diff --git a/.claude/commands/investigate-test-failure.md b/.claude/commands/investigate-test-failure.md new file mode 100644 index 00000000000..e9d6b2d2d2c --- /dev/null +++ b/.claude/commands/investigate-test-failure.md @@ -0,0 +1,49 @@ +--- +allowed-tools: Bash(python3:*), Bash(cargo run:*), Read, Grep, Glob, Bash(git add:*), Bash(git commit:*), Bash(cargo fmt:*), Bash(git diff:*), Task +--- + +# Investigate Test Failure + +Investigate why a specific test is failing and determine if it can be fixed or needs an issue. + +## Arguments +- `$ARGUMENTS`: Failed test identifier (e.g., `test_inspect.TestGetSourceBase.test_getsource_reload`) + +## Steps + +1. **Analyze failure cause** + - Read the test code + - Analyze failure message/traceback + - Check related RustPython code + +2. **Verify behavior in CPython** + - Run the test with `python3 -m unittest` to confirm expected behavior + - Document the expected output + +3. **Determine fix feasibility** + - **Simple fix** (import issues, small logic bugs): Fix code → Run `cargo fmt --all` → Pre-commit review → Commit + - **Complex fix** (major unimplemented features): Collect issue info and report to user + + **Pre-commit review process**: + - Run `git diff` to see the changes + - Use Task tool with `general-purpose` subagent to review: + - Compare implementation against cpython/ source code + - Verify the fix aligns with CPython behavior + - Check for any missed edge cases + - Proceed to commit only after review passes + +4. **For complex issues - Collect issue information** + Following `.github/ISSUE_TEMPLATE/report-incompatibility.md` format: + + - **Feature**: Description of missing/broken Python feature + - **Minimal reproduction code**: Smallest code that reproduces the issue + - **CPython behavior**: Result when running with python3 + - **RustPython behavior**: Result when running with cargo run + - **Python Documentation link**: Link to relevant CPython docs + + Report collected information to the user. Issue creation is done only upon user request. + + Example issue creation command: + ``` + gh issue create --template report-incompatibility.md --title "..." --body "..." + ``` diff --git a/.claude/commands/upgrade-pylib-next.md b/.claude/commands/upgrade-pylib-next.md new file mode 100644 index 00000000000..712b79433b3 --- /dev/null +++ b/.claude/commands/upgrade-pylib-next.md @@ -0,0 +1,33 @@ +--- +allowed-tools: Skill(upgrade-pylib), Bash(gh pr list:*) +--- + +# Upgrade Next Python Library + +Find the next Python library module ready for upgrade and run `/upgrade-pylib` for it. + +## Current TODO Status + +!`cargo run --release -- scripts/update_lib todo 2>/dev/null` + +## Open Upgrade PRs + +!`gh pr list --search "Update in:title" --json number,title --template '{{range .}}#{{.number}} {{.title}}{{"\n"}}{{end}}'` + +## Instructions + +From the TODO list above, find modules matching these patterns (in priority order): + +1. `[ ] [no deps]` - Modules with no dependencies (can be upgraded immediately) +2. `[ ] [0/n]` - Modules where all dependencies are already upgraded (e.g., `[0/3]`, `[0/5]`) + +These patterns indicate modules that are ready to upgrade without blocking dependencies. + +**Important**: Skip any modules that already have an open PR in the "Open Upgrade PRs" list above. + +**After identifying a suitable module**, run: +``` +/upgrade-pylib +``` + +If no modules match these criteria, inform the user that all eligible modules have dependencies that need to be upgraded first. diff --git a/.claude/commands/upgrade-pylib.md b/.claude/commands/upgrade-pylib.md new file mode 100644 index 00000000000..d54305d2616 --- /dev/null +++ b/.claude/commands/upgrade-pylib.md @@ -0,0 +1,157 @@ +--- +allowed-tools: Bash(git add:*), Bash(git commit:*), Bash(python3 scripts/update_lib quick:*), Bash(python3 scripts/update_lib auto-mark:*) +--- + +# Upgrade Python Library from CPython + +Upgrade a Python standard library module from CPython to RustPython. + +## Arguments +- `$ARGUMENTS`: Library name to upgrade (e.g., `inspect`, `asyncio`, `json`) + +## Important: Report Tool Issues First + +If during the upgrade process you encounter any of the following issues with `scripts/update_lib`: +- A feature that should be automated but isn't supported +- A bug or unexpected behavior in the tool +- Missing functionality that would make the upgrade easier + +**STOP the upgrade and report the issue first.** Describe: +1. What you were trying to do + - Library name + - The full command executed (e.g. python scripts/update_lib quick cpython/Lib/$ARGUMENTS.py) +2. What went wrong or what's missing +3. Expected vs actual behavior + +This helps improve the tooling for future upgrades. + +## Steps + +1. **Run quick upgrade with update_lib** + - Run: `python3 scripts/update_lib quick $ARGUMENTS` (module name) + - Or: `python3 scripts/update_lib quick cpython/Lib/$ARGUMENTS.py` (library file path) + - Or: `python3 scripts/update_lib quick cpython/Lib/$ARGUMENTS/` (library directory path) + - This will: + - Copy library files (delete existing `Lib/$ARGUMENTS.py` or `Lib/$ARGUMENTS/`, then copy from `cpython/Lib/`) + - Patch test files preserving existing RustPython markers + - Run tests and auto-mark new test failures (not regressions) + - Remove `@unittest.expectedFailure` from tests that now pass + - Create a git commit with the changes + - **Handle warnings**: If you see warnings like `WARNING: TestCFoo does not exist in remote file`, it means the class structure changed and markers couldn't be transferred automatically. These need to be manually restored in step 2 or added in step 3. + +2. **Review git diff and restore RUSTPYTHON-specific changes** + - Run `git diff Lib/test/test_$ARGUMENTS` to review all changes + - **Only restore changes that have explicit `RUSTPYTHON` comments**. Look for: + - `# XXX: RUSTPYTHON` or `# XXX RUSTPYTHON` - Comments marking RustPython-specific code modifications + - `# TODO: RUSTPYTHON` - Comments marking tests that need work + - Code changes with inline `# ... RUSTPYTHON` comments + - **Do NOT restore other diff changes** - these are likely upstream CPython changes, not RustPython-specific modifications + - When restoring, preserve the original context and formatting + +3. **Investigate test failures with subagent** + - First, get dependent tests using the deps command: + ``` + cargo run --release -- scripts/update_lib deps $ARGUMENTS + ``` + - Look for the line `- [ ] $ARGUMENTS: test_xxx test_yyy ...` to get the direct dependent tests + - Run those tests to collect failures: + ``` + cargo run --release -- -m test test_xxx test_yyy ... 2>&1 | grep -E "^(FAIL|ERROR):" + ``` + - For example, if deps output shows `- [ ] linecache: test_bdb test_inspect test_linecache test_traceback test_zipimport`, run: + ``` + cargo run --release -- -m test test_bdb test_inspect test_linecache test_traceback test_zipimport 2>&1 | grep -E "^(FAIL|ERROR):" + ``` + - For each failure, use the Task tool with `general-purpose` subagent to investigate: + - Subagent should follow the `/investigate-test-failure` skill workflow + - Pass the failed test identifier as the argument (e.g., `test_inspect.TestGetSourceBase.test_getsource_reload`) + - If subagent can fix the issue easily: fix and commit + - If complex issue: subagent collects issue info and reports back (issue creation on user request only) + - Using subagent prevents context pollution in the main conversation + +4. **Mark remaining test failures with auto-mark** + - Run: `python3 scripts/update_lib auto-mark Lib/test/test_$ARGUMENTS.py --mark-failure` + - Or for directory: `python3 scripts/update_lib auto-mark Lib/test/test_$ARGUMENTS/ --mark-failure` + - This will: + - Run tests and mark ALL failing tests with `@unittest.expectedFailure` + - Remove `@unittest.expectedFailure` from tests that now pass + - **Note**: The `--mark-failure` flag marks all failures including regressions. Review the changes before committing. + +5. **Handle panics manually** + - If any tests cause panics/crashes (not just assertion failures), they need `@unittest.skip` instead: + ```python + @unittest.skip("TODO: RUSTPYTHON; panics with 'index out of bounds'") + def test_crashes(self): + ... + ``` + - auto-mark cannot detect panics automatically - check the test output for crash messages + +6. **Handle class-specific failures** + - If a test fails only in the C implementation (TestCFoo) but passes in the Python implementation (TestPyFoo), or vice versa, move the marker to the specific subclass: + ```python + # Base class - no marker here + class TestFoo: + def test_something(self): + ... + + class TestPyFoo(TestFoo, PyTest): pass + + class TestCFoo(TestFoo, CTest): + # TODO: RUSTPYTHON + @unittest.expectedFailure + def test_something(self): + return super().test_something() + ``` + +7. **Commit the test fixes** + - Run: `git add -u && git commit -m "Mark failing tests"` + - This creates a separate commit for the test markers added in steps 2-6 + +## Example Usage +``` +# Using module names (recommended) +/upgrade-pylib inspect +/upgrade-pylib json +/upgrade-pylib asyncio + +# Using library paths (alternative) +/upgrade-pylib cpython/Lib/inspect.py +/upgrade-pylib cpython/Lib/json/ +``` + +## Example: Restoring RUSTPYTHON changes + +When git diff shows removed RUSTPYTHON-specific code like: +```diff +-# XXX RUSTPYTHON: we don't import _json as fresh since... +-cjson = import_helper.import_fresh_module('json') #, fresh=['_json']) ++cjson = import_helper.import_fresh_module('json', fresh=['_json']) +``` + +You should restore the RustPython version: +```python +# XXX RUSTPYTHON: we don't import _json as fresh since... +cjson = import_helper.import_fresh_module('json') #, fresh=['_json']) +``` + +## Notes +- The cpython/ directory should contain the CPython source that we're syncing from +- `scripts/update_lib` package handles patching and auto-marking: + - `quick` - Combined patch + auto-mark (recommended) + - `migrate` - Only migrate (patch), no test running + - `auto-mark` - Only run tests and mark failures + - `copy-lib` - Copy library files (not tests) +- The patching: + - Transfers `@unittest.expectedFailure` and `@unittest.skip` decorators with `TODO: RUSTPYTHON` markers + - Adds `import unittest # XXX: RUSTPYTHON` if needed for the decorators + - **Limitation**: If a class was restructured (e.g., method overrides removed), update_lib will warn and skip those markers +- The smart auto-mark: + - Marks NEW test failures automatically (tests that didn't exist before) + - Does NOT mark regressions (existing tests that now fail) - these are warnings + - Removes `@unittest.expectedFailure` from tests that now pass +- The script does NOT preserve all RustPython-specific changes - you must review `git diff` and restore them +- Common RustPython markers to look for: + - `# XXX: RUSTPYTHON` or `# XXX RUSTPYTHON` - Inline comments for code modifications + - `# TODO: RUSTPYTHON` - Test skip/failure markers + - Any code with `RUSTPYTHON` in comments that was removed in the diff +- **Important**: Not all changes in the git diff need to be restored. Only restore changes that have explicit `RUSTPYTHON` comments. Other changes are upstream CPython updates. diff --git a/.cspell.dict/cpython.txt b/.cspell.dict/cpython.txt index 2ac19a4fca0..c70e46cb207 100644 --- a/.cspell.dict/cpython.txt +++ b/.cspell.dict/cpython.txt @@ -1,3 +1,6 @@ +ADDOP +aftersign +argdefs argtypes asdl asname @@ -5,72 +8,180 @@ attro augassign badcert badsyntax +baseinfo basetype boolop +BUILDSTDLIB bxor +byteswap cached_tsver cadata cafile +calldepth +callinfo +callproc +capath +carg cellarg cellvar cellvars +cfield CLASSDEREF +classdict cmpop +codedepth +CODEUNIT +CONVFUNC +convparam +copyslot +cpucount +defaultdict denom +dictbytype DICTFLAG dictoffset distpoint +dynload elts +eofs +evalloop excepthandler +exceptiontable +fblock +fblocks +fdescr +ffi_argtypes +fielddesc +fieldlist fileutils finalbody +flowgraph formatfloat freevar freevars fromlist +getdict +getfunc +getiter +getsets +getslice +globalgetvar +HASARRAY +HASBITFIELD +HASPOINTER +HASSTRUCT +HASUNION heaptype HIGHRES +IFUNC IMMUTABLETYPE +INCREF +inlinedepth +inplace ismine +ISPOINTER +iteminfo Itertool keeped +kwnames kwonlyarg kwonlyargs lasti +libffi linearise +lineiterator +linetable +loadfast +localsplus +Lshift lsprof +MAXBLOCKS maxdepth +metavars +miscompiles mult multibytecodec +nameop +nconsts +newargs +newfree +NEWLOCALS newsemlockobject +nfrees nkwargs +nkwelts +Nondescriptor +noninteger +nops noraise +nseen +NSIGNALS numer +opname +opnames orelse +outparam +outparm +paramfunc +parg pathconfig patma +peepholer +phcount platstdlib posonlyarg posonlyargs prec preinitialized +pybuilddir +pycore pydecimal +Pyfunc +pylifecycle +pymain pyrepl +PYTHONTRACEMALLOC pythonw PYTHREAD_NAME releasebuffer +repr +resinfo +Rshift SA_ONSTACK +saveall +scls +setdict +setfunc +SETREF +setresult +setslice +SLOTDEFINED SOABI +SSLEOF stackdepth +staticbase stginfo +storefast stringlib structseq +subkwargs subparams +subscr +sval swappedbytes +templatelib ticketer +tmptype tok_oldval tvars +typeobject +typeparam +Typeparam +typeparams +typeslots unaryop +uncollectable +Unhandle unparse unparser VARKEYWORDS @@ -82,6 +193,7 @@ venvwlauncher venvwlaunchert wbits weakreflist +weakrefobject webpki withitem withs diff --git a/.cspell.dict/python-more.txt b/.cspell.dict/python-more.txt index a13f345eece..2dd31f8f579 100644 --- a/.cspell.dict/python-more.txt +++ b/.cspell.dict/python-more.txt @@ -5,6 +5,7 @@ aexit aiter anext anextawaitable +annotationlib appendleft argcount arrayiterator @@ -66,6 +67,7 @@ fnctl frombytes fromhex fromunicode +frozensets fset fspath fstring @@ -270,3 +272,29 @@ xmlcharrefreplace xoptions xopts yieldfrom +addcompare +altzone +classmethods +ctype +ctypes +genexpressions +getargs +getopt +getweakref +getweakrefs +inittab +Inittab +interpchannels +interpqueues +markupbase +mymodule +pydatetime +pyio +pymain +setprofileallthreads +settraceallthreads +sitebuiltins +subclassing +subpatterns +sysdict +weakrefset diff --git a/.cspell.dict/rust-more.txt b/.cspell.dict/rust-more.txt index ff2013e81a7..c3ebd61833a 100644 --- a/.cspell.dict/rust-more.txt +++ b/.cspell.dict/rust-more.txt @@ -28,6 +28,7 @@ hexf hexversion idents illumos +ilog indexmap insta keccak @@ -86,4 +87,7 @@ wasmer wasmtime widestring winapi +winresource winsock +bitvec +Bitvec diff --git a/.cspell.json b/.cspell.json index 89cde1ce775..ebed8664e58 100644 --- a/.cspell.json +++ b/.cspell.json @@ -69,10 +69,13 @@ "GetSet", "groupref", "internable", + "interps", "jitted", "jitting", "lossily", "makeunicodedata", + "microbenchmark", + "microbenchmarks", "miri", "notrace", "oparg", @@ -121,6 +124,7 @@ "uninit", "unraisable", "unresizable", + "varint", "wasi", "zelf", // unix @@ -139,7 +143,11 @@ "birthtime", "IFEXEC", // "stat" - "FIRMLINK" + "FIRMLINK", + // CPython internal names + "sysdict", + "settraceallthreads", + "setprofileallthreads" ], // flagWords - list of words to be always considered incorrect "flagWords": [ diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile index 339cdb69bbf..cdd54a47d5b 100644 --- a/.devcontainer/Dockerfile +++ b/.devcontainer/Dockerfile @@ -1,4 +1,4 @@ -FROM mcr.microsoft.com/vscode/devcontainers/rust:1-bullseye +FROM rust:bullseye # Install clang RUN apt-get update \ diff --git a/.gitattributes b/.gitattributes index d1dd182a9b0..71f807e8ed6 100644 --- a/.gitattributes +++ b/.gitattributes @@ -4,5 +4,65 @@ Cargo.lock linguist-generated vm/src/stdlib/ast/gen.rs linguist-generated -merge Lib/*.py text working-tree-encoding=UTF-8 eol=LF **/*.rs text working-tree-encoding=UTF-8 eol=LF -*.pck binary crates/rustpython_doc_db/src/*.inc.rs linguist-generated=true + +# Binary data types +*.aif binary +*.aifc binary +*.aiff binary +*.au binary +*.bmp binary +*.exe binary +*.icns binary +*.gif binary +*.ico binary +*.jpg binary +*.pck binary +*.pdf binary +*.png binary +*.psd binary +*.tar binary +*.wav binary +*.whl binary +*.zip binary + +# Text files that should not be subject to eol conversion +[attr]noeol -text + +Lib/test/cjkencodings/* noeol +Lib/test/tokenizedata/coding20731.py noeol +Lib/test/decimaltestdata/*.decTest noeol +Lib/test/test_email/data/*.txt noeol +Lib/test/xmltestdata/* noeol + +# Shell scripts should have LF even on Windows because of Cygwin +Lib/venv/scripts/common/activate text eol=lf +Lib/venv/scripts/posix/* text eol=lf + +# CRLF files +[attr]dos text eol=crlf + +# Language aware diff headers +# https://tekin.co.uk/2020/10/better-git-diff-output-for-ruby-python-elixir-and-more +# https://gist.github.com/tekin/12500956bd56784728e490d8cef9cb81 +*.css diff=css +*.html diff=html +*.py diff=python +*.md diff=markdown + +# Generated files +# https://github.com/github/linguist/blob/master/docs/overrides.md +# +# To always hide generated files in local diffs, mark them as binary: +# $ git config diff.generated.binary true +# +[attr]generated linguist-generated=true diff=generated + +Lib/_opcode_metadata.py generated +Lib/keyword.py generated +Lib/idlelib/help.html generated +Lib/test/certdata/*.pem generated +Lib/test/certdata/*.0 generated +Lib/test/levenshtein_examples.json generated +Lib/test/test_stable_abi_ctypes.py generated +Lib/token.py generated diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index a873aa2d29d..6a5ef501c1f 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -85,6 +85,7 @@ env: test_math test_operator test_ordered_dict + test_pep646_syntax test_pow test_raise test_richcmp @@ -99,14 +100,22 @@ env: test_subclassinit test_super test_syntax + test_tstring test_tuple test_types test_unary test_unpack + test_unpack_ex test_weakref test_yield_from + ENV_POLLUTING_TESTS_COMMON: >- + ENV_POLLUTING_TESTS_LINUX: >- + ENV_POLLUTING_TESTS_MACOS: >- + ENV_POLLUTING_TESTS_WINDOWS: >- # Python version targeted by the CI. - PYTHON_VERSION: "3.13.1" + PYTHON_VERSION: "3.14.2" + X86_64_PC_WINDOWS_MSVC_OPENSSL_LIB_DIR: C:\Program Files\OpenSSL\lib\VC\x64\MD + X86_64_PC_WINDOWS_MSVC_OPENSSL_INCLUDE_DIR: C:\Program Files\OpenSSL\include jobs: rust_tests: @@ -114,27 +123,19 @@ jobs: env: RUST_BACKTRACE: full name: Run rust tests - runs-on: ${{ matrix.os }} + runs-on: ${{ matrix.os }} timeout-minutes: 45 strategy: matrix: - os: [macos-latest, ubuntu-latest, windows-latest] + os: [macos-latest, ubuntu-latest, windows-2025] fail-fast: false steps: - - uses: actions/checkout@v6.0.1 + - uses: actions/checkout@v6.0.2 - uses: dtolnay/rust-toolchain@stable with: components: clippy - uses: Swatinem/rust-cache@v2 - # Only for OpenSSL builds - # - name: Set up the Windows environment - # shell: bash - # run: | - # git config --system core.longpaths true - # cargo install --target-dir=target -v cargo-vcpkg - # cargo vcpkg -v build - # if: runner.os == 'Windows' - name: Set up the Mac environment run: brew install autoconf automake libtool if: runner.os == 'macOS' @@ -153,8 +154,7 @@ jobs: run: cargo check ${{ env.CARGO_ARGS }} - name: Test openssl build - run: - cargo build --no-default-features --features ssl-openssl + run: cargo build --no-default-features --features ssl-openssl if: runner.os == 'Linux' # - name: Install tk-dev for tkinter build @@ -166,11 +166,17 @@ jobs: # if: runner.os == 'Linux' - name: Test example projects - run: + run: | cargo run --manifest-path example_projects/barebone/Cargo.toml cargo run --manifest-path example_projects/frozen_stdlib/Cargo.toml if: runner.os == 'Linux' + - name: run update_lib tests + run: cargo run -- -m unittest discover -s scripts/update_lib/tests -v + env: + PYTHONPATH: scripts + if: runner.os == 'Linux' + - name: prepare Intel MacOS build uses: dtolnay/rust-toolchain@stable with: @@ -194,7 +200,7 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 30 steps: - - uses: actions/checkout@v6.0.1 + - uses: actions/checkout@v6.0.2 - uses: dtolnay/rust-toolchain@stable with: target: i686-unknown-linux-gnu @@ -208,8 +214,19 @@ jobs: with: target: aarch64-linux-android + - name: Setup Android NDK + id: setup-ndk + uses: nttld/setup-ndk@v1 + with: + ndk-version: r27 + add-to-path: true + - name: Check compilation for android run: cargo check --target aarch64-linux-android ${{ env.CARGO_ARGS_NO_SSL }} + env: + CC_aarch64_linux_android: ${{ steps.setup-ndk.outputs.ndk-path }}/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang + AR_aarch64_linux_android: ${{ steps.setup-ndk.outputs.ndk-path }}/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar + CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER: ${{ steps.setup-ndk.outputs.ndk-path }}/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang - uses: dtolnay/rust-toolchain@stable with: @@ -241,6 +258,13 @@ jobs: - name: Check compilation for freeBSD run: cargo check --target x86_64-unknown-freebsd ${{ env.CARGO_ARGS_NO_SSL }} + - uses: dtolnay/rust-toolchain@stable + with: + target: wasm32-wasip2 + + - name: Check compilation for wasip2 + run: cargo check --target wasm32-wasip2 ${{ env.CARGO_ARGS_NO_SSL }} + # - name: Prepare repository for redox compilation # run: bash scripts/redox/uncomment-cargo.sh # - name: Check compilation for Redox @@ -254,26 +278,18 @@ jobs: env: RUST_BACKTRACE: full name: Run snippets and cpython tests - runs-on: ${{ matrix.os }} + runs-on: ${{ matrix.os }} strategy: matrix: - os: [macos-latest, ubuntu-latest, windows-latest] + os: [macos-latest, ubuntu-latest, windows-2025] fail-fast: false steps: - - uses: actions/checkout@v6.0.1 + - uses: actions/checkout@v6.0.2 - uses: dtolnay/rust-toolchain@stable - uses: Swatinem/rust-cache@v2 - - uses: actions/setup-python@v6.1.0 + - uses: actions/setup-python@v6.2.0 with: python-version: ${{ env.PYTHON_VERSION }} - # Only for OpenSSL builds - # - name: Set up the Windows environment - # shell: bash - # run: | - # git config --system core.longpaths true - # cargo install cargo-vcpkg - # cargo vcpkg build - # if: runner.os == 'Windows' - name: Set up the Mac environment run: brew install autoconf automake libtool openssl@3 if: runner.os == 'macOS' @@ -283,7 +299,7 @@ jobs: - name: build rustpython run: cargo build --release --verbose --features=threading ${{ env.CARGO_ARGS }},jit if: runner.os != 'macOS' - - uses: actions/setup-python@v6.1.0 + - uses: actions/setup-python@v6.2.0 with: python-version: ${{ env.PYTHON_VERSION }} - name: run snippets @@ -292,26 +308,116 @@ jobs: - if: runner.os == 'Linux' name: run cpython platform-independent tests - run: - target/release/rustpython -m test -j 1 -u all --slowest --fail-env-changed -v ${{ env.PLATFORM_INDEPENDENT_TESTS }} - timeout-minutes: 35 + env: + RUSTPYTHON_SKIP_ENV_POLLUTERS: true + run: target/release/rustpython -m test -j 1 -u all --slowest --fail-env-changed --timeout 600 -v ${{ env.PLATFORM_INDEPENDENT_TESTS }} + timeout-minutes: 45 - if: runner.os == 'Linux' name: run cpython platform-dependent tests (Linux) - run: target/release/rustpython -m test -j 1 -u all --slowest --fail-env-changed -v -x ${{ env.PLATFORM_INDEPENDENT_TESTS }} - timeout-minutes: 35 + env: + RUSTPYTHON_SKIP_ENV_POLLUTERS: true + run: target/release/rustpython -m test -j 1 -u all --slowest --fail-env-changed --timeout 600 -v -x ${{ env.PLATFORM_INDEPENDENT_TESTS }} + timeout-minutes: 45 - if: runner.os == 'macOS' name: run cpython platform-dependent tests (MacOS) - run: target/release/rustpython -m test -j 1 --slowest --fail-env-changed -v -x ${{ env.PLATFORM_INDEPENDENT_TESTS }} - timeout-minutes: 35 + env: + RUSTPYTHON_SKIP_ENV_POLLUTERS: true + run: target/release/rustpython -m test -j 1 --slowest --fail-env-changed --timeout 600 -v -x ${{ env.PLATFORM_INDEPENDENT_TESTS }} + timeout-minutes: 45 - if: runner.os == 'Windows' name: run cpython platform-dependent tests (windows partial - fixme) - run: - target/release/rustpython -m test -j 1 --slowest --fail-env-changed -v -x ${{ env.PLATFORM_INDEPENDENT_TESTS }} ${{ env.WINDOWS_SKIPS }} + env: + RUSTPYTHON_SKIP_ENV_POLLUTERS: true + run: target/release/rustpython -m test -j 1 --slowest --fail-env-changed --timeout 600 -v -x ${{ env.PLATFORM_INDEPENDENT_TESTS }} ${{ env.WINDOWS_SKIPS }} timeout-minutes: 45 + - if: runner.os == 'Linux' + name: run cpython tests to check if env polluters have stopped polluting (Common/Linux) + shell: bash + run: | + for thing in ${{ env.ENV_POLLUTING_TESTS_COMMON }} ${{ env.ENV_POLLUTING_TESTS_LINUX }}; do + for i in $(seq 1 10); do + set +e + target/release/rustpython -m test -j 1 --slowest --fail-env-changed --timeout 600 -v ${thing} + exit_code=$? + set -e + if [ ${exit_code} -eq 3 ]; then + echo "Test ${thing} polluted the environment on attempt ${i}." + break + fi + done + if [ ${exit_code} -ne 3 ]; then + echo "Test ${thing} is no longer polluting the environment after ${i} attempts!" + echo "Please remove ${thing} from either ENV_POLLUTING_TESTS_COMMON or ENV_POLLUTING_TESTS_LINUX in '.github/workflows/ci.yaml'." + echo "Please also remove the skip decorators that include the word 'POLLUTERS' in ${thing}." + if [ ${exit_code} -ne 0 ]; then + echo "Test ${thing} failed with exit code ${exit_code}." + echo "Please investigate which test item in ${thing} is failing and either mark it as an expected failure or a skip." + fi + exit 1 + fi + done + timeout-minutes: 15 + + - if: runner.os == 'macOS' + name: run cpython tests to check if env polluters have stopped polluting (Common/macOS) + shell: bash + run: | + for thing in ${{ env.ENV_POLLUTING_TESTS_COMMON }} ${{ env.ENV_POLLUTING_TESTS_MACOS }}; do + for i in $(seq 1 10); do + set +e + target/release/rustpython -m test -j 1 --slowest --fail-env-changed --timeout 600 -v ${thing} + exit_code=$? + set -e + if [ ${exit_code} -eq 3 ]; then + echo "Test ${thing} polluted the environment on attempt ${i}." + break + fi + done + if [ ${exit_code} -ne 3 ]; then + echo "Test ${thing} is no longer polluting the environment after ${i} attempts!" + echo "Please remove ${thing} from either ENV_POLLUTING_TESTS_COMMON or ENV_POLLUTING_TESTS_MACOS in '.github/workflows/ci.yaml'." + echo "Please also remove the skip decorators that include the word 'POLLUTERS' in ${thing}." + if [ ${exit_code} -ne 0 ]; then + echo "Test ${thing} failed with exit code ${exit_code}." + echo "Please investigate which test item in ${thing} is failing and either mark it as an expected failure or a skip." + fi + exit 1 + fi + done + timeout-minutes: 15 + + - if: runner.os == 'Windows' + name: run cpython tests to check if env polluters have stopped polluting (Common/windows) + shell: bash + run: | + for thing in ${{ env.ENV_POLLUTING_TESTS_COMMON }} ${{ env.ENV_POLLUTING_TESTS_WINDOWS }}; do + for i in $(seq 1 10); do + set +e + target/release/rustpython -m test -j 1 --slowest --fail-env-changed --timeout 600 -v ${thing} + exit_code=$? + set -e + if [ ${exit_code} -eq 3 ]; then + echo "Test ${thing} polluted the environment on attempt ${i}." + break + fi + done + if [ ${exit_code} -ne 3 ]; then + echo "Test ${thing} is no longer polluting the environment after ${i} attempts!" + echo "Please remove ${thing} from either ENV_POLLUTING_TESTS_COMMON or ENV_POLLUTING_TESTS_WINDOWS in '.github/workflows/ci.yaml'." + echo "Please also remove the skip decorators that include the word 'POLLUTERS' in ${thing}." + if [ ${exit_code} -ne 0 ]; then + echo "Test ${thing} failed with exit code ${exit_code}." + echo "Please investigate which test item in ${thing} is failing and either mark it as an expected failure or a skip." + fi + exit 1 + fi + done + timeout-minutes: 15 + - if: runner.os != 'Windows' name: check that --install-pip succeeds run: | @@ -327,27 +433,41 @@ jobs: run: | target/release/rustpython -m venv testvenv testvenv/bin/rustpython -m pip install wheel - - name: Check whats_left is not broken - run: python -I whats_left.py + - if: runner.os != 'macOS' + name: Check whats_left is not broken + shell: bash + run: python -I scripts/whats_left.py --no-default-features --features "$(sed -e 's/--[^ ]*//g' <<< "${{ env.CARGO_ARGS }}" | tr -d '[:space:]'),threading,jit" + - if: runner.os == 'macOS' # TODO fix jit on macOS + name: Check whats_left is not broken (macOS) + shell: bash + run: python -I scripts/whats_left.py --no-default-features --features "$(sed -e 's/--[^ ]*//g' <<< "${{ env.CARGO_ARGS }}" | tr -d '[:space:]'),threading" # no jit on macOS for now lint: name: Check Rust code with clippy runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6.0.1 + - uses: actions/checkout@v6.0.2 - uses: dtolnay/rust-toolchain@stable with: - components: clippy + components: clippy + - name: run clippy on wasm run: cargo clippy --manifest-path=crates/wasm/Cargo.toml -- -Dwarnings - name: Ensure docs generate no warnings - run: cargo doc + run: cargo doc --locked + + - name: Ensure Lib/_opcode_metadata is updated + run: | + python scripts/generate_opcode_metadata.py + if [ -n "$(git status --porcelain)" ]; then + exit 1 + fi - name: Install ruff - uses: astral-sh/ruff-action@57714a7c8a2e59f32539362ba31877a1957dded1 # v3.5.1 + uses: astral-sh/ruff-action@4919ec5cf1f49eff0871dbcea0da843445b837e6 # v3.6.1 with: - version: "0.14.9" + version: "0.14.11" args: "--version" - run: ruff check --diff @@ -366,7 +486,7 @@ jobs: - name: spell checker uses: streetsidesoftware/cspell-action@v8 with: - files: '**/*.rs' + files: "**/*.rs" incremental_files_only: true miri: @@ -377,21 +497,21 @@ jobs: env: NIGHTLY_CHANNEL: nightly steps: - - uses: actions/checkout@v6.0.1 + - uses: actions/checkout@v6.0.2 - uses: dtolnay/rust-toolchain@master with: - toolchain: ${{ env.NIGHTLY_CHANNEL }} - components: miri + toolchain: ${{ env.NIGHTLY_CHANNEL }} + components: miri - uses: Swatinem/rust-cache@v2 - name: Run tests under miri run: cargo +${{ env.NIGHTLY_CHANNEL }} miri test -p rustpython-vm -- miri_test env: - # miri-ignore-leaks because the type-object circular reference means that there will always be - # a memory leak, at least until we have proper cyclic gc - MIRIFLAGS: '-Zmiri-ignore-leaks' + # miri-ignore-leaks because the type-object circular reference means that there will always be + # a memory leak, at least until we have proper cyclic gc + MIRIFLAGS: "-Zmiri-ignore-leaks" wasm: if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip:ci') }} @@ -399,7 +519,7 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 30 steps: - - uses: actions/checkout@v6.0.1 + - uses: actions/checkout@v6.0.2 - uses: dtolnay/rust-toolchain@stable - uses: Swatinem/rust-cache@v2 @@ -410,7 +530,7 @@ jobs: wget https://github.com/mozilla/geckodriver/releases/download/v0.36.0/geckodriver-v0.36.0-linux64.tar.gz mkdir geckodriver tar -xzf geckodriver-v0.36.0-linux64.tar.gz -C geckodriver - - uses: actions/setup-python@v6.1.0 + - uses: actions/setup-python@v6.2.0 with: python-version: ${{ env.PYTHON_VERSION }} - run: python -m pip install -r requirements.txt @@ -462,7 +582,7 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 30 steps: - - uses: actions/checkout@v6.0.1 + - uses: actions/checkout@v6.0.2 - uses: dtolnay/rust-toolchain@stable with: target: wasm32-wasip1 diff --git a/.github/workflows/comment-commands.yml b/.github/workflows/comment-commands.yml index d1a457c73e6..3f3402270ea 100644 --- a/.github/workflows/comment-commands.yml +++ b/.github/workflows/comment-commands.yml @@ -7,7 +7,7 @@ on: jobs: issue_assign: if: (!github.event.issue.pull_request) && github.event.comment.body == 'take' - runs-on: ubuntu-latest + runs-on: ubuntu-slim concurrency: group: ${{ github.actor }}-issue-assign diff --git a/.github/workflows/cron-ci.yaml b/.github/workflows/cron-ci.yaml index 59d664e0ea1..0a546595a8c 100644 --- a/.github/workflows/cron-ci.yaml +++ b/.github/workflows/cron-ci.yaml @@ -5,12 +5,15 @@ on: push: paths: - .github/workflows/cron-ci.yaml + pull_request: + paths: + - .github/workflows/cron-ci.yaml name: Periodic checks/tasks env: - CARGO_ARGS: --no-default-features --features stdlib,importlib,encodings,ssl,jit - PYTHON_VERSION: "3.13.1" + CARGO_ARGS: --no-default-features --features stdlib,importlib,encodings,ssl-rustls,jit + PYTHON_VERSION: "3.14.2" jobs: # codecov collects code coverage data from the rust tests, python snippets and python test suite. @@ -21,15 +24,15 @@ jobs: # Disable this scheduled job when running on a fork. if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }} steps: - - uses: actions/checkout@v6.0.1 + - uses: actions/checkout@v6.0.2 - uses: dtolnay/rust-toolchain@stable - uses: taiki-e/install-action@cargo-llvm-cov - - uses: actions/setup-python@v6.1.0 + - uses: actions/setup-python@v6.2.0 with: python-version: ${{ env.PYTHON_VERSION }} - run: sudo apt-get update && sudo apt-get -y install lcov - name: Run cargo-llvm-cov with Rust tests. - run: cargo llvm-cov --no-report --workspace --exclude rustpython_wasm --verbose --no-default-features --features stdlib,importlib,encodings,ssl,jit + run: cargo llvm-cov --no-report --workspace --exclude rustpython_wasm --exclude rustpython-compiler-source --exclude rustpython-venvlauncher --verbose --no-default-features --features stdlib,importlib,encodings,ssl-rustls,jit - name: Run cargo-llvm-cov with Python snippets. run: python scripts/cargo-llvm-cov.py continue-on-error: true @@ -39,6 +42,7 @@ jobs: - name: Prepare code coverage data run: cargo llvm-cov report --lcov --output-path='codecov.lcov' - name: Upload to Codecov + if: ${{ github.event_name != 'pull_request' }} uses: codecov/codecov-action@v5 with: file: ./codecov.lcov @@ -49,7 +53,7 @@ jobs: # Disable this scheduled job when running on a fork. if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }} steps: - - uses: actions/checkout@v6.0.1 + - uses: actions/checkout@v6.0.2 - uses: dtolnay/rust-toolchain@stable - name: build rustpython run: cargo build --release --verbose @@ -58,6 +62,7 @@ jobs: env: RUSTPYTHONPATH: ${{ github.workspace }}/Lib - name: upload tests data to the website + if: ${{ github.event_name != 'pull_request' }} env: SSHKEY: ${{ secrets.ACTIONS_TESTS_DATA_DEPLOY_KEY }} GITHUB_ACTOR: ${{ github.actor }} @@ -80,20 +85,21 @@ jobs: # Disable this scheduled job when running on a fork. if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }} steps: - - uses: actions/checkout@v6.0.1 + - uses: actions/checkout@v6.0.2 - uses: dtolnay/rust-toolchain@stable - - uses: actions/setup-python@v6.1.0 + - uses: actions/setup-python@v6.2.0 with: python-version: ${{ env.PYTHON_VERSION }} - name: build rustpython run: cargo build --release --verbose - name: Collect what is left data run: | - chmod +x ./whats_left.py - ./whats_left.py --features "ssl,sqlite" > whats_left.temp + chmod +x ./scripts/whats_left.py + ./scripts/whats_left.py --features "ssl,sqlite" > whats_left.temp env: RUSTPYTHONPATH: ${{ github.workspace }}/Lib - name: Upload data to the website + if: ${{ github.event_name != 'pull_request' }} env: SSHKEY: ${{ secrets.ACTIONS_TESTS_DATA_DEPLOY_KEY }} GITHUB_ACTOR: ${{ github.actor }} @@ -137,11 +143,11 @@ jobs: # Disable this scheduled job when running on a fork. if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }} steps: - - uses: actions/checkout@v6.0.1 + - uses: actions/checkout@v6.0.2 - uses: dtolnay/rust-toolchain@stable - - uses: actions/setup-python@v6.1.0 + - uses: actions/setup-python@v6.2.0 with: - python-version: 3.9 + python-version: ${{ env.PYTHON_VERSION }} - run: cargo install cargo-criterion - name: build benchmarks run: cargo build --release --benches @@ -162,6 +168,7 @@ jobs: mv reports/* . rmdir reports - name: upload benchmark data to the website + if: ${{ github.event_name != 'pull_request' }} env: SSHKEY: ${{ secrets.ACTIONS_TESTS_DATA_DEPLOY_KEY }} run: | @@ -173,7 +180,11 @@ jobs: cd website rm -rf ./assets/criterion cp -r ../target/criterion ./assets/criterion - git add ./assets/criterion + printf '{\n "generated_at": "%s",\n "rustpython_commit": "%s",\n "rustpython_ref": "%s"\n}\n' \ + "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + "${{ github.sha }}" \ + "${{ github.ref_name }}" > ./_data/criterion-metadata.json + git add ./assets/criterion ./_data/criterion-metadata.json if git -c user.name="Github Actions" -c user.email="actions@github.com" commit -m "Update benchmark results"; then git push fi diff --git a/.github/workflows/lib-deps-check.yaml b/.github/workflows/lib-deps-check.yaml new file mode 100644 index 00000000000..a4b7128d830 --- /dev/null +++ b/.github/workflows/lib-deps-check.yaml @@ -0,0 +1,120 @@ +name: Lib Dependencies Check + +on: + pull_request_target: + types: [opened, synchronize, reopened] + paths: + - 'Lib/**' + +concurrency: + group: lib-deps-${{ github.event.pull_request.number }} + cancel-in-progress: true + +jobs: + check_deps: + permissions: + pull-requests: write + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Checkout base branch + uses: actions/checkout@v6.0.2 + with: + # Use base branch for scripts (security: don't run PR code with elevated permissions) + ref: ${{ github.event.pull_request.base.ref }} + fetch-depth: 0 + + - name: Fetch PR head + run: | + git fetch origin ${{ github.event.pull_request.head.sha }} + + - name: Checkout PR Lib files + run: | + # Checkout only Lib/ directory from PR head for accurate comparison + git checkout ${{ github.event.pull_request.head.sha }} -- Lib/ + + - name: Checkout CPython + run: | + git clone --depth 1 --branch v3.14.2 https://github.com/python/cpython.git cpython + + - name: Get changed Lib files + id: changed-files + run: | + # Get the list of changed files under Lib/ + changed=$(git diff --name-only ${{ github.event.pull_request.base.sha }} ${{ github.event.pull_request.head.sha }} -- 'Lib/*.py' 'Lib/**/*.py' | head -50) + echo "Changed files:" + echo "$changed" + + # Extract unique module names + modules="" + for file in $changed; do + if [[ "$file" == Lib/test/* ]]; then + # Test files: Lib/test/test_pydoc.py -> test_pydoc, Lib/test/test_pydoc/foo.py -> test_pydoc + module=$(echo "$file" | sed -E 's|^Lib/test/||; s|\.py$||; s|/.*||') + # Skip non-test files in test/ (e.g., support.py, __init__.py) + if [[ ! "$module" == test_* ]]; then + continue + fi + else + # Lib files: Lib/foo.py -> foo, Lib/foo/__init__.py -> foo + module=$(echo "$file" | sed -E 's|^Lib/||; s|/__init__\.py$||; s|\.py$||; s|/.*||') + fi + if [[ -n "$module" && ! " $modules " =~ " $module " ]]; then + modules="$modules $module" + fi + done + + modules=$(echo "$modules" | xargs) # trim whitespace + echo "Detected modules: $modules" + echo "modules=$modules" >> $GITHUB_OUTPUT + + - name: Setup Python + if: steps.changed-files.outputs.modules != '' + uses: actions/setup-python@v6.2.0 + with: + python-version: "3.12" + + - name: Run deps check + if: steps.changed-files.outputs.modules != '' + id: deps-check + run: | + # Run deps for all modules at once + python scripts/update_lib deps ${{ steps.changed-files.outputs.modules }} --depth 2 > /tmp/deps_output.txt 2>&1 || true + + # Read output for GitHub Actions + echo "deps_output<> $GITHUB_OUTPUT + cat /tmp/deps_output.txt >> $GITHUB_OUTPUT + echo "EOF" >> $GITHUB_OUTPUT + + # Check if there's any meaningful output + if [ -s /tmp/deps_output.txt ]; then + echo "has_output=true" >> $GITHUB_OUTPUT + else + echo "has_output=false" >> $GITHUB_OUTPUT + fi + + - name: Post comment + if: steps.deps-check.outputs.has_output == 'true' + uses: marocchino/sticky-pull-request-comment@v2 + with: + header: lib-deps-check + number: ${{ github.event.pull_request.number }} + recreate: true + message: | + ## 📦 Library Dependencies + + The following Lib/ modules were modified. Here are their dependencies: + + ${{ steps.deps-check.outputs.deps_output }} + + **Legend:** + - `[+]` path exists in CPython + - `[x]` up-to-date, `[ ]` outdated + + - name: Remove comment if no Lib changes + if: steps.changed-files.outputs.modules == '' + uses: marocchino/sticky-pull-request-comment@v2 + with: + header: lib-deps-check + number: ${{ github.event.pull_request.number }} + delete: true diff --git a/.github/workflows/pr-auto-commit.yaml b/.github/workflows/pr-auto-commit.yaml index 0cbd2bfefb0..e27cfe2ce16 100644 --- a/.github/workflows/pr-auto-commit.yaml +++ b/.github/workflows/pr-auto-commit.yaml @@ -14,7 +14,6 @@ concurrency: jobs: auto_format: - if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip:ci') }} permissions: contents: write pull-requests: write @@ -22,7 +21,7 @@ jobs: timeout-minutes: 60 steps: - name: Checkout PR branch - uses: actions/checkout@v6.0.1 + uses: actions/checkout@v6.0.2 with: ref: ${{ github.event.pull_request.head.sha }} repository: ${{ github.event.pull_request.head.repo.full_name }} @@ -34,41 +33,77 @@ jobs: with: components: rustfmt + - name: Configure git + run: | + git config user.name "github-actions[bot]" + git config user.email "github-actions[bot]@users.noreply.github.com" + echo "" > /tmp/committed_commands.txt + - name: Run cargo fmt run: | echo "Running cargo fmt --all on PR #${{ github.event.pull_request.number }}" cargo fmt --all + if [ -n "$(git status --porcelain)" ]; then + git add -u + git commit -m "Auto-format: cargo fmt --all" + echo "- \`cargo fmt --all\`" >> /tmp/committed_commands.txt + fi - name: Install ruff - uses: astral-sh/ruff-action@57714a7c8a2e59f32539362ba31877a1957dded1 # v3.5.1 + uses: astral-sh/ruff-action@4919ec5cf1f49eff0871dbcea0da843445b837e6 # v3.6.1 with: - version: "0.14.9" + version: "0.14.11" args: "--version" - - run: ruff format - - run: ruff check --select I --fix + - name: Run ruff format + run: | + ruff format + if [ -n "$(git status --porcelain)" ]; then + git add -u + git commit -m "Auto-format: ruff format" + echo "- \`ruff format\`" >> /tmp/committed_commands.txt + fi - - name: Configure git + - name: Run ruff check import sorting run: | - git config user.name "github-actions[bot]" - git config user.email "github-actions[bot]@users.noreply.github.com" + ruff check --select I --fix + if [ -n "$(git status --porcelain)" ]; then + git add -u + git commit -m "Auto-format: ruff check --select I --fix" + echo "- \`ruff check --select I --fix\`" >> /tmp/committed_commands.txt + fi + + - name: Run generate_opcode_metadata.py + run: | + python scripts/generate_opcode_metadata.py + if [ -n "$(git status --porcelain)" ]; then + git add -u + git commit -m "Auto-generate: generate_opcode_metadata.py" + echo "- \`python scripts/generate_opcode_metadata.py\`" >> /tmp/committed_commands.txt + fi - name: Check for changes id: check-changes run: | - if [ -n "$(git status --porcelain)" ]; then + if [ "$(git rev-parse HEAD)" != "${{ github.event.pull_request.head.sha }}" ]; then echo "has_changes=true" >> $GITHUB_OUTPUT else echo "has_changes=false" >> $GITHUB_OUTPUT fi - - name: Commit and push formatting changes + - name: Push formatting changes if: steps.check-changes.outputs.has_changes == 'true' run: | - git add -u - git commit -m "Auto-format: cargo fmt --all" git push origin HEAD:${{ github.event.pull_request.head.ref }} + - name: Read committed commands + id: committed-commands + if: steps.check-changes.outputs.has_changes == 'true' + run: | + echo "list<> $GITHUB_OUTPUT + cat /tmp/committed_commands.txt >> $GITHUB_OUTPUT + echo "EOF" >> $GITHUB_OUTPUT + - name: Comment on PR if: steps.check-changes.outputs.has_changes == 'true' uses: marocchino/sticky-pull-request-comment@v2 @@ -77,7 +112,8 @@ jobs: message: | **Code has been automatically formatted** - The code in this PR has been formatted using `cargo fmt --all`. + The code in this PR has been formatted using: + ${{ steps.committed-commands.outputs.list }} Please pull the latest changes before pushing again: ```bash git pull origin ${{ github.event.pull_request.head.ref }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 5ad2cf3f97a..9a6d0ad9838 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -17,6 +17,8 @@ permissions: env: CARGO_ARGS: --no-default-features --features stdlib,importlib,encodings,sqlite,ssl + X86_64_PC_WINDOWS_MSVC_OPENSSL_LIB_DIR: C:\Program Files\OpenSSL\lib\VC\x64\MD + X86_64_PC_WINDOWS_MSVC_OPENSSL_INCLUDE_DIR: C:\Program Files\OpenSSL\include jobs: build: @@ -42,28 +44,21 @@ jobs: target: aarch64-apple-darwin # - runner: macos-latest # target: x86_64-apple-darwin - - runner: windows-latest + - runner: windows-2025 target: x86_64-pc-windows-msvc -# - runner: windows-latest +# - runner: windows-2025 # target: i686-pc-windows-msvc -# - runner: windows-latest +# - runner: windows-2025 # target: aarch64-pc-windows-msvc fail-fast: false steps: - - uses: actions/checkout@v6.0.1 + - uses: actions/checkout@v6.0.2 - uses: dtolnay/rust-toolchain@stable - uses: cargo-bins/cargo-binstall@main - name: Set up Environment shell: bash run: rustup target add ${{ matrix.platform.target }} - - name: Set up Windows Environment - shell: bash - run: | - git config --global core.longpaths true - cargo install --target-dir=target -v cargo-vcpkg - cargo vcpkg -v build - if: runner.os == 'Windows' - name: Set up MacOS Environment run: brew install autoconf automake libtool if: runner.os == 'macOS' @@ -93,7 +88,7 @@ jobs: # Disable this scheduled job when running on a fork. if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }} steps: - - uses: actions/checkout@v6.0.1 + - uses: actions/checkout@v6.0.2 - uses: dtolnay/rust-toolchain@stable with: targets: wasm32-wasip1 @@ -144,7 +139,7 @@ jobs: if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }} needs: [build, build-wasm] steps: - - uses: actions/checkout@v6.0.1 + - uses: actions/checkout@v6.0.2 - name: Download Binary Artifacts uses: actions/download-artifact@v7.0.0 diff --git a/.github/workflows/update-doc-db.yml b/.github/workflows/update-doc-db.yml index c580e7d0eaf..1fd3b930985 100644 --- a/.github/workflows/update-doc-db.yml +++ b/.github/workflows/update-doc-db.yml @@ -9,7 +9,7 @@ on: python-version: description: Target python version to generate doc db for type: string - default: "3.13.9" + default: "3.14.2" ref: description: Branch to commit to (leave empty for current branch) type: string @@ -29,13 +29,13 @@ jobs: - windows-latest - macos-latest steps: - - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 + - uses: actions/checkout@0c366fd6a839edf440554fa01a7085ccba70ac98 # v6.0.1 with: persist-credentials: false sparse-checkout: | crates/doc - - uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v6.1.0 + - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 with: python-version: ${{ inputs.python-version }} @@ -54,7 +54,7 @@ jobs: runs-on: ubuntu-latest needs: generate steps: - - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 + - uses: actions/checkout@0c366fd6a839edf440554fa01a7085ccba70ac98 # v6.0.1 with: ref: ${{ inputs.ref || github.ref }} token: ${{ secrets.AUTO_COMMIT_PAT }} diff --git a/.gitignore b/.gitignore index fea93ace80a..c03ae1a997e 100644 --- a/.gitignore +++ b/.gitignore @@ -26,3 +26,5 @@ Lib/site-packages/* !Lib/site-packages/README.txt Lib/test/data/* !Lib/test/data/README +cpython/ + diff --git a/.github/copilot-instructions.md b/AGENTS.md similarity index 82% rename from .github/copilot-instructions.md rename to AGENTS.md index a2ab43a695c..89326ef35ad 100644 --- a/.github/copilot-instructions.md +++ b/AGENTS.md @@ -4,7 +4,7 @@ This document provides guidelines for working with GitHub Copilot when contribut ## Project Overview -RustPython is a Python 3 interpreter written in Rust, implementing Python 3.13.0+ compatibility. The project aims to provide: +RustPython is a Python 3 interpreter written in Rust, implementing Python 3.14.0+ compatibility. The project aims to provide: - A complete Python-3 environment entirely in Rust (not CPython bindings) - A clean implementation without compatibility hacks @@ -30,6 +30,14 @@ RustPython is a Python 3 interpreter written in Rust, implementing Python 3.13.0 - `jit/` - Experimental JIT compiler implementation - `pylib/` - Python standard library packaging (do not modify this directory directly - its contents are generated automatically) +## AI Agent Rules + +**CRITICAL: Git Operations** +- NEVER create pull requests directly without explicit user permission +- NEVER push commits to remote without explicit user permission +- Always ask the user before performing any git operations that affect the remote repository +- Commits can be created locally when requested, but pushing and PR creation require explicit approval + ## Important Development Notes ### Running Python Code @@ -87,11 +95,14 @@ rm -r target/debug/build/rustpython-* && find . | grep -E "\.pyc$" | xargs rm -r # Run Rust unit tests cargo test --workspace --exclude rustpython_wasm -# Run Python snippets tests +# Run Python snippets tests (debug mode recommended for faster compilation) +cargo run -- extra_tests/snippets/builtin_bytes.py + +# Run all Python snippets tests with pytest cd extra_tests pytest -v -# Run the Python test module +# Run the Python test module (release mode recommended for better performance) cargo run --release -- -m test ${TEST_MODULE} cargo run --release -- -m test test_unicode # to test test_unicode.py @@ -99,9 +110,11 @@ cargo run --release -- -m test test_unicode # to test test_unicode.py cargo run --release -- -m test test_unicode -k test_unicode_escape ``` +**Note**: For `extra_tests/snippets` tests, use debug mode (`cargo run`) as compilation is faster. For `unittest` (`-m test`), use release mode (`cargo run --release`) for better runtime performance. + ### Determining What to Implement -Run `./whats_left.py` to get a list of unimplemented methods, which is helpful when looking for contribution opportunities. +Run `./scripts/whats_left.py` to get a list of unimplemented methods, which is helpful when looking for contribution opportunities. ## Coding Guidelines @@ -184,6 +197,16 @@ cargo build --target wasm32-wasip1 --no-default-features --features freeze-stdli cargo run --features jit ``` +### Linux Build and Debug on macOS + +See the "Testing on Linux from macOS" section in [DEVELOPMENT.md](DEVELOPMENT.md#testing-on-linux-from-macos). + +### Building venvlauncher (Windows) + +See DEVELOPMENT.md "CPython Version Upgrade Checklist" section. + +**IMPORTANT**: All 4 venvlauncher binaries use the same source code. Do NOT add multiple `[[bin]]` entries to Cargo.toml. Build once and copy with different names. + ## Test Code Modification Rules **CRITICAL: Test code modification restrictions** diff --git a/Cargo.lock b/Cargo.lock index 55ade0703e7..03af941a50f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -142,6 +142,15 @@ dependencies = [ "num-traits", ] +[[package]] +name = "ar_archive_writer" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7eb93bbb63b9c227414f6eb3a0adfddca591a8ce1e9b60661bb08969b87e340b" +dependencies = [ + "object", +] + [[package]] name = "arbitrary" version = "1.4.2" @@ -166,7 +175,7 @@ dependencies = [ "nom", "num-traits", "rusticata-macros", - "thiserror 2.0.17", + "thiserror 2.0.18", "time", ] @@ -240,9 +249,9 @@ checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" [[package]] name = "aws-lc-fips-sys" -version = "0.13.10" +version = "0.13.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "57900537c00a0565a35b63c4c281b372edfc9744b072fd4a3b414350a8f5ed48" +checksum = "df6ea8e07e2df15b9f09f2ac5ee2977369b06d116f0c4eb5fa4ad443b73c7f53" dependencies = [ "bindgen 0.72.1", "cc", @@ -254,9 +263,9 @@ dependencies = [ [[package]] name = "aws-lc-rs" -version = "1.15.2" +version = "1.15.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6a88aab2464f1f25453baa7a07c84c5b7684e274054ba06817f382357f77a288" +checksum = "7b7b6141e96a8c160799cc2d5adecd5cbbe5054cb8c7c4af53da0f83bb7ad256" dependencies = [ "aws-lc-fips-sys", "aws-lc-sys", @@ -266,9 +275,9 @@ dependencies = [ [[package]] name = "aws-lc-sys" -version = "0.35.0" +version = "0.37.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b45afffdee1e7c9126814751f88dddc747f41d91da16c9551a0f1e8a11e788a1" +checksum = "5c34dda4df7017c8db52132f0f8a2e0f8161649d15723ed63fc00c82d0f2081a" dependencies = [ "cc", "cmake", @@ -284,9 +293,9 @@ checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" [[package]] name = "base64ct" -version = "1.8.1" +version = "1.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0e050f626429857a27ddccb31e0aca21356bfa709c04041aefddac081a8f068a" +checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" [[package]] name = "bindgen" @@ -443,9 +452,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.2.51" +version = "1.2.54" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7a0aeaff4ff1a90589618835a598e545176939b97874f7abc7851caa0618f203" +checksum = "6354c81bbfd62d9cfa9cb3c773c2b7b2a3a482d569de977fd0e961f6e7c00583" dependencies = [ "find-msvc-tools", "jobserver", @@ -482,9 +491,9 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" [[package]] name = "chrono" -version = "0.4.42" +version = "0.4.43" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "145052bdd345b87320e369255277e3fb5152762ad123a901ef5c262dd38fe8d2" +checksum = "fac4744fb15ae8337dc853fee7fb3f4e48c0fbaa23d0afe49c447b4fab126118" dependencies = [ "iana-time-zone", "js-sys", @@ -543,18 +552,18 @@ dependencies = [ [[package]] name = "clap" -version = "4.5.53" +version = "4.5.54" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c9e340e012a1bf4935f5282ed1436d1489548e8f72308207ea5df0e23d2d03f8" +checksum = "c6e6ff9dcd79cff5cd969a17a545d79e84ab086e444102a591e288a8aa3ce394" dependencies = [ "clap_builder", ] [[package]] name = "clap_builder" -version = "4.5.53" +version = "4.5.54" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d76b5d13eaa18c901fd2f7fca939fefe3a0727a953561fefdf3b2922b8569d00" +checksum = "fa42cf4d2b7a41bc8f663a7cab4031ebafa1bf3875705bfaf8466dc60ab52c00" dependencies = [ "anstyle", "clap_lex", @@ -562,9 +571,9 @@ dependencies = [ [[package]] name = "clap_lex" -version = "0.7.6" +version = "0.7.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a1d728cc89cf3aee9ff92b05e62b19ee65a02b5702cff7d5a377e32c6ae29d8d" +checksum = "c3e64b0cc0439b12df2fa678eae89a1c56a529fd067a9115f7827f1fffd22b32" [[package]] name = "clipboard-win" @@ -584,6 +593,15 @@ dependencies = [ "cc", ] +[[package]] +name = "cobs" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0fa961b519f0b462e3a3b4a34b64d119eeaca1d59af726fe450bbba07a9fc0a1" +dependencies = [ + "thiserror 2.0.18", +] + [[package]] name = "collection_literals" version = "1.0.3" @@ -691,9 +709,9 @@ dependencies = [ [[package]] name = "cranelift" -version = "0.127.0" +version = "0.128.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "513887fe45ce3979a4766ddc9992c3cdbf509add2a31906350649423a1d0d287" +checksum = "d483a248b5d971d1ef6a814385502a38d8dde8fbf08b4ad08b78c53b8d66f923" dependencies = [ "cranelift-codegen", "cranelift-frontend", @@ -702,42 +720,46 @@ dependencies = [ [[package]] name = "cranelift-assembler-x64" -version = "0.127.0" +version = "0.128.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8bd963a645179fa33834ba61fa63353998543b07f877e208da9eb47d4a70d1e7" +checksum = "d32b9105ce689b3e79ae288f62e9c2d0de66e4869176a11829e5c696da0f018f" dependencies = [ "cranelift-assembler-x64-meta", ] [[package]] name = "cranelift-assembler-x64-meta" -version = "0.127.0" +version = "0.128.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f6d5739c9dc6b5553ca758d78d87d127dd19f397f776efecf817b8ba8d0bb01" +checksum = "0e950e8dd96c1760f1c3a2b06d3d35584a3617239d034e73593ec096a1f3ea69" dependencies = [ "cranelift-srcgen", ] [[package]] name = "cranelift-bforest" -version = "0.127.0" +version = "0.128.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ff402c11bb1c9652b67a3e885e84b1b8d00c13472c8fd85211e06a41a63c3e03" +checksum = "d769576bc48246fccf7f07173739e5f7a7fb3270eb9ac363c0792cad8963c034" dependencies = [ "cranelift-entity", ] [[package]] name = "cranelift-bitset" -version = "0.127.0" +version = "0.128.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "769a0d88c2f5539e9c5536a93a7bf164b0dc68d91e3d00723e5b4ffc1440afdc" +checksum = "94d37c4589e52def48bd745c3b28b523d66ade8b074644ed3a366144c225f212" +dependencies = [ + "serde", + "serde_derive", +] [[package]] name = "cranelift-codegen" -version = "0.127.0" +version = "0.128.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d4351f721fb3b26add1c180f0a75c7474bab2f903c8b777c6ca65238ded59a78" +checksum = "c23b5ab93367eba82bddf49b63d841d8a0b8b39fb89d82829de6647b3a747108" dependencies = [ "bumpalo", "cranelift-assembler-x64", @@ -761,9 +783,9 @@ dependencies = [ [[package]] name = "cranelift-codegen-meta" -version = "0.127.0" +version = "0.128.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61f86c0ba5b96713643f4dd0de0df12844de9c7bb137d6829b174b706939aa74" +checksum = "6c6118d26dd046455d31374b9432947ea2ba445c21fd8724370edd072f51f3bd" dependencies = [ "cranelift-assembler-x64-meta", "cranelift-codegen-shared", @@ -773,33 +795,35 @@ dependencies = [ [[package]] name = "cranelift-codegen-shared" -version = "0.127.0" +version = "0.128.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f08605eee8d51fd976a970bd5b16c9529b51b624f8af68f80649ffb172eb85a4" +checksum = "a068c67f04f37de835fda87a10491e266eea9f9283d0887d8bd0a2c0726588a9" [[package]] name = "cranelift-control" -version = "0.127.0" +version = "0.128.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "623aab0a09e40f0cf0b5d35eb7832bae4c4f13e3768228e051a6c1a60e88ef5f" +checksum = "35ceb830549fcd7f05493a3b6d3d2bcfa4d43588b099e8c2393d2d140d6f7951" dependencies = [ "arbitrary", ] [[package]] name = "cranelift-entity" -version = "0.127.0" +version = "0.128.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ea0f066e07e3bcbe38884cc5c94c32c7a90267d69df80f187d9dfe421adaa7c4" +checksum = "2b130f0edd119e7665f1875b8d686bd3fccefd9d74d10e9005cbcd76392e1831" dependencies = [ "cranelift-bitset", + "serde", + "serde_derive", ] [[package]] name = "cranelift-frontend" -version = "0.127.0" +version = "0.128.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40865b02a0e52ca8e580ad64feef530cb1d05f6bb4972b4eef05e3eaeae81701" +checksum = "626a46aa207183bae011de3411a40951c494cea3fb2ef223d3118f75e13b23ca" dependencies = [ "cranelift-codegen", "log", @@ -809,15 +833,15 @@ dependencies = [ [[package]] name = "cranelift-isle" -version = "0.127.0" +version = "0.128.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "104b3c117ae513e9af1d90679842101193a5ccb96ac9f997966d85ea25be2852" +checksum = "d09dab08a5129cf59919fdd4567e599ea955de62191a852982150ac42ce4ab21" [[package]] name = "cranelift-jit" -version = "0.127.0" +version = "0.128.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3aa5f855cfb8e4253ed2d0dfc1a0b6ebe4912e67aa8b7ee14026ff55ca17f1fe" +checksum = "aaab95b37e712267c51ca968ed4fa83d1a79b9ff3bc86fb9469c764340f486e4" dependencies = [ "anyhow", "cranelift-codegen", @@ -835,9 +859,9 @@ dependencies = [ [[package]] name = "cranelift-module" -version = "0.127.0" +version = "0.128.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b1d01806b191b59f4fc4680293dd5f554caf2de5b62f95eff5beef7acb46c29c" +checksum = "4d53f2d6b64ef9fb21da36698d45715639e0df50224883baa1e9bd04f96f0716" dependencies = [ "anyhow", "cranelift-codegen", @@ -846,9 +870,9 @@ dependencies = [ [[package]] name = "cranelift-native" -version = "0.127.0" +version = "0.128.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e5c54e0a358bc05b48f2032e1c320e7f468da068604f2869b77052eab68eb0fe" +checksum = "847b8eaef0f7095b401d3ce80587036495b94e7a051904df9e28d6cd14e69b94" dependencies = [ "cranelift-codegen", "libc", @@ -857,9 +881,9 @@ dependencies = [ [[package]] name = "cranelift-srcgen" -version = "0.127.0" +version = "0.128.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cc6f4b039f453b66c75e9f7886e5a2af96276e151f44dc19b24b58f9a0c98009" +checksum = "15a4849e90e778f2fcc9fd1b93bd074dbf6b8b6f420951f9617c4774fe71e7fc" [[package]] name = "crc32fast" @@ -957,9 +981,9 @@ dependencies = [ [[package]] name = "data-encoding" -version = "2.9.0" +version = "2.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2a2330da5de22e8a3cb63252ce2abb30116bf5265e89c0e01bc17015ce30a476" +checksum = "d7a1e2f27636f116493b8b860f5546edb47c8d8f8ea73e1d2a20be88e28d1fea" [[package]] name = "der" @@ -1092,6 +1116,18 @@ version = "1.15.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719" +[[package]] +name = "embedded-io" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ef1a6892d9eef45c8fa6b9e0086428a2cca8491aca8f787c534a3d6d0bcb3ced" + +[[package]] +name = "embedded-io" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "edd0f118536f44f5ccd48bcb8b111bdc3de888b58c74639dfb034a357d0f206d" + [[package]] name = "encode_unicode" version = "1.0.0" @@ -1186,9 +1222,9 @@ dependencies = [ [[package]] name = "find-msvc-tools" -version = "0.1.6" +version = "0.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "645cbb3a84e60b7531617d5ae4e57f7e27308f6445f5abf653209ea76dec8dff" +checksum = "8591b0bcc8a98a64310a2fae1bb3e9b8564dd10e381e6e28010fde8e8e8568db" [[package]] name = "flagset" @@ -1234,15 +1270,21 @@ dependencies = [ [[package]] name = "flate2" -version = "1.1.5" +version = "1.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bfe33edd8e85a12a67454e37f8c75e730830d83e313556ab9ebf9ee7fbeb3bfb" +checksum = "b375d6465b98090a5f25b1c7703f3859783755aa9a80433b36e0379a3ec2f369" dependencies = [ "crc32fast", - "libz-rs-sys", "miniz_oxide", + "zlib-rs", ] +[[package]] +name = "foldhash" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" + [[package]] name = "foldhash" version = "0.2.0" @@ -1270,6 +1312,31 @@ version = "1.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c" +[[package]] +name = "futures-core" +version = "0.3.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "05f29059c0c2090612e8d742178b0580d2dc940c837851ad723096f87af6663e" + +[[package]] +name = "futures-task" +version = "0.3.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f90f7dce0722e95104fcb095585910c0977252f286e354b5e3bd38902cd99988" + +[[package]] +name = "futures-util" +version = "0.3.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9fa08315bb612088cc391249efdc3bc77536f16c91f6cf495e6fbe85b20a4a81" +dependencies = [ + "futures-core", + "futures-task", + "pin-project-lite", + "pin-utils", + "slab", +] + [[package]] name = "generic-array" version = "0.14.7" @@ -1282,9 +1349,9 @@ dependencies = [ [[package]] name = "get-size-derive2" -version = "0.7.3" +version = "0.7.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ab21d7bd2c625f2064f04ce54bcb88bc57c45724cde45cba326d784e22d3f71a" +checksum = "f2b6d1e2f75c16bfbcd0f95d84f99858a6e2f885c2287d1f5c3a96e8444a34b4" dependencies = [ "attribute-derive", "quote", @@ -1293,13 +1360,14 @@ dependencies = [ [[package]] name = "get-size2" -version = "0.7.3" +version = "0.7.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "879272b0de109e2b67b39fcfe3d25fdbba96ac07e44a254f5a0b4d7ff55340cb" +checksum = "49cf31a6d70300cf81461098f7797571362387ef4bf85d32ac47eaa59b3a5a1a" dependencies = [ "compact_str", "get-size-derive2", "hashbrown 0.16.1", + "ordermap", "smallvec", ] @@ -1324,9 +1392,9 @@ dependencies = [ [[package]] name = "getrandom" -version = "0.2.16" +version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "335ff9f135e4384c8150d6f27c6daed433577f86b4750418338c01a1a2528592" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" dependencies = [ "cfg-if", "libc", @@ -1380,6 +1448,10 @@ name = "hashbrown" version = "0.15.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" +dependencies = [ + "foldhash 0.1.5", + "serde", +] [[package]] name = "hashbrown" @@ -1387,7 +1459,7 @@ version = "0.16.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" dependencies = [ - "foldhash", + "foldhash 0.2.0", ] [[package]] @@ -1458,12 +1530,14 @@ dependencies = [ [[package]] name = "indexmap" -version = "2.12.1" +version = "2.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ad4bb2b565bca0645f4d68c5c9af97fba094e9791da685bf83cb5f3ce74acf2" +checksum = "7714e70437a7dc3ac8eb7e6f8df75fd8eb422675fc7678aff7364301092b1017" dependencies = [ "equivalent", "hashbrown 0.16.1", + "serde", + "serde_core", ] [[package]] @@ -1487,9 +1561,9 @@ dependencies = [ [[package]] name = "insta" -version = "1.46.0" +version = "1.46.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1b66886d14d18d420ab5052cbff544fc5d34d0b2cdd35eb5976aaa10a4a472e5" +checksum = "248b42847813a1550dafd15296fd9748c651d0c32194559dbc05d804d54b21e8" dependencies = [ "console", "once_cell", @@ -1547,9 +1621,9 @@ checksum = "92ecc6618181def0457392ccd0ee51198e065e016d1d527a7ac1b6dc7c1f09d2" [[package]] name = "jiff" -version = "0.2.17" +version = "0.2.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a87d9b8105c23642f50cbbae03d1f75d8422c5cb98ce7ee9271f7ff7505be6b8" +checksum = "e67e8da4c49d6d9909fe03361f9b620f58898859f5c7aded68351e85e71ecf50" dependencies = [ "jiff-static", "log", @@ -1560,9 +1634,9 @@ dependencies = [ [[package]] name = "jiff-static" -version = "0.2.17" +version = "0.2.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b787bebb543f8969132630c51fd0afab173a86c6abae56ff3b9e5e3e3f9f6e58" +checksum = "e0c84ee7f197eca9a86c6fd6cb771e55eb991632f15f2bc3ca6ec838929e6e78" dependencies = [ "proc-macro2", "quote", @@ -1603,9 +1677,9 @@ dependencies = [ [[package]] name = "js-sys" -version = "0.3.83" +version = "0.3.85" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "464a3709c7f55f1f721e5389aa6ea4e3bc6aba669353300af094b29ffbdde1d8" +checksum = "8c942ebf8e95485ca0d52d97da7c5a2c387d0e7f0ba4c35e93bfcaee045955b3" dependencies = [ "once_cell", "wasm-bindgen", @@ -1613,12 +1687,12 @@ dependencies = [ [[package]] name = "junction" -version = "1.3.0" +version = "1.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c52f6e1bf39a7894f618c9d378904a11dbd7e10fe3ec20d1173600e79b1408d8" +checksum = "642883fdc81cf2da15ee8183fa1d2c7da452414dd41541a0f3e1428069345447" dependencies = [ "scopeguard", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -1681,9 +1755,9 @@ checksum = "2c4a545a15244c7d945065b5d392b2d2d7f21526fba56ce51467b06ed445e8f7" [[package]] name = "libc" -version = "0.2.178" +version = "0.2.180" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "37c93d8daa9d8a012fd8ab92f088405fb202ea0b6ab73ee2482ae66af4f42091" +checksum = "bcc35a38544a891a5f7c865aca548a982ccb3b8650a5b06d0fd33a10283c56fc" [[package]] name = "libffi" @@ -1726,9 +1800,9 @@ dependencies = [ [[package]] name = "libm" -version = "0.2.15" +version = "0.2.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f9fbbcab51052fe104eb5e5d351cf728d30a5be1fe14d9be8a3b097481fb97de" +checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" [[package]] name = "libredox" @@ -1822,9 +1896,9 @@ dependencies = [ [[package]] name = "malachite-base" -version = "0.8.0" +version = "0.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c0c91cb6071ed9ac48669d3c79bd2792db596c7e542dbadd217b385bb359f42d" +checksum = "a8b6f86fdbb1eb9955946be91775239dfcb0acdb1a51bb07d5fc9b8c854f5ccd" dependencies = [ "hashbrown 0.16.1", "itertools 0.14.0", @@ -1834,9 +1908,9 @@ dependencies = [ [[package]] name = "malachite-bigint" -version = "0.8.0" +version = "0.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7ff3af5010102f29f2ef4ee6f7b1c5b3f08a6c261b5164e01c41cf43772b6f90" +checksum = "67fcd6e504ffc67db2b3c6d5e90e08054646e2b04f42115a5460bf1c1e37d3bc" dependencies = [ "malachite-base", "malachite-nz", @@ -1847,9 +1921,9 @@ dependencies = [ [[package]] name = "malachite-nz" -version = "0.8.0" +version = "0.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d9ecf4dd76246fd622de4811097966106aa43f9cd7cc36cb85e774fe84c8adc" +checksum = "0197a2f5cfee19d59178e282985c6ca79a9233e26a2adcf40acb693896aa09f6" dependencies = [ "itertools 0.14.0", "libm", @@ -1859,9 +1933,9 @@ dependencies = [ [[package]] name = "malachite-q" -version = "0.8.0" +version = "0.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b7bc9d9adf5b0a7999d84f761c809bec3dc46fe983e4de547725d2b7730462a0" +checksum = "be2add95162aede090c48f0ee51bea7d328847ce3180aa44588111f846cc116b" dependencies = [ "itertools 0.14.0", "malachite-base", @@ -1955,11 +2029,11 @@ dependencies = [ [[package]] name = "mt19937" -version = "3.1.0" +version = "3.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df7151a832e54d2d6b2c827a20e5bcdd80359281cd2c354e725d4b82e7c471de" +checksum = "56bc7ea7924ea1a79a9e817d0483e39295424cf2b1276cf2b968f9a6c9b63b54" dependencies = [ - "rand_core 0.9.3", + "rand_core 0.9.5", ] [[package]] @@ -2028,9 +2102,9 @@ dependencies = [ [[package]] name = "num-conv" -version = "0.1.0" +version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "51d515d32fb182ee37cda2ccdcb92950d6a3c2893aa280e540671c2cd0f3b1d9" +checksum = "cf97ec579c3c42f953ef76dbf8d55ac91fb219dde70e49aa4a6b7d74e9919050" [[package]] name = "num-integer" @@ -2081,6 +2155,15 @@ dependencies = [ "syn", ] +[[package]] +name = "object" +version = "0.37.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff76201f031d8863c38aa7f905eca4f53abbfa15f609db4277d44cd8938f33fe" +dependencies = [ + "memchr", +] + [[package]] name = "oid-registry" version = "0.8.1" @@ -2136,15 +2219,9 @@ dependencies = [ [[package]] name = "openssl-probe" -version = "0.1.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d05e27ee213611ffe7d6348b942e8f942b37114c00cc03cec254295a4a17852e" - -[[package]] -name = "openssl-probe" -version = "0.2.0" +version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f50d9b3dabb09ecd771ad0aa242ca6894994c130308ca3d7684634df8037391" +checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe" [[package]] name = "openssl-src" @@ -2174,6 +2251,15 @@ version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "978aa494585d3ca4ad74929863093e87cac9790d81fe7aba2b3dc2890643a0fc" +[[package]] +name = "ordermap" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfa78c92071bbd3628c22b1a964f7e0eb201dc1456555db072beb1662ecd6715" +dependencies = [ + "indexmap", +] + [[package]] name = "page_size" version = "0.6.0" @@ -2322,6 +2408,18 @@ dependencies = [ "siphasher", ] +[[package]] +name = "pin-project-lite" +version = "0.2.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b3cff922bd51709b605d9ead9aa71031d81447142d828eb4a6eba76fe619f9b" + +[[package]] +name = "pin-utils" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184" + [[package]] name = "pkcs5" version = "0.7.1" @@ -2409,6 +2507,18 @@ dependencies = [ "portable-atomic", ] +[[package]] +name = "postcard" +version = "1.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6764c3b5dd454e283a30e6dfe78e9b31096d9e32036b5d1eaac7a6119ccb9a24" +dependencies = [ + "cobs", + "embedded-io 0.4.0", + "embedded-io 0.6.1", + "serde", +] + [[package]] name = "powerfmt" version = "0.2.0" @@ -2447,20 +2557,35 @@ dependencies = [ [[package]] name = "proc-macro2" -version = "1.0.104" +version = "1.0.106" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9695f8df41bb4f3d222c95a67532365f569318332d03d5f3f67f37b20e6ebdf0" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" dependencies = [ "unicode-ident", ] +[[package]] +name = "psm" +version = "0.1.29" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fa96cb91275ed31d6da3e983447320c4eb219ac180fa1679a0889ff32861e2d" +dependencies = [ + "ar_archive_writer", + "cc", +] + [[package]] name = "pymath" -version = "0.0.2" +version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b66ab66a8610ce209d8b36cd0fecc3a15c494f715e0cb26f0586057f293abc9" +checksum = "bbfb6723b732fc7f0b29a0ee7150c7f70f947bf467b8c3e82530b13589a78b4c" dependencies = [ "libc", + "libm", + "malachite-bigint", + "num-complex", + "num-integer", + "num-traits", ] [[package]] @@ -2526,9 +2651,9 @@ dependencies = [ [[package]] name = "quote" -version = "1.0.42" +version = "1.0.44" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a338cc41d27e6cc6dce6cefc13a0729dfbb81c262b1f519331575dd80ef3067f" +checksum = "21b2ebcf727b7760c461f091f9f0f539b77b8e87f2fd88131e7f1b433b3cece4" dependencies = [ "proc-macro2", ] @@ -2598,7 +2723,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6db2770f06117d490610c7488547d543617b21bfa07796d7a12f6f1bd53850d1" dependencies = [ "rand_chacha 0.9.0", - "rand_core 0.9.3", + "rand_core 0.9.5", ] [[package]] @@ -2618,7 +2743,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" dependencies = [ "ppv-lite86", - "rand_core 0.9.3", + "rand_core 0.9.5", ] [[package]] @@ -2627,14 +2752,14 @@ version = "0.6.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" dependencies = [ - "getrandom 0.2.16", + "getrandom 0.2.17", ] [[package]] name = "rand_core" -version = "0.9.3" +version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "99d9a13982dcf210057a8a78572b2217b667c3beacbf3a0d8b454f6f82837d38" +checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" dependencies = [ "getrandom 0.3.4", ] @@ -2680,16 +2805,16 @@ version = "0.4.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ba009ff324d1fc1b900bd1fdb31564febe58a8ccc8a6fdbb93b543d33b13ca43" dependencies = [ - "getrandom 0.2.16", + "getrandom 0.2.17", "libredox", "thiserror 1.0.69", ] [[package]] name = "regalloc2" -version = "0.13.3" +version = "0.13.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4e249c660440317032a71ddac302f25f1d5dff387667bcc3978d1f77aa31ac34" +checksum = "08effbc1fa53aaebff69521a5c05640523fab037b34a4a2c109506bc938246fa" dependencies = [ "allocator-api2", "bumpalo", @@ -2769,7 +2894,7 @@ checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7" dependencies = [ "cc", "cfg-if", - "getrandom 0.2.16", + "getrandom 0.2.17", "libc", "untrusted 0.9.0", "windows-sys 0.52.0", @@ -2778,26 +2903,25 @@ dependencies = [ [[package]] name = "ruff_python_ast" version = "0.0.0" -source = "git+https://github.com/astral-sh/ruff.git?rev=45bbb4cbffe73cf925d4579c2e3eb413e0539390#45bbb4cbffe73cf925d4579c2e3eb413e0539390" +source = "git+https://github.com/astral-sh/ruff.git?rev=8b2e7b36f246b990fe473a84eef25ff429e59ecf#8b2e7b36f246b990fe473a84eef25ff429e59ecf" dependencies = [ "aho-corasick", "bitflags 2.10.0", "compact_str", "get-size2", "is-macro", - "itertools 0.14.0", "memchr", "ruff_python_trivia", "ruff_source_file", "ruff_text_size", "rustc-hash", - "thiserror 2.0.17", + "thiserror 2.0.18", ] [[package]] name = "ruff_python_parser" version = "0.0.0" -source = "git+https://github.com/astral-sh/ruff.git?rev=45bbb4cbffe73cf925d4579c2e3eb413e0539390#45bbb4cbffe73cf925d4579c2e3eb413e0539390" +source = "git+https://github.com/astral-sh/ruff.git?rev=8b2e7b36f246b990fe473a84eef25ff429e59ecf#8b2e7b36f246b990fe473a84eef25ff429e59ecf" dependencies = [ "bitflags 2.10.0", "bstr", @@ -2817,7 +2941,7 @@ dependencies = [ [[package]] name = "ruff_python_trivia" version = "0.0.0" -source = "git+https://github.com/astral-sh/ruff.git?rev=45bbb4cbffe73cf925d4579c2e3eb413e0539390#45bbb4cbffe73cf925d4579c2e3eb413e0539390" +source = "git+https://github.com/astral-sh/ruff.git?rev=8b2e7b36f246b990fe473a84eef25ff429e59ecf#8b2e7b36f246b990fe473a84eef25ff429e59ecf" dependencies = [ "itertools 0.14.0", "ruff_source_file", @@ -2828,7 +2952,7 @@ dependencies = [ [[package]] name = "ruff_source_file" version = "0.0.0" -source = "git+https://github.com/astral-sh/ruff.git?rev=45bbb4cbffe73cf925d4579c2e3eb413e0539390#45bbb4cbffe73cf925d4579c2e3eb413e0539390" +source = "git+https://github.com/astral-sh/ruff.git?rev=8b2e7b36f246b990fe473a84eef25ff429e59ecf#8b2e7b36f246b990fe473a84eef25ff429e59ecf" dependencies = [ "memchr", "ruff_text_size", @@ -2837,7 +2961,7 @@ dependencies = [ [[package]] name = "ruff_text_size" version = "0.0.0" -source = "git+https://github.com/astral-sh/ruff.git?rev=45bbb4cbffe73cf925d4579c2e3eb413e0539390#45bbb4cbffe73cf925d4579c2e3eb413e0539390" +source = "git+https://github.com/astral-sh/ruff.git?rev=8b2e7b36f246b990fe473a84eef25ff429e59ecf#8b2e7b36f246b990fe473a84eef25ff429e59ecf" dependencies = [ "get-size2", ] @@ -2872,9 +2996,9 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.35" +version = "0.23.36" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "533f54bc6a7d4f647e46ad909549eda97bf5afc1585190ef692b4286b198bd8f" +checksum = "c665f33d38cea657d9614f766881e4d510e0eda4239891eea56b4cadcf01801b" dependencies = [ "aws-lc-rs", "once_cell", @@ -2890,7 +3014,7 @@ version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "612460d5f7bea540c490b2b6395d8e34a953e52b491accd6c86c8164c5932a63" dependencies = [ - "openssl-probe 0.2.0", + "openssl-probe", "rustls-pki-types", "schannel", "security-framework", @@ -2907,9 +3031,9 @@ dependencies = [ [[package]] name = "rustls-pki-types" -version = "1.13.2" +version = "1.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "21e6f2ab2928ca4291b86736a8bd920a277a399bba1589409d72154ff87c1282" +checksum = "be040f8b0a225e40375822a563fa9524378b9d63112f53e19ffff34df5d33fdd" dependencies = [ "zeroize", ] @@ -2943,9 +3067,9 @@ checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f" [[package]] name = "rustls-webpki" -version = "0.103.8" +version = "0.103.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2ffdfa2f5286e2247234e03f680868ac2815974dc39e00ea15adc445d0aafe52" +checksum = "d7df23109aa6c1567d1c575b9952556388da57401e4ace1d15f79eedad0d8f53" dependencies = [ "aws-lc-rs", "ring", @@ -2996,7 +3120,7 @@ dependencies = [ "rustpython-compiler-core", "rustpython-literal", "rustpython-wtf8", - "thiserror 2.0.17", + "thiserror 2.0.18", "unicode_names2 2.0.0", ] @@ -3038,7 +3162,7 @@ dependencies = [ "ruff_text_size", "rustpython-codegen", "rustpython-compiler-core", - "thiserror 2.0.17", + "thiserror 2.0.18", ] [[package]] @@ -3105,7 +3229,8 @@ dependencies = [ "num-traits", "rustpython-compiler-core", "rustpython-derive", - "thiserror 2.0.17", + "rustpython-wtf8", + "thiserror 2.0.18", ] [[package]] @@ -3161,6 +3286,7 @@ dependencies = [ "digest", "dns-lookup", "dyn-clone", + "flame", "flate2", "foreign-types-shared", "gethostname", @@ -3184,7 +3310,7 @@ dependencies = [ "num_enum", "oid-registry", "openssl", - "openssl-probe 0.1.6", + "openssl-probe", "openssl-sys", "page_size", "parking_lot", @@ -3193,7 +3319,7 @@ dependencies = [ "phf 0.13.1", "pkcs8", "pymath", - "rand_core 0.9.3", + "rand_core 0.9.5", "rustix", "rustls", "rustls-native-certs", @@ -3279,6 +3405,7 @@ dependencies = [ "optional", "parking_lot", "paste", + "psm", "result-like", "ruff_python_ast", "ruff_python_parser", @@ -3295,11 +3422,11 @@ dependencies = [ "rustyline", "scoped-tls", "scopeguard", - "serde", + "serde_core", "static_assertions", "strum", "strum_macros", - "thiserror 2.0.17", + "thiserror 2.0.18", "thread_local", "timsort", "uname", @@ -3334,8 +3461,8 @@ dependencies = [ "rustpython-pylib", "rustpython-stdlib", "rustpython-vm", - "serde", "serde-wasm-bindgen", + "serde_core", "wasm-bindgen", "wasm-bindgen-futures", "web-sys", @@ -3377,9 +3504,9 @@ checksum = "a50f4cf475b65d88e057964e0e9bb1f0aa9bbb2036dc65c64596b42932536984" [[package]] name = "safe_arch" -version = "0.9.3" +version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "629516c85c29fe757770fa03f2074cf1eac43d44c02a3de9fc2ef7b0e207dfdd" +checksum = "1f7caad094bd561859bcd467734a720c3c1f5d1f338995351fefe2190c45efed" dependencies = [ "bytemuck", ] @@ -3500,9 +3627,9 @@ dependencies = [ [[package]] name = "serde_json" -version = "1.0.148" +version = "1.0.149" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3084b546a1dd6289475996f182a22aba973866ea8e8b02c51d9f46b1336a22da" +checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86" dependencies = [ "itoa", "memchr", @@ -3600,21 +3727,30 @@ checksum = "bbbb5d9659141646ae647b42fe094daf6c6192d1620870b449d9557f748b2daa" [[package]] name = "siphasher" -version = "1.0.1" +version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "56199f7ddabf13fe5074ce809e7d3f42b42ae711800501b5b16ea82ad029c39d" +checksum = "b2aa850e253778c88a04c3d7323b043aeda9d3e30d5971937c1855769763678e" + +[[package]] +name = "slab" +version = "0.4.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7a2ae44ef20feb57a68b23d846850f861394c2e02dc425a50098ae8c90267589" [[package]] name = "smallvec" version = "1.15.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03" +dependencies = [ + "serde", +] [[package]] name = "socket2" -version = "0.6.1" +version = "0.6.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "17129e116933cf371d018bb80ae557e889637989d8638274fb25622827b03881" +checksum = "86f4aa3ad99f2088c990dfa82d367e19cb29268ed67c574d10d0a4bfe71f07e0" dependencies = [ "libc", "windows-sys 0.60.2", @@ -3668,9 +3804,9 @@ checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" [[package]] name = "syn" -version = "2.0.113" +version = "2.0.114" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "678faa00651c9eb72dd2020cbdf275d92eccb2400d568e419efdd64838145cb4" +checksum = "d4d107df263a3013ef9b1879b0df87d706ff80f65a86ea879bd9c31f9b307c2a" dependencies = [ "proc-macro2", "quote", @@ -3774,11 +3910,11 @@ dependencies = [ [[package]] name = "thiserror" -version = "2.0.17" +version = "2.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f63587ca0f12b72a0600bcba1d40081f830876000bb46dd2337a3051618f4fc8" +checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" dependencies = [ - "thiserror-impl 2.0.17", + "thiserror-impl 2.0.18", ] [[package]] @@ -3794,9 +3930,9 @@ dependencies = [ [[package]] name = "thiserror-impl" -version = "2.0.17" +version = "2.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3ff15c8ecd7de3849db632e14d18d2571fa09dfc5ed93479bc4485c7a517c913" +checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" dependencies = [ "proc-macro2", "quote", @@ -3825,30 +3961,30 @@ dependencies = [ [[package]] name = "time" -version = "0.3.44" +version = "0.3.46" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91e7d9e3bb61134e77bde20dd4825b97c010155709965fedf0f49bb138e52a9d" +checksum = "9da98b7d9b7dad93488a84b8248efc35352b0b2657397d4167e7ad67e5d535e5" dependencies = [ "deranged", "itoa", "num-conv", "powerfmt", - "serde", + "serde_core", "time-core", "time-macros", ] [[package]] name = "time-core" -version = "0.1.6" +version = "0.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40868e7c1d2f0b8d73e4a8c7f0ff63af4f6d19be117e90bd73eb1d62cf831c6b" +checksum = "7694e1cfe791f8d31026952abf09c69ca6f6fa4e1a1229e18988f06a04a12dca" [[package]] name = "time-macros" -version = "0.2.24" +version = "0.2.26" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "30cfb0125f12d9c277f35663a0a33f8c30190f4e4574868a330595412d34ebf3" +checksum = "78cc610bac2dcee56805c99642447d4c5dbde4d01f752ffea0199aee1f601dc4" dependencies = [ "num-conv", "time-core", @@ -3917,9 +4053,9 @@ dependencies = [ [[package]] name = "toml" -version = "0.9.10+spec-1.1.0" +version = "0.9.11+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0825052159284a1a8b4d6c0c86cbc801f2da5afd2b225fa548c72f2e74002f48" +checksum = "f3afc9a848309fe1aaffaed6e1546a7a14de1f935dc9d89d32afd9a44bab7c46" dependencies = [ "indexmap", "serde_core", @@ -4193,9 +4329,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" -version = "1.19.0" +version = "1.20.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2e054861b4bd027cd373e18e8d8d8e6548085000e41290d95ce0c373a654b4a" +checksum = "ee48d38b119b0cd71fe4141b30f5ba9c7c5d9f4e7a3a8b4a674e4b6ef789976f" dependencies = [ "atomic", "js-sys", @@ -4232,18 +4368,18 @@ checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" [[package]] name = "wasip2" -version = "1.0.1+wasi-0.2.4" +version = "1.0.2+wasi-0.2.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0562428422c63773dad2c345a1882263bbf4d65cf3f42e90921f787ef5ad58e7" +checksum = "9517f9239f02c069db75e65f174b3da828fe5f5b945c4dd26bd25d89c03ebcf5" dependencies = [ "wit-bindgen", ] [[package]] name = "wasm-bindgen" -version = "0.2.106" +version = "0.2.108" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0d759f433fa64a2d763d1340820e46e111a7a5ab75f993d1852d70b03dbb80fd" +checksum = "64024a30ec1e37399cf85a7ffefebdb72205ca1c972291c51512360d90bd8566" dependencies = [ "cfg-if", "once_cell", @@ -4254,11 +4390,12 @@ dependencies = [ [[package]] name = "wasm-bindgen-futures" -version = "0.4.56" +version = "0.4.58" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "836d9622d604feee9e5de25ac10e3ea5f2d65b41eac0d9ce72eb5deae707ce7c" +checksum = "70a6e77fd0ae8029c9ea0063f87c46fde723e7d887703d74ad2616d792e51e6f" dependencies = [ "cfg-if", + "futures-util", "js-sys", "once_cell", "wasm-bindgen", @@ -4267,9 +4404,9 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro" -version = "0.2.106" +version = "0.2.108" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "48cb0d2638f8baedbc542ed444afc0644a29166f1595371af4fecf8ce1e7eeb3" +checksum = "008b239d9c740232e71bd39e8ef6429d27097518b6b30bdf9086833bd5b6d608" dependencies = [ "quote", "wasm-bindgen-macro-support", @@ -4277,9 +4414,9 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro-support" -version = "0.2.106" +version = "0.2.108" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cefb59d5cd5f92d9dcf80e4683949f15ca4b511f4ac0a6e14d4e1ac60c6ecd40" +checksum = "5256bae2d58f54820e6490f9839c49780dff84c65aeab9e772f15d5f0e913a55" dependencies = [ "bumpalo", "proc-macro2", @@ -4290,39 +4427,72 @@ dependencies = [ [[package]] name = "wasm-bindgen-shared" -version = "0.2.106" +version = "0.2.108" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cbc538057e648b67f72a982e708d485b2efa771e1ac05fec311f9f63e5800db4" +checksum = "1f01b580c9ac74c8d8f0c0e4afb04eeef2acf145458e52c03845ee9cd23e3d12" dependencies = [ "unicode-ident", ] [[package]] -name = "wasmtime-internal-jit-icache-coherence" -version = "40.0.0" +name = "wasmparser" +version = "0.243.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f6d8db401b0528ec316dfbe579e6ab4152d61739cfe076706d2009127970159d" +dependencies = [ + "bitflags 2.10.0", + "hashbrown 0.15.5", + "indexmap", + "serde", +] + +[[package]] +name = "wasmtime-environ" +version = "41.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0858b470463f3e7c73acd6049046049e64be17b98901c2db5047450cf83df1fe" +checksum = "37b9af430b11ff3cd63fbef54cf38e26154089c179316b8a5e400b8ba2d0ebf1" dependencies = [ "anyhow", + "cranelift-bitset", + "cranelift-entity", + "gimli", + "indexmap", + "log", + "object", + "postcard", + "serde", + "serde_derive", + "smallvec", + "target-lexicon", + "wasmparser", +] + +[[package]] +name = "wasmtime-internal-jit-icache-coherence" +version = "41.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85b46da671c07242b5f5eab491b12d6c25dd26929f1693c055fcca94489ef8f5" +dependencies = [ "cfg-if", "libc", + "wasmtime-environ", "windows-sys 0.61.2", ] [[package]] name = "wasmtime-internal-math" -version = "40.0.0" +version = "41.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "222e1a590ece4e898f20af1e541b61d2cb803f2557e7eaff23e6c1db5434454a" +checksum = "4d1f0763c6f6f78e410f964db9f53d9b84ab4cc336945e81f0b78717b0a9934e" dependencies = [ "libm", ] [[package]] name = "web-sys" -version = "0.3.83" +version = "0.3.85" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9b32828d774c412041098d182a8b38b16ea816958e07cf40eec2bc080ae137ac" +checksum = "312e32e551d92129218ea9a2452120f4aabc03529ef03e4d0d82fb2780608598" dependencies = [ "js-sys", "wasm-bindgen", @@ -4330,18 +4500,18 @@ dependencies = [ [[package]] name = "webpki-root-certs" -version = "1.0.4" +version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ee3e3b5f5e80bc89f30ce8d0343bf4e5f12341c51f3e26cbeecbc7c85443e85b" +checksum = "36a29fc0408b113f68cf32637857ab740edfafdf460c326cd2afaa2d84cc05dc" dependencies = [ "rustls-pki-types", ] [[package]] name = "webpki-roots" -version = "1.0.4" +version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b2878ef029c47c6e8cf779119f20fcf52bde7ad42a731b2a304bc221df17571e" +checksum = "12bed680863276c63889429bfd6cab3b99943659923822de1c8a39c49e4d722c" dependencies = [ "rustls-pki-types", ] @@ -4359,9 +4529,9 @@ dependencies = [ [[package]] name = "wide" -version = "0.8.3" +version = "1.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "13ca908d26e4786149c48efcf6c0ea09ab0e06d1fe3c17dc1b4b0f1ca4a7e788" +checksum = "ac11b009ebeae802ed758530b6496784ebfee7a87b9abfbcaf3bbe25b814eb25" dependencies = [ "bytemuck", "safe_arch", @@ -4702,9 +4872,9 @@ checksum = "5a5364e9d77fcdeeaa6062ced926ee3381faa2ee02d3eb83a5c27a8825540829" [[package]] name = "winresource" -version = "0.1.28" +version = "0.1.30" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6b021990998587d4438bb672b5c5f034cbc927f51b45e3807ab7323645ef4899" +checksum = "e287ced0f21cd11f4035fe946fd3af145f068d1acb708afd248100f89ec7432d" dependencies = [ "toml", "version_check", @@ -4718,9 +4888,9 @@ checksum = "d135d17ab770252ad95e9a872d365cf3090e3be864a34ab46f48555993efc904" [[package]] name = "wit-bindgen" -version = "0.46.0" +version = "0.51.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f17a85883d4e6d00e8a97c586de764dabcc06133f7f1d55dce5cdc070ad7fe59" +checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5" [[package]] name = "x509-cert" @@ -4749,15 +4919,15 @@ dependencies = [ "nom", "oid-registry", "rusticata-macros", - "thiserror 2.0.17", + "thiserror 2.0.18", "time", ] [[package]] name = "xml" -version = "1.2.0" +version = "1.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2df5825faced2427b2da74d9100f1e2e93c533fff063506a81ede1cf517b2e7e" +checksum = "b8aa498d22c9bbaf482329839bc5620c46be275a19a812e9a22a2b07529a642a" [[package]] name = "xz2" @@ -4770,18 +4940,18 @@ dependencies = [ [[package]] name = "zerocopy" -version = "0.8.31" +version = "0.8.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fd74ec98b9250adb3ca554bdde269adf631549f51d8a8f8f0a10b50f1cb298c3" +checksum = "71ddd76bcebeed25db614f82bf31a9f4222d3fbba300e6fb6c00afa26cbd4d9d" dependencies = [ "zerocopy-derive", ] [[package]] name = "zerocopy-derive" -version = "0.8.31" +version = "0.8.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d8a8d209fdf45cf5138cbb5a506f6b52522a25afccc534d1475dad8e31105c6a" +checksum = "d8187381b52e32220d50b255276aa16a084ec0a9017a0ca2152a1f55c539758d" dependencies = [ "proc-macro2", "quote", @@ -4816,6 +4986,6 @@ checksum = "40990edd51aae2c2b6907af74ffb635029d5788228222c4bb811e9351c0caad3" [[package]] name = "zmij" -version = "1.0.5" +version = "1.0.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e3280a1b827474fcd5dbef4b35a674deb52ba5c312363aef9135317df179d81b" +checksum = "02aae0f83f69aafc94776e879363e9771d7ecbffe2c7fbb6c14c5e00dfe88439" diff --git a/Cargo.toml b/Cargo.toml index 5bd392e9d26..35e03a93d74 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -15,7 +15,7 @@ importlib = ["rustpython-vm/importlib"] encodings = ["rustpython-vm/encodings"] stdio = ["rustpython-vm/stdio"] stdlib = ["rustpython-stdlib", "rustpython-pylib", "encodings"] -flame-it = ["rustpython-vm/flame-it", "flame", "flamescope"] +flame-it = ["rustpython-vm/flame-it", "rustpython-stdlib/flame-it", "flame", "flamescope"] freeze-stdlib = ["stdlib", "rustpython-vm/freeze-stdlib", "rustpython-pylib?/freeze-stdlib"] jit = ["rustpython-vm/jit"] threading = ["rustpython-vm/threading", "rustpython-stdlib/threading"] @@ -54,6 +54,7 @@ rustyline = { workspace = true } [dev-dependencies] criterion = { workspace = true } pyo3 = { version = "0.27", features = ["auto-initialize"] } +rustpython-stdlib = { workspace = true } [[bench]] name = "execution" @@ -91,16 +92,6 @@ lto = "thin" # REDOX START, Uncomment when you want to compile/check with redoxer # REDOX END -# Used only on Windows to build the vcpkg dependencies -[package.metadata.vcpkg] -git = "https://github.com/microsoft/vcpkg" -# The revision of the vcpkg repository to use -# https://github.com/microsoft/vcpkg/tags -rev = "2025.09.17" - -[package.metadata.vcpkg.target] -x86_64-pc-windows-msvc = { triplet = "x64-windows-static-md", dev-dependencies = ["openssl" ] } - [package.metadata.packager] product-name = "RustPython" identifier = "com.rustpython.rustpython" @@ -126,12 +117,13 @@ members = [ ".", "crates/*", ] +exclude = ["pymath"] [workspace.package] version = "0.4.0" authors = ["RustPython Team"] edition = "2024" -rust-version = "1.89.0" +rust-version = "1.93.0" repository = "https://github.com/RustPython/RustPython" license = "MIT" @@ -151,12 +143,12 @@ rustpython-sre_engine = { path = "crates/sre_engine", version = "0.4.0" } rustpython-wtf8 = { path = "crates/wtf8", version = "0.4.0" } rustpython-doc = { path = "crates/doc", version = "0.4.0" } -# Ruff tag 0.14.10 is based on commit 45bbb4cbffe73cf925d4579c2e3eb413e0539390 +# Ruff tag 0.14.14 is based on commit 8b2e7b36f246b990fe473a84eef25ff429e59ecf # at the time of this capture. We use the commit hash to ensure reproducible builds. -ruff_python_parser = { git = "https://github.com/astral-sh/ruff.git", rev = "45bbb4cbffe73cf925d4579c2e3eb413e0539390" } -ruff_python_ast = { git = "https://github.com/astral-sh/ruff.git", rev = "45bbb4cbffe73cf925d4579c2e3eb413e0539390" } -ruff_text_size = { git = "https://github.com/astral-sh/ruff.git", rev = "45bbb4cbffe73cf925d4579c2e3eb413e0539390" } -ruff_source_file = { git = "https://github.com/astral-sh/ruff.git", rev = "45bbb4cbffe73cf925d4579c2e3eb413e0539390" } +ruff_python_parser = { git = "https://github.com/astral-sh/ruff.git", rev = "8b2e7b36f246b990fe473a84eef25ff429e59ecf" } +ruff_python_ast = { git = "https://github.com/astral-sh/ruff.git", rev = "8b2e7b36f246b990fe473a84eef25ff429e59ecf" } +ruff_text_size = { git = "https://github.com/astral-sh/ruff.git", rev = "8b2e7b36f246b990fe473a84eef25ff429e59ecf" } +ruff_source_file = { git = "https://github.com/astral-sh/ruff.git", rev = "8b2e7b36f246b990fe473a84eef25ff429e59ecf" } phf = { version = "0.13.1", default-features = false, features = ["macros"]} ahash = "0.8.12" @@ -164,7 +156,7 @@ ascii = "1.1" bitflags = "2.9.4" bstr = "1" cfg-if = "1.0" -chrono = { version = "0.4.42", default-features = false, features = ["clock", "oldtime", "std"] } +chrono = { version = "0.4.43", default-features = false, features = ["clock", "oldtime", "std"] } constant_time_eq = "0.4" criterion = { version = "0.8", features = ["html_reports"] } crossbeam-utils = "0.8.21" @@ -172,18 +164,18 @@ flame = "0.2.2" getrandom = { version = "0.3", features = ["std"] } glob = "0.3" hex = "0.4.3" -indexmap = { version = "2.11.3", features = ["std"] } +indexmap = { version = "2.13.0", features = ["std"] } insta = "1.46" itertools = "0.14.0" is-macro = "0.3.7" -junction = "1.3.0" -libc = "0.2.178" +junction = "1.4.1" +libc = "0.2.180" libffi = "5" log = "0.4.29" nix = { version = "0.30", features = ["fs", "user", "process", "term", "time", "signal", "ioctl", "socket", "sched", "zerocopy", "dir", "hostname", "net", "poll"] } -malachite-bigint = "0.8" -malachite-q = "0.8" -malachite-base = "0.8" +malachite-bigint = "0.9.1" +malachite-q = "0.9.1" +malachite-base = "0.9.1" memchr = "2.7.4" num-complex = "0.4.6" num-integer = "0.1.46" @@ -193,15 +185,15 @@ optional = "0.5" once_cell = "1.20.3" parking_lot = "0.12.3" paste = "1.0.15" -proc-macro2 = "1.0.93" -pymath = "0.0.2" -quote = "1.0.38" +proc-macro2 = "1.0.105" +pymath = { version = "0.1.5", features = ["mul_add", "malachite-bigint", "complex"] } +quote = "1.0.44" radium = "1.1.1" rand = "0.9" rand_core = { version = "0.9", features = ["os_rng"] } rustix = { version = "1.1", features = ["event"] } rustyline = "17.0.1" -serde = { version = "1.0.225", default-features = false } +serde = { package = "serde_core", version = "1.0.225", default-features = false, features = ["alloc"] } schannel = "0.1.28" scoped-tls = "1" scopeguard = "1" diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index d5c675faca6..24c149eebef 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -19,13 +19,13 @@ The contents of the Development Guide include: RustPython requires the following: -- Rust latest stable version (e.g 1.69.0 as of Apr 20 2023) +- Rust latest stable version (e.g 1.92.0 as of Jan 7 2026) - To check Rust version: `rustc --version` - If you have `rustup` on your system, enter to update to the latest stable version: `rustup update stable` - If you do not have Rust installed, use [rustup](https://rustup.rs/) to do so. -- CPython version 3.13 or higher +- CPython version 3.14 or higher - CPython can be installed by your operating system's package manager, from the [Python website](https://www.python.org/downloads/), or using a third-party distribution, such as @@ -95,6 +95,41 @@ To run only `test_cmath` (located at `Lib/test/test_cmath`) verbosely: $ cargo run --release -- -m test test_cmath -v ``` +### Testing on Linux from macOS + +You can test RustPython on Linux from macOS using Apple's `container` CLI. + +**Setup (one-time):** + +```shell +# Install container CLI +$ brew install container + +# Disable Rosetta requirement for arm64-only builds +$ defaults write com.apple.container.defaults build.rosetta -bool false + +# Build the development image +$ container build --arch arm64 -t rustpython-dev -f .devcontainer/Dockerfile . +``` + +**Running tests:** + +```shell +# Start a persistent container in background (8GB memory, 4 CPUs for compilation) +$ container run -d --name rustpython-test -m 8G -c 4 \ + --mount type=bind,source=$(pwd),target=/workspace \ + -w /workspace rustpython-dev sleep infinity + +# Run tests inside the container +$ container exec rustpython-test sh -c "cargo run --release -- -m test test_ensurepip" + +# Run any command +$ container exec rustpython-test sh -c "cargo test --workspace" + +# Stop and remove the container when done +$ container rm -f rustpython-test +``` + ## Profiling To profile RustPython, build it in `release` mode with the `flame-it` feature. @@ -118,18 +153,17 @@ exists a raw html viewer which is currently broken, and we welcome a PR to fix i Understanding a new codebase takes time. Here's a brief view of the repository's structure: -- `compiler/src`: python compilation to bytecode - - `core/src`: python bytecode representation in rust structures - - `parser/src`: python lexing, parsing and ast -- `derive/src`: Rust language extensions and macros specific to rustpython +- `crates/compiler/src`: python compilation to bytecode + - `crates/compiler-core/src`: python bytecode representation in rust structures +- `crates/derive/src` and `crates/derive-impl/src`: Rust language extensions and macros specific to rustpython - `Lib`: Carefully selected / copied files from CPython sourcecode. This is the python side of the standard library. - `test`: CPython test suite -- `vm/src`: python virtual machine +- `crates/vm/src`: python virtual machine - `builtins`: Builtin functions and types - `stdlib`: Standard library parts implemented in rust. - `src`: using the other subcrates to bring rustpython to life. -- `wasm`: Binary crate and resources for WebAssembly build +- `crates/wasm`: Binary crate and resources for WebAssembly build - `extra_tests`: extra integration test snippets as a supplement to `Lib/test`. Add new RustPython-only regression tests here; do not place new tests under `Lib/test`. @@ -141,9 +175,9 @@ implementation is found in the `src` directory (specifically, `src/lib.rs`). The top-level `rustpython` binary depends on several lower-level crates including: -- `rustpython-parser` (implementation in `compiler/parser/src`) -- `rustpython-compiler` (implementation in `compiler/src`) -- `rustpython-vm` (implementation in `vm/src`) +- `ruff_python_parser` and `ruff_python_ast` (external dependencies from the Ruff project) +- `rustpython-compiler` (implementation in `crates/compiler/src`) +- `rustpython-vm` (implementation in `crates/vm/src`) Together, these crates provide the functions of a programming language and enable a line of code to go through a series of steps: @@ -154,31 +188,26 @@ enable a line of code to go through a series of steps: - compile the AST into bytecode - execute the bytecode in the virtual machine (VM). -### rustpython-parser +### Parser and AST -This crate contains the lexer and parser to convert a line of code to -an Abstract Syntax Tree (AST): +RustPython uses the Ruff project's parser and AST implementation: -- Lexer: `compiler/parser/src/lexer.rs` converts Python source code into tokens -- Parser: `compiler/parser/src/parser.rs` takes the tokens generated by the lexer and parses - the tokens into an AST (Abstract Syntax Tree) where the nodes of the syntax - tree are Rust structs and enums. - - The Parser relies on `LALRPOP`, a Rust parser generator framework. The - LALRPOP definition of Python's grammar is in `compiler/parser/src/python.lalrpop`. - - More information on parsers and a tutorial can be found in the - [LALRPOP book](https://lalrpop.github.io/lalrpop/). -- AST: `compiler/ast/` implements in Rust the Python types and expressions - represented by the AST nodes. +- Parser: `ruff_python_parser` is used to convert Python source code into tokens + and parse them into an Abstract Syntax Tree (AST) +- AST: `ruff_python_ast` provides the Rust types and expressions represented by + the AST nodes +- These are external dependencies maintained by the Ruff project +- For more information, visit the [Ruff GitHub repository](https://github.com/astral-sh/ruff) ### rustpython-compiler The `rustpython-compiler` crate's purpose is to transform the AST (Abstract Syntax Tree) to bytecode. The implementation of the compiler is found in the -`compiler/src` directory. The compiler implements Python's symbol table, +`crates/compiler/src` directory. The compiler implements Python's symbol table, ast->bytecode compiler, and bytecode optimizer in Rust. -Implementation of bytecode structure in Rust is found in the `compiler/core/src` -directory. `compiler/core/src/bytecode.rs` contains the representation of +Implementation of bytecode structure in Rust is found in the `crates/compiler-core/src` +directory. `crates/compiler-core/src/bytecode.rs` contains the representation of instructions and operations in Rust. Further information about Python's bytecode instructions can be found in the [Python documentation](https://docs.python.org/3/library/dis.html#bytecodes). @@ -186,14 +215,14 @@ bytecode instructions can be found in the ### rustpython-vm The `rustpython-vm` crate has the important job of running the virtual machine that -executes Python's instructions. The `vm/src` directory contains code to +executes Python's instructions. The `crates/vm/src` directory contains code to implement the read and evaluation loop that fetches and dispatches instructions. This directory also contains the implementation of the -Python Standard Library modules in Rust (`vm/src/stdlib`). In Python -everything can be represented as an object. The `vm/src/builtins` directory holds +Python Standard Library modules in Rust (`crates/vm/src/stdlib`). In Python +everything can be represented as an object. The `crates/vm/src/builtins` directory holds the Rust code used to represent different Python objects and their methods. The core implementation of what a Python object is can be found in -`vm/src/object/core.rs`. +`crates/vm/src/object/core.rs`. ### Code generation diff --git a/Lib/_ast_unparse.py b/Lib/_ast_unparse.py new file mode 100644 index 00000000000..1c8741b5a55 --- /dev/null +++ b/Lib/_ast_unparse.py @@ -0,0 +1,1161 @@ +# This module contains ``ast.unparse()``, defined here +# to improve the import time for the ``ast`` module. +import sys +from _ast import * +from ast import NodeVisitor +from contextlib import contextmanager, nullcontext +from enum import IntEnum, auto, _simple_enum + +# Large float and imaginary literals get turned into infinities in the AST. +# We unparse those infinities to INFSTR. +_INFSTR = "1e" + repr(sys.float_info.max_10_exp + 1) + +@_simple_enum(IntEnum) +class _Precedence: + """Precedence table that originated from python grammar.""" + + NAMED_EXPR = auto() # := + TUPLE = auto() # , + YIELD = auto() # 'yield', 'yield from' + TEST = auto() # 'if'-'else', 'lambda' + OR = auto() # 'or' + AND = auto() # 'and' + NOT = auto() # 'not' + CMP = auto() # '<', '>', '==', '>=', '<=', '!=', + # 'in', 'not in', 'is', 'is not' + EXPR = auto() + BOR = EXPR # '|' + BXOR = auto() # '^' + BAND = auto() # '&' + SHIFT = auto() # '<<', '>>' + ARITH = auto() # '+', '-' + TERM = auto() # '*', '@', '/', '%', '//' + FACTOR = auto() # unary '+', '-', '~' + POWER = auto() # '**' + AWAIT = auto() # 'await' + ATOM = auto() + + def next(self): + try: + return self.__class__(self + 1) + except ValueError: + return self + + +_SINGLE_QUOTES = ("'", '"') +_MULTI_QUOTES = ('"""', "'''") +_ALL_QUOTES = (*_SINGLE_QUOTES, *_MULTI_QUOTES) + +class Unparser(NodeVisitor): + """Methods in this class recursively traverse an AST and + output source code for the abstract syntax; original formatting + is disregarded.""" + + def __init__(self): + self._source = [] + self._precedences = {} + self._type_ignores = {} + self._indent = 0 + self._in_try_star = False + self._in_interactive = False + + def interleave(self, inter, f, seq): + """Call f on each item in seq, calling inter() in between.""" + seq = iter(seq) + try: + f(next(seq)) + except StopIteration: + pass + else: + for x in seq: + inter() + f(x) + + def items_view(self, traverser, items): + """Traverse and separate the given *items* with a comma and append it to + the buffer. If *items* is a single item sequence, a trailing comma + will be added.""" + if len(items) == 1: + traverser(items[0]) + self.write(",") + else: + self.interleave(lambda: self.write(", "), traverser, items) + + def maybe_newline(self): + """Adds a newline if it isn't the start of generated source""" + if self._source: + self.write("\n") + + def maybe_semicolon(self): + """Adds a "; " delimiter if it isn't the start of generated source""" + if self._source: + self.write("; ") + + def fill(self, text="", *, allow_semicolon=True): + """Indent a piece of text and append it, according to the current + indentation level, or only delineate with semicolon if applicable""" + if self._in_interactive and not self._indent and allow_semicolon: + self.maybe_semicolon() + self.write(text) + else: + self.maybe_newline() + self.write(" " * self._indent + text) + + def write(self, *text): + """Add new source parts""" + self._source.extend(text) + + @contextmanager + def buffered(self, buffer = None): + if buffer is None: + buffer = [] + + original_source = self._source + self._source = buffer + yield buffer + self._source = original_source + + @contextmanager + def block(self, *, extra = None): + """A context manager for preparing the source for blocks. It adds + the character':', increases the indentation on enter and decreases + the indentation on exit. If *extra* is given, it will be directly + appended after the colon character. + """ + self.write(":") + if extra: + self.write(extra) + self._indent += 1 + yield + self._indent -= 1 + + @contextmanager + def delimit(self, start, end): + """A context manager for preparing the source for expressions. It adds + *start* to the buffer and enters, after exit it adds *end*.""" + + self.write(start) + yield + self.write(end) + + def delimit_if(self, start, end, condition): + if condition: + return self.delimit(start, end) + else: + return nullcontext() + + def require_parens(self, precedence, node): + """Shortcut to adding precedence related parens""" + return self.delimit_if("(", ")", self.get_precedence(node) > precedence) + + def get_precedence(self, node): + return self._precedences.get(node, _Precedence.TEST) + + def set_precedence(self, precedence, *nodes): + for node in nodes: + self._precedences[node] = precedence + + def get_raw_docstring(self, node): + """If a docstring node is found in the body of the *node* parameter, + return that docstring node, None otherwise. + + Logic mirrored from ``_PyAST_GetDocString``.""" + if not isinstance( + node, (AsyncFunctionDef, FunctionDef, ClassDef, Module) + ) or len(node.body) < 1: + return None + node = node.body[0] + if not isinstance(node, Expr): + return None + node = node.value + if isinstance(node, Constant) and isinstance(node.value, str): + return node + + def get_type_comment(self, node): + comment = self._type_ignores.get(node.lineno) or node.type_comment + if comment is not None: + return f" # type: {comment}" + + def traverse(self, node): + if isinstance(node, list): + for item in node: + self.traverse(item) + else: + super().visit(node) + + # Note: as visit() resets the output text, do NOT rely on + # NodeVisitor.generic_visit to handle any nodes (as it calls back in to + # the subclass visit() method, which resets self._source to an empty list) + def visit(self, node): + """Outputs a source code string that, if converted back to an ast + (using ast.parse) will generate an AST equivalent to *node*""" + self._source = [] + self.traverse(node) + return "".join(self._source) + + def _write_docstring_and_traverse_body(self, node): + if (docstring := self.get_raw_docstring(node)): + self._write_docstring(docstring) + self.traverse(node.body[1:]) + else: + self.traverse(node.body) + + def visit_Module(self, node): + self._type_ignores = { + ignore.lineno: f"ignore{ignore.tag}" + for ignore in node.type_ignores + } + try: + self._write_docstring_and_traverse_body(node) + finally: + self._type_ignores.clear() + + def visit_Interactive(self, node): + self._in_interactive = True + try: + self._write_docstring_and_traverse_body(node) + finally: + self._in_interactive = False + + def visit_FunctionType(self, node): + with self.delimit("(", ")"): + self.interleave( + lambda: self.write(", "), self.traverse, node.argtypes + ) + + self.write(" -> ") + self.traverse(node.returns) + + def visit_Expr(self, node): + self.fill() + self.set_precedence(_Precedence.YIELD, node.value) + self.traverse(node.value) + + def visit_NamedExpr(self, node): + with self.require_parens(_Precedence.NAMED_EXPR, node): + self.set_precedence(_Precedence.ATOM, node.target, node.value) + self.traverse(node.target) + self.write(" := ") + self.traverse(node.value) + + def visit_Import(self, node): + self.fill("import ") + self.interleave(lambda: self.write(", "), self.traverse, node.names) + + def visit_ImportFrom(self, node): + self.fill("from ") + self.write("." * (node.level or 0)) + if node.module: + self.write(node.module) + self.write(" import ") + self.interleave(lambda: self.write(", "), self.traverse, node.names) + + def visit_Assign(self, node): + self.fill() + for target in node.targets: + self.set_precedence(_Precedence.TUPLE, target) + self.traverse(target) + self.write(" = ") + self.traverse(node.value) + if type_comment := self.get_type_comment(node): + self.write(type_comment) + + def visit_AugAssign(self, node): + self.fill() + self.traverse(node.target) + self.write(" " + self.binop[node.op.__class__.__name__] + "= ") + self.traverse(node.value) + + def visit_AnnAssign(self, node): + self.fill() + with self.delimit_if("(", ")", not node.simple and isinstance(node.target, Name)): + self.traverse(node.target) + self.write(": ") + self.traverse(node.annotation) + if node.value: + self.write(" = ") + self.traverse(node.value) + + def visit_Return(self, node): + self.fill("return") + if node.value: + self.write(" ") + self.traverse(node.value) + + def visit_Pass(self, node): + self.fill("pass") + + def visit_Break(self, node): + self.fill("break") + + def visit_Continue(self, node): + self.fill("continue") + + def visit_Delete(self, node): + self.fill("del ") + self.interleave(lambda: self.write(", "), self.traverse, node.targets) + + def visit_Assert(self, node): + self.fill("assert ") + self.traverse(node.test) + if node.msg: + self.write(", ") + self.traverse(node.msg) + + def visit_Global(self, node): + self.fill("global ") + self.interleave(lambda: self.write(", "), self.write, node.names) + + def visit_Nonlocal(self, node): + self.fill("nonlocal ") + self.interleave(lambda: self.write(", "), self.write, node.names) + + def visit_Await(self, node): + with self.require_parens(_Precedence.AWAIT, node): + self.write("await") + if node.value: + self.write(" ") + self.set_precedence(_Precedence.ATOM, node.value) + self.traverse(node.value) + + def visit_Yield(self, node): + with self.require_parens(_Precedence.YIELD, node): + self.write("yield") + if node.value: + self.write(" ") + self.set_precedence(_Precedence.ATOM, node.value) + self.traverse(node.value) + + def visit_YieldFrom(self, node): + with self.require_parens(_Precedence.YIELD, node): + self.write("yield from ") + if not node.value: + raise ValueError("Node can't be used without a value attribute.") + self.set_precedence(_Precedence.ATOM, node.value) + self.traverse(node.value) + + def visit_Raise(self, node): + self.fill("raise") + if not node.exc: + if node.cause: + raise ValueError(f"Node can't use cause without an exception.") + return + self.write(" ") + self.traverse(node.exc) + if node.cause: + self.write(" from ") + self.traverse(node.cause) + + def do_visit_try(self, node): + self.fill("try", allow_semicolon=False) + with self.block(): + self.traverse(node.body) + for ex in node.handlers: + self.traverse(ex) + if node.orelse: + self.fill("else", allow_semicolon=False) + with self.block(): + self.traverse(node.orelse) + if node.finalbody: + self.fill("finally", allow_semicolon=False) + with self.block(): + self.traverse(node.finalbody) + + def visit_Try(self, node): + prev_in_try_star = self._in_try_star + try: + self._in_try_star = False + self.do_visit_try(node) + finally: + self._in_try_star = prev_in_try_star + + def visit_TryStar(self, node): + prev_in_try_star = self._in_try_star + try: + self._in_try_star = True + self.do_visit_try(node) + finally: + self._in_try_star = prev_in_try_star + + def visit_ExceptHandler(self, node): + self.fill("except*" if self._in_try_star else "except", allow_semicolon=False) + if node.type: + self.write(" ") + self.traverse(node.type) + if node.name: + self.write(" as ") + self.write(node.name) + with self.block(): + self.traverse(node.body) + + def visit_ClassDef(self, node): + self.maybe_newline() + for deco in node.decorator_list: + self.fill("@", allow_semicolon=False) + self.traverse(deco) + self.fill("class " + node.name, allow_semicolon=False) + if hasattr(node, "type_params"): + self._type_params_helper(node.type_params) + with self.delimit_if("(", ")", condition = node.bases or node.keywords): + comma = False + for e in node.bases: + if comma: + self.write(", ") + else: + comma = True + self.traverse(e) + for e in node.keywords: + if comma: + self.write(", ") + else: + comma = True + self.traverse(e) + + with self.block(): + self._write_docstring_and_traverse_body(node) + + def visit_FunctionDef(self, node): + self._function_helper(node, "def") + + def visit_AsyncFunctionDef(self, node): + self._function_helper(node, "async def") + + def _function_helper(self, node, fill_suffix): + self.maybe_newline() + for deco in node.decorator_list: + self.fill("@", allow_semicolon=False) + self.traverse(deco) + def_str = fill_suffix + " " + node.name + self.fill(def_str, allow_semicolon=False) + if hasattr(node, "type_params"): + self._type_params_helper(node.type_params) + with self.delimit("(", ")"): + self.traverse(node.args) + if node.returns: + self.write(" -> ") + self.traverse(node.returns) + with self.block(extra=self.get_type_comment(node)): + self._write_docstring_and_traverse_body(node) + + def _type_params_helper(self, type_params): + if type_params is not None and len(type_params) > 0: + with self.delimit("[", "]"): + self.interleave(lambda: self.write(", "), self.traverse, type_params) + + def visit_TypeVar(self, node): + self.write(node.name) + if node.bound: + self.write(": ") + self.traverse(node.bound) + if node.default_value: + self.write(" = ") + self.traverse(node.default_value) + + def visit_TypeVarTuple(self, node): + self.write("*" + node.name) + if node.default_value: + self.write(" = ") + self.traverse(node.default_value) + + def visit_ParamSpec(self, node): + self.write("**" + node.name) + if node.default_value: + self.write(" = ") + self.traverse(node.default_value) + + def visit_TypeAlias(self, node): + self.fill("type ") + self.traverse(node.name) + self._type_params_helper(node.type_params) + self.write(" = ") + self.traverse(node.value) + + def visit_For(self, node): + self._for_helper("for ", node) + + def visit_AsyncFor(self, node): + self._for_helper("async for ", node) + + def _for_helper(self, fill, node): + self.fill(fill, allow_semicolon=False) + self.set_precedence(_Precedence.TUPLE, node.target) + self.traverse(node.target) + self.write(" in ") + self.traverse(node.iter) + with self.block(extra=self.get_type_comment(node)): + self.traverse(node.body) + if node.orelse: + self.fill("else", allow_semicolon=False) + with self.block(): + self.traverse(node.orelse) + + def visit_If(self, node): + self.fill("if ", allow_semicolon=False) + self.traverse(node.test) + with self.block(): + self.traverse(node.body) + # collapse nested ifs into equivalent elifs. + while node.orelse and len(node.orelse) == 1 and isinstance(node.orelse[0], If): + node = node.orelse[0] + self.fill("elif ", allow_semicolon=False) + self.traverse(node.test) + with self.block(): + self.traverse(node.body) + # final else + if node.orelse: + self.fill("else", allow_semicolon=False) + with self.block(): + self.traverse(node.orelse) + + def visit_While(self, node): + self.fill("while ", allow_semicolon=False) + self.traverse(node.test) + with self.block(): + self.traverse(node.body) + if node.orelse: + self.fill("else", allow_semicolon=False) + with self.block(): + self.traverse(node.orelse) + + def visit_With(self, node): + self.fill("with ", allow_semicolon=False) + self.interleave(lambda: self.write(", "), self.traverse, node.items) + with self.block(extra=self.get_type_comment(node)): + self.traverse(node.body) + + def visit_AsyncWith(self, node): + self.fill("async with ", allow_semicolon=False) + self.interleave(lambda: self.write(", "), self.traverse, node.items) + with self.block(extra=self.get_type_comment(node)): + self.traverse(node.body) + + def _str_literal_helper( + self, string, *, quote_types=_ALL_QUOTES, escape_special_whitespace=False + ): + """Helper for writing string literals, minimizing escapes. + Returns the tuple (string literal to write, possible quote types). + """ + def escape_char(c): + # \n and \t are non-printable, but we only escape them if + # escape_special_whitespace is True + if not escape_special_whitespace and c in "\n\t": + return c + # Always escape backslashes and other non-printable characters + if c == "\\" or not c.isprintable(): + return c.encode("unicode_escape").decode("ascii") + return c + + escaped_string = "".join(map(escape_char, string)) + possible_quotes = quote_types + if "\n" in escaped_string: + possible_quotes = [q for q in possible_quotes if q in _MULTI_QUOTES] + possible_quotes = [q for q in possible_quotes if q not in escaped_string] + if not possible_quotes: + # If there aren't any possible_quotes, fallback to using repr + # on the original string. Try to use a quote from quote_types, + # e.g., so that we use triple quotes for docstrings. + string = repr(string) + quote = next((q for q in quote_types if string[0] in q), string[0]) + return string[1:-1], [quote] + if escaped_string: + # Sort so that we prefer '''"''' over """\"""" + possible_quotes.sort(key=lambda q: q[0] == escaped_string[-1]) + # If we're using triple quotes and we'd need to escape a final + # quote, escape it + if possible_quotes[0][0] == escaped_string[-1]: + assert len(possible_quotes[0]) == 3 + escaped_string = escaped_string[:-1] + "\\" + escaped_string[-1] + return escaped_string, possible_quotes + + def _write_str_avoiding_backslashes(self, string, *, quote_types=_ALL_QUOTES): + """Write string literal value with a best effort attempt to avoid backslashes.""" + string, quote_types = self._str_literal_helper(string, quote_types=quote_types) + quote_type = quote_types[0] + self.write(f"{quote_type}{string}{quote_type}") + + def _ftstring_helper(self, parts): + new_parts = [] + quote_types = list(_ALL_QUOTES) + fallback_to_repr = False + for value, is_constant in parts: + if is_constant: + value, new_quote_types = self._str_literal_helper( + value, + quote_types=quote_types, + escape_special_whitespace=True, + ) + if set(new_quote_types).isdisjoint(quote_types): + fallback_to_repr = True + break + quote_types = new_quote_types + else: + if "\n" in value: + quote_types = [q for q in quote_types if q in _MULTI_QUOTES] + assert quote_types + + new_quote_types = [q for q in quote_types if q not in value] + if new_quote_types: + quote_types = new_quote_types + new_parts.append(value) + + if fallback_to_repr: + # If we weren't able to find a quote type that works for all parts + # of the JoinedStr, fallback to using repr and triple single quotes. + quote_types = ["'''"] + new_parts.clear() + for value, is_constant in parts: + if is_constant: + value = repr('"' + value) # force repr to use single quotes + expected_prefix = "'\"" + assert value.startswith(expected_prefix), repr(value) + value = value[len(expected_prefix):-1] + new_parts.append(value) + + value = "".join(new_parts) + quote_type = quote_types[0] + self.write(f"{quote_type}{value}{quote_type}") + + def _write_ftstring(self, values, prefix): + self.write(prefix) + fstring_parts = [] + for value in values: + with self.buffered() as buffer: + self._write_ftstring_inner(value) + fstring_parts.append( + ("".join(buffer), isinstance(value, Constant)) + ) + self._ftstring_helper(fstring_parts) + + def visit_JoinedStr(self, node): + self._write_ftstring(node.values, "f") + + def visit_TemplateStr(self, node): + self._write_ftstring(node.values, "t") + + def _write_ftstring_inner(self, node, is_format_spec=False): + if isinstance(node, JoinedStr): + # for both the f-string itself, and format_spec + for value in node.values: + self._write_ftstring_inner(value, is_format_spec=is_format_spec) + elif isinstance(node, Constant) and isinstance(node.value, str): + value = node.value.replace("{", "{{").replace("}", "}}") + + if is_format_spec: + value = value.replace("\\", "\\\\") + value = value.replace("'", "\\'") + value = value.replace('"', '\\"') + value = value.replace("\n", "\\n") + self.write(value) + elif isinstance(node, FormattedValue): + self.visit_FormattedValue(node) + elif isinstance(node, Interpolation): + self.visit_Interpolation(node) + else: + raise ValueError(f"Unexpected node inside JoinedStr, {node!r}") + + def _unparse_interpolation_value(self, inner): + unparser = type(self)() + unparser.set_precedence(_Precedence.TEST.next(), inner) + return unparser.visit(inner) + + def _write_interpolation(self, node, use_str_attr=False): + with self.delimit("{", "}"): + if use_str_attr: + expr = node.str + else: + expr = self._unparse_interpolation_value(node.value) + if expr.startswith("{"): + # Separate pair of opening brackets as "{ {" + self.write(" ") + self.write(expr) + if node.conversion != -1: + self.write(f"!{chr(node.conversion)}") + if node.format_spec: + self.write(":") + self._write_ftstring_inner(node.format_spec, is_format_spec=True) + + def visit_FormattedValue(self, node): + self._write_interpolation(node) + + def visit_Interpolation(self, node): + # If `str` is set to `None`, use the `value` to generate the source code. + self._write_interpolation(node, use_str_attr=node.str is not None) + + def visit_Name(self, node): + self.write(node.id) + + def _write_docstring(self, node): + self.fill(allow_semicolon=False) + if node.kind == "u": + self.write("u") + self._write_str_avoiding_backslashes(node.value, quote_types=_MULTI_QUOTES) + + def _write_constant(self, value): + if isinstance(value, (float, complex)): + # Substitute overflowing decimal literal for AST infinities, + # and inf - inf for NaNs. + self.write( + repr(value) + .replace("inf", _INFSTR) + .replace("nan", f"({_INFSTR}-{_INFSTR})") + ) + else: + self.write(repr(value)) + + def visit_Constant(self, node): + value = node.value + if isinstance(value, tuple): + with self.delimit("(", ")"): + self.items_view(self._write_constant, value) + elif value is ...: + self.write("...") + else: + if node.kind == "u": + self.write("u") + self._write_constant(node.value) + + def visit_List(self, node): + with self.delimit("[", "]"): + self.interleave(lambda: self.write(", "), self.traverse, node.elts) + + def visit_ListComp(self, node): + with self.delimit("[", "]"): + self.traverse(node.elt) + for gen in node.generators: + self.traverse(gen) + + def visit_GeneratorExp(self, node): + with self.delimit("(", ")"): + self.traverse(node.elt) + for gen in node.generators: + self.traverse(gen) + + def visit_SetComp(self, node): + with self.delimit("{", "}"): + self.traverse(node.elt) + for gen in node.generators: + self.traverse(gen) + + def visit_DictComp(self, node): + with self.delimit("{", "}"): + self.traverse(node.key) + self.write(": ") + self.traverse(node.value) + for gen in node.generators: + self.traverse(gen) + + def visit_comprehension(self, node): + if node.is_async: + self.write(" async for ") + else: + self.write(" for ") + self.set_precedence(_Precedence.TUPLE, node.target) + self.traverse(node.target) + self.write(" in ") + self.set_precedence(_Precedence.TEST.next(), node.iter, *node.ifs) + self.traverse(node.iter) + for if_clause in node.ifs: + self.write(" if ") + self.traverse(if_clause) + + def visit_IfExp(self, node): + with self.require_parens(_Precedence.TEST, node): + self.set_precedence(_Precedence.TEST.next(), node.body, node.test) + self.traverse(node.body) + self.write(" if ") + self.traverse(node.test) + self.write(" else ") + self.set_precedence(_Precedence.TEST, node.orelse) + self.traverse(node.orelse) + + def visit_Set(self, node): + if node.elts: + with self.delimit("{", "}"): + self.interleave(lambda: self.write(", "), self.traverse, node.elts) + else: + # `{}` would be interpreted as a dictionary literal, and + # `set` might be shadowed. Thus: + self.write('{*()}') + + def visit_Dict(self, node): + def write_key_value_pair(k, v): + self.traverse(k) + self.write(": ") + self.traverse(v) + + def write_item(item): + k, v = item + if k is None: + # for dictionary unpacking operator in dicts {**{'y': 2}} + # see PEP 448 for details + self.write("**") + self.set_precedence(_Precedence.EXPR, v) + self.traverse(v) + else: + write_key_value_pair(k, v) + + with self.delimit("{", "}"): + self.interleave( + lambda: self.write(", "), write_item, zip(node.keys, node.values) + ) + + def visit_Tuple(self, node): + with self.delimit_if( + "(", + ")", + len(node.elts) == 0 or self.get_precedence(node) > _Precedence.TUPLE + ): + self.items_view(self.traverse, node.elts) + + unop = {"Invert": "~", "Not": "not", "UAdd": "+", "USub": "-"} + unop_precedence = { + "not": _Precedence.NOT, + "~": _Precedence.FACTOR, + "+": _Precedence.FACTOR, + "-": _Precedence.FACTOR, + } + + def visit_UnaryOp(self, node): + operator = self.unop[node.op.__class__.__name__] + operator_precedence = self.unop_precedence[operator] + with self.require_parens(operator_precedence, node): + self.write(operator) + # factor prefixes (+, -, ~) shouldn't be separated + # from the value they belong, (e.g: +1 instead of + 1) + if operator_precedence is not _Precedence.FACTOR: + self.write(" ") + self.set_precedence(operator_precedence, node.operand) + self.traverse(node.operand) + + binop = { + "Add": "+", + "Sub": "-", + "Mult": "*", + "MatMult": "@", + "Div": "/", + "Mod": "%", + "LShift": "<<", + "RShift": ">>", + "BitOr": "|", + "BitXor": "^", + "BitAnd": "&", + "FloorDiv": "//", + "Pow": "**", + } + + binop_precedence = { + "+": _Precedence.ARITH, + "-": _Precedence.ARITH, + "*": _Precedence.TERM, + "@": _Precedence.TERM, + "/": _Precedence.TERM, + "%": _Precedence.TERM, + "<<": _Precedence.SHIFT, + ">>": _Precedence.SHIFT, + "|": _Precedence.BOR, + "^": _Precedence.BXOR, + "&": _Precedence.BAND, + "//": _Precedence.TERM, + "**": _Precedence.POWER, + } + + binop_rassoc = frozenset(("**",)) + def visit_BinOp(self, node): + operator = self.binop[node.op.__class__.__name__] + operator_precedence = self.binop_precedence[operator] + with self.require_parens(operator_precedence, node): + if operator in self.binop_rassoc: + left_precedence = operator_precedence.next() + right_precedence = operator_precedence + else: + left_precedence = operator_precedence + right_precedence = operator_precedence.next() + + self.set_precedence(left_precedence, node.left) + self.traverse(node.left) + self.write(f" {operator} ") + self.set_precedence(right_precedence, node.right) + self.traverse(node.right) + + cmpops = { + "Eq": "==", + "NotEq": "!=", + "Lt": "<", + "LtE": "<=", + "Gt": ">", + "GtE": ">=", + "Is": "is", + "IsNot": "is not", + "In": "in", + "NotIn": "not in", + } + + def visit_Compare(self, node): + with self.require_parens(_Precedence.CMP, node): + self.set_precedence(_Precedence.CMP.next(), node.left, *node.comparators) + self.traverse(node.left) + for o, e in zip(node.ops, node.comparators): + self.write(" " + self.cmpops[o.__class__.__name__] + " ") + self.traverse(e) + + boolops = {"And": "and", "Or": "or"} + boolop_precedence = {"and": _Precedence.AND, "or": _Precedence.OR} + + def visit_BoolOp(self, node): + operator = self.boolops[node.op.__class__.__name__] + operator_precedence = self.boolop_precedence[operator] + + def increasing_level_traverse(node): + nonlocal operator_precedence + operator_precedence = operator_precedence.next() + self.set_precedence(operator_precedence, node) + self.traverse(node) + + with self.require_parens(operator_precedence, node): + s = f" {operator} " + self.interleave(lambda: self.write(s), increasing_level_traverse, node.values) + + def visit_Attribute(self, node): + self.set_precedence(_Precedence.ATOM, node.value) + self.traverse(node.value) + # Special case: 3.__abs__() is a syntax error, so if node.value + # is an integer literal then we need to either parenthesize + # it or add an extra space to get 3 .__abs__(). + if isinstance(node.value, Constant) and isinstance(node.value.value, int): + self.write(" ") + self.write(".") + self.write(node.attr) + + def visit_Call(self, node): + self.set_precedence(_Precedence.ATOM, node.func) + self.traverse(node.func) + with self.delimit("(", ")"): + comma = False + for e in node.args: + if comma: + self.write(", ") + else: + comma = True + self.traverse(e) + for e in node.keywords: + if comma: + self.write(", ") + else: + comma = True + self.traverse(e) + + def visit_Subscript(self, node): + def is_non_empty_tuple(slice_value): + return ( + isinstance(slice_value, Tuple) + and slice_value.elts + ) + + self.set_precedence(_Precedence.ATOM, node.value) + self.traverse(node.value) + with self.delimit("[", "]"): + if is_non_empty_tuple(node.slice): + # parentheses can be omitted if the tuple isn't empty + self.items_view(self.traverse, node.slice.elts) + else: + self.traverse(node.slice) + + def visit_Starred(self, node): + self.write("*") + self.set_precedence(_Precedence.EXPR, node.value) + self.traverse(node.value) + + def visit_Ellipsis(self, node): + self.write("...") + + def visit_Slice(self, node): + if node.lower: + self.traverse(node.lower) + self.write(":") + if node.upper: + self.traverse(node.upper) + if node.step: + self.write(":") + self.traverse(node.step) + + def visit_Match(self, node): + self.fill("match ", allow_semicolon=False) + self.traverse(node.subject) + with self.block(): + for case in node.cases: + self.traverse(case) + + def visit_arg(self, node): + self.write(node.arg) + if node.annotation: + self.write(": ") + self.traverse(node.annotation) + + def visit_arguments(self, node): + first = True + # normal arguments + all_args = node.posonlyargs + node.args + defaults = [None] * (len(all_args) - len(node.defaults)) + node.defaults + for index, elements in enumerate(zip(all_args, defaults), 1): + a, d = elements + if first: + first = False + else: + self.write(", ") + self.traverse(a) + if d: + self.write("=") + self.traverse(d) + if index == len(node.posonlyargs): + self.write(", /") + + # varargs, or bare '*' if no varargs but keyword-only arguments present + if node.vararg or node.kwonlyargs: + if first: + first = False + else: + self.write(", ") + self.write("*") + if node.vararg: + self.write(node.vararg.arg) + if node.vararg.annotation: + self.write(": ") + self.traverse(node.vararg.annotation) + + # keyword-only arguments + if node.kwonlyargs: + for a, d in zip(node.kwonlyargs, node.kw_defaults): + self.write(", ") + self.traverse(a) + if d: + self.write("=") + self.traverse(d) + + # kwargs + if node.kwarg: + if first: + first = False + else: + self.write(", ") + self.write("**" + node.kwarg.arg) + if node.kwarg.annotation: + self.write(": ") + self.traverse(node.kwarg.annotation) + + def visit_keyword(self, node): + if node.arg is None: + self.write("**") + else: + self.write(node.arg) + self.write("=") + self.traverse(node.value) + + def visit_Lambda(self, node): + with self.require_parens(_Precedence.TEST, node): + self.write("lambda") + with self.buffered() as buffer: + self.traverse(node.args) + if buffer: + self.write(" ", *buffer) + self.write(": ") + self.set_precedence(_Precedence.TEST, node.body) + self.traverse(node.body) + + def visit_alias(self, node): + self.write(node.name) + if node.asname: + self.write(" as " + node.asname) + + def visit_withitem(self, node): + self.traverse(node.context_expr) + if node.optional_vars: + self.write(" as ") + self.traverse(node.optional_vars) + + def visit_match_case(self, node): + self.fill("case ", allow_semicolon=False) + self.traverse(node.pattern) + if node.guard: + self.write(" if ") + self.traverse(node.guard) + with self.block(): + self.traverse(node.body) + + def visit_MatchValue(self, node): + self.traverse(node.value) + + def visit_MatchSingleton(self, node): + self._write_constant(node.value) + + def visit_MatchSequence(self, node): + with self.delimit("[", "]"): + self.interleave( + lambda: self.write(", "), self.traverse, node.patterns + ) + + def visit_MatchStar(self, node): + name = node.name + if name is None: + name = "_" + self.write(f"*{name}") + + def visit_MatchMapping(self, node): + def write_key_pattern_pair(pair): + k, p = pair + self.traverse(k) + self.write(": ") + self.traverse(p) + + with self.delimit("{", "}"): + keys = node.keys + self.interleave( + lambda: self.write(", "), + write_key_pattern_pair, + zip(keys, node.patterns, strict=True), + ) + rest = node.rest + if rest is not None: + if keys: + self.write(", ") + self.write(f"**{rest}") + + def visit_MatchClass(self, node): + self.set_precedence(_Precedence.ATOM, node.cls) + self.traverse(node.cls) + with self.delimit("(", ")"): + patterns = node.patterns + self.interleave( + lambda: self.write(", "), self.traverse, patterns + ) + attrs = node.kwd_attrs + if attrs: + def write_attr_pattern(pair): + attr, pattern = pair + self.write(f"{attr}=") + self.traverse(pattern) + + if patterns: + self.write(", ") + self.interleave( + lambda: self.write(", "), + write_attr_pattern, + zip(attrs, node.kwd_patterns, strict=True), + ) + + def visit_MatchAs(self, node): + name = node.name + pattern = node.pattern + if name is None: + self.write("_") + elif pattern is None: + self.write(node.name) + else: + with self.require_parens(_Precedence.TEST, node): + self.set_precedence(_Precedence.BOR, node.pattern) + self.traverse(node.pattern) + self.write(f" as {node.name}") + + def visit_MatchOr(self, node): + with self.require_parens(_Precedence.BOR, node): + self.set_precedence(_Precedence.BOR.next(), *node.patterns) + self.interleave(lambda: self.write(" | "), self.traverse, node.patterns) diff --git a/Lib/_collections_abc.py b/Lib/_collections_abc.py index e02fc227384..241d40d5740 100644 --- a/Lib/_collections_abc.py +++ b/Lib/_collections_abc.py @@ -485,9 +485,10 @@ def __new__(cls, origin, args): def __repr__(self): if len(self.__args__) == 2 and _is_param_expr(self.__args__[0]): return super().__repr__() + from annotationlib import type_repr return (f'collections.abc.Callable' - f'[[{", ".join([_type_repr(a) for a in self.__args__[:-1]])}], ' - f'{_type_repr(self.__args__[-1])}]') + f'[[{", ".join([type_repr(a) for a in self.__args__[:-1]])}], ' + f'{type_repr(self.__args__[-1])}]') def __reduce__(self): args = self.__args__ @@ -512,10 +513,6 @@ def __getitem__(self, item): new_args = (t_args, t_result) return _CallableGenericAlias(Callable, tuple(new_args)) - # TODO: RUSTPYTHON; patch for common call - def __or__(self, other): - super().__or__(other) - def _is_param_expr(obj): """Checks if obj matches either a list of types, ``...``, ``ParamSpec`` or ``_ConcatenateGenericAlias`` from typing.py @@ -528,23 +525,6 @@ def _is_param_expr(obj): names = ('ParamSpec', '_ConcatenateGenericAlias') return obj.__module__ == 'typing' and any(obj.__name__ == name for name in names) -def _type_repr(obj): - """Return the repr() of an object, special-casing types (internal helper). - - Copied from :mod:`typing` since collections.abc - shouldn't depend on that module. - (Keep this roughly in sync with the typing version.) - """ - if isinstance(obj, type): - if obj.__module__ == 'builtins': - return obj.__qualname__ - return f'{obj.__module__}.{obj.__qualname__}' - if obj is Ellipsis: - return '...' - if isinstance(obj, FunctionType): - return obj.__name__ - return repr(obj) - class Callable(metaclass=ABCMeta): @@ -1077,6 +1057,7 @@ def count(self, value): Sequence.register(tuple) Sequence.register(str) +Sequence.register(bytes) Sequence.register(range) Sequence.register(memoryview) @@ -1183,4 +1164,4 @@ def __iadd__(self, values): MutableSequence.register(list) -MutableSequence.register(bytearray) # Multiply inheriting, see ByteString +MutableSequence.register(bytearray) diff --git a/Lib/_colorize.py b/Lib/_colorize.py index 9eb6f0933b8..d6673f6692f 100644 --- a/Lib/_colorize.py +++ b/Lib/_colorize.py @@ -1,13 +1,16 @@ -from __future__ import annotations -import io import os import sys +from collections.abc import Callable, Iterator, Mapping +from dataclasses import dataclass, field, Field + COLORIZE = True + # types if False: - from typing import IO + from typing import IO, Self, ClassVar + _theme: Theme class ANSIColors: @@ -17,11 +20,13 @@ class ANSIColors: BLUE = "\x1b[34m" CYAN = "\x1b[36m" GREEN = "\x1b[32m" + GREY = "\x1b[90m" MAGENTA = "\x1b[35m" RED = "\x1b[31m" WHITE = "\x1b[37m" # more like LIGHT GRAY YELLOW = "\x1b[33m" + BOLD = "\x1b[1m" BOLD_BLACK = "\x1b[1;30m" # DARK GRAY BOLD_BLUE = "\x1b[1;34m" BOLD_CYAN = "\x1b[1;36m" @@ -60,13 +65,196 @@ class ANSIColors: INTENSE_BACKGROUND_YELLOW = "\x1b[103m" +ColorCodes = set() NoColors = ANSIColors() -for attr in dir(NoColors): +for attr, code in ANSIColors.__dict__.items(): if not attr.startswith("__"): + ColorCodes.add(code) setattr(NoColors, attr, "") +# +# Experimental theming support (see gh-133346) +# + +# - Create a theme by copying an existing `Theme` with one or more sections +# replaced, using `default_theme.copy_with()`; +# - create a theme section by copying an existing `ThemeSection` with one or +# more colors replaced, using for example `default_theme.syntax.copy_with()`; +# - create a theme from scratch by instantiating a `Theme` data class with +# the required sections (which are also dataclass instances). +# +# Then call `_colorize.set_theme(your_theme)` to set it. +# +# Put your theme configuration in $PYTHONSTARTUP for the interactive shell, +# or sitecustomize.py in your virtual environment or Python installation for +# other uses. Your applications can call `_colorize.set_theme()` too. +# +# Note that thanks to the dataclasses providing default values for all fields, +# creating a new theme or theme section from scratch is possible without +# specifying all keys. +# +# For example, here's a theme that makes punctuation and operators less prominent: +# +# try: +# from _colorize import set_theme, default_theme, Syntax, ANSIColors +# except ImportError: +# pass +# else: +# theme_with_dim_operators = default_theme.copy_with( +# syntax=Syntax(op=ANSIColors.INTENSE_BLACK), +# ) +# set_theme(theme_with_dim_operators) +# del set_theme, default_theme, Syntax, ANSIColors, theme_with_dim_operators +# +# Guarding the import ensures that your .pythonstartup file will still work in +# Python 3.13 and older. Deleting the variables ensures they don't remain in your +# interactive shell's global scope. + +class ThemeSection(Mapping[str, str]): + """A mixin/base class for theme sections. + + It enables dictionary access to a section, as well as implements convenience + methods. + """ + + # The two types below are just that: types to inform the type checker that the + # mixin will work in context of those fields existing + __dataclass_fields__: ClassVar[dict[str, Field[str]]] + _name_to_value: Callable[[str], str] + + def __post_init__(self) -> None: + name_to_value = {} + for color_name in self.__dataclass_fields__: + name_to_value[color_name] = getattr(self, color_name) + super().__setattr__('_name_to_value', name_to_value.__getitem__) + + def copy_with(self, **kwargs: str) -> Self: + color_state: dict[str, str] = {} + for color_name in self.__dataclass_fields__: + color_state[color_name] = getattr(self, color_name) + color_state.update(kwargs) + return type(self)(**color_state) + + @classmethod + def no_colors(cls) -> Self: + color_state: dict[str, str] = {} + for color_name in cls.__dataclass_fields__: + color_state[color_name] = "" + return cls(**color_state) + + def __getitem__(self, key: str) -> str: + return self._name_to_value(key) + + def __len__(self) -> int: + return len(self.__dataclass_fields__) + + def __iter__(self) -> Iterator[str]: + return iter(self.__dataclass_fields__) + + +@dataclass(frozen=True, kw_only=True) +class Argparse(ThemeSection): + usage: str = ANSIColors.BOLD_BLUE + prog: str = ANSIColors.BOLD_MAGENTA + prog_extra: str = ANSIColors.MAGENTA + heading: str = ANSIColors.BOLD_BLUE + summary_long_option: str = ANSIColors.CYAN + summary_short_option: str = ANSIColors.GREEN + summary_label: str = ANSIColors.YELLOW + summary_action: str = ANSIColors.GREEN + long_option: str = ANSIColors.BOLD_CYAN + short_option: str = ANSIColors.BOLD_GREEN + label: str = ANSIColors.BOLD_YELLOW + action: str = ANSIColors.BOLD_GREEN + reset: str = ANSIColors.RESET + + +@dataclass(frozen=True) +class Syntax(ThemeSection): + prompt: str = ANSIColors.BOLD_MAGENTA + keyword: str = ANSIColors.BOLD_BLUE + keyword_constant: str = ANSIColors.BOLD_BLUE + builtin: str = ANSIColors.CYAN + comment: str = ANSIColors.RED + string: str = ANSIColors.GREEN + number: str = ANSIColors.YELLOW + op: str = ANSIColors.RESET + definition: str = ANSIColors.BOLD + soft_keyword: str = ANSIColors.BOLD_BLUE + reset: str = ANSIColors.RESET + + +@dataclass(frozen=True) +class Traceback(ThemeSection): + type: str = ANSIColors.BOLD_MAGENTA + message: str = ANSIColors.MAGENTA + filename: str = ANSIColors.MAGENTA + line_no: str = ANSIColors.MAGENTA + frame: str = ANSIColors.MAGENTA + error_highlight: str = ANSIColors.BOLD_RED + error_range: str = ANSIColors.RED + reset: str = ANSIColors.RESET + + +@dataclass(frozen=True) +class Unittest(ThemeSection): + passed: str = ANSIColors.GREEN + warn: str = ANSIColors.YELLOW + fail: str = ANSIColors.RED + fail_info: str = ANSIColors.BOLD_RED + reset: str = ANSIColors.RESET + + +@dataclass(frozen=True) +class Theme: + """A suite of themes for all sections of Python. + + When adding a new one, remember to also modify `copy_with` and `no_colors` + below. + """ + argparse: Argparse = field(default_factory=Argparse) + syntax: Syntax = field(default_factory=Syntax) + traceback: Traceback = field(default_factory=Traceback) + unittest: Unittest = field(default_factory=Unittest) + + def copy_with( + self, + *, + argparse: Argparse | None = None, + syntax: Syntax | None = None, + traceback: Traceback | None = None, + unittest: Unittest | None = None, + ) -> Self: + """Return a new Theme based on this instance with some sections replaced. + + Themes are immutable to protect against accidental modifications that + could lead to invalid terminal states. + """ + return type(self)( + argparse=argparse or self.argparse, + syntax=syntax or self.syntax, + traceback=traceback or self.traceback, + unittest=unittest or self.unittest, + ) + + @classmethod + def no_colors(cls) -> Self: + """Return a new Theme where colors in all sections are empty strings. + + This allows writing user code as if colors are always used. The color + fields will be ANSI color code strings when colorization is desired + and possible, and empty strings otherwise. + """ + return cls( + argparse=Argparse.no_colors(), + syntax=Syntax.no_colors(), + traceback=Traceback.no_colors(), + unittest=Unittest.no_colors(), + ) + + def get_colors( colorize: bool = False, *, file: IO[str] | IO[bytes] | None = None ) -> ANSIColors: @@ -76,22 +264,37 @@ def get_colors( return NoColors +def decolor(text: str) -> str: + """Remove ANSI color codes from a string.""" + for code in ColorCodes: + text = text.replace(code, "") + return text + + def can_colorize(*, file: IO[str] | IO[bytes] | None = None) -> bool: + + def _safe_getenv(k: str, fallback: str | None = None) -> str | None: + """Exception-safe environment retrieval. See gh-128636.""" + try: + return os.environ.get(k, fallback) + except Exception: + return fallback + if file is None: file = sys.stdout if not sys.flags.ignore_environment: - if os.environ.get("PYTHON_COLORS") == "0": + if _safe_getenv("PYTHON_COLORS") == "0": return False - if os.environ.get("PYTHON_COLORS") == "1": + if _safe_getenv("PYTHON_COLORS") == "1": return True - if os.environ.get("NO_COLOR"): + if _safe_getenv("NO_COLOR"): return False if not COLORIZE: return False - if os.environ.get("FORCE_COLOR"): + if _safe_getenv("FORCE_COLOR"): return True - if os.environ.get("TERM") == "dumb": + if _safe_getenv("TERM") == "dumb": return False if not hasattr(file, "fileno"): @@ -108,5 +311,45 @@ def can_colorize(*, file: IO[str] | IO[bytes] | None = None) -> bool: try: return os.isatty(file.fileno()) - except io.UnsupportedOperation: + except OSError: return hasattr(file, "isatty") and file.isatty() + + +default_theme = Theme() +theme_no_color = default_theme.no_colors() + + +def get_theme( + *, + tty_file: IO[str] | IO[bytes] | None = None, + force_color: bool = False, + force_no_color: bool = False, +) -> Theme: + """Returns the currently set theme, potentially in a zero-color variant. + + In cases where colorizing is not possible (see `can_colorize`), the returned + theme contains all empty strings in all color definitions. + See `Theme.no_colors()` for more information. + + It is recommended not to cache the result of this function for extended + periods of time because the user might influence theme selection by + the interactive shell, a debugger, or application-specific code. The + environment (including environment variable state and console configuration + on Windows) can also change in the course of the application life cycle. + """ + if force_color or (not force_no_color and + can_colorize(file=tty_file)): + return _theme + return theme_no_color + + +def set_theme(t: Theme) -> None: + global _theme + + if not isinstance(t, Theme): + raise ValueError(f"Expected Theme object, found {t}") + + _theme = t + + +set_theme(default_theme) diff --git a/Lib/_dummy_thread.py b/Lib/_dummy_thread.py index 424b0b3be5e..0630d4e59fa 100644 --- a/Lib/_dummy_thread.py +++ b/Lib/_dummy_thread.py @@ -11,15 +11,35 @@ import _dummy_thread as _thread """ + # Exports only things specified by thread documentation; # skipping obsolete synonyms allocate(), start_new(), exit_thread(). -__all__ = ['error', 'start_new_thread', 'exit', 'get_ident', 'allocate_lock', - 'interrupt_main', 'LockType', 'RLock', - '_count'] +__all__ = [ + "error", + "start_new_thread", + "exit", + "get_ident", + "allocate_lock", + "interrupt_main", + "LockType", + "RLock", + "_count", + "start_joinable_thread", + "daemon_threads_allowed", + "_shutdown", + "_make_thread_handle", + "_ThreadHandle", + "_get_main_thread_ident", + "_is_main_interpreter", + "_local", +] # A dummy value TIMEOUT_MAX = 2**31 +# Main thread ident for dummy implementation +_MAIN_THREAD_IDENT = -1 + # NOTE: this module can be imported early in the extension building process, # and so top level imports of other modules should be avoided. Instead, all # imports are done when needed on a function-by-function basis. Since threads @@ -27,6 +47,7 @@ error = RuntimeError + def start_new_thread(function, args, kwargs={}): """Dummy implementation of _thread.start_new_thread(). @@ -52,6 +73,7 @@ def start_new_thread(function, args, kwargs={}): pass except: import traceback + traceback.print_exc() _main = True global _interrupt @@ -59,10 +81,58 @@ def start_new_thread(function, args, kwargs={}): _interrupt = False raise KeyboardInterrupt + +def start_joinable_thread(function, handle=None, daemon=True): + """Dummy implementation of _thread.start_joinable_thread(). + + In dummy thread, we just run the function synchronously. + """ + if handle is None: + handle = _ThreadHandle() + try: + function() + except SystemExit: + pass + except: + import traceback + + traceback.print_exc() + handle._set_done() + return handle + + +def daemon_threads_allowed(): + """Dummy implementation of _thread.daemon_threads_allowed().""" + return True + + +def _shutdown(): + """Dummy implementation of _thread._shutdown().""" + pass + + +def _make_thread_handle(ident): + """Dummy implementation of _thread._make_thread_handle().""" + handle = _ThreadHandle() + handle._ident = ident + return handle + + +def _get_main_thread_ident(): + """Dummy implementation of _thread._get_main_thread_ident().""" + return _MAIN_THREAD_IDENT + + +def _is_main_interpreter(): + """Dummy implementation of _thread._is_main_interpreter().""" + return True + + def exit(): """Dummy implementation of _thread.exit().""" raise SystemExit + def get_ident(): """Dummy implementation of _thread.get_ident(). @@ -70,26 +140,31 @@ def get_ident(): available, it is safe to assume that the current process is the only thread. Thus a constant can be safely returned. """ - return -1 + return _MAIN_THREAD_IDENT + def allocate_lock(): """Dummy implementation of _thread.allocate_lock().""" return LockType() + def stack_size(size=None): """Dummy implementation of _thread.stack_size().""" if size is not None: raise error("setting thread stack size not supported") return 0 + def _set_sentinel(): """Dummy implementation of _thread._set_sentinel().""" return LockType() + def _count(): """Dummy implementation of _thread._count().""" return 0 + class LockType(object): """Class implementing dummy implementation of _thread.LockType. @@ -125,6 +200,7 @@ def acquire(self, waitflag=None, timeout=-1): else: if timeout > 0: import time + time.sleep(timeout) return False @@ -153,14 +229,41 @@ def __repr__(self): "locked" if self.locked_status else "unlocked", self.__class__.__module__, self.__class__.__qualname__, - hex(id(self)) + hex(id(self)), ) + +class _ThreadHandle: + """Dummy implementation of _thread._ThreadHandle.""" + + def __init__(self): + self._ident = _MAIN_THREAD_IDENT + self._done = False + + @property + def ident(self): + return self._ident + + def _set_done(self): + self._done = True + + def is_done(self): + return self._done + + def join(self, timeout=None): + # In dummy thread, thread is always done + return + + def __repr__(self): + return f"<_ThreadHandle ident={self._ident}>" + + # Used to signal that interrupt_main was called in a "thread" _interrupt = False # True when not executing in a "thread" _main = True + def interrupt_main(): """Set _interrupt flag to True to have start_new_thread raise KeyboardInterrupt upon exiting.""" @@ -170,6 +273,7 @@ def interrupt_main(): global _interrupt _interrupt = True + class RLock: def __init__(self): self.locked_count = 0 @@ -190,7 +294,7 @@ def release(self): return True def locked(self): - return self.locked_status != 0 + return self.locked_count != 0 def __repr__(self): return "<%s %s.%s object owner=%s count=%s at %s>" % ( @@ -199,5 +303,36 @@ def __repr__(self): self.__class__.__qualname__, get_ident() if self.locked_count else 0, self.locked_count, - hex(id(self)) + hex(id(self)), ) + + +class _local: + """Dummy implementation of _thread._local (thread-local storage).""" + + def __init__(self): + object.__setattr__(self, "_local__impl", {}) + + def __getattribute__(self, name): + if name.startswith("_local__"): + return object.__getattribute__(self, name) + impl = object.__getattribute__(self, "_local__impl") + try: + return impl[name] + except KeyError: + raise AttributeError(name) + + def __setattr__(self, name, value): + if name.startswith("_local__"): + return object.__setattr__(self, name, value) + impl = object.__getattribute__(self, "_local__impl") + impl[name] = value + + def __delattr__(self, name): + if name.startswith("_local__"): + return object.__delattr__(self, name) + impl = object.__getattribute__(self, "_local__impl") + try: + del impl[name] + except KeyError: + raise AttributeError(name) diff --git a/Lib/_opcode_metadata.py b/Lib/_opcode_metadata.py index 0d6b7f1109e..bb55ee423cf 100644 --- a/Lib/_opcode_metadata.py +++ b/Lib/_opcode_metadata.py @@ -8,155 +8,243 @@ opmap = { 'CACHE': 0, - 'BEFORE_ASYNC_WITH': 1, - 'BEFORE_WITH': 2, - 'RESERVED_3': 3, - 'BINARY_SLICE': 4, - 'BINARY_SUBSCR': 5, - 'CHECK_EG_MATCH': 6, - 'CHECK_EXC_MATCH': 7, - 'CLEANUP_THROW': 8, - 'DELETE_SUBSCR': 9, - 'END_ASYNC_FOR': 10, - 'END_FOR': 11, - 'END_SEND': 12, - 'EXIT_INIT_CHECK': 13, - 'FORMAT_SIMPLE': 14, - 'FORMAT_WITH_SPEC': 15, - 'GET_AITER': 16, - 'RESERVED_17': 17, - 'GET_ANEXT': 18, - 'GET_ITER': 19, - 'GET_LEN': 20, - 'GET_YIELD_FROM_ITER': 21, - 'INTERPRETER_EXIT': 22, - 'LOAD_ASSERTION_ERROR': 23, - 'LOAD_BUILD_CLASS': 24, - 'LOAD_LOCALS': 25, - 'MAKE_FUNCTION': 26, - 'MATCH_KEYS': 27, - 'MATCH_MAPPING': 28, - 'MATCH_SEQUENCE': 29, - 'NOP': 30, - 'POP_EXCEPT': 31, - 'POP_TOP': 32, - 'PUSH_EXC_INFO': 33, - 'PUSH_NULL': 34, - 'RETURN_GENERATOR': 35, - 'RETURN_VALUE': 36, - 'SETUP_ANNOTATIONS': 37, - 'STORE_SLICE': 38, - 'STORE_SUBSCR': 39, - 'TO_BOOL': 40, - 'UNARY_INVERT': 41, - 'UNARY_NEGATIVE': 42, - 'UNARY_NOT': 43, - 'WITH_EXCEPT_START': 44, - 'BINARY_OP': 45, - 'BUILD_CONST_KEY_MAP': 46, - 'BUILD_LIST': 47, - 'BUILD_MAP': 48, - 'BUILD_SET': 49, - 'BUILD_SLICE': 50, - 'BUILD_STRING': 51, - 'BUILD_TUPLE': 52, - 'CALL': 53, - 'CALL_FUNCTION_EX': 54, - 'CALL_INTRINSIC_1': 55, - 'CALL_INTRINSIC_2': 56, - 'CALL_KW': 57, - 'COMPARE_OP': 58, - 'CONTAINS_OP': 59, - 'CONVERT_VALUE': 60, - 'COPY': 61, - 'COPY_FREE_VARS': 62, - 'DELETE_ATTR': 63, - 'DELETE_DEREF': 64, - 'DELETE_FAST': 65, - 'DELETE_GLOBAL': 66, - 'DELETE_NAME': 67, - 'DICT_MERGE': 68, - 'DICT_UPDATE': 69, - 'ENTER_EXECUTOR': 70, - 'EXTENDED_ARG': 71, - 'FOR_ITER': 72, - 'GET_AWAITABLE': 73, - 'IMPORT_FROM': 74, - 'IMPORT_NAME': 75, - 'IS_OP': 76, - 'JUMP_BACKWARD': 77, - 'JUMP_BACKWARD_NO_INTERRUPT': 78, - 'JUMP_FORWARD': 79, - 'LIST_APPEND': 80, - 'LIST_EXTEND': 81, - 'LOAD_ATTR': 82, - 'LOAD_CONST': 83, - 'LOAD_DEREF': 84, - 'LOAD_FAST': 85, - 'LOAD_FAST_AND_CLEAR': 86, - 'LOAD_FAST_CHECK': 87, - 'LOAD_FAST_LOAD_FAST': 88, - 'LOAD_FROM_DICT_OR_DEREF': 89, - 'LOAD_FROM_DICT_OR_GLOBALS': 90, - 'LOAD_GLOBAL': 91, - 'LOAD_NAME': 92, - 'LOAD_SUPER_ATTR': 93, - 'MAKE_CELL': 94, - 'MAP_ADD': 95, - 'MATCH_CLASS': 96, - 'POP_JUMP_IF_FALSE': 97, - 'POP_JUMP_IF_NONE': 98, - 'POP_JUMP_IF_NOT_NONE': 99, - 'POP_JUMP_IF_TRUE': 100, - 'RAISE_VARARGS': 101, - 'RERAISE': 102, - 'RETURN_CONST': 103, - 'SEND': 104, - 'SET_ADD': 105, - 'SET_FUNCTION_ATTRIBUTE': 106, - 'SET_UPDATE': 107, - 'STORE_ATTR': 108, - 'STORE_DEREF': 109, - 'STORE_FAST': 110, - 'STORE_FAST_LOAD_FAST': 111, - 'STORE_FAST_STORE_FAST': 112, - 'STORE_GLOBAL': 113, - 'STORE_NAME': 114, - 'SWAP': 115, - 'UNPACK_EX': 116, - 'UNPACK_SEQUENCE': 117, - 'YIELD_VALUE': 118, - 'BREAK': 119, - 'BUILD_LIST_UNPACK': 120, - 'BUILD_MAP_FOR_CALL': 121, - 'BUILD_SET_UNPACK': 122, - 'BUILD_TUPLE_ITER': 123, - 'BUILD_TUPLE_UNPACK': 124, - 'CALL_METHOD': 125, - 'CALL_METHOD_KW': 126, - 'CALL_METHOD_EX': 127, - 'CONTINUE': 128, - 'JUMP': 129, - 'JUMP_IF_FALSE_OR_POP': 130, - 'JUMP_IF_TRUE_OR_POP': 131, - 'JUMP_IF_NOT_EXC_MATCH': 132, - 'LOAD_CLASSDEREF': 133, - 'LOAD_CLOSURE': 134, - 'LOAD_METHOD': 135, - 'POP_BLOCK': 136, - 'REVERSE': 137, - 'SET_EXC_INFO': 138, - 'SUBSCRIPT': 139, - 'UNARY_OP': 140, - 'RESERVED_141': 141, - 'RESERVED_142': 142, - 'RESERVED_143': 143, - 'RESERVED_144': 144, - 'RESERVED_145': 145, - 'RESERVED_146': 146, - 'RESERVED_147': 147, - 'RESERVED_148': 148, - 'RESUME': 149, + 'BINARY_SLICE': 1, + 'BUILD_TEMPLATE': 2, + 'BINARY_OP_INPLACE_ADD_UNICODE': 3, + 'CALL_FUNCTION_EX': 4, + 'CHECK_EG_MATCH': 5, + 'CHECK_EXC_MATCH': 6, + 'CLEANUP_THROW': 7, + 'DELETE_SUBSCR': 8, + 'END_FOR': 9, + 'END_SEND': 10, + 'EXIT_INIT_CHECK': 11, + 'FORMAT_SIMPLE': 12, + 'FORMAT_WITH_SPEC': 13, + 'GET_AITER': 14, + 'GET_ANEXT': 15, + 'GET_ITER': 16, + 'RESERVED': 17, + 'GET_LEN': 18, + 'GET_YIELD_FROM_ITER': 19, + 'INTERPRETER_EXIT': 20, + 'LOAD_BUILD_CLASS': 21, + 'LOAD_LOCALS': 22, + 'MAKE_FUNCTION': 23, + 'MATCH_KEYS': 24, + 'MATCH_MAPPING': 25, + 'MATCH_SEQUENCE': 26, + 'NOP': 27, + 'NOT_TAKEN': 28, + 'POP_EXCEPT': 29, + 'POP_ITER': 30, + 'POP_TOP': 31, + 'PUSH_EXC_INFO': 32, + 'PUSH_NULL': 33, + 'RETURN_GENERATOR': 34, + 'RETURN_VALUE': 35, + 'SETUP_ANNOTATIONS': 36, + 'STORE_SLICE': 37, + 'STORE_SUBSCR': 38, + 'TO_BOOL': 39, + 'UNARY_INVERT': 40, + 'UNARY_NEGATIVE': 41, + 'UNARY_NOT': 42, + 'WITH_EXCEPT_START': 43, + 'BINARY_OP': 44, + 'BUILD_INTERPOLATION': 45, + 'BUILD_LIST': 46, + 'BUILD_MAP': 47, + 'BUILD_SET': 48, + 'BUILD_SLICE': 49, + 'BUILD_STRING': 50, + 'BUILD_TUPLE': 51, + 'CALL': 52, + 'CALL_INTRINSIC_1': 53, + 'CALL_INTRINSIC_2': 54, + 'CALL_KW': 55, + 'COMPARE_OP': 56, + 'CONTAINS_OP': 57, + 'CONVERT_VALUE': 58, + 'COPY': 59, + 'COPY_FREE_VARS': 60, + 'DELETE_ATTR': 61, + 'DELETE_DEREF': 62, + 'DELETE_FAST': 63, + 'DELETE_GLOBAL': 64, + 'DELETE_NAME': 65, + 'DICT_MERGE': 66, + 'DICT_UPDATE': 67, + 'END_ASYNC_FOR': 68, + 'EXTENDED_ARG': 69, + 'FOR_ITER': 70, + 'GET_AWAITABLE': 71, + 'IMPORT_FROM': 72, + 'IMPORT_NAME': 73, + 'IS_OP': 74, + 'JUMP_BACKWARD': 75, + 'JUMP_BACKWARD_NO_INTERRUPT': 76, + 'JUMP_FORWARD': 77, + 'LIST_APPEND': 78, + 'LIST_EXTEND': 79, + 'LOAD_ATTR': 80, + 'LOAD_COMMON_CONSTANT': 81, + 'LOAD_CONST': 82, + 'LOAD_DEREF': 83, + 'LOAD_FAST': 84, + 'LOAD_FAST_AND_CLEAR': 85, + 'LOAD_FAST_BORROW': 86, + 'LOAD_FAST_BORROW_LOAD_FAST_BORROW': 87, + 'LOAD_FAST_CHECK': 88, + 'LOAD_FAST_LOAD_FAST': 89, + 'LOAD_FROM_DICT_OR_DEREF': 90, + 'LOAD_FROM_DICT_OR_GLOBALS': 91, + 'LOAD_GLOBAL': 92, + 'LOAD_NAME': 93, + 'LOAD_SMALL_INT': 94, + 'LOAD_SPECIAL': 95, + 'LOAD_SUPER_ATTR': 96, + 'MAKE_CELL': 97, + 'MAP_ADD': 98, + 'MATCH_CLASS': 99, + 'POP_JUMP_IF_FALSE': 100, + 'POP_JUMP_IF_NONE': 101, + 'POP_JUMP_IF_NOT_NONE': 102, + 'POP_JUMP_IF_TRUE': 103, + 'RAISE_VARARGS': 104, + 'RERAISE': 105, + 'SEND': 106, + 'SET_ADD': 107, + 'SET_FUNCTION_ATTRIBUTE': 108, + 'SET_UPDATE': 109, + 'STORE_ATTR': 110, + 'STORE_DEREF': 111, + 'STORE_FAST': 112, + 'STORE_FAST_LOAD_FAST': 113, + 'STORE_FAST_STORE_FAST': 114, + 'STORE_GLOBAL': 115, + 'STORE_NAME': 116, + 'SWAP': 117, + 'UNPACK_EX': 118, + 'UNPACK_SEQUENCE': 119, + 'YIELD_VALUE': 120, + 'RESUME': 128, + 'BINARY_OP_ADD_FLOAT': 129, + 'BINARY_OP_ADD_INT': 130, + 'BINARY_OP_ADD_UNICODE': 131, + 'BINARY_OP_EXTEND': 132, + 'BINARY_OP_MULTIPLY_FLOAT': 133, + 'BINARY_OP_MULTIPLY_INT': 134, + 'BINARY_OP_SUBSCR_DICT': 135, + 'BINARY_OP_SUBSCR_GETITEM': 136, + 'BINARY_OP_SUBSCR_LIST_INT': 137, + 'BINARY_OP_SUBSCR_LIST_SLICE': 138, + 'BINARY_OP_SUBSCR_STR_INT': 139, + 'BINARY_OP_SUBSCR_TUPLE_INT': 140, + 'BINARY_OP_SUBTRACT_FLOAT': 141, + 'BINARY_OP_SUBTRACT_INT': 142, + 'CALL_ALLOC_AND_ENTER_INIT': 143, + 'CALL_BOUND_METHOD_EXACT_ARGS': 144, + 'CALL_BOUND_METHOD_GENERAL': 145, + 'CALL_BUILTIN_CLASS': 146, + 'CALL_BUILTIN_FAST': 147, + 'CALL_BUILTIN_FAST_WITH_KEYWORDS': 148, + 'CALL_BUILTIN_O': 149, + 'CALL_ISINSTANCE': 150, + 'CALL_KW_BOUND_METHOD': 151, + 'CALL_KW_NON_PY': 152, + 'CALL_KW_PY': 153, + 'CALL_LEN': 154, + 'CALL_LIST_APPEND': 155, + 'CALL_METHOD_DESCRIPTOR_FAST': 156, + 'CALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS': 157, + 'CALL_METHOD_DESCRIPTOR_NOARGS': 158, + 'CALL_METHOD_DESCRIPTOR_O': 159, + 'CALL_NON_PY_GENERAL': 160, + 'CALL_PY_EXACT_ARGS': 161, + 'CALL_PY_GENERAL': 162, + 'CALL_STR_1': 163, + 'CALL_TUPLE_1': 164, + 'CALL_TYPE_1': 165, + 'COMPARE_OP_FLOAT': 166, + 'COMPARE_OP_INT': 167, + 'COMPARE_OP_STR': 168, + 'CONTAINS_OP_DICT': 169, + 'CONTAINS_OP_SET': 170, + 'FOR_ITER_GEN': 171, + 'FOR_ITER_LIST': 172, + 'FOR_ITER_RANGE': 173, + 'FOR_ITER_TUPLE': 174, + 'JUMP_BACKWARD_JIT': 175, + 'JUMP_BACKWARD_NO_JIT': 176, + 'LOAD_ATTR_CLASS': 177, + 'LOAD_ATTR_CLASS_WITH_METACLASS_CHECK': 178, + 'LOAD_ATTR_GETATTRIBUTE_OVERRIDDEN': 179, + 'LOAD_ATTR_INSTANCE_VALUE': 180, + 'LOAD_ATTR_METHOD_LAZY_DICT': 181, + 'LOAD_ATTR_METHOD_NO_DICT': 182, + 'LOAD_ATTR_METHOD_WITH_VALUES': 183, + 'LOAD_ATTR_MODULE': 184, + 'LOAD_ATTR_NONDESCRIPTOR_NO_DICT': 185, + 'LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES': 186, + 'LOAD_ATTR_PROPERTY': 187, + 'LOAD_ATTR_SLOT': 188, + 'LOAD_ATTR_WITH_HINT': 189, + 'LOAD_CONST_IMMORTAL': 190, + 'LOAD_CONST_MORTAL': 191, + 'LOAD_GLOBAL_BUILTIN': 192, + 'LOAD_GLOBAL_MODULE': 193, + 'LOAD_SUPER_ATTR_ATTR': 194, + 'LOAD_SUPER_ATTR_METHOD': 195, + 'RESUME_CHECK': 196, + 'SEND_GEN': 197, + 'STORE_ATTR_INSTANCE_VALUE': 198, + 'STORE_ATTR_SLOT': 199, + 'STORE_ATTR_WITH_HINT': 200, + 'STORE_SUBSCR_DICT': 201, + 'STORE_SUBSCR_LIST_INT': 202, + 'TO_BOOL_ALWAYS_TRUE': 203, + 'TO_BOOL_BOOL': 204, + 'TO_BOOL_INT': 205, + 'TO_BOOL_LIST': 206, + 'TO_BOOL_NONE': 207, + 'TO_BOOL_STR': 208, + 'UNPACK_SEQUENCE_LIST': 209, + 'UNPACK_SEQUENCE_TUPLE': 210, + 'UNPACK_SEQUENCE_TWO_TUPLE': 211, + 'INSTRUMENTED_END_FOR': 234, + 'INSTRUMENTED_POP_ITER': 235, + 'INSTRUMENTED_END_SEND': 236, + 'INSTRUMENTED_FOR_ITER': 237, + 'INSTRUMENTED_INSTRUCTION': 238, + 'INSTRUMENTED_JUMP_FORWARD': 239, + 'INSTRUMENTED_NOT_TAKEN': 240, + 'INSTRUMENTED_POP_JUMP_IF_TRUE': 241, + 'INSTRUMENTED_POP_JUMP_IF_FALSE': 242, + 'INSTRUMENTED_POP_JUMP_IF_NONE': 243, + 'INSTRUMENTED_POP_JUMP_IF_NOT_NONE': 244, + 'INSTRUMENTED_RESUME': 245, + 'INSTRUMENTED_RETURN_VALUE': 246, + 'INSTRUMENTED_YIELD_VALUE': 247, + 'INSTRUMENTED_END_ASYNC_FOR': 248, + 'INSTRUMENTED_LOAD_SUPER_ATTR': 249, + 'INSTRUMENTED_CALL': 250, + 'INSTRUMENTED_CALL_KW': 251, + 'INSTRUMENTED_CALL_FUNCTION_EX': 252, + 'INSTRUMENTED_JUMP_BACKWARD': 253, + 'INSTRUMENTED_LINE': 254, + 'ENTER_EXECUTOR': 255, + 'ANNOTATIONS_PLACEHOLDER': 256, + 'JUMP': 257, + 'JUMP_IF_FALSE': 258, + 'JUMP_IF_TRUE': 259, + 'JUMP_NO_INTERRUPT': 260, + 'LOAD_CLOSURE': 261, + 'POP_BLOCK': 262, + 'SETUP_CLEANUP': 263, + 'SETUP_FINALLY': 264, + 'SETUP_WITH': 265, + 'STORE_FAST_MAYBE_NULL': 266, } # CPython 3.13 compatible: opcodes < 44 have no argument diff --git a/Lib/_py_warnings.py b/Lib/_py_warnings.py new file mode 100644 index 00000000000..55f8c069591 --- /dev/null +++ b/Lib/_py_warnings.py @@ -0,0 +1,869 @@ +"""Python part of the warnings subsystem.""" + +import sys +import _contextvars +import _thread + + +__all__ = ["warn", "warn_explicit", "showwarning", + "formatwarning", "filterwarnings", "simplefilter", + "resetwarnings", "catch_warnings", "deprecated"] + + +# Normally '_wm' is sys.modules['warnings'] but for unit tests it can be +# a different module. User code is allowed to reassign global attributes +# of the 'warnings' module, commonly 'filters' or 'showwarning'. So we +# need to lookup these global attributes dynamically on the '_wm' object, +# rather than binding them earlier. The code in this module consistently uses +# '_wm.' rather than using the globals of this module. If the +# '_warnings' C extension is in use, some globals are replaced by functions +# and variables defined in that extension. +_wm = None + + +def _set_module(module): + global _wm + _wm = module + + +# filters contains a sequence of filter 5-tuples +# The components of the 5-tuple are: +# - an action: error, ignore, always, all, default, module, or once +# - a compiled regex that must match the warning message +# - a class representing the warning category +# - a compiled regex that must match the module that is being warned +# - a line number for the line being warning, or 0 to mean any line +# If either if the compiled regexs are None, match anything. +filters = [] + + +defaultaction = "default" +onceregistry = {} +_lock = _thread.RLock() +_filters_version = 1 + + +# If true, catch_warnings() will use a context var to hold the modified +# filters list. Otherwise, catch_warnings() will operate on the 'filters' +# global of the warnings module. +_use_context = sys.flags.context_aware_warnings + + +class _Context: + def __init__(self, filters): + self._filters = filters + self.log = None # if set to a list, logging is enabled + + def copy(self): + context = _Context(self._filters[:]) + if self.log is not None: + context.log = self.log + return context + + def _record_warning(self, msg): + self.log.append(msg) + + +class _GlobalContext(_Context): + def __init__(self): + self.log = None + + @property + def _filters(self): + # Since there is quite a lot of code that assigns to + # warnings.filters, this needs to return the current value of + # the module global. + try: + return _wm.filters + except AttributeError: + # 'filters' global was deleted. Do we need to actually handle this case? + return [] + + +_global_context = _GlobalContext() + + +_warnings_context = _contextvars.ContextVar('warnings_context') + + +def _get_context(): + if not _use_context: + return _global_context + try: + return _wm._warnings_context.get() + except LookupError: + return _global_context + + +def _set_context(context): + assert _use_context + _wm._warnings_context.set(context) + + +def _new_context(): + assert _use_context + old_context = _wm._get_context() + new_context = old_context.copy() + _wm._set_context(new_context) + return old_context, new_context + + +def _get_filters(): + """Return the current list of filters. This is a non-public API used by + module functions and by the unit tests.""" + return _wm._get_context()._filters + + +def _filters_mutated_lock_held(): + _wm._filters_version += 1 + + +def showwarning(message, category, filename, lineno, file=None, line=None): + """Hook to write a warning to a file; replace if you like.""" + msg = _wm.WarningMessage(message, category, filename, lineno, file, line) + _wm._showwarnmsg_impl(msg) + + +def formatwarning(message, category, filename, lineno, line=None): + """Function to format a warning the standard way.""" + msg = _wm.WarningMessage(message, category, filename, lineno, None, line) + return _wm._formatwarnmsg_impl(msg) + + +def _showwarnmsg_impl(msg): + context = _wm._get_context() + if context.log is not None: + context._record_warning(msg) + return + file = msg.file + if file is None: + file = sys.stderr + if file is None: + # sys.stderr is None when run with pythonw.exe: + # warnings get lost + return + text = _wm._formatwarnmsg(msg) + try: + file.write(text) + except OSError: + # the file (probably stderr) is invalid - this warning gets lost. + pass + + +def _formatwarnmsg_impl(msg): + category = msg.category.__name__ + s = f"{msg.filename}:{msg.lineno}: {category}: {msg.message}\n" + + if msg.line is None: + try: + import linecache + line = linecache.getline(msg.filename, msg.lineno) + except Exception: + # When a warning is logged during Python shutdown, linecache + # and the import machinery don't work anymore + line = None + linecache = None + else: + line = msg.line + if line: + line = line.strip() + s += " %s\n" % line + + if msg.source is not None: + try: + import tracemalloc + # Logging a warning should not raise a new exception: + # catch Exception, not only ImportError and RecursionError. + except Exception: + # don't suggest to enable tracemalloc if it's not available + suggest_tracemalloc = False + tb = None + else: + try: + suggest_tracemalloc = not tracemalloc.is_tracing() + tb = tracemalloc.get_object_traceback(msg.source) + except Exception: + # When a warning is logged during Python shutdown, tracemalloc + # and the import machinery don't work anymore + suggest_tracemalloc = False + tb = None + + if tb is not None: + s += 'Object allocated at (most recent call last):\n' + for frame in tb: + s += (' File "%s", lineno %s\n' + % (frame.filename, frame.lineno)) + + try: + if linecache is not None: + line = linecache.getline(frame.filename, frame.lineno) + else: + line = None + except Exception: + line = None + if line: + line = line.strip() + s += ' %s\n' % line + elif suggest_tracemalloc: + s += (f'{category}: Enable tracemalloc to get the object ' + f'allocation traceback\n') + return s + + +# Keep a reference to check if the function was replaced +_showwarning_orig = showwarning + + +def _showwarnmsg(msg): + """Hook to write a warning to a file; replace if you like.""" + try: + sw = _wm.showwarning + except AttributeError: + pass + else: + if sw is not _showwarning_orig: + # warnings.showwarning() was replaced + if not callable(sw): + raise TypeError("warnings.showwarning() must be set to a " + "function or method") + + sw(msg.message, msg.category, msg.filename, msg.lineno, + msg.file, msg.line) + return + _wm._showwarnmsg_impl(msg) + + +# Keep a reference to check if the function was replaced +_formatwarning_orig = formatwarning + + +def _formatwarnmsg(msg): + """Function to format a warning the standard way.""" + try: + fw = _wm.formatwarning + except AttributeError: + pass + else: + if fw is not _formatwarning_orig: + # warnings.formatwarning() was replaced + return fw(msg.message, msg.category, + msg.filename, msg.lineno, msg.line) + return _wm._formatwarnmsg_impl(msg) + + +def filterwarnings(action, message="", category=Warning, module="", lineno=0, + append=False): + """Insert an entry into the list of warnings filters (at the front). + + 'action' -- one of "error", "ignore", "always", "all", "default", "module", + or "once" + 'message' -- a regex that the warning message must match + 'category' -- a class that the warning must be a subclass of + 'module' -- a regex that the module name must match + 'lineno' -- an integer line number, 0 matches all warnings + 'append' -- if true, append to the list of filters + """ + if action not in {"error", "ignore", "always", "all", "default", "module", "once"}: + raise ValueError(f"invalid action: {action!r}") + if not isinstance(message, str): + raise TypeError("message must be a string") + if not isinstance(category, type) or not issubclass(category, Warning): + raise TypeError("category must be a Warning subclass") + if not isinstance(module, str): + raise TypeError("module must be a string") + if not isinstance(lineno, int): + raise TypeError("lineno must be an int") + if lineno < 0: + raise ValueError("lineno must be an int >= 0") + + if message or module: + import re + + if message: + message = re.compile(message, re.I) + else: + message = None + if module: + module = re.compile(module) + else: + module = None + + _wm._add_filter(action, message, category, module, lineno, append=append) + + +def simplefilter(action, category=Warning, lineno=0, append=False): + """Insert a simple entry into the list of warnings filters (at the front). + + A simple filter matches all modules and messages. + 'action' -- one of "error", "ignore", "always", "all", "default", "module", + or "once" + 'category' -- a class that the warning must be a subclass of + 'lineno' -- an integer line number, 0 matches all warnings + 'append' -- if true, append to the list of filters + """ + if action not in {"error", "ignore", "always", "all", "default", "module", "once"}: + raise ValueError(f"invalid action: {action!r}") + if not isinstance(lineno, int): + raise TypeError("lineno must be an int") + if lineno < 0: + raise ValueError("lineno must be an int >= 0") + _wm._add_filter(action, None, category, None, lineno, append=append) + + +def _filters_mutated(): + # Even though this function is not part of the public API, it's used by + # a fair amount of user code. + with _wm._lock: + _wm._filters_mutated_lock_held() + + +def _add_filter(*item, append): + with _wm._lock: + filters = _wm._get_filters() + if not append: + # Remove possible duplicate filters, so new one will be placed + # in correct place. If append=True and duplicate exists, do nothing. + try: + filters.remove(item) + except ValueError: + pass + filters.insert(0, item) + else: + if item not in filters: + filters.append(item) + _wm._filters_mutated_lock_held() + + +def resetwarnings(): + """Clear the list of warning filters, so that no filters are active.""" + with _wm._lock: + del _wm._get_filters()[:] + _wm._filters_mutated_lock_held() + + +class _OptionError(Exception): + """Exception used by option processing helpers.""" + pass + + +# Helper to process -W options passed via sys.warnoptions +def _processoptions(args): + for arg in args: + try: + _wm._setoption(arg) + except _wm._OptionError as msg: + print("Invalid -W option ignored:", msg, file=sys.stderr) + + +# Helper for _processoptions() +def _setoption(arg): + parts = arg.split(':') + if len(parts) > 5: + raise _wm._OptionError("too many fields (max 5): %r" % (arg,)) + while len(parts) < 5: + parts.append('') + action, message, category, module, lineno = [s.strip() + for s in parts] + action = _wm._getaction(action) + category = _wm._getcategory(category) + if message or module: + import re + if message: + message = re.escape(message) + if module: + module = re.escape(module) + r'\z' + if lineno: + try: + lineno = int(lineno) + if lineno < 0: + raise ValueError + except (ValueError, OverflowError): + raise _wm._OptionError("invalid lineno %r" % (lineno,)) from None + else: + lineno = 0 + _wm.filterwarnings(action, message, category, module, lineno) + + +# Helper for _setoption() +def _getaction(action): + if not action: + return "default" + for a in ('default', 'always', 'all', 'ignore', 'module', 'once', 'error'): + if a.startswith(action): + return a + raise _wm._OptionError("invalid action: %r" % (action,)) + + +# Helper for _setoption() +def _getcategory(category): + if not category: + return Warning + if '.' not in category: + import builtins as m + klass = category + else: + module, _, klass = category.rpartition('.') + try: + m = __import__(module, None, None, [klass]) + except ImportError: + raise _wm._OptionError("invalid module name: %r" % (module,)) from None + try: + cat = getattr(m, klass) + except AttributeError: + raise _wm._OptionError("unknown warning category: %r" % (category,)) from None + if not issubclass(cat, Warning): + raise _wm._OptionError("invalid warning category: %r" % (category,)) + return cat + + +def _is_internal_filename(filename): + return 'importlib' in filename and '_bootstrap' in filename + + +def _is_filename_to_skip(filename, skip_file_prefixes): + return any(filename.startswith(prefix) for prefix in skip_file_prefixes) + + +def _is_internal_frame(frame): + """Signal whether the frame is an internal CPython implementation detail.""" + return _is_internal_filename(frame.f_code.co_filename) + + +def _next_external_frame(frame, skip_file_prefixes): + """Find the next frame that doesn't involve Python or user internals.""" + frame = frame.f_back + while frame is not None and ( + _is_internal_filename(filename := frame.f_code.co_filename) or + _is_filename_to_skip(filename, skip_file_prefixes)): + frame = frame.f_back + return frame + + +# Code typically replaced by _warnings +def warn(message, category=None, stacklevel=1, source=None, + *, skip_file_prefixes=()): + """Issue a warning, or maybe ignore it or raise an exception.""" + # Check if message is already a Warning object + if isinstance(message, Warning): + category = message.__class__ + # Check category argument + if category is None: + category = UserWarning + if not (isinstance(category, type) and issubclass(category, Warning)): + raise TypeError("category must be a Warning subclass, " + "not '{:s}'".format(type(category).__name__)) + if not isinstance(skip_file_prefixes, tuple): + # The C version demands a tuple for implementation performance. + raise TypeError('skip_file_prefixes must be a tuple of strs.') + if skip_file_prefixes: + stacklevel = max(2, stacklevel) + # Get context information + try: + if stacklevel <= 1 or _is_internal_frame(sys._getframe(1)): + # If frame is too small to care or if the warning originated in + # internal code, then do not try to hide any frames. + frame = sys._getframe(stacklevel) + else: + frame = sys._getframe(1) + # Look for one frame less since the above line starts us off. + for x in range(stacklevel-1): + frame = _next_external_frame(frame, skip_file_prefixes) + if frame is None: + raise ValueError + except ValueError: + globals = sys.__dict__ + filename = "" + lineno = 0 + else: + globals = frame.f_globals + filename = frame.f_code.co_filename + lineno = frame.f_lineno + if '__name__' in globals: + module = globals['__name__'] + else: + module = "" + registry = globals.setdefault("__warningregistry__", {}) + _wm.warn_explicit( + message, + category, + filename, + lineno, + module, + registry, + globals, + source=source, + ) + + +def warn_explicit(message, category, filename, lineno, + module=None, registry=None, module_globals=None, + source=None): + lineno = int(lineno) + if module is None: + module = filename or "" + if module[-3:].lower() == ".py": + module = module[:-3] # XXX What about leading pathname? + if isinstance(message, Warning): + text = str(message) + category = message.__class__ + else: + text = message + message = category(message) + key = (text, category, lineno) + with _wm._lock: + if registry is None: + registry = {} + if registry.get('version', 0) != _wm._filters_version: + registry.clear() + registry['version'] = _wm._filters_version + # Quick test for common case + if registry.get(key): + return + # Search the filters + for item in _wm._get_filters(): + action, msg, cat, mod, ln = item + if ((msg is None or msg.match(text)) and + issubclass(category, cat) and + (mod is None or mod.match(module)) and + (ln == 0 or lineno == ln)): + break + else: + action = _wm.defaultaction + # Early exit actions + if action == "ignore": + return + + if action == "error": + raise message + # Other actions + if action == "once": + registry[key] = 1 + oncekey = (text, category) + if _wm.onceregistry.get(oncekey): + return + _wm.onceregistry[oncekey] = 1 + elif action in {"always", "all"}: + pass + elif action == "module": + registry[key] = 1 + altkey = (text, category, 0) + if registry.get(altkey): + return + registry[altkey] = 1 + elif action == "default": + registry[key] = 1 + else: + # Unrecognized actions are errors + raise RuntimeError( + "Unrecognized action (%r) in warnings.filters:\n %s" % + (action, item)) + + # Prime the linecache for formatting, in case the + # "file" is actually in a zipfile or something. + import linecache + linecache.getlines(filename, module_globals) + + # Print message and context + msg = _wm.WarningMessage(message, category, filename, lineno, source=source) + _wm._showwarnmsg(msg) + + +class WarningMessage(object): + + _WARNING_DETAILS = ("message", "category", "filename", "lineno", "file", + "line", "source") + + def __init__(self, message, category, filename, lineno, file=None, + line=None, source=None): + self.message = message + self.category = category + self.filename = filename + self.lineno = lineno + self.file = file + self.line = line + self.source = source + self._category_name = category.__name__ if category else None + + def __str__(self): + return ("{message : %r, category : %r, filename : %r, lineno : %s, " + "line : %r}" % (self.message, self._category_name, + self.filename, self.lineno, self.line)) + + def __repr__(self): + return f'<{type(self).__qualname__} {self}>' + + +class catch_warnings(object): + + """A context manager that copies and restores the warnings filter upon + exiting the context. + + The 'record' argument specifies whether warnings should be captured by a + custom implementation of warnings.showwarning() and be appended to a list + returned by the context manager. Otherwise None is returned by the context + manager. The objects appended to the list are arguments whose attributes + mirror the arguments to showwarning(). + + The 'module' argument is to specify an alternative module to the module + named 'warnings' and imported under that name. This argument is only useful + when testing the warnings module itself. + + If the 'action' argument is not None, the remaining arguments are passed + to warnings.simplefilter() as if it were called immediately on entering the + context. + """ + + def __init__(self, *, record=False, module=None, + action=None, category=Warning, lineno=0, append=False): + """Specify whether to record warnings and if an alternative module + should be used other than sys.modules['warnings']. + + """ + self._record = record + self._module = sys.modules['warnings'] if module is None else module + self._entered = False + if action is None: + self._filter = None + else: + self._filter = (action, category, lineno, append) + + def __repr__(self): + args = [] + if self._record: + args.append("record=True") + if self._module is not sys.modules['warnings']: + args.append("module=%r" % self._module) + name = type(self).__name__ + return "%s(%s)" % (name, ", ".join(args)) + + def __enter__(self): + if self._entered: + raise RuntimeError("Cannot enter %r twice" % self) + self._entered = True + with _wm._lock: + if _use_context: + self._saved_context, context = self._module._new_context() + else: + context = None + self._filters = self._module.filters + self._module.filters = self._filters[:] + self._showwarning = self._module.showwarning + self._showwarnmsg_impl = self._module._showwarnmsg_impl + self._module._filters_mutated_lock_held() + if self._record: + if _use_context: + context.log = log = [] + else: + log = [] + self._module._showwarnmsg_impl = log.append + # Reset showwarning() to the default implementation to make sure + # that _showwarnmsg() calls _showwarnmsg_impl() + self._module.showwarning = self._module._showwarning_orig + else: + log = None + if self._filter is not None: + self._module.simplefilter(*self._filter) + return log + + def __exit__(self, *exc_info): + if not self._entered: + raise RuntimeError("Cannot exit %r without entering first" % self) + with _wm._lock: + if _use_context: + self._module._warnings_context.set(self._saved_context) + else: + self._module.filters = self._filters + self._module.showwarning = self._showwarning + self._module._showwarnmsg_impl = self._showwarnmsg_impl + self._module._filters_mutated_lock_held() + + +class deprecated: + """Indicate that a class, function or overload is deprecated. + + When this decorator is applied to an object, the type checker + will generate a diagnostic on usage of the deprecated object. + + Usage: + + @deprecated("Use B instead") + class A: + pass + + @deprecated("Use g instead") + def f(): + pass + + @overload + @deprecated("int support is deprecated") + def g(x: int) -> int: ... + @overload + def g(x: str) -> int: ... + + The warning specified by *category* will be emitted at runtime + on use of deprecated objects. For functions, that happens on calls; + for classes, on instantiation and on creation of subclasses. + If the *category* is ``None``, no warning is emitted at runtime. + The *stacklevel* determines where the + warning is emitted. If it is ``1`` (the default), the warning + is emitted at the direct caller of the deprecated object; if it + is higher, it is emitted further up the stack. + Static type checker behavior is not affected by the *category* + and *stacklevel* arguments. + + The deprecation message passed to the decorator is saved in the + ``__deprecated__`` attribute on the decorated object. + If applied to an overload, the decorator + must be after the ``@overload`` decorator for the attribute to + exist on the overload as returned by ``get_overloads()``. + + See PEP 702 for details. + + """ + def __init__( + self, + message: str, + /, + *, + category: type[Warning] | None = DeprecationWarning, + stacklevel: int = 1, + ) -> None: + if not isinstance(message, str): + raise TypeError( + f"Expected an object of type str for 'message', not {type(message).__name__!r}" + ) + self.message = message + self.category = category + self.stacklevel = stacklevel + + def __call__(self, arg, /): + # Make sure the inner functions created below don't + # retain a reference to self. + msg = self.message + category = self.category + stacklevel = self.stacklevel + if category is None: + arg.__deprecated__ = msg + return arg + elif isinstance(arg, type): + import functools + from types import MethodType + + original_new = arg.__new__ + + @functools.wraps(original_new) + def __new__(cls, /, *args, **kwargs): + if cls is arg: + _wm.warn(msg, category=category, stacklevel=stacklevel + 1) + if original_new is not object.__new__: + return original_new(cls, *args, **kwargs) + # Mirrors a similar check in object.__new__. + elif cls.__init__ is object.__init__ and (args or kwargs): + raise TypeError(f"{cls.__name__}() takes no arguments") + else: + return original_new(cls) + + arg.__new__ = staticmethod(__new__) + + if "__init_subclass__" in arg.__dict__: + # __init_subclass__ is directly present on the decorated class. + # Synthesize a wrapper that calls this method directly. + original_init_subclass = arg.__init_subclass__ + # We need slightly different behavior if __init_subclass__ + # is a bound method (likely if it was implemented in Python). + # Otherwise, it likely means it's a builtin such as + # object's implementation of __init_subclass__. + if isinstance(original_init_subclass, MethodType): + original_init_subclass = original_init_subclass.__func__ + + @functools.wraps(original_init_subclass) + def __init_subclass__(*args, **kwargs): + _wm.warn(msg, category=category, stacklevel=stacklevel + 1) + return original_init_subclass(*args, **kwargs) + else: + def __init_subclass__(cls, *args, **kwargs): + _wm.warn(msg, category=category, stacklevel=stacklevel + 1) + return super(arg, cls).__init_subclass__(*args, **kwargs) + + arg.__init_subclass__ = classmethod(__init_subclass__) + + arg.__deprecated__ = __new__.__deprecated__ = msg + __init_subclass__.__deprecated__ = msg + return arg + elif callable(arg): + import functools + import inspect + + @functools.wraps(arg) + def wrapper(*args, **kwargs): + _wm.warn(msg, category=category, stacklevel=stacklevel + 1) + return arg(*args, **kwargs) + + if inspect.iscoroutinefunction(arg): + wrapper = inspect.markcoroutinefunction(wrapper) + + arg.__deprecated__ = wrapper.__deprecated__ = msg + return wrapper + else: + raise TypeError( + "@deprecated decorator with non-None category must be applied to " + f"a class or callable, not {arg!r}" + ) + + +_DEPRECATED_MSG = "{name!r} is deprecated and slated for removal in Python {remove}" + + +def _deprecated(name, message=_DEPRECATED_MSG, *, remove, _version=sys.version_info): + """Warn that *name* is deprecated or should be removed. + + RuntimeError is raised if *remove* specifies a major/minor tuple older than + the current Python version or the same version but past the alpha. + + The *message* argument is formatted with *name* and *remove* as a Python + version tuple (e.g. (3, 11)). + + """ + remove_formatted = f"{remove[0]}.{remove[1]}" + if (_version[:2] > remove) or (_version[:2] == remove and _version[3] != "alpha"): + msg = f"{name!r} was slated for removal after Python {remove_formatted} alpha" + raise RuntimeError(msg) + else: + msg = message.format(name=name, remove=remove_formatted) + _wm.warn(msg, DeprecationWarning, stacklevel=3) + + +# Private utility function called by _PyErr_WarnUnawaitedCoroutine +def _warn_unawaited_coroutine(coro): + msg_lines = [ + f"coroutine '{coro.__qualname__}' was never awaited\n" + ] + if coro.cr_origin is not None: + import linecache, traceback + def extract(): + for filename, lineno, funcname in reversed(coro.cr_origin): + line = linecache.getline(filename, lineno) + yield (filename, lineno, funcname, line) + msg_lines.append("Coroutine created at (most recent call last)\n") + msg_lines += traceback.format_list(list(extract())) + msg = "".join(msg_lines).rstrip("\n") + # Passing source= here means that if the user happens to have tracemalloc + # enabled and tracking where the coroutine was created, the warning will + # contain that traceback. This does mean that if they have *both* + # coroutine origin tracking *and* tracemalloc enabled, they'll get two + # partially-redundant tracebacks. If we wanted to be clever we could + # probably detect this case and avoid it, but for now we don't bother. + _wm.warn( + msg, category=RuntimeWarning, stacklevel=2, source=coro + ) + + +def _setup_defaults(): + # Several warning categories are ignored by default in regular builds + if hasattr(sys, 'gettotalrefcount'): + return + _wm.filterwarnings("default", category=DeprecationWarning, module="__main__", append=1) + _wm.simplefilter("ignore", category=DeprecationWarning, append=1) + _wm.simplefilter("ignore", category=PendingDeprecationWarning, append=1) + _wm.simplefilter("ignore", category=ImportWarning, append=1) + _wm.simplefilter("ignore", category=ResourceWarning, append=1) diff --git a/Lib/_pycodecs.py b/Lib/_pycodecs.py index d0efa9ad6bb..933d0e2ac71 100644 --- a/Lib/_pycodecs.py +++ b/Lib/_pycodecs.py @@ -357,6 +357,145 @@ def utf_16_be_decode( data, errors='strict', byteorder=0, final = 0): return res, consumed +def STORECHAR32(ch, byteorder): + """Store a 32-bit character as 4 bytes in the specified byte order.""" + b0 = ch & 0xff + b1 = (ch >> 8) & 0xff + b2 = (ch >> 16) & 0xff + b3 = (ch >> 24) & 0xff + if byteorder == 'little': + return [b0, b1, b2, b3] + else: # big-endian + return [b3, b2, b1, b0] + + +def PyUnicode_EncodeUTF32(s, size, errors, byteorder='little'): + """Encode a Unicode string to UTF-32.""" + p = [] + bom = sys.byteorder + + if byteorder == 'native': + bom = sys.byteorder + # Add BOM for native encoding + p += STORECHAR32(0xFEFF, bom) + + if size == 0: + return [] + + if byteorder == 'little': + bom = 'little' + elif byteorder == 'big': + bom = 'big' + + for c in s: + ch = ord(c) + # UTF-32 doesn't need surrogate pairs, each character is encoded directly + p += STORECHAR32(ch, bom) + + return p + + +def utf_32_encode(obj, errors='strict'): + """UTF-32 encoding with BOM.""" + res = PyUnicode_EncodeUTF32(obj, len(obj), errors, 'native') + res = bytes(res) + return res, len(obj) + + +def utf_32_le_encode(obj, errors='strict'): + """UTF-32 little-endian encoding without BOM.""" + res = PyUnicode_EncodeUTF32(obj, len(obj), errors, 'little') + res = bytes(res) + return res, len(obj) + + +def utf_32_be_encode(obj, errors='strict'): + """UTF-32 big-endian encoding without BOM.""" + res = PyUnicode_EncodeUTF32(obj, len(obj), errors, 'big') + res = bytes(res) + return res, len(obj) + + +def PyUnicode_DecodeUTF32Stateful(data, size, errors, byteorder='little', final=0): + """Decode UTF-32 encoded bytes to Unicode string.""" + if size == 0: + return [], 0, 0 + + if size % 4 != 0: + if not final: + # Incomplete data, return what we can decode + size = (size // 4) * 4 + if size == 0: + return [], 0, 0 + else: + # Final data must be complete + if errors == 'strict': + raise UnicodeDecodeError('utf-32', bytes(data), size - (size % 4), size, + 'truncated data') + elif errors == 'ignore': + size = (size // 4) * 4 + elif errors == 'replace': + size = (size // 4) * 4 + + result = [] + pos = 0 + + while pos + 3 < size: + if byteorder == 'little': + ch = data[pos] | (data[pos+1] << 8) | (data[pos+2] << 16) | (data[pos+3] << 24) + else: # big-endian + ch = (data[pos] << 24) | (data[pos+1] << 16) | (data[pos+2] << 8) | data[pos+3] + + # Validate code point + if ch > 0x10FFFF: + if errors == 'strict': + raise UnicodeDecodeError('utf-32', bytes(data), pos, pos+4, + 'codepoint not in range(0x110000)') + elif errors == 'replace': + result.append('\ufffd') + # 'ignore' - skip this character + else: + result.append(chr(ch)) + + pos += 4 + + return result, pos, 0 + + +def utf_32_decode(data, errors='strict', final=0): + """UTF-32 decoding with BOM detection.""" + if len(data) >= 4: + # Check for BOM + if data[0:4] == b'\xff\xfe\x00\x00': + # UTF-32 LE BOM + res, consumed, _ = PyUnicode_DecodeUTF32Stateful(data[4:], len(data)-4, errors, 'little', final) + res = ''.join(res) + return res, consumed + 4 + elif data[0:4] == b'\x00\x00\xfe\xff': + # UTF-32 BE BOM + res, consumed, _ = PyUnicode_DecodeUTF32Stateful(data[4:], len(data)-4, errors, 'big', final) + res = ''.join(res) + return res, consumed + 4 + + # Default to little-endian if no BOM + byteorder = 'little' if sys.byteorder == 'little' else 'big' + res, consumed, _ = PyUnicode_DecodeUTF32Stateful(data, len(data), errors, byteorder, final) + res = ''.join(res) + return res, consumed + + +def utf_32_le_decode(data, errors='strict', final=0): + """UTF-32 little-endian decoding without BOM.""" + res, consumed, _ = PyUnicode_DecodeUTF32Stateful(data, len(data), errors, 'little', final) + res = ''.join(res) + return res, consumed + + +def utf_32_be_decode(data, errors='strict', final=0): + """UTF-32 big-endian decoding without BOM.""" + res, consumed, _ = PyUnicode_DecodeUTF32Stateful(data, len(data), errors, 'big', final) + res = ''.join(res) + return res, consumed # ---------------------------------------------------------------------- @@ -677,8 +816,8 @@ def PyUnicode_AsASCIIString(unistr): if not type(unistr) == str: raise TypeError - return PyUnicode_EncodeASCII(str(unistr), - len(str), + return PyUnicode_EncodeASCII(unistr, + len(unistr), None) def PyUnicode_DecodeUTF16Stateful(s, size, errors, byteorder='native', final=True): @@ -815,7 +954,7 @@ def PyUnicode_EncodeUTF16(s, size, errors, byteorder='little'): p += STORECHAR(0xFEFF, bom) if (size == 0): - return "" + return [] if (byteorder == 'little' ): bom = 'little' @@ -1084,7 +1223,7 @@ def PyUnicode_EncodeRawUnicodeEscape(s, size): def charmapencode_output(c, mapping): rep = mapping[c] - if isinstance(rep, int) or isinstance(rep, int): + if isinstance(rep, int): if rep < 256: return [rep] else: diff --git a/Lib/_pydecimal.py b/Lib/_pydecimal.py index ff80180a79e..97a629fe92c 100644 --- a/Lib/_pydecimal.py +++ b/Lib/_pydecimal.py @@ -38,10 +38,10 @@ 'ROUND_FLOOR', 'ROUND_UP', 'ROUND_HALF_DOWN', 'ROUND_05UP', # Functions for manipulating contexts - 'setcontext', 'getcontext', 'localcontext', + 'setcontext', 'getcontext', 'localcontext', 'IEEEContext', # Limits for the C version for compatibility - 'MAX_PREC', 'MAX_EMAX', 'MIN_EMIN', 'MIN_ETINY', + 'MAX_PREC', 'MAX_EMAX', 'MIN_EMIN', 'MIN_ETINY', 'IEEE_CONTEXT_MAX_BITS', # C version: compile time choice that enables the thread local context (deprecated, now always true) 'HAVE_THREADS', @@ -83,10 +83,12 @@ MAX_PREC = 999999999999999999 MAX_EMAX = 999999999999999999 MIN_EMIN = -999999999999999999 + IEEE_CONTEXT_MAX_BITS = 512 else: MAX_PREC = 425000000 MAX_EMAX = 425000000 MIN_EMIN = -425000000 + IEEE_CONTEXT_MAX_BITS = 256 MIN_ETINY = MIN_EMIN - (MAX_PREC-1) @@ -417,6 +419,27 @@ def sin(x): return ctx_manager +def IEEEContext(bits, /): + """ + Return a context object initialized to the proper values for one of the + IEEE interchange formats. The argument must be a multiple of 32 and less + than IEEE_CONTEXT_MAX_BITS. + """ + if bits <= 0 or bits > IEEE_CONTEXT_MAX_BITS or bits % 32: + raise ValueError("argument must be a multiple of 32, " + f"with a maximum of {IEEE_CONTEXT_MAX_BITS}") + + ctx = Context() + ctx.prec = 9 * (bits//32) - 2 + ctx.Emax = 3 * (1 << (bits//16 + 3)) + ctx.Emin = 1 - ctx.Emax + ctx.rounding = ROUND_HALF_EVEN + ctx.clamp = 1 + ctx.traps = dict.fromkeys(_signals, False) + + return ctx + + ##### Decimal class ####################################################### # Do not subclass Decimal from numbers.Real and do not register it as such @@ -582,6 +605,21 @@ def __new__(cls, value="0", context=None): raise TypeError("Cannot convert %r to Decimal" % value) + @classmethod + def from_number(cls, number): + """Converts a real number to a decimal number, exactly. + + >>> Decimal.from_number(314) # int + Decimal('314') + >>> Decimal.from_number(0.1) # float + Decimal('0.1000000000000000055511151231257827021181583404541015625') + >>> Decimal.from_number(Decimal('3.14')) # another decimal instance + Decimal('3.14') + """ + if isinstance(number, (int, Decimal, float)): + return cls(number) + raise TypeError("Cannot convert %r to Decimal" % number) + @classmethod def from_float(cls, f): """Converts a float to a decimal number, exactly. @@ -2425,12 +2463,12 @@ def __pow__(self, other, modulo=None, context=None): return ans - def __rpow__(self, other, context=None): + def __rpow__(self, other, modulo=None, context=None): """Swaps self/other and returns __pow__.""" other = _convert_other(other) if other is NotImplemented: return other - return other.__pow__(self, context=context) + return other.__pow__(self, modulo, context=context) def normalize(self, context=None): """Normalize- strip trailing 0s, change anything equal to 0 to 0e0""" @@ -3302,7 +3340,10 @@ def _fill_logical(self, context, opa, opb): return opa, opb def logical_and(self, other, context=None): - """Applies an 'and' operation between self and other's digits.""" + """Applies an 'and' operation between self and other's digits. + + Both self and other must be logical numbers. + """ if context is None: context = getcontext() @@ -3319,14 +3360,20 @@ def logical_and(self, other, context=None): return _dec_from_triple(0, result.lstrip('0') or '0', 0) def logical_invert(self, context=None): - """Invert all its digits.""" + """Invert all its digits. + + The self must be logical number. + """ if context is None: context = getcontext() return self.logical_xor(_dec_from_triple(0,'1'*context.prec,0), context) def logical_or(self, other, context=None): - """Applies an 'or' operation between self and other's digits.""" + """Applies an 'or' operation between self and other's digits. + + Both self and other must be logical numbers. + """ if context is None: context = getcontext() @@ -3343,7 +3390,10 @@ def logical_or(self, other, context=None): return _dec_from_triple(0, result.lstrip('0') or '0', 0) def logical_xor(self, other, context=None): - """Applies an 'xor' operation between self and other's digits.""" + """Applies an 'xor' operation between self and other's digits. + + Both self and other must be logical numbers. + """ if context is None: context = getcontext() @@ -6058,7 +6108,7 @@ def _convert_for_comparison(self, other, equality_op=False): (?P\d*) # with (possibly empty) diagnostic info. ) # \s* - \Z + \z """, re.VERBOSE | re.IGNORECASE).match _all_zeros = re.compile('0*$').match @@ -6082,11 +6132,15 @@ def _convert_for_comparison(self, other, equality_op=False): (?Pz)? (?P\#)? (?P0)? -(?P(?!0)\d+)? -(?P,)? -(?:\.(?P0|(?!0)\d+))? +(?P\d+)? +(?P[,_])? +(?:\. + (?=[\d,_]) # lookahead for digit or separator + (?P\d+)? + (?P[,_])? +)? (?P[eEfFgGn%])? -\Z +\z """, re.VERBOSE|re.DOTALL) del re @@ -6177,6 +6231,9 @@ def _parse_format_specifier(format_spec, _localeconv=None): format_dict['grouping'] = [3, 0] format_dict['decimal_point'] = '.' + if format_dict['frac_separators'] is None: + format_dict['frac_separators'] = '' + return format_dict def _format_align(sign, body, spec): @@ -6296,6 +6353,11 @@ def _format_number(is_negative, intpart, fracpart, exp, spec): sign = _format_sign(is_negative, spec) + frac_sep = spec['frac_separators'] + if fracpart and frac_sep: + fracpart = frac_sep.join(fracpart[pos:pos + 3] + for pos in range(0, len(fracpart), 3)) + if fracpart or spec['alt']: fracpart = spec['decimal_point'] + fracpart diff --git a/Lib/_weakrefset.py b/Lib/_weakrefset.py index 489eec714e0..d1c7fcaeec9 100644 --- a/Lib/_weakrefset.py +++ b/Lib/_weakrefset.py @@ -8,69 +8,29 @@ __all__ = ['WeakSet'] -class _IterationGuard: - # This context manager registers itself in the current iterators of the - # weak container, such as to delay all removals until the context manager - # exits. - # This technique should be relatively thread-safe (since sets are). - - def __init__(self, weakcontainer): - # Don't create cycles - self.weakcontainer = ref(weakcontainer) - - def __enter__(self): - w = self.weakcontainer() - if w is not None: - w._iterating.add(self) - return self - - def __exit__(self, e, t, b): - w = self.weakcontainer() - if w is not None: - s = w._iterating - s.remove(self) - if not s: - w._commit_removals() - - class WeakSet: def __init__(self, data=None): self.data = set() + def _remove(item, selfref=ref(self)): self = selfref() if self is not None: - if self._iterating: - self._pending_removals.append(item) - else: - self.data.discard(item) + self.data.discard(item) + self._remove = _remove - # A list of keys to be removed - self._pending_removals = [] - self._iterating = set() if data is not None: self.update(data) - def _commit_removals(self): - pop = self._pending_removals.pop - discard = self.data.discard - while True: - try: - item = pop() - except IndexError: - return - discard(item) - def __iter__(self): - with _IterationGuard(self): - for itemref in self.data: - item = itemref() - if item is not None: - # Caveat: the iterator will keep a strong reference to - # `item` until it is resumed or closed. - yield item + for itemref in self.data.copy(): + item = itemref() + if item is not None: + # Caveat: the iterator will keep a strong reference to + # `item` until it is resumed or closed. + yield item def __len__(self): - return len(self.data) - len(self._pending_removals) + return len(self.data) def __contains__(self, item): try: @@ -83,21 +43,15 @@ def __reduce__(self): return self.__class__, (list(self),), self.__getstate__() def add(self, item): - if self._pending_removals: - self._commit_removals() self.data.add(ref(item, self._remove)) def clear(self): - if self._pending_removals: - self._commit_removals() self.data.clear() def copy(self): return self.__class__(self) def pop(self): - if self._pending_removals: - self._commit_removals() while True: try: itemref = self.data.pop() @@ -108,18 +62,12 @@ def pop(self): return item def remove(self, item): - if self._pending_removals: - self._commit_removals() self.data.remove(ref(item)) def discard(self, item): - if self._pending_removals: - self._commit_removals() self.data.discard(ref(item)) def update(self, other): - if self._pending_removals: - self._commit_removals() for element in other: self.add(element) @@ -136,8 +84,6 @@ def difference(self, other): def difference_update(self, other): self.__isub__(other) def __isub__(self, other): - if self._pending_removals: - self._commit_removals() if self is other: self.data.clear() else: @@ -151,8 +97,6 @@ def intersection(self, other): def intersection_update(self, other): self.__iand__(other) def __iand__(self, other): - if self._pending_removals: - self._commit_removals() self.data.intersection_update(ref(item) for item in other) return self @@ -184,8 +128,6 @@ def symmetric_difference(self, other): def symmetric_difference_update(self, other): self.__ixor__(other) def __ixor__(self, other): - if self._pending_removals: - self._commit_removals() if self is other: self.data.clear() else: diff --git a/Lib/annotationlib.py b/Lib/annotationlib.py new file mode 100644 index 00000000000..a5788cdbfae --- /dev/null +++ b/Lib/annotationlib.py @@ -0,0 +1,1143 @@ +"""Helpers for introspecting and wrapping annotations.""" + +import ast +import builtins +import enum +import keyword +import sys +import types + +__all__ = [ + "Format", + "ForwardRef", + "call_annotate_function", + "call_evaluate_function", + "get_annotate_from_class_namespace", + "get_annotations", + "annotations_to_string", + "type_repr", +] + + +class Format(enum.IntEnum): + VALUE = 1 + VALUE_WITH_FAKE_GLOBALS = 2 + FORWARDREF = 3 + STRING = 4 + + +_sentinel = object() +# Following `NAME_ERROR_MSG` in `ceval_macros.h`: +_NAME_ERROR_MSG = "name '{name:.200}' is not defined" + + +# Slots shared by ForwardRef and _Stringifier. The __forward__ names must be +# preserved for compatibility with the old typing.ForwardRef class. The remaining +# names are private. +_SLOTS = ( + "__forward_is_argument__", + "__forward_is_class__", + "__forward_module__", + "__weakref__", + "__arg__", + "__globals__", + "__extra_names__", + "__code__", + "__ast_node__", + "__cell__", + "__owner__", + "__stringifier_dict__", +) + + +class ForwardRef: + """Wrapper that holds a forward reference. + + Constructor arguments: + * arg: a string representing the code to be evaluated. + * module: the module where the forward reference was created. + Must be a string, not a module object. + * owner: The owning object (module, class, or function). + * is_argument: Does nothing, retained for compatibility. + * is_class: True if the forward reference was created in class scope. + + """ + + __slots__ = _SLOTS + + def __init__( + self, + arg, + *, + module=None, + owner=None, + is_argument=True, + is_class=False, + ): + if not isinstance(arg, str): + raise TypeError(f"Forward reference must be a string -- got {arg!r}") + + self.__arg__ = arg + self.__forward_is_argument__ = is_argument + self.__forward_is_class__ = is_class + self.__forward_module__ = module + self.__owner__ = owner + # These are always set to None here but may be non-None if a ForwardRef + # is created through __class__ assignment on a _Stringifier object. + self.__globals__ = None + # This may be either a cell object (for a ForwardRef referring to a single name) + # or a dict mapping cell names to cell objects (for a ForwardRef containing references + # to multiple names). + self.__cell__ = None + self.__extra_names__ = None + # These are initially None but serve as a cache and may be set to a non-None + # value later. + self.__code__ = None + self.__ast_node__ = None + + def __init_subclass__(cls, /, *args, **kwds): + raise TypeError("Cannot subclass ForwardRef") + + def evaluate( + self, + *, + globals=None, + locals=None, + type_params=None, + owner=None, + format=Format.VALUE, + ): + """Evaluate the forward reference and return the value. + + If the forward reference cannot be evaluated, raise an exception. + """ + match format: + case Format.STRING: + return self.__forward_arg__ + case Format.VALUE: + is_forwardref_format = False + case Format.FORWARDREF: + is_forwardref_format = True + case _: + raise NotImplementedError(format) + if isinstance(self.__cell__, types.CellType): + try: + return self.__cell__.cell_contents + except ValueError: + pass + if owner is None: + owner = self.__owner__ + + if globals is None and self.__forward_module__ is not None: + globals = getattr( + sys.modules.get(self.__forward_module__, None), "__dict__", None + ) + if globals is None: + globals = self.__globals__ + if globals is None: + if isinstance(owner, type): + module_name = getattr(owner, "__module__", None) + if module_name: + module = sys.modules.get(module_name, None) + if module: + globals = getattr(module, "__dict__", None) + elif isinstance(owner, types.ModuleType): + globals = getattr(owner, "__dict__", None) + elif callable(owner): + globals = getattr(owner, "__globals__", None) + + # If we pass None to eval() below, the globals of this module are used. + if globals is None: + globals = {} + + if type_params is None and owner is not None: + type_params = getattr(owner, "__type_params__", None) + + if locals is None: + locals = {} + if isinstance(owner, type): + locals.update(vars(owner)) + elif ( + type_params is not None + or isinstance(self.__cell__, dict) + or self.__extra_names__ + ): + # Create a new locals dict if necessary, + # to avoid mutating the argument. + locals = dict(locals) + + # "Inject" type parameters into the local namespace + # (unless they are shadowed by assignments *in* the local namespace), + # as a way of emulating annotation scopes when calling `eval()` + if type_params is not None: + for param in type_params: + locals.setdefault(param.__name__, param) + + # Similar logic can be used for nonlocals, which should not + # override locals. + if isinstance(self.__cell__, dict): + for cell_name, cell in self.__cell__.items(): + try: + cell_value = cell.cell_contents + except ValueError: + pass + else: + locals.setdefault(cell_name, cell_value) + + if self.__extra_names__: + locals.update(self.__extra_names__) + + arg = self.__forward_arg__ + if arg.isidentifier() and not keyword.iskeyword(arg): + if arg in locals: + return locals[arg] + elif arg in globals: + return globals[arg] + elif hasattr(builtins, arg): + return getattr(builtins, arg) + elif is_forwardref_format: + return self + else: + raise NameError(_NAME_ERROR_MSG.format(name=arg), name=arg) + else: + code = self.__forward_code__ + try: + return eval(code, globals=globals, locals=locals) + except Exception: + if not is_forwardref_format: + raise + + # All variables, in scoping order, should be checked before + # triggering __missing__ to create a _Stringifier. + new_locals = _StringifierDict( + {**builtins.__dict__, **globals, **locals}, + globals=globals, + owner=owner, + is_class=self.__forward_is_class__, + format=format, + ) + try: + result = eval(code, globals=globals, locals=new_locals) + except Exception: + return self + else: + new_locals.transmogrify(self.__cell__) + return result + + def _evaluate(self, globalns, localns, type_params=_sentinel, *, recursive_guard): + import typing + import warnings + + if type_params is _sentinel: + typing._deprecation_warning_for_no_type_params_passed( + "typing.ForwardRef._evaluate" + ) + type_params = () + warnings._deprecated( + "ForwardRef._evaluate", + "{name} is a private API and is retained for compatibility, but will be removed" + " in Python 3.16. Use ForwardRef.evaluate() or typing.evaluate_forward_ref() instead.", + remove=(3, 16), + ) + return typing.evaluate_forward_ref( + self, + globals=globalns, + locals=localns, + type_params=type_params, + _recursive_guard=recursive_guard, + ) + + @property + def __forward_arg__(self): + if self.__arg__ is not None: + return self.__arg__ + if self.__ast_node__ is not None: + self.__arg__ = ast.unparse(self.__ast_node__) + return self.__arg__ + raise AssertionError( + "Attempted to access '__forward_arg__' on an uninitialized ForwardRef" + ) + + @property + def __forward_code__(self): + if self.__code__ is not None: + return self.__code__ + arg = self.__forward_arg__ + try: + self.__code__ = compile(_rewrite_star_unpack(arg), "", "eval") + except SyntaxError: + raise SyntaxError(f"Forward reference must be an expression -- got {arg!r}") + return self.__code__ + + def __eq__(self, other): + if not isinstance(other, ForwardRef): + return NotImplemented + return ( + self.__forward_arg__ == other.__forward_arg__ + and self.__forward_module__ == other.__forward_module__ + # Use "is" here because we use id() for this in __hash__ + # because dictionaries are not hashable. + and self.__globals__ is other.__globals__ + and self.__forward_is_class__ == other.__forward_is_class__ + and self.__cell__ == other.__cell__ + and self.__owner__ == other.__owner__ + and ( + (tuple(sorted(self.__extra_names__.items())) if self.__extra_names__ else None) == + (tuple(sorted(other.__extra_names__.items())) if other.__extra_names__ else None) + ) + ) + + def __hash__(self): + return hash(( + self.__forward_arg__, + self.__forward_module__, + id(self.__globals__), # dictionaries are not hashable, so hash by identity + self.__forward_is_class__, + tuple(sorted(self.__cell__.items())) if isinstance(self.__cell__, dict) else self.__cell__, + self.__owner__, + tuple(sorted(self.__extra_names__.items())) if self.__extra_names__ else None, + )) + + def __or__(self, other): + return types.UnionType[self, other] + + def __ror__(self, other): + return types.UnionType[other, self] + + def __repr__(self): + extra = [] + if self.__forward_module__ is not None: + extra.append(f", module={self.__forward_module__!r}") + if self.__forward_is_class__: + extra.append(", is_class=True") + if self.__owner__ is not None: + extra.append(f", owner={self.__owner__!r}") + return f"ForwardRef({self.__forward_arg__!r}{''.join(extra)})" + + +_Template = type(t"") + + +class _Stringifier: + # Must match the slots on ForwardRef, so we can turn an instance of one into an + # instance of the other in place. + __slots__ = _SLOTS + + def __init__( + self, + node, + globals=None, + owner=None, + is_class=False, + cell=None, + *, + stringifier_dict, + extra_names=None, + ): + # Either an AST node or a simple str (for the common case where a ForwardRef + # represent a single name). + assert isinstance(node, (ast.AST, str)) + self.__arg__ = None + self.__forward_is_argument__ = False + self.__forward_is_class__ = is_class + self.__forward_module__ = None + self.__code__ = None + self.__ast_node__ = node + self.__globals__ = globals + self.__extra_names__ = extra_names + self.__cell__ = cell + self.__owner__ = owner + self.__stringifier_dict__ = stringifier_dict + + def __convert_to_ast(self, other): + if isinstance(other, _Stringifier): + if isinstance(other.__ast_node__, str): + return ast.Name(id=other.__ast_node__), other.__extra_names__ + return other.__ast_node__, other.__extra_names__ + elif type(other) is _Template: + return _template_to_ast(other), None + elif ( + # In STRING format we don't bother with the create_unique_name() dance; + # it's better to emit the repr() of the object instead of an opaque name. + self.__stringifier_dict__.format == Format.STRING + or other is None + or type(other) in (str, int, float, bool, complex) + ): + return ast.Constant(value=other), None + elif type(other) is dict: + extra_names = {} + keys = [] + values = [] + for key, value in other.items(): + new_key, new_extra_names = self.__convert_to_ast(key) + if new_extra_names is not None: + extra_names.update(new_extra_names) + keys.append(new_key) + new_value, new_extra_names = self.__convert_to_ast(value) + if new_extra_names is not None: + extra_names.update(new_extra_names) + values.append(new_value) + return ast.Dict(keys, values), extra_names + elif type(other) in (list, tuple, set): + extra_names = {} + elts = [] + for elt in other: + new_elt, new_extra_names = self.__convert_to_ast(elt) + if new_extra_names is not None: + extra_names.update(new_extra_names) + elts.append(new_elt) + ast_class = {list: ast.List, tuple: ast.Tuple, set: ast.Set}[type(other)] + return ast_class(elts), extra_names + else: + name = self.__stringifier_dict__.create_unique_name() + return ast.Name(id=name), {name: other} + + def __convert_to_ast_getitem(self, other): + if isinstance(other, slice): + extra_names = {} + + def conv(obj): + if obj is None: + return None + new_obj, new_extra_names = self.__convert_to_ast(obj) + if new_extra_names is not None: + extra_names.update(new_extra_names) + return new_obj + + return ast.Slice( + lower=conv(other.start), + upper=conv(other.stop), + step=conv(other.step), + ), extra_names + else: + return self.__convert_to_ast(other) + + def __get_ast(self): + node = self.__ast_node__ + if isinstance(node, str): + return ast.Name(id=node) + return node + + def __make_new(self, node, extra_names=None): + new_extra_names = {} + if self.__extra_names__ is not None: + new_extra_names.update(self.__extra_names__) + if extra_names is not None: + new_extra_names.update(extra_names) + stringifier = _Stringifier( + node, + self.__globals__, + self.__owner__, + self.__forward_is_class__, + stringifier_dict=self.__stringifier_dict__, + extra_names=new_extra_names or None, + ) + self.__stringifier_dict__.stringifiers.append(stringifier) + return stringifier + + # Must implement this since we set __eq__. We hash by identity so that + # stringifiers in dict keys are kept separate. + def __hash__(self): + return id(self) + + def __getitem__(self, other): + # Special case, to avoid stringifying references to class-scoped variables + # as '__classdict__["x"]'. + if self.__ast_node__ == "__classdict__": + raise KeyError + if isinstance(other, tuple): + extra_names = {} + elts = [] + for elt in other: + new_elt, new_extra_names = self.__convert_to_ast_getitem(elt) + if new_extra_names is not None: + extra_names.update(new_extra_names) + elts.append(new_elt) + other = ast.Tuple(elts) + else: + other, extra_names = self.__convert_to_ast_getitem(other) + assert isinstance(other, ast.AST), repr(other) + return self.__make_new(ast.Subscript(self.__get_ast(), other), extra_names) + + def __getattr__(self, attr): + return self.__make_new(ast.Attribute(self.__get_ast(), attr)) + + def __call__(self, *args, **kwargs): + extra_names = {} + ast_args = [] + for arg in args: + new_arg, new_extra_names = self.__convert_to_ast(arg) + if new_extra_names is not None: + extra_names.update(new_extra_names) + ast_args.append(new_arg) + ast_kwargs = [] + for key, value in kwargs.items(): + new_value, new_extra_names = self.__convert_to_ast(value) + if new_extra_names is not None: + extra_names.update(new_extra_names) + ast_kwargs.append(ast.keyword(key, new_value)) + return self.__make_new(ast.Call(self.__get_ast(), ast_args, ast_kwargs), extra_names) + + def __iter__(self): + yield self.__make_new(ast.Starred(self.__get_ast())) + + def __repr__(self): + if isinstance(self.__ast_node__, str): + return self.__ast_node__ + return ast.unparse(self.__ast_node__) + + def __format__(self, format_spec): + raise TypeError("Cannot stringify annotation containing string formatting") + + def _make_binop(op: ast.AST): + def binop(self, other): + rhs, extra_names = self.__convert_to_ast(other) + return self.__make_new( + ast.BinOp(self.__get_ast(), op, rhs), extra_names + ) + + return binop + + __add__ = _make_binop(ast.Add()) + __sub__ = _make_binop(ast.Sub()) + __mul__ = _make_binop(ast.Mult()) + __matmul__ = _make_binop(ast.MatMult()) + __truediv__ = _make_binop(ast.Div()) + __mod__ = _make_binop(ast.Mod()) + __lshift__ = _make_binop(ast.LShift()) + __rshift__ = _make_binop(ast.RShift()) + __or__ = _make_binop(ast.BitOr()) + __xor__ = _make_binop(ast.BitXor()) + __and__ = _make_binop(ast.BitAnd()) + __floordiv__ = _make_binop(ast.FloorDiv()) + __pow__ = _make_binop(ast.Pow()) + + del _make_binop + + def _make_rbinop(op: ast.AST): + def rbinop(self, other): + new_other, extra_names = self.__convert_to_ast(other) + return self.__make_new( + ast.BinOp(new_other, op, self.__get_ast()), extra_names + ) + + return rbinop + + __radd__ = _make_rbinop(ast.Add()) + __rsub__ = _make_rbinop(ast.Sub()) + __rmul__ = _make_rbinop(ast.Mult()) + __rmatmul__ = _make_rbinop(ast.MatMult()) + __rtruediv__ = _make_rbinop(ast.Div()) + __rmod__ = _make_rbinop(ast.Mod()) + __rlshift__ = _make_rbinop(ast.LShift()) + __rrshift__ = _make_rbinop(ast.RShift()) + __ror__ = _make_rbinop(ast.BitOr()) + __rxor__ = _make_rbinop(ast.BitXor()) + __rand__ = _make_rbinop(ast.BitAnd()) + __rfloordiv__ = _make_rbinop(ast.FloorDiv()) + __rpow__ = _make_rbinop(ast.Pow()) + + del _make_rbinop + + def _make_compare(op): + def compare(self, other): + rhs, extra_names = self.__convert_to_ast(other) + return self.__make_new( + ast.Compare( + left=self.__get_ast(), + ops=[op], + comparators=[rhs], + ), + extra_names, + ) + + return compare + + __lt__ = _make_compare(ast.Lt()) + __le__ = _make_compare(ast.LtE()) + __eq__ = _make_compare(ast.Eq()) + __ne__ = _make_compare(ast.NotEq()) + __gt__ = _make_compare(ast.Gt()) + __ge__ = _make_compare(ast.GtE()) + + del _make_compare + + def _make_unary_op(op): + def unary_op(self): + return self.__make_new(ast.UnaryOp(op, self.__get_ast())) + + return unary_op + + __invert__ = _make_unary_op(ast.Invert()) + __pos__ = _make_unary_op(ast.UAdd()) + __neg__ = _make_unary_op(ast.USub()) + + del _make_unary_op + + +def _template_to_ast_constructor(template): + """Convert a `template` instance to a non-literal AST.""" + args = [] + for part in template: + match part: + case str(): + args.append(ast.Constant(value=part)) + case _: + interp = ast.Call( + func=ast.Name(id="Interpolation"), + args=[ + ast.Constant(value=part.value), + ast.Constant(value=part.expression), + ast.Constant(value=part.conversion), + ast.Constant(value=part.format_spec), + ] + ) + args.append(interp) + return ast.Call(func=ast.Name(id="Template"), args=args, keywords=[]) + + +def _template_to_ast_literal(template, parsed): + """Convert a `template` instance to a t-string literal AST.""" + values = [] + interp_count = 0 + for part in template: + match part: + case str(): + values.append(ast.Constant(value=part)) + case _: + interp = ast.Interpolation( + str=part.expression, + value=parsed[interp_count], + conversion=ord(part.conversion) if part.conversion else -1, + format_spec=ast.Constant(value=part.format_spec) + if part.format_spec + else None, + ) + values.append(interp) + interp_count += 1 + return ast.TemplateStr(values=values) + + +def _template_to_ast(template): + """Make a best-effort conversion of a `template` instance to an AST.""" + # gh-138558: Not all Template instances can be represented as t-string + # literals. Return the most accurate AST we can. See issue for details. + + # If any expr is empty or whitespace only, we cannot convert to a literal. + if any(part.expression.strip() == "" for part in template.interpolations): + return _template_to_ast_constructor(template) + + try: + # Wrap in parens to allow whitespace inside interpolation curly braces + parsed = tuple( + ast.parse(f"({part.expression})", mode="eval").body + for part in template.interpolations + ) + except SyntaxError: + return _template_to_ast_constructor(template) + + return _template_to_ast_literal(template, parsed) + + +class _StringifierDict(dict): + def __init__(self, namespace, *, globals=None, owner=None, is_class=False, format): + super().__init__(namespace) + self.namespace = namespace + self.globals = globals + self.owner = owner + self.is_class = is_class + self.stringifiers = [] + self.next_id = 1 + self.format = format + + def __missing__(self, key): + fwdref = _Stringifier( + key, + globals=self.globals, + owner=self.owner, + is_class=self.is_class, + stringifier_dict=self, + ) + self.stringifiers.append(fwdref) + return fwdref + + def transmogrify(self, cell_dict): + for obj in self.stringifiers: + obj.__class__ = ForwardRef + obj.__stringifier_dict__ = None # not needed for ForwardRef + if isinstance(obj.__ast_node__, str): + obj.__arg__ = obj.__ast_node__ + obj.__ast_node__ = None + if cell_dict is not None and obj.__cell__ is None: + obj.__cell__ = cell_dict + + def create_unique_name(self): + name = f"__annotationlib_name_{self.next_id}__" + self.next_id += 1 + return name + + +def call_evaluate_function(evaluate, format, *, owner=None): + """Call an evaluate function. Evaluate functions are normally generated for + the value of type aliases and the bounds, constraints, and defaults of + type parameter objects. + """ + return call_annotate_function(evaluate, format, owner=owner, _is_evaluate=True) + + +def call_annotate_function(annotate, format, *, owner=None, _is_evaluate=False): + """Call an __annotate__ function. __annotate__ functions are normally + generated by the compiler to defer the evaluation of annotations. They + can be called with any of the format arguments in the Format enum, but + compiler-generated __annotate__ functions only support the VALUE format. + This function provides additional functionality to call __annotate__ + functions with the FORWARDREF and STRING formats. + + *annotate* must be an __annotate__ function, which takes a single argument + and returns a dict of annotations. + + *format* must be a member of the Format enum or one of the corresponding + integer values. + + *owner* can be the object that owns the annotations (i.e., the module, + class, or function that the __annotate__ function derives from). With the + FORWARDREF format, it is used to provide better evaluation capabilities + on the generated ForwardRef objects. + + """ + if format == Format.VALUE_WITH_FAKE_GLOBALS: + raise ValueError("The VALUE_WITH_FAKE_GLOBALS format is for internal use only") + try: + return annotate(format) + except NotImplementedError: + pass + if format == Format.STRING: + # STRING is implemented by calling the annotate function in a special + # environment where every name lookup results in an instance of _Stringifier. + # _Stringifier supports every dunder operation and returns a new _Stringifier. + # At the end, we get a dictionary that mostly contains _Stringifier objects (or + # possibly constants if the annotate function uses them directly). We then + # convert each of those into a string to get an approximation of the + # original source. + + # Attempt to call with VALUE_WITH_FAKE_GLOBALS to check if it is implemented + # See: https://github.com/python/cpython/issues/138764 + # Only fail on NotImplementedError + try: + annotate(Format.VALUE_WITH_FAKE_GLOBALS) + except NotImplementedError: + # Both STRING and VALUE_WITH_FAKE_GLOBALS are not implemented: fallback to VALUE + return annotations_to_string(annotate(Format.VALUE)) + except Exception: + pass + + globals = _StringifierDict({}, format=format) + is_class = isinstance(owner, type) + closure, _ = _build_closure( + annotate, owner, is_class, globals, allow_evaluation=False + ) + func = types.FunctionType( + annotate.__code__, + globals, + closure=closure, + argdefs=annotate.__defaults__, + kwdefaults=annotate.__kwdefaults__, + ) + annos = func(Format.VALUE_WITH_FAKE_GLOBALS) + if _is_evaluate: + return _stringify_single(annos) + return { + key: _stringify_single(val) + for key, val in annos.items() + } + elif format == Format.FORWARDREF: + # FORWARDREF is implemented similarly to STRING, but there are two changes, + # at the beginning and the end of the process. + # First, while STRING uses an empty dictionary as the namespace, so that all + # name lookups result in _Stringifier objects, FORWARDREF uses the globals + # and builtins, so that defined names map to their real values. + # Second, instead of returning strings, we want to return either real values + # or ForwardRef objects. To do this, we keep track of all _Stringifier objects + # created while the annotation is being evaluated, and at the end we convert + # them all to ForwardRef objects by assigning to __class__. To make this + # technique work, we have to ensure that the _Stringifier and ForwardRef + # classes share the same attributes. + # We use this technique because while the annotations are being evaluated, + # we want to support all operations that the language allows, including even + # __getattr__ and __eq__, and return new _Stringifier objects so we can accurately + # reconstruct the source. But in the dictionary that we eventually return, we + # want to return objects with more user-friendly behavior, such as an __eq__ + # that returns a bool and an defined set of attributes. + namespace = {**annotate.__builtins__, **annotate.__globals__} + is_class = isinstance(owner, type) + globals = _StringifierDict( + namespace, + globals=annotate.__globals__, + owner=owner, + is_class=is_class, + format=format, + ) + closure, cell_dict = _build_closure( + annotate, owner, is_class, globals, allow_evaluation=True + ) + func = types.FunctionType( + annotate.__code__, + globals, + closure=closure, + argdefs=annotate.__defaults__, + kwdefaults=annotate.__kwdefaults__, + ) + try: + result = func(Format.VALUE_WITH_FAKE_GLOBALS) + except NotImplementedError: + # FORWARDREF and VALUE_WITH_FAKE_GLOBALS not supported, fall back to VALUE + return annotate(Format.VALUE) + except Exception: + pass + else: + globals.transmogrify(cell_dict) + return result + + # Try again, but do not provide any globals. This allows us to return + # a value in certain cases where an exception gets raised during evaluation. + globals = _StringifierDict( + {}, + globals=annotate.__globals__, + owner=owner, + is_class=is_class, + format=format, + ) + closure, cell_dict = _build_closure( + annotate, owner, is_class, globals, allow_evaluation=False + ) + func = types.FunctionType( + annotate.__code__, + globals, + closure=closure, + argdefs=annotate.__defaults__, + kwdefaults=annotate.__kwdefaults__, + ) + result = func(Format.VALUE_WITH_FAKE_GLOBALS) + globals.transmogrify(cell_dict) + if _is_evaluate: + if isinstance(result, ForwardRef): + return result.evaluate(format=Format.FORWARDREF) + else: + return result + else: + return { + key: ( + val.evaluate(format=Format.FORWARDREF) + if isinstance(val, ForwardRef) + else val + ) + for key, val in result.items() + } + elif format == Format.VALUE: + # Should be impossible because __annotate__ functions must not raise + # NotImplementedError for this format. + raise RuntimeError("annotate function does not support VALUE format") + else: + raise ValueError(f"Invalid format: {format!r}") + + +def _build_closure(annotate, owner, is_class, stringifier_dict, *, allow_evaluation): + if not annotate.__closure__: + return None, None + new_closure = [] + cell_dict = {} + for name, cell in zip(annotate.__code__.co_freevars, annotate.__closure__, strict=True): + cell_dict[name] = cell + new_cell = None + if allow_evaluation: + try: + cell.cell_contents + except ValueError: + pass + else: + new_cell = cell + if new_cell is None: + fwdref = _Stringifier( + name, + cell=cell, + owner=owner, + globals=annotate.__globals__, + is_class=is_class, + stringifier_dict=stringifier_dict, + ) + stringifier_dict.stringifiers.append(fwdref) + new_cell = types.CellType(fwdref) + new_closure.append(new_cell) + return tuple(new_closure), cell_dict + + +def _stringify_single(anno): + if anno is ...: + return "..." + # We have to handle str specially to support PEP 563 stringified annotations. + elif isinstance(anno, str): + return anno + elif isinstance(anno, _Template): + return ast.unparse(_template_to_ast(anno)) + else: + return repr(anno) + + +def get_annotate_from_class_namespace(obj): + """Retrieve the annotate function from a class namespace dictionary. + + Return None if the namespace does not contain an annotate function. + This is useful in metaclass ``__new__`` methods to retrieve the annotate function. + """ + try: + return obj["__annotate__"] + except KeyError: + return obj.get("__annotate_func__", None) + + +def get_annotations( + obj, *, globals=None, locals=None, eval_str=False, format=Format.VALUE +): + """Compute the annotations dict for an object. + + obj may be a callable, class, module, or other object with + __annotate__ or __annotations__ attributes. + Passing any other object raises TypeError. + + The *format* parameter controls the format in which annotations are returned, + and must be a member of the Format enum or its integer equivalent. + For the VALUE format, the __annotations__ is tried first; if it + does not exist, the __annotate__ function is called. The + FORWARDREF format uses __annotations__ if it exists and can be + evaluated, and otherwise falls back to calling the __annotate__ function. + The SOURCE format tries __annotate__ first, and falls back to + using __annotations__, stringified using annotations_to_string(). + + This function handles several details for you: + + * If eval_str is true, values of type str will + be un-stringized using eval(). This is intended + for use with stringized annotations + ("from __future__ import annotations"). + * If obj doesn't have an annotations dict, returns an + empty dict. (Functions and methods always have an + annotations dict; classes, modules, and other types of + callables may not.) + * Ignores inherited annotations on classes. If a class + doesn't have its own annotations dict, returns an empty dict. + * All accesses to object members and dict values are done + using getattr() and dict.get() for safety. + * Always, always, always returns a freshly-created dict. + + eval_str controls whether or not values of type str are replaced + with the result of calling eval() on those values: + + * If eval_str is true, eval() is called on values of type str. + * If eval_str is false (the default), values of type str are unchanged. + + globals and locals are passed in to eval(); see the documentation + for eval() for more information. If either globals or locals is + None, this function may replace that value with a context-specific + default, contingent on type(obj): + + * If obj is a module, globals defaults to obj.__dict__. + * If obj is a class, globals defaults to + sys.modules[obj.__module__].__dict__ and locals + defaults to the obj class namespace. + * If obj is a callable, globals defaults to obj.__globals__, + although if obj is a wrapped function (using + functools.update_wrapper()) it is first unwrapped. + """ + if eval_str and format != Format.VALUE: + raise ValueError("eval_str=True is only supported with format=Format.VALUE") + + match format: + case Format.VALUE: + # For VALUE, we first look at __annotations__ + ann = _get_dunder_annotations(obj) + + # If it's not there, try __annotate__ instead + if ann is None: + ann = _get_and_call_annotate(obj, format) + case Format.FORWARDREF: + # For FORWARDREF, we use __annotations__ if it exists + try: + ann = _get_dunder_annotations(obj) + except Exception: + pass + else: + if ann is not None: + return dict(ann) + + # But if __annotations__ threw a NameError, we try calling __annotate__ + ann = _get_and_call_annotate(obj, format) + if ann is None: + # If that didn't work either, we have a very weird object: evaluating + # __annotations__ threw NameError and there is no __annotate__. In that case, + # we fall back to trying __annotations__ again. + ann = _get_dunder_annotations(obj) + case Format.STRING: + # For STRING, we try to call __annotate__ + ann = _get_and_call_annotate(obj, format) + if ann is not None: + return dict(ann) + # But if we didn't get it, we use __annotations__ instead. + ann = _get_dunder_annotations(obj) + if ann is not None: + return annotations_to_string(ann) + case Format.VALUE_WITH_FAKE_GLOBALS: + raise ValueError("The VALUE_WITH_FAKE_GLOBALS format is for internal use only") + case _: + raise ValueError(f"Unsupported format {format!r}") + + if ann is None: + if isinstance(obj, type) or callable(obj): + return {} + raise TypeError(f"{obj!r} does not have annotations") + + if not ann: + return {} + + if not eval_str: + return dict(ann) + + if globals is None or locals is None: + if isinstance(obj, type): + # class + obj_globals = None + module_name = getattr(obj, "__module__", None) + if module_name: + module = sys.modules.get(module_name, None) + if module: + obj_globals = getattr(module, "__dict__", None) + obj_locals = dict(vars(obj)) + unwrap = obj + elif isinstance(obj, types.ModuleType): + # module + obj_globals = getattr(obj, "__dict__") + obj_locals = None + unwrap = None + elif callable(obj): + # this includes types.Function, types.BuiltinFunctionType, + # types.BuiltinMethodType, functools.partial, functools.singledispatch, + # "class funclike" from Lib/test/test_inspect... on and on it goes. + obj_globals = getattr(obj, "__globals__", None) + obj_locals = None + unwrap = obj + else: + obj_globals = obj_locals = unwrap = None + + if unwrap is not None: + while True: + if hasattr(unwrap, "__wrapped__"): + unwrap = unwrap.__wrapped__ + continue + if functools := sys.modules.get("functools"): + if isinstance(unwrap, functools.partial): + unwrap = unwrap.func + continue + break + if hasattr(unwrap, "__globals__"): + obj_globals = unwrap.__globals__ + + if globals is None: + globals = obj_globals + if locals is None: + locals = obj_locals + + # "Inject" type parameters into the local namespace + # (unless they are shadowed by assignments *in* the local namespace), + # as a way of emulating annotation scopes when calling `eval()` + if type_params := getattr(obj, "__type_params__", ()): + if locals is None: + locals = {} + locals = {param.__name__: param for param in type_params} | locals + + return_value = { + key: value if not isinstance(value, str) + else eval(_rewrite_star_unpack(value), globals, locals) + for key, value in ann.items() + } + return return_value + + +def type_repr(value): + """Convert a Python value to a format suitable for use with the STRING format. + + This is intended as a helper for tools that support the STRING format but do + not have access to the code that originally produced the annotations. It uses + repr() for most objects. + + """ + if isinstance(value, (type, types.FunctionType, types.BuiltinFunctionType)): + if value.__module__ == "builtins": + return value.__qualname__ + return f"{value.__module__}.{value.__qualname__}" + elif isinstance(value, _Template): + tree = _template_to_ast(value) + return ast.unparse(tree) + if value is ...: + return "..." + return repr(value) + + +def annotations_to_string(annotations): + """Convert an annotation dict containing values to approximately the STRING format. + + Always returns a fresh a dictionary. + """ + return { + n: t if isinstance(t, str) else type_repr(t) + for n, t in annotations.items() + } + + +def _rewrite_star_unpack(arg): + """If the given argument annotation expression is a star unpack e.g. `'*Ts'` + rewrite it to a valid expression. + """ + if arg.startswith("*"): + return f"({arg},)[0]" # E.g. (*Ts,)[0] or (*tuple[int, int],)[0] + else: + return arg + + +def _get_and_call_annotate(obj, format): + """Get the __annotate__ function and call it. + + May not return a fresh dictionary. + """ + annotate = getattr(obj, "__annotate__", None) + if annotate is not None: + ann = call_annotate_function(annotate, format, owner=obj) + if not isinstance(ann, dict): + raise ValueError(f"{obj!r}.__annotate__ returned a non-dict") + return ann + return None + + +_BASE_GET_ANNOTATIONS = type.__dict__["__annotations__"].__get__ + + +def _get_dunder_annotations(obj): + """Return the annotations for an object, checking that it is a dictionary. + + Does not return a fresh dictionary. + """ + # This special case is needed to support types defined under + # from __future__ import annotations, where accessing the __annotations__ + # attribute directly might return annotations for the wrong class. + if isinstance(obj, type): + try: + ann = _BASE_GET_ANNOTATIONS(obj) + except AttributeError: + # For static types, the descriptor raises AttributeError. + return None + else: + ann = getattr(obj, "__annotations__", None) + if ann is None: + return None + + if not isinstance(ann, dict): + raise ValueError(f"{obj!r}.__annotations__ is neither a dict nor None") + return ann diff --git a/Lib/argparse.py b/Lib/argparse.py index bd088ea0e66..88c1f5a7ef3 100644 --- a/Lib/argparse.py +++ b/Lib/argparse.py @@ -18,11 +18,12 @@ 'integers', metavar='int', nargs='+', type=int, help='an integer to be summed') parser.add_argument( - '--log', default=sys.stdout, type=argparse.FileType('w'), + '--log', help='the file where the sum should be written') args = parser.parse_args() - args.log.write('%s' % sum(args.integers)) - args.log.close() + with (open(args.log, 'w') if args.log is not None + else contextlib.nullcontext(sys.stdout)) as log: + log.write('%s' % sum(args.integers)) The module contains the following public classes: @@ -39,7 +40,8 @@ - FileType -- A factory for defining types of files to be created. As the example above shows, instances of FileType are typically passed as - the type= argument of add_argument() calls. + the type= argument of add_argument() calls. Deprecated since + Python 3.14. - Action -- The base class for parser actions. Typically actions are selected by passing strings like 'store_true' or 'append_const' to @@ -159,18 +161,21 @@ class HelpFormatter(object): provided by the class are considered an implementation detail. """ - def __init__(self, - prog, - indent_increment=2, - max_help_position=24, - width=None): - + def __init__( + self, + prog, + indent_increment=2, + max_help_position=24, + width=None, + color=True, + ): # default setting for width if width is None: import shutil width = shutil.get_terminal_size().columns width -= 2 + self._set_color(color) self._prog = prog self._indent_increment = indent_increment self._max_help_position = min(max_help_position, @@ -187,6 +192,16 @@ def __init__(self, self._whitespace_matcher = _re.compile(r'\s+', _re.ASCII) self._long_break_matcher = _re.compile(r'\n\n\n+') + def _set_color(self, color): + from _colorize import can_colorize, decolor, get_theme + + if color and can_colorize(): + self._theme = get_theme(force_color=True).argparse + self._decolor = decolor + else: + self._theme = get_theme(force_no_color=True).argparse + self._decolor = lambda text: text + # =============================== # Section and indentation methods # =============================== @@ -225,7 +240,11 @@ def format_help(self): if self.heading is not SUPPRESS and self.heading is not None: current_indent = self.formatter._current_indent heading_text = _('%(heading)s:') % dict(heading=self.heading) - heading = '%*s%s\n' % (current_indent, '', heading_text) + t = self.formatter._theme + heading = ( + f'{" " * current_indent}' + f'{t.heading}{heading_text}{t.reset}\n' + ) else: heading = '' @@ -262,7 +281,7 @@ def add_argument(self, action): if action.help is not SUPPRESS: # find all invocations - get_invocation = self._format_action_invocation + get_invocation = lambda x: self._decolor(self._format_action_invocation(x)) invocation_lengths = [len(get_invocation(action)) + self._current_indent] for subaction in self._iter_indented_subactions(action): invocation_lengths.append(len(get_invocation(subaction)) + self._current_indent) @@ -296,16 +315,23 @@ def _join_parts(self, part_strings): if part and part is not SUPPRESS]) def _format_usage(self, usage, actions, groups, prefix): + t = self._theme + if prefix is None: prefix = _('usage: ') # if usage is specified, use that if usage is not None: - usage = usage % dict(prog=self._prog) + usage = ( + t.prog_extra + + usage + % {"prog": f"{t.prog}{self._prog}{t.reset}{t.prog_extra}"} + + t.reset + ) # if no optionals or positionals are available, usage is just prog elif usage is None and not actions: - usage = '%(prog)s' % dict(prog=self._prog) + usage = f"{t.prog}{self._prog}{t.reset}" # if optionals and positionals are available, calculate usage elif usage is None: @@ -327,7 +353,7 @@ def _format_usage(self, usage, actions, groups, prefix): # wrap the usage parts if it's too long text_width = self._width - self._current_indent - if len(prefix) + len(usage) > text_width: + if len(prefix) + len(self._decolor(usage)) > text_width: # break usage into wrappable parts opt_parts = self._get_actions_usage_parts(optionals, groups) @@ -343,12 +369,13 @@ def get_lines(parts, indent, prefix=None): else: line_len = indent_length - 1 for part in parts: - if line_len + 1 + len(part) > text_width and line: + part_len = len(self._decolor(part)) + if line_len + 1 + part_len > text_width and line: lines.append(indent + ' '.join(line)) line = [] line_len = indent_length - 1 line.append(part) - line_len += len(part) + 1 + line_len += part_len + 1 if line: lines.append(indent + ' '.join(line)) if prefix is not None: @@ -356,8 +383,9 @@ def get_lines(parts, indent, prefix=None): return lines # if prog is short, follow it with optionals or positionals - if len(prefix) + len(prog) <= 0.75 * text_width: - indent = ' ' * (len(prefix) + len(prog) + 1) + prog_len = len(self._decolor(prog)) + if len(prefix) + prog_len <= 0.75 * text_width: + indent = ' ' * (len(prefix) + prog_len + 1) if opt_parts: lines = get_lines([prog] + opt_parts, indent, prefix) lines.extend(get_lines(pos_parts, indent)) @@ -380,12 +408,18 @@ def get_lines(parts, indent, prefix=None): # join lines into usage usage = '\n'.join(lines) + usage = usage.removeprefix(prog) + usage = f"{t.prog}{prog}{t.reset}{usage}" + # prefix with 'usage:' - return '%s%s\n\n' % (prefix, usage) + return f'{t.usage}{prefix}{t.reset}{usage}\n\n' def _format_actions_usage(self, actions, groups): return ' '.join(self._get_actions_usage_parts(actions, groups)) + def _is_long_option(self, string): + return len(string) > 2 + def _get_actions_usage_parts(self, actions, groups): # find group indices and identify actions in groups group_actions = set() @@ -398,17 +432,18 @@ def _get_actions_usage_parts(self, actions, groups): continue try: - start = actions.index(group._group_actions[0]) + start = min(actions.index(item) for item in group._group_actions) except ValueError: continue else: end = start + len(group._group_actions) - if actions[start:end] == group._group_actions: + if set(actions[start:end]) == set(group._group_actions): group_actions.update(group._group_actions) inserts[start, end] = group # collect all actions format strings parts = [] + t = self._theme for action in actions: # suppressed arguments are marked with None @@ -418,7 +453,11 @@ def _get_actions_usage_parts(self, actions, groups): # produce all arg strings elif not action.option_strings: default = self._get_default_metavar_for_positional(action) - part = self._format_args(action, default) + part = ( + t.summary_action + + self._format_args(action, default) + + t.reset + ) # if it's in a group, strip the outer [] if action in group_actions: @@ -428,18 +467,26 @@ def _get_actions_usage_parts(self, actions, groups): # produce the first way to invoke the option in brackets else: option_string = action.option_strings[0] + if self._is_long_option(option_string): + option_color = t.summary_long_option + else: + option_color = t.summary_short_option # if the Optional doesn't take a value, format is: # -s or --long if action.nargs == 0: part = action.format_usage() + part = f"{option_color}{part}{t.reset}" # if the Optional takes a value, format is: # -s ARGS or --long ARGS else: default = self._get_default_metavar_for_optional(action) args_string = self._format_args(action, default) - part = '%s %s' % (option_string, args_string) + part = ( + f"{option_color}{option_string} " + f"{t.summary_label}{args_string}{t.reset}" + ) # make it look optional if it's not required or in a group if not action.required and action not in group_actions: @@ -486,6 +533,7 @@ def _format_action(self, action): help_width = max(self._width - help_position, 11) action_width = help_position - self._current_indent - 2 action_header = self._format_action_invocation(action) + action_header_no_color = self._decolor(action_header) # no help; start on same line and add a final newline if not action.help: @@ -493,9 +541,15 @@ def _format_action(self, action): action_header = '%*s%s\n' % tup # short action name; start on the same line and pad two spaces - elif len(action_header) <= action_width: - tup = self._current_indent, '', action_width, action_header + elif len(action_header_no_color) <= action_width: + # calculate widths without color codes + action_header_color = action_header + tup = self._current_indent, '', action_width, action_header_no_color action_header = '%*s%-*s ' % tup + # swap in the colored header + action_header = action_header.replace( + action_header_no_color, action_header_color + ) indent_first = 0 # long action name; start on the next line @@ -528,23 +582,42 @@ def _format_action(self, action): return self._join_parts(parts) def _format_action_invocation(self, action): + t = self._theme + if not action.option_strings: default = self._get_default_metavar_for_positional(action) - return ' '.join(self._metavar_formatter(action, default)(1)) + return ( + t.action + + ' '.join(self._metavar_formatter(action, default)(1)) + + t.reset + ) else: + def color_option_strings(strings): + parts = [] + for s in strings: + if self._is_long_option(s): + parts.append(f"{t.long_option}{s}{t.reset}") + else: + parts.append(f"{t.short_option}{s}{t.reset}") + return parts + # if the Optional doesn't take a value, format is: # -s, --long if action.nargs == 0: - return ', '.join(action.option_strings) + option_strings = color_option_strings(action.option_strings) + return ', '.join(option_strings) # if the Optional takes a value, format is: # -s, --long ARGS else: default = self._get_default_metavar_for_optional(action) - args_string = self._format_args(action, default) - return ', '.join(action.option_strings) + ' ' + args_string + option_strings = color_option_strings(action.option_strings) + args_string = ( + f"{t.label}{self._format_args(action, default)}{t.reset}" + ) + return ', '.join(option_strings) + ' ' + args_string def _metavar_formatter(self, action, default_metavar): if action.metavar is not None: @@ -590,16 +663,19 @@ def _format_args(self, action, default_metavar): return result def _expand_help(self, action): + help_string = self._get_help_string(action) + if '%' not in help_string: + return help_string params = dict(vars(action), prog=self._prog) for name in list(params): - if params[name] is SUPPRESS: + value = params[name] + if value is SUPPRESS: del params[name] - for name in list(params): - if hasattr(params[name], '__name__'): - params[name] = params[name].__name__ + elif hasattr(value, '__name__'): + params[name] = value.__name__ if params.get('choices') is not None: params['choices'] = ', '.join(map(str, params['choices'])) - return self._get_help_string(action) % params + return help_string % params def _iter_indented_subactions(self, action): try: @@ -844,22 +920,16 @@ def format_usage(self): return self.option_strings[0] def __call__(self, parser, namespace, values, option_string=None): - raise NotImplementedError(_('.__call__() not defined')) - + raise NotImplementedError('.__call__() not defined') -# FIXME: remove together with `BooleanOptionalAction` deprecated arguments. -_deprecated_default = object() class BooleanOptionalAction(Action): def __init__(self, option_strings, dest, default=None, - type=_deprecated_default, - choices=_deprecated_default, required=False, help=None, - metavar=_deprecated_default, deprecated=False): _option_strings = [] @@ -867,38 +937,19 @@ def __init__(self, _option_strings.append(option_string) if option_string.startswith('--'): + if option_string.startswith('--no-'): + raise ValueError(f'invalid option name {option_string!r} ' + f'for BooleanOptionalAction') option_string = '--no-' + option_string[2:] _option_strings.append(option_string) - # We need `_deprecated` special value to ban explicit arguments that - # match default value. Like: - # parser.add_argument('-f', action=BooleanOptionalAction, type=int) - for field_name in ('type', 'choices', 'metavar'): - if locals()[field_name] is not _deprecated_default: - import warnings - warnings._deprecated( - field_name, - "{name!r} is deprecated as of Python 3.12 and will be " - "removed in Python {remove}.", - remove=(3, 14)) - - if type is _deprecated_default: - type = None - if choices is _deprecated_default: - choices = None - if metavar is _deprecated_default: - metavar = None - super().__init__( option_strings=_option_strings, dest=dest, nargs=0, default=default, - type=type, - choices=choices, required=required, help=help, - metavar=metavar, deprecated=deprecated) @@ -1180,6 +1231,7 @@ def __init__(self, self._name_parser_map = {} self._choices_actions = [] self._deprecated = set() + self._color = True super(_SubParsersAction, self).__init__( option_strings=option_strings, @@ -1195,23 +1247,30 @@ def add_parser(self, name, *, deprecated=False, **kwargs): if kwargs.get('prog') is None: kwargs['prog'] = '%s %s' % (self._prog_prefix, name) + # set color + if kwargs.get('color') is None: + kwargs['color'] = self._color + aliases = kwargs.pop('aliases', ()) if name in self._name_parser_map: - raise ArgumentError(self, _('conflicting subparser: %s') % name) + raise ValueError(f'conflicting subparser: {name}') for alias in aliases: if alias in self._name_parser_map: - raise ArgumentError( - self, _('conflicting subparser alias: %s') % alias) + raise ValueError(f'conflicting subparser alias: {alias}') # create a pseudo-action to hold the choice help if 'help' in kwargs: help = kwargs.pop('help') choice_action = self._ChoicesPseudoAction(name, aliases, help) self._choices_actions.append(choice_action) + else: + choice_action = None # create the parser and add it to the map parser = self._parser_class(**kwargs) + if choice_action is not None: + parser._check_help(choice_action) self._name_parser_map[name] = parser # make parser available under aliases also @@ -1276,7 +1335,7 @@ def __call__(self, parser, namespace, values, option_string=None): # ============== class FileType(object): - """Factory for creating file object types + """Deprecated factory for creating file object types Instances of FileType are typically passed as type= arguments to the ArgumentParser add_argument() method. @@ -1293,6 +1352,12 @@ class FileType(object): """ def __init__(self, mode='r', bufsize=-1, encoding=None, errors=None): + import warnings + warnings.warn( + "FileType is deprecated. Simply open files after parsing arguments.", + category=PendingDeprecationWarning, + stacklevel=2 + ) self._mode = mode self._bufsize = bufsize self._encoding = encoding @@ -1396,7 +1461,7 @@ def __init__(self, self._defaults = {} # determines whether an "option" looks like a negative number - self._negative_number_matcher = _re.compile(r'^-\d+$|^-\d*\.\d+$') + self._negative_number_matcher = _re.compile(r'-\.?\d') # whether or not there are any optionals that look like negative # numbers -- uses a list so it can be shared and edited @@ -1449,7 +1514,8 @@ def add_argument(self, *args, **kwargs): chars = self.prefix_chars if not args or len(args) == 1 and args[0][0] not in chars: if args and 'dest' in kwargs: - raise ValueError('dest supplied twice for positional argument') + raise TypeError('dest supplied twice for positional argument,' + ' did you mean metavar?') kwargs = self._get_positional_kwargs(*args, **kwargs) # otherwise, we're adding an optional argument @@ -1465,27 +1531,34 @@ def add_argument(self, *args, **kwargs): kwargs['default'] = self.argument_default # create the action object, and add it to the parser + action_name = kwargs.get('action') action_class = self._pop_action_class(kwargs) if not callable(action_class): - raise ValueError('unknown action "%s"' % (action_class,)) + raise ValueError(f'unknown action {action_class!r}') action = action_class(**kwargs) + # raise an error if action for positional argument does not + # consume arguments + if not action.option_strings and action.nargs == 0: + raise ValueError(f'action {action_name!r} is not valid for positional arguments') + # raise an error if the action type is not callable type_func = self._registry_get('type', action.type, action.type) if not callable(type_func): - raise ValueError('%r is not callable' % (type_func,)) + raise TypeError(f'{type_func!r} is not callable') if type_func is FileType: - raise ValueError('%r is a FileType class object, instance of it' - ' must be passed' % (type_func,)) + raise TypeError(f'{type_func!r} is a FileType class object, ' + f'instance of it must be passed') # raise an error if the metavar does not match the type if hasattr(self, "_get_formatter"): + formatter = self._get_formatter() try: - self._get_formatter()._format_args(action, None) + formatter._format_args(action, None) except TypeError: raise ValueError("length of metavar tuple does not match nargs") - + self._check_help(action) return self._add_action(action) def add_argument_group(self, *args, **kwargs): @@ -1529,8 +1602,8 @@ def _add_container_actions(self, container): if group.title in title_group_map: # This branch could happen if a derived class added # groups with duplicated titles in __init__ - msg = _('cannot merge actions - two groups are named %r') - raise ValueError(msg % (group.title)) + msg = f'cannot merge actions - two groups are named {group.title!r}' + raise ValueError(msg) title_group_map[group.title] = group # map each action to its group @@ -1571,13 +1644,15 @@ def _add_container_actions(self, container): def _get_positional_kwargs(self, dest, **kwargs): # make sure required is not specified if 'required' in kwargs: - msg = _("'required' is an invalid argument for positionals") + msg = "'required' is an invalid argument for positionals" raise TypeError(msg) # mark positional arguments as required if at least one is # always required nargs = kwargs.get('nargs') - if nargs not in [OPTIONAL, ZERO_OR_MORE, REMAINDER, SUPPRESS, 0]: + if nargs == 0: + raise ValueError('nargs for positionals must be != 0') + if nargs not in [OPTIONAL, ZERO_OR_MORE, REMAINDER, SUPPRESS]: kwargs['required'] = True # return the keyword arguments with no option strings @@ -1590,11 +1665,9 @@ def _get_optional_kwargs(self, *args, **kwargs): for option_string in args: # error on strings that don't start with an appropriate prefix if not option_string[0] in self.prefix_chars: - args = {'option': option_string, - 'prefix_chars': self.prefix_chars} - msg = _('invalid option string %(option)r: ' - 'must start with a character %(prefix_chars)r') - raise ValueError(msg % args) + raise ValueError( + f'invalid option string {option_string!r}: ' + f'must start with a character {self.prefix_chars!r}') # strings starting with two prefix characters are long options option_strings.append(option_string) @@ -1610,8 +1683,8 @@ def _get_optional_kwargs(self, *args, **kwargs): dest_option_string = option_strings[0] dest = dest_option_string.lstrip(self.prefix_chars) if not dest: - msg = _('dest= is required for options like %r') - raise ValueError(msg % option_string) + msg = f'dest= is required for options like {option_string!r}' + raise TypeError(msg) dest = dest.replace('-', '_') # return the updated keyword arguments @@ -1627,8 +1700,8 @@ def _get_handler(self): try: return getattr(self, handler_func_name) except AttributeError: - msg = _('invalid conflict_resolution value: %r') - raise ValueError(msg % self.conflict_handler) + msg = f'invalid conflict_resolution value: {self.conflict_handler!r}' + raise ValueError(msg) def _check_conflict(self, action): @@ -1667,10 +1740,26 @@ def _handle_conflict_resolve(self, action, conflicting_actions): if not action.option_strings: action.container._remove_action(action) + def _check_help(self, action): + if action.help and hasattr(self, "_get_formatter"): + formatter = self._get_formatter() + try: + formatter._expand_help(action) + except (ValueError, TypeError, KeyError) as exc: + raise ValueError('badly formed help string') from exc + class _ArgumentGroup(_ActionsContainer): def __init__(self, container, title=None, description=None, **kwargs): + if 'prefix_chars' in kwargs: + import warnings + depr_msg = ( + "The use of the undocumented 'prefix_chars' parameter in " + "ArgumentParser.add_argument_group() is deprecated." + ) + warnings.warn(depr_msg, DeprecationWarning, stacklevel=3) + # add any missing keyword arguments by checking the container update = kwargs.setdefault update('conflict_handler', container.conflict_handler) @@ -1702,14 +1791,7 @@ def _remove_action(self, action): self._group_actions.remove(action) def add_argument_group(self, *args, **kwargs): - import warnings - warnings.warn( - "Nesting argument groups is deprecated.", - category=DeprecationWarning, - stacklevel=2 - ) - return super().add_argument_group(*args, **kwargs) - + raise ValueError('argument groups cannot be nested') class _MutuallyExclusiveGroup(_ArgumentGroup): @@ -1720,7 +1802,7 @@ def __init__(self, container, required=False): def _add_action(self, action): if action.required: - msg = _('mutually exclusive arguments must be optional') + msg = 'mutually exclusive arguments must be optional' raise ValueError(msg) action = self._container._add_action(action) self._group_actions.append(action) @@ -1730,14 +1812,29 @@ def _remove_action(self, action): self._container._remove_action(action) self._group_actions.remove(action) - def add_mutually_exclusive_group(self, *args, **kwargs): - import warnings - warnings.warn( - "Nesting mutually exclusive groups is deprecated.", - category=DeprecationWarning, - stacklevel=2 - ) - return super().add_mutually_exclusive_group(*args, **kwargs) + def add_mutually_exclusive_group(self, **kwargs): + raise ValueError('mutually exclusive groups cannot be nested') + +def _prog_name(prog=None): + if prog is not None: + return prog + arg0 = _sys.argv[0] + try: + modspec = _sys.modules['__main__'].__spec__ + except (KeyError, AttributeError): + # possibly PYTHONSTARTUP or -X presite or other weird edge case + # no good answer here, so fall back to the default + modspec = None + if modspec is None: + # simple script + return _os.path.basename(arg0) + py = _os.path.basename(_sys.executable) + if modspec.name != '__main__': + # imported module or package + modname = modspec.name.removesuffix('.__main__') + return f'{py} -m {modname}' + # directory or ZIP file + return f'{py} {arg0}' class ArgumentParser(_AttributeHolder, _ActionsContainer): @@ -1760,6 +1857,9 @@ class ArgumentParser(_AttributeHolder, _ActionsContainer): - allow_abbrev -- Allow long options to be abbreviated unambiguously - exit_on_error -- Determines whether or not ArgumentParser exits with error info when an error occurs + - suggest_on_error - Enables suggestions for mistyped argument choices + and subparser names (default: ``False``) + - color - Allow color output in help messages (default: ``False``) """ def __init__(self, @@ -1775,19 +1875,18 @@ def __init__(self, conflict_handler='error', add_help=True, allow_abbrev=True, - exit_on_error=True): - + exit_on_error=True, + *, + suggest_on_error=False, + color=True, + ): superinit = super(ArgumentParser, self).__init__ superinit(description=description, prefix_chars=prefix_chars, argument_default=argument_default, conflict_handler=conflict_handler) - # default setting for prog - if prog is None: - prog = _os.path.basename(_sys.argv[0]) - - self.prog = prog + self.prog = _prog_name(prog) self.usage = usage self.epilog = epilog self.formatter_class = formatter_class @@ -1795,6 +1894,8 @@ def __init__(self, self.add_help = add_help self.allow_abbrev = allow_abbrev self.exit_on_error = exit_on_error + self.suggest_on_error = suggest_on_error + self.color = color add_group = self.add_argument_group self._positionals = add_group(_('positional arguments')) @@ -1844,7 +1945,7 @@ def _get_kwargs(self): def add_subparsers(self, **kwargs): if self._subparsers is not None: - raise ArgumentError(None, _('cannot have multiple subparser arguments')) + raise ValueError('cannot have multiple subparser arguments') # add the parser class to the arguments if it's not present kwargs.setdefault('parser_class', type(self)) @@ -1859,15 +1960,19 @@ def add_subparsers(self, **kwargs): # prog defaults to the usage message of this parser, skipping # optional arguments and with no "usage:" prefix if kwargs.get('prog') is None: - formatter = self._get_formatter() + # Create formatter without color to avoid storing ANSI codes in prog + formatter = self.formatter_class(prog=self.prog) + formatter._set_color(False) positionals = self._get_positional_actions() groups = self._mutually_exclusive_groups - formatter.add_usage(self.usage, positionals, groups, '') + formatter.add_usage(None, positionals, groups, '') kwargs['prog'] = formatter.format_help().strip() # create the parsers action and add it to the positionals list parsers_class = self._pop_action_class(kwargs, 'parsers') action = parsers_class(option_strings=[], **kwargs) + action._color = self.color + self._check_help(action) self._subparsers._add_action(action) # return the created parsers action @@ -2498,7 +2603,6 @@ def _get_values(self, action, arg_strings): value = action.default if isinstance(value, str) and value is not SUPPRESS: value = self._get_value(action, value) - self._check_value(action, value) # when nargs='*' on a positional, if there were no command-line # args, use the default if it is anything other than None @@ -2506,11 +2610,8 @@ def _get_values(self, action, arg_strings): not action.option_strings): if action.default is not None: value = action.default - self._check_value(action, value) else: - # since arg_strings is always [] at this point - # there is no need to use self._check_value(action, value) - value = arg_strings + value = [] # single argument or optional argument produces a single value elif len(arg_strings) == 1 and action.nargs in [None, OPTIONAL]: @@ -2543,8 +2644,7 @@ def _get_values(self, action, arg_strings): def _get_value(self, action, arg_string): type_func = self._registry_get('type', action.type, action.type) if not callable(type_func): - msg = _('%r is not callable') - raise ArgumentError(action, msg % type_func) + raise TypeError(f'{type_func!r} is not callable') # convert the value to the appropriate type try: @@ -2568,14 +2668,27 @@ def _get_value(self, action, arg_string): def _check_value(self, action, value): # converted value must be one of the choices (if specified) choices = action.choices - if choices is not None: - if isinstance(choices, str): - choices = iter(choices) - if value not in choices: - args = {'value': str(value), - 'choices': ', '.join(map(str, action.choices))} - msg = _('invalid choice: %(value)r (choose from %(choices)s)') - raise ArgumentError(action, msg % args) + if choices is None: + return + + if isinstance(choices, str): + choices = iter(choices) + + if value not in choices: + args = {'value': str(value), + 'choices': ', '.join(map(str, action.choices))} + msg = _('invalid choice: %(value)r (choose from %(choices)s)') + + if self.suggest_on_error and isinstance(value, str): + if all(isinstance(choice, str) for choice in action.choices): + import difflib + suggestions = difflib.get_close_matches(value, action.choices, 1) + if suggestions: + args['closest'] = suggestions[0] + msg = _('invalid choice: %(value)r, maybe you meant %(closest)r? ' + '(choose from %(choices)s)') + + raise ArgumentError(action, msg % args) # ======================= # Help-formatting methods @@ -2611,7 +2724,9 @@ def format_help(self): return formatter.format_help() def _get_formatter(self): - return self.formatter_class(prog=self.prog) + formatter = self.formatter_class(prog=self.prog) + formatter._set_color(self.color) + return formatter # ===================== # Help-printing methods diff --git a/Lib/ast.py b/Lib/ast.py index 37b20206b8a..2f11683ecf7 100644 --- a/Lib/ast.py +++ b/Lib/ast.py @@ -20,11 +20,7 @@ :copyright: Copyright 2008 by Armin Ronacher. :license: Python License. """ -import sys -import re from _ast import * -from contextlib import contextmanager, nullcontext -from enum import IntEnum, auto, _simple_enum def parse(source, filename='', mode='exec', *, @@ -319,12 +315,18 @@ def get_docstring(node, clean=True): return text -_line_pattern = re.compile(r"(.*?(?:\r\n|\n|\r|$))") +_line_pattern = None def _splitlines_no_ff(source, maxlines=None): """Split a string into lines ignoring form feed and other chars. This mimics how the Python parser splits source code. """ + global _line_pattern + if _line_pattern is None: + # lazily computed to speedup import time of `ast` + import re + _line_pattern = re.compile(r"(.*?(?:\r\n|\n|\r|$))") + lines = [] for lineno, match in enumerate(_line_pattern.finditer(source), 1): if maxlines is not None and lineno > maxlines: @@ -395,6 +397,88 @@ def walk(node): yield node +def compare( + a, + b, + /, + *, + compare_attributes=False, +): + """Recursively compares two ASTs. + + compare_attributes affects whether AST attributes are considered + in the comparison. If compare_attributes is False (default), then + attributes are ignored. Otherwise they must all be equal. This + option is useful to check whether the ASTs are structurally equal but + might differ in whitespace or similar details. + """ + + sentinel = object() # handle the possibility of a missing attribute/field + + def _compare(a, b): + # Compare two fields on an AST object, which may themselves be + # AST objects, lists of AST objects, or primitive ASDL types + # like identifiers and constants. + if isinstance(a, AST): + return compare( + a, + b, + compare_attributes=compare_attributes, + ) + elif isinstance(a, list): + # If a field is repeated, then both objects will represent + # the value as a list. + if len(a) != len(b): + return False + for a_item, b_item in zip(a, b): + if not _compare(a_item, b_item): + return False + else: + return True + else: + return type(a) is type(b) and a == b + + def _compare_fields(a, b): + if a._fields != b._fields: + return False + for field in a._fields: + a_field = getattr(a, field, sentinel) + b_field = getattr(b, field, sentinel) + if a_field is sentinel and b_field is sentinel: + # both nodes are missing a field at runtime + continue + if a_field is sentinel or b_field is sentinel: + # one of the node is missing a field + return False + if not _compare(a_field, b_field): + return False + else: + return True + + def _compare_attributes(a, b): + if a._attributes != b._attributes: + return False + # Attributes are always ints. + for attr in a._attributes: + a_attr = getattr(a, attr, sentinel) + b_attr = getattr(b, attr, sentinel) + if a_attr is sentinel and b_attr is sentinel: + # both nodes are missing an attribute at runtime + continue + if a_attr != b_attr: + return False + else: + return True + + if type(a) is not type(b): + return False + if not _compare_fields(a, b): + return False + if compare_attributes and not _compare_attributes(a, b): + return False + return True + + class NodeVisitor(object): """ A node visitor base class that walks the abstract syntax tree and calls a @@ -431,27 +515,6 @@ def generic_visit(self, node): elif isinstance(value, AST): self.visit(value) - def visit_Constant(self, node): - value = node.value - type_name = _const_node_type_names.get(type(value)) - if type_name is None: - for cls, name in _const_node_type_names.items(): - if isinstance(value, cls): - type_name = name - break - if type_name is not None: - method = 'visit_' + type_name - try: - visitor = getattr(self, method) - except AttributeError: - pass - else: - import warnings - warnings.warn(f"{method} is deprecated; add visit_Constant", - DeprecationWarning, 2) - return visitor(node) - return self.generic_visit(node) - class NodeTransformer(NodeVisitor): """ @@ -511,151 +574,6 @@ def generic_visit(self, node): setattr(node, field, new_node) return node - -_DEPRECATED_VALUE_ALIAS_MESSAGE = ( - "{name} is deprecated and will be removed in Python {remove}; use value instead" -) -_DEPRECATED_CLASS_MESSAGE = ( - "{name} is deprecated and will be removed in Python {remove}; " - "use ast.Constant instead" -) - - -# If the ast module is loaded more than once, only add deprecated methods once -if not hasattr(Constant, 'n'): - # The following code is for backward compatibility. - # It will be removed in future. - - def _n_getter(self): - """Deprecated. Use value instead.""" - import warnings - warnings._deprecated( - "Attribute n", message=_DEPRECATED_VALUE_ALIAS_MESSAGE, remove=(3, 14) - ) - return self.value - - def _n_setter(self, value): - import warnings - warnings._deprecated( - "Attribute n", message=_DEPRECATED_VALUE_ALIAS_MESSAGE, remove=(3, 14) - ) - self.value = value - - def _s_getter(self): - """Deprecated. Use value instead.""" - import warnings - warnings._deprecated( - "Attribute s", message=_DEPRECATED_VALUE_ALIAS_MESSAGE, remove=(3, 14) - ) - return self.value - - def _s_setter(self, value): - import warnings - warnings._deprecated( - "Attribute s", message=_DEPRECATED_VALUE_ALIAS_MESSAGE, remove=(3, 14) - ) - self.value = value - - Constant.n = property(_n_getter, _n_setter) - Constant.s = property(_s_getter, _s_setter) - -class _ABC(type): - - def __init__(cls, *args): - cls.__doc__ = """Deprecated AST node class. Use ast.Constant instead""" - - def __instancecheck__(cls, inst): - if cls in _const_types: - import warnings - warnings._deprecated( - f"ast.{cls.__qualname__}", - message=_DEPRECATED_CLASS_MESSAGE, - remove=(3, 14) - ) - if not isinstance(inst, Constant): - return False - if cls in _const_types: - try: - value = inst.value - except AttributeError: - return False - else: - return ( - isinstance(value, _const_types[cls]) and - not isinstance(value, _const_types_not.get(cls, ())) - ) - return type.__instancecheck__(cls, inst) - -def _new(cls, *args, **kwargs): - for key in kwargs: - if key not in cls._fields: - # arbitrary keyword arguments are accepted - continue - pos = cls._fields.index(key) - if pos < len(args): - raise TypeError(f"{cls.__name__} got multiple values for argument {key!r}") - if cls in _const_types: - import warnings - warnings._deprecated( - f"ast.{cls.__qualname__}", message=_DEPRECATED_CLASS_MESSAGE, remove=(3, 14) - ) - return Constant(*args, **kwargs) - return Constant.__new__(cls, *args, **kwargs) - -class Num(Constant, metaclass=_ABC): - _fields = ('n',) - __new__ = _new - -class Str(Constant, metaclass=_ABC): - _fields = ('s',) - __new__ = _new - -class Bytes(Constant, metaclass=_ABC): - _fields = ('s',) - __new__ = _new - -class NameConstant(Constant, metaclass=_ABC): - __new__ = _new - -class Ellipsis(Constant, metaclass=_ABC): - _fields = () - - def __new__(cls, *args, **kwargs): - if cls is _ast_Ellipsis: - import warnings - warnings._deprecated( - "ast.Ellipsis", message=_DEPRECATED_CLASS_MESSAGE, remove=(3, 14) - ) - return Constant(..., *args, **kwargs) - return Constant.__new__(cls, *args, **kwargs) - -# Keep another reference to Ellipsis in the global namespace -# so it can be referenced in Ellipsis.__new__ -# (The original "Ellipsis" name is removed from the global namespace later on) -_ast_Ellipsis = Ellipsis - -_const_types = { - Num: (int, float, complex), - Str: (str,), - Bytes: (bytes,), - NameConstant: (type(None), bool), - Ellipsis: (type(...),), -} -_const_types_not = { - Num: (bool,), -} - -_const_node_type_names = { - bool: 'NameConstant', # should be before int - type(None): 'NameConstant', - int: 'Num', - float: 'Num', - complex: 'Num', - str: 'Str', - bytes: 'Bytes', - type(...): 'Ellipsis', -} - class slice(AST): """Deprecated AST node class.""" @@ -696,1147 +614,21 @@ class Param(expr_context): """Deprecated AST node class. Unused in Python 3.""" -# Large float and imaginary literals get turned into infinities in the AST. -# We unparse those infinities to INFSTR. -_INFSTR = "1e" + repr(sys.float_info.max_10_exp + 1) - -@_simple_enum(IntEnum) -class _Precedence: - """Precedence table that originated from python grammar.""" - - NAMED_EXPR = auto() # := - TUPLE = auto() # , - YIELD = auto() # 'yield', 'yield from' - TEST = auto() # 'if'-'else', 'lambda' - OR = auto() # 'or' - AND = auto() # 'and' - NOT = auto() # 'not' - CMP = auto() # '<', '>', '==', '>=', '<=', '!=', - # 'in', 'not in', 'is', 'is not' - EXPR = auto() - BOR = EXPR # '|' - BXOR = auto() # '^' - BAND = auto() # '&' - SHIFT = auto() # '<<', '>>' - ARITH = auto() # '+', '-' - TERM = auto() # '*', '@', '/', '%', '//' - FACTOR = auto() # unary '+', '-', '~' - POWER = auto() # '**' - AWAIT = auto() # 'await' - ATOM = auto() - - def next(self): - try: - return self.__class__(self + 1) - except ValueError: - return self - - -_SINGLE_QUOTES = ("'", '"') -_MULTI_QUOTES = ('"""', "'''") -_ALL_QUOTES = (*_SINGLE_QUOTES, *_MULTI_QUOTES) - -class _Unparser(NodeVisitor): - """Methods in this class recursively traverse an AST and - output source code for the abstract syntax; original formatting - is disregarded.""" - - def __init__(self): - self._source = [] - self._precedences = {} - self._type_ignores = {} - self._indent = 0 - self._in_try_star = False - - def interleave(self, inter, f, seq): - """Call f on each item in seq, calling inter() in between.""" - seq = iter(seq) - try: - f(next(seq)) - except StopIteration: - pass - else: - for x in seq: - inter() - f(x) - - def items_view(self, traverser, items): - """Traverse and separate the given *items* with a comma and append it to - the buffer. If *items* is a single item sequence, a trailing comma - will be added.""" - if len(items) == 1: - traverser(items[0]) - self.write(",") - else: - self.interleave(lambda: self.write(", "), traverser, items) - - def maybe_newline(self): - """Adds a newline if it isn't the start of generated source""" - if self._source: - self.write("\n") - - def fill(self, text=""): - """Indent a piece of text and append it, according to the current - indentation level""" - self.maybe_newline() - self.write(" " * self._indent + text) - - def write(self, *text): - """Add new source parts""" - self._source.extend(text) - - @contextmanager - def buffered(self, buffer = None): - if buffer is None: - buffer = [] - - original_source = self._source - self._source = buffer - yield buffer - self._source = original_source - - @contextmanager - def block(self, *, extra = None): - """A context manager for preparing the source for blocks. It adds - the character':', increases the indentation on enter and decreases - the indentation on exit. If *extra* is given, it will be directly - appended after the colon character. - """ - self.write(":") - if extra: - self.write(extra) - self._indent += 1 - yield - self._indent -= 1 - - @contextmanager - def delimit(self, start, end): - """A context manager for preparing the source for expressions. It adds - *start* to the buffer and enters, after exit it adds *end*.""" - - self.write(start) - yield - self.write(end) - - def delimit_if(self, start, end, condition): - if condition: - return self.delimit(start, end) - else: - return nullcontext() - - def require_parens(self, precedence, node): - """Shortcut to adding precedence related parens""" - return self.delimit_if("(", ")", self.get_precedence(node) > precedence) - - def get_precedence(self, node): - return self._precedences.get(node, _Precedence.TEST) - - def set_precedence(self, precedence, *nodes): - for node in nodes: - self._precedences[node] = precedence - - def get_raw_docstring(self, node): - """If a docstring node is found in the body of the *node* parameter, - return that docstring node, None otherwise. - - Logic mirrored from ``_PyAST_GetDocString``.""" - if not isinstance( - node, (AsyncFunctionDef, FunctionDef, ClassDef, Module) - ) or len(node.body) < 1: - return None - node = node.body[0] - if not isinstance(node, Expr): - return None - node = node.value - if isinstance(node, Constant) and isinstance(node.value, str): - return node - - def get_type_comment(self, node): - comment = self._type_ignores.get(node.lineno) or node.type_comment - if comment is not None: - return f" # type: {comment}" - - def traverse(self, node): - if isinstance(node, list): - for item in node: - self.traverse(item) - else: - super().visit(node) - - # Note: as visit() resets the output text, do NOT rely on - # NodeVisitor.generic_visit to handle any nodes (as it calls back in to - # the subclass visit() method, which resets self._source to an empty list) - def visit(self, node): - """Outputs a source code string that, if converted back to an ast - (using ast.parse) will generate an AST equivalent to *node*""" - self._source = [] - self.traverse(node) - return "".join(self._source) - - def _write_docstring_and_traverse_body(self, node): - if (docstring := self.get_raw_docstring(node)): - self._write_docstring(docstring) - self.traverse(node.body[1:]) - else: - self.traverse(node.body) - - def visit_Module(self, node): - self._type_ignores = { - ignore.lineno: f"ignore{ignore.tag}" - for ignore in node.type_ignores - } - self._write_docstring_and_traverse_body(node) - self._type_ignores.clear() - - def visit_FunctionType(self, node): - with self.delimit("(", ")"): - self.interleave( - lambda: self.write(", "), self.traverse, node.argtypes - ) - - self.write(" -> ") - self.traverse(node.returns) - - def visit_Expr(self, node): - self.fill() - self.set_precedence(_Precedence.YIELD, node.value) - self.traverse(node.value) - - def visit_NamedExpr(self, node): - with self.require_parens(_Precedence.NAMED_EXPR, node): - self.set_precedence(_Precedence.ATOM, node.target, node.value) - self.traverse(node.target) - self.write(" := ") - self.traverse(node.value) - - def visit_Import(self, node): - self.fill("import ") - self.interleave(lambda: self.write(", "), self.traverse, node.names) - - def visit_ImportFrom(self, node): - self.fill("from ") - self.write("." * (node.level or 0)) - if node.module: - self.write(node.module) - self.write(" import ") - self.interleave(lambda: self.write(", "), self.traverse, node.names) - - def visit_Assign(self, node): - self.fill() - for target in node.targets: - self.set_precedence(_Precedence.TUPLE, target) - self.traverse(target) - self.write(" = ") - self.traverse(node.value) - if type_comment := self.get_type_comment(node): - self.write(type_comment) - - def visit_AugAssign(self, node): - self.fill() - self.traverse(node.target) - self.write(" " + self.binop[node.op.__class__.__name__] + "= ") - self.traverse(node.value) - - def visit_AnnAssign(self, node): - self.fill() - with self.delimit_if("(", ")", not node.simple and isinstance(node.target, Name)): - self.traverse(node.target) - self.write(": ") - self.traverse(node.annotation) - if node.value: - self.write(" = ") - self.traverse(node.value) - - def visit_Return(self, node): - self.fill("return") - if node.value: - self.write(" ") - self.traverse(node.value) - - def visit_Pass(self, node): - self.fill("pass") - - def visit_Break(self, node): - self.fill("break") - - def visit_Continue(self, node): - self.fill("continue") - - def visit_Delete(self, node): - self.fill("del ") - self.interleave(lambda: self.write(", "), self.traverse, node.targets) - - def visit_Assert(self, node): - self.fill("assert ") - self.traverse(node.test) - if node.msg: - self.write(", ") - self.traverse(node.msg) - - def visit_Global(self, node): - self.fill("global ") - self.interleave(lambda: self.write(", "), self.write, node.names) - - def visit_Nonlocal(self, node): - self.fill("nonlocal ") - self.interleave(lambda: self.write(", "), self.write, node.names) - - def visit_Await(self, node): - with self.require_parens(_Precedence.AWAIT, node): - self.write("await") - if node.value: - self.write(" ") - self.set_precedence(_Precedence.ATOM, node.value) - self.traverse(node.value) - - def visit_Yield(self, node): - with self.require_parens(_Precedence.YIELD, node): - self.write("yield") - if node.value: - self.write(" ") - self.set_precedence(_Precedence.ATOM, node.value) - self.traverse(node.value) - - def visit_YieldFrom(self, node): - with self.require_parens(_Precedence.YIELD, node): - self.write("yield from ") - if not node.value: - raise ValueError("Node can't be used without a value attribute.") - self.set_precedence(_Precedence.ATOM, node.value) - self.traverse(node.value) - - def visit_Raise(self, node): - self.fill("raise") - if not node.exc: - if node.cause: - raise ValueError(f"Node can't use cause without an exception.") - return - self.write(" ") - self.traverse(node.exc) - if node.cause: - self.write(" from ") - self.traverse(node.cause) - - def do_visit_try(self, node): - self.fill("try") - with self.block(): - self.traverse(node.body) - for ex in node.handlers: - self.traverse(ex) - if node.orelse: - self.fill("else") - with self.block(): - self.traverse(node.orelse) - if node.finalbody: - self.fill("finally") - with self.block(): - self.traverse(node.finalbody) - - def visit_Try(self, node): - prev_in_try_star = self._in_try_star - try: - self._in_try_star = False - self.do_visit_try(node) - finally: - self._in_try_star = prev_in_try_star - - def visit_TryStar(self, node): - prev_in_try_star = self._in_try_star - try: - self._in_try_star = True - self.do_visit_try(node) - finally: - self._in_try_star = prev_in_try_star - - def visit_ExceptHandler(self, node): - self.fill("except*" if self._in_try_star else "except") - if node.type: - self.write(" ") - self.traverse(node.type) - if node.name: - self.write(" as ") - self.write(node.name) - with self.block(): - self.traverse(node.body) - - def visit_ClassDef(self, node): - self.maybe_newline() - for deco in node.decorator_list: - self.fill("@") - self.traverse(deco) - self.fill("class " + node.name) - if hasattr(node, "type_params"): - self._type_params_helper(node.type_params) - with self.delimit_if("(", ")", condition = node.bases or node.keywords): - comma = False - for e in node.bases: - if comma: - self.write(", ") - else: - comma = True - self.traverse(e) - for e in node.keywords: - if comma: - self.write(", ") - else: - comma = True - self.traverse(e) - - with self.block(): - self._write_docstring_and_traverse_body(node) - - def visit_FunctionDef(self, node): - self._function_helper(node, "def") - - def visit_AsyncFunctionDef(self, node): - self._function_helper(node, "async def") - - def _function_helper(self, node, fill_suffix): - self.maybe_newline() - for deco in node.decorator_list: - self.fill("@") - self.traverse(deco) - def_str = fill_suffix + " " + node.name - self.fill(def_str) - if hasattr(node, "type_params"): - self._type_params_helper(node.type_params) - with self.delimit("(", ")"): - self.traverse(node.args) - if node.returns: - self.write(" -> ") - self.traverse(node.returns) - with self.block(extra=self.get_type_comment(node)): - self._write_docstring_and_traverse_body(node) - - def _type_params_helper(self, type_params): - if type_params is not None and len(type_params) > 0: - with self.delimit("[", "]"): - self.interleave(lambda: self.write(", "), self.traverse, type_params) - - def visit_TypeVar(self, node): - self.write(node.name) - if node.bound: - self.write(": ") - self.traverse(node.bound) - if node.default_value: - self.write(" = ") - self.traverse(node.default_value) - - def visit_TypeVarTuple(self, node): - self.write("*" + node.name) - if node.default_value: - self.write(" = ") - self.traverse(node.default_value) - - def visit_ParamSpec(self, node): - self.write("**" + node.name) - if node.default_value: - self.write(" = ") - self.traverse(node.default_value) - - def visit_TypeAlias(self, node): - self.fill("type ") - self.traverse(node.name) - self._type_params_helper(node.type_params) - self.write(" = ") - self.traverse(node.value) - - def visit_For(self, node): - self._for_helper("for ", node) - - def visit_AsyncFor(self, node): - self._for_helper("async for ", node) - - def _for_helper(self, fill, node): - self.fill(fill) - self.set_precedence(_Precedence.TUPLE, node.target) - self.traverse(node.target) - self.write(" in ") - self.traverse(node.iter) - with self.block(extra=self.get_type_comment(node)): - self.traverse(node.body) - if node.orelse: - self.fill("else") - with self.block(): - self.traverse(node.orelse) - - def visit_If(self, node): - self.fill("if ") - self.traverse(node.test) - with self.block(): - self.traverse(node.body) - # collapse nested ifs into equivalent elifs. - while node.orelse and len(node.orelse) == 1 and isinstance(node.orelse[0], If): - node = node.orelse[0] - self.fill("elif ") - self.traverse(node.test) - with self.block(): - self.traverse(node.body) - # final else - if node.orelse: - self.fill("else") - with self.block(): - self.traverse(node.orelse) - - def visit_While(self, node): - self.fill("while ") - self.traverse(node.test) - with self.block(): - self.traverse(node.body) - if node.orelse: - self.fill("else") - with self.block(): - self.traverse(node.orelse) - - def visit_With(self, node): - self.fill("with ") - self.interleave(lambda: self.write(", "), self.traverse, node.items) - with self.block(extra=self.get_type_comment(node)): - self.traverse(node.body) - - def visit_AsyncWith(self, node): - self.fill("async with ") - self.interleave(lambda: self.write(", "), self.traverse, node.items) - with self.block(extra=self.get_type_comment(node)): - self.traverse(node.body) - - def _str_literal_helper( - self, string, *, quote_types=_ALL_QUOTES, escape_special_whitespace=False - ): - """Helper for writing string literals, minimizing escapes. - Returns the tuple (string literal to write, possible quote types). - """ - def escape_char(c): - # \n and \t are non-printable, but we only escape them if - # escape_special_whitespace is True - if not escape_special_whitespace and c in "\n\t": - return c - # Always escape backslashes and other non-printable characters - if c == "\\" or not c.isprintable(): - return c.encode("unicode_escape").decode("ascii") - return c - - escaped_string = "".join(map(escape_char, string)) - possible_quotes = quote_types - if "\n" in escaped_string: - possible_quotes = [q for q in possible_quotes if q in _MULTI_QUOTES] - possible_quotes = [q for q in possible_quotes if q not in escaped_string] - if not possible_quotes: - # If there aren't any possible_quotes, fallback to using repr - # on the original string. Try to use a quote from quote_types, - # e.g., so that we use triple quotes for docstrings. - string = repr(string) - quote = next((q for q in quote_types if string[0] in q), string[0]) - return string[1:-1], [quote] - if escaped_string: - # Sort so that we prefer '''"''' over """\"""" - possible_quotes.sort(key=lambda q: q[0] == escaped_string[-1]) - # If we're using triple quotes and we'd need to escape a final - # quote, escape it - if possible_quotes[0][0] == escaped_string[-1]: - assert len(possible_quotes[0]) == 3 - escaped_string = escaped_string[:-1] + "\\" + escaped_string[-1] - return escaped_string, possible_quotes - - def _write_str_avoiding_backslashes(self, string, *, quote_types=_ALL_QUOTES): - """Write string literal value with a best effort attempt to avoid backslashes.""" - string, quote_types = self._str_literal_helper(string, quote_types=quote_types) - quote_type = quote_types[0] - self.write(f"{quote_type}{string}{quote_type}") - - def visit_JoinedStr(self, node): - self.write("f") - - fstring_parts = [] - for value in node.values: - with self.buffered() as buffer: - self._write_fstring_inner(value) - fstring_parts.append( - ("".join(buffer), isinstance(value, Constant)) - ) - - new_fstring_parts = [] - quote_types = list(_ALL_QUOTES) - fallback_to_repr = False - for value, is_constant in fstring_parts: - if is_constant: - value, new_quote_types = self._str_literal_helper( - value, - quote_types=quote_types, - escape_special_whitespace=True, - ) - if set(new_quote_types).isdisjoint(quote_types): - fallback_to_repr = True - break - quote_types = new_quote_types - else: - if "\n" in value: - quote_types = [q for q in quote_types if q in _MULTI_QUOTES] - assert quote_types - - new_quote_types = [q for q in quote_types if q not in value] - if new_quote_types: - quote_types = new_quote_types - new_fstring_parts.append(value) - - if fallback_to_repr: - # If we weren't able to find a quote type that works for all parts - # of the JoinedStr, fallback to using repr and triple single quotes. - quote_types = ["'''"] - new_fstring_parts.clear() - for value, is_constant in fstring_parts: - if is_constant: - value = repr('"' + value) # force repr to use single quotes - expected_prefix = "'\"" - assert value.startswith(expected_prefix), repr(value) - value = value[len(expected_prefix):-1] - new_fstring_parts.append(value) - - value = "".join(new_fstring_parts) - quote_type = quote_types[0] - self.write(f"{quote_type}{value}{quote_type}") - - def _write_fstring_inner(self, node, is_format_spec=False): - if isinstance(node, JoinedStr): - # for both the f-string itself, and format_spec - for value in node.values: - self._write_fstring_inner(value, is_format_spec=is_format_spec) - elif isinstance(node, Constant) and isinstance(node.value, str): - value = node.value.replace("{", "{{").replace("}", "}}") - - if is_format_spec: - value = value.replace("\\", "\\\\") - value = value.replace("'", "\\'") - value = value.replace('"', '\\"') - value = value.replace("\n", "\\n") - self.write(value) - elif isinstance(node, FormattedValue): - self.visit_FormattedValue(node) - else: - raise ValueError(f"Unexpected node inside JoinedStr, {node!r}") - - def visit_FormattedValue(self, node): - def unparse_inner(inner): - unparser = type(self)() - unparser.set_precedence(_Precedence.TEST.next(), inner) - return unparser.visit(inner) - - with self.delimit("{", "}"): - expr = unparse_inner(node.value) - if expr.startswith("{"): - # Separate pair of opening brackets as "{ {" - self.write(" ") - self.write(expr) - if node.conversion != -1: - self.write(f"!{chr(node.conversion)}") - if node.format_spec: - self.write(":") - self._write_fstring_inner(node.format_spec, is_format_spec=True) - - def visit_Name(self, node): - self.write(node.id) - - def _write_docstring(self, node): - self.fill() - if node.kind == "u": - self.write("u") - self._write_str_avoiding_backslashes(node.value, quote_types=_MULTI_QUOTES) - - def _write_constant(self, value): - if isinstance(value, (float, complex)): - # Substitute overflowing decimal literal for AST infinities, - # and inf - inf for NaNs. - self.write( - repr(value) - .replace("inf", _INFSTR) - .replace("nan", f"({_INFSTR}-{_INFSTR})") - ) - else: - self.write(repr(value)) - - def visit_Constant(self, node): - value = node.value - if isinstance(value, tuple): - with self.delimit("(", ")"): - self.items_view(self._write_constant, value) - elif value is ...: - self.write("...") - else: - if node.kind == "u": - self.write("u") - self._write_constant(node.value) - - def visit_List(self, node): - with self.delimit("[", "]"): - self.interleave(lambda: self.write(", "), self.traverse, node.elts) - - def visit_ListComp(self, node): - with self.delimit("[", "]"): - self.traverse(node.elt) - for gen in node.generators: - self.traverse(gen) - - def visit_GeneratorExp(self, node): - with self.delimit("(", ")"): - self.traverse(node.elt) - for gen in node.generators: - self.traverse(gen) - - def visit_SetComp(self, node): - with self.delimit("{", "}"): - self.traverse(node.elt) - for gen in node.generators: - self.traverse(gen) - - def visit_DictComp(self, node): - with self.delimit("{", "}"): - self.traverse(node.key) - self.write(": ") - self.traverse(node.value) - for gen in node.generators: - self.traverse(gen) - - def visit_comprehension(self, node): - if node.is_async: - self.write(" async for ") - else: - self.write(" for ") - self.set_precedence(_Precedence.TUPLE, node.target) - self.traverse(node.target) - self.write(" in ") - self.set_precedence(_Precedence.TEST.next(), node.iter, *node.ifs) - self.traverse(node.iter) - for if_clause in node.ifs: - self.write(" if ") - self.traverse(if_clause) - - def visit_IfExp(self, node): - with self.require_parens(_Precedence.TEST, node): - self.set_precedence(_Precedence.TEST.next(), node.body, node.test) - self.traverse(node.body) - self.write(" if ") - self.traverse(node.test) - self.write(" else ") - self.set_precedence(_Precedence.TEST, node.orelse) - self.traverse(node.orelse) - - def visit_Set(self, node): - if node.elts: - with self.delimit("{", "}"): - self.interleave(lambda: self.write(", "), self.traverse, node.elts) - else: - # `{}` would be interpreted as a dictionary literal, and - # `set` might be shadowed. Thus: - self.write('{*()}') - - def visit_Dict(self, node): - def write_key_value_pair(k, v): - self.traverse(k) - self.write(": ") - self.traverse(v) - - def write_item(item): - k, v = item - if k is None: - # for dictionary unpacking operator in dicts {**{'y': 2}} - # see PEP 448 for details - self.write("**") - self.set_precedence(_Precedence.EXPR, v) - self.traverse(v) - else: - write_key_value_pair(k, v) - - with self.delimit("{", "}"): - self.interleave( - lambda: self.write(", "), write_item, zip(node.keys, node.values) - ) - - def visit_Tuple(self, node): - with self.delimit_if( - "(", - ")", - len(node.elts) == 0 or self.get_precedence(node) > _Precedence.TUPLE - ): - self.items_view(self.traverse, node.elts) - - unop = {"Invert": "~", "Not": "not", "UAdd": "+", "USub": "-"} - unop_precedence = { - "not": _Precedence.NOT, - "~": _Precedence.FACTOR, - "+": _Precedence.FACTOR, - "-": _Precedence.FACTOR, - } - - def visit_UnaryOp(self, node): - operator = self.unop[node.op.__class__.__name__] - operator_precedence = self.unop_precedence[operator] - with self.require_parens(operator_precedence, node): - self.write(operator) - # factor prefixes (+, -, ~) shouldn't be separated - # from the value they belong, (e.g: +1 instead of + 1) - if operator_precedence is not _Precedence.FACTOR: - self.write(" ") - self.set_precedence(operator_precedence, node.operand) - self.traverse(node.operand) - - binop = { - "Add": "+", - "Sub": "-", - "Mult": "*", - "MatMult": "@", - "Div": "/", - "Mod": "%", - "LShift": "<<", - "RShift": ">>", - "BitOr": "|", - "BitXor": "^", - "BitAnd": "&", - "FloorDiv": "//", - "Pow": "**", - } - - binop_precedence = { - "+": _Precedence.ARITH, - "-": _Precedence.ARITH, - "*": _Precedence.TERM, - "@": _Precedence.TERM, - "/": _Precedence.TERM, - "%": _Precedence.TERM, - "<<": _Precedence.SHIFT, - ">>": _Precedence.SHIFT, - "|": _Precedence.BOR, - "^": _Precedence.BXOR, - "&": _Precedence.BAND, - "//": _Precedence.TERM, - "**": _Precedence.POWER, - } - - binop_rassoc = frozenset(("**",)) - def visit_BinOp(self, node): - operator = self.binop[node.op.__class__.__name__] - operator_precedence = self.binop_precedence[operator] - with self.require_parens(operator_precedence, node): - if operator in self.binop_rassoc: - left_precedence = operator_precedence.next() - right_precedence = operator_precedence - else: - left_precedence = operator_precedence - right_precedence = operator_precedence.next() - - self.set_precedence(left_precedence, node.left) - self.traverse(node.left) - self.write(f" {operator} ") - self.set_precedence(right_precedence, node.right) - self.traverse(node.right) - - cmpops = { - "Eq": "==", - "NotEq": "!=", - "Lt": "<", - "LtE": "<=", - "Gt": ">", - "GtE": ">=", - "Is": "is", - "IsNot": "is not", - "In": "in", - "NotIn": "not in", - } - - def visit_Compare(self, node): - with self.require_parens(_Precedence.CMP, node): - self.set_precedence(_Precedence.CMP.next(), node.left, *node.comparators) - self.traverse(node.left) - for o, e in zip(node.ops, node.comparators): - self.write(" " + self.cmpops[o.__class__.__name__] + " ") - self.traverse(e) - - boolops = {"And": "and", "Or": "or"} - boolop_precedence = {"and": _Precedence.AND, "or": _Precedence.OR} - - def visit_BoolOp(self, node): - operator = self.boolops[node.op.__class__.__name__] - operator_precedence = self.boolop_precedence[operator] - - def increasing_level_traverse(node): - nonlocal operator_precedence - operator_precedence = operator_precedence.next() - self.set_precedence(operator_precedence, node) - self.traverse(node) - - with self.require_parens(operator_precedence, node): - s = f" {operator} " - self.interleave(lambda: self.write(s), increasing_level_traverse, node.values) - - def visit_Attribute(self, node): - self.set_precedence(_Precedence.ATOM, node.value) - self.traverse(node.value) - # Special case: 3.__abs__() is a syntax error, so if node.value - # is an integer literal then we need to either parenthesize - # it or add an extra space to get 3 .__abs__(). - if isinstance(node.value, Constant) and isinstance(node.value.value, int): - self.write(" ") - self.write(".") - self.write(node.attr) - - def visit_Call(self, node): - self.set_precedence(_Precedence.ATOM, node.func) - self.traverse(node.func) - with self.delimit("(", ")"): - comma = False - for e in node.args: - if comma: - self.write(", ") - else: - comma = True - self.traverse(e) - for e in node.keywords: - if comma: - self.write(", ") - else: - comma = True - self.traverse(e) - - def visit_Subscript(self, node): - def is_non_empty_tuple(slice_value): - return ( - isinstance(slice_value, Tuple) - and slice_value.elts - ) - - self.set_precedence(_Precedence.ATOM, node.value) - self.traverse(node.value) - with self.delimit("[", "]"): - if is_non_empty_tuple(node.slice): - # parentheses can be omitted if the tuple isn't empty - self.items_view(self.traverse, node.slice.elts) - else: - self.traverse(node.slice) - - def visit_Starred(self, node): - self.write("*") - self.set_precedence(_Precedence.EXPR, node.value) - self.traverse(node.value) - - def visit_Ellipsis(self, node): - self.write("...") - - def visit_Slice(self, node): - if node.lower: - self.traverse(node.lower) - self.write(":") - if node.upper: - self.traverse(node.upper) - if node.step: - self.write(":") - self.traverse(node.step) - - def visit_Match(self, node): - self.fill("match ") - self.traverse(node.subject) - with self.block(): - for case in node.cases: - self.traverse(case) - - def visit_arg(self, node): - self.write(node.arg) - if node.annotation: - self.write(": ") - self.traverse(node.annotation) - - def visit_arguments(self, node): - first = True - # normal arguments - all_args = node.posonlyargs + node.args - defaults = [None] * (len(all_args) - len(node.defaults)) + node.defaults - for index, elements in enumerate(zip(all_args, defaults), 1): - a, d = elements - if first: - first = False - else: - self.write(", ") - self.traverse(a) - if d: - self.write("=") - self.traverse(d) - if index == len(node.posonlyargs): - self.write(", /") - - # varargs, or bare '*' if no varargs but keyword-only arguments present - if node.vararg or node.kwonlyargs: - if first: - first = False - else: - self.write(", ") - self.write("*") - if node.vararg: - self.write(node.vararg.arg) - if node.vararg.annotation: - self.write(": ") - self.traverse(node.vararg.annotation) - - # keyword-only arguments - if node.kwonlyargs: - for a, d in zip(node.kwonlyargs, node.kw_defaults): - self.write(", ") - self.traverse(a) - if d: - self.write("=") - self.traverse(d) - - # kwargs - if node.kwarg: - if first: - first = False - else: - self.write(", ") - self.write("**" + node.kwarg.arg) - if node.kwarg.annotation: - self.write(": ") - self.traverse(node.kwarg.annotation) - - def visit_keyword(self, node): - if node.arg is None: - self.write("**") - else: - self.write(node.arg) - self.write("=") - self.traverse(node.value) - - def visit_Lambda(self, node): - with self.require_parens(_Precedence.TEST, node): - self.write("lambda") - with self.buffered() as buffer: - self.traverse(node.args) - if buffer: - self.write(" ", *buffer) - self.write(": ") - self.set_precedence(_Precedence.TEST, node.body) - self.traverse(node.body) - - def visit_alias(self, node): - self.write(node.name) - if node.asname: - self.write(" as " + node.asname) - - def visit_withitem(self, node): - self.traverse(node.context_expr) - if node.optional_vars: - self.write(" as ") - self.traverse(node.optional_vars) - - def visit_match_case(self, node): - self.fill("case ") - self.traverse(node.pattern) - if node.guard: - self.write(" if ") - self.traverse(node.guard) - with self.block(): - self.traverse(node.body) - - def visit_MatchValue(self, node): - self.traverse(node.value) - - def visit_MatchSingleton(self, node): - self._write_constant(node.value) - - def visit_MatchSequence(self, node): - with self.delimit("[", "]"): - self.interleave( - lambda: self.write(", "), self.traverse, node.patterns - ) - - def visit_MatchStar(self, node): - name = node.name - if name is None: - name = "_" - self.write(f"*{name}") - - def visit_MatchMapping(self, node): - def write_key_pattern_pair(pair): - k, p = pair - self.traverse(k) - self.write(": ") - self.traverse(p) - - with self.delimit("{", "}"): - keys = node.keys - self.interleave( - lambda: self.write(", "), - write_key_pattern_pair, - zip(keys, node.patterns, strict=True), - ) - rest = node.rest - if rest is not None: - if keys: - self.write(", ") - self.write(f"**{rest}") - - def visit_MatchClass(self, node): - self.set_precedence(_Precedence.ATOM, node.cls) - self.traverse(node.cls) - with self.delimit("(", ")"): - patterns = node.patterns - self.interleave( - lambda: self.write(", "), self.traverse, patterns - ) - attrs = node.kwd_attrs - if attrs: - def write_attr_pattern(pair): - attr, pattern = pair - self.write(f"{attr}=") - self.traverse(pattern) - - if patterns: - self.write(", ") - self.interleave( - lambda: self.write(", "), - write_attr_pattern, - zip(attrs, node.kwd_patterns, strict=True), - ) - - def visit_MatchAs(self, node): - name = node.name - pattern = node.pattern - if name is None: - self.write("_") - elif pattern is None: - self.write(node.name) - else: - with self.require_parens(_Precedence.TEST, node): - self.set_precedence(_Precedence.BOR, node.pattern) - self.traverse(node.pattern) - self.write(f" as {node.name}") - - def visit_MatchOr(self, node): - with self.require_parens(_Precedence.BOR, node): - self.set_precedence(_Precedence.BOR.next(), *node.patterns) - self.interleave(lambda: self.write(" | "), self.traverse, node.patterns) - def unparse(ast_obj): - unparser = _Unparser() + global _Unparser + try: + unparser = _Unparser() + except NameError: + from _ast_unparse import Unparser as _Unparser + unparser = _Unparser() return unparser.visit(ast_obj) -_deprecated_globals = { - name: globals().pop(name) - for name in ('Num', 'Str', 'Bytes', 'NameConstant', 'Ellipsis') -} - -def __getattr__(name): - if name in _deprecated_globals: - globals()[name] = value = _deprecated_globals[name] - import warnings - warnings._deprecated( - f"ast.{name}", message=_DEPRECATED_CLASS_MESSAGE, remove=(3, 14) - ) - return value - raise AttributeError(f"module 'ast' has no attribute '{name}'") - - -def main(): +def main(args=None): import argparse + import sys - parser = argparse.ArgumentParser(prog='python -m ast') + parser = argparse.ArgumentParser(color=True) parser.add_argument('infile', nargs='?', default='-', help='the file to parse; defaults to stdin') parser.add_argument('-m', '--mode', default='exec', @@ -1849,7 +641,16 @@ def main(): 'column offsets') parser.add_argument('-i', '--indent', type=int, default=3, help='indentation of nodes (number of spaces)') - args = parser.parse_args() + parser.add_argument('--feature-version', + type=str, default=None, metavar='VERSION', + help='Python version in the format 3.x ' + '(for example, 3.10)') + parser.add_argument('-O', '--optimize', + type=int, default=-1, metavar='LEVEL', + help='optimization level for parser (default -1)') + parser.add_argument('--show-empty', default=False, action='store_true', + help='show empty lists and fields in dump output') + args = parser.parse_args(args) if args.infile == '-': name = '' @@ -1858,8 +659,22 @@ def main(): name = args.infile with open(args.infile, 'rb') as infile: source = infile.read() - tree = parse(source, name, args.mode, type_comments=args.no_type_comments) - print(dump(tree, include_attributes=args.include_attributes, indent=args.indent)) + + # Process feature_version + feature_version = None + if args.feature_version: + try: + major, minor = map(int, args.feature_version.split('.', 1)) + except ValueError: + parser.error('Invalid format for --feature-version; ' + 'expected format 3.x (for example, 3.10)') + + feature_version = (major, minor) + + tree = parse(source, name, args.mode, type_comments=args.no_type_comments, + feature_version=feature_version, optimize=args.optimize) + print(dump(tree, include_attributes=args.include_attributes, + indent=args.indent, show_empty=args.show_empty)) if __name__ == '__main__': main() diff --git a/Lib/asyncio/__init__.py b/Lib/asyncio/__init__.py index 03165a425eb..32a5dbae03a 100644 --- a/Lib/asyncio/__init__.py +++ b/Lib/asyncio/__init__.py @@ -10,6 +10,7 @@ from .events import * from .exceptions import * from .futures import * +from .graph import * from .locks import * from .protocols import * from .runners import * @@ -27,6 +28,7 @@ events.__all__ + exceptions.__all__ + futures.__all__ + + graph.__all__ + locks.__all__ + protocols.__all__ + runners.__all__ + @@ -45,3 +47,28 @@ else: from .unix_events import * # pragma: no cover __all__ += unix_events.__all__ + +def __getattr__(name: str): + import warnings + + match name: + case "AbstractEventLoopPolicy": + warnings._deprecated(f"asyncio.{name}", remove=(3, 16)) + return events._AbstractEventLoopPolicy + case "DefaultEventLoopPolicy": + warnings._deprecated(f"asyncio.{name}", remove=(3, 16)) + if sys.platform == 'win32': + return windows_events._DefaultEventLoopPolicy + return unix_events._DefaultEventLoopPolicy + case "WindowsSelectorEventLoopPolicy": + if sys.platform == 'win32': + warnings._deprecated(f"asyncio.{name}", remove=(3, 16)) + return windows_events._WindowsSelectorEventLoopPolicy + # Else fall through to the AttributeError below. + case "WindowsProactorEventLoopPolicy": + if sys.platform == 'win32': + warnings._deprecated(f"asyncio.{name}", remove=(3, 16)) + return windows_events._WindowsProactorEventLoopPolicy + # Else fall through to the AttributeError below. + + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") diff --git a/Lib/asyncio/__main__.py b/Lib/asyncio/__main__.py index 18bb87a5bc4..e07dd52a2a5 100644 --- a/Lib/asyncio/__main__.py +++ b/Lib/asyncio/__main__.py @@ -1,41 +1,53 @@ +import argparse import ast import asyncio -import code +import asyncio.tools import concurrent.futures +import contextvars import inspect +import os +import site import sys import threading import types import warnings +from _colorize import get_theme +from _pyrepl.console import InteractiveColoredConsole + from . import futures -class AsyncIOInteractiveConsole(code.InteractiveConsole): +class AsyncIOInteractiveConsole(InteractiveColoredConsole): def __init__(self, locals, loop): - super().__init__(locals) + super().__init__(locals, filename="") self.compile.compiler.flags |= ast.PyCF_ALLOW_TOP_LEVEL_AWAIT self.loop = loop + self.context = contextvars.copy_context() def runcode(self, code): + global return_code future = concurrent.futures.Future() def callback(): + global return_code global repl_future - global repl_future_interrupted + global keyboard_interrupted repl_future = None - repl_future_interrupted = False + keyboard_interrupted = False func = types.FunctionType(code, self.locals) try: coro = func() - except SystemExit: - raise + except SystemExit as se: + return_code = se.code + self.loop.stop() + return except KeyboardInterrupt as ex: - repl_future_interrupted = True + keyboard_interrupted = True future.set_exception(ex) return except BaseException as ex: @@ -47,39 +59,72 @@ def callback(): return try: - repl_future = self.loop.create_task(coro) + repl_future = self.loop.create_task(coro, context=self.context) futures._chain_future(repl_future, future) except BaseException as exc: future.set_exception(exc) - loop.call_soon_threadsafe(callback) + self.loop.call_soon_threadsafe(callback, context=self.context) try: return future.result() - except SystemExit: - raise + except SystemExit as se: + return_code = se.code + self.loop.stop() + return except BaseException: - if repl_future_interrupted: - self.write("\nKeyboardInterrupt\n") + if keyboard_interrupted: + if not CAN_USE_PYREPL: + self.write("\nKeyboardInterrupt\n") else: self.showtraceback() - + return self.STATEMENT_FAILED class REPLThread(threading.Thread): def run(self): + global return_code + try: banner = ( f'asyncio REPL {sys.version} on {sys.platform}\n' f'Use "await" directly instead of "asyncio.run()".\n' f'Type "help", "copyright", "credits" or "license" ' f'for more information.\n' - f'{getattr(sys, "ps1", ">>> ")}import asyncio' ) - console.interact( - banner=banner, - exitmsg='exiting asyncio REPL...') + console.write(banner) + + if startup_path := os.getenv("PYTHONSTARTUP"): + sys.audit("cpython.run_startup", startup_path) + + import tokenize + with tokenize.open(startup_path) as f: + startup_code = compile(f.read(), startup_path, "exec") + exec(startup_code, console.locals) + + ps1 = getattr(sys, "ps1", ">>> ") + if CAN_USE_PYREPL: + theme = get_theme().syntax + ps1 = f"{theme.prompt}{ps1}{theme.reset}" + console.write(f"{ps1}import asyncio\n") + + if CAN_USE_PYREPL: + from _pyrepl.simple_interact import ( + run_multiline_interactive_console, + ) + try: + run_multiline_interactive_console(console) + except SystemExit: + # expected via the `exit` and `quit` commands + pass + except BaseException: + # unexpected issue + console.showtraceback() + console.write("Internal error, ") + return_code = 1 + else: + console.interact(banner="", exitmsg="") finally: warnings.filterwarnings( 'ignore', @@ -88,8 +133,56 @@ def run(self): loop.call_soon_threadsafe(loop.stop) + def interrupt(self) -> None: + if not CAN_USE_PYREPL: + return + + from _pyrepl.simple_interact import _get_reader + r = _get_reader() + if r.threading_hook is not None: + r.threading_hook.add("") # type: ignore + if __name__ == '__main__': + parser = argparse.ArgumentParser( + prog="python3 -m asyncio", + description="Interactive asyncio shell and CLI tools", + color=True, + ) + subparsers = parser.add_subparsers(help="sub-commands", dest="command") + ps = subparsers.add_parser( + "ps", help="Display a table of all pending tasks in a process" + ) + ps.add_argument("pid", type=int, help="Process ID to inspect") + pstree = subparsers.add_parser( + "pstree", help="Display a tree of all pending tasks in a process" + ) + pstree.add_argument("pid", type=int, help="Process ID to inspect") + args = parser.parse_args() + match args.command: + case "ps": + asyncio.tools.display_awaited_by_tasks_table(args.pid) + sys.exit(0) + case "pstree": + asyncio.tools.display_awaited_by_tasks_tree(args.pid) + sys.exit(0) + case None: + pass # continue to the interactive shell + case _: + # shouldn't happen as an invalid command-line wouldn't parse + # but let's keep it for the next person adding a command + print(f"error: unhandled command {args.command}", file=sys.stderr) + parser.print_usage(file=sys.stderr) + sys.exit(1) + + sys.audit("cpython.run_stdin") + + if os.getenv('PYTHON_BASIC_REPL'): + CAN_USE_PYREPL = False + else: + from _pyrepl.main import CAN_USE_PYREPL + + return_code = 0 loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) @@ -102,14 +195,31 @@ def run(self): console = AsyncIOInteractiveConsole(repl_locals, loop) repl_future = None - repl_future_interrupted = False + keyboard_interrupted = False try: import readline # NoQA except ImportError: - pass + readline = None - repl_thread = REPLThread() + interactive_hook = getattr(sys, "__interactivehook__", None) + + if interactive_hook is not None: + sys.audit("cpython.run_interactivehook", interactive_hook) + interactive_hook() + + if interactive_hook is site.register_readline: + # Fix the completer function to use the interactive console locals + try: + import rlcompleter + except: + pass + else: + if readline is not None: + completer = rlcompleter.Completer(console.locals) + readline.set_completer(completer.complete) + + repl_thread = REPLThread(name="Interactive thread") repl_thread.daemon = True repl_thread.start() @@ -117,9 +227,13 @@ def run(self): try: loop.run_forever() except KeyboardInterrupt: + keyboard_interrupted = True if repl_future and not repl_future.done(): repl_future.cancel() - repl_future_interrupted = True + repl_thread.interrupt() continue else: break + + console.write('exiting asyncio REPL...\n') + sys.exit(return_code) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py index 29eff0499cb..8cbb71f7085 100644 --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -17,7 +17,6 @@ import collections.abc import concurrent.futures import errno -import functools import heapq import itertools import os @@ -279,7 +278,9 @@ def __init__(self, loop, sockets, protocol_factory, ssl_context, backlog, ssl_handshake_timeout, ssl_shutdown_timeout=None): self._loop = loop self._sockets = sockets - self._active_count = 0 + # Weak references so we don't break Transport's ability to + # detect abandoned transports + self._clients = weakref.WeakSet() self._waiters = [] self._protocol_factory = protocol_factory self._backlog = backlog @@ -292,14 +293,13 @@ def __init__(self, loop, sockets, protocol_factory, ssl_context, backlog, def __repr__(self): return f'<{self.__class__.__name__} sockets={self.sockets!r}>' - def _attach(self): + def _attach(self, transport): assert self._sockets is not None - self._active_count += 1 + self._clients.add(transport) - def _detach(self): - assert self._active_count > 0 - self._active_count -= 1 - if self._active_count == 0 and self._sockets is None: + def _detach(self, transport): + self._clients.discard(transport) + if len(self._clients) == 0 and self._sockets is None: self._wakeup() def _wakeup(self): @@ -348,9 +348,17 @@ def close(self): self._serving_forever_fut.cancel() self._serving_forever_fut = None - if self._active_count == 0: + if len(self._clients) == 0: self._wakeup() + def close_clients(self): + for transport in self._clients.copy(): + transport.close() + + def abort_clients(self): + for transport in self._clients.copy(): + transport.abort() + async def start_serving(self): self._start_serving() # Skip one loop iteration so that all 'loop.add_reader' @@ -422,6 +430,8 @@ def __init__(self): self._clock_resolution = time.get_clock_info('monotonic').resolution self._exception_handler = None self.set_debug(coroutines._is_debug_mode()) + # The preserved state of async generator hooks. + self._old_agen_hooks = None # In debug mode, if the execution of a callback or a step of a task # exceed this duration in seconds, the slow callback/task is logged. self.slow_callback_duration = 0.1 @@ -448,26 +458,24 @@ def create_future(self): """Create a Future object attached to the loop.""" return futures.Future(loop=self) - def create_task(self, coro, *, name=None, context=None): - """Schedule a coroutine object. + def create_task(self, coro, **kwargs): + """Schedule or begin executing a coroutine object. Return a task object. """ self._check_closed() - if self._task_factory is None: - task = tasks.Task(coro, loop=self, name=name, context=context) - if task._source_traceback: - del task._source_traceback[-1] - else: - if context is None: - # Use legacy API if context is not needed - task = self._task_factory(self, coro) - else: - task = self._task_factory(self, coro, context=context) - - tasks._set_task_name(task, name) + if self._task_factory is not None: + return self._task_factory(self, coro, **kwargs) - return task + task = tasks.Task(coro, loop=self, **kwargs) + if task._source_traceback: + del task._source_traceback[-1] + try: + return task + finally: + # gh-128552: prevent a refcycle of + # task.exception().__traceback__->BaseEventLoop.create_task->task + del task def set_task_factory(self, factory): """Set a task factory that will be used by loop.create_task(). @@ -475,9 +483,10 @@ def set_task_factory(self, factory): If factory is None the default task factory will be set. If factory is a callable, it should have a signature matching - '(loop, coro)', where 'loop' will be a reference to the active - event loop, 'coro' will be a coroutine object. The callable - must return a Future. + '(loop, coro, **kwargs)', where 'loop' will be a reference to the active + event loop, 'coro' will be a coroutine object, and **kwargs will be + arbitrary keyword arguments that should be passed on to Task. + The callable must return a Task. """ if factory is not None and not callable(factory): raise TypeError('task factory must be a callable or None') @@ -624,29 +633,52 @@ def _check_running(self): raise RuntimeError( 'Cannot run the event loop while another loop is running') - def run_forever(self): - """Run until stop() is called.""" + def _run_forever_setup(self): + """Prepare the run loop to process events. + + This method exists so that custom event loop subclasses (e.g., event loops + that integrate a GUI event loop with Python's event loop) have access to all the + loop setup logic. + """ self._check_closed() self._check_running() self._set_coroutine_origin_tracking(self._debug) - old_agen_hooks = sys.get_asyncgen_hooks() - try: - self._thread_id = threading.get_ident() - sys.set_asyncgen_hooks(firstiter=self._asyncgen_firstiter_hook, - finalizer=self._asyncgen_finalizer_hook) + self._old_agen_hooks = sys.get_asyncgen_hooks() + self._thread_id = threading.get_ident() + sys.set_asyncgen_hooks( + firstiter=self._asyncgen_firstiter_hook, + finalizer=self._asyncgen_finalizer_hook + ) + + events._set_running_loop(self) + + def _run_forever_cleanup(self): + """Clean up after an event loop finishes the looping over events. + + This method exists so that custom event loop subclasses (e.g., event loops + that integrate a GUI event loop with Python's event loop) have access to all the + loop cleanup logic. + """ + self._stopping = False + self._thread_id = None + events._set_running_loop(None) + self._set_coroutine_origin_tracking(False) + # Restore any pre-existing async generator hooks. + if self._old_agen_hooks is not None: + sys.set_asyncgen_hooks(*self._old_agen_hooks) + self._old_agen_hooks = None - events._set_running_loop(self) + def run_forever(self): + """Run until stop() is called.""" + self._run_forever_setup() + try: while True: self._run_once() if self._stopping: break finally: - self._stopping = False - self._thread_id = None - events._set_running_loop(None) - self._set_coroutine_origin_tracking(False) - sys.set_asyncgen_hooks(*old_agen_hooks) + self._run_forever_cleanup() def run_until_complete(self, future): """Run until the Future is done. @@ -803,7 +835,7 @@ def call_soon(self, callback, *args, context=None): def _check_callback(self, callback, method): if (coroutines.iscoroutine(callback) or - coroutines.iscoroutinefunction(callback)): + coroutines._iscoroutinefunction(callback)): raise TypeError( f"coroutines cannot be used with {method}()") if not callable(callback): @@ -840,7 +872,10 @@ def call_soon_threadsafe(self, callback, *args, context=None): self._check_closed() if self._debug: self._check_callback(callback, 'call_soon_threadsafe') - handle = self._call_soon(callback, args, context) + handle = events._ThreadSafeHandle(callback, args, self, context) + self._ready.append(handle) + if handle._source_traceback: + del handle._source_traceback[-1] if handle._source_traceback: del handle._source_traceback[-1] self._write_to_self() @@ -981,39 +1016,43 @@ async def _connect_sock(self, exceptions, addr_info, local_addr_infos=None): family, type_, proto, _, address = addr_info sock = None try: - sock = socket.socket(family=family, type=type_, proto=proto) - sock.setblocking(False) - if local_addr_infos is not None: - for lfamily, _, _, _, laddr in local_addr_infos: - # skip local addresses of different family - if lfamily != family: - continue - try: - sock.bind(laddr) - break - except OSError as exc: - msg = ( - f'error while attempting to bind on ' - f'address {laddr!r}: ' - f'{exc.strerror.lower()}' - ) - exc = OSError(exc.errno, msg) - my_exceptions.append(exc) - else: # all bind attempts failed - if my_exceptions: - raise my_exceptions.pop() - else: - raise OSError(f"no matching local address with {family=} found") - await self.sock_connect(sock, address) - return sock - except OSError as exc: - my_exceptions.append(exc) - if sock is not None: - sock.close() - raise + try: + sock = socket.socket(family=family, type=type_, proto=proto) + sock.setblocking(False) + if local_addr_infos is not None: + for lfamily, _, _, _, laddr in local_addr_infos: + # skip local addresses of different family + if lfamily != family: + continue + try: + sock.bind(laddr) + break + except OSError as exc: + msg = ( + f'error while attempting to bind on ' + f'address {laddr!r}: {str(exc).lower()}' + ) + exc = OSError(exc.errno, msg) + my_exceptions.append(exc) + else: # all bind attempts failed + if my_exceptions: + raise my_exceptions.pop() + else: + raise OSError(f"no matching local address with {family=} found") + await self.sock_connect(sock, address) + return sock + except OSError as exc: + my_exceptions.append(exc) + raise except: if sock is not None: - sock.close() + try: + sock.close() + except OSError: + # An error when closing a newly created socket is + # not important, but it can overwrite more important + # non-OSError error. So ignore it. + pass raise finally: exceptions = my_exceptions = None @@ -1107,11 +1146,18 @@ async def create_connection( except OSError: continue else: # using happy eyeballs - sock, _, _ = await staggered.staggered_race( - (functools.partial(self._connect_sock, - exceptions, addrinfo, laddr_infos) - for addrinfo in infos), - happy_eyeballs_delay, loop=self) + sock = (await staggered.staggered_race( + ( + # can't use functools.partial as it keeps a reference + # to exceptions + lambda addrinfo=addrinfo: self._connect_sock( + exceptions, addrinfo, laddr_infos + ) + for addrinfo in infos + ), + happy_eyeballs_delay, + loop=self, + ))[0] # can't use sock, _, _ as it keeks a reference to exceptions if sock is None: exceptions = [exc for sub in exceptions for exc in sub] @@ -1120,7 +1166,7 @@ async def create_connection( raise ExceptionGroup("create_connection failed", exceptions) if len(exceptions) == 1: raise exceptions[0] - else: + elif exceptions: # If they all have the same str(), raise one. model = str(exceptions[0]) if all(str(exc) == model for exc in exceptions): @@ -1129,6 +1175,9 @@ async def create_connection( # the various error messages. raise OSError('Multiple exceptions: {}'.format( ', '.join(str(exc) for exc in exceptions))) + else: + # No exceptions were collected, raise a timeout error + raise TimeoutError('create_connection failed') finally: exceptions = None @@ -1254,8 +1303,8 @@ async def _sendfile_fallback(self, transp, file, offset, count): read = await self.run_in_executor(None, file.readinto, view) if not read: return total_sent # EOF - await proto.drain() transp.write(view[:read]) + await proto.drain() total_sent += read finally: if total_sent > 0 and hasattr(file, 'seek'): @@ -1474,6 +1523,7 @@ async def create_server( ssl=None, reuse_address=None, reuse_port=None, + keep_alive=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None, start_serving=True): @@ -1545,8 +1595,13 @@ async def create_server( if reuse_address: sock.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, True) - if reuse_port: + # Since Linux 6.12.9, SO_REUSEPORT is not allowed + # on other address families than AF_INET/AF_INET6. + if reuse_port and af in (socket.AF_INET, socket.AF_INET6): _set_reuseport(sock) + if keep_alive: + sock.setsockopt( + socket.SOL_SOCKET, socket.SO_KEEPALIVE, True) # Disable IPv4/IPv6 dual stack support (enabled by # default on Linux) which makes a single socket # listen on both address families. @@ -1561,7 +1616,7 @@ async def create_server( except OSError as err: msg = ('error while attempting ' 'to bind on address %r: %s' - % (sa, err.strerror.lower())) + % (sa, str(err).lower())) if err.errno == errno.EADDRNOTAVAIL: # Assume the family is not enabled (bpo-30945) sockets.pop() @@ -1619,8 +1674,7 @@ async def connect_accepted_socket( raise ValueError( 'ssl_shutdown_timeout is only meaningful with ssl') - if sock is not None: - _check_ssl_socket(sock) + _check_ssl_socket(sock) transport, protocol = await self._create_connection_transport( sock, protocol_factory, ssl, '', server_side=True, @@ -1833,6 +1887,8 @@ def call_exception_handler(self, context): - 'protocol' (optional): Protocol instance; - 'transport' (optional): Transport instance; - 'socket' (optional): Socket instance; + - 'source_traceback' (optional): Traceback of the source; + - 'handle_traceback' (optional): Traceback of the handle; - 'asyncgen' (optional): Asynchronous generator that caused the exception. @@ -1943,8 +1999,11 @@ def _run_once(self): timeout = 0 elif self._scheduled: # Compute the desired timeout. - when = self._scheduled[0]._when - timeout = min(max(0, when - self.time()), MAXIMUM_SELECT_TIMEOUT) + timeout = self._scheduled[0]._when - self.time() + if timeout > MAXIMUM_SELECT_TIMEOUT: + timeout = MAXIMUM_SELECT_TIMEOUT + elif timeout < 0: + timeout = 0 event_list = self._selector.select(timeout) self._process_events(event_list) diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py index 4c9b0dd5653..321a4e5d5d1 100644 --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -1,6 +1,9 @@ import collections import subprocess import warnings +import os +import signal +import sys from . import protocols from . import transports @@ -23,6 +26,7 @@ def __init__(self, loop, protocol, args, shell, self._pending_calls = collections.deque() self._pipes = {} self._finished = False + self._pipes_connected = False if stdin == subprocess.PIPE: self._pipes[0] = None @@ -101,7 +105,12 @@ def close(self): for proto in self._pipes.values(): if proto is None: continue - proto.pipe.close() + # See gh-114177 + # skip closing the pipe if loop is already closed + # this can happen e.g. when loop is closed immediately after + # process is killed + if self._loop and not self._loop.is_closed(): + proto.pipe.close() if (self._proc is not None and # has the child process finished? @@ -115,7 +124,8 @@ def close(self): try: self._proc.kill() - except ProcessLookupError: + except (ProcessLookupError, PermissionError): + # the process may have already exited or may be running setuid pass # Don't clear the _proc reference yet: _post_init() may still run @@ -141,17 +151,31 @@ def _check_proc(self): if self._proc is None: raise ProcessLookupError() - def send_signal(self, signal): - self._check_proc() - self._proc.send_signal(signal) + if sys.platform == 'win32': + def send_signal(self, signal): + self._check_proc() + self._proc.send_signal(signal) + + def terminate(self): + self._check_proc() + self._proc.terminate() + + def kill(self): + self._check_proc() + self._proc.kill() + else: + def send_signal(self, signal): + self._check_proc() + try: + os.kill(self._proc.pid, signal) + except ProcessLookupError: + pass - def terminate(self): - self._check_proc() - self._proc.terminate() + def terminate(self): + self.send_signal(signal.SIGTERM) - def kill(self): - self._check_proc() - self._proc.kill() + def kill(self): + self.send_signal(signal.SIGKILL) async def _connect_pipes(self, waiter): try: @@ -190,6 +214,7 @@ async def _connect_pipes(self, waiter): else: if waiter is not None and not waiter.cancelled(): waiter.set_result(None) + self._pipes_connected = True def _call(self, cb, *data): if self._pending_calls is not None: @@ -233,6 +258,15 @@ def _try_finish(self): assert not self._finished if self._returncode is None: return + if not self._pipes_connected: + # self._pipes_connected can be False if not all pipes were connected + # because either the process failed to start or the self._connect_pipes task + # got cancelled. In this broken state we consider all pipes disconnected and + # to avoid hanging forever in self._wait as otherwise _exit_waiters + # would never be woken up, we wake them up here. + for waiter in self._exit_waiters: + if not waiter.cancelled(): + waiter.set_result(self._returncode) if all(p is not None and p.disconnected for p in self._pipes.values()): self._finished = True diff --git a/Lib/asyncio/coroutines.py b/Lib/asyncio/coroutines.py index ab4f30eb51b..a51319cb72a 100644 --- a/Lib/asyncio/coroutines.py +++ b/Lib/asyncio/coroutines.py @@ -18,7 +18,16 @@ def _is_debug_mode(): def iscoroutinefunction(func): + import warnings """Return True if func is a decorated coroutine function.""" + warnings._deprecated("asyncio.iscoroutinefunction", + f"{warnings._DEPRECATED_MSG}; " + "use inspect.iscoroutinefunction() instead", + remove=(3,16)) + return _iscoroutinefunction(func) + + +def _iscoroutinefunction(func): return (inspect.iscoroutinefunction(func) or getattr(func, '_is_coroutine', None) is _is_coroutine) diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py index 016852880ca..a7fb55982ab 100644 --- a/Lib/asyncio/events.py +++ b/Lib/asyncio/events.py @@ -5,14 +5,18 @@ # SPDX-FileCopyrightText: Copyright (c) 2015-2021 MagicStack Inc. http://magic.io __all__ = ( - 'AbstractEventLoopPolicy', - 'AbstractEventLoop', 'AbstractServer', - 'Handle', 'TimerHandle', - 'get_event_loop_policy', 'set_event_loop_policy', - 'get_event_loop', 'set_event_loop', 'new_event_loop', - 'get_child_watcher', 'set_child_watcher', - '_set_running_loop', 'get_running_loop', - '_get_running_loop', + "AbstractEventLoop", + "AbstractServer", + "Handle", + "TimerHandle", + "get_event_loop_policy", + "set_event_loop_policy", + "get_event_loop", + "set_event_loop", + "new_event_loop", + "_set_running_loop", + "get_running_loop", + "_get_running_loop", ) import contextvars @@ -22,6 +26,7 @@ import subprocess import sys import threading +import warnings from . import format_helpers @@ -54,7 +59,8 @@ def _repr_info(self): info.append('cancelled') if self._callback is not None: info.append(format_helpers._format_callback_source( - self._callback, self._args)) + self._callback, self._args, + debug=self._loop.get_debug())) if self._source_traceback: frame = self._source_traceback[-1] info.append(f'created at {frame[0]}:{frame[1]}') @@ -90,7 +96,8 @@ def _run(self): raise except BaseException as exc: cb = format_helpers._format_callback_source( - self._callback, self._args) + self._callback, self._args, + debug=self._loop.get_debug()) msg = f'Exception in callback {cb}' context = { 'message': msg, @@ -102,6 +109,34 @@ def _run(self): self._loop.call_exception_handler(context) self = None # Needed to break cycles when an exception occurs. +# _ThreadSafeHandle is used for callbacks scheduled with call_soon_threadsafe +# and is thread safe unlike Handle which is not thread safe. +class _ThreadSafeHandle(Handle): + + __slots__ = ('_lock',) + + def __init__(self, callback, args, loop, context=None): + super().__init__(callback, args, loop, context) + self._lock = threading.RLock() + + def cancel(self): + with self._lock: + return super().cancel() + + def cancelled(self): + with self._lock: + return super().cancelled() + + def _run(self): + # The event loop checks for cancellation without holding the lock + # It is possible that the handle is cancelled after the check + # but before the callback is called so check it again after acquiring + # the lock and return without calling the callback if it is cancelled. + with self._lock: + if self._cancelled: + return + return super()._run() + class TimerHandle(Handle): """Object returned by timed callback registration methods.""" @@ -173,6 +208,14 @@ def close(self): """Stop serving. This leaves existing connections open.""" raise NotImplementedError + def close_clients(self): + """Close all active connections.""" + raise NotImplementedError + + def abort_clients(self): + """Close all active connections immediately.""" + raise NotImplementedError + def get_loop(self): """Get the event loop the Server object is attached to.""" raise NotImplementedError @@ -282,7 +325,7 @@ def create_future(self): # Method scheduling a coroutine object: create a task. - def create_task(self, coro, *, name=None, context=None): + def create_task(self, coro, **kwargs): raise NotImplementedError # Methods for interacting with threads. @@ -320,6 +363,7 @@ async def create_server( *, family=socket.AF_UNSPEC, flags=socket.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None, reuse_port=None, + keep_alive=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None, start_serving=True): @@ -358,6 +402,9 @@ async def create_server( they all set this flag when being created. This option is not supported on Windows. + keep_alive set to True keeps connections active by enabling the + periodic transmission of messages. + ssl_handshake_timeout is the time in seconds that an SSL server will wait for completion of the SSL handshake before aborting the connection. Default is 60s. @@ -615,7 +662,7 @@ def set_debug(self, enabled): raise NotImplementedError -class AbstractEventLoopPolicy: +class _AbstractEventLoopPolicy: """Abstract policy for accessing the event loop.""" def get_event_loop(self): @@ -638,18 +685,7 @@ def new_event_loop(self): the current context, set_event_loop must be called explicitly.""" raise NotImplementedError - # Child processes handling (Unix only). - - def get_child_watcher(self): - "Get the watcher for child processes." - raise NotImplementedError - - def set_child_watcher(self, watcher): - """Set the watcher for child processes.""" - raise NotImplementedError - - -class BaseDefaultEventLoopPolicy(AbstractEventLoopPolicy): +class _BaseDefaultEventLoopPolicy(_AbstractEventLoopPolicy): """Default policy implementation for accessing the event loop. In this policy, each thread has its own event loop. However, we @@ -666,7 +702,6 @@ class BaseDefaultEventLoopPolicy(AbstractEventLoopPolicy): class _Local(threading.local): _loop = None - _set_called = False def __init__(self): self._local = self._Local() @@ -676,28 +711,6 @@ def get_event_loop(self): Returns an instance of EventLoop or raises an exception. """ - if (self._local._loop is None and - not self._local._set_called and - threading.current_thread() is threading.main_thread()): - stacklevel = 2 - try: - f = sys._getframe(1) - except AttributeError: - pass - else: - # Move up the call stack so that the warning is attached - # to the line outside asyncio itself. - while f: - module = f.f_globals.get('__name__') - if not (module == 'asyncio' or module.startswith('asyncio.')): - break - f = f.f_back - stacklevel += 1 - import warnings - warnings.warn('There is no current event loop', - DeprecationWarning, stacklevel=stacklevel) - self.set_event_loop(self.new_event_loop()) - if self._local._loop is None: raise RuntimeError('There is no current event loop in thread %r.' % threading.current_thread().name) @@ -706,7 +719,6 @@ def get_event_loop(self): def set_event_loop(self, loop): """Set the event loop.""" - self._local._set_called = True if loop is not None and not isinstance(loop, AbstractEventLoop): raise TypeError(f"loop must be an instance of AbstractEventLoop or None, not '{type(loop).__name__}'") self._local._loop = loop @@ -776,26 +788,35 @@ def _init_event_loop_policy(): global _event_loop_policy with _lock: if _event_loop_policy is None: # pragma: no branch - from . import DefaultEventLoopPolicy - _event_loop_policy = DefaultEventLoopPolicy() + if sys.platform == 'win32': + from .windows_events import _DefaultEventLoopPolicy + else: + from .unix_events import _DefaultEventLoopPolicy + _event_loop_policy = _DefaultEventLoopPolicy() -def get_event_loop_policy(): +def _get_event_loop_policy(): """Get the current event loop policy.""" if _event_loop_policy is None: _init_event_loop_policy() return _event_loop_policy +def get_event_loop_policy(): + warnings._deprecated('asyncio.get_event_loop_policy', remove=(3, 16)) + return _get_event_loop_policy() -def set_event_loop_policy(policy): +def _set_event_loop_policy(policy): """Set the current event loop policy. If policy is None, the default policy is restored.""" global _event_loop_policy - if policy is not None and not isinstance(policy, AbstractEventLoopPolicy): + if policy is not None and not isinstance(policy, _AbstractEventLoopPolicy): raise TypeError(f"policy must be an instance of AbstractEventLoopPolicy or None, not '{type(policy).__name__}'") _event_loop_policy = policy +def set_event_loop_policy(policy): + warnings._deprecated('asyncio.set_event_loop_policy', remove=(3,16)) + _set_event_loop_policy(policy) def get_event_loop(): """Return an asyncio event loop. @@ -810,28 +831,17 @@ def get_event_loop(): current_loop = _get_running_loop() if current_loop is not None: return current_loop - return get_event_loop_policy().get_event_loop() + return _get_event_loop_policy().get_event_loop() def set_event_loop(loop): """Equivalent to calling get_event_loop_policy().set_event_loop(loop).""" - get_event_loop_policy().set_event_loop(loop) + _get_event_loop_policy().set_event_loop(loop) def new_event_loop(): """Equivalent to calling get_event_loop_policy().new_event_loop().""" - return get_event_loop_policy().new_event_loop() - - -def get_child_watcher(): - """Equivalent to calling get_event_loop_policy().get_child_watcher().""" - return get_event_loop_policy().get_child_watcher() - - -def set_child_watcher(watcher): - """Equivalent to calling - get_event_loop_policy().set_child_watcher(watcher).""" - return get_event_loop_policy().set_child_watcher(watcher) + return _get_event_loop_policy().new_event_loop() # Alias pure-Python implementations for testing purposes. @@ -861,7 +871,7 @@ def set_child_watcher(watcher): def on_fork(): # Reset the loop and wakeupfd in the forked child process. if _event_loop_policy is not None: - _event_loop_policy._local = BaseDefaultEventLoopPolicy._Local() + _event_loop_policy._local = _BaseDefaultEventLoopPolicy._Local() _set_running_loop(None) signal.set_wakeup_fd(-1) diff --git a/Lib/asyncio/format_helpers.py b/Lib/asyncio/format_helpers.py index 27d11fd4fa9..93737b7708a 100644 --- a/Lib/asyncio/format_helpers.py +++ b/Lib/asyncio/format_helpers.py @@ -19,19 +19,26 @@ def _get_function_source(func): return None -def _format_callback_source(func, args): - func_repr = _format_callback(func, args, None) +def _format_callback_source(func, args, *, debug=False): + func_repr = _format_callback(func, args, None, debug=debug) source = _get_function_source(func) if source: func_repr += f' at {source[0]}:{source[1]}' return func_repr -def _format_args_and_kwargs(args, kwargs): +def _format_args_and_kwargs(args, kwargs, *, debug=False): """Format function arguments and keyword arguments. Special case for a single parameter: ('hello',) is formatted as ('hello'). + + Note that this function only returns argument details when + debug=True is specified, as arguments may contain sensitive + information. """ + if not debug: + return '()' + # use reprlib to limit the length of the output items = [] if args: @@ -41,10 +48,11 @@ def _format_args_and_kwargs(args, kwargs): return '({})'.format(', '.join(items)) -def _format_callback(func, args, kwargs, suffix=''): +def _format_callback(func, args, kwargs, *, debug=False, suffix=''): if isinstance(func, functools.partial): - suffix = _format_args_and_kwargs(args, kwargs) + suffix - return _format_callback(func.func, func.args, func.keywords, suffix) + suffix = _format_args_and_kwargs(args, kwargs, debug=debug) + suffix + return _format_callback(func.func, func.args, func.keywords, + debug=debug, suffix=suffix) if hasattr(func, '__qualname__') and func.__qualname__: func_repr = func.__qualname__ @@ -53,7 +61,7 @@ def _format_callback(func, args, kwargs, suffix=''): else: func_repr = repr(func) - func_repr += _format_args_and_kwargs(args, kwargs) + func_repr += _format_args_and_kwargs(args, kwargs, debug=debug) if suffix: func_repr += suffix return func_repr diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py index 97fc4e3fcb6..d1df6707302 100644 --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -2,6 +2,7 @@ __all__ = ( 'Future', 'wrap_future', 'isfuture', + 'future_add_to_awaited_by', 'future_discard_from_awaited_by', ) import concurrent.futures @@ -43,7 +44,6 @@ class Future: - This class is not compatible with the wait() and as_completed() methods in the concurrent.futures package. - (In Python 3.4 or later we may be able to unify the implementations.) """ # Class variables serving as defaults for instance variables. @@ -61,12 +61,15 @@ class Future: # the Future protocol (i.e. is intended to be duck-type compatible). # The value must also be not-None, to enable a subclass to declare # that it is not compatible by setting this to None. - # - It is set by __iter__() below so that Task._step() can tell + # - It is set by __iter__() below so that Task.__step() can tell # the difference between - # `await Future()` or`yield from Future()` (correct) vs. + # `await Future()` or `yield from Future()` (correct) vs. # `yield Future()` (incorrect). _asyncio_future_blocking = False + # Used by the capture_call_stack() API. + __asyncio_awaited_by = None + __log_traceback = False def __init__(self, *, loop=None): @@ -116,6 +119,12 @@ def _log_traceback(self, val): raise ValueError('_log_traceback can only be set to False') self.__log_traceback = False + @property + def _asyncio_awaited_by(self): + if self.__asyncio_awaited_by is None: + return None + return frozenset(self.__asyncio_awaited_by) + def get_loop(self): """Return the event loop the Future is bound to.""" loop = self._loop @@ -138,9 +147,6 @@ def _make_cancelled_error(self): exc = exceptions.CancelledError() else: exc = exceptions.CancelledError(self._cancel_message) - exc.__context__ = self._cancelled_exc - # Remove the reference since we don't need this anymore. - self._cancelled_exc = None return exc def cancel(self, msg=None): @@ -194,8 +200,7 @@ def result(self): the future is done and has an exception set, this exception is raised. """ if self._state == _CANCELLED: - exc = self._make_cancelled_error() - raise exc + raise self._make_cancelled_error() if self._state != _FINISHED: raise exceptions.InvalidStateError('Result is not ready.') self.__log_traceback = False @@ -212,8 +217,7 @@ def exception(self): InvalidStateError. """ if self._state == _CANCELLED: - exc = self._make_cancelled_error() - raise exc + raise self._make_cancelled_error() if self._state != _FINISHED: raise exceptions.InvalidStateError('Exception is not set.') self.__log_traceback = False @@ -272,9 +276,13 @@ def set_exception(self, exception): raise exceptions.InvalidStateError(f'{self._state}: {self!r}') if isinstance(exception, type): exception = exception() - if type(exception) is StopIteration: - raise TypeError("StopIteration interacts badly with generators " - "and cannot be raised into a Future") + if isinstance(exception, StopIteration): + new_exc = RuntimeError("StopIteration interacts badly with " + "generators and cannot be raised into a " + "Future") + new_exc.__cause__ = exception + new_exc.__context__ = exception + exception = new_exc self._exception = exception self._exception_tb = exception.__traceback__ self._state = _FINISHED @@ -318,11 +326,9 @@ def _set_result_unless_cancelled(fut, result): def _convert_future_exc(exc): exc_class = type(exc) if exc_class is concurrent.futures.CancelledError: - return exceptions.CancelledError(*exc.args) - elif exc_class is concurrent.futures.TimeoutError: - return exceptions.TimeoutError(*exc.args) + return exceptions.CancelledError(*exc.args).with_traceback(exc.__traceback__) elif exc_class is concurrent.futures.InvalidStateError: - return exceptions.InvalidStateError(*exc.args) + return exceptions.InvalidStateError(*exc.args).with_traceback(exc.__traceback__) else: return exc @@ -419,6 +425,49 @@ def wrap_future(future, *, loop=None): return new_future +def future_add_to_awaited_by(fut, waiter, /): + """Record that `fut` is awaited on by `waiter`.""" + # For the sake of keeping the implementation minimal and assuming + # that most of asyncio users use the built-in Futures and Tasks + # (or their subclasses), we only support native Future objects + # and their subclasses. + # + # Longer version: tracking requires storing the caller-callee + # dependency somewhere. One obvious choice is to store that + # information right in the future itself in a dedicated attribute. + # This means that we'd have to require all duck-type compatible + # futures to implement a specific attribute used by asyncio for + # the book keeping. Another solution would be to store that in + # a global dictionary. The downside here is that that would create + # strong references and any scenario where the "add" call isn't + # followed by a "discard" call would lead to a memory leak. + # Using WeakDict would resolve that issue, but would complicate + # the C code (_asynciomodule.c). The bottom line here is that + # it's not clear that all this work would be worth the effort. + # + # Note that there's an accelerated version of this function + # shadowing this implementation later in this file. + if isinstance(fut, _PyFuture) and isinstance(waiter, _PyFuture): + if fut._Future__asyncio_awaited_by is None: + fut._Future__asyncio_awaited_by = set() + fut._Future__asyncio_awaited_by.add(waiter) + + +def future_discard_from_awaited_by(fut, waiter, /): + """Record that `fut` is no longer awaited on by `waiter`.""" + # See the comment in "future_add_to_awaited_by()" body for + # details on implementation. + # + # Note that there's an accelerated version of this function + # shadowing this implementation later in this file. + if isinstance(fut, _PyFuture) and isinstance(waiter, _PyFuture): + if fut._Future__asyncio_awaited_by is not None: + fut._Future__asyncio_awaited_by.discard(waiter) + + +_py_future_add_to_awaited_by = future_add_to_awaited_by +_py_future_discard_from_awaited_by = future_discard_from_awaited_by + try: import _asyncio except ImportError: @@ -426,3 +475,7 @@ def wrap_future(future, *, loop=None): else: # _CFuture is needed for tests. Future = _CFuture = _asyncio.Future + future_add_to_awaited_by = _asyncio.future_add_to_awaited_by + future_discard_from_awaited_by = _asyncio.future_discard_from_awaited_by + _c_future_add_to_awaited_by = future_add_to_awaited_by + _c_future_discard_from_awaited_by = future_discard_from_awaited_by diff --git a/Lib/asyncio/graph.py b/Lib/asyncio/graph.py new file mode 100644 index 00000000000..b5bfeb1630a --- /dev/null +++ b/Lib/asyncio/graph.py @@ -0,0 +1,276 @@ +"""Introspection utils for tasks call graphs.""" + +import dataclasses +import io +import sys +import types + +from . import events +from . import futures +from . import tasks + +__all__ = ( + 'capture_call_graph', + 'format_call_graph', + 'print_call_graph', + 'FrameCallGraphEntry', + 'FutureCallGraph', +) + +# Sadly, we can't re-use the traceback module's datastructures as those +# are tailored for error reporting, whereas we need to represent an +# async call graph. +# +# Going with pretty verbose names as we'd like to export them to the +# top level asyncio namespace, and want to avoid future name clashes. + + +@dataclasses.dataclass(frozen=True, slots=True) +class FrameCallGraphEntry: + frame: types.FrameType + + +@dataclasses.dataclass(frozen=True, slots=True) +class FutureCallGraph: + future: futures.Future + call_stack: tuple["FrameCallGraphEntry", ...] + awaited_by: tuple["FutureCallGraph", ...] + + +def _build_graph_for_future( + future: futures.Future, + *, + limit: int | None = None, +) -> FutureCallGraph: + if not isinstance(future, futures.Future): + raise TypeError( + f"{future!r} object does not appear to be compatible " + f"with asyncio.Future" + ) + + coro = None + if get_coro := getattr(future, 'get_coro', None): + coro = get_coro() if limit != 0 else None + + st: list[FrameCallGraphEntry] = [] + awaited_by: list[FutureCallGraph] = [] + + while coro is not None: + if hasattr(coro, 'cr_await'): + # A native coroutine or duck-type compatible iterator + st.append(FrameCallGraphEntry(coro.cr_frame)) + coro = coro.cr_await + elif hasattr(coro, 'ag_await'): + # A native async generator or duck-type compatible iterator + st.append(FrameCallGraphEntry(coro.cr_frame)) + coro = coro.ag_await + else: + break + + if future._asyncio_awaited_by: + for parent in future._asyncio_awaited_by: + awaited_by.append(_build_graph_for_future(parent, limit=limit)) + + if limit is not None: + if limit > 0: + st = st[:limit] + elif limit < 0: + st = st[limit:] + st.reverse() + return FutureCallGraph(future, tuple(st), tuple(awaited_by)) + + +def capture_call_graph( + future: futures.Future | None = None, + /, + *, + depth: int = 1, + limit: int | None = None, +) -> FutureCallGraph | None: + """Capture the async call graph for the current task or the provided Future. + + The graph is represented with three data structures: + + * FutureCallGraph(future, call_stack, awaited_by) + + Where 'future' is an instance of asyncio.Future or asyncio.Task. + + 'call_stack' is a tuple of FrameGraphEntry objects. + + 'awaited_by' is a tuple of FutureCallGraph objects. + + * FrameCallGraphEntry(frame) + + Where 'frame' is a frame object of a regular Python function + in the call stack. + + Receives an optional 'future' argument. If not passed, + the current task will be used. If there's no current task, the function + returns None. + + If "capture_call_graph()" is introspecting *the current task*, the + optional keyword-only 'depth' argument can be used to skip the specified + number of frames from top of the stack. + + If the optional keyword-only 'limit' argument is provided, each call stack + in the resulting graph is truncated to include at most ``abs(limit)`` + entries. If 'limit' is positive, the entries left are the closest to + the invocation point. If 'limit' is negative, the topmost entries are + left. If 'limit' is omitted or None, all entries are present. + If 'limit' is 0, the call stack is not captured at all, only + "awaited by" information is present. + """ + + loop = events._get_running_loop() + + if future is not None: + # Check if we're in a context of a running event loop; + # if yes - check if the passed future is the currently + # running task or not. + if loop is None or future is not tasks.current_task(loop=loop): + return _build_graph_for_future(future, limit=limit) + # else: future is the current task, move on. + else: + if loop is None: + raise RuntimeError( + 'capture_call_graph() is called outside of a running ' + 'event loop and no *future* to introspect was provided') + future = tasks.current_task(loop=loop) + + if future is None: + # This isn't a generic call stack introspection utility. If we + # can't determine the current task and none was provided, we + # just return. + return None + + if not isinstance(future, futures.Future): + raise TypeError( + f"{future!r} object does not appear to be compatible " + f"with asyncio.Future" + ) + + call_stack: list[FrameCallGraphEntry] = [] + + f = sys._getframe(depth) if limit != 0 else None + try: + while f is not None: + is_async = f.f_generator is not None + call_stack.append(FrameCallGraphEntry(f)) + + if is_async: + if f.f_back is not None and f.f_back.f_generator is None: + # We've reached the bottom of the coroutine stack, which + # must be the Task that runs it. + break + + f = f.f_back + finally: + del f + + awaited_by = [] + if future._asyncio_awaited_by: + for parent in future._asyncio_awaited_by: + awaited_by.append(_build_graph_for_future(parent, limit=limit)) + + if limit is not None: + limit *= -1 + if limit > 0: + call_stack = call_stack[:limit] + elif limit < 0: + call_stack = call_stack[limit:] + + return FutureCallGraph(future, tuple(call_stack), tuple(awaited_by)) + + +def format_call_graph( + future: futures.Future | None = None, + /, + *, + depth: int = 1, + limit: int | None = None, +) -> str: + """Return the async call graph as a string for `future`. + + If `future` is not provided, format the call graph for the current task. + """ + + def render_level(st: FutureCallGraph, buf: list[str], level: int) -> None: + def add_line(line: str) -> None: + buf.append(level * ' ' + line) + + if isinstance(st.future, tasks.Task): + add_line( + f'* Task(name={st.future.get_name()!r}, id={id(st.future):#x})' + ) + else: + add_line( + f'* Future(id={id(st.future):#x})' + ) + + if st.call_stack: + add_line( + f' + Call stack:' + ) + for ste in st.call_stack: + f = ste.frame + + if f.f_generator is None: + f = ste.frame + add_line( + f' | File {f.f_code.co_filename!r},' + f' line {f.f_lineno}, in' + f' {f.f_code.co_qualname}()' + ) + else: + c = f.f_generator + + try: + f = c.cr_frame + code = c.cr_code + tag = 'async' + except AttributeError: + try: + f = c.ag_frame + code = c.ag_code + tag = 'async generator' + except AttributeError: + f = c.gi_frame + code = c.gi_code + tag = 'generator' + + add_line( + f' | File {f.f_code.co_filename!r},' + f' line {f.f_lineno}, in' + f' {tag} {code.co_qualname}()' + ) + + if st.awaited_by: + add_line( + f' + Awaited by:' + ) + for fut in st.awaited_by: + render_level(fut, buf, level + 1) + + graph = capture_call_graph(future, depth=depth + 1, limit=limit) + if graph is None: + return "" + + buf: list[str] = [] + try: + render_level(graph, buf, 0) + finally: + # 'graph' has references to frames so we should + # make sure it's GC'ed as soon as we don't need it. + del graph + return '\n'.join(buf) + +def print_call_graph( + future: futures.Future | None = None, + /, + *, + file: io.Writer[str] | None = None, + depth: int = 1, + limit: int | None = None, +) -> None: + """Print the async call graph for the current task or the provided Future.""" + print(format_call_graph(future, depth=depth, limit=limit), file=file) diff --git a/Lib/asyncio/locks.py b/Lib/asyncio/locks.py index ce5d8d5bfb2..fa3a94764b5 100644 --- a/Lib/asyncio/locks.py +++ b/Lib/asyncio/locks.py @@ -24,25 +24,23 @@ class Lock(_ContextManagerMixin, mixins._LoopBoundMixin): """Primitive lock objects. A primitive lock is a synchronization primitive that is not owned - by a particular coroutine when locked. A primitive lock is in one + by a particular task when locked. A primitive lock is in one of two states, 'locked' or 'unlocked'. It is created in the unlocked state. It has two basic methods, acquire() and release(). When the state is unlocked, acquire() changes the state to locked and returns immediately. When the state is locked, acquire() blocks until a call to release() in - another coroutine changes it to unlocked, then the acquire() call + another task changes it to unlocked, then the acquire() call resets it to locked and returns. The release() method should only be called in the locked state; it changes the state to unlocked and returns immediately. If an attempt is made to release an unlocked lock, a RuntimeError will be raised. - When more than one coroutine is blocked in acquire() waiting for - the state to turn to unlocked, only one coroutine proceeds when a - release() call resets the state to unlocked; first coroutine which - is blocked in acquire() is being processed. - - acquire() is a coroutine and should be called with 'await'. + When more than one task is blocked in acquire() waiting for + the state to turn to unlocked, only one task proceeds when a + release() call resets the state to unlocked; successive release() + calls will unblock tasks in FIFO order. Locks also support the asynchronous context management protocol. 'async with lock' statement should be used. @@ -95,6 +93,8 @@ async def acquire(self): This method blocks until the lock is unlocked, then sets it to locked and returns True. """ + # Implement fair scheduling, where thread always waits + # its turn. Jumping the queue if all are cancelled is an optimization. if (not self._locked and (self._waiters is None or all(w.cancelled() for w in self._waiters))): self._locked = True @@ -105,19 +105,22 @@ async def acquire(self): fut = self._get_loop().create_future() self._waiters.append(fut) - # Finally block should be called before the CancelledError - # handling as we don't want CancelledError to call - # _wake_up_first() and attempt to wake up itself. try: try: await fut finally: self._waiters.remove(fut) except exceptions.CancelledError: + # Currently the only exception designed be able to occur here. + + # Ensure the lock invariant: If lock is not claimed (or about + # to be claimed by us) and there is a Task in waiters, + # ensure that the Task at the head will run. if not self._locked: self._wake_up_first() raise + # assert self._locked is False self._locked = True return True @@ -125,7 +128,7 @@ def release(self): """Release a lock. When the lock is locked, reset it to unlocked, and return. - If any other coroutines are blocked waiting for the lock to become + If any other tasks are blocked waiting for the lock to become unlocked, allow exactly one of them to proceed. When invoked on an unlocked lock, a RuntimeError is raised. @@ -139,7 +142,7 @@ def release(self): raise RuntimeError('Lock is not acquired.') def _wake_up_first(self): - """Wake up the first waiter if it isn't done.""" + """Ensure that the first waiter will wake up.""" if not self._waiters: return try: @@ -147,9 +150,7 @@ def _wake_up_first(self): except StopIteration: return - # .done() necessarily means that a waiter will wake up later on and - # either take the lock, or, if it was cancelled and lock wasn't - # taken already, will hit this again and wake up a new waiter. + # .done() means that the waiter is already set to wake up. if not fut.done(): fut.set_result(True) @@ -179,8 +180,8 @@ def is_set(self): return self._value def set(self): - """Set the internal flag to true. All coroutines waiting for it to - become true are awakened. Coroutine that call wait() once the flag is + """Set the internal flag to true. All tasks waiting for it to + become true are awakened. Tasks that call wait() once the flag is true will not block at all. """ if not self._value: @@ -191,7 +192,7 @@ def set(self): fut.set_result(True) def clear(self): - """Reset the internal flag to false. Subsequently, coroutines calling + """Reset the internal flag to false. Subsequently, tasks calling wait() will block until set() is called to set the internal flag to true again.""" self._value = False @@ -200,7 +201,7 @@ async def wait(self): """Block until the internal flag is true. If the internal flag is true on entry, return True - immediately. Otherwise, block until another coroutine calls + immediately. Otherwise, block until another task calls set() to set the flag to true, then return True. """ if self._value: @@ -219,8 +220,8 @@ class Condition(_ContextManagerMixin, mixins._LoopBoundMixin): """Asynchronous equivalent to threading.Condition. This class implements condition variable objects. A condition variable - allows one or more coroutines to wait until they are notified by another - coroutine. + allows one or more tasks to wait until they are notified by another + task. A new Lock object is created and used as the underlying lock. """ @@ -247,45 +248,64 @@ def __repr__(self): async def wait(self): """Wait until notified. - If the calling coroutine has not acquired the lock when this + If the calling task has not acquired the lock when this method is called, a RuntimeError is raised. This method releases the underlying lock, and then blocks until it is awakened by a notify() or notify_all() call for - the same condition variable in another coroutine. Once + the same condition variable in another task. Once awakened, it re-acquires the lock and returns True. + + This method may return spuriously, + which is why the caller should always + re-check the state and be prepared to wait() again. """ if not self.locked(): raise RuntimeError('cannot wait on un-acquired lock') + fut = self._get_loop().create_future() self.release() try: - fut = self._get_loop().create_future() - self._waiters.append(fut) try: - await fut - return True - finally: - self._waiters.remove(fut) - - finally: - # Must reacquire lock even if wait is cancelled - cancelled = False - while True: + self._waiters.append(fut) try: - await self.acquire() - break - except exceptions.CancelledError: - cancelled = True + await fut + return True + finally: + self._waiters.remove(fut) - if cancelled: - raise exceptions.CancelledError + finally: + # Must re-acquire lock even if wait is cancelled. + # We only catch CancelledError here, since we don't want any + # other (fatal) errors with the future to cause us to spin. + err = None + while True: + try: + await self.acquire() + break + except exceptions.CancelledError as e: + err = e + + if err is not None: + try: + raise err # Re-raise most recent exception instance. + finally: + err = None # Break reference cycles. + except BaseException: + # Any error raised out of here _may_ have occurred after this Task + # believed to have been successfully notified. + # Make sure to notify another Task instead. This may result + # in a "spurious wakeup", which is allowed as part of the + # Condition Variable protocol. + self._notify(1) + raise async def wait_for(self, predicate): """Wait until a predicate becomes true. - The predicate should be a callable which result will be - interpreted as a boolean value. The final predicate value is + The predicate should be a callable whose result will be + interpreted as a boolean value. The method will repeatedly + wait() until it evaluates to true. The final predicate value is the return value. """ result = predicate() @@ -295,20 +315,22 @@ async def wait_for(self, predicate): return result def notify(self, n=1): - """By default, wake up one coroutine waiting on this condition, if any. - If the calling coroutine has not acquired the lock when this method + """By default, wake up one task waiting on this condition, if any. + If the calling task has not acquired the lock when this method is called, a RuntimeError is raised. - This method wakes up at most n of the coroutines waiting for the - condition variable; it is a no-op if no coroutines are waiting. + This method wakes up n of the tasks waiting for the condition + variable; if fewer than n are waiting, they are all awoken. - Note: an awakened coroutine does not actually return from its + Note: an awakened task does not actually return from its wait() call until it can reacquire the lock. Since notify() does not release the lock, its caller should. """ if not self.locked(): raise RuntimeError('cannot notify on un-acquired lock') + self._notify(n) + def _notify(self, n): idx = 0 for fut in self._waiters: if idx >= n: @@ -319,9 +341,9 @@ def notify(self, n=1): fut.set_result(False) def notify_all(self): - """Wake up all threads waiting on this condition. This method acts - like notify(), but wakes up all waiting threads instead of one. If the - calling thread has not acquired the lock when this method is called, + """Wake up all tasks waiting on this condition. This method acts + like notify(), but wakes up all waiting tasks instead of one. If the + calling task has not acquired the lock when this method is called, a RuntimeError is raised. """ self.notify(len(self._waiters)) @@ -357,6 +379,7 @@ def __repr__(self): def locked(self): """Returns True if semaphore cannot be acquired immediately.""" + # Due to state, or FIFO rules (must allow others to run first). return self._value == 0 or ( any(not w.cancelled() for w in (self._waiters or ()))) @@ -365,11 +388,12 @@ async def acquire(self): If the internal counter is larger than zero on entry, decrement it by one and return True immediately. If it is - zero on entry, block, waiting until some other coroutine has + zero on entry, block, waiting until some other task has called release() to make it larger than 0, and then return True. """ if not self.locked(): + # Maintain FIFO, wait for others to start even if _value > 0. self._value -= 1 return True @@ -378,29 +402,34 @@ async def acquire(self): fut = self._get_loop().create_future() self._waiters.append(fut) - # Finally block should be called before the CancelledError - # handling as we don't want CancelledError to call - # _wake_up_first() and attempt to wake up itself. try: try: await fut finally: self._waiters.remove(fut) except exceptions.CancelledError: - if not fut.cancelled(): + # Currently the only exception designed be able to occur here. + if fut.done() and not fut.cancelled(): + # Our Future was successfully set to True via _wake_up_next(), + # but we are not about to successfully acquire(). Therefore we + # must undo the bookkeeping already done and attempt to wake + # up someone else. self._value += 1 - self._wake_up_next() raise - if self._value > 0: - self._wake_up_next() + finally: + # New waiters may have arrived but had to wait due to FIFO. + # Wake up as many as are allowed. + while self._value > 0: + if not self._wake_up_next(): + break # There was no-one to wake up. return True def release(self): """Release a semaphore, incrementing the internal counter by one. - When it was zero on entry and another coroutine is waiting for it to - become larger than zero again, wake up that coroutine. + When it was zero on entry and another task is waiting for it to + become larger than zero again, wake up that task. """ self._value += 1 self._wake_up_next() @@ -408,13 +437,15 @@ def release(self): def _wake_up_next(self): """Wake up the first waiter that isn't done.""" if not self._waiters: - return + return False for fut in self._waiters: if not fut.done(): self._value -= 1 fut.set_result(True) - return + # `fut` is now `done()` and not `cancelled()`. + return True + return False class BoundedSemaphore(Semaphore): @@ -454,7 +485,7 @@ class Barrier(mixins._LoopBoundMixin): def __init__(self, parties): """Create a barrier, initialised to 'parties' tasks.""" if parties < 1: - raise ValueError('parties must be > 0') + raise ValueError('parties must be >= 1') self._cond = Condition() # notify all tasks when state changes diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py index 1e2a730cf36..f404273c3ae 100644 --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -63,7 +63,7 @@ def __init__(self, loop, sock, protocol, waiter=None, self._called_connection_lost = False self._eof_written = False if self._server is not None: - self._server._attach() + self._server._attach(self) self._loop.call_soon(self._protocol.connection_made, self) if waiter is not None: # only wake up the waiter when connection_made() has been called @@ -167,7 +167,7 @@ def _call_connection_lost(self, exc): self._sock = None server = self._server if server is not None: - server._detach() + server._detach(self) self._server = None self._called_connection_lost = True @@ -460,6 +460,8 @@ def _pipe_closed(self, fut): class _ProactorDatagramTransport(_ProactorBasePipeTransport, transports.DatagramTransport): max_size = 256 * 1024 + _header_size = 8 + def __init__(self, loop, sock, protocol, address=None, waiter=None, extra=None): self._address = address @@ -487,9 +489,6 @@ def sendto(self, data, addr=None): raise TypeError('data argument must be bytes-like object (%r)', type(data)) - if not data: - return - if self._address is not None and addr not in (None, self._address): raise ValueError( f'Invalid address: must be None or {self._address}') @@ -502,7 +501,7 @@ def sendto(self, data, addr=None): # Ensure that what we buffer is immutable. self._buffer.append((bytes(data), addr)) - self._buffer_size += len(data) + self._buffer_size += len(data) + self._header_size if self._write_fut is None: # No current write operations are active, kick one off @@ -529,7 +528,7 @@ def _loop_writing(self, fut=None): return data, addr = self._buffer.popleft() - self._buffer_size -= len(data) + self._buffer_size -= len(data) + self._header_size if self._address is not None: self._write_fut = self._loop._proactor.send(self._sock, data) @@ -724,6 +723,8 @@ async def sock_sendto(self, sock, data, address): return await self._proactor.sendto(sock, data, 0, address) async def sock_connect(self, sock, address): + if self._debug and sock.gettimeout() != 0: + raise ValueError("the socket must be non-blocking") return await self._proactor.connect(sock, address) async def sock_accept(self, sock): diff --git a/Lib/asyncio/queues.py b/Lib/asyncio/queues.py index a9656a6df56..084fccaaff2 100644 --- a/Lib/asyncio/queues.py +++ b/Lib/asyncio/queues.py @@ -1,4 +1,11 @@ -__all__ = ('Queue', 'PriorityQueue', 'LifoQueue', 'QueueFull', 'QueueEmpty') +__all__ = ( + 'Queue', + 'PriorityQueue', + 'LifoQueue', + 'QueueFull', + 'QueueEmpty', + 'QueueShutDown', +) import collections import heapq @@ -18,6 +25,11 @@ class QueueFull(Exception): pass +class QueueShutDown(Exception): + """Raised when putting on to or getting from a shut-down Queue.""" + pass + + class Queue(mixins._LoopBoundMixin): """A queue, useful for coordinating producer and consumer coroutines. @@ -41,6 +53,7 @@ def __init__(self, maxsize=0): self._finished = locks.Event() self._finished.set() self._init(maxsize) + self._is_shutdown = False # These three are overridable in subclasses. @@ -81,6 +94,8 @@ def _format(self): result += f' _putters[{len(self._putters)}]' if self._unfinished_tasks: result += f' tasks={self._unfinished_tasks}' + if self._is_shutdown: + result += ' shutdown' return result def qsize(self): @@ -112,8 +127,12 @@ async def put(self, item): Put an item into the queue. If the queue is full, wait until a free slot is available before adding item. + + Raises QueueShutDown if the queue has been shut down. """ while self.full(): + if self._is_shutdown: + raise QueueShutDown putter = self._get_loop().create_future() self._putters.append(putter) try: @@ -125,7 +144,7 @@ async def put(self, item): self._putters.remove(putter) except ValueError: # The putter could be removed from self._putters by a - # previous get_nowait call. + # previous get_nowait call or a shutdown call. pass if not self.full() and not putter.cancelled(): # We were woken up by get_nowait(), but can't take @@ -138,7 +157,11 @@ def put_nowait(self, item): """Put an item into the queue without blocking. If no free slot is immediately available, raise QueueFull. + + Raises QueueShutDown if the queue has been shut down. """ + if self._is_shutdown: + raise QueueShutDown if self.full(): raise QueueFull self._put(item) @@ -150,8 +173,13 @@ async def get(self): """Remove and return an item from the queue. If queue is empty, wait until an item is available. + + Raises QueueShutDown if the queue has been shut down and is empty, or + if the queue has been shut down immediately. """ while self.empty(): + if self._is_shutdown and self.empty(): + raise QueueShutDown getter = self._get_loop().create_future() self._getters.append(getter) try: @@ -163,7 +191,7 @@ async def get(self): self._getters.remove(getter) except ValueError: # The getter could be removed from self._getters by a - # previous put_nowait call. + # previous put_nowait call, or a shutdown call. pass if not self.empty() and not getter.cancelled(): # We were woken up by put_nowait(), but can't take @@ -176,8 +204,13 @@ def get_nowait(self): """Remove and return an item from the queue. Return an item if one is immediately available, else raise QueueEmpty. + + Raises QueueShutDown if the queue has been shut down and is empty, or + if the queue has been shut down immediately. """ if self.empty(): + if self._is_shutdown: + raise QueueShutDown raise QueueEmpty item = self._get() self._wakeup_next(self._putters) @@ -214,6 +247,36 @@ async def join(self): if self._unfinished_tasks > 0: await self._finished.wait() + def shutdown(self, immediate=False): + """Shut-down the queue, making queue gets and puts raise QueueShutDown. + + By default, gets will only raise once the queue is empty. Set + 'immediate' to True to make gets raise immediately instead. + + All blocked callers of put() and get() will be unblocked. + + If 'immediate', the queue is drained and unfinished tasks + is reduced by the number of drained tasks. If unfinished tasks + is reduced to zero, callers of Queue.join are unblocked. + """ + self._is_shutdown = True + if immediate: + while not self.empty(): + self._get() + if self._unfinished_tasks > 0: + self._unfinished_tasks -= 1 + if self._unfinished_tasks == 0: + self._finished.set() + # All getters need to re-check queue-empty to raise ShutDown + while self._getters: + getter = self._getters.popleft() + if not getter.done(): + getter.set_result(None) + while self._putters: + putter = self._putters.popleft() + if not putter.done(): + putter.set_result(None) + class PriorityQueue(Queue): """A subclass of Queue; retrieves entries in priority order (lowest first). diff --git a/Lib/asyncio/runners.py b/Lib/asyncio/runners.py index 1b89236599a..ba37e003a65 100644 --- a/Lib/asyncio/runners.py +++ b/Lib/asyncio/runners.py @@ -3,6 +3,7 @@ import contextvars import enum import functools +import inspect import threading import signal from . import coroutines @@ -84,10 +85,7 @@ def get_loop(self): return self._loop def run(self, coro, *, context=None): - """Run a coroutine inside the embedded event loop.""" - if not coroutines.iscoroutine(coro): - raise ValueError("a coroutine was expected, got {!r}".format(coro)) - + """Run code in the embedded event loop.""" if events._get_running_loop() is not None: # fail fast with short traceback raise RuntimeError( @@ -95,8 +93,19 @@ def run(self, coro, *, context=None): self._lazy_init() + if not coroutines.iscoroutine(coro): + if inspect.isawaitable(coro): + async def _wrap_awaitable(awaitable): + return await awaitable + + coro = _wrap_awaitable(coro) + else: + raise TypeError('An asyncio.Future, a coroutine or an ' + 'awaitable is required') + if context is None: context = self._context + task = self._loop.create_task(coro, context=context) if (threading.current_thread() is threading.main_thread() @@ -168,6 +177,7 @@ def run(main, *, debug=None, loop_factory=None): running in the same thread. If debug is True, the event loop will be run in debug mode. + If loop_factory is passed, it is used for new event loop creation. This function always creates a new event loop and closes it at the end. It should be used as a main entry point for asyncio programs, and should diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py index 790711f8340..ff7e16df3c6 100644 --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -173,16 +173,20 @@ def _accept_connection( # listening socket has triggered an EVENT_READ. There may be multiple # connections waiting for an .accept() so it is called in a loop. # See https://bugs.python.org/issue27906 for more details. - for _ in range(backlog): + for _ in range(backlog + 1): try: conn, addr = sock.accept() if self._debug: logger.debug("%r got a new connection from %r: %r", server, addr, conn) conn.setblocking(False) - except (BlockingIOError, InterruptedError, ConnectionAbortedError): - # Early exit because the socket accept buffer is empty. - return None + except ConnectionAbortedError: + # Discard connections that were aborted before accept(). + continue + except (BlockingIOError, InterruptedError): + # Early exit because of a signal or + # the socket accept buffer is empty. + return except OSError as exc: # There's nowhere to send the error, so just log it. if exc.errno in (errno.EMFILE, errno.ENFILE, @@ -265,22 +269,17 @@ def _ensure_fd_no_transport(self, fd): except (AttributeError, TypeError, ValueError): # This code matches selectors._fileobj_to_fd function. raise ValueError(f"Invalid file object: {fd!r}") from None - try: - transport = self._transports[fileno] - except KeyError: - pass - else: - if not transport.is_closing(): - raise RuntimeError( - f'File descriptor {fd!r} is used by transport ' - f'{transport!r}') + transport = self._transports.get(fileno) + if transport and not transport.is_closing(): + raise RuntimeError( + f'File descriptor {fd!r} is used by transport ' + f'{transport!r}') def _add_reader(self, fd, callback, *args): self._check_closed() handle = events.Handle(callback, args, self, None) - try: - key = self._selector.get_key(fd) - except KeyError: + key = self._selector.get_map().get(fd) + if key is None: self._selector.register(fd, selectors.EVENT_READ, (handle, None)) else: @@ -294,30 +293,27 @@ def _add_reader(self, fd, callback, *args): def _remove_reader(self, fd): if self.is_closed(): return False - try: - key = self._selector.get_key(fd) - except KeyError: + key = self._selector.get_map().get(fd) + if key is None: return False + mask, (reader, writer) = key.events, key.data + mask &= ~selectors.EVENT_READ + if not mask: + self._selector.unregister(fd) else: - mask, (reader, writer) = key.events, key.data - mask &= ~selectors.EVENT_READ - if not mask: - self._selector.unregister(fd) - else: - self._selector.modify(fd, mask, (None, writer)) + self._selector.modify(fd, mask, (None, writer)) - if reader is not None: - reader.cancel() - return True - else: - return False + if reader is not None: + reader.cancel() + return True + else: + return False def _add_writer(self, fd, callback, *args): self._check_closed() handle = events.Handle(callback, args, self, None) - try: - key = self._selector.get_key(fd) - except KeyError: + key = self._selector.get_map().get(fd) + if key is None: self._selector.register(fd, selectors.EVENT_WRITE, (None, handle)) else: @@ -332,24 +328,22 @@ def _remove_writer(self, fd): """Remove a writer callback.""" if self.is_closed(): return False - try: - key = self._selector.get_key(fd) - except KeyError: + key = self._selector.get_map().get(fd) + if key is None: return False + mask, (reader, writer) = key.events, key.data + # Remove both writer and connector. + mask &= ~selectors.EVENT_WRITE + if not mask: + self._selector.unregister(fd) else: - mask, (reader, writer) = key.events, key.data - # Remove both writer and connector. - mask &= ~selectors.EVENT_WRITE - if not mask: - self._selector.unregister(fd) - else: - self._selector.modify(fd, mask, (reader, None)) + self._selector.modify(fd, mask, (reader, None)) - if writer is not None: - writer.cancel() - return True - else: - return False + if writer is not None: + writer.cancel() + return True + else: + return False def add_reader(self, fd, callback, *args): """Add a reader callback.""" @@ -801,7 +795,7 @@ def __init__(self, loop, sock, protocol, extra=None, server=None): self._paused = False # Set when pause_reading() called if self._server is not None: - self._server._attach() + self._server._attach(self) loop._transports[self._sock_fd] = self def __repr__(self): @@ -878,6 +872,8 @@ def __del__(self, _warn=warnings.warn): if self._sock is not None: _warn(f"unclosed transport {self!r}", ResourceWarning, source=self) self._sock.close() + if self._server is not None: + self._server._detach(self) def _fatal_error(self, exc, message='Fatal error on transport'): # Should be called from exception handler only. @@ -916,7 +912,7 @@ def _call_connection_lost(self, exc): self._loop = None server = self._server if server is not None: - server._detach() + server._detach(self) self._server = None def get_write_buffer_size(self): @@ -1054,8 +1050,8 @@ def _read_ready__on_eof(self): def write(self, data): if not isinstance(data, (bytes, bytearray, memoryview)): - raise TypeError(f'data argument must be a bytes-like object, ' - f'not {type(data).__name__!r}') + raise TypeError(f'data argument must be a bytes, bytearray, or memoryview ' + f'object, not {type(data).__name__!r}') if self._eof: raise RuntimeError('Cannot call write() after write_eof()') if self._empty_waiter is not None: @@ -1178,20 +1174,31 @@ def writelines(self, list_of_data): raise RuntimeError('unable to writelines; sendfile is in progress') if not list_of_data: return + + if self._conn_lost: + if self._conn_lost >= constants.LOG_THRESHOLD_FOR_CONNLOST_WRITES: + logger.warning('socket.send() raised exception.') + self._conn_lost += 1 + return + self._buffer.extend([memoryview(data) for data in list_of_data]) self._write_ready() # If the entire buffer couldn't be written, register a write handler if self._buffer: self._loop._add_writer(self._sock_fd, self._write_ready) + self._maybe_pause_protocol() def can_write_eof(self): return True def _call_connection_lost(self, exc): - super()._call_connection_lost(exc) - if self._empty_waiter is not None: - self._empty_waiter.set_exception( - ConnectionError("Connection is closed by peer")) + try: + super()._call_connection_lost(exc) + finally: + self._write_ready = None + if self._empty_waiter is not None: + self._empty_waiter.set_exception( + ConnectionError("Connection is closed by peer")) def _make_empty_waiter(self): if self._empty_waiter is not None: @@ -1206,13 +1213,13 @@ def _reset_empty_waiter(self): def close(self): self._read_ready_cb = None - self._write_ready = None super().close() class _SelectorDatagramTransport(_SelectorTransport, transports.DatagramTransport): _buffer_factory = collections.deque + _header_size = 8 def __init__(self, loop, sock, protocol, address=None, waiter=None, extra=None): @@ -1251,8 +1258,6 @@ def sendto(self, data, addr=None): if not isinstance(data, (bytes, bytearray, memoryview)): raise TypeError(f'data argument must be a bytes-like object, ' f'not {type(data).__name__!r}') - if not data: - return if self._address: if addr not in (None, self._address): @@ -1288,13 +1293,13 @@ def sendto(self, data, addr=None): # Ensure that what we buffer is immutable. self._buffer.append((bytes(data), addr)) - self._buffer_size += len(data) + self._buffer_size += len(data) + self._header_size self._maybe_pause_protocol() def _sendto_ready(self): while self._buffer: data, addr = self._buffer.popleft() - self._buffer_size -= len(data) + self._buffer_size -= len(data) + self._header_size try: if self._extra['peername']: self._sock.send(data) @@ -1302,7 +1307,7 @@ def _sendto_ready(self): self._sock.sendto(data, addr) except (BlockingIOError, InterruptedError): self._buffer.appendleft((data, addr)) # Try again later. - self._buffer_size += len(data) + self._buffer_size += len(data) + self._header_size break except OSError as exc: self._protocol.error_received(exc) diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py index e51669a2ab2..74c5f0d5ca0 100644 --- a/Lib/asyncio/sslproto.py +++ b/Lib/asyncio/sslproto.py @@ -101,7 +101,7 @@ def get_protocol(self): return self._ssl_protocol._app_protocol def is_closing(self): - return self._closed + return self._closed or self._ssl_protocol._is_transport_closing() def close(self): """Close the transport. @@ -379,6 +379,9 @@ def _get_app_transport(self): self._app_transport_created = True return self._app_transport + def _is_transport_closing(self): + return self._transport is not None and self._transport.is_closing() + def connection_made(self, transport): """Called when the low-level connection is made. @@ -542,7 +545,7 @@ def _start_handshake(self): # start handshake timeout count down self._handshake_timeout_handle = \ self._loop.call_later(self._ssl_handshake_timeout, - lambda: self._check_handshake_timeout()) + self._check_handshake_timeout) self._do_handshake() @@ -623,7 +626,7 @@ def _start_shutdown(self): self._set_state(SSLProtocolState.FLUSHING) self._shutdown_timeout_handle = self._loop.call_later( self._ssl_shutdown_timeout, - lambda: self._check_shutdown_timeout() + self._check_shutdown_timeout ) self._do_flush() @@ -762,7 +765,7 @@ def _do_read__buffered(self): else: break else: - self._loop.call_soon(lambda: self._do_read()) + self._loop.call_soon(self._do_read) except SSLAgainErrors: pass if offset > 0: diff --git a/Lib/asyncio/staggered.py b/Lib/asyncio/staggered.py index 451a53a16f3..2ad65d8648e 100644 --- a/Lib/asyncio/staggered.py +++ b/Lib/asyncio/staggered.py @@ -3,24 +3,15 @@ __all__ = 'staggered_race', import contextlib -import typing from . import events from . import exceptions as exceptions_mod from . import locks from . import tasks +from . import futures -async def staggered_race( - coro_fns: typing.Iterable[typing.Callable[[], typing.Awaitable]], - delay: typing.Optional[float], - *, - loop: events.AbstractEventLoop = None, -) -> typing.Tuple[ - typing.Any, - typing.Optional[int], - typing.List[typing.Optional[Exception]] -]: +async def staggered_race(coro_fns, delay, *, loop=None): """Run coroutines with staggered start times and take the first to finish. This method takes an iterable of coroutine functions. The first one is @@ -73,14 +64,38 @@ async def staggered_race( """ # TODO: when we have aiter() and anext(), allow async iterables in coro_fns. loop = loop or events.get_running_loop() + parent_task = tasks.current_task(loop) enum_coro_fns = enumerate(coro_fns) winner_result = None winner_index = None + unhandled_exceptions = [] exceptions = [] - running_tasks = [] + running_tasks = set() + on_completed_fut = None + + def task_done(task): + running_tasks.discard(task) + futures.future_discard_from_awaited_by(task, parent_task) + if ( + on_completed_fut is not None + and not on_completed_fut.done() + and not running_tasks + ): + on_completed_fut.set_result(None) + + if task.cancelled(): + return + + exc = task.exception() + if exc is None: + return + unhandled_exceptions.append(exc) - async def run_one_coro( - previous_failed: typing.Optional[locks.Event]) -> None: + async def run_one_coro(ok_to_start, previous_failed) -> None: + # in eager tasks this waits for the calling task to append this task + # to running_tasks, in regular tasks this wait is a no-op that does + # not yield a future. See gh-124309. + await ok_to_start.wait() # Wait for the previous task to finish, or for delay seconds if previous_failed is not None: with contextlib.suppress(exceptions_mod.TimeoutError): @@ -96,9 +111,14 @@ async def run_one_coro( return # Start task that will run the next coroutine this_failed = locks.Event() - next_task = loop.create_task(run_one_coro(this_failed)) - running_tasks.append(next_task) - assert len(running_tasks) == this_index + 2 + next_ok_to_start = locks.Event() + next_task = loop.create_task(run_one_coro(next_ok_to_start, this_failed)) + futures.future_add_to_awaited_by(next_task, parent_task) + running_tasks.add(next_task) + next_task.add_done_callback(task_done) + # next_task has been appended to running_tasks so next_task is ok to + # start. + next_ok_to_start.set() # Prepare place to put this coroutine's exceptions if not won exceptions.append(None) assert len(exceptions) == this_index + 1 @@ -123,27 +143,37 @@ async def run_one_coro( # up as done() == True, cancelled() == False, exception() == # asyncio.CancelledError. This behavior is specified in # https://bugs.python.org/issue30048 - for i, t in enumerate(running_tasks): - if i != this_index: + current_task = tasks.current_task(loop) + for t in running_tasks: + if t is not current_task: t.cancel() - first_task = loop.create_task(run_one_coro(None)) - running_tasks.append(first_task) + propagate_cancellation_error = None try: - # Wait for a growing list of tasks to all finish: poor man's version of - # curio's TaskGroup or trio's nursery - done_count = 0 - while done_count != len(running_tasks): - done, _ = await tasks.wait(running_tasks) - done_count = len(done) + ok_to_start = locks.Event() + first_task = loop.create_task(run_one_coro(ok_to_start, None)) + futures.future_add_to_awaited_by(first_task, parent_task) + running_tasks.add(first_task) + first_task.add_done_callback(task_done) + # first_task has been appended to running_tasks so first_task is ok to start. + ok_to_start.set() + propagate_cancellation_error = None + # Make sure no tasks are left running if we leave this function + while running_tasks: + on_completed_fut = loop.create_future() + try: + await on_completed_fut + except exceptions_mod.CancelledError as ex: + propagate_cancellation_error = ex + for task in running_tasks: + task.cancel(*ex.args) + on_completed_fut = None + if __debug__ and unhandled_exceptions: # If run_one_coro raises an unhandled exception, it's probably a # programming error, and I want to see it. - if __debug__: - for d in done: - if d.done() and not d.cancelled() and d.exception(): - raise d.exception() + raise ExceptionGroup("staggered race failed", unhandled_exceptions) + if propagate_cancellation_error is not None: + raise propagate_cancellation_error return winner_result, winner_index, exceptions finally: - # Make sure no tasks are left running if we leave this function - for t in running_tasks: - t.cancel() + del exceptions, propagate_cancellation_error, unhandled_exceptions, parent_task diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py index f310aa2f367..64aac4cc50d 100644 --- a/Lib/asyncio/streams.py +++ b/Lib/asyncio/streams.py @@ -201,7 +201,6 @@ def __init__(self, stream_reader, client_connected_cb=None, loop=None): # is established. self._strong_reader = stream_reader self._reject_connection = False - self._stream_writer = None self._task = None self._transport = None self._client_connected_cb = client_connected_cb @@ -214,10 +213,8 @@ def _stream_reader(self): return None return self._stream_reader_wr() - def _replace_writer(self, writer): + def _replace_transport(self, transport): loop = self._loop - transport = writer.transport - self._stream_writer = writer self._transport = transport self._over_ssl = transport.get_extra_info('sslcontext') is not None @@ -239,11 +236,8 @@ def connection_made(self, transport): reader.set_transport(transport) self._over_ssl = transport.get_extra_info('sslcontext') is not None if self._client_connected_cb is not None: - self._stream_writer = StreamWriter(transport, self, - reader, - self._loop) - res = self._client_connected_cb(reader, - self._stream_writer) + writer = StreamWriter(transport, self, reader, self._loop) + res = self._client_connected_cb(reader, writer) if coroutines.iscoroutine(res): def callback(task): if task.cancelled(): @@ -405,9 +399,9 @@ async def start_tls(self, sslcontext, *, ssl_handshake_timeout=ssl_handshake_timeout, ssl_shutdown_timeout=ssl_shutdown_timeout) self._transport = new_transport - protocol._replace_writer(self) + protocol._replace_transport(new_transport) - def __del__(self): + def __del__(self, warnings=warnings): if not self._transport.is_closing(): if self._loop.is_closed(): warnings.warn("loop is closed", ResourceWarning) @@ -596,20 +590,34 @@ async def readuntil(self, separator=b'\n'): If the data cannot be read because of over limit, a LimitOverrunError exception will be raised, and the data will be left in the internal buffer, so it can be read again. + + The ``separator`` may also be a tuple of separators. In this + case the return value will be the shortest possible that has any + separator as the suffix. For the purposes of LimitOverrunError, + the shortest possible separator is considered to be the one that + matched. """ - seplen = len(separator) - if seplen == 0: + if isinstance(separator, tuple): + # Makes sure shortest matches wins + separator = sorted(separator, key=len) + else: + separator = [separator] + if not separator: + raise ValueError('Separator should contain at least one element') + min_seplen = len(separator[0]) + max_seplen = len(separator[-1]) + if min_seplen == 0: raise ValueError('Separator should be at least one-byte string') if self._exception is not None: raise self._exception # Consume whole buffer except last bytes, which length is - # one less than seplen. Let's check corner cases with - # separator='SEPARATOR': + # one less than max_seplen. Let's check corner cases with + # separator[-1]='SEPARATOR': # * we have received almost complete separator (without last # byte). i.e buffer='some textSEPARATO'. In this case we - # can safely consume len(separator) - 1 bytes. + # can safely consume max_seplen - 1 bytes. # * last byte of buffer is first byte of separator, i.e. # buffer='abcdefghijklmnopqrS'. We may safely consume # everything except that last byte, but this require to @@ -622,26 +630,35 @@ async def readuntil(self, separator=b'\n'): # messages :) # `offset` is the number of bytes from the beginning of the buffer - # where there is no occurrence of `separator`. + # where there is no occurrence of any `separator`. offset = 0 - # Loop until we find `separator` in the buffer, exceed the buffer size, + # Loop until we find a `separator` in the buffer, exceed the buffer size, # or an EOF has happened. while True: buflen = len(self._buffer) - # Check if we now have enough data in the buffer for `separator` to - # fit. - if buflen - offset >= seplen: - isep = self._buffer.find(separator, offset) - - if isep != -1: - # `separator` is in the buffer. `isep` will be used later - # to retrieve the data. + # Check if we now have enough data in the buffer for shortest + # separator to fit. + if buflen - offset >= min_seplen: + match_start = None + match_end = None + for sep in separator: + isep = self._buffer.find(sep, offset) + + if isep != -1: + # `separator` is in the buffer. `match_start` and + # `match_end` will be used later to retrieve the + # data. + end = isep + len(sep) + if match_end is None or end < match_end: + match_end = end + match_start = isep + if match_end is not None: break # see upper comment for explanation. - offset = buflen + 1 - seplen + offset = max(0, buflen + 1 - max_seplen) if offset > self._limit: raise exceptions.LimitOverrunError( 'Separator is not found, and chunk exceed the limit', @@ -650,7 +667,7 @@ async def readuntil(self, separator=b'\n'): # Complete message (with full separator) may be present in buffer # even when EOF flag is set. This may happen when the last chunk # adds data which makes separator be found. That's why we check for - # EOF *ater* inspecting the buffer. + # EOF *after* inspecting the buffer. if self._eof: chunk = bytes(self._buffer) self._buffer.clear() @@ -659,12 +676,12 @@ async def readuntil(self, separator=b'\n'): # _wait_for_data() will resume reading if stream was paused. await self._wait_for_data('readuntil') - if isep > self._limit: + if match_start > self._limit: raise exceptions.LimitOverrunError( - 'Separator is found, but chunk is longer than limit', isep) + 'Separator is found, but chunk is longer than limit', match_start) - chunk = self._buffer[:isep + seplen] - del self._buffer[:isep + seplen] + chunk = self._buffer[:match_end] + del self._buffer[:match_end] self._maybe_resume_transport() return bytes(chunk) diff --git a/Lib/asyncio/taskgroups.py b/Lib/asyncio/taskgroups.py index d264e51f1fd..00e8f6d5d1a 100644 --- a/Lib/asyncio/taskgroups.py +++ b/Lib/asyncio/taskgroups.py @@ -6,6 +6,7 @@ from . import events from . import exceptions +from . import futures from . import tasks @@ -66,6 +67,20 @@ async def __aenter__(self): return self async def __aexit__(self, et, exc, tb): + tb = None + try: + return await self._aexit(et, exc) + finally: + # Exceptions are heavy objects that can have object + # cycles (bad for GC); let's not keep a reference to + # a bunch of them. It would be nicer to use a try/finally + # in __aexit__ directly but that introduced some diff noise + self._parent_task = None + self._errors = None + self._base_error = None + exc = None + + async def _aexit(self, et, exc): self._exiting = True if (exc is not None and @@ -73,14 +88,10 @@ async def __aexit__(self, et, exc, tb): self._base_error is None): self._base_error = exc - propagate_cancellation_error = \ - exc if et is exceptions.CancelledError else None - if self._parent_cancel_requested: - # If this flag is set we *must* call uncancel(). - if self._parent_task.uncancel() == 0: - # If there are no pending cancellations left, - # don't propagate CancelledError. - propagate_cancellation_error = None + if et is not None and issubclass(et, exceptions.CancelledError): + propagate_cancellation_error = exc + else: + propagate_cancellation_error = None if et is not None: if not self._aborting: @@ -126,51 +137,78 @@ async def __aexit__(self, et, exc, tb): assert not self._tasks if self._base_error is not None: - raise self._base_error + try: + raise self._base_error + finally: + exc = None + + if self._parent_cancel_requested: + # If this flag is set we *must* call uncancel(). + if self._parent_task.uncancel() == 0: + # If there are no pending cancellations left, + # don't propagate CancelledError. + propagate_cancellation_error = None # Propagate CancelledError if there is one, except if there # are other errors -- those have priority. - if propagate_cancellation_error and not self._errors: - raise propagate_cancellation_error - - if et is not None and et is not exceptions.CancelledError: + try: + if propagate_cancellation_error is not None and not self._errors: + try: + raise propagate_cancellation_error + finally: + exc = None + finally: + propagate_cancellation_error = None + + if et is not None and not issubclass(et, exceptions.CancelledError): self._errors.append(exc) if self._errors: - # Exceptions are heavy objects that can have object - # cycles (bad for GC); let's not keep a reference to - # a bunch of them. + # If the parent task is being cancelled from the outside + # of the taskgroup, un-cancel and re-cancel the parent task, + # which will keep the cancel count stable. + if self._parent_task.cancelling(): + self._parent_task.uncancel() + self._parent_task.cancel() try: - me = BaseExceptionGroup('unhandled errors in a TaskGroup', self._errors) - raise me from None + raise BaseExceptionGroup( + 'unhandled errors in a TaskGroup', + self._errors, + ) from None finally: - self._errors = None + exc = None - def create_task(self, coro, *, name=None, context=None): + + def create_task(self, coro, **kwargs): """Create a new task in this group and return it. Similar to `asyncio.create_task`. """ if not self._entered: + coro.close() raise RuntimeError(f"TaskGroup {self!r} has not been entered") if self._exiting and not self._tasks: + coro.close() raise RuntimeError(f"TaskGroup {self!r} is finished") if self._aborting: + coro.close() raise RuntimeError(f"TaskGroup {self!r} is shutting down") - if context is None: - task = self._loop.create_task(coro) - else: - task = self._loop.create_task(coro, context=context) - tasks._set_task_name(task, name) - # optimization: Immediately call the done callback if the task is + task = self._loop.create_task(coro, **kwargs) + + futures.future_add_to_awaited_by(task, self._parent_task) + + # Always schedule the done callback even if the task is # already done (e.g. if the coro was able to complete eagerly), - # and skip scheduling a done callback - if task.done(): - self._on_task_done(task) - else: - self._tasks.add(task) - task.add_done_callback(self._on_task_done) - return task + # otherwise if the task completes with an exception then it will cancel + # the current task too early. gh-128550, gh-128588 + self._tasks.add(task) + task.add_done_callback(self._on_task_done) + try: + return task + finally: + # gh-128552: prevent a refcycle of + # task.exception().__traceback__->TaskGroup.create_task->task + del task # Since Python 3.8 Tasks propagate all exceptions correctly, # except for KeyboardInterrupt and SystemExit which are @@ -190,6 +228,8 @@ def _abort(self): def _on_task_done(self, task): self._tasks.discard(task) + futures.future_discard_from_awaited_by(task, self._parent_task) + if self._on_completed_fut is not None and not self._tasks: if not self._on_completed_fut.done(): self._on_completed_fut.set_result(True) diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py index 0b22e28d8e0..fbd5c39a7c5 100644 --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -15,8 +15,8 @@ import functools import inspect import itertools +import math import types -import warnings import weakref from types import GenericAlias @@ -25,6 +25,7 @@ from . import events from . import exceptions from . import futures +from . import queues from . import timeouts # Helper to generate new task names @@ -47,37 +48,9 @@ def all_tasks(loop=None): # capturing the set of eager tasks first, so if an eager task "graduates" # to a regular task in another thread, we don't risk missing it. eager_tasks = list(_eager_tasks) - # Looping over the WeakSet isn't safe as it can be updated from another - # thread, therefore we cast it to list prior to filtering. The list cast - # itself requires iteration, so we repeat it several times ignoring - # RuntimeErrors (which are not very likely to occur). - # See issues 34970 and 36607 for details. - scheduled_tasks = None - i = 0 - while True: - try: - scheduled_tasks = list(_scheduled_tasks) - except RuntimeError: - i += 1 - if i >= 1000: - raise - else: - break - return {t for t in itertools.chain(scheduled_tasks, eager_tasks) - if futures._get_loop(t) is loop and not t.done()} - -def _set_task_name(task, name): - if name is not None: - try: - set_name = task.set_name - except AttributeError: - warnings.warn("Task.set_name() was added in Python 3.8, " - "the method support will be mandatory for third-party " - "task implementations since 3.13.", - DeprecationWarning, stacklevel=3) - else: - set_name(name) + return {t for t in itertools.chain(_scheduled_tasks, eager_tasks) + if futures._get_loop(t) is loop and not t.done()} class Task(futures._PyFuture): # Inherit Python Task implementation @@ -137,7 +110,7 @@ def __init__(self, coro, *, loop=None, name=None, context=None, self.__eager_start() else: self._loop.call_soon(self.__step, context=self._context) - _register_task(self) + _py_register_task(self) def __del__(self): if self._state == futures._PENDING and self._log_destroy_pending: @@ -267,42 +240,44 @@ def uncancel(self): """ if self._num_cancels_requested > 0: self._num_cancels_requested -= 1 + if self._num_cancels_requested == 0: + self._must_cancel = False return self._num_cancels_requested def __eager_start(self): - prev_task = _swap_current_task(self._loop, self) + prev_task = _py_swap_current_task(self._loop, self) try: - _register_eager_task(self) + _py_register_eager_task(self) try: self._context.run(self.__step_run_and_handle_result, None) finally: - _unregister_eager_task(self) + _py_unregister_eager_task(self) finally: try: - curtask = _swap_current_task(self._loop, prev_task) + curtask = _py_swap_current_task(self._loop, prev_task) assert curtask is self finally: if self.done(): self._coro = None self = None # Needed to break cycles when an exception occurs. else: - _register_task(self) + _py_register_task(self) def __step(self, exc=None): if self.done(): raise exceptions.InvalidStateError( - f'_step(): already done: {self!r}, {exc!r}') + f'__step(): already done: {self!r}, {exc!r}') if self._must_cancel: if not isinstance(exc, exceptions.CancelledError): exc = self._make_cancelled_error() self._must_cancel = False self._fut_waiter = None - _enter_task(self._loop, self) + _py_enter_task(self._loop, self) try: self.__step_run_and_handle_result(exc) finally: - _leave_task(self._loop, self) + _py_leave_task(self._loop, self) self = None # Needed to break cycles when an exception occurs. def __step_run_and_handle_result(self, exc): @@ -347,6 +322,7 @@ def __step_run_and_handle_result(self, exc): self._loop.call_soon( self.__step, new_exc, context=self._context) else: + futures.future_add_to_awaited_by(result, self) result._asyncio_future_blocking = False result.add_done_callback( self.__wakeup, context=self._context) @@ -381,6 +357,7 @@ def __step_run_and_handle_result(self, exc): self = None # Needed to break cycles when an exception occurs. def __wakeup(self, future): + futures.future_discard_from_awaited_by(future, self) try: future.result() except BaseException as exc: @@ -389,7 +366,7 @@ def __wakeup(self, future): else: # Don't pass the value of `future.result()` explicitly, # as `Future.__iter__` and `Future.__await__` don't need it. - # If we call `_step(value, None)` instead of `_step()`, + # If we call `__step(value, None)` instead of `__step()`, # Python eval loop would use `.send(value)` method call, # instead of `__next__()`, which is slower for futures # that return non-generator iterators from their `__iter__`. @@ -409,20 +386,13 @@ def __wakeup(self, future): Task = _CTask = _asyncio.Task -def create_task(coro, *, name=None, context=None): +def create_task(coro, **kwargs): """Schedule the execution of a coroutine object in a spawn task. Return a Task object. """ loop = events.get_running_loop() - if context is None: - # Use legacy API if context is not needed - task = loop.create_task(coro) - else: - task = loop.create_task(coro, context=context) - - _set_task_name(task, name) - return task + return loop.create_task(coro, **kwargs) # wait() and as_completed() similar to those in PEP 3148. @@ -437,8 +407,6 @@ async def wait(fs, *, timeout=None, return_when=ALL_COMPLETED): The fs iterable must not be empty. - Coroutines will be wrapped in Tasks. - Returns two sets of Future: (done, pending). Usage: @@ -530,6 +498,7 @@ async def _wait(fs, timeout, return_when, loop): if timeout is not None: timeout_handle = loop.call_later(timeout, _release_waiter, waiter) counter = len(fs) + cur_task = current_task() def _on_completion(f): nonlocal counter @@ -542,9 +511,11 @@ def _on_completion(f): timeout_handle.cancel() if not waiter.done(): waiter.set_result(None) + futures.future_discard_from_awaited_by(f, cur_task) for f in fs: f.add_done_callback(_on_completion) + futures.future_add_to_awaited_by(f, cur_task) try: await waiter @@ -580,62 +551,125 @@ async def _cancel_and_wait(fut): fut.remove_done_callback(cb) -# This is *not* a @coroutine! It is just an iterator (yielding Futures). +class _AsCompletedIterator: + """Iterator of awaitables representing tasks of asyncio.as_completed. + + As an asynchronous iterator, iteration yields futures as they finish. As a + plain iterator, new coroutines are yielded that will return or raise the + result of the next underlying future to complete. + """ + def __init__(self, aws, timeout): + self._done = queues.Queue() + self._timeout_handle = None + + loop = events.get_event_loop() + todo = {ensure_future(aw, loop=loop) for aw in set(aws)} + for f in todo: + f.add_done_callback(self._handle_completion) + if todo and timeout is not None: + self._timeout_handle = ( + loop.call_later(timeout, self._handle_timeout) + ) + self._todo = todo + self._todo_left = len(todo) + + def __aiter__(self): + return self + + def __iter__(self): + return self + + async def __anext__(self): + if not self._todo_left: + raise StopAsyncIteration + assert self._todo_left > 0 + self._todo_left -= 1 + return await self._wait_for_one() + + def __next__(self): + if not self._todo_left: + raise StopIteration + assert self._todo_left > 0 + self._todo_left -= 1 + return self._wait_for_one(resolve=True) + + def _handle_timeout(self): + for f in self._todo: + f.remove_done_callback(self._handle_completion) + self._done.put_nowait(None) # Sentinel for _wait_for_one(). + self._todo.clear() # Can't do todo.remove(f) in the loop. + + def _handle_completion(self, f): + if not self._todo: + return # _handle_timeout() was here first. + self._todo.remove(f) + self._done.put_nowait(f) + if not self._todo and self._timeout_handle is not None: + self._timeout_handle.cancel() + + async def _wait_for_one(self, resolve=False): + # Wait for the next future to be done and return it unless resolve is + # set, in which case return either the result of the future or raise + # an exception. + f = await self._done.get() + if f is None: + # Dummy value from _handle_timeout(). + raise exceptions.TimeoutError + return f.result() if resolve else f + + def as_completed(fs, *, timeout=None): - """Return an iterator whose values are coroutines. + """Create an iterator of awaitables or their results in completion order. - When waiting for the yielded coroutines you'll get the results (or - exceptions!) of the original Futures (or coroutines), in the order - in which and as soon as they complete. + Run the supplied awaitables concurrently. The returned object can be + iterated to obtain the results of the awaitables as they finish. - This differs from PEP 3148; the proper way to use this is: + The object returned can be iterated as an asynchronous iterator or a plain + iterator. When asynchronous iteration is used, the originally-supplied + awaitables are yielded if they are tasks or futures. This makes it easy to + correlate previously-scheduled tasks with their results: - for f in as_completed(fs): - result = await f # The 'await' may raise. - # Use result. + ipv4_connect = create_task(open_connection("127.0.0.1", 80)) + ipv6_connect = create_task(open_connection("::1", 80)) + tasks = [ipv4_connect, ipv6_connect] - If a timeout is specified, the 'await' will raise - TimeoutError when the timeout occurs before all Futures are done. + async for earliest_connect in as_completed(tasks): + # earliest_connect is done. The result can be obtained by + # awaiting it or calling earliest_connect.result() + reader, writer = await earliest_connect - Note: The futures 'f' are not necessarily members of fs. - """ - if futures.isfuture(fs) or coroutines.iscoroutine(fs): - raise TypeError(f"expect an iterable of futures, not {type(fs).__name__}") + if earliest_connect is ipv6_connect: + print("IPv6 connection established.") + else: + print("IPv4 connection established.") - from .queues import Queue # Import here to avoid circular import problem. - done = Queue() + During asynchronous iteration, implicitly-created tasks will be yielded for + supplied awaitables that aren't tasks or futures. - loop = events.get_event_loop() - todo = {ensure_future(f, loop=loop) for f in set(fs)} - timeout_handle = None + When used as a plain iterator, each iteration yields a new coroutine that + returns the result or raises the exception of the next completed awaitable. + This pattern is compatible with Python versions older than 3.13: - def _on_timeout(): - for f in todo: - f.remove_done_callback(_on_completion) - done.put_nowait(None) # Queue a dummy value for _wait_for_one(). - todo.clear() # Can't do todo.remove(f) in the loop. + ipv4_connect = create_task(open_connection("127.0.0.1", 80)) + ipv6_connect = create_task(open_connection("::1", 80)) + tasks = [ipv4_connect, ipv6_connect] - def _on_completion(f): - if not todo: - return # _on_timeout() was here first. - todo.remove(f) - done.put_nowait(f) - if not todo and timeout_handle is not None: - timeout_handle.cancel() + for next_connect in as_completed(tasks): + # next_connect is not one of the original task objects. It must be + # awaited to obtain the result value or raise the exception of the + # awaitable that finishes next. + reader, writer = await next_connect - async def _wait_for_one(): - f = await done.get() - if f is None: - # Dummy value from _on_timeout(). - raise exceptions.TimeoutError - return f.result() # May raise f.exception(). + A TimeoutError is raised if the timeout occurs before all awaitables are + done. This is raised by the async for loop during asynchronous iteration or + by the coroutines yielded during plain iteration. + """ + if inspect.isawaitable(fs): + raise TypeError( + f"expects an iterable of awaitables, not {type(fs).__name__}" + ) - for f in todo: - f.add_done_callback(_on_completion) - if todo and timeout is not None: - timeout_handle = loop.call_later(timeout, _on_timeout) - for _ in range(len(todo)): - yield _wait_for_one() + return _AsCompletedIterator(fs, timeout) @types.coroutine @@ -656,6 +690,9 @@ async def sleep(delay, result=None): await __sleep0() return result + if math.isnan(delay): + raise ValueError("Invalid delay: NaN (not a number)") + loop = events.get_running_loop() future = loop.create_future() h = loop.call_later(delay, @@ -764,10 +801,19 @@ def gather(*coros_or_futures, return_exceptions=False): outer.set_result([]) return outer - def _done_callback(fut): + loop = events._get_running_loop() + if loop is not None: + cur_task = current_task(loop) + else: + cur_task = None + + def _done_callback(fut, cur_task=cur_task): nonlocal nfinished nfinished += 1 + if cur_task is not None: + futures.future_discard_from_awaited_by(fut, cur_task) + if outer is None or outer.done(): if not fut.cancelled(): # Mark exception retrieved. @@ -824,7 +870,6 @@ def _done_callback(fut): nfuts = 0 nfinished = 0 done_futs = [] - loop = None outer = None # bpo-46672 for arg in coros_or_futures: if arg not in arg_to_fut: @@ -837,12 +882,13 @@ def _done_callback(fut): # can't control it, disable the "destroy pending task" # warning. fut._log_destroy_pending = False - nfuts += 1 arg_to_fut[arg] = fut if fut.done(): done_futs.append(fut) else: + if cur_task is not None: + futures.future_add_to_awaited_by(fut, cur_task) fut.add_done_callback(_done_callback) else: @@ -862,6 +908,25 @@ def _done_callback(fut): return outer +def _log_on_exception(fut): + if fut.cancelled(): + return + + exc = fut.exception() + if exc is None: + return + + context = { + 'message': + f'{exc.__class__.__name__} exception in shielded future', + 'exception': exc, + 'future': fut, + } + if fut._source_traceback: + context['source_traceback'] = fut._source_traceback + fut._loop.call_exception_handler(context) + + def shield(arg): """Wait for a future, shielding it from cancellation. @@ -902,11 +967,16 @@ def shield(arg): loop = futures._get_loop(inner) outer = loop.create_future() + if loop is not None and (cur_task := current_task(loop)) is not None: + futures.future_add_to_awaited_by(inner, cur_task) + else: + cur_task = None + + def _clear_awaited_by_callback(inner): + futures.future_discard_from_awaited_by(inner, cur_task) + def _inner_done_callback(inner): if outer.cancelled(): - if not inner.cancelled(): - # Mark inner's result as retrieved. - inner.exception() return if inner.cancelled(): @@ -918,10 +988,16 @@ def _inner_done_callback(inner): else: outer.set_result(inner.result()) - def _outer_done_callback(outer): if not inner.done(): inner.remove_done_callback(_inner_done_callback) + # Keep only one callback to log on cancel + inner.remove_done_callback(_log_on_exception) + inner.add_done_callback(_log_on_exception) + + if cur_task is not None: + inner.add_done_callback(_clear_awaited_by_callback) + inner.add_done_callback(_inner_done_callback) outer.add_done_callback(_outer_done_callback) @@ -970,9 +1046,9 @@ def create_eager_task_factory(custom_task_constructor): used. E.g. `loop.set_task_factory(asyncio.eager_task_factory)`. """ - def factory(loop, coro, *, name=None, context=None): + def factory(loop, coro, *, eager_start=True, **kwargs): return custom_task_constructor( - coro, loop=loop, name=name, context=context, eager_start=True) + coro, loop=loop, eager_start=eager_start, **kwargs) return factory @@ -1044,14 +1120,13 @@ def _unregister_eager_task(task): _py_enter_task = _enter_task _py_leave_task = _leave_task _py_swap_current_task = _swap_current_task - +_py_all_tasks = all_tasks try: from _asyncio import (_register_task, _register_eager_task, _unregister_task, _unregister_eager_task, _enter_task, _leave_task, _swap_current_task, - _scheduled_tasks, _eager_tasks, _current_tasks, - current_task) + current_task, all_tasks) except ImportError: pass else: @@ -1063,3 +1138,4 @@ def _unregister_eager_task(task): _c_enter_task = _enter_task _c_leave_task = _leave_task _c_swap_current_task = _swap_current_task + _c_all_tasks = all_tasks diff --git a/Lib/asyncio/timeouts.py b/Lib/asyncio/timeouts.py index 30042abb3ad..09342dc7c13 100644 --- a/Lib/asyncio/timeouts.py +++ b/Lib/asyncio/timeouts.py @@ -1,7 +1,6 @@ import enum from types import TracebackType -from typing import final, Optional, Type from . import events from . import exceptions @@ -23,14 +22,13 @@ class _State(enum.Enum): EXITED = "finished" -@final class Timeout: """Asynchronous context manager for cancelling overdue coroutines. Use `timeout()` or `timeout_at()` rather than instantiating this class directly. """ - def __init__(self, when: Optional[float]) -> None: + def __init__(self, when: float | None) -> None: """Schedule a timeout that will trigger at a given loop time. - If `when` is `None`, the timeout will never trigger. @@ -39,15 +37,15 @@ def __init__(self, when: Optional[float]) -> None: """ self._state = _State.CREATED - self._timeout_handler: Optional[events.TimerHandle] = None - self._task: Optional[tasks.Task] = None + self._timeout_handler: events.TimerHandle | None = None + self._task: tasks.Task | None = None self._when = when - def when(self) -> Optional[float]: + def when(self) -> float | None: """Return the current deadline.""" return self._when - def reschedule(self, when: Optional[float]) -> None: + def reschedule(self, when: float | None) -> None: """Reschedule the timeout.""" if self._state is not _State.ENTERED: if self._state is _State.CREATED: @@ -96,10 +94,10 @@ async def __aenter__(self) -> "Timeout": async def __aexit__( self, - exc_type: Optional[Type[BaseException]], - exc_val: Optional[BaseException], - exc_tb: Optional[TracebackType], - ) -> Optional[bool]: + exc_type: type[BaseException] | None, + exc_val: BaseException | None, + exc_tb: TracebackType | None, + ) -> bool | None: assert self._state in (_State.ENTERED, _State.EXPIRING) if self._timeout_handler is not None: @@ -109,10 +107,16 @@ async def __aexit__( if self._state is _State.EXPIRING: self._state = _State.EXPIRED - if self._task.uncancel() <= self._cancelling and exc_type is exceptions.CancelledError: + if self._task.uncancel() <= self._cancelling and exc_type is not None: # Since there are no new cancel requests, we're # handling this. - raise TimeoutError from exc_val + if issubclass(exc_type, exceptions.CancelledError): + raise TimeoutError from exc_val + elif exc_val is not None: + self._insert_timeout_error(exc_val) + if isinstance(exc_val, ExceptionGroup): + for exc in exc_val.exceptions: + self._insert_timeout_error(exc) elif self._state is _State.ENTERED: self._state = _State.EXITED @@ -125,8 +129,18 @@ def _on_timeout(self) -> None: # drop the reference early self._timeout_handler = None + @staticmethod + def _insert_timeout_error(exc_val: BaseException) -> None: + while exc_val.__context__ is not None: + if isinstance(exc_val.__context__, exceptions.CancelledError): + te = TimeoutError() + te.__context__ = te.__cause__ = exc_val.__context__ + exc_val.__context__ = te + break + exc_val = exc_val.__context__ -def timeout(delay: Optional[float]) -> Timeout: + +def timeout(delay: float | None) -> Timeout: """Timeout async context manager. Useful in cases when you want to apply timeout logic around block @@ -146,7 +160,7 @@ def timeout(delay: Optional[float]) -> Timeout: return Timeout(loop.time() + delay if delay is not None else None) -def timeout_at(when: Optional[float]) -> Timeout: +def timeout_at(when: float | None) -> Timeout: """Schedule the timeout at absolute time. Like timeout() but argument gives absolute time in the same clock system diff --git a/Lib/asyncio/tools.py b/Lib/asyncio/tools.py new file mode 100644 index 00000000000..f39e11fdd51 --- /dev/null +++ b/Lib/asyncio/tools.py @@ -0,0 +1,276 @@ +"""Tools to analyze tasks running in asyncio programs.""" + +from collections import defaultdict, namedtuple +from itertools import count +from enum import Enum +import sys +from _remote_debugging import RemoteUnwinder, FrameInfo + +class NodeType(Enum): + COROUTINE = 1 + TASK = 2 + + +class CycleFoundException(Exception): + """Raised when there is a cycle when drawing the call tree.""" + def __init__( + self, + cycles: list[list[int]], + id2name: dict[int, str], + ) -> None: + super().__init__(cycles, id2name) + self.cycles = cycles + self.id2name = id2name + + + +# ─── indexing helpers ─────────────────────────────────────────── +def _format_stack_entry(elem: str|FrameInfo) -> str: + if not isinstance(elem, str): + if elem.lineno == 0 and elem.filename == "": + return f"{elem.funcname}" + else: + return f"{elem.funcname} {elem.filename}:{elem.lineno}" + return elem + + +def _index(result): + id2name, awaits, task_stacks = {}, [], {} + for awaited_info in result: + for task_info in awaited_info.awaited_by: + task_id = task_info.task_id + task_name = task_info.task_name + id2name[task_id] = task_name + + # Store the internal coroutine stack for this task + if task_info.coroutine_stack: + for coro_info in task_info.coroutine_stack: + call_stack = coro_info.call_stack + internal_stack = [_format_stack_entry(frame) for frame in call_stack] + task_stacks[task_id] = internal_stack + + # Add the awaited_by relationships (external dependencies) + if task_info.awaited_by: + for coro_info in task_info.awaited_by: + call_stack = coro_info.call_stack + parent_task_id = coro_info.task_name + stack = [_format_stack_entry(frame) for frame in call_stack] + awaits.append((parent_task_id, stack, task_id)) + return id2name, awaits, task_stacks + + +def _build_tree(id2name, awaits, task_stacks): + id2label = {(NodeType.TASK, tid): name for tid, name in id2name.items()} + children = defaultdict(list) + cor_nodes = defaultdict(dict) # Maps parent -> {frame_name: node_key} + next_cor_id = count(1) + + def get_or_create_cor_node(parent, frame): + """Get existing coroutine node or create new one under parent""" + if frame in cor_nodes[parent]: + return cor_nodes[parent][frame] + + node_key = (NodeType.COROUTINE, f"c{next(next_cor_id)}") + id2label[node_key] = frame + children[parent].append(node_key) + cor_nodes[parent][frame] = node_key + return node_key + + # Build task dependency tree with coroutine frames + for parent_id, stack, child_id in awaits: + cur = (NodeType.TASK, parent_id) + for frame in reversed(stack): + cur = get_or_create_cor_node(cur, frame) + + child_key = (NodeType.TASK, child_id) + if child_key not in children[cur]: + children[cur].append(child_key) + + # Add coroutine stacks for leaf tasks + awaiting_tasks = {parent_id for parent_id, _, _ in awaits} + for task_id in id2name: + if task_id not in awaiting_tasks and task_id in task_stacks: + cur = (NodeType.TASK, task_id) + for frame in reversed(task_stacks[task_id]): + cur = get_or_create_cor_node(cur, frame) + + return id2label, children + + +def _roots(id2label, children): + all_children = {c for kids in children.values() for c in kids} + return [n for n in id2label if n not in all_children] + +# ─── detect cycles in the task-to-task graph ─────────────────────── +def _task_graph(awaits): + """Return {parent_task_id: {child_task_id, …}, …}.""" + g = defaultdict(set) + for parent_id, _stack, child_id in awaits: + g[parent_id].add(child_id) + return g + + +def _find_cycles(graph): + """ + Depth-first search for back-edges. + + Returns a list of cycles (each cycle is a list of task-ids) or an + empty list if the graph is acyclic. + """ + WHITE, GREY, BLACK = 0, 1, 2 + color = defaultdict(lambda: WHITE) + path, cycles = [], [] + + def dfs(v): + color[v] = GREY + path.append(v) + for w in graph.get(v, ()): + if color[w] == WHITE: + dfs(w) + elif color[w] == GREY: # back-edge → cycle! + i = path.index(w) + cycles.append(path[i:] + [w]) # make a copy + color[v] = BLACK + path.pop() + + for v in list(graph): + if color[v] == WHITE: + dfs(v) + return cycles + + +# ─── PRINT TREE FUNCTION ─────────────────────────────────────── +def get_all_awaited_by(pid): + unwinder = RemoteUnwinder(pid) + return unwinder.get_all_awaited_by() + + +def build_async_tree(result, task_emoji="(T)", cor_emoji=""): + """ + Build a list of strings for pretty-print an async call tree. + + The call tree is produced by `get_all_async_stacks()`, prefixing tasks + with `task_emoji` and coroutine frames with `cor_emoji`. + """ + id2name, awaits, task_stacks = _index(result) + g = _task_graph(awaits) + cycles = _find_cycles(g) + if cycles: + raise CycleFoundException(cycles, id2name) + labels, children = _build_tree(id2name, awaits, task_stacks) + + def pretty(node): + flag = task_emoji if node[0] == NodeType.TASK else cor_emoji + return f"{flag} {labels[node]}" + + def render(node, prefix="", last=True, buf=None): + if buf is None: + buf = [] + buf.append(f"{prefix}{'└── ' if last else '├── '}{pretty(node)}") + new_pref = prefix + (" " if last else "│ ") + kids = children.get(node, []) + for i, kid in enumerate(kids): + render(kid, new_pref, i == len(kids) - 1, buf) + return buf + + return [render(root) for root in _roots(labels, children)] + + +def build_task_table(result): + id2name, _, _ = _index(result) + table = [] + + for awaited_info in result: + thread_id = awaited_info.thread_id + for task_info in awaited_info.awaited_by: + # Get task info + task_id = task_info.task_id + task_name = task_info.task_name + + # Build coroutine stack string + frames = [frame for coro in task_info.coroutine_stack + for frame in coro.call_stack] + coro_stack = " -> ".join(_format_stack_entry(x).split(" ")[0] + for x in frames) + + # Handle tasks with no awaiters + if not task_info.awaited_by: + table.append([thread_id, hex(task_id), task_name, coro_stack, + "", "", "0x0"]) + continue + + # Handle tasks with awaiters + for coro_info in task_info.awaited_by: + parent_id = coro_info.task_name + awaiter_frames = [_format_stack_entry(x).split(" ")[0] + for x in coro_info.call_stack] + awaiter_chain = " -> ".join(awaiter_frames) + awaiter_name = id2name.get(parent_id, "Unknown") + parent_id_str = (hex(parent_id) if isinstance(parent_id, int) + else str(parent_id)) + + table.append([thread_id, hex(task_id), task_name, coro_stack, + awaiter_chain, awaiter_name, parent_id_str]) + + return table + +def _print_cycle_exception(exception: CycleFoundException): + print("ERROR: await-graph contains cycles - cannot print a tree!", file=sys.stderr) + print("", file=sys.stderr) + for c in exception.cycles: + inames = " → ".join(exception.id2name.get(tid, hex(tid)) for tid in c) + print(f"cycle: {inames}", file=sys.stderr) + + +def exit_with_permission_help_text(): + """ + Prints a message pointing to platform-specific permission help text and exits the program. + This function is called when a PermissionError is encountered while trying + to attach to a process. + """ + print( + "Error: The specified process cannot be attached to due to insufficient permissions.\n" + "See the Python documentation for details on required privileges and troubleshooting:\n" + "https://docs.python.org/3.14/howto/remote_debugging.html#permission-requirements\n" + ) + sys.exit(1) + + +def _get_awaited_by_tasks(pid: int) -> list: + try: + return get_all_awaited_by(pid) + except RuntimeError as e: + while e.__context__ is not None: + e = e.__context__ + print(f"Error retrieving tasks: {e}") + sys.exit(1) + except PermissionError as e: + exit_with_permission_help_text() + + +def display_awaited_by_tasks_table(pid: int) -> None: + """Build and print a table of all pending tasks under `pid`.""" + + tasks = _get_awaited_by_tasks(pid) + table = build_task_table(tasks) + # Print the table in a simple tabular format + print( + f"{'tid':<10} {'task id':<20} {'task name':<20} {'coroutine stack':<50} {'awaiter chain':<50} {'awaiter name':<15} {'awaiter id':<15}" + ) + print("-" * 180) + for row in table: + print(f"{row[0]:<10} {row[1]:<20} {row[2]:<20} {row[3]:<50} {row[4]:<50} {row[5]:<15} {row[6]:<15}") + + +def display_awaited_by_tasks_tree(pid: int) -> None: + """Build and print a tree of all pending tasks under `pid`.""" + + tasks = _get_awaited_by_tasks(pid) + try: + result = build_async_tree(tasks) + except CycleFoundException as e: + _print_cycle_exception(e) + sys.exit(1) + + for tree in result: + print("\n".join(tree)) diff --git a/Lib/asyncio/transports.py b/Lib/asyncio/transports.py index 30fd41d49af..34c7ad44ffd 100644 --- a/Lib/asyncio/transports.py +++ b/Lib/asyncio/transports.py @@ -181,6 +181,8 @@ def sendto(self, data, addr=None): to be sent out asynchronously. addr is target socket address. If addr is None use target address pointed on transport creation. + If data is an empty bytes object a zero-length datagram will be + sent. """ raise NotImplementedError diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py index f2e920ada46..1c1458127db 100644 --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -28,10 +28,7 @@ __all__ = ( 'SelectorEventLoop', - 'AbstractChildWatcher', 'SafeChildWatcher', - 'FastChildWatcher', 'PidfdChildWatcher', - 'MultiLoopChildWatcher', 'ThreadedChildWatcher', - 'DefaultEventLoopPolicy', + 'EventLoop', ) @@ -63,6 +60,11 @@ class _UnixSelectorEventLoop(selector_events.BaseSelectorEventLoop): def __init__(self, selector=None): super().__init__(selector) self._signal_handlers = {} + self._unix_server_sockets = {} + if can_use_pidfd(): + self._watcher = _PidfdChildWatcher() + else: + self._watcher = _ThreadedChildWatcher() def close(self): super().close() @@ -92,7 +94,7 @@ def add_signal_handler(self, sig, callback, *args): Raise RuntimeError if there is a problem setting up the handler. """ if (coroutines.iscoroutine(callback) or - coroutines.iscoroutinefunction(callback)): + coroutines._iscoroutinefunction(callback)): raise TypeError("coroutines cannot be used " "with add_signal_handler()") self._check_signal(sig) @@ -195,33 +197,22 @@ def _make_write_pipe_transport(self, pipe, protocol, waiter=None, async def _make_subprocess_transport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, extra=None, **kwargs): - with warnings.catch_warnings(): - warnings.simplefilter('ignore', DeprecationWarning) - watcher = events.get_child_watcher() - - with watcher: - if not watcher.is_active(): - # Check early. - # Raising exception before process creation - # prevents subprocess execution if the watcher - # is not ready to handle it. - raise RuntimeError("asyncio.get_child_watcher() is not activated, " - "subprocess support is not installed.") - waiter = self.create_future() - transp = _UnixSubprocessTransport(self, protocol, args, shell, - stdin, stdout, stderr, bufsize, - waiter=waiter, extra=extra, - **kwargs) - watcher.add_child_handler(transp.get_pid(), - self._child_watcher_callback, transp) - try: - await waiter - except (SystemExit, KeyboardInterrupt): - raise - except BaseException: - transp.close() - await transp._wait() - raise + watcher = self._watcher + waiter = self.create_future() + transp = _UnixSubprocessTransport(self, protocol, args, shell, + stdin, stdout, stderr, bufsize, + waiter=waiter, extra=extra, + **kwargs) + watcher.add_child_handler(transp.get_pid(), + self._child_watcher_callback, transp) + try: + await waiter + except (SystemExit, KeyboardInterrupt): + raise + except BaseException: + transp.close() + await transp._wait() + raise return transp @@ -283,7 +274,7 @@ async def create_unix_server( sock=None, backlog=100, ssl=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None, - start_serving=True): + start_serving=True, cleanup_socket=True): if isinstance(ssl, bool): raise TypeError('ssl argument must be an SSLContext or None') @@ -339,6 +330,15 @@ async def create_unix_server( raise ValueError( f'A UNIX Domain Stream Socket was expected, got {sock!r}') + if cleanup_socket: + path = sock.getsockname() + # Check for abstract socket. `str` and `bytes` paths are supported. + if path[0] not in (0, '\x00'): + try: + self._unix_server_sockets[sock] = os.stat(path).st_ino + except FileNotFoundError: + pass + sock.setblocking(False) server = base_events.Server(self, [sock], protocol_factory, ssl, backlog, ssl_handshake_timeout, @@ -393,6 +393,9 @@ def _sock_sendfile_native_impl(self, fut, registered_fd, sock, fileno, fut.set_result(total_sent) return + # On 32-bit architectures truncate to 1GiB to avoid OverflowError + blocksize = min(blocksize, sys.maxsize//2 + 1) + try: sent = os.sendfile(fd, fileno, offset, blocksize) except (BlockingIOError, InterruptedError): @@ -456,6 +459,27 @@ def cb(fut): self.remove_writer(fd) fut.add_done_callback(cb) + def _stop_serving(self, sock): + # Is this a unix socket that needs cleanup? + if sock in self._unix_server_sockets: + path = sock.getsockname() + else: + path = None + + super()._stop_serving(sock) + + if path is not None: + prev_ino = self._unix_server_sockets[sock] + del self._unix_server_sockets[sock] + try: + if os.stat(path).st_ino == prev_ino: + os.unlink(path) + except FileNotFoundError: + pass + except OSError as err: + logger.error('Unable to clean up listening UNIX socket ' + '%r: %r', path, err) + class _UnixReadPipeTransport(transports.ReadTransport): @@ -830,93 +854,7 @@ def _start(self, args, shell, stdin, stdout, stderr, bufsize, **kwargs): stdin_w.close() -class AbstractChildWatcher: - """Abstract base class for monitoring child processes. - - Objects derived from this class monitor a collection of subprocesses and - report their termination or interruption by a signal. - - New callbacks are registered with .add_child_handler(). Starting a new - process must be done within a 'with' block to allow the watcher to suspend - its activity until the new process if fully registered (this is needed to - prevent a race condition in some implementations). - - Example: - with watcher: - proc = subprocess.Popen("sleep 1") - watcher.add_child_handler(proc.pid, callback) - - Notes: - Implementations of this class must be thread-safe. - - Since child watcher objects may catch the SIGCHLD signal and call - waitpid(-1), there should be only one active object per process. - """ - - def __init_subclass__(cls) -> None: - if cls.__module__ != __name__: - warnings._deprecated("AbstractChildWatcher", - "{name!r} is deprecated as of Python 3.12 and will be " - "removed in Python {remove}.", - remove=(3, 14)) - - def add_child_handler(self, pid, callback, *args): - """Register a new child handler. - - Arrange for callback(pid, returncode, *args) to be called when - process 'pid' terminates. Specifying another callback for the same - process replaces the previous handler. - - Note: callback() must be thread-safe. - """ - raise NotImplementedError() - - def remove_child_handler(self, pid): - """Removes the handler for process 'pid'. - - The function returns True if the handler was successfully removed, - False if there was nothing to remove.""" - - raise NotImplementedError() - - def attach_loop(self, loop): - """Attach the watcher to an event loop. - - If the watcher was previously attached to an event loop, then it is - first detached before attaching to the new loop. - - Note: loop may be None. - """ - raise NotImplementedError() - - def close(self): - """Close the watcher. - - This must be called to make sure that any underlying resource is freed. - """ - raise NotImplementedError() - - def is_active(self): - """Return ``True`` if the watcher is active and is used by the event loop. - - Return True if the watcher is installed and ready to handle process exit - notifications. - - """ - raise NotImplementedError() - - def __enter__(self): - """Enter the watcher's context and allow starting new processes - - This function must return self""" - raise NotImplementedError() - - def __exit__(self, a, b, c): - """Exit the watcher's context""" - raise NotImplementedError() - - -class PidfdChildWatcher(AbstractChildWatcher): +class _PidfdChildWatcher: """Child watcher implementation using Linux's pid file descriptors. This child watcher polls process file descriptors (pidfds) to await child @@ -928,21 +866,6 @@ class PidfdChildWatcher(AbstractChildWatcher): recent (5.3+) kernels. """ - def __enter__(self): - return self - - def __exit__(self, exc_type, exc_value, exc_traceback): - pass - - def is_active(self): - return True - - def close(self): - pass - - def attach_loop(self, loop): - pass - def add_child_handler(self, pid, callback, *args): loop = events.get_running_loop() pidfd = os.pidfd_open(pid) @@ -967,386 +890,7 @@ def _do_wait(self, pid, pidfd, callback, args): os.close(pidfd) callback(pid, returncode, *args) - def remove_child_handler(self, pid): - # asyncio never calls remove_child_handler() !!! - # The method is no-op but is implemented because - # abstract base classes require it. - return True - - -class BaseChildWatcher(AbstractChildWatcher): - - def __init__(self): - self._loop = None - self._callbacks = {} - - def close(self): - self.attach_loop(None) - - def is_active(self): - return self._loop is not None and self._loop.is_running() - - def _do_waitpid(self, expected_pid): - raise NotImplementedError() - - def _do_waitpid_all(self): - raise NotImplementedError() - - def attach_loop(self, loop): - assert loop is None or isinstance(loop, events.AbstractEventLoop) - - if self._loop is not None and loop is None and self._callbacks: - warnings.warn( - 'A loop is being detached ' - 'from a child watcher with pending handlers', - RuntimeWarning) - - if self._loop is not None: - self._loop.remove_signal_handler(signal.SIGCHLD) - - self._loop = loop - if loop is not None: - loop.add_signal_handler(signal.SIGCHLD, self._sig_chld) - - # Prevent a race condition in case a child terminated - # during the switch. - self._do_waitpid_all() - - def _sig_chld(self): - try: - self._do_waitpid_all() - except (SystemExit, KeyboardInterrupt): - raise - except BaseException as exc: - # self._loop should always be available here - # as '_sig_chld' is added as a signal handler - # in 'attach_loop' - self._loop.call_exception_handler({ - 'message': 'Unknown exception in SIGCHLD handler', - 'exception': exc, - }) - - -class SafeChildWatcher(BaseChildWatcher): - """'Safe' child watcher implementation. - - This implementation avoids disrupting other code spawning processes by - polling explicitly each process in the SIGCHLD handler instead of calling - os.waitpid(-1). - - This is a safe solution but it has a significant overhead when handling a - big number of children (O(n) each time SIGCHLD is raised) - """ - - def __init__(self): - super().__init__() - warnings._deprecated("SafeChildWatcher", - "{name!r} is deprecated as of Python 3.12 and will be " - "removed in Python {remove}.", - remove=(3, 14)) - - def close(self): - self._callbacks.clear() - super().close() - - def __enter__(self): - return self - - def __exit__(self, a, b, c): - pass - - def add_child_handler(self, pid, callback, *args): - self._callbacks[pid] = (callback, args) - - # Prevent a race condition in case the child is already terminated. - self._do_waitpid(pid) - - def remove_child_handler(self, pid): - try: - del self._callbacks[pid] - return True - except KeyError: - return False - - def _do_waitpid_all(self): - - for pid in list(self._callbacks): - self._do_waitpid(pid) - - def _do_waitpid(self, expected_pid): - assert expected_pid > 0 - - try: - pid, status = os.waitpid(expected_pid, os.WNOHANG) - except ChildProcessError: - # The child process is already reaped - # (may happen if waitpid() is called elsewhere). - pid = expected_pid - returncode = 255 - logger.warning( - "Unknown child process pid %d, will report returncode 255", - pid) - else: - if pid == 0: - # The child process is still alive. - return - - returncode = waitstatus_to_exitcode(status) - if self._loop.get_debug(): - logger.debug('process %s exited with returncode %s', - expected_pid, returncode) - - try: - callback, args = self._callbacks.pop(pid) - except KeyError: # pragma: no cover - # May happen if .remove_child_handler() is called - # after os.waitpid() returns. - if self._loop.get_debug(): - logger.warning("Child watcher got an unexpected pid: %r", - pid, exc_info=True) - else: - callback(pid, returncode, *args) - - -class FastChildWatcher(BaseChildWatcher): - """'Fast' child watcher implementation. - - This implementation reaps every terminated processes by calling - os.waitpid(-1) directly, possibly breaking other code spawning processes - and waiting for their termination. - - There is no noticeable overhead when handling a big number of children - (O(1) each time a child terminates). - """ - def __init__(self): - super().__init__() - self._lock = threading.Lock() - self._zombies = {} - self._forks = 0 - warnings._deprecated("FastChildWatcher", - "{name!r} is deprecated as of Python 3.12 and will be " - "removed in Python {remove}.", - remove=(3, 14)) - - def close(self): - self._callbacks.clear() - self._zombies.clear() - super().close() - - def __enter__(self): - with self._lock: - self._forks += 1 - - return self - - def __exit__(self, a, b, c): - with self._lock: - self._forks -= 1 - - if self._forks or not self._zombies: - return - - collateral_victims = str(self._zombies) - self._zombies.clear() - - logger.warning( - "Caught subprocesses termination from unknown pids: %s", - collateral_victims) - - def add_child_handler(self, pid, callback, *args): - assert self._forks, "Must use the context manager" - - with self._lock: - try: - returncode = self._zombies.pop(pid) - except KeyError: - # The child is running. - self._callbacks[pid] = callback, args - return - - # The child is dead already. We can fire the callback. - callback(pid, returncode, *args) - - def remove_child_handler(self, pid): - try: - del self._callbacks[pid] - return True - except KeyError: - return False - - def _do_waitpid_all(self): - # Because of signal coalescing, we must keep calling waitpid() as - # long as we're able to reap a child. - while True: - try: - pid, status = os.waitpid(-1, os.WNOHANG) - except ChildProcessError: - # No more child processes exist. - return - else: - if pid == 0: - # A child process is still alive. - return - - returncode = waitstatus_to_exitcode(status) - - with self._lock: - try: - callback, args = self._callbacks.pop(pid) - except KeyError: - # unknown child - if self._forks: - # It may not be registered yet. - self._zombies[pid] = returncode - if self._loop.get_debug(): - logger.debug('unknown process %s exited ' - 'with returncode %s', - pid, returncode) - continue - callback = None - else: - if self._loop.get_debug(): - logger.debug('process %s exited with returncode %s', - pid, returncode) - - if callback is None: - logger.warning( - "Caught subprocess termination from unknown pid: " - "%d -> %d", pid, returncode) - else: - callback(pid, returncode, *args) - - -class MultiLoopChildWatcher(AbstractChildWatcher): - """A watcher that doesn't require running loop in the main thread. - - This implementation registers a SIGCHLD signal handler on - instantiation (which may conflict with other code that - install own handler for this signal). - - The solution is safe but it has a significant overhead when - handling a big number of processes (*O(n)* each time a - SIGCHLD is received). - """ - - # Implementation note: - # The class keeps compatibility with AbstractChildWatcher ABC - # To achieve this it has empty attach_loop() method - # and doesn't accept explicit loop argument - # for add_child_handler()/remove_child_handler() - # but retrieves the current loop by get_running_loop() - - def __init__(self): - self._callbacks = {} - self._saved_sighandler = None - warnings._deprecated("MultiLoopChildWatcher", - "{name!r} is deprecated as of Python 3.12 and will be " - "removed in Python {remove}.", - remove=(3, 14)) - - def is_active(self): - return self._saved_sighandler is not None - - def close(self): - self._callbacks.clear() - if self._saved_sighandler is None: - return - - handler = signal.getsignal(signal.SIGCHLD) - if handler != self._sig_chld: - logger.warning("SIGCHLD handler was changed by outside code") - else: - signal.signal(signal.SIGCHLD, self._saved_sighandler) - self._saved_sighandler = None - - def __enter__(self): - return self - - def __exit__(self, exc_type, exc_val, exc_tb): - pass - - def add_child_handler(self, pid, callback, *args): - loop = events.get_running_loop() - self._callbacks[pid] = (loop, callback, args) - - # Prevent a race condition in case the child is already terminated. - self._do_waitpid(pid) - - def remove_child_handler(self, pid): - try: - del self._callbacks[pid] - return True - except KeyError: - return False - - def attach_loop(self, loop): - # Don't save the loop but initialize itself if called first time - # The reason to do it here is that attach_loop() is called from - # unix policy only for the main thread. - # Main thread is required for subscription on SIGCHLD signal - if self._saved_sighandler is not None: - return - - self._saved_sighandler = signal.signal(signal.SIGCHLD, self._sig_chld) - if self._saved_sighandler is None: - logger.warning("Previous SIGCHLD handler was set by non-Python code, " - "restore to default handler on watcher close.") - self._saved_sighandler = signal.SIG_DFL - - # Set SA_RESTART to limit EINTR occurrences. - signal.siginterrupt(signal.SIGCHLD, False) - - def _do_waitpid_all(self): - for pid in list(self._callbacks): - self._do_waitpid(pid) - - def _do_waitpid(self, expected_pid): - assert expected_pid > 0 - - try: - pid, status = os.waitpid(expected_pid, os.WNOHANG) - except ChildProcessError: - # The child process is already reaped - # (may happen if waitpid() is called elsewhere). - pid = expected_pid - returncode = 255 - logger.warning( - "Unknown child process pid %d, will report returncode 255", - pid) - debug_log = False - else: - if pid == 0: - # The child process is still alive. - return - - returncode = waitstatus_to_exitcode(status) - debug_log = True - try: - loop, callback, args = self._callbacks.pop(pid) - except KeyError: # pragma: no cover - # May happen if .remove_child_handler() is called - # after os.waitpid() returns. - logger.warning("Child watcher got an unexpected pid: %r", - pid, exc_info=True) - else: - if loop.is_closed(): - logger.warning("Loop %r that handles pid %r is closed", loop, pid) - else: - if debug_log and loop.get_debug(): - logger.debug('process %s exited with returncode %s', - expected_pid, returncode) - loop.call_soon_threadsafe(callback, pid, returncode, *args) - - def _sig_chld(self, signum, frame): - try: - self._do_waitpid_all() - except (SystemExit, KeyboardInterrupt): - raise - except BaseException: - logger.warning('Unknown exception in SIGCHLD handler', exc_info=True) - - -class ThreadedChildWatcher(AbstractChildWatcher): +class _ThreadedChildWatcher: """Threaded child watcher implementation. The watcher uses a thread per process @@ -1363,18 +907,6 @@ def __init__(self): self._pid_counter = itertools.count(0) self._threads = {} - def is_active(self): - return True - - def close(self): - pass - - def __enter__(self): - return self - - def __exit__(self, exc_type, exc_val, exc_tb): - pass - def __del__(self, _warn=warnings.warn): threads = [thread for thread in list(self._threads.values()) if thread.is_alive()] @@ -1392,15 +924,6 @@ def add_child_handler(self, pid, callback, *args): self._threads[pid] = thread thread.start() - def remove_child_handler(self, pid): - # asyncio never calls remove_child_handler() !!! - # The method is no-op but is implemented because - # abstract base classes require it. - return True - - def attach_loop(self, loop): - pass - def _do_waitpid(self, loop, expected_pid, callback, args): assert expected_pid > 0 @@ -1439,62 +962,11 @@ def can_use_pidfd(): return True -class _UnixDefaultEventLoopPolicy(events.BaseDefaultEventLoopPolicy): - """UNIX event loop policy with a watcher for child processes.""" +class _UnixDefaultEventLoopPolicy(events._BaseDefaultEventLoopPolicy): + """UNIX event loop policy""" _loop_factory = _UnixSelectorEventLoop - def __init__(self): - super().__init__() - self._watcher = None - - def _init_watcher(self): - with events._lock: - if self._watcher is None: # pragma: no branch - if can_use_pidfd(): - self._watcher = PidfdChildWatcher() - else: - self._watcher = ThreadedChildWatcher() - - def set_event_loop(self, loop): - """Set the event loop. - - As a side effect, if a child watcher was set before, then calling - .set_event_loop() from the main thread will call .attach_loop(loop) on - the child watcher. - """ - - super().set_event_loop(loop) - - if (self._watcher is not None and - threading.current_thread() is threading.main_thread()): - self._watcher.attach_loop(loop) - - def get_child_watcher(self): - """Get the watcher for child processes. - - If not yet set, a ThreadedChildWatcher object is automatically created. - """ - if self._watcher is None: - self._init_watcher() - - warnings._deprecated("get_child_watcher", - "{name!r} is deprecated as of Python 3.12 and will be " - "removed in Python {remove}.", remove=(3, 14)) - return self._watcher - - def set_child_watcher(self, watcher): - """Set the watcher for child processes.""" - - assert watcher is None or isinstance(watcher, AbstractChildWatcher) - - if self._watcher is not None: - self._watcher.close() - - self._watcher = watcher - warnings._deprecated("set_child_watcher", - "{name!r} is deprecated as of Python 3.12 and will be " - "removed in Python {remove}.", remove=(3, 14)) - SelectorEventLoop = _UnixSelectorEventLoop -DefaultEventLoopPolicy = _UnixDefaultEventLoopPolicy +_DefaultEventLoopPolicy = _UnixDefaultEventLoopPolicy +EventLoop = SelectorEventLoop diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py index cb613451a58..5f75b17d8ca 100644 --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -29,8 +29,8 @@ __all__ = ( 'SelectorEventLoop', 'ProactorEventLoop', 'IocpProactor', - 'DefaultEventLoopPolicy', 'WindowsSelectorEventLoopPolicy', - 'WindowsProactorEventLoopPolicy', + '_DefaultEventLoopPolicy', '_WindowsSelectorEventLoopPolicy', + '_WindowsProactorEventLoopPolicy', 'EventLoop', ) @@ -315,24 +315,25 @@ def __init__(self, proactor=None): proactor = IocpProactor() super().__init__(proactor) - def run_forever(self): - try: - assert self._self_reading_future is None - self.call_soon(self._loop_self_reading) - super().run_forever() - finally: - if self._self_reading_future is not None: - ov = self._self_reading_future._ov - self._self_reading_future.cancel() - # self_reading_future always uses IOCP, so even though it's - # been cancelled, we need to make sure that the IOCP message - # is received so that the kernel is not holding on to the - # memory, possibly causing memory corruption later. Only - # unregister it if IO is complete in all respects. Otherwise - # we need another _poll() later to complete the IO. - if ov is not None and not ov.pending: - self._proactor._unregister(ov) - self._self_reading_future = None + def _run_forever_setup(self): + assert self._self_reading_future is None + self.call_soon(self._loop_self_reading) + super()._run_forever_setup() + + def _run_forever_cleanup(self): + super()._run_forever_cleanup() + if self._self_reading_future is not None: + ov = self._self_reading_future._ov + self._self_reading_future.cancel() + # self_reading_future always uses IOCP, so even though it's + # been cancelled, we need to make sure that the IOCP message + # is received so that the kernel is not holding on to the + # memory, possibly causing memory corruption later. Only + # unregister it if IO is complete in all respects. Otherwise + # we need another _poll() later to complete the IO. + if ov is not None and not ov.pending: + self._proactor._unregister(ov) + self._self_reading_future = None async def create_pipe_connection(self, protocol_factory, address): f = self._proactor.connect_pipe(address) @@ -890,12 +891,13 @@ def callback(f): SelectorEventLoop = _WindowsSelectorEventLoop -class WindowsSelectorEventLoopPolicy(events.BaseDefaultEventLoopPolicy): +class _WindowsSelectorEventLoopPolicy(events._BaseDefaultEventLoopPolicy): _loop_factory = SelectorEventLoop -class WindowsProactorEventLoopPolicy(events.BaseDefaultEventLoopPolicy): +class _WindowsProactorEventLoopPolicy(events._BaseDefaultEventLoopPolicy): _loop_factory = ProactorEventLoop -DefaultEventLoopPolicy = WindowsProactorEventLoopPolicy +_DefaultEventLoopPolicy = _WindowsProactorEventLoopPolicy +EventLoop = ProactorEventLoop diff --git a/Lib/base64.py b/Lib/base64.py old mode 100755 new mode 100644 index 5a7e790a193..f95132a4274 --- a/Lib/base64.py +++ b/Lib/base64.py @@ -1,12 +1,9 @@ -#! /usr/bin/env python3 - """Base16, Base32, Base64 (RFC 3548), Base85 and Ascii85 data encodings""" # Modified 04-Oct-1995 by Jack Jansen to use binascii module # Modified 30-Dec-2003 by Barry Warsaw to add full RFC 3548 support # Modified 22-May-2007 by Guido van Rossum to use bytes everywhere -import re import struct import binascii @@ -286,7 +283,7 @@ def b16decode(s, casefold=False): s = _bytes_from_decode_data(s) if casefold: s = s.upper() - if re.search(b'[^0-9A-F]', s): + if s.translate(None, delete=b'0123456789ABCDEF'): raise binascii.Error('Non-base16 digit found') return binascii.unhexlify(s) @@ -465,9 +462,12 @@ def b85decode(b): # Delay the initialization of tables to not waste memory # if the function is never called if _b85dec is None: - _b85dec = [None] * 256 + # we don't assign to _b85dec directly to avoid issues when + # multiple threads call this function simultaneously + b85dec_tmp = [None] * 256 for i, c in enumerate(_b85alphabet): - _b85dec[c] = i + b85dec_tmp[c] = i + _b85dec = b85dec_tmp b = _bytes_from_decode_data(b) padding = (-len(b)) % 5 @@ -604,7 +604,14 @@ def main(): with open(args[0], 'rb') as f: func(f, sys.stdout.buffer) else: - func(sys.stdin.buffer, sys.stdout.buffer) + if sys.stdin.isatty(): + # gh-138775: read terminal input data all at once to detect EOF + import io + data = sys.stdin.buffer.read() + buffer = io.BytesIO(data) + else: + buffer = sys.stdin.buffer + func(buffer, sys.stdout.buffer) if __name__ == '__main__': diff --git a/Lib/bz2.py b/Lib/bz2.py index 2420cd01906..eb58f4da596 100644 --- a/Lib/bz2.py +++ b/Lib/bz2.py @@ -10,9 +10,9 @@ __author__ = "Nadeem Vawda " from builtins import open as _builtin_open +from compression._common import _streams import io import os -import _compression from _bz2 import BZ2Compressor, BZ2Decompressor @@ -23,7 +23,7 @@ _MODE_WRITE = 3 -class BZ2File(_compression.BaseStream): +class BZ2File(_streams.BaseStream): """A file object providing transparent bzip2 (de)compression. @@ -88,7 +88,7 @@ def __init__(self, filename, mode="r", *, compresslevel=9): raise TypeError("filename must be a str, bytes, file or PathLike object") if self._mode == _MODE_READ: - raw = _compression.DecompressReader(self._fp, + raw = _streams.DecompressReader(self._fp, BZ2Decompressor, trailing_error=OSError) self._buffer = io.BufferedReader(raw) else: @@ -248,7 +248,7 @@ def writelines(self, seq): Line separators are not added between the written byte strings. """ - return _compression.BaseStream.writelines(self, seq) + return _streams.BaseStream.writelines(self, seq) def seek(self, offset, whence=io.SEEK_SET): """Change the file position. diff --git a/Lib/calendar.py b/Lib/calendar.py index 8c1c646da46..18f76d52ff8 100644 --- a/Lib/calendar.py +++ b/Lib/calendar.py @@ -428,6 +428,7 @@ def formatyear(self, theyear, w=2, l=1, c=6, m=3): headers = (header for k in months) a(formatstring(headers, colwidth, c).rstrip()) a('\n'*l) + # max number of weeks for this row height = max(len(cal) for cal in row) for j in range(height): @@ -646,6 +647,117 @@ def formatmonthname(self, theyear, themonth, withyear=True): with different_locale(self.locale): return super().formatmonthname(theyear, themonth, withyear) + +class _CLIDemoCalendar(TextCalendar): + def __init__(self, highlight_day=None, *args, **kwargs): + super().__init__(*args, **kwargs) + self.highlight_day = highlight_day + + def formatweek(self, theweek, width, *, highlight_day=None): + """ + Returns a single week in a string (no newline). + """ + if highlight_day: + from _colorize import get_colors + + ansi = get_colors() + highlight = f"{ansi.BLACK}{ansi.BACKGROUND_YELLOW}" + reset = ansi.RESET + else: + highlight = reset = "" + + return ' '.join( + ( + f"{highlight}{self.formatday(d, wd, width)}{reset}" + if d == highlight_day + else self.formatday(d, wd, width) + ) + for (d, wd) in theweek + ) + + def formatmonth(self, theyear, themonth, w=0, l=0): + """ + Return a month's calendar string (multi-line). + """ + if ( + self.highlight_day + and self.highlight_day.year == theyear + and self.highlight_day.month == themonth + ): + highlight_day = self.highlight_day.day + else: + highlight_day = None + w = max(2, w) + l = max(1, l) + s = self.formatmonthname(theyear, themonth, 7 * (w + 1) - 1) + s = s.rstrip() + s += '\n' * l + s += self.formatweekheader(w).rstrip() + s += '\n' * l + for week in self.monthdays2calendar(theyear, themonth): + s += self.formatweek(week, w, highlight_day=highlight_day).rstrip() + s += '\n' * l + return s + + def formatyear(self, theyear, w=2, l=1, c=6, m=3): + """ + Returns a year's calendar as a multi-line string. + """ + w = max(2, w) + l = max(1, l) + c = max(2, c) + colwidth = (w + 1) * 7 - 1 + v = [] + a = v.append + a(repr(theyear).center(colwidth*m+c*(m-1)).rstrip()) + a('\n'*l) + header = self.formatweekheader(w) + for (i, row) in enumerate(self.yeardays2calendar(theyear, m)): + # months in this row + months = range(m*i+1, min(m*(i+1)+1, 13)) + a('\n'*l) + names = (self.formatmonthname(theyear, k, colwidth, False) + for k in months) + a(formatstring(names, colwidth, c).rstrip()) + a('\n'*l) + headers = (header for k in months) + a(formatstring(headers, colwidth, c).rstrip()) + a('\n'*l) + + if ( + self.highlight_day + and self.highlight_day.year == theyear + and self.highlight_day.month in months + ): + month_pos = months.index(self.highlight_day.month) + else: + month_pos = None + + # max number of weeks for this row + height = max(len(cal) for cal in row) + for j in range(height): + weeks = [] + for k, cal in enumerate(row): + if j >= len(cal): + weeks.append('') + else: + day = ( + self.highlight_day.day if k == month_pos else None + ) + weeks.append( + self.formatweek(cal[j], w, highlight_day=day) + ) + a(formatstring(weeks, colwidth, c).rstrip()) + a('\n' * l) + return ''.join(v) + + +class _CLIDemoLocaleCalendar(LocaleTextCalendar, _CLIDemoCalendar): + def __init__(self, highlight_day=None, *args, **kwargs): + super().__init__(*args, **kwargs) + self.highlight_day = highlight_day + + # Support for old module level interface c = TextCalendar() @@ -698,7 +810,7 @@ def timegm(tuple): def main(args=None): import argparse - parser = argparse.ArgumentParser() + parser = argparse.ArgumentParser(color=True) textgroup = parser.add_argument_group('text only arguments') htmlgroup = parser.add_argument_group('html only arguments') textgroup.add_argument( @@ -765,6 +877,7 @@ def main(args=None): sys.exit(1) locale = options.locale, options.encoding + today = datetime.date.today() if options.type == "html": if options.month: @@ -781,23 +894,23 @@ def main(args=None): optdict = dict(encoding=encoding, css=options.css) write = sys.stdout.buffer.write if options.year is None: - write(cal.formatyearpage(datetime.date.today().year, **optdict)) + write(cal.formatyearpage(today.year, **optdict)) else: write(cal.formatyearpage(options.year, **optdict)) else: if options.locale: - cal = LocaleTextCalendar(locale=locale) + cal = _CLIDemoLocaleCalendar(highlight_day=today, locale=locale) else: - cal = TextCalendar() + cal = _CLIDemoCalendar(highlight_day=today) cal.setfirstweekday(options.first_weekday) optdict = dict(w=options.width, l=options.lines) if options.month is None: optdict["c"] = options.spacing optdict["m"] = options.months - if options.month is not None: + else: _validate_month(options.month) if options.year is None: - result = cal.formatyear(datetime.date.today().year, **optdict) + result = cal.formatyear(today.year, **optdict) elif options.month is None: result = cal.formatyear(options.year, **optdict) else: diff --git a/Lib/cmd.py b/Lib/cmd.py index a37d16cd7bd..51495fb3216 100644 --- a/Lib/cmd.py +++ b/Lib/cmd.py @@ -5,16 +5,16 @@ 1. End of file on input is processed as the command 'EOF'. 2. A command is parsed out of each line by collecting the prefix composed of characters in the identchars member. -3. A command `foo' is dispatched to a method 'do_foo()'; the do_ method +3. A command 'foo' is dispatched to a method 'do_foo()'; the do_ method is passed a single argument consisting of the remainder of the line. 4. Typing an empty line repeats the last command. (Actually, it calls the - method `emptyline', which may be overridden in a subclass.) -5. There is a predefined `help' method. Given an argument `topic', it - calls the command `help_topic'. With no arguments, it lists all topics + method 'emptyline', which may be overridden in a subclass.) +5. There is a predefined 'help' method. Given an argument 'topic', it + calls the command 'help_topic'. With no arguments, it lists all topics with defined help_ functions, broken into up to three topics; documented commands, miscellaneous help topics, and undocumented commands. -6. The command '?' is a synonym for `help'. The command '!' is a synonym - for `shell', if a do_shell method exists. +6. The command '?' is a synonym for 'help'. The command '!' is a synonym + for 'shell', if a do_shell method exists. 7. If completion is enabled, completing commands will be done automatically, and completing of commands args is done by calling complete_foo() with arguments text, line, begidx, endidx. text is string we are matching @@ -23,31 +23,34 @@ indexes of the text being matched, which could be used to provide different completion depending upon which position the argument is in. -The `default' method may be overridden to intercept commands for which there +The 'default' method may be overridden to intercept commands for which there is no do_ method. -The `completedefault' method may be overridden to intercept completions for +The 'completedefault' method may be overridden to intercept completions for commands that have no complete_ method. -The data member `self.ruler' sets the character used to draw separator lines +The data member 'self.ruler' sets the character used to draw separator lines in the help messages. If empty, no ruler line is drawn. It defaults to "=". -If the value of `self.intro' is nonempty when the cmdloop method is called, +If the value of 'self.intro' is nonempty when the cmdloop method is called, it is printed out on interpreter startup. This value may be overridden via an optional argument to the cmdloop() method. -The data members `self.doc_header', `self.misc_header', and -`self.undoc_header' set the headers used for the help function's +The data members 'self.doc_header', 'self.misc_header', and +'self.undoc_header' set the headers used for the help function's listings of documented functions, miscellaneous topics, and undocumented functions respectively. """ -import inspect, string, sys +import sys __all__ = ["Cmd"] PROMPT = '(Cmd) ' -IDENTCHARS = string.ascii_letters + string.digits + '_' +IDENTCHARS = ('ABCDEFGHIJKLMNOPQRSTUVWXYZ' + 'abcdefghijklmnopqrstuvwxyz' + '0123456789' + '_') class Cmd: """A simple framework for writing line-oriented command interpreters. @@ -270,7 +273,7 @@ def complete(self, text, state): endidx = readline.get_endidx() - stripped if begidx>0: cmd, args, foo = self.parseline(line) - if cmd == '': + if not cmd: compfunc = self.completedefault else: try: @@ -303,9 +306,11 @@ def do_help(self, arg): try: func = getattr(self, 'help_' + arg) except AttributeError: + from inspect import cleandoc + try: doc=getattr(self, 'do_' + arg).__doc__ - doc = inspect.cleandoc(doc) + doc = cleandoc(doc) if doc: self.stdout.write("%s\n"%str(doc)) return diff --git a/Lib/code.py b/Lib/code.py index 2777c311187..b134886dc26 100644 --- a/Lib/code.py +++ b/Lib/code.py @@ -13,7 +13,6 @@ __all__ = ["InteractiveInterpreter", "InteractiveConsole", "interact", "compile_command"] - class InteractiveInterpreter: """Base class for InteractiveConsole. @@ -126,7 +125,7 @@ def showtraceback(self): """ try: typ, value, tb = sys.exc_info() - self._showtraceback(typ, value, tb.tb_next, '') + self._showtraceback(typ, value, tb.tb_next, "") finally: typ = value = tb = None @@ -140,7 +139,7 @@ def _showtraceback(self, typ, value, tb, source): and not value.text and value.lineno is not None and len(lines) >= value.lineno): value.text = lines[value.lineno - 1] - sys.last_exc = sys.last_value = value = value.with_traceback(tb) + sys.last_exc = sys.last_value = value if sys.excepthook is sys.__excepthook__: self._excepthook(typ, value, tb) else: @@ -220,12 +219,17 @@ def interact(self, banner=None, exitmsg=None): """ try: sys.ps1 + delete_ps1_after = False except AttributeError: sys.ps1 = ">>> " + delete_ps1_after = True try: - sys.ps2 + _ps2 = sys.ps2 + delete_ps2_after = False except AttributeError: sys.ps2 = "... " + delete_ps2_after = True + cprt = 'Type "help", "copyright", "credits" or "license" for more information.' if banner is None: self.write("Python %s on %s\n%s\n(%s)\n" % @@ -288,6 +292,12 @@ def interact(self, banner=None, exitmsg=None): if _quit is not None: builtins.quit = _quit + if delete_ps1_after: + del sys.ps1 + + if delete_ps2_after: + del sys.ps2 + if exitmsg is None: self.write('now exiting %s...\n' % self.__class__.__name__) elif exitmsg != '': @@ -366,7 +376,7 @@ def interact(banner=None, readfunc=None, local=None, exitmsg=None, local_exit=Fa console.raw_input = readfunc else: try: - import readline + import readline # noqa: F401 except ImportError: pass console.interact(banner, exitmsg) @@ -375,9 +385,9 @@ def interact(banner=None, readfunc=None, local=None, exitmsg=None, local_exit=Fa if __name__ == "__main__": import argparse - parser = argparse.ArgumentParser() + parser = argparse.ArgumentParser(color=True) parser.add_argument('-q', action='store_true', - help="don't print version and copyright messages") + help="don't print version and copyright messages") args = parser.parse_args() if args.q or sys.flags.quiet: banner = '' diff --git a/Lib/codecs.py b/Lib/codecs.py index e4f4e1b5c02..e4a8010aba9 100644 --- a/Lib/codecs.py +++ b/Lib/codecs.py @@ -884,7 +884,6 @@ def __reduce_ex__(self, proto): ### Shortcuts def open(filename, mode='r', encoding=None, errors='strict', buffering=-1): - """ Open an encoded file using the given mode and return a wrapped version providing transparent encoding/decoding. @@ -912,8 +911,11 @@ def open(filename, mode='r', encoding=None, errors='strict', buffering=-1): .encoding which allows querying the used encoding. This attribute is only available if an encoding was specified as parameter. - """ + import warnings + warnings.warn("codecs.open() is deprecated. Use open() instead.", + DeprecationWarning, stacklevel=2) + if encoding is not None and \ 'b' not in mode: # Force opening of the file in binary mode @@ -1109,24 +1111,15 @@ def make_encoding_map(decoding_map): ### error handlers -try: - strict_errors = lookup_error("strict") - ignore_errors = lookup_error("ignore") - replace_errors = lookup_error("replace") - xmlcharrefreplace_errors = lookup_error("xmlcharrefreplace") - backslashreplace_errors = lookup_error("backslashreplace") - namereplace_errors = lookup_error("namereplace") -except LookupError: - # In --disable-unicode builds, these error handler are missing - strict_errors = None - ignore_errors = None - replace_errors = None - xmlcharrefreplace_errors = None - backslashreplace_errors = None - namereplace_errors = None +strict_errors = lookup_error("strict") +ignore_errors = lookup_error("ignore") +replace_errors = lookup_error("replace") +xmlcharrefreplace_errors = lookup_error("xmlcharrefreplace") +backslashreplace_errors = lookup_error("backslashreplace") +namereplace_errors = lookup_error("namereplace") # Tell modulefinder that using codecs probably needs the encodings # package _false = 0 if _false: - import encodings + import encodings # noqa: F401 diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py index f7348ee918d..db36248c4ff 100644 --- a/Lib/collections/__init__.py +++ b/Lib/collections/__init__.py @@ -29,6 +29,9 @@ import _collections_abc import sys as _sys +_sys.modules['collections.abc'] = _collections_abc +abc = _collections_abc + from itertools import chain as _chain from itertools import repeat as _repeat from itertools import starmap as _starmap @@ -46,7 +49,8 @@ _collections_abc.MutableSequence.register(deque) try: - from _collections import _deque_iterator + # Expose _deque_iterator to support pickling deque iterators + from _collections import _deque_iterator # noqa: F401 except ImportError: pass @@ -59,6 +63,8 @@ # with this instead. from ._defaultdict import defaultdict +heapq = None # Lazily imported + ################################################################################ ### OrderedDict @@ -461,7 +467,7 @@ def _make(cls, iterable): def _replace(self, /, **kwds): result = self._make(_map(kwds.pop, field_names, self)) if kwds: - raise ValueError(f'Got unexpected field names: {list(kwds)!r}') + raise TypeError(f'Got unexpected field names: {list(kwds)!r}') return result _replace.__doc__ = (f'Return a new {typename} object replacing specified ' @@ -499,6 +505,7 @@ def __getnewargs__(self): '_field_defaults': field_defaults, '__new__': __new__, '_make': _make, + '__replace__': _replace, '_replace': _replace, '__repr__': __repr__, '_asdict': _asdict, @@ -592,7 +599,7 @@ class Counter(dict): # References: # http://en.wikipedia.org/wiki/Multiset # http://www.gnu.org/software/smalltalk/manual-base/html_node/Bag.html - # http://www.demo2s.com/Tutorial/Cpp/0380__set-multiset/Catalog0380__set-multiset.htm + # http://www.java2s.com/Tutorial/Cpp/0380__set-multiset/Catalog0380__set-multiset.htm # http://code.activestate.com/recipes/259174/ # Knuth, TAOCP Vol. II section 4.6.3 @@ -632,7 +639,10 @@ def most_common(self, n=None): return sorted(self.items(), key=_itemgetter(1), reverse=True) # Lazy import to speedup Python startup time - import heapq + global heapq + if heapq is None: + import heapq + return heapq.nlargest(n, self.items(), key=_itemgetter(1)) def elements(self): @@ -642,7 +652,8 @@ def elements(self): >>> sorted(c.elements()) ['A', 'A', 'B', 'B', 'C', 'C'] - # Knuth's example for prime factors of 1836: 2**2 * 3**3 * 17**1 + Knuth's example for prime factors of 1836: 2**2 * 3**3 * 17**1 + >>> import math >>> prime_factors = Counter({2: 2, 3: 3, 17: 1}) >>> math.prod(prime_factors.elements()) @@ -683,7 +694,7 @@ def update(self, iterable=None, /, **kwds): ''' # The regular dict.update() operation makes no sense here because the - # replace behavior results in the some of original untouched counts + # replace behavior results in some of the original untouched counts # being mixed-in with all of the other counts for a mismash that # doesn't have a straight-forward interpretation in most counting # contexts. Instead, we implement straight-addition. Both the inputs @@ -1018,7 +1029,7 @@ def __getitem__(self, key): return self.__missing__(key) # support subclasses that define __missing__ def get(self, key, default=None): - return self[key] if key in self else default + return self[key] if key in self else default # needs to make use of __contains__ def __len__(self): return len(set().union(*self.maps)) # reuses stored hash values if possible @@ -1030,7 +1041,10 @@ def __iter__(self): return iter(d) def __contains__(self, key): - return any(key in m for m in self.maps) + for mapping in self.maps: + if key in mapping: + return True + return False def __bool__(self): return any(self.maps) @@ -1040,9 +1054,9 @@ def __repr__(self): return f'{self.__class__.__name__}({", ".join(map(repr, self.maps))})' @classmethod - def fromkeys(cls, iterable, *args): - 'Create a ChainMap with a single dict created from the iterable.' - return cls(dict.fromkeys(iterable, *args)) + def fromkeys(cls, iterable, value=None, /): + 'Create a new ChainMap with keys from iterable and values set to value.' + return cls(dict.fromkeys(iterable, value)) def copy(self): 'New ChainMap or subclass with a new copy of maps[0] and refs to maps[1:]' @@ -1485,6 +1499,8 @@ def format_map(self, mapping): return self.data.format_map(mapping) def index(self, sub, start=0, end=_sys.maxsize): + if isinstance(sub, UserString): + sub = sub.data return self.data.index(sub, start, end) def isalpha(self): @@ -1553,6 +1569,8 @@ def rfind(self, sub, start=0, end=_sys.maxsize): return self.data.rfind(sub, start, end) def rindex(self, sub, start=0, end=_sys.maxsize): + if isinstance(sub, UserString): + sub = sub.data return self.data.rindex(sub, start, end) def rjust(self, width, *args): diff --git a/Lib/collections/abc.py b/Lib/collections/abc.py deleted file mode 100644 index 86ca8b8a841..00000000000 --- a/Lib/collections/abc.py +++ /dev/null @@ -1,3 +0,0 @@ -from _collections_abc import * -from _collections_abc import __all__ -from _collections_abc import _CallableGenericAlias diff --git a/Lib/compression/__init__.py b/Lib/compression/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/Lib/compression/_common/__init__.py b/Lib/compression/_common/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/Lib/_compression.py b/Lib/compression/_common/_streams.py similarity index 98% rename from Lib/_compression.py rename to Lib/compression/_common/_streams.py index e8b70aa0a3e..9f367d4e304 100644 --- a/Lib/_compression.py +++ b/Lib/compression/_common/_streams.py @@ -1,4 +1,4 @@ -"""Internal classes used by the gzip, lzma and bz2 modules""" +"""Internal classes used by compression modules""" import io import sys diff --git a/Lib/compression/bz2.py b/Lib/compression/bz2.py new file mode 100644 index 00000000000..16815d6cd20 --- /dev/null +++ b/Lib/compression/bz2.py @@ -0,0 +1,5 @@ +import bz2 +__doc__ = bz2.__doc__ +del bz2 + +from bz2 import * diff --git a/Lib/compression/gzip.py b/Lib/compression/gzip.py new file mode 100644 index 00000000000..552f48f948a --- /dev/null +++ b/Lib/compression/gzip.py @@ -0,0 +1,5 @@ +import gzip +__doc__ = gzip.__doc__ +del gzip + +from gzip import * diff --git a/Lib/compression/lzma.py b/Lib/compression/lzma.py new file mode 100644 index 00000000000..b4bc7ccb1db --- /dev/null +++ b/Lib/compression/lzma.py @@ -0,0 +1,5 @@ +import lzma +__doc__ = lzma.__doc__ +del lzma + +from lzma import * diff --git a/Lib/compression/zlib.py b/Lib/compression/zlib.py new file mode 100644 index 00000000000..3aa7e2db90e --- /dev/null +++ b/Lib/compression/zlib.py @@ -0,0 +1,5 @@ +import zlib +__doc__ = zlib.__doc__ +del zlib + +from zlib import * diff --git a/Lib/compression/zstd/__init__.py b/Lib/compression/zstd/__init__.py new file mode 100644 index 00000000000..84b25914b0a --- /dev/null +++ b/Lib/compression/zstd/__init__.py @@ -0,0 +1,242 @@ +"""Python bindings to the Zstandard (zstd) compression library (RFC-8878).""" + +__all__ = ( + # compression.zstd + 'COMPRESSION_LEVEL_DEFAULT', + 'compress', + 'CompressionParameter', + 'decompress', + 'DecompressionParameter', + 'finalize_dict', + 'get_frame_info', + 'Strategy', + 'train_dict', + + # compression.zstd._zstdfile + 'open', + 'ZstdFile', + + # _zstd + 'get_frame_size', + 'zstd_version', + 'zstd_version_info', + 'ZstdCompressor', + 'ZstdDecompressor', + 'ZstdDict', + 'ZstdError', +) + +import _zstd +import enum +from _zstd import (ZstdCompressor, ZstdDecompressor, ZstdDict, ZstdError, + get_frame_size, zstd_version) +from compression.zstd._zstdfile import ZstdFile, open, _nbytes + +# zstd_version_number is (MAJOR * 100 * 100 + MINOR * 100 + RELEASE) +zstd_version_info = (*divmod(_zstd.zstd_version_number // 100, 100), + _zstd.zstd_version_number % 100) +"""Version number of the runtime zstd library as a tuple of integers.""" + +COMPRESSION_LEVEL_DEFAULT = _zstd.ZSTD_CLEVEL_DEFAULT +"""The default compression level for Zstandard, currently '3'.""" + + +class FrameInfo: + """Information about a Zstandard frame.""" + + __slots__ = 'decompressed_size', 'dictionary_id' + + def __init__(self, decompressed_size, dictionary_id): + super().__setattr__('decompressed_size', decompressed_size) + super().__setattr__('dictionary_id', dictionary_id) + + def __repr__(self): + return (f'FrameInfo(decompressed_size={self.decompressed_size}, ' + f'dictionary_id={self.dictionary_id})') + + def __setattr__(self, name, _): + raise AttributeError(f"can't set attribute {name!r}") + + +def get_frame_info(frame_buffer): + """Get Zstandard frame information from a frame header. + + *frame_buffer* is a bytes-like object. It should start from the beginning + of a frame, and needs to include at least the frame header (6 to 18 bytes). + + The returned FrameInfo object has two attributes. + 'decompressed_size' is the size in bytes of the data in the frame when + decompressed, or None when the decompressed size is unknown. + 'dictionary_id' is an int in the range (0, 2**32). The special value 0 + means that the dictionary ID was not recorded in the frame header, + the frame may or may not need a dictionary to be decoded, + and the ID of such a dictionary is not specified. + """ + return FrameInfo(*_zstd.get_frame_info(frame_buffer)) + + +def train_dict(samples, dict_size): + """Return a ZstdDict representing a trained Zstandard dictionary. + + *samples* is an iterable of samples, where a sample is a bytes-like + object representing a file. + + *dict_size* is the dictionary's maximum size, in bytes. + """ + if not isinstance(dict_size, int): + ds_cls = type(dict_size).__qualname__ + raise TypeError(f'dict_size must be an int object, not {ds_cls!r}.') + + samples = tuple(samples) + chunks = b''.join(samples) + chunk_sizes = tuple(_nbytes(sample) for sample in samples) + if not chunks: + raise ValueError("samples contained no data; can't train dictionary.") + dict_content = _zstd.train_dict(chunks, chunk_sizes, dict_size) + return ZstdDict(dict_content) + + +def finalize_dict(zstd_dict, /, samples, dict_size, level): + """Return a ZstdDict representing a finalized Zstandard dictionary. + + Given a custom content as a basis for dictionary, and a set of samples, + finalize *zstd_dict* by adding headers and statistics according to the + Zstandard dictionary format. + + You may compose an effective dictionary content by hand, which is used as + basis dictionary, and use some samples to finalize a dictionary. The basis + dictionary may be a "raw content" dictionary. See *is_raw* in ZstdDict. + + *samples* is an iterable of samples, where a sample is a bytes-like object + representing a file. + *dict_size* is the dictionary's maximum size, in bytes. + *level* is the expected compression level. The statistics for each + compression level differ, so tuning the dictionary to the compression level + can provide improvements. + """ + + if not isinstance(zstd_dict, ZstdDict): + raise TypeError('zstd_dict argument should be a ZstdDict object.') + if not isinstance(dict_size, int): + raise TypeError('dict_size argument should be an int object.') + if not isinstance(level, int): + raise TypeError('level argument should be an int object.') + + samples = tuple(samples) + chunks = b''.join(samples) + chunk_sizes = tuple(_nbytes(sample) for sample in samples) + if not chunks: + raise ValueError("The samples are empty content, can't finalize the " + "dictionary.") + dict_content = _zstd.finalize_dict(zstd_dict.dict_content, chunks, + chunk_sizes, dict_size, level) + return ZstdDict(dict_content) + + +def compress(data, level=None, options=None, zstd_dict=None): + """Return Zstandard compressed *data* as bytes. + + *level* is an int specifying the compression level to use, defaulting to + COMPRESSION_LEVEL_DEFAULT ('3'). + *options* is a dict object that contains advanced compression + parameters. See CompressionParameter for more on options. + *zstd_dict* is a ZstdDict object, a pre-trained Zstandard dictionary. See + the function train_dict for how to train a ZstdDict on sample data. + + For incremental compression, use a ZstdCompressor instead. + """ + comp = ZstdCompressor(level=level, options=options, zstd_dict=zstd_dict) + return comp.compress(data, mode=ZstdCompressor.FLUSH_FRAME) + + +def decompress(data, zstd_dict=None, options=None): + """Decompress one or more frames of Zstandard compressed *data*. + + *zstd_dict* is a ZstdDict object, a pre-trained Zstandard dictionary. See + the function train_dict for how to train a ZstdDict on sample data. + *options* is a dict object that contains advanced compression + parameters. See DecompressionParameter for more on options. + + For incremental decompression, use a ZstdDecompressor instead. + """ + results = [] + while True: + decomp = ZstdDecompressor(options=options, zstd_dict=zstd_dict) + results.append(decomp.decompress(data)) + if not decomp.eof: + raise ZstdError('Compressed data ended before the ' + 'end-of-stream marker was reached') + data = decomp.unused_data + if not data: + break + return b''.join(results) + + +class CompressionParameter(enum.IntEnum): + """Compression parameters.""" + + compression_level = _zstd.ZSTD_c_compressionLevel + window_log = _zstd.ZSTD_c_windowLog + hash_log = _zstd.ZSTD_c_hashLog + chain_log = _zstd.ZSTD_c_chainLog + search_log = _zstd.ZSTD_c_searchLog + min_match = _zstd.ZSTD_c_minMatch + target_length = _zstd.ZSTD_c_targetLength + strategy = _zstd.ZSTD_c_strategy + + enable_long_distance_matching = _zstd.ZSTD_c_enableLongDistanceMatching + ldm_hash_log = _zstd.ZSTD_c_ldmHashLog + ldm_min_match = _zstd.ZSTD_c_ldmMinMatch + ldm_bucket_size_log = _zstd.ZSTD_c_ldmBucketSizeLog + ldm_hash_rate_log = _zstd.ZSTD_c_ldmHashRateLog + + content_size_flag = _zstd.ZSTD_c_contentSizeFlag + checksum_flag = _zstd.ZSTD_c_checksumFlag + dict_id_flag = _zstd.ZSTD_c_dictIDFlag + + nb_workers = _zstd.ZSTD_c_nbWorkers + job_size = _zstd.ZSTD_c_jobSize + overlap_log = _zstd.ZSTD_c_overlapLog + + def bounds(self): + """Return the (lower, upper) int bounds of a compression parameter. + + Both the lower and upper bounds are inclusive. + """ + return _zstd.get_param_bounds(self.value, is_compress=True) + + +class DecompressionParameter(enum.IntEnum): + """Decompression parameters.""" + + window_log_max = _zstd.ZSTD_d_windowLogMax + + def bounds(self): + """Return the (lower, upper) int bounds of a decompression parameter. + + Both the lower and upper bounds are inclusive. + """ + return _zstd.get_param_bounds(self.value, is_compress=False) + + +class Strategy(enum.IntEnum): + """Compression strategies, listed from fastest to strongest. + + Note that new strategies might be added in the future. + Only the order (from fast to strong) is guaranteed, + the numeric value might change. + """ + + fast = _zstd.ZSTD_fast + dfast = _zstd.ZSTD_dfast + greedy = _zstd.ZSTD_greedy + lazy = _zstd.ZSTD_lazy + lazy2 = _zstd.ZSTD_lazy2 + btlazy2 = _zstd.ZSTD_btlazy2 + btopt = _zstd.ZSTD_btopt + btultra = _zstd.ZSTD_btultra + btultra2 = _zstd.ZSTD_btultra2 + + +# Check validity of the CompressionParameter & DecompressionParameter types +_zstd.set_parameter_types(CompressionParameter, DecompressionParameter) diff --git a/Lib/compression/zstd/_zstdfile.py b/Lib/compression/zstd/_zstdfile.py new file mode 100644 index 00000000000..d709f5efc65 --- /dev/null +++ b/Lib/compression/zstd/_zstdfile.py @@ -0,0 +1,345 @@ +import io +from os import PathLike +from _zstd import ZstdCompressor, ZstdDecompressor, ZSTD_DStreamOutSize +from compression._common import _streams + +__all__ = ('ZstdFile', 'open') + +_MODE_CLOSED = 0 +_MODE_READ = 1 +_MODE_WRITE = 2 + + +def _nbytes(dat, /): + if isinstance(dat, (bytes, bytearray)): + return len(dat) + with memoryview(dat) as mv: + return mv.nbytes + + +class ZstdFile(_streams.BaseStream): + """A file-like object providing transparent Zstandard (de)compression. + + A ZstdFile can act as a wrapper for an existing file object, or refer + directly to a named file on disk. + + ZstdFile provides a *binary* file interface. Data is read and returned as + bytes, and may only be written to objects that support the Buffer Protocol. + """ + + FLUSH_BLOCK = ZstdCompressor.FLUSH_BLOCK + FLUSH_FRAME = ZstdCompressor.FLUSH_FRAME + + def __init__(self, file, /, mode='r', *, + level=None, options=None, zstd_dict=None): + """Open a Zstandard compressed file in binary mode. + + *file* can be either an file-like object, or a file name to open. + + *mode* can be 'r' for reading (default), 'w' for (over)writing, 'x' for + creating exclusively, or 'a' for appending. These can equivalently be + given as 'rb', 'wb', 'xb' and 'ab' respectively. + + *level* is an optional int specifying the compression level to use, + or COMPRESSION_LEVEL_DEFAULT if not given. + + *options* is an optional dict for advanced compression parameters. + See CompressionParameter and DecompressionParameter for the possible + options. + + *zstd_dict* is an optional ZstdDict object, a pre-trained Zstandard + dictionary. See train_dict() to train ZstdDict on sample data. + """ + self._fp = None + self._close_fp = False + self._mode = _MODE_CLOSED + self._buffer = None + + if not isinstance(mode, str): + raise ValueError('mode must be a str') + if options is not None and not isinstance(options, dict): + raise TypeError('options must be a dict or None') + mode = mode.removesuffix('b') # handle rb, wb, xb, ab + if mode == 'r': + if level is not None: + raise TypeError('level is illegal in read mode') + self._mode = _MODE_READ + elif mode in {'w', 'a', 'x'}: + if level is not None and not isinstance(level, int): + raise TypeError('level must be int or None') + self._mode = _MODE_WRITE + self._compressor = ZstdCompressor(level=level, options=options, + zstd_dict=zstd_dict) + self._pos = 0 + else: + raise ValueError(f'Invalid mode: {mode!r}') + + if isinstance(file, (str, bytes, PathLike)): + self._fp = io.open(file, f'{mode}b') + self._close_fp = True + elif ((mode == 'r' and hasattr(file, 'read')) + or (mode != 'r' and hasattr(file, 'write'))): + self._fp = file + else: + raise TypeError('file must be a file-like object ' + 'or a str, bytes, or PathLike object') + + if self._mode == _MODE_READ: + raw = _streams.DecompressReader( + self._fp, + ZstdDecompressor, + zstd_dict=zstd_dict, + options=options, + ) + self._buffer = io.BufferedReader(raw) + + def close(self): + """Flush and close the file. + + May be called multiple times. Once the file has been closed, + any other operation on it will raise ValueError. + """ + if self._fp is None: + return + try: + if self._mode == _MODE_READ: + if getattr(self, '_buffer', None): + self._buffer.close() + self._buffer = None + elif self._mode == _MODE_WRITE: + self.flush(self.FLUSH_FRAME) + self._compressor = None + finally: + self._mode = _MODE_CLOSED + try: + if self._close_fp: + self._fp.close() + finally: + self._fp = None + self._close_fp = False + + def write(self, data, /): + """Write a bytes-like object *data* to the file. + + Returns the number of uncompressed bytes written, which is + always the length of data in bytes. Note that due to buffering, + the file on disk may not reflect the data written until .flush() + or .close() is called. + """ + self._check_can_write() + + length = _nbytes(data) + + compressed = self._compressor.compress(data) + self._fp.write(compressed) + self._pos += length + return length + + def flush(self, mode=FLUSH_BLOCK): + """Flush remaining data to the underlying stream. + + The mode argument can be FLUSH_BLOCK or FLUSH_FRAME. Abuse of this + method will reduce compression ratio, use it only when necessary. + + If the program is interrupted afterwards, all data can be recovered. + To ensure saving to disk, also need to use os.fsync(fd). + + This method does nothing in reading mode. + """ + if self._mode == _MODE_READ: + return + self._check_not_closed() + if mode not in {self.FLUSH_BLOCK, self.FLUSH_FRAME}: + raise ValueError('Invalid mode argument, expected either ' + 'ZstdFile.FLUSH_FRAME or ' + 'ZstdFile.FLUSH_BLOCK') + if self._compressor.last_mode == mode: + return + # Flush zstd block/frame, and write. + data = self._compressor.flush(mode) + self._fp.write(data) + if hasattr(self._fp, 'flush'): + self._fp.flush() + + def read(self, size=-1): + """Read up to size uncompressed bytes from the file. + + If size is negative or omitted, read until EOF is reached. + Returns b'' if the file is already at EOF. + """ + if size is None: + size = -1 + self._check_can_read() + return self._buffer.read(size) + + def read1(self, size=-1): + """Read up to size uncompressed bytes, while trying to avoid + making multiple reads from the underlying stream. Reads up to a + buffer's worth of data if size is negative. + + Returns b'' if the file is at EOF. + """ + self._check_can_read() + if size < 0: + # Note this should *not* be io.DEFAULT_BUFFER_SIZE. + # ZSTD_DStreamOutSize is the minimum amount to read guaranteeing + # a full block is read. + size = ZSTD_DStreamOutSize + return self._buffer.read1(size) + + def readinto(self, b): + """Read bytes into b. + + Returns the number of bytes read (0 for EOF). + """ + self._check_can_read() + return self._buffer.readinto(b) + + def readinto1(self, b): + """Read bytes into b, while trying to avoid making multiple reads + from the underlying stream. + + Returns the number of bytes read (0 for EOF). + """ + self._check_can_read() + return self._buffer.readinto1(b) + + def readline(self, size=-1): + """Read a line of uncompressed bytes from the file. + + The terminating newline (if present) is retained. If size is + non-negative, no more than size bytes will be read (in which + case the line may be incomplete). Returns b'' if already at EOF. + """ + self._check_can_read() + return self._buffer.readline(size) + + def seek(self, offset, whence=io.SEEK_SET): + """Change the file position. + + The new position is specified by offset, relative to the + position indicated by whence. Possible values for whence are: + + 0: start of stream (default): offset must not be negative + 1: current stream position + 2: end of stream; offset must not be positive + + Returns the new file position. + + Note that seeking is emulated, so depending on the arguments, + this operation may be extremely slow. + """ + self._check_can_read() + + # BufferedReader.seek() checks seekable + return self._buffer.seek(offset, whence) + + def peek(self, size=-1): + """Return buffered data without advancing the file position. + + Always returns at least one byte of data, unless at EOF. + The exact number of bytes returned is unspecified. + """ + # Relies on the undocumented fact that BufferedReader.peek() always + # returns at least one byte (except at EOF) + self._check_can_read() + return self._buffer.peek(size) + + def __next__(self): + if ret := self._buffer.readline(): + return ret + raise StopIteration + + def tell(self): + """Return the current file position.""" + self._check_not_closed() + if self._mode == _MODE_READ: + return self._buffer.tell() + elif self._mode == _MODE_WRITE: + return self._pos + + def fileno(self): + """Return the file descriptor for the underlying file.""" + self._check_not_closed() + return self._fp.fileno() + + @property + def name(self): + self._check_not_closed() + return self._fp.name + + @property + def mode(self): + return 'wb' if self._mode == _MODE_WRITE else 'rb' + + @property + def closed(self): + """True if this file is closed.""" + return self._mode == _MODE_CLOSED + + def seekable(self): + """Return whether the file supports seeking.""" + return self.readable() and self._buffer.seekable() + + def readable(self): + """Return whether the file was opened for reading.""" + self._check_not_closed() + return self._mode == _MODE_READ + + def writable(self): + """Return whether the file was opened for writing.""" + self._check_not_closed() + return self._mode == _MODE_WRITE + + +def open(file, /, mode='rb', *, level=None, options=None, zstd_dict=None, + encoding=None, errors=None, newline=None): + """Open a Zstandard compressed file in binary or text mode. + + file can be either a file name (given as a str, bytes, or PathLike object), + in which case the named file is opened, or it can be an existing file object + to read from or write to. + + The mode parameter can be 'r', 'rb' (default), 'w', 'wb', 'x', 'xb', 'a', + 'ab' for binary mode, or 'rt', 'wt', 'xt', 'at' for text mode. + + The level, options, and zstd_dict parameters specify the settings the same + as ZstdFile. + + When using read mode (decompression), the options parameter is a dict + representing advanced decompression options. The level parameter is not + supported in this case. When using write mode (compression), only one of + level, an int representing the compression level, or options, a dict + representing advanced compression options, may be passed. In both modes, + zstd_dict is a ZstdDict instance containing a trained Zstandard dictionary. + + For binary mode, this function is equivalent to the ZstdFile constructor: + ZstdFile(filename, mode, ...). In this case, the encoding, errors and + newline parameters must not be provided. + + For text mode, an ZstdFile object is created, and wrapped in an + io.TextIOWrapper instance with the specified encoding, error handling + behavior, and line ending(s). + """ + + text_mode = 't' in mode + mode = mode.replace('t', '') + + if text_mode: + if 'b' in mode: + raise ValueError(f'Invalid mode: {mode!r}') + else: + if encoding is not None: + raise ValueError('Argument "encoding" not supported in binary mode') + if errors is not None: + raise ValueError('Argument "errors" not supported in binary mode') + if newline is not None: + raise ValueError('Argument "newline" not supported in binary mode') + + binary_file = ZstdFile(file, mode, level=level, options=options, + zstd_dict=zstd_dict) + + if text_mode: + return io.TextIOWrapper(binary_file, encoding, errors, newline) + else: + return binary_file diff --git a/Lib/concurrent/futures/__init__.py b/Lib/concurrent/futures/__init__.py index d746aeac50a..72de617a5b6 100644 --- a/Lib/concurrent/futures/__init__.py +++ b/Lib/concurrent/futures/__init__.py @@ -23,6 +23,7 @@ 'ALL_COMPLETED', 'CancelledError', 'TimeoutError', + 'InvalidStateError', 'BrokenExecutor', 'Future', 'Executor', @@ -50,4 +51,4 @@ def __getattr__(name): ThreadPoolExecutor = te return te - raise AttributeError(f"module {__name__} has no attribute {name}") + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") diff --git a/Lib/concurrent/futures/_base.py b/Lib/concurrent/futures/_base.py index cf119ac6437..7d69a5baead 100644 --- a/Lib/concurrent/futures/_base.py +++ b/Lib/concurrent/futures/_base.py @@ -50,9 +50,7 @@ class CancelledError(Error): """The Future was cancelled.""" pass -class TimeoutError(Error): - """The operation exceeded the given deadline.""" - pass +TimeoutError = TimeoutError # make local alias for the standard exception class InvalidStateError(Error): """The operation is not allowed in this state.""" @@ -284,7 +282,7 @@ def wait(fs, timeout=None, return_when=ALL_COMPLETED): A named 2-tuple of sets. The first set, named 'done', contains the futures that completed (is finished or cancelled) before the wait completed. The second set, named 'not_done', contains uncompleted - futures. Duplicate futures given to *fs* are removed and will be + futures. Duplicate futures given to *fs* are removed and will be returned only once. """ fs = set(fs) @@ -312,6 +310,18 @@ def wait(fs, timeout=None, return_when=ALL_COMPLETED): done.update(waiter.finished_futures) return DoneAndNotDoneFutures(done, fs - done) + +def _result_or_cancel(fut, timeout=None): + try: + try: + return fut.result(timeout) + finally: + fut.cancel() + finally: + # Break a reference cycle with the exception in self._exception + del fut + + class Future(object): """Represents the result of an asynchronous computation.""" @@ -386,7 +396,7 @@ def done(self): return self._state in [CANCELLED, CANCELLED_AND_NOTIFIED, FINISHED] def __get_result(self): - if self._exception: + if self._exception is not None: try: raise self._exception finally: @@ -606,9 +616,9 @@ def result_iterator(): while fs: # Careful not to keep a reference to the popped future if timeout is None: - yield fs.pop().result() + yield _result_or_cancel(fs.pop()) else: - yield fs.pop().result(end_time - time.monotonic()) + yield _result_or_cancel(fs.pop(), end_time - time.monotonic()) finally: for future in fs: future.cancel() diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py index 57941e485d8..0dee8303ba2 100644 --- a/Lib/concurrent/futures/process.py +++ b/Lib/concurrent/futures/process.py @@ -49,6 +49,8 @@ from concurrent.futures import _base import queue import multiprocessing as mp +# This import is required to load the multiprocessing.connection submodule +# so that it can be accessed later as `mp.connection` import multiprocessing.connection from multiprocessing.queues import Queue import threading @@ -56,7 +58,7 @@ from functools import partial import itertools import sys -import traceback +from traceback import format_exception _threads_wakeups = weakref.WeakKeyDictionary() @@ -66,22 +68,31 @@ class _ThreadWakeup: def __init__(self): self._closed = False + self._lock = threading.Lock() self._reader, self._writer = mp.Pipe(duplex=False) def close(self): - if not self._closed: - self._closed = True - self._writer.close() - self._reader.close() + # Please note that we do not take the self._lock when + # calling clear() (to avoid deadlocking) so this method can + # only be called safely from the same thread as all calls to + # clear() even if you hold the lock. Otherwise we + # might try to read from the closed pipe. + with self._lock: + if not self._closed: + self._closed = True + self._writer.close() + self._reader.close() def wakeup(self): - if not self._closed: - self._writer.send_bytes(b"") + with self._lock: + if not self._closed: + self._writer.send_bytes(b"") def clear(self): - if not self._closed: - while self._reader.poll(): - self._reader.recv_bytes() + if self._closed: + raise RuntimeError('operation on closed _ThreadWakeup') + while self._reader.poll(): + self._reader.recv_bytes() def _python_exit(): @@ -123,8 +134,7 @@ def __str__(self): class _ExceptionWithTraceback: def __init__(self, exc, tb): - tb = traceback.format_exception(type(exc), exc, tb) - tb = ''.join(tb) + tb = ''.join(format_exception(type(exc), exc, tb)) self.exc = exc # Traceback object needs to be garbage-collected as its frames # contain references to all the objects in the exception scope @@ -145,10 +155,11 @@ def __init__(self, future, fn, args, kwargs): self.kwargs = kwargs class _ResultItem(object): - def __init__(self, work_id, exception=None, result=None): + def __init__(self, work_id, exception=None, result=None, exit_pid=None): self.work_id = work_id self.exception = exception self.result = result + self.exit_pid = exit_pid class _CallItem(object): def __init__(self, work_id, fn, args, kwargs): @@ -160,20 +171,17 @@ def __init__(self, work_id, fn, args, kwargs): class _SafeQueue(Queue): """Safe Queue set exception to the future object linked to a job""" - def __init__(self, max_size=0, *, ctx, pending_work_items, shutdown_lock, - thread_wakeup): + def __init__(self, max_size=0, *, ctx, pending_work_items, thread_wakeup): self.pending_work_items = pending_work_items - self.shutdown_lock = shutdown_lock self.thread_wakeup = thread_wakeup super().__init__(max_size, ctx=ctx) def _on_queue_feeder_error(self, e, obj): if isinstance(obj, _CallItem): - tb = traceback.format_exception(type(e), e, e.__traceback__) + tb = format_exception(type(e), e, e.__traceback__) e.__cause__ = _RemoteTraceback('\n"""\n{}"""'.format(''.join(tb))) work_item = self.pending_work_items.pop(obj.work_id, None) - with self.shutdown_lock: - self.thread_wakeup.wakeup() + self.thread_wakeup.wakeup() # work_item can be None if another process terminated. In this # case, the executor_manager_thread fails all work_items # with BrokenProcessPool @@ -183,16 +191,6 @@ def _on_queue_feeder_error(self, e, obj): super()._on_queue_feeder_error(e, obj) -def _get_chunks(*iterables, chunksize): - """ Iterates over zip()ed iterables in chunks. """ - it = zip(*iterables) - while True: - chunk = tuple(itertools.islice(it, chunksize)) - if not chunk: - return - yield chunk - - def _process_chunk(fn, chunk): """ Processes a chunk of an iterable passed to map. @@ -205,17 +203,19 @@ def _process_chunk(fn, chunk): return [fn(*args) for args in chunk] -def _sendback_result(result_queue, work_id, result=None, exception=None): +def _sendback_result(result_queue, work_id, result=None, exception=None, + exit_pid=None): """Safely send back the given result or exception""" try: result_queue.put(_ResultItem(work_id, result=result, - exception=exception)) + exception=exception, exit_pid=exit_pid)) except BaseException as e: exc = _ExceptionWithTraceback(e, e.__traceback__) - result_queue.put(_ResultItem(work_id, exception=exc)) + result_queue.put(_ResultItem(work_id, exception=exc, + exit_pid=exit_pid)) -def _process_worker(call_queue, result_queue, initializer, initargs): +def _process_worker(call_queue, result_queue, initializer, initargs, max_tasks=None): """Evaluates calls from call_queue and places the results in result_queue. This worker is run in a separate process. @@ -236,25 +236,38 @@ def _process_worker(call_queue, result_queue, initializer, initargs): # The parent will notice that the process stopped and # mark the pool broken return + num_tasks = 0 + exit_pid = None while True: call_item = call_queue.get(block=True) if call_item is None: # Wake up queue management thread result_queue.put(os.getpid()) return + + if max_tasks is not None: + num_tasks += 1 + if num_tasks >= max_tasks: + exit_pid = os.getpid() + try: r = call_item.fn(*call_item.args, **call_item.kwargs) except BaseException as e: exc = _ExceptionWithTraceback(e, e.__traceback__) - _sendback_result(result_queue, call_item.work_id, exception=exc) + _sendback_result(result_queue, call_item.work_id, exception=exc, + exit_pid=exit_pid) else: - _sendback_result(result_queue, call_item.work_id, result=r) + _sendback_result(result_queue, call_item.work_id, result=r, + exit_pid=exit_pid) del r # Liberate the resource as soon as possible, to avoid holding onto # open files or shared memory that is not needed anymore del call_item + if exit_pid is not None: + return + class _ExecutorManagerThread(threading.Thread): """Manages the communication between this process and the worker processes. @@ -284,11 +297,10 @@ def __init__(self, executor): # if there is no pending work item. def weakref_cb(_, thread_wakeup=self.thread_wakeup, - shutdown_lock=self.shutdown_lock): - mp.util.debug('Executor collected: triggering callback for' + mp_util_debug=mp.util.debug): + mp_util_debug('Executor collected: triggering callback for' ' QueueManager wakeup') - with shutdown_lock: - thread_wakeup.wakeup() + thread_wakeup.wakeup() self.executor_reference = weakref.ref(executor, weakref_cb) @@ -305,6 +317,10 @@ def weakref_cb(_, # A queue.Queue of work ids e.g. Queue([5, 6, ...]). self.work_ids_queue = executor._work_ids + # Maximum number of tasks a worker process can execute before + # exiting safely + self.max_tasks_per_child = executor._max_tasks_per_child + # A dict mapping work ids to _WorkItems e.g. # {5: <_WorkItem...>, 6: <_WorkItem...>, ...} self.pending_work_items = executor._pending_work_items @@ -315,7 +331,14 @@ def run(self): # Main loop for the executor manager thread. while True: - self.add_call_item_to_queue() + # gh-109047: During Python finalization, self.call_queue.put() + # creation of a thread can fail with RuntimeError. + try: + self.add_call_item_to_queue() + except BaseException as exc: + cause = format_exception(exc) + self.terminate_broken(cause) + return result_item, is_broken, cause = self.wait_result_broken_or_wakeup() @@ -324,19 +347,32 @@ def run(self): return if result_item is not None: self.process_result_item(result_item) + + process_exited = result_item.exit_pid is not None + if process_exited: + p = self.processes.pop(result_item.exit_pid) + p.join() + # Delete reference to result_item to avoid keeping references # while waiting on new results. del result_item - # attempt to increment idle process count - executor = self.executor_reference() - if executor is not None: - executor._idle_worker_semaphore.release() - del executor + if executor := self.executor_reference(): + if process_exited: + with self.shutdown_lock: + executor._adjust_process_count() + else: + executor._idle_worker_semaphore.release() + del executor if self.is_shutting_down(): self.flag_executor_shutting_down() + # When only canceled futures remain in pending_work_items, our + # next call to wait_result_broken_or_wakeup would hang forever. + # This makes sure we have some running futures or none at all. + self.add_call_item_to_queue() + # Since no new work items can be added, it is safe to shutdown # this thread if there are no pending work items. if not self.pending_work_items: @@ -386,14 +422,13 @@ def wait_result_broken_or_wakeup(self): try: result_item = result_reader.recv() is_broken = False - except BaseException as e: - cause = traceback.format_exception(type(e), e, e.__traceback__) + except BaseException as exc: + cause = format_exception(exc) elif wakeup_reader in ready: is_broken = False - with self.shutdown_lock: - self.thread_wakeup.clear() + self.thread_wakeup.clear() return result_item, is_broken, cause @@ -401,24 +436,14 @@ def process_result_item(self, result_item): # Process the received a result_item. This can be either the PID of a # worker that exited gracefully or a _ResultItem - if isinstance(result_item, int): - # Clean shutdown of a worker using its PID - # (avoids marking the executor broken) - assert self.is_shutting_down() - p = self.processes.pop(result_item) - p.join() - if not self.processes: - self.join_executor_internals() - return - else: - # Received a _ResultItem so mark the future as completed. - work_item = self.pending_work_items.pop(result_item.work_id, None) - # work_item can be None if another process terminated (see above) - if work_item is not None: - if result_item.exception: - work_item.future.set_exception(result_item.exception) - else: - work_item.future.set_result(result_item.result) + # Received a _ResultItem so mark the future as completed. + work_item = self.pending_work_items.pop(result_item.work_id, None) + # work_item can be None if another process terminated (see above) + if work_item is not None: + if result_item.exception is not None: + work_item.future.set_exception(result_item.exception) + else: + work_item.future.set_result(result_item.result) def is_shutting_down(self): # Check whether we should start shutting down the executor. @@ -430,7 +455,7 @@ def is_shutting_down(self): return (_global_shutdown or executor is None or executor._shutdown_thread) - def terminate_broken(self, cause): + def _terminate_broken(self, cause): # Terminate the executor because it is in a broken state. The cause # argument can be used to display more information on the error that # lead the executor into becoming broken. @@ -455,7 +480,14 @@ def terminate_broken(self, cause): # Mark pending tasks as failed. for work_id, work_item in self.pending_work_items.items(): - work_item.future.set_exception(bpe) + try: + work_item.future.set_exception(bpe) + except _base.InvalidStateError: + # set_exception() fails if the future is cancelled: ignore it. + # Trying to check if the future is cancelled before calling + # set_exception() would leave a race condition if the future is + # cancelled between the check and set_exception(). + pass # Delete references to object. See issue16284 del work_item self.pending_work_items.clear() @@ -465,8 +497,14 @@ def terminate_broken(self, cause): for p in self.processes.values(): p.terminate() + self.call_queue._terminate_broken() + # clean up resources - self.join_executor_internals() + self._join_executor_internals(broken=True) + + def terminate_broken(self, cause): + with self.shutdown_lock: + self._terminate_broken(cause) def flag_executor_shutting_down(self): # Flag the executor as shutting down and cancel remaining tasks if @@ -509,15 +547,24 @@ def shutdown_workers(self): break def join_executor_internals(self): - self.shutdown_workers() + with self.shutdown_lock: + self._join_executor_internals() + + def _join_executor_internals(self, broken=False): + # If broken, call_queue was closed and so can no longer be used. + if not broken: + self.shutdown_workers() + # Release the queue's resources as soon as possible. self.call_queue.close() self.call_queue.join_thread() - with self.shutdown_lock: - self.thread_wakeup.close() + self.thread_wakeup.close() + # If .join() is not called on the created processes then # some ctx.Queue methods may deadlock on Mac OS X. for p in self.processes.values(): + if broken: + p.terminate() p.join() def get_n_children_alive(self): @@ -582,22 +629,29 @@ class BrokenProcessPool(_base.BrokenExecutor): class ProcessPoolExecutor(_base.Executor): def __init__(self, max_workers=None, mp_context=None, - initializer=None, initargs=()): + initializer=None, initargs=(), *, max_tasks_per_child=None): """Initializes a new ProcessPoolExecutor instance. Args: max_workers: The maximum number of processes that can be used to execute the given calls. If None or not given then as many worker processes will be created as the machine has processors. - mp_context: A multiprocessing context to launch the workers. This + mp_context: A multiprocessing context to launch the workers created + using the multiprocessing.get_context('start method') API. This object should provide SimpleQueue, Queue and Process. initializer: A callable used to initialize worker processes. initargs: A tuple of arguments to pass to the initializer. + max_tasks_per_child: The maximum number of tasks a worker process + can complete before it will exit and be replaced with a fresh + worker process. The default of None means worker process will + live as long as the executor. Requires a non-'fork' mp_context + start method. When given, we default to using 'spawn' if no + mp_context is supplied. """ _check_system_limits() if max_workers is None: - self._max_workers = os.cpu_count() or 1 + self._max_workers = os.process_cpu_count() or 1 if sys.platform == 'win32': self._max_workers = min(_MAX_WINDOWS_WORKERS, self._max_workers) @@ -612,7 +666,10 @@ def __init__(self, max_workers=None, mp_context=None, self._max_workers = max_workers if mp_context is None: - mp_context = mp.get_context() + if max_tasks_per_child is not None: + mp_context = mp.get_context("spawn") + else: + mp_context = mp.get_context() self._mp_context = mp_context # https://github.com/python/cpython/issues/90622 @@ -624,6 +681,18 @@ def __init__(self, max_workers=None, mp_context=None, self._initializer = initializer self._initargs = initargs + if max_tasks_per_child is not None: + if not isinstance(max_tasks_per_child, int): + raise TypeError("max_tasks_per_child must be an integer") + elif max_tasks_per_child <= 0: + raise ValueError("max_tasks_per_child must be >= 1") + if self._mp_context.get_start_method(allow_none=False) == "fork": + # https://github.com/python/cpython/issues/90622 + raise ValueError("max_tasks_per_child is incompatible with" + " the 'fork' multiprocessing start method;" + " supply a different mp_context.") + self._max_tasks_per_child = max_tasks_per_child + # Management thread self._executor_manager_thread = None @@ -646,7 +715,9 @@ def __init__(self, max_workers=None, mp_context=None, # as it could result in a deadlock if a worker process dies with the # _result_queue write lock still acquired. # - # _shutdown_lock must be locked to access _ThreadWakeup. + # Care must be taken to only call clear and close from the + # executor_manager_thread, since _ThreadWakeup.clear() is not protected + # by a lock. self._executor_manager_thread_wakeup = _ThreadWakeup() # Create communication channels for the executor @@ -657,7 +728,6 @@ def __init__(self, max_workers=None, mp_context=None, self._call_queue = _SafeQueue( max_size=queue_size, ctx=self._mp_context, pending_work_items=self._pending_work_items, - shutdown_lock=self._shutdown_lock, thread_wakeup=self._executor_manager_thread_wakeup) # Killed worker processes can produce spurious "broken pipe" # tracebacks in the queue's own worker thread. But we detect killed @@ -677,6 +747,11 @@ def _start_executor_manager_thread(self): self._executor_manager_thread_wakeup def _adjust_process_count(self): + # gh-132969: avoid error when state is reset and executor is still running, + # which will happen when shutdown(wait=False) is called. + if self._processes is None: + return + # if there's an idle process, we don't need to spawn a new one. if self._idle_worker_semaphore.acquire(blocking=False): return @@ -705,7 +780,8 @@ def _spawn_process(self): args=(self._call_queue, self._result_queue, self._initializer, - self._initargs)) + self._initargs, + self._max_tasks_per_child)) p.start() self._processes[p.pid] = p @@ -759,7 +835,7 @@ def map(self, fn, *iterables, timeout=None, chunksize=1): raise ValueError("chunksize must be >= 1.") results = super().map(partial(_process_chunk, fn), - _get_chunks(*iterables, chunksize=chunksize), + itertools.batched(zip(*iterables), chunksize), timeout=timeout) return _chain_from_iterable_of_lists(results) diff --git a/Lib/concurrent/futures/thread.py b/Lib/concurrent/futures/thread.py index 493861d314d..9021dde48ef 100644 --- a/Lib/concurrent/futures/thread.py +++ b/Lib/concurrent/futures/thread.py @@ -37,14 +37,14 @@ def _python_exit(): threading._register_atexit(_python_exit) # At fork, reinitialize the `_global_shutdown_lock` lock in the child process -# TODO RUSTPYTHON - _at_fork_reinit is not implemented yet -if hasattr(os, 'register_at_fork') and hasattr(_global_shutdown_lock, '_at_fork_reinit'): +if hasattr(os, 'register_at_fork'): os.register_at_fork(before=_global_shutdown_lock.acquire, after_in_child=_global_shutdown_lock._at_fork_reinit, after_in_parent=_global_shutdown_lock.release) + os.register_at_fork(after_in_child=_threads_queues.clear) -class _WorkItem(object): +class _WorkItem: def __init__(self, future, fn, args, kwargs): self.future = future self.fn = fn @@ -79,17 +79,20 @@ def _worker(executor_reference, work_queue, initializer, initargs): return try: while True: - work_item = work_queue.get(block=True) - if work_item is not None: - work_item.run() - # Delete references to object. See issue16284 - del work_item - - # attempt to increment idle count + try: + work_item = work_queue.get_nowait() + except queue.Empty: + # attempt to increment idle count if queue is empty executor = executor_reference() if executor is not None: executor._idle_semaphore.release() del executor + work_item = work_queue.get(block=True) + + if work_item is not None: + work_item.run() + # Delete references to object. See GH-60488 + del work_item continue executor = executor_reference() @@ -137,10 +140,10 @@ def __init__(self, max_workers=None, thread_name_prefix='', # * CPU bound task which releases GIL # * I/O bound task (which releases GIL, of course) # - # We use cpu_count + 4 for both types of tasks. + # We use process_cpu_count + 4 for both types of tasks. # But we limit it to 32 to avoid consuming surprisingly large resource # on many core machine. - max_workers = min(32, (os.cpu_count() or 1) + 4) + max_workers = min(32, (os.process_cpu_count() or 1) + 4) if max_workers <= 0: raise ValueError("max_workers must be greater than 0") diff --git a/Lib/contextvars.py b/Lib/contextvars.py index d78c80dfe6f..14514f185e0 100644 --- a/Lib/contextvars.py +++ b/Lib/contextvars.py @@ -1,4 +1,8 @@ +import _collections_abc from _contextvars import Context, ContextVar, Token, copy_context __all__ = ('Context', 'ContextVar', 'Token', 'copy_context') + + +_collections_abc.Mapping.register(Context) diff --git a/Lib/copyreg.py b/Lib/copyreg.py index 578392409b4..a5e8add4a55 100644 --- a/Lib/copyreg.py +++ b/Lib/copyreg.py @@ -31,11 +31,16 @@ def pickle_complex(c): pickle(complex, pickle_complex, complex) def pickle_union(obj): - import functools, operator - return functools.reduce, (operator.or_, obj.__args__) + import typing, operator + return operator.getitem, (typing.Union, obj.__args__) pickle(type(int | str), pickle_union) +def pickle_super(obj): + return super, (obj.__thisclass__, obj.__self__) + +pickle(super, pickle_super) + # Support for pickling new-style objects def _reconstructor(cls, base, state): diff --git a/Lib/decimal.py b/Lib/decimal.py index ee3147f5dde..530bdfb3895 100644 --- a/Lib/decimal.py +++ b/Lib/decimal.py @@ -100,8 +100,8 @@ try: from _decimal import * - from _decimal import __version__ - from _decimal import __libmpdec_version__ + from _decimal import __version__ # noqa: F401 + from _decimal import __libmpdec_version__ # noqa: F401 except ImportError: import _pydecimal import sys diff --git a/Lib/dis.py b/Lib/dis.py index 6583cab62b8..d6d2c1386dd 100644 --- a/Lib/dis.py +++ b/Lib/dis.py @@ -11,15 +11,16 @@ _cache_format, _inline_cache_entries, _nb_ops, + _common_constants, _intrinsic_1_descs, _intrinsic_2_descs, + _special_method_names, _specializations, _specialized_opmap, ) from _opcode import get_executor - __all__ = ["code_info", "dis", "disassemble", "distb", "disco", "findlinestarts", "findlabels", "show_code", "get_instructions", "Instruction", "Bytecode"] + _opcodes_all @@ -31,12 +32,11 @@ CONVERT_VALUE = opmap['CONVERT_VALUE'] SET_FUNCTION_ATTRIBUTE = opmap['SET_FUNCTION_ATTRIBUTE'] -FUNCTION_ATTR_FLAGS = ('defaults', 'kwdefaults', 'annotations', 'closure') +FUNCTION_ATTR_FLAGS = ('defaults', 'kwdefaults', 'annotations', 'closure', 'annotate') ENTER_EXECUTOR = opmap['ENTER_EXECUTOR'] -LOAD_CONST = opmap['LOAD_CONST'] -RETURN_CONST = opmap['RETURN_CONST'] LOAD_GLOBAL = opmap['LOAD_GLOBAL'] +LOAD_SMALL_INT = opmap['LOAD_SMALL_INT'] BINARY_OP = opmap['BINARY_OP'] JUMP_BACKWARD = opmap['JUMP_BACKWARD'] FOR_ITER = opmap['FOR_ITER'] @@ -45,9 +45,15 @@ LOAD_SUPER_ATTR = opmap['LOAD_SUPER_ATTR'] CALL_INTRINSIC_1 = opmap['CALL_INTRINSIC_1'] CALL_INTRINSIC_2 = opmap['CALL_INTRINSIC_2'] +LOAD_COMMON_CONSTANT = opmap['LOAD_COMMON_CONSTANT'] +LOAD_SPECIAL = opmap['LOAD_SPECIAL'] LOAD_FAST_LOAD_FAST = opmap['LOAD_FAST_LOAD_FAST'] +LOAD_FAST_BORROW_LOAD_FAST_BORROW = opmap['LOAD_FAST_BORROW_LOAD_FAST_BORROW'] STORE_FAST_LOAD_FAST = opmap['STORE_FAST_LOAD_FAST'] STORE_FAST_STORE_FAST = opmap['STORE_FAST_STORE_FAST'] +IS_OP = opmap['IS_OP'] +CONTAINS_OP = opmap['CONTAINS_OP'] +END_ASYNC_FOR = opmap['END_ASYNC_FOR'] CACHE = opmap["CACHE"] @@ -77,7 +83,7 @@ def _try_compile(source, name): return compile(source, name, 'exec') def dis(x=None, *, file=None, depth=None, show_caches=False, adaptive=False, - show_offsets=False): + show_offsets=False, show_positions=False): """Disassemble classes, methods, functions, and other compiled objects. With no argument, disassemble the last traceback. @@ -88,7 +94,7 @@ def dis(x=None, *, file=None, depth=None, show_caches=False, adaptive=False, """ if x is None: distb(file=file, show_caches=show_caches, adaptive=adaptive, - show_offsets=show_offsets) + show_offsets=show_offsets, show_positions=show_positions) return # Extract functions from methods. if hasattr(x, '__func__'): @@ -109,12 +115,12 @@ def dis(x=None, *, file=None, depth=None, show_caches=False, adaptive=False, if isinstance(x1, _have_code): print("Disassembly of %s:" % name, file=file) try: - dis(x1, file=file, depth=depth, show_caches=show_caches, adaptive=adaptive, show_offsets=show_offsets) + dis(x1, file=file, depth=depth, show_caches=show_caches, adaptive=adaptive, show_offsets=show_offsets, show_positions=show_positions) except TypeError as msg: print("Sorry:", msg, file=file) print(file=file) elif hasattr(x, 'co_code'): # Code object - _disassemble_recursive(x, file=file, depth=depth, show_caches=show_caches, adaptive=adaptive, show_offsets=show_offsets) + _disassemble_recursive(x, file=file, depth=depth, show_caches=show_caches, adaptive=adaptive, show_offsets=show_offsets, show_positions=show_positions) elif isinstance(x, (bytes, bytearray)): # Raw bytecode labels_map = _make_labels_map(x) label_width = 4 + len(str(len(labels_map))) @@ -125,12 +131,12 @@ def dis(x=None, *, file=None, depth=None, show_caches=False, adaptive=False, arg_resolver = ArgResolver(labels_map=labels_map) _disassemble_bytes(x, arg_resolver=arg_resolver, formatter=formatter) elif isinstance(x, str): # Source code - _disassemble_str(x, file=file, depth=depth, show_caches=show_caches, adaptive=adaptive, show_offsets=show_offsets) + _disassemble_str(x, file=file, depth=depth, show_caches=show_caches, adaptive=adaptive, show_offsets=show_offsets, show_positions=show_positions) else: raise TypeError("don't know how to disassemble %s objects" % type(x).__name__) -def distb(tb=None, *, file=None, show_caches=False, adaptive=False, show_offsets=False): +def distb(tb=None, *, file=None, show_caches=False, adaptive=False, show_offsets=False, show_positions=False): """Disassemble a traceback (default: last traceback).""" if tb is None: try: @@ -141,22 +147,24 @@ def distb(tb=None, *, file=None, show_caches=False, adaptive=False, show_offsets except AttributeError: raise RuntimeError("no last traceback to disassemble") from None while tb.tb_next: tb = tb.tb_next - disassemble(tb.tb_frame.f_code, tb.tb_lasti, file=file, show_caches=show_caches, adaptive=adaptive, show_offsets=show_offsets) + disassemble(tb.tb_frame.f_code, tb.tb_lasti, file=file, show_caches=show_caches, adaptive=adaptive, show_offsets=show_offsets, show_positions=show_positions) # The inspect module interrogates this dictionary to build its # list of CO_* constants. It is also used by pretty_flags to # turn the co_flags field into a human readable list. COMPILER_FLAG_NAMES = { - 1: "OPTIMIZED", - 2: "NEWLOCALS", - 4: "VARARGS", - 8: "VARKEYWORDS", - 16: "NESTED", - 32: "GENERATOR", - 64: "NOFREE", - 128: "COROUTINE", - 256: "ITERABLE_COROUTINE", - 512: "ASYNC_GENERATOR", + 1: "OPTIMIZED", + 2: "NEWLOCALS", + 4: "VARARGS", + 8: "VARKEYWORDS", + 16: "NESTED", + 32: "GENERATOR", + 64: "NOFREE", + 128: "COROUTINE", + 256: "ITERABLE_COROUTINE", + 512: "ASYNC_GENERATOR", + 0x4000000: "HAS_DOCSTRING", + 0x8000000: "METHOD", } def pretty_flags(flags): @@ -370,6 +378,14 @@ class Instruction(_Instruction): entries (if any) """ + @staticmethod + def make( + opname, arg, argval, argrepr, offset, start_offset, starts_line, + line_number, label=None, positions=None, cache_info=None + ): + return Instruction(opname, _all_opmap[opname], arg, argval, argrepr, offset, + start_offset, starts_line, line_number, label, positions, cache_info) + @property def oparg(self): """Alias for Instruction.arg.""" @@ -424,21 +440,25 @@ def __str__(self): class Formatter: def __init__(self, file=None, lineno_width=0, offset_width=0, label_width=0, - line_offset=0, show_caches=False): + line_offset=0, show_caches=False, *, show_positions=False): """Create a Formatter *file* where to write the output - *lineno_width* sets the width of the line number field (0 omits it) + *lineno_width* sets the width of the source location field (0 omits it). + Should be large enough for a line number or full positions (depending + on the value of *show_positions*). *offset_width* sets the width of the instruction offset field *label_width* sets the width of the label field *show_caches* is a boolean indicating whether to display cache lines - + *show_positions* is a boolean indicating whether full positions should + be reported instead of only the line numbers. """ self.file = file self.lineno_width = lineno_width self.offset_width = offset_width self.label_width = label_width self.show_caches = show_caches + self.show_positions = show_positions def print_instruction(self, instr, mark_as_current=False): self.print_instruction_line(instr, mark_as_current) @@ -471,15 +491,27 @@ def print_instruction_line(self, instr, mark_as_current): print(file=self.file) fields = [] - # Column: Source code line number + # Column: Source code locations information if lineno_width: - if instr.starts_line: - lineno_fmt = "%%%dd" if instr.line_number is not None else "%%%ds" - lineno_fmt = lineno_fmt % lineno_width - lineno = _NO_LINENO if instr.line_number is None else instr.line_number - fields.append(lineno_fmt % lineno) + if self.show_positions: + # reporting positions instead of just line numbers + if instr_positions := instr.positions: + if all(p is None for p in instr_positions): + positions_str = _NO_LINENO + else: + ps = tuple('?' if p is None else p for p in instr_positions) + positions_str = f"{ps[0]}:{ps[2]}-{ps[1]}:{ps[3]}" + fields.append(f'{positions_str:{lineno_width}}') + else: + fields.append(' ' * lineno_width) else: - fields.append(' ' * lineno_width) + if instr.starts_line: + lineno_fmt = "%%%dd" if instr.line_number is not None else "%%%ds" + lineno_fmt = lineno_fmt % lineno_width + lineno = _NO_LINENO if instr.line_number is None else instr.line_number + fields.append(lineno_fmt % lineno) + else: + fields.append(' ' * lineno_width) # Column: Label if instr.label is not None: lbl = f"L{instr.label}:" @@ -575,8 +607,9 @@ def get_argval_argrepr(self, op, arg, offset): argval = self.offset_from_jump_arg(op, arg, offset) lbl = self.get_label_for_offset(argval) assert lbl is not None - argrepr = f"to L{lbl}" - elif deop in (LOAD_FAST_LOAD_FAST, STORE_FAST_LOAD_FAST, STORE_FAST_STORE_FAST): + preposition = "from" if deop == END_ASYNC_FOR else "to" + argrepr = f"{preposition} L{lbl}" + elif deop in (LOAD_FAST_LOAD_FAST, LOAD_FAST_BORROW_LOAD_FAST_BORROW, STORE_FAST_LOAD_FAST, STORE_FAST_STORE_FAST): arg1 = arg >> 4 arg2 = arg & 15 val1, argrepr1 = _get_name_info(arg1, self.varname_from_oparg) @@ -602,6 +635,18 @@ def get_argval_argrepr(self, op, arg, offset): argrepr = _intrinsic_1_descs[arg] elif deop == CALL_INTRINSIC_2: argrepr = _intrinsic_2_descs[arg] + elif deop == LOAD_COMMON_CONSTANT: + obj = _common_constants[arg] + if isinstance(obj, type): + argrepr = obj.__name__ + else: + argrepr = repr(obj) + elif deop == LOAD_SPECIAL: + argrepr = _special_method_names[arg] + elif deop == IS_OP: + argrepr = 'is not' if argval else 'is' + elif deop == CONTAINS_OP: + argrepr = 'not in' if argval else 'in' return argval, argrepr def get_instructions(x, *, first_line=None, show_caches=None, adaptive=False): @@ -641,8 +686,10 @@ def _get_const_value(op, arg, co_consts): Otherwise (if it is a LOAD_CONST and co_consts is not provided) returns the dis.UNKNOWN sentinel. """ - assert op in hasconst + assert op in hasconst or op == LOAD_SMALL_INT + if op == LOAD_SMALL_INT: + return arg argval = UNKNOWN if co_consts is not None: argval = co_consts[arg] @@ -701,7 +748,8 @@ def _parse_exception_table(code): def _is_backward_jump(op): return opname[op] in ('JUMP_BACKWARD', - 'JUMP_BACKWARD_NO_INTERRUPT') + 'JUMP_BACKWARD_NO_INTERRUPT', + 'END_ASYNC_FOR') # Not really a jump, but it has a "target" def _get_instructions_bytes(code, linestarts=None, line_offset=0, co_positions=None, original_code=None, arg_resolver=None): @@ -745,8 +793,10 @@ def _get_instructions_bytes(code, linestarts=None, line_offset=0, co_positions=N if caches: cache_info = [] + cache_offset = offset for name, size in _cache_format[opname[deop]].items(): - data = code[offset + 2: offset + 2 + 2 * size] + data = code[cache_offset + 2: cache_offset + 2 + 2 * size] + cache_offset += size * 2 cache_info.append((name, size, data)) else: cache_info = None @@ -758,17 +808,22 @@ def _get_instructions_bytes(code, linestarts=None, line_offset=0, co_positions=N def disassemble(co, lasti=-1, *, file=None, show_caches=False, adaptive=False, - show_offsets=False): + show_offsets=False, show_positions=False): """Disassemble a code object.""" linestarts = dict(findlinestarts(co)) exception_entries = _parse_exception_table(co) + if show_positions: + lineno_width = _get_positions_width(co) + else: + lineno_width = _get_lineno_width(linestarts) labels_map = _make_labels_map(co.co_code, exception_entries=exception_entries) label_width = 4 + len(str(len(labels_map))) formatter = Formatter(file=file, - lineno_width=_get_lineno_width(linestarts), + lineno_width=lineno_width, offset_width=len(str(max(len(co.co_code) - 2, 9999))) if show_offsets else 0, label_width=label_width, - show_caches=show_caches) + show_caches=show_caches, + show_positions=show_positions) arg_resolver = ArgResolver(co_consts=co.co_consts, names=co.co_names, varname_from_oparg=co._varname_from_oparg, @@ -777,8 +832,8 @@ def disassemble(co, lasti=-1, *, file=None, show_caches=False, adaptive=False, exception_entries=exception_entries, co_positions=co.co_positions(), original_code=co.co_code, arg_resolver=arg_resolver, formatter=formatter) -def _disassemble_recursive(co, *, file=None, depth=None, show_caches=False, adaptive=False, show_offsets=False): - disassemble(co, file=file, show_caches=show_caches, adaptive=adaptive, show_offsets=show_offsets) +def _disassemble_recursive(co, *, file=None, depth=None, show_caches=False, adaptive=False, show_offsets=False, show_positions=False): + disassemble(co, file=file, show_caches=show_caches, adaptive=adaptive, show_offsets=show_offsets, show_positions=show_positions) if depth is None or depth > 0: if depth is not None: depth = depth - 1 @@ -788,7 +843,7 @@ def _disassemble_recursive(co, *, file=None, depth=None, show_caches=False, adap print("Disassembly of %r:" % (x,), file=file) _disassemble_recursive( x, file=file, depth=depth, show_caches=show_caches, - adaptive=adaptive, show_offsets=show_offsets + adaptive=adaptive, show_offsets=show_offsets, show_positions=show_positions ) @@ -821,6 +876,22 @@ def _get_lineno_width(linestarts): lineno_width = len(_NO_LINENO) return lineno_width +def _get_positions_width(code): + # Positions are formatted as 'LINE:COL-ENDLINE:ENDCOL ' (note trailing space). + # A missing component appears as '?', and when all components are None, we + # render '_NO_LINENO'. thus the minimum width is 1 + len(_NO_LINENO). + # + # If all values are missing, positions are not printed (i.e. positions_width = 0). + has_value = False + values_width = 0 + for positions in code.co_positions(): + has_value |= any(isinstance(p, int) for p in positions) + width = sum(1 if p is None else len(str(p)) for p in positions) + values_width = max(width, values_width) + if has_value: + # 3 = number of separators in a normal format + return 1 + max(len(_NO_LINENO), 3 + values_width) + return 0 def _disassemble_bytes(code, lasti=-1, linestarts=None, *, line_offset=0, exception_entries=(), @@ -938,7 +1009,8 @@ def _find_imports(co): if op == IMPORT_NAME and i >= 2: from_op = opargs[i-1] level_op = opargs[i-2] - if (from_op[0] in hasconst and level_op[0] in hasconst): + if (from_op[0] in hasconst and + (level_op[0] in hasconst or level_op[0] == LOAD_SMALL_INT)): level = _get_const_value(level_op[0], level_op[1], consts) fromlist = _get_const_value(from_op[0], from_op[1], consts) yield (names[oparg], level, fromlist) @@ -967,7 +1039,7 @@ class Bytecode: Iterating over this yields the bytecode operations as Instruction instances. """ - def __init__(self, x, *, first_line=None, current_offset=None, show_caches=False, adaptive=False, show_offsets=False): + def __init__(self, x, *, first_line=None, current_offset=None, show_caches=False, adaptive=False, show_offsets=False, show_positions=False): self.codeobj = co = _get_code_object(x) if first_line is None: self.first_line = co.co_firstlineno @@ -982,6 +1054,7 @@ def __init__(self, x, *, first_line=None, current_offset=None, show_caches=False self.show_caches = show_caches self.adaptive = adaptive self.show_offsets = show_offsets + self.show_positions = show_positions def __iter__(self): co = self.codeobj @@ -1025,16 +1098,19 @@ def dis(self): with io.StringIO() as output: code = _get_code_array(co, self.adaptive) offset_width = len(str(max(len(code) - 2, 9999))) if self.show_offsets else 0 - - + if self.show_positions: + lineno_width = _get_positions_width(co) + else: + lineno_width = _get_lineno_width(self._linestarts) labels_map = _make_labels_map(co.co_code, self.exception_entries) label_width = 4 + len(str(len(labels_map))) formatter = Formatter(file=output, - lineno_width=_get_lineno_width(self._linestarts), + lineno_width=lineno_width, offset_width=offset_width, label_width=label_width, line_offset=self._line_offset, - show_caches=self.show_caches) + show_caches=self.show_caches, + show_positions=self.show_positions) arg_resolver = ArgResolver(co_consts=co.co_consts, names=co.co_names, @@ -1052,17 +1128,18 @@ def dis(self): return output.getvalue() -from _dis import * # TODO: RUSTPYTHON; Remove this import (and module) - - def main(args=None): import argparse - parser = argparse.ArgumentParser() + parser = argparse.ArgumentParser(color=True) parser.add_argument('-C', '--show-caches', action='store_true', help='show inline caches') parser.add_argument('-O', '--show-offsets', action='store_true', help='show instruction offsets') + parser.add_argument('-P', '--show-positions', action='store_true', + help='show instruction positions') + parser.add_argument('-S', '--specialized', action='store_true', + help='show specialized bytecode') parser.add_argument('infile', nargs='?', default='-') args = parser.parse_args(args=args) if args.infile == '-': @@ -1073,9 +1150,8 @@ def main(args=None): with open(args.infile, 'rb') as infile: source = infile.read() code = compile(source, name, "exec") - # TODO: RUSTPYTHON; Add support for `show_caches` & `show_offsets` arguments - # dis(code, show_caches=args.show_caches, show_offsets=args.show_offsets) - dis(code) + dis(code, show_caches=args.show_caches, adaptive=args.specialized, + show_offsets=args.show_offsets, show_positions=args.show_positions) if __name__ == "__main__": main() diff --git a/Lib/dummy_threading.py b/Lib/dummy_threading.py index 1bb7eee338a..662f3b89a9a 100644 --- a/Lib/dummy_threading.py +++ b/Lib/dummy_threading.py @@ -6,6 +6,7 @@ regardless of whether ``_thread`` was available which is not desired. """ + from sys import modules as sys_modules import _dummy_thread @@ -19,35 +20,38 @@ # Could have checked if ``_thread`` was not in sys.modules and gone # a different route, but decided to mirror technique used with # ``threading`` below. - if '_thread' in sys_modules: - held_thread = sys_modules['_thread'] + if "_thread" in sys_modules: + held_thread = sys_modules["_thread"] holding_thread = True # Must have some module named ``_thread`` that implements its API # in order to initially import ``threading``. - sys_modules['_thread'] = sys_modules['_dummy_thread'] + sys_modules["_thread"] = sys_modules["_dummy_thread"] - if 'threading' in sys_modules: + if "threading" in sys_modules: # If ``threading`` is already imported, might as well prevent # trying to import it more than needed by saving it if it is # already imported before deleting it. - held_threading = sys_modules['threading'] + held_threading = sys_modules["threading"] holding_threading = True - del sys_modules['threading'] + del sys_modules["threading"] - if '_threading_local' in sys_modules: + if "_threading_local" in sys_modules: # If ``_threading_local`` is already imported, might as well prevent # trying to import it more than needed by saving it if it is # already imported before deleting it. - held__threading_local = sys_modules['_threading_local'] + held__threading_local = sys_modules["_threading_local"] holding__threading_local = True - del sys_modules['_threading_local'] + del sys_modules["_threading_local"] import threading + # Need a copy of the code kept somewhere... - sys_modules['_dummy_threading'] = sys_modules['threading'] - del sys_modules['threading'] - sys_modules['_dummy__threading_local'] = sys_modules['_threading_local'] - del sys_modules['_threading_local'] + sys_modules["_dummy_threading"] = sys_modules["threading"] + del sys_modules["threading"] + # _threading_local may not be imported if _thread._local is available + if "_threading_local" in sys_modules: + sys_modules["_dummy__threading_local"] = sys_modules["_threading_local"] + del sys_modules["_threading_local"] from _dummy_threading import * from _dummy_threading import __all__ @@ -55,23 +59,23 @@ # Put back ``threading`` if we overwrote earlier if holding_threading: - sys_modules['threading'] = held_threading + sys_modules["threading"] = held_threading del held_threading del holding_threading # Put back ``_threading_local`` if we overwrote earlier if holding__threading_local: - sys_modules['_threading_local'] = held__threading_local + sys_modules["_threading_local"] = held__threading_local del held__threading_local del holding__threading_local # Put back ``thread`` if we overwrote, else del the entry we made if holding_thread: - sys_modules['_thread'] = held_thread + sys_modules["_thread"] = held_thread del held_thread else: - del sys_modules['_thread'] + del sys_modules["_thread"] del holding_thread del _dummy_thread diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py index ec2215a5e5f..91243378dc0 100644 --- a/Lib/email/_header_value_parser.py +++ b/Lib/email/_header_value_parser.py @@ -95,8 +95,16 @@ NLSET = {'\n', '\r'} SPECIALSNL = SPECIALS | NLSET + +def make_quoted_pairs(value): + """Escape dquote and backslash for use within a quoted-string.""" + return str(value).replace('\\', '\\\\').replace('"', '\\"') + + def quote_string(value): - return '"'+str(value).replace('\\', '\\\\').replace('"', r'\"')+'"' + escaped = make_quoted_pairs(value) + return f'"{escaped}"' + # Match a RFC 2047 word, looks like =?utf-8?q?someword?= rfc2047_matcher = re.compile(r''' @@ -1012,6 +1020,8 @@ def _get_ptext_to_endchars(value, endchars): a flag that is True iff there were any quoted printables decoded. """ + if not value: + return '', '', False fragment, *remainder = _wsp_splitter(value, 1) vchars = [] escape = False @@ -1045,7 +1055,7 @@ def get_fws(value): fws = WhiteSpaceTerminal(value[:len(value)-len(newvalue)], 'fws') return fws, newvalue -def get_encoded_word(value): +def get_encoded_word(value, terminal_type='vtext'): """ encoded-word = "=?" charset "?" encoding "?" encoded-text "?=" """ @@ -1084,7 +1094,7 @@ def get_encoded_word(value): ew.append(token) continue chars, *remainder = _wsp_splitter(text, 1) - vtext = ValueTerminal(chars, 'vtext') + vtext = ValueTerminal(chars, terminal_type) _validate_xtext(vtext) ew.append(vtext) text = ''.join(remainder) @@ -1126,7 +1136,7 @@ def get_unstructured(value): valid_ew = True if value.startswith('=?'): try: - token, value = get_encoded_word(value) + token, value = get_encoded_word(value, 'utext') except _InvalidEwError: valid_ew = False except errors.HeaderParseError: @@ -1155,7 +1165,7 @@ def get_unstructured(value): # the parser to go in an infinite loop. if valid_ew and rfc2047_matcher.search(tok): tok, *remainder = value.partition('=?') - vtext = ValueTerminal(tok, 'vtext') + vtext = ValueTerminal(tok, 'utext') _validate_xtext(vtext) unstructured.append(vtext) value = ''.join(remainder) @@ -1565,7 +1575,7 @@ def get_dtext(value): def _check_for_early_dl_end(value, domain_literal): if value: return False - domain_literal.append(errors.InvalidHeaderDefect( + domain_literal.defects.append(errors.InvalidHeaderDefect( "end of input inside domain-literal")) domain_literal.append(ValueTerminal(']', 'domain-literal-end')) return True @@ -1584,9 +1594,9 @@ def get_domain_literal(value): raise errors.HeaderParseError("expected '[' at start of domain-literal " "but found '{}'".format(value)) value = value[1:] + domain_literal.append(ValueTerminal('[', 'domain-literal-start')) if _check_for_early_dl_end(value, domain_literal): return domain_literal, value - domain_literal.append(ValueTerminal('[', 'domain-literal-start')) if value[0] in WSP: token, value = get_fws(value) domain_literal.append(token) @@ -2805,7 +2815,7 @@ def _refold_parse_tree(parse_tree, *, policy): continue tstr = str(part) if not want_encoding: - if part.token_type == 'ptext': + if part.token_type in ('ptext', 'vtext'): # Encode if tstr contains special characters. want_encoding = not SPECIALSNL.isdisjoint(tstr) else: @@ -2905,6 +2915,15 @@ def _refold_parse_tree(parse_tree, *, policy): if not hasattr(part, 'encode'): # It's not a terminal, try folding the subparts. newparts = list(part) + if part.token_type == 'bare-quoted-string': + # To fold a quoted string we need to create a list of terminal + # tokens that will render the leading and trailing quotes + # and use quoted pairs in the value as appropriate. + newparts = ( + [ValueTerminal('"', 'ptext')] + + [ValueTerminal(make_quoted_pairs(p), 'ptext') + for p in newparts] + + [ValueTerminal('"', 'ptext')]) if not part.as_ew_allowed: wrap_as_ew_blocked += 1 newparts.append(end_ew_not_allowed) diff --git a/Lib/email/_parseaddr.py b/Lib/email/_parseaddr.py index 0f1bf8e4253..565af0cf361 100644 --- a/Lib/email/_parseaddr.py +++ b/Lib/email/_parseaddr.py @@ -146,8 +146,9 @@ def _parsedate_tz(data): return None # Check for a yy specified in two-digit format, then convert it to the # appropriate four-digit format, according to the POSIX standard. RFC 822 - # calls for a two-digit yy, but RFC 2822 (which obsoletes RFC 822) - # mandates a 4-digit yy. For more information, see the documentation for + # calls for a two-digit yy, but RFC 2822 (which obsoletes RFC 822) already + # mandated a 4-digit yy, and RFC 5322 (which obsoletes RFC 2822) continues + # this requirement. For more information, see the documentation for # the time module. if yy < 100: # The year is between 1969 and 1999 (inclusive). @@ -233,9 +234,11 @@ def __init__(self, field): self.CR = '\r\n' self.FWS = self.LWS + self.CR self.atomends = self.specials + self.LWS + self.CR - # Note that RFC 2822 now specifies `.' as obs-phrase, meaning that it - # is obsolete syntax. RFC 2822 requires that we recognize obsolete - # syntax, so allow dots in phrases. + # Note that RFC 2822 section 4.1 introduced '.' as obs-phrase to handle + # existing practice (periods in display names), even though it was not + # allowed in RFC 822. RFC 5322 section 4.1 (which obsoletes RFC 2822) + # continues this requirement. We must recognize obsolete syntax, so + # allow dots in phrases. self.phraseends = self.atomends.replace('.', '') self.field = field self.commentlist = [] diff --git a/Lib/email/_policybase.py b/Lib/email/_policybase.py index c9f0d743090..0d486c90a9c 100644 --- a/Lib/email/_policybase.py +++ b/Lib/email/_policybase.py @@ -370,7 +370,7 @@ def _fold(self, name, value, sanitize): h = value if h is not None: # The Header class interprets a value of None for maxlinelen as the - # default value of 78, as recommended by RFC 2822. + # default value of 78, as recommended by RFC 5322 section 2.1.1. maxlinelen = 0 if self.max_line_length is not None: maxlinelen = self.max_line_length diff --git a/Lib/email/contentmanager.py b/Lib/email/contentmanager.py index b4f5830bead..11d1536db27 100644 --- a/Lib/email/contentmanager.py +++ b/Lib/email/contentmanager.py @@ -2,6 +2,7 @@ import email.charset import email.message import email.errors +import sys from email import quoprimime class ContentManager: @@ -142,13 +143,15 @@ def _encode_base64(data, max_line_length): def _encode_text(string, charset, cte, policy): + # If max_line_length is 0 or None, there is no limit. + maxlen = policy.max_line_length or sys.maxsize lines = string.encode(charset).splitlines() linesep = policy.linesep.encode('ascii') def embedded_body(lines): return linesep.join(lines) + linesep def normal_body(lines): return b'\n'.join(lines) + b'\n' if cte is None: # Use heuristics to decide on the "best" encoding. - if max((len(x) for x in lines), default=0) <= policy.max_line_length: + if max(map(len, lines), default=0) <= maxlen: try: return '7bit', normal_body(lines).decode('ascii') except UnicodeDecodeError: @@ -156,8 +159,7 @@ def normal_body(lines): return b'\n'.join(lines) + b'\n' if policy.cte_type == '8bit': return '8bit', normal_body(lines).decode('ascii', 'surrogateescape') sniff = embedded_body(lines[:10]) - sniff_qp = quoprimime.body_encode(sniff.decode('latin-1'), - policy.max_line_length) + sniff_qp = quoprimime.body_encode(sniff.decode('latin-1'), maxlen) sniff_base64 = binascii.b2a_base64(sniff) # This is a little unfair to qp; it includes lineseps, base64 doesn't. if len(sniff_qp) > len(sniff_base64): @@ -172,9 +174,9 @@ def normal_body(lines): return b'\n'.join(lines) + b'\n' data = normal_body(lines).decode('ascii', 'surrogateescape') elif cte == 'quoted-printable': data = quoprimime.body_encode(normal_body(lines).decode('latin-1'), - policy.max_line_length) + maxlen) elif cte == 'base64': - data = _encode_base64(embedded_body(lines), policy.max_line_length) + data = _encode_base64(embedded_body(lines), maxlen) else: raise ValueError("Unknown content transfer encoding {}".format(cte)) return cte, data diff --git a/Lib/email/feedparser.py b/Lib/email/feedparser.py index 06d6b4a3afc..bc773f38030 100644 --- a/Lib/email/feedparser.py +++ b/Lib/email/feedparser.py @@ -32,7 +32,7 @@ NLCRE_bol = re.compile(r'(\r\n|\r|\n)') NLCRE_eol = re.compile(r'(\r\n|\r|\n)\Z') NLCRE_crack = re.compile(r'(\r\n|\r|\n)') -# RFC 2822 $3.6.8 Optional fields. ftext is %d33-57 / %d59-126, Any character +# RFC 5322 section 3.6.8 Optional fields. ftext is %d33-57 / %d59-126, Any character # except controls, SP, and ":". headerRE = re.compile(r'^(From |[\041-\071\073-\176]*:|[\t ])') EMPTYSTRING = '' @@ -294,7 +294,7 @@ def _parsegen(self): return if self._cur.get_content_maintype() == 'message': # The message claims to be a message/* type, then what follows is - # another RFC 2822 message. + # another RFC 5322 message. for retval in self._parsegen(): if retval is NeedMoreData: yield NeedMoreData diff --git a/Lib/email/generator.py b/Lib/email/generator.py index 47b9df8f4e6..ce94f5c56fe 100644 --- a/Lib/email/generator.py +++ b/Lib/email/generator.py @@ -50,7 +50,7 @@ def __init__(self, outfp, mangle_from_=None, maxheaderlen=None, *, expanded to 8 spaces) than maxheaderlen, the header will split as defined in the Header class. Set maxheaderlen to zero to disable header wrapping. The default is 78, as recommended (but not required) - by RFC 2822. + by RFC 5322 section 2.1.1. The policy keyword specifies a policy object that controls a number of aspects of the generator's operation. If no policy is specified, diff --git a/Lib/email/header.py b/Lib/email/header.py index 984851a7d9a..a0aadb97ca6 100644 --- a/Lib/email/header.py +++ b/Lib/email/header.py @@ -59,16 +59,22 @@ def decode_header(header): """Decode a message header value without converting charset. - Returns a list of (string, charset) pairs containing each of the decoded - parts of the header. Charset is None for non-encoded parts of the header, - otherwise a lower-case string containing the name of the character set - specified in the encoded string. + For historical reasons, this function may return either: + + 1. A list of length 1 containing a pair (str, None). + 2. A list of (bytes, charset) pairs containing each of the decoded + parts of the header. Charset is None for non-encoded parts of the header, + otherwise a lower-case string containing the name of the character set + specified in the encoded string. header may be a string that may or may not contain RFC2047 encoded words, or it may be a Header object. An email.errors.HeaderParseError may be raised when certain decoding error occurs (e.g. a base64 decoding exception). + + This function exists for backwards compatibility only. For new code, we + recommend using email.headerregistry.HeaderRegistry instead. """ # If it is a Header object, we can just return the encoded chunks. if hasattr(header, '_chunks'): @@ -161,6 +167,9 @@ def make_header(decoded_seq, maxlinelen=None, header_name=None, This function takes one of those sequence of pairs and returns a Header instance. Optional maxlinelen, header_name, and continuation_ws are as in the Header constructor. + + This function exists for backwards compatibility only, and is not + recommended for use in new code. """ h = Header(maxlinelen=maxlinelen, header_name=header_name, continuation_ws=continuation_ws) diff --git a/Lib/email/message.py b/Lib/email/message.py index 46bb8c21942..80f01d66a33 100644 --- a/Lib/email/message.py +++ b/Lib/email/message.py @@ -74,19 +74,25 @@ def _parseparam(s): # RDM This might be a Header, so for now stringify it. s = ';' + str(s) plist = [] - while s[:1] == ';': - s = s[1:] - end = s.find(';') - while end > 0 and (s.count('"', 0, end) - s.count('\\"', 0, end)) % 2: - end = s.find(';', end + 1) + start = 0 + while s.find(';', start) == start: + start += 1 + end = s.find(';', start) + ind, diff = start, 0 + while end > 0: + diff += s.count('"', ind, end) - s.count('\\"', ind, end) + if diff % 2 == 0: + break + end, ind = ind, s.find(';', end + 1) if end < 0: end = len(s) - f = s[:end] - if '=' in f: - i = f.index('=') - f = f[:i].strip().lower() + '=' + f[i+1:].strip() + i = s.find('=', start, end) + if i == -1: + f = s[start:end] + else: + f = s[start:i].rstrip().lower() + '=' + s[i+1:end].lstrip() plist.append(f.strip()) - s = s[end:] + start = end return plist @@ -135,7 +141,7 @@ def _decode_uu(encoded): class Message: """Basic message object. - A message object is defined as something that has a bunch of RFC 2822 + A message object is defined as something that has a bunch of RFC 5322 headers and a payload. It may optionally have an envelope header (a.k.a. Unix-From or From_ header). If the message is a container (i.e. a multipart or a message/rfc822), then the payload is a list of Message @@ -286,8 +292,12 @@ def get_payload(self, i=None, decode=False): if i is not None and not isinstance(self._payload, list): raise TypeError('Expected list, got %s' % type(self._payload)) payload = self._payload - # cte might be a Header, so for now stringify it. - cte = str(self.get('content-transfer-encoding', '')).lower() + cte = self.get('content-transfer-encoding', '') + if hasattr(cte, 'cte'): + cte = cte.cte + else: + # cte might be a Header, so for now stringify it. + cte = str(cte).strip().lower() # payload may be bytes here. if not decode: if isinstance(payload, str) and utils._has_surrogates(payload): @@ -309,6 +319,8 @@ def get_payload(self, i=None, decode=False): # If it does happen, turn the string into bytes in a way # guaranteed not to fail. bpayload = payload.encode('raw-unicode-escape') + else: + bpayload = payload if cte == 'quoted-printable': return quopri.decodestring(bpayload) elif cte == 'base64': @@ -560,7 +572,7 @@ def add_header(self, _name, _value, **_params): msg.add_header('content-disposition', 'attachment', filename='bud.gif') msg.add_header('content-disposition', 'attachment', - filename=('utf-8', '', Fußballer.ppt')) + filename=('utf-8', '', 'Fußballer.ppt')) msg.add_header('content-disposition', 'attachment', filename='Fußballer.ppt')) """ diff --git a/Lib/email/parser.py b/Lib/email/parser.py index 06d99b17f2f..e3003118ce1 100644 --- a/Lib/email/parser.py +++ b/Lib/email/parser.py @@ -2,7 +2,7 @@ # Author: Barry Warsaw, Thomas Wouters, Anthony Baxter # Contact: email-sig@python.org -"""A parser of RFC 2822 and MIME email messages.""" +"""A parser of RFC 5322 and MIME email messages.""" __all__ = ['Parser', 'HeaderParser', 'BytesParser', 'BytesHeaderParser', 'FeedParser', 'BytesFeedParser'] @@ -15,14 +15,14 @@ class Parser: def __init__(self, _class=None, *, policy=compat32): - """Parser of RFC 2822 and MIME email messages. + """Parser of RFC 5322 and MIME email messages. Creates an in-memory object tree representing the email message, which can then be manipulated and turned over to a Generator to return the textual representation of the message. - The string must be formatted as a block of RFC 2822 headers and header - continuation lines, optionally preceded by a `Unix-from' header. The + The string must be formatted as a block of RFC 5322 headers and header + continuation lines, optionally preceded by a 'Unix-from' header. The header block is terminated either by the end of the string or by a blank line. @@ -75,14 +75,14 @@ def parsestr(self, text, headersonly=True): class BytesParser: def __init__(self, *args, **kw): - """Parser of binary RFC 2822 and MIME email messages. + """Parser of binary RFC 5322 and MIME email messages. Creates an in-memory object tree representing the email message, which can then be manipulated and turned over to a Generator to return the textual representation of the message. - The input must be formatted as a block of RFC 2822 headers and header - continuation lines, optionally preceded by a `Unix-from' header. The + The input must be formatted as a block of RFC 5322 headers and header + continuation lines, optionally preceded by a 'Unix-from' header. The header block is terminated either by the end of the input or by a blank line. diff --git a/Lib/email/utils.py b/Lib/email/utils.py index e42674fa4f3..e4d35f06abc 100644 --- a/Lib/email/utils.py +++ b/Lib/email/utils.py @@ -417,8 +417,14 @@ def decode_params(params): for name, continuations in rfc2231_params.items(): value = [] extended = False - # Sort by number - continuations.sort() + # Sort by number, treating None as 0 if there is no 0, + # and ignore it if there is already a 0. + has_zero = any(x[0] == 0 for x in continuations) + if has_zero: + continuations = [x for x in continuations if x[0] is not None] + else: + continuations = [(x[0] or 0, x[1], x[2]) for x in continuations] + continuations.sort(key=lambda x: x[0]) # And now append all values in numerical order, converting # %-encodings for the encoded segments. If any of the # continuation names ends in a *, then the entire string, after diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py index ab6d32478e4..21bbfad0fe6 100644 --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -10,7 +10,7 @@ __all__ = ["version", "bootstrap"] -_PIP_VERSION = "25.2" +_PIP_VERSION = "25.3" # Directory of system wheel packages. Some Linux distribution packaging # policies recommend against bundling dependencies. For example, Fedora @@ -205,7 +205,7 @@ def _uninstall_helper(*, verbosity=0): def _main(argv=None): import argparse - parser = argparse.ArgumentParser(prog="python -m ensurepip") + parser = argparse.ArgumentParser(color=True) parser.add_argument( "--version", action="version", diff --git a/Lib/ensurepip/_bundled/pip-25.2-py3-none-any.whl b/Lib/ensurepip/_bundled/pip-25.3-py3-none-any.whl similarity index 78% rename from Lib/ensurepip/_bundled/pip-25.2-py3-none-any.whl rename to Lib/ensurepip/_bundled/pip-25.3-py3-none-any.whl index 4db7e720718..755e1aa0c3d 100644 Binary files a/Lib/ensurepip/_bundled/pip-25.2-py3-none-any.whl and b/Lib/ensurepip/_bundled/pip-25.3-py3-none-any.whl differ diff --git a/Lib/ensurepip/_uninstall.py b/Lib/ensurepip/_uninstall.py index b257904328d..4183c28a809 100644 --- a/Lib/ensurepip/_uninstall.py +++ b/Lib/ensurepip/_uninstall.py @@ -6,7 +6,7 @@ def _main(argv=None): - parser = argparse.ArgumentParser(prog="python -m ensurepip._uninstall") + parser = argparse.ArgumentParser() parser.add_argument( "--version", action="version", diff --git a/Lib/enum.py b/Lib/enum.py index 3adb208c7e6..8a72c409b94 100644 --- a/Lib/enum.py +++ b/Lib/enum.py @@ -1,6 +1,5 @@ import sys import builtins as bltns -from functools import partial from types import MappingProxyType, DynamicClassAttribute @@ -38,7 +37,7 @@ def _is_descriptor(obj): """ Returns True if obj is a descriptor, False otherwise. """ - return not isinstance(obj, partial) and ( + return ( hasattr(obj, '__get__') or hasattr(obj, '__set__') or hasattr(obj, '__delete__') @@ -151,18 +150,6 @@ def bin(num, max_bits=None): digits = (sign[-1] * max_bits + digits)[-max_bits:] return "%s %s" % (sign, digits) -def _dedent(text): - """ - Like textwrap.dedent. Rewritten because we cannot import textwrap. - """ - lines = text.split('\n') - for i, ch in enumerate(lines[0]): - if ch != ' ': - break - for j, l in enumerate(lines): - lines[j] = l[i:] - return '\n'.join(lines) - class _not_given: def __repr__(self): return('') @@ -208,7 +195,7 @@ def __get__(self, instance, ownerclass=None): # use previous enum.property return self.fget(instance) elif self._attr_type == 'attr': - # look up previous attibute + # look up previous attribute return getattr(self._cls_type, self.name) elif self._attr_type == 'desc': # use previous descriptor @@ -406,12 +393,6 @@ def __setitem__(self, key, value): elif isinstance(value, nonmember): # unwrap value here; it won't be processed by the below `else` value = value.value - elif isinstance(value, partial): - import warnings - warnings.warn('functools.partial will be a method descriptor ' - 'in future Python versions; wrap it in ' - 'enum.member() if you want to preserve the ' - 'old behavior', FutureWarning, stacklevel=2) elif _is_descriptor(value): pass elif self._cls_name is not None and _is_internal_class(self._cls_name, value): @@ -1103,6 +1084,21 @@ def _add_member_(cls, name, member): # now add to _member_map_ (even aliases) cls._member_map_[name] = member + @property + def __signature__(cls): + from inspect import Parameter, Signature + if cls._member_names_: + return Signature([Parameter('values', Parameter.VAR_POSITIONAL)]) + else: + return Signature([Parameter('new_class_name', Parameter.POSITIONAL_ONLY), + Parameter('names', Parameter.POSITIONAL_OR_KEYWORD), + Parameter('module', Parameter.KEYWORD_ONLY, default=None), + Parameter('qualname', Parameter.KEYWORD_ONLY, default=None), + Parameter('type', Parameter.KEYWORD_ONLY, default=None), + Parameter('start', Parameter.KEYWORD_ONLY, default=1), + Parameter('boundary', Parameter.KEYWORD_ONLY, default=None)]) + + EnumMeta = EnumType # keep EnumMeta name for backwards compatibility @@ -1146,13 +1142,6 @@ class Enum(metaclass=EnumType): attributes -- see the documentation for details. """ - @classmethod - def __signature__(cls): - if cls._member_names_: - return '(*values)' - else: - return '(new_class_name, /, names, *, module=None, qualname=None, type=None, start=1, boundary=None)' - def __new__(cls, value): # all enum instances are actually created during class construction # without calling this method; this method is called by the metaclass' @@ -1214,9 +1203,6 @@ def __new__(cls, value): exc = None ve_exc = None - def __init__(self, *args, **kwds): - pass - def _add_alias_(self, name): self.__class__._add_member_(name, self) @@ -2041,8 +2027,7 @@ def _test_simple_enum(checked_enum, simple_enum): ... RED = auto() ... GREEN = auto() ... BLUE = auto() - ... # TODO: RUSTPYTHON - >>> _test_simple_enum(CheckedColor, Color) # doctest: +SKIP + >>> _test_simple_enum(CheckedColor, Color) If differences are found, a :exc:`TypeError` is raised. """ diff --git a/Lib/fractions.py b/Lib/fractions.py index 9d42e809875..a497ee19935 100644 --- a/Lib/fractions.py +++ b/Lib/fractions.py @@ -3,7 +3,6 @@ """Fraction, infinite-precision, rational numbers.""" -from decimal import Decimal import functools import math import numbers @@ -65,7 +64,7 @@ def _hash_algorithm(numerator, denominator): (?:\.(?P\d*|\d+(_\d+)*))? # an optional fractional part (?:E(?P[-+]?\d+(_\d+)*))? # and optional exponent ) - \s*\Z # and optional whitespace to finish + \s*\z # and optional whitespace to finish """, re.VERBOSE | re.IGNORECASE) @@ -169,9 +168,13 @@ def _round_to_figures(n, d, figures): # A '0' that's *not* followed by another digit is parsed as a minimum width # rather than a zeropad flag. (?P0(?=[0-9]))? - (?P0|[1-9][0-9]*)? + (?P[0-9]+)? (?P[,_])? - (?:\.(?P0|[1-9][0-9]*))? + (?:\. + (?=[,_0-9]) # lookahead for digit or separator + (?P[0-9]+)? + (?P[,_])? + )? (?P[eEfFgG%]) """, re.DOTALL | re.VERBOSE).fullmatch @@ -244,7 +247,9 @@ def __new__(cls, numerator=0, denominator=None): self._denominator = numerator.denominator return self - elif isinstance(numerator, (float, Decimal)): + elif (isinstance(numerator, float) or + (not isinstance(numerator, type) and + hasattr(numerator, 'as_integer_ratio'))): # Exact conversion self._numerator, self._denominator = numerator.as_integer_ratio() return self @@ -278,8 +283,8 @@ def __new__(cls, numerator=0, denominator=None): numerator = -numerator else: - raise TypeError("argument should be a string " - "or a Rational instance") + raise TypeError("argument should be a string or a Rational " + "instance or have the as_integer_ratio() method") elif type(numerator) is int is type(denominator): pass # *very* normal case @@ -305,6 +310,28 @@ def __new__(cls, numerator=0, denominator=None): self._denominator = denominator return self + @classmethod + def from_number(cls, number): + """Converts a finite real number to a rational number, exactly. + + Beware that Fraction.from_number(0.3) != Fraction(3, 10). + + """ + if type(number) is int: + return cls._from_coprime_ints(number, 1) + + elif isinstance(number, numbers.Rational): + return cls._from_coprime_ints(number.numerator, number.denominator) + + elif (isinstance(number, float) or + (not isinstance(number, type) and + hasattr(number, 'as_integer_ratio'))): + return cls._from_coprime_ints(*number.as_integer_ratio()) + + else: + raise TypeError("argument should be a Rational instance or " + "have the as_integer_ratio() method") + @classmethod def from_float(cls, f): """Converts a finite float to a rational number, exactly. @@ -476,6 +503,7 @@ def _format_float_style(self, match): minimumwidth = int(match["minimumwidth"] or "0") thousands_sep = match["thousands_sep"] precision = int(match["precision"] or "6") + frac_sep = match["frac_separators"] or "" presentation_type = match["presentation_type"] trim_zeros = presentation_type in "gG" and not alternate_form trim_point = not alternate_form @@ -532,6 +560,9 @@ def _format_float_style(self, match): if trim_zeros: frac_part = frac_part.rstrip("0") separator = "" if trim_point and not frac_part else "." + if frac_sep: + frac_part = frac_sep.join(frac_part[pos:pos + 3] + for pos in range(0, len(frac_part), 3)) trailing = separator + frac_part + suffix # Do zero padding if required. @@ -671,7 +702,7 @@ def forward(a, b): elif isinstance(b, float): return fallback_operator(float(a), b) elif handle_complex and isinstance(b, complex): - return fallback_operator(complex(a), b) + return fallback_operator(float(a), b) else: return NotImplemented forward.__name__ = '__' + fallback_operator.__name__ + '__' @@ -684,7 +715,7 @@ def reverse(b, a): elif isinstance(a, numbers.Real): return fallback_operator(float(a), float(b)) elif handle_complex and isinstance(a, numbers.Complex): - return fallback_operator(complex(a), complex(b)) + return fallback_operator(complex(a), float(b)) else: return NotImplemented reverse.__name__ = '__r' + fallback_operator.__name__ + '__' @@ -851,7 +882,7 @@ def _mod(a, b): __mod__, __rmod__ = _operator_fallbacks(_mod, operator.mod, False) - def __pow__(a, b): + def __pow__(a, b, modulo=None): """a ** b If b is not an integer, the result will be a float or complex @@ -859,6 +890,8 @@ def __pow__(a, b): result will be rational. """ + if modulo is not None: + return NotImplemented if isinstance(b, numbers.Rational): if b.denominator == 1: power = b.numerator @@ -883,8 +916,10 @@ def __pow__(a, b): else: return NotImplemented - def __rpow__(b, a): + def __rpow__(b, a, modulo=None): """a ** b""" + if modulo is not None: + return NotImplemented if b._denominator == 1 and b._numerator >= 0: # If a is an int, keep it that way if possible. return a ** b._numerator diff --git a/Lib/ftplib.py b/Lib/ftplib.py index 10c5d1ea08a..50771e8c17c 100644 --- a/Lib/ftplib.py +++ b/Lib/ftplib.py @@ -343,7 +343,7 @@ def ntransfercmd(self, cmd, rest=None): connection and the expected size of the transfer. The expected size may be None if it could not be determined. - Optional `rest' argument can be a string that is sent as the + Optional 'rest' argument can be a string that is sent as the argument to a REST command. This is essentially a server marker used to tell the server to skip over any data up to the given marker. diff --git a/Lib/functools.py b/Lib/functools.py index 4c1175b815d..df4660eef3f 100644 --- a/Lib/functools.py +++ b/Lib/functools.py @@ -6,23 +6,22 @@ # Written by Nick Coghlan , # Raymond Hettinger , # and Łukasz Langa . -# Copyright (C) 2006-2013 Python Software Foundation. +# Copyright (C) 2006 Python Software Foundation. # See C source code for _functools credits/copyright __all__ = ['update_wrapper', 'wraps', 'WRAPPER_ASSIGNMENTS', 'WRAPPER_UPDATES', 'total_ordering', 'cache', 'cmp_to_key', 'lru_cache', 'reduce', 'partial', 'partialmethod', 'singledispatch', 'singledispatchmethod', - 'cached_property'] + 'cached_property', 'Placeholder'] from abc import get_cache_token from collections import namedtuple -# import types, weakref # Deferred to single_dispatch() +# import weakref # Deferred to single_dispatch() +from operator import itemgetter from reprlib import recursive_repr +from types import GenericAlias, MethodType, MappingProxyType, UnionType from _thread import RLock -# Avoid importing types, so we can speedup import time -GenericAlias = type(list[int]) - ################################################################################ ### update_wrapper() and wraps() decorator ################################################################################ @@ -31,7 +30,7 @@ # wrapper functions that can handle naive introspection WRAPPER_ASSIGNMENTS = ('__module__', '__name__', '__qualname__', '__doc__', - '__annotations__', '__type_params__') + '__annotate__', '__type_params__') WRAPPER_UPDATES = ('__dict__',) def update_wrapper(wrapper, wrapped, @@ -237,7 +236,7 @@ def __ge__(self, other): def reduce(function, sequence, initial=_initial_missing): """ - reduce(function, iterable[, initial], /) -> value + reduce(function, iterable, /[, initial]) -> value Apply a function of two arguments cumulatively to the items of an iterable, from left to right. @@ -265,63 +264,138 @@ def reduce(function, sequence, initial=_initial_missing): return value -try: - from _functools import reduce -except ImportError: - pass - ################################################################################ ### partial() argument application ################################################################################ -# Purely functional, no descriptor behaviour -class partial: - """New function with partial application of the given arguments - and keywords. + +class _PlaceholderType: + """The type of the Placeholder singleton. + + Used as a placeholder for partial arguments. """ + __instance = None + __slots__ = () - __slots__ = "func", "args", "keywords", "__dict__", "__weakref__" + def __init_subclass__(cls, *args, **kwargs): + raise TypeError(f"type '{cls.__name__}' is not an acceptable base type") - def __new__(cls, func, /, *args, **keywords): + def __new__(cls): + if cls.__instance is None: + cls.__instance = object.__new__(cls) + return cls.__instance + + def __repr__(self): + return 'Placeholder' + + def __reduce__(self): + return 'Placeholder' + +Placeholder = _PlaceholderType() + +def _partial_prepare_merger(args): + if not args: + return 0, None + nargs = len(args) + order = [] + j = nargs + for i, a in enumerate(args): + if a is Placeholder: + order.append(j) + j += 1 + else: + order.append(i) + phcount = j - nargs + merger = itemgetter(*order) if phcount else None + return phcount, merger + +def _partial_new(cls, func, /, *args, **keywords): + if issubclass(cls, partial): + base_cls = partial if not callable(func): raise TypeError("the first argument must be callable") + else: + base_cls = partialmethod + # func could be a descriptor like classmethod which isn't callable + if not callable(func) and not hasattr(func, "__get__"): + raise TypeError(f"the first argument {func!r} must be a callable " + "or a descriptor") + if args and args[-1] is Placeholder: + raise TypeError("trailing Placeholders are not allowed") + for value in keywords.values(): + if value is Placeholder: + raise TypeError("Placeholder cannot be passed as a keyword argument") + if isinstance(func, base_cls): + pto_phcount = func._phcount + tot_args = func.args + if args: + tot_args += args + if pto_phcount: + # merge args with args of `func` which is `partial` + nargs = len(args) + if nargs < pto_phcount: + tot_args += (Placeholder,) * (pto_phcount - nargs) + tot_args = func._merger(tot_args) + if nargs > pto_phcount: + tot_args += args[pto_phcount:] + phcount, merger = _partial_prepare_merger(tot_args) + else: # works for both pto_phcount == 0 and != 0 + phcount, merger = pto_phcount, func._merger + keywords = {**func.keywords, **keywords} + func = func.func + else: + tot_args = args + phcount, merger = _partial_prepare_merger(tot_args) + + self = object.__new__(cls) + self.func = func + self.args = tot_args + self.keywords = keywords + self._phcount = phcount + self._merger = merger + return self + +def _partial_repr(self): + cls = type(self) + module = cls.__module__ + qualname = cls.__qualname__ + args = [repr(self.func)] + args.extend(map(repr, self.args)) + args.extend(f"{k}={v!r}" for k, v in self.keywords.items()) + return f"{module}.{qualname}({', '.join(args)})" - if isinstance(func, partial): - args = func.args + args - keywords = {**func.keywords, **keywords} - func = func.func +# Purely functional, no descriptor behaviour +class partial: + """New function with partial application of the given arguments + and keywords. + """ - self = super(partial, cls).__new__(cls) + __slots__ = ("func", "args", "keywords", "_phcount", "_merger", + "__dict__", "__weakref__") - self.func = func - self.args = args - self.keywords = keywords - return self + __new__ = _partial_new + __repr__ = recursive_repr()(_partial_repr) def __call__(self, /, *args, **keywords): + phcount = self._phcount + if phcount: + try: + pto_args = self._merger(self.args + args) + args = args[phcount:] + except IndexError: + raise TypeError("missing positional arguments " + "in 'partial' call; expected " + f"at least {phcount}, got {len(args)}") + else: + pto_args = self.args keywords = {**self.keywords, **keywords} - return self.func(*self.args, *args, **keywords) - - @recursive_repr() - def __repr__(self): - cls = type(self) - qualname = cls.__qualname__ - module = cls.__module__ - args = [repr(self.func)] - args.extend(repr(x) for x in self.args) - args.extend(f"{k}={v!r}" for (k, v) in self.keywords.items()) - return f"{module}.{qualname}({', '.join(args)})" + return self.func(*pto_args, *args, **keywords) def __get__(self, obj, objtype=None): if obj is None: return self - import warnings - warnings.warn('functools.partial will be a method descriptor in ' - 'future Python versions; wrap it in staticmethod() ' - 'if you want to preserve the old behavior', - FutureWarning, 2) - return self + return MethodType(self, obj) def __reduce__(self): return type(self), (self.func,), (self.func, self.args, @@ -338,6 +412,10 @@ def __setstate__(self, state): (namespace is not None and not isinstance(namespace, dict))): raise TypeError("invalid partial state") + if args and args[-1] is Placeholder: + raise TypeError("trailing Placeholders are not allowed") + phcount, merger = _partial_prepare_merger(args) + args = tuple(args) # just in case it's a subclass if kwds is None: kwds = {} @@ -350,56 +428,43 @@ def __setstate__(self, state): self.func = func self.args = args self.keywords = kwds + self._phcount = phcount + self._merger = merger __class_getitem__ = classmethod(GenericAlias) try: - from _functools import partial + from _functools import partial, Placeholder, _PlaceholderType except ImportError: pass # Descriptor version -class partialmethod(object): +class partialmethod: """Method descriptor with partial application of the given arguments and keywords. Supports wrapping existing descriptors and handles non-descriptor callables as instance methods. """ - - def __init__(self, func, /, *args, **keywords): - if not callable(func) and not hasattr(func, "__get__"): - raise TypeError("{!r} is not callable or a descriptor" - .format(func)) - - # func could be a descriptor like classmethod which isn't callable, - # so we can't inherit from partial (it verifies func is callable) - if isinstance(func, partialmethod): - # flattening is mandatory in order to place cls/self before all - # other arguments - # it's also more efficient since only one function will be called - self.func = func.func - self.args = func.args + args - self.keywords = {**func.keywords, **keywords} - else: - self.func = func - self.args = args - self.keywords = keywords - - def __repr__(self): - cls = type(self) - module = cls.__module__ - qualname = cls.__qualname__ - args = [repr(self.func)] - args.extend(map(repr, self.args)) - args.extend(f"{k}={v!r}" for k, v in self.keywords.items()) - return f"{module}.{qualname}({', '.join(args)})" + __new__ = _partial_new + __repr__ = _partial_repr def _make_unbound_method(self): def _method(cls_or_self, /, *args, **keywords): + phcount = self._phcount + if phcount: + try: + pto_args = self._merger(self.args + args) + args = args[phcount:] + except IndexError: + raise TypeError("missing positional arguments " + "in 'partialmethod' call; expected " + f"at least {phcount}, got {len(args)}") + else: + pto_args = self.args keywords = {**self.keywords, **keywords} - return self.func(cls_or_self, *self.args, *args, **keywords) + return self.func(cls_or_self, *pto_args, *args, **keywords) _method.__isabstractmethod__ = self.__isabstractmethod__ _method.__partialmethod__ = self return _method @@ -407,7 +472,7 @@ def _method(cls_or_self, /, *args, **keywords): def __get__(self, obj, cls=None): get = getattr(self.func, "__get__", None) result = None - if get is not None and not isinstance(self.func, partial): + if get is not None: new_func = get(obj, cls) if new_func is not self.func: # Assume __get__ returning something new indicates the @@ -454,22 +519,6 @@ def _unwrap_partialmethod(func): _CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "maxsize", "currsize"]) -class _HashedSeq(list): - """ This class guarantees that hash() will be called no more than once - per element. This is important because the lru_cache() will hash - the key multiple times on a cache miss. - - """ - - __slots__ = 'hashvalue' - - def __init__(self, tup, hash=hash): - self[:] = tup - self.hashvalue = hash(tup) - - def __hash__(self): - return self.hashvalue - def _make_key(args, kwds, typed, kwd_mark = (object(),), fasttypes = {int, str}, @@ -499,7 +548,7 @@ def _make_key(args, kwds, typed, key += tuple(type(v) for v in kwds.values()) elif len(key) == 1 and type(key[0]) in fasttypes: return key[0] - return _HashedSeq(key) + return key def lru_cache(maxsize=128, typed=False): """Least-recently-used cache decorator. @@ -835,7 +884,7 @@ def singledispatch(func): # There are many programs that use functools without singledispatch, so we # trade-off making singledispatch marginally slower for the benefit of # making start-up of such applications slightly faster. - import types, weakref + import weakref registry = {} dispatch_cache = weakref.WeakKeyDictionary() @@ -864,16 +913,11 @@ def dispatch(cls): dispatch_cache[cls] = impl return impl - def _is_union_type(cls): - from typing import get_origin, Union - return get_origin(cls) in {Union, types.UnionType} - def _is_valid_dispatch_type(cls): if isinstance(cls, type): return True - from typing import get_args - return (_is_union_type(cls) and - all(isinstance(arg, type) for arg in get_args(cls))) + return (isinstance(cls, UnionType) and + all(isinstance(arg, type) for arg in cls.__args__)) def register(cls, func=None): """generic_func.register(cls, func) -> func @@ -891,8 +935,8 @@ def register(cls, func=None): f"Invalid first argument to `register()`. " f"{cls!r} is not a class or union type." ) - ann = getattr(cls, '__annotations__', {}) - if not ann: + ann = getattr(cls, '__annotate__', None) + if ann is None: raise TypeError( f"Invalid first argument to `register()`: {cls!r}. " f"Use either `@register(some_class)` or plain `@register` " @@ -902,23 +946,27 @@ def register(cls, func=None): # only import typing if annotation parsing is necessary from typing import get_type_hints - argname, cls = next(iter(get_type_hints(func).items())) + from annotationlib import Format, ForwardRef + argname, cls = next(iter(get_type_hints(func, format=Format.FORWARDREF).items())) if not _is_valid_dispatch_type(cls): - if _is_union_type(cls): + if isinstance(cls, UnionType): raise TypeError( f"Invalid annotation for {argname!r}. " f"{cls!r} not all arguments are classes." ) + elif isinstance(cls, ForwardRef): + raise TypeError( + f"Invalid annotation for {argname!r}. " + f"{cls!r} is an unresolved forward reference." + ) else: raise TypeError( f"Invalid annotation for {argname!r}. " f"{cls!r} is not a class." ) - if _is_union_type(cls): - from typing import get_args - - for arg in get_args(cls): + if isinstance(cls, UnionType): + for arg in cls.__args__: registry[arg] = func else: registry[cls] = func @@ -937,7 +985,7 @@ def wrapper(*args, **kw): registry[object] = func wrapper.register = register wrapper.dispatch = dispatch - wrapper.registry = types.MappingProxyType(registry) + wrapper.registry = MappingProxyType(registry) wrapper._clear_cache = dispatch_cache.clear update_wrapper(wrapper, func) return wrapper @@ -947,8 +995,7 @@ def wrapper(*args, **kw): class singledispatchmethod: """Single-dispatch generic method descriptor. - Supports wrapping existing descriptors and handles non-descriptor - callables as instance methods. + Supports wrapping existing descriptors. """ def __init__(self, func): @@ -966,24 +1013,77 @@ def register(self, cls, method=None): return self.dispatcher.register(cls, func=method) def __get__(self, obj, cls=None): - dispatch = self.dispatcher.dispatch - funcname = getattr(self.func, '__name__', 'singledispatchmethod method') - def _method(*args, **kwargs): - if not args: - raise TypeError(f'{funcname} requires at least ' - '1 positional argument') - return dispatch(args[0].__class__).__get__(obj, cls)(*args, **kwargs) - - _method.__isabstractmethod__ = self.__isabstractmethod__ - _method.register = self.register - update_wrapper(_method, self.func) - - return _method + return _singledispatchmethod_get(self, obj, cls) @property def __isabstractmethod__(self): return getattr(self.func, '__isabstractmethod__', False) + def __repr__(self): + try: + name = self.func.__qualname__ + except AttributeError: + try: + name = self.func.__name__ + except AttributeError: + name = '?' + return f'' + +class _singledispatchmethod_get: + def __init__(self, unbound, obj, cls): + self._unbound = unbound + self._dispatch = unbound.dispatcher.dispatch + self._obj = obj + self._cls = cls + # Set instance attributes which cannot be handled in __getattr__() + # because they conflict with type descriptors. + func = unbound.func + try: + self.__module__ = func.__module__ + except AttributeError: + pass + try: + self.__doc__ = func.__doc__ + except AttributeError: + pass + + def __repr__(self): + try: + name = self.__qualname__ + except AttributeError: + try: + name = self.__name__ + except AttributeError: + name = '?' + if self._obj is not None: + return f'' + else: + return f'' + + def __call__(self, /, *args, **kwargs): + if not args: + funcname = getattr(self._unbound.func, '__name__', + 'singledispatchmethod method') + raise TypeError(f'{funcname} requires at least ' + '1 positional argument') + return self._dispatch(args[0].__class__).__get__(self._obj, self._cls)(*args, **kwargs) + + def __getattr__(self, name): + # Resolve these attributes lazily to speed up creation of + # the _singledispatchmethod_get instance. + if name not in {'__name__', '__qualname__', '__isabstractmethod__', + '__annotations__', '__type_params__'}: + raise AttributeError + return getattr(self._unbound.func, name) + + @property + def __wrapped__(self): + return self._unbound.func + + @property + def register(self): + return self._unbound.register + ################################################################################ ### cached_property() - property result cached as instance attribute @@ -1035,3 +1135,31 @@ def __get__(self, instance, owner=None): return val __class_getitem__ = classmethod(GenericAlias) + +def _warn_python_reduce_kwargs(py_reduce): + @wraps(py_reduce) + def wrapper(*args, **kwargs): + if 'function' in kwargs or 'sequence' in kwargs: + import os + import warnings + warnings.warn( + 'Calling functools.reduce with keyword arguments ' + '"function" or "sequence" ' + 'is deprecated in Python 3.14 and will be ' + 'forbidden in Python 3.16.', + DeprecationWarning, + skip_file_prefixes=(os.path.dirname(__file__),)) + return py_reduce(*args, **kwargs) + return wrapper + +reduce = _warn_python_reduce_kwargs(reduce) +del _warn_python_reduce_kwargs + +# The import of the C accelerated version of reduce() has been moved +# here due to gh-121676. In Python 3.16, _warn_python_reduce_kwargs() +# should be removed and the import block should be moved back right +# after the definition of reduce(). +try: + from _functools import reduce +except ImportError: + pass diff --git a/Lib/gettext.py b/Lib/gettext.py index 62cff81b7b3..6c11ab2b1eb 100644 --- a/Lib/gettext.py +++ b/Lib/gettext.py @@ -41,14 +41,10 @@ # to do binary searches and lazy initializations. Or you might want to use # the undocumented double-hash algorithm for .mo files with hash tables, but # you'll need to study the GNU gettext code to do this. -# -# - Support Solaris .mo file formats. Unfortunately, we've been unable to -# find this format documented anywhere. import operator import os -import re import sys @@ -70,22 +66,26 @@ # https://www.gnu.org/software/gettext/manual/gettext.html#Plural-forms # http://git.savannah.gnu.org/cgit/gettext.git/tree/gettext-runtime/intl/plural.y -_token_pattern = re.compile(r""" - (?P[ \t]+) | # spaces and horizontal tabs - (?P[0-9]+\b) | # decimal integer - (?Pn\b) | # only n is allowed - (?P[()]) | - (?P[-*/%+?:]|[>, - # <=, >=, ==, !=, &&, ||, - # ? : - # unary and bitwise ops - # not allowed - (?P\w+|.) # invalid token - """, re.VERBOSE|re.DOTALL) - +_token_pattern = None def _tokenize(plural): - for mo in re.finditer(_token_pattern, plural): + global _token_pattern + if _token_pattern is None: + import re + _token_pattern = re.compile(r""" + (?P[ \t]+) | # spaces and horizontal tabs + (?P[0-9]+\b) | # decimal integer + (?Pn\b) | # only n is allowed + (?P[()]) | + (?P[-*/%+?:]|[>, + # <=, >=, ==, !=, &&, ||, + # ? : + # unary and bitwise ops + # not allowed + (?P\w+|.) # invalid token + """, re.VERBOSE|re.DOTALL) + + for mo in _token_pattern.finditer(plural): kind = mo.lastgroup if kind == 'WHITESPACES': continue @@ -648,7 +648,7 @@ def npgettext(context, msgid1, msgid2, n): # import gettext # cat = gettext.Catalog(PACKAGE, localedir=LOCALEDIR) # _ = cat.gettext -# print _('Hello World') +# print(_('Hello World')) # The resulting catalog object currently don't support access through a # dictionary API, which was supported (but apparently unused) in GNOME diff --git a/Lib/graphlib.py b/Lib/graphlib.py index 9512865a8e5..7961c9c5cac 100644 --- a/Lib/graphlib.py +++ b/Lib/graphlib.py @@ -90,20 +90,24 @@ def prepare(self): still be used to obtain as many nodes as possible until cycles block more progress. After a call to this function, the graph cannot be modified and therefore no more nodes can be added using "add". + + Raise ValueError if nodes have already been passed out of the sorter. + """ - if self._ready_nodes is not None: - raise ValueError("cannot prepare() more than once") + if self._npassedout > 0: + raise ValueError("cannot prepare() after starting sort") - self._ready_nodes = [ - i.node for i in self._node2info.values() if i.npredecessors == 0 - ] + if self._ready_nodes is None: + self._ready_nodes = [ + i.node for i in self._node2info.values() if i.npredecessors == 0 + ] # ready_nodes is set before we look for cycles on purpose: # if the user wants to catch the CycleError, that's fine, # they can continue using the instance to grab as many # nodes as possible before cycles block more progress cycle = self._find_cycle() if cycle: - raise CycleError(f"nodes are in a cycle", cycle) + raise CycleError("nodes are in a cycle", cycle) def get_ready(self): """Return a tuple of all the nodes that are ready. diff --git a/Lib/gzip.py b/Lib/gzip.py index a550c20a7a0..c00f51858de 100644 --- a/Lib/gzip.py +++ b/Lib/gzip.py @@ -5,7 +5,6 @@ # based on Andrew Kuchling's minigzip.py distributed with the zlib module -import _compression import builtins import io import os @@ -14,6 +13,7 @@ import time import weakref import zlib +from compression._common import _streams __all__ = ["BadGzipFile", "GzipFile", "open", "compress", "decompress"] @@ -144,7 +144,7 @@ def writable(self): return True -class GzipFile(_compression.BaseStream): +class GzipFile(_streams.BaseStream): """The GzipFile class simulates most of the methods of a file object with the exception of the truncate() method. @@ -193,6 +193,11 @@ def __init__(self, filename=None, mode=None, """ + # Ensure attributes exist at __del__ + self.mode = None + self.fileobj = None + self._buffer = None + if mode and ('t' in mode or 'U' in mode): raise ValueError("Invalid mode: {!r}".format(mode)) if mode and 'b' not in mode: @@ -332,11 +337,15 @@ def _write_raw(self, data): return length - def read(self, size=-1): - self._check_not_closed() + def _check_read(self, caller): if self.mode != READ: import errno - raise OSError(errno.EBADF, "read() on write-only GzipFile object") + msg = f"{caller}() on write-only GzipFile object" + raise OSError(errno.EBADF, msg) + + def read(self, size=-1): + self._check_not_closed() + self._check_read("read") return self._buffer.read(size) def read1(self, size=-1): @@ -344,19 +353,25 @@ def read1(self, size=-1): Reads up to a buffer's worth of data if size is negative.""" self._check_not_closed() - if self.mode != READ: - import errno - raise OSError(errno.EBADF, "read1() on write-only GzipFile object") + self._check_read("read1") if size < 0: size = io.DEFAULT_BUFFER_SIZE return self._buffer.read1(size) + def readinto(self, b): + self._check_not_closed() + self._check_read("readinto") + return self._buffer.readinto(b) + + def readinto1(self, b): + self._check_not_closed() + self._check_read("readinto1") + return self._buffer.readinto1(b) + def peek(self, n): self._check_not_closed() - if self.mode != READ: - import errno - raise OSError(errno.EBADF, "peek() on write-only GzipFile object") + self._check_read("peek") return self._buffer.peek(n) @property @@ -365,7 +380,9 @@ def closed(self): def close(self): fileobj = self.fileobj - if fileobj is None or self._buffer.closed: + if fileobj is None: + return + if self._buffer is None or self._buffer.closed: return try: if self.mode == WRITE: @@ -445,6 +462,13 @@ def readline(self, size=-1): self._check_not_closed() return self._buffer.readline(size) + def __del__(self): + if self.mode == WRITE and not self.closed: + import warnings + warnings.warn("unclosed GzipFile", + ResourceWarning, source=self, stacklevel=2) + + super().__del__() def _read_exact(fp, n): '''Read exactly *n* bytes from `fp` @@ -499,7 +523,7 @@ def _read_gzip_header(fp): return last_mtime -class _GzipReader(_compression.DecompressReader): +class _GzipReader(_streams.DecompressReader): def __init__(self, fp): super().__init__(_PaddedFile(fp), zlib._ZlibDecompressor, wbits=-zlib.MAX_WBITS) @@ -597,12 +621,12 @@ def _rewind(self): self._new_member = True -def compress(data, compresslevel=_COMPRESS_LEVEL_BEST, *, mtime=None): +def compress(data, compresslevel=_COMPRESS_LEVEL_BEST, *, mtime=0): """Compress data in one shot and return the compressed string. compresslevel sets the compression level in range of 0-9. - mtime can be used to set the modification time. The modification time is - set to the current time by default. + mtime can be used to set the modification time. + The modification time is set to 0 by default, for reproducibility. """ # Wbits=31 automatically includes a gzip header and trailer. gzip_data = zlib.compress(data, level=compresslevel, wbits=31) @@ -643,7 +667,9 @@ def main(): from argparse import ArgumentParser parser = ArgumentParser(description= "A simple command line interface for the gzip module: act like gzip, " - "but do not delete the input file.") + "but do not delete the input file.", + color=True, + ) group = parser.add_mutually_exclusive_group() group.add_argument('--fast', action='store_true', help='compress faster') group.add_argument('--best', action='store_true', help='compress better') diff --git a/Lib/heapq.py b/Lib/heapq.py index 2fd9d1ff4bf..17f62dd2d58 100644 --- a/Lib/heapq.py +++ b/Lib/heapq.py @@ -42,7 +42,7 @@ property of a heap is that a[0] is always its smallest element. The strange invariant above is meant to be an efficient memory -representation for a tournament. The numbers below are `k', not a[k]: +representation for a tournament. The numbers below are 'k', not a[k]: 0 @@ -55,7 +55,7 @@ 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 -In the tree above, each cell `k' is topping `2*k+1' and `2*k+2'. In +In the tree above, each cell 'k' is topping '2*k+1' and '2*k+2'. In a usual binary tournament we see in sports, each cell is the winner over the two cells it tops, and we can trace the winner down the tree to see all opponents s/he had. However, in many computer applications @@ -78,7 +78,7 @@ not "better" than the last 0'th element you extracted. This is especially useful in simulation contexts, where the tree holds all incoming events, and the "win" condition means the smallest scheduled -time. When an event schedule other events for execution, they are +time. When an event schedules other events for execution, they are scheduled into the future, so they can easily go into the heap. So, a heap is a good structure for implementing schedulers (this is what I used for my MIDI sequencer :-). @@ -91,14 +91,14 @@ Heaps are also very useful in big disk sorts. You most probably all know that a big sort implies producing "runs" (which are pre-sorted -sequences, which size is usually related to the amount of CPU memory), +sequences, whose size is usually related to the amount of CPU memory), followed by a merging passes for these runs, which merging is often very cleverly organised[1]. It is very important that the initial sort produces the longest runs possible. Tournaments are a good way -to that. If, using all the memory available to hold a tournament, you -replace and percolate items that happen to fit the current run, you'll -produce runs which are twice the size of the memory for random input, -and much better for input fuzzily ordered. +to achieve that. If, using all the memory available to hold a +tournament, you replace and percolate items that happen to fit the +current run, you'll produce runs which are twice the size of the +memory for random input, and much better for input fuzzily ordered. Moreover, if you output the 0'th item on disk and get an input which may not fit in the current tournament (because the value "wins" over @@ -110,7 +110,7 @@ effective! In a word, heaps are useful memory structures to know. I use them in -a few applications, and I think it is good to keep a `heap' module +a few applications, and I think it is good to keep a 'heap' module around. :-) -------------------- @@ -126,8 +126,9 @@ From all times, sorting has always been a Great Art! :-) """ -__all__ = ['heappush', 'heappop', 'heapify', 'heapreplace', 'merge', - 'nlargest', 'nsmallest', 'heappushpop'] +__all__ = ['heappush', 'heappop', 'heapify', 'heapreplace', 'heappushpop', + 'heappush_max', 'heappop_max', 'heapify_max', 'heapreplace_max', + 'heappushpop_max', 'nlargest', 'nsmallest', 'merge'] def heappush(heap, item): """Push item onto heap, maintaining the heap invariant.""" @@ -178,7 +179,7 @@ def heapify(x): for i in reversed(range(n//2)): _siftup(x, i) -def _heappop_max(heap): +def heappop_max(heap): """Maxheap version of a heappop.""" lastelt = heap.pop() # raises appropriate IndexError if heap is empty if heap: @@ -188,19 +189,32 @@ def _heappop_max(heap): return returnitem return lastelt -def _heapreplace_max(heap, item): +def heapreplace_max(heap, item): """Maxheap version of a heappop followed by a heappush.""" returnitem = heap[0] # raises appropriate IndexError if heap is empty heap[0] = item _siftup_max(heap, 0) return returnitem -def _heapify_max(x): +def heappush_max(heap, item): + """Maxheap version of a heappush.""" + heap.append(item) + _siftdown_max(heap, 0, len(heap)-1) + +def heappushpop_max(heap, item): + """Maxheap fast version of a heappush followed by a heappop.""" + if heap and item < heap[0]: + item, heap[0] = heap[0], item + _siftup_max(heap, 0) + return item + +def heapify_max(x): """Transform list into a maxheap, in-place, in O(len(x)) time.""" n = len(x) for i in reversed(range(n//2)): _siftup_max(x, i) + # 'heap' is a heap at all indices >= startpos, except possibly for pos. pos # is the index of a leaf with a possibly out-of-order value. Restore the # heap invariant. @@ -335,9 +349,9 @@ def merge(*iterables, key=None, reverse=False): h_append = h.append if reverse: - _heapify = _heapify_max - _heappop = _heappop_max - _heapreplace = _heapreplace_max + _heapify = heapify_max + _heappop = heappop_max + _heapreplace = heapreplace_max direction = -1 else: _heapify = heapify @@ -490,10 +504,10 @@ def nsmallest(n, iterable, key=None): result = [(elem, i) for i, elem in zip(range(n), it)] if not result: return result - _heapify_max(result) + heapify_max(result) top = result[0][0] order = n - _heapreplace = _heapreplace_max + _heapreplace = heapreplace_max for elem in it: if elem < top: _heapreplace(result, (elem, order)) @@ -507,10 +521,10 @@ def nsmallest(n, iterable, key=None): result = [(key(elem), i, elem) for i, elem in zip(range(n), it)] if not result: return result - _heapify_max(result) + heapify_max(result) top = result[0][0] order = n - _heapreplace = _heapreplace_max + _heapreplace = heapreplace_max for elem in it: k = key(elem) if k < top: @@ -583,19 +597,13 @@ def nlargest(n, iterable, key=None): from _heapq import * except ImportError: pass -try: - from _heapq import _heapreplace_max -except ImportError: - pass -try: - from _heapq import _heapify_max -except ImportError: - pass -try: - from _heapq import _heappop_max -except ImportError: - pass +# For backwards compatibility +_heappop_max = heappop_max +_heapreplace_max = heapreplace_max +_heappush_max = heappush_max +_heappushpop_max = heappushpop_max +_heapify_max = heapify_max if __name__ == "__main__": diff --git a/Lib/hmac.py b/Lib/hmac.py index 8b4eb2fe741..16022c9ceb5 100644 --- a/Lib/hmac.py +++ b/Lib/hmac.py @@ -3,7 +3,6 @@ Implements the HMAC algorithm as described by RFC 2104. """ -import warnings as _warnings try: import _hashlib as _hashopenssl except ImportError: @@ -14,7 +13,10 @@ compare_digest = _hashopenssl.compare_digest _functype = type(_hashopenssl.openssl_sha256) # builtin type -import hashlib as _hashlib +try: + import _hmac +except ImportError: + _hmac = None trans_5C = bytes((x ^ 0x5C) for x in range(256)) trans_36 = bytes((x ^ 0x36) for x in range(256)) @@ -24,11 +26,27 @@ digest_size = None +def _get_digest_constructor(digest_like): + if callable(digest_like): + return digest_like + if isinstance(digest_like, str): + def digest_wrapper(d=b''): + import hashlib + return hashlib.new(digest_like, d) + else: + def digest_wrapper(d=b''): + return digest_like.new(d) + return digest_wrapper + + class HMAC: """RFC 2104 HMAC class. Also complies with RFC 4231. This supports the API for Cryptographic Hash Functions (PEP 247). """ + + # Note: self.blocksize is the default blocksize; self.block_size + # is effective block size as well as the public API attribute. blocksize = 64 # 512-bit HMAC; can be changed in subclasses. __slots__ = ( @@ -50,31 +68,47 @@ def __init__(self, key, msg=None, digestmod=''): """ if not isinstance(key, (bytes, bytearray)): - raise TypeError("key: expected bytes or bytearray, but got %r" % type(key).__name__) + raise TypeError(f"key: expected bytes or bytearray, " + f"but got {type(key).__name__!r}") if not digestmod: raise TypeError("Missing required argument 'digestmod'.") + self.__init(key, msg, digestmod) + + def __init(self, key, msg, digestmod): if _hashopenssl and isinstance(digestmod, (str, _functype)): try: - self._init_hmac(key, msg, digestmod) - except _hashopenssl.UnsupportedDigestmodError: - self._init_old(key, msg, digestmod) - else: - self._init_old(key, msg, digestmod) + self._init_openssl_hmac(key, msg, digestmod) + return + except _hashopenssl.UnsupportedDigestmodError: # pragma: no cover + pass + if _hmac and isinstance(digestmod, str): + try: + self._init_builtin_hmac(key, msg, digestmod) + return + except _hmac.UnknownHashError: # pragma: no cover + pass + self._init_old(key, msg, digestmod) - def _init_hmac(self, key, msg, digestmod): + def _init_openssl_hmac(self, key, msg, digestmod): self._hmac = _hashopenssl.hmac_new(key, msg, digestmod=digestmod) + self._inner = self._outer = None # because the slots are defined + self.digest_size = self._hmac.digest_size + self.block_size = self._hmac.block_size + + _init_hmac = _init_openssl_hmac # for backward compatibility (if any) + + def _init_builtin_hmac(self, key, msg, digestmod): + self._hmac = _hmac.new(key, msg, digestmod=digestmod) + self._inner = self._outer = None # because the slots are defined self.digest_size = self._hmac.digest_size self.block_size = self._hmac.block_size def _init_old(self, key, msg, digestmod): - if callable(digestmod): - digest_cons = digestmod - elif isinstance(digestmod, str): - digest_cons = lambda d=b'': _hashlib.new(digestmod, d) - else: - digest_cons = lambda d=b'': digestmod.new(d) + import warnings + + digest_cons = _get_digest_constructor(digestmod) self._hmac = None self._outer = digest_cons() @@ -84,21 +118,19 @@ def _init_old(self, key, msg, digestmod): if hasattr(self._inner, 'block_size'): blocksize = self._inner.block_size if blocksize < 16: - _warnings.warn('block_size of %d seems too small; using our ' - 'default of %d.' % (blocksize, self.blocksize), - RuntimeWarning, 2) - blocksize = self.blocksize + warnings.warn(f"block_size of {blocksize} seems too small; " + f"using our default of {self.blocksize}.", + RuntimeWarning, 2) + blocksize = self.blocksize # pragma: no cover else: - _warnings.warn('No block_size attribute on given digest object; ' - 'Assuming %d.' % (self.blocksize), - RuntimeWarning, 2) - blocksize = self.blocksize + warnings.warn("No block_size attribute on given digest object; " + f"Assuming {self.blocksize}.", + RuntimeWarning, 2) + blocksize = self.blocksize # pragma: no cover if len(key) > blocksize: key = digest_cons(key).digest() - # self.blocksize is the default blocksize. self.block_size is - # effective block size as well as the public API attribute. self.block_size = blocksize key = key.ljust(blocksize, b'\0') @@ -164,6 +196,7 @@ def hexdigest(self): h = self._current() return h.hexdigest() + def new(key, msg=None, digestmod=''): """Create a new hashing object and return it. @@ -193,25 +226,41 @@ def digest(key, msg, digest): A hashlib constructor returning a new hash object. *OR* A module supporting PEP 247. """ - if _hashopenssl is not None and isinstance(digest, (str, _functype)): + if _hashopenssl and isinstance(digest, (str, _functype)): try: return _hashopenssl.hmac_digest(key, msg, digest) + except OverflowError: + # OpenSSL's HMAC limits the size of the key to INT_MAX. + # Instead of falling back to HACL* implementation which + # may still not be supported due to a too large key, we + # directly switch to the pure Python fallback instead + # even if we could have used streaming HMAC for small keys + # but large messages. + return _compute_digest_fallback(key, msg, digest) except _hashopenssl.UnsupportedDigestmodError: pass - if callable(digest): - digest_cons = digest - elif isinstance(digest, str): - digest_cons = lambda d=b'': _hashlib.new(digest, d) - else: - digest_cons = lambda d=b'': digest.new(d) + if _hmac and isinstance(digest, str): + try: + return _hmac.compute_digest(key, msg, digest) + except (OverflowError, _hmac.UnknownHashError): + # HACL* HMAC limits the size of the key to UINT32_MAX + # so we fallback to the pure Python implementation even + # if streaming HMAC may have been used for small keys + # and large messages. + pass + + return _compute_digest_fallback(key, msg, digest) + +def _compute_digest_fallback(key, msg, digest): + digest_cons = _get_digest_constructor(digest) inner = digest_cons() outer = digest_cons() blocksize = getattr(inner, 'block_size', 64) if len(key) > blocksize: key = digest_cons(key).digest() - key = key + b'\x00' * (blocksize - len(key)) + key = key.ljust(blocksize, b'\0') inner.update(key.translate(trans_36)) outer.update(key.translate(trans_5C)) inner.update(msg) diff --git a/Lib/html/parser.py b/Lib/html/parser.py index 5d7050dad23..80fb8c3f929 100644 --- a/Lib/html/parser.py +++ b/Lib/html/parser.py @@ -24,6 +24,7 @@ entityref = re.compile('&([a-zA-Z][-.a-zA-Z0-9]*)[^a-zA-Z0-9]') charref = re.compile('&#(?:[0-9]+|[xX][0-9a-fA-F]+)[^0-9a-fA-F]') +incomplete_charref = re.compile('&#(?:[0-9]|[xX][0-9a-fA-F])') attr_charref = re.compile(r'&(#[0-9]+|#[xX][0-9a-fA-F]+|[a-zA-Z][a-zA-Z0-9]*)[;=]?') starttagopen = re.compile('<[a-zA-Z]') @@ -127,17 +128,25 @@ class HTMLParser(_markupbase.ParserBase): argument. """ - CDATA_CONTENT_ELEMENTS = ("script", "style") + # See the HTML5 specs section "13.4 Parsing HTML fragments". + # https://html.spec.whatwg.org/multipage/parsing.html#parsing-html-fragments + # CDATA_CONTENT_ELEMENTS are parsed in RAWTEXT mode + CDATA_CONTENT_ELEMENTS = ("script", "style", "xmp", "iframe", "noembed", "noframes") RCDATA_CONTENT_ELEMENTS = ("textarea", "title") - def __init__(self, *, convert_charrefs=True): + def __init__(self, *, convert_charrefs=True, scripting=False): """Initialize and reset this instance. - If convert_charrefs is True (the default), all character references + If convert_charrefs is true (the default), all character references are automatically converted to the corresponding Unicode characters. + + If *scripting* is false (the default), the content of the + ``noscript`` element is parsed normally; if it's true, + it's returned as is without being parsed. """ super().__init__() self.convert_charrefs = convert_charrefs + self.scripting = scripting self.reset() def reset(self): @@ -172,7 +181,9 @@ def get_starttag_text(self): def set_cdata_mode(self, elem, *, escapable=False): self.cdata_elem = elem.lower() self._escapable = escapable - if escapable and not self.convert_charrefs: + if self.cdata_elem == 'plaintext': + self.interesting = re.compile(r'\z') + elif escapable and not self.convert_charrefs: self.interesting = re.compile(r'&|])' % self.cdata_elem, re.IGNORECASE|re.ASCII) else: @@ -294,10 +305,20 @@ def goahead(self, end): k = k - 1 i = self.updatepos(i, k) continue + match = incomplete_charref.match(rawdata, i) + if match: + if end: + self.handle_charref(rawdata[i+2:]) + i = self.updatepos(i, n) + break + # incomplete + break + elif i + 3 < n: # larger than "&#x" + # not the end of the buffer, and can't be confused + # with some other construct + self.handle_data("&#") + i = self.updatepos(i, i + 2) else: - if ";" in rawdata[i:]: # bail by consuming &# - self.handle_data(rawdata[i:i+2]) - i = self.updatepos(i, i+2) break elif startswith('&', i): match = entityref.match(rawdata, i) @@ -311,15 +332,13 @@ def goahead(self, end): continue match = incomplete.match(rawdata, i) if match: - # match.group() will contain at least 2 chars - if end and match.group() == rawdata[i:]: - k = match.end() - if k <= i: - k = n - i = self.updatepos(i, i + 1) + if end: + self.handle_entityref(rawdata[i+1:]) + i = self.updatepos(i, n) + break # incomplete break - elif (i + 1) < n: + elif i + 1 < n: # not the end of the buffer, and can't be confused # with some other construct self.handle_data("&") @@ -444,8 +463,10 @@ def parse_starttag(self, i): self.handle_startendtag(tag, attrs) else: self.handle_starttag(tag, attrs) - if tag in self.CDATA_CONTENT_ELEMENTS: - self.set_cdata_mode(tag) + if (tag in self.CDATA_CONTENT_ELEMENTS or + (self.scripting and tag == "noscript") or + tag == "plaintext"): + self.set_cdata_mode(tag, escapable=False) elif tag in self.RCDATA_CONTENT_ELEMENTS: self.set_cdata_mode(tag, escapable=True) return endpos diff --git a/Lib/http/__init__.py b/Lib/http/__init__.py index bf8d7d68868..17a47b180e5 100644 --- a/Lib/http/__init__.py +++ b/Lib/http/__init__.py @@ -1,14 +1,15 @@ -from enum import IntEnum +from enum import StrEnum, IntEnum, _simple_enum -__all__ = ['HTTPStatus'] +__all__ = ['HTTPStatus', 'HTTPMethod'] -class HTTPStatus(IntEnum): +@_simple_enum(IntEnum) +class HTTPStatus: """HTTP status codes and reason phrases Status codes from the following RFCs are all observed: - * RFC 7231: Hypertext Transfer Protocol (HTTP/1.1), obsoletes 2616 + * RFC 9110: HTTP Semantics, obsoletes 7231, which obsoleted 2616 * RFC 6585: Additional HTTP Status Codes * RFC 3229: Delta encoding in HTTP * RFC 4918: HTTP Extensions for WebDAV, obsoletes 2518 @@ -25,11 +26,30 @@ class HTTPStatus(IntEnum): def __new__(cls, value, phrase, description=''): obj = int.__new__(cls, value) obj._value_ = value - obj.phrase = phrase obj.description = description return obj + @property + def is_informational(self): + return 100 <= self <= 199 + + @property + def is_success(self): + return 200 <= self <= 299 + + @property + def is_redirection(self): + return 300 <= self <= 399 + + @property + def is_client_error(self): + return 400 <= self <= 499 + + @property + def is_server_error(self): + return 500 <= self <= 599 + # informational CONTINUE = 100, 'Continue', 'Request received, please continue' SWITCHING_PROTOCOLS = (101, 'Switching Protocols', @@ -94,22 +114,25 @@ def __new__(cls, value, phrase, description=''): 'Client must specify Content-Length') PRECONDITION_FAILED = (412, 'Precondition Failed', 'Precondition in headers is false') - REQUEST_ENTITY_TOO_LARGE = (413, 'Request Entity Too Large', - 'Entity is too large') - REQUEST_URI_TOO_LONG = (414, 'Request-URI Too Long', + CONTENT_TOO_LARGE = (413, 'Content Too Large', + 'Content is too large') + REQUEST_ENTITY_TOO_LARGE = CONTENT_TOO_LARGE + URI_TOO_LONG = (414, 'URI Too Long', 'URI is too long') + REQUEST_URI_TOO_LONG = URI_TOO_LONG UNSUPPORTED_MEDIA_TYPE = (415, 'Unsupported Media Type', 'Entity body in unsupported format') - REQUESTED_RANGE_NOT_SATISFIABLE = (416, - 'Requested Range Not Satisfiable', + RANGE_NOT_SATISFIABLE = (416, 'Range Not Satisfiable', 'Cannot satisfy request range') + REQUESTED_RANGE_NOT_SATISFIABLE = RANGE_NOT_SATISFIABLE EXPECTATION_FAILED = (417, 'Expectation Failed', 'Expect condition could not be satisfied') IM_A_TEAPOT = (418, 'I\'m a Teapot', 'Server refuses to brew coffee because it is a teapot.') MISDIRECTED_REQUEST = (421, 'Misdirected Request', 'Server is not able to produce a response') - UNPROCESSABLE_ENTITY = 422, 'Unprocessable Entity' + UNPROCESSABLE_CONTENT = 422, 'Unprocessable Content' + UNPROCESSABLE_ENTITY = UNPROCESSABLE_CONTENT LOCKED = 423, 'Locked' FAILED_DEPENDENCY = 424, 'Failed Dependency' TOO_EARLY = 425, 'Too Early' @@ -148,3 +171,32 @@ def __new__(cls, value, phrase, description=''): NETWORK_AUTHENTICATION_REQUIRED = (511, 'Network Authentication Required', 'The client needs to authenticate to gain network access') + + +@_simple_enum(StrEnum) +class HTTPMethod: + """HTTP methods and descriptions + + Methods from the following RFCs are all observed: + + * RFC 9110: HTTP Semantics, obsoletes 7231, which obsoleted 2616 + * RFC 5789: PATCH Method for HTTP + """ + def __new__(cls, value, description): + obj = str.__new__(cls, value) + obj._value_ = value + obj.description = description + return obj + + def __repr__(self): + return "<%s.%s>" % (self.__class__.__name__, self._name_) + + CONNECT = 'CONNECT', 'Establish a connection to the server.' + DELETE = 'DELETE', 'Remove the target.' + GET = 'GET', 'Retrieve the target.' + HEAD = 'HEAD', 'Same as GET, but only retrieve the status line and header section.' + OPTIONS = 'OPTIONS', 'Describe the communication options for the target.' + PATCH = 'PATCH', 'Apply partial modifications to a target.' + POST = 'POST', 'Perform target-specific processing with the request payload.' + PUT = 'PUT', 'Replace the target with the request payload.' + TRACE = 'TRACE', 'Perform a message loop-back test along the path to the target.' diff --git a/Lib/http/client.py b/Lib/http/client.py index a6ab135b2c3..dd5f4136e9e 100644 --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -111,6 +111,11 @@ _MAXLINE = 65536 _MAXHEADERS = 100 +# Data larger than this will be read in chunks, to prevent extreme +# overallocation. +_MIN_READ_BUF_SIZE = 1 << 20 + + # Header name/value ABNF (http://tools.ietf.org/html/rfc7230#section-3.2) # # VCHAR = %x21-7E @@ -172,6 +177,13 @@ def _encode(data, name='data'): "if you want to send it encoded in UTF-8." % (name.title(), data[err.start:err.end], name)) from None +def _strip_ipv6_iface(enc_name: bytes) -> bytes: + """Remove interface scope from IPv6 address.""" + enc_name, percent, _ = enc_name.partition(b"%") + if percent: + assert enc_name.startswith(b'['), enc_name + enc_name += b']' + return enc_name class HTTPMessage(email.message.Message): # XXX The only usage of this method is in @@ -221,8 +233,9 @@ def _read_headers(fp): break return headers -def parse_headers(fp, _class=HTTPMessage): - """Parses only RFC2822 headers from a file pointer. +def _parse_header_lines(header_lines, _class=HTTPMessage): + """ + Parses only RFC 5322 headers from header lines. email Parser wants to see strings rather than bytes. But a TextIOWrapper around self.rfile would buffer too many bytes @@ -231,10 +244,15 @@ def parse_headers(fp, _class=HTTPMessage): to parse. """ - headers = _read_headers(fp) - hstring = b''.join(headers).decode('iso-8859-1') + hstring = b''.join(header_lines).decode('iso-8859-1') return email.parser.Parser(_class=_class).parsestr(hstring) +def parse_headers(fp, _class=HTTPMessage): + """Parses only RFC 5322 headers from a file pointer.""" + + headers = _read_headers(fp) + return _parse_header_lines(headers, _class) + class HTTPResponse(io.BufferedIOBase): @@ -448,6 +466,7 @@ def isclosed(self): return self.fp is None def read(self, amt=None): + """Read and return the response body, or up to the next amt bytes.""" if self.fp is None: return b"" @@ -458,7 +477,7 @@ def read(self, amt=None): if self.chunked: return self._read_chunked(amt) - if amt is not None: + if amt is not None and amt >= 0: if self.length is not None and amt > self.length: # clip the read to the "end of response" amt = self.length @@ -576,13 +595,11 @@ def _get_chunk_left(self): def _read_chunked(self, amt=None): assert self.chunked != _UNKNOWN + if amt is not None and amt < 0: + amt = None value = [] try: - while True: - chunk_left = self._get_chunk_left() - if chunk_left is None: - break - + while (chunk_left := self._get_chunk_left()) is not None: if amt is not None and amt <= chunk_left: value.append(self._safe_read(amt)) self.chunk_left = chunk_left - amt @@ -593,8 +610,8 @@ def _read_chunked(self, amt=None): amt -= chunk_left self.chunk_left = 0 return b''.join(value) - except IncompleteRead: - raise IncompleteRead(b''.join(value)) + except IncompleteRead as exc: + raise IncompleteRead(b''.join(value)) from exc def _readinto_chunked(self, b): assert self.chunked != _UNKNOWN @@ -627,10 +644,25 @@ def _safe_read(self, amt): reading. If the bytes are truly not available (due to EOF), then the IncompleteRead exception can be used to detect the problem. """ - data = self.fp.read(amt) - if len(data) < amt: - raise IncompleteRead(data, amt-len(data)) - return data + cursize = min(amt, _MIN_READ_BUF_SIZE) + data = self.fp.read(cursize) + if len(data) >= amt: + return data + if len(data) < cursize: + raise IncompleteRead(data, amt - len(data)) + + data = io.BytesIO(data) + data.seek(0, 2) + while True: + # This is a geometric increase in read size (never more than + # doubling out the current length of data per loop iteration). + delta = min(cursize, amt - cursize) + data.write(self.fp.read(delta)) + if data.tell() >= amt: + return data.getvalue() + cursize += delta + if data.tell() < cursize: + raise IncompleteRead(data.getvalue(), amt - data.tell()) def _safe_readinto(self, b): """Same as _safe_read, but for reading into a buffer.""" @@ -655,6 +687,8 @@ def read1(self, n=-1): self._close_conn() elif self.length is not None: self.length -= len(result) + if not self.length: + self._close_conn() return result def peek(self, n=-1): @@ -679,6 +713,8 @@ def readline(self, limit=-1): self._close_conn() elif self.length is not None: self.length -= len(result) + if not self.length: + self._close_conn() return result def _read1_chunked(self, n): @@ -786,6 +822,20 @@ def getcode(self): ''' return self.status + +def _create_https_context(http_version): + # Function also used by urllib.request to be able to set the check_hostname + # attribute on a context object. + context = ssl._create_default_https_context() + # send ALPN extension to indicate HTTP/1.1 protocol + if http_version == 11: + context.set_alpn_protocols(['http/1.1']) + # enable PHA for TLS 1.3 connections if available + if context.post_handshake_auth is not None: + context.post_handshake_auth = True + return context + + class HTTPConnection: _http_vsn = 11 @@ -847,6 +897,7 @@ def __init__(self, host, port=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT, self._tunnel_host = None self._tunnel_port = None self._tunnel_headers = {} + self._raw_proxy_headers = None (self.host, self.port) = self._get_hostport(host, port) @@ -859,9 +910,9 @@ def __init__(self, host, port=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT, def set_tunnel(self, host, port=None, headers=None): """Set up host and port for HTTP CONNECT tunnelling. - In a connection that uses HTTP CONNECT tunneling, the host passed to the - constructor is used as a proxy server that relays all communication to - the endpoint passed to `set_tunnel`. This done by sending an HTTP + In a connection that uses HTTP CONNECT tunnelling, the host passed to + the constructor is used as a proxy server that relays all communication + to the endpoint passed to `set_tunnel`. This done by sending an HTTP CONNECT request to the proxy server when the connection is established. This method must be called before the HTTP connection has been @@ -869,6 +920,13 @@ def set_tunnel(self, host, port=None, headers=None): The headers argument should be a mapping of extra HTTP headers to send with the CONNECT request. + + As HTTP/1.1 is used for HTTP CONNECT tunnelling request, as per the RFC + (https://tools.ietf.org/html/rfc7231#section-4.3.6), a HTTP Host: + header must be provided, matching the authority-form of the request + target provided as the destination for the CONNECT request. If a + HTTP Host: header is not provided via the headers argument, one + is generated and transmitted automatically. """ if self.sock: @@ -876,10 +934,15 @@ def set_tunnel(self, host, port=None, headers=None): self._tunnel_host, self._tunnel_port = self._get_hostport(host, port) if headers: - self._tunnel_headers = headers + self._tunnel_headers = headers.copy() else: self._tunnel_headers.clear() + if not any(header.lower() == "host" for header in self._tunnel_headers): + encoded_host = self._tunnel_host.encode("idna").decode("ascii") + self._tunnel_headers["Host"] = "%s:%d" % ( + encoded_host, self._tunnel_port) + def _get_hostport(self, host, port): if port is None: i = host.rfind(':') @@ -895,17 +958,24 @@ def _get_hostport(self, host, port): host = host[:i] else: port = self.default_port - if host and host[0] == '[' and host[-1] == ']': - host = host[1:-1] + if host and host[0] == '[' and host[-1] == ']': + host = host[1:-1] return (host, port) def set_debuglevel(self, level): self.debuglevel = level + def _wrap_ipv6(self, ip): + if b':' in ip and ip[0] != b'['[0]: + return b"[" + ip + b"]" + return ip + def _tunnel(self): - connect = b"CONNECT %s:%d HTTP/1.0\r\n" % ( - self._tunnel_host.encode("ascii"), self._tunnel_port) + connect = b"CONNECT %s:%d %s\r\n" % ( + self._wrap_ipv6(self._tunnel_host.encode("idna")), + self._tunnel_port, + self._http_vsn_str.encode("ascii")) headers = [connect] for header, value in self._tunnel_headers.items(): headers.append(f"{header}: {value}\r\n".encode("latin-1")) @@ -917,23 +987,35 @@ def _tunnel(self): del headers response = self.response_class(self.sock, method=self._method) - (version, code, message) = response._read_status() + try: + (version, code, message) = response._read_status() - if code != http.HTTPStatus.OK: - self.close() - raise OSError(f"Tunnel connection failed: {code} {message.strip()}") - while True: - line = response.fp.readline(_MAXLINE + 1) - if len(line) > _MAXLINE: - raise LineTooLong("header line") - if not line: - # for sites which EOF without sending a trailer - break - if line in (b'\r\n', b'\n', b''): - break + self._raw_proxy_headers = _read_headers(response.fp) if self.debuglevel > 0: - print('header:', line.decode()) + for header in self._raw_proxy_headers: + print('header:', header.decode()) + + if code != http.HTTPStatus.OK: + self.close() + raise OSError(f"Tunnel connection failed: {code} {message.strip()}") + + finally: + response.close() + + def get_proxy_response_headers(self): + """ + Returns a dictionary with the headers of the response + received from the proxy server to the CONNECT request + sent to set the tunnel. + + If the CONNECT request was not sent, the method returns None. + """ + return ( + _parse_header_lines(self._raw_proxy_headers) + if self._raw_proxy_headers is not None + else None + ) def connect(self): """Connect to the host and port specified in __init__.""" @@ -942,7 +1024,7 @@ def connect(self): (self.host,self.port), self.timeout, self.source_address) # Might fail in OSs that don't implement TCP_NODELAY try: - self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) + self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) except OSError as e: if e.errno != errno.ENOPROTOOPT: raise @@ -980,14 +1062,11 @@ def send(self, data): print("send:", repr(data)) if hasattr(data, "read") : if self.debuglevel > 0: - print("sendIng a read()able") + print("sending a readable") encode = self._is_textIO(data) if encode and self.debuglevel > 0: print("encoding file using iso-8859-1") - while 1: - datablock = data.read(self.blocksize) - if not datablock: - break + while datablock := data.read(self.blocksize): if encode: datablock = datablock.encode("iso-8859-1") sys.audit("http.client.send", self, datablock) @@ -1013,14 +1092,11 @@ def _output(self, s): def _read_readable(self, readable): if self.debuglevel > 0: - print("sendIng a read()able") + print("reading a readable") encode = self._is_textIO(readable) if encode and self.debuglevel > 0: print("encoding file using iso-8859-1") - while True: - datablock = readable.read(self.blocksize) - if not datablock: - break + while datablock := readable.read(self.blocksize): if encode: datablock = datablock.encode("iso-8859-1") yield datablock @@ -1157,7 +1233,7 @@ def putrequest(self, method, url, skip_host=False, netloc_enc = netloc.encode("ascii") except UnicodeEncodeError: netloc_enc = netloc.encode("idna") - self.putheader('Host', netloc_enc) + self.putheader('Host', _strip_ipv6_iface(netloc_enc)) else: if self._tunnel_host: host = self._tunnel_host @@ -1173,9 +1249,9 @@ def putrequest(self, method, url, skip_host=False, # As per RFC 273, IPv6 address should be wrapped with [] # when used as Host header - - if host.find(':') >= 0: - host_enc = b'[' + host_enc + b']' + host_enc = self._wrap_ipv6(host_enc) + if ":" in host: + host_enc = _strip_ipv6_iface(host_enc) if port == self.default_port: self.putheader('Host', host_enc) @@ -1400,46 +1476,15 @@ class HTTPSConnection(HTTPConnection): default_port = HTTPS_PORT - # XXX Should key_file and cert_file be deprecated in favour of context? - - def __init__(self, host, port=None, key_file=None, cert_file=None, - timeout=socket._GLOBAL_DEFAULT_TIMEOUT, - source_address=None, *, context=None, - check_hostname=None, blocksize=8192): + def __init__(self, host, port=None, + *, timeout=socket._GLOBAL_DEFAULT_TIMEOUT, + source_address=None, context=None, blocksize=8192): super(HTTPSConnection, self).__init__(host, port, timeout, source_address, blocksize=blocksize) - if (key_file is not None or cert_file is not None or - check_hostname is not None): - import warnings - warnings.warn("key_file, cert_file and check_hostname are " - "deprecated, use a custom context instead.", - DeprecationWarning, 2) - self.key_file = key_file - self.cert_file = cert_file if context is None: - context = ssl._create_default_https_context() - # send ALPN extension to indicate HTTP/1.1 protocol - if self._http_vsn == 11: - context.set_alpn_protocols(['http/1.1']) - # enable PHA for TLS 1.3 connections if available - if context.post_handshake_auth is not None: - context.post_handshake_auth = True - will_verify = context.verify_mode != ssl.CERT_NONE - if check_hostname is None: - check_hostname = context.check_hostname - if check_hostname and not will_verify: - raise ValueError("check_hostname needs a SSL context with " - "either CERT_OPTIONAL or CERT_REQUIRED") - if key_file or cert_file: - context.load_cert_chain(cert_file, key_file) - # cert and key file means the user wants to authenticate. - # enable TLS 1.3 PHA implicitly even for custom contexts. - if context.post_handshake_auth is not None: - context.post_handshake_auth = True + context = _create_https_context(self._http_vsn) self._context = context - if check_hostname is not None: - self._context.check_hostname = check_hostname def connect(self): "Connect to a host on a given (SSL) port." diff --git a/Lib/http/cookiejar.py b/Lib/http/cookiejar.py index 685f6a0b976..9a2f0fb851c 100644 --- a/Lib/http/cookiejar.py +++ b/Lib/http/cookiejar.py @@ -34,10 +34,7 @@ import re import time import urllib.parse, urllib.request -try: - import threading as _threading -except ImportError: - import dummy_threading as _threading +import threading as _threading import http.client # only for the default HTTP port from calendar import timegm @@ -92,8 +89,7 @@ def _timegm(tt): DAYS = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"] MONTHS = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"] -MONTHS_LOWER = [] -for month in MONTHS: MONTHS_LOWER.append(month.lower()) +MONTHS_LOWER = [month.lower() for month in MONTHS] def time2isoz(t=None): """Return a string representing time in seconds since epoch, t. @@ -108,9 +104,9 @@ def time2isoz(t=None): """ if t is None: - dt = datetime.datetime.utcnow() + dt = datetime.datetime.now(tz=datetime.UTC) else: - dt = datetime.datetime.utcfromtimestamp(t) + dt = datetime.datetime.fromtimestamp(t, tz=datetime.UTC) return "%04d-%02d-%02d %02d:%02d:%02dZ" % ( dt.year, dt.month, dt.day, dt.hour, dt.minute, dt.second) @@ -126,9 +122,9 @@ def time2netscape(t=None): """ if t is None: - dt = datetime.datetime.utcnow() + dt = datetime.datetime.now(tz=datetime.UTC) else: - dt = datetime.datetime.utcfromtimestamp(t) + dt = datetime.datetime.fromtimestamp(t, tz=datetime.UTC) return "%s, %02d-%s-%04d %02d:%02d:%02d GMT" % ( DAYS[dt.weekday()], dt.day, MONTHS[dt.month-1], dt.year, dt.hour, dt.minute, dt.second) @@ -434,6 +430,7 @@ def split_header_words(header_values): if pairs: result.append(pairs) return result +HEADER_JOIN_TOKEN_RE = re.compile(r"[!#$%&'*+\-.^_`|~0-9A-Za-z]+") HEADER_JOIN_ESCAPE_RE = re.compile(r"([\"\\])") def join_header_words(lists): """Do the inverse (almost) of the conversion done by split_header_words. @@ -441,10 +438,10 @@ def join_header_words(lists): Takes a list of lists of (key, value) pairs and produces a single header value. Attribute values are quoted if needed. - >>> join_header_words([[("text/plain", None), ("charset", "iso-8859-1")]]) - 'text/plain; charset="iso-8859-1"' - >>> join_header_words([[("text/plain", None)], [("charset", "iso-8859-1")]]) - 'text/plain, charset="iso-8859-1"' + >>> join_header_words([[("text/plain", None), ("charset", "iso-8859/1")]]) + 'text/plain; charset="iso-8859/1"' + >>> join_header_words([[("text/plain", None)], [("charset", "iso-8859/1")]]) + 'text/plain, charset="iso-8859/1"' """ headers = [] @@ -452,7 +449,7 @@ def join_header_words(lists): attr = [] for k, v in pairs: if v is not None: - if not re.search(r"^\w+$", v): + if not HEADER_JOIN_TOKEN_RE.fullmatch(v): v = HEADER_JOIN_ESCAPE_RE.sub(r"\\\1", v) # escape " and \ v = '"%s"' % v k = "%s=%s" % (k, v) @@ -644,7 +641,7 @@ def eff_request_host(request): """ erhn = req_host = request_host(request) - if req_host.find(".") == -1 and not IPV4_RE.search(req_host): + if "." not in req_host: erhn = req_host + ".local" return req_host, erhn @@ -1047,12 +1044,13 @@ def set_ok_domain(self, cookie, request): else: undotted_domain = domain embedded_dots = (undotted_domain.find(".") >= 0) - if not embedded_dots and domain != ".local": + if not embedded_dots and not erhn.endswith(".local"): _debug(" non-local domain %s contains no embedded dot", domain) return False if cookie.version == 0: - if (not erhn.endswith(domain) and + if (not (erhn.endswith(domain) or + erhn.endswith(f"{undotted_domain}.local")) and (not erhn.startswith(".") and not ("."+erhn).endswith(domain))): _debug(" effective request-host %s (even with added " @@ -1227,14 +1225,9 @@ def path_return_ok(self, path, request): _debug(" %s does not path-match %s", req_path, path) return False -def vals_sorted_by_key(adict): - keys = sorted(adict.keys()) - return map(adict.get, keys) - def deepvalues(mapping): - """Iterates over nested mapping, depth-first, in sorted order by key.""" - values = vals_sorted_by_key(mapping) - for obj in values: + """Iterates over nested mapping, depth-first""" + for obj in list(mapping.values()): mapping = False try: obj.items @@ -1898,7 +1891,10 @@ def save(self, filename=None, ignore_discard=False, ignore_expires=False): if self.filename is not None: filename = self.filename else: raise ValueError(MISSING_FILENAME_TEXT) - with open(filename, "w") as f: + with os.fdopen( + os.open(filename, os.O_CREAT | os.O_WRONLY | os.O_TRUNC, 0o600), + 'w', + ) as f: # There really isn't an LWP Cookies 2.0 format, but this indicates # that there is extra information in here (domain_dot and # port_spec) while still being compatible with libwww-perl, I hope. @@ -1923,9 +1919,7 @@ def _really_load(self, f, filename, ignore_discard, ignore_expires): "comment", "commenturl") try: - while 1: - line = f.readline() - if line == "": break + while (line := f.readline()) != "": if not line.startswith(header): continue line = line[len(header):].strip() @@ -1993,7 +1987,7 @@ class MozillaCookieJar(FileCookieJar): This class differs from CookieJar only in the format it uses to save and load cookies to and from a file. This class uses the Mozilla/Netscape - `cookies.txt' format. lynx uses this file format, too. + `cookies.txt' format. curl and lynx use this file format, too. Don't expect cookies saved while the browser is running to be noticed by the browser (in fact, Mozilla on unix will overwrite your saved cookies if @@ -2025,12 +2019,9 @@ def _really_load(self, f, filename, ignore_discard, ignore_expires): filename) try: - while 1: - line = f.readline() + while (line := f.readline()) != "": rest = {} - if line == "": break - # httponly is a cookie flag as defined in rfc6265 # when encoded in a netscape cookie file, # the line is prepended with "#HttpOnly_" @@ -2094,7 +2085,10 @@ def save(self, filename=None, ignore_discard=False, ignore_expires=False): if self.filename is not None: filename = self.filename else: raise ValueError(MISSING_FILENAME_TEXT) - with open(filename, "w") as f: + with os.fdopen( + os.open(filename, os.O_CREAT | os.O_WRONLY | os.O_TRUNC, 0o600), + 'w', + ) as f: f.write(NETSCAPE_HEADER_TEXT) now = time.time() for cookie in self: diff --git a/Lib/http/server.py b/Lib/http/server.py index 58abadf7377..0ec479003a4 100644 --- a/Lib/http/server.py +++ b/Lib/http/server.py @@ -2,18 +2,18 @@ Note: BaseHTTPRequestHandler doesn't implement any HTTP request; see SimpleHTTPRequestHandler for simple implementations of GET, HEAD and POST, -and CGIHTTPRequestHandler for CGI scripts. +and (deprecated) CGIHTTPRequestHandler for CGI scripts. -It does, however, optionally implement HTTP/1.1 persistent connections, -as of version 0.3. +It does, however, optionally implement HTTP/1.1 persistent connections. Notes on CGIHTTPRequestHandler ------------------------------ -This class implements GET and POST requests to cgi-bin scripts. +This class is deprecated. It implements GET and POST requests to cgi-bin scripts. -If the os.fork() function is not present (e.g. on Windows), -subprocess.Popen() is used as a fallback, with slightly altered semantics. +If the os.fork() function is not present (Windows), subprocess.Popen() is used, +with slightly altered but never documented semantics. Use from a threaded +process is likely to trigger a warning at os.fork() time. In all cases, the implementation is intentionally naive -- all requests are executed synchronously. @@ -93,6 +93,7 @@ import html import http.client import io +import itertools import mimetypes import os import posixpath @@ -109,11 +110,10 @@ # Default error message template DEFAULT_ERROR_MESSAGE = """\ - - + + - + Error response @@ -127,6 +127,10 @@ DEFAULT_ERROR_CONTENT_TYPE = "text/html;charset=utf-8" +# Data larger than this will be read in chunks, to prevent extreme +# overallocation. +_MIN_READ_BUF_SIZE = 1 << 20 + class HTTPServer(socketserver.TCPServer): allow_reuse_address = 1 # Seems to make sense in testing environment @@ -275,6 +279,7 @@ def parse_request(self): error response has already been sent back. """ + is_http_0_9 = False self.command = None # set in case of error on the first line self.request_version = version = self.default_request_version self.close_connection = True @@ -300,6 +305,10 @@ def parse_request(self): # - Leading zeros MUST be ignored by recipients. if len(version_number) != 2: raise ValueError + if any(not component.isdigit() for component in version_number): + raise ValueError("non digit in http version") + if any(len(component) > 10 for component in version_number): + raise ValueError("unreasonable length http version") version_number = int(version_number[0]), int(version_number[1]) except (ValueError, IndexError): self.send_error( @@ -328,8 +337,21 @@ def parse_request(self): HTTPStatus.BAD_REQUEST, "Bad HTTP/0.9 request type (%r)" % command) return False + is_http_0_9 = True self.command, self.path = command, path + # gh-87389: The purpose of replacing '//' with '/' is to protect + # against open redirect attacks possibly triggered if the path starts + # with '//' because http clients treat //path as an absolute URI + # without scheme (similar to http://path) rather than a path. + if self.path.startswith('//'): + self.path = '/' + self.path.lstrip('/') # Reduce to a single / + + # For HTTP/0.9, headers are not expected at all. + if is_http_0_9: + self.headers = {} + return True + # Examine the headers and look for a Connection directive. try: self.headers = http.client.parse_headers(self.rfile, @@ -556,6 +578,11 @@ def log_error(self, format, *args): self.log_message(format, *args) + # https://en.wikipedia.org/wiki/List_of_Unicode_characters#Control_codes + _control_char_table = str.maketrans( + {c: fr'\x{c:02x}' for c in itertools.chain(range(0x20), range(0x7f,0xa0))}) + _control_char_table[ord('\\')] = r'\\' + def log_message(self, format, *args): """Log an arbitrary message. @@ -571,12 +598,16 @@ def log_message(self, format, *args): The client ip and current date/time are prefixed to every message. + Unicode control characters are replaced with escaped hex + before writing the output to stderr. + """ + message = format % args sys.stderr.write("%s - - [%s] %s\n" % (self.address_string(), self.log_date_time_string(), - format%args)) + message.translate(self._control_char_table))) def version_string(self): """Return the server software version string.""" @@ -637,6 +668,7 @@ class SimpleHTTPRequestHandler(BaseHTTPRequestHandler): """ server_version = "SimpleHTTP/" + __version__ + index_pages = ("index.html", "index.htm") extensions_map = _encodings_map_default = { '.gz': 'application/gzip', '.Z': 'application/octet-stream', @@ -680,7 +712,7 @@ def send_head(self): f = None if os.path.isdir(path): parts = urllib.parse.urlsplit(self.path) - if not parts.path.endswith('/'): + if not parts.path.endswith(('/', '%2f', '%2F')): # redirect browser - doing basically what apache does self.send_response(HTTPStatus.MOVED_PERMANENTLY) new_parts = (parts[0], parts[1], parts[2] + '/', @@ -690,9 +722,9 @@ def send_head(self): self.send_header("Content-Length", "0") self.end_headers() return None - for index in "index.html", "index.htm": + for index in self.index_pages: index = os.path.join(path, index) - if os.path.exists(index): + if os.path.isfile(index): path = index break else: @@ -702,7 +734,7 @@ def send_head(self): # The test for this was added in test_httpserver.py # However, some OS platforms accept a trailingSlash as a filename # See discussion on python-dev and Issue34711 regarding - # parseing and rejection of filenames with a trailing slash + # parsing and rejection of filenames with a trailing slash if path.endswith("/"): self.send_error(HTTPStatus.NOT_FOUND, "File not found") return None @@ -770,21 +802,23 @@ def list_directory(self, path): return None list.sort(key=lambda a: a.lower()) r = [] + displaypath = self.path + displaypath = displaypath.split('#', 1)[0] + displaypath = displaypath.split('?', 1)[0] try: - displaypath = urllib.parse.unquote(self.path, + displaypath = urllib.parse.unquote(displaypath, errors='surrogatepass') except UnicodeDecodeError: - displaypath = urllib.parse.unquote(path) + displaypath = urllib.parse.unquote(displaypath) displaypath = html.escape(displaypath, quote=False) enc = sys.getfilesystemencoding() - title = 'Directory listing for %s' % displaypath - r.append('') - r.append('\n') - r.append('' % enc) - r.append('%s\n' % title) - r.append('\n

%s

' % title) + title = f'Directory listing for {displaypath}' + r.append('') + r.append('') + r.append('') + r.append(f'') + r.append(f'{title}\n') + r.append(f'\n

{title}

') r.append('
\n
    ') for name in list: fullname = os.path.join(path, name) @@ -820,14 +854,14 @@ def translate_path(self, path): """ # abandon query parameters - path = path.split('?',1)[0] - path = path.split('#',1)[0] + path = path.split('#', 1)[0] + path = path.split('?', 1)[0] # Don't forget explicit trailing slash when normalizing. Issue17324 - trailing_slash = path.rstrip().endswith('/') try: path = urllib.parse.unquote(path, errors='surrogatepass') except UnicodeDecodeError: path = urllib.parse.unquote(path) + trailing_slash = path.endswith('/') path = posixpath.normpath(path) words = path.split('/') words = filter(None, words) @@ -877,7 +911,7 @@ def guess_type(self, path): ext = ext.lower() if ext in self.extensions_map: return self.extensions_map[ext] - guess, _ = mimetypes.guess_type(path) + guess, _ = mimetypes.guess_file_type(path) if guess: return guess return 'application/octet-stream' @@ -966,6 +1000,12 @@ class CGIHTTPRequestHandler(SimpleHTTPRequestHandler): """ + def __init__(self, *args, **kwargs): + import warnings + warnings._deprecated("http.server.CGIHTTPRequestHandler", + remove=(3, 15)) + super().__init__(*args, **kwargs) + # Determine platform specifics have_fork = hasattr(os, 'fork') @@ -1078,7 +1118,7 @@ def run_cgi(self): "CGI script is not executable (%r)" % scriptname) return - # Reference: http://hoohoo.ncsa.uiuc.edu/cgi/env.html + # Reference: https://www6.uniovi.es/~antonio/ncsa_httpd/cgi/env.html # XXX Much of the following could be prepared ahead of time! env = copy.deepcopy(os.environ) env['SERVER_SOFTWARE'] = self.version_string() @@ -1198,7 +1238,18 @@ def run_cgi(self): env = env ) if self.command.lower() == "post" and nbytes > 0: - data = self.rfile.read(nbytes) + cursize = 0 + data = self.rfile.read(min(nbytes, _MIN_READ_BUF_SIZE)) + while len(data) < nbytes and len(data) != cursize: + cursize = len(data) + # This is a geometric increase in read size (never more + # than doubling out the current length of data per loop + # iteration). + delta = min(cursize, nbytes - cursize) + try: + data += self.rfile.read(delta) + except TimeoutError: + break else: data = None # throw away additional data [see bug #427345] @@ -1258,15 +1309,19 @@ def test(HandlerClass=BaseHTTPRequestHandler, parser = argparse.ArgumentParser() parser.add_argument('--cgi', action='store_true', help='run as CGI server') - parser.add_argument('--bind', '-b', metavar='ADDRESS', - help='specify alternate bind address ' + parser.add_argument('-b', '--bind', metavar='ADDRESS', + help='bind to this address ' '(default: all interfaces)') - parser.add_argument('--directory', '-d', default=os.getcwd(), - help='specify alternate directory ' + parser.add_argument('-d', '--directory', default=os.getcwd(), + help='serve this directory ' '(default: current directory)') - parser.add_argument('port', action='store', default=8000, type=int, - nargs='?', - help='specify alternate port (default: 8000)') + parser.add_argument('-p', '--protocol', metavar='VERSION', + default='HTTP/1.0', + help='conform to this HTTP version ' + '(default: %(default)s)') + parser.add_argument('port', default=8000, type=int, nargs='?', + help='bind to this port ' + '(default: %(default)s)') args = parser.parse_args() if args.cgi: handler_class = CGIHTTPRequestHandler @@ -1292,4 +1347,5 @@ def finish_request(self, request, client_address): ServerClass=DualStackServer, port=args.port, bind=args.bind, + protocol=args.protocol, ) diff --git a/Lib/imaplib.py b/Lib/imaplib.py new file mode 100644 index 00000000000..cbe129b3e7c --- /dev/null +++ b/Lib/imaplib.py @@ -0,0 +1,1967 @@ +"""IMAP4 client. + +Based on RFC 2060. + +Public class: IMAP4 +Public variable: Debug +Public functions: Internaldate2tuple + Int2AP + ParseFlags + Time2Internaldate +""" + +# Author: Piers Lauder December 1997. +# +# Authentication code contributed by Donn Cave June 1998. +# String method conversion by ESR, February 2001. +# GET/SETACL contributed by Anthony Baxter April 2001. +# IMAP4_SSL contributed by Tino Lange March 2002. +# GET/SETQUOTA contributed by Andreas Zeidler June 2002. +# PROXYAUTH contributed by Rick Holbert November 2002. +# GET/SETANNOTATION contributed by Tomas Lindroos June 2005. +# IDLE contributed by Forest August 2024. + +__version__ = "2.60" + +import binascii, errno, random, re, socket, subprocess, sys, time, calendar +from datetime import datetime, timezone, timedelta +from io import DEFAULT_BUFFER_SIZE + +try: + import ssl + HAVE_SSL = True +except ImportError: + HAVE_SSL = False + +__all__ = ["IMAP4", "IMAP4_stream", "Internaldate2tuple", + "Int2AP", "ParseFlags", "Time2Internaldate"] + +# Globals + +CRLF = b'\r\n' +Debug = 0 +IMAP4_PORT = 143 +IMAP4_SSL_PORT = 993 +AllowedVersions = ('IMAP4REV1', 'IMAP4') # Most recent first + +# Maximal line length when calling readline(). This is to prevent +# reading arbitrary length lines. RFC 3501 and 2060 (IMAP 4rev1) +# don't specify a line length. RFC 2683 suggests limiting client +# command lines to 1000 octets and that servers should be prepared +# to accept command lines up to 8000 octets, so we used to use 10K here. +# In the modern world (eg: gmail) the response to, for example, a +# search command can be quite large, so we now use 1M. +_MAXLINE = 1000000 + + +# Commands + +Commands = { + # name valid states + 'APPEND': ('AUTH', 'SELECTED'), + 'AUTHENTICATE': ('NONAUTH',), + 'CAPABILITY': ('NONAUTH', 'AUTH', 'SELECTED', 'LOGOUT'), + 'CHECK': ('SELECTED',), + 'CLOSE': ('SELECTED',), + 'COPY': ('SELECTED',), + 'CREATE': ('AUTH', 'SELECTED'), + 'DELETE': ('AUTH', 'SELECTED'), + 'DELETEACL': ('AUTH', 'SELECTED'), + 'ENABLE': ('AUTH', ), + 'EXAMINE': ('AUTH', 'SELECTED'), + 'EXPUNGE': ('SELECTED',), + 'FETCH': ('SELECTED',), + 'GETACL': ('AUTH', 'SELECTED'), + 'GETANNOTATION':('AUTH', 'SELECTED'), + 'GETQUOTA': ('AUTH', 'SELECTED'), + 'GETQUOTAROOT': ('AUTH', 'SELECTED'), + 'IDLE': ('AUTH', 'SELECTED'), + 'MYRIGHTS': ('AUTH', 'SELECTED'), + 'LIST': ('AUTH', 'SELECTED'), + 'LOGIN': ('NONAUTH',), + 'LOGOUT': ('NONAUTH', 'AUTH', 'SELECTED', 'LOGOUT'), + 'LSUB': ('AUTH', 'SELECTED'), + 'MOVE': ('SELECTED',), + 'NAMESPACE': ('AUTH', 'SELECTED'), + 'NOOP': ('NONAUTH', 'AUTH', 'SELECTED', 'LOGOUT'), + 'PARTIAL': ('SELECTED',), # NB: obsolete + 'PROXYAUTH': ('AUTH',), + 'RENAME': ('AUTH', 'SELECTED'), + 'SEARCH': ('SELECTED',), + 'SELECT': ('AUTH', 'SELECTED'), + 'SETACL': ('AUTH', 'SELECTED'), + 'SETANNOTATION':('AUTH', 'SELECTED'), + 'SETQUOTA': ('AUTH', 'SELECTED'), + 'SORT': ('SELECTED',), + 'STARTTLS': ('NONAUTH',), + 'STATUS': ('AUTH', 'SELECTED'), + 'STORE': ('SELECTED',), + 'SUBSCRIBE': ('AUTH', 'SELECTED'), + 'THREAD': ('SELECTED',), + 'UID': ('SELECTED',), + 'UNSUBSCRIBE': ('AUTH', 'SELECTED'), + 'UNSELECT': ('SELECTED',), + } + +# Patterns to match server responses + +Continuation = re.compile(br'\+( (?P.*))?') +Flags = re.compile(br'.*FLAGS \((?P[^\)]*)\)') +InternalDate = re.compile(br'.*INTERNALDATE "' + br'(?P[ 0123][0-9])-(?P[A-Z][a-z][a-z])-(?P[0-9][0-9][0-9][0-9])' + br' (?P[0-9][0-9]):(?P[0-9][0-9]):(?P[0-9][0-9])' + br' (?P[-+])(?P[0-9][0-9])(?P[0-9][0-9])' + br'"') +# Literal is no longer used; kept for backward compatibility. +Literal = re.compile(br'.*{(?P\d+)}$', re.ASCII) +MapCRLF = re.compile(br'\r\n|\r|\n') +# We no longer exclude the ']' character from the data portion of the response +# code, even though it violates the RFC. Popular IMAP servers such as Gmail +# allow flags with ']', and there are programs (including imaplib!) that can +# produce them. The problem with this is if the 'text' portion of the response +# includes a ']' we'll parse the response wrong (which is the point of the RFC +# restriction). However, that seems less likely to be a problem in practice +# than being unable to correctly parse flags that include ']' chars, which +# was reported as a real-world problem in issue #21815. +Response_code = re.compile(br'\[(?P[A-Z-]+)( (?P.*))?\]') +Untagged_response = re.compile(br'\* (?P[A-Z-]+)( (?P.*))?') +# Untagged_status is no longer used; kept for backward compatibility +Untagged_status = re.compile( + br'\* (?P\d+) (?P[A-Z-]+)( (?P.*))?', re.ASCII) +# We compile these in _mode_xxx. +_Literal = br'.*{(?P\d+)}$' +_Untagged_status = br'\* (?P\d+) (?P[A-Z-]+)( (?P.*))?' + + + +class IMAP4: + + r"""IMAP4 client class. + + Instantiate with: IMAP4([host[, port[, timeout=None]]]) + + host - host's name (default: localhost); + port - port number (default: standard IMAP4 port). + timeout - socket timeout (default: None) + If timeout is not given or is None, + the global default socket timeout is used + + All IMAP4rev1 commands are supported by methods of the same + name (in lowercase). + + All arguments to commands are converted to strings, except for + AUTHENTICATE, and the last argument to APPEND which is passed as + an IMAP4 literal. If necessary (the string contains any + non-printing characters or white-space and isn't enclosed with + either parentheses or double quotes) each string is quoted. + However, the 'password' argument to the LOGIN command is always + quoted. If you want to avoid having an argument string quoted + (eg: the 'flags' argument to STORE) then enclose the string in + parentheses (eg: "(\Deleted)"). + + Each command returns a tuple: (type, [data, ...]) where 'type' + is usually 'OK' or 'NO', and 'data' is either the text from the + tagged response, or untagged results from command. Each 'data' + is either a string, or a tuple. If a tuple, then the first part + is the header of the response, and the second part contains + the data (ie: 'literal' value). + + Errors raise the exception class .error(""). + IMAP4 server errors raise .abort(""), + which is a sub-class of 'error'. Mailbox status changes + from READ-WRITE to READ-ONLY raise the exception class + .readonly(""), which is a sub-class of 'abort'. + + "error" exceptions imply a program error. + "abort" exceptions imply the connection should be reset, and + the command re-tried. + "readonly" exceptions imply the command should be re-tried. + + Note: to use this module, you must read the RFCs pertaining to the + IMAP4 protocol, as the semantics of the arguments to each IMAP4 + command are left to the invoker, not to mention the results. Also, + most IMAP servers implement a sub-set of the commands available here. + """ + + class error(Exception): pass # Logical errors - debug required + class abort(error): pass # Service errors - close and retry + class readonly(abort): pass # Mailbox status changed to READ-ONLY + class _responsetimeout(TimeoutError): pass # No response during IDLE + + def __init__(self, host='', port=IMAP4_PORT, timeout=None): + self.debug = Debug + self.state = 'LOGOUT' + self.literal = None # A literal argument to a command + self.tagged_commands = {} # Tagged commands awaiting response + self.untagged_responses = {} # {typ: [data, ...], ...} + self.continuation_response = '' # Last continuation response + self._idle_responses = [] # Response queue for idle iteration + self._idle_capture = False # Whether to queue responses for idle + self.is_readonly = False # READ-ONLY desired state + self.tagnum = 0 + self._tls_established = False + self._mode_ascii() + self._readbuf = [] + + # Open socket to server. + + self.open(host, port, timeout) + + try: + self._connect() + except Exception: + try: + self.shutdown() + except OSError: + pass + raise + + def _mode_ascii(self): + self.utf8_enabled = False + self._encoding = 'ascii' + self.Literal = re.compile(_Literal, re.ASCII) + self.Untagged_status = re.compile(_Untagged_status, re.ASCII) + + + def _mode_utf8(self): + self.utf8_enabled = True + self._encoding = 'utf-8' + self.Literal = re.compile(_Literal) + self.Untagged_status = re.compile(_Untagged_status) + + + def _connect(self): + # Create unique tag for this session, + # and compile tagged response matcher. + + self.tagpre = Int2AP(random.randint(4096, 65535)) + self.tagre = re.compile(br'(?P' + + self.tagpre + + br'\d+) (?P[A-Z]+) (?P.*)', re.ASCII) + + # Get server welcome message, + # request and store CAPABILITY response. + + if __debug__: + self._cmd_log_len = 10 + self._cmd_log_idx = 0 + self._cmd_log = {} # Last '_cmd_log_len' interactions + if self.debug >= 1: + self._mesg('imaplib version %s' % __version__) + self._mesg('new IMAP4 connection, tag=%s' % self.tagpre) + + self.welcome = self._get_response() + if 'PREAUTH' in self.untagged_responses: + self.state = 'AUTH' + elif 'OK' in self.untagged_responses: + self.state = 'NONAUTH' + else: + raise self.error(self.welcome) + + self._get_capabilities() + if __debug__: + if self.debug >= 3: + self._mesg('CAPABILITIES: %r' % (self.capabilities,)) + + for version in AllowedVersions: + if not version in self.capabilities: + continue + self.PROTOCOL_VERSION = version + return + + raise self.error('server not IMAP4 compliant') + + + def __getattr__(self, attr): + # Allow UPPERCASE variants of IMAP4 command methods. + if attr in Commands: + return getattr(self, attr.lower()) + raise AttributeError("Unknown IMAP4 command: '%s'" % attr) + + def __enter__(self): + return self + + def __exit__(self, *args): + if self.state == "LOGOUT": + return + + try: + self.logout() + except OSError: + pass + + + # Overridable methods + + + def _create_socket(self, timeout): + # Default value of IMAP4.host is '', but socket.getaddrinfo() + # (which is used by socket.create_connection()) expects None + # as a default value for host. + if timeout is not None and not timeout: + raise ValueError('Non-blocking socket (timeout=0) is not supported') + host = None if not self.host else self.host + sys.audit("imaplib.open", self, self.host, self.port) + address = (host, self.port) + if timeout is not None: + return socket.create_connection(address, timeout) + return socket.create_connection(address) + + def open(self, host='', port=IMAP4_PORT, timeout=None): + """Setup connection to remote server on "host:port" + (default: localhost:standard IMAP4 port). + This connection will be used by the routines: + read, readline, send, shutdown. + """ + self.host = host + self.port = port + self.sock = self._create_socket(timeout) + self._file = self.sock.makefile('rb') + + + @property + def file(self): + # The old 'file' attribute is no longer used now that we do our own + # read() and readline() buffering, with which it conflicts. + # As an undocumented interface, it should never have been accessed by + # external code, and therefore does not warrant deprecation. + # Nevertheless, we provide this property for now, to avoid suddenly + # breaking any code in the wild that might have been using it in a + # harmless way. + import warnings + warnings.warn( + 'IMAP4.file is unsupported, can cause errors, and may be removed.', + RuntimeWarning, + stacklevel=2) + return self._file + + + def read(self, size): + """Read 'size' bytes from remote.""" + # We need buffered read() to continue working after socket timeouts, + # since we use them during IDLE. Unfortunately, the standard library's + # SocketIO implementation makes this impossible, by setting a permanent + # error condition instead of letting the caller decide how to handle a + # timeout. We therefore implement our own buffered read(). + # https://github.com/python/cpython/issues/51571 + # + # Reading in chunks instead of delegating to a single + # BufferedReader.read() call also means we avoid its preallocation + # of an unreasonably large memory block if a malicious server claims + # it will send a huge literal without actually sending one. + # https://github.com/python/cpython/issues/119511 + + parts = [] + + while size > 0: + + if len(parts) < len(self._readbuf): + buf = self._readbuf[len(parts)] + else: + try: + buf = self.sock.recv(DEFAULT_BUFFER_SIZE) + except ConnectionError: + break + if not buf: + break + self._readbuf.append(buf) + + if len(buf) >= size: + parts.append(buf[:size]) + self._readbuf = [buf[size:]] + self._readbuf[len(parts):] + break + parts.append(buf) + size -= len(buf) + + return b''.join(parts) + + + def readline(self): + """Read line from remote.""" + # The comment in read() explains why we implement our own readline(). + + LF = b'\n' + parts = [] + length = 0 + + while length < _MAXLINE: + + if len(parts) < len(self._readbuf): + buf = self._readbuf[len(parts)] + else: + try: + buf = self.sock.recv(DEFAULT_BUFFER_SIZE) + except ConnectionError: + break + if not buf: + break + self._readbuf.append(buf) + + pos = buf.find(LF) + if pos != -1: + pos += 1 + parts.append(buf[:pos]) + self._readbuf = [buf[pos:]] + self._readbuf[len(parts):] + break + parts.append(buf) + length += len(buf) + + line = b''.join(parts) + if len(line) > _MAXLINE: + raise self.error("got more than %d bytes" % _MAXLINE) + return line + + + def send(self, data): + """Send data to remote.""" + sys.audit("imaplib.send", self, data) + self.sock.sendall(data) + + + def shutdown(self): + """Close I/O established in "open".""" + self._file.close() + try: + self.sock.shutdown(socket.SHUT_RDWR) + except OSError as exc: + # The server might already have closed the connection. + # On Windows, this may result in WSAEINVAL (error 10022): + # An invalid operation was attempted. + if (exc.errno != errno.ENOTCONN + and getattr(exc, 'winerror', 0) != 10022): + raise + finally: + self.sock.close() + + + def socket(self): + """Return socket instance used to connect to IMAP4 server. + + socket = .socket() + """ + return self.sock + + + + # Utility methods + + + def recent(self): + """Return most recent 'RECENT' responses if any exist, + else prompt server for an update using the 'NOOP' command. + + (typ, [data]) = .recent() + + 'data' is None if no new messages, + else list of RECENT responses, most recent last. + """ + name = 'RECENT' + typ, dat = self._untagged_response('OK', [None], name) + if dat[-1]: + return typ, dat + typ, dat = self.noop() # Prod server for response + return self._untagged_response(typ, dat, name) + + + def response(self, code): + """Return data for response 'code' if received, or None. + + Old value for response 'code' is cleared. + + (code, [data]) = .response(code) + """ + return self._untagged_response(code, [None], code.upper()) + + + + # IMAP4 commands + + + def append(self, mailbox, flags, date_time, message): + """Append message to named mailbox. + + (typ, [data]) = .append(mailbox, flags, date_time, message) + + All args except 'message' can be None. + """ + name = 'APPEND' + if not mailbox: + mailbox = 'INBOX' + if flags: + if (flags[0],flags[-1]) != ('(',')'): + flags = '(%s)' % flags + else: + flags = None + if date_time: + date_time = Time2Internaldate(date_time) + else: + date_time = None + literal = MapCRLF.sub(CRLF, message) + self.literal = literal + return self._simple_command(name, mailbox, flags, date_time) + + + def authenticate(self, mechanism, authobject): + """Authenticate command - requires response processing. + + 'mechanism' specifies which authentication mechanism is to + be used - it must appear in .capabilities in the + form AUTH=. + + 'authobject' must be a callable object: + + data = authobject(response) + + It will be called to process server continuation responses; the + response argument it is passed will be a bytes. It should return bytes + data that will be base64 encoded and sent to the server. It should + return None if the client abort response '*' should be sent instead. + """ + mech = mechanism.upper() + # XXX: shouldn't this code be removed, not commented out? + #cap = 'AUTH=%s' % mech + #if not cap in self.capabilities: # Let the server decide! + # raise self.error("Server doesn't allow %s authentication." % mech) + self.literal = _Authenticator(authobject).process + typ, dat = self._simple_command('AUTHENTICATE', mech) + if typ != 'OK': + raise self.error(dat[-1].decode('utf-8', 'replace')) + self.state = 'AUTH' + return typ, dat + + + def capability(self): + """(typ, [data]) = .capability() + Fetch capabilities list from server.""" + + name = 'CAPABILITY' + typ, dat = self._simple_command(name) + return self._untagged_response(typ, dat, name) + + + def check(self): + """Checkpoint mailbox on server. + + (typ, [data]) = .check() + """ + return self._simple_command('CHECK') + + + def close(self): + """Close currently selected mailbox. + + Deleted messages are removed from writable mailbox. + This is the recommended command before 'LOGOUT'. + + (typ, [data]) = .close() + """ + try: + typ, dat = self._simple_command('CLOSE') + finally: + self.state = 'AUTH' + return typ, dat + + + def copy(self, message_set, new_mailbox): + """Copy 'message_set' messages onto end of 'new_mailbox'. + + (typ, [data]) = .copy(message_set, new_mailbox) + """ + return self._simple_command('COPY', message_set, new_mailbox) + + + def create(self, mailbox): + """Create new mailbox. + + (typ, [data]) = .create(mailbox) + """ + return self._simple_command('CREATE', mailbox) + + + def delete(self, mailbox): + """Delete old mailbox. + + (typ, [data]) = .delete(mailbox) + """ + return self._simple_command('DELETE', mailbox) + + def deleteacl(self, mailbox, who): + """Delete the ACLs (remove any rights) set for who on mailbox. + + (typ, [data]) = .deleteacl(mailbox, who) + """ + return self._simple_command('DELETEACL', mailbox, who) + + def enable(self, capability): + """Send an RFC5161 enable string to the server. + + (typ, [data]) = .enable(capability) + """ + if 'ENABLE' not in self.capabilities: + raise IMAP4.error("Server does not support ENABLE") + typ, data = self._simple_command('ENABLE', capability) + if typ == 'OK' and 'UTF8=ACCEPT' in capability.upper(): + self._mode_utf8() + return typ, data + + def expunge(self): + """Permanently remove deleted items from selected mailbox. + + Generates 'EXPUNGE' response for each deleted message. + + (typ, [data]) = .expunge() + + 'data' is list of 'EXPUNGE'd message numbers in order received. + """ + name = 'EXPUNGE' + typ, dat = self._simple_command(name) + return self._untagged_response(typ, dat, name) + + + def fetch(self, message_set, message_parts): + """Fetch (parts of) messages. + + (typ, [data, ...]) = .fetch(message_set, message_parts) + + 'message_parts' should be a string of selected parts + enclosed in parentheses, eg: "(UID BODY[TEXT])". + + 'data' are tuples of message part envelope and data. + """ + name = 'FETCH' + typ, dat = self._simple_command(name, message_set, message_parts) + return self._untagged_response(typ, dat, name) + + + def getacl(self, mailbox): + """Get the ACLs for a mailbox. + + (typ, [data]) = .getacl(mailbox) + """ + typ, dat = self._simple_command('GETACL', mailbox) + return self._untagged_response(typ, dat, 'ACL') + + + def getannotation(self, mailbox, entry, attribute): + """(typ, [data]) = .getannotation(mailbox, entry, attribute) + Retrieve ANNOTATIONs.""" + + typ, dat = self._simple_command('GETANNOTATION', mailbox, entry, attribute) + return self._untagged_response(typ, dat, 'ANNOTATION') + + + def getquota(self, root): + """Get the quota root's resource usage and limits. + + Part of the IMAP4 QUOTA extension defined in rfc2087. + + (typ, [data]) = .getquota(root) + """ + typ, dat = self._simple_command('GETQUOTA', root) + return self._untagged_response(typ, dat, 'QUOTA') + + + def getquotaroot(self, mailbox): + """Get the list of quota roots for the named mailbox. + + (typ, [[QUOTAROOT responses...], [QUOTA responses]]) = .getquotaroot(mailbox) + """ + typ, dat = self._simple_command('GETQUOTAROOT', mailbox) + typ, quota = self._untagged_response(typ, dat, 'QUOTA') + typ, quotaroot = self._untagged_response(typ, dat, 'QUOTAROOT') + return typ, [quotaroot, quota] + + + def idle(self, duration=None): + """Return an iterable IDLE context manager producing untagged responses. + If the argument is not None, limit iteration to 'duration' seconds. + + with M.idle(duration=29 * 60) as idler: + for typ, data in idler: + print(typ, data) + + Note: 'duration' requires a socket connection (not IMAP4_stream). + """ + return Idler(self, duration) + + + def list(self, directory='""', pattern='*'): + """List mailbox names in directory matching pattern. + + (typ, [data]) = .list(directory='""', pattern='*') + + 'data' is list of LIST responses. + """ + name = 'LIST' + typ, dat = self._simple_command(name, directory, pattern) + return self._untagged_response(typ, dat, name) + + + def login(self, user, password): + """Identify client using plaintext password. + + (typ, [data]) = .login(user, password) + + NB: 'password' will be quoted. + """ + typ, dat = self._simple_command('LOGIN', user, self._quote(password)) + if typ != 'OK': + raise self.error(dat[-1]) + self.state = 'AUTH' + return typ, dat + + + def login_cram_md5(self, user, password): + """ Force use of CRAM-MD5 authentication. + + (typ, [data]) = .login_cram_md5(user, password) + """ + self.user, self.password = user, password + return self.authenticate('CRAM-MD5', self._CRAM_MD5_AUTH) + + + def _CRAM_MD5_AUTH(self, challenge): + """ Authobject to use with CRAM-MD5 authentication. """ + import hmac + + if isinstance(self.password, str): + password = self.password.encode('utf-8') + else: + password = self.password + + try: + authcode = hmac.HMAC(password, challenge, 'md5') + except ValueError: # HMAC-MD5 is not available + raise self.error("CRAM-MD5 authentication is not supported") + return f"{self.user} {authcode.hexdigest()}" + + + def logout(self): + """Shutdown connection to server. + + (typ, [data]) = .logout() + + Returns server 'BYE' response. + """ + self.state = 'LOGOUT' + typ, dat = self._simple_command('LOGOUT') + self.shutdown() + return typ, dat + + + def lsub(self, directory='""', pattern='*'): + """List 'subscribed' mailbox names in directory matching pattern. + + (typ, [data, ...]) = .lsub(directory='""', pattern='*') + + 'data' are tuples of message part envelope and data. + """ + name = 'LSUB' + typ, dat = self._simple_command(name, directory, pattern) + return self._untagged_response(typ, dat, name) + + def myrights(self, mailbox): + """Show my ACLs for a mailbox (i.e. the rights that I have on mailbox). + + (typ, [data]) = .myrights(mailbox) + """ + typ,dat = self._simple_command('MYRIGHTS', mailbox) + return self._untagged_response(typ, dat, 'MYRIGHTS') + + def namespace(self): + """ Returns IMAP namespaces ala rfc2342 + + (typ, [data, ...]) = .namespace() + """ + name = 'NAMESPACE' + typ, dat = self._simple_command(name) + return self._untagged_response(typ, dat, name) + + + def noop(self): + """Send NOOP command. + + (typ, [data]) = .noop() + """ + if __debug__: + if self.debug >= 3: + self._dump_ur(self.untagged_responses) + return self._simple_command('NOOP') + + + def partial(self, message_num, message_part, start, length): + """Fetch truncated part of a message. + + (typ, [data, ...]) = .partial(message_num, message_part, start, length) + + 'data' is tuple of message part envelope and data. + """ + name = 'PARTIAL' + typ, dat = self._simple_command(name, message_num, message_part, start, length) + return self._untagged_response(typ, dat, 'FETCH') + + + def proxyauth(self, user): + """Assume authentication as "user". + + Allows an authorised administrator to proxy into any user's + mailbox. + + (typ, [data]) = .proxyauth(user) + """ + + name = 'PROXYAUTH' + return self._simple_command('PROXYAUTH', user) + + + def rename(self, oldmailbox, newmailbox): + """Rename old mailbox name to new. + + (typ, [data]) = .rename(oldmailbox, newmailbox) + """ + return self._simple_command('RENAME', oldmailbox, newmailbox) + + + def search(self, charset, *criteria): + """Search mailbox for matching messages. + + (typ, [data]) = .search(charset, criterion, ...) + + 'data' is space separated list of matching message numbers. + If UTF8 is enabled, charset MUST be None. + """ + name = 'SEARCH' + if charset: + if self.utf8_enabled: + raise IMAP4.error("Non-None charset not valid in UTF8 mode") + typ, dat = self._simple_command(name, 'CHARSET', charset, *criteria) + else: + typ, dat = self._simple_command(name, *criteria) + return self._untagged_response(typ, dat, name) + + + def select(self, mailbox='INBOX', readonly=False): + """Select a mailbox. + + Flush all untagged responses. + + (typ, [data]) = .select(mailbox='INBOX', readonly=False) + + 'data' is count of messages in mailbox ('EXISTS' response). + + Mandated responses are ('FLAGS', 'EXISTS', 'RECENT', 'UIDVALIDITY'), so + other responses should be obtained via .response('FLAGS') etc. + """ + self.untagged_responses = {} # Flush old responses. + self.is_readonly = readonly + if readonly: + name = 'EXAMINE' + else: + name = 'SELECT' + typ, dat = self._simple_command(name, mailbox) + if typ != 'OK': + self.state = 'AUTH' # Might have been 'SELECTED' + return typ, dat + self.state = 'SELECTED' + if 'READ-ONLY' in self.untagged_responses \ + and not readonly: + if __debug__: + if self.debug >= 1: + self._dump_ur(self.untagged_responses) + raise self.readonly('%s is not writable' % mailbox) + return typ, self.untagged_responses.get('EXISTS', [None]) + + + def setacl(self, mailbox, who, what): + """Set a mailbox acl. + + (typ, [data]) = .setacl(mailbox, who, what) + """ + return self._simple_command('SETACL', mailbox, who, what) + + + def setannotation(self, *args): + """(typ, [data]) = .setannotation(mailbox[, entry, attribute]+) + Set ANNOTATIONs.""" + + typ, dat = self._simple_command('SETANNOTATION', *args) + return self._untagged_response(typ, dat, 'ANNOTATION') + + + def setquota(self, root, limits): + """Set the quota root's resource limits. + + (typ, [data]) = .setquota(root, limits) + """ + typ, dat = self._simple_command('SETQUOTA', root, limits) + return self._untagged_response(typ, dat, 'QUOTA') + + + def sort(self, sort_criteria, charset, *search_criteria): + """IMAP4rev1 extension SORT command. + + (typ, [data]) = .sort(sort_criteria, charset, search_criteria, ...) + """ + name = 'SORT' + #if not name in self.capabilities: # Let the server decide! + # raise self.error('unimplemented extension command: %s' % name) + if (sort_criteria[0],sort_criteria[-1]) != ('(',')'): + sort_criteria = '(%s)' % sort_criteria + typ, dat = self._simple_command(name, sort_criteria, charset, *search_criteria) + return self._untagged_response(typ, dat, name) + + + def starttls(self, ssl_context=None): + name = 'STARTTLS' + if not HAVE_SSL: + raise self.error('SSL support missing') + if self._tls_established: + raise self.abort('TLS session already established') + if name not in self.capabilities: + raise self.abort('TLS not supported by server') + # Generate a default SSL context if none was passed. + if ssl_context is None: + ssl_context = ssl._create_stdlib_context() + typ, dat = self._simple_command(name) + if typ == 'OK': + self.sock = ssl_context.wrap_socket(self.sock, + server_hostname=self.host) + self._file = self.sock.makefile('rb') + self._tls_established = True + self._get_capabilities() + else: + raise self.error("Couldn't establish TLS session") + return self._untagged_response(typ, dat, name) + + + def status(self, mailbox, names): + """Request named status conditions for mailbox. + + (typ, [data]) = .status(mailbox, names) + """ + name = 'STATUS' + #if self.PROTOCOL_VERSION == 'IMAP4': # Let the server decide! + # raise self.error('%s unimplemented in IMAP4 (obtain IMAP4rev1 server, or re-code)' % name) + typ, dat = self._simple_command(name, mailbox, names) + return self._untagged_response(typ, dat, name) + + + def store(self, message_set, command, flags): + """Alters flag dispositions for messages in mailbox. + + (typ, [data]) = .store(message_set, command, flags) + """ + if (flags[0],flags[-1]) != ('(',')'): + flags = '(%s)' % flags # Avoid quoting the flags + typ, dat = self._simple_command('STORE', message_set, command, flags) + return self._untagged_response(typ, dat, 'FETCH') + + + def subscribe(self, mailbox): + """Subscribe to new mailbox. + + (typ, [data]) = .subscribe(mailbox) + """ + return self._simple_command('SUBSCRIBE', mailbox) + + + def thread(self, threading_algorithm, charset, *search_criteria): + """IMAPrev1 extension THREAD command. + + (type, [data]) = .thread(threading_algorithm, charset, search_criteria, ...) + """ + name = 'THREAD' + typ, dat = self._simple_command(name, threading_algorithm, charset, *search_criteria) + return self._untagged_response(typ, dat, name) + + + def uid(self, command, *args): + """Execute "command arg ..." with messages identified by UID, + rather than message number. + + (typ, [data]) = .uid(command, arg1, arg2, ...) + + Returns response appropriate to 'command'. + """ + command = command.upper() + if not command in Commands: + raise self.error("Unknown IMAP4 UID command: %s" % command) + if self.state not in Commands[command]: + raise self.error("command %s illegal in state %s, " + "only allowed in states %s" % + (command, self.state, + ', '.join(Commands[command]))) + name = 'UID' + typ, dat = self._simple_command(name, command, *args) + if command in ('SEARCH', 'SORT', 'THREAD'): + name = command + else: + name = 'FETCH' + return self._untagged_response(typ, dat, name) + + + def unsubscribe(self, mailbox): + """Unsubscribe from old mailbox. + + (typ, [data]) = .unsubscribe(mailbox) + """ + return self._simple_command('UNSUBSCRIBE', mailbox) + + + def unselect(self): + """Free server's resources associated with the selected mailbox + and returns the server to the authenticated state. + This command performs the same actions as CLOSE, except + that no messages are permanently removed from the currently + selected mailbox. + + (typ, [data]) = .unselect() + """ + try: + typ, data = self._simple_command('UNSELECT') + finally: + self.state = 'AUTH' + return typ, data + + + def xatom(self, name, *args): + """Allow simple extension commands + notified by server in CAPABILITY response. + + Assumes command is legal in current state. + + (typ, [data]) = .xatom(name, arg, ...) + + Returns response appropriate to extension command 'name'. + """ + name = name.upper() + #if not name in self.capabilities: # Let the server decide! + # raise self.error('unknown extension command: %s' % name) + if not name in Commands: + Commands[name] = (self.state,) + return self._simple_command(name, *args) + + + + # Private methods + + + def _append_untagged(self, typ, dat): + if dat is None: + dat = b'' + + # During idle, queue untagged responses for delivery via iteration + if self._idle_capture: + # Responses containing literal strings are passed to us one data + # fragment at a time, while others arrive in a single call. + if (not self._idle_responses or + isinstance(self._idle_responses[-1][1][-1], bytes)): + # We are not continuing a fragmented response; start a new one + self._idle_responses.append((typ, [dat])) + else: + # We are continuing a fragmented response; append the fragment + response = self._idle_responses[-1] + assert response[0] == typ + response[1].append(dat) + if __debug__ and self.debug >= 5: + self._mesg(f'idle: queue untagged {typ} {dat!r}') + return + + ur = self.untagged_responses + if __debug__: + if self.debug >= 5: + self._mesg('untagged_responses[%s] %s += ["%r"]' % + (typ, len(ur.get(typ,'')), dat)) + if typ in ur: + ur[typ].append(dat) + else: + ur[typ] = [dat] + + + def _check_bye(self): + bye = self.untagged_responses.get('BYE') + if bye: + raise self.abort(bye[-1].decode(self._encoding, 'replace')) + + + def _command(self, name, *args): + + if self.state not in Commands[name]: + self.literal = None + raise self.error("command %s illegal in state %s, " + "only allowed in states %s" % + (name, self.state, + ', '.join(Commands[name]))) + + for typ in ('OK', 'NO', 'BAD'): + if typ in self.untagged_responses: + del self.untagged_responses[typ] + + if 'READ-ONLY' in self.untagged_responses \ + and not self.is_readonly: + raise self.readonly('mailbox status changed to READ-ONLY') + + tag = self._new_tag() + name = bytes(name, self._encoding) + data = tag + b' ' + name + for arg in args: + if arg is None: continue + if isinstance(arg, str): + arg = bytes(arg, self._encoding) + data = data + b' ' + arg + + literal = self.literal + if literal is not None: + self.literal = None + if type(literal) is type(self._command): + literator = literal + else: + literator = None + if self.utf8_enabled: + data = data + bytes(' UTF8 (~{%s}' % len(literal), self._encoding) + literal = literal + b')' + else: + data = data + bytes(' {%s}' % len(literal), self._encoding) + + if __debug__: + if self.debug >= 4: + self._mesg('> %r' % data) + else: + self._log('> %r' % data) + + try: + self.send(data + CRLF) + except OSError as val: + raise self.abort('socket error: %s' % val) + + if literal is None: + return tag + + while 1: + # Wait for continuation response + + while self._get_response(): + if self.tagged_commands[tag]: # BAD/NO? + return tag + + # Send literal + + if literator: + literal = literator(self.continuation_response) + + if __debug__: + if self.debug >= 4: + self._mesg('write literal size %s' % len(literal)) + + try: + self.send(literal) + self.send(CRLF) + except OSError as val: + raise self.abort('socket error: %s' % val) + + if not literator: + break + + return tag + + + def _command_complete(self, name, tag): + logout = (name == 'LOGOUT') + # BYE is expected after LOGOUT + if not logout: + self._check_bye() + try: + typ, data = self._get_tagged_response(tag, expect_bye=logout) + except self.abort as val: + raise self.abort('command: %s => %s' % (name, val)) + except self.error as val: + raise self.error('command: %s => %s' % (name, val)) + if not logout: + self._check_bye() + if typ == 'BAD': + raise self.error('%s command error: %s %s' % (name, typ, data)) + return typ, data + + + def _get_capabilities(self): + typ, dat = self.capability() + if dat == [None]: + raise self.error('no CAPABILITY response from server') + dat = str(dat[-1], self._encoding) + dat = dat.upper() + self.capabilities = tuple(dat.split()) + + + def _get_response(self, start_timeout=False): + + # Read response and store. + # + # Returns None for continuation responses, + # otherwise first response line received. + # + # If start_timeout is given, temporarily uses it as a socket + # timeout while waiting for the start of a response, raising + # _responsetimeout if one doesn't arrive. (Used by Idler.) + + if start_timeout is not False and self.sock: + assert start_timeout is None or start_timeout > 0 + saved_timeout = self.sock.gettimeout() + self.sock.settimeout(start_timeout) + try: + resp = self._get_line() + except TimeoutError as err: + raise self._responsetimeout from err + finally: + self.sock.settimeout(saved_timeout) + else: + resp = self._get_line() + + # Command completion response? + + if self._match(self.tagre, resp): + tag = self.mo.group('tag') + if not tag in self.tagged_commands: + raise self.abort('unexpected tagged response: %r' % resp) + + typ = self.mo.group('type') + typ = str(typ, self._encoding) + dat = self.mo.group('data') + self.tagged_commands[tag] = (typ, [dat]) + else: + dat2 = None + + # '*' (untagged) responses? + + if not self._match(Untagged_response, resp): + if self._match(self.Untagged_status, resp): + dat2 = self.mo.group('data2') + + if self.mo is None: + # Only other possibility is '+' (continuation) response... + + if self._match(Continuation, resp): + self.continuation_response = self.mo.group('data') + return None # NB: indicates continuation + + raise self.abort("unexpected response: %r" % resp) + + typ = self.mo.group('type') + typ = str(typ, self._encoding) + dat = self.mo.group('data') + if dat is None: dat = b'' # Null untagged response + if dat2: dat = dat + b' ' + dat2 + + # Is there a literal to come? + + while self._match(self.Literal, dat): + + # Read literal direct from connection. + + size = int(self.mo.group('size')) + if __debug__: + if self.debug >= 4: + self._mesg('read literal size %s' % size) + data = self.read(size) + + # Store response with literal as tuple + + self._append_untagged(typ, (dat, data)) + + # Read trailer - possibly containing another literal + + dat = self._get_line() + + self._append_untagged(typ, dat) + + # Bracketed response information? + + if typ in ('OK', 'NO', 'BAD') and self._match(Response_code, dat): + typ = self.mo.group('type') + typ = str(typ, self._encoding) + self._append_untagged(typ, self.mo.group('data')) + + if __debug__: + if self.debug >= 1 and typ in ('NO', 'BAD', 'BYE'): + self._mesg('%s response: %r' % (typ, dat)) + + return resp + + + def _get_tagged_response(self, tag, expect_bye=False): + + while 1: + result = self.tagged_commands[tag] + if result is not None: + del self.tagged_commands[tag] + return result + + if expect_bye: + typ = 'BYE' + bye = self.untagged_responses.pop(typ, None) + if bye is not None: + # Server replies to the "LOGOUT" command with "BYE" + return (typ, bye) + + # If we've seen a BYE at this point, the socket will be + # closed, so report the BYE now. + self._check_bye() + + # Some have reported "unexpected response" exceptions. + # Note that ignoring them here causes loops. + # Instead, send me details of the unexpected response and + # I'll update the code in '_get_response()'. + + try: + self._get_response() + except self.abort as val: + if __debug__: + if self.debug >= 1: + self.print_log() + raise + + + def _get_line(self): + + line = self.readline() + if not line: + raise self.abort('socket error: EOF') + + # Protocol mandates all lines terminated by CRLF + if not line.endswith(b'\r\n'): + raise self.abort('socket error: unterminated line: %r' % line) + + line = line[:-2] + if __debug__: + if self.debug >= 4: + self._mesg('< %r' % line) + else: + self._log('< %r' % line) + return line + + + def _match(self, cre, s): + + # Run compiled regular expression match method on 's'. + # Save result, return success. + + self.mo = cre.match(s) + if __debug__: + if self.mo is not None and self.debug >= 5: + self._mesg("\tmatched %r => %r" % (cre.pattern, self.mo.groups())) + return self.mo is not None + + + def _new_tag(self): + + tag = self.tagpre + bytes(str(self.tagnum), self._encoding) + self.tagnum = self.tagnum + 1 + self.tagged_commands[tag] = None + return tag + + + def _quote(self, arg): + + arg = arg.replace('\\', '\\\\') + arg = arg.replace('"', '\\"') + + return '"' + arg + '"' + + + def _simple_command(self, name, *args): + + return self._command_complete(name, self._command(name, *args)) + + + def _untagged_response(self, typ, dat, name): + if typ == 'NO': + return typ, dat + if not name in self.untagged_responses: + return typ, [None] + data = self.untagged_responses.pop(name) + if __debug__: + if self.debug >= 5: + self._mesg('untagged_responses[%s] => %s' % (name, data)) + return typ, data + + + if __debug__: + + def _mesg(self, s, secs=None): + if secs is None: + secs = time.time() + tm = time.strftime('%M:%S', time.localtime(secs)) + sys.stderr.write(' %s.%02d %s\n' % (tm, (secs*100)%100, s)) + sys.stderr.flush() + + def _dump_ur(self, untagged_resp_dict): + if not untagged_resp_dict: + return + items = (f'{key}: {value!r}' + for key, value in untagged_resp_dict.items()) + self._mesg('untagged responses dump:' + '\n\t\t'.join(items)) + + def _log(self, line): + # Keep log of last '_cmd_log_len' interactions for debugging. + self._cmd_log[self._cmd_log_idx] = (line, time.time()) + self._cmd_log_idx += 1 + if self._cmd_log_idx >= self._cmd_log_len: + self._cmd_log_idx = 0 + + def print_log(self): + self._mesg('last %d IMAP4 interactions:' % len(self._cmd_log)) + i, n = self._cmd_log_idx, self._cmd_log_len + while n: + try: + self._mesg(*self._cmd_log[i]) + except: + pass + i += 1 + if i >= self._cmd_log_len: + i = 0 + n -= 1 + + +class Idler: + """Iterable IDLE context manager: start IDLE & produce untagged responses. + + An object of this type is returned by the IMAP4.idle() method. + + Note: The name and structure of this class are subject to change. + """ + + def __init__(self, imap, duration=None): + if 'IDLE' not in imap.capabilities: + raise imap.error("Server does not support IMAP4 IDLE") + if duration is not None and not imap.sock: + # IMAP4_stream pipes don't support timeouts + raise imap.error('duration requires a socket connection') + self._duration = duration + self._deadline = None + self._imap = imap + self._tag = None + self._saved_state = None + + def __enter__(self): + imap = self._imap + assert not imap._idle_responses + assert not imap._idle_capture + + if __debug__ and imap.debug >= 4: + imap._mesg(f'idle start duration={self._duration}') + + # Start capturing untagged responses before sending IDLE, + # so we can deliver via iteration any that arrive while + # the IDLE command continuation request is still pending. + imap._idle_capture = True + + try: + self._tag = imap._command('IDLE') + # As with any command, the server is allowed to send us unrelated, + # untagged responses before acting on IDLE. These lines will be + # returned by _get_response(). When the server is ready, it will + # send an IDLE continuation request, indicated by _get_response() + # returning None. We therefore process responses in a loop until + # this occurs. + while resp := imap._get_response(): + if imap.tagged_commands[self._tag]: + typ, data = imap.tagged_commands.pop(self._tag) + if typ == 'NO': + raise imap.error(f'idle denied: {data}') + raise imap.abort(f'unexpected status response: {resp}') + + if __debug__ and imap.debug >= 4: + prompt = imap.continuation_response + imap._mesg(f'idle continuation prompt: {prompt}') + except BaseException: + imap._idle_capture = False + raise + + if self._duration is not None: + self._deadline = time.monotonic() + self._duration + + self._saved_state = imap.state + imap.state = 'IDLING' + + return self + + def __exit__(self, exc_type, exc_val, exc_tb): + imap = self._imap + + if __debug__ and imap.debug >= 4: + imap._mesg('idle done') + imap.state = self._saved_state + + # Stop intercepting untagged responses before sending DONE, + # since we can no longer deliver them via iteration. + imap._idle_capture = False + + # If we captured untagged responses while the IDLE command + # continuation request was still pending, but the user did not + # iterate over them before exiting IDLE, we must put them + # someplace where the user can retrieve them. The only + # sensible place for this is the untagged_responses dict, + # despite its unfortunate inability to preserve the relative + # order of different response types. + if leftovers := len(imap._idle_responses): + if __debug__ and imap.debug >= 4: + imap._mesg(f'idle quit with {leftovers} leftover responses') + while imap._idle_responses: + typ, data = imap._idle_responses.pop(0) + # Append one fragment at a time, just as _get_response() does + for datum in data: + imap._append_untagged(typ, datum) + + try: + imap.send(b'DONE' + CRLF) + status, [msg] = imap._command_complete('IDLE', self._tag) + if __debug__ and imap.debug >= 4: + imap._mesg(f'idle status: {status} {msg!r}') + except OSError: + if not exc_type: + raise + + return False # Do not suppress context body exceptions + + def __iter__(self): + return self + + def _pop(self, timeout, default=('', None)): + # Get the next response, or a default value on timeout. + # The timeout arg can be an int or float, or None for no timeout. + # Timeouts require a socket connection (not IMAP4_stream). + # This method ignores self._duration. + + # Historical Note: + # The timeout was originally implemented using select() after + # checking for the presence of already-buffered data. + # That allowed timeouts on pipe connetions like IMAP4_stream. + # However, it seemed possible that SSL data arriving without any + # IMAP data afterward could cause select() to indicate available + # application data when there was none, leading to a read() call + # that would block with no timeout. It was unclear under what + # conditions this would happen in practice. Our implementation was + # changed to use socket timeouts instead of select(), just to be + # safe. + + imap = self._imap + if imap.state != 'IDLING': + raise imap.error('_pop() only works during IDLE') + + if imap._idle_responses: + # Response is ready to return to the user + resp = imap._idle_responses.pop(0) + if __debug__ and imap.debug >= 4: + imap._mesg(f'idle _pop({timeout}) de-queued {resp[0]}') + return resp + + if __debug__ and imap.debug >= 4: + imap._mesg(f'idle _pop({timeout}) reading') + + if timeout is not None: + if timeout <= 0: + return default + timeout = float(timeout) # Required by socket.settimeout() + + try: + imap._get_response(timeout) # Reads line, calls _append_untagged() + except IMAP4._responsetimeout: + if __debug__ and imap.debug >= 4: + imap._mesg(f'idle _pop({timeout}) done') + return default + + resp = imap._idle_responses.pop(0) + + if __debug__ and imap.debug >= 4: + imap._mesg(f'idle _pop({timeout}) read {resp[0]}') + return resp + + def __next__(self): + imap = self._imap + + if self._duration is None: + timeout = None + else: + timeout = self._deadline - time.monotonic() + typ, data = self._pop(timeout) + + if not typ: + if __debug__ and imap.debug >= 4: + imap._mesg('idle iterator exhausted') + raise StopIteration + + return typ, data + + def burst(self, interval=0.1): + """Yield a burst of responses no more than 'interval' seconds apart. + + with M.idle() as idler: + # get a response and any others following by < 0.1 seconds + batch = list(idler.burst()) + print(f'processing {len(batch)} responses...') + print(batch) + + Note: This generator requires a socket connection (not IMAP4_stream). + """ + if not self._imap.sock: + raise self._imap.error('burst() requires a socket connection') + + try: + yield next(self) + except StopIteration: + return + + while response := self._pop(interval, None): + yield response + + +if HAVE_SSL: + + class IMAP4_SSL(IMAP4): + + """IMAP4 client class over SSL connection + + Instantiate with: IMAP4_SSL([host[, port[, ssl_context[, timeout=None]]]]) + + host - host's name (default: localhost); + port - port number (default: standard IMAP4 SSL port); + ssl_context - a SSLContext object that contains your certificate chain + and private key (default: None) + timeout - socket timeout (default: None) If timeout is not given or is None, + the global default socket timeout is used + + for more documentation see the docstring of the parent class IMAP4. + """ + + + def __init__(self, host='', port=IMAP4_SSL_PORT, + *, ssl_context=None, timeout=None): + if ssl_context is None: + ssl_context = ssl._create_stdlib_context() + self.ssl_context = ssl_context + IMAP4.__init__(self, host, port, timeout) + + def _create_socket(self, timeout): + sock = IMAP4._create_socket(self, timeout) + return self.ssl_context.wrap_socket(sock, + server_hostname=self.host) + + def open(self, host='', port=IMAP4_SSL_PORT, timeout=None): + """Setup connection to remote server on "host:port". + (default: localhost:standard IMAP4 SSL port). + This connection will be used by the routines: + read, readline, send, shutdown. + """ + IMAP4.open(self, host, port, timeout) + + __all__.append("IMAP4_SSL") + + +class IMAP4_stream(IMAP4): + + """IMAP4 client class over a stream + + Instantiate with: IMAP4_stream(command) + + "command" - a string that can be passed to subprocess.Popen() + + for more documentation see the docstring of the parent class IMAP4. + """ + + + def __init__(self, command): + self.command = command + IMAP4.__init__(self) + + + def open(self, host=None, port=None, timeout=None): + """Setup a stream connection. + This connection will be used by the routines: + read, readline, send, shutdown. + """ + self.host = None # For compatibility with parent class + self.port = None + self.sock = None + self._file = None + self.process = subprocess.Popen(self.command, + bufsize=DEFAULT_BUFFER_SIZE, + stdin=subprocess.PIPE, stdout=subprocess.PIPE, + shell=True, close_fds=True) + self.writefile = self.process.stdin + self.readfile = self.process.stdout + + def read(self, size): + """Read 'size' bytes from remote.""" + return self.readfile.read(size) + + + def readline(self): + """Read line from remote.""" + return self.readfile.readline() + + + def send(self, data): + """Send data to remote.""" + self.writefile.write(data) + self.writefile.flush() + + + def shutdown(self): + """Close I/O established in "open".""" + self.readfile.close() + self.writefile.close() + self.process.wait() + + + +class _Authenticator: + + """Private class to provide en/decoding + for base64-based authentication conversation. + """ + + def __init__(self, mechinst): + self.mech = mechinst # Callable object to provide/process data + + def process(self, data): + ret = self.mech(self.decode(data)) + if ret is None: + return b'*' # Abort conversation + return self.encode(ret) + + def encode(self, inp): + # + # Invoke binascii.b2a_base64 iteratively with + # short even length buffers, strip the trailing + # line feed from the result and append. "Even" + # means a number that factors to both 6 and 8, + # so when it gets to the end of the 8-bit input + # there's no partial 6-bit output. + # + oup = b'' + if isinstance(inp, str): + inp = inp.encode('utf-8') + while inp: + if len(inp) > 48: + t = inp[:48] + inp = inp[48:] + else: + t = inp + inp = b'' + e = binascii.b2a_base64(t) + if e: + oup = oup + e[:-1] + return oup + + def decode(self, inp): + if not inp: + return b'' + return binascii.a2b_base64(inp) + +Months = ' Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec'.split(' ') +Mon2num = {s.encode():n+1 for n, s in enumerate(Months[1:])} + +def Internaldate2tuple(resp): + """Parse an IMAP4 INTERNALDATE string. + + Return corresponding local time. The return value is a + time.struct_time tuple or None if the string has wrong format. + """ + + mo = InternalDate.match(resp) + if not mo: + return None + + mon = Mon2num[mo.group('mon')] + zonen = mo.group('zonen') + + day = int(mo.group('day')) + year = int(mo.group('year')) + hour = int(mo.group('hour')) + min = int(mo.group('min')) + sec = int(mo.group('sec')) + zoneh = int(mo.group('zoneh')) + zonem = int(mo.group('zonem')) + + # INTERNALDATE timezone must be subtracted to get UT + + zone = (zoneh*60 + zonem)*60 + if zonen == b'-': + zone = -zone + + tt = (year, mon, day, hour, min, sec, -1, -1, -1) + utc = calendar.timegm(tt) - zone + + return time.localtime(utc) + + + +def Int2AP(num): + + """Convert integer to A-P string representation.""" + + val = b''; AP = b'ABCDEFGHIJKLMNOP' + num = int(abs(num)) + while num: + num, mod = divmod(num, 16) + val = AP[mod:mod+1] + val + return val + + + +def ParseFlags(resp): + + """Convert IMAP4 flags response to python tuple.""" + + mo = Flags.match(resp) + if not mo: + return () + + return tuple(mo.group('flags').split()) + + +def Time2Internaldate(date_time): + + """Convert date_time to IMAP4 INTERNALDATE representation. + + Return string in form: '"DD-Mmm-YYYY HH:MM:SS +HHMM"'. The + date_time argument can be a number (int or float) representing + seconds since epoch (as returned by time.time()), a 9-tuple + representing local time, an instance of time.struct_time (as + returned by time.localtime()), an aware datetime instance or a + double-quoted string. In the last case, it is assumed to already + be in the correct format. + """ + if isinstance(date_time, (int, float)): + dt = datetime.fromtimestamp(date_time, + timezone.utc).astimezone() + elif isinstance(date_time, tuple): + try: + gmtoff = date_time.tm_gmtoff + except AttributeError: + if time.daylight: + dst = date_time[8] + if dst == -1: + dst = time.localtime(time.mktime(date_time))[8] + gmtoff = -(time.timezone, time.altzone)[dst] + else: + gmtoff = -time.timezone + delta = timedelta(seconds=gmtoff) + dt = datetime(*date_time[:6], tzinfo=timezone(delta)) + elif isinstance(date_time, datetime): + if date_time.tzinfo is None: + raise ValueError("date_time must be aware") + dt = date_time + elif isinstance(date_time, str) and (date_time[0],date_time[-1]) == ('"','"'): + return date_time # Assume in correct format + else: + raise ValueError("date_time not of a known type") + fmt = '"%d-{}-%Y %H:%M:%S %z"'.format(Months[dt.month]) + return dt.strftime(fmt) + + + +if __name__ == '__main__': + + # To test: invoke either as 'python imaplib.py [IMAP4_server_hostname]' + # or 'python imaplib.py -s "rsh IMAP4_server_hostname exec /etc/rimapd"' + # to test the IMAP4_stream class + + import getopt, getpass + + try: + optlist, args = getopt.getopt(sys.argv[1:], 'd:s:') + except getopt.error as val: + optlist, args = (), () + + stream_command = None + for opt,val in optlist: + if opt == '-d': + Debug = int(val) + elif opt == '-s': + stream_command = val + if not args: args = (stream_command,) + + if not args: args = ('',) + + host = args[0] + + USER = getpass.getuser() + PASSWD = getpass.getpass("IMAP password for %s on %s: " % (USER, host or "localhost")) + + test_mesg = 'From: %(user)s@localhost%(lf)sSubject: IMAP4 test%(lf)s%(lf)sdata...%(lf)s' % {'user':USER, 'lf':'\n'} + test_seq1 = ( + ('login', (USER, PASSWD)), + ('create', ('/tmp/xxx 1',)), + ('rename', ('/tmp/xxx 1', '/tmp/yyy')), + ('CREATE', ('/tmp/yyz 2',)), + ('append', ('/tmp/yyz 2', None, None, test_mesg)), + ('list', ('/tmp', 'yy*')), + ('select', ('/tmp/yyz 2',)), + ('search', (None, 'SUBJECT', 'test')), + ('fetch', ('1', '(FLAGS INTERNALDATE RFC822)')), + ('store', ('1', 'FLAGS', r'(\Deleted)')), + ('namespace', ()), + ('expunge', ()), + ('recent', ()), + ('close', ()), + ) + + test_seq2 = ( + ('select', ()), + ('response',('UIDVALIDITY',)), + ('uid', ('SEARCH', 'ALL')), + ('response', ('EXISTS',)), + ('append', (None, None, None, test_mesg)), + ('recent', ()), + ('logout', ()), + ) + + def run(cmd, args): + M._mesg('%s %s' % (cmd, args)) + typ, dat = getattr(M, cmd)(*args) + M._mesg('%s => %s %s' % (cmd, typ, dat)) + if typ == 'NO': raise dat[0] + return dat + + try: + if stream_command: + M = IMAP4_stream(stream_command) + else: + M = IMAP4(host) + if M.state == 'AUTH': + test_seq1 = test_seq1[1:] # Login not needed + M._mesg('PROTOCOL_VERSION = %s' % M.PROTOCOL_VERSION) + M._mesg('CAPABILITIES = %r' % (M.capabilities,)) + + for cmd,args in test_seq1: + run(cmd, args) + + for ml in run('list', ('/tmp/', 'yy%')): + mo = re.match(r'.*"([^"]+)"$', ml) + if mo: path = mo.group(1) + else: path = ml.split()[-1] + run('delete', (path,)) + + for cmd,args in test_seq2: + dat = run(cmd, args) + + if (cmd,args) != ('uid', ('SEARCH', 'ALL')): + continue + + uid = dat[-1].split() + if not uid: continue + run('uid', ('FETCH', '%s' % uid[-1], + '(FLAGS INTERNALDATE RFC822.SIZE RFC822.HEADER RFC822.TEXT)')) + + print('\nAll tests OK.') + + except: + print('\nTests failed.') + + if not Debug: + print(''' +If you would like to see debugging output, +try: %s -d5 +''' % sys.argv[0]) + + raise diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py index ca73eb2fcf7..89ce8c09c94 100644 --- a/Lib/importlib/_bootstrap_external.py +++ b/Lib/importlib/_bootstrap_external.py @@ -489,7 +489,7 @@ def _write_atomic(path, data, mode=0o666): # Whenever MAGIC_NUMBER is changed, the ranges in the magic_values array # in PC/launcher.c must also be updated. -MAGIC_NUMBER = (2997).to_bytes(2, 'little') + b'\r\n' +MAGIC_NUMBER = (2996).to_bytes(2, 'little') + b'\r\n' _RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little') # For import.c diff --git a/Lib/inspect.py b/Lib/inspect.py index 5a814f97b5b..385fbc686b6 100644 --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -6,9 +6,9 @@ Here are some of the useful functions provided by this module: - ismodule(), isclass(), ismethod(), isfunction(), isgeneratorfunction(), - isgenerator(), istraceback(), isframe(), iscode(), isbuiltin(), - isroutine() - check object types + ismodule(), isclass(), ismethod(), ispackage(), isfunction(), + isgeneratorfunction(), isgenerator(), istraceback(), isframe(), + iscode(), isbuiltin(), isroutine() - check object types getmembers() - get members of an object that satisfy a given condition getfile(), getsourcefile(), getsource() - find an object's source code @@ -24,8 +24,6 @@ stack(), trace() - get info about frames on the stack or in a traceback signature() - get a Signature object for the callable - - get_annotations() - safely compute an object's annotations """ # This module is in the public domain. No warranties. @@ -58,6 +56,8 @@ "CO_OPTIMIZED", "CO_VARARGS", "CO_VARKEYWORDS", + "CO_HAS_DOCSTRING", + "CO_METHOD", "ClassFoundException", "ClosureVars", "EndOfBlock", @@ -130,6 +130,7 @@ "ismethoddescriptor", "ismethodwrapper", "ismodule", + "ispackage", "isroutine", "istraceback", "markcoroutinefunction", @@ -142,6 +143,8 @@ import abc +from annotationlib import Format, ForwardRef +from annotationlib import get_annotations # re-exported import ast import dis import collections.abc @@ -173,127 +176,6 @@ TPFLAGS_IS_ABSTRACT = 1 << 20 -def get_annotations(obj, *, globals=None, locals=None, eval_str=False): - """Compute the annotations dict for an object. - - obj may be a callable, class, or module. - Passing in an object of any other type raises TypeError. - - Returns a dict. get_annotations() returns a new dict every time - it's called; calling it twice on the same object will return two - different but equivalent dicts. - - This function handles several details for you: - - * If eval_str is true, values of type str will - be un-stringized using eval(). This is intended - for use with stringized annotations - ("from __future__ import annotations"). - * If obj doesn't have an annotations dict, returns an - empty dict. (Functions and methods always have an - annotations dict; classes, modules, and other types of - callables may not.) - * Ignores inherited annotations on classes. If a class - doesn't have its own annotations dict, returns an empty dict. - * All accesses to object members and dict values are done - using getattr() and dict.get() for safety. - * Always, always, always returns a freshly-created dict. - - eval_str controls whether or not values of type str are replaced - with the result of calling eval() on those values: - - * If eval_str is true, eval() is called on values of type str. - * If eval_str is false (the default), values of type str are unchanged. - - globals and locals are passed in to eval(); see the documentation - for eval() for more information. If either globals or locals is - None, this function may replace that value with a context-specific - default, contingent on type(obj): - - * If obj is a module, globals defaults to obj.__dict__. - * If obj is a class, globals defaults to - sys.modules[obj.__module__].__dict__ and locals - defaults to the obj class namespace. - * If obj is a callable, globals defaults to obj.__globals__, - although if obj is a wrapped function (using - functools.update_wrapper()) it is first unwrapped. - """ - if isinstance(obj, type): - # class - obj_dict = getattr(obj, '__dict__', None) - if obj_dict and hasattr(obj_dict, 'get'): - ann = obj_dict.get('__annotations__', None) - if isinstance(ann, types.GetSetDescriptorType): - ann = None - else: - ann = None - - obj_globals = None - module_name = getattr(obj, '__module__', None) - if module_name: - module = sys.modules.get(module_name, None) - if module: - obj_globals = getattr(module, '__dict__', None) - obj_locals = dict(vars(obj)) - unwrap = obj - elif isinstance(obj, types.ModuleType): - # module - ann = getattr(obj, '__annotations__', None) - obj_globals = getattr(obj, '__dict__') - obj_locals = None - unwrap = None - elif callable(obj): - # this includes types.Function, types.BuiltinFunctionType, - # types.BuiltinMethodType, functools.partial, functools.singledispatch, - # "class funclike" from Lib/test/test_inspect... on and on it goes. - ann = getattr(obj, '__annotations__', None) - obj_globals = getattr(obj, '__globals__', None) - obj_locals = None - unwrap = obj - else: - raise TypeError(f"{obj!r} is not a module, class, or callable.") - - if ann is None: - return {} - - if not isinstance(ann, dict): - raise ValueError(f"{obj!r}.__annotations__ is neither a dict nor None") - - if not ann: - return {} - - if not eval_str: - return dict(ann) - - if unwrap is not None: - while True: - if hasattr(unwrap, '__wrapped__'): - unwrap = unwrap.__wrapped__ - continue - if isinstance(unwrap, functools.partial): - unwrap = unwrap.func - continue - break - if hasattr(unwrap, "__globals__"): - obj_globals = unwrap.__globals__ - - if globals is None: - globals = obj_globals - if locals is None: - locals = obj_locals or {} - - # "Inject" type parameters into the local namespace - # (unless they are shadowed by assignments *in* the local namespace), - # as a way of emulating annotation scopes when calling `eval()` - if type_params := getattr(obj, "__type_params__", ()): - locals = {param.__name__: param for param in type_params} | locals - - return_value = {key: - value if not isinstance(value, str) else eval(value, globals, locals) - for key, value in ann.items() } - return return_value - - # ----------------------------------------------------------- type-checking def ismodule(object): """Return true if the object is a module.""" @@ -307,6 +189,10 @@ def ismethod(object): """Return true if the object is an instance method.""" return isinstance(object, types.MethodType) +def ispackage(object): + """Return true if the object is a package.""" + return ismodule(object) and hasattr(object, "__path__") + def ismethoddescriptor(object): """Return true if the object is a method descriptor. @@ -325,11 +211,6 @@ def ismethoddescriptor(object): if isclass(object) or ismethod(object) or isfunction(object): # mutual exclusion return False - if isinstance(object, functools.partial): - # Lie for children. The addition of partial.__get__ - # doesn't currently change the partial objects behaviour, - # not counting a warning about future changes. - return False tp = type(object) return (hasattr(tp, "__get__") and not hasattr(tp, "__set__") @@ -389,11 +270,16 @@ def isfunction(object): Function objects provide these attributes: __doc__ documentation string __name__ name with which this function was defined + __qualname__ qualified name of this function + __module__ name of the module the function was defined in or None __code__ code object containing compiled function bytecode __defaults__ tuple of any default values for arguments __globals__ global namespace in which this function was defined __annotations__ dict of parameter annotations - __kwdefaults__ dict of keyword only parameters with defaults""" + __kwdefaults__ dict of keyword only parameters with defaults + __dict__ namespace which is supporting arbitrary function attributes + __closure__ a tuple of cells or None + __type_params__ tuple of type parameters""" return isinstance(object, types.FunctionType) def _has_code_flag(f, flag): @@ -458,17 +344,18 @@ def isgenerator(object): """Return true if the object is a generator. Generator objects provide these attributes: - __iter__ defined to support iteration over container - close raises a new GeneratorExit exception inside the - generator to terminate the iteration gi_code code object gi_frame frame object or possibly None once the generator has been exhausted gi_running set to 1 when generator is executing, 0 otherwise - next return the next item from the container - send resumes the generator and "sends" a value that becomes + gi_yieldfrom object being iterated by yield from or None + + __iter__() defined to support iteration over container + close() raises a new GeneratorExit exception inside the + generator to terminate the iteration + send() resumes the generator and "sends" a value that becomes the result of the current yield-expression - throw used to raise an exception inside the generator""" + throw() used to raise an exception inside the generator""" return isinstance(object, types.GeneratorType) def iscoroutine(object): @@ -503,7 +390,11 @@ def isframe(object): f_lasti index of last attempted instruction in bytecode f_lineno current line number in Python source code f_locals local namespace seen by this frame - f_trace tracing function for this frame, or None""" + f_trace tracing function for this frame, or None + f_trace_lines is a tracing event triggered for each source line? + f_trace_opcodes are per-opcode events being requested? + + clear() used to clear all references to local variables""" return isinstance(object, types.FrameType) def iscode(object): @@ -520,6 +411,7 @@ def iscode(object): co_flags bitmap: 1=optimized | 2=newlocals | 4=*arg | 8=**arg | 16=nested | 32=generator | 64=nofree | 128=coroutine | 256=iterable_coroutine | 512=async_generator + | 0x4000000=has_docstring co_freevars tuple of names of free variables co_posonlyargcount number of positional only arguments co_kwonlyargcount number of keyword only arguments (not including ** arg) @@ -528,7 +420,12 @@ def iscode(object): co_names tuple of names other than arguments and function locals co_nlocals number of local variables co_stacksize virtual machine stack space required - co_varnames tuple of names of arguments and local variables""" + co_varnames tuple of names of arguments and local variables + co_qualname fully qualified function name + + co_lines() returns an iterator that yields successive bytecode ranges + co_positions() returns an iterator of source code positions for each bytecode instruction + replace() returns a copy of the code object with a new values""" return isinstance(object, types.CodeType) def isbuiltin(object): @@ -550,7 +447,8 @@ def isroutine(object): or isfunction(object) or ismethod(object) or ismethoddescriptor(object) - or ismethodwrapper(object)) + or ismethodwrapper(object) + or isinstance(object, functools._singledispatchmethod_get)) def isabstract(object): """Return true if the object is an abstract base class (ABC).""" @@ -961,8 +859,7 @@ def getsourcefile(object): Return None if no way can be identified to get the source. """ filename = getfile(object) - all_bytecode_suffixes = importlib.machinery.DEBUG_BYTECODE_SUFFIXES[:] - all_bytecode_suffixes += importlib.machinery.OPTIMIZED_BYTECODE_SUFFIXES[:] + all_bytecode_suffixes = importlib.machinery.BYTECODE_SUFFIXES[:] if any(filename.endswith(s) for s in all_bytecode_suffixes): filename = (os.path.splitext(filename)[0] + importlib.machinery.SOURCE_SUFFIXES[0]) @@ -1438,7 +1335,9 @@ def getargvalues(frame): args, varargs, varkw = getargs(frame.f_code) return ArgInfo(args, varargs, varkw, frame.f_locals) -def formatannotation(annotation, base_module=None): +def formatannotation(annotation, base_module=None, *, quote_annotation_strings=True): + if not quote_annotation_strings and isinstance(annotation, str): + return annotation if getattr(annotation, '__module__', None) == 'typing': def repl(match): text = match.group() @@ -1450,6 +1349,8 @@ def repl(match): if annotation.__module__ in ('builtins', base_module): return annotation.__qualname__ return annotation.__module__+'.'+annotation.__qualname__ + if isinstance(annotation, ForwardRef): + return annotation.__forward_arg__ return repr(annotation) def formatannotationrelativeto(object): @@ -2067,7 +1968,12 @@ def _signature_get_partial(wrapped_sig, partial, extra_args=()): if param.kind is _POSITIONAL_ONLY: # If positional-only parameter is bound by partial, # it effectively disappears from the signature - new_params.pop(param_name) + # However, if it is a Placeholder it is not removed + # And also looses default value + if arg_value is functools.Placeholder: + new_params[param_name] = param.replace(default=_empty) + else: + new_params.pop(param_name) continue if param.kind is _POSITIONAL_OR_KEYWORD: @@ -2089,7 +1995,17 @@ def _signature_get_partial(wrapped_sig, partial, extra_args=()): new_params[param_name] = param.replace(default=arg_value) else: # was passed as a positional argument - new_params.pop(param.name) + # Do not pop if it is a Placeholder + # also change kind to positional only + # and remove default + if arg_value is functools.Placeholder: + new_param = param.replace( + kind=_POSITIONAL_ONLY, + default=_empty + ) + new_params[param_name] = new_param + else: + new_params.pop(param_name) continue if param.kind is _KEYWORD_ONLY: @@ -2167,13 +2083,11 @@ def _signature_is_functionlike(obj): code = getattr(obj, '__code__', None) defaults = getattr(obj, '__defaults__', _void) # Important to use _void ... kwdefaults = getattr(obj, '__kwdefaults__', _void) # ... and not None here - annotations = getattr(obj, '__annotations__', None) return (isinstance(code, types.CodeType) and isinstance(name, str) and (defaults is None or isinstance(defaults, tuple)) and - (kwdefaults is None or isinstance(kwdefaults, dict)) and - (isinstance(annotations, (dict)) or annotations is None) ) + (kwdefaults is None or isinstance(kwdefaults, dict))) def _signature_strip_non_python_syntax(signature): @@ -2390,7 +2304,8 @@ def _signature_from_builtin(cls, func, skip_bound_arg=True): def _signature_from_function(cls, func, skip_bound_arg=True, - globals=None, locals=None, eval_str=False): + globals=None, locals=None, eval_str=False, + *, annotation_format=Format.VALUE): """Private helper: constructs Signature for the given python function.""" is_duck_function = False @@ -2416,7 +2331,8 @@ def _signature_from_function(cls, func, skip_bound_arg=True, positional = arg_names[:pos_count] keyword_only_count = func_code.co_kwonlyargcount keyword_only = arg_names[pos_count:pos_count + keyword_only_count] - annotations = get_annotations(func, globals=globals, locals=locals, eval_str=eval_str) + annotations = get_annotations(func, globals=globals, locals=locals, eval_str=eval_str, + format=annotation_format) defaults = func.__defaults__ kwdefaults = func.__kwdefaults__ @@ -2499,7 +2415,8 @@ def _signature_from_callable(obj, *, globals=None, locals=None, eval_str=False, - sigcls): + sigcls, + annotation_format=Format.VALUE): """Private helper function to get signature for arbitrary callable objects. @@ -2511,7 +2428,8 @@ def _signature_from_callable(obj, *, globals=globals, locals=locals, sigcls=sigcls, - eval_str=eval_str) + eval_str=eval_str, + annotation_format=annotation_format) if not callable(obj): raise TypeError('{!r} is not a callable object'.format(obj)) @@ -2544,18 +2462,10 @@ def _signature_from_callable(obj, *, pass else: if sig is not None: - # since __text_signature__ is not writable on classes, __signature__ - # may contain text (or be a callable that returns text); - # if so, convert it - o_sig = sig - if not isinstance(sig, (Signature, str)) and callable(sig): - sig = sig() - if isinstance(sig, str): - sig = _signature_fromstr(sigcls, obj, sig) if not isinstance(sig, Signature): raise TypeError( 'unexpected object {!r} in __signature__ ' - 'attribute'.format(o_sig)) + 'attribute'.format(sig)) return sig try: @@ -2583,6 +2493,11 @@ def _signature_from_callable(obj, *, sig_params = tuple(sig.parameters.values()) assert (not sig_params or first_wrapped_param is not sig_params[0]) + # If there were placeholders set, + # first param is transformed to positional only + if partialmethod.args.count(functools.Placeholder): + first_wrapped_param = first_wrapped_param.replace( + kind=Parameter.POSITIONAL_ONLY) new_params = (first_wrapped_param,) + sig_params return sig.replace(parameters=new_params) @@ -2595,7 +2510,8 @@ def _signature_from_callable(obj, *, # of a Python function (Cython functions, for instance), then: return _signature_from_function(sigcls, obj, skip_bound_arg=skip_bound_arg, - globals=globals, locals=locals, eval_str=eval_str) + globals=globals, locals=locals, eval_str=eval_str, + annotation_format=annotation_format) if _signature_is_builtin(obj): return _signature_from_builtin(sigcls, obj, @@ -2848,13 +2764,17 @@ def replace(self, *, name=_void, kind=_void, return type(self)(name, kind, default=default, annotation=annotation) def __str__(self): + return self._format() + + def _format(self, *, quote_annotation_strings=True): kind = self.kind formatted = self._name # Add annotation and default value if self._annotation is not _empty: - formatted = '{}: {}'.format(formatted, - formatannotation(self._annotation)) + annotation = formatannotation(self._annotation, + quote_annotation_strings=quote_annotation_strings) + formatted = '{}: {}'.format(formatted, annotation) if self._default is not _empty: if self._annotation is not _empty: @@ -3061,11 +2981,19 @@ def __init__(self, parameters=None, *, return_annotation=_empty, params = OrderedDict() top_kind = _POSITIONAL_ONLY seen_default = False + seen_var_parameters = set() for param in parameters: kind = param.kind name = param.name + if kind in (_VAR_POSITIONAL, _VAR_KEYWORD): + if kind in seen_var_parameters: + msg = f'more than one {kind.description} parameter' + raise ValueError(msg) + + seen_var_parameters.add(kind) + if kind < top_kind: msg = ( 'wrong parameter order: {} parameter before {} ' @@ -3102,11 +3030,13 @@ def __init__(self, parameters=None, *, return_annotation=_empty, @classmethod def from_callable(cls, obj, *, - follow_wrapped=True, globals=None, locals=None, eval_str=False): + follow_wrapped=True, globals=None, locals=None, eval_str=False, + annotation_format=Format.VALUE): """Constructs Signature for the given callable object.""" return _signature_from_callable(obj, sigcls=cls, follow_wrapper_chains=follow_wrapped, - globals=globals, locals=locals, eval_str=eval_str) + globals=globals, locals=locals, eval_str=eval_str, + annotation_format=annotation_format) @property def parameters(self): @@ -3324,19 +3254,24 @@ def __repr__(self): def __str__(self): return self.format() - def format(self, *, max_width=None): + def format(self, *, max_width=None, quote_annotation_strings=True): """Create a string representation of the Signature object. If *max_width* integer is passed, signature will try to fit into the *max_width*. If signature is longer than *max_width*, all parameters will be on separate lines. + + If *quote_annotation_strings* is False, annotations + in the signature are displayed without opening and closing quotation + marks. This is useful when the signature was created with the + STRING format or when ``from __future__ import annotations`` was used. """ result = [] render_pos_only_separator = False render_kw_only_separator = True for param in self.parameters.values(): - formatted = str(param) + formatted = param._format(quote_annotation_strings=quote_annotation_strings) kind = param.kind @@ -3373,16 +3308,19 @@ def format(self, *, max_width=None): rendered = '(\n {}\n)'.format(',\n '.join(result)) if self.return_annotation is not _empty: - anno = formatannotation(self.return_annotation) + anno = formatannotation(self.return_annotation, + quote_annotation_strings=quote_annotation_strings) rendered += ' -> {}'.format(anno) return rendered -def signature(obj, *, follow_wrapped=True, globals=None, locals=None, eval_str=False): +def signature(obj, *, follow_wrapped=True, globals=None, locals=None, eval_str=False, + annotation_format=Format.VALUE): """Get a signature object for the passed callable.""" return Signature.from_callable(obj, follow_wrapped=follow_wrapped, - globals=globals, locals=locals, eval_str=eval_str) + globals=globals, locals=locals, eval_str=eval_str, + annotation_format=annotation_format) class BufferFlags(enum.IntFlag): @@ -3412,7 +3350,7 @@ def _main(): import argparse import importlib - parser = argparse.ArgumentParser() + parser = argparse.ArgumentParser(color=True) parser.add_argument( 'object', help="The object to be analysed. " diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py index 67e45450fc1..ca732e4f2e8 100644 --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -239,7 +239,7 @@ def summarize_address_range(first, last): else: raise ValueError('unknown IP version') - ip_bits = first._max_prefixlen + ip_bits = first.max_prefixlen first_int = first._ip last_int = last._ip while first_int <= last_int: @@ -326,12 +326,12 @@ def collapse_addresses(addresses): # split IP addresses and networks for ip in addresses: if isinstance(ip, _BaseAddress): - if ips and ips[-1]._version != ip._version: + if ips and ips[-1].version != ip.version: raise TypeError("%s and %s are not of the same version" % ( ip, ips[-1])) ips.append(ip) - elif ip._prefixlen == ip._max_prefixlen: - if ips and ips[-1]._version != ip._version: + elif ip._prefixlen == ip.max_prefixlen: + if ips and ips[-1].version != ip.version: raise TypeError("%s and %s are not of the same version" % ( ip, ips[-1])) try: @@ -339,7 +339,7 @@ def collapse_addresses(addresses): except AttributeError: ips.append(ip.network_address) else: - if nets and nets[-1]._version != ip._version: + if nets and nets[-1].version != ip.version: raise TypeError("%s and %s are not of the same version" % ( ip, nets[-1])) nets.append(ip) @@ -407,26 +407,21 @@ def reverse_pointer(self): """ return self._reverse_pointer() - @property - def version(self): - msg = '%200s has no version specified' % (type(self),) - raise NotImplementedError(msg) - def _check_int_address(self, address): if address < 0: msg = "%d (< 0) is not permitted as an IPv%d address" - raise AddressValueError(msg % (address, self._version)) + raise AddressValueError(msg % (address, self.version)) if address > self._ALL_ONES: msg = "%d (>= 2**%d) is not permitted as an IPv%d address" - raise AddressValueError(msg % (address, self._max_prefixlen, - self._version)) + raise AddressValueError(msg % (address, self.max_prefixlen, + self.version)) def _check_packed_address(self, address, expected_len): address_len = len(address) if address_len != expected_len: msg = "%r (len %d != %d) is not permitted as an IPv%d address" raise AddressValueError(msg % (address, address_len, - expected_len, self._version)) + expected_len, self.version)) @classmethod def _ip_int_from_prefix(cls, prefixlen): @@ -455,12 +450,12 @@ def _prefix_from_ip_int(cls, ip_int): ValueError: If the input intermingles zeroes & ones """ trailing_zeroes = _count_righthand_zero_bits(ip_int, - cls._max_prefixlen) - prefixlen = cls._max_prefixlen - trailing_zeroes + cls.max_prefixlen) + prefixlen = cls.max_prefixlen - trailing_zeroes leading_ones = ip_int >> trailing_zeroes all_ones = (1 << prefixlen) - 1 if leading_ones != all_ones: - byteslen = cls._max_prefixlen // 8 + byteslen = cls.max_prefixlen // 8 details = ip_int.to_bytes(byteslen, 'big') msg = 'Netmask pattern %r mixes zeroes & ones' raise ValueError(msg % details) @@ -492,7 +487,7 @@ def _prefix_from_prefix_string(cls, prefixlen_str): prefixlen = int(prefixlen_str) except ValueError: cls._report_invalid_netmask(prefixlen_str) - if not (0 <= prefixlen <= cls._max_prefixlen): + if not (0 <= prefixlen <= cls.max_prefixlen): cls._report_invalid_netmask(prefixlen_str) return prefixlen @@ -542,7 +537,7 @@ def _split_addr_prefix(cls, address): """ # a packed address or integer if isinstance(address, (bytes, int)): - return address, cls._max_prefixlen + return address, cls.max_prefixlen if not isinstance(address, tuple): # Assume input argument to be string or any object representation @@ -552,7 +547,7 @@ def _split_addr_prefix(cls, address): # Constructing from a tuple (addr, [mask]) if len(address) > 1: return address - return address[0], cls._max_prefixlen + return address[0], cls.max_prefixlen def __reduce__(self): return self.__class__, (str(self),) @@ -577,14 +572,14 @@ def __int__(self): def __eq__(self, other): try: return (self._ip == other._ip - and self._version == other._version) + and self.version == other.version) except AttributeError: return NotImplemented def __lt__(self, other): if not isinstance(other, _BaseAddress): return NotImplemented - if self._version != other._version: + if self.version != other.version: raise TypeError('%s and %s are not of the same version' % ( self, other)) if self._ip != other._ip: @@ -613,7 +608,7 @@ def __hash__(self): return hash(hex(int(self._ip))) def _get_address_key(self): - return (self._version, self) + return (self.version, self) def __reduce__(self): return self.__class__, (self._ip,) @@ -649,15 +644,15 @@ def __format__(self, fmt): # Set some defaults if fmt_base == 'n': - if self._version == 4: + if self.version == 4: fmt_base = 'b' # Binary is default for ipv4 else: fmt_base = 'x' # Hex is default for ipv6 if fmt_base == 'b': - padlen = self._max_prefixlen + padlen = self.max_prefixlen else: - padlen = self._max_prefixlen // 4 + padlen = self.max_prefixlen // 4 if grouping: padlen += padlen // 4 - 1 @@ -716,7 +711,7 @@ def __getitem__(self, n): def __lt__(self, other): if not isinstance(other, _BaseNetwork): return NotImplemented - if self._version != other._version: + if self.version != other.version: raise TypeError('%s and %s are not of the same version' % ( self, other)) if self.network_address != other.network_address: @@ -727,7 +722,7 @@ def __lt__(self, other): def __eq__(self, other): try: - return (self._version == other._version and + return (self.version == other.version and self.network_address == other.network_address and int(self.netmask) == int(other.netmask)) except AttributeError: @@ -738,7 +733,7 @@ def __hash__(self): def __contains__(self, other): # always false if one is v4 and the other is v6. - if self._version != other._version: + if self.version != other.version: return False # dealing with another network. if isinstance(other, _BaseNetwork): @@ -829,7 +824,7 @@ def address_exclude(self, other): ValueError: If other is not completely contained by self. """ - if not self._version == other._version: + if not self.version == other.version: raise TypeError("%s and %s are not of the same version" % ( self, other)) @@ -901,10 +896,10 @@ def compare_networks(self, other): """ # does this need to raise a ValueError? - if self._version != other._version: + if self.version != other.version: raise TypeError('%s and %s are not of the same type' % ( self, other)) - # self._version == other._version below here: + # self.version == other.version below here: if self.network_address < other.network_address: return -1 if self.network_address > other.network_address: @@ -924,7 +919,7 @@ def _get_networks_key(self): and list.sort(). """ - return (self._version, self.network_address, self.netmask) + return (self.version, self.network_address, self.netmask) def subnets(self, prefixlen_diff=1, new_prefix=None): """The subnets which join to make the current subnet. @@ -952,7 +947,7 @@ def subnets(self, prefixlen_diff=1, new_prefix=None): number means a larger network) """ - if self._prefixlen == self._max_prefixlen: + if self._prefixlen == self.max_prefixlen: yield self return @@ -967,7 +962,7 @@ def subnets(self, prefixlen_diff=1, new_prefix=None): raise ValueError('prefix length diff must be > 0') new_prefixlen = self._prefixlen + prefixlen_diff - if new_prefixlen > self._max_prefixlen: + if new_prefixlen > self.max_prefixlen: raise ValueError( 'prefix length diff %d is invalid for netblock %s' % ( new_prefixlen, self)) @@ -1036,7 +1031,7 @@ def is_multicast(self): def _is_subnet_of(a, b): try: # Always false if one is v4 and the other is v6. - if a._version != b._version: + if a.version != b.version: raise TypeError(f"{a} and {b} are not of the same version") return (b.network_address <= a.network_address and b.broadcast_address >= a.broadcast_address) @@ -1146,11 +1141,11 @@ class _BaseV4: """ __slots__ = () - _version = 4 + version = 4 # Equivalent to 255.255.255.255 or 32 bits of 1's. _ALL_ONES = (2**IPV4LENGTH) - 1 - _max_prefixlen = IPV4LENGTH + max_prefixlen = IPV4LENGTH # There are only a handful of valid v4 netmasks, so we cache them all # when constructed (see _make_netmask()). _netmask_cache = {} @@ -1170,7 +1165,7 @@ def _make_netmask(cls, arg): if arg not in cls._netmask_cache: if isinstance(arg, int): prefixlen = arg - if not (0 <= prefixlen <= cls._max_prefixlen): + if not (0 <= prefixlen <= cls.max_prefixlen): cls._report_invalid_netmask(prefixlen) else: try: @@ -1268,15 +1263,6 @@ def _reverse_pointer(self): reverse_octets = str(self).split('.')[::-1] return '.'.join(reverse_octets) + '.in-addr.arpa' - @property - def max_prefixlen(self): - return self._max_prefixlen - - @property - def version(self): - return self._version - - class IPv4Address(_BaseV4, _BaseAddress): """Represent and manipulate single IPv4 Addresses.""" @@ -1556,10 +1542,10 @@ def __init__(self, address, strict=True): self.network_address = IPv4Address(packed & int(self.netmask)) - if self._prefixlen == (self._max_prefixlen - 1): + if self._prefixlen == (self.max_prefixlen - 1): self.hosts = self.__iter__ - elif self._prefixlen == (self._max_prefixlen): - self.hosts = lambda: [IPv4Address(addr)] + elif self._prefixlen == (self.max_prefixlen): + self.hosts = lambda: iter((IPv4Address(addr),)) @property @functools.lru_cache() @@ -1628,11 +1614,11 @@ class _BaseV6: """ __slots__ = () - _version = 6 + version = 6 _ALL_ONES = (2**IPV6LENGTH) - 1 _HEXTET_COUNT = 8 _HEX_DIGITS = frozenset('0123456789ABCDEFabcdef') - _max_prefixlen = IPV6LENGTH + max_prefixlen = IPV6LENGTH # There are only a bunch of valid v6 netmasks, so we cache them all # when constructed (see _make_netmask()). @@ -1650,7 +1636,7 @@ def _make_netmask(cls, arg): if arg not in cls._netmask_cache: if isinstance(arg, int): prefixlen = arg - if not (0 <= prefixlen <= cls._max_prefixlen): + if not (0 <= prefixlen <= cls.max_prefixlen): cls._report_invalid_netmask(prefixlen) else: prefixlen = cls._prefix_from_prefix_string(arg) @@ -1921,15 +1907,6 @@ def _split_scope_id(ip_str): raise AddressValueError('Invalid IPv6 address: "%r"' % ip_str) return addr, scope_id - @property - def max_prefixlen(self): - return self._max_prefixlen - - @property - def version(self): - return self._version - - class IPv6Address(_BaseV6, _BaseAddress): """Represent and manipulate single IPv6 Addresses.""" @@ -2356,10 +2333,10 @@ def __init__(self, address, strict=True): self.network_address = IPv6Address(packed & int(self.netmask)) - if self._prefixlen == (self._max_prefixlen - 1): + if self._prefixlen == (self.max_prefixlen - 1): self.hosts = self.__iter__ - elif self._prefixlen == self._max_prefixlen: - self.hosts = lambda: [IPv6Address(addr)] + elif self._prefixlen == self.max_prefixlen: + self.hosts = lambda: iter((IPv6Address(addr),)) def hosts(self): """Generate Iterator over usable hosts in a network. diff --git a/Lib/json/__init__.py b/Lib/json/__init__.py index ed2c74771ea..9eaa4f3fbc1 100644 --- a/Lib/json/__init__.py +++ b/Lib/json/__init__.py @@ -86,13 +86,13 @@ '[2.0, 1.0]' -Using json.tool from the shell to validate and pretty-print:: +Using json from the shell to validate and pretty-print:: - $ echo '{"json":"obj"}' | python -m json.tool + $ echo '{"json":"obj"}' | python -m json { "json": "obj" } - $ echo '{ 1.2:3.4}' | python -m json.tool + $ echo '{ 1.2:3.4}' | python -m json Expecting property name enclosed in double quotes: line 1 column 3 (char 2) """ __version__ = '2.0.9' @@ -128,8 +128,9 @@ def dump(obj, fp, *, skipkeys=False, ensure_ascii=True, check_circular=True, instead of raising a ``TypeError``. If ``ensure_ascii`` is false, then the strings written to ``fp`` can - contain non-ASCII characters if they appear in strings contained in - ``obj``. Otherwise, all such characters are escaped in JSON strings. + contain non-ASCII and non-printable characters if they appear in strings + contained in ``obj``. Otherwise, all such characters are escaped in JSON + strings. If ``check_circular`` is false, then the circular reference check for container types will be skipped and a circular reference will @@ -145,10 +146,11 @@ def dump(obj, fp, *, skipkeys=False, ensure_ascii=True, check_circular=True, level of 0 will only insert newlines. ``None`` is the most compact representation. - If specified, ``separators`` should be an ``(item_separator, key_separator)`` - tuple. The default is ``(', ', ': ')`` if *indent* is ``None`` and - ``(',', ': ')`` otherwise. To get the most compact JSON representation, - you should specify ``(',', ':')`` to eliminate whitespace. + If specified, ``separators`` should be an ``(item_separator, + key_separator)`` tuple. The default is ``(', ', ': ')`` if *indent* is + ``None`` and ``(',', ': ')`` otherwise. To get the most compact JSON + representation, you should specify ``(',', ':')`` to eliminate + whitespace. ``default(obj)`` is a function that should return a serializable version of obj or raise TypeError. The default simply raises TypeError. @@ -189,9 +191,10 @@ def dumps(obj, *, skipkeys=False, ensure_ascii=True, check_circular=True, (``str``, ``int``, ``float``, ``bool``, ``None``) will be skipped instead of raising a ``TypeError``. - If ``ensure_ascii`` is false, then the return value can contain non-ASCII - characters if they appear in strings contained in ``obj``. Otherwise, all - such characters are escaped in JSON strings. + If ``ensure_ascii`` is false, then the return value can contain + non-ASCII and non-printable characters if they appear in strings + contained in ``obj``. Otherwise, all such characters are escaped in + JSON strings. If ``check_circular`` is false, then the circular reference check for container types will be skipped and a circular reference will @@ -207,10 +210,11 @@ def dumps(obj, *, skipkeys=False, ensure_ascii=True, check_circular=True, level of 0 will only insert newlines. ``None`` is the most compact representation. - If specified, ``separators`` should be an ``(item_separator, key_separator)`` - tuple. The default is ``(', ', ': ')`` if *indent* is ``None`` and - ``(',', ': ')`` otherwise. To get the most compact JSON representation, - you should specify ``(',', ':')`` to eliminate whitespace. + If specified, ``separators`` should be an ``(item_separator, + key_separator)`` tuple. The default is ``(', ', ': ')`` if *indent* is + ``None`` and ``(',', ': ')`` otherwise. To get the most compact JSON + representation, you should specify ``(',', ':')`` to eliminate + whitespace. ``default(obj)`` is a function that should return a serializable version of obj or raise TypeError. The default simply raises TypeError. @@ -281,11 +285,12 @@ def load(fp, *, cls=None, object_hook=None, parse_float=None, ``object_hook`` will be used instead of the ``dict``. This feature can be used to implement custom decoders (e.g. JSON-RPC class hinting). - ``object_pairs_hook`` is an optional function that will be called with the - result of any object literal decoded with an ordered list of pairs. The - return value of ``object_pairs_hook`` will be used instead of the ``dict``. - This feature can be used to implement custom decoders. If ``object_hook`` - is also defined, the ``object_pairs_hook`` takes priority. + ``object_pairs_hook`` is an optional function that will be called with + the result of any object literal decoded with an ordered list of pairs. + The return value of ``object_pairs_hook`` will be used instead of the + ``dict``. This feature can be used to implement custom decoders. If + ``object_hook`` is also defined, the ``object_pairs_hook`` takes + priority. To use a custom ``JSONDecoder`` subclass, specify it with the ``cls`` kwarg; otherwise ``JSONDecoder`` is used. @@ -306,11 +311,12 @@ def loads(s, *, cls=None, object_hook=None, parse_float=None, ``object_hook`` will be used instead of the ``dict``. This feature can be used to implement custom decoders (e.g. JSON-RPC class hinting). - ``object_pairs_hook`` is an optional function that will be called with the - result of any object literal decoded with an ordered list of pairs. The - return value of ``object_pairs_hook`` will be used instead of the ``dict``. - This feature can be used to implement custom decoders. If ``object_hook`` - is also defined, the ``object_pairs_hook`` takes priority. + ``object_pairs_hook`` is an optional function that will be called with + the result of any object literal decoded with an ordered list of pairs. + The return value of ``object_pairs_hook`` will be used instead of the + ``dict``. This feature can be used to implement custom decoders. If + ``object_hook`` is also defined, the ``object_pairs_hook`` takes + priority. ``parse_float``, if specified, will be called with the string of every JSON float to be decoded. By default this is equivalent to diff --git a/Lib/json/__main__.py b/Lib/json/__main__.py new file mode 100644 index 00000000000..1808eaddb62 --- /dev/null +++ b/Lib/json/__main__.py @@ -0,0 +1,20 @@ +"""Command-line tool to validate and pretty-print JSON + +Usage:: + + $ echo '{"json":"obj"}' | python -m json + { + "json": "obj" + } + $ echo '{ 1.2:3.4}' | python -m json + Expecting property name enclosed in double quotes: line 1 column 3 (char 2) + +""" +import json.tool + + +if __name__ == '__main__': + try: + json.tool.main() + except BrokenPipeError as exc: + raise SystemExit(exc.errno) diff --git a/Lib/json/decoder.py b/Lib/json/decoder.py index 9e6ca981d76..db87724a897 100644 --- a/Lib/json/decoder.py +++ b/Lib/json/decoder.py @@ -311,10 +311,10 @@ def __init__(self, *, object_hook=None, parse_float=None, place of the given ``dict``. This can be used to provide custom deserializations (e.g. to support JSON-RPC class hinting). - ``object_pairs_hook``, if specified will be called with the result of - every JSON object decoded with an ordered list of pairs. The return - value of ``object_pairs_hook`` will be used instead of the ``dict``. - This feature can be used to implement custom decoders. + ``object_pairs_hook``, if specified will be called with the result + of every JSON object decoded with an ordered list of pairs. The + return value of ``object_pairs_hook`` will be used instead of the + ``dict``. This feature can be used to implement custom decoders. If ``object_hook`` is also defined, the ``object_pairs_hook`` takes priority. diff --git a/Lib/json/encoder.py b/Lib/json/encoder.py index 08ef39d1592..5cf6d64f3ea 100644 --- a/Lib/json/encoder.py +++ b/Lib/json/encoder.py @@ -111,9 +111,10 @@ def __init__(self, *, skipkeys=False, ensure_ascii=True, encoding of keys that are not str, int, float, bool or None. If skipkeys is True, such items are simply skipped. - If ensure_ascii is true, the output is guaranteed to be str - objects with all incoming non-ASCII characters escaped. If - ensure_ascii is false, the output can contain non-ASCII characters. + If ensure_ascii is true, the output is guaranteed to be str objects + with all incoming non-ASCII and non-printable characters escaped. + If ensure_ascii is false, the output can contain non-ASCII and + non-printable characters. If check_circular is true, then lists, dicts, and custom encoded objects will be checked for circular references during encoding to @@ -134,14 +135,15 @@ def __init__(self, *, skipkeys=False, ensure_ascii=True, indent level. An indent level of 0 will only insert newlines. None is the most compact representation. - If specified, separators should be an (item_separator, key_separator) - tuple. The default is (', ', ': ') if *indent* is ``None`` and - (',', ': ') otherwise. To get the most compact JSON representation, - you should specify (',', ':') to eliminate whitespace. + If specified, separators should be an (item_separator, + key_separator) tuple. The default is (', ', ': ') if *indent* is + ``None`` and (',', ': ') otherwise. To get the most compact JSON + representation, you should specify (',', ':') to eliminate + whitespace. If specified, default is a function that gets called for objects - that can't otherwise be serialized. It should return a JSON encodable - version of the object or raise a ``TypeError``. + that can't otherwise be serialized. It should return a JSON + encodable version of the object or raise a ``TypeError``. """ @@ -293,37 +295,40 @@ def _iterencode_list(lst, _current_indent_level): else: newline_indent = None separator = _item_separator - first = True - for value in lst: - if first: - first = False - else: + for i, value in enumerate(lst): + if i: buf = separator - if isinstance(value, str): - yield buf + _encoder(value) - elif value is None: - yield buf + 'null' - elif value is True: - yield buf + 'true' - elif value is False: - yield buf + 'false' - elif isinstance(value, int): - # Subclasses of int/float may override __repr__, but we still - # want to encode them as integers/floats in JSON. One example - # within the standard library is IntEnum. - yield buf + _intstr(value) - elif isinstance(value, float): - # see comment above for int - yield buf + _floatstr(value) - else: - yield buf - if isinstance(value, (list, tuple)): - chunks = _iterencode_list(value, _current_indent_level) - elif isinstance(value, dict): - chunks = _iterencode_dict(value, _current_indent_level) + try: + if isinstance(value, str): + yield buf + _encoder(value) + elif value is None: + yield buf + 'null' + elif value is True: + yield buf + 'true' + elif value is False: + yield buf + 'false' + elif isinstance(value, int): + # Subclasses of int/float may override __repr__, but we still + # want to encode them as integers/floats in JSON. One example + # within the standard library is IntEnum. + yield buf + _intstr(value) + elif isinstance(value, float): + # see comment above for int + yield buf + _floatstr(value) else: - chunks = _iterencode(value, _current_indent_level) - yield from chunks + yield buf + if isinstance(value, (list, tuple)): + chunks = _iterencode_list(value, _current_indent_level) + elif isinstance(value, dict): + chunks = _iterencode_dict(value, _current_indent_level) + else: + chunks = _iterencode(value, _current_indent_level) + yield from chunks + except GeneratorExit: + raise + except BaseException as exc: + exc.add_note(f'when serializing {type(lst).__name__} item {i}') + raise if newline_indent is not None: _current_indent_level -= 1 yield '\n' + _indent * _current_indent_level @@ -383,28 +388,34 @@ def _iterencode_dict(dct, _current_indent_level): yield item_separator yield _encoder(key) yield _key_separator - if isinstance(value, str): - yield _encoder(value) - elif value is None: - yield 'null' - elif value is True: - yield 'true' - elif value is False: - yield 'false' - elif isinstance(value, int): - # see comment for int/float in _make_iterencode - yield _intstr(value) - elif isinstance(value, float): - # see comment for int/float in _make_iterencode - yield _floatstr(value) - else: - if isinstance(value, (list, tuple)): - chunks = _iterencode_list(value, _current_indent_level) - elif isinstance(value, dict): - chunks = _iterencode_dict(value, _current_indent_level) + try: + if isinstance(value, str): + yield _encoder(value) + elif value is None: + yield 'null' + elif value is True: + yield 'true' + elif value is False: + yield 'false' + elif isinstance(value, int): + # see comment for int/float in _make_iterencode + yield _intstr(value) + elif isinstance(value, float): + # see comment for int/float in _make_iterencode + yield _floatstr(value) else: - chunks = _iterencode(value, _current_indent_level) - yield from chunks + if isinstance(value, (list, tuple)): + chunks = _iterencode_list(value, _current_indent_level) + elif isinstance(value, dict): + chunks = _iterencode_dict(value, _current_indent_level) + else: + chunks = _iterencode(value, _current_indent_level) + yield from chunks + except GeneratorExit: + raise + except BaseException as exc: + exc.add_note(f'when serializing {type(dct).__name__} item {key!r}') + raise if not first and newline_indent is not None: _current_indent_level -= 1 yield '\n' + _indent * _current_indent_level @@ -437,8 +448,14 @@ def _iterencode(o, _current_indent_level): if markerid in markers: raise ValueError("Circular reference detected") markers[markerid] = o - o = _default(o) - yield from _iterencode(o, _current_indent_level) + newobj = _default(o) + try: + yield from _iterencode(newobj, _current_indent_level) + except GeneratorExit: + raise + except BaseException as exc: + exc.add_note(f'when serializing {type(o).__name__} object') + raise if markers is not None: del markers[markerid] return _iterencode diff --git a/Lib/json/tool.py b/Lib/json/tool.py index fdfc3372bcc..1967817add8 100644 --- a/Lib/json/tool.py +++ b/Lib/json/tool.py @@ -1,25 +1,50 @@ -r"""Command-line tool to validate and pretty-print JSON - -Usage:: - - $ echo '{"json":"obj"}' | python -m json.tool - { - "json": "obj" - } - $ echo '{ 1.2:3.4}' | python -m json.tool - Expecting property name enclosed in double quotes: line 1 column 3 (char 2) +"""Command-line tool to validate and pretty-print JSON +See `json.__main__` for a usage example (invocation as +`python -m json.tool` is supported for backwards compatibility). """ import argparse import json +import re import sys +from _colorize import get_theme, can_colorize + + +# The string we are colorizing is valid JSON, +# so we can use a looser but simpler regex to match +# the various parts, most notably strings and numbers, +# where the regex given by the spec is much more complex. +_color_pattern = re.compile(r''' + (?P"(\\.|[^"\\])*")(?=:) | + (?P"(\\.|[^"\\])*") | + (?PNaN|-?Infinity|[0-9\-+.Ee]+) | + (?Ptrue|false) | + (?Pnull) +''', re.VERBOSE) + +_group_to_theme_color = { + "key": "definition", + "string": "string", + "number": "number", + "boolean": "keyword", + "null": "keyword", +} + + +def _colorize_json(json_str, theme): + def _replace_match_callback(match): + for group, color in _group_to_theme_color.items(): + if m := match.group(group): + return f"{theme[color]}{m}{theme.reset}" + return match.group() + + return re.sub(_color_pattern, _replace_match_callback, json_str) def main(): - prog = 'python -m json.tool' description = ('A simple command line interface for json module ' 'to validate and pretty-print JSON objects.') - parser = argparse.ArgumentParser(prog=prog, description=description) + parser = argparse.ArgumentParser(description=description, color=True) parser.add_argument('infile', nargs='?', help='a JSON file to be validated or pretty-printed', default='-') @@ -75,9 +100,16 @@ def main(): else: outfile = open(options.outfile, 'w', encoding='utf-8') with outfile: - for obj in objs: - json.dump(obj, outfile, **dump_args) - outfile.write('\n') + if can_colorize(file=outfile): + t = get_theme(tty_file=outfile).syntax + for obj in objs: + json_str = json.dumps(obj, **dump_args) + outfile.write(_colorize_json(json_str, t)) + outfile.write('\n') + else: + for obj in objs: + json.dump(obj, outfile, **dump_args) + outfile.write('\n') except ValueError as e: raise SystemExit(e) @@ -86,4 +118,4 @@ def main(): try: main() except BrokenPipeError as exc: - sys.exit(exc.errno) + raise SystemExit(exc.errno) diff --git a/Lib/linecache.py b/Lib/linecache.py index dc02de19eb6..2b5a31b3e75 100644 --- a/Lib/linecache.py +++ b/Lib/linecache.py @@ -63,6 +63,16 @@ def _getlines_from_code(code): return [] +def _source_unavailable(filename): + """Return True if the source code is unavailable for such file name.""" + return ( + not filename + or (filename.startswith('<') + and filename.endswith('>') + and not filename.startswith('')): + if _source_unavailable(filename): return [] - fullname = filename + if filename.startswith(' 1: + scheme = p.scheme + url = p.path + else: + return self.guess_file_type(url, strict=strict) if scheme == 'data': # syntax of data URLs: # dataurl := "data:" [ mediatype ] [ ";base64" ] "," data @@ -134,26 +159,43 @@ def guess_type(self, url, strict=True): if '=' in type or '/' not in type: type = 'text/plain' return type, None # never compressed, so encoding is None - base, ext = posixpath.splitext(url) - while ext in self.suffix_map: - base, ext = posixpath.splitext(base + self.suffix_map[ext]) + + # Lazy import to improve module import time + import posixpath + + return self._guess_file_type(url, strict, posixpath.splitext) + + def guess_file_type(self, path, *, strict=True): + """Guess the type of a file based on its path. + + Similar to guess_type(), but takes file path instead of URL. + """ + # Lazy import to improve module import time + import os + + path = os.fsdecode(path) + path = os.path.splitdrive(path)[1] + return self._guess_file_type(path, strict, os.path.splitext) + + def _guess_file_type(self, path, strict, splitext): + base, ext = splitext(path) + while (ext_lower := ext.lower()) in self.suffix_map: + base, ext = splitext(base + self.suffix_map[ext_lower]) + # encodings_map is case sensitive if ext in self.encodings_map: encoding = self.encodings_map[ext] - base, ext = posixpath.splitext(base) + base, ext = splitext(base) else: encoding = None + ext = ext.lower() types_map = self.types_map[True] if ext in types_map: return types_map[ext], encoding - elif ext.lower() in types_map: - return types_map[ext.lower()], encoding elif strict: return None, encoding types_map = self.types_map[False] if ext in types_map: return types_map[ext], encoding - elif ext.lower() in types_map: - return types_map[ext.lower()], encoding else: return None, encoding @@ -163,13 +205,13 @@ def guess_all_extensions(self, type, strict=True): Return value is a list of strings giving the possible filename extensions, including the leading dot ('.'). The extension is not guaranteed to have been associated with any particular data stream, - but would be mapped to the MIME type `type' by guess_type(). + but would be mapped to the MIME type 'type' by guess_type(). - Optional `strict' argument when false adds a bunch of commonly found, + Optional 'strict' argument when false adds a bunch of commonly found, but non-standard types. """ type = type.lower() - extensions = self.types_map_inv[True].get(type, []) + extensions = list(self.types_map_inv[True].get(type, [])) if not strict: for ext in self.types_map_inv[False].get(type, []): if ext not in extensions: @@ -182,11 +224,11 @@ def guess_extension(self, type, strict=True): Return value is a string giving a filename extension, including the leading dot ('.'). The extension is not guaranteed to have been associated with any particular data - stream, but would be mapped to the MIME type `type' by - guess_type(). If no extension can be guessed for `type', None + stream, but would be mapped to the MIME type 'type' by + guess_type(). If no extension can be guessed for 'type', None is returned. - Optional `strict' argument when false adds a bunch of commonly found, + Optional 'strict' argument when false adds a bunch of commonly found, but non-standard types. """ extensions = self.guess_all_extensions(type, strict) @@ -213,10 +255,7 @@ def readfp(self, fp, strict=True): list of standard types, else to the list of non-standard types. """ - while 1: - line = fp.readline() - if not line: - break + while line := fp.readline(): words = line.split() for i in range(len(words)): if words[i][0] == '#': @@ -237,10 +276,21 @@ def read_windows_registry(self, strict=True): types. """ - # Windows only - if not _winreg: + if not _mimetypes_read_windows_registry and not _winreg: return + add_type = self.add_type + if strict: + add_type = lambda type, ext: self.add_type(type, ext, True) + + # Accelerated function if it is available + if _mimetypes_read_windows_registry: + _mimetypes_read_windows_registry(add_type) + elif _winreg: + self._read_windows_registry(add_type) + + @classmethod + def _read_windows_registry(cls, add_type): def enum_types(mimedb): i = 0 while True: @@ -265,7 +315,7 @@ def enum_types(mimedb): subkey, 'Content Type') if datatype != _winreg.REG_SZ: continue - self.add_type(mimetype, subkeyname, strict) + add_type(mimetype, subkeyname) except OSError: continue @@ -284,7 +334,7 @@ def guess_type(url, strict=True): to ".tar.gz". (This is table-driven too, using the dictionary suffix_map). - Optional `strict' argument when false adds a bunch of commonly found, but + Optional 'strict' argument when false adds a bunch of commonly found, but non-standard types. """ if _db is None: @@ -292,17 +342,27 @@ def guess_type(url, strict=True): return _db.guess_type(url, strict) +def guess_file_type(path, *, strict=True): + """Guess the type of a file based on its path. + + Similar to guess_type(), but takes file path instead of URL. + """ + if _db is None: + init() + return _db.guess_file_type(path, strict=strict) + + def guess_all_extensions(type, strict=True): """Guess the extensions for a file based on its MIME type. Return value is a list of strings giving the possible filename extensions, including the leading dot ('.'). The extension is not guaranteed to have been associated with any particular data - stream, but would be mapped to the MIME type `type' by - guess_type(). If no extension can be guessed for `type', None + stream, but would be mapped to the MIME type 'type' by + guess_type(). If no extension can be guessed for 'type', None is returned. - Optional `strict' argument when false adds a bunch of commonly found, + Optional 'strict' argument when false adds a bunch of commonly found, but non-standard types. """ if _db is None: @@ -315,10 +375,10 @@ def guess_extension(type, strict=True): Return value is a string giving a filename extension, including the leading dot ('.'). The extension is not guaranteed to have been associated with any particular data stream, but would be mapped to the - MIME type `type' by guess_type(). If no extension can be guessed for - `type', None is returned. + MIME type 'type' by guess_type(). If no extension can be guessed for + 'type', None is returned. - Optional `strict' argument when false adds a bunch of commonly found, + Optional 'strict' argument when false adds a bunch of commonly found, but non-standard types. """ if _db is None: @@ -349,8 +409,8 @@ def init(files=None): if files is None or _db is None: db = MimeTypes() - if _winreg: - db.read_windows_registry() + # Quick return if not supported + db.read_windows_registry() if files is None: files = knownfiles @@ -359,6 +419,9 @@ def init(files=None): else: db = _db + # Lazy import to improve module import time + import os + for file in files: if os.path.isfile(file): db.read(file) @@ -401,23 +464,28 @@ def _default_mime_types(): '.Z': 'compress', '.bz2': 'bzip2', '.xz': 'xz', + '.br': 'br', } # Before adding new types, make sure they are either registered with IANA, - # at http://www.iana.org/assignments/media-types + # at https://www.iana.org/assignments/media-types/media-types.xhtml # or extensions, i.e. using the x- prefix # If you add to these, please keep them sorted by mime type. # Make sure the entry with the preferred file extension for a particular mime type # appears before any others of the same mimetype. types_map = _types_map_default = { - '.js' : 'application/javascript', - '.mjs' : 'application/javascript', + '.js' : 'text/javascript', + '.mjs' : 'text/javascript', + '.epub' : 'application/epub+zip', + '.gz' : 'application/gzip', '.json' : 'application/json', '.webmanifest': 'application/manifest+json', '.doc' : 'application/msword', '.dot' : 'application/msword', '.wiz' : 'application/msword', + '.nq' : 'application/n-quads', + '.nt' : 'application/n-triples', '.bin' : 'application/octet-stream', '.a' : 'application/octet-stream', '.dll' : 'application/octet-stream', @@ -426,24 +494,37 @@ def _default_mime_types(): '.obj' : 'application/octet-stream', '.so' : 'application/octet-stream', '.oda' : 'application/oda', + '.ogx' : 'application/ogg', '.pdf' : 'application/pdf', '.p7c' : 'application/pkcs7-mime', '.ps' : 'application/postscript', '.ai' : 'application/postscript', '.eps' : 'application/postscript', + '.trig' : 'application/trig', '.m3u' : 'application/vnd.apple.mpegurl', '.m3u8' : 'application/vnd.apple.mpegurl', '.xls' : 'application/vnd.ms-excel', '.xlb' : 'application/vnd.ms-excel', + '.eot' : 'application/vnd.ms-fontobject', '.ppt' : 'application/vnd.ms-powerpoint', '.pot' : 'application/vnd.ms-powerpoint', '.ppa' : 'application/vnd.ms-powerpoint', '.pps' : 'application/vnd.ms-powerpoint', '.pwz' : 'application/vnd.ms-powerpoint', + '.odg' : 'application/vnd.oasis.opendocument.graphics', + '.odp' : 'application/vnd.oasis.opendocument.presentation', + '.ods' : 'application/vnd.oasis.opendocument.spreadsheet', + '.odt' : 'application/vnd.oasis.opendocument.text', + '.pptx' : 'application/vnd.openxmlformats-officedocument.presentationml.presentation', + '.xlsx' : 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', + '.docx' : 'application/vnd.openxmlformats-officedocument.wordprocessingml.document', + '.rar' : 'application/vnd.rar', '.wasm' : 'application/wasm', + '.7z' : 'application/x-7z-compressed', '.bcpio' : 'application/x-bcpio', '.cpio' : 'application/x-cpio', '.csh' : 'application/x-csh', + '.deb' : 'application/x-debian-package', '.dvi' : 'application/x-dvi', '.gtar' : 'application/x-gtar', '.hdf' : 'application/x-hdf', @@ -453,10 +534,12 @@ def _default_mime_types(): '.cdf' : 'application/x-netcdf', '.nc' : 'application/x-netcdf', '.p12' : 'application/x-pkcs12', + '.php' : 'application/x-httpd-php', '.pfx' : 'application/x-pkcs12', '.ram' : 'application/x-pn-realaudio', '.pyc' : 'application/x-python-code', '.pyo' : 'application/x-python-code', + '.rpm' : 'application/x-rpm', '.sh' : 'application/x-sh', '.shar' : 'application/x-shar', '.swf' : 'application/x-shockwave-flash', @@ -479,29 +562,61 @@ def _default_mime_types(): '.rdf' : 'application/xml', '.wsdl' : 'application/xml', '.xpdl' : 'application/xml', + '.yaml' : 'application/yaml', + '.yml' : 'application/yaml', '.zip' : 'application/zip', + '.3gp' : 'audio/3gpp', + '.3gpp' : 'audio/3gpp', + '.3g2' : 'audio/3gpp2', + '.3gpp2' : 'audio/3gpp2', + '.aac' : 'audio/aac', + '.adts' : 'audio/aac', + '.loas' : 'audio/aac', + '.ass' : 'audio/aac', '.au' : 'audio/basic', '.snd' : 'audio/basic', + '.flac' : 'audio/flac', + '.mka' : 'audio/matroska', + '.m4a' : 'audio/mp4', '.mp3' : 'audio/mpeg', '.mp2' : 'audio/mpeg', + '.ogg' : 'audio/ogg', + '.opus' : 'audio/opus', '.aif' : 'audio/x-aiff', '.aifc' : 'audio/x-aiff', '.aiff' : 'audio/x-aiff', '.ra' : 'audio/x-pn-realaudio', - '.wav' : 'audio/x-wav', + '.wav' : 'audio/vnd.wave', + '.otf' : 'font/otf', + '.ttf' : 'font/ttf', + '.weba' : 'audio/webm', + '.woff' : 'font/woff', + '.woff2' : 'font/woff2', + '.avif' : 'image/avif', '.bmp' : 'image/bmp', + '.emf' : 'image/emf', + '.fits' : 'image/fits', + '.g3' : 'image/g3fax', '.gif' : 'image/gif', '.ief' : 'image/ief', + '.jp2' : 'image/jp2', '.jpg' : 'image/jpeg', '.jpe' : 'image/jpeg', '.jpeg' : 'image/jpeg', + '.jpm' : 'image/jpm', + '.jpx' : 'image/jpx', + '.heic' : 'image/heic', + '.heif' : 'image/heif', '.png' : 'image/png', '.svg' : 'image/svg+xml', + '.t38' : 'image/t38', '.tiff' : 'image/tiff', '.tif' : 'image/tiff', + '.tfx' : 'image/tiff-fx', '.ico' : 'image/vnd.microsoft.icon', + '.webp' : 'image/webp', + '.wmf' : 'image/wmf', '.ras' : 'image/x-cmu-raster', - '.bmp' : 'image/x-ms-bmp', '.pnm' : 'image/x-portable-anymap', '.pbm' : 'image/x-portable-bitmap', '.pgm' : 'image/x-portable-graymap', @@ -514,34 +629,49 @@ def _default_mime_types(): '.mht' : 'message/rfc822', '.mhtml' : 'message/rfc822', '.nws' : 'message/rfc822', + '.gltf' : 'model/gltf+json', + '.glb' : 'model/gltf-binary', + '.stl' : 'model/stl', '.css' : 'text/css', '.csv' : 'text/csv', '.html' : 'text/html', '.htm' : 'text/html', + '.md' : 'text/markdown', + '.markdown': 'text/markdown', + '.n3' : 'text/n3', '.txt' : 'text/plain', '.bat' : 'text/plain', '.c' : 'text/plain', '.h' : 'text/plain', '.ksh' : 'text/plain', '.pl' : 'text/plain', + '.srt' : 'text/plain', '.rtx' : 'text/richtext', + '.rtf' : 'text/rtf', '.tsv' : 'text/tab-separated-values', + '.vtt' : 'text/vtt', '.py' : 'text/x-python', + '.rst' : 'text/x-rst', '.etx' : 'text/x-setext', '.sgm' : 'text/x-sgml', '.sgml' : 'text/x-sgml', '.vcf' : 'text/x-vcard', '.xml' : 'text/xml', + '.mkv' : 'video/matroska', + '.mk3d' : 'video/matroska-3d', '.mp4' : 'video/mp4', '.mpeg' : 'video/mpeg', '.m1v' : 'video/mpeg', '.mpa' : 'video/mpeg', '.mpe' : 'video/mpeg', '.mpg' : 'video/mpeg', + '.ogv' : 'video/ogg', '.mov' : 'video/quicktime', '.qt' : 'video/quicktime', '.webm' : 'video/webm', - '.avi' : 'video/x-msvideo', + '.avi' : 'video/vnd.avi', + '.m4v' : 'video/x-m4v', + '.wmv' : 'video/x-ms-wmv', '.movie' : 'video/x-sgi-movie', } @@ -551,6 +681,7 @@ def _default_mime_types(): # Please sort these too common_types = _common_types_default = { '.rtf' : 'application/rtf', + '.apk' : 'application/vnd.android.package-archive', '.midi': 'audio/midi', '.mid' : 'audio/midi', '.jpg' : 'image/jpg', @@ -564,51 +695,53 @@ def _default_mime_types(): _default_mime_types() -def _main(): - import getopt - - USAGE = """\ -Usage: mimetypes.py [options] type - -Options: - --help / -h -- print this message and exit - --lenient / -l -- additionally search of some common, but non-standard - types. - --extension / -e -- guess extension instead of type - -More than one type argument may be given. -""" - - def usage(code, msg=''): - print(USAGE) - if msg: print(msg) - sys.exit(code) - - try: - opts, args = getopt.getopt(sys.argv[1:], 'hle', - ['help', 'lenient', 'extension']) - except getopt.error as msg: - usage(1, msg) - - strict = 1 - extension = 0 - for opt, arg in opts: - if opt in ('-h', '--help'): - usage(0) - elif opt in ('-l', '--lenient'): - strict = 0 - elif opt in ('-e', '--extension'): - extension = 1 - for gtype in args: - if extension: - guess = guess_extension(gtype, strict) - if not guess: print("I don't know anything about type", gtype) - else: print(guess) - else: - guess, encoding = guess_type(gtype, strict) - if not guess: print("I don't know anything about type", gtype) - else: print('type:', guess, 'encoding:', encoding) +def _parse_args(args): + from argparse import ArgumentParser + + parser = ArgumentParser( + description='map filename extensions to MIME types', color=True + ) + parser.add_argument( + '-e', '--extension', + action='store_true', + help='guess extension instead of type' + ) + parser.add_argument( + '-l', '--lenient', + action='store_true', + help='additionally search for common but non-standard types' + ) + parser.add_argument('type', nargs='+', help='a type to search') + args = parser.parse_args(args) + return args, parser.format_help() + + +def _main(args=None): + """Run the mimetypes command-line interface and return a text to print.""" + args, help_text = _parse_args(args) + + results = [] + if args.extension: + for gtype in args.type: + guess = guess_extension(gtype, not args.lenient) + if guess: + results.append(str(guess)) + else: + results.append(f"error: unknown type {gtype}") + return results + else: + for gtype in args.type: + guess, encoding = guess_type(gtype, not args.lenient) + if guess: + results.append(f"type: {guess} encoding: {encoding}") + else: + results.append(f"error: media type unknown for {gtype}") + return results if __name__ == '__main__': - _main() + import sys + + results = _main() + print("\n".join(results)) + sys.exit(any(result.startswith("error: ") for result in results)) diff --git a/Lib/ntpath.py b/Lib/ntpath.py index 9cdc16480f9..01f060e70be 100644 --- a/Lib/ntpath.py +++ b/Lib/ntpath.py @@ -400,17 +400,23 @@ def expanduser(path): # XXX With COMMAND.COM you can use any characters in a variable name, # XXX except '^|<>='. +_varpattern = r"'[^']*'?|%(%|[^%]*%?)|\$(\$|[-\w]+|\{[^}]*\}?)" +_varsub = None +_varsubb = None + def expandvars(path): """Expand shell variables of the forms $var, ${var} and %var%. Unknown variables are left unchanged.""" path = os.fspath(path) + global _varsub, _varsubb if isinstance(path, bytes): if b'$' not in path and b'%' not in path: return path - import string - varchars = bytes(string.ascii_letters + string.digits + '_-', 'ascii') - quote = b'\'' + if not _varsubb: + import re + _varsubb = re.compile(_varpattern.encode(), re.ASCII).sub + sub = _varsubb percent = b'%' brace = b'{' rbrace = b'}' @@ -419,94 +425,44 @@ def expandvars(path): else: if '$' not in path and '%' not in path: return path - import string - varchars = string.ascii_letters + string.digits + '_-' - quote = '\'' + if not _varsub: + import re + _varsub = re.compile(_varpattern, re.ASCII).sub + sub = _varsub percent = '%' brace = '{' rbrace = '}' dollar = '$' environ = os.environ - res = path[:0] - index = 0 - pathlen = len(path) - while index < pathlen: - c = path[index:index+1] - if c == quote: # no expansion within single quotes - path = path[index + 1:] - pathlen = len(path) - try: - index = path.index(c) - res += c + path[:index + 1] - except ValueError: - res += c + path - index = pathlen - 1 - elif c == percent: # variable or '%' - if path[index + 1:index + 2] == percent: - res += c - index += 1 - else: - path = path[index+1:] - pathlen = len(path) - try: - index = path.index(percent) - except ValueError: - res += percent + path - index = pathlen - 1 - else: - var = path[:index] - try: - if environ is None: - value = os.fsencode(os.environ[os.fsdecode(var)]) - else: - value = environ[var] - except KeyError: - value = percent + var + percent - res += value - elif c == dollar: # variable or '$$' - if path[index + 1:index + 2] == dollar: - res += c - index += 1 - elif path[index + 1:index + 2] == brace: - path = path[index+2:] - pathlen = len(path) - try: - index = path.index(rbrace) - except ValueError: - res += dollar + brace + path - index = pathlen - 1 - else: - var = path[:index] - try: - if environ is None: - value = os.fsencode(os.environ[os.fsdecode(var)]) - else: - value = environ[var] - except KeyError: - value = dollar + brace + var + rbrace - res += value - else: - var = path[:0] - index += 1 - c = path[index:index + 1] - while c and c in varchars: - var += c - index += 1 - c = path[index:index + 1] - try: - if environ is None: - value = os.fsencode(os.environ[os.fsdecode(var)]) - else: - value = environ[var] - except KeyError: - value = dollar + var - res += value - if c: - index -= 1 + + def repl(m): + lastindex = m.lastindex + if lastindex is None: + return m[0] + name = m[lastindex] + if lastindex == 1: + if name == percent: + return name + if not name.endswith(percent): + return m[0] + name = name[:-1] else: - res += c - index += 1 - return res + if name == dollar: + return name + if name.startswith(brace): + if not name.endswith(rbrace): + return m[0] + name = name[1:-1] + + try: + if environ is None: + return os.fsencode(os.environ[os.fsdecode(name)]) + else: + return environ[name] + except KeyError: + return m[0] + + return sub(repl, path) # Normalize a path, e.g. A//B, A/./B and A/foo/../B all become A\B. diff --git a/Lib/nturl2path.py b/Lib/nturl2path.py index 757fd01bec8..57c7858dff0 100644 --- a/Lib/nturl2path.py +++ b/Lib/nturl2path.py @@ -3,7 +3,15 @@ This module only exists to provide OS-specific code for urllib.requests, thus do not use directly. """ -# Testing is done through test_urllib. +# Testing is done through test_nturl2path. + +import warnings + + +warnings._deprecated( + __name__, + message=f"{warnings._DEPRECATED_MSG}; use 'urllib.request' instead", + remove=(3, 19)) def url2pathname(url): """OS-specific conversion from a relative URL of the 'file' scheme @@ -14,7 +22,7 @@ def url2pathname(url): # ///C:/foo/bar/spam.foo # become # C:\foo\bar\spam.foo - import string, urllib.parse + import urllib.parse if url[:3] == '///': # URL has an empty authority section, so the path begins on the third # character. @@ -25,19 +33,14 @@ def url2pathname(url): if url[:3] == '///': # Skip past extra slash before UNC drive in URL path. url = url[1:] - # Windows itself uses ":" even in URLs. - url = url.replace(':', '|') - if not '|' in url: - # No drive specifier, just convert slashes - # make sure not to convert quoted slashes :-) - return urllib.parse.unquote(url.replace('/', '\\')) - comp = url.split('|') - if len(comp) != 2 or comp[0][-1] not in string.ascii_letters: - error = 'Bad URL: ' + url - raise OSError(error) - drive = comp[0][-1].upper() - tail = urllib.parse.unquote(comp[1].replace('/', '\\')) - return drive + ':' + tail + else: + if url[:1] == '/' and url[2:3] in (':', '|'): + # Skip past extra slash before DOS drive in URL path. + url = url[1:] + if url[1:2] == '|': + # Older URLs use a pipe after a drive letter + url = url[:1] + ':' + url[2:] + return urllib.parse.unquote(url.replace('/', '\\')) def pathname2url(p): """OS-specific conversion from a file system path to a relative URL @@ -46,6 +49,7 @@ def pathname2url(p): # C:\foo\bar\spam.foo # becomes # ///C:/foo/bar/spam.foo + import ntpath import urllib.parse # First, clean up some special forms. We are going to sacrifice # the additional information anyway @@ -54,16 +58,17 @@ def pathname2url(p): p = p[4:] if p[:4].upper() == 'UNC/': p = '//' + p[4:] - elif p[1:2] != ':': - raise OSError('Bad path: ' + p) - if not ':' in p: - # No DOS drive specified, just quote the pathname - return urllib.parse.quote(p) - comp = p.split(':', maxsplit=2) - if len(comp) != 2 or len(comp[0]) > 1: - error = 'Bad path: ' + p - raise OSError(error) + drive, root, tail = ntpath.splitroot(p) + if drive: + if drive[1:] == ':': + # DOS drive specified. Add three slashes to the start, producing + # an authority section with a zero-length authority, and a path + # section starting with a single slash. + drive = f'///{drive}' + drive = urllib.parse.quote(drive, safe='/:') + elif root: + # Add explicitly empty authority to path beginning with one slash. + root = f'//{root}' - drive = urllib.parse.quote(comp[0].upper()) - tail = urllib.parse.quote(comp[1]) - return '///' + drive + ':' + tail + tail = urllib.parse.quote(tail) + return drive + root + tail diff --git a/Lib/opcode.py b/Lib/opcode.py index 5735686fa7f..0e9520b6832 100644 --- a/Lib/opcode.py +++ b/Lib/opcode.py @@ -9,16 +9,18 @@ "HAVE_ARGUMENT", "EXTENDED_ARG", "hasarg", "hasconst", "hasname", "hasjump", "hasjrel", "hasjabs", "hasfree", "haslocal", "hasexc"] +import builtins import _opcode from _opcode import stack_effect -from _opcode_metadata import (_specializations, _specialized_opmap, opmap, - HAVE_ARGUMENT, MIN_INSTRUMENTED_OPCODE) +from _opcode_metadata import (_specializations, _specialized_opmap, opmap, # noqa: F401 + HAVE_ARGUMENT, MIN_INSTRUMENTED_OPCODE) # noqa: F401 EXTENDED_ARG = opmap['EXTENDED_ARG'] opname = ['<%r>' % (op,) for op in range(max(opmap.values()) + 1)] -for op, i in opmap.items(): - opname[i] = op +for m in (opmap, _specialized_opmap): + for op, i in m.items(): + opname[i] = op cmp_op = ('<', '<=', '==', '!=', '>', '>=') @@ -36,6 +38,9 @@ _intrinsic_1_descs = _opcode.get_intrinsic1_descs() _intrinsic_2_descs = _opcode.get_intrinsic2_descs() +_special_method_names = _opcode.get_special_method_names() +_common_constants = [builtins.AssertionError, builtins.NotImplementedError, + builtins.tuple, builtins.all, builtins.any] _nb_ops = _opcode.get_nb_ops() hascompare = [opmap["COMPARE_OP"]] @@ -49,6 +54,7 @@ }, "BINARY_OP": { "counter": 1, + "descr": 4, }, "UNPACK_SEQUENCE": { "counter": 1, @@ -59,9 +65,6 @@ "CONTAINS_OP": { "counter": 1, }, - "BINARY_SUBSCR": { - "counter": 1, - }, "FOR_ITER": { "counter": 1, }, @@ -83,6 +86,10 @@ "counter": 1, "func_version": 2, }, + "CALL_KW": { + "counter": 1, + "func_version": 2, + }, "STORE_SUBSCR": { "counter": 1, }, diff --git a/Lib/operator.py b/Lib/operator.py index 02ccdaa13dd..1b765522f85 100644 --- a/Lib/operator.py +++ b/Lib/operator.py @@ -14,8 +14,8 @@ 'delitem', 'eq', 'floordiv', 'ge', 'getitem', 'gt', 'iadd', 'iand', 'iconcat', 'ifloordiv', 'ilshift', 'imatmul', 'imod', 'imul', 'index', 'indexOf', 'inv', 'invert', 'ior', 'ipow', 'irshift', - 'is_', 'is_not', 'isub', 'itemgetter', 'itruediv', 'ixor', 'le', - 'length_hint', 'lshift', 'lt', 'matmul', 'methodcaller', 'mod', + 'is_', 'is_none', 'is_not', 'is_not_none', 'isub', 'itemgetter', 'itruediv', + 'ixor', 'le', 'length_hint', 'lshift', 'lt', 'matmul', 'methodcaller', 'mod', 'mul', 'ne', 'neg', 'not_', 'or_', 'pos', 'pow', 'rshift', 'setitem', 'sub', 'truediv', 'truth', 'xor'] @@ -66,6 +66,14 @@ def is_not(a, b): "Same as a is not b." return a is not b +def is_none(a): + "Same as a is None." + return a is None + +def is_not_none(a): + "Same as a is not None." + return a is not None + # Mathematical/Bitwise Operations *********************************************# def abs(a): @@ -415,7 +423,7 @@ def ixor(a, b): except ImportError: pass else: - from _operator import __doc__ + from _operator import __doc__ # noqa: F401 # All of these "__func__ = func" assignments have to happen after importing # from _operator to make sure they're set to the right function diff --git a/Lib/optparse.py b/Lib/optparse.py index 1c450c6fcbe..38cf16d21ef 100644 --- a/Lib/optparse.py +++ b/Lib/optparse.py @@ -43,7 +43,7 @@ __copyright__ = """ Copyright (c) 2001-2006 Gregory P. Ward. All rights reserved. -Copyright (c) 2002-2006 Python Software Foundation. All rights reserved. +Copyright (c) 2002 Python Software Foundation. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are @@ -74,7 +74,8 @@ """ import sys, os -import textwrap +from gettext import gettext as _, ngettext + def _repr(self): return "<%s at 0x%x: %s>" % (self.__class__.__name__, id(self), self) @@ -86,19 +87,6 @@ def _repr(self): # Id: help.py 527 2006-07-23 15:21:30Z greg # Id: errors.py 509 2006-04-20 00:58:24Z gward -try: - from gettext import gettext, ngettext -except ImportError: - def gettext(message): - return message - - def ngettext(singular, plural, n): - if n == 1: - return singular - return plural - -_ = gettext - class OptParseError (Exception): def __init__(self, msg): @@ -263,6 +251,7 @@ def _format_text(self, text): Format a paragraph of free-form text for inclusion in the help output at the current indentation level. """ + import textwrap text_width = max(self.width - self.current_indent, 11) indent = " "*self.current_indent return textwrap.fill(text, @@ -319,6 +308,7 @@ def format_option(self, option): indent_first = 0 result.append(opts) if option.help: + import textwrap help_text = self.expand_default(option) help_lines = textwrap.wrap(help_text, self.help_width) result.append("%*s%s\n" % (indent_first, "", help_lines[0])) diff --git a/Lib/py_compile.py b/Lib/py_compile.py index 388614e51b1..43d8ec90ffb 100644 --- a/Lib/py_compile.py +++ b/Lib/py_compile.py @@ -177,7 +177,7 @@ def main(): import argparse description = 'A simple command-line interface for py_compile module.' - parser = argparse.ArgumentParser(description=description) + parser = argparse.ArgumentParser(description=description, color=True) parser.add_argument( '-q', '--quiet', action='store_true', diff --git a/Lib/pydoc_data/_pydoc.css b/Lib/pydoc_data/_pydoc.css index f036ef37a5a..a6aa2e4c1a0 100644 --- a/Lib/pydoc_data/_pydoc.css +++ b/Lib/pydoc_data/_pydoc.css @@ -4,3 +4,109 @@ Contents of this file are subject to change without notice. */ + +body { + background-color: #f0f0f8; +} + +table.heading tr { + background-color: #7799ee; +} + +.decor { + color: #ffffff; +} + +.title-decor { + background-color: #ffc8d8; + color: #000000; +} + +.pkg-content-decor { + background-color: #aa55cc; +} + +.index-decor { + background-color: #ee77aa; +} + +.functions-decor { + background-color: #eeaa77; +} + +.data-decor { + background-color: #55aa55; +} + +.author-decor { + background-color: #7799ee; +} + +.credits-decor { + background-color: #7799ee; +} + +.error-decor { + background-color: #bb0000; +} + +.grey { + color: #909090; +} + +.white { + color: #ffffff; +} + +.repr { + color: #c040c0; +} + +table.heading tr td.title { + vertical-align: bottom; +} + +table.heading tr td.extra { + vertical-align: bottom; + text-align: right; +} + +.heading-text { + font-family: helvetica, arial; +} + +.bigsection { + font-size: larger; +} + +.title { + font-size: x-large; +} + +.code { + font-family: monospace; +} + +table { + width: 100%; + border-spacing : 0; + border-collapse : collapse; + border: 0; +} + +td { + padding: 2; +} + +td.section-title { + vertical-align: bottom; +} + +td.multicolumn { + width: 25%; + vertical-align: bottom; +} + +td.singlecolumn { + width: 100%; +} diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py index e4c63058087..56317b8a724 100644 --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,13062 +1,14103 @@ -# -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Sun Dec 23 16:24:21 2018 -topics = {'assert': 'The "assert" statement\n' - '**********************\n' - '\n' - 'Assert statements are a convenient way to insert debugging ' - 'assertions\n' - 'into a program:\n' - '\n' - ' assert_stmt ::= "assert" expression ["," expression]\n' - '\n' - 'The simple form, "assert expression", is equivalent to\n' - '\n' - ' if __debug__:\n' - ' if not expression: raise AssertionError\n' - '\n' - 'The extended form, "assert expression1, expression2", is ' - 'equivalent to\n' - '\n' - ' if __debug__:\n' - ' if not expression1: raise AssertionError(expression2)\n' - '\n' - 'These equivalences assume that "__debug__" and "AssertionError" ' - 'refer\n' - 'to the built-in variables with those names. In the current\n' - 'implementation, the built-in variable "__debug__" is "True" under\n' - 'normal circumstances, "False" when optimization is requested ' - '(command\n' - 'line option "-O"). The current code generator emits no code for ' - 'an\n' - 'assert statement when optimization is requested at compile time. ' - 'Note\n' - 'that it is unnecessary to include the source code for the ' - 'expression\n' - 'that failed in the error message; it will be displayed as part of ' - 'the\n' - 'stack trace.\n' - '\n' - 'Assignments to "__debug__" are illegal. The value for the ' - 'built-in\n' - 'variable is determined when the interpreter starts.\n', - 'assignment': 'Assignment statements\n' - '*********************\n' - '\n' - 'Assignment statements are used to (re)bind names to values and ' - 'to\n' - 'modify attributes or items of mutable objects:\n' - '\n' - ' assignment_stmt ::= (target_list "=")+ (starred_expression ' - '| yield_expression)\n' - ' target_list ::= target ("," target)* [","]\n' - ' target ::= identifier\n' - ' | "(" [target_list] ")"\n' - ' | "[" [target_list] "]"\n' - ' | attributeref\n' - ' | subscription\n' - ' | slicing\n' - ' | "*" target\n' - '\n' - '(See section Primaries for the syntax definitions for ' - '*attributeref*,\n' - '*subscription*, and *slicing*.)\n' - '\n' - 'An assignment statement evaluates the expression list ' - '(remember that\n' - 'this can be a single expression or a comma-separated list, the ' - 'latter\n' - 'yielding a tuple) and assigns the single resulting object to ' - 'each of\n' - 'the target lists, from left to right.\n' - '\n' - 'Assignment is defined recursively depending on the form of the ' - 'target\n' - '(list). When a target is part of a mutable object (an ' - 'attribute\n' - 'reference, subscription or slicing), the mutable object must\n' - 'ultimately perform the assignment and decide about its ' - 'validity, and\n' - 'may raise an exception if the assignment is unacceptable. The ' - 'rules\n' - 'observed by various types and the exceptions raised are given ' - 'with the\n' - 'definition of the object types (see section The standard type\n' - 'hierarchy).\n' - '\n' - 'Assignment of an object to a target list, optionally enclosed ' - 'in\n' - 'parentheses or square brackets, is recursively defined as ' - 'follows.\n' - '\n' - '* If the target list is a single target with no trailing ' - 'comma,\n' - ' optionally in parentheses, the object is assigned to that ' - 'target.\n' - '\n' - '* Else: The object must be an iterable with the same number of ' - 'items\n' - ' as there are targets in the target list, and the items are ' - 'assigned,\n' - ' from left to right, to the corresponding targets.\n' - '\n' - ' * If the target list contains one target prefixed with an\n' - ' asterisk, called a “starred” target: The object must be ' - 'an\n' - ' iterable with at least as many items as there are targets ' - 'in the\n' - ' target list, minus one. The first items of the iterable ' - 'are\n' - ' assigned, from left to right, to the targets before the ' - 'starred\n' - ' target. The final items of the iterable are assigned to ' - 'the\n' - ' targets after the starred target. A list of the remaining ' - 'items\n' - ' in the iterable is then assigned to the starred target ' - '(the list\n' - ' can be empty).\n' - '\n' - ' * Else: The object must be an iterable with the same number ' - 'of\n' - ' items as there are targets in the target list, and the ' - 'items are\n' - ' assigned, from left to right, to the corresponding ' - 'targets.\n' - '\n' - 'Assignment of an object to a single target is recursively ' - 'defined as\n' - 'follows.\n' - '\n' - '* If the target is an identifier (name):\n' - '\n' - ' * If the name does not occur in a "global" or "nonlocal" ' - 'statement\n' - ' in the current code block: the name is bound to the object ' - 'in the\n' - ' current local namespace.\n' - '\n' - ' * Otherwise: the name is bound to the object in the global\n' - ' namespace or the outer namespace determined by ' - '"nonlocal",\n' - ' respectively.\n' - '\n' - ' The name is rebound if it was already bound. This may cause ' - 'the\n' - ' reference count for the object previously bound to the name ' - 'to reach\n' - ' zero, causing the object to be deallocated and its ' - 'destructor (if it\n' - ' has one) to be called.\n' - '\n' - '* If the target is an attribute reference: The primary ' - 'expression in\n' - ' the reference is evaluated. It should yield an object with\n' - ' assignable attributes; if this is not the case, "TypeError" ' - 'is\n' - ' raised. That object is then asked to assign the assigned ' - 'object to\n' - ' the given attribute; if it cannot perform the assignment, it ' - 'raises\n' - ' an exception (usually but not necessarily ' - '"AttributeError").\n' - '\n' - ' Note: If the object is a class instance and the attribute ' - 'reference\n' - ' occurs on both sides of the assignment operator, the RHS ' - 'expression,\n' - ' "a.x" can access either an instance attribute or (if no ' - 'instance\n' - ' attribute exists) a class attribute. The LHS target "a.x" ' - 'is always\n' - ' set as an instance attribute, creating it if necessary. ' - 'Thus, the\n' - ' two occurrences of "a.x" do not necessarily refer to the ' - 'same\n' - ' attribute: if the RHS expression refers to a class ' - 'attribute, the\n' - ' LHS creates a new instance attribute as the target of the\n' - ' assignment:\n' - '\n' - ' class Cls:\n' - ' x = 3 # class variable\n' - ' inst = Cls()\n' - ' inst.x = inst.x + 1 # writes inst.x as 4 leaving Cls.x ' - 'as 3\n' - '\n' - ' This description does not necessarily apply to descriptor\n' - ' attributes, such as properties created with "property()".\n' - '\n' - '* If the target is a subscription: The primary expression in ' - 'the\n' - ' reference is evaluated. It should yield either a mutable ' - 'sequence\n' - ' object (such as a list) or a mapping object (such as a ' - 'dictionary).\n' - ' Next, the subscript expression is evaluated.\n' - '\n' - ' If the primary is a mutable sequence object (such as a ' - 'list), the\n' - ' subscript must yield an integer. If it is negative, the ' - 'sequence’s\n' - ' length is added to it. The resulting value must be a ' - 'nonnegative\n' - ' integer less than the sequence’s length, and the sequence is ' - 'asked\n' - ' to assign the assigned object to its item with that index. ' - 'If the\n' - ' index is out of range, "IndexError" is raised (assignment to ' - 'a\n' - ' subscripted sequence cannot add new items to a list).\n' - '\n' - ' If the primary is a mapping object (such as a dictionary), ' - 'the\n' - ' subscript must have a type compatible with the mapping’s key ' - 'type,\n' - ' and the mapping is then asked to create a key/datum pair ' - 'which maps\n' - ' the subscript to the assigned object. This can either ' - 'replace an\n' - ' existing key/value pair with the same key value, or insert a ' - 'new\n' - ' key/value pair (if no key with the same value existed).\n' - '\n' - ' For user-defined objects, the "__setitem__()" method is ' - 'called with\n' - ' appropriate arguments.\n' - '\n' - '* If the target is a slicing: The primary expression in the\n' - ' reference is evaluated. It should yield a mutable sequence ' - 'object\n' - ' (such as a list). The assigned object should be a sequence ' - 'object\n' - ' of the same type. Next, the lower and upper bound ' - 'expressions are\n' - ' evaluated, insofar they are present; defaults are zero and ' - 'the\n' - ' sequence’s length. The bounds should evaluate to integers. ' - 'If\n' - ' either bound is negative, the sequence’s length is added to ' - 'it. The\n' - ' resulting bounds are clipped to lie between zero and the ' - 'sequence’s\n' - ' length, inclusive. Finally, the sequence object is asked to ' - 'replace\n' - ' the slice with the items of the assigned sequence. The ' - 'length of\n' - ' the slice may be different from the length of the assigned ' - 'sequence,\n' - ' thus changing the length of the target sequence, if the ' - 'target\n' - ' sequence allows it.\n' - '\n' - '**CPython implementation detail:** In the current ' - 'implementation, the\n' - 'syntax for targets is taken to be the same as for expressions, ' - 'and\n' - 'invalid syntax is rejected during the code generation phase, ' - 'causing\n' - 'less detailed error messages.\n' - '\n' - 'Although the definition of assignment implies that overlaps ' - 'between\n' - 'the left-hand side and the right-hand side are ‘simultaneous’ ' - '(for\n' - 'example "a, b = b, a" swaps two variables), overlaps *within* ' - 'the\n' - 'collection of assigned-to variables occur left-to-right, ' - 'sometimes\n' - 'resulting in confusion. For instance, the following program ' - 'prints\n' - '"[0, 2]":\n' - '\n' - ' x = [0, 1]\n' - ' i = 0\n' - ' i, x[i] = 1, 2 # i is updated, then x[i] is ' - 'updated\n' - ' print(x)\n' - '\n' - 'See also:\n' - '\n' - ' **PEP 3132** - Extended Iterable Unpacking\n' - ' The specification for the "*target" feature.\n' - '\n' - '\n' - 'Augmented assignment statements\n' - '===============================\n' - '\n' - 'Augmented assignment is the combination, in a single ' - 'statement, of a\n' - 'binary operation and an assignment statement:\n' - '\n' - ' augmented_assignment_stmt ::= augtarget augop ' - '(expression_list | yield_expression)\n' - ' augtarget ::= identifier | attributeref | ' - 'subscription | slicing\n' - ' augop ::= "+=" | "-=" | "*=" | "@=" | ' - '"/=" | "//=" | "%=" | "**="\n' - ' | ">>=" | "<<=" | "&=" | "^=" | "|="\n' - '\n' - '(See section Primaries for the syntax definitions of the last ' - 'three\n' - 'symbols.)\n' - '\n' - 'An augmented assignment evaluates the target (which, unlike ' - 'normal\n' - 'assignment statements, cannot be an unpacking) and the ' - 'expression\n' - 'list, performs the binary operation specific to the type of ' - 'assignment\n' - 'on the two operands, and assigns the result to the original ' - 'target.\n' - 'The target is only evaluated once.\n' - '\n' - 'An augmented assignment expression like "x += 1" can be ' - 'rewritten as\n' - '"x = x + 1" to achieve a similar, but not exactly equal ' - 'effect. In the\n' - 'augmented version, "x" is only evaluated once. Also, when ' - 'possible,\n' - 'the actual operation is performed *in-place*, meaning that ' - 'rather than\n' - 'creating a new object and assigning that to the target, the ' - 'old object\n' - 'is modified instead.\n' - '\n' - 'Unlike normal assignments, augmented assignments evaluate the ' - 'left-\n' - 'hand side *before* evaluating the right-hand side. For ' - 'example, "a[i]\n' - '+= f(x)" first looks-up "a[i]", then it evaluates "f(x)" and ' - 'performs\n' - 'the addition, and lastly, it writes the result back to ' - '"a[i]".\n' - '\n' - 'With the exception of assigning to tuples and multiple targets ' - 'in a\n' - 'single statement, the assignment done by augmented assignment\n' - 'statements is handled the same way as normal assignments. ' - 'Similarly,\n' - 'with the exception of the possible *in-place* behavior, the ' - 'binary\n' - 'operation performed by augmented assignment is the same as the ' - 'normal\n' - 'binary operations.\n' - '\n' - 'For targets which are attribute references, the same caveat ' - 'about\n' - 'class and instance attributes applies as for regular ' - 'assignments.\n' - '\n' - '\n' - 'Annotated assignment statements\n' - '===============================\n' - '\n' - 'Annotation assignment is the combination, in a single ' - 'statement, of a\n' - 'variable or attribute annotation and an optional assignment ' - 'statement:\n' - '\n' - ' annotated_assignment_stmt ::= augtarget ":" expression ["=" ' - 'expression]\n' - '\n' - 'The difference from normal Assignment statements is that only ' - 'single\n' - 'target and only single right hand side value is allowed.\n' - '\n' - 'For simple names as assignment targets, if in class or module ' - 'scope,\n' - 'the annotations are evaluated and stored in a special class or ' - 'module\n' - 'attribute "__annotations__" that is a dictionary mapping from ' - 'variable\n' - 'names (mangled if private) to evaluated annotations. This ' - 'attribute is\n' - 'writable and is automatically created at the start of class or ' - 'module\n' - 'body execution, if annotations are found statically.\n' - '\n' - 'For expressions as assignment targets, the annotations are ' - 'evaluated\n' - 'if in class or module scope, but not stored.\n' - '\n' - 'If a name is annotated in a function scope, then this name is ' - 'local\n' - 'for that scope. Annotations are never evaluated and stored in ' - 'function\n' - 'scopes.\n' - '\n' - 'If the right hand side is present, an annotated assignment ' - 'performs\n' - 'the actual assignment before evaluating annotations (where\n' - 'applicable). If the right hand side is not present for an ' - 'expression\n' - 'target, then the interpreter evaluates the target except for ' - 'the last\n' - '"__setitem__()" or "__setattr__()" call.\n' - '\n' - 'See also:\n' - '\n' - ' **PEP 526** - Syntax for Variable Annotations\n' - ' The proposal that added syntax for annotating the types ' - 'of\n' - ' variables (including class variables and instance ' - 'variables),\n' - ' instead of expressing them through comments.\n' - '\n' - ' **PEP 484** - Type hints\n' - ' The proposal that added the "typing" module to provide a ' - 'standard\n' - ' syntax for type annotations that can be used in static ' - 'analysis\n' - ' tools and IDEs.\n', - 'atom-identifiers': 'Identifiers (Names)\n' - '*******************\n' - '\n' - 'An identifier occurring as an atom is a name. See ' - 'section Identifiers\n' - 'and keywords for lexical definition and section Naming ' - 'and binding for\n' - 'documentation of naming and binding.\n' - '\n' - 'When the name is bound to an object, evaluation of the ' - 'atom yields\n' - 'that object. When a name is not bound, an attempt to ' - 'evaluate it\n' - 'raises a "NameError" exception.\n' - '\n' - '**Private name mangling:** When an identifier that ' - 'textually occurs in\n' - 'a class definition begins with two or more underscore ' - 'characters and\n' - 'does not end in two or more underscores, it is ' - 'considered a *private\n' - 'name* of that class. Private names are transformed to a ' - 'longer form\n' - 'before code is generated for them. The transformation ' - 'inserts the\n' - 'class name, with leading underscores removed and a ' - 'single underscore\n' - 'inserted, in front of the name. For example, the ' - 'identifier "__spam"\n' - 'occurring in a class named "Ham" will be transformed to ' - '"_Ham__spam".\n' - 'This transformation is independent of the syntactical ' - 'context in which\n' - 'the identifier is used. If the transformed name is ' - 'extremely long\n' - '(longer than 255 characters), implementation defined ' - 'truncation may\n' - 'happen. If the class name consists only of underscores, ' - 'no\n' - 'transformation is done.\n', - 'atom-literals': 'Literals\n' - '********\n' - '\n' - 'Python supports string and bytes literals and various ' - 'numeric\n' - 'literals:\n' - '\n' - ' literal ::= stringliteral | bytesliteral\n' - ' | integer | floatnumber | imagnumber\n' - '\n' - 'Evaluation of a literal yields an object of the given type ' - '(string,\n' - 'bytes, integer, floating point number, complex number) with ' - 'the given\n' - 'value. The value may be approximated in the case of ' - 'floating point\n' - 'and imaginary (complex) literals. See section Literals for ' - 'details.\n' - '\n' - 'All literals correspond to immutable data types, and hence ' - 'the\n' - 'object’s identity is less important than its value. ' - 'Multiple\n' - 'evaluations of literals with the same value (either the ' - 'same\n' - 'occurrence in the program text or a different occurrence) ' - 'may obtain\n' - 'the same object or a different object with the same ' - 'value.\n', - 'attribute-access': 'Customizing attribute access\n' - '****************************\n' - '\n' - 'The following methods can be defined to customize the ' - 'meaning of\n' - 'attribute access (use of, assignment to, or deletion of ' - '"x.name") for\n' - 'class instances.\n' - '\n' - 'object.__getattr__(self, name)\n' - '\n' - ' Called when the default attribute access fails with ' - 'an\n' - ' "AttributeError" (either "__getattribute__()" raises ' - 'an\n' - ' "AttributeError" because *name* is not an instance ' - 'attribute or an\n' - ' attribute in the class tree for "self"; or ' - '"__get__()" of a *name*\n' - ' property raises "AttributeError"). This method ' - 'should either\n' - ' return the (computed) attribute value or raise an ' - '"AttributeError"\n' - ' exception.\n' - '\n' - ' Note that if the attribute is found through the ' - 'normal mechanism,\n' - ' "__getattr__()" is not called. (This is an ' - 'intentional asymmetry\n' - ' between "__getattr__()" and "__setattr__()".) This is ' - 'done both for\n' - ' efficiency reasons and because otherwise ' - '"__getattr__()" would have\n' - ' no way to access other attributes of the instance. ' - 'Note that at\n' - ' least for instance variables, you can fake total ' - 'control by not\n' - ' inserting any values in the instance attribute ' - 'dictionary (but\n' - ' instead inserting them in another object). See the\n' - ' "__getattribute__()" method below for a way to ' - 'actually get total\n' - ' control over attribute access.\n' - '\n' - 'object.__getattribute__(self, name)\n' - '\n' - ' Called unconditionally to implement attribute ' - 'accesses for\n' - ' instances of the class. If the class also defines ' - '"__getattr__()",\n' - ' the latter will not be called unless ' - '"__getattribute__()" either\n' - ' calls it explicitly or raises an "AttributeError". ' - 'This method\n' - ' should return the (computed) attribute value or raise ' - 'an\n' - ' "AttributeError" exception. In order to avoid ' - 'infinite recursion in\n' - ' this method, its implementation should always call ' - 'the base class\n' - ' method with the same name to access any attributes it ' - 'needs, for\n' - ' example, "object.__getattribute__(self, name)".\n' - '\n' - ' Note: This method may still be bypassed when looking ' - 'up special\n' - ' methods as the result of implicit invocation via ' - 'language syntax\n' - ' or built-in functions. See Special method lookup.\n' - '\n' - 'object.__setattr__(self, name, value)\n' - '\n' - ' Called when an attribute assignment is attempted. ' - 'This is called\n' - ' instead of the normal mechanism (i.e. store the value ' - 'in the\n' - ' instance dictionary). *name* is the attribute name, ' - '*value* is the\n' - ' value to be assigned to it.\n' - '\n' - ' If "__setattr__()" wants to assign to an instance ' - 'attribute, it\n' - ' should call the base class method with the same name, ' - 'for example,\n' - ' "object.__setattr__(self, name, value)".\n' - '\n' - 'object.__delattr__(self, name)\n' - '\n' - ' Like "__setattr__()" but for attribute deletion ' - 'instead of\n' - ' assignment. This should only be implemented if "del ' - 'obj.name" is\n' - ' meaningful for the object.\n' - '\n' - 'object.__dir__(self)\n' - '\n' - ' Called when "dir()" is called on the object. A ' - 'sequence must be\n' - ' returned. "dir()" converts the returned sequence to a ' - 'list and\n' - ' sorts it.\n' - '\n' - '\n' - 'Customizing module attribute access\n' - '===================================\n' - '\n' - 'For a more fine grained customization of the module ' - 'behavior (setting\n' - 'attributes, properties, etc.), one can set the ' - '"__class__" attribute\n' - 'of a module object to a subclass of "types.ModuleType". ' - 'For example:\n' - '\n' - ' import sys\n' - ' from types import ModuleType\n' - '\n' - ' class VerboseModule(ModuleType):\n' - ' def __repr__(self):\n' - " return f'Verbose {self.__name__}'\n" - '\n' - ' def __setattr__(self, attr, value):\n' - " print(f'Setting {attr}...')\n" - ' setattr(self, attr, value)\n' - '\n' - ' sys.modules[__name__].__class__ = VerboseModule\n' - '\n' - 'Note: Setting module "__class__" only affects lookups ' - 'made using the\n' - ' attribute access syntax – directly accessing the ' - 'module globals\n' - ' (whether by code within the module, or via a reference ' - 'to the\n' - ' module’s globals dictionary) is unaffected.\n' - '\n' - 'Changed in version 3.5: "__class__" module attribute is ' - 'now writable.\n' - '\n' - '\n' - 'Implementing Descriptors\n' - '========================\n' - '\n' - 'The following methods only apply when an instance of the ' - 'class\n' - 'containing the method (a so-called *descriptor* class) ' - 'appears in an\n' - '*owner* class (the descriptor must be in either the ' - 'owner’s class\n' - 'dictionary or in the class dictionary for one of its ' - 'parents). In the\n' - 'examples below, “the attribute” refers to the attribute ' - 'whose name is\n' - 'the key of the property in the owner class’ "__dict__".\n' - '\n' - 'object.__get__(self, instance, owner)\n' - '\n' - ' Called to get the attribute of the owner class (class ' - 'attribute\n' - ' access) or of an instance of that class (instance ' - 'attribute\n' - ' access). *owner* is always the owner class, while ' - '*instance* is the\n' - ' instance that the attribute was accessed through, or ' - '"None" when\n' - ' the attribute is accessed through the *owner*. This ' - 'method should\n' - ' return the (computed) attribute value or raise an ' - '"AttributeError"\n' - ' exception.\n' - '\n' - 'object.__set__(self, instance, value)\n' - '\n' - ' Called to set the attribute on an instance *instance* ' - 'of the owner\n' - ' class to a new value, *value*.\n' - '\n' - 'object.__delete__(self, instance)\n' - '\n' - ' Called to delete the attribute on an instance ' - '*instance* of the\n' - ' owner class.\n' - '\n' - 'object.__set_name__(self, owner, name)\n' - '\n' - ' Called at the time the owning class *owner* is ' - 'created. The\n' - ' descriptor has been assigned to *name*.\n' - '\n' - ' New in version 3.6.\n' - '\n' - 'The attribute "__objclass__" is interpreted by the ' - '"inspect" module as\n' - 'specifying the class where this object was defined ' - '(setting this\n' - 'appropriately can assist in runtime introspection of ' - 'dynamic class\n' - 'attributes). For callables, it may indicate that an ' - 'instance of the\n' - 'given type (or a subclass) is expected or required as ' - 'the first\n' - 'positional argument (for example, CPython sets this ' - 'attribute for\n' - 'unbound methods that are implemented in C).\n' - '\n' - '\n' - 'Invoking Descriptors\n' - '====================\n' - '\n' - 'In general, a descriptor is an object attribute with ' - '“binding\n' - 'behavior”, one whose attribute access has been ' - 'overridden by methods\n' - 'in the descriptor protocol: "__get__()", "__set__()", ' - 'and\n' - '"__delete__()". If any of those methods are defined for ' - 'an object, it\n' - 'is said to be a descriptor.\n' - '\n' - 'The default behavior for attribute access is to get, ' - 'set, or delete\n' - 'the attribute from an object’s dictionary. For instance, ' - '"a.x" has a\n' - 'lookup chain starting with "a.__dict__[\'x\']", then\n' - '"type(a).__dict__[\'x\']", and continuing through the ' - 'base classes of\n' - '"type(a)" excluding metaclasses.\n' - '\n' - 'However, if the looked-up value is an object defining ' - 'one of the\n' - 'descriptor methods, then Python may override the default ' - 'behavior and\n' - 'invoke the descriptor method instead. Where this occurs ' - 'in the\n' - 'precedence chain depends on which descriptor methods ' - 'were defined and\n' - 'how they were called.\n' - '\n' - 'The starting point for descriptor invocation is a ' - 'binding, "a.x". How\n' - 'the arguments are assembled depends on "a":\n' - '\n' - 'Direct Call\n' - ' The simplest and least common call is when user code ' - 'directly\n' - ' invokes a descriptor method: "x.__get__(a)".\n' - '\n' - 'Instance Binding\n' - ' If binding to an object instance, "a.x" is ' - 'transformed into the\n' - ' call: "type(a).__dict__[\'x\'].__get__(a, type(a))".\n' - '\n' - 'Class Binding\n' - ' If binding to a class, "A.x" is transformed into the ' - 'call:\n' - ' "A.__dict__[\'x\'].__get__(None, A)".\n' - '\n' - 'Super Binding\n' - ' If "a" is an instance of "super", then the binding ' - '"super(B,\n' - ' obj).m()" searches "obj.__class__.__mro__" for the ' - 'base class "A"\n' - ' immediately preceding "B" and then invokes the ' - 'descriptor with the\n' - ' call: "A.__dict__[\'m\'].__get__(obj, ' - 'obj.__class__)".\n' - '\n' - 'For instance bindings, the precedence of descriptor ' - 'invocation depends\n' - 'on the which descriptor methods are defined. A ' - 'descriptor can define\n' - 'any combination of "__get__()", "__set__()" and ' - '"__delete__()". If it\n' - 'does not define "__get__()", then accessing the ' - 'attribute will return\n' - 'the descriptor object itself unless there is a value in ' - 'the object’s\n' - 'instance dictionary. If the descriptor defines ' - '"__set__()" and/or\n' - '"__delete__()", it is a data descriptor; if it defines ' - 'neither, it is\n' - 'a non-data descriptor. Normally, data descriptors ' - 'define both\n' - '"__get__()" and "__set__()", while non-data descriptors ' - 'have just the\n' - '"__get__()" method. Data descriptors with "__set__()" ' - 'and "__get__()"\n' - 'defined always override a redefinition in an instance ' - 'dictionary. In\n' - 'contrast, non-data descriptors can be overridden by ' - 'instances.\n' - '\n' - 'Python methods (including "staticmethod()" and ' - '"classmethod()") are\n' - 'implemented as non-data descriptors. Accordingly, ' - 'instances can\n' - 'redefine and override methods. This allows individual ' - 'instances to\n' - 'acquire behaviors that differ from other instances of ' - 'the same class.\n' - '\n' - 'The "property()" function is implemented as a data ' - 'descriptor.\n' - 'Accordingly, instances cannot override the behavior of a ' - 'property.\n' - '\n' - '\n' - '__slots__\n' - '=========\n' - '\n' - '*__slots__* allow us to explicitly declare data members ' - '(like\n' - 'properties) and deny the creation of *__dict__* and ' - '*__weakref__*\n' - '(unless explicitly declared in *__slots__* or available ' - 'in a parent.)\n' - '\n' - 'The space saved over using *__dict__* can be ' - 'significant.\n' - '\n' - 'object.__slots__\n' - '\n' - ' This class variable can be assigned a string, ' - 'iterable, or sequence\n' - ' of strings with variable names used by instances. ' - '*__slots__*\n' - ' reserves space for the declared variables and ' - 'prevents the\n' - ' automatic creation of *__dict__* and *__weakref__* ' - 'for each\n' - ' instance.\n' - '\n' - '\n' - 'Notes on using *__slots__*\n' - '--------------------------\n' - '\n' - '* When inheriting from a class without *__slots__*, the ' - '*__dict__*\n' - ' and *__weakref__* attribute of the instances will ' - 'always be\n' - ' accessible.\n' - '\n' - '* Without a *__dict__* variable, instances cannot be ' - 'assigned new\n' - ' variables not listed in the *__slots__* definition. ' - 'Attempts to\n' - ' assign to an unlisted variable name raises ' - '"AttributeError". If\n' - ' dynamic assignment of new variables is desired, then ' - 'add\n' - ' "\'__dict__\'" to the sequence of strings in the ' - '*__slots__*\n' - ' declaration.\n' - '\n' - '* Without a *__weakref__* variable for each instance, ' - 'classes\n' - ' defining *__slots__* do not support weak references to ' - 'its\n' - ' instances. If weak reference support is needed, then ' - 'add\n' - ' "\'__weakref__\'" to the sequence of strings in the ' - '*__slots__*\n' - ' declaration.\n' - '\n' - '* *__slots__* are implemented at the class level by ' - 'creating\n' - ' descriptors (Implementing Descriptors) for each ' - 'variable name. As a\n' - ' result, class attributes cannot be used to set default ' - 'values for\n' - ' instance variables defined by *__slots__*; otherwise, ' - 'the class\n' - ' attribute would overwrite the descriptor assignment.\n' - '\n' - '* The action of a *__slots__* declaration is not limited ' - 'to the\n' - ' class where it is defined. *__slots__* declared in ' - 'parents are\n' - ' available in child classes. However, child subclasses ' - 'will get a\n' - ' *__dict__* and *__weakref__* unless they also define ' - '*__slots__*\n' - ' (which should only contain names of any *additional* ' - 'slots).\n' - '\n' - '* If a class defines a slot also defined in a base ' - 'class, the\n' - ' instance variable defined by the base class slot is ' - 'inaccessible\n' - ' (except by retrieving its descriptor directly from the ' - 'base class).\n' - ' This renders the meaning of the program undefined. In ' - 'the future, a\n' - ' check may be added to prevent this.\n' - '\n' - '* Nonempty *__slots__* does not work for classes derived ' - 'from\n' - ' “variable-length” built-in types such as "int", ' - '"bytes" and "tuple".\n' - '\n' - '* Any non-string iterable may be assigned to ' - '*__slots__*. Mappings\n' - ' may also be used; however, in the future, special ' - 'meaning may be\n' - ' assigned to the values corresponding to each key.\n' - '\n' - '* *__class__* assignment works only if both classes have ' - 'the same\n' - ' *__slots__*.\n' - '\n' - '* Multiple inheritance with multiple slotted parent ' - 'classes can be\n' - ' used, but only one parent is allowed to have ' - 'attributes created by\n' - ' slots (the other bases must have empty slot layouts) - ' - 'violations\n' - ' raise "TypeError".\n', - 'attribute-references': 'Attribute references\n' - '********************\n' - '\n' - 'An attribute reference is a primary followed by a ' - 'period and a name:\n' - '\n' - ' attributeref ::= primary "." identifier\n' - '\n' - 'The primary must evaluate to an object of a type ' - 'that supports\n' - 'attribute references, which most objects do. This ' - 'object is then\n' - 'asked to produce the attribute whose name is the ' - 'identifier. This\n' - 'production can be customized by overriding the ' - '"__getattr__()" method.\n' - 'If this attribute is not available, the exception ' - '"AttributeError" is\n' - 'raised. Otherwise, the type and value of the object ' - 'produced is\n' - 'determined by the object. Multiple evaluations of ' - 'the same attribute\n' - 'reference may yield different objects.\n', - 'augassign': 'Augmented assignment statements\n' - '*******************************\n' - '\n' - 'Augmented assignment is the combination, in a single statement, ' - 'of a\n' - 'binary operation and an assignment statement:\n' - '\n' - ' augmented_assignment_stmt ::= augtarget augop ' - '(expression_list | yield_expression)\n' - ' augtarget ::= identifier | attributeref | ' - 'subscription | slicing\n' - ' augop ::= "+=" | "-=" | "*=" | "@=" | ' - '"/=" | "//=" | "%=" | "**="\n' - ' | ">>=" | "<<=" | "&=" | "^=" | "|="\n' - '\n' - '(See section Primaries for the syntax definitions of the last ' - 'three\n' - 'symbols.)\n' - '\n' - 'An augmented assignment evaluates the target (which, unlike ' - 'normal\n' - 'assignment statements, cannot be an unpacking) and the ' - 'expression\n' - 'list, performs the binary operation specific to the type of ' - 'assignment\n' - 'on the two operands, and assigns the result to the original ' - 'target.\n' - 'The target is only evaluated once.\n' - '\n' - 'An augmented assignment expression like "x += 1" can be ' - 'rewritten as\n' - '"x = x + 1" to achieve a similar, but not exactly equal effect. ' - 'In the\n' - 'augmented version, "x" is only evaluated once. Also, when ' - 'possible,\n' - 'the actual operation is performed *in-place*, meaning that ' - 'rather than\n' - 'creating a new object and assigning that to the target, the old ' - 'object\n' - 'is modified instead.\n' - '\n' - 'Unlike normal assignments, augmented assignments evaluate the ' - 'left-\n' - 'hand side *before* evaluating the right-hand side. For ' - 'example, "a[i]\n' - '+= f(x)" first looks-up "a[i]", then it evaluates "f(x)" and ' - 'performs\n' - 'the addition, and lastly, it writes the result back to "a[i]".\n' - '\n' - 'With the exception of assigning to tuples and multiple targets ' - 'in a\n' - 'single statement, the assignment done by augmented assignment\n' - 'statements is handled the same way as normal assignments. ' - 'Similarly,\n' - 'with the exception of the possible *in-place* behavior, the ' - 'binary\n' - 'operation performed by augmented assignment is the same as the ' - 'normal\n' - 'binary operations.\n' - '\n' - 'For targets which are attribute references, the same caveat ' - 'about\n' - 'class and instance attributes applies as for regular ' - 'assignments.\n', - 'binary': 'Binary arithmetic operations\n' - '****************************\n' - '\n' - 'The binary arithmetic operations have the conventional priority\n' - 'levels. Note that some of these operations also apply to certain ' - 'non-\n' - 'numeric types. Apart from the power operator, there are only two\n' - 'levels, one for multiplicative operators and one for additive\n' - 'operators:\n' - '\n' - ' m_expr ::= u_expr | m_expr "*" u_expr | m_expr "@" m_expr |\n' - ' m_expr "//" u_expr | m_expr "/" u_expr |\n' - ' m_expr "%" u_expr\n' - ' a_expr ::= m_expr | a_expr "+" m_expr | a_expr "-" m_expr\n' - '\n' - 'The "*" (multiplication) operator yields the product of its ' - 'arguments.\n' - 'The arguments must either both be numbers, or one argument must be ' - 'an\n' - 'integer and the other must be a sequence. In the former case, the\n' - 'numbers are converted to a common type and then multiplied ' - 'together.\n' - 'In the latter case, sequence repetition is performed; a negative\n' - 'repetition factor yields an empty sequence.\n' - '\n' - 'The "@" (at) operator is intended to be used for matrix\n' - 'multiplication. No builtin Python types implement this operator.\n' - '\n' - 'New in version 3.5.\n' - '\n' - 'The "/" (division) and "//" (floor division) operators yield the\n' - 'quotient of their arguments. The numeric arguments are first\n' - 'converted to a common type. Division of integers yields a float, ' - 'while\n' - 'floor division of integers results in an integer; the result is ' - 'that\n' - 'of mathematical division with the ‘floor’ function applied to the\n' - 'result. Division by zero raises the "ZeroDivisionError" ' - 'exception.\n' - '\n' - 'The "%" (modulo) operator yields the remainder from the division ' - 'of\n' - 'the first argument by the second. The numeric arguments are ' - 'first\n' - 'converted to a common type. A zero right argument raises the\n' - '"ZeroDivisionError" exception. The arguments may be floating ' - 'point\n' - 'numbers, e.g., "3.14%0.7" equals "0.34" (since "3.14" equals ' - '"4*0.7 +\n' - '0.34".) The modulo operator always yields a result with the same ' - 'sign\n' - 'as its second operand (or zero); the absolute value of the result ' - 'is\n' - 'strictly smaller than the absolute value of the second operand ' - '[1].\n' - '\n' - 'The floor division and modulo operators are connected by the ' - 'following\n' - 'identity: "x == (x//y)*y + (x%y)". Floor division and modulo are ' - 'also\n' - 'connected with the built-in function "divmod()": "divmod(x, y) ==\n' - '(x//y, x%y)". [2].\n' - '\n' - 'In addition to performing the modulo operation on numbers, the ' - '"%"\n' - 'operator is also overloaded by string objects to perform ' - 'old-style\n' - 'string formatting (also known as interpolation). The syntax for\n' - 'string formatting is described in the Python Library Reference,\n' - 'section printf-style String Formatting.\n' - '\n' - 'The floor division operator, the modulo operator, and the ' - '"divmod()"\n' - 'function are not defined for complex numbers. Instead, convert to ' - 'a\n' - 'floating point number using the "abs()" function if appropriate.\n' - '\n' - 'The "+" (addition) operator yields the sum of its arguments. The\n' - 'arguments must either both be numbers or both be sequences of the ' - 'same\n' - 'type. In the former case, the numbers are converted to a common ' - 'type\n' - 'and then added together. In the latter case, the sequences are\n' - 'concatenated.\n' - '\n' - 'The "-" (subtraction) operator yields the difference of its ' - 'arguments.\n' - 'The numeric arguments are first converted to a common type.\n', - 'bitwise': 'Binary bitwise operations\n' - '*************************\n' - '\n' - 'Each of the three bitwise operations has a different priority ' - 'level:\n' - '\n' - ' and_expr ::= shift_expr | and_expr "&" shift_expr\n' - ' xor_expr ::= and_expr | xor_expr "^" and_expr\n' - ' or_expr ::= xor_expr | or_expr "|" xor_expr\n' - '\n' - 'The "&" operator yields the bitwise AND of its arguments, which ' - 'must\n' - 'be integers.\n' - '\n' - 'The "^" operator yields the bitwise XOR (exclusive OR) of its\n' - 'arguments, which must be integers.\n' - '\n' - 'The "|" operator yields the bitwise (inclusive) OR of its ' - 'arguments,\n' - 'which must be integers.\n', - 'bltin-code-objects': 'Code Objects\n' - '************\n' - '\n' - 'Code objects are used by the implementation to ' - 'represent “pseudo-\n' - 'compiled” executable Python code such as a function ' - 'body. They differ\n' - 'from function objects because they don’t contain a ' - 'reference to their\n' - 'global execution environment. Code objects are ' - 'returned by the built-\n' - 'in "compile()" function and can be extracted from ' - 'function objects\n' - 'through their "__code__" attribute. See also the ' - '"code" module.\n' - '\n' - 'A code object can be executed or evaluated by passing ' - 'it (instead of a\n' - 'source string) to the "exec()" or "eval()" built-in ' - 'functions.\n' - '\n' - 'See The standard type hierarchy for more ' - 'information.\n', - 'bltin-ellipsis-object': 'The Ellipsis Object\n' - '*******************\n' - '\n' - 'This object is commonly used by slicing (see ' - 'Slicings). It supports\n' - 'no special operations. There is exactly one ' - 'ellipsis object, named\n' - '"Ellipsis" (a built-in name). "type(Ellipsis)()" ' - 'produces the\n' - '"Ellipsis" singleton.\n' - '\n' - 'It is written as "Ellipsis" or "...".\n', - 'bltin-null-object': 'The Null Object\n' - '***************\n' - '\n' - 'This object is returned by functions that don’t ' - 'explicitly return a\n' - 'value. It supports no special operations. There is ' - 'exactly one null\n' - 'object, named "None" (a built-in name). "type(None)()" ' - 'produces the\n' - 'same singleton.\n' - '\n' - 'It is written as "None".\n', - 'bltin-type-objects': 'Type Objects\n' - '************\n' - '\n' - 'Type objects represent the various object types. An ' - 'object’s type is\n' - 'accessed by the built-in function "type()". There are ' - 'no special\n' - 'operations on types. The standard module "types" ' - 'defines names for\n' - 'all standard built-in types.\n' - '\n' - 'Types are written like this: "".\n', - 'booleans': 'Boolean operations\n' - '******************\n' - '\n' - ' or_test ::= and_test | or_test "or" and_test\n' - ' and_test ::= not_test | and_test "and" not_test\n' - ' not_test ::= comparison | "not" not_test\n' - '\n' - 'In the context of Boolean operations, and also when expressions ' - 'are\n' - 'used by control flow statements, the following values are ' - 'interpreted\n' - 'as false: "False", "None", numeric zero of all types, and empty\n' - 'strings and containers (including strings, tuples, lists,\n' - 'dictionaries, sets and frozensets). All other values are ' - 'interpreted\n' - 'as true. User-defined objects can customize their truth value ' - 'by\n' - 'providing a "__bool__()" method.\n' - '\n' - 'The operator "not" yields "True" if its argument is false, ' - '"False"\n' - 'otherwise.\n' - '\n' - 'The expression "x and y" first evaluates *x*; if *x* is false, ' - 'its\n' - 'value is returned; otherwise, *y* is evaluated and the resulting ' - 'value\n' - 'is returned.\n' - '\n' - 'The expression "x or y" first evaluates *x*; if *x* is true, its ' - 'value\n' - 'is returned; otherwise, *y* is evaluated and the resulting value ' - 'is\n' - 'returned.\n' - '\n' - 'Note that neither "and" nor "or" restrict the value and type ' - 'they\n' - 'return to "False" and "True", but rather return the last ' - 'evaluated\n' - 'argument. This is sometimes useful, e.g., if "s" is a string ' - 'that\n' - 'should be replaced by a default value if it is empty, the ' - 'expression\n' - '"s or \'foo\'" yields the desired value. Because "not" has to ' - 'create a\n' - 'new value, it returns a boolean value regardless of the type of ' - 'its\n' - 'argument (for example, "not \'foo\'" produces "False" rather ' - 'than "\'\'".)\n', - 'break': 'The "break" statement\n' - '*********************\n' - '\n' - ' break_stmt ::= "break"\n' - '\n' - '"break" may only occur syntactically nested in a "for" or "while"\n' - 'loop, but not nested in a function or class definition within that\n' - 'loop.\n' - '\n' - 'It terminates the nearest enclosing loop, skipping the optional ' - '"else"\n' - 'clause if the loop has one.\n' - '\n' - 'If a "for" loop is terminated by "break", the loop control target\n' - 'keeps its current value.\n' - '\n' - 'When "break" passes control out of a "try" statement with a ' - '"finally"\n' - 'clause, that "finally" clause is executed before really leaving ' - 'the\n' - 'loop.\n', - 'callable-types': 'Emulating callable objects\n' - '**************************\n' - '\n' - 'object.__call__(self[, args...])\n' - '\n' - ' Called when the instance is “called” as a function; if ' - 'this method\n' - ' is defined, "x(arg1, arg2, ...)" is a shorthand for\n' - ' "x.__call__(arg1, arg2, ...)".\n', - 'calls': 'Calls\n' - '*****\n' - '\n' - 'A call calls a callable object (e.g., a *function*) with a ' - 'possibly\n' - 'empty series of *arguments*:\n' - '\n' - ' call ::= primary "(" [argument_list [","] | ' - 'comprehension] ")"\n' - ' argument_list ::= positional_arguments ["," ' - 'starred_and_keywords]\n' - ' ["," keywords_arguments]\n' - ' | starred_and_keywords ["," ' - 'keywords_arguments]\n' - ' | keywords_arguments\n' - ' positional_arguments ::= ["*"] expression ("," ["*"] ' - 'expression)*\n' - ' starred_and_keywords ::= ("*" expression | keyword_item)\n' - ' ("," "*" expression | "," ' - 'keyword_item)*\n' - ' keywords_arguments ::= (keyword_item | "**" expression)\n' - ' ("," keyword_item | "," "**" ' - 'expression)*\n' - ' keyword_item ::= identifier "=" expression\n' - '\n' - 'An optional trailing comma may be present after the positional and\n' - 'keyword arguments but does not affect the semantics.\n' - '\n' - 'The primary must evaluate to a callable object (user-defined\n' - 'functions, built-in functions, methods of built-in objects, class\n' - 'objects, methods of class instances, and all objects having a\n' - '"__call__()" method are callable). All argument expressions are\n' - 'evaluated before the call is attempted. Please refer to section\n' - 'Function definitions for the syntax of formal *parameter* lists.\n' - '\n' - 'If keyword arguments are present, they are first converted to\n' - 'positional arguments, as follows. First, a list of unfilled slots ' - 'is\n' - 'created for the formal parameters. If there are N positional\n' - 'arguments, they are placed in the first N slots. Next, for each\n' - 'keyword argument, the identifier is used to determine the\n' - 'corresponding slot (if the identifier is the same as the first ' - 'formal\n' - 'parameter name, the first slot is used, and so on). If the slot ' - 'is\n' - 'already filled, a "TypeError" exception is raised. Otherwise, the\n' - 'value of the argument is placed in the slot, filling it (even if ' - 'the\n' - 'expression is "None", it fills the slot). When all arguments have\n' - 'been processed, the slots that are still unfilled are filled with ' - 'the\n' - 'corresponding default value from the function definition. ' - '(Default\n' - 'values are calculated, once, when the function is defined; thus, a\n' - 'mutable object such as a list or dictionary used as default value ' - 'will\n' - 'be shared by all calls that don’t specify an argument value for ' - 'the\n' - 'corresponding slot; this should usually be avoided.) If there are ' - 'any\n' - 'unfilled slots for which no default value is specified, a ' - '"TypeError"\n' - 'exception is raised. Otherwise, the list of filled slots is used ' - 'as\n' - 'the argument list for the call.\n' - '\n' - '**CPython implementation detail:** An implementation may provide\n' - 'built-in functions whose positional parameters do not have names, ' - 'even\n' - 'if they are ‘named’ for the purpose of documentation, and which\n' - 'therefore cannot be supplied by keyword. In CPython, this is the ' - 'case\n' - 'for functions implemented in C that use "PyArg_ParseTuple()" to ' - 'parse\n' - 'their arguments.\n' - '\n' - 'If there are more positional arguments than there are formal ' - 'parameter\n' - 'slots, a "TypeError" exception is raised, unless a formal ' - 'parameter\n' - 'using the syntax "*identifier" is present; in this case, that ' - 'formal\n' - 'parameter receives a tuple containing the excess positional ' - 'arguments\n' - '(or an empty tuple if there were no excess positional arguments).\n' - '\n' - 'If any keyword argument does not correspond to a formal parameter\n' - 'name, a "TypeError" exception is raised, unless a formal parameter\n' - 'using the syntax "**identifier" is present; in this case, that ' - 'formal\n' - 'parameter receives a dictionary containing the excess keyword\n' - 'arguments (using the keywords as keys and the argument values as\n' - 'corresponding values), or a (new) empty dictionary if there were ' - 'no\n' - 'excess keyword arguments.\n' - '\n' - 'If the syntax "*expression" appears in the function call, ' - '"expression"\n' - 'must evaluate to an *iterable*. Elements from these iterables are\n' - 'treated as if they were additional positional arguments. For the ' - 'call\n' - '"f(x1, x2, *y, x3, x4)", if *y* evaluates to a sequence *y1*, …, ' - '*yM*,\n' - 'this is equivalent to a call with M+4 positional arguments *x1*, ' - '*x2*,\n' - '*y1*, …, *yM*, *x3*, *x4*.\n' - '\n' - 'A consequence of this is that although the "*expression" syntax ' - 'may\n' - 'appear *after* explicit keyword arguments, it is processed ' - '*before*\n' - 'the keyword arguments (and any "**expression" arguments – see ' - 'below).\n' - 'So:\n' - '\n' - ' >>> def f(a, b):\n' - ' ... print(a, b)\n' - ' ...\n' - ' >>> f(b=1, *(2,))\n' - ' 2 1\n' - ' >>> f(a=1, *(2,))\n' - ' Traceback (most recent call last):\n' - ' File "", line 1, in \n' - " TypeError: f() got multiple values for keyword argument 'a'\n" - ' >>> f(1, *(2,))\n' - ' 1 2\n' - '\n' - 'It is unusual for both keyword arguments and the "*expression" ' - 'syntax\n' - 'to be used in the same call, so in practice this confusion does ' - 'not\n' - 'arise.\n' - '\n' - 'If the syntax "**expression" appears in the function call,\n' - '"expression" must evaluate to a *mapping*, the contents of which ' - 'are\n' - 'treated as additional keyword arguments. If a keyword is already\n' - 'present (as an explicit keyword argument, or from another ' - 'unpacking),\n' - 'a "TypeError" exception is raised.\n' - '\n' - 'Formal parameters using the syntax "*identifier" or "**identifier"\n' - 'cannot be used as positional argument slots or as keyword argument\n' - 'names.\n' - '\n' - 'Changed in version 3.5: Function calls accept any number of "*" ' - 'and\n' - '"**" unpackings, positional arguments may follow iterable ' - 'unpackings\n' - '("*"), and keyword arguments may follow dictionary unpackings ' - '("**").\n' - 'Originally proposed by **PEP 448**.\n' - '\n' - 'A call always returns some value, possibly "None", unless it raises ' - 'an\n' - 'exception. How this value is computed depends on the type of the\n' - 'callable object.\n' - '\n' - 'If it is—\n' - '\n' - 'a user-defined function:\n' - ' The code block for the function is executed, passing it the\n' - ' argument list. The first thing the code block will do is bind ' - 'the\n' - ' formal parameters to the arguments; this is described in ' - 'section\n' - ' Function definitions. When the code block executes a "return"\n' - ' statement, this specifies the return value of the function ' - 'call.\n' - '\n' - 'a built-in function or method:\n' - ' The result is up to the interpreter; see Built-in Functions for ' - 'the\n' - ' descriptions of built-in functions and methods.\n' - '\n' - 'a class object:\n' - ' A new instance of that class is returned.\n' - '\n' - 'a class instance method:\n' - ' The corresponding user-defined function is called, with an ' - 'argument\n' - ' list that is one longer than the argument list of the call: the\n' - ' instance becomes the first argument.\n' - '\n' - 'a class instance:\n' - ' The class must define a "__call__()" method; the effect is then ' - 'the\n' - ' same as if that method was called.\n', - 'class': 'Class definitions\n' - '*****************\n' - '\n' - 'A class definition defines a class object (see section The ' - 'standard\n' - 'type hierarchy):\n' - '\n' - ' classdef ::= [decorators] "class" classname [inheritance] ":" ' - 'suite\n' - ' inheritance ::= "(" [argument_list] ")"\n' - ' classname ::= identifier\n' - '\n' - 'A class definition is an executable statement. The inheritance ' - 'list\n' - 'usually gives a list of base classes (see Metaclasses for more\n' - 'advanced uses), so each item in the list should evaluate to a ' - 'class\n' - 'object which allows subclassing. Classes without an inheritance ' - 'list\n' - 'inherit, by default, from the base class "object"; hence,\n' - '\n' - ' class Foo:\n' - ' pass\n' - '\n' - 'is equivalent to\n' - '\n' - ' class Foo(object):\n' - ' pass\n' - '\n' - 'The class’s suite is then executed in a new execution frame (see\n' - 'Naming and binding), using a newly created local namespace and the\n' - 'original global namespace. (Usually, the suite contains mostly\n' - 'function definitions.) When the class’s suite finishes execution, ' - 'its\n' - 'execution frame is discarded but its local namespace is saved. [3] ' - 'A\n' - 'class object is then created using the inheritance list for the ' - 'base\n' - 'classes and the saved local namespace for the attribute ' - 'dictionary.\n' - 'The class name is bound to this class object in the original local\n' - 'namespace.\n' - '\n' - 'The order in which attributes are defined in the class body is\n' - 'preserved in the new class’s "__dict__". Note that this is ' - 'reliable\n' - 'only right after the class is created and only for classes that ' - 'were\n' - 'defined using the definition syntax.\n' - '\n' - 'Class creation can be customized heavily using metaclasses.\n' - '\n' - 'Classes can also be decorated: just like when decorating ' - 'functions,\n' - '\n' - ' @f1(arg)\n' - ' @f2\n' - ' class Foo: pass\n' - '\n' - 'is roughly equivalent to\n' - '\n' - ' class Foo: pass\n' - ' Foo = f1(arg)(f2(Foo))\n' - '\n' - 'The evaluation rules for the decorator expressions are the same as ' - 'for\n' - 'function decorators. The result is then bound to the class name.\n' - '\n' - '**Programmer’s note:** Variables defined in the class definition ' - 'are\n' - 'class attributes; they are shared by instances. Instance ' - 'attributes\n' - 'can be set in a method with "self.name = value". Both class and\n' - 'instance attributes are accessible through the notation ' - '“"self.name"”,\n' - 'and an instance attribute hides a class attribute with the same ' - 'name\n' - 'when accessed in this way. Class attributes can be used as ' - 'defaults\n' - 'for instance attributes, but using mutable values there can lead ' - 'to\n' - 'unexpected results. Descriptors can be used to create instance\n' - 'variables with different implementation details.\n' - '\n' - 'See also:\n' - '\n' - ' **PEP 3115** - Metaclasses in Python 3000\n' - ' The proposal that changed the declaration of metaclasses to ' - 'the\n' - ' current syntax, and the semantics for how classes with\n' - ' metaclasses are constructed.\n' - '\n' - ' **PEP 3129** - Class Decorators\n' - ' The proposal that added class decorators. Function and ' - 'method\n' - ' decorators were introduced in **PEP 318**.\n', - 'comparisons': 'Comparisons\n' - '***********\n' - '\n' - 'Unlike C, all comparison operations in Python have the same ' - 'priority,\n' - 'which is lower than that of any arithmetic, shifting or ' - 'bitwise\n' - 'operation. Also unlike C, expressions like "a < b < c" have ' - 'the\n' - 'interpretation that is conventional in mathematics:\n' - '\n' - ' comparison ::= or_expr (comp_operator or_expr)*\n' - ' comp_operator ::= "<" | ">" | "==" | ">=" | "<=" | "!="\n' - ' | "is" ["not"] | ["not"] "in"\n' - '\n' - 'Comparisons yield boolean values: "True" or "False".\n' - '\n' - 'Comparisons can be chained arbitrarily, e.g., "x < y <= z" ' - 'is\n' - 'equivalent to "x < y and y <= z", except that "y" is ' - 'evaluated only\n' - 'once (but in both cases "z" is not evaluated at all when "x < ' - 'y" is\n' - 'found to be false).\n' - '\n' - 'Formally, if *a*, *b*, *c*, …, *y*, *z* are expressions and ' - '*op1*,\n' - '*op2*, …, *opN* are comparison operators, then "a op1 b op2 c ' - '... y\n' - 'opN z" is equivalent to "a op1 b and b op2 c and ... y opN ' - 'z", except\n' - 'that each expression is evaluated at most once.\n' - '\n' - 'Note that "a op1 b op2 c" doesn’t imply any kind of ' - 'comparison between\n' - '*a* and *c*, so that, e.g., "x < y > z" is perfectly legal ' - '(though\n' - 'perhaps not pretty).\n' - '\n' - '\n' - 'Value comparisons\n' - '=================\n' - '\n' - 'The operators "<", ">", "==", ">=", "<=", and "!=" compare ' - 'the values\n' - 'of two objects. The objects do not need to have the same ' - 'type.\n' - '\n' - 'Chapter Objects, values and types states that objects have a ' - 'value (in\n' - 'addition to type and identity). The value of an object is a ' - 'rather\n' - 'abstract notion in Python: For example, there is no canonical ' - 'access\n' - 'method for an object’s value. Also, there is no requirement ' - 'that the\n' - 'value of an object should be constructed in a particular way, ' - 'e.g.\n' - 'comprised of all its data attributes. Comparison operators ' - 'implement a\n' - 'particular notion of what the value of an object is. One can ' - 'think of\n' - 'them as defining the value of an object indirectly, by means ' - 'of their\n' - 'comparison implementation.\n' - '\n' - 'Because all types are (direct or indirect) subtypes of ' - '"object", they\n' - 'inherit the default comparison behavior from "object". Types ' - 'can\n' - 'customize their comparison behavior by implementing *rich ' - 'comparison\n' - 'methods* like "__lt__()", described in Basic customization.\n' - '\n' - 'The default behavior for equality comparison ("==" and "!=") ' - 'is based\n' - 'on the identity of the objects. Hence, equality comparison ' - 'of\n' - 'instances with the same identity results in equality, and ' - 'equality\n' - 'comparison of instances with different identities results in\n' - 'inequality. A motivation for this default behavior is the ' - 'desire that\n' - 'all objects should be reflexive (i.e. "x is y" implies "x == ' - 'y").\n' - '\n' - 'A default order comparison ("<", ">", "<=", and ">=") is not ' - 'provided;\n' - 'an attempt raises "TypeError". A motivation for this default ' - 'behavior\n' - 'is the lack of a similar invariant as for equality.\n' - '\n' - 'The behavior of the default equality comparison, that ' - 'instances with\n' - 'different identities are always unequal, may be in contrast ' - 'to what\n' - 'types will need that have a sensible definition of object ' - 'value and\n' - 'value-based equality. Such types will need to customize ' - 'their\n' - 'comparison behavior, and in fact, a number of built-in types ' - 'have done\n' - 'that.\n' - '\n' - 'The following list describes the comparison behavior of the ' - 'most\n' - 'important built-in types.\n' - '\n' - '* Numbers of built-in numeric types (Numeric Types — int, ' - 'float,\n' - ' complex) and of the standard library types ' - '"fractions.Fraction" and\n' - ' "decimal.Decimal" can be compared within and across their ' - 'types,\n' - ' with the restriction that complex numbers do not support ' - 'order\n' - ' comparison. Within the limits of the types involved, they ' - 'compare\n' - ' mathematically (algorithmically) correct without loss of ' - 'precision.\n' - '\n' - ' The not-a-number values "float(\'NaN\')" and ' - '"Decimal(\'NaN\')" are\n' - ' special. They are identical to themselves ("x is x" is ' - 'true) but\n' - ' are not equal to themselves ("x == x" is false). ' - 'Additionally,\n' - ' comparing any number to a not-a-number value will return ' - '"False".\n' - ' For example, both "3 < float(\'NaN\')" and "float(\'NaN\') ' - '< 3" will\n' - ' return "False".\n' - '\n' - '* Binary sequences (instances of "bytes" or "bytearray") can ' - 'be\n' - ' compared within and across their types. They compare\n' - ' lexicographically using the numeric values of their ' - 'elements.\n' - '\n' - '* Strings (instances of "str") compare lexicographically ' - 'using the\n' - ' numerical Unicode code points (the result of the built-in ' - 'function\n' - ' "ord()") of their characters. [3]\n' - '\n' - ' Strings and binary sequences cannot be directly compared.\n' - '\n' - '* Sequences (instances of "tuple", "list", or "range") can ' - 'be\n' - ' compared only within each of their types, with the ' - 'restriction that\n' - ' ranges do not support order comparison. Equality ' - 'comparison across\n' - ' these types results in inequality, and ordering comparison ' - 'across\n' - ' these types raises "TypeError".\n' - '\n' - ' Sequences compare lexicographically using comparison of\n' - ' corresponding elements, whereby reflexivity of the elements ' - 'is\n' - ' enforced.\n' - '\n' - ' In enforcing reflexivity of elements, the comparison of ' - 'collections\n' - ' assumes that for a collection element "x", "x == x" is ' - 'always true.\n' - ' Based on that assumption, element identity is compared ' - 'first, and\n' - ' element comparison is performed only for distinct ' - 'elements. This\n' - ' approach yields the same result as a strict element ' - 'comparison\n' - ' would, if the compared elements are reflexive. For ' - 'non-reflexive\n' - ' elements, the result is different than for strict element\n' - ' comparison, and may be surprising: The non-reflexive ' - 'not-a-number\n' - ' values for example result in the following comparison ' - 'behavior when\n' - ' used in a list:\n' - '\n' - " >>> nan = float('NaN')\n" - ' >>> nan is nan\n' - ' True\n' - ' >>> nan == nan\n' - ' False <-- the defined non-reflexive ' - 'behavior of NaN\n' - ' >>> [nan] == [nan]\n' - ' True <-- list enforces reflexivity and ' - 'tests identity first\n' - '\n' - ' Lexicographical comparison between built-in collections ' - 'works as\n' - ' follows:\n' - '\n' - ' * For two collections to compare equal, they must be of the ' - 'same\n' - ' type, have the same length, and each pair of ' - 'corresponding\n' - ' elements must compare equal (for example, "[1,2] == ' - '(1,2)" is\n' - ' false because the type is not the same).\n' - '\n' - ' * Collections that support order comparison are ordered the ' - 'same\n' - ' as their first unequal elements (for example, "[1,2,x] <= ' - '[1,2,y]"\n' - ' has the same value as "x <= y"). If a corresponding ' - 'element does\n' - ' not exist, the shorter collection is ordered first (for ' - 'example,\n' - ' "[1,2] < [1,2,3]" is true).\n' - '\n' - '* Mappings (instances of "dict") compare equal if and only if ' - 'they\n' - ' have equal *(key, value)* pairs. Equality comparison of the ' - 'keys and\n' - ' values enforces reflexivity.\n' - '\n' - ' Order comparisons ("<", ">", "<=", and ">=") raise ' - '"TypeError".\n' - '\n' - '* Sets (instances of "set" or "frozenset") can be compared ' - 'within\n' - ' and across their types.\n' - '\n' - ' They define order comparison operators to mean subset and ' - 'superset\n' - ' tests. Those relations do not define total orderings (for ' - 'example,\n' - ' the two sets "{1,2}" and "{2,3}" are not equal, nor subsets ' - 'of one\n' - ' another, nor supersets of one another). Accordingly, sets ' - 'are not\n' - ' appropriate arguments for functions which depend on total ' - 'ordering\n' - ' (for example, "min()", "max()", and "sorted()" produce ' - 'undefined\n' - ' results given a list of sets as inputs).\n' - '\n' - ' Comparison of sets enforces reflexivity of its elements.\n' - '\n' - '* Most other built-in types have no comparison methods ' - 'implemented,\n' - ' so they inherit the default comparison behavior.\n' - '\n' - 'User-defined classes that customize their comparison behavior ' - 'should\n' - 'follow some consistency rules, if possible:\n' - '\n' - '* Equality comparison should be reflexive. In other words, ' - 'identical\n' - ' objects should compare equal:\n' - '\n' - ' "x is y" implies "x == y"\n' - '\n' - '* Comparison should be symmetric. In other words, the ' - 'following\n' - ' expressions should have the same result:\n' - '\n' - ' "x == y" and "y == x"\n' - '\n' - ' "x != y" and "y != x"\n' - '\n' - ' "x < y" and "y > x"\n' - '\n' - ' "x <= y" and "y >= x"\n' - '\n' - '* Comparison should be transitive. The following ' - '(non-exhaustive)\n' - ' examples illustrate that:\n' - '\n' - ' "x > y and y > z" implies "x > z"\n' - '\n' - ' "x < y and y <= z" implies "x < z"\n' - '\n' - '* Inverse comparison should result in the boolean negation. ' - 'In other\n' - ' words, the following expressions should have the same ' - 'result:\n' - '\n' - ' "x == y" and "not x != y"\n' - '\n' - ' "x < y" and "not x >= y" (for total ordering)\n' - '\n' - ' "x > y" and "not x <= y" (for total ordering)\n' - '\n' - ' The last two expressions apply to totally ordered ' - 'collections (e.g.\n' - ' to sequences, but not to sets or mappings). See also the\n' - ' "total_ordering()" decorator.\n' - '\n' - '* The "hash()" result should be consistent with equality. ' - 'Objects\n' - ' that are equal should either have the same hash value, or ' - 'be marked\n' - ' as unhashable.\n' - '\n' - 'Python does not enforce these consistency rules. In fact, ' - 'the\n' - 'not-a-number values are an example for not following these ' - 'rules.\n' - '\n' - '\n' - 'Membership test operations\n' - '==========================\n' - '\n' - 'The operators "in" and "not in" test for membership. "x in ' - 's"\n' - 'evaluates to "True" if *x* is a member of *s*, and "False" ' - 'otherwise.\n' - '"x not in s" returns the negation of "x in s". All built-in ' - 'sequences\n' - 'and set types support this as well as dictionary, for which ' - '"in" tests\n' - 'whether the dictionary has a given key. For container types ' - 'such as\n' - 'list, tuple, set, frozenset, dict, or collections.deque, the\n' - 'expression "x in y" is equivalent to "any(x is e or x == e ' - 'for e in\n' - 'y)".\n' - '\n' - 'For the string and bytes types, "x in y" is "True" if and ' - 'only if *x*\n' - 'is a substring of *y*. An equivalent test is "y.find(x) != ' - '-1".\n' - 'Empty strings are always considered to be a substring of any ' - 'other\n' - 'string, so """ in "abc"" will return "True".\n' - '\n' - 'For user-defined classes which define the "__contains__()" ' - 'method, "x\n' - 'in y" returns "True" if "y.__contains__(x)" returns a true ' - 'value, and\n' - '"False" otherwise.\n' - '\n' - 'For user-defined classes which do not define "__contains__()" ' - 'but do\n' - 'define "__iter__()", "x in y" is "True" if some value "z" ' - 'with "x ==\n' - 'z" is produced while iterating over "y". If an exception is ' - 'raised\n' - 'during the iteration, it is as if "in" raised that ' - 'exception.\n' - '\n' - 'Lastly, the old-style iteration protocol is tried: if a class ' - 'defines\n' - '"__getitem__()", "x in y" is "True" if and only if there is a ' - 'non-\n' - 'negative integer index *i* such that "x == y[i]", and all ' - 'lower\n' - 'integer indices do not raise "IndexError" exception. (If any ' - 'other\n' - 'exception is raised, it is as if "in" raised that ' - 'exception).\n' - '\n' - 'The operator "not in" is defined to have the inverse true ' - 'value of\n' - '"in".\n' - '\n' - '\n' - 'Identity comparisons\n' - '====================\n' - '\n' - 'The operators "is" and "is not" test for object identity: "x ' - 'is y" is\n' - 'true if and only if *x* and *y* are the same object. Object ' - 'identity\n' - 'is determined using the "id()" function. "x is not y" yields ' - 'the\n' - 'inverse truth value. [4]\n', - 'compound': 'Compound statements\n' - '*******************\n' - '\n' - 'Compound statements contain (groups of) other statements; they ' - 'affect\n' - 'or control the execution of those other statements in some way. ' - 'In\n' - 'general, compound statements span multiple lines, although in ' - 'simple\n' - 'incarnations a whole compound statement may be contained in one ' - 'line.\n' - '\n' - 'The "if", "while" and "for" statements implement traditional ' - 'control\n' - 'flow constructs. "try" specifies exception handlers and/or ' - 'cleanup\n' - 'code for a group of statements, while the "with" statement ' - 'allows the\n' - 'execution of initialization and finalization code around a block ' - 'of\n' - 'code. Function and class definitions are also syntactically ' - 'compound\n' - 'statements.\n' - '\n' - 'A compound statement consists of one or more ‘clauses.’ A ' - 'clause\n' - 'consists of a header and a ‘suite.’ The clause headers of a\n' - 'particular compound statement are all at the same indentation ' - 'level.\n' - 'Each clause header begins with a uniquely identifying keyword ' - 'and ends\n' - 'with a colon. A suite is a group of statements controlled by a\n' - 'clause. A suite can be one or more semicolon-separated simple\n' - 'statements on the same line as the header, following the ' - 'header’s\n' - 'colon, or it can be one or more indented statements on ' - 'subsequent\n' - 'lines. Only the latter form of a suite can contain nested ' - 'compound\n' - 'statements; the following is illegal, mostly because it wouldn’t ' - 'be\n' - 'clear to which "if" clause a following "else" clause would ' - 'belong:\n' - '\n' - ' if test1: if test2: print(x)\n' - '\n' - 'Also note that the semicolon binds tighter than the colon in ' - 'this\n' - 'context, so that in the following example, either all or none of ' - 'the\n' - '"print()" calls are executed:\n' - '\n' - ' if x < y < z: print(x); print(y); print(z)\n' - '\n' - 'Summarizing:\n' - '\n' - ' compound_stmt ::= if_stmt\n' - ' | while_stmt\n' - ' | for_stmt\n' - ' | try_stmt\n' - ' | with_stmt\n' - ' | funcdef\n' - ' | classdef\n' - ' | async_with_stmt\n' - ' | async_for_stmt\n' - ' | async_funcdef\n' - ' suite ::= stmt_list NEWLINE | NEWLINE INDENT ' - 'statement+ DEDENT\n' - ' statement ::= stmt_list NEWLINE | compound_stmt\n' - ' stmt_list ::= simple_stmt (";" simple_stmt)* [";"]\n' - '\n' - 'Note that statements always end in a "NEWLINE" possibly followed ' - 'by a\n' - '"DEDENT". Also note that optional continuation clauses always ' - 'begin\n' - 'with a keyword that cannot start a statement, thus there are no\n' - 'ambiguities (the ‘dangling "else"’ problem is solved in Python ' - 'by\n' - 'requiring nested "if" statements to be indented).\n' - '\n' - 'The formatting of the grammar rules in the following sections ' - 'places\n' - 'each clause on a separate line for clarity.\n' - '\n' - '\n' - 'The "if" statement\n' - '==================\n' - '\n' - 'The "if" statement is used for conditional execution:\n' - '\n' - ' if_stmt ::= "if" expression ":" suite\n' - ' ("elif" expression ":" suite)*\n' - ' ["else" ":" suite]\n' - '\n' - 'It selects exactly one of the suites by evaluating the ' - 'expressions one\n' - 'by one until one is found to be true (see section Boolean ' - 'operations\n' - 'for the definition of true and false); then that suite is ' - 'executed\n' - '(and no other part of the "if" statement is executed or ' - 'evaluated).\n' - 'If all expressions are false, the suite of the "else" clause, ' - 'if\n' - 'present, is executed.\n' - '\n' - '\n' - 'The "while" statement\n' - '=====================\n' - '\n' - 'The "while" statement is used for repeated execution as long as ' - 'an\n' - 'expression is true:\n' - '\n' - ' while_stmt ::= "while" expression ":" suite\n' - ' ["else" ":" suite]\n' - '\n' - 'This repeatedly tests the expression and, if it is true, ' - 'executes the\n' - 'first suite; if the expression is false (which may be the first ' - 'time\n' - 'it is tested) the suite of the "else" clause, if present, is ' - 'executed\n' - 'and the loop terminates.\n' - '\n' - 'A "break" statement executed in the first suite terminates the ' - 'loop\n' - 'without executing the "else" clause’s suite. A "continue" ' - 'statement\n' - 'executed in the first suite skips the rest of the suite and goes ' - 'back\n' - 'to testing the expression.\n' - '\n' - '\n' - 'The "for" statement\n' - '===================\n' - '\n' - 'The "for" statement is used to iterate over the elements of a ' - 'sequence\n' - '(such as a string, tuple or list) or other iterable object:\n' - '\n' - ' for_stmt ::= "for" target_list "in" expression_list ":" ' - 'suite\n' - ' ["else" ":" suite]\n' - '\n' - 'The expression list is evaluated once; it should yield an ' - 'iterable\n' - 'object. An iterator is created for the result of the\n' - '"expression_list". The suite is then executed once for each ' - 'item\n' - 'provided by the iterator, in the order returned by the ' - 'iterator. Each\n' - 'item in turn is assigned to the target list using the standard ' - 'rules\n' - 'for assignments (see Assignment statements), and then the suite ' - 'is\n' - 'executed. When the items are exhausted (which is immediately ' - 'when the\n' - 'sequence is empty or an iterator raises a "StopIteration" ' - 'exception),\n' - 'the suite in the "else" clause, if present, is executed, and the ' - 'loop\n' - 'terminates.\n' - '\n' - 'A "break" statement executed in the first suite terminates the ' - 'loop\n' - 'without executing the "else" clause’s suite. A "continue" ' - 'statement\n' - 'executed in the first suite skips the rest of the suite and ' - 'continues\n' - 'with the next item, or with the "else" clause if there is no ' - 'next\n' - 'item.\n' - '\n' - 'The for-loop makes assignments to the variables(s) in the target ' - 'list.\n' - 'This overwrites all previous assignments to those variables ' - 'including\n' - 'those made in the suite of the for-loop:\n' - '\n' - ' for i in range(10):\n' - ' print(i)\n' - ' i = 5 # this will not affect the for-loop\n' - ' # because i will be overwritten with ' - 'the next\n' - ' # index in the range\n' - '\n' - 'Names in the target list are not deleted when the loop is ' - 'finished,\n' - 'but if the sequence is empty, they will not have been assigned ' - 'to at\n' - 'all by the loop. Hint: the built-in function "range()" returns ' - 'an\n' - 'iterator of integers suitable to emulate the effect of Pascal’s ' - '"for i\n' - ':= a to b do"; e.g., "list(range(3))" returns the list "[0, 1, ' - '2]".\n' - '\n' - 'Note: There is a subtlety when the sequence is being modified by ' - 'the\n' - ' loop (this can only occur for mutable sequences, e.g. lists). ' - 'An\n' - ' internal counter is used to keep track of which item is used ' - 'next,\n' - ' and this is incremented on each iteration. When this counter ' - 'has\n' - ' reached the length of the sequence the loop terminates. This ' - 'means\n' - ' that if the suite deletes the current (or a previous) item ' - 'from the\n' - ' sequence, the next item will be skipped (since it gets the ' - 'index of\n' - ' the current item which has already been treated). Likewise, ' - 'if the\n' - ' suite inserts an item in the sequence before the current item, ' - 'the\n' - ' current item will be treated again the next time through the ' - 'loop.\n' - ' This can lead to nasty bugs that can be avoided by making a\n' - ' temporary copy using a slice of the whole sequence, e.g.,\n' - '\n' - ' for x in a[:]:\n' - ' if x < 0: a.remove(x)\n' - '\n' - '\n' - 'The "try" statement\n' - '===================\n' - '\n' - 'The "try" statement specifies exception handlers and/or cleanup ' - 'code\n' - 'for a group of statements:\n' - '\n' - ' try_stmt ::= try1_stmt | try2_stmt\n' - ' try1_stmt ::= "try" ":" suite\n' - ' ("except" [expression ["as" identifier]] ":" ' - 'suite)+\n' - ' ["else" ":" suite]\n' - ' ["finally" ":" suite]\n' - ' try2_stmt ::= "try" ":" suite\n' - ' "finally" ":" suite\n' - '\n' - 'The "except" clause(s) specify one or more exception handlers. ' - 'When no\n' - 'exception occurs in the "try" clause, no exception handler is\n' - 'executed. When an exception occurs in the "try" suite, a search ' - 'for an\n' - 'exception handler is started. This search inspects the except ' - 'clauses\n' - 'in turn until one is found that matches the exception. An ' - 'expression-\n' - 'less except clause, if present, must be last; it matches any\n' - 'exception. For an except clause with an expression, that ' - 'expression\n' - 'is evaluated, and the clause matches the exception if the ' - 'resulting\n' - 'object is “compatible” with the exception. An object is ' - 'compatible\n' - 'with an exception if it is the class or a base class of the ' - 'exception\n' - 'object or a tuple containing an item compatible with the ' - 'exception.\n' - '\n' - 'If no except clause matches the exception, the search for an ' - 'exception\n' - 'handler continues in the surrounding code and on the invocation ' - 'stack.\n' - '[1]\n' - '\n' - 'If the evaluation of an expression in the header of an except ' - 'clause\n' - 'raises an exception, the original search for a handler is ' - 'canceled and\n' - 'a search starts for the new exception in the surrounding code ' - 'and on\n' - 'the call stack (it is treated as if the entire "try" statement ' - 'raised\n' - 'the exception).\n' - '\n' - 'When a matching except clause is found, the exception is ' - 'assigned to\n' - 'the target specified after the "as" keyword in that except ' - 'clause, if\n' - 'present, and the except clause’s suite is executed. All except\n' - 'clauses must have an executable block. When the end of this ' - 'block is\n' - 'reached, execution continues normally after the entire try ' - 'statement.\n' - '(This means that if two nested handlers exist for the same ' - 'exception,\n' - 'and the exception occurs in the try clause of the inner handler, ' - 'the\n' - 'outer handler will not handle the exception.)\n' - '\n' - 'When an exception has been assigned using "as target", it is ' - 'cleared\n' - 'at the end of the except clause. This is as if\n' - '\n' - ' except E as N:\n' - ' foo\n' - '\n' - 'was translated to\n' - '\n' - ' except E as N:\n' - ' try:\n' - ' foo\n' - ' finally:\n' - ' del N\n' - '\n' - 'This means the exception must be assigned to a different name to ' - 'be\n' - 'able to refer to it after the except clause. Exceptions are ' - 'cleared\n' - 'because with the traceback attached to them, they form a ' - 'reference\n' - 'cycle with the stack frame, keeping all locals in that frame ' - 'alive\n' - 'until the next garbage collection occurs.\n' - '\n' - 'Before an except clause’s suite is executed, details about the\n' - 'exception are stored in the "sys" module and can be accessed ' - 'via\n' - '"sys.exc_info()". "sys.exc_info()" returns a 3-tuple consisting ' - 'of the\n' - 'exception class, the exception instance and a traceback object ' - '(see\n' - 'section The standard type hierarchy) identifying the point in ' - 'the\n' - 'program where the exception occurred. "sys.exc_info()" values ' - 'are\n' - 'restored to their previous values (before the call) when ' - 'returning\n' - 'from a function that handled an exception.\n' - '\n' - 'The optional "else" clause is executed if the control flow ' - 'leaves the\n' - '"try" suite, no exception was raised, and no "return", ' - '"continue", or\n' - '"break" statement was executed. Exceptions in the "else" clause ' - 'are\n' - 'not handled by the preceding "except" clauses.\n' - '\n' - 'If "finally" is present, it specifies a ‘cleanup’ handler. The ' - '"try"\n' - 'clause is executed, including any "except" and "else" clauses. ' - 'If an\n' - 'exception occurs in any of the clauses and is not handled, the\n' - 'exception is temporarily saved. The "finally" clause is ' - 'executed. If\n' - 'there is a saved exception it is re-raised at the end of the ' - '"finally"\n' - 'clause. If the "finally" clause raises another exception, the ' - 'saved\n' - 'exception is set as the context of the new exception. If the ' - '"finally"\n' - 'clause executes a "return" or "break" statement, the saved ' - 'exception\n' - 'is discarded:\n' - '\n' - ' >>> def f():\n' - ' ... try:\n' - ' ... 1/0\n' - ' ... finally:\n' - ' ... return 42\n' - ' ...\n' - ' >>> f()\n' - ' 42\n' - '\n' - 'The exception information is not available to the program ' - 'during\n' - 'execution of the "finally" clause.\n' - '\n' - 'When a "return", "break" or "continue" statement is executed in ' - 'the\n' - '"try" suite of a "try"…"finally" statement, the "finally" clause ' - 'is\n' - 'also executed ‘on the way out.’ A "continue" statement is ' - 'illegal in\n' - 'the "finally" clause. (The reason is a problem with the current\n' - 'implementation — this restriction may be lifted in the future).\n' - '\n' - 'The return value of a function is determined by the last ' - '"return"\n' - 'statement executed. Since the "finally" clause always executes, ' - 'a\n' - '"return" statement executed in the "finally" clause will always ' - 'be the\n' - 'last one executed:\n' - '\n' - ' >>> def foo():\n' - ' ... try:\n' - " ... return 'try'\n" - ' ... finally:\n' - " ... return 'finally'\n" - ' ...\n' - ' >>> foo()\n' - " 'finally'\n" - '\n' - 'Additional information on exceptions can be found in section\n' - 'Exceptions, and information on using the "raise" statement to ' - 'generate\n' - 'exceptions may be found in section The raise statement.\n' - '\n' - '\n' - 'The "with" statement\n' - '====================\n' - '\n' - 'The "with" statement is used to wrap the execution of a block ' - 'with\n' - 'methods defined by a context manager (see section With ' - 'Statement\n' - 'Context Managers). This allows common "try"…"except"…"finally" ' - 'usage\n' - 'patterns to be encapsulated for convenient reuse.\n' - '\n' - ' with_stmt ::= "with" with_item ("," with_item)* ":" suite\n' - ' with_item ::= expression ["as" target]\n' - '\n' - 'The execution of the "with" statement with one “item” proceeds ' - 'as\n' - 'follows:\n' - '\n' - '1. The context expression (the expression given in the ' - '"with_item")\n' - ' is evaluated to obtain a context manager.\n' - '\n' - '2. The context manager’s "__exit__()" is loaded for later use.\n' - '\n' - '3. The context manager’s "__enter__()" method is invoked.\n' - '\n' - '4. If a target was included in the "with" statement, the return\n' - ' value from "__enter__()" is assigned to it.\n' - '\n' - ' Note: The "with" statement guarantees that if the ' - '"__enter__()"\n' - ' method returns without an error, then "__exit__()" will ' - 'always be\n' - ' called. Thus, if an error occurs during the assignment to ' - 'the\n' - ' target list, it will be treated the same as an error ' - 'occurring\n' - ' within the suite would be. See step 6 below.\n' - '\n' - '5. The suite is executed.\n' - '\n' - '6. The context manager’s "__exit__()" method is invoked. If an\n' - ' exception caused the suite to be exited, its type, value, ' - 'and\n' - ' traceback are passed as arguments to "__exit__()". Otherwise, ' - 'three\n' - ' "None" arguments are supplied.\n' - '\n' - ' If the suite was exited due to an exception, and the return ' - 'value\n' - ' from the "__exit__()" method was false, the exception is ' - 'reraised.\n' - ' If the return value was true, the exception is suppressed, ' - 'and\n' - ' execution continues with the statement following the "with"\n' - ' statement.\n' - '\n' - ' If the suite was exited for any reason other than an ' - 'exception, the\n' - ' return value from "__exit__()" is ignored, and execution ' - 'proceeds\n' - ' at the normal location for the kind of exit that was taken.\n' - '\n' - 'With more than one item, the context managers are processed as ' - 'if\n' - 'multiple "with" statements were nested:\n' - '\n' - ' with A() as a, B() as b:\n' - ' suite\n' - '\n' - 'is equivalent to\n' - '\n' - ' with A() as a:\n' - ' with B() as b:\n' - ' suite\n' - '\n' - 'Changed in version 3.1: Support for multiple context ' - 'expressions.\n' - '\n' - 'See also:\n' - '\n' - ' **PEP 343** - The “with” statement\n' - ' The specification, background, and examples for the Python ' - '"with"\n' - ' statement.\n' - '\n' - '\n' - 'Function definitions\n' - '====================\n' - '\n' - 'A function definition defines a user-defined function object ' - '(see\n' - 'section The standard type hierarchy):\n' - '\n' - ' funcdef ::= [decorators] "def" funcname "(" ' - '[parameter_list] ")"\n' - ' ["->" expression] ":" suite\n' - ' decorators ::= decorator+\n' - ' decorator ::= "@" dotted_name ["(" ' - '[argument_list [","]] ")"] NEWLINE\n' - ' dotted_name ::= identifier ("." identifier)*\n' - ' parameter_list ::= defparameter ("," defparameter)* ' - '["," [parameter_list_starargs]]\n' - ' | parameter_list_starargs\n' - ' parameter_list_starargs ::= "*" [parameter] ("," ' - 'defparameter)* ["," ["**" parameter [","]]]\n' - ' | "**" parameter [","]\n' - ' parameter ::= identifier [":" expression]\n' - ' defparameter ::= parameter ["=" expression]\n' - ' funcname ::= identifier\n' - '\n' - 'A function definition is an executable statement. Its execution ' - 'binds\n' - 'the function name in the current local namespace to a function ' - 'object\n' - '(a wrapper around the executable code for the function). This\n' - 'function object contains a reference to the current global ' - 'namespace\n' - 'as the global namespace to be used when the function is called.\n' - '\n' - 'The function definition does not execute the function body; this ' - 'gets\n' - 'executed only when the function is called. [2]\n' - '\n' - 'A function definition may be wrapped by one or more *decorator*\n' - 'expressions. Decorator expressions are evaluated when the ' - 'function is\n' - 'defined, in the scope that contains the function definition. ' - 'The\n' - 'result must be a callable, which is invoked with the function ' - 'object\n' - 'as the only argument. The returned value is bound to the ' - 'function name\n' - 'instead of the function object. Multiple decorators are applied ' - 'in\n' - 'nested fashion. For example, the following code\n' - '\n' - ' @f1(arg)\n' - ' @f2\n' - ' def func(): pass\n' - '\n' - 'is roughly equivalent to\n' - '\n' - ' def func(): pass\n' - ' func = f1(arg)(f2(func))\n' - '\n' - 'except that the original function is not temporarily bound to ' - 'the name\n' - '"func".\n' - '\n' - 'When one or more *parameters* have the form *parameter* "="\n' - '*expression*, the function is said to have “default parameter ' - 'values.”\n' - 'For a parameter with a default value, the corresponding ' - '*argument* may\n' - 'be omitted from a call, in which case the parameter’s default ' - 'value is\n' - 'substituted. If a parameter has a default value, all following\n' - 'parameters up until the “"*"” must also have a default value — ' - 'this is\n' - 'a syntactic restriction that is not expressed by the grammar.\n' - '\n' - '**Default parameter values are evaluated from left to right when ' - 'the\n' - 'function definition is executed.** This means that the ' - 'expression is\n' - 'evaluated once, when the function is defined, and that the same ' - '“pre-\n' - 'computed” value is used for each call. This is especially ' - 'important\n' - 'to understand when a default parameter is a mutable object, such ' - 'as a\n' - 'list or a dictionary: if the function modifies the object (e.g. ' - 'by\n' - 'appending an item to a list), the default value is in effect ' - 'modified.\n' - 'This is generally not what was intended. A way around this is ' - 'to use\n' - '"None" as the default, and explicitly test for it in the body of ' - 'the\n' - 'function, e.g.:\n' - '\n' - ' def whats_on_the_telly(penguin=None):\n' - ' if penguin is None:\n' - ' penguin = []\n' - ' penguin.append("property of the zoo")\n' - ' return penguin\n' - '\n' - 'Function call semantics are described in more detail in section ' - 'Calls.\n' - 'A function call always assigns values to all parameters ' - 'mentioned in\n' - 'the parameter list, either from position arguments, from ' - 'keyword\n' - 'arguments, or from default values. If the form “"*identifier"” ' - 'is\n' - 'present, it is initialized to a tuple receiving any excess ' - 'positional\n' - 'parameters, defaulting to the empty tuple. If the form\n' - '“"**identifier"” is present, it is initialized to a new ordered\n' - 'mapping receiving any excess keyword arguments, defaulting to a ' - 'new\n' - 'empty mapping of the same type. Parameters after “"*"” or\n' - '“"*identifier"” are keyword-only parameters and may only be ' - 'passed\n' - 'used keyword arguments.\n' - '\n' - 'Parameters may have annotations of the form “": expression"” ' - 'following\n' - 'the parameter name. Any parameter may have an annotation even ' - 'those\n' - 'of the form "*identifier" or "**identifier". Functions may ' - 'have\n' - '“return” annotation of the form “"-> expression"” after the ' - 'parameter\n' - 'list. These annotations can be any valid Python expression and ' - 'are\n' - 'evaluated when the function definition is executed. Annotations ' - 'may\n' - 'be evaluated in a different order than they appear in the source ' - 'code.\n' - 'The presence of annotations does not change the semantics of a\n' - 'function. The annotation values are available as values of a\n' - 'dictionary keyed by the parameters’ names in the ' - '"__annotations__"\n' - 'attribute of the function object.\n' - '\n' - 'It is also possible to create anonymous functions (functions not ' - 'bound\n' - 'to a name), for immediate use in expressions. This uses lambda\n' - 'expressions, described in section Lambdas. Note that the ' - 'lambda\n' - 'expression is merely a shorthand for a simplified function ' - 'definition;\n' - 'a function defined in a “"def"” statement can be passed around ' - 'or\n' - 'assigned to another name just like a function defined by a ' - 'lambda\n' - 'expression. The “"def"” form is actually more powerful since ' - 'it\n' - 'allows the execution of multiple statements and annotations.\n' - '\n' - '**Programmer’s note:** Functions are first-class objects. A ' - '“"def"”\n' - 'statement executed inside a function definition defines a local\n' - 'function that can be returned or passed around. Free variables ' - 'used\n' - 'in the nested function can access the local variables of the ' - 'function\n' - 'containing the def. See section Naming and binding for ' - 'details.\n' - '\n' - 'See also:\n' - '\n' - ' **PEP 3107** - Function Annotations\n' - ' The original specification for function annotations.\n' - '\n' - '\n' - 'Class definitions\n' - '=================\n' - '\n' - 'A class definition defines a class object (see section The ' - 'standard\n' - 'type hierarchy):\n' - '\n' - ' classdef ::= [decorators] "class" classname [inheritance] ' - '":" suite\n' - ' inheritance ::= "(" [argument_list] ")"\n' - ' classname ::= identifier\n' - '\n' - 'A class definition is an executable statement. The inheritance ' - 'list\n' - 'usually gives a list of base classes (see Metaclasses for more\n' - 'advanced uses), so each item in the list should evaluate to a ' - 'class\n' - 'object which allows subclassing. Classes without an inheritance ' - 'list\n' - 'inherit, by default, from the base class "object"; hence,\n' - '\n' - ' class Foo:\n' - ' pass\n' - '\n' - 'is equivalent to\n' - '\n' - ' class Foo(object):\n' - ' pass\n' - '\n' - 'The class’s suite is then executed in a new execution frame ' - '(see\n' - 'Naming and binding), using a newly created local namespace and ' - 'the\n' - 'original global namespace. (Usually, the suite contains mostly\n' - 'function definitions.) When the class’s suite finishes ' - 'execution, its\n' - 'execution frame is discarded but its local namespace is saved. ' - '[3] A\n' - 'class object is then created using the inheritance list for the ' - 'base\n' - 'classes and the saved local namespace for the attribute ' - 'dictionary.\n' - 'The class name is bound to this class object in the original ' - 'local\n' - 'namespace.\n' - '\n' - 'The order in which attributes are defined in the class body is\n' - 'preserved in the new class’s "__dict__". Note that this is ' - 'reliable\n' - 'only right after the class is created and only for classes that ' - 'were\n' - 'defined using the definition syntax.\n' - '\n' - 'Class creation can be customized heavily using metaclasses.\n' - '\n' - 'Classes can also be decorated: just like when decorating ' - 'functions,\n' - '\n' - ' @f1(arg)\n' - ' @f2\n' - ' class Foo: pass\n' - '\n' - 'is roughly equivalent to\n' - '\n' - ' class Foo: pass\n' - ' Foo = f1(arg)(f2(Foo))\n' - '\n' - 'The evaluation rules for the decorator expressions are the same ' - 'as for\n' - 'function decorators. The result is then bound to the class ' - 'name.\n' - '\n' - '**Programmer’s note:** Variables defined in the class definition ' - 'are\n' - 'class attributes; they are shared by instances. Instance ' - 'attributes\n' - 'can be set in a method with "self.name = value". Both class ' - 'and\n' - 'instance attributes are accessible through the notation ' - '“"self.name"”,\n' - 'and an instance attribute hides a class attribute with the same ' - 'name\n' - 'when accessed in this way. Class attributes can be used as ' - 'defaults\n' - 'for instance attributes, but using mutable values there can lead ' - 'to\n' - 'unexpected results. Descriptors can be used to create instance\n' - 'variables with different implementation details.\n' - '\n' - 'See also:\n' - '\n' - ' **PEP 3115** - Metaclasses in Python 3000\n' - ' The proposal that changed the declaration of metaclasses to ' - 'the\n' - ' current syntax, and the semantics for how classes with\n' - ' metaclasses are constructed.\n' - '\n' - ' **PEP 3129** - Class Decorators\n' - ' The proposal that added class decorators. Function and ' - 'method\n' - ' decorators were introduced in **PEP 318**.\n' - '\n' - '\n' - 'Coroutines\n' - '==========\n' - '\n' - 'New in version 3.5.\n' - '\n' - '\n' - 'Coroutine function definition\n' - '-----------------------------\n' - '\n' - ' async_funcdef ::= [decorators] "async" "def" funcname "(" ' - '[parameter_list] ")"\n' - ' ["->" expression] ":" suite\n' - '\n' - 'Execution of Python coroutines can be suspended and resumed at ' - 'many\n' - 'points (see *coroutine*). In the body of a coroutine, any ' - '"await" and\n' - '"async" identifiers become reserved keywords; "await" ' - 'expressions,\n' - '"async for" and "async with" can only be used in coroutine ' - 'bodies.\n' - '\n' - 'Functions defined with "async def" syntax are always coroutine\n' - 'functions, even if they do not contain "await" or "async" ' - 'keywords.\n' - '\n' - 'It is a "SyntaxError" to use "yield from" expressions in "async ' - 'def"\n' - 'coroutines.\n' - '\n' - 'An example of a coroutine function:\n' - '\n' - ' async def func(param1, param2):\n' - ' do_stuff()\n' - ' await some_coroutine()\n' - '\n' - '\n' - 'The "async for" statement\n' - '-------------------------\n' - '\n' - ' async_for_stmt ::= "async" for_stmt\n' - '\n' - 'An *asynchronous iterable* is able to call asynchronous code in ' - 'its\n' - '*iter* implementation, and *asynchronous iterator* can call\n' - 'asynchronous code in its *next* method.\n' - '\n' - 'The "async for" statement allows convenient iteration over\n' - 'asynchronous iterators.\n' - '\n' - 'The following code:\n' - '\n' - ' async for TARGET in ITER:\n' - ' BLOCK\n' - ' else:\n' - ' BLOCK2\n' - '\n' - 'Is semantically equivalent to:\n' - '\n' - ' iter = (ITER)\n' - ' iter = type(iter).__aiter__(iter)\n' - ' running = True\n' - ' while running:\n' - ' try:\n' - ' TARGET = await type(iter).__anext__(iter)\n' - ' except StopAsyncIteration:\n' - ' running = False\n' - ' else:\n' - ' BLOCK\n' - ' else:\n' - ' BLOCK2\n' - '\n' - 'See also "__aiter__()" and "__anext__()" for details.\n' - '\n' - 'It is a "SyntaxError" to use "async for" statement outside of ' - 'an\n' - '"async def" function.\n' - '\n' - '\n' - 'The "async with" statement\n' - '--------------------------\n' - '\n' - ' async_with_stmt ::= "async" with_stmt\n' - '\n' - 'An *asynchronous context manager* is a *context manager* that is ' - 'able\n' - 'to suspend execution in its *enter* and *exit* methods.\n' - '\n' - 'The following code:\n' - '\n' - ' async with EXPR as VAR:\n' - ' BLOCK\n' - '\n' - 'Is semantically equivalent to:\n' - '\n' - ' mgr = (EXPR)\n' - ' aexit = type(mgr).__aexit__\n' - ' aenter = type(mgr).__aenter__(mgr)\n' - '\n' - ' VAR = await aenter\n' - ' try:\n' - ' BLOCK\n' - ' except:\n' - ' if not await aexit(mgr, *sys.exc_info()):\n' - ' raise\n' - ' else:\n' - ' await aexit(mgr, None, None, None)\n' - '\n' - 'See also "__aenter__()" and "__aexit__()" for details.\n' - '\n' - 'It is a "SyntaxError" to use "async with" statement outside of ' - 'an\n' - '"async def" function.\n' - '\n' - 'See also:\n' - '\n' - ' **PEP 492** - Coroutines with async and await syntax\n' - ' The proposal that made coroutines a proper standalone ' - 'concept in\n' - ' Python, and added supporting syntax.\n' - '\n' - '-[ Footnotes ]-\n' - '\n' - '[1] The exception is propagated to the invocation stack unless\n' - ' there is a "finally" clause which happens to raise another\n' - ' exception. That new exception causes the old one to be ' - 'lost.\n' - '\n' - '[2] A string literal appearing as the first statement in the\n' - ' function body is transformed into the function’s "__doc__"\n' - ' attribute and therefore the function’s *docstring*.\n' - '\n' - '[3] A string literal appearing as the first statement in the ' - 'class\n' - ' body is transformed into the namespace’s "__doc__" item and\n' - ' therefore the class’s *docstring*.\n', - 'context-managers': 'With Statement Context Managers\n' - '*******************************\n' - '\n' - 'A *context manager* is an object that defines the ' - 'runtime context to\n' - 'be established when executing a "with" statement. The ' - 'context manager\n' - 'handles the entry into, and the exit from, the desired ' - 'runtime context\n' - 'for the execution of the block of code. Context ' - 'managers are normally\n' - 'invoked using the "with" statement (described in section ' - 'The with\n' - 'statement), but can also be used by directly invoking ' - 'their methods.\n' - '\n' - 'Typical uses of context managers include saving and ' - 'restoring various\n' - 'kinds of global state, locking and unlocking resources, ' - 'closing opened\n' - 'files, etc.\n' - '\n' - 'For more information on context managers, see Context ' - 'Manager Types.\n' - '\n' - 'object.__enter__(self)\n' - '\n' - ' Enter the runtime context related to this object. The ' - '"with"\n' - ' statement will bind this method’s return value to the ' - 'target(s)\n' - ' specified in the "as" clause of the statement, if ' - 'any.\n' - '\n' - 'object.__exit__(self, exc_type, exc_value, traceback)\n' - '\n' - ' Exit the runtime context related to this object. The ' - 'parameters\n' - ' describe the exception that caused the context to be ' - 'exited. If the\n' - ' context was exited without an exception, all three ' - 'arguments will\n' - ' be "None".\n' - '\n' - ' If an exception is supplied, and the method wishes to ' - 'suppress the\n' - ' exception (i.e., prevent it from being propagated), ' - 'it should\n' - ' return a true value. Otherwise, the exception will be ' - 'processed\n' - ' normally upon exit from this method.\n' - '\n' - ' Note that "__exit__()" methods should not reraise the ' - 'passed-in\n' - ' exception; this is the caller’s responsibility.\n' - '\n' - 'See also:\n' - '\n' - ' **PEP 343** - The “with” statement\n' - ' The specification, background, and examples for the ' - 'Python "with"\n' - ' statement.\n', - 'continue': 'The "continue" statement\n' - '************************\n' - '\n' - ' continue_stmt ::= "continue"\n' - '\n' - '"continue" may only occur syntactically nested in a "for" or ' - '"while"\n' - 'loop, but not nested in a function or class definition or ' - '"finally"\n' - 'clause within that loop. It continues with the next cycle of ' - 'the\n' - 'nearest enclosing loop.\n' - '\n' - 'When "continue" passes control out of a "try" statement with a\n' - '"finally" clause, that "finally" clause is executed before ' - 'really\n' - 'starting the next loop cycle.\n', - 'conversions': 'Arithmetic conversions\n' - '**********************\n' - '\n' - 'When a description of an arithmetic operator below uses the ' - 'phrase\n' - '“the numeric arguments are converted to a common type,” this ' - 'means\n' - 'that the operator implementation for built-in types works as ' - 'follows:\n' - '\n' - '* If either argument is a complex number, the other is ' - 'converted to\n' - ' complex;\n' - '\n' - '* otherwise, if either argument is a floating point number, ' - 'the\n' - ' other is converted to floating point;\n' - '\n' - '* otherwise, both must be integers and no conversion is ' - 'necessary.\n' - '\n' - 'Some additional rules apply for certain operators (e.g., a ' - 'string as a\n' - 'left argument to the ‘%’ operator). Extensions must define ' - 'their own\n' - 'conversion behavior.\n', - 'customization': 'Basic customization\n' - '*******************\n' - '\n' - 'object.__new__(cls[, ...])\n' - '\n' - ' Called to create a new instance of class *cls*. ' - '"__new__()" is a\n' - ' static method (special-cased so you need not declare it ' - 'as such)\n' - ' that takes the class of which an instance was requested ' - 'as its\n' - ' first argument. The remaining arguments are those ' - 'passed to the\n' - ' object constructor expression (the call to the class). ' - 'The return\n' - ' value of "__new__()" should be the new object instance ' - '(usually an\n' - ' instance of *cls*).\n' - '\n' - ' Typical implementations create a new instance of the ' - 'class by\n' - ' invoking the superclass’s "__new__()" method using\n' - ' "super().__new__(cls[, ...])" with appropriate arguments ' - 'and then\n' - ' modifying the newly-created instance as necessary before ' - 'returning\n' - ' it.\n' - '\n' - ' If "__new__()" returns an instance of *cls*, then the ' - 'new\n' - ' instance’s "__init__()" method will be invoked like\n' - ' "__init__(self[, ...])", where *self* is the new ' - 'instance and the\n' - ' remaining arguments are the same as were passed to ' - '"__new__()".\n' - '\n' - ' If "__new__()" does not return an instance of *cls*, ' - 'then the new\n' - ' instance’s "__init__()" method will not be invoked.\n' - '\n' - ' "__new__()" is intended mainly to allow subclasses of ' - 'immutable\n' - ' types (like int, str, or tuple) to customize instance ' - 'creation. It\n' - ' is also commonly overridden in custom metaclasses in ' - 'order to\n' - ' customize class creation.\n' - '\n' - 'object.__init__(self[, ...])\n' - '\n' - ' Called after the instance has been created (by ' - '"__new__()"), but\n' - ' before it is returned to the caller. The arguments are ' - 'those\n' - ' passed to the class constructor expression. If a base ' - 'class has an\n' - ' "__init__()" method, the derived class’s "__init__()" ' - 'method, if\n' - ' any, must explicitly call it to ensure proper ' - 'initialization of the\n' - ' base class part of the instance; for example:\n' - ' "super().__init__([args...])".\n' - '\n' - ' Because "__new__()" and "__init__()" work together in ' - 'constructing\n' - ' objects ("__new__()" to create it, and "__init__()" to ' - 'customize\n' - ' it), no non-"None" value may be returned by ' - '"__init__()"; doing so\n' - ' will cause a "TypeError" to be raised at runtime.\n' - '\n' - 'object.__del__(self)\n' - '\n' - ' Called when the instance is about to be destroyed. This ' - 'is also\n' - ' called a finalizer or (improperly) a destructor. If a ' - 'base class\n' - ' has a "__del__()" method, the derived class’s ' - '"__del__()" method,\n' - ' if any, must explicitly call it to ensure proper ' - 'deletion of the\n' - ' base class part of the instance.\n' - '\n' - ' It is possible (though not recommended!) for the ' - '"__del__()" method\n' - ' to postpone destruction of the instance by creating a ' - 'new reference\n' - ' to it. This is called object *resurrection*. It is\n' - ' implementation-dependent whether "__del__()" is called a ' - 'second\n' - ' time when a resurrected object is about to be destroyed; ' - 'the\n' - ' current *CPython* implementation only calls it once.\n' - '\n' - ' It is not guaranteed that "__del__()" methods are called ' - 'for\n' - ' objects that still exist when the interpreter exits.\n' - '\n' - ' Note: "del x" doesn’t directly call "x.__del__()" — the ' - 'former\n' - ' decrements the reference count for "x" by one, and the ' - 'latter is\n' - ' only called when "x"’s reference count reaches zero.\n' - '\n' - ' **CPython implementation detail:** It is possible for a ' - 'reference\n' - ' cycle to prevent the reference count of an object from ' - 'going to\n' - ' zero. In this case, the cycle will be later detected ' - 'and deleted\n' - ' by the *cyclic garbage collector*. A common cause of ' - 'reference\n' - ' cycles is when an exception has been caught in a local ' - 'variable.\n' - ' The frame’s locals then reference the exception, which ' - 'references\n' - ' its own traceback, which references the locals of all ' - 'frames caught\n' - ' in the traceback.\n' - '\n' - ' See also: Documentation for the "gc" module.\n' - '\n' - ' Warning: Due to the precarious circumstances under ' - 'which\n' - ' "__del__()" methods are invoked, exceptions that occur ' - 'during\n' - ' their execution are ignored, and a warning is printed ' - 'to\n' - ' "sys.stderr" instead. In particular:\n' - '\n' - ' * "__del__()" can be invoked when arbitrary code is ' - 'being\n' - ' executed, including from any arbitrary thread. If ' - '"__del__()"\n' - ' needs to take a lock or invoke any other blocking ' - 'resource, it\n' - ' may deadlock as the resource may already be taken by ' - 'the code\n' - ' that gets interrupted to execute "__del__()".\n' - '\n' - ' * "__del__()" can be executed during interpreter ' - 'shutdown. As\n' - ' a consequence, the global variables it needs to ' - 'access\n' - ' (including other modules) may already have been ' - 'deleted or set\n' - ' to "None". Python guarantees that globals whose name ' - 'begins\n' - ' with a single underscore are deleted from their ' - 'module before\n' - ' other globals are deleted; if no other references to ' - 'such\n' - ' globals exist, this may help in assuring that ' - 'imported modules\n' - ' are still available at the time when the "__del__()" ' - 'method is\n' - ' called.\n' - '\n' - 'object.__repr__(self)\n' - '\n' - ' Called by the "repr()" built-in function to compute the ' - '“official”\n' - ' string representation of an object. If at all possible, ' - 'this\n' - ' should look like a valid Python expression that could be ' - 'used to\n' - ' recreate an object with the same value (given an ' - 'appropriate\n' - ' environment). If this is not possible, a string of the ' - 'form\n' - ' "<...some useful description...>" should be returned. ' - 'The return\n' - ' value must be a string object. If a class defines ' - '"__repr__()" but\n' - ' not "__str__()", then "__repr__()" is also used when an ' - '“informal”\n' - ' string representation of instances of that class is ' - 'required.\n' - '\n' - ' This is typically used for debugging, so it is important ' - 'that the\n' - ' representation is information-rich and unambiguous.\n' - '\n' - 'object.__str__(self)\n' - '\n' - ' Called by "str(object)" and the built-in functions ' - '"format()" and\n' - ' "print()" to compute the “informal” or nicely printable ' - 'string\n' - ' representation of an object. The return value must be a ' - 'string\n' - ' object.\n' - '\n' - ' This method differs from "object.__repr__()" in that ' - 'there is no\n' - ' expectation that "__str__()" return a valid Python ' - 'expression: a\n' - ' more convenient or concise representation can be used.\n' - '\n' - ' The default implementation defined by the built-in type ' - '"object"\n' - ' calls "object.__repr__()".\n' - '\n' - 'object.__bytes__(self)\n' - '\n' - ' Called by bytes to compute a byte-string representation ' - 'of an\n' - ' object. This should return a "bytes" object.\n' - '\n' - 'object.__format__(self, format_spec)\n' - '\n' - ' Called by the "format()" built-in function, and by ' - 'extension,\n' - ' evaluation of formatted string literals and the ' - '"str.format()"\n' - ' method, to produce a “formatted” string representation ' - 'of an\n' - ' object. The "format_spec" argument is a string that ' - 'contains a\n' - ' description of the formatting options desired. The ' - 'interpretation\n' - ' of the "format_spec" argument is up to the type ' - 'implementing\n' - ' "__format__()", however most classes will either ' - 'delegate\n' - ' formatting to one of the built-in types, or use a ' - 'similar\n' - ' formatting option syntax.\n' - '\n' - ' See Format Specification Mini-Language for a description ' - 'of the\n' - ' standard formatting syntax.\n' - '\n' - ' The return value must be a string object.\n' - '\n' - ' Changed in version 3.4: The __format__ method of ' - '"object" itself\n' - ' raises a "TypeError" if passed any non-empty string.\n' - '\n' - 'object.__lt__(self, other)\n' - 'object.__le__(self, other)\n' - 'object.__eq__(self, other)\n' - 'object.__ne__(self, other)\n' - 'object.__gt__(self, other)\n' - 'object.__ge__(self, other)\n' - '\n' - ' These are the so-called “rich comparison” methods. The\n' - ' correspondence between operator symbols and method names ' - 'is as\n' - ' follows: "xy" calls\n' - ' "x.__gt__(y)", and "x>=y" calls "x.__ge__(y)".\n' - '\n' - ' A rich comparison method may return the singleton ' - '"NotImplemented"\n' - ' if it does not implement the operation for a given pair ' - 'of\n' - ' arguments. By convention, "False" and "True" are ' - 'returned for a\n' - ' successful comparison. However, these methods can return ' - 'any value,\n' - ' so if the comparison operator is used in a Boolean ' - 'context (e.g.,\n' - ' in the condition of an "if" statement), Python will call ' - '"bool()"\n' - ' on the value to determine if the result is true or ' - 'false.\n' - '\n' - ' By default, "__ne__()" delegates to "__eq__()" and ' - 'inverts the\n' - ' result unless it is "NotImplemented". There are no ' - 'other implied\n' - ' relationships among the comparison operators, for ' - 'example, the\n' - ' truth of "(x.__hash__".\n' - '\n' - ' If a class that does not override "__eq__()" wishes to ' - 'suppress\n' - ' hash support, it should include "__hash__ = None" in the ' - 'class\n' - ' definition. A class which defines its own "__hash__()" ' - 'that\n' - ' explicitly raises a "TypeError" would be incorrectly ' - 'identified as\n' - ' hashable by an "isinstance(obj, collections.Hashable)" ' - 'call.\n' - '\n' - ' Note: By default, the "__hash__()" values of str, bytes ' - 'and\n' - ' datetime objects are “salted” with an unpredictable ' - 'random value.\n' - ' Although they remain constant within an individual ' - 'Python\n' - ' process, they are not predictable between repeated ' - 'invocations of\n' - ' Python.This is intended to provide protection against ' - 'a denial-\n' - ' of-service caused by carefully-chosen inputs that ' - 'exploit the\n' - ' worst case performance of a dict insertion, O(n^2) ' - 'complexity.\n' - ' See ' - 'http://www.ocert.org/advisories/ocert-2011-003.html for\n' - ' details.Changing hash values affects the iteration ' - 'order of\n' - ' dicts, sets and other mappings. Python has never made ' - 'guarantees\n' - ' about this ordering (and it typically varies between ' - '32-bit and\n' - ' 64-bit builds).See also "PYTHONHASHSEED".\n' - '\n' - ' Changed in version 3.3: Hash randomization is enabled by ' - 'default.\n' - '\n' - 'object.__bool__(self)\n' - '\n' - ' Called to implement truth value testing and the built-in ' - 'operation\n' - ' "bool()"; should return "False" or "True". When this ' - 'method is not\n' - ' defined, "__len__()" is called, if it is defined, and ' - 'the object is\n' - ' considered true if its result is nonzero. If a class ' - 'defines\n' - ' neither "__len__()" nor "__bool__()", all its instances ' - 'are\n' - ' considered true.\n', - 'debugger': '"pdb" — The Python Debugger\n' - '***************************\n' - '\n' - '**Source code:** Lib/pdb.py\n' - '\n' - '======================================================================\n' - '\n' - 'The module "pdb" defines an interactive source code debugger ' - 'for\n' - 'Python programs. It supports setting (conditional) breakpoints ' - 'and\n' - 'single stepping at the source line level, inspection of stack ' - 'frames,\n' - 'source code listing, and evaluation of arbitrary Python code in ' - 'the\n' - 'context of any stack frame. It also supports post-mortem ' - 'debugging\n' - 'and can be called under program control.\n' - '\n' - 'The debugger is extensible – it is actually defined as the ' - 'class\n' - '"Pdb". This is currently undocumented but easily understood by ' - 'reading\n' - 'the source. The extension interface uses the modules "bdb" and ' - '"cmd".\n' - '\n' - 'The debugger’s prompt is "(Pdb)". Typical usage to run a program ' - 'under\n' - 'control of the debugger is:\n' - '\n' - ' >>> import pdb\n' - ' >>> import mymodule\n' - " >>> pdb.run('mymodule.test()')\n" - ' > (0)?()\n' - ' (Pdb) continue\n' - ' > (1)?()\n' - ' (Pdb) continue\n' - " NameError: 'spam'\n" - ' > (1)?()\n' - ' (Pdb)\n' - '\n' - 'Changed in version 3.3: Tab-completion via the "readline" module ' - 'is\n' - 'available for commands and command arguments, e.g. the current ' - 'global\n' - 'and local names are offered as arguments of the "p" command.\n' - '\n' - '"pdb.py" can also be invoked as a script to debug other ' - 'scripts. For\n' - 'example:\n' - '\n' - ' python3 -m pdb myscript.py\n' - '\n' - 'When invoked as a script, pdb will automatically enter ' - 'post-mortem\n' - 'debugging if the program being debugged exits abnormally. After ' - 'post-\n' - 'mortem debugging (or after normal exit of the program), pdb ' - 'will\n' - 'restart the program. Automatic restarting preserves pdb’s state ' - '(such\n' - 'as breakpoints) and in most cases is more useful than quitting ' - 'the\n' - 'debugger upon program’s exit.\n' - '\n' - 'New in version 3.2: "pdb.py" now accepts a "-c" option that ' - 'executes\n' - 'commands as if given in a ".pdbrc" file, see Debugger Commands.\n' - '\n' - 'The typical usage to break into the debugger from a running ' - 'program is\n' - 'to insert\n' - '\n' - ' import pdb; pdb.set_trace()\n' - '\n' - 'at the location you want to break into the debugger. You can ' - 'then\n' - 'step through the code following this statement, and continue ' - 'running\n' - 'without the debugger using the "continue" command.\n' - '\n' - 'The typical usage to inspect a crashed program is:\n' - '\n' - ' >>> import pdb\n' - ' >>> import mymodule\n' - ' >>> mymodule.test()\n' - ' Traceback (most recent call last):\n' - ' File "", line 1, in \n' - ' File "./mymodule.py", line 4, in test\n' - ' test2()\n' - ' File "./mymodule.py", line 3, in test2\n' - ' print(spam)\n' - ' NameError: spam\n' - ' >>> pdb.pm()\n' - ' > ./mymodule.py(3)test2()\n' - ' -> print(spam)\n' - ' (Pdb)\n' - '\n' - 'The module defines the following functions; each enters the ' - 'debugger\n' - 'in a slightly different way:\n' - '\n' - 'pdb.run(statement, globals=None, locals=None)\n' - '\n' - ' Execute the *statement* (given as a string or a code object) ' - 'under\n' - ' debugger control. The debugger prompt appears before any ' - 'code is\n' - ' executed; you can set breakpoints and type "continue", or you ' - 'can\n' - ' step through the statement using "step" or "next" (all these\n' - ' commands are explained below). The optional *globals* and ' - '*locals*\n' - ' arguments specify the environment in which the code is ' - 'executed; by\n' - ' default the dictionary of the module "__main__" is used. ' - '(See the\n' - ' explanation of the built-in "exec()" or "eval()" functions.)\n' - '\n' - 'pdb.runeval(expression, globals=None, locals=None)\n' - '\n' - ' Evaluate the *expression* (given as a string or a code ' - 'object)\n' - ' under debugger control. When "runeval()" returns, it returns ' - 'the\n' - ' value of the expression. Otherwise this function is similar ' - 'to\n' - ' "run()".\n' - '\n' - 'pdb.runcall(function, *args, **kwds)\n' - '\n' - ' Call the *function* (a function or method object, not a ' - 'string)\n' - ' with the given arguments. When "runcall()" returns, it ' - 'returns\n' - ' whatever the function call returned. The debugger prompt ' - 'appears\n' - ' as soon as the function is entered.\n' - '\n' - 'pdb.set_trace()\n' - '\n' - ' Enter the debugger at the calling stack frame. This is ' - 'useful to\n' - ' hard-code a breakpoint at a given point in a program, even if ' - 'the\n' - ' code is not otherwise being debugged (e.g. when an assertion\n' - ' fails).\n' - '\n' - 'pdb.post_mortem(traceback=None)\n' - '\n' - ' Enter post-mortem debugging of the given *traceback* object. ' - 'If no\n' - ' *traceback* is given, it uses the one of the exception that ' - 'is\n' - ' currently being handled (an exception must be being handled ' - 'if the\n' - ' default is to be used).\n' - '\n' - 'pdb.pm()\n' - '\n' - ' Enter post-mortem debugging of the traceback found in\n' - ' "sys.last_traceback".\n' - '\n' - 'The "run*" functions and "set_trace()" are aliases for ' - 'instantiating\n' - 'the "Pdb" class and calling the method of the same name. If you ' - 'want\n' - 'to access further features, you have to do this yourself:\n' - '\n' - "class pdb.Pdb(completekey='tab', stdin=None, stdout=None, " - 'skip=None, nosigint=False, readrc=True)\n' - '\n' - ' "Pdb" is the debugger class.\n' - '\n' - ' The *completekey*, *stdin* and *stdout* arguments are passed ' - 'to the\n' - ' underlying "cmd.Cmd" class; see the description there.\n' - '\n' - ' The *skip* argument, if given, must be an iterable of ' - 'glob-style\n' - ' module name patterns. The debugger will not step into frames ' - 'that\n' - ' originate in a module that matches one of these patterns. ' - '[1]\n' - '\n' - ' By default, Pdb sets a handler for the SIGINT signal (which ' - 'is sent\n' - ' when the user presses "Ctrl-C" on the console) when you give ' - 'a\n' - ' "continue" command. This allows you to break into the ' - 'debugger\n' - ' again by pressing "Ctrl-C". If you want Pdb not to touch ' - 'the\n' - ' SIGINT handler, set *nosigint* to true.\n' - '\n' - ' The *readrc* argument defaults to true and controls whether ' - 'Pdb\n' - ' will load .pdbrc files from the filesystem.\n' - '\n' - ' Example call to enable tracing with *skip*:\n' - '\n' - " import pdb; pdb.Pdb(skip=['django.*']).set_trace()\n" - '\n' - ' New in version 3.1: The *skip* argument.\n' - '\n' - ' New in version 3.2: The *nosigint* argument. Previously, a ' - 'SIGINT\n' - ' handler was never set by Pdb.\n' - '\n' - ' Changed in version 3.6: The *readrc* argument.\n' - '\n' - ' run(statement, globals=None, locals=None)\n' - ' runeval(expression, globals=None, locals=None)\n' - ' runcall(function, *args, **kwds)\n' - ' set_trace()\n' - '\n' - ' See the documentation for the functions explained above.\n' - '\n' - '\n' - 'Debugger Commands\n' - '=================\n' - '\n' - 'The commands recognized by the debugger are listed below. Most\n' - 'commands can be abbreviated to one or two letters as indicated; ' - 'e.g.\n' - '"h(elp)" means that either "h" or "help" can be used to enter ' - 'the help\n' - 'command (but not "he" or "hel", nor "H" or "Help" or "HELP").\n' - 'Arguments to commands must be separated by whitespace (spaces ' - 'or\n' - 'tabs). Optional arguments are enclosed in square brackets ' - '("[]") in\n' - 'the command syntax; the square brackets must not be typed.\n' - 'Alternatives in the command syntax are separated by a vertical ' - 'bar\n' - '("|").\n' - '\n' - 'Entering a blank line repeats the last command entered. ' - 'Exception: if\n' - 'the last command was a "list" command, the next 11 lines are ' - 'listed.\n' - '\n' - 'Commands that the debugger doesn’t recognize are assumed to be ' - 'Python\n' - 'statements and are executed in the context of the program being\n' - 'debugged. Python statements can also be prefixed with an ' - 'exclamation\n' - 'point ("!"). This is a powerful way to inspect the program ' - 'being\n' - 'debugged; it is even possible to change a variable or call a ' - 'function.\n' - 'When an exception occurs in such a statement, the exception name ' - 'is\n' - 'printed but the debugger’s state is not changed.\n' - '\n' - 'The debugger supports aliases. Aliases can have parameters ' - 'which\n' - 'allows one a certain level of adaptability to the context under\n' - 'examination.\n' - '\n' - 'Multiple commands may be entered on a single line, separated by ' - '";;".\n' - '(A single ";" is not used as it is the separator for multiple ' - 'commands\n' - 'in a line that is passed to the Python parser.) No intelligence ' - 'is\n' - 'applied to separating the commands; the input is split at the ' - 'first\n' - '";;" pair, even if it is in the middle of a quoted string.\n' - '\n' - 'If a file ".pdbrc" exists in the user’s home directory or in ' - 'the\n' - 'current directory, it is read in and executed as if it had been ' - 'typed\n' - 'at the debugger prompt. This is particularly useful for ' - 'aliases. If\n' - 'both files exist, the one in the home directory is read first ' - 'and\n' - 'aliases defined there can be overridden by the local file.\n' - '\n' - 'Changed in version 3.2: ".pdbrc" can now contain commands that\n' - 'continue debugging, such as "continue" or "next". Previously, ' - 'these\n' - 'commands had no effect.\n' - '\n' - 'h(elp) [command]\n' - '\n' - ' Without argument, print the list of available commands. With ' - 'a\n' - ' *command* as argument, print help about that command. "help ' - 'pdb"\n' - ' displays the full documentation (the docstring of the "pdb"\n' - ' module). Since the *command* argument must be an identifier, ' - '"help\n' - ' exec" must be entered to get help on the "!" command.\n' - '\n' - 'w(here)\n' - '\n' - ' Print a stack trace, with the most recent frame at the ' - 'bottom. An\n' - ' arrow indicates the current frame, which determines the ' - 'context of\n' - ' most commands.\n' - '\n' - 'd(own) [count]\n' - '\n' - ' Move the current frame *count* (default one) levels down in ' - 'the\n' - ' stack trace (to a newer frame).\n' - '\n' - 'u(p) [count]\n' - '\n' - ' Move the current frame *count* (default one) levels up in the ' - 'stack\n' - ' trace (to an older frame).\n' - '\n' - 'b(reak) [([filename:]lineno | function) [, condition]]\n' - '\n' - ' With a *lineno* argument, set a break there in the current ' - 'file.\n' - ' With a *function* argument, set a break at the first ' - 'executable\n' - ' statement within that function. The line number may be ' - 'prefixed\n' - ' with a filename and a colon, to specify a breakpoint in ' - 'another\n' - ' file (probably one that hasn’t been loaded yet). The file ' - 'is\n' - ' searched on "sys.path". Note that each breakpoint is ' - 'assigned a\n' - ' number to which all the other breakpoint commands refer.\n' - '\n' - ' If a second argument is present, it is an expression which ' - 'must\n' - ' evaluate to true before the breakpoint is honored.\n' - '\n' - ' Without argument, list all breaks, including for each ' - 'breakpoint,\n' - ' the number of times that breakpoint has been hit, the ' - 'current\n' - ' ignore count, and the associated condition if any.\n' - '\n' - 'tbreak [([filename:]lineno | function) [, condition]]\n' - '\n' - ' Temporary breakpoint, which is removed automatically when it ' - 'is\n' - ' first hit. The arguments are the same as for "break".\n' - '\n' - 'cl(ear) [filename:lineno | bpnumber [bpnumber ...]]\n' - '\n' - ' With a *filename:lineno* argument, clear all the breakpoints ' - 'at\n' - ' this line. With a space separated list of breakpoint numbers, ' - 'clear\n' - ' those breakpoints. Without argument, clear all breaks (but ' - 'first\n' - ' ask confirmation).\n' - '\n' - 'disable [bpnumber [bpnumber ...]]\n' - '\n' - ' Disable the breakpoints given as a space separated list of\n' - ' breakpoint numbers. Disabling a breakpoint means it cannot ' - 'cause\n' - ' the program to stop execution, but unlike clearing a ' - 'breakpoint, it\n' - ' remains in the list of breakpoints and can be (re-)enabled.\n' - '\n' - 'enable [bpnumber [bpnumber ...]]\n' - '\n' - ' Enable the breakpoints specified.\n' - '\n' - 'ignore bpnumber [count]\n' - '\n' - ' Set the ignore count for the given breakpoint number. If ' - 'count is\n' - ' omitted, the ignore count is set to 0. A breakpoint becomes ' - 'active\n' - ' when the ignore count is zero. When non-zero, the count is\n' - ' decremented each time the breakpoint is reached and the ' - 'breakpoint\n' - ' is not disabled and any associated condition evaluates to ' - 'true.\n' - '\n' - 'condition bpnumber [condition]\n' - '\n' - ' Set a new *condition* for the breakpoint, an expression which ' - 'must\n' - ' evaluate to true before the breakpoint is honored. If ' - '*condition*\n' - ' is absent, any existing condition is removed; i.e., the ' - 'breakpoint\n' - ' is made unconditional.\n' - '\n' - 'commands [bpnumber]\n' - '\n' - ' Specify a list of commands for breakpoint number *bpnumber*. ' - 'The\n' - ' commands themselves appear on the following lines. Type a ' - 'line\n' - ' containing just "end" to terminate the commands. An example:\n' - '\n' - ' (Pdb) commands 1\n' - ' (com) p some_variable\n' - ' (com) end\n' - ' (Pdb)\n' - '\n' - ' To remove all commands from a breakpoint, type commands and ' - 'follow\n' - ' it immediately with "end"; that is, give no commands.\n' - '\n' - ' With no *bpnumber* argument, commands refers to the last ' - 'breakpoint\n' - ' set.\n' - '\n' - ' You can use breakpoint commands to start your program up ' - 'again.\n' - ' Simply use the continue command, or step, or any other ' - 'command that\n' - ' resumes execution.\n' - '\n' - ' Specifying any command resuming execution (currently ' - 'continue,\n' - ' step, next, return, jump, quit and their abbreviations) ' - 'terminates\n' - ' the command list (as if that command was immediately followed ' - 'by\n' - ' end). This is because any time you resume execution (even ' - 'with a\n' - ' simple next or step), you may encounter another ' - 'breakpoint—which\n' - ' could have its own command list, leading to ambiguities about ' - 'which\n' - ' list to execute.\n' - '\n' - ' If you use the ‘silent’ command in the command list, the ' - 'usual\n' - ' message about stopping at a breakpoint is not printed. This ' - 'may be\n' - ' desirable for breakpoints that are to print a specific ' - 'message and\n' - ' then continue. If none of the other commands print anything, ' - 'you\n' - ' see no sign that the breakpoint was reached.\n' - '\n' - 's(tep)\n' - '\n' - ' Execute the current line, stop at the first possible ' - 'occasion\n' - ' (either in a function that is called or on the next line in ' - 'the\n' - ' current function).\n' - '\n' - 'n(ext)\n' - '\n' - ' Continue execution until the next line in the current ' - 'function is\n' - ' reached or it returns. (The difference between "next" and ' - '"step"\n' - ' is that "step" stops inside a called function, while "next"\n' - ' executes called functions at (nearly) full speed, only ' - 'stopping at\n' - ' the next line in the current function.)\n' - '\n' - 'unt(il) [lineno]\n' - '\n' - ' Without argument, continue execution until the line with a ' - 'number\n' - ' greater than the current one is reached.\n' - '\n' - ' With a line number, continue execution until a line with a ' - 'number\n' - ' greater or equal to that is reached. In both cases, also ' - 'stop when\n' - ' the current frame returns.\n' - '\n' - ' Changed in version 3.2: Allow giving an explicit line ' - 'number.\n' - '\n' - 'r(eturn)\n' - '\n' - ' Continue execution until the current function returns.\n' - '\n' - 'c(ont(inue))\n' - '\n' - ' Continue execution, only stop when a breakpoint is ' - 'encountered.\n' - '\n' - 'j(ump) lineno\n' - '\n' - ' Set the next line that will be executed. Only available in ' - 'the\n' - ' bottom-most frame. This lets you jump back and execute code ' - 'again,\n' - ' or jump forward to skip code that you don’t want to run.\n' - '\n' - ' It should be noted that not all jumps are allowed – for ' - 'instance it\n' - ' is not possible to jump into the middle of a "for" loop or ' - 'out of a\n' - ' "finally" clause.\n' - '\n' - 'l(ist) [first[, last]]\n' - '\n' - ' List source code for the current file. Without arguments, ' - 'list 11\n' - ' lines around the current line or continue the previous ' - 'listing.\n' - ' With "." as argument, list 11 lines around the current line. ' - 'With\n' - ' one argument, list 11 lines around at that line. With two\n' - ' arguments, list the given range; if the second argument is ' - 'less\n' - ' than the first, it is interpreted as a count.\n' - '\n' - ' The current line in the current frame is indicated by "->". ' - 'If an\n' - ' exception is being debugged, the line where the exception ' - 'was\n' - ' originally raised or propagated is indicated by ">>", if it ' - 'differs\n' - ' from the current line.\n' - '\n' - ' New in version 3.2: The ">>" marker.\n' - '\n' - 'll | longlist\n' - '\n' - ' List all source code for the current function or frame.\n' - ' Interesting lines are marked as for "list".\n' - '\n' - ' New in version 3.2.\n' - '\n' - 'a(rgs)\n' - '\n' - ' Print the argument list of the current function.\n' - '\n' - 'p expression\n' - '\n' - ' Evaluate the *expression* in the current context and print ' - 'its\n' - ' value.\n' - '\n' - ' Note: "print()" can also be used, but is not a debugger ' - 'command —\n' - ' this executes the Python "print()" function.\n' - '\n' - 'pp expression\n' - '\n' - ' Like the "p" command, except the value of the expression is ' - 'pretty-\n' - ' printed using the "pprint" module.\n' - '\n' - 'whatis expression\n' - '\n' - ' Print the type of the *expression*.\n' - '\n' - 'source expression\n' - '\n' - ' Try to get source code for the given object and display it.\n' - '\n' - ' New in version 3.2.\n' - '\n' - 'display [expression]\n' - '\n' - ' Display the value of the expression if it changed, each time\n' - ' execution stops in the current frame.\n' - '\n' - ' Without expression, list all display expressions for the ' - 'current\n' - ' frame.\n' - '\n' - ' New in version 3.2.\n' - '\n' - 'undisplay [expression]\n' - '\n' - ' Do not display the expression any more in the current frame.\n' - ' Without expression, clear all display expressions for the ' - 'current\n' - ' frame.\n' - '\n' - ' New in version 3.2.\n' - '\n' - 'interact\n' - '\n' - ' Start an interactive interpreter (using the "code" module) ' - 'whose\n' - ' global namespace contains all the (global and local) names ' - 'found in\n' - ' the current scope.\n' - '\n' - ' New in version 3.2.\n' - '\n' - 'alias [name [command]]\n' - '\n' - ' Create an alias called *name* that executes *command*. The ' - 'command\n' - ' must *not* be enclosed in quotes. Replaceable parameters can ' - 'be\n' - ' indicated by "%1", "%2", and so on, while "%*" is replaced by ' - 'all\n' - ' the parameters. If no command is given, the current alias ' - 'for\n' - ' *name* is shown. If no arguments are given, all aliases are ' - 'listed.\n' - '\n' - ' Aliases may be nested and can contain anything that can be ' - 'legally\n' - ' typed at the pdb prompt. Note that internal pdb commands ' - '*can* be\n' - ' overridden by aliases. Such a command is then hidden until ' - 'the\n' - ' alias is removed. Aliasing is recursively applied to the ' - 'first\n' - ' word of the command line; all other words in the line are ' - 'left\n' - ' alone.\n' - '\n' - ' As an example, here are two useful aliases (especially when ' - 'placed\n' - ' in the ".pdbrc" file):\n' - '\n' - ' # Print instance variables (usage "pi classInst")\n' - ' alias pi for k in %1.__dict__.keys(): ' - 'print("%1.",k,"=",%1.__dict__[k])\n' - ' # Print instance variables in self\n' - ' alias ps pi self\n' - '\n' - 'unalias name\n' - '\n' - ' Delete the specified alias.\n' - '\n' - '! statement\n' - '\n' - ' Execute the (one-line) *statement* in the context of the ' - 'current\n' - ' stack frame. The exclamation point can be omitted unless the ' - 'first\n' - ' word of the statement resembles a debugger command. To set ' - 'a\n' - ' global variable, you can prefix the assignment command with ' - 'a\n' - ' "global" statement on the same line, e.g.:\n' - '\n' - " (Pdb) global list_options; list_options = ['-l']\n" - ' (Pdb)\n' - '\n' - 'run [args ...]\n' - 'restart [args ...]\n' - '\n' - ' Restart the debugged Python program. If an argument is ' - 'supplied,\n' - ' it is split with "shlex" and the result is used as the new\n' - ' "sys.argv". History, breakpoints, actions and debugger ' - 'options are\n' - ' preserved. "restart" is an alias for "run".\n' - '\n' - 'q(uit)\n' - '\n' - ' Quit from the debugger. The program being executed is ' - 'aborted.\n' - '\n' - '-[ Footnotes ]-\n' - '\n' - '[1] Whether a frame is considered to originate in a certain ' - 'module\n' - ' is determined by the "__name__" in the frame globals.\n', - 'del': 'The "del" statement\n' - '*******************\n' - '\n' - ' del_stmt ::= "del" target_list\n' - '\n' - 'Deletion is recursively defined very similar to the way assignment ' - 'is\n' - 'defined. Rather than spelling it out in full details, here are some\n' - 'hints.\n' - '\n' - 'Deletion of a target list recursively deletes each target, from left\n' - 'to right.\n' - '\n' - 'Deletion of a name removes the binding of that name from the local ' - 'or\n' - 'global namespace, depending on whether the name occurs in a "global"\n' - 'statement in the same code block. If the name is unbound, a\n' - '"NameError" exception will be raised.\n' - '\n' - 'Deletion of attribute references, subscriptions and slicings is ' - 'passed\n' - 'to the primary object involved; deletion of a slicing is in general\n' - 'equivalent to assignment of an empty slice of the right type (but ' - 'even\n' - 'this is determined by the sliced object).\n' - '\n' - 'Changed in version 3.2: Previously it was illegal to delete a name\n' - 'from the local namespace if it occurs as a free variable in a nested\n' - 'block.\n', - 'dict': 'Dictionary displays\n' - '*******************\n' - '\n' - 'A dictionary display is a possibly empty series of key/datum pairs\n' - 'enclosed in curly braces:\n' - '\n' - ' dict_display ::= "{" [key_datum_list | dict_comprehension] ' - '"}"\n' - ' key_datum_list ::= key_datum ("," key_datum)* [","]\n' - ' key_datum ::= expression ":" expression | "**" or_expr\n' - ' dict_comprehension ::= expression ":" expression comp_for\n' - '\n' - 'A dictionary display yields a new dictionary object.\n' - '\n' - 'If a comma-separated sequence of key/datum pairs is given, they are\n' - 'evaluated from left to right to define the entries of the ' - 'dictionary:\n' - 'each key object is used as a key into the dictionary to store the\n' - 'corresponding datum. This means that you can specify the same key\n' - 'multiple times in the key/datum list, and the final dictionary’s ' - 'value\n' - 'for that key will be the last one given.\n' - '\n' - 'A double asterisk "**" denotes *dictionary unpacking*. Its operand\n' - 'must be a *mapping*. Each mapping item is added to the new\n' - 'dictionary. Later values replace values already set by earlier\n' - 'key/datum pairs and earlier dictionary unpackings.\n' - '\n' - 'New in version 3.5: Unpacking into dictionary displays, originally\n' - 'proposed by **PEP 448**.\n' - '\n' - 'A dict comprehension, in contrast to list and set comprehensions,\n' - 'needs two expressions separated with a colon followed by the usual\n' - '“for” and “if” clauses. When the comprehension is run, the ' - 'resulting\n' - 'key and value elements are inserted in the new dictionary in the ' - 'order\n' - 'they are produced.\n' - '\n' - 'Restrictions on the types of the key values are listed earlier in\n' - 'section The standard type hierarchy. (To summarize, the key type\n' - 'should be *hashable*, which excludes all mutable objects.) Clashes\n' - 'between duplicate keys are not detected; the last datum (textually\n' - 'rightmost in the display) stored for a given key value prevails.\n', - 'dynamic-features': 'Interaction with dynamic features\n' - '*********************************\n' - '\n' - 'Name resolution of free variables occurs at runtime, not ' - 'at compile\n' - 'time. This means that the following code will print 42:\n' - '\n' - ' i = 10\n' - ' def f():\n' - ' print(i)\n' - ' i = 42\n' - ' f()\n' - '\n' - 'The "eval()" and "exec()" functions do not have access ' - 'to the full\n' - 'environment for resolving names. Names may be resolved ' - 'in the local\n' - 'and global namespaces of the caller. Free variables are ' - 'not resolved\n' - 'in the nearest enclosing namespace, but in the global ' - 'namespace. [1]\n' - 'The "exec()" and "eval()" functions have optional ' - 'arguments to\n' - 'override the global and local namespace. If only one ' - 'namespace is\n' - 'specified, it is used for both.\n', - 'else': 'The "if" statement\n' - '******************\n' - '\n' - 'The "if" statement is used for conditional execution:\n' - '\n' - ' if_stmt ::= "if" expression ":" suite\n' - ' ("elif" expression ":" suite)*\n' - ' ["else" ":" suite]\n' - '\n' - 'It selects exactly one of the suites by evaluating the expressions ' - 'one\n' - 'by one until one is found to be true (see section Boolean ' - 'operations\n' - 'for the definition of true and false); then that suite is executed\n' - '(and no other part of the "if" statement is executed or evaluated).\n' - 'If all expressions are false, the suite of the "else" clause, if\n' - 'present, is executed.\n', - 'exceptions': 'Exceptions\n' - '**********\n' - '\n' - 'Exceptions are a means of breaking out of the normal flow of ' - 'control\n' - 'of a code block in order to handle errors or other ' - 'exceptional\n' - 'conditions. An exception is *raised* at the point where the ' - 'error is\n' - 'detected; it may be *handled* by the surrounding code block or ' - 'by any\n' - 'code block that directly or indirectly invoked the code block ' - 'where\n' - 'the error occurred.\n' - '\n' - 'The Python interpreter raises an exception when it detects a ' - 'run-time\n' - 'error (such as division by zero). A Python program can also\n' - 'explicitly raise an exception with the "raise" statement. ' - 'Exception\n' - 'handlers are specified with the "try" … "except" statement. ' - 'The\n' - '"finally" clause of such a statement can be used to specify ' - 'cleanup\n' - 'code which does not handle the exception, but is executed ' - 'whether an\n' - 'exception occurred or not in the preceding code.\n' - '\n' - 'Python uses the “termination” model of error handling: an ' - 'exception\n' - 'handler can find out what happened and continue execution at ' - 'an outer\n' - 'level, but it cannot repair the cause of the error and retry ' - 'the\n' - 'failing operation (except by re-entering the offending piece ' - 'of code\n' - 'from the top).\n' - '\n' - 'When an exception is not handled at all, the interpreter ' - 'terminates\n' - 'execution of the program, or returns to its interactive main ' - 'loop. In\n' - 'either case, it prints a stack backtrace, except when the ' - 'exception is\n' - '"SystemExit".\n' - '\n' - 'Exceptions are identified by class instances. The "except" ' - 'clause is\n' - 'selected depending on the class of the instance: it must ' - 'reference the\n' - 'class of the instance or a base class thereof. The instance ' - 'can be\n' - 'received by the handler and can carry additional information ' - 'about the\n' - 'exceptional condition.\n' - '\n' - 'Note: Exception messages are not part of the Python API. ' - 'Their\n' - ' contents may change from one version of Python to the next ' - 'without\n' - ' warning and should not be relied on by code which will run ' - 'under\n' - ' multiple versions of the interpreter.\n' - '\n' - 'See also the description of the "try" statement in section The ' - 'try\n' - 'statement and "raise" statement in section The raise ' - 'statement.\n' - '\n' - '-[ Footnotes ]-\n' - '\n' - '[1] This limitation occurs because the code that is executed ' - 'by\n' - ' these operations is not available at the time the module ' - 'is\n' - ' compiled.\n', - 'execmodel': 'Execution model\n' - '***************\n' - '\n' - '\n' - 'Structure of a program\n' - '======================\n' - '\n' - 'A Python program is constructed from code blocks. A *block* is ' - 'a piece\n' - 'of Python program text that is executed as a unit. The ' - 'following are\n' - 'blocks: a module, a function body, and a class definition. ' - 'Each\n' - 'command typed interactively is a block. A script file (a file ' - 'given\n' - 'as standard input to the interpreter or specified as a command ' - 'line\n' - 'argument to the interpreter) is a code block. A script command ' - '(a\n' - 'command specified on the interpreter command line with the ' - '"-c"\n' - 'option) is a code block. The string argument passed to the ' - 'built-in\n' - 'functions "eval()" and "exec()" is a code block.\n' - '\n' - 'A code block is executed in an *execution frame*. A frame ' - 'contains\n' - 'some administrative information (used for debugging) and ' - 'determines\n' - 'where and how execution continues after the code block’s ' - 'execution has\n' - 'completed.\n' - '\n' - '\n' - 'Naming and binding\n' - '==================\n' - '\n' - '\n' - 'Binding of names\n' - '----------------\n' - '\n' - '*Names* refer to objects. Names are introduced by name ' - 'binding\n' - 'operations.\n' - '\n' - 'The following constructs bind names: formal parameters to ' - 'functions,\n' - '"import" statements, class and function definitions (these bind ' - 'the\n' - 'class or function name in the defining block), and targets that ' - 'are\n' - 'identifiers if occurring in an assignment, "for" loop header, ' - 'or after\n' - '"as" in a "with" statement or "except" clause. The "import" ' - 'statement\n' - 'of the form "from ... import *" binds all names defined in the\n' - 'imported module, except those beginning with an underscore. ' - 'This form\n' - 'may only be used at the module level.\n' - '\n' - 'A target occurring in a "del" statement is also considered ' - 'bound for\n' - 'this purpose (though the actual semantics are to unbind the ' - 'name).\n' - '\n' - 'Each assignment or import statement occurs within a block ' - 'defined by a\n' - 'class or function definition or at the module level (the ' - 'top-level\n' - 'code block).\n' - '\n' - 'If a name is bound in a block, it is a local variable of that ' - 'block,\n' - 'unless declared as "nonlocal" or "global". If a name is bound ' - 'at the\n' - 'module level, it is a global variable. (The variables of the ' - 'module\n' - 'code block are local and global.) If a variable is used in a ' - 'code\n' - 'block but not defined there, it is a *free variable*.\n' - '\n' - 'Each occurrence of a name in the program text refers to the ' - '*binding*\n' - 'of that name established by the following name resolution ' - 'rules.\n' - '\n' - '\n' - 'Resolution of names\n' - '-------------------\n' - '\n' - 'A *scope* defines the visibility of a name within a block. If ' - 'a local\n' - 'variable is defined in a block, its scope includes that block. ' - 'If the\n' - 'definition occurs in a function block, the scope extends to any ' - 'blocks\n' - 'contained within the defining one, unless a contained block ' - 'introduces\n' - 'a different binding for the name.\n' - '\n' - 'When a name is used in a code block, it is resolved using the ' - 'nearest\n' - 'enclosing scope. The set of all such scopes visible to a code ' - 'block\n' - 'is called the block’s *environment*.\n' - '\n' - 'When a name is not found at all, a "NameError" exception is ' - 'raised. If\n' - 'the current scope is a function scope, and the name refers to a ' - 'local\n' - 'variable that has not yet been bound to a value at the point ' - 'where the\n' - 'name is used, an "UnboundLocalError" exception is raised.\n' - '"UnboundLocalError" is a subclass of "NameError".\n' - '\n' - 'If a name binding operation occurs anywhere within a code ' - 'block, all\n' - 'uses of the name within the block are treated as references to ' - 'the\n' - 'current block. This can lead to errors when a name is used ' - 'within a\n' - 'block before it is bound. This rule is subtle. Python lacks\n' - 'declarations and allows name binding operations to occur ' - 'anywhere\n' - 'within a code block. The local variables of a code block can ' - 'be\n' - 'determined by scanning the entire text of the block for name ' - 'binding\n' - 'operations.\n' - '\n' - 'If the "global" statement occurs within a block, all uses of ' - 'the name\n' - 'specified in the statement refer to the binding of that name in ' - 'the\n' - 'top-level namespace. Names are resolved in the top-level ' - 'namespace by\n' - 'searching the global namespace, i.e. the namespace of the ' - 'module\n' - 'containing the code block, and the builtins namespace, the ' - 'namespace\n' - 'of the module "builtins". The global namespace is searched ' - 'first. If\n' - 'the name is not found there, the builtins namespace is ' - 'searched. The\n' - '"global" statement must precede all uses of the name.\n' - '\n' - 'The "global" statement has the same scope as a name binding ' - 'operation\n' - 'in the same block. If the nearest enclosing scope for a free ' - 'variable\n' - 'contains a global statement, the free variable is treated as a ' - 'global.\n' - '\n' - 'The "nonlocal" statement causes corresponding names to refer ' - 'to\n' - 'previously bound variables in the nearest enclosing function ' - 'scope.\n' - '"SyntaxError" is raised at compile time if the given name does ' - 'not\n' - 'exist in any enclosing function scope.\n' - '\n' - 'The namespace for a module is automatically created the first ' - 'time a\n' - 'module is imported. The main module for a script is always ' - 'called\n' - '"__main__".\n' - '\n' - 'Class definition blocks and arguments to "exec()" and "eval()" ' - 'are\n' - 'special in the context of name resolution. A class definition ' - 'is an\n' - 'executable statement that may use and define names. These ' - 'references\n' - 'follow the normal rules for name resolution with an exception ' - 'that\n' - 'unbound local variables are looked up in the global namespace. ' - 'The\n' - 'namespace of the class definition becomes the attribute ' - 'dictionary of\n' - 'the class. The scope of names defined in a class block is ' - 'limited to\n' - 'the class block; it does not extend to the code blocks of ' - 'methods –\n' - 'this includes comprehensions and generator expressions since ' - 'they are\n' - 'implemented using a function scope. This means that the ' - 'following\n' - 'will fail:\n' - '\n' - ' class A:\n' - ' a = 42\n' - ' b = list(a + i for i in range(10))\n' - '\n' - '\n' - 'Builtins and restricted execution\n' - '---------------------------------\n' - '\n' - '**CPython implementation detail:** Users should not touch\n' - '"__builtins__"; it is strictly an implementation detail. ' - 'Users\n' - 'wanting to override values in the builtins namespace should ' - '"import"\n' - 'the "builtins" module and modify its attributes appropriately.\n' - '\n' - 'The builtins namespace associated with the execution of a code ' - 'block\n' - 'is actually found by looking up the name "__builtins__" in its ' - 'global\n' - 'namespace; this should be a dictionary or a module (in the ' - 'latter case\n' - 'the module’s dictionary is used). By default, when in the ' - '"__main__"\n' - 'module, "__builtins__" is the built-in module "builtins"; when ' - 'in any\n' - 'other module, "__builtins__" is an alias for the dictionary of ' - 'the\n' - '"builtins" module itself.\n' - '\n' - '\n' - 'Interaction with dynamic features\n' - '---------------------------------\n' - '\n' - 'Name resolution of free variables occurs at runtime, not at ' - 'compile\n' - 'time. This means that the following code will print 42:\n' - '\n' - ' i = 10\n' - ' def f():\n' - ' print(i)\n' - ' i = 42\n' - ' f()\n' - '\n' - 'The "eval()" and "exec()" functions do not have access to the ' - 'full\n' - 'environment for resolving names. Names may be resolved in the ' - 'local\n' - 'and global namespaces of the caller. Free variables are not ' - 'resolved\n' - 'in the nearest enclosing namespace, but in the global ' - 'namespace. [1]\n' - 'The "exec()" and "eval()" functions have optional arguments to\n' - 'override the global and local namespace. If only one namespace ' - 'is\n' - 'specified, it is used for both.\n' - '\n' - '\n' - 'Exceptions\n' - '==========\n' - '\n' - 'Exceptions are a means of breaking out of the normal flow of ' - 'control\n' - 'of a code block in order to handle errors or other exceptional\n' - 'conditions. An exception is *raised* at the point where the ' - 'error is\n' - 'detected; it may be *handled* by the surrounding code block or ' - 'by any\n' - 'code block that directly or indirectly invoked the code block ' - 'where\n' - 'the error occurred.\n' - '\n' - 'The Python interpreter raises an exception when it detects a ' - 'run-time\n' - 'error (such as division by zero). A Python program can also\n' - 'explicitly raise an exception with the "raise" statement. ' - 'Exception\n' - 'handlers are specified with the "try" … "except" statement. ' - 'The\n' - '"finally" clause of such a statement can be used to specify ' - 'cleanup\n' - 'code which does not handle the exception, but is executed ' - 'whether an\n' - 'exception occurred or not in the preceding code.\n' - '\n' - 'Python uses the “termination” model of error handling: an ' - 'exception\n' - 'handler can find out what happened and continue execution at an ' - 'outer\n' - 'level, but it cannot repair the cause of the error and retry ' - 'the\n' - 'failing operation (except by re-entering the offending piece of ' - 'code\n' - 'from the top).\n' - '\n' - 'When an exception is not handled at all, the interpreter ' - 'terminates\n' - 'execution of the program, or returns to its interactive main ' - 'loop. In\n' - 'either case, it prints a stack backtrace, except when the ' - 'exception is\n' - '"SystemExit".\n' - '\n' - 'Exceptions are identified by class instances. The "except" ' - 'clause is\n' - 'selected depending on the class of the instance: it must ' - 'reference the\n' - 'class of the instance or a base class thereof. The instance ' - 'can be\n' - 'received by the handler and can carry additional information ' - 'about the\n' - 'exceptional condition.\n' - '\n' - 'Note: Exception messages are not part of the Python API. ' - 'Their\n' - ' contents may change from one version of Python to the next ' - 'without\n' - ' warning and should not be relied on by code which will run ' - 'under\n' - ' multiple versions of the interpreter.\n' - '\n' - 'See also the description of the "try" statement in section The ' - 'try\n' - 'statement and "raise" statement in section The raise ' - 'statement.\n' - '\n' - '-[ Footnotes ]-\n' - '\n' - '[1] This limitation occurs because the code that is executed ' - 'by\n' - ' these operations is not available at the time the module ' - 'is\n' - ' compiled.\n', - 'exprlists': 'Expression lists\n' - '****************\n' - '\n' - ' expression_list ::= expression ("," expression)* [","]\n' - ' starred_list ::= starred_item ("," starred_item)* ' - '[","]\n' - ' starred_expression ::= expression | (starred_item ",")* ' - '[starred_item]\n' - ' starred_item ::= expression | "*" or_expr\n' - '\n' - 'Except when part of a list or set display, an expression list\n' - 'containing at least one comma yields a tuple. The length of ' - 'the tuple\n' - 'is the number of expressions in the list. The expressions are\n' - 'evaluated from left to right.\n' - '\n' - 'An asterisk "*" denotes *iterable unpacking*. Its operand must ' - 'be an\n' - '*iterable*. The iterable is expanded into a sequence of items, ' - 'which\n' - 'are included in the new tuple, list, or set, at the site of ' - 'the\n' - 'unpacking.\n' - '\n' - 'New in version 3.5: Iterable unpacking in expression lists, ' - 'originally\n' - 'proposed by **PEP 448**.\n' - '\n' - 'The trailing comma is required only to create a single tuple ' - '(a.k.a. a\n' - '*singleton*); it is optional in all other cases. A single ' - 'expression\n' - 'without a trailing comma doesn’t create a tuple, but rather ' - 'yields the\n' - 'value of that expression. (To create an empty tuple, use an ' - 'empty pair\n' - 'of parentheses: "()".)\n', - 'floating': 'Floating point literals\n' - '***********************\n' - '\n' - 'Floating point literals are described by the following lexical\n' - 'definitions:\n' - '\n' - ' floatnumber ::= pointfloat | exponentfloat\n' - ' pointfloat ::= [digitpart] fraction | digitpart "."\n' - ' exponentfloat ::= (digitpart | pointfloat) exponent\n' - ' digitpart ::= digit (["_"] digit)*\n' - ' fraction ::= "." digitpart\n' - ' exponent ::= ("e" | "E") ["+" | "-"] digitpart\n' - '\n' - 'Note that the integer and exponent parts are always interpreted ' - 'using\n' - 'radix 10. For example, "077e010" is legal, and denotes the same ' - 'number\n' - 'as "77e10". The allowed range of floating point literals is\n' - 'implementation-dependent. As in integer literals, underscores ' - 'are\n' - 'supported for digit grouping.\n' - '\n' - 'Some examples of floating point literals:\n' - '\n' - ' 3.14 10. .001 1e100 3.14e-10 0e0 ' - '3.14_15_93\n' - '\n' - 'Changed in version 3.6: Underscores are now allowed for ' - 'grouping\n' - 'purposes in literals.\n', - 'for': 'The "for" statement\n' - '*******************\n' - '\n' - 'The "for" statement is used to iterate over the elements of a ' - 'sequence\n' - '(such as a string, tuple or list) or other iterable object:\n' - '\n' - ' for_stmt ::= "for" target_list "in" expression_list ":" suite\n' - ' ["else" ":" suite]\n' - '\n' - 'The expression list is evaluated once; it should yield an iterable\n' - 'object. An iterator is created for the result of the\n' - '"expression_list". The suite is then executed once for each item\n' - 'provided by the iterator, in the order returned by the iterator. ' - 'Each\n' - 'item in turn is assigned to the target list using the standard rules\n' - 'for assignments (see Assignment statements), and then the suite is\n' - 'executed. When the items are exhausted (which is immediately when ' - 'the\n' - 'sequence is empty or an iterator raises a "StopIteration" ' - 'exception),\n' - 'the suite in the "else" clause, if present, is executed, and the ' - 'loop\n' - 'terminates.\n' - '\n' - 'A "break" statement executed in the first suite terminates the loop\n' - 'without executing the "else" clause’s suite. A "continue" statement\n' - 'executed in the first suite skips the rest of the suite and ' - 'continues\n' - 'with the next item, or with the "else" clause if there is no next\n' - 'item.\n' - '\n' - 'The for-loop makes assignments to the variables(s) in the target ' - 'list.\n' - 'This overwrites all previous assignments to those variables ' - 'including\n' - 'those made in the suite of the for-loop:\n' - '\n' - ' for i in range(10):\n' - ' print(i)\n' - ' i = 5 # this will not affect the for-loop\n' - ' # because i will be overwritten with the ' - 'next\n' - ' # index in the range\n' - '\n' - 'Names in the target list are not deleted when the loop is finished,\n' - 'but if the sequence is empty, they will not have been assigned to at\n' - 'all by the loop. Hint: the built-in function "range()" returns an\n' - 'iterator of integers suitable to emulate the effect of Pascal’s "for ' - 'i\n' - ':= a to b do"; e.g., "list(range(3))" returns the list "[0, 1, 2]".\n' - '\n' - 'Note: There is a subtlety when the sequence is being modified by the\n' - ' loop (this can only occur for mutable sequences, e.g. lists). An\n' - ' internal counter is used to keep track of which item is used next,\n' - ' and this is incremented on each iteration. When this counter has\n' - ' reached the length of the sequence the loop terminates. This ' - 'means\n' - ' that if the suite deletes the current (or a previous) item from ' - 'the\n' - ' sequence, the next item will be skipped (since it gets the index ' - 'of\n' - ' the current item which has already been treated). Likewise, if ' - 'the\n' - ' suite inserts an item in the sequence before the current item, the\n' - ' current item will be treated again the next time through the loop.\n' - ' This can lead to nasty bugs that can be avoided by making a\n' - ' temporary copy using a slice of the whole sequence, e.g.,\n' - '\n' - ' for x in a[:]:\n' - ' if x < 0: a.remove(x)\n', - 'formatstrings': 'Format String Syntax\n' - '********************\n' - '\n' - 'The "str.format()" method and the "Formatter" class share ' - 'the same\n' - 'syntax for format strings (although in the case of ' - '"Formatter",\n' - 'subclasses can define their own format string syntax). The ' - 'syntax is\n' - 'related to that of formatted string literals, but there ' - 'are\n' - 'differences.\n' - '\n' - 'Format strings contain “replacement fields” surrounded by ' - 'curly braces\n' - '"{}". Anything that is not contained in braces is ' - 'considered literal\n' - 'text, which is copied unchanged to the output. If you need ' - 'to include\n' - 'a brace character in the literal text, it can be escaped by ' - 'doubling:\n' - '"{{" and "}}".\n' - '\n' - 'The grammar for a replacement field is as follows:\n' - '\n' - ' replacement_field ::= "{" [field_name] ["!" ' - 'conversion] [":" format_spec] "}"\n' - ' field_name ::= arg_name ("." attribute_name | ' - '"[" element_index "]")*\n' - ' arg_name ::= [identifier | digit+]\n' - ' attribute_name ::= identifier\n' - ' element_index ::= digit+ | index_string\n' - ' index_string ::= +\n' - ' conversion ::= "r" | "s" | "a"\n' - ' format_spec ::= \n' - '\n' - 'In less formal terms, the replacement field can start with ' - 'a\n' - '*field_name* that specifies the object whose value is to be ' - 'formatted\n' - 'and inserted into the output instead of the replacement ' - 'field. The\n' - '*field_name* is optionally followed by a *conversion* ' - 'field, which is\n' - 'preceded by an exclamation point "\'!\'", and a ' - '*format_spec*, which is\n' - 'preceded by a colon "\':\'". These specify a non-default ' - 'format for the\n' - 'replacement value.\n' - '\n' - 'See also the Format Specification Mini-Language section.\n' - '\n' - 'The *field_name* itself begins with an *arg_name* that is ' - 'either a\n' - 'number or a keyword. If it’s a number, it refers to a ' - 'positional\n' - 'argument, and if it’s a keyword, it refers to a named ' - 'keyword\n' - 'argument. If the numerical arg_names in a format string ' - 'are 0, 1, 2,\n' - '… in sequence, they can all be omitted (not just some) and ' - 'the numbers\n' - '0, 1, 2, … will be automatically inserted in that order. ' - 'Because\n' - '*arg_name* is not quote-delimited, it is not possible to ' - 'specify\n' - 'arbitrary dictionary keys (e.g., the strings "\'10\'" or ' - '"\':-]\'") within\n' - 'a format string. The *arg_name* can be followed by any ' - 'number of index\n' - 'or attribute expressions. An expression of the form ' - '"\'.name\'" selects\n' - 'the named attribute using "getattr()", while an expression ' - 'of the form\n' - '"\'[index]\'" does an index lookup using "__getitem__()".\n' - '\n' - 'Changed in version 3.1: The positional argument specifiers ' - 'can be\n' - 'omitted for "str.format()", so "\'{} {}\'.format(a, b)" is ' - 'equivalent to\n' - '"\'{0} {1}\'.format(a, b)".\n' - '\n' - 'Changed in version 3.4: The positional argument specifiers ' - 'can be\n' - 'omitted for "Formatter".\n' - '\n' - 'Some simple format string examples:\n' - '\n' - ' "First, thou shalt count to {0}" # References first ' - 'positional argument\n' - ' "Bring me a {}" # Implicitly ' - 'references the first positional argument\n' - ' "From {} to {}" # Same as "From {0} to ' - '{1}"\n' - ' "My quest is {name}" # References keyword ' - "argument 'name'\n" - ' "Weight in tons {0.weight}" # \'weight\' attribute ' - 'of first positional arg\n' - ' "Units destroyed: {players[0]}" # First element of ' - "keyword argument 'players'.\n" - '\n' - 'The *conversion* field causes a type coercion before ' - 'formatting.\n' - 'Normally, the job of formatting a value is done by the ' - '"__format__()"\n' - 'method of the value itself. However, in some cases it is ' - 'desirable to\n' - 'force a type to be formatted as a string, overriding its ' - 'own\n' - 'definition of formatting. By converting the value to a ' - 'string before\n' - 'calling "__format__()", the normal formatting logic is ' - 'bypassed.\n' - '\n' - 'Three conversion flags are currently supported: "\'!s\'" ' - 'which calls\n' - '"str()" on the value, "\'!r\'" which calls "repr()" and ' - '"\'!a\'" which\n' - 'calls "ascii()".\n' - '\n' - 'Some examples:\n' - '\n' - ' "Harold\'s a clever {0!s}" # Calls str() on the ' - 'argument first\n' - ' "Bring out the holy {name!r}" # Calls repr() on the ' - 'argument first\n' - ' "More {!a}" # Calls ascii() on the ' - 'argument first\n' - '\n' - 'The *format_spec* field contains a specification of how the ' - 'value\n' - 'should be presented, including such details as field width, ' - 'alignment,\n' - 'padding, decimal precision and so on. Each value type can ' - 'define its\n' - 'own “formatting mini-language” or interpretation of the ' - '*format_spec*.\n' - '\n' - 'Most built-in types support a common formatting ' - 'mini-language, which\n' - 'is described in the next section.\n' - '\n' - 'A *format_spec* field can also include nested replacement ' - 'fields\n' - 'within it. These nested replacement fields may contain a ' - 'field name,\n' - 'conversion flag and format specification, but deeper ' - 'nesting is not\n' - 'allowed. The replacement fields within the format_spec ' - 'are\n' - 'substituted before the *format_spec* string is interpreted. ' - 'This\n' - 'allows the formatting of a value to be dynamically ' - 'specified.\n' - '\n' - 'See the Format examples section for some examples.\n' - '\n' - '\n' - 'Format Specification Mini-Language\n' - '==================================\n' - '\n' - '“Format specifications” are used within replacement fields ' - 'contained\n' - 'within a format string to define how individual values are ' - 'presented\n' - '(see Format String Syntax and Formatted string literals). ' - 'They can\n' - 'also be passed directly to the built-in "format()" ' - 'function. Each\n' - 'formattable type may define how the format specification is ' - 'to be\n' - 'interpreted.\n' - '\n' - 'Most built-in types implement the following options for ' - 'format\n' - 'specifications, although some of the formatting options are ' - 'only\n' - 'supported by the numeric types.\n' - '\n' - 'A general convention is that an empty format string ("""") ' - 'produces\n' - 'the same result as if you had called "str()" on the value. ' - 'A non-empty\n' - 'format string typically modifies the result.\n' - '\n' - 'The general form of a *standard format specifier* is:\n' - '\n' - ' format_spec ::= ' - '[[fill]align][sign][#][0][width][grouping_option][.precision][type]\n' - ' fill ::= \n' - ' align ::= "<" | ">" | "=" | "^"\n' - ' sign ::= "+" | "-" | " "\n' - ' width ::= digit+\n' - ' grouping_option ::= "_" | ","\n' - ' precision ::= digit+\n' - ' type ::= "b" | "c" | "d" | "e" | "E" | "f" | ' - '"F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"\n' - '\n' - 'If a valid *align* value is specified, it can be preceded ' - 'by a *fill*\n' - 'character that can be any character and defaults to a space ' - 'if\n' - 'omitted. It is not possible to use a literal curly brace ' - '(“"{"” or\n' - '“"}"”) as the *fill* character in a formatted string ' - 'literal or when\n' - 'using the "str.format()" method. However, it is possible ' - 'to insert a\n' - 'curly brace with a nested replacement field. This ' - 'limitation doesn’t\n' - 'affect the "format()" function.\n' - '\n' - 'The meaning of the various alignment options is as ' - 'follows:\n' - '\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | Option | ' - 'Meaning ' - '|\n' - ' ' - '+===========+============================================================+\n' - ' | "\'<\'" | Forces the field to be left-aligned ' - 'within the available |\n' - ' | | space (this is the default for most ' - 'objects). |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | "\'>\'" | Forces the field to be right-aligned ' - 'within the available |\n' - ' | | space (this is the default for ' - 'numbers). |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | "\'=\'" | Forces the padding to be placed after ' - 'the sign (if any) |\n' - ' | | but before the digits. This is used for ' - 'printing fields |\n' - ' | | in the form ‘+000000120’. This alignment ' - 'option is only |\n' - ' | | valid for numeric types. It becomes the ' - 'default when ‘0’ |\n' - ' | | immediately precedes the field ' - 'width. |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | "\'^\'" | Forces the field to be centered within ' - 'the available |\n' - ' | | ' - 'space. ' - '|\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - '\n' - 'Note that unless a minimum field width is defined, the ' - 'field width\n' - 'will always be the same size as the data to fill it, so ' - 'that the\n' - 'alignment option has no meaning in this case.\n' - '\n' - 'The *sign* option is only valid for number types, and can ' - 'be one of\n' - 'the following:\n' - '\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | Option | ' - 'Meaning ' - '|\n' - ' ' - '+===========+============================================================+\n' - ' | "\'+\'" | indicates that a sign should be used for ' - 'both positive as |\n' - ' | | well as negative ' - 'numbers. |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | "\'-\'" | indicates that a sign should be used ' - 'only for negative |\n' - ' | | numbers (this is the default ' - 'behavior). |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | space | indicates that a leading space should be ' - 'used on positive |\n' - ' | | numbers, and a minus sign on negative ' - 'numbers. |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - '\n' - 'The "\'#\'" option causes the “alternate form” to be used ' - 'for the\n' - 'conversion. The alternate form is defined differently for ' - 'different\n' - 'types. This option is only valid for integer, float, ' - 'complex and\n' - 'Decimal types. For integers, when binary, octal, or ' - 'hexadecimal output\n' - 'is used, this option adds the prefix respective "\'0b\'", ' - '"\'0o\'", or\n' - '"\'0x\'" to the output value. For floats, complex and ' - 'Decimal the\n' - 'alternate form causes the result of the conversion to ' - 'always contain a\n' - 'decimal-point character, even if no digits follow it. ' - 'Normally, a\n' - 'decimal-point character appears in the result of these ' - 'conversions\n' - 'only if a digit follows it. In addition, for "\'g\'" and ' - '"\'G\'"\n' - 'conversions, trailing zeros are not removed from the ' - 'result.\n' - '\n' - 'The "\',\'" option signals the use of a comma for a ' - 'thousands separator.\n' - 'For a locale aware separator, use the "\'n\'" integer ' - 'presentation type\n' - 'instead.\n' - '\n' - 'Changed in version 3.1: Added the "\',\'" option (see also ' - '**PEP 378**).\n' - '\n' - 'The "\'_\'" option signals the use of an underscore for a ' - 'thousands\n' - 'separator for floating point presentation types and for ' - 'integer\n' - 'presentation type "\'d\'". For integer presentation types ' - '"\'b\'", "\'o\'",\n' - '"\'x\'", and "\'X\'", underscores will be inserted every 4 ' - 'digits. For\n' - 'other presentation types, specifying this option is an ' - 'error.\n' - '\n' - 'Changed in version 3.6: Added the "\'_\'" option (see also ' - '**PEP 515**).\n' - '\n' - '*width* is a decimal integer defining the minimum field ' - 'width. If not\n' - 'specified, then the field width will be determined by the ' - 'content.\n' - '\n' - 'When no explicit alignment is given, preceding the *width* ' - 'field by a\n' - 'zero ("\'0\'") character enables sign-aware zero-padding ' - 'for numeric\n' - 'types. This is equivalent to a *fill* character of "\'0\'" ' - 'with an\n' - '*alignment* type of "\'=\'".\n' - '\n' - 'The *precision* is a decimal number indicating how many ' - 'digits should\n' - 'be displayed after the decimal point for a floating point ' - 'value\n' - 'formatted with "\'f\'" and "\'F\'", or before and after the ' - 'decimal point\n' - 'for a floating point value formatted with "\'g\'" or ' - '"\'G\'". For non-\n' - 'number types the field indicates the maximum field size - ' - 'in other\n' - 'words, how many characters will be used from the field ' - 'content. The\n' - '*precision* is not allowed for integer values.\n' - '\n' - 'Finally, the *type* determines how the data should be ' - 'presented.\n' - '\n' - 'The available string presentation types are:\n' - '\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | Type | ' - 'Meaning ' - '|\n' - ' ' - '+===========+============================================================+\n' - ' | "\'s\'" | String format. This is the default type ' - 'for strings and |\n' - ' | | may be ' - 'omitted. |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | None | The same as ' - '"\'s\'". |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - '\n' - 'The available integer presentation types are:\n' - '\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | Type | ' - 'Meaning ' - '|\n' - ' ' - '+===========+============================================================+\n' - ' | "\'b\'" | Binary format. Outputs the number in ' - 'base 2. |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | "\'c\'" | Character. Converts the integer to the ' - 'corresponding |\n' - ' | | unicode character before ' - 'printing. |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | "\'d\'" | Decimal Integer. Outputs the number in ' - 'base 10. |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | "\'o\'" | Octal format. Outputs the number in base ' - '8. |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | "\'x\'" | Hex format. Outputs the number in base ' - '16, using lower- |\n' - ' | | case letters for the digits above ' - '9. |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | "\'X\'" | Hex format. Outputs the number in base ' - '16, using upper- |\n' - ' | | case letters for the digits above ' - '9. |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | "\'n\'" | Number. This is the same as "\'d\'", ' - 'except that it uses the |\n' - ' | | current locale setting to insert the ' - 'appropriate number |\n' - ' | | separator ' - 'characters. |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | None | The same as ' - '"\'d\'". |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - '\n' - 'In addition to the above presentation types, integers can ' - 'be formatted\n' - 'with the floating point presentation types listed below ' - '(except "\'n\'"\n' - 'and "None"). When doing so, "float()" is used to convert ' - 'the integer\n' - 'to a floating point number before formatting.\n' - '\n' - 'The available presentation types for floating point and ' - 'decimal values\n' - 'are:\n' - '\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | Type | ' - 'Meaning ' - '|\n' - ' ' - '+===========+============================================================+\n' - ' | "\'e\'" | Exponent notation. Prints the number in ' - 'scientific |\n' - ' | | notation using the letter ‘e’ to indicate ' - 'the exponent. |\n' - ' | | The default precision is ' - '"6". |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | "\'E\'" | Exponent notation. Same as "\'e\'" ' - 'except it uses an upper |\n' - ' | | case ‘E’ as the separator ' - 'character. |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | "\'f\'" | Fixed-point notation. Displays the ' - 'number as a fixed-point |\n' - ' | | number. The default precision is ' - '"6". |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | "\'F\'" | Fixed-point notation. Same as "\'f\'", ' - 'but converts "nan" to |\n' - ' | | "NAN" and "inf" to ' - '"INF". |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | "\'g\'" | General format. For a given precision ' - '"p >= 1", this |\n' - ' | | rounds the number to "p" significant ' - 'digits and then |\n' - ' | | formats the result in either fixed-point ' - 'format or in |\n' - ' | | scientific notation, depending on its ' - 'magnitude. The |\n' - ' | | precise rules are as follows: suppose that ' - 'the result |\n' - ' | | formatted with presentation type "\'e\'" ' - 'and precision "p-1" |\n' - ' | | would have exponent "exp". Then if "-4 <= ' - 'exp < p", the |\n' - ' | | number is formatted with presentation type ' - '"\'f\'" and |\n' - ' | | precision "p-1-exp". Otherwise, the ' - 'number is formatted |\n' - ' | | with presentation type "\'e\'" and ' - 'precision "p-1". In both |\n' - ' | | cases insignificant trailing zeros are ' - 'removed from the |\n' - ' | | significand, and the decimal point is also ' - 'removed if |\n' - ' | | there are no remaining digits following ' - 'it. Positive and |\n' - ' | | negative infinity, positive and negative ' - 'zero, and nans, |\n' - ' | | are formatted as "inf", "-inf", "0", "-0" ' - 'and "nan" |\n' - ' | | respectively, regardless of the ' - 'precision. A precision of |\n' - ' | | "0" is treated as equivalent to a ' - 'precision of "1". The |\n' - ' | | default precision is ' - '"6". |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | "\'G\'" | General format. Same as "\'g\'" except ' - 'switches to "\'E\'" if |\n' - ' | | the number gets too large. The ' - 'representations of infinity |\n' - ' | | and NaN are uppercased, ' - 'too. |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | "\'n\'" | Number. This is the same as "\'g\'", ' - 'except that it uses the |\n' - ' | | current locale setting to insert the ' - 'appropriate number |\n' - ' | | separator ' - 'characters. |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | "\'%\'" | Percentage. Multiplies the number by 100 ' - 'and displays in |\n' - ' | | fixed ("\'f\'") format, followed by a ' - 'percent sign. |\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - ' | None | Similar to "\'g\'", except that ' - 'fixed-point notation, when |\n' - ' | | used, has at least one digit past the ' - 'decimal point. The |\n' - ' | | default precision is as high as needed to ' - 'represent the |\n' - ' | | particular value. The overall effect is to ' - 'match the |\n' - ' | | output of "str()" as altered by the other ' - 'format |\n' - ' | | ' - 'modifiers. ' - '|\n' - ' ' - '+-----------+------------------------------------------------------------+\n' - '\n' - '\n' - 'Format examples\n' - '===============\n' - '\n' - 'This section contains examples of the "str.format()" syntax ' - 'and\n' - 'comparison with the old "%"-formatting.\n' - '\n' - 'In most of the cases the syntax is similar to the old ' - '"%"-formatting,\n' - 'with the addition of the "{}" and with ":" used instead of ' - '"%". For\n' - 'example, "\'%03.2f\'" can be translated to "\'{:03.2f}\'".\n' - '\n' - 'The new format syntax also supports new and different ' - 'options, shown\n' - 'in the following examples.\n' - '\n' - 'Accessing arguments by position:\n' - '\n' - " >>> '{0}, {1}, {2}'.format('a', 'b', 'c')\n" - " 'a, b, c'\n" - " >>> '{}, {}, {}'.format('a', 'b', 'c') # 3.1+ only\n" - " 'a, b, c'\n" - " >>> '{2}, {1}, {0}'.format('a', 'b', 'c')\n" - " 'c, b, a'\n" - " >>> '{2}, {1}, {0}'.format(*'abc') # unpacking " - 'argument sequence\n' - " 'c, b, a'\n" - " >>> '{0}{1}{0}'.format('abra', 'cad') # arguments' " - 'indices can be repeated\n' - " 'abracadabra'\n" - '\n' - 'Accessing arguments by name:\n' - '\n' - " >>> 'Coordinates: {latitude}, " - "{longitude}'.format(latitude='37.24N', " - "longitude='-115.81W')\n" - " 'Coordinates: 37.24N, -115.81W'\n" - " >>> coord = {'latitude': '37.24N', 'longitude': " - "'-115.81W'}\n" - " >>> 'Coordinates: {latitude}, " - "{longitude}'.format(**coord)\n" - " 'Coordinates: 37.24N, -115.81W'\n" - '\n' - 'Accessing arguments’ attributes:\n' - '\n' - ' >>> c = 3-5j\n' - " >>> ('The complex number {0} is formed from the real " - "part {0.real} '\n" - " ... 'and the imaginary part {0.imag}.').format(c)\n" - " 'The complex number (3-5j) is formed from the real part " - "3.0 and the imaginary part -5.0.'\n" - ' >>> class Point:\n' - ' ... def __init__(self, x, y):\n' - ' ... self.x, self.y = x, y\n' - ' ... def __str__(self):\n' - " ... return 'Point({self.x}, " - "{self.y})'.format(self=self)\n" - ' ...\n' - ' >>> str(Point(4, 2))\n' - " 'Point(4, 2)'\n" - '\n' - 'Accessing arguments’ items:\n' - '\n' - ' >>> coord = (3, 5)\n' - " >>> 'X: {0[0]}; Y: {0[1]}'.format(coord)\n" - " 'X: 3; Y: 5'\n" - '\n' - 'Replacing "%s" and "%r":\n' - '\n' - ' >>> "repr() shows quotes: {!r}; str() doesn\'t: ' - '{!s}".format(\'test1\', \'test2\')\n' - ' "repr() shows quotes: \'test1\'; str() doesn\'t: test2"\n' - '\n' - 'Aligning the text and specifying a width:\n' - '\n' - " >>> '{:<30}'.format('left aligned')\n" - " 'left aligned '\n" - " >>> '{:>30}'.format('right aligned')\n" - " ' right aligned'\n" - " >>> '{:^30}'.format('centered')\n" - " ' centered '\n" - " >>> '{:*^30}'.format('centered') # use '*' as a fill " - 'char\n' - " '***********centered***********'\n" - '\n' - 'Replacing "%+f", "%-f", and "% f" and specifying a sign:\n' - '\n' - " >>> '{:+f}; {:+f}'.format(3.14, -3.14) # show it " - 'always\n' - " '+3.140000; -3.140000'\n" - " >>> '{: f}; {: f}'.format(3.14, -3.14) # show a space " - 'for positive numbers\n' - " ' 3.140000; -3.140000'\n" - " >>> '{:-f}; {:-f}'.format(3.14, -3.14) # show only the " - "minus -- same as '{:f}; {:f}'\n" - " '3.140000; -3.140000'\n" - '\n' - 'Replacing "%x" and "%o" and converting the value to ' - 'different bases:\n' - '\n' - ' >>> # format also supports binary numbers\n' - ' >>> "int: {0:d}; hex: {0:x}; oct: {0:o}; bin: ' - '{0:b}".format(42)\n' - " 'int: 42; hex: 2a; oct: 52; bin: 101010'\n" - ' >>> # with 0x, 0o, or 0b as prefix:\n' - ' >>> "int: {0:d}; hex: {0:#x}; oct: {0:#o}; bin: ' - '{0:#b}".format(42)\n' - " 'int: 42; hex: 0x2a; oct: 0o52; bin: 0b101010'\n" - '\n' - 'Using the comma as a thousands separator:\n' - '\n' - " >>> '{:,}'.format(1234567890)\n" - " '1,234,567,890'\n" - '\n' - 'Expressing a percentage:\n' - '\n' - ' >>> points = 19\n' - ' >>> total = 22\n' - " >>> 'Correct answers: {:.2%}'.format(points/total)\n" - " 'Correct answers: 86.36%'\n" - '\n' - 'Using type-specific formatting:\n' - '\n' - ' >>> import datetime\n' - ' >>> d = datetime.datetime(2010, 7, 4, 12, 15, 58)\n' - " >>> '{:%Y-%m-%d %H:%M:%S}'.format(d)\n" - " '2010-07-04 12:15:58'\n" - '\n' - 'Nesting arguments and more complex examples:\n' - '\n' - " >>> for align, text in zip('<^>', ['left', 'center', " - "'right']):\n" - " ... '{0:{fill}{align}16}'.format(text, fill=align, " - 'align=align)\n' - ' ...\n' - " 'left<<<<<<<<<<<<'\n" - " '^^^^^center^^^^^'\n" - " '>>>>>>>>>>>right'\n" - ' >>>\n' - ' >>> octets = [192, 168, 0, 1]\n' - " >>> '{:02X}{:02X}{:02X}{:02X}'.format(*octets)\n" - " 'C0A80001'\n" - ' >>> int(_, 16)\n' - ' 3232235521\n' - ' >>>\n' - ' >>> width = 5\n' - ' >>> for num in range(5,12): \n' - " ... for base in 'dXob':\n" - " ... print('{0:{width}{base}}'.format(num, " - "base=base, width=width), end=' ')\n" - ' ... print()\n' - ' ...\n' - ' 5 5 5 101\n' - ' 6 6 6 110\n' - ' 7 7 7 111\n' - ' 8 8 10 1000\n' - ' 9 9 11 1001\n' - ' 10 A 12 1010\n' - ' 11 B 13 1011\n', - 'function': 'Function definitions\n' - '********************\n' - '\n' - 'A function definition defines a user-defined function object ' - '(see\n' - 'section The standard type hierarchy):\n' - '\n' - ' funcdef ::= [decorators] "def" funcname "(" ' - '[parameter_list] ")"\n' - ' ["->" expression] ":" suite\n' - ' decorators ::= decorator+\n' - ' decorator ::= "@" dotted_name ["(" ' - '[argument_list [","]] ")"] NEWLINE\n' - ' dotted_name ::= identifier ("." identifier)*\n' - ' parameter_list ::= defparameter ("," defparameter)* ' - '["," [parameter_list_starargs]]\n' - ' | parameter_list_starargs\n' - ' parameter_list_starargs ::= "*" [parameter] ("," ' - 'defparameter)* ["," ["**" parameter [","]]]\n' - ' | "**" parameter [","]\n' - ' parameter ::= identifier [":" expression]\n' - ' defparameter ::= parameter ["=" expression]\n' - ' funcname ::= identifier\n' - '\n' - 'A function definition is an executable statement. Its execution ' - 'binds\n' - 'the function name in the current local namespace to a function ' - 'object\n' - '(a wrapper around the executable code for the function). This\n' - 'function object contains a reference to the current global ' - 'namespace\n' - 'as the global namespace to be used when the function is called.\n' - '\n' - 'The function definition does not execute the function body; this ' - 'gets\n' - 'executed only when the function is called. [2]\n' - '\n' - 'A function definition may be wrapped by one or more *decorator*\n' - 'expressions. Decorator expressions are evaluated when the ' - 'function is\n' - 'defined, in the scope that contains the function definition. ' - 'The\n' - 'result must be a callable, which is invoked with the function ' - 'object\n' - 'as the only argument. The returned value is bound to the ' - 'function name\n' - 'instead of the function object. Multiple decorators are applied ' - 'in\n' - 'nested fashion. For example, the following code\n' - '\n' - ' @f1(arg)\n' - ' @f2\n' - ' def func(): pass\n' - '\n' - 'is roughly equivalent to\n' - '\n' - ' def func(): pass\n' - ' func = f1(arg)(f2(func))\n' - '\n' - 'except that the original function is not temporarily bound to ' - 'the name\n' - '"func".\n' - '\n' - 'When one or more *parameters* have the form *parameter* "="\n' - '*expression*, the function is said to have “default parameter ' - 'values.”\n' - 'For a parameter with a default value, the corresponding ' - '*argument* may\n' - 'be omitted from a call, in which case the parameter’s default ' - 'value is\n' - 'substituted. If a parameter has a default value, all following\n' - 'parameters up until the “"*"” must also have a default value — ' - 'this is\n' - 'a syntactic restriction that is not expressed by the grammar.\n' - '\n' - '**Default parameter values are evaluated from left to right when ' - 'the\n' - 'function definition is executed.** This means that the ' - 'expression is\n' - 'evaluated once, when the function is defined, and that the same ' - '“pre-\n' - 'computed” value is used for each call. This is especially ' - 'important\n' - 'to understand when a default parameter is a mutable object, such ' - 'as a\n' - 'list or a dictionary: if the function modifies the object (e.g. ' - 'by\n' - 'appending an item to a list), the default value is in effect ' - 'modified.\n' - 'This is generally not what was intended. A way around this is ' - 'to use\n' - '"None" as the default, and explicitly test for it in the body of ' - 'the\n' - 'function, e.g.:\n' - '\n' - ' def whats_on_the_telly(penguin=None):\n' - ' if penguin is None:\n' - ' penguin = []\n' - ' penguin.append("property of the zoo")\n' - ' return penguin\n' - '\n' - 'Function call semantics are described in more detail in section ' - 'Calls.\n' - 'A function call always assigns values to all parameters ' - 'mentioned in\n' - 'the parameter list, either from position arguments, from ' - 'keyword\n' - 'arguments, or from default values. If the form “"*identifier"” ' - 'is\n' - 'present, it is initialized to a tuple receiving any excess ' - 'positional\n' - 'parameters, defaulting to the empty tuple. If the form\n' - '“"**identifier"” is present, it is initialized to a new ordered\n' - 'mapping receiving any excess keyword arguments, defaulting to a ' - 'new\n' - 'empty mapping of the same type. Parameters after “"*"” or\n' - '“"*identifier"” are keyword-only parameters and may only be ' - 'passed\n' - 'used keyword arguments.\n' - '\n' - 'Parameters may have annotations of the form “": expression"” ' - 'following\n' - 'the parameter name. Any parameter may have an annotation even ' - 'those\n' - 'of the form "*identifier" or "**identifier". Functions may ' - 'have\n' - '“return” annotation of the form “"-> expression"” after the ' - 'parameter\n' - 'list. These annotations can be any valid Python expression and ' - 'are\n' - 'evaluated when the function definition is executed. Annotations ' - 'may\n' - 'be evaluated in a different order than they appear in the source ' - 'code.\n' - 'The presence of annotations does not change the semantics of a\n' - 'function. The annotation values are available as values of a\n' - 'dictionary keyed by the parameters’ names in the ' - '"__annotations__"\n' - 'attribute of the function object.\n' - '\n' - 'It is also possible to create anonymous functions (functions not ' - 'bound\n' - 'to a name), for immediate use in expressions. This uses lambda\n' - 'expressions, described in section Lambdas. Note that the ' - 'lambda\n' - 'expression is merely a shorthand for a simplified function ' - 'definition;\n' - 'a function defined in a “"def"” statement can be passed around ' - 'or\n' - 'assigned to another name just like a function defined by a ' - 'lambda\n' - 'expression. The “"def"” form is actually more powerful since ' - 'it\n' - 'allows the execution of multiple statements and annotations.\n' - '\n' - '**Programmer’s note:** Functions are first-class objects. A ' - '“"def"”\n' - 'statement executed inside a function definition defines a local\n' - 'function that can be returned or passed around. Free variables ' - 'used\n' - 'in the nested function can access the local variables of the ' - 'function\n' - 'containing the def. See section Naming and binding for ' - 'details.\n' - '\n' - 'See also:\n' - '\n' - ' **PEP 3107** - Function Annotations\n' - ' The original specification for function annotations.\n', - 'global': 'The "global" statement\n' - '**********************\n' - '\n' - ' global_stmt ::= "global" identifier ("," identifier)*\n' - '\n' - 'The "global" statement is a declaration which holds for the ' - 'entire\n' - 'current code block. It means that the listed identifiers are to ' - 'be\n' - 'interpreted as globals. It would be impossible to assign to a ' - 'global\n' - 'variable without "global", although free variables may refer to\n' - 'globals without being declared global.\n' - '\n' - 'Names listed in a "global" statement must not be used in the same ' - 'code\n' - 'block textually preceding that "global" statement.\n' - '\n' - 'Names listed in a "global" statement must not be defined as ' - 'formal\n' - 'parameters or in a "for" loop control target, "class" definition,\n' - 'function definition, "import" statement, or variable annotation.\n' - '\n' - '**CPython implementation detail:** The current implementation does ' - 'not\n' - 'enforce some of these restrictions, but programs should not abuse ' - 'this\n' - 'freedom, as future implementations may enforce them or silently ' - 'change\n' - 'the meaning of the program.\n' - '\n' - '**Programmer’s note:** "global" is a directive to the parser. It\n' - 'applies only to code parsed at the same time as the "global"\n' - 'statement. In particular, a "global" statement contained in a ' - 'string\n' - 'or code object supplied to the built-in "exec()" function does ' - 'not\n' - 'affect the code block *containing* the function call, and code\n' - 'contained in such a string is unaffected by "global" statements in ' - 'the\n' - 'code containing the function call. The same applies to the ' - '"eval()"\n' - 'and "compile()" functions.\n', - 'id-classes': 'Reserved classes of identifiers\n' - '*******************************\n' - '\n' - 'Certain classes of identifiers (besides keywords) have ' - 'special\n' - 'meanings. These classes are identified by the patterns of ' - 'leading and\n' - 'trailing underscore characters:\n' - '\n' - '"_*"\n' - ' Not imported by "from module import *". The special ' - 'identifier "_"\n' - ' is used in the interactive interpreter to store the result ' - 'of the\n' - ' last evaluation; it is stored in the "builtins" module. ' - 'When not\n' - ' in interactive mode, "_" has no special meaning and is not ' - 'defined.\n' - ' See section The import statement.\n' - '\n' - ' Note: The name "_" is often used in conjunction with\n' - ' internationalization; refer to the documentation for the\n' - ' "gettext" module for more information on this ' - 'convention.\n' - '\n' - '"__*__"\n' - ' System-defined names. These names are defined by the ' - 'interpreter\n' - ' and its implementation (including the standard library). ' - 'Current\n' - ' system names are discussed in the Special method names ' - 'section and\n' - ' elsewhere. More will likely be defined in future versions ' - 'of\n' - ' Python. *Any* use of "__*__" names, in any context, that ' - 'does not\n' - ' follow explicitly documented use, is subject to breakage ' - 'without\n' - ' warning.\n' - '\n' - '"__*"\n' - ' Class-private names. Names in this category, when used ' - 'within the\n' - ' context of a class definition, are re-written to use a ' - 'mangled form\n' - ' to help avoid name clashes between “private” attributes of ' - 'base and\n' - ' derived classes. See section Identifiers (Names).\n', - 'identifiers': 'Identifiers and keywords\n' - '************************\n' - '\n' - 'Identifiers (also referred to as *names*) are described by ' - 'the\n' - 'following lexical definitions.\n' - '\n' - 'The syntax of identifiers in Python is based on the Unicode ' - 'standard\n' - 'annex UAX-31, with elaboration and changes as defined below; ' - 'see also\n' - '**PEP 3131** for further details.\n' - '\n' - 'Within the ASCII range (U+0001..U+007F), the valid characters ' - 'for\n' - 'identifiers are the same as in Python 2.x: the uppercase and ' - 'lowercase\n' - 'letters "A" through "Z", the underscore "_" and, except for ' - 'the first\n' - 'character, the digits "0" through "9".\n' - '\n' - 'Python 3.0 introduces additional characters from outside the ' - 'ASCII\n' - 'range (see **PEP 3131**). For these characters, the ' - 'classification\n' - 'uses the version of the Unicode Character Database as ' - 'included in the\n' - '"unicodedata" module.\n' - '\n' - 'Identifiers are unlimited in length. Case is significant.\n' - '\n' - ' identifier ::= xid_start xid_continue*\n' - ' id_start ::= \n' - ' id_continue ::= \n' - ' xid_start ::= \n' - ' xid_continue ::= \n' - '\n' - 'The Unicode category codes mentioned above stand for:\n' - '\n' - '* *Lu* - uppercase letters\n' - '\n' - '* *Ll* - lowercase letters\n' - '\n' - '* *Lt* - titlecase letters\n' - '\n' - '* *Lm* - modifier letters\n' - '\n' - '* *Lo* - other letters\n' - '\n' - '* *Nl* - letter numbers\n' - '\n' - '* *Mn* - nonspacing marks\n' - '\n' - '* *Mc* - spacing combining marks\n' - '\n' - '* *Nd* - decimal numbers\n' - '\n' - '* *Pc* - connector punctuations\n' - '\n' - '* *Other_ID_Start* - explicit list of characters in ' - 'PropList.txt to\n' - ' support backwards compatibility\n' - '\n' - '* *Other_ID_Continue* - likewise\n' - '\n' - 'All identifiers are converted into the normal form NFKC while ' - 'parsing;\n' - 'comparison of identifiers is based on NFKC.\n' - '\n' - 'A non-normative HTML file listing all valid identifier ' - 'characters for\n' - 'Unicode 4.1 can be found at https://www.dcl.hpi.uni-\n' - 'potsdam.de/home/loewis/table-3131.html.\n' - '\n' - '\n' - 'Keywords\n' - '========\n' - '\n' - 'The following identifiers are used as reserved words, or ' - '*keywords* of\n' - 'the language, and cannot be used as ordinary identifiers. ' - 'They must\n' - 'be spelled exactly as written here:\n' - '\n' - ' False class finally is return\n' - ' None continue for lambda try\n' - ' True def from nonlocal while\n' - ' and del global not with\n' - ' as elif if or yield\n' - ' assert else import pass\n' - ' break except in raise\n' - '\n' - '\n' - 'Reserved classes of identifiers\n' - '===============================\n' - '\n' - 'Certain classes of identifiers (besides keywords) have ' - 'special\n' - 'meanings. These classes are identified by the patterns of ' - 'leading and\n' - 'trailing underscore characters:\n' - '\n' - '"_*"\n' - ' Not imported by "from module import *". The special ' - 'identifier "_"\n' - ' is used in the interactive interpreter to store the result ' - 'of the\n' - ' last evaluation; it is stored in the "builtins" module. ' - 'When not\n' - ' in interactive mode, "_" has no special meaning and is not ' - 'defined.\n' - ' See section The import statement.\n' - '\n' - ' Note: The name "_" is often used in conjunction with\n' - ' internationalization; refer to the documentation for ' - 'the\n' - ' "gettext" module for more information on this ' - 'convention.\n' - '\n' - '"__*__"\n' - ' System-defined names. These names are defined by the ' - 'interpreter\n' - ' and its implementation (including the standard library). ' - 'Current\n' - ' system names are discussed in the Special method names ' - 'section and\n' - ' elsewhere. More will likely be defined in future versions ' - 'of\n' - ' Python. *Any* use of "__*__" names, in any context, that ' - 'does not\n' - ' follow explicitly documented use, is subject to breakage ' - 'without\n' - ' warning.\n' - '\n' - '"__*"\n' - ' Class-private names. Names in this category, when used ' - 'within the\n' - ' context of a class definition, are re-written to use a ' - 'mangled form\n' - ' to help avoid name clashes between “private” attributes of ' - 'base and\n' - ' derived classes. See section Identifiers (Names).\n', - 'if': 'The "if" statement\n' - '******************\n' - '\n' - 'The "if" statement is used for conditional execution:\n' - '\n' - ' if_stmt ::= "if" expression ":" suite\n' - ' ("elif" expression ":" suite)*\n' - ' ["else" ":" suite]\n' - '\n' - 'It selects exactly one of the suites by evaluating the expressions ' - 'one\n' - 'by one until one is found to be true (see section Boolean operations\n' - 'for the definition of true and false); then that suite is executed\n' - '(and no other part of the "if" statement is executed or evaluated).\n' - 'If all expressions are false, the suite of the "else" clause, if\n' - 'present, is executed.\n', - 'imaginary': 'Imaginary literals\n' - '******************\n' - '\n' - 'Imaginary literals are described by the following lexical ' - 'definitions:\n' - '\n' - ' imagnumber ::= (floatnumber | digitpart) ("j" | "J")\n' - '\n' - 'An imaginary literal yields a complex number with a real part ' - 'of 0.0.\n' - 'Complex numbers are represented as a pair of floating point ' - 'numbers\n' - 'and have the same restrictions on their range. To create a ' - 'complex\n' - 'number with a nonzero real part, add a floating point number to ' - 'it,\n' - 'e.g., "(3+4j)". Some examples of imaginary literals:\n' - '\n' - ' 3.14j 10.j 10j .001j 1e100j 3.14e-10j ' - '3.14_15_93j\n', - 'import': 'The "import" statement\n' - '**********************\n' - '\n' - ' import_stmt ::= "import" module ["as" identifier] ("," ' - 'module ["as" identifier])*\n' - ' | "from" relative_module "import" identifier ' - '["as" identifier]\n' - ' ("," identifier ["as" identifier])*\n' - ' | "from" relative_module "import" "(" ' - 'identifier ["as" identifier]\n' - ' ("," identifier ["as" identifier])* [","] ")"\n' - ' | "from" module "import" "*"\n' - ' module ::= (identifier ".")* identifier\n' - ' relative_module ::= "."* module | "."+\n' - '\n' - 'The basic import statement (no "from" clause) is executed in two\n' - 'steps:\n' - '\n' - '1. find a module, loading and initializing it if necessary\n' - '\n' - '2. define a name or names in the local namespace for the scope\n' - ' where the "import" statement occurs.\n' - '\n' - 'When the statement contains multiple clauses (separated by commas) ' - 'the\n' - 'two steps are carried out separately for each clause, just as ' - 'though\n' - 'the clauses had been separated out into individual import ' - 'statements.\n' - '\n' - 'The details of the first step, finding and loading modules are\n' - 'described in greater detail in the section on the import system, ' - 'which\n' - 'also describes the various types of packages and modules that can ' - 'be\n' - 'imported, as well as all the hooks that can be used to customize ' - 'the\n' - 'import system. Note that failures in this step may indicate ' - 'either\n' - 'that the module could not be located, *or* that an error occurred\n' - 'while initializing the module, which includes execution of the\n' - 'module’s code.\n' - '\n' - 'If the requested module is retrieved successfully, it will be ' - 'made\n' - 'available in the local namespace in one of three ways:\n' - '\n' - '* If the module name is followed by "as", then the name following\n' - ' "as" is bound directly to the imported module.\n' - '\n' - '* If no other name is specified, and the module being imported is ' - 'a\n' - ' top level module, the module’s name is bound in the local ' - 'namespace\n' - ' as a reference to the imported module\n' - '\n' - '* If the module being imported is *not* a top level module, then ' - 'the\n' - ' name of the top level package that contains the module is bound ' - 'in\n' - ' the local namespace as a reference to the top level package. ' - 'The\n' - ' imported module must be accessed using its full qualified name\n' - ' rather than directly\n' - '\n' - 'The "from" form uses a slightly more complex process:\n' - '\n' - '1. find the module specified in the "from" clause, loading and\n' - ' initializing it if necessary;\n' - '\n' - '2. for each of the identifiers specified in the "import" clauses:\n' - '\n' - ' 1. check if the imported module has an attribute by that name\n' - '\n' - ' 2. if not, attempt to import a submodule with that name and ' - 'then\n' - ' check the imported module again for that attribute\n' - '\n' - ' 3. if the attribute is not found, "ImportError" is raised.\n' - '\n' - ' 4. otherwise, a reference to that value is stored in the local\n' - ' namespace, using the name in the "as" clause if it is ' - 'present,\n' - ' otherwise using the attribute name\n' - '\n' - 'Examples:\n' - '\n' - ' import foo # foo imported and bound locally\n' - ' import foo.bar.baz # foo.bar.baz imported, foo bound ' - 'locally\n' - ' import foo.bar.baz as fbb # foo.bar.baz imported and bound as ' - 'fbb\n' - ' from foo.bar import baz # foo.bar.baz imported and bound as ' - 'baz\n' - ' from foo import attr # foo imported and foo.attr bound as ' - 'attr\n' - '\n' - 'If the list of identifiers is replaced by a star ("\'*\'"), all ' - 'public\n' - 'names defined in the module are bound in the local namespace for ' - 'the\n' - 'scope where the "import" statement occurs.\n' - '\n' - 'The *public names* defined by a module are determined by checking ' - 'the\n' - 'module’s namespace for a variable named "__all__"; if defined, it ' - 'must\n' - 'be a sequence of strings which are names defined or imported by ' - 'that\n' - 'module. The names given in "__all__" are all considered public ' - 'and\n' - 'are required to exist. If "__all__" is not defined, the set of ' - 'public\n' - 'names includes all names found in the module’s namespace which do ' - 'not\n' - 'begin with an underscore character ("\'_\'"). "__all__" should ' - 'contain\n' - 'the entire public API. It is intended to avoid accidentally ' - 'exporting\n' - 'items that are not part of the API (such as library modules which ' - 'were\n' - 'imported and used within the module).\n' - '\n' - 'The wild card form of import — "from module import *" — is only\n' - 'allowed at the module level. Attempting to use it in class or\n' - 'function definitions will raise a "SyntaxError".\n' - '\n' - 'When specifying what module to import you do not have to specify ' - 'the\n' - 'absolute name of the module. When a module or package is ' - 'contained\n' - 'within another package it is possible to make a relative import ' - 'within\n' - 'the same top package without having to mention the package name. ' - 'By\n' - 'using leading dots in the specified module or package after "from" ' - 'you\n' - 'can specify how high to traverse up the current package hierarchy\n' - 'without specifying exact names. One leading dot means the current\n' - 'package where the module making the import exists. Two dots means ' - 'up\n' - 'one package level. Three dots is up two levels, etc. So if you ' - 'execute\n' - '"from . import mod" from a module in the "pkg" package then you ' - 'will\n' - 'end up importing "pkg.mod". If you execute "from ..subpkg2 import ' - 'mod"\n' - 'from within "pkg.subpkg1" you will import "pkg.subpkg2.mod". The\n' - 'specification for relative imports is contained within **PEP ' - '328**.\n' - '\n' - '"importlib.import_module()" is provided to support applications ' - 'that\n' - 'determine dynamically the modules to be loaded.\n' - '\n' - '\n' - 'Future statements\n' - '=================\n' - '\n' - 'A *future statement* is a directive to the compiler that a ' - 'particular\n' - 'module should be compiled using syntax or semantics that will be\n' - 'available in a specified future release of Python where the ' - 'feature\n' - 'becomes standard.\n' - '\n' - 'The future statement is intended to ease migration to future ' - 'versions\n' - 'of Python that introduce incompatible changes to the language. ' - 'It\n' - 'allows use of the new features on a per-module basis before the\n' - 'release in which the feature becomes standard.\n' - '\n' - ' future_stmt ::= "from" "__future__" "import" feature ["as" ' - 'identifier]\n' - ' ("," feature ["as" identifier])*\n' - ' | "from" "__future__" "import" "(" feature ' - '["as" identifier]\n' - ' ("," feature ["as" identifier])* [","] ")"\n' - ' feature ::= identifier\n' - '\n' - 'A future statement must appear near the top of the module. The ' - 'only\n' - 'lines that can appear before a future statement are:\n' - '\n' - '* the module docstring (if any),\n' - '\n' - '* comments,\n' - '\n' - '* blank lines, and\n' - '\n' - '* other future statements.\n' - '\n' - 'The features recognized by Python 3.0 are "absolute_import",\n' - '"division", "generators", "unicode_literals", "print_function",\n' - '"nested_scopes" and "with_statement". They are all redundant ' - 'because\n' - 'they are always enabled, and only kept for backwards ' - 'compatibility.\n' - '\n' - 'A future statement is recognized and treated specially at compile\n' - 'time: Changes to the semantics of core constructs are often\n' - 'implemented by generating different code. It may even be the ' - 'case\n' - 'that a new feature introduces new incompatible syntax (such as a ' - 'new\n' - 'reserved word), in which case the compiler may need to parse the\n' - 'module differently. Such decisions cannot be pushed off until\n' - 'runtime.\n' - '\n' - 'For any given release, the compiler knows which feature names ' - 'have\n' - 'been defined, and raises a compile-time error if a future ' - 'statement\n' - 'contains a feature not known to it.\n' - '\n' - 'The direct runtime semantics are the same as for any import ' - 'statement:\n' - 'there is a standard module "__future__", described later, and it ' - 'will\n' - 'be imported in the usual way at the time the future statement is\n' - 'executed.\n' - '\n' - 'The interesting runtime semantics depend on the specific feature\n' - 'enabled by the future statement.\n' - '\n' - 'Note that there is nothing special about the statement:\n' - '\n' - ' import __future__ [as name]\n' - '\n' - 'That is not a future statement; it’s an ordinary import statement ' - 'with\n' - 'no special semantics or syntax restrictions.\n' - '\n' - 'Code compiled by calls to the built-in functions "exec()" and\n' - '"compile()" that occur in a module "M" containing a future ' - 'statement\n' - 'will, by default, use the new syntax or semantics associated with ' - 'the\n' - 'future statement. This can be controlled by optional arguments ' - 'to\n' - '"compile()" — see the documentation of that function for details.\n' - '\n' - 'A future statement typed at an interactive interpreter prompt ' - 'will\n' - 'take effect for the rest of the interpreter session. If an\n' - 'interpreter is started with the "-i" option, is passed a script ' - 'name\n' - 'to execute, and the script includes a future statement, it will be ' - 'in\n' - 'effect in the interactive session started after the script is\n' - 'executed.\n' - '\n' - 'See also:\n' - '\n' - ' **PEP 236** - Back to the __future__\n' - ' The original proposal for the __future__ mechanism.\n', - 'in': 'Membership test operations\n' - '**************************\n' - '\n' - 'The operators "in" and "not in" test for membership. "x in s"\n' - 'evaluates to "True" if *x* is a member of *s*, and "False" otherwise.\n' - '"x not in s" returns the negation of "x in s". All built-in ' - 'sequences\n' - 'and set types support this as well as dictionary, for which "in" ' - 'tests\n' - 'whether the dictionary has a given key. For container types such as\n' - 'list, tuple, set, frozenset, dict, or collections.deque, the\n' - 'expression "x in y" is equivalent to "any(x is e or x == e for e in\n' - 'y)".\n' - '\n' - 'For the string and bytes types, "x in y" is "True" if and only if *x*\n' - 'is a substring of *y*. An equivalent test is "y.find(x) != -1".\n' - 'Empty strings are always considered to be a substring of any other\n' - 'string, so """ in "abc"" will return "True".\n' - '\n' - 'For user-defined classes which define the "__contains__()" method, "x\n' - 'in y" returns "True" if "y.__contains__(x)" returns a true value, and\n' - '"False" otherwise.\n' - '\n' - 'For user-defined classes which do not define "__contains__()" but do\n' - 'define "__iter__()", "x in y" is "True" if some value "z" with "x ==\n' - 'z" is produced while iterating over "y". If an exception is raised\n' - 'during the iteration, it is as if "in" raised that exception.\n' - '\n' - 'Lastly, the old-style iteration protocol is tried: if a class defines\n' - '"__getitem__()", "x in y" is "True" if and only if there is a non-\n' - 'negative integer index *i* such that "x == y[i]", and all lower\n' - 'integer indices do not raise "IndexError" exception. (If any other\n' - 'exception is raised, it is as if "in" raised that exception).\n' - '\n' - 'The operator "not in" is defined to have the inverse true value of\n' - '"in".\n', - 'integers': 'Integer literals\n' - '****************\n' - '\n' - 'Integer literals are described by the following lexical ' - 'definitions:\n' - '\n' - ' integer ::= decinteger | bininteger | octinteger | ' - 'hexinteger\n' - ' decinteger ::= nonzerodigit (["_"] digit)* | "0"+ (["_"] ' - '"0")*\n' - ' bininteger ::= "0" ("b" | "B") (["_"] bindigit)+\n' - ' octinteger ::= "0" ("o" | "O") (["_"] octdigit)+\n' - ' hexinteger ::= "0" ("x" | "X") (["_"] hexdigit)+\n' - ' nonzerodigit ::= "1"..."9"\n' - ' digit ::= "0"..."9"\n' - ' bindigit ::= "0" | "1"\n' - ' octdigit ::= "0"..."7"\n' - ' hexdigit ::= digit | "a"..."f" | "A"..."F"\n' - '\n' - 'There is no limit for the length of integer literals apart from ' - 'what\n' - 'can be stored in available memory.\n' - '\n' - 'Underscores are ignored for determining the numeric value of ' - 'the\n' - 'literal. They can be used to group digits for enhanced ' - 'readability.\n' - 'One underscore can occur between digits, and after base ' - 'specifiers\n' - 'like "0x".\n' - '\n' - 'Note that leading zeros in a non-zero decimal number are not ' - 'allowed.\n' - 'This is for disambiguation with C-style octal literals, which ' - 'Python\n' - 'used before version 3.0.\n' - '\n' - 'Some examples of integer literals:\n' - '\n' - ' 7 2147483647 0o177 0b100110111\n' - ' 3 79228162514264337593543950336 0o377 0xdeadbeef\n' - ' 100_000_000_000 0b_1110_0101\n' - '\n' - 'Changed in version 3.6: Underscores are now allowed for ' - 'grouping\n' - 'purposes in literals.\n', - 'lambda': 'Lambdas\n' - '*******\n' - '\n' - ' lambda_expr ::= "lambda" [parameter_list] ":" ' - 'expression\n' - ' lambda_expr_nocond ::= "lambda" [parameter_list] ":" ' - 'expression_nocond\n' - '\n' - 'Lambda expressions (sometimes called lambda forms) are used to ' - 'create\n' - 'anonymous functions. The expression "lambda parameters: ' - 'expression"\n' - 'yields a function object. The unnamed object behaves like a ' - 'function\n' - 'object defined with:\n' - '\n' - ' def (parameters):\n' - ' return expression\n' - '\n' - 'See section Function definitions for the syntax of parameter ' - 'lists.\n' - 'Note that functions created with lambda expressions cannot ' - 'contain\n' - 'statements or annotations.\n', - 'lists': 'List displays\n' - '*************\n' - '\n' - 'A list display is a possibly empty series of expressions enclosed ' - 'in\n' - 'square brackets:\n' - '\n' - ' list_display ::= "[" [starred_list | comprehension] "]"\n' - '\n' - 'A list display yields a new list object, the contents being ' - 'specified\n' - 'by either a list of expressions or a comprehension. When a comma-\n' - 'separated list of expressions is supplied, its elements are ' - 'evaluated\n' - 'from left to right and placed into the list object in that order.\n' - 'When a comprehension is supplied, the list is constructed from the\n' - 'elements resulting from the comprehension.\n', - 'naming': 'Naming and binding\n' - '******************\n' - '\n' - '\n' - 'Binding of names\n' - '================\n' - '\n' - '*Names* refer to objects. Names are introduced by name binding\n' - 'operations.\n' - '\n' - 'The following constructs bind names: formal parameters to ' - 'functions,\n' - '"import" statements, class and function definitions (these bind ' - 'the\n' - 'class or function name in the defining block), and targets that ' - 'are\n' - 'identifiers if occurring in an assignment, "for" loop header, or ' - 'after\n' - '"as" in a "with" statement or "except" clause. The "import" ' - 'statement\n' - 'of the form "from ... import *" binds all names defined in the\n' - 'imported module, except those beginning with an underscore. This ' - 'form\n' - 'may only be used at the module level.\n' - '\n' - 'A target occurring in a "del" statement is also considered bound ' - 'for\n' - 'this purpose (though the actual semantics are to unbind the ' - 'name).\n' - '\n' - 'Each assignment or import statement occurs within a block defined ' - 'by a\n' - 'class or function definition or at the module level (the ' - 'top-level\n' - 'code block).\n' - '\n' - 'If a name is bound in a block, it is a local variable of that ' - 'block,\n' - 'unless declared as "nonlocal" or "global". If a name is bound at ' - 'the\n' - 'module level, it is a global variable. (The variables of the ' - 'module\n' - 'code block are local and global.) If a variable is used in a ' - 'code\n' - 'block but not defined there, it is a *free variable*.\n' - '\n' - 'Each occurrence of a name in the program text refers to the ' - '*binding*\n' - 'of that name established by the following name resolution rules.\n' - '\n' - '\n' - 'Resolution of names\n' - '===================\n' - '\n' - 'A *scope* defines the visibility of a name within a block. If a ' - 'local\n' - 'variable is defined in a block, its scope includes that block. If ' - 'the\n' - 'definition occurs in a function block, the scope extends to any ' - 'blocks\n' - 'contained within the defining one, unless a contained block ' - 'introduces\n' - 'a different binding for the name.\n' - '\n' - 'When a name is used in a code block, it is resolved using the ' - 'nearest\n' - 'enclosing scope. The set of all such scopes visible to a code ' - 'block\n' - 'is called the block’s *environment*.\n' - '\n' - 'When a name is not found at all, a "NameError" exception is ' - 'raised. If\n' - 'the current scope is a function scope, and the name refers to a ' - 'local\n' - 'variable that has not yet been bound to a value at the point where ' - 'the\n' - 'name is used, an "UnboundLocalError" exception is raised.\n' - '"UnboundLocalError" is a subclass of "NameError".\n' - '\n' - 'If a name binding operation occurs anywhere within a code block, ' - 'all\n' - 'uses of the name within the block are treated as references to ' - 'the\n' - 'current block. This can lead to errors when a name is used within ' - 'a\n' - 'block before it is bound. This rule is subtle. Python lacks\n' - 'declarations and allows name binding operations to occur anywhere\n' - 'within a code block. The local variables of a code block can be\n' - 'determined by scanning the entire text of the block for name ' - 'binding\n' - 'operations.\n' - '\n' - 'If the "global" statement occurs within a block, all uses of the ' - 'name\n' - 'specified in the statement refer to the binding of that name in ' - 'the\n' - 'top-level namespace. Names are resolved in the top-level ' - 'namespace by\n' - 'searching the global namespace, i.e. the namespace of the module\n' - 'containing the code block, and the builtins namespace, the ' - 'namespace\n' - 'of the module "builtins". The global namespace is searched ' - 'first. If\n' - 'the name is not found there, the builtins namespace is searched. ' - 'The\n' - '"global" statement must precede all uses of the name.\n' - '\n' - 'The "global" statement has the same scope as a name binding ' - 'operation\n' - 'in the same block. If the nearest enclosing scope for a free ' - 'variable\n' - 'contains a global statement, the free variable is treated as a ' - 'global.\n' - '\n' - 'The "nonlocal" statement causes corresponding names to refer to\n' - 'previously bound variables in the nearest enclosing function ' - 'scope.\n' - '"SyntaxError" is raised at compile time if the given name does ' - 'not\n' - 'exist in any enclosing function scope.\n' - '\n' - 'The namespace for a module is automatically created the first time ' - 'a\n' - 'module is imported. The main module for a script is always ' - 'called\n' - '"__main__".\n' - '\n' - 'Class definition blocks and arguments to "exec()" and "eval()" ' - 'are\n' - 'special in the context of name resolution. A class definition is ' - 'an\n' - 'executable statement that may use and define names. These ' - 'references\n' - 'follow the normal rules for name resolution with an exception ' - 'that\n' - 'unbound local variables are looked up in the global namespace. ' - 'The\n' - 'namespace of the class definition becomes the attribute dictionary ' - 'of\n' - 'the class. The scope of names defined in a class block is limited ' - 'to\n' - 'the class block; it does not extend to the code blocks of methods ' - '–\n' - 'this includes comprehensions and generator expressions since they ' - 'are\n' - 'implemented using a function scope. This means that the ' - 'following\n' - 'will fail:\n' - '\n' - ' class A:\n' - ' a = 42\n' - ' b = list(a + i for i in range(10))\n' - '\n' - '\n' - 'Builtins and restricted execution\n' - '=================================\n' - '\n' - '**CPython implementation detail:** Users should not touch\n' - '"__builtins__"; it is strictly an implementation detail. Users\n' - 'wanting to override values in the builtins namespace should ' - '"import"\n' - 'the "builtins" module and modify its attributes appropriately.\n' - '\n' - 'The builtins namespace associated with the execution of a code ' - 'block\n' - 'is actually found by looking up the name "__builtins__" in its ' - 'global\n' - 'namespace; this should be a dictionary or a module (in the latter ' - 'case\n' - 'the module’s dictionary is used). By default, when in the ' - '"__main__"\n' - 'module, "__builtins__" is the built-in module "builtins"; when in ' - 'any\n' - 'other module, "__builtins__" is an alias for the dictionary of ' - 'the\n' - '"builtins" module itself.\n' - '\n' - '\n' - 'Interaction with dynamic features\n' - '=================================\n' - '\n' - 'Name resolution of free variables occurs at runtime, not at ' - 'compile\n' - 'time. This means that the following code will print 42:\n' - '\n' - ' i = 10\n' - ' def f():\n' - ' print(i)\n' - ' i = 42\n' - ' f()\n' - '\n' - 'The "eval()" and "exec()" functions do not have access to the ' - 'full\n' - 'environment for resolving names. Names may be resolved in the ' - 'local\n' - 'and global namespaces of the caller. Free variables are not ' - 'resolved\n' - 'in the nearest enclosing namespace, but in the global namespace. ' - '[1]\n' - 'The "exec()" and "eval()" functions have optional arguments to\n' - 'override the global and local namespace. If only one namespace ' - 'is\n' - 'specified, it is used for both.\n', - 'nonlocal': 'The "nonlocal" statement\n' - '************************\n' - '\n' - ' nonlocal_stmt ::= "nonlocal" identifier ("," identifier)*\n' - '\n' - 'The "nonlocal" statement causes the listed identifiers to refer ' - 'to\n' - 'previously bound variables in the nearest enclosing scope ' - 'excluding\n' - 'globals. This is important because the default behavior for ' - 'binding is\n' - 'to search the local namespace first. The statement allows\n' - 'encapsulated code to rebind variables outside of the local ' - 'scope\n' - 'besides the global (module) scope.\n' - '\n' - 'Names listed in a "nonlocal" statement, unlike those listed in ' - 'a\n' - '"global" statement, must refer to pre-existing bindings in an\n' - 'enclosing scope (the scope in which a new binding should be ' - 'created\n' - 'cannot be determined unambiguously).\n' - '\n' - 'Names listed in a "nonlocal" statement must not collide with ' - 'pre-\n' - 'existing bindings in the local scope.\n' - '\n' - 'See also:\n' - '\n' - ' **PEP 3104** - Access to Names in Outer Scopes\n' - ' The specification for the "nonlocal" statement.\n', - 'numbers': 'Numeric literals\n' - '****************\n' - '\n' - 'There are three types of numeric literals: integers, floating ' - 'point\n' - 'numbers, and imaginary numbers. There are no complex literals\n' - '(complex numbers can be formed by adding a real number and an\n' - 'imaginary number).\n' - '\n' - 'Note that numeric literals do not include a sign; a phrase like ' - '"-1"\n' - 'is actually an expression composed of the unary operator ‘"-"‘ ' - 'and the\n' - 'literal "1".\n', - 'numeric-types': 'Emulating numeric types\n' - '***********************\n' - '\n' - 'The following methods can be defined to emulate numeric ' - 'objects.\n' - 'Methods corresponding to operations that are not supported ' - 'by the\n' - 'particular kind of number implemented (e.g., bitwise ' - 'operations for\n' - 'non-integral numbers) should be left undefined.\n' - '\n' - 'object.__add__(self, other)\n' - 'object.__sub__(self, other)\n' - 'object.__mul__(self, other)\n' - 'object.__matmul__(self, other)\n' - 'object.__truediv__(self, other)\n' - 'object.__floordiv__(self, other)\n' - 'object.__mod__(self, other)\n' - 'object.__divmod__(self, other)\n' - 'object.__pow__(self, other[, modulo])\n' - 'object.__lshift__(self, other)\n' - 'object.__rshift__(self, other)\n' - 'object.__and__(self, other)\n' - 'object.__xor__(self, other)\n' - 'object.__or__(self, other)\n' - '\n' - ' These methods are called to implement the binary ' - 'arithmetic\n' - ' operations ("+", "-", "*", "@", "/", "//", "%", ' - '"divmod()",\n' - ' "pow()", "**", "<<", ">>", "&", "^", "|"). For ' - 'instance, to\n' - ' evaluate the expression "x + y", where *x* is an ' - 'instance of a\n' - ' class that has an "__add__()" method, "x.__add__(y)" is ' - 'called.\n' - ' The "__divmod__()" method should be the equivalent to ' - 'using\n' - ' "__floordiv__()" and "__mod__()"; it should not be ' - 'related to\n' - ' "__truediv__()". Note that "__pow__()" should be ' - 'defined to accept\n' - ' an optional third argument if the ternary version of the ' - 'built-in\n' - ' "pow()" function is to be supported.\n' - '\n' - ' If one of those methods does not support the operation ' - 'with the\n' - ' supplied arguments, it should return "NotImplemented".\n' - '\n' - 'object.__radd__(self, other)\n' - 'object.__rsub__(self, other)\n' - 'object.__rmul__(self, other)\n' - 'object.__rmatmul__(self, other)\n' - 'object.__rtruediv__(self, other)\n' - 'object.__rfloordiv__(self, other)\n' - 'object.__rmod__(self, other)\n' - 'object.__rdivmod__(self, other)\n' - 'object.__rpow__(self, other)\n' - 'object.__rlshift__(self, other)\n' - 'object.__rrshift__(self, other)\n' - 'object.__rand__(self, other)\n' - 'object.__rxor__(self, other)\n' - 'object.__ror__(self, other)\n' - '\n' - ' These methods are called to implement the binary ' - 'arithmetic\n' - ' operations ("+", "-", "*", "@", "/", "//", "%", ' - '"divmod()",\n' - ' "pow()", "**", "<<", ">>", "&", "^", "|") with reflected ' - '(swapped)\n' - ' operands. These functions are only called if the left ' - 'operand does\n' - ' not support the corresponding operation [3] and the ' - 'operands are of\n' - ' different types. [4] For instance, to evaluate the ' - 'expression "x -\n' - ' y", where *y* is an instance of a class that has an ' - '"__rsub__()"\n' - ' method, "y.__rsub__(x)" is called if "x.__sub__(y)" ' - 'returns\n' - ' *NotImplemented*.\n' - '\n' - ' Note that ternary "pow()" will not try calling ' - '"__rpow__()" (the\n' - ' coercion rules would become too complicated).\n' - '\n' - ' Note: If the right operand’s type is a subclass of the ' - 'left\n' - ' operand’s type and that subclass provides the ' - 'reflected method\n' - ' for the operation, this method will be called before ' - 'the left\n' - ' operand’s non-reflected method. This behavior allows ' - 'subclasses\n' - ' to override their ancestors’ operations.\n' - '\n' - 'object.__iadd__(self, other)\n' - 'object.__isub__(self, other)\n' - 'object.__imul__(self, other)\n' - 'object.__imatmul__(self, other)\n' - 'object.__itruediv__(self, other)\n' - 'object.__ifloordiv__(self, other)\n' - 'object.__imod__(self, other)\n' - 'object.__ipow__(self, other[, modulo])\n' - 'object.__ilshift__(self, other)\n' - 'object.__irshift__(self, other)\n' - 'object.__iand__(self, other)\n' - 'object.__ixor__(self, other)\n' - 'object.__ior__(self, other)\n' - '\n' - ' These methods are called to implement the augmented ' - 'arithmetic\n' - ' assignments ("+=", "-=", "*=", "@=", "/=", "//=", "%=", ' - '"**=",\n' - ' "<<=", ">>=", "&=", "^=", "|="). These methods should ' - 'attempt to\n' - ' do the operation in-place (modifying *self*) and return ' - 'the result\n' - ' (which could be, but does not have to be, *self*). If a ' - 'specific\n' - ' method is not defined, the augmented assignment falls ' - 'back to the\n' - ' normal methods. For instance, if *x* is an instance of ' - 'a class\n' - ' with an "__iadd__()" method, "x += y" is equivalent to ' - '"x =\n' - ' x.__iadd__(y)" . Otherwise, "x.__add__(y)" and ' - '"y.__radd__(x)" are\n' - ' considered, as with the evaluation of "x + y". In ' - 'certain\n' - ' situations, augmented assignment can result in ' - 'unexpected errors\n' - ' (see Why does a_tuple[i] += [‘item’] raise an exception ' - 'when the\n' - ' addition works?), but this behavior is in fact part of ' - 'the data\n' - ' model.\n' - '\n' - 'object.__neg__(self)\n' - 'object.__pos__(self)\n' - 'object.__abs__(self)\n' - 'object.__invert__(self)\n' - '\n' - ' Called to implement the unary arithmetic operations ' - '("-", "+",\n' - ' "abs()" and "~").\n' - '\n' - 'object.__complex__(self)\n' - 'object.__int__(self)\n' - 'object.__float__(self)\n' - '\n' - ' Called to implement the built-in functions "complex()", ' - '"int()" and\n' - ' "float()". Should return a value of the appropriate ' - 'type.\n' - '\n' - 'object.__index__(self)\n' - '\n' - ' Called to implement "operator.index()", and whenever ' - 'Python needs\n' - ' to losslessly convert the numeric object to an integer ' - 'object (such\n' - ' as in slicing, or in the built-in "bin()", "hex()" and ' - '"oct()"\n' - ' functions). Presence of this method indicates that the ' - 'numeric\n' - ' object is an integer type. Must return an integer.\n' - '\n' - ' Note: In order to have a coherent integer type class, ' - 'when\n' - ' "__index__()" is defined "__int__()" should also be ' - 'defined, and\n' - ' both should return the same value.\n' - '\n' - 'object.__round__(self[, ndigits])\n' - 'object.__trunc__(self)\n' - 'object.__floor__(self)\n' - 'object.__ceil__(self)\n' - '\n' - ' Called to implement the built-in function "round()" and ' - '"math"\n' - ' functions "trunc()", "floor()" and "ceil()". Unless ' - '*ndigits* is\n' - ' passed to "__round__()" all these methods should return ' - 'the value\n' - ' of the object truncated to an "Integral" (typically an ' - '"int").\n' - '\n' - ' If "__int__()" is not defined then the built-in function ' - '"int()"\n' - ' falls back to "__trunc__()".\n', - 'objects': 'Objects, values and types\n' - '*************************\n' - '\n' - '*Objects* are Python’s abstraction for data. All data in a ' - 'Python\n' - 'program is represented by objects or by relations between ' - 'objects. (In\n' - 'a sense, and in conformance to Von Neumann’s model of a “stored\n' - 'program computer,” code is also represented by objects.)\n' - '\n' - 'Every object has an identity, a type and a value. An object’s\n' - '*identity* never changes once it has been created; you may think ' - 'of it\n' - 'as the object’s address in memory. The ‘"is"’ operator compares ' - 'the\n' - 'identity of two objects; the "id()" function returns an integer\n' - 'representing its identity.\n' - '\n' - '**CPython implementation detail:** For CPython, "id(x)" is the ' - 'memory\n' - 'address where "x" is stored.\n' - '\n' - 'An object’s type determines the operations that the object ' - 'supports\n' - '(e.g., “does it have a length?”) and also defines the possible ' - 'values\n' - 'for objects of that type. The "type()" function returns an ' - 'object’s\n' - 'type (which is an object itself). Like its identity, an ' - 'object’s\n' - '*type* is also unchangeable. [1]\n' - '\n' - 'The *value* of some objects can change. Objects whose value can\n' - 'change are said to be *mutable*; objects whose value is ' - 'unchangeable\n' - 'once they are created are called *immutable*. (The value of an\n' - 'immutable container object that contains a reference to a ' - 'mutable\n' - 'object can change when the latter’s value is changed; however ' - 'the\n' - 'container is still considered immutable, because the collection ' - 'of\n' - 'objects it contains cannot be changed. So, immutability is not\n' - 'strictly the same as having an unchangeable value, it is more ' - 'subtle.)\n' - 'An object’s mutability is determined by its type; for instance,\n' - 'numbers, strings and tuples are immutable, while dictionaries ' - 'and\n' - 'lists are mutable.\n' - '\n' - 'Objects are never explicitly destroyed; however, when they ' - 'become\n' - 'unreachable they may be garbage-collected. An implementation is\n' - 'allowed to postpone garbage collection or omit it altogether — it ' - 'is a\n' - 'matter of implementation quality how garbage collection is\n' - 'implemented, as long as no objects are collected that are still\n' - 'reachable.\n' - '\n' - '**CPython implementation detail:** CPython currently uses a ' - 'reference-\n' - 'counting scheme with (optional) delayed detection of cyclically ' - 'linked\n' - 'garbage, which collects most objects as soon as they become\n' - 'unreachable, but is not guaranteed to collect garbage containing\n' - 'circular references. See the documentation of the "gc" module ' - 'for\n' - 'information on controlling the collection of cyclic garbage. ' - 'Other\n' - 'implementations act differently and CPython may change. Do not ' - 'depend\n' - 'on immediate finalization of objects when they become unreachable ' - '(so\n' - 'you should always close files explicitly).\n' - '\n' - 'Note that the use of the implementation’s tracing or debugging\n' - 'facilities may keep objects alive that would normally be ' - 'collectable.\n' - 'Also note that catching an exception with a ‘"try"…"except"’ ' - 'statement\n' - 'may keep objects alive.\n' - '\n' - 'Some objects contain references to “external” resources such as ' - 'open\n' - 'files or windows. It is understood that these resources are ' - 'freed\n' - 'when the object is garbage-collected, but since garbage ' - 'collection is\n' - 'not guaranteed to happen, such objects also provide an explicit ' - 'way to\n' - 'release the external resource, usually a "close()" method. ' - 'Programs\n' - 'are strongly recommended to explicitly close such objects. The\n' - '‘"try"…"finally"’ statement and the ‘"with"’ statement provide\n' - 'convenient ways to do this.\n' - '\n' - 'Some objects contain references to other objects; these are ' - 'called\n' - '*containers*. Examples of containers are tuples, lists and\n' - 'dictionaries. The references are part of a container’s value. ' - 'In\n' - 'most cases, when we talk about the value of a container, we imply ' - 'the\n' - 'values, not the identities of the contained objects; however, ' - 'when we\n' - 'talk about the mutability of a container, only the identities of ' - 'the\n' - 'immediately contained objects are implied. So, if an immutable\n' - 'container (like a tuple) contains a reference to a mutable ' - 'object, its\n' - 'value changes if that mutable object is changed.\n' - '\n' - 'Types affect almost all aspects of object behavior. Even the\n' - 'importance of object identity is affected in some sense: for ' - 'immutable\n' - 'types, operations that compute new values may actually return a\n' - 'reference to any existing object with the same type and value, ' - 'while\n' - 'for mutable objects this is not allowed. E.g., after "a = 1; b = ' - '1",\n' - '"a" and "b" may or may not refer to the same object with the ' - 'value\n' - 'one, depending on the implementation, but after "c = []; d = []", ' - '"c"\n' - 'and "d" are guaranteed to refer to two different, unique, newly\n' - 'created empty lists. (Note that "c = d = []" assigns the same ' - 'object\n' - 'to both "c" and "d".)\n', - 'operator-summary': 'Operator precedence\n' - '*******************\n' - '\n' - 'The following table summarizes the operator precedence ' - 'in Python, from\n' - 'lowest precedence (least binding) to highest precedence ' - '(most\n' - 'binding). Operators in the same box have the same ' - 'precedence. Unless\n' - 'the syntax is explicitly given, operators are binary. ' - 'Operators in\n' - 'the same box group left to right (except for ' - 'exponentiation, which\n' - 'groups from right to left).\n' - '\n' - 'Note that comparisons, membership tests, and identity ' - 'tests, all have\n' - 'the same precedence and have a left-to-right chaining ' - 'feature as\n' - 'described in the Comparisons section.\n' - '\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| Operator | ' - 'Description |\n' - '+=================================================+=======================================+\n' - '| "lambda" | ' - 'Lambda expression |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| "if" – "else" | ' - 'Conditional expression |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| "or" | ' - 'Boolean OR |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| "and" | ' - 'Boolean AND |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| "not" "x" | ' - 'Boolean NOT |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| "in", "not in", "is", "is not", "<", "<=", ">", | ' - 'Comparisons, including membership |\n' - '| ">=", "!=", "==" | ' - 'tests and identity tests |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| "|" | ' - 'Bitwise OR |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| "^" | ' - 'Bitwise XOR |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| "&" | ' - 'Bitwise AND |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| "<<", ">>" | ' - 'Shifts |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| "+", "-" | ' - 'Addition and subtraction |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| "*", "@", "/", "//", "%" | ' - 'Multiplication, matrix |\n' - '| | ' - 'multiplication, division, floor |\n' - '| | ' - 'division, remainder [5] |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| "+x", "-x", "~x" | ' - 'Positive, negative, bitwise NOT |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| "**" | ' - 'Exponentiation [6] |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| "await" "x" | ' - 'Await expression |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| "x[index]", "x[index:index]", | ' - 'Subscription, slicing, call, |\n' - '| "x(arguments...)", "x.attribute" | ' - 'attribute reference |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| "(expressions...)", "[expressions...]", "{key: | ' - 'Binding or tuple display, list |\n' - '| value...}", "{expressions...}" | ' - 'display, dictionary display, set |\n' - '| | ' - 'display |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '\n' - '-[ Footnotes ]-\n' - '\n' - '[1] While "abs(x%y) < abs(y)" is true mathematically, ' - 'for floats\n' - ' it may not be true numerically due to roundoff. For ' - 'example, and\n' - ' assuming a platform on which a Python float is an ' - 'IEEE 754 double-\n' - ' precision number, in order that "-1e-100 % 1e100" ' - 'have the same\n' - ' sign as "1e100", the computed result is "-1e-100 + ' - '1e100", which\n' - ' is numerically exactly equal to "1e100". The ' - 'function\n' - ' "math.fmod()" returns a result whose sign matches ' - 'the sign of the\n' - ' first argument instead, and so returns "-1e-100" in ' - 'this case.\n' - ' Which approach is more appropriate depends on the ' - 'application.\n' - '\n' - '[2] If x is very close to an exact integer multiple of ' - 'y, it’s\n' - ' possible for "x//y" to be one larger than ' - '"(x-x%y)//y" due to\n' - ' rounding. In such cases, Python returns the latter ' - 'result, in\n' - ' order to preserve that "divmod(x,y)[0] * y + x % y" ' - 'be very close\n' - ' to "x".\n' - '\n' - '[3] The Unicode standard distinguishes between *code ' - 'points* (e.g.\n' - ' U+0041) and *abstract characters* (e.g. “LATIN ' - 'CAPITAL LETTER A”).\n' - ' While most abstract characters in Unicode are only ' - 'represented\n' - ' using one code point, there is a number of abstract ' - 'characters\n' - ' that can in addition be represented using a sequence ' - 'of more than\n' - ' one code point. For example, the abstract character ' - '“LATIN\n' - ' CAPITAL LETTER C WITH CEDILLA” can be represented as ' - 'a single\n' - ' *precomposed character* at code position U+00C7, or ' - 'as a sequence\n' - ' of a *base character* at code position U+0043 (LATIN ' - 'CAPITAL\n' - ' LETTER C), followed by a *combining character* at ' - 'code position\n' - ' U+0327 (COMBINING CEDILLA).\n' - '\n' - ' The comparison operators on strings compare at the ' - 'level of\n' - ' Unicode code points. This may be counter-intuitive ' - 'to humans. For\n' - ' example, ""\\u00C7" == "\\u0043\\u0327"" is "False", ' - 'even though both\n' - ' strings represent the same abstract character “LATIN ' - 'CAPITAL\n' - ' LETTER C WITH CEDILLA”.\n' - '\n' - ' To compare strings at the level of abstract ' - 'characters (that is,\n' - ' in a way intuitive to humans), use ' - '"unicodedata.normalize()".\n' - '\n' - '[4] Due to automatic garbage-collection, free lists, and ' - 'the\n' - ' dynamic nature of descriptors, you may notice ' - 'seemingly unusual\n' - ' behaviour in certain uses of the "is" operator, like ' - 'those\n' - ' involving comparisons between instance methods, or ' - 'constants.\n' - ' Check their documentation for more info.\n' - '\n' - '[5] The "%" operator is also used for string formatting; ' - 'the same\n' - ' precedence applies.\n' - '\n' - '[6] The power operator "**" binds less tightly than an ' - 'arithmetic\n' - ' or bitwise unary operator on its right, that is, ' - '"2**-1" is "0.5".\n', - 'pass': 'The "pass" statement\n' - '********************\n' - '\n' - ' pass_stmt ::= "pass"\n' - '\n' - '"pass" is a null operation — when it is executed, nothing happens. ' - 'It\n' - 'is useful as a placeholder when a statement is required ' - 'syntactically,\n' - 'but no code needs to be executed, for example:\n' - '\n' - ' def f(arg): pass # a function that does nothing (yet)\n' - '\n' - ' class C: pass # a class with no methods (yet)\n', - 'power': 'The power operator\n' - '******************\n' - '\n' - 'The power operator binds more tightly than unary operators on its\n' - 'left; it binds less tightly than unary operators on its right. ' - 'The\n' - 'syntax is:\n' - '\n' - ' power ::= (await_expr | primary) ["**" u_expr]\n' - '\n' - 'Thus, in an unparenthesized sequence of power and unary operators, ' - 'the\n' - 'operators are evaluated from right to left (this does not ' - 'constrain\n' - 'the evaluation order for the operands): "-1**2" results in "-1".\n' - '\n' - 'The power operator has the same semantics as the built-in "pow()"\n' - 'function, when called with two arguments: it yields its left ' - 'argument\n' - 'raised to the power of its right argument. The numeric arguments ' - 'are\n' - 'first converted to a common type, and the result is of that type.\n' - '\n' - 'For int operands, the result has the same type as the operands ' - 'unless\n' - 'the second argument is negative; in that case, all arguments are\n' - 'converted to float and a float result is delivered. For example,\n' - '"10**2" returns "100", but "10**-2" returns "0.01".\n' - '\n' - 'Raising "0.0" to a negative power results in a ' - '"ZeroDivisionError".\n' - 'Raising a negative number to a fractional power results in a ' - '"complex"\n' - 'number. (In earlier versions it raised a "ValueError".)\n', - 'raise': 'The "raise" statement\n' - '*********************\n' - '\n' - ' raise_stmt ::= "raise" [expression ["from" expression]]\n' - '\n' - 'If no expressions are present, "raise" re-raises the last ' - 'exception\n' - 'that was active in the current scope. If no exception is active ' - 'in\n' - 'the current scope, a "RuntimeError" exception is raised indicating\n' - 'that this is an error.\n' - '\n' - 'Otherwise, "raise" evaluates the first expression as the exception\n' - 'object. It must be either a subclass or an instance of\n' - '"BaseException". If it is a class, the exception instance will be\n' - 'obtained when needed by instantiating the class with no arguments.\n' - '\n' - 'The *type* of the exception is the exception instance’s class, the\n' - '*value* is the instance itself.\n' - '\n' - 'A traceback object is normally created automatically when an ' - 'exception\n' - 'is raised and attached to it as the "__traceback__" attribute, ' - 'which\n' - 'is writable. You can create an exception and set your own traceback ' - 'in\n' - 'one step using the "with_traceback()" exception method (which ' - 'returns\n' - 'the same exception instance, with its traceback set to its ' - 'argument),\n' - 'like so:\n' - '\n' - ' raise Exception("foo occurred").with_traceback(tracebackobj)\n' - '\n' - 'The "from" clause is used for exception chaining: if given, the ' - 'second\n' - '*expression* must be another exception class or instance, which ' - 'will\n' - 'then be attached to the raised exception as the "__cause__" ' - 'attribute\n' - '(which is writable). If the raised exception is not handled, both\n' - 'exceptions will be printed:\n' - '\n' - ' >>> try:\n' - ' ... print(1 / 0)\n' - ' ... except Exception as exc:\n' - ' ... raise RuntimeError("Something bad happened") from exc\n' - ' ...\n' - ' Traceback (most recent call last):\n' - ' File "", line 2, in \n' - ' ZeroDivisionError: division by zero\n' - '\n' - ' The above exception was the direct cause of the following ' - 'exception:\n' - '\n' - ' Traceback (most recent call last):\n' - ' File "", line 4, in \n' - ' RuntimeError: Something bad happened\n' - '\n' - 'A similar mechanism works implicitly if an exception is raised ' - 'inside\n' - 'an exception handler or a "finally" clause: the previous exception ' - 'is\n' - 'then attached as the new exception’s "__context__" attribute:\n' - '\n' - ' >>> try:\n' - ' ... print(1 / 0)\n' - ' ... except:\n' - ' ... raise RuntimeError("Something bad happened")\n' - ' ...\n' - ' Traceback (most recent call last):\n' - ' File "", line 2, in \n' - ' ZeroDivisionError: division by zero\n' - '\n' - ' During handling of the above exception, another exception ' - 'occurred:\n' - '\n' - ' Traceback (most recent call last):\n' - ' File "", line 4, in \n' - ' RuntimeError: Something bad happened\n' - '\n' - 'Exception chaining can be explicitly suppressed by specifying ' - '"None"\n' - 'in the "from" clause:\n' - '\n' - ' >>> try:\n' - ' ... print(1 / 0)\n' - ' ... except:\n' - ' ... raise RuntimeError("Something bad happened") from None\n' - ' ...\n' - ' Traceback (most recent call last):\n' - ' File "", line 4, in \n' - ' RuntimeError: Something bad happened\n' - '\n' - 'Additional information on exceptions can be found in section\n' - 'Exceptions, and information about handling exceptions is in ' - 'section\n' - 'The try statement.\n' - '\n' - 'Changed in version 3.3: "None" is now permitted as "Y" in "raise X\n' - 'from Y".\n' - '\n' - 'New in version 3.3: The "__suppress_context__" attribute to ' - 'suppress\n' - 'automatic display of the exception context.\n', - 'return': 'The "return" statement\n' - '**********************\n' - '\n' - ' return_stmt ::= "return" [expression_list]\n' - '\n' - '"return" may only occur syntactically nested in a function ' - 'definition,\n' - 'not within a nested class definition.\n' - '\n' - 'If an expression list is present, it is evaluated, else "None" is\n' - 'substituted.\n' - '\n' - '"return" leaves the current function call with the expression list ' - '(or\n' - '"None") as return value.\n' - '\n' - 'When "return" passes control out of a "try" statement with a ' - '"finally"\n' - 'clause, that "finally" clause is executed before really leaving ' - 'the\n' - 'function.\n' - '\n' - 'In a generator function, the "return" statement indicates that ' - 'the\n' - 'generator is done and will cause "StopIteration" to be raised. ' - 'The\n' - 'returned value (if any) is used as an argument to construct\n' - '"StopIteration" and becomes the "StopIteration.value" attribute.\n' - '\n' - 'In an asynchronous generator function, an empty "return" ' - 'statement\n' - 'indicates that the asynchronous generator is done and will cause\n' - '"StopAsyncIteration" to be raised. A non-empty "return" statement ' - 'is\n' - 'a syntax error in an asynchronous generator function.\n', - 'sequence-types': 'Emulating container types\n' - '*************************\n' - '\n' - 'The following methods can be defined to implement ' - 'container objects.\n' - 'Containers usually are sequences (such as lists or tuples) ' - 'or mappings\n' - '(like dictionaries), but can represent other containers as ' - 'well. The\n' - 'first set of methods is used either to emulate a sequence ' - 'or to\n' - 'emulate a mapping; the difference is that for a sequence, ' - 'the\n' - 'allowable keys should be the integers *k* for which "0 <= ' - 'k < N" where\n' - '*N* is the length of the sequence, or slice objects, which ' - 'define a\n' - 'range of items. It is also recommended that mappings ' - 'provide the\n' - 'methods "keys()", "values()", "items()", "get()", ' - '"clear()",\n' - '"setdefault()", "pop()", "popitem()", "copy()", and ' - '"update()"\n' - 'behaving similar to those for Python’s standard dictionary ' - 'objects.\n' - 'The "collections" module provides a "MutableMapping" ' - 'abstract base\n' - 'class to help create those methods from a base set of ' - '"__getitem__()",\n' - '"__setitem__()", "__delitem__()", and "keys()". Mutable ' - 'sequences\n' - 'should provide methods "append()", "count()", "index()", ' - '"extend()",\n' - '"insert()", "pop()", "remove()", "reverse()" and "sort()", ' - 'like Python\n' - 'standard list objects. Finally, sequence types should ' - 'implement\n' - 'addition (meaning concatenation) and multiplication ' - '(meaning\n' - 'repetition) by defining the methods "__add__()", ' - '"__radd__()",\n' - '"__iadd__()", "__mul__()", "__rmul__()" and "__imul__()" ' - 'described\n' - 'below; they should not define other numerical operators. ' - 'It is\n' - 'recommended that both mappings and sequences implement ' - 'the\n' - '"__contains__()" method to allow efficient use of the "in" ' - 'operator;\n' - 'for mappings, "in" should search the mapping’s keys; for ' - 'sequences, it\n' - 'should search through the values. It is further ' - 'recommended that both\n' - 'mappings and sequences implement the "__iter__()" method ' - 'to allow\n' - 'efficient iteration through the container; for mappings, ' - '"__iter__()"\n' - 'should be the same as "keys()"; for sequences, it should ' - 'iterate\n' - 'through the values.\n' - '\n' - 'object.__len__(self)\n' - '\n' - ' Called to implement the built-in function "len()". ' - 'Should return\n' - ' the length of the object, an integer ">=" 0. Also, an ' - 'object that\n' - ' doesn’t define a "__bool__()" method and whose ' - '"__len__()" method\n' - ' returns zero is considered to be false in a Boolean ' - 'context.\n' - '\n' - ' **CPython implementation detail:** In CPython, the ' - 'length is\n' - ' required to be at most "sys.maxsize". If the length is ' - 'larger than\n' - ' "sys.maxsize" some features (such as "len()") may ' - 'raise\n' - ' "OverflowError". To prevent raising "OverflowError" by ' - 'truth value\n' - ' testing, an object must define a "__bool__()" method.\n' - '\n' - 'object.__length_hint__(self)\n' - '\n' - ' Called to implement "operator.length_hint()". Should ' - 'return an\n' - ' estimated length for the object (which may be greater ' - 'or less than\n' - ' the actual length). The length must be an integer ">=" ' - '0. This\n' - ' method is purely an optimization and is never required ' - 'for\n' - ' correctness.\n' - '\n' - ' New in version 3.4.\n' - '\n' - 'Note: Slicing is done exclusively with the following three ' - 'methods.\n' - ' A call like\n' - '\n' - ' a[1:2] = b\n' - '\n' - ' is translated to\n' - '\n' - ' a[slice(1, 2, None)] = b\n' - '\n' - ' and so forth. Missing slice items are always filled in ' - 'with "None".\n' - '\n' - 'object.__getitem__(self, key)\n' - '\n' - ' Called to implement evaluation of "self[key]". For ' - 'sequence types,\n' - ' the accepted keys should be integers and slice ' - 'objects. Note that\n' - ' the special interpretation of negative indexes (if the ' - 'class wishes\n' - ' to emulate a sequence type) is up to the ' - '"__getitem__()" method. If\n' - ' *key* is of an inappropriate type, "TypeError" may be ' - 'raised; if of\n' - ' a value outside the set of indexes for the sequence ' - '(after any\n' - ' special interpretation of negative values), ' - '"IndexError" should be\n' - ' raised. For mapping types, if *key* is missing (not in ' - 'the\n' - ' container), "KeyError" should be raised.\n' - '\n' - ' Note: "for" loops expect that an "IndexError" will be ' - 'raised for\n' - ' illegal indexes to allow proper detection of the end ' - 'of the\n' - ' sequence.\n' - '\n' - 'object.__setitem__(self, key, value)\n' - '\n' - ' Called to implement assignment to "self[key]". Same ' - 'note as for\n' - ' "__getitem__()". This should only be implemented for ' - 'mappings if\n' - ' the objects support changes to the values for keys, or ' - 'if new keys\n' - ' can be added, or for sequences if elements can be ' - 'replaced. The\n' - ' same exceptions should be raised for improper *key* ' - 'values as for\n' - ' the "__getitem__()" method.\n' - '\n' - 'object.__delitem__(self, key)\n' - '\n' - ' Called to implement deletion of "self[key]". Same note ' - 'as for\n' - ' "__getitem__()". This should only be implemented for ' - 'mappings if\n' - ' the objects support removal of keys, or for sequences ' - 'if elements\n' - ' can be removed from the sequence. The same exceptions ' - 'should be\n' - ' raised for improper *key* values as for the ' - '"__getitem__()" method.\n' - '\n' - 'object.__missing__(self, key)\n' - '\n' - ' Called by "dict"."__getitem__()" to implement ' - '"self[key]" for dict\n' - ' subclasses when key is not in the dictionary.\n' - '\n' - 'object.__iter__(self)\n' - '\n' - ' This method is called when an iterator is required for ' - 'a container.\n' - ' This method should return a new iterator object that ' - 'can iterate\n' - ' over all the objects in the container. For mappings, ' - 'it should\n' - ' iterate over the keys of the container.\n' - '\n' - ' Iterator objects also need to implement this method; ' - 'they are\n' - ' required to return themselves. For more information on ' - 'iterator\n' - ' objects, see Iterator Types.\n' - '\n' - 'object.__reversed__(self)\n' - '\n' - ' Called (if present) by the "reversed()" built-in to ' - 'implement\n' - ' reverse iteration. It should return a new iterator ' - 'object that\n' - ' iterates over all the objects in the container in ' - 'reverse order.\n' - '\n' - ' If the "__reversed__()" method is not provided, the ' - '"reversed()"\n' - ' built-in will fall back to using the sequence protocol ' - '("__len__()"\n' - ' and "__getitem__()"). Objects that support the ' - 'sequence protocol\n' - ' should only provide "__reversed__()" if they can ' - 'provide an\n' - ' implementation that is more efficient than the one ' - 'provided by\n' - ' "reversed()".\n' - '\n' - 'The membership test operators ("in" and "not in") are ' - 'normally\n' - 'implemented as an iteration through a sequence. However, ' - 'container\n' - 'objects can supply the following special method with a ' - 'more efficient\n' - 'implementation, which also does not require the object be ' - 'a sequence.\n' - '\n' - 'object.__contains__(self, item)\n' - '\n' - ' Called to implement membership test operators. Should ' - 'return true\n' - ' if *item* is in *self*, false otherwise. For mapping ' - 'objects, this\n' - ' should consider the keys of the mapping rather than the ' - 'values or\n' - ' the key-item pairs.\n' - '\n' - ' For objects that don’t define "__contains__()", the ' - 'membership test\n' - ' first tries iteration via "__iter__()", then the old ' - 'sequence\n' - ' iteration protocol via "__getitem__()", see this ' - 'section in the\n' - ' language reference.\n', - 'shifting': 'Shifting operations\n' - '*******************\n' - '\n' - 'The shifting operations have lower priority than the arithmetic\n' - 'operations:\n' - '\n' - ' shift_expr ::= a_expr | shift_expr ("<<" | ">>") a_expr\n' - '\n' - 'These operators accept integers as arguments. They shift the ' - 'first\n' - 'argument to the left or right by the number of bits given by ' - 'the\n' - 'second argument.\n' - '\n' - 'A right shift by *n* bits is defined as floor division by ' - '"pow(2,n)".\n' - 'A left shift by *n* bits is defined as multiplication with ' - '"pow(2,n)".\n' - '\n' - 'Note: In the current implementation, the right-hand operand is\n' - ' required to be at most "sys.maxsize". If the right-hand ' - 'operand is\n' - ' larger than "sys.maxsize" an "OverflowError" exception is ' - 'raised.\n', - 'slicings': 'Slicings\n' - '********\n' - '\n' - 'A slicing selects a range of items in a sequence object (e.g., ' - 'a\n' - 'string, tuple or list). Slicings may be used as expressions or ' - 'as\n' - 'targets in assignment or "del" statements. The syntax for a ' - 'slicing:\n' - '\n' - ' slicing ::= primary "[" slice_list "]"\n' - ' slice_list ::= slice_item ("," slice_item)* [","]\n' - ' slice_item ::= expression | proper_slice\n' - ' proper_slice ::= [lower_bound] ":" [upper_bound] [ ":" ' - '[stride] ]\n' - ' lower_bound ::= expression\n' - ' upper_bound ::= expression\n' - ' stride ::= expression\n' - '\n' - 'There is ambiguity in the formal syntax here: anything that ' - 'looks like\n' - 'an expression list also looks like a slice list, so any ' - 'subscription\n' - 'can be interpreted as a slicing. Rather than further ' - 'complicating the\n' - 'syntax, this is disambiguated by defining that in this case the\n' - 'interpretation as a subscription takes priority over the\n' - 'interpretation as a slicing (this is the case if the slice list\n' - 'contains no proper slice).\n' - '\n' - 'The semantics for a slicing are as follows. The primary is ' - 'indexed\n' - '(using the same "__getitem__()" method as normal subscription) ' - 'with a\n' - 'key that is constructed from the slice list, as follows. If the ' - 'slice\n' - 'list contains at least one comma, the key is a tuple containing ' - 'the\n' - 'conversion of the slice items; otherwise, the conversion of the ' - 'lone\n' - 'slice item is the key. The conversion of a slice item that is ' - 'an\n' - 'expression is that expression. The conversion of a proper slice ' - 'is a\n' - 'slice object (see section The standard type hierarchy) whose ' - '"start",\n' - '"stop" and "step" attributes are the values of the expressions ' - 'given\n' - 'as lower bound, upper bound and stride, respectively, ' - 'substituting\n' - '"None" for missing expressions.\n', - 'specialattrs': 'Special Attributes\n' - '******************\n' - '\n' - 'The implementation adds a few special read-only attributes ' - 'to several\n' - 'object types, where they are relevant. Some of these are ' - 'not reported\n' - 'by the "dir()" built-in function.\n' - '\n' - 'object.__dict__\n' - '\n' - ' A dictionary or other mapping object used to store an ' - 'object’s\n' - ' (writable) attributes.\n' - '\n' - 'instance.__class__\n' - '\n' - ' The class to which a class instance belongs.\n' - '\n' - 'class.__bases__\n' - '\n' - ' The tuple of base classes of a class object.\n' - '\n' - 'definition.__name__\n' - '\n' - ' The name of the class, function, method, descriptor, or ' - 'generator\n' - ' instance.\n' - '\n' - 'definition.__qualname__\n' - '\n' - ' The *qualified name* of the class, function, method, ' - 'descriptor, or\n' - ' generator instance.\n' - '\n' - ' New in version 3.3.\n' - '\n' - 'class.__mro__\n' - '\n' - ' This attribute is a tuple of classes that are considered ' - 'when\n' - ' looking for base classes during method resolution.\n' - '\n' - 'class.mro()\n' - '\n' - ' This method can be overridden by a metaclass to customize ' - 'the\n' - ' method resolution order for its instances. It is called ' - 'at class\n' - ' instantiation, and its result is stored in "__mro__".\n' - '\n' - 'class.__subclasses__()\n' - '\n' - ' Each class keeps a list of weak references to its ' - 'immediate\n' - ' subclasses. This method returns a list of all those ' - 'references\n' - ' still alive. Example:\n' - '\n' - ' >>> int.__subclasses__()\n' - " []\n" - '\n' - '-[ Footnotes ]-\n' - '\n' - '[1] Additional information on these special methods may be ' - 'found\n' - ' in the Python Reference Manual (Basic customization).\n' - '\n' - '[2] As a consequence, the list "[1, 2]" is considered equal ' - 'to\n' - ' "[1.0, 2.0]", and similarly for tuples.\n' - '\n' - '[3] They must have since the parser can’t tell the type of ' - 'the\n' - ' operands.\n' - '\n' - '[4] Cased characters are those with general category ' - 'property\n' - ' being one of “Lu” (Letter, uppercase), “Ll” (Letter, ' - 'lowercase),\n' - ' or “Lt” (Letter, titlecase).\n' - '\n' - '[5] To format only a tuple you should therefore provide a\n' - ' singleton tuple whose only element is the tuple to be ' - 'formatted.\n', - 'specialnames': 'Special method names\n' - '********************\n' - '\n' - 'A class can implement certain operations that are invoked by ' - 'special\n' - 'syntax (such as arithmetic operations or subscripting and ' - 'slicing) by\n' - 'defining methods with special names. This is Python’s ' - 'approach to\n' - '*operator overloading*, allowing classes to define their own ' - 'behavior\n' - 'with respect to language operators. For instance, if a ' - 'class defines\n' - 'a method named "__getitem__()", and "x" is an instance of ' - 'this class,\n' - 'then "x[i]" is roughly equivalent to "type(x).__getitem__(x, ' - 'i)".\n' - 'Except where mentioned, attempts to execute an operation ' - 'raise an\n' - 'exception when no appropriate method is defined (typically\n' - '"AttributeError" or "TypeError").\n' - '\n' - 'Setting a special method to "None" indicates that the ' - 'corresponding\n' - 'operation is not available. For example, if a class sets ' - '"__iter__()"\n' - 'to "None", the class is not iterable, so calling "iter()" on ' - 'its\n' - 'instances will raise a "TypeError" (without falling back to\n' - '"__getitem__()"). [2]\n' - '\n' - 'When implementing a class that emulates any built-in type, ' - 'it is\n' - 'important that the emulation only be implemented to the ' - 'degree that it\n' - 'makes sense for the object being modelled. For example, ' - 'some\n' - 'sequences may work well with retrieval of individual ' - 'elements, but\n' - 'extracting a slice may not make sense. (One example of this ' - 'is the\n' - '"NodeList" interface in the W3C’s Document Object Model.)\n' - '\n' - '\n' - 'Basic customization\n' - '===================\n' - '\n' - 'object.__new__(cls[, ...])\n' - '\n' - ' Called to create a new instance of class *cls*. ' - '"__new__()" is a\n' - ' static method (special-cased so you need not declare it ' - 'as such)\n' - ' that takes the class of which an instance was requested ' - 'as its\n' - ' first argument. The remaining arguments are those passed ' - 'to the\n' - ' object constructor expression (the call to the class). ' - 'The return\n' - ' value of "__new__()" should be the new object instance ' - '(usually an\n' - ' instance of *cls*).\n' - '\n' - ' Typical implementations create a new instance of the ' - 'class by\n' - ' invoking the superclass’s "__new__()" method using\n' - ' "super().__new__(cls[, ...])" with appropriate arguments ' - 'and then\n' - ' modifying the newly-created instance as necessary before ' - 'returning\n' - ' it.\n' - '\n' - ' If "__new__()" returns an instance of *cls*, then the ' - 'new\n' - ' instance’s "__init__()" method will be invoked like\n' - ' "__init__(self[, ...])", where *self* is the new instance ' - 'and the\n' - ' remaining arguments are the same as were passed to ' - '"__new__()".\n' - '\n' - ' If "__new__()" does not return an instance of *cls*, then ' - 'the new\n' - ' instance’s "__init__()" method will not be invoked.\n' - '\n' - ' "__new__()" is intended mainly to allow subclasses of ' - 'immutable\n' - ' types (like int, str, or tuple) to customize instance ' - 'creation. It\n' - ' is also commonly overridden in custom metaclasses in ' - 'order to\n' - ' customize class creation.\n' - '\n' - 'object.__init__(self[, ...])\n' - '\n' - ' Called after the instance has been created (by ' - '"__new__()"), but\n' - ' before it is returned to the caller. The arguments are ' - 'those\n' - ' passed to the class constructor expression. If a base ' - 'class has an\n' - ' "__init__()" method, the derived class’s "__init__()" ' - 'method, if\n' - ' any, must explicitly call it to ensure proper ' - 'initialization of the\n' - ' base class part of the instance; for example:\n' - ' "super().__init__([args...])".\n' - '\n' - ' Because "__new__()" and "__init__()" work together in ' - 'constructing\n' - ' objects ("__new__()" to create it, and "__init__()" to ' - 'customize\n' - ' it), no non-"None" value may be returned by "__init__()"; ' - 'doing so\n' - ' will cause a "TypeError" to be raised at runtime.\n' - '\n' - 'object.__del__(self)\n' - '\n' - ' Called when the instance is about to be destroyed. This ' - 'is also\n' - ' called a finalizer or (improperly) a destructor. If a ' - 'base class\n' - ' has a "__del__()" method, the derived class’s "__del__()" ' - 'method,\n' - ' if any, must explicitly call it to ensure proper deletion ' - 'of the\n' - ' base class part of the instance.\n' - '\n' - ' It is possible (though not recommended!) for the ' - '"__del__()" method\n' - ' to postpone destruction of the instance by creating a new ' - 'reference\n' - ' to it. This is called object *resurrection*. It is\n' - ' implementation-dependent whether "__del__()" is called a ' - 'second\n' - ' time when a resurrected object is about to be destroyed; ' - 'the\n' - ' current *CPython* implementation only calls it once.\n' - '\n' - ' It is not guaranteed that "__del__()" methods are called ' - 'for\n' - ' objects that still exist when the interpreter exits.\n' - '\n' - ' Note: "del x" doesn’t directly call "x.__del__()" — the ' - 'former\n' - ' decrements the reference count for "x" by one, and the ' - 'latter is\n' - ' only called when "x"’s reference count reaches zero.\n' - '\n' - ' **CPython implementation detail:** It is possible for a ' - 'reference\n' - ' cycle to prevent the reference count of an object from ' - 'going to\n' - ' zero. In this case, the cycle will be later detected and ' - 'deleted\n' - ' by the *cyclic garbage collector*. A common cause of ' - 'reference\n' - ' cycles is when an exception has been caught in a local ' - 'variable.\n' - ' The frame’s locals then reference the exception, which ' - 'references\n' - ' its own traceback, which references the locals of all ' - 'frames caught\n' - ' in the traceback.\n' - '\n' - ' See also: Documentation for the "gc" module.\n' - '\n' - ' Warning: Due to the precarious circumstances under which\n' - ' "__del__()" methods are invoked, exceptions that occur ' - 'during\n' - ' their execution are ignored, and a warning is printed ' - 'to\n' - ' "sys.stderr" instead. In particular:\n' - '\n' - ' * "__del__()" can be invoked when arbitrary code is ' - 'being\n' - ' executed, including from any arbitrary thread. If ' - '"__del__()"\n' - ' needs to take a lock or invoke any other blocking ' - 'resource, it\n' - ' may deadlock as the resource may already be taken by ' - 'the code\n' - ' that gets interrupted to execute "__del__()".\n' - '\n' - ' * "__del__()" can be executed during interpreter ' - 'shutdown. As\n' - ' a consequence, the global variables it needs to ' - 'access\n' - ' (including other modules) may already have been ' - 'deleted or set\n' - ' to "None". Python guarantees that globals whose name ' - 'begins\n' - ' with a single underscore are deleted from their ' - 'module before\n' - ' other globals are deleted; if no other references to ' - 'such\n' - ' globals exist, this may help in assuring that ' - 'imported modules\n' - ' are still available at the time when the "__del__()" ' - 'method is\n' - ' called.\n' - '\n' - 'object.__repr__(self)\n' - '\n' - ' Called by the "repr()" built-in function to compute the ' - '“official”\n' - ' string representation of an object. If at all possible, ' - 'this\n' - ' should look like a valid Python expression that could be ' - 'used to\n' - ' recreate an object with the same value (given an ' - 'appropriate\n' - ' environment). If this is not possible, a string of the ' - 'form\n' - ' "<...some useful description...>" should be returned. The ' - 'return\n' - ' value must be a string object. If a class defines ' - '"__repr__()" but\n' - ' not "__str__()", then "__repr__()" is also used when an ' - '“informal”\n' - ' string representation of instances of that class is ' - 'required.\n' - '\n' - ' This is typically used for debugging, so it is important ' - 'that the\n' - ' representation is information-rich and unambiguous.\n' - '\n' - 'object.__str__(self)\n' - '\n' - ' Called by "str(object)" and the built-in functions ' - '"format()" and\n' - ' "print()" to compute the “informal” or nicely printable ' - 'string\n' - ' representation of an object. The return value must be a ' - 'string\n' - ' object.\n' - '\n' - ' This method differs from "object.__repr__()" in that ' - 'there is no\n' - ' expectation that "__str__()" return a valid Python ' - 'expression: a\n' - ' more convenient or concise representation can be used.\n' - '\n' - ' The default implementation defined by the built-in type ' - '"object"\n' - ' calls "object.__repr__()".\n' - '\n' - 'object.__bytes__(self)\n' - '\n' - ' Called by bytes to compute a byte-string representation ' - 'of an\n' - ' object. This should return a "bytes" object.\n' - '\n' - 'object.__format__(self, format_spec)\n' - '\n' - ' Called by the "format()" built-in function, and by ' - 'extension,\n' - ' evaluation of formatted string literals and the ' - '"str.format()"\n' - ' method, to produce a “formatted” string representation of ' - 'an\n' - ' object. The "format_spec" argument is a string that ' - 'contains a\n' - ' description of the formatting options desired. The ' - 'interpretation\n' - ' of the "format_spec" argument is up to the type ' - 'implementing\n' - ' "__format__()", however most classes will either ' - 'delegate\n' - ' formatting to one of the built-in types, or use a ' - 'similar\n' - ' formatting option syntax.\n' - '\n' - ' See Format Specification Mini-Language for a description ' - 'of the\n' - ' standard formatting syntax.\n' - '\n' - ' The return value must be a string object.\n' - '\n' - ' Changed in version 3.4: The __format__ method of "object" ' - 'itself\n' - ' raises a "TypeError" if passed any non-empty string.\n' - '\n' - 'object.__lt__(self, other)\n' - 'object.__le__(self, other)\n' - 'object.__eq__(self, other)\n' - 'object.__ne__(self, other)\n' - 'object.__gt__(self, other)\n' - 'object.__ge__(self, other)\n' - '\n' - ' These are the so-called “rich comparison” methods. The\n' - ' correspondence between operator symbols and method names ' - 'is as\n' - ' follows: "xy" calls\n' - ' "x.__gt__(y)", and "x>=y" calls "x.__ge__(y)".\n' - '\n' - ' A rich comparison method may return the singleton ' - '"NotImplemented"\n' - ' if it does not implement the operation for a given pair ' - 'of\n' - ' arguments. By convention, "False" and "True" are returned ' - 'for a\n' - ' successful comparison. However, these methods can return ' - 'any value,\n' - ' so if the comparison operator is used in a Boolean ' - 'context (e.g.,\n' - ' in the condition of an "if" statement), Python will call ' - '"bool()"\n' - ' on the value to determine if the result is true or ' - 'false.\n' - '\n' - ' By default, "__ne__()" delegates to "__eq__()" and ' - 'inverts the\n' - ' result unless it is "NotImplemented". There are no other ' - 'implied\n' - ' relationships among the comparison operators, for ' - 'example, the\n' - ' truth of "(x.__hash__".\n' - '\n' - ' If a class that does not override "__eq__()" wishes to ' - 'suppress\n' - ' hash support, it should include "__hash__ = None" in the ' - 'class\n' - ' definition. A class which defines its own "__hash__()" ' - 'that\n' - ' explicitly raises a "TypeError" would be incorrectly ' - 'identified as\n' - ' hashable by an "isinstance(obj, collections.Hashable)" ' - 'call.\n' - '\n' - ' Note: By default, the "__hash__()" values of str, bytes ' - 'and\n' - ' datetime objects are “salted” with an unpredictable ' - 'random value.\n' - ' Although they remain constant within an individual ' - 'Python\n' - ' process, they are not predictable between repeated ' - 'invocations of\n' - ' Python.This is intended to provide protection against a ' - 'denial-\n' - ' of-service caused by carefully-chosen inputs that ' - 'exploit the\n' - ' worst case performance of a dict insertion, O(n^2) ' - 'complexity.\n' - ' See http://www.ocert.org/advisories/ocert-2011-003.html ' - 'for\n' - ' details.Changing hash values affects the iteration ' - 'order of\n' - ' dicts, sets and other mappings. Python has never made ' - 'guarantees\n' - ' about this ordering (and it typically varies between ' - '32-bit and\n' - ' 64-bit builds).See also "PYTHONHASHSEED".\n' - '\n' - ' Changed in version 3.3: Hash randomization is enabled by ' - 'default.\n' - '\n' - 'object.__bool__(self)\n' - '\n' - ' Called to implement truth value testing and the built-in ' - 'operation\n' - ' "bool()"; should return "False" or "True". When this ' - 'method is not\n' - ' defined, "__len__()" is called, if it is defined, and the ' - 'object is\n' - ' considered true if its result is nonzero. If a class ' - 'defines\n' - ' neither "__len__()" nor "__bool__()", all its instances ' - 'are\n' - ' considered true.\n' - '\n' - '\n' - 'Customizing attribute access\n' - '============================\n' - '\n' - 'The following methods can be defined to customize the ' - 'meaning of\n' - 'attribute access (use of, assignment to, or deletion of ' - '"x.name") for\n' - 'class instances.\n' - '\n' - 'object.__getattr__(self, name)\n' - '\n' - ' Called when the default attribute access fails with an\n' - ' "AttributeError" (either "__getattribute__()" raises an\n' - ' "AttributeError" because *name* is not an instance ' - 'attribute or an\n' - ' attribute in the class tree for "self"; or "__get__()" of ' - 'a *name*\n' - ' property raises "AttributeError"). This method should ' - 'either\n' - ' return the (computed) attribute value or raise an ' - '"AttributeError"\n' - ' exception.\n' - '\n' - ' Note that if the attribute is found through the normal ' - 'mechanism,\n' - ' "__getattr__()" is not called. (This is an intentional ' - 'asymmetry\n' - ' between "__getattr__()" and "__setattr__()".) This is ' - 'done both for\n' - ' efficiency reasons and because otherwise "__getattr__()" ' - 'would have\n' - ' no way to access other attributes of the instance. Note ' - 'that at\n' - ' least for instance variables, you can fake total control ' - 'by not\n' - ' inserting any values in the instance attribute dictionary ' - '(but\n' - ' instead inserting them in another object). See the\n' - ' "__getattribute__()" method below for a way to actually ' - 'get total\n' - ' control over attribute access.\n' - '\n' - 'object.__getattribute__(self, name)\n' - '\n' - ' Called unconditionally to implement attribute accesses ' - 'for\n' - ' instances of the class. If the class also defines ' - '"__getattr__()",\n' - ' the latter will not be called unless "__getattribute__()" ' - 'either\n' - ' calls it explicitly or raises an "AttributeError". This ' - 'method\n' - ' should return the (computed) attribute value or raise an\n' - ' "AttributeError" exception. In order to avoid infinite ' - 'recursion in\n' - ' this method, its implementation should always call the ' - 'base class\n' - ' method with the same name to access any attributes it ' - 'needs, for\n' - ' example, "object.__getattribute__(self, name)".\n' - '\n' - ' Note: This method may still be bypassed when looking up ' - 'special\n' - ' methods as the result of implicit invocation via ' - 'language syntax\n' - ' or built-in functions. See Special method lookup.\n' - '\n' - 'object.__setattr__(self, name, value)\n' - '\n' - ' Called when an attribute assignment is attempted. This ' - 'is called\n' - ' instead of the normal mechanism (i.e. store the value in ' - 'the\n' - ' instance dictionary). *name* is the attribute name, ' - '*value* is the\n' - ' value to be assigned to it.\n' - '\n' - ' If "__setattr__()" wants to assign to an instance ' - 'attribute, it\n' - ' should call the base class method with the same name, for ' - 'example,\n' - ' "object.__setattr__(self, name, value)".\n' - '\n' - 'object.__delattr__(self, name)\n' - '\n' - ' Like "__setattr__()" but for attribute deletion instead ' - 'of\n' - ' assignment. This should only be implemented if "del ' - 'obj.name" is\n' - ' meaningful for the object.\n' - '\n' - 'object.__dir__(self)\n' - '\n' - ' Called when "dir()" is called on the object. A sequence ' - 'must be\n' - ' returned. "dir()" converts the returned sequence to a ' - 'list and\n' - ' sorts it.\n' - '\n' - '\n' - 'Customizing module attribute access\n' - '-----------------------------------\n' - '\n' - 'For a more fine grained customization of the module behavior ' - '(setting\n' - 'attributes, properties, etc.), one can set the "__class__" ' - 'attribute\n' - 'of a module object to a subclass of "types.ModuleType". For ' - 'example:\n' - '\n' - ' import sys\n' - ' from types import ModuleType\n' - '\n' - ' class VerboseModule(ModuleType):\n' - ' def __repr__(self):\n' - " return f'Verbose {self.__name__}'\n" - '\n' - ' def __setattr__(self, attr, value):\n' - " print(f'Setting {attr}...')\n" - ' setattr(self, attr, value)\n' - '\n' - ' sys.modules[__name__].__class__ = VerboseModule\n' - '\n' - 'Note: Setting module "__class__" only affects lookups made ' - 'using the\n' - ' attribute access syntax – directly accessing the module ' - 'globals\n' - ' (whether by code within the module, or via a reference to ' - 'the\n' - ' module’s globals dictionary) is unaffected.\n' - '\n' - 'Changed in version 3.5: "__class__" module attribute is now ' - 'writable.\n' - '\n' - '\n' - 'Implementing Descriptors\n' - '------------------------\n' - '\n' - 'The following methods only apply when an instance of the ' - 'class\n' - 'containing the method (a so-called *descriptor* class) ' - 'appears in an\n' - '*owner* class (the descriptor must be in either the owner’s ' - 'class\n' - 'dictionary or in the class dictionary for one of its ' - 'parents). In the\n' - 'examples below, “the attribute” refers to the attribute ' - 'whose name is\n' - 'the key of the property in the owner class’ "__dict__".\n' - '\n' - 'object.__get__(self, instance, owner)\n' - '\n' - ' Called to get the attribute of the owner class (class ' - 'attribute\n' - ' access) or of an instance of that class (instance ' - 'attribute\n' - ' access). *owner* is always the owner class, while ' - '*instance* is the\n' - ' instance that the attribute was accessed through, or ' - '"None" when\n' - ' the attribute is accessed through the *owner*. This ' - 'method should\n' - ' return the (computed) attribute value or raise an ' - '"AttributeError"\n' - ' exception.\n' - '\n' - 'object.__set__(self, instance, value)\n' - '\n' - ' Called to set the attribute on an instance *instance* of ' - 'the owner\n' - ' class to a new value, *value*.\n' - '\n' - 'object.__delete__(self, instance)\n' - '\n' - ' Called to delete the attribute on an instance *instance* ' - 'of the\n' - ' owner class.\n' - '\n' - 'object.__set_name__(self, owner, name)\n' - '\n' - ' Called at the time the owning class *owner* is created. ' - 'The\n' - ' descriptor has been assigned to *name*.\n' - '\n' - ' New in version 3.6.\n' - '\n' - 'The attribute "__objclass__" is interpreted by the "inspect" ' - 'module as\n' - 'specifying the class where this object was defined (setting ' - 'this\n' - 'appropriately can assist in runtime introspection of dynamic ' - 'class\n' - 'attributes). For callables, it may indicate that an instance ' - 'of the\n' - 'given type (or a subclass) is expected or required as the ' - 'first\n' - 'positional argument (for example, CPython sets this ' - 'attribute for\n' - 'unbound methods that are implemented in C).\n' - '\n' - '\n' - 'Invoking Descriptors\n' - '--------------------\n' - '\n' - 'In general, a descriptor is an object attribute with ' - '“binding\n' - 'behavior”, one whose attribute access has been overridden by ' - 'methods\n' - 'in the descriptor protocol: "__get__()", "__set__()", and\n' - '"__delete__()". If any of those methods are defined for an ' - 'object, it\n' - 'is said to be a descriptor.\n' - '\n' - 'The default behavior for attribute access is to get, set, or ' - 'delete\n' - 'the attribute from an object’s dictionary. For instance, ' - '"a.x" has a\n' - 'lookup chain starting with "a.__dict__[\'x\']", then\n' - '"type(a).__dict__[\'x\']", and continuing through the base ' - 'classes of\n' - '"type(a)" excluding metaclasses.\n' - '\n' - 'However, if the looked-up value is an object defining one of ' - 'the\n' - 'descriptor methods, then Python may override the default ' - 'behavior and\n' - 'invoke the descriptor method instead. Where this occurs in ' - 'the\n' - 'precedence chain depends on which descriptor methods were ' - 'defined and\n' - 'how they were called.\n' - '\n' - 'The starting point for descriptor invocation is a binding, ' - '"a.x". How\n' - 'the arguments are assembled depends on "a":\n' - '\n' - 'Direct Call\n' - ' The simplest and least common call is when user code ' - 'directly\n' - ' invokes a descriptor method: "x.__get__(a)".\n' - '\n' - 'Instance Binding\n' - ' If binding to an object instance, "a.x" is transformed ' - 'into the\n' - ' call: "type(a).__dict__[\'x\'].__get__(a, type(a))".\n' - '\n' - 'Class Binding\n' - ' If binding to a class, "A.x" is transformed into the ' - 'call:\n' - ' "A.__dict__[\'x\'].__get__(None, A)".\n' - '\n' - 'Super Binding\n' - ' If "a" is an instance of "super", then the binding ' - '"super(B,\n' - ' obj).m()" searches "obj.__class__.__mro__" for the base ' - 'class "A"\n' - ' immediately preceding "B" and then invokes the descriptor ' - 'with the\n' - ' call: "A.__dict__[\'m\'].__get__(obj, obj.__class__)".\n' - '\n' - 'For instance bindings, the precedence of descriptor ' - 'invocation depends\n' - 'on the which descriptor methods are defined. A descriptor ' - 'can define\n' - 'any combination of "__get__()", "__set__()" and ' - '"__delete__()". If it\n' - 'does not define "__get__()", then accessing the attribute ' - 'will return\n' - 'the descriptor object itself unless there is a value in the ' - 'object’s\n' - 'instance dictionary. If the descriptor defines "__set__()" ' - 'and/or\n' - '"__delete__()", it is a data descriptor; if it defines ' - 'neither, it is\n' - 'a non-data descriptor. Normally, data descriptors define ' - 'both\n' - '"__get__()" and "__set__()", while non-data descriptors have ' - 'just the\n' - '"__get__()" method. Data descriptors with "__set__()" and ' - '"__get__()"\n' - 'defined always override a redefinition in an instance ' - 'dictionary. In\n' - 'contrast, non-data descriptors can be overridden by ' - 'instances.\n' - '\n' - 'Python methods (including "staticmethod()" and ' - '"classmethod()") are\n' - 'implemented as non-data descriptors. Accordingly, instances ' - 'can\n' - 'redefine and override methods. This allows individual ' - 'instances to\n' - 'acquire behaviors that differ from other instances of the ' - 'same class.\n' - '\n' - 'The "property()" function is implemented as a data ' - 'descriptor.\n' - 'Accordingly, instances cannot override the behavior of a ' - 'property.\n' - '\n' - '\n' - '__slots__\n' - '---------\n' - '\n' - '*__slots__* allow us to explicitly declare data members ' - '(like\n' - 'properties) and deny the creation of *__dict__* and ' - '*__weakref__*\n' - '(unless explicitly declared in *__slots__* or available in a ' - 'parent.)\n' - '\n' - 'The space saved over using *__dict__* can be significant.\n' - '\n' - 'object.__slots__\n' - '\n' - ' This class variable can be assigned a string, iterable, ' - 'or sequence\n' - ' of strings with variable names used by instances. ' - '*__slots__*\n' - ' reserves space for the declared variables and prevents ' - 'the\n' - ' automatic creation of *__dict__* and *__weakref__* for ' - 'each\n' - ' instance.\n' - '\n' - '\n' - 'Notes on using *__slots__*\n' - '~~~~~~~~~~~~~~~~~~~~~~~~~~\n' - '\n' - '* When inheriting from a class without *__slots__*, the ' - '*__dict__*\n' - ' and *__weakref__* attribute of the instances will always ' - 'be\n' - ' accessible.\n' - '\n' - '* Without a *__dict__* variable, instances cannot be ' - 'assigned new\n' - ' variables not listed in the *__slots__* definition. ' - 'Attempts to\n' - ' assign to an unlisted variable name raises ' - '"AttributeError". If\n' - ' dynamic assignment of new variables is desired, then add\n' - ' "\'__dict__\'" to the sequence of strings in the ' - '*__slots__*\n' - ' declaration.\n' - '\n' - '* Without a *__weakref__* variable for each instance, ' - 'classes\n' - ' defining *__slots__* do not support weak references to ' - 'its\n' - ' instances. If weak reference support is needed, then add\n' - ' "\'__weakref__\'" to the sequence of strings in the ' - '*__slots__*\n' - ' declaration.\n' - '\n' - '* *__slots__* are implemented at the class level by ' - 'creating\n' - ' descriptors (Implementing Descriptors) for each variable ' - 'name. As a\n' - ' result, class attributes cannot be used to set default ' - 'values for\n' - ' instance variables defined by *__slots__*; otherwise, the ' - 'class\n' - ' attribute would overwrite the descriptor assignment.\n' - '\n' - '* The action of a *__slots__* declaration is not limited to ' - 'the\n' - ' class where it is defined. *__slots__* declared in ' - 'parents are\n' - ' available in child classes. However, child subclasses will ' - 'get a\n' - ' *__dict__* and *__weakref__* unless they also define ' - '*__slots__*\n' - ' (which should only contain names of any *additional* ' - 'slots).\n' - '\n' - '* If a class defines a slot also defined in a base class, ' - 'the\n' - ' instance variable defined by the base class slot is ' - 'inaccessible\n' - ' (except by retrieving its descriptor directly from the ' - 'base class).\n' - ' This renders the meaning of the program undefined. In the ' - 'future, a\n' - ' check may be added to prevent this.\n' - '\n' - '* Nonempty *__slots__* does not work for classes derived ' - 'from\n' - ' “variable-length” built-in types such as "int", "bytes" ' - 'and "tuple".\n' - '\n' - '* Any non-string iterable may be assigned to *__slots__*. ' - 'Mappings\n' - ' may also be used; however, in the future, special meaning ' - 'may be\n' - ' assigned to the values corresponding to each key.\n' - '\n' - '* *__class__* assignment works only if both classes have the ' - 'same\n' - ' *__slots__*.\n' - '\n' - '* Multiple inheritance with multiple slotted parent classes ' - 'can be\n' - ' used, but only one parent is allowed to have attributes ' - 'created by\n' - ' slots (the other bases must have empty slot layouts) - ' - 'violations\n' - ' raise "TypeError".\n' - '\n' - '\n' - 'Customizing class creation\n' - '==========================\n' - '\n' - 'Whenever a class inherits from another class, ' - '*__init_subclass__* is\n' - 'called on that class. This way, it is possible to write ' - 'classes which\n' - 'change the behavior of subclasses. This is closely related ' - 'to class\n' - 'decorators, but where class decorators only affect the ' - 'specific class\n' - 'they’re applied to, "__init_subclass__" solely applies to ' - 'future\n' - 'subclasses of the class defining the method.\n' - '\n' - 'classmethod object.__init_subclass__(cls)\n' - '\n' - ' This method is called whenever the containing class is ' - 'subclassed.\n' - ' *cls* is then the new subclass. If defined as a normal ' - 'instance\n' - ' method, this method is implicitly converted to a class ' - 'method.\n' - '\n' - ' Keyword arguments which are given to a new class are ' - 'passed to the\n' - ' parent’s class "__init_subclass__". For compatibility ' - 'with other\n' - ' classes using "__init_subclass__", one should take out ' - 'the needed\n' - ' keyword arguments and pass the others over to the base ' - 'class, as\n' - ' in:\n' - '\n' - ' class Philosopher:\n' - ' def __init_subclass__(cls, default_name, ' - '**kwargs):\n' - ' super().__init_subclass__(**kwargs)\n' - ' cls.default_name = default_name\n' - '\n' - ' class AustralianPhilosopher(Philosopher, ' - 'default_name="Bruce"):\n' - ' pass\n' - '\n' - ' The default implementation "object.__init_subclass__" ' - 'does nothing,\n' - ' but raises an error if it is called with any arguments.\n' - '\n' - ' Note: The metaclass hint "metaclass" is consumed by the ' - 'rest of\n' - ' the type machinery, and is never passed to ' - '"__init_subclass__"\n' - ' implementations. The actual metaclass (rather than the ' - 'explicit\n' - ' hint) can be accessed as "type(cls)".\n' - '\n' - ' New in version 3.6.\n' - '\n' - '\n' - 'Metaclasses\n' - '-----------\n' - '\n' - 'By default, classes are constructed using "type()". The ' - 'class body is\n' - 'executed in a new namespace and the class name is bound ' - 'locally to the\n' - 'result of "type(name, bases, namespace)".\n' - '\n' - 'The class creation process can be customized by passing the\n' - '"metaclass" keyword argument in the class definition line, ' - 'or by\n' - 'inheriting from an existing class that included such an ' - 'argument. In\n' - 'the following example, both "MyClass" and "MySubclass" are ' - 'instances\n' - 'of "Meta":\n' - '\n' - ' class Meta(type):\n' - ' pass\n' - '\n' - ' class MyClass(metaclass=Meta):\n' - ' pass\n' - '\n' - ' class MySubclass(MyClass):\n' - ' pass\n' - '\n' - 'Any other keyword arguments that are specified in the class ' - 'definition\n' - 'are passed through to all metaclass operations described ' - 'below.\n' - '\n' - 'When a class definition is executed, the following steps ' - 'occur:\n' - '\n' - '* the appropriate metaclass is determined\n' - '\n' - '* the class namespace is prepared\n' - '\n' - '* the class body is executed\n' - '\n' - '* the class object is created\n' - '\n' - '\n' - 'Determining the appropriate metaclass\n' - '-------------------------------------\n' - '\n' - 'The appropriate metaclass for a class definition is ' - 'determined as\n' - 'follows:\n' - '\n' - '* if no bases and no explicit metaclass are given, then ' - '"type()" is\n' - ' used\n' - '\n' - '* if an explicit metaclass is given and it is *not* an ' - 'instance of\n' - ' "type()", then it is used directly as the metaclass\n' - '\n' - '* if an instance of "type()" is given as the explicit ' - 'metaclass, or\n' - ' bases are defined, then the most derived metaclass is ' - 'used\n' - '\n' - 'The most derived metaclass is selected from the explicitly ' - 'specified\n' - 'metaclass (if any) and the metaclasses (i.e. "type(cls)") of ' - 'all\n' - 'specified base classes. The most derived metaclass is one ' - 'which is a\n' - 'subtype of *all* of these candidate metaclasses. If none of ' - 'the\n' - 'candidate metaclasses meets that criterion, then the class ' - 'definition\n' - 'will fail with "TypeError".\n' - '\n' - '\n' - 'Preparing the class namespace\n' - '-----------------------------\n' - '\n' - 'Once the appropriate metaclass has been identified, then the ' - 'class\n' - 'namespace is prepared. If the metaclass has a "__prepare__" ' - 'attribute,\n' - 'it is called as "namespace = metaclass.__prepare__(name, ' - 'bases,\n' - '**kwds)" (where the additional keyword arguments, if any, ' - 'come from\n' - 'the class definition).\n' - '\n' - 'If the metaclass has no "__prepare__" attribute, then the ' - 'class\n' - 'namespace is initialised as an empty ordered mapping.\n' - '\n' - 'See also:\n' - '\n' - ' **PEP 3115** - Metaclasses in Python 3000\n' - ' Introduced the "__prepare__" namespace hook\n' - '\n' - '\n' - 'Executing the class body\n' - '------------------------\n' - '\n' - 'The class body is executed (approximately) as "exec(body, ' - 'globals(),\n' - 'namespace)". The key difference from a normal call to ' - '"exec()" is that\n' - 'lexical scoping allows the class body (including any ' - 'methods) to\n' - 'reference names from the current and outer scopes when the ' - 'class\n' - 'definition occurs inside a function.\n' - '\n' - 'However, even when the class definition occurs inside the ' - 'function,\n' - 'methods defined inside the class still cannot see names ' - 'defined at the\n' - 'class scope. Class variables must be accessed through the ' - 'first\n' - 'parameter of instance or class methods, or through the ' - 'implicit\n' - 'lexically scoped "__class__" reference described in the next ' - 'section.\n' - '\n' - '\n' - 'Creating the class object\n' - '-------------------------\n' - '\n' - 'Once the class namespace has been populated by executing the ' - 'class\n' - 'body, the class object is created by calling ' - '"metaclass(name, bases,\n' - 'namespace, **kwds)" (the additional keywords passed here are ' - 'the same\n' - 'as those passed to "__prepare__").\n' - '\n' - 'This class object is the one that will be referenced by the ' - 'zero-\n' - 'argument form of "super()". "__class__" is an implicit ' - 'closure\n' - 'reference created by the compiler if any methods in a class ' - 'body refer\n' - 'to either "__class__" or "super". This allows the zero ' - 'argument form\n' - 'of "super()" to correctly identify the class being defined ' - 'based on\n' - 'lexical scoping, while the class or instance that was used ' - 'to make the\n' - 'current call is identified based on the first argument ' - 'passed to the\n' - 'method.\n' - '\n' - '**CPython implementation detail:** In CPython 3.6 and later, ' - 'the\n' - '"__class__" cell is passed to the metaclass as a ' - '"__classcell__" entry\n' - 'in the class namespace. If present, this must be propagated ' - 'up to the\n' - '"type.__new__" call in order for the class to be ' - 'initialised\n' - 'correctly. Failing to do so will result in a ' - '"DeprecationWarning" in\n' - 'Python 3.6, and a "RuntimeError" in Python 3.8.\n' - '\n' - 'When using the default metaclass "type", or any metaclass ' - 'that\n' - 'ultimately calls "type.__new__", the following additional\n' - 'customisation steps are invoked after creating the class ' - 'object:\n' - '\n' - '* first, "type.__new__" collects all of the descriptors in ' - 'the class\n' - ' namespace that define a "__set_name__()" method;\n' - '\n' - '* second, all of these "__set_name__" methods are called ' - 'with the\n' - ' class being defined and the assigned name of that ' - 'particular\n' - ' descriptor; and\n' - '\n' - '* finally, the "__init_subclass__()" hook is called on the ' - 'immediate\n' - ' parent of the new class in its method resolution order.\n' - '\n' - 'After the class object is created, it is passed to the ' - 'class\n' - 'decorators included in the class definition (if any) and the ' - 'resulting\n' - 'object is bound in the local namespace as the defined ' - 'class.\n' - '\n' - 'When a new class is created by "type.__new__", the object ' - 'provided as\n' - 'the namespace parameter is copied to a new ordered mapping ' - 'and the\n' - 'original object is discarded. The new copy is wrapped in a ' - 'read-only\n' - 'proxy, which becomes the "__dict__" attribute of the class ' - 'object.\n' - '\n' - 'See also:\n' - '\n' - ' **PEP 3135** - New super\n' - ' Describes the implicit "__class__" closure reference\n' - '\n' - '\n' - 'Uses for metaclasses\n' - '--------------------\n' - '\n' - 'The potential uses for metaclasses are boundless. Some ideas ' - 'that have\n' - 'been explored include enum, logging, interface checking, ' - 'automatic\n' - 'delegation, automatic property creation, proxies, ' - 'frameworks, and\n' - 'automatic resource locking/synchronization.\n' - '\n' - '\n' - 'Customizing instance and subclass checks\n' - '========================================\n' - '\n' - 'The following methods are used to override the default ' - 'behavior of the\n' - '"isinstance()" and "issubclass()" built-in functions.\n' - '\n' - 'In particular, the metaclass "abc.ABCMeta" implements these ' - 'methods in\n' - 'order to allow the addition of Abstract Base Classes (ABCs) ' - 'as\n' - '“virtual base classes” to any class or type (including ' - 'built-in\n' - 'types), including other ABCs.\n' - '\n' - 'class.__instancecheck__(self, instance)\n' - '\n' - ' Return true if *instance* should be considered a (direct ' - 'or\n' - ' indirect) instance of *class*. If defined, called to ' - 'implement\n' - ' "isinstance(instance, class)".\n' - '\n' - 'class.__subclasscheck__(self, subclass)\n' - '\n' - ' Return true if *subclass* should be considered a (direct ' - 'or\n' - ' indirect) subclass of *class*. If defined, called to ' - 'implement\n' - ' "issubclass(subclass, class)".\n' - '\n' - 'Note that these methods are looked up on the type ' - '(metaclass) of a\n' - 'class. They cannot be defined as class methods in the ' - 'actual class.\n' - 'This is consistent with the lookup of special methods that ' - 'are called\n' - 'on instances, only in this case the instance is itself a ' - 'class.\n' - '\n' - 'See also:\n' - '\n' - ' **PEP 3119** - Introducing Abstract Base Classes\n' - ' Includes the specification for customizing ' - '"isinstance()" and\n' - ' "issubclass()" behavior through "__instancecheck__()" ' - 'and\n' - ' "__subclasscheck__()", with motivation for this ' - 'functionality in\n' - ' the context of adding Abstract Base Classes (see the ' - '"abc"\n' - ' module) to the language.\n' - '\n' - '\n' - 'Emulating callable objects\n' - '==========================\n' - '\n' - 'object.__call__(self[, args...])\n' - '\n' - ' Called when the instance is “called” as a function; if ' - 'this method\n' - ' is defined, "x(arg1, arg2, ...)" is a shorthand for\n' - ' "x.__call__(arg1, arg2, ...)".\n' - '\n' - '\n' - 'Emulating container types\n' - '=========================\n' - '\n' - 'The following methods can be defined to implement container ' - 'objects.\n' - 'Containers usually are sequences (such as lists or tuples) ' - 'or mappings\n' - '(like dictionaries), but can represent other containers as ' - 'well. The\n' - 'first set of methods is used either to emulate a sequence or ' - 'to\n' - 'emulate a mapping; the difference is that for a sequence, ' - 'the\n' - 'allowable keys should be the integers *k* for which "0 <= k ' - '< N" where\n' - '*N* is the length of the sequence, or slice objects, which ' - 'define a\n' - 'range of items. It is also recommended that mappings ' - 'provide the\n' - 'methods "keys()", "values()", "items()", "get()", ' - '"clear()",\n' - '"setdefault()", "pop()", "popitem()", "copy()", and ' - '"update()"\n' - 'behaving similar to those for Python’s standard dictionary ' - 'objects.\n' - 'The "collections" module provides a "MutableMapping" ' - 'abstract base\n' - 'class to help create those methods from a base set of ' - '"__getitem__()",\n' - '"__setitem__()", "__delitem__()", and "keys()". Mutable ' - 'sequences\n' - 'should provide methods "append()", "count()", "index()", ' - '"extend()",\n' - '"insert()", "pop()", "remove()", "reverse()" and "sort()", ' - 'like Python\n' - 'standard list objects. Finally, sequence types should ' - 'implement\n' - 'addition (meaning concatenation) and multiplication ' - '(meaning\n' - 'repetition) by defining the methods "__add__()", ' - '"__radd__()",\n' - '"__iadd__()", "__mul__()", "__rmul__()" and "__imul__()" ' - 'described\n' - 'below; they should not define other numerical operators. It ' - 'is\n' - 'recommended that both mappings and sequences implement the\n' - '"__contains__()" method to allow efficient use of the "in" ' - 'operator;\n' - 'for mappings, "in" should search the mapping’s keys; for ' - 'sequences, it\n' - 'should search through the values. It is further recommended ' - 'that both\n' - 'mappings and sequences implement the "__iter__()" method to ' - 'allow\n' - 'efficient iteration through the container; for mappings, ' - '"__iter__()"\n' - 'should be the same as "keys()"; for sequences, it should ' - 'iterate\n' - 'through the values.\n' - '\n' - 'object.__len__(self)\n' - '\n' - ' Called to implement the built-in function "len()". ' - 'Should return\n' - ' the length of the object, an integer ">=" 0. Also, an ' - 'object that\n' - ' doesn’t define a "__bool__()" method and whose ' - '"__len__()" method\n' - ' returns zero is considered to be false in a Boolean ' - 'context.\n' - '\n' - ' **CPython implementation detail:** In CPython, the length ' - 'is\n' - ' required to be at most "sys.maxsize". If the length is ' - 'larger than\n' - ' "sys.maxsize" some features (such as "len()") may raise\n' - ' "OverflowError". To prevent raising "OverflowError" by ' - 'truth value\n' - ' testing, an object must define a "__bool__()" method.\n' - '\n' - 'object.__length_hint__(self)\n' - '\n' - ' Called to implement "operator.length_hint()". Should ' - 'return an\n' - ' estimated length for the object (which may be greater or ' - 'less than\n' - ' the actual length). The length must be an integer ">=" 0. ' - 'This\n' - ' method is purely an optimization and is never required ' - 'for\n' - ' correctness.\n' - '\n' - ' New in version 3.4.\n' - '\n' - 'Note: Slicing is done exclusively with the following three ' - 'methods.\n' - ' A call like\n' - '\n' - ' a[1:2] = b\n' - '\n' - ' is translated to\n' - '\n' - ' a[slice(1, 2, None)] = b\n' - '\n' - ' and so forth. Missing slice items are always filled in ' - 'with "None".\n' - '\n' - 'object.__getitem__(self, key)\n' - '\n' - ' Called to implement evaluation of "self[key]". For ' - 'sequence types,\n' - ' the accepted keys should be integers and slice objects. ' - 'Note that\n' - ' the special interpretation of negative indexes (if the ' - 'class wishes\n' - ' to emulate a sequence type) is up to the "__getitem__()" ' - 'method. If\n' - ' *key* is of an inappropriate type, "TypeError" may be ' - 'raised; if of\n' - ' a value outside the set of indexes for the sequence ' - '(after any\n' - ' special interpretation of negative values), "IndexError" ' - 'should be\n' - ' raised. For mapping types, if *key* is missing (not in ' - 'the\n' - ' container), "KeyError" should be raised.\n' - '\n' - ' Note: "for" loops expect that an "IndexError" will be ' - 'raised for\n' - ' illegal indexes to allow proper detection of the end of ' - 'the\n' - ' sequence.\n' - '\n' - 'object.__setitem__(self, key, value)\n' - '\n' - ' Called to implement assignment to "self[key]". Same note ' - 'as for\n' - ' "__getitem__()". This should only be implemented for ' - 'mappings if\n' - ' the objects support changes to the values for keys, or if ' - 'new keys\n' - ' can be added, or for sequences if elements can be ' - 'replaced. The\n' - ' same exceptions should be raised for improper *key* ' - 'values as for\n' - ' the "__getitem__()" method.\n' - '\n' - 'object.__delitem__(self, key)\n' - '\n' - ' Called to implement deletion of "self[key]". Same note ' - 'as for\n' - ' "__getitem__()". This should only be implemented for ' - 'mappings if\n' - ' the objects support removal of keys, or for sequences if ' - 'elements\n' - ' can be removed from the sequence. The same exceptions ' - 'should be\n' - ' raised for improper *key* values as for the ' - '"__getitem__()" method.\n' - '\n' - 'object.__missing__(self, key)\n' - '\n' - ' Called by "dict"."__getitem__()" to implement "self[key]" ' - 'for dict\n' - ' subclasses when key is not in the dictionary.\n' - '\n' - 'object.__iter__(self)\n' - '\n' - ' This method is called when an iterator is required for a ' - 'container.\n' - ' This method should return a new iterator object that can ' - 'iterate\n' - ' over all the objects in the container. For mappings, it ' - 'should\n' - ' iterate over the keys of the container.\n' - '\n' - ' Iterator objects also need to implement this method; they ' - 'are\n' - ' required to return themselves. For more information on ' - 'iterator\n' - ' objects, see Iterator Types.\n' - '\n' - 'object.__reversed__(self)\n' - '\n' - ' Called (if present) by the "reversed()" built-in to ' - 'implement\n' - ' reverse iteration. It should return a new iterator ' - 'object that\n' - ' iterates over all the objects in the container in reverse ' - 'order.\n' - '\n' - ' If the "__reversed__()" method is not provided, the ' - '"reversed()"\n' - ' built-in will fall back to using the sequence protocol ' - '("__len__()"\n' - ' and "__getitem__()"). Objects that support the sequence ' - 'protocol\n' - ' should only provide "__reversed__()" if they can provide ' - 'an\n' - ' implementation that is more efficient than the one ' - 'provided by\n' - ' "reversed()".\n' - '\n' - 'The membership test operators ("in" and "not in") are ' - 'normally\n' - 'implemented as an iteration through a sequence. However, ' - 'container\n' - 'objects can supply the following special method with a more ' - 'efficient\n' - 'implementation, which also does not require the object be a ' - 'sequence.\n' - '\n' - 'object.__contains__(self, item)\n' - '\n' - ' Called to implement membership test operators. Should ' - 'return true\n' - ' if *item* is in *self*, false otherwise. For mapping ' - 'objects, this\n' - ' should consider the keys of the mapping rather than the ' - 'values or\n' - ' the key-item pairs.\n' - '\n' - ' For objects that don’t define "__contains__()", the ' - 'membership test\n' - ' first tries iteration via "__iter__()", then the old ' - 'sequence\n' - ' iteration protocol via "__getitem__()", see this section ' - 'in the\n' - ' language reference.\n' - '\n' - '\n' - 'Emulating numeric types\n' - '=======================\n' - '\n' - 'The following methods can be defined to emulate numeric ' - 'objects.\n' - 'Methods corresponding to operations that are not supported ' - 'by the\n' - 'particular kind of number implemented (e.g., bitwise ' - 'operations for\n' - 'non-integral numbers) should be left undefined.\n' - '\n' - 'object.__add__(self, other)\n' - 'object.__sub__(self, other)\n' - 'object.__mul__(self, other)\n' - 'object.__matmul__(self, other)\n' - 'object.__truediv__(self, other)\n' - 'object.__floordiv__(self, other)\n' - 'object.__mod__(self, other)\n' - 'object.__divmod__(self, other)\n' - 'object.__pow__(self, other[, modulo])\n' - 'object.__lshift__(self, other)\n' - 'object.__rshift__(self, other)\n' - 'object.__and__(self, other)\n' - 'object.__xor__(self, other)\n' - 'object.__or__(self, other)\n' - '\n' - ' These methods are called to implement the binary ' - 'arithmetic\n' - ' operations ("+", "-", "*", "@", "/", "//", "%", ' - '"divmod()",\n' - ' "pow()", "**", "<<", ">>", "&", "^", "|"). For instance, ' - 'to\n' - ' evaluate the expression "x + y", where *x* is an instance ' - 'of a\n' - ' class that has an "__add__()" method, "x.__add__(y)" is ' - 'called.\n' - ' The "__divmod__()" method should be the equivalent to ' - 'using\n' - ' "__floordiv__()" and "__mod__()"; it should not be ' - 'related to\n' - ' "__truediv__()". Note that "__pow__()" should be defined ' - 'to accept\n' - ' an optional third argument if the ternary version of the ' - 'built-in\n' - ' "pow()" function is to be supported.\n' - '\n' - ' If one of those methods does not support the operation ' - 'with the\n' - ' supplied arguments, it should return "NotImplemented".\n' - '\n' - 'object.__radd__(self, other)\n' - 'object.__rsub__(self, other)\n' - 'object.__rmul__(self, other)\n' - 'object.__rmatmul__(self, other)\n' - 'object.__rtruediv__(self, other)\n' - 'object.__rfloordiv__(self, other)\n' - 'object.__rmod__(self, other)\n' - 'object.__rdivmod__(self, other)\n' - 'object.__rpow__(self, other)\n' - 'object.__rlshift__(self, other)\n' - 'object.__rrshift__(self, other)\n' - 'object.__rand__(self, other)\n' - 'object.__rxor__(self, other)\n' - 'object.__ror__(self, other)\n' - '\n' - ' These methods are called to implement the binary ' - 'arithmetic\n' - ' operations ("+", "-", "*", "@", "/", "//", "%", ' - '"divmod()",\n' - ' "pow()", "**", "<<", ">>", "&", "^", "|") with reflected ' - '(swapped)\n' - ' operands. These functions are only called if the left ' - 'operand does\n' - ' not support the corresponding operation [3] and the ' - 'operands are of\n' - ' different types. [4] For instance, to evaluate the ' - 'expression "x -\n' - ' y", where *y* is an instance of a class that has an ' - '"__rsub__()"\n' - ' method, "y.__rsub__(x)" is called if "x.__sub__(y)" ' - 'returns\n' - ' *NotImplemented*.\n' - '\n' - ' Note that ternary "pow()" will not try calling ' - '"__rpow__()" (the\n' - ' coercion rules would become too complicated).\n' - '\n' - ' Note: If the right operand’s type is a subclass of the ' - 'left\n' - ' operand’s type and that subclass provides the reflected ' - 'method\n' - ' for the operation, this method will be called before ' - 'the left\n' - ' operand’s non-reflected method. This behavior allows ' - 'subclasses\n' - ' to override their ancestors’ operations.\n' - '\n' - 'object.__iadd__(self, other)\n' - 'object.__isub__(self, other)\n' - 'object.__imul__(self, other)\n' - 'object.__imatmul__(self, other)\n' - 'object.__itruediv__(self, other)\n' - 'object.__ifloordiv__(self, other)\n' - 'object.__imod__(self, other)\n' - 'object.__ipow__(self, other[, modulo])\n' - 'object.__ilshift__(self, other)\n' - 'object.__irshift__(self, other)\n' - 'object.__iand__(self, other)\n' - 'object.__ixor__(self, other)\n' - 'object.__ior__(self, other)\n' - '\n' - ' These methods are called to implement the augmented ' - 'arithmetic\n' - ' assignments ("+=", "-=", "*=", "@=", "/=", "//=", "%=", ' - '"**=",\n' - ' "<<=", ">>=", "&=", "^=", "|="). These methods should ' - 'attempt to\n' - ' do the operation in-place (modifying *self*) and return ' - 'the result\n' - ' (which could be, but does not have to be, *self*). If a ' - 'specific\n' - ' method is not defined, the augmented assignment falls ' - 'back to the\n' - ' normal methods. For instance, if *x* is an instance of a ' - 'class\n' - ' with an "__iadd__()" method, "x += y" is equivalent to "x ' - '=\n' - ' x.__iadd__(y)" . Otherwise, "x.__add__(y)" and ' - '"y.__radd__(x)" are\n' - ' considered, as with the evaluation of "x + y". In ' - 'certain\n' - ' situations, augmented assignment can result in unexpected ' - 'errors\n' - ' (see Why does a_tuple[i] += [‘item’] raise an exception ' - 'when the\n' - ' addition works?), but this behavior is in fact part of ' - 'the data\n' - ' model.\n' - '\n' - 'object.__neg__(self)\n' - 'object.__pos__(self)\n' - 'object.__abs__(self)\n' - 'object.__invert__(self)\n' - '\n' - ' Called to implement the unary arithmetic operations ("-", ' - '"+",\n' - ' "abs()" and "~").\n' - '\n' - 'object.__complex__(self)\n' - 'object.__int__(self)\n' - 'object.__float__(self)\n' - '\n' - ' Called to implement the built-in functions "complex()", ' - '"int()" and\n' - ' "float()". Should return a value of the appropriate ' - 'type.\n' - '\n' - 'object.__index__(self)\n' - '\n' - ' Called to implement "operator.index()", and whenever ' - 'Python needs\n' - ' to losslessly convert the numeric object to an integer ' - 'object (such\n' - ' as in slicing, or in the built-in "bin()", "hex()" and ' - '"oct()"\n' - ' functions). Presence of this method indicates that the ' - 'numeric\n' - ' object is an integer type. Must return an integer.\n' - '\n' - ' Note: In order to have a coherent integer type class, ' - 'when\n' - ' "__index__()" is defined "__int__()" should also be ' - 'defined, and\n' - ' both should return the same value.\n' - '\n' - 'object.__round__(self[, ndigits])\n' - 'object.__trunc__(self)\n' - 'object.__floor__(self)\n' - 'object.__ceil__(self)\n' - '\n' - ' Called to implement the built-in function "round()" and ' - '"math"\n' - ' functions "trunc()", "floor()" and "ceil()". Unless ' - '*ndigits* is\n' - ' passed to "__round__()" all these methods should return ' - 'the value\n' - ' of the object truncated to an "Integral" (typically an ' - '"int").\n' - '\n' - ' If "__int__()" is not defined then the built-in function ' - '"int()"\n' - ' falls back to "__trunc__()".\n' - '\n' - '\n' - 'With Statement Context Managers\n' - '===============================\n' - '\n' - 'A *context manager* is an object that defines the runtime ' - 'context to\n' - 'be established when executing a "with" statement. The ' - 'context manager\n' - 'handles the entry into, and the exit from, the desired ' - 'runtime context\n' - 'for the execution of the block of code. Context managers ' - 'are normally\n' - 'invoked using the "with" statement (described in section The ' - 'with\n' - 'statement), but can also be used by directly invoking their ' - 'methods.\n' - '\n' - 'Typical uses of context managers include saving and ' - 'restoring various\n' - 'kinds of global state, locking and unlocking resources, ' - 'closing opened\n' - 'files, etc.\n' - '\n' - 'For more information on context managers, see Context ' - 'Manager Types.\n' - '\n' - 'object.__enter__(self)\n' - '\n' - ' Enter the runtime context related to this object. The ' - '"with"\n' - ' statement will bind this method’s return value to the ' - 'target(s)\n' - ' specified in the "as" clause of the statement, if any.\n' - '\n' - 'object.__exit__(self, exc_type, exc_value, traceback)\n' - '\n' - ' Exit the runtime context related to this object. The ' - 'parameters\n' - ' describe the exception that caused the context to be ' - 'exited. If the\n' - ' context was exited without an exception, all three ' - 'arguments will\n' - ' be "None".\n' - '\n' - ' If an exception is supplied, and the method wishes to ' - 'suppress the\n' - ' exception (i.e., prevent it from being propagated), it ' - 'should\n' - ' return a true value. Otherwise, the exception will be ' - 'processed\n' - ' normally upon exit from this method.\n' - '\n' - ' Note that "__exit__()" methods should not reraise the ' - 'passed-in\n' - ' exception; this is the caller’s responsibility.\n' - '\n' - 'See also:\n' - '\n' - ' **PEP 343** - The “with” statement\n' - ' The specification, background, and examples for the ' - 'Python "with"\n' - ' statement.\n' - '\n' - '\n' - 'Special method lookup\n' - '=====================\n' - '\n' - 'For custom classes, implicit invocations of special methods ' - 'are only\n' - 'guaranteed to work correctly if defined on an object’s type, ' - 'not in\n' - 'the object’s instance dictionary. That behaviour is the ' - 'reason why\n' - 'the following code raises an exception:\n' - '\n' - ' >>> class C:\n' - ' ... pass\n' - ' ...\n' - ' >>> c = C()\n' - ' >>> c.__len__ = lambda: 5\n' - ' >>> len(c)\n' - ' Traceback (most recent call last):\n' - ' File "", line 1, in \n' - " TypeError: object of type 'C' has no len()\n" - '\n' - 'The rationale behind this behaviour lies with a number of ' - 'special\n' - 'methods such as "__hash__()" and "__repr__()" that are ' - 'implemented by\n' - 'all objects, including type objects. If the implicit lookup ' - 'of these\n' - 'methods used the conventional lookup process, they would ' - 'fail when\n' - 'invoked on the type object itself:\n' - '\n' - ' >>> 1 .__hash__() == hash(1)\n' - ' True\n' - ' >>> int.__hash__() == hash(int)\n' - ' Traceback (most recent call last):\n' - ' File "", line 1, in \n' - " TypeError: descriptor '__hash__' of 'int' object needs an " - 'argument\n' - '\n' - 'Incorrectly attempting to invoke an unbound method of a ' - 'class in this\n' - 'way is sometimes referred to as ‘metaclass confusion’, and ' - 'is avoided\n' - 'by bypassing the instance when looking up special methods:\n' - '\n' - ' >>> type(1).__hash__(1) == hash(1)\n' - ' True\n' - ' >>> type(int).__hash__(int) == hash(int)\n' - ' True\n' - '\n' - 'In addition to bypassing any instance attributes in the ' - 'interest of\n' - 'correctness, implicit special method lookup generally also ' - 'bypasses\n' - 'the "__getattribute__()" method even of the object’s ' - 'metaclass:\n' - '\n' - ' >>> class Meta(type):\n' - ' ... def __getattribute__(*args):\n' - ' ... print("Metaclass getattribute invoked")\n' - ' ... return type.__getattribute__(*args)\n' - ' ...\n' - ' >>> class C(object, metaclass=Meta):\n' - ' ... def __len__(self):\n' - ' ... return 10\n' - ' ... def __getattribute__(*args):\n' - ' ... print("Class getattribute invoked")\n' - ' ... return object.__getattribute__(*args)\n' - ' ...\n' - ' >>> c = C()\n' - ' >>> c.__len__() # Explicit lookup via ' - 'instance\n' - ' Class getattribute invoked\n' - ' 10\n' - ' >>> type(c).__len__(c) # Explicit lookup via ' - 'type\n' - ' Metaclass getattribute invoked\n' - ' 10\n' - ' >>> len(c) # Implicit lookup\n' - ' 10\n' - '\n' - 'Bypassing the "__getattribute__()" machinery in this fashion ' - 'provides\n' - 'significant scope for speed optimisations within the ' - 'interpreter, at\n' - 'the cost of some flexibility in the handling of special ' - 'methods (the\n' - 'special method *must* be set on the class object itself in ' - 'order to be\n' - 'consistently invoked by the interpreter).\n', - 'string-methods': 'String Methods\n' - '**************\n' - '\n' - 'Strings implement all of the common sequence operations, ' - 'along with\n' - 'the additional methods described below.\n' - '\n' - 'Strings also support two styles of string formatting, one ' - 'providing a\n' - 'large degree of flexibility and customization (see ' - '"str.format()",\n' - 'Format String Syntax and Custom String Formatting) and the ' - 'other based\n' - 'on C "printf" style formatting that handles a narrower ' - 'range of types\n' - 'and is slightly harder to use correctly, but is often ' - 'faster for the\n' - 'cases it can handle (printf-style String Formatting).\n' - '\n' - 'The Text Processing Services section of the standard ' - 'library covers a\n' - 'number of other modules that provide various text related ' - 'utilities\n' - '(including regular expression support in the "re" ' - 'module).\n' - '\n' - 'str.capitalize()\n' - '\n' - ' Return a copy of the string with its first character ' - 'capitalized\n' - ' and the rest lowercased.\n' - '\n' - 'str.casefold()\n' - '\n' - ' Return a casefolded copy of the string. Casefolded ' - 'strings may be\n' - ' used for caseless matching.\n' - '\n' - ' Casefolding is similar to lowercasing but more ' - 'aggressive because\n' - ' it is intended to remove all case distinctions in a ' - 'string. For\n' - ' example, the German lowercase letter "\'ß\'" is ' - 'equivalent to ""ss"".\n' - ' Since it is already lowercase, "lower()" would do ' - 'nothing to "\'ß\'";\n' - ' "casefold()" converts it to ""ss"".\n' - '\n' - ' The casefolding algorithm is described in section 3.13 ' - 'of the\n' - ' Unicode Standard.\n' - '\n' - ' New in version 3.3.\n' - '\n' - 'str.center(width[, fillchar])\n' - '\n' - ' Return centered in a string of length *width*. Padding ' - 'is done\n' - ' using the specified *fillchar* (default is an ASCII ' - 'space). The\n' - ' original string is returned if *width* is less than or ' - 'equal to\n' - ' "len(s)".\n' - '\n' - 'str.count(sub[, start[, end]])\n' - '\n' - ' Return the number of non-overlapping occurrences of ' - 'substring *sub*\n' - ' in the range [*start*, *end*]. Optional arguments ' - '*start* and\n' - ' *end* are interpreted as in slice notation.\n' - '\n' - 'str.encode(encoding="utf-8", errors="strict")\n' - '\n' - ' Return an encoded version of the string as a bytes ' - 'object. Default\n' - ' encoding is "\'utf-8\'". *errors* may be given to set a ' - 'different\n' - ' error handling scheme. The default for *errors* is ' - '"\'strict\'",\n' - ' meaning that encoding errors raise a "UnicodeError". ' - 'Other possible\n' - ' values are "\'ignore\'", "\'replace\'", ' - '"\'xmlcharrefreplace\'",\n' - ' "\'backslashreplace\'" and any other name registered ' - 'via\n' - ' "codecs.register_error()", see section Error Handlers. ' - 'For a list\n' - ' of possible encodings, see section Standard Encodings.\n' - '\n' - ' Changed in version 3.1: Support for keyword arguments ' - 'added.\n' - '\n' - 'str.endswith(suffix[, start[, end]])\n' - '\n' - ' Return "True" if the string ends with the specified ' - '*suffix*,\n' - ' otherwise return "False". *suffix* can also be a tuple ' - 'of suffixes\n' - ' to look for. With optional *start*, test beginning at ' - 'that\n' - ' position. With optional *end*, stop comparing at that ' - 'position.\n' - '\n' - 'str.expandtabs(tabsize=8)\n' - '\n' - ' Return a copy of the string where all tab characters ' - 'are replaced\n' - ' by one or more spaces, depending on the current column ' - 'and the\n' - ' given tab size. Tab positions occur every *tabsize* ' - 'characters\n' - ' (default is 8, giving tab positions at columns 0, 8, 16 ' - 'and so on).\n' - ' To expand the string, the current column is set to zero ' - 'and the\n' - ' string is examined character by character. If the ' - 'character is a\n' - ' tab ("\\t"), one or more space characters are inserted ' - 'in the result\n' - ' until the current column is equal to the next tab ' - 'position. (The\n' - ' tab character itself is not copied.) If the character ' - 'is a newline\n' - ' ("\\n") or return ("\\r"), it is copied and the current ' - 'column is\n' - ' reset to zero. Any other character is copied unchanged ' - 'and the\n' - ' current column is incremented by one regardless of how ' - 'the\n' - ' character is represented when printed.\n' - '\n' - " >>> '01\\t012\\t0123\\t01234'.expandtabs()\n" - " '01 012 0123 01234'\n" - " >>> '01\\t012\\t0123\\t01234'.expandtabs(4)\n" - " '01 012 0123 01234'\n" - '\n' - 'str.find(sub[, start[, end]])\n' - '\n' - ' Return the lowest index in the string where substring ' - '*sub* is\n' - ' found within the slice "s[start:end]". Optional ' - 'arguments *start*\n' - ' and *end* are interpreted as in slice notation. Return ' - '"-1" if\n' - ' *sub* is not found.\n' - '\n' - ' Note: The "find()" method should be used only if you ' - 'need to know\n' - ' the position of *sub*. To check if *sub* is a ' - 'substring or not,\n' - ' use the "in" operator:\n' - '\n' - " >>> 'Py' in 'Python'\n" - ' True\n' - '\n' - 'str.format(*args, **kwargs)\n' - '\n' - ' Perform a string formatting operation. The string on ' - 'which this\n' - ' method is called can contain literal text or ' - 'replacement fields\n' - ' delimited by braces "{}". Each replacement field ' - 'contains either\n' - ' the numeric index of a positional argument, or the name ' - 'of a\n' - ' keyword argument. Returns a copy of the string where ' - 'each\n' - ' replacement field is replaced with the string value of ' - 'the\n' - ' corresponding argument.\n' - '\n' - ' >>> "The sum of 1 + 2 is {0}".format(1+2)\n' - " 'The sum of 1 + 2 is 3'\n" - '\n' - ' See Format String Syntax for a description of the ' - 'various\n' - ' formatting options that can be specified in format ' - 'strings.\n' - '\n' - ' Note: When formatting a number ("int", "float", ' - '"complex",\n' - ' "decimal.Decimal" and subclasses) with the "n" type ' - '(ex:\n' - ' "\'{:n}\'.format(1234)"), the function temporarily ' - 'sets the\n' - ' "LC_CTYPE" locale to the "LC_NUMERIC" locale to ' - 'decode\n' - ' "decimal_point" and "thousands_sep" fields of ' - '"localeconv()" if\n' - ' they are non-ASCII or longer than 1 byte, and the ' - '"LC_NUMERIC"\n' - ' locale is different than the "LC_CTYPE" locale. This ' - 'temporary\n' - ' change affects other threads.\n' - '\n' - ' Changed in version 3.6.5: When formatting a number with ' - 'the "n"\n' - ' type, the function sets temporarily the "LC_CTYPE" ' - 'locale to the\n' - ' "LC_NUMERIC" locale in some cases.\n' - '\n' - 'str.format_map(mapping)\n' - '\n' - ' Similar to "str.format(**mapping)", except that ' - '"mapping" is used\n' - ' directly and not copied to a "dict". This is useful if ' - 'for example\n' - ' "mapping" is a dict subclass:\n' - '\n' - ' >>> class Default(dict):\n' - ' ... def __missing__(self, key):\n' - ' ... return key\n' - ' ...\n' - " >>> '{name} was born in " - "{country}'.format_map(Default(name='Guido'))\n" - " 'Guido was born in country'\n" - '\n' - ' New in version 3.2.\n' - '\n' - 'str.index(sub[, start[, end]])\n' - '\n' - ' Like "find()", but raise "ValueError" when the ' - 'substring is not\n' - ' found.\n' - '\n' - 'str.isalnum()\n' - '\n' - ' Return true if all characters in the string are ' - 'alphanumeric and\n' - ' there is at least one character, false otherwise. A ' - 'character "c"\n' - ' is alphanumeric if one of the following returns ' - '"True":\n' - ' "c.isalpha()", "c.isdecimal()", "c.isdigit()", or ' - '"c.isnumeric()".\n' - '\n' - 'str.isalpha()\n' - '\n' - ' Return true if all characters in the string are ' - 'alphabetic and\n' - ' there is at least one character, false otherwise. ' - 'Alphabetic\n' - ' characters are those characters defined in the Unicode ' - 'character\n' - ' database as “Letter”, i.e., those with general category ' - 'property\n' - ' being one of “Lm”, “Lt”, “Lu”, “Ll”, or “Lo”. Note ' - 'that this is\n' - ' different from the “Alphabetic” property defined in the ' - 'Unicode\n' - ' Standard.\n' - '\n' - 'str.isdecimal()\n' - '\n' - ' Return true if all characters in the string are decimal ' - 'characters\n' - ' and there is at least one character, false otherwise. ' - 'Decimal\n' - ' characters are those that can be used to form numbers ' - 'in base 10,\n' - ' e.g. U+0660, ARABIC-INDIC DIGIT ZERO. Formally a ' - 'decimal character\n' - ' is a character in the Unicode General Category “Nd”.\n' - '\n' - 'str.isdigit()\n' - '\n' - ' Return true if all characters in the string are digits ' - 'and there is\n' - ' at least one character, false otherwise. Digits ' - 'include decimal\n' - ' characters and digits that need special handling, such ' - 'as the\n' - ' compatibility superscript digits. This covers digits ' - 'which cannot\n' - ' be used to form numbers in base 10, like the Kharosthi ' - 'numbers.\n' - ' Formally, a digit is a character that has the property ' - 'value\n' - ' Numeric_Type=Digit or Numeric_Type=Decimal.\n' - '\n' - 'str.isidentifier()\n' - '\n' - ' Return true if the string is a valid identifier ' - 'according to the\n' - ' language definition, section Identifiers and keywords.\n' - '\n' - ' Use "keyword.iskeyword()" to test for reserved ' - 'identifiers such as\n' - ' "def" and "class".\n' - '\n' - 'str.islower()\n' - '\n' - ' Return true if all cased characters [4] in the string ' - 'are lowercase\n' - ' and there is at least one cased character, false ' - 'otherwise.\n' - '\n' - 'str.isnumeric()\n' - '\n' - ' Return true if all characters in the string are numeric ' - 'characters,\n' - ' and there is at least one character, false otherwise. ' - 'Numeric\n' - ' characters include digit characters, and all characters ' - 'that have\n' - ' the Unicode numeric value property, e.g. U+2155, VULGAR ' - 'FRACTION\n' - ' ONE FIFTH. Formally, numeric characters are those with ' - 'the\n' - ' property value Numeric_Type=Digit, Numeric_Type=Decimal ' - 'or\n' - ' Numeric_Type=Numeric.\n' - '\n' - 'str.isprintable()\n' - '\n' - ' Return true if all characters in the string are ' - 'printable or the\n' - ' string is empty, false otherwise. Nonprintable ' - 'characters are\n' - ' those characters defined in the Unicode character ' - 'database as\n' - ' “Other” or “Separator”, excepting the ASCII space ' - '(0x20) which is\n' - ' considered printable. (Note that printable characters ' - 'in this\n' - ' context are those which should not be escaped when ' - '"repr()" is\n' - ' invoked on a string. It has no bearing on the handling ' - 'of strings\n' - ' written to "sys.stdout" or "sys.stderr".)\n' - '\n' - 'str.isspace()\n' - '\n' - ' Return true if there are only whitespace characters in ' - 'the string\n' - ' and there is at least one character, false otherwise. ' - 'Whitespace\n' - ' characters are those characters defined in the Unicode ' - 'character\n' - ' database as “Other” or “Separator” and those with ' - 'bidirectional\n' - ' property being one of “WS”, “B”, or “S”.\n' - '\n' - 'str.istitle()\n' - '\n' - ' Return true if the string is a titlecased string and ' - 'there is at\n' - ' least one character, for example uppercase characters ' - 'may only\n' - ' follow uncased characters and lowercase characters only ' - 'cased ones.\n' - ' Return false otherwise.\n' - '\n' - 'str.isupper()\n' - '\n' - ' Return true if all cased characters [4] in the string ' - 'are uppercase\n' - ' and there is at least one cased character, false ' - 'otherwise.\n' - '\n' - 'str.join(iterable)\n' - '\n' - ' Return a string which is the concatenation of the ' - 'strings in\n' - ' *iterable*. A "TypeError" will be raised if there are ' - 'any non-\n' - ' string values in *iterable*, including "bytes" ' - 'objects. The\n' - ' separator between elements is the string providing this ' - 'method.\n' - '\n' - 'str.ljust(width[, fillchar])\n' - '\n' - ' Return the string left justified in a string of length ' - '*width*.\n' - ' Padding is done using the specified *fillchar* (default ' - 'is an ASCII\n' - ' space). The original string is returned if *width* is ' - 'less than or\n' - ' equal to "len(s)".\n' - '\n' - 'str.lower()\n' - '\n' - ' Return a copy of the string with all the cased ' - 'characters [4]\n' - ' converted to lowercase.\n' - '\n' - ' The lowercasing algorithm used is described in section ' - '3.13 of the\n' - ' Unicode Standard.\n' - '\n' - 'str.lstrip([chars])\n' - '\n' - ' Return a copy of the string with leading characters ' - 'removed. The\n' - ' *chars* argument is a string specifying the set of ' - 'characters to be\n' - ' removed. If omitted or "None", the *chars* argument ' - 'defaults to\n' - ' removing whitespace. The *chars* argument is not a ' - 'prefix; rather,\n' - ' all combinations of its values are stripped:\n' - '\n' - " >>> ' spacious '.lstrip()\n" - " 'spacious '\n" - " >>> 'www.example.com'.lstrip('cmowz.')\n" - " 'example.com'\n" - '\n' - 'static str.maketrans(x[, y[, z]])\n' - '\n' - ' This static method returns a translation table usable ' - 'for\n' - ' "str.translate()".\n' - '\n' - ' If there is only one argument, it must be a dictionary ' - 'mapping\n' - ' Unicode ordinals (integers) or characters (strings of ' - 'length 1) to\n' - ' Unicode ordinals, strings (of arbitrary lengths) or ' - '"None".\n' - ' Character keys will then be converted to ordinals.\n' - '\n' - ' If there are two arguments, they must be strings of ' - 'equal length,\n' - ' and in the resulting dictionary, each character in x ' - 'will be mapped\n' - ' to the character at the same position in y. If there ' - 'is a third\n' - ' argument, it must be a string, whose characters will be ' - 'mapped to\n' - ' "None" in the result.\n' - '\n' - 'str.partition(sep)\n' - '\n' - ' Split the string at the first occurrence of *sep*, and ' - 'return a\n' - ' 3-tuple containing the part before the separator, the ' - 'separator\n' - ' itself, and the part after the separator. If the ' - 'separator is not\n' - ' found, return a 3-tuple containing the string itself, ' - 'followed by\n' - ' two empty strings.\n' - '\n' - 'str.replace(old, new[, count])\n' - '\n' - ' Return a copy of the string with all occurrences of ' - 'substring *old*\n' - ' replaced by *new*. If the optional argument *count* is ' - 'given, only\n' - ' the first *count* occurrences are replaced.\n' - '\n' - 'str.rfind(sub[, start[, end]])\n' - '\n' - ' Return the highest index in the string where substring ' - '*sub* is\n' - ' found, such that *sub* is contained within ' - '"s[start:end]".\n' - ' Optional arguments *start* and *end* are interpreted as ' - 'in slice\n' - ' notation. Return "-1" on failure.\n' - '\n' - 'str.rindex(sub[, start[, end]])\n' - '\n' - ' Like "rfind()" but raises "ValueError" when the ' - 'substring *sub* is\n' - ' not found.\n' - '\n' - 'str.rjust(width[, fillchar])\n' - '\n' - ' Return the string right justified in a string of length ' - '*width*.\n' - ' Padding is done using the specified *fillchar* (default ' - 'is an ASCII\n' - ' space). The original string is returned if *width* is ' - 'less than or\n' - ' equal to "len(s)".\n' - '\n' - 'str.rpartition(sep)\n' - '\n' - ' Split the string at the last occurrence of *sep*, and ' - 'return a\n' - ' 3-tuple containing the part before the separator, the ' - 'separator\n' - ' itself, and the part after the separator. If the ' - 'separator is not\n' - ' found, return a 3-tuple containing two empty strings, ' - 'followed by\n' - ' the string itself.\n' - '\n' - 'str.rsplit(sep=None, maxsplit=-1)\n' - '\n' - ' Return a list of the words in the string, using *sep* ' - 'as the\n' - ' delimiter string. If *maxsplit* is given, at most ' - '*maxsplit* splits\n' - ' are done, the *rightmost* ones. If *sep* is not ' - 'specified or\n' - ' "None", any whitespace string is a separator. Except ' - 'for splitting\n' - ' from the right, "rsplit()" behaves like "split()" which ' - 'is\n' - ' described in detail below.\n' - '\n' - 'str.rstrip([chars])\n' - '\n' - ' Return a copy of the string with trailing characters ' - 'removed. The\n' - ' *chars* argument is a string specifying the set of ' - 'characters to be\n' - ' removed. If omitted or "None", the *chars* argument ' - 'defaults to\n' - ' removing whitespace. The *chars* argument is not a ' - 'suffix; rather,\n' - ' all combinations of its values are stripped:\n' - '\n' - " >>> ' spacious '.rstrip()\n" - " ' spacious'\n" - " >>> 'mississippi'.rstrip('ipz')\n" - " 'mississ'\n" - '\n' - 'str.split(sep=None, maxsplit=-1)\n' - '\n' - ' Return a list of the words in the string, using *sep* ' - 'as the\n' - ' delimiter string. If *maxsplit* is given, at most ' - '*maxsplit*\n' - ' splits are done (thus, the list will have at most ' - '"maxsplit+1"\n' - ' elements). If *maxsplit* is not specified or "-1", ' - 'then there is\n' - ' no limit on the number of splits (all possible splits ' - 'are made).\n' - '\n' - ' If *sep* is given, consecutive delimiters are not ' - 'grouped together\n' - ' and are deemed to delimit empty strings (for example,\n' - ' "\'1,,2\'.split(\',\')" returns "[\'1\', \'\', ' - '\'2\']"). The *sep* argument\n' - ' may consist of multiple characters (for example,\n' - ' "\'1<>2<>3\'.split(\'<>\')" returns "[\'1\', \'2\', ' - '\'3\']"). Splitting an\n' - ' empty string with a specified separator returns ' - '"[\'\']".\n' - '\n' - ' For example:\n' - '\n' - " >>> '1,2,3'.split(',')\n" - " ['1', '2', '3']\n" - " >>> '1,2,3'.split(',', maxsplit=1)\n" - " ['1', '2,3']\n" - " >>> '1,2,,3,'.split(',')\n" - " ['1', '2', '', '3', '']\n" - '\n' - ' If *sep* is not specified or is "None", a different ' - 'splitting\n' - ' algorithm is applied: runs of consecutive whitespace ' - 'are regarded\n' - ' as a single separator, and the result will contain no ' - 'empty strings\n' - ' at the start or end if the string has leading or ' - 'trailing\n' - ' whitespace. Consequently, splitting an empty string or ' - 'a string\n' - ' consisting of just whitespace with a "None" separator ' - 'returns "[]".\n' - '\n' - ' For example:\n' - '\n' - " >>> '1 2 3'.split()\n" - " ['1', '2', '3']\n" - " >>> '1 2 3'.split(maxsplit=1)\n" - " ['1', '2 3']\n" - " >>> ' 1 2 3 '.split()\n" - " ['1', '2', '3']\n" - '\n' - 'str.splitlines([keepends])\n' - '\n' - ' Return a list of the lines in the string, breaking at ' - 'line\n' - ' boundaries. Line breaks are not included in the ' - 'resulting list\n' - ' unless *keepends* is given and true.\n' - '\n' - ' This method splits on the following line boundaries. ' - 'In\n' - ' particular, the boundaries are a superset of *universal ' - 'newlines*.\n' - '\n' - ' ' - '+-------------------------+-------------------------------+\n' - ' | Representation | ' - 'Description |\n' - ' ' - '+=========================+===============================+\n' - ' | "\\n" | Line ' - 'Feed |\n' - ' ' - '+-------------------------+-------------------------------+\n' - ' | "\\r" | Carriage ' - 'Return |\n' - ' ' - '+-------------------------+-------------------------------+\n' - ' | "\\r\\n" | Carriage Return + Line ' - 'Feed |\n' - ' ' - '+-------------------------+-------------------------------+\n' - ' | "\\v" or "\\x0b" | Line ' - 'Tabulation |\n' - ' ' - '+-------------------------+-------------------------------+\n' - ' | "\\f" or "\\x0c" | Form ' - 'Feed |\n' - ' ' - '+-------------------------+-------------------------------+\n' - ' | "\\x1c" | File ' - 'Separator |\n' - ' ' - '+-------------------------+-------------------------------+\n' - ' | "\\x1d" | Group ' - 'Separator |\n' - ' ' - '+-------------------------+-------------------------------+\n' - ' | "\\x1e" | Record ' - 'Separator |\n' - ' ' - '+-------------------------+-------------------------------+\n' - ' | "\\x85" | Next Line (C1 Control ' - 'Code) |\n' - ' ' - '+-------------------------+-------------------------------+\n' - ' | "\\u2028" | Line ' - 'Separator |\n' - ' ' - '+-------------------------+-------------------------------+\n' - ' | "\\u2029" | Paragraph ' - 'Separator |\n' - ' ' - '+-------------------------+-------------------------------+\n' - '\n' - ' Changed in version 3.2: "\\v" and "\\f" added to list ' - 'of line\n' - ' boundaries.\n' - '\n' - ' For example:\n' - '\n' - " >>> 'ab c\\n\\nde fg\\rkl\\r\\n'.splitlines()\n" - " ['ab c', '', 'de fg', 'kl']\n" - " >>> 'ab c\\n\\nde " - "fg\\rkl\\r\\n'.splitlines(keepends=True)\n" - " ['ab c\\n', '\\n', 'de fg\\r', 'kl\\r\\n']\n" - '\n' - ' Unlike "split()" when a delimiter string *sep* is ' - 'given, this\n' - ' method returns an empty list for the empty string, and ' - 'a terminal\n' - ' line break does not result in an extra line:\n' - '\n' - ' >>> "".splitlines()\n' - ' []\n' - ' >>> "One line\\n".splitlines()\n' - " ['One line']\n" - '\n' - ' For comparison, "split(\'\\n\')" gives:\n' - '\n' - " >>> ''.split('\\n')\n" - " ['']\n" - " >>> 'Two lines\\n'.split('\\n')\n" - " ['Two lines', '']\n" - '\n' - 'str.startswith(prefix[, start[, end]])\n' - '\n' - ' Return "True" if string starts with the *prefix*, ' - 'otherwise return\n' - ' "False". *prefix* can also be a tuple of prefixes to ' - 'look for.\n' - ' With optional *start*, test string beginning at that ' - 'position.\n' - ' With optional *end*, stop comparing string at that ' - 'position.\n' - '\n' - 'str.strip([chars])\n' - '\n' - ' Return a copy of the string with the leading and ' - 'trailing\n' - ' characters removed. The *chars* argument is a string ' - 'specifying the\n' - ' set of characters to be removed. If omitted or "None", ' - 'the *chars*\n' - ' argument defaults to removing whitespace. The *chars* ' - 'argument is\n' - ' not a prefix or suffix; rather, all combinations of its ' - 'values are\n' - ' stripped:\n' - '\n' - " >>> ' spacious '.strip()\n" - " 'spacious'\n" - " >>> 'www.example.com'.strip('cmowz.')\n" - " 'example'\n" - '\n' - ' The outermost leading and trailing *chars* argument ' - 'values are\n' - ' stripped from the string. Characters are removed from ' - 'the leading\n' - ' end until reaching a string character that is not ' - 'contained in the\n' - ' set of characters in *chars*. A similar action takes ' - 'place on the\n' - ' trailing end. For example:\n' - '\n' - " >>> comment_string = '#....... Section 3.2.1 Issue " - "#32 .......'\n" - " >>> comment_string.strip('.#! ')\n" - " 'Section 3.2.1 Issue #32'\n" - '\n' - 'str.swapcase()\n' - '\n' - ' Return a copy of the string with uppercase characters ' - 'converted to\n' - ' lowercase and vice versa. Note that it is not ' - 'necessarily true that\n' - ' "s.swapcase().swapcase() == s".\n' - '\n' - 'str.title()\n' - '\n' - ' Return a titlecased version of the string where words ' - 'start with an\n' - ' uppercase character and the remaining characters are ' - 'lowercase.\n' - '\n' - ' For example:\n' - '\n' - " >>> 'Hello world'.title()\n" - " 'Hello World'\n" - '\n' - ' The algorithm uses a simple language-independent ' - 'definition of a\n' - ' word as groups of consecutive letters. The definition ' - 'works in\n' - ' many contexts but it means that apostrophes in ' - 'contractions and\n' - ' possessives form word boundaries, which may not be the ' - 'desired\n' - ' result:\n' - '\n' - ' >>> "they\'re bill\'s friends from the UK".title()\n' - ' "They\'Re Bill\'S Friends From The Uk"\n' - '\n' - ' A workaround for apostrophes can be constructed using ' - 'regular\n' - ' expressions:\n' - '\n' - ' >>> import re\n' - ' >>> def titlecase(s):\n' - ' ... return re.sub(r"[A-Za-z]+(\'[A-Za-z]+)?",\n' - ' ... lambda mo: ' - 'mo.group(0)[0].upper() +\n' - ' ... ' - 'mo.group(0)[1:].lower(),\n' - ' ... s)\n' - ' ...\n' - ' >>> titlecase("they\'re bill\'s friends.")\n' - ' "They\'re Bill\'s Friends."\n' - '\n' - 'str.translate(table)\n' - '\n' - ' Return a copy of the string in which each character has ' - 'been mapped\n' - ' through the given translation table. The table must be ' - 'an object\n' - ' that implements indexing via "__getitem__()", typically ' - 'a *mapping*\n' - ' or *sequence*. When indexed by a Unicode ordinal (an ' - 'integer), the\n' - ' table object can do any of the following: return a ' - 'Unicode ordinal\n' - ' or a string, to map the character to one or more other ' - 'characters;\n' - ' return "None", to delete the character from the return ' - 'string; or\n' - ' raise a "LookupError" exception, to map the character ' - 'to itself.\n' - '\n' - ' You can use "str.maketrans()" to create a translation ' - 'map from\n' - ' character-to-character mappings in different formats.\n' - '\n' - ' See also the "codecs" module for a more flexible ' - 'approach to custom\n' - ' character mappings.\n' - '\n' - 'str.upper()\n' - '\n' - ' Return a copy of the string with all the cased ' - 'characters [4]\n' - ' converted to uppercase. Note that ' - '"s.upper().isupper()" might be\n' - ' "False" if "s" contains uncased characters or if the ' - 'Unicode\n' - ' category of the resulting character(s) is not “Lu” ' - '(Letter,\n' - ' uppercase), but e.g. “Lt” (Letter, titlecase).\n' - '\n' - ' The uppercasing algorithm used is described in section ' - '3.13 of the\n' - ' Unicode Standard.\n' - '\n' - 'str.zfill(width)\n' - '\n' - ' Return a copy of the string left filled with ASCII ' - '"\'0\'" digits to\n' - ' make a string of length *width*. A leading sign prefix\n' - ' ("\'+\'"/"\'-\'") is handled by inserting the padding ' - '*after* the sign\n' - ' character rather than before. The original string is ' - 'returned if\n' - ' *width* is less than or equal to "len(s)".\n' - '\n' - ' For example:\n' - '\n' - ' >>> "42".zfill(5)\n' - " '00042'\n" - ' >>> "-42".zfill(5)\n' - " '-0042'\n", - 'strings': 'String and Bytes literals\n' - '*************************\n' - '\n' - 'String literals are described by the following lexical ' - 'definitions:\n' - '\n' - ' stringliteral ::= [stringprefix](shortstring | longstring)\n' - ' stringprefix ::= "r" | "u" | "R" | "U" | "f" | "F"\n' - ' | "fr" | "Fr" | "fR" | "FR" | "rf" | "rF" | ' - '"Rf" | "RF"\n' - ' shortstring ::= "\'" shortstringitem* "\'" | \'"\' ' - 'shortstringitem* \'"\'\n' - ' longstring ::= "\'\'\'" longstringitem* "\'\'\'" | ' - '\'"""\' longstringitem* \'"""\'\n' - ' shortstringitem ::= shortstringchar | stringescapeseq\n' - ' longstringitem ::= longstringchar | stringescapeseq\n' - ' shortstringchar ::= \n' - ' longstringchar ::= \n' - ' stringescapeseq ::= "\\" \n' - '\n' - ' bytesliteral ::= bytesprefix(shortbytes | longbytes)\n' - ' bytesprefix ::= "b" | "B" | "br" | "Br" | "bR" | "BR" | ' - '"rb" | "rB" | "Rb" | "RB"\n' - ' shortbytes ::= "\'" shortbytesitem* "\'" | \'"\' ' - 'shortbytesitem* \'"\'\n' - ' longbytes ::= "\'\'\'" longbytesitem* "\'\'\'" | \'"""\' ' - 'longbytesitem* \'"""\'\n' - ' shortbytesitem ::= shortbyteschar | bytesescapeseq\n' - ' longbytesitem ::= longbyteschar | bytesescapeseq\n' - ' shortbyteschar ::= \n' - ' longbyteschar ::= \n' - ' bytesescapeseq ::= "\\" \n' - '\n' - 'One syntactic restriction not indicated by these productions is ' - 'that\n' - 'whitespace is not allowed between the "stringprefix" or ' - '"bytesprefix"\n' - 'and the rest of the literal. The source character set is defined ' - 'by\n' - 'the encoding declaration; it is UTF-8 if no encoding declaration ' - 'is\n' - 'given in the source file; see section Encoding declarations.\n' - '\n' - 'In plain English: Both types of literals can be enclosed in ' - 'matching\n' - 'single quotes ("\'") or double quotes ("""). They can also be ' - 'enclosed\n' - 'in matching groups of three single or double quotes (these are\n' - 'generally referred to as *triple-quoted strings*). The ' - 'backslash\n' - '("\\") character is used to escape characters that otherwise have ' - 'a\n' - 'special meaning, such as newline, backslash itself, or the quote\n' - 'character.\n' - '\n' - 'Bytes literals are always prefixed with "\'b\'" or "\'B\'"; they ' - 'produce\n' - 'an instance of the "bytes" type instead of the "str" type. They ' - 'may\n' - 'only contain ASCII characters; bytes with a numeric value of 128 ' - 'or\n' - 'greater must be expressed with escapes.\n' - '\n' - 'Both string and bytes literals may optionally be prefixed with a\n' - 'letter "\'r\'" or "\'R\'"; such strings are called *raw strings* ' - 'and treat\n' - 'backslashes as literal characters. As a result, in string ' - 'literals,\n' - '"\'\\U\'" and "\'\\u\'" escapes in raw strings are not treated ' - 'specially.\n' - 'Given that Python 2.x’s raw unicode literals behave differently ' - 'than\n' - 'Python 3.x’s the "\'ur\'" syntax is not supported.\n' - '\n' - 'New in version 3.3: The "\'rb\'" prefix of raw bytes literals has ' - 'been\n' - 'added as a synonym of "\'br\'".\n' - '\n' - 'New in version 3.3: Support for the unicode legacy literal\n' - '("u\'value\'") was reintroduced to simplify the maintenance of ' - 'dual\n' - 'Python 2.x and 3.x codebases. See **PEP 414** for more ' - 'information.\n' - '\n' - 'A string literal with "\'f\'" or "\'F\'" in its prefix is a ' - '*formatted\n' - 'string literal*; see Formatted string literals. The "\'f\'" may ' - 'be\n' - 'combined with "\'r\'", but not with "\'b\'" or "\'u\'", therefore ' - 'raw\n' - 'formatted strings are possible, but formatted bytes literals are ' - 'not.\n' - '\n' - 'In triple-quoted literals, unescaped newlines and quotes are ' - 'allowed\n' - '(and are retained), except that three unescaped quotes in a row\n' - 'terminate the literal. (A “quote” is the character used to open ' - 'the\n' - 'literal, i.e. either "\'" or """.)\n' - '\n' - 'Unless an "\'r\'" or "\'R\'" prefix is present, escape sequences ' - 'in string\n' - 'and bytes literals are interpreted according to rules similar to ' - 'those\n' - 'used by Standard C. The recognized escape sequences are:\n' - '\n' - '+-------------------+-----------------------------------+---------+\n' - '| Escape Sequence | Meaning | Notes ' - '|\n' - '+===================+===================================+=========+\n' - '| "\\newline" | Backslash and newline ignored ' - '| |\n' - '+-------------------+-----------------------------------+---------+\n' - '| "\\\\" | Backslash ("\\") ' - '| |\n' - '+-------------------+-----------------------------------+---------+\n' - '| "\\\'" | Single quote ("\'") ' - '| |\n' - '+-------------------+-----------------------------------+---------+\n' - '| "\\"" | Double quote (""") ' - '| |\n' - '+-------------------+-----------------------------------+---------+\n' - '| "\\a" | ASCII Bell (BEL) ' - '| |\n' - '+-------------------+-----------------------------------+---------+\n' - '| "\\b" | ASCII Backspace (BS) ' - '| |\n' - '+-------------------+-----------------------------------+---------+\n' - '| "\\f" | ASCII Formfeed (FF) ' - '| |\n' - '+-------------------+-----------------------------------+---------+\n' - '| "\\n" | ASCII Linefeed (LF) ' - '| |\n' - '+-------------------+-----------------------------------+---------+\n' - '| "\\r" | ASCII Carriage Return (CR) ' - '| |\n' - '+-------------------+-----------------------------------+---------+\n' - '| "\\t" | ASCII Horizontal Tab (TAB) ' - '| |\n' - '+-------------------+-----------------------------------+---------+\n' - '| "\\v" | ASCII Vertical Tab (VT) ' - '| |\n' - '+-------------------+-----------------------------------+---------+\n' - '| "\\ooo" | Character with octal value *ooo* | ' - '(1,3) |\n' - '+-------------------+-----------------------------------+---------+\n' - '| "\\xhh" | Character with hex value *hh* | ' - '(2,3) |\n' - '+-------------------+-----------------------------------+---------+\n' - '\n' - 'Escape sequences only recognized in string literals are:\n' - '\n' - '+-------------------+-----------------------------------+---------+\n' - '| Escape Sequence | Meaning | Notes ' - '|\n' - '+===================+===================================+=========+\n' - '| "\\N{name}" | Character named *name* in the | ' - '(4) |\n' - '| | Unicode database | ' - '|\n' - '+-------------------+-----------------------------------+---------+\n' - '| "\\uxxxx" | Character with 16-bit hex value | ' - '(5) |\n' - '| | *xxxx* | ' - '|\n' - '+-------------------+-----------------------------------+---------+\n' - '| "\\Uxxxxxxxx" | Character with 32-bit hex value | ' - '(6) |\n' - '| | *xxxxxxxx* | ' - '|\n' - '+-------------------+-----------------------------------+---------+\n' - '\n' - 'Notes:\n' - '\n' - '1. As in Standard C, up to three octal digits are accepted.\n' - '\n' - '2. Unlike in Standard C, exactly two hex digits are required.\n' - '\n' - '3. In a bytes literal, hexadecimal and octal escapes denote the\n' - ' byte with the given value. In a string literal, these escapes\n' - ' denote a Unicode character with the given value.\n' - '\n' - '4. Changed in version 3.3: Support for name aliases [1] has been\n' - ' added.\n' - '\n' - '5. Exactly four hex digits are required.\n' - '\n' - '6. Any Unicode character can be encoded this way. Exactly eight\n' - ' hex digits are required.\n' - '\n' - 'Unlike Standard C, all unrecognized escape sequences are left in ' - 'the\n' - 'string unchanged, i.e., *the backslash is left in the result*. ' - '(This\n' - 'behavior is useful when debugging: if an escape sequence is ' - 'mistyped,\n' - 'the resulting output is more easily recognized as broken.) It is ' - 'also\n' - 'important to note that the escape sequences only recognized in ' - 'string\n' - 'literals fall into the category of unrecognized escapes for ' - 'bytes\n' - 'literals.\n' - '\n' - ' Changed in version 3.6: Unrecognized escape sequences produce ' - 'a\n' - ' DeprecationWarning. In some future version of Python they ' - 'will be\n' - ' a SyntaxError.\n' - '\n' - 'Even in a raw literal, quotes can be escaped with a backslash, ' - 'but the\n' - 'backslash remains in the result; for example, "r"\\""" is a ' - 'valid\n' - 'string literal consisting of two characters: a backslash and a ' - 'double\n' - 'quote; "r"\\"" is not a valid string literal (even a raw string ' - 'cannot\n' - 'end in an odd number of backslashes). Specifically, *a raw ' - 'literal\n' - 'cannot end in a single backslash* (since the backslash would ' - 'escape\n' - 'the following quote character). Note also that a single ' - 'backslash\n' - 'followed by a newline is interpreted as those two characters as ' - 'part\n' - 'of the literal, *not* as a line continuation.\n', - 'subscriptions': 'Subscriptions\n' - '*************\n' - '\n' - 'A subscription selects an item of a sequence (string, tuple ' - 'or list)\n' - 'or mapping (dictionary) object:\n' - '\n' - ' subscription ::= primary "[" expression_list "]"\n' - '\n' - 'The primary must evaluate to an object that supports ' - 'subscription\n' - '(lists or dictionaries for example). User-defined objects ' - 'can support\n' - 'subscription by defining a "__getitem__()" method.\n' - '\n' - 'For built-in objects, there are two types of objects that ' - 'support\n' - 'subscription:\n' - '\n' - 'If the primary is a mapping, the expression list must ' - 'evaluate to an\n' - 'object whose value is one of the keys of the mapping, and ' - 'the\n' - 'subscription selects the value in the mapping that ' - 'corresponds to that\n' - 'key. (The expression list is a tuple except if it has ' - 'exactly one\n' - 'item.)\n' - '\n' - 'If the primary is a sequence, the expression list must ' - 'evaluate to an\n' - 'integer or a slice (as discussed in the following ' - 'section).\n' - '\n' - 'The formal syntax makes no special provision for negative ' - 'indices in\n' - 'sequences; however, built-in sequences all provide a ' - '"__getitem__()"\n' - 'method that interprets negative indices by adding the ' - 'length of the\n' - 'sequence to the index (so that "x[-1]" selects the last ' - 'item of "x").\n' - 'The resulting value must be a nonnegative integer less than ' - 'the number\n' - 'of items in the sequence, and the subscription selects the ' - 'item whose\n' - 'index is that value (counting from zero). Since the support ' - 'for\n' - 'negative indices and slicing occurs in the object’s ' - '"__getitem__()"\n' - 'method, subclasses overriding this method will need to ' - 'explicitly add\n' - 'that support.\n' - '\n' - 'A string’s items are characters. A character is not a ' - 'separate data\n' - 'type but a string of exactly one character.\n', - 'truth': 'Truth Value Testing\n' - '*******************\n' - '\n' - 'Any object can be tested for truth value, for use in an "if" or\n' - '"while" condition or as operand of the Boolean operations below.\n' - '\n' - 'By default, an object is considered true unless its class defines\n' - 'either a "__bool__()" method that returns "False" or a "__len__()"\n' - 'method that returns zero, when called with the object. [1] Here ' - 'are\n' - 'most of the built-in objects considered false:\n' - '\n' - '* constants defined to be false: "None" and "False".\n' - '\n' - '* zero of any numeric type: "0", "0.0", "0j", "Decimal(0)",\n' - ' "Fraction(0, 1)"\n' - '\n' - '* empty sequences and collections: "\'\'", "()", "[]", "{}", ' - '"set()",\n' - ' "range(0)"\n' - '\n' - 'Operations and built-in functions that have a Boolean result ' - 'always\n' - 'return "0" or "False" for false and "1" or "True" for true, unless\n' - 'otherwise stated. (Important exception: the Boolean operations ' - '"or"\n' - 'and "and" always return one of their operands.)\n', - 'try': 'The "try" statement\n' - '*******************\n' - '\n' - 'The "try" statement specifies exception handlers and/or cleanup code\n' - 'for a group of statements:\n' - '\n' - ' try_stmt ::= try1_stmt | try2_stmt\n' - ' try1_stmt ::= "try" ":" suite\n' - ' ("except" [expression ["as" identifier]] ":" ' - 'suite)+\n' - ' ["else" ":" suite]\n' - ' ["finally" ":" suite]\n' - ' try2_stmt ::= "try" ":" suite\n' - ' "finally" ":" suite\n' - '\n' - 'The "except" clause(s) specify one or more exception handlers. When ' - 'no\n' - 'exception occurs in the "try" clause, no exception handler is\n' - 'executed. When an exception occurs in the "try" suite, a search for ' - 'an\n' - 'exception handler is started. This search inspects the except ' - 'clauses\n' - 'in turn until one is found that matches the exception. An ' - 'expression-\n' - 'less except clause, if present, must be last; it matches any\n' - 'exception. For an except clause with an expression, that expression\n' - 'is evaluated, and the clause matches the exception if the resulting\n' - 'object is “compatible” with the exception. An object is compatible\n' - 'with an exception if it is the class or a base class of the ' - 'exception\n' - 'object or a tuple containing an item compatible with the exception.\n' - '\n' - 'If no except clause matches the exception, the search for an ' - 'exception\n' - 'handler continues in the surrounding code and on the invocation ' - 'stack.\n' - '[1]\n' - '\n' - 'If the evaluation of an expression in the header of an except clause\n' - 'raises an exception, the original search for a handler is canceled ' - 'and\n' - 'a search starts for the new exception in the surrounding code and on\n' - 'the call stack (it is treated as if the entire "try" statement ' - 'raised\n' - 'the exception).\n' - '\n' - 'When a matching except clause is found, the exception is assigned to\n' - 'the target specified after the "as" keyword in that except clause, ' - 'if\n' - 'present, and the except clause’s suite is executed. All except\n' - 'clauses must have an executable block. When the end of this block ' - 'is\n' - 'reached, execution continues normally after the entire try ' - 'statement.\n' - '(This means that if two nested handlers exist for the same ' - 'exception,\n' - 'and the exception occurs in the try clause of the inner handler, the\n' - 'outer handler will not handle the exception.)\n' - '\n' - 'When an exception has been assigned using "as target", it is cleared\n' - 'at the end of the except clause. This is as if\n' - '\n' - ' except E as N:\n' - ' foo\n' - '\n' - 'was translated to\n' - '\n' - ' except E as N:\n' - ' try:\n' - ' foo\n' - ' finally:\n' - ' del N\n' - '\n' - 'This means the exception must be assigned to a different name to be\n' - 'able to refer to it after the except clause. Exceptions are cleared\n' - 'because with the traceback attached to them, they form a reference\n' - 'cycle with the stack frame, keeping all locals in that frame alive\n' - 'until the next garbage collection occurs.\n' - '\n' - 'Before an except clause’s suite is executed, details about the\n' - 'exception are stored in the "sys" module and can be accessed via\n' - '"sys.exc_info()". "sys.exc_info()" returns a 3-tuple consisting of ' - 'the\n' - 'exception class, the exception instance and a traceback object (see\n' - 'section The standard type hierarchy) identifying the point in the\n' - 'program where the exception occurred. "sys.exc_info()" values are\n' - 'restored to their previous values (before the call) when returning\n' - 'from a function that handled an exception.\n' - '\n' - 'The optional "else" clause is executed if the control flow leaves ' - 'the\n' - '"try" suite, no exception was raised, and no "return", "continue", ' - 'or\n' - '"break" statement was executed. Exceptions in the "else" clause are\n' - 'not handled by the preceding "except" clauses.\n' - '\n' - 'If "finally" is present, it specifies a ‘cleanup’ handler. The ' - '"try"\n' - 'clause is executed, including any "except" and "else" clauses. If ' - 'an\n' - 'exception occurs in any of the clauses and is not handled, the\n' - 'exception is temporarily saved. The "finally" clause is executed. ' - 'If\n' - 'there is a saved exception it is re-raised at the end of the ' - '"finally"\n' - 'clause. If the "finally" clause raises another exception, the saved\n' - 'exception is set as the context of the new exception. If the ' - '"finally"\n' - 'clause executes a "return" or "break" statement, the saved exception\n' - 'is discarded:\n' - '\n' - ' >>> def f():\n' - ' ... try:\n' - ' ... 1/0\n' - ' ... finally:\n' - ' ... return 42\n' - ' ...\n' - ' >>> f()\n' - ' 42\n' - '\n' - 'The exception information is not available to the program during\n' - 'execution of the "finally" clause.\n' - '\n' - 'When a "return", "break" or "continue" statement is executed in the\n' - '"try" suite of a "try"…"finally" statement, the "finally" clause is\n' - 'also executed ‘on the way out.’ A "continue" statement is illegal in\n' - 'the "finally" clause. (The reason is a problem with the current\n' - 'implementation — this restriction may be lifted in the future).\n' - '\n' - 'The return value of a function is determined by the last "return"\n' - 'statement executed. Since the "finally" clause always executes, a\n' - '"return" statement executed in the "finally" clause will always be ' - 'the\n' - 'last one executed:\n' - '\n' - ' >>> def foo():\n' - ' ... try:\n' - " ... return 'try'\n" - ' ... finally:\n' - " ... return 'finally'\n" - ' ...\n' - ' >>> foo()\n' - " 'finally'\n" - '\n' - 'Additional information on exceptions can be found in section\n' - 'Exceptions, and information on using the "raise" statement to ' - 'generate\n' - 'exceptions may be found in section The raise statement.\n', - 'types': 'The standard type hierarchy\n' - '***************************\n' - '\n' - 'Below is a list of the types that are built into Python. ' - 'Extension\n' - 'modules (written in C, Java, or other languages, depending on the\n' - 'implementation) can define additional types. Future versions of\n' - 'Python may add types to the type hierarchy (e.g., rational ' - 'numbers,\n' - 'efficiently stored arrays of integers, etc.), although such ' - 'additions\n' - 'will often be provided via the standard library instead.\n' - '\n' - 'Some of the type descriptions below contain a paragraph listing\n' - '‘special attributes.’ These are attributes that provide access to ' - 'the\n' - 'implementation and are not intended for general use. Their ' - 'definition\n' - 'may change in the future.\n' - '\n' - 'None\n' - ' This type has a single value. There is a single object with ' - 'this\n' - ' value. This object is accessed through the built-in name "None". ' - 'It\n' - ' is used to signify the absence of a value in many situations, ' - 'e.g.,\n' - ' it is returned from functions that don’t explicitly return\n' - ' anything. Its truth value is false.\n' - '\n' - 'NotImplemented\n' - ' This type has a single value. There is a single object with ' - 'this\n' - ' value. This object is accessed through the built-in name\n' - ' "NotImplemented". Numeric methods and rich comparison methods\n' - ' should return this value if they do not implement the operation ' - 'for\n' - ' the operands provided. (The interpreter will then try the\n' - ' reflected operation, or some other fallback, depending on the\n' - ' operator.) Its truth value is true.\n' - '\n' - ' See Implementing the arithmetic operations for more details.\n' - '\n' - 'Ellipsis\n' - ' This type has a single value. There is a single object with ' - 'this\n' - ' value. This object is accessed through the literal "..." or the\n' - ' built-in name "Ellipsis". Its truth value is true.\n' - '\n' - '"numbers.Number"\n' - ' These are created by numeric literals and returned as results ' - 'by\n' - ' arithmetic operators and arithmetic built-in functions. ' - 'Numeric\n' - ' objects are immutable; once created their value never changes.\n' - ' Python numbers are of course strongly related to mathematical\n' - ' numbers, but subject to the limitations of numerical ' - 'representation\n' - ' in computers.\n' - '\n' - ' Python distinguishes between integers, floating point numbers, ' - 'and\n' - ' complex numbers:\n' - '\n' - ' "numbers.Integral"\n' - ' These represent elements from the mathematical set of ' - 'integers\n' - ' (positive and negative).\n' - '\n' - ' There are two types of integers:\n' - '\n' - ' Integers ("int")\n' - '\n' - ' These represent numbers in an unlimited range, subject to\n' - ' available (virtual) memory only. For the purpose of ' - 'shift\n' - ' and mask operations, a binary representation is assumed, ' - 'and\n' - ' negative numbers are represented in a variant of 2’s\n' - ' complement which gives the illusion of an infinite string ' - 'of\n' - ' sign bits extending to the left.\n' - '\n' - ' Booleans ("bool")\n' - ' These represent the truth values False and True. The two\n' - ' objects representing the values "False" and "True" are ' - 'the\n' - ' only Boolean objects. The Boolean type is a subtype of ' - 'the\n' - ' integer type, and Boolean values behave like the values 0 ' - 'and\n' - ' 1, respectively, in almost all contexts, the exception ' - 'being\n' - ' that when converted to a string, the strings ""False"" or\n' - ' ""True"" are returned, respectively.\n' - '\n' - ' The rules for integer representation are intended to give ' - 'the\n' - ' most meaningful interpretation of shift and mask operations\n' - ' involving negative integers.\n' - '\n' - ' "numbers.Real" ("float")\n' - ' These represent machine-level double precision floating ' - 'point\n' - ' numbers. You are at the mercy of the underlying machine\n' - ' architecture (and C or Java implementation) for the accepted\n' - ' range and handling of overflow. Python does not support ' - 'single-\n' - ' precision floating point numbers; the savings in processor ' - 'and\n' - ' memory usage that are usually the reason for using these are\n' - ' dwarfed by the overhead of using objects in Python, so there ' - 'is\n' - ' no reason to complicate the language with two kinds of ' - 'floating\n' - ' point numbers.\n' - '\n' - ' "numbers.Complex" ("complex")\n' - ' These represent complex numbers as a pair of machine-level\n' - ' double precision floating point numbers. The same caveats ' - 'apply\n' - ' as for floating point numbers. The real and imaginary parts ' - 'of a\n' - ' complex number "z" can be retrieved through the read-only\n' - ' attributes "z.real" and "z.imag".\n' - '\n' - 'Sequences\n' - ' These represent finite ordered sets indexed by non-negative\n' - ' numbers. The built-in function "len()" returns the number of ' - 'items\n' - ' of a sequence. When the length of a sequence is *n*, the index ' - 'set\n' - ' contains the numbers 0, 1, …, *n*-1. Item *i* of sequence *a* ' - 'is\n' - ' selected by "a[i]".\n' - '\n' - ' Sequences also support slicing: "a[i:j]" selects all items with\n' - ' index *k* such that *i* "<=" *k* "<" *j*. When used as an\n' - ' expression, a slice is a sequence of the same type. This ' - 'implies\n' - ' that the index set is renumbered so that it starts at 0.\n' - '\n' - ' Some sequences also support “extended slicing” with a third ' - '“step”\n' - ' parameter: "a[i:j:k]" selects all items of *a* with index *x* ' - 'where\n' - ' "x = i + n*k", *n* ">=" "0" and *i* "<=" *x* "<" *j*.\n' - '\n' - ' Sequences are distinguished according to their mutability:\n' - '\n' - ' Immutable sequences\n' - ' An object of an immutable sequence type cannot change once it ' - 'is\n' - ' created. (If the object contains references to other ' - 'objects,\n' - ' these other objects may be mutable and may be changed; ' - 'however,\n' - ' the collection of objects directly referenced by an ' - 'immutable\n' - ' object cannot change.)\n' - '\n' - ' The following types are immutable sequences:\n' - '\n' - ' Strings\n' - ' A string is a sequence of values that represent Unicode ' - 'code\n' - ' points. All the code points in the range "U+0000 - ' - 'U+10FFFF"\n' - ' can be represented in a string. Python doesn’t have a ' - '"char"\n' - ' type; instead, every code point in the string is ' - 'represented\n' - ' as a string object with length "1". The built-in ' - 'function\n' - ' "ord()" converts a code point from its string form to an\n' - ' integer in the range "0 - 10FFFF"; "chr()" converts an\n' - ' integer in the range "0 - 10FFFF" to the corresponding ' - 'length\n' - ' "1" string object. "str.encode()" can be used to convert ' - 'a\n' - ' "str" to "bytes" using the given text encoding, and\n' - ' "bytes.decode()" can be used to achieve the opposite.\n' - '\n' - ' Tuples\n' - ' The items of a tuple are arbitrary Python objects. Tuples ' - 'of\n' - ' two or more items are formed by comma-separated lists of\n' - ' expressions. A tuple of one item (a ‘singleton’) can be\n' - ' formed by affixing a comma to an expression (an expression ' - 'by\n' - ' itself does not create a tuple, since parentheses must be\n' - ' usable for grouping of expressions). An empty tuple can ' - 'be\n' - ' formed by an empty pair of parentheses.\n' - '\n' - ' Bytes\n' - ' A bytes object is an immutable array. The items are ' - '8-bit\n' - ' bytes, represented by integers in the range 0 <= x < 256.\n' - ' Bytes literals (like "b\'abc\'") and the built-in ' - '"bytes()"\n' - ' constructor can be used to create bytes objects. Also, ' - 'bytes\n' - ' objects can be decoded to strings via the "decode()" ' - 'method.\n' - '\n' - ' Mutable sequences\n' - ' Mutable sequences can be changed after they are created. ' - 'The\n' - ' subscription and slicing notations can be used as the target ' - 'of\n' - ' assignment and "del" (delete) statements.\n' - '\n' - ' There are currently two intrinsic mutable sequence types:\n' - '\n' - ' Lists\n' - ' The items of a list are arbitrary Python objects. Lists ' - 'are\n' - ' formed by placing a comma-separated list of expressions ' - 'in\n' - ' square brackets. (Note that there are no special cases ' - 'needed\n' - ' to form lists of length 0 or 1.)\n' - '\n' - ' Byte Arrays\n' - ' A bytearray object is a mutable array. They are created ' - 'by\n' - ' the built-in "bytearray()" constructor. Aside from being\n' - ' mutable (and hence unhashable), byte arrays otherwise ' - 'provide\n' - ' the same interface and functionality as immutable "bytes"\n' - ' objects.\n' - '\n' - ' The extension module "array" provides an additional example ' - 'of a\n' - ' mutable sequence type, as does the "collections" module.\n' - '\n' - 'Set types\n' - ' These represent unordered, finite sets of unique, immutable\n' - ' objects. As such, they cannot be indexed by any subscript. ' - 'However,\n' - ' they can be iterated over, and the built-in function "len()"\n' - ' returns the number of items in a set. Common uses for sets are ' - 'fast\n' - ' membership testing, removing duplicates from a sequence, and\n' - ' computing mathematical operations such as intersection, union,\n' - ' difference, and symmetric difference.\n' - '\n' - ' For set elements, the same immutability rules apply as for\n' - ' dictionary keys. Note that numeric types obey the normal rules ' - 'for\n' - ' numeric comparison: if two numbers compare equal (e.g., "1" and\n' - ' "1.0"), only one of them can be contained in a set.\n' - '\n' - ' There are currently two intrinsic set types:\n' - '\n' - ' Sets\n' - ' These represent a mutable set. They are created by the ' - 'built-in\n' - ' "set()" constructor and can be modified afterwards by ' - 'several\n' - ' methods, such as "add()".\n' - '\n' - ' Frozen sets\n' - ' These represent an immutable set. They are created by the\n' - ' built-in "frozenset()" constructor. As a frozenset is ' - 'immutable\n' - ' and *hashable*, it can be used again as an element of ' - 'another\n' - ' set, or as a dictionary key.\n' - '\n' - 'Mappings\n' - ' These represent finite sets of objects indexed by arbitrary ' - 'index\n' - ' sets. The subscript notation "a[k]" selects the item indexed by ' - '"k"\n' - ' from the mapping "a"; this can be used in expressions and as ' - 'the\n' - ' target of assignments or "del" statements. The built-in ' - 'function\n' - ' "len()" returns the number of items in a mapping.\n' - '\n' - ' There is currently a single intrinsic mapping type:\n' - '\n' - ' Dictionaries\n' - ' These represent finite sets of objects indexed by nearly\n' - ' arbitrary values. The only types of values not acceptable ' - 'as\n' - ' keys are values containing lists or dictionaries or other\n' - ' mutable types that are compared by value rather than by ' - 'object\n' - ' identity, the reason being that the efficient implementation ' - 'of\n' - ' dictionaries requires a key’s hash value to remain constant.\n' - ' Numeric types used for keys obey the normal rules for ' - 'numeric\n' - ' comparison: if two numbers compare equal (e.g., "1" and ' - '"1.0")\n' - ' then they can be used interchangeably to index the same\n' - ' dictionary entry.\n' - '\n' - ' Dictionaries are mutable; they can be created by the "{...}"\n' - ' notation (see section Dictionary displays).\n' - '\n' - ' The extension modules "dbm.ndbm" and "dbm.gnu" provide\n' - ' additional examples of mapping types, as does the ' - '"collections"\n' - ' module.\n' - '\n' - 'Callable types\n' - ' These are the types to which the function call operation (see\n' - ' section Calls) can be applied:\n' - '\n' - ' User-defined functions\n' - ' A user-defined function object is created by a function\n' - ' definition (see section Function definitions). It should be\n' - ' called with an argument list containing the same number of ' - 'items\n' - ' as the function’s formal parameter list.\n' - '\n' - ' Special attributes:\n' - '\n' - ' ' - '+---------------------------+---------------------------------+-------------+\n' - ' | Attribute | Meaning ' - '| |\n' - ' ' - '+===========================+=================================+=============+\n' - ' | "__doc__" | The function’s documentation ' - '| Writable |\n' - ' | | string, or "None" if ' - '| |\n' - ' | | unavailable; not inherited by ' - '| |\n' - ' | | subclasses ' - '| |\n' - ' ' - '+---------------------------+---------------------------------+-------------+\n' - ' | "__name__" | The function’s name ' - '| Writable |\n' - ' ' - '+---------------------------+---------------------------------+-------------+\n' - ' | "__qualname__" | The function’s *qualified name* ' - '| Writable |\n' - ' | | New in version 3.3. ' - '| |\n' - ' ' - '+---------------------------+---------------------------------+-------------+\n' - ' | "__module__" | The name of the module the ' - '| Writable |\n' - ' | | function was defined in, or ' - '| |\n' - ' | | "None" if unavailable. ' - '| |\n' - ' ' - '+---------------------------+---------------------------------+-------------+\n' - ' | "__defaults__" | A tuple containing default ' - '| Writable |\n' - ' | | argument values for those ' - '| |\n' - ' | | arguments that have defaults, ' - '| |\n' - ' | | or "None" if no arguments have ' - '| |\n' - ' | | a default value ' - '| |\n' - ' ' - '+---------------------------+---------------------------------+-------------+\n' - ' | "__code__" | The code object representing ' - '| Writable |\n' - ' | | the compiled function body. ' - '| |\n' - ' ' - '+---------------------------+---------------------------------+-------------+\n' - ' | "__globals__" | A reference to the dictionary ' - '| Read-only |\n' - ' | | that holds the function’s ' - '| |\n' - ' | | global variables — the global ' - '| |\n' - ' | | namespace of the module in ' - '| |\n' - ' | | which the function was defined. ' - '| |\n' - ' ' - '+---------------------------+---------------------------------+-------------+\n' - ' | "__dict__" | The namespace supporting ' - '| Writable |\n' - ' | | arbitrary function attributes. ' - '| |\n' - ' ' - '+---------------------------+---------------------------------+-------------+\n' - ' | "__closure__" | "None" or a tuple of cells that ' - '| Read-only |\n' - ' | | contain bindings for the ' - '| |\n' - ' | | function’s free variables. ' - '| |\n' - ' ' - '+---------------------------+---------------------------------+-------------+\n' - ' | "__annotations__" | A dict containing annotations ' - '| Writable |\n' - ' | | of parameters. The keys of the ' - '| |\n' - ' | | dict are the parameter names, ' - '| |\n' - ' | | and "\'return\'" for the ' - 'return | |\n' - ' | | annotation, if provided. ' - '| |\n' - ' ' - '+---------------------------+---------------------------------+-------------+\n' - ' | "__kwdefaults__" | A dict containing defaults for ' - '| Writable |\n' - ' | | keyword-only parameters. ' - '| |\n' - ' ' - '+---------------------------+---------------------------------+-------------+\n' - '\n' - ' Most of the attributes labelled “Writable” check the type of ' - 'the\n' - ' assigned value.\n' - '\n' - ' Function objects also support getting and setting arbitrary\n' - ' attributes, which can be used, for example, to attach ' - 'metadata\n' - ' to functions. Regular attribute dot-notation is used to get ' - 'and\n' - ' set such attributes. *Note that the current implementation ' - 'only\n' - ' supports function attributes on user-defined functions. ' - 'Function\n' - ' attributes on built-in functions may be supported in the\n' - ' future.*\n' - '\n' - ' Additional information about a function’s definition can be\n' - ' retrieved from its code object; see the description of ' - 'internal\n' - ' types below.\n' - '\n' - ' Instance methods\n' - ' An instance method object combines a class, a class instance ' - 'and\n' - ' any callable object (normally a user-defined function).\n' - '\n' - ' Special read-only attributes: "__self__" is the class ' - 'instance\n' - ' object, "__func__" is the function object; "__doc__" is the\n' - ' method’s documentation (same as "__func__.__doc__"); ' - '"__name__"\n' - ' is the method name (same as "__func__.__name__"); ' - '"__module__"\n' - ' is the name of the module the method was defined in, or ' - '"None"\n' - ' if unavailable.\n' - '\n' - ' Methods also support accessing (but not setting) the ' - 'arbitrary\n' - ' function attributes on the underlying function object.\n' - '\n' - ' User-defined method objects may be created when getting an\n' - ' attribute of a class (perhaps via an instance of that class), ' - 'if\n' - ' that attribute is a user-defined function object or a class\n' - ' method object.\n' - '\n' - ' When an instance method object is created by retrieving a ' - 'user-\n' - ' defined function object from a class via one of its ' - 'instances,\n' - ' its "__self__" attribute is the instance, and the method ' - 'object\n' - ' is said to be bound. The new method’s "__func__" attribute ' - 'is\n' - ' the original function object.\n' - '\n' - ' When a user-defined method object is created by retrieving\n' - ' another method object from a class or instance, the behaviour ' - 'is\n' - ' the same as for a function object, except that the ' - '"__func__"\n' - ' attribute of the new instance is not the original method ' - 'object\n' - ' but its "__func__" attribute.\n' - '\n' - ' When an instance method object is created by retrieving a ' - 'class\n' - ' method object from a class or instance, its "__self__" ' - 'attribute\n' - ' is the class itself, and its "__func__" attribute is the\n' - ' function object underlying the class method.\n' - '\n' - ' When an instance method object is called, the underlying\n' - ' function ("__func__") is called, inserting the class ' - 'instance\n' - ' ("__self__") in front of the argument list. For instance, ' - 'when\n' - ' "C" is a class which contains a definition for a function ' - '"f()",\n' - ' and "x" is an instance of "C", calling "x.f(1)" is equivalent ' - 'to\n' - ' calling "C.f(x, 1)".\n' - '\n' - ' When an instance method object is derived from a class ' - 'method\n' - ' object, the “class instance” stored in "__self__" will ' - 'actually\n' - ' be the class itself, so that calling either "x.f(1)" or ' - '"C.f(1)"\n' - ' is equivalent to calling "f(C,1)" where "f" is the ' - 'underlying\n' - ' function.\n' - '\n' - ' Note that the transformation from function object to ' - 'instance\n' - ' method object happens each time the attribute is retrieved ' - 'from\n' - ' the instance. In some cases, a fruitful optimization is to\n' - ' assign the attribute to a local variable and call that local\n' - ' variable. Also notice that this transformation only happens ' - 'for\n' - ' user-defined functions; other callable objects (and all non-\n' - ' callable objects) are retrieved without transformation. It ' - 'is\n' - ' also important to note that user-defined functions which are\n' - ' attributes of a class instance are not converted to bound\n' - ' methods; this *only* happens when the function is an ' - 'attribute\n' - ' of the class.\n' - '\n' - ' Generator functions\n' - ' A function or method which uses the "yield" statement (see\n' - ' section The yield statement) is called a *generator ' - 'function*.\n' - ' Such a function, when called, always returns an iterator ' - 'object\n' - ' which can be used to execute the body of the function: ' - 'calling\n' - ' the iterator’s "iterator.__next__()" method will cause the\n' - ' function to execute until it provides a value using the ' - '"yield"\n' - ' statement. When the function executes a "return" statement ' - 'or\n' - ' falls off the end, a "StopIteration" exception is raised and ' - 'the\n' - ' iterator will have reached the end of the set of values to ' - 'be\n' - ' returned.\n' - '\n' - ' Coroutine functions\n' - ' A function or method which is defined using "async def" is\n' - ' called a *coroutine function*. Such a function, when ' - 'called,\n' - ' returns a *coroutine* object. It may contain "await"\n' - ' expressions, as well as "async with" and "async for" ' - 'statements.\n' - ' See also the Coroutine Objects section.\n' - '\n' - ' Asynchronous generator functions\n' - ' A function or method which is defined using "async def" and\n' - ' which uses the "yield" statement is called a *asynchronous\n' - ' generator function*. Such a function, when called, returns ' - 'an\n' - ' asynchronous iterator object which can be used in an "async ' - 'for"\n' - ' statement to execute the body of the function.\n' - '\n' - ' Calling the asynchronous iterator’s "aiterator.__anext__()"\n' - ' method will return an *awaitable* which when awaited will\n' - ' execute until it provides a value using the "yield" ' - 'expression.\n' - ' When the function executes an empty "return" statement or ' - 'falls\n' - ' off the end, a "StopAsyncIteration" exception is raised and ' - 'the\n' - ' asynchronous iterator will have reached the end of the set ' - 'of\n' - ' values to be yielded.\n' - '\n' - ' Built-in functions\n' - ' A built-in function object is a wrapper around a C function.\n' - ' Examples of built-in functions are "len()" and "math.sin()"\n' - ' ("math" is a standard built-in module). The number and type ' - 'of\n' - ' the arguments are determined by the C function. Special ' - 'read-\n' - ' only attributes: "__doc__" is the function’s documentation\n' - ' string, or "None" if unavailable; "__name__" is the ' - 'function’s\n' - ' name; "__self__" is set to "None" (but see the next item);\n' - ' "__module__" is the name of the module the function was ' - 'defined\n' - ' in or "None" if unavailable.\n' - '\n' - ' Built-in methods\n' - ' This is really a different disguise of a built-in function, ' - 'this\n' - ' time containing an object passed to the C function as an\n' - ' implicit extra argument. An example of a built-in method is\n' - ' "alist.append()", assuming *alist* is a list object. In this\n' - ' case, the special read-only attribute "__self__" is set to ' - 'the\n' - ' object denoted by *alist*.\n' - '\n' - ' Classes\n' - ' Classes are callable. These objects normally act as ' - 'factories\n' - ' for new instances of themselves, but variations are possible ' - 'for\n' - ' class types that override "__new__()". The arguments of the\n' - ' call are passed to "__new__()" and, in the typical case, to\n' - ' "__init__()" to initialize the new instance.\n' - '\n' - ' Class Instances\n' - ' Instances of arbitrary classes can be made callable by ' - 'defining\n' - ' a "__call__()" method in their class.\n' - '\n' - 'Modules\n' - ' Modules are a basic organizational unit of Python code, and are\n' - ' created by the import system as invoked either by the "import"\n' - ' statement (see "import"), or by calling functions such as\n' - ' "importlib.import_module()" and built-in "__import__()". A ' - 'module\n' - ' object has a namespace implemented by a dictionary object (this ' - 'is\n' - ' the dictionary referenced by the "__globals__" attribute of\n' - ' functions defined in the module). Attribute references are\n' - ' translated to lookups in this dictionary, e.g., "m.x" is ' - 'equivalent\n' - ' to "m.__dict__["x"]". A module object does not contain the code\n' - ' object used to initialize the module (since it isn’t needed ' - 'once\n' - ' the initialization is done).\n' - '\n' - ' Attribute assignment updates the module’s namespace dictionary,\n' - ' e.g., "m.x = 1" is equivalent to "m.__dict__["x"] = 1".\n' - '\n' - ' Predefined (writable) attributes: "__name__" is the module’s ' - 'name;\n' - ' "__doc__" is the module’s documentation string, or "None" if\n' - ' unavailable; "__annotations__" (optional) is a dictionary\n' - ' containing *variable annotations* collected during module body\n' - ' execution; "__file__" is the pathname of the file from which ' - 'the\n' - ' module was loaded, if it was loaded from a file. The "__file__"\n' - ' attribute may be missing for certain types of modules, such as ' - 'C\n' - ' modules that are statically linked into the interpreter; for\n' - ' extension modules loaded dynamically from a shared library, it ' - 'is\n' - ' the pathname of the shared library file.\n' - '\n' - ' Special read-only attribute: "__dict__" is the module’s ' - 'namespace\n' - ' as a dictionary object.\n' - '\n' - ' **CPython implementation detail:** Because of the way CPython\n' - ' clears module dictionaries, the module dictionary will be ' - 'cleared\n' - ' when the module falls out of scope even if the dictionary still ' - 'has\n' - ' live references. To avoid this, copy the dictionary or keep ' - 'the\n' - ' module around while using its dictionary directly.\n' - '\n' - 'Custom classes\n' - ' Custom class types are typically created by class definitions ' - '(see\n' - ' section Class definitions). A class has a namespace implemented ' - 'by\n' - ' a dictionary object. Class attribute references are translated ' - 'to\n' - ' lookups in this dictionary, e.g., "C.x" is translated to\n' - ' "C.__dict__["x"]" (although there are a number of hooks which ' - 'allow\n' - ' for other means of locating attributes). When the attribute name ' - 'is\n' - ' not found there, the attribute search continues in the base\n' - ' classes. This search of the base classes uses the C3 method\n' - ' resolution order which behaves correctly even in the presence ' - 'of\n' - ' ‘diamond’ inheritance structures where there are multiple\n' - ' inheritance paths leading back to a common ancestor. Additional\n' - ' details on the C3 MRO used by Python can be found in the\n' - ' documentation accompanying the 2.3 release at\n' - ' https://www.python.org/download/releases/2.3/mro/.\n' - '\n' - ' When a class attribute reference (for class "C", say) would ' - 'yield a\n' - ' class method object, it is transformed into an instance method\n' - ' object whose "__self__" attribute is "C". When it would yield ' - 'a\n' - ' static method object, it is transformed into the object wrapped ' - 'by\n' - ' the static method object. See section Implementing Descriptors ' - 'for\n' - ' another way in which attributes retrieved from a class may ' - 'differ\n' - ' from those actually contained in its "__dict__".\n' - '\n' - ' Class attribute assignments update the class’s dictionary, ' - 'never\n' - ' the dictionary of a base class.\n' - '\n' - ' A class object can be called (see above) to yield a class ' - 'instance\n' - ' (see below).\n' - '\n' - ' Special attributes: "__name__" is the class name; "__module__" ' - 'is\n' - ' the module name in which the class was defined; "__dict__" is ' - 'the\n' - ' dictionary containing the class’s namespace; "__bases__" is a ' - 'tuple\n' - ' containing the base classes, in the order of their occurrence ' - 'in\n' - ' the base class list; "__doc__" is the class’s documentation ' - 'string,\n' - ' or "None" if undefined; "__annotations__" (optional) is a\n' - ' dictionary containing *variable annotations* collected during ' - 'class\n' - ' body execution.\n' - '\n' - 'Class instances\n' - ' A class instance is created by calling a class object (see ' - 'above).\n' - ' A class instance has a namespace implemented as a dictionary ' - 'which\n' - ' is the first place in which attribute references are searched.\n' - ' When an attribute is not found there, and the instance’s class ' - 'has\n' - ' an attribute by that name, the search continues with the class\n' - ' attributes. If a class attribute is found that is a ' - 'user-defined\n' - ' function object, it is transformed into an instance method ' - 'object\n' - ' whose "__self__" attribute is the instance. Static method and\n' - ' class method objects are also transformed; see above under\n' - ' “Classes”. See section Implementing Descriptors for another way ' - 'in\n' - ' which attributes of a class retrieved via its instances may ' - 'differ\n' - ' from the objects actually stored in the class’s "__dict__". If ' - 'no\n' - ' class attribute is found, and the object’s class has a\n' - ' "__getattr__()" method, that is called to satisfy the lookup.\n' - '\n' - ' Attribute assignments and deletions update the instance’s\n' - ' dictionary, never a class’s dictionary. If the class has a\n' - ' "__setattr__()" or "__delattr__()" method, this is called ' - 'instead\n' - ' of updating the instance dictionary directly.\n' - '\n' - ' Class instances can pretend to be numbers, sequences, or ' - 'mappings\n' - ' if they have methods with certain special names. See section\n' - ' Special method names.\n' - '\n' - ' Special attributes: "__dict__" is the attribute dictionary;\n' - ' "__class__" is the instance’s class.\n' - '\n' - 'I/O objects (also known as file objects)\n' - ' A *file object* represents an open file. Various shortcuts are\n' - ' available to create file objects: the "open()" built-in ' - 'function,\n' - ' and also "os.popen()", "os.fdopen()", and the "makefile()" ' - 'method\n' - ' of socket objects (and perhaps by other functions or methods\n' - ' provided by extension modules).\n' - '\n' - ' The objects "sys.stdin", "sys.stdout" and "sys.stderr" are\n' - ' initialized to file objects corresponding to the interpreter’s\n' - ' standard input, output and error streams; they are all open in ' - 'text\n' - ' mode and therefore follow the interface defined by the\n' - ' "io.TextIOBase" abstract class.\n' - '\n' - 'Internal types\n' - ' A few types used internally by the interpreter are exposed to ' - 'the\n' - ' user. Their definitions may change with future versions of the\n' - ' interpreter, but they are mentioned here for completeness.\n' - '\n' - ' Code objects\n' - ' Code objects represent *byte-compiled* executable Python ' - 'code,\n' - ' or *bytecode*. The difference between a code object and a\n' - ' function object is that the function object contains an ' - 'explicit\n' - ' reference to the function’s globals (the module in which it ' - 'was\n' - ' defined), while a code object contains no context; also the\n' - ' default argument values are stored in the function object, ' - 'not\n' - ' in the code object (because they represent values calculated ' - 'at\n' - ' run-time). Unlike function objects, code objects are ' - 'immutable\n' - ' and contain no references (directly or indirectly) to ' - 'mutable\n' - ' objects.\n' - '\n' - ' Special read-only attributes: "co_name" gives the function ' - 'name;\n' - ' "co_argcount" is the number of positional arguments ' - '(including\n' - ' arguments with default values); "co_nlocals" is the number ' - 'of\n' - ' local variables used by the function (including arguments);\n' - ' "co_varnames" is a tuple containing the names of the local\n' - ' variables (starting with the argument names); "co_cellvars" ' - 'is a\n' - ' tuple containing the names of local variables that are\n' - ' referenced by nested functions; "co_freevars" is a tuple\n' - ' containing the names of free variables; "co_code" is a ' - 'string\n' - ' representing the sequence of bytecode instructions; ' - '"co_consts"\n' - ' is a tuple containing the literals used by the bytecode;\n' - ' "co_names" is a tuple containing the names used by the ' - 'bytecode;\n' - ' "co_filename" is the filename from which the code was ' - 'compiled;\n' - ' "co_firstlineno" is the first line number of the function;\n' - ' "co_lnotab" is a string encoding the mapping from bytecode\n' - ' offsets to line numbers (for details see the source code of ' - 'the\n' - ' interpreter); "co_stacksize" is the required stack size\n' - ' (including local variables); "co_flags" is an integer ' - 'encoding a\n' - ' number of flags for the interpreter.\n' - '\n' - ' The following flag bits are defined for "co_flags": bit ' - '"0x04"\n' - ' is set if the function uses the "*arguments" syntax to accept ' - 'an\n' - ' arbitrary number of positional arguments; bit "0x08" is set ' - 'if\n' - ' the function uses the "**keywords" syntax to accept ' - 'arbitrary\n' - ' keyword arguments; bit "0x20" is set if the function is a\n' - ' generator.\n' - '\n' - ' Future feature declarations ("from __future__ import ' - 'division")\n' - ' also use bits in "co_flags" to indicate whether a code ' - 'object\n' - ' was compiled with a particular feature enabled: bit "0x2000" ' - 'is\n' - ' set if the function was compiled with future division ' - 'enabled;\n' - ' bits "0x10" and "0x1000" were used in earlier versions of\n' - ' Python.\n' - '\n' - ' Other bits in "co_flags" are reserved for internal use.\n' - '\n' - ' If a code object represents a function, the first item in\n' - ' "co_consts" is the documentation string of the function, or\n' - ' "None" if undefined.\n' - '\n' - ' Frame objects\n' - ' Frame objects represent execution frames. They may occur in\n' - ' traceback objects (see below).\n' - '\n' - ' Special read-only attributes: "f_back" is to the previous ' - 'stack\n' - ' frame (towards the caller), or "None" if this is the bottom\n' - ' stack frame; "f_code" is the code object being executed in ' - 'this\n' - ' frame; "f_locals" is the dictionary used to look up local\n' - ' variables; "f_globals" is used for global variables;\n' - ' "f_builtins" is used for built-in (intrinsic) names; ' - '"f_lasti"\n' - ' gives the precise instruction (this is an index into the\n' - ' bytecode string of the code object).\n' - '\n' - ' Special writable attributes: "f_trace", if not "None", is a\n' - ' function called at the start of each source code line (this ' - 'is\n' - ' used by the debugger); "f_lineno" is the current line number ' - 'of\n' - ' the frame — writing to this from within a trace function ' - 'jumps\n' - ' to the given line (only for the bottom-most frame). A ' - 'debugger\n' - ' can implement a Jump command (aka Set Next Statement) by ' - 'writing\n' - ' to f_lineno.\n' - '\n' - ' Frame objects support one method:\n' - '\n' - ' frame.clear()\n' - '\n' - ' This method clears all references to local variables held ' - 'by\n' - ' the frame. Also, if the frame belonged to a generator, ' - 'the\n' - ' generator is finalized. This helps break reference ' - 'cycles\n' - ' involving frame objects (for example when catching an\n' - ' exception and storing its traceback for later use).\n' - '\n' - ' "RuntimeError" is raised if the frame is currently ' - 'executing.\n' - '\n' - ' New in version 3.4.\n' - '\n' - ' Traceback objects\n' - ' Traceback objects represent a stack trace of an exception. ' - 'A\n' - ' traceback object is created when an exception occurs. When ' - 'the\n' - ' search for an exception handler unwinds the execution stack, ' - 'at\n' - ' each unwound level a traceback object is inserted in front ' - 'of\n' - ' the current traceback. When an exception handler is ' - 'entered,\n' - ' the stack trace is made available to the program. (See ' - 'section\n' - ' The try statement.) It is accessible as the third item of ' - 'the\n' - ' tuple returned by "sys.exc_info()". When the program contains ' - 'no\n' - ' suitable handler, the stack trace is written (nicely ' - 'formatted)\n' - ' to the standard error stream; if the interpreter is ' - 'interactive,\n' - ' it is also made available to the user as ' - '"sys.last_traceback".\n' - '\n' - ' Special read-only attributes: "tb_next" is the next level in ' - 'the\n' - ' stack trace (towards the frame where the exception occurred), ' - 'or\n' - ' "None" if there is no next level; "tb_frame" points to the\n' - ' execution frame of the current level; "tb_lineno" gives the ' - 'line\n' - ' number where the exception occurred; "tb_lasti" indicates ' - 'the\n' - ' precise instruction. The line number and last instruction ' - 'in\n' - ' the traceback may differ from the line number of its frame\n' - ' object if the exception occurred in a "try" statement with ' - 'no\n' - ' matching except clause or with a finally clause.\n' - '\n' - ' Slice objects\n' - ' Slice objects are used to represent slices for ' - '"__getitem__()"\n' - ' methods. They are also created by the built-in "slice()"\n' - ' function.\n' - '\n' - ' Special read-only attributes: "start" is the lower bound; ' - '"stop"\n' - ' is the upper bound; "step" is the step value; each is "None" ' - 'if\n' - ' omitted. These attributes can have any type.\n' - '\n' - ' Slice objects support one method:\n' - '\n' - ' slice.indices(self, length)\n' - '\n' - ' This method takes a single integer argument *length* and\n' - ' computes information about the slice that the slice ' - 'object\n' - ' would describe if applied to a sequence of *length* ' - 'items.\n' - ' It returns a tuple of three integers; respectively these ' - 'are\n' - ' the *start* and *stop* indices and the *step* or stride\n' - ' length of the slice. Missing or out-of-bounds indices are\n' - ' handled in a manner consistent with regular slices.\n' - '\n' - ' Static method objects\n' - ' Static method objects provide a way of defeating the\n' - ' transformation of function objects to method objects ' - 'described\n' - ' above. A static method object is a wrapper around any other\n' - ' object, usually a user-defined method object. When a static\n' - ' method object is retrieved from a class or a class instance, ' - 'the\n' - ' object actually returned is the wrapped object, which is not\n' - ' subject to any further transformation. Static method objects ' - 'are\n' - ' not themselves callable, although the objects they wrap ' - 'usually\n' - ' are. Static method objects are created by the built-in\n' - ' "staticmethod()" constructor.\n' - '\n' - ' Class method objects\n' - ' A class method object, like a static method object, is a ' - 'wrapper\n' - ' around another object that alters the way in which that ' - 'object\n' - ' is retrieved from classes and class instances. The behaviour ' - 'of\n' - ' class method objects upon such retrieval is described above,\n' - ' under “User-defined methods”. Class method objects are ' - 'created\n' - ' by the built-in "classmethod()" constructor.\n', - 'typesfunctions': 'Functions\n' - '*********\n' - '\n' - 'Function objects are created by function definitions. The ' - 'only\n' - 'operation on a function object is to call it: ' - '"func(argument-list)".\n' - '\n' - 'There are really two flavors of function objects: built-in ' - 'functions\n' - 'and user-defined functions. Both support the same ' - 'operation (to call\n' - 'the function), but the implementation is different, hence ' - 'the\n' - 'different object types.\n' - '\n' - 'See Function definitions for more information.\n', - 'typesmapping': 'Mapping Types — "dict"\n' - '**********************\n' - '\n' - 'A *mapping* object maps *hashable* values to arbitrary ' - 'objects.\n' - 'Mappings are mutable objects. There is currently only one ' - 'standard\n' - 'mapping type, the *dictionary*. (For other containers see ' - 'the built-\n' - 'in "list", "set", and "tuple" classes, and the "collections" ' - 'module.)\n' - '\n' - 'A dictionary’s keys are *almost* arbitrary values. Values ' - 'that are\n' - 'not *hashable*, that is, values containing lists, ' - 'dictionaries or\n' - 'other mutable types (that are compared by value rather than ' - 'by object\n' - 'identity) may not be used as keys. Numeric types used for ' - 'keys obey\n' - 'the normal rules for numeric comparison: if two numbers ' - 'compare equal\n' - '(such as "1" and "1.0") then they can be used ' - 'interchangeably to index\n' - 'the same dictionary entry. (Note however, that since ' - 'computers store\n' - 'floating-point numbers as approximations it is usually ' - 'unwise to use\n' - 'them as dictionary keys.)\n' - '\n' - 'Dictionaries can be created by placing a comma-separated ' - 'list of "key:\n' - 'value" pairs within braces, for example: "{\'jack\': 4098, ' - "'sjoerd':\n" - '4127}" or "{4098: \'jack\', 4127: \'sjoerd\'}", or by the ' - '"dict"\n' - 'constructor.\n' - '\n' - 'class dict(**kwarg)\n' - 'class dict(mapping, **kwarg)\n' - 'class dict(iterable, **kwarg)\n' - '\n' - ' Return a new dictionary initialized from an optional ' - 'positional\n' - ' argument and a possibly empty set of keyword arguments.\n' - '\n' - ' If no positional argument is given, an empty dictionary ' - 'is created.\n' - ' If a positional argument is given and it is a mapping ' - 'object, a\n' - ' dictionary is created with the same key-value pairs as ' - 'the mapping\n' - ' object. Otherwise, the positional argument must be an ' - '*iterable*\n' - ' object. Each item in the iterable must itself be an ' - 'iterable with\n' - ' exactly two objects. The first object of each item ' - 'becomes a key\n' - ' in the new dictionary, and the second object the ' - 'corresponding\n' - ' value. If a key occurs more than once, the last value ' - 'for that key\n' - ' becomes the corresponding value in the new dictionary.\n' - '\n' - ' If keyword arguments are given, the keyword arguments and ' - 'their\n' - ' values are added to the dictionary created from the ' - 'positional\n' - ' argument. If a key being added is already present, the ' - 'value from\n' - ' the keyword argument replaces the value from the ' - 'positional\n' - ' argument.\n' - '\n' - ' To illustrate, the following examples all return a ' - 'dictionary equal\n' - ' to "{"one": 1, "two": 2, "three": 3}":\n' - '\n' - ' >>> a = dict(one=1, two=2, three=3)\n' - " >>> b = {'one': 1, 'two': 2, 'three': 3}\n" - " >>> c = dict(zip(['one', 'two', 'three'], [1, 2, 3]))\n" - " >>> d = dict([('two', 2), ('one', 1), ('three', 3)])\n" - " >>> e = dict({'three': 3, 'one': 1, 'two': 2})\n" - ' >>> a == b == c == d == e\n' - ' True\n' - '\n' - ' Providing keyword arguments as in the first example only ' - 'works for\n' - ' keys that are valid Python identifiers. Otherwise, any ' - 'valid keys\n' - ' can be used.\n' - '\n' - ' These are the operations that dictionaries support (and ' - 'therefore,\n' - ' custom mapping types should support too):\n' - '\n' - ' len(d)\n' - '\n' - ' Return the number of items in the dictionary *d*.\n' - '\n' - ' d[key]\n' - '\n' - ' Return the item of *d* with key *key*. Raises a ' - '"KeyError" if\n' - ' *key* is not in the map.\n' - '\n' - ' If a subclass of dict defines a method "__missing__()" ' - 'and *key*\n' - ' is not present, the "d[key]" operation calls that ' - 'method with\n' - ' the key *key* as argument. The "d[key]" operation ' - 'then returns\n' - ' or raises whatever is returned or raised by the\n' - ' "__missing__(key)" call. No other operations or ' - 'methods invoke\n' - ' "__missing__()". If "__missing__()" is not defined, ' - '"KeyError"\n' - ' is raised. "__missing__()" must be a method; it cannot ' - 'be an\n' - ' instance variable:\n' - '\n' - ' >>> class Counter(dict):\n' - ' ... def __missing__(self, key):\n' - ' ... return 0\n' - ' >>> c = Counter()\n' - " >>> c['red']\n" - ' 0\n' - " >>> c['red'] += 1\n" - " >>> c['red']\n" - ' 1\n' - '\n' - ' The example above shows part of the implementation of\n' - ' "collections.Counter". A different "__missing__" ' - 'method is used\n' - ' by "collections.defaultdict".\n' - '\n' - ' d[key] = value\n' - '\n' - ' Set "d[key]" to *value*.\n' - '\n' - ' del d[key]\n' - '\n' - ' Remove "d[key]" from *d*. Raises a "KeyError" if ' - '*key* is not\n' - ' in the map.\n' - '\n' - ' key in d\n' - '\n' - ' Return "True" if *d* has a key *key*, else "False".\n' - '\n' - ' key not in d\n' - '\n' - ' Equivalent to "not key in d".\n' - '\n' - ' iter(d)\n' - '\n' - ' Return an iterator over the keys of the dictionary. ' - 'This is a\n' - ' shortcut for "iter(d.keys())".\n' - '\n' - ' clear()\n' - '\n' - ' Remove all items from the dictionary.\n' - '\n' - ' copy()\n' - '\n' - ' Return a shallow copy of the dictionary.\n' - '\n' - ' classmethod fromkeys(seq[, value])\n' - '\n' - ' Create a new dictionary with keys from *seq* and ' - 'values set to\n' - ' *value*.\n' - '\n' - ' "fromkeys()" is a class method that returns a new ' - 'dictionary.\n' - ' *value* defaults to "None".\n' - '\n' - ' get(key[, default])\n' - '\n' - ' Return the value for *key* if *key* is in the ' - 'dictionary, else\n' - ' *default*. If *default* is not given, it defaults to ' - '"None", so\n' - ' that this method never raises a "KeyError".\n' - '\n' - ' items()\n' - '\n' - ' Return a new view of the dictionary’s items ("(key, ' - 'value)"\n' - ' pairs). See the documentation of view objects.\n' - '\n' - ' keys()\n' - '\n' - ' Return a new view of the dictionary’s keys. See the\n' - ' documentation of view objects.\n' - '\n' - ' pop(key[, default])\n' - '\n' - ' If *key* is in the dictionary, remove it and return ' - 'its value,\n' - ' else return *default*. If *default* is not given and ' - '*key* is\n' - ' not in the dictionary, a "KeyError" is raised.\n' - '\n' - ' popitem()\n' - '\n' - ' Remove and return an arbitrary "(key, value)" pair ' - 'from the\n' - ' dictionary.\n' - '\n' - ' "popitem()" is useful to destructively iterate over a\n' - ' dictionary, as often used in set algorithms. If the ' - 'dictionary\n' - ' is empty, calling "popitem()" raises a "KeyError".\n' - '\n' - ' setdefault(key[, default])\n' - '\n' - ' If *key* is in the dictionary, return its value. If ' - 'not, insert\n' - ' *key* with a value of *default* and return *default*. ' - '*default*\n' - ' defaults to "None".\n' - '\n' - ' update([other])\n' - '\n' - ' Update the dictionary with the key/value pairs from ' - '*other*,\n' - ' overwriting existing keys. Return "None".\n' - '\n' - ' "update()" accepts either another dictionary object or ' - 'an\n' - ' iterable of key/value pairs (as tuples or other ' - 'iterables of\n' - ' length two). If keyword arguments are specified, the ' - 'dictionary\n' - ' is then updated with those key/value pairs: ' - '"d.update(red=1,\n' - ' blue=2)".\n' - '\n' - ' values()\n' - '\n' - ' Return a new view of the dictionary’s values. See ' - 'the\n' - ' documentation of view objects.\n' - '\n' - ' Dictionaries compare equal if and only if they have the ' - 'same "(key,\n' - ' value)" pairs. Order comparisons (‘<’, ‘<=’, ‘>=’, ‘>’) ' - 'raise\n' - ' "TypeError".\n' - '\n' - 'See also: "types.MappingProxyType" can be used to create a ' - 'read-only\n' - ' view of a "dict".\n' - '\n' - '\n' - 'Dictionary view objects\n' - '=======================\n' - '\n' - 'The objects returned by "dict.keys()", "dict.values()" and\n' - '"dict.items()" are *view objects*. They provide a dynamic ' - 'view on the\n' - 'dictionary’s entries, which means that when the dictionary ' - 'changes,\n' - 'the view reflects these changes.\n' - '\n' - 'Dictionary views can be iterated over to yield their ' - 'respective data,\n' - 'and support membership tests:\n' - '\n' - 'len(dictview)\n' - '\n' - ' Return the number of entries in the dictionary.\n' - '\n' - 'iter(dictview)\n' - '\n' - ' Return an iterator over the keys, values or items ' - '(represented as\n' - ' tuples of "(key, value)") in the dictionary.\n' - '\n' - ' Keys and values are iterated over in an arbitrary order ' - 'which is\n' - ' non-random, varies across Python implementations, and ' - 'depends on\n' - ' the dictionary’s history of insertions and deletions. If ' - 'keys,\n' - ' values and items views are iterated over with no ' - 'intervening\n' - ' modifications to the dictionary, the order of items will ' - 'directly\n' - ' correspond. This allows the creation of "(value, key)" ' - 'pairs using\n' - ' "zip()": "pairs = zip(d.values(), d.keys())". Another ' - 'way to\n' - ' create the same list is "pairs = [(v, k) for (k, v) in ' - 'd.items()]".\n' - '\n' - ' Iterating views while adding or deleting entries in the ' - 'dictionary\n' - ' may raise a "RuntimeError" or fail to iterate over all ' - 'entries.\n' - '\n' - 'x in dictview\n' - '\n' - ' Return "True" if *x* is in the underlying dictionary’s ' - 'keys, values\n' - ' or items (in the latter case, *x* should be a "(key, ' - 'value)"\n' - ' tuple).\n' - '\n' - 'Keys views are set-like since their entries are unique and ' - 'hashable.\n' - 'If all values are hashable, so that "(key, value)" pairs are ' - 'unique\n' - 'and hashable, then the items view is also set-like. (Values ' - 'views are\n' - 'not treated as set-like since the entries are generally not ' - 'unique.)\n' - 'For set-like views, all of the operations defined for the ' - 'abstract\n' - 'base class "collections.abc.Set" are available (for example, ' - '"==",\n' - '"<", or "^").\n' - '\n' - 'An example of dictionary view usage:\n' - '\n' - " >>> dishes = {'eggs': 2, 'sausage': 1, 'bacon': 1, " - "'spam': 500}\n" - ' >>> keys = dishes.keys()\n' - ' >>> values = dishes.values()\n' - '\n' - ' >>> # iteration\n' - ' >>> n = 0\n' - ' >>> for val in values:\n' - ' ... n += val\n' - ' >>> print(n)\n' - ' 504\n' - '\n' - ' >>> # keys and values are iterated over in the same ' - 'order\n' - ' >>> list(keys)\n' - " ['eggs', 'bacon', 'sausage', 'spam']\n" - ' >>> list(values)\n' - ' [2, 1, 1, 500]\n' - '\n' - ' >>> # view objects are dynamic and reflect dict changes\n' - " >>> del dishes['eggs']\n" - " >>> del dishes['sausage']\n" - ' >>> list(keys)\n' - " ['spam', 'bacon']\n" - '\n' - ' >>> # set operations\n' - " >>> keys & {'eggs', 'bacon', 'salad'}\n" - " {'bacon'}\n" - " >>> keys ^ {'sausage', 'juice'}\n" - " {'juice', 'sausage', 'bacon', 'spam'}\n", - 'typesmethods': 'Methods\n' - '*******\n' - '\n' - 'Methods are functions that are called using the attribute ' - 'notation.\n' - 'There are two flavors: built-in methods (such as "append()" ' - 'on lists)\n' - 'and class instance methods. Built-in methods are described ' - 'with the\n' - 'types that support them.\n' - '\n' - 'If you access a method (a function defined in a class ' - 'namespace)\n' - 'through an instance, you get a special object: a *bound ' - 'method* (also\n' - 'called *instance method*) object. When called, it will add ' - 'the "self"\n' - 'argument to the argument list. Bound methods have two ' - 'special read-\n' - 'only attributes: "m.__self__" is the object on which the ' - 'method\n' - 'operates, and "m.__func__" is the function implementing the ' - 'method.\n' - 'Calling "m(arg-1, arg-2, ..., arg-n)" is completely ' - 'equivalent to\n' - 'calling "m.__func__(m.__self__, arg-1, arg-2, ..., arg-n)".\n' - '\n' - 'Like function objects, bound method objects support getting ' - 'arbitrary\n' - 'attributes. However, since method attributes are actually ' - 'stored on\n' - 'the underlying function object ("meth.__func__"), setting ' - 'method\n' - 'attributes on bound methods is disallowed. Attempting to ' - 'set an\n' - 'attribute on a method results in an "AttributeError" being ' - 'raised. In\n' - 'order to set a method attribute, you need to explicitly set ' - 'it on the\n' - 'underlying function object:\n' - '\n' - ' >>> class C:\n' - ' ... def method(self):\n' - ' ... pass\n' - ' ...\n' - ' >>> c = C()\n' - " >>> c.method.whoami = 'my name is method' # can't set on " - 'the method\n' - ' Traceback (most recent call last):\n' - ' File "", line 1, in \n' - " AttributeError: 'method' object has no attribute " - "'whoami'\n" - " >>> c.method.__func__.whoami = 'my name is method'\n" - ' >>> c.method.whoami\n' - " 'my name is method'\n" - '\n' - 'See The standard type hierarchy for more information.\n', - 'typesmodules': 'Modules\n' - '*******\n' - '\n' - 'The only special operation on a module is attribute access: ' - '"m.name",\n' - 'where *m* is a module and *name* accesses a name defined in ' - '*m*’s\n' - 'symbol table. Module attributes can be assigned to. (Note ' - 'that the\n' - '"import" statement is not, strictly speaking, an operation ' - 'on a module\n' - 'object; "import foo" does not require a module object named ' - '*foo* to\n' - 'exist, rather it requires an (external) *definition* for a ' - 'module\n' - 'named *foo* somewhere.)\n' - '\n' - 'A special attribute of every module is "__dict__". This is ' - 'the\n' - 'dictionary containing the module’s symbol table. Modifying ' - 'this\n' - 'dictionary will actually change the module’s symbol table, ' - 'but direct\n' - 'assignment to the "__dict__" attribute is not possible (you ' - 'can write\n' - '"m.__dict__[\'a\'] = 1", which defines "m.a" to be "1", but ' - 'you can’t\n' - 'write "m.__dict__ = {}"). Modifying "__dict__" directly is ' - 'not\n' - 'recommended.\n' - '\n' - 'Modules built into the interpreter are written like this: ' - '"". If loaded from a file, they are ' - 'written as\n' - '"".\n', - 'typesseq': 'Sequence Types — "list", "tuple", "range"\n' - '*****************************************\n' - '\n' - 'There are three basic sequence types: lists, tuples, and range\n' - 'objects. Additional sequence types tailored for processing of ' - 'binary\n' - 'data and text strings are described in dedicated sections.\n' - '\n' - '\n' - 'Common Sequence Operations\n' - '==========================\n' - '\n' - 'The operations in the following table are supported by most ' - 'sequence\n' - 'types, both mutable and immutable. The ' - '"collections.abc.Sequence" ABC\n' - 'is provided to make it easier to correctly implement these ' - 'operations\n' - 'on custom sequence types.\n' - '\n' - 'This table lists the sequence operations sorted in ascending ' - 'priority.\n' - 'In the table, *s* and *t* are sequences of the same type, *n*, ' - '*i*,\n' - '*j* and *k* are integers and *x* is an arbitrary object that ' - 'meets any\n' - 'type and value restrictions imposed by *s*.\n' - '\n' - 'The "in" and "not in" operations have the same priorities as ' - 'the\n' - 'comparison operations. The "+" (concatenation) and "*" ' - '(repetition)\n' - 'operations have the same priority as the corresponding numeric\n' - 'operations. [3]\n' - '\n' - '+----------------------------+----------------------------------+------------+\n' - '| Operation | Result ' - '| Notes |\n' - '+============================+==================================+============+\n' - '| "x in s" | "True" if an item of *s* is ' - '| (1) |\n' - '| | equal to *x*, else "False" ' - '| |\n' - '+----------------------------+----------------------------------+------------+\n' - '| "x not in s" | "False" if an item of *s* is ' - '| (1) |\n' - '| | equal to *x*, else "True" ' - '| |\n' - '+----------------------------+----------------------------------+------------+\n' - '| "s + t" | the concatenation of *s* and *t* ' - '| (6)(7) |\n' - '+----------------------------+----------------------------------+------------+\n' - '| "s * n" or "n * s" | equivalent to adding *s* to ' - '| (2)(7) |\n' - '| | itself *n* times ' - '| |\n' - '+----------------------------+----------------------------------+------------+\n' - '| "s[i]" | *i*th item of *s*, origin 0 ' - '| (3) |\n' - '+----------------------------+----------------------------------+------------+\n' - '| "s[i:j]" | slice of *s* from *i* to *j* ' - '| (3)(4) |\n' - '+----------------------------+----------------------------------+------------+\n' - '| "s[i:j:k]" | slice of *s* from *i* to *j* ' - '| (3)(5) |\n' - '| | with step *k* ' - '| |\n' - '+----------------------------+----------------------------------+------------+\n' - '| "len(s)" | length of *s* ' - '| |\n' - '+----------------------------+----------------------------------+------------+\n' - '| "min(s)" | smallest item of *s* ' - '| |\n' - '+----------------------------+----------------------------------+------------+\n' - '| "max(s)" | largest item of *s* ' - '| |\n' - '+----------------------------+----------------------------------+------------+\n' - '| "s.index(x[, i[, j]])" | index of the first occurrence of ' - '| (8) |\n' - '| | *x* in *s* (at or after index ' - '| |\n' - '| | *i* and before index *j*) ' - '| |\n' - '+----------------------------+----------------------------------+------------+\n' - '| "s.count(x)" | total number of occurrences of ' - '| |\n' - '| | *x* in *s* ' - '| |\n' - '+----------------------------+----------------------------------+------------+\n' - '\n' - 'Sequences of the same type also support comparisons. In ' - 'particular,\n' - 'tuples and lists are compared lexicographically by comparing\n' - 'corresponding elements. This means that to compare equal, every\n' - 'element must compare equal and the two sequences must be of the ' - 'same\n' - 'type and have the same length. (For full details see ' - 'Comparisons in\n' - 'the language reference.)\n' - '\n' - 'Notes:\n' - '\n' - '1. While the "in" and "not in" operations are used only for ' - 'simple\n' - ' containment testing in the general case, some specialised ' - 'sequences\n' - ' (such as "str", "bytes" and "bytearray") also use them for\n' - ' subsequence testing:\n' - '\n' - ' >>> "gg" in "eggs"\n' - ' True\n' - '\n' - '2. Values of *n* less than "0" are treated as "0" (which yields ' - 'an\n' - ' empty sequence of the same type as *s*). Note that items in ' - 'the\n' - ' sequence *s* are not copied; they are referenced multiple ' - 'times.\n' - ' This often haunts new Python programmers; consider:\n' - '\n' - ' >>> lists = [[]] * 3\n' - ' >>> lists\n' - ' [[], [], []]\n' - ' >>> lists[0].append(3)\n' - ' >>> lists\n' - ' [[3], [3], [3]]\n' - '\n' - ' What has happened is that "[[]]" is a one-element list ' - 'containing\n' - ' an empty list, so all three elements of "[[]] * 3" are ' - 'references\n' - ' to this single empty list. Modifying any of the elements of\n' - ' "lists" modifies this single list. You can create a list of\n' - ' different lists this way:\n' - '\n' - ' >>> lists = [[] for i in range(3)]\n' - ' >>> lists[0].append(3)\n' - ' >>> lists[1].append(5)\n' - ' >>> lists[2].append(7)\n' - ' >>> lists\n' - ' [[3], [5], [7]]\n' - '\n' - ' Further explanation is available in the FAQ entry How do I ' - 'create a\n' - ' multidimensional list?.\n' - '\n' - '3. If *i* or *j* is negative, the index is relative to the end ' - 'of\n' - ' sequence *s*: "len(s) + i" or "len(s) + j" is substituted. ' - 'But\n' - ' note that "-0" is still "0".\n' - '\n' - '4. The slice of *s* from *i* to *j* is defined as the sequence ' - 'of\n' - ' items with index *k* such that "i <= k < j". If *i* or *j* ' - 'is\n' - ' greater than "len(s)", use "len(s)". If *i* is omitted or ' - '"None",\n' - ' use "0". If *j* is omitted or "None", use "len(s)". If *i* ' - 'is\n' - ' greater than or equal to *j*, the slice is empty.\n' - '\n' - '5. The slice of *s* from *i* to *j* with step *k* is defined as ' - 'the\n' - ' sequence of items with index "x = i + n*k" such that "0 <= n ' - '<\n' - ' (j-i)/k". In other words, the indices are "i", "i+k", ' - '"i+2*k",\n' - ' "i+3*k" and so on, stopping when *j* is reached (but never\n' - ' including *j*). When *k* is positive, *i* and *j* are ' - 'reduced to\n' - ' "len(s)" if they are greater. When *k* is negative, *i* and ' - '*j* are\n' - ' reduced to "len(s) - 1" if they are greater. If *i* or *j* ' - 'are\n' - ' omitted or "None", they become “end” values (which end ' - 'depends on\n' - ' the sign of *k*). Note, *k* cannot be zero. If *k* is ' - '"None", it\n' - ' is treated like "1".\n' - '\n' - '6. Concatenating immutable sequences always results in a new\n' - ' object. This means that building up a sequence by repeated\n' - ' concatenation will have a quadratic runtime cost in the ' - 'total\n' - ' sequence length. To get a linear runtime cost, you must ' - 'switch to\n' - ' one of the alternatives below:\n' - '\n' - ' * if concatenating "str" objects, you can build a list and ' - 'use\n' - ' "str.join()" at the end or else write to an "io.StringIO"\n' - ' instance and retrieve its value when complete\n' - '\n' - ' * if concatenating "bytes" objects, you can similarly use\n' - ' "bytes.join()" or "io.BytesIO", or you can do in-place\n' - ' concatenation with a "bytearray" object. "bytearray" ' - 'objects are\n' - ' mutable and have an efficient overallocation mechanism\n' - '\n' - ' * if concatenating "tuple" objects, extend a "list" instead\n' - '\n' - ' * for other types, investigate the relevant class ' - 'documentation\n' - '\n' - '7. Some sequence types (such as "range") only support item\n' - ' sequences that follow specific patterns, and hence don’t ' - 'support\n' - ' sequence concatenation or repetition.\n' - '\n' - '8. "index" raises "ValueError" when *x* is not found in *s*. ' - 'Not\n' - ' all implementations support passing the additional arguments ' - '*i*\n' - ' and *j*. These arguments allow efficient searching of ' - 'subsections\n' - ' of the sequence. Passing the extra arguments is roughly ' - 'equivalent\n' - ' to using "s[i:j].index(x)", only without copying any data and ' - 'with\n' - ' the returned index being relative to the start of the ' - 'sequence\n' - ' rather than the start of the slice.\n' - '\n' - '\n' - 'Immutable Sequence Types\n' - '========================\n' - '\n' - 'The only operation that immutable sequence types generally ' - 'implement\n' - 'that is not also implemented by mutable sequence types is ' - 'support for\n' - 'the "hash()" built-in.\n' - '\n' - 'This support allows immutable sequences, such as "tuple" ' - 'instances, to\n' - 'be used as "dict" keys and stored in "set" and "frozenset" ' - 'instances.\n' - '\n' - 'Attempting to hash an immutable sequence that contains ' - 'unhashable\n' - 'values will result in "TypeError".\n' - '\n' - '\n' - 'Mutable Sequence Types\n' - '======================\n' - '\n' - 'The operations in the following table are defined on mutable ' - 'sequence\n' - 'types. The "collections.abc.MutableSequence" ABC is provided to ' - 'make\n' - 'it easier to correctly implement these operations on custom ' - 'sequence\n' - 'types.\n' - '\n' - 'In the table *s* is an instance of a mutable sequence type, *t* ' - 'is any\n' - 'iterable object and *x* is an arbitrary object that meets any ' - 'type and\n' - 'value restrictions imposed by *s* (for example, "bytearray" ' - 'only\n' - 'accepts integers that meet the value restriction "0 <= x <= ' - '255").\n' - '\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| Operation | ' - 'Result | Notes |\n' - '+================================+==================================+=======================+\n' - '| "s[i] = x" | item *i* of *s* is replaced ' - 'by | |\n' - '| | ' - '*x* | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s[i:j] = t" | slice of *s* from *i* to *j* ' - 'is | |\n' - '| | replaced by the contents of ' - 'the | |\n' - '| | iterable ' - '*t* | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "del s[i:j]" | same as "s[i:j] = ' - '[]" | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s[i:j:k] = t" | the elements of "s[i:j:k]" ' - 'are | (1) |\n' - '| | replaced by those of ' - '*t* | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "del s[i:j:k]" | removes the elements ' - 'of | |\n' - '| | "s[i:j:k]" from the ' - 'list | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.append(x)" | appends *x* to the end of ' - 'the | |\n' - '| | sequence (same ' - 'as | |\n' - '| | "s[len(s):len(s)] = ' - '[x]") | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.clear()" | removes all items from *s* ' - '(same | (5) |\n' - '| | as "del ' - 's[:]") | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.copy()" | creates a shallow copy of ' - '*s* | (5) |\n' - '| | (same as ' - '"s[:]") | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.extend(t)" or "s += t" | extends *s* with the contents ' - 'of | |\n' - '| | *t* (for the most part the ' - 'same | |\n' - '| | as "s[len(s):len(s)] = ' - 't") | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s *= n" | updates *s* with its ' - 'contents | (6) |\n' - '| | repeated *n* ' - 'times | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.insert(i, x)" | inserts *x* into *s* at ' - 'the | |\n' - '| | index given by *i* (same ' - 'as | |\n' - '| | "s[i:i] = ' - '[x]") | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.pop([i])" | retrieves the item at *i* ' - 'and | (2) |\n' - '| | also removes it from ' - '*s* | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.remove(x)" | remove the first item from ' - '*s* | (3) |\n' - '| | where "s[i] == ' - 'x" | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.reverse()" | reverses the items of *s* ' - 'in | (4) |\n' - '| | ' - 'place | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '\n' - 'Notes:\n' - '\n' - '1. *t* must have the same length as the slice it is replacing.\n' - '\n' - '2. The optional argument *i* defaults to "-1", so that by ' - 'default\n' - ' the last item is removed and returned.\n' - '\n' - '3. "remove" raises "ValueError" when *x* is not found in *s*.\n' - '\n' - '4. The "reverse()" method modifies the sequence in place for\n' - ' economy of space when reversing a large sequence. To remind ' - 'users\n' - ' that it operates by side effect, it does not return the ' - 'reversed\n' - ' sequence.\n' - '\n' - '5. "clear()" and "copy()" are included for consistency with the\n' - ' interfaces of mutable containers that don’t support slicing\n' - ' operations (such as "dict" and "set")\n' - '\n' - ' New in version 3.3: "clear()" and "copy()" methods.\n' - '\n' - '6. The value *n* is an integer, or an object implementing\n' - ' "__index__()". Zero and negative values of *n* clear the ' - 'sequence.\n' - ' Items in the sequence are not copied; they are referenced ' - 'multiple\n' - ' times, as explained for "s * n" under Common Sequence ' - 'Operations.\n' - '\n' - '\n' - 'Lists\n' - '=====\n' - '\n' - 'Lists are mutable sequences, typically used to store collections ' - 'of\n' - 'homogeneous items (where the precise degree of similarity will ' - 'vary by\n' - 'application).\n' - '\n' - 'class list([iterable])\n' - '\n' - ' Lists may be constructed in several ways:\n' - '\n' - ' * Using a pair of square brackets to denote the empty list: ' - '"[]"\n' - '\n' - ' * Using square brackets, separating items with commas: ' - '"[a]",\n' - ' "[a, b, c]"\n' - '\n' - ' * Using a list comprehension: "[x for x in iterable]"\n' - '\n' - ' * Using the type constructor: "list()" or "list(iterable)"\n' - '\n' - ' The constructor builds a list whose items are the same and in ' - 'the\n' - ' same order as *iterable*’s items. *iterable* may be either ' - 'a\n' - ' sequence, a container that supports iteration, or an ' - 'iterator\n' - ' object. If *iterable* is already a list, a copy is made and\n' - ' returned, similar to "iterable[:]". For example, ' - '"list(\'abc\')"\n' - ' returns "[\'a\', \'b\', \'c\']" and "list( (1, 2, 3) )" ' - 'returns "[1, 2,\n' - ' 3]". If no argument is given, the constructor creates a new ' - 'empty\n' - ' list, "[]".\n' - '\n' - ' Many other operations also produce lists, including the ' - '"sorted()"\n' - ' built-in.\n' - '\n' - ' Lists implement all of the common and mutable sequence ' - 'operations.\n' - ' Lists also provide the following additional method:\n' - '\n' - ' sort(*, key=None, reverse=False)\n' - '\n' - ' This method sorts the list in place, using only "<" ' - 'comparisons\n' - ' between items. Exceptions are not suppressed - if any ' - 'comparison\n' - ' operations fail, the entire sort operation will fail (and ' - 'the\n' - ' list will likely be left in a partially modified state).\n' - '\n' - ' "sort()" accepts two arguments that can only be passed by\n' - ' keyword (keyword-only arguments):\n' - '\n' - ' *key* specifies a function of one argument that is used ' - 'to\n' - ' extract a comparison key from each list element (for ' - 'example,\n' - ' "key=str.lower"). The key corresponding to each item in ' - 'the list\n' - ' is calculated once and then used for the entire sorting ' - 'process.\n' - ' The default value of "None" means that list items are ' - 'sorted\n' - ' directly without calculating a separate key value.\n' - '\n' - ' The "functools.cmp_to_key()" utility is available to ' - 'convert a\n' - ' 2.x style *cmp* function to a *key* function.\n' - '\n' - ' *reverse* is a boolean value. If set to "True", then the ' - 'list\n' - ' elements are sorted as if each comparison were reversed.\n' - '\n' - ' This method modifies the sequence in place for economy of ' - 'space\n' - ' when sorting a large sequence. To remind users that it ' - 'operates\n' - ' by side effect, it does not return the sorted sequence ' - '(use\n' - ' "sorted()" to explicitly request a new sorted list ' - 'instance).\n' - '\n' - ' The "sort()" method is guaranteed to be stable. A sort ' - 'is\n' - ' stable if it guarantees not to change the relative order ' - 'of\n' - ' elements that compare equal — this is helpful for sorting ' - 'in\n' - ' multiple passes (for example, sort by department, then by ' - 'salary\n' - ' grade).\n' - '\n' - ' **CPython implementation detail:** While a list is being ' - 'sorted,\n' - ' the effect of attempting to mutate, or even inspect, the ' - 'list is\n' - ' undefined. The C implementation of Python makes the list ' - 'appear\n' - ' empty for the duration, and raises "ValueError" if it can ' - 'detect\n' - ' that the list has been mutated during a sort.\n' - '\n' - '\n' - 'Tuples\n' - '======\n' - '\n' - 'Tuples are immutable sequences, typically used to store ' - 'collections of\n' - 'heterogeneous data (such as the 2-tuples produced by the ' - '"enumerate()"\n' - 'built-in). Tuples are also used for cases where an immutable ' - 'sequence\n' - 'of homogeneous data is needed (such as allowing storage in a ' - '"set" or\n' - '"dict" instance).\n' - '\n' - 'class tuple([iterable])\n' - '\n' - ' Tuples may be constructed in a number of ways:\n' - '\n' - ' * Using a pair of parentheses to denote the empty tuple: ' - '"()"\n' - '\n' - ' * Using a trailing comma for a singleton tuple: "a," or ' - '"(a,)"\n' - '\n' - ' * Separating items with commas: "a, b, c" or "(a, b, c)"\n' - '\n' - ' * Using the "tuple()" built-in: "tuple()" or ' - '"tuple(iterable)"\n' - '\n' - ' The constructor builds a tuple whose items are the same and ' - 'in the\n' - ' same order as *iterable*’s items. *iterable* may be either ' - 'a\n' - ' sequence, a container that supports iteration, or an ' - 'iterator\n' - ' object. If *iterable* is already a tuple, it is returned\n' - ' unchanged. For example, "tuple(\'abc\')" returns "(\'a\', ' - '\'b\', \'c\')"\n' - ' and "tuple( [1, 2, 3] )" returns "(1, 2, 3)". If no argument ' - 'is\n' - ' given, the constructor creates a new empty tuple, "()".\n' - '\n' - ' Note that it is actually the comma which makes a tuple, not ' - 'the\n' - ' parentheses. The parentheses are optional, except in the ' - 'empty\n' - ' tuple case, or when they are needed to avoid syntactic ' - 'ambiguity.\n' - ' For example, "f(a, b, c)" is a function call with three ' - 'arguments,\n' - ' while "f((a, b, c))" is a function call with a 3-tuple as the ' - 'sole\n' - ' argument.\n' - '\n' - ' Tuples implement all of the common sequence operations.\n' - '\n' - 'For heterogeneous collections of data where access by name is ' - 'clearer\n' - 'than access by index, "collections.namedtuple()" may be a more\n' - 'appropriate choice than a simple tuple object.\n' - '\n' - '\n' - 'Ranges\n' - '======\n' - '\n' - 'The "range" type represents an immutable sequence of numbers and ' - 'is\n' - 'commonly used for looping a specific number of times in "for" ' - 'loops.\n' - '\n' - 'class range(stop)\n' - 'class range(start, stop[, step])\n' - '\n' - ' The arguments to the range constructor must be integers ' - '(either\n' - ' built-in "int" or any object that implements the "__index__"\n' - ' special method). If the *step* argument is omitted, it ' - 'defaults to\n' - ' "1". If the *start* argument is omitted, it defaults to "0". ' - 'If\n' - ' *step* is zero, "ValueError" is raised.\n' - '\n' - ' For a positive *step*, the contents of a range "r" are ' - 'determined\n' - ' by the formula "r[i] = start + step*i" where "i >= 0" and ' - '"r[i] <\n' - ' stop".\n' - '\n' - ' For a negative *step*, the contents of the range are still\n' - ' determined by the formula "r[i] = start + step*i", but the\n' - ' constraints are "i >= 0" and "r[i] > stop".\n' - '\n' - ' A range object will be empty if "r[0]" does not meet the ' - 'value\n' - ' constraint. Ranges do support negative indices, but these ' - 'are\n' - ' interpreted as indexing from the end of the sequence ' - 'determined by\n' - ' the positive indices.\n' - '\n' - ' Ranges containing absolute values larger than "sys.maxsize" ' - 'are\n' - ' permitted but some features (such as "len()") may raise\n' - ' "OverflowError".\n' - '\n' - ' Range examples:\n' - '\n' - ' >>> list(range(10))\n' - ' [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n' - ' >>> list(range(1, 11))\n' - ' [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]\n' - ' >>> list(range(0, 30, 5))\n' - ' [0, 5, 10, 15, 20, 25]\n' - ' >>> list(range(0, 10, 3))\n' - ' [0, 3, 6, 9]\n' - ' >>> list(range(0, -10, -1))\n' - ' [0, -1, -2, -3, -4, -5, -6, -7, -8, -9]\n' - ' >>> list(range(0))\n' - ' []\n' - ' >>> list(range(1, 0))\n' - ' []\n' - '\n' - ' Ranges implement all of the common sequence operations ' - 'except\n' - ' concatenation and repetition (due to the fact that range ' - 'objects\n' - ' can only represent sequences that follow a strict pattern ' - 'and\n' - ' repetition and concatenation will usually violate that ' - 'pattern).\n' - '\n' - ' start\n' - '\n' - ' The value of the *start* parameter (or "0" if the ' - 'parameter was\n' - ' not supplied)\n' - '\n' - ' stop\n' - '\n' - ' The value of the *stop* parameter\n' - '\n' - ' step\n' - '\n' - ' The value of the *step* parameter (or "1" if the parameter ' - 'was\n' - ' not supplied)\n' - '\n' - 'The advantage of the "range" type over a regular "list" or ' - '"tuple" is\n' - 'that a "range" object will always take the same (small) amount ' - 'of\n' - 'memory, no matter the size of the range it represents (as it ' - 'only\n' - 'stores the "start", "stop" and "step" values, calculating ' - 'individual\n' - 'items and subranges as needed).\n' - '\n' - 'Range objects implement the "collections.abc.Sequence" ABC, and\n' - 'provide features such as containment tests, element index ' - 'lookup,\n' - 'slicing and support for negative indices (see Sequence Types — ' - 'list,\n' - 'tuple, range):\n' - '\n' - '>>> r = range(0, 20, 2)\n' - '>>> r\n' - 'range(0, 20, 2)\n' - '>>> 11 in r\n' - 'False\n' - '>>> 10 in r\n' - 'True\n' - '>>> r.index(10)\n' - '5\n' - '>>> r[5]\n' - '10\n' - '>>> r[:5]\n' - 'range(0, 10, 2)\n' - '>>> r[-1]\n' - '18\n' - '\n' - 'Testing range objects for equality with "==" and "!=" compares ' - 'them as\n' - 'sequences. That is, two range objects are considered equal if ' - 'they\n' - 'represent the same sequence of values. (Note that two range ' - 'objects\n' - 'that compare equal might have different "start", "stop" and ' - '"step"\n' - 'attributes, for example "range(0) == range(2, 1, 3)" or ' - '"range(0, 3,\n' - '2) == range(0, 4, 2)".)\n' - '\n' - 'Changed in version 3.2: Implement the Sequence ABC. Support ' - 'slicing\n' - 'and negative indices. Test "int" objects for membership in ' - 'constant\n' - 'time instead of iterating through all items.\n' - '\n' - 'Changed in version 3.3: Define ‘==’ and ‘!=’ to compare range ' - 'objects\n' - 'based on the sequence of values they define (instead of ' - 'comparing\n' - 'based on object identity).\n' - '\n' - 'New in version 3.3: The "start", "stop" and "step" attributes.\n' - '\n' - 'See also:\n' - '\n' - ' * The linspace recipe shows how to implement a lazy version ' - 'of\n' - ' range suitable for floating point applications.\n', - 'typesseq-mutable': 'Mutable Sequence Types\n' - '**********************\n' - '\n' - 'The operations in the following table are defined on ' - 'mutable sequence\n' - 'types. The "collections.abc.MutableSequence" ABC is ' - 'provided to make\n' - 'it easier to correctly implement these operations on ' - 'custom sequence\n' - 'types.\n' - '\n' - 'In the table *s* is an instance of a mutable sequence ' - 'type, *t* is any\n' - 'iterable object and *x* is an arbitrary object that ' - 'meets any type and\n' - 'value restrictions imposed by *s* (for example, ' - '"bytearray" only\n' - 'accepts integers that meet the value restriction "0 <= x ' - '<= 255").\n' - '\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| Operation | ' - 'Result | Notes ' - '|\n' - '+================================+==================================+=======================+\n' - '| "s[i] = x" | item *i* of *s* is ' - 'replaced by | |\n' - '| | ' - '*x* | ' - '|\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s[i:j] = t" | slice of *s* from *i* ' - 'to *j* is | |\n' - '| | replaced by the ' - 'contents of the | |\n' - '| | iterable ' - '*t* | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "del s[i:j]" | same as "s[i:j] = ' - '[]" | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s[i:j:k] = t" | the elements of ' - '"s[i:j:k]" are | (1) |\n' - '| | replaced by those of ' - '*t* | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "del s[i:j:k]" | removes the elements ' - 'of | |\n' - '| | "s[i:j:k]" from the ' - 'list | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.append(x)" | appends *x* to the ' - 'end of the | |\n' - '| | sequence (same ' - 'as | |\n' - '| | "s[len(s):len(s)] = ' - '[x]") | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.clear()" | removes all items ' - 'from *s* (same | (5) |\n' - '| | as "del ' - 's[:]") | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.copy()" | creates a shallow ' - 'copy of *s* | (5) |\n' - '| | (same as ' - '"s[:]") | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.extend(t)" or "s += t" | extends *s* with the ' - 'contents of | |\n' - '| | *t* (for the most ' - 'part the same | |\n' - '| | as "s[len(s):len(s)] ' - '= t") | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s *= n" | updates *s* with its ' - 'contents | (6) |\n' - '| | repeated *n* ' - 'times | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.insert(i, x)" | inserts *x* into *s* ' - 'at the | |\n' - '| | index given by *i* ' - '(same as | |\n' - '| | "s[i:i] = ' - '[x]") | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.pop([i])" | retrieves the item at ' - '*i* and | (2) |\n' - '| | also removes it from ' - '*s* | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.remove(x)" | remove the first item ' - 'from *s* | (3) |\n' - '| | where "s[i] == ' - 'x" | |\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.reverse()" | reverses the items of ' - '*s* in | (4) |\n' - '| | ' - 'place | ' - '|\n' - '+--------------------------------+----------------------------------+-----------------------+\n' - '\n' - 'Notes:\n' - '\n' - '1. *t* must have the same length as the slice it is ' - 'replacing.\n' - '\n' - '2. The optional argument *i* defaults to "-1", so that ' - 'by default\n' - ' the last item is removed and returned.\n' - '\n' - '3. "remove" raises "ValueError" when *x* is not found in ' - '*s*.\n' - '\n' - '4. The "reverse()" method modifies the sequence in place ' - 'for\n' - ' economy of space when reversing a large sequence. To ' - 'remind users\n' - ' that it operates by side effect, it does not return ' - 'the reversed\n' - ' sequence.\n' - '\n' - '5. "clear()" and "copy()" are included for consistency ' - 'with the\n' - ' interfaces of mutable containers that don’t support ' - 'slicing\n' - ' operations (such as "dict" and "set")\n' - '\n' - ' New in version 3.3: "clear()" and "copy()" methods.\n' - '\n' - '6. The value *n* is an integer, or an object ' - 'implementing\n' - ' "__index__()". Zero and negative values of *n* clear ' - 'the sequence.\n' - ' Items in the sequence are not copied; they are ' - 'referenced multiple\n' - ' times, as explained for "s * n" under Common Sequence ' - 'Operations.\n', - 'unary': 'Unary arithmetic and bitwise operations\n' - '***************************************\n' - '\n' - 'All unary arithmetic and bitwise operations have the same ' - 'priority:\n' - '\n' - ' u_expr ::= power | "-" u_expr | "+" u_expr | "~" u_expr\n' - '\n' - 'The unary "-" (minus) operator yields the negation of its numeric\n' - 'argument.\n' - '\n' - 'The unary "+" (plus) operator yields its numeric argument ' - 'unchanged.\n' - '\n' - 'The unary "~" (invert) operator yields the bitwise inversion of ' - 'its\n' - 'integer argument. The bitwise inversion of "x" is defined as\n' - '"-(x+1)". It only applies to integral numbers.\n' - '\n' - 'In all three cases, if the argument does not have the proper type, ' - 'a\n' - '"TypeError" exception is raised.\n', - 'while': 'The "while" statement\n' - '*********************\n' - '\n' - 'The "while" statement is used for repeated execution as long as an\n' - 'expression is true:\n' - '\n' - ' while_stmt ::= "while" expression ":" suite\n' - ' ["else" ":" suite]\n' - '\n' - 'This repeatedly tests the expression and, if it is true, executes ' - 'the\n' - 'first suite; if the expression is false (which may be the first ' - 'time\n' - 'it is tested) the suite of the "else" clause, if present, is ' - 'executed\n' - 'and the loop terminates.\n' - '\n' - 'A "break" statement executed in the first suite terminates the ' - 'loop\n' - 'without executing the "else" clause’s suite. A "continue" ' - 'statement\n' - 'executed in the first suite skips the rest of the suite and goes ' - 'back\n' - 'to testing the expression.\n', - 'with': 'The "with" statement\n' - '********************\n' - '\n' - 'The "with" statement is used to wrap the execution of a block with\n' - 'methods defined by a context manager (see section With Statement\n' - 'Context Managers). This allows common "try"…"except"…"finally" ' - 'usage\n' - 'patterns to be encapsulated for convenient reuse.\n' - '\n' - ' with_stmt ::= "with" with_item ("," with_item)* ":" suite\n' - ' with_item ::= expression ["as" target]\n' - '\n' - 'The execution of the "with" statement with one “item” proceeds as\n' - 'follows:\n' - '\n' - '1. The context expression (the expression given in the "with_item")\n' - ' is evaluated to obtain a context manager.\n' - '\n' - '2. The context manager’s "__exit__()" is loaded for later use.\n' - '\n' - '3. The context manager’s "__enter__()" method is invoked.\n' - '\n' - '4. If a target was included in the "with" statement, the return\n' - ' value from "__enter__()" is assigned to it.\n' - '\n' - ' Note: The "with" statement guarantees that if the "__enter__()"\n' - ' method returns without an error, then "__exit__()" will always ' - 'be\n' - ' called. Thus, if an error occurs during the assignment to the\n' - ' target list, it will be treated the same as an error occurring\n' - ' within the suite would be. See step 6 below.\n' - '\n' - '5. The suite is executed.\n' - '\n' - '6. The context manager’s "__exit__()" method is invoked. If an\n' - ' exception caused the suite to be exited, its type, value, and\n' - ' traceback are passed as arguments to "__exit__()". Otherwise, ' - 'three\n' - ' "None" arguments are supplied.\n' - '\n' - ' If the suite was exited due to an exception, and the return ' - 'value\n' - ' from the "__exit__()" method was false, the exception is ' - 'reraised.\n' - ' If the return value was true, the exception is suppressed, and\n' - ' execution continues with the statement following the "with"\n' - ' statement.\n' - '\n' - ' If the suite was exited for any reason other than an exception, ' - 'the\n' - ' return value from "__exit__()" is ignored, and execution ' - 'proceeds\n' - ' at the normal location for the kind of exit that was taken.\n' - '\n' - 'With more than one item, the context managers are processed as if\n' - 'multiple "with" statements were nested:\n' - '\n' - ' with A() as a, B() as b:\n' - ' suite\n' - '\n' - 'is equivalent to\n' - '\n' - ' with A() as a:\n' - ' with B() as b:\n' - ' suite\n' - '\n' - 'Changed in version 3.1: Support for multiple context expressions.\n' - '\n' - 'See also:\n' - '\n' - ' **PEP 343** - The “with” statement\n' - ' The specification, background, and examples for the Python ' - '"with"\n' - ' statement.\n', - 'yield': 'The "yield" statement\n' - '*********************\n' - '\n' - ' yield_stmt ::= yield_expression\n' - '\n' - 'A "yield" statement is semantically equivalent to a yield ' - 'expression.\n' - 'The yield statement can be used to omit the parentheses that would\n' - 'otherwise be required in the equivalent yield expression ' - 'statement.\n' - 'For example, the yield statements\n' - '\n' - ' yield \n' - ' yield from \n' - '\n' - 'are equivalent to the yield expression statements\n' - '\n' - ' (yield )\n' - ' (yield from )\n' - '\n' - 'Yield expressions and statements are only used when defining a\n' - '*generator* function, and are only used in the body of the ' - 'generator\n' - 'function. Using yield in a function definition is sufficient to ' - 'cause\n' - 'that definition to create a generator function instead of a normal\n' - 'function.\n' - '\n' - 'For full details of "yield" semantics, refer to the Yield ' - 'expressions\n' - 'section.\n'} +# Autogenerated by Sphinx on Fri Dec 5 18:49:09 2025 +# as part of the release process. + +topics = { + 'assert': r'''The "assert" statement +********************** + +Assert statements are a convenient way to insert debugging assertions +into a program: + + assert_stmt: "assert" expression ["," expression] + +The simple form, "assert expression", is equivalent to + + if __debug__: + if not expression: raise AssertionError + +The extended form, "assert expression1, expression2", is equivalent to + + if __debug__: + if not expression1: raise AssertionError(expression2) + +These equivalences assume that "__debug__" and "AssertionError" refer +to the built-in variables with those names. In the current +implementation, the built-in variable "__debug__" is "True" under +normal circumstances, "False" when optimization is requested (command +line option "-O"). The current code generator emits no code for an +"assert" statement when optimization is requested at compile time. +Note that it is unnecessary to include the source code for the +expression that failed in the error message; it will be displayed as +part of the stack trace. + +Assignments to "__debug__" are illegal. The value for the built-in +variable is determined when the interpreter starts. +''', + 'assignment': r'''Assignment statements +********************* + +Assignment statements are used to (re)bind names to values and to +modify attributes or items of mutable objects: + + assignment_stmt: (target_list "=")+ (starred_expression | yield_expression) + target_list: target ("," target)* [","] + target: identifier + | "(" [target_list] ")" + | "[" [target_list] "]" + | attributeref + | subscription + | slicing + | "*" target + +(See section Primaries for the syntax definitions for *attributeref*, +*subscription*, and *slicing*.) + +An assignment statement evaluates the expression list (remember that +this can be a single expression or a comma-separated list, the latter +yielding a tuple) and assigns the single resulting object to each of +the target lists, from left to right. + +Assignment is defined recursively depending on the form of the target +(list). When a target is part of a mutable object (an attribute +reference, subscription or slicing), the mutable object must +ultimately perform the assignment and decide about its validity, and +may raise an exception if the assignment is unacceptable. The rules +observed by various types and the exceptions raised are given with the +definition of the object types (see section The standard type +hierarchy). + +Assignment of an object to a target list, optionally enclosed in +parentheses or square brackets, is recursively defined as follows. + +* If the target list is a single target with no trailing comma, + optionally in parentheses, the object is assigned to that target. + +* Else: + + * If the target list contains one target prefixed with an asterisk, + called a “starred” target: The object must be an iterable with at + least as many items as there are targets in the target list, minus + one. The first items of the iterable are assigned, from left to + right, to the targets before the starred target. The final items + of the iterable are assigned to the targets after the starred + target. A list of the remaining items in the iterable is then + assigned to the starred target (the list can be empty). + + * Else: The object must be an iterable with the same number of items + as there are targets in the target list, and the items are + assigned, from left to right, to the corresponding targets. + +Assignment of an object to a single target is recursively defined as +follows. + +* If the target is an identifier (name): + + * If the name does not occur in a "global" or "nonlocal" statement + in the current code block: the name is bound to the object in the + current local namespace. + + * Otherwise: the name is bound to the object in the global namespace + or the outer namespace determined by "nonlocal", respectively. + + The name is rebound if it was already bound. This may cause the + reference count for the object previously bound to the name to reach + zero, causing the object to be deallocated and its destructor (if it + has one) to be called. + +* If the target is an attribute reference: The primary expression in + the reference is evaluated. It should yield an object with + assignable attributes; if this is not the case, "TypeError" is + raised. That object is then asked to assign the assigned object to + the given attribute; if it cannot perform the assignment, it raises + an exception (usually but not necessarily "AttributeError"). + + Note: If the object is a class instance and the attribute reference + occurs on both sides of the assignment operator, the right-hand side + expression, "a.x" can access either an instance attribute or (if no + instance attribute exists) a class attribute. The left-hand side + target "a.x" is always set as an instance attribute, creating it if + necessary. Thus, the two occurrences of "a.x" do not necessarily + refer to the same attribute: if the right-hand side expression + refers to a class attribute, the left-hand side creates a new + instance attribute as the target of the assignment: + + class Cls: + x = 3 # class variable + inst = Cls() + inst.x = inst.x + 1 # writes inst.x as 4 leaving Cls.x as 3 + + This description does not necessarily apply to descriptor + attributes, such as properties created with "property()". + +* If the target is a subscription: The primary expression in the + reference is evaluated. It should yield either a mutable sequence + object (such as a list) or a mapping object (such as a dictionary). + Next, the subscript expression is evaluated. + + If the primary is a mutable sequence object (such as a list), the + subscript must yield an integer. If it is negative, the sequence’s + length is added to it. The resulting value must be a nonnegative + integer less than the sequence’s length, and the sequence is asked + to assign the assigned object to its item with that index. If the + index is out of range, "IndexError" is raised (assignment to a + subscripted sequence cannot add new items to a list). + + If the primary is a mapping object (such as a dictionary), the + subscript must have a type compatible with the mapping’s key type, + and the mapping is then asked to create a key/value pair which maps + the subscript to the assigned object. This can either replace an + existing key/value pair with the same key value, or insert a new + key/value pair (if no key with the same value existed). + + For user-defined objects, the "__setitem__()" method is called with + appropriate arguments. + +* If the target is a slicing: The primary expression in the reference + is evaluated. It should yield a mutable sequence object (such as a + list). The assigned object should be a sequence object of the same + type. Next, the lower and upper bound expressions are evaluated, + insofar they are present; defaults are zero and the sequence’s + length. The bounds should evaluate to integers. If either bound is + negative, the sequence’s length is added to it. The resulting + bounds are clipped to lie between zero and the sequence’s length, + inclusive. Finally, the sequence object is asked to replace the + slice with the items of the assigned sequence. The length of the + slice may be different from the length of the assigned sequence, + thus changing the length of the target sequence, if the target + sequence allows it. + +**CPython implementation detail:** In the current implementation, the +syntax for targets is taken to be the same as for expressions, and +invalid syntax is rejected during the code generation phase, causing +less detailed error messages. + +Although the definition of assignment implies that overlaps between +the left-hand side and the right-hand side are ‘simultaneous’ (for +example "a, b = b, a" swaps two variables), overlaps *within* the +collection of assigned-to variables occur left-to-right, sometimes +resulting in confusion. For instance, the following program prints +"[0, 2]": + + x = [0, 1] + i = 0 + i, x[i] = 1, 2 # i is updated, then x[i] is updated + print(x) + +See also: + + **PEP 3132** - Extended Iterable Unpacking + The specification for the "*target" feature. + + +Augmented assignment statements +=============================== + +Augmented assignment is the combination, in a single statement, of a +binary operation and an assignment statement: + + augmented_assignment_stmt: augtarget augop (expression_list | yield_expression) + augtarget: identifier | attributeref | subscription | slicing + augop: "+=" | "-=" | "*=" | "@=" | "/=" | "//=" | "%=" | "**=" + | ">>=" | "<<=" | "&=" | "^=" | "|=" + +(See section Primaries for the syntax definitions of the last three +symbols.) + +An augmented assignment evaluates the target (which, unlike normal +assignment statements, cannot be an unpacking) and the expression +list, performs the binary operation specific to the type of assignment +on the two operands, and assigns the result to the original target. +The target is only evaluated once. + +An augmented assignment statement like "x += 1" can be rewritten as "x += x + 1" to achieve a similar, but not exactly equal effect. In the +augmented version, "x" is only evaluated once. Also, when possible, +the actual operation is performed *in-place*, meaning that rather than +creating a new object and assigning that to the target, the old object +is modified instead. + +Unlike normal assignments, augmented assignments evaluate the left- +hand side *before* evaluating the right-hand side. For example, "a[i] ++= f(x)" first looks-up "a[i]", then it evaluates "f(x)" and performs +the addition, and lastly, it writes the result back to "a[i]". + +With the exception of assigning to tuples and multiple targets in a +single statement, the assignment done by augmented assignment +statements is handled the same way as normal assignments. Similarly, +with the exception of the possible *in-place* behavior, the binary +operation performed by augmented assignment is the same as the normal +binary operations. + +For targets which are attribute references, the same caveat about +class and instance attributes applies as for regular assignments. + + +Annotated assignment statements +=============================== + +*Annotation* assignment is the combination, in a single statement, of +a variable or attribute annotation and an optional assignment +statement: + + annotated_assignment_stmt: augtarget ":" expression + ["=" (starred_expression | yield_expression)] + +The difference from normal Assignment statements is that only a single +target is allowed. + +The assignment target is considered “simple” if it consists of a +single name that is not enclosed in parentheses. For simple assignment +targets, if in class or module scope, the annotations are gathered in +a lazily evaluated annotation scope. The annotations can be evaluated +using the "__annotations__" attribute of a class or module, or using +the facilities in the "annotationlib" module. + +If the assignment target is not simple (an attribute, subscript node, +or parenthesized name), the annotation is never evaluated. + +If a name is annotated in a function scope, then this name is local +for that scope. Annotations are never evaluated and stored in function +scopes. + +If the right hand side is present, an annotated assignment performs +the actual assignment as if there was no annotation present. If the +right hand side is not present for an expression target, then the +interpreter evaluates the target except for the last "__setitem__()" +or "__setattr__()" call. + +See also: + + **PEP 526** - Syntax for Variable Annotations + The proposal that added syntax for annotating the types of + variables (including class variables and instance variables), + instead of expressing them through comments. + + **PEP 484** - Type hints + The proposal that added the "typing" module to provide a standard + syntax for type annotations that can be used in static analysis + tools and IDEs. + +Changed in version 3.8: Now annotated assignments allow the same +expressions in the right hand side as regular assignments. Previously, +some expressions (like un-parenthesized tuple expressions) caused a +syntax error. + +Changed in version 3.14: Annotations are now lazily evaluated in a +separate annotation scope. If the assignment target is not simple, +annotations are never evaluated. +''', + 'assignment-expressions': r'''Assignment expressions +********************** + + assignment_expression: [identifier ":="] expression + +An assignment expression (sometimes also called a “named expression” +or “walrus”) assigns an "expression" to an "identifier", while also +returning the value of the "expression". + +One common use case is when handling matched regular expressions: + + if matching := pattern.search(data): + do_something(matching) + +Or, when processing a file stream in chunks: + + while chunk := file.read(9000): + process(chunk) + +Assignment expressions must be surrounded by parentheses when used as +expression statements and when used as sub-expressions in slicing, +conditional, lambda, keyword-argument, and comprehension-if +expressions and in "assert", "with", and "assignment" statements. In +all other places where they can be used, parentheses are not required, +including in "if" and "while" statements. + +Added in version 3.8: See **PEP 572** for more details about +assignment expressions. +''', + 'async': r'''Coroutines +********** + +Added in version 3.5. + + +Coroutine function definition +============================= + + async_funcdef: [decorators] "async" "def" funcname "(" [parameter_list] ")" + ["->" expression] ":" suite + +Execution of Python coroutines can be suspended and resumed at many +points (see *coroutine*). "await" expressions, "async for" and "async +with" can only be used in the body of a coroutine function. + +Functions defined with "async def" syntax are always coroutine +functions, even if they do not contain "await" or "async" keywords. + +It is a "SyntaxError" to use a "yield from" expression inside the body +of a coroutine function. + +An example of a coroutine function: + + async def func(param1, param2): + do_stuff() + await some_coroutine() + +Changed in version 3.7: "await" and "async" are now keywords; +previously they were only treated as such inside the body of a +coroutine function. + + +The "async for" statement +========================= + + async_for_stmt: "async" for_stmt + +An *asynchronous iterable* provides an "__aiter__" method that +directly returns an *asynchronous iterator*, which can call +asynchronous code in its "__anext__" method. + +The "async for" statement allows convenient iteration over +asynchronous iterables. + +The following code: + + async for TARGET in ITER: + SUITE + else: + SUITE2 + +Is semantically equivalent to: + + iter = (ITER) + iter = type(iter).__aiter__(iter) + running = True + + while running: + try: + TARGET = await type(iter).__anext__(iter) + except StopAsyncIteration: + running = False + else: + SUITE + else: + SUITE2 + +See also "__aiter__()" and "__anext__()" for details. + +It is a "SyntaxError" to use an "async for" statement outside the body +of a coroutine function. + + +The "async with" statement +========================== + + async_with_stmt: "async" with_stmt + +An *asynchronous context manager* is a *context manager* that is able +to suspend execution in its *enter* and *exit* methods. + +The following code: + + async with EXPRESSION as TARGET: + SUITE + +is semantically equivalent to: + + manager = (EXPRESSION) + aenter = type(manager).__aenter__ + aexit = type(manager).__aexit__ + value = await aenter(manager) + hit_except = False + + try: + TARGET = value + SUITE + except: + hit_except = True + if not await aexit(manager, *sys.exc_info()): + raise + finally: + if not hit_except: + await aexit(manager, None, None, None) + +See also "__aenter__()" and "__aexit__()" for details. + +It is a "SyntaxError" to use an "async with" statement outside the +body of a coroutine function. + +See also: + + **PEP 492** - Coroutines with async and await syntax + The proposal that made coroutines a proper standalone concept in + Python, and added supporting syntax. +''', + 'atom-identifiers': r'''Identifiers (Names) +******************* + +An identifier occurring as an atom is a name. See section Names +(identifiers and keywords) for lexical definition and section Naming +and binding for documentation of naming and binding. + +When the name is bound to an object, evaluation of the atom yields +that object. When a name is not bound, an attempt to evaluate it +raises a "NameError" exception. + + +Private name mangling +===================== + +When an identifier that textually occurs in a class definition begins +with two or more underscore characters and does not end in two or more +underscores, it is considered a *private name* of that class. + +See also: The class specifications. + +More precisely, private names are transformed to a longer form before +code is generated for them. If the transformed name is longer than +255 characters, implementation-defined truncation may happen. + +The transformation is independent of the syntactical context in which +the identifier is used but only the following private identifiers are +mangled: + +* Any name used as the name of a variable that is assigned or read or + any name of an attribute being accessed. + + The "__name__" attribute of nested functions, classes, and type + aliases is however not mangled. + +* The name of imported modules, e.g., "__spam" in "import __spam". If + the module is part of a package (i.e., its name contains a dot), the + name is *not* mangled, e.g., the "__foo" in "import __foo.bar" is + not mangled. + +* The name of an imported member, e.g., "__f" in "from spam import + __f". + +The transformation rule is defined as follows: + +* The class name, with leading underscores removed and a single + leading underscore inserted, is inserted in front of the identifier, + e.g., the identifier "__spam" occurring in a class named "Foo", + "_Foo" or "__Foo" is transformed to "_Foo__spam". + +* If the class name consists only of underscores, the transformation + is the identity, e.g., the identifier "__spam" occurring in a class + named "_" or "__" is left as is. +''', + 'atom-literals': r'''Literals +******** + +Python supports string and bytes literals and various numeric +literals: + + literal: strings | NUMBER + +Evaluation of a literal yields an object of the given type (string, +bytes, integer, floating-point number, complex number) with the given +value. The value may be approximated in the case of floating-point +and imaginary (complex) literals. See section Literals for details. +See section String literal concatenation for details on "strings". + +All literals correspond to immutable data types, and hence the +object’s identity is less important than its value. Multiple +evaluations of literals with the same value (either the same +occurrence in the program text or a different occurrence) may obtain +the same object or a different object with the same value. + + +String literal concatenation +============================ + +Multiple adjacent string or bytes literals (delimited by whitespace), +possibly using different quoting conventions, are allowed, and their +meaning is the same as their concatenation: + + >>> "hello" 'world' + "helloworld" + +Formally: + + strings: ( STRING | fstring)+ | tstring+ + +This feature is defined at the syntactical level, so it only works +with literals. To concatenate string expressions at run time, the ‘+’ +operator may be used: + + >>> greeting = "Hello" + >>> space = " " + >>> name = "Blaise" + >>> print(greeting + space + name) # not: print(greeting space name) + Hello Blaise + +Literal concatenation can freely mix raw strings, triple-quoted +strings, and formatted string literals. For example: + + >>> "Hello" r', ' f"{name}!" + "Hello, Blaise!" + +This feature can be used to reduce the number of backslashes needed, +to split long strings conveniently across long lines, or even to add +comments to parts of strings. For example: + + re.compile("[A-Za-z_]" # letter or underscore + "[A-Za-z0-9_]*" # letter, digit or underscore + ) + +However, bytes literals may only be combined with other byte literals; +not with string literals of any kind. Also, template string literals +may only be combined with other template string literals: + + >>> t"Hello" t"{name}!" + Template(strings=('Hello', '!'), interpolations=(...)) +''', + 'attribute-access': r'''Customizing attribute access +**************************** + +The following methods can be defined to customize the meaning of +attribute access (use of, assignment to, or deletion of "x.name") for +class instances. + +object.__getattr__(self, name) + + Called when the default attribute access fails with an + "AttributeError" (either "__getattribute__()" raises an + "AttributeError" because *name* is not an instance attribute or an + attribute in the class tree for "self"; or "__get__()" of a *name* + property raises "AttributeError"). This method should either + return the (computed) attribute value or raise an "AttributeError" + exception. The "object" class itself does not provide this method. + + Note that if the attribute is found through the normal mechanism, + "__getattr__()" is not called. (This is an intentional asymmetry + between "__getattr__()" and "__setattr__()".) This is done both for + efficiency reasons and because otherwise "__getattr__()" would have + no way to access other attributes of the instance. Note that at + least for instance variables, you can take total control by not + inserting any values in the instance attribute dictionary (but + instead inserting them in another object). See the + "__getattribute__()" method below for a way to actually get total + control over attribute access. + +object.__getattribute__(self, name) + + Called unconditionally to implement attribute accesses for + instances of the class. If the class also defines "__getattr__()", + the latter will not be called unless "__getattribute__()" either + calls it explicitly or raises an "AttributeError". This method + should return the (computed) attribute value or raise an + "AttributeError" exception. In order to avoid infinite recursion in + this method, its implementation should always call the base class + method with the same name to access any attributes it needs, for + example, "object.__getattribute__(self, name)". + + Note: + + This method may still be bypassed when looking up special methods + as the result of implicit invocation via language syntax or + built-in functions. See Special method lookup. + + For certain sensitive attribute accesses, raises an auditing event + "object.__getattr__" with arguments "obj" and "name". + +object.__setattr__(self, name, value) + + Called when an attribute assignment is attempted. This is called + instead of the normal mechanism (i.e. store the value in the + instance dictionary). *name* is the attribute name, *value* is the + value to be assigned to it. + + If "__setattr__()" wants to assign to an instance attribute, it + should call the base class method with the same name, for example, + "object.__setattr__(self, name, value)". + + For certain sensitive attribute assignments, raises an auditing + event "object.__setattr__" with arguments "obj", "name", "value". + +object.__delattr__(self, name) + + Like "__setattr__()" but for attribute deletion instead of + assignment. This should only be implemented if "del obj.name" is + meaningful for the object. + + For certain sensitive attribute deletions, raises an auditing event + "object.__delattr__" with arguments "obj" and "name". + +object.__dir__(self) + + Called when "dir()" is called on the object. An iterable must be + returned. "dir()" converts the returned iterable to a list and + sorts it. + + +Customizing module attribute access +=================================== + +module.__getattr__() +module.__dir__() + +Special names "__getattr__" and "__dir__" can be also used to +customize access to module attributes. The "__getattr__" function at +the module level should accept one argument which is the name of an +attribute and return the computed value or raise an "AttributeError". +If an attribute is not found on a module object through the normal +lookup, i.e. "object.__getattribute__()", then "__getattr__" is +searched in the module "__dict__" before raising an "AttributeError". +If found, it is called with the attribute name and the result is +returned. + +The "__dir__" function should accept no arguments, and return an +iterable of strings that represents the names accessible on module. If +present, this function overrides the standard "dir()" search on a +module. + +module.__class__ + +For a more fine grained customization of the module behavior (setting +attributes, properties, etc.), one can set the "__class__" attribute +of a module object to a subclass of "types.ModuleType". For example: + + import sys + from types import ModuleType + + class VerboseModule(ModuleType): + def __repr__(self): + return f'Verbose {self.__name__}' + + def __setattr__(self, attr, value): + print(f'Setting {attr}...') + super().__setattr__(attr, value) + + sys.modules[__name__].__class__ = VerboseModule + +Note: + + Defining module "__getattr__" and setting module "__class__" only + affect lookups made using the attribute access syntax – directly + accessing the module globals (whether by code within the module, or + via a reference to the module’s globals dictionary) is unaffected. + +Changed in version 3.5: "__class__" module attribute is now writable. + +Added in version 3.7: "__getattr__" and "__dir__" module attributes. + +See also: + + **PEP 562** - Module __getattr__ and __dir__ + Describes the "__getattr__" and "__dir__" functions on modules. + + +Implementing Descriptors +======================== + +The following methods only apply when an instance of the class +containing the method (a so-called *descriptor* class) appears in an +*owner* class (the descriptor must be in either the owner’s class +dictionary or in the class dictionary for one of its parents). In the +examples below, “the attribute” refers to the attribute whose name is +the key of the property in the owner class’ "__dict__". The "object" +class itself does not implement any of these protocols. + +object.__get__(self, instance, owner=None) + + Called to get the attribute of the owner class (class attribute + access) or of an instance of that class (instance attribute + access). The optional *owner* argument is the owner class, while + *instance* is the instance that the attribute was accessed through, + or "None" when the attribute is accessed through the *owner*. + + This method should return the computed attribute value or raise an + "AttributeError" exception. + + **PEP 252** specifies that "__get__()" is callable with one or two + arguments. Python’s own built-in descriptors support this + specification; however, it is likely that some third-party tools + have descriptors that require both arguments. Python’s own + "__getattribute__()" implementation always passes in both arguments + whether they are required or not. + +object.__set__(self, instance, value) + + Called to set the attribute on an instance *instance* of the owner + class to a new value, *value*. + + Note, adding "__set__()" or "__delete__()" changes the kind of + descriptor to a “data descriptor”. See Invoking Descriptors for + more details. + +object.__delete__(self, instance) + + Called to delete the attribute on an instance *instance* of the + owner class. + +Instances of descriptors may also have the "__objclass__" attribute +present: + +object.__objclass__ + + The attribute "__objclass__" is interpreted by the "inspect" module + as specifying the class where this object was defined (setting this + appropriately can assist in runtime introspection of dynamic class + attributes). For callables, it may indicate that an instance of the + given type (or a subclass) is expected or required as the first + positional argument (for example, CPython sets this attribute for + unbound methods that are implemented in C). + + +Invoking Descriptors +==================== + +In general, a descriptor is an object attribute with “binding +behavior”, one whose attribute access has been overridden by methods +in the descriptor protocol: "__get__()", "__set__()", and +"__delete__()". If any of those methods are defined for an object, it +is said to be a descriptor. + +The default behavior for attribute access is to get, set, or delete +the attribute from an object’s dictionary. For instance, "a.x" has a +lookup chain starting with "a.__dict__['x']", then +"type(a).__dict__['x']", and continuing through the base classes of +"type(a)" excluding metaclasses. + +However, if the looked-up value is an object defining one of the +descriptor methods, then Python may override the default behavior and +invoke the descriptor method instead. Where this occurs in the +precedence chain depends on which descriptor methods were defined and +how they were called. + +The starting point for descriptor invocation is a binding, "a.x". How +the arguments are assembled depends on "a": + +Direct Call + The simplest and least common call is when user code directly + invokes a descriptor method: "x.__get__(a)". + +Instance Binding + If binding to an object instance, "a.x" is transformed into the + call: "type(a).__dict__['x'].__get__(a, type(a))". + +Class Binding + If binding to a class, "A.x" is transformed into the call: + "A.__dict__['x'].__get__(None, A)". + +Super Binding + A dotted lookup such as "super(A, a).x" searches + "a.__class__.__mro__" for a base class "B" following "A" and then + returns "B.__dict__['x'].__get__(a, A)". If not a descriptor, "x" + is returned unchanged. + +For instance bindings, the precedence of descriptor invocation depends +on which descriptor methods are defined. A descriptor can define any +combination of "__get__()", "__set__()" and "__delete__()". If it +does not define "__get__()", then accessing the attribute will return +the descriptor object itself unless there is a value in the object’s +instance dictionary. If the descriptor defines "__set__()" and/or +"__delete__()", it is a data descriptor; if it defines neither, it is +a non-data descriptor. Normally, data descriptors define both +"__get__()" and "__set__()", while non-data descriptors have just the +"__get__()" method. Data descriptors with "__get__()" and "__set__()" +(and/or "__delete__()") defined always override a redefinition in an +instance dictionary. In contrast, non-data descriptors can be +overridden by instances. + +Python methods (including those decorated with "@staticmethod" and +"@classmethod") are implemented as non-data descriptors. Accordingly, +instances can redefine and override methods. This allows individual +instances to acquire behaviors that differ from other instances of the +same class. + +The "property()" function is implemented as a data descriptor. +Accordingly, instances cannot override the behavior of a property. + + +__slots__ +========= + +*__slots__* allow us to explicitly declare data members (like +properties) and deny the creation of "__dict__" and *__weakref__* +(unless explicitly declared in *__slots__* or available in a parent.) + +The space saved over using "__dict__" can be significant. Attribute +lookup speed can be significantly improved as well. + +object.__slots__ + + This class variable can be assigned a string, iterable, or sequence + of strings with variable names used by instances. *__slots__* + reserves space for the declared variables and prevents the + automatic creation of "__dict__" and *__weakref__* for each + instance. + +Notes on using *__slots__*: + +* When inheriting from a class without *__slots__*, the "__dict__" and + *__weakref__* attribute of the instances will always be accessible. + +* Without a "__dict__" variable, instances cannot be assigned new + variables not listed in the *__slots__* definition. Attempts to + assign to an unlisted variable name raises "AttributeError". If + dynamic assignment of new variables is desired, then add + "'__dict__'" to the sequence of strings in the *__slots__* + declaration. + +* Without a *__weakref__* variable for each instance, classes defining + *__slots__* do not support "weak references" to its instances. If + weak reference support is needed, then add "'__weakref__'" to the + sequence of strings in the *__slots__* declaration. + +* *__slots__* are implemented at the class level by creating + descriptors for each variable name. As a result, class attributes + cannot be used to set default values for instance variables defined + by *__slots__*; otherwise, the class attribute would overwrite the + descriptor assignment. + +* The action of a *__slots__* declaration is not limited to the class + where it is defined. *__slots__* declared in parents are available + in child classes. However, instances of a child subclass will get a + "__dict__" and *__weakref__* unless the subclass also defines + *__slots__* (which should only contain names of any *additional* + slots). + +* If a class defines a slot also defined in a base class, the instance + variable defined by the base class slot is inaccessible (except by + retrieving its descriptor directly from the base class). This + renders the meaning of the program undefined. In the future, a + check may be added to prevent this. + +* "TypeError" will be raised if nonempty *__slots__* are defined for a + class derived from a ""variable-length" built-in type" such as + "int", "bytes", and "tuple". + +* Any non-string *iterable* may be assigned to *__slots__*. + +* If a "dictionary" is used to assign *__slots__*, the dictionary keys + will be used as the slot names. The values of the dictionary can be + used to provide per-attribute docstrings that will be recognised by + "inspect.getdoc()" and displayed in the output of "help()". + +* "__class__" assignment works only if both classes have the same + *__slots__*. + +* Multiple inheritance with multiple slotted parent classes can be + used, but only one parent is allowed to have attributes created by + slots (the other bases must have empty slot layouts) - violations + raise "TypeError". + +* If an *iterator* is used for *__slots__* then a *descriptor* is + created for each of the iterator’s values. However, the *__slots__* + attribute will be an empty iterator. +''', + 'attribute-references': r'''Attribute references +******************** + +An attribute reference is a primary followed by a period and a name: + + attributeref: primary "." identifier + +The primary must evaluate to an object of a type that supports +attribute references, which most objects do. This object is then +asked to produce the attribute whose name is the identifier. The type +and value produced is determined by the object. Multiple evaluations +of the same attribute reference may yield different objects. + +This production can be customized by overriding the +"__getattribute__()" method or the "__getattr__()" method. The +"__getattribute__()" method is called first and either returns a value +or raises "AttributeError" if the attribute is not available. + +If an "AttributeError" is raised and the object has a "__getattr__()" +method, that method is called as a fallback. +''', + 'augassign': r'''Augmented assignment statements +******************************* + +Augmented assignment is the combination, in a single statement, of a +binary operation and an assignment statement: + + augmented_assignment_stmt: augtarget augop (expression_list | yield_expression) + augtarget: identifier | attributeref | subscription | slicing + augop: "+=" | "-=" | "*=" | "@=" | "/=" | "//=" | "%=" | "**=" + | ">>=" | "<<=" | "&=" | "^=" | "|=" + +(See section Primaries for the syntax definitions of the last three +symbols.) + +An augmented assignment evaluates the target (which, unlike normal +assignment statements, cannot be an unpacking) and the expression +list, performs the binary operation specific to the type of assignment +on the two operands, and assigns the result to the original target. +The target is only evaluated once. + +An augmented assignment statement like "x += 1" can be rewritten as "x += x + 1" to achieve a similar, but not exactly equal effect. In the +augmented version, "x" is only evaluated once. Also, when possible, +the actual operation is performed *in-place*, meaning that rather than +creating a new object and assigning that to the target, the old object +is modified instead. + +Unlike normal assignments, augmented assignments evaluate the left- +hand side *before* evaluating the right-hand side. For example, "a[i] ++= f(x)" first looks-up "a[i]", then it evaluates "f(x)" and performs +the addition, and lastly, it writes the result back to "a[i]". + +With the exception of assigning to tuples and multiple targets in a +single statement, the assignment done by augmented assignment +statements is handled the same way as normal assignments. Similarly, +with the exception of the possible *in-place* behavior, the binary +operation performed by augmented assignment is the same as the normal +binary operations. + +For targets which are attribute references, the same caveat about +class and instance attributes applies as for regular assignments. +''', + 'await': r'''Await expression +**************** + +Suspend the execution of *coroutine* on an *awaitable* object. Can +only be used inside a *coroutine function*. + + await_expr: "await" primary + +Added in version 3.5. +''', + 'binary': r'''Binary arithmetic operations +**************************** + +The binary arithmetic operations have the conventional priority +levels. Note that some of these operations also apply to certain non- +numeric types. Apart from the power operator, there are only two +levels, one for multiplicative operators and one for additive +operators: + + m_expr: u_expr | m_expr "*" u_expr | m_expr "@" m_expr | + m_expr "//" u_expr | m_expr "/" u_expr | + m_expr "%" u_expr + a_expr: m_expr | a_expr "+" m_expr | a_expr "-" m_expr + +The "*" (multiplication) operator yields the product of its arguments. +The arguments must either both be numbers, or one argument must be an +integer and the other must be a sequence. In the former case, the +numbers are converted to a common real type and then multiplied +together. In the latter case, sequence repetition is performed; a +negative repetition factor yields an empty sequence. + +This operation can be customized using the special "__mul__()" and +"__rmul__()" methods. + +Changed in version 3.14: If only one operand is a complex number, the +other operand is converted to a floating-point number. + +The "@" (at) operator is intended to be used for matrix +multiplication. No builtin Python types implement this operator. + +This operation can be customized using the special "__matmul__()" and +"__rmatmul__()" methods. + +Added in version 3.5. + +The "/" (division) and "//" (floor division) operators yield the +quotient of their arguments. The numeric arguments are first +converted to a common type. Division of integers yields a float, while +floor division of integers results in an integer; the result is that +of mathematical division with the ‘floor’ function applied to the +result. Division by zero raises the "ZeroDivisionError" exception. + +The division operation can be customized using the special +"__truediv__()" and "__rtruediv__()" methods. The floor division +operation can be customized using the special "__floordiv__()" and +"__rfloordiv__()" methods. + +The "%" (modulo) operator yields the remainder from the division of +the first argument by the second. The numeric arguments are first +converted to a common type. A zero right argument raises the +"ZeroDivisionError" exception. The arguments may be floating-point +numbers, e.g., "3.14%0.7" equals "0.34" (since "3.14" equals "4*0.7 + +0.34".) The modulo operator always yields a result with the same sign +as its second operand (or zero); the absolute value of the result is +strictly smaller than the absolute value of the second operand [1]. + +The floor division and modulo operators are connected by the following +identity: "x == (x//y)*y + (x%y)". Floor division and modulo are also +connected with the built-in function "divmod()": "divmod(x, y) == +(x//y, x%y)". [2]. + +In addition to performing the modulo operation on numbers, the "%" +operator is also overloaded by string objects to perform old-style +string formatting (also known as interpolation). The syntax for +string formatting is described in the Python Library Reference, +section printf-style String Formatting. + +The *modulo* operation can be customized using the special "__mod__()" +and "__rmod__()" methods. + +The floor division operator, the modulo operator, and the "divmod()" +function are not defined for complex numbers. Instead, convert to a +floating-point number using the "abs()" function if appropriate. + +The "+" (addition) operator yields the sum of its arguments. The +arguments must either both be numbers or both be sequences of the same +type. In the former case, the numbers are converted to a common real +type and then added together. In the latter case, the sequences are +concatenated. + +This operation can be customized using the special "__add__()" and +"__radd__()" methods. + +Changed in version 3.14: If only one operand is a complex number, the +other operand is converted to a floating-point number. + +The "-" (subtraction) operator yields the difference of its arguments. +The numeric arguments are first converted to a common real type. + +This operation can be customized using the special "__sub__()" and +"__rsub__()" methods. + +Changed in version 3.14: If only one operand is a complex number, the +other operand is converted to a floating-point number. +''', + 'bitwise': r'''Binary bitwise operations +************************* + +Each of the three bitwise operations has a different priority level: + + and_expr: shift_expr | and_expr "&" shift_expr + xor_expr: and_expr | xor_expr "^" and_expr + or_expr: xor_expr | or_expr "|" xor_expr + +The "&" operator yields the bitwise AND of its arguments, which must +be integers or one of them must be a custom object overriding +"__and__()" or "__rand__()" special methods. + +The "^" operator yields the bitwise XOR (exclusive OR) of its +arguments, which must be integers or one of them must be a custom +object overriding "__xor__()" or "__rxor__()" special methods. + +The "|" operator yields the bitwise (inclusive) OR of its arguments, +which must be integers or one of them must be a custom object +overriding "__or__()" or "__ror__()" special methods. +''', + 'bltin-code-objects': r'''Code Objects +************ + +Code objects are used by the implementation to represent “pseudo- +compiled” executable Python code such as a function body. They differ +from function objects because they don’t contain a reference to their +global execution environment. Code objects are returned by the built- +in "compile()" function and can be extracted from function objects +through their "__code__" attribute. See also the "code" module. + +Accessing "__code__" raises an auditing event "object.__getattr__" +with arguments "obj" and ""__code__"". + +A code object can be executed or evaluated by passing it (instead of a +source string) to the "exec()" or "eval()" built-in functions. + +See The standard type hierarchy for more information. +''', + 'bltin-ellipsis-object': r'''The Ellipsis Object +******************* + +This object is commonly used to indicate that something is omitted. It +supports no special operations. There is exactly one ellipsis object, +named "Ellipsis" (a built-in name). "type(Ellipsis)()" produces the +"Ellipsis" singleton. + +It is written as "Ellipsis" or "...". + +In typical use, "..." as the "Ellipsis" object appears in a few +different places, for instance: + +* In type annotations, such as callable arguments or tuple elements. + +* As the body of a function instead of a pass statement. + +* In third-party libraries, such as Numpy’s slicing and striding. + +Python also uses three dots in ways that are not "Ellipsis" objects, +for instance: + +* Doctest’s "ELLIPSIS", as a pattern for missing content. + +* The default Python prompt of the *interactive* shell when partial + input is incomplete. + +Lastly, the Python documentation often uses three dots in conventional +English usage to mean omitted content, even in code examples that also +use them as the "Ellipsis". +''', + 'bltin-null-object': r'''The Null Object +*************** + +This object is returned by functions that don’t explicitly return a +value. It supports no special operations. There is exactly one null +object, named "None" (a built-in name). "type(None)()" produces the +same singleton. + +It is written as "None". +''', + 'bltin-type-objects': r'''Type Objects +************ + +Type objects represent the various object types. An object’s type is +accessed by the built-in function "type()". There are no special +operations on types. The standard module "types" defines names for +all standard built-in types. + +Types are written like this: "". +''', + 'booleans': r'''Boolean operations +****************** + + or_test: and_test | or_test "or" and_test + and_test: not_test | and_test "and" not_test + not_test: comparison | "not" not_test + +In the context of Boolean operations, and also when expressions are +used by control flow statements, the following values are interpreted +as false: "False", "None", numeric zero of all types, and empty +strings and containers (including strings, tuples, lists, +dictionaries, sets and frozensets). All other values are interpreted +as true. User-defined objects can customize their truth value by +providing a "__bool__()" method. + +The operator "not" yields "True" if its argument is false, "False" +otherwise. + +The expression "x and y" first evaluates *x*; if *x* is false, its +value is returned; otherwise, *y* is evaluated and the resulting value +is returned. + +The expression "x or y" first evaluates *x*; if *x* is true, its value +is returned; otherwise, *y* is evaluated and the resulting value is +returned. + +Note that neither "and" nor "or" restrict the value and type they +return to "False" and "True", but rather return the last evaluated +argument. This is sometimes useful, e.g., if "s" is a string that +should be replaced by a default value if it is empty, the expression +"s or 'foo'" yields the desired value. Because "not" has to create a +new value, it returns a boolean value regardless of the type of its +argument (for example, "not 'foo'" produces "False" rather than "''".) +''', + 'break': r'''The "break" statement +********************* + + break_stmt: "break" + +"break" may only occur syntactically nested in a "for" or "while" +loop, but not nested in a function or class definition within that +loop. + +It terminates the nearest enclosing loop, skipping the optional "else" +clause if the loop has one. + +If a "for" loop is terminated by "break", the loop control target +keeps its current value. + +When "break" passes control out of a "try" statement with a "finally" +clause, that "finally" clause is executed before really leaving the +loop. +''', + 'callable-types': r'''Emulating callable objects +************************** + +object.__call__(self[, args...]) + + Called when the instance is “called” as a function; if this method + is defined, "x(arg1, arg2, ...)" roughly translates to + "type(x).__call__(x, arg1, ...)". The "object" class itself does + not provide this method. +''', + 'calls': r'''Calls +***** + +A call calls a callable object (e.g., a *function*) with a possibly +empty series of *arguments*: + + call: primary "(" [argument_list [","] | comprehension] ")" + argument_list: positional_arguments ["," starred_and_keywords] + ["," keywords_arguments] + | starred_and_keywords ["," keywords_arguments] + | keywords_arguments + positional_arguments: positional_item ("," positional_item)* + positional_item: assignment_expression | "*" expression + starred_and_keywords: ("*" expression | keyword_item) + ("," "*" expression | "," keyword_item)* + keywords_arguments: (keyword_item | "**" expression) + ("," keyword_item | "," "**" expression)* + keyword_item: identifier "=" expression + +An optional trailing comma may be present after the positional and +keyword arguments but does not affect the semantics. + +The primary must evaluate to a callable object (user-defined +functions, built-in functions, methods of built-in objects, class +objects, methods of class instances, and all objects having a +"__call__()" method are callable). All argument expressions are +evaluated before the call is attempted. Please refer to section +Function definitions for the syntax of formal *parameter* lists. + +If keyword arguments are present, they are first converted to +positional arguments, as follows. First, a list of unfilled slots is +created for the formal parameters. If there are N positional +arguments, they are placed in the first N slots. Next, for each +keyword argument, the identifier is used to determine the +corresponding slot (if the identifier is the same as the first formal +parameter name, the first slot is used, and so on). If the slot is +already filled, a "TypeError" exception is raised. Otherwise, the +argument is placed in the slot, filling it (even if the expression is +"None", it fills the slot). When all arguments have been processed, +the slots that are still unfilled are filled with the corresponding +default value from the function definition. (Default values are +calculated, once, when the function is defined; thus, a mutable object +such as a list or dictionary used as default value will be shared by +all calls that don’t specify an argument value for the corresponding +slot; this should usually be avoided.) If there are any unfilled +slots for which no default value is specified, a "TypeError" exception +is raised. Otherwise, the list of filled slots is used as the +argument list for the call. + +**CPython implementation detail:** An implementation may provide +built-in functions whose positional parameters do not have names, even +if they are ‘named’ for the purpose of documentation, and which +therefore cannot be supplied by keyword. In CPython, this is the case +for functions implemented in C that use "PyArg_ParseTuple()" to parse +their arguments. + +If there are more positional arguments than there are formal parameter +slots, a "TypeError" exception is raised, unless a formal parameter +using the syntax "*identifier" is present; in this case, that formal +parameter receives a tuple containing the excess positional arguments +(or an empty tuple if there were no excess positional arguments). + +If any keyword argument does not correspond to a formal parameter +name, a "TypeError" exception is raised, unless a formal parameter +using the syntax "**identifier" is present; in this case, that formal +parameter receives a dictionary containing the excess keyword +arguments (using the keywords as keys and the argument values as +corresponding values), or a (new) empty dictionary if there were no +excess keyword arguments. + +If the syntax "*expression" appears in the function call, "expression" +must evaluate to an *iterable*. Elements from these iterables are +treated as if they were additional positional arguments. For the call +"f(x1, x2, *y, x3, x4)", if *y* evaluates to a sequence *y1*, …, *yM*, +this is equivalent to a call with M+4 positional arguments *x1*, *x2*, +*y1*, …, *yM*, *x3*, *x4*. + +A consequence of this is that although the "*expression" syntax may +appear *after* explicit keyword arguments, it is processed *before* +the keyword arguments (and any "**expression" arguments – see below). +So: + + >>> def f(a, b): + ... print(a, b) + ... + >>> f(b=1, *(2,)) + 2 1 + >>> f(a=1, *(2,)) + Traceback (most recent call last): + File "", line 1, in + TypeError: f() got multiple values for keyword argument 'a' + >>> f(1, *(2,)) + 1 2 + +It is unusual for both keyword arguments and the "*expression" syntax +to be used in the same call, so in practice this confusion does not +often arise. + +If the syntax "**expression" appears in the function call, +"expression" must evaluate to a *mapping*, the contents of which are +treated as additional keyword arguments. If a parameter matching a key +has already been given a value (by an explicit keyword argument, or +from another unpacking), a "TypeError" exception is raised. + +When "**expression" is used, each key in this mapping must be a +string. Each value from the mapping is assigned to the first formal +parameter eligible for keyword assignment whose name is equal to the +key. A key need not be a Python identifier (e.g. ""max-temp °F"" is +acceptable, although it will not match any formal parameter that could +be declared). If there is no match to a formal parameter the key-value +pair is collected by the "**" parameter, if there is one, or if there +is not, a "TypeError" exception is raised. + +Formal parameters using the syntax "*identifier" or "**identifier" +cannot be used as positional argument slots or as keyword argument +names. + +Changed in version 3.5: Function calls accept any number of "*" and +"**" unpackings, positional arguments may follow iterable unpackings +("*"), and keyword arguments may follow dictionary unpackings ("**"). +Originally proposed by **PEP 448**. + +A call always returns some value, possibly "None", unless it raises an +exception. How this value is computed depends on the type of the +callable object. + +If it is— + +a user-defined function: + The code block for the function is executed, passing it the + argument list. The first thing the code block will do is bind the + formal parameters to the arguments; this is described in section + Function definitions. When the code block executes a "return" + statement, this specifies the return value of the function call. + If execution reaches the end of the code block without executing a + "return" statement, the return value is "None". + +a built-in function or method: + The result is up to the interpreter; see Built-in Functions for the + descriptions of built-in functions and methods. + +a class object: + A new instance of that class is returned. + +a class instance method: + The corresponding user-defined function is called, with an argument + list that is one longer than the argument list of the call: the + instance becomes the first argument. + +a class instance: + The class must define a "__call__()" method; the effect is then the + same as if that method was called. +''', + 'class': r'''Class definitions +***************** + +A class definition defines a class object (see section The standard +type hierarchy): + + classdef: [decorators] "class" classname [type_params] [inheritance] ":" suite + inheritance: "(" [argument_list] ")" + classname: identifier + +A class definition is an executable statement. The inheritance list +usually gives a list of base classes (see Metaclasses for more +advanced uses), so each item in the list should evaluate to a class +object which allows subclassing. Classes without an inheritance list +inherit, by default, from the base class "object"; hence, + + class Foo: + pass + +is equivalent to + + class Foo(object): + pass + +The class’s suite is then executed in a new execution frame (see +Naming and binding), using a newly created local namespace and the +original global namespace. (Usually, the suite contains mostly +function definitions.) When the class’s suite finishes execution, its +execution frame is discarded but its local namespace is saved. [5] A +class object is then created using the inheritance list for the base +classes and the saved local namespace for the attribute dictionary. +The class name is bound to this class object in the original local +namespace. + +The order in which attributes are defined in the class body is +preserved in the new class’s "__dict__". Note that this is reliable +only right after the class is created and only for classes that were +defined using the definition syntax. + +Class creation can be customized heavily using metaclasses. + +Classes can also be decorated: just like when decorating functions, + + @f1(arg) + @f2 + class Foo: pass + +is roughly equivalent to + + class Foo: pass + Foo = f1(arg)(f2(Foo)) + +The evaluation rules for the decorator expressions are the same as for +function decorators. The result is then bound to the class name. + +Changed in version 3.9: Classes may be decorated with any valid +"assignment_expression". Previously, the grammar was much more +restrictive; see **PEP 614** for details. + +A list of type parameters may be given in square brackets immediately +after the class’s name. This indicates to static type checkers that +the class is generic. At runtime, the type parameters can be retrieved +from the class’s "__type_params__" attribute. See Generic classes for +more. + +Changed in version 3.12: Type parameter lists are new in Python 3.12. + +**Programmer’s note:** Variables defined in the class definition are +class attributes; they are shared by instances. Instance attributes +can be set in a method with "self.name = value". Both class and +instance attributes are accessible through the notation “"self.name"”, +and an instance attribute hides a class attribute with the same name +when accessed in this way. Class attributes can be used as defaults +for instance attributes, but using mutable values there can lead to +unexpected results. Descriptors can be used to create instance +variables with different implementation details. + +See also: + + **PEP 3115** - Metaclasses in Python 3000 + The proposal that changed the declaration of metaclasses to the + current syntax, and the semantics for how classes with + metaclasses are constructed. + + **PEP 3129** - Class Decorators + The proposal that added class decorators. Function and method + decorators were introduced in **PEP 318**. +''', + 'comparisons': r'''Comparisons +*********** + +Unlike C, all comparison operations in Python have the same priority, +which is lower than that of any arithmetic, shifting or bitwise +operation. Also unlike C, expressions like "a < b < c" have the +interpretation that is conventional in mathematics: + + comparison: or_expr (comp_operator or_expr)* + comp_operator: "<" | ">" | "==" | ">=" | "<=" | "!=" + | "is" ["not"] | ["not"] "in" + +Comparisons yield boolean values: "True" or "False". Custom *rich +comparison methods* may return non-boolean values. In this case Python +will call "bool()" on such value in boolean contexts. + +Comparisons can be chained arbitrarily, e.g., "x < y <= z" is +equivalent to "x < y and y <= z", except that "y" is evaluated only +once (but in both cases "z" is not evaluated at all when "x < y" is +found to be false). + +Formally, if *a*, *b*, *c*, …, *y*, *z* are expressions and *op1*, +*op2*, …, *opN* are comparison operators, then "a op1 b op2 c ... y +opN z" is equivalent to "a op1 b and b op2 c and ... y opN z", except +that each expression is evaluated at most once. + +Note that "a op1 b op2 c" doesn’t imply any kind of comparison between +*a* and *c*, so that, e.g., "x < y > z" is perfectly legal (though +perhaps not pretty). + + +Value comparisons +================= + +The operators "<", ">", "==", ">=", "<=", and "!=" compare the values +of two objects. The objects do not need to have the same type. + +Chapter Objects, values and types states that objects have a value (in +addition to type and identity). The value of an object is a rather +abstract notion in Python: For example, there is no canonical access +method for an object’s value. Also, there is no requirement that the +value of an object should be constructed in a particular way, e.g. +comprised of all its data attributes. Comparison operators implement a +particular notion of what the value of an object is. One can think of +them as defining the value of an object indirectly, by means of their +comparison implementation. + +Because all types are (direct or indirect) subtypes of "object", they +inherit the default comparison behavior from "object". Types can +customize their comparison behavior by implementing *rich comparison +methods* like "__lt__()", described in Basic customization. + +The default behavior for equality comparison ("==" and "!=") is based +on the identity of the objects. Hence, equality comparison of +instances with the same identity results in equality, and equality +comparison of instances with different identities results in +inequality. A motivation for this default behavior is the desire that +all objects should be reflexive (i.e. "x is y" implies "x == y"). + +A default order comparison ("<", ">", "<=", and ">=") is not provided; +an attempt raises "TypeError". A motivation for this default behavior +is the lack of a similar invariant as for equality. + +The behavior of the default equality comparison, that instances with +different identities are always unequal, may be in contrast to what +types will need that have a sensible definition of object value and +value-based equality. Such types will need to customize their +comparison behavior, and in fact, a number of built-in types have done +that. + +The following list describes the comparison behavior of the most +important built-in types. + +* Numbers of built-in numeric types (Numeric Types — int, float, + complex) and of the standard library types "fractions.Fraction" and + "decimal.Decimal" can be compared within and across their types, + with the restriction that complex numbers do not support order + comparison. Within the limits of the types involved, they compare + mathematically (algorithmically) correct without loss of precision. + + The not-a-number values "float('NaN')" and "decimal.Decimal('NaN')" + are special. Any ordered comparison of a number to a not-a-number + value is false. A counter-intuitive implication is that not-a-number + values are not equal to themselves. For example, if "x = + float('NaN')", "3 < x", "x < 3" and "x == x" are all false, while "x + != x" is true. This behavior is compliant with IEEE 754. + +* "None" and "NotImplemented" are singletons. **PEP 8** advises that + comparisons for singletons should always be done with "is" or "is + not", never the equality operators. + +* Binary sequences (instances of "bytes" or "bytearray") can be + compared within and across their types. They compare + lexicographically using the numeric values of their elements. + +* Strings (instances of "str") compare lexicographically using the + numerical Unicode code points (the result of the built-in function + "ord()") of their characters. [3] + + Strings and binary sequences cannot be directly compared. + +* Sequences (instances of "tuple", "list", or "range") can be compared + only within each of their types, with the restriction that ranges do + not support order comparison. Equality comparison across these + types results in inequality, and ordering comparison across these + types raises "TypeError". + + Sequences compare lexicographically using comparison of + corresponding elements. The built-in containers typically assume + identical objects are equal to themselves. That lets them bypass + equality tests for identical objects to improve performance and to + maintain their internal invariants. + + Lexicographical comparison between built-in collections works as + follows: + + * For two collections to compare equal, they must be of the same + type, have the same length, and each pair of corresponding + elements must compare equal (for example, "[1,2] == (1,2)" is + false because the type is not the same). + + * Collections that support order comparison are ordered the same as + their first unequal elements (for example, "[1,2,x] <= [1,2,y]" + has the same value as "x <= y"). If a corresponding element does + not exist, the shorter collection is ordered first (for example, + "[1,2] < [1,2,3]" is true). + +* Mappings (instances of "dict") compare equal if and only if they + have equal "(key, value)" pairs. Equality comparison of the keys and + values enforces reflexivity. + + Order comparisons ("<", ">", "<=", and ">=") raise "TypeError". + +* Sets (instances of "set" or "frozenset") can be compared within and + across their types. + + They define order comparison operators to mean subset and superset + tests. Those relations do not define total orderings (for example, + the two sets "{1,2}" and "{2,3}" are not equal, nor subsets of one + another, nor supersets of one another). Accordingly, sets are not + appropriate arguments for functions which depend on total ordering + (for example, "min()", "max()", and "sorted()" produce undefined + results given a list of sets as inputs). + + Comparison of sets enforces reflexivity of its elements. + +* Most other built-in types have no comparison methods implemented, so + they inherit the default comparison behavior. + +User-defined classes that customize their comparison behavior should +follow some consistency rules, if possible: + +* Equality comparison should be reflexive. In other words, identical + objects should compare equal: + + "x is y" implies "x == y" + +* Comparison should be symmetric. In other words, the following + expressions should have the same result: + + "x == y" and "y == x" + + "x != y" and "y != x" + + "x < y" and "y > x" + + "x <= y" and "y >= x" + +* Comparison should be transitive. The following (non-exhaustive) + examples illustrate that: + + "x > y and y > z" implies "x > z" + + "x < y and y <= z" implies "x < z" + +* Inverse comparison should result in the boolean negation. In other + words, the following expressions should have the same result: + + "x == y" and "not x != y" + + "x < y" and "not x >= y" (for total ordering) + + "x > y" and "not x <= y" (for total ordering) + + The last two expressions apply to totally ordered collections (e.g. + to sequences, but not to sets or mappings). See also the + "total_ordering()" decorator. + +* The "hash()" result should be consistent with equality. Objects that + are equal should either have the same hash value, or be marked as + unhashable. + +Python does not enforce these consistency rules. In fact, the +not-a-number values are an example for not following these rules. + + +Membership test operations +========================== + +The operators "in" and "not in" test for membership. "x in s" +evaluates to "True" if *x* is a member of *s*, and "False" otherwise. +"x not in s" returns the negation of "x in s". All built-in sequences +and set types support this as well as dictionary, for which "in" tests +whether the dictionary has a given key. For container types such as +list, tuple, set, frozenset, dict, or collections.deque, the +expression "x in y" is equivalent to "any(x is e or x == e for e in +y)". + +For the string and bytes types, "x in y" is "True" if and only if *x* +is a substring of *y*. An equivalent test is "y.find(x) != -1". +Empty strings are always considered to be a substring of any other +string, so """ in "abc"" will return "True". + +For user-defined classes which define the "__contains__()" method, "x +in y" returns "True" if "y.__contains__(x)" returns a true value, and +"False" otherwise. + +For user-defined classes which do not define "__contains__()" but do +define "__iter__()", "x in y" is "True" if some value "z", for which +the expression "x is z or x == z" is true, is produced while iterating +over "y". If an exception is raised during the iteration, it is as if +"in" raised that exception. + +Lastly, the old-style iteration protocol is tried: if a class defines +"__getitem__()", "x in y" is "True" if and only if there is a non- +negative integer index *i* such that "x is y[i] or x == y[i]", and no +lower integer index raises the "IndexError" exception. (If any other +exception is raised, it is as if "in" raised that exception). + +The operator "not in" is defined to have the inverse truth value of +"in". + + +Identity comparisons +==================== + +The operators "is" and "is not" test for an object’s identity: "x is +y" is true if and only if *x* and *y* are the same object. An +Object’s identity is determined using the "id()" function. "x is not +y" yields the inverse truth value. [4] +''', + 'compound': r'''Compound statements +******************* + +Compound statements contain (groups of) other statements; they affect +or control the execution of those other statements in some way. In +general, compound statements span multiple lines, although in simple +incarnations a whole compound statement may be contained in one line. + +The "if", "while" and "for" statements implement traditional control +flow constructs. "try" specifies exception handlers and/or cleanup +code for a group of statements, while the "with" statement allows the +execution of initialization and finalization code around a block of +code. Function and class definitions are also syntactically compound +statements. + +A compound statement consists of one or more ‘clauses.’ A clause +consists of a header and a ‘suite.’ The clause headers of a +particular compound statement are all at the same indentation level. +Each clause header begins with a uniquely identifying keyword and ends +with a colon. A suite is a group of statements controlled by a +clause. A suite can be one or more semicolon-separated simple +statements on the same line as the header, following the header’s +colon, or it can be one or more indented statements on subsequent +lines. Only the latter form of a suite can contain nested compound +statements; the following is illegal, mostly because it wouldn’t be +clear to which "if" clause a following "else" clause would belong: + + if test1: if test2: print(x) + +Also note that the semicolon binds tighter than the colon in this +context, so that in the following example, either all or none of the +"print()" calls are executed: + + if x < y < z: print(x); print(y); print(z) + +Summarizing: + + compound_stmt: if_stmt + | while_stmt + | for_stmt + | try_stmt + | with_stmt + | match_stmt + | funcdef + | classdef + | async_with_stmt + | async_for_stmt + | async_funcdef + suite: stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENT + statement: stmt_list NEWLINE | compound_stmt + stmt_list: simple_stmt (";" simple_stmt)* [";"] + +Note that statements always end in a "NEWLINE" possibly followed by a +"DEDENT". Also note that optional continuation clauses always begin +with a keyword that cannot start a statement, thus there are no +ambiguities (the ‘dangling "else"’ problem is solved in Python by +requiring nested "if" statements to be indented). + +The formatting of the grammar rules in the following sections places +each clause on a separate line for clarity. + + +The "if" statement +================== + +The "if" statement is used for conditional execution: + + if_stmt: "if" assignment_expression ":" suite + ("elif" assignment_expression ":" suite)* + ["else" ":" suite] + +It selects exactly one of the suites by evaluating the expressions one +by one until one is found to be true (see section Boolean operations +for the definition of true and false); then that suite is executed +(and no other part of the "if" statement is executed or evaluated). +If all expressions are false, the suite of the "else" clause, if +present, is executed. + + +The "while" statement +===================== + +The "while" statement is used for repeated execution as long as an +expression is true: + + while_stmt: "while" assignment_expression ":" suite + ["else" ":" suite] + +This repeatedly tests the expression and, if it is true, executes the +first suite; if the expression is false (which may be the first time +it is tested) the suite of the "else" clause, if present, is executed +and the loop terminates. + +A "break" statement executed in the first suite terminates the loop +without executing the "else" clause’s suite. A "continue" statement +executed in the first suite skips the rest of the suite and goes back +to testing the expression. + + +The "for" statement +=================== + +The "for" statement is used to iterate over the elements of a sequence +(such as a string, tuple or list) or other iterable object: + + for_stmt: "for" target_list "in" starred_expression_list ":" suite + ["else" ":" suite] + +The "starred_expression_list" expression is evaluated once; it should +yield an *iterable* object. An *iterator* is created for that +iterable. The first item provided by the iterator is then assigned to +the target list using the standard rules for assignments (see +Assignment statements), and the suite is executed. This repeats for +each item provided by the iterator. When the iterator is exhausted, +the suite in the "else" clause, if present, is executed, and the loop +terminates. + +A "break" statement executed in the first suite terminates the loop +without executing the "else" clause’s suite. A "continue" statement +executed in the first suite skips the rest of the suite and continues +with the next item, or with the "else" clause if there is no next +item. + +The for-loop makes assignments to the variables in the target list. +This overwrites all previous assignments to those variables including +those made in the suite of the for-loop: + + for i in range(10): + print(i) + i = 5 # this will not affect the for-loop + # because i will be overwritten with the next + # index in the range + +Names in the target list are not deleted when the loop is finished, +but if the sequence is empty, they will not have been assigned to at +all by the loop. Hint: the built-in type "range()" represents +immutable arithmetic sequences of integers. For instance, iterating +"range(3)" successively yields 0, 1, and then 2. + +Changed in version 3.11: Starred elements are now allowed in the +expression list. + + +The "try" statement +=================== + +The "try" statement specifies exception handlers and/or cleanup code +for a group of statements: + + try_stmt: try1_stmt | try2_stmt | try3_stmt + try1_stmt: "try" ":" suite + ("except" [expression ["as" identifier]] ":" suite)+ + ["else" ":" suite] + ["finally" ":" suite] + try2_stmt: "try" ":" suite + ("except" "*" expression ["as" identifier] ":" suite)+ + ["else" ":" suite] + ["finally" ":" suite] + try3_stmt: "try" ":" suite + "finally" ":" suite + +Additional information on exceptions can be found in section +Exceptions, and information on using the "raise" statement to generate +exceptions may be found in section The raise statement. + +Changed in version 3.14: Support for optionally dropping grouping +parentheses when using multiple exception types. See **PEP 758**. + + +"except" clause +--------------- + +The "except" clause(s) specify one or more exception handlers. When no +exception occurs in the "try" clause, no exception handler is +executed. When an exception occurs in the "try" suite, a search for an +exception handler is started. This search inspects the "except" +clauses in turn until one is found that matches the exception. An +expression-less "except" clause, if present, must be last; it matches +any exception. + +For an "except" clause with an expression, the expression must +evaluate to an exception type or a tuple of exception types. +Parentheses can be dropped if multiple exception types are provided +and the "as" clause is not used. The raised exception matches an +"except" clause whose expression evaluates to the class or a *non- +virtual base class* of the exception object, or to a tuple that +contains such a class. + +If no "except" clause matches the exception, the search for an +exception handler continues in the surrounding code and on the +invocation stack. [1] + +If the evaluation of an expression in the header of an "except" clause +raises an exception, the original search for a handler is canceled and +a search starts for the new exception in the surrounding code and on +the call stack (it is treated as if the entire "try" statement raised +the exception). + +When a matching "except" clause is found, the exception is assigned to +the target specified after the "as" keyword in that "except" clause, +if present, and the "except" clause’s suite is executed. All "except" +clauses must have an executable block. When the end of this block is +reached, execution continues normally after the entire "try" +statement. (This means that if two nested handlers exist for the same +exception, and the exception occurs in the "try" clause of the inner +handler, the outer handler will not handle the exception.) + +When an exception has been assigned using "as target", it is cleared +at the end of the "except" clause. This is as if + + except E as N: + foo + +was translated to + + except E as N: + try: + foo + finally: + del N + +This means the exception must be assigned to a different name to be +able to refer to it after the "except" clause. Exceptions are cleared +because with the traceback attached to them, they form a reference +cycle with the stack frame, keeping all locals in that frame alive +until the next garbage collection occurs. + +Before an "except" clause’s suite is executed, the exception is stored +in the "sys" module, where it can be accessed from within the body of +the "except" clause by calling "sys.exception()". When leaving an +exception handler, the exception stored in the "sys" module is reset +to its previous value: + + >>> print(sys.exception()) + None + >>> try: + ... raise TypeError + ... except: + ... print(repr(sys.exception())) + ... try: + ... raise ValueError + ... except: + ... print(repr(sys.exception())) + ... print(repr(sys.exception())) + ... + TypeError() + ValueError() + TypeError() + >>> print(sys.exception()) + None + + +"except*" clause +---------------- + +The "except*" clause(s) specify one or more handlers for groups of +exceptions ("BaseExceptionGroup" instances). A "try" statement can +have either "except" or "except*" clauses, but not both. The exception +type for matching is mandatory in the case of "except*", so "except*:" +is a syntax error. The type is interpreted as in the case of "except", +but matching is performed on the exceptions contained in the group +that is being handled. An "TypeError" is raised if a matching type is +a subclass of "BaseExceptionGroup", because that would have ambiguous +semantics. + +When an exception group is raised in the try block, each "except*" +clause splits (see "split()") it into the subgroups of matching and +non-matching exceptions. If the matching subgroup is not empty, it +becomes the handled exception (the value returned from +"sys.exception()") and assigned to the target of the "except*" clause +(if there is one). Then, the body of the "except*" clause executes. If +the non-matching subgroup is not empty, it is processed by the next +"except*" in the same manner. This continues until all exceptions in +the group have been matched, or the last "except*" clause has run. + +After all "except*" clauses execute, the group of unhandled exceptions +is merged with any exceptions that were raised or re-raised from +within "except*" clauses. This merged exception group propagates on.: + + >>> try: + ... raise ExceptionGroup("eg", + ... [ValueError(1), TypeError(2), OSError(3), OSError(4)]) + ... except* TypeError as e: + ... print(f'caught {type(e)} with nested {e.exceptions}') + ... except* OSError as e: + ... print(f'caught {type(e)} with nested {e.exceptions}') + ... + caught with nested (TypeError(2),) + caught with nested (OSError(3), OSError(4)) + + Exception Group Traceback (most recent call last): + | File "", line 2, in + | raise ExceptionGroup("eg", + | [ValueError(1), TypeError(2), OSError(3), OSError(4)]) + | ExceptionGroup: eg (1 sub-exception) + +-+---------------- 1 ---------------- + | ValueError: 1 + +------------------------------------ + +If the exception raised from the "try" block is not an exception group +and its type matches one of the "except*" clauses, it is caught and +wrapped by an exception group with an empty message string. This +ensures that the type of the target "e" is consistently +"BaseExceptionGroup": + + >>> try: + ... raise BlockingIOError + ... except* BlockingIOError as e: + ... print(repr(e)) + ... + ExceptionGroup('', (BlockingIOError())) + +"break", "continue" and "return" cannot appear in an "except*" clause. + + +"else" clause +------------- + +The optional "else" clause is executed if the control flow leaves the +"try" suite, no exception was raised, and no "return", "continue", or +"break" statement was executed. Exceptions in the "else" clause are +not handled by the preceding "except" clauses. + + +"finally" clause +---------------- + +If "finally" is present, it specifies a ‘cleanup’ handler. The "try" +clause is executed, including any "except" and "else" clauses. If an +exception occurs in any of the clauses and is not handled, the +exception is temporarily saved. The "finally" clause is executed. If +there is a saved exception it is re-raised at the end of the "finally" +clause. If the "finally" clause raises another exception, the saved +exception is set as the context of the new exception. If the "finally" +clause executes a "return", "break" or "continue" statement, the saved +exception is discarded. For example, this function returns 42. + + def f(): + try: + 1/0 + finally: + return 42 + +The exception information is not available to the program during +execution of the "finally" clause. + +When a "return", "break" or "continue" statement is executed in the +"try" suite of a "try"…"finally" statement, the "finally" clause is +also executed ‘on the way out.’ + +The return value of a function is determined by the last "return" +statement executed. Since the "finally" clause always executes, a +"return" statement executed in the "finally" clause will always be the +last one executed. The following function returns ‘finally’. + + def foo(): + try: + return 'try' + finally: + return 'finally' + +Changed in version 3.8: Prior to Python 3.8, a "continue" statement +was illegal in the "finally" clause due to a problem with the +implementation. + +Changed in version 3.14: The compiler emits a "SyntaxWarning" when a +"return", "break" or "continue" appears in a "finally" block (see +**PEP 765**). + + +The "with" statement +==================== + +The "with" statement is used to wrap the execution of a block with +methods defined by a context manager (see section With Statement +Context Managers). This allows common "try"…"except"…"finally" usage +patterns to be encapsulated for convenient reuse. + + with_stmt: "with" ( "(" with_stmt_contents ","? ")" | with_stmt_contents ) ":" suite + with_stmt_contents: with_item ("," with_item)* + with_item: expression ["as" target] + +The execution of the "with" statement with one “item” proceeds as +follows: + +1. The context expression (the expression given in the "with_item") is + evaluated to obtain a context manager. + +2. The context manager’s "__enter__()" is loaded for later use. + +3. The context manager’s "__exit__()" is loaded for later use. + +4. The context manager’s "__enter__()" method is invoked. + +5. If a target was included in the "with" statement, the return value + from "__enter__()" is assigned to it. + + Note: + + The "with" statement guarantees that if the "__enter__()" method + returns without an error, then "__exit__()" will always be + called. Thus, if an error occurs during the assignment to the + target list, it will be treated the same as an error occurring + within the suite would be. See step 7 below. + +6. The suite is executed. + +7. The context manager’s "__exit__()" method is invoked. If an + exception caused the suite to be exited, its type, value, and + traceback are passed as arguments to "__exit__()". Otherwise, three + "None" arguments are supplied. + + If the suite was exited due to an exception, and the return value + from the "__exit__()" method was false, the exception is reraised. + If the return value was true, the exception is suppressed, and + execution continues with the statement following the "with" + statement. + + If the suite was exited for any reason other than an exception, the + return value from "__exit__()" is ignored, and execution proceeds + at the normal location for the kind of exit that was taken. + +The following code: + + with EXPRESSION as TARGET: + SUITE + +is semantically equivalent to: + + manager = (EXPRESSION) + enter = type(manager).__enter__ + exit = type(manager).__exit__ + value = enter(manager) + hit_except = False + + try: + TARGET = value + SUITE + except: + hit_except = True + if not exit(manager, *sys.exc_info()): + raise + finally: + if not hit_except: + exit(manager, None, None, None) + +With more than one item, the context managers are processed as if +multiple "with" statements were nested: + + with A() as a, B() as b: + SUITE + +is semantically equivalent to: + + with A() as a: + with B() as b: + SUITE + +You can also write multi-item context managers in multiple lines if +the items are surrounded by parentheses. For example: + + with ( + A() as a, + B() as b, + ): + SUITE + +Changed in version 3.1: Support for multiple context expressions. + +Changed in version 3.10: Support for using grouping parentheses to +break the statement in multiple lines. + +See also: + + **PEP 343** - The “with” statement + The specification, background, and examples for the Python "with" + statement. + + +The "match" statement +===================== + +Added in version 3.10. + +The match statement is used for pattern matching. Syntax: + + match_stmt: 'match' subject_expr ":" NEWLINE INDENT case_block+ DEDENT + subject_expr: `!star_named_expression` "," `!star_named_expressions`? + | `!named_expression` + case_block: 'case' patterns [guard] ":" `!block` + +Note: + + This section uses single quotes to denote soft keywords. + +Pattern matching takes a pattern as input (following "case") and a +subject value (following "match"). The pattern (which may contain +subpatterns) is matched against the subject value. The outcomes are: + +* A match success or failure (also termed a pattern success or + failure). + +* Possible binding of matched values to a name. The prerequisites for + this are further discussed below. + +The "match" and "case" keywords are soft keywords. + +See also: + + * **PEP 634** – Structural Pattern Matching: Specification + + * **PEP 636** – Structural Pattern Matching: Tutorial + + +Overview +-------- + +Here’s an overview of the logical flow of a match statement: + +1. The subject expression "subject_expr" is evaluated and a resulting + subject value obtained. If the subject expression contains a comma, + a tuple is constructed using the standard rules. + +2. Each pattern in a "case_block" is attempted to match with the + subject value. The specific rules for success or failure are + described below. The match attempt can also bind some or all of the + standalone names within the pattern. The precise pattern binding + rules vary per pattern type and are specified below. **Name + bindings made during a successful pattern match outlive the + executed block and can be used after the match statement**. + + Note: + + During failed pattern matches, some subpatterns may succeed. Do + not rely on bindings being made for a failed match. Conversely, + do not rely on variables remaining unchanged after a failed + match. The exact behavior is dependent on implementation and may + vary. This is an intentional decision made to allow different + implementations to add optimizations. + +3. If the pattern succeeds, the corresponding guard (if present) is + evaluated. In this case all name bindings are guaranteed to have + happened. + + * If the guard evaluates as true or is missing, the "block" inside + "case_block" is executed. + + * Otherwise, the next "case_block" is attempted as described above. + + * If there are no further case blocks, the match statement is + completed. + +Note: + + Users should generally never rely on a pattern being evaluated. + Depending on implementation, the interpreter may cache values or use + other optimizations which skip repeated evaluations. + +A sample match statement: + + >>> flag = False + >>> match (100, 200): + ... case (100, 300): # Mismatch: 200 != 300 + ... print('Case 1') + ... case (100, 200) if flag: # Successful match, but guard fails + ... print('Case 2') + ... case (100, y): # Matches and binds y to 200 + ... print(f'Case 3, y: {y}') + ... case _: # Pattern not attempted + ... print('Case 4, I match anything!') + ... + Case 3, y: 200 + +In this case, "if flag" is a guard. Read more about that in the next +section. + + +Guards +------ + + guard: "if" `!named_expression` + +A "guard" (which is part of the "case") must succeed for code inside +the "case" block to execute. It takes the form: "if" followed by an +expression. + +The logical flow of a "case" block with a "guard" follows: + +1. Check that the pattern in the "case" block succeeded. If the + pattern failed, the "guard" is not evaluated and the next "case" + block is checked. + +2. If the pattern succeeded, evaluate the "guard". + + * If the "guard" condition evaluates as true, the case block is + selected. + + * If the "guard" condition evaluates as false, the case block is + not selected. + + * If the "guard" raises an exception during evaluation, the + exception bubbles up. + +Guards are allowed to have side effects as they are expressions. +Guard evaluation must proceed from the first to the last case block, +one at a time, skipping case blocks whose pattern(s) don’t all +succeed. (I.e., guard evaluation must happen in order.) Guard +evaluation must stop once a case block is selected. + + +Irrefutable Case Blocks +----------------------- + +An irrefutable case block is a match-all case block. A match +statement may have at most one irrefutable case block, and it must be +last. + +A case block is considered irrefutable if it has no guard and its +pattern is irrefutable. A pattern is considered irrefutable if we can +prove from its syntax alone that it will always succeed. Only the +following patterns are irrefutable: + +* AS Patterns whose left-hand side is irrefutable + +* OR Patterns containing at least one irrefutable pattern + +* Capture Patterns + +* Wildcard Patterns + +* parenthesized irrefutable patterns + + +Patterns +-------- + +Note: + + This section uses grammar notations beyond standard EBNF: + + * the notation "SEP.RULE+" is shorthand for "RULE (SEP RULE)*" + + * the notation "!RULE" is shorthand for a negative lookahead + assertion + +The top-level syntax for "patterns" is: + + patterns: open_sequence_pattern | pattern + pattern: as_pattern | or_pattern + closed_pattern: | literal_pattern + | capture_pattern + | wildcard_pattern + | value_pattern + | group_pattern + | sequence_pattern + | mapping_pattern + | class_pattern + +The descriptions below will include a description “in simple terms” of +what a pattern does for illustration purposes (credits to Raymond +Hettinger for a document that inspired most of the descriptions). Note +that these descriptions are purely for illustration purposes and **may +not** reflect the underlying implementation. Furthermore, they do not +cover all valid forms. + + +OR Patterns +~~~~~~~~~~~ + +An OR pattern is two or more patterns separated by vertical bars "|". +Syntax: + + or_pattern: "|".closed_pattern+ + +Only the final subpattern may be irrefutable, and each subpattern must +bind the same set of names to avoid ambiguity. + +An OR pattern matches each of its subpatterns in turn to the subject +value, until one succeeds. The OR pattern is then considered +successful. Otherwise, if none of the subpatterns succeed, the OR +pattern fails. + +In simple terms, "P1 | P2 | ..." will try to match "P1", if it fails +it will try to match "P2", succeeding immediately if any succeeds, +failing otherwise. + + +AS Patterns +~~~~~~~~~~~ + +An AS pattern matches an OR pattern on the left of the "as" keyword +against a subject. Syntax: + + as_pattern: or_pattern "as" capture_pattern + +If the OR pattern fails, the AS pattern fails. Otherwise, the AS +pattern binds the subject to the name on the right of the as keyword +and succeeds. "capture_pattern" cannot be a "_". + +In simple terms "P as NAME" will match with "P", and on success it +will set "NAME = ". + + +Literal Patterns +~~~~~~~~~~~~~~~~ + +A literal pattern corresponds to most literals in Python. Syntax: + + literal_pattern: signed_number + | signed_number "+" NUMBER + | signed_number "-" NUMBER + | strings + | "None" + | "True" + | "False" + signed_number: ["-"] NUMBER + +The rule "strings" and the token "NUMBER" are defined in the standard +Python grammar. Triple-quoted strings are supported. Raw strings and +byte strings are supported. f-strings and t-strings are not +supported. + +The forms "signed_number '+' NUMBER" and "signed_number '-' NUMBER" +are for expressing complex numbers; they require a real number on the +left and an imaginary number on the right. E.g. "3 + 4j". + +In simple terms, "LITERAL" will succeed only if " == +LITERAL". For the singletons "None", "True" and "False", the "is" +operator is used. + + +Capture Patterns +~~~~~~~~~~~~~~~~ + +A capture pattern binds the subject value to a name. Syntax: + + capture_pattern: !'_' NAME + +A single underscore "_" is not a capture pattern (this is what "!'_'" +expresses). It is instead treated as a "wildcard_pattern". + +In a given pattern, a given name can only be bound once. E.g. "case +x, x: ..." is invalid while "case [x] | x: ..." is allowed. + +Capture patterns always succeed. The binding follows scoping rules +established by the assignment expression operator in **PEP 572**; the +name becomes a local variable in the closest containing function scope +unless there’s an applicable "global" or "nonlocal" statement. + +In simple terms "NAME" will always succeed and it will set "NAME = +". + + +Wildcard Patterns +~~~~~~~~~~~~~~~~~ + +A wildcard pattern always succeeds (matches anything) and binds no +name. Syntax: + + wildcard_pattern: '_' + +"_" is a soft keyword within any pattern, but only within patterns. +It is an identifier, as usual, even within "match" subject +expressions, "guard"s, and "case" blocks. + +In simple terms, "_" will always succeed. + + +Value Patterns +~~~~~~~~~~~~~~ + +A value pattern represents a named value in Python. Syntax: + + value_pattern: attr + attr: name_or_attr "." NAME + name_or_attr: attr | NAME + +The dotted name in the pattern is looked up using standard Python name +resolution rules. The pattern succeeds if the value found compares +equal to the subject value (using the "==" equality operator). + +In simple terms "NAME1.NAME2" will succeed only if " == +NAME1.NAME2" + +Note: + + If the same value occurs multiple times in the same match statement, + the interpreter may cache the first value found and reuse it rather + than repeat the same lookup. This cache is strictly tied to a given + execution of a given match statement. + + +Group Patterns +~~~~~~~~~~~~~~ + +A group pattern allows users to add parentheses around patterns to +emphasize the intended grouping. Otherwise, it has no additional +syntax. Syntax: + + group_pattern: "(" pattern ")" + +In simple terms "(P)" has the same effect as "P". + + +Sequence Patterns +~~~~~~~~~~~~~~~~~ + +A sequence pattern contains several subpatterns to be matched against +sequence elements. The syntax is similar to the unpacking of a list or +tuple. + + sequence_pattern: "[" [maybe_sequence_pattern] "]" + | "(" [open_sequence_pattern] ")" + open_sequence_pattern: maybe_star_pattern "," [maybe_sequence_pattern] + maybe_sequence_pattern: ",".maybe_star_pattern+ ","? + maybe_star_pattern: star_pattern | pattern + star_pattern: "*" (capture_pattern | wildcard_pattern) + +There is no difference if parentheses or square brackets are used for +sequence patterns (i.e. "(...)" vs "[...]" ). + +Note: + + A single pattern enclosed in parentheses without a trailing comma + (e.g. "(3 | 4)") is a group pattern. While a single pattern enclosed + in square brackets (e.g. "[3 | 4]") is still a sequence pattern. + +At most one star subpattern may be in a sequence pattern. The star +subpattern may occur in any position. If no star subpattern is +present, the sequence pattern is a fixed-length sequence pattern; +otherwise it is a variable-length sequence pattern. + +The following is the logical flow for matching a sequence pattern +against a subject value: + +1. If the subject value is not a sequence [2], the sequence pattern + fails. + +2. If the subject value is an instance of "str", "bytes" or + "bytearray" the sequence pattern fails. + +3. The subsequent steps depend on whether the sequence pattern is + fixed or variable-length. + + If the sequence pattern is fixed-length: + + 1. If the length of the subject sequence is not equal to the number + of subpatterns, the sequence pattern fails + + 2. Subpatterns in the sequence pattern are matched to their + corresponding items in the subject sequence from left to right. + Matching stops as soon as a subpattern fails. If all + subpatterns succeed in matching their corresponding item, the + sequence pattern succeeds. + + Otherwise, if the sequence pattern is variable-length: + + 1. If the length of the subject sequence is less than the number of + non-star subpatterns, the sequence pattern fails. + + 2. The leading non-star subpatterns are matched to their + corresponding items as for fixed-length sequences. + + 3. If the previous step succeeds, the star subpattern matches a + list formed of the remaining subject items, excluding the + remaining items corresponding to non-star subpatterns following + the star subpattern. + + 4. Remaining non-star subpatterns are matched to their + corresponding subject items, as for a fixed-length sequence. + + Note: + + The length of the subject sequence is obtained via "len()" (i.e. + via the "__len__()" protocol). This length may be cached by the + interpreter in a similar manner as value patterns. + +In simple terms "[P1, P2, P3," … ", P]" matches only if all the +following happens: + +* check "" is a sequence + +* "len(subject) == " + +* "P1" matches "[0]" (note that this match can also bind + names) + +* "P2" matches "[1]" (note that this match can also bind + names) + +* … and so on for the corresponding pattern/element. + + +Mapping Patterns +~~~~~~~~~~~~~~~~ + +A mapping pattern contains one or more key-value patterns. The syntax +is similar to the construction of a dictionary. Syntax: + + mapping_pattern: "{" [items_pattern] "}" + items_pattern: ",".key_value_pattern+ ","? + key_value_pattern: (literal_pattern | value_pattern) ":" pattern + | double_star_pattern + double_star_pattern: "**" capture_pattern + +At most one double star pattern may be in a mapping pattern. The +double star pattern must be the last subpattern in the mapping +pattern. + +Duplicate keys in mapping patterns are disallowed. Duplicate literal +keys will raise a "SyntaxError". Two keys that otherwise have the same +value will raise a "ValueError" at runtime. + +The following is the logical flow for matching a mapping pattern +against a subject value: + +1. If the subject value is not a mapping [3],the mapping pattern + fails. + +2. If every key given in the mapping pattern is present in the subject + mapping, and the pattern for each key matches the corresponding + item of the subject mapping, the mapping pattern succeeds. + +3. If duplicate keys are detected in the mapping pattern, the pattern + is considered invalid. A "SyntaxError" is raised for duplicate + literal values; or a "ValueError" for named keys of the same value. + +Note: + + Key-value pairs are matched using the two-argument form of the + mapping subject’s "get()" method. Matched key-value pairs must + already be present in the mapping, and not created on-the-fly via + "__missing__()" or "__getitem__()". + +In simple terms "{KEY1: P1, KEY2: P2, ... }" matches only if all the +following happens: + +* check "" is a mapping + +* "KEY1 in " + +* "P1" matches "[KEY1]" + +* … and so on for the corresponding KEY/pattern pair. + + +Class Patterns +~~~~~~~~~~~~~~ + +A class pattern represents a class and its positional and keyword +arguments (if any). Syntax: + + class_pattern: name_or_attr "(" [pattern_arguments ","?] ")" + pattern_arguments: positional_patterns ["," keyword_patterns] + | keyword_patterns + positional_patterns: ",".pattern+ + keyword_patterns: ",".keyword_pattern+ + keyword_pattern: NAME "=" pattern + +The same keyword should not be repeated in class patterns. + +The following is the logical flow for matching a class pattern against +a subject value: + +1. If "name_or_attr" is not an instance of the builtin "type" , raise + "TypeError". + +2. If the subject value is not an instance of "name_or_attr" (tested + via "isinstance()"), the class pattern fails. + +3. If no pattern arguments are present, the pattern succeeds. + Otherwise, the subsequent steps depend on whether keyword or + positional argument patterns are present. + + For a number of built-in types (specified below), a single + positional subpattern is accepted which will match the entire + subject; for these types keyword patterns also work as for other + types. + + If only keyword patterns are present, they are processed as + follows, one by one: + + 1. The keyword is looked up as an attribute on the subject. + + * If this raises an exception other than "AttributeError", the + exception bubbles up. + + * If this raises "AttributeError", the class pattern has failed. + + * Else, the subpattern associated with the keyword pattern is + matched against the subject’s attribute value. If this fails, + the class pattern fails; if this succeeds, the match proceeds + to the next keyword. + + 2. If all keyword patterns succeed, the class pattern succeeds. + + If any positional patterns are present, they are converted to + keyword patterns using the "__match_args__" attribute on the class + "name_or_attr" before matching: + + 1. The equivalent of "getattr(cls, "__match_args__", ())" is + called. + + * If this raises an exception, the exception bubbles up. + + * If the returned value is not a tuple, the conversion fails and + "TypeError" is raised. + + * If there are more positional patterns than + "len(cls.__match_args__)", "TypeError" is raised. + + * Otherwise, positional pattern "i" is converted to a keyword + pattern using "__match_args__[i]" as the keyword. + "__match_args__[i]" must be a string; if not "TypeError" is + raised. + + * If there are duplicate keywords, "TypeError" is raised. + + See also: + + Customizing positional arguments in class pattern matching + + 2. Once all positional patterns have been converted to keyword + patterns, the match proceeds as if there were only keyword + patterns. + + For the following built-in types the handling of positional + subpatterns is different: + + * "bool" + + * "bytearray" + + * "bytes" + + * "dict" + + * "float" + + * "frozenset" + + * "int" + + * "list" + + * "set" + + * "str" + + * "tuple" + + These classes accept a single positional argument, and the pattern + there is matched against the whole object rather than an attribute. + For example "int(0|1)" matches the value "0", but not the value + "0.0". + +In simple terms "CLS(P1, attr=P2)" matches only if the following +happens: + +* "isinstance(, CLS)" + +* convert "P1" to a keyword pattern using "CLS.__match_args__" + +* For each keyword argument "attr=P2": + + * "hasattr(, "attr")" + + * "P2" matches ".attr" + +* … and so on for the corresponding keyword argument/pattern pair. + +See also: + + * **PEP 634** – Structural Pattern Matching: Specification + + * **PEP 636** – Structural Pattern Matching: Tutorial + + +Function definitions +==================== + +A function definition defines a user-defined function object (see +section The standard type hierarchy): + + funcdef: [decorators] "def" funcname [type_params] "(" [parameter_list] ")" + ["->" expression] ":" suite + decorators: decorator+ + decorator: "@" assignment_expression NEWLINE + parameter_list: defparameter ("," defparameter)* "," "/" ["," [parameter_list_no_posonly]] + | parameter_list_no_posonly + parameter_list_no_posonly: defparameter ("," defparameter)* ["," [parameter_list_starargs]] + | parameter_list_starargs + parameter_list_starargs: "*" [star_parameter] ("," defparameter)* ["," [parameter_star_kwargs]] + | "*" ("," defparameter)+ ["," [parameter_star_kwargs]] + | parameter_star_kwargs + parameter_star_kwargs: "**" parameter [","] + parameter: identifier [":" expression] + star_parameter: identifier [":" ["*"] expression] + defparameter: parameter ["=" expression] + funcname: identifier + +A function definition is an executable statement. Its execution binds +the function name in the current local namespace to a function object +(a wrapper around the executable code for the function). This +function object contains a reference to the current global namespace +as the global namespace to be used when the function is called. + +The function definition does not execute the function body; this gets +executed only when the function is called. [4] + +A function definition may be wrapped by one or more *decorator* +expressions. Decorator expressions are evaluated when the function is +defined, in the scope that contains the function definition. The +result must be a callable, which is invoked with the function object +as the only argument. The returned value is bound to the function name +instead of the function object. Multiple decorators are applied in +nested fashion. For example, the following code + + @f1(arg) + @f2 + def func(): pass + +is roughly equivalent to + + def func(): pass + func = f1(arg)(f2(func)) + +except that the original function is not temporarily bound to the name +"func". + +Changed in version 3.9: Functions may be decorated with any valid +"assignment_expression". Previously, the grammar was much more +restrictive; see **PEP 614** for details. + +A list of type parameters may be given in square brackets between the +function’s name and the opening parenthesis for its parameter list. +This indicates to static type checkers that the function is generic. +At runtime, the type parameters can be retrieved from the function’s +"__type_params__" attribute. See Generic functions for more. + +Changed in version 3.12: Type parameter lists are new in Python 3.12. + +When one or more *parameters* have the form *parameter* "=" +*expression*, the function is said to have “default parameter values.” +For a parameter with a default value, the corresponding *argument* may +be omitted from a call, in which case the parameter’s default value is +substituted. If a parameter has a default value, all following +parameters up until the “"*"” must also have a default value — this is +a syntactic restriction that is not expressed by the grammar. + +**Default parameter values are evaluated from left to right when the +function definition is executed.** This means that the expression is +evaluated once, when the function is defined, and that the same “pre- +computed” value is used for each call. This is especially important +to understand when a default parameter value is a mutable object, such +as a list or a dictionary: if the function modifies the object (e.g. +by appending an item to a list), the default parameter value is in +effect modified. This is generally not what was intended. A way +around this is to use "None" as the default, and explicitly test for +it in the body of the function, e.g.: + + def whats_on_the_telly(penguin=None): + if penguin is None: + penguin = [] + penguin.append("property of the zoo") + return penguin + +Function call semantics are described in more detail in section Calls. +A function call always assigns values to all parameters mentioned in +the parameter list, either from positional arguments, from keyword +arguments, or from default values. If the form “"*identifier"” is +present, it is initialized to a tuple receiving any excess positional +parameters, defaulting to the empty tuple. If the form +“"**identifier"” is present, it is initialized to a new ordered +mapping receiving any excess keyword arguments, defaulting to a new +empty mapping of the same type. Parameters after “"*"” or +“"*identifier"” are keyword-only parameters and may only be passed by +keyword arguments. Parameters before “"/"” are positional-only +parameters and may only be passed by positional arguments. + +Changed in version 3.8: The "/" function parameter syntax may be used +to indicate positional-only parameters. See **PEP 570** for details. + +Parameters may have an *annotation* of the form “": expression"” +following the parameter name. Any parameter may have an annotation, +even those of the form "*identifier" or "**identifier". (As a special +case, parameters of the form "*identifier" may have an annotation “": +*expression"”.) Functions may have “return” annotation of the form +“"-> expression"” after the parameter list. These annotations can be +any valid Python expression. The presence of annotations does not +change the semantics of a function. See Annotations for more +information on annotations. + +Changed in version 3.11: Parameters of the form “"*identifier"” may +have an annotation “": *expression"”. See **PEP 646**. + +It is also possible to create anonymous functions (functions not bound +to a name), for immediate use in expressions. This uses lambda +expressions, described in section Lambdas. Note that the lambda +expression is merely a shorthand for a simplified function definition; +a function defined in a “"def"” statement can be passed around or +assigned to another name just like a function defined by a lambda +expression. The “"def"” form is actually more powerful since it +allows the execution of multiple statements and annotations. + +**Programmer’s note:** Functions are first-class objects. A “"def"” +statement executed inside a function definition defines a local +function that can be returned or passed around. Free variables used +in the nested function can access the local variables of the function +containing the def. See section Naming and binding for details. + +See also: + + **PEP 3107** - Function Annotations + The original specification for function annotations. + + **PEP 484** - Type Hints + Definition of a standard meaning for annotations: type hints. + + **PEP 526** - Syntax for Variable Annotations + Ability to type hint variable declarations, including class + variables and instance variables. + + **PEP 563** - Postponed Evaluation of Annotations + Support for forward references within annotations by preserving + annotations in a string form at runtime instead of eager + evaluation. + + **PEP 318** - Decorators for Functions and Methods + Function and method decorators were introduced. Class decorators + were introduced in **PEP 3129**. + + +Class definitions +================= + +A class definition defines a class object (see section The standard +type hierarchy): + + classdef: [decorators] "class" classname [type_params] [inheritance] ":" suite + inheritance: "(" [argument_list] ")" + classname: identifier + +A class definition is an executable statement. The inheritance list +usually gives a list of base classes (see Metaclasses for more +advanced uses), so each item in the list should evaluate to a class +object which allows subclassing. Classes without an inheritance list +inherit, by default, from the base class "object"; hence, + + class Foo: + pass + +is equivalent to + + class Foo(object): + pass + +The class’s suite is then executed in a new execution frame (see +Naming and binding), using a newly created local namespace and the +original global namespace. (Usually, the suite contains mostly +function definitions.) When the class’s suite finishes execution, its +execution frame is discarded but its local namespace is saved. [5] A +class object is then created using the inheritance list for the base +classes and the saved local namespace for the attribute dictionary. +The class name is bound to this class object in the original local +namespace. + +The order in which attributes are defined in the class body is +preserved in the new class’s "__dict__". Note that this is reliable +only right after the class is created and only for classes that were +defined using the definition syntax. + +Class creation can be customized heavily using metaclasses. + +Classes can also be decorated: just like when decorating functions, + + @f1(arg) + @f2 + class Foo: pass + +is roughly equivalent to + + class Foo: pass + Foo = f1(arg)(f2(Foo)) + +The evaluation rules for the decorator expressions are the same as for +function decorators. The result is then bound to the class name. + +Changed in version 3.9: Classes may be decorated with any valid +"assignment_expression". Previously, the grammar was much more +restrictive; see **PEP 614** for details. + +A list of type parameters may be given in square brackets immediately +after the class’s name. This indicates to static type checkers that +the class is generic. At runtime, the type parameters can be retrieved +from the class’s "__type_params__" attribute. See Generic classes for +more. + +Changed in version 3.12: Type parameter lists are new in Python 3.12. + +**Programmer’s note:** Variables defined in the class definition are +class attributes; they are shared by instances. Instance attributes +can be set in a method with "self.name = value". Both class and +instance attributes are accessible through the notation “"self.name"”, +and an instance attribute hides a class attribute with the same name +when accessed in this way. Class attributes can be used as defaults +for instance attributes, but using mutable values there can lead to +unexpected results. Descriptors can be used to create instance +variables with different implementation details. + +See also: + + **PEP 3115** - Metaclasses in Python 3000 + The proposal that changed the declaration of metaclasses to the + current syntax, and the semantics for how classes with + metaclasses are constructed. + + **PEP 3129** - Class Decorators + The proposal that added class decorators. Function and method + decorators were introduced in **PEP 318**. + + +Coroutines +========== + +Added in version 3.5. + + +Coroutine function definition +----------------------------- + + async_funcdef: [decorators] "async" "def" funcname "(" [parameter_list] ")" + ["->" expression] ":" suite + +Execution of Python coroutines can be suspended and resumed at many +points (see *coroutine*). "await" expressions, "async for" and "async +with" can only be used in the body of a coroutine function. + +Functions defined with "async def" syntax are always coroutine +functions, even if they do not contain "await" or "async" keywords. + +It is a "SyntaxError" to use a "yield from" expression inside the body +of a coroutine function. + +An example of a coroutine function: + + async def func(param1, param2): + do_stuff() + await some_coroutine() + +Changed in version 3.7: "await" and "async" are now keywords; +previously they were only treated as such inside the body of a +coroutine function. + + +The "async for" statement +------------------------- + + async_for_stmt: "async" for_stmt + +An *asynchronous iterable* provides an "__aiter__" method that +directly returns an *asynchronous iterator*, which can call +asynchronous code in its "__anext__" method. + +The "async for" statement allows convenient iteration over +asynchronous iterables. + +The following code: + + async for TARGET in ITER: + SUITE + else: + SUITE2 + +Is semantically equivalent to: + + iter = (ITER) + iter = type(iter).__aiter__(iter) + running = True + + while running: + try: + TARGET = await type(iter).__anext__(iter) + except StopAsyncIteration: + running = False + else: + SUITE + else: + SUITE2 + +See also "__aiter__()" and "__anext__()" for details. + +It is a "SyntaxError" to use an "async for" statement outside the body +of a coroutine function. + + +The "async with" statement +-------------------------- + + async_with_stmt: "async" with_stmt + +An *asynchronous context manager* is a *context manager* that is able +to suspend execution in its *enter* and *exit* methods. + +The following code: + + async with EXPRESSION as TARGET: + SUITE + +is semantically equivalent to: + + manager = (EXPRESSION) + aenter = type(manager).__aenter__ + aexit = type(manager).__aexit__ + value = await aenter(manager) + hit_except = False + + try: + TARGET = value + SUITE + except: + hit_except = True + if not await aexit(manager, *sys.exc_info()): + raise + finally: + if not hit_except: + await aexit(manager, None, None, None) + +See also "__aenter__()" and "__aexit__()" for details. + +It is a "SyntaxError" to use an "async with" statement outside the +body of a coroutine function. + +See also: + + **PEP 492** - Coroutines with async and await syntax + The proposal that made coroutines a proper standalone concept in + Python, and added supporting syntax. + + +Type parameter lists +==================== + +Added in version 3.12. + +Changed in version 3.13: Support for default values was added (see +**PEP 696**). + + type_params: "[" type_param ("," type_param)* "]" + type_param: typevar | typevartuple | paramspec + typevar: identifier (":" expression)? ("=" expression)? + typevartuple: "*" identifier ("=" expression)? + paramspec: "**" identifier ("=" expression)? + +Functions (including coroutines), classes and type aliases may contain +a type parameter list: + + def max[T](args: list[T]) -> T: + ... + + async def amax[T](args: list[T]) -> T: + ... + + class Bag[T]: + def __iter__(self) -> Iterator[T]: + ... + + def add(self, arg: T) -> None: + ... + + type ListOrSet[T] = list[T] | set[T] + +Semantically, this indicates that the function, class, or type alias +is generic over a type variable. This information is primarily used by +static type checkers, and at runtime, generic objects behave much like +their non-generic counterparts. + +Type parameters are declared in square brackets ("[]") immediately +after the name of the function, class, or type alias. The type +parameters are accessible within the scope of the generic object, but +not elsewhere. Thus, after a declaration "def func[T](): pass", the +name "T" is not available in the module scope. Below, the semantics of +generic objects are described with more precision. The scope of type +parameters is modeled with a special function (technically, an +annotation scope) that wraps the creation of the generic object. + +Generic functions, classes, and type aliases have a "__type_params__" +attribute listing their type parameters. + +Type parameters come in three kinds: + +* "typing.TypeVar", introduced by a plain name (e.g., "T"). + Semantically, this represents a single type to a type checker. + +* "typing.TypeVarTuple", introduced by a name prefixed with a single + asterisk (e.g., "*Ts"). Semantically, this stands for a tuple of any + number of types. + +* "typing.ParamSpec", introduced by a name prefixed with two asterisks + (e.g., "**P"). Semantically, this stands for the parameters of a + callable. + +"typing.TypeVar" declarations can define *bounds* and *constraints* +with a colon (":") followed by an expression. A single expression +after the colon indicates a bound (e.g. "T: int"). Semantically, this +means that the "typing.TypeVar" can only represent types that are a +subtype of this bound. A parenthesized tuple of expressions after the +colon indicates a set of constraints (e.g. "T: (str, bytes)"). Each +member of the tuple should be a type (again, this is not enforced at +runtime). Constrained type variables can only take on one of the types +in the list of constraints. + +For "typing.TypeVar"s declared using the type parameter list syntax, +the bound and constraints are not evaluated when the generic object is +created, but only when the value is explicitly accessed through the +attributes "__bound__" and "__constraints__". To accomplish this, the +bounds or constraints are evaluated in a separate annotation scope. + +"typing.TypeVarTuple"s and "typing.ParamSpec"s cannot have bounds or +constraints. + +All three flavors of type parameters can also have a *default value*, +which is used when the type parameter is not explicitly provided. This +is added by appending a single equals sign ("=") followed by an +expression. Like the bounds and constraints of type variables, the +default value is not evaluated when the object is created, but only +when the type parameter’s "__default__" attribute is accessed. To this +end, the default value is evaluated in a separate annotation scope. If +no default value is specified for a type parameter, the "__default__" +attribute is set to the special sentinel object "typing.NoDefault". + +The following example indicates the full set of allowed type parameter +declarations: + + def overly_generic[ + SimpleTypeVar, + TypeVarWithDefault = int, + TypeVarWithBound: int, + TypeVarWithConstraints: (str, bytes), + *SimpleTypeVarTuple = (int, float), + **SimpleParamSpec = (str, bytearray), + ]( + a: SimpleTypeVar, + b: TypeVarWithDefault, + c: TypeVarWithBound, + d: Callable[SimpleParamSpec, TypeVarWithConstraints], + *e: SimpleTypeVarTuple, + ): ... + + +Generic functions +----------------- + +Generic functions are declared as follows: + + def func[T](arg: T): ... + +This syntax is equivalent to: + + annotation-def TYPE_PARAMS_OF_func(): + T = typing.TypeVar("T") + def func(arg: T): ... + func.__type_params__ = (T,) + return func + func = TYPE_PARAMS_OF_func() + +Here "annotation-def" indicates an annotation scope, which is not +actually bound to any name at runtime. (One other liberty is taken in +the translation: the syntax does not go through attribute access on +the "typing" module, but creates an instance of "typing.TypeVar" +directly.) + +The annotations of generic functions are evaluated within the +annotation scope used for declaring the type parameters, but the +function’s defaults and decorators are not. + +The following example illustrates the scoping rules for these cases, +as well as for additional flavors of type parameters: + + @decorator + def func[T: int, *Ts, **P](*args: *Ts, arg: Callable[P, T] = some_default): + ... + +Except for the lazy evaluation of the "TypeVar" bound, this is +equivalent to: + + DEFAULT_OF_arg = some_default + + annotation-def TYPE_PARAMS_OF_func(): + + annotation-def BOUND_OF_T(): + return int + # In reality, BOUND_OF_T() is evaluated only on demand. + T = typing.TypeVar("T", bound=BOUND_OF_T()) + + Ts = typing.TypeVarTuple("Ts") + P = typing.ParamSpec("P") + + def func(*args: *Ts, arg: Callable[P, T] = DEFAULT_OF_arg): + ... + + func.__type_params__ = (T, Ts, P) + return func + func = decorator(TYPE_PARAMS_OF_func()) + +The capitalized names like "DEFAULT_OF_arg" are not actually bound at +runtime. + + +Generic classes +--------------- + +Generic classes are declared as follows: + + class Bag[T]: ... + +This syntax is equivalent to: + + annotation-def TYPE_PARAMS_OF_Bag(): + T = typing.TypeVar("T") + class Bag(typing.Generic[T]): + __type_params__ = (T,) + ... + return Bag + Bag = TYPE_PARAMS_OF_Bag() + +Here again "annotation-def" (not a real keyword) indicates an +annotation scope, and the name "TYPE_PARAMS_OF_Bag" is not actually +bound at runtime. + +Generic classes implicitly inherit from "typing.Generic". The base +classes and keyword arguments of generic classes are evaluated within +the type scope for the type parameters, and decorators are evaluated +outside that scope. This is illustrated by this example: + + @decorator + class Bag(Base[T], arg=T): ... + +This is equivalent to: + + annotation-def TYPE_PARAMS_OF_Bag(): + T = typing.TypeVar("T") + class Bag(Base[T], typing.Generic[T], arg=T): + __type_params__ = (T,) + ... + return Bag + Bag = decorator(TYPE_PARAMS_OF_Bag()) + + +Generic type aliases +-------------------- + +The "type" statement can also be used to create a generic type alias: + + type ListOrSet[T] = list[T] | set[T] + +Except for the lazy evaluation of the value, this is equivalent to: + + annotation-def TYPE_PARAMS_OF_ListOrSet(): + T = typing.TypeVar("T") + + annotation-def VALUE_OF_ListOrSet(): + return list[T] | set[T] + # In reality, the value is lazily evaluated + return typing.TypeAliasType("ListOrSet", VALUE_OF_ListOrSet(), type_params=(T,)) + ListOrSet = TYPE_PARAMS_OF_ListOrSet() + +Here, "annotation-def" (not a real keyword) indicates an annotation +scope. The capitalized names like "TYPE_PARAMS_OF_ListOrSet" are not +actually bound at runtime. + + +Annotations +=========== + +Changed in version 3.14: Annotations are now lazily evaluated by +default. + +Variables and function parameters may carry *annotations*, created by +adding a colon after the name, followed by an expression: + + x: annotation = 1 + def f(param: annotation): ... + +Functions may also carry a return annotation following an arrow: + + def f() -> annotation: ... + +Annotations are conventionally used for *type hints*, but this is not +enforced by the language, and in general annotations may contain +arbitrary expressions. The presence of annotations does not change the +runtime semantics of the code, except if some mechanism is used that +introspects and uses the annotations (such as "dataclasses" or +"functools.singledispatch()"). + +By default, annotations are lazily evaluated in an annotation scope. +This means that they are not evaluated when the code containing the +annotation is evaluated. Instead, the interpreter saves information +that can be used to evaluate the annotation later if requested. The +"annotationlib" module provides tools for evaluating annotations. + +If the future statement "from __future__ import annotations" is +present, all annotations are instead stored as strings: + + >>> from __future__ import annotations + >>> def f(param: annotation): ... + >>> f.__annotations__ + {'param': 'annotation'} + +This future statement will be deprecated and removed in a future +version of Python, but not before Python 3.13 reaches its end of life +(see **PEP 749**). When it is used, introspection tools like +"annotationlib.get_annotations()" and "typing.get_type_hints()" are +less likely to be able to resolve annotations at runtime. + +-[ Footnotes ]- + +[1] The exception is propagated to the invocation stack unless there + is a "finally" clause which happens to raise another exception. + That new exception causes the old one to be lost. + +[2] In pattern matching, a sequence is defined as one of the + following: + + * a class that inherits from "collections.abc.Sequence" + + * a Python class that has been registered as + "collections.abc.Sequence" + + * a builtin class that has its (CPython) "Py_TPFLAGS_SEQUENCE" bit + set + + * a class that inherits from any of the above + + The following standard library classes are sequences: + + * "array.array" + + * "collections.deque" + + * "list" + + * "memoryview" + + * "range" + + * "tuple" + + Note: + + Subject values of type "str", "bytes", and "bytearray" do not + match sequence patterns. + +[3] In pattern matching, a mapping is defined as one of the following: + + * a class that inherits from "collections.abc.Mapping" + + * a Python class that has been registered as + "collections.abc.Mapping" + + * a builtin class that has its (CPython) "Py_TPFLAGS_MAPPING" bit + set + + * a class that inherits from any of the above + + The standard library classes "dict" and "types.MappingProxyType" + are mappings. + +[4] A string literal appearing as the first statement in the function + body is transformed into the function’s "__doc__" attribute and + therefore the function’s *docstring*. + +[5] A string literal appearing as the first statement in the class + body is transformed into the namespace’s "__doc__" item and + therefore the class’s *docstring*. +''', + 'context-managers': r'''With Statement Context Managers +******************************* + +A *context manager* is an object that defines the runtime context to +be established when executing a "with" statement. The context manager +handles the entry into, and the exit from, the desired runtime context +for the execution of the block of code. Context managers are normally +invoked using the "with" statement (described in section The with +statement), but can also be used by directly invoking their methods. + +Typical uses of context managers include saving and restoring various +kinds of global state, locking and unlocking resources, closing opened +files, etc. + +For more information on context managers, see Context Manager Types. +The "object" class itself does not provide the context manager +methods. + +object.__enter__(self) + + Enter the runtime context related to this object. The "with" + statement will bind this method’s return value to the target(s) + specified in the "as" clause of the statement, if any. + +object.__exit__(self, exc_type, exc_value, traceback) + + Exit the runtime context related to this object. The parameters + describe the exception that caused the context to be exited. If the + context was exited without an exception, all three arguments will + be "None". + + If an exception is supplied, and the method wishes to suppress the + exception (i.e., prevent it from being propagated), it should + return a true value. Otherwise, the exception will be processed + normally upon exit from this method. + + Note that "__exit__()" methods should not reraise the passed-in + exception; this is the caller’s responsibility. + +See also: + + **PEP 343** - The “with” statement + The specification, background, and examples for the Python "with" + statement. +''', + 'continue': r'''The "continue" statement +************************ + + continue_stmt: "continue" + +"continue" may only occur syntactically nested in a "for" or "while" +loop, but not nested in a function or class definition within that +loop. It continues with the next cycle of the nearest enclosing loop. + +When "continue" passes control out of a "try" statement with a +"finally" clause, that "finally" clause is executed before really +starting the next loop cycle. +''', + 'conversions': r'''Arithmetic conversions +********************** + +When a description of an arithmetic operator below uses the phrase +“the numeric arguments are converted to a common real type”, this +means that the operator implementation for built-in types works as +follows: + +* If both arguments are complex numbers, no conversion is performed; + +* if either argument is a complex or a floating-point number, the + other is converted to a floating-point number; + +* otherwise, both must be integers and no conversion is necessary. + +Some additional rules apply for certain operators (e.g., a string as a +left argument to the ‘%’ operator). Extensions must define their own +conversion behavior. +''', + 'customization': r'''Basic customization +******************* + +object.__new__(cls[, ...]) + + Called to create a new instance of class *cls*. "__new__()" is a + static method (special-cased so you need not declare it as such) + that takes the class of which an instance was requested as its + first argument. The remaining arguments are those passed to the + object constructor expression (the call to the class). The return + value of "__new__()" should be the new object instance (usually an + instance of *cls*). + + Typical implementations create a new instance of the class by + invoking the superclass’s "__new__()" method using + "super().__new__(cls[, ...])" with appropriate arguments and then + modifying the newly created instance as necessary before returning + it. + + If "__new__()" is invoked during object construction and it returns + an instance of *cls*, then the new instance’s "__init__()" method + will be invoked like "__init__(self[, ...])", where *self* is the + new instance and the remaining arguments are the same as were + passed to the object constructor. + + If "__new__()" does not return an instance of *cls*, then the new + instance’s "__init__()" method will not be invoked. + + "__new__()" is intended mainly to allow subclasses of immutable + types (like int, str, or tuple) to customize instance creation. It + is also commonly overridden in custom metaclasses in order to + customize class creation. + +object.__init__(self[, ...]) + + Called after the instance has been created (by "__new__()"), but + before it is returned to the caller. The arguments are those + passed to the class constructor expression. If a base class has an + "__init__()" method, the derived class’s "__init__()" method, if + any, must explicitly call it to ensure proper initialization of the + base class part of the instance; for example: + "super().__init__([args...])". + + Because "__new__()" and "__init__()" work together in constructing + objects ("__new__()" to create it, and "__init__()" to customize + it), no non-"None" value may be returned by "__init__()"; doing so + will cause a "TypeError" to be raised at runtime. + +object.__del__(self) + + Called when the instance is about to be destroyed. This is also + called a finalizer or (improperly) a destructor. If a base class + has a "__del__()" method, the derived class’s "__del__()" method, + if any, must explicitly call it to ensure proper deletion of the + base class part of the instance. + + It is possible (though not recommended!) for the "__del__()" method + to postpone destruction of the instance by creating a new reference + to it. This is called object *resurrection*. It is + implementation-dependent whether "__del__()" is called a second + time when a resurrected object is about to be destroyed; the + current *CPython* implementation only calls it once. + + It is not guaranteed that "__del__()" methods are called for + objects that still exist when the interpreter exits. + "weakref.finalize" provides a straightforward way to register a + cleanup function to be called when an object is garbage collected. + + Note: + + "del x" doesn’t directly call "x.__del__()" — the former + decrements the reference count for "x" by one, and the latter is + only called when "x"’s reference count reaches zero. + + **CPython implementation detail:** It is possible for a reference + cycle to prevent the reference count of an object from going to + zero. In this case, the cycle will be later detected and deleted + by the *cyclic garbage collector*. A common cause of reference + cycles is when an exception has been caught in a local variable. + The frame’s locals then reference the exception, which references + its own traceback, which references the locals of all frames caught + in the traceback. + + See also: Documentation for the "gc" module. + + Warning: + + Due to the precarious circumstances under which "__del__()" + methods are invoked, exceptions that occur during their execution + are ignored, and a warning is printed to "sys.stderr" instead. + In particular: + + * "__del__()" can be invoked when arbitrary code is being + executed, including from any arbitrary thread. If "__del__()" + needs to take a lock or invoke any other blocking resource, it + may deadlock as the resource may already be taken by the code + that gets interrupted to execute "__del__()". + + * "__del__()" can be executed during interpreter shutdown. As a + consequence, the global variables it needs to access (including + other modules) may already have been deleted or set to "None". + Python guarantees that globals whose name begins with a single + underscore are deleted from their module before other globals + are deleted; if no other references to such globals exist, this + may help in assuring that imported modules are still available + at the time when the "__del__()" method is called. + +object.__repr__(self) + + Called by the "repr()" built-in function to compute the “official” + string representation of an object. If at all possible, this + should look like a valid Python expression that could be used to + recreate an object with the same value (given an appropriate + environment). If this is not possible, a string of the form + "<...some useful description...>" should be returned. The return + value must be a string object. If a class defines "__repr__()" but + not "__str__()", then "__repr__()" is also used when an “informal” + string representation of instances of that class is required. + + This is typically used for debugging, so it is important that the + representation is information-rich and unambiguous. A default + implementation is provided by the "object" class itself. + +object.__str__(self) + + Called by "str(object)", the default "__format__()" implementation, + and the built-in function "print()", to compute the “informal” or + nicely printable string representation of an object. The return + value must be a str object. + + This method differs from "object.__repr__()" in that there is no + expectation that "__str__()" return a valid Python expression: a + more convenient or concise representation can be used. + + The default implementation defined by the built-in type "object" + calls "object.__repr__()". + +object.__bytes__(self) + + Called by bytes to compute a byte-string representation of an + object. This should return a "bytes" object. The "object" class + itself does not provide this method. + +object.__format__(self, format_spec) + + Called by the "format()" built-in function, and by extension, + evaluation of formatted string literals and the "str.format()" + method, to produce a “formatted” string representation of an + object. The *format_spec* argument is a string that contains a + description of the formatting options desired. The interpretation + of the *format_spec* argument is up to the type implementing + "__format__()", however most classes will either delegate + formatting to one of the built-in types, or use a similar + formatting option syntax. + + See Format Specification Mini-Language for a description of the + standard formatting syntax. + + The return value must be a string object. + + The default implementation by the "object" class should be given an + empty *format_spec* string. It delegates to "__str__()". + + Changed in version 3.4: The __format__ method of "object" itself + raises a "TypeError" if passed any non-empty string. + + Changed in version 3.7: "object.__format__(x, '')" is now + equivalent to "str(x)" rather than "format(str(x), '')". + +object.__lt__(self, other) +object.__le__(self, other) +object.__eq__(self, other) +object.__ne__(self, other) +object.__gt__(self, other) +object.__ge__(self, other) + + These are the so-called “rich comparison” methods. The + correspondence between operator symbols and method names is as + follows: "xy" calls + "x.__gt__(y)", and "x>=y" calls "x.__ge__(y)". + + A rich comparison method may return the singleton "NotImplemented" + if it does not implement the operation for a given pair of + arguments. By convention, "False" and "True" are returned for a + successful comparison. However, these methods can return any value, + so if the comparison operator is used in a Boolean context (e.g., + in the condition of an "if" statement), Python will call "bool()" + on the value to determine if the result is true or false. + + By default, "object" implements "__eq__()" by using "is", returning + "NotImplemented" in the case of a false comparison: "True if x is y + else NotImplemented". For "__ne__()", by default it delegates to + "__eq__()" and inverts the result unless it is "NotImplemented". + There are no other implied relationships among the comparison + operators or default implementations; for example, the truth of + "(x.__hash__". + + If a class that does not override "__eq__()" wishes to suppress + hash support, it should include "__hash__ = None" in the class + definition. A class which defines its own "__hash__()" that + explicitly raises a "TypeError" would be incorrectly identified as + hashable by an "isinstance(obj, collections.abc.Hashable)" call. + + Note: + + By default, the "__hash__()" values of str and bytes objects are + “salted” with an unpredictable random value. Although they + remain constant within an individual Python process, they are not + predictable between repeated invocations of Python.This is + intended to provide protection against a denial-of-service caused + by carefully chosen inputs that exploit the worst case + performance of a dict insertion, *O*(*n*^2) complexity. See + http://ocert.org/advisories/ocert-2011-003.html for + details.Changing hash values affects the iteration order of sets. + Python has never made guarantees about this ordering (and it + typically varies between 32-bit and 64-bit builds).See also + "PYTHONHASHSEED". + + Changed in version 3.3: Hash randomization is enabled by default. + +object.__bool__(self) + + Called to implement truth value testing and the built-in operation + "bool()"; should return "False" or "True". When this method is not + defined, "__len__()" is called, if it is defined, and the object is + considered true if its result is nonzero. If a class defines + neither "__len__()" nor "__bool__()" (which is true of the "object" + class itself), all its instances are considered true. +''', + 'debugger': r'''"pdb" — The Python Debugger +*************************** + +**Source code:** Lib/pdb.py + +====================================================================== + +The module "pdb" defines an interactive source code debugger for +Python programs. It supports setting (conditional) breakpoints and +single stepping at the source line level, inspection of stack frames, +source code listing, and evaluation of arbitrary Python code in the +context of any stack frame. It also supports post-mortem debugging +and can be called under program control. + +The debugger is extensible – it is actually defined as the class +"Pdb". This is currently undocumented but easily understood by reading +the source. The extension interface uses the modules "bdb" and "cmd". + +See also: + + Module "faulthandler" + Used to dump Python tracebacks explicitly, on a fault, after a + timeout, or on a user signal. + + Module "traceback" + Standard interface to extract, format and print stack traces of + Python programs. + +The typical usage to break into the debugger is to insert: + + import pdb; pdb.set_trace() + +Or: + + breakpoint() + +at the location you want to break into the debugger, and then run the +program. You can then step through the code following this statement, +and continue running without the debugger using the "continue" +command. + +Changed in version 3.7: The built-in "breakpoint()", when called with +defaults, can be used instead of "import pdb; pdb.set_trace()". + + def double(x): + breakpoint() + return x * 2 + val = 3 + print(f"{val} * 2 is {double(val)}") + +The debugger’s prompt is "(Pdb)", which is the indicator that you are +in debug mode: + + > ...(2)double() + -> breakpoint() + (Pdb) p x + 3 + (Pdb) continue + 3 * 2 is 6 + +Changed in version 3.3: Tab-completion via the "readline" module is +available for commands and command arguments, e.g. the current global +and local names are offered as arguments of the "p" command. + + +Command-line interface +====================== + +You can also invoke "pdb" from the command line to debug other +scripts. For example: + + python -m pdb [-c command] (-m module | -p pid | pyfile) [args ...] + +When invoked as a module, pdb will automatically enter post-mortem +debugging if the program being debugged exits abnormally. After post- +mortem debugging (or after normal exit of the program), pdb will +restart the program. Automatic restarting preserves pdb’s state (such +as breakpoints) and in most cases is more useful than quitting the +debugger upon program’s exit. + +-c, --command + + To execute commands as if given in a ".pdbrc" file; see Debugger + commands. + + Changed in version 3.2: Added the "-c" option. + +-m + + To execute modules similar to the way "python -m" does. As with a + script, the debugger will pause execution just before the first + line of the module. + + Changed in version 3.7: Added the "-m" option. + +-p, --pid + + Attach to the process with the specified PID. + + Added in version 3.14. + +To attach to a running Python process for remote debugging, use the +"-p" or "--pid" option with the target process’s PID: + + python -m pdb -p 1234 + +Note: + + Attaching to a process that is blocked in a system call or waiting + for I/O will only work once the next bytecode instruction is + executed or when the process receives a signal. + +Typical usage to execute a statement under control of the debugger is: + + >>> import pdb + >>> def f(x): + ... print(1 / x) + >>> pdb.run("f(2)") + > (1)() + (Pdb) continue + 0.5 + >>> + +The typical usage to inspect a crashed program is: + + >>> import pdb + >>> def f(x): + ... print(1 / x) + ... + >>> f(0) + Traceback (most recent call last): + File "", line 1, in + File "", line 2, in f + ZeroDivisionError: division by zero + >>> pdb.pm() + > (2)f() + (Pdb) p x + 0 + (Pdb) + +Changed in version 3.13: The implementation of **PEP 667** means that +name assignments made via "pdb" will immediately affect the active +scope, even when running inside an *optimized scope*. + +The module defines the following functions; each enters the debugger +in a slightly different way: + +pdb.run(statement, globals=None, locals=None) + + Execute the *statement* (given as a string or a code object) under + debugger control. The debugger prompt appears before any code is + executed; you can set breakpoints and type "continue", or you can + step through the statement using "step" or "next" (all these + commands are explained below). The optional *globals* and *locals* + arguments specify the environment in which the code is executed; by + default the dictionary of the module "__main__" is used. (See the + explanation of the built-in "exec()" or "eval()" functions.) + +pdb.runeval(expression, globals=None, locals=None) + + Evaluate the *expression* (given as a string or a code object) + under debugger control. When "runeval()" returns, it returns the + value of the *expression*. Otherwise this function is similar to + "run()". + +pdb.runcall(function, *args, **kwds) + + Call the *function* (a function or method object, not a string) + with the given arguments. When "runcall()" returns, it returns + whatever the function call returned. The debugger prompt appears + as soon as the function is entered. + +pdb.set_trace(*, header=None, commands=None) + + Enter the debugger at the calling stack frame. This is useful to + hard-code a breakpoint at a given point in a program, even if the + code is not otherwise being debugged (e.g. when an assertion + fails). If given, *header* is printed to the console just before + debugging begins. The *commands* argument, if given, is a list of + commands to execute when the debugger starts. + + Changed in version 3.7: The keyword-only argument *header*. + + Changed in version 3.13: "set_trace()" will enter the debugger + immediately, rather than on the next line of code to be executed. + + Added in version 3.14: The *commands* argument. + +awaitable pdb.set_trace_async(*, header=None, commands=None) + + async version of "set_trace()". This function should be used inside + an async function with "await". + + async def f(): + await pdb.set_trace_async() + + "await" statements are supported if the debugger is invoked by this + function. + + Added in version 3.14. + +pdb.post_mortem(t=None) + + Enter post-mortem debugging of the given exception or traceback + object. If no value is given, it uses the exception that is + currently being handled, or raises "ValueError" if there isn’t one. + + Changed in version 3.13: Support for exception objects was added. + +pdb.pm() + + Enter post-mortem debugging of the exception found in + "sys.last_exc". + +pdb.set_default_backend(backend) + + There are two supported backends for pdb: "'settrace'" and + "'monitoring'". See "bdb.Bdb" for details. The user can set the + default backend to use if none is specified when instantiating + "Pdb". If no backend is specified, the default is "'settrace'". + + Note: + + "breakpoint()" and "set_trace()" will not be affected by this + function. They always use "'monitoring'" backend. + + Added in version 3.14. + +pdb.get_default_backend() + + Returns the default backend for pdb. + + Added in version 3.14. + +The "run*" functions and "set_trace()" are aliases for instantiating +the "Pdb" class and calling the method of the same name. If you want +to access further features, you have to do this yourself: + +class pdb.Pdb(completekey='tab', stdin=None, stdout=None, skip=None, nosigint=False, readrc=True, mode=None, backend=None, colorize=False) + + "Pdb" is the debugger class. + + The *completekey*, *stdin* and *stdout* arguments are passed to the + underlying "cmd.Cmd" class; see the description there. + + The *skip* argument, if given, must be an iterable of glob-style + module name patterns. The debugger will not step into frames that + originate in a module that matches one of these patterns. [1] + + By default, Pdb sets a handler for the SIGINT signal (which is sent + when the user presses "Ctrl"-"C" on the console) when you give a + "continue" command. This allows you to break into the debugger + again by pressing "Ctrl"-"C". If you want Pdb not to touch the + SIGINT handler, set *nosigint* to true. + + The *readrc* argument defaults to true and controls whether Pdb + will load .pdbrc files from the filesystem. + + The *mode* argument specifies how the debugger was invoked. It + impacts the workings of some debugger commands. Valid values are + "'inline'" (used by the breakpoint() builtin), "'cli'" (used by the + command line invocation) or "None" (for backwards compatible + behaviour, as before the *mode* argument was added). + + The *backend* argument specifies the backend to use for the + debugger. If "None" is passed, the default backend will be used. + See "set_default_backend()". Otherwise the supported backends are + "'settrace'" and "'monitoring'". + + The *colorize* argument, if set to "True", will enable colorized + output in the debugger, if color is supported. This will highlight + source code displayed in pdb. + + Example call to enable tracing with *skip*: + + import pdb; pdb.Pdb(skip=['django.*']).set_trace() + + Raises an auditing event "pdb.Pdb" with no arguments. + + Changed in version 3.1: Added the *skip* parameter. + + Changed in version 3.2: Added the *nosigint* parameter. Previously, + a SIGINT handler was never set by Pdb. + + Changed in version 3.6: The *readrc* argument. + + Added in version 3.14: Added the *mode* argument. + + Added in version 3.14: Added the *backend* argument. + + Added in version 3.14: Added the *colorize* argument. + + Changed in version 3.14: Inline breakpoints like "breakpoint()" or + "pdb.set_trace()" will always stop the program at calling frame, + ignoring the *skip* pattern (if any). + + run(statement, globals=None, locals=None) + runeval(expression, globals=None, locals=None) + runcall(function, *args, **kwds) + set_trace() + + See the documentation for the functions explained above. + + +Debugger commands +================= + +The commands recognized by the debugger are listed below. Most +commands can be abbreviated to one or two letters as indicated; e.g. +"h(elp)" means that either "h" or "help" can be used to enter the help +command (but not "he" or "hel", nor "H" or "Help" or "HELP"). +Arguments to commands must be separated by whitespace (spaces or +tabs). Optional arguments are enclosed in square brackets ("[]") in +the command syntax; the square brackets must not be typed. +Alternatives in the command syntax are separated by a vertical bar +("|"). + +Entering a blank line repeats the last command entered. Exception: if +the last command was a "list" command, the next 11 lines are listed. + +Commands that the debugger doesn’t recognize are assumed to be Python +statements and are executed in the context of the program being +debugged. Python statements can also be prefixed with an exclamation +point ("!"). This is a powerful way to inspect the program being +debugged; it is even possible to change a variable or call a function. +When an exception occurs in such a statement, the exception name is +printed but the debugger’s state is not changed. + +Changed in version 3.13: Expressions/Statements whose prefix is a pdb +command are now correctly identified and executed. + +The debugger supports aliases. Aliases can have parameters which +allows one a certain level of adaptability to the context under +examination. + +Multiple commands may be entered on a single line, separated by ";;". +(A single ";" is not used as it is the separator for multiple commands +in a line that is passed to the Python parser.) No intelligence is +applied to separating the commands; the input is split at the first +";;" pair, even if it is in the middle of a quoted string. A +workaround for strings with double semicolons is to use implicit +string concatenation "';'';'" or "";"";"". + +To set a temporary global variable, use a *convenience variable*. A +*convenience variable* is a variable whose name starts with "$". For +example, "$foo = 1" sets a global variable "$foo" which you can use in +the debugger session. The *convenience variables* are cleared when +the program resumes execution so it’s less likely to interfere with +your program compared to using normal variables like "foo = 1". + +There are four preset *convenience variables*: + +* "$_frame": the current frame you are debugging + +* "$_retval": the return value if the frame is returning + +* "$_exception": the exception if the frame is raising an exception + +* "$_asynctask": the asyncio task if pdb stops in an async function + +Added in version 3.12: Added the *convenience variable* feature. + +Added in version 3.14: Added the "$_asynctask" convenience variable. + +If a file ".pdbrc" exists in the user’s home directory or in the +current directory, it is read with "'utf-8'" encoding and executed as +if it had been typed at the debugger prompt, with the exception that +empty lines and lines starting with "#" are ignored. This is +particularly useful for aliases. If both files exist, the one in the +home directory is read first and aliases defined there can be +overridden by the local file. + +Changed in version 3.2: ".pdbrc" can now contain commands that +continue debugging, such as "continue" or "next". Previously, these +commands had no effect. + +Changed in version 3.11: ".pdbrc" is now read with "'utf-8'" encoding. +Previously, it was read with the system locale encoding. + +h(elp) [command] + + Without argument, print the list of available commands. With a + *command* as argument, print help about that command. "help pdb" + displays the full documentation (the docstring of the "pdb" + module). Since the *command* argument must be an identifier, "help + exec" must be entered to get help on the "!" command. + +w(here) [count] + + Print a stack trace, with the most recent frame at the bottom. if + *count* is 0, print the current frame entry. If *count* is + negative, print the least recent - *count* frames. If *count* is + positive, print the most recent *count* frames. An arrow (">") + indicates the current frame, which determines the context of most + commands. + + Changed in version 3.14: *count* argument is added. + +d(own) [count] + + Move the current frame *count* (default one) levels down in the + stack trace (to a newer frame). + +u(p) [count] + + Move the current frame *count* (default one) levels up in the stack + trace (to an older frame). + +b(reak) [([filename:]lineno | function) [, condition]] + + With a *lineno* argument, set a break at line *lineno* in the + current file. The line number may be prefixed with a *filename* and + a colon, to specify a breakpoint in another file (possibly one that + hasn’t been loaded yet). The file is searched on "sys.path". + Acceptable forms of *filename* are "/abspath/to/file.py", + "relpath/file.py", "module" and "package.module". + + With a *function* argument, set a break at the first executable + statement within that function. *function* can be any expression + that evaluates to a function in the current namespace. + + If a second argument is present, it is an expression which must + evaluate to true before the breakpoint is honored. + + Without argument, list all breaks, including for each breakpoint, + the number of times that breakpoint has been hit, the current + ignore count, and the associated condition if any. + + Each breakpoint is assigned a number to which all the other + breakpoint commands refer. + +tbreak [([filename:]lineno | function) [, condition]] + + Temporary breakpoint, which is removed automatically when it is + first hit. The arguments are the same as for "break". + +cl(ear) [filename:lineno | bpnumber ...] + + With a *filename:lineno* argument, clear all the breakpoints at + this line. With a space separated list of breakpoint numbers, clear + those breakpoints. Without argument, clear all breaks (but first + ask confirmation). + +disable bpnumber [bpnumber ...] + + Disable the breakpoints given as a space separated list of + breakpoint numbers. Disabling a breakpoint means it cannot cause + the program to stop execution, but unlike clearing a breakpoint, it + remains in the list of breakpoints and can be (re-)enabled. + +enable bpnumber [bpnumber ...] + + Enable the breakpoints specified. + +ignore bpnumber [count] + + Set the ignore count for the given breakpoint number. If *count* + is omitted, the ignore count is set to 0. A breakpoint becomes + active when the ignore count is zero. When non-zero, the *count* + is decremented each time the breakpoint is reached and the + breakpoint is not disabled and any associated condition evaluates + to true. + +condition bpnumber [condition] + + Set a new *condition* for the breakpoint, an expression which must + evaluate to true before the breakpoint is honored. If *condition* + is absent, any existing condition is removed; i.e., the breakpoint + is made unconditional. + +commands [bpnumber] + + Specify a list of commands for breakpoint number *bpnumber*. The + commands themselves appear on the following lines. Type a line + containing just "end" to terminate the commands. An example: + + (Pdb) commands 1 + (com) p some_variable + (com) end + (Pdb) + + To remove all commands from a breakpoint, type "commands" and + follow it immediately with "end"; that is, give no commands. + + With no *bpnumber* argument, "commands" refers to the last + breakpoint set. + + You can use breakpoint commands to start your program up again. + Simply use the "continue" command, or "step", or any other command + that resumes execution. + + Specifying any command resuming execution (currently "continue", + "step", "next", "return", "until", "jump", "quit" and their + abbreviations) terminates the command list (as if that command was + immediately followed by end). This is because any time you resume + execution (even with a simple next or step), you may encounter + another breakpoint—which could have its own command list, leading + to ambiguities about which list to execute. + + If the list of commands contains the "silent" command, or a command + that resumes execution, then the breakpoint message containing + information about the frame is not displayed. + + Changed in version 3.14: Frame information will not be displayed if + a command that resumes execution is present in the command list. + +s(tep) + + Execute the current line, stop at the first possible occasion + (either in a function that is called or on the next line in the + current function). + +n(ext) + + Continue execution until the next line in the current function is + reached or it returns. (The difference between "next" and "step" + is that "step" stops inside a called function, while "next" + executes called functions at (nearly) full speed, only stopping at + the next line in the current function.) + +unt(il) [lineno] + + Without argument, continue execution until the line with a number + greater than the current one is reached. + + With *lineno*, continue execution until a line with a number + greater or equal to *lineno* is reached. In both cases, also stop + when the current frame returns. + + Changed in version 3.2: Allow giving an explicit line number. + +r(eturn) + + Continue execution until the current function returns. + +c(ont(inue)) + + Continue execution, only stop when a breakpoint is encountered. + +j(ump) lineno + + Set the next line that will be executed. Only available in the + bottom-most frame. This lets you jump back and execute code again, + or jump forward to skip code that you don’t want to run. + + It should be noted that not all jumps are allowed – for instance it + is not possible to jump into the middle of a "for" loop or out of a + "finally" clause. + +l(ist) [first[, last]] + + List source code for the current file. Without arguments, list 11 + lines around the current line or continue the previous listing. + With "." as argument, list 11 lines around the current line. With + one argument, list 11 lines around at that line. With two + arguments, list the given range; if the second argument is less + than the first, it is interpreted as a count. + + The current line in the current frame is indicated by "->". If an + exception is being debugged, the line where the exception was + originally raised or propagated is indicated by ">>", if it differs + from the current line. + + Changed in version 3.2: Added the ">>" marker. + +ll | longlist + + List all source code for the current function or frame. + Interesting lines are marked as for "list". + + Added in version 3.2. + +a(rgs) + + Print the arguments of the current function and their current + values. + +p expression + + Evaluate *expression* in the current context and print its value. + + Note: + + "print()" can also be used, but is not a debugger command — this + executes the Python "print()" function. + +pp expression + + Like the "p" command, except the value of *expression* is pretty- + printed using the "pprint" module. + +whatis expression + + Print the type of *expression*. + +source expression + + Try to get source code of *expression* and display it. + + Added in version 3.2. + +display [expression] + + Display the value of *expression* if it changed, each time + execution stops in the current frame. + + Without *expression*, list all display expressions for the current + frame. + + Note: + + Display evaluates *expression* and compares to the result of the + previous evaluation of *expression*, so when the result is + mutable, display may not be able to pick up the changes. + + Example: + + lst = [] + breakpoint() + pass + lst.append(1) + print(lst) + + Display won’t realize "lst" has been changed because the result of + evaluation is modified in place by "lst.append(1)" before being + compared: + + > example.py(3)() + -> pass + (Pdb) display lst + display lst: [] + (Pdb) n + > example.py(4)() + -> lst.append(1) + (Pdb) n + > example.py(5)() + -> print(lst) + (Pdb) + + You can do some tricks with copy mechanism to make it work: + + > example.py(3)() + -> pass + (Pdb) display lst[:] + display lst[:]: [] + (Pdb) n + > example.py(4)() + -> lst.append(1) + (Pdb) n + > example.py(5)() + -> print(lst) + display lst[:]: [1] [old: []] + (Pdb) + + Added in version 3.2. + +undisplay [expression] + + Do not display *expression* anymore in the current frame. Without + *expression*, clear all display expressions for the current frame. + + Added in version 3.2. + +interact + + Start an interactive interpreter (using the "code" module) in a new + global namespace initialised from the local and global namespaces + for the current scope. Use "exit()" or "quit()" to exit the + interpreter and return to the debugger. + + Note: + + As "interact" creates a new dedicated namespace for code + execution, assignments to variables will not affect the original + namespaces. However, modifications to any referenced mutable + objects will be reflected in the original namespaces as usual. + + Added in version 3.2. + + Changed in version 3.13: "exit()" and "quit()" can be used to exit + the "interact" command. + + Changed in version 3.13: "interact" directs its output to the + debugger’s output channel rather than "sys.stderr". + +alias [name [command]] + + Create an alias called *name* that executes *command*. The + *command* must *not* be enclosed in quotes. Replaceable parameters + can be indicated by "%1", "%2", … and "%9", while "%*" is replaced + by all the parameters. If *command* is omitted, the current alias + for *name* is shown. If no arguments are given, all aliases are + listed. + + Aliases may be nested and can contain anything that can be legally + typed at the pdb prompt. Note that internal pdb commands *can* be + overridden by aliases. Such a command is then hidden until the + alias is removed. Aliasing is recursively applied to the first + word of the command line; all other words in the line are left + alone. + + As an example, here are two useful aliases (especially when placed + in the ".pdbrc" file): + + # Print instance variables (usage "pi classInst") + alias pi for k in %1.__dict__.keys(): print(f"%1.{k} = {%1.__dict__[k]}") + # Print instance variables in self + alias ps pi self + +unalias name + + Delete the specified alias *name*. + +! statement + + Execute the (one-line) *statement* in the context of the current + stack frame. The exclamation point can be omitted unless the first + word of the statement resembles a debugger command, e.g.: + + (Pdb) ! n=42 + (Pdb) + + To set a global variable, you can prefix the assignment command + with a "global" statement on the same line, e.g.: + + (Pdb) global list_options; list_options = ['-l'] + (Pdb) + +run [args ...] +restart [args ...] + + Restart the debugged Python program. If *args* is supplied, it is + split with "shlex" and the result is used as the new "sys.argv". + History, breakpoints, actions and debugger options are preserved. + "restart" is an alias for "run". + + Changed in version 3.14: "run" and "restart" commands are disabled + when the debugger is invoked in "'inline'" mode. + +q(uit) + + Quit from the debugger. The program being executed is aborted. An + end-of-file input is equivalent to "quit". + + A confirmation prompt will be shown if the debugger is invoked in + "'inline'" mode. Either "y", "Y", "" or "EOF" will confirm + the quit. + + Changed in version 3.14: A confirmation prompt will be shown if the + debugger is invoked in "'inline'" mode. After the confirmation, the + debugger will call "sys.exit()" immediately, instead of raising + "bdb.BdbQuit" in the next trace event. + +debug code + + Enter a recursive debugger that steps through *code* (which is an + arbitrary expression or statement to be executed in the current + environment). + +retval + + Print the return value for the last return of the current function. + +exceptions [excnumber] + + List or jump between chained exceptions. + + When using "pdb.pm()" or "Pdb.post_mortem(...)" with a chained + exception instead of a traceback, it allows the user to move + between the chained exceptions using "exceptions" command to list + exceptions, and "exceptions " to switch to that exception. + + Example: + + def out(): + try: + middle() + except Exception as e: + raise ValueError("reraise middle() error") from e + + def middle(): + try: + return inner(0) + except Exception as e: + raise ValueError("Middle fail") + + def inner(x): + 1 / x + + out() + + calling "pdb.pm()" will allow to move between exceptions: + + > example.py(5)out() + -> raise ValueError("reraise middle() error") from e + + (Pdb) exceptions + 0 ZeroDivisionError('division by zero') + 1 ValueError('Middle fail') + > 2 ValueError('reraise middle() error') + + (Pdb) exceptions 0 + > example.py(16)inner() + -> 1 / x + + (Pdb) up + > example.py(10)middle() + -> return inner(0) + + Added in version 3.13. + +-[ Footnotes ]- + +[1] Whether a frame is considered to originate in a certain module is + determined by the "__name__" in the frame globals. +''', + 'del': r'''The "del" statement +******************* + + del_stmt: "del" target_list + +Deletion is recursively defined very similar to the way assignment is +defined. Rather than spelling it out in full details, here are some +hints. + +Deletion of a target list recursively deletes each target, from left +to right. + +Deletion of a name removes the binding of that name from the local or +global namespace, depending on whether the name occurs in a "global" +statement in the same code block. Trying to delete an unbound name +raises a "NameError" exception. + +Deletion of attribute references, subscriptions and slicings is passed +to the primary object involved; deletion of a slicing is in general +equivalent to assignment of an empty slice of the right type (but even +this is determined by the sliced object). + +Changed in version 3.2: Previously it was illegal to delete a name +from the local namespace if it occurs as a free variable in a nested +block. +''', + 'dict': r'''Dictionary displays +******************* + +A dictionary display is a possibly empty series of dict items +(key/value pairs) enclosed in curly braces: + + dict_display: "{" [dict_item_list | dict_comprehension] "}" + dict_item_list: dict_item ("," dict_item)* [","] + dict_item: expression ":" expression | "**" or_expr + dict_comprehension: expression ":" expression comp_for + +A dictionary display yields a new dictionary object. + +If a comma-separated sequence of dict items is given, they are +evaluated from left to right to define the entries of the dictionary: +each key object is used as a key into the dictionary to store the +corresponding value. This means that you can specify the same key +multiple times in the dict item list, and the final dictionary’s value +for that key will be the last one given. + +A double asterisk "**" denotes *dictionary unpacking*. Its operand +must be a *mapping*. Each mapping item is added to the new +dictionary. Later values replace values already set by earlier dict +items and earlier dictionary unpackings. + +Added in version 3.5: Unpacking into dictionary displays, originally +proposed by **PEP 448**. + +A dict comprehension, in contrast to list and set comprehensions, +needs two expressions separated with a colon followed by the usual +“for” and “if” clauses. When the comprehension is run, the resulting +key and value elements are inserted in the new dictionary in the order +they are produced. + +Restrictions on the types of the key values are listed earlier in +section The standard type hierarchy. (To summarize, the key type +should be *hashable*, which excludes all mutable objects.) Clashes +between duplicate keys are not detected; the last value (textually +rightmost in the display) stored for a given key value prevails. + +Changed in version 3.8: Prior to Python 3.8, in dict comprehensions, +the evaluation order of key and value was not well-defined. In +CPython, the value was evaluated before the key. Starting with 3.8, +the key is evaluated before the value, as proposed by **PEP 572**. +''', + 'dynamic-features': r'''Interaction with dynamic features +********************************* + +Name resolution of free variables occurs at runtime, not at compile +time. This means that the following code will print 42: + + i = 10 + def f(): + print(i) + i = 42 + f() + +The "eval()" and "exec()" functions do not have access to the full +environment for resolving names. Names may be resolved in the local +and global namespaces of the caller. Free variables are not resolved +in the nearest enclosing namespace, but in the global namespace. [1] +The "exec()" and "eval()" functions have optional arguments to +override the global and local namespace. If only one namespace is +specified, it is used for both. +''', + 'else': r'''The "if" statement +****************** + +The "if" statement is used for conditional execution: + + if_stmt: "if" assignment_expression ":" suite + ("elif" assignment_expression ":" suite)* + ["else" ":" suite] + +It selects exactly one of the suites by evaluating the expressions one +by one until one is found to be true (see section Boolean operations +for the definition of true and false); then that suite is executed +(and no other part of the "if" statement is executed or evaluated). +If all expressions are false, the suite of the "else" clause, if +present, is executed. +''', + 'exceptions': r'''Exceptions +********** + +Exceptions are a means of breaking out of the normal flow of control +of a code block in order to handle errors or other exceptional +conditions. An exception is *raised* at the point where the error is +detected; it may be *handled* by the surrounding code block or by any +code block that directly or indirectly invoked the code block where +the error occurred. + +The Python interpreter raises an exception when it detects a run-time +error (such as division by zero). A Python program can also +explicitly raise an exception with the "raise" statement. Exception +handlers are specified with the "try" … "except" statement. The +"finally" clause of such a statement can be used to specify cleanup +code which does not handle the exception, but is executed whether an +exception occurred or not in the preceding code. + +Python uses the “termination” model of error handling: an exception +handler can find out what happened and continue execution at an outer +level, but it cannot repair the cause of the error and retry the +failing operation (except by re-entering the offending piece of code +from the top). + +When an exception is not handled at all, the interpreter terminates +execution of the program, or returns to its interactive main loop. In +either case, it prints a stack traceback, except when the exception is +"SystemExit". + +Exceptions are identified by class instances. The "except" clause is +selected depending on the class of the instance: it must reference the +class of the instance or a *non-virtual base class* thereof. The +instance can be received by the handler and can carry additional +information about the exceptional condition. + +Note: + + Exception messages are not part of the Python API. Their contents + may change from one version of Python to the next without warning + and should not be relied on by code which will run under multiple + versions of the interpreter. + +See also the description of the "try" statement in section The try +statement and "raise" statement in section The raise statement. +''', + 'execmodel': r'''Execution model +*************** + + +Structure of a program +====================== + +A Python program is constructed from code blocks. A *block* is a piece +of Python program text that is executed as a unit. The following are +blocks: a module, a function body, and a class definition. Each +command typed interactively is a block. A script file (a file given +as standard input to the interpreter or specified as a command line +argument to the interpreter) is a code block. A script command (a +command specified on the interpreter command line with the "-c" +option) is a code block. A module run as a top level script (as module +"__main__") from the command line using a "-m" argument is also a code +block. The string argument passed to the built-in functions "eval()" +and "exec()" is a code block. + +A code block is executed in an *execution frame*. A frame contains +some administrative information (used for debugging) and determines +where and how execution continues after the code block’s execution has +completed. + + +Naming and binding +================== + + +Binding of names +---------------- + +*Names* refer to objects. Names are introduced by name binding +operations. + +The following constructs bind names: + +* formal parameters to functions, + +* class definitions, + +* function definitions, + +* assignment expressions, + +* targets that are identifiers if occurring in an assignment: + + * "for" loop header, + + * after "as" in a "with" statement, "except" clause, "except*" + clause, or in the as-pattern in structural pattern matching, + + * in a capture pattern in structural pattern matching + +* "import" statements. + +* "type" statements. + +* type parameter lists. + +The "import" statement of the form "from ... import *" binds all names +defined in the imported module, except those beginning with an +underscore. This form may only be used at the module level. + +A target occurring in a "del" statement is also considered bound for +this purpose (though the actual semantics are to unbind the name). + +Each assignment or import statement occurs within a block defined by a +class or function definition or at the module level (the top-level +code block). + +If a name is bound in a block, it is a local variable of that block, +unless declared as "nonlocal" or "global". If a name is bound at the +module level, it is a global variable. (The variables of the module +code block are local and global.) If a variable is used in a code +block but not defined there, it is a *free variable*. + +Each occurrence of a name in the program text refers to the *binding* +of that name established by the following name resolution rules. + + +Resolution of names +------------------- + +A *scope* defines the visibility of a name within a block. If a local +variable is defined in a block, its scope includes that block. If the +definition occurs in a function block, the scope extends to any blocks +contained within the defining one, unless a contained block introduces +a different binding for the name. + +When a name is used in a code block, it is resolved using the nearest +enclosing scope. The set of all such scopes visible to a code block +is called the block’s *environment*. + +When a name is not found at all, a "NameError" exception is raised. If +the current scope is a function scope, and the name refers to a local +variable that has not yet been bound to a value at the point where the +name is used, an "UnboundLocalError" exception is raised. +"UnboundLocalError" is a subclass of "NameError". + +If a name binding operation occurs anywhere within a code block, all +uses of the name within the block are treated as references to the +current block. This can lead to errors when a name is used within a +block before it is bound. This rule is subtle. Python lacks +declarations and allows name binding operations to occur anywhere +within a code block. The local variables of a code block can be +determined by scanning the entire text of the block for name binding +operations. See the FAQ entry on UnboundLocalError for examples. + +If the "global" statement occurs within a block, all uses of the names +specified in the statement refer to the bindings of those names in the +top-level namespace. Names are resolved in the top-level namespace by +searching the global namespace, i.e. the namespace of the module +containing the code block, and the builtins namespace, the namespace +of the module "builtins". The global namespace is searched first. If +the names are not found there, the builtins namespace is searched +next. If the names are also not found in the builtins namespace, new +variables are created in the global namespace. The global statement +must precede all uses of the listed names. + +The "global" statement has the same scope as a name binding operation +in the same block. If the nearest enclosing scope for a free variable +contains a global statement, the free variable is treated as a global. + +The "nonlocal" statement causes corresponding names to refer to +previously bound variables in the nearest enclosing function scope. +"SyntaxError" is raised at compile time if the given name does not +exist in any enclosing function scope. Type parameters cannot be +rebound with the "nonlocal" statement. + +The namespace for a module is automatically created the first time a +module is imported. The main module for a script is always called +"__main__". + +Class definition blocks and arguments to "exec()" and "eval()" are +special in the context of name resolution. A class definition is an +executable statement that may use and define names. These references +follow the normal rules for name resolution with an exception that +unbound local variables are looked up in the global namespace. The +namespace of the class definition becomes the attribute dictionary of +the class. The scope of names defined in a class block is limited to +the class block; it does not extend to the code blocks of methods. +This includes comprehensions and generator expressions, but it does +not include annotation scopes, which have access to their enclosing +class scopes. This means that the following will fail: + + class A: + a = 42 + b = list(a + i for i in range(10)) + +However, the following will succeed: + + class A: + type Alias = Nested + class Nested: pass + + print(A.Alias.__value__) # + + +Annotation scopes +----------------- + +*Annotations*, type parameter lists and "type" statements introduce +*annotation scopes*, which behave mostly like function scopes, but +with some exceptions discussed below. + +Annotation scopes are used in the following contexts: + +* *Function annotations*. + +* *Variable annotations*. + +* Type parameter lists for generic type aliases. + +* Type parameter lists for generic functions. A generic function’s + annotations are executed within the annotation scope, but its + defaults and decorators are not. + +* Type parameter lists for generic classes. A generic class’s base + classes and keyword arguments are executed within the annotation + scope, but its decorators are not. + +* The bounds, constraints, and default values for type parameters + (lazily evaluated). + +* The value of type aliases (lazily evaluated). + +Annotation scopes differ from function scopes in the following ways: + +* Annotation scopes have access to their enclosing class namespace. If + an annotation scope is immediately within a class scope, or within + another annotation scope that is immediately within a class scope, + the code in the annotation scope can use names defined in the class + scope as if it were executed directly within the class body. This + contrasts with regular functions defined within classes, which + cannot access names defined in the class scope. + +* Expressions in annotation scopes cannot contain "yield", "yield + from", "await", or ":=" expressions. (These expressions are allowed + in other scopes contained within the annotation scope.) + +* Names defined in annotation scopes cannot be rebound with "nonlocal" + statements in inner scopes. This includes only type parameters, as + no other syntactic elements that can appear within annotation scopes + can introduce new names. + +* While annotation scopes have an internal name, that name is not + reflected in the *qualified name* of objects defined within the + scope. Instead, the "__qualname__" of such objects is as if the + object were defined in the enclosing scope. + +Added in version 3.12: Annotation scopes were introduced in Python +3.12 as part of **PEP 695**. + +Changed in version 3.13: Annotation scopes are also used for type +parameter defaults, as introduced by **PEP 696**. + +Changed in version 3.14: Annotation scopes are now also used for +annotations, as specified in **PEP 649** and **PEP 749**. + + +Lazy evaluation +--------------- + +Most annotation scopes are *lazily evaluated*. This includes +annotations, the values of type aliases created through the "type" +statement, and the bounds, constraints, and default values of type +variables created through the type parameter syntax. This means that +they are not evaluated when the type alias or type variable is +created, or when the object carrying annotations is created. Instead, +they are only evaluated when necessary, for example when the +"__value__" attribute on a type alias is accessed. + +Example: + + >>> type Alias = 1/0 + >>> Alias.__value__ + Traceback (most recent call last): + ... + ZeroDivisionError: division by zero + >>> def func[T: 1/0](): pass + >>> T = func.__type_params__[0] + >>> T.__bound__ + Traceback (most recent call last): + ... + ZeroDivisionError: division by zero + +Here the exception is raised only when the "__value__" attribute of +the type alias or the "__bound__" attribute of the type variable is +accessed. + +This behavior is primarily useful for references to types that have +not yet been defined when the type alias or type variable is created. +For example, lazy evaluation enables creation of mutually recursive +type aliases: + + from typing import Literal + + type SimpleExpr = int | Parenthesized + type Parenthesized = tuple[Literal["("], Expr, Literal[")"]] + type Expr = SimpleExpr | tuple[SimpleExpr, Literal["+", "-"], Expr] + +Lazily evaluated values are evaluated in annotation scope, which means +that names that appear inside the lazily evaluated value are looked up +as if they were used in the immediately enclosing scope. + +Added in version 3.12. + + +Builtins and restricted execution +--------------------------------- + +**CPython implementation detail:** Users should not touch +"__builtins__"; it is strictly an implementation detail. Users +wanting to override values in the builtins namespace should "import" +the "builtins" module and modify its attributes appropriately. + +The builtins namespace associated with the execution of a code block +is actually found by looking up the name "__builtins__" in its global +namespace; this should be a dictionary or a module (in the latter case +the module’s dictionary is used). By default, when in the "__main__" +module, "__builtins__" is the built-in module "builtins"; when in any +other module, "__builtins__" is an alias for the dictionary of the +"builtins" module itself. + + +Interaction with dynamic features +--------------------------------- + +Name resolution of free variables occurs at runtime, not at compile +time. This means that the following code will print 42: + + i = 10 + def f(): + print(i) + i = 42 + f() + +The "eval()" and "exec()" functions do not have access to the full +environment for resolving names. Names may be resolved in the local +and global namespaces of the caller. Free variables are not resolved +in the nearest enclosing namespace, but in the global namespace. [1] +The "exec()" and "eval()" functions have optional arguments to +override the global and local namespace. If only one namespace is +specified, it is used for both. + + +Exceptions +========== + +Exceptions are a means of breaking out of the normal flow of control +of a code block in order to handle errors or other exceptional +conditions. An exception is *raised* at the point where the error is +detected; it may be *handled* by the surrounding code block or by any +code block that directly or indirectly invoked the code block where +the error occurred. + +The Python interpreter raises an exception when it detects a run-time +error (such as division by zero). A Python program can also +explicitly raise an exception with the "raise" statement. Exception +handlers are specified with the "try" … "except" statement. The +"finally" clause of such a statement can be used to specify cleanup +code which does not handle the exception, but is executed whether an +exception occurred or not in the preceding code. + +Python uses the “termination” model of error handling: an exception +handler can find out what happened and continue execution at an outer +level, but it cannot repair the cause of the error and retry the +failing operation (except by re-entering the offending piece of code +from the top). + +When an exception is not handled at all, the interpreter terminates +execution of the program, or returns to its interactive main loop. In +either case, it prints a stack traceback, except when the exception is +"SystemExit". + +Exceptions are identified by class instances. The "except" clause is +selected depending on the class of the instance: it must reference the +class of the instance or a *non-virtual base class* thereof. The +instance can be received by the handler and can carry additional +information about the exceptional condition. + +Note: + + Exception messages are not part of the Python API. Their contents + may change from one version of Python to the next without warning + and should not be relied on by code which will run under multiple + versions of the interpreter. + +See also the description of the "try" statement in section The try +statement and "raise" statement in section The raise statement. + + +Runtime Components +================== + + +General Computing Model +----------------------- + +Python’s execution model does not operate in a vacuum. It runs on a +host machine and through that host’s runtime environment, including +its operating system (OS), if there is one. When a program runs, the +conceptual layers of how it runs on the host look something like this: + + **host machine** + **process** (global resources) + **thread** (runs machine code) + +Each process represents a program running on the host. Think of each +process itself as the data part of its program. Think of the process’ +threads as the execution part of the program. This distinction will +be important to understand the conceptual Python runtime. + +The process, as the data part, is the execution context in which the +program runs. It mostly consists of the set of resources assigned to +the program by the host, including memory, signals, file handles, +sockets, and environment variables. + +Processes are isolated and independent from one another. (The same is +true for hosts.) The host manages the process’ access to its assigned +resources, in addition to coordinating between processes. + +Each thread represents the actual execution of the program’s machine +code, running relative to the resources assigned to the program’s +process. It’s strictly up to the host how and when that execution +takes place. + +From the point of view of Python, a program always starts with exactly +one thread. However, the program may grow to run in multiple +simultaneous threads. Not all hosts support multiple threads per +process, but most do. Unlike processes, threads in a process are not +isolated and independent from one another. Specifically, all threads +in a process share all of the process’ resources. + +The fundamental point of threads is that each one does *run* +independently, at the same time as the others. That may be only +conceptually at the same time (“concurrently”) or physically (“in +parallel”). Either way, the threads effectively run at a non- +synchronized rate. + +Note: + + That non-synchronized rate means none of the process’ memory is + guaranteed to stay consistent for the code running in any given + thread. Thus multi-threaded programs must take care to coordinate + access to intentionally shared resources. Likewise, they must take + care to be absolutely diligent about not accessing any *other* + resources in multiple threads; otherwise two threads running at the + same time might accidentally interfere with each other’s use of some + shared data. All this is true for both Python programs and the + Python runtime.The cost of this broad, unstructured requirement is + the tradeoff for the kind of raw concurrency that threads provide. + The alternative to the required discipline generally means dealing + with non-deterministic bugs and data corruption. + + +Python Runtime Model +-------------------- + +The same conceptual layers apply to each Python program, with some +extra data layers specific to Python: + + **host machine** + **process** (global resources) + Python global runtime (*state*) + Python interpreter (*state*) + **thread** (runs Python bytecode and “C-API”) + Python thread *state* + +At the conceptual level: when a Python program starts, it looks +exactly like that diagram, with one of each. The runtime may grow to +include multiple interpreters, and each interpreter may grow to +include multiple thread states. + +Note: + + A Python implementation won’t necessarily implement the runtime + layers distinctly or even concretely. The only exception is places + where distinct layers are directly specified or exposed to users, + like through the "threading" module. + +Note: + + The initial interpreter is typically called the “main” interpreter. + Some Python implementations, like CPython, assign special roles to + the main interpreter.Likewise, the host thread where the runtime was + initialized is known as the “main” thread. It may be different from + the process’ initial thread, though they are often the same. In + some cases “main thread” may be even more specific and refer to the + initial thread state. A Python runtime might assign specific + responsibilities to the main thread, such as handling signals. + +As a whole, the Python runtime consists of the global runtime state, +interpreters, and thread states. The runtime ensures all that state +stays consistent over its lifetime, particularly when used with +multiple host threads. + +The global runtime, at the conceptual level, is just a set of +interpreters. While those interpreters are otherwise isolated and +independent from one another, they may share some data or other +resources. The runtime is responsible for managing these global +resources safely. The actual nature and management of these resources +is implementation-specific. Ultimately, the external utility of the +global runtime is limited to managing interpreters. + +In contrast, an “interpreter” is conceptually what we would normally +think of as the (full-featured) “Python runtime”. When machine code +executing in a host thread interacts with the Python runtime, it calls +into Python in the context of a specific interpreter. + +Note: + + The term “interpreter” here is not the same as the “bytecode + interpreter”, which is what regularly runs in threads, executing + compiled Python code.In an ideal world, “Python runtime” would refer + to what we currently call “interpreter”. However, it’s been called + “interpreter” at least since introduced in 1997 (CPython:a027efa5b). + +Each interpreter completely encapsulates all of the non-process- +global, non-thread-specific state needed for the Python runtime to +work. Notably, the interpreter’s state persists between uses. It +includes fundamental data like "sys.modules". The runtime ensures +multiple threads using the same interpreter will safely share it +between them. + +A Python implementation may support using multiple interpreters at the +same time in the same process. They are independent and isolated from +one another. For example, each interpreter has its own "sys.modules". + +For thread-specific runtime state, each interpreter has a set of +thread states, which it manages, in the same way the global runtime +contains a set of interpreters. It can have thread states for as many +host threads as it needs. It may even have multiple thread states for +the same host thread, though that isn’t as common. + +Each thread state, conceptually, has all the thread-specific runtime +data an interpreter needs to operate in one host thread. The thread +state includes the current raised exception and the thread’s Python +call stack. It may include other thread-specific resources. + +Note: + + The term “Python thread” can sometimes refer to a thread state, but + normally it means a thread created using the "threading" module. + +Each thread state, over its lifetime, is always tied to exactly one +interpreter and exactly one host thread. It will only ever be used in +that thread and with that interpreter. + +Multiple thread states may be tied to the same host thread, whether +for different interpreters or even the same interpreter. However, for +any given host thread, only one of the thread states tied to it can be +used by the thread at a time. + +Thread states are isolated and independent from one another and don’t +share any data, except for possibly sharing an interpreter and objects +or other resources belonging to that interpreter. + +Once a program is running, new Python threads can be created using the +"threading" module (on platforms and Python implementations that +support threads). Additional processes can be created using the "os", +"subprocess", and "multiprocessing" modules. Interpreters can be +created and used with the "interpreters" module. Coroutines (async) +can be run using "asyncio" in each interpreter, typically only in a +single thread (often the main thread). + +-[ Footnotes ]- + +[1] This limitation occurs because the code that is executed by these + operations is not available at the time the module is compiled. +''', + 'exprlists': r'''Expression lists +**************** + + starred_expression: "*" or_expr | expression + flexible_expression: assignment_expression | starred_expression + flexible_expression_list: flexible_expression ("," flexible_expression)* [","] + starred_expression_list: starred_expression ("," starred_expression)* [","] + expression_list: expression ("," expression)* [","] + yield_list: expression_list | starred_expression "," [starred_expression_list] + +Except when part of a list or set display, an expression list +containing at least one comma yields a tuple. The length of the tuple +is the number of expressions in the list. The expressions are +evaluated from left to right. + +An asterisk "*" denotes *iterable unpacking*. Its operand must be an +*iterable*. The iterable is expanded into a sequence of items, which +are included in the new tuple, list, or set, at the site of the +unpacking. + +Added in version 3.5: Iterable unpacking in expression lists, +originally proposed by **PEP 448**. + +Added in version 3.11: Any item in an expression list may be starred. +See **PEP 646**. + +A trailing comma is required only to create a one-item tuple, such as +"1,"; it is optional in all other cases. A single expression without a +trailing comma doesn’t create a tuple, but rather yields the value of +that expression. (To create an empty tuple, use an empty pair of +parentheses: "()".) +''', + 'floating': r'''Floating-point literals +*********************** + +Floating-point (float) literals, such as "3.14" or "1.5", denote +approximations of real numbers. + +They consist of *integer* and *fraction* parts, each composed of +decimal digits. The parts are separated by a decimal point, ".": + + 2.71828 + 4.0 + +Unlike in integer literals, leading zeros are allowed. For example, +"077.010" is legal, and denotes the same number as "77.01". + +As in integer literals, single underscores may occur between digits to +help readability: + + 96_485.332_123 + 3.14_15_93 + +Either of these parts, but not both, can be empty. For example: + + 10. # (equivalent to 10.0) + .001 # (equivalent to 0.001) + +Optionally, the integer and fraction may be followed by an *exponent*: +the letter "e" or "E", followed by an optional sign, "+" or "-", and a +number in the same format as the integer and fraction parts. The "e" +or "E" represents “times ten raised to the power of”: + + 1.0e3 # (represents 1.0×10³, or 1000.0) + 1.166e-5 # (represents 1.166×10⁻⁵, or 0.00001166) + 6.02214076e+23 # (represents 6.02214076×10²³, or 602214076000000000000000.) + +In floats with only integer and exponent parts, the decimal point may +be omitted: + + 1e3 # (equivalent to 1.e3 and 1.0e3) + 0e0 # (equivalent to 0.) + +Formally, floating-point literals are described by the following +lexical definitions: + + floatnumber: + | digitpart "." [digitpart] [exponent] + | "." digitpart [exponent] + | digitpart exponent + digitpart: digit (["_"] digit)* + exponent: ("e" | "E") ["+" | "-"] digitpart + +Changed in version 3.6: Underscores are now allowed for grouping +purposes in literals. +''', + 'for': r'''The "for" statement +******************* + +The "for" statement is used to iterate over the elements of a sequence +(such as a string, tuple or list) or other iterable object: + + for_stmt: "for" target_list "in" starred_expression_list ":" suite + ["else" ":" suite] + +The "starred_expression_list" expression is evaluated once; it should +yield an *iterable* object. An *iterator* is created for that +iterable. The first item provided by the iterator is then assigned to +the target list using the standard rules for assignments (see +Assignment statements), and the suite is executed. This repeats for +each item provided by the iterator. When the iterator is exhausted, +the suite in the "else" clause, if present, is executed, and the loop +terminates. + +A "break" statement executed in the first suite terminates the loop +without executing the "else" clause’s suite. A "continue" statement +executed in the first suite skips the rest of the suite and continues +with the next item, or with the "else" clause if there is no next +item. + +The for-loop makes assignments to the variables in the target list. +This overwrites all previous assignments to those variables including +those made in the suite of the for-loop: + + for i in range(10): + print(i) + i = 5 # this will not affect the for-loop + # because i will be overwritten with the next + # index in the range + +Names in the target list are not deleted when the loop is finished, +but if the sequence is empty, they will not have been assigned to at +all by the loop. Hint: the built-in type "range()" represents +immutable arithmetic sequences of integers. For instance, iterating +"range(3)" successively yields 0, 1, and then 2. + +Changed in version 3.11: Starred elements are now allowed in the +expression list. +''', + 'formatstrings': r'''Format String Syntax +******************** + +The "str.format()" method and the "Formatter" class share the same +syntax for format strings (although in the case of "Formatter", +subclasses can define their own format string syntax). The syntax is +related to that of formatted string literals and template string +literals, but it is less sophisticated and, in particular, does not +support arbitrary expressions in interpolations. + +Format strings contain “replacement fields” surrounded by curly braces +"{}". Anything that is not contained in braces is considered literal +text, which is copied unchanged to the output. If you need to include +a brace character in the literal text, it can be escaped by doubling: +"{{" and "}}". + +The grammar for a replacement field is as follows: + + replacement_field: "{" [field_name] ["!" conversion] [":" format_spec] "}" + field_name: arg_name ("." attribute_name | "[" element_index "]")* + arg_name: [identifier | digit+] + attribute_name: identifier + element_index: digit+ | index_string + index_string: + + conversion: "r" | "s" | "a" + format_spec: format-spec:format_spec + +In less formal terms, the replacement field can start with a +*field_name* that specifies the object whose value is to be formatted +and inserted into the output instead of the replacement field. The +*field_name* is optionally followed by a *conversion* field, which is +preceded by an exclamation point "'!'", and a *format_spec*, which is +preceded by a colon "':'". These specify a non-default format for the +replacement value. + +See also the Format Specification Mini-Language section. + +The *field_name* itself begins with an *arg_name* that is either a +number or a keyword. If it’s a number, it refers to a positional +argument, and if it’s a keyword, it refers to a named keyword +argument. An *arg_name* is treated as a number if a call to +"str.isdecimal()" on the string would return true. If the numerical +arg_names in a format string are 0, 1, 2, … in sequence, they can all +be omitted (not just some) and the numbers 0, 1, 2, … will be +automatically inserted in that order. Because *arg_name* is not quote- +delimited, it is not possible to specify arbitrary dictionary keys +(e.g., the strings "'10'" or "':-]'") within a format string. The +*arg_name* can be followed by any number of index or attribute +expressions. An expression of the form "'.name'" selects the named +attribute using "getattr()", while an expression of the form +"'[index]'" does an index lookup using "__getitem__()". + +Changed in version 3.1: The positional argument specifiers can be +omitted for "str.format()", so "'{} {}'.format(a, b)" is equivalent to +"'{0} {1}'.format(a, b)". + +Changed in version 3.4: The positional argument specifiers can be +omitted for "Formatter". + +Some simple format string examples: + + "First, thou shalt count to {0}" # References first positional argument + "Bring me a {}" # Implicitly references the first positional argument + "From {} to {}" # Same as "From {0} to {1}" + "My quest is {name}" # References keyword argument 'name' + "Weight in tons {0.weight}" # 'weight' attribute of first positional arg + "Units destroyed: {players[0]}" # First element of keyword argument 'players'. + +The *conversion* field causes a type coercion before formatting. +Normally, the job of formatting a value is done by the "__format__()" +method of the value itself. However, in some cases it is desirable to +force a type to be formatted as a string, overriding its own +definition of formatting. By converting the value to a string before +calling "__format__()", the normal formatting logic is bypassed. + +Three conversion flags are currently supported: "'!s'" which calls +"str()" on the value, "'!r'" which calls "repr()" and "'!a'" which +calls "ascii()". + +Some examples: + + "Harold's a clever {0!s}" # Calls str() on the argument first + "Bring out the holy {name!r}" # Calls repr() on the argument first + "More {!a}" # Calls ascii() on the argument first + +The *format_spec* field contains a specification of how the value +should be presented, including such details as field width, alignment, +padding, decimal precision and so on. Each value type can define its +own “formatting mini-language” or interpretation of the *format_spec*. + +Most built-in types support a common formatting mini-language, which +is described in the next section. + +A *format_spec* field can also include nested replacement fields +within it. These nested replacement fields may contain a field name, +conversion flag and format specification, but deeper nesting is not +allowed. The replacement fields within the format_spec are +substituted before the *format_spec* string is interpreted. This +allows the formatting of a value to be dynamically specified. + +See the Format examples section for some examples. + + +Format Specification Mini-Language +================================== + +“Format specifications” are used within replacement fields contained +within a format string to define how individual values are presented +(see Format String Syntax, f-strings, and t-strings). They can also be +passed directly to the built-in "format()" function. Each formattable +type may define how the format specification is to be interpreted. + +Most built-in types implement the following options for format +specifications, although some of the formatting options are only +supported by the numeric types. + +A general convention is that an empty format specification produces +the same result as if you had called "str()" on the value. A non-empty +format specification typically modifies the result. + +The general form of a *standard format specifier* is: + + format_spec: [options][width_and_precision][type] + options: [[fill]align][sign]["z"]["#"]["0"] + fill: + align: "<" | ">" | "=" | "^" + sign: "+" | "-" | " " + width_and_precision: [width_with_grouping][precision_with_grouping] + width_with_grouping: [width][grouping] + precision_with_grouping: "." [precision][grouping] | "." grouping + width: digit+ + precision: digit+ + grouping: "," | "_" + type: "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" + | "G" | "n" | "o" | "s" | "x" | "X" | "%" + +If a valid *align* value is specified, it can be preceded by a *fill* +character that can be any character and defaults to a space if +omitted. It is not possible to use a literal curly brace (”"{"” or +“"}"”) as the *fill* character in a formatted string literal or when +using the "str.format()" method. However, it is possible to insert a +curly brace with a nested replacement field. This limitation doesn’t +affect the "format()" function. + +The meaning of the various alignment options is as follows: + ++-----------+------------------------------------------------------------+ +| Option | Meaning | +|===========|============================================================| +| "'<'" | Forces the field to be left-aligned within the available | +| | space (this is the default for most objects). | ++-----------+------------------------------------------------------------+ +| "'>'" | Forces the field to be right-aligned within the available | +| | space (this is the default for numbers). | ++-----------+------------------------------------------------------------+ +| "'='" | Forces the padding to be placed after the sign (if any) | +| | but before the digits. This is used for printing fields | +| | in the form ‘+000000120’. This alignment option is only | +| | valid for numeric types, excluding "complex". It becomes | +| | the default for numbers when ‘0’ immediately precedes the | +| | field width. | ++-----------+------------------------------------------------------------+ +| "'^'" | Forces the field to be centered within the available | +| | space. | ++-----------+------------------------------------------------------------+ + +Note that unless a minimum field width is defined, the field width +will always be the same size as the data to fill it, so that the +alignment option has no meaning in this case. + +The *sign* option is only valid for number types, and can be one of +the following: + ++-----------+------------------------------------------------------------+ +| Option | Meaning | +|===========|============================================================| +| "'+'" | Indicates that a sign should be used for both positive as | +| | well as negative numbers. | ++-----------+------------------------------------------------------------+ +| "'-'" | Indicates that a sign should be used only for negative | +| | numbers (this is the default behavior). | ++-----------+------------------------------------------------------------+ +| space | Indicates that a leading space should be used on positive | +| | numbers, and a minus sign on negative numbers. | ++-----------+------------------------------------------------------------+ + +The "'z'" option coerces negative zero floating-point values to +positive zero after rounding to the format precision. This option is +only valid for floating-point presentation types. + +Changed in version 3.11: Added the "'z'" option (see also **PEP +682**). + +The "'#'" option causes the “alternate form” to be used for the +conversion. The alternate form is defined differently for different +types. This option is only valid for integer, float and complex +types. For integers, when binary, octal, or hexadecimal output is +used, this option adds the respective prefix "'0b'", "'0o'", "'0x'", +or "'0X'" to the output value. For float and complex the alternate +form causes the result of the conversion to always contain a decimal- +point character, even if no digits follow it. Normally, a decimal- +point character appears in the result of these conversions only if a +digit follows it. In addition, for "'g'" and "'G'" conversions, +trailing zeros are not removed from the result. + +The *width* is a decimal integer defining the minimum total field +width, including any prefixes, separators, and other formatting +characters. If not specified, then the field width will be determined +by the content. + +When no explicit alignment is given, preceding the *width* field by a +zero ("'0'") character enables sign-aware zero-padding for numeric +types, excluding "complex". This is equivalent to a *fill* character +of "'0'" with an *alignment* type of "'='". + +Changed in version 3.10: Preceding the *width* field by "'0'" no +longer affects the default alignment for strings. + +The *precision* is a decimal integer indicating how many digits should +be displayed after the decimal point for presentation types "'f'" and +"'F'", or before and after the decimal point for presentation types +"'g'" or "'G'". For string presentation types the field indicates the +maximum field size - in other words, how many characters will be used +from the field content. The *precision* is not allowed for integer +presentation types. + +The *grouping* option after *width* and *precision* fields specifies a +digit group separator for the integral and fractional parts of a +number respectively. It can be one of the following: + ++-----------+------------------------------------------------------------+ +| Option | Meaning | +|===========|============================================================| +| "','" | Inserts a comma every 3 digits for integer presentation | +| | type "'d'" and floating-point presentation types, | +| | excluding "'n'". For other presentation types, this option | +| | is not supported. | ++-----------+------------------------------------------------------------+ +| "'_'" | Inserts an underscore every 3 digits for integer | +| | presentation type "'d'" and floating-point presentation | +| | types, excluding "'n'". For integer presentation types | +| | "'b'", "'o'", "'x'", and "'X'", underscores are inserted | +| | every 4 digits. For other presentation types, this option | +| | is not supported. | ++-----------+------------------------------------------------------------+ + +For a locale aware separator, use the "'n'" presentation type instead. + +Changed in version 3.1: Added the "','" option (see also **PEP 378**). + +Changed in version 3.6: Added the "'_'" option (see also **PEP 515**). + +Changed in version 3.14: Support the *grouping* option for the +fractional part. + +Finally, the *type* determines how the data should be presented. + +The available string presentation types are: + + +-----------+------------------------------------------------------------+ + | Type | Meaning | + |===========|============================================================| + | "'s'" | String format. This is the default type for strings and | + | | may be omitted. | + +-----------+------------------------------------------------------------+ + | None | The same as "'s'". | + +-----------+------------------------------------------------------------+ + +The available integer presentation types are: + + +-----------+------------------------------------------------------------+ + | Type | Meaning | + |===========|============================================================| + | "'b'" | Binary format. Outputs the number in base 2. | + +-----------+------------------------------------------------------------+ + | "'c'" | Character. Converts the integer to the corresponding | + | | unicode character before printing. | + +-----------+------------------------------------------------------------+ + | "'d'" | Decimal Integer. Outputs the number in base 10. | + +-----------+------------------------------------------------------------+ + | "'o'" | Octal format. Outputs the number in base 8. | + +-----------+------------------------------------------------------------+ + | "'x'" | Hex format. Outputs the number in base 16, using lower- | + | | case letters for the digits above 9. | + +-----------+------------------------------------------------------------+ + | "'X'" | Hex format. Outputs the number in base 16, using upper- | + | | case letters for the digits above 9. In case "'#'" is | + | | specified, the prefix "'0x'" will be upper-cased to "'0X'" | + | | as well. | + +-----------+------------------------------------------------------------+ + | "'n'" | Number. This is the same as "'d'", except that it uses the | + | | current locale setting to insert the appropriate digit | + | | group separators. | + +-----------+------------------------------------------------------------+ + | None | The same as "'d'". | + +-----------+------------------------------------------------------------+ + +In addition to the above presentation types, integers can be formatted +with the floating-point presentation types listed below (except "'n'" +and "None"). When doing so, "float()" is used to convert the integer +to a floating-point number before formatting. + +The available presentation types for "float" and "Decimal" values are: + + +-----------+------------------------------------------------------------+ + | Type | Meaning | + |===========|============================================================| + | "'e'" | Scientific notation. For a given precision "p", formats | + | | the number in scientific notation with the letter ‘e’ | + | | separating the coefficient from the exponent. The | + | | coefficient has one digit before and "p" digits after the | + | | decimal point, for a total of "p + 1" significant digits. | + | | With no precision given, uses a precision of "6" digits | + | | after the decimal point for "float", and shows all | + | | coefficient digits for "Decimal". If "p=0", the decimal | + | | point is omitted unless the "#" option is used. | + +-----------+------------------------------------------------------------+ + | "'E'" | Scientific notation. Same as "'e'" except it uses an upper | + | | case ‘E’ as the separator character. | + +-----------+------------------------------------------------------------+ + | "'f'" | Fixed-point notation. For a given precision "p", formats | + | | the number as a decimal number with exactly "p" digits | + | | following the decimal point. With no precision given, uses | + | | a precision of "6" digits after the decimal point for | + | | "float", and uses a precision large enough to show all | + | | coefficient digits for "Decimal". If "p=0", the decimal | + | | point is omitted unless the "#" option is used. | + +-----------+------------------------------------------------------------+ + | "'F'" | Fixed-point notation. Same as "'f'", but converts "nan" to | + | | "NAN" and "inf" to "INF". | + +-----------+------------------------------------------------------------+ + | "'g'" | General format. For a given precision "p >= 1", this | + | | rounds the number to "p" significant digits and then | + | | formats the result in either fixed-point format or in | + | | scientific notation, depending on its magnitude. A | + | | precision of "0" is treated as equivalent to a precision | + | | of "1". The precise rules are as follows: suppose that | + | | the result formatted with presentation type "'e'" and | + | | precision "p-1" would have exponent "exp". Then, if "m <= | + | | exp < p", where "m" is -4 for floats and -6 for | + | | "Decimals", the number is formatted with presentation type | + | | "'f'" and precision "p-1-exp". Otherwise, the number is | + | | formatted with presentation type "'e'" and precision | + | | "p-1". In both cases insignificant trailing zeros are | + | | removed from the significand, and the decimal point is | + | | also removed if there are no remaining digits following | + | | it, unless the "'#'" option is used. With no precision | + | | given, uses a precision of "6" significant digits for | + | | "float". For "Decimal", the coefficient of the result is | + | | formed from the coefficient digits of the value; | + | | scientific notation is used for values smaller than "1e-6" | + | | in absolute value and values where the place value of the | + | | least significant digit is larger than 1, and fixed-point | + | | notation is used otherwise. Positive and negative | + | | infinity, positive and negative zero, and nans, are | + | | formatted as "inf", "-inf", "0", "-0" and "nan" | + | | respectively, regardless of the precision. | + +-----------+------------------------------------------------------------+ + | "'G'" | General format. Same as "'g'" except switches to "'E'" if | + | | the number gets too large. The representations of infinity | + | | and NaN are uppercased, too. | + +-----------+------------------------------------------------------------+ + | "'n'" | Number. This is the same as "'g'", except that it uses the | + | | current locale setting to insert the appropriate digit | + | | group separators for the integral part of a number. | + +-----------+------------------------------------------------------------+ + | "'%'" | Percentage. Multiplies the number by 100 and displays in | + | | fixed ("'f'") format, followed by a percent sign. | + +-----------+------------------------------------------------------------+ + | None | For "float" this is like the "'g'" type, except that when | + | | fixed- point notation is used to format the result, it | + | | always includes at least one digit past the decimal point, | + | | and switches to the scientific notation when "exp >= p - | + | | 1". When the precision is not specified, the latter will | + | | be as large as needed to represent the given value | + | | faithfully. For "Decimal", this is the same as either | + | | "'g'" or "'G'" depending on the value of | + | | "context.capitals" for the current decimal context. The | + | | overall effect is to match the output of "str()" as | + | | altered by the other format modifiers. | + +-----------+------------------------------------------------------------+ + +The result should be correctly rounded to a given precision "p" of +digits after the decimal point. The rounding mode for "float" matches +that of the "round()" builtin. For "Decimal", the rounding mode of +the current context will be used. + +The available presentation types for "complex" are the same as those +for "float" ("'%'" is not allowed). Both the real and imaginary +components of a complex number are formatted as floating-point +numbers, according to the specified presentation type. They are +separated by the mandatory sign of the imaginary part, the latter +being terminated by a "j" suffix. If the presentation type is +missing, the result will match the output of "str()" (complex numbers +with a non-zero real part are also surrounded by parentheses), +possibly altered by other format modifiers. + + +Format examples +=============== + +This section contains examples of the "str.format()" syntax and +comparison with the old "%"-formatting. + +In most of the cases the syntax is similar to the old "%"-formatting, +with the addition of the "{}" and with ":" used instead of "%". For +example, "'%03.2f'" can be translated to "'{:03.2f}'". + +The new format syntax also supports new and different options, shown +in the following examples. + +Accessing arguments by position: + + >>> '{0}, {1}, {2}'.format('a', 'b', 'c') + 'a, b, c' + >>> '{}, {}, {}'.format('a', 'b', 'c') # 3.1+ only + 'a, b, c' + >>> '{2}, {1}, {0}'.format('a', 'b', 'c') + 'c, b, a' + >>> '{2}, {1}, {0}'.format(*'abc') # unpacking argument sequence + 'c, b, a' + >>> '{0}{1}{0}'.format('abra', 'cad') # arguments' indices can be repeated + 'abracadabra' + +Accessing arguments by name: + + >>> 'Coordinates: {latitude}, {longitude}'.format(latitude='37.24N', longitude='-115.81W') + 'Coordinates: 37.24N, -115.81W' + >>> coord = {'latitude': '37.24N', 'longitude': '-115.81W'} + >>> 'Coordinates: {latitude}, {longitude}'.format(**coord) + 'Coordinates: 37.24N, -115.81W' + +Accessing arguments’ attributes: + + >>> c = 3-5j + >>> ('The complex number {0} is formed from the real part {0.real} ' + ... 'and the imaginary part {0.imag}.').format(c) + 'The complex number (3-5j) is formed from the real part 3.0 and the imaginary part -5.0.' + >>> class Point: + ... def __init__(self, x, y): + ... self.x, self.y = x, y + ... def __str__(self): + ... return 'Point({self.x}, {self.y})'.format(self=self) + ... + >>> str(Point(4, 2)) + 'Point(4, 2)' + +Accessing arguments’ items: + + >>> coord = (3, 5) + >>> 'X: {0[0]}; Y: {0[1]}'.format(coord) + 'X: 3; Y: 5' + +Replacing "%s" and "%r": + + >>> "repr() shows quotes: {!r}; str() doesn't: {!s}".format('test1', 'test2') + "repr() shows quotes: 'test1'; str() doesn't: test2" + +Aligning the text and specifying a width: + + >>> '{:<30}'.format('left aligned') + 'left aligned ' + >>> '{:>30}'.format('right aligned') + ' right aligned' + >>> '{:^30}'.format('centered') + ' centered ' + >>> '{:*^30}'.format('centered') # use '*' as a fill char + '***********centered***********' + +Replacing "%+f", "%-f", and "% f" and specifying a sign: + + >>> '{:+f}; {:+f}'.format(3.14, -3.14) # show it always + '+3.140000; -3.140000' + >>> '{: f}; {: f}'.format(3.14, -3.14) # show a space for positive numbers + ' 3.140000; -3.140000' + >>> '{:-f}; {:-f}'.format(3.14, -3.14) # show only the minus -- same as '{:f}; {:f}' + '3.140000; -3.140000' + +Replacing "%x" and "%o" and converting the value to different bases: + + >>> # format also supports binary numbers + >>> "int: {0:d}; hex: {0:x}; oct: {0:o}; bin: {0:b}".format(42) + 'int: 42; hex: 2a; oct: 52; bin: 101010' + >>> # with 0x, 0o, or 0b as prefix: + >>> "int: {0:d}; hex: {0:#x}; oct: {0:#o}; bin: {0:#b}".format(42) + 'int: 42; hex: 0x2a; oct: 0o52; bin: 0b101010' + +Using the comma or the underscore as a digit group separator: + + >>> '{:,}'.format(1234567890) + '1,234,567,890' + >>> '{:_}'.format(1234567890) + '1_234_567_890' + >>> '{:_b}'.format(1234567890) + '100_1001_1001_0110_0000_0010_1101_0010' + >>> '{:_x}'.format(1234567890) + '4996_02d2' + >>> '{:_}'.format(123456789.123456789) + '123_456_789.12345679' + >>> '{:.,}'.format(123456789.123456789) + '123456789.123,456,79' + >>> '{:,._}'.format(123456789.123456789) + '123,456,789.123_456_79' + +Expressing a percentage: + + >>> points = 19 + >>> total = 22 + >>> 'Correct answers: {:.2%}'.format(points/total) + 'Correct answers: 86.36%' + +Using type-specific formatting: + + >>> import datetime + >>> d = datetime.datetime(2010, 7, 4, 12, 15, 58) + >>> '{:%Y-%m-%d %H:%M:%S}'.format(d) + '2010-07-04 12:15:58' + +Nesting arguments and more complex examples: + + >>> for align, text in zip('<^>', ['left', 'center', 'right']): + ... '{0:{fill}{align}16}'.format(text, fill=align, align=align) + ... + 'left<<<<<<<<<<<<' + '^^^^^center^^^^^' + '>>>>>>>>>>>right' + >>> + >>> octets = [192, 168, 0, 1] + >>> '{:02X}{:02X}{:02X}{:02X}'.format(*octets) + 'C0A80001' + >>> int(_, 16) + 3232235521 + >>> + >>> width = 5 + >>> for num in range(5,12): + ... for base in 'dXob': + ... print('{0:{width}{base}}'.format(num, base=base, width=width), end=' ') + ... print() + ... + 5 5 5 101 + 6 6 6 110 + 7 7 7 111 + 8 8 10 1000 + 9 9 11 1001 + 10 A 12 1010 + 11 B 13 1011 +''', + 'function': r'''Function definitions +******************** + +A function definition defines a user-defined function object (see +section The standard type hierarchy): + + funcdef: [decorators] "def" funcname [type_params] "(" [parameter_list] ")" + ["->" expression] ":" suite + decorators: decorator+ + decorator: "@" assignment_expression NEWLINE + parameter_list: defparameter ("," defparameter)* "," "/" ["," [parameter_list_no_posonly]] + | parameter_list_no_posonly + parameter_list_no_posonly: defparameter ("," defparameter)* ["," [parameter_list_starargs]] + | parameter_list_starargs + parameter_list_starargs: "*" [star_parameter] ("," defparameter)* ["," [parameter_star_kwargs]] + | "*" ("," defparameter)+ ["," [parameter_star_kwargs]] + | parameter_star_kwargs + parameter_star_kwargs: "**" parameter [","] + parameter: identifier [":" expression] + star_parameter: identifier [":" ["*"] expression] + defparameter: parameter ["=" expression] + funcname: identifier + +A function definition is an executable statement. Its execution binds +the function name in the current local namespace to a function object +(a wrapper around the executable code for the function). This +function object contains a reference to the current global namespace +as the global namespace to be used when the function is called. + +The function definition does not execute the function body; this gets +executed only when the function is called. [4] + +A function definition may be wrapped by one or more *decorator* +expressions. Decorator expressions are evaluated when the function is +defined, in the scope that contains the function definition. The +result must be a callable, which is invoked with the function object +as the only argument. The returned value is bound to the function name +instead of the function object. Multiple decorators are applied in +nested fashion. For example, the following code + + @f1(arg) + @f2 + def func(): pass + +is roughly equivalent to + + def func(): pass + func = f1(arg)(f2(func)) + +except that the original function is not temporarily bound to the name +"func". + +Changed in version 3.9: Functions may be decorated with any valid +"assignment_expression". Previously, the grammar was much more +restrictive; see **PEP 614** for details. + +A list of type parameters may be given in square brackets between the +function’s name and the opening parenthesis for its parameter list. +This indicates to static type checkers that the function is generic. +At runtime, the type parameters can be retrieved from the function’s +"__type_params__" attribute. See Generic functions for more. + +Changed in version 3.12: Type parameter lists are new in Python 3.12. + +When one or more *parameters* have the form *parameter* "=" +*expression*, the function is said to have “default parameter values.” +For a parameter with a default value, the corresponding *argument* may +be omitted from a call, in which case the parameter’s default value is +substituted. If a parameter has a default value, all following +parameters up until the “"*"” must also have a default value — this is +a syntactic restriction that is not expressed by the grammar. + +**Default parameter values are evaluated from left to right when the +function definition is executed.** This means that the expression is +evaluated once, when the function is defined, and that the same “pre- +computed” value is used for each call. This is especially important +to understand when a default parameter value is a mutable object, such +as a list or a dictionary: if the function modifies the object (e.g. +by appending an item to a list), the default parameter value is in +effect modified. This is generally not what was intended. A way +around this is to use "None" as the default, and explicitly test for +it in the body of the function, e.g.: + + def whats_on_the_telly(penguin=None): + if penguin is None: + penguin = [] + penguin.append("property of the zoo") + return penguin + +Function call semantics are described in more detail in section Calls. +A function call always assigns values to all parameters mentioned in +the parameter list, either from positional arguments, from keyword +arguments, or from default values. If the form “"*identifier"” is +present, it is initialized to a tuple receiving any excess positional +parameters, defaulting to the empty tuple. If the form +“"**identifier"” is present, it is initialized to a new ordered +mapping receiving any excess keyword arguments, defaulting to a new +empty mapping of the same type. Parameters after “"*"” or +“"*identifier"” are keyword-only parameters and may only be passed by +keyword arguments. Parameters before “"/"” are positional-only +parameters and may only be passed by positional arguments. + +Changed in version 3.8: The "/" function parameter syntax may be used +to indicate positional-only parameters. See **PEP 570** for details. + +Parameters may have an *annotation* of the form “": expression"” +following the parameter name. Any parameter may have an annotation, +even those of the form "*identifier" or "**identifier". (As a special +case, parameters of the form "*identifier" may have an annotation “": +*expression"”.) Functions may have “return” annotation of the form +“"-> expression"” after the parameter list. These annotations can be +any valid Python expression. The presence of annotations does not +change the semantics of a function. See Annotations for more +information on annotations. + +Changed in version 3.11: Parameters of the form “"*identifier"” may +have an annotation “": *expression"”. See **PEP 646**. + +It is also possible to create anonymous functions (functions not bound +to a name), for immediate use in expressions. This uses lambda +expressions, described in section Lambdas. Note that the lambda +expression is merely a shorthand for a simplified function definition; +a function defined in a “"def"” statement can be passed around or +assigned to another name just like a function defined by a lambda +expression. The “"def"” form is actually more powerful since it +allows the execution of multiple statements and annotations. + +**Programmer’s note:** Functions are first-class objects. A “"def"” +statement executed inside a function definition defines a local +function that can be returned or passed around. Free variables used +in the nested function can access the local variables of the function +containing the def. See section Naming and binding for details. + +See also: + + **PEP 3107** - Function Annotations + The original specification for function annotations. + + **PEP 484** - Type Hints + Definition of a standard meaning for annotations: type hints. + + **PEP 526** - Syntax for Variable Annotations + Ability to type hint variable declarations, including class + variables and instance variables. + + **PEP 563** - Postponed Evaluation of Annotations + Support for forward references within annotations by preserving + annotations in a string form at runtime instead of eager + evaluation. + + **PEP 318** - Decorators for Functions and Methods + Function and method decorators were introduced. Class decorators + were introduced in **PEP 3129**. +''', + 'global': r'''The "global" statement +********************** + + global_stmt: "global" identifier ("," identifier)* + +The "global" statement causes the listed identifiers to be interpreted +as globals. It would be impossible to assign to a global variable +without "global", although free variables may refer to globals without +being declared global. + +The "global" statement applies to the entire current scope (module, +function body or class definition). A "SyntaxError" is raised if a +variable is used or assigned to prior to its global declaration in the +scope. + +At the module level, all variables are global, so a "global" statement +has no effect. However, variables must still not be used or assigned +to prior to their "global" declaration. This requirement is relaxed in +the interactive prompt (*REPL*). + +**Programmer’s note:** "global" is a directive to the parser. It +applies only to code parsed at the same time as the "global" +statement. In particular, a "global" statement contained in a string +or code object supplied to the built-in "exec()" function does not +affect the code block *containing* the function call, and code +contained in such a string is unaffected by "global" statements in the +code containing the function call. The same applies to the "eval()" +and "compile()" functions. +''', + 'id-classes': r'''Reserved classes of identifiers +******************************* + +Certain classes of identifiers (besides keywords) have special +meanings. These classes are identified by the patterns of leading and +trailing underscore characters: + +"_*" + Not imported by "from module import *". + +"_" + In a "case" pattern within a "match" statement, "_" is a soft + keyword that denotes a wildcard. + + Separately, the interactive interpreter makes the result of the + last evaluation available in the variable "_". (It is stored in the + "builtins" module, alongside built-in functions like "print".) + + Elsewhere, "_" is a regular identifier. It is often used to name + “special” items, but it is not special to Python itself. + + Note: + + The name "_" is often used in conjunction with + internationalization; refer to the documentation for the + "gettext" module for more information on this convention.It is + also commonly used for unused variables. + +"__*__" + System-defined names, informally known as “dunder” names. These + names are defined by the interpreter and its implementation + (including the standard library). Current system names are + discussed in the Special method names section and elsewhere. More + will likely be defined in future versions of Python. *Any* use of + "__*__" names, in any context, that does not follow explicitly + documented use, is subject to breakage without warning. + +"__*" + Class-private names. Names in this category, when used within the + context of a class definition, are re-written to use a mangled form + to help avoid name clashes between “private” attributes of base and + derived classes. See section Identifiers (Names). +''', + 'identifiers': r'''Names (identifiers and keywords) +******************************** + +"NAME" tokens represent *identifiers*, *keywords*, and *soft +keywords*. + +Names are composed of the following characters: + +* uppercase and lowercase letters ("A-Z" and "a-z"), + +* the underscore ("_"), + +* digits ("0" through "9"), which cannot appear as the first + character, and + +* non-ASCII characters. Valid names may only contain “letter-like” and + “digit-like” characters; see Non-ASCII characters in names for + details. + +Names must contain at least one character, but have no upper length +limit. Case is significant. + +Formally, names are described by the following lexical definitions: + + NAME: name_start name_continue* + name_start: "a"..."z" | "A"..."Z" | "_" | + name_continue: name_start | "0"..."9" + identifier: + +Note that not all names matched by this grammar are valid; see Non- +ASCII characters in names for details. + + +Keywords +======== + +The following names are used as reserved words, or *keywords* of the +language, and cannot be used as ordinary identifiers. They must be +spelled exactly as written here: + + False await else import pass + None break except in raise + True class finally is return + and continue for lambda try + as def from nonlocal while + assert del global not with + async elif if or yield + + +Soft Keywords +============= + +Added in version 3.10. + +Some names are only reserved under specific contexts. These are known +as *soft keywords*: + +* "match", "case", and "_", when used in the "match" statement. + +* "type", when used in the "type" statement. + +These syntactically act as keywords in their specific contexts, but +this distinction is done at the parser level, not when tokenizing. + +As soft keywords, their use in the grammar is possible while still +preserving compatibility with existing code that uses these names as +identifier names. + +Changed in version 3.12: "type" is now a soft keyword. + + +Reserved classes of identifiers +=============================== + +Certain classes of identifiers (besides keywords) have special +meanings. These classes are identified by the patterns of leading and +trailing underscore characters: + +"_*" + Not imported by "from module import *". + +"_" + In a "case" pattern within a "match" statement, "_" is a soft + keyword that denotes a wildcard. + + Separately, the interactive interpreter makes the result of the + last evaluation available in the variable "_". (It is stored in the + "builtins" module, alongside built-in functions like "print".) + + Elsewhere, "_" is a regular identifier. It is often used to name + “special” items, but it is not special to Python itself. + + Note: + + The name "_" is often used in conjunction with + internationalization; refer to the documentation for the + "gettext" module for more information on this convention.It is + also commonly used for unused variables. + +"__*__" + System-defined names, informally known as “dunder” names. These + names are defined by the interpreter and its implementation + (including the standard library). Current system names are + discussed in the Special method names section and elsewhere. More + will likely be defined in future versions of Python. *Any* use of + "__*__" names, in any context, that does not follow explicitly + documented use, is subject to breakage without warning. + +"__*" + Class-private names. Names in this category, when used within the + context of a class definition, are re-written to use a mangled form + to help avoid name clashes between “private” attributes of base and + derived classes. See section Identifiers (Names). + + +Non-ASCII characters in names +============================= + +Names that contain non-ASCII characters need additional normalization +and validation beyond the rules and grammar explained above. For +example, "ř_1", "蛇", or "साँप" are valid names, but "r〰2", "€", or +"🐍" are not. + +This section explains the exact rules. + +All names are converted into the normalization form NFKC while +parsing. This means that, for example, some typographic variants of +characters are converted to their “basic” form. For example, +"fiⁿₐˡᵢᶻₐᵗᵢᵒₙ" normalizes to "finalization", so Python treats them as +the same name: + + >>> fiⁿₐˡᵢᶻₐᵗᵢᵒₙ = 3 + >>> finalization + 3 + +Note: + + Normalization is done at the lexical level only. Run-time functions + that take names as *strings* generally do not normalize their + arguments. For example, the variable defined above is accessible at + run time in the "globals()" dictionary as + "globals()["finalization"]" but not "globals()["fiⁿₐˡᵢᶻₐᵗᵢᵒₙ"]". + +Similarly to how ASCII-only names must contain only letters, digits +and the underscore, and cannot start with a digit, a valid name must +start with a character in the “letter-like” set "xid_start", and the +remaining characters must be in the “letter- and digit-like” set +"xid_continue". + +These sets based on the *XID_Start* and *XID_Continue* sets as defined +by the Unicode standard annex UAX-31. Python’s "xid_start" +additionally includes the underscore ("_"). Note that Python does not +necessarily conform to UAX-31. + +A non-normative listing of characters in the *XID_Start* and +*XID_Continue* sets as defined by Unicode is available in the +DerivedCoreProperties.txt file in the Unicode Character Database. For +reference, the construction rules for the "xid_*" sets are given +below. + +The set "id_start" is defined as the union of: + +* Unicode category "" - uppercase letters (includes "A" to "Z") + +* Unicode category "" - lowercase letters (includes "a" to "z") + +* Unicode category "" - titlecase letters + +* Unicode category "" - modifier letters + +* Unicode category "" - other letters + +* Unicode category "" - letter numbers + +* {""_""} - the underscore + +* "" - an explicit set of characters in PropList.txt + to support backwards compatibility + +The set "xid_start" then closes this set under NFKC normalization, by +removing all characters whose normalization is not of the form +"id_start id_continue*". + +The set "id_continue" is defined as the union of: + +* "id_start" (see above) + +* Unicode category "" - decimal numbers (includes "0" to "9") + +* Unicode category "" - connector punctuations + +* Unicode category "" - nonspacing marks + +* Unicode category "" - spacing combining marks + +* "" - another explicit set of characters in + PropList.txt to support backwards compatibility + +Again, "xid_continue" closes this set under NFKC normalization. + +Unicode categories use the version of the Unicode Character Database +as included in the "unicodedata" module. + +See also: + + * **PEP 3131** – Supporting Non-ASCII Identifiers + + * **PEP 672** – Unicode-related Security Considerations for Python +''', + 'if': r'''The "if" statement +****************** + +The "if" statement is used for conditional execution: + + if_stmt: "if" assignment_expression ":" suite + ("elif" assignment_expression ":" suite)* + ["else" ":" suite] + +It selects exactly one of the suites by evaluating the expressions one +by one until one is found to be true (see section Boolean operations +for the definition of true and false); then that suite is executed +(and no other part of the "if" statement is executed or evaluated). +If all expressions are false, the suite of the "else" clause, if +present, is executed. +''', + 'imaginary': r'''Imaginary literals +****************** + +Python has complex number objects, but no complex literals. Instead, +*imaginary literals* denote complex numbers with a zero real part. + +For example, in math, the complex number 3+4.2*i* is written as the +real number 3 added to the imaginary number 4.2*i*. Python uses a +similar syntax, except the imaginary unit is written as "j" rather +than *i*: + + 3+4.2j + +This is an expression composed of the integer literal "3", the +operator ‘"+"’, and the imaginary literal "4.2j". Since these are +three separate tokens, whitespace is allowed between them: + + 3 + 4.2j + +No whitespace is allowed *within* each token. In particular, the "j" +suffix, may not be separated from the number before it. + +The number before the "j" has the same syntax as a floating-point +literal. Thus, the following are valid imaginary literals: + + 4.2j + 3.14j + 10.j + .001j + 1e100j + 3.14e-10j + 3.14_15_93j + +Unlike in a floating-point literal the decimal point can be omitted if +the imaginary number only has an integer part. The number is still +evaluated as a floating-point number, not an integer: + + 10j + 0j + 1000000000000000000000000j # equivalent to 1e+24j + +The "j" suffix is case-insensitive. That means you can use "J" +instead: + + 3.14J # equivalent to 3.14j + +Formally, imaginary literals are described by the following lexical +definition: + + imagnumber: (floatnumber | digitpart) ("j" | "J") +''', + 'import': r'''The "import" statement +********************** + + import_stmt: "import" module ["as" identifier] ("," module ["as" identifier])* + | "from" relative_module "import" identifier ["as" identifier] + ("," identifier ["as" identifier])* + | "from" relative_module "import" "(" identifier ["as" identifier] + ("," identifier ["as" identifier])* [","] ")" + | "from" relative_module "import" "*" + module: (identifier ".")* identifier + relative_module: "."* module | "."+ + +The basic import statement (no "from" clause) is executed in two +steps: + +1. find a module, loading and initializing it if necessary + +2. define a name or names in the local namespace for the scope where + the "import" statement occurs. + +When the statement contains multiple clauses (separated by commas) the +two steps are carried out separately for each clause, just as though +the clauses had been separated out into individual import statements. + +The details of the first step, finding and loading modules, are +described in greater detail in the section on the import system, which +also describes the various types of packages and modules that can be +imported, as well as all the hooks that can be used to customize the +import system. Note that failures in this step may indicate either +that the module could not be located, *or* that an error occurred +while initializing the module, which includes execution of the +module’s code. + +If the requested module is retrieved successfully, it will be made +available in the local namespace in one of three ways: + +* If the module name is followed by "as", then the name following "as" + is bound directly to the imported module. + +* If no other name is specified, and the module being imported is a + top level module, the module’s name is bound in the local namespace + as a reference to the imported module + +* If the module being imported is *not* a top level module, then the + name of the top level package that contains the module is bound in + the local namespace as a reference to the top level package. The + imported module must be accessed using its full qualified name + rather than directly + +The "from" form uses a slightly more complex process: + +1. find the module specified in the "from" clause, loading and + initializing it if necessary; + +2. for each of the identifiers specified in the "import" clauses: + + 1. check if the imported module has an attribute by that name + + 2. if not, attempt to import a submodule with that name and then + check the imported module again for that attribute + + 3. if the attribute is not found, "ImportError" is raised. + + 4. otherwise, a reference to that value is stored in the local + namespace, using the name in the "as" clause if it is present, + otherwise using the attribute name + +Examples: + + import foo # foo imported and bound locally + import foo.bar.baz # foo, foo.bar, and foo.bar.baz imported, foo bound locally + import foo.bar.baz as fbb # foo, foo.bar, and foo.bar.baz imported, foo.bar.baz bound as fbb + from foo.bar import baz # foo, foo.bar, and foo.bar.baz imported, foo.bar.baz bound as baz + from foo import attr # foo imported and foo.attr bound as attr + +If the list of identifiers is replaced by a star ("'*'"), all public +names defined in the module are bound in the local namespace for the +scope where the "import" statement occurs. + +The *public names* defined by a module are determined by checking the +module’s namespace for a variable named "__all__"; if defined, it must +be a sequence of strings which are names defined or imported by that +module. The names given in "__all__" are all considered public and +are required to exist. If "__all__" is not defined, the set of public +names includes all names found in the module’s namespace which do not +begin with an underscore character ("'_'"). "__all__" should contain +the entire public API. It is intended to avoid accidentally exporting +items that are not part of the API (such as library modules which were +imported and used within the module). + +The wild card form of import — "from module import *" — is only +allowed at the module level. Attempting to use it in class or +function definitions will raise a "SyntaxError". + +When specifying what module to import you do not have to specify the +absolute name of the module. When a module or package is contained +within another package it is possible to make a relative import within +the same top package without having to mention the package name. By +using leading dots in the specified module or package after "from" you +can specify how high to traverse up the current package hierarchy +without specifying exact names. One leading dot means the current +package where the module making the import exists. Two dots means up +one package level. Three dots is up two levels, etc. So if you execute +"from . import mod" from a module in the "pkg" package then you will +end up importing "pkg.mod". If you execute "from ..subpkg2 import mod" +from within "pkg.subpkg1" you will import "pkg.subpkg2.mod". The +specification for relative imports is contained in the Package +Relative Imports section. + +"importlib.import_module()" is provided to support applications that +determine dynamically the modules to be loaded. + +Raises an auditing event "import" with arguments "module", "filename", +"sys.path", "sys.meta_path", "sys.path_hooks". + + +Future statements +================= + +A *future statement* is a directive to the compiler that a particular +module should be compiled using syntax or semantics that will be +available in a specified future release of Python where the feature +becomes standard. + +The future statement is intended to ease migration to future versions +of Python that introduce incompatible changes to the language. It +allows use of the new features on a per-module basis before the +release in which the feature becomes standard. + + future_stmt: "from" "__future__" "import" feature ["as" identifier] + ("," feature ["as" identifier])* + | "from" "__future__" "import" "(" feature ["as" identifier] + ("," feature ["as" identifier])* [","] ")" + feature: identifier + +A future statement must appear near the top of the module. The only +lines that can appear before a future statement are: + +* the module docstring (if any), + +* comments, + +* blank lines, and + +* other future statements. + +The only feature that requires using the future statement is +"annotations" (see **PEP 563**). + +All historical features enabled by the future statement are still +recognized by Python 3. The list includes "absolute_import", +"division", "generators", "generator_stop", "unicode_literals", +"print_function", "nested_scopes" and "with_statement". They are all +redundant because they are always enabled, and only kept for backwards +compatibility. + +A future statement is recognized and treated specially at compile +time: Changes to the semantics of core constructs are often +implemented by generating different code. It may even be the case +that a new feature introduces new incompatible syntax (such as a new +reserved word), in which case the compiler may need to parse the +module differently. Such decisions cannot be pushed off until +runtime. + +For any given release, the compiler knows which feature names have +been defined, and raises a compile-time error if a future statement +contains a feature not known to it. + +The direct runtime semantics are the same as for any import statement: +there is a standard module "__future__", described later, and it will +be imported in the usual way at the time the future statement is +executed. + +The interesting runtime semantics depend on the specific feature +enabled by the future statement. + +Note that there is nothing special about the statement: + + import __future__ [as name] + +That is not a future statement; it’s an ordinary import statement with +no special semantics or syntax restrictions. + +Code compiled by calls to the built-in functions "exec()" and +"compile()" that occur in a module "M" containing a future statement +will, by default, use the new syntax or semantics associated with the +future statement. This can be controlled by optional arguments to +"compile()" — see the documentation of that function for details. + +A future statement typed at an interactive interpreter prompt will +take effect for the rest of the interpreter session. If an +interpreter is started with the "-i" option, is passed a script name +to execute, and the script includes a future statement, it will be in +effect in the interactive session started after the script is +executed. + +See also: + + **PEP 236** - Back to the __future__ + The original proposal for the __future__ mechanism. +''', + 'in': r'''Membership test operations +************************** + +The operators "in" and "not in" test for membership. "x in s" +evaluates to "True" if *x* is a member of *s*, and "False" otherwise. +"x not in s" returns the negation of "x in s". All built-in sequences +and set types support this as well as dictionary, for which "in" tests +whether the dictionary has a given key. For container types such as +list, tuple, set, frozenset, dict, or collections.deque, the +expression "x in y" is equivalent to "any(x is e or x == e for e in +y)". + +For the string and bytes types, "x in y" is "True" if and only if *x* +is a substring of *y*. An equivalent test is "y.find(x) != -1". +Empty strings are always considered to be a substring of any other +string, so """ in "abc"" will return "True". + +For user-defined classes which define the "__contains__()" method, "x +in y" returns "True" if "y.__contains__(x)" returns a true value, and +"False" otherwise. + +For user-defined classes which do not define "__contains__()" but do +define "__iter__()", "x in y" is "True" if some value "z", for which +the expression "x is z or x == z" is true, is produced while iterating +over "y". If an exception is raised during the iteration, it is as if +"in" raised that exception. + +Lastly, the old-style iteration protocol is tried: if a class defines +"__getitem__()", "x in y" is "True" if and only if there is a non- +negative integer index *i* such that "x is y[i] or x == y[i]", and no +lower integer index raises the "IndexError" exception. (If any other +exception is raised, it is as if "in" raised that exception). + +The operator "not in" is defined to have the inverse truth value of +"in". +''', + 'integers': r'''Integer literals +**************** + +Integer literals denote whole numbers. For example: + + 7 + 3 + 2147483647 + +There is no limit for the length of integer literals apart from what +can be stored in available memory: + + 7922816251426433759354395033679228162514264337593543950336 + +Underscores can be used to group digits for enhanced readability, and +are ignored for determining the numeric value of the literal. For +example, the following literals are equivalent: + + 100_000_000_000 + 100000000000 + 1_00_00_00_00_000 + +Underscores can only occur between digits. For example, "_123", +"321_", and "123__321" are *not* valid literals. + +Integers can be specified in binary (base 2), octal (base 8), or +hexadecimal (base 16) using the prefixes "0b", "0o" and "0x", +respectively. Hexadecimal digits 10 through 15 are represented by +letters "A"-"F", case-insensitive. For example: + + 0b100110111 + 0b_1110_0101 + 0o177 + 0o377 + 0xdeadbeef + 0xDead_Beef + +An underscore can follow the base specifier. For example, "0x_1f" is a +valid literal, but "0_x1f" and "0x__1f" are not. + +Leading zeros in a non-zero decimal number are not allowed. For +example, "0123" is not a valid literal. This is for disambiguation +with C-style octal literals, which Python used before version 3.0. + +Formally, integer literals are described by the following lexical +definitions: + + integer: decinteger | bininteger | octinteger | hexinteger | zerointeger + decinteger: nonzerodigit (["_"] digit)* + bininteger: "0" ("b" | "B") (["_"] bindigit)+ + octinteger: "0" ("o" | "O") (["_"] octdigit)+ + hexinteger: "0" ("x" | "X") (["_"] hexdigit)+ + zerointeger: "0"+ (["_"] "0")* + nonzerodigit: "1"..."9" + digit: "0"..."9" + bindigit: "0" | "1" + octdigit: "0"..."7" + hexdigit: digit | "a"..."f" | "A"..."F" + +Changed in version 3.6: Underscores are now allowed for grouping +purposes in literals. +''', + 'lambda': r'''Lambdas +******* + + lambda_expr: "lambda" [parameter_list] ":" expression + +Lambda expressions (sometimes called lambda forms) are used to create +anonymous functions. The expression "lambda parameters: expression" +yields a function object. The unnamed object behaves like a function +object defined with: + + def (parameters): + return expression + +See section Function definitions for the syntax of parameter lists. +Note that functions created with lambda expressions cannot contain +statements or annotations. +''', + 'lists': r'''List displays +************* + +A list display is a possibly empty series of expressions enclosed in +square brackets: + + list_display: "[" [flexible_expression_list | comprehension] "]" + +A list display yields a new list object, the contents being specified +by either a list of expressions or a comprehension. When a comma- +separated list of expressions is supplied, its elements are evaluated +from left to right and placed into the list object in that order. +When a comprehension is supplied, the list is constructed from the +elements resulting from the comprehension. +''', + 'naming': r'''Naming and binding +****************** + + +Binding of names +================ + +*Names* refer to objects. Names are introduced by name binding +operations. + +The following constructs bind names: + +* formal parameters to functions, + +* class definitions, + +* function definitions, + +* assignment expressions, + +* targets that are identifiers if occurring in an assignment: + + * "for" loop header, + + * after "as" in a "with" statement, "except" clause, "except*" + clause, or in the as-pattern in structural pattern matching, + + * in a capture pattern in structural pattern matching + +* "import" statements. + +* "type" statements. + +* type parameter lists. + +The "import" statement of the form "from ... import *" binds all names +defined in the imported module, except those beginning with an +underscore. This form may only be used at the module level. + +A target occurring in a "del" statement is also considered bound for +this purpose (though the actual semantics are to unbind the name). + +Each assignment or import statement occurs within a block defined by a +class or function definition or at the module level (the top-level +code block). + +If a name is bound in a block, it is a local variable of that block, +unless declared as "nonlocal" or "global". If a name is bound at the +module level, it is a global variable. (The variables of the module +code block are local and global.) If a variable is used in a code +block but not defined there, it is a *free variable*. + +Each occurrence of a name in the program text refers to the *binding* +of that name established by the following name resolution rules. + + +Resolution of names +=================== + +A *scope* defines the visibility of a name within a block. If a local +variable is defined in a block, its scope includes that block. If the +definition occurs in a function block, the scope extends to any blocks +contained within the defining one, unless a contained block introduces +a different binding for the name. + +When a name is used in a code block, it is resolved using the nearest +enclosing scope. The set of all such scopes visible to a code block +is called the block’s *environment*. + +When a name is not found at all, a "NameError" exception is raised. If +the current scope is a function scope, and the name refers to a local +variable that has not yet been bound to a value at the point where the +name is used, an "UnboundLocalError" exception is raised. +"UnboundLocalError" is a subclass of "NameError". + +If a name binding operation occurs anywhere within a code block, all +uses of the name within the block are treated as references to the +current block. This can lead to errors when a name is used within a +block before it is bound. This rule is subtle. Python lacks +declarations and allows name binding operations to occur anywhere +within a code block. The local variables of a code block can be +determined by scanning the entire text of the block for name binding +operations. See the FAQ entry on UnboundLocalError for examples. + +If the "global" statement occurs within a block, all uses of the names +specified in the statement refer to the bindings of those names in the +top-level namespace. Names are resolved in the top-level namespace by +searching the global namespace, i.e. the namespace of the module +containing the code block, and the builtins namespace, the namespace +of the module "builtins". The global namespace is searched first. If +the names are not found there, the builtins namespace is searched +next. If the names are also not found in the builtins namespace, new +variables are created in the global namespace. The global statement +must precede all uses of the listed names. + +The "global" statement has the same scope as a name binding operation +in the same block. If the nearest enclosing scope for a free variable +contains a global statement, the free variable is treated as a global. + +The "nonlocal" statement causes corresponding names to refer to +previously bound variables in the nearest enclosing function scope. +"SyntaxError" is raised at compile time if the given name does not +exist in any enclosing function scope. Type parameters cannot be +rebound with the "nonlocal" statement. + +The namespace for a module is automatically created the first time a +module is imported. The main module for a script is always called +"__main__". + +Class definition blocks and arguments to "exec()" and "eval()" are +special in the context of name resolution. A class definition is an +executable statement that may use and define names. These references +follow the normal rules for name resolution with an exception that +unbound local variables are looked up in the global namespace. The +namespace of the class definition becomes the attribute dictionary of +the class. The scope of names defined in a class block is limited to +the class block; it does not extend to the code blocks of methods. +This includes comprehensions and generator expressions, but it does +not include annotation scopes, which have access to their enclosing +class scopes. This means that the following will fail: + + class A: + a = 42 + b = list(a + i for i in range(10)) + +However, the following will succeed: + + class A: + type Alias = Nested + class Nested: pass + + print(A.Alias.__value__) # + + +Annotation scopes +================= + +*Annotations*, type parameter lists and "type" statements introduce +*annotation scopes*, which behave mostly like function scopes, but +with some exceptions discussed below. + +Annotation scopes are used in the following contexts: + +* *Function annotations*. + +* *Variable annotations*. + +* Type parameter lists for generic type aliases. + +* Type parameter lists for generic functions. A generic function’s + annotations are executed within the annotation scope, but its + defaults and decorators are not. + +* Type parameter lists for generic classes. A generic class’s base + classes and keyword arguments are executed within the annotation + scope, but its decorators are not. + +* The bounds, constraints, and default values for type parameters + (lazily evaluated). + +* The value of type aliases (lazily evaluated). + +Annotation scopes differ from function scopes in the following ways: + +* Annotation scopes have access to their enclosing class namespace. If + an annotation scope is immediately within a class scope, or within + another annotation scope that is immediately within a class scope, + the code in the annotation scope can use names defined in the class + scope as if it were executed directly within the class body. This + contrasts with regular functions defined within classes, which + cannot access names defined in the class scope. + +* Expressions in annotation scopes cannot contain "yield", "yield + from", "await", or ":=" expressions. (These expressions are allowed + in other scopes contained within the annotation scope.) + +* Names defined in annotation scopes cannot be rebound with "nonlocal" + statements in inner scopes. This includes only type parameters, as + no other syntactic elements that can appear within annotation scopes + can introduce new names. + +* While annotation scopes have an internal name, that name is not + reflected in the *qualified name* of objects defined within the + scope. Instead, the "__qualname__" of such objects is as if the + object were defined in the enclosing scope. + +Added in version 3.12: Annotation scopes were introduced in Python +3.12 as part of **PEP 695**. + +Changed in version 3.13: Annotation scopes are also used for type +parameter defaults, as introduced by **PEP 696**. + +Changed in version 3.14: Annotation scopes are now also used for +annotations, as specified in **PEP 649** and **PEP 749**. + + +Lazy evaluation +=============== + +Most annotation scopes are *lazily evaluated*. This includes +annotations, the values of type aliases created through the "type" +statement, and the bounds, constraints, and default values of type +variables created through the type parameter syntax. This means that +they are not evaluated when the type alias or type variable is +created, or when the object carrying annotations is created. Instead, +they are only evaluated when necessary, for example when the +"__value__" attribute on a type alias is accessed. + +Example: + + >>> type Alias = 1/0 + >>> Alias.__value__ + Traceback (most recent call last): + ... + ZeroDivisionError: division by zero + >>> def func[T: 1/0](): pass + >>> T = func.__type_params__[0] + >>> T.__bound__ + Traceback (most recent call last): + ... + ZeroDivisionError: division by zero + +Here the exception is raised only when the "__value__" attribute of +the type alias or the "__bound__" attribute of the type variable is +accessed. + +This behavior is primarily useful for references to types that have +not yet been defined when the type alias or type variable is created. +For example, lazy evaluation enables creation of mutually recursive +type aliases: + + from typing import Literal + + type SimpleExpr = int | Parenthesized + type Parenthesized = tuple[Literal["("], Expr, Literal[")"]] + type Expr = SimpleExpr | tuple[SimpleExpr, Literal["+", "-"], Expr] + +Lazily evaluated values are evaluated in annotation scope, which means +that names that appear inside the lazily evaluated value are looked up +as if they were used in the immediately enclosing scope. + +Added in version 3.12. + + +Builtins and restricted execution +================================= + +**CPython implementation detail:** Users should not touch +"__builtins__"; it is strictly an implementation detail. Users +wanting to override values in the builtins namespace should "import" +the "builtins" module and modify its attributes appropriately. + +The builtins namespace associated with the execution of a code block +is actually found by looking up the name "__builtins__" in its global +namespace; this should be a dictionary or a module (in the latter case +the module’s dictionary is used). By default, when in the "__main__" +module, "__builtins__" is the built-in module "builtins"; when in any +other module, "__builtins__" is an alias for the dictionary of the +"builtins" module itself. + + +Interaction with dynamic features +================================= + +Name resolution of free variables occurs at runtime, not at compile +time. This means that the following code will print 42: + + i = 10 + def f(): + print(i) + i = 42 + f() + +The "eval()" and "exec()" functions do not have access to the full +environment for resolving names. Names may be resolved in the local +and global namespaces of the caller. Free variables are not resolved +in the nearest enclosing namespace, but in the global namespace. [1] +The "exec()" and "eval()" functions have optional arguments to +override the global and local namespace. If only one namespace is +specified, it is used for both. +''', + 'nonlocal': r'''The "nonlocal" statement +************************ + + nonlocal_stmt: "nonlocal" identifier ("," identifier)* + +When the definition of a function or class is nested (enclosed) within +the definitions of other functions, its nonlocal scopes are the local +scopes of the enclosing functions. The "nonlocal" statement causes the +listed identifiers to refer to names previously bound in nonlocal +scopes. It allows encapsulated code to rebind such nonlocal +identifiers. If a name is bound in more than one nonlocal scope, the +nearest binding is used. If a name is not bound in any nonlocal scope, +or if there is no nonlocal scope, a "SyntaxError" is raised. + +The "nonlocal" statement applies to the entire scope of a function or +class body. A "SyntaxError" is raised if a variable is used or +assigned to prior to its nonlocal declaration in the scope. + +See also: + + **PEP 3104** - Access to Names in Outer Scopes + The specification for the "nonlocal" statement. + +**Programmer’s note:** "nonlocal" is a directive to the parser and +applies only to code parsed along with it. See the note for the +"global" statement. +''', + 'numbers': r'''Numeric literals +**************** + +"NUMBER" tokens represent numeric literals, of which there are three +types: integers, floating-point numbers, and imaginary numbers. + + NUMBER: integer | floatnumber | imagnumber + +The numeric value of a numeric literal is the same as if it were +passed as a string to the "int", "float" or "complex" class +constructor, respectively. Note that not all valid inputs for those +constructors are also valid literals. + +Numeric literals do not include a sign; a phrase like "-1" is actually +an expression composed of the unary operator ‘"-"’ and the literal +"1". + + +Integer literals +================ + +Integer literals denote whole numbers. For example: + + 7 + 3 + 2147483647 + +There is no limit for the length of integer literals apart from what +can be stored in available memory: + + 7922816251426433759354395033679228162514264337593543950336 + +Underscores can be used to group digits for enhanced readability, and +are ignored for determining the numeric value of the literal. For +example, the following literals are equivalent: + + 100_000_000_000 + 100000000000 + 1_00_00_00_00_000 + +Underscores can only occur between digits. For example, "_123", +"321_", and "123__321" are *not* valid literals. + +Integers can be specified in binary (base 2), octal (base 8), or +hexadecimal (base 16) using the prefixes "0b", "0o" and "0x", +respectively. Hexadecimal digits 10 through 15 are represented by +letters "A"-"F", case-insensitive. For example: + + 0b100110111 + 0b_1110_0101 + 0o177 + 0o377 + 0xdeadbeef + 0xDead_Beef + +An underscore can follow the base specifier. For example, "0x_1f" is a +valid literal, but "0_x1f" and "0x__1f" are not. + +Leading zeros in a non-zero decimal number are not allowed. For +example, "0123" is not a valid literal. This is for disambiguation +with C-style octal literals, which Python used before version 3.0. + +Formally, integer literals are described by the following lexical +definitions: + + integer: decinteger | bininteger | octinteger | hexinteger | zerointeger + decinteger: nonzerodigit (["_"] digit)* + bininteger: "0" ("b" | "B") (["_"] bindigit)+ + octinteger: "0" ("o" | "O") (["_"] octdigit)+ + hexinteger: "0" ("x" | "X") (["_"] hexdigit)+ + zerointeger: "0"+ (["_"] "0")* + nonzerodigit: "1"..."9" + digit: "0"..."9" + bindigit: "0" | "1" + octdigit: "0"..."7" + hexdigit: digit | "a"..."f" | "A"..."F" + +Changed in version 3.6: Underscores are now allowed for grouping +purposes in literals. + + +Floating-point literals +======================= + +Floating-point (float) literals, such as "3.14" or "1.5", denote +approximations of real numbers. + +They consist of *integer* and *fraction* parts, each composed of +decimal digits. The parts are separated by a decimal point, ".": + + 2.71828 + 4.0 + +Unlike in integer literals, leading zeros are allowed. For example, +"077.010" is legal, and denotes the same number as "77.01". + +As in integer literals, single underscores may occur between digits to +help readability: + + 96_485.332_123 + 3.14_15_93 + +Either of these parts, but not both, can be empty. For example: + + 10. # (equivalent to 10.0) + .001 # (equivalent to 0.001) + +Optionally, the integer and fraction may be followed by an *exponent*: +the letter "e" or "E", followed by an optional sign, "+" or "-", and a +number in the same format as the integer and fraction parts. The "e" +or "E" represents “times ten raised to the power of”: + + 1.0e3 # (represents 1.0×10³, or 1000.0) + 1.166e-5 # (represents 1.166×10⁻⁵, or 0.00001166) + 6.02214076e+23 # (represents 6.02214076×10²³, or 602214076000000000000000.) + +In floats with only integer and exponent parts, the decimal point may +be omitted: + + 1e3 # (equivalent to 1.e3 and 1.0e3) + 0e0 # (equivalent to 0.) + +Formally, floating-point literals are described by the following +lexical definitions: + + floatnumber: + | digitpart "." [digitpart] [exponent] + | "." digitpart [exponent] + | digitpart exponent + digitpart: digit (["_"] digit)* + exponent: ("e" | "E") ["+" | "-"] digitpart + +Changed in version 3.6: Underscores are now allowed for grouping +purposes in literals. + + +Imaginary literals +================== + +Python has complex number objects, but no complex literals. Instead, +*imaginary literals* denote complex numbers with a zero real part. + +For example, in math, the complex number 3+4.2*i* is written as the +real number 3 added to the imaginary number 4.2*i*. Python uses a +similar syntax, except the imaginary unit is written as "j" rather +than *i*: + + 3+4.2j + +This is an expression composed of the integer literal "3", the +operator ‘"+"’, and the imaginary literal "4.2j". Since these are +three separate tokens, whitespace is allowed between them: + + 3 + 4.2j + +No whitespace is allowed *within* each token. In particular, the "j" +suffix, may not be separated from the number before it. + +The number before the "j" has the same syntax as a floating-point +literal. Thus, the following are valid imaginary literals: + + 4.2j + 3.14j + 10.j + .001j + 1e100j + 3.14e-10j + 3.14_15_93j + +Unlike in a floating-point literal the decimal point can be omitted if +the imaginary number only has an integer part. The number is still +evaluated as a floating-point number, not an integer: + + 10j + 0j + 1000000000000000000000000j # equivalent to 1e+24j + +The "j" suffix is case-insensitive. That means you can use "J" +instead: + + 3.14J # equivalent to 3.14j + +Formally, imaginary literals are described by the following lexical +definition: + + imagnumber: (floatnumber | digitpart) ("j" | "J") +''', + 'numeric-types': r'''Emulating numeric types +*********************** + +The following methods can be defined to emulate numeric objects. +Methods corresponding to operations that are not supported by the +particular kind of number implemented (e.g., bitwise operations for +non-integral numbers) should be left undefined. + +object.__add__(self, other) +object.__sub__(self, other) +object.__mul__(self, other) +object.__matmul__(self, other) +object.__truediv__(self, other) +object.__floordiv__(self, other) +object.__mod__(self, other) +object.__divmod__(self, other) +object.__pow__(self, other[, modulo]) +object.__lshift__(self, other) +object.__rshift__(self, other) +object.__and__(self, other) +object.__xor__(self, other) +object.__or__(self, other) + + These methods are called to implement the binary arithmetic + operations ("+", "-", "*", "@", "/", "//", "%", "divmod()", + "pow()", "**", "<<", ">>", "&", "^", "|"). For instance, to + evaluate the expression "x + y", where *x* is an instance of a + class that has an "__add__()" method, "type(x).__add__(x, y)" is + called. The "__divmod__()" method should be the equivalent to + using "__floordiv__()" and "__mod__()"; it should not be related to + "__truediv__()". Note that "__pow__()" should be defined to accept + an optional third argument if the three-argument version of the + built-in "pow()" function is to be supported. + + If one of those methods does not support the operation with the + supplied arguments, it should return "NotImplemented". + +object.__radd__(self, other) +object.__rsub__(self, other) +object.__rmul__(self, other) +object.__rmatmul__(self, other) +object.__rtruediv__(self, other) +object.__rfloordiv__(self, other) +object.__rmod__(self, other) +object.__rdivmod__(self, other) +object.__rpow__(self, other[, modulo]) +object.__rlshift__(self, other) +object.__rrshift__(self, other) +object.__rand__(self, other) +object.__rxor__(self, other) +object.__ror__(self, other) + + These methods are called to implement the binary arithmetic + operations ("+", "-", "*", "@", "/", "//", "%", "divmod()", + "pow()", "**", "<<", ">>", "&", "^", "|") with reflected (swapped) + operands. These functions are only called if the operands are of + different types, when the left operand does not support the + corresponding operation [3], or the right operand’s class is + derived from the left operand’s class. [4] For instance, to + evaluate the expression "x - y", where *y* is an instance of a + class that has an "__rsub__()" method, "type(y).__rsub__(y, x)" is + called if "type(x).__sub__(x, y)" returns "NotImplemented" or + "type(y)" is a subclass of "type(x)". [5] + + Note that "__rpow__()" should be defined to accept an optional + third argument if the three-argument version of the built-in + "pow()" function is to be supported. + + Changed in version 3.14: Three-argument "pow()" now try calling + "__rpow__()" if necessary. Previously it was only called in two- + argument "pow()" and the binary power operator. + + Note: + + If the right operand’s type is a subclass of the left operand’s + type and that subclass provides a different implementation of the + reflected method for the operation, this method will be called + before the left operand’s non-reflected method. This behavior + allows subclasses to override their ancestors’ operations. + +object.__iadd__(self, other) +object.__isub__(self, other) +object.__imul__(self, other) +object.__imatmul__(self, other) +object.__itruediv__(self, other) +object.__ifloordiv__(self, other) +object.__imod__(self, other) +object.__ipow__(self, other[, modulo]) +object.__ilshift__(self, other) +object.__irshift__(self, other) +object.__iand__(self, other) +object.__ixor__(self, other) +object.__ior__(self, other) + + These methods are called to implement the augmented arithmetic + assignments ("+=", "-=", "*=", "@=", "/=", "//=", "%=", "**=", + "<<=", ">>=", "&=", "^=", "|="). These methods should attempt to + do the operation in-place (modifying *self*) and return the result + (which could be, but does not have to be, *self*). If a specific + method is not defined, or if that method returns "NotImplemented", + the augmented assignment falls back to the normal methods. For + instance, if *x* is an instance of a class with an "__iadd__()" + method, "x += y" is equivalent to "x = x.__iadd__(y)" . If + "__iadd__()" does not exist, or if "x.__iadd__(y)" returns + "NotImplemented", "x.__add__(y)" and "y.__radd__(x)" are + considered, as with the evaluation of "x + y". In certain + situations, augmented assignment can result in unexpected errors + (see Why does a_tuple[i] += [‘item’] raise an exception when the + addition works?), but this behavior is in fact part of the data + model. + +object.__neg__(self) +object.__pos__(self) +object.__abs__(self) +object.__invert__(self) + + Called to implement the unary arithmetic operations ("-", "+", + "abs()" and "~"). + +object.__complex__(self) +object.__int__(self) +object.__float__(self) + + Called to implement the built-in functions "complex()", "int()" and + "float()". Should return a value of the appropriate type. + +object.__index__(self) + + Called to implement "operator.index()", and whenever Python needs + to losslessly convert the numeric object to an integer object (such + as in slicing, or in the built-in "bin()", "hex()" and "oct()" + functions). Presence of this method indicates that the numeric + object is an integer type. Must return an integer. + + If "__int__()", "__float__()" and "__complex__()" are not defined + then corresponding built-in functions "int()", "float()" and + "complex()" fall back to "__index__()". + +object.__round__(self[, ndigits]) +object.__trunc__(self) +object.__floor__(self) +object.__ceil__(self) + + Called to implement the built-in function "round()" and "math" + functions "trunc()", "floor()" and "ceil()". Unless *ndigits* is + passed to "__round__()" all these methods should return the value + of the object truncated to an "Integral" (typically an "int"). + + Changed in version 3.14: "int()" no longer delegates to the + "__trunc__()" method. +''', + 'objects': r'''Objects, values and types +************************* + +*Objects* are Python’s abstraction for data. All data in a Python +program is represented by objects or by relations between objects. +Even code is represented by objects. + +Every object has an identity, a type and a value. An object’s +*identity* never changes once it has been created; you may think of it +as the object’s address in memory. The "is" operator compares the +identity of two objects; the "id()" function returns an integer +representing its identity. + +**CPython implementation detail:** For CPython, "id(x)" is the memory +address where "x" is stored. + +An object’s type determines the operations that the object supports +(e.g., “does it have a length?”) and also defines the possible values +for objects of that type. The "type()" function returns an object’s +type (which is an object itself). Like its identity, an object’s +*type* is also unchangeable. [1] + +The *value* of some objects can change. Objects whose value can +change are said to be *mutable*; objects whose value is unchangeable +once they are created are called *immutable*. (The value of an +immutable container object that contains a reference to a mutable +object can change when the latter’s value is changed; however the +container is still considered immutable, because the collection of +objects it contains cannot be changed. So, immutability is not +strictly the same as having an unchangeable value, it is more subtle.) +An object’s mutability is determined by its type; for instance, +numbers, strings and tuples are immutable, while dictionaries and +lists are mutable. + +Objects are never explicitly destroyed; however, when they become +unreachable they may be garbage-collected. An implementation is +allowed to postpone garbage collection or omit it altogether — it is a +matter of implementation quality how garbage collection is +implemented, as long as no objects are collected that are still +reachable. + +**CPython implementation detail:** CPython currently uses a reference- +counting scheme with (optional) delayed detection of cyclically linked +garbage, which collects most objects as soon as they become +unreachable, but is not guaranteed to collect garbage containing +circular references. See the documentation of the "gc" module for +information on controlling the collection of cyclic garbage. Other +implementations act differently and CPython may change. Do not depend +on immediate finalization of objects when they become unreachable (so +you should always close files explicitly). + +Note that the use of the implementation’s tracing or debugging +facilities may keep objects alive that would normally be collectable. +Also note that catching an exception with a "try"…"except" statement +may keep objects alive. + +Some objects contain references to “external” resources such as open +files or windows. It is understood that these resources are freed +when the object is garbage-collected, but since garbage collection is +not guaranteed to happen, such objects also provide an explicit way to +release the external resource, usually a "close()" method. Programs +are strongly recommended to explicitly close such objects. The +"try"…"finally" statement and the "with" statement provide convenient +ways to do this. + +Some objects contain references to other objects; these are called +*containers*. Examples of containers are tuples, lists and +dictionaries. The references are part of a container’s value. In +most cases, when we talk about the value of a container, we imply the +values, not the identities of the contained objects; however, when we +talk about the mutability of a container, only the identities of the +immediately contained objects are implied. So, if an immutable +container (like a tuple) contains a reference to a mutable object, its +value changes if that mutable object is changed. + +Types affect almost all aspects of object behavior. Even the +importance of object identity is affected in some sense: for immutable +types, operations that compute new values may actually return a +reference to any existing object with the same type and value, while +for mutable objects this is not allowed. For example, after "a = 1; b += 1", *a* and *b* may or may not refer to the same object with the +value one, depending on the implementation. This is because "int" is +an immutable type, so the reference to "1" can be reused. This +behaviour depends on the implementation used, so should not be relied +upon, but is something to be aware of when making use of object +identity tests. However, after "c = []; d = []", *c* and *d* are +guaranteed to refer to two different, unique, newly created empty +lists. (Note that "e = f = []" assigns the *same* object to both *e* +and *f*.) +''', + 'operator-summary': r'''Operator precedence +******************* + +The following table summarizes the operator precedence in Python, from +highest precedence (most binding) to lowest precedence (least +binding). Operators in the same box have the same precedence. Unless +the syntax is explicitly given, operators are binary. Operators in +the same box group left to right (except for exponentiation and +conditional expressions, which group from right to left). + +Note that comparisons, membership tests, and identity tests, all have +the same precedence and have a left-to-right chaining feature as +described in the Comparisons section. + ++-------------------------------------------------+---------------------------------------+ +| Operator | Description | +|=================================================|=======================================| +| "(expressions...)", "[expressions...]", "{key: | Binding or parenthesized expression, | +| value...}", "{expressions...}" | list display, dictionary display, set | +| | display | ++-------------------------------------------------+---------------------------------------+ +| "x[index]", "x[index:index]", | Subscription, slicing, call, | +| "x(arguments...)", "x.attribute" | attribute reference | ++-------------------------------------------------+---------------------------------------+ +| "await x" | Await expression | ++-------------------------------------------------+---------------------------------------+ +| "**" | Exponentiation [5] | ++-------------------------------------------------+---------------------------------------+ +| "+x", "-x", "~x" | Positive, negative, bitwise NOT | ++-------------------------------------------------+---------------------------------------+ +| "*", "@", "/", "//", "%" | Multiplication, matrix | +| | multiplication, division, floor | +| | division, remainder [6] | ++-------------------------------------------------+---------------------------------------+ +| "+", "-" | Addition and subtraction | ++-------------------------------------------------+---------------------------------------+ +| "<<", ">>" | Shifts | ++-------------------------------------------------+---------------------------------------+ +| "&" | Bitwise AND | ++-------------------------------------------------+---------------------------------------+ +| "^" | Bitwise XOR | ++-------------------------------------------------+---------------------------------------+ +| "|" | Bitwise OR | ++-------------------------------------------------+---------------------------------------+ +| "in", "not in", "is", "is not", "<", "<=", ">", | Comparisons, including membership | +| ">=", "!=", "==" | tests and identity tests | ++-------------------------------------------------+---------------------------------------+ +| "not x" | Boolean NOT | ++-------------------------------------------------+---------------------------------------+ +| "and" | Boolean AND | ++-------------------------------------------------+---------------------------------------+ +| "or" | Boolean OR | ++-------------------------------------------------+---------------------------------------+ +| "if" – "else" | Conditional expression | ++-------------------------------------------------+---------------------------------------+ +| "lambda" | Lambda expression | ++-------------------------------------------------+---------------------------------------+ +| ":=" | Assignment expression | ++-------------------------------------------------+---------------------------------------+ + +-[ Footnotes ]- + +[1] While "abs(x%y) < abs(y)" is true mathematically, for floats it + may not be true numerically due to roundoff. For example, and + assuming a platform on which a Python float is an IEEE 754 double- + precision number, in order that "-1e-100 % 1e100" have the same + sign as "1e100", the computed result is "-1e-100 + 1e100", which + is numerically exactly equal to "1e100". The function + "math.fmod()" returns a result whose sign matches the sign of the + first argument instead, and so returns "-1e-100" in this case. + Which approach is more appropriate depends on the application. + +[2] If x is very close to an exact integer multiple of y, it’s + possible for "x//y" to be one larger than "(x-x%y)//y" due to + rounding. In such cases, Python returns the latter result, in + order to preserve that "divmod(x,y)[0] * y + x % y" be very close + to "x". + +[3] The Unicode standard distinguishes between *code points* (e.g. + U+0041) and *abstract characters* (e.g. “LATIN CAPITAL LETTER A”). + While most abstract characters in Unicode are only represented + using one code point, there is a number of abstract characters + that can in addition be represented using a sequence of more than + one code point. For example, the abstract character “LATIN + CAPITAL LETTER C WITH CEDILLA” can be represented as a single + *precomposed character* at code position U+00C7, or as a sequence + of a *base character* at code position U+0043 (LATIN CAPITAL + LETTER C), followed by a *combining character* at code position + U+0327 (COMBINING CEDILLA). + + The comparison operators on strings compare at the level of + Unicode code points. This may be counter-intuitive to humans. For + example, ""\u00C7" == "\u0043\u0327"" is "False", even though both + strings represent the same abstract character “LATIN CAPITAL + LETTER C WITH CEDILLA”. + + To compare strings at the level of abstract characters (that is, + in a way intuitive to humans), use "unicodedata.normalize()". + +[4] Due to automatic garbage-collection, free lists, and the dynamic + nature of descriptors, you may notice seemingly unusual behaviour + in certain uses of the "is" operator, like those involving + comparisons between instance methods, or constants. Check their + documentation for more info. + +[5] The power operator "**" binds less tightly than an arithmetic or + bitwise unary operator on its right, that is, "2**-1" is "0.5". + +[6] The "%" operator is also used for string formatting; the same + precedence applies. +''', + 'pass': r'''The "pass" statement +******************** + + pass_stmt: "pass" + +"pass" is a null operation — when it is executed, nothing happens. It +is useful as a placeholder when a statement is required syntactically, +but no code needs to be executed, for example: + + def f(arg): pass # a function that does nothing (yet) + + class C: pass # a class with no methods (yet) +''', + 'power': r'''The power operator +****************** + +The power operator binds more tightly than unary operators on its +left; it binds less tightly than unary operators on its right. The +syntax is: + + power: (await_expr | primary) ["**" u_expr] + +Thus, in an unparenthesized sequence of power and unary operators, the +operators are evaluated from right to left (this does not constrain +the evaluation order for the operands): "-1**2" results in "-1". + +The power operator has the same semantics as the built-in "pow()" +function, when called with two arguments: it yields its left argument +raised to the power of its right argument. The numeric arguments are +first converted to a common type, and the result is of that type. + +For int operands, the result has the same type as the operands unless +the second argument is negative; in that case, all arguments are +converted to float and a float result is delivered. For example, +"10**2" returns "100", but "10**-2" returns "0.01". + +Raising "0.0" to a negative power results in a "ZeroDivisionError". +Raising a negative number to a fractional power results in a "complex" +number. (In earlier versions it raised a "ValueError".) + +This operation can be customized using the special "__pow__()" and +"__rpow__()" methods. +''', + 'raise': r'''The "raise" statement +********************* + + raise_stmt: "raise" [expression ["from" expression]] + +If no expressions are present, "raise" re-raises the exception that is +currently being handled, which is also known as the *active +exception*. If there isn’t currently an active exception, a +"RuntimeError" exception is raised indicating that this is an error. + +Otherwise, "raise" evaluates the first expression as the exception +object. It must be either a subclass or an instance of +"BaseException". If it is a class, the exception instance will be +obtained when needed by instantiating the class with no arguments. + +The *type* of the exception is the exception instance’s class, the +*value* is the instance itself. + +A traceback object is normally created automatically when an exception +is raised and attached to it as the "__traceback__" attribute. You can +create an exception and set your own traceback in one step using the +"with_traceback()" exception method (which returns the same exception +instance, with its traceback set to its argument), like so: + + raise Exception("foo occurred").with_traceback(tracebackobj) + +The "from" clause is used for exception chaining: if given, the second +*expression* must be another exception class or instance. If the +second expression is an exception instance, it will be attached to the +raised exception as the "__cause__" attribute (which is writable). If +the expression is an exception class, the class will be instantiated +and the resulting exception instance will be attached to the raised +exception as the "__cause__" attribute. If the raised exception is not +handled, both exceptions will be printed: + + >>> try: + ... print(1 / 0) + ... except Exception as exc: + ... raise RuntimeError("Something bad happened") from exc + ... + Traceback (most recent call last): + File "", line 2, in + print(1 / 0) + ~~^~~ + ZeroDivisionError: division by zero + + The above exception was the direct cause of the following exception: + + Traceback (most recent call last): + File "", line 4, in + raise RuntimeError("Something bad happened") from exc + RuntimeError: Something bad happened + +A similar mechanism works implicitly if a new exception is raised when +an exception is already being handled. An exception may be handled +when an "except" or "finally" clause, or a "with" statement, is used. +The previous exception is then attached as the new exception’s +"__context__" attribute: + + >>> try: + ... print(1 / 0) + ... except: + ... raise RuntimeError("Something bad happened") + ... + Traceback (most recent call last): + File "", line 2, in + print(1 / 0) + ~~^~~ + ZeroDivisionError: division by zero + + During handling of the above exception, another exception occurred: + + Traceback (most recent call last): + File "", line 4, in + raise RuntimeError("Something bad happened") + RuntimeError: Something bad happened + +Exception chaining can be explicitly suppressed by specifying "None" +in the "from" clause: + + >>> try: + ... print(1 / 0) + ... except: + ... raise RuntimeError("Something bad happened") from None + ... + Traceback (most recent call last): + File "", line 4, in + RuntimeError: Something bad happened + +Additional information on exceptions can be found in section +Exceptions, and information about handling exceptions is in section +The try statement. + +Changed in version 3.3: "None" is now permitted as "Y" in "raise X +from Y".Added the "__suppress_context__" attribute to suppress +automatic display of the exception context. + +Changed in version 3.11: If the traceback of the active exception is +modified in an "except" clause, a subsequent "raise" statement re- +raises the exception with the modified traceback. Previously, the +exception was re-raised with the traceback it had when it was caught. +''', + 'return': r'''The "return" statement +********************** + + return_stmt: "return" [expression_list] + +"return" may only occur syntactically nested in a function definition, +not within a nested class definition. + +If an expression list is present, it is evaluated, else "None" is +substituted. + +"return" leaves the current function call with the expression list (or +"None") as return value. + +When "return" passes control out of a "try" statement with a "finally" +clause, that "finally" clause is executed before really leaving the +function. + +In a generator function, the "return" statement indicates that the +generator is done and will cause "StopIteration" to be raised. The +returned value (if any) is used as an argument to construct +"StopIteration" and becomes the "StopIteration.value" attribute. + +In an asynchronous generator function, an empty "return" statement +indicates that the asynchronous generator is done and will cause +"StopAsyncIteration" to be raised. A non-empty "return" statement is +a syntax error in an asynchronous generator function. +''', + 'sequence-types': r'''Emulating container types +************************* + +The following methods can be defined to implement container objects. +None of them are provided by the "object" class itself. Containers +usually are *sequences* (such as "lists" or "tuples") or *mappings* +(like *dictionaries*), but can represent other containers as well. +The first set of methods is used either to emulate a sequence or to +emulate a mapping; the difference is that for a sequence, the +allowable keys should be the integers *k* for which "0 <= k < N" where +*N* is the length of the sequence, or "slice" objects, which define a +range of items. It is also recommended that mappings provide the +methods "keys()", "values()", "items()", "get()", "clear()", +"setdefault()", "pop()", "popitem()", "copy()", and "update()" +behaving similar to those for Python’s standard "dictionary" objects. +The "collections.abc" module provides a "MutableMapping" *abstract +base class* to help create those methods from a base set of +"__getitem__()", "__setitem__()", "__delitem__()", and "keys()". + +Mutable sequences should provide methods "append()", "clear()", +"count()", "extend()", "index()", "insert()", "pop()", "remove()", and +"reverse()", like Python standard "list" objects. Finally, sequence +types should implement addition (meaning concatenation) and +multiplication (meaning repetition) by defining the methods +"__add__()", "__radd__()", "__iadd__()", "__mul__()", "__rmul__()" and +"__imul__()" described below; they should not define other numerical +operators. + +It is recommended that both mappings and sequences implement the +"__contains__()" method to allow efficient use of the "in" operator; +for mappings, "in" should search the mapping’s keys; for sequences, it +should search through the values. It is further recommended that both +mappings and sequences implement the "__iter__()" method to allow +efficient iteration through the container; for mappings, "__iter__()" +should iterate through the object’s keys; for sequences, it should +iterate through the values. + +object.__len__(self) + + Called to implement the built-in function "len()". Should return + the length of the object, an integer ">=" 0. Also, an object that + doesn’t define a "__bool__()" method and whose "__len__()" method + returns zero is considered to be false in a Boolean context. + + **CPython implementation detail:** In CPython, the length is + required to be at most "sys.maxsize". If the length is larger than + "sys.maxsize" some features (such as "len()") may raise + "OverflowError". To prevent raising "OverflowError" by truth value + testing, an object must define a "__bool__()" method. + +object.__length_hint__(self) + + Called to implement "operator.length_hint()". Should return an + estimated length for the object (which may be greater or less than + the actual length). The length must be an integer ">=" 0. The + return value may also be "NotImplemented", which is treated the + same as if the "__length_hint__" method didn’t exist at all. This + method is purely an optimization and is never required for + correctness. + + Added in version 3.4. + +Note: + + Slicing is done exclusively with the following three methods. A + call like + + a[1:2] = b + + is translated to + + a[slice(1, 2, None)] = b + + and so forth. Missing slice items are always filled in with "None". + +object.__getitem__(self, key) + + Called to implement evaluation of "self[key]". For *sequence* + types, the accepted keys should be integers. Optionally, they may + support "slice" objects as well. Negative index support is also + optional. If *key* is of an inappropriate type, "TypeError" may be + raised; if *key* is a value outside the set of indexes for the + sequence (after any special interpretation of negative values), + "IndexError" should be raised. For *mapping* types, if *key* is + missing (not in the container), "KeyError" should be raised. + + Note: + + "for" loops expect that an "IndexError" will be raised for + illegal indexes to allow proper detection of the end of the + sequence. + + Note: + + When subscripting a *class*, the special class method + "__class_getitem__()" may be called instead of "__getitem__()". + See __class_getitem__ versus __getitem__ for more details. + +object.__setitem__(self, key, value) + + Called to implement assignment to "self[key]". Same note as for + "__getitem__()". This should only be implemented for mappings if + the objects support changes to the values for keys, or if new keys + can be added, or for sequences if elements can be replaced. The + same exceptions should be raised for improper *key* values as for + the "__getitem__()" method. + +object.__delitem__(self, key) + + Called to implement deletion of "self[key]". Same note as for + "__getitem__()". This should only be implemented for mappings if + the objects support removal of keys, or for sequences if elements + can be removed from the sequence. The same exceptions should be + raised for improper *key* values as for the "__getitem__()" method. + +object.__missing__(self, key) + + Called by "dict"."__getitem__()" to implement "self[key]" for dict + subclasses when key is not in the dictionary. + +object.__iter__(self) + + This method is called when an *iterator* is required for a + container. This method should return a new iterator object that can + iterate over all the objects in the container. For mappings, it + should iterate over the keys of the container. + +object.__reversed__(self) + + Called (if present) by the "reversed()" built-in to implement + reverse iteration. It should return a new iterator object that + iterates over all the objects in the container in reverse order. + + If the "__reversed__()" method is not provided, the "reversed()" + built-in will fall back to using the sequence protocol ("__len__()" + and "__getitem__()"). Objects that support the sequence protocol + should only provide "__reversed__()" if they can provide an + implementation that is more efficient than the one provided by + "reversed()". + +The membership test operators ("in" and "not in") are normally +implemented as an iteration through a container. However, container +objects can supply the following special method with a more efficient +implementation, which also does not require the object be iterable. + +object.__contains__(self, item) + + Called to implement membership test operators. Should return true + if *item* is in *self*, false otherwise. For mapping objects, this + should consider the keys of the mapping rather than the values or + the key-item pairs. + + For objects that don’t define "__contains__()", the membership test + first tries iteration via "__iter__()", then the old sequence + iteration protocol via "__getitem__()", see this section in the + language reference. +''', + 'shifting': r'''Shifting operations +******************* + +The shifting operations have lower priority than the arithmetic +operations: + + shift_expr: a_expr | shift_expr ("<<" | ">>") a_expr + +These operators accept integers as arguments. They shift the first +argument to the left or right by the number of bits given by the +second argument. + +The left shift operation can be customized using the special +"__lshift__()" and "__rlshift__()" methods. The right shift operation +can be customized using the special "__rshift__()" and "__rrshift__()" +methods. + +A right shift by *n* bits is defined as floor division by "pow(2,n)". +A left shift by *n* bits is defined as multiplication with "pow(2,n)". +''', + 'slicings': r'''Slicings +******** + +A slicing selects a range of items in a sequence object (e.g., a +string, tuple or list). Slicings may be used as expressions or as +targets in assignment or "del" statements. The syntax for a slicing: + + slicing: primary "[" slice_list "]" + slice_list: slice_item ("," slice_item)* [","] + slice_item: expression | proper_slice + proper_slice: [lower_bound] ":" [upper_bound] [ ":" [stride] ] + lower_bound: expression + upper_bound: expression + stride: expression + +There is ambiguity in the formal syntax here: anything that looks like +an expression list also looks like a slice list, so any subscription +can be interpreted as a slicing. Rather than further complicating the +syntax, this is disambiguated by defining that in this case the +interpretation as a subscription takes priority over the +interpretation as a slicing (this is the case if the slice list +contains no proper slice). + +The semantics for a slicing are as follows. The primary is indexed +(using the same "__getitem__()" method as normal subscription) with a +key that is constructed from the slice list, as follows. If the slice +list contains at least one comma, the key is a tuple containing the +conversion of the slice items; otherwise, the conversion of the lone +slice item is the key. The conversion of a slice item that is an +expression is that expression. The conversion of a proper slice is a +slice object (see section The standard type hierarchy) whose "start", +"stop" and "step" attributes are the values of the expressions given +as lower bound, upper bound and stride, respectively, substituting +"None" for missing expressions. +''', + 'specialattrs': r'''Special Attributes +****************** + +The implementation adds a few special read-only attributes to several +object types, where they are relevant. Some of these are not reported +by the "dir()" built-in function. + +definition.__name__ + + The name of the class, function, method, descriptor, or generator + instance. + +definition.__qualname__ + + The *qualified name* of the class, function, method, descriptor, or + generator instance. + + Added in version 3.3. + +definition.__module__ + + The name of the module in which a class or function was defined. + +definition.__doc__ + + The documentation string of a class or function, or "None" if + undefined. + +definition.__type_params__ + + The type parameters of generic classes, functions, and type + aliases. For classes and functions that are not generic, this will + be an empty tuple. + + Added in version 3.12. +''', + 'specialnames': r'''Special method names +******************** + +A class can implement certain operations that are invoked by special +syntax (such as arithmetic operations or subscripting and slicing) by +defining methods with special names. This is Python’s approach to +*operator overloading*, allowing classes to define their own behavior +with respect to language operators. For instance, if a class defines +a method named "__getitem__()", and "x" is an instance of this class, +then "x[i]" is roughly equivalent to "type(x).__getitem__(x, i)". +Except where mentioned, attempts to execute an operation raise an +exception when no appropriate method is defined (typically +"AttributeError" or "TypeError"). + +Setting a special method to "None" indicates that the corresponding +operation is not available. For example, if a class sets "__iter__()" +to "None", the class is not iterable, so calling "iter()" on its +instances will raise a "TypeError" (without falling back to +"__getitem__()"). [2] + +When implementing a class that emulates any built-in type, it is +important that the emulation only be implemented to the degree that it +makes sense for the object being modelled. For example, some +sequences may work well with retrieval of individual elements, but +extracting a slice may not make sense. (One example of this is the +NodeList interface in the W3C’s Document Object Model.) + + +Basic customization +=================== + +object.__new__(cls[, ...]) + + Called to create a new instance of class *cls*. "__new__()" is a + static method (special-cased so you need not declare it as such) + that takes the class of which an instance was requested as its + first argument. The remaining arguments are those passed to the + object constructor expression (the call to the class). The return + value of "__new__()" should be the new object instance (usually an + instance of *cls*). + + Typical implementations create a new instance of the class by + invoking the superclass’s "__new__()" method using + "super().__new__(cls[, ...])" with appropriate arguments and then + modifying the newly created instance as necessary before returning + it. + + If "__new__()" is invoked during object construction and it returns + an instance of *cls*, then the new instance’s "__init__()" method + will be invoked like "__init__(self[, ...])", where *self* is the + new instance and the remaining arguments are the same as were + passed to the object constructor. + + If "__new__()" does not return an instance of *cls*, then the new + instance’s "__init__()" method will not be invoked. + + "__new__()" is intended mainly to allow subclasses of immutable + types (like int, str, or tuple) to customize instance creation. It + is also commonly overridden in custom metaclasses in order to + customize class creation. + +object.__init__(self[, ...]) + + Called after the instance has been created (by "__new__()"), but + before it is returned to the caller. The arguments are those + passed to the class constructor expression. If a base class has an + "__init__()" method, the derived class’s "__init__()" method, if + any, must explicitly call it to ensure proper initialization of the + base class part of the instance; for example: + "super().__init__([args...])". + + Because "__new__()" and "__init__()" work together in constructing + objects ("__new__()" to create it, and "__init__()" to customize + it), no non-"None" value may be returned by "__init__()"; doing so + will cause a "TypeError" to be raised at runtime. + +object.__del__(self) + + Called when the instance is about to be destroyed. This is also + called a finalizer or (improperly) a destructor. If a base class + has a "__del__()" method, the derived class’s "__del__()" method, + if any, must explicitly call it to ensure proper deletion of the + base class part of the instance. + + It is possible (though not recommended!) for the "__del__()" method + to postpone destruction of the instance by creating a new reference + to it. This is called object *resurrection*. It is + implementation-dependent whether "__del__()" is called a second + time when a resurrected object is about to be destroyed; the + current *CPython* implementation only calls it once. + + It is not guaranteed that "__del__()" methods are called for + objects that still exist when the interpreter exits. + "weakref.finalize" provides a straightforward way to register a + cleanup function to be called when an object is garbage collected. + + Note: + + "del x" doesn’t directly call "x.__del__()" — the former + decrements the reference count for "x" by one, and the latter is + only called when "x"’s reference count reaches zero. + + **CPython implementation detail:** It is possible for a reference + cycle to prevent the reference count of an object from going to + zero. In this case, the cycle will be later detected and deleted + by the *cyclic garbage collector*. A common cause of reference + cycles is when an exception has been caught in a local variable. + The frame’s locals then reference the exception, which references + its own traceback, which references the locals of all frames caught + in the traceback. + + See also: Documentation for the "gc" module. + + Warning: + + Due to the precarious circumstances under which "__del__()" + methods are invoked, exceptions that occur during their execution + are ignored, and a warning is printed to "sys.stderr" instead. + In particular: + + * "__del__()" can be invoked when arbitrary code is being + executed, including from any arbitrary thread. If "__del__()" + needs to take a lock or invoke any other blocking resource, it + may deadlock as the resource may already be taken by the code + that gets interrupted to execute "__del__()". + + * "__del__()" can be executed during interpreter shutdown. As a + consequence, the global variables it needs to access (including + other modules) may already have been deleted or set to "None". + Python guarantees that globals whose name begins with a single + underscore are deleted from their module before other globals + are deleted; if no other references to such globals exist, this + may help in assuring that imported modules are still available + at the time when the "__del__()" method is called. + +object.__repr__(self) + + Called by the "repr()" built-in function to compute the “official” + string representation of an object. If at all possible, this + should look like a valid Python expression that could be used to + recreate an object with the same value (given an appropriate + environment). If this is not possible, a string of the form + "<...some useful description...>" should be returned. The return + value must be a string object. If a class defines "__repr__()" but + not "__str__()", then "__repr__()" is also used when an “informal” + string representation of instances of that class is required. + + This is typically used for debugging, so it is important that the + representation is information-rich and unambiguous. A default + implementation is provided by the "object" class itself. + +object.__str__(self) + + Called by "str(object)", the default "__format__()" implementation, + and the built-in function "print()", to compute the “informal” or + nicely printable string representation of an object. The return + value must be a str object. + + This method differs from "object.__repr__()" in that there is no + expectation that "__str__()" return a valid Python expression: a + more convenient or concise representation can be used. + + The default implementation defined by the built-in type "object" + calls "object.__repr__()". + +object.__bytes__(self) + + Called by bytes to compute a byte-string representation of an + object. This should return a "bytes" object. The "object" class + itself does not provide this method. + +object.__format__(self, format_spec) + + Called by the "format()" built-in function, and by extension, + evaluation of formatted string literals and the "str.format()" + method, to produce a “formatted” string representation of an + object. The *format_spec* argument is a string that contains a + description of the formatting options desired. The interpretation + of the *format_spec* argument is up to the type implementing + "__format__()", however most classes will either delegate + formatting to one of the built-in types, or use a similar + formatting option syntax. + + See Format Specification Mini-Language for a description of the + standard formatting syntax. + + The return value must be a string object. + + The default implementation by the "object" class should be given an + empty *format_spec* string. It delegates to "__str__()". + + Changed in version 3.4: The __format__ method of "object" itself + raises a "TypeError" if passed any non-empty string. + + Changed in version 3.7: "object.__format__(x, '')" is now + equivalent to "str(x)" rather than "format(str(x), '')". + +object.__lt__(self, other) +object.__le__(self, other) +object.__eq__(self, other) +object.__ne__(self, other) +object.__gt__(self, other) +object.__ge__(self, other) + + These are the so-called “rich comparison” methods. The + correspondence between operator symbols and method names is as + follows: "xy" calls + "x.__gt__(y)", and "x>=y" calls "x.__ge__(y)". + + A rich comparison method may return the singleton "NotImplemented" + if it does not implement the operation for a given pair of + arguments. By convention, "False" and "True" are returned for a + successful comparison. However, these methods can return any value, + so if the comparison operator is used in a Boolean context (e.g., + in the condition of an "if" statement), Python will call "bool()" + on the value to determine if the result is true or false. + + By default, "object" implements "__eq__()" by using "is", returning + "NotImplemented" in the case of a false comparison: "True if x is y + else NotImplemented". For "__ne__()", by default it delegates to + "__eq__()" and inverts the result unless it is "NotImplemented". + There are no other implied relationships among the comparison + operators or default implementations; for example, the truth of + "(x.__hash__". + + If a class that does not override "__eq__()" wishes to suppress + hash support, it should include "__hash__ = None" in the class + definition. A class which defines its own "__hash__()" that + explicitly raises a "TypeError" would be incorrectly identified as + hashable by an "isinstance(obj, collections.abc.Hashable)" call. + + Note: + + By default, the "__hash__()" values of str and bytes objects are + “salted” with an unpredictable random value. Although they + remain constant within an individual Python process, they are not + predictable between repeated invocations of Python.This is + intended to provide protection against a denial-of-service caused + by carefully chosen inputs that exploit the worst case + performance of a dict insertion, *O*(*n*^2) complexity. See + http://ocert.org/advisories/ocert-2011-003.html for + details.Changing hash values affects the iteration order of sets. + Python has never made guarantees about this ordering (and it + typically varies between 32-bit and 64-bit builds).See also + "PYTHONHASHSEED". + + Changed in version 3.3: Hash randomization is enabled by default. + +object.__bool__(self) + + Called to implement truth value testing and the built-in operation + "bool()"; should return "False" or "True". When this method is not + defined, "__len__()" is called, if it is defined, and the object is + considered true if its result is nonzero. If a class defines + neither "__len__()" nor "__bool__()" (which is true of the "object" + class itself), all its instances are considered true. + + +Customizing attribute access +============================ + +The following methods can be defined to customize the meaning of +attribute access (use of, assignment to, or deletion of "x.name") for +class instances. + +object.__getattr__(self, name) + + Called when the default attribute access fails with an + "AttributeError" (either "__getattribute__()" raises an + "AttributeError" because *name* is not an instance attribute or an + attribute in the class tree for "self"; or "__get__()" of a *name* + property raises "AttributeError"). This method should either + return the (computed) attribute value or raise an "AttributeError" + exception. The "object" class itself does not provide this method. + + Note that if the attribute is found through the normal mechanism, + "__getattr__()" is not called. (This is an intentional asymmetry + between "__getattr__()" and "__setattr__()".) This is done both for + efficiency reasons and because otherwise "__getattr__()" would have + no way to access other attributes of the instance. Note that at + least for instance variables, you can take total control by not + inserting any values in the instance attribute dictionary (but + instead inserting them in another object). See the + "__getattribute__()" method below for a way to actually get total + control over attribute access. + +object.__getattribute__(self, name) + + Called unconditionally to implement attribute accesses for + instances of the class. If the class also defines "__getattr__()", + the latter will not be called unless "__getattribute__()" either + calls it explicitly or raises an "AttributeError". This method + should return the (computed) attribute value or raise an + "AttributeError" exception. In order to avoid infinite recursion in + this method, its implementation should always call the base class + method with the same name to access any attributes it needs, for + example, "object.__getattribute__(self, name)". + + Note: + + This method may still be bypassed when looking up special methods + as the result of implicit invocation via language syntax or + built-in functions. See Special method lookup. + + For certain sensitive attribute accesses, raises an auditing event + "object.__getattr__" with arguments "obj" and "name". + +object.__setattr__(self, name, value) + + Called when an attribute assignment is attempted. This is called + instead of the normal mechanism (i.e. store the value in the + instance dictionary). *name* is the attribute name, *value* is the + value to be assigned to it. + + If "__setattr__()" wants to assign to an instance attribute, it + should call the base class method with the same name, for example, + "object.__setattr__(self, name, value)". + + For certain sensitive attribute assignments, raises an auditing + event "object.__setattr__" with arguments "obj", "name", "value". + +object.__delattr__(self, name) + + Like "__setattr__()" but for attribute deletion instead of + assignment. This should only be implemented if "del obj.name" is + meaningful for the object. + + For certain sensitive attribute deletions, raises an auditing event + "object.__delattr__" with arguments "obj" and "name". + +object.__dir__(self) + + Called when "dir()" is called on the object. An iterable must be + returned. "dir()" converts the returned iterable to a list and + sorts it. + + +Customizing module attribute access +----------------------------------- + +module.__getattr__() +module.__dir__() + +Special names "__getattr__" and "__dir__" can be also used to +customize access to module attributes. The "__getattr__" function at +the module level should accept one argument which is the name of an +attribute and return the computed value or raise an "AttributeError". +If an attribute is not found on a module object through the normal +lookup, i.e. "object.__getattribute__()", then "__getattr__" is +searched in the module "__dict__" before raising an "AttributeError". +If found, it is called with the attribute name and the result is +returned. + +The "__dir__" function should accept no arguments, and return an +iterable of strings that represents the names accessible on module. If +present, this function overrides the standard "dir()" search on a +module. + +module.__class__ + +For a more fine grained customization of the module behavior (setting +attributes, properties, etc.), one can set the "__class__" attribute +of a module object to a subclass of "types.ModuleType". For example: + + import sys + from types import ModuleType + + class VerboseModule(ModuleType): + def __repr__(self): + return f'Verbose {self.__name__}' + + def __setattr__(self, attr, value): + print(f'Setting {attr}...') + super().__setattr__(attr, value) + + sys.modules[__name__].__class__ = VerboseModule + +Note: + + Defining module "__getattr__" and setting module "__class__" only + affect lookups made using the attribute access syntax – directly + accessing the module globals (whether by code within the module, or + via a reference to the module’s globals dictionary) is unaffected. + +Changed in version 3.5: "__class__" module attribute is now writable. + +Added in version 3.7: "__getattr__" and "__dir__" module attributes. + +See also: + + **PEP 562** - Module __getattr__ and __dir__ + Describes the "__getattr__" and "__dir__" functions on modules. + + +Implementing Descriptors +------------------------ + +The following methods only apply when an instance of the class +containing the method (a so-called *descriptor* class) appears in an +*owner* class (the descriptor must be in either the owner’s class +dictionary or in the class dictionary for one of its parents). In the +examples below, “the attribute” refers to the attribute whose name is +the key of the property in the owner class’ "__dict__". The "object" +class itself does not implement any of these protocols. + +object.__get__(self, instance, owner=None) + + Called to get the attribute of the owner class (class attribute + access) or of an instance of that class (instance attribute + access). The optional *owner* argument is the owner class, while + *instance* is the instance that the attribute was accessed through, + or "None" when the attribute is accessed through the *owner*. + + This method should return the computed attribute value or raise an + "AttributeError" exception. + + **PEP 252** specifies that "__get__()" is callable with one or two + arguments. Python’s own built-in descriptors support this + specification; however, it is likely that some third-party tools + have descriptors that require both arguments. Python’s own + "__getattribute__()" implementation always passes in both arguments + whether they are required or not. + +object.__set__(self, instance, value) + + Called to set the attribute on an instance *instance* of the owner + class to a new value, *value*. + + Note, adding "__set__()" or "__delete__()" changes the kind of + descriptor to a “data descriptor”. See Invoking Descriptors for + more details. + +object.__delete__(self, instance) + + Called to delete the attribute on an instance *instance* of the + owner class. + +Instances of descriptors may also have the "__objclass__" attribute +present: + +object.__objclass__ + + The attribute "__objclass__" is interpreted by the "inspect" module + as specifying the class where this object was defined (setting this + appropriately can assist in runtime introspection of dynamic class + attributes). For callables, it may indicate that an instance of the + given type (or a subclass) is expected or required as the first + positional argument (for example, CPython sets this attribute for + unbound methods that are implemented in C). + + +Invoking Descriptors +-------------------- + +In general, a descriptor is an object attribute with “binding +behavior”, one whose attribute access has been overridden by methods +in the descriptor protocol: "__get__()", "__set__()", and +"__delete__()". If any of those methods are defined for an object, it +is said to be a descriptor. + +The default behavior for attribute access is to get, set, or delete +the attribute from an object’s dictionary. For instance, "a.x" has a +lookup chain starting with "a.__dict__['x']", then +"type(a).__dict__['x']", and continuing through the base classes of +"type(a)" excluding metaclasses. + +However, if the looked-up value is an object defining one of the +descriptor methods, then Python may override the default behavior and +invoke the descriptor method instead. Where this occurs in the +precedence chain depends on which descriptor methods were defined and +how they were called. + +The starting point for descriptor invocation is a binding, "a.x". How +the arguments are assembled depends on "a": + +Direct Call + The simplest and least common call is when user code directly + invokes a descriptor method: "x.__get__(a)". + +Instance Binding + If binding to an object instance, "a.x" is transformed into the + call: "type(a).__dict__['x'].__get__(a, type(a))". + +Class Binding + If binding to a class, "A.x" is transformed into the call: + "A.__dict__['x'].__get__(None, A)". + +Super Binding + A dotted lookup such as "super(A, a).x" searches + "a.__class__.__mro__" for a base class "B" following "A" and then + returns "B.__dict__['x'].__get__(a, A)". If not a descriptor, "x" + is returned unchanged. + +For instance bindings, the precedence of descriptor invocation depends +on which descriptor methods are defined. A descriptor can define any +combination of "__get__()", "__set__()" and "__delete__()". If it +does not define "__get__()", then accessing the attribute will return +the descriptor object itself unless there is a value in the object’s +instance dictionary. If the descriptor defines "__set__()" and/or +"__delete__()", it is a data descriptor; if it defines neither, it is +a non-data descriptor. Normally, data descriptors define both +"__get__()" and "__set__()", while non-data descriptors have just the +"__get__()" method. Data descriptors with "__get__()" and "__set__()" +(and/or "__delete__()") defined always override a redefinition in an +instance dictionary. In contrast, non-data descriptors can be +overridden by instances. + +Python methods (including those decorated with "@staticmethod" and +"@classmethod") are implemented as non-data descriptors. Accordingly, +instances can redefine and override methods. This allows individual +instances to acquire behaviors that differ from other instances of the +same class. + +The "property()" function is implemented as a data descriptor. +Accordingly, instances cannot override the behavior of a property. + + +__slots__ +--------- + +*__slots__* allow us to explicitly declare data members (like +properties) and deny the creation of "__dict__" and *__weakref__* +(unless explicitly declared in *__slots__* or available in a parent.) + +The space saved over using "__dict__" can be significant. Attribute +lookup speed can be significantly improved as well. + +object.__slots__ + + This class variable can be assigned a string, iterable, or sequence + of strings with variable names used by instances. *__slots__* + reserves space for the declared variables and prevents the + automatic creation of "__dict__" and *__weakref__* for each + instance. + +Notes on using *__slots__*: + +* When inheriting from a class without *__slots__*, the "__dict__" and + *__weakref__* attribute of the instances will always be accessible. + +* Without a "__dict__" variable, instances cannot be assigned new + variables not listed in the *__slots__* definition. Attempts to + assign to an unlisted variable name raises "AttributeError". If + dynamic assignment of new variables is desired, then add + "'__dict__'" to the sequence of strings in the *__slots__* + declaration. + +* Without a *__weakref__* variable for each instance, classes defining + *__slots__* do not support "weak references" to its instances. If + weak reference support is needed, then add "'__weakref__'" to the + sequence of strings in the *__slots__* declaration. + +* *__slots__* are implemented at the class level by creating + descriptors for each variable name. As a result, class attributes + cannot be used to set default values for instance variables defined + by *__slots__*; otherwise, the class attribute would overwrite the + descriptor assignment. + +* The action of a *__slots__* declaration is not limited to the class + where it is defined. *__slots__* declared in parents are available + in child classes. However, instances of a child subclass will get a + "__dict__" and *__weakref__* unless the subclass also defines + *__slots__* (which should only contain names of any *additional* + slots). + +* If a class defines a slot also defined in a base class, the instance + variable defined by the base class slot is inaccessible (except by + retrieving its descriptor directly from the base class). This + renders the meaning of the program undefined. In the future, a + check may be added to prevent this. + +* "TypeError" will be raised if nonempty *__slots__* are defined for a + class derived from a ""variable-length" built-in type" such as + "int", "bytes", and "tuple". + +* Any non-string *iterable* may be assigned to *__slots__*. + +* If a "dictionary" is used to assign *__slots__*, the dictionary keys + will be used as the slot names. The values of the dictionary can be + used to provide per-attribute docstrings that will be recognised by + "inspect.getdoc()" and displayed in the output of "help()". + +* "__class__" assignment works only if both classes have the same + *__slots__*. + +* Multiple inheritance with multiple slotted parent classes can be + used, but only one parent is allowed to have attributes created by + slots (the other bases must have empty slot layouts) - violations + raise "TypeError". + +* If an *iterator* is used for *__slots__* then a *descriptor* is + created for each of the iterator’s values. However, the *__slots__* + attribute will be an empty iterator. + + +Customizing class creation +========================== + +Whenever a class inherits from another class, "__init_subclass__()" is +called on the parent class. This way, it is possible to write classes +which change the behavior of subclasses. This is closely related to +class decorators, but where class decorators only affect the specific +class they’re applied to, "__init_subclass__" solely applies to future +subclasses of the class defining the method. + +classmethod object.__init_subclass__(cls) + + This method is called whenever the containing class is subclassed. + *cls* is then the new subclass. If defined as a normal instance + method, this method is implicitly converted to a class method. + + Keyword arguments which are given to a new class are passed to the + parent class’s "__init_subclass__". For compatibility with other + classes using "__init_subclass__", one should take out the needed + keyword arguments and pass the others over to the base class, as + in: + + class Philosopher: + def __init_subclass__(cls, /, default_name, **kwargs): + super().__init_subclass__(**kwargs) + cls.default_name = default_name + + class AustralianPhilosopher(Philosopher, default_name="Bruce"): + pass + + The default implementation "object.__init_subclass__" does nothing, + but raises an error if it is called with any arguments. + + Note: + + The metaclass hint "metaclass" is consumed by the rest of the + type machinery, and is never passed to "__init_subclass__" + implementations. The actual metaclass (rather than the explicit + hint) can be accessed as "type(cls)". + + Added in version 3.6. + +When a class is created, "type.__new__()" scans the class variables +and makes callbacks to those with a "__set_name__()" hook. + +object.__set_name__(self, owner, name) + + Automatically called at the time the owning class *owner* is + created. The object has been assigned to *name* in that class: + + class A: + x = C() # Automatically calls: x.__set_name__(A, 'x') + + If the class variable is assigned after the class is created, + "__set_name__()" will not be called automatically. If needed, + "__set_name__()" can be called directly: + + class A: + pass + + c = C() + A.x = c # The hook is not called + c.__set_name__(A, 'x') # Manually invoke the hook + + See Creating the class object for more details. + + Added in version 3.6. + + +Metaclasses +----------- + +By default, classes are constructed using "type()". The class body is +executed in a new namespace and the class name is bound locally to the +result of "type(name, bases, namespace)". + +The class creation process can be customized by passing the +"metaclass" keyword argument in the class definition line, or by +inheriting from an existing class that included such an argument. In +the following example, both "MyClass" and "MySubclass" are instances +of "Meta": + + class Meta(type): + pass + + class MyClass(metaclass=Meta): + pass + + class MySubclass(MyClass): + pass + +Any other keyword arguments that are specified in the class definition +are passed through to all metaclass operations described below. + +When a class definition is executed, the following steps occur: + +* MRO entries are resolved; + +* the appropriate metaclass is determined; + +* the class namespace is prepared; + +* the class body is executed; + +* the class object is created. + + +Resolving MRO entries +--------------------- + +object.__mro_entries__(self, bases) + + If a base that appears in a class definition is not an instance of + "type", then an "__mro_entries__()" method is searched on the base. + If an "__mro_entries__()" method is found, the base is substituted + with the result of a call to "__mro_entries__()" when creating the + class. The method is called with the original bases tuple passed to + the *bases* parameter, and must return a tuple of classes that will + be used instead of the base. The returned tuple may be empty: in + these cases, the original base is ignored. + +See also: + + "types.resolve_bases()" + Dynamically resolve bases that are not instances of "type". + + "types.get_original_bases()" + Retrieve a class’s “original bases” prior to modifications by + "__mro_entries__()". + + **PEP 560** + Core support for typing module and generic types. + + +Determining the appropriate metaclass +------------------------------------- + +The appropriate metaclass for a class definition is determined as +follows: + +* if no bases and no explicit metaclass are given, then "type()" is + used; + +* if an explicit metaclass is given and it is *not* an instance of + "type()", then it is used directly as the metaclass; + +* if an instance of "type()" is given as the explicit metaclass, or + bases are defined, then the most derived metaclass is used. + +The most derived metaclass is selected from the explicitly specified +metaclass (if any) and the metaclasses (i.e. "type(cls)") of all +specified base classes. The most derived metaclass is one which is a +subtype of *all* of these candidate metaclasses. If none of the +candidate metaclasses meets that criterion, then the class definition +will fail with "TypeError". + + +Preparing the class namespace +----------------------------- + +Once the appropriate metaclass has been identified, then the class +namespace is prepared. If the metaclass has a "__prepare__" attribute, +it is called as "namespace = metaclass.__prepare__(name, bases, +**kwds)" (where the additional keyword arguments, if any, come from +the class definition). The "__prepare__" method should be implemented +as a "classmethod". The namespace returned by "__prepare__" is passed +in to "__new__", but when the final class object is created the +namespace is copied into a new "dict". + +If the metaclass has no "__prepare__" attribute, then the class +namespace is initialised as an empty ordered mapping. + +See also: + + **PEP 3115** - Metaclasses in Python 3000 + Introduced the "__prepare__" namespace hook + + +Executing the class body +------------------------ + +The class body is executed (approximately) as "exec(body, globals(), +namespace)". The key difference from a normal call to "exec()" is that +lexical scoping allows the class body (including any methods) to +reference names from the current and outer scopes when the class +definition occurs inside a function. + +However, even when the class definition occurs inside the function, +methods defined inside the class still cannot see names defined at the +class scope. Class variables must be accessed through the first +parameter of instance or class methods, or through the implicit +lexically scoped "__class__" reference described in the next section. + + +Creating the class object +------------------------- + +Once the class namespace has been populated by executing the class +body, the class object is created by calling "metaclass(name, bases, +namespace, **kwds)" (the additional keywords passed here are the same +as those passed to "__prepare__"). + +This class object is the one that will be referenced by the zero- +argument form of "super()". "__class__" is an implicit closure +reference created by the compiler if any methods in a class body refer +to either "__class__" or "super". This allows the zero argument form +of "super()" to correctly identify the class being defined based on +lexical scoping, while the class or instance that was used to make the +current call is identified based on the first argument passed to the +method. + +**CPython implementation detail:** In CPython 3.6 and later, the +"__class__" cell is passed to the metaclass as a "__classcell__" entry +in the class namespace. If present, this must be propagated up to the +"type.__new__" call in order for the class to be initialised +correctly. Failing to do so will result in a "RuntimeError" in Python +3.8. + +When using the default metaclass "type", or any metaclass that +ultimately calls "type.__new__", the following additional +customization steps are invoked after creating the class object: + +1. The "type.__new__" method collects all of the attributes in the + class namespace that define a "__set_name__()" method; + +2. Those "__set_name__" methods are called with the class being + defined and the assigned name of that particular attribute; + +3. The "__init_subclass__()" hook is called on the immediate parent of + the new class in its method resolution order. + +After the class object is created, it is passed to the class +decorators included in the class definition (if any) and the resulting +object is bound in the local namespace as the defined class. + +When a new class is created by "type.__new__", the object provided as +the namespace parameter is copied to a new ordered mapping and the +original object is discarded. The new copy is wrapped in a read-only +proxy, which becomes the "__dict__" attribute of the class object. + +See also: + + **PEP 3135** - New super + Describes the implicit "__class__" closure reference + + +Uses for metaclasses +-------------------- + +The potential uses for metaclasses are boundless. Some ideas that have +been explored include enum, logging, interface checking, automatic +delegation, automatic property creation, proxies, frameworks, and +automatic resource locking/synchronization. + + +Customizing instance and subclass checks +======================================== + +The following methods are used to override the default behavior of the +"isinstance()" and "issubclass()" built-in functions. + +In particular, the metaclass "abc.ABCMeta" implements these methods in +order to allow the addition of Abstract Base Classes (ABCs) as +“virtual base classes” to any class or type (including built-in +types), including other ABCs. + +type.__instancecheck__(self, instance) + + Return true if *instance* should be considered a (direct or + indirect) instance of *class*. If defined, called to implement + "isinstance(instance, class)". + +type.__subclasscheck__(self, subclass) + + Return true if *subclass* should be considered a (direct or + indirect) subclass of *class*. If defined, called to implement + "issubclass(subclass, class)". + +Note that these methods are looked up on the type (metaclass) of a +class. They cannot be defined as class methods in the actual class. +This is consistent with the lookup of special methods that are called +on instances, only in this case the instance is itself a class. + +See also: + + **PEP 3119** - Introducing Abstract Base Classes + Includes the specification for customizing "isinstance()" and + "issubclass()" behavior through "__instancecheck__()" and + "__subclasscheck__()", with motivation for this functionality in + the context of adding Abstract Base Classes (see the "abc" + module) to the language. + + +Emulating generic types +======================= + +When using *type annotations*, it is often useful to *parameterize* a +*generic type* using Python’s square-brackets notation. For example, +the annotation "list[int]" might be used to signify a "list" in which +all the elements are of type "int". + +See also: + + **PEP 484** - Type Hints + Introducing Python’s framework for type annotations + + Generic Alias Types + Documentation for objects representing parameterized generic + classes + + Generics, user-defined generics and "typing.Generic" + Documentation on how to implement generic classes that can be + parameterized at runtime and understood by static type-checkers. + +A class can *generally* only be parameterized if it defines the +special class method "__class_getitem__()". + +classmethod object.__class_getitem__(cls, key) + + Return an object representing the specialization of a generic class + by type arguments found in *key*. + + When defined on a class, "__class_getitem__()" is automatically a + class method. As such, there is no need for it to be decorated with + "@classmethod" when it is defined. + + +The purpose of *__class_getitem__* +---------------------------------- + +The purpose of "__class_getitem__()" is to allow runtime +parameterization of standard-library generic classes in order to more +easily apply *type hints* to these classes. + +To implement custom generic classes that can be parameterized at +runtime and understood by static type-checkers, users should either +inherit from a standard library class that already implements +"__class_getitem__()", or inherit from "typing.Generic", which has its +own implementation of "__class_getitem__()". + +Custom implementations of "__class_getitem__()" on classes defined +outside of the standard library may not be understood by third-party +type-checkers such as mypy. Using "__class_getitem__()" on any class +for purposes other than type hinting is discouraged. + + +*__class_getitem__* versus *__getitem__* +---------------------------------------- + +Usually, the subscription of an object using square brackets will call +the "__getitem__()" instance method defined on the object’s class. +However, if the object being subscribed is itself a class, the class +method "__class_getitem__()" may be called instead. +"__class_getitem__()" should return a GenericAlias object if it is +properly defined. + +Presented with the *expression* "obj[x]", the Python interpreter +follows something like the following process to decide whether +"__getitem__()" or "__class_getitem__()" should be called: + + from inspect import isclass + + def subscribe(obj, x): + """Return the result of the expression 'obj[x]'""" + + class_of_obj = type(obj) + + # If the class of obj defines __getitem__, + # call class_of_obj.__getitem__(obj, x) + if hasattr(class_of_obj, '__getitem__'): + return class_of_obj.__getitem__(obj, x) + + # Else, if obj is a class and defines __class_getitem__, + # call obj.__class_getitem__(x) + elif isclass(obj) and hasattr(obj, '__class_getitem__'): + return obj.__class_getitem__(x) + + # Else, raise an exception + else: + raise TypeError( + f"'{class_of_obj.__name__}' object is not subscriptable" + ) + +In Python, all classes are themselves instances of other classes. The +class of a class is known as that class’s *metaclass*, and most +classes have the "type" class as their metaclass. "type" does not +define "__getitem__()", meaning that expressions such as "list[int]", +"dict[str, float]" and "tuple[str, bytes]" all result in +"__class_getitem__()" being called: + + >>> # list has class "type" as its metaclass, like most classes: + >>> type(list) + + >>> type(dict) == type(list) == type(tuple) == type(str) == type(bytes) + True + >>> # "list[int]" calls "list.__class_getitem__(int)" + >>> list[int] + list[int] + >>> # list.__class_getitem__ returns a GenericAlias object: + >>> type(list[int]) + + +However, if a class has a custom metaclass that defines +"__getitem__()", subscribing the class may result in different +behaviour. An example of this can be found in the "enum" module: + + >>> from enum import Enum + >>> class Menu(Enum): + ... """A breakfast menu""" + ... SPAM = 'spam' + ... BACON = 'bacon' + ... + >>> # Enum classes have a custom metaclass: + >>> type(Menu) + + >>> # EnumMeta defines __getitem__, + >>> # so __class_getitem__ is not called, + >>> # and the result is not a GenericAlias object: + >>> Menu['SPAM'] + + >>> type(Menu['SPAM']) + + +See also: + + **PEP 560** - Core Support for typing module and generic types + Introducing "__class_getitem__()", and outlining when a + subscription results in "__class_getitem__()" being called + instead of "__getitem__()" + + +Emulating callable objects +========================== + +object.__call__(self[, args...]) + + Called when the instance is “called” as a function; if this method + is defined, "x(arg1, arg2, ...)" roughly translates to + "type(x).__call__(x, arg1, ...)". The "object" class itself does + not provide this method. + + +Emulating container types +========================= + +The following methods can be defined to implement container objects. +None of them are provided by the "object" class itself. Containers +usually are *sequences* (such as "lists" or "tuples") or *mappings* +(like *dictionaries*), but can represent other containers as well. +The first set of methods is used either to emulate a sequence or to +emulate a mapping; the difference is that for a sequence, the +allowable keys should be the integers *k* for which "0 <= k < N" where +*N* is the length of the sequence, or "slice" objects, which define a +range of items. It is also recommended that mappings provide the +methods "keys()", "values()", "items()", "get()", "clear()", +"setdefault()", "pop()", "popitem()", "copy()", and "update()" +behaving similar to those for Python’s standard "dictionary" objects. +The "collections.abc" module provides a "MutableMapping" *abstract +base class* to help create those methods from a base set of +"__getitem__()", "__setitem__()", "__delitem__()", and "keys()". + +Mutable sequences should provide methods "append()", "clear()", +"count()", "extend()", "index()", "insert()", "pop()", "remove()", and +"reverse()", like Python standard "list" objects. Finally, sequence +types should implement addition (meaning concatenation) and +multiplication (meaning repetition) by defining the methods +"__add__()", "__radd__()", "__iadd__()", "__mul__()", "__rmul__()" and +"__imul__()" described below; they should not define other numerical +operators. + +It is recommended that both mappings and sequences implement the +"__contains__()" method to allow efficient use of the "in" operator; +for mappings, "in" should search the mapping’s keys; for sequences, it +should search through the values. It is further recommended that both +mappings and sequences implement the "__iter__()" method to allow +efficient iteration through the container; for mappings, "__iter__()" +should iterate through the object’s keys; for sequences, it should +iterate through the values. + +object.__len__(self) + + Called to implement the built-in function "len()". Should return + the length of the object, an integer ">=" 0. Also, an object that + doesn’t define a "__bool__()" method and whose "__len__()" method + returns zero is considered to be false in a Boolean context. + + **CPython implementation detail:** In CPython, the length is + required to be at most "sys.maxsize". If the length is larger than + "sys.maxsize" some features (such as "len()") may raise + "OverflowError". To prevent raising "OverflowError" by truth value + testing, an object must define a "__bool__()" method. + +object.__length_hint__(self) + + Called to implement "operator.length_hint()". Should return an + estimated length for the object (which may be greater or less than + the actual length). The length must be an integer ">=" 0. The + return value may also be "NotImplemented", which is treated the + same as if the "__length_hint__" method didn’t exist at all. This + method is purely an optimization and is never required for + correctness. + + Added in version 3.4. + +Note: + + Slicing is done exclusively with the following three methods. A + call like + + a[1:2] = b + + is translated to + + a[slice(1, 2, None)] = b + + and so forth. Missing slice items are always filled in with "None". + +object.__getitem__(self, key) + + Called to implement evaluation of "self[key]". For *sequence* + types, the accepted keys should be integers. Optionally, they may + support "slice" objects as well. Negative index support is also + optional. If *key* is of an inappropriate type, "TypeError" may be + raised; if *key* is a value outside the set of indexes for the + sequence (after any special interpretation of negative values), + "IndexError" should be raised. For *mapping* types, if *key* is + missing (not in the container), "KeyError" should be raised. + + Note: + + "for" loops expect that an "IndexError" will be raised for + illegal indexes to allow proper detection of the end of the + sequence. + + Note: + + When subscripting a *class*, the special class method + "__class_getitem__()" may be called instead of "__getitem__()". + See __class_getitem__ versus __getitem__ for more details. + +object.__setitem__(self, key, value) + + Called to implement assignment to "self[key]". Same note as for + "__getitem__()". This should only be implemented for mappings if + the objects support changes to the values for keys, or if new keys + can be added, or for sequences if elements can be replaced. The + same exceptions should be raised for improper *key* values as for + the "__getitem__()" method. + +object.__delitem__(self, key) + + Called to implement deletion of "self[key]". Same note as for + "__getitem__()". This should only be implemented for mappings if + the objects support removal of keys, or for sequences if elements + can be removed from the sequence. The same exceptions should be + raised for improper *key* values as for the "__getitem__()" method. + +object.__missing__(self, key) + + Called by "dict"."__getitem__()" to implement "self[key]" for dict + subclasses when key is not in the dictionary. + +object.__iter__(self) + + This method is called when an *iterator* is required for a + container. This method should return a new iterator object that can + iterate over all the objects in the container. For mappings, it + should iterate over the keys of the container. + +object.__reversed__(self) + + Called (if present) by the "reversed()" built-in to implement + reverse iteration. It should return a new iterator object that + iterates over all the objects in the container in reverse order. + + If the "__reversed__()" method is not provided, the "reversed()" + built-in will fall back to using the sequence protocol ("__len__()" + and "__getitem__()"). Objects that support the sequence protocol + should only provide "__reversed__()" if they can provide an + implementation that is more efficient than the one provided by + "reversed()". + +The membership test operators ("in" and "not in") are normally +implemented as an iteration through a container. However, container +objects can supply the following special method with a more efficient +implementation, which also does not require the object be iterable. + +object.__contains__(self, item) + + Called to implement membership test operators. Should return true + if *item* is in *self*, false otherwise. For mapping objects, this + should consider the keys of the mapping rather than the values or + the key-item pairs. + + For objects that don’t define "__contains__()", the membership test + first tries iteration via "__iter__()", then the old sequence + iteration protocol via "__getitem__()", see this section in the + language reference. + + +Emulating numeric types +======================= + +The following methods can be defined to emulate numeric objects. +Methods corresponding to operations that are not supported by the +particular kind of number implemented (e.g., bitwise operations for +non-integral numbers) should be left undefined. + +object.__add__(self, other) +object.__sub__(self, other) +object.__mul__(self, other) +object.__matmul__(self, other) +object.__truediv__(self, other) +object.__floordiv__(self, other) +object.__mod__(self, other) +object.__divmod__(self, other) +object.__pow__(self, other[, modulo]) +object.__lshift__(self, other) +object.__rshift__(self, other) +object.__and__(self, other) +object.__xor__(self, other) +object.__or__(self, other) + + These methods are called to implement the binary arithmetic + operations ("+", "-", "*", "@", "/", "//", "%", "divmod()", + "pow()", "**", "<<", ">>", "&", "^", "|"). For instance, to + evaluate the expression "x + y", where *x* is an instance of a + class that has an "__add__()" method, "type(x).__add__(x, y)" is + called. The "__divmod__()" method should be the equivalent to + using "__floordiv__()" and "__mod__()"; it should not be related to + "__truediv__()". Note that "__pow__()" should be defined to accept + an optional third argument if the three-argument version of the + built-in "pow()" function is to be supported. + + If one of those methods does not support the operation with the + supplied arguments, it should return "NotImplemented". + +object.__radd__(self, other) +object.__rsub__(self, other) +object.__rmul__(self, other) +object.__rmatmul__(self, other) +object.__rtruediv__(self, other) +object.__rfloordiv__(self, other) +object.__rmod__(self, other) +object.__rdivmod__(self, other) +object.__rpow__(self, other[, modulo]) +object.__rlshift__(self, other) +object.__rrshift__(self, other) +object.__rand__(self, other) +object.__rxor__(self, other) +object.__ror__(self, other) + + These methods are called to implement the binary arithmetic + operations ("+", "-", "*", "@", "/", "//", "%", "divmod()", + "pow()", "**", "<<", ">>", "&", "^", "|") with reflected (swapped) + operands. These functions are only called if the operands are of + different types, when the left operand does not support the + corresponding operation [3], or the right operand’s class is + derived from the left operand’s class. [4] For instance, to + evaluate the expression "x - y", where *y* is an instance of a + class that has an "__rsub__()" method, "type(y).__rsub__(y, x)" is + called if "type(x).__sub__(x, y)" returns "NotImplemented" or + "type(y)" is a subclass of "type(x)". [5] + + Note that "__rpow__()" should be defined to accept an optional + third argument if the three-argument version of the built-in + "pow()" function is to be supported. + + Changed in version 3.14: Three-argument "pow()" now try calling + "__rpow__()" if necessary. Previously it was only called in two- + argument "pow()" and the binary power operator. + + Note: + + If the right operand’s type is a subclass of the left operand’s + type and that subclass provides a different implementation of the + reflected method for the operation, this method will be called + before the left operand’s non-reflected method. This behavior + allows subclasses to override their ancestors’ operations. + +object.__iadd__(self, other) +object.__isub__(self, other) +object.__imul__(self, other) +object.__imatmul__(self, other) +object.__itruediv__(self, other) +object.__ifloordiv__(self, other) +object.__imod__(self, other) +object.__ipow__(self, other[, modulo]) +object.__ilshift__(self, other) +object.__irshift__(self, other) +object.__iand__(self, other) +object.__ixor__(self, other) +object.__ior__(self, other) + + These methods are called to implement the augmented arithmetic + assignments ("+=", "-=", "*=", "@=", "/=", "//=", "%=", "**=", + "<<=", ">>=", "&=", "^=", "|="). These methods should attempt to + do the operation in-place (modifying *self*) and return the result + (which could be, but does not have to be, *self*). If a specific + method is not defined, or if that method returns "NotImplemented", + the augmented assignment falls back to the normal methods. For + instance, if *x* is an instance of a class with an "__iadd__()" + method, "x += y" is equivalent to "x = x.__iadd__(y)" . If + "__iadd__()" does not exist, or if "x.__iadd__(y)" returns + "NotImplemented", "x.__add__(y)" and "y.__radd__(x)" are + considered, as with the evaluation of "x + y". In certain + situations, augmented assignment can result in unexpected errors + (see Why does a_tuple[i] += [‘item’] raise an exception when the + addition works?), but this behavior is in fact part of the data + model. + +object.__neg__(self) +object.__pos__(self) +object.__abs__(self) +object.__invert__(self) + + Called to implement the unary arithmetic operations ("-", "+", + "abs()" and "~"). + +object.__complex__(self) +object.__int__(self) +object.__float__(self) + + Called to implement the built-in functions "complex()", "int()" and + "float()". Should return a value of the appropriate type. + +object.__index__(self) + + Called to implement "operator.index()", and whenever Python needs + to losslessly convert the numeric object to an integer object (such + as in slicing, or in the built-in "bin()", "hex()" and "oct()" + functions). Presence of this method indicates that the numeric + object is an integer type. Must return an integer. + + If "__int__()", "__float__()" and "__complex__()" are not defined + then corresponding built-in functions "int()", "float()" and + "complex()" fall back to "__index__()". + +object.__round__(self[, ndigits]) +object.__trunc__(self) +object.__floor__(self) +object.__ceil__(self) + + Called to implement the built-in function "round()" and "math" + functions "trunc()", "floor()" and "ceil()". Unless *ndigits* is + passed to "__round__()" all these methods should return the value + of the object truncated to an "Integral" (typically an "int"). + + Changed in version 3.14: "int()" no longer delegates to the + "__trunc__()" method. + + +With Statement Context Managers +=============================== + +A *context manager* is an object that defines the runtime context to +be established when executing a "with" statement. The context manager +handles the entry into, and the exit from, the desired runtime context +for the execution of the block of code. Context managers are normally +invoked using the "with" statement (described in section The with +statement), but can also be used by directly invoking their methods. + +Typical uses of context managers include saving and restoring various +kinds of global state, locking and unlocking resources, closing opened +files, etc. + +For more information on context managers, see Context Manager Types. +The "object" class itself does not provide the context manager +methods. + +object.__enter__(self) + + Enter the runtime context related to this object. The "with" + statement will bind this method’s return value to the target(s) + specified in the "as" clause of the statement, if any. + +object.__exit__(self, exc_type, exc_value, traceback) + + Exit the runtime context related to this object. The parameters + describe the exception that caused the context to be exited. If the + context was exited without an exception, all three arguments will + be "None". + + If an exception is supplied, and the method wishes to suppress the + exception (i.e., prevent it from being propagated), it should + return a true value. Otherwise, the exception will be processed + normally upon exit from this method. + + Note that "__exit__()" methods should not reraise the passed-in + exception; this is the caller’s responsibility. + +See also: + + **PEP 343** - The “with” statement + The specification, background, and examples for the Python "with" + statement. + + +Customizing positional arguments in class pattern matching +========================================================== + +When using a class name in a pattern, positional arguments in the +pattern are not allowed by default, i.e. "case MyClass(x, y)" is +typically invalid without special support in "MyClass". To be able to +use that kind of pattern, the class needs to define a *__match_args__* +attribute. + +object.__match_args__ + + This class variable can be assigned a tuple of strings. When this + class is used in a class pattern with positional arguments, each + positional argument will be converted into a keyword argument, + using the corresponding value in *__match_args__* as the keyword. + The absence of this attribute is equivalent to setting it to "()". + +For example, if "MyClass.__match_args__" is "("left", "center", +"right")" that means that "case MyClass(x, y)" is equivalent to "case +MyClass(left=x, center=y)". Note that the number of arguments in the +pattern must be smaller than or equal to the number of elements in +*__match_args__*; if it is larger, the pattern match attempt will +raise a "TypeError". + +Added in version 3.10. + +See also: + + **PEP 634** - Structural Pattern Matching + The specification for the Python "match" statement. + + +Emulating buffer types +====================== + +The buffer protocol provides a way for Python objects to expose +efficient access to a low-level memory array. This protocol is +implemented by builtin types such as "bytes" and "memoryview", and +third-party libraries may define additional buffer types. + +While buffer types are usually implemented in C, it is also possible +to implement the protocol in Python. + +object.__buffer__(self, flags) + + Called when a buffer is requested from *self* (for example, by the + "memoryview" constructor). The *flags* argument is an integer + representing the kind of buffer requested, affecting for example + whether the returned buffer is read-only or writable. + "inspect.BufferFlags" provides a convenient way to interpret the + flags. The method must return a "memoryview" object. + +object.__release_buffer__(self, buffer) + + Called when a buffer is no longer needed. The *buffer* argument is + a "memoryview" object that was previously returned by + "__buffer__()". The method must release any resources associated + with the buffer. This method should return "None". Buffer objects + that do not need to perform any cleanup are not required to + implement this method. + +Added in version 3.12. + +See also: + + **PEP 688** - Making the buffer protocol accessible in Python + Introduces the Python "__buffer__" and "__release_buffer__" + methods. + + "collections.abc.Buffer" + ABC for buffer types. + + +Annotations +=========== + +Functions, classes, and modules may contain *annotations*, which are a +way to associate information (usually *type hints*) with a symbol. + +object.__annotations__ + + This attribute contains the annotations for an object. It is lazily + evaluated, so accessing the attribute may execute arbitrary code + and raise exceptions. If evaluation is successful, the attribute is + set to a dictionary mapping from variable names to annotations. + + Changed in version 3.14: Annotations are now lazily evaluated. + +object.__annotate__(format) + + An *annotate function*. Returns a new dictionary object mapping + attribute/parameter names to their annotation values. + + Takes a format parameter specifying the format in which annotations + values should be provided. It must be a member of the + "annotationlib.Format" enum, or an integer with a value + corresponding to a member of the enum. + + If an annotate function doesn’t support the requested format, it + must raise "NotImplementedError". Annotate functions must always + support "VALUE" format; they must not raise "NotImplementedError()" + when called with this format. + + When called with "VALUE" format, an annotate function may raise + "NameError"; it must not raise "NameError" when called requesting + any other format. + + If an object does not have any annotations, "__annotate__" should + preferably be set to "None" (it can’t be deleted), rather than set + to a function that returns an empty dict. + + Added in version 3.14. + +See also: + + **PEP 649** — Deferred evaluation of annotation using descriptors + Introduces lazy evaluation of annotations and the "__annotate__" + function. + + +Special method lookup +===================== + +For custom classes, implicit invocations of special methods are only +guaranteed to work correctly if defined on an object’s type, not in +the object’s instance dictionary. That behaviour is the reason why +the following code raises an exception: + + >>> class C: + ... pass + ... + >>> c = C() + >>> c.__len__ = lambda: 5 + >>> len(c) + Traceback (most recent call last): + File "", line 1, in + TypeError: object of type 'C' has no len() + +The rationale behind this behaviour lies with a number of special +methods such as "__hash__()" and "__repr__()" that are implemented by +all objects, including type objects. If the implicit lookup of these +methods used the conventional lookup process, they would fail when +invoked on the type object itself: + + >>> 1 .__hash__() == hash(1) + True + >>> int.__hash__() == hash(int) + Traceback (most recent call last): + File "", line 1, in + TypeError: descriptor '__hash__' of 'int' object needs an argument + +Incorrectly attempting to invoke an unbound method of a class in this +way is sometimes referred to as ‘metaclass confusion’, and is avoided +by bypassing the instance when looking up special methods: + + >>> type(1).__hash__(1) == hash(1) + True + >>> type(int).__hash__(int) == hash(int) + True + +In addition to bypassing any instance attributes in the interest of +correctness, implicit special method lookup generally also bypasses +the "__getattribute__()" method even of the object’s metaclass: + + >>> class Meta(type): + ... def __getattribute__(*args): + ... print("Metaclass getattribute invoked") + ... return type.__getattribute__(*args) + ... + >>> class C(object, metaclass=Meta): + ... def __len__(self): + ... return 10 + ... def __getattribute__(*args): + ... print("Class getattribute invoked") + ... return object.__getattribute__(*args) + ... + >>> c = C() + >>> c.__len__() # Explicit lookup via instance + Class getattribute invoked + 10 + >>> type(c).__len__(c) # Explicit lookup via type + Metaclass getattribute invoked + 10 + >>> len(c) # Implicit lookup + 10 + +Bypassing the "__getattribute__()" machinery in this fashion provides +significant scope for speed optimisations within the interpreter, at +the cost of some flexibility in the handling of special methods (the +special method *must* be set on the class object itself in order to be +consistently invoked by the interpreter). +''', + 'string-methods': r'''String Methods +************** + +Strings implement all of the common sequence operations, along with +the additional methods described below. + +Strings also support two styles of string formatting, one providing a +large degree of flexibility and customization (see "str.format()", +Format String Syntax and Custom String Formatting) and the other based +on C "printf" style formatting that handles a narrower range of types +and is slightly harder to use correctly, but is often faster for the +cases it can handle (printf-style String Formatting). + +The Text Processing Services section of the standard library covers a +number of other modules that provide various text related utilities +(including regular expression support in the "re" module). + +str.capitalize() + + Return a copy of the string with its first character capitalized + and the rest lowercased. + + Changed in version 3.8: The first character is now put into + titlecase rather than uppercase. This means that characters like + digraphs will only have their first letter capitalized, instead of + the full character. + +str.casefold() + + Return a casefolded copy of the string. Casefolded strings may be + used for caseless matching. + + Casefolding is similar to lowercasing but more aggressive because + it is intended to remove all case distinctions in a string. For + example, the German lowercase letter "'ß'" is equivalent to ""ss"". + Since it is already lowercase, "lower()" would do nothing to "'ß'"; + "casefold()" converts it to ""ss"". + + The casefolding algorithm is described in section 3.13 ‘Default + Case Folding’ of the Unicode Standard. + + Added in version 3.3. + +str.center(width, fillchar=' ', /) + + Return centered in a string of length *width*. Padding is done + using the specified *fillchar* (default is an ASCII space). The + original string is returned if *width* is less than or equal to + "len(s)". For example: + + >>> 'Python'.center(10) + ' Python ' + >>> 'Python'.center(10, '-') + '--Python--' + >>> 'Python'.center(4) + 'Python' + +str.count(sub[, start[, end]]) + + Return the number of non-overlapping occurrences of substring *sub* + in the range [*start*, *end*]. Optional arguments *start* and + *end* are interpreted as in slice notation. + + If *sub* is empty, returns the number of empty strings between + characters which is the length of the string plus one. For example: + + >>> 'spam, spam, spam'.count('spam') + 3 + >>> 'spam, spam, spam'.count('spam', 5) + 2 + >>> 'spam, spam, spam'.count('spam', 5, 10) + 1 + >>> 'spam, spam, spam'.count('eggs') + 0 + >>> 'spam, spam, spam'.count('') + 17 + +str.encode(encoding='utf-8', errors='strict') + + Return the string encoded to "bytes". + + *encoding* defaults to "'utf-8'"; see Standard Encodings for + possible values. + + *errors* controls how encoding errors are handled. If "'strict'" + (the default), a "UnicodeError" exception is raised. Other possible + values are "'ignore'", "'replace'", "'xmlcharrefreplace'", + "'backslashreplace'" and any other name registered via + "codecs.register_error()". See Error Handlers for details. + + For performance reasons, the value of *errors* is not checked for + validity unless an encoding error actually occurs, Python + Development Mode is enabled or a debug build is used. For example: + + >>> encoded_str_to_bytes = 'Python'.encode() + >>> type(encoded_str_to_bytes) + + >>> encoded_str_to_bytes + b'Python' + + Changed in version 3.1: Added support for keyword arguments. + + Changed in version 3.9: The value of the *errors* argument is now + checked in Python Development Mode and in debug mode. + +str.endswith(suffix[, start[, end]]) + + Return "True" if the string ends with the specified *suffix*, + otherwise return "False". *suffix* can also be a tuple of suffixes + to look for. With optional *start*, test beginning at that + position. With optional *end*, stop comparing at that position. + Using *start* and *end* is equivalent to + "str[start:end].endswith(suffix)". For example: + + >>> 'Python'.endswith('on') + True + >>> 'a tuple of suffixes'.endswith(('at', 'in')) + False + >>> 'a tuple of suffixes'.endswith(('at', 'es')) + True + >>> 'Python is amazing'.endswith('is', 0, 9) + True + + See also "startswith()" and "removesuffix()". + +str.expandtabs(tabsize=8) + + Return a copy of the string where all tab characters are replaced + by one or more spaces, depending on the current column and the + given tab size. Tab positions occur every *tabsize* characters + (default is 8, giving tab positions at columns 0, 8, 16 and so on). + To expand the string, the current column is set to zero and the + string is examined character by character. If the character is a + tab ("\t"), one or more space characters are inserted in the result + until the current column is equal to the next tab position. (The + tab character itself is not copied.) If the character is a newline + ("\n") or return ("\r"), it is copied and the current column is + reset to zero. Any other character is copied unchanged and the + current column is incremented by one regardless of how the + character is represented when printed. For example: + + >>> '01\t012\t0123\t01234'.expandtabs() + '01 012 0123 01234' + >>> '01\t012\t0123\t01234'.expandtabs(4) + '01 012 0123 01234' + >>> print('01\t012\n0123\t01234'.expandtabs(4)) + 01 012 + 0123 01234 + +str.find(sub[, start[, end]]) + + Return the lowest index in the string where substring *sub* is + found within the slice "s[start:end]". Optional arguments *start* + and *end* are interpreted as in slice notation. Return "-1" if + *sub* is not found. For example: + + >>> 'spam, spam, spam'.find('sp') + 0 + >>> 'spam, spam, spam'.find('sp', 5) + 6 + + See also "rfind()" and "index()". + + Note: + + The "find()" method should be used only if you need to know the + position of *sub*. To check if *sub* is a substring or not, use + the "in" operator: + + >>> 'Py' in 'Python' + True + +str.format(*args, **kwargs) + + Perform a string formatting operation. The string on which this + method is called can contain literal text or replacement fields + delimited by braces "{}". Each replacement field contains either + the numeric index of a positional argument, or the name of a + keyword argument. Returns a copy of the string where each + replacement field is replaced with the string value of the + corresponding argument. For example: + + >>> "The sum of 1 + 2 is {0}".format(1+2) + 'The sum of 1 + 2 is 3' + >>> "The sum of {a} + {b} is {answer}".format(answer=1+2, a=1, b=2) + 'The sum of 1 + 2 is 3' + >>> "{1} expects the {0} Inquisition!".format("Spanish", "Nobody") + 'Nobody expects the Spanish Inquisition!' + + See Format String Syntax for a description of the various + formatting options that can be specified in format strings. + + Note: + + When formatting a number ("int", "float", "complex", + "decimal.Decimal" and subclasses) with the "n" type (ex: + "'{:n}'.format(1234)"), the function temporarily sets the + "LC_CTYPE" locale to the "LC_NUMERIC" locale to decode + "decimal_point" and "thousands_sep" fields of "localeconv()" if + they are non-ASCII or longer than 1 byte, and the "LC_NUMERIC" + locale is different than the "LC_CTYPE" locale. This temporary + change affects other threads. + + Changed in version 3.7: When formatting a number with the "n" type, + the function sets temporarily the "LC_CTYPE" locale to the + "LC_NUMERIC" locale in some cases. + +str.format_map(mapping, /) + + Similar to "str.format(**mapping)", except that "mapping" is used + directly and not copied to a "dict". This is useful if for example + "mapping" is a dict subclass: + + >>> class Default(dict): + ... def __missing__(self, key): + ... return key + ... + >>> '{name} was born in {country}'.format_map(Default(name='Guido')) + 'Guido was born in country' + + Added in version 3.2. + +str.index(sub[, start[, end]]) + + Like "find()", but raise "ValueError" when the substring is not + found. + +str.isalnum() + + Return "True" if all characters in the string are alphanumeric and + there is at least one character, "False" otherwise. A character + "c" is alphanumeric if one of the following returns "True": + "c.isalpha()", "c.isdecimal()", "c.isdigit()", or "c.isnumeric()". + +str.isalpha() + + Return "True" if all characters in the string are alphabetic and + there is at least one character, "False" otherwise. Alphabetic + characters are those characters defined in the Unicode character + database as “Letter”, i.e., those with general category property + being one of “Lm”, “Lt”, “Lu”, “Ll”, or “Lo”. Note that this is + different from the Alphabetic property defined in the section 4.10 + ‘Letters, Alphabetic, and Ideographic’ of the Unicode Standard. For + example: + + >>> 'Letters and spaces'.isalpha() + False + >>> 'LettersOnly'.isalpha() + True + >>> 'µ'.isalpha() # non-ASCII characters can be considered alphabetical too + True + + See Unicode Properties. + +str.isascii() + + Return "True" if the string is empty or all characters in the + string are ASCII, "False" otherwise. ASCII characters have code + points in the range U+0000-U+007F. For example: + + >>> 'ASCII characters'.isascii() + True + >>> 'µ'.isascii() + False + + Added in version 3.7. + +str.isdecimal() + + Return "True" if all characters in the string are decimal + characters and there is at least one character, "False" otherwise. + Decimal characters are those that can be used to form numbers in + base 10, such as U+0660, ARABIC-INDIC DIGIT ZERO. Formally a + decimal character is a character in the Unicode General Category + “Nd”. For example: + + >>> '0123456789'.isdecimal() + True + >>> '٠١٢٣٤٥٦٧٨٩'.isdecimal() # Arabic-Indic digits zero to nine + True + >>> 'alphabetic'.isdecimal() + False + +str.isdigit() + + Return "True" if all characters in the string are digits and there + is at least one character, "False" otherwise. Digits include + decimal characters and digits that need special handling, such as + the compatibility superscript digits. This covers digits which + cannot be used to form numbers in base 10, like the Kharosthi + numbers. Formally, a digit is a character that has the property + value Numeric_Type=Digit or Numeric_Type=Decimal. + +str.isidentifier() + + Return "True" if the string is a valid identifier according to the + language definition, section Names (identifiers and keywords). + + "keyword.iskeyword()" can be used to test whether string "s" is a + reserved identifier, such as "def" and "class". + + Example: + + >>> from keyword import iskeyword + + >>> 'hello'.isidentifier(), iskeyword('hello') + (True, False) + >>> 'def'.isidentifier(), iskeyword('def') + (True, True) + +str.islower() + + Return "True" if all cased characters [4] in the string are + lowercase and there is at least one cased character, "False" + otherwise. + +str.isnumeric() + + Return "True" if all characters in the string are numeric + characters, and there is at least one character, "False" otherwise. + Numeric characters include digit characters, and all characters + that have the Unicode numeric value property, e.g. U+2155, VULGAR + FRACTION ONE FIFTH. Formally, numeric characters are those with + the property value Numeric_Type=Digit, Numeric_Type=Decimal or + Numeric_Type=Numeric. + +str.isprintable() + + Return "True" if all characters in the string are printable, + "False" if it contains at least one non-printable character. + + Here “printable” means the character is suitable for "repr()" to + use in its output; “non-printable” means that "repr()" on built-in + types will hex-escape the character. It has no bearing on the + handling of strings written to "sys.stdout" or "sys.stderr". + + The printable characters are those which in the Unicode character + database (see "unicodedata") have a general category in group + Letter, Mark, Number, Punctuation, or Symbol (L, M, N, P, or S); + plus the ASCII space 0x20. Nonprintable characters are those in + group Separator or Other (Z or C), except the ASCII space. + +str.isspace() + + Return "True" if there are only whitespace characters in the string + and there is at least one character, "False" otherwise. + + A character is *whitespace* if in the Unicode character database + (see "unicodedata"), either its general category is "Zs" + (“Separator, space”), or its bidirectional class is one of "WS", + "B", or "S". + +str.istitle() + + Return "True" if the string is a titlecased string and there is at + least one character, for example uppercase characters may only + follow uncased characters and lowercase characters only cased ones. + Return "False" otherwise. + + For example: + + >>> 'Spam, Spam, Spam'.istitle() + True + >>> 'spam, spam, spam'.istitle() + False + >>> 'SPAM, SPAM, SPAM'.istitle() + False + + See also "title()". + +str.isupper() + + Return "True" if all cased characters [4] in the string are + uppercase and there is at least one cased character, "False" + otherwise. + + >>> 'BANANA'.isupper() + True + >>> 'banana'.isupper() + False + >>> 'baNana'.isupper() + False + >>> ' '.isupper() + False + +str.join(iterable, /) + + Return a string which is the concatenation of the strings in + *iterable*. A "TypeError" will be raised if there are any non- + string values in *iterable*, including "bytes" objects. The + separator between elements is the string providing this method. For + example: + + >>> ', '.join(['spam', 'spam', 'spam']) + 'spam, spam, spam' + >>> '-'.join('Python') + 'P-y-t-h-o-n' + + See also "split()". + +str.ljust(width, fillchar=' ', /) + + Return the string left justified in a string of length *width*. + Padding is done using the specified *fillchar* (default is an ASCII + space). The original string is returned if *width* is less than or + equal to "len(s)". + +str.lower() + + Return a copy of the string with all the cased characters [4] + converted to lowercase. + + The lowercasing algorithm used is described in section 3.13 + ‘Default Case Folding’ of the Unicode Standard. + +str.lstrip(chars=None, /) + + Return a copy of the string with leading characters removed. The + *chars* argument is a string specifying the set of characters to be + removed. If omitted or "None", the *chars* argument defaults to + removing whitespace. The *chars* argument is not a prefix; rather, + all combinations of its values are stripped: + + >>> ' spacious '.lstrip() + 'spacious ' + >>> 'www.example.com'.lstrip('cmowz.') + 'example.com' + + See "str.removeprefix()" for a method that will remove a single + prefix string rather than all of a set of characters. For example: + + >>> 'Arthur: three!'.lstrip('Arthur: ') + 'ee!' + >>> 'Arthur: three!'.removeprefix('Arthur: ') + 'three!' + +static str.maketrans(dict, /) +static str.maketrans(from, to, remove='', /) + + This static method returns a translation table usable for + "str.translate()". + + If there is only one argument, it must be a dictionary mapping + Unicode ordinals (integers) or characters (strings of length 1) to + Unicode ordinals, strings (of arbitrary lengths) or "None". + Character keys will then be converted to ordinals. + + If there are two arguments, they must be strings of equal length, + and in the resulting dictionary, each character in *from* will be + mapped to the character at the same position in *to*. If there is + a third argument, it must be a string, whose characters will be + mapped to "None" in the result. + +str.partition(sep, /) + + Split the string at the first occurrence of *sep*, and return a + 3-tuple containing the part before the separator, the separator + itself, and the part after the separator. If the separator is not + found, return a 3-tuple containing the string itself, followed by + two empty strings. + +str.removeprefix(prefix, /) + + If the string starts with the *prefix* string, return + "string[len(prefix):]". Otherwise, return a copy of the original + string: + + >>> 'TestHook'.removeprefix('Test') + 'Hook' + >>> 'BaseTestCase'.removeprefix('Test') + 'BaseTestCase' + + Added in version 3.9. + +str.removesuffix(suffix, /) + + If the string ends with the *suffix* string and that *suffix* is + not empty, return "string[:-len(suffix)]". Otherwise, return a copy + of the original string: + + >>> 'MiscTests'.removesuffix('Tests') + 'Misc' + >>> 'TmpDirMixin'.removesuffix('Tests') + 'TmpDirMixin' + + Added in version 3.9. + +str.replace(old, new, /, count=-1) + + Return a copy of the string with all occurrences of substring *old* + replaced by *new*. If *count* is given, only the first *count* + occurrences are replaced. If *count* is not specified or "-1", then + all occurrences are replaced. + + Changed in version 3.13: *count* is now supported as a keyword + argument. + +str.rfind(sub[, start[, end]]) + + Return the highest index in the string where substring *sub* is + found, such that *sub* is contained within "s[start:end]". + Optional arguments *start* and *end* are interpreted as in slice + notation. Return "-1" on failure. + +str.rindex(sub[, start[, end]]) + + Like "rfind()" but raises "ValueError" when the substring *sub* is + not found. + +str.rjust(width, fillchar=' ', /) + + Return the string right justified in a string of length *width*. + Padding is done using the specified *fillchar* (default is an ASCII + space). The original string is returned if *width* is less than or + equal to "len(s)". + +str.rpartition(sep, /) + + Split the string at the last occurrence of *sep*, and return a + 3-tuple containing the part before the separator, the separator + itself, and the part after the separator. If the separator is not + found, return a 3-tuple containing two empty strings, followed by + the string itself. + +str.rsplit(sep=None, maxsplit=-1) + + Return a list of the words in the string, using *sep* as the + delimiter string. If *maxsplit* is given, at most *maxsplit* splits + are done, the *rightmost* ones. If *sep* is not specified or + "None", any whitespace string is a separator. Except for splitting + from the right, "rsplit()" behaves like "split()" which is + described in detail below. + +str.rstrip(chars=None, /) + + Return a copy of the string with trailing characters removed. The + *chars* argument is a string specifying the set of characters to be + removed. If omitted or "None", the *chars* argument defaults to + removing whitespace. The *chars* argument is not a suffix; rather, + all combinations of its values are stripped: + + >>> ' spacious '.rstrip() + ' spacious' + >>> 'mississippi'.rstrip('ipz') + 'mississ' + + See "str.removesuffix()" for a method that will remove a single + suffix string rather than all of a set of characters. For example: + + >>> 'Monty Python'.rstrip(' Python') + 'M' + >>> 'Monty Python'.removesuffix(' Python') + 'Monty' + +str.split(sep=None, maxsplit=-1) + + Return a list of the words in the string, using *sep* as the + delimiter string. If *maxsplit* is given, at most *maxsplit* + splits are done (thus, the list will have at most "maxsplit+1" + elements). If *maxsplit* is not specified or "-1", then there is + no limit on the number of splits (all possible splits are made). + + If *sep* is given, consecutive delimiters are not grouped together + and are deemed to delimit empty strings (for example, + "'1,,2'.split(',')" returns "['1', '', '2']"). The *sep* argument + may consist of multiple characters as a single delimiter (to split + with multiple delimiters, use "re.split()"). Splitting an empty + string with a specified separator returns "['']". + + For example: + + >>> '1,2,3'.split(',') + ['1', '2', '3'] + >>> '1,2,3'.split(',', maxsplit=1) + ['1', '2,3'] + >>> '1,2,,3,'.split(',') + ['1', '2', '', '3', ''] + >>> '1<>2<>3<4'.split('<>') + ['1', '2', '3<4'] + + If *sep* is not specified or is "None", a different splitting + algorithm is applied: runs of consecutive whitespace are regarded + as a single separator, and the result will contain no empty strings + at the start or end if the string has leading or trailing + whitespace. Consequently, splitting an empty string or a string + consisting of just whitespace with a "None" separator returns "[]". + + For example: + + >>> '1 2 3'.split() + ['1', '2', '3'] + >>> '1 2 3'.split(maxsplit=1) + ['1', '2 3'] + >>> ' 1 2 3 '.split() + ['1', '2', '3'] + + If *sep* is not specified or is "None" and *maxsplit* is "0", only + leading runs of consecutive whitespace are considered. + + For example: + + >>> "".split(None, 0) + [] + >>> " ".split(None, 0) + [] + >>> " foo ".split(maxsplit=0) + ['foo '] + + See also "join()". + +str.splitlines(keepends=False) + + Return a list of the lines in the string, breaking at line + boundaries. Line breaks are not included in the resulting list + unless *keepends* is given and true. + + This method splits on the following line boundaries. In + particular, the boundaries are a superset of *universal newlines*. + + +-------------------------+-------------------------------+ + | Representation | Description | + |=========================|===============================| + | "\n" | Line Feed | + +-------------------------+-------------------------------+ + | "\r" | Carriage Return | + +-------------------------+-------------------------------+ + | "\r\n" | Carriage Return + Line Feed | + +-------------------------+-------------------------------+ + | "\v" or "\x0b" | Line Tabulation | + +-------------------------+-------------------------------+ + | "\f" or "\x0c" | Form Feed | + +-------------------------+-------------------------------+ + | "\x1c" | File Separator | + +-------------------------+-------------------------------+ + | "\x1d" | Group Separator | + +-------------------------+-------------------------------+ + | "\x1e" | Record Separator | + +-------------------------+-------------------------------+ + | "\x85" | Next Line (C1 Control Code) | + +-------------------------+-------------------------------+ + | "\u2028" | Line Separator | + +-------------------------+-------------------------------+ + | "\u2029" | Paragraph Separator | + +-------------------------+-------------------------------+ + + Changed in version 3.2: "\v" and "\f" added to list of line + boundaries. + + For example: + + >>> 'ab c\n\nde fg\rkl\r\n'.splitlines() + ['ab c', '', 'de fg', 'kl'] + >>> 'ab c\n\nde fg\rkl\r\n'.splitlines(keepends=True) + ['ab c\n', '\n', 'de fg\r', 'kl\r\n'] + + Unlike "split()" when a delimiter string *sep* is given, this + method returns an empty list for the empty string, and a terminal + line break does not result in an extra line: + + >>> "".splitlines() + [] + >>> "One line\n".splitlines() + ['One line'] + + For comparison, "split('\n')" gives: + + >>> ''.split('\n') + [''] + >>> 'Two lines\n'.split('\n') + ['Two lines', ''] + +str.startswith(prefix[, start[, end]]) + + Return "True" if string starts with the *prefix*, otherwise return + "False". *prefix* can also be a tuple of prefixes to look for. + With optional *start*, test string beginning at that position. + With optional *end*, stop comparing string at that position. + +str.strip(chars=None, /) + + Return a copy of the string with the leading and trailing + characters removed. The *chars* argument is a string specifying the + set of characters to be removed. If omitted or "None", the *chars* + argument defaults to removing whitespace. The *chars* argument is + not a prefix or suffix; rather, all combinations of its values are + stripped: + + >>> ' spacious '.strip() + 'spacious' + >>> 'www.example.com'.strip('cmowz.') + 'example' + + The outermost leading and trailing *chars* argument values are + stripped from the string. Characters are removed from the leading + end until reaching a string character that is not contained in the + set of characters in *chars*. A similar action takes place on the + trailing end. For example: + + >>> comment_string = '#....... Section 3.2.1 Issue #32 .......' + >>> comment_string.strip('.#! ') + 'Section 3.2.1 Issue #32' + +str.swapcase() + + Return a copy of the string with uppercase characters converted to + lowercase and vice versa. Note that it is not necessarily true that + "s.swapcase().swapcase() == s". + +str.title() + + Return a titlecased version of the string where words start with an + uppercase character and the remaining characters are lowercase. + + For example: + + >>> 'Hello world'.title() + 'Hello World' + + The algorithm uses a simple language-independent definition of a + word as groups of consecutive letters. The definition works in + many contexts but it means that apostrophes in contractions and + possessives form word boundaries, which may not be the desired + result: + + >>> "they're bill's friends from the UK".title() + "They'Re Bill'S Friends From The Uk" + + The "string.capwords()" function does not have this problem, as it + splits words on spaces only. + + Alternatively, a workaround for apostrophes can be constructed + using regular expressions: + + >>> import re + >>> def titlecase(s): + ... return re.sub(r"[A-Za-z]+('[A-Za-z]+)?", + ... lambda mo: mo.group(0).capitalize(), + ... s) + ... + >>> titlecase("they're bill's friends.") + "They're Bill's Friends." + + See also "istitle()". + +str.translate(table, /) + + Return a copy of the string in which each character has been mapped + through the given translation table. The table must be an object + that implements indexing via "__getitem__()", typically a *mapping* + or *sequence*. When indexed by a Unicode ordinal (an integer), the + table object can do any of the following: return a Unicode ordinal + or a string, to map the character to one or more other characters; + return "None", to delete the character from the return string; or + raise a "LookupError" exception, to map the character to itself. + + You can use "str.maketrans()" to create a translation map from + character-to-character mappings in different formats. + + See also the "codecs" module for a more flexible approach to custom + character mappings. + +str.upper() + + Return a copy of the string with all the cased characters [4] + converted to uppercase. Note that "s.upper().isupper()" might be + "False" if "s" contains uncased characters or if the Unicode + category of the resulting character(s) is not “Lu” (Letter, + uppercase), but e.g. “Lt” (Letter, titlecase). + + The uppercasing algorithm used is described in section 3.13 + ‘Default Case Folding’ of the Unicode Standard. + +str.zfill(width, /) + + Return a copy of the string left filled with ASCII "'0'" digits to + make a string of length *width*. A leading sign prefix + ("'+'"/"'-'") is handled by inserting the padding *after* the sign + character rather than before. The original string is returned if + *width* is less than or equal to "len(s)". + + For example: + + >>> "42".zfill(5) + '00042' + >>> "-42".zfill(5) + '-0042' +''', + 'strings': '''String and Bytes literals +************************* + +String literals are text enclosed in single quotes ("'") or double +quotes ("""). For example: + + "spam" + 'eggs' + +The quote used to start the literal also terminates it, so a string +literal can only contain the other quote (except with escape +sequences, see below). For example: + + 'Say "Hello", please.' + "Don't do that!" + +Except for this limitation, the choice of quote character ("'" or """) +does not affect how the literal is parsed. + +Inside a string literal, the backslash ("\\") character introduces an +*escape sequence*, which has special meaning depending on the +character after the backslash. For example, "\\"" denotes the double +quote character, and does *not* end the string: + + >>> print("Say \\"Hello\\" to everyone!") + Say "Hello" to everyone! + +See escape sequences below for a full list of such sequences, and more +details. + + +Triple-quoted strings +===================== + +Strings can also be enclosed in matching groups of three single or +double quotes. These are generally referred to as *triple-quoted +strings*: + + """This is a triple-quoted string.""" + +In triple-quoted literals, unescaped quotes are allowed (and are +retained), except that three unescaped quotes in a row terminate the +literal, if they are of the same kind ("'" or """) used at the start: + + """This string has "quotes" inside.""" + +Unescaped newlines are also allowed and retained: + + \'\'\'This triple-quoted string + continues on the next line.\'\'\' + + +String prefixes +=============== + +String literals can have an optional *prefix* that influences how the +content of the literal is parsed, for example: + + b"data" + f'{result=}' + +The allowed prefixes are: + +* "b": Bytes literal + +* "r": Raw string + +* "f": Formatted string literal (“f-string”) + +* "t": Template string literal (“t-string”) + +* "u": No effect (allowed for backwards compatibility) + +See the linked sections for details on each type. + +Prefixes are case-insensitive (for example, ‘"B"’ works the same as +‘"b"’). The ‘"r"’ prefix can be combined with ‘"f"’, ‘"t"’ or ‘"b"’, +so ‘"fr"’, ‘"rf"’, ‘"tr"’, ‘"rt"’, ‘"br"’, and ‘"rb"’ are also valid +prefixes. + +Added in version 3.3: The "'rb'" prefix of raw bytes literals has been +added as a synonym of "'br'".Support for the unicode legacy literal +("u'value'") was reintroduced to simplify the maintenance of dual +Python 2.x and 3.x codebases. See **PEP 414** for more information. + + +Formal grammar +============== + +String literals, except “f-strings” and “t-strings”, are described by +the following lexical definitions. + +These definitions use negative lookaheads ("!") to indicate that an +ending quote ends the literal. + + STRING: [stringprefix] (stringcontent) + stringprefix: <("r" | "u" | "b" | "br" | "rb"), case-insensitive> + stringcontent: + | "\'\'\'" ( !"\'\'\'" longstringitem)* "\'\'\'" + | '"""' ( !'"""' longstringitem)* '"""' + | "'" ( !"'" stringitem)* "'" + | '"' ( !'"' stringitem)* '"' + stringitem: stringchar | stringescapeseq + stringchar: + longstringitem: stringitem | newline + stringescapeseq: "\\" + +Note that as in all lexical definitions, whitespace is significant. In +particular, the prefix (if any) must be immediately followed by the +starting quote. + + +Escape sequences +================ + +Unless an ‘"r"’ or ‘"R"’ prefix is present, escape sequences in string +and bytes literals are interpreted according to rules similar to those +used by Standard C. The recognized escape sequences are: + ++----------------------------------------------------+----------------------------------------------------+ +| Escape Sequence | Meaning | +|====================================================|====================================================| +| "\\" | Ignored end of line | ++----------------------------------------------------+----------------------------------------------------+ +| "\\\\" | Backslash | ++----------------------------------------------------+----------------------------------------------------+ +| "\\'" | Single quote | ++----------------------------------------------------+----------------------------------------------------+ +| "\\"" | Double quote | ++----------------------------------------------------+----------------------------------------------------+ +| "\\a" | ASCII Bell (BEL) | ++----------------------------------------------------+----------------------------------------------------+ +| "\\b" | ASCII Backspace (BS) | ++----------------------------------------------------+----------------------------------------------------+ +| "\\f" | ASCII Formfeed (FF) | ++----------------------------------------------------+----------------------------------------------------+ +| "\\n" | ASCII Linefeed (LF) | ++----------------------------------------------------+----------------------------------------------------+ +| "\\r" | ASCII Carriage Return (CR) | ++----------------------------------------------------+----------------------------------------------------+ +| "\\t" | ASCII Horizontal Tab (TAB) | ++----------------------------------------------------+----------------------------------------------------+ +| "\\v" | ASCII Vertical Tab (VT) | ++----------------------------------------------------+----------------------------------------------------+ +| "\\*ooo*" | Octal character | ++----------------------------------------------------+----------------------------------------------------+ +| "\\x*hh*" | Hexadecimal character | ++----------------------------------------------------+----------------------------------------------------+ +| "\\N{*name*}" | Named Unicode character | ++----------------------------------------------------+----------------------------------------------------+ +| "\\u*xxxx*" | Hexadecimal Unicode character | ++----------------------------------------------------+----------------------------------------------------+ +| "\\U*xxxxxxxx*" | Hexadecimal Unicode character | ++----------------------------------------------------+----------------------------------------------------+ + + +Ignored end of line +------------------- + +A backslash can be added at the end of a line to ignore the newline: + + >>> 'This string will not include \\ + ... backslashes or newline characters.' + 'This string will not include backslashes or newline characters.' + +The same result can be achieved using triple-quoted strings, or +parentheses and string literal concatenation. + + +Escaped characters +------------------ + +To include a backslash in a non-raw Python string literal, it must be +doubled. The "\\\\" escape sequence denotes a single backslash +character: + + >>> print('C:\\\\Program Files') + C:\\Program Files + +Similarly, the "\\'" and "\\"" sequences denote the single and double +quote character, respectively: + + >>> print('\\' and \\"') + ' and " + + +Octal character +--------------- + +The sequence "\\*ooo*" denotes a *character* with the octal (base 8) +value *ooo*: + + >>> '\\120' + 'P' + +Up to three octal digits (0 through 7) are accepted. + +In a bytes literal, *character* means a *byte* with the given value. +In a string literal, it means a Unicode character with the given +value. + +Changed in version 3.11: Octal escapes with value larger than "0o377" +(255) produce a "DeprecationWarning". + +Changed in version 3.12: Octal escapes with value larger than "0o377" +(255) produce a "SyntaxWarning". In a future Python version they will +raise a "SyntaxError". + + +Hexadecimal character +--------------------- + +The sequence "\\x*hh*" denotes a *character* with the hex (base 16) +value *hh*: + + >>> '\\x50' + 'P' + +Unlike in Standard C, exactly two hex digits are required. + +In a bytes literal, *character* means a *byte* with the given value. +In a string literal, it means a Unicode character with the given +value. + + +Named Unicode character +----------------------- + +The sequence "\\N{*name*}" denotes a Unicode character with the given +*name*: + + >>> '\\N{LATIN CAPITAL LETTER P}' + 'P' + >>> '\\N{SNAKE}' + '🐍' + +This sequence cannot appear in bytes literals. + +Changed in version 3.3: Support for name aliases has been added. + + +Hexadecimal Unicode characters +------------------------------ + +These sequences "\\u*xxxx*" and "\\U*xxxxxxxx*" denote the Unicode +character with the given hex (base 16) value. Exactly four digits are +required for "\\u"; exactly eight digits are required for "\\U". The +latter can encode any Unicode character. + + >>> '\\u1234' + 'ሴ' + >>> '\\U0001f40d' + '🐍' + +These sequences cannot appear in bytes literals. + + +Unrecognized escape sequences +----------------------------- + +Unlike in Standard C, all unrecognized escape sequences are left in +the string unchanged, that is, *the backslash is left in the result*: + + >>> print('\\q') + \\q + >>> list('\\q') + ['\\\\', 'q'] + +Note that for bytes literals, the escape sequences only recognized in +string literals ("\\N...", "\\u...", "\\U...") fall into the category of +unrecognized escapes. + +Changed in version 3.6: Unrecognized escape sequences produce a +"DeprecationWarning". + +Changed in version 3.12: Unrecognized escape sequences produce a +"SyntaxWarning". In a future Python version they will raise a +"SyntaxError". + + +Bytes literals +============== + +*Bytes literals* are always prefixed with ‘"b"’ or ‘"B"’; they produce +an instance of the "bytes" type instead of the "str" type. They may +only contain ASCII characters; bytes with a numeric value of 128 or +greater must be expressed with escape sequences (typically Hexadecimal +character or Octal character): + + >>> b'\\x89PNG\\r\\n\\x1a\\n' + b'\\x89PNG\\r\\n\\x1a\\n' + >>> list(b'\\x89PNG\\r\\n\\x1a\\n') + [137, 80, 78, 71, 13, 10, 26, 10] + +Similarly, a zero byte must be expressed using an escape sequence +(typically "\\0" or "\\x00"). + + +Raw string literals +=================== + +Both string and bytes literals may optionally be prefixed with a +letter ‘"r"’ or ‘"R"’; such constructs are called *raw string +literals* and *raw bytes literals* respectively and treat backslashes +as literal characters. As a result, in raw string literals, escape +sequences are not treated specially: + + >>> r'\\d{4}-\\d{2}-\\d{2}' + '\\\\d{4}-\\\\d{2}-\\\\d{2}' + +Even in a raw literal, quotes can be escaped with a backslash, but the +backslash remains in the result; for example, "r"\\""" is a valid +string literal consisting of two characters: a backslash and a double +quote; "r"\\"" is not a valid string literal (even a raw string cannot +end in an odd number of backslashes). Specifically, *a raw literal +cannot end in a single backslash* (since the backslash would escape +the following quote character). Note also that a single backslash +followed by a newline is interpreted as those two characters as part +of the literal, *not* as a line continuation. + + +f-strings +========= + +Added in version 3.6. + +Changed in version 3.7: The "await" and "async for" can be used in +expressions within f-strings. + +Changed in version 3.8: Added the debug specifier ("=") + +Changed in version 3.12: Many restrictions on expressions within +f-strings have been removed. Notably, nested strings, comments, and +backslashes are now permitted. + +A *formatted string literal* or *f-string* is a string literal that is +prefixed with ‘"f"’ or ‘"F"’. Unlike other string literals, f-strings +do not have a constant value. They may contain *replacement fields* +delimited by curly braces "{}". Replacement fields contain expressions +which are evaluated at run time. For example: + + >>> who = 'nobody' + >>> nationality = 'Spanish' + >>> f'{who.title()} expects the {nationality} Inquisition!' + 'Nobody expects the Spanish Inquisition!' + +Any doubled curly braces ("{{" or "}}") outside replacement fields are +replaced with the corresponding single curly brace: + + >>> print(f'{{...}}') + {...} + +Other characters outside replacement fields are treated like in +ordinary string literals. This means that escape sequences are decoded +(except when a literal is also marked as a raw string), and newlines +are possible in triple-quoted f-strings: + + >>> name = 'Galahad' + >>> favorite_color = 'blue' + >>> print(f'{name}:\\t{favorite_color}') + Galahad: blue + >>> print(rf"C:\\Users\\{name}") + C:\\Users\\Galahad + >>> print(f\'\'\'Three shall be the number of the counting + ... and the number of the counting shall be three.\'\'\') + Three shall be the number of the counting + and the number of the counting shall be three. + +Expressions in formatted string literals are treated like regular +Python expressions. Each expression is evaluated in the context where +the formatted string literal appears, in order from left to right. An +empty expression is not allowed, and both "lambda" and assignment +expressions ":=" must be surrounded by explicit parentheses: + + >>> f'{(half := 1/2)}, {half * 42}' + '0.5, 21.0' + +Reusing the outer f-string quoting type inside a replacement field is +permitted: + + >>> a = dict(x=2) + >>> f"abc {a["x"]} def" + 'abc 2 def' + +Backslashes are also allowed in replacement fields and are evaluated +the same way as in any other context: + + >>> a = ["a", "b", "c"] + >>> print(f"List a contains:\\n{"\\n".join(a)}") + List a contains: + a + b + c + +It is possible to nest f-strings: + + >>> name = 'world' + >>> f'Repeated:{f' hello {name}' * 3}' + 'Repeated: hello world hello world hello world' + +Portable Python programs should not use more than 5 levels of nesting. + +**CPython implementation detail:** CPython does not limit nesting of +f-strings. + +Replacement expressions can contain newlines in both single-quoted and +triple-quoted f-strings and they can contain comments. Everything that +comes after a "#" inside a replacement field is a comment (even +closing braces and quotes). This means that replacement fields with +comments must be closed in a different line: + + >>> a = 2 + >>> f"abc{a # This comment }" continues until the end of the line + ... + 3}" + 'abc5' + +After the expression, replacement fields may optionally contain: + +* a *debug specifier* – an equal sign ("="), optionally surrounded by + whitespace on one or both sides; + +* a *conversion specifier* – "!s", "!r" or "!a"; and/or + +* a *format specifier* prefixed with a colon (":"). + +See the Standard Library section on f-strings for details on how these +fields are evaluated. + +As that section explains, *format specifiers* are passed as the second +argument to the "format()" function to format a replacement field +value. For example, they can be used to specify a field width and +padding characters using the Format Specification Mini-Language: + + >>> number = 14.3 + >>> f'{number:20.7f}' + ' 14.3000000' + +Top-level format specifiers may include nested replacement fields: + + >>> field_size = 20 + >>> precision = 7 + >>> f'{number:{field_size}.{precision}f}' + ' 14.3000000' + +These nested fields may include their own conversion fields and format +specifiers: + + >>> number = 3 + >>> f'{number:{field_size}}' + ' 3' + >>> f'{number:{field_size:05}}' + '00000000000000000003' + +However, these nested fields may not include more deeply nested +replacement fields. + +Formatted string literals cannot be used as *docstrings*, even if they +do not include expressions: + + >>> def foo(): + ... f"Not a docstring" + ... + >>> print(foo.__doc__) + None + +See also: + + * **PEP 498** – Literal String Interpolation + + * **PEP 701** – Syntactic formalization of f-strings + + * "str.format()", which uses a related format string mechanism. + + +t-strings +========= + +Added in version 3.14. + +A *template string literal* or *t-string* is a string literal that is +prefixed with ‘"t"’ or ‘"T"’. These strings follow the same syntax +rules as formatted string literals. For differences in evaluation +rules, see the Standard Library section on t-strings + + +Formal grammar for f-strings +============================ + +F-strings are handled partly by the *lexical analyzer*, which produces +the tokens "FSTRING_START", "FSTRING_MIDDLE" and "FSTRING_END", and +partly by the parser, which handles expressions in the replacement +field. The exact way the work is split is a CPython implementation +detail. + +Correspondingly, the f-string grammar is a mix of lexical and +syntactic definitions. + +Whitespace is significant in these situations: + +* There may be no whitespace in "FSTRING_START" (between the prefix + and quote). + +* Whitespace in "FSTRING_MIDDLE" is part of the literal string + contents. + +* In "fstring_replacement_field", if "f_debug_specifier" is present, + all whitespace after the opening brace until the + "f_debug_specifier", as well as whitespace immediatelly following + "f_debug_specifier", is retained as part of the expression. + + **CPython implementation detail:** The expression is not handled in + the tokenization phase; it is retrieved from the source code using + locations of the "{" token and the token after "=". + +The "FSTRING_MIDDLE" definition uses negative lookaheads ("!") to +indicate special characters (backslash, newline, "{", "}") and +sequences ("f_quote"). + + fstring: FSTRING_START fstring_middle* FSTRING_END + + FSTRING_START: fstringprefix ("'" | '"' | "\'\'\'" | '"""') + FSTRING_END: f_quote + fstringprefix: <("f" | "fr" | "rf"), case-insensitive> + f_debug_specifier: '=' + f_quote: + + fstring_middle: + | fstring_replacement_field + | FSTRING_MIDDLE + FSTRING_MIDDLE: + | (!"\\" !newline !'{' !'}' !f_quote) source_character + | stringescapeseq + | "{{" + | "}}" + | + fstring_replacement_field: + | '{' f_expression [f_debug_specifier] [fstring_conversion] + [fstring_full_format_spec] '}' + fstring_conversion: + | "!" ("s" | "r" | "a") + fstring_full_format_spec: + | ':' fstring_format_spec* + fstring_format_spec: + | FSTRING_MIDDLE + | fstring_replacement_field + f_expression: + | ','.(conditional_expression | "*" or_expr)+ [","] + | yield_expression + +Note: + + In the above grammar snippet, the "f_quote" and "FSTRING_MIDDLE" + rules are context-sensitive – they depend on the contents of + "FSTRING_START" of the nearest enclosing "fstring".Constructing a + more traditional formal grammar from this template is left as an + exercise for the reader. + +The grammar for t-strings is identical to the one for f-strings, with +*t* instead of *f* at the beginning of rule and token names and in the +prefix. + + tstring: TSTRING_START tstring_middle* TSTRING_END + + +''', + 'subscriptions': r'''Subscriptions +************* + +The subscription of an instance of a container class will generally +select an element from the container. The subscription of a *generic +class* will generally return a GenericAlias object. + + subscription: primary "[" flexible_expression_list "]" + +When an object is subscripted, the interpreter will evaluate the +primary and the expression list. + +The primary must evaluate to an object that supports subscription. An +object may support subscription through defining one or both of +"__getitem__()" and "__class_getitem__()". When the primary is +subscripted, the evaluated result of the expression list will be +passed to one of these methods. For more details on when +"__class_getitem__" is called instead of "__getitem__", see +__class_getitem__ versus __getitem__. + +If the expression list contains at least one comma, or if any of the +expressions are starred, the expression list will evaluate to a +"tuple" containing the items of the expression list. Otherwise, the +expression list will evaluate to the value of the list’s sole member. + +Changed in version 3.11: Expressions in an expression list may be +starred. See **PEP 646**. + +For built-in objects, there are two types of objects that support +subscription via "__getitem__()": + +1. Mappings. If the primary is a *mapping*, the expression list must + evaluate to an object whose value is one of the keys of the + mapping, and the subscription selects the value in the mapping that + corresponds to that key. An example of a builtin mapping class is + the "dict" class. + +2. Sequences. If the primary is a *sequence*, the expression list must + evaluate to an "int" or a "slice" (as discussed in the following + section). Examples of builtin sequence classes include the "str", + "list" and "tuple" classes. + +The formal syntax makes no special provision for negative indices in +*sequences*. However, built-in sequences all provide a "__getitem__()" +method that interprets negative indices by adding the length of the +sequence to the index so that, for example, "x[-1]" selects the last +item of "x". The resulting value must be a nonnegative integer less +than the number of items in the sequence, and the subscription selects +the item whose index is that value (counting from zero). Since the +support for negative indices and slicing occurs in the object’s +"__getitem__()" method, subclasses overriding this method will need to +explicitly add that support. + +A "string" is a special kind of sequence whose items are *characters*. +A character is not a separate data type but a string of exactly one +character. +''', + 'truth': r'''Truth Value Testing +******************* + +Any object can be tested for truth value, for use in an "if" or +"while" condition or as operand of the Boolean operations below. + +By default, an object is considered true unless its class defines +either a "__bool__()" method that returns "False" or a "__len__()" +method that returns zero, when called with the object. [1] Here are +most of the built-in objects considered false: + +* constants defined to be false: "None" and "False" + +* zero of any numeric type: "0", "0.0", "0j", "Decimal(0)", + "Fraction(0, 1)" + +* empty sequences and collections: "''", "()", "[]", "{}", "set()", + "range(0)" + +Operations and built-in functions that have a Boolean result always +return "0" or "False" for false and "1" or "True" for true, unless +otherwise stated. (Important exception: the Boolean operations "or" +and "and" always return one of their operands.) +''', + 'try': r'''The "try" statement +******************* + +The "try" statement specifies exception handlers and/or cleanup code +for a group of statements: + + try_stmt: try1_stmt | try2_stmt | try3_stmt + try1_stmt: "try" ":" suite + ("except" [expression ["as" identifier]] ":" suite)+ + ["else" ":" suite] + ["finally" ":" suite] + try2_stmt: "try" ":" suite + ("except" "*" expression ["as" identifier] ":" suite)+ + ["else" ":" suite] + ["finally" ":" suite] + try3_stmt: "try" ":" suite + "finally" ":" suite + +Additional information on exceptions can be found in section +Exceptions, and information on using the "raise" statement to generate +exceptions may be found in section The raise statement. + +Changed in version 3.14: Support for optionally dropping grouping +parentheses when using multiple exception types. See **PEP 758**. + + +"except" clause +=============== + +The "except" clause(s) specify one or more exception handlers. When no +exception occurs in the "try" clause, no exception handler is +executed. When an exception occurs in the "try" suite, a search for an +exception handler is started. This search inspects the "except" +clauses in turn until one is found that matches the exception. An +expression-less "except" clause, if present, must be last; it matches +any exception. + +For an "except" clause with an expression, the expression must +evaluate to an exception type or a tuple of exception types. +Parentheses can be dropped if multiple exception types are provided +and the "as" clause is not used. The raised exception matches an +"except" clause whose expression evaluates to the class or a *non- +virtual base class* of the exception object, or to a tuple that +contains such a class. + +If no "except" clause matches the exception, the search for an +exception handler continues in the surrounding code and on the +invocation stack. [1] + +If the evaluation of an expression in the header of an "except" clause +raises an exception, the original search for a handler is canceled and +a search starts for the new exception in the surrounding code and on +the call stack (it is treated as if the entire "try" statement raised +the exception). + +When a matching "except" clause is found, the exception is assigned to +the target specified after the "as" keyword in that "except" clause, +if present, and the "except" clause’s suite is executed. All "except" +clauses must have an executable block. When the end of this block is +reached, execution continues normally after the entire "try" +statement. (This means that if two nested handlers exist for the same +exception, and the exception occurs in the "try" clause of the inner +handler, the outer handler will not handle the exception.) + +When an exception has been assigned using "as target", it is cleared +at the end of the "except" clause. This is as if + + except E as N: + foo + +was translated to + + except E as N: + try: + foo + finally: + del N + +This means the exception must be assigned to a different name to be +able to refer to it after the "except" clause. Exceptions are cleared +because with the traceback attached to them, they form a reference +cycle with the stack frame, keeping all locals in that frame alive +until the next garbage collection occurs. + +Before an "except" clause’s suite is executed, the exception is stored +in the "sys" module, where it can be accessed from within the body of +the "except" clause by calling "sys.exception()". When leaving an +exception handler, the exception stored in the "sys" module is reset +to its previous value: + + >>> print(sys.exception()) + None + >>> try: + ... raise TypeError + ... except: + ... print(repr(sys.exception())) + ... try: + ... raise ValueError + ... except: + ... print(repr(sys.exception())) + ... print(repr(sys.exception())) + ... + TypeError() + ValueError() + TypeError() + >>> print(sys.exception()) + None + + +"except*" clause +================ + +The "except*" clause(s) specify one or more handlers for groups of +exceptions ("BaseExceptionGroup" instances). A "try" statement can +have either "except" or "except*" clauses, but not both. The exception +type for matching is mandatory in the case of "except*", so "except*:" +is a syntax error. The type is interpreted as in the case of "except", +but matching is performed on the exceptions contained in the group +that is being handled. An "TypeError" is raised if a matching type is +a subclass of "BaseExceptionGroup", because that would have ambiguous +semantics. + +When an exception group is raised in the try block, each "except*" +clause splits (see "split()") it into the subgroups of matching and +non-matching exceptions. If the matching subgroup is not empty, it +becomes the handled exception (the value returned from +"sys.exception()") and assigned to the target of the "except*" clause +(if there is one). Then, the body of the "except*" clause executes. If +the non-matching subgroup is not empty, it is processed by the next +"except*" in the same manner. This continues until all exceptions in +the group have been matched, or the last "except*" clause has run. + +After all "except*" clauses execute, the group of unhandled exceptions +is merged with any exceptions that were raised or re-raised from +within "except*" clauses. This merged exception group propagates on.: + + >>> try: + ... raise ExceptionGroup("eg", + ... [ValueError(1), TypeError(2), OSError(3), OSError(4)]) + ... except* TypeError as e: + ... print(f'caught {type(e)} with nested {e.exceptions}') + ... except* OSError as e: + ... print(f'caught {type(e)} with nested {e.exceptions}') + ... + caught with nested (TypeError(2),) + caught with nested (OSError(3), OSError(4)) + + Exception Group Traceback (most recent call last): + | File "", line 2, in + | raise ExceptionGroup("eg", + | [ValueError(1), TypeError(2), OSError(3), OSError(4)]) + | ExceptionGroup: eg (1 sub-exception) + +-+---------------- 1 ---------------- + | ValueError: 1 + +------------------------------------ + +If the exception raised from the "try" block is not an exception group +and its type matches one of the "except*" clauses, it is caught and +wrapped by an exception group with an empty message string. This +ensures that the type of the target "e" is consistently +"BaseExceptionGroup": + + >>> try: + ... raise BlockingIOError + ... except* BlockingIOError as e: + ... print(repr(e)) + ... + ExceptionGroup('', (BlockingIOError())) + +"break", "continue" and "return" cannot appear in an "except*" clause. + + +"else" clause +============= + +The optional "else" clause is executed if the control flow leaves the +"try" suite, no exception was raised, and no "return", "continue", or +"break" statement was executed. Exceptions in the "else" clause are +not handled by the preceding "except" clauses. + + +"finally" clause +================ + +If "finally" is present, it specifies a ‘cleanup’ handler. The "try" +clause is executed, including any "except" and "else" clauses. If an +exception occurs in any of the clauses and is not handled, the +exception is temporarily saved. The "finally" clause is executed. If +there is a saved exception it is re-raised at the end of the "finally" +clause. If the "finally" clause raises another exception, the saved +exception is set as the context of the new exception. If the "finally" +clause executes a "return", "break" or "continue" statement, the saved +exception is discarded. For example, this function returns 42. + + def f(): + try: + 1/0 + finally: + return 42 + +The exception information is not available to the program during +execution of the "finally" clause. + +When a "return", "break" or "continue" statement is executed in the +"try" suite of a "try"…"finally" statement, the "finally" clause is +also executed ‘on the way out.’ + +The return value of a function is determined by the last "return" +statement executed. Since the "finally" clause always executes, a +"return" statement executed in the "finally" clause will always be the +last one executed. The following function returns ‘finally’. + + def foo(): + try: + return 'try' + finally: + return 'finally' + +Changed in version 3.8: Prior to Python 3.8, a "continue" statement +was illegal in the "finally" clause due to a problem with the +implementation. + +Changed in version 3.14: The compiler emits a "SyntaxWarning" when a +"return", "break" or "continue" appears in a "finally" block (see +**PEP 765**). +''', + 'types': r'''The standard type hierarchy +*************************** + +Below is a list of the types that are built into Python. Extension +modules (written in C, Java, or other languages, depending on the +implementation) can define additional types. Future versions of +Python may add types to the type hierarchy (e.g., rational numbers, +efficiently stored arrays of integers, etc.), although such additions +will often be provided via the standard library instead. + +Some of the type descriptions below contain a paragraph listing +‘special attributes.’ These are attributes that provide access to the +implementation and are not intended for general use. Their definition +may change in the future. + + +None +==== + +This type has a single value. There is a single object with this +value. This object is accessed through the built-in name "None". It is +used to signify the absence of a value in many situations, e.g., it is +returned from functions that don’t explicitly return anything. Its +truth value is false. + + +NotImplemented +============== + +This type has a single value. There is a single object with this +value. This object is accessed through the built-in name +"NotImplemented". Numeric methods and rich comparison methods should +return this value if they do not implement the operation for the +operands provided. (The interpreter will then try the reflected +operation, or some other fallback, depending on the operator.) It +should not be evaluated in a boolean context. + +See Implementing the arithmetic operations for more details. + +Changed in version 3.9: Evaluating "NotImplemented" in a boolean +context was deprecated. + +Changed in version 3.14: Evaluating "NotImplemented" in a boolean +context now raises a "TypeError". It previously evaluated to "True" +and emitted a "DeprecationWarning" since Python 3.9. + + +Ellipsis +======== + +This type has a single value. There is a single object with this +value. This object is accessed through the literal "..." or the built- +in name "Ellipsis". Its truth value is true. + + +"numbers.Number" +================ + +These are created by numeric literals and returned as results by +arithmetic operators and arithmetic built-in functions. Numeric +objects are immutable; once created their value never changes. Python +numbers are of course strongly related to mathematical numbers, but +subject to the limitations of numerical representation in computers. + +The string representations of the numeric classes, computed by +"__repr__()" and "__str__()", have the following properties: + +* They are valid numeric literals which, when passed to their class + constructor, produce an object having the value of the original + numeric. + +* The representation is in base 10, when possible. + +* Leading zeros, possibly excepting a single zero before a decimal + point, are not shown. + +* Trailing zeros, possibly excepting a single zero after a decimal + point, are not shown. + +* A sign is shown only when the number is negative. + +Python distinguishes between integers, floating-point numbers, and +complex numbers: + + +"numbers.Integral" +------------------ + +These represent elements from the mathematical set of integers +(positive and negative). + +Note: + + The rules for integer representation are intended to give the most + meaningful interpretation of shift and mask operations involving + negative integers. + +There are two types of integers: + +Integers ("int") + These represent numbers in an unlimited range, subject to available + (virtual) memory only. For the purpose of shift and mask + operations, a binary representation is assumed, and negative + numbers are represented in a variant of 2’s complement which gives + the illusion of an infinite string of sign bits extending to the + left. + +Booleans ("bool") + These represent the truth values False and True. The two objects + representing the values "False" and "True" are the only Boolean + objects. The Boolean type is a subtype of the integer type, and + Boolean values behave like the values 0 and 1, respectively, in + almost all contexts, the exception being that when converted to a + string, the strings ""False"" or ""True"" are returned, + respectively. + + +"numbers.Real" ("float") +------------------------ + +These represent machine-level double precision floating-point numbers. +You are at the mercy of the underlying machine architecture (and C or +Java implementation) for the accepted range and handling of overflow. +Python does not support single-precision floating-point numbers; the +savings in processor and memory usage that are usually the reason for +using these are dwarfed by the overhead of using objects in Python, so +there is no reason to complicate the language with two kinds of +floating-point numbers. + + +"numbers.Complex" ("complex") +----------------------------- + +These represent complex numbers as a pair of machine-level double +precision floating-point numbers. The same caveats apply as for +floating-point numbers. The real and imaginary parts of a complex +number "z" can be retrieved through the read-only attributes "z.real" +and "z.imag". + + +Sequences +========= + +These represent finite ordered sets indexed by non-negative numbers. +The built-in function "len()" returns the number of items of a +sequence. When the length of a sequence is *n*, the index set contains +the numbers 0, 1, …, *n*-1. Item *i* of sequence *a* is selected by +"a[i]". Some sequences, including built-in sequences, interpret +negative subscripts by adding the sequence length. For example, +"a[-2]" equals "a[n-2]", the second to last item of sequence a with +length "n". + +Sequences also support slicing: "a[i:j]" selects all items with index +*k* such that *i* "<=" *k* "<" *j*. When used as an expression, a +slice is a sequence of the same type. The comment above about negative +indexes also applies to negative slice positions. + +Some sequences also support “extended slicing” with a third “step” +parameter: "a[i:j:k]" selects all items of *a* with index *x* where "x += i + n*k", *n* ">=" "0" and *i* "<=" *x* "<" *j*. + +Sequences are distinguished according to their mutability: + + +Immutable sequences +------------------- + +An object of an immutable sequence type cannot change once it is +created. (If the object contains references to other objects, these +other objects may be mutable and may be changed; however, the +collection of objects directly referenced by an immutable object +cannot change.) + +The following types are immutable sequences: + +Strings + A string is a sequence of values that represent Unicode code + points. All the code points in the range "U+0000 - U+10FFFF" can be + represented in a string. Python doesn’t have a char type; instead, + every code point in the string is represented as a string object + with length "1". The built-in function "ord()" converts a code + point from its string form to an integer in the range "0 - 10FFFF"; + "chr()" converts an integer in the range "0 - 10FFFF" to the + corresponding length "1" string object. "str.encode()" can be used + to convert a "str" to "bytes" using the given text encoding, and + "bytes.decode()" can be used to achieve the opposite. + +Tuples + The items of a tuple are arbitrary Python objects. Tuples of two or + more items are formed by comma-separated lists of expressions. A + tuple of one item (a ‘singleton’) can be formed by affixing a comma + to an expression (an expression by itself does not create a tuple, + since parentheses must be usable for grouping of expressions). An + empty tuple can be formed by an empty pair of parentheses. + +Bytes + A bytes object is an immutable array. The items are 8-bit bytes, + represented by integers in the range 0 <= x < 256. Bytes literals + (like "b'abc'") and the built-in "bytes()" constructor can be used + to create bytes objects. Also, bytes objects can be decoded to + strings via the "decode()" method. + + +Mutable sequences +----------------- + +Mutable sequences can be changed after they are created. The +subscription and slicing notations can be used as the target of +assignment and "del" (delete) statements. + +Note: + + The "collections" and "array" module provide additional examples of + mutable sequence types. + +There are currently two intrinsic mutable sequence types: + +Lists + The items of a list are arbitrary Python objects. Lists are formed + by placing a comma-separated list of expressions in square + brackets. (Note that there are no special cases needed to form + lists of length 0 or 1.) + +Byte Arrays + A bytearray object is a mutable array. They are created by the + built-in "bytearray()" constructor. Aside from being mutable (and + hence unhashable), byte arrays otherwise provide the same interface + and functionality as immutable "bytes" objects. + + +Set types +========= + +These represent unordered, finite sets of unique, immutable objects. +As such, they cannot be indexed by any subscript. However, they can be +iterated over, and the built-in function "len()" returns the number of +items in a set. Common uses for sets are fast membership testing, +removing duplicates from a sequence, and computing mathematical +operations such as intersection, union, difference, and symmetric +difference. + +For set elements, the same immutability rules apply as for dictionary +keys. Note that numeric types obey the normal rules for numeric +comparison: if two numbers compare equal (e.g., "1" and "1.0"), only +one of them can be contained in a set. + +There are currently two intrinsic set types: + +Sets + These represent a mutable set. They are created by the built-in + "set()" constructor and can be modified afterwards by several + methods, such as "add()". + +Frozen sets + These represent an immutable set. They are created by the built-in + "frozenset()" constructor. As a frozenset is immutable and + *hashable*, it can be used again as an element of another set, or + as a dictionary key. + + +Mappings +======== + +These represent finite sets of objects indexed by arbitrary index +sets. The subscript notation "a[k]" selects the item indexed by "k" +from the mapping "a"; this can be used in expressions and as the +target of assignments or "del" statements. The built-in function +"len()" returns the number of items in a mapping. + +There is currently a single intrinsic mapping type: + + +Dictionaries +------------ + +These represent finite sets of objects indexed by nearly arbitrary +values. The only types of values not acceptable as keys are values +containing lists or dictionaries or other mutable types that are +compared by value rather than by object identity, the reason being +that the efficient implementation of dictionaries requires a key’s +hash value to remain constant. Numeric types used for keys obey the +normal rules for numeric comparison: if two numbers compare equal +(e.g., "1" and "1.0") then they can be used interchangeably to index +the same dictionary entry. + +Dictionaries preserve insertion order, meaning that keys will be +produced in the same order they were added sequentially over the +dictionary. Replacing an existing key does not change the order, +however removing a key and re-inserting it will add it to the end +instead of keeping its old place. + +Dictionaries are mutable; they can be created by the "{}" notation +(see section Dictionary displays). + +The extension modules "dbm.ndbm" and "dbm.gnu" provide additional +examples of mapping types, as does the "collections" module. + +Changed in version 3.7: Dictionaries did not preserve insertion order +in versions of Python before 3.6. In CPython 3.6, insertion order was +preserved, but it was considered an implementation detail at that time +rather than a language guarantee. + + +Callable types +============== + +These are the types to which the function call operation (see section +Calls) can be applied: + + +User-defined functions +---------------------- + +A user-defined function object is created by a function definition +(see section Function definitions). It should be called with an +argument list containing the same number of items as the function’s +formal parameter list. + + +Special read-only attributes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + ++----------------------------------------------------+----------------------------------------------------+ +| Attribute | Meaning | +|====================================================|====================================================| +| function.__globals__ | A reference to the "dictionary" that holds the | +| | function’s global variables – the global namespace | +| | of the module in which the function was defined. | ++----------------------------------------------------+----------------------------------------------------+ +| function.__closure__ | "None" or a "tuple" of cells that contain bindings | +| | for the names specified in the "co_freevars" | +| | attribute of the function’s "code object". A cell | +| | object has the attribute "cell_contents". This can | +| | be used to get the value of the cell, as well as | +| | set the value. | ++----------------------------------------------------+----------------------------------------------------+ + + +Special writable attributes +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Most of these attributes check the type of the assigned value: + ++----------------------------------------------------+----------------------------------------------------+ +| Attribute | Meaning | +|====================================================|====================================================| +| function.__doc__ | The function’s documentation string, or "None" if | +| | unavailable. | ++----------------------------------------------------+----------------------------------------------------+ +| function.__name__ | The function’s name. See also: "__name__ | +| | attributes". | ++----------------------------------------------------+----------------------------------------------------+ +| function.__qualname__ | The function’s *qualified name*. See also: | +| | "__qualname__ attributes". Added in version 3.3. | ++----------------------------------------------------+----------------------------------------------------+ +| function.__module__ | The name of the module the function was defined | +| | in, or "None" if unavailable. | ++----------------------------------------------------+----------------------------------------------------+ +| function.__defaults__ | A "tuple" containing default *parameter* values | +| | for those parameters that have defaults, or "None" | +| | if no parameters have a default value. | ++----------------------------------------------------+----------------------------------------------------+ +| function.__code__ | The code object representing the compiled function | +| | body. | ++----------------------------------------------------+----------------------------------------------------+ +| function.__dict__ | The namespace supporting arbitrary function | +| | attributes. See also: "__dict__ attributes". | ++----------------------------------------------------+----------------------------------------------------+ +| function.__annotations__ | A "dictionary" containing annotations of | +| | *parameters*. The keys of the dictionary are the | +| | parameter names, and "'return'" for the return | +| | annotation, if provided. See also: | +| | "object.__annotations__". Changed in version | +| | 3.14: Annotations are now lazily evaluated. See | +| | **PEP 649**. | ++----------------------------------------------------+----------------------------------------------------+ +| function.__annotate__ | The *annotate function* for this function, or | +| | "None" if the function has no annotations. See | +| | "object.__annotate__". Added in version 3.14. | ++----------------------------------------------------+----------------------------------------------------+ +| function.__kwdefaults__ | A "dictionary" containing defaults for keyword- | +| | only *parameters*. | ++----------------------------------------------------+----------------------------------------------------+ +| function.__type_params__ | A "tuple" containing the type parameters of a | +| | generic function. Added in version 3.12. | ++----------------------------------------------------+----------------------------------------------------+ + +Function objects also support getting and setting arbitrary +attributes, which can be used, for example, to attach metadata to +functions. Regular attribute dot-notation is used to get and set such +attributes. + +**CPython implementation detail:** CPython’s current implementation +only supports function attributes on user-defined functions. Function +attributes on built-in functions may be supported in the future. + +Additional information about a function’s definition can be retrieved +from its code object (accessible via the "__code__" attribute). + + +Instance methods +---------------- + +An instance method object combines a class, a class instance and any +callable object (normally a user-defined function). + +Special read-only attributes: + ++----------------------------------------------------+----------------------------------------------------+ +| method.__self__ | Refers to the class instance object to which the | +| | method is bound | ++----------------------------------------------------+----------------------------------------------------+ +| method.__func__ | Refers to the original function object | ++----------------------------------------------------+----------------------------------------------------+ +| method.__doc__ | The method’s documentation (same as | +| | "method.__func__.__doc__"). A "string" if the | +| | original function had a docstring, else "None". | ++----------------------------------------------------+----------------------------------------------------+ +| method.__name__ | The name of the method (same as | +| | "method.__func__.__name__") | ++----------------------------------------------------+----------------------------------------------------+ +| method.__module__ | The name of the module the method was defined in, | +| | or "None" if unavailable. | ++----------------------------------------------------+----------------------------------------------------+ + +Methods also support accessing (but not setting) the arbitrary +function attributes on the underlying function object. + +User-defined method objects may be created when getting an attribute +of a class (perhaps via an instance of that class), if that attribute +is a user-defined function object or a "classmethod" object. + +When an instance method object is created by retrieving a user-defined +function object from a class via one of its instances, its "__self__" +attribute is the instance, and the method object is said to be +*bound*. The new method’s "__func__" attribute is the original +function object. + +When an instance method object is created by retrieving a +"classmethod" object from a class or instance, its "__self__" +attribute is the class itself, and its "__func__" attribute is the +function object underlying the class method. + +When an instance method object is called, the underlying function +("__func__") is called, inserting the class instance ("__self__") in +front of the argument list. For instance, when "C" is a class which +contains a definition for a function "f()", and "x" is an instance of +"C", calling "x.f(1)" is equivalent to calling "C.f(x, 1)". + +When an instance method object is derived from a "classmethod" object, +the “class instance” stored in "__self__" will actually be the class +itself, so that calling either "x.f(1)" or "C.f(1)" is equivalent to +calling "f(C,1)" where "f" is the underlying function. + +It is important to note that user-defined functions which are +attributes of a class instance are not converted to bound methods; +this *only* happens when the function is an attribute of the class. + + +Generator functions +------------------- + +A function or method which uses the "yield" statement (see section The +yield statement) is called a *generator function*. Such a function, +when called, always returns an *iterator* object which can be used to +execute the body of the function: calling the iterator’s +"iterator.__next__()" method will cause the function to execute until +it provides a value using the "yield" statement. When the function +executes a "return" statement or falls off the end, a "StopIteration" +exception is raised and the iterator will have reached the end of the +set of values to be returned. + + +Coroutine functions +------------------- + +A function or method which is defined using "async def" is called a +*coroutine function*. Such a function, when called, returns a +*coroutine* object. It may contain "await" expressions, as well as +"async with" and "async for" statements. See also the Coroutine +Objects section. + + +Asynchronous generator functions +-------------------------------- + +A function or method which is defined using "async def" and which uses +the "yield" statement is called a *asynchronous generator function*. +Such a function, when called, returns an *asynchronous iterator* +object which can be used in an "async for" statement to execute the +body of the function. + +Calling the asynchronous iterator’s "aiterator.__anext__" method will +return an *awaitable* which when awaited will execute until it +provides a value using the "yield" expression. When the function +executes an empty "return" statement or falls off the end, a +"StopAsyncIteration" exception is raised and the asynchronous iterator +will have reached the end of the set of values to be yielded. + + +Built-in functions +------------------ + +A built-in function object is a wrapper around a C function. Examples +of built-in functions are "len()" and "math.sin()" ("math" is a +standard built-in module). The number and type of the arguments are +determined by the C function. Special read-only attributes: + +* "__doc__" is the function’s documentation string, or "None" if + unavailable. See "function.__doc__". + +* "__name__" is the function’s name. See "function.__name__". + +* "__self__" is set to "None" (but see the next item). + +* "__module__" is the name of the module the function was defined in + or "None" if unavailable. See "function.__module__". + + +Built-in methods +---------------- + +This is really a different disguise of a built-in function, this time +containing an object passed to the C function as an implicit extra +argument. An example of a built-in method is "alist.append()", +assuming *alist* is a list object. In this case, the special read-only +attribute "__self__" is set to the object denoted by *alist*. (The +attribute has the same semantics as it does with "other instance +methods".) + + +Classes +------- + +Classes are callable. These objects normally act as factories for new +instances of themselves, but variations are possible for class types +that override "__new__()". The arguments of the call are passed to +"__new__()" and, in the typical case, to "__init__()" to initialize +the new instance. + + +Class Instances +--------------- + +Instances of arbitrary classes can be made callable by defining a +"__call__()" method in their class. + + +Modules +======= + +Modules are a basic organizational unit of Python code, and are +created by the import system as invoked either by the "import" +statement, or by calling functions such as "importlib.import_module()" +and built-in "__import__()". A module object has a namespace +implemented by a "dictionary" object (this is the dictionary +referenced by the "__globals__" attribute of functions defined in the +module). Attribute references are translated to lookups in this +dictionary, e.g., "m.x" is equivalent to "m.__dict__["x"]". A module +object does not contain the code object used to initialize the module +(since it isn’t needed once the initialization is done). + +Attribute assignment updates the module’s namespace dictionary, e.g., +"m.x = 1" is equivalent to "m.__dict__["x"] = 1". + + +Import-related attributes on module objects +------------------------------------------- + +Module objects have the following attributes that relate to the import +system. When a module is created using the machinery associated with +the import system, these attributes are filled in based on the +module’s *spec*, before the *loader* executes and loads the module. + +To create a module dynamically rather than using the import system, +it’s recommended to use "importlib.util.module_from_spec()", which +will set the various import-controlled attributes to appropriate +values. It’s also possible to use the "types.ModuleType" constructor +to create modules directly, but this technique is more error-prone, as +most attributes must be manually set on the module object after it has +been created when using this approach. + +Caution: + + With the exception of "__name__", it is **strongly** recommended + that you rely on "__spec__" and its attributes instead of any of the + other individual attributes listed in this subsection. Note that + updating an attribute on "__spec__" will not update the + corresponding attribute on the module itself: + + >>> import typing + >>> typing.__name__, typing.__spec__.name + ('typing', 'typing') + >>> typing.__spec__.name = 'spelling' + >>> typing.__name__, typing.__spec__.name + ('typing', 'spelling') + >>> typing.__name__ = 'keyboard_smashing' + >>> typing.__name__, typing.__spec__.name + ('keyboard_smashing', 'spelling') + +module.__name__ + + The name used to uniquely identify the module in the import system. + For a directly executed module, this will be set to ""__main__"". + + This attribute must be set to the fully qualified name of the + module. It is expected to match the value of + "module.__spec__.name". + +module.__spec__ + + A record of the module’s import-system-related state. + + Set to the "module spec" that was used when importing the module. + See Module specs for more details. + + Added in version 3.4. + +module.__package__ + + The *package* a module belongs to. + + If the module is top-level (that is, not a part of any specific + package) then the attribute should be set to "''" (the empty + string). Otherwise, it should be set to the name of the module’s + package (which can be equal to "module.__name__" if the module + itself is a package). See **PEP 366** for further details. + + This attribute is used instead of "__name__" to calculate explicit + relative imports for main modules. It defaults to "None" for + modules created dynamically using the "types.ModuleType" + constructor; use "importlib.util.module_from_spec()" instead to + ensure the attribute is set to a "str". + + It is **strongly** recommended that you use + "module.__spec__.parent" instead of "module.__package__". + "__package__" is now only used as a fallback if "__spec__.parent" + is not set, and this fallback path is deprecated. + + Changed in version 3.4: This attribute now defaults to "None" for + modules created dynamically using the "types.ModuleType" + constructor. Previously the attribute was optional. + + Changed in version 3.6: The value of "__package__" is expected to + be the same as "__spec__.parent". "__package__" is now only used as + a fallback during import resolution if "__spec__.parent" is not + defined. + + Changed in version 3.10: "ImportWarning" is raised if an import + resolution falls back to "__package__" instead of + "__spec__.parent". + + Changed in version 3.12: Raise "DeprecationWarning" instead of + "ImportWarning" when falling back to "__package__" during import + resolution. + + Deprecated since version 3.13, will be removed in version 3.15: + "__package__" will cease to be set or taken into consideration by + the import system or standard library. + +module.__loader__ + + The *loader* object that the import machinery used to load the + module. + + This attribute is mostly useful for introspection, but can be used + for additional loader-specific functionality, for example getting + data associated with a loader. + + "__loader__" defaults to "None" for modules created dynamically + using the "types.ModuleType" constructor; use + "importlib.util.module_from_spec()" instead to ensure the attribute + is set to a *loader* object. + + It is **strongly** recommended that you use + "module.__spec__.loader" instead of "module.__loader__". + + Changed in version 3.4: This attribute now defaults to "None" for + modules created dynamically using the "types.ModuleType" + constructor. Previously the attribute was optional. + + Deprecated since version 3.12, will be removed in version 3.16: + Setting "__loader__" on a module while failing to set + "__spec__.loader" is deprecated. In Python 3.16, "__loader__" will + cease to be set or taken into consideration by the import system or + the standard library. + +module.__path__ + + A (possibly empty) *sequence* of strings enumerating the locations + where the package’s submodules will be found. Non-package modules + should not have a "__path__" attribute. See __path__ attributes on + modules for more details. + + It is **strongly** recommended that you use + "module.__spec__.submodule_search_locations" instead of + "module.__path__". + +module.__file__ + +module.__cached__ + + "__file__" and "__cached__" are both optional attributes that may + or may not be set. Both attributes should be a "str" when they are + available. + + "__file__" indicates the pathname of the file from which the module + was loaded (if loaded from a file), or the pathname of the shared + library file for extension modules loaded dynamically from a shared + library. It might be missing for certain types of modules, such as + C modules that are statically linked into the interpreter, and the + import system may opt to leave it unset if it has no semantic + meaning (for example, a module loaded from a database). + + If "__file__" is set then the "__cached__" attribute might also be + set, which is the path to any compiled version of the code (for + example, a byte-compiled file). The file does not need to exist to + set this attribute; the path can simply point to where the compiled + file *would* exist (see **PEP 3147**). + + Note that "__cached__" may be set even if "__file__" is not set. + However, that scenario is quite atypical. Ultimately, the *loader* + is what makes use of the module spec provided by the *finder* (from + which "__file__" and "__cached__" are derived). So if a loader can + load from a cached module but otherwise does not load from a file, + that atypical scenario may be appropriate. + + It is **strongly** recommended that you use + "module.__spec__.cached" instead of "module.__cached__". + + Deprecated since version 3.13, will be removed in version 3.15: + Setting "__cached__" on a module while failing to set + "__spec__.cached" is deprecated. In Python 3.15, "__cached__" will + cease to be set or taken into consideration by the import system or + standard library. + + +Other writable attributes on module objects +------------------------------------------- + +As well as the import-related attributes listed above, module objects +also have the following writable attributes: + +module.__doc__ + + The module’s documentation string, or "None" if unavailable. See + also: "__doc__ attributes". + +module.__annotations__ + + A dictionary containing *variable annotations* collected during + module body execution. For best practices on working with + "__annotations__", see "annotationlib". + + Changed in version 3.14: Annotations are now lazily evaluated. See + **PEP 649**. + +module.__annotate__ + + The *annotate function* for this module, or "None" if the module + has no annotations. See also: "__annotate__" attributes. + + Added in version 3.14. + + +Module dictionaries +------------------- + +Module objects also have the following special read-only attribute: + +module.__dict__ + + The module’s namespace as a dictionary object. Uniquely among the + attributes listed here, "__dict__" cannot be accessed as a global + variable from within a module; it can only be accessed as an + attribute on module objects. + + **CPython implementation detail:** Because of the way CPython + clears module dictionaries, the module dictionary will be cleared + when the module falls out of scope even if the dictionary still has + live references. To avoid this, copy the dictionary or keep the + module around while using its dictionary directly. + + +Custom classes +============== + +Custom class types are typically created by class definitions (see +section Class definitions). A class has a namespace implemented by a +dictionary object. Class attribute references are translated to +lookups in this dictionary, e.g., "C.x" is translated to +"C.__dict__["x"]" (although there are a number of hooks which allow +for other means of locating attributes). When the attribute name is +not found there, the attribute search continues in the base classes. +This search of the base classes uses the C3 method resolution order +which behaves correctly even in the presence of ‘diamond’ inheritance +structures where there are multiple inheritance paths leading back to +a common ancestor. Additional details on the C3 MRO used by Python can +be found at The Python 2.3 Method Resolution Order. + +When a class attribute reference (for class "C", say) would yield a +class method object, it is transformed into an instance method object +whose "__self__" attribute is "C". When it would yield a +"staticmethod" object, it is transformed into the object wrapped by +the static method object. See section Implementing Descriptors for +another way in which attributes retrieved from a class may differ from +those actually contained in its "__dict__". + +Class attribute assignments update the class’s dictionary, never the +dictionary of a base class. + +A class object can be called (see above) to yield a class instance +(see below). + + +Special attributes +------------------ + ++----------------------------------------------------+----------------------------------------------------+ +| Attribute | Meaning | +|====================================================|====================================================| +| type.__name__ | The class’s name. See also: "__name__ attributes". | ++----------------------------------------------------+----------------------------------------------------+ +| type.__qualname__ | The class’s *qualified name*. See also: | +| | "__qualname__ attributes". | ++----------------------------------------------------+----------------------------------------------------+ +| type.__module__ | The name of the module in which the class was | +| | defined. | ++----------------------------------------------------+----------------------------------------------------+ +| type.__dict__ | A "mapping proxy" providing a read-only view of | +| | the class’s namespace. See also: "__dict__ | +| | attributes". | ++----------------------------------------------------+----------------------------------------------------+ +| type.__bases__ | A "tuple" containing the class’s bases. In most | +| | cases, for a class defined as "class X(A, B, C)", | +| | "X.__bases__" will be exactly equal to "(A, B, | +| | C)". | ++----------------------------------------------------+----------------------------------------------------+ +| type.__base__ | **CPython implementation detail:** The single base | +| | class in the inheritance chain that is responsible | +| | for the memory layout of instances. This attribute | +| | corresponds to "tp_base" at the C level. | ++----------------------------------------------------+----------------------------------------------------+ +| type.__doc__ | The class’s documentation string, or "None" if | +| | undefined. Not inherited by subclasses. | ++----------------------------------------------------+----------------------------------------------------+ +| type.__annotations__ | A dictionary containing *variable annotations* | +| | collected during class body execution. See also: | +| | "__annotations__ attributes". For best practices | +| | on working with "__annotations__", please see | +| | "annotationlib". Use | +| | "annotationlib.get_annotations()" instead of | +| | accessing this attribute directly. Warning: | +| | Accessing the "__annotations__" attribute directly | +| | on a class object may return annotations for the | +| | wrong class, specifically in certain cases where | +| | the class, its base class, or a metaclass is | +| | defined under "from __future__ import | +| | annotations". See **749** for details.This | +| | attribute does not exist on certain builtin | +| | classes. On user-defined classes without | +| | "__annotations__", it is an empty dictionary. | +| | Changed in version 3.14: Annotations are now | +| | lazily evaluated. See **PEP 649**. | ++----------------------------------------------------+----------------------------------------------------+ +| type.__annotate__() | The *annotate function* for this class, or "None" | +| | if the class has no annotations. See also: | +| | "__annotate__ attributes". Added in version 3.14. | ++----------------------------------------------------+----------------------------------------------------+ +| type.__type_params__ | A "tuple" containing the type parameters of a | +| | generic class. Added in version 3.12. | ++----------------------------------------------------+----------------------------------------------------+ +| type.__static_attributes__ | A "tuple" containing names of attributes of this | +| | class which are assigned through "self.X" from any | +| | function in its body. Added in version 3.13. | ++----------------------------------------------------+----------------------------------------------------+ +| type.__firstlineno__ | The line number of the first line of the class | +| | definition, including decorators. Setting the | +| | "__module__" attribute removes the | +| | "__firstlineno__" item from the type’s dictionary. | +| | Added in version 3.13. | ++----------------------------------------------------+----------------------------------------------------+ +| type.__mro__ | The "tuple" of classes that are considered when | +| | looking for base classes during method resolution. | ++----------------------------------------------------+----------------------------------------------------+ + + +Special methods +--------------- + +In addition to the special attributes described above, all Python +classes also have the following two methods available: + +type.mro() + + This method can be overridden by a metaclass to customize the + method resolution order for its instances. It is called at class + instantiation, and its result is stored in "__mro__". + +type.__subclasses__() + + Each class keeps a list of weak references to its immediate + subclasses. This method returns a list of all those references + still alive. The list is in definition order. Example: + + >>> class A: pass + >>> class B(A): pass + >>> A.__subclasses__() + [] + + +Class instances +=============== + +A class instance is created by calling a class object (see above). A +class instance has a namespace implemented as a dictionary which is +the first place in which attribute references are searched. When an +attribute is not found there, and the instance’s class has an +attribute by that name, the search continues with the class +attributes. If a class attribute is found that is a user-defined +function object, it is transformed into an instance method object +whose "__self__" attribute is the instance. Static method and class +method objects are also transformed; see above under “Classes”. See +section Implementing Descriptors for another way in which attributes +of a class retrieved via its instances may differ from the objects +actually stored in the class’s "__dict__". If no class attribute is +found, and the object’s class has a "__getattr__()" method, that is +called to satisfy the lookup. + +Attribute assignments and deletions update the instance’s dictionary, +never a class’s dictionary. If the class has a "__setattr__()" or +"__delattr__()" method, this is called instead of updating the +instance dictionary directly. + +Class instances can pretend to be numbers, sequences, or mappings if +they have methods with certain special names. See section Special +method names. + + +Special attributes +------------------ + +object.__class__ + + The class to which a class instance belongs. + +object.__dict__ + + A dictionary or other mapping object used to store an object’s + (writable) attributes. Not all instances have a "__dict__" + attribute; see the section on __slots__ for more details. + + +I/O objects (also known as file objects) +======================================== + +A *file object* represents an open file. Various shortcuts are +available to create file objects: the "open()" built-in function, and +also "os.popen()", "os.fdopen()", and the "makefile()" method of +socket objects (and perhaps by other functions or methods provided by +extension modules). + +The objects "sys.stdin", "sys.stdout" and "sys.stderr" are initialized +to file objects corresponding to the interpreter’s standard input, +output and error streams; they are all open in text mode and therefore +follow the interface defined by the "io.TextIOBase" abstract class. + + +Internal types +============== + +A few types used internally by the interpreter are exposed to the +user. Their definitions may change with future versions of the +interpreter, but they are mentioned here for completeness. + + +Code objects +------------ + +Code objects represent *byte-compiled* executable Python code, or +*bytecode*. The difference between a code object and a function object +is that the function object contains an explicit reference to the +function’s globals (the module in which it was defined), while a code +object contains no context; also the default argument values are +stored in the function object, not in the code object (because they +represent values calculated at run-time). Unlike function objects, +code objects are immutable and contain no references (directly or +indirectly) to mutable objects. + + +Special read-only attributes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + ++----------------------------------------------------+----------------------------------------------------+ +| codeobject.co_name | The function name | ++----------------------------------------------------+----------------------------------------------------+ +| codeobject.co_qualname | The fully qualified function name Added in | +| | version 3.11. | ++----------------------------------------------------+----------------------------------------------------+ +| codeobject.co_argcount | The total number of positional *parameters* | +| | (including positional-only parameters and | +| | parameters with default values) that the function | +| | has | ++----------------------------------------------------+----------------------------------------------------+ +| codeobject.co_posonlyargcount | The number of positional-only *parameters* | +| | (including arguments with default values) that the | +| | function has | ++----------------------------------------------------+----------------------------------------------------+ +| codeobject.co_kwonlyargcount | The number of keyword-only *parameters* (including | +| | arguments with default values) that the function | +| | has | ++----------------------------------------------------+----------------------------------------------------+ +| codeobject.co_nlocals | The number of local variables used by the function | +| | (including parameters) | ++----------------------------------------------------+----------------------------------------------------+ +| codeobject.co_varnames | A "tuple" containing the names of the local | +| | variables in the function (starting with the | +| | parameter names) | ++----------------------------------------------------+----------------------------------------------------+ +| codeobject.co_cellvars | A "tuple" containing the names of local variables | +| | that are referenced from at least one *nested | +| | scope* inside the function | ++----------------------------------------------------+----------------------------------------------------+ +| codeobject.co_freevars | A "tuple" containing the names of *free (closure) | +| | variables* that a *nested scope* references in an | +| | outer scope. See also "function.__closure__". | +| | Note: references to global and builtin names are | +| | *not* included. | ++----------------------------------------------------+----------------------------------------------------+ +| codeobject.co_code | A string representing the sequence of *bytecode* | +| | instructions in the function | ++----------------------------------------------------+----------------------------------------------------+ +| codeobject.co_consts | A "tuple" containing the literals used by the | +| | *bytecode* in the function | ++----------------------------------------------------+----------------------------------------------------+ +| codeobject.co_names | A "tuple" containing the names used by the | +| | *bytecode* in the function | ++----------------------------------------------------+----------------------------------------------------+ +| codeobject.co_filename | The name of the file from which the code was | +| | compiled | ++----------------------------------------------------+----------------------------------------------------+ +| codeobject.co_firstlineno | The line number of the first line of the function | ++----------------------------------------------------+----------------------------------------------------+ +| codeobject.co_lnotab | A string encoding the mapping from *bytecode* | +| | offsets to line numbers. For details, see the | +| | source code of the interpreter. Deprecated since | +| | version 3.12: This attribute of code objects is | +| | deprecated, and may be removed in Python 3.15. | ++----------------------------------------------------+----------------------------------------------------+ +| codeobject.co_stacksize | The required stack size of the code object | ++----------------------------------------------------+----------------------------------------------------+ +| codeobject.co_flags | An "integer" encoding a number of flags for the | +| | interpreter. | ++----------------------------------------------------+----------------------------------------------------+ + +The following flag bits are defined for "co_flags": bit "0x04" is set +if the function uses the "*arguments" syntax to accept an arbitrary +number of positional arguments; bit "0x08" is set if the function uses +the "**keywords" syntax to accept arbitrary keyword arguments; bit +"0x20" is set if the function is a generator. See Code Objects Bit +Flags for details on the semantics of each flags that might be +present. + +Future feature declarations (for example, "from __future__ import +division") also use bits in "co_flags" to indicate whether a code +object was compiled with a particular feature enabled. See +"compiler_flag". + +Other bits in "co_flags" are reserved for internal use. + +If a code object represents a function and has a docstring, the +"CO_HAS_DOCSTRING" bit is set in "co_flags" and the first item in +"co_consts" is the docstring of the function. + + +Methods on code objects +~~~~~~~~~~~~~~~~~~~~~~~ + +codeobject.co_positions() + + Returns an iterable over the source code positions of each + *bytecode* instruction in the code object. + + The iterator returns "tuple"s containing the "(start_line, + end_line, start_column, end_column)". The *i-th* tuple corresponds + to the position of the source code that compiled to the *i-th* code + unit. Column information is 0-indexed utf-8 byte offsets on the + given source line. + + This positional information can be missing. A non-exhaustive lists + of cases where this may happen: + + * Running the interpreter with "-X" "no_debug_ranges". + + * Loading a pyc file compiled while using "-X" "no_debug_ranges". + + * Position tuples corresponding to artificial instructions. + + * Line and column numbers that can’t be represented due to + implementation specific limitations. + + When this occurs, some or all of the tuple elements can be "None". + + Added in version 3.11. + + Note: + + This feature requires storing column positions in code objects + which may result in a small increase of disk usage of compiled + Python files or interpreter memory usage. To avoid storing the + extra information and/or deactivate printing the extra traceback + information, the "-X" "no_debug_ranges" command line flag or the + "PYTHONNODEBUGRANGES" environment variable can be used. + +codeobject.co_lines() + + Returns an iterator that yields information about successive ranges + of *bytecode*s. Each item yielded is a "(start, end, lineno)" + "tuple": + + * "start" (an "int") represents the offset (inclusive) of the start + of the *bytecode* range + + * "end" (an "int") represents the offset (exclusive) of the end of + the *bytecode* range + + * "lineno" is an "int" representing the line number of the + *bytecode* range, or "None" if the bytecodes in the given range + have no line number + + The items yielded will have the following properties: + + * The first range yielded will have a "start" of 0. + + * The "(start, end)" ranges will be non-decreasing and consecutive. + That is, for any pair of "tuple"s, the "start" of the second will + be equal to the "end" of the first. + + * No range will be backwards: "end >= start" for all triples. + + * The last "tuple" yielded will have "end" equal to the size of the + *bytecode*. + + Zero-width ranges, where "start == end", are allowed. Zero-width + ranges are used for lines that are present in the source code, but + have been eliminated by the *bytecode* compiler. + + Added in version 3.10. + + See also: + + **PEP 626** - Precise line numbers for debugging and other tools. + The PEP that introduced the "co_lines()" method. + +codeobject.replace(**kwargs) + + Return a copy of the code object with new values for the specified + fields. + + Code objects are also supported by the generic function + "copy.replace()". + + Added in version 3.8. + + +Frame objects +------------- + +Frame objects represent execution frames. They may occur in traceback +objects, and are also passed to registered trace functions. + + +Special read-only attributes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + ++----------------------------------------------------+----------------------------------------------------+ +| frame.f_back | Points to the previous stack frame (towards the | +| | caller), or "None" if this is the bottom stack | +| | frame | ++----------------------------------------------------+----------------------------------------------------+ +| frame.f_code | The code object being executed in this frame. | +| | Accessing this attribute raises an auditing event | +| | "object.__getattr__" with arguments "obj" and | +| | ""f_code"". | ++----------------------------------------------------+----------------------------------------------------+ +| frame.f_locals | The mapping used by the frame to look up local | +| | variables. If the frame refers to an *optimized | +| | scope*, this may return a write-through proxy | +| | object. Changed in version 3.13: Return a proxy | +| | for optimized scopes. | ++----------------------------------------------------+----------------------------------------------------+ +| frame.f_globals | The dictionary used by the frame to look up global | +| | variables | ++----------------------------------------------------+----------------------------------------------------+ +| frame.f_builtins | The dictionary used by the frame to look up built- | +| | in (intrinsic) names | ++----------------------------------------------------+----------------------------------------------------+ +| frame.f_lasti | The “precise instruction” of the frame object | +| | (this is an index into the *bytecode* string of | +| | the code object) | ++----------------------------------------------------+----------------------------------------------------+ +| frame.f_generator | The *generator* or *coroutine* object that owns | +| | this frame, or "None" if the frame is a normal | +| | function. Added in version 3.14. | ++----------------------------------------------------+----------------------------------------------------+ + + +Special writable attributes +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + ++----------------------------------------------------+----------------------------------------------------+ +| frame.f_trace | If not "None", this is a function called for | +| | various events during code execution (this is used | +| | by debuggers). Normally an event is triggered for | +| | each new source line (see "f_trace_lines"). | ++----------------------------------------------------+----------------------------------------------------+ +| frame.f_trace_lines | Set this attribute to "False" to disable | +| | triggering a tracing event for each source line. | ++----------------------------------------------------+----------------------------------------------------+ +| frame.f_trace_opcodes | Set this attribute to "True" to allow per-opcode | +| | events to be requested. Note that this may lead to | +| | undefined interpreter behaviour if exceptions | +| | raised by the trace function escape to the | +| | function being traced. | ++----------------------------------------------------+----------------------------------------------------+ +| frame.f_lineno | The current line number of the frame – writing to | +| | this from within a trace function jumps to the | +| | given line (only for the bottom-most frame). A | +| | debugger can implement a Jump command (aka Set | +| | Next Statement) by writing to this attribute. | ++----------------------------------------------------+----------------------------------------------------+ + + +Frame object methods +~~~~~~~~~~~~~~~~~~~~ + +Frame objects support one method: + +frame.clear() + + This method clears all references to local variables held by the + frame. Also, if the frame belonged to a *generator*, the generator + is finalized. This helps break reference cycles involving frame + objects (for example when catching an exception and storing its + traceback for later use). + + "RuntimeError" is raised if the frame is currently executing or + suspended. + + Added in version 3.4. + + Changed in version 3.13: Attempting to clear a suspended frame + raises "RuntimeError" (as has always been the case for executing + frames). + + +Traceback objects +----------------- + +Traceback objects represent the stack trace of an exception. A +traceback object is implicitly created when an exception occurs, and +may also be explicitly created by calling "types.TracebackType". + +Changed in version 3.7: Traceback objects can now be explicitly +instantiated from Python code. + +For implicitly created tracebacks, when the search for an exception +handler unwinds the execution stack, at each unwound level a traceback +object is inserted in front of the current traceback. When an +exception handler is entered, the stack trace is made available to the +program. (See section The try statement.) It is accessible as the +third item of the tuple returned by "sys.exc_info()", and as the +"__traceback__" attribute of the caught exception. + +When the program contains no suitable handler, the stack trace is +written (nicely formatted) to the standard error stream; if the +interpreter is interactive, it is also made available to the user as +"sys.last_traceback". + +For explicitly created tracebacks, it is up to the creator of the +traceback to determine how the "tb_next" attributes should be linked +to form a full stack trace. + +Special read-only attributes: + ++----------------------------------------------------+----------------------------------------------------+ +| traceback.tb_frame | Points to the execution frame of the current | +| | level. Accessing this attribute raises an | +| | auditing event "object.__getattr__" with arguments | +| | "obj" and ""tb_frame"". | ++----------------------------------------------------+----------------------------------------------------+ +| traceback.tb_lineno | Gives the line number where the exception occurred | ++----------------------------------------------------+----------------------------------------------------+ +| traceback.tb_lasti | Indicates the “precise instruction”. | ++----------------------------------------------------+----------------------------------------------------+ + +The line number and last instruction in the traceback may differ from +the line number of its frame object if the exception occurred in a +"try" statement with no matching except clause or with a "finally" +clause. + +traceback.tb_next + + The special writable attribute "tb_next" is the next level in the + stack trace (towards the frame where the exception occurred), or + "None" if there is no next level. + + Changed in version 3.7: This attribute is now writable + + +Slice objects +------------- + +Slice objects are used to represent slices for "__getitem__()" +methods. They are also created by the built-in "slice()" function. + +Special read-only attributes: "start" is the lower bound; "stop" is +the upper bound; "step" is the step value; each is "None" if omitted. +These attributes can have any type. + +Slice objects support one method: + +slice.indices(self, length) + + This method takes a single integer argument *length* and computes + information about the slice that the slice object would describe if + applied to a sequence of *length* items. It returns a tuple of + three integers; respectively these are the *start* and *stop* + indices and the *step* or stride length of the slice. Missing or + out-of-bounds indices are handled in a manner consistent with + regular slices. + + +Static method objects +--------------------- + +Static method objects provide a way of defeating the transformation of +function objects to method objects described above. A static method +object is a wrapper around any other object, usually a user-defined +method object. When a static method object is retrieved from a class +or a class instance, the object actually returned is the wrapped +object, which is not subject to any further transformation. Static +method objects are also callable. Static method objects are created by +the built-in "staticmethod()" constructor. + + +Class method objects +-------------------- + +A class method object, like a static method object, is a wrapper +around another object that alters the way in which that object is +retrieved from classes and class instances. The behaviour of class +method objects upon such retrieval is described above, under “instance +methods”. Class method objects are created by the built-in +"classmethod()" constructor. +''', + 'typesfunctions': r'''Functions +********* + +Function objects are created by function definitions. The only +operation on a function object is to call it: "func(argument-list)". + +There are really two flavors of function objects: built-in functions +and user-defined functions. Both support the same operation (to call +the function), but the implementation is different, hence the +different object types. + +See Function definitions for more information. +''', + 'typesmapping': r'''Mapping Types — "dict" +********************** + +A *mapping* object maps *hashable* values to arbitrary objects. +Mappings are mutable objects. There is currently only one standard +mapping type, the *dictionary*. (For other containers see the built- +in "list", "set", and "tuple" classes, and the "collections" module.) + +A dictionary’s keys are *almost* arbitrary values. Values that are +not *hashable*, that is, values containing lists, dictionaries or +other mutable types (that are compared by value rather than by object +identity) may not be used as keys. Values that compare equal (such as +"1", "1.0", and "True") can be used interchangeably to index the same +dictionary entry. + +class dict(**kwargs) +class dict(mapping, /, **kwargs) +class dict(iterable, /, **kwargs) + + Return a new dictionary initialized from an optional positional + argument and a possibly empty set of keyword arguments. + + Dictionaries can be created by several means: + + * Use a comma-separated list of "key: value" pairs within braces: + "{'jack': 4098, 'sjoerd': 4127}" or "{4098: 'jack', 4127: + 'sjoerd'}" + + * Use a dict comprehension: "{}", "{x: x ** 2 for x in range(10)}" + + * Use the type constructor: "dict()", "dict([('foo', 100), ('bar', + 200)])", "dict(foo=100, bar=200)" + + If no positional argument is given, an empty dictionary is created. + If a positional argument is given and it defines a "keys()" method, + a dictionary is created by calling "__getitem__()" on the argument + with each returned key from the method. Otherwise, the positional + argument must be an *iterable* object. Each item in the iterable + must itself be an iterable with exactly two elements. The first + element of each item becomes a key in the new dictionary, and the + second element the corresponding value. If a key occurs more than + once, the last value for that key becomes the corresponding value + in the new dictionary. + + If keyword arguments are given, the keyword arguments and their + values are added to the dictionary created from the positional + argument. If a key being added is already present, the value from + the keyword argument replaces the value from the positional + argument. + + Providing keyword arguments as in the first example only works for + keys that are valid Python identifiers. Otherwise, any valid keys + can be used. + + Dictionaries compare equal if and only if they have the same "(key, + value)" pairs (regardless of ordering). Order comparisons (‘<’, + ‘<=’, ‘>=’, ‘>’) raise "TypeError". To illustrate dictionary + creation and equality, the following examples all return a + dictionary equal to "{"one": 1, "two": 2, "three": 3}": + + >>> a = dict(one=1, two=2, three=3) + >>> b = {'one': 1, 'two': 2, 'three': 3} + >>> c = dict(zip(['one', 'two', 'three'], [1, 2, 3])) + >>> d = dict([('two', 2), ('one', 1), ('three', 3)]) + >>> e = dict({'three': 3, 'one': 1, 'two': 2}) + >>> f = dict({'one': 1, 'three': 3}, two=2) + >>> a == b == c == d == e == f + True + + Providing keyword arguments as in the first example only works for + keys that are valid Python identifiers. Otherwise, any valid keys + can be used. + + Dictionaries preserve insertion order. Note that updating a key + does not affect the order. Keys added after deletion are inserted + at the end. + + >>> d = {"one": 1, "two": 2, "three": 3, "four": 4} + >>> d + {'one': 1, 'two': 2, 'three': 3, 'four': 4} + >>> list(d) + ['one', 'two', 'three', 'four'] + >>> list(d.values()) + [1, 2, 3, 4] + >>> d["one"] = 42 + >>> d + {'one': 42, 'two': 2, 'three': 3, 'four': 4} + >>> del d["two"] + >>> d["two"] = None + >>> d + {'one': 42, 'three': 3, 'four': 4, 'two': None} + + Changed in version 3.7: Dictionary order is guaranteed to be + insertion order. This behavior was an implementation detail of + CPython from 3.6. + + These are the operations that dictionaries support (and therefore, + custom mapping types should support too): + + list(d) + + Return a list of all the keys used in the dictionary *d*. + + len(d) + + Return the number of items in the dictionary *d*. + + d[key] + + Return the item of *d* with key *key*. Raises a "KeyError" if + *key* is not in the map. + + If a subclass of dict defines a method "__missing__()" and *key* + is not present, the "d[key]" operation calls that method with + the key *key* as argument. The "d[key]" operation then returns + or raises whatever is returned or raised by the + "__missing__(key)" call. No other operations or methods invoke + "__missing__()". If "__missing__()" is not defined, "KeyError" + is raised. "__missing__()" must be a method; it cannot be an + instance variable: + + >>> class Counter(dict): + ... def __missing__(self, key): + ... return 0 + ... + >>> c = Counter() + >>> c['red'] + 0 + >>> c['red'] += 1 + >>> c['red'] + 1 + + The example above shows part of the implementation of + "collections.Counter". A different "__missing__()" method is + used by "collections.defaultdict". + + d[key] = value + + Set "d[key]" to *value*. + + del d[key] + + Remove "d[key]" from *d*. Raises a "KeyError" if *key* is not + in the map. + + key in d + + Return "True" if *d* has a key *key*, else "False". + + key not in d + + Equivalent to "not key in d". + + iter(d) + + Return an iterator over the keys of the dictionary. This is a + shortcut for "iter(d.keys())". + + clear() + + Remove all items from the dictionary. + + copy() + + Return a shallow copy of the dictionary. + + classmethod fromkeys(iterable, value=None, /) + + Create a new dictionary with keys from *iterable* and values set + to *value*. + + "fromkeys()" is a class method that returns a new dictionary. + *value* defaults to "None". All of the values refer to just a + single instance, so it generally doesn’t make sense for *value* + to be a mutable object such as an empty list. To get distinct + values, use a dict comprehension instead. + + get(key, default=None, /) + + Return the value for *key* if *key* is in the dictionary, else + *default*. If *default* is not given, it defaults to "None", so + that this method never raises a "KeyError". + + items() + + Return a new view of the dictionary’s items ("(key, value)" + pairs). See the documentation of view objects. + + keys() + + Return a new view of the dictionary’s keys. See the + documentation of view objects. + + pop(key, /) + pop(key, default, /) + + If *key* is in the dictionary, remove it and return its value, + else return *default*. If *default* is not given and *key* is + not in the dictionary, a "KeyError" is raised. + + popitem() + + Remove and return a "(key, value)" pair from the dictionary. + Pairs are returned in LIFO (last-in, first-out) order. + + "popitem()" is useful to destructively iterate over a + dictionary, as often used in set algorithms. If the dictionary + is empty, calling "popitem()" raises a "KeyError". + + Changed in version 3.7: LIFO order is now guaranteed. In prior + versions, "popitem()" would return an arbitrary key/value pair. + + reversed(d) + + Return a reverse iterator over the keys of the dictionary. This + is a shortcut for "reversed(d.keys())". + + Added in version 3.8. + + setdefault(key, default=None, /) + + If *key* is in the dictionary, return its value. If not, insert + *key* with a value of *default* and return *default*. *default* + defaults to "None". + + update(**kwargs) + update(mapping, /, **kwargs) + update(iterable, /, **kwargs) + + Update the dictionary with the key/value pairs from *mapping* or + *iterable* and *kwargs*, overwriting existing keys. Return + "None". + + "update()" accepts either another object with a "keys()" method + (in which case "__getitem__()" is called with every key returned + from the method) or an iterable of key/value pairs (as tuples or + other iterables of length two). If keyword arguments are + specified, the dictionary is then updated with those key/value + pairs: "d.update(red=1, blue=2)". + + values() + + Return a new view of the dictionary’s values. See the + documentation of view objects. + + An equality comparison between one "dict.values()" view and + another will always return "False". This also applies when + comparing "dict.values()" to itself: + + >>> d = {'a': 1} + >>> d.values() == d.values() + False + + d | other + + Create a new dictionary with the merged keys and values of *d* + and *other*, which must both be dictionaries. The values of + *other* take priority when *d* and *other* share keys. + + Added in version 3.9. + + d |= other + + Update the dictionary *d* with keys and values from *other*, + which may be either a *mapping* or an *iterable* of key/value + pairs. The values of *other* take priority when *d* and *other* + share keys. + + Added in version 3.9. + + Dictionaries and dictionary views are reversible. + + >>> d = {"one": 1, "two": 2, "three": 3, "four": 4} + >>> d + {'one': 1, 'two': 2, 'three': 3, 'four': 4} + >>> list(reversed(d)) + ['four', 'three', 'two', 'one'] + >>> list(reversed(d.values())) + [4, 3, 2, 1] + >>> list(reversed(d.items())) + [('four', 4), ('three', 3), ('two', 2), ('one', 1)] + + Changed in version 3.8: Dictionaries are now reversible. + +See also: + + "types.MappingProxyType" can be used to create a read-only view of a + "dict". + + +Dictionary view objects +======================= + +The objects returned by "dict.keys()", "dict.values()" and +"dict.items()" are *view objects*. They provide a dynamic view on the +dictionary’s entries, which means that when the dictionary changes, +the view reflects these changes. + +Dictionary views can be iterated over to yield their respective data, +and support membership tests: + +len(dictview) + + Return the number of entries in the dictionary. + +iter(dictview) + + Return an iterator over the keys, values or items (represented as + tuples of "(key, value)") in the dictionary. + + Keys and values are iterated over in insertion order. This allows + the creation of "(value, key)" pairs using "zip()": "pairs = + zip(d.values(), d.keys())". Another way to create the same list is + "pairs = [(v, k) for (k, v) in d.items()]". + + Iterating views while adding or deleting entries in the dictionary + may raise a "RuntimeError" or fail to iterate over all entries. + + Changed in version 3.7: Dictionary order is guaranteed to be + insertion order. + +x in dictview + + Return "True" if *x* is in the underlying dictionary’s keys, values + or items (in the latter case, *x* should be a "(key, value)" + tuple). + +reversed(dictview) + + Return a reverse iterator over the keys, values or items of the + dictionary. The view will be iterated in reverse order of the + insertion. + + Changed in version 3.8: Dictionary views are now reversible. + +dictview.mapping + + Return a "types.MappingProxyType" that wraps the original + dictionary to which the view refers. + + Added in version 3.10. + +Keys views are set-like since their entries are unique and *hashable*. +Items views also have set-like operations since the (key, value) pairs +are unique and the keys are hashable. If all values in an items view +are hashable as well, then the items view can interoperate with other +sets. (Values views are not treated as set-like since the entries are +generally not unique.) For set-like views, all of the operations +defined for the abstract base class "collections.abc.Set" are +available (for example, "==", "<", or "^"). While using set +operators, set-like views accept any iterable as the other operand, +unlike sets which only accept sets as the input. + +An example of dictionary view usage: + + >>> dishes = {'eggs': 2, 'sausage': 1, 'bacon': 1, 'spam': 500} + >>> keys = dishes.keys() + >>> values = dishes.values() + + >>> # iteration + >>> n = 0 + >>> for val in values: + ... n += val + ... + >>> print(n) + 504 + + >>> # keys and values are iterated over in the same order (insertion order) + >>> list(keys) + ['eggs', 'sausage', 'bacon', 'spam'] + >>> list(values) + [2, 1, 1, 500] + + >>> # view objects are dynamic and reflect dict changes + >>> del dishes['eggs'] + >>> del dishes['sausage'] + >>> list(keys) + ['bacon', 'spam'] + + >>> # set operations + >>> keys & {'eggs', 'bacon', 'salad'} + {'bacon'} + >>> keys ^ {'sausage', 'juice'} == {'juice', 'sausage', 'bacon', 'spam'} + True + >>> keys | ['juice', 'juice', 'juice'] == {'bacon', 'spam', 'juice'} + True + + >>> # get back a read-only proxy for the original dictionary + >>> values.mapping + mappingproxy({'bacon': 1, 'spam': 500}) + >>> values.mapping['spam'] + 500 +''', + 'typesmethods': r'''Methods +******* + +Methods are functions that are called using the attribute notation. +There are two flavors: built-in methods (such as "append()" on lists) +and class instance method. Built-in methods are described with the +types that support them. + +If you access a method (a function defined in a class namespace) +through an instance, you get a special object: a *bound method* (also +called instance method) object. When called, it will add the "self" +argument to the argument list. Bound methods have two special read- +only attributes: "m.__self__" is the object on which the method +operates, and "m.__func__" is the function implementing the method. +Calling "m(arg-1, arg-2, ..., arg-n)" is completely equivalent to +calling "m.__func__(m.__self__, arg-1, arg-2, ..., arg-n)". + +Like function objects, bound method objects support getting arbitrary +attributes. However, since method attributes are actually stored on +the underlying function object ("method.__func__"), setting method +attributes on bound methods is disallowed. Attempting to set an +attribute on a method results in an "AttributeError" being raised. In +order to set a method attribute, you need to explicitly set it on the +underlying function object: + + >>> class C: + ... def method(self): + ... pass + ... + >>> c = C() + >>> c.method.whoami = 'my name is method' # can't set on the method + Traceback (most recent call last): + File "", line 1, in + AttributeError: 'method' object has no attribute 'whoami' + >>> c.method.__func__.whoami = 'my name is method' + >>> c.method.whoami + 'my name is method' + +See Instance methods for more information. +''', + 'typesmodules': r'''Modules +******* + +The only special operation on a module is attribute access: "m.name", +where *m* is a module and *name* accesses a name defined in *m*’s +symbol table. Module attributes can be assigned to. (Note that the +"import" statement is not, strictly speaking, an operation on a module +object; "import foo" does not require a module object named *foo* to +exist, rather it requires an (external) *definition* for a module +named *foo* somewhere.) + +A special attribute of every module is "__dict__". This is the +dictionary containing the module’s symbol table. Modifying this +dictionary will actually change the module’s symbol table, but direct +assignment to the "__dict__" attribute is not possible (you can write +"m.__dict__['a'] = 1", which defines "m.a" to be "1", but you can’t +write "m.__dict__ = {}"). Modifying "__dict__" directly is not +recommended. + +Modules built into the interpreter are written like this: "". If loaded from a file, they are written as +"". +''', + 'typesseq': r'''Sequence Types — "list", "tuple", "range" +***************************************** + +There are three basic sequence types: lists, tuples, and range +objects. Additional sequence types tailored for processing of binary +data and text strings are described in dedicated sections. + + +Common Sequence Operations +========================== + +The operations in the following table are supported by most sequence +types, both mutable and immutable. The "collections.abc.Sequence" ABC +is provided to make it easier to correctly implement these operations +on custom sequence types. + +This table lists the sequence operations sorted in ascending priority. +In the table, *s* and *t* are sequences of the same type, *n*, *i*, +*j* and *k* are integers and *x* is an arbitrary object that meets any +type and value restrictions imposed by *s*. + +The "in" and "not in" operations have the same priorities as the +comparison operations. The "+" (concatenation) and "*" (repetition) +operations have the same priority as the corresponding numeric +operations. [3] + ++----------------------------+----------------------------------+------------+ +| Operation | Result | Notes | +|============================|==================================|============| +| "x in s" | "True" if an item of *s* is | (1) | +| | equal to *x*, else "False" | | ++----------------------------+----------------------------------+------------+ +| "x not in s" | "False" if an item of *s* is | (1) | +| | equal to *x*, else "True" | | ++----------------------------+----------------------------------+------------+ +| "s + t" | the concatenation of *s* and *t* | (6)(7) | ++----------------------------+----------------------------------+------------+ +| "s * n" or "n * s" | equivalent to adding *s* to | (2)(7) | +| | itself *n* times | | ++----------------------------+----------------------------------+------------+ +| "s[i]" | *i*th item of *s*, origin 0 | (3)(8) | ++----------------------------+----------------------------------+------------+ +| "s[i:j]" | slice of *s* from *i* to *j* | (3)(4) | ++----------------------------+----------------------------------+------------+ +| "s[i:j:k]" | slice of *s* from *i* to *j* | (3)(5) | +| | with step *k* | | ++----------------------------+----------------------------------+------------+ +| "len(s)" | length of *s* | | ++----------------------------+----------------------------------+------------+ +| "min(s)" | smallest item of *s* | | ++----------------------------+----------------------------------+------------+ +| "max(s)" | largest item of *s* | | ++----------------------------+----------------------------------+------------+ + +Sequences of the same type also support comparisons. In particular, +tuples and lists are compared lexicographically by comparing +corresponding elements. This means that to compare equal, every +element must compare equal and the two sequences must be of the same +type and have the same length. (For full details see Comparisons in +the language reference.) + +Forward and reversed iterators over mutable sequences access values +using an index. That index will continue to march forward (or +backward) even if the underlying sequence is mutated. The iterator +terminates only when an "IndexError" or a "StopIteration" is +encountered (or when the index drops below zero). + +Notes: + +1. While the "in" and "not in" operations are used only for simple + containment testing in the general case, some specialised sequences + (such as "str", "bytes" and "bytearray") also use them for + subsequence testing: + + >>> "gg" in "eggs" + True + +2. Values of *n* less than "0" are treated as "0" (which yields an + empty sequence of the same type as *s*). Note that items in the + sequence *s* are not copied; they are referenced multiple times. + This often haunts new Python programmers; consider: + + >>> lists = [[]] * 3 + >>> lists + [[], [], []] + >>> lists[0].append(3) + >>> lists + [[3], [3], [3]] + + What has happened is that "[[]]" is a one-element list containing + an empty list, so all three elements of "[[]] * 3" are references + to this single empty list. Modifying any of the elements of + "lists" modifies this single list. You can create a list of + different lists this way: + + >>> lists = [[] for i in range(3)] + >>> lists[0].append(3) + >>> lists[1].append(5) + >>> lists[2].append(7) + >>> lists + [[3], [5], [7]] + + Further explanation is available in the FAQ entry How do I create a + multidimensional list?. + +3. If *i* or *j* is negative, the index is relative to the end of + sequence *s*: "len(s) + i" or "len(s) + j" is substituted. But + note that "-0" is still "0". + +4. The slice of *s* from *i* to *j* is defined as the sequence of + items with index *k* such that "i <= k < j". If *i* or *j* is + greater than "len(s)", use "len(s)". If *i* is omitted or "None", + use "0". If *j* is omitted or "None", use "len(s)". If *i* is + greater than or equal to *j*, the slice is empty. + +5. The slice of *s* from *i* to *j* with step *k* is defined as the + sequence of items with index "x = i + n*k" such that "0 <= n < + (j-i)/k". In other words, the indices are "i", "i+k", "i+2*k", + "i+3*k" and so on, stopping when *j* is reached (but never + including *j*). When *k* is positive, *i* and *j* are reduced to + "len(s)" if they are greater. When *k* is negative, *i* and *j* are + reduced to "len(s) - 1" if they are greater. If *i* or *j* are + omitted or "None", they become “end” values (which end depends on + the sign of *k*). Note, *k* cannot be zero. If *k* is "None", it + is treated like "1". + +6. Concatenating immutable sequences always results in a new object. + This means that building up a sequence by repeated concatenation + will have a quadratic runtime cost in the total sequence length. + To get a linear runtime cost, you must switch to one of the + alternatives below: + + * if concatenating "str" objects, you can build a list and use + "str.join()" at the end or else write to an "io.StringIO" + instance and retrieve its value when complete + + * if concatenating "bytes" objects, you can similarly use + "bytes.join()" or "io.BytesIO", or you can do in-place + concatenation with a "bytearray" object. "bytearray" objects are + mutable and have an efficient overallocation mechanism + + * if concatenating "tuple" objects, extend a "list" instead + + * for other types, investigate the relevant class documentation + +7. Some sequence types (such as "range") only support item sequences + that follow specific patterns, and hence don’t support sequence + concatenation or repetition. + +8. An "IndexError" is raised if *i* is outside the sequence range. + +-[ Sequence Methods ]- + +Sequence types also support the following methods: + +sequence.count(value, /) + + Return the total number of occurrences of *value* in *sequence*. + +sequence.index(value[, start[, stop]) + + Return the index of the first occurrence of *value* in *sequence*. + + Raises "ValueError" if *value* is not found in *sequence*. + + The *start* or *stop* arguments allow for efficient searching of + subsections of the sequence, beginning at *start* and ending at + *stop*. This is roughly equivalent to "start + + sequence[start:stop].index(value)", only without copying any data. + + Caution: + + Not all sequence types support passing the *start* and *stop* + arguments. + + +Immutable Sequence Types +======================== + +The only operation that immutable sequence types generally implement +that is not also implemented by mutable sequence types is support for +the "hash()" built-in. + +This support allows immutable sequences, such as "tuple" instances, to +be used as "dict" keys and stored in "set" and "frozenset" instances. + +Attempting to hash an immutable sequence that contains unhashable +values will result in "TypeError". + + +Mutable Sequence Types +====================== + +The operations in the following table are defined on mutable sequence +types. The "collections.abc.MutableSequence" ABC is provided to make +it easier to correctly implement these operations on custom sequence +types. + +In the table *s* is an instance of a mutable sequence type, *t* is any +iterable object and *x* is an arbitrary object that meets any type and +value restrictions imposed by *s* (for example, "bytearray" only +accepts integers that meet the value restriction "0 <= x <= 255"). + ++--------------------------------+----------------------------------+-----------------------+ +| Operation | Result | Notes | +|================================|==================================|=======================| +| "s[i] = x" | item *i* of *s* is replaced by | | +| | *x* | | ++--------------------------------+----------------------------------+-----------------------+ +| "del s[i]" | removes item *i* of *s* | | ++--------------------------------+----------------------------------+-----------------------+ +| "s[i:j] = t" | slice of *s* from *i* to *j* is | | +| | replaced by the contents of the | | +| | iterable *t* | | ++--------------------------------+----------------------------------+-----------------------+ +| "del s[i:j]" | removes the elements of "s[i:j]" | | +| | from the list (same as "s[i:j] = | | +| | []") | | ++--------------------------------+----------------------------------+-----------------------+ +| "s[i:j:k] = t" | the elements of "s[i:j:k]" are | (1) | +| | replaced by those of *t* | | ++--------------------------------+----------------------------------+-----------------------+ +| "del s[i:j:k]" | removes the elements of | | +| | "s[i:j:k]" from the list | | ++--------------------------------+----------------------------------+-----------------------+ +| "s += t" | extends *s* with the contents of | | +| | *t* (for the most part the same | | +| | as "s[len(s):len(s)] = t") | | ++--------------------------------+----------------------------------+-----------------------+ +| "s *= n" | updates *s* with its contents | (2) | +| | repeated *n* times | | ++--------------------------------+----------------------------------+-----------------------+ + +Notes: + +1. If *k* is not equal to "1", *t* must have the same length as the + slice it is replacing. + +2. The value *n* is an integer, or an object implementing + "__index__()". Zero and negative values of *n* clear the sequence. + Items in the sequence are not copied; they are referenced multiple + times, as explained for "s * n" under Common Sequence Operations. + +-[ Mutable Sequence Methods ]- + +Mutable sequence types also support the following methods: + +sequence.append(value, /) + + Append *value* to the end of the sequence This is equivalent to + writing "seq[len(seq):len(seq)] = [value]". + +sequence.clear() + + Added in version 3.3. + + Remove all items from *sequence*. This is equivalent to writing + "del sequence[:]". + +sequence.copy() + + Added in version 3.3. + + Create a shallow copy of *sequence*. This is equivalent to writing + "sequence[:]". + + Hint: + + The "copy()" method is not part of the "MutableSequence" "ABC", + but most concrete mutable sequence types provide it. + +sequence.extend(iterable, /) + + Extend *sequence* with the contents of *iterable*. For the most + part, this is the same as writing "seq[len(seq):len(seq)] = + iterable". + +sequence.insert(index, value, /) + + Insert *value* into *sequence* at the given *index*. This is + equivalent to writing "sequence[index:index] = [value]". + +sequence.pop(index=-1, /) + + Retrieve the item at *index* and also removes it from *sequence*. + By default, the last item in *sequence* is removed and returned. + +sequence.remove(value, /) + + Remove the first item from *sequence* where "sequence[i] == value". + + Raises "ValueError" if *value* is not found in *sequence*. + +sequence.reverse() + + Reverse the items of *sequence* in place. This method maintains + economy of space when reversing a large sequence. To remind users + that it operates by side-effect, it returns "None". + + +Lists +===== + +Lists are mutable sequences, typically used to store collections of +homogeneous items (where the precise degree of similarity will vary by +application). + +class list(iterable=(), /) + + Lists may be constructed in several ways: + + * Using a pair of square brackets to denote the empty list: "[]" + + * Using square brackets, separating items with commas: "[a]", "[a, + b, c]" + + * Using a list comprehension: "[x for x in iterable]" + + * Using the type constructor: "list()" or "list(iterable)" + + The constructor builds a list whose items are the same and in the + same order as *iterable*’s items. *iterable* may be either a + sequence, a container that supports iteration, or an iterator + object. If *iterable* is already a list, a copy is made and + returned, similar to "iterable[:]". For example, "list('abc')" + returns "['a', 'b', 'c']" and "list( (1, 2, 3) )" returns "[1, 2, + 3]". If no argument is given, the constructor creates a new empty + list, "[]". + + Many other operations also produce lists, including the "sorted()" + built-in. + + Lists implement all of the common and mutable sequence operations. + Lists also provide the following additional method: + + sort(*, key=None, reverse=False) + + This method sorts the list in place, using only "<" comparisons + between items. Exceptions are not suppressed - if any comparison + operations fail, the entire sort operation will fail (and the + list will likely be left in a partially modified state). + + "sort()" accepts two arguments that can only be passed by + keyword (keyword-only arguments): + + *key* specifies a function of one argument that is used to + extract a comparison key from each list element (for example, + "key=str.lower"). The key corresponding to each item in the list + is calculated once and then used for the entire sorting process. + The default value of "None" means that list items are sorted + directly without calculating a separate key value. + + The "functools.cmp_to_key()" utility is available to convert a + 2.x style *cmp* function to a *key* function. + + *reverse* is a boolean value. If set to "True", then the list + elements are sorted as if each comparison were reversed. + + This method modifies the sequence in place for economy of space + when sorting a large sequence. To remind users that it operates + by side effect, it does not return the sorted sequence (use + "sorted()" to explicitly request a new sorted list instance). + + The "sort()" method is guaranteed to be stable. A sort is + stable if it guarantees not to change the relative order of + elements that compare equal — this is helpful for sorting in + multiple passes (for example, sort by department, then by salary + grade). + + For sorting examples and a brief sorting tutorial, see Sorting + Techniques. + + **CPython implementation detail:** While a list is being sorted, + the effect of attempting to mutate, or even inspect, the list is + undefined. The C implementation of Python makes the list appear + empty for the duration, and raises "ValueError" if it can detect + that the list has been mutated during a sort. + + +Tuples +====== + +Tuples are immutable sequences, typically used to store collections of +heterogeneous data (such as the 2-tuples produced by the "enumerate()" +built-in). Tuples are also used for cases where an immutable sequence +of homogeneous data is needed (such as allowing storage in a "set" or +"dict" instance). + +class tuple(iterable=(), /) + + Tuples may be constructed in a number of ways: + + * Using a pair of parentheses to denote the empty tuple: "()" + + * Using a trailing comma for a singleton tuple: "a," or "(a,)" + + * Separating items with commas: "a, b, c" or "(a, b, c)" + + * Using the "tuple()" built-in: "tuple()" or "tuple(iterable)" + + The constructor builds a tuple whose items are the same and in the + same order as *iterable*’s items. *iterable* may be either a + sequence, a container that supports iteration, or an iterator + object. If *iterable* is already a tuple, it is returned + unchanged. For example, "tuple('abc')" returns "('a', 'b', 'c')" + and "tuple( [1, 2, 3] )" returns "(1, 2, 3)". If no argument is + given, the constructor creates a new empty tuple, "()". + + Note that it is actually the comma which makes a tuple, not the + parentheses. The parentheses are optional, except in the empty + tuple case, or when they are needed to avoid syntactic ambiguity. + For example, "f(a, b, c)" is a function call with three arguments, + while "f((a, b, c))" is a function call with a 3-tuple as the sole + argument. + + Tuples implement all of the common sequence operations. + +For heterogeneous collections of data where access by name is clearer +than access by index, "collections.namedtuple()" may be a more +appropriate choice than a simple tuple object. + + +Ranges +====== + +The "range" type represents an immutable sequence of numbers and is +commonly used for looping a specific number of times in "for" loops. + +class range(stop, /) +class range(start, stop, step=1, /) + + The arguments to the range constructor must be integers (either + built-in "int" or any object that implements the "__index__()" + special method). If the *step* argument is omitted, it defaults to + "1". If the *start* argument is omitted, it defaults to "0". If + *step* is zero, "ValueError" is raised. + + For a positive *step*, the contents of a range "r" are determined + by the formula "r[i] = start + step*i" where "i >= 0" and "r[i] < + stop". + + For a negative *step*, the contents of the range are still + determined by the formula "r[i] = start + step*i", but the + constraints are "i >= 0" and "r[i] > stop". + + A range object will be empty if "r[0]" does not meet the value + constraint. Ranges do support negative indices, but these are + interpreted as indexing from the end of the sequence determined by + the positive indices. + + Ranges containing absolute values larger than "sys.maxsize" are + permitted but some features (such as "len()") may raise + "OverflowError". + + Range examples: + + >>> list(range(10)) + [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] + >>> list(range(1, 11)) + [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] + >>> list(range(0, 30, 5)) + [0, 5, 10, 15, 20, 25] + >>> list(range(0, 10, 3)) + [0, 3, 6, 9] + >>> list(range(0, -10, -1)) + [0, -1, -2, -3, -4, -5, -6, -7, -8, -9] + >>> list(range(0)) + [] + >>> list(range(1, 0)) + [] + + Ranges implement all of the common sequence operations except + concatenation and repetition (due to the fact that range objects + can only represent sequences that follow a strict pattern and + repetition and concatenation will usually violate that pattern). + + start + + The value of the *start* parameter (or "0" if the parameter was + not supplied) + + stop + + The value of the *stop* parameter + + step + + The value of the *step* parameter (or "1" if the parameter was + not supplied) + +The advantage of the "range" type over a regular "list" or "tuple" is +that a "range" object will always take the same (small) amount of +memory, no matter the size of the range it represents (as it only +stores the "start", "stop" and "step" values, calculating individual +items and subranges as needed). + +Range objects implement the "collections.abc.Sequence" ABC, and +provide features such as containment tests, element index lookup, +slicing and support for negative indices (see Sequence Types — list, +tuple, range): + +>>> r = range(0, 20, 2) +>>> r +range(0, 20, 2) +>>> 11 in r +False +>>> 10 in r +True +>>> r.index(10) +5 +>>> r[5] +10 +>>> r[:5] +range(0, 10, 2) +>>> r[-1] +18 + +Testing range objects for equality with "==" and "!=" compares them as +sequences. That is, two range objects are considered equal if they +represent the same sequence of values. (Note that two range objects +that compare equal might have different "start", "stop" and "step" +attributes, for example "range(0) == range(2, 1, 3)" or "range(0, 3, +2) == range(0, 4, 2)".) + +Changed in version 3.2: Implement the Sequence ABC. Support slicing +and negative indices. Test "int" objects for membership in constant +time instead of iterating through all items. + +Changed in version 3.3: Define ‘==’ and ‘!=’ to compare range objects +based on the sequence of values they define (instead of comparing +based on object identity).Added the "start", "stop" and "step" +attributes. + +See also: + + * The linspace recipe shows how to implement a lazy version of range + suitable for floating-point applications. +''', + 'typesseq-mutable': r'''Mutable Sequence Types +********************** + +The operations in the following table are defined on mutable sequence +types. The "collections.abc.MutableSequence" ABC is provided to make +it easier to correctly implement these operations on custom sequence +types. + +In the table *s* is an instance of a mutable sequence type, *t* is any +iterable object and *x* is an arbitrary object that meets any type and +value restrictions imposed by *s* (for example, "bytearray" only +accepts integers that meet the value restriction "0 <= x <= 255"). + ++--------------------------------+----------------------------------+-----------------------+ +| Operation | Result | Notes | +|================================|==================================|=======================| +| "s[i] = x" | item *i* of *s* is replaced by | | +| | *x* | | ++--------------------------------+----------------------------------+-----------------------+ +| "del s[i]" | removes item *i* of *s* | | ++--------------------------------+----------------------------------+-----------------------+ +| "s[i:j] = t" | slice of *s* from *i* to *j* is | | +| | replaced by the contents of the | | +| | iterable *t* | | ++--------------------------------+----------------------------------+-----------------------+ +| "del s[i:j]" | removes the elements of "s[i:j]" | | +| | from the list (same as "s[i:j] = | | +| | []") | | ++--------------------------------+----------------------------------+-----------------------+ +| "s[i:j:k] = t" | the elements of "s[i:j:k]" are | (1) | +| | replaced by those of *t* | | ++--------------------------------+----------------------------------+-----------------------+ +| "del s[i:j:k]" | removes the elements of | | +| | "s[i:j:k]" from the list | | ++--------------------------------+----------------------------------+-----------------------+ +| "s += t" | extends *s* with the contents of | | +| | *t* (for the most part the same | | +| | as "s[len(s):len(s)] = t") | | ++--------------------------------+----------------------------------+-----------------------+ +| "s *= n" | updates *s* with its contents | (2) | +| | repeated *n* times | | ++--------------------------------+----------------------------------+-----------------------+ + +Notes: + +1. If *k* is not equal to "1", *t* must have the same length as the + slice it is replacing. + +2. The value *n* is an integer, or an object implementing + "__index__()". Zero and negative values of *n* clear the sequence. + Items in the sequence are not copied; they are referenced multiple + times, as explained for "s * n" under Common Sequence Operations. + +-[ Mutable Sequence Methods ]- + +Mutable sequence types also support the following methods: + +sequence.append(value, /) + + Append *value* to the end of the sequence This is equivalent to + writing "seq[len(seq):len(seq)] = [value]". + +sequence.clear() + + Added in version 3.3. + + Remove all items from *sequence*. This is equivalent to writing + "del sequence[:]". + +sequence.copy() + + Added in version 3.3. + + Create a shallow copy of *sequence*. This is equivalent to writing + "sequence[:]". + + Hint: + + The "copy()" method is not part of the "MutableSequence" "ABC", + but most concrete mutable sequence types provide it. + +sequence.extend(iterable, /) + + Extend *sequence* with the contents of *iterable*. For the most + part, this is the same as writing "seq[len(seq):len(seq)] = + iterable". + +sequence.insert(index, value, /) + + Insert *value* into *sequence* at the given *index*. This is + equivalent to writing "sequence[index:index] = [value]". + +sequence.pop(index=-1, /) + + Retrieve the item at *index* and also removes it from *sequence*. + By default, the last item in *sequence* is removed and returned. + +sequence.remove(value, /) + + Remove the first item from *sequence* where "sequence[i] == value". + + Raises "ValueError" if *value* is not found in *sequence*. + +sequence.reverse() + + Reverse the items of *sequence* in place. This method maintains + economy of space when reversing a large sequence. To remind users + that it operates by side-effect, it returns "None". +''', + 'unary': r'''Unary arithmetic and bitwise operations +*************************************** + +All unary arithmetic and bitwise operations have the same priority: + + u_expr: power | "-" u_expr | "+" u_expr | "~" u_expr + +The unary "-" (minus) operator yields the negation of its numeric +argument; the operation can be overridden with the "__neg__()" special +method. + +The unary "+" (plus) operator yields its numeric argument unchanged; +the operation can be overridden with the "__pos__()" special method. + +The unary "~" (invert) operator yields the bitwise inversion of its +integer argument. The bitwise inversion of "x" is defined as +"-(x+1)". It only applies to integral numbers or to custom objects +that override the "__invert__()" special method. + +In all three cases, if the argument does not have the proper type, a +"TypeError" exception is raised. +''', + 'while': r'''The "while" statement +********************* + +The "while" statement is used for repeated execution as long as an +expression is true: + + while_stmt: "while" assignment_expression ":" suite + ["else" ":" suite] + +This repeatedly tests the expression and, if it is true, executes the +first suite; if the expression is false (which may be the first time +it is tested) the suite of the "else" clause, if present, is executed +and the loop terminates. + +A "break" statement executed in the first suite terminates the loop +without executing the "else" clause’s suite. A "continue" statement +executed in the first suite skips the rest of the suite and goes back +to testing the expression. +''', + 'with': r'''The "with" statement +******************** + +The "with" statement is used to wrap the execution of a block with +methods defined by a context manager (see section With Statement +Context Managers). This allows common "try"…"except"…"finally" usage +patterns to be encapsulated for convenient reuse. + + with_stmt: "with" ( "(" with_stmt_contents ","? ")" | with_stmt_contents ) ":" suite + with_stmt_contents: with_item ("," with_item)* + with_item: expression ["as" target] + +The execution of the "with" statement with one “item” proceeds as +follows: + +1. The context expression (the expression given in the "with_item") is + evaluated to obtain a context manager. + +2. The context manager’s "__enter__()" is loaded for later use. + +3. The context manager’s "__exit__()" is loaded for later use. + +4. The context manager’s "__enter__()" method is invoked. + +5. If a target was included in the "with" statement, the return value + from "__enter__()" is assigned to it. + + Note: + + The "with" statement guarantees that if the "__enter__()" method + returns without an error, then "__exit__()" will always be + called. Thus, if an error occurs during the assignment to the + target list, it will be treated the same as an error occurring + within the suite would be. See step 7 below. + +6. The suite is executed. + +7. The context manager’s "__exit__()" method is invoked. If an + exception caused the suite to be exited, its type, value, and + traceback are passed as arguments to "__exit__()". Otherwise, three + "None" arguments are supplied. + + If the suite was exited due to an exception, and the return value + from the "__exit__()" method was false, the exception is reraised. + If the return value was true, the exception is suppressed, and + execution continues with the statement following the "with" + statement. + + If the suite was exited for any reason other than an exception, the + return value from "__exit__()" is ignored, and execution proceeds + at the normal location for the kind of exit that was taken. + +The following code: + + with EXPRESSION as TARGET: + SUITE + +is semantically equivalent to: + + manager = (EXPRESSION) + enter = type(manager).__enter__ + exit = type(manager).__exit__ + value = enter(manager) + hit_except = False + + try: + TARGET = value + SUITE + except: + hit_except = True + if not exit(manager, *sys.exc_info()): + raise + finally: + if not hit_except: + exit(manager, None, None, None) + +With more than one item, the context managers are processed as if +multiple "with" statements were nested: + + with A() as a, B() as b: + SUITE + +is semantically equivalent to: + + with A() as a: + with B() as b: + SUITE + +You can also write multi-item context managers in multiple lines if +the items are surrounded by parentheses. For example: + + with ( + A() as a, + B() as b, + ): + SUITE + +Changed in version 3.1: Support for multiple context expressions. + +Changed in version 3.10: Support for using grouping parentheses to +break the statement in multiple lines. + +See also: + + **PEP 343** - The “with” statement + The specification, background, and examples for the Python "with" + statement. +''', + 'yield': r'''The "yield" statement +********************* + + yield_stmt: yield_expression + +A "yield" statement is semantically equivalent to a yield expression. +The "yield" statement can be used to omit the parentheses that would +otherwise be required in the equivalent yield expression statement. +For example, the yield statements + + yield + yield from + +are equivalent to the yield expression statements + + (yield ) + (yield from ) + +Yield expressions and statements are only used when defining a +*generator* function, and are only used in the body of the generator +function. Using "yield" in a function definition is sufficient to +cause that definition to create a generator function instead of a +normal function. + +For full details of "yield" semantics, refer to the Yield expressions +section. +''', +} diff --git a/Lib/quopri.py b/Lib/quopri.py old mode 100755 new mode 100644 index f36cf7b3951..129fd2f5c7c --- a/Lib/quopri.py +++ b/Lib/quopri.py @@ -1,5 +1,3 @@ -#! /usr/bin/env python3 - """Conversions to/from quoted-printable transport encoding as per RFC 1521.""" # (Dec 1991 version). diff --git a/Lib/re/__init__.py b/Lib/re/__init__.py index 7e8abbf6ffe..af2808a77da 100644 --- a/Lib/re/__init__.py +++ b/Lib/re/__init__.py @@ -61,7 +61,7 @@ resulting RE will match the second character. \number Matches the contents of the group of the same number. \A Matches only at the start of the string. - \Z Matches only at the end of the string. + \z Matches only at the end of the string. \b Matches the empty string, but only at the start or end of a word. \B Matches the empty string, but not at the start or end of a word. \d Matches any decimal digit; equivalent to the set [0-9] in diff --git a/Lib/re/_compiler.py b/Lib/re/_compiler.py index 1b1aaa7714b..20dd561d1c1 100644 --- a/Lib/re/_compiler.py +++ b/Lib/re/_compiler.py @@ -28,6 +28,8 @@ POSSESSIVE_REPEAT: (POSSESSIVE_REPEAT, SUCCESS, POSSESSIVE_REPEAT_ONE), } +_CHARSET_ALL = [(NEGATE, None)] + def _combine_flags(flags, add_flags, del_flags, TYPE_FLAGS=_parser.TYPE_FLAGS): if add_flags & TYPE_FLAGS: @@ -84,17 +86,22 @@ def _compile(code, pattern, flags): code[skip] = _len(code) - skip elif op is IN: charset, hascased = _optimize_charset(av, iscased, tolower, fixes) - if flags & SRE_FLAG_IGNORECASE and flags & SRE_FLAG_LOCALE: - emit(IN_LOC_IGNORE) - elif not hascased: - emit(IN) - elif not fixes: # ascii - emit(IN_IGNORE) + if not charset: + emit(FAILURE) + elif charset == _CHARSET_ALL: + emit(ANY_ALL) else: - emit(IN_UNI_IGNORE) - skip = _len(code); emit(0) - _compile_charset(charset, flags, code) - code[skip] = _len(code) - skip + if flags & SRE_FLAG_IGNORECASE and flags & SRE_FLAG_LOCALE: + emit(IN_LOC_IGNORE) + elif not hascased: + emit(IN) + elif not fixes: # ascii + emit(IN_IGNORE) + else: + emit(IN_UNI_IGNORE) + skip = _len(code); emit(0) + _compile_charset(charset, flags, code) + code[skip] = _len(code) - skip elif op is ANY: if flags & SRE_FLAG_DOTALL: emit(ANY_ALL) @@ -277,6 +284,10 @@ def _optimize_charset(charset, iscased=None, fixup=None, fixes=None): charmap[i] = 1 elif op is NEGATE: out.append((op, av)) + elif op is CATEGORY and tail and (CATEGORY, CH_NEGATE[av]) in tail: + # Optimize [\s\S] etc. + out = [] if out else _CHARSET_ALL + return out, False else: tail.append((op, av)) except IndexError: @@ -524,13 +535,18 @@ def _compile_info(code, pattern, flags): # look for a literal prefix prefix = [] prefix_skip = 0 - charset = [] # not used + charset = None # not used if not (flags & SRE_FLAG_IGNORECASE and flags & SRE_FLAG_LOCALE): # look for literal prefix prefix, prefix_skip, got_all = _get_literal_prefix(pattern, flags) # if no prefix, look for charset prefix if not prefix: charset = _get_charset_prefix(pattern, flags) + if charset: + charset, hascased = _optimize_charset(charset) + assert not hascased + if charset == _CHARSET_ALL: + charset = None ## if prefix: ## print("*** PREFIX", prefix, prefix_skip) ## if charset: @@ -565,8 +581,6 @@ def _compile_info(code, pattern, flags): # generate overlap table code.extend(_generate_overlap_table(prefix)) elif charset: - charset, hascased = _optimize_charset(charset) - assert not hascased _compile_charset(charset, flags, code) code[skip] = len(code) - skip diff --git a/Lib/re/_constants.py b/Lib/re/_constants.py index 9c3c294ba44..d6f32302d37 100644 --- a/Lib/re/_constants.py +++ b/Lib/re/_constants.py @@ -15,7 +15,7 @@ MAGIC = 20230612 -from _sre import MAXREPEAT, MAXGROUPS +from _sre import MAXREPEAT, MAXGROUPS # noqa: F401 # SRE standard exception (access as sre.error) # should this really be here? @@ -206,6 +206,8 @@ def _makecodes(*names): CATEGORY_NOT_LINEBREAK: CATEGORY_UNI_NOT_LINEBREAK } +CH_NEGATE = dict(zip(CHCODES[::2] + CHCODES[1::2], CHCODES[1::2] + CHCODES[::2])) + # flags SRE_FLAG_IGNORECASE = 2 # case insensitive SRE_FLAG_LOCALE = 4 # honour system locale diff --git a/Lib/re/_parser.py b/Lib/re/_parser.py index f3c779340fe..35ab7ede2a7 100644 --- a/Lib/re/_parser.py +++ b/Lib/re/_parser.py @@ -49,7 +49,8 @@ r"\S": (IN, [(CATEGORY, CATEGORY_NOT_SPACE)]), r"\w": (IN, [(CATEGORY, CATEGORY_WORD)]), r"\W": (IN, [(CATEGORY, CATEGORY_NOT_WORD)]), - r"\Z": (AT, AT_END_STRING), # end of string + r"\z": (AT, AT_END_STRING), # end of string + r"\Z": (AT, AT_END_STRING), # end of string (obsolete) } FLAGS = { @@ -807,14 +808,6 @@ def _parse(source, state, verbose, nested, first=False): state.grouprefpos[condgroup] = ( source.tell() - len(condname) - 1 ) - if not (condname.isdecimal() and condname.isascii()): - import warnings - warnings.warn( - "bad character in group name %s at position %d" % - (repr(condname) if source.istext else ascii(condname), - source.tell() - len(condname) - 1), - DeprecationWarning, stacklevel=nested + 6 - ) state.checklookbehindgroup(condgroup, source) item_yes = _parse(source, state, verbose, nested + 1) if source.match("|"): @@ -1038,14 +1031,6 @@ def addgroup(index, pos): if index >= MAXGROUPS: raise s.error("invalid group reference %d" % index, len(name) + 1) - if not (name.isdecimal() and name.isascii()): - import warnings - warnings.warn( - "bad character in group name %s at position %d" % - (repr(name) if s.istext else ascii(name), - s.tell() - len(name) - 1), - DeprecationWarning, stacklevel=5 - ) addgroup(index, len(name) + 1) elif c == "0": if s.next in OCTDIGITS: diff --git a/Lib/reprlib.py b/Lib/reprlib.py index 19dbe3a07eb..ab18247682b 100644 --- a/Lib/reprlib.py +++ b/Lib/reprlib.py @@ -28,7 +28,7 @@ def wrapper(self): wrapper.__doc__ = getattr(user_function, '__doc__') wrapper.__name__ = getattr(user_function, '__name__') wrapper.__qualname__ = getattr(user_function, '__qualname__') - wrapper.__annotations__ = getattr(user_function, '__annotations__', {}) + wrapper.__annotate__ = getattr(user_function, '__annotate__', None) wrapper.__type_params__ = getattr(user_function, '__type_params__', ()) wrapper.__wrapped__ = user_function return wrapper @@ -181,7 +181,22 @@ def repr_str(self, x, level): return s def repr_int(self, x, level): - s = builtins.repr(x) # XXX Hope this isn't too slow... + try: + s = builtins.repr(x) + except ValueError as exc: + assert 'sys.set_int_max_str_digits()' in str(exc) + # Those imports must be deferred due to Python's build system + # where the reprlib module is imported before the math module. + import math, sys + # Integers with more than sys.get_int_max_str_digits() digits + # are rendered differently as their repr() raises a ValueError. + # See https://github.com/python/cpython/issues/135487. + k = 1 + int(math.log10(abs(x))) + # Note: math.log10(abs(x)) may be overestimated or underestimated, + # but for simplicity, we do not compute the exact number of digits. + max_digits = sys.get_int_max_str_digits() + return (f'<{x.__class__.__name__} instance with roughly {k} ' + f'digits (limit at {max_digits}) at 0x{id(x):x}>') if len(s) > self.maxlong: i = max(0, (self.maxlong-3)//2) j = max(0, self.maxlong-3-i) diff --git a/Lib/shlex.py b/Lib/shlex.py index f4821616b62..5959f52dd12 100644 --- a/Lib/shlex.py +++ b/Lib/shlex.py @@ -7,11 +7,7 @@ # iterator interface by Gustavo Niemeyer, April 2003. # changes to tokenize more like Posix shells by Vinay Sajip, July 2016. -import os -import re import sys -from collections import deque - from io import StringIO __all__ = ["shlex", "split", "quote", "join"] @@ -20,6 +16,8 @@ class shlex: "A lexical analyzer class for simple shell-like syntaxes." def __init__(self, instream=None, infile=None, posix=False, punctuation_chars=False): + from collections import deque # deferred import for performance + if isinstance(instream, str): instream = StringIO(instream) if instream is not None: @@ -278,6 +276,7 @@ def read_token(self): def sourcehook(self, newfile): "Hook called on a filename to be sourced." + import os.path if newfile[0] == '"': newfile = newfile[1:-1] # This implements cpp-like semantics for relative-path inclusion. @@ -318,13 +317,20 @@ def join(split_command): return ' '.join(quote(arg) for arg in split_command) -_find_unsafe = re.compile(r'[^\w@%+=:,./-]', re.ASCII).search - def quote(s): """Return a shell-escaped version of the string *s*.""" if not s: return "''" - if _find_unsafe(s) is None: + + if not isinstance(s, str): + raise TypeError(f"expected string object, got {type(s).__name__!r}") + + # Use bytes.translate() for performance + safe_chars = (b'%+,-./0123456789:=@' + b'ABCDEFGHIJKLMNOPQRSTUVWXYZ_' + b'abcdefghijklmnopqrstuvwxyz') + # No quoting is needed if `s` is an ASCII string consisting only of `safe_chars` + if s.isascii() and not s.encode().translate(None, delete=safe_chars): return s # use single quotes, and put single quotes into double quotes diff --git a/Lib/site.py b/Lib/site.py index 2983ca71544..5305d67b3b8 100644 --- a/Lib/site.py +++ b/Lib/site.py @@ -73,7 +73,7 @@ import os import builtins import _sitebuiltins -import io +import _io as io import stat import errno @@ -95,6 +95,12 @@ def _trace(message): print(message, file=sys.stderr) +def _warn(*args, **kwargs): + import warnings + + warnings.warn(*args, **kwargs) + + def makepath(*paths): dir = os.path.join(*paths) try: @@ -444,9 +450,9 @@ def setcopyright(): """Set 'copyright' and 'credits' in builtins""" builtins.copyright = _sitebuiltins._Printer("copyright", sys.copyright) builtins.credits = _sitebuiltins._Printer("credits", """\ - Thanks to CWI, CNRI, BeOpen, Zope Corporation, the Python Software - Foundation, and a cast of thousands for supporting Python - development. See www.python.org for more information.""") +Thanks to CWI, CNRI, BeOpen, Zope Corporation, the Python Software +Foundation, and a cast of thousands for supporting Python +development. See www.python.org for more information.""") files, dirs = [], [] # Not all modules are required to have a __file__ attribute. See # PEP 420 for more details. @@ -574,7 +580,7 @@ def register_readline(): def write_history(): try: readline_module.write_history_file(history) - except (FileNotFoundError, PermissionError): + except FileNotFoundError, PermissionError: # home directory does not exist or is not writable # https://bugs.python.org/issue19891 pass @@ -626,17 +632,17 @@ def venv(known_paths): elif key == 'home': sys._home = value - sys.prefix = sys.exec_prefix = site_prefix + if sys.prefix != site_prefix: + _warn(f'Unexpected value in sys.prefix, expected {site_prefix}, got {sys.prefix}', RuntimeWarning) + if sys.exec_prefix != site_prefix: + _warn(f'Unexpected value in sys.exec_prefix, expected {site_prefix}, got {sys.exec_prefix}', RuntimeWarning) # Doing this here ensures venv takes precedence over user-site addsitepackages(known_paths, [sys.prefix]) - # addsitepackages will process site_prefix again if its in PREFIXES, - # but that's ok; known_paths will prevent anything being added twice if system_site == "true": - PREFIXES.insert(0, sys.prefix) + PREFIXES += [sys.base_prefix, sys.base_exec_prefix] else: - PREFIXES = [sys.prefix] ENABLE_USER_SITE = False return known_paths @@ -646,7 +652,7 @@ def execsitecustomize(): """Run custom site specific code, if available.""" try: try: - import sitecustomize + import sitecustomize # noqa: F401 except ImportError as exc: if exc.name == 'sitecustomize': pass @@ -666,7 +672,7 @@ def execusercustomize(): """Run custom user specific code, if available.""" try: try: - import usercustomize + import usercustomize # noqa: F401 except ImportError as exc: if exc.name == 'usercustomize': pass diff --git a/Lib/socket.py b/Lib/socket.py index 35d87eff34d..727b0e75f03 100644 --- a/Lib/socket.py +++ b/Lib/socket.py @@ -52,7 +52,9 @@ import _socket from _socket import * -import os, sys, io, selectors +import io +import os +import sys from enum import IntEnum, IntFlag try: @@ -110,102 +112,103 @@ def _intenum_converter(value, enum_klass): # WSA error codes if sys.platform.lower().startswith("win"): - errorTab = {} - errorTab[6] = "Specified event object handle is invalid." - errorTab[8] = "Insufficient memory available." - errorTab[87] = "One or more parameters are invalid." - errorTab[995] = "Overlapped operation aborted." - errorTab[996] = "Overlapped I/O event object not in signaled state." - errorTab[997] = "Overlapped operation will complete later." - errorTab[10004] = "The operation was interrupted." - errorTab[10009] = "A bad file handle was passed." - errorTab[10013] = "Permission denied." - errorTab[10014] = "A fault occurred on the network??" # WSAEFAULT - errorTab[10022] = "An invalid operation was attempted." - errorTab[10024] = "Too many open files." - errorTab[10035] = "The socket operation would block." - errorTab[10036] = "A blocking operation is already in progress." - errorTab[10037] = "Operation already in progress." - errorTab[10038] = "Socket operation on nonsocket." - errorTab[10039] = "Destination address required." - errorTab[10040] = "Message too long." - errorTab[10041] = "Protocol wrong type for socket." - errorTab[10042] = "Bad protocol option." - errorTab[10043] = "Protocol not supported." - errorTab[10044] = "Socket type not supported." - errorTab[10045] = "Operation not supported." - errorTab[10046] = "Protocol family not supported." - errorTab[10047] = "Address family not supported by protocol family." - errorTab[10048] = "The network address is in use." - errorTab[10049] = "Cannot assign requested address." - errorTab[10050] = "Network is down." - errorTab[10051] = "Network is unreachable." - errorTab[10052] = "Network dropped connection on reset." - errorTab[10053] = "Software caused connection abort." - errorTab[10054] = "The connection has been reset." - errorTab[10055] = "No buffer space available." - errorTab[10056] = "Socket is already connected." - errorTab[10057] = "Socket is not connected." - errorTab[10058] = "The network has been shut down." - errorTab[10059] = "Too many references." - errorTab[10060] = "The operation timed out." - errorTab[10061] = "Connection refused." - errorTab[10062] = "Cannot translate name." - errorTab[10063] = "The name is too long." - errorTab[10064] = "The host is down." - errorTab[10065] = "The host is unreachable." - errorTab[10066] = "Directory not empty." - errorTab[10067] = "Too many processes." - errorTab[10068] = "User quota exceeded." - errorTab[10069] = "Disk quota exceeded." - errorTab[10070] = "Stale file handle reference." - errorTab[10071] = "Item is remote." - errorTab[10091] = "Network subsystem is unavailable." - errorTab[10092] = "Winsock.dll version out of range." - errorTab[10093] = "Successful WSAStartup not yet performed." - errorTab[10101] = "Graceful shutdown in progress." - errorTab[10102] = "No more results from WSALookupServiceNext." - errorTab[10103] = "Call has been canceled." - errorTab[10104] = "Procedure call table is invalid." - errorTab[10105] = "Service provider is invalid." - errorTab[10106] = "Service provider failed to initialize." - errorTab[10107] = "System call failure." - errorTab[10108] = "Service not found." - errorTab[10109] = "Class type not found." - errorTab[10110] = "No more results from WSALookupServiceNext." - errorTab[10111] = "Call was canceled." - errorTab[10112] = "Database query was refused." - errorTab[11001] = "Host not found." - errorTab[11002] = "Nonauthoritative host not found." - errorTab[11003] = "This is a nonrecoverable error." - errorTab[11004] = "Valid name, no data record requested type." - errorTab[11005] = "QoS receivers." - errorTab[11006] = "QoS senders." - errorTab[11007] = "No QoS senders." - errorTab[11008] = "QoS no receivers." - errorTab[11009] = "QoS request confirmed." - errorTab[11010] = "QoS admission error." - errorTab[11011] = "QoS policy failure." - errorTab[11012] = "QoS bad style." - errorTab[11013] = "QoS bad object." - errorTab[11014] = "QoS traffic control error." - errorTab[11015] = "QoS generic error." - errorTab[11016] = "QoS service type error." - errorTab[11017] = "QoS flowspec error." - errorTab[11018] = "Invalid QoS provider buffer." - errorTab[11019] = "Invalid QoS filter style." - errorTab[11020] = "Invalid QoS filter style." - errorTab[11021] = "Incorrect QoS filter count." - errorTab[11022] = "Invalid QoS object length." - errorTab[11023] = "Incorrect QoS flow count." - errorTab[11024] = "Unrecognized QoS object." - errorTab[11025] = "Invalid QoS policy object." - errorTab[11026] = "Invalid QoS flow descriptor." - errorTab[11027] = "Invalid QoS provider-specific flowspec." - errorTab[11028] = "Invalid QoS provider-specific filterspec." - errorTab[11029] = "Invalid QoS shape discard mode object." - errorTab[11030] = "Invalid QoS shaping rate object." - errorTab[11031] = "Reserved policy QoS element type." + errorTab = { + 6: "Specified event object handle is invalid.", + 8: "Insufficient memory available.", + 87: "One or more parameters are invalid.", + 995: "Overlapped operation aborted.", + 996: "Overlapped I/O event object not in signaled state.", + 997: "Overlapped operation will complete later.", + 10004: "The operation was interrupted.", + 10009: "A bad file handle was passed.", + 10013: "Permission denied.", + 10014: "A fault occurred on the network??", + 10022: "An invalid operation was attempted.", + 10024: "Too many open files.", + 10035: "The socket operation would block.", + 10036: "A blocking operation is already in progress.", + 10037: "Operation already in progress.", + 10038: "Socket operation on nonsocket.", + 10039: "Destination address required.", + 10040: "Message too long.", + 10041: "Protocol wrong type for socket.", + 10042: "Bad protocol option.", + 10043: "Protocol not supported.", + 10044: "Socket type not supported.", + 10045: "Operation not supported.", + 10046: "Protocol family not supported.", + 10047: "Address family not supported by protocol family.", + 10048: "The network address is in use.", + 10049: "Cannot assign requested address.", + 10050: "Network is down.", + 10051: "Network is unreachable.", + 10052: "Network dropped connection on reset.", + 10053: "Software caused connection abort.", + 10054: "The connection has been reset.", + 10055: "No buffer space available.", + 10056: "Socket is already connected.", + 10057: "Socket is not connected.", + 10058: "The network has been shut down.", + 10059: "Too many references.", + 10060: "The operation timed out.", + 10061: "Connection refused.", + 10062: "Cannot translate name.", + 10063: "The name is too long.", + 10064: "The host is down.", + 10065: "The host is unreachable.", + 10066: "Directory not empty.", + 10067: "Too many processes.", + 10068: "User quota exceeded.", + 10069: "Disk quota exceeded.", + 10070: "Stale file handle reference.", + 10071: "Item is remote.", + 10091: "Network subsystem is unavailable.", + 10092: "Winsock.dll version out of range.", + 10093: "Successful WSAStartup not yet performed.", + 10101: "Graceful shutdown in progress.", + 10102: "No more results from WSALookupServiceNext.", + 10103: "Call has been canceled.", + 10104: "Procedure call table is invalid.", + 10105: "Service provider is invalid.", + 10106: "Service provider failed to initialize.", + 10107: "System call failure.", + 10108: "Service not found.", + 10109: "Class type not found.", + 10110: "No more results from WSALookupServiceNext.", + 10111: "Call was canceled.", + 10112: "Database query was refused.", + 11001: "Host not found.", + 11002: "Nonauthoritative host not found.", + 11003: "This is a nonrecoverable error.", + 11004: "Valid name, no data record requested type.", + 11005: "QoS receivers.", + 11006: "QoS senders.", + 11007: "No QoS senders.", + 11008: "QoS no receivers.", + 11009: "QoS request confirmed.", + 11010: "QoS admission error.", + 11011: "QoS policy failure.", + 11012: "QoS bad style.", + 11013: "QoS bad object.", + 11014: "QoS traffic control error.", + 11015: "QoS generic error.", + 11016: "QoS service type error.", + 11017: "QoS flowspec error.", + 11018: "Invalid QoS provider buffer.", + 11019: "Invalid QoS filter style.", + 11020: "Invalid QoS filter style.", + 11021: "Incorrect QoS filter count.", + 11022: "Invalid QoS object length.", + 11023: "Incorrect QoS flow count.", + 11024: "Unrecognized QoS object.", + 11025: "Invalid QoS policy object.", + 11026: "Invalid QoS flow descriptor.", + 11027: "Invalid QoS provider-specific flowspec.", + 11028: "Invalid QoS provider-specific filterspec.", + 11029: "Invalid QoS shape discard mode object.", + 11030: "Invalid QoS shaping rate object.", + 11031: "Reserved policy QoS element type." + } __all__.append("errorTab") @@ -348,6 +351,9 @@ def makefile(self, mode="r", buffering=None, *, if hasattr(os, 'sendfile'): def _sendfile_use_sendfile(self, file, offset=0, count=None): + # Lazy import to improve module import time + import selectors + self._check_sendfile_params(file, offset, count) sockno = self.fileno() try: @@ -549,20 +555,18 @@ def fromfd(fd, family, type, proto=0): return socket(family, type, proto, nfd) if hasattr(_socket.socket, "sendmsg"): - import array - def send_fds(sock, buffers, fds, flags=0, address=None): """ send_fds(sock, buffers, fds[, flags[, address]]) -> integer Send the list of file descriptors fds over an AF_UNIX socket. """ + import array + return sock.sendmsg(buffers, [(_socket.SOL_SOCKET, _socket.SCM_RIGHTS, array.array("i", fds))]) __all__.append("send_fds") if hasattr(_socket.socket, "recvmsg"): - import array - def recv_fds(sock, bufsize, maxfds, flags=0): """ recv_fds(sock, bufsize, maxfds[, flags]) -> (data, list of file descriptors, msg_flags, address) @@ -570,6 +574,8 @@ def recv_fds(sock, bufsize, maxfds, flags=0): Receive up to maxfds file descriptors returning the message data and a list containing the descriptors. """ + import array + # Array of ints fds = array.array("i") msg, ancdata, flags, addr = sock.recvmsg(bufsize, diff --git a/Lib/sqlite3/__init__.py b/Lib/sqlite3/__init__.py index 927267cf0b9..ed727fae609 100644 --- a/Lib/sqlite3/__init__.py +++ b/Lib/sqlite3/__init__.py @@ -22,7 +22,7 @@ """ The sqlite3 extension module provides a DB-API 2.0 (PEP 249) compliant -interface to the SQLite library, and requires SQLite 3.7.15 or newer. +interface to the SQLite library, and requires SQLite 3.15.2 or newer. To use the module, start by creating a database Connection object: @@ -55,16 +55,3 @@ """ from sqlite3.dbapi2 import * -from sqlite3.dbapi2 import (_deprecated_names, - _deprecated_version_info, - _deprecated_version) - - -def __getattr__(name): - if name in _deprecated_names: - from warnings import warn - - warn(f"{name} is deprecated and will be removed in Python 3.14", - DeprecationWarning, stacklevel=2) - return globals()[f"_deprecated_{name}"] - raise AttributeError(f"module {__name__!r} has no attribute {name!r}") diff --git a/Lib/sqlite3/__main__.py b/Lib/sqlite3/__main__.py index f8a5cca24e5..4ccf292ddf2 100644 --- a/Lib/sqlite3/__main__.py +++ b/Lib/sqlite3/__main__.py @@ -46,26 +46,34 @@ def runsource(self, source, filename="", symbol="single"): """Override runsource, the core of the InteractiveConsole REPL. Return True if more input is needed; buffering is done automatically. - Return False is input is a complete statement ready for execution. + Return False if input is a complete statement ready for execution. """ - match source: - case ".version": - print(f"{sqlite3.sqlite_version}") - case ".help": - print("Enter SQL code and press enter.") - case ".quit": - sys.exit(0) - case _: - if not sqlite3.complete_statement(source): - return True - execute(self._cur, source) + if not source or source.isspace(): + return False + if source[0] == ".": + match source[1:].strip(): + case "version": + print(f"{sqlite3.sqlite_version}") + case "help": + print("Enter SQL code and press enter.") + case "quit": + sys.exit(0) + case "": + pass + case _ as unknown: + self.write("Error: unknown command or invalid arguments:" + f' "{unknown}".\n') + else: + if not sqlite3.complete_statement(source): + return True + execute(self._cur, source) return False -def main(): +def main(*args): parser = ArgumentParser( description="Python sqlite3 CLI", - prog="python -m sqlite3", + color=True, ) parser.add_argument( "filename", type=str, default=":memory:", nargs="?", @@ -86,7 +94,7 @@ def main(): version=f"SQLite version {sqlite3.sqlite_version}", help="Print underlying SQLite library version", ) - args = parser.parse_args() + args = parser.parse_args(*args) if args.filename == ":memory:": db_name = "a transient in-memory database" @@ -94,12 +102,16 @@ def main(): db_name = repr(args.filename) # Prepare REPL banner and prompts. + if sys.platform == "win32" and "idlelib.run" not in sys.modules: + eofkey = "CTRL-Z" + else: + eofkey = "CTRL-D" banner = dedent(f""" sqlite3 shell, running on SQLite version {sqlite3.sqlite_version} Connected to {db_name} Each command will be run using execute() on the cursor. - Type ".help" for more information; type ".quit" or CTRL-D to quit. + Type ".help" for more information; type ".quit" or {eofkey} to quit. """).strip() sys.ps1 = "sqlite> " sys.ps2 = " ... " @@ -112,9 +124,16 @@ def main(): else: # No SQL provided; start the REPL. console = SqliteInteractiveConsole(con) + try: + import readline # noqa: F401 + except ImportError: + pass console.interact(banner, exitmsg="") finally: con.close() + sys.exit(0) + -main() +if __name__ == "__main__": + main(sys.argv[1:]) diff --git a/Lib/sqlite3/dbapi2.py b/Lib/sqlite3/dbapi2.py index 56fc0461e6c..0315760516e 100644 --- a/Lib/sqlite3/dbapi2.py +++ b/Lib/sqlite3/dbapi2.py @@ -25,9 +25,6 @@ import collections.abc from _sqlite3 import * -from _sqlite3 import _deprecated_version - -_deprecated_names = frozenset({"version", "version_info"}) paramstyle = "qmark" @@ -48,7 +45,7 @@ def TimeFromTicks(ticks): def TimestampFromTicks(ticks): return Timestamp(*time.localtime(ticks)[:6]) -_deprecated_version_info = tuple(map(int, _deprecated_version.split("."))) + sqlite_version_info = tuple([int(x) for x in sqlite_version.split(".")]) Binary = memoryview @@ -97,12 +94,3 @@ def convert_timestamp(val): # Clean up namespace del(register_adapters_and_converters) - -def __getattr__(name): - if name in _deprecated_names: - from warnings import warn - - warn(f"{name} is deprecated and will be removed in Python 3.14", - DeprecationWarning, stacklevel=2) - return globals()[f"_deprecated_{name}"] - raise AttributeError(f"module {__name__!r} has no attribute {name!r}") diff --git a/Lib/sqlite3/dump.py b/Lib/sqlite3/dump.py index 07b9da10b92..57e6a3b4f1e 100644 --- a/Lib/sqlite3/dump.py +++ b/Lib/sqlite3/dump.py @@ -7,7 +7,15 @@ # future enhancements, you should normally quote any identifier that # is an English language word, even if you do not have to." -def _iterdump(connection): +def _quote_name(name): + return '"{0}"'.format(name.replace('"', '""')) + + +def _quote_value(value): + return "'{0}'".format(value.replace("'", "''")) + + +def _iterdump(connection, *, filter=None): """ Returns an iterator to the dump of the database in an SQL text format. @@ -16,64 +24,87 @@ def _iterdump(connection): directly but instead called from the Connection method, iterdump(). """ + writeable_schema = False cu = connection.cursor() + cu.row_factory = None # Make sure we get predictable results. + # Disable foreign key constraints, if there is any foreign key violation. + violations = cu.execute("PRAGMA foreign_key_check").fetchall() + if violations: + yield('PRAGMA foreign_keys=OFF;') yield('BEGIN TRANSACTION;') + if filter: + # Return database objects which match the filter pattern. + filter_name_clause = 'AND "name" LIKE ?' + params = [filter] + else: + filter_name_clause = "" + params = [] # sqlite_master table contains the SQL CREATE statements for the database. - q = """ + q = f""" SELECT "name", "type", "sql" FROM "sqlite_master" WHERE "sql" NOT NULL AND "type" == 'table' + {filter_name_clause} ORDER BY "name" """ - schema_res = cu.execute(q) + schema_res = cu.execute(q, params) sqlite_sequence = [] for table_name, type, sql in schema_res.fetchall(): if table_name == 'sqlite_sequence': - rows = cu.execute('SELECT * FROM "sqlite_sequence";').fetchall() + rows = cu.execute('SELECT * FROM "sqlite_sequence";') sqlite_sequence = ['DELETE FROM "sqlite_sequence"'] sqlite_sequence += [ - f'INSERT INTO "sqlite_sequence" VALUES(\'{row[0]}\',{row[1]})' - for row in rows + f'INSERT INTO "sqlite_sequence" VALUES({_quote_value(table_name)},{seq_value})' + for table_name, seq_value in rows.fetchall() ] continue elif table_name == 'sqlite_stat1': yield('ANALYZE "sqlite_master";') elif table_name.startswith('sqlite_'): continue - # NOTE: Virtual table support not implemented - #elif sql.startswith('CREATE VIRTUAL TABLE'): - # qtable = table_name.replace("'", "''") - # yield("INSERT INTO sqlite_master(type,name,tbl_name,rootpage,sql)"\ - # "VALUES('table','{0}','{0}',0,'{1}');".format( - # qtable, - # sql.replace("''"))) + elif sql.startswith('CREATE VIRTUAL TABLE'): + if not writeable_schema: + writeable_schema = True + yield('PRAGMA writable_schema=ON;') + yield("INSERT INTO sqlite_master(type,name,tbl_name,rootpage,sql)" + "VALUES('table',{0},{0},0,{1});".format( + _quote_value(table_name), + _quote_value(sql), + )) else: yield('{0};'.format(sql)) # Build the insert statement for each row of the current table - table_name_ident = table_name.replace('"', '""') - res = cu.execute('PRAGMA table_info("{0}")'.format(table_name_ident)) + table_name_ident = _quote_name(table_name) + res = cu.execute(f'PRAGMA table_info({table_name_ident})') column_names = [str(table_info[1]) for table_info in res.fetchall()] - q = """SELECT 'INSERT INTO "{0}" VALUES({1})' FROM "{0}";""".format( + q = "SELECT 'INSERT INTO {0} VALUES('{1}')' FROM {0};".format( table_name_ident, - ",".join("""'||quote("{0}")||'""".format(col.replace('"', '""')) for col in column_names)) + "','".join( + "||quote({0})||".format(_quote_name(col)) for col in column_names + ) + ) query_res = cu.execute(q) for row in query_res: yield("{0};".format(row[0])) # Now when the type is 'index', 'trigger', or 'view' - q = """ + q = f""" SELECT "name", "type", "sql" FROM "sqlite_master" WHERE "sql" NOT NULL AND "type" IN ('index', 'trigger', 'view') + {filter_name_clause} """ - schema_res = cu.execute(q) + schema_res = cu.execute(q, params) for name, type, sql in schema_res.fetchall(): yield('{0};'.format(sql)) + if writeable_schema: + yield('PRAGMA writable_schema=OFF;') + # gh-79009: Yield statements concerning the sqlite_sequence table at the # end of the transaction. for row in sqlite_sequence: diff --git a/Lib/string.py b/Lib/string/__init__.py similarity index 87% rename from Lib/string.py rename to Lib/string/__init__.py index 2eab6d4f595..eab5067c9b1 100644 --- a/Lib/string.py +++ b/Lib/string/__init__.py @@ -49,11 +49,18 @@ def capwords(s, sep=None): #################################################################### -import re as _re -from collections import ChainMap as _ChainMap - _sentinel_dict = {} + +class _TemplatePattern: + # This descriptor is overwritten in ``Template._compile_pattern()``. + def __get__(self, instance, cls=None): + if cls is None: + return self + return cls._compile_pattern() +_TemplatePattern = _TemplatePattern() + + class Template: """A string class for supporting $-substitutions.""" @@ -64,14 +71,21 @@ class Template: # See https://bugs.python.org/issue31672 idpattern = r'(?a:[_a-z][_a-z0-9]*)' braceidpattern = None - flags = _re.IGNORECASE + flags = None # default: re.IGNORECASE + + pattern = _TemplatePattern # use a descriptor to compile the pattern def __init_subclass__(cls): super().__init_subclass__() - if 'pattern' in cls.__dict__: - pattern = cls.pattern - else: - delim = _re.escape(cls.delimiter) + cls._compile_pattern() + + @classmethod + def _compile_pattern(cls): + import re # deferred import, for performance + + pattern = cls.__dict__.get('pattern', _TemplatePattern) + if pattern is _TemplatePattern: + delim = re.escape(cls.delimiter) id = cls.idpattern bid = cls.braceidpattern or cls.idpattern pattern = fr""" @@ -82,7 +96,10 @@ def __init_subclass__(cls): (?P) # Other ill-formed delimiter exprs ) """ - cls.pattern = _re.compile(pattern, cls.flags | _re.VERBOSE) + if cls.flags is None: + cls.flags = re.IGNORECASE + pat = cls.pattern = re.compile(pattern, cls.flags | re.VERBOSE) + return pat def __init__(self, template): self.template = template @@ -105,7 +122,8 @@ def substitute(self, mapping=_sentinel_dict, /, **kws): if mapping is _sentinel_dict: mapping = kws elif kws: - mapping = _ChainMap(kws, mapping) + from collections import ChainMap + mapping = ChainMap(kws, mapping) # Helper function for .sub() def convert(mo): # Check the most common path first. @@ -124,7 +142,8 @@ def safe_substitute(self, mapping=_sentinel_dict, /, **kws): if mapping is _sentinel_dict: mapping = kws elif kws: - mapping = _ChainMap(kws, mapping) + from collections import ChainMap + mapping = ChainMap(kws, mapping) # Helper function for .sub() def convert(mo): named = mo.group('named') or mo.group('braced') @@ -170,10 +189,6 @@ def get_identifiers(self): self.pattern) return ids -# Initialize Template.pattern. __init_subclass__() is automatically called -# only for subclasses, not for the Template class itself. -Template.__init_subclass__() - ######################################################################## # the Formatter class @@ -212,19 +227,20 @@ def _vformat(self, format_string, args, kwargs, used_args, recursion_depth, # this is some markup, find the object and do # the formatting - # handle arg indexing when empty field_names are given. - if field_name == '': + # handle arg indexing when empty field first parts are given. + field_first, _ = _string.formatter_field_name_split(field_name) + if field_first == '': if auto_arg_index is False: raise ValueError('cannot switch from manual field ' 'specification to automatic field ' 'numbering') - field_name = str(auto_arg_index) + field_name = str(auto_arg_index) + field_name auto_arg_index += 1 - elif field_name.isdigit(): + elif isinstance(field_first, int): if auto_arg_index: - raise ValueError('cannot switch from manual field ' - 'specification to automatic field ' - 'numbering') + raise ValueError('cannot switch from automatic field ' + 'numbering to manual field ' + 'specification') # disable auto arg incrementing, if it gets # used later on, then an exception will be raised auto_arg_index = False diff --git a/Lib/string/templatelib.py b/Lib/string/templatelib.py new file mode 100644 index 00000000000..8164872432a --- /dev/null +++ b/Lib/string/templatelib.py @@ -0,0 +1,33 @@ +"""Support for template string literals (t-strings).""" + +t = t"{0}" +Template = type(t) +Interpolation = type(t.interpolations[0]) +del t + +def convert(obj, /, conversion): + """Convert *obj* using formatted string literal semantics.""" + if conversion is None: + return obj + if conversion == 'r': + return repr(obj) + if conversion == 's': + return str(obj) + if conversion == 'a': + return ascii(obj) + raise ValueError(f'invalid conversion specifier: {conversion}') + +def _template_unpickle(*args): + import itertools + + if len(args) != 2: + raise ValueError('Template expects tuple of length 2 to unpickle') + + strings, interpolations = args + parts = [] + for string, interpolation in itertools.zip_longest(strings, interpolations): + if string is not None: + parts.append(string) + if interpolation is not None: + parts.append(interpolation) + return Template(*parts) diff --git a/Lib/struct.py b/Lib/struct.py index d6bba588636..ff98e8c4cb3 100644 --- a/Lib/struct.py +++ b/Lib/struct.py @@ -11,5 +11,5 @@ ] from _struct import * -from _struct import _clearcache -from _struct import __doc__ +from _struct import _clearcache # noqa: F401 +from _struct import __doc__ # noqa: F401 diff --git a/Lib/symtable.py b/Lib/symtable.py index 672ec0ce1ff..7a30e1ac4ca 100644 --- a/Lib/symtable.py +++ b/Lib/symtable.py @@ -1,9 +1,16 @@ """Interface to the compiler's internal symbol tables""" import _symtable -from _symtable import (USE, DEF_GLOBAL, DEF_NONLOCAL, DEF_LOCAL, DEF_PARAM, - DEF_IMPORT, DEF_BOUND, DEF_ANNOT, SCOPE_OFF, SCOPE_MASK, FREE, - LOCAL, GLOBAL_IMPLICIT, GLOBAL_EXPLICIT, CELL) +from _symtable import ( + USE, + DEF_GLOBAL, # noqa: F401 + DEF_NONLOCAL, DEF_LOCAL, + DEF_PARAM, DEF_TYPE_PARAM, DEF_FREE_CLASS, + DEF_IMPORT, DEF_BOUND, DEF_ANNOT, + DEF_COMP_ITER, DEF_COMP_CELL, + SCOPE_OFF, SCOPE_MASK, + FREE, LOCAL, GLOBAL_IMPLICIT, GLOBAL_EXPLICIT, CELL +) import weakref from enum import StrEnum @@ -165,6 +172,10 @@ def get_children(self): for st in self._table.children] +def _get_scope(flags): # like _PyST_GetScope() + return (flags >> SCOPE_OFF) & SCOPE_MASK + + class Function(SymbolTable): # Default values for instance variables @@ -190,7 +201,7 @@ def get_locals(self): """ if self.__locals is None: locs = (LOCAL, CELL) - test = lambda x: ((x >> SCOPE_OFF) & SCOPE_MASK) in locs + test = lambda x: _get_scope(x) in locs self.__locals = self.__idents_matching(test) return self.__locals @@ -199,7 +210,7 @@ def get_globals(self): """ if self.__globals is None: glob = (GLOBAL_IMPLICIT, GLOBAL_EXPLICIT) - test = lambda x:((x >> SCOPE_OFF) & SCOPE_MASK) in glob + test = lambda x: _get_scope(x) in glob self.__globals = self.__idents_matching(test) return self.__globals @@ -214,7 +225,7 @@ def get_frees(self): """Return a tuple of free variables in the function. """ if self.__frees is None: - is_free = lambda x:((x >> SCOPE_OFF) & SCOPE_MASK) == FREE + is_free = lambda x: _get_scope(x) == FREE self.__frees = self.__idents_matching(is_free) return self.__frees @@ -226,6 +237,12 @@ class Class(SymbolTable): def get_methods(self): """Return a tuple of methods declared in the class. """ + import warnings + typename = f'{self.__class__.__module__}.{self.__class__.__name__}' + warnings.warn(f'{typename}.get_methods() is deprecated ' + f'and will be removed in Python 3.16.', + DeprecationWarning, stacklevel=2) + if self.__methods is None: d = {} @@ -268,7 +285,7 @@ class Symbol: def __init__(self, name, flags, namespaces=None, *, module_scope=False): self.__name = name self.__flags = flags - self.__scope = (flags >> SCOPE_OFF) & SCOPE_MASK # like PyST_GetScope() + self.__scope = _get_scope(flags) self.__namespaces = namespaces or () self.__module_scope = module_scope @@ -293,13 +310,18 @@ def is_referenced(self): """Return *True* if the symbol is used in its block. """ - return bool(self.__flags & _symtable.USE) + return bool(self.__flags & USE) def is_parameter(self): """Return *True* if the symbol is a parameter. """ return bool(self.__flags & DEF_PARAM) + def is_type_parameter(self): + """Return *True* if the symbol is a type parameter. + """ + return bool(self.__flags & DEF_TYPE_PARAM) + def is_global(self): """Return *True* if the symbol is global. """ @@ -332,6 +354,11 @@ def is_free(self): """ return bool(self.__scope == FREE) + def is_free_class(self): + """Return *True* if a class-scoped symbol is free from + the perspective of a method.""" + return bool(self.__flags & DEF_FREE_CLASS) + def is_imported(self): """Return *True* if the symbol is created from an import statement. @@ -342,6 +369,16 @@ def is_assigned(self): """Return *True* if a symbol is assigned to.""" return bool(self.__flags & DEF_LOCAL) + def is_comp_iter(self): + """Return *True* if the symbol is a comprehension iteration variable. + """ + return bool(self.__flags & DEF_COMP_ITER) + + def is_comp_cell(self): + """Return *True* if the symbol is a cell in an inlined comprehension. + """ + return bool(self.__flags & DEF_COMP_CELL) + def is_namespace(self): """Returns *True* if name binding introduces new namespace. diff --git a/Lib/tabnanny.py b/Lib/tabnanny.py old mode 100755 new mode 100644 index d06c4c221e9..c0097351b26 --- a/Lib/tabnanny.py +++ b/Lib/tabnanny.py @@ -1,5 +1,3 @@ -#! /usr/bin/env python3 - """The Tab Nanny despises ambiguous indentation. She knows no mercy. tabnanny -- Detection of ambiguous indentation diff --git a/Lib/test/__main__.py b/Lib/test/__main__.py index 19a6b2b8904..82b50ad2c6e 100644 --- a/Lib/test/__main__.py +++ b/Lib/test/__main__.py @@ -1,2 +1,2 @@ -from test.libregrtest import main -main() +from test.libregrtest.main import main +main(_add_python_opts=True) diff --git a/Lib/test/_test_eintr.py b/Lib/test/_test_eintr.py new file mode 100644 index 00000000000..c8f04e9625c --- /dev/null +++ b/Lib/test/_test_eintr.py @@ -0,0 +1,550 @@ +""" +This test suite exercises some system calls subject to interruption with EINTR, +to check that it is actually handled transparently. +It is intended to be run by the main test suite within a child process, to +ensure there is no background thread running (so that signals are delivered to +the correct thread). +Signals are generated in-process using setitimer(ITIMER_REAL), which allows +sub-second periodicity (contrarily to signal()). +""" + +import contextlib +import faulthandler +import fcntl +import os +import platform +import select +import signal +import socket +import subprocess +import sys +import textwrap +import time +import unittest + +from test import support +from test.support import os_helper +from test.support import socket_helper + + +# gh-109592: Tolerate a difference of 20 ms when comparing timings +# (clock resolution) +CLOCK_RES = 0.020 + + +@contextlib.contextmanager +def kill_on_error(proc): + """Context manager killing the subprocess if a Python exception is raised.""" + with proc: + try: + yield proc + except: + proc.kill() + raise + + +@unittest.skipUnless(hasattr(signal, "setitimer"), "requires setitimer()") +class EINTRBaseTest(unittest.TestCase): + """ Base class for EINTR tests. """ + + # delay for initial signal delivery + signal_delay = 0.1 + # signal delivery periodicity + signal_period = 0.1 + # default sleep time for tests - should obviously have: + # sleep_time > signal_period + sleep_time = 0.2 + + def sighandler(self, signum, frame): + self.signals += 1 + + def setUp(self): + self.signals = 0 + self.orig_handler = signal.signal(signal.SIGALRM, self.sighandler) + signal.setitimer(signal.ITIMER_REAL, self.signal_delay, + self.signal_period) + + # Use faulthandler as watchdog to debug when a test hangs + # (timeout of 10 minutes) + faulthandler.dump_traceback_later(10 * 60, exit=True, + file=sys.__stderr__) + + @staticmethod + def stop_alarm(): + signal.setitimer(signal.ITIMER_REAL, 0, 0) + + def tearDown(self): + self.stop_alarm() + signal.signal(signal.SIGALRM, self.orig_handler) + faulthandler.cancel_dump_traceback_later() + + def subprocess(self, *args, **kw): + cmd_args = (sys.executable, '-c') + args + return subprocess.Popen(cmd_args, **kw) + + def check_elapsed_time(self, elapsed): + self.assertGreaterEqual(elapsed, self.sleep_time - CLOCK_RES) + + +@unittest.skipUnless(hasattr(signal, "setitimer"), "requires setitimer()") +class OSEINTRTest(EINTRBaseTest): + """ EINTR tests for the os module. """ + + def new_sleep_process(self): + code = 'import time; time.sleep(%r)' % self.sleep_time + return self.subprocess(code) + + def _test_wait_multiple(self, wait_func): + num = 3 + processes = [self.new_sleep_process() for _ in range(num)] + for _ in range(num): + wait_func() + # Call the Popen method to avoid a ResourceWarning + for proc in processes: + proc.wait() + + def test_wait(self): + self._test_wait_multiple(os.wait) + + @unittest.skipUnless(hasattr(os, 'wait3'), 'requires wait3()') + def test_wait3(self): + self._test_wait_multiple(lambda: os.wait3(0)) + + def _test_wait_single(self, wait_func): + proc = self.new_sleep_process() + wait_func(proc.pid) + # Call the Popen method to avoid a ResourceWarning + proc.wait() + + def test_waitpid(self): + self._test_wait_single(lambda pid: os.waitpid(pid, 0)) + + @unittest.skipUnless(hasattr(os, 'wait4'), 'requires wait4()') + def test_wait4(self): + self._test_wait_single(lambda pid: os.wait4(pid, 0)) + + def test_read(self): + rd, wr = os.pipe() + self.addCleanup(os.close, rd) + # wr closed explicitly by parent + + # the payload below are smaller than PIPE_BUF, hence the writes will be + # atomic + datas = [b"hello", b"world", b"spam"] + + code = '\n'.join(( + 'import os, sys, time', + '', + 'wr = int(sys.argv[1])', + 'datas = %r' % datas, + 'sleep_time = %r' % self.sleep_time, + '', + 'for data in datas:', + ' # let the parent block on read()', + ' time.sleep(sleep_time)', + ' os.write(wr, data)', + )) + + proc = self.subprocess(code, str(wr), pass_fds=[wr]) + with kill_on_error(proc): + os.close(wr) + for data in datas: + self.assertEqual(data, os.read(rd, len(data))) + self.assertEqual(proc.wait(), 0) + + @unittest.expectedFailure # TODO: RUSTPYTHON InterruptedError: [Errno 4] Interrupted system call + def test_write(self): + rd, wr = os.pipe() + self.addCleanup(os.close, wr) + # rd closed explicitly by parent + + # we must write enough data for the write() to block + data = b"x" * support.PIPE_MAX_SIZE + + code = '\n'.join(( + 'import io, os, sys, time', + '', + 'rd = int(sys.argv[1])', + 'sleep_time = %r' % self.sleep_time, + 'data = b"x" * %s' % support.PIPE_MAX_SIZE, + 'data_len = len(data)', + '', + '# let the parent block on write()', + 'time.sleep(sleep_time)', + '', + 'read_data = io.BytesIO()', + 'while len(read_data.getvalue()) < data_len:', + ' chunk = os.read(rd, 2 * data_len)', + ' read_data.write(chunk)', + '', + 'value = read_data.getvalue()', + 'if value != data:', + ' raise Exception("read error: %s vs %s bytes"', + ' % (len(value), data_len))', + )) + + proc = self.subprocess(code, str(rd), pass_fds=[rd]) + with kill_on_error(proc): + os.close(rd) + written = 0 + while written < len(data): + written += os.write(wr, memoryview(data)[written:]) + self.assertEqual(proc.wait(), 0) + + +@unittest.skipUnless(hasattr(signal, "setitimer"), "requires setitimer()") +class SocketEINTRTest(EINTRBaseTest): + """ EINTR tests for the socket module. """ + + @unittest.skipUnless(hasattr(socket, 'socketpair'), 'needs socketpair()') + def _test_recv(self, recv_func): + rd, wr = socket.socketpair() + self.addCleanup(rd.close) + # wr closed explicitly by parent + + # single-byte payload guard us against partial recv + datas = [b"x", b"y", b"z"] + + code = '\n'.join(( + 'import os, socket, sys, time', + '', + 'fd = int(sys.argv[1])', + 'family = %s' % int(wr.family), + 'sock_type = %s' % int(wr.type), + 'datas = %r' % datas, + 'sleep_time = %r' % self.sleep_time, + '', + 'wr = socket.fromfd(fd, family, sock_type)', + 'os.close(fd)', + '', + 'with wr:', + ' for data in datas:', + ' # let the parent block on recv()', + ' time.sleep(sleep_time)', + ' wr.sendall(data)', + )) + + fd = wr.fileno() + proc = self.subprocess(code, str(fd), pass_fds=[fd]) + with kill_on_error(proc): + wr.close() + for data in datas: + self.assertEqual(data, recv_func(rd, len(data))) + self.assertEqual(proc.wait(), 0) + + def test_recv(self): + self._test_recv(socket.socket.recv) + + @unittest.skipUnless(hasattr(socket.socket, 'recvmsg'), 'needs recvmsg()') + def test_recvmsg(self): + self._test_recv(lambda sock, data: sock.recvmsg(data)[0]) + + def _test_send(self, send_func): + rd, wr = socket.socketpair() + self.addCleanup(wr.close) + # rd closed explicitly by parent + + # we must send enough data for the send() to block + data = b"xyz" * (support.SOCK_MAX_SIZE // 3) + + code = '\n'.join(( + 'import os, socket, sys, time', + '', + 'fd = int(sys.argv[1])', + 'family = %s' % int(rd.family), + 'sock_type = %s' % int(rd.type), + 'sleep_time = %r' % self.sleep_time, + 'data = b"xyz" * %s' % (support.SOCK_MAX_SIZE // 3), + 'data_len = len(data)', + '', + 'rd = socket.fromfd(fd, family, sock_type)', + 'os.close(fd)', + '', + 'with rd:', + ' # let the parent block on send()', + ' time.sleep(sleep_time)', + '', + ' received_data = bytearray(data_len)', + ' n = 0', + ' while n < data_len:', + ' n += rd.recv_into(memoryview(received_data)[n:])', + '', + 'if received_data != data:', + ' raise Exception("recv error: %s vs %s bytes"', + ' % (len(received_data), data_len))', + )) + + fd = rd.fileno() + proc = self.subprocess(code, str(fd), pass_fds=[fd]) + with kill_on_error(proc): + rd.close() + written = 0 + while written < len(data): + sent = send_func(wr, memoryview(data)[written:]) + # sendall() returns None + written += len(data) if sent is None else sent + self.assertEqual(proc.wait(), 0) + + def test_send(self): + self._test_send(socket.socket.send) + + def test_sendall(self): + self._test_send(socket.socket.sendall) + + @unittest.skipUnless(hasattr(socket.socket, 'sendmsg'), 'needs sendmsg()') + def test_sendmsg(self): + self._test_send(lambda sock, data: sock.sendmsg([data])) + + def test_accept(self): + sock = socket.create_server((socket_helper.HOST, 0)) + self.addCleanup(sock.close) + port = sock.getsockname()[1] + + code = '\n'.join(( + 'import socket, time', + '', + 'host = %r' % socket_helper.HOST, + 'port = %s' % port, + 'sleep_time = %r' % self.sleep_time, + '', + '# let parent block on accept()', + 'time.sleep(sleep_time)', + 'with socket.create_connection((host, port)):', + ' time.sleep(sleep_time)', + )) + + proc = self.subprocess(code) + with kill_on_error(proc): + client_sock, _ = sock.accept() + client_sock.close() + self.assertEqual(proc.wait(), 0) + + # Issue #25122: There is a race condition in the FreeBSD kernel on + # handling signals in the FIFO device. Skip the test until the bug is + # fixed in the kernel. + # https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203162 + @support.requires_freebsd_version(10, 3) + @unittest.skipUnless(hasattr(os, 'mkfifo'), 'needs mkfifo()') + def _test_open(self, do_open_close_reader, do_open_close_writer): + filename = os_helper.TESTFN + + # Use a fifo: until the child opens it for reading, the parent will + # block when trying to open it for writing. + os_helper.unlink(filename) + try: + os.mkfifo(filename) + except PermissionError as e: + self.skipTest('os.mkfifo(): %s' % e) + self.addCleanup(os_helper.unlink, filename) + + code = '\n'.join(( + 'import os, time', + '', + 'path = %a' % filename, + 'sleep_time = %r' % self.sleep_time, + '', + '# let the parent block', + 'time.sleep(sleep_time)', + '', + do_open_close_reader, + )) + + proc = self.subprocess(code) + with kill_on_error(proc): + do_open_close_writer(filename) + self.assertEqual(proc.wait(), 0) + + def python_open(self, path): + fp = open(path, 'w') + fp.close() + + @unittest.skipIf(sys.platform == "darwin", + "hangs under macOS; see bpo-25234, bpo-35363") + def test_open(self): + self._test_open("fp = open(path, 'r')\nfp.close()", + self.python_open) + + def os_open(self, path): + fd = os.open(path, os.O_WRONLY) + os.close(fd) + + @unittest.skipIf(sys.platform == "darwin", + "hangs under macOS; see bpo-25234, bpo-35363") + @unittest.skipIf(sys.platform.startswith('netbsd'), + "hangs on NetBSD; see gh-137397") + def test_os_open(self): + self._test_open("fd = os.open(path, os.O_RDONLY)\nos.close(fd)", + self.os_open) + + +@unittest.skipUnless(hasattr(signal, "setitimer"), "requires setitimer()") +class TimeEINTRTest(EINTRBaseTest): + """ EINTR tests for the time module. """ + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_sleep(self): + t0 = time.monotonic() + time.sleep(self.sleep_time) + self.stop_alarm() + dt = time.monotonic() - t0 + self.check_elapsed_time(dt) + + +@unittest.skipUnless(hasattr(signal, "setitimer"), "requires setitimer()") +# bpo-30320: Need pthread_sigmask() to block the signal, otherwise the test +# is vulnerable to a race condition between the child and the parent processes. +@unittest.skipUnless(hasattr(signal, 'pthread_sigmask'), + 'need signal.pthread_sigmask()') +class SignalEINTRTest(EINTRBaseTest): + """ EINTR tests for the signal module. """ + + def check_sigwait(self, wait_func): + signum = signal.SIGUSR1 + pid = os.getpid() + + old_handler = signal.signal(signum, lambda *args: None) + self.addCleanup(signal.signal, signum, old_handler) + + code = '\n'.join(( + 'import os, time', + 'pid = %s' % os.getpid(), + 'signum = %s' % int(signum), + 'sleep_time = %r' % self.sleep_time, + 'time.sleep(sleep_time)', + 'os.kill(pid, signum)', + )) + + old_mask = signal.pthread_sigmask(signal.SIG_BLOCK, [signum]) + self.addCleanup(signal.pthread_sigmask, signal.SIG_UNBLOCK, [signum]) + + proc = self.subprocess(code) + with kill_on_error(proc): + wait_func(signum) + + self.assertEqual(proc.wait(), 0) + + @unittest.skipUnless(hasattr(signal, 'sigwaitinfo'), + 'need signal.sigwaitinfo()') + def test_sigwaitinfo(self): + def wait_func(signum): + signal.sigwaitinfo([signum]) + + self.check_sigwait(wait_func) + + @unittest.skipUnless(hasattr(signal, 'sigtimedwait'), + 'need signal.sigwaitinfo()') + def test_sigtimedwait(self): + def wait_func(signum): + signal.sigtimedwait([signum], 120.0) + + self.check_sigwait(wait_func) + + +@unittest.skipUnless(hasattr(signal, "setitimer"), "requires setitimer()") +class SelectEINTRTest(EINTRBaseTest): + """ EINTR tests for the select module. """ + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_select(self): + t0 = time.monotonic() + select.select([], [], [], self.sleep_time) + dt = time.monotonic() - t0 + self.stop_alarm() + self.check_elapsed_time(dt) + + @unittest.skip('TODO: RUSTPYTHON timed out at the 10 minute mark') + @unittest.skipIf(sys.platform == "darwin", + "poll may fail on macOS; see issue #28087") + @unittest.skipUnless(hasattr(select, 'poll'), 'need select.poll') + def test_poll(self): + poller = select.poll() + + t0 = time.monotonic() + poller.poll(self.sleep_time * 1e3) + dt = time.monotonic() - t0 + self.stop_alarm() + self.check_elapsed_time(dt) + + @unittest.skipUnless(hasattr(select, 'epoll'), 'need select.epoll') + def test_epoll(self): + poller = select.epoll() + self.addCleanup(poller.close) + + t0 = time.monotonic() + poller.poll(self.sleep_time) + dt = time.monotonic() - t0 + self.stop_alarm() + self.check_elapsed_time(dt) + + @unittest.skipUnless(hasattr(select, 'kqueue'), 'need select.kqueue') + def test_kqueue(self): + kqueue = select.kqueue() + self.addCleanup(kqueue.close) + + t0 = time.monotonic() + kqueue.control(None, 1, self.sleep_time) + dt = time.monotonic() - t0 + self.stop_alarm() + self.check_elapsed_time(dt) + + @unittest.skipUnless(hasattr(select, 'devpoll'), 'need select.devpoll') + def test_devpoll(self): + poller = select.devpoll() + self.addCleanup(poller.close) + + t0 = time.monotonic() + poller.poll(self.sleep_time * 1e3) + dt = time.monotonic() - t0 + self.stop_alarm() + self.check_elapsed_time(dt) + + +class FCNTLEINTRTest(EINTRBaseTest): + def _lock(self, lock_func, lock_name): + self.addCleanup(os_helper.unlink, os_helper.TESTFN) + rd1, wr1 = os.pipe() + rd2, wr2 = os.pipe() + for fd in (rd1, wr1, rd2, wr2): + self.addCleanup(os.close, fd) + code = textwrap.dedent(f""" + import fcntl, os, time + with open('{os_helper.TESTFN}', 'wb') as f: + fcntl.{lock_name}(f, fcntl.LOCK_EX) + os.write({wr1}, b"ok") + _ = os.read({rd2}, 2) # wait for parent process + time.sleep({self.sleep_time}) + """) + proc = self.subprocess(code, pass_fds=[wr1, rd2]) + with kill_on_error(proc): + with open(os_helper.TESTFN, 'wb') as f: + # synchronize the subprocess + ok = os.read(rd1, 2) + self.assertEqual(ok, b"ok") + + # notify the child that the parent is ready + start_time = time.monotonic() + os.write(wr2, b"go") + + # the child locked the file just a moment ago for 'sleep_time' seconds + # that means that the lock below will block for 'sleep_time' minus some + # potential context switch delay + lock_func(f, fcntl.LOCK_EX) + dt = time.monotonic() - start_time + self.stop_alarm() + self.check_elapsed_time(dt) + proc.wait() + + @unittest.expectedFailure # TODO: RUSTPYTHON InterruptedError: [Errno 4] Interrupted system call + # Issue 35633: See https://bugs.python.org/issue35633#msg333662 + # skip test rather than accept PermissionError from all platforms + @unittest.skipIf(platform.system() == "AIX", "AIX returns PermissionError") + def test_lockf(self): + self._lock(fcntl.lockf, "lockf") + + @unittest.expectedFailure # TODO: RUSTPYTHON InterruptedError: [Errno 4] Interrupted system call + def test_flock(self): + self._lock(fcntl.flock, "flock") + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index 2fc206f7d53..87cb2ca7895 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -1459,7 +1459,7 @@ def _acquire_release(lock, timeout, l=None, n=1): for _ in range(n): lock.release() - @unittest.skipIf(sys.platform == 'darwin', "TODO: RUSTPYTHON; flaky on darwin") + @unittest.skip("TODO: RUSTPYTHON; flaky timeout") def test_repr_rlock(self): if self.TYPE != 'processes': self.skipTest('test not appropriate for {}'.format(self.TYPE)) @@ -4057,8 +4057,6 @@ def test_heap(self): self.assertEqual(len(heap._allocated_blocks), 0, heap._allocated_blocks) self.assertEqual(len(heap._len_to_seq), 0) - # TODO: RUSTPYTHON - gc.enable() not implemented - @unittest.expectedFailure def test_free_from_gc(self): # Check that freeing of blocks by the garbage collector doesn't deadlock # (issue #12352). @@ -4417,6 +4415,7 @@ def test_shared_memory_across_processes(self): sms.close() + @unittest.skip("TODO: RUSTPYTHON; flaky") @unittest.skipIf(os.name != "posix", "not feasible in non-posix platforms") def test_shared_memory_SharedMemoryServer_ignores_sigint(self): # bpo-36368: protect SharedMemoryManager server process from @@ -6591,13 +6590,15 @@ def tearDownClass(cls): # cycles. Trigger a garbage collection to break these cycles. test.support.gc_collect() - processes = set(multiprocessing.process._dangling) - set(cls.dangling[0]) + # TODO: RUSTPYTHON: Filter out stopped processes since gc.collect() is a no-op + processes = {p for p in multiprocessing.process._dangling if p.is_alive()} - {p for p in cls.dangling[0] if p.is_alive()} if processes: test.support.environment_altered = True support.print_warning(f'Dangling processes: {processes}') processes = None - threads = set(threading._dangling) - set(cls.dangling[1]) + # TODO: RUSTPYTHON: Filter out stopped threads since gc.collect() is a no-op + threads = {t for t in threading._dangling if t.is_alive()} - {t for t in cls.dangling[1] if t.is_alive()} if threads: test.support.environment_altered = True support.print_warning(f'Dangling threads: {threads}') @@ -6787,14 +6788,16 @@ def tearDownModule(): multiprocessing.set_start_method(old_start_method[0], force=True) # pause a bit so we don't get warning about dangling threads/processes - processes = set(multiprocessing.process._dangling) - set(dangling[0]) + # TODO: RUSTPYTHON: Filter out stopped processes since gc.collect() is a no-op + processes = {p for p in multiprocessing.process._dangling if p.is_alive()} - {p for p in dangling[0] if p.is_alive()} if processes: need_sleep = True test.support.environment_altered = True support.print_warning(f'Dangling processes: {processes}') processes = None - threads = set(threading._dangling) - set(dangling[1]) + # TODO: RUSTPYTHON: Filter out stopped threads since gc.collect() is a no-op + threads = {t for t in threading._dangling if t.is_alive()} - {t for t in dangling[1] if t.is_alive()} if threads: need_sleep = True test.support.environment_altered = True diff --git a/Lib/test/autotest.py b/Lib/test/autotest.py new file mode 100644 index 00000000000..b5a1fab404c --- /dev/null +++ b/Lib/test/autotest.py @@ -0,0 +1,5 @@ +# This should be equivalent to running regrtest.py from the cmdline. +# It can be especially handy if you're in an interactive shell, e.g., +# from test import autotest. +from test.libregrtest.main import main +main() diff --git a/Lib/test/certdata/keycert3.pem.reference b/Lib/test/certdata/keycert3.pem.reference new file mode 100644 index 00000000000..84d2ca29953 --- /dev/null +++ b/Lib/test/certdata/keycert3.pem.reference @@ -0,0 +1,15 @@ +{'OCSP': ('http://testca.pythontest.net/testca/ocsp/',), + 'caIssuers': ('http://testca.pythontest.net/testca/pycacert.cer',), + 'crlDistributionPoints': ('http://testca.pythontest.net/testca/revocation.crl',), + 'issuer': ((('countryName', 'XY'),), + (('organizationName', 'Python Software Foundation CA'),), + (('commonName', 'our-ca-server'),)), + 'notAfter': 'Oct 28 14:23:16 2525 GMT', + 'notBefore': 'Aug 29 14:23:16 2018 GMT', + 'serialNumber': 'CB2D80995A69525C', + 'subject': ((('countryName', 'XY'),), + (('localityName', 'Castle Anthrax'),), + (('organizationName', 'Python Software Foundation'),), + (('commonName', 'localhost'),)), + 'subjectAltName': (('DNS', 'localhost'),), + 'version': 3} diff --git a/Lib/test/libregrtest/__init__.py b/Lib/test/libregrtest/__init__.py index 3427b51b60a..8b137891791 100644 --- a/Lib/test/libregrtest/__init__.py +++ b/Lib/test/libregrtest/__init__.py @@ -1,5 +1 @@ -# We import importlib *ASAP* in order to test #15386 -import importlib -from test.libregrtest.cmdline import _parse_args, RESOURCE_NAMES, ALL_RESOURCES -from test.libregrtest.main import main diff --git a/Lib/test/libregrtest/cmdline.py b/Lib/test/libregrtest/cmdline.py index 0a97c8c19b1..e7a12e4d0b6 100644 --- a/Lib/test/libregrtest/cmdline.py +++ b/Lib/test/libregrtest/cmdline.py @@ -1,8 +1,9 @@ import argparse -import os +import os.path +import shlex import sys -from test import support -from test.support import os_helper +from test.support import os_helper, Py_DEBUG +from .utils import ALL_RESOURCES, RESOURCE_NAMES, TestFilter USAGE = """\ @@ -27,8 +28,10 @@ Additional option details: -r randomizes test execution order. You can use --randseed=int to provide an -int seed value for the randomizer; this is useful for reproducing troublesome -test orders. +int seed value for the randomizer. The randseed value will be used +to set seeds for all random usages in tests +(including randomizing the tests order if -r is set). +By default we always set random seed, but do not randomize test order. -s On the first invocation of regrtest using -s, the first test file found or the first test file given on the command line is run, and the name of @@ -41,11 +44,19 @@ doing memory analysis on the Python interpreter, which process tends to consume too many resources to run the full regression test non-stop. --S is used to continue running tests after an aborted run. It will -maintain the order a standard run (ie, this assumes -r is not used). +-S is used to resume running tests after an interrupted run. It will +maintain the order a standard run (i.e. it assumes -r is not used). This is useful after the tests have prematurely stopped for some external -reason and you want to start running from where you left off rather -than starting from the beginning. +reason and you want to resume the run from where you left off rather +than starting from the beginning. Note: this is different from --prioritize. + +--prioritize is used to influence the order of selected tests, such that +the tests listed as an argument are executed first. This is especially +useful when combined with -j and -r to pin the longest-running tests +to start at the beginning of a test run. Pass --prioritize=test_a,test_b +to make test_a run first, followed by test_b, and then the other tests. +If test_a wasn't selected for execution by regular means, --prioritize will +not make it execute. -f reads the names of tests from the file given as f's argument, one or more test names per line. Whitespace is ignored. Blank lines and @@ -84,36 +95,40 @@ The argument is a comma-separated list of words indicating the resources to test. Currently only the following are defined: - all - Enable all special resources. + all - Enable all special resources. + + none - Disable all special resources (this is the default). + + audio - Tests that use the audio device. (There are known + cases of broken audio drivers that can crash Python or + even the Linux kernel.) - none - Disable all special resources (this is the default). + curses - Tests that use curses and will modify the terminal's + state and output modes. - audio - Tests that use the audio device. (There are known - cases of broken audio drivers that can crash Python or - even the Linux kernel.) + largefile - It is okay to run some test that may create huge + files. These tests can take a long time and may + consume >2 GiB of disk space temporarily. - curses - Tests that use curses and will modify the terminal's - state and output modes. + extralargefile - Like 'largefile', but even larger (and slower). - largefile - It is okay to run some test that may create huge - files. These tests can take a long time and may - consume >2 GiB of disk space temporarily. + network - It is okay to run tests that use external network + resource, e.g. testing SSL support for sockets. - network - It is okay to run tests that use external network - resource, e.g. testing SSL support for sockets. + decimal - Test the decimal module against a large suite that + verifies compliance with standards. - decimal - Test the decimal module against a large suite that - verifies compliance with standards. + cpu - Used for certain CPU-heavy tests. - cpu - Used for certain CPU-heavy tests. + walltime - Long running but not CPU-bound tests. - subprocess Run all tests for the subprocess module. + subprocess Run all tests for the subprocess module. - urlfetch - It is okay to download files required on testing. + urlfetch - It is okay to download files required on testing. - gui - Run tests that require a running GUI. + gui - Run tests that require a running GUI. - tzdata - Run tests that require timezone data. + tzdata - Run tests that require timezone data. To enable all resources except one, use '-uall,-'. For example, to run all the tests except for the gui tests, give the @@ -128,17 +143,53 @@ """ -ALL_RESOURCES = ('audio', 'curses', 'largefile', 'network', - 'decimal', 'cpu', 'subprocess', 'urlfetch', 'gui') +class Namespace(argparse.Namespace): + def __init__(self, **kwargs) -> None: + self.ci = False + self.testdir = None + self.verbose = 0 + self.quiet = False + self.exclude = False + self.cleanup = False + self.wait = False + self.list_cases = False + self.list_tests = False + self.single = False + self.randomize = False + self.fromfile = None + self.fail_env_changed = False + self.use_resources: list[str] = [] + self.trace = False + self.coverdir = 'coverage' + self.runleaks = False + self.huntrleaks: tuple[int, int, str] | None = None + self.rerun = False + self.verbose3 = False + self.print_slow = False + self.random_seed = None + self.use_mp = None + self.parallel_threads = None + self.forever = False + self.header = False + self.failfast = False + self.match_tests: TestFilter = [] + self.pgo = False + self.pgo_extended = False + self.tsan = False + self.tsan_parallel = False + self.worker_json = None + self.start = None + self.timeout = None + self.memlimit = None + self.threshold = None + self.fail_rerun = False + self.tempdir = None + self._add_python_opts = True + self.xmlpath = None + self.single_process = False + + super().__init__(**kwargs) -# Other resources excluded from --use=all: -# -# - extralagefile (ex: test_zipfile64): really too slow to be enabled -# "by default" -# - tzdata: while needed to validate fully test_datetime, it makes -# test_datetime too slow (15-20 min on some buildbots) and so is disabled by -# default (see bpo-30822). -RESOURCE_NAMES = ALL_RESOURCES + ('extralargefile', 'tzdata') class _ArgParser(argparse.ArgumentParser): @@ -146,6 +197,20 @@ def error(self, message): super().error(message + "\nPass -h or --help for complete help.") +class FilterAction(argparse.Action): + def __call__(self, parser, namespace, value, option_string=None): + items = getattr(namespace, self.dest) + items.append((value, self.const)) + + +class FromFileFilterAction(argparse.Action): + def __call__(self, parser, namespace, value, option_string=None): + items = getattr(namespace, self.dest) + with open(value, encoding='utf-8') as fp: + for line in fp: + items.append((line.strip(), self.const)) + + def _create_parser(): # Set prog to prevent the uninformative "__main__.py" from displaying in # error messages when using "python -m test ...". @@ -155,6 +220,7 @@ def _create_parser(): epilog=EPILOG, add_help=False, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.set_defaults(match_tests=[]) # Arguments with this clause added to its help are described further in # the epilog's "Additional option details" section. @@ -164,23 +230,35 @@ def _create_parser(): # We add help explicitly to control what argument group it renders under. group.add_argument('-h', '--help', action='help', help='show this help message and exit') - group.add_argument('--timeout', metavar='TIMEOUT', type=float, + group.add_argument('--fast-ci', action='store_true', + help='Fast Continuous Integration (CI) mode used by ' + 'GitHub Actions') + group.add_argument('--slow-ci', action='store_true', + help='Slow Continuous Integration (CI) mode used by ' + 'buildbot workers') + group.add_argument('--timeout', metavar='TIMEOUT', help='dump the traceback and exit if a test takes ' 'more than TIMEOUT seconds; disabled if TIMEOUT ' 'is negative or equals to zero') group.add_argument('--wait', action='store_true', help='wait for user input, e.g., allow a debugger ' 'to be attached') - group.add_argument('--worker-args', metavar='ARGS') group.add_argument('-S', '--start', metavar='START', - help='the name of the test at which to start.' + + help='resume an interrupted run at the following test.' + more_details) + group.add_argument('-p', '--python', metavar='PYTHON', + help='Command to run Python test subprocesses with.') + group.add_argument('--randseed', metavar='SEED', + dest='random_seed', type=int, + help='pass a global random seed') group = parser.add_argument_group('Verbosity') group.add_argument('-v', '--verbose', action='count', help='run tests in verbose mode with output to stdout') - group.add_argument('-w', '--verbose2', action='store_true', + group.add_argument('-w', '--rerun', action='store_true', help='re-run failed tests in verbose mode') + group.add_argument('--verbose2', action='store_true', dest='rerun', + help='deprecated alias to --rerun') group.add_argument('-W', '--verbose3', action='store_true', help='display test output on failure') group.add_argument('-q', '--quiet', action='store_true', @@ -193,10 +271,13 @@ def _create_parser(): group = parser.add_argument_group('Selecting tests') group.add_argument('-r', '--randomize', action='store_true', help='randomize test execution order.' + more_details) - group.add_argument('--randseed', metavar='SEED', - dest='random_seed', type=int, - help='pass a random seed to reproduce a previous ' - 'random run') + group.add_argument('--no-randomize', dest='no_randomize', action='store_true', + help='do not randomize test execution order, even if ' + 'it would be implied by another option') + group.add_argument('--prioritize', metavar='TEST1,TEST2,...', + action='append', type=priority_list, + help='select these tests first, even if the order is' + ' randomized.' + more_details) group.add_argument('-f', '--fromfile', metavar='FILE', help='read names of tests to run from a file.' + more_details) @@ -206,12 +287,21 @@ def _create_parser(): help='single step through a set of tests.' + more_details) group.add_argument('-m', '--match', metavar='PAT', - dest='match_tests', action='append', + dest='match_tests', action=FilterAction, const=True, help='match test cases and methods with glob pattern PAT') + group.add_argument('-i', '--ignore', metavar='PAT', + dest='match_tests', action=FilterAction, const=False, + help='ignore test cases and methods with glob pattern PAT') group.add_argument('--matchfile', metavar='FILENAME', - dest='match_filename', + dest='match_tests', + action=FromFileFilterAction, const=True, help='similar to --match but get patterns from a ' 'text file, one pattern per line') + group.add_argument('--ignorefile', metavar='FILENAME', + dest='match_tests', + action=FromFileFilterAction, const=False, + help='similar to --matchfile but it receives patterns ' + 'from text file to ignore') group.add_argument('-G', '--failfast', action='store_true', help='fail as soon as a test fails (only with -v or -W)') group.add_argument('-u', '--use', metavar='RES1,RES2,...', @@ -227,9 +317,6 @@ def _create_parser(): '(instead of the Python stdlib test suite)') group = parser.add_argument_group('Special runs') - group.add_argument('-l', '--findleaks', action='store_const', const=2, - default=1, - help='deprecated alias to --fail-env-changed') group.add_argument('-L', '--runleaks', action='store_true', help='run the leaks(1) command just before exit.' + more_details) @@ -240,6 +327,16 @@ def _create_parser(): group.add_argument('-j', '--multiprocess', metavar='PROCESSES', dest='use_mp', type=int, help='run PROCESSES processes at once') + group.add_argument('--single-process', action='store_true', + dest='single_process', + help='always run all tests sequentially in ' + 'a single process, ignore -jN option, ' + 'and failed tests are also rerun sequentially ' + 'in the same process') + group.add_argument('--parallel-threads', metavar='PARALLEL_THREADS', + type=int, + help='run copies of each test in PARALLEL_THREADS at ' + 'once') group.add_argument('-T', '--coverage', action='store_true', dest='trace', help='turn on code coverage tracing using the trace ' @@ -257,7 +354,7 @@ def _create_parser(): help='suppress error message boxes on Windows') group.add_argument('-F', '--forever', action='store_true', help='run the specified tests in a loop, until an ' - 'error happens') + 'error happens; imply --failfast') group.add_argument('--list-tests', action='store_true', help="only write the name of tests that will be run, " "don't execute them") @@ -265,16 +362,33 @@ def _create_parser(): help='only write the name of test cases that will be run' ' , don\'t execute them') group.add_argument('-P', '--pgo', dest='pgo', action='store_true', - help='enable Profile Guided Optimization training') + help='enable Profile Guided Optimization (PGO) training') + group.add_argument('--pgo-extended', action='store_true', + help='enable extended PGO training (slower training)') + group.add_argument('--tsan', dest='tsan', action='store_true', + help='run a subset of test cases that are proper for the TSAN test') + group.add_argument('--tsan-parallel', action='store_true', + help='run a subset of test cases that are appropriate ' + 'for TSAN with `--parallel-threads=N`') group.add_argument('--fail-env-changed', action='store_true', help='if a test file alters the environment, mark ' 'the test as failed') + group.add_argument('--fail-rerun', action='store_true', + help='if a test failed and then passed when re-run, ' + 'mark the tests as failed') group.add_argument('--junit-xml', dest='xmlpath', metavar='FILENAME', help='writes JUnit-style XML results to the specified ' 'file') - group.add_argument('--tempdir', dest='tempdir', metavar='PATH', + group.add_argument('--tempdir', metavar='PATH', help='override the working directory for the test run') + group.add_argument('--cleanup', action='store_true', + help='remove old test_python_* directories') + group.add_argument('--bisect', action='store_true', + help='if some tests fail, run test.bisect_cmd on them') + group.add_argument('--dont-add-python-opts', dest='_add_python_opts', + action='store_false', + help="internal option, don't use it") return parser @@ -307,21 +421,18 @@ def resources_list(string): return u +def priority_list(string): + return string.split(",") + + def _parse_args(args, **kwargs): # Defaults - ns = argparse.Namespace(testdir=None, verbose=0, quiet=False, - exclude=False, single=False, randomize=False, fromfile=None, - findleaks=1, use_resources=None, trace=False, coverdir='coverage', - runleaks=False, huntrleaks=False, verbose2=False, print_slow=False, - random_seed=None, use_mp=None, verbose3=False, forever=False, - header=False, failfast=False, match_tests=None, pgo=False) + ns = Namespace() for k, v in kwargs.items(): if not hasattr(ns, k): raise TypeError('%r is an invalid keyword argument ' 'for this function' % k) setattr(ns, k, v) - if ns.use_resources is None: - ns.use_resources = [] parser = _create_parser() # Issue #14191: argparse doesn't support "intermixed" positional and @@ -330,19 +441,81 @@ def _parse_args(args, **kwargs): for arg in ns.args: if arg.startswith('-'): parser.error("unrecognized arguments: %s" % arg) - sys.exit(1) - if ns.findleaks > 1: - # --findleaks implies --fail-env-changed + if ns.timeout is not None: + # Support "--timeout=" (no value) so Makefile.pre.pre TESTTIMEOUT + # can be used by "make buildbottest" and "make test". + if ns.timeout != "": + try: + ns.timeout = float(ns.timeout) + except ValueError: + parser.error(f"invalid timeout value: {ns.timeout!r}") + else: + ns.timeout = None + + # Continuous Integration (CI): common options for fast/slow CI modes + if ns.slow_ci or ns.fast_ci: + # Similar to options: + # -j0 --randomize --fail-env-changed --rerun --slowest --verbose3 + if ns.use_mp is None: + ns.use_mp = 0 + ns.randomize = True ns.fail_env_changed = True + if ns.python is None: + ns.rerun = True + ns.print_slow = True + if not ns.verbose: + ns.verbose3 = True + else: + # --verbose has the priority over --verbose3 + pass + else: + ns._add_python_opts = False + + # --singleprocess overrides -jN option + if ns.single_process: + ns.use_mp = None + + # When both --slow-ci and --fast-ci options are present, + # --slow-ci has the priority + if ns.slow_ci: + # Similar to: -u "all" --timeout=1200 + if ns.use is None: + ns.use = [] + ns.use.insert(0, ['all']) + if ns.timeout is None: + ns.timeout = 1200 # 20 minutes + elif ns.fast_ci: + # Similar to: -u "all,-cpu" --timeout=600 + if ns.use is None: + ns.use = [] + ns.use.insert(0, ['all', '-cpu']) + if ns.timeout is None: + ns.timeout = 600 # 10 minutes + if ns.single and ns.fromfile: parser.error("-s and -f don't go together!") - if ns.use_mp is not None and ns.trace: - parser.error("-T and -j don't go together!") + if ns.trace: + if ns.use_mp is not None: + if not Py_DEBUG: + parser.error("need --with-pydebug to use -T and -j together") + else: + print( + "Warning: collecting coverage without -j is imprecise. Configure" + " --with-pydebug and run -m test -T -j for best results.", + file=sys.stderr + ) + if ns.python is not None: + if ns.use_mp is None: + parser.error("-p requires -j!") + # The "executable" may be two or more parts, e.g. "node python.js" + ns.python = shlex.split(ns.python) if ns.failfast and not (ns.verbose or ns.verbose3): parser.error("-G/--failfast needs either -v or -W") - if ns.pgo and (ns.verbose or ns.verbose2 or ns.verbose3): + if ns.pgo and (ns.verbose or ns.rerun or ns.verbose3): parser.error("--pgo/-v don't go together!") + if ns.pgo_extended: + ns.pgo = True # pgo_extended implies pgo if ns.nowindows: print("Warning: the --nowindows (-n) option is deprecated. " @@ -353,10 +526,6 @@ def _parse_args(args, **kwargs): if ns.timeout is not None: if ns.timeout <= 0: ns.timeout = None - if ns.use_mp is not None: - if ns.use_mp <= 0: - # Use all cores + extras for tests that like to sleep - ns.use_mp = 2 + (os.cpu_count() or 1) if ns.use: for a in ns.use: for r in a: @@ -377,18 +546,40 @@ def _parse_args(args, **kwargs): ns.use_resources.append(r) if ns.random_seed is not None: ns.randomize = True + if ns.no_randomize: + ns.randomize = False if ns.verbose: ns.header = True - if ns.huntrleaks and ns.verbose3: + + # When -jN option is used, a worker process does not use --verbose3 + # and so -R 3:3 -jN --verbose3 just works as expected: there is no false + # alarm about memory leak. + if ns.huntrleaks and ns.verbose3 and ns.use_mp is None: + # run_single_test() replaces sys.stdout with io.StringIO if verbose3 + # is true. In this case, huntrleaks sees an write into StringIO as + # a memory leak, whereas it is not (gh-71290). ns.verbose3 = False print("WARNING: Disable --verbose3 because it's incompatible with " - "--huntrleaks: see http://bugs.python.org/issue27103", + "--huntrleaks without -jN option", file=sys.stderr) - if ns.match_filename: - if ns.match_tests is None: - ns.match_tests = [] - with open(ns.match_filename) as fp: - for line in fp: - ns.match_tests.append(line.strip()) + + if ns.forever: + # --forever implies --failfast + ns.failfast = True + + if ns.huntrleaks: + warmup, repetitions, _ = ns.huntrleaks + if warmup < 1 or repetitions < 1: + msg = ("Invalid values for the --huntrleaks/-R parameters. The " + "number of warmups and repetitions must be at least 1 " + "each (1:1).") + print(msg, file=sys.stderr, flush=True) + sys.exit(2) + + ns.prioritize = [ + test + for test_list in (ns.prioritize or ()) + for test in test_list + ] return ns diff --git a/Lib/test/libregrtest/filter.py b/Lib/test/libregrtest/filter.py new file mode 100644 index 00000000000..41372e427ff --- /dev/null +++ b/Lib/test/libregrtest/filter.py @@ -0,0 +1,77 @@ +import itertools +import operator +import re + + +# By default, don't filter tests +_test_matchers = () +_test_patterns = () + + +def match_test(test): + # Function used by support.run_unittest() and regrtest --list-cases + result = False + for matcher, result in reversed(_test_matchers): + if matcher(test.id()): + return result + return not result + + +def _is_full_match_test(pattern): + # If a pattern contains at least one dot, it's considered + # as a full test identifier. + # Example: 'test.test_os.FileTests.test_access'. + # + # ignore patterns which contain fnmatch patterns: '*', '?', '[...]' + # or '[!...]'. For example, ignore 'test_access*'. + return ('.' in pattern) and (not re.search(r'[?*\[\]]', pattern)) + + +def get_match_tests(): + global _test_patterns + return _test_patterns + + +def set_match_tests(patterns): + global _test_matchers, _test_patterns + + if not patterns: + _test_matchers = () + _test_patterns = () + else: + itemgetter = operator.itemgetter + patterns = tuple(patterns) + if patterns != _test_patterns: + _test_matchers = [ + (_compile_match_function(map(itemgetter(0), it)), result) + for result, it in itertools.groupby(patterns, itemgetter(1)) + ] + _test_patterns = patterns + + +def _compile_match_function(patterns): + patterns = list(patterns) + + if all(map(_is_full_match_test, patterns)): + # Simple case: all patterns are full test identifier. + # The test.bisect_cmd utility only uses such full test identifiers. + return set(patterns).__contains__ + else: + import fnmatch + regex = '|'.join(map(fnmatch.translate, patterns)) + # The search *is* case sensitive on purpose: + # don't use flags=re.IGNORECASE + regex_match = re.compile(regex).match + + def match_test_regex(test_id, regex_match=regex_match): + if regex_match(test_id): + # The regex matches the whole identifier, for example + # 'test.test_os.FileTests.test_access'. + return True + else: + # Try to match parts of the test identifier. + # For example, split 'test.test_os.FileTests.test_access' + # into: 'test', 'test_os', 'FileTests' and 'test_access'. + return any(map(regex_match, test_id.split("."))) + + return match_test_regex diff --git a/Lib/test/libregrtest/findtests.py b/Lib/test/libregrtest/findtests.py new file mode 100644 index 00000000000..f01c1240774 --- /dev/null +++ b/Lib/test/libregrtest/findtests.py @@ -0,0 +1,110 @@ +import os +import sys +import unittest +from collections.abc import Container + +from test import support + +from .filter import match_test, set_match_tests +from .utils import ( + StrPath, TestName, TestTuple, TestList, TestFilter, + abs_module_name, count, printlist) + + +# If these test directories are encountered recurse into them and treat each +# "test_*.py" file or each sub-directory as a separate test module. This can +# increase parallelism. +# +# Beware this can't generally be done for any directory with sub-tests as the +# __init__.py may do things which alter what tests are to be run. +SPLITTESTDIRS: set[TestName] = { + "test_asyncio", + "test_concurrent_futures", + "test_doctests", + "test_future_stmt", + "test_gdb", + "test_inspect", + "test_pydoc", + "test_multiprocessing_fork", + "test_multiprocessing_forkserver", + "test_multiprocessing_spawn", +} + + +def findtestdir(path: StrPath | None = None) -> StrPath: + return path or os.path.dirname(os.path.dirname(__file__)) or os.curdir + + +def findtests(*, testdir: StrPath | None = None, exclude: Container[str] = (), + split_test_dirs: set[TestName] = SPLITTESTDIRS, + base_mod: str = "") -> TestList: + """Return a list of all applicable test modules.""" + testdir = findtestdir(testdir) + tests = [] + for name in os.listdir(testdir): + mod, ext = os.path.splitext(name) + if (not mod.startswith("test_")) or (mod in exclude): + continue + if base_mod: + fullname = f"{base_mod}.{mod}" + else: + fullname = mod + if fullname in split_test_dirs: + subdir = os.path.join(testdir, mod) + if not base_mod: + fullname = f"test.{mod}" + tests.extend(findtests(testdir=subdir, exclude=exclude, + split_test_dirs=split_test_dirs, + base_mod=fullname)) + elif ext in (".py", ""): + tests.append(fullname) + return sorted(tests) + + +def split_test_packages(tests, *, testdir: StrPath | None = None, + exclude: Container[str] = (), + split_test_dirs=SPLITTESTDIRS) -> list[TestName]: + testdir = findtestdir(testdir) + splitted = [] + for name in tests: + if name in split_test_dirs: + subdir = os.path.join(testdir, name) + splitted.extend(findtests(testdir=subdir, exclude=exclude, + split_test_dirs=split_test_dirs, + base_mod=name)) + else: + splitted.append(name) + return splitted + + +def _list_cases(suite: unittest.TestSuite) -> None: + for test in suite: + if isinstance(test, unittest.loader._FailedTest): # type: ignore[attr-defined] + continue + if isinstance(test, unittest.TestSuite): + _list_cases(test) + elif isinstance(test, unittest.TestCase): + if match_test(test): + print(test.id()) + +def list_cases(tests: TestTuple, *, + match_tests: TestFilter | None = None, + test_dir: StrPath | None = None) -> None: + support.verbose = False + set_match_tests(match_tests) + + skipped = [] + for test_name in tests: + module_name = abs_module_name(test_name, test_dir) + try: + suite = unittest.defaultTestLoader.loadTestsFromName(module_name) + _list_cases(suite) + except unittest.SkipTest: + skipped.append(test_name) + + if skipped: + sys.stdout.flush() + stderr = sys.stderr + print(file=stderr) + print(count(len(skipped), "test"), "skipped:", file=stderr) + printlist(skipped, file=stderr) diff --git a/Lib/test/libregrtest/logger.py b/Lib/test/libregrtest/logger.py new file mode 100644 index 00000000000..fa1d4d575c8 --- /dev/null +++ b/Lib/test/libregrtest/logger.py @@ -0,0 +1,89 @@ +import os +import time + +from test.support import MS_WINDOWS +from .results import TestResults +from .runtests import RunTests +from .utils import print_warning + +if MS_WINDOWS: + from .win_utils import WindowsLoadTracker + + +class Logger: + def __init__(self, results: TestResults, quiet: bool, pgo: bool): + self.start_time = time.perf_counter() + self.test_count_text = '' + self.test_count_width = 3 + self.win_load_tracker: WindowsLoadTracker | None = None + self._results: TestResults = results + self._quiet: bool = quiet + self._pgo: bool = pgo + + def log(self, line: str = '') -> None: + empty = not line + + # add the system load prefix: "load avg: 1.80 " + load_avg = self.get_load_avg() + if load_avg is not None: + line = f"load avg: {load_avg:.2f} {line}" + + # add the timestamp prefix: "0:01:05 " + log_time = time.perf_counter() - self.start_time + + mins, secs = divmod(int(log_time), 60) + hours, mins = divmod(mins, 60) + formatted_log_time = "%d:%02d:%02d" % (hours, mins, secs) + + line = f"{formatted_log_time} {line}" + if empty: + line = line[:-1] + + print(line, flush=True) + + def get_load_avg(self) -> float | None: + if hasattr(os, 'getloadavg'): + try: + return os.getloadavg()[0] + except OSError: + pass + if self.win_load_tracker is not None: + return self.win_load_tracker.getloadavg() + return None + + def display_progress(self, test_index: int, text: str) -> None: + if self._quiet: + return + results = self._results + + # "[ 51/405/1] test_tcl passed" + line = f"{test_index:{self.test_count_width}}{self.test_count_text}" + fails = len(results.bad) + len(results.env_changed) + if fails and not self._pgo: + line = f"{line}/{fails}" + self.log(f"[{line}] {text}") + + def set_tests(self, runtests: RunTests) -> None: + if runtests.forever: + self.test_count_text = '' + self.test_count_width = 3 + else: + self.test_count_text = '/{}'.format(len(runtests.tests)) + self.test_count_width = len(self.test_count_text) - 1 + + def start_load_tracker(self) -> None: + if not MS_WINDOWS: + return + + try: + self.win_load_tracker = WindowsLoadTracker() + except PermissionError as error: + # Standard accounts may not have access to the performance + # counters. + print_warning(f'Failed to create WindowsLoadTracker: {error}') + + def stop_load_tracker(self) -> None: + if self.win_load_tracker is None: + return + self.win_load_tracker.close() + self.win_load_tracker = None diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py index e1d19e1e4ac..0fc2548789e 100644 --- a/Lib/test/libregrtest/main.py +++ b/Lib/test/libregrtest/main.py @@ -1,42 +1,33 @@ -import datetime -import faulthandler -import json -import locale import os -import platform import random import re +import shlex import sys import sysconfig -import tempfile import time -import unittest -from test.libregrtest.cmdline import _parse_args -from test.libregrtest.runtest import ( - findtests, runtest, get_abs_module, - STDTESTS, NOTTESTS, PASSED, FAILED, ENV_CHANGED, SKIPPED, RESOURCE_DENIED, - INTERRUPTED, CHILD_ERROR, TEST_DID_NOT_RUN, - PROGRESS_MIN_TIME, format_test_result) -from test.libregrtest.setup import setup_tests -from test.libregrtest.utils import removepy, count, format_duration, printlist -from test import support -from test.support import os_helper, import_helper - - -# When tests are run from the Python build directory, it is best practice -# to keep the test files in a subfolder. This eases the cleanup of leftover -# files using the "make distclean" command. -if sysconfig.is_python_build(): - TEMPDIR = sysconfig.get_config_var('abs_builddir') - if TEMPDIR is None: - # bpo-30284: On Windows, only srcdir is available. Using abs_builddir - # mostly matters on UNIX when building Python out of the source tree, - # especially when the source tree is read only. - TEMPDIR = sysconfig.get_config_var('srcdir') - TEMPDIR = os.path.join(TEMPDIR, 'build') -else: - TEMPDIR = tempfile.gettempdir() -TEMPDIR = os.path.abspath(TEMPDIR) +import trace +from _colorize import get_colors # type: ignore[import-not-found] +from typing import NoReturn + +from test.support import os_helper, MS_WINDOWS, flush_std_streams + +from .cmdline import _parse_args, Namespace +from .findtests import findtests, split_test_packages, list_cases +from .logger import Logger +from .pgo import setup_pgo_tests +from .result import TestResult +from .results import TestResults, EXITCODE_INTERRUPTED +from .runtests import RunTests, HuntRefleak +from .setup import setup_process, setup_test_dir +from .single import run_single_test, PROGRESS_MIN_TIME +from .tsan import setup_tsan_tests, setup_tsan_parallel_tests +from .utils import ( + StrPath, StrJSON, TestName, TestList, TestTuple, TestFilter, + strip_py_suffix, count, format_duration, + printlist, get_temp_dir, get_work_dir, exit_timeout, + display_header, cleanup_temp_dir, print_warning, + is_cross_compiled, get_host_runner, + EXIT_TIMEOUT) class Regrtest: @@ -57,357 +48,379 @@ class Regrtest: files beginning with test_ will be used. The other default arguments (verbose, quiet, exclude, - single, randomize, findleaks, use_resources, trace, coverdir, + single, randomize, use_resources, trace, coverdir, print_slow, and random_seed) allow programmers calling main() directly to set the values that would normally be set by flags on the command line. """ - def __init__(self): - # Namespace of command line options - self.ns = None - - # tests - self.tests = [] - self.selected = [] - - # test results - self.good = [] - self.bad = [] - self.skipped = [] - self.resource_denieds = [] - self.environment_changed = [] - self.run_no_tests = [] - self.rerun = [] - self.first_result = None - self.interrupted = False - - # used by --slow - self.test_times = [] - - # used by --coverage, trace.Trace instance - self.tracer = None - - # used to display the progress bar "[ 3/100]" - self.start_time = time.monotonic() - self.test_count = '' - self.test_count_width = 1 - - # used by --single - self.next_single_test = None - self.next_single_filename = None - - # used by --junit-xml - self.testsuite_xml = None - - self.win_load_tracker = None - - def get_executed(self): - return (set(self.good) | set(self.bad) | set(self.skipped) - | set(self.resource_denieds) | set(self.environment_changed) - | set(self.run_no_tests)) - - def accumulate_result(self, result, rerun=False): - test_name = result.test_name - ok = result.result - - if ok not in (CHILD_ERROR, INTERRUPTED) and not rerun: - self.test_times.append((result.test_time, test_name)) - - if ok == PASSED: - self.good.append(test_name) - elif ok in (FAILED, CHILD_ERROR): - if not rerun: - self.bad.append(test_name) - elif ok == ENV_CHANGED: - self.environment_changed.append(test_name) - elif ok == SKIPPED: - self.skipped.append(test_name) - elif ok == RESOURCE_DENIED: - self.skipped.append(test_name) - self.resource_denieds.append(test_name) - elif ok == TEST_DID_NOT_RUN: - self.run_no_tests.append(test_name) - elif ok == INTERRUPTED: - self.interrupted = True + def __init__(self, ns: Namespace, _add_python_opts: bool = False): + # Log verbosity + self.verbose: int = int(ns.verbose) + self.quiet: bool = ns.quiet + self.pgo: bool = ns.pgo + self.pgo_extended: bool = ns.pgo_extended + self.tsan: bool = ns.tsan + self.tsan_parallel: bool = ns.tsan_parallel + + # Test results + self.results: TestResults = TestResults() + self.first_state: str | None = None + + # Logger + self.logger = Logger(self.results, self.quiet, self.pgo) + + # Actions + self.want_header: bool = ns.header + self.want_list_tests: bool = ns.list_tests + self.want_list_cases: bool = ns.list_cases + self.want_wait: bool = ns.wait + self.want_cleanup: bool = ns.cleanup + self.want_rerun: bool = ns.rerun + self.want_run_leaks: bool = ns.runleaks + self.want_bisect: bool = ns.bisect + + self.ci_mode: bool = (ns.fast_ci or ns.slow_ci) + self.want_add_python_opts: bool = (_add_python_opts + and ns._add_python_opts) + + # Select tests + self.match_tests: TestFilter = ns.match_tests + self.exclude: bool = ns.exclude + self.fromfile: StrPath | None = ns.fromfile + self.starting_test: TestName | None = ns.start + self.cmdline_args: TestList = ns.args + + # Workers + self.single_process: bool = ns.single_process + if self.single_process or ns.use_mp is None: + num_workers = 0 # run sequentially in a single process + elif ns.use_mp <= 0: + num_workers = -1 # run in parallel, use the number of CPUs else: - raise ValueError("invalid test result: %r" % ok) - - if rerun and ok not in {FAILED, CHILD_ERROR, INTERRUPTED}: - self.bad.remove(test_name) - - xml_data = result.xml_data - if xml_data: - import xml.etree.ElementTree as ET - for e in xml_data: - try: - self.testsuite_xml.append(ET.fromstring(e)) - except ET.ParseError: - print(xml_data, file=sys.__stderr__) - raise - - def display_progress(self, test_index, text): - if self.ns.quiet: - return - - # "[ 51/405/1] test_tcl passed" - line = f"{test_index:{self.test_count_width}}{self.test_count}" - fails = len(self.bad) + len(self.environment_changed) - if fails and not self.ns.pgo: - line = f"{line}/{fails}" - line = f"[{line}] {text}" - - # add the system load prefix: "load avg: 1.80 " - load_avg = self.getloadavg() - if load_avg is not None: - line = f"load avg: {load_avg:.2f} {line}" - - # add the timestamp prefix: "0:01:05 " - test_time = time.monotonic() - self.start_time - test_time = datetime.timedelta(seconds=int(test_time)) - line = f"{test_time} {line}" - print(line, flush=True) - - def parse_args(self, kwargs): - ns = _parse_args(sys.argv[1:], **kwargs) - - if ns.timeout and not hasattr(faulthandler, 'dump_traceback_later'): - print("Warning: The timeout option requires " - "faulthandler.dump_traceback_later", file=sys.stderr) - ns.timeout = None + num_workers = ns.use_mp # run in parallel + self.num_workers: int = num_workers + self.worker_json: StrJSON | None = ns.worker_json + + # Options to run tests + self.fail_fast: bool = ns.failfast + self.fail_env_changed: bool = ns.fail_env_changed + self.fail_rerun: bool = ns.fail_rerun + self.forever: bool = ns.forever + self.output_on_failure: bool = ns.verbose3 + self.timeout: float | None = ns.timeout + if ns.huntrleaks: + warmups, runs, filename = ns.huntrleaks + filename = os.path.abspath(filename) + self.hunt_refleak: HuntRefleak | None = HuntRefleak(warmups, runs, filename) + else: + self.hunt_refleak = None + self.test_dir: StrPath | None = ns.testdir + self.junit_filename: StrPath | None = ns.xmlpath + self.memory_limit: str | None = ns.memlimit + self.gc_threshold: int | None = ns.threshold + self.use_resources: tuple[str, ...] = tuple(ns.use_resources) + if ns.python: + self.python_cmd: tuple[str, ...] | None = tuple(ns.python) + else: + self.python_cmd = None + self.coverage: bool = ns.trace + self.coverage_dir: StrPath | None = ns.coverdir + self._tmp_dir: StrPath | None = ns.tempdir + + # Randomize + self.randomize: bool = ns.randomize + if ('SOURCE_DATE_EPOCH' in os.environ + # don't use the variable if empty + and os.environ['SOURCE_DATE_EPOCH'] + ): + self.randomize = False + # SOURCE_DATE_EPOCH should be an integer, but use a string to not + # fail if it's not integer. random.seed() accepts a string. + # https://reproducible-builds.org/docs/source-date-epoch/ + self.random_seed: int | str = os.environ['SOURCE_DATE_EPOCH'] + elif ns.random_seed is None: + self.random_seed = random.getrandbits(32) + else: + self.random_seed = ns.random_seed + self.prioritize_tests: tuple[str, ...] = tuple(ns.prioritize) - if ns.xmlpath: - support.junit_xml_list = self.testsuite_xml = [] + self.parallel_threads = ns.parallel_threads - # Strip .py extensions. - removepy(ns.args) + # tests + self.first_runtests: RunTests | None = None - return ns + # used by --slowest + self.print_slowest: bool = ns.print_slow - def find_tests(self, tests): - self.tests = tests + # used to display the progress bar "[ 3/100]" + self.start_time = time.perf_counter() - if self.ns.single: - self.next_single_filename = os.path.join(TEMPDIR, 'pynexttest') + # used by --single + self.single_test_run: bool = ns.single + self.next_single_test: TestName | None = None + self.next_single_filename: StrPath | None = None + + def log(self, line: str = '') -> None: + self.logger.log(line) + + def find_tests(self, tests: TestList | None = None) -> tuple[TestTuple, TestList | None]: + if tests is None: + tests = [] + if self.single_test_run: + self.next_single_filename = os.path.join(self.tmp_dir, 'pynexttest') try: with open(self.next_single_filename, 'r') as fp: next_test = fp.read().strip() - self.tests = [next_test] + tests = [next_test] except OSError: pass - if self.ns.fromfile: - self.tests = [] + if self.fromfile: + tests = [] # regex to match 'test_builtin' in line: # '0:00:00 [ 4/400] test_builtin -- test_dict took 1 sec' regex = re.compile(r'\btest_[a-zA-Z0-9_]+\b') - with open(os.path.join(os_helper.SAVEDCWD, self.ns.fromfile)) as fp: + with open(os.path.join(os_helper.SAVEDCWD, self.fromfile)) as fp: for line in fp: line = line.split('#', 1)[0] line = line.strip() match = regex.search(line) if match is not None: - self.tests.append(match.group()) - - removepy(self.tests) - - stdtests = STDTESTS[:] - nottests = NOTTESTS.copy() - if self.ns.exclude: - for arg in self.ns.args: - if arg in stdtests: - stdtests.remove(arg) - nottests.add(arg) - self.ns.args = [] - - # if testdir is set, then we are not running the python tests suite, so - # don't add default tests to be executed or skipped (pass empty values) - if self.ns.testdir: - alltests = findtests(self.ns.testdir, list(), set()) - else: - alltests = findtests(self.ns.testdir, stdtests, nottests) + tests.append(match.group()) - if not self.ns.fromfile: - self.selected = self.tests or self.ns.args or alltests + strip_py_suffix(tests) + + exclude_tests = set() + if self.exclude: + for arg in self.cmdline_args: + exclude_tests.add(arg) + self.cmdline_args = [] + + if self.pgo: + # add default PGO tests if no tests are specified + setup_pgo_tests(self.cmdline_args, self.pgo_extended) + + if self.tsan: + setup_tsan_tests(self.cmdline_args) + + if self.tsan_parallel: + setup_tsan_parallel_tests(self.cmdline_args) + + alltests = findtests(testdir=self.test_dir, + exclude=exclude_tests) + + if not self.fromfile: + selected = tests or self.cmdline_args + if exclude_tests: + # Support "--pgo/--tsan -x test_xxx" command + selected = [name for name in selected + if name not in exclude_tests] + if selected: + selected = split_test_packages(selected) + else: + selected = alltests else: - self.selected = self.tests - if self.ns.single: - self.selected = self.selected[:1] + selected = tests + + if self.single_test_run: + selected = selected[:1] try: - pos = alltests.index(self.selected[0]) + pos = alltests.index(selected[0]) self.next_single_test = alltests[pos + 1] except IndexError: pass # Remove all the selected tests that precede start if it's set. - if self.ns.start: + if self.starting_test: try: - del self.selected[:self.selected.index(self.ns.start)] + del selected[:selected.index(self.starting_test)] except ValueError: - print("Couldn't find starting test (%s), using all tests" - % self.ns.start, file=sys.stderr) + print(f"Cannot find starting test: {self.starting_test}") + sys.exit(1) - if self.ns.randomize: - if self.ns.random_seed is None: - self.ns.random_seed = random.randrange(10000000) - random.seed(self.ns.random_seed) - random.shuffle(self.selected) + random.seed(self.random_seed) + if self.randomize: + random.shuffle(selected) - def list_tests(self): - for name in self.selected: - print(name) - - def _list_cases(self, suite): - for test in suite: - if isinstance(test, unittest.loader._FailedTest): - continue - if isinstance(test, unittest.TestSuite): - self._list_cases(test) - elif isinstance(test, unittest.TestCase): - if support.match_test(test): - print(test.id()) - - def list_cases(self): - support.verbose = False - support.set_match_tests(self.ns.match_tests) - - for test_name in self.selected: - abstest = get_abs_module(self.ns, test_name) + for priority_test in reversed(self.prioritize_tests): try: - suite = unittest.defaultTestLoader.loadTestsFromName(abstest) - self._list_cases(suite) - except unittest.SkipTest: - self.skipped.append(test_name) + selected.remove(priority_test) + except ValueError: + print(f"warning: --prioritize={priority_test} used" + f" but test not actually selected") + continue + else: + selected.insert(0, priority_test) - if self.skipped: - print(file=sys.stderr) - print(count(len(self.skipped), "test"), "skipped:", file=sys.stderr) - printlist(self.skipped, file=sys.stderr) + return (tuple(selected), tests) - def rerun_failed_tests(self): - self.ns.verbose = True - self.ns.failfast = False - self.ns.verbose3 = False + @staticmethod + def list_tests(tests: TestTuple) -> None: + for name in tests: + print(name) - self.first_result = self.get_tests_result() + def _rerun_failed_tests(self, runtests: RunTests) -> RunTests: + # Configure the runner to re-run tests + if self.num_workers == 0 and not self.single_process: + # Always run tests in fresh processes to have more deterministic + # initial state. Don't re-run tests in parallel but limit to a + # single worker process to have side effects (on the system load + # and timings) between tests. + self.num_workers = 1 + + tests, match_tests_dict = self.results.prepare_rerun() + + # Re-run failed tests + runtests = runtests.copy( + tests=tests, + rerun=True, + verbose=True, + forever=False, + fail_fast=False, + match_tests_dict=match_tests_dict, + output_on_failure=False) + self.logger.set_tests(runtests) + + msg = f"Re-running {len(tests)} failed tests in verbose mode" + if not self.single_process: + msg = f"{msg} in subprocesses" + self.log(msg) + self._run_tests_mp(runtests, self.num_workers) + else: + self.log(msg) + self.run_tests_sequentially(runtests) + return runtests + + def rerun_failed_tests(self, runtests: RunTests) -> None: + ansi = get_colors() + red, reset = ansi.BOLD_RED, ansi.RESET + + if self.python_cmd: + # Temp patch for https://github.com/python/cpython/issues/94052 + self.log( + "Re-running failed tests is not supported with --python " + "host runner option." + ) + return + + self.first_state = self.get_state() print() - print("Re-running failed tests in verbose mode") - self.rerun = self.bad[:] - for test_name in self.rerun: - print(f"Re-running {test_name} in verbose mode", flush=True) - self.ns.verbose = True - result = runtest(self.ns, test_name) + rerun_runtests = self._rerun_failed_tests(runtests) - self.accumulate_result(result, rerun=True) + if self.results.bad: + print( + f"{red}{count(len(self.results.bad), 'test')} " + f"failed again:{reset}" + ) + printlist(self.results.bad) - if result.result == INTERRUPTED: - break + self.display_result(rerun_runtests) - if self.bad: - print(count(len(self.bad), 'test'), "failed again:") - printlist(self.bad) + def _run_bisect(self, runtests: RunTests, test: str, progress: str) -> bool: + print() + title = f"Bisect {test}" + if progress: + title = f"{title} ({progress})" + print(title) + print("#" * len(title)) + print() - self.display_result() + cmd = runtests.create_python_cmd() + cmd.extend([ + "-u", "-m", "test.bisect_cmd", + # Limit to 25 iterations (instead of 100) to not abuse CI resources + "--max-iter", "25", + "-v", + # runtests.match_tests is not used (yet) for bisect_cmd -i arg + ]) + cmd.extend(runtests.bisect_cmd_args()) + cmd.append(test) + print("+", shlex.join(cmd), flush=True) + + flush_std_streams() + + import subprocess + proc = subprocess.run(cmd, timeout=runtests.timeout) + exitcode = proc.returncode + + title = f"{title}: exit code {exitcode}" + print(title) + print("#" * len(title)) + print(flush=True) + + if exitcode: + print(f"Bisect failed with exit code {exitcode}") + return False + + return True + + def run_bisect(self, runtests: RunTests) -> None: + tests, _ = self.results.prepare_rerun(clear=False) + + for index, name in enumerate(tests, 1): + if len(tests) > 1: + progress = f"{index}/{len(tests)}" + else: + progress = "" + if not self._run_bisect(runtests, name, progress): + return - def display_result(self): + def display_result(self, runtests: RunTests) -> None: # If running the test suite for PGO then no one cares about results. - if self.ns.pgo: + if runtests.pgo: return + state = self.get_state() print() - print("== Tests result: %s ==" % self.get_tests_result()) - - if self.interrupted: - print("Test suite interrupted by signal SIGINT.") - - omitted = set(self.selected) - self.get_executed() - if omitted: - print() - print(count(len(omitted), "test"), "omitted:") - printlist(omitted) - - if self.good and not self.ns.quiet: - print() - if (not self.bad - and not self.skipped - and not self.interrupted - and len(self.good) > 1): - print("All", end=' ') - print(count(len(self.good), "test"), "OK.") - - if self.ns.print_slow: - self.test_times.sort(reverse=True) - print() - print("10 slowest tests:") - for test_time, test in self.test_times[:10]: - print("- %s: %s" % (test, format_duration(test_time))) - - if self.bad: - print() - print(count(len(self.bad), "test"), "failed:") - printlist(self.bad) - - if self.environment_changed: - print() - print("{} altered the execution environment:".format( - count(len(self.environment_changed), "test"))) - printlist(self.environment_changed) - - if self.skipped and not self.ns.quiet: - print() - print(count(len(self.skipped), "test"), "skipped:") - printlist(self.skipped) - - if self.rerun: - print() - print("%s:" % count(len(self.rerun), "re-run test")) - printlist(self.rerun) - - if self.run_no_tests: - print() - print(count(len(self.run_no_tests), "test"), "run no tests:") - printlist(self.run_no_tests) - - def run_tests_sequential(self): - if self.ns.trace: - import trace - self.tracer = trace.Trace(trace=False, count=True) + print(f"== Tests result: {state} ==") + + self.results.display_result(runtests.tests, + self.quiet, self.print_slowest) + + def run_test( + self, test_name: TestName, runtests: RunTests, tracer: trace.Trace | None + ) -> TestResult: + if tracer is not None: + # If we're tracing code coverage, then we don't exit with status + # if on a false return value from main. + cmd = ('result = run_single_test(test_name, runtests)') + namespace = dict(locals()) + tracer.runctx(cmd, globals=globals(), locals=namespace) + result = namespace['result'] + result.covered_lines = list(tracer.counts) + else: + result = run_single_test(test_name, runtests) + + self.results.accumulate_result(result, runtests) + + return result + + def run_tests_sequentially(self, runtests: RunTests) -> None: + if self.coverage: + tracer = trace.Trace(trace=False, count=True) + else: + tracer = None save_modules = set(sys.modules) - print("Run tests sequentially") - - previous_test = None - for test_index, test_name in enumerate(self.tests, 1): - start_time = time.monotonic() - - text = test_name - if previous_test: - text = '%s -- %s' % (text, previous_test) - self.display_progress(test_index, text) - - if self.tracer: - # If we're tracing code coverage, then we don't exit with status - # if on a false return value from main. - cmd = ('result = runtest(self.ns, test_name); ' - 'self.accumulate_result(result)') - ns = dict(locals()) - self.tracer.runctx(cmd, globals=globals(), locals=ns) - result = ns['result'] - else: - result = runtest(self.ns, test_name) - self.accumulate_result(result) + jobs = runtests.get_jobs() + if jobs is not None: + tests = count(jobs, 'test') + else: + tests = 'tests' + msg = f"Run {tests} sequentially in a single process" + if runtests.timeout: + msg += " (timeout: %s)" % format_duration(runtests.timeout) + self.log(msg) - if result.result == INTERRUPTED: - break + tests_iter = runtests.iter_tests() + for test_index, test_name in enumerate(tests_iter, 1): + start_time = time.perf_counter() - previous_test = format_test_result(result) - test_time = time.monotonic() - start_time - if test_time >= PROGRESS_MIN_TIME: - previous_test = "%s in %s" % (previous_test, format_duration(test_time)) - elif result[0] == PASSED: - # be quiet: say nothing if the test passed shortly - previous_test = None + self.logger.display_progress(test_index, test_name) + + result = self.run_test(test_name, runtests, tracer) # Unload the newly imported test modules (best effort finalization) new_modules = [module for module in sys.modules @@ -422,95 +435,26 @@ def run_tests_sequential(self): except (KeyError, AttributeError): pass - if previous_test: - print(previous_test) - - def _test_forever(self, tests): - while True: - for test_name in tests: - yield test_name - if self.bad: - return - if self.ns.fail_env_changed and self.environment_changed: - return - - def display_header(self): - # Print basic platform information - print("==", platform.python_implementation(), *sys.version.split()) - try: - print("==", platform.platform(aliased=True), - "%s-endian" % sys.byteorder) - except: - print("== RustPython: Need to fix platform.platform") - print("== cwd:", os.getcwd()) - cpu_count = os.cpu_count() - if cpu_count: - print("== CPU count:", cpu_count) - try: - print("== encodings: locale=%s, FS=%s" - % (locale.getpreferredencoding(False), - sys.getfilesystemencoding())) - except: - print("== RustPython: Need to fix encoding stuff") - - def get_tests_result(self): - result = [] - if self.bad: - result.append("FAILURE") - elif self.ns.fail_env_changed and self.environment_changed: - result.append("ENV CHANGED") - elif not any((self.good, self.bad, self.skipped, self.interrupted, - self.environment_changed)): - result.append("NO TEST RUN") - - if self.interrupted: - result.append("INTERRUPTED") - - if not result: - result.append("SUCCESS") - - result = ', '.join(result) - if self.first_result: - result = '%s then %s' % (self.first_result, result) - return result + text = str(result) + test_time = time.perf_counter() - start_time + if test_time >= PROGRESS_MIN_TIME: + text = f"{text} in {format_duration(test_time)}" + self.logger.display_progress(test_index, text) - def run_tests(self): - # For a partial run, we do not need to clutter the output. - if (self.ns.header - or not(self.ns.pgo or self.ns.quiet or self.ns.single - or self.tests or self.ns.args)): - self.display_header() - - if self.ns.huntrleaks: - warmup, repetitions, _ = self.ns.huntrleaks - if warmup < 3: - msg = ("WARNING: Running tests with --huntrleaks/-R and less than " - "3 warmup repetitions can give false positives!") - print(msg, file=sys.stdout, flush=True) - - if self.ns.randomize: - print("Using random seed", self.ns.random_seed) - - if self.ns.forever: - self.tests = self._test_forever(list(self.selected)) - self.test_count = '' - self.test_count_width = 3 - else: - self.tests = iter(self.selected) - self.test_count = '/{}'.format(len(self.selected)) - self.test_count_width = len(self.test_count) - 1 + if result.must_stop(self.fail_fast, self.fail_env_changed): + break - if self.ns.use_mp: - from test.libregrtest.runtest_mp import run_tests_multiprocess - run_tests_multiprocess(self) - else: - self.run_tests_sequential() + def get_state(self) -> str: + state = self.results.get_state(self.fail_env_changed) + if self.first_state: + state = f'{self.first_state} then {state}' + return state - def finalize(self): - if self.win_load_tracker is not None: - self.win_load_tracker.close() - self.win_load_tracker = None + def _run_tests_mp(self, runtests: RunTests, num_workers: int) -> None: + from .run_workers import RunWorkers + RunWorkers(num_workers, runtests, self.logger, self.results).run() + def finalize_tests(self, coverage: trace.CoverageResults | None) -> None: if self.next_single_filename: if self.next_single_test: with open(self.next_single_filename, 'w') as fp: @@ -518,141 +462,326 @@ def finalize(self): else: os.unlink(self.next_single_filename) - if self.tracer: - r = self.tracer.results() - r.write_results(show_missing=True, summary=True, - coverdir=self.ns.coverdir) + if coverage is not None: + # uses a new-in-Python 3.13 keyword argument that mypy doesn't know about yet: + coverage.write_results(show_missing=True, summary=True, # type: ignore[call-arg] + coverdir=self.coverage_dir, + ignore_missing_files=True) + + if self.want_run_leaks: + os.system("leaks %d" % os.getpid()) + + if self.junit_filename: + self.results.write_junit(self.junit_filename) + def display_summary(self) -> None: + if self.first_runtests is None: + raise ValueError( + "Should never call `display_summary()` before calling `_run_test()`" + ) + + duration = time.perf_counter() - self.logger.start_time + filtered = bool(self.match_tests) + + # Total duration print() - duration = time.monotonic() - self.start_time print("Total duration: %s" % format_duration(duration)) - print("Tests result: %s" % self.get_tests_result()) - if self.ns.runleaks: - os.system("leaks %d" % os.getpid()) + self.results.display_summary(self.first_runtests, filtered) + + # Result + state = self.get_state() + print(f"Result: {state}") + + def create_run_tests(self, tests: TestTuple) -> RunTests: + return RunTests( + tests, + fail_fast=self.fail_fast, + fail_env_changed=self.fail_env_changed, + match_tests=self.match_tests, + match_tests_dict=None, + rerun=False, + forever=self.forever, + pgo=self.pgo, + pgo_extended=self.pgo_extended, + output_on_failure=self.output_on_failure, + timeout=self.timeout, + verbose=self.verbose, + quiet=self.quiet, + hunt_refleak=self.hunt_refleak, + test_dir=self.test_dir, + use_junit=(self.junit_filename is not None), + coverage=self.coverage, + memory_limit=self.memory_limit, + gc_threshold=self.gc_threshold, + use_resources=self.use_resources, + python_cmd=self.python_cmd, + randomize=self.randomize, + random_seed=self.random_seed, + parallel_threads=self.parallel_threads, + ) + + def _run_tests(self, selected: TestTuple, tests: TestList | None) -> int: + if self.hunt_refleak and self.hunt_refleak.warmups < 3: + msg = ("WARNING: Running tests with --huntrleaks/-R and " + "less than 3 warmup repetitions can give false positives!") + print(msg, file=sys.stdout, flush=True) + + if self.num_workers < 0: + # Use all CPUs + 2 extra worker processes for tests + # that like to sleep + # + # os.process.cpu_count() is new in Python 3.13; + # mypy doesn't know about it yet + self.num_workers = (os.process_cpu_count() or 1) + 2 # type: ignore[attr-defined] - def save_xml_result(self): - if not self.ns.xmlpath and not self.testsuite_xml: - return + # For a partial run, we do not need to clutter the output. + if (self.want_header + or not(self.pgo or self.quiet or self.single_test_run + or tests or self.cmdline_args)): + display_header(self.use_resources, self.python_cmd) - import xml.etree.ElementTree as ET - root = ET.Element("testsuites") + print("Using random seed:", self.random_seed) - # Manually count the totals for the overall summary - totals = {'tests': 0, 'errors': 0, 'failures': 0} - for suite in self.testsuite_xml: - root.append(suite) - for k in totals: - try: - totals[k] += int(suite.get(k, 0)) - except ValueError: - pass + runtests = self.create_run_tests(selected) + self.first_runtests = runtests + self.logger.set_tests(runtests) - for k, v in totals.items(): - root.set(k, str(v)) - - xmlpath = os.path.join(os_helper.SAVEDCWD, self.ns.xmlpath) - with open(xmlpath, 'wb') as f: - for s in ET.tostringlist(root): - f.write(s) - - def main(self, tests=None, **kwargs): - global TEMPDIR - self.ns = self.parse_args(kwargs) - - if self.ns.tempdir: - TEMPDIR = self.ns.tempdir - elif self.ns.worker_args: - ns_dict, _ = json.loads(self.ns.worker_args) - TEMPDIR = ns_dict.get("tempdir") or TEMPDIR - - os.makedirs(TEMPDIR, exist_ok=True) - - # Define a writable temp dir that will be used as cwd while running - # the tests. The name of the dir includes the pid to allow parallel - # testing (see the -j option). - test_cwd = 'test_python_{}'.format(os.getpid()) - test_cwd = os.path.join(TEMPDIR, test_cwd) - - # Run the tests in a context manager that temporarily changes the CWD to a - # temporary and writable directory. If it's not possible to create or - # change the CWD, the original CWD will be used. The original CWD is - # available from os_helper.SAVEDCWD. - with os_helper.temp_cwd(test_cwd, quiet=True): - self._main(tests, kwargs) - - def getloadavg(self): - if self.win_load_tracker is not None: - return self.win_load_tracker.getloadavg() - - if hasattr(os, 'getloadavg'): - return os.getloadavg()[0] - - return None - - def _main(self, tests, kwargs): - if self.ns.huntrleaks: - warmup, repetitions, _ = self.ns.huntrleaks - if warmup < 1 or repetitions < 1: - msg = ("Invalid values for the --huntrleaks/-R parameters. The " - "number of warmups and repetitions must be at least 1 " - "each (1:1).") - print(msg, file=sys.stderr, flush=True) - sys.exit(2) - - if self.ns.worker_args is not None: - from test.libregrtest.runtest_mp import run_tests_worker - run_tests_worker(self.ns.worker_args) - - if self.ns.wait: - input("Press any key to continue...") + if (runtests.hunt_refleak is not None) and (not self.num_workers): + # gh-109739: WindowsLoadTracker thread interferes with refleak check + use_load_tracker = False + else: + # WindowsLoadTracker is only needed on Windows + use_load_tracker = MS_WINDOWS + + if use_load_tracker: + self.logger.start_load_tracker() + try: + if self.num_workers: + self._run_tests_mp(runtests, self.num_workers) + else: + self.run_tests_sequentially(runtests) + + coverage = self.results.get_coverage_results() + self.display_result(runtests) + + if self.want_rerun and self.results.need_rerun(): + self.rerun_failed_tests(runtests) + + if self.want_bisect and self.results.need_rerun(): + self.run_bisect(runtests) + finally: + if use_load_tracker: + self.logger.stop_load_tracker() + + self.display_summary() + self.finalize_tests(coverage) + + return self.results.get_exitcode(self.fail_env_changed, + self.fail_rerun) + + def run_tests(self, selected: TestTuple, tests: TestList | None) -> int: + os.makedirs(self.tmp_dir, exist_ok=True) + work_dir = get_work_dir(self.tmp_dir) + + # Put a timeout on Python exit + with exit_timeout(): + # Run the tests in a context manager that temporarily changes the + # CWD to a temporary and writable directory. If it's not possible + # to create or change the CWD, the original CWD will be used. + # The original CWD is available from os_helper.SAVEDCWD. + with os_helper.temp_cwd(work_dir, quiet=True): + # When using multiprocessing, worker processes will use + # work_dir as their parent temporary directory. So when the + # main process exit, it removes also subdirectories of worker + # processes. + return self._run_tests(selected, tests) + + def _add_cross_compile_opts(self, regrtest_opts): + # WASM/WASI buildbot builders pass multiple PYTHON environment + # variables such as PYTHONPATH and _PYTHON_HOSTRUNNER. + keep_environ = bool(self.python_cmd) + environ = None + + # Are we using cross-compilation? + cross_compile = is_cross_compiled() + + # Get HOSTRUNNER + hostrunner = get_host_runner() + + if cross_compile: + # emulate -E, but keep PYTHONPATH + cross compile env vars, + # so test executable can load correct sysconfigdata file. + keep = { + '_PYTHON_PROJECT_BASE', + '_PYTHON_HOST_PLATFORM', + '_PYTHON_SYSCONFIGDATA_NAME', + "_PYTHON_SYSCONFIGDATA_PATH", + 'PYTHONPATH' + } + old_environ = os.environ + new_environ = { + name: value for name, value in os.environ.items() + if not name.startswith(('PYTHON', '_PYTHON')) or name in keep + } + # Only set environ if at least one variable was removed + if new_environ != old_environ: + environ = new_environ + keep_environ = True + + if cross_compile and hostrunner: + if self.num_workers == 0 and not self.single_process: + # For now use only two cores for cross-compiled builds; + # hostrunner can be expensive. + regrtest_opts.extend(['-j', '2']) + + # If HOSTRUNNER is set and -p/--python option is not given, then + # use hostrunner to execute python binary for tests. + if not self.python_cmd: + buildpython = sysconfig.get_config_var("BUILDPYTHON") + python_cmd = f"{hostrunner} {buildpython}" + regrtest_opts.extend(["--python", python_cmd]) + keep_environ = True + + return (environ, keep_environ) + + def _add_ci_python_opts(self, python_opts, keep_environ): + # --fast-ci and --slow-ci add options to Python. + # + # Some platforms run tests in embedded mode and cannot change options + # after startup, so if this function changes, consider also updating: + # * gradle_task in Android/android.py + + # Unbuffered stdout and stderr. This isn't helpful on Android, because + # it would cause lines to be split into multiple log messages. + if not sys.stdout.write_through and sys.platform != "android": + python_opts.append('-u') + + # Add warnings filter 'error', unless the user specified a different + # filter. Ignore BytesWarning since it's controlled by '-b' below. + if not [ + opt for opt in sys.warnoptions + if not opt.endswith("::BytesWarning") + ]: + python_opts.extend(('-W', 'error')) + + # Error on bytes/str comparison + if sys.flags.bytes_warning < 2: + python_opts.append('-bb') + + if not keep_environ: + # Ignore PYTHON* environment variables + if not sys.flags.ignore_environment: + python_opts.append('-E') + + def _execute_python(self, cmd, environ): + # Make sure that messages before execv() are logged + sys.stdout.flush() + sys.stderr.flush() + + cmd_text = shlex.join(cmd) + try: + # Android and iOS run tests in embedded mode. To update their + # Python options, see the comment in _add_ci_python_opts. + if not cmd[0]: + raise ValueError("No Python executable is present") + + print(f"+ {cmd_text}", flush=True) + if hasattr(os, 'execv') and not MS_WINDOWS: + os.execv(cmd[0], cmd) + # On success, execv() do no return. + # On error, it raises an OSError. + else: + import subprocess + with subprocess.Popen(cmd, env=environ) as proc: + try: + proc.wait() + except KeyboardInterrupt: + # There is no need to call proc.terminate(): on CTRL+C, + # SIGTERM is also sent to the child process. + try: + proc.wait(timeout=EXIT_TIMEOUT) + except subprocess.TimeoutExpired: + proc.kill() + proc.wait() + sys.exit(EXITCODE_INTERRUPTED) + + sys.exit(proc.returncode) + except Exception as exc: + print_warning(f"Failed to change Python options: {exc!r}\n" + f"Command: {cmd_text}") + # continue executing main() + + def _add_python_opts(self) -> None: + python_opts: list[str] = [] + regrtest_opts: list[str] = [] + + environ, keep_environ = self._add_cross_compile_opts(regrtest_opts) + if self.ci_mode: + self._add_ci_python_opts(python_opts, keep_environ) + + if (not python_opts) and (not regrtest_opts) and (environ is None): + # Nothing changed: nothing to do + return - support.PGO = self.ns.pgo + # Create new command line + cmd = list(sys.orig_argv) + if python_opts: + cmd[1:1] = python_opts + if regrtest_opts: + cmd.extend(regrtest_opts) + cmd.append("--dont-add-python-opts") - setup_tests(self.ns) + self._execute_python(cmd, environ) - self.find_tests(tests) + def _init(self): + setup_process() - if self.ns.list_tests: - self.list_tests() - sys.exit(0) + if self.junit_filename and not os.path.isabs(self.junit_filename): + self.junit_filename = os.path.abspath(self.junit_filename) - if self.ns.list_cases: - self.list_cases() - sys.exit(0) + strip_py_suffix(self.cmdline_args) - # If we're on windows and this is the parent runner (not a worker), - # track the load average. - # TODO: RUSTPYTHON - # if sys.platform == 'win32' and (self.ns.worker_args is None): - # from test.libregrtest.win_utils import WindowsLoadTracker + self._tmp_dir = get_temp_dir(self._tmp_dir) - # try: - # self.win_load_tracker = WindowsLoadTracker() - # except FileNotFoundError as error: - # # Windows IoT Core and Windows Nano Server do not provide - # # typeperf.exe for x64, x86 or ARM - # print(f'Failed to create WindowsLoadTracker: {error}') + @property + def tmp_dir(self) -> StrPath: + if self._tmp_dir is None: + raise ValueError( + "Should never use `.tmp_dir` before calling `.main()`" + ) + return self._tmp_dir - self.run_tests() - self.display_result() + def main(self, tests: TestList | None = None) -> NoReturn: + if self.want_add_python_opts: + self._add_python_opts() - if self.ns.verbose2 and self.bad: - self.rerun_failed_tests() + self._init() - self.finalize() + if self.want_cleanup: + cleanup_temp_dir(self.tmp_dir) + sys.exit(0) + + if self.want_wait: + input("Press any key to continue...") - self.save_xml_result() + setup_test_dir(self.test_dir) + selected, tests = self.find_tests(tests) + + exitcode = 0 + if self.want_list_tests: + self.list_tests(selected) + elif self.want_list_cases: + list_cases(selected, + match_tests=self.match_tests, + test_dir=self.test_dir) + else: + exitcode = self.run_tests(selected, tests) - if self.bad: - sys.exit(2) - if self.interrupted: - sys.exit(130) - if self.ns.fail_env_changed and self.environment_changed: - sys.exit(3) - sys.exit(0) + sys.exit(exitcode) -def main(tests=None, **kwargs): +def main(tests=None, _add_python_opts=False, **kwargs) -> NoReturn: """Run the Python suite.""" - Regrtest().main(tests=tests, **kwargs) + ns = _parse_args(sys.argv[1:], **kwargs) + Regrtest(ns, _add_python_opts=_add_python_opts).main(tests=tests) diff --git a/Lib/test/libregrtest/mypy.ini b/Lib/test/libregrtest/mypy.ini new file mode 100644 index 00000000000..3fa9afcb7a4 --- /dev/null +++ b/Lib/test/libregrtest/mypy.ini @@ -0,0 +1,26 @@ +# Config file for running mypy on libregrtest. +# Run mypy by invoking `mypy --config-file Lib/test/libregrtest/mypy.ini` +# on the command-line from the repo root + +[mypy] +files = Lib/test/libregrtest +explicit_package_bases = True +python_version = 3.12 +platform = linux +pretty = True + +# Enable most stricter settings +enable_error_code = ignore-without-code +strict = True + +# Various stricter settings that we can't yet enable +# Try to enable these in the following order: +disallow_incomplete_defs = False +disallow_untyped_calls = False +disallow_untyped_defs = False +check_untyped_defs = False +warn_return_any = False + +# Various internal modules that typeshed deliberately doesn't have stubs for: +[mypy-_abc.*,_opcode.*,_overlapped.*,_testcapi.*,_testinternalcapi.*,test.*] +ignore_missing_imports = True diff --git a/Lib/test/libregrtest/parallel_case.py b/Lib/test/libregrtest/parallel_case.py new file mode 100644 index 00000000000..8eb3c314916 --- /dev/null +++ b/Lib/test/libregrtest/parallel_case.py @@ -0,0 +1,78 @@ +"""Run a test case multiple times in parallel threads.""" + +import copy +import threading +import unittest + +from unittest import TestCase + + +class ParallelTestCase(TestCase): + def __init__(self, test_case: TestCase, num_threads: int): + self.test_case = test_case + self.num_threads = num_threads + self._testMethodName = test_case._testMethodName + self._testMethodDoc = test_case._testMethodDoc + + def __str__(self): + return f"{str(self.test_case)} [threads={self.num_threads}]" + + def run_worker(self, test_case: TestCase, result: unittest.TestResult, + barrier: threading.Barrier): + barrier.wait() + test_case.run(result) + + def run(self, result=None): + if result is None: + result = test_case.defaultTestResult() + startTestRun = getattr(result, 'startTestRun', None) + stopTestRun = getattr(result, 'stopTestRun', None) + if startTestRun is not None: + startTestRun() + else: + stopTestRun = None + + # Called at the beginning of each test. See TestCase.run. + result.startTest(self) + + cases = [copy.copy(self.test_case) for _ in range(self.num_threads)] + results = [unittest.TestResult() for _ in range(self.num_threads)] + + barrier = threading.Barrier(self.num_threads) + threads = [] + for i, (case, r) in enumerate(zip(cases, results)): + thread = threading.Thread(target=self.run_worker, + args=(case, r, barrier), + name=f"{str(self.test_case)}-{i}", + daemon=True) + threads.append(thread) + + for thread in threads: + thread.start() + + for threads in threads: + threads.join() + + # Aggregate test results + if all(r.wasSuccessful() for r in results): + result.addSuccess(self) + + # Note: We can't call result.addError, result.addFailure, etc. because + # we no longer have the original exception, just the string format. + for r in results: + if len(r.errors) > 0 or len(r.failures) > 0: + result._mirrorOutput = True + result.errors.extend(r.errors) + result.failures.extend(r.failures) + result.skipped.extend(r.skipped) + result.expectedFailures.extend(r.expectedFailures) + result.unexpectedSuccesses.extend(r.unexpectedSuccesses) + result.collectedDurations.extend(r.collectedDurations) + + if any(r.shouldStop for r in results): + result.stop() + + # Test has finished running + result.stopTest(self) + if stopTestRun is not None: + stopTestRun() diff --git a/Lib/test/libregrtest/pgo.py b/Lib/test/libregrtest/pgo.py new file mode 100644 index 00000000000..04803ddf644 --- /dev/null +++ b/Lib/test/libregrtest/pgo.py @@ -0,0 +1,55 @@ +# Set of tests run by default if --pgo is specified. The tests below were +# chosen based on the following criteria: either they exercise a commonly used +# C extension module or type, or they run some relatively typical Python code. +# Long running tests should be avoided because the PGO instrumented executable +# runs slowly. +PGO_TESTS = [ + 'test_array', + 'test_base64', + 'test_binascii', + 'test_binop', + 'test_bisect', + 'test_bytes', + 'test_bz2', + 'test_cmath', + 'test_codecs', + 'test_collections', + 'test_complex', + 'test_dataclasses', + 'test_datetime', + 'test_decimal', + 'test_difflib', + 'test_float', + 'test_fstring', + 'test_functools', + 'test_generators', + 'test_hashlib', + 'test_heapq', + 'test_int', + 'test_itertools', + 'test_json', + 'test_long', + 'test_lzma', + 'test_math', + 'test_memoryview', + 'test_operator', + 'test_ordered_dict', + 'test_patma', + 'test_pickle', + 'test_pprint', + 'test_re', + 'test_set', + 'test_sqlite3', + 'test_statistics', + 'test_str', + 'test_struct', + 'test_tabnanny', + 'test_time', + 'test_xml_etree', + 'test_xml_etree_c', +] + +def setup_pgo_tests(cmdline_args, pgo_extended: bool) -> None: + if not cmdline_args and not pgo_extended: + # run default set of tests for PGO training + cmdline_args[:] = PGO_TESTS[:] diff --git a/Lib/test/libregrtest/refleak.py b/Lib/test/libregrtest/refleak.py index 03747f7f757..5c78515506d 100644 --- a/Lib/test/libregrtest/refleak.py +++ b/Lib/test/libregrtest/refleak.py @@ -1,9 +1,17 @@ import os -import re import sys import warnings from inspect import isabstract +from typing import Any +import linecache + from test import support +from test.support import os_helper +from test.support import refleak_helper + +from .runtests import HuntRefleak +from .utils import clear_caches + try: from _abc import _get_dump except ImportError: @@ -17,7 +25,33 @@ def _get_dump(cls): cls._abc_negative_cache, cls._abc_negative_cache_version) -def dash_R(ns, test_name, test_func): +def save_support_xml(filename): + if support.junit_xml_list is None: + return + + import pickle + with open(filename, 'xb') as fp: + pickle.dump(support.junit_xml_list, fp) + support.junit_xml_list = None + + +def restore_support_xml(filename): + try: + fp = open(filename, 'rb') + except FileNotFoundError: + return + + import pickle + with fp: + xml_list = pickle.load(fp) + os.unlink(filename) + + support.junit_xml_list = xml_list + + +def runtest_refleak(test_name, test_func, + hunt_refleak: HuntRefleak, + quiet: bool): """Run a test multiple times, looking for reference leaks. Returns: @@ -39,12 +73,19 @@ def dash_R(ns, test_name, test_func): fs = warnings.filters[:] ps = copyreg.dispatch_table.copy() pic = sys.path_importer_cache.copy() + zdc: dict[str, Any] | None + # Linecache holds a cache with the source of interactive code snippets + # (e.g. code typed in the REPL). This cache is not cleared by + # linecache.clearcache(). We need to save and restore it to avoid false + # positives. + linecache_data = linecache.cache.copy(), linecache._interactive_cache.copy() # type: ignore[attr-defined] try: import zipimport except ImportError: zdc = None # Run unmodified on platforms without zipimport support else: - zdc = zipimport._zip_directory_cache.copy() + # private attribute that mypy doesn't know about: + zdc = zipimport._zip_directory_cache.copy() # type: ignore[attr-defined] abcs = {} for abc in [getattr(collections.abc, a) for a in collections.abc.__all__]: if not isabstract(abc): @@ -60,9 +101,10 @@ def dash_R(ns, test_name, test_func): def get_pooled_int(value): return int_pool.setdefault(value, value) - nwarmup, ntracked, fname = ns.huntrleaks - fname = os.path.join(os_helper.SAVEDCWD, fname) - repcount = nwarmup + ntracked + warmups = hunt_refleak.warmups + runs = hunt_refleak.runs + filename = hunt_refleak.filename + repcount = warmups + runs # Pre-allocate to ensure that the loop doesn't allocate anything new rep_range = list(range(repcount)) @@ -71,45 +113,81 @@ def get_pooled_int(value): fd_deltas = [0] * repcount getallocatedblocks = sys.getallocatedblocks gettotalrefcount = sys.gettotalrefcount - fd_count = support.fd_count - + getunicodeinternedsize = sys.getunicodeinternedsize + fd_count = os_helper.fd_count # initialize variables to make pyflakes quiet - rc_before = alloc_before = fd_before = 0 + rc_before = alloc_before = fd_before = interned_immortal_before = 0 - if not ns.quiet: - print("beginning", repcount, "repetitions", file=sys.stderr) - print(("1234567890"*(repcount//10 + 1))[:repcount], file=sys.stderr, - flush=True) + if not quiet: + print("beginning", repcount, "repetitions. Showing number of leaks " + "(. for 0 or less, X for 10 or more)", + file=sys.stderr) + numbers = ("1234567890"*(repcount//10 + 1))[:repcount] + numbers = numbers[:warmups] + ':' + numbers[warmups:] + print(numbers, file=sys.stderr, flush=True) - dash_R_cleanup(fs, ps, pic, zdc, abcs) + xml_filename = 'refleak-xml.tmp' + result = None + dash_R_cleanup(fs, ps, pic, zdc, abcs, linecache_data) for i in rep_range: - test_func() - dash_R_cleanup(fs, ps, pic, zdc, abcs) - - # dash_R_cleanup() ends with collecting cyclic trash: - # read memory statistics immediately after. - alloc_after = getallocatedblocks() + support.gc_collect() + current = refleak_helper._hunting_for_refleaks + refleak_helper._hunting_for_refleaks = True + try: + result = test_func() + finally: + refleak_helper._hunting_for_refleaks = current + + save_support_xml(xml_filename) + dash_R_cleanup(fs, ps, pic, zdc, abcs, linecache_data) + support.gc_collect() + + # Read memory statistics immediately after the garbage collection. + # Also, readjust the reference counts and alloc blocks by ignoring + # any strings that might have been interned during test_func. These + # strings will be deallocated at runtime shutdown + interned_immortal_after = getunicodeinternedsize( + # Use an internal-only keyword argument that mypy doesn't know yet + _only_immortal=True) # type: ignore[call-arg] + alloc_after = getallocatedblocks() - interned_immortal_after rc_after = gettotalrefcount() fd_after = fd_count() - if not ns.quiet: - print('.', end='', file=sys.stderr, flush=True) - rc_deltas[i] = get_pooled_int(rc_after - rc_before) alloc_deltas[i] = get_pooled_int(alloc_after - alloc_before) fd_deltas[i] = get_pooled_int(fd_after - fd_before) + if not quiet: + # use max, not sum, so total_leaks is one of the pooled ints + total_leaks = max(rc_deltas[i], alloc_deltas[i], fd_deltas[i]) + if total_leaks <= 0: + symbol = '.' + elif total_leaks < 10: + symbol = ( + '.', '1', '2', '3', '4', '5', '6', '7', '8', '9', + )[total_leaks] + else: + symbol = 'X' + if i == warmups: + print(' ', end='', file=sys.stderr, flush=True) + print(symbol, end='', file=sys.stderr, flush=True) + del total_leaks + del symbol + alloc_before = alloc_after rc_before = rc_after fd_before = fd_after + interned_immortal_before = interned_immortal_after + + restore_support_xml(xml_filename) - if not ns.quiet: + if not quiet: print(file=sys.stderr) # These checkers return False on success, True on failure def check_rc_deltas(deltas): - # Checker for reference counters and memomry blocks. + # Checker for reference counters and memory blocks. # # bpo-30776: Try to ignore false positives: # @@ -133,19 +211,25 @@ def check_fd_deltas(deltas): (fd_deltas, 'file descriptors', check_fd_deltas) ]: # ignore warmup runs - deltas = deltas[nwarmup:] - if checker(deltas): + deltas = deltas[warmups:] + failing = checker(deltas) + suspicious = any(deltas) + if failing or suspicious: msg = '%s leaked %s %s, sum=%s' % ( test_name, deltas, item_name, sum(deltas)) - print(msg, file=sys.stderr, flush=True) - with open(fname, "a") as refrep: - print(msg, file=refrep) - refrep.flush() - failed = True - return failed - - -def dash_R_cleanup(fs, ps, pic, zdc, abcs): + print(msg, end='', file=sys.stderr) + if failing: + print(file=sys.stderr, flush=True) + with open(filename, "a", encoding="utf-8") as refrep: + print(msg, file=refrep) + refrep.flush() + failed = True + else: + print(' (this is fine)', file=sys.stderr, flush=True) + return (failed, result) + + +def dash_R_cleanup(fs, ps, pic, zdc, abcs, linecache_data): import copyreg import collections.abc @@ -155,6 +239,11 @@ def dash_R_cleanup(fs, ps, pic, zdc, abcs): copyreg.dispatch_table.update(ps) sys.path_importer_cache.clear() sys.path_importer_cache.update(pic) + lcache, linteractive = linecache_data + linecache._interactive_cache.clear() + linecache._interactive_cache.update(linteractive) + linecache.cache.clear() + linecache.cache.update(lcache) try: import zipimport except ImportError: @@ -163,121 +252,28 @@ def dash_R_cleanup(fs, ps, pic, zdc, abcs): zipimport._zip_directory_cache.clear() zipimport._zip_directory_cache.update(zdc) - # clear type cache - sys._clear_type_cache() - # Clear ABC registries, restoring previously saved ABC registries. abs_classes = [getattr(collections.abc, a) for a in collections.abc.__all__] abs_classes = filter(isabstract, abs_classes) for abc in abs_classes: for obj in abc.__subclasses__() + [abc]: - for ref in abcs.get(obj, set()): - if ref() is not None: - obj.register(ref()) + refs = abcs.get(obj, None) + if refs is not None: + obj._abc_registry_clear() + for ref in refs: + subclass = ref() + if subclass is not None: + obj.register(subclass) obj._abc_caches_clear() + # Clear caches clear_caches() - -def clear_caches(): - # Clear the warnings registry, so they can be displayed again - for mod in sys.modules.values(): - if hasattr(mod, '__warningregistry__'): - del mod.__warningregistry__ - - # Flush standard output, so that buffered data is sent to the OS and - # associated Python objects are reclaimed. - for stream in (sys.stdout, sys.stderr, sys.__stdout__, sys.__stderr__): - if stream is not None: - stream.flush() - - # Clear assorted module caches. - # Don't worry about resetting the cache if the module is not loaded - try: - distutils_dir_util = sys.modules['distutils.dir_util'] - except KeyError: - pass - else: - distutils_dir_util._path_created.clear() - re.purge() - - try: - _strptime = sys.modules['_strptime'] - except KeyError: - pass - else: - _strptime._regex_cache.clear() - - try: - urllib_parse = sys.modules['urllib.parse'] - except KeyError: - pass - else: - urllib_parse.clear_cache() - - try: - urllib_request = sys.modules['urllib.request'] - except KeyError: - pass - else: - urllib_request.urlcleanup() - - try: - linecache = sys.modules['linecache'] - except KeyError: - pass - else: - linecache.clearcache() - - try: - mimetypes = sys.modules['mimetypes'] - except KeyError: - pass - else: - mimetypes._default_mime_types() - - try: - filecmp = sys.modules['filecmp'] - except KeyError: - pass - else: - filecmp._cache.clear() - - try: - struct = sys.modules['struct'] - except KeyError: - pass - else: - # TODO: fix - # struct._clearcache() - pass - - try: - doctest = sys.modules['doctest'] - except KeyError: - pass - else: - doctest.master = None - - try: - ctypes = sys.modules['ctypes'] - except KeyError: - pass - else: - ctypes._reset_cache() - - try: - typing = sys.modules['typing'] - except KeyError: - pass - else: - for f in typing._cleanups: - f() - - support.gc_collect() + # Clear other caches last (previous function calls can re-populate them): + sys._clear_internal_caches() -def warm_caches(): +def warm_caches() -> None: # char cache s = bytes(range(256)) for i in range(256): diff --git a/Lib/test/libregrtest/result.py b/Lib/test/libregrtest/result.py new file mode 100644 index 00000000000..daf7624366e --- /dev/null +++ b/Lib/test/libregrtest/result.py @@ -0,0 +1,243 @@ +import dataclasses +import json +from _colorize import get_colors # type: ignore[import-not-found] +from typing import Any + +from .utils import ( + StrJSON, TestName, FilterTuple, + format_duration, normalize_test_name, print_warning) + + +@dataclasses.dataclass(slots=True) +class TestStats: + tests_run: int = 0 + failures: int = 0 + skipped: int = 0 + + @staticmethod + def from_unittest(result): + return TestStats(result.testsRun, + len(result.failures), + len(result.skipped)) + + @staticmethod + def from_doctest(results): + return TestStats(results.attempted, + results.failed, + results.skipped) + + def accumulate(self, stats): + self.tests_run += stats.tests_run + self.failures += stats.failures + self.skipped += stats.skipped + + +# Avoid enum.Enum to reduce the number of imports when tests are run +class State: + PASSED = "PASSED" + FAILED = "FAILED" + SKIPPED = "SKIPPED" + UNCAUGHT_EXC = "UNCAUGHT_EXC" + REFLEAK = "REFLEAK" + ENV_CHANGED = "ENV_CHANGED" + RESOURCE_DENIED = "RESOURCE_DENIED" + INTERRUPTED = "INTERRUPTED" + WORKER_FAILED = "WORKER_FAILED" # non-zero worker process exit code + WORKER_BUG = "WORKER_BUG" # exception when running a worker + DID_NOT_RUN = "DID_NOT_RUN" + TIMEOUT = "TIMEOUT" + + @staticmethod + def is_failed(state): + return state in { + State.FAILED, + State.UNCAUGHT_EXC, + State.REFLEAK, + State.WORKER_FAILED, + State.WORKER_BUG, + State.TIMEOUT} + + @staticmethod + def has_meaningful_duration(state): + # Consider that the duration is meaningless for these cases. + # For example, if a whole test file is skipped, its duration + # is unlikely to be the duration of executing its tests, + # but just the duration to execute code which skips the test. + return state not in { + State.SKIPPED, + State.RESOURCE_DENIED, + State.INTERRUPTED, + State.WORKER_FAILED, + State.WORKER_BUG, + State.DID_NOT_RUN} + + @staticmethod + def must_stop(state): + return state in { + State.INTERRUPTED, + State.WORKER_BUG, + } + + +FileName = str +LineNo = int +Location = tuple[FileName, LineNo] + + +@dataclasses.dataclass(slots=True) +class TestResult: + test_name: TestName + state: str | None = None + # Test duration in seconds + duration: float | None = None + xml_data: list[str] | None = None + stats: TestStats | None = None + + # errors and failures copied from support.TestFailedWithDetails + errors: list[tuple[str, str]] | None = None + failures: list[tuple[str, str]] | None = None + + # partial coverage in a worker run; not used by sequential in-process runs + covered_lines: list[Location] | None = None + + def is_failed(self, fail_env_changed: bool) -> bool: + if self.state == State.ENV_CHANGED: + return fail_env_changed + return State.is_failed(self.state) + + def _format_failed(self): + ansi = get_colors() + red, reset = ansi.BOLD_RED, ansi.RESET + if self.errors and self.failures: + le = len(self.errors) + lf = len(self.failures) + error_s = "error" + ("s" if le > 1 else "") + failure_s = "failure" + ("s" if lf > 1 else "") + return ( + f"{red}{self.test_name} failed " + f"({le} {error_s}, {lf} {failure_s}){reset}" + ) + + if self.errors: + le = len(self.errors) + error_s = "error" + ("s" if le > 1 else "") + return f"{red}{self.test_name} failed ({le} {error_s}){reset}" + + if self.failures: + lf = len(self.failures) + failure_s = "failure" + ("s" if lf > 1 else "") + return f"{red}{self.test_name} failed ({lf} {failure_s}){reset}" + + return f"{red}{self.test_name} failed{reset}" + + def __str__(self) -> str: + ansi = get_colors() + green = ansi.GREEN + red = ansi.BOLD_RED + reset = ansi.RESET + yellow = ansi.YELLOW + + match self.state: + case State.PASSED: + return f"{green}{self.test_name} passed{reset}" + case State.FAILED: + return f"{red}{self._format_failed()}{reset}" + case State.SKIPPED: + return f"{yellow}{self.test_name} skipped{reset}" + case State.UNCAUGHT_EXC: + return ( + f"{red}{self.test_name} failed (uncaught exception){reset}" + ) + case State.REFLEAK: + return f"{red}{self.test_name} failed (reference leak){reset}" + case State.ENV_CHANGED: + return f"{red}{self.test_name} failed (env changed){reset}" + case State.RESOURCE_DENIED: + return f"{yellow}{self.test_name} skipped (resource denied){reset}" + case State.INTERRUPTED: + return f"{yellow}{self.test_name} interrupted{reset}" + case State.WORKER_FAILED: + return ( + f"{red}{self.test_name} worker non-zero exit code{reset}" + ) + case State.WORKER_BUG: + return f"{red}{self.test_name} worker bug{reset}" + case State.DID_NOT_RUN: + return f"{yellow}{self.test_name} ran no tests{reset}" + case State.TIMEOUT: + assert self.duration is not None, "self.duration is None" + return f"{self.test_name} timed out ({format_duration(self.duration)})" + case _: + raise ValueError( + f"{red}unknown result state: {{state!r}}{reset}" + ) + + def has_meaningful_duration(self): + return State.has_meaningful_duration(self.state) + + def set_env_changed(self): + if self.state is None or self.state == State.PASSED: + self.state = State.ENV_CHANGED + + def must_stop(self, fail_fast: bool, fail_env_changed: bool) -> bool: + if State.must_stop(self.state): + return True + if fail_fast and self.is_failed(fail_env_changed): + return True + return False + + def get_rerun_match_tests(self) -> FilterTuple | None: + match_tests = [] + + errors = self.errors or [] + failures = self.failures or [] + for error_list, is_error in ( + (errors, True), + (failures, False), + ): + for full_name, *_ in error_list: + match_name = normalize_test_name(full_name, is_error=is_error) + if match_name is None: + # 'setUpModule (test.test_sys)': don't filter tests + return None + if not match_name: + error_type = "ERROR" if is_error else "FAIL" + print_warning(f"rerun failed to parse {error_type} test name: " + f"{full_name!r}: don't filter tests") + return None + match_tests.append(match_name) + + if not match_tests: + return None + return tuple(match_tests) + + def write_json_into(self, file) -> None: + json.dump(self, file, cls=_EncodeTestResult) + + @staticmethod + def from_json(worker_json: StrJSON) -> 'TestResult': + return json.loads(worker_json, object_hook=_decode_test_result) + + +class _EncodeTestResult(json.JSONEncoder): + def default(self, o: Any) -> dict[str, Any]: + if isinstance(o, TestResult): + result = dataclasses.asdict(o) + result["__test_result__"] = o.__class__.__name__ + return result + else: + return super().default(o) + + +def _decode_test_result(data: dict[str, Any]) -> TestResult | dict[str, Any]: + if "__test_result__" in data: + data.pop('__test_result__') + if data['stats'] is not None: + data['stats'] = TestStats(**data['stats']) + if data['covered_lines'] is not None: + data['covered_lines'] = [ + tuple(loc) for loc in data['covered_lines'] + ] + return TestResult(**data) + else: + return data diff --git a/Lib/test/libregrtest/results.py b/Lib/test/libregrtest/results.py new file mode 100644 index 00000000000..a35934fc2c9 --- /dev/null +++ b/Lib/test/libregrtest/results.py @@ -0,0 +1,309 @@ +import sys +import trace +from _colorize import get_colors # type: ignore[import-not-found] +from typing import TYPE_CHECKING + +from .runtests import RunTests +from .result import State, TestResult, TestStats, Location +from .utils import ( + StrPath, TestName, TestTuple, TestList, FilterDict, + printlist, count, format_duration) + +if TYPE_CHECKING: + from xml.etree.ElementTree import Element + + +# Python uses exit code 1 when an exception is not caught +# argparse.ArgumentParser.error() uses exit code 2 +EXITCODE_BAD_TEST = 2 +EXITCODE_ENV_CHANGED = 3 +EXITCODE_NO_TESTS_RAN = 4 +EXITCODE_RERUN_FAIL = 5 +EXITCODE_INTERRUPTED = 130 # 128 + signal.SIGINT=2 + + +class TestResults: + def __init__(self) -> None: + self.bad: TestList = [] + self.good: TestList = [] + self.rerun_bad: TestList = [] + self.skipped: TestList = [] + self.resource_denied: TestList = [] + self.env_changed: TestList = [] + self.run_no_tests: TestList = [] + self.rerun: TestList = [] + self.rerun_results: list[TestResult] = [] + + self.interrupted: bool = False + self.worker_bug: bool = False + self.test_times: list[tuple[float, TestName]] = [] + self.stats = TestStats() + # used by --junit-xml + self.testsuite_xml: list['Element'] = [] + # used by -T with -j + self.covered_lines: set[Location] = set() + + def is_all_good(self) -> bool: + return (not self.bad + and not self.skipped + and not self.interrupted + and not self.worker_bug) + + def get_executed(self) -> set[TestName]: + return (set(self.good) | set(self.bad) | set(self.skipped) + | set(self.resource_denied) | set(self.env_changed) + | set(self.run_no_tests)) + + def no_tests_run(self) -> bool: + return not any((self.good, self.bad, self.skipped, self.interrupted, + self.env_changed)) + + def get_state(self, fail_env_changed: bool) -> str: + state = [] + ansi = get_colors() + green = ansi.GREEN + red = ansi.BOLD_RED + reset = ansi.RESET + yellow = ansi.YELLOW + if self.bad: + state.append(f"{red}FAILURE{reset}") + elif fail_env_changed and self.env_changed: + state.append(f"{yellow}ENV CHANGED{reset}") + elif self.no_tests_run(): + state.append(f"{yellow}NO TESTS RAN{reset}") + + if self.interrupted: + state.append(f"{yellow}INTERRUPTED{reset}") + if self.worker_bug: + state.append(f"{red}WORKER BUG{reset}") + if not state: + state.append(f"{green}SUCCESS{reset}") + + return ', '.join(state) + + def get_exitcode(self, fail_env_changed: bool, fail_rerun: bool) -> int: + exitcode = 0 + if self.bad: + exitcode = EXITCODE_BAD_TEST + elif self.interrupted: + exitcode = EXITCODE_INTERRUPTED + elif fail_env_changed and self.env_changed: + exitcode = EXITCODE_ENV_CHANGED + elif self.no_tests_run(): + exitcode = EXITCODE_NO_TESTS_RAN + elif fail_rerun and self.rerun: + exitcode = EXITCODE_RERUN_FAIL + elif self.worker_bug: + exitcode = EXITCODE_BAD_TEST + return exitcode + + def accumulate_result(self, result: TestResult, runtests: RunTests) -> None: + test_name = result.test_name + rerun = runtests.rerun + fail_env_changed = runtests.fail_env_changed + + match result.state: + case State.PASSED: + self.good.append(test_name) + case State.ENV_CHANGED: + self.env_changed.append(test_name) + self.rerun_results.append(result) + case State.SKIPPED: + self.skipped.append(test_name) + case State.RESOURCE_DENIED: + self.resource_denied.append(test_name) + case State.INTERRUPTED: + self.interrupted = True + case State.DID_NOT_RUN: + self.run_no_tests.append(test_name) + case _: + if result.is_failed(fail_env_changed): + self.bad.append(test_name) + self.rerun_results.append(result) + else: + raise ValueError(f"invalid test state: {result.state!r}") + + if result.state == State.WORKER_BUG: + self.worker_bug = True + + if result.has_meaningful_duration() and not rerun: + if result.duration is None: + raise ValueError("result.duration is None") + self.test_times.append((result.duration, test_name)) + if result.stats is not None: + self.stats.accumulate(result.stats) + if rerun: + self.rerun.append(test_name) + if result.covered_lines: + # we don't care about trace counts so we don't have to sum them up + self.covered_lines.update(result.covered_lines) + xml_data = result.xml_data + if xml_data: + self.add_junit(xml_data) + + def get_coverage_results(self) -> trace.CoverageResults: + counts = {loc: 1 for loc in self.covered_lines} + return trace.CoverageResults(counts=counts) + + def need_rerun(self) -> bool: + return bool(self.rerun_results) + + def prepare_rerun(self, *, clear: bool = True) -> tuple[TestTuple, FilterDict]: + tests: TestList = [] + match_tests_dict = {} + for result in self.rerun_results: + tests.append(result.test_name) + + match_tests = result.get_rerun_match_tests() + # ignore empty match list + if match_tests: + match_tests_dict[result.test_name] = match_tests + + if clear: + # Clear previously failed tests + self.rerun_bad.extend(self.bad) + self.bad.clear() + self.env_changed.clear() + self.rerun_results.clear() + + return (tuple(tests), match_tests_dict) + + def add_junit(self, xml_data: list[str]) -> None: + import xml.etree.ElementTree as ET + for e in xml_data: + try: + self.testsuite_xml.append(ET.fromstring(e)) + except ET.ParseError: + print(xml_data, file=sys.__stderr__) + raise + + def write_junit(self, filename: StrPath) -> None: + if not self.testsuite_xml: + # Don't create empty XML file + return + + import xml.etree.ElementTree as ET + root = ET.Element("testsuites") + + # Manually count the totals for the overall summary + totals = {'tests': 0, 'errors': 0, 'failures': 0} + for suite in self.testsuite_xml: + root.append(suite) + for k in totals: + try: + totals[k] += int(suite.get(k, 0)) + except ValueError: + pass + + for k, v in totals.items(): + root.set(k, str(v)) + + with open(filename, 'wb') as f: + for s in ET.tostringlist(root): + f.write(s) + + def display_result(self, tests: TestTuple, quiet: bool, print_slowest: bool) -> None: + ansi = get_colors() + green = ansi.GREEN + red = ansi.BOLD_RED + reset = ansi.RESET + yellow = ansi.YELLOW + + if print_slowest: + self.test_times.sort(reverse=True) + print() + print(f"{yellow}10 slowest tests:{reset}") + for test_time, test in self.test_times[:10]: + print(f"- {test}: {format_duration(test_time)}") + + all_tests = [] + omitted = set(tests) - self.get_executed() + + # less important + all_tests.append( + (sorted(omitted), "test", f"{yellow}{{}} omitted:{reset}") + ) + if not quiet: + all_tests.append( + (self.skipped, "test", f"{yellow}{{}} skipped:{reset}") + ) + all_tests.append( + ( + self.resource_denied, + "test", + f"{yellow}{{}} skipped (resource denied):{reset}", + ) + ) + all_tests.append( + (self.run_no_tests, "test", f"{yellow}{{}} run no tests:{reset}") + ) + + # more important + all_tests.append( + ( + self.env_changed, + "test", + f"{yellow}{{}} altered the execution environment (env changed):{reset}", + ) + ) + all_tests.append((self.rerun, "re-run test", f"{yellow}{{}}:{reset}")) + all_tests.append((self.bad, "test", f"{red}{{}} failed:{reset}")) + + for tests_list, count_text, title_format in all_tests: + if tests_list: + print() + count_text = count(len(tests_list), count_text) + print(title_format.format(count_text)) + printlist(tests_list) + + if self.good and not quiet: + print() + text = count(len(self.good), "test") + text = f"{green}{text} OK.{reset}" + if self.is_all_good() and len(self.good) > 1: + text = f"All {text}" + print(text) + + if self.interrupted: + print() + print(f"{yellow}Test suite interrupted by signal SIGINT.{reset}") + + def display_summary(self, first_runtests: RunTests, filtered: bool) -> None: + # Total tests + ansi = get_colors() + red, reset, yellow = ansi.RED, ansi.RESET, ansi.YELLOW + + stats = self.stats + text = f'run={stats.tests_run:,}' + if filtered: + text = f"{text} (filtered)" + report = [text] + if stats.failures: + report.append(f'{red}failures={stats.failures:,}{reset}') + if stats.skipped: + report.append(f'{yellow}skipped={stats.skipped:,}{reset}') + print(f"Total tests: {' '.join(report)}") + + # Total test files + all_tests = [self.good, self.bad, self.rerun, + self.skipped, + self.env_changed, self.run_no_tests] + run = sum(map(len, all_tests)) + text = f'run={run}' + if not first_runtests.forever: + ntest = len(first_runtests.tests) + text = f"{text}/{ntest}" + if filtered: + text = f"{text} (filtered)" + report = [text] + for name, tests, color in ( + ('failed', self.bad, red), + ('env_changed', self.env_changed, yellow), + ('skipped', self.skipped, yellow), + ('resource_denied', self.resource_denied, yellow), + ('rerun', self.rerun, yellow), + ('run_no_tests', self.run_no_tests, yellow), + ): + if tests: + report.append(f'{color}{name}={len(tests)}{reset}') + print(f"Total test files: {' '.join(report)}") diff --git a/Lib/test/libregrtest/run_workers.py b/Lib/test/libregrtest/run_workers.py new file mode 100644 index 00000000000..424085a0050 --- /dev/null +++ b/Lib/test/libregrtest/run_workers.py @@ -0,0 +1,627 @@ +import contextlib +import dataclasses +import faulthandler +import os.path +import queue +import signal +import subprocess +import sys +import tempfile +import threading +import time +import traceback +from typing import Any, Literal, TextIO + +from test import support +from test.support import os_helper, MS_WINDOWS + +from .logger import Logger +from .result import TestResult, State +from .results import TestResults +from .runtests import RunTests, WorkerRunTests, JsonFile, JsonFileType +from .single import PROGRESS_MIN_TIME +from .utils import ( + StrPath, TestName, + format_duration, print_warning, count, plural) +from .worker import create_worker_process, USE_PROCESS_GROUP + +if MS_WINDOWS: + import locale + import msvcrt + + + +# Display the running tests if nothing happened last N seconds +PROGRESS_UPDATE = 30.0 # seconds +assert PROGRESS_UPDATE >= PROGRESS_MIN_TIME + +# Kill the main process after 5 minutes. It is supposed to write an update +# every PROGRESS_UPDATE seconds. Tolerate 5 minutes for Python slowest +# buildbot workers. +MAIN_PROCESS_TIMEOUT = 5 * 60.0 +assert MAIN_PROCESS_TIMEOUT >= PROGRESS_UPDATE + +# Time to wait until a worker completes: should be immediate +WAIT_COMPLETED_TIMEOUT = 30.0 # seconds + +# Time to wait a killed process (in seconds) +WAIT_KILLED_TIMEOUT = 60.0 + + +# We do not use a generator so multiple threads can call next(). +class MultiprocessIterator: + + """A thread-safe iterator over tests for multiprocess mode.""" + + def __init__(self, tests_iter): + self.lock = threading.Lock() + self.tests_iter = tests_iter + + def __iter__(self): + return self + + def __next__(self): + with self.lock: + if self.tests_iter is None: + raise StopIteration + return next(self.tests_iter) + + def stop(self): + with self.lock: + self.tests_iter = None + + +@dataclasses.dataclass(slots=True, frozen=True) +class MultiprocessResult: + result: TestResult + # bpo-45410: stderr is written into stdout to keep messages order + worker_stdout: str | None = None + err_msg: str | None = None + + +class WorkerThreadExited: + """Indicates that a worker thread has exited""" + +ExcStr = str +QueueOutput = tuple[Literal[False], MultiprocessResult] | tuple[Literal[True], ExcStr] +QueueContent = QueueOutput | WorkerThreadExited + + +class ExitThread(Exception): + pass + + +class WorkerError(Exception): + def __init__(self, + test_name: TestName, + err_msg: str | None, + stdout: str | None, + state: str): + result = TestResult(test_name, state=state) + self.mp_result = MultiprocessResult(result, stdout, err_msg) + super().__init__() + + +_NOT_RUNNING = "" + + +class WorkerThread(threading.Thread): + def __init__(self, worker_id: int, runner: "RunWorkers") -> None: + super().__init__() + self.worker_id = worker_id + self.runtests = runner.runtests + self.pending = runner.pending + self.output = runner.output + self.timeout = runner.worker_timeout + self.log = runner.log + self.test_name = _NOT_RUNNING + self.start_time = time.monotonic() + self._popen: subprocess.Popen[str] | None = None + self._killed = False + self._stopped = False + + def __repr__(self) -> str: + info = [f'WorkerThread #{self.worker_id}'] + if self.is_alive(): + info.append("running") + else: + info.append('stopped') + test = self.test_name + if test: + info.append(f'test={test}') + popen = self._popen + if popen is not None: + dt = time.monotonic() - self.start_time + info.extend((f'pid={popen.pid}', + f'time={format_duration(dt)}')) + return '<%s>' % ' '.join(info) + + def _kill(self) -> None: + popen = self._popen + if popen is None: + return + + if self._killed: + return + self._killed = True + + use_killpg = USE_PROCESS_GROUP + if use_killpg: + parent_sid = os.getsid(0) + sid = os.getsid(popen.pid) + use_killpg = (sid != parent_sid) + + if use_killpg: + what = f"{self} process group" + else: + what = f"{self} process" + + print(f"Kill {what}", file=sys.stderr, flush=True) + try: + if use_killpg: + os.killpg(popen.pid, signal.SIGKILL) + else: + popen.kill() + except ProcessLookupError: + # popen.kill(): the process completed, the WorkerThread thread + # read its exit status, but Popen.send_signal() read the returncode + # just before Popen.wait() set returncode. + pass + except OSError as exc: + print_warning(f"Failed to kill {what}: {exc!r}") + + def stop(self) -> None: + # Method called from a different thread to stop this thread + self._stopped = True + self._kill() + + def _run_process(self, runtests: WorkerRunTests, output_fd: int, + tmp_dir: StrPath | None = None) -> int | None: + popen = create_worker_process(runtests, output_fd, tmp_dir) + self._popen = popen + self._killed = False + + try: + if self._stopped: + # If kill() has been called before self._popen is set, + # self._popen is still running. Call again kill() + # to ensure that the process is killed. + self._kill() + raise ExitThread + + try: + # gh-94026: stdout+stderr are written to tempfile + retcode = popen.wait(timeout=self.timeout) + assert retcode is not None + return retcode + except subprocess.TimeoutExpired: + if self._stopped: + # kill() has been called: communicate() fails on reading + # closed stdout + raise ExitThread + + # On timeout, kill the process + self._kill() + + # None means TIMEOUT for the caller + retcode = None + # bpo-38207: Don't attempt to call communicate() again: on it + # can hang until all child processes using stdout + # pipes completes. + except OSError: + if self._stopped: + # kill() has been called: communicate() fails + # on reading closed stdout + raise ExitThread + raise + return None + except: + self._kill() + raise + finally: + self._wait_completed() + self._popen = None + + def create_stdout(self, stack: contextlib.ExitStack) -> TextIO: + """Create stdout temporary file (file descriptor).""" + + if MS_WINDOWS: + # gh-95027: When stdout is not a TTY, Python uses the ANSI code + # page for the sys.stdout encoding. If the main process runs in a + # terminal, sys.stdout uses WindowsConsoleIO with UTF-8 encoding. + encoding = locale.getencoding() + else: + encoding = sys.stdout.encoding + + # gh-94026: Write stdout+stderr to a tempfile as workaround for + # non-blocking pipes on Emscripten with NodeJS. + # gh-109425: Use "backslashreplace" error handler: log corrupted + # stdout+stderr, instead of failing with a UnicodeDecodeError and not + # logging stdout+stderr at all. + stdout_file = tempfile.TemporaryFile('w+', + encoding=encoding, + errors='backslashreplace') + stack.enter_context(stdout_file) + return stdout_file + + def create_json_file(self, stack: contextlib.ExitStack) -> tuple[JsonFile, TextIO | None]: + """Create JSON file.""" + + json_file_use_stdout = self.runtests.json_file_use_stdout() + if json_file_use_stdout: + json_file = JsonFile(None, JsonFileType.STDOUT) + json_tmpfile = None + else: + json_tmpfile = tempfile.TemporaryFile('w+', encoding='utf8') + stack.enter_context(json_tmpfile) + + json_fd = json_tmpfile.fileno() + if MS_WINDOWS: + # The msvcrt module is only available on Windows; + # we run mypy with `--platform=linux` in CI + json_handle: int = msvcrt.get_osfhandle(json_fd) # type: ignore[attr-defined] + json_file = JsonFile(json_handle, + JsonFileType.WINDOWS_HANDLE) + else: + json_file = JsonFile(json_fd, JsonFileType.UNIX_FD) + return (json_file, json_tmpfile) + + def create_worker_runtests(self, test_name: TestName, json_file: JsonFile) -> WorkerRunTests: + tests = (test_name,) + if self.runtests.rerun: + match_tests = self.runtests.get_match_tests(test_name) + else: + match_tests = None + + kwargs: dict[str, Any] = {} + if match_tests: + kwargs['match_tests'] = [(test, True) for test in match_tests] + if self.runtests.output_on_failure: + kwargs['verbose'] = True + kwargs['output_on_failure'] = False + return self.runtests.create_worker_runtests( + tests=tests, + json_file=json_file, + **kwargs) + + def run_tmp_files(self, worker_runtests: WorkerRunTests, + stdout_fd: int) -> tuple[int | None, list[StrPath]]: + # gh-93353: Check for leaked temporary files in the parent process, + # since the deletion of temporary files can happen late during + # Python finalization: too late for libregrtest. + if not support.is_wasi: + # Don't check for leaked temporary files and directories if Python is + # run on WASI. WASI doesn't pass environment variables like TMPDIR to + # worker processes. + tmp_dir = tempfile.mkdtemp(prefix="test_python_") + tmp_dir = os.path.abspath(tmp_dir) + try: + retcode = self._run_process(worker_runtests, + stdout_fd, tmp_dir) + finally: + tmp_files = os.listdir(tmp_dir) + os_helper.rmtree(tmp_dir) + else: + retcode = self._run_process(worker_runtests, stdout_fd) + tmp_files = [] + + return (retcode, tmp_files) + + def read_stdout(self, stdout_file: TextIO) -> str: + stdout_file.seek(0) + try: + return stdout_file.read().strip() + except Exception as exc: + # gh-101634: Catch UnicodeDecodeError if stdout cannot be + # decoded from encoding + raise WorkerError(self.test_name, + f"Cannot read process stdout: {exc}", + stdout=None, + state=State.WORKER_BUG) + + def read_json(self, json_file: JsonFile, json_tmpfile: TextIO | None, + stdout: str) -> tuple[TestResult, str]: + try: + if json_tmpfile is not None: + json_tmpfile.seek(0) + worker_json = json_tmpfile.read() + elif json_file.file_type == JsonFileType.STDOUT: + stdout, _, worker_json = stdout.rpartition("\n") + stdout = stdout.rstrip() + else: + with json_file.open(encoding='utf8') as json_fp: + worker_json = json_fp.read() + except Exception as exc: + # gh-101634: Catch UnicodeDecodeError if stdout cannot be + # decoded from encoding + err_msg = f"Failed to read worker process JSON: {exc}" + raise WorkerError(self.test_name, err_msg, stdout, + state=State.WORKER_BUG) + + if not worker_json: + raise WorkerError(self.test_name, "empty JSON", stdout, + state=State.WORKER_BUG) + + try: + result = TestResult.from_json(worker_json) + except Exception as exc: + # gh-101634: Catch UnicodeDecodeError if stdout cannot be + # decoded from encoding + err_msg = f"Failed to parse worker process JSON: {exc}" + raise WorkerError(self.test_name, err_msg, stdout, + state=State.WORKER_BUG) + + return (result, stdout) + + def _runtest(self, test_name: TestName) -> MultiprocessResult: + with contextlib.ExitStack() as stack: + stdout_file = self.create_stdout(stack) + json_file, json_tmpfile = self.create_json_file(stack) + worker_runtests = self.create_worker_runtests(test_name, json_file) + + retcode: str | int | None + retcode, tmp_files = self.run_tmp_files(worker_runtests, + stdout_file.fileno()) + + stdout = self.read_stdout(stdout_file) + + if retcode is None: + raise WorkerError(self.test_name, stdout=stdout, + err_msg=None, + state=State.TIMEOUT) + if retcode != 0: + name = support.get_signal_name(retcode) + if name: + retcode = f"{retcode} ({name})" + raise WorkerError(self.test_name, f"Exit code {retcode}", stdout, + state=State.WORKER_FAILED) + + result, stdout = self.read_json(json_file, json_tmpfile, stdout) + + if tmp_files: + msg = (f'\n\n' + f'Warning -- {test_name} leaked temporary files ' + f'({len(tmp_files)}): {", ".join(sorted(tmp_files))}') + stdout += msg + result.set_env_changed() + + return MultiprocessResult(result, stdout) + + def run(self) -> None: + fail_fast = self.runtests.fail_fast + fail_env_changed = self.runtests.fail_env_changed + try: + while not self._stopped: + try: + test_name = next(self.pending) + except StopIteration: + break + + self.start_time = time.monotonic() + self.test_name = test_name + try: + mp_result = self._runtest(test_name) + except WorkerError as exc: + mp_result = exc.mp_result + finally: + self.test_name = _NOT_RUNNING + mp_result.result.duration = time.monotonic() - self.start_time + self.output.put((False, mp_result)) + + if mp_result.result.must_stop(fail_fast, fail_env_changed): + break + except ExitThread: + pass + except BaseException: + self.output.put((True, traceback.format_exc())) + finally: + self.output.put(WorkerThreadExited()) + + def _wait_completed(self) -> None: + popen = self._popen + # only needed for mypy: + if popen is None: + raise ValueError("Should never access `._popen` before calling `.run()`") + + try: + popen.wait(WAIT_COMPLETED_TIMEOUT) + except (subprocess.TimeoutExpired, OSError) as exc: + print_warning(f"Failed to wait for {self} completion " + f"(timeout={format_duration(WAIT_COMPLETED_TIMEOUT)}): " + f"{exc!r}") + + def wait_stopped(self, start_time: float) -> None: + # bpo-38207: RunWorkers.stop_workers() called self.stop() + # which killed the process. Sometimes, killing the process from the + # main thread does not interrupt popen.communicate() in + # WorkerThread thread. This loop with a timeout is a workaround + # for that. + # + # Moreover, if this method fails to join the thread, it is likely + # that Python will hang at exit while calling threading._shutdown() + # which tries again to join the blocked thread. Regrtest.main() + # uses EXIT_TIMEOUT to workaround this second bug. + while True: + # Write a message every second + self.join(1.0) + if not self.is_alive(): + break + dt = time.monotonic() - start_time + self.log(f"Waiting for {self} thread for {format_duration(dt)}") + if dt > WAIT_KILLED_TIMEOUT: + print_warning(f"Failed to join {self} in {format_duration(dt)}") + break + + +def get_running(workers: list[WorkerThread]) -> str | None: + running: list[str] = [] + for worker in workers: + test_name = worker.test_name + if test_name == _NOT_RUNNING: + continue + dt = time.monotonic() - worker.start_time + if dt >= PROGRESS_MIN_TIME: + text = f'{test_name} ({format_duration(dt)})' + running.append(text) + if not running: + return None + return f"running ({len(running)}): {', '.join(running)}" + + +class RunWorkers: + def __init__(self, num_workers: int, runtests: RunTests, + logger: Logger, results: TestResults) -> None: + self.num_workers = num_workers + self.runtests = runtests + self.log = logger.log + self.display_progress = logger.display_progress + self.results: TestResults = results + self.live_worker_count = 0 + + self.output: queue.Queue[QueueContent] = queue.Queue() + tests_iter = runtests.iter_tests() + self.pending = MultiprocessIterator(tests_iter) + self.timeout = runtests.timeout + if self.timeout is not None: + # Rely on faulthandler to kill a worker process. This timouet is + # when faulthandler fails to kill a worker process. Give a maximum + # of 5 minutes to faulthandler to kill the worker. + self.worker_timeout: float | None = min(self.timeout * 1.5, self.timeout + 5 * 60) + else: + self.worker_timeout = None + self.workers: list[WorkerThread] = [] + + jobs = self.runtests.get_jobs() + if jobs is not None: + # Don't spawn more threads than the number of jobs: + # these worker threads would never get anything to do. + self.num_workers = min(self.num_workers, jobs) + + def start_workers(self) -> None: + self.workers = [WorkerThread(index, self) + for index in range(1, self.num_workers + 1)] + jobs = self.runtests.get_jobs() + if jobs is not None: + tests = count(jobs, 'test') + else: + tests = 'tests' + nworkers = len(self.workers) + processes = plural(nworkers, "process", "processes") + msg = (f"Run {tests} in parallel using " + f"{nworkers} worker {processes}") + if self.timeout and self.worker_timeout is not None: + msg += (" (timeout: %s, worker timeout: %s)" + % (format_duration(self.timeout), + format_duration(self.worker_timeout))) + self.log(msg) + for worker in self.workers: + worker.start() + self.live_worker_count += 1 + + def stop_workers(self) -> None: + start_time = time.monotonic() + for worker in self.workers: + worker.stop() + for worker in self.workers: + worker.wait_stopped(start_time) + + def _get_result(self) -> QueueOutput | None: + pgo = self.runtests.pgo + use_faulthandler = (self.timeout is not None) + + # bpo-46205: check the status of workers every iteration to avoid + # waiting forever on an empty queue. + while self.live_worker_count > 0: + if use_faulthandler: + faulthandler.dump_traceback_later(MAIN_PROCESS_TIMEOUT, + exit=True) + + # wait for a thread + try: + result = self.output.get(timeout=PROGRESS_UPDATE) + if isinstance(result, WorkerThreadExited): + self.live_worker_count -= 1 + continue + return result + except queue.Empty: + pass + + if not pgo: + # display progress + running = get_running(self.workers) + if running: + self.log(running) + return None + + def display_result(self, mp_result: MultiprocessResult) -> None: + result = mp_result.result + pgo = self.runtests.pgo + + text = str(result) + if mp_result.err_msg: + # WORKER_BUG + text += ' (%s)' % mp_result.err_msg + elif (result.duration and result.duration >= PROGRESS_MIN_TIME and not pgo): + text += ' (%s)' % format_duration(result.duration) + if not pgo: + running = get_running(self.workers) + if running: + text += f' -- {running}' + self.display_progress(self.test_index, text) + + def _process_result(self, item: QueueOutput) -> TestResult: + """Returns True if test runner must stop.""" + if item[0]: + # Thread got an exception + format_exc = item[1] + print_warning(f"regrtest worker thread failed: {format_exc}") + result = TestResult("", state=State.WORKER_BUG) + self.results.accumulate_result(result, self.runtests) + return result + + self.test_index += 1 + mp_result = item[1] + result = mp_result.result + self.results.accumulate_result(result, self.runtests) + self.display_result(mp_result) + + # Display worker stdout + if not self.runtests.output_on_failure: + show_stdout = True + else: + # --verbose3 ignores stdout on success + show_stdout = (result.state != State.PASSED) + if show_stdout: + stdout = mp_result.worker_stdout + if stdout: + print(stdout, flush=True) + + return result + + def run(self) -> None: + fail_fast = self.runtests.fail_fast + fail_env_changed = self.runtests.fail_env_changed + + self.start_workers() + + self.test_index = 0 + try: + while True: + item = self._get_result() + if item is None: + break + + result = self._process_result(item) + if result.must_stop(fail_fast, fail_env_changed): + break + except KeyboardInterrupt: + print() + self.results.interrupted = True + finally: + if self.timeout is not None: + faulthandler.cancel_dump_traceback_later() + + # Always ensure that all worker processes are no longer + # worker when we exit this function + self.pending.stop() + self.stop_workers() diff --git a/Lib/test/libregrtest/runtest.py b/Lib/test/libregrtest/runtest.py deleted file mode 100644 index e2af18f3499..00000000000 --- a/Lib/test/libregrtest/runtest.py +++ /dev/null @@ -1,328 +0,0 @@ -import collections -import faulthandler -import functools -# import gc -import importlib -import io -import os -import sys -import time -import traceback -import unittest - -from test import support -from test.support import os_helper, import_helper -from test.libregrtest.refleak import dash_R, clear_caches -from test.libregrtest.save_env import saved_test_environment -from test.libregrtest.utils import print_warning - - -# Test result constants. -PASSED = 1 -FAILED = 0 -ENV_CHANGED = -1 -SKIPPED = -2 -RESOURCE_DENIED = -3 -INTERRUPTED = -4 -CHILD_ERROR = -5 # error in a child process -TEST_DID_NOT_RUN = -6 # error in a child process - -_FORMAT_TEST_RESULT = { - PASSED: '%s passed', - FAILED: '%s failed', - ENV_CHANGED: '%s failed (env changed)', - SKIPPED: '%s skipped', - RESOURCE_DENIED: '%s skipped (resource denied)', - INTERRUPTED: '%s interrupted', - CHILD_ERROR: '%s crashed', - TEST_DID_NOT_RUN: '%s run no tests', -} - -# Minimum duration of a test to display its duration or to mention that -# the test is running in background -PROGRESS_MIN_TIME = 30.0 # seconds - -# small set of tests to determine if we have a basically functioning interpreter -# (i.e. if any of these fail, then anything else is likely to follow) -STDTESTS = [ - # 'test_grammar', - # 'test_opcodes', - # 'test_dict', - # 'test_builtin', - # 'test_exceptions', - # 'test_types', - # 'test_unittest', - # 'test_doctest', - # 'test_doctest2', - # 'test_support' -] - -# set of tests that we don't want to be executed when using regrtest -NOTTESTS = set() - - -# used by --findleaks, store for gc.garbage -FOUND_GARBAGE = [] - - -def format_test_result(result): - fmt = _FORMAT_TEST_RESULT.get(result.result, "%s") - return fmt % result.test_name - - -def findtestdir(path=None): - return path or os.path.dirname(os.path.dirname(__file__)) or os.curdir - - -def findtests(testdir=None, stdtests=STDTESTS, nottests=NOTTESTS): - """Return a list of all applicable test modules.""" - testdir = findtestdir(testdir) - names = os.listdir(testdir) - tests = [] - others = set(stdtests) | nottests - for name in names: - mod, ext = os.path.splitext(name) - if mod[:5] == "test_" and ext in (".py", "") and mod not in others: - tests.append(mod) - return stdtests + sorted(tests) - - -def get_abs_module(ns, test_name): - if test_name.startswith('test.') or ns.testdir: - return test_name - else: - # Import it from the test package - return 'test.' + test_name - - -TestResult = collections.namedtuple('TestResult', - 'test_name result test_time xml_data') - -def _runtest(ns, test_name): - # Handle faulthandler timeout, capture stdout+stderr, XML serialization - # and measure time. - - output_on_failure = ns.verbose3 - - use_timeout = (ns.timeout is not None) - if use_timeout: - faulthandler.dump_traceback_later(ns.timeout, exit=True) - - start_time = time.perf_counter() - try: - support.set_match_tests(ns.match_tests) - support.junit_xml_list = xml_list = [] if ns.xmlpath else None - if ns.failfast: - support.failfast = True - - if output_on_failure: - support.verbose = True - - stream = io.StringIO() - orig_stdout = sys.stdout - orig_stderr = sys.stderr - try: - sys.stdout = stream - sys.stderr = stream - result = _runtest_inner(ns, test_name, - display_failure=False) - if result != PASSED: - output = stream.getvalue() - orig_stderr.write(output) - orig_stderr.flush() - finally: - sys.stdout = orig_stdout - sys.stderr = orig_stderr - else: - # Tell tests to be moderately quiet - support.verbose = ns.verbose - - result = _runtest_inner(ns, test_name, - display_failure=not ns.verbose) - - if xml_list: - import xml.etree.ElementTree as ET - xml_data = [ET.tostring(x).decode('us-ascii') for x in xml_list] - else: - xml_data = None - - test_time = time.perf_counter() - start_time - - return TestResult(test_name, result, test_time, xml_data) - finally: - if use_timeout: - faulthandler.cancel_dump_traceback_later() - support.junit_xml_list = None - - -def runtest(ns, test_name): - """Run a single test. - - ns -- regrtest namespace of options - test_name -- the name of the test - - Returns the tuple (result, test_time, xml_data), where result is one - of the constants: - - INTERRUPTED KeyboardInterrupt - RESOURCE_DENIED test skipped because resource denied - SKIPPED test skipped for some other reason - ENV_CHANGED test failed because it changed the execution environment - FAILED test failed - PASSED test passed - EMPTY_TEST_SUITE test ran no subtests. - - If ns.xmlpath is not None, xml_data is a list containing each - generated testsuite element. - """ - try: - return _runtest(ns, test_name) - except: - if not ns.pgo: - msg = traceback.format_exc() - print(f"test {test_name} crashed -- {msg}", - file=sys.stderr, flush=True) - return TestResult(test_name, FAILED, 0.0, None) - - -def _test_module(the_module): - loader = unittest.TestLoader() - tests = loader.loadTestsFromModule(the_module) - for error in loader.errors: - print(error, file=sys.stderr) - if loader.errors: - raise Exception("errors while loading tests") - support.run_unittest(tests) - - -def _runtest_inner2(ns, test_name): - # Load the test function, run the test function, handle huntrleaks - # and findleaks to detect leaks - - abstest = get_abs_module(ns, test_name) - - # remove the module from sys.module to reload it if it was already imported - import_helper.unload(abstest) - - the_module = importlib.import_module(abstest) - - # If the test has a test_main, that will run the appropriate - # tests. If not, use normal unittest test loading. - test_runner = getattr(the_module, "test_main", None) - if test_runner is None: - test_runner = functools.partial(_test_module, the_module) - - try: - if ns.huntrleaks: - # Return True if the test leaked references - refleak = dash_R(ns, test_name, test_runner) - else: - test_runner() - refleak = False - finally: - cleanup_test_droppings(test_name, ns.verbose) - - support.gc_collect() - - # if gc.garbage: - # support.environment_altered = True - # print_warning(f"{test_name} created {len(gc.garbage)} " - # f"uncollectable object(s).") - - # # move the uncollectable objects somewhere, - # # so we don't see them again - # FOUND_GARBAGE.extend(gc.garbage) - # gc.garbage.clear() - - support.reap_children() - - return refleak - - -def _runtest_inner(ns, test_name, display_failure=True): - # Detect environment changes, handle exceptions. - - # Reset the environment_altered flag to detect if a test altered - # the environment - support.environment_altered = False - - if ns.pgo: - display_failure = False - - try: - clear_caches() - - # with saved_test_environment(test_name, ns.verbose, ns.quiet, pgo=ns.pgo) as environment: - refleak = _runtest_inner2(ns, test_name) - except support.ResourceDenied as msg: - if not ns.quiet and not ns.pgo: - print(f"{test_name} skipped -- {msg}", flush=True) - return RESOURCE_DENIED - except unittest.SkipTest as msg: - if not ns.quiet and not ns.pgo: - print(f"{test_name} skipped -- {msg}", flush=True) - return SKIPPED - except support.TestFailed as exc: - msg = f"test {test_name} failed" - if display_failure: - msg = f"{msg} -- {exc}" - print(msg, file=sys.stderr, flush=True) - return FAILED - except support.TestDidNotRun: - return TEST_DID_NOT_RUN - except KeyboardInterrupt: - print() - return INTERRUPTED - except: - if not ns.pgo: - msg = traceback.format_exc() - print(f"test {test_name} crashed -- {msg}", - file=sys.stderr, flush=True) - return FAILED - - if refleak: - return FAILED - # if environment.changed: - # return ENV_CHANGED - return PASSED - - -def cleanup_test_droppings(test_name, verbose): - # First kill any dangling references to open files etc. - # This can also issue some ResourceWarnings which would otherwise get - # triggered during the following test run, and possibly produce failures. - support.gc_collect() - - # Try to clean up junk commonly left behind. While tests shouldn't leave - # any files or directories behind, when a test fails that can be tedious - # for it to arrange. The consequences can be especially nasty on Windows, - # since if a test leaves a file open, it cannot be deleted by name (while - # there's nothing we can do about that here either, we can display the - # name of the offending test, which is a real help). - for name in (os_helper.TESTFN, - "db_home", - ): - if not os.path.exists(name): - continue - - if os.path.isdir(name): - import shutil - kind, nuker = "directory", shutil.rmtree - elif os.path.isfile(name): - kind, nuker = "file", os.unlink - else: - raise RuntimeError(f"os.path says {name!r} exists but is neither " - f"directory nor file") - - if verbose: - print_warning("%r left behind %s %r" % (test_name, kind, name)) - support.environment_altered = True - - try: - import stat - # fix possible permissions problems that might prevent cleanup - os.chmod(name, stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO) - nuker(name) - except Exception as exc: - print_warning(f"{test_name} left behind {kind} {name!r} " - f"and it couldn't be removed: {exc}") diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py deleted file mode 100644 index c2177d99955..00000000000 --- a/Lib/test/libregrtest/runtest_mp.py +++ /dev/null @@ -1,288 +0,0 @@ -import collections -import faulthandler -import json -import os -import queue -import subprocess -import sys -import threading -import time -import traceback -import types -from test import support - -from test.libregrtest.runtest import ( - runtest, INTERRUPTED, CHILD_ERROR, PROGRESS_MIN_TIME, - format_test_result, TestResult) -from test.libregrtest.setup import setup_tests -from test.libregrtest.utils import format_duration -from test.support import os_helper - - -# Display the running tests if nothing happened last N seconds -PROGRESS_UPDATE = 30.0 # seconds - - -def must_stop(result): - return result.result in (INTERRUPTED, CHILD_ERROR) - - -def run_test_in_subprocess(testname, ns): - ns_dict = vars(ns) - worker_args = (ns_dict, testname) - worker_args = json.dumps(worker_args) - - cmd = [sys.executable, *support.args_from_interpreter_flags(), - '-u', # Unbuffered stdout and stderr - '-m', 'test.regrtest', - '--worker-args', worker_args] - if ns.pgo: - cmd += ['--pgo'] - - # Running the child from the same working directory as regrtest's original - # invocation ensures that TEMPDIR for the child is the same when - # sysconfig.is_python_build() is true. See issue 15300. - return subprocess.Popen(cmd, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - universal_newlines=True, - close_fds=(os.name != 'nt'), - cwd=os_helper.SAVEDCWD) - - -def run_tests_worker(worker_args): - ns_dict, testname = json.loads(worker_args) - ns = types.SimpleNamespace(**ns_dict) - - setup_tests(ns) - - result = runtest(ns, testname) - print() # Force a newline (just in case) - print(json.dumps(result), flush=True) - sys.exit(0) - - -# We do not use a generator so multiple threads can call next(). -class MultiprocessIterator: - - """A thread-safe iterator over tests for multiprocess mode.""" - - def __init__(self, tests): - self.lock = threading.Lock() - self.tests = tests - - def __iter__(self): - return self - - def __next__(self): - with self.lock: - return next(self.tests) - - -MultiprocessResult = collections.namedtuple('MultiprocessResult', - 'result stdout stderr error_msg') - -class MultiprocessThread(threading.Thread): - def __init__(self, pending, output, ns): - super().__init__() - self.pending = pending - self.output = output - self.ns = ns - self.current_test_name = None - self.start_time = None - self._popen = None - - def kill(self): - if not self.is_alive(): - return - if self._popen is not None: - self._popen.kill() - - def _runtest(self, test_name): - try: - self.start_time = time.monotonic() - self.current_test_name = test_name - - popen = run_test_in_subprocess(test_name, self.ns) - self._popen = popen - with popen: - try: - stdout, stderr = popen.communicate() - except: - popen.kill() - popen.wait() - raise - - retcode = popen.wait() - finally: - self.current_test_name = None - self._popen = None - - stdout = stdout.strip() - stderr = stderr.rstrip() - - err_msg = None - if retcode != 0: - err_msg = "Exit code %s" % retcode - else: - stdout, _, result = stdout.rpartition("\n") - stdout = stdout.rstrip() - if not result: - err_msg = "Failed to parse worker stdout" - else: - try: - # deserialize run_tests_worker() output - result = json.loads(result) - result = TestResult(*result) - except Exception as exc: - err_msg = "Failed to parse worker JSON: %s" % exc - - if err_msg is not None: - test_time = time.monotonic() - self.start_time - result = TestResult(test_name, CHILD_ERROR, test_time, None) - - return MultiprocessResult(result, stdout, stderr, err_msg) - - def run(self): - while True: - try: - try: - test_name = next(self.pending) - except StopIteration: - break - - mp_result = self._runtest(test_name) - self.output.put((False, mp_result)) - - if must_stop(mp_result.result): - break - except BaseException: - self.output.put((True, traceback.format_exc())) - break - - -def get_running(workers): - running = [] - for worker in workers: - current_test_name = worker.current_test_name - if not current_test_name: - continue - dt = time.monotonic() - worker.start_time - if dt >= PROGRESS_MIN_TIME: - text = '%s (%s)' % (current_test_name, format_duration(dt)) - running.append(text) - return running - - -class MultiprocessRunner: - def __init__(self, regrtest): - self.regrtest = regrtest - self.ns = regrtest.ns - self.output = queue.Queue() - self.pending = MultiprocessIterator(self.regrtest.tests) - if self.ns.timeout is not None: - self.test_timeout = self.ns.timeout * 1.5 - else: - self.test_timeout = None - self.workers = None - - def start_workers(self): - self.workers = [MultiprocessThread(self.pending, self.output, self.ns) - for _ in range(self.ns.use_mp)] - print("Run tests in parallel using %s child processes" - % len(self.workers)) - for worker in self.workers: - worker.start() - - def wait_workers(self): - for worker in self.workers: - worker.kill() - for worker in self.workers: - worker.join() - - def _get_result(self): - if not any(worker.is_alive() for worker in self.workers): - # all worker threads are done: consume pending results - try: - return self.output.get(timeout=0) - except queue.Empty: - return None - - while True: - if self.test_timeout is not None: - faulthandler.dump_traceback_later(self.test_timeout, exit=True) - - # wait for a thread - timeout = max(PROGRESS_UPDATE, PROGRESS_MIN_TIME) - try: - return self.output.get(timeout=timeout) - except queue.Empty: - pass - - # display progress - running = get_running(self.workers) - if running and not self.ns.pgo: - print('running: %s' % ', '.join(running), flush=True) - - def display_result(self, mp_result): - result = mp_result.result - - text = format_test_result(result) - if mp_result.error_msg is not None: - # CHILD_ERROR - text += ' (%s)' % mp_result.error_msg - elif (result.test_time >= PROGRESS_MIN_TIME and not self.ns.pgo): - text += ' (%s)' % format_duration(result.test_time) - running = get_running(self.workers) - if running and not self.ns.pgo: - text += ' -- running: %s' % ', '.join(running) - self.regrtest.display_progress(self.test_index, text) - - def _process_result(self, item): - if item[0]: - # Thread got an exception - format_exc = item[1] - print(f"regrtest worker thread failed: {format_exc}", - file=sys.stderr, flush=True) - return True - - self.test_index += 1 - mp_result = item[1] - self.regrtest.accumulate_result(mp_result.result) - self.display_result(mp_result) - - if mp_result.stdout: - print(mp_result.stdout, flush=True) - if mp_result.stderr and not self.ns.pgo: - print(mp_result.stderr, file=sys.stderr, flush=True) - - if must_stop(mp_result.result): - return True - - return False - - def run_tests(self): - self.start_workers() - - self.test_index = 0 - try: - while True: - item = self._get_result() - if item is None: - break - - stop = self._process_result(item) - if stop: - break - except KeyboardInterrupt: - print() - self.regrtest.interrupted = True - finally: - if self.test_timeout is not None: - faulthandler.cancel_dump_traceback_later() - - self.wait_workers() - - -def run_tests_multiprocess(regrtest): - MultiprocessRunner(regrtest).run_tests() diff --git a/Lib/test/libregrtest/runtests.py b/Lib/test/libregrtest/runtests.py new file mode 100644 index 00000000000..759f24fc25e --- /dev/null +++ b/Lib/test/libregrtest/runtests.py @@ -0,0 +1,225 @@ +import contextlib +import dataclasses +import json +import os +import shlex +import subprocess +import sys +from typing import Any, Iterator + +from test import support + +from .utils import ( + StrPath, StrJSON, TestTuple, TestName, TestFilter, FilterTuple, FilterDict) + + +class JsonFileType: + UNIX_FD = "UNIX_FD" + WINDOWS_HANDLE = "WINDOWS_HANDLE" + STDOUT = "STDOUT" + + +@dataclasses.dataclass(slots=True, frozen=True) +class JsonFile: + # file type depends on file_type: + # - UNIX_FD: file descriptor (int) + # - WINDOWS_HANDLE: handle (int) + # - STDOUT: use process stdout (None) + file: int | None + file_type: str + + def configure_subprocess(self, popen_kwargs: dict[str, Any]) -> None: + match self.file_type: + case JsonFileType.UNIX_FD: + # Unix file descriptor + popen_kwargs['pass_fds'] = [self.file] + case JsonFileType.WINDOWS_HANDLE: + # Windows handle + # We run mypy with `--platform=linux` so it complains about this: + startupinfo = subprocess.STARTUPINFO() # type: ignore[attr-defined] + startupinfo.lpAttributeList = {"handle_list": [self.file]} + popen_kwargs['startupinfo'] = startupinfo + + @contextlib.contextmanager + def inherit_subprocess(self) -> Iterator[None]: + if sys.platform == 'win32' and self.file_type == JsonFileType.WINDOWS_HANDLE: + os.set_handle_inheritable(self.file, True) + try: + yield + finally: + os.set_handle_inheritable(self.file, False) + else: + yield + + def open(self, mode='r', *, encoding): + if self.file_type == JsonFileType.STDOUT: + raise ValueError("for STDOUT file type, just use sys.stdout") + + file = self.file + if self.file_type == JsonFileType.WINDOWS_HANDLE: + import msvcrt + # Create a file descriptor from the handle + file = msvcrt.open_osfhandle(file, os.O_WRONLY) + return open(file, mode, encoding=encoding) + + +@dataclasses.dataclass(slots=True, frozen=True) +class HuntRefleak: + warmups: int + runs: int + filename: StrPath + + def bisect_cmd_args(self) -> list[str]: + # Ignore filename since it can contain colon (":"), + # and usually it's not used. Use the default filename. + return ["-R", f"{self.warmups}:{self.runs}:"] + + +@dataclasses.dataclass(slots=True, frozen=True) +class RunTests: + tests: TestTuple + fail_fast: bool + fail_env_changed: bool + match_tests: TestFilter + match_tests_dict: FilterDict | None + rerun: bool + forever: bool + pgo: bool + pgo_extended: bool + output_on_failure: bool + timeout: float | None + verbose: int + quiet: bool + hunt_refleak: HuntRefleak | None + test_dir: StrPath | None + use_junit: bool + coverage: bool + memory_limit: str | None + gc_threshold: int | None + use_resources: tuple[str, ...] + python_cmd: tuple[str, ...] | None + randomize: bool + random_seed: int | str + parallel_threads: int | None + + def copy(self, **override) -> 'RunTests': + state = dataclasses.asdict(self) + state.update(override) + return RunTests(**state) + + def create_worker_runtests(self, **override) -> WorkerRunTests: + state = dataclasses.asdict(self) + state.update(override) + return WorkerRunTests(**state) + + def get_match_tests(self, test_name: TestName) -> FilterTuple | None: + if self.match_tests_dict is not None: + return self.match_tests_dict.get(test_name, None) + else: + return None + + def get_jobs(self) -> int | None: + # Number of run_single_test() calls needed to run all tests. + # None means that there is not bound limit (--forever option). + if self.forever: + return None + return len(self.tests) + + def iter_tests(self) -> Iterator[TestName]: + if self.forever: + while True: + yield from self.tests + else: + yield from self.tests + + def json_file_use_stdout(self) -> bool: + # Use STDOUT in two cases: + # + # - If --python command line option is used; + # - On Emscripten and WASI. + # + # On other platforms, UNIX_FD or WINDOWS_HANDLE can be used. + return ( + bool(self.python_cmd) + or support.is_emscripten + or support.is_wasi + ) + + def create_python_cmd(self) -> list[str]: + python_opts = support.args_from_interpreter_flags() + if self.python_cmd is not None: + executable = self.python_cmd + # Remove -E option, since --python=COMMAND can set PYTHON + # environment variables, such as PYTHONPATH, in the worker + # process. + python_opts = [opt for opt in python_opts if opt != "-E"] + else: + executable = (sys.executable,) + cmd = [*executable, *python_opts] + if '-u' not in python_opts: + cmd.append('-u') # Unbuffered stdout and stderr + if self.coverage: + cmd.append("-Xpresite=test.cov") + return cmd + + def bisect_cmd_args(self) -> list[str]: + args = [] + if self.fail_fast: + args.append("--failfast") + if self.fail_env_changed: + args.append("--fail-env-changed") + if self.timeout: + args.append(f"--timeout={self.timeout}") + if self.hunt_refleak is not None: + args.extend(self.hunt_refleak.bisect_cmd_args()) + if self.test_dir: + args.extend(("--testdir", self.test_dir)) + if self.memory_limit: + args.extend(("--memlimit", self.memory_limit)) + if self.gc_threshold: + args.append(f"--threshold={self.gc_threshold}") + if self.use_resources: + args.extend(("-u", ','.join(self.use_resources))) + if self.python_cmd: + cmd = shlex.join(self.python_cmd) + args.extend(("--python", cmd)) + if self.randomize: + args.append(f"--randomize") + if self.parallel_threads: + args.append(f"--parallel-threads={self.parallel_threads}") + args.append(f"--randseed={self.random_seed}") + return args + + +@dataclasses.dataclass(slots=True, frozen=True) +class WorkerRunTests(RunTests): + json_file: JsonFile + + def as_json(self) -> StrJSON: + return json.dumps(self, cls=_EncodeRunTests) + + @staticmethod + def from_json(worker_json: StrJSON) -> 'WorkerRunTests': + return json.loads(worker_json, object_hook=_decode_runtests) + + +class _EncodeRunTests(json.JSONEncoder): + def default(self, o: Any) -> dict[str, Any]: + if isinstance(o, WorkerRunTests): + result = dataclasses.asdict(o) + result["__runtests__"] = True + return result + else: + return super().default(o) + + +def _decode_runtests(data: dict[str, Any]) -> RunTests | dict[str, Any]: + if "__runtests__" in data: + data.pop('__runtests__') + if data['hunt_refleak']: + data['hunt_refleak'] = HuntRefleak(**data['hunt_refleak']) + if data['json_file']: + data['json_file'] = JsonFile(**data['json_file']) + return WorkerRunTests(**data) + else: + return data diff --git a/Lib/test/libregrtest/save_env.py b/Lib/test/libregrtest/save_env.py index b9a1c0b3926..0ec12d7c475 100644 --- a/Lib/test/libregrtest/save_env.py +++ b/Lib/test/libregrtest/save_env.py @@ -1,20 +1,24 @@ -import asyncio import builtins import locale -import logging import os -import shutil import sys -import sysconfig import threading -import warnings + from test import support from test.support import os_helper -from test.libregrtest.utils import print_warning + +from .utils import print_warning + +# Import termios to save and restore terminal echo. This is only available on +# Unix, and it's fine if the module can't be found. try: - import _multiprocessing, multiprocessing.process -except ImportError: - multiprocessing = None + import termios # noqa: F401 +except ModuleNotFoundError: + pass + + +class SkipTestEnvironment(Exception): + pass # Unit tests are supposed to leave the execution environment unchanged @@ -28,21 +32,19 @@ class saved_test_environment: """Save bits of the test environment and restore them at block exit. - with saved_test_environment(testname, verbose, quiet): + with saved_test_environment(test_name, verbose, quiet): #stuff Unless quiet is True, a warning is printed to stderr if any of - the saved items was changed by the test. The attribute 'changed' - is initially False, but is set to True if a change is detected. + the saved items was changed by the test. The support.environment_altered + attribute is set to True if a change is detected. If verbose is more than 1, the before and after state of changed items is also printed. """ - changed = False - - def __init__(self, testname, verbose=0, quiet=False, *, pgo=False): - self.testname = testname + def __init__(self, test_name, verbose, quiet, *, pgo): + self.test_name = test_name self.verbose = verbose self.quiet = quiet self.pgo = pgo @@ -69,12 +71,41 @@ def __init__(self, testname, verbose=0, quiet=False, *, pgo=False): 'files', 'locale', 'warnings.showwarning', 'shutil_archive_formats', 'shutil_unpack_formats', 'asyncio.events._event_loop_policy', + 'urllib.requests._url_tempfiles', 'urllib.requests._opener', + 'stty_echo', ) + def get_module(self, name): + # function for restore() methods + return sys.modules[name] + + def try_get_module(self, name): + # function for get() methods + try: + return self.get_module(name) + except KeyError: + raise SkipTestEnvironment + + def get_urllib_requests__url_tempfiles(self): + urllib_request = self.try_get_module('urllib.request') + return list(urllib_request._url_tempfiles) + def restore_urllib_requests__url_tempfiles(self, tempfiles): + for filename in tempfiles: + os_helper.unlink(filename) + + def get_urllib_requests__opener(self): + urllib_request = self.try_get_module('urllib.request') + return urllib_request._opener + def restore_urllib_requests__opener(self, opener): + urllib_request = self.get_module('urllib.request') + urllib_request._opener = opener + def get_asyncio_events__event_loop_policy(self): + self.try_get_module('asyncio') return support.maybe_get_event_loop_policy() def restore_asyncio_events__event_loop_policy(self, policy): - asyncio.set_event_loop_policy(policy) + asyncio = self.get_module('asyncio') + asyncio.events._set_event_loop_policy(policy) def get_sys_argv(self): return id(sys.argv), sys.argv, sys.argv[:] @@ -132,39 +163,46 @@ def restore___import__(self, import_): builtins.__import__ = import_ def get_warnings_filters(self): + warnings = self.try_get_module('warnings') return id(warnings.filters), warnings.filters, warnings.filters[:] def restore_warnings_filters(self, saved_filters): + warnings = self.get_module('warnings') warnings.filters = saved_filters[1] warnings.filters[:] = saved_filters[2] def get_asyncore_socket_map(self): - asyncore = sys.modules.get('asyncore') + asyncore = sys.modules.get('test.support.asyncore') # XXX Making a copy keeps objects alive until __exit__ gets called. return asyncore and asyncore.socket_map.copy() or {} def restore_asyncore_socket_map(self, saved_map): - asyncore = sys.modules.get('asyncore') + asyncore = sys.modules.get('test.support.asyncore') if asyncore is not None: asyncore.close_all(ignore_all=True) asyncore.socket_map.update(saved_map) def get_shutil_archive_formats(self): + shutil = self.try_get_module('shutil') # we could call get_archives_formats() but that only returns the # registry keys; we want to check the values too (the functions that # are registered) return shutil._ARCHIVE_FORMATS, shutil._ARCHIVE_FORMATS.copy() def restore_shutil_archive_formats(self, saved): + shutil = self.get_module('shutil') shutil._ARCHIVE_FORMATS = saved[0] shutil._ARCHIVE_FORMATS.clear() shutil._ARCHIVE_FORMATS.update(saved[1]) def get_shutil_unpack_formats(self): + shutil = self.try_get_module('shutil') return shutil._UNPACK_FORMATS, shutil._UNPACK_FORMATS.copy() def restore_shutil_unpack_formats(self, saved): + shutil = self.get_module('shutil') shutil._UNPACK_FORMATS = saved[0] shutil._UNPACK_FORMATS.clear() shutil._UNPACK_FORMATS.update(saved[1]) def get_logging__handlers(self): + logging = self.try_get_module('logging') # _handlers is a WeakValueDictionary return id(logging._handlers), logging._handlers, logging._handlers.copy() def restore_logging__handlers(self, saved_handlers): @@ -172,6 +210,7 @@ def restore_logging__handlers(self, saved_handlers): pass def get_logging__handlerList(self): + logging = self.try_get_module('logging') # _handlerList is a list of weakrefs to handlers return id(logging._handlerList), logging._handlerList, logging._handlerList[:] def restore_logging__handlerList(self, saved_handlerList): @@ -188,46 +227,52 @@ def restore_sys_warnoptions(self, saved_options): # to track reference leaks. def get_threading__dangling(self): # This copies the weakrefs without making any strong reference - return threading._dangling.copy() + # XXX: RUSTPYTHON - filter out dead threads since gc doesn't clean WeakSet. Revert this line when we have a GC + # return threading._dangling.copy() + return {t for t in threading._dangling if t.is_alive()} def restore_threading__dangling(self, saved): threading._dangling.clear() threading._dangling.update(saved) # Same for Process objects def get_multiprocessing_process__dangling(self): - if not multiprocessing: - return None + multiprocessing_process = self.try_get_module('multiprocessing.process') # Unjoined process objects can survive after process exits - multiprocessing.process._cleanup() + multiprocessing_process._cleanup() # This copies the weakrefs without making any strong reference - return multiprocessing.process._dangling.copy() + return multiprocessing_process._dangling.copy() def restore_multiprocessing_process__dangling(self, saved): - if not multiprocessing: - return - multiprocessing.process._dangling.clear() - multiprocessing.process._dangling.update(saved) + multiprocessing_process = self.get_module('multiprocessing.process') + multiprocessing_process._dangling.clear() + multiprocessing_process._dangling.update(saved) def get_sysconfig__CONFIG_VARS(self): # make sure the dict is initialized + sysconfig = self.try_get_module('sysconfig') sysconfig.get_config_var('prefix') return (id(sysconfig._CONFIG_VARS), sysconfig._CONFIG_VARS, dict(sysconfig._CONFIG_VARS)) def restore_sysconfig__CONFIG_VARS(self, saved): + sysconfig = self.get_module('sysconfig') sysconfig._CONFIG_VARS = saved[1] sysconfig._CONFIG_VARS.clear() sysconfig._CONFIG_VARS.update(saved[2]) def get_sysconfig__INSTALL_SCHEMES(self): + sysconfig = self.try_get_module('sysconfig') return (id(sysconfig._INSTALL_SCHEMES), sysconfig._INSTALL_SCHEMES, sysconfig._INSTALL_SCHEMES.copy()) def restore_sysconfig__INSTALL_SCHEMES(self, saved): + sysconfig = self.get_module('sysconfig') sysconfig._INSTALL_SCHEMES = saved[1] sysconfig._INSTALL_SCHEMES.clear() sysconfig._INSTALL_SCHEMES.update(saved[2]) def get_files(self): + # XXX: Maybe add an allow-list here? return sorted(fn + ('/' if os.path.isdir(fn) else '') - for fn in os.listdir()) + for fn in os.listdir() + if not fn.startswith(".hypothesis")) def restore_files(self, saved_value): fn = os_helper.TESTFN if fn not in saved_value and (fn + '/') not in saved_value: @@ -251,10 +296,30 @@ def restore_locale(self, saved): locale.setlocale(lc, setting) def get_warnings_showwarning(self): + warnings = self.try_get_module('warnings') return warnings.showwarning def restore_warnings_showwarning(self, fxn): + warnings = self.get_module('warnings') warnings.showwarning = fxn + def get_stty_echo(self): + termios = self.try_get_module('termios') + if not os.isatty(fd := sys.__stdin__.fileno()): + return None + attrs = termios.tcgetattr(fd) + lflags = attrs[3] + return bool(lflags & termios.ECHO) + def restore_stty_echo(self, echo): + termios = self.get_module('termios') + attrs = termios.tcgetattr(fd := sys.__stdin__.fileno()) + if echo: + # Turn echo on. + attrs[3] |= termios.ECHO + else: + # Turn echo off. + attrs[3] &= ~termios.ECHO + termios.tcsetattr(fd, termios.TCSADRAIN, attrs) + def resource_info(self): for name in self.resources: method_suffix = name.replace('.', '_') @@ -263,29 +328,32 @@ def resource_info(self): yield name, getattr(self, get_name), getattr(self, restore_name) def __enter__(self): - self.saved_values = dict((name, get()) for name, get, restore - in self.resource_info()) + self.saved_values = [] + for name, get, restore in self.resource_info(): + try: + original = get() + except SkipTestEnvironment: + continue + + self.saved_values.append((name, get, restore, original)) return self def __exit__(self, exc_type, exc_val, exc_tb): saved_values = self.saved_values - del self.saved_values + self.saved_values = None # Some resources use weak references support.gc_collect() - # Read support.environment_altered, set by support helper functions - self.changed |= support.environment_altered - - for name, get, restore in self.resource_info(): + for name, get, restore, original in saved_values: current = get() - original = saved_values.pop(name) # Check for changes to the resource's value if current != original: - self.changed = True + support.environment_altered = True restore(original) if not self.quiet and not self.pgo: - print_warning(f"{name} was modified by {self.testname}") - print(f" Before: {original}\n After: {current} ", - file=sys.stderr, flush=True) + print_warning( + f"{name} was modified by {self.test_name}\n" + f" Before: {original}\n" + f" After: {current} ") return False diff --git a/Lib/test/libregrtest/setup.py b/Lib/test/libregrtest/setup.py index b1a5ded5254..b9b76a44e3b 100644 --- a/Lib/test/libregrtest/setup.py +++ b/Lib/test/libregrtest/setup.py @@ -1,17 +1,34 @@ -import atexit import faulthandler +import gc +import io import os +import random import signal import sys import unittest from test import support -try: - import gc -except ImportError: - gc = None +from test.support.os_helper import TESTFN_UNDECODABLE, FS_NONASCII +from _colorize import can_colorize # type: ignore[import-not-found] +from .filter import set_match_tests +from .runtests import RunTests +from .utils import ( + setup_unraisable_hook, setup_threading_excepthook, + adjust_rlimit_nofile) -def setup_tests(ns): + +UNICODE_GUARD_ENV = "PYTHONREGRTEST_UNICODE_GUARD" + + +def setup_test_dir(testdir: str | None) -> None: + if testdir: + # Prepend test directory to sys.path, so runtest() will be able + # to locate tests + sys.path.insert(0, os.path.abspath(testdir)) + + +def setup_process() -> None: + assert sys.__stderr__ is not None, "sys.__stderr__ is None" try: stderr_fd = sys.__stderr__.fileno() except (ValueError, AttributeError): @@ -19,13 +36,13 @@ def setup_tests(ns): # and ValueError on a closed stream. # # Catch AttributeError for stderr being None. - stderr_fd = None + pass else: # Display the Python traceback on fatal errors (e.g. segfault) faulthandler.enable(all_threads=True, file=stderr_fd) # Display the Python traceback on SIGALRM or SIGUSR1 signal - signals = [] + signals: list[signal.Signals] = [] if hasattr(signal, 'SIGALRM'): signals.append(signal.SIGALRM) if hasattr(signal, 'SIGUSR1'): @@ -33,13 +50,17 @@ def setup_tests(ns): for signum in signals: faulthandler.register(signum, chain=True, file=stderr_fd) - # replace_stdout() - # support.record_original_stdout(sys.stdout) + adjust_rlimit_nofile() - if ns.testdir: - # Prepend test directory to sys.path, so runtest() will be able - # to locate tests - sys.path.insert(0, os.path.abspath(ns.testdir)) + support.record_original_stdout(sys.stdout) + + # Set sys.stdout encoder error handler to backslashreplace, + # similar to sys.stderr error handler, to avoid UnicodeEncodeError + # when printing a traceback or any other non-encodable character. + # + # Use an assertion to fix mypy error. + assert isinstance(sys.stdout, io.TextIOWrapper) + sys.stdout.reconfigure(errors="backslashreplace") # Some times __path__ and __file__ are not absolute (e.g. while running from # Lib/) and, if we change the CWD to run the tests in a temporary dir, some @@ -56,79 +77,73 @@ def setup_tests(ns): for index, path in enumerate(module.__path__): module.__path__[index] = os.path.abspath(path) if getattr(module, '__file__', None): - module.__file__ = os.path.abspath(module.__file__) - - # MacOSX (a.k.a. Darwin) has a default stack size that is too small - # for deeply recursive regular expressions. We see this as crashes in - # the Python test suite when running test_re.py and test_sre.py. The - # fix is to set the stack limit to 2048. - # This approach may also be useful for other Unixy platforms that - # suffer from small default stack limits. - if sys.platform == 'darwin': - try: - import resource - except ImportError: + module.__file__ = os.path.abspath(module.__file__) # type: ignore[type-var] + + if hasattr(sys, 'addaudithook'): + # Add an auditing hook for all tests to ensure PySys_Audit is tested + def _test_audit_hook(name, args): pass - else: - soft, hard = resource.getrlimit(resource.RLIMIT_STACK) - newsoft = min(hard, max(soft, 1024*2048)) - resource.setrlimit(resource.RLIMIT_STACK, (newsoft, hard)) + sys.addaudithook(_test_audit_hook) - if ns.huntrleaks: - unittest.BaseTestSuite._cleanup = False + setup_unraisable_hook() + setup_threading_excepthook() - if ns.memlimit is not None: - support.set_memlimit(ns.memlimit) + # Ensure there's a non-ASCII character in env vars at all times to force + # tests consider this case. See BPO-44647 for details. + if TESTFN_UNDECODABLE and os.supports_bytes_environ: + os.environb.setdefault(UNICODE_GUARD_ENV.encode(), TESTFN_UNDECODABLE) + elif FS_NONASCII: + os.environ.setdefault(UNICODE_GUARD_ENV, FS_NONASCII) - if ns.threshold is not None: - gc.set_threshold(ns.threshold) - try: - import msvcrt - except ImportError: - pass +def setup_tests(runtests: RunTests) -> None: + support.verbose = runtests.verbose + support.failfast = runtests.fail_fast + support.PGO = runtests.pgo + support.PGO_EXTENDED = runtests.pgo_extended + + set_match_tests(runtests.match_tests) + + if runtests.use_junit: + support.junit_xml_list = [] + from .testresult import RegressionTestResult + RegressionTestResult.USE_XML = True else: - msvcrt.SetErrorMode(msvcrt.SEM_FAILCRITICALERRORS| - msvcrt.SEM_NOALIGNMENTFAULTEXCEPT| - msvcrt.SEM_NOGPFAULTERRORBOX| - msvcrt.SEM_NOOPENFILEERRORBOX) - try: - msvcrt.CrtSetReportMode - except AttributeError: - # release build - pass - else: - for m in [msvcrt.CRT_WARN, msvcrt.CRT_ERROR, msvcrt.CRT_ASSERT]: - if ns.verbose and ns.verbose >= 2: - msvcrt.CrtSetReportMode(m, msvcrt.CRTDBG_MODE_FILE) - msvcrt.CrtSetReportFile(m, msvcrt.CRTDBG_FILE_STDERR) - else: - msvcrt.CrtSetReportMode(m, 0) + support.junit_xml_list = None - support.use_resources = ns.use_resources + if runtests.memory_limit is not None: + support.set_memlimit(runtests.memory_limit) + support.suppress_msvcrt_asserts(runtests.verbose >= 2) -def replace_stdout(): - """Set stdout encoder error handler to backslashreplace (as stderr error - handler) to avoid UnicodeEncodeError when printing a traceback""" - stdout = sys.stdout - try: - fd = stdout.fileno() - except ValueError: - # On IDLE, sys.stdout has no file descriptor and is not a TextIOWrapper - # object. Leaving sys.stdout unchanged. - # - # Catch ValueError to catch io.UnsupportedOperation on TextIOBase - # and ValueError on a closed stream. - return - - sys.stdout = open(fd, 'w', - encoding=stdout.encoding, - errors="backslashreplace", - closefd=False, - newline='\n') - - def restore_stdout(): - sys.stdout.close() - sys.stdout = stdout - atexit.register(restore_stdout) + support.use_resources = runtests.use_resources + + timeout = runtests.timeout + if timeout is not None: + # For a slow buildbot worker, increase SHORT_TIMEOUT and LONG_TIMEOUT + support.LOOPBACK_TIMEOUT = max(support.LOOPBACK_TIMEOUT, timeout / 120) + # don't increase INTERNET_TIMEOUT + support.SHORT_TIMEOUT = max(support.SHORT_TIMEOUT, timeout / 40) + support.LONG_TIMEOUT = max(support.LONG_TIMEOUT, timeout / 4) + + # If --timeout is short: reduce timeouts + support.LOOPBACK_TIMEOUT = min(support.LOOPBACK_TIMEOUT, timeout) + support.INTERNET_TIMEOUT = min(support.INTERNET_TIMEOUT, timeout) + support.SHORT_TIMEOUT = min(support.SHORT_TIMEOUT, timeout) + support.LONG_TIMEOUT = min(support.LONG_TIMEOUT, timeout) + + if runtests.hunt_refleak: + # private attribute that mypy doesn't know about: + unittest.BaseTestSuite._cleanup = False # type: ignore[attr-defined] + + if runtests.gc_threshold is not None: + gc.set_threshold(runtests.gc_threshold) + + random.seed(runtests.random_seed) + + # sys.stdout is redirected to a StringIO in single process mode on which + # color auto-detect fails as StringIO is not a TTY. If the original + # sys.stdout supports color pass that through with FORCE_COLOR so that when + # results are printed, such as with -W, they get color. + if can_colorize(file=sys.stdout): + os.environ['FORCE_COLOR'] = "1" diff --git a/Lib/test/libregrtest/single.py b/Lib/test/libregrtest/single.py new file mode 100644 index 00000000000..3dfb0b01dc1 --- /dev/null +++ b/Lib/test/libregrtest/single.py @@ -0,0 +1,361 @@ +import faulthandler +import gc +import importlib +import io +import sys +import time +import traceback +import unittest + +from _colorize import get_colors # type: ignore[import-not-found] +from test import support +from test.support import threading_helper + +from .filter import match_test +from .result import State, TestResult, TestStats +from .runtests import RunTests +from .save_env import saved_test_environment +from .setup import setup_tests +from .testresult import get_test_runner +from .parallel_case import ParallelTestCase +from .utils import ( + TestName, + clear_caches, remove_testfn, abs_module_name, print_warning) + + +# Minimum duration of a test to display its duration or to mention that +# the test is running in background +PROGRESS_MIN_TIME = 30.0 # seconds + + +def run_unittest(test_mod, runtests: RunTests): + loader = unittest.TestLoader() + tests = loader.loadTestsFromModule(test_mod) + + for error in loader.errors: + print(error, file=sys.stderr) + if loader.errors: + raise Exception("errors while loading tests") + _filter_suite(tests, match_test) + if runtests.parallel_threads: + _parallelize_tests(tests, runtests.parallel_threads) + return _run_suite(tests) + +def _filter_suite(suite, pred): + """Recursively filter test cases in a suite based on a predicate.""" + newtests = [] + for test in suite._tests: + if isinstance(test, unittest.TestSuite): + _filter_suite(test, pred) + newtests.append(test) + else: + if pred(test): + newtests.append(test) + suite._tests = newtests + +def _parallelize_tests(suite, parallel_threads: int): + def is_thread_unsafe(test): + test_method = getattr(test, test._testMethodName) + instance = test_method.__self__ + return (getattr(test_method, "__unittest_thread_unsafe__", False) or + getattr(instance, "__unittest_thread_unsafe__", False)) + + newtests: list[object] = [] + for test in suite._tests: + if isinstance(test, unittest.TestSuite): + _parallelize_tests(test, parallel_threads) + newtests.append(test) + continue + + if is_thread_unsafe(test): + # Don't parallelize thread-unsafe tests + newtests.append(test) + continue + + newtests.append(ParallelTestCase(test, parallel_threads)) + suite._tests = newtests + +def _run_suite(suite): + """Run tests from a unittest.TestSuite-derived class.""" + runner = get_test_runner(sys.stdout, + verbosity=support.verbose, + capture_output=(support.junit_xml_list is not None)) + + result = runner.run(suite) + + if support.junit_xml_list is not None: + import xml.etree.ElementTree as ET + xml_elem = result.get_xml_element() + xml_str = ET.tostring(xml_elem).decode('ascii') + support.junit_xml_list.append(xml_str) + + if not result.testsRun and not result.skipped and not result.errors: + raise support.TestDidNotRun + if not result.wasSuccessful(): + stats = TestStats.from_unittest(result) + if len(result.errors) == 1 and not result.failures: + err = result.errors[0][1] + elif len(result.failures) == 1 and not result.errors: + err = result.failures[0][1] + else: + err = "multiple errors occurred" + if not support.verbose: err += "; run in verbose mode for details" + errors = [(str(tc), exc_str) for tc, exc_str in result.errors] + failures = [(str(tc), exc_str) for tc, exc_str in result.failures] + raise support.TestFailedWithDetails(err, errors, failures, stats=stats) + return result + + +def regrtest_runner(result: TestResult, test_func, runtests: RunTests) -> None: + # Run test_func(), collect statistics, and detect reference and memory + # leaks. + if runtests.hunt_refleak: + from .refleak import runtest_refleak + refleak, test_result = runtest_refleak(result.test_name, test_func, + runtests.hunt_refleak, + runtests.quiet) + else: + test_result = test_func() + refleak = False + + if refleak: + result.state = State.REFLEAK + + stats: TestStats | None + + match test_result: + case TestStats(): + stats = test_result + case unittest.TestResult(): + stats = TestStats.from_unittest(test_result) + case None: + print_warning(f"{result.test_name} test runner returned None: {test_func}") + stats = None + case _: + # Don't import doctest at top level since only few tests return + # a doctest.TestResult instance. + import doctest + if isinstance(test_result, doctest.TestResults): + stats = TestStats.from_doctest(test_result) + else: + print_warning(f"Unknown test result type: {type(test_result)}") + stats = None + + result.stats = stats + + +# Storage of uncollectable GC objects (gc.garbage) +GC_GARBAGE = [] + + +def _load_run_test(result: TestResult, runtests: RunTests) -> None: + # Load the test module and run the tests. + test_name = result.test_name + module_name = abs_module_name(test_name, runtests.test_dir) + test_mod = importlib.import_module(module_name) + + if hasattr(test_mod, "test_main"): + # https://github.com/python/cpython/issues/89392 + raise Exception(f"Module {test_name} defines test_main() which " + f"is no longer supported by regrtest") + def test_func(): + return run_unittest(test_mod, runtests) + + try: + regrtest_runner(result, test_func, runtests) + finally: + # First kill any dangling references to open files etc. + # This can also issue some ResourceWarnings which would otherwise get + # triggered during the following test run, and possibly produce + # failures. + support.gc_collect() + + remove_testfn(test_name, runtests.verbose) + + # XXX: RUSTPYTHON, build a functional garbage collector into the interpreter + # if gc.garbage: + # support.environment_altered = True + # print_warning(f"{test_name} created {len(gc.garbage)} " + # f"uncollectable object(s)") + + # # move the uncollectable objects somewhere, + # # so we don't see them again + # GC_GARBAGE.extend(gc.garbage) + # gc.garbage.clear() + + support.reap_children() + + +def _runtest_env_changed_exc(result: TestResult, runtests: RunTests, + display_failure: bool = True) -> None: + # Handle exceptions, detect environment changes. + stdout = get_colors(file=sys.stdout) + stderr = get_colors(file=sys.stderr) + + # Reset the environment_altered flag to detect if a test altered + # the environment + support.environment_altered = False + + pgo = runtests.pgo + if pgo: + display_failure = False + quiet = runtests.quiet + + test_name = result.test_name + try: + clear_caches() + support.gc_collect() + + with saved_test_environment(test_name, + runtests.verbose, quiet, pgo=pgo): + _load_run_test(result, runtests) + except support.ResourceDenied as exc: + if not quiet and not pgo: + print( + f"{stdout.YELLOW}{test_name} skipped -- {exc}{stdout.RESET}", + flush=True, + ) + result.state = State.RESOURCE_DENIED + return + except unittest.SkipTest as exc: + if not quiet and not pgo: + print( + f"{stdout.YELLOW}{test_name} skipped -- {exc}{stdout.RESET}", + flush=True, + ) + result.state = State.SKIPPED + return + except support.TestFailedWithDetails as exc: + msg = f"{stderr.RED}test {test_name} failed{stderr.RESET}" + if display_failure: + msg = f"{stderr.RED}{msg} -- {exc}{stderr.RESET}" + print(msg, file=sys.stderr, flush=True) + result.state = State.FAILED + result.errors = exc.errors + result.failures = exc.failures + result.stats = exc.stats + return + except support.TestFailed as exc: + msg = f"{stderr.RED}test {test_name} failed{stderr.RESET}" + if display_failure: + msg = f"{stderr.RED}{msg} -- {exc}{stderr.RESET}" + print(msg, file=sys.stderr, flush=True) + result.state = State.FAILED + result.stats = exc.stats + return + except support.TestDidNotRun: + result.state = State.DID_NOT_RUN + return + except KeyboardInterrupt: + print() + result.state = State.INTERRUPTED + return + except: + if not pgo: + msg = traceback.format_exc() + print( + f"{stderr.RED}test {test_name} crashed -- {msg}{stderr.RESET}", + file=sys.stderr, + flush=True, + ) + result.state = State.UNCAUGHT_EXC + return + + if support.environment_altered: + result.set_env_changed() + # Don't override the state if it was already set (REFLEAK or ENV_CHANGED) + if result.state is None: + result.state = State.PASSED + + +def _runtest(result: TestResult, runtests: RunTests) -> None: + # Capture stdout and stderr, set faulthandler timeout, + # and create JUnit XML report. + verbose = runtests.verbose + output_on_failure = runtests.output_on_failure + timeout = runtests.timeout + + if timeout is not None and threading_helper.can_start_thread: + use_timeout = True + faulthandler.dump_traceback_later(timeout, exit=True) + else: + use_timeout = False + + try: + setup_tests(runtests) + + if output_on_failure or runtests.pgo: + support.verbose = True + + stream = io.StringIO() + orig_stdout = sys.stdout + orig_stderr = sys.stderr + print_warning = support.print_warning + orig_print_warnings_stderr = print_warning.orig_stderr + + output = None + try: + sys.stdout = stream + sys.stderr = stream + # print_warning() writes into the temporary stream to preserve + # messages order. If support.environment_altered becomes true, + # warnings will be written to sys.stderr below. + print_warning.orig_stderr = stream + + _runtest_env_changed_exc(result, runtests, display_failure=False) + # Ignore output if the test passed successfully + if result.state != State.PASSED: + output = stream.getvalue() + finally: + sys.stdout = orig_stdout + sys.stderr = orig_stderr + print_warning.orig_stderr = orig_print_warnings_stderr + + if output is not None: + sys.stderr.write(output) + sys.stderr.flush() + else: + # Tell tests to be moderately quiet + support.verbose = verbose + _runtest_env_changed_exc(result, runtests, + display_failure=not verbose) + + xml_list = support.junit_xml_list + if xml_list: + result.xml_data = xml_list + finally: + if use_timeout: + faulthandler.cancel_dump_traceback_later() + support.junit_xml_list = None + + +def run_single_test(test_name: TestName, runtests: RunTests) -> TestResult: + """Run a single test. + + test_name -- the name of the test + + Returns a TestResult. + + If runtests.use_junit, xml_data is a list containing each generated + testsuite element. + """ + ansi = get_colors(file=sys.stderr) + red, reset, yellow = ansi.BOLD_RED, ansi.RESET, ansi.YELLOW + + start_time = time.perf_counter() + result = TestResult(test_name) + pgo = runtests.pgo + try: + _runtest(result, runtests) + except: + if not pgo: + msg = traceback.format_exc() + print(f"{red}test {test_name} crashed -- {msg}{reset}", + file=sys.stderr, flush=True) + result.state = State.UNCAUGHT_EXC + + sys.stdout.flush() + sys.stderr.flush() + + result.duration = time.perf_counter() - start_time + return result diff --git a/Lib/test/support/testresult.py b/Lib/test/libregrtest/testresult.py similarity index 94% rename from Lib/test/support/testresult.py rename to Lib/test/libregrtest/testresult.py index de23fdd59de..1820f354572 100644 --- a/Lib/test/support/testresult.py +++ b/Lib/test/libregrtest/testresult.py @@ -9,6 +9,7 @@ import traceback import unittest from test import support +from test.libregrtest.utils import sanitize_xml class RegressionTestResult(unittest.TextTestResult): USE_XML = False @@ -65,23 +66,24 @@ def _add_result(self, test, capture=False, **args): if capture: if self._stdout_buffer is not None: stdout = self._stdout_buffer.getvalue().rstrip() - ET.SubElement(e, 'system-out').text = stdout + ET.SubElement(e, 'system-out').text = sanitize_xml(stdout) if self._stderr_buffer is not None: stderr = self._stderr_buffer.getvalue().rstrip() - ET.SubElement(e, 'system-err').text = stderr + ET.SubElement(e, 'system-err').text = sanitize_xml(stderr) for k, v in args.items(): if not k or not v: continue + e2 = ET.SubElement(e, k) if hasattr(v, 'items'): for k2, v2 in v.items(): if k2: - e2.set(k2, str(v2)) + e2.set(k2, sanitize_xml(str(v2))) else: - e2.text = str(v2) + e2.text = sanitize_xml(str(v2)) else: - e2.text = str(v) + e2.text = sanitize_xml(str(v)) @classmethod def __makeErrorDict(cls, err_type, err_value, err_tb): diff --git a/Lib/test/libregrtest/tsan.py b/Lib/test/libregrtest/tsan.py new file mode 100644 index 00000000000..d984a735bdf --- /dev/null +++ b/Lib/test/libregrtest/tsan.py @@ -0,0 +1,51 @@ +# Set of tests run by default if --tsan is specified. The tests below were +# chosen because they use threads and run in a reasonable amount of time. + +TSAN_TESTS = [ + 'test_asyncio', + # TODO: enable more of test_capi once bugs are fixed (GH-116908, GH-116909). + 'test_capi.test_mem', + 'test_capi.test_pyatomic', + 'test_code', + 'test_ctypes', + # 'test_concurrent_futures', # gh-130605: too many data races + 'test_enum', + 'test_functools', + 'test_httpservers', + 'test_imaplib', + 'test_importlib', + 'test_io', + 'test_logging', + 'test_opcache', + 'test_queue', + 'test_signal', + 'test_socket', + 'test_sqlite3', + 'test_ssl', + 'test_syslog', + 'test_thread', + 'test_thread_local_bytecode', + 'test_threadedtempfile', + 'test_threading', + 'test_threading_local', + 'test_threadsignals', + 'test_weakref', + 'test_free_threading', +] + +# Tests that should be run with `--parallel-threads=N` under TSAN. These tests +# typically do not use threads, but are run multiple times in parallel by +# the regression test runner with the `--parallel-threads` option enabled. +TSAN_PARALLEL_TESTS = [ + 'test_abc', + 'test_hashlib', +] + + +def setup_tsan_tests(cmdline_args) -> None: + if not cmdline_args: + cmdline_args[:] = TSAN_TESTS[:] + +def setup_tsan_parallel_tests(cmdline_args) -> None: + if not cmdline_args: + cmdline_args[:] = TSAN_PARALLEL_TESTS[:] diff --git a/Lib/test/libregrtest/utils.py b/Lib/test/libregrtest/utils.py index fb9971a64f6..d94fb84a743 100644 --- a/Lib/test/libregrtest/utils.py +++ b/Lib/test/libregrtest/utils.py @@ -1,10 +1,63 @@ +import contextlib +import faulthandler +import locale import math import os.path +import platform +import random +import re +import shlex +import subprocess import sys +import sysconfig +import tempfile import textwrap +from collections.abc import Callable, Iterable +from test import support +from test.support import os_helper +from test.support import threading_helper -def format_duration(seconds): + +# All temporary files and temporary directories created by libregrtest should +# use TMP_PREFIX so cleanup_temp_dir() can remove them all. +TMP_PREFIX = 'test_python_' +WORK_DIR_PREFIX = TMP_PREFIX +WORKER_WORK_DIR_PREFIX = WORK_DIR_PREFIX + 'worker_' + +# bpo-38203: Maximum delay in seconds to exit Python (call Py_Finalize()). +# Used to protect against threading._shutdown() hang. +# Must be smaller than buildbot "1200 seconds without output" limit. +EXIT_TIMEOUT = 120.0 + + +ALL_RESOURCES = ('audio', 'console', 'curses', 'largefile', 'network', + 'decimal', 'cpu', 'subprocess', 'urlfetch', 'gui', 'walltime') + +# Other resources excluded from --use=all: +# +# - extralagefile (ex: test_zipfile64): really too slow to be enabled +# "by default" +# - tzdata: while needed to validate fully test_datetime, it makes +# test_datetime too slow (15-20 min on some buildbots) and so is disabled by +# default (see bpo-30822). +RESOURCE_NAMES = ALL_RESOURCES + ('extralargefile', 'tzdata') + + +# Types for types hints +StrPath = str +TestName = str +StrJSON = str +TestTuple = tuple[TestName, ...] +TestList = list[TestName] +# --match and --ignore options: list of patterns +# ('*' joker character can be used) +TestFilter = list[tuple[TestName, bool]] +FilterTuple = tuple[TestName, ...] +FilterDict = dict[TestName, FilterTuple] + + +def format_duration(seconds: float) -> str: ms = math.ceil(seconds * 1e3) seconds, ms = divmod(ms, 1000) minutes, seconds = divmod(seconds, 60) @@ -16,17 +69,20 @@ def format_duration(seconds): if minutes: parts.append('%s min' % minutes) if seconds: - parts.append('%s sec' % seconds) - if ms: - parts.append('%s ms' % ms) + if parts: + # 2 min 1 sec + parts.append('%s sec' % seconds) + else: + # 1.0 sec + parts.append('%.1f sec' % (seconds + ms / 1000)) if not parts: - return '0 ms' + return '%s ms' % ms parts = parts[:2] return ' '.join(parts) -def removepy(names): +def strip_py_suffix(names: list[str] | None) -> None: if not names: return for idx, name in enumerate(names): @@ -35,11 +91,20 @@ def removepy(names): names[idx] = basename -def count(n, word): +def plural(n: int, singular: str, plural: str | None = None) -> str: if n == 1: - return "%d %s" % (n, word) + return singular + elif plural is not None: + return plural else: - return "%d %ss" % (n, word) + return singular + 's' + + +def count(n: int, word: str) -> str: + if n == 1: + return f"{n} {word}" + else: + return f"{n} {word}s" def printlist(x, width=70, indent=4, file=None): @@ -57,5 +122,605 @@ def printlist(x, width=70, indent=4, file=None): file=file) -def print_warning(msg): - print(f"Warning -- {msg}", file=sys.stderr, flush=True) +def print_warning(msg: str) -> None: + support.print_warning(msg) + + +orig_unraisablehook: Callable[..., None] | None = None + + +def regrtest_unraisable_hook(unraisable) -> None: + global orig_unraisablehook + support.environment_altered = True + support.print_warning("Unraisable exception") + old_stderr = sys.stderr + try: + support.flush_std_streams() + sys.stderr = support.print_warning.orig_stderr + assert orig_unraisablehook is not None, "orig_unraisablehook not set" + orig_unraisablehook(unraisable) + sys.stderr.flush() + finally: + sys.stderr = old_stderr + + +def setup_unraisable_hook() -> None: + global orig_unraisablehook + orig_unraisablehook = sys.unraisablehook + sys.unraisablehook = regrtest_unraisable_hook + + +orig_threading_excepthook: Callable[..., None] | None = None + + +def regrtest_threading_excepthook(args) -> None: + global orig_threading_excepthook + support.environment_altered = True + support.print_warning(f"Uncaught thread exception: {args.exc_type.__name__}") + old_stderr = sys.stderr + try: + support.flush_std_streams() + sys.stderr = support.print_warning.orig_stderr + assert orig_threading_excepthook is not None, "orig_threading_excepthook not set" + orig_threading_excepthook(args) + sys.stderr.flush() + finally: + sys.stderr = old_stderr + + +def setup_threading_excepthook() -> None: + global orig_threading_excepthook + import threading + orig_threading_excepthook = threading.excepthook + threading.excepthook = regrtest_threading_excepthook + + +def clear_caches(): + # Clear the warnings registry, so they can be displayed again + for mod in sys.modules.values(): + if hasattr(mod, '__warningregistry__'): + del mod.__warningregistry__ + + # Flush standard output, so that buffered data is sent to the OS and + # associated Python objects are reclaimed. + for stream in (sys.stdout, sys.stderr, sys.__stdout__, sys.__stderr__): + if stream is not None: + stream.flush() + + try: + re = sys.modules['re'] + except KeyError: + pass + else: + re.purge() + + try: + _strptime = sys.modules['_strptime'] + except KeyError: + pass + else: + _strptime._regex_cache.clear() + + try: + urllib_parse = sys.modules['urllib.parse'] + except KeyError: + pass + else: + urllib_parse.clear_cache() + + try: + urllib_request = sys.modules['urllib.request'] + except KeyError: + pass + else: + urllib_request.urlcleanup() + + try: + linecache = sys.modules['linecache'] + except KeyError: + pass + else: + linecache.clearcache() + + try: + mimetypes = sys.modules['mimetypes'] + except KeyError: + pass + else: + mimetypes._default_mime_types() + + try: + filecmp = sys.modules['filecmp'] + except KeyError: + pass + else: + filecmp._cache.clear() + + try: + struct = sys.modules['struct'] + except KeyError: + pass + else: + struct._clearcache() + + try: + doctest = sys.modules['doctest'] + except KeyError: + pass + else: + doctest.master = None + + try: + ctypes = sys.modules['ctypes'] + except KeyError: + pass + else: + ctypes._reset_cache() + + try: + typing = sys.modules['typing'] + except KeyError: + pass + else: + for f in typing._cleanups: + f() + + import inspect + abs_classes = filter(inspect.isabstract, typing.__dict__.values()) + for abc in abs_classes: + for obj in abc.__subclasses__() + [abc]: + obj._abc_caches_clear() + + try: + fractions = sys.modules['fractions'] + except KeyError: + pass + else: + fractions._hash_algorithm.cache_clear() + + try: + inspect = sys.modules['inspect'] + except KeyError: + pass + else: + inspect._shadowed_dict_from_weakref_mro_tuple.cache_clear() + inspect._filesbymodname.clear() + inspect.modulesbyfile.clear() + + try: + importlib_metadata = sys.modules['importlib.metadata'] + except KeyError: + pass + else: + importlib_metadata.FastPath.__new__.cache_clear() + + +def get_build_info(): + # Get most important configure and build options as a list of strings. + # Example: ['debug', 'ASAN+MSAN'] or ['release', 'LTO+PGO']. + + config_args = sysconfig.get_config_var('CONFIG_ARGS') or '' + cflags = sysconfig.get_config_var('PY_CFLAGS') or '' + cflags += ' ' + (sysconfig.get_config_var('PY_CFLAGS_NODIST') or '') + ldflags_nodist = sysconfig.get_config_var('PY_LDFLAGS_NODIST') or '' + + build = [] + + # --disable-gil + if sysconfig.get_config_var('Py_GIL_DISABLED'): + if not sys.flags.ignore_environment: + PYTHON_GIL = os.environ.get('PYTHON_GIL', None) + if PYTHON_GIL: + PYTHON_GIL = (PYTHON_GIL == '1') + else: + PYTHON_GIL = None + + free_threading = "free_threading" + if PYTHON_GIL is not None: + free_threading = f"{free_threading} GIL={int(PYTHON_GIL)}" + build.append(free_threading) + + if hasattr(sys, 'gettotalrefcount'): + # --with-pydebug + build.append('debug') + + if '-DNDEBUG' in cflags: + build.append('without_assert') + else: + build.append('release') + + if '--with-assertions' in config_args: + build.append('with_assert') + elif '-DNDEBUG' not in cflags: + build.append('with_assert') + + # --enable-experimental-jit + if sys._jit.is_available(): + if sys._jit.is_enabled(): + build.append("JIT") + else: + build.append("JIT (disabled)") + + # --enable-framework=name + framework = sysconfig.get_config_var('PYTHONFRAMEWORK') + if framework: + build.append(f'framework={framework}') + + # --enable-shared + shared = int(sysconfig.get_config_var('PY_ENABLE_SHARED') or '0') + if shared: + build.append('shared') + + # --with-lto + optimizations = [] + if '-flto=thin' in ldflags_nodist: + optimizations.append('ThinLTO') + elif '-flto' in ldflags_nodist: + optimizations.append('LTO') + + if support.check_cflags_pgo(): + # PGO (--enable-optimizations) + optimizations.append('PGO') + + if support.check_bolt_optimized(): + # BOLT (--enable-bolt) + optimizations.append('BOLT') + + if optimizations: + build.append('+'.join(optimizations)) + + # --with-address-sanitizer + sanitizers = [] + if support.check_sanitizer(address=True): + sanitizers.append("ASAN") + # --with-memory-sanitizer + if support.check_sanitizer(memory=True): + sanitizers.append("MSAN") + # --with-undefined-behavior-sanitizer + if support.check_sanitizer(ub=True): + sanitizers.append("UBSAN") + # --with-thread-sanitizer + if support.check_sanitizer(thread=True): + sanitizers.append("TSAN") + if sanitizers: + build.append('+'.join(sanitizers)) + + # --with-trace-refs + if hasattr(sys, 'getobjects'): + build.append("TraceRefs") + # --enable-pystats + if hasattr(sys, '_stats_on'): + build.append("pystats") + # --with-valgrind + if sysconfig.get_config_var('WITH_VALGRIND'): + build.append("valgrind") + # --with-dtrace + if sysconfig.get_config_var('WITH_DTRACE'): + build.append("dtrace") + + return build + + +def get_temp_dir(tmp_dir: StrPath | None = None) -> StrPath: + if tmp_dir: + tmp_dir = os.path.expanduser(tmp_dir) + else: + # When tests are run from the Python build directory, it is best practice + # to keep the test files in a subfolder. This eases the cleanup of leftover + # files using the "make distclean" command. + if sysconfig.is_python_build(): + if not support.is_wasi: + tmp_dir = sysconfig.get_config_var('abs_builddir') + if tmp_dir is None: + tmp_dir = sysconfig.get_config_var('abs_srcdir') + if not tmp_dir: + # gh-74470: On Windows, only srcdir is available. Using + # abs_builddir mostly matters on UNIX when building + # Python out of the source tree, especially when the + # source tree is read only. + tmp_dir = sysconfig.get_config_var('srcdir') + if not tmp_dir: + raise RuntimeError( + "Could not determine the correct value for tmp_dir" + ) + tmp_dir = os.path.join(tmp_dir, 'build') + else: + # WASI platform + tmp_dir = sysconfig.get_config_var('projectbase') + if not tmp_dir: + raise RuntimeError( + "sysconfig.get_config_var('projectbase') " + f"unexpectedly returned {tmp_dir!r} on WASI" + ) + tmp_dir = os.path.join(tmp_dir, 'build') + + # When get_temp_dir() is called in a worker process, + # get_temp_dir() path is different than in the parent process + # which is not a WASI process. So the parent does not create + # the same "tmp_dir" than the test worker process. + os.makedirs(tmp_dir, exist_ok=True) + else: + tmp_dir = tempfile.gettempdir() + + return os.path.abspath(tmp_dir) + + +def get_work_dir(parent_dir: StrPath, worker: bool = False) -> StrPath: + # Define a writable temp dir that will be used as cwd while running + # the tests. The name of the dir includes the pid to allow parallel + # testing (see the -j option). + # Emscripten and WASI have stubbed getpid(), Emscripten has only + # millisecond clock resolution. Use randint() instead. + if support.is_emscripten or support.is_wasi: + nounce = random.randint(0, 1_000_000) + else: + nounce = os.getpid() + + if worker: + work_dir = WORK_DIR_PREFIX + str(nounce) + else: + work_dir = WORKER_WORK_DIR_PREFIX + str(nounce) + work_dir += os_helper.FS_NONASCII + work_dir = os.path.join(parent_dir, work_dir) + return work_dir + + +@contextlib.contextmanager +def exit_timeout(): + try: + yield + except SystemExit as exc: + # bpo-38203: Python can hang at exit in Py_Finalize(), especially + # on threading._shutdown() call: put a timeout + if threading_helper.can_start_thread: + faulthandler.dump_traceback_later(EXIT_TIMEOUT, exit=True) + sys.exit(exc.code) + + +def remove_testfn(test_name: TestName, verbose: int) -> None: + # Try to clean up os_helper.TESTFN if left behind. + # + # While tests shouldn't leave any files or directories behind, when a test + # fails that can be tedious for it to arrange. The consequences can be + # especially nasty on Windows, since if a test leaves a file open, it + # cannot be deleted by name (while there's nothing we can do about that + # here either, we can display the name of the offending test, which is a + # real help). + name = os_helper.TESTFN + if not os.path.exists(name): + return + + nuker: Callable[[str], None] + if os.path.isdir(name): + import shutil + kind, nuker = "directory", shutil.rmtree + elif os.path.isfile(name): + kind, nuker = "file", os.unlink + else: + raise RuntimeError(f"os.path says {name!r} exists but is neither " + f"directory nor file") + + if verbose: + print_warning(f"{test_name} left behind {kind} {name!r}") + support.environment_altered = True + + try: + import stat + # fix possible permissions problems that might prevent cleanup + os.chmod(name, stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO) + nuker(name) + except Exception as exc: + print_warning(f"{test_name} left behind {kind} {name!r} " + f"and it couldn't be removed: {exc}") + + +def abs_module_name(test_name: TestName, test_dir: StrPath | None) -> TestName: + if test_name.startswith('test.') or test_dir: + return test_name + else: + # Import it from the test package + return 'test.' + test_name + + +# gh-90681: When rerunning tests, we might need to rerun the whole +# class or module suite if some its life-cycle hooks fail. +# Test level hooks are not affected. +_TEST_LIFECYCLE_HOOKS = frozenset(( + 'setUpClass', 'tearDownClass', + 'setUpModule', 'tearDownModule', +)) + +def normalize_test_name(test_full_name: str, *, + is_error: bool = False) -> str | None: + short_name = test_full_name.split(" ")[0] + if is_error and short_name in _TEST_LIFECYCLE_HOOKS: + if test_full_name.startswith(('setUpModule (', 'tearDownModule (')): + # if setUpModule() or tearDownModule() failed, don't filter + # tests with the test file name, don't use filters. + return None + + # This means that we have a failure in a life-cycle hook, + # we need to rerun the whole module or class suite. + # Basically the error looks like this: + # ERROR: setUpClass (test.test_reg_ex.RegTest) + # or + # ERROR: setUpModule (test.test_reg_ex) + # So, we need to parse the class / module name. + lpar = test_full_name.index('(') + rpar = test_full_name.index(')') + return test_full_name[lpar + 1: rpar].split('.')[-1] + return short_name + + +def adjust_rlimit_nofile() -> None: + """ + On macOS the default fd limit (RLIMIT_NOFILE) is sometimes too low (256) + for our test suite to succeed. Raise it to something more reasonable. 1024 + is a common Linux default. + """ + try: + import resource + except ImportError: + return + + fd_limit, max_fds = resource.getrlimit(resource.RLIMIT_NOFILE) + + desired_fds = 1024 + + if fd_limit < desired_fds and fd_limit < max_fds: + new_fd_limit = min(desired_fds, max_fds) + try: + resource.setrlimit(resource.RLIMIT_NOFILE, + (new_fd_limit, max_fds)) + print(f"Raised RLIMIT_NOFILE: {fd_limit} -> {new_fd_limit}") + except (ValueError, OSError) as err: + print_warning(f"Unable to raise RLIMIT_NOFILE from {fd_limit} to " + f"{new_fd_limit}: {err}.") + + +def get_host_runner() -> str: + if (hostrunner := os.environ.get("_PYTHON_HOSTRUNNER")) is None: + hostrunner = sysconfig.get_config_var("HOSTRUNNER") + return hostrunner + + +def is_cross_compiled() -> bool: + return ('_PYTHON_HOST_PLATFORM' in os.environ) + + +def format_resources(use_resources: Iterable[str]) -> str: + use_resources = set(use_resources) + all_resources = set(ALL_RESOURCES) + + # Express resources relative to "all" + relative_all = ['all'] + for name in sorted(all_resources - use_resources): + relative_all.append(f'-{name}') + for name in sorted(use_resources - all_resources): + relative_all.append(f'{name}') + all_text = ','.join(relative_all) + all_text = f"resources: {all_text}" + + # List of enabled resources + text = ','.join(sorted(use_resources)) + text = f"resources ({len(use_resources)}): {text}" + + # Pick the shortest string (prefer relative to all if lengths are equal) + if len(all_text) <= len(text): + return all_text + else: + return text + + +def display_header(use_resources: tuple[str, ...], + python_cmd: tuple[str, ...] | None) -> None: + # Print basic platform information + print("==", platform.python_implementation(), *sys.version.split()) + print("==", platform.platform(aliased=True), + "%s-endian" % sys.byteorder) + print("== Python build:", ' '.join(get_build_info())) + print("== cwd:", os.getcwd()) + + cpu_count: object = os.cpu_count() + if cpu_count: + # The function is new in Python 3.13; mypy doesn't know about it yet: + process_cpu_count = os.process_cpu_count() # type: ignore[attr-defined] + if process_cpu_count and process_cpu_count != cpu_count: + cpu_count = f"{process_cpu_count} (process) / {cpu_count} (system)" + print("== CPU count:", cpu_count) + print("== encodings: locale=%s FS=%s" + % (locale.getencoding(), sys.getfilesystemencoding())) + + if use_resources: + text = format_resources(use_resources) + print(f"== {text}") + else: + print("== resources: all test resources are disabled, " + "use -u option to unskip tests") + + cross_compile = is_cross_compiled() + if cross_compile: + print("== cross compiled: Yes") + if python_cmd: + cmd = shlex.join(python_cmd) + print(f"== host python: {cmd}") + + get_cmd = [*python_cmd, '-m', 'platform'] + proc = subprocess.run( + get_cmd, + stdout=subprocess.PIPE, + text=True, + cwd=os_helper.SAVEDCWD) + stdout = proc.stdout.replace('\n', ' ').strip() + if stdout: + print(f"== host platform: {stdout}") + elif proc.returncode: + print(f"== host platform: ") + else: + hostrunner = get_host_runner() + if hostrunner: + print(f"== host runner: {hostrunner}") + + # This makes it easier to remember what to set in your local + # environment when trying to reproduce a sanitizer failure. + asan = support.check_sanitizer(address=True) + msan = support.check_sanitizer(memory=True) + ubsan = support.check_sanitizer(ub=True) + tsan = support.check_sanitizer(thread=True) + sanitizers = [] + if asan: + sanitizers.append("address") + if msan: + sanitizers.append("memory") + if ubsan: + sanitizers.append("undefined behavior") + if tsan: + sanitizers.append("thread") + if sanitizers: + print(f"== sanitizers: {', '.join(sanitizers)}") + for sanitizer, env_var in ( + (asan, "ASAN_OPTIONS"), + (msan, "MSAN_OPTIONS"), + (ubsan, "UBSAN_OPTIONS"), + (tsan, "TSAN_OPTIONS"), + ): + options= os.environ.get(env_var) + if sanitizer and options is not None: + print(f"== {env_var}={options!r}") + + print(flush=True) + + +def cleanup_temp_dir(tmp_dir: StrPath) -> None: + import glob + + path = os.path.join(glob.escape(tmp_dir), TMP_PREFIX + '*') + print("Cleanup %s directory" % tmp_dir) + for name in glob.glob(path): + if os.path.isdir(name): + print("Remove directory: %s" % name) + os_helper.rmtree(name) + else: + print("Remove file: %s" % name) + os_helper.unlink(name) + + +ILLEGAL_XML_CHARS_RE = re.compile( + '[' + # Control characters; newline (\x0A and \x0D) and TAB (\x09) are legal + '\x00-\x08\x0B\x0C\x0E-\x1F' + # Surrogate characters + '\uD800-\uDFFF' + # Special Unicode characters + '\uFFFE' + '\uFFFF' + # Match multiple sequential invalid characters for better efficiency + ']+') + +def _sanitize_xml_replace(regs): + text = regs[0] + return ''.join(f'\\x{ord(ch):02x}' if ch <= '\xff' else ascii(ch)[1:-1] + for ch in text) + +def sanitize_xml(text: str) -> str: + return ILLEGAL_XML_CHARS_RE.sub(_sanitize_xml_replace, text) diff --git a/Lib/test/libregrtest/win_utils.py b/Lib/test/libregrtest/win_utils.py index 95db3def36f..b51fde0af57 100644 --- a/Lib/test/libregrtest/win_utils.py +++ b/Lib/test/libregrtest/win_utils.py @@ -1,105 +1,128 @@ +import _overlapped +import _thread import _winapi -import msvcrt -import os -import subprocess -import uuid -from test import support +import math +import struct +import winreg -# Max size of asynchronous reads -BUFSIZE = 8192 -# Exponential damping factor (see below) -LOAD_FACTOR_1 = 0.9200444146293232478931553241 # Seconds per measurement -SAMPLING_INTERVAL = 5 -COUNTER_NAME = r'\System\Processor Queue Length' +SAMPLING_INTERVAL = 1 +# Exponential damping factor to compute exponentially weighted moving average +# on 1 minute (60 seconds) +LOAD_FACTOR_1 = 1 / math.exp(SAMPLING_INTERVAL / 60) +# Initialize the load using the arithmetic mean of the first NVALUE values +# of the Processor Queue Length +NVALUE = 5 class WindowsLoadTracker(): """ - This class asynchronously interacts with the `typeperf` command to read - the system load on Windows. Mulitprocessing and threads can't be used - here because they interfere with the test suite's cases for those - modules. + This class asynchronously reads the performance counters to calculate + the system load on Windows. A "raw" thread is used here to prevent + interference with the test suite's cases for the threading module. """ def __init__(self): - self.load = 0.0 - self.start() - - def start(self): - # Create a named pipe which allows for asynchronous IO in Windows - pipe_name = r'\\.\pipe\typeperf_output_' + str(uuid.uuid4()) - - open_mode = _winapi.PIPE_ACCESS_INBOUND - open_mode |= _winapi.FILE_FLAG_FIRST_PIPE_INSTANCE - open_mode |= _winapi.FILE_FLAG_OVERLAPPED - - # This is the read end of the pipe, where we will be grabbing output - self.pipe = _winapi.CreateNamedPipe( - pipe_name, open_mode, _winapi.PIPE_WAIT, - 1, BUFSIZE, BUFSIZE, _winapi.NMPWAIT_WAIT_FOREVER, _winapi.NULL - ) - # The write end of the pipe which is passed to the created process - pipe_write_end = _winapi.CreateFile( - pipe_name, _winapi.GENERIC_WRITE, 0, _winapi.NULL, - _winapi.OPEN_EXISTING, 0, _winapi.NULL - ) - # Open up the handle as a python file object so we can pass it to - # subprocess - command_stdout = msvcrt.open_osfhandle(pipe_write_end, 0) - - # Connect to the read end of the pipe in overlap/async mode - overlap = _winapi.ConnectNamedPipe(self.pipe, overlapped=True) - overlap.GetOverlappedResult(True) - - # Spawn off the load monitor - command = ['typeperf', COUNTER_NAME, '-si', str(SAMPLING_INTERVAL)] - self.p = subprocess.Popen(command, stdout=command_stdout, cwd=os_helper.SAVEDCWD) - - # Close our copy of the write end of the pipe - os.close(command_stdout) - - def close(self): - if self.p is None: + # make __del__ not fail if pre-flight test fails + self._running = None + self._stopped = None + + # Pre-flight test for access to the performance data; + # `PermissionError` will be raised if not allowed + winreg.QueryInfoKey(winreg.HKEY_PERFORMANCE_DATA) + + self._values = [] + self._load = None + self._running = _overlapped.CreateEvent(None, True, False, None) + self._stopped = _overlapped.CreateEvent(None, True, False, None) + + _thread.start_new_thread(self._update_load, (), {}) + + def _update_load(self, + # localize module access to prevent shutdown errors + _wait=_winapi.WaitForSingleObject, + _signal=_overlapped.SetEvent): + # run until signaled to stop + while _wait(self._running, 1000): + self._calculate_load() + # notify stopped + _signal(self._stopped) + + def _calculate_load(self, + # localize module access to prevent shutdown errors + _query=winreg.QueryValueEx, + _hkey=winreg.HKEY_PERFORMANCE_DATA, + _unpack=struct.unpack_from): + # get the 'System' object + data, _ = _query(_hkey, '2') + # PERF_DATA_BLOCK { + # WCHAR Signature[4] 8 + + # DWOWD LittleEndian 4 + + # DWORD Version 4 + + # DWORD Revision 4 + + # DWORD TotalByteLength 4 + + # DWORD HeaderLength = 24 byte offset + # ... + # } + obj_start, = _unpack('L', data, 24) + # PERF_OBJECT_TYPE { + # DWORD TotalByteLength + # DWORD DefinitionLength + # DWORD HeaderLength + # ... + # } + data_start, defn_start = _unpack('4xLL', data, obj_start) + data_base = obj_start + data_start + defn_base = obj_start + defn_start + # find the 'Processor Queue Length' counter (index=44) + while defn_base < data_base: + # PERF_COUNTER_DEFINITION { + # DWORD ByteLength + # DWORD CounterNameTitleIndex + # ... [7 DWORDs/28 bytes] + # DWORD CounterOffset + # } + size, idx, offset = _unpack('LL28xL', data, defn_base) + defn_base += size + if idx == 44: + counter_offset = data_base + offset + # the counter is known to be PERF_COUNTER_RAWCOUNT (DWORD) + processor_queue_length, = _unpack('L', data, counter_offset) + break + else: return - self.p.kill() - self.p.wait() - self.p = None - def __del__(self): - self.close() - - def read_output(self): - import _winapi - - overlapped, _ = _winapi.ReadFile(self.pipe, BUFSIZE, True) - bytes_read, res = overlapped.GetOverlappedResult(False) - if res != 0: - return - - return overlapped.getbuffer().decode() + # We use an exponentially weighted moving average, imitating the + # load calculation on Unix systems. + # https://en.wikipedia.org/wiki/Load_(computing)#Unix-style_load_calculation + # https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average + if self._load is not None: + self._load = (self._load * LOAD_FACTOR_1 + + processor_queue_length * (1.0 - LOAD_FACTOR_1)) + elif len(self._values) < NVALUE: + self._values.append(processor_queue_length) + else: + self._load = sum(self._values) / len(self._values) + + def close(self, kill=True): + self.__del__() + return + + def __del__(self, + # localize module access to prevent shutdown errors + _wait=_winapi.WaitForSingleObject, + _close=_winapi.CloseHandle, + _signal=_overlapped.SetEvent): + if self._running is not None: + # tell the update thread to quit + _signal(self._running) + # wait for the update thread to signal done + _wait(self._stopped, -1) + # cleanup events + _close(self._running) + _close(self._stopped) + self._running = self._stopped = None def getloadavg(self): - typeperf_output = self.read_output() - # Nothing to update, just return the current load - if not typeperf_output: - return self.load - - # Process the backlog of load values - for line in typeperf_output.splitlines(): - # typeperf outputs in a CSV format like this: - # "07/19/2018 01:32:26.605","3.000000" - toks = line.split(',') - # Ignore blank lines and the initial header - if line.strip() == '' or (COUNTER_NAME in line) or len(toks) != 2: - continue - - load = float(toks[1].replace('"', '')) - # We use an exponentially weighted moving average, imitating the - # load calculation on Unix systems. - # https://en.wikipedia.org/wiki/Load_(computing)#Unix-style_load_calculation - new_load = self.load * LOAD_FACTOR_1 + load * (1.0 - LOAD_FACTOR_1) - self.load = new_load - - return self.load + return self._load diff --git a/Lib/test/libregrtest/worker.py b/Lib/test/libregrtest/worker.py new file mode 100644 index 00000000000..1ad67e1cebf --- /dev/null +++ b/Lib/test/libregrtest/worker.py @@ -0,0 +1,138 @@ +import subprocess +import sys +import os +from _colorize import can_colorize # type: ignore[import-not-found] +from typing import Any, NoReturn + +from test.support import os_helper, Py_DEBUG + +from .setup import setup_process, setup_test_dir +from .runtests import WorkerRunTests, JsonFile, JsonFileType +from .single import run_single_test +from .utils import ( + StrPath, StrJSON, TestFilter, + get_temp_dir, get_work_dir, exit_timeout) + + +USE_PROCESS_GROUP = (hasattr(os, "setsid") and hasattr(os, "killpg")) +NEED_TTY = { + 'test_ioctl', +} + + +def create_worker_process(runtests: WorkerRunTests, output_fd: int, + tmp_dir: StrPath | None = None) -> subprocess.Popen[str]: + worker_json = runtests.as_json() + + cmd = runtests.create_python_cmd() + cmd.extend(['-m', 'test.libregrtest.worker', worker_json]) + + env = dict(os.environ) + if tmp_dir is not None: + env['TMPDIR'] = tmp_dir + env['TEMP'] = tmp_dir + env['TMP'] = tmp_dir + + # The subcommand is run with a temporary output which means it is not a TTY + # and won't auto-color. The test results are printed to stdout so if we can + # color that have the subprocess use color. + if can_colorize(file=sys.stdout): + env['FORCE_COLOR'] = '1' + + # Running the child from the same working directory as regrtest's original + # invocation ensures that TEMPDIR for the child is the same when + # sysconfig.is_python_build() is true. See issue 15300. + # + # Emscripten and WASI Python must start in the Python source code directory + # to get 'python.js' or 'python.wasm' file. Then worker_process() changes + # to a temporary directory created to run tests. + work_dir = os_helper.SAVEDCWD + + kwargs: dict[str, Any] = dict( + env=env, + stdout=output_fd, + # bpo-45410: Write stderr into stdout to keep messages order + stderr=output_fd, + text=True, + close_fds=True, + cwd=work_dir, + ) + + # Don't use setsid() in tests using TTY + test_name = runtests.tests[0] + if USE_PROCESS_GROUP and test_name not in NEED_TTY: + kwargs['start_new_session'] = True + + # Include the test name in the TSAN log file name + if 'TSAN_OPTIONS' in env: + parts = env['TSAN_OPTIONS'].split(' ') + for i, part in enumerate(parts): + if part.startswith('log_path='): + parts[i] = f'{part}.{test_name}' + break + env['TSAN_OPTIONS'] = ' '.join(parts) + + # Pass json_file to the worker process + json_file = runtests.json_file + json_file.configure_subprocess(kwargs) + + with json_file.inherit_subprocess(): + return subprocess.Popen(cmd, **kwargs) + + +def worker_process(worker_json: StrJSON) -> NoReturn: + runtests = WorkerRunTests.from_json(worker_json) + test_name = runtests.tests[0] + match_tests: TestFilter = runtests.match_tests + json_file: JsonFile = runtests.json_file + + setup_test_dir(runtests.test_dir) + setup_process() + + if runtests.rerun: + if match_tests: + matching = "matching: " + ", ".join(pattern for pattern, result in match_tests if result) + print(f"Re-running {test_name} in verbose mode ({matching})", flush=True) + else: + print(f"Re-running {test_name} in verbose mode", flush=True) + + result = run_single_test(test_name, runtests) + if runtests.coverage: + if "test.cov" in sys.modules: # imported by -Xpresite= + result.covered_lines = list(sys.modules["test.cov"].coverage) + elif not Py_DEBUG: + print( + "Gathering coverage in worker processes requires --with-pydebug", + flush=True, + ) + else: + raise LookupError( + "`test.cov` not found in sys.modules but coverage wanted" + ) + + if json_file.file_type == JsonFileType.STDOUT: + print() + result.write_json_into(sys.stdout) + else: + with json_file.open('w', encoding='utf-8') as json_fp: + result.write_json_into(json_fp) + + sys.exit(0) + + +def main() -> NoReturn: + if len(sys.argv) != 2: + print("usage: python -m test.libregrtest.worker JSON") + sys.exit(1) + worker_json = sys.argv[1] + + tmp_dir = get_temp_dir() + work_dir = get_work_dir(tmp_dir, worker=True) + + with exit_timeout(): + with os_helper.temp_cwd(work_dir, quiet=True): + worker_process(worker_json) + + +if __name__ == "__main__": + main() diff --git a/Lib/test/lock_tests.py b/Lib/test/lock_tests.py index 09b91147801..4031bfaeb64 100644 --- a/Lib/test/lock_tests.py +++ b/Lib/test/lock_tests.py @@ -2,6 +2,7 @@ Various tests for synchronization primitives. """ +import gc import sys import time from _thread import start_new_thread, TIMEOUT_MAX @@ -13,54 +14,79 @@ from test.support import threading_helper -def _wait(): - # A crude wait/yield function not relying on synchronization primitives. - time.sleep(0.01) +requires_fork = unittest.skipUnless(support.has_fork_support, + "platform doesn't support fork " + "(no _at_fork_reinit method)") + + +def wait_threads_blocked(nthread): + # Arbitrary sleep to wait until N threads are blocked, + # like waiting for a lock. + time.sleep(0.010 * nthread) + class Bunch(object): """ A bunch of threads. """ - def __init__(self, f, n, wait_before_exit=False): + def __init__(self, func, nthread, wait_before_exit=False): """ - Construct a bunch of `n` threads running the same function `f`. + Construct a bunch of `nthread` threads running the same function `func`. If `wait_before_exit` is True, the threads won't terminate until do_finish() is called. """ - self.f = f - self.n = n + self.func = func + self.nthread = nthread self.started = [] self.finished = [] + self.exceptions = [] self._can_exit = not wait_before_exit - self.wait_thread = threading_helper.wait_threads_exit() - self.wait_thread.__enter__() + self._wait_thread = None - def task(): - tid = threading.get_ident() - self.started.append(tid) - try: - f() - finally: - self.finished.append(tid) - while not self._can_exit: - _wait() + def task(self): + tid = threading.get_ident() + self.started.append(tid) + try: + self.func() + except BaseException as exc: + self.exceptions.append(exc) + finally: + self.finished.append(tid) + for _ in support.sleeping_retry(support.SHORT_TIMEOUT): + if self._can_exit: + break + + def __enter__(self): + self._wait_thread = threading_helper.wait_threads_exit(support.SHORT_TIMEOUT) + self._wait_thread.__enter__() try: - for i in range(n): - start_new_thread(task, ()) + for _ in range(self.nthread): + start_new_thread(self.task, ()) except: self._can_exit = True raise - def wait_for_started(self): - while len(self.started) < self.n: - _wait() + for _ in support.sleeping_retry(support.SHORT_TIMEOUT): + if len(self.started) >= self.nthread: + break + + return self + + def __exit__(self, exc_type, exc_value, traceback): + for _ in support.sleeping_retry(support.SHORT_TIMEOUT): + if len(self.finished) >= self.nthread: + break + + # Wait until threads completely exit according to _thread._count() + self._wait_thread.__exit__(None, None, None) - def wait_for_finished(self): - while len(self.finished) < self.n: - _wait() - # Wait for threads exit - self.wait_thread.__exit__(None, None, None) + # Break reference cycle + exceptions = self.exceptions + self.exceptions = None + if exceptions: + raise ExceptionGroup(f"{self.func} threads raised exceptions", + exceptions) def do_finish(self): self._can_exit = True @@ -88,6 +114,12 @@ class BaseLockTests(BaseTestCase): Tests for both recursive and non-recursive locks. """ + def wait_phase(self, phase, expected): + for _ in support.sleeping_retry(support.SHORT_TIMEOUT): + if len(phase) >= expected: + break + self.assertEqual(len(phase), expected) + def test_constructor(self): lock = self.locktype() del lock @@ -125,44 +157,60 @@ def test_try_acquire_contended(self): result = [] def f(): result.append(lock.acquire(False)) - Bunch(f, 1).wait_for_finished() + with Bunch(f, 1): + pass self.assertFalse(result[0]) lock.release() - @unittest.skip("TODO: RUSTPYTHON, sometimes hangs") + @unittest.skip('TODO: RUSTPYTHON; sometimes hangs') def test_acquire_contended(self): lock = self.locktype() lock.acquire() - N = 5 def f(): lock.acquire() lock.release() - b = Bunch(f, N) - b.wait_for_started() - _wait() - self.assertEqual(len(b.finished), 0) - lock.release() - b.wait_for_finished() - self.assertEqual(len(b.finished), N) + N = 5 + with Bunch(f, N) as bunch: + # Threads block on lock.acquire() + wait_threads_blocked(N) + self.assertEqual(len(bunch.finished), 0) + + # Threads unblocked + lock.release() + + self.assertEqual(len(bunch.finished), N) def test_with(self): lock = self.locktype() def f(): lock.acquire() lock.release() - def _with(err=None): + + def with_lock(err=None): with lock: if err is not None: raise err - _with() - # Check the lock is unacquired - Bunch(f, 1).wait_for_finished() - self.assertRaises(TypeError, _with, TypeError) - # Check the lock is unacquired - Bunch(f, 1).wait_for_finished() - @unittest.skip("TODO: RUSTPYTHON, sometimes hangs") + # Acquire the lock, do nothing, with releases the lock + with lock: + pass + + # Check that the lock is unacquired + with Bunch(f, 1): + pass + + # Acquire the lock, raise an exception, with releases the lock + with self.assertRaises(TypeError): + with lock: + raise TypeError + + # Check that the lock is unacquired even if after an exception + # was raised in the previous "with lock:" block + with Bunch(f, 1): + pass + + @unittest.skip('TODO: RUSTPYTHON; sometimes hangs') def test_thread_leak(self): # The lock shouldn't leak a Thread instance when used from a foreign # (non-threading) thread. @@ -170,22 +218,16 @@ def test_thread_leak(self): def f(): lock.acquire() lock.release() - n = len(threading.enumerate()) + # We run many threads in the hope that existing threads ids won't # be recycled. - Bunch(f, 15).wait_for_finished() - if len(threading.enumerate()) != n: - # There is a small window during which a Thread instance's - # target function has finished running, but the Thread is still - # alive and registered. Avoid spurious failures by waiting a - # bit more (seen on a buildbot). - time.sleep(0.4) - self.assertEqual(n, len(threading.enumerate())) + with Bunch(f, 15): + pass def test_timeout(self): lock = self.locktype() # Can't set timeout if not blocking - self.assertRaises(ValueError, lock.acquire, 0, 1) + self.assertRaises(ValueError, lock.acquire, False, 1) # Invalid timeout values self.assertRaises(ValueError, lock.acquire, timeout=-100) self.assertRaises(OverflowError, lock.acquire, timeout=1e100) @@ -204,7 +246,8 @@ def f(): results.append(lock.acquire(timeout=0.5)) t2 = time.monotonic() results.append(t2 - t1) - Bunch(f, 1).wait_for_finished() + with Bunch(f, 1): + pass self.assertFalse(results[0]) self.assertTimeout(results[1], 0.5) @@ -217,6 +260,7 @@ def test_weakref_deleted(self): lock = self.locktype() ref = weakref.ref(lock) del lock + gc.collect() # For PyPy or other GCs. self.assertIsNone(ref()) @@ -237,15 +281,13 @@ def f(): phase.append(None) with threading_helper.wait_threads_exit(): + # Thread blocked on lock.acquire() start_new_thread(f, ()) - while len(phase) == 0: - _wait() - _wait() - self.assertEqual(len(phase), 1) + self.wait_phase(phase, 1) + + # Thread unblocked lock.release() - while len(phase) == 1: - _wait() - self.assertEqual(len(phase), 2) + self.wait_phase(phase, 2) def test_different_thread(self): # Lock can be released from a different thread. @@ -253,8 +295,8 @@ def test_different_thread(self): lock.acquire() def f(): lock.release() - b = Bunch(f, 1) - b.wait_for_finished() + with Bunch(f, 1): + pass lock.acquire() lock.release() @@ -268,6 +310,25 @@ def test_state_after_timeout(self): self.assertFalse(lock.locked()) self.assertTrue(lock.acquire(blocking=False)) + @requires_fork + def test_at_fork_reinit(self): + def use_lock(lock): + # make sure that the lock still works normally + # after _at_fork_reinit() + lock.acquire() + lock.release() + + # unlocked + lock = self.locktype() + lock._at_fork_reinit() + use_lock(lock) + + # locked: _at_fork_reinit() resets the lock to the unlocked state + lock2 = self.locktype() + lock2.acquire() + lock2._at_fork_reinit() + use_lock(lock2) + class RLockTests(BaseLockTests): """ @@ -306,17 +367,52 @@ def test_release_save_unacquired(self): lock.release() self.assertRaises(RuntimeError, lock._release_save) + def test_recursion_count(self): + lock = self.locktype() + self.assertEqual(0, lock._recursion_count()) + lock.acquire() + self.assertEqual(1, lock._recursion_count()) + lock.acquire() + lock.acquire() + self.assertEqual(3, lock._recursion_count()) + lock.release() + self.assertEqual(2, lock._recursion_count()) + lock.release() + lock.release() + self.assertEqual(0, lock._recursion_count()) + + phase = [] + + def f(): + lock.acquire() + phase.append(None) + + self.wait_phase(phase, 2) + lock.release() + phase.append(None) + + with threading_helper.wait_threads_exit(): + # Thread blocked on lock.acquire() + start_new_thread(f, ()) + self.wait_phase(phase, 1) + self.assertEqual(0, lock._recursion_count()) + + # Thread unblocked + phase.append(None) + self.wait_phase(phase, 3) + self.assertEqual(0, lock._recursion_count()) + def test_different_thread(self): # Cannot release from a different thread lock = self.locktype() def f(): lock.acquire() - b = Bunch(f, 1, True) - try: - self.assertRaises(RuntimeError, lock.release) - finally: - b.do_finish() - b.wait_for_finished() + + with Bunch(f, 1, True) as bunch: + try: + self.assertRaises(RuntimeError, lock.release) + finally: + bunch.do_finish() def test__is_owned(self): lock = self.locktype() @@ -328,7 +424,8 @@ def test__is_owned(self): result = [] def f(): result.append(lock._is_owned()) - Bunch(f, 1).wait_for_finished() + with Bunch(f, 1): + pass self.assertFalse(result[0]) lock.release() self.assertTrue(lock._is_owned()) @@ -361,12 +458,15 @@ def _check_notify(self, evt): def f(): results1.append(evt.wait()) results2.append(evt.wait()) - b = Bunch(f, N) - b.wait_for_started() - _wait() - self.assertEqual(len(results1), 0) - evt.set() - b.wait_for_finished() + + with Bunch(f, N): + # Threads blocked on first evt.wait() + wait_threads_blocked(N) + self.assertEqual(len(results1), 0) + + # Threads unblocked + evt.set() + self.assertEqual(results1, [True] * N) self.assertEqual(results2, [True] * N) @@ -389,46 +489,61 @@ def f(): r = evt.wait(0.5) t2 = time.monotonic() results2.append((r, t2 - t1)) - Bunch(f, N).wait_for_finished() + + with Bunch(f, N): + pass + self.assertEqual(results1, [False] * N) for r, dt in results2: self.assertFalse(r) self.assertTimeout(dt, 0.5) + # The event is set results1 = [] results2 = [] evt.set() - Bunch(f, N).wait_for_finished() + with Bunch(f, N): + pass + self.assertEqual(results1, [True] * N) for r, dt in results2: self.assertTrue(r) def test_set_and_clear(self): - # Issue #13502: check that wait() returns true even when the event is + # gh-57711: check that wait() returns true even when the event is # cleared before the waiting thread is woken up. - evt = self.eventtype() + event = self.eventtype() results = [] - timeout = 0.250 - N = 5 def f(): - results.append(evt.wait(timeout * 4)) - b = Bunch(f, N) - b.wait_for_started() - time.sleep(timeout) - evt.set() - evt.clear() - b.wait_for_finished() + results.append(event.wait(support.LONG_TIMEOUT)) + + N = 5 + with Bunch(f, N): + # Threads blocked on event.wait() + wait_threads_blocked(N) + + # Threads unblocked + event.set() + event.clear() + self.assertEqual(results, [True] * N) - def test_reset_internal_locks(self): + @requires_fork + def test_at_fork_reinit(self): # ensure that condition is still using a Lock after reset evt = self.eventtype() with evt._cond: self.assertFalse(evt._cond.acquire(False)) - evt._reset_internal_locks() + evt._at_fork_reinit() with evt._cond: self.assertFalse(evt._cond.acquire(False)) + def test_repr(self): + evt = self.eventtype() + self.assertRegex(repr(evt), r"<\w+\.Event at .*: unset>") + evt.set() + self.assertRegex(repr(evt), r"<\w+\.Event at .*: set>") + class ConditionTests(BaseTestCase): """ @@ -466,15 +581,14 @@ def _check_notify(self, cond): # Note that this test is sensitive to timing. If the worker threads # don't execute in a timely fashion, the main thread may think they # are further along then they are. The main thread therefore issues - # _wait() statements to try to make sure that it doesn't race ahead - # of the workers. + # wait_threads_blocked() statements to try to make sure that it doesn't + # race ahead of the workers. # Secondly, this test assumes that condition variables are not subject # to spurious wakeups. The absence of spurious wakeups is an implementation # detail of Condition Variables in current CPython, but in general, not # a guaranteed property of condition variables as a programming # construct. In particular, it is possible that this can no longer # be conveniently guaranteed should their implementation ever change. - N = 5 ready = [] results1 = [] results2 = [] @@ -483,58 +597,83 @@ def f(): cond.acquire() ready.append(phase_num) result = cond.wait() + cond.release() results1.append((result, phase_num)) + cond.acquire() ready.append(phase_num) + result = cond.wait() cond.release() results2.append((result, phase_num)) - b = Bunch(f, N) - b.wait_for_started() - # first wait, to ensure all workers settle into cond.wait() before - # we continue. See issues #8799 and #30727. - while len(ready) < 5: - _wait() - ready.clear() - self.assertEqual(results1, []) - # Notify 3 threads at first - cond.acquire() - cond.notify(3) - _wait() - phase_num = 1 - cond.release() - while len(results1) < 3: - _wait() - self.assertEqual(results1, [(True, 1)] * 3) - self.assertEqual(results2, []) - # make sure all awaken workers settle into cond.wait() - while len(ready) < 3: - _wait() - # Notify 5 threads: they might be in their first or second wait - cond.acquire() - cond.notify(5) - _wait() - phase_num = 2 - cond.release() - while len(results1) + len(results2) < 8: - _wait() - self.assertEqual(results1, [(True, 1)] * 3 + [(True, 2)] * 2) - self.assertEqual(results2, [(True, 2)] * 3) - # make sure all workers settle into cond.wait() - while len(ready) < 5: - _wait() - # Notify all threads: they are all in their second wait - cond.acquire() - cond.notify_all() - _wait() - phase_num = 3 - cond.release() - while len(results2) < 5: - _wait() - self.assertEqual(results1, [(True, 1)] * 3 + [(True,2)] * 2) - self.assertEqual(results2, [(True, 2)] * 3 + [(True, 3)] * 2) - b.wait_for_finished() + + N = 5 + with Bunch(f, N): + # first wait, to ensure all workers settle into cond.wait() before + # we continue. See issues #8799 and #30727. + for _ in support.sleeping_retry(support.SHORT_TIMEOUT): + if len(ready) >= N: + break + + ready.clear() + self.assertEqual(results1, []) + + # Notify 3 threads at first + count1 = 3 + cond.acquire() + cond.notify(count1) + wait_threads_blocked(count1) + + # Phase 1 + phase_num = 1 + cond.release() + for _ in support.sleeping_retry(support.SHORT_TIMEOUT): + if len(results1) >= count1: + break + + self.assertEqual(results1, [(True, 1)] * count1) + self.assertEqual(results2, []) + + # Wait until awaken workers are blocked on cond.wait() + for _ in support.sleeping_retry(support.SHORT_TIMEOUT): + if len(ready) >= count1 : + break + + # Notify 5 threads: they might be in their first or second wait + cond.acquire() + cond.notify(5) + wait_threads_blocked(N) + + # Phase 2 + phase_num = 2 + cond.release() + for _ in support.sleeping_retry(support.SHORT_TIMEOUT): + if len(results1) + len(results2) >= (N + count1): + break + + count2 = N - count1 + self.assertEqual(results1, [(True, 1)] * count1 + [(True, 2)] * count2) + self.assertEqual(results2, [(True, 2)] * count1) + + # Make sure all workers settle into cond.wait() + for _ in support.sleeping_retry(support.SHORT_TIMEOUT): + if len(ready) >= N: + break + + # Notify all threads: they are all in their second wait + cond.acquire() + cond.notify_all() + wait_threads_blocked(N) + + # Phase 3 + phase_num = 3 + cond.release() + for _ in support.sleeping_retry(support.SHORT_TIMEOUT): + if len(results2) >= N: + break + self.assertEqual(results1, [(True, 1)] * count1 + [(True, 2)] * count2) + self.assertEqual(results2, [(True, 2)] * count1 + [(True, 3)] * count2) def test_notify(self): cond = self.condtype() @@ -544,19 +683,23 @@ def test_notify(self): def test_timeout(self): cond = self.condtype() + timeout = 0.5 results = [] - N = 5 def f(): cond.acquire() t1 = time.monotonic() - result = cond.wait(0.5) + result = cond.wait(timeout) t2 = time.monotonic() cond.release() results.append((t2 - t1, result)) - Bunch(f, N).wait_for_finished() + + N = 5 + with Bunch(f, N): + pass self.assertEqual(len(results), N) + for dt, result in results: - self.assertTimeout(dt, 0.5) + self.assertTimeout(dt, timeout) # Note that conceptually (that"s the condition variable protocol) # a wait() may succeed even if no one notifies us and before any # timeout occurs. Spurious wakeups can occur. @@ -569,17 +712,16 @@ def test_waitfor(self): state = 0 def f(): with cond: - result = cond.wait_for(lambda : state==4) + result = cond.wait_for(lambda: state == 4) self.assertTrue(result) self.assertEqual(state, 4) - b = Bunch(f, 1) - b.wait_for_started() - for i in range(4): - time.sleep(0.01) - with cond: - state += 1 - cond.notify() - b.wait_for_finished() + + with Bunch(f, 1): + for i in range(4): + time.sleep(0.010) + with cond: + state += 1 + cond.notify() def test_waitfor_timeout(self): cond = self.condtype() @@ -593,15 +735,15 @@ def f(): self.assertFalse(result) self.assertTimeout(dt, 0.1) success.append(None) - b = Bunch(f, 1) - b.wait_for_started() - # Only increment 3 times, so state == 4 is never reached. - for i in range(3): - time.sleep(0.01) - with cond: - state += 1 - cond.notify() - b.wait_for_finished() + + with Bunch(f, 1): + # Only increment 3 times, so state == 4 is never reached. + for i in range(3): + time.sleep(0.010) + with cond: + state += 1 + cond.notify() + self.assertEqual(len(success), 1) @@ -630,41 +772,107 @@ def test_acquire_destroy(self): del sem def test_acquire_contended(self): - sem = self.semtype(7) + sem_value = 7 + sem = self.semtype(sem_value) sem.acquire() - N = 10 + sem_results = [] results1 = [] results2 = [] phase_num = 0 - def f(): + + def func(): sem_results.append(sem.acquire()) results1.append(phase_num) + sem_results.append(sem.acquire()) results2.append(phase_num) - b = Bunch(f, 10) - b.wait_for_started() - while len(results1) + len(results2) < 6: - _wait() - self.assertEqual(results1 + results2, [0] * 6) - phase_num = 1 - for i in range(7): + + def wait_count(count): + for _ in support.sleeping_retry(support.SHORT_TIMEOUT): + if len(results1) + len(results2) >= count: + break + + N = 10 + with Bunch(func, N): + # Phase 0 + count1 = sem_value - 1 + wait_count(count1) + self.assertEqual(results1 + results2, [0] * count1) + + # Phase 1 + phase_num = 1 + for i in range(sem_value): + sem.release() + count2 = sem_value + wait_count(count1 + count2) + self.assertEqual(sorted(results1 + results2), + [0] * count1 + [1] * count2) + + # Phase 2 + phase_num = 2 + count3 = (sem_value - 1) + for i in range(count3): + sem.release() + wait_count(count1 + count2 + count3) + self.assertEqual(sorted(results1 + results2), + [0] * count1 + [1] * count2 + [2] * count3) + # The semaphore is still locked + self.assertFalse(sem.acquire(False)) + + # Final release, to let the last thread finish + count4 = 1 sem.release() - while len(results1) + len(results2) < 13: - _wait() - self.assertEqual(sorted(results1 + results2), [0] * 6 + [1] * 7) - phase_num = 2 - for i in range(6): + + self.assertEqual(sem_results, + [True] * (count1 + count2 + count3 + count4)) + + def test_multirelease(self): + sem_value = 7 + sem = self.semtype(sem_value) + sem.acquire() + + results1 = [] + results2 = [] + phase_num = 0 + def func(): + sem.acquire() + results1.append(phase_num) + + sem.acquire() + results2.append(phase_num) + + def wait_count(count): + for _ in support.sleeping_retry(support.SHORT_TIMEOUT): + if len(results1) + len(results2) >= count: + break + + with Bunch(func, 10): + # Phase 0 + count1 = sem_value - 1 + wait_count(count1) + self.assertEqual(results1 + results2, [0] * count1) + + # Phase 1 + phase_num = 1 + count2 = sem_value + sem.release(count2) + wait_count(count1 + count2) + self.assertEqual(sorted(results1 + results2), + [0] * count1 + [1] * count2) + + # Phase 2 + phase_num = 2 + count3 = sem_value - 1 + sem.release(count3) + wait_count(count1 + count2 + count3) + self.assertEqual(sorted(results1 + results2), + [0] * count1 + [1] * count2 + [2] * count3) + # The semaphore is still locked + self.assertFalse(sem.acquire(False)) + + # Final release, to let the last thread finish sem.release() - while len(results1) + len(results2) < 19: - _wait() - self.assertEqual(sorted(results1 + results2), [0] * 6 + [1] * 7 + [2] * 6) - # The semaphore is still locked - self.assertFalse(sem.acquire(False)) - # Final release, to let the last thread finish - sem.release() - b.wait_for_finished() - self.assertEqual(sem_results, [True] * (6 + 7 + 6 + 1)) def test_try_acquire(self): sem = self.semtype(2) @@ -681,7 +889,8 @@ def test_try_acquire_contended(self): def f(): results.append(sem.acquire(False)) results.append(sem.acquire(False)) - Bunch(f, 5).wait_for_finished() + with Bunch(f, 5): + pass # There can be a thread switch between acquiring the semaphore and # appending the result, therefore results will not necessarily be # ordered. @@ -707,12 +916,14 @@ def test_default_value(self): def f(): sem.acquire() sem.release() - b = Bunch(f, 1) - b.wait_for_started() - _wait() - self.assertFalse(b.finished) - sem.release() - b.wait_for_finished() + + with Bunch(f, 1) as bunch: + # Thread blocked on sem.acquire() + wait_threads_blocked(1) + self.assertFalse(bunch.finished) + + # Thread unblocked + sem.release() def test_with(self): sem = self.semtype(2) @@ -744,6 +955,15 @@ def test_release_unacquired(self): sem.acquire() sem.release() + def test_repr(self): + sem = self.semtype(3) + self.assertRegex(repr(sem), r"<\w+\.Semaphore at .*: value=3>") + sem.acquire() + self.assertRegex(repr(sem), r"<\w+\.Semaphore at .*: value=2>") + sem.release() + sem.release() + self.assertRegex(repr(sem), r"<\w+\.Semaphore at .*: value=4>") + class BoundedSemaphoreTests(BaseSemaphoreTests): """ @@ -758,6 +978,12 @@ def test_release_unacquired(self): sem.release() self.assertRaises(ValueError, sem.release) + def test_repr(self): + sem = self.semtype(3) + self.assertRegex(repr(sem), r"<\w+\.BoundedSemaphore at .*: value=3/3>") + sem.acquire() + self.assertRegex(repr(sem), r"<\w+\.BoundedSemaphore at .*: value=2/3>") + class BarrierTests(BaseTestCase): """ @@ -768,13 +994,13 @@ class BarrierTests(BaseTestCase): def setUp(self): self.barrier = self.barriertype(self.N, timeout=self.defaultTimeout) + def tearDown(self): self.barrier.abort() def run_threads(self, f): - b = Bunch(f, self.N-1) - f() - b.wait_for_finished() + with Bunch(f, self.N): + pass def multipass(self, results, n): m = self.barrier.parties @@ -789,6 +1015,10 @@ def multipass(self, results, n): self.assertEqual(self.barrier.n_waiting, 0) self.assertFalse(self.barrier.broken) + def test_constructor(self): + self.assertRaises(ValueError, self.barriertype, parties=0) + self.assertRaises(ValueError, self.barriertype, parties=-1) + def test_barrier(self, passes=1): """ Test that a barrier is passed in lockstep @@ -865,8 +1095,9 @@ def f(): i = self.barrier.wait() if i == self.N//2: # Wait until the other threads are all in the barrier. - while self.barrier.n_waiting < self.N-1: - time.sleep(0.001) + for _ in support.sleeping_retry(support.SHORT_TIMEOUT): + if self.barrier.n_waiting >= (self.N - 1): + break self.barrier.reset() else: try: @@ -926,27 +1157,56 @@ def f(): i = self.barrier.wait() if i == self.N // 2: # One thread is late! - time.sleep(1.0) + time.sleep(self.defaultTimeout / 2) # Default timeout is 2.0, so this is shorter. self.assertRaises(threading.BrokenBarrierError, - self.barrier.wait, 0.5) + self.barrier.wait, self.defaultTimeout / 4) self.run_threads(f) def test_default_timeout(self): """ Test the barrier's default timeout """ - # create a barrier with a low default timeout - barrier = self.barriertype(self.N, timeout=0.3) + timeout = 0.100 + barrier = self.barriertype(2, timeout=timeout) def f(): - i = barrier.wait() - if i == self.N // 2: - # One thread is later than the default timeout of 0.3s. - time.sleep(1.0) - self.assertRaises(threading.BrokenBarrierError, barrier.wait) - self.run_threads(f) + self.assertRaises(threading.BrokenBarrierError, + barrier.wait) + + start_time = time.monotonic() + with Bunch(f, 1): + pass + dt = time.monotonic() - start_time + self.assertGreaterEqual(dt, timeout) def test_single_thread(self): b = self.barriertype(1) b.wait() b.wait() + + def test_repr(self): + barrier = self.barriertype(3) + timeout = support.LONG_TIMEOUT + self.assertRegex(repr(barrier), r"<\w+\.Barrier at .*: waiters=0/3>") + def f(): + barrier.wait(timeout) + + N = 2 + with Bunch(f, N): + # Threads blocked on barrier.wait() + for _ in support.sleeping_retry(support.SHORT_TIMEOUT): + if barrier.n_waiting >= N: + break + self.assertRegex(repr(barrier), + r"<\w+\.Barrier at .*: waiters=2/3>") + + # Threads unblocked + barrier.wait(timeout) + + self.assertRegex(repr(barrier), + r"<\w+\.Barrier at .*: waiters=0/3>") + + # Abort the barrier + barrier.abort() + self.assertRegex(repr(barrier), + r"<\w+\.Barrier at .*: broken>") diff --git a/Lib/test/mathdata/cmath_testcases.txt b/Lib/test/mathdata/cmath_testcases.txt index 0165e17634f..7b98b5a2998 100644 --- a/Lib/test/mathdata/cmath_testcases.txt +++ b/Lib/test/mathdata/cmath_testcases.txt @@ -371,9 +371,9 @@ acosh1002 acosh 0.0 inf -> inf 1.5707963267948966 acosh1003 acosh 2.3 inf -> inf 1.5707963267948966 acosh1004 acosh -0.0 inf -> inf 1.5707963267948966 acosh1005 acosh -2.3 inf -> inf 1.5707963267948966 -acosh1006 acosh 0.0 nan -> nan nan +acosh1006 acosh 0.0 nan -> nan 1.5707963267948966 ignore-imag-sign acosh1007 acosh 2.3 nan -> nan nan -acosh1008 acosh -0.0 nan -> nan nan +acosh1008 acosh -0.0 nan -> nan 1.5707963267948966 ignore-imag-sign acosh1009 acosh -2.3 nan -> nan nan acosh1010 acosh -inf 0.0 -> inf 3.1415926535897931 acosh1011 acosh -inf 2.3 -> inf 3.1415926535897931 @@ -1992,9 +1992,9 @@ tanh0065 tanh 1.797e+308 0.0 -> 1.0 0.0 --special values tanh1000 tanh 0.0 0.0 -> 0.0 0.0 -tanh1001 tanh 0.0 inf -> nan nan invalid +tanh1001 tanh 0.0 inf -> 0.0 nan invalid tanh1002 tanh 2.3 inf -> nan nan invalid -tanh1003 tanh 0.0 nan -> nan nan +tanh1003 tanh 0.0 nan -> 0.0 nan tanh1004 tanh 2.3 nan -> nan nan tanh1005 tanh inf 0.0 -> 1.0 0.0 tanh1006 tanh inf 0.7 -> 1.0 0.0 @@ -2009,7 +2009,7 @@ tanh1014 tanh nan 2.3 -> nan nan tanh1015 tanh nan inf -> nan nan tanh1016 tanh nan nan -> nan nan tanh1017 tanh 0.0 -0.0 -> 0.0 -0.0 -tanh1018 tanh 0.0 -inf -> nan nan invalid +tanh1018 tanh 0.0 -inf -> 0.0 nan invalid tanh1019 tanh 2.3 -inf -> nan nan invalid tanh1020 tanh inf -0.0 -> 1.0 -0.0 tanh1021 tanh inf -0.7 -> 1.0 -0.0 @@ -2022,9 +2022,9 @@ tanh1027 tanh nan -0.0 -> nan -0.0 tanh1028 tanh nan -2.3 -> nan nan tanh1029 tanh nan -inf -> nan nan tanh1030 tanh -0.0 -0.0 -> -0.0 -0.0 -tanh1031 tanh -0.0 -inf -> nan nan invalid +tanh1031 tanh -0.0 -inf -> -0.0 nan invalid tanh1032 tanh -2.3 -inf -> nan nan invalid -tanh1033 tanh -0.0 nan -> nan nan +tanh1033 tanh -0.0 nan -> -0.0 nan tanh1034 tanh -2.3 nan -> nan nan tanh1035 tanh -inf -0.0 -> -1.0 -0.0 tanh1036 tanh -inf -0.7 -> -1.0 -0.0 @@ -2035,7 +2035,7 @@ tanh1040 tanh -inf -3.5 -> -1.0 -0.0 tanh1041 tanh -inf -inf -> -1.0 0.0 ignore-imag-sign tanh1042 tanh -inf nan -> -1.0 0.0 ignore-imag-sign tanh1043 tanh -0.0 0.0 -> -0.0 0.0 -tanh1044 tanh -0.0 inf -> nan nan invalid +tanh1044 tanh -0.0 inf -> -0.0 nan invalid tanh1045 tanh -2.3 inf -> nan nan invalid tanh1046 tanh -inf 0.0 -> -1.0 0.0 tanh1047 tanh -inf 0.7 -> -1.0 0.0 @@ -2307,9 +2307,9 @@ tan0066 tan -8.79645943005142 0.0 -> 0.7265425280053614098 0.0 -- special values tan1000 tan -0.0 0.0 -> -0.0 0.0 -tan1001 tan -inf 0.0 -> nan nan invalid +tan1001 tan -inf 0.0 -> nan 0.0 invalid tan1002 tan -inf 2.2999999999999998 -> nan nan invalid -tan1003 tan nan 0.0 -> nan nan +tan1003 tan nan 0.0 -> nan 0.0 tan1004 tan nan 2.2999999999999998 -> nan nan tan1005 tan -0.0 inf -> -0.0 1.0 tan1006 tan -0.69999999999999996 inf -> -0.0 1.0 @@ -2324,7 +2324,7 @@ tan1014 tan -2.2999999999999998 nan -> nan nan tan1015 tan -inf nan -> nan nan tan1016 tan nan nan -> nan nan tan1017 tan 0.0 0.0 -> 0.0 0.0 -tan1018 tan inf 0.0 -> nan nan invalid +tan1018 tan inf 0.0 -> nan 0.0 invalid tan1019 tan inf 2.2999999999999998 -> nan nan invalid tan1020 tan 0.0 inf -> 0.0 1.0 tan1021 tan 0.69999999999999996 inf -> 0.0 1.0 @@ -2337,9 +2337,9 @@ tan1027 tan 0.0 nan -> 0.0 nan tan1028 tan 2.2999999999999998 nan -> nan nan tan1029 tan inf nan -> nan nan tan1030 tan 0.0 -0.0 -> 0.0 -0.0 -tan1031 tan inf -0.0 -> nan nan invalid +tan1031 tan inf -0.0 -> nan -0.0 invalid tan1032 tan inf -2.2999999999999998 -> nan nan invalid -tan1033 tan nan -0.0 -> nan nan +tan1033 tan nan -0.0 -> nan -0.0 tan1034 tan nan -2.2999999999999998 -> nan nan tan1035 tan 0.0 -inf -> 0.0 -1.0 tan1036 tan 0.69999999999999996 -inf -> 0.0 -1.0 @@ -2350,7 +2350,7 @@ tan1040 tan 3.5 -inf -> 0.0 -1.0 tan1041 tan inf -inf -> -0.0 -1.0 ignore-real-sign tan1042 tan nan -inf -> -0.0 -1.0 ignore-real-sign tan1043 tan -0.0 -0.0 -> -0.0 -0.0 -tan1044 tan -inf -0.0 -> nan nan invalid +tan1044 tan -inf -0.0 -> nan -0.0 invalid tan1045 tan -inf -2.2999999999999998 -> nan nan invalid tan1046 tan -0.0 -inf -> -0.0 -1.0 tan1047 tan -0.69999999999999996 -inf -> -0.0 -1.0 diff --git a/Lib/test/mathdata/ieee754.txt b/Lib/test/mathdata/ieee754.txt index 3e986cdb102..9be667826a6 100644 --- a/Lib/test/mathdata/ieee754.txt +++ b/Lib/test/mathdata/ieee754.txt @@ -116,7 +116,7 @@ inf >>> 0 ** -1 Traceback (most recent call last): ... -ZeroDivisionError: 0.0 cannot be raised to a negative power +ZeroDivisionError: zero to a negative power >>> pow(0, NAN) nan @@ -127,31 +127,31 @@ Trigonometric Functions >>> sin(INF) Traceback (most recent call last): ... -ValueError: math domain error +ValueError: expected a finite input, got inf >>> sin(NINF) Traceback (most recent call last): ... -ValueError: math domain error +ValueError: expected a finite input, got -inf >>> sin(NAN) nan >>> cos(INF) Traceback (most recent call last): ... -ValueError: math domain error +ValueError: expected a finite input, got inf >>> cos(NINF) Traceback (most recent call last): ... -ValueError: math domain error +ValueError: expected a finite input, got -inf >>> cos(NAN) nan >>> tan(INF) Traceback (most recent call last): ... -ValueError: math domain error +ValueError: expected a finite input, got inf >>> tan(NINF) Traceback (most recent call last): ... -ValueError: math domain error +ValueError: expected a finite input, got -inf >>> tan(NAN) nan @@ -169,11 +169,11 @@ True >>> asin(INF), asin(NINF) Traceback (most recent call last): ... -ValueError: math domain error +ValueError: expected a number in range from -1 up to 1, got inf >>> acos(INF), acos(NINF) Traceback (most recent call last): ... -ValueError: math domain error +ValueError: expected a number in range from -1 up to 1, got inf >>> equal(atan(INF), PI/2), equal(atan(NINF), -PI/2) (True, True) diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py index 21b0edfd073..dd61b051354 100755 --- a/Lib/test/regrtest.py +++ b/Lib/test/regrtest.py @@ -6,13 +6,10 @@ Run this script with -h or --help for documentation. """ -# We import importlib *ASAP* in order to test #15386 -import importlib - import os import sys -from test.libregrtest import main +from test.libregrtest.main import main # Alias for backward compatibility (just in case) main_in_temp_cwd = main diff --git a/Lib/test/regrtestdata/import_from_tests/test_regrtest_a.py b/Lib/test/regrtestdata/import_from_tests/test_regrtest_a.py new file mode 100644 index 00000000000..9c3d0c7cf4b --- /dev/null +++ b/Lib/test/regrtestdata/import_from_tests/test_regrtest_a.py @@ -0,0 +1,11 @@ +import sys +import unittest +import test_regrtest_b.util + +class Test(unittest.TestCase): + def test(self): + test_regrtest_b.util # does not fail + self.assertIn('test_regrtest_a', sys.modules) + self.assertIs(sys.modules['test_regrtest_b'], test_regrtest_b) + self.assertIs(sys.modules['test_regrtest_b.util'], test_regrtest_b.util) + self.assertNotIn('test_regrtest_c', sys.modules) diff --git a/Lib/test/regrtestdata/import_from_tests/test_regrtest_b/__init__.py b/Lib/test/regrtestdata/import_from_tests/test_regrtest_b/__init__.py new file mode 100644 index 00000000000..3dfba253455 --- /dev/null +++ b/Lib/test/regrtestdata/import_from_tests/test_regrtest_b/__init__.py @@ -0,0 +1,9 @@ +import sys +import unittest + +class Test(unittest.TestCase): + def test(self): + self.assertNotIn('test_regrtest_a', sys.modules) + self.assertIn('test_regrtest_b', sys.modules) + self.assertNotIn('test_regrtest_b.util', sys.modules) + self.assertNotIn('test_regrtest_c', sys.modules) diff --git a/Lib/test/regrtestdata/import_from_tests/test_regrtest_b/util.py b/Lib/test/regrtestdata/import_from_tests/test_regrtest_b/util.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/Lib/test/regrtestdata/import_from_tests/test_regrtest_c.py b/Lib/test/regrtestdata/import_from_tests/test_regrtest_c.py new file mode 100644 index 00000000000..de80769118d --- /dev/null +++ b/Lib/test/regrtestdata/import_from_tests/test_regrtest_c.py @@ -0,0 +1,11 @@ +import sys +import unittest +import test_regrtest_b.util + +class Test(unittest.TestCase): + def test(self): + test_regrtest_b.util # does not fail + self.assertNotIn('test_regrtest_a', sys.modules) + self.assertIs(sys.modules['test_regrtest_b'], test_regrtest_b) + self.assertIs(sys.modules['test_regrtest_b.util'], test_regrtest_b.util) + self.assertIn('test_regrtest_c', sys.modules) diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index 444ca2219cf..cc5a48738fd 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -3,11 +3,11 @@ if __name__ != 'test.support': raise ImportError('support must be imported from the test package') +import _opcode import contextlib -import dataclasses import functools +import inspect import logging -import _opcode import os import re import stat @@ -19,6 +19,7 @@ import unittest import warnings +import annotationlib __all__ = [ # globals @@ -32,7 +33,7 @@ "is_resource_enabled", "requires", "requires_freebsd_version", "requires_gil_enabled", "requires_linux_version", "requires_mac_ver", "check_syntax_error", - "requires_gzip", "requires_bz2", "requires_lzma", + "requires_gzip", "requires_bz2", "requires_lzma", "requires_zstd", "bigmemtest", "bigaddrspacetest", "cpython_only", "get_attribute", "requires_IEEE_754", "requires_zlib", "has_fork_support", "requires_fork", @@ -41,10 +42,11 @@ "anticipate_failure", "load_package_tests", "detect_api_mismatch", "check__all__", "skip_if_buggy_ucrt_strfptime", "check_disallow_instantiation", "check_sanitizer", "skip_if_sanitizer", - "requires_limited_api", "requires_specialization", + "requires_limited_api", "requires_specialization", "thread_unsafe", # sys "MS_WINDOWS", "is_jython", "is_android", "is_emscripten", "is_wasi", "is_apple_mobile", "check_impl_detail", "unix_shell", "setswitchinterval", + "support_remote_exec_only", # os "get_pagesize", # network @@ -57,13 +59,16 @@ "run_with_tz", "PGO", "missing_compiler_executable", "ALWAYS_EQ", "NEVER_EQ", "LARGEST", "SMALLEST", "LOOPBACK_TIMEOUT", "INTERNET_TIMEOUT", "SHORT_TIMEOUT", "LONG_TIMEOUT", - "Py_DEBUG", "exceeds_recursion_limit", "get_c_recursion_limit", - "skip_on_s390x", - "without_optimizer", + "Py_DEBUG", "exceeds_recursion_limit", "skip_on_s390x", + "requires_jit_enabled", + "requires_jit_disabled", "force_not_colorized", "force_not_colorized_test_class", "make_clean_env", "BrokenIter", + "in_systemd_nspawn_sync_suppressed", + "run_no_yield_async_fn", "run_yielding_async_fn", "async_yield", + "reset_code", "on_github_actions" ] @@ -389,6 +394,21 @@ def wrapper(*args, **kw): return decorator +def thread_unsafe(reason): + """Mark a test as not thread safe. When the test runner is run with + --parallel-threads=N, the test will be run in a single thread.""" + def decorator(test_item): + test_item.__unittest_thread_unsafe__ = True + # the reason is not currently used + test_item.__unittest_thread_unsafe__why__ = reason + return test_item + if isinstance(reason, types.FunctionType): + test_item = reason + reason = '' + return decorator(test_item) + return decorator + + def skip_if_buildbot(reason=None): """Decorator raising SkipTest if running on a buildbot.""" import getpass @@ -401,7 +421,8 @@ def skip_if_buildbot(reason=None): isbuildbot = False return unittest.skipIf(isbuildbot, reason) -def check_sanitizer(*, address=False, memory=False, ub=False, thread=False): +def check_sanitizer(*, address=False, memory=False, ub=False, thread=False, + function=True): """Returns True if Python is compiled with sanitizer support""" if not (address or memory or ub or thread): raise ValueError('At least one of address, memory, ub or thread must be True') @@ -425,11 +446,15 @@ def check_sanitizer(*, address=False, memory=False, ub=False, thread=False): '-fsanitize=thread' in cflags or '--with-thread-sanitizer' in config_args ) + function_sanitizer = ( + '-fsanitize=function' in cflags + ) return ( (memory and memory_sanitizer) or (address and address_sanitizer) or (ub and ub_sanitizer) or - (thread and thread_sanitizer) + (thread and thread_sanitizer) or + (function and function_sanitizer) ) @@ -514,13 +539,19 @@ def requires_lzma(reason='requires lzma'): lzma = None # XXX: RUSTPYTHON; xz is not supported yet return unittest.skipUnless(lzma, reason) +def requires_zstd(reason='requires zstd'): + try: + from compression import zstd + except ImportError: + zstd = None + return unittest.skipUnless(zstd, reason) + def has_no_debug_ranges(): try: - import _testinternalcapi + import _testcapi except ImportError: raise unittest.SkipTest("_testinternalcapi required") - config = _testinternalcapi.get_config() - return not bool(config['code_debug_ranges']) + return not _testcapi.config_get('code_debug_ranges') def requires_debug_ranges(reason='requires co_positions / debug_ranges'): try: @@ -531,6 +562,7 @@ def requires_debug_ranges(reason='requires co_positions / debug_ranges'): return unittest.skipIf(skip, reason) +# XXX: RUSTPYTHON; this is not belong to 3.14 def can_use_suppress_immortalization(suppress=True): """Check if suppress_immortalization(suppress) can be used. @@ -583,6 +615,11 @@ def skip_if_suppress_immortalization(): is_android = sys.platform == "android" +def skip_android_selinux(name): + return unittest.skipIf( + sys.platform == "android", f"Android blocks {name} with SELinux" + ) + if sys.platform not in {"win32", "vxworks", "ios", "tvos", "watchos"}: unix_shell = '/system/bin/sh' if is_android else '/bin/sh' else: @@ -593,6 +630,15 @@ def skip_if_suppress_immortalization(): is_emscripten = sys.platform == "emscripten" is_wasi = sys.platform == "wasi" +# Use is_wasm32 as a generic check for WebAssembly platforms. +is_wasm32 = is_emscripten or is_wasi + +def skip_emscripten_stack_overflow(): + return unittest.skipIf(is_emscripten, "Exhausts stack on Emscripten") + +def skip_wasi_stack_overflow(): + return unittest.skipIf(is_wasi, "Exhausts stack on WASI") + is_apple_mobile = sys.platform in {"ios", "tvos", "watchos"} is_apple = is_apple_mobile or sys.platform == "darwin" @@ -715,9 +761,11 @@ def sortdict(dict): return "{%s}" % withcommas -def run_code(code: str) -> dict[str, object]: +def run_code(code: str, extra_names: dict[str, object] | None = None) -> dict[str, object]: """Run a piece of code after dedenting it, and return its global namespace.""" ns = {} + if extra_names: + ns.update(extra_names) exec(textwrap.dedent(code), ns) return ns @@ -735,7 +783,9 @@ def check_syntax_error(testcase, statement, errtext='', *, lineno=None, offset=N def open_urlresource(url, *args, **kw): - import urllib.request, urllib.parse + import urllib.parse + import urllib.request + from .os_helper import unlink try: import gzip @@ -953,8 +1003,16 @@ def calcvobjsize(fmt): return struct.calcsize(_vheader + fmt + _align) -_TPFLAGS_HAVE_GC = 1<<14 +_TPFLAGS_STATIC_BUILTIN = 1<<1 +_TPFLAGS_DISALLOW_INSTANTIATION = 1<<7 +_TPFLAGS_IMMUTABLETYPE = 1<<8 _TPFLAGS_HEAPTYPE = 1<<9 +_TPFLAGS_BASETYPE = 1<<10 +_TPFLAGS_READY = 1<<12 +_TPFLAGS_READYING = 1<<13 +_TPFLAGS_HAVE_GC = 1<<14 +_TPFLAGS_BASE_EXC_SUBCLASS = 1<<30 +_TPFLAGS_TYPE_SUBCLASS = 1<<31 def check_sizeof(test, o, size): try: @@ -1318,6 +1376,26 @@ def coverage_wrapper(*args, **kwargs): return coverage_wrapper +def no_rerun(reason): + """Skip rerunning for a particular test. + + WARNING: Use this decorator with care; skipping rerunning makes it + impossible to find reference leaks. Provide a clear reason for skipping the + test using the 'reason' parameter. + """ + def deco(func): + assert not isinstance(func, type), func + _has_run = False + def wrapper(self): + nonlocal _has_run + if _has_run: + self.skipTest(reason) + func(self) + _has_run = True + return wrapper + return deco + + def refcount_test(test): """Decorator for tests which involve reference counting. @@ -1331,8 +1409,9 @@ def refcount_test(test): def requires_limited_api(test): try: - import _testcapi - import _testlimitedcapi + import _testcapi # noqa: F401 + + import _testlimitedcapi # noqa: F401 except ImportError: return unittest.skip('needs _testcapi and _testlimitedcapi modules')(test) return test @@ -1347,6 +1426,18 @@ def requires_specialization(test): _opcode.ENABLE_SPECIALIZATION, "requires specialization")(test) +def requires_specialization_ft(test): + return unittest.skipUnless( + _opcode.ENABLE_SPECIALIZATION_FT, "requires specialization")(test) + + +def reset_code(f: types.FunctionType) -> types.FunctionType: + """Clear all specializations, local instrumentation, and JIT code for the given function.""" + f.__code__ = f.__code__.replace() + return f + +on_github_actions = "GITHUB_ACTIONS" in os.environ + #======================================================================= # Check for the presence of docstrings. @@ -1575,8 +1666,8 @@ def __init__(self, link=None): if sys.platform == "win32": def _platform_specific(self): - import glob import _winapi + import glob if os.path.lexists(self.real) and not os.path.exists(self.real): # App symlink appears to not exist, but we want the @@ -1972,10 +2063,11 @@ def missing_compiler_executable(cmd_names=[]): missing. """ - from setuptools._distutils import ccompiler, sysconfig - from setuptools import errors import shutil + from setuptools import errors + from setuptools._distutils import ccompiler, sysconfig + compiler = ccompiler.new_compiler() sysconfig.customize_compiler(compiler) if compiler.compiler_type == "msvc": @@ -2304,7 +2396,15 @@ def skip_if_broken_multiprocessing_synchronize(): # bpo-38377: On Linux, creating a semaphore fails with OSError # if the current user does not have the permission to create # a file in /dev/shm/ directory. - synchronize.Lock(ctx=None) + import multiprocessing + synchronize.Lock(ctx=multiprocessing.get_context('fork')) + # The explicit fork mp context is required in order for + # TestResourceTracker.test_resource_tracker_reused to work. + # synchronize creates a new multiprocessing.resource_tracker + # process at module import time via the above call in that + # scenario. Awkward. This enables gh-84559. No code involved + # should have threads at that point so fork() should be safe. + except OSError as exc: raise unittest.SkipTest(f"broken multiprocessing SemLock: {exc!r}") @@ -2396,8 +2496,9 @@ def clear_ignored_deprecations(*tokens: object) -> None: raise ValueError("Provide token or tokens returned by ignore_deprecations_from") new_filters = [] + old_filters = warnings._get_filters() endswith = tuple(rf"(?#support{id(token)})" for token in tokens) - for action, message, category, module, lineno in warnings.filters: + for action, message, category, module, lineno in old_filters: if action == "ignore" and category is DeprecationWarning: if isinstance(message, re.Pattern): msg = message.pattern @@ -2406,8 +2507,8 @@ def clear_ignored_deprecations(*tokens: object) -> None: if msg.endswith(endswith): continue new_filters.append((action, message, category, module, lineno)) - if warnings.filters != new_filters: - warnings.filters[:] = new_filters + if old_filters != new_filters: + old_filters[:] = new_filters warnings._filters_mutated() @@ -2415,7 +2516,7 @@ def clear_ignored_deprecations(*tokens: object) -> None: def requires_venv_with_pip(): # ensurepip requires zlib to open ZIP archives (.whl binary wheel packages) try: - import zlib + import zlib # noqa: F401 except ImportError: return unittest.skipIf(True, "venv: ensurepip requires zlib") @@ -2455,6 +2556,7 @@ def _findwheel(pkgname): @contextlib.contextmanager def setup_venv_with_pip_setuptools(venv_dir): import subprocess + from .os_helper import temp_cwd def run_command(cmd): @@ -2610,30 +2712,30 @@ def sleeping_retry(timeout, err_msg=None, /, delay = min(delay * 2, max_delay) -class CPUStopwatch: +class Stopwatch: """Context manager to roughly time a CPU-bound operation. - Disables GC. Uses CPU time if it can (i.e. excludes sleeps & time of - other processes). + Disables GC. Uses perf_counter, which is a clock with the highest + available resolution. It is chosen even though it does include + time elapsed during sleep and is system-wide, because the + resolution of process_time is too coarse on Windows and + process_time does not exist everywhere (for example, WASM). - N.B.: - - This *includes* time spent in other threads. + Note: + - This *includes* time spent in other threads/processes. - Some systems only have a coarse resolution; check - stopwatch.clock_info.rseolution if. + stopwatch.clock_info.resolution when using the results. Usage: - with ProcessStopwatch() as stopwatch: + with Stopwatch() as stopwatch: ... elapsed = stopwatch.seconds resolution = stopwatch.clock_info.resolution """ def __enter__(self): - get_time = time.process_time - clock_info = time.get_clock_info('process_time') - if get_time() <= 0: # some platforms like WASM lack process_time() - get_time = time.monotonic - clock_info = time.get_clock_info('monotonic') + get_time = time.perf_counter + clock_info = time.get_clock_info('perf_counter') self.context = disable_gc() self.context.__enter__() self.get_time = get_time @@ -2661,6 +2763,7 @@ def adjust_int_max_str_digits(max_digits): sys.set_int_max_str_digits(current) +# XXX: RUSTPYTHON; removed in 3.14 def get_c_recursion_limit(): try: import _testcapi @@ -2671,7 +2774,7 @@ def get_c_recursion_limit(): def exceeds_recursion_limit(): """For recursion tests, easily exceeds default recursion limit.""" - return get_c_recursion_limit() * 3 + return 150_000 # Windows doesn't have os.uname() but it doesn't support s390x. @@ -2680,21 +2783,9 @@ def exceeds_recursion_limit(): Py_TRACE_REFS = hasattr(sys, 'getobjects') -# Decorator to disable optimizer while a function run -def without_optimizer(func): - try: - from _testinternalcapi import get_optimizer, set_optimizer - except ImportError: - return func - @functools.wraps(func) - def wrapper(*args, **kwargs): - save_opt = get_optimizer() - try: - set_optimizer(None) - return func(*args, **kwargs) - finally: - set_optimizer(save_opt) - return wrapper +_JIT_ENABLED = sys._jit.is_enabled() +requires_jit_enabled = unittest.skipUnless(_JIT_ENABLED, "requires JIT enabled") +requires_jit_disabled = unittest.skipIf(_JIT_ENABLED, "requires JIT disabled") _BASE_COPY_SRC_DIR_IGNORED_NAMES = frozenset({ @@ -2724,19 +2815,121 @@ def copy_python_src_ignore(path, names): return ignored -def iter_builtin_types(): - for obj in __builtins__.values(): - if not isinstance(obj, type): +# XXX Move this to the inspect module? +def walk_class_hierarchy(top, *, topdown=True): + # This is based on the logic in os.walk(). + assert isinstance(top, type), repr(top) + stack = [top] + while stack: + top = stack.pop() + if isinstance(top, tuple): + yield top continue - cls = obj - if cls.__module__ != 'builtins': + + subs = type(top).__subclasses__(top) + if topdown: + # Yield before subclass traversal if going top down. + yield top, subs + # Traverse into subclasses. + for sub in reversed(subs): + stack.append(sub) + else: + # Yield after subclass traversal if going bottom up. + stack.append((top, subs)) + # Traverse into subclasses. + for sub in reversed(subs): + stack.append(sub) + + +def iter_builtin_types(): + # First try the explicit route. + try: + import _testinternalcapi + except ImportError: + _testinternalcapi = None + if _testinternalcapi is not None: + yield from _testinternalcapi.get_static_builtin_types() + return + + # Fall back to making a best-effort guess. + if hasattr(object, '__flags__'): + # Look for any type object with the Py_TPFLAGS_STATIC_BUILTIN flag set. + import datetime + seen = set() + for cls, subs in walk_class_hierarchy(object): + if cls in seen: + continue + seen.add(cls) + if not (cls.__flags__ & _TPFLAGS_STATIC_BUILTIN): + # Do not walk its subclasses. + subs[:] = [] + continue + yield cls + else: + # Fall back to a naive approach. + seen = set() + for obj in __builtins__.values(): + if not isinstance(obj, type): + continue + cls = obj + # XXX? + if cls.__module__ != 'builtins': + continue + if cls == ExceptionGroup: + # It's a heap type. + continue + if cls in seen: + continue + seen.add(cls) + yield cls + + +# XXX Move this to the inspect module? +def iter_name_in_mro(cls, name): + """Yield matching items found in base.__dict__ across the MRO. + + The descriptor protocol is not invoked. + + list(iter_name_in_mro(cls, name))[0] is roughly equivalent to + find_name_in_mro() in Objects/typeobject.c (AKA PyType_Lookup()). + + inspect.getattr_static() is similar. + """ + # This can fail if "cls" is weird. + for base in inspect._static_getmro(cls): + # This can fail if "base" is weird. + ns = inspect._get_dunder_dict_of_class(base) + try: + obj = ns[name] + except KeyError: continue - yield cls + yield obj, base -def iter_slot_wrappers(cls): - assert cls.__module__ == 'builtins', cls +# XXX Move this to the inspect module? +def find_name_in_mro(cls, name, default=inspect._sentinel): + for res in iter_name_in_mro(cls, name): + # Return the first one. + return res + if default is not inspect._sentinel: + return default, None + raise AttributeError(name) + +# XXX The return value should always be exactly the same... +def identify_type_slot_wrappers(): + try: + import _testinternalcapi + except ImportError: + _testinternalcapi = None + if _testinternalcapi is not None: + names = {n: None for n in _testinternalcapi.identify_type_slot_wrappers()} + return list(names) + else: + raise NotImplementedError + + +def iter_slot_wrappers(cls): def is_slot_wrapper(name, value): if not isinstance(value, types.WrapperDescriptorType): assert not repr(value).startswith(' dict[str, str]: return clean_env -def initialized_with_pyrepl(): - """Detect whether PyREPL was used during Python initialization.""" - # If the main module has a __file__ attribute it's a Python module, which means PyREPL. - return hasattr(sys.modules["__main__"], "__file__") +WINDOWS_STATUS = { + 0xC0000005: "STATUS_ACCESS_VIOLATION", + 0xC00000FD: "STATUS_STACK_OVERFLOW", + 0xC000013A: "STATUS_CONTROL_C_EXIT", +} + +def get_signal_name(exitcode): + import signal + + if exitcode < 0: + signum = -exitcode + try: + return signal.Signals(signum).name + except ValueError: + pass + + # Shell exit code (ex: WASI build) + if 128 < exitcode < 256: + signum = exitcode - 128 + try: + return signal.Signals(signum).name + except ValueError: + pass + + try: + return WINDOWS_STATUS[exitcode] + except KeyError: + pass + return None class BrokenIter: def __init__(self, init_raises=False, next_raises=False, iter_raises=False): @@ -2849,222 +3104,166 @@ def __iter__(self): return self -def linked_to_musl(): +def in_systemd_nspawn_sync_suppressed() -> bool: """ - Test if the Python executable is linked to the musl C library. + Test whether the test suite is runing in systemd-nspawn + with ``--suppress-sync=true``. + + This can be used to skip tests that rely on ``fsync()`` calls + and similar not being intercepted. """ - if sys.platform != 'linux': + + if not hasattr(os, "O_SYNC"): return False - import subprocess - exe = getattr(sys, '_base_executable', sys.executable) - cmd = ['ldd', exe] try: - stdout = subprocess.check_output(cmd, - text=True, - stderr=subprocess.STDOUT) - except (OSError, subprocess.CalledProcessError): + with open("/run/systemd/container", "rb") as fp: + if fp.read().rstrip() != b"systemd-nspawn": + return False + except FileNotFoundError: return False - return ('musl' in stdout) + # If systemd-nspawn is used, O_SYNC flag will immediately + # trigger EINVAL. Otherwise, ENOENT will be given instead. + import errno + try: + fd = os.open(__file__, os.O_RDONLY | os.O_SYNC) + except OSError as err: + if err.errno == errno.EINVAL: + return True + else: + os.close(fd) -# TODO: RUSTPYTHON -# Every line of code below allowed us to update `Lib/test/support/__init__.py` without -# needing to update `libregtest` and its dependencies. -# Ideally we want to remove all code below and update `libregtest`. -# -# Code below was copied from: https://github.com/RustPython/RustPython/blob/9499d39f55b73535e2405bf208d5380241f79ada/Lib/test/support/__init__.py + return False -from .testresult import get_test_runner +def run_no_yield_async_fn(async_fn, /, *args, **kwargs): + coro = async_fn(*args, **kwargs) + try: + coro.send(None) + except StopIteration as e: + return e.value + else: + raise AssertionError("coroutine did not complete") + finally: + coro.close() -def _filter_suite(suite, pred): - """Recursively filter test cases in a suite based on a predicate.""" - newtests = [] - for test in suite._tests: - if isinstance(test, unittest.TestSuite): - _filter_suite(test, pred) - newtests.append(test) - else: - if pred(test): - newtests.append(test) - suite._tests = newtests -# By default, don't filter tests -_match_test_func = None +@types.coroutine +def async_yield(v): + return (yield v) -_accept_test_patterns = None -_ignore_test_patterns = None -def match_test(test): - # Function used by support.run_unittest() and regrtest --list-cases - if _match_test_func is None: - return True - else: - return _match_test_func(test.id()) +def run_yielding_async_fn(async_fn, /, *args, **kwargs): + coro = async_fn(*args, **kwargs) + try: + while True: + try: + coro.send(None) + except StopIteration as e: + return e.value + finally: + coro.close() -def _is_full_match_test(pattern): - # If a pattern contains at least one dot, it's considered - # as a full test identifier. - # Example: 'test.test_os.FileTests.test_access'. - # - # ignore patterns which contain fnmatch patterns: '*', '?', '[...]' - # or '[!...]'. For example, ignore 'test_access*'. - return ('.' in pattern) and (not re.search(r'[?*\[\]]', pattern)) - -def set_match_tests(accept_patterns=None, ignore_patterns=None): - global _match_test_func, _accept_test_patterns, _ignore_test_patterns - - if accept_patterns is None: - accept_patterns = () - if ignore_patterns is None: - ignore_patterns = () - - accept_func = ignore_func = None - - if accept_patterns != _accept_test_patterns: - accept_patterns, accept_func = _compile_match_function(accept_patterns) - if ignore_patterns != _ignore_test_patterns: - ignore_patterns, ignore_func = _compile_match_function(ignore_patterns) - - # Create a copy since patterns can be mutable and so modified later - _accept_test_patterns = tuple(accept_patterns) - _ignore_test_patterns = tuple(ignore_patterns) - - if accept_func is not None or ignore_func is not None: - def match_function(test_id): - accept = True - ignore = False - if accept_func: - accept = accept_func(test_id) - if ignore_func: - ignore = ignore_func(test_id) - return accept and not ignore - - _match_test_func = match_function - -def _compile_match_function(patterns): - if not patterns: - func = None - # set_match_tests(None) behaves as set_match_tests(()) - patterns = () - elif all(map(_is_full_match_test, patterns)): - # Simple case: all patterns are full test identifier. - # The test.bisect_cmd utility only uses such full test identifiers. - func = set(patterns).__contains__ - else: - import fnmatch - regex = '|'.join(map(fnmatch.translate, patterns)) - # The search *is* case sensitive on purpose: - # don't use flags=re.IGNORECASE - regex_match = re.compile(regex).match - - def match_test_regex(test_id): - if regex_match(test_id): - # The regex matches the whole identifier, for example - # 'test.test_os.FileTests.test_access'. - return True - else: - # Try to match parts of the test identifier. - # For example, split 'test.test_os.FileTests.test_access' - # into: 'test', 'test_os', 'FileTests' and 'test_access'. - return any(map(regex_match, test_id.split("."))) - - func = match_test_regex - - return patterns, func - -def run_unittest(*classes): - """Run tests from unittest.TestCase-derived classes.""" - valid_types = (unittest.TestSuite, unittest.TestCase) - loader = unittest.TestLoader() - suite = unittest.TestSuite() - for cls in classes: - if isinstance(cls, str): - if cls in sys.modules: - suite.addTest(loader.loadTestsFromModule(sys.modules[cls])) - else: - raise ValueError("str arguments must be keys in sys.modules") - elif isinstance(cls, valid_types): - suite.addTest(cls) - else: - suite.addTest(loader.loadTestsFromTestCase(cls)) - _filter_suite(suite, match_test) - return _run_suite(suite) - -def _run_suite(suite): - """Run tests from a unittest.TestSuite-derived class.""" - runner = get_test_runner(sys.stdout, - verbosity=verbose, - capture_output=(junit_xml_list is not None)) - - result = runner.run(suite) - - if junit_xml_list is not None: - junit_xml_list.append(result.get_xml_element()) - - if not result.testsRun and not result.skipped and not result.errors: - raise TestDidNotRun - if not result.wasSuccessful(): - stats = TestStats.from_unittest(result) - if len(result.errors) == 1 and not result.failures: - err = result.errors[0][1] - elif len(result.failures) == 1 and not result.errors: - err = result.failures[0][1] - else: - err = "multiple errors occurred" - if not verbose: err += "; run in verbose mode for details" - errors = [(str(tc), exc_str) for tc, exc_str in result.errors] - failures = [(str(tc), exc_str) for tc, exc_str in result.failures] - raise TestFailedWithDetails(err, errors, failures, stats=stats) - return result -@dataclasses.dataclass(slots=True) -class TestStats: - tests_run: int = 0 - failures: int = 0 - skipped: int = 0 +def is_libssl_fips_mode(): + try: + from _hashlib import get_fips_mode # ask _hashopenssl.c + except ImportError: + return False # more of a maybe, unless we add this to the _ssl module. + return get_fips_mode() != 0 + +def _supports_remote_attaching(): + PROCESS_VM_READV_SUPPORTED = False + + try: + from _remote_debugging import PROCESS_VM_READV_SUPPORTED + except ImportError: + pass - @staticmethod - def from_unittest(result): - return TestStats(result.testsRun, - len(result.failures), - len(result.skipped)) + return PROCESS_VM_READV_SUPPORTED - @staticmethod - def from_doctest(results): - return TestStats(results.attempted, - results.failed) +def _support_remote_exec_only_impl(): + if not sys.is_remote_debug_enabled(): + return unittest.skip("Remote debugging is not enabled") + if sys.platform not in ("darwin", "linux", "win32"): + return unittest.skip("Test only runs on Linux, Windows and macOS") + if sys.platform == "linux" and not _supports_remote_attaching(): + return unittest.skip("Test only runs on Linux with process_vm_readv support") + return _id - def accumulate(self, stats): - self.tests_run += stats.tests_run - self.failures += stats.failures - self.skipped += stats.skipped +def support_remote_exec_only(test): + return _support_remote_exec_only_impl()(test) +class EqualToForwardRef: + """Helper to ease use of annotationlib.ForwardRef in tests. -def run_doctest(module, verbosity=None, optionflags=0): - """Run doctest on the given module. Return (#failures, #tests). + This checks only attributes that can be set using the constructor. - If optional argument verbosity is not specified (or is None), pass - support's belief about verbosity on to doctest. Else doctest's - usual behavior is used (it searches sys.argv for -v). """ - import doctest + def __init__( + self, + arg, + *, + module=None, + owner=None, + is_class=False, + ): + self.__forward_arg__ = arg + self.__forward_is_class__ = is_class + self.__forward_module__ = module + self.__owner__ = owner - if verbosity is None: - verbosity = verbose - else: - verbosity = None - - results = doctest.testmod(module, - verbose=verbosity, - optionflags=optionflags) - if results.failed: - stats = TestStats.from_doctest(results) - raise TestFailed(f"{results.failed} of {results.attempted} " - f"doctests failed", - stats=stats) - if verbose: - print('doctest (%s) ... %d tests with zero failures' % - (module.__name__, results.attempted)) - return results + def __eq__(self, other): + if not isinstance(other, (EqualToForwardRef, annotationlib.ForwardRef)): + return NotImplemented + return ( + self.__forward_arg__ == other.__forward_arg__ + and self.__forward_module__ == other.__forward_module__ + and self.__forward_is_class__ == other.__forward_is_class__ + and self.__owner__ == other.__owner__ + ) + + def __repr__(self): + extra = [] + if self.__forward_module__ is not None: + extra.append(f", module={self.__forward_module__!r}") + if self.__forward_is_class__: + extra.append(", is_class=True") + if self.__owner__ is not None: + extra.append(f", owner={self.__owner__!r}") + return f"EqualToForwardRef({self.__forward_arg__!r}{''.join(extra)})" + + +_linked_to_musl = None +def linked_to_musl(): + """ + Report if the Python executable is linked to the musl C library. + + Return False if we don't think it is, or a version triple otherwise. + """ + # This is can be a relatively expensive check, so we use a cache. + global _linked_to_musl + if _linked_to_musl is not None: + return _linked_to_musl + + # emscripten (at least as far as we're concerned) and wasi use musl, + # but platform doesn't know how to get the version, so set it to zero. + if is_wasm32: + _linked_to_musl = (0, 0, 0) + return _linked_to_musl + + # On all other non-linux platforms assume no musl. + if sys.platform != 'linux': + _linked_to_musl = False + return _linked_to_musl + + # On linux, we'll depend on the platform module to do the check, so new + # musl platforms should add support in that module if possible. + import platform + lib, version = platform.libc_ver() + if lib != 'musl': + _linked_to_musl = False + return _linked_to_musl + _linked_to_musl = tuple(map(int, version.split('.'))) + return _linked_to_musl diff --git a/Lib/test/support/_hypothesis_stubs/__init__.py b/Lib/test/support/_hypothesis_stubs/__init__.py index 6ba5bb814b9..9a57c309616 100644 --- a/Lib/test/support/_hypothesis_stubs/__init__.py +++ b/Lib/test/support/_hypothesis_stubs/__init__.py @@ -1,6 +1,6 @@ -from enum import Enum import functools import unittest +from enum import Enum __all__ = [ "given", diff --git a/Lib/test/support/ast_helper.py b/Lib/test/support/ast_helper.py index 8a0415b6aae..98eaf0b2721 100644 --- a/Lib/test/support/ast_helper.py +++ b/Lib/test/support/ast_helper.py @@ -1,5 +1,6 @@ import ast + class ASTTestMixin: """Test mixing to have basic assertions for AST nodes.""" @@ -16,6 +17,9 @@ def traverse_compare(a, b, missing=object()): self.fail(f"{type(a)!r} is not {type(b)!r}") if isinstance(a, ast.AST): for field in a._fields: + if isinstance(a, ast.Constant) and field == "kind": + # Skip the 'kind' field for ast.Constant + continue value1 = getattr(a, field, missing) value2 = getattr(b, field, missing) # Singletons are equal by definition, so further diff --git a/Lib/test/support/asynchat.py b/Lib/test/support/asynchat.py index 38c47a1fda6..a8c6b28a9e1 100644 --- a/Lib/test/support/asynchat.py +++ b/Lib/test/support/asynchat.py @@ -1,5 +1,5 @@ # TODO: This module was deprecated and removed from CPython 3.12 -# Now it is a test-only helper. Any attempts to rewrite exising tests that +# Now it is a test-only helper. Any attempts to rewrite existing tests that # are using this module and remove it completely are appreciated! # See: https://github.com/python/cpython/issues/72719 diff --git a/Lib/test/support/asyncore.py b/Lib/test/support/asyncore.py index b397aca5568..658c22fdcee 100644 --- a/Lib/test/support/asyncore.py +++ b/Lib/test/support/asyncore.py @@ -1,5 +1,5 @@ # TODO: This module was deprecated and removed from CPython 3.12 -# Now it is a test-only helper. Any attempts to rewrite exising tests that +# Now it is a test-only helper. Any attempts to rewrite existing tests that # are using this module and remove it completely are appreciated! # See: https://github.com/python/cpython/issues/72719 @@ -51,17 +51,27 @@ sophisticated high-performance network servers and clients a snap. """ +import os import select import socket import sys import time import warnings - -import os -from errno import EALREADY, EINPROGRESS, EWOULDBLOCK, ECONNRESET, EINVAL, \ - ENOTCONN, ESHUTDOWN, EISCONN, EBADF, ECONNABORTED, EPIPE, EAGAIN, \ - errorcode - +from errno import ( + EAGAIN, + EALREADY, + EBADF, + ECONNABORTED, + ECONNRESET, + EINPROGRESS, + EINVAL, + EISCONN, + ENOTCONN, + EPIPE, + ESHUTDOWN, + EWOULDBLOCK, + errorcode, +) _DISCONNECTED = frozenset({ECONNRESET, ENOTCONN, ESHUTDOWN, ECONNABORTED, EPIPE, EBADF}) diff --git a/Lib/test/support/bytecode_helper.py b/Lib/test/support/bytecode_helper.py index 85bcd1f0f1c..4a3c8c2c4f1 100644 --- a/Lib/test/support/bytecode_helper.py +++ b/Lib/test/support/bytecode_helper.py @@ -1,9 +1,10 @@ """bytecode_helper - support tools for testing correct bytecode generation""" -import unittest import dis import io import opcode +import unittest + try: import _testinternalcapi except ImportError: @@ -71,7 +72,7 @@ class Label: def assertInstructionsMatch(self, actual_seq, expected): # get an InstructionSequence and an expected list, where each - # entry is a label or an instruction tuple. Construct an expcted + # entry is a label or an instruction tuple. Construct an expected # instruction sequence and compare with the one given. self.assertIsInstance(expected, list) diff --git a/Lib/test/support/interpreters/channels.py b/Lib/test/support/channels.py similarity index 73% rename from Lib/test/support/interpreters/channels.py rename to Lib/test/support/channels.py index d2bd93d77f7..3f7b46030fd 100644 --- a/Lib/test/support/interpreters/channels.py +++ b/Lib/test/support/channels.py @@ -1,19 +1,23 @@ """Cross-interpreter Channels High Level Module.""" import time +from concurrent.interpreters import _crossinterp +from concurrent.interpreters._crossinterp import ( + UNBOUND_ERROR, + UNBOUND_REMOVE, +) + import _interpchannels as _channels -from . import _crossinterp # aliases: from _interpchannels import ( - ChannelError, ChannelNotFoundError, ChannelClosedError, - ChannelEmptyError, ChannelNotEmptyError, -) -from ._crossinterp import ( - UNBOUND_ERROR, UNBOUND_REMOVE, + ChannelClosedError, + ChannelEmptyError, + ChannelError, + ChannelNotEmptyError, + ChannelNotFoundError, ) - __all__ = [ 'UNBOUND', 'UNBOUND_ERROR', 'UNBOUND_REMOVE', 'create', 'list_all', @@ -55,15 +59,23 @@ def create(*, unbounditems=UNBOUND): """ unbound = _serialize_unbound(unbounditems) unboundop, = unbound - cid = _channels.create(unboundop) - recv, send = RecvChannel(cid), SendChannel(cid, _unbound=unbound) + cid = _channels.create(unboundop, -1) + recv, send = RecvChannel(cid), SendChannel(cid) + send._set_unbound(unboundop, unbounditems) return recv, send def list_all(): """Return a list of (recv, send) for all open channels.""" - return [(RecvChannel(cid), SendChannel(cid, _unbound=unbound)) - for cid, unbound in _channels.list_all()] + channels = [] + for cid, unboundop, _ in _channels.list_all(): + chan = _, send = RecvChannel(cid), SendChannel(cid) + if not hasattr(send, '_unboundop'): + send._set_unbound(unboundop) + else: + assert send._unbound[0] == unboundop + channels.append(chan) + return channels class _ChannelEnd: @@ -97,12 +109,8 @@ def __eq__(self, other): return other._id == self._id # for pickling: - def __getnewargs__(self): - return (int(self._id),) - - # for pickling: - def __getstate__(self): - return None + def __reduce__(self): + return (type(self), (int(self._id),)) @property def id(self): @@ -175,16 +183,33 @@ class SendChannel(_ChannelEnd): _end = 'send' - def __new__(cls, cid, *, _unbound=None): - if _unbound is None: - try: - op = _channels.get_channel_defaults(cid) - _unbound = (op,) - except ChannelNotFoundError: - _unbound = _serialize_unbound(UNBOUND) - self = super().__new__(cls, cid) - self._unbound = _unbound - return self +# def __new__(cls, cid, *, _unbound=None): +# if _unbound is None: +# try: +# op = _channels.get_channel_defaults(cid) +# _unbound = (op,) +# except ChannelNotFoundError: +# _unbound = _serialize_unbound(UNBOUND) +# self = super().__new__(cls, cid) +# self._unbound = _unbound +# return self + + def _set_unbound(self, op, items=None): + assert not hasattr(self, '_unbound') + if items is None: + items = _resolve_unbound(op) + unbound = (op, items) + self._unbound = unbound + return unbound + + @property + def unbounditems(self): + try: + _, items = self._unbound + except AttributeError: + op, _ = _channels.get_queue_defaults(self._id) + _, items = self._set_unbound(op) + return items @property def is_closed(self): @@ -192,61 +217,61 @@ def is_closed(self): return info.closed or info.closing def send(self, obj, timeout=None, *, - unbound=None, + unbounditems=None, ): """Send the object (i.e. its data) to the channel's receiving end. This blocks until the object is received. """ - if unbound is None: - unboundop, = self._unbound + if unbounditems is None: + unboundop = -1 else: - unboundop, = _serialize_unbound(unbound) + unboundop, = _serialize_unbound(unbounditems) _channels.send(self._id, obj, unboundop, timeout=timeout, blocking=True) def send_nowait(self, obj, *, - unbound=None, + unbounditems=None, ): """Send the object to the channel's receiving end. If the object is immediately received then return True (else False). Otherwise this is the same as send(). """ - if unbound is None: - unboundop, = self._unbound + if unbounditems is None: + unboundop = -1 else: - unboundop, = _serialize_unbound(unbound) + unboundop, = _serialize_unbound(unbounditems) # XXX Note that at the moment channel_send() only ever returns # None. This should be fixed when channel_send_wait() is added. # See bpo-32604 and gh-19829. return _channels.send(self._id, obj, unboundop, blocking=False) def send_buffer(self, obj, timeout=None, *, - unbound=None, + unbounditems=None, ): """Send the object's buffer to the channel's receiving end. This blocks until the object is received. """ - if unbound is None: - unboundop, = self._unbound + if unbounditems is None: + unboundop = -1 else: - unboundop, = _serialize_unbound(unbound) + unboundop, = _serialize_unbound(unbounditems) _channels.send_buffer(self._id, obj, unboundop, timeout=timeout, blocking=True) def send_buffer_nowait(self, obj, *, - unbound=None, + unbounditems=None, ): """Send the object's buffer to the channel's receiving end. If the object is immediately received then return True (else False). Otherwise this is the same as send(). """ - if unbound is None: - unboundop, = self._unbound + if unbounditems is None: + unboundop = -1 else: - unboundop, = _serialize_unbound(unbound) + unboundop, = _serialize_unbound(unbounditems) return _channels.send_buffer(self._id, obj, unboundop, blocking=False) def close(self): diff --git a/Lib/test/support/hashlib_helper.py b/Lib/test/support/hashlib_helper.py index a4e6c92203a..75dc2ba7506 100644 --- a/Lib/test/support/hashlib_helper.py +++ b/Lib/test/support/hashlib_helper.py @@ -1,51 +1,330 @@ import functools import hashlib +import importlib import unittest +from test.support.import_helper import import_module + try: import _hashlib except ImportError: _hashlib = None +try: + import _hmac +except ImportError: + _hmac = None + + +def requires_hashlib(): + return unittest.skipIf(_hashlib is None, "requires _hashlib") + + +def requires_builtin_hmac(): + return unittest.skipIf(_hmac is None, "requires _hmac") + + +def _missing_hash(digestname, implementation=None, *, exc=None): + parts = ["missing", implementation, f"hash algorithm: {digestname!r}"] + msg = " ".join(filter(None, parts)) + raise unittest.SkipTest(msg) from exc + + +def _openssl_availabillity(digestname, *, usedforsecurity): + try: + _hashlib.new(digestname, usedforsecurity=usedforsecurity) + except AttributeError: + assert _hashlib is None + _missing_hash(digestname, "OpenSSL") + except ValueError as exc: + _missing_hash(digestname, "OpenSSL", exc=exc) + + +def _decorate_func_or_class(func_or_class, decorator_func): + if not isinstance(func_or_class, type): + return decorator_func(func_or_class) + + decorated_class = func_or_class + setUpClass = decorated_class.__dict__.get('setUpClass') + if setUpClass is None: + def setUpClass(cls): + super(decorated_class, cls).setUpClass() + setUpClass.__qualname__ = decorated_class.__qualname__ + '.setUpClass' + setUpClass.__module__ = decorated_class.__module__ + else: + setUpClass = setUpClass.__func__ + setUpClass = classmethod(decorator_func(setUpClass)) + decorated_class.setUpClass = setUpClass + return decorated_class + def requires_hashdigest(digestname, openssl=None, usedforsecurity=True): - """Decorator raising SkipTest if a hashing algorithm is not available + """Decorator raising SkipTest if a hashing algorithm is not available. - The hashing algorithm could be missing or blocked by a strict crypto - policy. + The hashing algorithm may be missing, blocked by a strict crypto policy, + or Python may be configured with `--with-builtin-hashlib-hashes=no`. If 'openssl' is True, then the decorator checks that OpenSSL provides - the algorithm. Otherwise the check falls back to built-in - implementations. The usedforsecurity flag is passed to the constructor. + the algorithm. Otherwise the check falls back to (optional) built-in + HACL* implementations. + The usedforsecurity flag is passed to the constructor but has no effect + on HACL* implementations. + + Examples of exceptions being suppressed: ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS ValueError: unsupported hash type md4 """ + if openssl and _hashlib is not None: + def test_availability(): + _hashlib.new(digestname, usedforsecurity=usedforsecurity) + else: + def test_availability(): + hashlib.new(digestname, usedforsecurity=usedforsecurity) + + def decorator_func(func): + @functools.wraps(func) + def wrapper(*args, **kwargs): + try: + test_availability() + except ValueError as exc: + _missing_hash(digestname, exc=exc) + return func(*args, **kwargs) + return wrapper + + def decorator(func_or_class): + return _decorate_func_or_class(func_or_class, decorator_func) + return decorator + + +def requires_openssl_hashdigest(digestname, *, usedforsecurity=True): + """Decorator raising SkipTest if an OpenSSL hashing algorithm is missing. + + The hashing algorithm may be missing or blocked by a strict crypto policy. + """ + def decorator_func(func): + @requires_hashlib() # avoid checking at each call + @functools.wraps(func) + def wrapper(*args, **kwargs): + _openssl_availabillity(digestname, usedforsecurity=usedforsecurity) + return func(*args, **kwargs) + return wrapper + def decorator(func_or_class): - if isinstance(func_or_class, type): - setUpClass = func_or_class.__dict__.get('setUpClass') - if setUpClass is None: - def setUpClass(cls): - super(func_or_class, cls).setUpClass() - setUpClass.__qualname__ = func_or_class.__qualname__ + '.setUpClass' - setUpClass.__module__ = func_or_class.__module__ - else: - setUpClass = setUpClass.__func__ - setUpClass = classmethod(decorator(setUpClass)) - func_or_class.setUpClass = setUpClass - return func_or_class - - @functools.wraps(func_or_class) + return _decorate_func_or_class(func_or_class, decorator_func) + return decorator + + +def find_openssl_hashdigest_constructor(digestname, *, usedforsecurity=True): + """Find the OpenSSL hash function constructor by its name.""" + assert isinstance(digestname, str), digestname + _openssl_availabillity(digestname, usedforsecurity=usedforsecurity) + # This returns a function of the form _hashlib.openssl_ and + # not a lambda function as it is rejected by _hashlib.hmac_new(). + return getattr(_hashlib, f"openssl_{digestname}") + + +def requires_builtin_hashdigest( + module_name, digestname, *, usedforsecurity=True +): + """Decorator raising SkipTest if a HACL* hashing algorithm is missing. + + - The *module_name* is the C extension module name based on HACL*. + - The *digestname* is one of its member, e.g., 'md5'. + """ + def decorator_func(func): + @functools.wraps(func) def wrapper(*args, **kwargs): + module = import_module(module_name) try: - if openssl and _hashlib is not None: - _hashlib.new(digestname, usedforsecurity=usedforsecurity) - else: - hashlib.new(digestname, usedforsecurity=usedforsecurity) - except ValueError: - raise unittest.SkipTest( - f"hash digest '{digestname}' is not available." - ) - return func_or_class(*args, **kwargs) + getattr(module, digestname) + except AttributeError: + fullname = f'{module_name}.{digestname}' + _missing_hash(fullname, implementation="HACL") + return func(*args, **kwargs) return wrapper + + def decorator(func_or_class): + return _decorate_func_or_class(func_or_class, decorator_func) return decorator + + +def find_builtin_hashdigest_constructor( + module_name, digestname, *, usedforsecurity=True +): + """Find the HACL* hash function constructor. + + - The *module_name* is the C extension module name based on HACL*. + - The *digestname* is one of its member, e.g., 'md5'. + """ + module = import_module(module_name) + try: + constructor = getattr(module, digestname) + constructor(b'', usedforsecurity=usedforsecurity) + except (AttributeError, TypeError, ValueError): + _missing_hash(f'{module_name}.{digestname}', implementation="HACL") + return constructor + + +class HashFunctionsTrait: + """Mixin trait class containing hash functions. + + This class is assumed to have all unitest.TestCase methods but should + not directly inherit from it to prevent the test suite being run on it. + + Subclasses should implement the hash functions by returning an object + that can be recognized as a valid digestmod parameter for both hashlib + and HMAC. In particular, it cannot be a lambda function as it will not + be recognized by hashlib (it will still be accepted by the pure Python + implementation of HMAC). + """ + + ALGORITHMS = [ + 'md5', 'sha1', + 'sha224', 'sha256', 'sha384', 'sha512', + 'sha3_224', 'sha3_256', 'sha3_384', 'sha3_512', + ] + + # Default 'usedforsecurity' to use when looking up a hash function. + usedforsecurity = True + + def _find_constructor(self, name): + # By default, a missing algorithm skips the test that uses it. + self.assertIn(name, self.ALGORITHMS) + self.skipTest(f"missing hash function: {name}") + + @property + def md5(self): + return self._find_constructor("md5") + + @property + def sha1(self): + return self._find_constructor("sha1") + + @property + def sha224(self): + return self._find_constructor("sha224") + + @property + def sha256(self): + return self._find_constructor("sha256") + + @property + def sha384(self): + return self._find_constructor("sha384") + + @property + def sha512(self): + return self._find_constructor("sha512") + + @property + def sha3_224(self): + return self._find_constructor("sha3_224") + + @property + def sha3_256(self): + return self._find_constructor("sha3_256") + + @property + def sha3_384(self): + return self._find_constructor("sha3_384") + + @property + def sha3_512(self): + return self._find_constructor("sha3_512") + + +class NamedHashFunctionsTrait(HashFunctionsTrait): + """Trait containing named hash functions. + + Hash functions are available if and only if they are available in hashlib. + """ + + def _find_constructor(self, name): + self.assertIn(name, self.ALGORITHMS) + return name + + +class OpenSSLHashFunctionsTrait(HashFunctionsTrait): + """Trait containing OpenSSL hash functions. + + Hash functions are available if and only if they are available in _hashlib. + """ + + def _find_constructor(self, name): + self.assertIn(name, self.ALGORITHMS) + return find_openssl_hashdigest_constructor( + name, usedforsecurity=self.usedforsecurity + ) + + +class BuiltinHashFunctionsTrait(HashFunctionsTrait): + """Trait containing HACL* hash functions. + + Hash functions are available if and only if they are available in C. + In particular, HACL* HMAC-MD5 may be available even though HACL* md5 + is not since the former is unconditionally built. + """ + + def _find_constructor_in(self, module, name): + self.assertIn(name, self.ALGORITHMS) + return find_builtin_hashdigest_constructor(module, name) + + @property + def md5(self): + return self._find_constructor_in("_md5", "md5") + + @property + def sha1(self): + return self._find_constructor_in("_sha1", "sha1") + + @property + def sha224(self): + return self._find_constructor_in("_sha2", "sha224") + + @property + def sha256(self): + return self._find_constructor_in("_sha2", "sha256") + + @property + def sha384(self): + return self._find_constructor_in("_sha2", "sha384") + + @property + def sha512(self): + return self._find_constructor_in("_sha2", "sha512") + + @property + def sha3_224(self): + return self._find_constructor_in("_sha3", "sha3_224") + + @property + def sha3_256(self): + return self._find_constructor_in("_sha3","sha3_256") + + @property + def sha3_384(self): + return self._find_constructor_in("_sha3","sha3_384") + + @property + def sha3_512(self): + return self._find_constructor_in("_sha3","sha3_512") + + +def find_gil_minsize(modules_names, default=2048): + """Get the largest GIL_MINSIZE value for the given cryptographic modules. + + The valid module names are the following: + + - _hashlib + - _md5, _sha1, _sha2, _sha3, _blake2 + - _hmac + """ + sizes = [] + for module_name in modules_names: + try: + module = importlib.import_module(module_name) + except ImportError: + continue + sizes.append(getattr(module, '_GIL_MINSIZE', default)) + return max(sizes, default=default) diff --git a/Lib/test/support/hypothesis_helper.py b/Lib/test/support/hypothesis_helper.py index a99a4963ffe..6e9e168f63a 100644 --- a/Lib/test/support/hypothesis_helper.py +++ b/Lib/test/support/hypothesis_helper.py @@ -7,9 +7,10 @@ else: # Regrtest changes to use a tempdir as the working directory, so we have # to tell Hypothesis to use the original in order to persist the database. + from hypothesis.configuration import set_hypothesis_home_dir + from test.support import has_socket_support from test.support.os_helper import SAVEDCWD - from hypothesis.configuration import set_hypothesis_home_dir set_hypothesis_home_dir(os.path.join(SAVEDCWD, ".hypothesis")) diff --git a/Lib/test/support/i18n_helper.py b/Lib/test/support/i18n_helper.py index 2e304f29e8b..af97cdc9cb5 100644 --- a/Lib/test/support/i18n_helper.py +++ b/Lib/test/support/i18n_helper.py @@ -3,10 +3,10 @@ import sys import unittest from pathlib import Path + from test.support import REPO_ROOT, TEST_HOME_DIR, requires_subprocess from test.test_tools import skip_if_missing - pygettext = Path(REPO_ROOT) / 'Tools' / 'i18n' / 'pygettext.py' msgid_pattern = re.compile(r'msgid(.*?)(?:msgid_plural|msgctxt|msgstr)', diff --git a/Lib/test/support/import_helper.py b/Lib/test/support/import_helper.py index 2b91bdcf9cd..2d80b663dd5 100644 --- a/Lib/test/support/import_helper.py +++ b/Lib/test/support/import_helper.py @@ -1,14 +1,16 @@ -import contextlib import _imp +import contextlib import importlib +import importlib.machinery import importlib.util import os import shutil import sys +import textwrap import unittest import warnings -from .os_helper import unlink, temp_dir +from .os_helper import temp_dir, unlink @contextlib.contextmanager @@ -303,9 +305,138 @@ def ready_to_import(name=None, source=""): try: sys.path.insert(0, tempdir) yield name, path - sys.path.remove(tempdir) finally: + sys.path.remove(tempdir) if old_module is not None: sys.modules[name] = old_module else: sys.modules.pop(name, None) + + +def ensure_lazy_imports(imported_module, modules_to_block): + """Test that when imported_module is imported, none of the modules in + modules_to_block are imported as a side effect.""" + modules_to_block = frozenset(modules_to_block) + script = textwrap.dedent( + f""" + import sys + modules_to_block = {modules_to_block} + if unexpected := modules_to_block & sys.modules.keys(): + startup = ", ".join(unexpected) + raise AssertionError(f'unexpectedly imported at startup: {{startup}}') + + import {imported_module} + if unexpected := modules_to_block & sys.modules.keys(): + after = ", ".join(unexpected) + raise AssertionError(f'unexpectedly imported after importing {imported_module}: {{after}}') + """ + ) + from .script_helper import assert_python_ok + assert_python_ok("-S", "-c", script) + + +@contextlib.contextmanager +def module_restored(name): + """A context manager that restores a module to the original state.""" + missing = object() + orig = sys.modules.get(name, missing) + if orig is None: + mod = importlib.import_module(name) + else: + mod = type(sys)(name) + mod.__dict__.update(orig.__dict__) + sys.modules[name] = mod + try: + yield mod + finally: + if orig is missing: + sys.modules.pop(name, None) + else: + sys.modules[name] = orig + + +def create_module(name, loader=None, *, ispkg=False): + """Return a new, empty module.""" + spec = importlib.machinery.ModuleSpec( + name, + loader, + origin='', + is_package=ispkg, + ) + return importlib.util.module_from_spec(spec) + + +def _ensure_module(name, ispkg, addparent, clearnone): + try: + mod = orig = sys.modules[name] + except KeyError: + mod = orig = None + missing = True + else: + missing = False + if mod is not None: + # It was already imported. + return mod, orig, missing + # Otherwise, None means it was explicitly disabled. + + assert name != '__main__' + if not missing: + assert orig is None, (name, sys.modules[name]) + if not clearnone: + raise ModuleNotFoundError(name) + del sys.modules[name] + # Try normal import, then fall back to adding the module. + try: + mod = importlib.import_module(name) + except ModuleNotFoundError: + if addparent and not clearnone: + addparent = None + mod = _add_module(name, ispkg, addparent) + return mod, orig, missing + + +def _add_module(spec, ispkg, addparent): + if isinstance(spec, str): + name = spec + mod = create_module(name, ispkg=ispkg) + spec = mod.__spec__ + else: + name = spec.name + mod = importlib.util.module_from_spec(spec) + sys.modules[name] = mod + if addparent is not False and spec.parent: + _ensure_module(spec.parent, True, addparent, bool(addparent)) + return mod + + +def add_module(spec, *, parents=True): + """Return the module after creating it and adding it to sys.modules. + + If parents is True then also create any missing parents. + """ + return _add_module(spec, False, parents) + + +def add_package(spec, *, parents=True): + """Return the module after creating it and adding it to sys.modules. + + If parents is True then also create any missing parents. + """ + return _add_module(spec, True, parents) + + +def ensure_module_imported(name, *, clearnone=True): + """Return the corresponding module. + + If it was already imported then return that. Otherwise, try + importing it (optionally clear it first if None). If that fails + then create a new empty module. + + It can be helpful to combine this with ready_to_import() and/or + isolated_modules(). + """ + if sys.modules.get(name) is not None: + mod = sys.modules[name] + else: + mod, _, _ = _ensure_module(name, False, True, clearnone) + return mod diff --git a/Lib/test/support/interpreters/__init__.py b/Lib/test/support/interpreters/__init__.py deleted file mode 100644 index e067f259364..00000000000 --- a/Lib/test/support/interpreters/__init__.py +++ /dev/null @@ -1,258 +0,0 @@ -"""Subinterpreters High Level Module.""" - -import threading -import weakref -import _interpreters - -# aliases: -from _interpreters import ( - InterpreterError, InterpreterNotFoundError, NotShareableError, - is_shareable, -) - - -__all__ = [ - 'get_current', 'get_main', 'create', 'list_all', 'is_shareable', - 'Interpreter', - 'InterpreterError', 'InterpreterNotFoundError', 'ExecutionFailed', - 'NotShareableError', - 'create_queue', 'Queue', 'QueueEmpty', 'QueueFull', -] - - -_queuemod = None - -def __getattr__(name): - if name in ('Queue', 'QueueEmpty', 'QueueFull', 'create_queue'): - global create_queue, Queue, QueueEmpty, QueueFull - ns = globals() - from .queues import ( - create as create_queue, - Queue, QueueEmpty, QueueFull, - ) - return ns[name] - else: - raise AttributeError(name) - - -_EXEC_FAILURE_STR = """ -{superstr} - -Uncaught in the interpreter: - -{formatted} -""".strip() - -class ExecutionFailed(InterpreterError): - """An unhandled exception happened during execution. - - This is raised from Interpreter.exec() and Interpreter.call(). - """ - - def __init__(self, excinfo): - msg = excinfo.formatted - if not msg: - if excinfo.type and excinfo.msg: - msg = f'{excinfo.type.__name__}: {excinfo.msg}' - else: - msg = excinfo.type.__name__ or excinfo.msg - super().__init__(msg) - self.excinfo = excinfo - - def __str__(self): - try: - formatted = self.excinfo.errdisplay - except Exception: - return super().__str__() - else: - return _EXEC_FAILURE_STR.format( - superstr=super().__str__(), - formatted=formatted, - ) - - -def create(): - """Return a new (idle) Python interpreter.""" - id = _interpreters.create(reqrefs=True) - return Interpreter(id, _ownsref=True) - - -def list_all(): - """Return all existing interpreters.""" - return [Interpreter(id, _whence=whence) - for id, whence in _interpreters.list_all(require_ready=True)] - - -def get_current(): - """Return the currently running interpreter.""" - id, whence = _interpreters.get_current() - return Interpreter(id, _whence=whence) - - -def get_main(): - """Return the main interpreter.""" - id, whence = _interpreters.get_main() - assert whence == _interpreters.WHENCE_RUNTIME, repr(whence) - return Interpreter(id, _whence=whence) - - -_known = weakref.WeakValueDictionary() - -class Interpreter: - """A single Python interpreter. - - Attributes: - - "id" - the unique process-global ID number for the interpreter - "whence" - indicates where the interpreter was created - - If the interpreter wasn't created by this module - then any method that modifies the interpreter will fail, - i.e. .close(), .prepare_main(), .exec(), and .call() - """ - - _WHENCE_TO_STR = { - _interpreters.WHENCE_UNKNOWN: 'unknown', - _interpreters.WHENCE_RUNTIME: 'runtime init', - _interpreters.WHENCE_LEGACY_CAPI: 'legacy C-API', - _interpreters.WHENCE_CAPI: 'C-API', - _interpreters.WHENCE_XI: 'cross-interpreter C-API', - _interpreters.WHENCE_STDLIB: '_interpreters module', - } - - def __new__(cls, id, /, _whence=None, _ownsref=None): - # There is only one instance for any given ID. - if not isinstance(id, int): - raise TypeError(f'id must be an int, got {id!r}') - id = int(id) - if _whence is None: - if _ownsref: - _whence = _interpreters.WHENCE_STDLIB - else: - _whence = _interpreters.whence(id) - assert _whence in cls._WHENCE_TO_STR, repr(_whence) - if _ownsref is None: - _ownsref = (_whence == _interpreters.WHENCE_STDLIB) - try: - self = _known[id] - assert hasattr(self, '_ownsref') - except KeyError: - self = super().__new__(cls) - _known[id] = self - self._id = id - self._whence = _whence - self._ownsref = _ownsref - if _ownsref: - # This may raise InterpreterNotFoundError: - _interpreters.incref(id) - return self - - def __repr__(self): - return f'{type(self).__name__}({self.id})' - - def __hash__(self): - return hash(self._id) - - def __del__(self): - self._decref() - - # for pickling: - def __getnewargs__(self): - return (self._id,) - - # for pickling: - def __getstate__(self): - return None - - def _decref(self): - if not self._ownsref: - return - self._ownsref = False - try: - _interpreters.decref(self._id) - except InterpreterNotFoundError: - pass - - @property - def id(self): - return self._id - - @property - def whence(self): - return self._WHENCE_TO_STR[self._whence] - - def is_running(self): - """Return whether or not the identified interpreter is running.""" - return _interpreters.is_running(self._id) - - # Everything past here is available only to interpreters created by - # interpreters.create(). - - def close(self): - """Finalize and destroy the interpreter. - - Attempting to destroy the current interpreter results - in an InterpreterError. - """ - return _interpreters.destroy(self._id, restrict=True) - - def prepare_main(self, ns=None, /, **kwargs): - """Bind the given values into the interpreter's __main__. - - The values must be shareable. - """ - ns = dict(ns, **kwargs) if ns is not None else kwargs - _interpreters.set___main___attrs(self._id, ns, restrict=True) - - def exec(self, code, /): - """Run the given source code in the interpreter. - - This is essentially the same as calling the builtin "exec" - with this interpreter, using the __dict__ of its __main__ - module as both globals and locals. - - There is no return value. - - If the code raises an unhandled exception then an ExecutionFailed - exception is raised, which summarizes the unhandled exception. - The actual exception is discarded because objects cannot be - shared between interpreters. - - This blocks the current Python thread until done. During - that time, the previous interpreter is allowed to run - in other threads. - """ - excinfo = _interpreters.exec(self._id, code, restrict=True) - if excinfo is not None: - raise ExecutionFailed(excinfo) - - def call(self, callable, /): - """Call the object in the interpreter with given args/kwargs. - - Only functions that take no arguments and have no closure - are supported. - - The return value is discarded. - - If the callable raises an exception then the error display - (including full traceback) is send back between the interpreters - and an ExecutionFailed exception is raised, much like what - happens with Interpreter.exec(). - """ - # XXX Support args and kwargs. - # XXX Support arbitrary callables. - # XXX Support returning the return value (e.g. via pickle). - excinfo = _interpreters.call(self._id, callable, restrict=True) - if excinfo is not None: - raise ExecutionFailed(excinfo) - - def call_in_thread(self, callable, /): - """Return a new thread that calls the object in the interpreter. - - The return value and any raised exception are discarded. - """ - def task(): - self.call(callable) - t = threading.Thread(target=task) - t.start() - return t diff --git a/Lib/test/support/interpreters/_crossinterp.py b/Lib/test/support/interpreters/_crossinterp.py deleted file mode 100644 index 544e197ba4c..00000000000 --- a/Lib/test/support/interpreters/_crossinterp.py +++ /dev/null @@ -1,102 +0,0 @@ -"""Common code between queues and channels.""" - - -class ItemInterpreterDestroyed(Exception): - """Raised when trying to get an item whose interpreter was destroyed.""" - - -class classonly: - """A non-data descriptor that makes a value only visible on the class. - - This is like the "classmethod" builtin, but does not show up on - instances of the class. It may be used as a decorator. - """ - - def __init__(self, value): - self.value = value - self.getter = classmethod(value).__get__ - self.name = None - - def __set_name__(self, cls, name): - if self.name is not None: - raise TypeError('already used') - self.name = name - - def __get__(self, obj, cls): - if obj is not None: - raise AttributeError(self.name) - # called on the class - return self.getter(None, cls) - - -class UnboundItem: - """Represents a cross-interpreter item no longer bound to an interpreter. - - An item is unbound when the interpreter that added it to the - cross-interpreter container is destroyed. - """ - - __slots__ = () - - @classonly - def singleton(cls, kind, module, name='UNBOUND'): - doc = cls.__doc__.replace('cross-interpreter container', kind) - doc = doc.replace('cross-interpreter', kind) - subclass = type( - f'Unbound{kind.capitalize()}Item', - (cls,), - dict( - _MODULE=module, - _NAME=name, - __doc__=doc, - ), - ) - return object.__new__(subclass) - - _MODULE = __name__ - _NAME = 'UNBOUND' - - def __new__(cls): - raise Exception(f'use {cls._MODULE}.{cls._NAME}') - - def __repr__(self): - return f'{self._MODULE}.{self._NAME}' -# return f'interpreters.queues.UNBOUND' - - -UNBOUND = object.__new__(UnboundItem) -UNBOUND_ERROR = object() -UNBOUND_REMOVE = object() - -_UNBOUND_CONSTANT_TO_FLAG = { - UNBOUND_REMOVE: 1, - UNBOUND_ERROR: 2, - UNBOUND: 3, -} -_UNBOUND_FLAG_TO_CONSTANT = {v: k - for k, v in _UNBOUND_CONSTANT_TO_FLAG.items()} - - -def serialize_unbound(unbound): - op = unbound - try: - flag = _UNBOUND_CONSTANT_TO_FLAG[op] - except KeyError: - raise NotImplementedError(f'unsupported unbound replacement op {op!r}') - return flag, - - -def resolve_unbound(flag, exctype_destroyed): - try: - op = _UNBOUND_FLAG_TO_CONSTANT[flag] - except KeyError: - raise NotImplementedError(f'unsupported unbound replacement op {flag!r}') - if op is UNBOUND_REMOVE: - # "remove" not possible here - raise NotImplementedError - elif op is UNBOUND_ERROR: - raise exctype_destroyed("item's original interpreter destroyed") - elif op is UNBOUND: - return UNBOUND - else: - raise NotImplementedError(repr(op)) diff --git a/Lib/test/support/interpreters/queues.py b/Lib/test/support/interpreters/queues.py deleted file mode 100644 index deb8e8613af..00000000000 --- a/Lib/test/support/interpreters/queues.py +++ /dev/null @@ -1,313 +0,0 @@ -"""Cross-interpreter Queues High Level Module.""" - -import pickle -import queue -import time -import weakref -import _interpqueues as _queues -from . import _crossinterp - -# aliases: -from _interpqueues import ( - QueueError, QueueNotFoundError, -) -from ._crossinterp import ( - UNBOUND_ERROR, UNBOUND_REMOVE, -) - -__all__ = [ - 'UNBOUND', 'UNBOUND_ERROR', 'UNBOUND_REMOVE', - 'create', 'list_all', - 'Queue', - 'QueueError', 'QueueNotFoundError', 'QueueEmpty', 'QueueFull', - 'ItemInterpreterDestroyed', -] - - -class QueueEmpty(QueueError, queue.Empty): - """Raised from get_nowait() when the queue is empty. - - It is also raised from get() if it times out. - """ - - -class QueueFull(QueueError, queue.Full): - """Raised from put_nowait() when the queue is full. - - It is also raised from put() if it times out. - """ - - -class ItemInterpreterDestroyed(QueueError, - _crossinterp.ItemInterpreterDestroyed): - """Raised from get() and get_nowait().""" - - -_SHARED_ONLY = 0 -_PICKLED = 1 - - -UNBOUND = _crossinterp.UnboundItem.singleton('queue', __name__) - - -def _serialize_unbound(unbound): - if unbound is UNBOUND: - unbound = _crossinterp.UNBOUND - return _crossinterp.serialize_unbound(unbound) - - -def _resolve_unbound(flag): - resolved = _crossinterp.resolve_unbound(flag, ItemInterpreterDestroyed) - if resolved is _crossinterp.UNBOUND: - resolved = UNBOUND - return resolved - - -def create(maxsize=0, *, syncobj=False, unbounditems=UNBOUND): - """Return a new cross-interpreter queue. - - The queue may be used to pass data safely between interpreters. - - "syncobj" sets the default for Queue.put() - and Queue.put_nowait(). - - "unbounditems" likewise sets the default. See Queue.put() for - supported values. The default value is UNBOUND, which replaces - the unbound item. - """ - fmt = _SHARED_ONLY if syncobj else _PICKLED - unbound = _serialize_unbound(unbounditems) - unboundop, = unbound - qid = _queues.create(maxsize, fmt, unboundop) - return Queue(qid, _fmt=fmt, _unbound=unbound) - - -def list_all(): - """Return a list of all open queues.""" - return [Queue(qid, _fmt=fmt, _unbound=(unboundop,)) - for qid, fmt, unboundop in _queues.list_all()] - - -_known_queues = weakref.WeakValueDictionary() - -class Queue: - """A cross-interpreter queue.""" - - def __new__(cls, id, /, *, _fmt=None, _unbound=None): - # There is only one instance for any given ID. - if isinstance(id, int): - id = int(id) - else: - raise TypeError(f'id must be an int, got {id!r}') - if _fmt is None: - if _unbound is None: - _fmt, op = _queues.get_queue_defaults(id) - _unbound = (op,) - else: - _fmt, _ = _queues.get_queue_defaults(id) - elif _unbound is None: - _, op = _queues.get_queue_defaults(id) - _unbound = (op,) - try: - self = _known_queues[id] - except KeyError: - self = super().__new__(cls) - self._id = id - self._fmt = _fmt - self._unbound = _unbound - _known_queues[id] = self - _queues.bind(id) - return self - - def __del__(self): - try: - _queues.release(self._id) - except QueueNotFoundError: - pass - try: - del _known_queues[self._id] - except KeyError: - pass - - def __repr__(self): - return f'{type(self).__name__}({self.id})' - - def __hash__(self): - return hash(self._id) - - # for pickling: - def __getnewargs__(self): - return (self._id,) - - # for pickling: - def __getstate__(self): - return None - - @property - def id(self): - return self._id - - @property - def maxsize(self): - try: - return self._maxsize - except AttributeError: - self._maxsize = _queues.get_maxsize(self._id) - return self._maxsize - - def empty(self): - return self.qsize() == 0 - - def full(self): - return _queues.is_full(self._id) - - def qsize(self): - return _queues.get_count(self._id) - - def put(self, obj, timeout=None, *, - syncobj=None, - unbound=None, - _delay=10 / 1000, # 10 milliseconds - ): - """Add the object to the queue. - - This blocks while the queue is full. - - If "syncobj" is None (the default) then it uses the - queue's default, set with create_queue(). - - If "syncobj" is false then all objects are supported, - at the expense of worse performance. - - If "syncobj" is true then the object must be "shareable". - Examples of "shareable" objects include the builtin singletons, - str, and memoryview. One benefit is that such objects are - passed through the queue efficiently. - - The key difference, though, is conceptual: the corresponding - object returned from Queue.get() will be strictly equivalent - to the given obj. In other words, the two objects will be - effectively indistinguishable from each other, even if the - object is mutable. The received object may actually be the - same object, or a copy (immutable values only), or a proxy. - Regardless, the received object should be treated as though - the original has been shared directly, whether or not it - actually is. That's a slightly different and stronger promise - than just (initial) equality, which is all "syncobj=False" - can promise. - - "unbound" controls the behavior of Queue.get() for the given - object if the current interpreter (calling put()) is later - destroyed. - - If "unbound" is None (the default) then it uses the - queue's default, set with create_queue(), - which is usually UNBOUND. - - If "unbound" is UNBOUND_ERROR then get() will raise an - ItemInterpreterDestroyed exception if the original interpreter - has been destroyed. This does not otherwise affect the queue; - the next call to put() will work like normal, returning the next - item in the queue. - - If "unbound" is UNBOUND_REMOVE then the item will be removed - from the queue as soon as the original interpreter is destroyed. - Be aware that this will introduce an imbalance between put() - and get() calls. - - If "unbound" is UNBOUND then it is returned by get() in place - of the unbound item. - """ - if syncobj is None: - fmt = self._fmt - else: - fmt = _SHARED_ONLY if syncobj else _PICKLED - if unbound is None: - unboundop, = self._unbound - else: - unboundop, = _serialize_unbound(unbound) - if timeout is not None: - timeout = int(timeout) - if timeout < 0: - raise ValueError(f'timeout value must be non-negative') - end = time.time() + timeout - if fmt is _PICKLED: - obj = pickle.dumps(obj) - while True: - try: - _queues.put(self._id, obj, fmt, unboundop) - except QueueFull as exc: - if timeout is not None and time.time() >= end: - raise # re-raise - time.sleep(_delay) - else: - break - - def put_nowait(self, obj, *, syncobj=None, unbound=None): - if syncobj is None: - fmt = self._fmt - else: - fmt = _SHARED_ONLY if syncobj else _PICKLED - if unbound is None: - unboundop, = self._unbound - else: - unboundop, = _serialize_unbound(unbound) - if fmt is _PICKLED: - obj = pickle.dumps(obj) - _queues.put(self._id, obj, fmt, unboundop) - - def get(self, timeout=None, *, - _delay=10 / 1000, # 10 milliseconds - ): - """Return the next object from the queue. - - This blocks while the queue is empty. - - If the next item's original interpreter has been destroyed - then the "next object" is determined by the value of the - "unbound" argument to put(). - """ - if timeout is not None: - timeout = int(timeout) - if timeout < 0: - raise ValueError(f'timeout value must be non-negative') - end = time.time() + timeout - while True: - try: - obj, fmt, unboundop = _queues.get(self._id) - except QueueEmpty as exc: - if timeout is not None and time.time() >= end: - raise # re-raise - time.sleep(_delay) - else: - break - if unboundop is not None: - assert obj is None, repr(obj) - return _resolve_unbound(unboundop) - if fmt == _PICKLED: - obj = pickle.loads(obj) - else: - assert fmt == _SHARED_ONLY - return obj - - def get_nowait(self): - """Return the next object from the channel. - - If the queue is empty then raise QueueEmpty. Otherwise this - is the same as get(). - """ - try: - obj, fmt, unboundop = _queues.get(self._id) - except QueueEmpty as exc: - raise # re-raise - if unboundop is not None: - assert obj is None, repr(obj) - return _resolve_unbound(unboundop) - if fmt == _PICKLED: - obj = pickle.loads(obj) - else: - assert fmt == _SHARED_ONLY - return obj - - -_queues._register_heap_types(Queue, QueueEmpty, QueueFull) diff --git a/Lib/test/support/logging_helper.py b/Lib/test/support/logging_helper.py index 12fcca4f0f0..db556c7f5ad 100644 --- a/Lib/test/support/logging_helper.py +++ b/Lib/test/support/logging_helper.py @@ -1,5 +1,6 @@ import logging.handlers + class TestHandler(logging.handlers.BufferingHandler): def __init__(self, matcher): # BufferingHandler takes a "capacity" argument diff --git a/Lib/test/support/os_helper.py b/Lib/test/support/os_helper.py index 26c467a7ad2..d3d6fa632f9 100644 --- a/Lib/test/support/os_helper.py +++ b/Lib/test/support/os_helper.py @@ -13,7 +13,6 @@ from test import support - # Filename used for testing TESTFN_ASCII = '@test' @@ -295,6 +294,33 @@ def skip_unless_working_chmod(test): return test if ok else unittest.skip(msg)(test) +@contextlib.contextmanager +def save_mode(path, *, quiet=False): + """Context manager that restores the mode (permissions) of *path* on exit. + + Arguments: + + path: Path of the file to restore the mode of. + + quiet: if False (the default), the context manager raises an exception + on error. Otherwise, it issues only a warning and keeps the current + working directory the same. + + """ + saved_mode = os.stat(path) + try: + yield + finally: + try: + os.chmod(path, saved_mode.st_mode) + except OSError as exc: + if not quiet: + raise + warnings.warn(f'tests may fail, unable to restore the mode of ' + f'{path!r} to {saved_mode.st_mode}: {exc}', + RuntimeWarning, stacklevel=3) + + # Check whether the current effective user has the capability to override # DAC (discretionary access control). Typically user root is able to # bypass file read, write, and execute permission checks. The capability diff --git a/Lib/test/support/pty_helper.py b/Lib/test/support/pty_helper.py index 6587fd40333..7e1ae9e59b8 100644 --- a/Lib/test/support/pty_helper.py +++ b/Lib/test/support/pty_helper.py @@ -10,6 +10,7 @@ from test.support.import_helper import import_module + def run_pty(script, input=b"dummy input\r", env=None): pty = import_module('pty') output = bytearray() diff --git a/Lib/test/support/script_helper.py b/Lib/test/support/script_helper.py index 04458077d51..a338f484449 100644 --- a/Lib/test/support/script_helper.py +++ b/Lib/test/support/script_helper.py @@ -3,17 +3,16 @@ import collections import importlib -import sys import os import os.path -import subprocess import py_compile - +import subprocess +import sys from importlib.util import source_from_cache + from test import support from test.support.import_helper import make_legacy_pyc - # Cached result of the expensive test performed in the function below. __cached_interp_requires_environment = None @@ -70,23 +69,25 @@ def fail(self, cmd_line): out = b'(... truncated stdout ...)' + out[-maxlen:] if len(err) > maxlen: err = b'(... truncated stderr ...)' + err[-maxlen:] - out = out.decode('ascii', 'replace').rstrip() - err = err.decode('ascii', 'replace').rstrip() - raise AssertionError("Process return code is %d\n" - "command line: %r\n" - "\n" - "stdout:\n" - "---\n" - "%s\n" - "---\n" - "\n" - "stderr:\n" - "---\n" - "%s\n" - "---" - % (self.rc, cmd_line, - out, - err)) + out = out.decode('utf8', 'replace').rstrip() + err = err.decode('utf8', 'replace').rstrip() + + exitcode = self.rc + signame = support.get_signal_name(exitcode) + if signame: + exitcode = f"{exitcode} ({signame})" + raise AssertionError(f"Process return code is {exitcode}\n" + f"command line: {cmd_line!r}\n" + f"\n" + f"stdout:\n" + f"---\n" + f"{out}\n" + f"---\n" + f"\n" + f"stderr:\n" + f"---\n" + f"{err}\n" + f"---") # Executing the interpreter in a subprocess diff --git a/Lib/test/support/smtpd.py b/Lib/test/support/smtpd.py old mode 100644 new mode 100755 index 6052232ec2b..cf333aaf6b0 --- a/Lib/test/support/smtpd.py +++ b/Lib/test/support/smtpd.py @@ -7,7 +7,7 @@ --nosetuid -n - This program generally tries to setuid `nobody', unless this flag is + This program generally tries to setuid 'nobody', unless this flag is set. The setuid call will fail if this program is not run as root (in which case, use this flag). @@ -17,7 +17,7 @@ --class classname -c classname - Use `classname' as the concrete SMTP proxy class. Uses `PureProxy' by + Use 'classname' as the concrete SMTP proxy class. Uses 'PureProxy' by default. --size limit @@ -39,8 +39,8 @@ Version: %(__version__)s -If localhost is not given then `localhost' is used, and if localport is not -given then 8025 is used. If remotehost is not given then `localhost' is used, +If localhost is not given then 'localhost' is used, and if localport is not +given then 8025 is used. If remotehost is not given then 'localhost' is used, and if remoteport is not given, then 25 is used. """ @@ -70,16 +70,17 @@ # - Handle more ESMTP extensions # - handle error codes from the backend smtpd -import sys -import os +import collections import errno import getopt -import time +import os import socket -import collections -from test.support import asyncore, asynchat -from warnings import warn +import sys +import time from email._header_value_parser import get_addr_spec, get_angle_addr +from warnings import warn + +from test.support import asynchat, asyncore __all__ = [ "SMTPChannel", "SMTPServer", "DebuggingServer", "PureProxy", @@ -633,7 +634,8 @@ def __init__(self, localaddr, remoteaddr, " be set to True at the same time") asyncore.dispatcher.__init__(self, map=map) try: - gai_results = socket.getaddrinfo(*localaddr, + family = 0 if socket.has_ipv6 else socket.AF_INET + gai_results = socket.getaddrinfo(*localaddr, family=family, type=socket.SOCK_STREAM) self.create_socket(gai_results[0][0], gai_results[0][1]) # try to re-use a server port if possible @@ -672,9 +674,9 @@ def process_message(self, peer, mailfrom, rcpttos, data, **kwargs): message to. data is a string containing the entire full text of the message, - headers (if supplied) and all. It has been `de-transparencied' + headers (if supplied) and all. It has been 'de-transparencied' according to RFC 821, Section 4.5.2. In other words, a line - containing a `.' followed by other text has had the leading dot + containing a '.' followed by other text has had the leading dot removed. kwargs is a dictionary containing additional information. It is @@ -685,7 +687,7 @@ def process_message(self, peer, mailfrom, rcpttos, data, **kwargs): ['BODY=8BITMIME', 'SMTPUTF8']. 'rcpt_options': same, for the rcpt command. - This function should return None for a normal `250 Ok' response; + This function should return None for a normal '250 Ok' response; otherwise, it should return the desired response string in RFC 821 format. diff --git a/Lib/test/support/socket_helper.py b/Lib/test/support/socket_helper.py index a41e487f3e4..655ffbea0db 100644 --- a/Lib/test/support/socket_helper.py +++ b/Lib/test/support/socket_helper.py @@ -2,8 +2,8 @@ import errno import os.path import socket -import sys import subprocess +import sys import tempfile import unittest diff --git a/Lib/test/support/strace_helper.py b/Lib/test/support/strace_helper.py new file mode 100644 index 00000000000..abc93dee2ce --- /dev/null +++ b/Lib/test/support/strace_helper.py @@ -0,0 +1,210 @@ +import os +import re +import sys +import textwrap +import unittest +from dataclasses import dataclass +from functools import cache + +from test import support +from test.support.script_helper import run_python_until_end + +_strace_binary = "/usr/bin/strace" +_syscall_regex = re.compile( + r"(?P[^(]*)\((?P[^)]*)\)\s*[=]\s*(?P.+)") +_returncode_regex = re.compile( + br"\+\+\+ exited with (?P\d+) \+\+\+") + + +@dataclass +class StraceEvent: + syscall: str + args: list[str] + returncode: str + + +@dataclass +class StraceResult: + strace_returncode: int + python_returncode: int + + """The event messages generated by strace. This is very similar to the + stderr strace produces with returncode marker section removed.""" + event_bytes: bytes + stdout: bytes + stderr: bytes + + def events(self): + """Parse event_bytes data into system calls for easier processing. + + This assumes the program under inspection doesn't print any non-utf8 + strings which would mix into the strace output.""" + decoded_events = self.event_bytes.decode('utf-8', 'surrogateescape') + matches = [ + _syscall_regex.match(event) + for event in decoded_events.splitlines() + ] + return [ + StraceEvent(match["syscall"], + [arg.strip() for arg in (match["args"].split(","))], + match["returncode"]) for match in matches if match + ] + + def sections(self): + """Find all "MARK " writes and use them to make groups of events. + + This is useful to avoid variable / overhead events, like those at + interpreter startup or when opening a file so a test can verify just + the small case under study.""" + current_section = "__startup" + sections = {current_section: []} + for event in self.events(): + if event.syscall == 'write' and len( + event.args) > 2 and event.args[1].startswith("\"MARK "): + # Found a new section, don't include the write in the section + # but all events until next mark should be in that section + current_section = event.args[1].split( + " ", 1)[1].removesuffix('\\n"') + if current_section not in sections: + sections[current_section] = list() + else: + sections[current_section].append(event) + + return sections + +def _filter_memory_call(call): + # mmap can operate on a fd or "MAP_ANONYMOUS" which gives a block of memory. + # Ignore "MAP_ANONYMOUS + the "MAP_ANON" alias. + if call.syscall == "mmap" and "MAP_ANON" in call.args[3]: + return True + + if call.syscall in ("munmap", "mprotect"): + return True + + return False + + +def filter_memory(syscalls): + """Filter out memory allocation calls from File I/O calls. + + Some calls (mmap, munmap, etc) can be used on files or to just get a block + of memory. Use this function to filter out the memory related calls from + other calls.""" + + return [call for call in syscalls if not _filter_memory_call(call)] + + +@support.requires_subprocess() +def strace_python(code, strace_flags, check=True): + """Run strace and return the trace. + + Sets strace_returncode and python_returncode to `-1` on error.""" + res = None + + def _make_error(reason, details): + return StraceResult( + strace_returncode=-1, + python_returncode=-1, + event_bytes= f"error({reason},details={details!r}) = -1".encode('utf-8'), + stdout=res.out if res else b"", + stderr=res.err if res else b"") + + # Run strace, and get out the raw text + try: + res, cmd_line = run_python_until_end( + "-c", + textwrap.dedent(code), + __run_using_command=[_strace_binary] + strace_flags, + ) + except OSError as err: + return _make_error("Caught OSError", err) + + if check and res.rc: + res.fail(cmd_line) + + # Get out program returncode + stripped = res.err.strip() + output = stripped.rsplit(b"\n", 1) + if len(output) != 2: + return _make_error("Expected strace events and exit code line", + stripped[-50:]) + + returncode_match = _returncode_regex.match(output[1]) + if not returncode_match: + return _make_error("Expected to find returncode in last line.", + output[1][:50]) + + python_returncode = int(returncode_match["returncode"]) + if check and python_returncode: + res.fail(cmd_line) + + return StraceResult(strace_returncode=res.rc, + python_returncode=python_returncode, + event_bytes=output[0], + stdout=res.out, + stderr=res.err) + + +def get_events(code, strace_flags, prelude, cleanup): + # NOTE: The flush is currently required to prevent the prints from getting + # buffered and done all at once at exit + prelude = textwrap.dedent(prelude) + code = textwrap.dedent(code) + cleanup = textwrap.dedent(cleanup) + to_run = f""" +print("MARK prelude", flush=True) +{prelude} +print("MARK code", flush=True) +{code} +print("MARK cleanup", flush=True) +{cleanup} +print("MARK __shutdown", flush=True) + """ + trace = strace_python(to_run, strace_flags) + all_sections = trace.sections() + return all_sections['code'] + + +def get_syscalls(code, strace_flags, prelude="", cleanup="", + ignore_memory=True): + """Get the syscalls which a given chunk of python code generates""" + events = get_events(code, strace_flags, prelude=prelude, cleanup=cleanup) + + if ignore_memory: + events = filter_memory(events) + + return [ev.syscall for ev in events] + + +# Moderately expensive (spawns a subprocess), so share results when possible. +@cache +def _can_strace(): + res = strace_python("import sys; sys.exit(0)", + # --trace option needs strace 5.5 (gh-133741) + ["--trace=%process"], + check=False) + if res.strace_returncode == 0 and res.python_returncode == 0: + assert res.events(), "Should have parsed multiple calls" + return True + return False + + +def requires_strace(): + if sys.platform != "linux": + return unittest.skip("Linux only, requires strace.") + + if "LD_PRELOAD" in os.environ: + # Distribution packaging (ex. Debian `fakeroot` and Gentoo `sandbox`) + # use LD_PRELOAD to intercept system calls, which changes the overall + # set of system calls which breaks tests expecting a specific set of + # system calls). + return unittest.skip("Not supported when LD_PRELOAD is intercepting system calls.") + + if support.check_sanitizer(address=True, memory=True): + return unittest.skip("LeakSanitizer does not work under ptrace (strace, gdb, etc)") + + return unittest.skipUnless(_can_strace(), "Requires working strace") + + +__all__ = ["filter_memory", "get_events", "get_syscalls", "requires_strace", + "strace_python", "StraceEvent", "StraceResult"] diff --git a/Lib/test/support/testcase.py b/Lib/test/support/testcase.py index fd32457d146..e617b19b6ac 100644 --- a/Lib/test/support/testcase.py +++ b/Lib/test/support/testcase.py @@ -1,6 +1,7 @@ from math import copysign, isnan +# XXX: RUSTPYTHON: removed in 3.14 class ExtraAssertions: def assertIsSubclass(self, cls, superclass, msg=None): diff --git a/Lib/test/support/threading_helper.py b/Lib/test/support/threading_helper.py index afa25a76f63..9b2b8f2dff0 100644 --- a/Lib/test/support/threading_helper.py +++ b/Lib/test/support/threading_helper.py @@ -8,7 +8,6 @@ from test import support - #======================================================================= # Threading support to prevent reporting refleaks when running regrtest.py -R @@ -248,3 +247,27 @@ def requires_working_threading(*, module=False): raise unittest.SkipTest(msg) else: return unittest.skipUnless(can_start_thread, msg) + + +def run_concurrently(worker_func, nthreads, args=(), kwargs={}): + """ + Run the worker function concurrently in multiple threads. + """ + barrier = threading.Barrier(nthreads) + + def wrapper_func(*args, **kwargs): + # Wait for all threads to reach this point before proceeding. + barrier.wait() + worker_func(*args, **kwargs) + + with catch_threading_exception() as cm: + workers = [ + threading.Thread(target=wrapper_func, args=args, kwargs=kwargs) + for _ in range(nthreads) + ] + with start_threads(workers): + pass + + # If a worker thread raises an exception, re-raise it. + if cm.exc_value is not None: + raise cm.exc_value diff --git a/Lib/test/support/venv.py b/Lib/test/support/venv.py index 78e6a51ec18..b60f6097e65 100644 --- a/Lib/test/support/venv.py +++ b/Lib/test/support/venv.py @@ -1,8 +1,8 @@ import contextlib import logging import os -import subprocess import shlex +import subprocess import sys import sysconfig import tempfile @@ -68,3 +68,14 @@ def run(self, *args, **subprocess_args): raise else: return result + + +class VirtualEnvironmentMixin: + def venv(self, name=None, **venv_create_args): + venv_name = self.id() + if name: + venv_name += f'-{name}' + return VirtualEnvironment.from_tmpdir( + prefix=f'{venv_name}-venv-', + **venv_create_args, + ) diff --git a/Lib/test/support/warnings_helper.py b/Lib/test/support/warnings_helper.py index c1bf0562300..5f6f14afd74 100644 --- a/Lib/test/support/warnings_helper.py +++ b/Lib/test/support/warnings_helper.py @@ -23,8 +23,7 @@ def check_syntax_warning(testcase, statement, errtext='', testcase.assertEqual(len(warns), 1, warns) warn, = warns - testcase.assertTrue(issubclass(warn.category, SyntaxWarning), - warn.category) + testcase.assertIsSubclass(warn.category, SyntaxWarning) if errtext: testcase.assertRegex(str(warn.message), errtext) testcase.assertEqual(warn.filename, '') @@ -160,11 +159,12 @@ def _filterwarnings(filters, quiet=False): registry = frame.f_globals.get('__warningregistry__') if registry: registry.clear() - with warnings.catch_warnings(record=True) as w: - # Set filter "always" to record all warnings. Because - # test_warnings swap the module, we need to look up in - # the sys.modules dictionary. - sys.modules['warnings'].simplefilter("always") + # Because test_warnings swap the module, we need to look up in the + # sys.modules dictionary. + wmod = sys.modules['warnings'] + with wmod.catch_warnings(record=True) as w: + # Set filter "always" to record all warnings. + wmod.simplefilter("always") yield WarningsRecorder(w) # Filter the recorded warnings reraise = list(w) diff --git a/Lib/test/test__colorize.py b/Lib/test/test__colorize.py index b2f0bb1386f..026277267e0 100644 --- a/Lib/test/test__colorize.py +++ b/Lib/test/test__colorize.py @@ -1,4 +1,5 @@ import contextlib +import dataclasses import io import sys import unittest @@ -21,6 +22,41 @@ def supports_virtual_terminal(): return contextlib.nullcontext() +class TestTheme(unittest.TestCase): + + def test_attributes(self): + # only theme configurations attributes by default + for field in dataclasses.fields(_colorize.Theme): + with self.subTest(field.name): + self.assertIsSubclass(field.type, _colorize.ThemeSection) + self.assertIsNotNone(field.default_factory) + + def test_copy_with(self): + theme = _colorize.Theme() + + copy = theme.copy_with() + self.assertEqual(theme, copy) + + unittest_no_colors = _colorize.Unittest.no_colors() + copy = theme.copy_with(unittest=unittest_no_colors) + self.assertEqual(copy.argparse, theme.argparse) + self.assertEqual(copy.syntax, theme.syntax) + self.assertEqual(copy.traceback, theme.traceback) + self.assertEqual(copy.unittest, unittest_no_colors) + + def test_no_colors(self): + # idempotence test + theme_no_colors = _colorize.Theme().no_colors() + theme_no_colors_no_colors = theme_no_colors.no_colors() + self.assertEqual(theme_no_colors, theme_no_colors_no_colors) + + # attributes check + for section in dataclasses.fields(_colorize.Theme): + with self.subTest(section.name): + section_theme = getattr(theme_no_colors, section.name) + self.assertEqual(section_theme, section.type.no_colors()) + + class TestColorizeFunction(unittest.TestCase): def test_colorized_detection_checks_for_environment_variables(self): def check(env, fallback, expected): @@ -129,6 +165,17 @@ def test_colorized_detection_checks_for_file(self): file.isatty.return_value = False self.assertEqual(_colorize.can_colorize(file=file), False) + # The documentation for file.fileno says: + # > An OSError is raised if the IO object does not use a file descriptor. + # gh-141570: Check OSError is caught and handled + with unittest.mock.patch("os.isatty", side_effect=ZeroDivisionError): + file = unittest.mock.MagicMock() + file.fileno.side_effect = OSError + file.isatty.return_value = True + self.assertEqual(_colorize.can_colorize(file=file), True) + file.isatty.return_value = False + self.assertEqual(_colorize.can_colorize(file=file), False) + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test__locale.py b/Lib/test/test__locale.py new file mode 100644 index 00000000000..11b2c9545a1 --- /dev/null +++ b/Lib/test/test__locale.py @@ -0,0 +1,300 @@ +from _locale import (setlocale, LC_ALL, LC_CTYPE, LC_NUMERIC, LC_TIME, localeconv, Error) +try: + from _locale import (RADIXCHAR, THOUSEP, nl_langinfo) +except ImportError: + nl_langinfo = None + +import locale +import sys +import unittest +from platform import uname + +from test import support + +if uname().system == "Darwin": + maj, min, mic = [int(part) for part in uname().release.split(".")] + if (maj, min, mic) < (8, 0, 0): + raise unittest.SkipTest("locale support broken for OS X < 10.4") + +candidate_locales = ['es_UY', 'fr_FR', 'fi_FI', 'es_CO', 'pt_PT', 'it_IT', + 'et_EE', 'es_PY', 'no_NO', 'nl_NL', 'lv_LV', 'el_GR', 'be_BY', 'fr_BE', + 'ro_RO', 'ru_UA', 'ru_RU', 'es_VE', 'ca_ES', 'se_NO', 'es_EC', 'id_ID', + 'ka_GE', 'es_CL', 'wa_BE', 'hu_HU', 'lt_LT', 'sl_SI', 'hr_HR', 'es_AR', + 'es_ES', 'oc_FR', 'gl_ES', 'bg_BG', 'is_IS', 'mk_MK', 'de_AT', 'pt_BR', + 'da_DK', 'nn_NO', 'cs_CZ', 'de_LU', 'es_BO', 'sq_AL', 'sk_SK', 'fr_CH', + 'de_DE', 'sr_YU', 'br_FR', 'nl_BE', 'sv_FI', 'pl_PL', 'fr_CA', 'fo_FO', + 'bs_BA', 'fr_LU', 'kl_GL', 'fa_IR', 'de_BE', 'sv_SE', 'it_CH', 'uk_UA', + 'eu_ES', 'vi_VN', 'af_ZA', 'nb_NO', 'en_DK', 'tg_TJ', 'ps_AF', 'en_US', + 'fr_FR.ISO8859-1', 'fr_FR.UTF-8', 'fr_FR.ISO8859-15@euro', + 'ru_RU.KOI8-R', 'ko_KR.eucKR', + 'ja_JP.UTF-8', 'lzh_TW.UTF-8', 'my_MM.UTF-8', 'or_IN.UTF-8', 'shn_MM.UTF-8', + 'ar_AE.UTF-8', 'bn_IN.UTF-8', 'mr_IN.UTF-8', 'th_TH.TIS620', +] + +def setUpModule(): + global candidate_locales + # Issue #13441: Skip some locales (e.g. cs_CZ and hu_HU) on Solaris to + # workaround a mbstowcs() bug. For example, on Solaris, the hu_HU locale uses + # the locale encoding ISO-8859-2, the thousands separator is b'\xA0' and it is + # decoded as U+30000020 (an invalid character) by mbstowcs(). + if sys.platform == 'sunos5': + old_locale = locale.setlocale(locale.LC_ALL) + try: + locales = [] + for loc in candidate_locales: + try: + locale.setlocale(locale.LC_ALL, loc) + except Error: + continue + encoding = locale.getencoding() + try: + localeconv() + except Exception as err: + print("WARNING: Skip locale %s (encoding %s): [%s] %s" + % (loc, encoding, type(err), err)) + else: + locales.append(loc) + candidate_locales = locales + finally: + locale.setlocale(locale.LC_ALL, old_locale) + + # Workaround for MSVC6(debug) crash bug + if "MSC v.1200" in sys.version: + def accept(loc): + a = loc.split(".") + return not(len(a) == 2 and len(a[-1]) >= 9) + candidate_locales = [loc for loc in candidate_locales if accept(loc)] + +# List known locale values to test against when available. +# Dict formatted as `` : (, )``. If a +# value is not known, use '' . +known_numerics = { + 'en_US': ('.', ','), + 'de_DE' : (',', '.'), + # The French thousands separator may be a breaking or non-breaking space + # depending on the platform, so do not test it + 'fr_FR' : (',', ''), + 'ps_AF': ('\u066b', '\u066c'), +} + +known_alt_digits = { + 'C': (0, {}), + 'en_US': (0, {}), + 'fa_IR': (100, {0: '\u06f0\u06f0', 10: '\u06f1\u06f0', 99: '\u06f9\u06f9'}), + 'ja_JP': (100, {1: '\u4e00', 10: '\u5341', 99: '\u4e5d\u5341\u4e5d'}), + 'lzh_TW': (32, {0: '\u3007', 10: '\u5341', 31: '\u5345\u4e00'}), + 'my_MM': (100, {0: '\u1040\u1040', 10: '\u1041\u1040', 99: '\u1049\u1049'}), + 'or_IN': (100, {0: '\u0b66', 10: '\u0b67\u0b66', 99: '\u0b6f\u0b6f'}), + 'shn_MM': (100, {0: '\u1090\u1090', 10: '\u1091\u1090', 99: '\u1099\u1099'}), + 'ar_AE': (100, {0: '\u0660', 10: '\u0661\u0660', 99: '\u0669\u0669'}), + 'bn_IN': (100, {0: '\u09e6', 10: '\u09e7\u09e6', 99: '\u09ef\u09ef'}), +} + +known_era = { + 'C': (0, ''), + 'en_US': (0, ''), + 'ja_JP': (11, '+:1:2019/05/01:2019/12/31:令和:%EC元年'), + 'zh_TW': (3, '+:1:1912/01/01:1912/12/31:民國:%EC元年'), + 'th_TW': (1, '+:1:-543/01/01:+*:พ.ศ.:%EC %Ey'), +} + +if sys.platform == 'win32': + # ps_AF doesn't work on Windows: see bpo-38324 (msg361830) + del known_numerics['ps_AF'] + +if sys.platform == 'sunos5': + # On Solaris, Japanese ERAs start with the year 1927, + # and thus there's less of them. + known_era['ja_JP'] = (5, '+:1:2019/05/01:2019/12/31:令和:%EC元年') + +class _LocaleTests(unittest.TestCase): + + def setUp(self): + self.oldlocale = setlocale(LC_ALL) + + def tearDown(self): + setlocale(LC_ALL, self.oldlocale) + + # Want to know what value was calculated, what it was compared against, + # what function was used for the calculation, what type of data was used, + # the locale that was supposedly set, and the actual locale that is set. + lc_numeric_err_msg = "%s != %s (%s for %s; set to %s, using %s)" + + def numeric_tester(self, calc_type, calc_value, data_type, used_locale): + """Compare calculation against known value, if available""" + try: + set_locale = setlocale(LC_NUMERIC) + except Error: + set_locale = "" + known_value = known_numerics.get(used_locale, + ('', ''))[data_type == 'thousands_sep'] + if known_value and calc_value: + self.assertEqual(calc_value, known_value, + self.lc_numeric_err_msg % ( + calc_value, known_value, + calc_type, data_type, set_locale, + used_locale)) + return True + + @unittest.skipUnless(nl_langinfo, "nl_langinfo is not available") + @unittest.skipIf(support.linked_to_musl(), "musl libc issue, bpo-46390") + def test_lc_numeric_nl_langinfo(self): + # Test nl_langinfo against known values + tested = False + oldloc = setlocale(LC_CTYPE) + for loc in candidate_locales: + try: + setlocale(LC_NUMERIC, loc) + except Error: + continue + for li, lc in ((RADIXCHAR, "decimal_point"), + (THOUSEP, "thousands_sep")): + if self.numeric_tester('nl_langinfo', nl_langinfo(li), lc, loc): + tested = True + self.assertEqual(setlocale(LC_CTYPE), oldloc) + if not tested: + self.skipTest('no suitable locales') + + @unittest.skipIf(support.linked_to_musl(), "musl libc issue, bpo-46390") + def test_lc_numeric_localeconv(self): + # Test localeconv against known values + tested = False + oldloc = setlocale(LC_CTYPE) + for loc in candidate_locales: + try: + setlocale(LC_NUMERIC, loc) + except Error: + continue + formatting = localeconv() + for lc in ("decimal_point", + "thousands_sep"): + if self.numeric_tester('localeconv', formatting[lc], lc, loc): + tested = True + self.assertEqual(setlocale(LC_CTYPE), oldloc) + if not tested: + self.skipTest('no suitable locales') + + @unittest.skipUnless(nl_langinfo, "nl_langinfo is not available") + def test_lc_numeric_basic(self): + # Test nl_langinfo against localeconv + tested = False + oldloc = setlocale(LC_CTYPE) + for loc in candidate_locales: + try: + setlocale(LC_NUMERIC, loc) + except Error: + continue + for li, lc in ((RADIXCHAR, "decimal_point"), + (THOUSEP, "thousands_sep")): + nl_radixchar = nl_langinfo(li) + li_radixchar = localeconv()[lc] + try: + set_locale = setlocale(LC_NUMERIC) + except Error: + set_locale = "" + self.assertEqual(nl_radixchar, li_radixchar, + "%s (nl_langinfo) != %s (localeconv) " + "(set to %s, using %s)" % ( + nl_radixchar, li_radixchar, + loc, set_locale)) + tested = True + self.assertEqual(setlocale(LC_CTYPE), oldloc) + if not tested: + self.skipTest('no suitable locales') + + @unittest.skipUnless(nl_langinfo, "nl_langinfo is not available") + @unittest.skipUnless(hasattr(locale, 'ALT_DIGITS'), "requires locale.ALT_DIGITS") + @unittest.skipIf(support.linked_to_musl(), "musl libc issue, bpo-46390") + def test_alt_digits_nl_langinfo(self): + # Test nl_langinfo(ALT_DIGITS) + tested = False + for loc in candidate_locales: + with self.subTest(locale=loc): + try: + setlocale(LC_TIME, loc) + except Error: + self.skipTest(f'no locale {loc!r}') + continue + + with self.subTest(locale=loc): + alt_digits = nl_langinfo(locale.ALT_DIGITS) + self.assertIsInstance(alt_digits, str) + alt_digits = alt_digits.split(';') if alt_digits else [] + if alt_digits: + self.assertGreaterEqual(len(alt_digits), 10, alt_digits) + loc1 = loc.split('.', 1)[0] + if loc1 in known_alt_digits: + count, samples = known_alt_digits[loc1] + if count and not alt_digits: + self.skipTest(f'ALT_DIGITS is not set for locale {loc!r} on this platform') + self.assertEqual(len(alt_digits), count, alt_digits) + for i in samples: + self.assertEqual(alt_digits[i], samples[i]) + tested = True + if not tested: + self.skipTest('no suitable locales') + + @unittest.skipUnless(nl_langinfo, "nl_langinfo is not available") + @unittest.skipUnless(hasattr(locale, 'ERA'), "requires locale.ERA") + @unittest.skipIf(support.linked_to_musl(), "musl libc issue, bpo-46390") + def test_era_nl_langinfo(self): + # Test nl_langinfo(ERA) + tested = False + for loc in candidate_locales: + with self.subTest(locale=loc): + try: + setlocale(LC_TIME, loc) + except Error: + self.skipTest(f'no locale {loc!r}') + continue + + with self.subTest(locale=loc): + era = nl_langinfo(locale.ERA) + self.assertIsInstance(era, str) + if era: + self.assertEqual(era.count(':'), (era.count(';') + 1) * 5, era) + + loc1 = loc.split('.', 1)[0] + if loc1 in known_era: + count, sample = known_era[loc1] + if count: + if not era: + self.skipTest(f'ERA is not set for locale {loc!r} on this platform') + self.assertGreaterEqual(era.count(';') + 1, count) + self.assertIn(sample, era) + else: + self.assertEqual(era, '') + tested = True + if not tested: + self.skipTest('no suitable locales') + + def test_float_parsing(self): + # Bug #1391872: Test whether float parsing is okay on European + # locales. + tested = False + oldloc = setlocale(LC_CTYPE) + for loc in candidate_locales: + try: + setlocale(LC_NUMERIC, loc) + except Error: + continue + + # Ignore buggy locale databases. (Mac OS 10.4 and some other BSDs) + if loc == 'eu_ES' and localeconv()['decimal_point'] == "' ": + continue + + self.assertEqual(int(eval('3.14') * 100), 314, + "using eval('3.14') failed for %s" % loc) + self.assertEqual(int(float('3.14') * 100), 314, + "using float('3.14') failed for %s" % loc) + if localeconv()['decimal_point'] != '.': + self.assertRaises(ValueError, float, + localeconv()['decimal_point'].join(['1', '23'])) + tested = True + self.assertEqual(setlocale(LC_CTYPE), oldloc) + if not tested: + self.skipTest('no suitable locales') + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test__opcode.py b/Lib/test/test__opcode.py index 60dcdc6cd70..43d475baa5d 100644 --- a/Lib/test/test__opcode.py +++ b/Lib/test/test__opcode.py @@ -17,7 +17,7 @@ def check_bool_function_result(self, func, ops, expected): self.assertEqual(func(op), expected) def test_invalid_opcodes(self): - invalid = [-100, -1, 255, 512, 513, 1000] + invalid = [-100, -1, 512, 513, 1000] self.check_bool_function_result(_opcode.is_valid, invalid, False) self.check_bool_function_result(_opcode.has_arg, invalid, False) self.check_bool_function_result(_opcode.has_const, invalid, False) @@ -27,7 +27,6 @@ def test_invalid_opcodes(self): self.check_bool_function_result(_opcode.has_local, invalid, False) self.check_bool_function_result(_opcode.has_exc, invalid, False) - @unittest.expectedFailure # TODO: RUSTPYTHON - no instrumented opcodes def test_is_valid(self): names = [ 'CACHE', @@ -39,6 +38,14 @@ def test_is_valid(self): opcodes = [dis.opmap[opname] for opname in names] self.check_bool_function_result(_opcode.is_valid, opcodes, True) + @unittest.expectedFailure # TODO: RUSTPYTHON; KeyError: 'BINARY_OP_ADD_INT' + def test_opmaps(self): + def check_roundtrip(name, map): + return self.assertEqual(opcode.opname[map[name]], name) + + check_roundtrip('BINARY_OP', opcode.opmap) + check_roundtrip('BINARY_OP_ADD_INT', opcode._specialized_opmap) + def test_oplists(self): def check_function(self, func, expected): for op in [-10, 520]: @@ -59,8 +66,7 @@ def check_function(self, func, expected): class StackEffectTests(unittest.TestCase): def test_stack_effect(self): self.assertEqual(stack_effect(dis.opmap['POP_TOP']), -1) - self.assertEqual(stack_effect(dis.opmap['BUILD_SLICE'], 0), -1) - self.assertEqual(stack_effect(dis.opmap['BUILD_SLICE'], 1), -1) + self.assertEqual(stack_effect(dis.opmap['BUILD_SLICE'], 2), -1) self.assertEqual(stack_effect(dis.opmap['BUILD_SLICE'], 3), -2) self.assertRaises(ValueError, stack_effect, 30000) # All defined opcodes @@ -77,7 +83,6 @@ def test_stack_effect(self): self.assertRaises(ValueError, stack_effect, code) self.assertRaises(ValueError, stack_effect, code, 0) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_stack_effect_jump(self): FOR_ITER = dis.opmap['FOR_ITER'] self.assertEqual(stack_effect(FOR_ITER, 0), 1) @@ -111,7 +116,7 @@ def test_stack_effect_jump(self): class SpecializationStatsTests(unittest.TestCase): - @unittest.expectedFailure # TODO: RUSTPYTHON - no specialization stats + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 'load_attr' not found in [] def test_specialization_stats(self): stat_names = ["success", "failure", "hit", "deferred", "miss", "deopt"] specialized_opcodes = [ @@ -120,7 +125,7 @@ def test_specialization_stats(self): if opcode._inline_cache_entries.get(op, 0) ] self.assertIn('load_attr', specialized_opcodes) - self.assertIn('binary_subscr', specialized_opcodes) + self.assertIn('binary_op', specialized_opcodes) stats = _opcode.get_specialization_stats() if stats is not None: diff --git a/Lib/test/test_apple.py b/Lib/test/test_apple.py new file mode 100644 index 00000000000..ab5296afad1 --- /dev/null +++ b/Lib/test/test_apple.py @@ -0,0 +1,155 @@ +import unittest +from _apple_support import SystemLog +from test.support import is_apple +from unittest.mock import Mock, call + +if not is_apple: + raise unittest.SkipTest("Apple-specific") + + +# Test redirection of stdout and stderr to the Apple system log. +class TestAppleSystemLogOutput(unittest.TestCase): + maxDiff = None + + def assert_writes(self, output): + self.assertEqual( + self.log_write.mock_calls, + [ + call(self.log_level, line) + for line in output + ] + ) + + self.log_write.reset_mock() + + def setUp(self): + self.log_write = Mock() + self.log_level = 42 + self.log = SystemLog(self.log_write, self.log_level, errors="replace") + + def test_repr(self): + self.assertEqual(repr(self.log), "") + self.assertEqual(repr(self.log.buffer), "") + + def test_log_config(self): + self.assertIs(self.log.writable(), True) + self.assertIs(self.log.readable(), False) + + self.assertEqual("UTF-8", self.log.encoding) + self.assertEqual("replace", self.log.errors) + + self.assertIs(self.log.line_buffering, True) + self.assertIs(self.log.write_through, False) + + def test_empty_str(self): + self.log.write("") + self.log.flush() + + self.assert_writes([]) + + def test_simple_str(self): + self.log.write("hello world\n") + + self.assert_writes([b"hello world\n"]) + + def test_buffered_str(self): + self.log.write("h") + self.log.write("ello") + self.log.write(" ") + self.log.write("world\n") + self.log.write("goodbye.") + self.log.flush() + + self.assert_writes([b"hello world\n", b"goodbye."]) + + def test_manual_flush(self): + self.log.write("Hello") + + self.assert_writes([]) + + self.log.write(" world\nHere for a while...\nGoodbye") + self.assert_writes([b"Hello world\n", b"Here for a while...\n"]) + + self.log.write(" world\nHello again") + self.assert_writes([b"Goodbye world\n"]) + + self.log.flush() + self.assert_writes([b"Hello again"]) + + def test_non_ascii(self): + # Spanish + self.log.write("ol\u00e9\n") + self.assert_writes([b"ol\xc3\xa9\n"]) + + # Chinese + self.log.write("\u4e2d\u6587\n") + self.assert_writes([b"\xe4\xb8\xad\xe6\x96\x87\n"]) + + # Printing Non-BMP emoji + self.log.write("\U0001f600\n") + self.assert_writes([b"\xf0\x9f\x98\x80\n"]) + + # Non-encodable surrogates are replaced + self.log.write("\ud800\udc00\n") + self.assert_writes([b"??\n"]) + + def test_modified_null(self): + # Null characters are logged using "modified UTF-8". + self.log.write("\u0000\n") + self.assert_writes([b"\xc0\x80\n"]) + self.log.write("a\u0000\n") + self.assert_writes([b"a\xc0\x80\n"]) + self.log.write("\u0000b\n") + self.assert_writes([b"\xc0\x80b\n"]) + self.log.write("a\u0000b\n") + self.assert_writes([b"a\xc0\x80b\n"]) + + def test_nonstandard_str(self): + # String subclasses are accepted, but they should be converted + # to a standard str without calling any of their methods. + class CustomStr(str): + def splitlines(self, *args, **kwargs): + raise AssertionError() + + def __len__(self): + raise AssertionError() + + def __str__(self): + raise AssertionError() + + self.log.write(CustomStr("custom\n")) + self.assert_writes([b"custom\n"]) + + def test_non_str(self): + # Non-string classes are not accepted. + for obj in [b"", b"hello", None, 42]: + with self.subTest(obj=obj): + with self.assertRaisesRegex( + TypeError, + fr"write\(\) argument must be str, not " + fr"{type(obj).__name__}" + ): + self.log.write(obj) + + def test_byteslike_in_buffer(self): + # The underlying buffer *can* accept bytes-like objects + self.log.buffer.write(bytearray(b"hello")) + self.log.flush() + + self.log.buffer.write(b"") + self.log.flush() + + self.log.buffer.write(b"goodbye") + self.log.flush() + + self.assert_writes([b"hello", b"goodbye"]) + + def test_non_byteslike_in_buffer(self): + for obj in ["hello", None, 42]: + with self.subTest(obj=obj): + with self.assertRaisesRegex( + TypeError, + fr"write\(\) argument must be bytes-like, not " + fr"{type(obj).__name__}" + ): + self.log.buffer.write(obj) diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py index b7e995334fe..9b8179ef969 100644 --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -1,11 +1,13 @@ # Author: Steven J. Bethard . +import _colorize import contextlib import functools import inspect import io import operator import os +import py_compile import shutil import stat import sys @@ -16,12 +18,22 @@ import warnings from enum import StrEnum -from test.support import captured_stderr +from test.support import ( + captured_stderr, + force_not_colorized, + force_not_colorized_test_class, + swap_attr, +) +from test.support import import_helper from test.support import os_helper +from test.support import script_helper from test.support.i18n_helper import TestTranslationsBase, update_translation_snapshots from unittest import mock +py = os.path.basename(sys.executable) + + class StdIOBuffer(io.TextIOWrapper): '''Replacement for writable io.StringIO that behaves more like real file @@ -631,9 +643,9 @@ class TestOptionalsNargsOptional(ParserTestCase): Sig('-w', nargs='?'), Sig('-x', nargs='?', const=42), Sig('-y', nargs='?', default='spam'), - Sig('-z', nargs='?', type=int, const='42', default='84'), + Sig('-z', nargs='?', type=int, const='42', default='84', choices=[1, 2]), ] - failures = ['2'] + failures = ['2', '-z a', '-z 42', '-z 84'] successes = [ ('', NS(w=None, x=None, y='spam', z=84)), ('-w', NS(w=None, x=None, y='spam', z=84)), @@ -777,48 +789,12 @@ def test_const(self): self.assertIn("got an unexpected keyword argument 'const'", str(cm.exception)) - def test_deprecated_init_kw(self): - # See gh-92248 + def test_invalid_name(self): parser = argparse.ArgumentParser() - - with self.assertWarns(DeprecationWarning): - parser.add_argument( - '-a', - action=argparse.BooleanOptionalAction, - type=None, - ) - with self.assertWarns(DeprecationWarning): - parser.add_argument( - '-b', - action=argparse.BooleanOptionalAction, - type=bool, - ) - - with self.assertWarns(DeprecationWarning): - parser.add_argument( - '-c', - action=argparse.BooleanOptionalAction, - metavar=None, - ) - with self.assertWarns(DeprecationWarning): - parser.add_argument( - '-d', - action=argparse.BooleanOptionalAction, - metavar='d', - ) - - with self.assertWarns(DeprecationWarning): - parser.add_argument( - '-e', - action=argparse.BooleanOptionalAction, - choices=None, - ) - with self.assertWarns(DeprecationWarning): - parser.add_argument( - '-f', - action=argparse.BooleanOptionalAction, - choices=(), - ) + with self.assertRaises(ValueError) as cm: + parser.add_argument('--no-foo', action=argparse.BooleanOptionalAction) + self.assertEqual(str(cm.exception), + "invalid option name '--no-foo' for BooleanOptionalAction") class TestBooleanOptionalActionRequired(ParserTestCase): """Tests BooleanOptionalAction required""" @@ -1036,6 +1012,7 @@ def test_parse_enum_value(self): args = parser.parse_args(['--color', 'red']) self.assertEqual(args.color, self.Color.RED) + @force_not_colorized def test_help_message_contains_enum_choices(self): parser = argparse.ArgumentParser() parser.add_argument('--color', choices=self.Color, help='Choose a color') @@ -1101,8 +1078,8 @@ class TestPositionalsNargsZeroOrMore(ParserTestCase): class TestPositionalsNargsZeroOrMoreDefault(ParserTestCase): """Test a Positional that specifies unlimited nargs and a default""" - argument_signatures = [Sig('foo', nargs='*', default='bar')] - failures = ['-x'] + argument_signatures = [Sig('foo', nargs='*', default='bar', choices=['a', 'b'])] + failures = ['-x', 'bar', 'a c'] successes = [ ('', NS(foo='bar')), ('a', NS(foo=['a'])), @@ -1135,8 +1112,8 @@ class TestPositionalsNargsOptional(ParserTestCase): class TestPositionalsNargsOptionalDefault(ParserTestCase): """Tests an Optional Positional with a default value""" - argument_signatures = [Sig('foo', nargs='?', default=42)] - failures = ['-x', 'a b'] + argument_signatures = [Sig('foo', nargs='?', default=42, choices=['a', 'b'])] + failures = ['-x', 'a b', '42'] successes = [ ('', NS(foo=42)), ('a', NS(foo='a')), @@ -1149,9 +1126,9 @@ class TestPositionalsNargsOptionalConvertedDefault(ParserTestCase): """ argument_signatures = [ - Sig('foo', nargs='?', type=int, default='42'), + Sig('foo', nargs='?', type=int, default='42', choices=[1, 2]), ] - failures = ['-x', 'a b', '1 2'] + failures = ['-x', 'a b', '1 2', '42'] successes = [ ('', NS(foo=42)), ('1', NS(foo=1)), @@ -1811,27 +1788,43 @@ def convert_arg_line_to_args(self, arg_line): # Type conversion tests # ===================== +def FileType(*args, **kwargs): + with warnings.catch_warnings(): + warnings.filterwarnings('ignore', 'FileType is deprecated', + PendingDeprecationWarning, __name__) + return argparse.FileType(*args, **kwargs) + + +class TestFileTypeDeprecation(TestCase): + + def test(self): + with self.assertWarns(PendingDeprecationWarning) as cm: + argparse.FileType() + self.assertIn('FileType is deprecated', str(cm.warning)) + self.assertEqual(cm.filename, __file__) + + class TestFileTypeRepr(TestCase): def test_r(self): - type = argparse.FileType('r') + type = FileType('r') self.assertEqual("FileType('r')", repr(type)) def test_wb_1(self): - type = argparse.FileType('wb', 1) + type = FileType('wb', 1) self.assertEqual("FileType('wb', 1)", repr(type)) def test_r_latin(self): - type = argparse.FileType('r', encoding='latin_1') + type = FileType('r', encoding='latin_1') self.assertEqual("FileType('r', encoding='latin_1')", repr(type)) def test_w_big5_ignore(self): - type = argparse.FileType('w', encoding='big5', errors='ignore') + type = FileType('w', encoding='big5', errors='ignore') self.assertEqual("FileType('w', encoding='big5', errors='ignore')", repr(type)) def test_r_1_replace(self): - type = argparse.FileType('r', 1, errors='replace') + type = FileType('r', 1, errors='replace') self.assertEqual("FileType('r', 1, errors='replace')", repr(type)) @@ -1885,7 +1878,6 @@ def __eq__(self, other): text = text.decode('ascii') return self.name == other.name == text - class TestFileTypeR(TempDirMixin, ParserTestCase): """Test the FileType option/argument type for reading files""" @@ -1898,8 +1890,8 @@ def setUp(self): self.create_readonly_file('readonly') argument_signatures = [ - Sig('-x', type=argparse.FileType()), - Sig('spam', type=argparse.FileType('r')), + Sig('-x', type=FileType()), + Sig('spam', type=FileType('r')), ] failures = ['-x', '', 'non-existent-file.txt'] successes = [ @@ -1919,7 +1911,7 @@ def setUp(self): file.close() argument_signatures = [ - Sig('-c', type=argparse.FileType('r'), default='no-file.txt'), + Sig('-c', type=FileType('r'), default='no-file.txt'), ] # should provoke no such file error failures = [''] @@ -1938,8 +1930,8 @@ def setUp(self): file.write(file_name) argument_signatures = [ - Sig('-x', type=argparse.FileType('rb')), - Sig('spam', type=argparse.FileType('rb')), + Sig('-x', type=FileType('rb')), + Sig('spam', type=FileType('rb')), ] failures = ['-x', ''] successes = [ @@ -1977,8 +1969,8 @@ def setUp(self): self.create_writable_file('writable') argument_signatures = [ - Sig('-x', type=argparse.FileType('w')), - Sig('spam', type=argparse.FileType('w')), + Sig('-x', type=FileType('w')), + Sig('spam', type=FileType('w')), ] failures = ['-x', '', 'readonly'] successes = [ @@ -2000,8 +1992,8 @@ def setUp(self): self.create_writable_file('writable') argument_signatures = [ - Sig('-x', type=argparse.FileType('x')), - Sig('spam', type=argparse.FileType('x')), + Sig('-x', type=FileType('x')), + Sig('spam', type=FileType('x')), ] failures = ['-x', '', 'readonly', 'writable'] successes = [ @@ -2015,8 +2007,8 @@ class TestFileTypeWB(TempDirMixin, ParserTestCase): """Test the FileType option/argument type for writing binary files""" argument_signatures = [ - Sig('-x', type=argparse.FileType('wb')), - Sig('spam', type=argparse.FileType('wb')), + Sig('-x', type=FileType('wb')), + Sig('spam', type=FileType('wb')), ] failures = ['-x', ''] successes = [ @@ -2032,8 +2024,8 @@ class TestFileTypeXB(TestFileTypeX): "Test the FileType option/argument type for writing new binary files only" argument_signatures = [ - Sig('-x', type=argparse.FileType('xb')), - Sig('spam', type=argparse.FileType('xb')), + Sig('-x', type=FileType('xb')), + Sig('spam', type=FileType('xb')), ] successes = [ ('-x foo bar', NS(x=WFile('foo'), spam=WFile('bar'))), @@ -2045,7 +2037,7 @@ class TestFileTypeOpenArgs(TestCase): """Test that open (the builtin) is correctly called""" def test_open_args(self): - FT = argparse.FileType + FT = FileType cases = [ (FT('rb'), ('rb', -1, None, None)), (FT('w', 1), ('w', 1, None, None)), @@ -2060,7 +2052,7 @@ def test_open_args(self): def test_invalid_file_type(self): with self.assertRaises(ValueError): - argparse.FileType('b')('-test') + FileType('b')('-test') class TestFileTypeMissingInitialization(TestCase): @@ -2071,7 +2063,7 @@ class TestFileTypeMissingInitialization(TestCase): def test(self): parser = argparse.ArgumentParser() - with self.assertRaises(ValueError) as cm: + with self.assertRaises(TypeError) as cm: parser.add_argument('-x', type=argparse.FileType) self.assertEqual( @@ -2256,6 +2248,130 @@ class TestActionExtend(ParserTestCase): ] +class TestNegativeNumber(ParserTestCase): + """Test parsing negative numbers""" + + argument_signatures = [ + Sig('--int', type=int), + Sig('--float', type=float), + Sig('--complex', type=complex), + ] + failures = [ + '--float -_.45', + '--float -1__000.0', + '--float -1.0.0', + '--int -1__000', + '--int -1.0', + '--complex -1__000.0j', + '--complex -1.0jj', + '--complex -_.45j', + ] + successes = [ + ('--int -1000 --float -1000.0', NS(int=-1000, float=-1000.0, complex=None)), + ('--int -1_000 --float -1_000.0', NS(int=-1000, float=-1000.0, complex=None)), + ('--int -1_000_000 --float -1_000_000.0', NS(int=-1000000, float=-1000000.0, complex=None)), + ('--float -1_000.0', NS(int=None, float=-1000.0, complex=None)), + ('--float -1_000_000.0_0', NS(int=None, float=-1000000.0, complex=None)), + ('--float -.5', NS(int=None, float=-0.5, complex=None)), + ('--float -.5_000', NS(int=None, float=-0.5, complex=None)), + ('--float -1e3', NS(int=None, float=-1000, complex=None)), + ('--float -1e-3', NS(int=None, float=-0.001, complex=None)), + ('--complex -1j', NS(int=None, float=None, complex=-1j)), + ('--complex -1_000j', NS(int=None, float=None, complex=-1000j)), + ('--complex -1_000.0j', NS(int=None, float=None, complex=-1000.0j)), + ('--complex -1e3j', NS(int=None, float=None, complex=-1000j)), + ('--complex -1e-3j', NS(int=None, float=None, complex=-0.001j)), + ] + +class TestArgumentAndSubparserSuggestions(TestCase): + """Test error handling and suggestion when a user makes a typo""" + + def test_wrong_argument_error_with_suggestions(self): + parser = ErrorRaisingArgumentParser(suggest_on_error=True) + parser.add_argument('foo', choices=['bar', 'baz']) + with self.assertRaises(ArgumentParserError) as excinfo: + parser.parse_args(('bazz',)) + self.assertIn( + "error: argument foo: invalid choice: 'bazz', maybe you meant 'baz'? (choose from bar, baz)", + excinfo.exception.stderr + ) + + def test_wrong_argument_error_no_suggestions(self): + parser = ErrorRaisingArgumentParser(suggest_on_error=False) + parser.add_argument('foo', choices=['bar', 'baz']) + with self.assertRaises(ArgumentParserError) as excinfo: + parser.parse_args(('bazz',)) + self.assertIn( + "error: argument foo: invalid choice: 'bazz' (choose from bar, baz)", + excinfo.exception.stderr, + ) + + def test_wrong_argument_subparsers_with_suggestions(self): + parser = ErrorRaisingArgumentParser(suggest_on_error=True) + subparsers = parser.add_subparsers(required=True) + subparsers.add_parser('foo') + subparsers.add_parser('bar') + with self.assertRaises(ArgumentParserError) as excinfo: + parser.parse_args(('baz',)) + self.assertIn( + "error: argument {foo,bar}: invalid choice: 'baz', maybe you meant" + " 'bar'? (choose from foo, bar)", + excinfo.exception.stderr, + ) + + def test_wrong_argument_subparsers_no_suggestions(self): + parser = ErrorRaisingArgumentParser(suggest_on_error=False) + subparsers = parser.add_subparsers(required=True) + subparsers.add_parser('foo') + subparsers.add_parser('bar') + with self.assertRaises(ArgumentParserError) as excinfo: + parser.parse_args(('baz',)) + self.assertIn( + "error: argument {foo,bar}: invalid choice: 'baz' (choose from foo, bar)", + excinfo.exception.stderr, + ) + + def test_wrong_argument_no_suggestion_implicit(self): + parser = ErrorRaisingArgumentParser() + parser.add_argument('foo', choices=['bar', 'baz']) + with self.assertRaises(ArgumentParserError) as excinfo: + parser.parse_args(('bazz',)) + self.assertIn( + "error: argument foo: invalid choice: 'bazz' (choose from bar, baz)", + excinfo.exception.stderr, + ) + + def test_suggestions_choices_empty(self): + parser = ErrorRaisingArgumentParser(suggest_on_error=True) + parser.add_argument('foo', choices=[]) + with self.assertRaises(ArgumentParserError) as excinfo: + parser.parse_args(('bazz',)) + self.assertIn( + "error: argument foo: invalid choice: 'bazz' (choose from )", + excinfo.exception.stderr, + ) + + def test_suggestions_choices_int(self): + parser = ErrorRaisingArgumentParser(suggest_on_error=True) + parser.add_argument('foo', choices=[1, 2]) + with self.assertRaises(ArgumentParserError) as excinfo: + parser.parse_args(('3',)) + self.assertIn( + "error: argument foo: invalid choice: '3' (choose from 1, 2)", + excinfo.exception.stderr, + ) + + def test_suggestions_choices_mixed_types(self): + parser = ErrorRaisingArgumentParser(suggest_on_error=True) + parser.add_argument('foo', choices=[1, '2']) + with self.assertRaises(ArgumentParserError) as excinfo: + parser.parse_args(('3',)) + self.assertIn( + "error: argument foo: invalid choice: '3' (choose from 1, 2)", + excinfo.exception.stderr, + ) + + class TestInvalidAction(TestCase): """Test invalid user defined Action""" @@ -2269,17 +2385,31 @@ def test_invalid_type(self): self.assertRaises(NotImplementedError, parser.parse_args, ['--foo', 'bar']) def test_modified_invalid_action(self): - parser = ErrorRaisingArgumentParser() + parser = argparse.ArgumentParser(exit_on_error=False) action = parser.add_argument('--foo') # Someone got crazy and did this action.type = 1 - self.assertRaises(ArgumentParserError, parser.parse_args, ['--foo', 'bar']) + self.assertRaisesRegex(TypeError, '1 is not callable', + parser.parse_args, ['--foo', 'bar']) + action.type = () + self.assertRaisesRegex(TypeError, r'\(\) is not callable', + parser.parse_args, ['--foo', 'bar']) + # It is impossible to distinguish a TypeError raised due to a mismatch + # of the required function arguments from a TypeError raised for an incorrect + # argument value, and using the heavy inspection machinery is not worthwhile + # as it does not reliably work in all cases. + # Therefore, a generic ArgumentError is raised to handle this logical error. + action.type = pow + self.assertRaisesRegex(argparse.ArgumentError, + "argument --foo: invalid pow value: 'bar'", + parser.parse_args, ['--foo', 'bar']) # ================ # Subparsers tests # ================ +@force_not_colorized_test_class class TestAddSubparsers(TestCase): """Test the add_subparsers method""" @@ -2287,16 +2417,17 @@ def assertArgumentParserError(self, *args, **kwargs): self.assertRaises(ArgumentParserError, *args, **kwargs) def _get_parser(self, subparser_help=False, prefix_chars=None, - aliases=False): + aliases=False, usage=None): # create a parser with a subparsers argument if prefix_chars: parser = ErrorRaisingArgumentParser( - prog='PROG', description='main description', prefix_chars=prefix_chars) + prog='PROG', description='main description', usage=usage, + prefix_chars=prefix_chars) parser.add_argument( prefix_chars[0] * 2 + 'foo', action='store_true', help='foo help') else: parser = ErrorRaisingArgumentParser( - prog='PROG', description='main description') + prog='PROG', description='main description', usage=usage) parser.add_argument( '--foo', action='store_true', help='foo help') parser.add_argument( @@ -2310,7 +2441,7 @@ def _get_parser(self, subparser_help=False, prefix_chars=None, else: subparsers_kwargs['help'] = 'command help' subparsers = parser.add_subparsers(**subparsers_kwargs) - self.assertRaisesRegex(argparse.ArgumentError, + self.assertRaisesRegex(ValueError, 'cannot have multiple subparser arguments', parser.add_subparsers) @@ -2333,7 +2464,8 @@ def _get_parser(self, subparser_help=False, prefix_chars=None, parser2.add_argument('z', type=complex, nargs='*', help='z help') # add third sub-parser - parser3_kwargs = dict(description='3 description') + parser3_kwargs = dict(description='3 description', + usage='PROG --foo bar 3 t ...') if subparser_help: parser3_kwargs['help'] = '3 help' parser3 = subparsers.add_parser('3', **parser3_kwargs) @@ -2355,6 +2487,47 @@ def test_parse_args_failures(self): args = args_str.split() self.assertArgumentParserError(self.parser.parse_args, args) + def test_parse_args_failures_details(self): + for args_str, usage_str, error_str in [ + ('', + 'usage: PROG [-h] [--foo] bar {1,2,3} ...', + 'PROG: error: the following arguments are required: bar'), + ('0.5 1 -y', + 'usage: PROG bar 1 [-h] [-w W] {a,b,c}', + 'PROG bar 1: error: the following arguments are required: x'), + ('0.5 3', + 'usage: PROG --foo bar 3 t ...', + 'PROG bar 3: error: the following arguments are required: t'), + ]: + with self.subTest(args_str): + args = args_str.split() + with self.assertRaises(ArgumentParserError) as cm: + self.parser.parse_args(args) + self.assertEqual(cm.exception.args[0], 'SystemExit') + self.assertEqual(cm.exception.args[2], f'{usage_str}\n{error_str}\n') + + def test_parse_args_failures_details_custom_usage(self): + parser = self._get_parser(usage='PROG [--foo] bar 1 [-w W] {a,b,c}\n' + ' PROG --foo bar 3 t ...') + for args_str, usage_str, error_str in [ + ('', + 'usage: PROG [--foo] bar 1 [-w W] {a,b,c}\n' + ' PROG --foo bar 3 t ...', + 'PROG: error: the following arguments are required: bar'), + ('0.5 1 -y', + 'usage: PROG bar 1 [-h] [-w W] {a,b,c}', + 'PROG bar 1: error: the following arguments are required: x'), + ('0.5 3', + 'usage: PROG --foo bar 3 t ...', + 'PROG bar 3: error: the following arguments are required: t'), + ]: + with self.subTest(args_str): + args = args_str.split() + with self.assertRaises(ArgumentParserError) as cm: + parser.parse_args(args) + self.assertEqual(cm.exception.args[0], 'SystemExit') + self.assertEqual(cm.exception.args[2], f'{usage_str}\n{error_str}\n') + def test_parse_args(self): # check some non-failure cases: self.assertEqual( @@ -2508,18 +2681,6 @@ def test_required_subparsers_no_destination_error(self): 'error: the following arguments are required: {foo,bar}\n$' ) - def test_wrong_argument_subparsers_no_destination_error(self): - parser = ErrorRaisingArgumentParser() - subparsers = parser.add_subparsers(required=True) - subparsers.add_parser('foo') - subparsers.add_parser('bar') - with self.assertRaises(ArgumentParserError) as excinfo: - parser.parse_args(('baz',)) - self.assertRegex( - excinfo.exception.stderr, - r"error: argument {foo,bar}: invalid choice: 'baz' \(choose from foo, bar\)\n$" - ) - def test_optional_subparsers(self): parser = ErrorRaisingArgumentParser() subparsers = parser.add_subparsers(dest='command', required=False) @@ -2655,6 +2816,29 @@ def test_parser_command_help(self): --foo foo help ''')) + def assert_bad_help(self, context_type, func, *args, **kwargs): + with self.assertRaisesRegex(ValueError, 'badly formed help string') as cm: + func(*args, **kwargs) + self.assertIsInstance(cm.exception.__context__, context_type) + + def test_invalid_subparsers_help(self): + parser = ErrorRaisingArgumentParser(prog='PROG') + self.assert_bad_help(ValueError, parser.add_subparsers, help='%Y-%m-%d') + parser = ErrorRaisingArgumentParser(prog='PROG') + self.assert_bad_help(KeyError, parser.add_subparsers, help='%(spam)s') + parser = ErrorRaisingArgumentParser(prog='PROG') + self.assert_bad_help(TypeError, parser.add_subparsers, help='%(prog)d') + + def test_invalid_subparser_help(self): + parser = ErrorRaisingArgumentParser(prog='PROG') + subparsers = parser.add_subparsers() + self.assert_bad_help(ValueError, subparsers.add_parser, '1', + help='%Y-%m-%d') + self.assert_bad_help(KeyError, subparsers.add_parser, '1', + help='%(spam)s') + self.assert_bad_help(TypeError, subparsers.add_parser, '1', + help='%(prog)d') + def test_subparser_title_help(self): parser = ErrorRaisingArgumentParser(prog='PROG', description='main description') @@ -2796,10 +2980,43 @@ def test_interleaved_groups(self): result = parser.parse_args('1 2 3 4'.split()) self.assertEqual(expected, result) +class TestGroupConstructor(TestCase): + def test_group_prefix_chars(self): + parser = ErrorRaisingArgumentParser() + msg = ( + "The use of the undocumented 'prefix_chars' parameter in " + "ArgumentParser.add_argument_group() is deprecated." + ) + with self.assertWarns(DeprecationWarning) as cm: + parser.add_argument_group(prefix_chars='-+') + self.assertEqual(msg, str(cm.warning)) + self.assertEqual(cm.filename, __file__) + + def test_group_prefix_chars_default(self): + # "default" isn't quite the right word here, but it's the same as + # the parser's default prefix so it's a good test + parser = ErrorRaisingArgumentParser() + msg = ( + "The use of the undocumented 'prefix_chars' parameter in " + "ArgumentParser.add_argument_group() is deprecated." + ) + with self.assertWarns(DeprecationWarning) as cm: + parser.add_argument_group(prefix_chars='-') + self.assertEqual(msg, str(cm.warning)) + self.assertEqual(cm.filename, __file__) + + def test_nested_argument_group(self): + parser = argparse.ArgumentParser() + g = parser.add_argument_group() + self.assertRaisesRegex(ValueError, + 'argument groups cannot be nested', + g.add_argument_group) + # =================== # Parent parser tests # =================== +@force_not_colorized_test_class class TestParentParsers(TestCase): """Tests that parsers can be created with parent parsers""" @@ -2832,8 +3049,6 @@ def setUp(self): group.add_argument('-a', action='store_true') group.add_argument('-b', action='store_true') - self.main_program = os.path.basename(sys.argv[0]) - def test_single_parent(self): parser = ErrorRaisingArgumentParser(parents=[self.wxyz_parent]) self.assertEqual(parser.parse_args('-y 1 2 --w 3'.split()), @@ -2844,7 +3059,7 @@ def test_single_parent_mutex(self): parser = ErrorRaisingArgumentParser(parents=[self.ab_mutex_parent]) self._test_mutex_ab(parser.parse_args) - def test_single_granparent_mutex(self): + def test_single_grandparent_mutex(self): parents = [self.ab_mutex_parent] parser = ErrorRaisingArgumentParser(add_help=False, parents=parents) parser = ErrorRaisingArgumentParser(parents=[parser]) @@ -2923,11 +3138,10 @@ def test_subparser_parents_mutex(self): def test_parent_help(self): parents = [self.abcd_parent, self.wxyz_parent] - parser = ErrorRaisingArgumentParser(parents=parents) + parser = ErrorRaisingArgumentParser(prog='PROG', parents=parents) parser_help = parser.format_help() - progname = self.main_program self.assertEqual(parser_help, textwrap.dedent('''\ - usage: {}{}[-h] [-b B] [--d D] [--w W] [-y Y] a z + usage: PROG [-h] [-b B] [--d D] [--w W] [-y Y] a z positional arguments: a @@ -2943,7 +3157,7 @@ def test_parent_help(self): x: -y Y - '''.format(progname, ' ' if progname else '' ))) + ''')) def test_groups_parents(self): parent = ErrorRaisingArgumentParser(add_help=False) @@ -2953,15 +3167,14 @@ def test_groups_parents(self): m = parent.add_mutually_exclusive_group() m.add_argument('-y') m.add_argument('-z') - parser = ErrorRaisingArgumentParser(parents=[parent]) + parser = ErrorRaisingArgumentParser(prog='PROG', parents=[parent]) self.assertRaises(ArgumentParserError, parser.parse_args, ['-y', 'Y', '-z', 'Z']) parser_help = parser.format_help() - progname = self.main_program self.assertEqual(parser_help, textwrap.dedent('''\ - usage: {}{}[-h] [-w W] [-x X] [-y Y | -z Z] + usage: PROG [-h] [-w W] [-x X] [-y Y | -z Z] options: -h, --help show this help message and exit @@ -2973,7 +3186,7 @@ def test_groups_parents(self): -w W -x X - '''.format(progname, ' ' if progname else '' ))) + ''')) def test_wrong_type_parents(self): self.assertRaises(TypeError, ErrorRaisingArgumentParser, parents=[1]) @@ -3011,6 +3224,7 @@ def test_mutex_groups_parents(self): # Mutually exclusive group tests # ============================== +@force_not_colorized_test_class class TestMutuallyExclusiveGroupErrors(TestCase): def test_invalid_add_argument_group(self): @@ -3049,6 +3263,29 @@ def test_help(self): ''' self.assertEqual(parser.format_help(), textwrap.dedent(expected)) + def test_optional_order(self): + parser = ErrorRaisingArgumentParser(prog='PROG') + group = parser.add_mutually_exclusive_group(required=True) + group.add_argument('--foo') + group.add_argument('bar', nargs='?') + expected = '''\ + usage: PROG [-h] (--foo FOO | bar) + + positional arguments: + bar + + options: + -h, --help show this help message and exit + --foo FOO + ''' + self.assertEqual(parser.format_help(), textwrap.dedent(expected)) + + parser = ErrorRaisingArgumentParser(prog='PROG') + group = parser.add_mutually_exclusive_group(required=True) + group.add_argument('bar', nargs='?') + group.add_argument('--foo') + self.assertEqual(parser.format_help(), textwrap.dedent(expected)) + def test_help_subparser_all_mutually_exclusive_group_members_suppressed(self): self.maxDiff = None parser = ErrorRaisingArgumentParser(prog='PROG') @@ -3077,6 +3314,14 @@ def test_empty_group(self): with self.assertRaises(ValueError): parser.parse_args(['-h']) + def test_nested_mutex_groups(self): + parser = argparse.ArgumentParser(prog='PROG') + g = parser.add_mutually_exclusive_group() + g.add_argument("--spam") + self.assertRaisesRegex(ValueError, + 'mutually exclusive groups cannot be nested', + g.add_mutually_exclusive_group) + class MEMixin(object): def test_failures_when_not_required(self): @@ -3108,21 +3353,25 @@ def test_successes_when_required(self): actual_ns = parse_args(args_string.split()) self.assertEqual(actual_ns, expected_ns) + @force_not_colorized def test_usage_when_not_required(self): format_usage = self.get_parser(required=False).format_usage expected_usage = self.usage_when_not_required self.assertEqual(format_usage(), textwrap.dedent(expected_usage)) + @force_not_colorized def test_usage_when_required(self): format_usage = self.get_parser(required=True).format_usage expected_usage = self.usage_when_required self.assertEqual(format_usage(), textwrap.dedent(expected_usage)) + @force_not_colorized def test_help_when_not_required(self): format_help = self.get_parser(required=False).format_help help = self.usage_when_not_required + self.help self.assertEqual(format_help(), textwrap.dedent(help)) + @force_not_colorized def test_help_when_required(self): format_help = self.get_parser(required=True).format_help help = self.usage_when_required + self.help @@ -3444,55 +3693,6 @@ def get_parser(self, required): -c c help ''' -class TestMutuallyExclusiveNested(MEMixin, TestCase): - - # Nesting mutually exclusive groups is an undocumented feature - # that came about by accident through inheritance and has been - # the source of many bugs. It is deprecated and this test should - # eventually be removed along with it. - - def get_parser(self, required): - parser = ErrorRaisingArgumentParser(prog='PROG') - group = parser.add_mutually_exclusive_group(required=required) - group.add_argument('-a') - group.add_argument('-b') - with warnings.catch_warnings(): - warnings.simplefilter('ignore', DeprecationWarning) - group2 = group.add_mutually_exclusive_group(required=required) - group2.add_argument('-c') - group2.add_argument('-d') - with warnings.catch_warnings(): - warnings.simplefilter('ignore', DeprecationWarning) - group3 = group2.add_mutually_exclusive_group(required=required) - group3.add_argument('-e') - group3.add_argument('-f') - return parser - - usage_when_not_required = '''\ - usage: PROG [-h] [-a A | -b B | [-c C | -d D | [-e E | -f F]]] - ''' - usage_when_required = '''\ - usage: PROG [-h] (-a A | -b B | (-c C | -d D | (-e E | -f F))) - ''' - - help = '''\ - - options: - -h, --help show this help message and exit - -a A - -b B - -c C - -d D - -e E - -f F - ''' - - # We are only interested in testing the behavior of format_usage(). - test_failures_when_not_required = None - test_failures_when_required = None - test_successes_when_not_required = None - test_successes_when_required = None - class TestMutuallyExclusiveOptionalOptional(MEMixin, TestCase): def get_parser(self, required=None): @@ -3843,11 +4043,13 @@ def _test(self, tester, parser_text): tester.maxDiff = None tester.assertEqual(expected_text, parser_text) + @force_not_colorized def test_format(self, tester): parser = self._get_parser(tester) format = getattr(parser, 'format_%s' % self.func_suffix) self._test(tester, format()) + @force_not_colorized def test_print(self, tester): parser = self._get_parser(tester) print_ = getattr(parser, 'print_%s' % self.func_suffix) @@ -3860,6 +4062,7 @@ def test_print(self, tester): setattr(sys, self.std_name, old_stream) self._test(tester, parser_text) + @force_not_colorized def test_print_file(self, tester): parser = self._get_parser(tester) print_ = getattr(parser, 'print_%s' % self.func_suffix) @@ -4601,6 +4804,7 @@ class TestHelpUsageMetavarsSpacesParentheses(HelpTestCase): version = '' +@force_not_colorized_test_class class TestHelpUsageNoWhitespaceCrash(TestCase): def test_all_suppressed_mutex_followed_by_long_arg(self): @@ -4663,25 +4867,6 @@ def test_all_suppressed_mutex_with_optional_nargs(self): usage = 'usage: PROG [-h]\n' self.assertEqual(parser.format_usage(), usage) - def test_nested_mutex_groups(self): - parser = argparse.ArgumentParser(prog='PROG') - g = parser.add_mutually_exclusive_group() - g.add_argument("--spam") - with warnings.catch_warnings(): - warnings.simplefilter('ignore', DeprecationWarning) - gg = g.add_mutually_exclusive_group() - gg.add_argument("--hax") - gg.add_argument("--hox", help=argparse.SUPPRESS) - gg.add_argument("--hex") - g.add_argument("--eggs") - parser.add_argument("--num") - - usage = textwrap.dedent('''\ - usage: PROG [-h] [--spam SPAM | [--hax HAX | --hex HEX] | --eggs EGGS] - [--num NUM] - ''') - self.assertEqual(parser.format_usage(), usage) - def test_long_mutex_groups_wrap(self): parser = argparse.ArgumentParser(prog='PROG') g = parser.add_mutually_exclusive_group() @@ -5301,11 +5486,61 @@ def custom_type(string): version = '' -class TestHelpUsageLongSubparserCommand(TestCase): - """Test that subparser commands are formatted correctly in help""" +@force_not_colorized_test_class +class TestHelpCustomHelpFormatter(TestCase): maxDiff = None - def test_parent_help(self): + def test_custom_formatter_function(self): + def custom_formatter(prog): + return argparse.RawTextHelpFormatter(prog, indent_increment=5) + + parser = argparse.ArgumentParser( + prog='PROG', + prefix_chars='-+', + formatter_class=custom_formatter + ) + parser.add_argument('+f', '++foo', help="foo help") + parser.add_argument('spam', help="spam help") + + parser_help = parser.format_help() + self.assertEqual(parser_help, textwrap.dedent('''\ + usage: PROG [-h] [+f FOO] spam + + positional arguments: + spam spam help + + options: + -h, --help show this help message and exit + +f, ++foo FOO foo help + ''')) + + def test_custom_formatter_class(self): + class CustomFormatter(argparse.RawTextHelpFormatter): + def __init__(self, prog): + super().__init__(prog, indent_increment=5) + + parser = argparse.ArgumentParser( + prog='PROG', + prefix_chars='-+', + formatter_class=CustomFormatter + ) + parser.add_argument('+f', '++foo', help="foo help") + parser.add_argument('spam', help="spam help") + + parser_help = parser.format_help() + self.assertEqual(parser_help, textwrap.dedent('''\ + usage: PROG [-h] [+f FOO] spam + + positional arguments: + spam spam help + + options: + -h, --help show this help message and exit + +f, ++foo FOO foo help + ''')) + + def test_usage_long_subparser_command(self): + """Test that subparser commands are formatted correctly in help""" def custom_formatter(prog): return argparse.RawTextHelpFormatter(prog, max_help_position=50) @@ -5371,29 +5606,45 @@ def test_missing_destination(self): self.assertTypeError(action=action) def test_invalid_option_strings(self): - self.assertValueError('--') - self.assertValueError('---') + self.assertTypeError('-', errmsg='dest= is required') + self.assertTypeError('--', errmsg='dest= is required') + self.assertTypeError('---', errmsg='dest= is required') def test_invalid_prefix(self): - self.assertValueError('--foo', '+foo') + self.assertValueError('--foo', '+foo', + errmsg='must start with a character') def test_invalid_type(self): - self.assertValueError('--foo', type='int') - self.assertValueError('--foo', type=(int, float)) + self.assertTypeError('--foo', type='int', + errmsg="'int' is not callable") + self.assertTypeError('--foo', type=(int, float), + errmsg='is not callable') def test_invalid_action(self): - self.assertValueError('-x', action='foo') - self.assertValueError('foo', action='baz') - self.assertValueError('--foo', action=('store', 'append')) + self.assertValueError('-x', action='foo', + errmsg='unknown action') + self.assertValueError('foo', action='baz', + errmsg='unknown action') + self.assertValueError('--foo', action=('store', 'append'), + errmsg='unknown action') self.assertValueError('--foo', action="store-true", errmsg='unknown action') + def test_invalid_help(self): + self.assertValueError('--foo', help='%Y-%m-%d', + errmsg='badly formed help string') + self.assertValueError('--foo', help='%(spam)s', + errmsg='badly formed help string') + self.assertValueError('--foo', help='%(prog)d', + errmsg='badly formed help string') + def test_multiple_dest(self): parser = argparse.ArgumentParser() parser.add_argument(dest='foo') - with self.assertRaises(ValueError) as cm: + with self.assertRaises(TypeError) as cm: parser.add_argument('bar', dest='baz') - self.assertIn('dest supplied twice for positional argument', + self.assertIn('dest supplied twice for positional argument,' + ' did you mean metavar?', str(cm.exception)) def test_no_argument_actions(self): @@ -5405,8 +5656,11 @@ def test_no_argument_actions(self): with self.subTest(attrs=attrs): self.assertTypeError('-x', action=action, **attrs) self.assertTypeError('x', action=action, **attrs) + self.assertValueError('x', action=action, + errmsg=f"action '{action}' is not valid for positional arguments") self.assertTypeError('-x', action=action, nargs=0) - self.assertTypeError('x', action=action, nargs=0) + self.assertValueError('x', action=action, nargs=0, + errmsg='nargs for positionals must be != 0') def test_no_argument_no_const_actions(self): # options with zero arguments @@ -5426,7 +5680,7 @@ def test_more_than_one_argument_actions(self): self.assertValueError('-x', nargs=0, action=action, errmsg=f'nargs for {action_name} actions must be != 0') self.assertValueError('spam', nargs=0, action=action, - errmsg=f'nargs for {action_name} actions must be != 0') + errmsg='nargs for positionals must be != 0') # const is disallowed with non-optional arguments for nargs in [1, '*', '+']: @@ -5529,6 +5783,7 @@ def test_conflict_error(self): self.assertRaises(argparse.ArgumentError, parser.add_argument, '--spam') + @force_not_colorized def test_resolve_error(self): get_parser = argparse.ArgumentParser parser = get_parser(prog='PROG', conflict_handler='resolve') @@ -5558,20 +5813,25 @@ def test_subparser_conflict(self): parser = argparse.ArgumentParser() sp = parser.add_subparsers() sp.add_parser('fullname', aliases=['alias']) - self.assertRaises(argparse.ArgumentError, - sp.add_parser, 'fullname') - self.assertRaises(argparse.ArgumentError, - sp.add_parser, 'alias') - self.assertRaises(argparse.ArgumentError, - sp.add_parser, 'other', aliases=['fullname']) - self.assertRaises(argparse.ArgumentError, - sp.add_parser, 'other', aliases=['alias']) + self.assertRaisesRegex(ValueError, + 'conflicting subparser: fullname', + sp.add_parser, 'fullname') + self.assertRaisesRegex(ValueError, + 'conflicting subparser: alias', + sp.add_parser, 'alias') + self.assertRaisesRegex(ValueError, + 'conflicting subparser alias: fullname', + sp.add_parser, 'other', aliases=['fullname']) + self.assertRaisesRegex(ValueError, + 'conflicting subparser alias: alias', + sp.add_parser, 'other', aliases=['alias']) # ============================= # Help and Version option tests # ============================= +@force_not_colorized_test_class class TestOptionalsHelpVersionActions(TestCase): """Test the help and version actions""" @@ -5791,6 +6051,7 @@ def test_argument_error(self): class TestArgumentTypeError(TestCase): + @force_not_colorized def test_argument_type_error(self): def spam(string): @@ -6565,7 +6826,7 @@ class TestImportStar(TestCase): def test(self): for name in argparse.__all__: - self.assertTrue(hasattr(argparse, name)) + self.assertHasAttr(argparse, name) def test_all_exports_everything_but_modules(self): items = [ @@ -6589,6 +6850,7 @@ def setUp(self): metavar = '' self.parser.add_argument('--proxy', metavar=metavar) + @force_not_colorized def test_help_with_metavar(self): help_text = self.parser.format_help() self.assertEqual(help_text, textwrap.dedent('''\ @@ -6754,6 +7016,99 @@ def test_os_error(self): self.parser.parse_args, ['@no-such-file']) +@force_not_colorized_test_class +class TestProgName(TestCase): + source = textwrap.dedent('''\ + import argparse + parser = argparse.ArgumentParser() + parser.parse_args() + ''') + + def setUp(self): + self.dirname = 'package' + os_helper.FS_NONASCII + self.addCleanup(os_helper.rmtree, self.dirname) + os.mkdir(self.dirname) + + def make_script(self, dirname, basename, *, compiled=False): + script_name = script_helper.make_script(dirname, basename, self.source) + if not compiled: + return script_name + py_compile.compile(script_name, doraise=True) + os.remove(script_name) + pyc_file = import_helper.make_legacy_pyc(script_name) + return pyc_file + + def make_zip_script(self, script_name, name_in_zip=None): + zip_name, _ = script_helper.make_zip_script(self.dirname, 'test_zip', + script_name, name_in_zip) + return zip_name + + def check_usage(self, expected, *args, **kwargs): + res = script_helper.assert_python_ok('-Xutf8', *args, '-h', **kwargs) + self.assertEqual(os.fsdecode(res.out.splitlines()[0]), + f'usage: {expected} [-h]') + + def test_script(self, compiled=False): + basename = os_helper.TESTFN + script_name = self.make_script(self.dirname, basename, compiled=compiled) + self.check_usage(os.path.basename(script_name), script_name, '-h') + + def test_script_compiled(self): + self.test_script(compiled=True) + + def test_directory(self, compiled=False): + dirname = os.path.join(self.dirname, os_helper.TESTFN) + os.mkdir(dirname) + self.make_script(dirname, '__main__', compiled=compiled) + self.check_usage(f'{py} {dirname}', dirname) + dirname2 = os.path.join(os.curdir, dirname) + self.check_usage(f'{py} {dirname2}', dirname2) + + def test_directory_compiled(self): + self.test_directory(compiled=True) + + def test_module(self, compiled=False): + basename = 'module' + os_helper.FS_NONASCII + modulename = f'{self.dirname}.{basename}' + self.make_script(self.dirname, basename, compiled=compiled) + self.check_usage(f'{py} -m {modulename}', + '-m', modulename, PYTHONPATH=os.curdir) + + def test_module_compiled(self): + self.test_module(compiled=True) + + def test_package(self, compiled=False): + basename = 'subpackage' + os_helper.FS_NONASCII + packagename = f'{self.dirname}.{basename}' + subdirname = os.path.join(self.dirname, basename) + os.mkdir(subdirname) + self.make_script(subdirname, '__main__', compiled=compiled) + self.check_usage(f'{py} -m {packagename}', + '-m', packagename, PYTHONPATH=os.curdir) + self.check_usage(f'{py} -m {packagename}', + '-m', packagename + '.__main__', PYTHONPATH=os.curdir) + + def test_package_compiled(self): + self.test_package(compiled=True) + + def test_zipfile(self, compiled=False): + script_name = self.make_script(self.dirname, '__main__', compiled=compiled) + zip_name = self.make_zip_script(script_name) + self.check_usage(f'{py} {zip_name}', zip_name) + + def test_zipfile_compiled(self): + self.test_zipfile(compiled=True) + + def test_directory_in_zipfile(self, compiled=False): + script_name = self.make_script(self.dirname, '__main__', compiled=compiled) + name_in_zip = 'package/subpackage/__main__' + ('.py', '.pyc')[compiled] + zip_name = self.make_zip_script(script_name, name_in_zip) + dirname = os.path.join(zip_name, 'package', 'subpackage') + self.check_usage(f'{py} {dirname}', dirname) + + def test_directory_in_zipfile_compiled(self): + self.test_directory_in_zipfile(compiled=True) + # ================= # Translation tests # ================= @@ -6764,6 +7119,257 @@ def test_translations(self): self.assertMsgidsEqual(argparse) +# =========== +# Color tests +# =========== + + +class TestColorized(TestCase): + maxDiff = None + + def setUp(self): + super().setUp() + # Ensure color even if ran with NO_COLOR=1 + self.enterContext(swap_attr(_colorize, 'can_colorize', + lambda *args, **kwargs: True)) + self.theme = _colorize.get_theme(force_color=True).argparse + + def test_argparse_color(self): + # Arrange: create a parser with a bit of everything + parser = argparse.ArgumentParser( + color=True, + description="Colorful help", + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + prefix_chars="-+", + prog="PROG", + ) + group = parser.add_mutually_exclusive_group() + group.add_argument( + "-v", "--verbose", action="store_true", help="more spam" + ) + group.add_argument( + "-q", "--quiet", action="store_true", help="less spam" + ) + parser.add_argument("x", type=int, help="the base") + parser.add_argument( + "y", type=int, help="the exponent", deprecated=True + ) + parser.add_argument( + "this_indeed_is_a_very_long_action_name", + type=int, + help="the exponent", + ) + parser.add_argument( + "-o", "--optional1", action="store_true", deprecated=True + ) + parser.add_argument("--optional2", help="pick one") + parser.add_argument("--optional3", choices=("X", "Y", "Z")) + parser.add_argument( + "--optional4", choices=("X", "Y", "Z"), help="pick one" + ) + parser.add_argument( + "--optional5", choices=("X", "Y", "Z"), help="pick one" + ) + parser.add_argument( + "--optional6", choices=("X", "Y", "Z"), help="pick one" + ) + parser.add_argument( + "-p", + "--optional7", + choices=("Aaaaa", "Bbbbb", "Ccccc", "Ddddd"), + help="pick one", + ) + + parser.add_argument("+f") + parser.add_argument("++bar") + parser.add_argument("-+baz") + parser.add_argument("-c", "--count") + + subparsers = parser.add_subparsers( + title="subcommands", + description="valid subcommands", + help="additional help", + ) + subparsers.add_parser("sub1", deprecated=True, help="sub1 help") + sub2 = subparsers.add_parser("sub2", deprecated=True, help="sub2 help") + sub2.add_argument("--baz", choices=("X", "Y", "Z"), help="baz help") + + prog = self.theme.prog + heading = self.theme.heading + long = self.theme.summary_long_option + short = self.theme.summary_short_option + label = self.theme.summary_label + pos = self.theme.summary_action + long_b = self.theme.long_option + short_b = self.theme.short_option + label_b = self.theme.label + pos_b = self.theme.action + reset = self.theme.reset + + # Act + help_text = parser.format_help() + + # Assert + self.assertEqual( + help_text, + textwrap.dedent( + f"""\ + {heading}usage: {reset}{prog}PROG{reset} [{short}-h{reset}] [{short}-v{reset} | {short}-q{reset}] [{short}-o{reset}] [{long}--optional2 {label}OPTIONAL2{reset}] [{long}--optional3 {label}{{X,Y,Z}}{reset}] + [{long}--optional4 {label}{{X,Y,Z}}{reset}] [{long}--optional5 {label}{{X,Y,Z}}{reset}] [{long}--optional6 {label}{{X,Y,Z}}{reset}] + [{short}-p {label}{{Aaaaa,Bbbbb,Ccccc,Ddddd}}{reset}] [{short}+f {label}F{reset}] [{long}++bar {label}BAR{reset}] [{long}-+baz {label}BAZ{reset}] + [{short}-c {label}COUNT{reset}] + {pos}x{reset} {pos}y{reset} {pos}this_indeed_is_a_very_long_action_name{reset} {pos}{{sub1,sub2}} ...{reset} + + Colorful help + + {heading}positional arguments:{reset} + {pos_b}x{reset} the base + {pos_b}y{reset} the exponent + {pos_b}this_indeed_is_a_very_long_action_name{reset} + the exponent + + {heading}options:{reset} + {short_b}-h{reset}, {long_b}--help{reset} show this help message and exit + {short_b}-v{reset}, {long_b}--verbose{reset} more spam (default: False) + {short_b}-q{reset}, {long_b}--quiet{reset} less spam (default: False) + {short_b}-o{reset}, {long_b}--optional1{reset} + {long_b}--optional2{reset} {label_b}OPTIONAL2{reset} + pick one (default: None) + {long_b}--optional3{reset} {label_b}{{X,Y,Z}}{reset} + {long_b}--optional4{reset} {label_b}{{X,Y,Z}}{reset} pick one (default: None) + {long_b}--optional5{reset} {label_b}{{X,Y,Z}}{reset} pick one (default: None) + {long_b}--optional6{reset} {label_b}{{X,Y,Z}}{reset} pick one (default: None) + {short_b}-p{reset}, {long_b}--optional7{reset} {label_b}{{Aaaaa,Bbbbb,Ccccc,Ddddd}}{reset} + pick one (default: None) + {short_b}+f{reset} {label_b}F{reset} + {long_b}++bar{reset} {label_b}BAR{reset} + {long_b}-+baz{reset} {label_b}BAZ{reset} + {short_b}-c{reset}, {long_b}--count{reset} {label_b}COUNT{reset} + + {heading}subcommands:{reset} + valid subcommands + + {pos_b}{{sub1,sub2}}{reset} additional help + {pos_b}sub1{reset} sub1 help + {pos_b}sub2{reset} sub2 help + """ + ), + ) + + def test_argparse_color_usage(self): + # Arrange + parser = argparse.ArgumentParser( + add_help=False, + color=True, + description="Test prog and usage colors", + prog="PROG", + usage="[prefix] %(prog)s [suffix]", + ) + heading = self.theme.heading + prog = self.theme.prog + reset = self.theme.reset + usage = self.theme.prog_extra + + # Act + help_text = parser.format_help() + + # Assert + self.assertEqual( + help_text, + textwrap.dedent( + f"""\ + {heading}usage: {reset}{usage}[prefix] {prog}PROG{reset}{usage} [suffix]{reset} + + Test prog and usage colors + """ + ), + ) + + def test_custom_formatter_function(self): + def custom_formatter(prog): + return argparse.RawTextHelpFormatter(prog, indent_increment=5) + + parser = argparse.ArgumentParser( + prog="PROG", + prefix_chars="-+", + formatter_class=custom_formatter, + color=True, + ) + parser.add_argument('+f', '++foo', help="foo help") + parser.add_argument('spam', help="spam help") + + prog = self.theme.prog + heading = self.theme.heading + short = self.theme.summary_short_option + label = self.theme.summary_label + pos = self.theme.summary_action + long_b = self.theme.long_option + short_b = self.theme.short_option + label_b = self.theme.label + pos_b = self.theme.action + reset = self.theme.reset + + parser_help = parser.format_help() + self.assertEqual(parser_help, textwrap.dedent(f'''\ + {heading}usage: {reset}{prog}PROG{reset} [{short}-h{reset}] [{short}+f {label}FOO{reset}] {pos}spam{reset} + + {heading}positional arguments:{reset} + {pos_b}spam{reset} spam help + + {heading}options:{reset} + {short_b}-h{reset}, {long_b}--help{reset} show this help message and exit + {short_b}+f{reset}, {long_b}++foo{reset} {label_b}FOO{reset} foo help + ''')) + + def test_custom_formatter_class(self): + class CustomFormatter(argparse.RawTextHelpFormatter): + def __init__(self, prog): + super().__init__(prog, indent_increment=5) + + parser = argparse.ArgumentParser( + prog="PROG", + prefix_chars="-+", + formatter_class=CustomFormatter, + color=True, + ) + parser.add_argument('+f', '++foo', help="foo help") + parser.add_argument('spam', help="spam help") + + prog = self.theme.prog + heading = self.theme.heading + short = self.theme.summary_short_option + label = self.theme.summary_label + pos = self.theme.summary_action + long_b = self.theme.long_option + short_b = self.theme.short_option + label_b = self.theme.label + pos_b = self.theme.action + reset = self.theme.reset + + parser_help = parser.format_help() + self.assertEqual(parser_help, textwrap.dedent(f'''\ + {heading}usage: {reset}{prog}PROG{reset} [{short}-h{reset}] [{short}+f {label}FOO{reset}] {pos}spam{reset} + + {heading}positional arguments:{reset} + {pos_b}spam{reset} spam help + + {heading}options:{reset} + {short_b}-h{reset}, {long_b}--help{reset} show this help message and exit + {short_b}+f{reset}, {long_b}++foo{reset} {label_b}FOO{reset} foo help + ''')) + + def test_subparser_prog_is_stored_without_color(self): + parser = argparse.ArgumentParser(prog='complex', color=True) + sub = parser.add_subparsers(dest='command') + demo_parser = sub.add_parser('demo') + + self.assertNotIn('\x1b[', demo_parser.prog) + + demo_parser.color = False + help_text = demo_parser.format_help() + self.assertNotIn('\x1b[', help_text) + + def tearDownModule(): # Remove global references to avoid looking like we have refleaks. RFile.seen = {} diff --git a/Lib/test/test_array.py b/Lib/test/test_array.py index 0c20e27cfda..0376d7ff9b7 100644 --- a/Lib/test/test_array.py +++ b/Lib/test/test_array.py @@ -176,8 +176,6 @@ def test_numbers(self): self.assertEqual(a, b, msg="{0!r} != {1!r}; testcase={2!r}".format(a, b, testcase)) - # TODO: RUSTPYTHON - requires UTF-32 encoding support in codecs and proper array reconstructor implementation - @unittest.expectedFailure def test_unicode(self): teststr = "Bonne Journ\xe9e \U0002030a\U00020347" testcases = ( diff --git a/Lib/test/test_ast/snippets.py b/Lib/test/test_ast/snippets.py index 28d32b2941f..b76f98901d2 100644 --- a/Lib/test/test_ast/snippets.py +++ b/Lib/test/test_ast/snippets.py @@ -364,6 +364,12 @@ "f'{a:.2f}'", "f'{a!r}'", "f'foo({a})'", + # TemplateStr and Interpolation + "t'{a}'", + "t'{a:.2f}'", + "t'{a!r}'", + "t'{a!r:.2f}'", + "t'foo({a})'", ] @@ -597,5 +603,10 @@ def main(): ('Expression', ('JoinedStr', (1, 0, 1, 10), [('FormattedValue', (1, 2, 1, 9), ('Name', (1, 3, 1, 4), 'a', ('Load',)), -1, ('JoinedStr', (1, 4, 1, 8), [('Constant', (1, 5, 1, 8), '.2f', None)]))])), ('Expression', ('JoinedStr', (1, 0, 1, 8), [('FormattedValue', (1, 2, 1, 7), ('Name', (1, 3, 1, 4), 'a', ('Load',)), 114, None)])), ('Expression', ('JoinedStr', (1, 0, 1, 11), [('Constant', (1, 2, 1, 6), 'foo(', None), ('FormattedValue', (1, 6, 1, 9), ('Name', (1, 7, 1, 8), 'a', ('Load',)), -1, None), ('Constant', (1, 9, 1, 10), ')', None)])), +('Expression', ('TemplateStr', (1, 0, 1, 6), [('Interpolation', (1, 2, 1, 5), ('Name', (1, 3, 1, 4), 'a', ('Load',)), 'a', -1, None)])), +('Expression', ('TemplateStr', (1, 0, 1, 10), [('Interpolation', (1, 2, 1, 9), ('Name', (1, 3, 1, 4), 'a', ('Load',)), 'a', -1, ('JoinedStr', (1, 4, 1, 8), [('Constant', (1, 5, 1, 8), '.2f', None)]))])), +('Expression', ('TemplateStr', (1, 0, 1, 8), [('Interpolation', (1, 2, 1, 7), ('Name', (1, 3, 1, 4), 'a', ('Load',)), 'a', 114, None)])), +('Expression', ('TemplateStr', (1, 0, 1, 12), [('Interpolation', (1, 2, 1, 11), ('Name', (1, 3, 1, 4), 'a', ('Load',)), 'a', 114, ('JoinedStr', (1, 6, 1, 10), [('Constant', (1, 7, 1, 10), '.2f', None)]))])), +('Expression', ('TemplateStr', (1, 0, 1, 11), [('Constant', (1, 2, 1, 6), 'foo(', None), ('Interpolation', (1, 6, 1, 9), ('Name', (1, 7, 1, 8), 'a', ('Load',)), 'a', -1, None), ('Constant', (1, 9, 1, 10), ')', None)])), ] main() diff --git a/Lib/test/test_ast/test_ast.py b/Lib/test/test_ast/test_ast.py index 3ff1cea2bf8..9100cf44335 100644 --- a/Lib/test/test_ast/test_ast.py +++ b/Lib/test/test_ast/test_ast.py @@ -1,34 +1,59 @@ +import _ast_unparse import ast import builtins +import contextlib import copy import dis import enum +import itertools import os import re import sys +import tempfile import textwrap import types import unittest -import warnings import weakref -from functools import partial +from io import StringIO +from pathlib import Path from textwrap import dedent - try: import _testinternalcapi except ImportError: _testinternalcapi = None from test import support -from test.support.import_helper import import_fresh_module -from test.support import os_helper, script_helper +from test.support import os_helper +from test.support import skip_emscripten_stack_overflow, skip_wasi_stack_overflow from test.support.ast_helper import ASTTestMixin +from test.support.import_helper import ensure_lazy_imports from test.test_ast.utils import to_tuple from test.test_ast.snippets import ( eval_tests, eval_results, exec_tests, exec_results, single_tests, single_results ) +STDLIB = os.path.dirname(ast.__file__) +STDLIB_FILES = [fn for fn in os.listdir(STDLIB) if fn.endswith(".py")] +STDLIB_FILES.extend(["test/test_grammar.py", "test/test_unpack_ex.py"]) + +AST_REPR_DATA_FILE = Path(__file__).parent / "data" / "ast_repr.txt" + +def ast_repr_get_test_cases() -> list[str]: + return exec_tests + eval_tests + + +def ast_repr_update_snapshots() -> None: + data = [repr(ast.parse(test)) for test in ast_repr_get_test_cases()] + AST_REPR_DATA_FILE.write_text("\n".join(data)) + + +class LazyImportTest(unittest.TestCase): + @support.cpython_only + def test_lazy_import(self): + ensure_lazy_imports("ast", {"contextlib", "enum", "inspect", "re", "collections", "argparse"}) + + class AST_Tests(unittest.TestCase): maxDiff = None @@ -37,7 +62,7 @@ def _is_ast_node(self, name, node): return False if "ast" not in node.__module__: return False - return name != "AST" and name[0].isupper() + return name != 'AST' and name[0].isupper() def _assertTrueorder(self, ast_node, parent_pos): if not isinstance(ast_node, ast.AST) or ast_node._fields is None: @@ -50,7 +75,7 @@ def _assertTrueorder(self, ast_node, parent_pos): value = getattr(ast_node, name) if isinstance(value, list): first_pos = parent_pos - if value and name == "decorator_list": + if value and name == 'decorator_list': first_pos = (value[0].lineno, value[0].col_offset) for child in value: self._assertTrueorder(child, first_pos) @@ -72,7 +97,7 @@ def test_AST_objects(self): # "ast.AST constructor takes 0 positional arguments" ast.AST(2) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_AST_fields_NULL_check(self): # See: https://github.com/python/cpython/issues/126105 old_value = ast.AST._fields @@ -90,11 +115,10 @@ def cleanup(): with self.assertRaisesRegex(AttributeError, msg): ast.AST() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_AST_garbage_collection(self): class X: pass - a = ast.AST() a.x = X() a.x.a = a @@ -103,13 +127,11 @@ class X: support.gc_collect() self.assertIsNone(ref()) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_snippets(self): - for input, output, kind in ( - (exec_tests, exec_results, "exec"), - (single_tests, single_results, "single"), - (eval_tests, eval_results, "eval"), - ): + for input, output, kind in ((exec_tests, exec_results, "exec"), + (single_tests, single_results, "single"), + (eval_tests, eval_results, "eval")): for i, o in zip(input, output): with self.subTest(action="parsing", input=i): ast_tree = compile(i, "?", kind, ast.PyCF_ONLY_AST) @@ -118,18 +140,24 @@ def test_snippets(self): with self.subTest(action="compiling", input=i, kind=kind): compile(ast_tree, "?", kind) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_ast_validation(self): # compile() is the only function that calls PyAST_Validate snippets_to_validate = exec_tests + single_tests + eval_tests for snippet in snippets_to_validate: tree = ast.parse(snippet) - compile(tree, "", "exec") + compile(tree, '', 'exec') + + def test_parse_invalid_ast(self): + # see gh-130139 + for optval in (-1, 0, 1, 2): + self.assertRaises(TypeError, ast.parse, ast.Constant(42), + optimize=optval) - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags def test_optimization_levels__debug__(self): - cases = [(-1, "__debug__"), (0, "__debug__"), (1, False), (2, False)] - for optval, expected in cases: + cases = [(-1, '__debug__'), (0, '__debug__'), (1, False), (2, False)] + for (optval, expected) in cases: with self.subTest(optval=optval, expected=expected): res1 = ast.parse("__debug__", optimize=optval) res2 = ast.parse(ast.parse("__debug__"), optimize=optval) @@ -142,33 +170,11 @@ def test_optimization_levels__debug__(self): self.assertIsInstance(res.body[0].value, ast.Name) self.assertEqual(res.body[0].value.id, expected) - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags - def test_optimization_levels_const_folding(self): - folded = ("Expr", (1, 0, 1, 5), ("Constant", (1, 0, 1, 5), 3, None)) - not_folded = ( - "Expr", - (1, 0, 1, 5), - ( - "BinOp", - (1, 0, 1, 5), - ("Constant", (1, 0, 1, 1), 1, None), - ("Add",), - ("Constant", (1, 4, 1, 5), 2, None), - ), - ) - - cases = [(-1, not_folded), (0, not_folded), (1, folded), (2, folded)] - for optval, expected in cases: - with self.subTest(optval=optval): - tree1 = ast.parse("1 + 2", optimize=optval) - tree2 = ast.parse(ast.parse("1 + 2"), optimize=optval) - for tree in [tree1, tree2]: - res = to_tuple(tree.body[0]) - self.assertEqual(res, expected) - - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_invalid_position_information(self): - invalid_linenos = [(10, 1), (-10, -11), (10, -11), (-5, -2), (-5, 1)] + invalid_linenos = [ + (10, 1), (-10, -11), (10, -11), (-5, -2), (-5, 1) + ] for lineno, end_lineno in invalid_linenos: with self.subTest(f"Check invalid linenos {lineno}:{end_lineno}"): @@ -177,36 +183,25 @@ def test_invalid_position_information(self): tree.body[0].lineno = lineno tree.body[0].end_lineno = end_lineno with self.assertRaises(ValueError): - compile(tree, "", "exec") + compile(tree, '', 'exec') - invalid_col_offsets = [(10, 1), (-10, -11), (10, -11), (-5, -2), (-5, 1)] + invalid_col_offsets = [ + (10, 1), (-10, -11), (10, -11), (-5, -2), (-5, 1) + ] for col_offset, end_col_offset in invalid_col_offsets: - with self.subTest( - f"Check invalid col_offset {col_offset}:{end_col_offset}" - ): + with self.subTest(f"Check invalid col_offset {col_offset}:{end_col_offset}"): snippet = "a = 1" tree = ast.parse(snippet) tree.body[0].col_offset = col_offset tree.body[0].end_col_offset = end_col_offset with self.assertRaises(ValueError): - compile(tree, "", "exec") + compile(tree, '', 'exec') def test_compilation_of_ast_nodes_with_default_end_position_values(self): - tree = ast.Module( - body=[ - ast.Import( - names=[ast.alias(name="builtins", lineno=1, col_offset=0)], - lineno=1, - col_offset=0, - ), - ast.Import( - names=[ast.alias(name="traceback", lineno=0, col_offset=0)], - lineno=0, - col_offset=1, - ), - ], - type_ignores=[], - ) + tree = ast.Module(body=[ + ast.Import(names=[ast.alias(name='builtins', lineno=1, col_offset=0)], lineno=1, col_offset=0), + ast.Import(names=[ast.alias(name='traceback', lineno=0, col_offset=0)], lineno=0, col_offset=1) + ], type_ignores=[]) # Check that compilation doesn't crash. Note: this may crash explicitly only on debug mode. compile(tree, "", "exec") @@ -231,6 +226,133 @@ def test_negative_locations_for_compile(self): # This also must not crash: ast.parse(tree, optimize=2) + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_docstring_optimization_single_node(self): + # https://github.com/python/cpython/issues/137308 + class_example1 = textwrap.dedent(''' + class A: + """Docstring""" + ''') + class_example2 = textwrap.dedent(''' + class A: + """ + Docstring""" + ''') + def_example1 = textwrap.dedent(''' + def some(): + """Docstring""" + ''') + def_example2 = textwrap.dedent(''' + def some(): + """Docstring + """ + ''') + async_def_example1 = textwrap.dedent(''' + async def some(): + """Docstring""" + ''') + async_def_example2 = textwrap.dedent(''' + async def some(): + """ + Docstring + """ + ''') + for code in [ + class_example1, + class_example2, + def_example1, + def_example2, + async_def_example1, + async_def_example2, + ]: + for opt_level in [0, 1, 2]: + with self.subTest(code=code, opt_level=opt_level): + mod = ast.parse(code, optimize=opt_level) + self.assertEqual(len(mod.body[0].body), 1) + if opt_level == 2: + pass_stmt = mod.body[0].body[0] + self.assertIsInstance(pass_stmt, ast.Pass) + self.assertEqual( + vars(pass_stmt), + { + 'lineno': 3, + 'col_offset': 4, + 'end_lineno': 3, + 'end_col_offset': 8, + }, + ) + else: + self.assertIsInstance(mod.body[0].body[0], ast.Expr) + self.assertIsInstance( + mod.body[0].body[0].value, + ast.Constant, + ) + + compile(code, "a", "exec") + compile(code, "a", "exec", optimize=opt_level) + compile(mod, "a", "exec") + compile(mod, "a", "exec", optimize=opt_level) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_docstring_optimization_multiple_nodes(self): + # https://github.com/python/cpython/issues/137308 + class_example = textwrap.dedent( + """ + class A: + ''' + Docstring + ''' + x = 1 + """ + ) + + def_example = textwrap.dedent( + """ + def some(): + ''' + Docstring + + ''' + x = 1 + """ + ) + + async_def_example = textwrap.dedent( + """ + async def some(): + + '''Docstring + + ''' + x = 1 + """ + ) + + for code in [ + class_example, + def_example, + async_def_example, + ]: + for opt_level in [0, 1, 2]: + with self.subTest(code=code, opt_level=opt_level): + mod = ast.parse(code, optimize=opt_level) + if opt_level == 2: + self.assertNotIsInstance( + mod.body[0].body[0], + (ast.Pass, ast.Expr), + ) + else: + self.assertIsInstance(mod.body[0].body[0], ast.Expr) + self.assertIsInstance( + mod.body[0].body[0].value, + ast.Constant, + ) + + compile(code, "a", "exec") + compile(code, "a", "exec", optimize=opt_level) + compile(mod, "a", "exec") + compile(mod, "a", "exec", optimize=opt_level) + def test_slice(self): slc = ast.parse("x[::]").body[0].value.slice self.assertIsNone(slc.upper) @@ -251,7 +373,7 @@ def test_alias(self): im = ast.parse("from bar import y").body[0] self.assertEqual(len(im.names), 1) alias = im.names[0] - self.assertEqual(alias.name, "y") + self.assertEqual(alias.name, 'y') self.assertIsNone(alias.asname) self.assertEqual(alias.lineno, 1) self.assertEqual(alias.end_lineno, 1) @@ -260,7 +382,7 @@ def test_alias(self): im = ast.parse("from bar import *").body[0] alias = im.names[0] - self.assertEqual(alias.name, "*") + self.assertEqual(alias.name, '*') self.assertIsNone(alias.asname) self.assertEqual(alias.lineno, 1) self.assertEqual(alias.end_lineno, 1) @@ -286,45 +408,18 @@ def test_alias(self): self.assertEqual(alias.end_col_offset, 17) def test_base_classes(self): - self.assertTrue(issubclass(ast.For, ast.stmt)) - self.assertTrue(issubclass(ast.Name, ast.expr)) - self.assertTrue(issubclass(ast.stmt, ast.AST)) - self.assertTrue(issubclass(ast.expr, ast.AST)) - self.assertTrue(issubclass(ast.comprehension, ast.AST)) - self.assertTrue(issubclass(ast.Gt, ast.AST)) - - def test_import_deprecated(self): - ast = import_fresh_module("ast") - depr_regex = ( - r"ast\.{} is deprecated and will be removed in Python 3.14; " - r"use ast\.Constant instead" - ) - for name in "Num", "Str", "Bytes", "NameConstant", "Ellipsis": - with self.assertWarnsRegex(DeprecationWarning, depr_regex.format(name)): - getattr(ast, name) - - def test_field_attr_existence_deprecated(self): - with warnings.catch_warnings(): - warnings.filterwarnings("ignore", "", DeprecationWarning) - from ast import Num, Str, Bytes, NameConstant, Ellipsis - - for name in ("Num", "Str", "Bytes", "NameConstant", "Ellipsis"): - item = getattr(ast, name) - if self._is_ast_node(name, item): - with self.subTest(item): - with self.assertWarns(DeprecationWarning): - x = item() - if isinstance(x, ast.AST): - self.assertIs(type(x._fields), tuple) - - @unittest.expectedFailure # TODO: RUSTPYTHON; type object 'Module' has no attribute '__annotations__' + self.assertIsSubclass(ast.For, ast.stmt) + self.assertIsSubclass(ast.Name, ast.expr) + self.assertIsSubclass(ast.stmt, ast.AST) + self.assertIsSubclass(ast.expr, ast.AST) + self.assertIsSubclass(ast.comprehension, ast.AST) + self.assertIsSubclass(ast.Gt, ast.AST) + + @unittest.expectedFailure # TODO: RUSTPYTHON; type object 'Module' has no attribute '__annotations__' def test_field_attr_existence(self): for name, item in ast.__dict__.items(): - # These emit DeprecationWarnings - if name in {"Num", "Str", "Bytes", "NameConstant", "Ellipsis"}: - continue # constructor has a different signature - if name == "Index": + if name == 'Index': continue if self._is_ast_node(name, item): x = self._construct_ast_class(item) @@ -335,42 +430,29 @@ def _construct_ast_class(self, cls): kwargs = {} for name, typ in cls.__annotations__.items(): if typ is str: - kwargs[name] = "capybara" + kwargs[name] = 'capybara' elif typ is int: kwargs[name] = 42 elif typ is object: - kwargs[name] = b"capybara" + kwargs[name] = b'capybara' elif isinstance(typ, type) and issubclass(typ, ast.AST): kwargs[name] = self._construct_ast_class(typ) return cls(**kwargs) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_arguments(self): x = ast.arguments() - self.assertEqual( - x._fields, - ( - "posonlyargs", - "args", - "vararg", - "kwonlyargs", - "kw_defaults", - "kwarg", - "defaults", - ), - ) - self.assertEqual( - x.__annotations__, - { - "posonlyargs": list[ast.arg], - "args": list[ast.arg], - "vararg": ast.arg | None, - "kwonlyargs": list[ast.arg], - "kw_defaults": list[ast.expr], - "kwarg": ast.arg | None, - "defaults": list[ast.expr], - }, - ) + self.assertEqual(x._fields, ('posonlyargs', 'args', 'vararg', 'kwonlyargs', + 'kw_defaults', 'kwarg', 'defaults')) + self.assertEqual(ast.arguments.__annotations__, { + 'posonlyargs': list[ast.arg], + 'args': list[ast.arg], + 'vararg': ast.arg | None, + 'kwonlyargs': list[ast.arg], + 'kw_defaults': list[ast.expr], + 'kwarg': ast.arg | None, + 'defaults': list[ast.expr], + }) self.assertEqual(x.args, []) self.assertIsNone(x.vararg) @@ -379,117 +461,17 @@ def test_arguments(self): self.assertEqual(x.args, 2) self.assertEqual(x.vararg, 3) - def test_field_attr_writable_deprecated(self): - with warnings.catch_warnings(): - warnings.filterwarnings("ignore", "", DeprecationWarning) - x = ast.Num() - # We can assign to _fields - x._fields = 666 - self.assertEqual(x._fields, 666) - def test_field_attr_writable(self): x = ast.Constant(1) # We can assign to _fields x._fields = 666 self.assertEqual(x._fields, 666) - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_classattrs_deprecated(self): - with warnings.catch_warnings(): - warnings.filterwarnings("ignore", "", DeprecationWarning) - from ast import Num, Str, Bytes, NameConstant, Ellipsis - - with warnings.catch_warnings(record=True) as wlog: - warnings.filterwarnings("always", "", DeprecationWarning) - x = ast.Num() - self.assertEqual(x._fields, ("value", "kind")) - - with self.assertRaises(AttributeError): - x.value - - with self.assertRaises(AttributeError): - x.n - - x = ast.Num(42) - self.assertEqual(x.value, 42) - self.assertEqual(x.n, 42) - - with self.assertRaises(AttributeError): - x.lineno - - with self.assertRaises(AttributeError): - x.foobar - - x = ast.Num(lineno=2) - self.assertEqual(x.lineno, 2) - - x = ast.Num(42, lineno=0) - self.assertEqual(x.lineno, 0) - self.assertEqual(x._fields, ("value", "kind")) - self.assertEqual(x.value, 42) - self.assertEqual(x.n, 42) - - self.assertRaises(TypeError, ast.Num, 1, None, 2) - self.assertRaises(TypeError, ast.Num, 1, None, 2, lineno=0) - - # Arbitrary keyword arguments are supported - self.assertEqual(ast.Num(1, foo="bar").foo, "bar") - - with self.assertRaisesRegex( - TypeError, "Num got multiple values for argument 'n'" - ): - ast.Num(1, n=2) - - self.assertEqual(ast.Num(42).n, 42) - self.assertEqual(ast.Num(4.25).n, 4.25) - self.assertEqual(ast.Num(4.25j).n, 4.25j) - self.assertEqual(ast.Str("42").s, "42") - self.assertEqual(ast.Bytes(b"42").s, b"42") - self.assertIs(ast.NameConstant(True).value, True) - self.assertIs(ast.NameConstant(False).value, False) - self.assertIs(ast.NameConstant(None).value, None) - - self.assertEqual( - [str(w.message) for w in wlog], - [ - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "Constant.__init__ missing 1 required positional argument: 'value'. This will become " - "an error in Python 3.15.", - "Attribute n is deprecated and will be removed in Python 3.14; use value instead", - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "Attribute n is deprecated and will be removed in Python 3.14; use value instead", - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "Constant.__init__ missing 1 required positional argument: 'value'. This will become " - "an error in Python 3.15.", - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "Attribute n is deprecated and will be removed in Python 3.14; use value instead", - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "Constant.__init__ got an unexpected keyword argument 'foo'. Support for " - "arbitrary keyword arguments is deprecated and will be removed in Python " - "3.15.", - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "Attribute n is deprecated and will be removed in Python 3.14; use value instead", - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "Attribute n is deprecated and will be removed in Python 3.14; use value instead", - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "Attribute n is deprecated and will be removed in Python 3.14; use value instead", - "ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "Attribute s is deprecated and will be removed in Python 3.14; use value instead", - "ast.Bytes is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "Attribute s is deprecated and will be removed in Python 3.14; use value instead", - "ast.NameConstant is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "ast.NameConstant is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "ast.NameConstant is deprecated and will be removed in Python 3.14; use ast.Constant instead", - ], - ) - - @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_classattrs(self): with self.assertWarns(DeprecationWarning): x = ast.Constant() - self.assertEqual(x._fields, ("value", "kind")) + self.assertEqual(x._fields, ('value', 'kind')) with self.assertRaises(AttributeError): x.value @@ -508,7 +490,7 @@ def test_classattrs(self): x = ast.Constant(42, lineno=0) self.assertEqual(x.lineno, 0) - self.assertEqual(x._fields, ("value", "kind")) + self.assertEqual(x._fields, ('value', 'kind')) self.assertEqual(x.value, 42) self.assertRaises(TypeError, ast.Constant, 1, None, 2) @@ -516,234 +498,32 @@ def test_classattrs(self): # Arbitrary keyword arguments are supported (but deprecated) with self.assertWarns(DeprecationWarning): - self.assertEqual(ast.Constant(1, foo="bar").foo, "bar") + self.assertEqual(ast.Constant(1, foo='bar').foo, 'bar') - with self.assertRaisesRegex( - TypeError, "Constant got multiple values for argument 'value'" - ): + with self.assertRaisesRegex(TypeError, "Constant got multiple values for argument 'value'"): ast.Constant(1, value=2) self.assertEqual(ast.Constant(42).value, 42) self.assertEqual(ast.Constant(4.25).value, 4.25) self.assertEqual(ast.Constant(4.25j).value, 4.25j) - self.assertEqual(ast.Constant("42").value, "42") - self.assertEqual(ast.Constant(b"42").value, b"42") + self.assertEqual(ast.Constant('42').value, '42') + self.assertEqual(ast.Constant(b'42').value, b'42') self.assertIs(ast.Constant(True).value, True) self.assertIs(ast.Constant(False).value, False) self.assertIs(ast.Constant(None).value, None) self.assertIs(ast.Constant(...).value, ...) - def test_realtype(self): - with warnings.catch_warnings(): - warnings.filterwarnings("ignore", "", DeprecationWarning) - from ast import Num, Str, Bytes, NameConstant, Ellipsis - - with warnings.catch_warnings(record=True) as wlog: - warnings.filterwarnings("always", "", DeprecationWarning) - self.assertIs(type(ast.Num(42)), ast.Constant) - self.assertIs(type(ast.Num(4.25)), ast.Constant) - self.assertIs(type(ast.Num(4.25j)), ast.Constant) - self.assertIs(type(ast.Str("42")), ast.Constant) - self.assertIs(type(ast.Bytes(b"42")), ast.Constant) - self.assertIs(type(ast.NameConstant(True)), ast.Constant) - self.assertIs(type(ast.NameConstant(False)), ast.Constant) - self.assertIs(type(ast.NameConstant(None)), ast.Constant) - self.assertIs(type(ast.Ellipsis()), ast.Constant) - - self.assertEqual( - [str(w.message) for w in wlog], - [ - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "ast.Bytes is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "ast.NameConstant is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "ast.NameConstant is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "ast.NameConstant is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "ast.Ellipsis is deprecated and will be removed in Python 3.14; use ast.Constant instead", - ], - ) - - def test_isinstance(self): - from ast import Constant - - with warnings.catch_warnings(): - warnings.filterwarnings("ignore", "", DeprecationWarning) - from ast import Num, Str, Bytes, NameConstant, Ellipsis - - cls_depr_msg = ( - "ast.{} is deprecated and will be removed in Python 3.14; " - "use ast.Constant instead" - ) - - assertNumDeprecated = partial( - self.assertWarnsRegex, DeprecationWarning, cls_depr_msg.format("Num") - ) - assertStrDeprecated = partial( - self.assertWarnsRegex, DeprecationWarning, cls_depr_msg.format("Str") - ) - assertBytesDeprecated = partial( - self.assertWarnsRegex, DeprecationWarning, cls_depr_msg.format("Bytes") - ) - assertNameConstantDeprecated = partial( - self.assertWarnsRegex, - DeprecationWarning, - cls_depr_msg.format("NameConstant"), - ) - assertEllipsisDeprecated = partial( - self.assertWarnsRegex, DeprecationWarning, cls_depr_msg.format("Ellipsis") - ) - - for arg in 42, 4.2, 4.2j: - with self.subTest(arg=arg): - with assertNumDeprecated(): - n = Num(arg) - with assertNumDeprecated(): - self.assertIsInstance(n, Num) - - with assertStrDeprecated(): - s = Str("42") - with assertStrDeprecated(): - self.assertIsInstance(s, Str) - - with assertBytesDeprecated(): - b = Bytes(b"42") - with assertBytesDeprecated(): - self.assertIsInstance(b, Bytes) - - for arg in True, False, None: - with self.subTest(arg=arg): - with assertNameConstantDeprecated(): - n = NameConstant(arg) - with assertNameConstantDeprecated(): - self.assertIsInstance(n, NameConstant) - - with assertEllipsisDeprecated(): - e = Ellipsis() - with assertEllipsisDeprecated(): - self.assertIsInstance(e, Ellipsis) - - for arg in 42, 4.2, 4.2j: - with self.subTest(arg=arg): - with assertNumDeprecated(): - self.assertIsInstance(Constant(arg), Num) - - with assertStrDeprecated(): - self.assertIsInstance(Constant("42"), Str) - - with assertBytesDeprecated(): - self.assertIsInstance(Constant(b"42"), Bytes) - - for arg in True, False, None: - with self.subTest(arg=arg): - with assertNameConstantDeprecated(): - self.assertIsInstance(Constant(arg), NameConstant) - - with assertEllipsisDeprecated(): - self.assertIsInstance(Constant(...), Ellipsis) - - with assertStrDeprecated(): - s = Str("42") - assertNumDeprecated(self.assertNotIsInstance, s, Num) - assertBytesDeprecated(self.assertNotIsInstance, s, Bytes) - - with assertNumDeprecated(): - n = Num(42) - assertStrDeprecated(self.assertNotIsInstance, n, Str) - assertNameConstantDeprecated(self.assertNotIsInstance, n, NameConstant) - assertEllipsisDeprecated(self.assertNotIsInstance, n, Ellipsis) - - with assertNameConstantDeprecated(): - n = NameConstant(True) - with assertNumDeprecated(): - self.assertNotIsInstance(n, Num) - - with assertNameConstantDeprecated(): - n = NameConstant(False) - with assertNumDeprecated(): - self.assertNotIsInstance(n, Num) - - for arg in "42", True, False: - with self.subTest(arg=arg): - with assertNumDeprecated(): - self.assertNotIsInstance(Constant(arg), Num) - - assertStrDeprecated(self.assertNotIsInstance, Constant(42), Str) - assertBytesDeprecated(self.assertNotIsInstance, Constant("42"), Bytes) - assertNameConstantDeprecated( - self.assertNotIsInstance, Constant(42), NameConstant - ) - assertEllipsisDeprecated(self.assertNotIsInstance, Constant(42), Ellipsis) - assertNumDeprecated(self.assertNotIsInstance, Constant(None), Num) - assertStrDeprecated(self.assertNotIsInstance, Constant(None), Str) - assertBytesDeprecated(self.assertNotIsInstance, Constant(None), Bytes) - assertNameConstantDeprecated( - self.assertNotIsInstance, Constant(1), NameConstant - ) - assertEllipsisDeprecated(self.assertNotIsInstance, Constant(None), Ellipsis) - - class S(str): - pass - - with assertStrDeprecated(): - self.assertIsInstance(Constant(S("42")), Str) - with assertNumDeprecated(): - self.assertNotIsInstance(Constant(S("42")), Num) - - @unittest.expectedFailure # TODO: RUSTPYTHON; will be removed in Python 3.14 - def test_constant_subclasses_deprecated(self): - with warnings.catch_warnings(): - warnings.filterwarnings("ignore", "", DeprecationWarning) - from ast import Num - - with warnings.catch_warnings(record=True) as wlog: - warnings.filterwarnings("always", "", DeprecationWarning) - - class N(ast.Num): - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self.z = "spam" - - class N2(ast.Num): - pass - - n = N(42) - self.assertEqual(n.n, 42) - self.assertEqual(n.z, "spam") - self.assertIs(type(n), N) - self.assertIsInstance(n, N) - self.assertIsInstance(n, ast.Num) - self.assertNotIsInstance(n, N2) - self.assertNotIsInstance(ast.Num(42), N) - n = N(n=42) - self.assertEqual(n.n, 42) - self.assertIs(type(n), N) - - self.assertEqual( - [str(w.message) for w in wlog], - [ - "Attribute n is deprecated and will be removed in Python 3.14; use value instead", - "Attribute n is deprecated and will be removed in Python 3.14; use value instead", - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "Attribute n is deprecated and will be removed in Python 3.14; use value instead", - "Attribute n is deprecated and will be removed in Python 3.14; use value instead", - ], - ) - def test_constant_subclasses(self): class N(ast.Constant): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) - self.z = "spam" - + self.z = 'spam' class N2(ast.Constant): pass n = N(42) self.assertEqual(n.value, 42) - self.assertEqual(n.z, "spam") + self.assertEqual(n.z, 'spam') self.assertEqual(type(n), N) self.assertTrue(isinstance(n, N)) self.assertTrue(isinstance(n, ast.Constant)) @@ -758,12 +538,12 @@ def test_module(self): x = ast.Module(body, []) self.assertEqual(x.body, body) - @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_nodeclasses(self): # Zero arguments constructor explicitly allowed (but deprecated) with self.assertWarns(DeprecationWarning): x = ast.BinOp() - self.assertEqual(x._fields, ("left", "op", "right")) + self.assertEqual(x._fields, ('left', 'op', 'right')) # Random attribute allowed too x.foobarbaz = 5 @@ -810,15 +590,15 @@ def test_no_fields(self): x = ast.Sub() self.assertEqual(x._fields, ()) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_invalid_sum(self): pos = dict(lineno=2, col_offset=3) m = ast.Module([ast.Expr(ast.expr(**pos), **pos)], []) with self.assertRaises(TypeError) as cm: compile(m, "", "exec") - self.assertIn("but got ", "eval") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_empty_yield_from(self): # Issue 16546: yield from value is not optional. empty_yield_from = ast.parse("def f():\n yield from g()") @@ -847,15 +629,13 @@ def test_issue31592(self): # There shouldn't be an assertion failure in case of a bad # unicodedata.normalize(). import unicodedata - def bad_normalize(*args): return None - - with support.swap_attr(unicodedata, "normalize", bad_normalize): - self.assertRaises(TypeError, ast.parse, "\u03d5") + with support.swap_attr(unicodedata, 'normalize', bad_normalize): + self.assertRaises(TypeError, ast.parse, '\u03D5') def test_issue18374_binop_col_offset(self): - tree = ast.parse("4+5+6+7") + tree = ast.parse('4+5+6+7') parent_binop = tree.body[0].value child_binop = parent_binop.left grandchild_binop = child_binop.left @@ -866,7 +646,7 @@ def test_issue18374_binop_col_offset(self): self.assertEqual(grandchild_binop.col_offset, 0) self.assertEqual(grandchild_binop.end_col_offset, 3) - tree = ast.parse("4+5-\\\n 6-7") + tree = ast.parse('4+5-\\\n 6-7') parent_binop = tree.body[0].value child_binop = parent_binop.left grandchild_binop = child_binop.left @@ -886,62 +666,274 @@ def test_issue18374_binop_col_offset(self): self.assertEqual(grandchild_binop.end_lineno, 1) def test_issue39579_dotted_name_end_col_offset(self): - tree = ast.parse("@a.b.c\ndef f(): pass") + tree = ast.parse('@a.b.c\ndef f(): pass') attr_b = tree.body[0].decorator_list[0].value self.assertEqual(attr_b.end_col_offset, 4) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_ast_asdl_signature(self): - self.assertEqual( - ast.withitem.__doc__, "withitem(expr context_expr, expr? optional_vars)" - ) + self.assertEqual(ast.withitem.__doc__, "withitem(expr context_expr, expr? optional_vars)") self.assertEqual(ast.GtE.__doc__, "GtE") self.assertEqual(ast.Name.__doc__, "Name(identifier id, expr_context ctx)") - self.assertEqual( - ast.cmpop.__doc__, - "cmpop = Eq | NotEq | Lt | LtE | Gt | GtE | Is | IsNot | In | NotIn", - ) + self.assertEqual(ast.cmpop.__doc__, "cmpop = Eq | NotEq | Lt | LtE | Gt | GtE | Is | IsNot | In | NotIn") expressions = [f" | {node.__doc__}" for node in ast.expr.__subclasses__()] expressions[0] = f"expr = {ast.expr.__subclasses__()[0].__doc__}" self.assertCountEqual(ast.expr.__doc__.split("\n"), expressions) - @unittest.expectedFailure # TODO: RUSTPYTHON; SyntaxError not raised + def test_compare_basics(self): + self.assertTrue(ast.compare(ast.parse("x = 10"), ast.parse("x = 10"))) + self.assertFalse(ast.compare(ast.parse("x = 10"), ast.parse(""))) + self.assertFalse(ast.compare(ast.parse("x = 10"), ast.parse("x"))) + self.assertFalse( + ast.compare(ast.parse("x = 10;y = 20"), ast.parse("class C:pass")) + ) + + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: Cannot add list and tuple + def test_compare_modified_ast(self): + # The ast API is a bit underspecified. The objects are mutable, + # and even _fields and _attributes are mutable. The compare() does + # some simple things to accommodate mutability. + a = ast.parse("m * x + b", mode="eval") + b = ast.parse("m * x + b", mode="eval") + self.assertTrue(ast.compare(a, b)) + + a._fields = a._fields + ("spam",) + a.spam = "Spam" + self.assertNotEqual(a._fields, b._fields) + self.assertFalse(ast.compare(a, b)) + self.assertFalse(ast.compare(b, a)) + + b._fields = a._fields + b.spam = a.spam + self.assertTrue(ast.compare(a, b)) + self.assertTrue(ast.compare(b, a)) + + b._attributes = b._attributes + ("eggs",) + b.eggs = "eggs" + self.assertNotEqual(a._attributes, b._attributes) + self.assertFalse(ast.compare(a, b, compare_attributes=True)) + self.assertFalse(ast.compare(b, a, compare_attributes=True)) + + a._attributes = b._attributes + a.eggs = b.eggs + self.assertTrue(ast.compare(a, b, compare_attributes=True)) + self.assertTrue(ast.compare(b, a, compare_attributes=True)) + + def test_compare_literals(self): + constants = ( + -20, + 20, + 20.0, + 1, + 1.0, + True, + 0, + False, + frozenset(), + tuple(), + "ABCD", + "abcd", + "中文字", + 1e1000, + -1e1000, + ) + for next_index, constant in enumerate(constants[:-1], 1): + next_constant = constants[next_index] + with self.subTest(literal=constant, next_literal=next_constant): + self.assertTrue( + ast.compare(ast.Constant(constant), ast.Constant(constant)) + ) + self.assertFalse( + ast.compare( + ast.Constant(constant), ast.Constant(next_constant) + ) + ) + + same_looking_literal_cases = [ + {1, 1.0, True, 1 + 0j}, + {0, 0.0, False, 0 + 0j}, + ] + for same_looking_literals in same_looking_literal_cases: + for literal in same_looking_literals: + for same_looking_literal in same_looking_literals - {literal}: + self.assertFalse( + ast.compare( + ast.Constant(literal), + ast.Constant(same_looking_literal), + ) + ) + + def test_compare_fieldless(self): + self.assertTrue(ast.compare(ast.Add(), ast.Add())) + self.assertFalse(ast.compare(ast.Sub(), ast.Add())) + + # test that missing runtime fields is handled in ast.compare() + a1, a2 = ast.Name('a'), ast.Name('a') + self.assertTrue(ast.compare(a1, a2)) + self.assertTrue(ast.compare(a1, a2)) + del a1.id + self.assertFalse(ast.compare(a1, a2)) + del a2.id + self.assertTrue(ast.compare(a1, a2)) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: type object '_ast.Module' has no attribute '_field_types' + def test_compare_modes(self): + for mode, sources in ( + ("exec", exec_tests), + ("eval", eval_tests), + ("single", single_tests), + ): + for source in sources: + a = ast.parse(source, mode=mode) + b = ast.parse(source, mode=mode) + self.assertTrue( + ast.compare(a, b), f"{ast.dump(a)} != {ast.dump(b)}" + ) + + def test_compare_attributes_option(self): + def parse(a, b): + return ast.parse(a), ast.parse(b) + + a, b = parse("2 + 2", "2+2") + self.assertTrue(ast.compare(a, b)) + self.assertTrue(ast.compare(a, b, compare_attributes=False)) + self.assertFalse(ast.compare(a, b, compare_attributes=True)) + + def test_compare_attributes_option_missing_attribute(self): + # test that missing runtime attributes is handled in ast.compare() + a1, a2 = ast.Name('a', lineno=1), ast.Name('a', lineno=1) + self.assertTrue(ast.compare(a1, a2)) + self.assertTrue(ast.compare(a1, a2, compare_attributes=True)) + del a1.lineno + self.assertFalse(ast.compare(a1, a2, compare_attributes=True)) + del a2.lineno + self.assertTrue(ast.compare(a1, a2, compare_attributes=True)) + + @unittest.expectedFailure # TODO: RUSTPYTHON; SyntaxError not raised def test_positional_only_feature_version(self): - ast.parse("def foo(x, /): ...", feature_version=(3, 8)) - ast.parse("def bar(x=1, /): ...", feature_version=(3, 8)) + ast.parse('def foo(x, /): ...', feature_version=(3, 8)) + ast.parse('def bar(x=1, /): ...', feature_version=(3, 8)) with self.assertRaises(SyntaxError): - ast.parse("def foo(x, /): ...", feature_version=(3, 7)) + ast.parse('def foo(x, /): ...', feature_version=(3, 7)) with self.assertRaises(SyntaxError): - ast.parse("def bar(x=1, /): ...", feature_version=(3, 7)) + ast.parse('def bar(x=1, /): ...', feature_version=(3, 7)) - ast.parse("lambda x, /: ...", feature_version=(3, 8)) - ast.parse("lambda x=1, /: ...", feature_version=(3, 8)) + ast.parse('lambda x, /: ...', feature_version=(3, 8)) + ast.parse('lambda x=1, /: ...', feature_version=(3, 8)) with self.assertRaises(SyntaxError): - ast.parse("lambda x, /: ...", feature_version=(3, 7)) + ast.parse('lambda x, /: ...', feature_version=(3, 7)) with self.assertRaises(SyntaxError): - ast.parse("lambda x=1, /: ...", feature_version=(3, 7)) + ast.parse('lambda x=1, /: ...', feature_version=(3, 7)) - @unittest.expectedFailure # TODO: RUSTPYTHON; SyntaxError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; SyntaxError not raised def test_assignment_expression_feature_version(self): - ast.parse("(x := 0)", feature_version=(3, 8)) + ast.parse('(x := 0)', feature_version=(3, 8)) + with self.assertRaises(SyntaxError): + ast.parse('(x := 0)', feature_version=(3, 7)) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: SyntaxError not raised + def test_pep750_tstring(self): + code = 't""' + ast.parse(code, feature_version=(3, 14)) + with self.assertRaises(SyntaxError): + ast.parse(code, feature_version=(3, 13)) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: SyntaxError not raised + def test_pep758_except_without_parens(self): + code = textwrap.dedent(""" + try: + ... + except ValueError, TypeError: + ... + """) + ast.parse(code, feature_version=(3, 14)) + with self.assertRaises(SyntaxError): + ast.parse(code, feature_version=(3, 13)) + + def test_pep758_except_with_single_expr(self): + single_expr = textwrap.dedent(""" + try: + ... + except{0} TypeError: + ... + """) + + single_expr_with_as = textwrap.dedent(""" + try: + ... + except{0} TypeError as exc: + ... + """) + + single_tuple_expr = textwrap.dedent(""" + try: + ... + except{0} (TypeError,): + ... + """) + + single_tuple_expr_with_as = textwrap.dedent(""" + try: + ... + except{0} (TypeError,) as exc: + ... + """) + + single_parens_expr = textwrap.dedent(""" + try: + ... + except{0} (TypeError): + ... + """) + + single_parens_expr_with_as = textwrap.dedent(""" + try: + ... + except{0} (TypeError) as exc: + ... + """) + + for code in [ + single_expr, + single_expr_with_as, + single_tuple_expr, + single_tuple_expr_with_as, + single_parens_expr, + single_parens_expr_with_as, + ]: + for star in [True, False]: + code = code.format('*' if star else '') + with self.subTest(code=code, star=star): + ast.parse(code, feature_version=(3, 14)) + ast.parse(code, feature_version=(3, 13)) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: SyntaxError not raised + def test_pep758_except_star_without_parens(self): + code = textwrap.dedent(""" + try: + ... + except* ValueError, TypeError: + ... + """) + ast.parse(code, feature_version=(3, 14)) with self.assertRaises(SyntaxError): - ast.parse("(x := 0)", feature_version=(3, 7)) + ast.parse(code, feature_version=(3, 13)) def test_conditional_context_managers_parse_with_low_feature_version(self): # regression test for gh-115881 - ast.parse("with (x() if y else z()): ...", feature_version=(3, 8)) + ast.parse('with (x() if y else z()): ...', feature_version=(3, 8)) - @unittest.expectedFailure # TODO: RUSTPYTHON; SyntaxError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; SyntaxError not raised def test_exception_groups_feature_version(self): - code = dedent(""" + code = dedent(''' try: ... except* Exception: ... - """) + ''') ast.parse(code) with self.assertRaises(SyntaxError): ast.parse(code, feature_version=(3, 10)) - @unittest.expectedFailure # TODO: RUSTPYTHON; SyntaxError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; SyntaxError not raised def test_type_params_feature_version(self): samples = [ "type X = int", @@ -954,7 +946,7 @@ def test_type_params_feature_version(self): with self.assertRaises(SyntaxError): ast.parse(sample, feature_version=(3, 11)) - @unittest.expectedFailure # TODO: RUSTPYTHON; SyntaxError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; SyntaxError not raised def test_type_params_default_feature_version(self): samples = [ "type X[*Ts=int] = int", @@ -969,21 +961,19 @@ def test_type_params_default_feature_version(self): def test_invalid_major_feature_version(self): with self.assertRaises(ValueError): - ast.parse("pass", feature_version=(2, 7)) + ast.parse('pass', feature_version=(2, 7)) with self.assertRaises(ValueError): - ast.parse("pass", feature_version=(4, 0)) + ast.parse('pass', feature_version=(4, 0)) - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_constant_as_name(self): for constant in "True", "False", "None": expr = ast.Expression(ast.Name(constant, ast.Load())) ast.fix_missing_locations(expr) - with self.assertRaisesRegex( - ValueError, f"identifier field can't represent '{constant}' constant" - ): + with self.assertRaisesRegex(ValueError, f"identifier field can't represent '{constant}' constant"): compile(expr, "", "eval") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_constant_as_unicode_name(self): constants = [ ("True", b"Tru\xe1\xb5\x89"), @@ -998,41 +988,39 @@ def test_constant_as_unicode_name(self): def test_precedence_enum(self): class _Precedence(enum.IntEnum): """Precedence table that originated from python grammar.""" - - NAMED_EXPR = enum.auto() # := - TUPLE = enum.auto() # , - YIELD = enum.auto() # 'yield', 'yield from' - TEST = enum.auto() # 'if'-'else', 'lambda' - OR = enum.auto() # 'or' - AND = enum.auto() # 'and' - NOT = enum.auto() # 'not' - CMP = enum.auto() # '<', '>', '==', '>=', '<=', '!=', - # 'in', 'not in', 'is', 'is not' + NAMED_EXPR = enum.auto() # := + TUPLE = enum.auto() # , + YIELD = enum.auto() # 'yield', 'yield from' + TEST = enum.auto() # 'if'-'else', 'lambda' + OR = enum.auto() # 'or' + AND = enum.auto() # 'and' + NOT = enum.auto() # 'not' + CMP = enum.auto() # '<', '>', '==', '>=', '<=', '!=', + # 'in', 'not in', 'is', 'is not' EXPR = enum.auto() - BOR = EXPR # '|' - BXOR = enum.auto() # '^' - BAND = enum.auto() # '&' - SHIFT = enum.auto() # '<<', '>>' - ARITH = enum.auto() # '+', '-' - TERM = enum.auto() # '*', '@', '/', '%', '//' - FACTOR = enum.auto() # unary '+', '-', '~' - POWER = enum.auto() # '**' - AWAIT = enum.auto() # 'await' + BOR = EXPR # '|' + BXOR = enum.auto() # '^' + BAND = enum.auto() # '&' + SHIFT = enum.auto() # '<<', '>>' + ARITH = enum.auto() # '+', '-' + TERM = enum.auto() # '*', '@', '/', '%', '//' + FACTOR = enum.auto() # unary '+', '-', '~' + POWER = enum.auto() # '**' + AWAIT = enum.auto() # 'await' ATOM = enum.auto() - def next(self): try: return self.__class__(self + 1) except ValueError: return self - - enum._test_simple_enum(_Precedence, ast._Precedence) + enum._test_simple_enum(_Precedence, _ast_unparse._Precedence) @support.cpython_only + @skip_wasi_stack_overflow() + @skip_emscripten_stack_overflow() def test_ast_recursion_limit(self): - fail_depth = support.exceeds_recursion_limit() - crash_depth = 100_000 - success_depth = int(support.get_c_recursion_limit() * 0.8) + crash_depth = 500_000 + success_depth = 200 if _testinternalcapi is not None: remaining = _testinternalcapi.get_c_recursion_remaining() success_depth = min(success_depth, remaining) @@ -1040,12 +1028,13 @@ def test_ast_recursion_limit(self): def check_limit(prefix, repeated): expect_ok = prefix + repeated * success_depth ast.parse(expect_ok) - for depth in (fail_depth, crash_depth): - broken = prefix + repeated * depth - details = "Compiling ({!r} + {!r} * {})".format(prefix, repeated, depth) - with self.assertRaises(RecursionError, msg=details): - with support.infinite_recursion(): - ast.parse(broken) + + broken = prefix + repeated * crash_depth + details = "Compiling ({!r} + {!r} * {})".format( + prefix, repeated, crash_depth) + with self.assertRaises(RecursionError, msg=details): + with support.infinite_recursion(): + ast.parse(broken) check_limit("a", "()") check_limit("a", ".b") @@ -1053,9 +1042,8 @@ def check_limit(prefix, repeated): check_limit("a", "*a") def test_null_bytes(self): - with self.assertRaises( - SyntaxError, msg="source code string cannot contain null bytes" - ): + with self.assertRaises(SyntaxError, + msg="source code string cannot contain null bytes"): ast.parse("a\0b") def assert_none_check(self, node: type[ast.AST], attr: str, source: str) -> None: @@ -1071,7 +1059,7 @@ def assert_none_check(self, node: type[ast.AST], attr: str, source: str) -> None with self.assertRaisesRegex(ValueError, f"^{e}$"): compile(tree, "", "exec") - @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: expected some sort of expr, but got None + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: expected some sort of expr, but got None def test_none_checks(self) -> None: tests = [ (ast.alias, "name", "import spam as SPAM"), @@ -1085,11 +1073,69 @@ def test_none_checks(self) -> None: for node, attr, source in tests: self.assert_none_check(node, attr, source) + @unittest.expectedFailure # TODO: RUSTPYTHON; FileNotFoundError: [Errno 2] No such file or directory: '/Users/youknowone/Projects/RustPython/crates/pylib/Lib/test/test_ast/data/ast_repr.txt' + def test_repr(self) -> None: + snapshots = AST_REPR_DATA_FILE.read_text().split("\n") + for test, snapshot in zip(ast_repr_get_test_cases(), snapshots, strict=True): + with self.subTest(test_input=test): + self.assertEqual(repr(ast.parse(test)), snapshot) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: ValueError not raised + def test_repr_large_input_crash(self): + # gh-125010: Fix use-after-free in ast repr() + source = "0x0" + "e" * 10_000 + with self.assertRaisesRegex(ValueError, + r"Exceeds the limit \(\d+ digits\)"): + repr(ast.Constant(value=eval(source))) + + def test_tstring(self): + # Test AST structure for simple t-string + tree = ast.parse('t"Hello"') + self.assertIsInstance(tree.body[0].value, ast.TemplateStr) + self.assertIsInstance(tree.body[0].value.values[0], ast.Constant) + + # Test AST for t-string with interpolation + tree = ast.parse('t"Hello {name}"') + self.assertIsInstance(tree.body[0].value, ast.TemplateStr) + self.assertIsInstance(tree.body[0].value.values[0], ast.Constant) + self.assertIsInstance(tree.body[0].value.values[1], ast.Interpolation) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_classattrs_deprecated(self): + return super().test_classattrs_deprecated() + + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags + def test_optimization_levels_const_folding(self): + return super().test_optimization_levels_const_folding() + + @unittest.expectedFailure # TODO: RUSTPYTHON; will be removed in Python 3.14 + def test_constant_subclasses_deprecated(self): + return super().test_constant_subclasses_deprecated() + class CopyTests(unittest.TestCase): """Test copying and pickling AST nodes.""" - @unittest.expectedFailure # TODO: RUSTPYTHON + @staticmethod + def iter_ast_classes(): + """Iterate over the (native) subclasses of ast.AST recursively. + + This excludes the special class ast.Index since its constructor + returns an integer. + """ + def do(cls): + if cls.__module__ != 'ast': + return + if cls is ast.Index: + return + + yield cls + for sub in cls.__subclasses__(): + yield from do(sub) + + yield from do(ast.AST) + + @unittest.expectedFailure # TODO: RUSTPYTHON def test_pickling(self): import pickle @@ -1154,66 +1200,317 @@ def test_copy_with_parents(self): for node in ast.walk(tree2): for child in ast.iter_child_nodes(node): - if hasattr(child, "parent") and not isinstance( - child, - ( - ast.expr_context, - ast.boolop, - ast.unaryop, - ast.cmpop, - ast.operator, - ), - ): + if hasattr(child, "parent") and not isinstance(child, ( + ast.expr_context, ast.boolop, ast.unaryop, ast.cmpop, ast.operator, + )): self.assertEqual(to_tuple(child.parent), to_tuple(node)) + def test_replace_interface(self): + for klass in self.iter_ast_classes(): + with self.subTest(klass=klass): + self.assertHasAttr(klass, '__replace__') + + fields = set(klass._fields) + with self.subTest(klass=klass, fields=fields): + node = klass(**dict.fromkeys(fields)) + # forbid positional arguments in replace() + self.assertRaises(TypeError, copy.replace, node, 1) + self.assertRaises(TypeError, node.__replace__, 1) + + def test_replace_native(self): + for klass in self.iter_ast_classes(): + fields = set(klass._fields) + attributes = set(klass._attributes) + + with self.subTest(klass=klass, fields=fields, attributes=attributes): + # use of object() to ensure that '==' and 'is' + # behave similarly in ast.compare(node, repl) + old_fields = {field: object() for field in fields} + old_attrs = {attr: object() for attr in attributes} + + # check shallow copy + node = klass(**old_fields) + repl = copy.replace(node) + self.assertTrue(ast.compare(node, repl, compare_attributes=True)) + # check when passing using attributes (they may be optional!) + node = klass(**old_fields, **old_attrs) + repl = copy.replace(node) + self.assertTrue(ast.compare(node, repl, compare_attributes=True)) + + for field in fields: + # check when we sometimes have attributes and sometimes not + for init_attrs in [{}, old_attrs]: + node = klass(**old_fields, **init_attrs) + # only change a single field (do not change attributes) + new_value = object() + repl = copy.replace(node, **{field: new_value}) + for f in fields: + old_value = old_fields[f] + # assert that there is no side-effect + self.assertIs(getattr(node, f), old_value) + # check the changes + if f != field: + self.assertIs(getattr(repl, f), old_value) + else: + self.assertIs(getattr(repl, f), new_value) + self.assertFalse(ast.compare(node, repl, compare_attributes=True)) + + for attribute in attributes: + node = klass(**old_fields, **old_attrs) + # only change a single attribute (do not change fields) + new_attr = object() + repl = copy.replace(node, **{attribute: new_attr}) + for a in attributes: + old_attr = old_attrs[a] + # assert that there is no side-effect + self.assertIs(getattr(node, a), old_attr) + # check the changes + if a != attribute: + self.assertIs(getattr(repl, a), old_attr) + else: + self.assertIs(getattr(repl, a), new_attr) + self.assertFalse(ast.compare(node, repl, compare_attributes=True)) + + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: replace() does not support Name objects + def test_replace_accept_known_class_fields(self): + nid, ctx = object(), object() + + node = ast.Name(id=nid, ctx=ctx) + self.assertIs(node.id, nid) + self.assertIs(node.ctx, ctx) + + new_nid = object() + repl = copy.replace(node, id=new_nid) + # assert that there is no side-effect + self.assertIs(node.id, nid) + self.assertIs(node.ctx, ctx) + # check the changes + self.assertIs(repl.id, new_nid) + self.assertIs(repl.ctx, node.ctx) # no changes + + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: replace() does not support Name objects + def test_replace_accept_known_class_attributes(self): + node = ast.parse('x').body[0].value + self.assertEqual(node.id, 'x') + self.assertEqual(node.lineno, 1) + + # constructor allows any type so replace() should do the same + lineno = object() + repl = copy.replace(node, lineno=lineno) + # assert that there is no side-effect + self.assertEqual(node.lineno, 1) + # check the changes + self.assertEqual(repl.id, node.id) + self.assertEqual(repl.ctx, node.ctx) + self.assertEqual(repl.lineno, lineno) + + _, _, state = node.__reduce__() + self.assertEqual(state['id'], 'x') + self.assertEqual(state['ctx'], node.ctx) + self.assertEqual(state['lineno'], 1) + + _, _, state = repl.__reduce__() + self.assertEqual(state['id'], 'x') + self.assertEqual(state['ctx'], node.ctx) + self.assertEqual(state['lineno'], lineno) + + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: replace() does not support MyNode objects + def test_replace_accept_known_custom_class_fields(self): + class MyNode(ast.AST): + _fields = ('name', 'data') + __annotations__ = {'name': str, 'data': object} + __match_args__ = ('name', 'data') + + name, data = 'name', object() + + node = MyNode(name, data) + self.assertIs(node.name, name) + self.assertIs(node.data, data) + # check shallow copy + repl = copy.replace(node) + # assert that there is no side-effect + self.assertIs(node.name, name) + self.assertIs(node.data, data) + # check the shallow copy + self.assertIs(repl.name, name) + self.assertIs(repl.data, data) + + node = MyNode(name, data) + repl_data = object() + # replace custom but known field + repl = copy.replace(node, data=repl_data) + # assert that there is no side-effect + self.assertIs(node.name, name) + self.assertIs(node.data, data) + # check the changes + self.assertIs(repl.name, node.name) + self.assertIs(repl.data, repl_data) + + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: replace() does not support MyNode objects + def test_replace_accept_known_custom_class_attributes(self): + class MyNode(ast.AST): + x = 0 + y = 1 + _attributes = ('x', 'y') + + node = MyNode() + self.assertEqual(node.x, 0) + self.assertEqual(node.y, 1) + + y = object() + repl = copy.replace(node, y=y) + # assert that there is no side-effect + self.assertEqual(node.x, 0) + self.assertEqual(node.y, 1) + # check the changes + self.assertEqual(repl.x, 0) + self.assertEqual(repl.y, y) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 'x' is not 'x' + def test_replace_ignore_known_custom_instance_fields(self): + node = ast.parse('x').body[0].value + node.extra = extra = object() # add instance 'extra' field + context = node.ctx + + # assert initial values + self.assertIs(node.id, 'x') + self.assertIs(node.ctx, context) + self.assertIs(node.extra, extra) + # shallow copy, but drops extra fields + repl = copy.replace(node) + # assert that there is no side-effect + self.assertIs(node.id, 'x') + self.assertIs(node.ctx, context) + self.assertIs(node.extra, extra) + # verify that the 'extra' field is not kept + self.assertIs(repl.id, 'x') + self.assertIs(repl.ctx, context) + self.assertRaises(AttributeError, getattr, repl, 'extra') + + # change known native field + repl = copy.replace(node, id='y') + # assert that there is no side-effect + self.assertIs(node.id, 'x') + self.assertIs(node.ctx, context) + self.assertIs(node.extra, extra) + # verify that the 'extra' field is not kept + self.assertIs(repl.id, 'y') + self.assertIs(repl.ctx, context) + self.assertRaises(AttributeError, getattr, repl, 'extra') + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: "Name\.__replace__\ missing\ 1\ keyword\ argument:\ 'id'\." does not match "replace() does not support Name objects" + def test_replace_reject_missing_field(self): + # case: warn if deleted field is not replaced + node = ast.parse('x').body[0].value + context = node.ctx + del node.id + + self.assertRaises(AttributeError, getattr, node, 'id') + self.assertIs(node.ctx, context) + msg = "Name.__replace__ missing 1 keyword argument: 'id'." + with self.assertRaisesRegex(TypeError, re.escape(msg)): + copy.replace(node) + # assert that there is no side-effect + self.assertRaises(AttributeError, getattr, node, 'id') + self.assertIs(node.ctx, context) + + # case: do not raise if deleted field is replaced + node = ast.parse('x').body[0].value + context = node.ctx + del node.id + + self.assertRaises(AttributeError, getattr, node, 'id') + self.assertIs(node.ctx, context) + repl = copy.replace(node, id='y') + # assert that there is no side-effect + self.assertRaises(AttributeError, getattr, node, 'id') + self.assertIs(node.ctx, context) + self.assertIs(repl.id, 'y') + self.assertIs(repl.ctx, context) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'FunctionDef' object has no attribute 'returns' + def test_replace_accept_missing_field_with_default(self): + node = ast.FunctionDef(name="foo", args=ast.arguments()) + self.assertIs(node.returns, None) + self.assertEqual(node.decorator_list, []) + node2 = copy.replace(node, name="bar") + self.assertEqual(node2.name, "bar") + self.assertIs(node2.returns, None) + self.assertEqual(node2.decorator_list, []) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: "Name\.__replace__\ got\ an\ unexpected\ keyword\ argument\ 'extra'\." does not match "replace() does not support Name objects" + def test_replace_reject_known_custom_instance_fields_commits(self): + node = ast.parse('x').body[0].value + node.extra = extra = object() # add instance 'extra' field + context = node.ctx + + # explicit rejection of known instance fields + self.assertHasAttr(node, 'extra') + msg = "Name.__replace__ got an unexpected keyword argument 'extra'." + with self.assertRaisesRegex(TypeError, re.escape(msg)): + copy.replace(node, extra=1) + # assert that there is no side-effect + self.assertIs(node.id, 'x') + self.assertIs(node.ctx, context) + self.assertIs(node.extra, extra) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: "Name\.__replace__\ got\ an\ unexpected\ keyword\ argument\ 'unknown'\." does not match "replace() does not support Name objects" + def test_replace_reject_unknown_instance_fields(self): + node = ast.parse('x').body[0].value + context = node.ctx + + # explicit rejection of unknown extra fields + self.assertRaises(AttributeError, getattr, node, 'unknown') + msg = "Name.__replace__ got an unexpected keyword argument 'unknown'." + with self.assertRaisesRegex(TypeError, re.escape(msg)): + copy.replace(node, unknown=1) + # assert that there is no side-effect + self.assertIs(node.id, 'x') + self.assertIs(node.ctx, context) + self.assertRaises(AttributeError, getattr, node, 'unknown') class ASTHelpers_Test(unittest.TestCase): maxDiff = None - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_parse(self): - a = ast.parse("foo(1 + 1)") - b = compile("foo(1 + 1)", "", "exec", ast.PyCF_ONLY_AST) + a = ast.parse('foo(1 + 1)') + b = compile('foo(1 + 1)', '', 'exec', ast.PyCF_ONLY_AST) self.assertEqual(ast.dump(a), ast.dump(b)) def test_parse_in_error(self): try: - 1 / 0 + 1/0 except Exception: with self.assertRaises(SyntaxError) as e: ast.literal_eval(r"'\U'") self.assertIsNotNone(e.exception.__context__) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_dump(self): node = ast.parse('spam(eggs, "and cheese")') - self.assertEqual( - ast.dump(node), + self.assertEqual(ast.dump(node), "Module(body=[Expr(value=Call(func=Name(id='spam', ctx=Load()), " - "args=[Name(id='eggs', ctx=Load()), Constant(value='and cheese')]))])", + "args=[Name(id='eggs', ctx=Load()), Constant(value='and cheese')]))])" ) - self.assertEqual( - ast.dump(node, annotate_fields=False), + self.assertEqual(ast.dump(node, annotate_fields=False), "Module([Expr(Call(Name('spam', Load()), [Name('eggs', Load()), " - "Constant('and cheese')]))])", + "Constant('and cheese')]))])" ) - self.assertEqual( - ast.dump(node, include_attributes=True), + self.assertEqual(ast.dump(node, include_attributes=True), "Module(body=[Expr(value=Call(func=Name(id='spam', ctx=Load(), " "lineno=1, col_offset=0, end_lineno=1, end_col_offset=4), " "args=[Name(id='eggs', ctx=Load(), lineno=1, col_offset=5, " "end_lineno=1, end_col_offset=9), Constant(value='and cheese', " "lineno=1, col_offset=11, end_lineno=1, end_col_offset=23)], " "lineno=1, col_offset=0, end_lineno=1, end_col_offset=24), " - "lineno=1, col_offset=0, end_lineno=1, end_col_offset=24)])", + "lineno=1, col_offset=0, end_lineno=1, end_col_offset=24)])" ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_dump_indent(self): node = ast.parse('spam(eggs, "and cheese")') - self.assertEqual( - ast.dump(node, indent=3), - """\ + self.assertEqual(ast.dump(node, indent=3), """\ Module( body=[ Expr( @@ -1221,11 +1518,8 @@ def test_dump_indent(self): func=Name(id='spam', ctx=Load()), args=[ Name(id='eggs', ctx=Load()), - Constant(value='and cheese')]))])""", - ) - self.assertEqual( - ast.dump(node, annotate_fields=False, indent="\t"), - """\ + Constant(value='and cheese')]))])""") + self.assertEqual(ast.dump(node, annotate_fields=False, indent='\t'), """\ Module( \t[ \t\tExpr( @@ -1233,11 +1527,8 @@ def test_dump_indent(self): \t\t\t\tName('spam', Load()), \t\t\t\t[ \t\t\t\t\tName('eggs', Load()), -\t\t\t\t\tConstant('and cheese')]))])""", - ) - self.assertEqual( - ast.dump(node, include_attributes=True, indent=3), - """\ +\t\t\t\t\tConstant('and cheese')]))])""") + self.assertEqual(ast.dump(node, include_attributes=True, indent=3), """\ Module( body=[ Expr( @@ -1270,78 +1561,74 @@ def test_dump_indent(self): lineno=1, col_offset=0, end_lineno=1, - end_col_offset=24)])""", - ) + end_col_offset=24)])""") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_dump_incomplete(self): node = ast.Raise(lineno=3, col_offset=4) - self.assertEqual(ast.dump(node), "Raise()") - self.assertEqual( - ast.dump(node, include_attributes=True), "Raise(lineno=3, col_offset=4)" + self.assertEqual(ast.dump(node), + "Raise()" + ) + self.assertEqual(ast.dump(node, include_attributes=True), + "Raise(lineno=3, col_offset=4)" ) - node = ast.Raise(exc=ast.Name(id="e", ctx=ast.Load()), lineno=3, col_offset=4) - self.assertEqual(ast.dump(node), "Raise(exc=Name(id='e', ctx=Load()))") - self.assertEqual( - ast.dump(node, annotate_fields=False), "Raise(Name('e', Load()))" + node = ast.Raise(exc=ast.Name(id='e', ctx=ast.Load()), lineno=3, col_offset=4) + self.assertEqual(ast.dump(node), + "Raise(exc=Name(id='e', ctx=Load()))" ) - self.assertEqual( - ast.dump(node, include_attributes=True), - "Raise(exc=Name(id='e', ctx=Load()), lineno=3, col_offset=4)", + self.assertEqual(ast.dump(node, annotate_fields=False), + "Raise(Name('e', Load()))" ) - self.assertEqual( - ast.dump(node, annotate_fields=False, include_attributes=True), - "Raise(Name('e', Load()), lineno=3, col_offset=4)", + self.assertEqual(ast.dump(node, include_attributes=True), + "Raise(exc=Name(id='e', ctx=Load()), lineno=3, col_offset=4)" ) - node = ast.Raise(cause=ast.Name(id="e", ctx=ast.Load())) - self.assertEqual(ast.dump(node), "Raise(cause=Name(id='e', ctx=Load()))") - self.assertEqual( - ast.dump(node, annotate_fields=False), "Raise(cause=Name('e', Load()))" + self.assertEqual(ast.dump(node, annotate_fields=False, include_attributes=True), + "Raise(Name('e', Load()), lineno=3, col_offset=4)" + ) + node = ast.Raise(cause=ast.Name(id='e', ctx=ast.Load())) + self.assertEqual(ast.dump(node), + "Raise(cause=Name(id='e', ctx=Load()))" + ) + self.assertEqual(ast.dump(node, annotate_fields=False), + "Raise(cause=Name('e', Load()))" ) # Arguments: node = ast.arguments(args=[ast.arg("x")]) - self.assertEqual( - ast.dump(node, annotate_fields=False), + self.assertEqual(ast.dump(node, annotate_fields=False), "arguments([], [arg('x')])", ) node = ast.arguments(posonlyargs=[ast.arg("x")]) - self.assertEqual( - ast.dump(node, annotate_fields=False), + self.assertEqual(ast.dump(node, annotate_fields=False), "arguments([arg('x')])", ) - node = ast.arguments(posonlyargs=[ast.arg("x")], kwonlyargs=[ast.arg("y")]) - self.assertEqual( - ast.dump(node, annotate_fields=False), + node = ast.arguments(posonlyargs=[ast.arg("x")], kwonlyargs=[ast.arg('y')]) + self.assertEqual(ast.dump(node, annotate_fields=False), "arguments([arg('x')], kwonlyargs=[arg('y')])", ) - node = ast.arguments(args=[ast.arg("x")], kwonlyargs=[ast.arg("y")]) - self.assertEqual( - ast.dump(node, annotate_fields=False), + node = ast.arguments(args=[ast.arg("x")], kwonlyargs=[ast.arg('y')]) + self.assertEqual(ast.dump(node, annotate_fields=False), "arguments([], [arg('x')], kwonlyargs=[arg('y')])", ) node = ast.arguments() - self.assertEqual( - ast.dump(node, annotate_fields=False), + self.assertEqual(ast.dump(node, annotate_fields=False), "arguments()", ) # Classes: node = ast.ClassDef( - "T", + 'T', [], - [ast.keyword("a", ast.Constant(None))], + [ast.keyword('a', ast.Constant(None))], [], - [ast.Name("dataclass", ctx=ast.Load())], + [ast.Name('dataclass', ctx=ast.Load())], ) - self.assertEqual( - ast.dump(node), + self.assertEqual(ast.dump(node), "ClassDef(name='T', keywords=[keyword(arg='a', value=Constant(value=None))], decorator_list=[Name(id='dataclass', ctx=Load())])", ) - self.assertEqual( - ast.dump(node, annotate_fields=False), + self.assertEqual(ast.dump(node, annotate_fields=False), "ClassDef('T', [], [keyword('a', Constant(None))], [], [Name('dataclass', Load())])", ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_dump_show_empty(self): def check_node(node, empty, full, **kwargs): with self.subTest(show_empty=False): @@ -1366,7 +1653,7 @@ def check_text(code, empty, full, **kwargs): check_node( # Corner case: there are no real `Name` instances with `id=''`: - ast.Name(id="", ctx=ast.Load()), + ast.Name(id='', ctx=ast.Load()), empty="Name(id='', ctx=Load())", full="Name(id='', ctx=Load())", ) @@ -1396,11 +1683,23 @@ def check_text(code, empty, full, **kwargs): ) check_node( - ast.Constant(value=""), + ast.Constant(value=''), empty="Constant(value='')", full="Constant(value='')", ) + check_node( + ast.Interpolation(value=ast.Constant(42), str=None, conversion=-1), + empty="Interpolation(value=Constant(value=42), str=None, conversion=-1)", + full="Interpolation(value=Constant(value=42), str=None, conversion=-1)", + ) + + check_node( + ast.Interpolation(value=ast.Constant(42), str=[], conversion=-1), + empty="Interpolation(value=Constant(value=42), str=[], conversion=-1)", + full="Interpolation(value=Constant(value=42), str=[], conversion=-1)", + ) + check_text( "def a(b: int = 0, *, c): ...", empty="Module(body=[FunctionDef(name='a', args=arguments(args=[arg(arg='b', annotation=Name(id='int', ctx=Load()))], kwonlyargs=[arg(arg='c')], kw_defaults=[None], defaults=[Constant(value=0)]), body=[Expr(value=Constant(value=Ellipsis))])])", @@ -1432,37 +1731,32 @@ def check_text(code, empty, full, **kwargs): full="Module(body=[Import(names=[alias(name='_ast', asname='ast')]), ImportFrom(module='module', names=[alias(name='sub')], level=0)], type_ignores=[])", ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_copy_location(self): - src = ast.parse("1 + 1", mode="eval") + src = ast.parse('1 + 1', mode='eval') src.body.right = ast.copy_location(ast.Constant(2), src.body.right) - self.assertEqual( - ast.dump(src, include_attributes=True), - "Expression(body=BinOp(left=Constant(value=1, lineno=1, col_offset=0, " - "end_lineno=1, end_col_offset=1), op=Add(), right=Constant(value=2, " - "lineno=1, col_offset=4, end_lineno=1, end_col_offset=5), lineno=1, " - "col_offset=0, end_lineno=1, end_col_offset=5))", - ) - func = ast.Name("spam", ast.Load()) - src = ast.Call( - col_offset=1, lineno=1, end_lineno=1, end_col_offset=1, func=func - ) + self.assertEqual(ast.dump(src, include_attributes=True), + 'Expression(body=BinOp(left=Constant(value=1, lineno=1, col_offset=0, ' + 'end_lineno=1, end_col_offset=1), op=Add(), right=Constant(value=2, ' + 'lineno=1, col_offset=4, end_lineno=1, end_col_offset=5), lineno=1, ' + 'col_offset=0, end_lineno=1, end_col_offset=5))' + ) + func = ast.Name('spam', ast.Load()) + src = ast.Call(col_offset=1, lineno=1, end_lineno=1, end_col_offset=1, func=func) new = ast.copy_location(src, ast.Call(col_offset=None, lineno=None, func=func)) self.assertIsNone(new.end_lineno) self.assertIsNone(new.end_col_offset) self.assertEqual(new.lineno, 1) self.assertEqual(new.col_offset, 1) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_fix_missing_locations(self): src = ast.parse('write("spam")') - src.body.append( - ast.Expr(ast.Call(ast.Name("spam", ast.Load()), [ast.Constant("eggs")], [])) - ) + src.body.append(ast.Expr(ast.Call(ast.Name('spam', ast.Load()), + [ast.Constant('eggs')], []))) self.assertEqual(src, ast.fix_missing_locations(src)) self.maxDiff = None - self.assertEqual( - ast.dump(src, include_attributes=True), + self.assertEqual(ast.dump(src, include_attributes=True), "Module(body=[Expr(value=Call(func=Name(id='write', ctx=Load(), " "lineno=1, col_offset=0, end_lineno=1, end_col_offset=5), " "args=[Constant(value='spam', lineno=1, col_offset=6, end_lineno=1, " @@ -1472,29 +1766,27 @@ def test_fix_missing_locations(self): "lineno=1, col_offset=0, end_lineno=1, end_col_offset=0), " "args=[Constant(value='eggs', lineno=1, col_offset=0, end_lineno=1, " "end_col_offset=0)], lineno=1, col_offset=0, end_lineno=1, " - "end_col_offset=0), lineno=1, col_offset=0, end_lineno=1, end_col_offset=0)])", + "end_col_offset=0), lineno=1, col_offset=0, end_lineno=1, end_col_offset=0)])" ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_increment_lineno(self): - src = ast.parse("1 + 1", mode="eval") + src = ast.parse('1 + 1', mode='eval') self.assertEqual(ast.increment_lineno(src, n=3), src) - self.assertEqual( - ast.dump(src, include_attributes=True), - "Expression(body=BinOp(left=Constant(value=1, lineno=4, col_offset=0, " - "end_lineno=4, end_col_offset=1), op=Add(), right=Constant(value=1, " - "lineno=4, col_offset=4, end_lineno=4, end_col_offset=5), lineno=4, " - "col_offset=0, end_lineno=4, end_col_offset=5))", + self.assertEqual(ast.dump(src, include_attributes=True), + 'Expression(body=BinOp(left=Constant(value=1, lineno=4, col_offset=0, ' + 'end_lineno=4, end_col_offset=1), op=Add(), right=Constant(value=1, ' + 'lineno=4, col_offset=4, end_lineno=4, end_col_offset=5), lineno=4, ' + 'col_offset=0, end_lineno=4, end_col_offset=5))' ) # issue10869: do not increment lineno of root twice - src = ast.parse("1 + 1", mode="eval") + src = ast.parse('1 + 1', mode='eval') self.assertEqual(ast.increment_lineno(src.body, n=3), src.body) - self.assertEqual( - ast.dump(src, include_attributes=True), - "Expression(body=BinOp(left=Constant(value=1, lineno=4, col_offset=0, " - "end_lineno=4, end_col_offset=1), op=Add(), right=Constant(value=1, " - "lineno=4, col_offset=4, end_lineno=4, end_col_offset=5), lineno=4, " - "col_offset=0, end_lineno=4, end_col_offset=5))", + self.assertEqual(ast.dump(src, include_attributes=True), + 'Expression(body=BinOp(left=Constant(value=1, lineno=4, col_offset=0, ' + 'end_lineno=4, end_col_offset=1), op=Add(), right=Constant(value=1, ' + 'lineno=4, col_offset=4, end_lineno=4, end_col_offset=5), lineno=4, ' + 'col_offset=0, end_lineno=4, end_col_offset=5))' ) src = ast.Call( func=ast.Name("test", ast.Load()), args=[], keywords=[], lineno=1 @@ -1502,85 +1794,84 @@ def test_increment_lineno(self): self.assertEqual(ast.increment_lineno(src).lineno, 2) self.assertIsNone(ast.increment_lineno(src).end_lineno) - @unittest.expectedFailure # TODO: RUSTPYTHON; IndexError: index out of range + @unittest.expectedFailure # TODO: RUSTPYTHON; IndexError: index out of range def test_increment_lineno_on_module(self): - src = ast.parse( - dedent("""\ + src = ast.parse(dedent("""\ a = 1 b = 2 # type: ignore c = 3 d = 4 # type: ignore@tag - """), - type_comments=True, - ) + """), type_comments=True) ast.increment_lineno(src, n=5) self.assertEqual(src.type_ignores[0].lineno, 7) self.assertEqual(src.type_ignores[1].lineno, 9) - self.assertEqual(src.type_ignores[1].tag, "@tag") + self.assertEqual(src.type_ignores[1].tag, '@tag') def test_iter_fields(self): - node = ast.parse("foo()", mode="eval") + node = ast.parse('foo()', mode='eval') d = dict(ast.iter_fields(node.body)) - self.assertEqual(d.pop("func").id, "foo") - self.assertEqual(d, {"keywords": [], "args": []}) + self.assertEqual(d.pop('func').id, 'foo') + self.assertEqual(d, {'keywords': [], 'args': []}) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_iter_child_nodes(self): - node = ast.parse("spam(23, 42, eggs='leek')", mode="eval") + node = ast.parse("spam(23, 42, eggs='leek')", mode='eval') self.assertEqual(len(list(ast.iter_child_nodes(node.body))), 4) iterator = ast.iter_child_nodes(node.body) - self.assertEqual(next(iterator).id, "spam") + self.assertEqual(next(iterator).id, 'spam') self.assertEqual(next(iterator).value, 23) self.assertEqual(next(iterator).value, 42) - self.assertEqual( - ast.dump(next(iterator)), - "keyword(arg='eggs', value=Constant(value='leek'))", + self.assertEqual(ast.dump(next(iterator)), + "keyword(arg='eggs', value=Constant(value='leek'))" ) def test_get_docstring(self): node = ast.parse('"""line one\n line two"""') - self.assertEqual(ast.get_docstring(node), "line one\nline two") + self.assertEqual(ast.get_docstring(node), + 'line one\nline two') node = ast.parse('class foo:\n """line one\n line two"""') - self.assertEqual(ast.get_docstring(node.body[0]), "line one\nline two") + self.assertEqual(ast.get_docstring(node.body[0]), + 'line one\nline two') node = ast.parse('def foo():\n """line one\n line two"""') - self.assertEqual(ast.get_docstring(node.body[0]), "line one\nline two") + self.assertEqual(ast.get_docstring(node.body[0]), + 'line one\nline two') node = ast.parse('async def foo():\n """spam\n ham"""') - self.assertEqual(ast.get_docstring(node.body[0]), "spam\nham") + self.assertEqual(ast.get_docstring(node.body[0]), 'spam\nham') node = ast.parse('async def foo():\n """spam\n ham"""') - self.assertEqual(ast.get_docstring(node.body[0], clean=False), "spam\n ham") + self.assertEqual(ast.get_docstring(node.body[0], clean=False), 'spam\n ham') - node = ast.parse("x") + node = ast.parse('x') self.assertRaises(TypeError, ast.get_docstring, node.body[0]) def test_get_docstring_none(self): - self.assertIsNone(ast.get_docstring(ast.parse(""))) + self.assertIsNone(ast.get_docstring(ast.parse(''))) node = ast.parse('x = "not docstring"') self.assertIsNone(ast.get_docstring(node)) - node = ast.parse("def foo():\n pass") + node = ast.parse('def foo():\n pass') self.assertIsNone(ast.get_docstring(node)) - node = ast.parse("class foo:\n pass") + node = ast.parse('class foo:\n pass') self.assertIsNone(ast.get_docstring(node.body[0])) node = ast.parse('class foo:\n x = "not docstring"') self.assertIsNone(ast.get_docstring(node.body[0])) - node = ast.parse("class foo:\n def bar(self): pass") + node = ast.parse('class foo:\n def bar(self): pass') self.assertIsNone(ast.get_docstring(node.body[0])) - node = ast.parse("def foo():\n pass") + node = ast.parse('def foo():\n pass') self.assertIsNone(ast.get_docstring(node.body[0])) node = ast.parse('def foo():\n x = "not docstring"') self.assertIsNone(ast.get_docstring(node.body[0])) - node = ast.parse("async def foo():\n pass") + node = ast.parse('async def foo():\n pass') self.assertIsNone(ast.get_docstring(node.body[0])) node = ast.parse('async def foo():\n x = "not docstring"') self.assertIsNone(ast.get_docstring(node.body[0])) - node = ast.parse("async def foo():\n 42") + node = ast.parse('async def foo():\n 42') self.assertIsNone(ast.get_docstring(node.body[0])) def test_multi_line_docstring_col_offset_and_lineno_issue16806(self): @@ -1603,83 +1894,79 @@ def test_multi_line_docstring_col_offset_and_lineno_issue16806(self): self.assertEqual(node.body[2].lineno, 13) def test_elif_stmt_start_position(self): - node = ast.parse("if a:\n pass\nelif b:\n pass\n") + node = ast.parse('if a:\n pass\nelif b:\n pass\n') elif_stmt = node.body[0].orelse[0] self.assertEqual(elif_stmt.lineno, 3) self.assertEqual(elif_stmt.col_offset, 0) def test_elif_stmt_start_position_with_else(self): - node = ast.parse("if a:\n pass\nelif b:\n pass\nelse:\n pass\n") + node = ast.parse('if a:\n pass\nelif b:\n pass\nelse:\n pass\n') elif_stmt = node.body[0].orelse[0] self.assertEqual(elif_stmt.lineno, 3) self.assertEqual(elif_stmt.col_offset, 0) def test_starred_expr_end_position_within_call(self): - node = ast.parse("f(*[0, 1])") + node = ast.parse('f(*[0, 1])') starred_expr = node.body[0].value.args[0] self.assertEqual(starred_expr.end_lineno, 1) self.assertEqual(starred_expr.end_col_offset, 9) def test_literal_eval(self): - self.assertEqual(ast.literal_eval("[1, 2, 3]"), [1, 2, 3]) + self.assertEqual(ast.literal_eval('[1, 2, 3]'), [1, 2, 3]) self.assertEqual(ast.literal_eval('{"foo": 42}'), {"foo": 42}) - self.assertEqual(ast.literal_eval("(True, False, None)"), (True, False, None)) - self.assertEqual(ast.literal_eval("{1, 2, 3}"), {1, 2, 3}) + self.assertEqual(ast.literal_eval('(True, False, None)'), (True, False, None)) + self.assertEqual(ast.literal_eval('{1, 2, 3}'), {1, 2, 3}) self.assertEqual(ast.literal_eval('b"hi"'), b"hi") - self.assertEqual(ast.literal_eval("set()"), set()) - self.assertRaises(ValueError, ast.literal_eval, "foo()") - self.assertEqual(ast.literal_eval("6"), 6) - self.assertEqual(ast.literal_eval("+6"), 6) - self.assertEqual(ast.literal_eval("-6"), -6) - self.assertEqual(ast.literal_eval("3.25"), 3.25) - self.assertEqual(ast.literal_eval("+3.25"), 3.25) - self.assertEqual(ast.literal_eval("-3.25"), -3.25) - self.assertEqual(repr(ast.literal_eval("-0.0")), "-0.0") - self.assertRaises(ValueError, ast.literal_eval, "++6") - self.assertRaises(ValueError, ast.literal_eval, "+True") - self.assertRaises(ValueError, ast.literal_eval, "2+3") - - @unittest.expectedFailure # TODO: RUSTPYTHON; SyntaxError not raised + self.assertEqual(ast.literal_eval('set()'), set()) + self.assertRaises(ValueError, ast.literal_eval, 'foo()') + self.assertEqual(ast.literal_eval('6'), 6) + self.assertEqual(ast.literal_eval('+6'), 6) + self.assertEqual(ast.literal_eval('-6'), -6) + self.assertEqual(ast.literal_eval('3.25'), 3.25) + self.assertEqual(ast.literal_eval('+3.25'), 3.25) + self.assertEqual(ast.literal_eval('-3.25'), -3.25) + self.assertEqual(repr(ast.literal_eval('-0.0')), '-0.0') + self.assertRaises(ValueError, ast.literal_eval, '++6') + self.assertRaises(ValueError, ast.literal_eval, '+True') + self.assertRaises(ValueError, ast.literal_eval, '2+3') + + @unittest.expectedFailure # TODO: RUSTPYTHON; SyntaxError not raised def test_literal_eval_str_int_limit(self): with support.adjust_int_max_str_digits(4000): - ast.literal_eval("3" * 4000) # no error + ast.literal_eval('3'*4000) # no error with self.assertRaises(SyntaxError) as err_ctx: - ast.literal_eval("3" * 4001) - self.assertIn("Exceeds the limit ", str(err_ctx.exception)) - self.assertIn(" Consider hexadecimal ", str(err_ctx.exception)) + ast.literal_eval('3'*4001) + self.assertIn('Exceeds the limit ', str(err_ctx.exception)) + self.assertIn(' Consider hexadecimal ', str(err_ctx.exception)) def test_literal_eval_complex(self): # Issue #4907 - self.assertEqual(ast.literal_eval("6j"), 6j) - self.assertEqual(ast.literal_eval("-6j"), -6j) - self.assertEqual(ast.literal_eval("6.75j"), 6.75j) - self.assertEqual(ast.literal_eval("-6.75j"), -6.75j) - self.assertEqual(ast.literal_eval("3+6j"), 3 + 6j) - self.assertEqual(ast.literal_eval("-3+6j"), -3 + 6j) - self.assertEqual(ast.literal_eval("3-6j"), 3 - 6j) - self.assertEqual(ast.literal_eval("-3-6j"), -3 - 6j) - self.assertEqual(ast.literal_eval("3.25+6.75j"), 3.25 + 6.75j) - self.assertEqual(ast.literal_eval("-3.25+6.75j"), -3.25 + 6.75j) - self.assertEqual(ast.literal_eval("3.25-6.75j"), 3.25 - 6.75j) - self.assertEqual(ast.literal_eval("-3.25-6.75j"), -3.25 - 6.75j) - self.assertEqual(ast.literal_eval("(3+6j)"), 3 + 6j) - self.assertRaises(ValueError, ast.literal_eval, "-6j+3") - self.assertRaises(ValueError, ast.literal_eval, "-6j+3j") - self.assertRaises(ValueError, ast.literal_eval, "3+-6j") - self.assertRaises(ValueError, ast.literal_eval, "3+(0+6j)") - self.assertRaises(ValueError, ast.literal_eval, "-(3+6j)") + self.assertEqual(ast.literal_eval('6j'), 6j) + self.assertEqual(ast.literal_eval('-6j'), -6j) + self.assertEqual(ast.literal_eval('6.75j'), 6.75j) + self.assertEqual(ast.literal_eval('-6.75j'), -6.75j) + self.assertEqual(ast.literal_eval('3+6j'), 3+6j) + self.assertEqual(ast.literal_eval('-3+6j'), -3+6j) + self.assertEqual(ast.literal_eval('3-6j'), 3-6j) + self.assertEqual(ast.literal_eval('-3-6j'), -3-6j) + self.assertEqual(ast.literal_eval('3.25+6.75j'), 3.25+6.75j) + self.assertEqual(ast.literal_eval('-3.25+6.75j'), -3.25+6.75j) + self.assertEqual(ast.literal_eval('3.25-6.75j'), 3.25-6.75j) + self.assertEqual(ast.literal_eval('-3.25-6.75j'), -3.25-6.75j) + self.assertEqual(ast.literal_eval('(3+6j)'), 3+6j) + self.assertRaises(ValueError, ast.literal_eval, '-6j+3') + self.assertRaises(ValueError, ast.literal_eval, '-6j+3j') + self.assertRaises(ValueError, ast.literal_eval, '3+-6j') + self.assertRaises(ValueError, ast.literal_eval, '3+(0+6j)') + self.assertRaises(ValueError, ast.literal_eval, '-(3+6j)') def test_literal_eval_malformed_dict_nodes(self): - malformed = ast.Dict( - keys=[ast.Constant(1), ast.Constant(2)], values=[ast.Constant(3)] - ) + malformed = ast.Dict(keys=[ast.Constant(1), ast.Constant(2)], values=[ast.Constant(3)]) self.assertRaises(ValueError, ast.literal_eval, malformed) - malformed = ast.Dict( - keys=[ast.Constant(1)], values=[ast.Constant(2), ast.Constant(3)] - ) + malformed = ast.Dict(keys=[ast.Constant(1)], values=[ast.Constant(2), ast.Constant(3)]) self.assertRaises(ValueError, ast.literal_eval, malformed) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_literal_eval_trailing_ws(self): self.assertEqual(ast.literal_eval(" -1"), -1) self.assertEqual(ast.literal_eval("\t\t-1"), -1) @@ -1687,58 +1974,51 @@ def test_literal_eval_trailing_ws(self): self.assertRaises(IndentationError, ast.literal_eval, "\n -1") def test_literal_eval_malformed_lineno(self): - msg = r"malformed node or string on line 3:" + msg = r'malformed node or string on line 3:' with self.assertRaisesRegex(ValueError, msg): ast.literal_eval("{'a': 1,\n'b':2,\n'c':++3,\n'd':4}") - node = ast.UnaryOp(ast.UAdd(), ast.UnaryOp(ast.UAdd(), ast.Constant(6))) - self.assertIsNone(getattr(node, "lineno", None)) - msg = r"malformed node or string:" + node = ast.UnaryOp( + ast.UAdd(), ast.UnaryOp(ast.UAdd(), ast.Constant(6))) + self.assertIsNone(getattr(node, 'lineno', None)) + msg = r'malformed node or string:' with self.assertRaisesRegex(ValueError, msg): ast.literal_eval(node) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_literal_eval_syntax_errors(self): with self.assertRaisesRegex(SyntaxError, "unexpected indent"): - ast.literal_eval(r""" + ast.literal_eval(r''' \ (\ - \ """) + \ ''') - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_bad_integer(self): # issue13436: Bad error message with invalid numeric values - body = [ - ast.ImportFrom( - module="time", - names=[ast.alias(name="sleep")], - level=None, - lineno=None, - col_offset=None, - ) - ] + body = [ast.ImportFrom(module='time', + names=[ast.alias(name='sleep')], + level=None, + lineno=None, col_offset=None)] mod = ast.Module(body, []) with self.assertRaises(ValueError) as cm: - compile(mod, "test", "exec") + compile(mod, 'test', 'exec') self.assertIn("invalid integer value: None", str(cm.exception)) def test_level_as_none(self): - body = [ - ast.ImportFrom( - module="time", - names=[ast.alias(name="sleep", lineno=0, col_offset=0)], - level=None, - lineno=0, - col_offset=0, - ) - ] + body = [ast.ImportFrom(module='time', + names=[ast.alias(name='sleep', + lineno=0, col_offset=0)], + level=None, + lineno=0, col_offset=0)] mod = ast.Module(body, []) - code = compile(mod, "test", "exec") + code = compile(mod, 'test', 'exec') ns = {} exec(code, ns) - self.assertIn("sleep", ns) + self.assertIn('sleep', ns) - @unittest.skip('TODO: RUSTPYTHON; crash') + @unittest.skip("TODO: RUSTPYTHON; crash") + @skip_emscripten_stack_overflow() def test_recursion_direct(self): e = ast.UnaryOp(op=ast.Not(), lineno=0, col_offset=0, operand=ast.Constant(1)) e.operand = e @@ -1746,7 +2026,8 @@ def test_recursion_direct(self): with support.infinite_recursion(): compile(ast.Expression(e), "", "eval") - @unittest.skip('TODO: RUSTPYTHON; crash') + @unittest.skip("TODO: RUSTPYTHON; crash") + @skip_emscripten_stack_overflow() def test_recursion_indirect(self): e = ast.UnaryOp(op=ast.Not(), lineno=0, col_offset=0, operand=ast.Constant(1)) f = ast.UnaryOp(op=ast.Not(), lineno=0, col_offset=0, operand=ast.Constant(1)) @@ -1758,6 +2039,7 @@ def test_recursion_indirect(self): class ASTValidatorTests(unittest.TestCase): + def mod(self, mod, msg=None, mode="exec", *, exc=ValueError): mod.lineno = mod.col_offset = 0 ast.fix_missing_locations(mod) @@ -1776,7 +2058,7 @@ def stmt(self, stmt, msg=None): mod = ast.Module([stmt], []) self.mod(mod, msg) - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_module(self): m = ast.Interactive([ast.Expr(ast.Name("x", ast.Store()))]) self.mod(m, "must have Load context", "single") @@ -1784,15 +2066,9 @@ def test_module(self): self.mod(m, "must have Load context", "eval") def _check_arguments(self, fac, check): - def arguments( - args=None, - posonlyargs=None, - vararg=None, - kwonlyargs=None, - kwarg=None, - defaults=None, - kw_defaults=None, - ): + def arguments(args=None, posonlyargs=None, vararg=None, + kwonlyargs=None, kwarg=None, + defaults=None, kw_defaults=None): if args is None: args = [] if posonlyargs is None: @@ -1803,69 +2079,51 @@ def arguments( defaults = [] if kw_defaults is None: kw_defaults = [] - args = ast.arguments( - args, posonlyargs, vararg, kwonlyargs, kw_defaults, kwarg, defaults - ) + args = ast.arguments(args, posonlyargs, vararg, kwonlyargs, + kw_defaults, kwarg, defaults) return fac(args) - args = [ast.arg("x", ast.Name("x", ast.Store()))] check(arguments(args=args), "must have Load context") check(arguments(posonlyargs=args), "must have Load context") check(arguments(kwonlyargs=args), "must have Load context") - check( - arguments(defaults=[ast.Constant(3)]), "more positional defaults than args" - ) - check( - arguments(kw_defaults=[ast.Constant(4)]), - "length of kwonlyargs is not the same as kw_defaults", - ) + check(arguments(defaults=[ast.Constant(3)]), + "more positional defaults than args") + check(arguments(kw_defaults=[ast.Constant(4)]), + "length of kwonlyargs is not the same as kw_defaults") args = [ast.arg("x", ast.Name("x", ast.Load()))] - check( - arguments(args=args, defaults=[ast.Name("x", ast.Store())]), - "must have Load context", - ) - args = [ - ast.arg("a", ast.Name("x", ast.Load())), - ast.arg("b", ast.Name("y", ast.Load())), - ] - check( - arguments(kwonlyargs=args, kw_defaults=[None, ast.Name("x", ast.Store())]), - "must have Load context", - ) - - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + check(arguments(args=args, defaults=[ast.Name("x", ast.Store())]), + "must have Load context") + args = [ast.arg("a", ast.Name("x", ast.Load())), + ast.arg("b", ast.Name("y", ast.Load()))] + check(arguments(kwonlyargs=args, + kw_defaults=[None, ast.Name("x", ast.Store())]), + "must have Load context") + + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_funcdef(self): a = ast.arguments([], [], None, [], [], None, []) f = ast.FunctionDef("x", a, [], [], None, None, []) self.stmt(f, "empty body on FunctionDef") - f = ast.FunctionDef( - "x", a, [ast.Pass()], [ast.Name("x", ast.Store())], None, None, [] - ) + f = ast.FunctionDef("x", a, [ast.Pass()], [ast.Name("x", ast.Store())], None, None, []) self.stmt(f, "must have Load context") - f = ast.FunctionDef( - "x", a, [ast.Pass()], [], ast.Name("x", ast.Store()), None, [] - ) + f = ast.FunctionDef("x", a, [ast.Pass()], [], + ast.Name("x", ast.Store()), None, []) self.stmt(f, "must have Load context") f = ast.FunctionDef("x", ast.arguments(), [ast.Pass()]) self.stmt(f) - def fac(args): return ast.FunctionDef("x", args, [ast.Pass()], [], None, None, []) - self._check_arguments(fac, self.stmt) - @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: class pattern defines no positional sub-patterns (__match_args__ missing) + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: class pattern defines no positional sub-patterns (__match_args__ missing) def test_funcdef_pattern_matching(self): # gh-104799: New fields on FunctionDef should be added at the end def matcher(node): match node: - case ast.FunctionDef( - "foo", - ast.arguments(args=[ast.arg("bar")]), - [ast.Pass()], - [ast.Name("capybara", ast.Load())], - ast.Name("pacarana", ast.Load()), - ): + case ast.FunctionDef("foo", ast.arguments(args=[ast.arg("bar")]), + [ast.Pass()], + [ast.Name("capybara", ast.Load())], + ast.Name("pacarana", ast.Load())): return True case _: return False @@ -1880,11 +2138,9 @@ def foo(bar) -> pacarana: self.assertIsInstance(funcdef, ast.FunctionDef) self.assertTrue(matcher(funcdef)) - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_classdef(self): - def cls( - bases=None, keywords=None, body=None, decorator_list=None, type_params=None - ): + def cls(bases=None, keywords=None, body=None, decorator_list=None, type_params=None): if bases is None: bases = [] if keywords is None: @@ -1895,94 +2151,78 @@ def cls( decorator_list = [] if type_params is None: type_params = [] - return ast.ClassDef( - "myclass", bases, keywords, body, decorator_list, type_params - ) - - self.stmt(cls(bases=[ast.Name("x", ast.Store())]), "must have Load context") - self.stmt( - cls(keywords=[ast.keyword("x", ast.Name("x", ast.Store()))]), - "must have Load context", - ) + return ast.ClassDef("myclass", bases, keywords, + body, decorator_list, type_params) + self.stmt(cls(bases=[ast.Name("x", ast.Store())]), + "must have Load context") + self.stmt(cls(keywords=[ast.keyword("x", ast.Name("x", ast.Store()))]), + "must have Load context") self.stmt(cls(body=[]), "empty body on ClassDef") self.stmt(cls(body=[None]), "None disallowed") - self.stmt( - cls(decorator_list=[ast.Name("x", ast.Store())]), "must have Load context" - ) + self.stmt(cls(decorator_list=[ast.Name("x", ast.Store())]), + "must have Load context") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_delete(self): self.stmt(ast.Delete([]), "empty targets on Delete") self.stmt(ast.Delete([None]), "None disallowed") - self.stmt(ast.Delete([ast.Name("x", ast.Load())]), "must have Del context") + self.stmt(ast.Delete([ast.Name("x", ast.Load())]), + "must have Del context") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_assign(self): self.stmt(ast.Assign([], ast.Constant(3)), "empty targets on Assign") self.stmt(ast.Assign([None], ast.Constant(3)), "None disallowed") - self.stmt( - ast.Assign([ast.Name("x", ast.Load())], ast.Constant(3)), - "must have Store context", - ) - self.stmt( - ast.Assign([ast.Name("x", ast.Store())], ast.Name("y", ast.Store())), - "must have Load context", - ) + self.stmt(ast.Assign([ast.Name("x", ast.Load())], ast.Constant(3)), + "must have Store context") + self.stmt(ast.Assign([ast.Name("x", ast.Store())], + ast.Name("y", ast.Store())), + "must have Load context") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_augassign(self): - aug = ast.AugAssign( - ast.Name("x", ast.Load()), ast.Add(), ast.Name("y", ast.Load()) - ) + aug = ast.AugAssign(ast.Name("x", ast.Load()), ast.Add(), + ast.Name("y", ast.Load())) self.stmt(aug, "must have Store context") - aug = ast.AugAssign( - ast.Name("x", ast.Store()), ast.Add(), ast.Name("y", ast.Store()) - ) + aug = ast.AugAssign(ast.Name("x", ast.Store()), ast.Add(), + ast.Name("y", ast.Store())) self.stmt(aug, "must have Load context") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_for(self): x = ast.Name("x", ast.Store()) y = ast.Name("y", ast.Load()) p = ast.Pass() self.stmt(ast.For(x, y, [], []), "empty body on For") - self.stmt( - ast.For(ast.Name("x", ast.Load()), y, [p], []), "must have Store context" - ) - self.stmt( - ast.For(x, ast.Name("y", ast.Store()), [p], []), "must have Load context" - ) + self.stmt(ast.For(ast.Name("x", ast.Load()), y, [p], []), + "must have Store context") + self.stmt(ast.For(x, ast.Name("y", ast.Store()), [p], []), + "must have Load context") e = ast.Expr(ast.Name("x", ast.Store())) self.stmt(ast.For(x, y, [e], []), "must have Load context") self.stmt(ast.For(x, y, [p], [e]), "must have Load context") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_while(self): self.stmt(ast.While(ast.Constant(3), [], []), "empty body on While") - self.stmt( - ast.While(ast.Name("x", ast.Store()), [ast.Pass()], []), - "must have Load context", - ) - self.stmt( - ast.While( - ast.Constant(3), [ast.Pass()], [ast.Expr(ast.Name("x", ast.Store()))] - ), - "must have Load context", - ) + self.stmt(ast.While(ast.Name("x", ast.Store()), [ast.Pass()], []), + "must have Load context") + self.stmt(ast.While(ast.Constant(3), [ast.Pass()], + [ast.Expr(ast.Name("x", ast.Store()))]), + "must have Load context") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_if(self): self.stmt(ast.If(ast.Constant(3), [], []), "empty body on If") i = ast.If(ast.Name("x", ast.Store()), [ast.Pass()], []) self.stmt(i, "must have Load context") i = ast.If(ast.Constant(3), [ast.Expr(ast.Name("x", ast.Store()))], []) self.stmt(i, "must have Load context") - i = ast.If( - ast.Constant(3), [ast.Pass()], [ast.Expr(ast.Name("x", ast.Store()))] - ) + i = ast.If(ast.Constant(3), [ast.Pass()], + [ast.Expr(ast.Name("x", ast.Store()))]) self.stmt(i, "must have Load context") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_with(self): p = ast.Pass() self.stmt(ast.With([], [p]), "empty items on With") @@ -1993,7 +2233,7 @@ def test_with(self): i = ast.withitem(ast.Constant(3), ast.Name("x", ast.Load())) self.stmt(ast.With([i], [p]), "must have Store context") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_raise(self): r = ast.Raise(None, ast.Constant(3)) self.stmt(r, "Raise with cause but no exception") @@ -2002,7 +2242,7 @@ def test_raise(self): r = ast.Raise(ast.Constant(4), ast.Name("x", ast.Store())) self.stmt(r, "must have Load context") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_try(self): p = ast.Pass() t = ast.Try([], [], [], [p]) @@ -2023,7 +2263,7 @@ def test_try(self): t = ast.Try([p], e, [p], [ast.Expr(ast.Name("x", ast.Store()))]) self.stmt(t, "must have Load context") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_try_star(self): p = ast.Pass() t = ast.TryStar([], [], [], [p]) @@ -2044,38 +2284,38 @@ def test_try_star(self): t = ast.TryStar([p], e, [p], [ast.Expr(ast.Name("x", ast.Store()))]) self.stmt(t, "must have Load context") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_assert(self): - self.stmt( - ast.Assert(ast.Name("x", ast.Store()), None), "must have Load context" - ) - assrt = ast.Assert(ast.Name("x", ast.Load()), ast.Name("y", ast.Store())) + self.stmt(ast.Assert(ast.Name("x", ast.Store()), None), + "must have Load context") + assrt = ast.Assert(ast.Name("x", ast.Load()), + ast.Name("y", ast.Store())) self.stmt(assrt, "must have Load context") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_import(self): self.stmt(ast.Import([]), "empty names on Import") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_importfrom(self): imp = ast.ImportFrom(None, [ast.alias("x", None)], -42) self.stmt(imp, "Negative ImportFrom level") self.stmt(ast.ImportFrom(None, [], 0), "empty names on ImportFrom") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_global(self): self.stmt(ast.Global([]), "empty names on Global") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_nonlocal(self): self.stmt(ast.Nonlocal([]), "empty names on Nonlocal") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_expr(self): e = ast.Expr(ast.Name("x", ast.Store())) self.stmt(e, "must have Load context") - @unittest.skip('TODO: RUSTPYTHON; called `Option::unwrap()` on a `None` value') + @unittest.skip("TODO: RUSTPYTHON; called `Option::unwrap()` on a `None` value") def test_boolop(self): b = ast.BoolOp(ast.And(), []) self.expr(b, "less than 2 values") @@ -2086,36 +2326,35 @@ def test_boolop(self): b = ast.BoolOp(ast.And(), [ast.Constant(4), ast.Name("x", ast.Store())]) self.expr(b, "must have Load context") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_unaryop(self): u = ast.UnaryOp(ast.Not(), ast.Name("x", ast.Store())) self.expr(u, "must have Load context") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_lambda(self): a = ast.arguments([], [], None, [], [], None, []) - self.expr(ast.Lambda(a, ast.Name("x", ast.Store())), "must have Load context") - + self.expr(ast.Lambda(a, ast.Name("x", ast.Store())), + "must have Load context") def fac(args): return ast.Lambda(args, ast.Name("x", ast.Load())) - self._check_arguments(fac, self.expr) - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_ifexp(self): l = ast.Name("x", ast.Load()) s = ast.Name("y", ast.Store()) for args in (s, l, l), (l, s, l), (l, l, s): self.expr(ast.IfExp(*args), "must have Load context") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_dict(self): d = ast.Dict([], [ast.Name("x", ast.Load())]) self.expr(d, "same number of keys as values") d = ast.Dict([ast.Name("x", ast.Load())], [None]) self.expr(d, "None disallowed") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_set(self): self.expr(ast.Set([None]), "None disallowed") s = ast.Set([ast.Name("x", ast.Store())]) @@ -2123,13 +2362,11 @@ def test_set(self): def _check_comprehension(self, fac): self.expr(fac([]), "comprehension with no generators") - g = ast.comprehension( - ast.Name("x", ast.Load()), ast.Name("x", ast.Load()), [], 0 - ) + g = ast.comprehension(ast.Name("x", ast.Load()), + ast.Name("x", ast.Load()), [], 0) self.expr(fac([g]), "must have Store context") - g = ast.comprehension( - ast.Name("x", ast.Store()), ast.Name("x", ast.Store()), [], 0 - ) + g = ast.comprehension(ast.Name("x", ast.Store()), + ast.Name("x", ast.Store()), [], 0) self.expr(fac([g]), "must have Load context") x = ast.Name("x", ast.Store()) y = ast.Name("y", ast.Load()) @@ -2139,46 +2376,43 @@ def _check_comprehension(self, fac): self.expr(fac([g]), "must have Load context") def _simple_comp(self, fac): - g = ast.comprehension( - ast.Name("x", ast.Store()), ast.Name("x", ast.Load()), [], 0 - ) - self.expr(fac(ast.Name("x", ast.Store()), [g]), "must have Load context") - + g = ast.comprehension(ast.Name("x", ast.Store()), + ast.Name("x", ast.Load()), [], 0) + self.expr(fac(ast.Name("x", ast.Store()), [g]), + "must have Load context") def wrap(gens): return fac(ast.Name("x", ast.Store()), gens) - self._check_comprehension(wrap) - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_listcomp(self): self._simple_comp(ast.ListComp) - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_setcomp(self): self._simple_comp(ast.SetComp) - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_generatorexp(self): self._simple_comp(ast.GeneratorExp) - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_dictcomp(self): - g = ast.comprehension( - ast.Name("y", ast.Store()), ast.Name("p", ast.Load()), [], 0 - ) - c = ast.DictComp(ast.Name("x", ast.Store()), ast.Name("y", ast.Load()), [g]) + g = ast.comprehension(ast.Name("y", ast.Store()), + ast.Name("p", ast.Load()), [], 0) + c = ast.DictComp(ast.Name("x", ast.Store()), + ast.Name("y", ast.Load()), [g]) self.expr(c, "must have Load context") - c = ast.DictComp(ast.Name("x", ast.Load()), ast.Name("y", ast.Store()), [g]) + c = ast.DictComp(ast.Name("x", ast.Load()), + ast.Name("y", ast.Store()), [g]) self.expr(c, "must have Load context") - def factory(comps): k = ast.Name("x", ast.Load()) v = ast.Name("y", ast.Load()) return ast.DictComp(k, v, comps) - self._check_comprehension(factory) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_yield(self): self.expr(ast.Yield(ast.Name("x", ast.Store())), "must have Load") self.expr(ast.YieldFrom(ast.Name("x", ast.Store())), "must have Load") @@ -2195,7 +2429,7 @@ def test_compare(self): comp = ast.Compare(left, [ast.In()], [ast.Constant("blah")]) self.expr(comp) - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_call(self): func = ast.Name("x", ast.Load()) args = [ast.Name("y", ast.Load())] @@ -2208,202 +2442,209 @@ def test_call(self): call = ast.Call(func, args, bad_keywords) self.expr(call, "must have Load context") - def test_num(self): - with warnings.catch_warnings(record=True) as wlog: - warnings.filterwarnings("ignore", "", DeprecationWarning) - from ast import Num - - with warnings.catch_warnings(record=True) as wlog: - warnings.filterwarnings("always", "", DeprecationWarning) - - class subint(int): - pass - - class subfloat(float): - pass - - class subcomplex(complex): - pass - - for obj in "0", "hello": - self.expr(ast.Num(obj)) - for obj in subint(), subfloat(), subcomplex(): - self.expr(ast.Num(obj), "invalid type", exc=TypeError) - - self.assertEqual( - [str(w.message) for w in wlog], - [ - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - "ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead", - ], - ) - - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_attribute(self): attr = ast.Attribute(ast.Name("x", ast.Store()), "y", ast.Load()) self.expr(attr, "must have Load context") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_subscript(self): - sub = ast.Subscript(ast.Name("x", ast.Store()), ast.Constant(3), ast.Load()) + sub = ast.Subscript(ast.Name("x", ast.Store()), ast.Constant(3), + ast.Load()) self.expr(sub, "must have Load context") x = ast.Name("x", ast.Load()) - sub = ast.Subscript(x, ast.Name("y", ast.Store()), ast.Load()) + sub = ast.Subscript(x, ast.Name("y", ast.Store()), + ast.Load()) self.expr(sub, "must have Load context") s = ast.Name("x", ast.Store()) for args in (s, None, None), (None, s, None), (None, None, s): sl = ast.Slice(*args) - self.expr(ast.Subscript(x, sl, ast.Load()), "must have Load context") + self.expr(ast.Subscript(x, sl, ast.Load()), + "must have Load context") sl = ast.Tuple([], ast.Load()) self.expr(ast.Subscript(x, sl, ast.Load())) sl = ast.Tuple([s], ast.Load()) self.expr(ast.Subscript(x, sl, ast.Load()), "must have Load context") - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError not raised def test_starred(self): - left = ast.List( - [ast.Starred(ast.Name("x", ast.Load()), ast.Store())], ast.Store() - ) + left = ast.List([ast.Starred(ast.Name("x", ast.Load()), ast.Store())], + ast.Store()) assign = ast.Assign([left], ast.Constant(4)) self.stmt(assign, "must have Store context") def _sequence(self, fac): self.expr(fac([None], ast.Load()), "None disallowed") - self.expr( - fac([ast.Name("x", ast.Store())], ast.Load()), "must have Load context" - ) + self.expr(fac([ast.Name("x", ast.Store())], ast.Load()), + "must have Load context") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_list(self): self._sequence(ast.List) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_tuple(self): self._sequence(ast.Tuple) - def test_nameconstant(self): - with warnings.catch_warnings(record=True) as wlog: - warnings.filterwarnings("ignore", "", DeprecationWarning) - from ast import NameConstant - - with warnings.catch_warnings(record=True) as wlog: - warnings.filterwarnings("always", "", DeprecationWarning) - self.expr(ast.NameConstant(4)) - - self.assertEqual( - [str(w.message) for w in wlog], - [ - "ast.NameConstant is deprecated and will be removed in Python 3.14; use ast.Constant instead", - ], - ) - - @unittest.expectedFailure # TODO: RUSTPYTHON - @support.requires_resource("cpu") + @unittest.expectedFailure # TODO: RUSTPYTHON + @support.requires_resource('cpu') def test_stdlib_validates(self): - stdlib = os.path.dirname(ast.__file__) - tests = [fn for fn in os.listdir(stdlib) if fn.endswith(".py")] - tests.extend(["test/test_grammar.py", "test/test_unpack_ex.py"]) - for module in tests: + for module in STDLIB_FILES: with self.subTest(module): - fn = os.path.join(stdlib, module) + fn = os.path.join(STDLIB, module) with open(fn, "r", encoding="utf-8") as fp: source = fp.read() mod = ast.parse(source, fn) compile(mod, fn, "exec") + mod2 = ast.parse(source, fn) + self.assertTrue(ast.compare(mod, mod2)) constant_1 = ast.Constant(1) pattern_1 = ast.MatchValue(constant_1) - constant_x = ast.Constant("x") + constant_x = ast.Constant('x') pattern_x = ast.MatchValue(constant_x) constant_true = ast.Constant(True) pattern_true = ast.MatchSingleton(True) - name_carter = ast.Name("carter", ast.Load()) + name_carter = ast.Name('carter', ast.Load()) _MATCH_PATTERNS = [ ast.MatchValue( ast.Attribute( - ast.Attribute(ast.Name("x", ast.Store()), "y", ast.Load()), - "z", - ast.Load(), + ast.Attribute( + ast.Name('x', ast.Store()), + 'y', ast.Load() + ), + 'z', ast.Load() ) ), ast.MatchValue( ast.Attribute( - ast.Attribute(ast.Name("x", ast.Load()), "y", ast.Store()), - "z", - ast.Load(), + ast.Attribute( + ast.Name('x', ast.Load()), + 'y', ast.Store() + ), + 'z', ast.Load() ) ), - ast.MatchValue(ast.Constant(...)), - ast.MatchValue(ast.Constant(True)), - ast.MatchValue(ast.Constant((1, 2, 3))), - ast.MatchSingleton("string"), - ast.MatchSequence([ast.MatchSingleton("string")]), - ast.MatchSequence([ast.MatchSequence([ast.MatchSingleton("string")])]), - ast.MatchMapping([constant_1, constant_true], [pattern_x]), + ast.MatchValue( + ast.Constant(...) + ), + ast.MatchValue( + ast.Constant(True) + ), + ast.MatchValue( + ast.Constant((1,2,3)) + ), + ast.MatchSingleton('string'), + ast.MatchSequence([ + ast.MatchSingleton('string') + ]), + ast.MatchSequence( + [ + ast.MatchSequence( + [ + ast.MatchSingleton('string') + ] + ) + ] + ), ast.MatchMapping( - [constant_true, constant_1], [pattern_x, pattern_1], rest="True" + [constant_1, constant_true], + [pattern_x] ), ast.MatchMapping( - [constant_true, ast.Starred(ast.Name("lol", ast.Load()), ast.Load())], + [constant_true, constant_1], [pattern_x, pattern_1], - rest="legit", + rest='True' + ), + ast.MatchMapping( + [constant_true, ast.Starred(ast.Name('lol', ast.Load()), ast.Load())], + [pattern_x, pattern_1], + rest='legit' ), ast.MatchClass( - ast.Attribute(ast.Attribute(constant_x, "y", ast.Load()), "z", ast.Load()), - patterns=[], - kwd_attrs=[], - kwd_patterns=[], + ast.Attribute( + ast.Attribute( + constant_x, + 'y', ast.Load()), + 'z', ast.Load()), + patterns=[], kwd_attrs=[], kwd_patterns=[] ), ast.MatchClass( - name_carter, patterns=[], kwd_attrs=["True"], kwd_patterns=[pattern_1] + name_carter, + patterns=[], + kwd_attrs=['True'], + kwd_patterns=[pattern_1] ), ast.MatchClass( - name_carter, patterns=[], kwd_attrs=[], kwd_patterns=[pattern_1] + name_carter, + patterns=[], + kwd_attrs=[], + kwd_patterns=[pattern_1] ), ast.MatchClass( name_carter, - patterns=[ast.MatchSingleton("string")], + patterns=[ast.MatchSingleton('string')], kwd_attrs=[], - kwd_patterns=[], + kwd_patterns=[] ), ast.MatchClass( - name_carter, patterns=[ast.MatchStar()], kwd_attrs=[], kwd_patterns=[] + name_carter, + patterns=[ast.MatchStar()], + kwd_attrs=[], + kwd_patterns=[] ), ast.MatchClass( - name_carter, patterns=[], kwd_attrs=[], kwd_patterns=[ast.MatchStar()] + name_carter, + patterns=[], + kwd_attrs=[], + kwd_patterns=[ast.MatchStar()] ), ast.MatchClass( constant_true, # invalid name patterns=[], - kwd_attrs=["True"], - kwd_patterns=[pattern_1], + kwd_attrs=['True'], + kwd_patterns=[pattern_1] + ), + ast.MatchSequence( + [ + ast.MatchStar("True") + ] + ), + ast.MatchAs( + name='False' + ), + ast.MatchOr( + [] + ), + ast.MatchOr( + [pattern_1] + ), + ast.MatchOr( + [pattern_1, pattern_x, ast.MatchSingleton('xxx')] ), - ast.MatchSequence([ast.MatchStar("True")]), - ast.MatchAs(name="False"), - ast.MatchOr([]), - ast.MatchOr([pattern_1]), - ast.MatchOr([pattern_1, pattern_x, ast.MatchSingleton("xxx")]), ast.MatchAs(name="_"), ast.MatchStar(name="x"), ast.MatchSequence([ast.MatchStar("_")]), ast.MatchMapping([], [], rest="_"), ] - @unittest.skip("TODO: RUSTPYTHON; thread 'main' panicked") + @unittest.expectedFailure # TODO: RUSTPYTHON def test_match_validation_pattern(self): - name_x = ast.Name("x", ast.Load()) + name_x = ast.Name('x', ast.Load()) for pattern in self._MATCH_PATTERNS: with self.subTest(ast.dump(pattern, indent=4)): node = ast.Match( subject=name_x, - cases=[ast.match_case(pattern=pattern, body=[ast.Pass()])], + cases = [ + ast.match_case( + pattern=pattern, + body = [ast.Pass()] + ) + ] ) node = ast.fix_missing_locations(node) module = ast.Module([node], []) @@ -2426,44 +2667,38 @@ def compile_constant(self, value): ns = {} exec(code, ns) - return ns["x"] + return ns['x'] def test_validation(self): with self.assertRaises(TypeError) as cm: self.compile_constant([1, 2, 3]) - self.assertEqual(str(cm.exception), "got an invalid type in Constant: list") + self.assertEqual(str(cm.exception), + "got an invalid type in Constant: list") - @unittest.expectedFailure # TODO: RUSTPYTHON; b'' is not b'' + @unittest.expectedFailure # TODO: RUSTPYTHON; b'' is not b'' def test_singletons(self): - for const in (None, False, True, Ellipsis, b"", frozenset()): + for const in (None, False, True, Ellipsis, b''): with self.subTest(const=const): value = self.compile_constant(const) self.assertIs(value, const) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_values(self): nested_tuple = (1,) nested_frozenset = frozenset({1}) for level in range(3): nested_tuple = (nested_tuple, 2) nested_frozenset = frozenset({nested_frozenset, 2}) - values = ( - 123, - 123.0, - 123j, - "unicode", - b"bytes", - tuple("tuple"), - frozenset("frozenset"), - nested_tuple, - nested_frozenset, - ) + values = (123, 123.0, 123j, + "unicode", b'bytes', + tuple("tuple"), frozenset("frozenset"), + nested_tuple, nested_frozenset) for value in values: with self.subTest(value=value): result = self.compile_constant(value) self.assertEqual(result, value) - @unittest.expectedFailure # TODO: RUSTPYTHON; SyntaxError: cannot assign to literal + @unittest.expectedFailure # TODO: RUSTPYTHON; SyntaxError: cannot assign to literal def test_assign_to_constant(self): tree = ast.parse("x = 1") @@ -2474,35 +2709,42 @@ def test_assign_to_constant(self): with self.assertRaises(ValueError) as cm: compile(tree, "string", "exec") - self.assertEqual( - str(cm.exception), - "expression which can't be assigned " "to in Store context", - ) + self.assertEqual(str(cm.exception), + "expression which can't be assigned " + "to in Store context") def test_get_docstring(self): tree = ast.parse("'docstring'\nx = 1") - self.assertEqual(ast.get_docstring(tree), "docstring") + self.assertEqual(ast.get_docstring(tree), 'docstring') def get_load_const(self, tree): # Compile to bytecode, disassemble and get parameter of LOAD_CONST # instructions - co = compile(tree, "", "exec") + co = compile(tree, '', 'exec') consts = [] for instr in dis.get_instructions(co): - if instr.opname == "LOAD_CONST" or instr.opname == "RETURN_CONST": + if instr.opcode in dis.hasconst: consts.append(instr.argval) return consts @support.cpython_only def test_load_const(self): - consts = [None, True, False, 124, 2.0, 3j, "unicode", b"bytes", (1, 2, 3)] - - code = "\n".join(["x={!r}".format(const) for const in consts]) - code += "\nx = ..." + consts = [None, + True, False, + 1000, + 2.0, + 3j, + "unicode", + b'bytes', + (1, 2, 3)] + + code = '\n'.join(['x={!r}'.format(const) for const in consts]) + code += '\nx = ...' consts.extend((Ellipsis, None)) tree = ast.parse(code) - self.assertEqual(self.get_load_const(tree), consts) + self.assertEqual(self.get_load_const(tree), + consts) # Replace expression nodes with constants for assign, const in zip(tree.body, consts): @@ -2511,7 +2753,8 @@ def test_load_const(self): ast.copy_location(new_node, assign.value) assign.value = new_node - self.assertEqual(self.get_load_const(tree), consts) + self.assertEqual(self.get_load_const(tree), + consts) def test_literal_eval(self): tree = ast.parse("1 + 2") @@ -2525,22 +2768,22 @@ def test_literal_eval(self): ast.copy_location(new_right, binop.right) binop.right = new_right - self.assertEqual(ast.literal_eval(binop), 10 + 20j) + self.assertEqual(ast.literal_eval(binop), 10+20j) def test_string_kind(self): - c = ast.parse('"x"', mode="eval").body + c = ast.parse('"x"', mode='eval').body self.assertEqual(c.value, "x") self.assertEqual(c.kind, None) - c = ast.parse('u"x"', mode="eval").body + c = ast.parse('u"x"', mode='eval').body self.assertEqual(c.value, "x") self.assertEqual(c.kind, "u") - c = ast.parse('r"x"', mode="eval").body + c = ast.parse('r"x"', mode='eval').body self.assertEqual(c.value, "x") self.assertEqual(c.kind, None) - c = ast.parse('b"x"', mode="eval").body + c = ast.parse('b"x"', mode='eval').body self.assertEqual(c.value, b"x") self.assertEqual(c.kind, None) @@ -2551,7 +2794,6 @@ class EndPositionTests(unittest.TestCase): Testing end positions of nodes requires a bit of extra care because of how LL parsers work. """ - def _check_end_pos(self, ast_node, end_lineno, end_col_offset): self.assertEqual(ast_node.end_lineno, end_lineno) self.assertEqual(ast_node.end_col_offset, end_col_offset) @@ -2565,55 +2807,55 @@ def _parse_value(self, s): return ast.parse(s).body[0].value def test_lambda(self): - s = "lambda x, *y: None" + s = 'lambda x, *y: None' lam = self._parse_value(s) - self._check_content(s, lam.body, "None") - self._check_content(s, lam.args.args[0], "x") - self._check_content(s, lam.args.vararg, "y") + self._check_content(s, lam.body, 'None') + self._check_content(s, lam.args.args[0], 'x') + self._check_content(s, lam.args.vararg, 'y') def test_func_def(self): - s = dedent(""" + s = dedent(''' def func(x: int, *args: str, z: float = 0, **kwargs: Any) -> bool: return True - """).strip() + ''').strip() fdef = ast.parse(s).body[0] self._check_end_pos(fdef, 5, 15) - self._check_content(s, fdef.body[0], "return True") - self._check_content(s, fdef.args.args[0], "x: int") - self._check_content(s, fdef.args.args[0].annotation, "int") - self._check_content(s, fdef.args.kwarg, "kwargs: Any") - self._check_content(s, fdef.args.kwarg.annotation, "Any") + self._check_content(s, fdef.body[0], 'return True') + self._check_content(s, fdef.args.args[0], 'x: int') + self._check_content(s, fdef.args.args[0].annotation, 'int') + self._check_content(s, fdef.args.kwarg, 'kwargs: Any') + self._check_content(s, fdef.args.kwarg.annotation, 'Any') def test_call(self): - s = "func(x, y=2, **kw)" + s = 'func(x, y=2, **kw)' call = self._parse_value(s) - self._check_content(s, call.func, "func") - self._check_content(s, call.keywords[0].value, "2") - self._check_content(s, call.keywords[1].value, "kw") + self._check_content(s, call.func, 'func') + self._check_content(s, call.keywords[0].value, '2') + self._check_content(s, call.keywords[1].value, 'kw') def test_call_noargs(self): - s = "x[0]()" + s = 'x[0]()' call = self._parse_value(s) - self._check_content(s, call.func, "x[0]") + self._check_content(s, call.func, 'x[0]') self._check_end_pos(call, 1, 6) def test_class_def(self): - s = dedent(""" + s = dedent(''' class C(A, B): x: int = 0 - """).strip() + ''').strip() cdef = ast.parse(s).body[0] self._check_end_pos(cdef, 2, 14) - self._check_content(s, cdef.bases[1], "B") - self._check_content(s, cdef.body[0], "x: int = 0") + self._check_content(s, cdef.bases[1], 'B') + self._check_content(s, cdef.body[0], 'x: int = 0') def test_class_kw(self): - s = "class S(metaclass=abc.ABCMeta): pass" + s = 'class S(metaclass=abc.ABCMeta): pass' cdef = ast.parse(s).body[0] - self._check_content(s, cdef.keywords[0].value, "abc.ABCMeta") + self._check_content(s, cdef.keywords[0].value, 'abc.ABCMeta') def test_multi_line_str(self): s = dedent(''' @@ -2626,10 +2868,10 @@ def test_multi_line_str(self): self._check_end_pos(assign.value, 3, 40) def test_continued_str(self): - s = dedent(""" + s = dedent(''' x = "first part" \\ "second part" - """).strip() + ''').strip() assign = ast.parse(s).body[0] self._check_end_pos(assign, 2, 13) self._check_end_pos(assign.value, 2, 13) @@ -2637,7 +2879,7 @@ def test_continued_str(self): def test_suites(self): # We intentionally put these into the same string to check # that empty lines are not part of the suite. - s = dedent(""" + s = dedent(''' while True: pass @@ -2657,7 +2899,7 @@ def test_suites(self): pass pass - """).strip() + ''').strip() mod = ast.parse(s) while_loop = mod.body[0] if_stmt = mod.body[1] @@ -2671,18 +2913,18 @@ def test_suites(self): self._check_end_pos(try_stmt, 17, 8) self._check_end_pos(pass_stmt, 19, 4) - self._check_content(s, while_loop.test, "True") - self._check_content(s, if_stmt.body[0], "x = None") - self._check_content(s, if_stmt.orelse[0].test, "other()") - self._check_content(s, for_loop.target, "x, y") - self._check_content(s, try_stmt.body[0], "raise RuntimeError") - self._check_content(s, try_stmt.handlers[0].type, "TypeError") + self._check_content(s, while_loop.test, 'True') + self._check_content(s, if_stmt.body[0], 'x = None') + self._check_content(s, if_stmt.orelse[0].test, 'other()') + self._check_content(s, for_loop.target, 'x, y') + self._check_content(s, try_stmt.body[0], 'raise RuntimeError') + self._check_content(s, try_stmt.handlers[0].type, 'TypeError') def test_fstring(self): s = 'x = f"abc {x + y} abc"' fstr = self._parse_value(s) binop = fstr.values[1].value - self._check_content(s, binop, "x + y") + self._check_content(s, binop, 'x + y') def test_fstring_multi_line(self): s = dedent(''' @@ -2697,198 +2939,200 @@ def test_fstring_multi_line(self): fstr = self._parse_value(s) binop = fstr.values[1].value self._check_end_pos(binop, 5, 7) - self._check_content(s, binop.left, "arg_one") - self._check_content(s, binop.right, "arg_two") + self._check_content(s, binop.left, 'arg_one') + self._check_content(s, binop.right, 'arg_two') def test_import_from_multi_line(self): - s = dedent(""" + s = dedent(''' from x.y.z import ( a, b, c as c ) - """).strip() + ''').strip() imp = ast.parse(s).body[0] self._check_end_pos(imp, 3, 1) self._check_end_pos(imp.names[2], 2, 16) def test_slices(self): - s1 = "f()[1, 2] [0]" - s2 = "x[ a.b: c.d]" - sm = dedent(""" + s1 = 'f()[1, 2] [0]' + s2 = 'x[ a.b: c.d]' + sm = dedent(''' x[ a.b: f () , g () : c.d ] - """).strip() + ''').strip() i1, i2, im = map(self._parse_value, (s1, s2, sm)) - self._check_content(s1, i1.value, "f()[1, 2]") - self._check_content(s1, i1.value.slice, "1, 2") - self._check_content(s2, i2.slice.lower, "a.b") - self._check_content(s2, i2.slice.upper, "c.d") - self._check_content(sm, im.slice.elts[0].upper, "f ()") - self._check_content(sm, im.slice.elts[1].lower, "g ()") + self._check_content(s1, i1.value, 'f()[1, 2]') + self._check_content(s1, i1.value.slice, '1, 2') + self._check_content(s2, i2.slice.lower, 'a.b') + self._check_content(s2, i2.slice.upper, 'c.d') + self._check_content(sm, im.slice.elts[0].upper, 'f ()') + self._check_content(sm, im.slice.elts[1].lower, 'g ()') self._check_end_pos(im, 3, 3) def test_binop(self): - s = dedent(""" + s = dedent(''' (1 * 2 + (3 ) + 4 ) - """).strip() + ''').strip() binop = self._parse_value(s) self._check_end_pos(binop, 2, 6) - self._check_content(s, binop.right, "4") - self._check_content(s, binop.left, "1 * 2 + (3 )") - self._check_content(s, binop.left.right, "3") + self._check_content(s, binop.right, '4') + self._check_content(s, binop.left, '1 * 2 + (3 )') + self._check_content(s, binop.left.right, '3') def test_boolop(self): - s = dedent(""" + s = dedent(''' if (one_condition and (other_condition or yet_another_one)): pass - """).strip() + ''').strip() bop = ast.parse(s).body[0].test self._check_end_pos(bop, 2, 44) - self._check_content(s, bop.values[1], "other_condition or yet_another_one") + self._check_content(s, bop.values[1], + 'other_condition or yet_another_one') def test_tuples(self): - s1 = "x = () ;" - s2 = "x = 1 , ;" - s3 = "x = (1 , 2 ) ;" - sm = dedent(""" + s1 = 'x = () ;' + s2 = 'x = 1 , ;' + s3 = 'x = (1 , 2 ) ;' + sm = dedent(''' x = ( a, b, ) - """).strip() + ''').strip() t1, t2, t3, tm = map(self._parse_value, (s1, s2, s3, sm)) - self._check_content(s1, t1, "()") - self._check_content(s2, t2, "1 ,") - self._check_content(s3, t3, "(1 , 2 )") + self._check_content(s1, t1, '()') + self._check_content(s2, t2, '1 ,') + self._check_content(s3, t3, '(1 , 2 )') self._check_end_pos(tm, 3, 1) def test_attribute_spaces(self): - s = "func(x. y .z)" + s = 'func(x. y .z)' call = self._parse_value(s) self._check_content(s, call, s) - self._check_content(s, call.args[0], "x. y .z") + self._check_content(s, call.args[0], 'x. y .z') def test_redundant_parenthesis(self): - s = "( ( ( a + b ) ) )" + s = '( ( ( a + b ) ) )' v = ast.parse(s).body[0].value - self.assertEqual(type(v).__name__, "BinOp") - self._check_content(s, v, "a + b") - s2 = "await " + s + self.assertEqual(type(v).__name__, 'BinOp') + self._check_content(s, v, 'a + b') + s2 = 'await ' + s v = ast.parse(s2).body[0].value.value - self.assertEqual(type(v).__name__, "BinOp") - self._check_content(s2, v, "a + b") + self.assertEqual(type(v).__name__, 'BinOp') + self._check_content(s2, v, 'a + b') def test_trailers_with_redundant_parenthesis(self): tests = ( - ("( ( ( a ) ) ) ( )", "Call"), - ("( ( ( a ) ) ) ( b )", "Call"), - ("( ( ( a ) ) ) [ b ]", "Subscript"), - ("( ( ( a ) ) ) . b", "Attribute"), + ('( ( ( a ) ) ) ( )', 'Call'), + ('( ( ( a ) ) ) ( b )', 'Call'), + ('( ( ( a ) ) ) [ b ]', 'Subscript'), + ('( ( ( a ) ) ) . b', 'Attribute'), ) for s, t in tests: with self.subTest(s): v = ast.parse(s).body[0].value self.assertEqual(type(v).__name__, t) self._check_content(s, v, s) - s2 = "await " + s + s2 = 'await ' + s v = ast.parse(s2).body[0].value.value self.assertEqual(type(v).__name__, t) self._check_content(s2, v, s) def test_displays(self): - s1 = "[{}, {1, }, {1, 2,} ]" - s2 = "{a: b, f (): g () ,}" + s1 = '[{}, {1, }, {1, 2,} ]' + s2 = '{a: b, f (): g () ,}' c1 = self._parse_value(s1) c2 = self._parse_value(s2) - self._check_content(s1, c1.elts[0], "{}") - self._check_content(s1, c1.elts[1], "{1, }") - self._check_content(s1, c1.elts[2], "{1, 2,}") - self._check_content(s2, c2.keys[1], "f ()") - self._check_content(s2, c2.values[1], "g ()") + self._check_content(s1, c1.elts[0], '{}') + self._check_content(s1, c1.elts[1], '{1, }') + self._check_content(s1, c1.elts[2], '{1, 2,}') + self._check_content(s2, c2.keys[1], 'f ()') + self._check_content(s2, c2.values[1], 'g ()') def test_comprehensions(self): - s = dedent(""" + s = dedent(''' x = [{x for x, y in stuff if cond.x} for stuff in things] - """).strip() + ''').strip() cmp = self._parse_value(s) self._check_end_pos(cmp, 2, 37) - self._check_content(s, cmp.generators[0].iter, "things") - self._check_content(s, cmp.elt.generators[0].iter, "stuff") - self._check_content(s, cmp.elt.generators[0].ifs[0], "cond.x") - self._check_content(s, cmp.elt.generators[0].target, "x, y") + self._check_content(s, cmp.generators[0].iter, 'things') + self._check_content(s, cmp.elt.generators[0].iter, 'stuff') + self._check_content(s, cmp.elt.generators[0].ifs[0], 'cond.x') + self._check_content(s, cmp.elt.generators[0].target, 'x, y') def test_yield_await(self): - s = dedent(""" + s = dedent(''' async def f(): yield x await y - """).strip() + ''').strip() fdef = ast.parse(s).body[0] - self._check_content(s, fdef.body[0].value, "yield x") - self._check_content(s, fdef.body[1].value, "await y") + self._check_content(s, fdef.body[0].value, 'yield x') + self._check_content(s, fdef.body[1].value, 'await y') def test_source_segment_multi(self): - s_orig = dedent(""" + s_orig = dedent(''' x = ( a, b, ) + () - """).strip() - s_tuple = dedent(""" + ''').strip() + s_tuple = dedent(''' ( a, b, ) - """).strip() + ''').strip() binop = self._parse_value(s_orig) self.assertEqual(ast.get_source_segment(s_orig, binop.left), s_tuple) def test_source_segment_padded(self): - s_orig = dedent(""" + s_orig = dedent(''' class C: def fun(self) -> None: "ЖЖЖЖЖ" - """).strip() - s_method = " def fun(self) -> None:\n" ' "ЖЖЖЖЖ"' + ''').strip() + s_method = ' def fun(self) -> None:\n' \ + ' "ЖЖЖЖЖ"' cdef = ast.parse(s_orig).body[0] - self.assertEqual( - ast.get_source_segment(s_orig, cdef.body[0], padded=True), s_method - ) + self.assertEqual(ast.get_source_segment(s_orig, cdef.body[0], padded=True), + s_method) def test_source_segment_endings(self): - s = "v = 1\r\nw = 1\nx = 1\n\ry = 1\rz = 1\r\n" + s = 'v = 1\r\nw = 1\nx = 1\n\ry = 1\rz = 1\r\n' v, w, x, y, z = ast.parse(s).body - self._check_content(s, v, "v = 1") - self._check_content(s, w, "w = 1") - self._check_content(s, x, "x = 1") - self._check_content(s, y, "y = 1") - self._check_content(s, z, "z = 1") + self._check_content(s, v, 'v = 1') + self._check_content(s, w, 'w = 1') + self._check_content(s, x, 'x = 1') + self._check_content(s, y, 'y = 1') + self._check_content(s, z, 'z = 1') def test_source_segment_tabs(self): - s = dedent(""" + s = dedent(''' class C: \t\f def fun(self) -> None: \t\f pass - """).strip() - s_method = " \t\f def fun(self) -> None:\n" " \t\f pass" + ''').strip() + s_method = ' \t\f def fun(self) -> None:\n' \ + ' \t\f pass' cdef = ast.parse(s).body[0] self.assertEqual(ast.get_source_segment(s, cdef.body[0], padded=True), s_method) def test_source_segment_newlines(self): - s = "def f():\n pass\ndef g():\r pass\r\ndef h():\r\n pass\r\n" + s = 'def f():\n pass\ndef g():\r pass\r\ndef h():\r\n pass\r\n' f, g, h = ast.parse(s).body - self._check_content(s, f, "def f():\n pass") - self._check_content(s, g, "def g():\r pass") - self._check_content(s, h, "def h():\r\n pass") + self._check_content(s, f, 'def f():\n pass') + self._check_content(s, g, 'def g():\r pass') + self._check_content(s, h, 'def h():\r\n pass') - s = "def f():\n a = 1\r b = 2\r\n c = 3\n" + s = 'def f():\n a = 1\r b = 2\r\n c = 3\n' f = ast.parse(s).body[0] self._check_content(s, f, s.rstrip()) def test_source_segment_missing_info(self): - s = "v = 1\r\nw = 1\nx = 1\n\ry = 1\r\n" + s = 'v = 1\r\nw = 1\nx = 1\n\ry = 1\r\n' v, w, x, y = ast.parse(s).body del v.lineno del w.end_lineno @@ -2900,102 +3144,28 @@ def test_source_segment_missing_info(self): self.assertIsNone(ast.get_source_segment(s, y)) -class BaseNodeVisitorCases: - # Both `NodeVisitor` and `NodeTranformer` must raise these warnings: - def test_old_constant_nodes(self): - class Visitor(self.visitor_class): - def visit_Num(self, node): - log.append((node.lineno, "Num", node.n)) - - def visit_Str(self, node): - log.append((node.lineno, "Str", node.s)) - - def visit_Bytes(self, node): - log.append((node.lineno, "Bytes", node.s)) - - def visit_NameConstant(self, node): - log.append((node.lineno, "NameConstant", node.value)) - - def visit_Ellipsis(self, node): - log.append((node.lineno, "Ellipsis", ...)) - - mod = ast.parse( - dedent("""\ - i = 42 - f = 4.25 - c = 4.25j - s = 'string' - b = b'bytes' - t = True - n = None - e = ... - """) - ) - visitor = Visitor() - log = [] - with warnings.catch_warnings(record=True) as wlog: - warnings.filterwarnings("always", "", DeprecationWarning) - visitor.visit(mod) - self.assertEqual( - log, - [ - (1, "Num", 42), - (2, "Num", 4.25), - (3, "Num", 4.25j), - (4, "Str", "string"), - (5, "Bytes", b"bytes"), - (6, "NameConstant", True), - (7, "NameConstant", None), - (8, "Ellipsis", ...), - ], - ) - self.assertEqual( - [str(w.message) for w in wlog], - [ - "visit_Num is deprecated; add visit_Constant", - "Attribute n is deprecated and will be removed in Python 3.14; use value instead", - "visit_Num is deprecated; add visit_Constant", - "Attribute n is deprecated and will be removed in Python 3.14; use value instead", - "visit_Num is deprecated; add visit_Constant", - "Attribute n is deprecated and will be removed in Python 3.14; use value instead", - "visit_Str is deprecated; add visit_Constant", - "Attribute s is deprecated and will be removed in Python 3.14; use value instead", - "visit_Bytes is deprecated; add visit_Constant", - "Attribute s is deprecated and will be removed in Python 3.14; use value instead", - "visit_NameConstant is deprecated; add visit_Constant", - "visit_NameConstant is deprecated; add visit_Constant", - "visit_Ellipsis is deprecated; add visit_Constant", - ], - ) - - -class NodeVisitorTests(BaseNodeVisitorCases, unittest.TestCase): - visitor_class = ast.NodeVisitor - - -class NodeTransformerTests(ASTTestMixin, BaseNodeVisitorCases, unittest.TestCase): - visitor_class = ast.NodeTransformer - - def assertASTTransformation(self, tranformer_class, initial_code, expected_code): +class NodeTransformerTests(ASTTestMixin, unittest.TestCase): + def assertASTTransformation(self, transformer_class, + initial_code, expected_code): initial_ast = ast.parse(dedent(initial_code)) expected_ast = ast.parse(dedent(expected_code)) - tranformer = tranformer_class() - result_ast = ast.fix_missing_locations(tranformer.visit(initial_ast)) + transformer = transformer_class() + result_ast = ast.fix_missing_locations(transformer.visit(initial_ast)) self.assertASTEqual(result_ast, expected_ast) - @unittest.expectedFailure # TODO: RUSTPYTHON; is not + @unittest.expectedFailure # TODO: RUSTPYTHON; is not def test_node_remove_single(self): - code = "def func(arg) -> SomeType: ..." - expected = "def func(arg): ..." + code = 'def func(arg) -> SomeType: ...' + expected = 'def func(arg): ...' # Since `FunctionDef.returns` is defined as a single value, we test # the `if isinstance(old_value, AST):` branch here. class SomeTypeRemover(ast.NodeTransformer): def visit_Name(self, node: ast.Name): self.generic_visit(node) - if node.id == "SomeType": + if node.id == 'SomeType': return None return node @@ -3023,7 +3193,6 @@ def visit_Expr(self, node: ast.Expr): self.assertASTTransformation(YieldRemover, code, expected) - @unittest.expectedFailure # TODO: RUSTPYTHON; is not def test_node_return_list(self): code = """ class DSL(Base, kw1=True): ... @@ -3035,11 +3204,11 @@ class DSL(Base, kw1=True, kw2=True, kw3=False): ... class ExtendKeywords(ast.NodeTransformer): def visit_keyword(self, node: ast.keyword): self.generic_visit(node) - if node.arg == "kw1": + if node.arg == 'kw1': return [ node, - ast.keyword("kw2", ast.Constant(True)), - ast.keyword("kw3", ast.Constant(False)), + ast.keyword('kw2', ast.Constant(True)), + ast.keyword('kw3', ast.Constant(False)), ] return node @@ -3058,13 +3227,12 @@ def func(arg): class PrintToLog(ast.NodeTransformer): def visit_Call(self, node: ast.Call): self.generic_visit(node) - if isinstance(node.func, ast.Name) and node.func.id == "print": - node.func.id = "log" + if isinstance(node.func, ast.Name) and node.func.id == 'print': + node.func.id = 'log' return node self.assertASTTransformation(PrintToLog, code, expected) - @unittest.expectedFailure # TODO: RUSTPYTHON; is not def test_node_replace(self): code = """ def func(arg): @@ -3078,15 +3246,15 @@ def func(arg): class PrintToLog(ast.NodeTransformer): def visit_Call(self, node: ast.Call): self.generic_visit(node) - if isinstance(node.func, ast.Name) and node.func.id == "print": + if isinstance(node.func, ast.Name) and node.func.id == 'print': return ast.Call( func=ast.Attribute( - ast.Name("logger", ctx=ast.Load()), - attr="log", + ast.Name('logger', ctx=ast.Load()), + attr='log', ctx=ast.Load(), ), args=node.args, - keywords=[ast.keyword("debug", ast.Constant(True))], + keywords=[ast.keyword('debug', ast.Constant(True))], ) return node @@ -3096,23 +3264,21 @@ def visit_Call(self, node: ast.Call): class ASTConstructorTests(unittest.TestCase): """Test the autogenerated constructors for AST nodes.""" - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_FunctionDef(self): args = ast.arguments() self.assertEqual(args.args, []) self.assertEqual(args.posonlyargs, []) - with self.assertWarnsRegex( - DeprecationWarning, - r"FunctionDef\.__init__ missing 1 required positional argument: 'name'", - ): + with self.assertWarnsRegex(DeprecationWarning, + r"FunctionDef\.__init__ missing 1 required positional argument: 'name'"): node = ast.FunctionDef(args=args) - self.assertFalse(hasattr(node, "name")) + self.assertNotHasAttr(node, "name") self.assertEqual(node.decorator_list, []) - node = ast.FunctionDef(name="foo", args=args) - self.assertEqual(node.name, "foo") + node = ast.FunctionDef(name='foo', args=args) + self.assertEqual(node.name, 'foo') self.assertEqual(node.decorator_list, []) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_expr_context(self): name = ast.Name("x") self.assertEqual(name.id, "x") @@ -3126,10 +3292,8 @@ def test_expr_context(self): self.assertEqual(name3.id, "x") self.assertIsInstance(name3.ctx, ast.Del) - with self.assertWarnsRegex( - DeprecationWarning, - r"Name\.__init__ missing 1 required positional argument: 'id'", - ): + with self.assertWarnsRegex(DeprecationWarning, + r"Name\.__init__ missing 1 required positional argument: 'id'"): name3 = ast.Name() def test_custom_subclass_with_no_fields(self): @@ -3142,7 +3306,7 @@ class NoInit(ast.AST): def test_fields_but_no_field_types(self): class Fields(ast.AST): - _fields = ("a",) + _fields = ('a',) obj = Fields() with self.assertRaises(AttributeError): @@ -3152,8 +3316,8 @@ class Fields(ast.AST): def test_fields_and_types(self): class FieldsAndTypes(ast.AST): - _fields = ("a",) - _field_types = {"a": int | None} + _fields = ('a',) + _field_types = {'a': int | None} a: int | None = None obj = FieldsAndTypes() @@ -3161,7 +3325,7 @@ class FieldsAndTypes(ast.AST): obj = FieldsAndTypes(a=1) self.assertEqual(obj.a, 1) - @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_custom_attributes(self): class MyAttrs(ast.AST): _attributes = ("a", "b") @@ -3170,39 +3334,35 @@ class MyAttrs(ast.AST): self.assertEqual(obj.a, 1) self.assertEqual(obj.b, 2) - with self.assertWarnsRegex( - DeprecationWarning, - r"MyAttrs.__init__ got an unexpected keyword argument 'c'.", - ): + with self.assertWarnsRegex(DeprecationWarning, + r"MyAttrs.__init__ got an unexpected keyword argument 'c'."): obj = MyAttrs(c=3) - @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_fields_and_types_no_default(self): class FieldsAndTypesNoDefault(ast.AST): - _fields = ("a",) - _field_types = {"a": int} + _fields = ('a',) + _field_types = {'a': int} - with self.assertWarnsRegex( - DeprecationWarning, - r"FieldsAndTypesNoDefault\.__init__ missing 1 required positional argument: 'a'\.", - ): + with self.assertWarnsRegex(DeprecationWarning, + r"FieldsAndTypesNoDefault\.__init__ missing 1 required positional argument: 'a'\."): obj = FieldsAndTypesNoDefault() with self.assertRaises(AttributeError): obj.a obj = FieldsAndTypesNoDefault(a=1) self.assertEqual(obj.a, 1) - @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_incomplete_field_types(self): class MoreFieldsThanTypes(ast.AST): - _fields = ("a", "b") - _field_types = {"a": int | None} + _fields = ('a', 'b') + _field_types = {'a': int | None} a: int | None = None b: int | None = None with self.assertWarnsRegex( DeprecationWarning, - r"Field 'b' is missing from MoreFieldsThanTypes\._field_types", + r"Field 'b' is missing from MoreFieldsThanTypes\._field_types" ): obj = MoreFieldsThanTypes() self.assertIs(obj.a, None) @@ -3212,11 +3372,21 @@ class MoreFieldsThanTypes(ast.AST): self.assertEqual(obj.a, 1) self.assertEqual(obj.b, 2) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: Expected type 'str' but 'bytes' found. + def test_malformed_fields_with_bytes(self): + class BadFields(ast.AST): + _fields = (b'\xff'*64,) + _field_types = {'a': int} + + # This should not crash + with self.assertWarnsRegex(DeprecationWarning, r"Field b'\\xff\\xff.*' .*"): + obj = BadFields() + + @unittest.expectedFailure # TODO: RUSTPYTHON def test_complete_field_types(self): class _AllFieldTypes(ast.AST): - _fields = ("a", "b") - _field_types = {"a": int | None, "b": list[str]} + _fields = ('a', 'b') + _field_types = {'a': int | None, 'b': list[str]} # This must be set explicitly a: int | None = None # This will add an implicit empty list default @@ -3233,9 +3403,9 @@ class ModuleStateTests(unittest.TestCase): def check_ast_module(self): # Check that the _ast module still works as expected - code = "x + 1" - filename = "" - mode = "eval" + code = 'x + 1' + filename = '' + mode = 'eval' # Create _ast.AST subclasses instances ast_tree = compile(code, filename, mode, flags=ast.PyCF_ONLY_AST) @@ -3246,11 +3416,11 @@ def check_ast_module(self): def test_reload_module(self): # bpo-41194: Importing the _ast module twice must not crash. - with support.swap_item(sys.modules, "_ast", None): - del sys.modules["_ast"] + with support.swap_item(sys.modules, '_ast', None): + del sys.modules['_ast'] import _ast as ast1 - del sys.modules["_ast"] + del sys.modules['_ast'] import _ast as ast2 self.check_ast_module() @@ -3271,23 +3441,22 @@ def my_import(name, *args, **kw): sys.modules[name] = lazy_mod return lazy_mod - with support.swap_item(sys.modules, "_ast", None): - del sys.modules["_ast"] + with support.swap_item(sys.modules, '_ast', None): + del sys.modules['_ast'] - with support.swap_attr(builtins, "__import__", my_import): + with support.swap_attr(builtins, '__import__', my_import): # Test that compile() does not import the _ast module self.check_ast_module() - self.assertNotIn("_ast", sys.modules) + self.assertNotIn('_ast', sys.modules) # Sanity check of the test itself import _ast - self.assertIs(_ast, lazy_mod) def test_subinterpreter(self): # bpo-41631: Importing and using the _ast module in a subinterpreter # must not crash. - code = dedent(""" + code = dedent(''' import _ast import ast import gc @@ -3305,54 +3474,284 @@ def test_subinterpreter(self): del ast, _ast del sys.modules['ast'], sys.modules['_ast'] gc.collect() - """) + ''') res = support.run_in_subinterp(code) self.assertEqual(res, 0) -class ASTMainTests(unittest.TestCase): - # Tests `ast.main()` function. +class CommandLineTests(unittest.TestCase): + def setUp(self): + self.filename = tempfile.mktemp() + self.addCleanup(os_helper.unlink, self.filename) - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_cli_file_input(self): - code = "print(1, 2, 3)" - expected = ast.dump(ast.parse(code), indent=3) + @staticmethod + def text_normalize(string): + return textwrap.dedent(string).strip() - with os_helper.temp_dir() as tmp_dir: - filename = os.path.join(tmp_dir, "test_module.py") - with open(filename, "w", encoding="utf-8") as f: - f.write(code) - res, _ = script_helper.run_python_until_end("-m", "ast", filename) + def set_source(self, content): + Path(self.filename).write_text(self.text_normalize(content)) - self.assertEqual(res.err, b"") - self.assertEqual(expected.splitlines(), res.out.decode("utf8").splitlines()) - self.assertEqual(res.rc, 0) + def invoke_ast(self, *flags): + stderr = StringIO() + stdout = StringIO() + with ( + contextlib.redirect_stdout(stdout), + contextlib.redirect_stderr(stderr), + ): + ast.main(args=[*flags, self.filename]) + self.assertEqual(stderr.getvalue(), '') + return stdout.getvalue().strip() + + def check_output(self, source, expect, *flags): + self.set_source(source) + res = self.invoke_ast(*flags) + expect = self.text_normalize(expect) + self.assertEqual(res, expect) + + @unittest.expectedFailure # TODO: RUSTPYTHON + @support.requires_resource('cpu') + def test_invocation(self): + # test various combinations of parameters + base_flags = ( + ('-m=exec', '--mode=exec'), + ('--no-type-comments', '--no-type-comments'), + ('-a', '--include-attributes'), + ('-i=4', '--indent=4'), + ('--feature-version=3.13', '--feature-version=3.13'), + ('-O=-1', '--optimize=-1'), + ('--show-empty', '--show-empty'), + ) + self.set_source(''' + print(1, 2, 3) + def f(x: int) -> int: + x -= 1 + return x + ''') + + for r in range(1, len(base_flags) + 1): + for choices in itertools.combinations(base_flags, r=r): + for args in itertools.product(*choices): + with self.subTest(flags=args): + self.invoke_ast(*args) + + @support.force_not_colorized + def test_help_message(self): + for flag in ('-h', '--help', '--unknown'): + with self.subTest(flag=flag): + output = StringIO() + with self.assertRaises(SystemExit): + with contextlib.redirect_stderr(output): + ast.main(args=flag) + self.assertStartsWith(output.getvalue(), 'usage: ') + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_exec_mode_flag(self): + # test 'python -m ast -m/--mode exec' + source = 'x: bool = 1 # type: ignore[assignment]' + expect = ''' + Module( + body=[ + AnnAssign( + target=Name(id='x', ctx=Store()), + annotation=Name(id='bool', ctx=Load()), + value=Constant(value=1), + simple=1)], + type_ignores=[ + TypeIgnore(lineno=1, tag='[assignment]')]) + ''' + for flag in ('-m=exec', '--mode=exec'): + with self.subTest(flag=flag): + self.check_output(source, expect, flag) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_single_mode_flag(self): + # test 'python -m ast -m/--mode single' + source = 'pass' + expect = ''' + Interactive( + body=[ + Pass()]) + ''' + for flag in ('-m=single', '--mode=single'): + with self.subTest(flag=flag): + self.check_output(source, expect, flag) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_eval_mode_flag(self): + # test 'python -m ast -m/--mode eval' + source = 'print(1, 2, 3)' + expect = ''' + Expression( + body=Call( + func=Name(id='print', ctx=Load()), + args=[ + Constant(value=1), + Constant(value=2), + Constant(value=3)])) + ''' + for flag in ('-m=eval', '--mode=eval'): + with self.subTest(flag=flag): + self.check_output(source, expect, flag) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_func_type_mode_flag(self): + # test 'python -m ast -m/--mode func_type' + source = '(int, str) -> list[int]' + expect = ''' + FunctionType( + argtypes=[ + Name(id='int', ctx=Load()), + Name(id='str', ctx=Load())], + returns=Subscript( + value=Name(id='list', ctx=Load()), + slice=Name(id='int', ctx=Load()), + ctx=Load())) + ''' + for flag in ('-m=func_type', '--mode=func_type'): + with self.subTest(flag=flag): + self.check_output(source, expect, flag) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: type object '_ast.Module' has no attribute '_field_types' + def test_no_type_comments_flag(self): + # test 'python -m ast --no-type-comments' + source = 'x: bool = 1 # type: ignore[assignment]' + expect = ''' + Module( + body=[ + AnnAssign( + target=Name(id='x', ctx=Store()), + annotation=Name(id='bool', ctx=Load()), + value=Constant(value=1), + simple=1)]) + ''' + self.check_output(source, expect, '--no-type-comments') + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_include_attributes_flag(self): + # test 'python -m ast -a/--include-attributes' + source = 'pass' + expect = ''' + Module( + body=[ + Pass( + lineno=1, + col_offset=0, + end_lineno=1, + end_col_offset=4)]) + ''' + for flag in ('-a', '--include-attributes'): + with self.subTest(flag=flag): + self.check_output(source, expect, flag) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_indent_flag(self): + # test 'python -m ast -i/--indent 0' + source = 'pass' + expect = ''' + Module( + body=[ + Pass()]) + ''' + for flag in ('-i=0', '--indent=0'): + with self.subTest(flag=flag): + self.check_output(source, expect, flag) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: type object '_ast.Module' has no attribute '_field_types' + def test_feature_version_flag(self): + # test 'python -m ast --feature-version 3.9/3.10' + source = ''' + match x: + case 1: + pass + ''' + expect = ''' + Module( + body=[ + Match( + subject=Name(id='x', ctx=Load()), + cases=[ + match_case( + pattern=MatchValue( + value=Constant(value=1)), + body=[ + Pass()])])]) + ''' + self.check_output(source, expect, '--feature-version=3.10') + with self.assertRaises(SyntaxError): + self.invoke_ast('--feature-version=3.9') + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_no_optimize_flag(self): + # test 'python -m ast -O/--optimize -1/0' + source = ''' + match a: + case 1+2j: + pass + ''' + expect = ''' + Module( + body=[ + Match( + subject=Name(id='a', ctx=Load()), + cases=[ + match_case( + pattern=MatchValue( + value=BinOp( + left=Constant(value=1), + op=Add(), + right=Constant(value=2j))), + body=[ + Pass()])])]) + ''' + for flag in ('-O=-1', '--optimize=-1', '-O=0', '--optimize=0'): + with self.subTest(flag=flag): + self.check_output(source, expect, flag) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_optimize_flag(self): + # test 'python -m ast -O/--optimize 1/2' + source = ''' + match a: + case 1+2j: + pass + ''' + expect = ''' + Module( + body=[ + Match( + subject=Name(id='a', ctx=Load()), + cases=[ + match_case( + pattern=MatchValue( + value=Constant(value=(1+2j))), + body=[ + Pass()])])]) + ''' + for flag in ('-O=1', '--optimize=1', '-O=2', '--optimize=2'): + with self.subTest(flag=flag): + self.check_output(source, expect, flag) + + @unittest.expectedFailure # TODO: RUSTPYTHON; type_ignores=[]) + def test_show_empty_flag(self): + # test 'python -m ast --show-empty' + source = 'print(1, 2, 3)' + expect = ''' + Module( + body=[ + Expr( + value=Call( + func=Name(id='print', ctx=Load()), + args=[ + Constant(value=1), + Constant(value=2), + Constant(value=3)], + keywords=[]))], + type_ignores=[]) + ''' + self.check_output(source, expect, '--show-empty') -def compare(left, right): - return ast.dump(left) == ast.dump(right) class ASTOptimizationTests(unittest.TestCase): - binop = { - "+": ast.Add(), - "-": ast.Sub(), - "*": ast.Mult(), - "/": ast.Div(), - "%": ast.Mod(), - "<<": ast.LShift(), - ">>": ast.RShift(), - "|": ast.BitOr(), - "^": ast.BitXor(), - "&": ast.BitAnd(), - "//": ast.FloorDiv(), - "**": ast.Pow(), - } - - unaryop = { - "~": ast.Invert(), - "+": ast.UAdd(), - "-": ast.USub(), - } - def wrap_expr(self, expr): return ast.Module(body=[ast.Expr(value=expr)]) @@ -3360,112 +3759,31 @@ def wrap_statement(self, statement): return ast.Module(body=[statement]) def assert_ast(self, code, non_optimized_target, optimized_target): - non_optimized_tree = ast.parse(code, optimize=-1) optimized_tree = ast.parse(code, optimize=1) # Is a non-optimized tree equal to a non-optimized target? self.assertTrue( - compare(non_optimized_tree, non_optimized_target), + ast.compare(non_optimized_tree, non_optimized_target), f"{ast.dump(non_optimized_target)} must equal " f"{ast.dump(non_optimized_tree)}", ) # Is a optimized tree equal to a non-optimized target? self.assertFalse( - compare(optimized_tree, non_optimized_target), + ast.compare(optimized_tree, non_optimized_target), f"{ast.dump(non_optimized_target)} must not equal " f"{ast.dump(non_optimized_tree)}" ) # Is a optimized tree is equal to an optimized target? self.assertTrue( - compare(optimized_tree, optimized_target), + ast.compare(optimized_tree, optimized_target), f"{ast.dump(optimized_target)} must equal " f"{ast.dump(optimized_tree)}", ) - def create_binop(self, operand, left=ast.Constant(1), right=ast.Constant(1)): - return ast.BinOp(left=left, op=self.binop[operand], right=right) - - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags - def test_folding_binop(self): - code = "1 %s 1" - operators = self.binop.keys() - - for op in operators: - result_code = code % op - non_optimized_target = self.wrap_expr(self.create_binop(op)) - optimized_target = self.wrap_expr(ast.Constant(value=eval(result_code))) - - with self.subTest( - result_code=result_code, - non_optimized_target=non_optimized_target, - optimized_target=optimized_target - ): - self.assert_ast(result_code, non_optimized_target, optimized_target) - - # Multiplication of constant tuples must be folded - code = "(1,) * 3" - non_optimized_target = self.wrap_expr(self.create_binop("*", ast.Tuple(elts=[ast.Constant(value=1)]), ast.Constant(value=3))) - optimized_target = self.wrap_expr(ast.Constant(eval(code))) - - self.assert_ast(code, non_optimized_target, optimized_target) - - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags - def test_folding_unaryop(self): - code = "%s1" - operators = self.unaryop.keys() - - def create_unaryop(operand): - return ast.UnaryOp(op=self.unaryop[operand], operand=ast.Constant(1)) - - for op in operators: - result_code = code % op - non_optimized_target = self.wrap_expr(create_unaryop(op)) - optimized_target = self.wrap_expr(ast.Constant(eval(result_code))) - - with self.subTest( - result_code=result_code, - non_optimized_target=non_optimized_target, - optimized_target=optimized_target - ): - self.assert_ast(result_code, non_optimized_target, optimized_target) - - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags - def test_folding_not(self): - code = "not (1 %s (1,))" - operators = { - "in": ast.In(), - "is": ast.Is(), - } - opt_operators = { - "is": ast.IsNot(), - "in": ast.NotIn(), - } - - def create_notop(operand): - return ast.UnaryOp(op=ast.Not(), operand=ast.Compare( - left=ast.Constant(value=1), - ops=[operators[operand]], - comparators=[ast.Tuple(elts=[ast.Constant(value=1)])] - )) - - for op in operators.keys(): - result_code = code % op - non_optimized_target = self.wrap_expr(create_notop(op)) - optimized_target = self.wrap_expr( - ast.Compare(left=ast.Constant(1), ops=[opt_operators[op]], comparators=[ast.Constant(value=(1,))]) - ) - - with self.subTest( - result_code=result_code, - non_optimized_target=non_optimized_target, - optimized_target=optimized_target - ): - self.assert_ast(result_code, non_optimized_target, optimized_target) - - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags def test_folding_format(self): code = "'%s' % (a,)" @@ -3485,158 +3803,124 @@ def test_folding_format(self): self.assert_ast(code, non_optimized_target, optimized_target) + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_folding_match_case_allowed_expressions(self): + def get_match_case_values(node): + result = [] + if isinstance(node, ast.Constant): + result.append(node.value) + elif isinstance(node, ast.MatchValue): + result.extend(get_match_case_values(node.value)) + elif isinstance(node, ast.MatchMapping): + for key in node.keys: + result.extend(get_match_case_values(key)) + elif isinstance(node, ast.MatchSequence): + for pat in node.patterns: + result.extend(get_match_case_values(pat)) + else: + self.fail(f"Unexpected node {node}") + return result - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags - def test_folding_tuple(self): - code = "(1,)" + tests = [ + ("-0", [0]), + ("-0.1", [-0.1]), + ("-0j", [complex(0, 0)]), + ("-0.1j", [complex(0, -0.1)]), + ("1 + 2j", [complex(1, 2)]), + ("1 - 2j", [complex(1, -2)]), + ("1.1 + 2.1j", [complex(1.1, 2.1)]), + ("1.1 - 2.1j", [complex(1.1, -2.1)]), + ("-0 + 1j", [complex(0, 1)]), + ("-0 - 1j", [complex(0, -1)]), + ("-0.1 + 1.1j", [complex(-0.1, 1.1)]), + ("-0.1 - 1.1j", [complex(-0.1, -1.1)]), + ("{-0: 0}", [0]), + ("{-0.1: 0}", [-0.1]), + ("{-0j: 0}", [complex(0, 0)]), + ("{-0.1j: 0}", [complex(0, -0.1)]), + ("{1 + 2j: 0}", [complex(1, 2)]), + ("{1 - 2j: 0}", [complex(1, -2)]), + ("{1.1 + 2.1j: 0}", [complex(1.1, 2.1)]), + ("{1.1 - 2.1j: 0}", [complex(1.1, -2.1)]), + ("{-0 + 1j: 0}", [complex(0, 1)]), + ("{-0 - 1j: 0}", [complex(0, -1)]), + ("{-0.1 + 1.1j: 0}", [complex(-0.1, 1.1)]), + ("{-0.1 - 1.1j: 0}", [complex(-0.1, -1.1)]), + ("{-0: 0, 0 + 1j: 0, 0.1 + 1j: 0}", [0, complex(0, 1), complex(0.1, 1)]), + ("[-0, -0.1, -0j, -0.1j]", [0, -0.1, complex(0, 0), complex(0, -0.1)]), + ("[[[[-0, -0.1, -0j, -0.1j]]]]", [0, -0.1, complex(0, 0), complex(0, -0.1)]), + ("[[-0, -0.1], -0j, -0.1j]", [0, -0.1, complex(0, 0), complex(0, -0.1)]), + ("[[-0, -0.1], [-0j, -0.1j]]", [0, -0.1, complex(0, 0), complex(0, -0.1)]), + ("(-0, -0.1, -0j, -0.1j)", [0, -0.1, complex(0, 0), complex(0, -0.1)]), + ("((((-0, -0.1, -0j, -0.1j))))", [0, -0.1, complex(0, 0), complex(0, -0.1)]), + ("((-0, -0.1), -0j, -0.1j)", [0, -0.1, complex(0, 0), complex(0, -0.1)]), + ("((-0, -0.1), (-0j, -0.1j))", [0, -0.1, complex(0, 0), complex(0, -0.1)]), + ] + for match_expr, constants in tests: + with self.subTest(match_expr): + src = f"match 0:\n\t case {match_expr}: pass" + tree = ast.parse(src, optimize=1) + match_stmt = tree.body[0] + case = match_stmt.cases[0] + values = get_match_case_values(case.pattern) + self.assertListEqual(constants, values) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: type object '_ast.Module' has no attribute '_field_types' + def test_match_case_not_folded_in_unoptimized_ast(self): + src = textwrap.dedent(""" + match a: + case 1+2j: + pass + """) - non_optimized_target = self.wrap_expr(ast.Tuple(elts=[ast.Constant(1)])) - optimized_target = self.wrap_expr(ast.Constant(value=(1,))) + unfolded = "MatchValue(value=BinOp(left=Constant(value=1), op=Add(), right=Constant(value=2j))" + folded = "MatchValue(value=Constant(value=(1+2j)))" + for optval in (0, 1, 2): + self.assertIn(folded if optval else unfolded, ast.dump(ast.parse(src, optimize=optval))) - self.assert_ast(code, non_optimized_target, optimized_target) + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags + def test_folding_binop(self): + return super().test_folding_binop() - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags def test_folding_comparator(self): - code = "1 %s %s1%s" - operators = [("in", ast.In()), ("not in", ast.NotIn())] - braces = [ - ("[", "]", ast.List, (1,)), - ("{", "}", ast.Set, frozenset({1})), - ] - for left, right, non_optimized_comparator, optimized_comparator in braces: - for op, node in operators: - non_optimized_target = self.wrap_expr(ast.Compare( - left=ast.Constant(1), ops=[node], - comparators=[non_optimized_comparator(elts=[ast.Constant(1)])] - )) - optimized_target = self.wrap_expr(ast.Compare( - left=ast.Constant(1), ops=[node], - comparators=[ast.Constant(value=optimized_comparator)] - )) - self.assert_ast(code % (op, left, right), non_optimized_target, optimized_target) - - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags - def test_folding_iter(self): - code = "for _ in %s1%s: pass" - braces = [ - ("[", "]", ast.List, (1,)), - ("{", "}", ast.Set, frozenset({1})), - ] - - for left, right, ast_cls, optimized_iter in braces: - non_optimized_target = self.wrap_statement(ast.For( - target=ast.Name(id="_", ctx=ast.Store()), - iter=ast_cls(elts=[ast.Constant(1)]), - body=[ast.Pass()] - )) - optimized_target = self.wrap_statement(ast.For( - target=ast.Name(id="_", ctx=ast.Store()), - iter=ast.Constant(value=optimized_iter), - body=[ast.Pass()] - )) - - self.assert_ast(code % (left, right), non_optimized_target, optimized_target) - - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags - def test_folding_subscript(self): - code = "(1,)[0]" - - non_optimized_target = self.wrap_expr( - ast.Subscript(value=ast.Tuple(elts=[ast.Constant(value=1)]), slice=ast.Constant(value=0)) - ) - optimized_target = self.wrap_expr(ast.Constant(value=1)) + return super().test_folding_comparator() - self.assert_ast(code, non_optimized_target, optimized_target) + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags + def test_folding_iter(self): + return super().test_folding_iter() - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags - def test_folding_type_param_in_function_def(self): - code = "def foo[%s = 1 + 1](): pass" + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags + def test_folding_not(self): + return super().test_folding_not() - unoptimized_binop = self.create_binop("+") - unoptimized_type_params = [ - ("T", "T", ast.TypeVar), - ("**P", "P", ast.ParamSpec), - ("*Ts", "Ts", ast.TypeVarTuple), - ] + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags + def test_folding_subscript(self): + return super().test_folding_subscript() - for type, name, type_param in unoptimized_type_params: - result_code = code % type - optimized_target = self.wrap_statement( - ast.FunctionDef( - name='foo', - args=ast.arguments(), - body=[ast.Pass()], - type_params=[type_param(name=name, default_value=ast.Constant(2))] - ) - ) - non_optimized_target = self.wrap_statement( - ast.FunctionDef( - name='foo', - args=ast.arguments(), - body=[ast.Pass()], - type_params=[type_param(name=name, default_value=unoptimized_binop)] - ) - ) - self.assert_ast(result_code, non_optimized_target, optimized_target) + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags + def test_folding_tuple(self): + return super().test_folding_tuple() - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags def test_folding_type_param_in_class_def(self): - code = "class foo[%s = 1 + 1]: pass" - - unoptimized_binop = self.create_binop("+") - unoptimized_type_params = [ - ("T", "T", ast.TypeVar), - ("**P", "P", ast.ParamSpec), - ("*Ts", "Ts", ast.TypeVarTuple), - ] + return super().test_folding_type_param_in_class_def() - for type, name, type_param in unoptimized_type_params: - result_code = code % type - optimized_target = self.wrap_statement( - ast.ClassDef( - name='foo', - body=[ast.Pass()], - type_params=[type_param(name=name, default_value=ast.Constant(2))] - ) - ) - non_optimized_target = self.wrap_statement( - ast.ClassDef( - name='foo', - body=[ast.Pass()], - type_params=[type_param(name=name, default_value=unoptimized_binop)] - ) - ) - self.assert_ast(result_code, non_optimized_target, optimized_target) + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags + def test_folding_type_param_in_function_def(self): + return super().test_folding_type_param_in_function_def() - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags def test_folding_type_param_in_type_alias(self): - code = "type foo[%s = 1 + 1] = 1" - - unoptimized_binop = self.create_binop("+") - unoptimized_type_params = [ - ("T", "T", ast.TypeVar), - ("**P", "P", ast.ParamSpec), - ("*Ts", "Ts", ast.TypeVarTuple), - ] + return super().test_folding_type_param_in_type_alias() - for type, name, type_param in unoptimized_type_params: - result_code = code % type - optimized_target = self.wrap_statement( - ast.TypeAlias( - name=ast.Name(id='foo', ctx=ast.Store()), - type_params=[type_param(name=name, default_value=ast.Constant(2))], - value=ast.Constant(value=1), - ) - ) - non_optimized_target = self.wrap_statement( - ast.TypeAlias( - name=ast.Name(id='foo', ctx=ast.Store()), - type_params=[type_param(name=name, default_value=unoptimized_binop)], - value=ast.Constant(value=1), - ) - ) - self.assert_ast(result_code, non_optimized_target, optimized_target) + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags + def test_folding_unaryop(self): + return super().test_folding_unaryop() -if __name__ == "__main__": +if __name__ == '__main__': + if len(sys.argv) > 1 and sys.argv[1] == '--snapshot-update': + ast_repr_update_snapshots() + sys.exit(0) unittest.main() diff --git a/Lib/test/test_ast/utils.py b/Lib/test/test_ast/utils.py index 145e89ee94e..e7054f3f710 100644 --- a/Lib/test/test_ast/utils.py +++ b/Lib/test/test_ast/utils.py @@ -1,5 +1,5 @@ def to_tuple(t): - if t is None or isinstance(t, (str, int, complex, float, bytes)) or t is Ellipsis: + if t is None or isinstance(t, (str, int, complex, float, bytes, tuple)) or t is Ellipsis: return t elif isinstance(t, list): return [to_tuple(e) for e in t] diff --git a/Lib/test/test_asyncgen.py b/Lib/test/test_asyncgen.py index 45d220e3e02..181476e0989 100644 --- a/Lib/test/test_asyncgen.py +++ b/Lib/test/test_asyncgen.py @@ -4,10 +4,12 @@ import contextlib from test.support.import_helper import import_module -from test.support import gc_collect +from test.support import gc_collect, requires_working_socket asyncio = import_module("asyncio") +requires_working_socket(module=True) + _no_default = object() @@ -375,6 +377,178 @@ async def async_gen_wrapper(): self.compare_generators(sync_gen_wrapper(), async_gen_wrapper()) + def test_async_gen_exception_12(self): + async def gen(): + with self.assertWarnsRegex(RuntimeWarning, + f"coroutine method 'asend' of '{gen.__qualname__}' " + f"was never awaited"): + await anext(me) + yield 123 + + me = gen() + ai = me.__aiter__() + an = ai.__anext__() + + with self.assertRaisesRegex(RuntimeError, + r'anext\(\): asynchronous generator is already running'): + an.__next__() + + with self.assertRaisesRegex(RuntimeError, + r"cannot reuse already awaited __anext__\(\)/asend\(\)"): + an.send(None) + + def test_async_gen_asend_throw_concurrent_with_send(self): + import types + + @types.coroutine + def _async_yield(v): + return (yield v) + + class MyExc(Exception): + pass + + async def agenfn(): + while True: + try: + await _async_yield(None) + except MyExc: + pass + return + yield + + + agen = agenfn() + gen = agen.asend(None) + gen.send(None) + gen2 = agen.asend(None) + + with self.assertRaisesRegex(RuntimeError, + r'anext\(\): asynchronous generator is already running'): + gen2.throw(MyExc) + + with self.assertRaisesRegex(RuntimeError, + r"cannot reuse already awaited __anext__\(\)/asend\(\)"): + gen2.send(None) + + def test_async_gen_athrow_throw_concurrent_with_send(self): + import types + + @types.coroutine + def _async_yield(v): + return (yield v) + + class MyExc(Exception): + pass + + async def agenfn(): + while True: + try: + await _async_yield(None) + except MyExc: + pass + return + yield + + + agen = agenfn() + gen = agen.asend(None) + gen.send(None) + gen2 = agen.athrow(MyExc) + + with self.assertRaisesRegex(RuntimeError, + r'athrow\(\): asynchronous generator is already running'): + gen2.throw(MyExc) + + with self.assertRaisesRegex(RuntimeError, + r"cannot reuse already awaited aclose\(\)/athrow\(\)"): + gen2.send(None) + + def test_async_gen_asend_throw_concurrent_with_throw(self): + import types + + @types.coroutine + def _async_yield(v): + return (yield v) + + class MyExc(Exception): + pass + + async def agenfn(): + try: + yield + except MyExc: + pass + while True: + try: + await _async_yield(None) + except MyExc: + pass + + + agen = agenfn() + with self.assertRaises(StopIteration): + agen.asend(None).send(None) + + gen = agen.athrow(MyExc) + gen.throw(MyExc) + gen2 = agen.asend(MyExc) + + with self.assertRaisesRegex(RuntimeError, + r'anext\(\): asynchronous generator is already running'): + gen2.throw(MyExc) + + with self.assertRaisesRegex(RuntimeError, + r"cannot reuse already awaited __anext__\(\)/asend\(\)"): + gen2.send(None) + + def test_async_gen_athrow_throw_concurrent_with_throw(self): + import types + + @types.coroutine + def _async_yield(v): + return (yield v) + + class MyExc(Exception): + pass + + async def agenfn(): + try: + yield + except MyExc: + pass + while True: + try: + await _async_yield(None) + except MyExc: + pass + + agen = agenfn() + with self.assertRaises(StopIteration): + agen.asend(None).send(None) + + gen = agen.athrow(MyExc) + gen.throw(MyExc) + gen2 = agen.athrow(None) + + with self.assertRaisesRegex(RuntimeError, + r'athrow\(\): asynchronous generator is already running'): + gen2.throw(MyExc) + + with self.assertRaisesRegex(RuntimeError, + r"cannot reuse already awaited aclose\(\)/athrow\(\)"): + gen2.send(None) + + def test_async_gen_3_arg_deprecation_warning(self): + async def gen(): + yield 123 + + with self.assertWarns(DeprecationWarning): + x = gen().athrow(GeneratorExit, GeneratorExit(), None) + with self.assertRaises(GeneratorExit): + x.send(None) + del x + gc_collect() + def test_async_gen_api_01(self): async def gen(): yield 123 @@ -393,8 +567,57 @@ async def gen(): self.assertIsInstance(g.ag_frame, types.FrameType) self.assertFalse(g.ag_running) self.assertIsInstance(g.ag_code, types.CodeType) + aclose = g.aclose() + self.assertTrue(inspect.isawaitable(aclose)) + aclose.close() + + def test_async_gen_asend_close_runtime_error(self): + import types + + @types.coroutine + def _async_yield(v): + return (yield v) - self.assertTrue(inspect.isawaitable(g.aclose())) + async def agenfn(): + try: + await _async_yield(None) + except GeneratorExit: + await _async_yield(None) + return + yield + + agen = agenfn() + gen = agen.asend(None) + gen.send(None) + with self.assertRaisesRegex(RuntimeError, "coroutine ignored GeneratorExit"): + gen.close() + + def test_async_gen_athrow_close_runtime_error(self): + import types + + @types.coroutine + def _async_yield(v): + return (yield v) + + class MyExc(Exception): + pass + + async def agenfn(): + try: + yield + except MyExc: + try: + await _async_yield(None) + except GeneratorExit: + await _async_yield(None) + + agen = agenfn() + with self.assertRaises(StopIteration): + agen.asend(None).send(None) + gen = agen.athrow(MyExc) + gen.send(None) + with self.assertRaisesRegex(RuntimeError, "coroutine ignored GeneratorExit"): + gen.close() class AsyncGenAsyncioTest(unittest.TestCase): @@ -406,7 +629,7 @@ def setUp(self): def tearDown(self): self.loop.close() self.loop = None - asyncio.set_event_loop_policy(None) + asyncio.events._set_event_loop_policy(None) def check_async_iterator_anext(self, ait_class): with self.subTest(anext="pure-Python"): @@ -648,7 +871,7 @@ def test1(anext): agen = agenfn() with contextlib.closing(anext(agen, "default").__await__()) as g: self.assertEqual(g.send(None), 1) - self.assertEqual(g.throw(MyError, MyError(), None), 2) + self.assertEqual(g.throw(MyError()), 2) try: g.send(None) except StopIteration as e: @@ -661,9 +884,9 @@ def test2(anext): agen = agenfn() with contextlib.closing(anext(agen, "default").__await__()) as g: self.assertEqual(g.send(None), 1) - self.assertEqual(g.throw(MyError, MyError(), None), 2) + self.assertEqual(g.throw(MyError()), 2) with self.assertRaises(MyError): - g.throw(MyError, MyError(), None) + g.throw(MyError()) def test3(anext): agen = agenfn() @@ -690,9 +913,9 @@ async def agenfn(): agen = agenfn() with contextlib.closing(anext(agen, "default").__await__()) as g: self.assertEqual(g.send(None), 10) - self.assertEqual(g.throw(MyError, MyError(), None), 20) + self.assertEqual(g.throw(MyError()), 20) with self.assertRaisesRegex(MyError, 'val'): - g.throw(MyError, MyError('val'), None) + g.throw(MyError('val')) def test5(anext): @types.coroutine @@ -711,7 +934,7 @@ async def agenfn(): with contextlib.closing(anext(agen, "default").__await__()) as g: self.assertEqual(g.send(None), 10) with self.assertRaisesRegex(StopIteration, 'default'): - g.throw(MyError, MyError(), None) + g.throw(MyError()) def test6(anext): @types.coroutine @@ -726,7 +949,7 @@ async def agenfn(): agen = agenfn() with contextlib.closing(anext(agen, "default").__await__()) as g: with self.assertRaises(MyError): - g.throw(MyError, MyError(), None) + g.throw(MyError()) def run_test(test): with self.subTest('pure-Python anext()'): @@ -929,6 +1152,43 @@ async def run(): self.loop.run_until_complete(run()) + def test_async_gen_asyncio_anext_tuple_no_exceptions(self): + # StopAsyncIteration exceptions should be cleared. + # See: https://github.com/python/cpython/issues/128078. + + async def foo(): + if False: + yield (1, 2) + + async def run(): + it = foo().__aiter__() + with self.assertRaises(StopAsyncIteration): + await it.__anext__() + res = await anext(it, ('a', 'b')) + self.assertTupleEqual(res, ('a', 'b')) + + self.loop.run_until_complete(run()) + + def test_sync_anext_raises_exception(self): + # See: https://github.com/python/cpython/issues/131670 + msg = 'custom' + for exc_type in [ + StopAsyncIteration, + StopIteration, + ValueError, + Exception, + ]: + exc = exc_type(msg) + with self.subTest(exc=exc): + class A: + def __anext__(self): + raise exc + + with self.assertRaisesRegex(exc_type, msg): + anext(A()) + with self.assertRaisesRegex(exc_type, msg): + anext(A(), 1) + def test_async_gen_asyncio_anext_stopiteration(self): async def foo(): try: @@ -1035,8 +1295,7 @@ async def gen(): while True: yield 1 finally: - await asyncio.sleep(0.01) - await asyncio.sleep(0.01) + await asyncio.sleep(0) DONE = 1 async def run(): @@ -1046,7 +1305,10 @@ async def run(): del g gc_collect() # For PyPy or other GCs. - await asyncio.sleep(0.1) + # Starts running the aclose task + await asyncio.sleep(0) + # For asyncio.sleep(0) in finally block + await asyncio.sleep(0) self.loop.run_until_complete(run()) self.assertEqual(DONE, 1) @@ -1539,6 +1801,8 @@ async def main(): self.assertIsInstance(message['exception'], ZeroDivisionError) self.assertIn('unhandled exception during asyncio.run() shutdown', message['message']) + del message, messages + gc_collect() def test_async_gen_expression_01(self): async def arange(n): @@ -1556,21 +1820,35 @@ async def run(): res = self.loop.run_until_complete(run()) self.assertEqual(res, [i * 2 for i in range(10)]) - # TODO: RUSTPYTHON: async for gen expression compilation - # def test_async_gen_expression_02(self): - # async def wrap(n): - # await asyncio.sleep(0.01) - # return n + def test_async_gen_expression_02(self): + async def wrap(n): + await asyncio.sleep(0.01) + return n - # def make_arange(n): - # # This syntax is legal starting with Python 3.7 - # return (i * 2 for i in range(n) if await wrap(i)) + def make_arange(n): + # This syntax is legal starting with Python 3.7 + return (i * 2 for i in range(n) if await wrap(i)) - # async def run(): - # return [i async for i in make_arange(10)] + async def run(): + return [i async for i in make_arange(10)] + + res = self.loop.run_until_complete(run()) + self.assertEqual(res, [i * 2 for i in range(1, 10)]) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: __aiter__ + def test_async_gen_expression_incorrect(self): + async def ag(): + yield 42 - # res = self.loop.run_until_complete(run()) - # self.assertEqual(res, [i * 2 for i in range(1, 10)]) + async def run(arg): + (x async for x in arg) + + err_msg_async = "'async for' requires an object with " \ + "__aiter__ method, got .*" + + self.loop.run_until_complete(run(ag())) + with self.assertRaisesRegex(TypeError, err_msg_async): + self.loop.run_until_complete(run(None)) def test_asyncgen_nonstarted_hooks_are_cancellable(self): # See https://bugs.python.org/issue38013 @@ -1593,6 +1871,7 @@ async def main(): asyncio.run(main()) self.assertEqual([], messages) + gc_collect() def test_async_gen_await_same_anext_coro_twice(self): async def async_iterate(): @@ -1630,6 +1909,62 @@ async def run(): self.loop.run_until_complete(run()) + def test_async_gen_throw_same_aclose_coro_twice(self): + async def async_iterate(): + yield 1 + yield 2 + + it = async_iterate() + nxt = it.aclose() + with self.assertRaises(StopIteration): + nxt.throw(GeneratorExit) + + with self.assertRaisesRegex( + RuntimeError, + r"cannot reuse already awaited aclose\(\)/athrow\(\)" + ): + nxt.throw(GeneratorExit) + + def test_async_gen_throw_custom_same_aclose_coro_twice(self): + async def async_iterate(): + yield 1 + yield 2 + + it = async_iterate() + + class MyException(Exception): + pass + + nxt = it.aclose() + with self.assertRaises(MyException): + nxt.throw(MyException) + + with self.assertRaisesRegex( + RuntimeError, + r"cannot reuse already awaited aclose\(\)/athrow\(\)" + ): + nxt.throw(MyException) + + def test_async_gen_throw_custom_same_athrow_coro_twice(self): + async def async_iterate(): + yield 1 + yield 2 + + it = async_iterate() + + class MyException(Exception): + pass + + nxt = it.athrow(MyException) + with self.assertRaises(MyException): + nxt.throw(MyException) + + with self.assertRaisesRegex( + RuntimeError, + r"cannot reuse already awaited aclose\(\)/athrow\(\)" + ): + nxt.throw(MyException) + def test_async_gen_aclose_twice_with_different_coros(self): # Regression test for https://bugs.python.org/issue39606 async def async_iterate(): @@ -1672,5 +2007,109 @@ async def run(): self.loop.run_until_complete(run()) +class TestUnawaitedWarnings(unittest.TestCase): + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: RuntimeWarning not triggered + def test_asend(self): + async def gen(): + yield 1 + + # gh-113753: asend objects allocated from a free-list should warn. + # Ensure there is a finalized 'asend' object ready to be reused. + try: + g = gen() + g.asend(None).send(None) + except StopIteration: + pass + + msg = f"coroutine method 'asend' of '{gen.__qualname__}' was never awaited" + with self.assertWarnsRegex(RuntimeWarning, msg): + g = gen() + g.asend(None) + gc_collect() + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: RuntimeWarning not triggered + def test_athrow(self): + async def gen(): + yield 1 + + msg = f"coroutine method 'athrow' of '{gen.__qualname__}' was never awaited" + with self.assertWarnsRegex(RuntimeWarning, msg): + g = gen() + g.athrow(RuntimeError) + gc_collect() + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: RuntimeWarning not triggered + def test_aclose(self): + async def gen(): + yield 1 + + msg = f"coroutine method 'aclose' of '{gen.__qualname__}' was never awaited" + with self.assertWarnsRegex(RuntimeWarning, msg): + g = gen() + g.aclose() + gc_collect() + + def test_aclose_throw(self): + async def gen(): + return + yield + + class MyException(Exception): + pass + + g = gen() + with self.assertRaises(MyException): + g.aclose().throw(MyException) + + del g + gc_collect() # does not warn unawaited + + def test_asend_send_already_running(self): + @types.coroutine + def _async_yield(v): + return (yield v) + + async def agenfn(): + while True: + await _async_yield(1) + return + yield + + agen = agenfn() + gen = agen.asend(None) + gen.send(None) + gen2 = agen.asend(None) + + with self.assertRaisesRegex(RuntimeError, + r'anext\(\): asynchronous generator is already running'): + gen2.send(None) + + del gen2 + gc_collect() # does not warn unawaited + + + def test_athrow_send_already_running(self): + @types.coroutine + def _async_yield(v): + return (yield v) + + async def agenfn(): + while True: + await _async_yield(1) + return + yield + + agen = agenfn() + gen = agen.asend(None) + gen.send(None) + gen2 = agen.athrow(Exception) + + with self.assertRaisesRegex(RuntimeError, + r'athrow\(\): asynchronous generator is already running'): + gen2.send(None) + + del gen2 + gc_collect() # does not warn unawaited + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_asyncio/__init__.py b/Lib/test/test_asyncio/__init__.py new file mode 100644 index 00000000000..ab0b5aa9489 --- /dev/null +++ b/Lib/test/test_asyncio/__init__.py @@ -0,0 +1,12 @@ +import os +from test import support +from test.support import load_package_tests +from test.support import import_helper + +support.requires_working_socket(module=True) + +# Skip tests if we don't have concurrent.futures. +import_helper.import_module('concurrent.futures') + +def load_tests(*args): + return load_package_tests(os.path.dirname(__file__), *args) diff --git a/Lib/test/test_asyncio/__main__.py b/Lib/test/test_asyncio/__main__.py new file mode 100644 index 00000000000..40a23a297ec --- /dev/null +++ b/Lib/test/test_asyncio/__main__.py @@ -0,0 +1,4 @@ +from . import load_tests +import unittest + +unittest.main() diff --git a/Lib/test/test_asyncio/echo.py b/Lib/test/test_asyncio/echo.py new file mode 100644 index 00000000000..006364bb007 --- /dev/null +++ b/Lib/test/test_asyncio/echo.py @@ -0,0 +1,8 @@ +import os + +if __name__ == '__main__': + while True: + buf = os.read(0, 1024) + if not buf: + break + os.write(1, buf) diff --git a/Lib/test/test_asyncio/echo2.py b/Lib/test/test_asyncio/echo2.py new file mode 100644 index 00000000000..e83ca09fb7a --- /dev/null +++ b/Lib/test/test_asyncio/echo2.py @@ -0,0 +1,6 @@ +import os + +if __name__ == '__main__': + buf = os.read(0, 1024) + os.write(1, b'OUT:'+buf) + os.write(2, b'ERR:'+buf) diff --git a/Lib/test/test_asyncio/echo3.py b/Lib/test/test_asyncio/echo3.py new file mode 100644 index 00000000000..064496736bf --- /dev/null +++ b/Lib/test/test_asyncio/echo3.py @@ -0,0 +1,11 @@ +import os + +if __name__ == '__main__': + while True: + buf = os.read(0, 1024) + if not buf: + break + try: + os.write(1, b'OUT:'+buf) + except OSError as ex: + os.write(2, b'ERR:' + ex.__class__.__name__.encode('ascii')) diff --git a/Lib/test/test_asyncio/functional.py b/Lib/test/test_asyncio/functional.py new file mode 100644 index 00000000000..96dc9ab4401 --- /dev/null +++ b/Lib/test/test_asyncio/functional.py @@ -0,0 +1,268 @@ +import asyncio +import asyncio.events +import contextlib +import os +import pprint +import select +import socket +import tempfile +import threading +from test import support + + +class FunctionalTestCaseMixin: + + def new_loop(self): + return asyncio.new_event_loop() + + def run_loop_briefly(self, *, delay=0.01): + self.loop.run_until_complete(asyncio.sleep(delay)) + + def loop_exception_handler(self, loop, context): + self.__unhandled_exceptions.append(context) + self.loop.default_exception_handler(context) + + def setUp(self): + self.loop = self.new_loop() + asyncio.set_event_loop(None) + + self.loop.set_exception_handler(self.loop_exception_handler) + self.__unhandled_exceptions = [] + + def tearDown(self): + try: + self.loop.close() + + if self.__unhandled_exceptions: + print('Unexpected calls to loop.call_exception_handler():') + pprint.pprint(self.__unhandled_exceptions) + self.fail('unexpected calls to loop.call_exception_handler()') + + finally: + asyncio.set_event_loop(None) + self.loop = None + + def tcp_server(self, server_prog, *, + family=socket.AF_INET, + addr=None, + timeout=support.LOOPBACK_TIMEOUT, + backlog=1, + max_clients=10): + + if addr is None: + if hasattr(socket, 'AF_UNIX') and family == socket.AF_UNIX: + with tempfile.NamedTemporaryFile() as tmp: + addr = tmp.name + else: + addr = ('127.0.0.1', 0) + + sock = socket.create_server(addr, family=family, backlog=backlog) + if timeout is None: + raise RuntimeError('timeout is required') + if timeout <= 0: + raise RuntimeError('only blocking sockets are supported') + sock.settimeout(timeout) + + return TestThreadedServer( + self, sock, server_prog, timeout, max_clients) + + def tcp_client(self, client_prog, + family=socket.AF_INET, + timeout=support.LOOPBACK_TIMEOUT): + + sock = socket.socket(family, socket.SOCK_STREAM) + + if timeout is None: + raise RuntimeError('timeout is required') + if timeout <= 0: + raise RuntimeError('only blocking sockets are supported') + sock.settimeout(timeout) + + return TestThreadedClient( + self, sock, client_prog, timeout) + + def unix_server(self, *args, **kwargs): + if not hasattr(socket, 'AF_UNIX'): + raise NotImplementedError + return self.tcp_server(*args, family=socket.AF_UNIX, **kwargs) + + def unix_client(self, *args, **kwargs): + if not hasattr(socket, 'AF_UNIX'): + raise NotImplementedError + return self.tcp_client(*args, family=socket.AF_UNIX, **kwargs) + + @contextlib.contextmanager + def unix_sock_name(self): + with tempfile.TemporaryDirectory() as td: + fn = os.path.join(td, 'sock') + try: + yield fn + finally: + try: + os.unlink(fn) + except OSError: + pass + + def _abort_socket_test(self, ex): + try: + self.loop.stop() + finally: + self.fail(ex) + + +############################################################################## +# Socket Testing Utilities +############################################################################## + + +class TestSocketWrapper: + + def __init__(self, sock): + self.__sock = sock + + def recv_all(self, n): + buf = b'' + while len(buf) < n: + data = self.recv(n - len(buf)) + if data == b'': + raise ConnectionAbortedError + buf += data + return buf + + def start_tls(self, ssl_context, *, + server_side=False, + server_hostname=None): + + ssl_sock = ssl_context.wrap_socket( + self.__sock, server_side=server_side, + server_hostname=server_hostname, + do_handshake_on_connect=False) + + try: + ssl_sock.do_handshake() + except: + ssl_sock.close() + raise + finally: + self.__sock.close() + + self.__sock = ssl_sock + + def __getattr__(self, name): + return getattr(self.__sock, name) + + def __repr__(self): + return '<{} {!r}>'.format(type(self).__name__, self.__sock) + + +class SocketThread(threading.Thread): + + def stop(self): + self._active = False + self.join() + + def __enter__(self): + self.start() + return self + + def __exit__(self, *exc): + self.stop() + + +class TestThreadedClient(SocketThread): + + def __init__(self, test, sock, prog, timeout): + threading.Thread.__init__(self, None, None, 'test-client') + self.daemon = True + + self._timeout = timeout + self._sock = sock + self._active = True + self._prog = prog + self._test = test + + def run(self): + try: + self._prog(TestSocketWrapper(self._sock)) + except Exception as ex: + self._test._abort_socket_test(ex) + + +class TestThreadedServer(SocketThread): + + def __init__(self, test, sock, prog, timeout, max_clients): + threading.Thread.__init__(self, None, None, 'test-server') + self.daemon = True + + self._clients = 0 + self._finished_clients = 0 + self._max_clients = max_clients + self._timeout = timeout + self._sock = sock + self._active = True + + self._prog = prog + + self._s1, self._s2 = socket.socketpair() + self._s1.setblocking(False) + + self._test = test + + def stop(self): + try: + if self._s2 and self._s2.fileno() != -1: + try: + self._s2.send(b'stop') + except OSError: + pass + finally: + super().stop() + self._sock.close() + self._s1.close() + self._s2.close() + + + def run(self): + self._sock.setblocking(False) + self._run() + + def _run(self): + while self._active: + if self._clients >= self._max_clients: + return + + r, w, x = select.select( + [self._sock, self._s1], [], [], self._timeout) + + if self._s1 in r: + return + + if self._sock in r: + try: + conn, addr = self._sock.accept() + except BlockingIOError: + continue + except TimeoutError: + if not self._active: + return + else: + raise + else: + self._clients += 1 + conn.settimeout(self._timeout) + try: + with conn: + self._handle_client(conn) + except Exception as ex: + self._active = False + try: + raise + finally: + self._test._abort_socket_test(ex) + + def _handle_client(self, sock): + self._prog(TestSocketWrapper(sock)) + + @property + def addr(self): + return self._sock.getsockname() diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py new file mode 100644 index 00000000000..92895bbb420 --- /dev/null +++ b/Lib/test/test_asyncio/test_base_events.py @@ -0,0 +1,2297 @@ +"""Tests for base_events.py""" + +import concurrent.futures +import errno +import math +import platform +import socket +import sys +import threading +import time +import unittest +from unittest import mock + +import asyncio +from asyncio import base_events +from asyncio import constants +from test.test_asyncio import utils as test_utils +from test import support +from test.support.script_helper import assert_python_ok +from test.support import os_helper +from test.support import socket_helper +import warnings + +MOCK_ANY = mock.ANY + + +class CustomError(Exception): + pass + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +def mock_socket_module(): + m_socket = mock.MagicMock(spec=socket) + for name in ( + 'AF_INET', 'AF_INET6', 'AF_UNSPEC', 'IPPROTO_TCP', 'IPPROTO_UDP', + 'SOCK_STREAM', 'SOCK_DGRAM', 'SOL_SOCKET', 'SO_REUSEADDR', 'inet_pton' + ): + if hasattr(socket, name): + setattr(m_socket, name, getattr(socket, name)) + else: + delattr(m_socket, name) + + m_socket.socket = mock.MagicMock() + m_socket.socket.return_value = test_utils.mock_nonblocking_socket() + + return m_socket + + +def patch_socket(f): + return mock.patch('asyncio.base_events.socket', + new_callable=mock_socket_module)(f) + + +class BaseEventTests(test_utils.TestCase): + + def test_ipaddr_info(self): + UNSPEC = socket.AF_UNSPEC + INET = socket.AF_INET + INET6 = socket.AF_INET6 + STREAM = socket.SOCK_STREAM + DGRAM = socket.SOCK_DGRAM + TCP = socket.IPPROTO_TCP + UDP = socket.IPPROTO_UDP + + self.assertEqual( + (INET, STREAM, TCP, '', ('1.2.3.4', 1)), + base_events._ipaddr_info('1.2.3.4', 1, INET, STREAM, TCP)) + + self.assertEqual( + (INET, STREAM, TCP, '', ('1.2.3.4', 1)), + base_events._ipaddr_info(b'1.2.3.4', 1, INET, STREAM, TCP)) + + self.assertEqual( + (INET, STREAM, TCP, '', ('1.2.3.4', 1)), + base_events._ipaddr_info('1.2.3.4', 1, UNSPEC, STREAM, TCP)) + + self.assertEqual( + (INET, DGRAM, UDP, '', ('1.2.3.4', 1)), + base_events._ipaddr_info('1.2.3.4', 1, UNSPEC, DGRAM, UDP)) + + # Socket type STREAM implies TCP protocol. + self.assertEqual( + (INET, STREAM, TCP, '', ('1.2.3.4', 1)), + base_events._ipaddr_info('1.2.3.4', 1, UNSPEC, STREAM, 0)) + + # Socket type DGRAM implies UDP protocol. + self.assertEqual( + (INET, DGRAM, UDP, '', ('1.2.3.4', 1)), + base_events._ipaddr_info('1.2.3.4', 1, UNSPEC, DGRAM, 0)) + + # No socket type. + self.assertIsNone( + base_events._ipaddr_info('1.2.3.4', 1, UNSPEC, 0, 0)) + + if socket_helper.IPV6_ENABLED: + # IPv4 address with family IPv6. + self.assertIsNone( + base_events._ipaddr_info('1.2.3.4', 1, INET6, STREAM, TCP)) + + self.assertEqual( + (INET6, STREAM, TCP, '', ('::3', 1, 0, 0)), + base_events._ipaddr_info('::3', 1, INET6, STREAM, TCP)) + + self.assertEqual( + (INET6, STREAM, TCP, '', ('::3', 1, 0, 0)), + base_events._ipaddr_info('::3', 1, UNSPEC, STREAM, TCP)) + + # IPv6 address with family IPv4. + self.assertIsNone( + base_events._ipaddr_info('::3', 1, INET, STREAM, TCP)) + + # IPv6 address with zone index. + self.assertIsNone( + base_events._ipaddr_info('::3%lo0', 1, INET6, STREAM, TCP)) + + def test_port_parameter_types(self): + # Test obscure kinds of arguments for "port". + INET = socket.AF_INET + STREAM = socket.SOCK_STREAM + TCP = socket.IPPROTO_TCP + + self.assertEqual( + (INET, STREAM, TCP, '', ('1.2.3.4', 0)), + base_events._ipaddr_info('1.2.3.4', None, INET, STREAM, TCP)) + + self.assertEqual( + (INET, STREAM, TCP, '', ('1.2.3.4', 0)), + base_events._ipaddr_info('1.2.3.4', b'', INET, STREAM, TCP)) + + self.assertEqual( + (INET, STREAM, TCP, '', ('1.2.3.4', 0)), + base_events._ipaddr_info('1.2.3.4', '', INET, STREAM, TCP)) + + self.assertEqual( + (INET, STREAM, TCP, '', ('1.2.3.4', 1)), + base_events._ipaddr_info('1.2.3.4', '1', INET, STREAM, TCP)) + + self.assertEqual( + (INET, STREAM, TCP, '', ('1.2.3.4', 1)), + base_events._ipaddr_info('1.2.3.4', b'1', INET, STREAM, TCP)) + + @patch_socket + def test_ipaddr_info_no_inet_pton(self, m_socket): + del m_socket.inet_pton + self.assertIsNone(base_events._ipaddr_info('1.2.3.4', 1, + socket.AF_INET, + socket.SOCK_STREAM, + socket.IPPROTO_TCP)) + + +class BaseEventLoopTests(test_utils.TestCase): + + def setUp(self): + super().setUp() + self.loop = base_events.BaseEventLoop() + self.loop._selector = mock.Mock() + self.loop._selector.select.return_value = () + self.set_event_loop(self.loop) + + def test_not_implemented(self): + m = mock.Mock() + self.assertRaises( + NotImplementedError, + self.loop._make_socket_transport, m, m) + self.assertRaises( + NotImplementedError, + self.loop._make_ssl_transport, m, m, m, m) + self.assertRaises( + NotImplementedError, + self.loop._make_datagram_transport, m, m) + self.assertRaises( + NotImplementedError, self.loop._process_events, []) + self.assertRaises( + NotImplementedError, self.loop._write_to_self) + self.assertRaises( + NotImplementedError, + self.loop._make_read_pipe_transport, m, m) + self.assertRaises( + NotImplementedError, + self.loop._make_write_pipe_transport, m, m) + gen = self.loop._make_subprocess_transport(m, m, m, m, m, m, m) + with self.assertRaises(NotImplementedError): + gen.send(None) + + def test_close(self): + self.assertFalse(self.loop.is_closed()) + self.loop.close() + self.assertTrue(self.loop.is_closed()) + + # it should be possible to call close() more than once + self.loop.close() + self.loop.close() + + # operation blocked when the loop is closed + f = self.loop.create_future() + self.assertRaises(RuntimeError, self.loop.run_forever) + self.assertRaises(RuntimeError, self.loop.run_until_complete, f) + + def test__add_callback_handle(self): + h = asyncio.Handle(lambda: False, (), self.loop, None) + + self.loop._add_callback(h) + self.assertFalse(self.loop._scheduled) + self.assertIn(h, self.loop._ready) + + def test__add_callback_cancelled_handle(self): + h = asyncio.Handle(lambda: False, (), self.loop, None) + h.cancel() + + self.loop._add_callback(h) + self.assertFalse(self.loop._scheduled) + self.assertFalse(self.loop._ready) + + def test_set_default_executor(self): + class DummyExecutor(concurrent.futures.ThreadPoolExecutor): + def submit(self, fn, *args, **kwargs): + raise NotImplementedError( + 'cannot submit into a dummy executor') + + self.loop._process_events = mock.Mock() + self.loop._write_to_self = mock.Mock() + + executor = DummyExecutor() + self.loop.set_default_executor(executor) + self.assertIs(executor, self.loop._default_executor) + + def test_set_default_executor_error(self): + executor = mock.Mock() + + msg = 'executor must be ThreadPoolExecutor instance' + with self.assertRaisesRegex(TypeError, msg): + self.loop.set_default_executor(executor) + + self.assertIsNone(self.loop._default_executor) + + def test_shutdown_default_executor_timeout(self): + event = threading.Event() + + class DummyExecutor(concurrent.futures.ThreadPoolExecutor): + def shutdown(self, wait=True, *, cancel_futures=False): + if wait: + event.wait() + + self.loop._process_events = mock.Mock() + self.loop._write_to_self = mock.Mock() + executor = DummyExecutor() + self.loop.set_default_executor(executor) + + try: + with self.assertWarnsRegex(RuntimeWarning, + "The executor did not finishing joining"): + self.loop.run_until_complete( + self.loop.shutdown_default_executor(timeout=0.01)) + finally: + event.set() + + def test_call_soon(self): + def cb(): + pass + + h = self.loop.call_soon(cb) + self.assertEqual(h._callback, cb) + self.assertIsInstance(h, asyncio.Handle) + self.assertIn(h, self.loop._ready) + + def test_call_soon_non_callable(self): + self.loop.set_debug(True) + with self.assertRaisesRegex(TypeError, 'a callable object'): + self.loop.call_soon(1) + + def test_call_later(self): + def cb(): + pass + + h = self.loop.call_later(10.0, cb) + self.assertIsInstance(h, asyncio.TimerHandle) + self.assertIn(h, self.loop._scheduled) + self.assertNotIn(h, self.loop._ready) + with self.assertRaises(TypeError, msg="delay must not be None"): + self.loop.call_later(None, cb) + + def test_call_later_negative_delays(self): + calls = [] + + def cb(arg): + calls.append(arg) + + self.loop._process_events = mock.Mock() + self.loop.call_later(-1, cb, 'a') + self.loop.call_later(-2, cb, 'b') + test_utils.run_briefly(self.loop) + self.assertEqual(calls, ['b', 'a']) + + def test_time_and_call_at(self): + def cb(): + self.loop.stop() + + self.loop._process_events = mock.Mock() + delay = 0.100 + + when = self.loop.time() + delay + self.loop.call_at(when, cb) + t0 = self.loop.time() + self.loop.run_forever() + dt = self.loop.time() - t0 + + # 50 ms: maximum granularity of the event loop + self.assertGreaterEqual(dt, delay - test_utils.CLOCK_RES) + with self.assertRaises(TypeError, msg="when cannot be None"): + self.loop.call_at(None, cb) + + def check_thread(self, loop, debug): + def cb(): + pass + + loop.set_debug(debug) + if debug: + msg = ("Non-thread-safe operation invoked on an event loop other " + "than the current one") + with self.assertRaisesRegex(RuntimeError, msg): + loop.call_soon(cb) + with self.assertRaisesRegex(RuntimeError, msg): + loop.call_later(60, cb) + with self.assertRaisesRegex(RuntimeError, msg): + loop.call_at(loop.time() + 60, cb) + else: + loop.call_soon(cb) + loop.call_later(60, cb) + loop.call_at(loop.time() + 60, cb) + + def test_check_thread(self): + def check_in_thread(loop, event, debug, create_loop, fut): + # wait until the event loop is running + event.wait() + + try: + if create_loop: + loop2 = base_events.BaseEventLoop() + try: + asyncio.set_event_loop(loop2) + self.check_thread(loop, debug) + finally: + asyncio.set_event_loop(None) + loop2.close() + else: + self.check_thread(loop, debug) + except Exception as exc: + loop.call_soon_threadsafe(fut.set_exception, exc) + else: + loop.call_soon_threadsafe(fut.set_result, None) + + def test_thread(loop, debug, create_loop=False): + event = threading.Event() + fut = loop.create_future() + loop.call_soon(event.set) + args = (loop, event, debug, create_loop, fut) + thread = threading.Thread(target=check_in_thread, args=args) + thread.start() + loop.run_until_complete(fut) + thread.join() + + self.loop._process_events = mock.Mock() + self.loop._write_to_self = mock.Mock() + + # raise RuntimeError if the thread has no event loop + test_thread(self.loop, True) + + # check disabled if debug mode is disabled + test_thread(self.loop, False) + + # raise RuntimeError if the event loop of the thread is not the called + # event loop + test_thread(self.loop, True, create_loop=True) + + # check disabled if debug mode is disabled + test_thread(self.loop, False, create_loop=True) + + def test__run_once(self): + h1 = asyncio.TimerHandle(time.monotonic() + 5.0, lambda: True, (), + self.loop, None) + h2 = asyncio.TimerHandle(time.monotonic() + 10.0, lambda: True, (), + self.loop, None) + + h1.cancel() + + self.loop._process_events = mock.Mock() + self.loop._scheduled.append(h1) + self.loop._scheduled.append(h2) + self.loop._run_once() + + t = self.loop._selector.select.call_args[0][0] + self.assertTrue(9.5 < t < 10.5, t) + self.assertEqual([h2], self.loop._scheduled) + self.assertTrue(self.loop._process_events.called) + + def test_set_debug(self): + self.loop.set_debug(True) + self.assertTrue(self.loop.get_debug()) + self.loop.set_debug(False) + self.assertFalse(self.loop.get_debug()) + + def test__run_once_schedule_handle(self): + handle = None + processed = False + + def cb(loop): + nonlocal processed, handle + processed = True + handle = loop.call_soon(lambda: True) + + h = asyncio.TimerHandle(time.monotonic() - 1, cb, (self.loop,), + self.loop, None) + + self.loop._process_events = mock.Mock() + self.loop._scheduled.append(h) + self.loop._run_once() + + self.assertTrue(processed) + self.assertEqual([handle], list(self.loop._ready)) + + def test__run_once_cancelled_event_cleanup(self): + self.loop._process_events = mock.Mock() + + self.assertTrue( + 0 < base_events._MIN_CANCELLED_TIMER_HANDLES_FRACTION < 1.0) + + def cb(): + pass + + # Set up one "blocking" event that will not be cancelled to + # ensure later cancelled events do not make it to the head + # of the queue and get cleaned. + not_cancelled_count = 1 + self.loop.call_later(3000, cb) + + # Add less than threshold (base_events._MIN_SCHEDULED_TIMER_HANDLES) + # cancelled handles, ensure they aren't removed + + cancelled_count = 2 + for x in range(2): + h = self.loop.call_later(3600, cb) + h.cancel() + + # Add some cancelled events that will be at head and removed + cancelled_count += 2 + for x in range(2): + h = self.loop.call_later(100, cb) + h.cancel() + + # This test is invalid if _MIN_SCHEDULED_TIMER_HANDLES is too low + self.assertLessEqual(cancelled_count + not_cancelled_count, + base_events._MIN_SCHEDULED_TIMER_HANDLES) + + self.assertEqual(self.loop._timer_cancelled_count, cancelled_count) + + self.loop._run_once() + + cancelled_count -= 2 + + self.assertEqual(self.loop._timer_cancelled_count, cancelled_count) + + self.assertEqual(len(self.loop._scheduled), + cancelled_count + not_cancelled_count) + + # Need enough events to pass _MIN_CANCELLED_TIMER_HANDLES_FRACTION + # so that deletion of cancelled events will occur on next _run_once + add_cancel_count = int(math.ceil( + base_events._MIN_SCHEDULED_TIMER_HANDLES * + base_events._MIN_CANCELLED_TIMER_HANDLES_FRACTION)) + 1 + + add_not_cancel_count = max(base_events._MIN_SCHEDULED_TIMER_HANDLES - + add_cancel_count, 0) + + # Add some events that will not be cancelled + not_cancelled_count += add_not_cancel_count + for x in range(add_not_cancel_count): + self.loop.call_later(3600, cb) + + # Add enough cancelled events + cancelled_count += add_cancel_count + for x in range(add_cancel_count): + h = self.loop.call_later(3600, cb) + h.cancel() + + # Ensure all handles are still scheduled + self.assertEqual(len(self.loop._scheduled), + cancelled_count + not_cancelled_count) + + self.loop._run_once() + + # Ensure cancelled events were removed + self.assertEqual(len(self.loop._scheduled), not_cancelled_count) + + # Ensure only uncancelled events remain scheduled + self.assertTrue(all([not x._cancelled for x in self.loop._scheduled])) + + def test_run_until_complete_type_error(self): + self.assertRaises(TypeError, + self.loop.run_until_complete, 'blah') + + def test_run_until_complete_loop(self): + task = self.loop.create_future() + other_loop = self.new_test_loop() + self.addCleanup(other_loop.close) + self.assertRaises(ValueError, + other_loop.run_until_complete, task) + + def test_run_until_complete_loop_orphan_future_close_loop(self): + class ShowStopper(SystemExit): + pass + + async def foo(delay): + await asyncio.sleep(delay) + + def throw(): + raise ShowStopper + + self.loop._process_events = mock.Mock() + self.loop.call_soon(throw) + with self.assertRaises(ShowStopper): + self.loop.run_until_complete(foo(0.1)) + + # This call fails if run_until_complete does not clean up + # done-callback for the previous future. + self.loop.run_until_complete(foo(0.2)) + + def test_subprocess_exec_invalid_args(self): + args = [sys.executable, '-c', 'pass'] + + # missing program parameter (empty args) + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_exec, + asyncio.SubprocessProtocol) + + # expected multiple arguments, not a list + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_exec, + asyncio.SubprocessProtocol, args) + + # program arguments must be strings, not int + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_exec, + asyncio.SubprocessProtocol, sys.executable, 123) + + # universal_newlines, shell, bufsize must not be set + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_exec, + asyncio.SubprocessProtocol, *args, universal_newlines=True) + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_exec, + asyncio.SubprocessProtocol, *args, shell=True) + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_exec, + asyncio.SubprocessProtocol, *args, bufsize=4096) + + def test_subprocess_shell_invalid_args(self): + # expected a string, not an int or a list + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_shell, + asyncio.SubprocessProtocol, 123) + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_shell, + asyncio.SubprocessProtocol, [sys.executable, '-c', 'pass']) + + # universal_newlines, shell, bufsize must not be set + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_shell, + asyncio.SubprocessProtocol, 'exit 0', universal_newlines=True) + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_shell, + asyncio.SubprocessProtocol, 'exit 0', shell=True) + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_shell, + asyncio.SubprocessProtocol, 'exit 0', bufsize=4096) + + def test_default_exc_handler_callback(self): + self.loop._process_events = mock.Mock() + + def zero_error(fut): + fut.set_result(True) + 1/0 + + # Test call_soon (events.Handle) + with mock.patch('asyncio.base_events.logger') as log: + fut = self.loop.create_future() + self.loop.call_soon(zero_error, fut) + fut.add_done_callback(lambda fut: self.loop.stop()) + self.loop.run_forever() + log.error.assert_called_with( + test_utils.MockPattern('Exception in callback.*zero'), + exc_info=(ZeroDivisionError, MOCK_ANY, MOCK_ANY)) + + # Test call_later (events.TimerHandle) + with mock.patch('asyncio.base_events.logger') as log: + fut = self.loop.create_future() + self.loop.call_later(0.01, zero_error, fut) + fut.add_done_callback(lambda fut: self.loop.stop()) + self.loop.run_forever() + log.error.assert_called_with( + test_utils.MockPattern('Exception in callback.*zero'), + exc_info=(ZeroDivisionError, MOCK_ANY, MOCK_ANY)) + + def test_default_exc_handler_coro(self): + self.loop._process_events = mock.Mock() + + async def zero_error_coro(): + await asyncio.sleep(0.01) + 1/0 + + # Test Future.__del__ + with mock.patch('asyncio.base_events.logger') as log: + fut = asyncio.ensure_future(zero_error_coro(), loop=self.loop) + fut.add_done_callback(lambda *args: self.loop.stop()) + self.loop.run_forever() + fut = None # Trigger Future.__del__ or futures._TracebackLogger + support.gc_collect() + # Future.__del__ in logs error with an actual exception context + log.error.assert_called_with( + test_utils.MockPattern('.*exception was never retrieved'), + exc_info=(ZeroDivisionError, MOCK_ANY, MOCK_ANY)) + + def test_set_exc_handler_invalid(self): + with self.assertRaisesRegex(TypeError, 'A callable object or None'): + self.loop.set_exception_handler('spam') + + def test_set_exc_handler_custom(self): + def zero_error(): + 1/0 + + def run_loop(): + handle = self.loop.call_soon(zero_error) + self.loop._run_once() + return handle + + self.loop.set_debug(True) + self.loop._process_events = mock.Mock() + + self.assertIsNone(self.loop.get_exception_handler()) + mock_handler = mock.Mock() + self.loop.set_exception_handler(mock_handler) + self.assertIs(self.loop.get_exception_handler(), mock_handler) + handle = run_loop() + mock_handler.assert_called_with(self.loop, { + 'exception': MOCK_ANY, + 'message': test_utils.MockPattern( + 'Exception in callback.*zero_error'), + 'handle': handle, + 'source_traceback': handle._source_traceback, + }) + mock_handler.reset_mock() + + self.loop.set_exception_handler(None) + with mock.patch('asyncio.base_events.logger') as log: + run_loop() + log.error.assert_called_with( + test_utils.MockPattern( + 'Exception in callback.*zero'), + exc_info=(ZeroDivisionError, MOCK_ANY, MOCK_ANY)) + + self.assertFalse(mock_handler.called) + + def test_set_exc_handler_broken(self): + def run_loop(): + def zero_error(): + 1/0 + self.loop.call_soon(zero_error) + self.loop._run_once() + + def handler(loop, context): + raise AttributeError('spam') + + self.loop._process_events = mock.Mock() + + self.loop.set_exception_handler(handler) + + with mock.patch('asyncio.base_events.logger') as log: + run_loop() + log.error.assert_called_with( + test_utils.MockPattern( + 'Unhandled error in exception handler'), + exc_info=(AttributeError, MOCK_ANY, MOCK_ANY)) + + def test_default_exc_handler_broken(self): + _context = None + + class Loop(base_events.BaseEventLoop): + + _selector = mock.Mock() + _process_events = mock.Mock() + + def default_exception_handler(self, context): + nonlocal _context + _context = context + # Simulates custom buggy "default_exception_handler" + raise ValueError('spam') + + loop = Loop() + self.addCleanup(loop.close) + asyncio.set_event_loop(loop) + + def run_loop(): + def zero_error(): + 1/0 + loop.call_soon(zero_error) + loop._run_once() + + with mock.patch('asyncio.base_events.logger') as log: + run_loop() + log.error.assert_called_with( + 'Exception in default exception handler', + exc_info=True) + + def custom_handler(loop, context): + raise ValueError('ham') + + _context = None + loop.set_exception_handler(custom_handler) + with mock.patch('asyncio.base_events.logger') as log: + run_loop() + log.error.assert_called_with( + test_utils.MockPattern('Exception in default exception.*' + 'while handling.*in custom'), + exc_info=True) + + # Check that original context was passed to default + # exception handler. + self.assertIn('context', _context) + self.assertIs(type(_context['context']['exception']), + ZeroDivisionError) + + def test_set_task_factory_invalid(self): + with self.assertRaisesRegex( + TypeError, 'task factory must be a callable or None'): + + self.loop.set_task_factory(1) + + self.assertIsNone(self.loop.get_task_factory()) + + def test_set_task_factory(self): + self.loop._process_events = mock.Mock() + + class MyTask(asyncio.Task): + pass + + async def coro(): + pass + + factory = lambda loop, coro: MyTask(coro, loop=loop) + + self.assertIsNone(self.loop.get_task_factory()) + self.loop.set_task_factory(factory) + self.assertIs(self.loop.get_task_factory(), factory) + + task = self.loop.create_task(coro()) + self.assertTrue(isinstance(task, MyTask)) + self.loop.run_until_complete(task) + + self.loop.set_task_factory(None) + self.assertIsNone(self.loop.get_task_factory()) + + task = self.loop.create_task(coro()) + self.assertTrue(isinstance(task, asyncio.Task)) + self.assertFalse(isinstance(task, MyTask)) + self.loop.run_until_complete(task) + + def test_env_var_debug(self): + code = '\n'.join(( + 'import asyncio', + 'loop = asyncio.new_event_loop()', + 'print(loop.get_debug())')) + + # Test with -E to not fail if the unit test was run with + # PYTHONASYNCIODEBUG set to a non-empty string + sts, stdout, stderr = assert_python_ok('-E', '-c', code) + self.assertEqual(stdout.rstrip(), b'False') + + sts, stdout, stderr = assert_python_ok('-c', code, + PYTHONASYNCIODEBUG='', + PYTHONDEVMODE='') + self.assertEqual(stdout.rstrip(), b'False') + + sts, stdout, stderr = assert_python_ok('-c', code, + PYTHONASYNCIODEBUG='1', + PYTHONDEVMODE='') + self.assertEqual(stdout.rstrip(), b'True') + + sts, stdout, stderr = assert_python_ok('-E', '-c', code, + PYTHONASYNCIODEBUG='1') + self.assertEqual(stdout.rstrip(), b'False') + + # -X dev + sts, stdout, stderr = assert_python_ok('-E', '-X', 'dev', + '-c', code) + self.assertEqual(stdout.rstrip(), b'True') + + def test_create_task(self): + class MyTask(asyncio.Task): + pass + + async def test(): + pass + + class EventLoop(base_events.BaseEventLoop): + def create_task(self, coro): + return MyTask(coro, loop=loop) + + loop = EventLoop() + self.set_event_loop(loop) + + coro = test() + task = asyncio.ensure_future(coro, loop=loop) + self.assertIsInstance(task, MyTask) + + # make warnings quiet + task._log_destroy_pending = False + coro.close() + + def test_create_task_error_closes_coro(self): + async def test(): + pass + loop = asyncio.new_event_loop() + loop.close() + with warnings.catch_warnings(record=True) as w: + with self.assertRaises(RuntimeError): + asyncio.ensure_future(test(), loop=loop) + self.assertEqual(len(w), 0) + + + def test_create_named_task_with_default_factory(self): + async def test(): + pass + + loop = asyncio.new_event_loop() + task = loop.create_task(test(), name='test_task') + try: + self.assertEqual(task.get_name(), 'test_task') + finally: + loop.run_until_complete(task) + loop.close() + + def test_create_named_task_with_custom_factory(self): + def task_factory(loop, coro, **kwargs): + return asyncio.Task(coro, loop=loop, **kwargs) + + async def test(): + pass + + loop = asyncio.new_event_loop() + loop.set_task_factory(task_factory) + task = loop.create_task(test(), name='test_task') + try: + self.assertEqual(task.get_name(), 'test_task') + finally: + loop.run_until_complete(task) + loop.close() + + def test_run_forever_keyboard_interrupt(self): + # Python issue #22601: ensure that the temporary task created by + # run_forever() consumes the KeyboardInterrupt and so don't log + # a warning + async def raise_keyboard_interrupt(): + raise KeyboardInterrupt + + self.loop._process_events = mock.Mock() + self.loop.call_exception_handler = mock.Mock() + + try: + self.loop.run_until_complete(raise_keyboard_interrupt()) + except KeyboardInterrupt: + pass + self.loop.close() + support.gc_collect() + + self.assertFalse(self.loop.call_exception_handler.called) + + def test_run_until_complete_baseexception(self): + # Python issue #22429: run_until_complete() must not schedule a pending + # call to stop() if the future raised a BaseException + async def raise_keyboard_interrupt(): + raise KeyboardInterrupt + + self.loop._process_events = mock.Mock() + + with self.assertRaises(KeyboardInterrupt): + self.loop.run_until_complete(raise_keyboard_interrupt()) + + def func(): + self.loop.stop() + func.called = True + func.called = False + self.loop.call_soon(self.loop.call_soon, func) + self.loop.run_forever() + self.assertTrue(func.called) + + def test_single_selecter_event_callback_after_stopping(self): + # Python issue #25593: A stopped event loop may cause event callbacks + # to run more than once. + event_sentinel = object() + callcount = 0 + doer = None + + def proc_events(event_list): + nonlocal doer + if event_sentinel in event_list: + doer = self.loop.call_soon(do_event) + + def do_event(): + nonlocal callcount + callcount += 1 + self.loop.call_soon(clear_selector) + + def clear_selector(): + doer.cancel() + self.loop._selector.select.return_value = () + + self.loop._process_events = proc_events + self.loop._selector.select.return_value = (event_sentinel,) + + for i in range(1, 3): + with self.subTest('Loop %d/2' % i): + self.loop.call_soon(self.loop.stop) + self.loop.run_forever() + self.assertEqual(callcount, 1) + + def test_run_once(self): + # Simple test for test_utils.run_once(). It may seem strange + # to have a test for this (the function isn't even used!) but + # it's a de-factor standard API for library tests. This tests + # the idiom: loop.call_soon(loop.stop); loop.run_forever(). + count = 0 + + def callback(): + nonlocal count + count += 1 + + self.loop._process_events = mock.Mock() + self.loop.call_soon(callback) + test_utils.run_once(self.loop) + self.assertEqual(count, 1) + + def test_run_forever_pre_stopped(self): + # Test that the old idiom for pre-stopping the loop works. + self.loop._process_events = mock.Mock() + self.loop.stop() + self.loop.run_forever() + self.loop._selector.select.assert_called_once_with(0) + + def test_custom_run_forever_integration(self): + # Test that the run_forever_setup() and run_forever_cleanup() primitives + # can be used to implement a custom run_forever loop. + self.loop._process_events = mock.Mock() + + count = 0 + + def callback(): + nonlocal count + count += 1 + + self.loop.call_soon(callback) + + # Set up the custom event loop + self.loop._run_forever_setup() + + # Confirm the loop has been started + self.assertEqual(asyncio.get_running_loop(), self.loop) + self.assertTrue(self.loop.is_running()) + + # Our custom "event loop" just iterates 10 times before exiting. + for i in range(10): + self.loop._run_once() + + # Clean up the event loop + self.loop._run_forever_cleanup() + + # Confirm the loop has been cleaned up + with self.assertRaises(RuntimeError): + asyncio.get_running_loop() + self.assertFalse(self.loop.is_running()) + + # Confirm the loop actually did run, processing events 10 times, + # and invoking the callback once. + self.assertEqual(self.loop._process_events.call_count, 10) + self.assertEqual(count, 1) + + async def leave_unfinalized_asyncgen(self): + # Create an async generator, iterate it partially, and leave it + # to be garbage collected. + # Used in async generator finalization tests. + # Depends on implementation details of garbage collector. Changes + # in gc may break this function. + status = {'started': False, + 'stopped': False, + 'finalized': False} + + async def agen(): + status['started'] = True + try: + for item in ['ZERO', 'ONE', 'TWO', 'THREE', 'FOUR']: + yield item + finally: + status['finalized'] = True + + ag = agen() + ai = ag.__aiter__() + + async def iter_one(): + try: + item = await ai.__anext__() + except StopAsyncIteration: + return + if item == 'THREE': + status['stopped'] = True + return + asyncio.create_task(iter_one()) + + asyncio.create_task(iter_one()) + return status + + @unittest.expectedFailure # TODO: RUSTPYTHON; - GC doesn't finalize async generators + def test_asyncgen_finalization_by_gc(self): + # Async generators should be finalized when garbage collected. + self.loop._process_events = mock.Mock() + self.loop._write_to_self = mock.Mock() + with support.disable_gc(): + status = self.loop.run_until_complete(self.leave_unfinalized_asyncgen()) + while not status['stopped']: + test_utils.run_briefly(self.loop) + self.assertTrue(status['started']) + self.assertTrue(status['stopped']) + self.assertFalse(status['finalized']) + support.gc_collect() + test_utils.run_briefly(self.loop) + self.assertTrue(status['finalized']) + + @unittest.expectedFailure # TODO: RUSTPYTHON; - GC doesn't finalize async generators + def test_asyncgen_finalization_by_gc_in_other_thread(self): + # Python issue 34769: If garbage collector runs in another + # thread, async generators will not finalize in debug + # mode. + self.loop._process_events = mock.Mock() + self.loop._write_to_self = mock.Mock() + self.loop.set_debug(True) + with support.disable_gc(): + status = self.loop.run_until_complete(self.leave_unfinalized_asyncgen()) + while not status['stopped']: + test_utils.run_briefly(self.loop) + self.assertTrue(status['started']) + self.assertTrue(status['stopped']) + self.assertFalse(status['finalized']) + self.loop.run_until_complete( + self.loop.run_in_executor(None, support.gc_collect)) + test_utils.run_briefly(self.loop) + self.assertTrue(status['finalized']) + + +class MyProto(asyncio.Protocol): + done = None + + def __init__(self, create_future=False): + self.state = 'INITIAL' + self.nbytes = 0 + if create_future: + self.done = asyncio.get_running_loop().create_future() + + def _assert_state(self, *expected): + if self.state not in expected: + raise AssertionError(f'state: {self.state!r}, expected: {expected!r}') + + def connection_made(self, transport): + self.transport = transport + self._assert_state('INITIAL') + self.state = 'CONNECTED' + transport.write(b'GET / HTTP/1.0\r\nHost: example.com\r\n\r\n') + + def data_received(self, data): + self._assert_state('CONNECTED') + self.nbytes += len(data) + + def eof_received(self): + self._assert_state('CONNECTED') + self.state = 'EOF' + + def connection_lost(self, exc): + self._assert_state('CONNECTED', 'EOF') + self.state = 'CLOSED' + if self.done: + self.done.set_result(None) + + +class MyDatagramProto(asyncio.DatagramProtocol): + done = None + + def __init__(self, create_future=False, loop=None): + self.state = 'INITIAL' + self.nbytes = 0 + if create_future: + self.done = loop.create_future() + + def _assert_state(self, expected): + if self.state != expected: + raise AssertionError(f'state: {self.state!r}, expected: {expected!r}') + + def connection_made(self, transport): + self.transport = transport + self._assert_state('INITIAL') + self.state = 'INITIALIZED' + + def datagram_received(self, data, addr): + self._assert_state('INITIALIZED') + self.nbytes += len(data) + + def error_received(self, exc): + self._assert_state('INITIALIZED') + + def connection_lost(self, exc): + self._assert_state('INITIALIZED') + self.state = 'CLOSED' + if self.done: + self.done.set_result(None) + + +class BaseEventLoopWithSelectorTests(test_utils.TestCase): + + def setUp(self): + super().setUp() + self.loop = asyncio.SelectorEventLoop() + self.set_event_loop(self.loop) + + @mock.patch('socket.getnameinfo') + def test_getnameinfo(self, m_gai): + m_gai.side_effect = lambda *args: 42 + r = self.loop.run_until_complete(self.loop.getnameinfo(('abc', 123))) + self.assertEqual(r, 42) + + @patch_socket + def test_create_connection_multiple_errors(self, m_socket): + + class MyProto(asyncio.Protocol): + pass + + async def getaddrinfo(*args, **kw): + return [(2, 1, 6, '', ('107.6.106.82', 80)), + (2, 1, 6, '', ('107.6.106.82', 80))] + + def getaddrinfo_task(*args, **kwds): + return self.loop.create_task(getaddrinfo(*args, **kwds)) + + idx = -1 + errors = ['err1', 'err2'] + + def _socket(*args, **kw): + nonlocal idx, errors + idx += 1 + raise OSError(errors[idx]) + + m_socket.socket = _socket + + self.loop.getaddrinfo = getaddrinfo_task + + coro = self.loop.create_connection(MyProto, 'example.com', 80) + with self.assertRaises(OSError) as cm: + self.loop.run_until_complete(coro) + + self.assertEqual(str(cm.exception), 'Multiple exceptions: err1, err2') + + idx = -1 + coro = self.loop.create_connection(MyProto, 'example.com', 80, all_errors=True) + with self.assertRaises(ExceptionGroup) as cm: + self.loop.run_until_complete(coro) + + self.assertIsInstance(cm.exception, ExceptionGroup) + for e in cm.exception.exceptions: + self.assertIsInstance(e, OSError) + + @patch_socket + def test_create_connection_timeout(self, m_socket): + # Ensure that the socket is closed on timeout + sock = mock.Mock() + m_socket.socket.return_value = sock + + def getaddrinfo(*args, **kw): + fut = self.loop.create_future() + addr = (socket.AF_INET, socket.SOCK_STREAM, 0, '', + ('127.0.0.1', 80)) + fut.set_result([addr]) + return fut + self.loop.getaddrinfo = getaddrinfo + + with mock.patch.object(self.loop, 'sock_connect', + side_effect=asyncio.TimeoutError): + coro = self.loop.create_connection(MyProto, '127.0.0.1', 80) + with self.assertRaises(asyncio.TimeoutError): + self.loop.run_until_complete(coro) + self.assertTrue(sock.close.called) + + @patch_socket + def test_create_connection_happy_eyeballs_empty_exceptions(self, m_socket): + # See gh-135836: Fix IndexError when Happy Eyeballs algorithm + # results in empty exceptions list + + async def getaddrinfo(*args, **kw): + return [(socket.AF_INET, socket.SOCK_STREAM, 0, '', ('127.0.0.1', 80)), + (socket.AF_INET6, socket.SOCK_STREAM, 0, '', ('::1', 80))] + + def getaddrinfo_task(*args, **kwds): + return self.loop.create_task(getaddrinfo(*args, **kwds)) + + self.loop.getaddrinfo = getaddrinfo_task + + # Mock staggered_race to return empty exceptions list + # This simulates the scenario where Happy Eyeballs algorithm + # cancels all attempts but doesn't properly collect exceptions + with mock.patch('asyncio.staggered.staggered_race') as mock_staggered: + # Return (None, []) - no winner, empty exceptions list + async def mock_race(coro_fns, delay, loop): + return None, [] + mock_staggered.side_effect = mock_race + + coro = self.loop.create_connection( + MyProto, 'example.com', 80, happy_eyeballs_delay=0.1) + + # Should raise TimeoutError instead of IndexError + with self.assertRaisesRegex(TimeoutError, "create_connection failed"): + self.loop.run_until_complete(coro) + + def test_create_connection_host_port_sock(self): + coro = self.loop.create_connection( + MyProto, 'example.com', 80, sock=object()) + self.assertRaises(ValueError, self.loop.run_until_complete, coro) + + def test_create_connection_wrong_sock(self): + sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + with sock: + coro = self.loop.create_connection(MyProto, sock=sock) + with self.assertRaisesRegex(ValueError, + 'A Stream Socket was expected'): + self.loop.run_until_complete(coro) + + def test_create_server_wrong_sock(self): + sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + with sock: + coro = self.loop.create_server(MyProto, sock=sock) + with self.assertRaisesRegex(ValueError, + 'A Stream Socket was expected'): + self.loop.run_until_complete(coro) + + def test_create_server_ssl_timeout_for_plain_socket(self): + coro = self.loop.create_server( + MyProto, 'example.com', 80, ssl_handshake_timeout=1) + with self.assertRaisesRegex( + ValueError, + 'ssl_handshake_timeout is only meaningful with ssl'): + self.loop.run_until_complete(coro) + + @unittest.skipUnless(hasattr(socket, 'SOCK_NONBLOCK'), + 'no socket.SOCK_NONBLOCK (linux only)') + def test_create_server_stream_bittype(self): + sock = socket.socket( + socket.AF_INET, socket.SOCK_STREAM | socket.SOCK_NONBLOCK) + with sock: + coro = self.loop.create_server(lambda: None, sock=sock) + srv = self.loop.run_until_complete(coro) + srv.close() + self.loop.run_until_complete(srv.wait_closed()) + + @unittest.skipUnless(socket_helper.IPV6_ENABLED, 'no IPv6 support') + def test_create_server_ipv6(self): + async def main(): + srv = await asyncio.start_server(lambda: None, '::1', 0) + try: + self.assertGreater(len(srv.sockets), 0) + finally: + srv.close() + await srv.wait_closed() + + try: + self.loop.run_until_complete(main()) + except OSError as ex: + if (hasattr(errno, 'EADDRNOTAVAIL') and + ex.errno == errno.EADDRNOTAVAIL): + self.skipTest('failed to bind to ::1') + else: + raise + + def test_create_datagram_endpoint_wrong_sock(self): + sock = socket.socket(socket.AF_INET) + with sock: + coro = self.loop.create_datagram_endpoint(MyProto, sock=sock) + with self.assertRaisesRegex(ValueError, + 'A datagram socket was expected'): + self.loop.run_until_complete(coro) + + def test_create_connection_no_host_port_sock(self): + coro = self.loop.create_connection(MyProto) + self.assertRaises(ValueError, self.loop.run_until_complete, coro) + + def test_create_connection_no_getaddrinfo(self): + async def getaddrinfo(*args, **kw): + return [] + + def getaddrinfo_task(*args, **kwds): + return self.loop.create_task(getaddrinfo(*args, **kwds)) + + self.loop.getaddrinfo = getaddrinfo_task + coro = self.loop.create_connection(MyProto, 'example.com', 80) + self.assertRaises( + OSError, self.loop.run_until_complete, coro) + + def test_create_connection_connect_err(self): + async def getaddrinfo(*args, **kw): + return [(2, 1, 6, '', ('107.6.106.82', 80))] + + def getaddrinfo_task(*args, **kwds): + return self.loop.create_task(getaddrinfo(*args, **kwds)) + + self.loop.getaddrinfo = getaddrinfo_task + self.loop.sock_connect = mock.Mock() + self.loop.sock_connect.side_effect = OSError + + coro = self.loop.create_connection(MyProto, 'example.com', 80) + self.assertRaises( + OSError, self.loop.run_until_complete, coro) + + coro = self.loop.create_connection(MyProto, 'example.com', 80, all_errors=True) + with self.assertRaises(ExceptionGroup) as cm: + self.loop.run_until_complete(coro) + + self.assertIsInstance(cm.exception, ExceptionGroup) + self.assertEqual(len(cm.exception.exceptions), 1) + self.assertIsInstance(cm.exception.exceptions[0], OSError) + + @patch_socket + def test_create_connection_connect_non_os_err_close_err(self, m_socket): + # Test the case when sock_connect() raises non-OSError exception + # and sock.close() raises OSError. + async def getaddrinfo(*args, **kw): + return [(2, 1, 6, '', ('107.6.106.82', 80))] + + def getaddrinfo_task(*args, **kwds): + return self.loop.create_task(getaddrinfo(*args, **kwds)) + + self.loop.getaddrinfo = getaddrinfo_task + self.loop.sock_connect = mock.Mock() + self.loop.sock_connect.side_effect = CustomError + sock = mock.Mock() + m_socket.socket.return_value = sock + sock.close.side_effect = OSError + + coro = self.loop.create_connection(MyProto, 'example.com', 80) + self.assertRaises( + CustomError, self.loop.run_until_complete, coro) + + coro = self.loop.create_connection(MyProto, 'example.com', 80, all_errors=True) + self.assertRaises( + CustomError, self.loop.run_until_complete, coro) + + def test_create_connection_multiple(self): + async def getaddrinfo(*args, **kw): + return [(2, 1, 6, '', ('0.0.0.1', 80)), + (2, 1, 6, '', ('0.0.0.2', 80))] + + def getaddrinfo_task(*args, **kwds): + return self.loop.create_task(getaddrinfo(*args, **kwds)) + + self.loop.getaddrinfo = getaddrinfo_task + self.loop.sock_connect = mock.Mock() + self.loop.sock_connect.side_effect = OSError + + coro = self.loop.create_connection( + MyProto, 'example.com', 80, family=socket.AF_INET) + with self.assertRaises(OSError): + self.loop.run_until_complete(coro) + + coro = self.loop.create_connection( + MyProto, 'example.com', 80, family=socket.AF_INET, all_errors=True) + with self.assertRaises(ExceptionGroup) as cm: + self.loop.run_until_complete(coro) + + self.assertIsInstance(cm.exception, ExceptionGroup) + for e in cm.exception.exceptions: + self.assertIsInstance(e, OSError) + + @patch_socket + def test_create_connection_multiple_errors_local_addr(self, m_socket): + + def bind(addr): + if addr[0] == '0.0.0.1': + err = OSError('Err') + err.strerror = 'Err' + raise err + + m_socket.socket.return_value.bind = bind + + async def getaddrinfo(*args, **kw): + return [(2, 1, 6, '', ('0.0.0.1', 80)), + (2, 1, 6, '', ('0.0.0.2', 80))] + + def getaddrinfo_task(*args, **kwds): + return self.loop.create_task(getaddrinfo(*args, **kwds)) + + self.loop.getaddrinfo = getaddrinfo_task + self.loop.sock_connect = mock.Mock() + self.loop.sock_connect.side_effect = OSError('Err2') + + coro = self.loop.create_connection( + MyProto, 'example.com', 80, family=socket.AF_INET, + local_addr=(None, 8080)) + with self.assertRaises(OSError) as cm: + self.loop.run_until_complete(coro) + + self.assertStartsWith(str(cm.exception), 'Multiple exceptions: ') + self.assertTrue(m_socket.socket.return_value.close.called) + + coro = self.loop.create_connection( + MyProto, 'example.com', 80, family=socket.AF_INET, + local_addr=(None, 8080), all_errors=True) + with self.assertRaises(ExceptionGroup) as cm: + self.loop.run_until_complete(coro) + + self.assertIsInstance(cm.exception, ExceptionGroup) + for e in cm.exception.exceptions: + self.assertIsInstance(e, OSError) + + def _test_create_connection_ip_addr(self, m_socket, allow_inet_pton): + # Test the fallback code, even if this system has inet_pton. + if not allow_inet_pton: + del m_socket.inet_pton + + m_socket.getaddrinfo = socket.getaddrinfo + sock = m_socket.socket.return_value + + self.loop._add_reader = mock.Mock() + self.loop._add_writer = mock.Mock() + + coro = self.loop.create_connection(asyncio.Protocol, '1.2.3.4', 80) + t, p = self.loop.run_until_complete(coro) + try: + sock.connect.assert_called_with(('1.2.3.4', 80)) + _, kwargs = m_socket.socket.call_args + self.assertEqual(kwargs['family'], m_socket.AF_INET) + self.assertEqual(kwargs['type'], m_socket.SOCK_STREAM) + finally: + t.close() + test_utils.run_briefly(self.loop) # allow transport to close + + if socket_helper.IPV6_ENABLED: + sock.family = socket.AF_INET6 + coro = self.loop.create_connection(asyncio.Protocol, '::1', 80) + t, p = self.loop.run_until_complete(coro) + try: + # Without inet_pton we use getaddrinfo, which transforms + # ('::1', 80) to ('::1', 80, 0, 0). The last 0s are flow info, + # scope id. + [address] = sock.connect.call_args[0] + host, port = address[:2] + self.assertRegex(host, r'::(0\.)*1') + self.assertEqual(port, 80) + _, kwargs = m_socket.socket.call_args + self.assertEqual(kwargs['family'], m_socket.AF_INET6) + self.assertEqual(kwargs['type'], m_socket.SOCK_STREAM) + finally: + t.close() + test_utils.run_briefly(self.loop) # allow transport to close + + @unittest.skipUnless(socket_helper.IPV6_ENABLED, 'no IPv6 support') + @unittest.skipIf(sys.platform.startswith('aix'), + "bpo-25545: IPv6 scope id and getaddrinfo() behave differently on AIX") + @patch_socket + def test_create_connection_ipv6_scope(self, m_socket): + m_socket.getaddrinfo = socket.getaddrinfo + sock = m_socket.socket.return_value + sock.family = socket.AF_INET6 + + self.loop._add_reader = mock.Mock() + self.loop._add_writer = mock.Mock() + + coro = self.loop.create_connection(asyncio.Protocol, 'fe80::1%1', 80) + t, p = self.loop.run_until_complete(coro) + try: + sock.connect.assert_called_with(('fe80::1', 80, 0, 1)) + _, kwargs = m_socket.socket.call_args + self.assertEqual(kwargs['family'], m_socket.AF_INET6) + self.assertEqual(kwargs['type'], m_socket.SOCK_STREAM) + finally: + t.close() + test_utils.run_briefly(self.loop) # allow transport to close + + @patch_socket + def test_create_connection_ip_addr(self, m_socket): + self._test_create_connection_ip_addr(m_socket, True) + + @patch_socket + def test_create_connection_no_inet_pton(self, m_socket): + self._test_create_connection_ip_addr(m_socket, False) + + @patch_socket + @unittest.skipIf( + support.is_android and platform.android_ver().api_level < 23, + "Issue gh-71123: this fails on Android before API level 23" + ) + def test_create_connection_service_name(self, m_socket): + m_socket.getaddrinfo = socket.getaddrinfo + sock = m_socket.socket.return_value + + self.loop._add_reader = mock.Mock() + self.loop._add_writer = mock.Mock() + + for service, port in ('http', 80), (b'http', 80): + coro = self.loop.create_connection(asyncio.Protocol, + '127.0.0.1', service) + + t, p = self.loop.run_until_complete(coro) + try: + sock.connect.assert_called_with(('127.0.0.1', port)) + _, kwargs = m_socket.socket.call_args + self.assertEqual(kwargs['family'], m_socket.AF_INET) + self.assertEqual(kwargs['type'], m_socket.SOCK_STREAM) + finally: + t.close() + test_utils.run_briefly(self.loop) # allow transport to close + + for service in 'nonsense', b'nonsense': + coro = self.loop.create_connection(asyncio.Protocol, + '127.0.0.1', service) + + with self.assertRaises(OSError): + self.loop.run_until_complete(coro) + + def test_create_connection_no_local_addr(self): + async def getaddrinfo(host, *args, **kw): + if host == 'example.com': + return [(2, 1, 6, '', ('107.6.106.82', 80)), + (2, 1, 6, '', ('107.6.106.82', 80))] + else: + return [] + + def getaddrinfo_task(*args, **kwds): + return self.loop.create_task(getaddrinfo(*args, **kwds)) + self.loop.getaddrinfo = getaddrinfo_task + + coro = self.loop.create_connection( + MyProto, 'example.com', 80, family=socket.AF_INET, + local_addr=(None, 8080)) + self.assertRaises( + OSError, self.loop.run_until_complete, coro) + + @patch_socket + def test_create_connection_bluetooth(self, m_socket): + # See http://bugs.python.org/issue27136, fallback to getaddrinfo when + # we can't recognize an address is resolved, e.g. a Bluetooth address. + addr = ('00:01:02:03:04:05', 1) + + def getaddrinfo(host, port, *args, **kw): + self.assertEqual((host, port), addr) + return [(999, 1, 999, '', (addr, 1))] + + m_socket.getaddrinfo = getaddrinfo + sock = m_socket.socket() + coro = self.loop.sock_connect(sock, addr) + self.loop.run_until_complete(coro) + + def test_create_connection_ssl_server_hostname_default(self): + self.loop.getaddrinfo = mock.Mock() + + def mock_getaddrinfo(*args, **kwds): + f = self.loop.create_future() + f.set_result([(socket.AF_INET, socket.SOCK_STREAM, + socket.SOL_TCP, '', ('1.2.3.4', 80))]) + return f + + self.loop.getaddrinfo.side_effect = mock_getaddrinfo + self.loop.sock_connect = mock.Mock() + self.loop.sock_connect.return_value = self.loop.create_future() + self.loop.sock_connect.return_value.set_result(None) + self.loop._make_ssl_transport = mock.Mock() + + class _SelectorTransportMock: + _sock = None + + def get_extra_info(self, key): + return mock.Mock() + + def close(self): + self._sock.close() + + def mock_make_ssl_transport(sock, protocol, sslcontext, waiter, + **kwds): + waiter.set_result(None) + transport = _SelectorTransportMock() + transport._sock = sock + return transport + + self.loop._make_ssl_transport.side_effect = mock_make_ssl_transport + ANY = mock.ANY + handshake_timeout = object() + shutdown_timeout = object() + # First try the default server_hostname. + self.loop._make_ssl_transport.reset_mock() + coro = self.loop.create_connection( + MyProto, 'python.org', 80, ssl=True, + ssl_handshake_timeout=handshake_timeout, + ssl_shutdown_timeout=shutdown_timeout) + transport, _ = self.loop.run_until_complete(coro) + transport.close() + self.loop._make_ssl_transport.assert_called_with( + ANY, ANY, ANY, ANY, + server_side=False, + server_hostname='python.org', + ssl_handshake_timeout=handshake_timeout, + ssl_shutdown_timeout=shutdown_timeout) + # Next try an explicit server_hostname. + self.loop._make_ssl_transport.reset_mock() + coro = self.loop.create_connection( + MyProto, 'python.org', 80, ssl=True, + server_hostname='perl.com', + ssl_handshake_timeout=handshake_timeout, + ssl_shutdown_timeout=shutdown_timeout) + transport, _ = self.loop.run_until_complete(coro) + transport.close() + self.loop._make_ssl_transport.assert_called_with( + ANY, ANY, ANY, ANY, + server_side=False, + server_hostname='perl.com', + ssl_handshake_timeout=handshake_timeout, + ssl_shutdown_timeout=shutdown_timeout) + # Finally try an explicit empty server_hostname. + self.loop._make_ssl_transport.reset_mock() + coro = self.loop.create_connection( + MyProto, 'python.org', 80, ssl=True, + server_hostname='', + ssl_handshake_timeout=handshake_timeout, + ssl_shutdown_timeout=shutdown_timeout) + transport, _ = self.loop.run_until_complete(coro) + transport.close() + self.loop._make_ssl_transport.assert_called_with( + ANY, ANY, ANY, ANY, + server_side=False, + server_hostname='', + ssl_handshake_timeout=handshake_timeout, + ssl_shutdown_timeout=shutdown_timeout) + + def test_create_connection_no_ssl_server_hostname_errors(self): + # When not using ssl, server_hostname must be None. + coro = self.loop.create_connection(MyProto, 'python.org', 80, + server_hostname='') + self.assertRaises(ValueError, self.loop.run_until_complete, coro) + coro = self.loop.create_connection(MyProto, 'python.org', 80, + server_hostname='python.org') + self.assertRaises(ValueError, self.loop.run_until_complete, coro) + + def test_create_connection_ssl_server_hostname_errors(self): + # When using ssl, server_hostname may be None if host is non-empty. + coro = self.loop.create_connection(MyProto, '', 80, ssl=True) + self.assertRaises(ValueError, self.loop.run_until_complete, coro) + coro = self.loop.create_connection(MyProto, None, 80, ssl=True) + self.assertRaises(ValueError, self.loop.run_until_complete, coro) + sock = socket.socket() + coro = self.loop.create_connection(MyProto, None, None, + ssl=True, sock=sock) + self.addCleanup(sock.close) + self.assertRaises(ValueError, self.loop.run_until_complete, coro) + + def test_create_connection_ssl_timeout_for_plain_socket(self): + coro = self.loop.create_connection( + MyProto, 'example.com', 80, ssl_handshake_timeout=1) + with self.assertRaisesRegex( + ValueError, + 'ssl_handshake_timeout is only meaningful with ssl'): + self.loop.run_until_complete(coro) + + def test_create_server_empty_host(self): + # if host is empty string use None instead + host = object() + + async def getaddrinfo(*args, **kw): + nonlocal host + host = args[0] + return [] + + def getaddrinfo_task(*args, **kwds): + return self.loop.create_task(getaddrinfo(*args, **kwds)) + + self.loop.getaddrinfo = getaddrinfo_task + fut = self.loop.create_server(MyProto, '', 0) + self.assertRaises(OSError, self.loop.run_until_complete, fut) + self.assertIsNone(host) + + def test_create_server_host_port_sock(self): + fut = self.loop.create_server( + MyProto, '0.0.0.0', 0, sock=object()) + self.assertRaises(ValueError, self.loop.run_until_complete, fut) + + def test_create_server_no_host_port_sock(self): + fut = self.loop.create_server(MyProto) + self.assertRaises(ValueError, self.loop.run_until_complete, fut) + + def test_create_server_no_getaddrinfo(self): + getaddrinfo = self.loop.getaddrinfo = mock.Mock() + getaddrinfo.return_value = self.loop.create_future() + getaddrinfo.return_value.set_result(None) + + f = self.loop.create_server(MyProto, 'python.org', 0) + self.assertRaises(OSError, self.loop.run_until_complete, f) + + @patch_socket + def test_create_server_nosoreuseport(self, m_socket): + m_socket.getaddrinfo = socket.getaddrinfo + del m_socket.SO_REUSEPORT + m_socket.socket.return_value = mock.Mock() + + f = self.loop.create_server( + MyProto, '0.0.0.0', 0, reuse_port=True) + + self.assertRaises(ValueError, self.loop.run_until_complete, f) + + @patch_socket + def test_create_server_soreuseport_only_defined(self, m_socket): + m_socket.getaddrinfo = socket.getaddrinfo + m_socket.socket.return_value = mock.Mock() + m_socket.SO_REUSEPORT = -1 + + f = self.loop.create_server( + MyProto, '0.0.0.0', 0, reuse_port=True) + + self.assertRaises(ValueError, self.loop.run_until_complete, f) + + @patch_socket + def test_create_server_cant_bind(self, m_socket): + + class Err(OSError): + strerror = 'error' + + m_socket.getaddrinfo.return_value = [ + (2, 1, 6, '', ('127.0.0.1', 10100))] + m_sock = m_socket.socket.return_value = mock.Mock() + m_sock.bind.side_effect = Err + + fut = self.loop.create_server(MyProto, '0.0.0.0', 0) + self.assertRaises(OSError, self.loop.run_until_complete, fut) + self.assertTrue(m_sock.close.called) + + @patch_socket + def test_create_datagram_endpoint_no_addrinfo(self, m_socket): + m_socket.getaddrinfo.return_value = [] + + coro = self.loop.create_datagram_endpoint( + MyDatagramProto, local_addr=('localhost', 0)) + self.assertRaises( + OSError, self.loop.run_until_complete, coro) + + def test_create_datagram_endpoint_addr_error(self): + coro = self.loop.create_datagram_endpoint( + MyDatagramProto, local_addr='localhost') + self.assertRaises( + TypeError, self.loop.run_until_complete, coro) + coro = self.loop.create_datagram_endpoint( + MyDatagramProto, local_addr=('localhost', 1, 2, 3)) + self.assertRaises( + TypeError, self.loop.run_until_complete, coro) + + def test_create_datagram_endpoint_connect_err(self): + self.loop.sock_connect = mock.Mock() + self.loop.sock_connect.side_effect = OSError + + coro = self.loop.create_datagram_endpoint( + asyncio.DatagramProtocol, remote_addr=('127.0.0.1', 0)) + self.assertRaises( + OSError, self.loop.run_until_complete, coro) + + def test_create_datagram_endpoint_allow_broadcast(self): + protocol = MyDatagramProto(create_future=True, loop=self.loop) + self.loop.sock_connect = sock_connect = mock.Mock() + sock_connect.return_value = [] + + coro = self.loop.create_datagram_endpoint( + lambda: protocol, + remote_addr=('127.0.0.1', 0), + allow_broadcast=True) + + transport, _ = self.loop.run_until_complete(coro) + self.assertFalse(sock_connect.called) + + transport.close() + self.loop.run_until_complete(protocol.done) + self.assertEqual('CLOSED', protocol.state) + + @patch_socket + def test_create_datagram_endpoint_socket_err(self, m_socket): + m_socket.getaddrinfo = socket.getaddrinfo + m_socket.socket.side_effect = OSError + + coro = self.loop.create_datagram_endpoint( + asyncio.DatagramProtocol, family=socket.AF_INET) + self.assertRaises( + OSError, self.loop.run_until_complete, coro) + + coro = self.loop.create_datagram_endpoint( + asyncio.DatagramProtocol, local_addr=('127.0.0.1', 0)) + self.assertRaises( + OSError, self.loop.run_until_complete, coro) + + @unittest.skipUnless(socket_helper.IPV6_ENABLED, 'IPv6 not supported or enabled') + def test_create_datagram_endpoint_no_matching_family(self): + coro = self.loop.create_datagram_endpoint( + asyncio.DatagramProtocol, + remote_addr=('127.0.0.1', 0), local_addr=('::1', 0)) + self.assertRaises( + ValueError, self.loop.run_until_complete, coro) + + @patch_socket + def test_create_datagram_endpoint_setblk_err(self, m_socket): + m_socket.socket.return_value.setblocking.side_effect = OSError + + coro = self.loop.create_datagram_endpoint( + asyncio.DatagramProtocol, family=socket.AF_INET) + self.assertRaises( + OSError, self.loop.run_until_complete, coro) + self.assertTrue( + m_socket.socket.return_value.close.called) + + def test_create_datagram_endpoint_noaddr_nofamily(self): + coro = self.loop.create_datagram_endpoint( + asyncio.DatagramProtocol) + self.assertRaises(ValueError, self.loop.run_until_complete, coro) + + @patch_socket + def test_create_datagram_endpoint_cant_bind(self, m_socket): + class Err(OSError): + pass + + m_socket.getaddrinfo = socket.getaddrinfo + m_sock = m_socket.socket.return_value = mock.Mock() + m_sock.bind.side_effect = Err + + fut = self.loop.create_datagram_endpoint( + MyDatagramProto, + local_addr=('127.0.0.1', 0), family=socket.AF_INET) + self.assertRaises(Err, self.loop.run_until_complete, fut) + self.assertTrue(m_sock.close.called) + + def test_create_datagram_endpoint_sock(self): + sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + sock.bind(('127.0.0.1', 0)) + fut = self.loop.create_datagram_endpoint( + lambda: MyDatagramProto(create_future=True, loop=self.loop), + sock=sock) + transport, protocol = self.loop.run_until_complete(fut) + transport.close() + self.loop.run_until_complete(protocol.done) + self.assertEqual('CLOSED', protocol.state) + + @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') + def test_create_datagram_endpoint_sock_unix(self): + fut = self.loop.create_datagram_endpoint( + lambda: MyDatagramProto(create_future=True, loop=self.loop), + family=socket.AF_UNIX) + transport, protocol = self.loop.run_until_complete(fut) + self.assertEqual(transport._sock.family, socket.AF_UNIX) + transport.close() + self.loop.run_until_complete(protocol.done) + self.assertEqual('CLOSED', protocol.state) + + @socket_helper.skip_unless_bind_unix_socket + def test_create_datagram_endpoint_existing_sock_unix(self): + with test_utils.unix_socket_path() as path: + sock = socket.socket(socket.AF_UNIX, type=socket.SOCK_DGRAM) + sock.bind(path) + sock.close() + + coro = self.loop.create_datagram_endpoint( + lambda: MyDatagramProto(create_future=True, loop=self.loop), + path, family=socket.AF_UNIX) + transport, protocol = self.loop.run_until_complete(coro) + transport.close() + self.loop.run_until_complete(protocol.done) + + def test_create_datagram_endpoint_sock_sockopts(self): + class FakeSock: + type = socket.SOCK_DGRAM + + fut = self.loop.create_datagram_endpoint( + MyDatagramProto, local_addr=('127.0.0.1', 0), sock=FakeSock()) + self.assertRaises(ValueError, self.loop.run_until_complete, fut) + + fut = self.loop.create_datagram_endpoint( + MyDatagramProto, remote_addr=('127.0.0.1', 0), sock=FakeSock()) + self.assertRaises(ValueError, self.loop.run_until_complete, fut) + + fut = self.loop.create_datagram_endpoint( + MyDatagramProto, family=1, sock=FakeSock()) + self.assertRaises(ValueError, self.loop.run_until_complete, fut) + + fut = self.loop.create_datagram_endpoint( + MyDatagramProto, proto=1, sock=FakeSock()) + self.assertRaises(ValueError, self.loop.run_until_complete, fut) + + fut = self.loop.create_datagram_endpoint( + MyDatagramProto, flags=1, sock=FakeSock()) + self.assertRaises(ValueError, self.loop.run_until_complete, fut) + + fut = self.loop.create_datagram_endpoint( + MyDatagramProto, reuse_port=True, sock=FakeSock()) + self.assertRaises(ValueError, self.loop.run_until_complete, fut) + + fut = self.loop.create_datagram_endpoint( + MyDatagramProto, allow_broadcast=True, sock=FakeSock()) + self.assertRaises(ValueError, self.loop.run_until_complete, fut) + + @unittest.skipIf(sys.platform == 'vxworks', + "SO_BROADCAST is enabled by default on VxWorks") + def test_create_datagram_endpoint_sockopts(self): + # Socket options should not be applied unless asked for. + # SO_REUSEPORT is not available on all platforms. + + coro = self.loop.create_datagram_endpoint( + lambda: MyDatagramProto(create_future=True, loop=self.loop), + local_addr=('127.0.0.1', 0)) + transport, protocol = self.loop.run_until_complete(coro) + sock = transport.get_extra_info('socket') + + reuseport_supported = hasattr(socket, 'SO_REUSEPORT') + + if reuseport_supported: + self.assertFalse( + sock.getsockopt( + socket.SOL_SOCKET, socket.SO_REUSEPORT)) + self.assertFalse( + sock.getsockopt( + socket.SOL_SOCKET, socket.SO_BROADCAST)) + + transport.close() + self.loop.run_until_complete(protocol.done) + self.assertEqual('CLOSED', protocol.state) + + coro = self.loop.create_datagram_endpoint( + lambda: MyDatagramProto(create_future=True, loop=self.loop), + local_addr=('127.0.0.1', 0), + reuse_port=reuseport_supported, + allow_broadcast=True) + transport, protocol = self.loop.run_until_complete(coro) + sock = transport.get_extra_info('socket') + + self.assertFalse( + sock.getsockopt( + socket.SOL_SOCKET, socket.SO_REUSEADDR)) + if reuseport_supported: + self.assertTrue( + sock.getsockopt( + socket.SOL_SOCKET, socket.SO_REUSEPORT)) + self.assertTrue( + sock.getsockopt( + socket.SOL_SOCKET, socket.SO_BROADCAST)) + + transport.close() + self.loop.run_until_complete(protocol.done) + self.assertEqual('CLOSED', protocol.state) + + @patch_socket + def test_create_datagram_endpoint_nosoreuseport(self, m_socket): + del m_socket.SO_REUSEPORT + m_socket.socket.return_value = mock.Mock() + + coro = self.loop.create_datagram_endpoint( + lambda: MyDatagramProto(loop=self.loop), + local_addr=('127.0.0.1', 0), + reuse_port=True) + + self.assertRaises(ValueError, self.loop.run_until_complete, coro) + + @patch_socket + def test_create_datagram_endpoint_ip_addr(self, m_socket): + def getaddrinfo(*args, **kw): + self.fail('should not have called getaddrinfo') + + m_socket.getaddrinfo = getaddrinfo + m_socket.socket.return_value.bind = bind = mock.Mock() + self.loop._add_reader = mock.Mock() + + reuseport_supported = hasattr(socket, 'SO_REUSEPORT') + coro = self.loop.create_datagram_endpoint( + lambda: MyDatagramProto(loop=self.loop), + local_addr=('1.2.3.4', 0), + reuse_port=reuseport_supported) + + t, p = self.loop.run_until_complete(coro) + try: + bind.assert_called_with(('1.2.3.4', 0)) + m_socket.socket.assert_called_with(family=m_socket.AF_INET, + proto=m_socket.IPPROTO_UDP, + type=m_socket.SOCK_DGRAM) + finally: + t.close() + test_utils.run_briefly(self.loop) # allow transport to close + + def test_accept_connection_retry(self): + sock = mock.Mock() + sock.accept.side_effect = BlockingIOError() + + self.loop._accept_connection(MyProto, sock) + self.assertFalse(sock.close.called) + + @mock.patch('asyncio.base_events.logger') + def test_accept_connection_exception(self, m_log): + sock = mock.Mock() + sock.fileno.return_value = 10 + sock.accept.side_effect = OSError(errno.EMFILE, 'Too many open files') + self.loop._remove_reader = mock.Mock() + self.loop.call_later = mock.Mock() + + self.loop._accept_connection(MyProto, sock) + self.assertTrue(m_log.error.called) + self.assertFalse(sock.close.called) + self.loop._remove_reader.assert_called_with(10) + self.loop.call_later.assert_called_with( + constants.ACCEPT_RETRY_DELAY, + # self.loop._start_serving + mock.ANY, + MyProto, sock, None, None, mock.ANY, mock.ANY, mock.ANY) + + def test_call_coroutine(self): + async def simple_coroutine(): + pass + + self.loop.set_debug(True) + coro_func = simple_coroutine + coro_obj = coro_func() + self.addCleanup(coro_obj.close) + for func in (coro_func, coro_obj): + with self.assertRaises(TypeError): + self.loop.call_soon(func) + with self.assertRaises(TypeError): + self.loop.call_soon_threadsafe(func) + with self.assertRaises(TypeError): + self.loop.call_later(60, func) + with self.assertRaises(TypeError): + self.loop.call_at(self.loop.time() + 60, func) + with self.assertRaises(TypeError): + self.loop.run_until_complete( + self.loop.run_in_executor(None, func)) + + @mock.patch('asyncio.base_events.logger') + def test_log_slow_callbacks(self, m_logger): + def stop_loop_cb(loop): + loop.stop() + + async def stop_loop_coro(loop): + loop.stop() + + asyncio.set_event_loop(self.loop) + self.loop.set_debug(True) + self.loop.slow_callback_duration = 0.0 + + # slow callback + self.loop.call_soon(stop_loop_cb, self.loop) + self.loop.run_forever() + fmt, *args = m_logger.warning.call_args[0] + self.assertRegex(fmt % tuple(args), + "^Executing " + "took .* seconds$") + + # slow task + asyncio.ensure_future(stop_loop_coro(self.loop), loop=self.loop) + self.loop.run_forever() + fmt, *args = m_logger.warning.call_args[0] + self.assertRegex(fmt % tuple(args), + "^Executing " + "took .* seconds$") + + +class RunningLoopTests(unittest.TestCase): + + def test_running_loop_within_a_loop(self): + async def runner(loop): + loop.run_forever() + + loop = asyncio.new_event_loop() + outer_loop = asyncio.new_event_loop() + try: + with self.assertRaisesRegex(RuntimeError, + 'while another loop is running'): + outer_loop.run_until_complete(runner(loop)) + finally: + loop.close() + outer_loop.close() + + +class BaseLoopSockSendfileTests(test_utils.TestCase): + + DATA = b"12345abcde" * 16 * 1024 # 160 KiB + + class MyProto(asyncio.Protocol): + + def __init__(self, loop): + self.started = False + self.closed = False + self.data = bytearray() + self.fut = loop.create_future() + self.transport = None + + def connection_made(self, transport): + self.started = True + self.transport = transport + + def data_received(self, data): + self.data.extend(data) + + def connection_lost(self, exc): + self.closed = True + self.fut.set_result(None) + self.transport = None + + async def wait_closed(self): + await self.fut + + @classmethod + def setUpClass(cls): + cls.__old_bufsize = constants.SENDFILE_FALLBACK_READBUFFER_SIZE + constants.SENDFILE_FALLBACK_READBUFFER_SIZE = 1024 * 16 + with open(os_helper.TESTFN, 'wb') as fp: + fp.write(cls.DATA) + super().setUpClass() + + @classmethod + def tearDownClass(cls): + constants.SENDFILE_FALLBACK_READBUFFER_SIZE = cls.__old_bufsize + os_helper.unlink(os_helper.TESTFN) + super().tearDownClass() + + def setUp(self): + from asyncio.selector_events import BaseSelectorEventLoop + # BaseSelectorEventLoop() has no native implementation + self.loop = BaseSelectorEventLoop() + self.set_event_loop(self.loop) + self.file = open(os_helper.TESTFN, 'rb') + self.addCleanup(self.file.close) + super().setUp() + + def make_socket(self, blocking=False): + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + sock.setblocking(blocking) + self.addCleanup(sock.close) + return sock + + def run_loop(self, coro): + return self.loop.run_until_complete(coro) + + def prepare(self): + sock = self.make_socket() + proto = self.MyProto(self.loop) + server = self.run_loop(self.loop.create_server( + lambda: proto, socket_helper.HOST, 0, family=socket.AF_INET)) + addr = server.sockets[0].getsockname() + + for _ in range(10): + try: + self.run_loop(self.loop.sock_connect(sock, addr)) + except OSError: + self.run_loop(asyncio.sleep(0.5)) + continue + else: + break + else: + # One last try, so we get the exception + self.run_loop(self.loop.sock_connect(sock, addr)) + + def cleanup(): + server.close() + sock.close() + if proto.transport is not None: + proto.transport.close() + self.run_loop(proto.wait_closed()) + self.run_loop(server.wait_closed()) + + self.addCleanup(cleanup) + + return sock, proto + + def test__sock_sendfile_native_failure(self): + sock, proto = self.prepare() + + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, + "sendfile is not available"): + self.run_loop(self.loop._sock_sendfile_native(sock, self.file, + 0, None)) + + self.assertEqual(proto.data, b'') + self.assertEqual(self.file.tell(), 0) + + def test_sock_sendfile_no_fallback(self): + sock, proto = self.prepare() + + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, + "sendfile is not available"): + self.run_loop(self.loop.sock_sendfile(sock, self.file, + fallback=False)) + + self.assertEqual(self.file.tell(), 0) + self.assertEqual(proto.data, b'') + + def test_sock_sendfile_fallback(self): + sock, proto = self.prepare() + + ret = self.run_loop(self.loop.sock_sendfile(sock, self.file)) + sock.close() + self.run_loop(proto.wait_closed()) + + self.assertEqual(ret, len(self.DATA)) + self.assertEqual(self.file.tell(), len(self.DATA)) + self.assertEqual(proto.data, self.DATA) + + def test_sock_sendfile_fallback_offset_and_count(self): + sock, proto = self.prepare() + + ret = self.run_loop(self.loop.sock_sendfile(sock, self.file, + 1000, 2000)) + sock.close() + self.run_loop(proto.wait_closed()) + + self.assertEqual(ret, 2000) + self.assertEqual(self.file.tell(), 3000) + self.assertEqual(proto.data, self.DATA[1000:3000]) + + def test_blocking_socket(self): + self.loop.set_debug(True) + sock = self.make_socket(blocking=True) + with self.assertRaisesRegex(ValueError, "must be non-blocking"): + self.run_loop(self.loop.sock_sendfile(sock, self.file)) + + def test_nonbinary_file(self): + sock = self.make_socket() + with open(os_helper.TESTFN, encoding="utf-8") as f: + with self.assertRaisesRegex(ValueError, "binary mode"): + self.run_loop(self.loop.sock_sendfile(sock, f)) + + def test_nonstream_socket(self): + sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + sock.setblocking(False) + self.addCleanup(sock.close) + with self.assertRaisesRegex(ValueError, "only SOCK_STREAM type"): + self.run_loop(self.loop.sock_sendfile(sock, self.file)) + + def test_notint_count(self): + sock = self.make_socket() + with self.assertRaisesRegex(TypeError, + "count must be a positive integer"): + self.run_loop(self.loop.sock_sendfile(sock, self.file, 0, 'count')) + + def test_negative_count(self): + sock = self.make_socket() + with self.assertRaisesRegex(ValueError, + "count must be a positive integer"): + self.run_loop(self.loop.sock_sendfile(sock, self.file, 0, -1)) + + def test_notint_offset(self): + sock = self.make_socket() + with self.assertRaisesRegex(TypeError, + "offset must be a non-negative integer"): + self.run_loop(self.loop.sock_sendfile(sock, self.file, 'offset')) + + def test_negative_offset(self): + sock = self.make_socket() + with self.assertRaisesRegex(ValueError, + "offset must be a non-negative integer"): + self.run_loop(self.loop.sock_sendfile(sock, self.file, -1)) + + +class TestSelectorUtils(test_utils.TestCase): + def check_set_nodelay(self, sock): + opt = sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY) + self.assertFalse(opt) + + base_events._set_nodelay(sock) + + opt = sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY) + self.assertTrue(opt) + + @unittest.skipUnless(hasattr(socket, 'TCP_NODELAY'), + 'need socket.TCP_NODELAY') + def test_set_nodelay(self): + sock = socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM, + proto=socket.IPPROTO_TCP) + with sock: + self.check_set_nodelay(sock) + + sock = socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM, + proto=socket.IPPROTO_TCP) + with sock: + sock.setblocking(False) + self.check_set_nodelay(sock) + + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_buffered_proto.py b/Lib/test/test_asyncio/test_buffered_proto.py new file mode 100644 index 00000000000..6d3edcc36f5 --- /dev/null +++ b/Lib/test/test_asyncio/test_buffered_proto.py @@ -0,0 +1,89 @@ +import asyncio +import unittest + +from test.test_asyncio import functional as func_tests + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +class ReceiveStuffProto(asyncio.BufferedProtocol): + def __init__(self, cb, con_lost_fut): + self.cb = cb + self.con_lost_fut = con_lost_fut + + def get_buffer(self, sizehint): + self.buffer = bytearray(100) + return self.buffer + + def buffer_updated(self, nbytes): + self.cb(self.buffer[:nbytes]) + + def connection_lost(self, exc): + if exc is None: + self.con_lost_fut.set_result(None) + else: + self.con_lost_fut.set_exception(exc) + + +class BaseTestBufferedProtocol(func_tests.FunctionalTestCaseMixin): + + def new_loop(self): + raise NotImplementedError + + def test_buffered_proto_create_connection(self): + + NOISE = b'12345678+' * 1024 + + async def client(addr): + data = b'' + + def on_buf(buf): + nonlocal data + data += buf + if data == NOISE: + tr.write(b'1') + + conn_lost_fut = self.loop.create_future() + + tr, pr = await self.loop.create_connection( + lambda: ReceiveStuffProto(on_buf, conn_lost_fut), *addr) + + await conn_lost_fut + + async def on_server_client(reader, writer): + writer.write(NOISE) + await reader.readexactly(1) + writer.close() + await writer.wait_closed() + + srv = self.loop.run_until_complete( + asyncio.start_server( + on_server_client, '127.0.0.1', 0)) + + addr = srv.sockets[0].getsockname() + self.loop.run_until_complete( + asyncio.wait_for(client(addr), 5)) + + srv.close() + self.loop.run_until_complete(srv.wait_closed()) + + +class BufferedProtocolSelectorTests(BaseTestBufferedProtocol, + unittest.TestCase): + + def new_loop(self): + return asyncio.SelectorEventLoop() + + +@unittest.skipUnless(hasattr(asyncio, 'ProactorEventLoop'), 'Windows only') +class BufferedProtocolProactorTests(BaseTestBufferedProtocol, + unittest.TestCase): + + def new_loop(self): + return asyncio.ProactorEventLoop() + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_context.py b/Lib/test/test_asyncio/test_context.py new file mode 100644 index 00000000000..f85f39839cb --- /dev/null +++ b/Lib/test/test_asyncio/test_context.py @@ -0,0 +1,38 @@ +import asyncio +import decimal +import unittest + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +@unittest.skipUnless(decimal.HAVE_CONTEXTVAR, "decimal is built with a thread-local context") +class DecimalContextTest(unittest.TestCase): + + def test_asyncio_task_decimal_context(self): + async def fractions(t, precision, x, y): + with decimal.localcontext() as ctx: + ctx.prec = precision + a = decimal.Decimal(x) / decimal.Decimal(y) + await asyncio.sleep(t) + b = decimal.Decimal(x) / decimal.Decimal(y ** 2) + return a, b + + async def main(): + r1, r2 = await asyncio.gather( + fractions(0.1, 3, 1, 3), fractions(0.2, 6, 1, 3)) + + return r1, r2 + + r1, r2 = asyncio.run(main()) + + self.assertEqual(str(r1[0]), '0.333') + self.assertEqual(str(r1[1]), '0.111') + + self.assertEqual(str(r2[0]), '0.333333') + self.assertEqual(str(r2[1]), '0.111111') + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_eager_task_factory.py b/Lib/test/test_asyncio/test_eager_task_factory.py new file mode 100644 index 00000000000..0561b54a3f1 --- /dev/null +++ b/Lib/test/test_asyncio/test_eager_task_factory.py @@ -0,0 +1,545 @@ +"""Tests for base_events.py""" + +import asyncio +import contextvars +import unittest + +from unittest import mock +from asyncio import tasks +from test.test_asyncio import utils as test_utils +from test.support.script_helper import assert_python_ok + +MOCK_ANY = mock.ANY + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +class EagerTaskFactoryLoopTests: + + Task = None + + def run_coro(self, coro): + """ + Helper method to run the `coro` coroutine in the test event loop. + It helps with making sure the event loop is running before starting + to execute `coro`. This is important for testing the eager step + functionality, since an eager step is taken only if the event loop + is already running. + """ + + async def coro_runner(): + self.assertTrue(asyncio.get_event_loop().is_running()) + return await coro + + return self.loop.run_until_complete(coro) + + def setUp(self): + super().setUp() + self.loop = asyncio.new_event_loop() + self.eager_task_factory = asyncio.create_eager_task_factory(self.Task) + self.loop.set_task_factory(self.eager_task_factory) + self.set_event_loop(self.loop) + + def test_eager_task_factory_set(self): + self.assertIsNotNone(self.eager_task_factory) + self.assertIs(self.loop.get_task_factory(), self.eager_task_factory) + + async def noop(): pass + + async def run(): + t = self.loop.create_task(noop()) + self.assertIsInstance(t, self.Task) + await t + + self.run_coro(run()) + + def test_await_future_during_eager_step(self): + + async def set_result(fut, val): + fut.set_result(val) + + async def run(): + fut = self.loop.create_future() + t = self.loop.create_task(set_result(fut, 'my message')) + # assert the eager step completed the task + self.assertTrue(t.done()) + return await fut + + self.assertEqual(self.run_coro(run()), 'my message') + + def test_eager_completion(self): + + async def coro(): + return 'hello' + + async def run(): + t = self.loop.create_task(coro()) + # assert the eager step completed the task + self.assertTrue(t.done()) + return await t + + self.assertEqual(self.run_coro(run()), 'hello') + + def test_block_after_eager_step(self): + + async def coro(): + await asyncio.sleep(0.1) + return 'finished after blocking' + + async def run(): + t = self.loop.create_task(coro()) + self.assertFalse(t.done()) + result = await t + self.assertTrue(t.done()) + return result + + self.assertEqual(self.run_coro(run()), 'finished after blocking') + + def test_cancellation_after_eager_completion(self): + + async def coro(): + return 'finished without blocking' + + async def run(): + t = self.loop.create_task(coro()) + t.cancel() + result = await t + # finished task can't be cancelled + self.assertFalse(t.cancelled()) + return result + + self.assertEqual(self.run_coro(run()), 'finished without blocking') + + def test_cancellation_after_eager_step_blocks(self): + + async def coro(): + await asyncio.sleep(0.1) + return 'finished after blocking' + + async def run(): + t = self.loop.create_task(coro()) + t.cancel('cancellation message') + self.assertGreater(t.cancelling(), 0) + result = await t + + with self.assertRaises(asyncio.CancelledError) as cm: + self.run_coro(run()) + + self.assertEqual('cancellation message', cm.exception.args[0]) + + def test_current_task(self): + captured_current_task = None + + async def coro(): + nonlocal captured_current_task + captured_current_task = asyncio.current_task() + # verify the task before and after blocking is identical + await asyncio.sleep(0.1) + self.assertIs(asyncio.current_task(), captured_current_task) + + async def run(): + t = self.loop.create_task(coro()) + self.assertIs(captured_current_task, t) + await t + + self.run_coro(run()) + captured_current_task = None + + def test_all_tasks_with_eager_completion(self): + captured_all_tasks = None + + async def coro(): + nonlocal captured_all_tasks + captured_all_tasks = asyncio.all_tasks() + + async def run(): + t = self.loop.create_task(coro()) + self.assertIn(t, captured_all_tasks) + self.assertNotIn(t, asyncio.all_tasks()) + + self.run_coro(run()) + + def test_all_tasks_with_blocking(self): + captured_eager_all_tasks = None + + async def coro(fut1, fut2): + nonlocal captured_eager_all_tasks + captured_eager_all_tasks = asyncio.all_tasks() + await fut1 + fut2.set_result(None) + + async def run(): + fut1 = self.loop.create_future() + fut2 = self.loop.create_future() + t = self.loop.create_task(coro(fut1, fut2)) + self.assertIn(t, captured_eager_all_tasks) + self.assertIn(t, asyncio.all_tasks()) + fut1.set_result(None) + await fut2 + self.assertNotIn(t, asyncio.all_tasks()) + + self.run_coro(run()) + + def test_context_vars(self): + cv = contextvars.ContextVar('cv', default=0) + + coro_first_step_ran = False + coro_second_step_ran = False + + async def coro(): + nonlocal coro_first_step_ran + nonlocal coro_second_step_ran + self.assertEqual(cv.get(), 1) + cv.set(2) + self.assertEqual(cv.get(), 2) + coro_first_step_ran = True + await asyncio.sleep(0.1) + self.assertEqual(cv.get(), 2) + cv.set(3) + self.assertEqual(cv.get(), 3) + coro_second_step_ran = True + + async def run(): + cv.set(1) + t = self.loop.create_task(coro()) + self.assertTrue(coro_first_step_ran) + self.assertFalse(coro_second_step_ran) + self.assertEqual(cv.get(), 1) + await t + self.assertTrue(coro_second_step_ran) + self.assertEqual(cv.get(), 1) + + self.run_coro(run()) + + def test_staggered_race_with_eager_tasks(self): + # See https://github.com/python/cpython/issues/124309 + + async def fail(): + await asyncio.sleep(0) + raise ValueError("no good") + + async def blocked(): + fut = asyncio.Future() + await fut + + async def run(): + winner, index, excs = await asyncio.staggered.staggered_race( + [ + lambda: blocked(), + lambda: asyncio.sleep(1, result="sleep1"), + lambda: fail() + ], + delay=0.25 + ) + self.assertEqual(winner, 'sleep1') + self.assertEqual(index, 1) + self.assertIsNone(excs[index]) + self.assertIsInstance(excs[0], asyncio.CancelledError) + self.assertIsInstance(excs[2], ValueError) + + self.run_coro(run()) + + def test_staggered_race_with_eager_tasks_no_delay(self): + # See https://github.com/python/cpython/issues/124309 + async def fail(): + raise ValueError("no good") + + async def run(): + winner, index, excs = await asyncio.staggered.staggered_race( + [ + lambda: fail(), + lambda: asyncio.sleep(1, result="sleep1"), + lambda: asyncio.sleep(0, result="sleep0"), + ], + delay=None + ) + self.assertEqual(winner, 'sleep1') + self.assertEqual(index, 1) + self.assertIsNone(excs[index]) + self.assertIsInstance(excs[0], ValueError) + self.assertEqual(len(excs), 2) + + self.run_coro(run()) + + def test_eager_start_false(self): + name = None + + async def asyncfn(): + nonlocal name + name = asyncio.current_task().get_name() + + async def main(): + t = asyncio.get_running_loop().create_task( + asyncfn(), eager_start=False, name="example" + ) + self.assertFalse(t.done()) + self.assertIsNone(name) + await t + self.assertEqual(name, "example") + + self.run_coro(main()) + + +class PyEagerTaskFactoryLoopTests(EagerTaskFactoryLoopTests, test_utils.TestCase): + Task = tasks._PyTask + + def setUp(self): + self._all_tasks = asyncio.all_tasks + self._current_task = asyncio.current_task + asyncio.current_task = asyncio.tasks.current_task = asyncio.tasks._py_current_task + asyncio.all_tasks = asyncio.tasks.all_tasks = asyncio.tasks._py_all_tasks + return super().setUp() + + def tearDown(self): + asyncio.current_task = asyncio.tasks.current_task = self._current_task + asyncio.all_tasks = asyncio.tasks.all_tasks = self._all_tasks + return super().tearDown() + + + +@unittest.skipUnless(hasattr(tasks, '_CTask'), + 'requires the C _asyncio module') +class CEagerTaskFactoryLoopTests(EagerTaskFactoryLoopTests, test_utils.TestCase): + Task = getattr(tasks, '_CTask', None) + + def setUp(self): + self._current_task = asyncio.current_task + self._all_tasks = asyncio.all_tasks + asyncio.current_task = asyncio.tasks.current_task = asyncio.tasks._c_current_task + asyncio.all_tasks = asyncio.tasks.all_tasks = asyncio.tasks._c_all_tasks + return super().setUp() + + def tearDown(self): + asyncio.current_task = asyncio.tasks.current_task = self._current_task + asyncio.all_tasks = asyncio.tasks.all_tasks = self._all_tasks + return super().tearDown() + + def test_issue105987(self): + code = """if 1: + from _asyncio import _swap_current_task, _set_running_loop + + class DummyTask: + pass + + class DummyLoop: + pass + + l = DummyLoop() + _set_running_loop(l) + _swap_current_task(l, DummyTask()) + t = _swap_current_task(l, None) + """ + + _, out, err = assert_python_ok("-c", code) + self.assertFalse(err) + + def test_issue122332(self): + async def coro(): + pass + + async def run(): + task = self.loop.create_task(coro()) + await task + self.assertIsNone(task.get_coro()) + + self.run_coro(run()) + + def test_name(self): + name = None + async def coro(): + nonlocal name + name = asyncio.current_task().get_name() + + async def main(): + task = self.loop.create_task(coro(), name="test name") + self.assertEqual(name, "test name") + await task + + self.run_coro(coro()) + +class AsyncTaskCounter: + def __init__(self, loop, *, task_class, eager): + self.suspense_count = 0 + self.task_count = 0 + + def CountingTask(*args, eager_start=False, **kwargs): + if not eager_start: + self.task_count += 1 + kwargs["eager_start"] = eager_start + return task_class(*args, **kwargs) + + if eager: + factory = asyncio.create_eager_task_factory(CountingTask) + else: + def factory(loop, coro, **kwargs): + return CountingTask(coro, loop=loop, **kwargs) + loop.set_task_factory(factory) + + def get(self): + return self.task_count + + +async def awaitable_chain(depth): + if depth == 0: + return 0 + return 1 + await awaitable_chain(depth - 1) + + +async def recursive_taskgroups(width, depth): + if depth == 0: + return + + async with asyncio.TaskGroup() as tg: + futures = [ + tg.create_task(recursive_taskgroups(width, depth - 1)) + for _ in range(width) + ] + + +async def recursive_gather(width, depth): + if depth == 0: + return + + await asyncio.gather( + *[recursive_gather(width, depth - 1) for _ in range(width)] + ) + + +class BaseTaskCountingTests: + + Task = None + eager = None + expected_task_count = None + + def setUp(self): + super().setUp() + self.loop = asyncio.new_event_loop() + self.counter = AsyncTaskCounter(self.loop, task_class=self.Task, eager=self.eager) + self.set_event_loop(self.loop) + + def test_awaitables_chain(self): + observed_depth = self.loop.run_until_complete(awaitable_chain(100)) + self.assertEqual(observed_depth, 100) + self.assertEqual(self.counter.get(), 0 if self.eager else 1) + + def test_recursive_taskgroups(self): + num_tasks = self.loop.run_until_complete(recursive_taskgroups(5, 4)) + self.assertEqual(self.counter.get(), self.expected_task_count) + + def test_recursive_gather(self): + self.loop.run_until_complete(recursive_gather(5, 4)) + self.assertEqual(self.counter.get(), self.expected_task_count) + + +class BaseNonEagerTaskFactoryTests(BaseTaskCountingTests): + eager = False + expected_task_count = 781 # 1 + 5 + 5^2 + 5^3 + 5^4 + + +class BaseEagerTaskFactoryTests(BaseTaskCountingTests): + eager = True + expected_task_count = 0 + + +class NonEagerTests(BaseNonEagerTaskFactoryTests, test_utils.TestCase): + Task = asyncio.tasks._CTask + + def setUp(self): + self._current_task = asyncio.current_task + asyncio.current_task = asyncio.tasks.current_task = asyncio.tasks._c_current_task + return super().setUp() + + def tearDown(self): + asyncio.current_task = asyncio.tasks.current_task = self._current_task + return super().tearDown() + +class EagerTests(BaseEagerTaskFactoryTests, test_utils.TestCase): + Task = asyncio.tasks._CTask + + def setUp(self): + self._current_task = asyncio.current_task + asyncio.current_task = asyncio.tasks.current_task = asyncio.tasks._c_current_task + return super().setUp() + + def tearDown(self): + asyncio.current_task = asyncio.tasks.current_task = self._current_task + return super().tearDown() + + +class NonEagerPyTaskTests(BaseNonEagerTaskFactoryTests, test_utils.TestCase): + Task = tasks._PyTask + + def setUp(self): + self._current_task = asyncio.current_task + asyncio.current_task = asyncio.tasks.current_task = asyncio.tasks._py_current_task + return super().setUp() + + def tearDown(self): + asyncio.current_task = asyncio.tasks.current_task = self._current_task + return super().tearDown() + + +class EagerPyTaskTests(BaseEagerTaskFactoryTests, test_utils.TestCase): + Task = tasks._PyTask + + def setUp(self): + self._current_task = asyncio.current_task + asyncio.current_task = asyncio.tasks.current_task = asyncio.tasks._py_current_task + return super().setUp() + + def tearDown(self): + asyncio.current_task = asyncio.tasks.current_task = self._current_task + return super().tearDown() + +@unittest.skipUnless(hasattr(tasks, '_CTask'), + 'requires the C _asyncio module') +class NonEagerCTaskTests(BaseNonEagerTaskFactoryTests, test_utils.TestCase): + Task = getattr(tasks, '_CTask', None) + + def setUp(self): + self._current_task = asyncio.current_task + asyncio.current_task = asyncio.tasks.current_task = asyncio.tasks._c_current_task + return super().setUp() + + def tearDown(self): + asyncio.current_task = asyncio.tasks.current_task = self._current_task + return super().tearDown() + + +@unittest.skipUnless(hasattr(tasks, '_CTask'), + 'requires the C _asyncio module') +class EagerCTaskTests(BaseEagerTaskFactoryTests, test_utils.TestCase): + Task = getattr(tasks, '_CTask', None) + + def setUp(self): + self._current_task = asyncio.current_task + asyncio.current_task = asyncio.tasks.current_task = asyncio.tasks._c_current_task + return super().setUp() + + def tearDown(self): + asyncio.current_task = asyncio.tasks.current_task = self._current_task + return super().tearDown() + + +class DefaultTaskFactoryEagerStart(test_utils.TestCase): + def test_eager_start_true_with_default_factory(self): + name = None + + async def asyncfn(): + nonlocal name + name = asyncio.current_task().get_name() + + async def main(): + t = asyncio.get_running_loop().create_task( + asyncfn(), eager_start=True, name="example" + ) + self.assertTrue(t.done()) + self.assertEqual(name, "example") + await t + + asyncio.run(main(), loop_factory=asyncio.EventLoop) + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py new file mode 100644 index 00000000000..1a06b426f71 --- /dev/null +++ b/Lib/test/test_asyncio/test_events.py @@ -0,0 +1,3180 @@ +"""Tests for events.py.""" + +import concurrent.futures +import contextlib +import functools +import io +import multiprocessing +import os +import platform +import re +import signal +import socket +try: + import ssl +except ImportError: + ssl = None +import subprocess +import sys +import threading +import time +import types +import errno +import unittest +from unittest import mock +import weakref +if sys.platform not in ('win32', 'vxworks'): + import tty + +import asyncio +from asyncio import coroutines +from asyncio import events +from asyncio import selector_events +from multiprocessing.util import _cleanup_tests as multiprocessing_cleanup_tests +from test.test_asyncio import utils as test_utils +from test import support +from test.support import socket_helper +from test.support import threading_helper +from test.support import ALWAYS_EQ, LARGEST, SMALLEST + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +def broken_unix_getsockname(): + """Return True if the platform is Mac OS 10.4 or older.""" + if sys.platform.startswith("aix"): + return True + elif sys.platform != 'darwin': + return False + version = platform.mac_ver()[0] + version = tuple(map(int, version.split('.'))) + return version < (10, 5) + + +def _test_get_event_loop_new_process__sub_proc(): + async def doit(): + return 'hello' + + with contextlib.closing(asyncio.new_event_loop()) as loop: + asyncio.set_event_loop(loop) + return loop.run_until_complete(doit()) + + +class CoroLike: + def send(self, v): + pass + + def throw(self, *exc): + pass + + def close(self): + pass + + def __await__(self): + pass + + +class MyBaseProto(asyncio.Protocol): + connected = None + done = None + + def __init__(self, loop=None): + self.transport = None + self.state = 'INITIAL' + self.nbytes = 0 + if loop is not None: + self.connected = loop.create_future() + self.done = loop.create_future() + + def _assert_state(self, *expected): + if self.state not in expected: + raise AssertionError(f'state: {self.state!r}, expected: {expected!r}') + + def connection_made(self, transport): + self.transport = transport + self._assert_state('INITIAL') + self.state = 'CONNECTED' + if self.connected: + self.connected.set_result(None) + + def data_received(self, data): + self._assert_state('CONNECTED') + self.nbytes += len(data) + + def eof_received(self): + self._assert_state('CONNECTED') + self.state = 'EOF' + + def connection_lost(self, exc): + self._assert_state('CONNECTED', 'EOF') + self.state = 'CLOSED' + if self.done: + self.done.set_result(None) + + +class MyProto(MyBaseProto): + def connection_made(self, transport): + super().connection_made(transport) + transport.write(b'GET / HTTP/1.0\r\nHost: example.com\r\n\r\n') + + +class MyDatagramProto(asyncio.DatagramProtocol): + done = None + + def __init__(self, loop=None): + self.state = 'INITIAL' + self.nbytes = 0 + if loop is not None: + self.done = loop.create_future() + + def _assert_state(self, expected): + if self.state != expected: + raise AssertionError(f'state: {self.state!r}, expected: {expected!r}') + + def connection_made(self, transport): + self.transport = transport + self._assert_state('INITIAL') + self.state = 'INITIALIZED' + + def datagram_received(self, data, addr): + self._assert_state('INITIALIZED') + self.nbytes += len(data) + + def error_received(self, exc): + self._assert_state('INITIALIZED') + + def connection_lost(self, exc): + self._assert_state('INITIALIZED') + self.state = 'CLOSED' + if self.done: + self.done.set_result(None) + + +class MyReadPipeProto(asyncio.Protocol): + done = None + + def __init__(self, loop=None): + self.state = ['INITIAL'] + self.nbytes = 0 + self.transport = None + if loop is not None: + self.done = loop.create_future() + + def _assert_state(self, expected): + if self.state != expected: + raise AssertionError(f'state: {self.state!r}, expected: {expected!r}') + + def connection_made(self, transport): + self.transport = transport + self._assert_state(['INITIAL']) + self.state.append('CONNECTED') + + def data_received(self, data): + self._assert_state(['INITIAL', 'CONNECTED']) + self.nbytes += len(data) + + def eof_received(self): + self._assert_state(['INITIAL', 'CONNECTED']) + self.state.append('EOF') + + def connection_lost(self, exc): + if 'EOF' not in self.state: + self.state.append('EOF') # It is okay if EOF is missed. + self._assert_state(['INITIAL', 'CONNECTED', 'EOF']) + self.state.append('CLOSED') + if self.done: + self.done.set_result(None) + + +class MyWritePipeProto(asyncio.BaseProtocol): + done = None + + def __init__(self, loop=None): + self.state = 'INITIAL' + self.transport = None + if loop is not None: + self.done = loop.create_future() + + def _assert_state(self, expected): + if self.state != expected: + raise AssertionError(f'state: {self.state!r}, expected: {expected!r}') + + def connection_made(self, transport): + self.transport = transport + self._assert_state('INITIAL') + self.state = 'CONNECTED' + + def connection_lost(self, exc): + self._assert_state('CONNECTED') + self.state = 'CLOSED' + if self.done: + self.done.set_result(None) + + +class MySubprocessProtocol(asyncio.SubprocessProtocol): + + def __init__(self, loop): + self.state = 'INITIAL' + self.transport = None + self.connected = loop.create_future() + self.completed = loop.create_future() + self.disconnects = {fd: loop.create_future() for fd in range(3)} + self.data = {1: b'', 2: b''} + self.returncode = None + self.got_data = {1: asyncio.Event(), + 2: asyncio.Event()} + + def _assert_state(self, expected): + if self.state != expected: + raise AssertionError(f'state: {self.state!r}, expected: {expected!r}') + + def connection_made(self, transport): + self.transport = transport + self._assert_state('INITIAL') + self.state = 'CONNECTED' + self.connected.set_result(None) + + def connection_lost(self, exc): + self._assert_state('CONNECTED') + self.state = 'CLOSED' + self.completed.set_result(None) + + def pipe_data_received(self, fd, data): + self._assert_state('CONNECTED') + self.data[fd] += data + self.got_data[fd].set() + + def pipe_connection_lost(self, fd, exc): + self._assert_state('CONNECTED') + if exc: + self.disconnects[fd].set_exception(exc) + else: + self.disconnects[fd].set_result(exc) + + def process_exited(self): + self._assert_state('CONNECTED') + self.returncode = self.transport.get_returncode() + + +class EventLoopTestsMixin: + + def setUp(self): + super().setUp() + self.loop = self.create_event_loop() + self.set_event_loop(self.loop) + + def tearDown(self): + # just in case if we have transport close callbacks + if not self.loop.is_closed(): + test_utils.run_briefly(self.loop) + + self.doCleanups() + support.gc_collect() + super().tearDown() + + @unittest.expectedFailure # TODO: RUSTPYTHON; - RuntimeWarning for unawaited coroutine not triggered + def test_run_until_complete_nesting(self): + async def coro1(): + await asyncio.sleep(0) + + async def coro2(): + self.assertTrue(self.loop.is_running()) + self.loop.run_until_complete(coro1()) + + with self.assertWarnsRegex( + RuntimeWarning, + r"coroutine \S+ was never awaited" + ): + self.assertRaises( + RuntimeError, self.loop.run_until_complete, coro2()) + + # Note: because of the default Windows timing granularity of + # 15.6 msec, we use fairly long sleep times here (~100 msec). + + def test_run_until_complete(self): + delay = 0.100 + t0 = self.loop.time() + self.loop.run_until_complete(asyncio.sleep(delay)) + dt = self.loop.time() - t0 + self.assertGreaterEqual(dt, delay - test_utils.CLOCK_RES) + + def test_run_until_complete_stopped(self): + + async def cb(): + self.loop.stop() + await asyncio.sleep(0.1) + task = cb() + self.assertRaises(RuntimeError, + self.loop.run_until_complete, task) + + def test_call_later(self): + results = [] + + def callback(arg): + results.append(arg) + self.loop.stop() + + self.loop.call_later(0.1, callback, 'hello world') + self.loop.run_forever() + self.assertEqual(results, ['hello world']) + + def test_call_soon(self): + results = [] + + def callback(arg1, arg2): + results.append((arg1, arg2)) + self.loop.stop() + + self.loop.call_soon(callback, 'hello', 'world') + self.loop.run_forever() + self.assertEqual(results, [('hello', 'world')]) + + def test_call_soon_threadsafe(self): + results = [] + lock = threading.Lock() + + def callback(arg): + results.append(arg) + if len(results) >= 2: + self.loop.stop() + + def run_in_thread(): + self.loop.call_soon_threadsafe(callback, 'hello') + lock.release() + + lock.acquire() + t = threading.Thread(target=run_in_thread) + t.start() + + with lock: + self.loop.call_soon(callback, 'world') + self.loop.run_forever() + t.join() + self.assertEqual(results, ['hello', 'world']) + + def test_call_soon_threadsafe_handle_block_check_cancelled(self): + results = [] + + callback_started = threading.Event() + callback_finished = threading.Event() + def callback(arg): + callback_started.set() + results.append(arg) + time.sleep(1) + callback_finished.set() + + def run_in_thread(): + handle = self.loop.call_soon_threadsafe(callback, 'hello') + self.assertIsInstance(handle, events._ThreadSafeHandle) + callback_started.wait() + # callback started so it should block checking for cancellation + # until it finishes + self.assertFalse(handle.cancelled()) + self.assertTrue(callback_finished.is_set()) + self.loop.call_soon_threadsafe(self.loop.stop) + + t = threading.Thread(target=run_in_thread) + t.start() + + self.loop.run_forever() + t.join() + self.assertEqual(results, ['hello']) + + def test_call_soon_threadsafe_handle_block_cancellation(self): + results = [] + + callback_started = threading.Event() + callback_finished = threading.Event() + def callback(arg): + callback_started.set() + results.append(arg) + time.sleep(1) + callback_finished.set() + + def run_in_thread(): + handle = self.loop.call_soon_threadsafe(callback, 'hello') + self.assertIsInstance(handle, events._ThreadSafeHandle) + callback_started.wait() + # callback started so it cannot be cancelled from other thread until + # it finishes + handle.cancel() + self.assertTrue(callback_finished.is_set()) + self.loop.call_soon_threadsafe(self.loop.stop) + + t = threading.Thread(target=run_in_thread) + t.start() + + self.loop.run_forever() + t.join() + self.assertEqual(results, ['hello']) + + def test_call_soon_threadsafe_handle_cancel_same_thread(self): + results = [] + callback_started = threading.Event() + callback_finished = threading.Event() + + fut = concurrent.futures.Future() + def callback(arg): + callback_started.set() + handle = fut.result() + handle.cancel() + results.append(arg) + callback_finished.set() + self.loop.stop() + + def run_in_thread(): + handle = self.loop.call_soon_threadsafe(callback, 'hello') + fut.set_result(handle) + self.assertIsInstance(handle, events._ThreadSafeHandle) + callback_started.wait() + # callback cancels itself from same thread so it has no effect + # it runs to completion + self.assertTrue(handle.cancelled()) + self.assertTrue(callback_finished.is_set()) + self.loop.call_soon_threadsafe(self.loop.stop) + + t = threading.Thread(target=run_in_thread) + t.start() + + self.loop.run_forever() + t.join() + self.assertEqual(results, ['hello']) + + def test_call_soon_threadsafe_handle_cancel_other_thread(self): + results = [] + ev = threading.Event() + + callback_finished = threading.Event() + def callback(arg): + results.append(arg) + callback_finished.set() + self.loop.stop() + + def run_in_thread(): + handle = self.loop.call_soon_threadsafe(callback, 'hello') + # handle can be cancelled from other thread if not started yet + self.assertIsInstance(handle, events._ThreadSafeHandle) + handle.cancel() + self.assertTrue(handle.cancelled()) + self.assertFalse(callback_finished.is_set()) + ev.set() + self.loop.call_soon_threadsafe(self.loop.stop) + + # block the main loop until the callback is added and cancelled in the + # other thread + self.loop.call_soon(ev.wait) + t = threading.Thread(target=run_in_thread) + t.start() + self.loop.run_forever() + t.join() + self.assertEqual(results, []) + self.assertFalse(callback_finished.is_set()) + + def test_call_soon_threadsafe_same_thread(self): + results = [] + + def callback(arg): + results.append(arg) + if len(results) >= 2: + self.loop.stop() + + self.loop.call_soon_threadsafe(callback, 'hello') + self.loop.call_soon(callback, 'world') + self.loop.run_forever() + self.assertEqual(results, ['hello', 'world']) + + def test_run_in_executor(self): + def run(arg): + return (arg, threading.get_ident()) + f2 = self.loop.run_in_executor(None, run, 'yo') + res, thread_id = self.loop.run_until_complete(f2) + self.assertEqual(res, 'yo') + self.assertNotEqual(thread_id, threading.get_ident()) + + def test_run_in_executor_cancel(self): + called = False + + def patched_call_soon(*args): + nonlocal called + called = True + + def run(): + time.sleep(0.05) + + f2 = self.loop.run_in_executor(None, run) + f2.cancel() + self.loop.run_until_complete( + self.loop.shutdown_default_executor()) + self.loop.close() + self.loop.call_soon = patched_call_soon + self.loop.call_soon_threadsafe = patched_call_soon + time.sleep(0.4) + self.assertFalse(called) + + def test_reader_callback(self): + r, w = socket.socketpair() + r.setblocking(False) + bytes_read = bytearray() + + def reader(): + try: + data = r.recv(1024) + except BlockingIOError: + # Spurious readiness notifications are possible + # at least on Linux -- see man select. + return + if data: + bytes_read.extend(data) + else: + self.assertTrue(self.loop.remove_reader(r.fileno())) + r.close() + + self.loop.add_reader(r.fileno(), reader) + self.loop.call_soon(w.send, b'abc') + test_utils.run_until(self.loop, lambda: len(bytes_read) >= 3) + self.loop.call_soon(w.send, b'def') + test_utils.run_until(self.loop, lambda: len(bytes_read) >= 6) + self.loop.call_soon(w.close) + self.loop.call_soon(self.loop.stop) + self.loop.run_forever() + self.assertEqual(bytes_read, b'abcdef') + + def test_writer_callback(self): + r, w = socket.socketpair() + w.setblocking(False) + + def writer(data): + w.send(data) + self.loop.stop() + + data = b'x' * 1024 + self.loop.add_writer(w.fileno(), writer, data) + self.loop.run_forever() + + self.assertTrue(self.loop.remove_writer(w.fileno())) + self.assertFalse(self.loop.remove_writer(w.fileno())) + + w.close() + read = r.recv(len(data) * 2) + r.close() + self.assertEqual(read, data) + + @unittest.expectedFailure # TODO: RUSTPYTHON; - signal handler implementation differs + @unittest.skipUnless(hasattr(signal, 'SIGKILL'), 'No SIGKILL') + def test_add_signal_handler(self): + caught = 0 + + def my_handler(): + nonlocal caught + caught += 1 + + # Check error behavior first. + self.assertRaises( + TypeError, self.loop.add_signal_handler, 'boom', my_handler) + self.assertRaises( + TypeError, self.loop.remove_signal_handler, 'boom') + self.assertRaises( + ValueError, self.loop.add_signal_handler, signal.NSIG+1, + my_handler) + self.assertRaises( + ValueError, self.loop.remove_signal_handler, signal.NSIG+1) + self.assertRaises( + ValueError, self.loop.add_signal_handler, 0, my_handler) + self.assertRaises( + ValueError, self.loop.remove_signal_handler, 0) + self.assertRaises( + ValueError, self.loop.add_signal_handler, -1, my_handler) + self.assertRaises( + ValueError, self.loop.remove_signal_handler, -1) + self.assertRaises( + RuntimeError, self.loop.add_signal_handler, signal.SIGKILL, + my_handler) + # Removing SIGKILL doesn't raise, since we don't call signal(). + self.assertFalse(self.loop.remove_signal_handler(signal.SIGKILL)) + # Now set a handler and handle it. + self.loop.add_signal_handler(signal.SIGINT, my_handler) + + os.kill(os.getpid(), signal.SIGINT) + test_utils.run_until(self.loop, lambda: caught) + + # Removing it should restore the default handler. + self.assertTrue(self.loop.remove_signal_handler(signal.SIGINT)) + self.assertEqual(signal.getsignal(signal.SIGINT), + signal.default_int_handler) + # Removing again returns False. + self.assertFalse(self.loop.remove_signal_handler(signal.SIGINT)) + + @unittest.skipUnless(hasattr(signal, 'SIGALRM'), 'No SIGALRM') + @unittest.skipUnless(hasattr(signal, 'setitimer'), + 'need signal.setitimer()') + def test_signal_handling_while_selecting(self): + # Test with a signal actually arriving during a select() call. + caught = 0 + + def my_handler(): + nonlocal caught + caught += 1 + self.loop.stop() + + self.loop.add_signal_handler(signal.SIGALRM, my_handler) + + signal.setitimer(signal.ITIMER_REAL, 0.01, 0) # Send SIGALRM once. + self.loop.call_later(60, self.loop.stop) + self.loop.run_forever() + self.assertEqual(caught, 1) + + @unittest.skipUnless(hasattr(signal, 'SIGALRM'), 'No SIGALRM') + @unittest.skipUnless(hasattr(signal, 'setitimer'), + 'need signal.setitimer()') + def test_signal_handling_args(self): + some_args = (42,) + caught = 0 + + def my_handler(*args): + nonlocal caught + caught += 1 + self.assertEqual(args, some_args) + self.loop.stop() + + self.loop.add_signal_handler(signal.SIGALRM, my_handler, *some_args) + + signal.setitimer(signal.ITIMER_REAL, 0.1, 0) # Send SIGALRM once. + self.loop.call_later(60, self.loop.stop) + self.loop.run_forever() + self.assertEqual(caught, 1) + + def _basetest_create_connection(self, connection_fut, check_sockname=True): + tr, pr = self.loop.run_until_complete(connection_fut) + self.assertIsInstance(tr, asyncio.Transport) + self.assertIsInstance(pr, asyncio.Protocol) + self.assertIs(pr.transport, tr) + if check_sockname: + self.assertIsNotNone(tr.get_extra_info('sockname')) + self.loop.run_until_complete(pr.done) + self.assertGreater(pr.nbytes, 0) + tr.close() + + def test_create_connection(self): + with test_utils.run_test_server() as httpd: + conn_fut = self.loop.create_connection( + lambda: MyProto(loop=self.loop), *httpd.address) + self._basetest_create_connection(conn_fut) + + @socket_helper.skip_unless_bind_unix_socket + def test_create_unix_connection(self): + # Issue #20682: On Mac OS X Tiger, getsockname() returns a + # zero-length address for UNIX socket. + check_sockname = not broken_unix_getsockname() + + with test_utils.run_test_unix_server() as httpd: + conn_fut = self.loop.create_unix_connection( + lambda: MyProto(loop=self.loop), httpd.address) + self._basetest_create_connection(conn_fut, check_sockname) + + def check_ssl_extra_info(self, client, check_sockname=True, + peername=None, peercert={}): + if check_sockname: + self.assertIsNotNone(client.get_extra_info('sockname')) + if peername: + self.assertEqual(peername, + client.get_extra_info('peername')) + else: + self.assertIsNotNone(client.get_extra_info('peername')) + self.assertEqual(peercert, + client.get_extra_info('peercert')) + + # test SSL cipher + cipher = client.get_extra_info('cipher') + self.assertIsInstance(cipher, tuple) + self.assertEqual(len(cipher), 3, cipher) + self.assertIsInstance(cipher[0], str) + self.assertIsInstance(cipher[1], str) + self.assertIsInstance(cipher[2], int) + + # test SSL object + sslobj = client.get_extra_info('ssl_object') + self.assertIsNotNone(sslobj) + self.assertEqual(sslobj.compression(), + client.get_extra_info('compression')) + self.assertEqual(sslobj.cipher(), + client.get_extra_info('cipher')) + self.assertEqual(sslobj.getpeercert(), + client.get_extra_info('peercert')) + self.assertEqual(sslobj.compression(), + client.get_extra_info('compression')) + + def _basetest_create_ssl_connection(self, connection_fut, + check_sockname=True, + peername=None): + tr, pr = self.loop.run_until_complete(connection_fut) + self.assertIsInstance(tr, asyncio.Transport) + self.assertIsInstance(pr, asyncio.Protocol) + self.assertTrue('ssl' in tr.__class__.__name__.lower()) + self.check_ssl_extra_info(tr, check_sockname, peername) + self.loop.run_until_complete(pr.done) + self.assertGreater(pr.nbytes, 0) + tr.close() + + def _test_create_ssl_connection(self, httpd, create_connection, + check_sockname=True, peername=None): + conn_fut = create_connection(ssl=test_utils.dummy_ssl_context()) + self._basetest_create_ssl_connection(conn_fut, check_sockname, + peername) + + # ssl.Purpose was introduced in Python 3.4 + if hasattr(ssl, 'Purpose'): + def _dummy_ssl_create_context(purpose=ssl.Purpose.SERVER_AUTH, *, + cafile=None, capath=None, + cadata=None): + """ + A ssl.create_default_context() replacement that doesn't enable + cert validation. + """ + self.assertEqual(purpose, ssl.Purpose.SERVER_AUTH) + return test_utils.dummy_ssl_context() + + # With ssl=True, ssl.create_default_context() should be called + with mock.patch('ssl.create_default_context', + side_effect=_dummy_ssl_create_context) as m: + conn_fut = create_connection(ssl=True) + self._basetest_create_ssl_connection(conn_fut, check_sockname, + peername) + self.assertEqual(m.call_count, 1) + + # With the real ssl.create_default_context(), certificate + # validation will fail + with self.assertRaises(ssl.SSLError) as cm: + conn_fut = create_connection(ssl=True) + # Ignore the "SSL handshake failed" log in debug mode + with test_utils.disable_logger(): + self._basetest_create_ssl_connection(conn_fut, check_sockname, + peername) + + self.assertEqual(cm.exception.reason, 'CERTIFICATE_VERIFY_FAILED') + + @unittest.skipIf(ssl is None, 'No ssl module') + def test_create_ssl_connection(self): + with test_utils.run_test_server(use_ssl=True) as httpd: + create_connection = functools.partial( + self.loop.create_connection, + lambda: MyProto(loop=self.loop), + *httpd.address) + self._test_create_ssl_connection(httpd, create_connection, + peername=httpd.address) + + @socket_helper.skip_unless_bind_unix_socket + @unittest.skipIf(ssl is None, 'No ssl module') + def test_create_ssl_unix_connection(self): + # Issue #20682: On Mac OS X Tiger, getsockname() returns a + # zero-length address for UNIX socket. + check_sockname = not broken_unix_getsockname() + + with test_utils.run_test_unix_server(use_ssl=True) as httpd: + create_connection = functools.partial( + self.loop.create_unix_connection, + lambda: MyProto(loop=self.loop), httpd.address, + server_hostname='127.0.0.1') + + self._test_create_ssl_connection(httpd, create_connection, + check_sockname, + peername=httpd.address) + + def test_create_connection_local_addr(self): + with test_utils.run_test_server() as httpd: + port = socket_helper.find_unused_port() + f = self.loop.create_connection( + lambda: MyProto(loop=self.loop), + *httpd.address, local_addr=(httpd.address[0], port)) + tr, pr = self.loop.run_until_complete(f) + expected = pr.transport.get_extra_info('sockname')[1] + self.assertEqual(port, expected) + tr.close() + + @socket_helper.skip_if_tcp_blackhole + def test_create_connection_local_addr_skip_different_family(self): + # See https://github.com/python/cpython/issues/86508 + port1 = socket_helper.find_unused_port() + port2 = socket_helper.find_unused_port() + getaddrinfo_orig = self.loop.getaddrinfo + + async def getaddrinfo(host, port, *args, **kwargs): + if port == port2: + return [(socket.AF_INET6, socket.SOCK_STREAM, 0, '', ('::1', 0, 0, 0)), + (socket.AF_INET, socket.SOCK_STREAM, 0, '', ('127.0.0.1', 0))] + return await getaddrinfo_orig(host, port, *args, **kwargs) + + self.loop.getaddrinfo = getaddrinfo + + f = self.loop.create_connection( + lambda: MyProto(loop=self.loop), + 'localhost', port1, local_addr=('localhost', port2)) + + with self.assertRaises(OSError): + self.loop.run_until_complete(f) + + @socket_helper.skip_if_tcp_blackhole + def test_create_connection_local_addr_nomatch_family(self): + # See https://github.com/python/cpython/issues/86508 + port1 = socket_helper.find_unused_port() + port2 = socket_helper.find_unused_port() + getaddrinfo_orig = self.loop.getaddrinfo + + async def getaddrinfo(host, port, *args, **kwargs): + if port == port2: + return [(socket.AF_INET6, socket.SOCK_STREAM, 0, '', ('::1', 0, 0, 0))] + return await getaddrinfo_orig(host, port, *args, **kwargs) + + self.loop.getaddrinfo = getaddrinfo + + f = self.loop.create_connection( + lambda: MyProto(loop=self.loop), + 'localhost', port1, local_addr=('localhost', port2)) + + with self.assertRaises(OSError): + self.loop.run_until_complete(f) + + def test_create_connection_local_addr_in_use(self): + with test_utils.run_test_server() as httpd: + f = self.loop.create_connection( + lambda: MyProto(loop=self.loop), + *httpd.address, local_addr=httpd.address) + with self.assertRaises(OSError) as cm: + self.loop.run_until_complete(f) + self.assertEqual(cm.exception.errno, errno.EADDRINUSE) + self.assertIn(str(httpd.address), cm.exception.strerror) + + def test_connect_accepted_socket(self, server_ssl=None, client_ssl=None): + loop = self.loop + + class MyProto(MyBaseProto): + + def connection_lost(self, exc): + super().connection_lost(exc) + loop.call_soon(loop.stop) + + def data_received(self, data): + super().data_received(data) + self.transport.write(expected_response) + + lsock = socket.create_server(('127.0.0.1', 0), backlog=1) + addr = lsock.getsockname() + + message = b'test data' + response = None + expected_response = b'roger' + + def client(): + nonlocal response + try: + csock = socket.socket() + if client_ssl is not None: + csock = client_ssl.wrap_socket(csock) + csock.connect(addr) + csock.sendall(message) + response = csock.recv(99) + csock.close() + except Exception as exc: + print( + "Failure in client thread in test_connect_accepted_socket", + exc) + + thread = threading.Thread(target=client, daemon=True) + thread.start() + + conn, _ = lsock.accept() + proto = MyProto(loop=loop) + proto.loop = loop + loop.run_until_complete( + loop.connect_accepted_socket( + (lambda: proto), conn, ssl=server_ssl)) + loop.run_forever() + proto.transport.close() + lsock.close() + + threading_helper.join_thread(thread) + self.assertFalse(thread.is_alive()) + self.assertEqual(proto.state, 'CLOSED') + self.assertEqual(proto.nbytes, len(message)) + self.assertEqual(response, expected_response) + + @unittest.skipIf(ssl is None, 'No ssl module') + def test_ssl_connect_accepted_socket(self): + server_context = test_utils.simple_server_sslcontext() + client_context = test_utils.simple_client_sslcontext() + + self.test_connect_accepted_socket(server_context, client_context) + + def test_connect_accepted_socket_ssl_timeout_for_plain_socket(self): + sock = socket.socket() + self.addCleanup(sock.close) + coro = self.loop.connect_accepted_socket( + MyProto, sock, ssl_handshake_timeout=support.LOOPBACK_TIMEOUT) + with self.assertRaisesRegex( + ValueError, + 'ssl_handshake_timeout is only meaningful with ssl'): + self.loop.run_until_complete(coro) + + @mock.patch('asyncio.base_events.socket') + def create_server_multiple_hosts(self, family, hosts, mock_sock): + async def getaddrinfo(host, port, *args, **kw): + if family == socket.AF_INET: + return [(family, socket.SOCK_STREAM, 6, '', (host, port))] + else: + return [(family, socket.SOCK_STREAM, 6, '', (host, port, 0, 0))] + + def getaddrinfo_task(*args, **kwds): + return self.loop.create_task(getaddrinfo(*args, **kwds)) + + unique_hosts = set(hosts) + + if family == socket.AF_INET: + mock_sock.socket().getsockbyname.side_effect = [ + (host, 80) for host in unique_hosts] + else: + mock_sock.socket().getsockbyname.side_effect = [ + (host, 80, 0, 0) for host in unique_hosts] + self.loop.getaddrinfo = getaddrinfo_task + self.loop._start_serving = mock.Mock() + self.loop._stop_serving = mock.Mock() + f = self.loop.create_server(lambda: MyProto(self.loop), hosts, 80) + server = self.loop.run_until_complete(f) + self.addCleanup(server.close) + server_hosts = {sock.getsockbyname()[0] for sock in server.sockets} + self.assertEqual(server_hosts, unique_hosts) + + def test_create_server_multiple_hosts_ipv4(self): + self.create_server_multiple_hosts(socket.AF_INET, + ['1.2.3.4', '5.6.7.8', '1.2.3.4']) + + def test_create_server_multiple_hosts_ipv6(self): + self.create_server_multiple_hosts(socket.AF_INET6, + ['::1', '::2', '::1']) + + def test_create_server(self): + proto = MyProto(self.loop) + f = self.loop.create_server(lambda: proto, '0.0.0.0', 0) + server = self.loop.run_until_complete(f) + self.assertEqual(len(server.sockets), 1) + sock = server.sockets[0] + host, port = sock.getsockname() + self.assertEqual(host, '0.0.0.0') + client = socket.socket() + client.connect(('127.0.0.1', port)) + client.sendall(b'xxx') + + self.loop.run_until_complete(proto.connected) + self.assertEqual('CONNECTED', proto.state) + + test_utils.run_until(self.loop, lambda: proto.nbytes > 0) + self.assertEqual(3, proto.nbytes) + + # extra info is available + self.assertIsNotNone(proto.transport.get_extra_info('sockname')) + self.assertEqual('127.0.0.1', + proto.transport.get_extra_info('peername')[0]) + + # close connection + proto.transport.close() + self.loop.run_until_complete(proto.done) + + self.assertEqual('CLOSED', proto.state) + + # the client socket must be closed after to avoid ECONNRESET upon + # recv()/send() on the serving socket + client.close() + + # close server + server.close() + + def test_create_server_trsock(self): + proto = MyProto(self.loop) + f = self.loop.create_server(lambda: proto, '0.0.0.0', 0) + server = self.loop.run_until_complete(f) + self.assertEqual(len(server.sockets), 1) + sock = server.sockets[0] + self.assertIsInstance(sock, asyncio.trsock.TransportSocket) + host, port = sock.getsockname() + self.assertEqual(host, '0.0.0.0') + dup = sock.dup() + self.addCleanup(dup.close) + self.assertIsInstance(dup, socket.socket) + self.assertFalse(sock.get_inheritable()) + with self.assertRaises(ValueError): + sock.settimeout(1) + sock.settimeout(0) + self.assertEqual(sock.gettimeout(), 0) + with self.assertRaises(ValueError): + sock.setblocking(True) + sock.setblocking(False) + server.close() + + + @unittest.skipUnless(hasattr(socket, 'SO_REUSEPORT'), 'No SO_REUSEPORT') + def test_create_server_reuse_port(self): + proto = MyProto(self.loop) + f = self.loop.create_server( + lambda: proto, '0.0.0.0', 0) + server = self.loop.run_until_complete(f) + self.assertEqual(len(server.sockets), 1) + sock = server.sockets[0] + self.assertFalse( + sock.getsockopt( + socket.SOL_SOCKET, socket.SO_REUSEPORT)) + server.close() + + test_utils.run_briefly(self.loop) + + proto = MyProto(self.loop) + f = self.loop.create_server( + lambda: proto, '0.0.0.0', 0, reuse_port=True) + server = self.loop.run_until_complete(f) + self.assertEqual(len(server.sockets), 1) + sock = server.sockets[0] + self.assertTrue( + sock.getsockopt( + socket.SOL_SOCKET, socket.SO_REUSEPORT)) + server.close() + + def _make_unix_server(self, factory, **kwargs): + path = test_utils.gen_unix_socket_path() + self.addCleanup(lambda: os.path.exists(path) and os.unlink(path)) + + f = self.loop.create_unix_server(factory, path, **kwargs) + server = self.loop.run_until_complete(f) + + return server, path + + @socket_helper.skip_unless_bind_unix_socket + def test_create_unix_server(self): + proto = MyProto(loop=self.loop) + server, path = self._make_unix_server(lambda: proto) + self.assertEqual(len(server.sockets), 1) + + client = socket.socket(socket.AF_UNIX) + client.connect(path) + client.sendall(b'xxx') + + self.loop.run_until_complete(proto.connected) + self.assertEqual('CONNECTED', proto.state) + test_utils.run_until(self.loop, lambda: proto.nbytes > 0) + self.assertEqual(3, proto.nbytes) + + # close connection + proto.transport.close() + self.loop.run_until_complete(proto.done) + + self.assertEqual('CLOSED', proto.state) + + # the client socket must be closed after to avoid ECONNRESET upon + # recv()/send() on the serving socket + client.close() + + # close server + server.close() + + @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') + def test_create_unix_server_path_socket_error(self): + proto = MyProto(loop=self.loop) + sock = socket.socket() + with sock: + f = self.loop.create_unix_server(lambda: proto, '/test', sock=sock) + with self.assertRaisesRegex(ValueError, + 'path and sock can not be specified ' + 'at the same time'): + self.loop.run_until_complete(f) + + def _create_ssl_context(self, certfile, keyfile=None): + sslcontext = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) + sslcontext.options |= ssl.OP_NO_SSLv2 + sslcontext.load_cert_chain(certfile, keyfile) + return sslcontext + + def _make_ssl_server(self, factory, certfile, keyfile=None): + sslcontext = self._create_ssl_context(certfile, keyfile) + + f = self.loop.create_server(factory, '127.0.0.1', 0, ssl=sslcontext) + server = self.loop.run_until_complete(f) + + sock = server.sockets[0] + host, port = sock.getsockname() + self.assertEqual(host, '127.0.0.1') + return server, host, port + + def _make_ssl_unix_server(self, factory, certfile, keyfile=None): + sslcontext = self._create_ssl_context(certfile, keyfile) + return self._make_unix_server(factory, ssl=sslcontext) + + @unittest.skipIf(ssl is None, 'No ssl module') + def test_create_server_ssl(self): + proto = MyProto(loop=self.loop) + server, host, port = self._make_ssl_server( + lambda: proto, test_utils.ONLYCERT, test_utils.ONLYKEY) + + f_c = self.loop.create_connection(MyBaseProto, host, port, + ssl=test_utils.dummy_ssl_context()) + client, pr = self.loop.run_until_complete(f_c) + + client.write(b'xxx') + self.loop.run_until_complete(proto.connected) + self.assertEqual('CONNECTED', proto.state) + + test_utils.run_until(self.loop, lambda: proto.nbytes > 0) + self.assertEqual(3, proto.nbytes) + + # extra info is available + self.check_ssl_extra_info(client, peername=(host, port)) + + # close connection + proto.transport.close() + self.loop.run_until_complete(proto.done) + self.assertEqual('CLOSED', proto.state) + + # the client socket must be closed after to avoid ECONNRESET upon + # recv()/send() on the serving socket + client.close() + + # stop serving + server.close() + + @socket_helper.skip_unless_bind_unix_socket + @unittest.skipIf(ssl is None, 'No ssl module') + def test_create_unix_server_ssl(self): + proto = MyProto(loop=self.loop) + server, path = self._make_ssl_unix_server( + lambda: proto, test_utils.ONLYCERT, test_utils.ONLYKEY) + + f_c = self.loop.create_unix_connection( + MyBaseProto, path, ssl=test_utils.dummy_ssl_context(), + server_hostname='') + + client, pr = self.loop.run_until_complete(f_c) + + client.write(b'xxx') + self.loop.run_until_complete(proto.connected) + self.assertEqual('CONNECTED', proto.state) + test_utils.run_until(self.loop, lambda: proto.nbytes > 0) + self.assertEqual(3, proto.nbytes) + + # close connection + proto.transport.close() + self.loop.run_until_complete(proto.done) + self.assertEqual('CLOSED', proto.state) + + # the client socket must be closed after to avoid ECONNRESET upon + # recv()/send() on the serving socket + client.close() + + # stop serving + server.close() + + @unittest.skipIf(ssl is None, 'No ssl module') + def test_create_server_ssl_verify_failed(self): + proto = MyProto(loop=self.loop) + server, host, port = self._make_ssl_server( + lambda: proto, test_utils.SIGNED_CERTFILE) + + sslcontext_client = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + sslcontext_client.options |= ssl.OP_NO_SSLv2 + sslcontext_client.verify_mode = ssl.CERT_REQUIRED + if hasattr(sslcontext_client, 'check_hostname'): + sslcontext_client.check_hostname = True + + + # no CA loaded + f_c = self.loop.create_connection(MyProto, host, port, + ssl=sslcontext_client) + with mock.patch.object(self.loop, 'call_exception_handler'): + with test_utils.disable_logger(): + with self.assertRaisesRegex(ssl.SSLError, + '(?i)certificate.verify.failed'): + self.loop.run_until_complete(f_c) + + # execute the loop to log the connection error + test_utils.run_briefly(self.loop) + + # close connection + self.assertIsNone(proto.transport) + server.close() + + @socket_helper.skip_unless_bind_unix_socket + @unittest.skipIf(ssl is None, 'No ssl module') + def test_create_unix_server_ssl_verify_failed(self): + proto = MyProto(loop=self.loop) + server, path = self._make_ssl_unix_server( + lambda: proto, test_utils.SIGNED_CERTFILE) + + sslcontext_client = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + sslcontext_client.options |= ssl.OP_NO_SSLv2 + sslcontext_client.verify_mode = ssl.CERT_REQUIRED + if hasattr(sslcontext_client, 'check_hostname'): + sslcontext_client.check_hostname = True + + # no CA loaded + f_c = self.loop.create_unix_connection(MyProto, path, + ssl=sslcontext_client, + server_hostname='invalid') + with mock.patch.object(self.loop, 'call_exception_handler'): + with test_utils.disable_logger(): + with self.assertRaisesRegex(ssl.SSLError, + '(?i)certificate.verify.failed'): + self.loop.run_until_complete(f_c) + + # execute the loop to log the connection error + test_utils.run_briefly(self.loop) + + # close connection + self.assertIsNone(proto.transport) + server.close() + + @unittest.skipIf(ssl is None, 'No ssl module') + def test_create_server_ssl_match_failed(self): + proto = MyProto(loop=self.loop) + server, host, port = self._make_ssl_server( + lambda: proto, test_utils.SIGNED_CERTFILE) + + sslcontext_client = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + sslcontext_client.options |= ssl.OP_NO_SSLv2 + sslcontext_client.verify_mode = ssl.CERT_REQUIRED + sslcontext_client.load_verify_locations( + cafile=test_utils.SIGNING_CA) + if hasattr(sslcontext_client, 'check_hostname'): + sslcontext_client.check_hostname = True + + # incorrect server_hostname + f_c = self.loop.create_connection(MyProto, host, port, + ssl=sslcontext_client) + + # Allow for flexible libssl error messages. + regex = re.compile(r"""( + IP address mismatch, certificate is not valid for '127.0.0.1' # OpenSSL + | + CERTIFICATE_VERIFY_FAILED # AWS-LC + )""", re.X) + with mock.patch.object(self.loop, 'call_exception_handler'): + with test_utils.disable_logger(): + with self.assertRaisesRegex(ssl.CertificateError, regex): + self.loop.run_until_complete(f_c) + + # close connection + # transport is None because TLS ALERT aborted the handshake + self.assertIsNone(proto.transport) + server.close() + + @socket_helper.skip_unless_bind_unix_socket + @unittest.skipIf(ssl is None, 'No ssl module') + def test_create_unix_server_ssl_verified(self): + proto = MyProto(loop=self.loop) + server, path = self._make_ssl_unix_server( + lambda: proto, test_utils.SIGNED_CERTFILE) + + sslcontext_client = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + sslcontext_client.options |= ssl.OP_NO_SSLv2 + sslcontext_client.verify_mode = ssl.CERT_REQUIRED + sslcontext_client.load_verify_locations(cafile=test_utils.SIGNING_CA) + if hasattr(sslcontext_client, 'check_hostname'): + sslcontext_client.check_hostname = True + + # Connection succeeds with correct CA and server hostname. + f_c = self.loop.create_unix_connection(MyProto, path, + ssl=sslcontext_client, + server_hostname='localhost') + client, pr = self.loop.run_until_complete(f_c) + self.loop.run_until_complete(proto.connected) + + # close connection + proto.transport.close() + client.close() + server.close() + self.loop.run_until_complete(proto.done) + + @unittest.expectedFailure # TODO: RUSTPYTHON; - SSL peer certificate format differs + @unittest.skipIf(ssl is None, 'No ssl module') + def test_create_server_ssl_verified(self): + proto = MyProto(loop=self.loop) + server, host, port = self._make_ssl_server( + lambda: proto, test_utils.SIGNED_CERTFILE) + + sslcontext_client = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + sslcontext_client.options |= ssl.OP_NO_SSLv2 + sslcontext_client.verify_mode = ssl.CERT_REQUIRED + sslcontext_client.load_verify_locations(cafile=test_utils.SIGNING_CA) + if hasattr(sslcontext_client, 'check_hostname'): + sslcontext_client.check_hostname = True + + # Connection succeeds with correct CA and server hostname. + f_c = self.loop.create_connection(MyProto, host, port, + ssl=sslcontext_client, + server_hostname='localhost') + client, pr = self.loop.run_until_complete(f_c) + self.loop.run_until_complete(proto.connected) + + # extra info is available + self.check_ssl_extra_info(client, peername=(host, port), + peercert=test_utils.PEERCERT) + + # close connection + proto.transport.close() + client.close() + server.close() + self.loop.run_until_complete(proto.done) + + def test_create_server_sock(self): + proto = self.loop.create_future() + + class TestMyProto(MyProto): + def connection_made(self, transport): + super().connection_made(transport) + proto.set_result(self) + + sock_ob = socket.create_server(('0.0.0.0', 0)) + + f = self.loop.create_server(TestMyProto, sock=sock_ob) + server = self.loop.run_until_complete(f) + sock = server.sockets[0] + self.assertEqual(sock.fileno(), sock_ob.fileno()) + + host, port = sock.getsockname() + self.assertEqual(host, '0.0.0.0') + client = socket.socket() + client.connect(('127.0.0.1', port)) + client.send(b'xxx') + client.close() + server.close() + + def test_create_server_addr_in_use(self): + sock_ob = socket.create_server(('0.0.0.0', 0)) + + f = self.loop.create_server(MyProto, sock=sock_ob) + server = self.loop.run_until_complete(f) + sock = server.sockets[0] + host, port = sock.getsockname() + + f = self.loop.create_server(MyProto, host=host, port=port) + with self.assertRaises(OSError) as cm: + self.loop.run_until_complete(f) + self.assertEqual(cm.exception.errno, errno.EADDRINUSE) + + server.close() + + @unittest.skipUnless(socket_helper.IPV6_ENABLED, 'IPv6 not supported or enabled') + def test_create_server_dual_stack(self): + f_proto = self.loop.create_future() + + class TestMyProto(MyProto): + def connection_made(self, transport): + super().connection_made(transport) + f_proto.set_result(self) + + try_count = 0 + while True: + try: + port = socket_helper.find_unused_port() + f = self.loop.create_server(TestMyProto, host=None, port=port) + server = self.loop.run_until_complete(f) + except OSError as ex: + if ex.errno == errno.EADDRINUSE: + try_count += 1 + self.assertGreaterEqual(5, try_count) + continue + else: + raise + else: + break + client = socket.socket() + client.connect(('127.0.0.1', port)) + client.send(b'xxx') + proto = self.loop.run_until_complete(f_proto) + proto.transport.close() + client.close() + + f_proto = self.loop.create_future() + client = socket.socket(socket.AF_INET6) + client.connect(('::1', port)) + client.send(b'xxx') + proto = self.loop.run_until_complete(f_proto) + proto.transport.close() + client.close() + + server.close() + + @socket_helper.skip_if_tcp_blackhole + def test_server_close(self): + f = self.loop.create_server(MyProto, '0.0.0.0', 0) + server = self.loop.run_until_complete(f) + sock = server.sockets[0] + host, port = sock.getsockname() + + client = socket.socket() + client.connect(('127.0.0.1', port)) + client.send(b'xxx') + client.close() + + server.close() + + client = socket.socket() + self.assertRaises( + ConnectionRefusedError, client.connect, ('127.0.0.1', port)) + client.close() + + def _test_create_datagram_endpoint(self, local_addr, family): + class TestMyDatagramProto(MyDatagramProto): + def __init__(inner_self): + super().__init__(loop=self.loop) + + def datagram_received(self, data, addr): + super().datagram_received(data, addr) + self.transport.sendto(b'resp:'+data, addr) + + coro = self.loop.create_datagram_endpoint( + TestMyDatagramProto, local_addr=local_addr, family=family) + s_transport, server = self.loop.run_until_complete(coro) + sockname = s_transport.get_extra_info('sockname') + host, port = socket.getnameinfo( + sockname, socket.NI_NUMERICHOST|socket.NI_NUMERICSERV) + + self.assertIsInstance(s_transport, asyncio.Transport) + self.assertIsInstance(server, TestMyDatagramProto) + self.assertEqual('INITIALIZED', server.state) + self.assertIs(server.transport, s_transport) + + coro = self.loop.create_datagram_endpoint( + lambda: MyDatagramProto(loop=self.loop), + remote_addr=(host, port)) + transport, client = self.loop.run_until_complete(coro) + + self.assertIsInstance(transport, asyncio.Transport) + self.assertIsInstance(client, MyDatagramProto) + self.assertEqual('INITIALIZED', client.state) + self.assertIs(client.transport, transport) + + transport.sendto(b'xxx') + test_utils.run_until(self.loop, lambda: server.nbytes) + self.assertEqual(3, server.nbytes) + test_utils.run_until(self.loop, lambda: client.nbytes) + + # received + self.assertEqual(8, client.nbytes) + + # extra info is available + self.assertIsNotNone(transport.get_extra_info('sockname')) + + # close connection + transport.close() + self.loop.run_until_complete(client.done) + self.assertEqual('CLOSED', client.state) + server.transport.close() + + def test_create_datagram_endpoint(self): + self._test_create_datagram_endpoint(('127.0.0.1', 0), socket.AF_INET) + + @unittest.skipUnless(socket_helper.IPV6_ENABLED, 'IPv6 not supported or enabled') + def test_create_datagram_endpoint_ipv6(self): + self._test_create_datagram_endpoint(('::1', 0), socket.AF_INET6) + + def test_create_datagram_endpoint_sock(self): + sock = None + local_address = ('127.0.0.1', 0) + infos = self.loop.run_until_complete( + self.loop.getaddrinfo( + *local_address, type=socket.SOCK_DGRAM)) + for family, type, proto, cname, address in infos: + try: + sock = socket.socket(family=family, type=type, proto=proto) + sock.setblocking(False) + sock.bind(address) + except: + pass + else: + break + else: + self.fail('Can not create socket.') + + f = self.loop.create_datagram_endpoint( + lambda: MyDatagramProto(loop=self.loop), sock=sock) + tr, pr = self.loop.run_until_complete(f) + self.assertIsInstance(tr, asyncio.Transport) + self.assertIsInstance(pr, MyDatagramProto) + tr.close() + self.loop.run_until_complete(pr.done) + + def test_datagram_send_to_non_listening_address(self): + # see: + # https://github.com/python/cpython/issues/91227 + # https://github.com/python/cpython/issues/88906 + # https://bugs.python.org/issue47071 + # https://bugs.python.org/issue44743 + # The Proactor event loop would fail to receive datagram messages after + # sending a message to an address that wasn't listening. + loop = self.loop + + class Protocol(asyncio.DatagramProtocol): + + _received_datagram = None + + def datagram_received(self, data, addr): + self._received_datagram.set_result(data) + + async def wait_for_datagram_received(self): + self._received_datagram = loop.create_future() + result = await asyncio.wait_for(self._received_datagram, 10) + self._received_datagram = None + return result + + def create_socket(): + sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + sock.setblocking(False) + sock.bind(('127.0.0.1', 0)) + return sock + + socket_1 = create_socket() + transport_1, protocol_1 = loop.run_until_complete( + loop.create_datagram_endpoint(Protocol, sock=socket_1) + ) + addr_1 = socket_1.getsockname() + + socket_2 = create_socket() + transport_2, protocol_2 = loop.run_until_complete( + loop.create_datagram_endpoint(Protocol, sock=socket_2) + ) + addr_2 = socket_2.getsockname() + + # creating and immediately closing this to try to get an address that + # is not listening + socket_3 = create_socket() + transport_3, protocol_3 = loop.run_until_complete( + loop.create_datagram_endpoint(Protocol, sock=socket_3) + ) + addr_3 = socket_3.getsockname() + transport_3.abort() + + transport_1.sendto(b'a', addr=addr_2) + self.assertEqual(loop.run_until_complete( + protocol_2.wait_for_datagram_received() + ), b'a') + + transport_2.sendto(b'b', addr=addr_1) + self.assertEqual(loop.run_until_complete( + protocol_1.wait_for_datagram_received() + ), b'b') + + # this should send to an address that isn't listening + transport_1.sendto(b'c', addr=addr_3) + loop.run_until_complete(asyncio.sleep(0)) + + # transport 1 should still be able to receive messages after sending to + # an address that wasn't listening + transport_2.sendto(b'd', addr=addr_1) + self.assertEqual(loop.run_until_complete( + protocol_1.wait_for_datagram_received() + ), b'd') + + transport_1.close() + transport_2.close() + + def test_internal_fds(self): + loop = self.create_event_loop() + if not isinstance(loop, selector_events.BaseSelectorEventLoop): + loop.close() + self.skipTest('loop is not a BaseSelectorEventLoop') + + self.assertEqual(1, loop._internal_fds) + loop.close() + self.assertEqual(0, loop._internal_fds) + self.assertIsNone(loop._csock) + self.assertIsNone(loop._ssock) + + @unittest.skipUnless(sys.platform != 'win32', + "Don't support pipes for Windows") + def test_read_pipe(self): + proto = MyReadPipeProto(loop=self.loop) + + rpipe, wpipe = os.pipe() + pipeobj = io.open(rpipe, 'rb', 1024) + + async def connect(): + t, p = await self.loop.connect_read_pipe( + lambda: proto, pipeobj) + self.assertIs(p, proto) + self.assertIs(t, proto.transport) + self.assertEqual(['INITIAL', 'CONNECTED'], proto.state) + self.assertEqual(0, proto.nbytes) + + self.loop.run_until_complete(connect()) + + os.write(wpipe, b'1') + test_utils.run_until(self.loop, lambda: proto.nbytes >= 1) + self.assertEqual(1, proto.nbytes) + + os.write(wpipe, b'2345') + test_utils.run_until(self.loop, lambda: proto.nbytes >= 5) + self.assertEqual(['INITIAL', 'CONNECTED'], proto.state) + self.assertEqual(5, proto.nbytes) + + os.close(wpipe) + self.loop.run_until_complete(proto.done) + self.assertEqual( + ['INITIAL', 'CONNECTED', 'EOF', 'CLOSED'], proto.state) + # extra info is available + self.assertIsNotNone(proto.transport.get_extra_info('pipe')) + + @unittest.skipUnless(sys.platform != 'win32', + "Don't support pipes for Windows") + def test_unclosed_pipe_transport(self): + # This test reproduces the issue #314 on GitHub + loop = self.create_event_loop() + read_proto = MyReadPipeProto(loop=loop) + write_proto = MyWritePipeProto(loop=loop) + + rpipe, wpipe = os.pipe() + rpipeobj = io.open(rpipe, 'rb', 1024) + wpipeobj = io.open(wpipe, 'w', 1024, encoding="utf-8") + + async def connect(): + read_transport, _ = await loop.connect_read_pipe( + lambda: read_proto, rpipeobj) + write_transport, _ = await loop.connect_write_pipe( + lambda: write_proto, wpipeobj) + return read_transport, write_transport + + # Run and close the loop without closing the transports + read_transport, write_transport = loop.run_until_complete(connect()) + loop.close() + + # These 'repr' calls used to raise an AttributeError + # See Issue #314 on GitHub + self.assertIn('open', repr(read_transport)) + self.assertIn('open', repr(write_transport)) + + # Clean up (avoid ResourceWarning) + rpipeobj.close() + wpipeobj.close() + read_transport._pipe = None + write_transport._pipe = None + + @unittest.skipUnless(sys.platform != 'win32', + "Don't support pipes for Windows") + @unittest.skipUnless(hasattr(os, 'openpty'), 'need os.openpty()') + def test_read_pty_output(self): + proto = MyReadPipeProto(loop=self.loop) + + master, slave = os.openpty() + master_read_obj = io.open(master, 'rb', 0) + + async def connect(): + t, p = await self.loop.connect_read_pipe(lambda: proto, + master_read_obj) + self.assertIs(p, proto) + self.assertIs(t, proto.transport) + self.assertEqual(['INITIAL', 'CONNECTED'], proto.state) + self.assertEqual(0, proto.nbytes) + + self.loop.run_until_complete(connect()) + + os.write(slave, b'1') + test_utils.run_until(self.loop, lambda: proto.nbytes) + self.assertEqual(1, proto.nbytes) + + os.write(slave, b'2345') + test_utils.run_until(self.loop, lambda: proto.nbytes >= 5) + self.assertEqual(['INITIAL', 'CONNECTED'], proto.state) + self.assertEqual(5, proto.nbytes) + + os.close(slave) + proto.transport.close() + self.loop.run_until_complete(proto.done) + self.assertEqual( + ['INITIAL', 'CONNECTED', 'EOF', 'CLOSED'], proto.state) + # extra info is available + self.assertIsNotNone(proto.transport.get_extra_info('pipe')) + + @unittest.skipUnless(sys.platform != 'win32', + "Don't support pipes for Windows") + def test_write_pipe(self): + rpipe, wpipe = os.pipe() + pipeobj = io.open(wpipe, 'wb', 1024) + + proto = MyWritePipeProto(loop=self.loop) + connect = self.loop.connect_write_pipe(lambda: proto, pipeobj) + transport, p = self.loop.run_until_complete(connect) + self.assertIs(p, proto) + self.assertIs(transport, proto.transport) + self.assertEqual('CONNECTED', proto.state) + + transport.write(b'1') + + data = bytearray() + def reader(data): + chunk = os.read(rpipe, 1024) + data += chunk + return len(data) + + test_utils.run_until(self.loop, lambda: reader(data) >= 1) + self.assertEqual(b'1', data) + + transport.write(b'2345') + test_utils.run_until(self.loop, lambda: reader(data) >= 5) + self.assertEqual(b'12345', data) + self.assertEqual('CONNECTED', proto.state) + + os.close(rpipe) + + # extra info is available + self.assertIsNotNone(proto.transport.get_extra_info('pipe')) + + # close connection + proto.transport.close() + self.loop.run_until_complete(proto.done) + self.assertEqual('CLOSED', proto.state) + + @unittest.skipUnless(sys.platform != 'win32', + "Don't support pipes for Windows") + def test_write_pipe_disconnect_on_close(self): + rsock, wsock = socket.socketpair() + rsock.setblocking(False) + pipeobj = io.open(wsock.detach(), 'wb', 1024) + + proto = MyWritePipeProto(loop=self.loop) + connect = self.loop.connect_write_pipe(lambda: proto, pipeobj) + transport, p = self.loop.run_until_complete(connect) + self.assertIs(p, proto) + self.assertIs(transport, proto.transport) + self.assertEqual('CONNECTED', proto.state) + + transport.write(b'1') + data = self.loop.run_until_complete(self.loop.sock_recv(rsock, 1024)) + self.assertEqual(b'1', data) + + rsock.close() + + self.loop.run_until_complete(proto.done) + self.assertEqual('CLOSED', proto.state) + + @unittest.skipUnless(sys.platform != 'win32', + "Don't support pipes for Windows") + @unittest.skipUnless(hasattr(os, 'openpty'), 'need os.openpty()') + # select, poll and kqueue don't support character devices (PTY) on Mac OS X + # older than 10.6 (Snow Leopard) + @support.requires_mac_ver(10, 6) + def test_write_pty(self): + master, slave = os.openpty() + slave_write_obj = io.open(slave, 'wb', 0) + + proto = MyWritePipeProto(loop=self.loop) + connect = self.loop.connect_write_pipe(lambda: proto, slave_write_obj) + transport, p = self.loop.run_until_complete(connect) + self.assertIs(p, proto) + self.assertIs(transport, proto.transport) + self.assertEqual('CONNECTED', proto.state) + + transport.write(b'1') + + data = bytearray() + def reader(data): + chunk = os.read(master, 1024) + data += chunk + return len(data) + + test_utils.run_until(self.loop, lambda: reader(data) >= 1, + timeout=support.SHORT_TIMEOUT) + self.assertEqual(b'1', data) + + transport.write(b'2345') + test_utils.run_until(self.loop, lambda: reader(data) >= 5, + timeout=support.SHORT_TIMEOUT) + self.assertEqual(b'12345', data) + self.assertEqual('CONNECTED', proto.state) + + os.close(master) + + # extra info is available + self.assertIsNotNone(proto.transport.get_extra_info('pipe')) + + # close connection + proto.transport.close() + self.loop.run_until_complete(proto.done) + self.assertEqual('CLOSED', proto.state) + + @unittest.skipUnless(sys.platform != 'win32', + "Don't support pipes for Windows") + @unittest.skipUnless(hasattr(os, 'openpty'), 'need os.openpty()') + # select, poll and kqueue don't support character devices (PTY) on Mac OS X + # older than 10.6 (Snow Leopard) + @support.requires_mac_ver(10, 6) + def test_bidirectional_pty(self): + master, read_slave = os.openpty() + write_slave = os.dup(read_slave) + tty.setraw(read_slave) + + slave_read_obj = io.open(read_slave, 'rb', 0) + read_proto = MyReadPipeProto(loop=self.loop) + read_connect = self.loop.connect_read_pipe(lambda: read_proto, + slave_read_obj) + read_transport, p = self.loop.run_until_complete(read_connect) + self.assertIs(p, read_proto) + self.assertIs(read_transport, read_proto.transport) + self.assertEqual(['INITIAL', 'CONNECTED'], read_proto.state) + self.assertEqual(0, read_proto.nbytes) + + + slave_write_obj = io.open(write_slave, 'wb', 0) + write_proto = MyWritePipeProto(loop=self.loop) + write_connect = self.loop.connect_write_pipe(lambda: write_proto, + slave_write_obj) + write_transport, p = self.loop.run_until_complete(write_connect) + self.assertIs(p, write_proto) + self.assertIs(write_transport, write_proto.transport) + self.assertEqual('CONNECTED', write_proto.state) + + data = bytearray() + def reader(data): + chunk = os.read(master, 1024) + data += chunk + return len(data) + + write_transport.write(b'1') + test_utils.run_until(self.loop, lambda: reader(data) >= 1, + timeout=support.SHORT_TIMEOUT) + self.assertEqual(b'1', data) + self.assertEqual(['INITIAL', 'CONNECTED'], read_proto.state) + self.assertEqual('CONNECTED', write_proto.state) + + os.write(master, b'a') + test_utils.run_until(self.loop, lambda: read_proto.nbytes >= 1, + timeout=support.SHORT_TIMEOUT) + self.assertEqual(['INITIAL', 'CONNECTED'], read_proto.state) + self.assertEqual(1, read_proto.nbytes) + self.assertEqual('CONNECTED', write_proto.state) + + write_transport.write(b'2345') + test_utils.run_until(self.loop, lambda: reader(data) >= 5, + timeout=support.SHORT_TIMEOUT) + self.assertEqual(b'12345', data) + self.assertEqual(['INITIAL', 'CONNECTED'], read_proto.state) + self.assertEqual('CONNECTED', write_proto.state) + + os.write(master, b'bcde') + test_utils.run_until(self.loop, lambda: read_proto.nbytes >= 5, + timeout=support.SHORT_TIMEOUT) + self.assertEqual(['INITIAL', 'CONNECTED'], read_proto.state) + self.assertEqual(5, read_proto.nbytes) + self.assertEqual('CONNECTED', write_proto.state) + + os.close(master) + + read_transport.close() + self.loop.run_until_complete(read_proto.done) + self.assertEqual( + ['INITIAL', 'CONNECTED', 'EOF', 'CLOSED'], read_proto.state) + + write_transport.close() + self.loop.run_until_complete(write_proto.done) + self.assertEqual('CLOSED', write_proto.state) + + def test_prompt_cancellation(self): + r, w = socket.socketpair() + r.setblocking(False) + f = self.loop.create_task(self.loop.sock_recv(r, 1)) + ov = getattr(f, 'ov', None) + if ov is not None: + self.assertTrue(ov.pending) + + async def main(): + try: + self.loop.call_soon(f.cancel) + await f + except asyncio.CancelledError: + res = 'cancelled' + else: + res = None + finally: + self.loop.stop() + return res + + t = self.loop.create_task(main()) + self.loop.run_forever() + + self.assertEqual(t.result(), 'cancelled') + self.assertRaises(asyncio.CancelledError, f.result) + if ov is not None: + self.assertFalse(ov.pending) + self.loop._stop_serving(r) + + r.close() + w.close() + + def test_timeout_rounding(self): + def _run_once(): + self.loop._run_once_counter += 1 + orig_run_once() + + orig_run_once = self.loop._run_once + self.loop._run_once_counter = 0 + self.loop._run_once = _run_once + + async def wait(): + await asyncio.sleep(1e-2) + await asyncio.sleep(1e-4) + await asyncio.sleep(1e-6) + await asyncio.sleep(1e-8) + await asyncio.sleep(1e-10) + + self.loop.run_until_complete(wait()) + # The ideal number of call is 12, but on some platforms, the selector + # may sleep at little bit less than timeout depending on the resolution + # of the clock used by the kernel. Tolerate a few useless calls on + # these platforms. + self.assertLessEqual(self.loop._run_once_counter, 20, + {'clock_resolution': self.loop._clock_resolution, + 'selector': self.loop._selector.__class__.__name__}) + + def test_remove_fds_after_closing(self): + loop = self.create_event_loop() + callback = lambda: None + r, w = socket.socketpair() + self.addCleanup(r.close) + self.addCleanup(w.close) + loop.add_reader(r, callback) + loop.add_writer(w, callback) + loop.close() + self.assertFalse(loop.remove_reader(r)) + self.assertFalse(loop.remove_writer(w)) + + def test_add_fds_after_closing(self): + loop = self.create_event_loop() + callback = lambda: None + r, w = socket.socketpair() + self.addCleanup(r.close) + self.addCleanup(w.close) + loop.close() + with self.assertRaises(RuntimeError): + loop.add_reader(r, callback) + with self.assertRaises(RuntimeError): + loop.add_writer(w, callback) + + def test_close_running_event_loop(self): + async def close_loop(loop): + self.loop.close() + + coro = close_loop(self.loop) + with self.assertRaises(RuntimeError): + self.loop.run_until_complete(coro) + + def test_close(self): + self.loop.close() + + async def test(): + pass + + func = lambda: False + coro = test() + self.addCleanup(coro.close) + + # operation blocked when the loop is closed + with self.assertRaises(RuntimeError): + self.loop.run_forever() + with self.assertRaises(RuntimeError): + fut = self.loop.create_future() + self.loop.run_until_complete(fut) + with self.assertRaises(RuntimeError): + self.loop.call_soon(func) + with self.assertRaises(RuntimeError): + self.loop.call_soon_threadsafe(func) + with self.assertRaises(RuntimeError): + self.loop.call_later(1.0, func) + with self.assertRaises(RuntimeError): + self.loop.call_at(self.loop.time() + .0, func) + with self.assertRaises(RuntimeError): + self.loop.create_task(coro) + with self.assertRaises(RuntimeError): + self.loop.add_signal_handler(signal.SIGTERM, func) + + # run_in_executor test is tricky: the method is a coroutine, + # but run_until_complete cannot be called on closed loop. + # Thus iterate once explicitly. + with self.assertRaises(RuntimeError): + it = self.loop.run_in_executor(None, func).__await__() + next(it) + + +class SubprocessTestsMixin: + + def check_terminated(self, returncode): + if sys.platform == 'win32': + self.assertIsInstance(returncode, int) + # expect 1 but sometimes get 0 + else: + self.assertEqual(-signal.SIGTERM, returncode) + + def check_killed(self, returncode): + if sys.platform == 'win32': + self.assertIsInstance(returncode, int) + # expect 1 but sometimes get 0 + else: + self.assertEqual(-signal.SIGKILL, returncode) + + @support.requires_subprocess() + def test_subprocess_exec(self): + prog = os.path.join(os.path.dirname(__file__), 'echo.py') + + connect = self.loop.subprocess_exec( + functools.partial(MySubprocessProtocol, self.loop), + sys.executable, prog) + + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) + self.loop.run_until_complete(proto.connected) + self.assertEqual('CONNECTED', proto.state) + + stdin = transp.get_pipe_transport(0) + stdin.write(b'Python The Winner') + self.loop.run_until_complete(proto.got_data[1].wait()) + with test_utils.disable_logger(): + transp.close() + self.loop.run_until_complete(proto.completed) + self.check_killed(proto.returncode) + self.assertEqual(b'Python The Winner', proto.data[1]) + + @support.requires_subprocess() + def test_subprocess_interactive(self): + prog = os.path.join(os.path.dirname(__file__), 'echo.py') + + connect = self.loop.subprocess_exec( + functools.partial(MySubprocessProtocol, self.loop), + sys.executable, prog) + + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) + self.loop.run_until_complete(proto.connected) + self.assertEqual('CONNECTED', proto.state) + + stdin = transp.get_pipe_transport(0) + stdin.write(b'Python ') + self.loop.run_until_complete(proto.got_data[1].wait()) + proto.got_data[1].clear() + self.assertEqual(b'Python ', proto.data[1]) + + stdin.write(b'The Winner') + self.loop.run_until_complete(proto.got_data[1].wait()) + self.assertEqual(b'Python The Winner', proto.data[1]) + + with test_utils.disable_logger(): + transp.close() + self.loop.run_until_complete(proto.completed) + self.check_killed(proto.returncode) + + @support.requires_subprocess() + def test_subprocess_shell(self): + connect = self.loop.subprocess_shell( + functools.partial(MySubprocessProtocol, self.loop), + 'echo Python') + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) + self.loop.run_until_complete(proto.connected) + + transp.get_pipe_transport(0).close() + self.loop.run_until_complete(proto.completed) + self.assertEqual(0, proto.returncode) + self.assertTrue(all(f.done() for f in proto.disconnects.values())) + self.assertEqual(proto.data[1].rstrip(b'\r\n'), b'Python') + self.assertEqual(proto.data[2], b'') + transp.close() + + @support.requires_subprocess() + def test_subprocess_exitcode(self): + connect = self.loop.subprocess_shell( + functools.partial(MySubprocessProtocol, self.loop), + 'exit 7', stdin=None, stdout=None, stderr=None) + + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) + self.loop.run_until_complete(proto.completed) + self.assertEqual(7, proto.returncode) + transp.close() + + @support.requires_subprocess() + def test_subprocess_close_after_finish(self): + connect = self.loop.subprocess_shell( + functools.partial(MySubprocessProtocol, self.loop), + 'exit 7', stdin=None, stdout=None, stderr=None) + + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) + self.assertIsNone(transp.get_pipe_transport(0)) + self.assertIsNone(transp.get_pipe_transport(1)) + self.assertIsNone(transp.get_pipe_transport(2)) + self.loop.run_until_complete(proto.completed) + self.assertEqual(7, proto.returncode) + self.assertIsNone(transp.close()) + + @support.requires_subprocess() + def test_subprocess_kill(self): + prog = os.path.join(os.path.dirname(__file__), 'echo.py') + + connect = self.loop.subprocess_exec( + functools.partial(MySubprocessProtocol, self.loop), + sys.executable, prog) + + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) + self.loop.run_until_complete(proto.connected) + + transp.kill() + self.loop.run_until_complete(proto.completed) + self.check_killed(proto.returncode) + transp.close() + + @support.requires_subprocess() + def test_subprocess_terminate(self): + prog = os.path.join(os.path.dirname(__file__), 'echo.py') + + connect = self.loop.subprocess_exec( + functools.partial(MySubprocessProtocol, self.loop), + sys.executable, prog) + + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) + self.loop.run_until_complete(proto.connected) + + transp.terminate() + self.loop.run_until_complete(proto.completed) + self.check_terminated(proto.returncode) + transp.close() + + @unittest.skipIf(sys.platform == 'win32', "Don't have SIGHUP") + @support.requires_subprocess() + def test_subprocess_send_signal(self): + # bpo-31034: Make sure that we get the default signal handler (killing + # the process). The parent process may have decided to ignore SIGHUP, + # and signal handlers are inherited. + old_handler = signal.signal(signal.SIGHUP, signal.SIG_DFL) + try: + prog = os.path.join(os.path.dirname(__file__), 'echo.py') + + connect = self.loop.subprocess_exec( + functools.partial(MySubprocessProtocol, self.loop), + sys.executable, prog) + + + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) + self.loop.run_until_complete(proto.connected) + + transp.send_signal(signal.SIGHUP) + self.loop.run_until_complete(proto.completed) + self.assertEqual(-signal.SIGHUP, proto.returncode) + transp.close() + finally: + signal.signal(signal.SIGHUP, old_handler) + + @support.requires_subprocess() + def test_subprocess_stderr(self): + prog = os.path.join(os.path.dirname(__file__), 'echo2.py') + + connect = self.loop.subprocess_exec( + functools.partial(MySubprocessProtocol, self.loop), + sys.executable, prog) + + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) + self.loop.run_until_complete(proto.connected) + + stdin = transp.get_pipe_transport(0) + stdin.write(b'test') + + self.loop.run_until_complete(proto.completed) + + transp.close() + self.assertEqual(b'OUT:test', proto.data[1]) + self.assertStartsWith(proto.data[2], b'ERR:test') + self.assertEqual(0, proto.returncode) + + @support.requires_subprocess() + def test_subprocess_stderr_redirect_to_stdout(self): + prog = os.path.join(os.path.dirname(__file__), 'echo2.py') + + connect = self.loop.subprocess_exec( + functools.partial(MySubprocessProtocol, self.loop), + sys.executable, prog, stderr=subprocess.STDOUT) + + + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) + self.loop.run_until_complete(proto.connected) + + stdin = transp.get_pipe_transport(0) + self.assertIsNotNone(transp.get_pipe_transport(1)) + self.assertIsNone(transp.get_pipe_transport(2)) + + stdin.write(b'test') + self.loop.run_until_complete(proto.completed) + self.assertStartsWith(proto.data[1], b'OUT:testERR:test') + self.assertEqual(b'', proto.data[2]) + + transp.close() + self.assertEqual(0, proto.returncode) + + @support.requires_subprocess() + def test_subprocess_close_client_stream(self): + prog = os.path.join(os.path.dirname(__file__), 'echo3.py') + + connect = self.loop.subprocess_exec( + functools.partial(MySubprocessProtocol, self.loop), + sys.executable, prog) + + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) + self.loop.run_until_complete(proto.connected) + + stdin = transp.get_pipe_transport(0) + stdout = transp.get_pipe_transport(1) + stdin.write(b'test') + self.loop.run_until_complete(proto.got_data[1].wait()) + self.assertEqual(b'OUT:test', proto.data[1]) + + stdout.close() + self.loop.run_until_complete(proto.disconnects[1]) + stdin.write(b'xxx') + self.loop.run_until_complete(proto.got_data[2].wait()) + if sys.platform != 'win32': + self.assertEqual(b'ERR:BrokenPipeError', proto.data[2]) + else: + # After closing the read-end of a pipe, writing to the + # write-end using os.write() fails with errno==EINVAL and + # GetLastError()==ERROR_INVALID_NAME on Windows!?! (Using + # WriteFile() we get ERROR_BROKEN_PIPE as expected.) + self.assertEqual(b'ERR:OSError', proto.data[2]) + with test_utils.disable_logger(): + transp.close() + self.loop.run_until_complete(proto.completed) + self.check_killed(proto.returncode) + + @support.requires_subprocess() + def test_subprocess_wait_no_same_group(self): + # start the new process in a new session + connect = self.loop.subprocess_shell( + functools.partial(MySubprocessProtocol, self.loop), + 'exit 7', stdin=None, stdout=None, stderr=None, + start_new_session=True) + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) + self.loop.run_until_complete(proto.completed) + self.assertEqual(7, proto.returncode) + transp.close() + + @support.requires_subprocess() + def test_subprocess_exec_invalid_args(self): + async def connect(**kwds): + await self.loop.subprocess_exec( + asyncio.SubprocessProtocol, + 'pwd', **kwds) + + with self.assertRaises(ValueError): + self.loop.run_until_complete(connect(universal_newlines=True)) + with self.assertRaises(ValueError): + self.loop.run_until_complete(connect(bufsize=4096)) + with self.assertRaises(ValueError): + self.loop.run_until_complete(connect(shell=True)) + + @support.requires_subprocess() + def test_subprocess_shell_invalid_args(self): + + async def connect(cmd=None, **kwds): + if not cmd: + cmd = 'pwd' + await self.loop.subprocess_shell( + asyncio.SubprocessProtocol, + cmd, **kwds) + + with self.assertRaises(ValueError): + self.loop.run_until_complete(connect(['ls', '-l'])) + with self.assertRaises(ValueError): + self.loop.run_until_complete(connect(universal_newlines=True)) + with self.assertRaises(ValueError): + self.loop.run_until_complete(connect(bufsize=4096)) + with self.assertRaises(ValueError): + self.loop.run_until_complete(connect(shell=False)) + + +if sys.platform == 'win32': + + class SelectEventLoopTests(EventLoopTestsMixin, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.SelectorEventLoop() + + class ProactorEventLoopTests(EventLoopTestsMixin, + SubprocessTestsMixin, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.ProactorEventLoop() + + def test_reader_callback(self): + raise unittest.SkipTest("IocpEventLoop does not have add_reader()") + + def test_reader_callback_cancel(self): + raise unittest.SkipTest("IocpEventLoop does not have add_reader()") + + def test_writer_callback(self): + raise unittest.SkipTest("IocpEventLoop does not have add_writer()") + + def test_writer_callback_cancel(self): + raise unittest.SkipTest("IocpEventLoop does not have add_writer()") + + def test_remove_fds_after_closing(self): + raise unittest.SkipTest("IocpEventLoop does not have add_reader()") +else: + import selectors + + if hasattr(selectors, 'KqueueSelector'): + class KqueueEventLoopTests(EventLoopTestsMixin, + SubprocessTestsMixin, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.SelectorEventLoop( + selectors.KqueueSelector()) + + # kqueue doesn't support character devices (PTY) on Mac OS X older + # than 10.9 (Maverick) + @support.requires_mac_ver(10, 9) + # Issue #20667: KqueueEventLoopTests.test_read_pty_output() + # hangs on OpenBSD 5.5 + @unittest.skipIf(sys.platform.startswith('openbsd'), + 'test hangs on OpenBSD') + def test_read_pty_output(self): + super().test_read_pty_output() + + # kqueue doesn't support character devices (PTY) on Mac OS X older + # than 10.9 (Maverick) + @support.requires_mac_ver(10, 9) + def test_write_pty(self): + super().test_write_pty() + + if hasattr(selectors, 'EpollSelector'): + class EPollEventLoopTests(EventLoopTestsMixin, + SubprocessTestsMixin, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.SelectorEventLoop(selectors.EpollSelector()) + + if hasattr(selectors, 'PollSelector'): + class PollEventLoopTests(EventLoopTestsMixin, + SubprocessTestsMixin, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.SelectorEventLoop(selectors.PollSelector()) + + # Should always exist. + class SelectEventLoopTests(EventLoopTestsMixin, + SubprocessTestsMixin, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.SelectorEventLoop(selectors.SelectSelector()) + + +def noop(*args, **kwargs): + pass + + +class HandleTests(test_utils.TestCase): + + def setUp(self): + super().setUp() + self.loop = mock.Mock() + self.loop.get_debug.return_value = True + + def test_handle(self): + def callback(*args): + return args + + args = () + h = asyncio.Handle(callback, args, self.loop) + self.assertIs(h._callback, callback) + self.assertIs(h._args, args) + self.assertFalse(h.cancelled()) + + h.cancel() + self.assertTrue(h.cancelled()) + + def test_callback_with_exception(self): + def callback(): + raise ValueError() + + self.loop = mock.Mock() + self.loop.call_exception_handler = mock.Mock() + + h = asyncio.Handle(callback, (), self.loop) + h._run() + + self.loop.call_exception_handler.assert_called_with({ + 'message': test_utils.MockPattern('Exception in callback.*'), + 'exception': mock.ANY, + 'handle': h, + 'source_traceback': h._source_traceback, + }) + + def test_handle_weakref(self): + wd = weakref.WeakValueDictionary() + h = asyncio.Handle(lambda: None, (), self.loop) + wd['h'] = h # Would fail without __weakref__ slot. + + def test_handle_repr(self): + self.loop.get_debug.return_value = False + + # simple function + h = asyncio.Handle(noop, (1, 2), self.loop) + filename, lineno = test_utils.get_function_source(noop) + self.assertEqual(repr(h), + '' + % (filename, lineno)) + + # cancelled handle + h.cancel() + self.assertEqual(repr(h), + '') + + # decorated function + cb = types.coroutine(noop) + h = asyncio.Handle(cb, (), self.loop) + self.assertEqual(repr(h), + '' + % (filename, lineno)) + + # partial function + cb = functools.partial(noop, 1, 2) + h = asyncio.Handle(cb, (3,), self.loop) + regex = (r'^$' + % (re.escape(filename), lineno)) + self.assertRegex(repr(h), regex) + + # partial function with keyword args + cb = functools.partial(noop, x=1) + h = asyncio.Handle(cb, (2, 3), self.loop) + regex = (r'^$' + % (re.escape(filename), lineno)) + self.assertRegex(repr(h), regex) + + # partial method + method = HandleTests.test_handle_repr + cb = functools.partialmethod(method) + filename, lineno = test_utils.get_function_source(method) + h = asyncio.Handle(cb, (), self.loop) + + cb_regex = r'' + cb_regex = fr'functools.partialmethod\({cb_regex}\)\(\)' + regex = fr'^$' + self.assertRegex(repr(h), regex) + + def test_handle_repr_debug(self): + self.loop.get_debug.return_value = True + + # simple function + create_filename = __file__ + create_lineno = sys._getframe().f_lineno + 1 + h = asyncio.Handle(noop, (1, 2), self.loop) + filename, lineno = test_utils.get_function_source(noop) + self.assertEqual(repr(h), + '' + % (filename, lineno, create_filename, create_lineno)) + + # cancelled handle + h.cancel() + self.assertEqual( + repr(h), + '' + % (filename, lineno, create_filename, create_lineno)) + + # double cancellation won't overwrite _repr + h.cancel() + self.assertEqual( + repr(h), + '' + % (filename, lineno, create_filename, create_lineno)) + + # partial function + cb = functools.partial(noop, 1, 2) + create_lineno = sys._getframe().f_lineno + 1 + h = asyncio.Handle(cb, (3,), self.loop) + regex = (r'^$' + % (re.escape(filename), lineno, + re.escape(create_filename), create_lineno)) + self.assertRegex(repr(h), regex) + + # partial function with keyword args + cb = functools.partial(noop, x=1) + create_lineno = sys._getframe().f_lineno + 1 + h = asyncio.Handle(cb, (2, 3), self.loop) + regex = (r'^$' + % (re.escape(filename), lineno, + re.escape(create_filename), create_lineno)) + self.assertRegex(repr(h), regex) + + def test_handle_source_traceback(self): + loop = asyncio.new_event_loop() + loop.set_debug(True) + self.set_event_loop(loop) + + def check_source_traceback(h): + lineno = sys._getframe(1).f_lineno - 1 + self.assertIsInstance(h._source_traceback, list) + self.assertEqual(h._source_traceback[-1][:3], + (__file__, + lineno, + 'test_handle_source_traceback')) + + # call_soon + h = loop.call_soon(noop) + check_source_traceback(h) + + # call_soon_threadsafe + h = loop.call_soon_threadsafe(noop) + check_source_traceback(h) + + # call_later + h = loop.call_later(0, noop) + check_source_traceback(h) + + # call_at + h = loop.call_later(0, noop) + check_source_traceback(h) + + def test_coroutine_like_object_debug_formatting(self): + # Test that asyncio can format coroutines that are instances of + # collections.abc.Coroutine, but lack cr_core or gi_code attributes + # (such as ones compiled with Cython). + + coro = CoroLike() + coro.__name__ = 'AAA' + self.assertTrue(asyncio.iscoroutine(coro)) + self.assertEqual(coroutines._format_coroutine(coro), 'AAA()') + + coro.__qualname__ = 'BBB' + self.assertEqual(coroutines._format_coroutine(coro), 'BBB()') + + coro.cr_running = True + self.assertEqual(coroutines._format_coroutine(coro), 'BBB() running') + + coro.__name__ = coro.__qualname__ = None + self.assertEqual(coroutines._format_coroutine(coro), + '() running') + + coro = CoroLike() + coro.__qualname__ = 'CoroLike' + # Some coroutines might not have '__name__', such as + # built-in async_gen.asend(). + self.assertEqual(coroutines._format_coroutine(coro), 'CoroLike()') + + coro = CoroLike() + coro.__qualname__ = 'AAA' + coro.cr_code = None + self.assertEqual(coroutines._format_coroutine(coro), 'AAA()') + + +class TimerTests(unittest.TestCase): + + def setUp(self): + super().setUp() + self.loop = mock.Mock() + + def test_hash(self): + when = time.monotonic() + h = asyncio.TimerHandle(when, lambda: False, (), + mock.Mock()) + self.assertEqual(hash(h), hash(when)) + + def test_when(self): + when = time.monotonic() + h = asyncio.TimerHandle(when, lambda: False, (), + mock.Mock()) + self.assertEqual(when, h.when()) + + def test_timer(self): + def callback(*args): + return args + + args = (1, 2, 3) + when = time.monotonic() + h = asyncio.TimerHandle(when, callback, args, mock.Mock()) + self.assertIs(h._callback, callback) + self.assertIs(h._args, args) + self.assertFalse(h.cancelled()) + + # cancel + h.cancel() + self.assertTrue(h.cancelled()) + self.assertIsNone(h._callback) + self.assertIsNone(h._args) + + + def test_timer_repr(self): + self.loop.get_debug.return_value = False + + # simple function + h = asyncio.TimerHandle(123, noop, (), self.loop) + src = test_utils.get_function_source(noop) + self.assertEqual(repr(h), + '' % src) + + # cancelled handle + h.cancel() + self.assertEqual(repr(h), + '') + + def test_timer_repr_debug(self): + self.loop.get_debug.return_value = True + + # simple function + create_filename = __file__ + create_lineno = sys._getframe().f_lineno + 1 + h = asyncio.TimerHandle(123, noop, (), self.loop) + filename, lineno = test_utils.get_function_source(noop) + self.assertEqual(repr(h), + '' + % (filename, lineno, create_filename, create_lineno)) + + # cancelled handle + h.cancel() + self.assertEqual(repr(h), + '' + % (filename, lineno, create_filename, create_lineno)) + + + def test_timer_comparison(self): + def callback(*args): + return args + + when = time.monotonic() + + h1 = asyncio.TimerHandle(when, callback, (), self.loop) + h2 = asyncio.TimerHandle(when, callback, (), self.loop) + with self.assertRaises(AssertionError): + self.assertLess(h1, h2) + with self.assertRaises(AssertionError): + self.assertLess(h2, h1) + with self.assertRaises(AssertionError): + self.assertGreater(h1, h2) + with self.assertRaises(AssertionError): + self.assertGreater(h2, h1) + with self.assertRaises(AssertionError): + self.assertNotEqual(h1, h2) + + self.assertLessEqual(h1, h2) + self.assertLessEqual(h2, h1) + self.assertGreaterEqual(h1, h2) + self.assertGreaterEqual(h2, h1) + self.assertEqual(h1, h2) + + h2.cancel() + with self.assertRaises(AssertionError): + self.assertEqual(h1, h2) + self.assertNotEqual(h1, h2) + + h1 = asyncio.TimerHandle(when, callback, (), self.loop) + h2 = asyncio.TimerHandle(when + 10.0, callback, (), self.loop) + with self.assertRaises(AssertionError): + self.assertLess(h2, h1) + with self.assertRaises(AssertionError): + self.assertLessEqual(h2, h1) + with self.assertRaises(AssertionError): + self.assertGreater(h1, h2) + with self.assertRaises(AssertionError): + self.assertGreaterEqual(h1, h2) + with self.assertRaises(AssertionError): + self.assertEqual(h1, h2) + + self.assertLess(h1, h2) + self.assertGreater(h2, h1) + self.assertLessEqual(h1, h2) + self.assertGreaterEqual(h2, h1) + self.assertNotEqual(h1, h2) + + h3 = asyncio.Handle(callback, (), self.loop) + self.assertIs(NotImplemented, h1.__eq__(h3)) + self.assertIs(NotImplemented, h1.__ne__(h3)) + + with self.assertRaises(TypeError): + h1 < () + with self.assertRaises(TypeError): + h1 > () + with self.assertRaises(TypeError): + h1 <= () + with self.assertRaises(TypeError): + h1 >= () + with self.assertRaises(AssertionError): + self.assertEqual(h1, ()) + with self.assertRaises(AssertionError): + self.assertNotEqual(h1, ALWAYS_EQ) + with self.assertRaises(AssertionError): + self.assertGreater(h1, LARGEST) + with self.assertRaises(AssertionError): + self.assertGreaterEqual(h1, LARGEST) + with self.assertRaises(AssertionError): + self.assertLess(h1, SMALLEST) + with self.assertRaises(AssertionError): + self.assertLessEqual(h1, SMALLEST) + + self.assertNotEqual(h1, ()) + self.assertEqual(h1, ALWAYS_EQ) + self.assertLess(h1, LARGEST) + self.assertLessEqual(h1, LARGEST) + self.assertGreaterEqual(h1, SMALLEST) + self.assertGreater(h1, SMALLEST) + + +class AbstractEventLoopTests(unittest.TestCase): + + def test_not_implemented(self): + f = mock.Mock() + loop = asyncio.AbstractEventLoop() + self.assertRaises( + NotImplementedError, loop.run_forever) + self.assertRaises( + NotImplementedError, loop.run_until_complete, None) + self.assertRaises( + NotImplementedError, loop.stop) + self.assertRaises( + NotImplementedError, loop.is_running) + self.assertRaises( + NotImplementedError, loop.is_closed) + self.assertRaises( + NotImplementedError, loop.close) + self.assertRaises( + NotImplementedError, loop.create_task, None) + self.assertRaises( + NotImplementedError, loop.call_later, None, None) + self.assertRaises( + NotImplementedError, loop.call_at, f, f) + self.assertRaises( + NotImplementedError, loop.call_soon, None) + self.assertRaises( + NotImplementedError, loop.time) + self.assertRaises( + NotImplementedError, loop.call_soon_threadsafe, None) + self.assertRaises( + NotImplementedError, loop.set_default_executor, f) + self.assertRaises( + NotImplementedError, loop.add_reader, 1, f) + self.assertRaises( + NotImplementedError, loop.remove_reader, 1) + self.assertRaises( + NotImplementedError, loop.add_writer, 1, f) + self.assertRaises( + NotImplementedError, loop.remove_writer, 1) + self.assertRaises( + NotImplementedError, loop.add_signal_handler, 1, f) + self.assertRaises( + NotImplementedError, loop.remove_signal_handler, 1) + self.assertRaises( + NotImplementedError, loop.remove_signal_handler, 1) + self.assertRaises( + NotImplementedError, loop.set_exception_handler, f) + self.assertRaises( + NotImplementedError, loop.default_exception_handler, f) + self.assertRaises( + NotImplementedError, loop.call_exception_handler, f) + self.assertRaises( + NotImplementedError, loop.get_debug) + self.assertRaises( + NotImplementedError, loop.set_debug, f) + + def test_not_implemented_async(self): + + async def inner(): + f = mock.Mock() + loop = asyncio.AbstractEventLoop() + + with self.assertRaises(NotImplementedError): + await loop.run_in_executor(f, f) + with self.assertRaises(NotImplementedError): + await loop.getaddrinfo('localhost', 8080) + with self.assertRaises(NotImplementedError): + await loop.getnameinfo(('localhost', 8080)) + with self.assertRaises(NotImplementedError): + await loop.create_connection(f) + with self.assertRaises(NotImplementedError): + await loop.create_server(f) + with self.assertRaises(NotImplementedError): + await loop.create_datagram_endpoint(f) + with self.assertRaises(NotImplementedError): + await loop.sock_recv(f, 10) + with self.assertRaises(NotImplementedError): + await loop.sock_recv_into(f, 10) + with self.assertRaises(NotImplementedError): + await loop.sock_sendall(f, 10) + with self.assertRaises(NotImplementedError): + await loop.sock_connect(f, f) + with self.assertRaises(NotImplementedError): + await loop.sock_accept(f) + with self.assertRaises(NotImplementedError): + await loop.sock_sendfile(f, f) + with self.assertRaises(NotImplementedError): + await loop.sendfile(f, f) + with self.assertRaises(NotImplementedError): + await loop.connect_read_pipe(f, mock.sentinel.pipe) + with self.assertRaises(NotImplementedError): + await loop.connect_write_pipe(f, mock.sentinel.pipe) + with self.assertRaises(NotImplementedError): + await loop.subprocess_shell(f, mock.sentinel) + with self.assertRaises(NotImplementedError): + await loop.subprocess_exec(f) + + loop = asyncio.new_event_loop() + loop.run_until_complete(inner()) + loop.close() + + +class PolicyTests(unittest.TestCase): + + def test_abstract_event_loop_policy_deprecation(self): + with self.assertWarnsRegex( + DeprecationWarning, "'asyncio.AbstractEventLoopPolicy' is deprecated"): + policy = asyncio.AbstractEventLoopPolicy() + self.assertIsInstance(policy, asyncio.AbstractEventLoopPolicy) + + def test_default_event_loop_policy_deprecation(self): + with self.assertWarnsRegex( + DeprecationWarning, "'asyncio.DefaultEventLoopPolicy' is deprecated"): + policy = asyncio.DefaultEventLoopPolicy() + self.assertIsInstance(policy, asyncio.DefaultEventLoopPolicy) + + def test_event_loop_policy(self): + policy = asyncio.events._AbstractEventLoopPolicy() + self.assertRaises(NotImplementedError, policy.get_event_loop) + self.assertRaises(NotImplementedError, policy.set_event_loop, object()) + self.assertRaises(NotImplementedError, policy.new_event_loop) + + def test_get_event_loop(self): + policy = test_utils.DefaultEventLoopPolicy() + self.assertIsNone(policy._local._loop) + + with self.assertRaises(RuntimeError): + loop = policy.get_event_loop() + self.assertIsNone(policy._local._loop) + + def test_get_event_loop_does_not_call_set_event_loop(self): + policy = test_utils.DefaultEventLoopPolicy() + + with mock.patch.object( + policy, "set_event_loop", + wraps=policy.set_event_loop) as m_set_event_loop: + + with self.assertRaises(RuntimeError): + loop = policy.get_event_loop() + + m_set_event_loop.assert_not_called() + + def test_get_event_loop_after_set_none(self): + policy = test_utils.DefaultEventLoopPolicy() + policy.set_event_loop(None) + self.assertRaises(RuntimeError, policy.get_event_loop) + + @unittest.expectedFailure # TODO: RUSTPYTHON; - mock.patch doesn't work correctly with threading.current_thread + @mock.patch('asyncio.events.threading.current_thread') + def test_get_event_loop_thread(self, m_current_thread): + + def f(): + policy = test_utils.DefaultEventLoopPolicy() + self.assertRaises(RuntimeError, policy.get_event_loop) + + th = threading.Thread(target=f) + th.start() + th.join() + + def test_new_event_loop(self): + policy = test_utils.DefaultEventLoopPolicy() + + loop = policy.new_event_loop() + self.assertIsInstance(loop, asyncio.AbstractEventLoop) + loop.close() + + def test_set_event_loop(self): + policy = test_utils.DefaultEventLoopPolicy() + old_loop = policy.new_event_loop() + policy.set_event_loop(old_loop) + + self.assertRaises(TypeError, policy.set_event_loop, object()) + + loop = policy.new_event_loop() + policy.set_event_loop(loop) + self.assertIs(loop, policy.get_event_loop()) + self.assertIsNot(old_loop, policy.get_event_loop()) + loop.close() + old_loop.close() + + def test_get_event_loop_policy(self): + with self.assertWarnsRegex( + DeprecationWarning, "'asyncio.get_event_loop_policy' is deprecated"): + policy = asyncio.get_event_loop_policy() + self.assertIsInstance(policy, asyncio.events._AbstractEventLoopPolicy) + self.assertIs(policy, asyncio.get_event_loop_policy()) + + def test_set_event_loop_policy(self): + with self.assertWarnsRegex( + DeprecationWarning, "'asyncio.set_event_loop_policy' is deprecated"): + self.assertRaises( + TypeError, asyncio.set_event_loop_policy, object()) + + with self.assertWarnsRegex( + DeprecationWarning, "'asyncio.get_event_loop_policy' is deprecated"): + old_policy = asyncio.get_event_loop_policy() + + policy = test_utils.DefaultEventLoopPolicy() + with self.assertWarnsRegex( + DeprecationWarning, "'asyncio.set_event_loop_policy' is deprecated"): + asyncio.set_event_loop_policy(policy) + + with self.assertWarnsRegex( + DeprecationWarning, "'asyncio.get_event_loop_policy' is deprecated"): + self.assertIs(policy, asyncio.get_event_loop_policy()) + self.assertIsNot(policy, old_policy) + + +class GetEventLoopTestsMixin: + + _get_running_loop_impl = None + _set_running_loop_impl = None + get_running_loop_impl = None + get_event_loop_impl = None + + Task = None + Future = None + + def setUp(self): + self._get_running_loop_saved = events._get_running_loop + self._set_running_loop_saved = events._set_running_loop + self.get_running_loop_saved = events.get_running_loop + self.get_event_loop_saved = events.get_event_loop + self._Task_saved = asyncio.Task + self._Future_saved = asyncio.Future + + events._get_running_loop = type(self)._get_running_loop_impl + events._set_running_loop = type(self)._set_running_loop_impl + events.get_running_loop = type(self).get_running_loop_impl + events.get_event_loop = type(self).get_event_loop_impl + + asyncio._get_running_loop = type(self)._get_running_loop_impl + asyncio._set_running_loop = type(self)._set_running_loop_impl + asyncio.get_running_loop = type(self).get_running_loop_impl + asyncio.get_event_loop = type(self).get_event_loop_impl + + asyncio.Task = asyncio.tasks.Task = type(self).Task + asyncio.Future = asyncio.futures.Future = type(self).Future + super().setUp() + + self.loop = asyncio.new_event_loop() + asyncio.set_event_loop(self.loop) + + def tearDown(self): + try: + super().tearDown() + finally: + self.loop.close() + asyncio.set_event_loop(None) + + events._get_running_loop = self._get_running_loop_saved + events._set_running_loop = self._set_running_loop_saved + events.get_running_loop = self.get_running_loop_saved + events.get_event_loop = self.get_event_loop_saved + + asyncio._get_running_loop = self._get_running_loop_saved + asyncio._set_running_loop = self._set_running_loop_saved + asyncio.get_running_loop = self.get_running_loop_saved + asyncio.get_event_loop = self.get_event_loop_saved + + asyncio.Task = asyncio.tasks.Task = self._Task_saved + asyncio.Future = asyncio.futures.Future = self._Future_saved + + if sys.platform != 'win32': + def test_get_event_loop_new_process(self): + # bpo-32126: The multiprocessing module used by + # ProcessPoolExecutor is not functional when the + # multiprocessing.synchronize module cannot be imported. + support.skip_if_broken_multiprocessing_synchronize() + + self.addCleanup(multiprocessing_cleanup_tests) + + async def main(): + if multiprocessing.get_start_method() == 'fork': + # Avoid 'fork' DeprecationWarning. + mp_context = multiprocessing.get_context('forkserver') + else: + mp_context = None + pool = concurrent.futures.ProcessPoolExecutor( + mp_context=mp_context) + result = await self.loop.run_in_executor( + pool, _test_get_event_loop_new_process__sub_proc) + pool.shutdown() + return result + + self.assertEqual( + self.loop.run_until_complete(main()), + 'hello') + + def test_get_running_loop_already_running(self): + async def main(): + running_loop = asyncio.get_running_loop() + with contextlib.closing(asyncio.new_event_loop()) as loop: + try: + loop.run_forever() + except RuntimeError: + pass + else: + self.fail("RuntimeError not raised") + + self.assertIs(asyncio.get_running_loop(), running_loop) + + self.loop.run_until_complete(main()) + + + def test_get_event_loop_returns_running_loop(self): + class TestError(Exception): + pass + + class Policy(test_utils.DefaultEventLoopPolicy): + def get_event_loop(self): + raise TestError + + old_policy = asyncio.events._get_event_loop_policy() + try: + asyncio.events._set_event_loop_policy(Policy()) + loop = asyncio.new_event_loop() + + with self.assertRaises(TestError): + asyncio.get_event_loop() + asyncio.set_event_loop(None) + with self.assertRaises(TestError): + asyncio.get_event_loop() + + with self.assertRaisesRegex(RuntimeError, 'no running'): + asyncio.get_running_loop() + self.assertIs(asyncio._get_running_loop(), None) + + async def func(): + self.assertIs(asyncio.get_event_loop(), loop) + self.assertIs(asyncio.get_running_loop(), loop) + self.assertIs(asyncio._get_running_loop(), loop) + + loop.run_until_complete(func()) + + asyncio.set_event_loop(loop) + with self.assertRaises(TestError): + asyncio.get_event_loop() + asyncio.set_event_loop(None) + with self.assertRaises(TestError): + asyncio.get_event_loop() + + finally: + asyncio.events._set_event_loop_policy(old_policy) + if loop is not None: + loop.close() + + with self.assertRaisesRegex(RuntimeError, 'no running'): + asyncio.get_running_loop() + + self.assertIs(asyncio._get_running_loop(), None) + + def test_get_event_loop_returns_running_loop2(self): + old_policy = asyncio.events._get_event_loop_policy() + try: + asyncio.events._set_event_loop_policy(test_utils.DefaultEventLoopPolicy()) + loop = asyncio.new_event_loop() + self.addCleanup(loop.close) + + with self.assertRaisesRegex(RuntimeError, 'no current'): + asyncio.get_event_loop() + + asyncio.set_event_loop(None) + with self.assertRaisesRegex(RuntimeError, 'no current'): + asyncio.get_event_loop() + + async def func(): + self.assertIs(asyncio.get_event_loop(), loop) + self.assertIs(asyncio.get_running_loop(), loop) + self.assertIs(asyncio._get_running_loop(), loop) + + loop.run_until_complete(func()) + + asyncio.set_event_loop(loop) + self.assertIs(asyncio.get_event_loop(), loop) + + asyncio.set_event_loop(None) + with self.assertRaisesRegex(RuntimeError, 'no current'): + asyncio.get_event_loop() + + finally: + asyncio.events._set_event_loop_policy(old_policy) + if loop is not None: + loop.close() + + with self.assertRaisesRegex(RuntimeError, 'no running'): + asyncio.get_running_loop() + + self.assertIs(asyncio._get_running_loop(), None) + + +class TestPyGetEventLoop(GetEventLoopTestsMixin, unittest.TestCase): + + _get_running_loop_impl = events._py__get_running_loop + _set_running_loop_impl = events._py__set_running_loop + get_running_loop_impl = events._py_get_running_loop + get_event_loop_impl = events._py_get_event_loop + + Task = asyncio.tasks._PyTask + Future = asyncio.futures._PyFuture + +try: + import _asyncio # NoQA +except ImportError: + pass +else: + + class TestCGetEventLoop(GetEventLoopTestsMixin, unittest.TestCase): + + _get_running_loop_impl = events._c__get_running_loop + _set_running_loop_impl = events._c__set_running_loop + get_running_loop_impl = events._c_get_running_loop + get_event_loop_impl = events._c_get_event_loop + + Task = asyncio.tasks._CTask + Future = asyncio.futures._CFuture + +class TestServer(unittest.TestCase): + + def test_get_loop(self): + loop = asyncio.new_event_loop() + self.addCleanup(loop.close) + proto = MyProto(loop) + server = loop.run_until_complete(loop.create_server(lambda: proto, '0.0.0.0', 0)) + self.assertEqual(server.get_loop(), loop) + server.close() + loop.run_until_complete(server.wait_closed()) + + +class TestAbstractServer(unittest.TestCase): + + def test_close(self): + with self.assertRaises(NotImplementedError): + events.AbstractServer().close() + + def test_wait_closed(self): + loop = asyncio.new_event_loop() + self.addCleanup(loop.close) + + with self.assertRaises(NotImplementedError): + loop.run_until_complete(events.AbstractServer().wait_closed()) + + def test_get_loop(self): + with self.assertRaises(NotImplementedError): + events.AbstractServer().get_loop() + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_free_threading.py b/Lib/test/test_asyncio/test_free_threading.py new file mode 100644 index 00000000000..c8de0d24499 --- /dev/null +++ b/Lib/test/test_asyncio/test_free_threading.py @@ -0,0 +1,235 @@ +import asyncio +import threading +import unittest +from threading import Thread +from unittest import TestCase +import weakref +from test import support +from test.support import threading_helper + +threading_helper.requires_working_threading(module=True) + + +class MyException(Exception): + pass + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +class TestFreeThreading: + def test_all_tasks_race(self) -> None: + async def main(): + loop = asyncio.get_running_loop() + future = loop.create_future() + + async def coro(): + await future + + tasks = set() + + async with asyncio.TaskGroup() as tg: + for _ in range(100): + tasks.add(tg.create_task(coro())) + + all_tasks = asyncio.all_tasks(loop) + self.assertEqual(len(all_tasks), 101) + + for task in all_tasks: + self.assertEqual(task.get_loop(), loop) + self.assertFalse(task.done()) + + current = asyncio.current_task() + self.assertEqual(current.get_loop(), loop) + self.assertSetEqual(all_tasks, tasks | {current}) + future.set_result(None) + + def runner(): + with asyncio.Runner() as runner: + loop = runner.get_loop() + loop.set_task_factory(self.factory) + runner.run(main()) + + threads = [] + + for _ in range(10): + thread = Thread(target=runner) + threads.append(thread) + + with threading_helper.start_threads(threads): + pass + + def test_all_tasks_different_thread(self) -> None: + loop = None + started = threading.Event() + done = threading.Event() # used for main task not finishing early + async def coro(): + await asyncio.Future() + + lock = threading.Lock() + tasks = set() + + async def main(): + nonlocal tasks, loop + loop = asyncio.get_running_loop() + started.set() + for i in range(1000): + with lock: + asyncio.create_task(coro()) + tasks = asyncio.all_tasks(loop) + done.wait() + + runner = threading.Thread(target=lambda: asyncio.run(main())) + + def check(): + started.wait() + with lock: + self.assertSetEqual(tasks & asyncio.all_tasks(loop), tasks) + + threads = [threading.Thread(target=check) for _ in range(10)] + runner.start() + + with threading_helper.start_threads(threads): + pass + + done.set() + runner.join() + + def test_task_different_thread_finalized(self) -> None: + task = None + async def func(): + nonlocal task + task = asyncio.current_task() + def runner(): + with asyncio.Runner() as runner: + loop = runner.get_loop() + loop.set_task_factory(self.factory) + runner.run(func()) + thread = Thread(target=runner) + thread.start() + thread.join() + wr = weakref.ref(task) + del thread + del task + # task finalization in different thread shouldn't crash + support.gc_collect() + self.assertIsNone(wr()) + + def test_run_coroutine_threadsafe(self) -> None: + results = [] + + def in_thread(loop: asyncio.AbstractEventLoop): + coro = asyncio.sleep(0.1, result=42) + fut = asyncio.run_coroutine_threadsafe(coro, loop) + result = fut.result() + self.assertEqual(result, 42) + results.append(result) + + async def main(): + loop = asyncio.get_running_loop() + async with asyncio.TaskGroup() as tg: + for _ in range(10): + tg.create_task(asyncio.to_thread(in_thread, loop)) + self.assertEqual(results, [42] * 10) + + with asyncio.Runner() as r: + loop = r.get_loop() + loop.set_task_factory(self.factory) + r.run(main()) + + def test_run_coroutine_threadsafe_exception(self) -> None: + async def coro(): + await asyncio.sleep(0) + raise MyException("test") + + def in_thread(loop: asyncio.AbstractEventLoop): + fut = asyncio.run_coroutine_threadsafe(coro(), loop) + return fut.result() + + async def main(): + loop = asyncio.get_running_loop() + tasks = [] + for _ in range(10): + task = loop.create_task(asyncio.to_thread(in_thread, loop)) + tasks.append(task) + results = await asyncio.gather(*tasks, return_exceptions=True) + + self.assertEqual(len(results), 10) + for result in results: + self.assertIsInstance(result, MyException) + self.assertEqual(str(result), "test") + + with asyncio.Runner() as r: + loop = r.get_loop() + loop.set_task_factory(self.factory) + r.run(main()) + + +class TestPyFreeThreading(TestFreeThreading, TestCase): + + def setUp(self): + self._old_current_task = asyncio.current_task + asyncio.current_task = asyncio.tasks.current_task = asyncio.tasks._py_current_task + self._old_all_tasks = asyncio.all_tasks + asyncio.all_tasks = asyncio.tasks.all_tasks = asyncio.tasks._py_all_tasks + self._old_Task = asyncio.Task + asyncio.Task = asyncio.tasks.Task = asyncio.tasks._PyTask + self._old_Future = asyncio.Future + asyncio.Future = asyncio.futures.Future = asyncio.futures._PyFuture + return super().setUp() + + def tearDown(self): + asyncio.current_task = asyncio.tasks.current_task = self._old_current_task + asyncio.all_tasks = asyncio.tasks.all_tasks = self._old_all_tasks + asyncio.Task = asyncio.tasks.Task = self._old_Task + asyncio.Future = asyncio.tasks.Future = self._old_Future + return super().tearDown() + + def factory(self, loop, coro, **kwargs): + return asyncio.tasks._PyTask(coro, loop=loop, **kwargs) + + @unittest.skip("TODO: RUSTPYTHON; hangs - Python _current_tasks dict not thread-safe") + def test_all_tasks_race(self): + return super().test_all_tasks_race() + + +@unittest.skipUnless(hasattr(asyncio.tasks, "_c_all_tasks"), "requires _asyncio") +class TestCFreeThreading(TestFreeThreading, TestCase): + + def setUp(self): + self._old_current_task = asyncio.current_task + asyncio.current_task = asyncio.tasks.current_task = asyncio.tasks._c_current_task + self._old_all_tasks = asyncio.all_tasks + asyncio.all_tasks = asyncio.tasks.all_tasks = asyncio.tasks._c_all_tasks + self._old_Task = asyncio.Task + asyncio.Task = asyncio.tasks.Task = asyncio.tasks._CTask + self._old_Future = asyncio.Future + asyncio.Future = asyncio.futures.Future = asyncio.futures._CFuture + return super().setUp() + + def tearDown(self): + asyncio.current_task = asyncio.tasks.current_task = self._old_current_task + asyncio.all_tasks = asyncio.tasks.all_tasks = self._old_all_tasks + asyncio.Task = asyncio.tasks.Task = self._old_Task + asyncio.Future = asyncio.futures.Future = self._old_Future + return super().tearDown() + + + def factory(self, loop, coro, **kwargs): + return asyncio.tasks._CTask(coro, loop=loop, **kwargs) + + +class TestEagerPyFreeThreading(TestPyFreeThreading): + def factory(self, loop, coro, eager_start=True, **kwargs): + return asyncio.tasks._PyTask(coro, loop=loop, **kwargs, eager_start=eager_start) + + @unittest.skip("TODO: RUSTPYTHON; hangs - Python _current_tasks dict not thread-safe") + def test_all_tasks_race(self): + return super().test_all_tasks_race() + + +@unittest.skipUnless(hasattr(asyncio.tasks, "_c_all_tasks"), "requires _asyncio") +class TestEagerCFreeThreading(TestCFreeThreading, TestCase): + def factory(self, loop, coro, eager_start=True, **kwargs): + return asyncio.tasks._CTask(coro, loop=loop, **kwargs, eager_start=eager_start) diff --git a/Lib/test/test_asyncio/test_futures.py b/Lib/test/test_asyncio/test_futures.py new file mode 100644 index 00000000000..54bf824fef7 --- /dev/null +++ b/Lib/test/test_asyncio/test_futures.py @@ -0,0 +1,1144 @@ +"""Tests for futures.py.""" + +import concurrent.futures +import gc +import re +import sys +import threading +import traceback +import unittest +from unittest import mock +from types import GenericAlias +import asyncio +from asyncio import futures +import warnings +from test.test_asyncio import utils as test_utils +from test import support + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +def _fakefunc(f): + return f + + +def first_cb(): + pass + + +def last_cb(): + pass + + +class ReachableCode(Exception): + """Exception to raise to indicate that some code was reached. + + Use this exception if using mocks is not a good alternative. + """ + + +class SimpleEvilEventLoop(asyncio.base_events.BaseEventLoop): + """Base class for UAF and other evil stuff requiring an evil event loop.""" + + def get_debug(self): # to suppress tracebacks + return False + + def __del__(self): + # Automatically close the evil event loop to avoid warnings. + if not self.is_closed() and not self.is_running(): + self.close() + + +class DuckFuture: + # Class that does not inherit from Future but aims to be duck-type + # compatible with it. + + _asyncio_future_blocking = False + __cancelled = False + __result = None + __exception = None + + def cancel(self): + if self.done(): + return False + self.__cancelled = True + return True + + def cancelled(self): + return self.__cancelled + + def done(self): + return (self.__cancelled + or self.__result is not None + or self.__exception is not None) + + def result(self): + self.assertFalse(self.cancelled()) + if self.__exception is not None: + raise self.__exception + return self.__result + + def exception(self): + self.assertFalse(self.cancelled()) + return self.__exception + + def set_result(self, result): + self.assertFalse(self.done()) + self.assertIsNotNone(result) + self.__result = result + + def set_exception(self, exception): + self.assertFalse(self.done()) + self.assertIsNotNone(exception) + self.__exception = exception + + def __iter__(self): + if not self.done(): + self._asyncio_future_blocking = True + yield self + self.assertTrue(self.done()) + return self.result() + + +class DuckTests(test_utils.TestCase): + + def setUp(self): + super().setUp() + self.loop = self.new_test_loop() + self.addCleanup(self.loop.close) + + def test_wrap_future(self): + f = DuckFuture() + g = asyncio.wrap_future(f) + self.assertIs(g, f) + + def test_ensure_future(self): + f = DuckFuture() + g = asyncio.ensure_future(f) + self.assertIs(g, f) + + +class BaseFutureTests: + + def _new_future(self, *args, **kwargs): + return self.cls(*args, **kwargs) + + def setUp(self): + super().setUp() + self.loop = self.new_test_loop() + self.addCleanup(self.loop.close) + + def test_generic_alias(self): + future = self.cls[str] + self.assertEqual(future.__args__, (str,)) + self.assertIsInstance(future, GenericAlias) + + def test_isfuture(self): + class MyFuture: + _asyncio_future_blocking = None + + def __init__(self): + self._asyncio_future_blocking = False + + self.assertFalse(asyncio.isfuture(MyFuture)) + self.assertTrue(asyncio.isfuture(MyFuture())) + self.assertFalse(asyncio.isfuture(1)) + + # As `isinstance(Mock(), Future)` returns `False` + self.assertFalse(asyncio.isfuture(mock.Mock())) + + f = self._new_future(loop=self.loop) + self.assertTrue(asyncio.isfuture(f)) + self.assertFalse(asyncio.isfuture(type(f))) + + # As `isinstance(Mock(Future), Future)` returns `True` + self.assertTrue(asyncio.isfuture(mock.Mock(type(f)))) + + f.cancel() + + def test_initial_state(self): + f = self._new_future(loop=self.loop) + self.assertFalse(f.cancelled()) + self.assertFalse(f.done()) + f.cancel() + self.assertTrue(f.cancelled()) + + def test_constructor_without_loop(self): + with self.assertRaisesRegex(RuntimeError, 'no current event loop'): + self._new_future() + + def test_constructor_use_running_loop(self): + async def test(): + return self._new_future() + f = self.loop.run_until_complete(test()) + self.assertIs(f._loop, self.loop) + self.assertIs(f.get_loop(), self.loop) + + def test_constructor_use_global_loop(self): + # Deprecated in 3.10, undeprecated in 3.12 + asyncio.set_event_loop(self.loop) + self.addCleanup(asyncio.set_event_loop, None) + f = self._new_future() + self.assertIs(f._loop, self.loop) + self.assertIs(f.get_loop(), self.loop) + + def test_constructor_positional(self): + # Make sure Future doesn't accept a positional argument + self.assertRaises(TypeError, self._new_future, 42) + + def test_uninitialized(self): + # Test that C Future doesn't crash when Future.__init__() + # call was skipped. + + fut = self.cls.__new__(self.cls, loop=self.loop) + self.assertRaises(asyncio.InvalidStateError, fut.result) + + fut = self.cls.__new__(self.cls, loop=self.loop) + self.assertRaises(asyncio.InvalidStateError, fut.exception) + + fut = self.cls.__new__(self.cls, loop=self.loop) + with self.assertRaises((RuntimeError, AttributeError)): + fut.set_result(None) + + fut = self.cls.__new__(self.cls, loop=self.loop) + with self.assertRaises((RuntimeError, AttributeError)): + fut.set_exception(Exception) + + fut = self.cls.__new__(self.cls, loop=self.loop) + with self.assertRaises((RuntimeError, AttributeError)): + fut.cancel() + + fut = self.cls.__new__(self.cls, loop=self.loop) + with self.assertRaises((RuntimeError, AttributeError)): + fut.add_done_callback(lambda f: None) + + fut = self.cls.__new__(self.cls, loop=self.loop) + with self.assertRaises((RuntimeError, AttributeError)): + fut.remove_done_callback(lambda f: None) + + fut = self.cls.__new__(self.cls, loop=self.loop) + try: + repr(fut) + except (RuntimeError, AttributeError): + pass + + fut = self.cls.__new__(self.cls, loop=self.loop) + try: + fut.__await__() + except RuntimeError: + pass + + fut = self.cls.__new__(self.cls, loop=self.loop) + try: + iter(fut) + except RuntimeError: + pass + + fut = self.cls.__new__(self.cls, loop=self.loop) + self.assertFalse(fut.cancelled()) + self.assertFalse(fut.done()) + + def test_future_cancel_message_getter(self): + f = self._new_future(loop=self.loop) + self.assertHasAttr(f, '_cancel_message') + self.assertEqual(f._cancel_message, None) + + f.cancel('my message') + with self.assertRaises(asyncio.CancelledError): + self.loop.run_until_complete(f) + self.assertEqual(f._cancel_message, 'my message') + + def test_future_cancel_message_setter(self): + f = self._new_future(loop=self.loop) + f.cancel('my message') + f._cancel_message = 'my new message' + self.assertEqual(f._cancel_message, 'my new message') + + # Also check that the value is used for cancel(). + with self.assertRaises(asyncio.CancelledError): + self.loop.run_until_complete(f) + self.assertEqual(f._cancel_message, 'my new message') + + def test_cancel(self): + f = self._new_future(loop=self.loop) + self.assertTrue(f.cancel()) + self.assertTrue(f.cancelled()) + self.assertTrue(f.done()) + self.assertRaises(asyncio.CancelledError, f.result) + self.assertRaises(asyncio.CancelledError, f.exception) + self.assertRaises(asyncio.InvalidStateError, f.set_result, None) + self.assertRaises(asyncio.InvalidStateError, f.set_exception, None) + self.assertFalse(f.cancel()) + + def test_result(self): + f = self._new_future(loop=self.loop) + self.assertRaises(asyncio.InvalidStateError, f.result) + + f.set_result(42) + self.assertFalse(f.cancelled()) + self.assertTrue(f.done()) + self.assertEqual(f.result(), 42) + self.assertEqual(f.exception(), None) + self.assertRaises(asyncio.InvalidStateError, f.set_result, None) + self.assertRaises(asyncio.InvalidStateError, f.set_exception, None) + self.assertFalse(f.cancel()) + + def test_exception(self): + exc = RuntimeError() + f = self._new_future(loop=self.loop) + self.assertRaises(asyncio.InvalidStateError, f.exception) + + f.set_exception(exc) + self.assertFalse(f.cancelled()) + self.assertTrue(f.done()) + self.assertRaises(RuntimeError, f.result) + self.assertEqual(f.exception(), exc) + self.assertRaises(asyncio.InvalidStateError, f.set_result, None) + self.assertRaises(asyncio.InvalidStateError, f.set_exception, None) + self.assertFalse(f.cancel()) + + def test_stop_iteration_exception(self, stop_iteration_class=StopIteration): + exc = stop_iteration_class() + f = self._new_future(loop=self.loop) + f.set_exception(exc) + self.assertFalse(f.cancelled()) + self.assertTrue(f.done()) + self.assertRaises(RuntimeError, f.result) + exc = f.exception() + cause = exc.__cause__ + self.assertIsInstance(exc, RuntimeError) + self.assertRegex(str(exc), 'StopIteration .* cannot be raised') + self.assertIsInstance(cause, stop_iteration_class) + + def test_stop_iteration_subclass_exception(self): + class MyStopIteration(StopIteration): + pass + + self.test_stop_iteration_exception(MyStopIteration) + + def test_exception_class(self): + f = self._new_future(loop=self.loop) + f.set_exception(RuntimeError) + self.assertIsInstance(f.exception(), RuntimeError) + + def test_yield_from_twice(self): + f = self._new_future(loop=self.loop) + + def fixture(): + yield 'A' + x = yield from f + yield 'B', x + y = yield from f + yield 'C', y + + g = fixture() + self.assertEqual(next(g), 'A') # yield 'A'. + self.assertEqual(next(g), f) # First yield from f. + f.set_result(42) + self.assertEqual(next(g), ('B', 42)) # yield 'B', x. + # The second "yield from f" does not yield f. + self.assertEqual(next(g), ('C', 42)) # yield 'C', y. + + def test_future_repr(self): + self.loop.set_debug(True) + f_pending_debug = self._new_future(loop=self.loop) + frame = f_pending_debug._source_traceback[-1] + self.assertEqual( + repr(f_pending_debug), + f'<{self.cls.__name__} pending created at {frame[0]}:{frame[1]}>') + f_pending_debug.cancel() + + self.loop.set_debug(False) + f_pending = self._new_future(loop=self.loop) + self.assertEqual(repr(f_pending), f'<{self.cls.__name__} pending>') + f_pending.cancel() + + f_cancelled = self._new_future(loop=self.loop) + f_cancelled.cancel() + self.assertEqual(repr(f_cancelled), f'<{self.cls.__name__} cancelled>') + + f_result = self._new_future(loop=self.loop) + f_result.set_result(4) + self.assertEqual( + repr(f_result), f'<{self.cls.__name__} finished result=4>') + self.assertEqual(f_result.result(), 4) + + exc = RuntimeError() + f_exception = self._new_future(loop=self.loop) + f_exception.set_exception(exc) + self.assertEqual( + repr(f_exception), + f'<{self.cls.__name__} finished exception=RuntimeError()>') + self.assertIs(f_exception.exception(), exc) + + def func_repr(func): + filename, lineno = test_utils.get_function_source(func) + text = '%s() at %s:%s' % (func.__qualname__, filename, lineno) + return re.escape(text) + + f_one_callbacks = self._new_future(loop=self.loop) + f_one_callbacks.add_done_callback(_fakefunc) + fake_repr = func_repr(_fakefunc) + self.assertRegex( + repr(f_one_callbacks), + r'<' + self.cls.__name__ + r' pending cb=\[%s\]>' % fake_repr) + f_one_callbacks.cancel() + self.assertEqual(repr(f_one_callbacks), + f'<{self.cls.__name__} cancelled>') + + f_two_callbacks = self._new_future(loop=self.loop) + f_two_callbacks.add_done_callback(first_cb) + f_two_callbacks.add_done_callback(last_cb) + first_repr = func_repr(first_cb) + last_repr = func_repr(last_cb) + self.assertRegex(repr(f_two_callbacks), + r'<' + self.cls.__name__ + r' pending cb=\[%s, %s\]>' + % (first_repr, last_repr)) + + f_many_callbacks = self._new_future(loop=self.loop) + f_many_callbacks.add_done_callback(first_cb) + for i in range(8): + f_many_callbacks.add_done_callback(_fakefunc) + f_many_callbacks.add_done_callback(last_cb) + cb_regex = r'%s, <8 more>, %s' % (first_repr, last_repr) + self.assertRegex( + repr(f_many_callbacks), + r'<' + self.cls.__name__ + r' pending cb=\[%s\]>' % cb_regex) + f_many_callbacks.cancel() + self.assertEqual(repr(f_many_callbacks), + f'<{self.cls.__name__} cancelled>') + + def test_copy_state(self): + from asyncio.futures import _copy_future_state + + f = self._new_future(loop=self.loop) + f.set_result(10) + + newf = self._new_future(loop=self.loop) + _copy_future_state(f, newf) + self.assertTrue(newf.done()) + self.assertEqual(newf.result(), 10) + + f_exception = self._new_future(loop=self.loop) + f_exception.set_exception(RuntimeError()) + + newf_exception = self._new_future(loop=self.loop) + _copy_future_state(f_exception, newf_exception) + self.assertTrue(newf_exception.done()) + self.assertRaises(RuntimeError, newf_exception.result) + + f_cancelled = self._new_future(loop=self.loop) + f_cancelled.cancel() + + newf_cancelled = self._new_future(loop=self.loop) + _copy_future_state(f_cancelled, newf_cancelled) + self.assertTrue(newf_cancelled.cancelled()) + + try: + raise concurrent.futures.InvalidStateError + except BaseException as e: + f_exc = e + + f_conexc = self._new_future(loop=self.loop) + f_conexc.set_exception(f_exc) + + newf_conexc = self._new_future(loop=self.loop) + _copy_future_state(f_conexc, newf_conexc) + self.assertTrue(newf_conexc.done()) + try: + newf_conexc.result() + except BaseException as e: + newf_exc = e # assertRaises context manager drops the traceback + newf_tb = ''.join(traceback.format_tb(newf_exc.__traceback__)) + self.assertEqual(newf_tb.count('raise concurrent.futures.InvalidStateError'), 1) + + def test_iter(self): + fut = self._new_future(loop=self.loop) + + def coro(): + yield from fut + + def test(): + arg1, arg2 = coro() + + with self.assertRaisesRegex(RuntimeError, "await wasn't used"): + test() + fut.cancel() + + def test_log_traceback(self): + fut = self._new_future(loop=self.loop) + with self.assertRaisesRegex(ValueError, 'can only be set to False'): + fut._log_traceback = True + + @mock.patch('asyncio.base_events.logger') + def test_tb_logger_abandoned(self, m_log): + fut = self._new_future(loop=self.loop) + del fut + self.assertFalse(m_log.error.called) + + @mock.patch('asyncio.base_events.logger') + def test_tb_logger_not_called_after_cancel(self, m_log): + fut = self._new_future(loop=self.loop) + fut.set_exception(Exception()) + fut.cancel() + del fut + self.assertFalse(m_log.error.called) + + @mock.patch('asyncio.base_events.logger') + def test_tb_logger_result_unretrieved(self, m_log): + fut = self._new_future(loop=self.loop) + fut.set_result(42) + del fut + self.assertFalse(m_log.error.called) + + @mock.patch('asyncio.base_events.logger') + def test_tb_logger_result_retrieved(self, m_log): + fut = self._new_future(loop=self.loop) + fut.set_result(42) + fut.result() + del fut + self.assertFalse(m_log.error.called) + + @mock.patch('asyncio.base_events.logger') + def test_tb_logger_exception_unretrieved(self, m_log): + fut = self._new_future(loop=self.loop) + fut.set_exception(RuntimeError('boom')) + del fut + test_utils.run_briefly(self.loop) + support.gc_collect() + self.assertTrue(m_log.error.called) + + @mock.patch('asyncio.base_events.logger') + def test_tb_logger_exception_retrieved(self, m_log): + fut = self._new_future(loop=self.loop) + fut.set_exception(RuntimeError('boom')) + fut.exception() + del fut + self.assertFalse(m_log.error.called) + + @mock.patch('asyncio.base_events.logger') + def test_tb_logger_exception_result_retrieved(self, m_log): + fut = self._new_future(loop=self.loop) + fut.set_exception(RuntimeError('boom')) + self.assertRaises(RuntimeError, fut.result) + del fut + self.assertFalse(m_log.error.called) + + def test_wrap_future(self): + + def run(arg): + return (arg, threading.get_ident()) + ex = concurrent.futures.ThreadPoolExecutor(1) + f1 = ex.submit(run, 'oi') + f2 = asyncio.wrap_future(f1, loop=self.loop) + res, ident = self.loop.run_until_complete(f2) + self.assertTrue(asyncio.isfuture(f2)) + self.assertEqual(res, 'oi') + self.assertNotEqual(ident, threading.get_ident()) + ex.shutdown(wait=True) + + def test_wrap_future_future(self): + f1 = self._new_future(loop=self.loop) + f2 = asyncio.wrap_future(f1) + self.assertIs(f1, f2) + + def test_wrap_future_without_loop(self): + def run(arg): + return (arg, threading.get_ident()) + ex = concurrent.futures.ThreadPoolExecutor(1) + f1 = ex.submit(run, 'oi') + with self.assertRaisesRegex(RuntimeError, 'no current event loop'): + asyncio.wrap_future(f1) + ex.shutdown(wait=True) + + def test_wrap_future_use_running_loop(self): + def run(arg): + return (arg, threading.get_ident()) + ex = concurrent.futures.ThreadPoolExecutor(1) + f1 = ex.submit(run, 'oi') + async def test(): + return asyncio.wrap_future(f1) + f2 = self.loop.run_until_complete(test()) + self.assertIs(self.loop, f2._loop) + ex.shutdown(wait=True) + + def test_wrap_future_use_global_loop(self): + # Deprecated in 3.10, undeprecated in 3.12 + asyncio.set_event_loop(self.loop) + self.addCleanup(asyncio.set_event_loop, None) + def run(arg): + return (arg, threading.get_ident()) + ex = concurrent.futures.ThreadPoolExecutor(1) + f1 = ex.submit(run, 'oi') + f2 = asyncio.wrap_future(f1) + self.assertIs(self.loop, f2._loop) + ex.shutdown(wait=True) + + def test_wrap_future_cancel(self): + f1 = concurrent.futures.Future() + f2 = asyncio.wrap_future(f1, loop=self.loop) + f2.cancel() + test_utils.run_briefly(self.loop) + self.assertTrue(f1.cancelled()) + self.assertTrue(f2.cancelled()) + + def test_wrap_future_cancel2(self): + f1 = concurrent.futures.Future() + f2 = asyncio.wrap_future(f1, loop=self.loop) + f1.set_result(42) + f2.cancel() + test_utils.run_briefly(self.loop) + self.assertFalse(f1.cancelled()) + self.assertEqual(f1.result(), 42) + self.assertTrue(f2.cancelled()) + + def test_future_source_traceback(self): + self.loop.set_debug(True) + + future = self._new_future(loop=self.loop) + lineno = sys._getframe().f_lineno - 1 + self.assertIsInstance(future._source_traceback, list) + self.assertEqual(future._source_traceback[-2][:3], + (__file__, + lineno, + 'test_future_source_traceback')) + + @mock.patch('asyncio.base_events.logger') + def check_future_exception_never_retrieved(self, debug, m_log): + self.loop.set_debug(debug) + + def memory_error(): + try: + raise MemoryError() + except BaseException as exc: + return exc + exc = memory_error() + + future = self._new_future(loop=self.loop) + future.set_exception(exc) + future = None + test_utils.run_briefly(self.loop) + support.gc_collect() + + regex = f'^{self.cls.__name__} exception was never retrieved\n' + exc_info = (type(exc), exc, exc.__traceback__) + m_log.error.assert_called_once_with(mock.ANY, exc_info=exc_info) + + message = m_log.error.call_args[0][0] + self.assertRegex(message, re.compile(regex, re.DOTALL)) + + def test_future_exception_never_retrieved(self): + self.check_future_exception_never_retrieved(False) + + def test_future_exception_never_retrieved_debug(self): + self.check_future_exception_never_retrieved(True) + + def test_set_result_unless_cancelled(self): + fut = self._new_future(loop=self.loop) + fut.cancel() + futures._set_result_unless_cancelled(fut, 2) + self.assertTrue(fut.cancelled()) + + def test_future_stop_iteration_args(self): + fut = self._new_future(loop=self.loop) + fut.set_result((1, 2)) + fi = fut.__iter__() + result = None + try: + fi.send(None) + except StopIteration as ex: + result = ex.args[0] + else: + self.fail('StopIteration was expected') + self.assertEqual(result, (1, 2)) + + def test_future_iter_throw(self): + fut = self._new_future(loop=self.loop) + fi = iter(fut) + with self.assertWarns(DeprecationWarning): + self.assertRaises(Exception, fi.throw, Exception, Exception("zebra"), None) + with warnings.catch_warnings(): + warnings.filterwarnings("ignore", category=DeprecationWarning) + self.assertRaises(TypeError, fi.throw, + Exception, Exception("elephant"), 32) + self.assertRaises(TypeError, fi.throw, + Exception("elephant"), Exception("elephant")) + # https://github.com/python/cpython/issues/101326 + self.assertRaises(ValueError, fi.throw, ValueError, None, None) + self.assertRaises(TypeError, fi.throw, list) + + def test_future_del_collect(self): + class Evil: + def __del__(self): + gc.collect() + + for i in range(100): + fut = self._new_future(loop=self.loop) + fut.set_result(Evil()) + + def test_future_cancelled_result_refcycles(self): + f = self._new_future(loop=self.loop) + f.cancel() + exc = None + try: + f.result() + except asyncio.CancelledError as e: + exc = e + self.assertIsNotNone(exc) + self.assertListEqual(gc.get_referrers(exc), []) + + def test_future_cancelled_exception_refcycles(self): + f = self._new_future(loop=self.loop) + f.cancel() + exc = None + try: + f.exception() + except asyncio.CancelledError as e: + exc = e + self.assertIsNotNone(exc) + self.assertListEqual(gc.get_referrers(exc), []) + + +@unittest.skipUnless(hasattr(futures, '_CFuture'), + 'requires the C _asyncio module') +class CFutureTests(BaseFutureTests, test_utils.TestCase): + try: + cls = futures._CFuture + except AttributeError: + cls = None + + def test_future_del_segfault(self): + fut = self._new_future(loop=self.loop) + with self.assertRaises(AttributeError): + del fut._asyncio_future_blocking + with self.assertRaises(AttributeError): + del fut._log_traceback + + def test_callbacks_copy(self): + # See https://github.com/python/cpython/issues/125789 + # In C implementation, the `_callbacks` attribute + # always returns a new list to avoid mutations of internal state + + fut = self._new_future(loop=self.loop) + f1 = lambda _: 1 + f2 = lambda _: 2 + fut.add_done_callback(f1) + fut.add_done_callback(f2) + callbacks = fut._callbacks + self.assertIsNot(callbacks, fut._callbacks) + fut.remove_done_callback(f1) + callbacks = fut._callbacks + self.assertIsNot(callbacks, fut._callbacks) + fut.remove_done_callback(f2) + self.assertIsNone(fut._callbacks) + + @unittest.expectedFailure # TODO: RUSTPYTHON; - gc.get_referents not implemented + def test_future_iter_get_referents_segfault(self): + return super().test_future_iter_get_referents_segfault() + + +@unittest.skipUnless(hasattr(futures, '_CFuture'), + 'requires the C _asyncio module') +class CSubFutureTests(BaseFutureTests, test_utils.TestCase): + try: + class CSubFuture(futures._CFuture): + pass + + cls = CSubFuture + except AttributeError: + cls = None + + +class PyFutureTests(BaseFutureTests, test_utils.TestCase): + cls = futures._PyFuture + + +class BaseFutureDoneCallbackTests(): + + def setUp(self): + super().setUp() + self.loop = self.new_test_loop() + + def run_briefly(self): + test_utils.run_briefly(self.loop) + + def _make_callback(self, bag, thing): + # Create a callback function that appends thing to bag. + def bag_appender(future): + bag.append(thing) + return bag_appender + + def _new_future(self): + raise NotImplementedError + + def test_callbacks_remove_first_callback(self): + bag = [] + f = self._new_future() + + cb1 = self._make_callback(bag, 42) + cb2 = self._make_callback(bag, 17) + cb3 = self._make_callback(bag, 100) + + f.add_done_callback(cb1) + f.add_done_callback(cb2) + f.add_done_callback(cb3) + + f.remove_done_callback(cb1) + f.remove_done_callback(cb1) + + self.assertEqual(bag, []) + f.set_result('foo') + + self.run_briefly() + + self.assertEqual(bag, [17, 100]) + self.assertEqual(f.result(), 'foo') + + def test_callbacks_remove_first_and_second_callback(self): + bag = [] + f = self._new_future() + + cb1 = self._make_callback(bag, 42) + cb2 = self._make_callback(bag, 17) + cb3 = self._make_callback(bag, 100) + + f.add_done_callback(cb1) + f.add_done_callback(cb2) + f.add_done_callback(cb3) + + f.remove_done_callback(cb1) + f.remove_done_callback(cb2) + f.remove_done_callback(cb1) + + self.assertEqual(bag, []) + f.set_result('foo') + + self.run_briefly() + + self.assertEqual(bag, [100]) + self.assertEqual(f.result(), 'foo') + + def test_callbacks_remove_third_callback(self): + bag = [] + f = self._new_future() + + cb1 = self._make_callback(bag, 42) + cb2 = self._make_callback(bag, 17) + cb3 = self._make_callback(bag, 100) + + f.add_done_callback(cb1) + f.add_done_callback(cb2) + f.add_done_callback(cb3) + + f.remove_done_callback(cb3) + f.remove_done_callback(cb3) + + self.assertEqual(bag, []) + f.set_result('foo') + + self.run_briefly() + + self.assertEqual(bag, [42, 17]) + self.assertEqual(f.result(), 'foo') + + def test_callbacks_invoked_on_set_result(self): + bag = [] + f = self._new_future() + f.add_done_callback(self._make_callback(bag, 42)) + f.add_done_callback(self._make_callback(bag, 17)) + + self.assertEqual(bag, []) + f.set_result('foo') + + self.run_briefly() + + self.assertEqual(bag, [42, 17]) + self.assertEqual(f.result(), 'foo') + + def test_callbacks_invoked_on_set_exception(self): + bag = [] + f = self._new_future() + f.add_done_callback(self._make_callback(bag, 100)) + + self.assertEqual(bag, []) + exc = RuntimeError() + f.set_exception(exc) + + self.run_briefly() + + self.assertEqual(bag, [100]) + self.assertEqual(f.exception(), exc) + + def test_remove_done_callback(self): + bag = [] + f = self._new_future() + cb1 = self._make_callback(bag, 1) + cb2 = self._make_callback(bag, 2) + cb3 = self._make_callback(bag, 3) + + # Add one cb1 and one cb2. + f.add_done_callback(cb1) + f.add_done_callback(cb2) + + # One instance of cb2 removed. Now there's only one cb1. + self.assertEqual(f.remove_done_callback(cb2), 1) + + # Never had any cb3 in there. + self.assertEqual(f.remove_done_callback(cb3), 0) + + # After this there will be 6 instances of cb1 and one of cb2. + f.add_done_callback(cb2) + for i in range(5): + f.add_done_callback(cb1) + + # Remove all instances of cb1. One cb2 remains. + self.assertEqual(f.remove_done_callback(cb1), 6) + + self.assertEqual(bag, []) + f.set_result('foo') + + self.run_briefly() + + self.assertEqual(bag, [2]) + self.assertEqual(f.result(), 'foo') + + def test_remove_done_callbacks_list_mutation(self): + # see http://bugs.python.org/issue28963 for details + + fut = self._new_future() + fut.add_done_callback(str) + + for _ in range(63): + fut.add_done_callback(id) + + class evil: + def __eq__(self, other): + fut.remove_done_callback(id) + return False + + fut.remove_done_callback(evil()) + + def test_remove_done_callbacks_list_clear(self): + # see https://github.com/python/cpython/issues/97592 for details + + fut = self._new_future() + fut.add_done_callback(str) + + for _ in range(63): + fut.add_done_callback(id) + + class evil: + def __eq__(self, other): + fut.remove_done_callback(other) + + fut.remove_done_callback(evil()) + + def test_schedule_callbacks_list_mutation_1(self): + # see http://bugs.python.org/issue28963 for details + + def mut(f): + f.remove_done_callback(str) + + fut = self._new_future() + fut.add_done_callback(mut) + fut.add_done_callback(str) + fut.add_done_callback(str) + fut.set_result(1) + test_utils.run_briefly(self.loop) + + def test_schedule_callbacks_list_mutation_2(self): + # see http://bugs.python.org/issue30828 for details + + fut = self._new_future() + fut.add_done_callback(str) + + for _ in range(63): + fut.add_done_callback(id) + + max_extra_cbs = 100 + extra_cbs = 0 + + class evil: + def __eq__(self, other): + nonlocal extra_cbs + extra_cbs += 1 + if extra_cbs < max_extra_cbs: + fut.add_done_callback(id) + return False + + fut.remove_done_callback(evil()) + + def test_evil_call_soon_list_mutation(self): + # see: https://github.com/python/cpython/issues/125969 + called_on_fut_callback0 = False + + pad = lambda: ... + + def evil_call_soon(*args, **kwargs): + nonlocal called_on_fut_callback0 + if called_on_fut_callback0: + # Called when handling fut->fut_callbacks[0] + # and mutates the length fut->fut_callbacks. + fut.remove_done_callback(int) + fut.remove_done_callback(pad) + else: + called_on_fut_callback0 = True + + fake_event_loop = SimpleEvilEventLoop() + fake_event_loop.call_soon = evil_call_soon + + with mock.patch.object(self, 'loop', fake_event_loop): + fut = self._new_future() + self.assertIs(fut.get_loop(), fake_event_loop) + + fut.add_done_callback(str) # sets fut->fut_callback0 + fut.add_done_callback(int) # sets fut->fut_callbacks[0] + fut.add_done_callback(pad) # sets fut->fut_callbacks[1] + fut.add_done_callback(pad) # sets fut->fut_callbacks[2] + fut.set_result("boom") + + # When there are no more callbacks, the Python implementation + # returns an empty list but the C implementation returns None. + self.assertIn(fut._callbacks, (None, [])) + + def test_use_after_free_on_fut_callback_0_with_evil__eq__(self): + # Special thanks to Nico-Posada for the original PoC. + # See https://github.com/python/cpython/issues/125966. + + fut = self._new_future() + + class cb_pad: + def __eq__(self, other): + return True + + class evil(cb_pad): + def __eq__(self, other): + fut.remove_done_callback(None) + return NotImplemented + + fut.add_done_callback(cb_pad()) + fut.remove_done_callback(evil()) + + def test_use_after_free_on_fut_callback_0_with_evil__getattribute__(self): + # see: https://github.com/python/cpython/issues/125984 + + class EvilEventLoop(SimpleEvilEventLoop): + def call_soon(self, *args, **kwargs): + super().call_soon(*args, **kwargs) + raise ReachableCode + + def __getattribute__(self, name): + nonlocal fut_callback_0 + if name == 'call_soon': + fut.remove_done_callback(fut_callback_0) + del fut_callback_0 + return object.__getattribute__(self, name) + + evil_loop = EvilEventLoop() + with mock.patch.object(self, 'loop', evil_loop): + fut = self._new_future() + self.assertIs(fut.get_loop(), evil_loop) + + fut_callback_0 = lambda: ... + fut.add_done_callback(fut_callback_0) + self.assertRaises(ReachableCode, fut.set_result, "boom") + + def test_use_after_free_on_fut_context_0_with_evil__getattribute__(self): + # see: https://github.com/python/cpython/issues/125984 + + class EvilEventLoop(SimpleEvilEventLoop): + def call_soon(self, *args, **kwargs): + super().call_soon(*args, **kwargs) + raise ReachableCode + + def __getattribute__(self, name): + if name == 'call_soon': + # resets the future's event loop + fut.__init__(loop=SimpleEvilEventLoop()) + return object.__getattribute__(self, name) + + evil_loop = EvilEventLoop() + with mock.patch.object(self, 'loop', evil_loop): + fut = self._new_future() + self.assertIs(fut.get_loop(), evil_loop) + + fut_callback_0 = mock.Mock() + fut_context_0 = mock.Mock() + fut.add_done_callback(fut_callback_0, context=fut_context_0) + del fut_context_0 + del fut_callback_0 + self.assertRaises(ReachableCode, fut.set_result, "boom") + + +@unittest.skipUnless(hasattr(futures, '_CFuture'), + 'requires the C _asyncio module') +class CFutureDoneCallbackTests(BaseFutureDoneCallbackTests, + test_utils.TestCase): + + def _new_future(self): + return futures._CFuture(loop=self.loop) + + +@unittest.skipUnless(hasattr(futures, '_CFuture'), + 'requires the C _asyncio module') +class CSubFutureDoneCallbackTests(BaseFutureDoneCallbackTests, + test_utils.TestCase): + + def _new_future(self): + class CSubFuture(futures._CFuture): + pass + return CSubFuture(loop=self.loop) + + +class PyFutureDoneCallbackTests(BaseFutureDoneCallbackTests, + test_utils.TestCase): + + def _new_future(self): + return futures._PyFuture(loop=self.loop) + + +class BaseFutureInheritanceTests: + + def _get_future_cls(self): + raise NotImplementedError + + def setUp(self): + super().setUp() + self.loop = self.new_test_loop() + self.addCleanup(self.loop.close) + + def test_inherit_without_calling_super_init(self): + # See https://bugs.python.org/issue38785 for the context + cls = self._get_future_cls() + + class MyFut(cls): + def __init__(self, *args, **kwargs): + # don't call super().__init__() + pass + + fut = MyFut(loop=self.loop) + with self.assertRaisesRegex( + RuntimeError, + "Future object is not initialized." + ): + fut.get_loop() + + +class PyFutureInheritanceTests(BaseFutureInheritanceTests, + test_utils.TestCase): + def _get_future_cls(self): + return futures._PyFuture + + +@unittest.skipUnless(hasattr(futures, '_CFuture'), + 'requires the C _asyncio module') +class CFutureInheritanceTests(BaseFutureInheritanceTests, + test_utils.TestCase): + def _get_future_cls(self): + return futures._CFuture + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_futures2.py b/Lib/test/test_asyncio/test_futures2.py new file mode 100644 index 00000000000..c7c0ebdac1b --- /dev/null +++ b/Lib/test/test_asyncio/test_futures2.py @@ -0,0 +1,95 @@ +# IsolatedAsyncioTestCase based tests +import asyncio +import contextvars +import traceback +import unittest +from asyncio import tasks + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +class FutureTests: + + async def test_future_traceback(self): + + async def raise_exc(): + raise TypeError(42) + + future = self.cls(raise_exc()) + + for _ in range(5): + try: + await future + except TypeError as e: + tb = ''.join(traceback.format_tb(e.__traceback__)) + self.assertEqual(tb.count("await future"), 1) + else: + self.fail('TypeError was not raised') + + async def test_task_exc_handler_correct_context(self): + # see https://github.com/python/cpython/issues/96704 + name = contextvars.ContextVar('name', default='foo') + exc_handler_called = False + + def exc_handler(*args): + self.assertEqual(name.get(), 'bar') + nonlocal exc_handler_called + exc_handler_called = True + + async def task(): + name.set('bar') + 1/0 + + loop = asyncio.get_running_loop() + loop.set_exception_handler(exc_handler) + self.cls(task()) + await asyncio.sleep(0) + self.assertTrue(exc_handler_called) + + async def test_handle_exc_handler_correct_context(self): + # see https://github.com/python/cpython/issues/96704 + name = contextvars.ContextVar('name', default='foo') + exc_handler_called = False + + def exc_handler(*args): + self.assertEqual(name.get(), 'bar') + nonlocal exc_handler_called + exc_handler_called = True + + def callback(): + name.set('bar') + 1/0 + + loop = asyncio.get_running_loop() + loop.set_exception_handler(exc_handler) + loop.call_soon(callback) + await asyncio.sleep(0) + self.assertTrue(exc_handler_called) + +@unittest.skipUnless(hasattr(tasks, '_CTask'), + 'requires the C _asyncio module') +class CFutureTests(FutureTests, unittest.IsolatedAsyncioTestCase): + cls = tasks._CTask + +class PyFutureTests(FutureTests, unittest.IsolatedAsyncioTestCase): + cls = tasks._PyTask + +class FutureReprTests(unittest.IsolatedAsyncioTestCase): + + async def test_recursive_repr_for_pending_tasks(self): + # The call crashes if the guard for recursive call + # in base_futures:_future_repr_info is absent + # See Also: https://bugs.python.org/issue42183 + + async def func(): + return asyncio.all_tasks() + + # The repr() call should not raise RecursionError at first. + waiter = await asyncio.wait_for(asyncio.Task(func()),timeout=10) + self.assertIn('...', repr(waiter)) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_graph.py b/Lib/test/test_asyncio/test_graph.py new file mode 100644 index 00000000000..2f22fbccba4 --- /dev/null +++ b/Lib/test/test_asyncio/test_graph.py @@ -0,0 +1,445 @@ +import asyncio +import io +import unittest + + +# To prevent a warning "test altered the execution environment" +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +def capture_test_stack(*, fut=None, depth=1): + + def walk(s): + ret = [ + (f"T<{n}>" if '-' not in (n := s.future.get_name()) else 'T') + if isinstance(s.future, asyncio.Task) else 'F' + ] + + ret.append( + [ + ( + f"s {entry.frame.f_code.co_name}" + if entry.frame.f_generator is None else + ( + f"a {entry.frame.f_generator.cr_code.co_name}" + if hasattr(entry.frame.f_generator, 'cr_code') else + f"ag {entry.frame.f_generator.ag_code.co_name}" + ) + ) for entry in s.call_stack + ] + ) + + ret.append( + sorted([ + walk(ab) for ab in s.awaited_by + ], key=lambda entry: entry[0]) + ) + + return ret + + buf = io.StringIO() + asyncio.print_call_graph(fut, file=buf, depth=depth+1) + + stack = asyncio.capture_call_graph(fut, depth=depth) + return walk(stack), buf.getvalue() + + +class CallStackTestBase: + + async def test_stack_tgroup(self): + + stack_for_c5 = None + + def c5(): + nonlocal stack_for_c5 + stack_for_c5 = capture_test_stack(depth=2) + + async def c4(): + await asyncio.sleep(0) + c5() + + async def c3(): + await c4() + + async def c2(): + await c3() + + async def c1(task): + await task + + async def main(): + async with asyncio.TaskGroup() as tg: + task = tg.create_task(c2(), name="c2_root") + tg.create_task(c1(task), name="sub_main_1") + tg.create_task(c1(task), name="sub_main_2") + + await main() + + self.assertEqual(stack_for_c5[0], [ + # task name + 'T', + # call stack + ['s c5', 'a c4', 'a c3', 'a c2'], + # awaited by + [ + ['T', + ['a _aexit', 'a __aexit__', 'a main', 'a test_stack_tgroup'], [] + ], + ['T', + ['a c1'], + [ + ['T', + ['a _aexit', 'a __aexit__', 'a main', 'a test_stack_tgroup'], [] + ] + ] + ], + ['T', + ['a c1'], + [ + ['T', + ['a _aexit', 'a __aexit__', 'a main', 'a test_stack_tgroup'], [] + ] + ] + ] + ] + ]) + + self.assertIn( + ' async CallStackTestBase.test_stack_tgroup()', + stack_for_c5[1]) + + + async def test_stack_async_gen(self): + + stack_for_gen_nested_call = None + + async def gen_nested_call(): + nonlocal stack_for_gen_nested_call + stack_for_gen_nested_call = capture_test_stack() + + async def gen(): + for num in range(2): + yield num + if num == 1: + await gen_nested_call() + + async def main(): + async for el in gen(): + pass + + await main() + + self.assertEqual(stack_for_gen_nested_call[0], [ + 'T', + [ + 's capture_test_stack', + 'a gen_nested_call', + 'ag gen', + 'a main', + 'a test_stack_async_gen' + ], + [] + ]) + + self.assertIn( + 'async generator CallStackTestBase.test_stack_async_gen..gen()', + stack_for_gen_nested_call[1]) + + async def test_stack_gather(self): + + stack_for_deep = None + + async def deep(): + await asyncio.sleep(0) + nonlocal stack_for_deep + stack_for_deep = capture_test_stack() + + async def c1(): + await asyncio.sleep(0) + await deep() + + async def c2(): + await asyncio.sleep(0) + + async def main(): + await asyncio.gather(c1(), c2()) + + await main() + + self.assertEqual(stack_for_deep[0], [ + 'T', + ['s capture_test_stack', 'a deep', 'a c1'], + [ + ['T', ['a main', 'a test_stack_gather'], []] + ] + ]) + + async def test_stack_shield(self): + + stack_for_shield = None + + async def deep(): + await asyncio.sleep(0) + nonlocal stack_for_shield + stack_for_shield = capture_test_stack() + + async def c1(): + await asyncio.sleep(0) + await deep() + + async def main(): + await asyncio.shield(c1()) + + await main() + + self.assertEqual(stack_for_shield[0], [ + 'T', + ['s capture_test_stack', 'a deep', 'a c1'], + [ + ['T', ['a main', 'a test_stack_shield'], []] + ] + ]) + + async def test_stack_timeout(self): + + stack_for_inner = None + + async def inner(): + await asyncio.sleep(0) + nonlocal stack_for_inner + stack_for_inner = capture_test_stack() + + async def c1(): + async with asyncio.timeout(1): + await asyncio.sleep(0) + await inner() + + async def main(): + await asyncio.shield(c1()) + + await main() + + self.assertEqual(stack_for_inner[0], [ + 'T', + ['s capture_test_stack', 'a inner', 'a c1'], + [ + ['T', ['a main', 'a test_stack_timeout'], []] + ] + ]) + + async def test_stack_wait(self): + + stack_for_inner = None + + async def inner(): + await asyncio.sleep(0) + nonlocal stack_for_inner + stack_for_inner = capture_test_stack() + + async def c1(): + async with asyncio.timeout(1): + await asyncio.sleep(0) + await inner() + + async def c2(): + for i in range(3): + await asyncio.sleep(0) + + async def main(t1, t2): + while True: + _, pending = await asyncio.wait([t1, t2]) + if not pending: + break + + t1 = asyncio.create_task(c1()) + t2 = asyncio.create_task(c2()) + try: + await main(t1, t2) + finally: + await t1 + await t2 + + self.assertEqual(stack_for_inner[0], [ + 'T', + ['s capture_test_stack', 'a inner', 'a c1'], + [ + ['T', + ['a _wait', 'a wait', 'a main', 'a test_stack_wait'], + [] + ] + ] + ]) + + async def test_stack_task(self): + + stack_for_inner = None + + async def inner(): + await asyncio.sleep(0) + nonlocal stack_for_inner + stack_for_inner = capture_test_stack() + + async def c1(): + await inner() + + async def c2(): + await asyncio.create_task(c1(), name='there there') + + async def main(): + await c2() + + await main() + + self.assertEqual(stack_for_inner[0], [ + 'T', + ['s capture_test_stack', 'a inner', 'a c1'], + [['T', ['a c2', 'a main', 'a test_stack_task'], []]] + ]) + + async def test_stack_future(self): + + stack_for_fut = None + + async def a2(fut): + await fut + + async def a1(fut): + await a2(fut) + + async def b1(fut): + await fut + + async def main(): + nonlocal stack_for_fut + + fut = asyncio.Future() + async with asyncio.TaskGroup() as g: + g.create_task(a1(fut), name="task A") + g.create_task(b1(fut), name='task B') + + for _ in range(5): + # Do a few iterations to ensure that both a1 and b1 + # await on the future + await asyncio.sleep(0) + + stack_for_fut = capture_test_stack(fut=fut) + fut.set_result(None) + + await main() + + self.assertEqual(stack_for_fut[0], + ['F', + [], + [ + ['T', + ['a a2', 'a a1'], + [['T', ['a test_stack_future'], []]] + ], + ['T', + ['a b1'], + [['T', ['a test_stack_future'], []]] + ], + ]] + ) + + self.assertTrue(stack_for_fut[1].startswith('* Future(id=')) + + +@unittest.skipIf( + not hasattr(asyncio.futures, "_c_future_add_to_awaited_by"), + "C-accelerated asyncio call graph backend missing", +) +class TestCallStackC(CallStackTestBase, unittest.IsolatedAsyncioTestCase): + def setUp(self): + futures = asyncio.futures + tasks = asyncio.tasks + + self._Future = asyncio.Future + asyncio.Future = futures.Future = futures._CFuture + + self._Task = asyncio.Task + asyncio.Task = tasks.Task = tasks._CTask + + self._future_add_to_awaited_by = asyncio.future_add_to_awaited_by + futures.future_add_to_awaited_by = futures._c_future_add_to_awaited_by + asyncio.future_add_to_awaited_by = futures.future_add_to_awaited_by + + self._future_discard_from_awaited_by = asyncio.future_discard_from_awaited_by + futures.future_discard_from_awaited_by = futures._c_future_discard_from_awaited_by + asyncio.future_discard_from_awaited_by = futures.future_discard_from_awaited_by + + self._current_task = asyncio.current_task + asyncio.current_task = asyncio.tasks.current_task = tasks._c_current_task + + def tearDown(self): + futures = asyncio.futures + tasks = asyncio.tasks + + futures.future_discard_from_awaited_by = self._future_discard_from_awaited_by + asyncio.future_discard_from_awaited_by = self._future_discard_from_awaited_by + del self._future_discard_from_awaited_by + + futures.future_add_to_awaited_by = self._future_add_to_awaited_by + asyncio.future_add_to_awaited_by = self._future_add_to_awaited_by + del self._future_add_to_awaited_by + + asyncio.Task = self._Task + tasks.Task = self._Task + del self._Task + + asyncio.Future = self._Future + futures.Future = self._Future + del self._Future + + asyncio.current_task = asyncio.tasks.current_task = self._current_task + + +@unittest.skipIf( + not hasattr(asyncio.futures, "_py_future_add_to_awaited_by"), + "Pure Python asyncio call graph backend missing", +) +class TestCallStackPy(CallStackTestBase, unittest.IsolatedAsyncioTestCase): + def setUp(self): + futures = asyncio.futures + tasks = asyncio.tasks + + self._Future = asyncio.Future + asyncio.Future = futures.Future = futures._PyFuture + + self._Task = asyncio.Task + asyncio.Task = tasks.Task = tasks._PyTask + + self._future_add_to_awaited_by = asyncio.future_add_to_awaited_by + futures.future_add_to_awaited_by = futures._py_future_add_to_awaited_by + asyncio.future_add_to_awaited_by = futures.future_add_to_awaited_by + + self._future_discard_from_awaited_by = asyncio.future_discard_from_awaited_by + futures.future_discard_from_awaited_by = futures._py_future_discard_from_awaited_by + asyncio.future_discard_from_awaited_by = futures.future_discard_from_awaited_by + + self._current_task = asyncio.current_task + asyncio.current_task = asyncio.tasks.current_task = tasks._py_current_task + + + def tearDown(self): + futures = asyncio.futures + tasks = asyncio.tasks + + futures.future_discard_from_awaited_by = self._future_discard_from_awaited_by + asyncio.future_discard_from_awaited_by = self._future_discard_from_awaited_by + del self._future_discard_from_awaited_by + + futures.future_add_to_awaited_by = self._future_add_to_awaited_by + asyncio.future_add_to_awaited_by = self._future_add_to_awaited_by + del self._future_add_to_awaited_by + + asyncio.Task = self._Task + tasks.Task = self._Task + del self._Task + + asyncio.Future = self._Future + futures.Future = self._Future + del self._Future + + asyncio.current_task = asyncio.tasks.current_task = self._current_task diff --git a/Lib/test/test_asyncio/test_locks.py b/Lib/test/test_asyncio/test_locks.py new file mode 100644 index 00000000000..e025d2990a3 --- /dev/null +++ b/Lib/test/test_asyncio/test_locks.py @@ -0,0 +1,1825 @@ +"""Tests for locks.py""" + +import unittest +from unittest import mock +import re + +import asyncio +import collections + +STR_RGX_REPR = ( + r'^<(?P.*?) object at (?P
    .*?)' + r'\[(?P' + r'(set|unset|locked|unlocked|filling|draining|resetting|broken)' + r'(, value:\d)?' + r'(, waiters:\d+)?' + r'(, waiters:\d+\/\d+)?' # barrier + r')\]>\z' +) +RGX_REPR = re.compile(STR_RGX_REPR) + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +class LockTests(unittest.IsolatedAsyncioTestCase): + + async def test_repr(self): + lock = asyncio.Lock() + self.assertEndsWith(repr(lock), '[unlocked]>') + self.assertTrue(RGX_REPR.match(repr(lock))) + + await lock.acquire() + self.assertEndsWith(repr(lock), '[locked]>') + self.assertTrue(RGX_REPR.match(repr(lock))) + + async def test_lock(self): + lock = asyncio.Lock() + + with self.assertRaisesRegex( + TypeError, + "'Lock' object can't be awaited" + ): + await lock + + self.assertFalse(lock.locked()) + + async def test_lock_doesnt_accept_loop_parameter(self): + primitives_cls = [ + asyncio.Lock, + asyncio.Condition, + asyncio.Event, + asyncio.Semaphore, + asyncio.BoundedSemaphore, + ] + + loop = asyncio.get_running_loop() + + for cls in primitives_cls: + with self.assertRaisesRegex( + TypeError, + rf"{cls.__name__}\.__init__\(\) got an unexpected " + rf"keyword argument 'loop'" + ): + cls(loop=loop) + + async def test_lock_by_with_statement(self): + primitives = [ + asyncio.Lock(), + asyncio.Condition(), + asyncio.Semaphore(), + asyncio.BoundedSemaphore(), + ] + + for lock in primitives: + await asyncio.sleep(0.01) + self.assertFalse(lock.locked()) + with self.assertRaisesRegex( + TypeError, + r"'\w+' object can't be awaited" + ): + with await lock: + pass + self.assertFalse(lock.locked()) + + async def test_acquire(self): + lock = asyncio.Lock() + result = [] + + self.assertTrue(await lock.acquire()) + + async def c1(result): + if await lock.acquire(): + result.append(1) + return True + + async def c2(result): + if await lock.acquire(): + result.append(2) + return True + + async def c3(result): + if await lock.acquire(): + result.append(3) + return True + + t1 = asyncio.create_task(c1(result)) + t2 = asyncio.create_task(c2(result)) + + await asyncio.sleep(0) + self.assertEqual([], result) + + lock.release() + await asyncio.sleep(0) + self.assertEqual([1], result) + + await asyncio.sleep(0) + self.assertEqual([1], result) + + t3 = asyncio.create_task(c3(result)) + + lock.release() + await asyncio.sleep(0) + self.assertEqual([1, 2], result) + + lock.release() + await asyncio.sleep(0) + self.assertEqual([1, 2, 3], result) + + self.assertTrue(t1.done()) + self.assertTrue(t1.result()) + self.assertTrue(t2.done()) + self.assertTrue(t2.result()) + self.assertTrue(t3.done()) + self.assertTrue(t3.result()) + + async def test_acquire_cancel(self): + lock = asyncio.Lock() + self.assertTrue(await lock.acquire()) + + task = asyncio.create_task(lock.acquire()) + asyncio.get_running_loop().call_soon(task.cancel) + with self.assertRaises(asyncio.CancelledError): + await task + self.assertFalse(lock._waiters) + + async def test_cancel_race(self): + # Several tasks: + # - A acquires the lock + # - B is blocked in acquire() + # - C is blocked in acquire() + # + # Now, concurrently: + # - B is cancelled + # - A releases the lock + # + # If B's waiter is marked cancelled but not yet removed from + # _waiters, A's release() call will crash when trying to set + # B's waiter; instead, it should move on to C's waiter. + + # Setup: A has the lock, b and c are waiting. + lock = asyncio.Lock() + + async def lockit(name, blocker): + await lock.acquire() + try: + if blocker is not None: + await blocker + finally: + lock.release() + + fa = asyncio.get_running_loop().create_future() + ta = asyncio.create_task(lockit('A', fa)) + await asyncio.sleep(0) + self.assertTrue(lock.locked()) + tb = asyncio.create_task(lockit('B', None)) + await asyncio.sleep(0) + self.assertEqual(len(lock._waiters), 1) + tc = asyncio.create_task(lockit('C', None)) + await asyncio.sleep(0) + self.assertEqual(len(lock._waiters), 2) + + # Create the race and check. + # Without the fix this failed at the last assert. + fa.set_result(None) + tb.cancel() + self.assertTrue(lock._waiters[0].cancelled()) + await asyncio.sleep(0) + self.assertFalse(lock.locked()) + self.assertTrue(ta.done()) + self.assertTrue(tb.cancelled()) + await tc + + async def test_cancel_release_race(self): + # Issue 32734 + # Acquire 4 locks, cancel second, release first + # and 2 locks are taken at once. + loop = asyncio.get_running_loop() + lock = asyncio.Lock() + lock_count = 0 + call_count = 0 + + async def lockit(): + nonlocal lock_count + nonlocal call_count + call_count += 1 + await lock.acquire() + lock_count += 1 + + def trigger(): + t1.cancel() + lock.release() + + await lock.acquire() + + t1 = asyncio.create_task(lockit()) + t2 = asyncio.create_task(lockit()) + t3 = asyncio.create_task(lockit()) + + # Start scheduled tasks + await asyncio.sleep(0) + + loop.call_soon(trigger) + with self.assertRaises(asyncio.CancelledError): + # Wait for cancellation + await t1 + + # Make sure only one lock was taken + self.assertEqual(lock_count, 1) + # While 3 calls were made to lockit() + self.assertEqual(call_count, 3) + self.assertTrue(t1.cancelled() and t2.done()) + + # Cleanup the task that is stuck on acquire. + t3.cancel() + await asyncio.sleep(0) + self.assertTrue(t3.cancelled()) + + async def test_finished_waiter_cancelled(self): + lock = asyncio.Lock() + + await lock.acquire() + self.assertTrue(lock.locked()) + + tb = asyncio.create_task(lock.acquire()) + await asyncio.sleep(0) + self.assertEqual(len(lock._waiters), 1) + + # Create a second waiter, wake up the first, and cancel it. + # Without the fix, the second was not woken up. + tc = asyncio.create_task(lock.acquire()) + tb.cancel() + lock.release() + await asyncio.sleep(0) + + self.assertTrue(lock.locked()) + self.assertTrue(tb.cancelled()) + + # Cleanup + await tc + + async def test_release_not_acquired(self): + lock = asyncio.Lock() + + self.assertRaises(RuntimeError, lock.release) + + async def test_release_no_waiters(self): + lock = asyncio.Lock() + await lock.acquire() + self.assertTrue(lock.locked()) + + lock.release() + self.assertFalse(lock.locked()) + + async def test_context_manager(self): + lock = asyncio.Lock() + self.assertFalse(lock.locked()) + + async with lock: + self.assertTrue(lock.locked()) + + self.assertFalse(lock.locked()) + + +class EventTests(unittest.IsolatedAsyncioTestCase): + + def test_repr(self): + ev = asyncio.Event() + self.assertEndsWith(repr(ev), '[unset]>') + match = RGX_REPR.match(repr(ev)) + self.assertEqual(match.group('extras'), 'unset') + + ev.set() + self.assertEndsWith(repr(ev), '[set]>') + self.assertTrue(RGX_REPR.match(repr(ev))) + + ev._waiters.append(mock.Mock()) + self.assertTrue('waiters:1' in repr(ev)) + self.assertTrue(RGX_REPR.match(repr(ev))) + + async def test_wait(self): + ev = asyncio.Event() + self.assertFalse(ev.is_set()) + + result = [] + + async def c1(result): + if await ev.wait(): + result.append(1) + + async def c2(result): + if await ev.wait(): + result.append(2) + + async def c3(result): + if await ev.wait(): + result.append(3) + + t1 = asyncio.create_task(c1(result)) + t2 = asyncio.create_task(c2(result)) + + await asyncio.sleep(0) + self.assertEqual([], result) + + t3 = asyncio.create_task(c3(result)) + + ev.set() + await asyncio.sleep(0) + self.assertEqual([3, 1, 2], result) + + self.assertTrue(t1.done()) + self.assertIsNone(t1.result()) + self.assertTrue(t2.done()) + self.assertIsNone(t2.result()) + self.assertTrue(t3.done()) + self.assertIsNone(t3.result()) + + async def test_wait_on_set(self): + ev = asyncio.Event() + ev.set() + + res = await ev.wait() + self.assertTrue(res) + + async def test_wait_cancel(self): + ev = asyncio.Event() + + wait = asyncio.create_task(ev.wait()) + asyncio.get_running_loop().call_soon(wait.cancel) + with self.assertRaises(asyncio.CancelledError): + await wait + self.assertFalse(ev._waiters) + + async def test_clear(self): + ev = asyncio.Event() + self.assertFalse(ev.is_set()) + + ev.set() + self.assertTrue(ev.is_set()) + + ev.clear() + self.assertFalse(ev.is_set()) + + async def test_clear_with_waiters(self): + ev = asyncio.Event() + result = [] + + async def c1(result): + if await ev.wait(): + result.append(1) + return True + + t = asyncio.create_task(c1(result)) + await asyncio.sleep(0) + self.assertEqual([], result) + + ev.set() + ev.clear() + self.assertFalse(ev.is_set()) + + ev.set() + ev.set() + self.assertEqual(1, len(ev._waiters)) + + await asyncio.sleep(0) + self.assertEqual([1], result) + self.assertEqual(0, len(ev._waiters)) + + self.assertTrue(t.done()) + self.assertTrue(t.result()) + + +class ConditionTests(unittest.IsolatedAsyncioTestCase): + + async def test_wait(self): + cond = asyncio.Condition() + result = [] + + async def c1(result): + await cond.acquire() + if await cond.wait(): + result.append(1) + return True + + async def c2(result): + await cond.acquire() + if await cond.wait(): + result.append(2) + return True + + async def c3(result): + await cond.acquire() + if await cond.wait(): + result.append(3) + return True + + t1 = asyncio.create_task(c1(result)) + t2 = asyncio.create_task(c2(result)) + t3 = asyncio.create_task(c3(result)) + + await asyncio.sleep(0) + self.assertEqual([], result) + self.assertFalse(cond.locked()) + + self.assertTrue(await cond.acquire()) + cond.notify() + await asyncio.sleep(0) + self.assertEqual([], result) + self.assertTrue(cond.locked()) + + cond.release() + await asyncio.sleep(0) + self.assertEqual([1], result) + self.assertTrue(cond.locked()) + + cond.notify(2) + await asyncio.sleep(0) + self.assertEqual([1], result) + self.assertTrue(cond.locked()) + + cond.release() + await asyncio.sleep(0) + self.assertEqual([1, 2], result) + self.assertTrue(cond.locked()) + + cond.release() + await asyncio.sleep(0) + self.assertEqual([1, 2, 3], result) + self.assertTrue(cond.locked()) + + self.assertTrue(t1.done()) + self.assertTrue(t1.result()) + self.assertTrue(t2.done()) + self.assertTrue(t2.result()) + self.assertTrue(t3.done()) + self.assertTrue(t3.result()) + + async def test_wait_cancel(self): + cond = asyncio.Condition() + await cond.acquire() + + wait = asyncio.create_task(cond.wait()) + asyncio.get_running_loop().call_soon(wait.cancel) + with self.assertRaises(asyncio.CancelledError): + await wait + self.assertFalse(cond._waiters) + self.assertTrue(cond.locked()) + + async def test_wait_cancel_contested(self): + cond = asyncio.Condition() + + await cond.acquire() + self.assertTrue(cond.locked()) + + wait_task = asyncio.create_task(cond.wait()) + await asyncio.sleep(0) + self.assertFalse(cond.locked()) + + # Notify, but contest the lock before cancelling + await cond.acquire() + self.assertTrue(cond.locked()) + cond.notify() + asyncio.get_running_loop().call_soon(wait_task.cancel) + asyncio.get_running_loop().call_soon(cond.release) + + try: + await wait_task + except asyncio.CancelledError: + # Should not happen, since no cancellation points + pass + + self.assertTrue(cond.locked()) + + async def test_wait_cancel_after_notify(self): + # See bpo-32841 + waited = False + + cond = asyncio.Condition() + + async def wait_on_cond(): + nonlocal waited + async with cond: + waited = True # Make sure this area was reached + await cond.wait() + + waiter = asyncio.create_task(wait_on_cond()) + await asyncio.sleep(0) # Start waiting + + await cond.acquire() + cond.notify() + await asyncio.sleep(0) # Get to acquire() + waiter.cancel() + await asyncio.sleep(0) # Activate cancellation + cond.release() + await asyncio.sleep(0) # Cancellation should occur + + self.assertTrue(waiter.cancelled()) + self.assertTrue(waited) + + async def test_wait_unacquired(self): + cond = asyncio.Condition() + with self.assertRaises(RuntimeError): + await cond.wait() + + async def test_wait_for(self): + cond = asyncio.Condition() + presult = False + + def predicate(): + return presult + + result = [] + + async def c1(result): + await cond.acquire() + if await cond.wait_for(predicate): + result.append(1) + cond.release() + return True + + t = asyncio.create_task(c1(result)) + + await asyncio.sleep(0) + self.assertEqual([], result) + + await cond.acquire() + cond.notify() + cond.release() + await asyncio.sleep(0) + self.assertEqual([], result) + + presult = True + await cond.acquire() + cond.notify() + cond.release() + await asyncio.sleep(0) + self.assertEqual([1], result) + + self.assertTrue(t.done()) + self.assertTrue(t.result()) + + async def test_wait_for_unacquired(self): + cond = asyncio.Condition() + + # predicate can return true immediately + res = await cond.wait_for(lambda: [1, 2, 3]) + self.assertEqual([1, 2, 3], res) + + with self.assertRaises(RuntimeError): + await cond.wait_for(lambda: False) + + async def test_notify(self): + cond = asyncio.Condition() + result = [] + + async def c1(result): + await cond.acquire() + if await cond.wait(): + result.append(1) + cond.release() + return True + + async def c2(result): + await cond.acquire() + if await cond.wait(): + result.append(2) + cond.release() + return True + + async def c3(result): + await cond.acquire() + if await cond.wait(): + result.append(3) + cond.release() + return True + + t1 = asyncio.create_task(c1(result)) + t2 = asyncio.create_task(c2(result)) + t3 = asyncio.create_task(c3(result)) + + await asyncio.sleep(0) + self.assertEqual([], result) + + await cond.acquire() + cond.notify(1) + cond.release() + await asyncio.sleep(0) + self.assertEqual([1], result) + + await cond.acquire() + cond.notify(1) + cond.notify(2048) + cond.release() + await asyncio.sleep(0) + self.assertEqual([1, 2, 3], result) + + self.assertTrue(t1.done()) + self.assertTrue(t1.result()) + self.assertTrue(t2.done()) + self.assertTrue(t2.result()) + self.assertTrue(t3.done()) + self.assertTrue(t3.result()) + + async def test_notify_all(self): + cond = asyncio.Condition() + + result = [] + + async def c1(result): + await cond.acquire() + if await cond.wait(): + result.append(1) + cond.release() + return True + + async def c2(result): + await cond.acquire() + if await cond.wait(): + result.append(2) + cond.release() + return True + + t1 = asyncio.create_task(c1(result)) + t2 = asyncio.create_task(c2(result)) + + await asyncio.sleep(0) + self.assertEqual([], result) + + await cond.acquire() + cond.notify_all() + cond.release() + await asyncio.sleep(0) + self.assertEqual([1, 2], result) + + self.assertTrue(t1.done()) + self.assertTrue(t1.result()) + self.assertTrue(t2.done()) + self.assertTrue(t2.result()) + + def test_notify_unacquired(self): + cond = asyncio.Condition() + self.assertRaises(RuntimeError, cond.notify) + + def test_notify_all_unacquired(self): + cond = asyncio.Condition() + self.assertRaises(RuntimeError, cond.notify_all) + + async def test_repr(self): + cond = asyncio.Condition() + self.assertTrue('unlocked' in repr(cond)) + self.assertTrue(RGX_REPR.match(repr(cond))) + + await cond.acquire() + self.assertTrue('locked' in repr(cond)) + + cond._waiters.append(mock.Mock()) + self.assertTrue('waiters:1' in repr(cond)) + self.assertTrue(RGX_REPR.match(repr(cond))) + + cond._waiters.append(mock.Mock()) + self.assertTrue('waiters:2' in repr(cond)) + self.assertTrue(RGX_REPR.match(repr(cond))) + + async def test_context_manager(self): + cond = asyncio.Condition() + self.assertFalse(cond.locked()) + async with cond: + self.assertTrue(cond.locked()) + self.assertFalse(cond.locked()) + + async def test_explicit_lock(self): + async def f(lock=None, cond=None): + if lock is None: + lock = asyncio.Lock() + if cond is None: + cond = asyncio.Condition(lock) + self.assertIs(cond._lock, lock) + self.assertFalse(lock.locked()) + self.assertFalse(cond.locked()) + async with cond: + self.assertTrue(lock.locked()) + self.assertTrue(cond.locked()) + self.assertFalse(lock.locked()) + self.assertFalse(cond.locked()) + async with lock: + self.assertTrue(lock.locked()) + self.assertTrue(cond.locked()) + self.assertFalse(lock.locked()) + self.assertFalse(cond.locked()) + + # All should work in the same way. + await f() + await f(asyncio.Lock()) + lock = asyncio.Lock() + await f(lock, asyncio.Condition(lock)) + + async def test_ambiguous_loops(self): + loop = asyncio.new_event_loop() + self.addCleanup(loop.close) + + async def wrong_loop_in_lock(): + with self.assertRaises(TypeError): + asyncio.Lock(loop=loop) # actively disallowed since 3.10 + lock = asyncio.Lock() + lock._loop = loop # use private API for testing + async with lock: + # acquired immediately via the fast-path + # without interaction with any event loop. + cond = asyncio.Condition(lock) + # cond.acquire() will trigger waiting on the lock + # and it will discover the event loop mismatch. + with self.assertRaisesRegex( + RuntimeError, + "is bound to a different event loop", + ): + await cond.acquire() + + async def wrong_loop_in_cond(): + # Same analogy here with the condition's loop. + lock = asyncio.Lock() + async with lock: + with self.assertRaises(TypeError): + asyncio.Condition(lock, loop=loop) + cond = asyncio.Condition(lock) + cond._loop = loop + with self.assertRaisesRegex( + RuntimeError, + "is bound to a different event loop", + ): + await cond.wait() + + await wrong_loop_in_lock() + await wrong_loop_in_cond() + + async def test_timeout_in_block(self): + condition = asyncio.Condition() + async with condition: + with self.assertRaises(asyncio.TimeoutError): + await asyncio.wait_for(condition.wait(), timeout=0.5) + + async def test_cancelled_error_wakeup(self): + # Test that a cancelled error, received when awaiting wakeup, + # will be re-raised un-modified. + wake = False + raised = None + cond = asyncio.Condition() + + async def func(): + nonlocal raised + async with cond: + with self.assertRaises(asyncio.CancelledError) as err: + await cond.wait_for(lambda: wake) + raised = err.exception + raise raised + + task = asyncio.create_task(func()) + await asyncio.sleep(0) + # Task is waiting on the condition, cancel it there. + task.cancel(msg="foo") + with self.assertRaises(asyncio.CancelledError) as err: + await task + self.assertEqual(err.exception.args, ("foo",)) + # We should have got the _same_ exception instance as the one + # originally raised. + self.assertIs(err.exception, raised) + + async def test_cancelled_error_re_aquire(self): + # Test that a cancelled error, received when re-aquiring lock, + # will be re-raised un-modified. + wake = False + raised = None + cond = asyncio.Condition() + + async def func(): + nonlocal raised + async with cond: + with self.assertRaises(asyncio.CancelledError) as err: + await cond.wait_for(lambda: wake) + raised = err.exception + raise raised + + task = asyncio.create_task(func()) + await asyncio.sleep(0) + # Task is waiting on the condition + await cond.acquire() + wake = True + cond.notify() + await asyncio.sleep(0) + # Task is now trying to re-acquire the lock, cancel it there. + task.cancel(msg="foo") + cond.release() + with self.assertRaises(asyncio.CancelledError) as err: + await task + self.assertEqual(err.exception.args, ("foo",)) + # We should have got the _same_ exception instance as the one + # originally raised. + self.assertIs(err.exception, raised) + + async def test_cancelled_wakeup(self): + # Test that a task cancelled at the "same" time as it is woken + # up as part of a Condition.notify() does not result in a lost wakeup. + # This test simulates a cancel while the target task is awaiting initial + # wakeup on the wakeup queue. + condition = asyncio.Condition() + state = 0 + async def consumer(): + nonlocal state + async with condition: + while True: + await condition.wait_for(lambda: state != 0) + if state < 0: + return + state -= 1 + + # create two consumers + c = [asyncio.create_task(consumer()) for _ in range(2)] + # wait for them to settle + await asyncio.sleep(0) + async with condition: + # produce one item and wake up one + state += 1 + condition.notify(1) + + # Cancel it while it is awaiting to be run. + # This cancellation could come from the outside + c[0].cancel() + + # now wait for the item to be consumed + # if it doesn't means that our "notify" didn"t take hold. + # because it raced with a cancel() + try: + async with asyncio.timeout(0.01): + await condition.wait_for(lambda: state == 0) + except TimeoutError: + pass + self.assertEqual(state, 0) + + # clean up + state = -1 + condition.notify_all() + await c[1] + + async def test_cancelled_wakeup_relock(self): + # Test that a task cancelled at the "same" time as it is woken + # up as part of a Condition.notify() does not result in a lost wakeup. + # This test simulates a cancel while the target task is acquiring the lock + # again. + condition = asyncio.Condition() + state = 0 + async def consumer(): + nonlocal state + async with condition: + while True: + await condition.wait_for(lambda: state != 0) + if state < 0: + return + state -= 1 + + # create two consumers + c = [asyncio.create_task(consumer()) for _ in range(2)] + # wait for them to settle + await asyncio.sleep(0) + async with condition: + # produce one item and wake up one + state += 1 + condition.notify(1) + + # now we sleep for a bit. This allows the target task to wake up and + # settle on re-aquiring the lock + await asyncio.sleep(0) + + # Cancel it while awaiting the lock + # This cancel could come the outside. + c[0].cancel() + + # now wait for the item to be consumed + # if it doesn't means that our "notify" didn"t take hold. + # because it raced with a cancel() + try: + async with asyncio.timeout(0.01): + await condition.wait_for(lambda: state == 0) + except TimeoutError: + pass + self.assertEqual(state, 0) + + # clean up + state = -1 + condition.notify_all() + await c[1] + +class SemaphoreTests(unittest.IsolatedAsyncioTestCase): + + def test_initial_value_zero(self): + sem = asyncio.Semaphore(0) + self.assertTrue(sem.locked()) + + async def test_repr(self): + sem = asyncio.Semaphore() + self.assertEndsWith(repr(sem), '[unlocked, value:1]>') + self.assertTrue(RGX_REPR.match(repr(sem))) + + await sem.acquire() + self.assertEndsWith(repr(sem), '[locked]>') + self.assertTrue('waiters' not in repr(sem)) + self.assertTrue(RGX_REPR.match(repr(sem))) + + if sem._waiters is None: + sem._waiters = collections.deque() + + sem._waiters.append(mock.Mock()) + self.assertTrue('waiters:1' in repr(sem)) + self.assertTrue(RGX_REPR.match(repr(sem))) + + sem._waiters.append(mock.Mock()) + self.assertTrue('waiters:2' in repr(sem)) + self.assertTrue(RGX_REPR.match(repr(sem))) + + async def test_semaphore(self): + sem = asyncio.Semaphore() + self.assertEqual(1, sem._value) + + with self.assertRaisesRegex( + TypeError, + "'Semaphore' object can't be awaited", + ): + await sem + + self.assertFalse(sem.locked()) + self.assertEqual(1, sem._value) + + def test_semaphore_value(self): + self.assertRaises(ValueError, asyncio.Semaphore, -1) + + async def test_acquire(self): + sem = asyncio.Semaphore(3) + result = [] + + self.assertTrue(await sem.acquire()) + self.assertTrue(await sem.acquire()) + self.assertFalse(sem.locked()) + + async def c1(result): + await sem.acquire() + result.append(1) + return True + + async def c2(result): + await sem.acquire() + result.append(2) + return True + + async def c3(result): + await sem.acquire() + result.append(3) + return True + + async def c4(result): + await sem.acquire() + result.append(4) + return True + + t1 = asyncio.create_task(c1(result)) + t2 = asyncio.create_task(c2(result)) + t3 = asyncio.create_task(c3(result)) + + await asyncio.sleep(0) + self.assertEqual([1], result) + self.assertTrue(sem.locked()) + self.assertEqual(2, len(sem._waiters)) + self.assertEqual(0, sem._value) + + t4 = asyncio.create_task(c4(result)) + + sem.release() + sem.release() + self.assertEqual(0, sem._value) + + await asyncio.sleep(0) + self.assertEqual(0, sem._value) + self.assertEqual(3, len(result)) + self.assertTrue(sem.locked()) + self.assertEqual(1, len(sem._waiters)) + self.assertEqual(0, sem._value) + + self.assertTrue(t1.done()) + self.assertTrue(t1.result()) + race_tasks = [t2, t3, t4] + done_tasks = [t for t in race_tasks if t.done() and t.result()] + self.assertEqual(2, len(done_tasks)) + + # cleanup locked semaphore + sem.release() + await asyncio.gather(*race_tasks) + + async def test_acquire_cancel(self): + sem = asyncio.Semaphore() + await sem.acquire() + + acquire = asyncio.create_task(sem.acquire()) + asyncio.get_running_loop().call_soon(acquire.cancel) + with self.assertRaises(asyncio.CancelledError): + await acquire + self.assertTrue((not sem._waiters) or + all(waiter.done() for waiter in sem._waiters)) + + async def test_acquire_cancel_before_awoken(self): + sem = asyncio.Semaphore(value=0) + + t1 = asyncio.create_task(sem.acquire()) + t2 = asyncio.create_task(sem.acquire()) + t3 = asyncio.create_task(sem.acquire()) + t4 = asyncio.create_task(sem.acquire()) + + await asyncio.sleep(0) + + t1.cancel() + t2.cancel() + sem.release() + + await asyncio.sleep(0) + await asyncio.sleep(0) + num_done = sum(t.done() for t in [t3, t4]) + self.assertEqual(num_done, 1) + self.assertTrue(t3.done()) + self.assertFalse(t4.done()) + + t3.cancel() + t4.cancel() + await asyncio.sleep(0) + + async def test_acquire_hang(self): + sem = asyncio.Semaphore(value=0) + + t1 = asyncio.create_task(sem.acquire()) + t2 = asyncio.create_task(sem.acquire()) + await asyncio.sleep(0) + + t1.cancel() + sem.release() + await asyncio.sleep(0) + await asyncio.sleep(0) + self.assertTrue(sem.locked()) + self.assertTrue(t2.done()) + + async def test_acquire_no_hang(self): + + sem = asyncio.Semaphore(1) + + async def c1(): + async with sem: + await asyncio.sleep(0) + t2.cancel() + + async def c2(): + async with sem: + self.assertFalse(True) + + t1 = asyncio.create_task(c1()) + t2 = asyncio.create_task(c2()) + + r1, r2 = await asyncio.gather(t1, t2, return_exceptions=True) + self.assertTrue(r1 is None) + self.assertTrue(isinstance(r2, asyncio.CancelledError)) + + await asyncio.wait_for(sem.acquire(), timeout=1.0) + + def test_release_not_acquired(self): + sem = asyncio.BoundedSemaphore() + + self.assertRaises(ValueError, sem.release) + + async def test_release_no_waiters(self): + sem = asyncio.Semaphore() + await sem.acquire() + self.assertTrue(sem.locked()) + + sem.release() + self.assertFalse(sem.locked()) + + async def test_acquire_fifo_order(self): + sem = asyncio.Semaphore(1) + result = [] + + async def coro(tag): + await sem.acquire() + result.append(f'{tag}_1') + await asyncio.sleep(0.01) + sem.release() + + await sem.acquire() + result.append(f'{tag}_2') + await asyncio.sleep(0.01) + sem.release() + + async with asyncio.TaskGroup() as tg: + tg.create_task(coro('c1')) + tg.create_task(coro('c2')) + tg.create_task(coro('c3')) + + self.assertEqual( + ['c1_1', 'c2_1', 'c3_1', 'c1_2', 'c2_2', 'c3_2'], + result + ) + + async def test_acquire_fifo_order_2(self): + sem = asyncio.Semaphore(1) + result = [] + + async def c1(result): + await sem.acquire() + result.append(1) + return True + + async def c2(result): + await sem.acquire() + result.append(2) + sem.release() + await sem.acquire() + result.append(4) + return True + + async def c3(result): + await sem.acquire() + result.append(3) + return True + + t1 = asyncio.create_task(c1(result)) + t2 = asyncio.create_task(c2(result)) + t3 = asyncio.create_task(c3(result)) + + await asyncio.sleep(0) + + sem.release() + sem.release() + + tasks = [t1, t2, t3] + await asyncio.gather(*tasks) + self.assertEqual([1, 2, 3, 4], result) + + async def test_acquire_fifo_order_3(self): + sem = asyncio.Semaphore(0) + result = [] + + async def c1(result): + await sem.acquire() + result.append(1) + return True + + async def c2(result): + await sem.acquire() + result.append(2) + return True + + async def c3(result): + await sem.acquire() + result.append(3) + return True + + t1 = asyncio.create_task(c1(result)) + t2 = asyncio.create_task(c2(result)) + t3 = asyncio.create_task(c3(result)) + + await asyncio.sleep(0) + + t1.cancel() + + await asyncio.sleep(0) + + sem.release() + sem.release() + + tasks = [t1, t2, t3] + await asyncio.gather(*tasks, return_exceptions=True) + self.assertEqual([2, 3], result) + + async def test_acquire_fifo_order_4(self): + # Test that a successful `acquire()` will wake up multiple Tasks + # that were waiting in the Semaphore queue due to FIFO rules. + sem = asyncio.Semaphore(0) + result = [] + count = 0 + + async def c1(result): + # First task immediately waits for semaphore. It will be awoken by c2. + self.assertEqual(sem._value, 0) + await sem.acquire() + # We should have woken up all waiting tasks now. + self.assertEqual(sem._value, 0) + # Create a fourth task. It should run after c3, not c2. + nonlocal t4 + t4 = asyncio.create_task(c4(result)) + result.append(1) + return True + + async def c2(result): + # The second task begins by releasing semaphore three times, + # for c1, c2, and c3. + sem.release() + sem.release() + sem.release() + self.assertEqual(sem._value, 2) + # It is locked, because c1 hasn't woken up yet. + self.assertTrue(sem.locked()) + await sem.acquire() + result.append(2) + return True + + async def c3(result): + await sem.acquire() + self.assertTrue(sem.locked()) + result.append(3) + return True + + async def c4(result): + result.append(4) + return True + + t1 = asyncio.create_task(c1(result)) + t2 = asyncio.create_task(c2(result)) + t3 = asyncio.create_task(c3(result)) + t4 = None + + await asyncio.sleep(0) + # Three tasks are in the queue, the first hasn't woken up yet. + self.assertEqual(sem._value, 2) + self.assertEqual(len(sem._waiters), 3) + await asyncio.sleep(0) + + tasks = [t1, t2, t3, t4] + await asyncio.gather(*tasks) + self.assertEqual([1, 2, 3, 4], result) + +class BarrierTests(unittest.IsolatedAsyncioTestCase): + + async def asyncSetUp(self): + await super().asyncSetUp() + self.N = 5 + + def make_tasks(self, n, coro): + tasks = [asyncio.create_task(coro()) for _ in range(n)] + return tasks + + async def gather_tasks(self, n, coro): + tasks = self.make_tasks(n, coro) + res = await asyncio.gather(*tasks) + return res, tasks + + async def test_barrier(self): + barrier = asyncio.Barrier(self.N) + self.assertIn("filling", repr(barrier)) + with self.assertRaisesRegex( + TypeError, + "'Barrier' object can't be awaited", + ): + await barrier + + self.assertIn("filling", repr(barrier)) + + async def test_repr(self): + barrier = asyncio.Barrier(self.N) + + self.assertTrue(RGX_REPR.match(repr(barrier))) + self.assertIn("filling", repr(barrier)) + + waiters = [] + async def wait(barrier): + await barrier.wait() + + incr = 2 + for i in range(incr): + waiters.append(asyncio.create_task(wait(barrier))) + await asyncio.sleep(0) + + self.assertTrue(RGX_REPR.match(repr(barrier))) + self.assertTrue(f"waiters:{incr}/{self.N}" in repr(barrier)) + self.assertIn("filling", repr(barrier)) + + # create missing waiters + for i in range(barrier.parties - barrier.n_waiting): + waiters.append(asyncio.create_task(wait(barrier))) + await asyncio.sleep(0) + + self.assertTrue(RGX_REPR.match(repr(barrier))) + self.assertIn("draining", repr(barrier)) + + # add a part of waiters + for i in range(incr): + waiters.append(asyncio.create_task(wait(barrier))) + await asyncio.sleep(0) + # and reset + await barrier.reset() + + self.assertTrue(RGX_REPR.match(repr(barrier))) + self.assertIn("resetting", repr(barrier)) + + # add a part of waiters again + for i in range(incr): + waiters.append(asyncio.create_task(wait(barrier))) + await asyncio.sleep(0) + # and abort + await barrier.abort() + + self.assertTrue(RGX_REPR.match(repr(barrier))) + self.assertIn("broken", repr(barrier)) + self.assertTrue(barrier.broken) + + # suppress unhandled exceptions + await asyncio.gather(*waiters, return_exceptions=True) + + async def test_barrier_parties(self): + self.assertRaises(ValueError, lambda: asyncio.Barrier(0)) + self.assertRaises(ValueError, lambda: asyncio.Barrier(-4)) + + self.assertIsInstance(asyncio.Barrier(self.N), asyncio.Barrier) + + async def test_context_manager(self): + self.N = 3 + barrier = asyncio.Barrier(self.N) + results = [] + + async def coro(): + async with barrier as i: + results.append(i) + + await self.gather_tasks(self.N, coro) + + self.assertListEqual(sorted(results), list(range(self.N))) + self.assertEqual(barrier.n_waiting, 0) + self.assertFalse(barrier.broken) + + async def test_filling_one_task(self): + barrier = asyncio.Barrier(1) + + async def f(): + async with barrier as i: + return True + + ret = await f() + + self.assertTrue(ret) + self.assertEqual(barrier.n_waiting, 0) + self.assertFalse(barrier.broken) + + async def test_filling_one_task_twice(self): + barrier = asyncio.Barrier(1) + + t1 = asyncio.create_task(barrier.wait()) + await asyncio.sleep(0) + self.assertEqual(barrier.n_waiting, 0) + + t2 = asyncio.create_task(barrier.wait()) + await asyncio.sleep(0) + + self.assertEqual(t1.result(), t2.result()) + self.assertEqual(t1.done(), t2.done()) + + self.assertEqual(barrier.n_waiting, 0) + self.assertFalse(barrier.broken) + + async def test_filling_task_by_task(self): + self.N = 3 + barrier = asyncio.Barrier(self.N) + + t1 = asyncio.create_task(barrier.wait()) + await asyncio.sleep(0) + self.assertEqual(barrier.n_waiting, 1) + self.assertIn("filling", repr(barrier)) + + t2 = asyncio.create_task(barrier.wait()) + await asyncio.sleep(0) + self.assertEqual(barrier.n_waiting, 2) + self.assertIn("filling", repr(barrier)) + + t3 = asyncio.create_task(barrier.wait()) + await asyncio.sleep(0) + + await asyncio.wait([t1, t2, t3]) + + self.assertEqual(barrier.n_waiting, 0) + self.assertFalse(barrier.broken) + + async def test_filling_tasks_wait_twice(self): + barrier = asyncio.Barrier(self.N) + results = [] + + async def coro(): + async with barrier: + results.append(True) + + async with barrier: + results.append(False) + + await self.gather_tasks(self.N, coro) + + self.assertEqual(len(results), self.N*2) + self.assertEqual(results.count(True), self.N) + self.assertEqual(results.count(False), self.N) + + self.assertEqual(barrier.n_waiting, 0) + self.assertFalse(barrier.broken) + + async def test_filling_tasks_check_return_value(self): + barrier = asyncio.Barrier(self.N) + results1 = [] + results2 = [] + + async def coro(): + async with barrier: + results1.append(True) + + async with barrier as i: + results2.append(True) + return i + + res, _ = await self.gather_tasks(self.N, coro) + + self.assertEqual(len(results1), self.N) + self.assertTrue(all(results1)) + self.assertEqual(len(results2), self.N) + self.assertTrue(all(results2)) + self.assertListEqual(sorted(res), list(range(self.N))) + + self.assertEqual(barrier.n_waiting, 0) + self.assertFalse(barrier.broken) + + async def test_draining_state(self): + barrier = asyncio.Barrier(self.N) + results = [] + + async def coro(): + async with barrier: + # barrier state change to filling for the last task release + results.append("draining" in repr(barrier)) + + await self.gather_tasks(self.N, coro) + + self.assertEqual(len(results), self.N) + self.assertEqual(results[-1], False) + self.assertTrue(all(results[:self.N-1])) + + self.assertEqual(barrier.n_waiting, 0) + self.assertFalse(barrier.broken) + + async def test_blocking_tasks_while_draining(self): + rewait = 2 + barrier = asyncio.Barrier(self.N) + barrier_nowaiting = asyncio.Barrier(self.N - rewait) + results = [] + rewait_n = rewait + counter = 0 + + async def coro(): + nonlocal rewait_n + + # first time waiting + await barrier.wait() + + # after waiting once for all tasks + if rewait_n > 0: + rewait_n -= 1 + # wait again only for rewait tasks + await barrier.wait() + else: + # wait for end of draining state + await barrier_nowaiting.wait() + # wait for other waiting tasks + await barrier.wait() + + # a success means that barrier_nowaiting + # was waited for exactly N-rewait=3 times + await self.gather_tasks(self.N, coro) + + async def test_filling_tasks_cancel_one(self): + self.N = 3 + barrier = asyncio.Barrier(self.N) + results = [] + + async def coro(): + await barrier.wait() + results.append(True) + + t1 = asyncio.create_task(coro()) + await asyncio.sleep(0) + self.assertEqual(barrier.n_waiting, 1) + + t2 = asyncio.create_task(coro()) + await asyncio.sleep(0) + self.assertEqual(barrier.n_waiting, 2) + + t1.cancel() + await asyncio.sleep(0) + self.assertEqual(barrier.n_waiting, 1) + with self.assertRaises(asyncio.CancelledError): + await t1 + self.assertTrue(t1.cancelled()) + + t3 = asyncio.create_task(coro()) + await asyncio.sleep(0) + self.assertEqual(barrier.n_waiting, 2) + + t4 = asyncio.create_task(coro()) + await asyncio.gather(t2, t3, t4) + + self.assertEqual(len(results), self.N) + self.assertTrue(all(results)) + + self.assertEqual(barrier.n_waiting, 0) + self.assertFalse(barrier.broken) + + async def test_reset_barrier(self): + barrier = asyncio.Barrier(1) + + asyncio.create_task(barrier.reset()) + await asyncio.sleep(0) + + self.assertEqual(barrier.n_waiting, 0) + self.assertFalse(barrier.broken) + + async def test_reset_barrier_while_tasks_waiting(self): + barrier = asyncio.Barrier(self.N) + results = [] + + async def coro(): + try: + await barrier.wait() + except asyncio.BrokenBarrierError: + results.append(True) + + async def coro_reset(): + await barrier.reset() + + # N-1 tasks waiting on barrier with N parties + tasks = self.make_tasks(self.N-1, coro) + await asyncio.sleep(0) + + # reset the barrier + asyncio.create_task(coro_reset()) + await asyncio.gather(*tasks) + + self.assertEqual(len(results), self.N-1) + self.assertTrue(all(results)) + self.assertEqual(barrier.n_waiting, 0) + self.assertNotIn("resetting", repr(barrier)) + self.assertFalse(barrier.broken) + + async def test_reset_barrier_when_tasks_half_draining(self): + barrier = asyncio.Barrier(self.N) + results1 = [] + rest_of_tasks = self.N//2 + + async def coro(): + try: + await barrier.wait() + except asyncio.BrokenBarrierError: + # catch here waiting tasks + results1.append(True) + else: + # here drained task outside the barrier + if rest_of_tasks == barrier._count: + # tasks outside the barrier + await barrier.reset() + + await self.gather_tasks(self.N, coro) + + self.assertEqual(results1, [True]*rest_of_tasks) + self.assertEqual(barrier.n_waiting, 0) + self.assertNotIn("resetting", repr(barrier)) + self.assertFalse(barrier.broken) + + async def test_reset_barrier_when_tasks_half_draining_half_blocking(self): + barrier = asyncio.Barrier(self.N) + results1 = [] + results2 = [] + blocking_tasks = self.N//2 + count = 0 + + async def coro(): + nonlocal count + try: + await barrier.wait() + except asyncio.BrokenBarrierError: + # here catch still waiting tasks + results1.append(True) + + # so now waiting again to reach nb_parties + await barrier.wait() + else: + count += 1 + if count > blocking_tasks: + # reset now: raise asyncio.BrokenBarrierError for waiting tasks + await barrier.reset() + + # so now waiting again to reach nb_parties + await barrier.wait() + else: + try: + await barrier.wait() + except asyncio.BrokenBarrierError: + # here no catch - blocked tasks go to wait + results2.append(True) + + await self.gather_tasks(self.N, coro) + + self.assertEqual(results1, [True]*blocking_tasks) + self.assertEqual(results2, []) + self.assertEqual(barrier.n_waiting, 0) + self.assertNotIn("resetting", repr(barrier)) + self.assertFalse(barrier.broken) + + async def test_reset_barrier_while_tasks_waiting_and_waiting_again(self): + barrier = asyncio.Barrier(self.N) + results1 = [] + results2 = [] + + async def coro1(): + try: + await barrier.wait() + except asyncio.BrokenBarrierError: + results1.append(True) + finally: + await barrier.wait() + results2.append(True) + + async def coro2(): + async with barrier: + results2.append(True) + + tasks = self.make_tasks(self.N-1, coro1) + + # reset barrier, N-1 waiting tasks raise an BrokenBarrierError + asyncio.create_task(barrier.reset()) + await asyncio.sleep(0) + + # complete waiting tasks in the `finally` + asyncio.create_task(coro2()) + + await asyncio.gather(*tasks) + + self.assertFalse(barrier.broken) + self.assertEqual(len(results1), self.N-1) + self.assertTrue(all(results1)) + self.assertEqual(len(results2), self.N) + self.assertTrue(all(results2)) + + self.assertEqual(barrier.n_waiting, 0) + + + async def test_reset_barrier_while_tasks_draining(self): + barrier = asyncio.Barrier(self.N) + results1 = [] + results2 = [] + results3 = [] + count = 0 + + async def coro(): + nonlocal count + + i = await barrier.wait() + count += 1 + if count == self.N: + # last task exited from barrier + await barrier.reset() + + # wait here to reach the `parties` + await barrier.wait() + else: + try: + # second waiting + await barrier.wait() + + # N-1 tasks here + results1.append(True) + except Exception as e: + # never goes here + results2.append(True) + + # Now, pass the barrier again + # last wait, must be completed + k = await barrier.wait() + results3.append(True) + + await self.gather_tasks(self.N, coro) + + self.assertFalse(barrier.broken) + self.assertTrue(all(results1)) + self.assertEqual(len(results1), self.N-1) + self.assertEqual(len(results2), 0) + self.assertEqual(len(results3), self.N) + self.assertTrue(all(results3)) + + self.assertEqual(barrier.n_waiting, 0) + + async def test_abort_barrier(self): + barrier = asyncio.Barrier(1) + + asyncio.create_task(barrier.abort()) + await asyncio.sleep(0) + + self.assertEqual(barrier.n_waiting, 0) + self.assertTrue(barrier.broken) + + async def test_abort_barrier_when_tasks_half_draining_half_blocking(self): + barrier = asyncio.Barrier(self.N) + results1 = [] + results2 = [] + blocking_tasks = self.N//2 + count = 0 + + async def coro(): + nonlocal count + try: + await barrier.wait() + except asyncio.BrokenBarrierError: + # here catch tasks waiting to drain + results1.append(True) + else: + count += 1 + if count > blocking_tasks: + # abort now: raise asyncio.BrokenBarrierError for all tasks + await barrier.abort() + else: + try: + await barrier.wait() + except asyncio.BrokenBarrierError: + # here catch blocked tasks (already drained) + results2.append(True) + + await self.gather_tasks(self.N, coro) + + self.assertTrue(barrier.broken) + self.assertEqual(results1, [True]*blocking_tasks) + self.assertEqual(results2, [True]*(self.N-blocking_tasks-1)) + self.assertEqual(barrier.n_waiting, 0) + self.assertNotIn("resetting", repr(barrier)) + + async def test_abort_barrier_when_exception(self): + # test from threading.Barrier: see `lock_tests.test_reset` + barrier = asyncio.Barrier(self.N) + results1 = [] + results2 = [] + + async def coro(): + try: + async with barrier as i : + if i == self.N//2: + raise RuntimeError + async with barrier: + results1.append(True) + except asyncio.BrokenBarrierError: + results2.append(True) + except RuntimeError: + await barrier.abort() + + await self.gather_tasks(self.N, coro) + + self.assertTrue(barrier.broken) + self.assertEqual(len(results1), 0) + self.assertEqual(len(results2), self.N-1) + self.assertTrue(all(results2)) + self.assertEqual(barrier.n_waiting, 0) + + async def test_abort_barrier_when_exception_then_resetting(self): + # test from threading.Barrier: see `lock_tests.test_abort_and_reset` + barrier1 = asyncio.Barrier(self.N) + barrier2 = asyncio.Barrier(self.N) + results1 = [] + results2 = [] + results3 = [] + + async def coro(): + try: + i = await barrier1.wait() + if i == self.N//2: + raise RuntimeError + await barrier1.wait() + results1.append(True) + except asyncio.BrokenBarrierError: + results2.append(True) + except RuntimeError: + await barrier1.abort() + + # Synchronize and reset the barrier. Must synchronize first so + # that everyone has left it when we reset, and after so that no + # one enters it before the reset. + i = await barrier2.wait() + if i == self.N//2: + await barrier1.reset() + await barrier2.wait() + await barrier1.wait() + results3.append(True) + + await self.gather_tasks(self.N, coro) + + self.assertFalse(barrier1.broken) + self.assertEqual(len(results1), 0) + self.assertEqual(len(results2), self.N-1) + self.assertTrue(all(results2)) + self.assertEqual(len(results3), self.N) + self.assertTrue(all(results3)) + + self.assertEqual(barrier1.n_waiting, 0) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_pep492.py b/Lib/test/test_asyncio/test_pep492.py new file mode 100644 index 00000000000..a0c8434c945 --- /dev/null +++ b/Lib/test/test_asyncio/test_pep492.py @@ -0,0 +1,212 @@ +"""Tests support for new syntax introduced by PEP 492.""" + +import sys +import types +import unittest + +from unittest import mock + +import asyncio +from test.test_asyncio import utils as test_utils + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +# Test that asyncio.iscoroutine() uses collections.abc.Coroutine +class FakeCoro: + def send(self, value): + pass + + def throw(self, typ, val=None, tb=None): + pass + + def close(self): + pass + + def __await__(self): + yield + + +class BaseTest(test_utils.TestCase): + + def setUp(self): + super().setUp() + self.loop = asyncio.BaseEventLoop() + self.loop._process_events = mock.Mock() + self.loop._selector = mock.Mock() + self.loop._selector.select.return_value = () + self.set_event_loop(self.loop) + + +class LockTests(BaseTest): + + def test_context_manager_async_with(self): + primitives = [ + asyncio.Lock(), + asyncio.Condition(), + asyncio.Semaphore(), + asyncio.BoundedSemaphore(), + ] + + async def test(lock): + await asyncio.sleep(0.01) + self.assertFalse(lock.locked()) + async with lock as _lock: + self.assertIs(_lock, None) + self.assertTrue(lock.locked()) + await asyncio.sleep(0.01) + self.assertTrue(lock.locked()) + self.assertFalse(lock.locked()) + + for primitive in primitives: + self.loop.run_until_complete(test(primitive)) + self.assertFalse(primitive.locked()) + + def test_context_manager_with_await(self): + primitives = [ + asyncio.Lock(), + asyncio.Condition(), + asyncio.Semaphore(), + asyncio.BoundedSemaphore(), + ] + + async def test(lock): + await asyncio.sleep(0.01) + self.assertFalse(lock.locked()) + with self.assertRaisesRegex( + TypeError, + "can't be awaited" + ): + with await lock: + pass + + for primitive in primitives: + self.loop.run_until_complete(test(primitive)) + self.assertFalse(primitive.locked()) + + +class StreamReaderTests(BaseTest): + + def test_readline(self): + DATA = b'line1\nline2\nline3' + + stream = asyncio.StreamReader(loop=self.loop) + stream.feed_data(DATA) + stream.feed_eof() + + async def reader(): + data = [] + async for line in stream: + data.append(line) + return data + + data = self.loop.run_until_complete(reader()) + self.assertEqual(data, [b'line1\n', b'line2\n', b'line3']) + + +class CoroutineTests(BaseTest): + + def test_iscoroutine(self): + async def foo(): pass + + f = foo() + try: + self.assertTrue(asyncio.iscoroutine(f)) + finally: + f.close() # silence warning + + self.assertTrue(asyncio.iscoroutine(FakeCoro())) + + def test_iscoroutine_generator(self): + def foo(): yield + + self.assertFalse(asyncio.iscoroutine(foo())) + + def test_iscoroutinefunction(self): + async def foo(): pass + with self.assertWarns(DeprecationWarning): + self.assertTrue(asyncio.iscoroutinefunction(foo)) + + def test_async_def_coroutines(self): + async def bar(): + return 'spam' + async def foo(): + return await bar() + + # production mode + data = self.loop.run_until_complete(foo()) + self.assertEqual(data, 'spam') + + # debug mode + self.loop.set_debug(True) + data = self.loop.run_until_complete(foo()) + self.assertEqual(data, 'spam') + + def test_debug_mode_manages_coroutine_origin_tracking(self): + async def start(): + self.assertTrue(sys.get_coroutine_origin_tracking_depth() > 0) + + self.assertEqual(sys.get_coroutine_origin_tracking_depth(), 0) + self.loop.set_debug(True) + self.loop.run_until_complete(start()) + self.assertEqual(sys.get_coroutine_origin_tracking_depth(), 0) + + def test_types_coroutine(self): + def gen(): + yield from () + return 'spam' + + @types.coroutine + def func(): + return gen() + + async def coro(): + wrapper = func() + self.assertIsInstance(wrapper, types._GeneratorWrapper) + return await wrapper + + data = self.loop.run_until_complete(coro()) + self.assertEqual(data, 'spam') + + def test_task_print_stack(self): + T = None + + async def foo(): + f = T.get_stack(limit=1) + try: + self.assertEqual(f[0].f_code.co_name, 'foo') + finally: + f = None + + async def runner(): + nonlocal T + T = asyncio.ensure_future(foo(), loop=self.loop) + await T + + self.loop.run_until_complete(runner()) + + def test_double_await(self): + async def afunc(): + await asyncio.sleep(0.1) + + async def runner(): + coro = afunc() + t = self.loop.create_task(coro) + try: + await asyncio.sleep(0) + await coro + finally: + t.cancel() + + self.loop.set_debug(True) + with self.assertRaises( + RuntimeError, + msg='coroutine is being awaited already'): + + self.loop.run_until_complete(runner()) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py new file mode 100644 index 00000000000..edfad5e11db --- /dev/null +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -0,0 +1,1094 @@ +"""Tests for proactor_events.py""" + +import io +import socket +import unittest +import sys +from unittest import mock + +import asyncio +from asyncio.proactor_events import BaseProactorEventLoop +from asyncio.proactor_events import _ProactorSocketTransport +from asyncio.proactor_events import _ProactorWritePipeTransport +from asyncio.proactor_events import _ProactorDuplexPipeTransport +from asyncio.proactor_events import _ProactorDatagramTransport +from test.support import os_helper +from test.support import socket_helper +from test.test_asyncio import utils as test_utils + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +def close_transport(transport): + # Don't call transport.close() because the event loop and the IOCP proactor + # are mocked + if transport._sock is None: + return + transport._sock.close() + transport._sock = None + + +class ProactorSocketTransportTests(test_utils.TestCase): + + def setUp(self): + super().setUp() + self.loop = self.new_test_loop() + self.addCleanup(self.loop.close) + self.proactor = mock.Mock() + self.loop._proactor = self.proactor + self.protocol = test_utils.make_test_protocol(asyncio.Protocol) + self.sock = mock.Mock(socket.socket) + self.buffer_size = 65536 + + def socket_transport(self, waiter=None): + transport = _ProactorSocketTransport(self.loop, self.sock, + self.protocol, waiter=waiter) + self.addCleanup(close_transport, transport) + return transport + + def test_ctor(self): + fut = self.loop.create_future() + tr = self.socket_transport(waiter=fut) + test_utils.run_briefly(self.loop) + self.assertIsNone(fut.result()) + self.protocol.connection_made(tr) + self.proactor.recv_into.assert_called_with(self.sock, bytearray(self.buffer_size)) + + def test_loop_reading(self): + tr = self.socket_transport() + tr._loop_reading() + self.loop._proactor.recv_into.assert_called_with(self.sock, bytearray(self.buffer_size)) + self.assertFalse(self.protocol.data_received.called) + self.assertFalse(self.protocol.eof_received.called) + + def test_loop_reading_data(self): + buf = b'data' + res = self.loop.create_future() + res.set_result(len(buf)) + + tr = self.socket_transport() + tr._read_fut = res + tr._data[:len(buf)] = buf + tr._loop_reading(res) + called_buf = bytearray(self.buffer_size) + called_buf[:len(buf)] = buf + self.loop._proactor.recv_into.assert_called_with(self.sock, called_buf) + self.protocol.data_received.assert_called_with(buf) + # assert_called_with maps bytearray and bytes to the same thing so check manually + # regression test for https://github.com/python/cpython/issues/99941 + self.assertIsInstance(self.protocol.data_received.call_args.args[0], bytes) + + @unittest.skipIf(sys.flags.optimize, "Assertions are disabled in optimized mode") + def test_loop_reading_no_data(self): + res = self.loop.create_future() + res.set_result(0) + + tr = self.socket_transport() + self.assertRaises(AssertionError, tr._loop_reading, res) + + tr.close = mock.Mock() + tr._read_fut = res + tr._loop_reading(res) + self.assertFalse(self.loop._proactor.recv_into.called) + self.assertTrue(self.protocol.eof_received.called) + self.assertTrue(tr.close.called) + + def test_loop_reading_aborted(self): + err = self.loop._proactor.recv_into.side_effect = ConnectionAbortedError() + + tr = self.socket_transport() + tr._fatal_error = mock.Mock() + tr._loop_reading() + tr._fatal_error.assert_called_with( + err, + 'Fatal read error on pipe transport') + + def test_loop_reading_aborted_closing(self): + self.loop._proactor.recv_into.side_effect = ConnectionAbortedError() + + tr = self.socket_transport() + tr._closing = True + tr._fatal_error = mock.Mock() + tr._loop_reading() + self.assertFalse(tr._fatal_error.called) + + def test_loop_reading_aborted_is_fatal(self): + self.loop._proactor.recv_into.side_effect = ConnectionAbortedError() + tr = self.socket_transport() + tr._closing = False + tr._fatal_error = mock.Mock() + tr._loop_reading() + self.assertTrue(tr._fatal_error.called) + + def test_loop_reading_conn_reset_lost(self): + err = self.loop._proactor.recv_into.side_effect = ConnectionResetError() + + tr = self.socket_transport() + tr._closing = False + tr._fatal_error = mock.Mock() + tr._force_close = mock.Mock() + tr._loop_reading() + self.assertFalse(tr._fatal_error.called) + tr._force_close.assert_called_with(err) + + def test_loop_reading_exception(self): + err = self.loop._proactor.recv_into.side_effect = (OSError()) + + tr = self.socket_transport() + tr._fatal_error = mock.Mock() + tr._loop_reading() + tr._fatal_error.assert_called_with( + err, + 'Fatal read error on pipe transport') + + def test_write(self): + tr = self.socket_transport() + tr._loop_writing = mock.Mock() + tr.write(b'data') + self.assertEqual(tr._buffer, None) + tr._loop_writing.assert_called_with(data=b'data') + + def test_write_no_data(self): + tr = self.socket_transport() + tr.write(b'') + self.assertFalse(tr._buffer) + + def test_write_more(self): + tr = self.socket_transport() + tr._write_fut = mock.Mock() + tr._loop_writing = mock.Mock() + tr.write(b'data') + self.assertEqual(tr._buffer, b'data') + self.assertFalse(tr._loop_writing.called) + + def test_loop_writing(self): + tr = self.socket_transport() + tr._buffer = bytearray(b'data') + tr._loop_writing() + self.loop._proactor.send.assert_called_with(self.sock, b'data') + self.loop._proactor.send.return_value.add_done_callback.\ + assert_called_with(tr._loop_writing) + + @mock.patch('asyncio.proactor_events.logger') + def test_loop_writing_err(self, m_log): + err = self.loop._proactor.send.side_effect = OSError() + tr = self.socket_transport() + tr._fatal_error = mock.Mock() + tr._buffer = [b'da', b'ta'] + tr._loop_writing() + tr._fatal_error.assert_called_with( + err, + 'Fatal write error on pipe transport') + tr._conn_lost = 1 + + tr.write(b'data') + tr.write(b'data') + tr.write(b'data') + tr.write(b'data') + tr.write(b'data') + self.assertEqual(tr._buffer, None) + m_log.warning.assert_called_with('socket.send() raised exception.') + + def test_loop_writing_stop(self): + fut = self.loop.create_future() + fut.set_result(b'data') + + tr = self.socket_transport() + tr._write_fut = fut + tr._loop_writing(fut) + self.assertIsNone(tr._write_fut) + + def test_loop_writing_closing(self): + fut = self.loop.create_future() + fut.set_result(1) + + tr = self.socket_transport() + tr._write_fut = fut + tr.close() + tr._loop_writing(fut) + self.assertIsNone(tr._write_fut) + test_utils.run_briefly(self.loop) + self.protocol.connection_lost.assert_called_with(None) + + def test_abort(self): + tr = self.socket_transport() + tr._force_close = mock.Mock() + tr.abort() + tr._force_close.assert_called_with(None) + + def test_close(self): + tr = self.socket_transport() + tr.close() + test_utils.run_briefly(self.loop) + self.protocol.connection_lost.assert_called_with(None) + self.assertTrue(tr.is_closing()) + self.assertEqual(tr._conn_lost, 1) + + self.protocol.connection_lost.reset_mock() + tr.close() + test_utils.run_briefly(self.loop) + self.assertFalse(self.protocol.connection_lost.called) + + def test_close_write_fut(self): + tr = self.socket_transport() + tr._write_fut = mock.Mock() + tr.close() + test_utils.run_briefly(self.loop) + self.assertFalse(self.protocol.connection_lost.called) + + def test_close_buffer(self): + tr = self.socket_transport() + tr._buffer = [b'data'] + tr.close() + test_utils.run_briefly(self.loop) + self.assertFalse(self.protocol.connection_lost.called) + + def test_close_invalid_sockobj(self): + tr = self.socket_transport() + self.sock.fileno.return_value = -1 + tr.close() + test_utils.run_briefly(self.loop) + self.protocol.connection_lost.assert_called_with(None) + self.assertFalse(self.sock.shutdown.called) + + @mock.patch('asyncio.base_events.logger') + def test_fatal_error(self, m_logging): + tr = self.socket_transport() + tr._force_close = mock.Mock() + tr._fatal_error(None) + self.assertTrue(tr._force_close.called) + self.assertTrue(m_logging.error.called) + + def test_force_close(self): + tr = self.socket_transport() + tr._buffer = [b'data'] + read_fut = tr._read_fut = mock.Mock() + write_fut = tr._write_fut = mock.Mock() + tr._force_close(None) + + read_fut.cancel.assert_called_with() + write_fut.cancel.assert_called_with() + test_utils.run_briefly(self.loop) + self.protocol.connection_lost.assert_called_with(None) + self.assertEqual(None, tr._buffer) + self.assertEqual(tr._conn_lost, 1) + + def test_loop_writing_force_close(self): + exc_handler = mock.Mock() + self.loop.set_exception_handler(exc_handler) + fut = self.loop.create_future() + fut.set_result(1) + self.proactor.send.return_value = fut + + tr = self.socket_transport() + tr.write(b'data') + tr._force_close(None) + test_utils.run_briefly(self.loop) + exc_handler.assert_not_called() + + def test_force_close_idempotent(self): + tr = self.socket_transport() + tr._closing = True + tr._force_close(None) + test_utils.run_briefly(self.loop) + # See https://github.com/python/cpython/issues/89237 + # `protocol.connection_lost` should be called even if + # the transport was closed forcefully otherwise + # the resources held by protocol will never be freed + # and waiters will never be notified leading to hang. + self.assertTrue(self.protocol.connection_lost.called) + + def test_force_close_protocol_connection_lost_once(self): + tr = self.socket_transport() + self.assertFalse(self.protocol.connection_lost.called) + tr._closing = True + # Calling _force_close twice should not call + # protocol.connection_lost twice + tr._force_close(None) + tr._force_close(None) + test_utils.run_briefly(self.loop) + self.assertEqual(1, self.protocol.connection_lost.call_count) + + def test_close_protocol_connection_lost_once(self): + tr = self.socket_transport() + self.assertFalse(self.protocol.connection_lost.called) + # Calling close twice should not call + # protocol.connection_lost twice + tr.close() + tr.close() + test_utils.run_briefly(self.loop) + self.assertEqual(1, self.protocol.connection_lost.call_count) + + def test_fatal_error_2(self): + tr = self.socket_transport() + tr._buffer = [b'data'] + tr._force_close(None) + + test_utils.run_briefly(self.loop) + self.protocol.connection_lost.assert_called_with(None) + self.assertEqual(None, tr._buffer) + + def test_call_connection_lost(self): + tr = self.socket_transport() + tr._call_connection_lost(None) + self.assertTrue(self.protocol.connection_lost.called) + self.assertTrue(self.sock.close.called) + + def test_write_eof(self): + tr = self.socket_transport() + self.assertTrue(tr.can_write_eof()) + tr.write_eof() + self.sock.shutdown.assert_called_with(socket.SHUT_WR) + tr.write_eof() + self.assertEqual(self.sock.shutdown.call_count, 1) + tr.close() + + def test_write_eof_buffer(self): + tr = self.socket_transport() + f = self.loop.create_future() + tr._loop._proactor.send.return_value = f + tr.write(b'data') + tr.write_eof() + self.assertTrue(tr._eof_written) + self.assertFalse(self.sock.shutdown.called) + tr._loop._proactor.send.assert_called_with(self.sock, b'data') + f.set_result(4) + self.loop._run_once() + self.sock.shutdown.assert_called_with(socket.SHUT_WR) + tr.close() + + def test_write_eof_write_pipe(self): + tr = _ProactorWritePipeTransport( + self.loop, self.sock, self.protocol) + self.assertTrue(tr.can_write_eof()) + tr.write_eof() + self.assertTrue(tr.is_closing()) + self.loop._run_once() + self.assertTrue(self.sock.close.called) + tr.close() + + def test_write_eof_buffer_write_pipe(self): + tr = _ProactorWritePipeTransport(self.loop, self.sock, self.protocol) + f = self.loop.create_future() + tr._loop._proactor.send.return_value = f + tr.write(b'data') + tr.write_eof() + self.assertTrue(tr.is_closing()) + self.assertFalse(self.sock.shutdown.called) + tr._loop._proactor.send.assert_called_with(self.sock, b'data') + f.set_result(4) + self.loop._run_once() + self.loop._run_once() + self.assertTrue(self.sock.close.called) + tr.close() + + def test_write_eof_duplex_pipe(self): + tr = _ProactorDuplexPipeTransport( + self.loop, self.sock, self.protocol) + self.assertFalse(tr.can_write_eof()) + with self.assertRaises(NotImplementedError): + tr.write_eof() + close_transport(tr) + + def test_pause_resume_reading(self): + tr = self.socket_transport() + index = 0 + msgs = [b'data1', b'data2', b'data3', b'data4', b'data5', b''] + reversed_msgs = list(reversed(msgs)) + + def recv_into(sock, data): + f = self.loop.create_future() + msg = reversed_msgs.pop() + + result = f.result + def monkey(): + data[:len(msg)] = msg + return result() + f.result = monkey + + f.set_result(len(msg)) + return f + + self.loop._proactor.recv_into.side_effect = recv_into + self.loop._run_once() + self.assertFalse(tr._paused) + self.assertTrue(tr.is_reading()) + + for msg in msgs[:2]: + self.loop._run_once() + self.protocol.data_received.assert_called_with(bytearray(msg)) + + tr.pause_reading() + tr.pause_reading() + self.assertTrue(tr._paused) + self.assertFalse(tr.is_reading()) + for i in range(10): + self.loop._run_once() + self.protocol.data_received.assert_called_with(bytearray(msgs[1])) + + tr.resume_reading() + tr.resume_reading() + self.assertFalse(tr._paused) + self.assertTrue(tr.is_reading()) + + for msg in msgs[2:4]: + self.loop._run_once() + self.protocol.data_received.assert_called_with(bytearray(msg)) + + tr.pause_reading() + tr.resume_reading() + self.loop.call_exception_handler = mock.Mock() + self.loop._run_once() + self.loop.call_exception_handler.assert_not_called() + self.protocol.data_received.assert_called_with(bytearray(msgs[4])) + tr.close() + + self.assertFalse(tr.is_reading()) + + def test_pause_reading_connection_made(self): + tr = self.socket_transport() + self.protocol.connection_made.side_effect = lambda _: tr.pause_reading() + test_utils.run_briefly(self.loop) + self.assertFalse(tr.is_reading()) + self.loop.assert_no_reader(7) + + tr.resume_reading() + self.assertTrue(tr.is_reading()) + + tr.close() + self.assertFalse(tr.is_reading()) + + + def pause_writing_transport(self, high): + tr = self.socket_transport() + tr.set_write_buffer_limits(high=high) + + self.assertEqual(tr.get_write_buffer_size(), 0) + self.assertFalse(self.protocol.pause_writing.called) + self.assertFalse(self.protocol.resume_writing.called) + return tr + + def test_pause_resume_writing(self): + tr = self.pause_writing_transport(high=4) + + # write a large chunk, must pause writing + fut = self.loop.create_future() + self.loop._proactor.send.return_value = fut + tr.write(b'large data') + self.loop._run_once() + self.assertTrue(self.protocol.pause_writing.called) + + # flush the buffer + fut.set_result(None) + self.loop._run_once() + self.assertEqual(tr.get_write_buffer_size(), 0) + self.assertTrue(self.protocol.resume_writing.called) + + def test_pause_writing_2write(self): + tr = self.pause_writing_transport(high=4) + + # first short write, the buffer is not full (3 <= 4) + fut1 = self.loop.create_future() + self.loop._proactor.send.return_value = fut1 + tr.write(b'123') + self.loop._run_once() + self.assertEqual(tr.get_write_buffer_size(), 3) + self.assertFalse(self.protocol.pause_writing.called) + + # fill the buffer, must pause writing (6 > 4) + tr.write(b'abc') + self.loop._run_once() + self.assertEqual(tr.get_write_buffer_size(), 6) + self.assertTrue(self.protocol.pause_writing.called) + + def test_pause_writing_3write(self): + tr = self.pause_writing_transport(high=4) + + # first short write, the buffer is not full (1 <= 4) + fut = self.loop.create_future() + self.loop._proactor.send.return_value = fut + tr.write(b'1') + self.loop._run_once() + self.assertEqual(tr.get_write_buffer_size(), 1) + self.assertFalse(self.protocol.pause_writing.called) + + # second short write, the buffer is not full (3 <= 4) + tr.write(b'23') + self.loop._run_once() + self.assertEqual(tr.get_write_buffer_size(), 3) + self.assertFalse(self.protocol.pause_writing.called) + + # fill the buffer, must pause writing (6 > 4) + tr.write(b'abc') + self.loop._run_once() + self.assertEqual(tr.get_write_buffer_size(), 6) + self.assertTrue(self.protocol.pause_writing.called) + + def test_dont_pause_writing(self): + tr = self.pause_writing_transport(high=4) + + # write a large chunk which completes immediately, + # it should not pause writing + fut = self.loop.create_future() + fut.set_result(None) + self.loop._proactor.send.return_value = fut + tr.write(b'very large data') + self.loop._run_once() + self.assertEqual(tr.get_write_buffer_size(), 0) + self.assertFalse(self.protocol.pause_writing.called) + + +class ProactorDatagramTransportTests(test_utils.TestCase): + + def setUp(self): + super().setUp() + self.loop = self.new_test_loop() + self.proactor = mock.Mock() + self.loop._proactor = self.proactor + self.protocol = test_utils.make_test_protocol(asyncio.DatagramProtocol) + self.sock = mock.Mock(spec_set=socket.socket) + self.sock.fileno.return_value = 7 + + def datagram_transport(self, address=None): + self.sock.getpeername.side_effect = None if address else OSError + transport = _ProactorDatagramTransport(self.loop, self.sock, + self.protocol, + address=address) + self.addCleanup(close_transport, transport) + return transport + + def test_sendto(self): + data = b'data' + transport = self.datagram_transport() + transport.sendto(data, ('0.0.0.0', 1234)) + self.assertTrue(self.proactor.sendto.called) + self.proactor.sendto.assert_called_with( + self.sock, data, addr=('0.0.0.0', 1234)) + self.assertFalse(transport._buffer) + self.assertEqual(0, transport._buffer_size) + + def test_sendto_bytearray(self): + data = bytearray(b'data') + transport = self.datagram_transport() + transport.sendto(data, ('0.0.0.0', 1234)) + self.assertTrue(self.proactor.sendto.called) + self.proactor.sendto.assert_called_with( + self.sock, b'data', addr=('0.0.0.0', 1234)) + + def test_sendto_memoryview(self): + data = memoryview(b'data') + transport = self.datagram_transport() + transport.sendto(data, ('0.0.0.0', 1234)) + self.assertTrue(self.proactor.sendto.called) + self.proactor.sendto.assert_called_with( + self.sock, b'data', addr=('0.0.0.0', 1234)) + + def test_sendto_no_data(self): + transport = self.datagram_transport() + transport.sendto(b'', ('0.0.0.0', 1234)) + self.assertTrue(self.proactor.sendto.called) + self.proactor.sendto.assert_called_with( + self.sock, b'', addr=('0.0.0.0', 1234)) + + def test_sendto_buffer(self): + transport = self.datagram_transport() + transport._buffer.append((b'data1', ('0.0.0.0', 12345))) + transport._write_fut = object() + transport.sendto(b'data2', ('0.0.0.0', 12345)) + self.assertFalse(self.proactor.sendto.called) + self.assertEqual( + [(b'data1', ('0.0.0.0', 12345)), + (b'data2', ('0.0.0.0', 12345))], + list(transport._buffer)) + + def test_sendto_buffer_bytearray(self): + data2 = bytearray(b'data2') + transport = self.datagram_transport() + transport._buffer.append((b'data1', ('0.0.0.0', 12345))) + transport._write_fut = object() + transport.sendto(data2, ('0.0.0.0', 12345)) + self.assertFalse(self.proactor.sendto.called) + self.assertEqual( + [(b'data1', ('0.0.0.0', 12345)), + (b'data2', ('0.0.0.0', 12345))], + list(transport._buffer)) + self.assertIsInstance(transport._buffer[1][0], bytes) + + def test_sendto_buffer_memoryview(self): + data2 = memoryview(b'data2') + transport = self.datagram_transport() + transport._buffer.append((b'data1', ('0.0.0.0', 12345))) + transport._write_fut = object() + transport.sendto(data2, ('0.0.0.0', 12345)) + self.assertFalse(self.proactor.sendto.called) + self.assertEqual( + [(b'data1', ('0.0.0.0', 12345)), + (b'data2', ('0.0.0.0', 12345))], + list(transport._buffer)) + self.assertIsInstance(transport._buffer[1][0], bytes) + + def test_sendto_buffer_nodata(self): + data2 = b'' + transport = self.datagram_transport() + transport._buffer.append((b'data1', ('0.0.0.0', 12345))) + transport._write_fut = object() + transport.sendto(data2, ('0.0.0.0', 12345)) + self.assertFalse(self.proactor.sendto.called) + self.assertEqual( + [(b'data1', ('0.0.0.0', 12345)), + (b'', ('0.0.0.0', 12345))], + list(transport._buffer)) + self.assertIsInstance(transport._buffer[1][0], bytes) + + @mock.patch('asyncio.proactor_events.logger') + def test_sendto_exception(self, m_log): + data = b'data' + err = self.proactor.sendto.side_effect = RuntimeError() + + transport = self.datagram_transport() + transport._fatal_error = mock.Mock() + transport.sendto(data, ()) + + self.assertTrue(transport._fatal_error.called) + transport._fatal_error.assert_called_with( + err, + 'Fatal write error on datagram transport') + transport._conn_lost = 1 + + transport._address = ('123',) + transport.sendto(data) + transport.sendto(data) + transport.sendto(data) + transport.sendto(data) + transport.sendto(data) + m_log.warning.assert_called_with('socket.sendto() raised exception.') + + def test_sendto_error_received(self): + data = b'data' + + self.sock.sendto.side_effect = ConnectionRefusedError + + transport = self.datagram_transport() + transport._fatal_error = mock.Mock() + transport.sendto(data, ()) + + self.assertEqual(transport._conn_lost, 0) + self.assertFalse(transport._fatal_error.called) + + def test_sendto_error_received_connected(self): + data = b'data' + + self.proactor.send.side_effect = ConnectionRefusedError + + transport = self.datagram_transport(address=('0.0.0.0', 1)) + transport._fatal_error = mock.Mock() + transport.sendto(data) + + self.assertFalse(transport._fatal_error.called) + self.assertTrue(self.protocol.error_received.called) + + def test_sendto_str(self): + transport = self.datagram_transport() + self.assertRaises(TypeError, transport.sendto, 'str', ()) + + def test_sendto_connected_addr(self): + transport = self.datagram_transport(address=('0.0.0.0', 1)) + self.assertRaises( + ValueError, transport.sendto, b'str', ('0.0.0.0', 2)) + + def test_sendto_closing(self): + transport = self.datagram_transport(address=(1,)) + transport.close() + self.assertEqual(transport._conn_lost, 1) + transport.sendto(b'data', (1,)) + self.assertEqual(transport._conn_lost, 2) + + def test__loop_writing_closing(self): + transport = self.datagram_transport() + transport._closing = True + transport._loop_writing() + self.assertIsNone(transport._write_fut) + test_utils.run_briefly(self.loop) + self.sock.close.assert_called_with() + self.protocol.connection_lost.assert_called_with(None) + + def test__loop_writing_exception(self): + err = self.proactor.sendto.side_effect = RuntimeError() + + transport = self.datagram_transport() + transport._fatal_error = mock.Mock() + transport._buffer.append((b'data', ())) + transport._loop_writing() + + transport._fatal_error.assert_called_with( + err, + 'Fatal write error on datagram transport') + + def test__loop_writing_error_received(self): + self.proactor.sendto.side_effect = ConnectionRefusedError + + transport = self.datagram_transport() + transport._fatal_error = mock.Mock() + transport._buffer.append((b'data', ())) + transport._loop_writing() + + self.assertFalse(transport._fatal_error.called) + + def test__loop_writing_error_received_connection(self): + self.proactor.send.side_effect = ConnectionRefusedError + + transport = self.datagram_transport(address=('0.0.0.0', 1)) + transport._fatal_error = mock.Mock() + transport._buffer.append((b'data', ())) + transport._loop_writing() + + self.assertFalse(transport._fatal_error.called) + self.assertTrue(self.protocol.error_received.called) + + @mock.patch('asyncio.base_events.logger.error') + def test_fatal_error_connected(self, m_exc): + transport = self.datagram_transport(address=('0.0.0.0', 1)) + err = ConnectionRefusedError() + transport._fatal_error(err) + self.assertFalse(self.protocol.error_received.called) + m_exc.assert_not_called() + + +class BaseProactorEventLoopTests(test_utils.TestCase): + + def setUp(self): + super().setUp() + + self.sock = test_utils.mock_nonblocking_socket() + self.proactor = mock.Mock() + + self.ssock, self.csock = mock.Mock(), mock.Mock() + + with mock.patch('asyncio.proactor_events.socket.socketpair', + return_value=(self.ssock, self.csock)): + with mock.patch('signal.set_wakeup_fd'): + self.loop = BaseProactorEventLoop(self.proactor) + self.set_event_loop(self.loop) + + @mock.patch('asyncio.proactor_events.socket.socketpair') + def test_ctor(self, socketpair): + ssock, csock = socketpair.return_value = ( + mock.Mock(), mock.Mock()) + with mock.patch('signal.set_wakeup_fd'): + loop = BaseProactorEventLoop(self.proactor) + self.assertIs(loop._ssock, ssock) + self.assertIs(loop._csock, csock) + self.assertEqual(loop._internal_fds, 1) + loop.close() + + def test_close_self_pipe(self): + self.loop._close_self_pipe() + self.assertEqual(self.loop._internal_fds, 0) + self.assertTrue(self.ssock.close.called) + self.assertTrue(self.csock.close.called) + self.assertIsNone(self.loop._ssock) + self.assertIsNone(self.loop._csock) + + # Don't call close(): _close_self_pipe() cannot be called twice + self.loop._closed = True + + def test_close(self): + self.loop._close_self_pipe = mock.Mock() + self.loop.close() + self.assertTrue(self.loop._close_self_pipe.called) + self.assertTrue(self.proactor.close.called) + self.assertIsNone(self.loop._proactor) + + self.loop._close_self_pipe.reset_mock() + self.loop.close() + self.assertFalse(self.loop._close_self_pipe.called) + + def test_make_socket_transport(self): + tr = self.loop._make_socket_transport(self.sock, asyncio.Protocol()) + self.assertIsInstance(tr, _ProactorSocketTransport) + close_transport(tr) + + def test_loop_self_reading(self): + self.loop._loop_self_reading() + self.proactor.recv.assert_called_with(self.ssock, 4096) + self.proactor.recv.return_value.add_done_callback.assert_called_with( + self.loop._loop_self_reading) + + def test_loop_self_reading_fut(self): + fut = mock.Mock() + self.loop._self_reading_future = fut + self.loop._loop_self_reading(fut) + self.assertTrue(fut.result.called) + self.proactor.recv.assert_called_with(self.ssock, 4096) + self.proactor.recv.return_value.add_done_callback.assert_called_with( + self.loop._loop_self_reading) + + def test_loop_self_reading_exception(self): + self.loop.call_exception_handler = mock.Mock() + self.proactor.recv.side_effect = OSError() + self.loop._loop_self_reading() + self.assertTrue(self.loop.call_exception_handler.called) + + def test_write_to_self(self): + self.loop._write_to_self() + self.csock.send.assert_called_with(b'\0') + + def test_process_events(self): + self.loop._process_events([]) + + @mock.patch('asyncio.base_events.logger') + def test_create_server(self, m_log): + pf = mock.Mock() + call_soon = self.loop.call_soon = mock.Mock() + + self.loop._start_serving(pf, self.sock) + self.assertTrue(call_soon.called) + + # callback + loop = call_soon.call_args[0][0] + loop() + self.proactor.accept.assert_called_with(self.sock) + + # conn + fut = mock.Mock() + fut.result.return_value = (mock.Mock(), mock.Mock()) + + make_tr = self.loop._make_socket_transport = mock.Mock() + loop(fut) + self.assertTrue(fut.result.called) + self.assertTrue(make_tr.called) + + # exception + fut.result.side_effect = OSError() + loop(fut) + self.assertTrue(self.sock.close.called) + self.assertTrue(m_log.error.called) + + def test_create_server_cancel(self): + pf = mock.Mock() + call_soon = self.loop.call_soon = mock.Mock() + + self.loop._start_serving(pf, self.sock) + loop = call_soon.call_args[0][0] + + # cancelled + fut = self.loop.create_future() + fut.cancel() + loop(fut) + self.assertTrue(self.sock.close.called) + + def test_stop_serving(self): + sock1 = mock.Mock() + future1 = mock.Mock() + sock2 = mock.Mock() + future2 = mock.Mock() + self.loop._accept_futures = { + sock1.fileno(): future1, + sock2.fileno(): future2 + } + + self.loop._stop_serving(sock1) + self.assertTrue(sock1.close.called) + self.assertTrue(future1.cancel.called) + self.proactor._stop_serving.assert_called_with(sock1) + self.assertFalse(sock2.close.called) + self.assertFalse(future2.cancel.called) + + def datagram_transport(self): + self.protocol = test_utils.make_test_protocol(asyncio.DatagramProtocol) + return self.loop._make_datagram_transport(self.sock, self.protocol) + + def test_make_datagram_transport(self): + tr = self.datagram_transport() + self.assertIsInstance(tr, _ProactorDatagramTransport) + self.assertIsInstance(tr, asyncio.DatagramTransport) + close_transport(tr) + + def test_datagram_loop_writing(self): + tr = self.datagram_transport() + tr._buffer.appendleft((b'data', ('127.0.0.1', 12068))) + tr._loop_writing() + self.loop._proactor.sendto.assert_called_with(self.sock, b'data', addr=('127.0.0.1', 12068)) + self.loop._proactor.sendto.return_value.add_done_callback.\ + assert_called_with(tr._loop_writing) + + close_transport(tr) + + def test_datagram_loop_reading(self): + tr = self.datagram_transport() + tr._loop_reading() + self.loop._proactor.recvfrom.assert_called_with(self.sock, 256 * 1024) + self.assertFalse(self.protocol.datagram_received.called) + self.assertFalse(self.protocol.error_received.called) + close_transport(tr) + + def test_datagram_loop_reading_data(self): + res = self.loop.create_future() + res.set_result((b'data', ('127.0.0.1', 12068))) + + tr = self.datagram_transport() + tr._read_fut = res + tr._loop_reading(res) + self.loop._proactor.recvfrom.assert_called_with(self.sock, 256 * 1024) + self.protocol.datagram_received.assert_called_with(b'data', ('127.0.0.1', 12068)) + close_transport(tr) + + @unittest.skipIf(sys.flags.optimize, "Assertions are disabled in optimized mode") + def test_datagram_loop_reading_no_data(self): + res = self.loop.create_future() + res.set_result((b'', ('127.0.0.1', 12068))) + + tr = self.datagram_transport() + self.assertRaises(AssertionError, tr._loop_reading, res) + + tr.close = mock.Mock() + tr._read_fut = res + tr._loop_reading(res) + self.assertTrue(self.loop._proactor.recvfrom.called) + self.assertFalse(self.protocol.error_received.called) + self.assertFalse(tr.close.called) + close_transport(tr) + + def test_datagram_loop_reading_aborted(self): + err = self.loop._proactor.recvfrom.side_effect = ConnectionAbortedError() + + tr = self.datagram_transport() + tr._fatal_error = mock.Mock() + tr._protocol.error_received = mock.Mock() + tr._loop_reading() + tr._protocol.error_received.assert_called_with(err) + close_transport(tr) + + def test_datagram_loop_writing_aborted(self): + err = self.loop._proactor.sendto.side_effect = ConnectionAbortedError() + + tr = self.datagram_transport() + tr._fatal_error = mock.Mock() + tr._protocol.error_received = mock.Mock() + tr._buffer.appendleft((b'Hello', ('127.0.0.1', 12068))) + tr._loop_writing() + tr._protocol.error_received.assert_called_with(err) + close_transport(tr) + + +@unittest.skipIf(sys.platform != 'win32', + 'Proactor is supported on Windows only') +class ProactorEventLoopUnixSockSendfileTests(test_utils.TestCase): + DATA = b"12345abcde" * 16 * 1024 # 160 KiB + + class MyProto(asyncio.Protocol): + + def __init__(self, loop): + self.started = False + self.closed = False + self.data = bytearray() + self.fut = loop.create_future() + self.transport = None + + def connection_made(self, transport): + self.started = True + self.transport = transport + + def data_received(self, data): + self.data.extend(data) + + def connection_lost(self, exc): + self.closed = True + self.fut.set_result(None) + + async def wait_closed(self): + await self.fut + + @classmethod + def setUpClass(cls): + with open(os_helper.TESTFN, 'wb') as fp: + fp.write(cls.DATA) + super().setUpClass() + + @classmethod + def tearDownClass(cls): + os_helper.unlink(os_helper.TESTFN) + super().tearDownClass() + + def setUp(self): + self.loop = asyncio.ProactorEventLoop() + self.set_event_loop(self.loop) + self.addCleanup(self.loop.close) + self.file = open(os_helper.TESTFN, 'rb') + self.addCleanup(self.file.close) + super().setUp() + + def make_socket(self, cleanup=True, blocking=False): + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + sock.setblocking(blocking) + sock.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 1024) + sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 1024) + if cleanup: + self.addCleanup(sock.close) + return sock + + def run_loop(self, coro): + return self.loop.run_until_complete(coro) + + def prepare(self): + sock = self.make_socket() + proto = self.MyProto(self.loop) + port = socket_helper.find_unused_port() + srv_sock = self.make_socket(cleanup=False) + srv_sock.bind(('127.0.0.1', port)) + server = self.run_loop(self.loop.create_server( + lambda: proto, sock=srv_sock)) + self.run_loop(self.loop.sock_connect(sock, srv_sock.getsockname())) + + def cleanup(): + if proto.transport is not None: + # can be None if the task was cancelled before + # connection_made callback + proto.transport.close() + self.run_loop(proto.wait_closed()) + + server.close() + self.run_loop(server.wait_closed()) + + self.addCleanup(cleanup) + + return sock, proto + + def test_sock_sendfile_not_a_file(self): + sock, proto = self.prepare() + f = object() + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, + "not a regular file"): + self.run_loop(self.loop._sock_sendfile_native(sock, f, + 0, None)) + self.assertEqual(self.file.tell(), 0) + + def test_sock_sendfile_iobuffer(self): + sock, proto = self.prepare() + f = io.BytesIO() + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, + "not a regular file"): + self.run_loop(self.loop._sock_sendfile_native(sock, f, + 0, None)) + self.assertEqual(self.file.tell(), 0) + + def test_sock_sendfile_not_regular_file(self): + sock, proto = self.prepare() + f = mock.Mock() + f.fileno.return_value = -1 + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, + "not a regular file"): + self.run_loop(self.loop._sock_sendfile_native(sock, f, + 0, None)) + self.assertEqual(self.file.tell(), 0) + + def test_blocking_socket(self): + self.loop.set_debug(True) + sock = self.make_socket(blocking=True) + with self.assertRaisesRegex(ValueError, "must be non-blocking"): + self.run_loop(self.loop.sock_sendfile(sock, self.file)) + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_protocols.py b/Lib/test/test_asyncio/test_protocols.py new file mode 100644 index 00000000000..29d3bd22705 --- /dev/null +++ b/Lib/test/test_asyncio/test_protocols.py @@ -0,0 +1,67 @@ +import unittest +from unittest import mock + +import asyncio + + +def tearDownModule(): + # not needed for the test file but added for uniformness with all other + # asyncio test files for the sake of unified cleanup + asyncio.events._set_event_loop_policy(None) + + +class ProtocolsAbsTests(unittest.TestCase): + + def test_base_protocol(self): + f = mock.Mock() + p = asyncio.BaseProtocol() + self.assertIsNone(p.connection_made(f)) + self.assertIsNone(p.connection_lost(f)) + self.assertIsNone(p.pause_writing()) + self.assertIsNone(p.resume_writing()) + self.assertNotHasAttr(p, '__dict__') + + def test_protocol(self): + f = mock.Mock() + p = asyncio.Protocol() + self.assertIsNone(p.connection_made(f)) + self.assertIsNone(p.connection_lost(f)) + self.assertIsNone(p.data_received(f)) + self.assertIsNone(p.eof_received()) + self.assertIsNone(p.pause_writing()) + self.assertIsNone(p.resume_writing()) + self.assertNotHasAttr(p, '__dict__') + + def test_buffered_protocol(self): + f = mock.Mock() + p = asyncio.BufferedProtocol() + self.assertIsNone(p.connection_made(f)) + self.assertIsNone(p.connection_lost(f)) + self.assertIsNone(p.get_buffer(100)) + self.assertIsNone(p.buffer_updated(150)) + self.assertIsNone(p.pause_writing()) + self.assertIsNone(p.resume_writing()) + self.assertNotHasAttr(p, '__dict__') + + def test_datagram_protocol(self): + f = mock.Mock() + dp = asyncio.DatagramProtocol() + self.assertIsNone(dp.connection_made(f)) + self.assertIsNone(dp.connection_lost(f)) + self.assertIsNone(dp.error_received(f)) + self.assertIsNone(dp.datagram_received(f, f)) + self.assertNotHasAttr(dp, '__dict__') + + def test_subprocess_protocol(self): + f = mock.Mock() + sp = asyncio.SubprocessProtocol() + self.assertIsNone(sp.connection_made(f)) + self.assertIsNone(sp.connection_lost(f)) + self.assertIsNone(sp.pipe_data_received(1, f)) + self.assertIsNone(sp.pipe_connection_lost(1, f)) + self.assertIsNone(sp.process_exited()) + self.assertNotHasAttr(sp, '__dict__') + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_queues.py b/Lib/test/test_asyncio/test_queues.py new file mode 100644 index 00000000000..54bbe79f81f --- /dev/null +++ b/Lib/test/test_asyncio/test_queues.py @@ -0,0 +1,725 @@ +"""Tests for queues.py""" + +import asyncio +import unittest +from types import GenericAlias + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +class QueueBasicTests(unittest.IsolatedAsyncioTestCase): + + async def _test_repr_or_str(self, fn, expect_id): + """Test Queue's repr or str. + + fn is repr or str. expect_id is True if we expect the Queue's id to + appear in fn(Queue()). + """ + q = asyncio.Queue() + self.assertStartsWith(fn(q), ' 0 + + self.assertEqual(q.qsize(), 0) + + # Ensure join() task successfully finishes + await q.join() + + # Ensure get() task is finished, and raised ShutDown + await asyncio.sleep(0) + self.assertTrue(get_task.done()) + with self.assertRaisesShutdown(): + await get_task + + # Ensure put() and get() raise ShutDown + with self.assertRaisesShutdown(): + await q.put("data") + with self.assertRaisesShutdown(): + q.put_nowait("data") + + with self.assertRaisesShutdown(): + await q.get() + with self.assertRaisesShutdown(): + q.get_nowait() + + async def test_shutdown_nonempty(self): + # Test shutting down a non-empty queue + + # Setup full queue with 1 item, and join() and put() tasks + q = self.q_class(maxsize=1) + loop = asyncio.get_running_loop() + + q.put_nowait("data") + join_task = loop.create_task(q.join()) + put_task = loop.create_task(q.put("data2")) + + # Ensure put() task is not finished + await asyncio.sleep(0) + self.assertFalse(put_task.done()) + + # Perform shut-down + q.shutdown(immediate=False) # unfinished tasks: 1 -> 1 + + self.assertEqual(q.qsize(), 1) + + # Ensure put() task is finished, and raised ShutDown + await asyncio.sleep(0) + self.assertTrue(put_task.done()) + with self.assertRaisesShutdown(): + await put_task + + # Ensure get() succeeds on enqueued item + self.assertEqual(await q.get(), "data") + + # Ensure join() task is not finished + await asyncio.sleep(0) + self.assertFalse(join_task.done()) + + # Ensure put() and get() raise ShutDown + with self.assertRaisesShutdown(): + await q.put("data") + with self.assertRaisesShutdown(): + q.put_nowait("data") + + with self.assertRaisesShutdown(): + await q.get() + with self.assertRaisesShutdown(): + q.get_nowait() + + # Ensure there is 1 unfinished task, and join() task succeeds + q.task_done() + + await asyncio.sleep(0) + self.assertTrue(join_task.done()) + await join_task + + with self.assertRaises( + ValueError, msg="Didn't appear to mark all tasks done" + ): + q.task_done() + + async def test_shutdown_immediate(self): + # Test immediately shutting down a queue + + # Setup queue with 1 item, and a join() task + q = self.q_class() + loop = asyncio.get_running_loop() + q.put_nowait("data") + join_task = loop.create_task(q.join()) + + # Perform shut-down + q.shutdown(immediate=True) # unfinished tasks: 1 -> 0 + + self.assertEqual(q.qsize(), 0) + + # Ensure join() task has successfully finished + await asyncio.sleep(0) + self.assertTrue(join_task.done()) + await join_task + + # Ensure put() and get() raise ShutDown + with self.assertRaisesShutdown(): + await q.put("data") + with self.assertRaisesShutdown(): + q.put_nowait("data") + + with self.assertRaisesShutdown(): + await q.get() + with self.assertRaisesShutdown(): + q.get_nowait() + + # Ensure there are no unfinished tasks + with self.assertRaises( + ValueError, msg="Didn't appear to mark all tasks done" + ): + q.task_done() + + async def test_shutdown_immediate_with_unfinished(self): + # Test immediately shutting down a queue with unfinished tasks + + # Setup queue with 2 items (1 retrieved), and a join() task + q = self.q_class() + loop = asyncio.get_running_loop() + q.put_nowait("data") + q.put_nowait("data") + join_task = loop.create_task(q.join()) + self.assertEqual(await q.get(), "data") + + # Perform shut-down + q.shutdown(immediate=True) # unfinished tasks: 2 -> 1 + + self.assertEqual(q.qsize(), 0) + + # Ensure join() task is not finished + await asyncio.sleep(0) + self.assertFalse(join_task.done()) + + # Ensure put() and get() raise ShutDown + with self.assertRaisesShutdown(): + await q.put("data") + with self.assertRaisesShutdown(): + q.put_nowait("data") + + with self.assertRaisesShutdown(): + await q.get() + with self.assertRaisesShutdown(): + q.get_nowait() + + # Ensure there is 1 unfinished task + q.task_done() + with self.assertRaises( + ValueError, msg="Didn't appear to mark all tasks done" + ): + q.task_done() + + # Ensure join() task has successfully finished + await asyncio.sleep(0) + self.assertTrue(join_task.done()) + await join_task + + +class QueueShutdownTests( + _QueueShutdownTestMixin, unittest.IsolatedAsyncioTestCase +): + q_class = asyncio.Queue + + +class LifoQueueShutdownTests( + _QueueShutdownTestMixin, unittest.IsolatedAsyncioTestCase +): + q_class = asyncio.LifoQueue + + +class PriorityQueueShutdownTests( + _QueueShutdownTestMixin, unittest.IsolatedAsyncioTestCase +): + q_class = asyncio.PriorityQueue + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_runners.py b/Lib/test/test_asyncio/test_runners.py new file mode 100644 index 00000000000..de489c2dc43 --- /dev/null +++ b/Lib/test/test_asyncio/test_runners.py @@ -0,0 +1,528 @@ +import _thread +import asyncio +import contextvars +import re +import signal +import sys +import threading +import unittest +from test.test_asyncio import utils as test_utils +from unittest import mock +from unittest.mock import patch + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +def interrupt_self(): + _thread.interrupt_main() + + +class TestPolicy(asyncio.events._AbstractEventLoopPolicy): + + def __init__(self, loop_factory): + self.loop_factory = loop_factory + self.loop = None + + def get_event_loop(self): + # shouldn't ever be called by asyncio.run() + raise RuntimeError + + def new_event_loop(self): + return self.loop_factory() + + def set_event_loop(self, loop): + if loop is not None: + # we want to check if the loop is closed + # in BaseTest.tearDown + self.loop = loop + + +class BaseTest(unittest.TestCase): + + def new_loop(self): + loop = asyncio.BaseEventLoop() + loop._process_events = mock.Mock() + # Mock waking event loop from select + loop._write_to_self = mock.Mock() + loop._write_to_self.return_value = None + loop._selector = mock.Mock() + loop._selector.select.return_value = () + loop.shutdown_ag_run = False + + async def shutdown_asyncgens(): + loop.shutdown_ag_run = True + loop.shutdown_asyncgens = shutdown_asyncgens + + return loop + + def setUp(self): + super().setUp() + + policy = TestPolicy(self.new_loop) + asyncio.events._set_event_loop_policy(policy) + + def tearDown(self): + policy = asyncio.events._get_event_loop_policy() + if policy.loop is not None: + self.assertTrue(policy.loop.is_closed()) + self.assertTrue(policy.loop.shutdown_ag_run) + + asyncio.events._set_event_loop_policy(None) + super().tearDown() + + +class RunTests(BaseTest): + + def test_asyncio_run_return(self): + async def main(): + await asyncio.sleep(0) + return 42 + + self.assertEqual(asyncio.run(main()), 42) + + def test_asyncio_run_raises(self): + async def main(): + await asyncio.sleep(0) + raise ValueError('spam') + + with self.assertRaisesRegex(ValueError, 'spam'): + asyncio.run(main()) + + def test_asyncio_run_only_coro(self): + for o in {1, lambda: None}: + with self.subTest(obj=o), \ + self.assertRaisesRegex(TypeError, + 'an awaitable is required'): + asyncio.run(o) + + def test_asyncio_run_debug(self): + async def main(expected): + loop = asyncio.get_event_loop() + self.assertIs(loop.get_debug(), expected) + + asyncio.run(main(False), debug=False) + asyncio.run(main(True), debug=True) + with mock.patch('asyncio.coroutines._is_debug_mode', lambda: True): + asyncio.run(main(True)) + asyncio.run(main(False), debug=False) + with mock.patch('asyncio.coroutines._is_debug_mode', lambda: False): + asyncio.run(main(True), debug=True) + asyncio.run(main(False)) + + def test_asyncio_run_from_running_loop(self): + async def main(): + coro = main() + try: + asyncio.run(coro) + finally: + coro.close() # Suppress ResourceWarning + + with self.assertRaisesRegex(RuntimeError, + 'cannot be called from a running'): + asyncio.run(main()) + + def test_asyncio_run_cancels_hanging_tasks(self): + lo_task = None + + async def leftover(): + await asyncio.sleep(0.1) + + async def main(): + nonlocal lo_task + lo_task = asyncio.create_task(leftover()) + return 123 + + self.assertEqual(asyncio.run(main()), 123) + self.assertTrue(lo_task.done()) + + def test_asyncio_run_reports_hanging_tasks_errors(self): + lo_task = None + call_exc_handler_mock = mock.Mock() + + async def leftover(): + try: + await asyncio.sleep(0.1) + except asyncio.CancelledError: + 1 / 0 + + async def main(): + loop = asyncio.get_running_loop() + loop.call_exception_handler = call_exc_handler_mock + + nonlocal lo_task + lo_task = asyncio.create_task(leftover()) + return 123 + + self.assertEqual(asyncio.run(main()), 123) + self.assertTrue(lo_task.done()) + + call_exc_handler_mock.assert_called_with({ + 'message': test_utils.MockPattern(r'asyncio.run.*shutdown'), + 'task': lo_task, + 'exception': test_utils.MockInstanceOf(ZeroDivisionError) + }) + + def test_asyncio_run_closes_gens_after_hanging_tasks_errors(self): + spinner = None + lazyboy = None + + class FancyExit(Exception): + pass + + async def fidget(): + while True: + yield 1 + await asyncio.sleep(1) + + async def spin(): + nonlocal spinner + spinner = fidget() + try: + async for the_meaning_of_life in spinner: # NoQA + pass + except asyncio.CancelledError: + 1 / 0 + + async def main(): + loop = asyncio.get_running_loop() + loop.call_exception_handler = mock.Mock() + + nonlocal lazyboy + lazyboy = asyncio.create_task(spin()) + raise FancyExit + + with self.assertRaises(FancyExit): + asyncio.run(main()) + + self.assertTrue(lazyboy.done()) + + self.assertIsNone(spinner.ag_frame) + self.assertFalse(spinner.ag_running) + + def test_asyncio_run_set_event_loop(self): + #See https://github.com/python/cpython/issues/93896 + + async def main(): + await asyncio.sleep(0) + return 42 + + policy = asyncio.events._get_event_loop_policy() + policy.set_event_loop = mock.Mock() + asyncio.run(main()) + self.assertTrue(policy.set_event_loop.called) + + def test_asyncio_run_without_uncancel(self): + # See https://github.com/python/cpython/issues/95097 + class Task: + def __init__(self, loop, coro, **kwargs): + self._task = asyncio.Task(coro, loop=loop, **kwargs) + + def cancel(self, *args, **kwargs): + return self._task.cancel(*args, **kwargs) + + def add_done_callback(self, *args, **kwargs): + return self._task.add_done_callback(*args, **kwargs) + + def remove_done_callback(self, *args, **kwargs): + return self._task.remove_done_callback(*args, **kwargs) + + @property + def _asyncio_future_blocking(self): + return self._task._asyncio_future_blocking + + def result(self, *args, **kwargs): + return self._task.result(*args, **kwargs) + + def done(self, *args, **kwargs): + return self._task.done(*args, **kwargs) + + def cancelled(self, *args, **kwargs): + return self._task.cancelled(*args, **kwargs) + + def exception(self, *args, **kwargs): + return self._task.exception(*args, **kwargs) + + def get_loop(self, *args, **kwargs): + return self._task.get_loop(*args, **kwargs) + + def set_name(self, *args, **kwargs): + return self._task.set_name(*args, **kwargs) + + async def main(): + interrupt_self() + await asyncio.Event().wait() + + def new_event_loop(): + loop = self.new_loop() + loop.set_task_factory(Task) + return loop + + asyncio.events._set_event_loop_policy(TestPolicy(new_event_loop)) + with self.assertRaises(asyncio.CancelledError): + asyncio.run(main()) + + def test_asyncio_run_loop_factory(self): + factory = mock.Mock() + loop = factory.return_value = self.new_loop() + + async def main(): + self.assertEqual(asyncio.get_running_loop(), loop) + + asyncio.run(main(), loop_factory=factory) + factory.assert_called_once_with() + + def test_loop_factory_default_event_loop(self): + async def main(): + if sys.platform == "win32": + self.assertIsInstance(asyncio.get_running_loop(), asyncio.ProactorEventLoop) + else: + self.assertIsInstance(asyncio.get_running_loop(), asyncio.SelectorEventLoop) + + + asyncio.run(main(), loop_factory=asyncio.EventLoop) + + +class RunnerTests(BaseTest): + + def test_non_debug(self): + with asyncio.Runner(debug=False) as runner: + self.assertFalse(runner.get_loop().get_debug()) + + def test_debug(self): + with asyncio.Runner(debug=True) as runner: + self.assertTrue(runner.get_loop().get_debug()) + + def test_custom_factory(self): + loop = mock.Mock() + with asyncio.Runner(loop_factory=lambda: loop) as runner: + self.assertIs(runner.get_loop(), loop) + + def test_run(self): + async def f(): + await asyncio.sleep(0) + return 'done' + + with asyncio.Runner() as runner: + self.assertEqual('done', runner.run(f())) + loop = runner.get_loop() + + with self.assertRaisesRegex( + RuntimeError, + "Runner is closed" + ): + runner.get_loop() + + self.assertTrue(loop.is_closed()) + + def test_run_non_coro(self): + with asyncio.Runner() as runner: + with self.assertRaisesRegex( + TypeError, + "an awaitable is required" + ): + runner.run(123) + + def test_run_future(self): + with asyncio.Runner() as runner: + fut = runner.get_loop().create_future() + fut.set_result('done') + self.assertEqual('done', runner.run(fut)) + + def test_run_awaitable(self): + class MyAwaitable: + def __await__(self): + return self.run().__await__() + + @staticmethod + async def run(): + return 'done' + + with asyncio.Runner() as runner: + self.assertEqual('done', runner.run(MyAwaitable())) + + def test_explicit_close(self): + runner = asyncio.Runner() + loop = runner.get_loop() + runner.close() + with self.assertRaisesRegex( + RuntimeError, + "Runner is closed" + ): + runner.get_loop() + + self.assertTrue(loop.is_closed()) + + def test_double_close(self): + runner = asyncio.Runner() + loop = runner.get_loop() + + runner.close() + self.assertTrue(loop.is_closed()) + + # the second call is no-op + runner.close() + self.assertTrue(loop.is_closed()) + + def test_second_with_block_raises(self): + ret = [] + + async def f(arg): + ret.append(arg) + + runner = asyncio.Runner() + with runner: + runner.run(f(1)) + + with self.assertRaisesRegex( + RuntimeError, + "Runner is closed" + ): + with runner: + runner.run(f(2)) + + self.assertEqual([1], ret) + + def test_run_keeps_context(self): + cvar = contextvars.ContextVar("cvar", default=-1) + + async def f(val): + old = cvar.get() + await asyncio.sleep(0) + cvar.set(val) + return old + + async def get_context(): + return contextvars.copy_context() + + with asyncio.Runner() as runner: + self.assertEqual(-1, runner.run(f(1))) + self.assertEqual(1, runner.run(f(2))) + + self.assertEqual(2, runner.run(get_context()).get(cvar)) + + @unittest.expectedFailure # TODO: RUSTPYTHON; - RuntimeWarning for unawaited coroutine not triggered + def test_recursive_run(self): + async def g(): + pass + + async def f(): + runner.run(g()) + + with asyncio.Runner() as runner: + with self.assertWarnsRegex( + RuntimeWarning, + "coroutine .+ was never awaited", + ): + with self.assertRaisesRegex( + RuntimeError, + re.escape( + "Runner.run() cannot be called from a running event loop" + ), + ): + runner.run(f()) + + def test_interrupt_call_soon(self): + # The only case when task is not suspended by waiting a future + # or another task + assert threading.current_thread() is threading.main_thread() + + async def coro(): + with self.assertRaises(asyncio.CancelledError): + while True: + await asyncio.sleep(0) + raise asyncio.CancelledError() + + with asyncio.Runner() as runner: + runner.get_loop().call_later(0.1, interrupt_self) + with self.assertRaises(KeyboardInterrupt): + runner.run(coro()) + + def test_interrupt_wait(self): + # interrupting when waiting a future cancels both future and main task + assert threading.current_thread() is threading.main_thread() + + async def coro(fut): + with self.assertRaises(asyncio.CancelledError): + await fut + raise asyncio.CancelledError() + + with asyncio.Runner() as runner: + fut = runner.get_loop().create_future() + runner.get_loop().call_later(0.1, interrupt_self) + + with self.assertRaises(KeyboardInterrupt): + runner.run(coro(fut)) + + self.assertTrue(fut.cancelled()) + + def test_interrupt_cancelled_task(self): + # interrupting cancelled main task doesn't raise KeyboardInterrupt + assert threading.current_thread() is threading.main_thread() + + async def subtask(task): + await asyncio.sleep(0) + task.cancel() + interrupt_self() + + async def coro(): + asyncio.create_task(subtask(asyncio.current_task())) + await asyncio.sleep(10) + + with asyncio.Runner() as runner: + with self.assertRaises(asyncio.CancelledError): + runner.run(coro()) + + def test_signal_install_not_supported_ok(self): + # signal.signal() can throw if the "main thread" doesn't have signals enabled + assert threading.current_thread() is threading.main_thread() + + async def coro(): + pass + + with asyncio.Runner() as runner: + with patch.object( + signal, + "signal", + side_effect=ValueError( + "signal only works in main thread of the main interpreter" + ) + ): + runner.run(coro()) + + def test_set_event_loop_called_once(self): + # See https://github.com/python/cpython/issues/95736 + async def coro(): + pass + + policy = asyncio.events._get_event_loop_policy() + policy.set_event_loop = mock.Mock() + runner = asyncio.Runner() + runner.run(coro()) + runner.run(coro()) + + self.assertEqual(1, policy.set_event_loop.call_count) + runner.close() + + def test_no_repr_is_call_on_the_task_result(self): + # See https://github.com/python/cpython/issues/112559. + class MyResult: + def __init__(self): + self.repr_count = 0 + def __repr__(self): + self.repr_count += 1 + return super().__repr__() + + async def coro(): + return MyResult() + + + with asyncio.Runner() as runner: + result = runner.run(coro()) + + self.assertEqual(0, result.repr_count) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py new file mode 100644 index 00000000000..4bb5d4fb816 --- /dev/null +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -0,0 +1,1660 @@ +"""Tests for selector_events.py""" + +import collections +import selectors +import socket +import sys +import unittest +from asyncio import selector_events +from unittest import mock + +try: + import ssl +except ImportError: + ssl = None + +import asyncio +from asyncio.selector_events import (BaseSelectorEventLoop, + _SelectorDatagramTransport, + _SelectorSocketTransport, + _SelectorTransport) +from test.test_asyncio import utils as test_utils + +MOCK_ANY = mock.ANY + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +class TestBaseSelectorEventLoop(BaseSelectorEventLoop): + + def _make_self_pipe(self): + self._ssock = mock.Mock() + self._csock = mock.Mock() + self._internal_fds += 1 + + def _close_self_pipe(self): + pass + + +def list_to_buffer(l=()): + buffer = collections.deque() + buffer.extend((memoryview(i) for i in l)) + return buffer + + + +def close_transport(transport): + # Don't call transport.close() because the event loop and the selector + # are mocked + if transport._sock is None: + return + transport._sock.close() + transport._sock = None + + +class BaseSelectorEventLoopTests(test_utils.TestCase): + + def setUp(self): + super().setUp() + self.selector = mock.Mock() + self.selector.select.return_value = [] + self.loop = TestBaseSelectorEventLoop(self.selector) + self.set_event_loop(self.loop) + + def test_make_socket_transport(self): + m = mock.Mock() + self.loop.add_reader = mock.Mock() + self.loop._ensure_fd_no_transport = mock.Mock() + transport = self.loop._make_socket_transport(m, asyncio.Protocol()) + self.assertIsInstance(transport, _SelectorSocketTransport) + self.assertEqual(self.loop._ensure_fd_no_transport.call_count, 1) + + # Calling repr() must not fail when the event loop is closed + self.loop.close() + repr(transport) + + close_transport(transport) + + @mock.patch('asyncio.selector_events.ssl', None) + @mock.patch('asyncio.sslproto.ssl', None) + def test_make_ssl_transport_without_ssl_error(self): + m = mock.Mock() + self.loop.add_reader = mock.Mock() + self.loop.add_writer = mock.Mock() + self.loop.remove_reader = mock.Mock() + self.loop.remove_writer = mock.Mock() + self.loop._ensure_fd_no_transport = mock.Mock() + with self.assertRaises(RuntimeError): + self.loop._make_ssl_transport(m, m, m, m) + self.assertEqual(self.loop._ensure_fd_no_transport.call_count, 1) + + def test_close(self): + class EventLoop(BaseSelectorEventLoop): + def _make_self_pipe(self): + self._ssock = mock.Mock() + self._csock = mock.Mock() + self._internal_fds += 1 + + self.loop = EventLoop(self.selector) + self.set_event_loop(self.loop) + + ssock = self.loop._ssock + ssock.fileno.return_value = 7 + csock = self.loop._csock + csock.fileno.return_value = 1 + remove_reader = self.loop._remove_reader = mock.Mock() + + self.loop._selector.close() + self.loop._selector = selector = mock.Mock() + self.assertFalse(self.loop.is_closed()) + + self.loop.close() + self.assertTrue(self.loop.is_closed()) + self.assertIsNone(self.loop._selector) + self.assertIsNone(self.loop._csock) + self.assertIsNone(self.loop._ssock) + selector.close.assert_called_with() + ssock.close.assert_called_with() + csock.close.assert_called_with() + remove_reader.assert_called_with(7) + + # it should be possible to call close() more than once + self.loop.close() + self.loop.close() + + # operation blocked when the loop is closed + f = self.loop.create_future() + self.assertRaises(RuntimeError, self.loop.run_forever) + self.assertRaises(RuntimeError, self.loop.run_until_complete, f) + fd = 0 + def callback(): + pass + self.assertRaises(RuntimeError, self.loop.add_reader, fd, callback) + self.assertRaises(RuntimeError, self.loop.add_writer, fd, callback) + + def test_close_no_selector(self): + self.loop.remove_reader = mock.Mock() + self.loop._selector.close() + self.loop._selector = None + self.loop.close() + self.assertIsNone(self.loop._selector) + + def test_read_from_self_tryagain(self): + self.loop._ssock.recv.side_effect = BlockingIOError + self.assertIsNone(self.loop._read_from_self()) + + def test_read_from_self_exception(self): + self.loop._ssock.recv.side_effect = OSError + self.assertRaises(OSError, self.loop._read_from_self) + + def test_write_to_self_tryagain(self): + self.loop._csock.send.side_effect = BlockingIOError + with test_utils.disable_logger(): + self.assertIsNone(self.loop._write_to_self()) + + def test_write_to_self_exception(self): + # _write_to_self() swallows OSError + self.loop._csock.send.side_effect = RuntimeError() + self.assertRaises(RuntimeError, self.loop._write_to_self) + + @mock.patch('socket.getaddrinfo') + def test_sock_connect_resolve_using_socket_params(self, m_gai): + addr = ('need-resolution.com', 8080) + for sock_type in [socket.SOCK_STREAM, socket.SOCK_DGRAM]: + with self.subTest(sock_type): + sock = test_utils.mock_nonblocking_socket(type=sock_type) + + m_gai.side_effect = \ + lambda *args: [(None, None, None, None, ('127.0.0.1', 0))] + + con = self.loop.create_task(self.loop.sock_connect(sock, addr)) + self.loop.run_until_complete(con) + m_gai.assert_called_with( + addr[0], addr[1], sock.family, sock.type, sock.proto, 0) + + self.loop.run_until_complete(con) + sock.connect.assert_called_with(('127.0.0.1', 0)) + + def test_add_reader(self): + self.loop._selector.get_map.return_value = {} + cb = lambda: True + self.loop.add_reader(1, cb) + + self.assertTrue(self.loop._selector.register.called) + fd, mask, (r, w) = self.loop._selector.register.call_args[0] + self.assertEqual(1, fd) + self.assertEqual(selectors.EVENT_READ, mask) + self.assertEqual(cb, r._callback) + self.assertIsNone(w) + + def test_add_reader_existing(self): + reader = mock.Mock() + writer = mock.Mock() + self.loop._selector.get_map.return_value = {1: selectors.SelectorKey( + 1, 1, selectors.EVENT_WRITE, (reader, writer))} + cb = lambda: True + self.loop.add_reader(1, cb) + + self.assertTrue(reader.cancel.called) + self.assertFalse(self.loop._selector.register.called) + self.assertTrue(self.loop._selector.modify.called) + fd, mask, (r, w) = self.loop._selector.modify.call_args[0] + self.assertEqual(1, fd) + self.assertEqual(selectors.EVENT_WRITE | selectors.EVENT_READ, mask) + self.assertEqual(cb, r._callback) + self.assertEqual(writer, w) + + def test_add_reader_existing_writer(self): + writer = mock.Mock() + self.loop._selector.get_map.return_value = {1: selectors.SelectorKey( + 1, 1, selectors.EVENT_WRITE, (None, writer))} + cb = lambda: True + self.loop.add_reader(1, cb) + + self.assertFalse(self.loop._selector.register.called) + self.assertTrue(self.loop._selector.modify.called) + fd, mask, (r, w) = self.loop._selector.modify.call_args[0] + self.assertEqual(1, fd) + self.assertEqual(selectors.EVENT_WRITE | selectors.EVENT_READ, mask) + self.assertEqual(cb, r._callback) + self.assertEqual(writer, w) + + def test_remove_reader(self): + self.loop._selector.get_map.return_value = {1: selectors.SelectorKey( + 1, 1, selectors.EVENT_READ, (None, None))} + self.assertFalse(self.loop.remove_reader(1)) + + self.assertTrue(self.loop._selector.unregister.called) + + def test_remove_reader_read_write(self): + reader = mock.Mock() + writer = mock.Mock() + self.loop._selector.get_map.return_value = {1: selectors.SelectorKey( + 1, 1, selectors.EVENT_READ | selectors.EVENT_WRITE, + (reader, writer))} + self.assertTrue( + self.loop.remove_reader(1)) + + self.assertFalse(self.loop._selector.unregister.called) + self.assertEqual( + (1, selectors.EVENT_WRITE, (None, writer)), + self.loop._selector.modify.call_args[0]) + + def test_remove_reader_unknown(self): + self.loop._selector.get_map.return_value = {} + self.assertFalse( + self.loop.remove_reader(1)) + + def test_add_writer(self): + self.loop._selector.get_map.return_value = {} + cb = lambda: True + self.loop.add_writer(1, cb) + + self.assertTrue(self.loop._selector.register.called) + fd, mask, (r, w) = self.loop._selector.register.call_args[0] + self.assertEqual(1, fd) + self.assertEqual(selectors.EVENT_WRITE, mask) + self.assertIsNone(r) + self.assertEqual(cb, w._callback) + + def test_add_writer_existing(self): + reader = mock.Mock() + writer = mock.Mock() + self.loop._selector.get_map.return_value = {1: selectors.SelectorKey( + 1, 1, selectors.EVENT_READ, (reader, writer))} + cb = lambda: True + self.loop.add_writer(1, cb) + + self.assertTrue(writer.cancel.called) + self.assertFalse(self.loop._selector.register.called) + self.assertTrue(self.loop._selector.modify.called) + fd, mask, (r, w) = self.loop._selector.modify.call_args[0] + self.assertEqual(1, fd) + self.assertEqual(selectors.EVENT_WRITE | selectors.EVENT_READ, mask) + self.assertEqual(reader, r) + self.assertEqual(cb, w._callback) + + def test_remove_writer(self): + self.loop._selector.get_map.return_value = {1: selectors.SelectorKey( + 1, 1, selectors.EVENT_WRITE, (None, None))} + self.assertFalse(self.loop.remove_writer(1)) + + self.assertTrue(self.loop._selector.unregister.called) + + def test_remove_writer_read_write(self): + reader = mock.Mock() + writer = mock.Mock() + self.loop._selector.get_map.return_value = {1: selectors.SelectorKey( + 1, 1, selectors.EVENT_READ | selectors.EVENT_WRITE, + (reader, writer))} + self.assertTrue( + self.loop.remove_writer(1)) + + self.assertFalse(self.loop._selector.unregister.called) + self.assertEqual( + (1, selectors.EVENT_READ, (reader, None)), + self.loop._selector.modify.call_args[0]) + + def test_remove_writer_unknown(self): + self.loop._selector.get_map.return_value = {} + self.assertFalse( + self.loop.remove_writer(1)) + + def test_process_events_read(self): + reader = mock.Mock() + reader._cancelled = False + + self.loop._add_callback = mock.Mock() + self.loop._process_events( + [(selectors.SelectorKey( + 1, 1, selectors.EVENT_READ, (reader, None)), + selectors.EVENT_READ)]) + self.assertTrue(self.loop._add_callback.called) + self.loop._add_callback.assert_called_with(reader) + + def test_process_events_read_cancelled(self): + reader = mock.Mock() + reader.cancelled = True + + self.loop._remove_reader = mock.Mock() + self.loop._process_events( + [(selectors.SelectorKey( + 1, 1, selectors.EVENT_READ, (reader, None)), + selectors.EVENT_READ)]) + self.loop._remove_reader.assert_called_with(1) + + def test_process_events_write(self): + writer = mock.Mock() + writer._cancelled = False + + self.loop._add_callback = mock.Mock() + self.loop._process_events( + [(selectors.SelectorKey(1, 1, selectors.EVENT_WRITE, + (None, writer)), + selectors.EVENT_WRITE)]) + self.loop._add_callback.assert_called_with(writer) + + def test_process_events_write_cancelled(self): + writer = mock.Mock() + writer.cancelled = True + self.loop._remove_writer = mock.Mock() + + self.loop._process_events( + [(selectors.SelectorKey(1, 1, selectors.EVENT_WRITE, + (None, writer)), + selectors.EVENT_WRITE)]) + self.loop._remove_writer.assert_called_with(1) + + def test_accept_connection_zero_one(self): + for backlog in [0, 1]: + sock = mock.Mock() + sock.accept.return_value = (mock.Mock(), mock.Mock()) + with self.subTest(backlog): + mock_obj = mock.patch.object + with mock_obj(self.loop, '_accept_connection2') as accept2_mock: + self.loop._accept_connection( + mock.Mock(), sock, backlog=backlog) + self.loop.run_until_complete(asyncio.sleep(0)) + self.assertEqual(sock.accept.call_count, backlog + 1) + + def test_accept_connection_multiple(self): + sock = mock.Mock() + sock.accept.return_value = (mock.Mock(), mock.Mock()) + backlog = 100 + # Mock the coroutine generation for a connection to prevent + # warnings related to un-awaited coroutines. _accept_connection2 + # is an async function that is patched with AsyncMock. create_task + # creates a task out of coroutine returned by AsyncMock, so use + # asyncio.sleep(0) to ensure created tasks are complete to avoid + # task pending warnings. + mock_obj = mock.patch.object + with mock_obj(self.loop, '_accept_connection2') as accept2_mock: + self.loop._accept_connection( + mock.Mock(), sock, backlog=backlog) + self.loop.run_until_complete(asyncio.sleep(0)) + self.assertEqual(sock.accept.call_count, backlog + 1) + + def test_accept_connection_skip_connectionabortederror(self): + sock = mock.Mock() + + def mock_sock_accept(): + # mock accept(2) returning -ECONNABORTED every-other + # time that it's called. This applies most to OpenBSD + # whose sockets generate this errno more reproducibly than + # Linux and other OS. + if sock.accept.call_count % 2 == 0: + raise ConnectionAbortedError + return (mock.Mock(), mock.Mock()) + + sock.accept.side_effect = mock_sock_accept + backlog = 100 + # test that _accept_connection's loop calls sock.accept + # all 100 times, continuing past ConnectionAbortedError + # instead of unnecessarily returning early + mock_obj = mock.patch.object + with mock_obj(self.loop, '_accept_connection2') as accept2_mock: + self.loop._accept_connection( + mock.Mock(), sock, backlog=backlog) + # as in test_accept_connection_multiple avoid task pending + # warnings by using asyncio.sleep(0) + self.loop.run_until_complete(asyncio.sleep(0)) + self.assertEqual(sock.accept.call_count, backlog + 1) + +class SelectorTransportTests(test_utils.TestCase): + + def setUp(self): + super().setUp() + self.loop = self.new_test_loop() + self.protocol = test_utils.make_test_protocol(asyncio.Protocol) + self.sock = mock.Mock(socket.socket) + self.sock.fileno.return_value = 7 + + def create_transport(self): + transport = _SelectorTransport(self.loop, self.sock, self.protocol, + None) + self.addCleanup(close_transport, transport) + return transport + + def test_ctor(self): + tr = self.create_transport() + self.assertIs(tr._loop, self.loop) + self.assertIs(tr._sock, self.sock) + self.assertIs(tr._sock_fd, 7) + + def test_abort(self): + tr = self.create_transport() + tr._force_close = mock.Mock() + + tr.abort() + tr._force_close.assert_called_with(None) + + def test_close(self): + tr = self.create_transport() + tr.close() + + self.assertTrue(tr.is_closing()) + self.assertEqual(1, self.loop.remove_reader_count[7]) + self.protocol.connection_lost(None) + self.assertEqual(tr._conn_lost, 1) + + tr.close() + self.assertEqual(tr._conn_lost, 1) + self.assertEqual(1, self.loop.remove_reader_count[7]) + + def test_close_write_buffer(self): + tr = self.create_transport() + tr._buffer.extend(b'data') + tr.close() + + self.assertFalse(self.loop.readers) + test_utils.run_briefly(self.loop) + self.assertFalse(self.protocol.connection_lost.called) + + def test_force_close(self): + tr = self.create_transport() + tr._buffer.extend(b'1') + self.loop._add_reader(7, mock.sentinel) + self.loop._add_writer(7, mock.sentinel) + tr._force_close(None) + + self.assertTrue(tr.is_closing()) + self.assertEqual(tr._buffer, list_to_buffer()) + self.assertFalse(self.loop.readers) + self.assertFalse(self.loop.writers) + + # second close should not remove reader + tr._force_close(None) + self.assertFalse(self.loop.readers) + self.assertEqual(1, self.loop.remove_reader_count[7]) + + @mock.patch('asyncio.log.logger.error') + def test_fatal_error(self, m_exc): + exc = OSError() + tr = self.create_transport() + tr._force_close = mock.Mock() + tr._fatal_error(exc) + + m_exc.assert_not_called() + + tr._force_close.assert_called_with(exc) + + @mock.patch('asyncio.log.logger.error') + def test_fatal_error_custom_exception(self, m_exc): + class MyError(Exception): + pass + exc = MyError() + tr = self.create_transport() + tr._force_close = mock.Mock() + tr._fatal_error(exc) + + m_exc.assert_called_with( + test_utils.MockPattern( + 'Fatal error on transport\nprotocol:.*\ntransport:.*'), + exc_info=(MyError, MOCK_ANY, MOCK_ANY)) + + tr._force_close.assert_called_with(exc) + + def test_connection_lost(self): + exc = OSError() + tr = self.create_transport() + self.assertIsNotNone(tr._protocol) + self.assertIsNotNone(tr._loop) + tr._call_connection_lost(exc) + + self.protocol.connection_lost.assert_called_with(exc) + self.sock.close.assert_called_with() + self.assertIsNone(tr._sock) + + self.assertIsNone(tr._protocol) + self.assertIsNone(tr._loop) + + def test__add_reader(self): + tr = self.create_transport() + tr._buffer.extend(b'1') + tr._add_reader(7, mock.sentinel) + self.assertTrue(self.loop.readers) + + tr._force_close(None) + + self.assertTrue(tr.is_closing()) + self.assertFalse(self.loop.readers) + + # can not add readers after closing + tr._add_reader(7, mock.sentinel) + self.assertFalse(self.loop.readers) + + +class SelectorSocketTransportTests(test_utils.TestCase): + + def setUp(self): + super().setUp() + self.loop = self.new_test_loop() + self.protocol = test_utils.make_test_protocol(asyncio.Protocol) + self.sock = mock.Mock(socket.socket) + self.sock_fd = self.sock.fileno.return_value = 7 + + def socket_transport(self, waiter=None, sendmsg=False): + transport = _SelectorSocketTransport(self.loop, self.sock, + self.protocol, waiter=waiter) + if sendmsg: + transport._write_ready = transport._write_sendmsg + else: + transport._write_ready = transport._write_send + self.addCleanup(close_transport, transport) + return transport + + def test_ctor(self): + waiter = self.loop.create_future() + tr = self.socket_transport(waiter=waiter) + self.loop.run_until_complete(waiter) + + self.loop.assert_reader(7, tr._read_ready) + test_utils.run_briefly(self.loop) + self.protocol.connection_made.assert_called_with(tr) + + def test_ctor_with_waiter(self): + waiter = self.loop.create_future() + self.socket_transport(waiter=waiter) + self.loop.run_until_complete(waiter) + + self.assertIsNone(waiter.result()) + + def test_pause_resume_reading(self): + tr = self.socket_transport() + test_utils.run_briefly(self.loop) + self.assertFalse(tr._paused) + self.assertTrue(tr.is_reading()) + self.loop.assert_reader(7, tr._read_ready) + + tr.pause_reading() + tr.pause_reading() + self.assertTrue(tr._paused) + self.assertFalse(tr.is_reading()) + self.loop.assert_no_reader(7) + + tr.resume_reading() + tr.resume_reading() + self.assertFalse(tr._paused) + self.assertTrue(tr.is_reading()) + self.loop.assert_reader(7, tr._read_ready) + + tr.close() + self.assertFalse(tr.is_reading()) + self.loop.assert_no_reader(7) + + def test_pause_reading_connection_made(self): + tr = self.socket_transport() + self.protocol.connection_made.side_effect = lambda _: tr.pause_reading() + test_utils.run_briefly(self.loop) + self.assertFalse(tr.is_reading()) + self.loop.assert_no_reader(7) + + tr.resume_reading() + self.assertTrue(tr.is_reading()) + self.loop.assert_reader(7, tr._read_ready) + + tr.close() + self.assertFalse(tr.is_reading()) + self.loop.assert_no_reader(7) + + + def test_read_eof_received_error(self): + transport = self.socket_transport() + transport.close = mock.Mock() + transport._fatal_error = mock.Mock() + + self.loop.call_exception_handler = mock.Mock() + + self.protocol.eof_received.side_effect = LookupError() + + self.sock.recv.return_value = b'' + transport._read_ready() + + self.protocol.eof_received.assert_called_with() + self.assertTrue(transport._fatal_error.called) + + def test_data_received_error(self): + transport = self.socket_transport() + transport._fatal_error = mock.Mock() + + self.loop.call_exception_handler = mock.Mock() + self.protocol.data_received.side_effect = LookupError() + + self.sock.recv.return_value = b'data' + transport._read_ready() + + self.assertTrue(transport._fatal_error.called) + self.assertTrue(self.protocol.data_received.called) + + def test_read_ready(self): + transport = self.socket_transport() + + self.sock.recv.return_value = b'data' + transport._read_ready() + + self.protocol.data_received.assert_called_with(b'data') + + def test_read_ready_eof(self): + transport = self.socket_transport() + transport.close = mock.Mock() + + self.sock.recv.return_value = b'' + transport._read_ready() + + self.protocol.eof_received.assert_called_with() + transport.close.assert_called_with() + + def test_read_ready_eof_keep_open(self): + transport = self.socket_transport() + transport.close = mock.Mock() + + self.sock.recv.return_value = b'' + self.protocol.eof_received.return_value = True + transport._read_ready() + + self.protocol.eof_received.assert_called_with() + self.assertFalse(transport.close.called) + + @mock.patch('logging.exception') + def test_read_ready_tryagain(self, m_exc): + self.sock.recv.side_effect = BlockingIOError + + transport = self.socket_transport() + transport._fatal_error = mock.Mock() + transport._read_ready() + + self.assertFalse(transport._fatal_error.called) + + @mock.patch('logging.exception') + def test_read_ready_tryagain_interrupted(self, m_exc): + self.sock.recv.side_effect = InterruptedError + + transport = self.socket_transport() + transport._fatal_error = mock.Mock() + transport._read_ready() + + self.assertFalse(transport._fatal_error.called) + + @mock.patch('logging.exception') + def test_read_ready_conn_reset(self, m_exc): + err = self.sock.recv.side_effect = ConnectionResetError() + + transport = self.socket_transport() + transport._force_close = mock.Mock() + with test_utils.disable_logger(): + transport._read_ready() + transport._force_close.assert_called_with(err) + + @mock.patch('logging.exception') + def test_read_ready_err(self, m_exc): + err = self.sock.recv.side_effect = OSError() + + transport = self.socket_transport() + transport._fatal_error = mock.Mock() + transport._read_ready() + + transport._fatal_error.assert_called_with( + err, + 'Fatal read error on socket transport') + + def test_write(self): + data = b'data' + self.sock.send.return_value = len(data) + + transport = self.socket_transport() + transport.write(data) + self.sock.send.assert_called_with(data) + + def test_write_bytearray(self): + data = bytearray(b'data') + self.sock.send.return_value = len(data) + + transport = self.socket_transport() + transport.write(data) + self.sock.send.assert_called_with(data) + self.assertEqual(data, bytearray(b'data')) # Hasn't been mutated. + + def test_write_memoryview(self): + data = memoryview(b'data') + self.sock.send.return_value = len(data) + + transport = self.socket_transport() + transport.write(data) + self.sock.send.assert_called_with(data) + + def test_write_no_data(self): + transport = self.socket_transport() + transport._buffer.append(memoryview(b'data')) + transport.write(b'') + self.assertFalse(self.sock.send.called) + self.assertEqual(list_to_buffer([b'data']), transport._buffer) + + def test_write_buffer(self): + transport = self.socket_transport() + transport._buffer.append(b'data1') + transport.write(b'data2') + self.assertFalse(self.sock.send.called) + self.assertEqual(list_to_buffer([b'data1', b'data2']), + transport._buffer) + + def test_write_partial(self): + data = b'data' + self.sock.send.return_value = 2 + + transport = self.socket_transport() + transport.write(data) + + self.loop.assert_writer(7, transport._write_ready) + self.assertEqual(list_to_buffer([b'ta']), transport._buffer) + + def test_write_partial_bytearray(self): + data = bytearray(b'data') + self.sock.send.return_value = 2 + + transport = self.socket_transport() + transport.write(data) + + self.loop.assert_writer(7, transport._write_ready) + self.assertEqual(list_to_buffer([b'ta']), transport._buffer) + self.assertEqual(data, bytearray(b'data')) # Hasn't been mutated. + + def test_write_partial_memoryview(self): + data = memoryview(b'data') + self.sock.send.return_value = 2 + + transport = self.socket_transport() + transport.write(data) + + self.loop.assert_writer(7, transport._write_ready) + self.assertEqual(list_to_buffer([b'ta']), transport._buffer) + + def test_write_partial_none(self): + data = b'data' + self.sock.send.return_value = 0 + self.sock.fileno.return_value = 7 + + transport = self.socket_transport() + transport.write(data) + + self.loop.assert_writer(7, transport._write_ready) + self.assertEqual(list_to_buffer([b'data']), transport._buffer) + + def test_write_tryagain(self): + self.sock.send.side_effect = BlockingIOError + + data = b'data' + transport = self.socket_transport() + transport.write(data) + + self.loop.assert_writer(7, transport._write_ready) + self.assertEqual(list_to_buffer([b'data']), transport._buffer) + + def test_write_sendmsg_no_data(self): + self.sock.sendmsg = mock.Mock() + self.sock.sendmsg.return_value = 0 + transport = self.socket_transport(sendmsg=True) + transport._buffer.append(memoryview(b'data')) + transport.write(b'') + self.assertFalse(self.sock.sendmsg.called) + self.assertEqual(list_to_buffer([b'data']), transport._buffer) + + @unittest.skipUnless(selector_events._HAS_SENDMSG, 'no sendmsg') + def test_writelines_sendmsg_full(self): + data = memoryview(b'data') + self.sock.sendmsg = mock.Mock() + self.sock.sendmsg.return_value = len(data) + + transport = self.socket_transport(sendmsg=True) + transport.writelines([data]) + self.assertTrue(self.sock.sendmsg.called) + self.assertFalse(self.loop.writers) + + @unittest.skipUnless(selector_events._HAS_SENDMSG, 'no sendmsg') + def test_writelines_sendmsg_partial(self): + data = memoryview(b'data') + self.sock.sendmsg = mock.Mock() + self.sock.sendmsg.return_value = 2 + + transport = self.socket_transport(sendmsg=True) + transport.writelines([data]) + self.assertTrue(self.sock.sendmsg.called) + self.assertTrue(self.loop.writers) + + def test_writelines_send_full(self): + data = memoryview(b'data') + self.sock.send.return_value = len(data) + self.sock.send.fileno.return_value = 7 + + transport = self.socket_transport() + transport.writelines([data]) + self.assertTrue(self.sock.send.called) + self.assertFalse(self.loop.writers) + + def test_writelines_send_partial(self): + data = memoryview(b'data') + self.sock.send.return_value = 2 + self.sock.send.fileno.return_value = 7 + + transport = self.socket_transport() + transport.writelines([data]) + self.assertTrue(self.sock.send.called) + self.assertTrue(self.loop.writers) + + def test_writelines_pauses_protocol(self): + data = memoryview(b'data') + self.sock.send.return_value = 2 + self.sock.send.fileno.return_value = 7 + + transport = self.socket_transport() + transport._high_water = 1 + transport.writelines([data]) + self.assertTrue(self.protocol.pause_writing.called) + self.assertTrue(self.sock.send.called) + self.assertTrue(self.loop.writers) + + def test_writelines_after_connection_lost(self): + # GH-136234 + transport = self.socket_transport() + self.sock.send = mock.Mock() + self.sock.send.side_effect = ConnectionResetError + transport.write(b'data1') # Will fail immediately, causing connection lost + + transport.writelines([b'data2']) + self.assertFalse(transport._buffer) + self.assertFalse(self.loop.writers) + + test_utils.run_briefly(self.loop) # Allow _call_connection_lost to run + transport.writelines([b'data2']) + self.assertFalse(transport._buffer) + self.assertFalse(self.loop.writers) + + @unittest.skipUnless(selector_events._HAS_SENDMSG, 'no sendmsg') + def test_write_sendmsg_full(self): + data = memoryview(b'data') + self.sock.sendmsg = mock.Mock() + self.sock.sendmsg.return_value = len(data) + + transport = self.socket_transport(sendmsg=True) + transport._buffer.append(data) + self.loop._add_writer(7, transport._write_ready) + transport._write_ready() + self.assertTrue(self.sock.sendmsg.called) + self.assertFalse(self.loop.writers) + + @unittest.skipUnless(selector_events._HAS_SENDMSG, 'no sendmsg') + def test_write_sendmsg_partial(self): + + data = memoryview(b'data') + self.sock.sendmsg = mock.Mock() + # Sent partial data + self.sock.sendmsg.return_value = 2 + + transport = self.socket_transport(sendmsg=True) + transport._buffer.append(data) + self.loop._add_writer(7, transport._write_ready) + transport._write_ready() + self.assertTrue(self.sock.sendmsg.called) + self.assertTrue(self.loop.writers) + self.assertEqual(list_to_buffer([b'ta']), transport._buffer) + + @unittest.skipUnless(selector_events._HAS_SENDMSG, 'no sendmsg') + def test_write_sendmsg_half_buffer(self): + data = [memoryview(b'data1'), memoryview(b'data2')] + self.sock.sendmsg = mock.Mock() + # Sent partial data + self.sock.sendmsg.return_value = 2 + + transport = self.socket_transport(sendmsg=True) + transport._buffer.extend(data) + self.loop._add_writer(7, transport._write_ready) + transport._write_ready() + self.assertTrue(self.sock.sendmsg.called) + self.assertTrue(self.loop.writers) + self.assertEqual(list_to_buffer([b'ta1', b'data2']), transport._buffer) + + @unittest.skipUnless(selector_events._HAS_SENDMSG, 'no sendmsg') + def test_write_sendmsg_OSError(self): + data = memoryview(b'data') + self.sock.sendmsg = mock.Mock() + err = self.sock.sendmsg.side_effect = OSError() + + transport = self.socket_transport(sendmsg=True) + transport._fatal_error = mock.Mock() + transport._buffer.extend(data) + # Calls _fatal_error and clears the buffer + transport._write_ready() + self.assertTrue(self.sock.sendmsg.called) + self.assertFalse(self.loop.writers) + self.assertEqual(list_to_buffer([]), transport._buffer) + transport._fatal_error.assert_called_with( + err, + 'Fatal write error on socket transport') + + @mock.patch('asyncio.selector_events.logger') + def test_write_exception(self, m_log): + err = self.sock.send.side_effect = OSError() + + data = b'data' + transport = self.socket_transport() + transport._fatal_error = mock.Mock() + transport.write(data) + transport._fatal_error.assert_called_with( + err, + 'Fatal write error on socket transport') + transport._conn_lost = 1 + + self.sock.reset_mock() + transport.write(data) + self.assertFalse(self.sock.send.called) + self.assertEqual(transport._conn_lost, 2) + transport.write(data) + transport.write(data) + transport.write(data) + transport.write(data) + m_log.warning.assert_called_with('socket.send() raised exception.') + + def test_write_str(self): + transport = self.socket_transport() + self.assertRaises(TypeError, transport.write, 'str') + + def test_write_closing(self): + transport = self.socket_transport() + transport.close() + self.assertEqual(transport._conn_lost, 1) + transport.write(b'data') + self.assertEqual(transport._conn_lost, 2) + + def test_write_ready(self): + data = b'data' + self.sock.send.return_value = len(data) + + transport = self.socket_transport() + transport._buffer.append(data) + self.loop._add_writer(7, transport._write_ready) + transport._write_ready() + self.assertTrue(self.sock.send.called) + self.assertFalse(self.loop.writers) + + def test_write_ready_closing(self): + data = memoryview(b'data') + self.sock.send.return_value = len(data) + + transport = self.socket_transport() + transport._closing = True + transport._buffer.append(data) + self.loop._add_writer(7, transport._write_ready) + transport._write_ready() + self.assertTrue(self.sock.send.called) + self.assertFalse(self.loop.writers) + self.sock.close.assert_called_with() + self.protocol.connection_lost.assert_called_with(None) + + @unittest.skipIf(sys.flags.optimize, "Assertions are disabled in optimized mode") + def test_write_ready_no_data(self): + transport = self.socket_transport() + # This is an internal error. + self.assertRaises(AssertionError, transport._write_ready) + + def test_write_ready_partial(self): + data = memoryview(b'data') + self.sock.send.return_value = 2 + + transport = self.socket_transport() + transport._buffer.append(data) + self.loop._add_writer(7, transport._write_ready) + transport._write_ready() + self.loop.assert_writer(7, transport._write_ready) + self.assertEqual(list_to_buffer([b'ta']), transport._buffer) + + def test_write_ready_partial_none(self): + data = b'data' + self.sock.send.return_value = 0 + + transport = self.socket_transport() + transport._buffer.append(data) + self.loop._add_writer(7, transport._write_ready) + transport._write_ready() + self.loop.assert_writer(7, transport._write_ready) + self.assertEqual(list_to_buffer([b'data']), transport._buffer) + + def test_write_ready_tryagain(self): + self.sock.send.side_effect = BlockingIOError + + transport = self.socket_transport() + buffer = list_to_buffer([b'data1', b'data2']) + transport._buffer = buffer + self.loop._add_writer(7, transport._write_ready) + transport._write_ready() + + self.loop.assert_writer(7, transport._write_ready) + self.assertEqual(buffer, transport._buffer) + + def test_write_ready_exception(self): + err = self.sock.send.side_effect = OSError() + + transport = self.socket_transport() + transport._fatal_error = mock.Mock() + transport._buffer.extend(b'data') + transport._write_ready() + transport._fatal_error.assert_called_with( + err, + 'Fatal write error on socket transport') + + def test_write_eof(self): + tr = self.socket_transport() + self.assertTrue(tr.can_write_eof()) + tr.write_eof() + self.sock.shutdown.assert_called_with(socket.SHUT_WR) + tr.write_eof() + self.assertEqual(self.sock.shutdown.call_count, 1) + tr.close() + + def test_write_eof_buffer(self): + tr = self.socket_transport() + self.sock.send.side_effect = BlockingIOError + tr.write(b'data') + tr.write_eof() + self.assertEqual(tr._buffer, list_to_buffer([b'data'])) + self.assertTrue(tr._eof) + self.assertFalse(self.sock.shutdown.called) + self.sock.send.side_effect = lambda _: 4 + tr._write_ready() + self.assertTrue(self.sock.send.called) + self.sock.shutdown.assert_called_with(socket.SHUT_WR) + tr.close() + + def test_write_eof_after_close(self): + tr = self.socket_transport() + tr.close() + self.loop.run_until_complete(asyncio.sleep(0)) + tr.write_eof() + + @mock.patch('asyncio.base_events.logger') + def test_transport_close_remove_writer(self, m_log): + remove_writer = self.loop._remove_writer = mock.Mock() + + transport = self.socket_transport() + transport.close() + remove_writer.assert_called_with(self.sock_fd) + + def test_write_buffer_after_close(self): + # gh-115514: If the transport is closed while: + # * Transport write buffer is not empty + # * Transport is paused + # * Protocol has data in its buffer, like SSLProtocol in self._outgoing + # The data is still written out. + + # Also tested with real SSL transport in + # test.test_asyncio.test_ssl.TestSSL.test_remote_shutdown_receives_trailing_data + + data = memoryview(b'data') + self.sock.send.return_value = 2 + self.sock.send.fileno.return_value = 7 + + def _resume_writing(): + transport.write(b"data") + self.protocol.resume_writing.side_effect = None + + self.protocol.resume_writing.side_effect = _resume_writing + + transport = self.socket_transport() + transport._high_water = 1 + + transport.write(data) + + self.assertTrue(transport._protocol_paused) + self.assertTrue(self.sock.send.called) + self.loop.assert_writer(7, transport._write_ready) + + transport.close() + + # not called, we still have data in write buffer + self.assertFalse(self.protocol.connection_lost.called) + + self.loop.writers[7]._run() + # during this ^ run, the _resume_writing mock above was called and added more data + + self.assertEqual(transport.get_write_buffer_size(), 2) + self.loop.writers[7]._run() + + self.assertEqual(transport.get_write_buffer_size(), 0) + self.assertTrue(self.protocol.connection_lost.called) + +class SelectorSocketTransportBufferedProtocolTests(test_utils.TestCase): + + def setUp(self): + super().setUp() + self.loop = self.new_test_loop() + + self.protocol = test_utils.make_test_protocol(asyncio.BufferedProtocol) + self.buf = bytearray(1) + self.protocol.get_buffer.side_effect = lambda hint: self.buf + + self.sock = mock.Mock(socket.socket) + self.sock_fd = self.sock.fileno.return_value = 7 + + def socket_transport(self, waiter=None): + transport = _SelectorSocketTransport(self.loop, self.sock, + self.protocol, waiter=waiter) + self.addCleanup(close_transport, transport) + return transport + + def test_ctor(self): + waiter = self.loop.create_future() + tr = self.socket_transport(waiter=waiter) + self.loop.run_until_complete(waiter) + + self.loop.assert_reader(7, tr._read_ready) + test_utils.run_briefly(self.loop) + self.protocol.connection_made.assert_called_with(tr) + + def test_get_buffer_error(self): + transport = self.socket_transport() + transport._fatal_error = mock.Mock() + + self.loop.call_exception_handler = mock.Mock() + self.protocol.get_buffer.side_effect = LookupError() + + transport._read_ready() + + self.assertTrue(transport._fatal_error.called) + self.assertTrue(self.protocol.get_buffer.called) + self.assertFalse(self.protocol.buffer_updated.called) + + def test_get_buffer_zerosized(self): + transport = self.socket_transport() + transport._fatal_error = mock.Mock() + + self.loop.call_exception_handler = mock.Mock() + self.protocol.get_buffer.side_effect = lambda hint: bytearray(0) + + transport._read_ready() + + self.assertTrue(transport._fatal_error.called) + self.assertTrue(self.protocol.get_buffer.called) + self.assertFalse(self.protocol.buffer_updated.called) + + def test_proto_type_switch(self): + self.protocol = test_utils.make_test_protocol(asyncio.Protocol) + transport = self.socket_transport() + + self.sock.recv.return_value = b'data' + transport._read_ready() + + self.protocol.data_received.assert_called_with(b'data') + + # switch protocol to a BufferedProtocol + + buf_proto = test_utils.make_test_protocol(asyncio.BufferedProtocol) + buf = bytearray(4) + buf_proto.get_buffer.side_effect = lambda hint: buf + + transport.set_protocol(buf_proto) + + self.sock.recv_into.return_value = 10 + transport._read_ready() + + buf_proto.get_buffer.assert_called_with(-1) + buf_proto.buffer_updated.assert_called_with(10) + + def test_buffer_updated_error(self): + transport = self.socket_transport() + transport._fatal_error = mock.Mock() + + self.loop.call_exception_handler = mock.Mock() + self.protocol.buffer_updated.side_effect = LookupError() + + self.sock.recv_into.return_value = 10 + transport._read_ready() + + self.assertTrue(transport._fatal_error.called) + self.assertTrue(self.protocol.get_buffer.called) + self.assertTrue(self.protocol.buffer_updated.called) + + def test_read_eof_received_error(self): + transport = self.socket_transport() + transport.close = mock.Mock() + transport._fatal_error = mock.Mock() + + self.loop.call_exception_handler = mock.Mock() + + self.protocol.eof_received.side_effect = LookupError() + + self.sock.recv_into.return_value = 0 + transport._read_ready() + + self.protocol.eof_received.assert_called_with() + self.assertTrue(transport._fatal_error.called) + + def test_read_ready(self): + transport = self.socket_transport() + + self.sock.recv_into.return_value = 10 + transport._read_ready() + + self.protocol.get_buffer.assert_called_with(-1) + self.protocol.buffer_updated.assert_called_with(10) + + def test_read_ready_eof(self): + transport = self.socket_transport() + transport.close = mock.Mock() + + self.sock.recv_into.return_value = 0 + transport._read_ready() + + self.protocol.eof_received.assert_called_with() + transport.close.assert_called_with() + + def test_read_ready_eof_keep_open(self): + transport = self.socket_transport() + transport.close = mock.Mock() + + self.sock.recv_into.return_value = 0 + self.protocol.eof_received.return_value = True + transport._read_ready() + + self.protocol.eof_received.assert_called_with() + self.assertFalse(transport.close.called) + + @mock.patch('logging.exception') + def test_read_ready_tryagain(self, m_exc): + self.sock.recv_into.side_effect = BlockingIOError + + transport = self.socket_transport() + transport._fatal_error = mock.Mock() + transport._read_ready() + + self.assertFalse(transport._fatal_error.called) + + @mock.patch('logging.exception') + def test_read_ready_tryagain_interrupted(self, m_exc): + self.sock.recv_into.side_effect = InterruptedError + + transport = self.socket_transport() + transport._fatal_error = mock.Mock() + transport._read_ready() + + self.assertFalse(transport._fatal_error.called) + + @mock.patch('logging.exception') + def test_read_ready_conn_reset(self, m_exc): + err = self.sock.recv_into.side_effect = ConnectionResetError() + + transport = self.socket_transport() + transport._force_close = mock.Mock() + with test_utils.disable_logger(): + transport._read_ready() + transport._force_close.assert_called_with(err) + + @mock.patch('logging.exception') + def test_read_ready_err(self, m_exc): + err = self.sock.recv_into.side_effect = OSError() + + transport = self.socket_transport() + transport._fatal_error = mock.Mock() + transport._read_ready() + + transport._fatal_error.assert_called_with( + err, + 'Fatal read error on socket transport') + + +class SelectorDatagramTransportTests(test_utils.TestCase): + + def setUp(self): + super().setUp() + self.loop = self.new_test_loop() + self.protocol = test_utils.make_test_protocol(asyncio.DatagramProtocol) + self.sock = mock.Mock(spec_set=socket.socket) + self.sock.fileno.return_value = 7 + + def datagram_transport(self, address=None): + self.sock.getpeername.side_effect = None if address else OSError + transport = _SelectorDatagramTransport(self.loop, self.sock, + self.protocol, + address=address) + self.addCleanup(close_transport, transport) + return transport + + def test_read_ready(self): + transport = self.datagram_transport() + + self.sock.recvfrom.return_value = (b'data', ('0.0.0.0', 1234)) + transport._read_ready() + + self.protocol.datagram_received.assert_called_with( + b'data', ('0.0.0.0', 1234)) + + def test_transport_inheritance(self): + transport = self.datagram_transport() + self.assertIsInstance(transport, asyncio.DatagramTransport) + + def test_read_ready_tryagain(self): + transport = self.datagram_transport() + + self.sock.recvfrom.side_effect = BlockingIOError + transport._fatal_error = mock.Mock() + transport._read_ready() + + self.assertFalse(transport._fatal_error.called) + + def test_read_ready_err(self): + transport = self.datagram_transport() + + err = self.sock.recvfrom.side_effect = RuntimeError() + transport._fatal_error = mock.Mock() + transport._read_ready() + + transport._fatal_error.assert_called_with( + err, + 'Fatal read error on datagram transport') + + def test_read_ready_oserr(self): + transport = self.datagram_transport() + + err = self.sock.recvfrom.side_effect = OSError() + transport._fatal_error = mock.Mock() + transport._read_ready() + + self.assertFalse(transport._fatal_error.called) + self.protocol.error_received.assert_called_with(err) + + def test_sendto(self): + data = b'data' + transport = self.datagram_transport() + transport.sendto(data, ('0.0.0.0', 1234)) + self.assertTrue(self.sock.sendto.called) + self.assertEqual( + self.sock.sendto.call_args[0], (data, ('0.0.0.0', 1234))) + + def test_sendto_bytearray(self): + data = bytearray(b'data') + transport = self.datagram_transport() + transport.sendto(data, ('0.0.0.0', 1234)) + self.assertTrue(self.sock.sendto.called) + self.assertEqual( + self.sock.sendto.call_args[0], (data, ('0.0.0.0', 1234))) + + def test_sendto_memoryview(self): + data = memoryview(b'data') + transport = self.datagram_transport() + transport.sendto(data, ('0.0.0.0', 1234)) + self.assertTrue(self.sock.sendto.called) + self.assertEqual( + self.sock.sendto.call_args[0], (data, ('0.0.0.0', 1234))) + + def test_sendto_no_data(self): + transport = self.datagram_transport() + transport.sendto(b'', ('0.0.0.0', 1234)) + self.assertTrue(self.sock.sendto.called) + self.assertEqual( + self.sock.sendto.call_args[0], (b'', ('0.0.0.0', 1234))) + + def test_sendto_buffer(self): + transport = self.datagram_transport() + transport._buffer.append((b'data1', ('0.0.0.0', 12345))) + transport.sendto(b'data2', ('0.0.0.0', 12345)) + self.assertFalse(self.sock.sendto.called) + self.assertEqual( + [(b'data1', ('0.0.0.0', 12345)), + (b'data2', ('0.0.0.0', 12345))], + list(transport._buffer)) + + def test_sendto_buffer_bytearray(self): + data2 = bytearray(b'data2') + transport = self.datagram_transport() + transport._buffer.append((b'data1', ('0.0.0.0', 12345))) + transport.sendto(data2, ('0.0.0.0', 12345)) + self.assertFalse(self.sock.sendto.called) + self.assertEqual( + [(b'data1', ('0.0.0.0', 12345)), + (b'data2', ('0.0.0.0', 12345))], + list(transport._buffer)) + self.assertIsInstance(transport._buffer[1][0], bytes) + + def test_sendto_buffer_memoryview(self): + data2 = memoryview(b'data2') + transport = self.datagram_transport() + transport._buffer.append((b'data1', ('0.0.0.0', 12345))) + transport.sendto(data2, ('0.0.0.0', 12345)) + self.assertFalse(self.sock.sendto.called) + self.assertEqual( + [(b'data1', ('0.0.0.0', 12345)), + (b'data2', ('0.0.0.0', 12345))], + list(transport._buffer)) + self.assertIsInstance(transport._buffer[1][0], bytes) + + def test_sendto_buffer_nodata(self): + data2 = b'' + transport = self.datagram_transport() + transport._buffer.append((b'data1', ('0.0.0.0', 12345))) + transport.sendto(data2, ('0.0.0.0', 12345)) + self.assertFalse(self.sock.sendto.called) + self.assertEqual( + [(b'data1', ('0.0.0.0', 12345)), + (b'', ('0.0.0.0', 12345))], + list(transport._buffer)) + self.assertIsInstance(transport._buffer[1][0], bytes) + + def test_sendto_tryagain(self): + data = b'data' + + self.sock.sendto.side_effect = BlockingIOError + + transport = self.datagram_transport() + transport.sendto(data, ('0.0.0.0', 12345)) + + self.loop.assert_writer(7, transport._sendto_ready) + self.assertEqual( + [(b'data', ('0.0.0.0', 12345))], list(transport._buffer)) + + @mock.patch('asyncio.selector_events.logger') + def test_sendto_exception(self, m_log): + data = b'data' + err = self.sock.sendto.side_effect = RuntimeError() + + transport = self.datagram_transport() + transport._fatal_error = mock.Mock() + transport.sendto(data, ()) + + self.assertTrue(transport._fatal_error.called) + transport._fatal_error.assert_called_with( + err, + 'Fatal write error on datagram transport') + transport._conn_lost = 1 + + transport._address = ('123',) + transport.sendto(data) + transport.sendto(data) + transport.sendto(data) + transport.sendto(data) + transport.sendto(data) + m_log.warning.assert_called_with('socket.send() raised exception.') + + def test_sendto_error_received(self): + data = b'data' + + self.sock.sendto.side_effect = ConnectionRefusedError + + transport = self.datagram_transport() + transport._fatal_error = mock.Mock() + transport.sendto(data, ()) + + self.assertEqual(transport._conn_lost, 0) + self.assertFalse(transport._fatal_error.called) + + def test_sendto_error_received_connected(self): + data = b'data' + + self.sock.send.side_effect = ConnectionRefusedError + + transport = self.datagram_transport(address=('0.0.0.0', 1)) + transport._fatal_error = mock.Mock() + transport.sendto(data) + + self.assertFalse(transport._fatal_error.called) + self.assertTrue(self.protocol.error_received.called) + + def test_sendto_str(self): + transport = self.datagram_transport() + self.assertRaises(TypeError, transport.sendto, 'str', ()) + + def test_sendto_connected_addr(self): + transport = self.datagram_transport(address=('0.0.0.0', 1)) + self.assertRaises( + ValueError, transport.sendto, b'str', ('0.0.0.0', 2)) + + def test_sendto_closing(self): + transport = self.datagram_transport(address=(1,)) + transport.close() + self.assertEqual(transport._conn_lost, 1) + transport.sendto(b'data', (1,)) + self.assertEqual(transport._conn_lost, 2) + + def test_sendto_sendto_ready(self): + data = b'data' + + # First queue up the buffer by having the socket blocked + self.sock.sendto.side_effect = BlockingIOError + transport = self.datagram_transport() + transport.sendto(data, ('0.0.0.0', 12345)) + self.loop.assert_writer(7, transport._sendto_ready) + self.assertEqual(1, len(transport._buffer)) + self.assertEqual(transport._buffer_size, len(data) + transport._header_size) + + # Now let the socket send the buffer + self.sock.sendto.side_effect = None + transport._sendto_ready() + self.assertTrue(self.sock.sendto.called) + self.assertEqual( + self.sock.sendto.call_args[0], (data, ('0.0.0.0', 12345))) + self.assertFalse(self.loop.writers) + self.assertFalse(transport._buffer) + self.assertEqual(transport._buffer_size, 0) + + def test_sendto_sendto_ready_blocked(self): + data = b'data' + + # First queue up the buffer by having the socket blocked + self.sock.sendto.side_effect = BlockingIOError + transport = self.datagram_transport() + transport.sendto(data, ('0.0.0.0', 12345)) + self.loop.assert_writer(7, transport._sendto_ready) + self.assertEqual(1, len(transport._buffer)) + self.assertEqual(transport._buffer_size, len(data) + transport._header_size) + + # Now try to send the buffer, it will be added to buffer again if it fails + transport._sendto_ready() + self.assertTrue(self.sock.sendto.called) + self.assertEqual( + self.sock.sendto.call_args[0], (data, ('0.0.0.0', 12345))) + self.assertTrue(self.loop.writers) + self.assertEqual(1, len(transport._buffer)) + self.assertEqual(transport._buffer_size, len(data) + transport._header_size) + + def test_sendto_ready(self): + data = b'data' + self.sock.sendto.return_value = len(data) + + transport = self.datagram_transport() + transport._buffer.append((data, ('0.0.0.0', 12345))) + self.loop._add_writer(7, transport._sendto_ready) + transport._sendto_ready() + self.assertTrue(self.sock.sendto.called) + self.assertEqual( + self.sock.sendto.call_args[0], (data, ('0.0.0.0', 12345))) + self.assertFalse(self.loop.writers) + + def test_sendto_ready_closing(self): + data = b'data' + self.sock.send.return_value = len(data) + + transport = self.datagram_transport() + transport._closing = True + transport._buffer.append((data, ())) + self.loop._add_writer(7, transport._sendto_ready) + transport._sendto_ready() + self.sock.sendto.assert_called_with(data, ()) + self.assertFalse(self.loop.writers) + self.sock.close.assert_called_with() + self.protocol.connection_lost.assert_called_with(None) + + def test_sendto_ready_no_data(self): + transport = self.datagram_transport() + self.loop._add_writer(7, transport._sendto_ready) + transport._sendto_ready() + self.assertFalse(self.sock.sendto.called) + self.assertFalse(self.loop.writers) + + def test_sendto_ready_tryagain(self): + self.sock.sendto.side_effect = BlockingIOError + + transport = self.datagram_transport() + transport._buffer.extend([(b'data1', ()), (b'data2', ())]) + self.loop._add_writer(7, transport._sendto_ready) + transport._sendto_ready() + + self.loop.assert_writer(7, transport._sendto_ready) + self.assertEqual( + [(b'data1', ()), (b'data2', ())], + list(transport._buffer)) + + def test_sendto_ready_exception(self): + err = self.sock.sendto.side_effect = RuntimeError() + + transport = self.datagram_transport() + transport._fatal_error = mock.Mock() + transport._buffer.append((b'data', ())) + transport._sendto_ready() + + transport._fatal_error.assert_called_with( + err, + 'Fatal write error on datagram transport') + + def test_sendto_ready_error_received(self): + self.sock.sendto.side_effect = ConnectionRefusedError + + transport = self.datagram_transport() + transport._fatal_error = mock.Mock() + transport._buffer.append((b'data', ())) + transport._sendto_ready() + + self.assertFalse(transport._fatal_error.called) + + def test_sendto_ready_error_received_connection(self): + self.sock.send.side_effect = ConnectionRefusedError + + transport = self.datagram_transport(address=('0.0.0.0', 1)) + transport._fatal_error = mock.Mock() + transport._buffer.append((b'data', ())) + transport._sendto_ready() + + self.assertFalse(transport._fatal_error.called) + self.assertTrue(self.protocol.error_received.called) + + @mock.patch('asyncio.base_events.logger.error') + def test_fatal_error_connected(self, m_exc): + transport = self.datagram_transport(address=('0.0.0.0', 1)) + err = ConnectionRefusedError() + transport._fatal_error(err) + self.assertFalse(self.protocol.error_received.called) + m_exc.assert_not_called() + + @mock.patch('asyncio.base_events.logger.error') + def test_fatal_error_connected_custom_error(self, m_exc): + class MyException(Exception): + pass + transport = self.datagram_transport(address=('0.0.0.0', 1)) + err = MyException() + transport._fatal_error(err) + self.assertFalse(self.protocol.error_received.called) + m_exc.assert_called_with( + test_utils.MockPattern( + 'Fatal error on transport\nprotocol:.*\ntransport:.*'), + exc_info=(MyException, MOCK_ANY, MOCK_ANY)) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_sendfile.py b/Lib/test/test_asyncio/test_sendfile.py new file mode 100644 index 00000000000..dcd963b3355 --- /dev/null +++ b/Lib/test/test_asyncio/test_sendfile.py @@ -0,0 +1,585 @@ +"""Tests for sendfile functionality.""" + +import asyncio +import errno +import os +import socket +import sys +import tempfile +import unittest +from asyncio import base_events +from asyncio import constants +from unittest import mock +from test import support +from test.support import os_helper +from test.support import socket_helper +from test.test_asyncio import utils as test_utils + +try: + import ssl +except ImportError: + ssl = None + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +class MySendfileProto(asyncio.Protocol): + + def __init__(self, loop=None, close_after=0): + self.transport = None + self.state = 'INITIAL' + self.nbytes = 0 + if loop is not None: + self.connected = loop.create_future() + self.done = loop.create_future() + self.data = bytearray() + self.close_after = close_after + + def _assert_state(self, *expected): + if self.state not in expected: + raise AssertionError(f'state: {self.state!r}, expected: {expected!r}') + + def connection_made(self, transport): + self.transport = transport + self._assert_state('INITIAL') + self.state = 'CONNECTED' + if self.connected: + self.connected.set_result(None) + + def eof_received(self): + self._assert_state('CONNECTED') + self.state = 'EOF' + + def connection_lost(self, exc): + self._assert_state('CONNECTED', 'EOF') + self.state = 'CLOSED' + if self.done: + self.done.set_result(None) + + def data_received(self, data): + self._assert_state('CONNECTED') + self.nbytes += len(data) + self.data.extend(data) + super().data_received(data) + if self.close_after and self.nbytes >= self.close_after: + self.transport.close() + + +class MyProto(asyncio.Protocol): + + def __init__(self, loop): + self.started = False + self.closed = False + self.data = bytearray() + self.fut = loop.create_future() + self.transport = None + + def connection_made(self, transport): + self.started = True + self.transport = transport + + def data_received(self, data): + self.data.extend(data) + + def connection_lost(self, exc): + self.closed = True + self.fut.set_result(None) + + async def wait_closed(self): + await self.fut + + +class SendfileBase: + + # Linux >= 6.10 seems buffering up to 17 pages of data. + # So DATA should be large enough to make this test reliable even with a + # 64 KiB page configuration. + DATA = b"x" * (1024 * 17 * 64 + 1) + # Reduce socket buffer size to test on relative small data sets. + BUF_SIZE = 4 * 1024 # 4 KiB + + def create_event_loop(self): + raise NotImplementedError + + @classmethod + def setUpClass(cls): + with open(os_helper.TESTFN, 'wb') as fp: + fp.write(cls.DATA) + super().setUpClass() + + @classmethod + def tearDownClass(cls): + os_helper.unlink(os_helper.TESTFN) + super().tearDownClass() + + def setUp(self): + self.file = open(os_helper.TESTFN, 'rb') + self.addCleanup(self.file.close) + self.loop = self.create_event_loop() + self.set_event_loop(self.loop) + super().setUp() + + def tearDown(self): + # just in case if we have transport close callbacks + if not self.loop.is_closed(): + test_utils.run_briefly(self.loop) + + self.doCleanups() + support.gc_collect() + super().tearDown() + + def run_loop(self, coro): + return self.loop.run_until_complete(coro) + + +class SockSendfileMixin(SendfileBase): + + @classmethod + def setUpClass(cls): + cls.__old_bufsize = constants.SENDFILE_FALLBACK_READBUFFER_SIZE + constants.SENDFILE_FALLBACK_READBUFFER_SIZE = 1024 * 16 + super().setUpClass() + + @classmethod + def tearDownClass(cls): + constants.SENDFILE_FALLBACK_READBUFFER_SIZE = cls.__old_bufsize + super().tearDownClass() + + def make_socket(self, cleanup=True): + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + sock.setblocking(False) + if cleanup: + self.addCleanup(sock.close) + return sock + + def reduce_receive_buffer_size(self, sock): + # Reduce receive socket buffer size to test on relative + # small data sets. + sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, self.BUF_SIZE) + + def reduce_send_buffer_size(self, sock, transport=None): + # Reduce send socket buffer size to test on relative small data sets. + + # On macOS, SO_SNDBUF is reset by connect(). So this method + # should be called after the socket is connected. + sock.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, self.BUF_SIZE) + + if transport is not None: + transport.set_write_buffer_limits(high=self.BUF_SIZE) + + def prepare_socksendfile(self): + proto = MyProto(self.loop) + port = socket_helper.find_unused_port() + srv_sock = self.make_socket(cleanup=False) + srv_sock.bind((socket_helper.HOST, port)) + server = self.run_loop(self.loop.create_server( + lambda: proto, sock=srv_sock)) + self.reduce_receive_buffer_size(srv_sock) + + sock = self.make_socket() + self.run_loop(self.loop.sock_connect(sock, ('127.0.0.1', port))) + self.reduce_send_buffer_size(sock) + + def cleanup(): + if proto.transport is not None: + # can be None if the task was cancelled before + # connection_made callback + proto.transport.close() + self.run_loop(proto.wait_closed()) + + server.close() + self.run_loop(server.wait_closed()) + + self.addCleanup(cleanup) + + return sock, proto + + def test_sock_sendfile_success(self): + sock, proto = self.prepare_socksendfile() + ret = self.run_loop(self.loop.sock_sendfile(sock, self.file)) + sock.close() + self.run_loop(proto.wait_closed()) + + self.assertEqual(ret, len(self.DATA)) + self.assertEqual(proto.data, self.DATA) + self.assertEqual(self.file.tell(), len(self.DATA)) + + def test_sock_sendfile_with_offset_and_count(self): + sock, proto = self.prepare_socksendfile() + ret = self.run_loop(self.loop.sock_sendfile(sock, self.file, + 1000, 2000)) + sock.close() + self.run_loop(proto.wait_closed()) + + self.assertEqual(proto.data, self.DATA[1000:3000]) + self.assertEqual(self.file.tell(), 3000) + self.assertEqual(ret, 2000) + + def test_sock_sendfile_zero_size(self): + sock, proto = self.prepare_socksendfile() + with tempfile.TemporaryFile() as f: + ret = self.run_loop(self.loop.sock_sendfile(sock, f, + 0, None)) + sock.close() + self.run_loop(proto.wait_closed()) + + self.assertEqual(ret, 0) + self.assertEqual(self.file.tell(), 0) + + def test_sock_sendfile_mix_with_regular_send(self): + buf = b"mix_regular_send" * (4 * 1024) # 64 KiB + sock, proto = self.prepare_socksendfile() + self.run_loop(self.loop.sock_sendall(sock, buf)) + ret = self.run_loop(self.loop.sock_sendfile(sock, self.file)) + self.run_loop(self.loop.sock_sendall(sock, buf)) + sock.close() + self.run_loop(proto.wait_closed()) + + self.assertEqual(ret, len(self.DATA)) + expected = buf + self.DATA + buf + self.assertEqual(proto.data, expected) + self.assertEqual(self.file.tell(), len(self.DATA)) + + +class SendfileMixin(SendfileBase): + + # Note: sendfile via SSL transport is equal to sendfile fallback + + def prepare_sendfile(self, *, is_ssl=False, close_after=0): + port = socket_helper.find_unused_port() + srv_proto = MySendfileProto(loop=self.loop, + close_after=close_after) + if is_ssl: + if not ssl: + self.skipTest("No ssl module") + srv_ctx = test_utils.simple_server_sslcontext() + cli_ctx = test_utils.simple_client_sslcontext() + else: + srv_ctx = None + cli_ctx = None + srv_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + srv_sock.bind((socket_helper.HOST, port)) + server = self.run_loop(self.loop.create_server( + lambda: srv_proto, sock=srv_sock, ssl=srv_ctx)) + self.reduce_receive_buffer_size(srv_sock) + + if is_ssl: + server_hostname = socket_helper.HOST + else: + server_hostname = None + cli_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + cli_sock.connect((socket_helper.HOST, port)) + + cli_proto = MySendfileProto(loop=self.loop) + tr, pr = self.run_loop(self.loop.create_connection( + lambda: cli_proto, sock=cli_sock, + ssl=cli_ctx, server_hostname=server_hostname)) + self.reduce_send_buffer_size(cli_sock, transport=tr) + + def cleanup(): + srv_proto.transport.close() + cli_proto.transport.close() + self.run_loop(srv_proto.done) + self.run_loop(cli_proto.done) + + server.close() + self.run_loop(server.wait_closed()) + + self.addCleanup(cleanup) + return srv_proto, cli_proto + + @unittest.skipIf(sys.platform == 'win32', "UDP sockets are not supported") + def test_sendfile_not_supported(self): + tr, pr = self.run_loop( + self.loop.create_datagram_endpoint( + asyncio.DatagramProtocol, + family=socket.AF_INET)) + try: + with self.assertRaisesRegex(RuntimeError, "not supported"): + self.run_loop( + self.loop.sendfile(tr, self.file)) + self.assertEqual(0, self.file.tell()) + finally: + # don't use self.addCleanup because it produces resource warning + tr.close() + + def test_sendfile(self): + srv_proto, cli_proto = self.prepare_sendfile() + ret = self.run_loop( + self.loop.sendfile(cli_proto.transport, self.file)) + cli_proto.transport.close() + self.run_loop(srv_proto.done) + self.assertEqual(ret, len(self.DATA)) + self.assertEqual(srv_proto.nbytes, len(self.DATA)) + self.assertEqual(srv_proto.data, self.DATA) + self.assertEqual(self.file.tell(), len(self.DATA)) + + def test_sendfile_force_fallback(self): + srv_proto, cli_proto = self.prepare_sendfile() + + def sendfile_native(transp, file, offset, count): + # to raise SendfileNotAvailableError + return base_events.BaseEventLoop._sendfile_native( + self.loop, transp, file, offset, count) + + self.loop._sendfile_native = sendfile_native + + ret = self.run_loop( + self.loop.sendfile(cli_proto.transport, self.file)) + cli_proto.transport.close() + self.run_loop(srv_proto.done) + self.assertEqual(ret, len(self.DATA)) + self.assertEqual(srv_proto.nbytes, len(self.DATA)) + self.assertEqual(srv_proto.data, self.DATA) + self.assertEqual(self.file.tell(), len(self.DATA)) + + def test_sendfile_force_unsupported_native(self): + if sys.platform == 'win32': + if isinstance(self.loop, asyncio.ProactorEventLoop): + self.skipTest("Fails on proactor event loop") + srv_proto, cli_proto = self.prepare_sendfile() + + def sendfile_native(transp, file, offset, count): + # to raise SendfileNotAvailableError + return base_events.BaseEventLoop._sendfile_native( + self.loop, transp, file, offset, count) + + self.loop._sendfile_native = sendfile_native + + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, + "not supported"): + self.run_loop( + self.loop.sendfile(cli_proto.transport, self.file, + fallback=False)) + + cli_proto.transport.close() + self.run_loop(srv_proto.done) + self.assertEqual(srv_proto.nbytes, 0) + self.assertEqual(self.file.tell(), 0) + + def test_sendfile_ssl(self): + srv_proto, cli_proto = self.prepare_sendfile(is_ssl=True) + ret = self.run_loop( + self.loop.sendfile(cli_proto.transport, self.file)) + cli_proto.transport.close() + self.run_loop(srv_proto.done) + self.assertEqual(ret, len(self.DATA)) + self.assertEqual(srv_proto.nbytes, len(self.DATA)) + self.assertEqual(srv_proto.data, self.DATA) + self.assertEqual(self.file.tell(), len(self.DATA)) + + def test_sendfile_for_closing_transp(self): + srv_proto, cli_proto = self.prepare_sendfile() + cli_proto.transport.close() + with self.assertRaisesRegex(RuntimeError, "is closing"): + self.run_loop(self.loop.sendfile(cli_proto.transport, self.file)) + self.run_loop(srv_proto.done) + self.assertEqual(srv_proto.nbytes, 0) + self.assertEqual(self.file.tell(), 0) + + def test_sendfile_pre_and_post_data(self): + srv_proto, cli_proto = self.prepare_sendfile() + PREFIX = b'PREFIX__' * 1024 # 8 KiB + SUFFIX = b'--SUFFIX' * 1024 # 8 KiB + cli_proto.transport.write(PREFIX) + ret = self.run_loop( + self.loop.sendfile(cli_proto.transport, self.file)) + cli_proto.transport.write(SUFFIX) + cli_proto.transport.close() + self.run_loop(srv_proto.done) + self.assertEqual(ret, len(self.DATA)) + self.assertEqual(srv_proto.data, PREFIX + self.DATA + SUFFIX) + self.assertEqual(self.file.tell(), len(self.DATA)) + + def test_sendfile_ssl_pre_and_post_data(self): + srv_proto, cli_proto = self.prepare_sendfile(is_ssl=True) + PREFIX = b'zxcvbnm' * 1024 + SUFFIX = b'0987654321' * 1024 + cli_proto.transport.write(PREFIX) + ret = self.run_loop( + self.loop.sendfile(cli_proto.transport, self.file)) + cli_proto.transport.write(SUFFIX) + cli_proto.transport.close() + self.run_loop(srv_proto.done) + self.assertEqual(ret, len(self.DATA)) + self.assertEqual(srv_proto.data, PREFIX + self.DATA + SUFFIX) + self.assertEqual(self.file.tell(), len(self.DATA)) + + def test_sendfile_partial(self): + srv_proto, cli_proto = self.prepare_sendfile() + ret = self.run_loop( + self.loop.sendfile(cli_proto.transport, self.file, 1000, 100)) + cli_proto.transport.close() + self.run_loop(srv_proto.done) + self.assertEqual(ret, 100) + self.assertEqual(srv_proto.nbytes, 100) + self.assertEqual(srv_proto.data, self.DATA[1000:1100]) + self.assertEqual(self.file.tell(), 1100) + + def test_sendfile_ssl_partial(self): + srv_proto, cli_proto = self.prepare_sendfile(is_ssl=True) + ret = self.run_loop( + self.loop.sendfile(cli_proto.transport, self.file, 1000, 100)) + cli_proto.transport.close() + self.run_loop(srv_proto.done) + self.assertEqual(ret, 100) + self.assertEqual(srv_proto.nbytes, 100) + self.assertEqual(srv_proto.data, self.DATA[1000:1100]) + self.assertEqual(self.file.tell(), 1100) + + def test_sendfile_close_peer_after_receiving(self): + srv_proto, cli_proto = self.prepare_sendfile( + close_after=len(self.DATA)) + ret = self.run_loop( + self.loop.sendfile(cli_proto.transport, self.file)) + cli_proto.transport.close() + self.run_loop(srv_proto.done) + self.assertEqual(ret, len(self.DATA)) + self.assertEqual(srv_proto.nbytes, len(self.DATA)) + self.assertEqual(srv_proto.data, self.DATA) + self.assertEqual(self.file.tell(), len(self.DATA)) + + def test_sendfile_ssl_close_peer_after_receiving(self): + srv_proto, cli_proto = self.prepare_sendfile( + is_ssl=True, close_after=len(self.DATA)) + ret = self.run_loop( + self.loop.sendfile(cli_proto.transport, self.file)) + self.run_loop(srv_proto.done) + self.assertEqual(ret, len(self.DATA)) + self.assertEqual(srv_proto.nbytes, len(self.DATA)) + self.assertEqual(srv_proto.data, self.DATA) + self.assertEqual(self.file.tell(), len(self.DATA)) + + # On Solaris, lowering SO_RCVBUF on a TCP connection after it has been + # established has no effect. Due to its age, this bug affects both Oracle + # Solaris as well as all other OpenSolaris forks (unless they fixed it + # themselves). + @unittest.skipIf(sys.platform.startswith('sunos'), + "Doesn't work on Solaris") + def test_sendfile_close_peer_in_the_middle_of_receiving(self): + srv_proto, cli_proto = self.prepare_sendfile(close_after=1024) + with self.assertRaises(ConnectionError): + self.run_loop( + self.loop.sendfile(cli_proto.transport, self.file)) + self.run_loop(srv_proto.done) + + self.assertTrue(1024 <= srv_proto.nbytes < len(self.DATA), + srv_proto.nbytes) + if not (sys.platform == 'win32' + and isinstance(self.loop, asyncio.ProactorEventLoop)): + # On Windows, Proactor uses transmitFile, which does not update tell() + self.assertTrue(1024 <= self.file.tell() < len(self.DATA), + self.file.tell()) + self.assertTrue(cli_proto.transport.is_closing()) + + def test_sendfile_fallback_close_peer_in_the_middle_of_receiving(self): + + def sendfile_native(transp, file, offset, count): + # to raise SendfileNotAvailableError + return base_events.BaseEventLoop._sendfile_native( + self.loop, transp, file, offset, count) + + self.loop._sendfile_native = sendfile_native + + srv_proto, cli_proto = self.prepare_sendfile(close_after=1024) + with self.assertRaises(ConnectionError): + try: + self.run_loop( + self.loop.sendfile(cli_proto.transport, self.file)) + except OSError as e: + # macOS may raise OSError of EPROTOTYPE when writing to a + # socket that is in the process of closing down. + if e.errno == errno.EPROTOTYPE and sys.platform == "darwin": + raise ConnectionError + else: + raise + + self.run_loop(srv_proto.done) + + self.assertTrue(1024 <= srv_proto.nbytes < len(self.DATA), + srv_proto.nbytes) + self.assertTrue(1024 <= self.file.tell() < len(self.DATA), + self.file.tell()) + + @unittest.skipIf(not hasattr(os, 'sendfile'), + "Don't have native sendfile support") + def test_sendfile_prevents_bare_write(self): + srv_proto, cli_proto = self.prepare_sendfile() + fut = self.loop.create_future() + + async def coro(): + fut.set_result(None) + return await self.loop.sendfile(cli_proto.transport, self.file) + + t = self.loop.create_task(coro()) + self.run_loop(fut) + with self.assertRaisesRegex(RuntimeError, + "sendfile is in progress"): + cli_proto.transport.write(b'data') + ret = self.run_loop(t) + self.assertEqual(ret, len(self.DATA)) + + def test_sendfile_no_fallback_for_fallback_transport(self): + transport = mock.Mock() + transport.is_closing.side_effect = lambda: False + transport._sendfile_compatible = constants._SendfileMode.FALLBACK + with self.assertRaisesRegex(RuntimeError, 'fallback is disabled'): + self.loop.run_until_complete( + self.loop.sendfile(transport, None, fallback=False)) + + +class SendfileTestsBase(SendfileMixin, SockSendfileMixin): + pass + + +if sys.platform == 'win32': + + class SelectEventLoopTests(SendfileTestsBase, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.SelectorEventLoop() + + class ProactorEventLoopTests(SendfileTestsBase, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.ProactorEventLoop() + +else: + import selectors + + if hasattr(selectors, 'KqueueSelector'): + class KqueueEventLoopTests(SendfileTestsBase, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.SelectorEventLoop( + selectors.KqueueSelector()) + + if hasattr(selectors, 'EpollSelector'): + class EPollEventLoopTests(SendfileTestsBase, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.SelectorEventLoop(selectors.EpollSelector()) + + if hasattr(selectors, 'PollSelector'): + class PollEventLoopTests(SendfileTestsBase, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.SelectorEventLoop(selectors.PollSelector()) + + # Should always exist. + class SelectEventLoopTests(SendfileTestsBase, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.SelectorEventLoop(selectors.SelectSelector()) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_server.py b/Lib/test/test_asyncio/test_server.py new file mode 100644 index 00000000000..5bd0f7e2af4 --- /dev/null +++ b/Lib/test/test_asyncio/test_server.py @@ -0,0 +1,352 @@ +import asyncio +import os +import socket +import time +import threading +import unittest + +from test.support import socket_helper +from test.test_asyncio import utils as test_utils +from test.test_asyncio import functional as func_tests + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +class BaseStartServer(func_tests.FunctionalTestCaseMixin): + + def new_loop(self): + raise NotImplementedError + + def test_start_server_1(self): + HELLO_MSG = b'1' * 1024 * 5 + b'\n' + + def client(sock, addr): + for i in range(10): + time.sleep(0.2) + if srv.is_serving(): + break + else: + raise RuntimeError + + sock.settimeout(2) + sock.connect(addr) + sock.send(HELLO_MSG) + sock.recv_all(1) + sock.close() + + async def serve(reader, writer): + await reader.readline() + main_task.cancel() + writer.write(b'1') + writer.close() + await writer.wait_closed() + + async def main(srv): + async with srv: + await srv.serve_forever() + + srv = self.loop.run_until_complete(asyncio.start_server( + serve, socket_helper.HOSTv4, 0, start_serving=False)) + + self.assertFalse(srv.is_serving()) + + main_task = self.loop.create_task(main(srv)) + + addr = srv.sockets[0].getsockname() + with self.assertRaises(asyncio.CancelledError): + with self.tcp_client(lambda sock: client(sock, addr)): + self.loop.run_until_complete(main_task) + + self.assertEqual(srv.sockets, ()) + + self.assertIsNone(srv._sockets) + self.assertIsNone(srv._waiters) + self.assertFalse(srv.is_serving()) + + with self.assertRaisesRegex(RuntimeError, r'is closed'): + self.loop.run_until_complete(srv.serve_forever()) + + +class SelectorStartServerTests(BaseStartServer, unittest.TestCase): + + def new_loop(self): + return asyncio.SelectorEventLoop() + + @socket_helper.skip_unless_bind_unix_socket + def test_start_unix_server_1(self): + HELLO_MSG = b'1' * 1024 * 5 + b'\n' + started = threading.Event() + + def client(sock, addr): + sock.settimeout(2) + started.wait(5) + sock.connect(addr) + sock.send(HELLO_MSG) + sock.recv_all(1) + sock.close() + + async def serve(reader, writer): + await reader.readline() + main_task.cancel() + writer.write(b'1') + writer.close() + await writer.wait_closed() + + async def main(srv): + async with srv: + self.assertFalse(srv.is_serving()) + await srv.start_serving() + self.assertTrue(srv.is_serving()) + started.set() + await srv.serve_forever() + + with test_utils.unix_socket_path() as addr: + srv = self.loop.run_until_complete(asyncio.start_unix_server( + serve, addr, start_serving=False)) + + main_task = self.loop.create_task(main(srv)) + + with self.assertRaises(asyncio.CancelledError): + with self.unix_client(lambda sock: client(sock, addr)): + self.loop.run_until_complete(main_task) + + self.assertEqual(srv.sockets, ()) + + self.assertIsNone(srv._sockets) + self.assertIsNone(srv._waiters) + self.assertFalse(srv.is_serving()) + + with self.assertRaisesRegex(RuntimeError, r'is closed'): + self.loop.run_until_complete(srv.serve_forever()) + + +class TestServer2(unittest.IsolatedAsyncioTestCase): + + async def test_wait_closed_basic(self): + async def serve(rd, wr): + try: + await rd.read() + finally: + wr.close() + await wr.wait_closed() + + srv = await asyncio.start_server(serve, socket_helper.HOSTv4, 0) + self.addCleanup(srv.close) + + # active count = 0, not closed: should block + task1 = asyncio.create_task(srv.wait_closed()) + await asyncio.sleep(0) + self.assertFalse(task1.done()) + + # active count != 0, not closed: should block + addr = srv.sockets[0].getsockname() + (rd, wr) = await asyncio.open_connection(addr[0], addr[1]) + task2 = asyncio.create_task(srv.wait_closed()) + await asyncio.sleep(0) + self.assertFalse(task1.done()) + self.assertFalse(task2.done()) + + srv.close() + await asyncio.sleep(0) + # active count != 0, closed: should block + task3 = asyncio.create_task(srv.wait_closed()) + await asyncio.sleep(0) + self.assertFalse(task1.done()) + self.assertFalse(task2.done()) + self.assertFalse(task3.done()) + + wr.close() + await wr.wait_closed() + # active count == 0, closed: should unblock + await task1 + await task2 + await task3 + await srv.wait_closed() # Return immediately + + async def test_wait_closed_race(self): + # Test a regression in 3.12.0, should be fixed in 3.12.1 + async def serve(rd, wr): + try: + await rd.read() + finally: + wr.close() + await wr.wait_closed() + + srv = await asyncio.start_server(serve, socket_helper.HOSTv4, 0) + self.addCleanup(srv.close) + + task = asyncio.create_task(srv.wait_closed()) + await asyncio.sleep(0) + self.assertFalse(task.done()) + addr = srv.sockets[0].getsockname() + (rd, wr) = await asyncio.open_connection(addr[0], addr[1]) + loop = asyncio.get_running_loop() + loop.call_soon(srv.close) + loop.call_soon(wr.close) + await srv.wait_closed() + + async def test_close_clients(self): + async def serve(rd, wr): + try: + await rd.read() + finally: + wr.close() + await wr.wait_closed() + + srv = await asyncio.start_server(serve, socket_helper.HOSTv4, 0) + self.addCleanup(srv.close) + + addr = srv.sockets[0].getsockname() + (rd, wr) = await asyncio.open_connection(addr[0], addr[1]) + self.addCleanup(wr.close) + + task = asyncio.create_task(srv.wait_closed()) + await asyncio.sleep(0) + self.assertFalse(task.done()) + + srv.close() + srv.close_clients() + await asyncio.sleep(0) + await asyncio.sleep(0) + self.assertTrue(task.done()) + + async def test_abort_clients(self): + async def serve(rd, wr): + fut.set_result((rd, wr)) + await wr.wait_closed() + + fut = asyncio.Future() + srv = await asyncio.start_server(serve, socket_helper.HOSTv4, 0) + self.addCleanup(srv.close) + + addr = srv.sockets[0].getsockname() + (c_rd, c_wr) = await asyncio.open_connection(addr[0], addr[1], limit=4096) + self.addCleanup(c_wr.close) + + (s_rd, s_wr) = await fut + + # Limit the socket buffers so we can more reliably overfill them + s_sock = s_wr.get_extra_info('socket') + s_sock.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 65536) + c_sock = c_wr.get_extra_info('socket') + c_sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 65536) + + # Get the reader in to a paused state by sending more than twice + # the configured limit + s_wr.write(b'a' * 4096) + s_wr.write(b'a' * 4096) + s_wr.write(b'a' * 4096) + while c_wr.transport.is_reading(): + await asyncio.sleep(0) + + # Get the writer in a waiting state by sending data until the + # kernel stops accepting more data in the send buffer. + # gh-122136: getsockopt() does not reliably report the buffer size + # available for message content. + # We loop until we start filling up the asyncio buffer. + # To avoid an infinite loop we cap at 10 times the expected value + c_bufsize = c_sock.getsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF) + s_bufsize = s_sock.getsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF) + for i in range(10): + s_wr.write(b'a' * c_bufsize) + s_wr.write(b'a' * s_bufsize) + if s_wr.transport.get_write_buffer_size() > 0: + break + self.assertNotEqual(s_wr.transport.get_write_buffer_size(), 0) + + task = asyncio.create_task(srv.wait_closed()) + await asyncio.sleep(0) + self.assertFalse(task.done()) + + srv.close() + srv.abort_clients() + await asyncio.sleep(0) + await asyncio.sleep(0) + self.assertTrue(task.done()) + + +# Test the various corner cases of Unix server socket removal +class UnixServerCleanupTests(unittest.IsolatedAsyncioTestCase): + @socket_helper.skip_unless_bind_unix_socket + async def test_unix_server_addr_cleanup(self): + # Default scenario + with test_utils.unix_socket_path() as addr: + async def serve(*args): + pass + + srv = await asyncio.start_unix_server(serve, addr) + + srv.close() + self.assertFalse(os.path.exists(addr)) + + @socket_helper.skip_unless_bind_unix_socket + async def test_unix_server_sock_cleanup(self): + # Using already bound socket + with test_utils.unix_socket_path() as addr: + async def serve(*args): + pass + + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as sock: + sock.bind(addr) + + srv = await asyncio.start_unix_server(serve, sock=sock) + + srv.close() + self.assertFalse(os.path.exists(addr)) + + @socket_helper.skip_unless_bind_unix_socket + async def test_unix_server_cleanup_gone(self): + # Someone else has already cleaned up the socket + with test_utils.unix_socket_path() as addr: + async def serve(*args): + pass + + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as sock: + sock.bind(addr) + + srv = await asyncio.start_unix_server(serve, sock=sock) + + os.unlink(addr) + + srv.close() + + @socket_helper.skip_unless_bind_unix_socket + async def test_unix_server_cleanup_replaced(self): + # Someone else has replaced the socket with their own + with test_utils.unix_socket_path() as addr: + async def serve(*args): + pass + + srv = await asyncio.start_unix_server(serve, addr) + + os.unlink(addr) + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as sock: + sock.bind(addr) + + srv.close() + self.assertTrue(os.path.exists(addr)) + + @socket_helper.skip_unless_bind_unix_socket + async def test_unix_server_cleanup_prevented(self): + # Automatic cleanup explicitly disabled + with test_utils.unix_socket_path() as addr: + async def serve(*args): + pass + + srv = await asyncio.start_unix_server(serve, addr, cleanup_socket=False) + + srv.close() + self.assertTrue(os.path.exists(addr)) + + +@unittest.skipUnless(hasattr(asyncio, 'ProactorEventLoop'), 'Windows only') +class ProactorStartServerTests(BaseStartServer, unittest.TestCase): + + def new_loop(self): + return asyncio.ProactorEventLoop() + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_sock_lowlevel.py b/Lib/test/test_asyncio/test_sock_lowlevel.py new file mode 100644 index 00000000000..df4ec794897 --- /dev/null +++ b/Lib/test/test_asyncio/test_sock_lowlevel.py @@ -0,0 +1,679 @@ +import socket +import asyncio +import sys +import unittest + +from asyncio import proactor_events +from itertools import cycle, islice +from unittest.mock import Mock +from test.test_asyncio import utils as test_utils +from test import support +from test.support import socket_helper + +if socket_helper.tcp_blackhole(): + raise unittest.SkipTest('Not relevant to ProactorEventLoop') + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +class MyProto(asyncio.Protocol): + connected = None + done = None + + def __init__(self, loop=None): + self.transport = None + self.state = 'INITIAL' + self.nbytes = 0 + if loop is not None: + self.connected = loop.create_future() + self.done = loop.create_future() + + def _assert_state(self, *expected): + if self.state not in expected: + raise AssertionError(f'state: {self.state!r}, expected: {expected!r}') + + def connection_made(self, transport): + self.transport = transport + self._assert_state('INITIAL') + self.state = 'CONNECTED' + if self.connected: + self.connected.set_result(None) + transport.write(b'GET / HTTP/1.0\r\nHost: example.com\r\n\r\n') + + def data_received(self, data): + self._assert_state('CONNECTED') + self.nbytes += len(data) + + def eof_received(self): + self._assert_state('CONNECTED') + self.state = 'EOF' + + def connection_lost(self, exc): + self._assert_state('CONNECTED', 'EOF') + self.state = 'CLOSED' + if self.done: + self.done.set_result(None) + + +class BaseSockTestsMixin: + + def create_event_loop(self): + raise NotImplementedError + + def setUp(self): + self.loop = self.create_event_loop() + self.set_event_loop(self.loop) + super().setUp() + + def tearDown(self): + # just in case if we have transport close callbacks + if not self.loop.is_closed(): + test_utils.run_briefly(self.loop) + + self.doCleanups() + support.gc_collect() + super().tearDown() + + def _basetest_sock_client_ops(self, httpd, sock): + if not isinstance(self.loop, proactor_events.BaseProactorEventLoop): + # in debug mode, socket operations must fail + # if the socket is not in blocking mode + self.loop.set_debug(True) + sock.setblocking(True) + with self.assertRaises(ValueError): + self.loop.run_until_complete( + self.loop.sock_connect(sock, httpd.address)) + with self.assertRaises(ValueError): + self.loop.run_until_complete( + self.loop.sock_sendall(sock, b'GET / HTTP/1.0\r\n\r\n')) + with self.assertRaises(ValueError): + self.loop.run_until_complete( + self.loop.sock_recv(sock, 1024)) + with self.assertRaises(ValueError): + self.loop.run_until_complete( + self.loop.sock_recv_into(sock, bytearray())) + with self.assertRaises(ValueError): + self.loop.run_until_complete( + self.loop.sock_accept(sock)) + + # test in non-blocking mode + sock.setblocking(False) + self.loop.run_until_complete( + self.loop.sock_connect(sock, httpd.address)) + self.loop.run_until_complete( + self.loop.sock_sendall(sock, b'GET / HTTP/1.0\r\n\r\n')) + data = self.loop.run_until_complete( + self.loop.sock_recv(sock, 1024)) + # consume data + self.loop.run_until_complete( + self.loop.sock_recv(sock, 1024)) + sock.close() + self.assertStartsWith(data, b'HTTP/1.0 200 OK') + + def _basetest_sock_recv_into(self, httpd, sock): + # same as _basetest_sock_client_ops, but using sock_recv_into + sock.setblocking(False) + self.loop.run_until_complete( + self.loop.sock_connect(sock, httpd.address)) + self.loop.run_until_complete( + self.loop.sock_sendall(sock, b'GET / HTTP/1.0\r\n\r\n')) + data = bytearray(1024) + with memoryview(data) as buf: + nbytes = self.loop.run_until_complete( + self.loop.sock_recv_into(sock, buf[:1024])) + # consume data + self.loop.run_until_complete( + self.loop.sock_recv_into(sock, buf[nbytes:])) + sock.close() + self.assertStartsWith(data, b'HTTP/1.0 200 OK') + + def test_sock_client_ops(self): + with test_utils.run_test_server() as httpd: + sock = socket.socket() + self._basetest_sock_client_ops(httpd, sock) + sock = socket.socket() + self._basetest_sock_recv_into(httpd, sock) + + async def _basetest_sock_recv_racing(self, httpd, sock): + sock.setblocking(False) + await self.loop.sock_connect(sock, httpd.address) + + task = asyncio.create_task(self.loop.sock_recv(sock, 1024)) + await asyncio.sleep(0) + task.cancel() + + asyncio.create_task( + self.loop.sock_sendall(sock, b'GET / HTTP/1.0\r\n\r\n')) + data = await self.loop.sock_recv(sock, 1024) + # consume data + await self.loop.sock_recv(sock, 1024) + + self.assertStartsWith(data, b'HTTP/1.0 200 OK') + + async def _basetest_sock_recv_into_racing(self, httpd, sock): + sock.setblocking(False) + await self.loop.sock_connect(sock, httpd.address) + + data = bytearray(1024) + with memoryview(data) as buf: + task = asyncio.create_task( + self.loop.sock_recv_into(sock, buf[:1024])) + await asyncio.sleep(0) + task.cancel() + + task = asyncio.create_task( + self.loop.sock_sendall(sock, b'GET / HTTP/1.0\r\n\r\n')) + nbytes = await self.loop.sock_recv_into(sock, buf[:1024]) + # consume data + await self.loop.sock_recv_into(sock, buf[nbytes:]) + self.assertStartsWith(data, b'HTTP/1.0 200 OK') + + await task + + async def _basetest_sock_send_racing(self, listener, sock): + listener.bind(('127.0.0.1', 0)) + listener.listen(1) + + # make connection + sock.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 1024) + sock.setblocking(False) + task = asyncio.create_task( + self.loop.sock_connect(sock, listener.getsockname())) + await asyncio.sleep(0) + server = listener.accept()[0] + server.setblocking(False) + + with server: + await task + + # fill the buffer until sending 5 chars would block + size = 8192 + while size >= 4: + with self.assertRaises(BlockingIOError): + while True: + sock.send(b' ' * size) + size = int(size / 2) + + # cancel a blocked sock_sendall + task = asyncio.create_task( + self.loop.sock_sendall(sock, b'hello')) + await asyncio.sleep(0) + task.cancel() + + # receive everything that is not a space + async def recv_all(): + rv = b'' + while True: + buf = await self.loop.sock_recv(server, 8192) + if not buf: + return rv + rv += buf.strip() + task = asyncio.create_task(recv_all()) + + # immediately make another sock_sendall call + await self.loop.sock_sendall(sock, b'world') + sock.shutdown(socket.SHUT_WR) + data = await task + # ProactorEventLoop could deliver hello, so endswith is necessary + self.assertEndsWith(data, b'world') + + # After the first connect attempt before the listener is ready, + # the socket needs time to "recover" to make the next connect call. + # On Linux, a second retry will do. On Windows, the waiting time is + # unpredictable; and on FreeBSD the socket may never come back + # because it's a loopback address. Here we'll just retry for a few + # times, and have to skip the test if it's not working. See also: + # https://stackoverflow.com/a/54437602/3316267 + # https://lists.freebsd.org/pipermail/freebsd-current/2005-May/049876.html + async def _basetest_sock_connect_racing(self, listener, sock): + listener.bind(('127.0.0.1', 0)) + addr = listener.getsockname() + sock.setblocking(False) + + task = asyncio.create_task(self.loop.sock_connect(sock, addr)) + await asyncio.sleep(0) + task.cancel() + + listener.listen(1) + + skip_reason = "Max retries reached" + for i in range(128): + try: + await self.loop.sock_connect(sock, addr) + except ConnectionRefusedError as e: + skip_reason = e + except OSError as e: + skip_reason = e + + # Retry only for this error: + # [WinError 10022] An invalid argument was supplied + if getattr(e, 'winerror', 0) != 10022: + break + else: + # success + return + + self.skipTest(skip_reason) + + def test_sock_client_racing(self): + with test_utils.run_test_server() as httpd: + sock = socket.socket() + with sock: + self.loop.run_until_complete(asyncio.wait_for( + self._basetest_sock_recv_racing(httpd, sock), 10)) + sock = socket.socket() + with sock: + self.loop.run_until_complete(asyncio.wait_for( + self._basetest_sock_recv_into_racing(httpd, sock), 10)) + listener = socket.socket() + sock = socket.socket() + with listener, sock: + self.loop.run_until_complete(asyncio.wait_for( + self._basetest_sock_send_racing(listener, sock), 10)) + + def test_sock_client_connect_racing(self): + listener = socket.socket() + sock = socket.socket() + with listener, sock: + self.loop.run_until_complete(asyncio.wait_for( + self._basetest_sock_connect_racing(listener, sock), 10)) + + async def _basetest_huge_content(self, address): + sock = socket.socket() + sock.setblocking(False) + DATA_SIZE = 10_000_00 + + chunk = b'0123456789' * (DATA_SIZE // 10) + + await self.loop.sock_connect(sock, address) + await self.loop.sock_sendall(sock, + (b'POST /loop HTTP/1.0\r\n' + + b'Content-Length: %d\r\n' % DATA_SIZE + + b'\r\n')) + + task = asyncio.create_task(self.loop.sock_sendall(sock, chunk)) + + data = await self.loop.sock_recv(sock, DATA_SIZE) + # HTTP headers size is less than MTU, + # they are sent by the first packet always + self.assertStartsWith(data, b'HTTP/1.0 200 OK') + while data.find(b'\r\n\r\n') == -1: + data += await self.loop.sock_recv(sock, DATA_SIZE) + # Strip headers + headers = data[:data.index(b'\r\n\r\n') + 4] + data = data[len(headers):] + + size = DATA_SIZE + checker = cycle(b'0123456789') + + expected = bytes(islice(checker, len(data))) + self.assertEqual(data, expected) + size -= len(data) + + while True: + data = await self.loop.sock_recv(sock, DATA_SIZE) + if not data: + break + expected = bytes(islice(checker, len(data))) + self.assertEqual(data, expected) + size -= len(data) + self.assertEqual(size, 0) + + await task + sock.close() + + def test_huge_content(self): + with test_utils.run_test_server() as httpd: + self.loop.run_until_complete( + self._basetest_huge_content(httpd.address)) + + async def _basetest_huge_content_recvinto(self, address): + sock = socket.socket() + sock.setblocking(False) + DATA_SIZE = 10_000_00 + + chunk = b'0123456789' * (DATA_SIZE // 10) + + await self.loop.sock_connect(sock, address) + await self.loop.sock_sendall(sock, + (b'POST /loop HTTP/1.0\r\n' + + b'Content-Length: %d\r\n' % DATA_SIZE + + b'\r\n')) + + task = asyncio.create_task(self.loop.sock_sendall(sock, chunk)) + + array = bytearray(DATA_SIZE) + buf = memoryview(array) + + nbytes = await self.loop.sock_recv_into(sock, buf) + data = bytes(buf[:nbytes]) + # HTTP headers size is less than MTU, + # they are sent by the first packet always + self.assertStartsWith(data, b'HTTP/1.0 200 OK') + while data.find(b'\r\n\r\n') == -1: + nbytes = await self.loop.sock_recv_into(sock, buf) + data = bytes(buf[:nbytes]) + # Strip headers + headers = data[:data.index(b'\r\n\r\n') + 4] + data = data[len(headers):] + + size = DATA_SIZE + checker = cycle(b'0123456789') + + expected = bytes(islice(checker, len(data))) + self.assertEqual(data, expected) + size -= len(data) + + while True: + nbytes = await self.loop.sock_recv_into(sock, buf) + data = buf[:nbytes] + if not data: + break + expected = bytes(islice(checker, len(data))) + self.assertEqual(data, expected) + size -= len(data) + self.assertEqual(size, 0) + + await task + sock.close() + + def test_huge_content_recvinto(self): + with test_utils.run_test_server() as httpd: + self.loop.run_until_complete( + self._basetest_huge_content_recvinto(httpd.address)) + + async def _basetest_datagram_recvfrom(self, server_address): + # Happy path, sock.sendto() returns immediately + data = b'\x01' * 4096 + with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock: + sock.setblocking(False) + await self.loop.sock_sendto(sock, data, server_address) + received_data, from_addr = await self.loop.sock_recvfrom( + sock, 4096) + self.assertEqual(received_data, data) + self.assertEqual(from_addr, server_address) + + def test_recvfrom(self): + with test_utils.run_udp_echo_server() as server_address: + self.loop.run_until_complete( + self._basetest_datagram_recvfrom(server_address)) + + async def _basetest_datagram_recvfrom_into(self, server_address): + # Happy path, sock.sendto() returns immediately + with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock: + sock.setblocking(False) + + buf = bytearray(4096) + data = b'\x01' * 4096 + await self.loop.sock_sendto(sock, data, server_address) + num_bytes, from_addr = await self.loop.sock_recvfrom_into( + sock, buf) + self.assertEqual(num_bytes, 4096) + self.assertEqual(buf, data) + self.assertEqual(from_addr, server_address) + + buf = bytearray(8192) + await self.loop.sock_sendto(sock, data, server_address) + num_bytes, from_addr = await self.loop.sock_recvfrom_into( + sock, buf, 4096) + self.assertEqual(num_bytes, 4096) + self.assertEqual(buf[:4096], data[:4096]) + self.assertEqual(from_addr, server_address) + + def test_recvfrom_into(self): + with test_utils.run_udp_echo_server() as server_address: + self.loop.run_until_complete( + self._basetest_datagram_recvfrom_into(server_address)) + + async def _basetest_datagram_sendto_blocking(self, server_address): + # Sad path, sock.sendto() raises BlockingIOError + # This involves patching sock.sendto() to raise BlockingIOError but + # sendto() is not used by the proactor event loop + data = b'\x01' * 4096 + with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock: + sock.setblocking(False) + mock_sock = Mock(sock) + mock_sock.gettimeout = sock.gettimeout + mock_sock.sendto.configure_mock(side_effect=BlockingIOError) + mock_sock.fileno = sock.fileno + self.loop.call_soon( + lambda: setattr(mock_sock, 'sendto', sock.sendto) + ) + await self.loop.sock_sendto(mock_sock, data, server_address) + + received_data, from_addr = await self.loop.sock_recvfrom( + sock, 4096) + self.assertEqual(received_data, data) + self.assertEqual(from_addr, server_address) + + def test_sendto_blocking(self): + if sys.platform == 'win32': + if isinstance(self.loop, asyncio.ProactorEventLoop): + raise unittest.SkipTest('Not relevant to ProactorEventLoop') + + with test_utils.run_udp_echo_server() as server_address: + self.loop.run_until_complete( + self._basetest_datagram_sendto_blocking(server_address)) + + @socket_helper.skip_unless_bind_unix_socket + def test_unix_sock_client_ops(self): + with test_utils.run_test_unix_server() as httpd: + sock = socket.socket(socket.AF_UNIX) + self._basetest_sock_client_ops(httpd, sock) + sock = socket.socket(socket.AF_UNIX) + self._basetest_sock_recv_into(httpd, sock) + + def test_sock_client_fail(self): + # Make sure that we will get an unused port + address = None + try: + s = socket.socket() + s.bind(('127.0.0.1', 0)) + address = s.getsockname() + finally: + s.close() + + sock = socket.socket() + sock.setblocking(False) + with self.assertRaises(ConnectionRefusedError): + self.loop.run_until_complete( + self.loop.sock_connect(sock, address)) + sock.close() + + def test_sock_accept(self): + listener = socket.socket() + listener.setblocking(False) + listener.bind(('127.0.0.1', 0)) + listener.listen(1) + client = socket.socket() + client.connect(listener.getsockname()) + + f = self.loop.sock_accept(listener) + conn, addr = self.loop.run_until_complete(f) + self.assertEqual(conn.gettimeout(), 0) + self.assertEqual(addr, client.getsockname()) + self.assertEqual(client.getpeername(), listener.getsockname()) + client.close() + conn.close() + listener.close() + + def test_cancel_sock_accept(self): + listener = socket.socket() + listener.setblocking(False) + listener.bind(('127.0.0.1', 0)) + listener.listen(1) + sockaddr = listener.getsockname() + f = asyncio.wait_for(self.loop.sock_accept(listener), 0.1) + with self.assertRaises(asyncio.TimeoutError): + self.loop.run_until_complete(f) + + listener.close() + client = socket.socket() + client.setblocking(False) + f = self.loop.sock_connect(client, sockaddr) + with self.assertRaises(ConnectionRefusedError): + self.loop.run_until_complete(f) + + client.close() + + def test_create_connection_sock(self): + with test_utils.run_test_server() as httpd: + sock = None + infos = self.loop.run_until_complete( + self.loop.getaddrinfo( + *httpd.address, type=socket.SOCK_STREAM)) + for family, type, proto, cname, address in infos: + try: + sock = socket.socket(family=family, type=type, proto=proto) + sock.setblocking(False) + self.loop.run_until_complete( + self.loop.sock_connect(sock, address)) + except BaseException: + pass + else: + break + else: + self.fail('Can not create socket.') + + f = self.loop.create_connection( + lambda: MyProto(loop=self.loop), sock=sock) + tr, pr = self.loop.run_until_complete(f) + self.assertIsInstance(tr, asyncio.Transport) + self.assertIsInstance(pr, asyncio.Protocol) + self.loop.run_until_complete(pr.done) + self.assertGreater(pr.nbytes, 0) + tr.close() + + +if sys.platform == 'win32': + + class SelectEventLoopTests(BaseSockTestsMixin, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.SelectorEventLoop() + + + class ProactorEventLoopTests(BaseSockTestsMixin, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.ProactorEventLoop() + + + async def _basetest_datagram_send_to_non_listening_address(self, + recvfrom): + # see: + # https://github.com/python/cpython/issues/91227 + # https://github.com/python/cpython/issues/88906 + # https://bugs.python.org/issue47071 + # https://bugs.python.org/issue44743 + # The Proactor event loop would fail to receive datagram messages + # after sending a message to an address that wasn't listening. + + def create_socket(): + sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + sock.setblocking(False) + sock.bind(('127.0.0.1', 0)) + return sock + + socket_1 = create_socket() + addr_1 = socket_1.getsockname() + + socket_2 = create_socket() + addr_2 = socket_2.getsockname() + + # creating and immediately closing this to try to get an address + # that is not listening + socket_3 = create_socket() + addr_3 = socket_3.getsockname() + socket_3.shutdown(socket.SHUT_RDWR) + socket_3.close() + + socket_1_recv_task = self.loop.create_task(recvfrom(socket_1)) + socket_2_recv_task = self.loop.create_task(recvfrom(socket_2)) + await asyncio.sleep(0) + + await self.loop.sock_sendto(socket_1, b'a', addr_2) + self.assertEqual(await socket_2_recv_task, b'a') + + await self.loop.sock_sendto(socket_2, b'b', addr_1) + self.assertEqual(await socket_1_recv_task, b'b') + socket_1_recv_task = self.loop.create_task(recvfrom(socket_1)) + await asyncio.sleep(0) + + # this should send to an address that isn't listening + await self.loop.sock_sendto(socket_1, b'c', addr_3) + self.assertEqual(await socket_1_recv_task, b'') + socket_1_recv_task = self.loop.create_task(recvfrom(socket_1)) + await asyncio.sleep(0) + + # socket 1 should still be able to receive messages after sending + # to an address that wasn't listening + socket_2.sendto(b'd', addr_1) + self.assertEqual(await socket_1_recv_task, b'd') + + socket_1.shutdown(socket.SHUT_RDWR) + socket_1.close() + socket_2.shutdown(socket.SHUT_RDWR) + socket_2.close() + + + def test_datagram_send_to_non_listening_address_recvfrom(self): + async def recvfrom(socket): + data, _ = await self.loop.sock_recvfrom(socket, 4096) + return data + + self.loop.run_until_complete( + self._basetest_datagram_send_to_non_listening_address( + recvfrom)) + + + def test_datagram_send_to_non_listening_address_recvfrom_into(self): + async def recvfrom_into(socket): + buf = bytearray(4096) + length, _ = await self.loop.sock_recvfrom_into(socket, buf, + 4096) + return buf[:length] + + self.loop.run_until_complete( + self._basetest_datagram_send_to_non_listening_address( + recvfrom_into)) + +else: + import selectors + + if hasattr(selectors, 'KqueueSelector'): + class KqueueEventLoopTests(BaseSockTestsMixin, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.SelectorEventLoop( + selectors.KqueueSelector()) + + if hasattr(selectors, 'EpollSelector'): + class EPollEventLoopTests(BaseSockTestsMixin, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.SelectorEventLoop(selectors.EpollSelector()) + + if hasattr(selectors, 'PollSelector'): + class PollEventLoopTests(BaseSockTestsMixin, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.SelectorEventLoop(selectors.PollSelector()) + + # Should always exist. + class SelectEventLoopTests(BaseSockTestsMixin, + test_utils.TestCase): + + def create_event_loop(self): + return asyncio.SelectorEventLoop(selectors.SelectSelector()) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_ssl.py b/Lib/test/test_asyncio/test_ssl.py new file mode 100644 index 00000000000..e5d3b63b94f --- /dev/null +++ b/Lib/test/test_asyncio/test_ssl.py @@ -0,0 +1,1906 @@ +# Contains code from https://github.com/MagicStack/uvloop/tree/v0.16.0 +# SPDX-License-Identifier: PSF-2.0 AND (MIT OR Apache-2.0) +# SPDX-FileCopyrightText: Copyright (c) 2015-2021 MagicStack Inc. http://magic.io + +import asyncio +import contextlib +import gc +import logging +import select +import socket +import sys +import tempfile +import threading +import time +import unittest.mock +import weakref +import unittest + +try: + import ssl +except ImportError: + ssl = None + +from test import support +from test.test_asyncio import utils as test_utils + + +MACOS = (sys.platform == 'darwin') +BUF_MULTIPLIER = 1024 if not MACOS else 64 + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +class MyBaseProto(asyncio.Protocol): + connected = None + done = None + + def __init__(self, loop=None): + self.transport = None + self.state = 'INITIAL' + self.nbytes = 0 + if loop is not None: + self.connected = asyncio.Future(loop=loop) + self.done = asyncio.Future(loop=loop) + + def connection_made(self, transport): + self.transport = transport + assert self.state == 'INITIAL', self.state + self.state = 'CONNECTED' + if self.connected: + self.connected.set_result(None) + + def data_received(self, data): + assert self.state == 'CONNECTED', self.state + self.nbytes += len(data) + + def eof_received(self): + assert self.state == 'CONNECTED', self.state + self.state = 'EOF' + + def connection_lost(self, exc): + assert self.state in ('CONNECTED', 'EOF'), self.state + self.state = 'CLOSED' + if self.done: + self.done.set_result(None) + + +class MessageOutFilter(logging.Filter): + def __init__(self, msg): + self.msg = msg + + def filter(self, record): + if self.msg in record.msg: + return False + return True + + +@unittest.skipIf(ssl is None, 'No ssl module') +class TestSSL(test_utils.TestCase): + + PAYLOAD_SIZE = 1024 * 100 + TIMEOUT = support.LONG_TIMEOUT + + def setUp(self): + super().setUp() + self.loop = asyncio.new_event_loop() + self.set_event_loop(self.loop) + self.addCleanup(self.loop.close) + + def tearDown(self): + # just in case if we have transport close callbacks + if not self.loop.is_closed(): + test_utils.run_briefly(self.loop) + + self.doCleanups() + support.gc_collect() + super().tearDown() + + def tcp_server(self, server_prog, *, + family=socket.AF_INET, + addr=None, + timeout=support.SHORT_TIMEOUT, + backlog=1, + max_clients=10): + + if addr is None: + if family == getattr(socket, "AF_UNIX", None): + with tempfile.NamedTemporaryFile() as tmp: + addr = tmp.name + else: + addr = ('127.0.0.1', 0) + + sock = socket.socket(family, socket.SOCK_STREAM) + + if timeout is None: + raise RuntimeError('timeout is required') + if timeout <= 0: + raise RuntimeError('only blocking sockets are supported') + sock.settimeout(timeout) + + try: + sock.bind(addr) + sock.listen(backlog) + except OSError as ex: + sock.close() + raise ex + + return TestThreadedServer( + self, sock, server_prog, timeout, max_clients) + + def tcp_client(self, client_prog, + family=socket.AF_INET, + timeout=support.SHORT_TIMEOUT): + + sock = socket.socket(family, socket.SOCK_STREAM) + + if timeout is None: + raise RuntimeError('timeout is required') + if timeout <= 0: + raise RuntimeError('only blocking sockets are supported') + sock.settimeout(timeout) + + return TestThreadedClient( + self, sock, client_prog, timeout) + + def unix_server(self, *args, **kwargs): + return self.tcp_server(*args, family=socket.AF_UNIX, **kwargs) + + def unix_client(self, *args, **kwargs): + return self.tcp_client(*args, family=socket.AF_UNIX, **kwargs) + + def _create_server_ssl_context(self, certfile, keyfile=None): + sslcontext = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) + sslcontext.options |= ssl.OP_NO_SSLv2 + sslcontext.load_cert_chain(certfile, keyfile) + return sslcontext + + def _create_client_ssl_context(self, *, disable_verify=True): + sslcontext = ssl.create_default_context() + sslcontext.check_hostname = False + if disable_verify: + sslcontext.verify_mode = ssl.CERT_NONE + return sslcontext + + @contextlib.contextmanager + def _silence_eof_received_warning(self): + # TODO This warning has to be fixed in asyncio. + logger = logging.getLogger('asyncio') + filter = MessageOutFilter('has no effect when using ssl') + logger.addFilter(filter) + try: + yield + finally: + logger.removeFilter(filter) + + def _abort_socket_test(self, ex): + try: + self.loop.stop() + finally: + self.fail(ex) + + def new_loop(self): + return asyncio.new_event_loop() + + def new_policy(self): + return asyncio.DefaultEventLoopPolicy() + + async def wait_closed(self, obj): + if not isinstance(obj, asyncio.StreamWriter): + return + try: + await obj.wait_closed() + except (BrokenPipeError, ConnectionError): + pass + + @support.bigmemtest(size=25, memuse=90*2**20, dry_run=False) + def test_create_server_ssl_1(self, size): + CNT = 0 # number of clients that were successful + TOTAL_CNT = size # total number of clients that test will create + TIMEOUT = support.LONG_TIMEOUT # timeout for this test + + A_DATA = b'A' * 1024 * BUF_MULTIPLIER + B_DATA = b'B' * 1024 * BUF_MULTIPLIER + + sslctx = self._create_server_ssl_context( + test_utils.ONLYCERT, test_utils.ONLYKEY + ) + client_sslctx = self._create_client_ssl_context() + + clients = [] + + async def handle_client(reader, writer): + nonlocal CNT + + data = await reader.readexactly(len(A_DATA)) + self.assertEqual(data, A_DATA) + writer.write(b'OK') + + data = await reader.readexactly(len(B_DATA)) + self.assertEqual(data, B_DATA) + writer.writelines([b'SP', bytearray(b'A'), memoryview(b'M')]) + + await writer.drain() + writer.close() + + CNT += 1 + + async def test_client(addr): + fut = asyncio.Future() + + def prog(sock): + try: + sock.starttls(client_sslctx) + sock.connect(addr) + sock.send(A_DATA) + + data = sock.recv_all(2) + self.assertEqual(data, b'OK') + + sock.send(B_DATA) + data = sock.recv_all(4) + self.assertEqual(data, b'SPAM') + + sock.close() + + except Exception as ex: + self.loop.call_soon_threadsafe(fut.set_exception, ex) + else: + self.loop.call_soon_threadsafe(fut.set_result, None) + + client = self.tcp_client(prog) + client.start() + clients.append(client) + + await fut + + async def start_server(): + extras = {} + extras = dict(ssl_handshake_timeout=support.SHORT_TIMEOUT) + + srv = await asyncio.start_server( + handle_client, + '127.0.0.1', 0, + family=socket.AF_INET, + ssl=sslctx, + **extras) + + try: + srv_socks = srv.sockets + self.assertTrue(srv_socks) + + addr = srv_socks[0].getsockname() + + tasks = [] + for _ in range(TOTAL_CNT): + tasks.append(test_client(addr)) + + await asyncio.wait_for(asyncio.gather(*tasks), TIMEOUT) + + finally: + self.loop.call_soon(srv.close) + await srv.wait_closed() + + with self._silence_eof_received_warning(): + self.loop.run_until_complete(start_server()) + + self.assertEqual(CNT, TOTAL_CNT) + + for client in clients: + client.stop() + + def test_create_connection_ssl_1(self): + self.loop.set_exception_handler(None) + + CNT = 0 + TOTAL_CNT = 25 + + A_DATA = b'A' * 1024 * BUF_MULTIPLIER + B_DATA = b'B' * 1024 * BUF_MULTIPLIER + + sslctx = self._create_server_ssl_context( + test_utils.ONLYCERT, + test_utils.ONLYKEY + ) + client_sslctx = self._create_client_ssl_context() + + def server(sock): + sock.starttls( + sslctx, + server_side=True) + + data = sock.recv_all(len(A_DATA)) + self.assertEqual(data, A_DATA) + sock.send(b'OK') + + data = sock.recv_all(len(B_DATA)) + self.assertEqual(data, B_DATA) + sock.send(b'SPAM') + + sock.close() + + async def client(addr): + extras = {} + extras = dict(ssl_handshake_timeout=support.SHORT_TIMEOUT) + + reader, writer = await asyncio.open_connection( + *addr, + ssl=client_sslctx, + server_hostname='', + **extras) + + writer.write(A_DATA) + self.assertEqual(await reader.readexactly(2), b'OK') + + writer.write(B_DATA) + self.assertEqual(await reader.readexactly(4), b'SPAM') + + nonlocal CNT + CNT += 1 + + writer.close() + await self.wait_closed(writer) + + async def client_sock(addr): + sock = socket.socket() + sock.connect(addr) + reader, writer = await asyncio.open_connection( + sock=sock, + ssl=client_sslctx, + server_hostname='') + + writer.write(A_DATA) + self.assertEqual(await reader.readexactly(2), b'OK') + + writer.write(B_DATA) + self.assertEqual(await reader.readexactly(4), b'SPAM') + + nonlocal CNT + CNT += 1 + + writer.close() + await self.wait_closed(writer) + sock.close() + + def run(coro): + nonlocal CNT + CNT = 0 + + async def _gather(*tasks): + # trampoline + return await asyncio.gather(*tasks) + + with self.tcp_server(server, + max_clients=TOTAL_CNT, + backlog=TOTAL_CNT) as srv: + tasks = [] + for _ in range(TOTAL_CNT): + tasks.append(coro(srv.addr)) + + self.loop.run_until_complete(_gather(*tasks)) + + self.assertEqual(CNT, TOTAL_CNT) + + with self._silence_eof_received_warning(): + run(client) + + with self._silence_eof_received_warning(): + run(client_sock) + + def test_create_connection_ssl_slow_handshake(self): + client_sslctx = self._create_client_ssl_context() + + # silence error logger + self.loop.set_exception_handler(lambda *args: None) + + def server(sock): + try: + sock.recv_all(1024 * 1024) + except ConnectionAbortedError: + pass + finally: + sock.close() + + async def client(addr): + reader, writer = await asyncio.open_connection( + *addr, + ssl=client_sslctx, + server_hostname='', + ssl_handshake_timeout=1.0) + writer.close() + await self.wait_closed(writer) + + with self.tcp_server(server, + max_clients=1, + backlog=1) as srv: + + with self.assertRaisesRegex( + ConnectionAbortedError, + r'SSL handshake.*is taking longer'): + + self.loop.run_until_complete(client(srv.addr)) + + def test_create_connection_ssl_failed_certificate(self): + # silence error logger + self.loop.set_exception_handler(lambda *args: None) + + sslctx = self._create_server_ssl_context( + test_utils.ONLYCERT, + test_utils.ONLYKEY + ) + client_sslctx = self._create_client_ssl_context(disable_verify=False) + + def server(sock): + try: + sock.starttls( + sslctx, + server_side=True) + sock.connect() + except (ssl.SSLError, OSError): + pass + finally: + sock.close() + + async def client(addr): + reader, writer = await asyncio.open_connection( + *addr, + ssl=client_sslctx, + server_hostname='', + ssl_handshake_timeout=support.SHORT_TIMEOUT) + writer.close() + await self.wait_closed(writer) + + with self.tcp_server(server, + max_clients=1, + backlog=1) as srv: + + with self.assertRaises(ssl.SSLCertVerificationError): + self.loop.run_until_complete(client(srv.addr)) + + def test_ssl_handshake_timeout(self): + # bpo-29970: Check that a connection is aborted if handshake is not + # completed in timeout period, instead of remaining open indefinitely + client_sslctx = test_utils.simple_client_sslcontext() + + # silence error logger + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + + server_side_aborted = False + + def server(sock): + nonlocal server_side_aborted + try: + sock.recv_all(1024 * 1024) + except ConnectionAbortedError: + server_side_aborted = True + finally: + sock.close() + + async def client(addr): + await asyncio.wait_for( + self.loop.create_connection( + asyncio.Protocol, + *addr, + ssl=client_sslctx, + server_hostname='', + ssl_handshake_timeout=10.0), + 0.5) + + with self.tcp_server(server, + max_clients=1, + backlog=1) as srv: + + with self.assertRaises(asyncio.TimeoutError): + self.loop.run_until_complete(client(srv.addr)) + + self.assertTrue(server_side_aborted) + + # Python issue #23197: cancelling a handshake must not raise an + # exception or log an error, even if the handshake failed + self.assertEqual(messages, []) + + def test_ssl_handshake_connection_lost(self): + # #246: make sure that no connection_lost() is called before + # connection_made() is called first + + client_sslctx = test_utils.simple_client_sslcontext() + + # silence error logger + self.loop.set_exception_handler(lambda loop, ctx: None) + + connection_made_called = False + connection_lost_called = False + + def server(sock): + sock.recv(1024) + # break the connection during handshake + sock.close() + + class ClientProto(asyncio.Protocol): + def connection_made(self, transport): + nonlocal connection_made_called + connection_made_called = True + + def connection_lost(self, exc): + nonlocal connection_lost_called + connection_lost_called = True + + async def client(addr): + await self.loop.create_connection( + ClientProto, + *addr, + ssl=client_sslctx, + server_hostname=''), + + with self.tcp_server(server, + max_clients=1, + backlog=1) as srv: + + with self.assertRaises(ConnectionResetError): + self.loop.run_until_complete(client(srv.addr)) + + if connection_lost_called: + if connection_made_called: + self.fail("unexpected call to connection_lost()") + else: + self.fail("unexpected call to connection_lost() without" + "calling connection_made()") + elif connection_made_called: + self.fail("unexpected call to connection_made()") + + def test_ssl_connect_accepted_socket(self): + proto = ssl.PROTOCOL_TLS_SERVER + server_context = ssl.SSLContext(proto) + server_context.load_cert_chain(test_utils.ONLYCERT, test_utils.ONLYKEY) + if hasattr(server_context, 'check_hostname'): + server_context.check_hostname = False + server_context.verify_mode = ssl.CERT_NONE + + client_context = ssl.SSLContext(proto) + if hasattr(server_context, 'check_hostname'): + client_context.check_hostname = False + client_context.verify_mode = ssl.CERT_NONE + + def test_connect_accepted_socket(self, server_ssl=None, client_ssl=None): + loop = self.loop + + class MyProto(MyBaseProto): + + def connection_lost(self, exc): + super().connection_lost(exc) + loop.call_soon(loop.stop) + + def data_received(self, data): + super().data_received(data) + self.transport.write(expected_response) + + lsock = socket.socket(socket.AF_INET) + lsock.bind(('127.0.0.1', 0)) + lsock.listen(1) + addr = lsock.getsockname() + + message = b'test data' + response = None + expected_response = b'roger' + + def client(): + nonlocal response + try: + csock = socket.socket(socket.AF_INET) + if client_ssl is not None: + csock = client_ssl.wrap_socket(csock) + csock.connect(addr) + csock.sendall(message) + response = csock.recv(99) + csock.close() + except Exception as exc: + print( + "Failure in client thread in test_connect_accepted_socket", + exc) + + thread = threading.Thread(target=client, daemon=True) + thread.start() + + conn, _ = lsock.accept() + proto = MyProto(loop=loop) + proto.loop = loop + + extras = {} + if server_ssl: + extras = dict(ssl_handshake_timeout=support.SHORT_TIMEOUT) + + f = loop.create_task( + loop.connect_accepted_socket( + (lambda: proto), conn, ssl=server_ssl, + **extras)) + loop.run_forever() + conn.close() + lsock.close() + + thread.join(1) + self.assertFalse(thread.is_alive()) + self.assertEqual(proto.state, 'CLOSED') + self.assertEqual(proto.nbytes, len(message)) + self.assertEqual(response, expected_response) + tr, _ = f.result() + + if server_ssl: + self.assertIn('SSL', tr.__class__.__name__) + + tr.close() + # let it close + self.loop.run_until_complete(asyncio.sleep(0.1)) + + def test_start_tls_client_corrupted_ssl(self): + self.loop.set_exception_handler(lambda loop, ctx: None) + + sslctx = test_utils.simple_server_sslcontext() + client_sslctx = test_utils.simple_client_sslcontext() + + def server(sock): + orig_sock = sock.dup() + try: + sock.starttls( + sslctx, + server_side=True) + sock.sendall(b'A\n') + sock.recv_all(1) + orig_sock.send(b'please corrupt the SSL connection') + except ssl.SSLError: + pass + finally: + sock.close() + orig_sock.close() + + async def client(addr): + reader, writer = await asyncio.open_connection( + *addr, + ssl=client_sslctx, + server_hostname='') + + self.assertEqual(await reader.readline(), b'A\n') + writer.write(b'B') + with self.assertRaises(ssl.SSLError): + await reader.readline() + writer.close() + try: + await self.wait_closed(writer) + except ssl.SSLError: + pass + return 'OK' + + with self.tcp_server(server, + max_clients=1, + backlog=1) as srv: + + res = self.loop.run_until_complete(client(srv.addr)) + + self.assertEqual(res, 'OK') + + def test_start_tls_client_reg_proto_1(self): + HELLO_MSG = b'1' * self.PAYLOAD_SIZE + + server_context = test_utils.simple_server_sslcontext() + client_context = test_utils.simple_client_sslcontext() + + def serve(sock): + sock.settimeout(self.TIMEOUT) + + data = sock.recv_all(len(HELLO_MSG)) + self.assertEqual(len(data), len(HELLO_MSG)) + + sock.starttls(server_context, server_side=True) + + sock.sendall(b'O') + data = sock.recv_all(len(HELLO_MSG)) + self.assertEqual(len(data), len(HELLO_MSG)) + + sock.unwrap() + sock.close() + + class ClientProto(asyncio.Protocol): + def __init__(self, on_data, on_eof): + self.on_data = on_data + self.on_eof = on_eof + self.con_made_cnt = 0 + + def connection_made(proto, tr): + proto.con_made_cnt += 1 + # Ensure connection_made gets called only once. + self.assertEqual(proto.con_made_cnt, 1) + + def data_received(self, data): + self.on_data.set_result(data) + + def eof_received(self): + self.on_eof.set_result(True) + + async def client(addr): + await asyncio.sleep(0.5) + + on_data = self.loop.create_future() + on_eof = self.loop.create_future() + + tr, proto = await self.loop.create_connection( + lambda: ClientProto(on_data, on_eof), *addr) + + tr.write(HELLO_MSG) + new_tr = await self.loop.start_tls(tr, proto, client_context) + + self.assertEqual(await on_data, b'O') + new_tr.write(HELLO_MSG) + await on_eof + + new_tr.close() + + with self.tcp_server(serve, timeout=self.TIMEOUT) as srv: + self.loop.run_until_complete( + asyncio.wait_for(client(srv.addr), + timeout=support.SHORT_TIMEOUT)) + + @unittest.expectedFailure # TODO: RUSTPYTHON; - gc.collect() doesn't release SSLContext properly + def test_create_connection_memory_leak(self): + HELLO_MSG = b'1' * self.PAYLOAD_SIZE + + server_context = self._create_server_ssl_context( + test_utils.ONLYCERT, test_utils.ONLYKEY) + client_context = self._create_client_ssl_context() + + def serve(sock): + sock.settimeout(self.TIMEOUT) + + sock.starttls(server_context, server_side=True) + + sock.sendall(b'O') + data = sock.recv_all(len(HELLO_MSG)) + self.assertEqual(len(data), len(HELLO_MSG)) + + sock.unwrap() + sock.close() + + class ClientProto(asyncio.Protocol): + def __init__(self, on_data, on_eof): + self.on_data = on_data + self.on_eof = on_eof + self.con_made_cnt = 0 + + def connection_made(proto, tr): + # XXX: We assume user stores the transport in protocol + proto.tr = tr + proto.con_made_cnt += 1 + # Ensure connection_made gets called only once. + self.assertEqual(proto.con_made_cnt, 1) + + def data_received(self, data): + self.on_data.set_result(data) + + def eof_received(self): + self.on_eof.set_result(True) + + async def client(addr): + await asyncio.sleep(0.5) + + on_data = self.loop.create_future() + on_eof = self.loop.create_future() + + tr, proto = await self.loop.create_connection( + lambda: ClientProto(on_data, on_eof), *addr, + ssl=client_context) + + self.assertEqual(await on_data, b'O') + tr.write(HELLO_MSG) + await on_eof + + tr.close() + + with self.tcp_server(serve, timeout=self.TIMEOUT) as srv: + self.loop.run_until_complete( + asyncio.wait_for(client(srv.addr), + timeout=support.SHORT_TIMEOUT)) + + # No garbage is left for SSL client from loop.create_connection, even + # if user stores the SSLTransport in corresponding protocol instance + client_context = weakref.ref(client_context) + self.assertIsNone(client_context()) + + def test_start_tls_client_buf_proto_1(self): + HELLO_MSG = b'1' * self.PAYLOAD_SIZE + + server_context = test_utils.simple_server_sslcontext() + client_context = test_utils.simple_client_sslcontext() + + client_con_made_calls = 0 + + def serve(sock): + sock.settimeout(self.TIMEOUT) + + data = sock.recv_all(len(HELLO_MSG)) + self.assertEqual(len(data), len(HELLO_MSG)) + + sock.starttls(server_context, server_side=True) + + sock.sendall(b'O') + data = sock.recv_all(len(HELLO_MSG)) + self.assertEqual(len(data), len(HELLO_MSG)) + + sock.sendall(b'2') + data = sock.recv_all(len(HELLO_MSG)) + self.assertEqual(len(data), len(HELLO_MSG)) + + sock.unwrap() + sock.close() + + class ClientProtoFirst(asyncio.BufferedProtocol): + def __init__(self, on_data): + self.on_data = on_data + self.buf = bytearray(1) + + def connection_made(self, tr): + nonlocal client_con_made_calls + client_con_made_calls += 1 + + def get_buffer(self, sizehint): + return self.buf + + def buffer_updated(self, nsize): + assert nsize == 1 + self.on_data.set_result(bytes(self.buf[:nsize])) + + def eof_received(self): + pass + + class ClientProtoSecond(asyncio.Protocol): + def __init__(self, on_data, on_eof): + self.on_data = on_data + self.on_eof = on_eof + self.con_made_cnt = 0 + + def connection_made(self, tr): + nonlocal client_con_made_calls + client_con_made_calls += 1 + + def data_received(self, data): + self.on_data.set_result(data) + + def eof_received(self): + self.on_eof.set_result(True) + + async def client(addr): + await asyncio.sleep(0.5) + + on_data1 = self.loop.create_future() + on_data2 = self.loop.create_future() + on_eof = self.loop.create_future() + + tr, proto = await self.loop.create_connection( + lambda: ClientProtoFirst(on_data1), *addr) + + tr.write(HELLO_MSG) + new_tr = await self.loop.start_tls(tr, proto, client_context) + + self.assertEqual(await on_data1, b'O') + new_tr.write(HELLO_MSG) + + new_tr.set_protocol(ClientProtoSecond(on_data2, on_eof)) + self.assertEqual(await on_data2, b'2') + new_tr.write(HELLO_MSG) + await on_eof + + new_tr.close() + + # connection_made() should be called only once -- when + # we establish connection for the first time. Start TLS + # doesn't call connection_made() on application protocols. + self.assertEqual(client_con_made_calls, 1) + + with self.tcp_server(serve, timeout=self.TIMEOUT) as srv: + self.loop.run_until_complete( + asyncio.wait_for(client(srv.addr), + timeout=self.TIMEOUT)) + + def test_start_tls_slow_client_cancel(self): + HELLO_MSG = b'1' * self.PAYLOAD_SIZE + + client_context = test_utils.simple_client_sslcontext() + server_waits_on_handshake = self.loop.create_future() + + def serve(sock): + sock.settimeout(self.TIMEOUT) + + data = sock.recv_all(len(HELLO_MSG)) + self.assertEqual(len(data), len(HELLO_MSG)) + + try: + self.loop.call_soon_threadsafe( + server_waits_on_handshake.set_result, None) + data = sock.recv_all(1024 * 1024) + except ConnectionAbortedError: + pass + finally: + sock.close() + + class ClientProto(asyncio.Protocol): + def __init__(self, on_data, on_eof): + self.on_data = on_data + self.on_eof = on_eof + self.con_made_cnt = 0 + + def connection_made(proto, tr): + proto.con_made_cnt += 1 + # Ensure connection_made gets called only once. + self.assertEqual(proto.con_made_cnt, 1) + + def data_received(self, data): + self.on_data.set_result(data) + + def eof_received(self): + self.on_eof.set_result(True) + + async def client(addr): + await asyncio.sleep(0.5) + + on_data = self.loop.create_future() + on_eof = self.loop.create_future() + + tr, proto = await self.loop.create_connection( + lambda: ClientProto(on_data, on_eof), *addr) + + tr.write(HELLO_MSG) + + await server_waits_on_handshake + + with self.assertRaises(asyncio.TimeoutError): + await asyncio.wait_for( + self.loop.start_tls(tr, proto, client_context), + 0.5) + + with self.tcp_server(serve, timeout=self.TIMEOUT) as srv: + self.loop.run_until_complete( + asyncio.wait_for(client(srv.addr), + timeout=support.SHORT_TIMEOUT)) + + def test_start_tls_server_1(self): + HELLO_MSG = b'1' * self.PAYLOAD_SIZE + + server_context = test_utils.simple_server_sslcontext() + client_context = test_utils.simple_client_sslcontext() + + def client(sock, addr): + sock.settimeout(self.TIMEOUT) + + sock.connect(addr) + data = sock.recv_all(len(HELLO_MSG)) + self.assertEqual(len(data), len(HELLO_MSG)) + + sock.starttls(client_context) + sock.sendall(HELLO_MSG) + + sock.unwrap() + sock.close() + + class ServerProto(asyncio.Protocol): + def __init__(self, on_con, on_eof, on_con_lost): + self.on_con = on_con + self.on_eof = on_eof + self.on_con_lost = on_con_lost + self.data = b'' + + def connection_made(self, tr): + self.on_con.set_result(tr) + + def data_received(self, data): + self.data += data + + def eof_received(self): + self.on_eof.set_result(1) + + def connection_lost(self, exc): + if exc is None: + self.on_con_lost.set_result(None) + else: + self.on_con_lost.set_exception(exc) + + async def main(proto, on_con, on_eof, on_con_lost): + tr = await on_con + tr.write(HELLO_MSG) + + self.assertEqual(proto.data, b'') + + new_tr = await self.loop.start_tls( + tr, proto, server_context, + server_side=True, + ssl_handshake_timeout=self.TIMEOUT) + + await on_eof + await on_con_lost + self.assertEqual(proto.data, HELLO_MSG) + new_tr.close() + + async def run_main(): + on_con = self.loop.create_future() + on_eof = self.loop.create_future() + on_con_lost = self.loop.create_future() + proto = ServerProto(on_con, on_eof, on_con_lost) + + server = await self.loop.create_server( + lambda: proto, '127.0.0.1', 0) + addr = server.sockets[0].getsockname() + + with self.tcp_client(lambda sock: client(sock, addr), + timeout=self.TIMEOUT): + await asyncio.wait_for( + main(proto, on_con, on_eof, on_con_lost), + timeout=self.TIMEOUT) + + server.close() + await server.wait_closed() + + self.loop.run_until_complete(run_main()) + + @support.bigmemtest(size=25, memuse=90*2**20, dry_run=False) + def test_create_server_ssl_over_ssl(self, size): + CNT = 0 # number of clients that were successful + TOTAL_CNT = size # total number of clients that test will create + TIMEOUT = support.LONG_TIMEOUT # timeout for this test + + A_DATA = b'A' * 1024 * BUF_MULTIPLIER + B_DATA = b'B' * 1024 * BUF_MULTIPLIER + + sslctx_1 = self._create_server_ssl_context( + test_utils.ONLYCERT, test_utils.ONLYKEY) + client_sslctx_1 = self._create_client_ssl_context() + sslctx_2 = self._create_server_ssl_context( + test_utils.ONLYCERT, test_utils.ONLYKEY) + client_sslctx_2 = self._create_client_ssl_context() + + clients = [] + + async def handle_client(reader, writer): + nonlocal CNT + + data = await reader.readexactly(len(A_DATA)) + self.assertEqual(data, A_DATA) + writer.write(b'OK') + + data = await reader.readexactly(len(B_DATA)) + self.assertEqual(data, B_DATA) + writer.writelines([b'SP', bytearray(b'A'), memoryview(b'M')]) + + await writer.drain() + writer.close() + + CNT += 1 + + class ServerProtocol(asyncio.StreamReaderProtocol): + def connection_made(self, transport): + super_ = super() + transport.pause_reading() + fut = self._loop.create_task(self._loop.start_tls( + transport, self, sslctx_2, server_side=True)) + + def cb(_): + try: + tr = fut.result() + except Exception as ex: + super_.connection_lost(ex) + else: + super_.connection_made(tr) + fut.add_done_callback(cb) + + def server_protocol_factory(): + reader = asyncio.StreamReader() + protocol = ServerProtocol(reader, handle_client) + return protocol + + async def test_client(addr): + fut = asyncio.Future() + + def prog(sock): + try: + sock.connect(addr) + sock.starttls(client_sslctx_1) + + # because wrap_socket() doesn't work correctly on + # SSLSocket, we have to do the 2nd level SSL manually + incoming = ssl.MemoryBIO() + outgoing = ssl.MemoryBIO() + sslobj = client_sslctx_2.wrap_bio(incoming, outgoing) + + def do(func, *args): + while True: + try: + rv = func(*args) + break + except ssl.SSLWantReadError: + if outgoing.pending: + sock.send(outgoing.read()) + incoming.write(sock.recv(65536)) + if outgoing.pending: + sock.send(outgoing.read()) + return rv + + do(sslobj.do_handshake) + + do(sslobj.write, A_DATA) + data = do(sslobj.read, 2) + self.assertEqual(data, b'OK') + + do(sslobj.write, B_DATA) + data = b'' + while True: + chunk = do(sslobj.read, 4) + if not chunk: + break + data += chunk + self.assertEqual(data, b'SPAM') + + do(sslobj.unwrap) + sock.close() + + except Exception as ex: + self.loop.call_soon_threadsafe(fut.set_exception, ex) + sock.close() + else: + self.loop.call_soon_threadsafe(fut.set_result, None) + + client = self.tcp_client(prog) + client.start() + clients.append(client) + + await fut + + async def start_server(): + extras = {} + + srv = await self.loop.create_server( + server_protocol_factory, + '127.0.0.1', 0, + family=socket.AF_INET, + ssl=sslctx_1, + **extras) + + try: + srv_socks = srv.sockets + self.assertTrue(srv_socks) + + addr = srv_socks[0].getsockname() + + tasks = [] + for _ in range(TOTAL_CNT): + tasks.append(test_client(addr)) + + await asyncio.wait_for(asyncio.gather(*tasks), TIMEOUT) + + finally: + self.loop.call_soon(srv.close) + await srv.wait_closed() + + with self._silence_eof_received_warning(): + self.loop.run_until_complete(start_server()) + + self.assertEqual(CNT, TOTAL_CNT) + + for client in clients: + client.stop() + + def test_shutdown_cleanly(self): + CNT = 0 + TOTAL_CNT = 25 + + A_DATA = b'A' * 1024 * BUF_MULTIPLIER + + sslctx = self._create_server_ssl_context( + test_utils.ONLYCERT, test_utils.ONLYKEY) + client_sslctx = self._create_client_ssl_context() + + def server(sock): + sock.starttls( + sslctx, + server_side=True) + + data = sock.recv_all(len(A_DATA)) + self.assertEqual(data, A_DATA) + sock.send(b'OK') + + sock.unwrap() + + sock.close() + + async def client(addr): + extras = {} + extras = dict(ssl_handshake_timeout=support.SHORT_TIMEOUT) + + reader, writer = await asyncio.open_connection( + *addr, + ssl=client_sslctx, + server_hostname='', + **extras) + + writer.write(A_DATA) + self.assertEqual(await reader.readexactly(2), b'OK') + + self.assertEqual(await reader.read(), b'') + + nonlocal CNT + CNT += 1 + + writer.close() + await self.wait_closed(writer) + + def run(coro): + nonlocal CNT + CNT = 0 + + async def _gather(*tasks): + return await asyncio.gather(*tasks) + + with self.tcp_server(server, + max_clients=TOTAL_CNT, + backlog=TOTAL_CNT) as srv: + tasks = [] + for _ in range(TOTAL_CNT): + tasks.append(coro(srv.addr)) + + self.loop.run_until_complete( + _gather(*tasks)) + + self.assertEqual(CNT, TOTAL_CNT) + + with self._silence_eof_received_warning(): + run(client) + + def test_flush_before_shutdown(self): + CHUNK = 1024 * 128 + SIZE = 32 + + sslctx = self._create_server_ssl_context( + test_utils.ONLYCERT, test_utils.ONLYKEY) + client_sslctx = self._create_client_ssl_context() + + future = None + + def server(sock): + sock.starttls(sslctx, server_side=True) + self.assertEqual(sock.recv_all(4), b'ping') + sock.send(b'pong') + time.sleep(0.5) # hopefully stuck the TCP buffer + data = sock.recv_all(CHUNK * SIZE) + self.assertEqual(len(data), CHUNK * SIZE) + sock.close() + + def run(meth): + def wrapper(sock): + try: + meth(sock) + except Exception as ex: + self.loop.call_soon_threadsafe(future.set_exception, ex) + else: + self.loop.call_soon_threadsafe(future.set_result, None) + return wrapper + + async def client(addr): + nonlocal future + future = self.loop.create_future() + reader, writer = await asyncio.open_connection( + *addr, + ssl=client_sslctx, + server_hostname='') + sslprotocol = writer.transport._ssl_protocol + writer.write(b'ping') + data = await reader.readexactly(4) + self.assertEqual(data, b'pong') + + sslprotocol.pause_writing() + for _ in range(SIZE): + writer.write(b'x' * CHUNK) + + writer.close() + sslprotocol.resume_writing() + + await self.wait_closed(writer) + try: + data = await reader.read() + self.assertEqual(data, b'') + except ConnectionResetError: + pass + await future + + with self.tcp_server(run(server)) as srv: + self.loop.run_until_complete(client(srv.addr)) + + def test_remote_shutdown_receives_trailing_data(self): + CHUNK = 1024 * 128 + SIZE = 32 + + sslctx = self._create_server_ssl_context( + test_utils.ONLYCERT, + test_utils.ONLYKEY + ) + client_sslctx = self._create_client_ssl_context() + future = None + + def server(sock): + incoming = ssl.MemoryBIO() + outgoing = ssl.MemoryBIO() + sslobj = sslctx.wrap_bio(incoming, outgoing, server_side=True) + + while True: + try: + sslobj.do_handshake() + except ssl.SSLWantReadError: + if outgoing.pending: + sock.send(outgoing.read()) + incoming.write(sock.recv(16384)) + else: + if outgoing.pending: + sock.send(outgoing.read()) + break + + while True: + try: + data = sslobj.read(4) + except ssl.SSLWantReadError: + incoming.write(sock.recv(16384)) + else: + break + + self.assertEqual(data, b'ping') + sslobj.write(b'pong') + sock.send(outgoing.read()) + + time.sleep(0.2) # wait for the peer to fill its backlog + + # send close_notify but don't wait for response + with self.assertRaises(ssl.SSLWantReadError): + sslobj.unwrap() + sock.send(outgoing.read()) + + # should receive all data + data_len = 0 + while True: + try: + chunk = len(sslobj.read(16384)) + data_len += chunk + except ssl.SSLWantReadError: + incoming.write(sock.recv(16384)) + except ssl.SSLZeroReturnError: + break + + self.assertEqual(data_len, CHUNK * SIZE) + + # verify that close_notify is received + sslobj.unwrap() + + sock.close() + + def eof_server(sock): + sock.starttls(sslctx, server_side=True) + self.assertEqual(sock.recv_all(4), b'ping') + sock.send(b'pong') + + time.sleep(0.2) # wait for the peer to fill its backlog + + # send EOF + sock.shutdown(socket.SHUT_WR) + + # should receive all data + data = sock.recv_all(CHUNK * SIZE) + self.assertEqual(len(data), CHUNK * SIZE) + + sock.close() + + async def client(addr): + nonlocal future + future = self.loop.create_future() + + reader, writer = await asyncio.open_connection( + *addr, + ssl=client_sslctx, + server_hostname='') + writer.write(b'ping') + data = await reader.readexactly(4) + self.assertEqual(data, b'pong') + + # fill write backlog in a hacky way - renegotiation won't help + for _ in range(SIZE): + writer.transport._test__append_write_backlog(b'x' * CHUNK) + + try: + data = await reader.read() + self.assertEqual(data, b'') + except (BrokenPipeError, ConnectionResetError): + pass + + await future + + writer.close() + await self.wait_closed(writer) + + def run(meth): + def wrapper(sock): + try: + meth(sock) + except Exception as ex: + self.loop.call_soon_threadsafe(future.set_exception, ex) + else: + self.loop.call_soon_threadsafe(future.set_result, None) + return wrapper + + with self.tcp_server(run(server)) as srv: + self.loop.run_until_complete(client(srv.addr)) + + with self.tcp_server(run(eof_server)) as srv: + self.loop.run_until_complete(client(srv.addr)) + + def test_remote_shutdown_receives_trailing_data_on_slow_socket(self): + # This test is the same as test_remote_shutdown_receives_trailing_data, + # except it simulates a socket that is not able to write data in time, + # thus triggering different code path in _SelectorSocketTransport. + # This triggers bug gh-115514, also tested using mocks in + # test.test_asyncio.test_selector_events.SelectorSocketTransportTests.test_write_buffer_after_close + # The slow path is triggered here by setting SO_SNDBUF, see code and comment below. + + CHUNK = 1024 * 128 + SIZE = 32 + + sslctx = self._create_server_ssl_context( + test_utils.ONLYCERT, + test_utils.ONLYKEY + ) + client_sslctx = self._create_client_ssl_context() + future = None + + def server(sock): + incoming = ssl.MemoryBIO() + outgoing = ssl.MemoryBIO() + sslobj = sslctx.wrap_bio(incoming, outgoing, server_side=True) + + while True: + try: + sslobj.do_handshake() + except ssl.SSLWantReadError: + if outgoing.pending: + sock.send(outgoing.read()) + incoming.write(sock.recv(16384)) + else: + if outgoing.pending: + sock.send(outgoing.read()) + break + + while True: + try: + data = sslobj.read(4) + except ssl.SSLWantReadError: + incoming.write(sock.recv(16384)) + else: + break + + self.assertEqual(data, b'ping') + sslobj.write(b'pong') + sock.send(outgoing.read()) + + time.sleep(0.2) # wait for the peer to fill its backlog + + # send close_notify but don't wait for response + with self.assertRaises(ssl.SSLWantReadError): + sslobj.unwrap() + sock.send(outgoing.read()) + + # should receive all data + data_len = 0 + while True: + try: + chunk = len(sslobj.read(16384)) + data_len += chunk + except ssl.SSLWantReadError: + incoming.write(sock.recv(16384)) + except ssl.SSLZeroReturnError: + break + + self.assertEqual(data_len, CHUNK * SIZE*2) + + # verify that close_notify is received + sslobj.unwrap() + + sock.close() + + def eof_server(sock): + sock.starttls(sslctx, server_side=True) + self.assertEqual(sock.recv_all(4), b'ping') + sock.send(b'pong') + + time.sleep(0.2) # wait for the peer to fill its backlog + + # send EOF + sock.shutdown(socket.SHUT_WR) + + # should receive all data + data = sock.recv_all(CHUNK * SIZE) + self.assertEqual(len(data), CHUNK * SIZE) + + sock.close() + + async def client(addr): + nonlocal future + future = self.loop.create_future() + + reader, writer = await asyncio.open_connection( + *addr, + ssl=client_sslctx, + server_hostname='') + writer.write(b'ping') + data = await reader.readexactly(4) + self.assertEqual(data, b'pong') + + # fill write backlog in a hacky way - renegotiation won't help + for _ in range(SIZE*2): + writer.transport._test__append_write_backlog(b'x' * CHUNK) + + try: + data = await reader.read() + self.assertEqual(data, b'') + except (BrokenPipeError, ConnectionResetError): + pass + + # Make sure _SelectorSocketTransport enters the delayed write + # path in its `write` method by wrapping socket in a fake class + # that acts as if there is not enough space in socket buffer. + # This triggers bug gh-115514, also tested using mocks in + # test.test_asyncio.test_selector_events.SelectorSocketTransportTests.test_write_buffer_after_close + socket_transport = writer.transport._ssl_protocol._transport + + class SocketWrapper: + def __init__(self, sock) -> None: + self.sock = sock + + def __getattr__(self, name): + return getattr(self.sock, name) + + def send(self, data): + # Fake that our write buffer is full, send only half + to_send = len(data)//2 + return self.sock.send(data[:to_send]) + + def _fake_full_write_buffer(data): + if socket_transport._read_ready_cb is None and not isinstance(socket_transport._sock, SocketWrapper): + socket_transport._sock = SocketWrapper(socket_transport._sock) + return unittest.mock.DEFAULT + + with unittest.mock.patch.object( + socket_transport, "write", + wraps=socket_transport.write, + side_effect=_fake_full_write_buffer + ): + await future + + writer.close() + await self.wait_closed(writer) + + def run(meth): + def wrapper(sock): + try: + meth(sock) + except Exception as ex: + self.loop.call_soon_threadsafe(future.set_exception, ex) + else: + self.loop.call_soon_threadsafe(future.set_result, None) + return wrapper + + with self.tcp_server(run(server)) as srv: + self.loop.run_until_complete(client(srv.addr)) + + with self.tcp_server(run(eof_server)) as srv: + self.loop.run_until_complete(client(srv.addr)) + + def test_connect_timeout_warning(self): + s = socket.socket(socket.AF_INET) + s.bind(('127.0.0.1', 0)) + addr = s.getsockname() + + async def test(): + try: + await asyncio.wait_for( + self.loop.create_connection(asyncio.Protocol, + *addr, ssl=True), + 0.1) + except (ConnectionRefusedError, asyncio.TimeoutError): + pass + else: + self.fail('TimeoutError is not raised') + + with s: + try: + with self.assertWarns(ResourceWarning) as cm: + self.loop.run_until_complete(test()) + gc.collect() + gc.collect() + gc.collect() + except AssertionError as e: + self.assertEqual(str(e), 'ResourceWarning not triggered') + else: + self.fail('Unexpected ResourceWarning: {}'.format(cm.warning)) + + @unittest.expectedFailure # TODO: RUSTPYTHON; - gc.collect() doesn't release SSLContext properly + def test_handshake_timeout_handler_leak(self): + s = socket.socket(socket.AF_INET) + s.bind(('127.0.0.1', 0)) + s.listen(1) + addr = s.getsockname() + + async def test(ctx): + try: + await asyncio.wait_for( + self.loop.create_connection(asyncio.Protocol, *addr, + ssl=ctx), + 0.1) + except (ConnectionRefusedError, asyncio.TimeoutError): + pass + else: + self.fail('TimeoutError is not raised') + + with s: + ctx = ssl.create_default_context() + self.loop.run_until_complete(test(ctx)) + ctx = weakref.ref(ctx) + + # SSLProtocol should be DECREF to 0 + self.assertIsNone(ctx()) + + @unittest.expectedFailure # TODO: RUSTPYTHON; - gc.collect() doesn't release SSLContext properly + def test_shutdown_timeout_handler_leak(self): + loop = self.loop + + def server(sock): + sslctx = self._create_server_ssl_context( + test_utils.ONLYCERT, + test_utils.ONLYKEY + ) + sock = sslctx.wrap_socket(sock, server_side=True) + sock.recv(32) + sock.close() + + class Protocol(asyncio.Protocol): + def __init__(self): + self.fut = asyncio.Future(loop=loop) + + def connection_lost(self, exc): + self.fut.set_result(None) + + async def client(addr, ctx): + tr, pr = await loop.create_connection(Protocol, *addr, ssl=ctx) + tr.close() + await pr.fut + + with self.tcp_server(server) as srv: + ctx = self._create_client_ssl_context() + loop.run_until_complete(client(srv.addr, ctx)) + ctx = weakref.ref(ctx) + + # asyncio has no shutdown timeout, but it ends up with a circular + # reference loop - not ideal (introduces gc glitches), but at least + # not leaking + gc.collect() + gc.collect() + gc.collect() + + # SSLProtocol should be DECREF to 0 + self.assertIsNone(ctx()) + + def test_shutdown_timeout_handler_not_set(self): + loop = self.loop + eof = asyncio.Event() + extra = None + + def server(sock): + sslctx = self._create_server_ssl_context( + test_utils.ONLYCERT, + test_utils.ONLYKEY + ) + sock = sslctx.wrap_socket(sock, server_side=True) + sock.send(b'hello') + assert sock.recv(1024) == b'world' + sock.send(b'extra bytes') + # sending EOF here + sock.shutdown(socket.SHUT_WR) + loop.call_soon_threadsafe(eof.set) + # make sure we have enough time to reproduce the issue + assert sock.recv(1024) == b'' + sock.close() + + class Protocol(asyncio.Protocol): + def __init__(self): + self.fut = asyncio.Future(loop=loop) + self.transport = None + + def connection_made(self, transport): + self.transport = transport + + def data_received(self, data): + if data == b'hello': + self.transport.write(b'world') + # pause reading would make incoming data stay in the sslobj + self.transport.pause_reading() + else: + nonlocal extra + extra = data + + def connection_lost(self, exc): + if exc is None: + self.fut.set_result(None) + else: + self.fut.set_exception(exc) + + async def client(addr): + ctx = self._create_client_ssl_context() + tr, pr = await loop.create_connection(Protocol, *addr, ssl=ctx) + await eof.wait() + tr.resume_reading() + await pr.fut + tr.close() + assert extra == b'extra bytes' + + with self.tcp_server(server) as srv: + loop.run_until_complete(client(srv.addr)) + + +############################################################################### +# Socket Testing Utilities +############################################################################### + + +class TestSocketWrapper: + + def __init__(self, sock): + self.__sock = sock + + def recv_all(self, n): + buf = b'' + while len(buf) < n: + data = self.recv(n - len(buf)) + if data == b'': + raise ConnectionAbortedError + buf += data + return buf + + def starttls(self, ssl_context, *, + server_side=False, + server_hostname=None, + do_handshake_on_connect=True): + + assert isinstance(ssl_context, ssl.SSLContext) + + ssl_sock = ssl_context.wrap_socket( + self.__sock, server_side=server_side, + server_hostname=server_hostname, + do_handshake_on_connect=do_handshake_on_connect) + + if server_side: + ssl_sock.do_handshake() + + self.__sock.close() + self.__sock = ssl_sock + + def __getattr__(self, name): + return getattr(self.__sock, name) + + def __repr__(self): + return '<{} {!r}>'.format(type(self).__name__, self.__sock) + + +class SocketThread(threading.Thread): + + def stop(self): + self._active = False + self.join() + + def __enter__(self): + self.start() + return self + + def __exit__(self, *exc): + self.stop() + + +class TestThreadedClient(SocketThread): + + def __init__(self, test, sock, prog, timeout): + threading.Thread.__init__(self, None, None, 'test-client') + self.daemon = True + + self._timeout = timeout + self._sock = sock + self._active = True + self._prog = prog + self._test = test + + def run(self): + try: + self._prog(TestSocketWrapper(self._sock)) + except (KeyboardInterrupt, SystemExit): + raise + except BaseException as ex: + self._test._abort_socket_test(ex) + + +class TestThreadedServer(SocketThread): + + def __init__(self, test, sock, prog, timeout, max_clients): + threading.Thread.__init__(self, None, None, 'test-server') + self.daemon = True + + self._clients = 0 + self._finished_clients = 0 + self._max_clients = max_clients + self._timeout = timeout + self._sock = sock + self._active = True + + self._prog = prog + + self._s1, self._s2 = socket.socketpair() + self._s1.setblocking(False) + + self._test = test + + def stop(self): + try: + if self._s2 and self._s2.fileno() != -1: + try: + self._s2.send(b'stop') + except OSError: + pass + finally: + super().stop() + self._sock.close() + self._s1.close() + self._s2.close() + + def run(self): + self._sock.setblocking(False) + self._run() + + def _run(self): + while self._active: + if self._clients >= self._max_clients: + return + + r, w, x = select.select( + [self._sock, self._s1], [], [], self._timeout) + + if self._s1 in r: + return + + if self._sock in r: + try: + conn, addr = self._sock.accept() + except BlockingIOError: + continue + except socket.timeout: + if not self._active: + return + else: + raise + else: + self._clients += 1 + conn.settimeout(self._timeout) + try: + with conn: + self._handle_client(conn) + except (KeyboardInterrupt, SystemExit): + raise + except BaseException as ex: + self._active = False + try: + raise + finally: + self._test._abort_socket_test(ex) + + def _handle_client(self, sock): + self._prog(TestSocketWrapper(sock)) + + @property + def addr(self): + return self._sock.getsockname() diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py new file mode 100644 index 00000000000..7ab6e1511d7 --- /dev/null +++ b/Lib/test/test_asyncio/test_sslproto.py @@ -0,0 +1,846 @@ +"""Tests for asyncio/sslproto.py.""" + +import logging +import socket +import unittest +import weakref +from test import support +from test.support import socket_helper +from unittest import mock +try: + import ssl +except ImportError: + ssl = None + +import asyncio +from asyncio import log +from asyncio import protocols +from asyncio import sslproto +from test.test_asyncio import utils as test_utils +from test.test_asyncio import functional as func_tests + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +@unittest.skipIf(ssl is None, 'No ssl module') +class SslProtoHandshakeTests(test_utils.TestCase): + + def setUp(self): + super().setUp() + self.loop = asyncio.new_event_loop() + self.set_event_loop(self.loop) + + def ssl_protocol(self, *, waiter=None, proto=None): + sslcontext = test_utils.dummy_ssl_context() + if proto is None: # app protocol + proto = asyncio.Protocol() + ssl_proto = sslproto.SSLProtocol(self.loop, proto, sslcontext, waiter, + ssl_handshake_timeout=0.1) + self.assertIs(ssl_proto._app_transport.get_protocol(), proto) + self.addCleanup(ssl_proto._app_transport.close) + return ssl_proto + + def connection_made(self, ssl_proto, *, do_handshake=None): + transport = mock.Mock() + sslobj = mock.Mock() + # emulate reading decompressed data + sslobj.read.side_effect = ssl.SSLWantReadError + sslobj.write.side_effect = ssl.SSLWantReadError + if do_handshake is not None: + sslobj.do_handshake = do_handshake + ssl_proto._sslobj = sslobj + ssl_proto.connection_made(transport) + return transport + + def test_handshake_timeout_zero(self): + sslcontext = test_utils.dummy_ssl_context() + app_proto = mock.Mock() + waiter = mock.Mock() + with self.assertRaisesRegex(ValueError, 'a positive number'): + sslproto.SSLProtocol(self.loop, app_proto, sslcontext, waiter, + ssl_handshake_timeout=0) + + def test_handshake_timeout_negative(self): + sslcontext = test_utils.dummy_ssl_context() + app_proto = mock.Mock() + waiter = mock.Mock() + with self.assertRaisesRegex(ValueError, 'a positive number'): + sslproto.SSLProtocol(self.loop, app_proto, sslcontext, waiter, + ssl_handshake_timeout=-10) + + def test_eof_received_waiter(self): + waiter = self.loop.create_future() + ssl_proto = self.ssl_protocol(waiter=waiter) + self.connection_made( + ssl_proto, + do_handshake=mock.Mock(side_effect=ssl.SSLWantReadError) + ) + ssl_proto.eof_received() + test_utils.run_briefly(self.loop) + self.assertIsInstance(waiter.exception(), ConnectionResetError) + + def test_fatal_error_no_name_error(self): + # From issue #363. + # _fatal_error() generates a NameError if sslproto.py + # does not import base_events. + waiter = self.loop.create_future() + ssl_proto = self.ssl_protocol(waiter=waiter) + # Temporarily turn off error logging so as not to spoil test output. + log_level = log.logger.getEffectiveLevel() + log.logger.setLevel(logging.FATAL) + try: + ssl_proto._fatal_error(None) + finally: + # Restore error logging. + log.logger.setLevel(log_level) + + def test_connection_lost(self): + # From issue #472. + # yield from waiter hang if lost_connection was called. + waiter = self.loop.create_future() + ssl_proto = self.ssl_protocol(waiter=waiter) + self.connection_made( + ssl_proto, + do_handshake=mock.Mock(side_effect=ssl.SSLWantReadError) + ) + ssl_proto.connection_lost(ConnectionAbortedError) + test_utils.run_briefly(self.loop) + self.assertIsInstance(waiter.exception(), ConnectionAbortedError) + + def test_connection_lost_when_busy(self): + # gh-118950: SSLProtocol.connection_lost not being called when OSError + # is thrown on asyncio.write. + sock = mock.Mock() + sock.fileno = mock.Mock(return_value=12345) + sock.send = mock.Mock(side_effect=BrokenPipeError) + + # construct StreamWriter chain that contains loop dependant logic this emulates + # what _make_ssl_transport() does in BaseSelectorEventLoop + reader = asyncio.StreamReader(limit=2 ** 16, loop=self.loop) + protocol = asyncio.StreamReaderProtocol(reader, loop=self.loop) + ssl_proto = self.ssl_protocol(proto=protocol) + + # emulate reading decompressed data + sslobj = mock.Mock() + sslobj.read.side_effect = ssl.SSLWantReadError + sslobj.write.side_effect = ssl.SSLWantReadError + ssl_proto._sslobj = sslobj + + # emulate outgoing data + data = b'An interesting message' + + outgoing = mock.Mock() + outgoing.read = mock.Mock(return_value=data) + outgoing.pending = len(data) + ssl_proto._outgoing = outgoing + + # use correct socket transport to initialize the SSLProtocol + self.loop._make_socket_transport(sock, ssl_proto) + + transport = ssl_proto._app_transport + writer = asyncio.StreamWriter(transport, protocol, reader, self.loop) + + async def main(): + # writes data to transport + async def write(): + writer.write(data) + await writer.drain() + + # try to write for the first time + await write() + # try to write for the second time, this raises as the connection_lost + # callback should be done with error + with self.assertRaises(ConnectionResetError): + await write() + + self.loop.run_until_complete(main()) + + def test_close_during_handshake(self): + # bpo-29743 Closing transport during handshake process leaks socket + waiter = self.loop.create_future() + ssl_proto = self.ssl_protocol(waiter=waiter) + + transport = self.connection_made( + ssl_proto, + do_handshake=mock.Mock(side_effect=ssl.SSLWantReadError) + ) + test_utils.run_briefly(self.loop) + + ssl_proto._app_transport.close() + self.assertTrue(transport._force_close.called) + + def test_close_during_ssl_over_ssl(self): + # gh-113214: passing exceptions from the inner wrapped SSL protocol to the + # shim transport provided by the outer SSL protocol should not raise + # attribute errors + outer = self.ssl_protocol(proto=self.ssl_protocol()) + self.connection_made(outer) + # Closing the outer app transport should not raise an exception + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + outer._app_transport.close() + self.assertEqual(messages, []) + + def test_get_extra_info_on_closed_connection(self): + waiter = self.loop.create_future() + ssl_proto = self.ssl_protocol(waiter=waiter) + self.assertIsNone(ssl_proto._get_extra_info('socket')) + default = object() + self.assertIs(ssl_proto._get_extra_info('socket', default), default) + self.connection_made(ssl_proto) + self.assertIsNotNone(ssl_proto._get_extra_info('socket')) + ssl_proto.connection_lost(None) + self.assertIsNone(ssl_proto._get_extra_info('socket')) + + def test_set_new_app_protocol(self): + waiter = self.loop.create_future() + ssl_proto = self.ssl_protocol(waiter=waiter) + new_app_proto = asyncio.Protocol() + ssl_proto._app_transport.set_protocol(new_app_proto) + self.assertIs(ssl_proto._app_transport.get_protocol(), new_app_proto) + self.assertIs(ssl_proto._app_protocol, new_app_proto) + + def test_data_received_after_closing(self): + ssl_proto = self.ssl_protocol() + self.connection_made(ssl_proto) + transp = ssl_proto._app_transport + + transp.close() + + # should not raise + self.assertIsNone(ssl_proto.buffer_updated(5)) + + def test_write_after_closing(self): + ssl_proto = self.ssl_protocol() + self.connection_made(ssl_proto) + transp = ssl_proto._app_transport + transp.close() + + # should not raise + self.assertIsNone(transp.write(b'data')) + + +############################################################################## +# Start TLS Tests +############################################################################## + + +class BaseStartTLS(func_tests.FunctionalTestCaseMixin): + + PAYLOAD_SIZE = 1024 * 100 + TIMEOUT = support.LONG_TIMEOUT + + def new_loop(self): + raise NotImplementedError + + def test_buf_feed_data(self): + + class Proto(asyncio.BufferedProtocol): + + def __init__(self, bufsize, usemv): + self.buf = bytearray(bufsize) + self.mv = memoryview(self.buf) + self.data = b'' + self.usemv = usemv + + def get_buffer(self, sizehint): + if self.usemv: + return self.mv + else: + return self.buf + + def buffer_updated(self, nsize): + if self.usemv: + self.data += self.mv[:nsize] + else: + self.data += self.buf[:nsize] + + for usemv in [False, True]: + proto = Proto(1, usemv) + protocols._feed_data_to_buffered_proto(proto, b'12345') + self.assertEqual(proto.data, b'12345') + + proto = Proto(2, usemv) + protocols._feed_data_to_buffered_proto(proto, b'12345') + self.assertEqual(proto.data, b'12345') + + proto = Proto(2, usemv) + protocols._feed_data_to_buffered_proto(proto, b'1234') + self.assertEqual(proto.data, b'1234') + + proto = Proto(4, usemv) + protocols._feed_data_to_buffered_proto(proto, b'1234') + self.assertEqual(proto.data, b'1234') + + proto = Proto(100, usemv) + protocols._feed_data_to_buffered_proto(proto, b'12345') + self.assertEqual(proto.data, b'12345') + + proto = Proto(0, usemv) + with self.assertRaisesRegex(RuntimeError, 'empty buffer'): + protocols._feed_data_to_buffered_proto(proto, b'12345') + + @unittest.expectedFailure # TODO: RUSTPYTHON; - gc.collect() doesn't release SSLContext properly + def test_start_tls_client_reg_proto_1(self): + HELLO_MSG = b'1' * self.PAYLOAD_SIZE + + server_context = test_utils.simple_server_sslcontext() + client_context = test_utils.simple_client_sslcontext() + + def serve(sock): + sock.settimeout(self.TIMEOUT) + + data = sock.recv_all(len(HELLO_MSG)) + self.assertEqual(len(data), len(HELLO_MSG)) + + sock.start_tls(server_context, server_side=True) + + sock.sendall(b'O') + data = sock.recv_all(len(HELLO_MSG)) + self.assertEqual(len(data), len(HELLO_MSG)) + + sock.shutdown(socket.SHUT_RDWR) + sock.close() + + class ClientProto(asyncio.Protocol): + def __init__(self, on_data, on_eof): + self.on_data = on_data + self.on_eof = on_eof + self.con_made_cnt = 0 + + def connection_made(proto, tr): + proto.con_made_cnt += 1 + # Ensure connection_made gets called only once. + self.assertEqual(proto.con_made_cnt, 1) + + def data_received(self, data): + self.on_data.set_result(data) + + def eof_received(self): + self.on_eof.set_result(True) + + async def client(addr): + await asyncio.sleep(0.5) + + on_data = self.loop.create_future() + on_eof = self.loop.create_future() + + tr, proto = await self.loop.create_connection( + lambda: ClientProto(on_data, on_eof), *addr) + + tr.write(HELLO_MSG) + new_tr = await self.loop.start_tls(tr, proto, client_context) + + self.assertEqual(await on_data, b'O') + new_tr.write(HELLO_MSG) + await on_eof + + new_tr.close() + + with self.tcp_server(serve, timeout=self.TIMEOUT) as srv: + self.loop.run_until_complete( + asyncio.wait_for(client(srv.addr), + timeout=support.SHORT_TIMEOUT)) + + # No garbage is left if SSL is closed uncleanly + client_context = weakref.ref(client_context) + support.gc_collect() + self.assertIsNone(client_context()) + + @unittest.expectedFailure # TODO: RUSTPYTHON; - gc.collect() doesn't release SSLContext properly + def test_create_connection_memory_leak(self): + HELLO_MSG = b'1' * self.PAYLOAD_SIZE + + server_context = test_utils.simple_server_sslcontext() + client_context = test_utils.simple_client_sslcontext() + + def serve(sock): + sock.settimeout(self.TIMEOUT) + + sock.start_tls(server_context, server_side=True) + + sock.sendall(b'O') + data = sock.recv_all(len(HELLO_MSG)) + self.assertEqual(len(data), len(HELLO_MSG)) + + sock.shutdown(socket.SHUT_RDWR) + sock.close() + + class ClientProto(asyncio.Protocol): + def __init__(self, on_data, on_eof): + self.on_data = on_data + self.on_eof = on_eof + self.con_made_cnt = 0 + + def connection_made(proto, tr): + # XXX: We assume user stores the transport in protocol + proto.tr = tr + proto.con_made_cnt += 1 + # Ensure connection_made gets called only once. + self.assertEqual(proto.con_made_cnt, 1) + + def data_received(self, data): + self.on_data.set_result(data) + + def eof_received(self): + self.on_eof.set_result(True) + + async def client(addr): + await asyncio.sleep(0.5) + + on_data = self.loop.create_future() + on_eof = self.loop.create_future() + + tr, proto = await self.loop.create_connection( + lambda: ClientProto(on_data, on_eof), *addr, + ssl=client_context) + + self.assertEqual(await on_data, b'O') + tr.write(HELLO_MSG) + await on_eof + + tr.close() + + with self.tcp_server(serve, timeout=self.TIMEOUT) as srv: + self.loop.run_until_complete( + asyncio.wait_for(client(srv.addr), + timeout=support.SHORT_TIMEOUT)) + + # No garbage is left for SSL client from loop.create_connection, even + # if user stores the SSLTransport in corresponding protocol instance + client_context = weakref.ref(client_context) + support.gc_collect() + self.assertIsNone(client_context()) + + @socket_helper.skip_if_tcp_blackhole + def test_start_tls_client_buf_proto_1(self): + HELLO_MSG = b'1' * self.PAYLOAD_SIZE + + server_context = test_utils.simple_server_sslcontext() + client_context = test_utils.simple_client_sslcontext() + client_con_made_calls = 0 + + def serve(sock): + sock.settimeout(self.TIMEOUT) + + data = sock.recv_all(len(HELLO_MSG)) + self.assertEqual(len(data), len(HELLO_MSG)) + + sock.start_tls(server_context, server_side=True) + + sock.sendall(b'O') + data = sock.recv_all(len(HELLO_MSG)) + self.assertEqual(len(data), len(HELLO_MSG)) + + sock.sendall(b'2') + data = sock.recv_all(len(HELLO_MSG)) + self.assertEqual(len(data), len(HELLO_MSG)) + + sock.shutdown(socket.SHUT_RDWR) + sock.close() + + class ClientProtoFirst(asyncio.BufferedProtocol): + def __init__(self, on_data): + self.on_data = on_data + self.buf = bytearray(1) + + def connection_made(self, tr): + nonlocal client_con_made_calls + client_con_made_calls += 1 + + def get_buffer(self, sizehint): + return self.buf + + def buffer_updated(slf, nsize): + self.assertEqual(nsize, 1) + slf.on_data.set_result(bytes(slf.buf[:nsize])) + + class ClientProtoSecond(asyncio.Protocol): + def __init__(self, on_data, on_eof): + self.on_data = on_data + self.on_eof = on_eof + self.con_made_cnt = 0 + + def connection_made(self, tr): + nonlocal client_con_made_calls + client_con_made_calls += 1 + + def data_received(self, data): + self.on_data.set_result(data) + + def eof_received(self): + self.on_eof.set_result(True) + + async def client(addr): + await asyncio.sleep(0.5) + + on_data1 = self.loop.create_future() + on_data2 = self.loop.create_future() + on_eof = self.loop.create_future() + + tr, proto = await self.loop.create_connection( + lambda: ClientProtoFirst(on_data1), *addr) + + tr.write(HELLO_MSG) + new_tr = await self.loop.start_tls(tr, proto, client_context) + + self.assertEqual(await on_data1, b'O') + new_tr.write(HELLO_MSG) + + new_tr.set_protocol(ClientProtoSecond(on_data2, on_eof)) + self.assertEqual(await on_data2, b'2') + new_tr.write(HELLO_MSG) + await on_eof + + new_tr.close() + + # connection_made() should be called only once -- when + # we establish connection for the first time. Start TLS + # doesn't call connection_made() on application protocols. + self.assertEqual(client_con_made_calls, 1) + + with self.tcp_server(serve, timeout=self.TIMEOUT) as srv: + self.loop.run_until_complete( + asyncio.wait_for(client(srv.addr), + timeout=self.TIMEOUT)) + + def test_start_tls_slow_client_cancel(self): + HELLO_MSG = b'1' * self.PAYLOAD_SIZE + + client_context = test_utils.simple_client_sslcontext() + server_waits_on_handshake = self.loop.create_future() + + def serve(sock): + sock.settimeout(self.TIMEOUT) + + data = sock.recv_all(len(HELLO_MSG)) + self.assertEqual(len(data), len(HELLO_MSG)) + + try: + self.loop.call_soon_threadsafe( + server_waits_on_handshake.set_result, None) + data = sock.recv_all(1024 * 1024) + except ConnectionAbortedError: + pass + finally: + sock.close() + + class ClientProto(asyncio.Protocol): + def __init__(self, on_data, on_eof): + self.on_data = on_data + self.on_eof = on_eof + self.con_made_cnt = 0 + + def connection_made(proto, tr): + proto.con_made_cnt += 1 + # Ensure connection_made gets called only once. + self.assertEqual(proto.con_made_cnt, 1) + + def data_received(self, data): + self.on_data.set_result(data) + + def eof_received(self): + self.on_eof.set_result(True) + + async def client(addr): + await asyncio.sleep(0.5) + + on_data = self.loop.create_future() + on_eof = self.loop.create_future() + + tr, proto = await self.loop.create_connection( + lambda: ClientProto(on_data, on_eof), *addr) + + tr.write(HELLO_MSG) + + await server_waits_on_handshake + + with self.assertRaises(asyncio.TimeoutError): + await asyncio.wait_for( + self.loop.start_tls(tr, proto, client_context), + 0.5) + + with self.tcp_server(serve, timeout=self.TIMEOUT) as srv: + self.loop.run_until_complete( + asyncio.wait_for(client(srv.addr), + timeout=support.SHORT_TIMEOUT)) + + @socket_helper.skip_if_tcp_blackhole + def test_start_tls_server_1(self): + HELLO_MSG = b'1' * self.PAYLOAD_SIZE + ANSWER = b'answer' + + server_context = test_utils.simple_server_sslcontext() + client_context = test_utils.simple_client_sslcontext() + answer = None + + def client(sock, addr): + nonlocal answer + sock.settimeout(self.TIMEOUT) + + sock.connect(addr) + data = sock.recv_all(len(HELLO_MSG)) + self.assertEqual(len(data), len(HELLO_MSG)) + + sock.start_tls(client_context) + sock.sendall(HELLO_MSG) + answer = sock.recv_all(len(ANSWER)) + sock.close() + + class ServerProto(asyncio.Protocol): + def __init__(self, on_con, on_con_lost, on_got_hello): + self.on_con = on_con + self.on_con_lost = on_con_lost + self.on_got_hello = on_got_hello + self.data = b'' + self.transport = None + + def connection_made(self, tr): + self.transport = tr + self.on_con.set_result(tr) + + def replace_transport(self, tr): + self.transport = tr + + def data_received(self, data): + self.data += data + if len(self.data) >= len(HELLO_MSG): + self.on_got_hello.set_result(None) + + def connection_lost(self, exc): + self.transport = None + if exc is None: + self.on_con_lost.set_result(None) + else: + self.on_con_lost.set_exception(exc) + + async def main(proto, on_con, on_con_lost, on_got_hello): + tr = await on_con + tr.write(HELLO_MSG) + + self.assertEqual(proto.data, b'') + + new_tr = await self.loop.start_tls( + tr, proto, server_context, + server_side=True, + ssl_handshake_timeout=self.TIMEOUT) + proto.replace_transport(new_tr) + + await on_got_hello + new_tr.write(ANSWER) + + await on_con_lost + self.assertEqual(proto.data, HELLO_MSG) + new_tr.close() + + async def run_main(): + on_con = self.loop.create_future() + on_con_lost = self.loop.create_future() + on_got_hello = self.loop.create_future() + proto = ServerProto(on_con, on_con_lost, on_got_hello) + + server = await self.loop.create_server( + lambda: proto, '127.0.0.1', 0) + addr = server.sockets[0].getsockname() + + with self.tcp_client(lambda sock: client(sock, addr), + timeout=self.TIMEOUT): + await asyncio.wait_for( + main(proto, on_con, on_con_lost, on_got_hello), + timeout=self.TIMEOUT) + + server.close() + await server.wait_closed() + self.assertEqual(answer, ANSWER) + + self.loop.run_until_complete(run_main()) + + def test_start_tls_wrong_args(self): + async def main(): + with self.assertRaisesRegex(TypeError, 'SSLContext, got'): + await self.loop.start_tls(None, None, None) + + sslctx = test_utils.simple_server_sslcontext() + with self.assertRaisesRegex(TypeError, 'is not supported'): + await self.loop.start_tls(None, None, sslctx) + + self.loop.run_until_complete(main()) + + @unittest.expectedFailure # TODO: RUSTPYTHON; - gc.collect() doesn't release SSLContext properly + def test_handshake_timeout(self): + # bpo-29970: Check that a connection is aborted if handshake is not + # completed in timeout period, instead of remaining open indefinitely + client_sslctx = test_utils.simple_client_sslcontext() + + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + + server_side_aborted = False + + def server(sock): + nonlocal server_side_aborted + try: + sock.recv_all(1024 * 1024) + except ConnectionAbortedError: + server_side_aborted = True + finally: + sock.close() + + async def client(addr): + await asyncio.wait_for( + self.loop.create_connection( + asyncio.Protocol, + *addr, + ssl=client_sslctx, + server_hostname='', + ssl_handshake_timeout=support.SHORT_TIMEOUT), + 0.5) + + with self.tcp_server(server, + max_clients=1, + backlog=1) as srv: + + with self.assertRaises(asyncio.TimeoutError): + self.loop.run_until_complete(client(srv.addr)) + + self.assertTrue(server_side_aborted) + + # Python issue #23197: cancelling a handshake must not raise an + # exception or log an error, even if the handshake failed + self.assertEqual(messages, []) + + # The 10s handshake timeout should be cancelled to free related + # objects without really waiting for 10s + client_sslctx = weakref.ref(client_sslctx) + support.gc_collect() + self.assertIsNone(client_sslctx()) + + def test_create_connection_ssl_slow_handshake(self): + client_sslctx = test_utils.simple_client_sslcontext() + + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + + def server(sock): + try: + sock.recv_all(1024 * 1024) + except ConnectionAbortedError: + pass + finally: + sock.close() + + async def client(addr): + reader, writer = await asyncio.open_connection( + *addr, + ssl=client_sslctx, + server_hostname='', + ssl_handshake_timeout=1.0) + + with self.tcp_server(server, + max_clients=1, + backlog=1) as srv: + + with self.assertRaisesRegex( + ConnectionAbortedError, + r'SSL handshake.*is taking longer'): + + self.loop.run_until_complete(client(srv.addr)) + + self.assertEqual(messages, []) + + def test_create_connection_ssl_failed_certificate(self): + self.loop.set_exception_handler(lambda loop, ctx: None) + + sslctx = test_utils.simple_server_sslcontext() + client_sslctx = test_utils.simple_client_sslcontext( + disable_verify=False) + + def server(sock): + try: + sock.start_tls( + sslctx, + server_side=True) + except ssl.SSLError: + pass + except OSError: + pass + finally: + sock.close() + + async def client(addr): + reader, writer = await asyncio.open_connection( + *addr, + ssl=client_sslctx, + server_hostname='', + ssl_handshake_timeout=support.LOOPBACK_TIMEOUT) + + with self.tcp_server(server, + max_clients=1, + backlog=1) as srv: + + with self.assertRaises(ssl.SSLCertVerificationError): + self.loop.run_until_complete(client(srv.addr)) + + def test_start_tls_client_corrupted_ssl(self): + self.loop.set_exception_handler(lambda loop, ctx: None) + + sslctx = test_utils.simple_server_sslcontext() + client_sslctx = test_utils.simple_client_sslcontext() + + def server(sock): + orig_sock = sock.dup() + try: + sock.start_tls( + sslctx, + server_side=True) + sock.sendall(b'A\n') + sock.recv_all(1) + orig_sock.send(b'please corrupt the SSL connection') + except ssl.SSLError: + pass + finally: + orig_sock.close() + sock.close() + + async def client(addr): + reader, writer = await asyncio.open_connection( + *addr, + ssl=client_sslctx, + server_hostname='') + + self.assertEqual(await reader.readline(), b'A\n') + writer.write(b'B') + with self.assertRaises(ssl.SSLError): + await reader.readline() + + writer.close() + return 'OK' + + with self.tcp_server(server, + max_clients=1, + backlog=1) as srv: + + res = self.loop.run_until_complete(client(srv.addr)) + + self.assertEqual(res, 'OK') + + +@unittest.skipIf(ssl is None, 'No ssl module') +class SelectorStartTLSTests(BaseStartTLS, unittest.TestCase): + + def new_loop(self): + return asyncio.SelectorEventLoop() + + +@unittest.skipIf(ssl is None, 'No ssl module') +@unittest.skipUnless(hasattr(asyncio, 'ProactorEventLoop'), 'Windows only') +class ProactorStartTLSTests(BaseStartTLS, unittest.TestCase): + + def new_loop(self): + return asyncio.ProactorEventLoop() + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_staggered.py b/Lib/test/test_asyncio/test_staggered.py new file mode 100644 index 00000000000..32e4817b70d --- /dev/null +++ b/Lib/test/test_asyncio/test_staggered.py @@ -0,0 +1,151 @@ +import asyncio +import unittest +from asyncio.staggered import staggered_race + +from test import support + +support.requires_working_socket(module=True) + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +class StaggeredTests(unittest.IsolatedAsyncioTestCase): + async def test_empty(self): + winner, index, excs = await staggered_race( + [], + delay=None, + ) + + self.assertIs(winner, None) + self.assertIs(index, None) + self.assertEqual(excs, []) + + async def test_one_successful(self): + async def coro(index): + return f'Res: {index}' + + winner, index, excs = await staggered_race( + [ + lambda: coro(0), + lambda: coro(1), + ], + delay=None, + ) + + self.assertEqual(winner, 'Res: 0') + self.assertEqual(index, 0) + self.assertEqual(excs, [None]) + + async def test_first_error_second_successful(self): + async def coro(index): + if index == 0: + raise ValueError(index) + return f'Res: {index}' + + winner, index, excs = await staggered_race( + [ + lambda: coro(0), + lambda: coro(1), + ], + delay=None, + ) + + self.assertEqual(winner, 'Res: 1') + self.assertEqual(index, 1) + self.assertEqual(len(excs), 2) + self.assertIsInstance(excs[0], ValueError) + self.assertIs(excs[1], None) + + async def test_first_timeout_second_successful(self): + async def coro(index): + if index == 0: + await asyncio.sleep(10) # much bigger than delay + return f'Res: {index}' + + winner, index, excs = await staggered_race( + [ + lambda: coro(0), + lambda: coro(1), + ], + delay=0.1, + ) + + self.assertEqual(winner, 'Res: 1') + self.assertEqual(index, 1) + self.assertEqual(len(excs), 2) + self.assertIsInstance(excs[0], asyncio.CancelledError) + self.assertIs(excs[1], None) + + async def test_none_successful(self): + async def coro(index): + raise ValueError(index) + + winner, index, excs = await staggered_race( + [ + lambda: coro(0), + lambda: coro(1), + ], + delay=None, + ) + + self.assertIs(winner, None) + self.assertIs(index, None) + self.assertEqual(len(excs), 2) + self.assertIsInstance(excs[0], ValueError) + self.assertIsInstance(excs[1], ValueError) + + + async def test_multiple_winners(self): + event = asyncio.Event() + + async def coro(index): + await event.wait() + return index + + async def do_set(): + event.set() + await asyncio.Event().wait() + + winner, index, excs = await staggered_race( + [ + lambda: coro(0), + lambda: coro(1), + do_set, + ], + delay=0.1, + ) + self.assertIs(winner, 0) + self.assertIs(index, 0) + self.assertEqual(len(excs), 3) + self.assertIsNone(excs[0], None) + self.assertIsInstance(excs[1], asyncio.CancelledError) + self.assertIsInstance(excs[2], asyncio.CancelledError) + + + async def test_cancelled(self): + log = [] + with self.assertRaises(TimeoutError): + async with asyncio.timeout(None) as cs_outer, asyncio.timeout(None) as cs_inner: + async def coro_fn(): + cs_inner.reschedule(-1) + await asyncio.sleep(0) + try: + await asyncio.sleep(0) + except asyncio.CancelledError: + log.append("cancelled 1") + + cs_outer.reschedule(-1) + await asyncio.sleep(0) + try: + await asyncio.sleep(0) + except asyncio.CancelledError: + log.append("cancelled 2") + try: + await staggered_race([coro_fn], delay=None) + except asyncio.CancelledError: + log.append("cancelled 3") + raise + + self.assertListEqual(log, ["cancelled 1", "cancelled 2", "cancelled 3"]) diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py new file mode 100644 index 00000000000..5f0fc6a7a9d --- /dev/null +++ b/Lib/test/test_asyncio/test_streams.py @@ -0,0 +1,1221 @@ +"""Tests for streams.py.""" + +import gc +import queue +import pickle +import socket +import threading +import unittest +from unittest import mock +try: + import ssl +except ImportError: + ssl = None + +import asyncio +from test.test_asyncio import utils as test_utils +from test.support import socket_helper + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +class StreamTests(test_utils.TestCase): + + DATA = b'line1\nline2\nline3\n' + + def setUp(self): + super().setUp() + self.loop = asyncio.new_event_loop() + self.set_event_loop(self.loop) + + def tearDown(self): + # just in case if we have transport close callbacks + test_utils.run_briefly(self.loop) + + # set_event_loop() takes care of closing self.loop in a safe way + super().tearDown() + + def _basetest_open_connection(self, open_connection_fut): + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + reader, writer = self.loop.run_until_complete(open_connection_fut) + writer.write(b'GET / HTTP/1.0\r\n\r\n') + f = reader.readline() + data = self.loop.run_until_complete(f) + self.assertEqual(data, b'HTTP/1.0 200 OK\r\n') + f = reader.read() + data = self.loop.run_until_complete(f) + self.assertEndsWith(data, b'\r\n\r\nTest message') + writer.close() + self.assertEqual(messages, []) + + def test_open_connection(self): + with test_utils.run_test_server() as httpd: + conn_fut = asyncio.open_connection(*httpd.address) + self._basetest_open_connection(conn_fut) + + @socket_helper.skip_unless_bind_unix_socket + def test_open_unix_connection(self): + with test_utils.run_test_unix_server() as httpd: + conn_fut = asyncio.open_unix_connection(httpd.address) + self._basetest_open_connection(conn_fut) + + def _basetest_open_connection_no_loop_ssl(self, open_connection_fut): + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + try: + reader, writer = self.loop.run_until_complete(open_connection_fut) + finally: + asyncio.set_event_loop(None) + writer.write(b'GET / HTTP/1.0\r\n\r\n') + f = reader.read() + data = self.loop.run_until_complete(f) + self.assertEndsWith(data, b'\r\n\r\nTest message') + + writer.close() + self.assertEqual(messages, []) + + @unittest.skipIf(ssl is None, 'No ssl module') + def test_open_connection_no_loop_ssl(self): + with test_utils.run_test_server(use_ssl=True) as httpd: + conn_fut = asyncio.open_connection( + *httpd.address, + ssl=test_utils.dummy_ssl_context()) + + self._basetest_open_connection_no_loop_ssl(conn_fut) + + @socket_helper.skip_unless_bind_unix_socket + @unittest.skipIf(ssl is None, 'No ssl module') + def test_open_unix_connection_no_loop_ssl(self): + with test_utils.run_test_unix_server(use_ssl=True) as httpd: + conn_fut = asyncio.open_unix_connection( + httpd.address, + ssl=test_utils.dummy_ssl_context(), + server_hostname='', + ) + + self._basetest_open_connection_no_loop_ssl(conn_fut) + + def _basetest_open_connection_error(self, open_connection_fut): + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + reader, writer = self.loop.run_until_complete(open_connection_fut) + writer._protocol.connection_lost(ZeroDivisionError()) + f = reader.read() + with self.assertRaises(ZeroDivisionError): + self.loop.run_until_complete(f) + writer.close() + test_utils.run_briefly(self.loop) + self.assertEqual(messages, []) + + def test_open_connection_error(self): + with test_utils.run_test_server() as httpd: + conn_fut = asyncio.open_connection(*httpd.address) + self._basetest_open_connection_error(conn_fut) + + @socket_helper.skip_unless_bind_unix_socket + def test_open_unix_connection_error(self): + with test_utils.run_test_unix_server() as httpd: + conn_fut = asyncio.open_unix_connection(httpd.address) + self._basetest_open_connection_error(conn_fut) + + def test_feed_empty_data(self): + stream = asyncio.StreamReader(loop=self.loop) + + stream.feed_data(b'') + self.assertEqual(b'', stream._buffer) + + def test_feed_nonempty_data(self): + stream = asyncio.StreamReader(loop=self.loop) + + stream.feed_data(self.DATA) + self.assertEqual(self.DATA, stream._buffer) + + def test_read_zero(self): + # Read zero bytes. + stream = asyncio.StreamReader(loop=self.loop) + stream.feed_data(self.DATA) + + data = self.loop.run_until_complete(stream.read(0)) + self.assertEqual(b'', data) + self.assertEqual(self.DATA, stream._buffer) + + def test_read(self): + # Read bytes. + stream = asyncio.StreamReader(loop=self.loop) + read_task = self.loop.create_task(stream.read(30)) + + def cb(): + stream.feed_data(self.DATA) + self.loop.call_soon(cb) + + data = self.loop.run_until_complete(read_task) + self.assertEqual(self.DATA, data) + self.assertEqual(b'', stream._buffer) + + def test_read_line_breaks(self): + # Read bytes without line breaks. + stream = asyncio.StreamReader(loop=self.loop) + stream.feed_data(b'line1') + stream.feed_data(b'line2') + + data = self.loop.run_until_complete(stream.read(5)) + + self.assertEqual(b'line1', data) + self.assertEqual(b'line2', stream._buffer) + + def test_read_eof(self): + # Read bytes, stop at eof. + stream = asyncio.StreamReader(loop=self.loop) + read_task = self.loop.create_task(stream.read(1024)) + + def cb(): + stream.feed_eof() + self.loop.call_soon(cb) + + data = self.loop.run_until_complete(read_task) + self.assertEqual(b'', data) + self.assertEqual(b'', stream._buffer) + + def test_read_until_eof(self): + # Read all bytes until eof. + stream = asyncio.StreamReader(loop=self.loop) + read_task = self.loop.create_task(stream.read(-1)) + + def cb(): + stream.feed_data(b'chunk1\n') + stream.feed_data(b'chunk2') + stream.feed_eof() + self.loop.call_soon(cb) + + data = self.loop.run_until_complete(read_task) + + self.assertEqual(b'chunk1\nchunk2', data) + self.assertEqual(b'', stream._buffer) + + def test_read_exception(self): + stream = asyncio.StreamReader(loop=self.loop) + stream.feed_data(b'line\n') + + data = self.loop.run_until_complete(stream.read(2)) + self.assertEqual(b'li', data) + + stream.set_exception(ValueError()) + self.assertRaises( + ValueError, self.loop.run_until_complete, stream.read(2)) + + def test_invalid_limit(self): + with self.assertRaisesRegex(ValueError, 'imit'): + asyncio.StreamReader(limit=0, loop=self.loop) + + with self.assertRaisesRegex(ValueError, 'imit'): + asyncio.StreamReader(limit=-1, loop=self.loop) + + def test_read_limit(self): + stream = asyncio.StreamReader(limit=3, loop=self.loop) + stream.feed_data(b'chunk') + data = self.loop.run_until_complete(stream.read(5)) + self.assertEqual(b'chunk', data) + self.assertEqual(b'', stream._buffer) + + def test_readline(self): + # Read one line. 'readline' will need to wait for the data + # to come from 'cb' + stream = asyncio.StreamReader(loop=self.loop) + stream.feed_data(b'chunk1 ') + read_task = self.loop.create_task(stream.readline()) + + def cb(): + stream.feed_data(b'chunk2 ') + stream.feed_data(b'chunk3 ') + stream.feed_data(b'\n chunk4') + self.loop.call_soon(cb) + + line = self.loop.run_until_complete(read_task) + self.assertEqual(b'chunk1 chunk2 chunk3 \n', line) + self.assertEqual(b' chunk4', stream._buffer) + + def test_readline_limit_with_existing_data(self): + # Read one line. The data is in StreamReader's buffer + # before the event loop is run. + + stream = asyncio.StreamReader(limit=3, loop=self.loop) + stream.feed_data(b'li') + stream.feed_data(b'ne1\nline2\n') + + self.assertRaises( + ValueError, self.loop.run_until_complete, stream.readline()) + # The buffer should contain the remaining data after exception + self.assertEqual(b'line2\n', stream._buffer) + + stream = asyncio.StreamReader(limit=3, loop=self.loop) + stream.feed_data(b'li') + stream.feed_data(b'ne1') + stream.feed_data(b'li') + + self.assertRaises( + ValueError, self.loop.run_until_complete, stream.readline()) + # No b'\n' at the end. The 'limit' is set to 3. So before + # waiting for the new data in buffer, 'readline' will consume + # the entire buffer, and since the length of the consumed data + # is more than 3, it will raise a ValueError. The buffer is + # expected to be empty now. + self.assertEqual(b'', stream._buffer) + + def test_at_eof(self): + stream = asyncio.StreamReader(loop=self.loop) + self.assertFalse(stream.at_eof()) + + stream.feed_data(b'some data\n') + self.assertFalse(stream.at_eof()) + + self.loop.run_until_complete(stream.readline()) + self.assertFalse(stream.at_eof()) + + stream.feed_data(b'some data\n') + stream.feed_eof() + self.loop.run_until_complete(stream.readline()) + self.assertTrue(stream.at_eof()) + + def test_readline_limit(self): + # Read one line. StreamReaders are fed with data after + # their 'readline' methods are called. + + stream = asyncio.StreamReader(limit=7, loop=self.loop) + def cb(): + stream.feed_data(b'chunk1') + stream.feed_data(b'chunk2') + stream.feed_data(b'chunk3\n') + stream.feed_eof() + self.loop.call_soon(cb) + + self.assertRaises( + ValueError, self.loop.run_until_complete, stream.readline()) + # The buffer had just one line of data, and after raising + # a ValueError it should be empty. + self.assertEqual(b'', stream._buffer) + + stream = asyncio.StreamReader(limit=7, loop=self.loop) + def cb(): + stream.feed_data(b'chunk1') + stream.feed_data(b'chunk2\n') + stream.feed_data(b'chunk3\n') + stream.feed_eof() + self.loop.call_soon(cb) + + self.assertRaises( + ValueError, self.loop.run_until_complete, stream.readline()) + self.assertEqual(b'chunk3\n', stream._buffer) + + # check strictness of the limit + stream = asyncio.StreamReader(limit=7, loop=self.loop) + stream.feed_data(b'1234567\n') + line = self.loop.run_until_complete(stream.readline()) + self.assertEqual(b'1234567\n', line) + self.assertEqual(b'', stream._buffer) + + stream.feed_data(b'12345678\n') + with self.assertRaises(ValueError) as cm: + self.loop.run_until_complete(stream.readline()) + self.assertEqual(b'', stream._buffer) + + stream.feed_data(b'12345678') + with self.assertRaises(ValueError) as cm: + self.loop.run_until_complete(stream.readline()) + self.assertEqual(b'', stream._buffer) + + def test_readline_nolimit_nowait(self): + # All needed data for the first 'readline' call will be + # in the buffer. + stream = asyncio.StreamReader(loop=self.loop) + stream.feed_data(self.DATA[:6]) + stream.feed_data(self.DATA[6:]) + + line = self.loop.run_until_complete(stream.readline()) + + self.assertEqual(b'line1\n', line) + self.assertEqual(b'line2\nline3\n', stream._buffer) + + def test_readline_eof(self): + stream = asyncio.StreamReader(loop=self.loop) + stream.feed_data(b'some data') + stream.feed_eof() + + line = self.loop.run_until_complete(stream.readline()) + self.assertEqual(b'some data', line) + + def test_readline_empty_eof(self): + stream = asyncio.StreamReader(loop=self.loop) + stream.feed_eof() + + line = self.loop.run_until_complete(stream.readline()) + self.assertEqual(b'', line) + + def test_readline_read_byte_count(self): + stream = asyncio.StreamReader(loop=self.loop) + stream.feed_data(self.DATA) + + self.loop.run_until_complete(stream.readline()) + + data = self.loop.run_until_complete(stream.read(7)) + + self.assertEqual(b'line2\nl', data) + self.assertEqual(b'ine3\n', stream._buffer) + + def test_readline_exception(self): + stream = asyncio.StreamReader(loop=self.loop) + stream.feed_data(b'line\n') + + data = self.loop.run_until_complete(stream.readline()) + self.assertEqual(b'line\n', data) + + stream.set_exception(ValueError()) + self.assertRaises( + ValueError, self.loop.run_until_complete, stream.readline()) + self.assertEqual(b'', stream._buffer) + + def test_readuntil_separator(self): + stream = asyncio.StreamReader(loop=self.loop) + with self.assertRaisesRegex(ValueError, 'Separator should be'): + self.loop.run_until_complete(stream.readuntil(separator=b'')) + with self.assertRaisesRegex(ValueError, 'Separator should be'): + self.loop.run_until_complete(stream.readuntil(separator=(b'',))) + with self.assertRaisesRegex(ValueError, 'Separator should contain'): + self.loop.run_until_complete(stream.readuntil(separator=())) + + def test_readuntil_multi_chunks(self): + stream = asyncio.StreamReader(loop=self.loop) + + stream.feed_data(b'lineAAA') + data = self.loop.run_until_complete(stream.readuntil(separator=b'AAA')) + self.assertEqual(b'lineAAA', data) + self.assertEqual(b'', stream._buffer) + + stream.feed_data(b'lineAAA') + data = self.loop.run_until_complete(stream.readuntil(b'AAA')) + self.assertEqual(b'lineAAA', data) + self.assertEqual(b'', stream._buffer) + + stream.feed_data(b'lineAAAxxx') + data = self.loop.run_until_complete(stream.readuntil(b'AAA')) + self.assertEqual(b'lineAAA', data) + self.assertEqual(b'xxx', stream._buffer) + + def test_readuntil_multi_chunks_1(self): + stream = asyncio.StreamReader(loop=self.loop) + + stream.feed_data(b'QWEaa') + stream.feed_data(b'XYaa') + stream.feed_data(b'a') + data = self.loop.run_until_complete(stream.readuntil(b'aaa')) + self.assertEqual(b'QWEaaXYaaa', data) + self.assertEqual(b'', stream._buffer) + + stream.feed_data(b'QWEaa') + stream.feed_data(b'XYa') + stream.feed_data(b'aa') + data = self.loop.run_until_complete(stream.readuntil(b'aaa')) + self.assertEqual(b'QWEaaXYaaa', data) + self.assertEqual(b'', stream._buffer) + + stream.feed_data(b'aaa') + data = self.loop.run_until_complete(stream.readuntil(b'aaa')) + self.assertEqual(b'aaa', data) + self.assertEqual(b'', stream._buffer) + + stream.feed_data(b'Xaaa') + data = self.loop.run_until_complete(stream.readuntil(b'aaa')) + self.assertEqual(b'Xaaa', data) + self.assertEqual(b'', stream._buffer) + + stream.feed_data(b'XXX') + stream.feed_data(b'a') + stream.feed_data(b'a') + stream.feed_data(b'a') + data = self.loop.run_until_complete(stream.readuntil(b'aaa')) + self.assertEqual(b'XXXaaa', data) + self.assertEqual(b'', stream._buffer) + + def test_readuntil_eof(self): + stream = asyncio.StreamReader(loop=self.loop) + data = b'some dataAA' + stream.feed_data(data) + stream.feed_eof() + + with self.assertRaisesRegex(asyncio.IncompleteReadError, + 'undefined expected bytes') as cm: + self.loop.run_until_complete(stream.readuntil(b'AAA')) + self.assertEqual(cm.exception.partial, data) + self.assertIsNone(cm.exception.expected) + self.assertEqual(b'', stream._buffer) + + def test_readuntil_limit_found_sep(self): + stream = asyncio.StreamReader(loop=self.loop, limit=3) + stream.feed_data(b'some dataAA') + with self.assertRaisesRegex(asyncio.LimitOverrunError, + 'not found') as cm: + self.loop.run_until_complete(stream.readuntil(b'AAA')) + + self.assertEqual(b'some dataAA', stream._buffer) + + stream.feed_data(b'A') + with self.assertRaisesRegex(asyncio.LimitOverrunError, + 'is found') as cm: + self.loop.run_until_complete(stream.readuntil(b'AAA')) + + self.assertEqual(b'some dataAAA', stream._buffer) + + def test_readuntil_multi_separator(self): + stream = asyncio.StreamReader(loop=self.loop) + + # Simple case + stream.feed_data(b'line 1\nline 2\r') + data = self.loop.run_until_complete(stream.readuntil((b'\r', b'\n'))) + self.assertEqual(b'line 1\n', data) + data = self.loop.run_until_complete(stream.readuntil((b'\r', b'\n'))) + self.assertEqual(b'line 2\r', data) + self.assertEqual(b'', stream._buffer) + + # First end position matches, even if that's a longer match + stream.feed_data(b'ABCDEFG') + data = self.loop.run_until_complete(stream.readuntil((b'DEF', b'BCDE'))) + self.assertEqual(b'ABCDE', data) + self.assertEqual(b'FG', stream._buffer) + + def test_readuntil_multi_separator_limit(self): + stream = asyncio.StreamReader(loop=self.loop, limit=3) + stream.feed_data(b'some dataA') + + with self.assertRaisesRegex(asyncio.LimitOverrunError, + 'is found') as cm: + self.loop.run_until_complete(stream.readuntil((b'A', b'ome dataA'))) + + self.assertEqual(b'some dataA', stream._buffer) + + def test_readuntil_multi_separator_negative_offset(self): + # If the buffer is big enough for the smallest separator (but does + # not contain it) but too small for the largest, `offset` must not + # become negative. + stream = asyncio.StreamReader(loop=self.loop) + stream.feed_data(b'data') + + readuntil_task = self.loop.create_task(stream.readuntil((b'A', b'long sep'))) + self.loop.call_soon(stream.feed_data, b'Z') + self.loop.call_soon(stream.feed_data, b'Aaaa') + + data = self.loop.run_until_complete(readuntil_task) + self.assertEqual(b'dataZA', data) + self.assertEqual(b'aaa', stream._buffer) + + def test_readuntil_bytearray(self): + stream = asyncio.StreamReader(loop=self.loop) + stream.feed_data(b'some data\r\n') + data = self.loop.run_until_complete(stream.readuntil(bytearray(b'\r\n'))) + self.assertEqual(b'some data\r\n', data) + self.assertEqual(b'', stream._buffer) + + def test_readexactly_zero_or_less(self): + # Read exact number of bytes (zero or less). + stream = asyncio.StreamReader(loop=self.loop) + stream.feed_data(self.DATA) + + data = self.loop.run_until_complete(stream.readexactly(0)) + self.assertEqual(b'', data) + self.assertEqual(self.DATA, stream._buffer) + + with self.assertRaisesRegex(ValueError, 'less than zero'): + self.loop.run_until_complete(stream.readexactly(-1)) + self.assertEqual(self.DATA, stream._buffer) + + def test_readexactly(self): + # Read exact number of bytes. + stream = asyncio.StreamReader(loop=self.loop) + + n = 2 * len(self.DATA) + read_task = self.loop.create_task(stream.readexactly(n)) + + def cb(): + stream.feed_data(self.DATA) + stream.feed_data(self.DATA) + stream.feed_data(self.DATA) + self.loop.call_soon(cb) + + data = self.loop.run_until_complete(read_task) + self.assertEqual(self.DATA + self.DATA, data) + self.assertEqual(self.DATA, stream._buffer) + + def test_readexactly_limit(self): + stream = asyncio.StreamReader(limit=3, loop=self.loop) + stream.feed_data(b'chunk') + data = self.loop.run_until_complete(stream.readexactly(5)) + self.assertEqual(b'chunk', data) + self.assertEqual(b'', stream._buffer) + + def test_readexactly_eof(self): + # Read exact number of bytes (eof). + stream = asyncio.StreamReader(loop=self.loop) + n = 2 * len(self.DATA) + read_task = self.loop.create_task(stream.readexactly(n)) + + def cb(): + stream.feed_data(self.DATA) + stream.feed_eof() + self.loop.call_soon(cb) + + with self.assertRaises(asyncio.IncompleteReadError) as cm: + self.loop.run_until_complete(read_task) + self.assertEqual(cm.exception.partial, self.DATA) + self.assertEqual(cm.exception.expected, n) + self.assertEqual(str(cm.exception), + '18 bytes read on a total of 36 expected bytes') + self.assertEqual(b'', stream._buffer) + + def test_readexactly_exception(self): + stream = asyncio.StreamReader(loop=self.loop) + stream.feed_data(b'line\n') + + data = self.loop.run_until_complete(stream.readexactly(2)) + self.assertEqual(b'li', data) + + stream.set_exception(ValueError()) + self.assertRaises( + ValueError, self.loop.run_until_complete, stream.readexactly(2)) + + def test_exception(self): + stream = asyncio.StreamReader(loop=self.loop) + self.assertIsNone(stream.exception()) + + exc = ValueError() + stream.set_exception(exc) + self.assertIs(stream.exception(), exc) + + def test_exception_waiter(self): + stream = asyncio.StreamReader(loop=self.loop) + + async def set_err(): + stream.set_exception(ValueError()) + + t1 = self.loop.create_task(stream.readline()) + t2 = self.loop.create_task(set_err()) + + self.loop.run_until_complete(asyncio.wait([t1, t2])) + + self.assertRaises(ValueError, t1.result) + + def test_exception_cancel(self): + stream = asyncio.StreamReader(loop=self.loop) + + t = self.loop.create_task(stream.readline()) + test_utils.run_briefly(self.loop) + t.cancel() + test_utils.run_briefly(self.loop) + # The following line fails if set_exception() isn't careful. + stream.set_exception(RuntimeError('message')) + test_utils.run_briefly(self.loop) + self.assertIs(stream._waiter, None) + + def test_start_server(self): + + class MyServer: + + def __init__(self, loop): + self.server = None + self.loop = loop + + async def handle_client(self, client_reader, client_writer): + data = await client_reader.readline() + client_writer.write(data) + await client_writer.drain() + client_writer.close() + await client_writer.wait_closed() + + def start(self): + sock = socket.create_server(('127.0.0.1', 0)) + self.server = self.loop.run_until_complete( + asyncio.start_server(self.handle_client, + sock=sock)) + return sock.getsockname() + + def handle_client_callback(self, client_reader, client_writer): + self.loop.create_task(self.handle_client(client_reader, + client_writer)) + + def start_callback(self): + sock = socket.create_server(('127.0.0.1', 0)) + addr = sock.getsockname() + sock.close() + self.server = self.loop.run_until_complete( + asyncio.start_server(self.handle_client_callback, + host=addr[0], port=addr[1])) + return addr + + def stop(self): + if self.server is not None: + self.server.close() + self.loop.run_until_complete(self.server.wait_closed()) + self.server = None + + async def client(addr): + reader, writer = await asyncio.open_connection(*addr) + # send a line + writer.write(b"hello world!\n") + # read it back + msgback = await reader.readline() + writer.close() + await writer.wait_closed() + return msgback + + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + + # test the server variant with a coroutine as client handler + server = MyServer(self.loop) + addr = server.start() + msg = self.loop.run_until_complete(self.loop.create_task(client(addr))) + server.stop() + self.assertEqual(msg, b"hello world!\n") + + # test the server variant with a callback as client handler + server = MyServer(self.loop) + addr = server.start_callback() + msg = self.loop.run_until_complete(self.loop.create_task(client(addr))) + server.stop() + self.assertEqual(msg, b"hello world!\n") + + self.assertEqual(messages, []) + + @socket_helper.skip_unless_bind_unix_socket + def test_start_unix_server(self): + + class MyServer: + + def __init__(self, loop, path): + self.server = None + self.loop = loop + self.path = path + + async def handle_client(self, client_reader, client_writer): + data = await client_reader.readline() + client_writer.write(data) + await client_writer.drain() + client_writer.close() + await client_writer.wait_closed() + + def start(self): + self.server = self.loop.run_until_complete( + asyncio.start_unix_server(self.handle_client, + path=self.path)) + + def handle_client_callback(self, client_reader, client_writer): + self.loop.create_task(self.handle_client(client_reader, + client_writer)) + + def start_callback(self): + start = asyncio.start_unix_server(self.handle_client_callback, + path=self.path) + self.server = self.loop.run_until_complete(start) + + def stop(self): + if self.server is not None: + self.server.close() + self.loop.run_until_complete(self.server.wait_closed()) + self.server = None + + async def client(path): + reader, writer = await asyncio.open_unix_connection(path) + # send a line + writer.write(b"hello world!\n") + # read it back + msgback = await reader.readline() + writer.close() + await writer.wait_closed() + return msgback + + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + + # test the server variant with a coroutine as client handler + with test_utils.unix_socket_path() as path: + server = MyServer(self.loop, path) + server.start() + msg = self.loop.run_until_complete( + self.loop.create_task(client(path))) + server.stop() + self.assertEqual(msg, b"hello world!\n") + + # test the server variant with a callback as client handler + with test_utils.unix_socket_path() as path: + server = MyServer(self.loop, path) + server.start_callback() + msg = self.loop.run_until_complete( + self.loop.create_task(client(path))) + server.stop() + self.assertEqual(msg, b"hello world!\n") + + self.assertEqual(messages, []) + + @unittest.skipIf(ssl is None, 'No ssl module') + def test_start_tls(self): + + class MyServer: + + def __init__(self, loop): + self.server = None + self.loop = loop + + async def handle_client(self, client_reader, client_writer): + data1 = await client_reader.readline() + client_writer.write(data1) + await client_writer.drain() + assert client_writer.get_extra_info('sslcontext') is None + await client_writer.start_tls( + test_utils.simple_server_sslcontext()) + assert client_writer.get_extra_info('sslcontext') is not None + data2 = await client_reader.readline() + client_writer.write(data2) + await client_writer.drain() + client_writer.close() + await client_writer.wait_closed() + + def start(self): + sock = socket.create_server(('127.0.0.1', 0)) + self.server = self.loop.run_until_complete( + asyncio.start_server(self.handle_client, + sock=sock)) + return sock.getsockname() + + def stop(self): + if self.server is not None: + self.server.close() + self.loop.run_until_complete(self.server.wait_closed()) + self.server = None + + async def client(addr): + reader, writer = await asyncio.open_connection(*addr) + writer.write(b"hello world 1!\n") + await writer.drain() + msgback1 = await reader.readline() + assert writer.get_extra_info('sslcontext') is None + await writer.start_tls(test_utils.simple_client_sslcontext()) + assert writer.get_extra_info('sslcontext') is not None + writer.write(b"hello world 2!\n") + await writer.drain() + msgback2 = await reader.readline() + writer.close() + await writer.wait_closed() + return msgback1, msgback2 + + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + + server = MyServer(self.loop) + addr = server.start() + msg1, msg2 = self.loop.run_until_complete(client(addr)) + server.stop() + + self.assertEqual(messages, []) + self.assertEqual(msg1, b"hello world 1!\n") + self.assertEqual(msg2, b"hello world 2!\n") + + def test_streamreader_constructor_without_loop(self): + with self.assertRaisesRegex(RuntimeError, 'no current event loop'): + asyncio.StreamReader() + + def test_streamreader_constructor_use_running_loop(self): + # asyncio issue #184: Ensure that StreamReaderProtocol constructor + # retrieves the current loop if the loop parameter is not set + async def test(): + return asyncio.StreamReader() + + reader = self.loop.run_until_complete(test()) + self.assertIs(reader._loop, self.loop) + + def test_streamreader_constructor_use_global_loop(self): + # asyncio issue #184: Ensure that StreamReaderProtocol constructor + # retrieves the current loop if the loop parameter is not set + # Deprecated in 3.10, undeprecated in 3.12 + self.addCleanup(asyncio.set_event_loop, None) + asyncio.set_event_loop(self.loop) + reader = asyncio.StreamReader() + self.assertIs(reader._loop, self.loop) + + + def test_streamreaderprotocol_constructor_without_loop(self): + reader = mock.Mock() + with self.assertRaisesRegex(RuntimeError, 'no current event loop'): + asyncio.StreamReaderProtocol(reader) + + def test_streamreaderprotocol_constructor_use_running_loop(self): + # asyncio issue #184: Ensure that StreamReaderProtocol constructor + # retrieves the current loop if the loop parameter is not set + reader = mock.Mock() + async def test(): + return asyncio.StreamReaderProtocol(reader) + protocol = self.loop.run_until_complete(test()) + self.assertIs(protocol._loop, self.loop) + + def test_streamreaderprotocol_constructor_use_global_loop(self): + # asyncio issue #184: Ensure that StreamReaderProtocol constructor + # retrieves the current loop if the loop parameter is not set + # Deprecated in 3.10, undeprecated in 3.12 + self.addCleanup(asyncio.set_event_loop, None) + asyncio.set_event_loop(self.loop) + reader = mock.Mock() + protocol = asyncio.StreamReaderProtocol(reader) + self.assertIs(protocol._loop, self.loop) + + def test_multiple_drain(self): + # See https://github.com/python/cpython/issues/74116 + drained = 0 + + async def drainer(stream): + nonlocal drained + await stream._drain_helper() + drained += 1 + + async def main(): + loop = asyncio.get_running_loop() + stream = asyncio.streams.FlowControlMixin(loop) + stream.pause_writing() + loop.call_later(0.1, stream.resume_writing) + await asyncio.gather(*[drainer(stream) for _ in range(10)]) + self.assertEqual(drained, 10) + + self.loop.run_until_complete(main()) + + def test_drain_raises(self): + # See http://bugs.python.org/issue25441 + + # This test should not use asyncio for the mock server; the + # whole point of the test is to test for a bug in drain() + # where it never gives up the event loop but the socket is + # closed on the server side. + + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + q = queue.Queue() + + def server(): + # Runs in a separate thread. + with socket.create_server(('localhost', 0)) as sock: + addr = sock.getsockname() + q.put(addr) + clt, _ = sock.accept() + clt.close() + + async def client(host, port): + reader, writer = await asyncio.open_connection(host, port) + + while True: + writer.write(b"foo\n") + await writer.drain() + + # Start the server thread and wait for it to be listening. + thread = threading.Thread(target=server) + thread.daemon = True + thread.start() + addr = q.get() + + # Should not be stuck in an infinite loop. + with self.assertRaises((ConnectionResetError, ConnectionAbortedError, + BrokenPipeError)): + self.loop.run_until_complete(client(*addr)) + + # Clean up the thread. (Only on success; on failure, it may + # be stuck in accept().) + thread.join() + self.assertEqual([], messages) + + def test___repr__(self): + stream = asyncio.StreamReader(loop=self.loop) + self.assertEqual("", repr(stream)) + + def test___repr__nondefault_limit(self): + stream = asyncio.StreamReader(loop=self.loop, limit=123) + self.assertEqual("", repr(stream)) + + def test___repr__eof(self): + stream = asyncio.StreamReader(loop=self.loop) + stream.feed_eof() + self.assertEqual("", repr(stream)) + + def test___repr__data(self): + stream = asyncio.StreamReader(loop=self.loop) + stream.feed_data(b'data') + self.assertEqual("", repr(stream)) + + def test___repr__exception(self): + stream = asyncio.StreamReader(loop=self.loop) + exc = RuntimeError() + stream.set_exception(exc) + self.assertEqual("", + repr(stream)) + + def test___repr__waiter(self): + stream = asyncio.StreamReader(loop=self.loop) + stream._waiter = asyncio.Future(loop=self.loop) + self.assertRegex( + repr(stream), + r">") + stream._waiter.set_result(None) + self.loop.run_until_complete(stream._waiter) + stream._waiter = None + self.assertEqual("", repr(stream)) + + def test___repr__transport(self): + stream = asyncio.StreamReader(loop=self.loop) + stream._transport = mock.Mock() + stream._transport.__repr__ = mock.Mock() + stream._transport.__repr__.return_value = "" + self.assertEqual(">", repr(stream)) + + def test_IncompleteReadError_pickleable(self): + e = asyncio.IncompleteReadError(b'abc', 10) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(pickle_protocol=proto): + e2 = pickle.loads(pickle.dumps(e, protocol=proto)) + self.assertEqual(str(e), str(e2)) + self.assertEqual(e.partial, e2.partial) + self.assertEqual(e.expected, e2.expected) + + def test_LimitOverrunError_pickleable(self): + e = asyncio.LimitOverrunError('message', 10) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(pickle_protocol=proto): + e2 = pickle.loads(pickle.dumps(e, protocol=proto)) + self.assertEqual(str(e), str(e2)) + self.assertEqual(e.consumed, e2.consumed) + + def test_wait_closed_on_close(self): + with test_utils.run_test_server() as httpd: + rd, wr = self.loop.run_until_complete( + asyncio.open_connection(*httpd.address)) + + wr.write(b'GET / HTTP/1.0\r\n\r\n') + f = rd.readline() + data = self.loop.run_until_complete(f) + self.assertEqual(data, b'HTTP/1.0 200 OK\r\n') + f = rd.read() + data = self.loop.run_until_complete(f) + self.assertEndsWith(data, b'\r\n\r\nTest message') + self.assertFalse(wr.is_closing()) + wr.close() + self.assertTrue(wr.is_closing()) + self.loop.run_until_complete(wr.wait_closed()) + + def test_wait_closed_on_close_with_unread_data(self): + with test_utils.run_test_server() as httpd: + rd, wr = self.loop.run_until_complete( + asyncio.open_connection(*httpd.address)) + + wr.write(b'GET / HTTP/1.0\r\n\r\n') + f = rd.readline() + data = self.loop.run_until_complete(f) + self.assertEqual(data, b'HTTP/1.0 200 OK\r\n') + wr.close() + self.loop.run_until_complete(wr.wait_closed()) + + def test_async_writer_api(self): + async def inner(httpd): + rd, wr = await asyncio.open_connection(*httpd.address) + + wr.write(b'GET / HTTP/1.0\r\n\r\n') + data = await rd.readline() + self.assertEqual(data, b'HTTP/1.0 200 OK\r\n') + data = await rd.read() + self.assertEndsWith(data, b'\r\n\r\nTest message') + wr.close() + await wr.wait_closed() + + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + + with test_utils.run_test_server() as httpd: + self.loop.run_until_complete(inner(httpd)) + + self.assertEqual(messages, []) + + def test_async_writer_api_exception_after_close(self): + async def inner(httpd): + rd, wr = await asyncio.open_connection(*httpd.address) + + wr.write(b'GET / HTTP/1.0\r\n\r\n') + data = await rd.readline() + self.assertEqual(data, b'HTTP/1.0 200 OK\r\n') + data = await rd.read() + self.assertEndsWith(data, b'\r\n\r\nTest message') + wr.close() + with self.assertRaises(ConnectionResetError): + wr.write(b'data') + await wr.drain() + + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + + with test_utils.run_test_server() as httpd: + self.loop.run_until_complete(inner(httpd)) + + self.assertEqual(messages, []) + + def test_eof_feed_when_closing_writer(self): + # See http://bugs.python.org/issue35065 + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + + with test_utils.run_test_server() as httpd: + rd, wr = self.loop.run_until_complete( + asyncio.open_connection(*httpd.address)) + + wr.close() + f = wr.wait_closed() + self.loop.run_until_complete(f) + self.assertTrue(rd.at_eof()) + f = rd.read() + data = self.loop.run_until_complete(f) + self.assertEqual(data, b'') + + self.assertEqual(messages, []) + + def test_unclosed_resource_warnings(self): + async def inner(httpd): + rd, wr = await asyncio.open_connection(*httpd.address) + + wr.write(b'GET / HTTP/1.0\r\n\r\n') + data = await rd.readline() + self.assertEqual(data, b'HTTP/1.0 200 OK\r\n') + data = await rd.read() + self.assertEndsWith(data, b'\r\n\r\nTest message') + with self.assertWarns(ResourceWarning) as cm: + del wr + gc.collect() + self.assertEqual(len(cm.warnings), 1) + self.assertStartsWith(str(cm.warnings[0].message), "unclosed " + ) + transport._returncode = None + self.assertEqual( + repr(transport), + "" + ) + transport._pid = None + transport._returncode = None + self.assertEqual( + repr(transport), + "" + ) + transport.close() + + +class SubprocessMixin: + + def test_stdin_stdout(self): + args = PROGRAM_CAT + + async def run(data): + proc = await asyncio.create_subprocess_exec( + *args, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + ) + + # feed data + proc.stdin.write(data) + await proc.stdin.drain() + proc.stdin.close() + + # get output and exitcode + data = await proc.stdout.read() + exitcode = await proc.wait() + return (exitcode, data) + + task = run(b'some data') + task = asyncio.wait_for(task, 60.0) + exitcode, stdout = self.loop.run_until_complete(task) + self.assertEqual(exitcode, 0) + self.assertEqual(stdout, b'some data') + + def test_communicate(self): + args = PROGRAM_CAT + + async def run(data): + proc = await asyncio.create_subprocess_exec( + *args, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + ) + stdout, stderr = await proc.communicate(data) + return proc.returncode, stdout + + task = run(b'some data') + task = asyncio.wait_for(task, support.LONG_TIMEOUT) + exitcode, stdout = self.loop.run_until_complete(task) + self.assertEqual(exitcode, 0) + self.assertEqual(stdout, b'some data') + + def test_communicate_none_input(self): + args = PROGRAM_CAT + + async def run(): + proc = await asyncio.create_subprocess_exec( + *args, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + ) + stdout, stderr = await proc.communicate() + return proc.returncode, stdout + + task = run() + task = asyncio.wait_for(task, support.LONG_TIMEOUT) + exitcode, stdout = self.loop.run_until_complete(task) + self.assertEqual(exitcode, 0) + self.assertEqual(stdout, b'') + + def test_shell(self): + proc = self.loop.run_until_complete( + asyncio.create_subprocess_shell('exit 7') + ) + exitcode = self.loop.run_until_complete(proc.wait()) + self.assertEqual(exitcode, 7) + + def test_start_new_session(self): + # start the new process in a new session + proc = self.loop.run_until_complete( + asyncio.create_subprocess_shell( + 'exit 8', + start_new_session=True, + ) + ) + exitcode = self.loop.run_until_complete(proc.wait()) + self.assertEqual(exitcode, 8) + + def test_kill(self): + args = PROGRAM_BLOCKED + proc = self.loop.run_until_complete( + asyncio.create_subprocess_exec(*args) + ) + proc.kill() + returncode = self.loop.run_until_complete(proc.wait()) + if sys.platform == 'win32': + self.assertIsInstance(returncode, int) + # expect 1 but sometimes get 0 + else: + self.assertEqual(-signal.SIGKILL, returncode) + + def test_kill_issue43884(self): + if sys.platform == 'win32': + blocking_shell_command = f'"{sys.executable}" -c "import time; time.sleep(2)"' + else: + blocking_shell_command = 'sleep 1; sleep 1' + creationflags = 0 + if sys.platform == 'win32': + from subprocess import CREATE_NEW_PROCESS_GROUP + # On windows create a new process group so that killing process + # kills the process and all its children. + creationflags = CREATE_NEW_PROCESS_GROUP + proc = self.loop.run_until_complete( + asyncio.create_subprocess_shell(blocking_shell_command, stdout=asyncio.subprocess.PIPE, + creationflags=creationflags) + ) + self.loop.run_until_complete(asyncio.sleep(1)) + if sys.platform == 'win32': + proc.send_signal(signal.CTRL_BREAK_EVENT) + # On windows it is an alias of terminate which sets the return code + proc.kill() + returncode = self.loop.run_until_complete(proc.wait()) + if sys.platform == 'win32': + self.assertIsInstance(returncode, int) + # expect 1 but sometimes get 0 + else: + self.assertEqual(-signal.SIGKILL, returncode) + + def test_terminate(self): + args = PROGRAM_BLOCKED + proc = self.loop.run_until_complete( + asyncio.create_subprocess_exec(*args) + ) + proc.terminate() + returncode = self.loop.run_until_complete(proc.wait()) + if sys.platform == 'win32': + self.assertIsInstance(returncode, int) + # expect 1 but sometimes get 0 + else: + self.assertEqual(-signal.SIGTERM, returncode) + + @unittest.skipIf(sys.platform == 'win32', "Don't have SIGHUP") + def test_send_signal(self): + # bpo-31034: Make sure that we get the default signal handler (killing + # the process). The parent process may have decided to ignore SIGHUP, + # and signal handlers are inherited. + old_handler = signal.signal(signal.SIGHUP, signal.SIG_DFL) + try: + code = 'import time; print("sleeping", flush=True); time.sleep(3600)' + args = [sys.executable, '-c', code] + proc = self.loop.run_until_complete( + asyncio.create_subprocess_exec( + *args, + stdout=subprocess.PIPE, + ) + ) + + async def send_signal(proc): + # basic synchronization to wait until the program is sleeping + line = await proc.stdout.readline() + self.assertEqual(line, b'sleeping\n') + + proc.send_signal(signal.SIGHUP) + returncode = await proc.wait() + return returncode + + returncode = self.loop.run_until_complete(send_signal(proc)) + self.assertEqual(-signal.SIGHUP, returncode) + finally: + signal.signal(signal.SIGHUP, old_handler) + + def test_stdin_broken_pipe(self): + # buffer large enough to feed the whole pipe buffer + large_data = b'x' * support.PIPE_MAX_SIZE + + rfd, wfd = os.pipe() + self.addCleanup(os.close, rfd) + self.addCleanup(os.close, wfd) + if support.MS_WINDOWS: + handle = msvcrt.get_osfhandle(rfd) + os.set_handle_inheritable(handle, True) + code = textwrap.dedent(f''' + import os, msvcrt + handle = {handle} + fd = msvcrt.open_osfhandle(handle, os.O_RDONLY) + os.read(fd, 1) + ''') + from subprocess import STARTUPINFO + startupinfo = STARTUPINFO() + startupinfo.lpAttributeList = {"handle_list": [handle]} + kwargs = dict(startupinfo=startupinfo) + else: + code = f'import os; fd = {rfd}; os.read(fd, 1)' + kwargs = dict(pass_fds=(rfd,)) + + # the program ends before the stdin can be fed + proc = self.loop.run_until_complete( + asyncio.create_subprocess_exec( + sys.executable, '-c', code, + stdin=subprocess.PIPE, + **kwargs + ) + ) + + async def write_stdin(proc, data): + proc.stdin.write(data) + # Only exit the child process once the write buffer is filled + os.write(wfd, b'go') + await proc.stdin.drain() + + coro = write_stdin(proc, large_data) + # drain() must raise BrokenPipeError or ConnectionResetError + with test_utils.disable_logger(): + self.assertRaises((BrokenPipeError, ConnectionResetError), + self.loop.run_until_complete, coro) + self.loop.run_until_complete(proc.wait()) + + def test_communicate_ignore_broken_pipe(self): + # buffer large enough to feed the whole pipe buffer + large_data = b'x' * support.PIPE_MAX_SIZE + + # the program ends before the stdin can be fed + proc = self.loop.run_until_complete( + asyncio.create_subprocess_exec( + sys.executable, '-c', 'pass', + stdin=subprocess.PIPE, + ) + ) + + # communicate() must ignore BrokenPipeError when feeding stdin + self.loop.set_exception_handler(lambda loop, msg: None) + self.loop.run_until_complete(proc.communicate(large_data)) + self.loop.run_until_complete(proc.wait()) + + def test_pause_reading(self): + limit = 10 + size = (limit * 2 + 1) + + async def test_pause_reading(): + code = '\n'.join(( + 'import sys', + 'sys.stdout.write("x" * %s)' % size, + 'sys.stdout.flush()', + )) + + connect_read_pipe = self.loop.connect_read_pipe + + async def connect_read_pipe_mock(*args, **kw): + transport, protocol = await connect_read_pipe(*args, **kw) + transport.pause_reading = mock.Mock() + transport.resume_reading = mock.Mock() + return (transport, protocol) + + self.loop.connect_read_pipe = connect_read_pipe_mock + + proc = await asyncio.create_subprocess_exec( + sys.executable, '-c', code, + stdin=asyncio.subprocess.PIPE, + stdout=asyncio.subprocess.PIPE, + limit=limit, + ) + stdout_transport = proc._transport.get_pipe_transport(1) + + stdout, stderr = await proc.communicate() + + # The child process produced more than limit bytes of output, + # the stream reader transport should pause the protocol to not + # allocate too much memory. + return (stdout, stdout_transport) + + # Issue #22685: Ensure that the stream reader pauses the protocol + # when the child process produces too much data + stdout, transport = self.loop.run_until_complete(test_pause_reading()) + + self.assertEqual(stdout, b'x' * size) + self.assertTrue(transport.pause_reading.called) + self.assertTrue(transport.resume_reading.called) + + def test_stdin_not_inheritable(self): + # asyncio issue #209: stdin must not be inheritable, otherwise + # the Process.communicate() hangs + async def len_message(message): + code = 'import sys; data = sys.stdin.read(); print(len(data))' + proc = await asyncio.create_subprocess_exec( + sys.executable, '-c', code, + stdin=asyncio.subprocess.PIPE, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + close_fds=False, + ) + stdout, stderr = await proc.communicate(message) + exitcode = await proc.wait() + return (stdout, exitcode) + + output, exitcode = self.loop.run_until_complete(len_message(b'abc')) + self.assertEqual(output.rstrip(), b'3') + self.assertEqual(exitcode, 0) + + def test_empty_input(self): + + async def empty_input(): + code = 'import sys; data = sys.stdin.read(); print(len(data))' + proc = await asyncio.create_subprocess_exec( + sys.executable, '-c', code, + stdin=asyncio.subprocess.PIPE, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + close_fds=False, + ) + stdout, stderr = await proc.communicate(b'') + exitcode = await proc.wait() + return (stdout, exitcode) + + output, exitcode = self.loop.run_until_complete(empty_input()) + self.assertEqual(output.rstrip(), b'0') + self.assertEqual(exitcode, 0) + + def test_devnull_input(self): + + async def empty_input(): + code = 'import sys; data = sys.stdin.read(); print(len(data))' + proc = await asyncio.create_subprocess_exec( + sys.executable, '-c', code, + stdin=asyncio.subprocess.DEVNULL, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + close_fds=False, + ) + stdout, stderr = await proc.communicate() + exitcode = await proc.wait() + return (stdout, exitcode) + + output, exitcode = self.loop.run_until_complete(empty_input()) + self.assertEqual(output.rstrip(), b'0') + self.assertEqual(exitcode, 0) + + def test_devnull_output(self): + + async def empty_output(): + code = 'import sys; data = sys.stdin.read(); print(len(data))' + proc = await asyncio.create_subprocess_exec( + sys.executable, '-c', code, + stdin=asyncio.subprocess.PIPE, + stdout=asyncio.subprocess.DEVNULL, + stderr=asyncio.subprocess.PIPE, + close_fds=False, + ) + stdout, stderr = await proc.communicate(b"abc") + exitcode = await proc.wait() + return (stdout, exitcode) + + output, exitcode = self.loop.run_until_complete(empty_output()) + self.assertEqual(output, None) + self.assertEqual(exitcode, 0) + + def test_devnull_error(self): + + async def empty_error(): + code = 'import sys; data = sys.stdin.read(); print(len(data))' + proc = await asyncio.create_subprocess_exec( + sys.executable, '-c', code, + stdin=asyncio.subprocess.PIPE, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.DEVNULL, + close_fds=False, + ) + stdout, stderr = await proc.communicate(b"abc") + exitcode = await proc.wait() + return (stderr, exitcode) + + output, exitcode = self.loop.run_until_complete(empty_error()) + self.assertEqual(output, None) + self.assertEqual(exitcode, 0) + + @unittest.skipIf(sys.platform not in ('linux', 'android'), + "Don't have /dev/stdin") + def test_devstdin_input(self): + + async def devstdin_input(message): + code = 'file = open("/dev/stdin"); data = file.read(); print(len(data))' + proc = await asyncio.create_subprocess_exec( + sys.executable, '-c', code, + stdin=asyncio.subprocess.PIPE, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + close_fds=False, + ) + stdout, stderr = await proc.communicate(message) + exitcode = await proc.wait() + return (stdout, exitcode) + + output, exitcode = self.loop.run_until_complete(devstdin_input(b'abc')) + self.assertEqual(output.rstrip(), b'3') + self.assertEqual(exitcode, 0) + + def test_cancel_process_wait(self): + # Issue #23140: cancel Process.wait() + + async def cancel_wait(): + proc = await asyncio.create_subprocess_exec(*PROGRAM_BLOCKED) + + # Create an internal future waiting on the process exit + task = self.loop.create_task(proc.wait()) + self.loop.call_soon(task.cancel) + try: + await task + except asyncio.CancelledError: + pass + + # Cancel the future + task.cancel() + + # Kill the process and wait until it is done + proc.kill() + await proc.wait() + + self.loop.run_until_complete(cancel_wait()) + + def test_cancel_make_subprocess_transport_exec(self): + + async def cancel_make_transport(): + coro = asyncio.create_subprocess_exec(*PROGRAM_BLOCKED) + task = self.loop.create_task(coro) + + self.loop.call_soon(task.cancel) + try: + await task + except asyncio.CancelledError: + pass + + # ignore the log: + # "Exception during subprocess creation, kill the subprocess" + with test_utils.disable_logger(): + self.loop.run_until_complete(cancel_make_transport()) + + def test_cancel_post_init(self): + + async def cancel_make_transport(): + coro = self.loop.subprocess_exec(asyncio.SubprocessProtocol, + *PROGRAM_BLOCKED) + task = self.loop.create_task(coro) + + self.loop.call_soon(task.cancel) + try: + await task + except asyncio.CancelledError: + pass + + # ignore the log: + # "Exception during subprocess creation, kill the subprocess" + with test_utils.disable_logger(): + self.loop.run_until_complete(cancel_make_transport()) + test_utils.run_briefly(self.loop) + + def test_close_kill_running(self): + + async def kill_running(): + create = self.loop.subprocess_exec(asyncio.SubprocessProtocol, + *PROGRAM_BLOCKED) + transport, protocol = await create + + kill_called = False + def kill(): + nonlocal kill_called + kill_called = True + orig_kill() + + proc = transport.get_extra_info('subprocess') + orig_kill = proc.kill + proc.kill = kill + returncode = transport.get_returncode() + transport.close() + await asyncio.wait_for(transport._wait(), 5) + return (returncode, kill_called) + + # Ignore "Close running child process: kill ..." log + with test_utils.disable_logger(): + try: + returncode, killed = self.loop.run_until_complete( + kill_running() + ) + except asyncio.TimeoutError: + self.skipTest( + "Timeout failure on waiting for subprocess stopping" + ) + self.assertIsNone(returncode) + + # transport.close() must kill the process if it is still running + self.assertTrue(killed) + test_utils.run_briefly(self.loop) + + def test_close_dont_kill_finished(self): + + async def kill_running(): + create = self.loop.subprocess_exec(asyncio.SubprocessProtocol, + *PROGRAM_BLOCKED) + transport, protocol = await create + proc = transport.get_extra_info('subprocess') + + # kill the process (but asyncio is not notified immediately) + proc.kill() + proc.wait() + + proc.kill = mock.Mock() + proc_returncode = proc.poll() + transport_returncode = transport.get_returncode() + transport.close() + return (proc_returncode, transport_returncode, proc.kill.called) + + # Ignore "Unknown child process pid ..." log of SafeChildWatcher, + # emitted because the test already consumes the exit status: + # proc.wait() + with test_utils.disable_logger(): + result = self.loop.run_until_complete(kill_running()) + test_utils.run_briefly(self.loop) + + proc_returncode, transport_return_code, killed = result + + self.assertIsNotNone(proc_returncode) + self.assertIsNone(transport_return_code) + + # transport.close() must not kill the process if it finished, even if + # the transport was not notified yet + self.assertFalse(killed) + + async def _test_popen_error(self, stdin): + if sys.platform == 'win32': + target = 'asyncio.windows_utils.Popen' + else: + target = 'subprocess.Popen' + with mock.patch(target) as popen: + exc = ZeroDivisionError + popen.side_effect = exc + + with warnings.catch_warnings(record=True) as warns: + with self.assertRaises(exc): + await asyncio.create_subprocess_exec( + sys.executable, + '-c', + 'pass', + stdin=stdin + ) + self.assertEqual(warns, []) + + def test_popen_error(self): + # Issue #24763: check that the subprocess transport is closed + # when BaseSubprocessTransport fails + self.loop.run_until_complete(self._test_popen_error(stdin=None)) + + def test_popen_error_with_stdin_pipe(self): + # Issue #35721: check that newly created socket pair is closed when + # Popen fails + self.loop.run_until_complete( + self._test_popen_error(stdin=subprocess.PIPE)) + + def test_read_stdout_after_process_exit(self): + + async def execute(): + code = '\n'.join(['import sys', + 'for _ in range(64):', + ' sys.stdout.write("x" * 4096)', + 'sys.stdout.flush()', + 'sys.exit(1)']) + + process = await asyncio.create_subprocess_exec( + sys.executable, '-c', code, + stdout=asyncio.subprocess.PIPE, + ) + + while True: + data = await process.stdout.read(65536) + if data: + await asyncio.sleep(0.3) + else: + break + + self.loop.run_until_complete(execute()) + + def test_create_subprocess_exec_text_mode_fails(self): + async def execute(): + with self.assertRaises(ValueError): + await subprocess.create_subprocess_exec(sys.executable, + text=True) + + with self.assertRaises(ValueError): + await subprocess.create_subprocess_exec(sys.executable, + encoding="utf-8") + + with self.assertRaises(ValueError): + await subprocess.create_subprocess_exec(sys.executable, + errors="strict") + + self.loop.run_until_complete(execute()) + + def test_create_subprocess_shell_text_mode_fails(self): + + async def execute(): + with self.assertRaises(ValueError): + await subprocess.create_subprocess_shell(sys.executable, + text=True) + + with self.assertRaises(ValueError): + await subprocess.create_subprocess_shell(sys.executable, + encoding="utf-8") + + with self.assertRaises(ValueError): + await subprocess.create_subprocess_shell(sys.executable, + errors="strict") + + self.loop.run_until_complete(execute()) + + def test_create_subprocess_exec_with_path(self): + async def execute(): + p = await subprocess.create_subprocess_exec( + os_helper.FakePath(sys.executable), '-c', 'pass') + await p.wait() + p = await subprocess.create_subprocess_exec( + sys.executable, '-c', 'pass', os_helper.FakePath('.')) + await p.wait() + + self.assertIsNone(self.loop.run_until_complete(execute())) + + async def check_stdout_output(self, coro, output): + proc = await coro + stdout, _ = await proc.communicate() + self.assertEqual(stdout, output) + self.assertEqual(proc.returncode, 0) + task = asyncio.create_task(proc.wait()) + await asyncio.sleep(0) + self.assertEqual(task.result(), proc.returncode) + + def test_create_subprocess_env_shell(self) -> None: + async def main() -> None: + executable = sys.executable + if sys.platform == "win32": + executable = f'"{executable}"' + cmd = f'''{executable} -c "import os, sys; sys.stdout.write(os.getenv('FOO'))"''' + env = os.environ.copy() + env["FOO"] = "bar" + proc = await asyncio.create_subprocess_shell( + cmd, env=env, stdout=subprocess.PIPE + ) + return proc + + self.loop.run_until_complete(self.check_stdout_output(main(), b'bar')) + + def test_create_subprocess_env_exec(self) -> None: + async def main() -> None: + cmd = [sys.executable, "-c", + "import os, sys; sys.stdout.write(os.getenv('FOO'))"] + env = os.environ.copy() + env["FOO"] = "baz" + proc = await asyncio.create_subprocess_exec( + *cmd, env=env, stdout=subprocess.PIPE + ) + return proc + + self.loop.run_until_complete(self.check_stdout_output(main(), b'baz')) + + + def test_subprocess_concurrent_wait(self) -> None: + async def main() -> None: + proc = await asyncio.create_subprocess_exec( + *PROGRAM_CAT, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + ) + stdout, _ = await proc.communicate(b'some data') + self.assertEqual(stdout, b"some data") + self.assertEqual(proc.returncode, 0) + self.assertEqual(await asyncio.gather(*[proc.wait() for _ in range(10)]), + [proc.returncode] * 10) + + self.loop.run_until_complete(main()) + + def test_subprocess_protocol_events(self): + # gh-108973: Test that all subprocess protocol methods are called. + # The protocol methods are not called in a deterministic order. + # The order depends on the event loop and the operating system. + events = [] + fds = [1, 2] + expected = [ + ('pipe_data_received', 1, b'stdout'), + ('pipe_data_received', 2, b'stderr'), + ('pipe_connection_lost', 1), + ('pipe_connection_lost', 2), + 'process_exited', + ] + per_fd_expected = [ + 'pipe_data_received', + 'pipe_connection_lost', + ] + + class MyProtocol(asyncio.SubprocessProtocol): + def __init__(self, exit_future: asyncio.Future) -> None: + self.exit_future = exit_future + + def pipe_data_received(self, fd, data) -> None: + events.append(('pipe_data_received', fd, data)) + self.exit_maybe() + + def pipe_connection_lost(self, fd, exc) -> None: + events.append(('pipe_connection_lost', fd)) + self.exit_maybe() + + def process_exited(self) -> None: + events.append('process_exited') + self.exit_maybe() + + def exit_maybe(self): + # Only exit when we got all expected events + if len(events) >= len(expected): + self.exit_future.set_result(True) + + async def main() -> None: + loop = asyncio.get_running_loop() + exit_future = asyncio.Future() + code = 'import sys; sys.stdout.write("stdout"); sys.stderr.write("stderr")' + transport, _ = await loop.subprocess_exec(lambda: MyProtocol(exit_future), + sys.executable, '-c', code, stdin=None) + await exit_future + transport.close() + + return events + + events = self.loop.run_until_complete(main()) + + # First, make sure that we received all events + self.assertSetEqual(set(events), set(expected)) + + # Second, check order of pipe events per file descriptor + per_fd_events = {fd: [] for fd in fds} + for event in events: + if event == 'process_exited': + continue + name, fd = event[:2] + per_fd_events[fd].append(name) + + for fd in fds: + self.assertEqual(per_fd_events[fd], per_fd_expected, (fd, events)) + + def test_subprocess_communicate_stdout(self): + # See https://github.com/python/cpython/issues/100133 + async def get_command_stdout(cmd, *args): + proc = await asyncio.create_subprocess_exec( + cmd, *args, stdout=asyncio.subprocess.PIPE, + ) + stdout, _ = await proc.communicate() + return stdout.decode().strip() + + async def main(): + outputs = [f'foo{i}' for i in range(10)] + res = await asyncio.gather(*[get_command_stdout(sys.executable, '-c', + f'print({out!r})') for out in outputs]) + self.assertEqual(res, outputs) + + self.loop.run_until_complete(main()) + + @unittest.skipIf(sys.platform != 'linux', "Linux only") + def test_subprocess_send_signal_race(self): + # See https://github.com/python/cpython/issues/87744 + async def main(): + for _ in range(10): + proc = await asyncio.create_subprocess_exec('sleep', '0.1') + await asyncio.sleep(0.1) + try: + proc.send_signal(signal.SIGUSR1) + except ProcessLookupError: + pass + self.assertNotEqual(await proc.wait(), 255) + + self.loop.run_until_complete(main()) + + @warnings_helper.ignore_warnings(category=ResourceWarning) + def test_subprocess_read_pipe_cancelled(self): + async def main(): + loop = asyncio.get_running_loop() + loop.connect_read_pipe = mock.AsyncMock(side_effect=asyncio.CancelledError) + with self.assertRaises(asyncio.CancelledError): + await asyncio.create_subprocess_exec(*PROGRAM_BLOCKED, stderr=asyncio.subprocess.PIPE) + + asyncio.run(main()) + gc_collect() + + @warnings_helper.ignore_warnings(category=ResourceWarning) + def test_subprocess_write_pipe_cancelled(self): + async def main(): + loop = asyncio.get_running_loop() + loop.connect_write_pipe = mock.AsyncMock(side_effect=asyncio.CancelledError) + with self.assertRaises(asyncio.CancelledError): + await asyncio.create_subprocess_exec(*PROGRAM_BLOCKED, stdin=asyncio.subprocess.PIPE) + + asyncio.run(main()) + gc_collect() + + @warnings_helper.ignore_warnings(category=ResourceWarning) + def test_subprocess_read_write_pipe_cancelled(self): + async def main(): + loop = asyncio.get_running_loop() + loop.connect_read_pipe = mock.AsyncMock(side_effect=asyncio.CancelledError) + loop.connect_write_pipe = mock.AsyncMock(side_effect=asyncio.CancelledError) + with self.assertRaises(asyncio.CancelledError): + await asyncio.create_subprocess_exec( + *PROGRAM_BLOCKED, + stdin=asyncio.subprocess.PIPE, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + + asyncio.run(main()) + gc_collect() + +if sys.platform != 'win32': + # Unix + class SubprocessWatcherMixin(SubprocessMixin): + + def setUp(self): + super().setUp() + self.loop = asyncio.new_event_loop() + self.set_event_loop(self.loop) + + def test_watcher_implementation(self): + loop = self.loop + watcher = loop._watcher + if unix_events.can_use_pidfd(): + self.assertIsInstance(watcher, unix_events._PidfdChildWatcher) + else: + self.assertIsInstance(watcher, unix_events._ThreadedChildWatcher) + + + class SubprocessThreadedWatcherTests(SubprocessWatcherMixin, + test_utils.TestCase): + def setUp(self): + self._original_can_use_pidfd = unix_events.can_use_pidfd + # Force the use of the threaded child watcher + unix_events.can_use_pidfd = mock.Mock(return_value=False) + super().setUp() + + def tearDown(self): + unix_events.can_use_pidfd = self._original_can_use_pidfd + return super().tearDown() + + @unittest.skipUnless( + unix_events.can_use_pidfd(), + "operating system does not support pidfds", + ) + class SubprocessPidfdWatcherTests(SubprocessWatcherMixin, + test_utils.TestCase): + + pass + +else: + # Windows + class SubprocessProactorTests(SubprocessMixin, test_utils.TestCase): + + def setUp(self): + super().setUp() + self.loop = asyncio.ProactorEventLoop() + self.set_event_loop(self.loop) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_taskgroups.py b/Lib/test/test_asyncio/test_taskgroups.py new file mode 100644 index 00000000000..d4b2554dda9 --- /dev/null +++ b/Lib/test/test_asyncio/test_taskgroups.py @@ -0,0 +1,1166 @@ +# Adapted with permission from the EdgeDB project; +# license: PSFL. + +import weakref +import sys +import gc +import asyncio +import contextvars +import contextlib +from asyncio import taskgroups +import unittest +import warnings + +from test.test_asyncio.utils import await_without_task + +# To prevent a warning "test altered the execution environment" +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +class MyExc(Exception): + pass + + +class MyBaseExc(BaseException): + pass + + +def get_error_types(eg): + return {type(exc) for exc in eg.exceptions} + + +def no_other_refs(): + # due to gh-124392 coroutines now refer to their locals + coro = asyncio.current_task().get_coro() + frame = sys._getframe(1) + while coro.cr_frame != frame: + coro = coro.cr_await + return [coro] + + +def set_gc_state(enabled): + was_enabled = gc.isenabled() + if enabled: + gc.enable() + else: + gc.disable() + return was_enabled + + +@contextlib.contextmanager +def disable_gc(): + was_enabled = set_gc_state(enabled=False) + try: + yield + finally: + set_gc_state(enabled=was_enabled) + + +class BaseTestTaskGroup: + + async def test_taskgroup_01(self): + + async def foo1(): + await asyncio.sleep(0.1) + return 42 + + async def foo2(): + await asyncio.sleep(0.2) + return 11 + + async with taskgroups.TaskGroup() as g: + t1 = g.create_task(foo1()) + t2 = g.create_task(foo2()) + + self.assertEqual(t1.result(), 42) + self.assertEqual(t2.result(), 11) + + async def test_taskgroup_02(self): + + async def foo1(): + await asyncio.sleep(0.1) + return 42 + + async def foo2(): + await asyncio.sleep(0.2) + return 11 + + async with taskgroups.TaskGroup() as g: + t1 = g.create_task(foo1()) + await asyncio.sleep(0.15) + t2 = g.create_task(foo2()) + + self.assertEqual(t1.result(), 42) + self.assertEqual(t2.result(), 11) + + async def test_taskgroup_03(self): + + async def foo1(): + await asyncio.sleep(1) + return 42 + + async def foo2(): + await asyncio.sleep(0.2) + return 11 + + async with taskgroups.TaskGroup() as g: + t1 = g.create_task(foo1()) + await asyncio.sleep(0.15) + # cancel t1 explicitly, i.e. everything should continue + # working as expected. + t1.cancel() + + t2 = g.create_task(foo2()) + + self.assertTrue(t1.cancelled()) + self.assertEqual(t2.result(), 11) + + async def test_taskgroup_04(self): + + NUM = 0 + t2_cancel = False + t2 = None + + async def foo1(): + await asyncio.sleep(0.1) + 1 / 0 + + async def foo2(): + nonlocal NUM, t2_cancel + try: + await asyncio.sleep(1) + except asyncio.CancelledError: + t2_cancel = True + raise + NUM += 1 + + async def runner(): + nonlocal NUM, t2 + + async with taskgroups.TaskGroup() as g: + g.create_task(foo1()) + t2 = g.create_task(foo2()) + + NUM += 10 + + with self.assertRaises(ExceptionGroup) as cm: + await asyncio.create_task(runner()) + + self.assertEqual(get_error_types(cm.exception), {ZeroDivisionError}) + + self.assertEqual(NUM, 0) + self.assertTrue(t2_cancel) + self.assertTrue(t2.cancelled()) + + async def test_cancel_children_on_child_error(self): + # When a child task raises an error, the rest of the children + # are cancelled and the errors are gathered into an EG. + + NUM = 0 + t2_cancel = False + runner_cancel = False + + async def foo1(): + await asyncio.sleep(0.1) + 1 / 0 + + async def foo2(): + nonlocal NUM, t2_cancel + try: + await asyncio.sleep(5) + except asyncio.CancelledError: + t2_cancel = True + raise + NUM += 1 + + async def runner(): + nonlocal NUM, runner_cancel + + async with taskgroups.TaskGroup() as g: + g.create_task(foo1()) + g.create_task(foo1()) + g.create_task(foo1()) + g.create_task(foo2()) + try: + await asyncio.sleep(10) + except asyncio.CancelledError: + runner_cancel = True + raise + + NUM += 10 + + # The 3 foo1 sub tasks can be racy when the host is busy - if the + # cancellation happens in the middle, we'll see partial sub errors here + with self.assertRaises(ExceptionGroup) as cm: + await asyncio.create_task(runner()) + + self.assertEqual(get_error_types(cm.exception), {ZeroDivisionError}) + self.assertEqual(NUM, 0) + self.assertTrue(t2_cancel) + self.assertTrue(runner_cancel) + + async def test_cancellation(self): + + NUM = 0 + + async def foo(): + nonlocal NUM + try: + await asyncio.sleep(5) + except asyncio.CancelledError: + NUM += 1 + raise + + async def runner(): + async with taskgroups.TaskGroup() as g: + for _ in range(5): + g.create_task(foo()) + + r = asyncio.create_task(runner()) + await asyncio.sleep(0.1) + + self.assertFalse(r.done()) + r.cancel() + with self.assertRaises(asyncio.CancelledError) as cm: + await r + + self.assertEqual(NUM, 5) + + async def test_taskgroup_07(self): + + NUM = 0 + + async def foo(): + nonlocal NUM + try: + await asyncio.sleep(5) + except asyncio.CancelledError: + NUM += 1 + raise + + async def runner(): + nonlocal NUM + async with taskgroups.TaskGroup() as g: + for _ in range(5): + g.create_task(foo()) + + try: + await asyncio.sleep(10) + except asyncio.CancelledError: + NUM += 10 + raise + + r = asyncio.create_task(runner()) + await asyncio.sleep(0.1) + + self.assertFalse(r.done()) + r.cancel() + with self.assertRaises(asyncio.CancelledError): + await r + + self.assertEqual(NUM, 15) + + async def test_taskgroup_08(self): + + async def foo(): + try: + await asyncio.sleep(10) + finally: + 1 / 0 + + async def runner(): + async with taskgroups.TaskGroup() as g: + for _ in range(5): + g.create_task(foo()) + + await asyncio.sleep(10) + + r = asyncio.create_task(runner()) + await asyncio.sleep(0.1) + + self.assertFalse(r.done()) + r.cancel() + with self.assertRaises(ExceptionGroup) as cm: + await r + self.assertEqual(get_error_types(cm.exception), {ZeroDivisionError}) + + async def test_taskgroup_09(self): + + t1 = t2 = None + + async def foo1(): + await asyncio.sleep(1) + return 42 + + async def foo2(): + await asyncio.sleep(2) + return 11 + + async def runner(): + nonlocal t1, t2 + async with taskgroups.TaskGroup() as g: + t1 = g.create_task(foo1()) + t2 = g.create_task(foo2()) + await asyncio.sleep(0.1) + 1 / 0 + + try: + await runner() + except ExceptionGroup as t: + self.assertEqual(get_error_types(t), {ZeroDivisionError}) + else: + self.fail('ExceptionGroup was not raised') + + self.assertTrue(t1.cancelled()) + self.assertTrue(t2.cancelled()) + + async def test_taskgroup_10(self): + + t1 = t2 = None + + async def foo1(): + await asyncio.sleep(1) + return 42 + + async def foo2(): + await asyncio.sleep(2) + return 11 + + async def runner(): + nonlocal t1, t2 + async with taskgroups.TaskGroup() as g: + t1 = g.create_task(foo1()) + t2 = g.create_task(foo2()) + 1 / 0 + + try: + await runner() + except ExceptionGroup as t: + self.assertEqual(get_error_types(t), {ZeroDivisionError}) + else: + self.fail('ExceptionGroup was not raised') + + self.assertTrue(t1.cancelled()) + self.assertTrue(t2.cancelled()) + + async def test_taskgroup_11(self): + + async def foo(): + try: + await asyncio.sleep(10) + finally: + 1 / 0 + + async def runner(): + async with taskgroups.TaskGroup(): + async with taskgroups.TaskGroup() as g2: + for _ in range(5): + g2.create_task(foo()) + + await asyncio.sleep(10) + + r = asyncio.create_task(runner()) + await asyncio.sleep(0.1) + + self.assertFalse(r.done()) + r.cancel() + with self.assertRaises(ExceptionGroup) as cm: + await r + + self.assertEqual(get_error_types(cm.exception), {ExceptionGroup}) + self.assertEqual(get_error_types(cm.exception.exceptions[0]), {ZeroDivisionError}) + + async def test_taskgroup_12(self): + + async def foo(): + try: + await asyncio.sleep(10) + finally: + 1 / 0 + + async def runner(): + async with taskgroups.TaskGroup() as g1: + g1.create_task(asyncio.sleep(10)) + + async with taskgroups.TaskGroup() as g2: + for _ in range(5): + g2.create_task(foo()) + + await asyncio.sleep(10) + + r = asyncio.create_task(runner()) + await asyncio.sleep(0.1) + + self.assertFalse(r.done()) + r.cancel() + with self.assertRaises(ExceptionGroup) as cm: + await r + + self.assertEqual(get_error_types(cm.exception), {ExceptionGroup}) + self.assertEqual(get_error_types(cm.exception.exceptions[0]), {ZeroDivisionError}) + + async def test_taskgroup_13(self): + + async def crash_after(t): + await asyncio.sleep(t) + raise ValueError(t) + + async def runner(): + async with taskgroups.TaskGroup() as g1: + g1.create_task(crash_after(0.1)) + + async with taskgroups.TaskGroup() as g2: + g2.create_task(crash_after(10)) + + r = asyncio.create_task(runner()) + with self.assertRaises(ExceptionGroup) as cm: + await r + + self.assertEqual(get_error_types(cm.exception), {ValueError}) + + async def test_taskgroup_14(self): + + async def crash_after(t): + await asyncio.sleep(t) + raise ValueError(t) + + async def runner(): + async with taskgroups.TaskGroup() as g1: + g1.create_task(crash_after(10)) + + async with taskgroups.TaskGroup() as g2: + g2.create_task(crash_after(0.1)) + + r = asyncio.create_task(runner()) + with self.assertRaises(ExceptionGroup) as cm: + await r + + self.assertEqual(get_error_types(cm.exception), {ExceptionGroup}) + self.assertEqual(get_error_types(cm.exception.exceptions[0]), {ValueError}) + + async def test_taskgroup_15(self): + + async def crash_soon(): + await asyncio.sleep(0.3) + 1 / 0 + + async def runner(): + async with taskgroups.TaskGroup() as g1: + g1.create_task(crash_soon()) + try: + await asyncio.sleep(10) + except asyncio.CancelledError: + await asyncio.sleep(0.5) + raise + + r = asyncio.create_task(runner()) + await asyncio.sleep(0.1) + + self.assertFalse(r.done()) + r.cancel() + with self.assertRaises(ExceptionGroup) as cm: + await r + self.assertEqual(get_error_types(cm.exception), {ZeroDivisionError}) + + async def test_taskgroup_16(self): + + async def crash_soon(): + await asyncio.sleep(0.3) + 1 / 0 + + async def nested_runner(): + async with taskgroups.TaskGroup() as g1: + g1.create_task(crash_soon()) + try: + await asyncio.sleep(10) + except asyncio.CancelledError: + await asyncio.sleep(0.5) + raise + + async def runner(): + t = asyncio.create_task(nested_runner()) + await t + + r = asyncio.create_task(runner()) + await asyncio.sleep(0.1) + + self.assertFalse(r.done()) + r.cancel() + with self.assertRaises(ExceptionGroup) as cm: + await r + self.assertEqual(get_error_types(cm.exception), {ZeroDivisionError}) + + async def test_taskgroup_17(self): + NUM = 0 + + async def runner(): + nonlocal NUM + async with taskgroups.TaskGroup(): + try: + await asyncio.sleep(10) + except asyncio.CancelledError: + NUM += 10 + raise + + r = asyncio.create_task(runner()) + await asyncio.sleep(0.1) + + self.assertFalse(r.done()) + r.cancel() + with self.assertRaises(asyncio.CancelledError): + await r + + self.assertEqual(NUM, 10) + + async def test_taskgroup_18(self): + NUM = 0 + + async def runner(): + nonlocal NUM + async with taskgroups.TaskGroup(): + try: + await asyncio.sleep(10) + except asyncio.CancelledError: + NUM += 10 + # This isn't a good idea, but we have to support + # this weird case. + raise MyExc + + r = asyncio.create_task(runner()) + await asyncio.sleep(0.1) + + self.assertFalse(r.done()) + r.cancel() + + try: + await r + except ExceptionGroup as t: + self.assertEqual(get_error_types(t),{MyExc}) + else: + self.fail('ExceptionGroup was not raised') + + self.assertEqual(NUM, 10) + + async def test_taskgroup_19(self): + async def crash_soon(): + await asyncio.sleep(0.1) + 1 / 0 + + async def nested(): + try: + await asyncio.sleep(10) + finally: + raise MyExc + + async def runner(): + async with taskgroups.TaskGroup() as g: + g.create_task(crash_soon()) + await nested() + + r = asyncio.create_task(runner()) + try: + await r + except ExceptionGroup as t: + self.assertEqual(get_error_types(t), {MyExc, ZeroDivisionError}) + else: + self.fail('TasgGroupError was not raised') + + async def test_taskgroup_20(self): + async def crash_soon(): + await asyncio.sleep(0.1) + 1 / 0 + + async def nested(): + try: + await asyncio.sleep(10) + finally: + raise KeyboardInterrupt + + async def runner(): + async with taskgroups.TaskGroup() as g: + g.create_task(crash_soon()) + await nested() + + with self.assertRaises(KeyboardInterrupt): + await runner() + + async def test_taskgroup_20a(self): + async def crash_soon(): + await asyncio.sleep(0.1) + 1 / 0 + + async def nested(): + try: + await asyncio.sleep(10) + finally: + raise MyBaseExc + + async def runner(): + async with taskgroups.TaskGroup() as g: + g.create_task(crash_soon()) + await nested() + + with self.assertRaises(BaseExceptionGroup) as cm: + await runner() + + self.assertEqual( + get_error_types(cm.exception), {MyBaseExc, ZeroDivisionError} + ) + + async def _test_taskgroup_21(self): + # This test doesn't work as asyncio, currently, doesn't + # correctly propagate KeyboardInterrupt (or SystemExit) -- + # those cause the event loop itself to crash. + # (Compare to the previous (passing) test -- that one raises + # a plain exception but raises KeyboardInterrupt in nested(); + # this test does it the other way around.) + + async def crash_soon(): + await asyncio.sleep(0.1) + raise KeyboardInterrupt + + async def nested(): + try: + await asyncio.sleep(10) + finally: + raise TypeError + + async def runner(): + async with taskgroups.TaskGroup() as g: + g.create_task(crash_soon()) + await nested() + + with self.assertRaises(KeyboardInterrupt): + await runner() + + async def test_taskgroup_21a(self): + + async def crash_soon(): + await asyncio.sleep(0.1) + raise MyBaseExc + + async def nested(): + try: + await asyncio.sleep(10) + finally: + raise TypeError + + async def runner(): + async with taskgroups.TaskGroup() as g: + g.create_task(crash_soon()) + await nested() + + with self.assertRaises(BaseExceptionGroup) as cm: + await runner() + + self.assertEqual(get_error_types(cm.exception), {MyBaseExc, TypeError}) + + async def test_taskgroup_22(self): + + async def foo1(): + await asyncio.sleep(1) + return 42 + + async def foo2(): + await asyncio.sleep(2) + return 11 + + async def runner(): + async with taskgroups.TaskGroup() as g: + g.create_task(foo1()) + g.create_task(foo2()) + + r = asyncio.create_task(runner()) + await asyncio.sleep(0.05) + r.cancel() + + with self.assertRaises(asyncio.CancelledError): + await r + + async def test_taskgroup_23(self): + + async def do_job(delay): + await asyncio.sleep(delay) + + async with taskgroups.TaskGroup() as g: + for count in range(10): + await asyncio.sleep(0.1) + g.create_task(do_job(0.3)) + if count == 5: + self.assertLess(len(g._tasks), 5) + await asyncio.sleep(1.35) + self.assertEqual(len(g._tasks), 0) + + async def test_taskgroup_24(self): + + async def root(g): + await asyncio.sleep(0.1) + g.create_task(coro1(0.1)) + g.create_task(coro1(0.2)) + + async def coro1(delay): + await asyncio.sleep(delay) + + async def runner(): + async with taskgroups.TaskGroup() as g: + g.create_task(root(g)) + + await runner() + + async def test_taskgroup_25(self): + nhydras = 0 + + async def hydra(g): + nonlocal nhydras + nhydras += 1 + await asyncio.sleep(0.01) + g.create_task(hydra(g)) + g.create_task(hydra(g)) + + async def hercules(): + while nhydras < 10: + await asyncio.sleep(0.015) + 1 / 0 + + async def runner(): + async with taskgroups.TaskGroup() as g: + g.create_task(hydra(g)) + g.create_task(hercules()) + + with self.assertRaises(ExceptionGroup) as cm: + await runner() + + self.assertEqual(get_error_types(cm.exception), {ZeroDivisionError}) + self.assertGreaterEqual(nhydras, 10) + + async def test_taskgroup_task_name(self): + async def coro(): + await asyncio.sleep(0) + async with taskgroups.TaskGroup() as g: + t = g.create_task(coro(), name="yolo") + self.assertEqual(t.get_name(), "yolo") + + async def test_taskgroup_task_context(self): + cvar = contextvars.ContextVar('cvar') + + async def coro(val): + await asyncio.sleep(0) + cvar.set(val) + + async with taskgroups.TaskGroup() as g: + ctx = contextvars.copy_context() + self.assertIsNone(ctx.get(cvar)) + t1 = g.create_task(coro(1), context=ctx) + await t1 + self.assertEqual(1, ctx.get(cvar)) + t2 = g.create_task(coro(2), context=ctx) + await t2 + self.assertEqual(2, ctx.get(cvar)) + + async def test_taskgroup_no_create_task_after_failure(self): + async def coro1(): + await asyncio.sleep(0.001) + 1 / 0 + async def coro2(g): + try: + await asyncio.sleep(1) + except asyncio.CancelledError: + with self.assertRaises(RuntimeError): + g.create_task(coro1()) + + with self.assertRaises(ExceptionGroup) as cm: + async with taskgroups.TaskGroup() as g: + g.create_task(coro1()) + g.create_task(coro2(g)) + + self.assertEqual(get_error_types(cm.exception), {ZeroDivisionError}) + + async def test_taskgroup_context_manager_exit_raises(self): + # See https://github.com/python/cpython/issues/95289 + class CustomException(Exception): + pass + + async def raise_exc(): + raise CustomException + + @contextlib.asynccontextmanager + async def database(): + try: + yield + finally: + raise CustomException + + async def main(): + task = asyncio.current_task() + try: + async with taskgroups.TaskGroup() as tg: + async with database(): + tg.create_task(raise_exc()) + await asyncio.sleep(1) + except* CustomException as err: + self.assertEqual(task.cancelling(), 0) + self.assertEqual(len(err.exceptions), 2) + + else: + self.fail('CustomException not raised') + + await asyncio.create_task(main()) + + async def test_taskgroup_already_entered(self): + tg = taskgroups.TaskGroup() + async with tg: + with self.assertRaisesRegex(RuntimeError, "has already been entered"): + async with tg: + pass + + async def test_taskgroup_double_enter(self): + tg = taskgroups.TaskGroup() + async with tg: + pass + with self.assertRaisesRegex(RuntimeError, "has already been entered"): + async with tg: + pass + + async def test_taskgroup_finished(self): + async def create_task_after_tg_finish(): + tg = taskgroups.TaskGroup() + async with tg: + pass + coro = asyncio.sleep(0) + with self.assertRaisesRegex(RuntimeError, "is finished"): + tg.create_task(coro) + + # Make sure the coroutine was closed when submitted to the inactive tg + # (if not closed, a RuntimeWarning should have been raised) + with warnings.catch_warnings(record=True) as w: + await create_task_after_tg_finish() + self.assertEqual(len(w), 0) + + async def test_taskgroup_not_entered(self): + tg = taskgroups.TaskGroup() + coro = asyncio.sleep(0) + with self.assertRaisesRegex(RuntimeError, "has not been entered"): + tg.create_task(coro) + + async def test_taskgroup_without_parent_task(self): + tg = taskgroups.TaskGroup() + with self.assertRaisesRegex(RuntimeError, "parent task"): + await await_without_task(tg.__aenter__()) + coro = asyncio.sleep(0) + with self.assertRaisesRegex(RuntimeError, "has not been entered"): + tg.create_task(coro) + + async def test_coro_closed_when_tg_closed(self): + async def run_coro_after_tg_closes(): + async with taskgroups.TaskGroup() as tg: + pass + coro = asyncio.sleep(0) + with self.assertRaisesRegex(RuntimeError, "is finished"): + tg.create_task(coro) + + await run_coro_after_tg_closes() + + async def test_cancelling_level_preserved(self): + async def raise_after(t, e): + await asyncio.sleep(t) + raise e() + + try: + async with asyncio.TaskGroup() as tg: + tg.create_task(raise_after(0.0, RuntimeError)) + except* RuntimeError: + pass + self.assertEqual(asyncio.current_task().cancelling(), 0) + + async def test_nested_groups_both_cancelled(self): + async def raise_after(t, e): + await asyncio.sleep(t) + raise e() + + try: + async with asyncio.TaskGroup() as outer_tg: + try: + async with asyncio.TaskGroup() as inner_tg: + inner_tg.create_task(raise_after(0, RuntimeError)) + outer_tg.create_task(raise_after(0, ValueError)) + except* RuntimeError: + pass + else: + self.fail("RuntimeError not raised") + self.assertEqual(asyncio.current_task().cancelling(), 1) + except* ValueError: + pass + else: + self.fail("ValueError not raised") + self.assertEqual(asyncio.current_task().cancelling(), 0) + + async def test_error_and_cancel(self): + event = asyncio.Event() + + async def raise_error(): + event.set() + await asyncio.sleep(0) + raise RuntimeError() + + async def inner(): + try: + async with taskgroups.TaskGroup() as tg: + tg.create_task(raise_error()) + await asyncio.sleep(1) + self.fail("Sleep in group should have been cancelled") + except* RuntimeError: + self.assertEqual(asyncio.current_task().cancelling(), 1) + self.assertEqual(asyncio.current_task().cancelling(), 1) + await asyncio.sleep(1) + self.fail("Sleep after group should have been cancelled") + + async def outer(): + t = asyncio.create_task(inner()) + await event.wait() + self.assertEqual(t.cancelling(), 0) + t.cancel() + self.assertEqual(t.cancelling(), 1) + with self.assertRaises(asyncio.CancelledError): + await t + self.assertTrue(t.cancelled()) + + await outer() + + async def test_exception_refcycles_direct(self): + """Test that TaskGroup doesn't keep a reference to the raised ExceptionGroup""" + tg = asyncio.TaskGroup() + exc = None + + class _Done(Exception): + pass + + try: + async with tg: + raise _Done + except ExceptionGroup as e: + exc = e + + self.assertIsNotNone(exc) + self.assertListEqual(gc.get_referrers(exc), no_other_refs()) + + + async def test_exception_refcycles_errors(self): + """Test that TaskGroup deletes self._errors, and __aexit__ args""" + tg = asyncio.TaskGroup() + exc = None + + class _Done(Exception): + pass + + try: + async with tg: + raise _Done + except* _Done as excs: + exc = excs.exceptions[0] + + self.assertIsInstance(exc, _Done) + self.assertListEqual(gc.get_referrers(exc), no_other_refs()) + + + async def test_exception_refcycles_parent_task(self): + """Test that TaskGroup deletes self._parent_task""" + tg = asyncio.TaskGroup() + exc = None + + class _Done(Exception): + pass + + async def coro_fn(): + async with tg: + raise _Done + + try: + async with asyncio.TaskGroup() as tg2: + tg2.create_task(coro_fn()) + except* _Done as excs: + exc = excs.exceptions[0].exceptions[0] + + self.assertIsInstance(exc, _Done) + self.assertListEqual(gc.get_referrers(exc), no_other_refs()) + + + async def test_exception_refcycles_parent_task_wr(self): + """Test that TaskGroup deletes self._parent_task and create_task() deletes task""" + tg = asyncio.TaskGroup() + exc = None + + class _Done(Exception): + pass + + async def coro_fn(): + async with tg: + raise _Done + + with disable_gc(): + try: + async with asyncio.TaskGroup() as tg2: + task_wr = weakref.ref(tg2.create_task(coro_fn())) + except* _Done as excs: + exc = excs.exceptions[0].exceptions[0] + + self.assertIsNone(task_wr()) + self.assertIsInstance(exc, _Done) + self.assertListEqual(gc.get_referrers(exc), no_other_refs()) + + async def test_exception_refcycles_propagate_cancellation_error(self): + """Test that TaskGroup deletes propagate_cancellation_error""" + tg = asyncio.TaskGroup() + exc = None + + try: + async with asyncio.timeout(-1): + async with tg: + await asyncio.sleep(0) + except TimeoutError as e: + exc = e.__cause__ + + self.assertIsInstance(exc, asyncio.CancelledError) + self.assertListEqual(gc.get_referrers(exc), no_other_refs()) + + async def test_exception_refcycles_base_error(self): + """Test that TaskGroup deletes self._base_error""" + class MyKeyboardInterrupt(KeyboardInterrupt): + pass + + tg = asyncio.TaskGroup() + exc = None + + try: + async with tg: + raise MyKeyboardInterrupt + except MyKeyboardInterrupt as e: + exc = e + + self.assertIsNotNone(exc) + self.assertListEqual(gc.get_referrers(exc), no_other_refs()) + + async def test_name(self): + name = None + + async def asyncfn(): + nonlocal name + name = asyncio.current_task().get_name() + + async with asyncio.TaskGroup() as tg: + tg.create_task(asyncfn(), name="example name") + + self.assertEqual(name, "example name") + + + async def test_cancels_task_if_created_during_creation(self): + # regression test for gh-128550 + ran = False + class MyError(Exception): + pass + + exc = None + try: + async with asyncio.TaskGroup() as tg: + async def third_task(): + raise MyError("third task failed") + + async def second_task(): + nonlocal ran + tg.create_task(third_task()) + with self.assertRaises(asyncio.CancelledError): + await asyncio.sleep(0) # eager tasks cancel here + await asyncio.sleep(0) # lazy tasks cancel here + ran = True + + tg.create_task(second_task()) + except* MyError as excs: + exc = excs.exceptions[0] + + self.assertTrue(ran) + self.assertIsInstance(exc, MyError) + + + async def test_cancellation_does_not_leak_out_of_tg(self): + class MyError(Exception): + pass + + async def throw_error(): + raise MyError + + try: + async with asyncio.TaskGroup() as tg: + tg.create_task(throw_error()) + except* MyError: + pass + else: + self.fail("should have raised one MyError in group") + + # if this test fails this current task will be cancelled + # outside the task group and inside unittest internals + # we yield to the event loop with sleep(0) so that + # cancellation happens here and error is more understandable + await asyncio.sleep(0) + + +class TestTaskGroup(BaseTestTaskGroup, unittest.IsolatedAsyncioTestCase): + loop_factory = asyncio.EventLoop + + @unittest.expectedFailure # TODO: RUSTPYTHON; Test that TaskGroup deletes propagate_cancellation_error + async def test_exception_refcycles_propagate_cancellation_error(self): + return await super().test_exception_refcycles_propagate_cancellation_error() + + @unittest.expectedFailure # TODO: RUSTPYTHON; Test that TaskGroup deletes self._base_error + async def test_exception_refcycles_base_error(self): + return await super().test_exception_refcycles_base_error() + + @unittest.expectedFailure # TODO: RUSTPYTHON; Test that TaskGroup deletes self._errors, and __aexit__ args + async def test_exception_refcycles_errors(self): + return await super().test_exception_refcycles_errors() + + @unittest.expectedFailure # TODO: RUSTPYTHON; Test that TaskGroup deletes self._parent_task + async def test_exception_refcycles_parent_task(self): + return await super().test_exception_refcycles_parent_task() + + @unittest.expectedFailure # TODO: RUSTPYTHON; Test that TaskGroup deletes self._parent_task and create_task() deletes task + async def test_exception_refcycles_parent_task_wr(self): + return await super().test_exception_refcycles_parent_task_wr() + + @unittest.expectedFailure # TODO: RUSTPYTHON; Test that TaskGroup doesn't keep a reference to the raised ExceptionGroup + async def test_exception_refcycles_direct(self): + return await super().test_exception_refcycles_direct() + +class TestEagerTaskTaskGroup(BaseTestTaskGroup, unittest.IsolatedAsyncioTestCase): + @staticmethod + def loop_factory(): + loop = asyncio.EventLoop() + loop.set_task_factory(asyncio.eager_task_factory) + return loop + + @unittest.expectedFailure # TODO: RUSTPYTHON; Test that TaskGroup deletes propagate_cancellation_error + async def test_exception_refcycles_propagate_cancellation_error(self): + return await super().test_exception_refcycles_propagate_cancellation_error() + + @unittest.expectedFailure # TODO: RUSTPYTHON; Test that TaskGroup deletes self._base_error + async def test_exception_refcycles_base_error(self): + return await super().test_exception_refcycles_base_error() + + @unittest.expectedFailure # TODO: RUSTPYTHON; Test that TaskGroup deletes self._errors, and __aexit__ args + async def test_exception_refcycles_errors(self): + return await super().test_exception_refcycles_errors() + + @unittest.expectedFailure # TODO: RUSTPYTHON; Test that TaskGroup deletes self._parent_task + async def test_exception_refcycles_parent_task(self): + return await super().test_exception_refcycles_parent_task() + + @unittest.expectedFailure # TODO: RUSTPYTHON; Test that TaskGroup deletes self._parent_task and create_task() deletes task + async def test_exception_refcycles_parent_task_wr(self): + return await super().test_exception_refcycles_parent_task_wr() + + @unittest.expectedFailure # TODO: RUSTPYTHON; Test that TaskGroup doesn't keep a reference to the raised ExceptionGroup + async def test_exception_refcycles_direct(self): + return await super().test_exception_refcycles_direct() + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py new file mode 100644 index 00000000000..b0ca67d6716 --- /dev/null +++ b/Lib/test/test_asyncio/test_tasks.py @@ -0,0 +1,3761 @@ +"""Tests for tasks.py.""" + +import collections +import contextlib +import contextvars +import gc +import io +import random +import re +import sys +import traceback +import types +import unittest +from unittest import mock +from types import GenericAlias + +import asyncio +from asyncio import futures +from asyncio import tasks +from test.test_asyncio import utils as test_utils +from test import support +from test.support.script_helper import assert_python_ok +from test.support.warnings_helper import ignore_warnings + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +async def coroutine_function(): + pass + + +def format_coroutine(qualname, state, src, source_traceback, generator=False): + if generator: + state = '%s' % state + else: + state = '%s, defined' % state + if source_traceback is not None: + frame = source_traceback[-1] + return ('coro=<%s() %s at %s> created at %s:%s' + % (qualname, state, src, frame[0], frame[1])) + else: + return 'coro=<%s() %s at %s>' % (qualname, state, src) + + +def get_innermost_context(exc): + """ + Return information about the innermost exception context in the chain. + """ + depth = 0 + while True: + context = exc.__context__ + if context is None: + break + + exc = context + depth += 1 + + return (type(exc), exc.args, depth) + + +class Dummy: + + def __repr__(self): + return '' + + def __call__(self, *args): + pass + + +class CoroLikeObject: + def send(self, v): + raise StopIteration(42) + + def throw(self, *exc): + pass + + def close(self): + pass + + def __await__(self): + return self + + +class BaseTaskTests: + + Task = None + Future = None + all_tasks = None + + def new_task(self, loop, coro, name='TestTask', context=None, eager_start=None): + return self.__class__.Task(coro, loop=loop, name=name, context=context, eager_start=eager_start) + + def new_future(self, loop): + return self.__class__.Future(loop=loop) + + def setUp(self): + super().setUp() + self.loop = self.new_test_loop() + self.loop.set_task_factory(self.new_task) + self.loop.create_future = lambda: self.new_future(self.loop) + + def test_generic_alias(self): + task = self.__class__.Task[str] + self.assertEqual(task.__args__, (str,)) + self.assertIsInstance(task, GenericAlias) + + def test_task_cancel_message_getter(self): + async def coro(): + pass + t = self.new_task(self.loop, coro()) + self.assertHasAttr(t, '_cancel_message') + self.assertEqual(t._cancel_message, None) + + t.cancel('my message') + self.assertEqual(t._cancel_message, 'my message') + + with self.assertRaises(asyncio.CancelledError) as cm: + self.loop.run_until_complete(t) + + self.assertEqual('my message', cm.exception.args[0]) + + def test_task_cancel_message_setter(self): + async def coro(): + pass + t = self.new_task(self.loop, coro()) + t.cancel('my message') + t._cancel_message = 'my new message' + self.assertEqual(t._cancel_message, 'my new message') + + with self.assertRaises(asyncio.CancelledError) as cm: + self.loop.run_until_complete(t) + + self.assertEqual('my new message', cm.exception.args[0]) + + def test_task_del_collect(self): + class Evil: + def __del__(self): + gc.collect() + + async def run(): + return Evil() + + self.loop.run_until_complete( + asyncio.gather(*[ + self.new_task(self.loop, run()) for _ in range(100) + ])) + + def test_other_loop_future(self): + other_loop = asyncio.new_event_loop() + fut = self.new_future(other_loop) + + async def run(fut): + await fut + + try: + with self.assertRaisesRegex(RuntimeError, + r'Task .* got Future .* attached'): + self.loop.run_until_complete(run(fut)) + finally: + other_loop.close() + + def test_task_awaits_on_itself(self): + + async def test(): + await task + + task = asyncio.ensure_future(test(), loop=self.loop) + + with self.assertRaisesRegex(RuntimeError, + 'Task cannot await on itself'): + self.loop.run_until_complete(task) + + def test_task_class(self): + async def notmuch(): + return 'ok' + t = self.new_task(self.loop, notmuch()) + self.loop.run_until_complete(t) + self.assertTrue(t.done()) + self.assertEqual(t.result(), 'ok') + self.assertIs(t._loop, self.loop) + self.assertIs(t.get_loop(), self.loop) + + loop = asyncio.new_event_loop() + self.set_event_loop(loop) + t = self.new_task(loop, notmuch()) + self.assertIs(t._loop, loop) + loop.run_until_complete(t) + loop.close() + + def test_ensure_future_coroutine(self): + async def notmuch(): + return 'ok' + t = asyncio.ensure_future(notmuch(), loop=self.loop) + self.assertIs(t._loop, self.loop) + self.loop.run_until_complete(t) + self.assertTrue(t.done()) + self.assertEqual(t.result(), 'ok') + + a = notmuch() + self.addCleanup(a.close) + with self.assertRaisesRegex(RuntimeError, 'no current event loop'): + asyncio.ensure_future(a) + + async def test(): + return asyncio.ensure_future(notmuch()) + t = self.loop.run_until_complete(test()) + self.assertIs(t._loop, self.loop) + self.loop.run_until_complete(t) + self.assertTrue(t.done()) + self.assertEqual(t.result(), 'ok') + + # Deprecated in 3.10, undeprecated in 3.12 + asyncio.set_event_loop(self.loop) + self.addCleanup(asyncio.set_event_loop, None) + t = asyncio.ensure_future(notmuch()) + self.assertIs(t._loop, self.loop) + self.loop.run_until_complete(t) + self.assertTrue(t.done()) + self.assertEqual(t.result(), 'ok') + + def test_ensure_future_future(self): + f_orig = self.new_future(self.loop) + f_orig.set_result('ko') + + f = asyncio.ensure_future(f_orig) + self.loop.run_until_complete(f) + self.assertTrue(f.done()) + self.assertEqual(f.result(), 'ko') + self.assertIs(f, f_orig) + + loop = asyncio.new_event_loop() + self.set_event_loop(loop) + + with self.assertRaises(ValueError): + f = asyncio.ensure_future(f_orig, loop=loop) + + loop.close() + + f = asyncio.ensure_future(f_orig, loop=self.loop) + self.assertIs(f, f_orig) + + def test_ensure_future_task(self): + async def notmuch(): + return 'ok' + t_orig = self.new_task(self.loop, notmuch()) + t = asyncio.ensure_future(t_orig) + self.loop.run_until_complete(t) + self.assertTrue(t.done()) + self.assertEqual(t.result(), 'ok') + self.assertIs(t, t_orig) + + loop = asyncio.new_event_loop() + self.set_event_loop(loop) + + with self.assertRaises(ValueError): + t = asyncio.ensure_future(t_orig, loop=loop) + + loop.close() + + t = asyncio.ensure_future(t_orig, loop=self.loop) + self.assertIs(t, t_orig) + + def test_ensure_future_awaitable(self): + class Aw: + def __init__(self, coro): + self.coro = coro + def __await__(self): + return self.coro.__await__() + + async def coro(): + return 'ok' + + loop = asyncio.new_event_loop() + self.set_event_loop(loop) + fut = asyncio.ensure_future(Aw(coro()), loop=loop) + loop.run_until_complete(fut) + self.assertEqual(fut.result(), 'ok') + + def test_ensure_future_task_awaitable(self): + class Aw: + def __await__(self): + return asyncio.sleep(0, result='ok').__await__() + + loop = asyncio.new_event_loop() + self.set_event_loop(loop) + task = asyncio.ensure_future(Aw(), loop=loop) + loop.run_until_complete(task) + self.assertTrue(task.done()) + self.assertEqual(task.result(), 'ok') + self.assertIsInstance(task.get_coro(), types.CoroutineType) + loop.close() + + def test_ensure_future_neither(self): + with self.assertRaises(TypeError): + asyncio.ensure_future('ok') + + def test_ensure_future_error_msg(self): + loop = asyncio.new_event_loop() + f = self.new_future(self.loop) + with self.assertRaisesRegex(ValueError, 'The future belongs to a ' + 'different loop than the one specified as ' + 'the loop argument'): + asyncio.ensure_future(f, loop=loop) + loop.close() + + def test_get_stack(self): + T = None + + async def foo(): + await bar() + + async def bar(): + # test get_stack() + f = T.get_stack(limit=1) + try: + self.assertEqual(f[0].f_code.co_name, 'foo') + finally: + f = None + + # test print_stack() + file = io.StringIO() + T.print_stack(limit=1, file=file) + file.seek(0) + tb = file.read() + self.assertRegex(tb, r'foo\(\) running') + + async def runner(): + nonlocal T + T = asyncio.ensure_future(foo(), loop=self.loop) + await T + + self.loop.run_until_complete(runner()) + + def test_task_repr(self): + self.loop.set_debug(False) + + async def notmuch(): + return 'abc' + + # test coroutine function + self.assertEqual(notmuch.__name__, 'notmuch') + self.assertRegex(notmuch.__qualname__, + r'\w+.test_task_repr..notmuch') + self.assertEqual(notmuch.__module__, __name__) + + filename, lineno = test_utils.get_function_source(notmuch) + src = "%s:%s" % (filename, lineno) + + # test coroutine object + gen = notmuch() + coro_qualname = 'BaseTaskTests.test_task_repr..notmuch' + self.assertEqual(gen.__name__, 'notmuch') + self.assertEqual(gen.__qualname__, coro_qualname) + + # test pending Task + t = self.new_task(self.loop, gen) + t.add_done_callback(Dummy()) + + coro = format_coroutine(coro_qualname, 'running', src, + t._source_traceback, generator=True) + self.assertEqual(repr(t), + "()]>" % coro) + + # test cancelling Task + t.cancel() # Does not take immediate effect! + self.assertEqual(repr(t), + "()]>" % coro) + + # test cancelled Task + self.assertRaises(asyncio.CancelledError, + self.loop.run_until_complete, t) + coro = format_coroutine(coro_qualname, 'done', src, + t._source_traceback) + self.assertEqual(repr(t), + "" % coro) + + # test finished Task + t = self.new_task(self.loop, notmuch()) + self.loop.run_until_complete(t) + coro = format_coroutine(coro_qualname, 'done', src, + t._source_traceback) + self.assertEqual(repr(t), + "" % coro) + + def test_task_repr_autogenerated(self): + async def notmuch(): + return 123 + + t1 = self.new_task(self.loop, notmuch(), None) + t2 = self.new_task(self.loop, notmuch(), None) + self.assertNotEqual(repr(t1), repr(t2)) + + match1 = re.match(r"^' % re.escape(repr(fut))) + + fut.set_result(None) + self.loop.run_until_complete(task) + + def test_task_basics(self): + + async def outer(): + a = await inner1() + b = await inner2() + return a+b + + async def inner1(): + return 42 + + async def inner2(): + return 1000 + + t = outer() + self.assertEqual(self.loop.run_until_complete(t), 1042) + + def test_exception_chaining_after_await(self): + # Test that when awaiting on a task when an exception is already + # active, if the task raises an exception it will be chained + # with the original. + loop = asyncio.new_event_loop() + self.set_event_loop(loop) + + async def raise_error(): + raise ValueError + + async def run(): + try: + raise KeyError(3) + except Exception as exc: + task = self.new_task(loop, raise_error()) + try: + await task + except Exception as exc: + self.assertEqual(type(exc), ValueError) + chained = exc.__context__ + self.assertEqual((type(chained), chained.args), + (KeyError, (3,))) + + try: + task = self.new_task(loop, run()) + loop.run_until_complete(task) + finally: + loop.close() + + def test_exception_chaining_after_await_with_context_cycle(self): + # Check trying to create an exception context cycle: + # https://bugs.python.org/issue40696 + has_cycle = None + loop = asyncio.new_event_loop() + self.set_event_loop(loop) + + async def process_exc(exc): + raise exc + + async def run(): + nonlocal has_cycle + try: + raise KeyError('a') + except Exception as exc: + task = self.new_task(loop, process_exc(exc)) + try: + await task + except BaseException as exc: + has_cycle = (exc is exc.__context__) + # Prevent a hang if has_cycle is True. + exc.__context__ = None + + try: + task = self.new_task(loop, run()) + loop.run_until_complete(task) + finally: + loop.close() + # This also distinguishes from the initial has_cycle=None. + self.assertEqual(has_cycle, False) + + + def test_cancelling(self): + loop = asyncio.new_event_loop() + + async def task(): + await asyncio.sleep(10) + + try: + t = self.new_task(loop, task()) + self.assertFalse(t.cancelling()) + self.assertNotIn(" cancelling ", repr(t)) + self.assertTrue(t.cancel()) + self.assertTrue(t.cancelling()) + self.assertIn(" cancelling ", repr(t)) + + # Since we commented out two lines from Task.cancel(), + # this t.cancel() call now returns True. + # self.assertFalse(t.cancel()) + self.assertTrue(t.cancel()) + + with self.assertRaises(asyncio.CancelledError): + loop.run_until_complete(t) + finally: + loop.close() + + def test_uncancel_basic(self): + loop = asyncio.new_event_loop() + + async def task(): + try: + await asyncio.sleep(10) + except asyncio.CancelledError: + self.current_task().uncancel() + await asyncio.sleep(10) + + try: + t = self.new_task(loop, task()) + loop.run_until_complete(asyncio.sleep(0.01)) + + # Cancel first sleep + self.assertTrue(t.cancel()) + self.assertIn(" cancelling ", repr(t)) + self.assertEqual(t.cancelling(), 1) + self.assertFalse(t.cancelled()) # Task is still not complete + loop.run_until_complete(asyncio.sleep(0.01)) + + # after .uncancel() + self.assertNotIn(" cancelling ", repr(t)) + self.assertEqual(t.cancelling(), 0) + self.assertFalse(t.cancelled()) # Task is still not complete + + # Cancel second sleep + self.assertTrue(t.cancel()) + self.assertEqual(t.cancelling(), 1) + self.assertFalse(t.cancelled()) # Task is still not complete + with self.assertRaises(asyncio.CancelledError): + loop.run_until_complete(t) + self.assertTrue(t.cancelled()) # Finally, task complete + self.assertTrue(t.done()) + + # uncancel is no longer effective after the task is complete + t.uncancel() + self.assertTrue(t.cancelled()) + self.assertTrue(t.done()) + finally: + loop.close() + + def test_uncancel_structured_blocks(self): + # This test recreates the following high-level structure using uncancel():: + # + # async def make_request_with_timeout(): + # try: + # async with asyncio.timeout(1): + # # Structured block affected by the timeout: + # await make_request() + # await make_another_request() + # except TimeoutError: + # pass # There was a timeout + # # Outer code not affected by the timeout: + # await unrelated_code() + + loop = asyncio.new_event_loop() + + async def make_request_with_timeout(*, sleep: float, timeout: float): + task = self.current_task() + loop = task.get_loop() + + timed_out = False + structured_block_finished = False + outer_code_reached = False + + def on_timeout(): + nonlocal timed_out + timed_out = True + task.cancel() + + timeout_handle = loop.call_later(timeout, on_timeout) + try: + try: + # Structured block affected by the timeout + await asyncio.sleep(sleep) + structured_block_finished = True + finally: + timeout_handle.cancel() + if ( + timed_out + and task.uncancel() == 0 + and type(sys.exception()) is asyncio.CancelledError + ): + # Note the five rules that are needed here to satisfy proper + # uncancellation: + # + # 1. handle uncancellation in a `finally:` block to allow for + # plain returns; + # 2. our `timed_out` flag is set, meaning that it was our event + # that triggered the need to uncancel the task, regardless of + # what exception is raised; + # 3. we can call `uncancel()` because *we* called `cancel()` + # before; + # 4. we call `uncancel()` but we only continue converting the + # CancelledError to TimeoutError if `uncancel()` caused the + # cancellation request count go down to 0. We need to look + # at the counter vs having a simple boolean flag because our + # code might have been nested (think multiple timeouts). See + # commit 7fce1063b6e5a366f8504e039a8ccdd6944625cd for + # details. + # 5. we only convert CancelledError to TimeoutError; for other + # exceptions raised due to the cancellation (like + # a ConnectionLostError from a database client), simply + # propagate them. + # + # Those checks need to take place in this exact order to make + # sure the `cancelling()` counter always stays in sync. + # + # Additionally, the original stimulus to `cancel()` the task + # needs to be unscheduled to avoid re-cancelling the task later. + # Here we do it by cancelling `timeout_handle` in the `finally:` + # block. + raise TimeoutError + except TimeoutError: + self.assertTrue(timed_out) + + # Outer code not affected by the timeout: + outer_code_reached = True + await asyncio.sleep(0) + return timed_out, structured_block_finished, outer_code_reached + + try: + # Test which timed out. + t1 = self.new_task(loop, make_request_with_timeout(sleep=10.0, timeout=0.1)) + timed_out, structured_block_finished, outer_code_reached = ( + loop.run_until_complete(t1) + ) + self.assertTrue(timed_out) + self.assertFalse(structured_block_finished) # it was cancelled + self.assertTrue(outer_code_reached) # task got uncancelled after leaving + # the structured block and continued until + # completion + self.assertEqual(t1.cancelling(), 0) # no pending cancellation of the outer task + + # Test which did not time out. + t2 = self.new_task(loop, make_request_with_timeout(sleep=0, timeout=10.0)) + timed_out, structured_block_finished, outer_code_reached = ( + loop.run_until_complete(t2) + ) + self.assertFalse(timed_out) + self.assertTrue(structured_block_finished) + self.assertTrue(outer_code_reached) + self.assertEqual(t2.cancelling(), 0) + finally: + loop.close() + + def test_uncancel_resets_must_cancel(self): + + async def coro(): + await fut + return 42 + + loop = asyncio.new_event_loop() + fut = asyncio.Future(loop=loop) + task = self.new_task(loop, coro()) + loop.run_until_complete(asyncio.sleep(0)) # Get task waiting for fut + fut.set_result(None) # Make task runnable + try: + task.cancel() # Enter cancelled state + self.assertEqual(task.cancelling(), 1) + self.assertTrue(task._must_cancel) + + task.uncancel() # Undo cancellation + self.assertEqual(task.cancelling(), 0) + self.assertFalse(task._must_cancel) + finally: + res = loop.run_until_complete(task) + self.assertEqual(res, 42) + loop.close() + + def test_cancel(self): + + def gen(): + when = yield + self.assertAlmostEqual(10.0, when) + yield 0 + + loop = self.new_test_loop(gen) + + async def task(): + await asyncio.sleep(10.0) + return 12 + + t = self.new_task(loop, task()) + loop.call_soon(t.cancel) + with self.assertRaises(asyncio.CancelledError): + loop.run_until_complete(t) + self.assertTrue(t.done()) + self.assertTrue(t.cancelled()) + self.assertFalse(t.cancel()) + + def test_cancel_with_message_then_future_result(self): + # Test Future.result() after calling cancel() with a message. + cases = [ + ((), ()), + ((None,), ()), + (('my message',), ('my message',)), + # Non-string values should roundtrip. + ((5,), (5,)), + ] + for cancel_args, expected_args in cases: + with self.subTest(cancel_args=cancel_args): + loop = asyncio.new_event_loop() + self.set_event_loop(loop) + + async def sleep(): + await asyncio.sleep(10) + + async def coro(): + task = self.new_task(loop, sleep()) + await asyncio.sleep(0) + task.cancel(*cancel_args) + done, pending = await asyncio.wait([task]) + task.result() + + task = self.new_task(loop, coro()) + with self.assertRaises(asyncio.CancelledError) as cm: + loop.run_until_complete(task) + exc = cm.exception + self.assertEqual(exc.args, expected_args) + + actual = get_innermost_context(exc) + self.assertEqual(actual, + (asyncio.CancelledError, expected_args, 0)) + + def test_cancel_with_message_then_future_exception(self): + # Test Future.exception() after calling cancel() with a message. + cases = [ + ((), ()), + ((None,), ()), + (('my message',), ('my message',)), + # Non-string values should roundtrip. + ((5,), (5,)), + ] + for cancel_args, expected_args in cases: + with self.subTest(cancel_args=cancel_args): + loop = asyncio.new_event_loop() + self.set_event_loop(loop) + + async def sleep(): + await asyncio.sleep(10) + + async def coro(): + task = self.new_task(loop, sleep()) + await asyncio.sleep(0) + task.cancel(*cancel_args) + done, pending = await asyncio.wait([task]) + task.exception() + + task = self.new_task(loop, coro()) + with self.assertRaises(asyncio.CancelledError) as cm: + loop.run_until_complete(task) + exc = cm.exception + self.assertEqual(exc.args, expected_args) + + actual = get_innermost_context(exc) + self.assertEqual(actual, + (asyncio.CancelledError, expected_args, 0)) + + def test_cancellation_exception_context(self): + loop = asyncio.new_event_loop() + self.set_event_loop(loop) + fut = loop.create_future() + + async def sleep(): + fut.set_result(None) + await asyncio.sleep(10) + + async def coro(): + inner_task = self.new_task(loop, sleep()) + await fut + loop.call_soon(inner_task.cancel, 'msg') + try: + await inner_task + except asyncio.CancelledError as ex: + raise ValueError("cancelled") from ex + + task = self.new_task(loop, coro()) + with self.assertRaises(ValueError) as cm: + loop.run_until_complete(task) + exc = cm.exception + self.assertEqual(exc.args, ('cancelled',)) + + actual = get_innermost_context(exc) + self.assertEqual(actual, + (asyncio.CancelledError, ('msg',), 1)) + + def test_cancel_with_message_before_starting_task(self): + loop = asyncio.new_event_loop() + self.set_event_loop(loop) + + async def sleep(): + await asyncio.sleep(10) + + async def coro(): + task = self.new_task(loop, sleep()) + # We deliberately leave out the sleep here. + task.cancel('my message') + done, pending = await asyncio.wait([task]) + task.exception() + + task = self.new_task(loop, coro()) + with self.assertRaises(asyncio.CancelledError) as cm: + loop.run_until_complete(task) + exc = cm.exception + self.assertEqual(exc.args, ('my message',)) + + actual = get_innermost_context(exc) + self.assertEqual(actual, + (asyncio.CancelledError, ('my message',), 0)) + + def test_cancel_yield(self): + async def task(): + await asyncio.sleep(0) + await asyncio.sleep(0) + return 12 + + t = self.new_task(self.loop, task()) + test_utils.run_briefly(self.loop) # start coro + t.cancel() + self.assertRaises( + asyncio.CancelledError, self.loop.run_until_complete, t) + self.assertTrue(t.done()) + self.assertTrue(t.cancelled()) + self.assertFalse(t.cancel()) + + def test_cancel_inner_future(self): + f = self.new_future(self.loop) + + async def task(): + await f + return 12 + + t = self.new_task(self.loop, task()) + test_utils.run_briefly(self.loop) # start task + f.cancel() + with self.assertRaises(asyncio.CancelledError): + self.loop.run_until_complete(t) + self.assertTrue(f.cancelled()) + self.assertTrue(t.cancelled()) + + def test_cancel_both_task_and_inner_future(self): + f = self.new_future(self.loop) + + async def task(): + await f + return 12 + + t = self.new_task(self.loop, task()) + test_utils.run_briefly(self.loop) + + f.cancel() + t.cancel() + + with self.assertRaises(asyncio.CancelledError): + self.loop.run_until_complete(t) + + self.assertTrue(t.done()) + self.assertTrue(f.cancelled()) + self.assertTrue(t.cancelled()) + + def test_cancel_task_catching(self): + fut1 = self.new_future(self.loop) + fut2 = self.new_future(self.loop) + + async def task(): + await fut1 + try: + await fut2 + except asyncio.CancelledError: + return 42 + + t = self.new_task(self.loop, task()) + test_utils.run_briefly(self.loop) + self.assertIs(t._fut_waiter, fut1) # White-box test. + fut1.set_result(None) + test_utils.run_briefly(self.loop) + self.assertIs(t._fut_waiter, fut2) # White-box test. + t.cancel() + self.assertTrue(fut2.cancelled()) + res = self.loop.run_until_complete(t) + self.assertEqual(res, 42) + self.assertFalse(t.cancelled()) + + def test_cancel_task_ignoring(self): + fut1 = self.new_future(self.loop) + fut2 = self.new_future(self.loop) + fut3 = self.new_future(self.loop) + + async def task(): + await fut1 + try: + await fut2 + except asyncio.CancelledError: + pass + res = await fut3 + return res + + t = self.new_task(self.loop, task()) + test_utils.run_briefly(self.loop) + self.assertIs(t._fut_waiter, fut1) # White-box test. + fut1.set_result(None) + test_utils.run_briefly(self.loop) + self.assertIs(t._fut_waiter, fut2) # White-box test. + t.cancel() + self.assertTrue(fut2.cancelled()) + test_utils.run_briefly(self.loop) + self.assertIs(t._fut_waiter, fut3) # White-box test. + fut3.set_result(42) + res = self.loop.run_until_complete(t) + self.assertEqual(res, 42) + self.assertFalse(fut3.cancelled()) + self.assertFalse(t.cancelled()) + + def test_cancel_current_task(self): + loop = asyncio.new_event_loop() + self.set_event_loop(loop) + + async def task(): + t.cancel() + self.assertTrue(t._must_cancel) # White-box test. + # The sleep should be cancelled immediately. + await asyncio.sleep(100) + return 12 + + t = self.new_task(loop, task()) + self.assertFalse(t.cancelled()) + self.assertRaises( + asyncio.CancelledError, loop.run_until_complete, t) + self.assertTrue(t.done()) + self.assertTrue(t.cancelled()) + self.assertFalse(t._must_cancel) # White-box test. + self.assertFalse(t.cancel()) + + def test_cancel_at_end(self): + """coroutine end right after task is cancelled""" + loop = asyncio.new_event_loop() + self.set_event_loop(loop) + + async def task(): + t.cancel() + self.assertTrue(t._must_cancel) # White-box test. + return 12 + + t = self.new_task(loop, task()) + self.assertFalse(t.cancelled()) + self.assertRaises( + asyncio.CancelledError, loop.run_until_complete, t) + self.assertTrue(t.done()) + self.assertTrue(t.cancelled()) + self.assertFalse(t._must_cancel) # White-box test. + self.assertFalse(t.cancel()) + + def test_cancel_awaited_task(self): + # This tests for a relatively rare condition when + # a task cancellation is requested for a task which is not + # currently blocked, such as a task cancelling itself. + # In this situation we must ensure that whatever next future + # or task the cancelled task blocks on is cancelled correctly + # as well. See also bpo-34872. + loop = asyncio.new_event_loop() + self.addCleanup(lambda: loop.close()) + + task = nested_task = None + fut = self.new_future(loop) + + async def nested(): + await fut + + async def coro(): + nonlocal nested_task + # Create a sub-task and wait for it to run. + nested_task = self.new_task(loop, nested()) + await asyncio.sleep(0) + + # Request the current task to be cancelled. + task.cancel() + # Block on the nested task, which should be immediately + # cancelled. + await nested_task + + task = self.new_task(loop, coro()) + with self.assertRaises(asyncio.CancelledError): + loop.run_until_complete(task) + + self.assertTrue(task.cancelled()) + self.assertTrue(nested_task.cancelled()) + self.assertTrue(fut.cancelled()) + + def assert_text_contains(self, text, substr): + if substr not in text: + raise RuntimeError(f'text {substr!r} not found in:\n>>>{text}<<<') + + def test_cancel_traceback_for_future_result(self): + # When calling Future.result() on a cancelled task, check that the + # line of code that was interrupted is included in the traceback. + loop = asyncio.new_event_loop() + self.set_event_loop(loop) + + async def nested(): + # This will get cancelled immediately. + await asyncio.sleep(10) + + async def coro(): + task = self.new_task(loop, nested()) + await asyncio.sleep(0) + task.cancel() + await task # search target + + task = self.new_task(loop, coro()) + try: + loop.run_until_complete(task) + except asyncio.CancelledError: + tb = traceback.format_exc() + self.assert_text_contains(tb, "await asyncio.sleep(10)") + # The intermediate await should also be included. + self.assert_text_contains(tb, "await task # search target") + else: + self.fail('CancelledError did not occur') + + def test_cancel_traceback_for_future_exception(self): + # When calling Future.exception() on a cancelled task, check that the + # line of code that was interrupted is included in the traceback. + loop = asyncio.new_event_loop() + self.set_event_loop(loop) + + async def nested(): + # This will get cancelled immediately. + await asyncio.sleep(10) + + async def coro(): + task = self.new_task(loop, nested()) + await asyncio.sleep(0) + task.cancel() + done, pending = await asyncio.wait([task]) + task.exception() # search target + + task = self.new_task(loop, coro()) + try: + loop.run_until_complete(task) + except asyncio.CancelledError: + tb = traceback.format_exc() + self.assert_text_contains(tb, "await asyncio.sleep(10)") + # The intermediate await should also be included. + self.assert_text_contains(tb, + "task.exception() # search target") + else: + self.fail('CancelledError did not occur') + + def test_stop_while_run_in_complete(self): + + def gen(): + when = yield + self.assertAlmostEqual(0.1, when) + when = yield 0.1 + self.assertAlmostEqual(0.2, when) + when = yield 0.1 + self.assertAlmostEqual(0.3, when) + yield 0.1 + + loop = self.new_test_loop(gen) + + x = 0 + + async def task(): + nonlocal x + while x < 10: + await asyncio.sleep(0.1) + x += 1 + if x == 2: + loop.stop() + + t = self.new_task(loop, task()) + with self.assertRaises(RuntimeError) as cm: + loop.run_until_complete(t) + self.assertEqual(str(cm.exception), + 'Event loop stopped before Future completed.') + self.assertFalse(t.done()) + self.assertEqual(x, 2) + self.assertAlmostEqual(0.3, loop.time()) + + t.cancel() + self.assertRaises(asyncio.CancelledError, loop.run_until_complete, t) + + def test_log_traceback(self): + async def coro(): + pass + + task = self.new_task(self.loop, coro()) + with self.assertRaisesRegex(ValueError, 'can only be set to False'): + task._log_traceback = True + self.loop.run_until_complete(task) + + def test_wait(self): + + def gen(): + when = yield + self.assertAlmostEqual(0.1, when) + when = yield 0 + self.assertAlmostEqual(0.15, when) + yield 0.15 + + loop = self.new_test_loop(gen) + + a = self.new_task(loop, asyncio.sleep(0.1)) + b = self.new_task(loop, asyncio.sleep(0.15)) + + async def foo(): + done, pending = await asyncio.wait([b, a]) + self.assertEqual(done, set([a, b])) + self.assertEqual(pending, set()) + return 42 + + res = loop.run_until_complete(self.new_task(loop, foo())) + self.assertEqual(res, 42) + self.assertAlmostEqual(0.15, loop.time()) + + # Doing it again should take no time and exercise a different path. + res = loop.run_until_complete(self.new_task(loop, foo())) + self.assertAlmostEqual(0.15, loop.time()) + self.assertEqual(res, 42) + + def test_wait_duplicate_coroutines(self): + + async def coro(s): + return s + c = self.loop.create_task(coro('test')) + task = self.new_task( + self.loop, + asyncio.wait([c, c, self.loop.create_task(coro('spam'))])) + + done, pending = self.loop.run_until_complete(task) + + self.assertFalse(pending) + self.assertEqual(set(f.result() for f in done), {'test', 'spam'}) + + def test_wait_errors(self): + self.assertRaises( + ValueError, self.loop.run_until_complete, + asyncio.wait(set())) + + # -1 is an invalid return_when value + sleep_coro = asyncio.sleep(10.0) + wait_coro = asyncio.wait([sleep_coro], return_when=-1) + self.assertRaises(ValueError, + self.loop.run_until_complete, wait_coro) + + sleep_coro.close() + + def test_wait_first_completed(self): + + def gen(): + when = yield + self.assertAlmostEqual(10.0, when) + when = yield 0 + self.assertAlmostEqual(0.1, when) + yield 0.1 + + loop = self.new_test_loop(gen) + + a = self.new_task(loop, asyncio.sleep(10.0)) + b = self.new_task(loop, asyncio.sleep(0.1)) + task = self.new_task( + loop, + asyncio.wait([b, a], return_when=asyncio.FIRST_COMPLETED)) + + done, pending = loop.run_until_complete(task) + self.assertEqual({b}, done) + self.assertEqual({a}, pending) + self.assertFalse(a.done()) + self.assertTrue(b.done()) + self.assertIsNone(b.result()) + self.assertAlmostEqual(0.1, loop.time()) + + # move forward to close generator + loop.advance_time(10) + loop.run_until_complete(asyncio.wait([a, b])) + + def test_wait_really_done(self): + # there is possibility that some tasks in the pending list + # became done but their callbacks haven't all been called yet + + async def coro1(): + await asyncio.sleep(0) + + async def coro2(): + await asyncio.sleep(0) + await asyncio.sleep(0) + + a = self.new_task(self.loop, coro1()) + b = self.new_task(self.loop, coro2()) + task = self.new_task( + self.loop, + asyncio.wait([b, a], return_when=asyncio.FIRST_COMPLETED)) + + done, pending = self.loop.run_until_complete(task) + self.assertEqual({a, b}, done) + self.assertTrue(a.done()) + self.assertIsNone(a.result()) + self.assertTrue(b.done()) + self.assertIsNone(b.result()) + + def test_wait_first_exception(self): + + def gen(): + when = yield + self.assertAlmostEqual(10.0, when) + yield 0 + + loop = self.new_test_loop(gen) + + # first_exception, task already has exception + a = self.new_task(loop, asyncio.sleep(10.0)) + + async def exc(): + raise ZeroDivisionError('err') + + b = self.new_task(loop, exc()) + task = self.new_task( + loop, + asyncio.wait([b, a], return_when=asyncio.FIRST_EXCEPTION)) + + done, pending = loop.run_until_complete(task) + self.assertEqual({b}, done) + self.assertEqual({a}, pending) + self.assertAlmostEqual(0, loop.time()) + + # move forward to close generator + loop.advance_time(10) + loop.run_until_complete(asyncio.wait([a, b])) + + def test_wait_first_exception_in_wait(self): + + def gen(): + when = yield + self.assertAlmostEqual(10.0, when) + when = yield 0 + self.assertAlmostEqual(0.01, when) + yield 0.01 + + loop = self.new_test_loop(gen) + + # first_exception, exception during waiting + a = self.new_task(loop, asyncio.sleep(10.0)) + + async def exc(): + await asyncio.sleep(0.01) + raise ZeroDivisionError('err') + + b = self.new_task(loop, exc()) + task = asyncio.wait([b, a], return_when=asyncio.FIRST_EXCEPTION) + + done, pending = loop.run_until_complete(task) + self.assertEqual({b}, done) + self.assertEqual({a}, pending) + self.assertAlmostEqual(0.01, loop.time()) + + # move forward to close generator + loop.advance_time(10) + loop.run_until_complete(asyncio.wait([a, b])) + + def test_wait_with_exception(self): + + def gen(): + when = yield + self.assertAlmostEqual(0.1, when) + when = yield 0 + self.assertAlmostEqual(0.15, when) + yield 0.15 + + loop = self.new_test_loop(gen) + + a = self.new_task(loop, asyncio.sleep(0.1)) + + async def sleeper(): + await asyncio.sleep(0.15) + raise ZeroDivisionError('really') + + b = self.new_task(loop, sleeper()) + + async def foo(): + done, pending = await asyncio.wait([b, a]) + self.assertEqual(len(done), 2) + self.assertEqual(pending, set()) + errors = set(f for f in done if f.exception() is not None) + self.assertEqual(len(errors), 1) + + loop.run_until_complete(self.new_task(loop, foo())) + self.assertAlmostEqual(0.15, loop.time()) + + loop.run_until_complete(self.new_task(loop, foo())) + self.assertAlmostEqual(0.15, loop.time()) + + def test_wait_with_timeout(self): + + def gen(): + when = yield + self.assertAlmostEqual(0.1, when) + when = yield 0 + self.assertAlmostEqual(0.15, when) + when = yield 0 + self.assertAlmostEqual(0.11, when) + yield 0.11 + + loop = self.new_test_loop(gen) + + a = self.new_task(loop, asyncio.sleep(0.1)) + b = self.new_task(loop, asyncio.sleep(0.15)) + + async def foo(): + done, pending = await asyncio.wait([b, a], timeout=0.11) + self.assertEqual(done, set([a])) + self.assertEqual(pending, set([b])) + + loop.run_until_complete(self.new_task(loop, foo())) + self.assertAlmostEqual(0.11, loop.time()) + + # move forward to close generator + loop.advance_time(10) + loop.run_until_complete(asyncio.wait([a, b])) + + def test_wait_concurrent_complete(self): + + def gen(): + when = yield + self.assertAlmostEqual(0.1, when) + when = yield 0 + self.assertAlmostEqual(0.15, when) + when = yield 0 + self.assertAlmostEqual(0.1, when) + yield 0.1 + + loop = self.new_test_loop(gen) + + a = self.new_task(loop, asyncio.sleep(0.1)) + b = self.new_task(loop, asyncio.sleep(0.15)) + + done, pending = loop.run_until_complete( + asyncio.wait([b, a], timeout=0.1)) + + self.assertEqual(done, set([a])) + self.assertEqual(pending, set([b])) + self.assertAlmostEqual(0.1, loop.time()) + + # move forward to close generator + loop.advance_time(10) + loop.run_until_complete(asyncio.wait([a, b])) + + def test_wait_with_iterator_of_tasks(self): + + def gen(): + when = yield + self.assertAlmostEqual(0.1, when) + when = yield 0 + self.assertAlmostEqual(0.15, when) + yield 0.15 + + loop = self.new_test_loop(gen) + + a = self.new_task(loop, asyncio.sleep(0.1)) + b = self.new_task(loop, asyncio.sleep(0.15)) + + async def foo(): + done, pending = await asyncio.wait(iter([b, a])) + self.assertEqual(done, set([a, b])) + self.assertEqual(pending, set()) + return 42 + + res = loop.run_until_complete(self.new_task(loop, foo())) + self.assertEqual(res, 42) + self.assertAlmostEqual(0.15, loop.time()) + + + def test_wait_generator(self): + async def func(a): + return a + + loop = self.new_test_loop() + + async def main(): + tasks = (self.new_task(loop, func(i)) for i in range(10)) + done, pending = await asyncio.wait(tasks, return_when=asyncio.ALL_COMPLETED) + self.assertEqual(len(done), 10) + self.assertEqual(len(pending), 0) + + loop.run_until_complete(main()) + + + def test_as_completed(self): + + def gen(): + yield 0 + yield 0 + yield 0.01 + yield 0 + + async def sleeper(dt, x): + nonlocal time_shifted + await asyncio.sleep(dt) + completed.add(x) + if not time_shifted and 'a' in completed and 'b' in completed: + time_shifted = True + loop.advance_time(0.14) + return x + + async def try_iterator(awaitables): + values = [] + for f in asyncio.as_completed(awaitables): + values.append(await f) + return values + + async def try_async_iterator(awaitables): + values = [] + async for f in asyncio.as_completed(awaitables): + values.append(await f) + return values + + for foo in try_iterator, try_async_iterator: + with self.subTest(method=foo.__name__): + loop = self.new_test_loop(gen) + # disable "slow callback" warning + loop.slow_callback_duration = 1.0 + + completed = set() + time_shifted = False + + a = sleeper(0.01, 'a') + b = sleeper(0.01, 'b') + c = sleeper(0.15, 'c') + + res = loop.run_until_complete(self.new_task(loop, foo([b, c, a]))) + self.assertAlmostEqual(0.15, loop.time()) + self.assertTrue('a' in res[:2]) + self.assertTrue('b' in res[:2]) + self.assertEqual(res[2], 'c') + + def test_as_completed_same_tasks_in_as_out(self): + # Ensures that asynchronously iterating as_completed's iterator + # yields awaitables are the same awaitables that were passed in when + # those awaitables are futures. + async def try_async_iterator(awaitables): + awaitables_out = set() + async for out_aw in asyncio.as_completed(awaitables): + awaitables_out.add(out_aw) + return awaitables_out + + async def coro(i): + return i + + with contextlib.closing(asyncio.new_event_loop()) as loop: + # Coroutines shouldn't be yielded back as finished coroutines + # can't be re-used. + awaitables_in = frozenset( + (coro(0), coro(1), coro(2), coro(3)) + ) + awaitables_out = loop.run_until_complete( + try_async_iterator(awaitables_in) + ) + if awaitables_in - awaitables_out != awaitables_in: + raise self.failureException('Got original coroutines ' + 'out of as_completed iterator.') + + # Tasks should be yielded back. + coro_obj_a = coro('a') + task_b = loop.create_task(coro('b')) + coro_obj_c = coro('c') + task_d = loop.create_task(coro('d')) + awaitables_in = frozenset( + (coro_obj_a, task_b, coro_obj_c, task_d) + ) + awaitables_out = loop.run_until_complete( + try_async_iterator(awaitables_in) + ) + if awaitables_in & awaitables_out != {task_b, task_d}: + raise self.failureException('Only tasks should be yielded ' + 'from as_completed iterator ' + 'as-is.') + + def test_as_completed_with_timeout(self): + + def gen(): + yield + yield 0 + yield 0 + yield 0.1 + + async def try_iterator(): + values = [] + for f in asyncio.as_completed([a, b], timeout=0.12): + if values: + loop.advance_time(0.02) + try: + v = await f + values.append((1, v)) + except asyncio.TimeoutError as exc: + values.append((2, exc)) + return values + + async def try_async_iterator(): + values = [] + try: + async for f in asyncio.as_completed([a, b], timeout=0.12): + v = await f + values.append((1, v)) + loop.advance_time(0.02) + except asyncio.TimeoutError as exc: + values.append((2, exc)) + return values + + for foo in try_iterator, try_async_iterator: + with self.subTest(method=foo.__name__): + loop = self.new_test_loop(gen) + a = loop.create_task(asyncio.sleep(0.1, 'a')) + b = loop.create_task(asyncio.sleep(0.15, 'b')) + + res = loop.run_until_complete(self.new_task(loop, foo())) + self.assertEqual(len(res), 2, res) + self.assertEqual(res[0], (1, 'a')) + self.assertEqual(res[1][0], 2) + self.assertIsInstance(res[1][1], asyncio.TimeoutError) + self.assertAlmostEqual(0.12, loop.time()) + + # move forward to close generator + loop.advance_time(10) + loop.run_until_complete(asyncio.wait([a, b])) + + def test_as_completed_with_unused_timeout(self): + + def gen(): + yield + yield 0 + yield 0.01 + + async def try_iterator(): + for f in asyncio.as_completed([a], timeout=1): + v = await f + self.assertEqual(v, 'a') + + async def try_async_iterator(): + async for f in asyncio.as_completed([a], timeout=1): + v = await f + self.assertEqual(v, 'a') + + for foo in try_iterator, try_async_iterator: + with self.subTest(method=foo.__name__): + a = asyncio.sleep(0.01, 'a') + loop = self.new_test_loop(gen) + loop.run_until_complete(self.new_task(loop, foo())) + loop.close() + + def test_as_completed_resume_iterator(self): + # Test that as_completed returns an iterator that can be resumed + # the next time iteration is performed (i.e. if __iter__ is called + # again) + async def try_iterator(awaitables): + iterations = 0 + iterator = asyncio.as_completed(awaitables) + collected = [] + for f in iterator: + collected.append(await f) + iterations += 1 + if iterations == 2: + break + self.assertEqual(len(collected), 2) + + # Resume same iterator: + for f in iterator: + collected.append(await f) + return collected + + async def try_async_iterator(awaitables): + iterations = 0 + iterator = asyncio.as_completed(awaitables) + collected = [] + async for f in iterator: + collected.append(await f) + iterations += 1 + if iterations == 2: + break + self.assertEqual(len(collected), 2) + + # Resume same iterator: + async for f in iterator: + collected.append(await f) + return collected + + async def coro(i): + return i + + with contextlib.closing(asyncio.new_event_loop()) as loop: + for foo in try_iterator, try_async_iterator: + with self.subTest(method=foo.__name__): + results = loop.run_until_complete( + foo((coro(0), coro(1), coro(2), coro(3))) + ) + self.assertCountEqual(results, (0, 1, 2, 3)) + + def test_as_completed_reverse_wait(self): + # Tests the plain iterator style of as_completed iteration to + # ensure that the first future awaited resolves to the first + # completed awaitable from the set we passed in, even if it wasn't + # the first future generated by as_completed. + def gen(): + yield 0 + yield 0.05 + yield 0 + + loop = self.new_test_loop(gen) + + a = asyncio.sleep(0.05, 'a') + b = asyncio.sleep(0.10, 'b') + fs = {a, b} + + async def test(): + futs = list(asyncio.as_completed(fs)) + self.assertEqual(len(futs), 2) + + x = await futs[1] + self.assertEqual(x, 'a') + self.assertAlmostEqual(0.05, loop.time()) + loop.advance_time(0.05) + y = await futs[0] + self.assertEqual(y, 'b') + self.assertAlmostEqual(0.10, loop.time()) + + loop.run_until_complete(test()) + + def test_as_completed_concurrent(self): + # Ensure that more than one future or coroutine yielded from + # as_completed can be awaited concurrently. + def gen(): + when = yield + self.assertAlmostEqual(0.05, when) + when = yield 0 + self.assertAlmostEqual(0.05, when) + yield 0.05 + + async def try_iterator(fs): + return list(asyncio.as_completed(fs)) + + async def try_async_iterator(fs): + return [f async for f in asyncio.as_completed(fs)] + + for runner in try_iterator, try_async_iterator: + with self.subTest(method=runner.__name__): + a = asyncio.sleep(0.05, 'a') + b = asyncio.sleep(0.05, 'b') + fs = {a, b} + + async def test(): + futs = await runner(fs) + self.assertEqual(len(futs), 2) + done, pending = await asyncio.wait( + [asyncio.ensure_future(fut) for fut in futs] + ) + self.assertEqual(set(f.result() for f in done), {'a', 'b'}) + + loop = self.new_test_loop(gen) + loop.run_until_complete(test()) + + def test_as_completed_duplicate_coroutines(self): + + async def coro(s): + return s + + async def try_iterator(): + result = [] + c = coro('ham') + for f in asyncio.as_completed([c, c, coro('spam')]): + result.append(await f) + return result + + async def try_async_iterator(): + result = [] + c = coro('ham') + async for f in asyncio.as_completed([c, c, coro('spam')]): + result.append(await f) + return result + + for runner in try_iterator, try_async_iterator: + with self.subTest(method=runner.__name__): + fut = self.new_task(self.loop, runner()) + self.loop.run_until_complete(fut) + result = fut.result() + self.assertEqual(set(result), {'ham', 'spam'}) + self.assertEqual(len(result), 2) + + def test_as_completed_coroutine_without_loop(self): + async def coro(): + return 42 + + a = coro() + self.addCleanup(a.close) + + with self.assertRaisesRegex(RuntimeError, 'no current event loop'): + futs = asyncio.as_completed([a]) + list(futs) + + def test_as_completed_coroutine_use_running_loop(self): + loop = self.new_test_loop() + + async def coro(): + return 42 + + async def test(): + futs = list(asyncio.as_completed([coro()])) + self.assertEqual(len(futs), 1) + self.assertEqual(await futs[0], 42) + + loop.run_until_complete(test()) + + def test_sleep(self): + + def gen(): + when = yield + self.assertAlmostEqual(0.05, when) + when = yield 0.05 + self.assertAlmostEqual(0.1, when) + yield 0.05 + + loop = self.new_test_loop(gen) + + async def sleeper(dt, arg): + await asyncio.sleep(dt/2) + res = await asyncio.sleep(dt/2, arg) + return res + + t = self.new_task(loop, sleeper(0.1, 'yeah')) + loop.run_until_complete(t) + self.assertTrue(t.done()) + self.assertEqual(t.result(), 'yeah') + self.assertAlmostEqual(0.1, loop.time()) + + def test_sleep_when_delay_is_nan(self): + + def gen(): + yield + + loop = self.new_test_loop(gen) + + async def sleeper(): + await asyncio.sleep(float("nan")) + + t = self.new_task(loop, sleeper()) + + with self.assertRaises(ValueError): + loop.run_until_complete(t) + + def test_sleep_cancel(self): + + def gen(): + when = yield + self.assertAlmostEqual(10.0, when) + yield 0 + + loop = self.new_test_loop(gen) + + t = self.new_task(loop, asyncio.sleep(10.0, 'yeah')) + + handle = None + orig_call_later = loop.call_later + + def call_later(delay, callback, *args): + nonlocal handle + handle = orig_call_later(delay, callback, *args) + return handle + + loop.call_later = call_later + test_utils.run_briefly(loop) + + self.assertFalse(handle._cancelled) + + t.cancel() + test_utils.run_briefly(loop) + self.assertTrue(handle._cancelled) + + def test_task_cancel_sleeping_task(self): + + def gen(): + when = yield + self.assertAlmostEqual(0.1, when) + when = yield 0 + self.assertAlmostEqual(5000, when) + yield 0.1 + + loop = self.new_test_loop(gen) + + async def sleep(dt): + await asyncio.sleep(dt) + + async def doit(): + sleeper = self.new_task(loop, sleep(5000)) + loop.call_later(0.1, sleeper.cancel) + try: + await sleeper + except asyncio.CancelledError: + return 'cancelled' + else: + return 'slept in' + + doer = doit() + self.assertEqual(loop.run_until_complete(doer), 'cancelled') + self.assertAlmostEqual(0.1, loop.time()) + + def test_task_cancel_waiter_future(self): + fut = self.new_future(self.loop) + + async def coro(): + await fut + + task = self.new_task(self.loop, coro()) + test_utils.run_briefly(self.loop) + self.assertIs(task._fut_waiter, fut) + + task.cancel() + test_utils.run_briefly(self.loop) + self.assertRaises( + asyncio.CancelledError, self.loop.run_until_complete, task) + self.assertIsNone(task._fut_waiter) + self.assertTrue(fut.cancelled()) + + def test_task_set_methods(self): + async def notmuch(): + return 'ko' + + gen = notmuch() + task = self.new_task(self.loop, gen) + + with self.assertRaisesRegex(RuntimeError, 'not support set_result'): + task.set_result('ok') + + with self.assertRaisesRegex(RuntimeError, 'not support set_exception'): + task.set_exception(ValueError()) + + self.assertEqual( + self.loop.run_until_complete(task), + 'ko') + + def test_step_result_future(self): + # If coroutine returns future, task waits on this future. + + class Fut(asyncio.Future): + def __init__(self, *args, **kwds): + self.cb_added = False + super().__init__(*args, **kwds) + + def add_done_callback(self, *args, **kwargs): + self.cb_added = True + super().add_done_callback(*args, **kwargs) + + fut = Fut(loop=self.loop) + result = None + + async def wait_for_future(): + nonlocal result + result = await fut + + t = self.new_task(self.loop, wait_for_future()) + test_utils.run_briefly(self.loop) + self.assertTrue(fut.cb_added) + + res = object() + fut.set_result(res) + test_utils.run_briefly(self.loop) + self.assertIs(res, result) + self.assertTrue(t.done()) + self.assertIsNone(t.result()) + + def test_baseexception_during_cancel(self): + + def gen(): + when = yield + self.assertAlmostEqual(10.0, when) + yield 0 + + loop = self.new_test_loop(gen) + + async def sleeper(): + await asyncio.sleep(10) + + base_exc = SystemExit() + + async def notmutch(): + try: + await sleeper() + except asyncio.CancelledError: + raise base_exc + + task = self.new_task(loop, notmutch()) + test_utils.run_briefly(loop) + + task.cancel() + self.assertFalse(task.done()) + + self.assertRaises(SystemExit, test_utils.run_briefly, loop) + + self.assertTrue(task.done()) + self.assertFalse(task.cancelled()) + self.assertIs(task.exception(), base_exc) + + @ignore_warnings(category=DeprecationWarning) + def test_iscoroutinefunction(self): + def fn(): + pass + + self.assertFalse(asyncio.iscoroutinefunction(fn)) + + def fn1(): + yield + self.assertFalse(asyncio.iscoroutinefunction(fn1)) + + async def fn2(): + pass + self.assertTrue(asyncio.iscoroutinefunction(fn2)) + + self.assertFalse(asyncio.iscoroutinefunction(mock.Mock())) + self.assertTrue(asyncio.iscoroutinefunction(mock.AsyncMock())) + + @ignore_warnings(category=DeprecationWarning) + def test_coroutine_non_gen_function(self): + async def func(): + return 'test' + + self.assertTrue(asyncio.iscoroutinefunction(func)) + + coro = func() + self.assertTrue(asyncio.iscoroutine(coro)) + + res = self.loop.run_until_complete(coro) + self.assertEqual(res, 'test') + + def test_coroutine_non_gen_function_return_future(self): + fut = self.new_future(self.loop) + + async def func(): + return fut + + async def coro(): + fut.set_result('test') + + t1 = self.new_task(self.loop, func()) + t2 = self.new_task(self.loop, coro()) + res = self.loop.run_until_complete(t1) + self.assertEqual(res, fut) + self.assertIsNone(t2.result()) + + def test_current_task(self): + self.assertIsNone(self.current_task(loop=self.loop)) + + async def coro(loop): + self.assertIs(self.current_task(), task) + + self.assertIs(self.current_task(None), task) + self.assertIs(self.current_task(), task) + + task = self.new_task(self.loop, coro(self.loop)) + self.loop.run_until_complete(task) + self.assertIsNone(self.current_task(loop=self.loop)) + + def test_current_task_with_interleaving_tasks(self): + self.assertIsNone(self.current_task(loop=self.loop)) + + fut1 = self.new_future(self.loop) + fut2 = self.new_future(self.loop) + + async def coro1(loop): + self.assertTrue(self.current_task() is task1) + await fut1 + self.assertTrue(self.current_task() is task1) + fut2.set_result(True) + + async def coro2(loop): + self.assertTrue(self.current_task() is task2) + fut1.set_result(True) + await fut2 + self.assertTrue(self.current_task() is task2) + + task1 = self.new_task(self.loop, coro1(self.loop)) + task2 = self.new_task(self.loop, coro2(self.loop)) + + self.loop.run_until_complete(asyncio.wait((task1, task2))) + self.assertIsNone(self.current_task(loop=self.loop)) + + # Some thorough tests for cancellation propagation through + # coroutines, tasks and wait(). + + def test_yield_future_passes_cancel(self): + # Cancelling outer() cancels inner() cancels waiter. + proof = 0 + waiter = self.new_future(self.loop) + + async def inner(): + nonlocal proof + try: + await waiter + except asyncio.CancelledError: + proof += 1 + raise + else: + self.fail('got past sleep() in inner()') + + async def outer(): + nonlocal proof + try: + await inner() + except asyncio.CancelledError: + proof += 100 # Expect this path. + else: + proof += 10 + + f = asyncio.ensure_future(outer(), loop=self.loop) + test_utils.run_briefly(self.loop) + f.cancel() + self.loop.run_until_complete(f) + self.assertEqual(proof, 101) + self.assertTrue(waiter.cancelled()) + + def test_yield_wait_does_not_shield_cancel(self): + # Cancelling outer() makes wait() return early, leaves inner() + # running. + proof = 0 + waiter = self.new_future(self.loop) + + async def inner(): + nonlocal proof + await waiter + proof += 1 + + async def outer(): + nonlocal proof + with self.assertWarns(DeprecationWarning): + d, p = await asyncio.wait([asyncio.create_task(inner())]) + proof += 100 + + f = asyncio.ensure_future(outer(), loop=self.loop) + test_utils.run_briefly(self.loop) + f.cancel() + self.assertRaises( + asyncio.CancelledError, self.loop.run_until_complete, f) + waiter.set_result(None) + test_utils.run_briefly(self.loop) + self.assertEqual(proof, 1) + + def test_shield_result(self): + inner = self.new_future(self.loop) + outer = asyncio.shield(inner) + inner.set_result(42) + res = self.loop.run_until_complete(outer) + self.assertEqual(res, 42) + + def test_shield_exception(self): + inner = self.new_future(self.loop) + outer = asyncio.shield(inner) + test_utils.run_briefly(self.loop) + exc = RuntimeError('expected') + inner.set_exception(exc) + test_utils.run_briefly(self.loop) + self.assertIs(outer.exception(), exc) + + def test_shield_cancel_inner(self): + inner = self.new_future(self.loop) + outer = asyncio.shield(inner) + test_utils.run_briefly(self.loop) + inner.cancel() + test_utils.run_briefly(self.loop) + self.assertTrue(outer.cancelled()) + + def test_shield_cancel_outer(self): + inner = self.new_future(self.loop) + outer = asyncio.shield(inner) + test_utils.run_briefly(self.loop) + outer.cancel() + test_utils.run_briefly(self.loop) + self.assertTrue(outer.cancelled()) + self.assertEqual(0, 0 if outer._callbacks is None else len(outer._callbacks)) + + def test_shield_cancel_outer_result(self): + mock_handler = mock.Mock() + self.loop.set_exception_handler(mock_handler) + inner = self.new_future(self.loop) + outer = asyncio.shield(inner) + test_utils.run_briefly(self.loop) + outer.cancel() + test_utils.run_briefly(self.loop) + inner.set_result(1) + test_utils.run_briefly(self.loop) + mock_handler.assert_not_called() + + def test_shield_cancel_outer_exception(self): + mock_handler = mock.Mock() + self.loop.set_exception_handler(mock_handler) + inner = self.new_future(self.loop) + outer = asyncio.shield(inner) + test_utils.run_briefly(self.loop) + outer.cancel() + test_utils.run_briefly(self.loop) + inner.set_exception(Exception('foo')) + test_utils.run_briefly(self.loop) + mock_handler.assert_called_once() + + def test_shield_duplicate_log_once(self): + mock_handler = mock.Mock() + self.loop.set_exception_handler(mock_handler) + inner = self.new_future(self.loop) + outer = asyncio.shield(inner) + test_utils.run_briefly(self.loop) + outer.cancel() + test_utils.run_briefly(self.loop) + outer = asyncio.shield(inner) + test_utils.run_briefly(self.loop) + outer.cancel() + test_utils.run_briefly(self.loop) + inner.set_exception(Exception('foo')) + test_utils.run_briefly(self.loop) + mock_handler.assert_called_once() + + def test_shield_shortcut(self): + fut = self.new_future(self.loop) + fut.set_result(42) + res = self.loop.run_until_complete(asyncio.shield(fut)) + self.assertEqual(res, 42) + + def test_shield_effect(self): + # Cancelling outer() does not affect inner(). + proof = 0 + waiter = self.new_future(self.loop) + + async def inner(): + nonlocal proof + await waiter + proof += 1 + + async def outer(): + nonlocal proof + await asyncio.shield(inner()) + proof += 100 + + f = asyncio.ensure_future(outer(), loop=self.loop) + test_utils.run_briefly(self.loop) + f.cancel() + with self.assertRaises(asyncio.CancelledError): + self.loop.run_until_complete(f) + waiter.set_result(None) + test_utils.run_briefly(self.loop) + self.assertEqual(proof, 1) + + def test_shield_gather(self): + child1 = self.new_future(self.loop) + child2 = self.new_future(self.loop) + parent = asyncio.gather(child1, child2) + outer = asyncio.shield(parent) + test_utils.run_briefly(self.loop) + outer.cancel() + test_utils.run_briefly(self.loop) + self.assertTrue(outer.cancelled()) + child1.set_result(1) + child2.set_result(2) + test_utils.run_briefly(self.loop) + self.assertEqual(parent.result(), [1, 2]) + + def test_gather_shield(self): + child1 = self.new_future(self.loop) + child2 = self.new_future(self.loop) + inner1 = asyncio.shield(child1) + inner2 = asyncio.shield(child2) + parent = asyncio.gather(inner1, inner2) + test_utils.run_briefly(self.loop) + parent.cancel() + # This should cancel inner1 and inner2 but bot child1 and child2. + test_utils.run_briefly(self.loop) + self.assertIsInstance(parent.exception(), asyncio.CancelledError) + self.assertTrue(inner1.cancelled()) + self.assertTrue(inner2.cancelled()) + child1.set_result(1) + child2.set_result(2) + test_utils.run_briefly(self.loop) + + def test_shield_coroutine_without_loop(self): + async def coro(): + return 42 + + inner = coro() + self.addCleanup(inner.close) + with self.assertRaisesRegex(RuntimeError, 'no current event loop'): + asyncio.shield(inner) + + def test_shield_coroutine_use_running_loop(self): + async def coro(): + return 42 + + async def test(): + return asyncio.shield(coro()) + outer = self.loop.run_until_complete(test()) + self.assertEqual(outer._loop, self.loop) + res = self.loop.run_until_complete(outer) + self.assertEqual(res, 42) + + def test_shield_coroutine_use_global_loop(self): + # Deprecated in 3.10, undeprecated in 3.12 + async def coro(): + return 42 + + asyncio.set_event_loop(self.loop) + self.addCleanup(asyncio.set_event_loop, None) + outer = asyncio.shield(coro()) + self.assertEqual(outer._loop, self.loop) + res = self.loop.run_until_complete(outer) + self.assertEqual(res, 42) + + def test_as_completed_invalid_args(self): + # as_completed() expects a list of futures, not a future instance + # TypeError should be raised either on iterator construction or first + # iteration + + # Plain iterator + fut = self.new_future(self.loop) + with self.assertRaises(TypeError): + iterator = asyncio.as_completed(fut) + next(iterator) + coro = coroutine_function() + with self.assertRaises(TypeError): + iterator = asyncio.as_completed(coro) + next(iterator) + coro.close() + + # Async iterator + async def try_async_iterator(aw): + async for f in asyncio.as_completed(aw): + break + + fut = self.new_future(self.loop) + with self.assertRaises(TypeError): + self.loop.run_until_complete(try_async_iterator(fut)) + coro = coroutine_function() + with self.assertRaises(TypeError): + self.loop.run_until_complete(try_async_iterator(coro)) + coro.close() + + def test_wait_invalid_args(self): + fut = self.new_future(self.loop) + + # wait() expects a list of futures, not a future instance + self.assertRaises(TypeError, self.loop.run_until_complete, + asyncio.wait(fut)) + coro = coroutine_function() + self.assertRaises(TypeError, self.loop.run_until_complete, + asyncio.wait(coro)) + coro.close() + + # wait() expects at least a future + self.assertRaises(ValueError, self.loop.run_until_complete, + asyncio.wait([])) + + def test_log_destroyed_pending_task(self): + + async def kill_me(loop): + future = self.new_future(loop) + await future + # at this point, the only reference to kill_me() task is + # the Task._wakeup() method in future._callbacks + raise Exception("code never reached") + + mock_handler = mock.Mock() + self.loop.set_debug(True) + self.loop.set_exception_handler(mock_handler) + + # schedule the task + coro = kill_me(self.loop) + task = self.new_task(self.loop, coro) + + self.assertEqual(self.all_tasks(loop=self.loop), {task}) + + # execute the task so it waits for future + self.loop.run_until_complete(asyncio.sleep(0)) + self.assertEqual(len(self.loop._ready), 0) + + coro = None + source_traceback = task._source_traceback + task = None + + # no more reference to kill_me() task: the task is destroyed by the GC + support.gc_collect() + + mock_handler.assert_called_with(self.loop, { + 'message': 'Task was destroyed but it is pending!', + 'task': mock.ANY, + 'source_traceback': source_traceback, + }) + mock_handler.reset_mock() + # task got resurrected by the exception handler + support.gc_collect() + + self.assertEqual(self.all_tasks(loop=self.loop), set()) + + def test_task_not_crash_without_finalization(self): + Task = self.__class__.Task + + class Subclass(Task): + def __del__(self): + pass + + async def corofn(): + await asyncio.sleep(0.01) + + coro = corofn() + task = Subclass(coro, loop = self.loop) + task._log_destroy_pending = False + + del task + + support.gc_collect() + + coro.close() + + @mock.patch('asyncio.base_events.logger') + def test_tb_logger_not_called_after_cancel(self, m_log): + loop = asyncio.new_event_loop() + self.set_event_loop(loop) + + async def coro(): + raise TypeError + + async def runner(): + task = self.new_task(loop, coro()) + await asyncio.sleep(0.05) + task.cancel() + task = None + + loop.run_until_complete(runner()) + self.assertFalse(m_log.error.called) + + def test_task_source_traceback(self): + self.loop.set_debug(True) + + task = self.new_task(self.loop, coroutine_function()) + lineno = sys._getframe().f_lineno - 1 + self.assertIsInstance(task._source_traceback, list) + self.assertEqual(task._source_traceback[-2][:3], + (__file__, + lineno, + 'test_task_source_traceback')) + self.loop.run_until_complete(task) + + def test_cancel_gather_1(self): + """Ensure that a gathering future refuses to be cancelled once all + children are done""" + loop = asyncio.new_event_loop() + self.addCleanup(loop.close) + + fut = self.new_future(loop) + async def create(): + # The indirection fut->child_coro is needed since otherwise the + # gathering task is done at the same time as the child future + async def child_coro(): + return await fut + gather_future = asyncio.gather(child_coro()) + return asyncio.ensure_future(gather_future) + gather_task = loop.run_until_complete(create()) + + cancel_result = None + def cancelling_callback(_): + nonlocal cancel_result + cancel_result = gather_task.cancel() + fut.add_done_callback(cancelling_callback) + + fut.set_result(42) # calls the cancelling_callback after fut is done() + + # At this point the task should complete. + loop.run_until_complete(gather_task) + + # Python issue #26923: asyncio.gather drops cancellation + self.assertEqual(cancel_result, False) + self.assertFalse(gather_task.cancelled()) + self.assertEqual(gather_task.result(), [42]) + + def test_cancel_gather_2(self): + cases = [ + ((), ()), + ((None,), ()), + (('my message',), ('my message',)), + # Non-string values should roundtrip. + ((5,), (5,)), + ] + for cancel_args, expected_args in cases: + with self.subTest(cancel_args=cancel_args): + loop = asyncio.new_event_loop() + self.addCleanup(loop.close) + + async def test(): + time = 0 + while True: + time += 0.05 + await asyncio.gather(asyncio.sleep(0.05), + return_exceptions=True) + if time > 1: + return + + async def main(): + qwe = self.new_task(loop, test()) + await asyncio.sleep(0.2) + qwe.cancel(*cancel_args) + await qwe + + try: + loop.run_until_complete(main()) + except asyncio.CancelledError as exc: + self.assertEqual(exc.args, expected_args) + actual = get_innermost_context(exc) + self.assertEqual( + actual, + (asyncio.CancelledError, expected_args, 0), + ) + else: + self.fail( + 'gather() does not propagate CancelledError ' + 'raised by inner task to the gather() caller.' + ) + + def test_exception_traceback(self): + # See http://bugs.python.org/issue28843 + + async def foo(): + 1 / 0 + + async def main(): + task = self.new_task(self.loop, foo()) + await asyncio.sleep(0) # skip one loop iteration + self.assertIsNotNone(task.exception().__traceback__) + + self.loop.run_until_complete(main()) + + @mock.patch('asyncio.base_events.logger') + def test_error_in_call_soon(self, m_log): + def call_soon(callback, *args, **kwargs): + raise ValueError + self.loop.call_soon = call_soon + + async def coro(): + pass + + self.assertFalse(m_log.error.called) + + with self.assertRaises(ValueError): + gen = coro() + try: + self.new_task(self.loop, gen) + finally: + gen.close() + gc.collect() # For PyPy or other GCs. + + self.assertTrue(m_log.error.called) + message = m_log.error.call_args[0][0] + self.assertIn('Task was destroyed but it is pending', message) + + self.assertEqual(self.all_tasks(self.loop), set()) + + def test_create_task_with_noncoroutine(self): + with self.assertRaisesRegex(TypeError, + "a coroutine was expected, got 123"): + self.new_task(self.loop, 123) + + # test it for the second time to ensure that caching + # in asyncio.iscoroutine() doesn't break things. + with self.assertRaisesRegex(TypeError, + "a coroutine was expected, got 123"): + self.new_task(self.loop, 123) + + def test_create_task_with_async_function(self): + + async def coro(): + pass + + task = self.new_task(self.loop, coro()) + self.assertIsInstance(task, self.Task) + self.loop.run_until_complete(task) + + # test it for the second time to ensure that caching + # in asyncio.iscoroutine() doesn't break things. + task = self.new_task(self.loop, coro()) + self.assertIsInstance(task, self.Task) + self.loop.run_until_complete(task) + + def test_create_task_with_asynclike_function(self): + task = self.new_task(self.loop, CoroLikeObject()) + self.assertIsInstance(task, self.Task) + self.assertEqual(self.loop.run_until_complete(task), 42) + + # test it for the second time to ensure that caching + # in asyncio.iscoroutine() doesn't break things. + task = self.new_task(self.loop, CoroLikeObject()) + self.assertIsInstance(task, self.Task) + self.assertEqual(self.loop.run_until_complete(task), 42) + + def test_bare_create_task(self): + + async def inner(): + return 1 + + async def coro(): + task = asyncio.create_task(inner()) + self.assertIsInstance(task, self.Task) + ret = await task + self.assertEqual(1, ret) + + self.loop.run_until_complete(coro()) + + def test_bare_create_named_task(self): + + async def coro_noop(): + pass + + async def coro(): + task = asyncio.create_task(coro_noop(), name='No-op') + self.assertEqual(task.get_name(), 'No-op') + await task + + self.loop.run_until_complete(coro()) + + def test_context_1(self): + cvar = contextvars.ContextVar('cvar', default='nope') + + async def sub(): + await asyncio.sleep(0.01) + self.assertEqual(cvar.get(), 'nope') + cvar.set('something else') + + async def main(): + self.assertEqual(cvar.get(), 'nope') + subtask = self.new_task(loop, sub()) + cvar.set('yes') + self.assertEqual(cvar.get(), 'yes') + await subtask + self.assertEqual(cvar.get(), 'yes') + + loop = asyncio.new_event_loop() + try: + task = self.new_task(loop, main()) + loop.run_until_complete(task) + finally: + loop.close() + + def test_context_2(self): + cvar = contextvars.ContextVar('cvar', default='nope') + + async def main(): + def fut_on_done(fut): + # This change must not pollute the context + # of the "main()" task. + cvar.set('something else') + + self.assertEqual(cvar.get(), 'nope') + + for j in range(2): + fut = self.new_future(loop) + fut.add_done_callback(fut_on_done) + cvar.set(f'yes{j}') + loop.call_soon(fut.set_result, None) + await fut + self.assertEqual(cvar.get(), f'yes{j}') + + for i in range(3): + # Test that task passed its context to add_done_callback: + cvar.set(f'yes{i}-{j}') + await asyncio.sleep(0.001) + self.assertEqual(cvar.get(), f'yes{i}-{j}') + + loop = asyncio.new_event_loop() + try: + task = self.new_task(loop, main()) + loop.run_until_complete(task) + finally: + loop.close() + + self.assertEqual(cvar.get(), 'nope') + + def test_context_3(self): + # Run 100 Tasks in parallel, each modifying cvar. + + cvar = contextvars.ContextVar('cvar', default=-1) + + async def sub(num): + for i in range(10): + cvar.set(num + i) + await asyncio.sleep(random.uniform(0.001, 0.05)) + self.assertEqual(cvar.get(), num + i) + + async def main(): + tasks = [] + for i in range(100): + task = loop.create_task(sub(random.randint(0, 10))) + tasks.append(task) + + await asyncio.gather(*tasks) + + loop = asyncio.new_event_loop() + try: + loop.run_until_complete(main()) + finally: + loop.close() + + self.assertEqual(cvar.get(), -1) + + def test_context_4(self): + cvar = contextvars.ContextVar('cvar') + + async def coro(val): + await asyncio.sleep(0) + cvar.set(val) + + async def main(): + ret = [] + ctx = contextvars.copy_context() + ret.append(ctx.get(cvar)) + t1 = self.new_task(loop, coro(1), context=ctx) + await t1 + ret.append(ctx.get(cvar)) + t2 = self.new_task(loop, coro(2), context=ctx) + await t2 + ret.append(ctx.get(cvar)) + return ret + + loop = asyncio.new_event_loop() + try: + task = self.new_task(loop, main()) + ret = loop.run_until_complete(task) + finally: + loop.close() + + self.assertEqual([None, 1, 2], ret) + + def test_context_5(self): + cvar = contextvars.ContextVar('cvar') + + async def coro(val): + await asyncio.sleep(0) + cvar.set(val) + + async def main(): + ret = [] + ctx = contextvars.copy_context() + ret.append(ctx.get(cvar)) + t1 = asyncio.create_task(coro(1), context=ctx) + await t1 + ret.append(ctx.get(cvar)) + t2 = asyncio.create_task(coro(2), context=ctx) + await t2 + ret.append(ctx.get(cvar)) + return ret + + loop = asyncio.new_event_loop() + try: + task = self.new_task(loop, main()) + ret = loop.run_until_complete(task) + finally: + loop.close() + + self.assertEqual([None, 1, 2], ret) + + def test_context_6(self): + cvar = contextvars.ContextVar('cvar') + + async def coro(val): + await asyncio.sleep(0) + cvar.set(val) + + async def main(): + ret = [] + ctx = contextvars.copy_context() + ret.append(ctx.get(cvar)) + t1 = loop.create_task(coro(1), context=ctx) + await t1 + ret.append(ctx.get(cvar)) + t2 = loop.create_task(coro(2), context=ctx) + await t2 + ret.append(ctx.get(cvar)) + return ret + + loop = asyncio.new_event_loop() + try: + task = loop.create_task(main()) + ret = loop.run_until_complete(task) + finally: + loop.close() + + self.assertEqual([None, 1, 2], ret) + + def test_eager_start_true(self): + name = None + + async def asyncfn(): + nonlocal name + name = self.current_task().get_name() + + async def main(): + t = self.new_task(coro=asyncfn(), loop=asyncio.get_running_loop(), eager_start=True, name="example") + self.assertTrue(t.done()) + self.assertEqual(name, "example") + await t + + def test_eager_start_false(self): + name = None + + async def asyncfn(): + nonlocal name + name = self.current_task().get_name() + + async def main(): + t = self.new_task(coro=asyncfn(), loop=asyncio.get_running_loop(), eager_start=False, name="example") + self.assertFalse(t.done()) + self.assertIsNone(name) + await t + self.assertEqual(name, "example") + + asyncio.run(main(), loop_factory=asyncio.EventLoop) + + def test_get_coro(self): + loop = asyncio.new_event_loop() + coro = coroutine_function() + try: + task = self.new_task(loop, coro) + loop.run_until_complete(task) + self.assertIs(task.get_coro(), coro) + finally: + loop.close() + + def test_get_context(self): + loop = asyncio.new_event_loop() + coro = coroutine_function() + context = contextvars.copy_context() + try: + task = self.new_task(loop, coro, context=context) + loop.run_until_complete(task) + self.assertIs(task.get_context(), context) + finally: + loop.close() + + def test_proper_refcounts(self): + # see: https://github.com/python/cpython/issues/126083 + class Break: + def __str__(self): + raise RuntimeError("break") + + obj = object() + initial_refcount = sys.getrefcount(obj) + + coro = coroutine_function() + with contextlib.closing(asyncio.EventLoop()) as loop: + task = asyncio.Task.__new__(asyncio.Task) + for _ in range(5): + with self.assertRaisesRegex(RuntimeError, 'break'): + task.__init__(coro, loop=loop, context=obj, name=Break()) + + coro.close() + task._log_destroy_pending = False + del task + + self.assertEqual(sys.getrefcount(obj), initial_refcount) + + +def add_subclass_tests(cls): + BaseTask = cls.Task + BaseFuture = cls.Future + + if BaseTask is None or BaseFuture is None: + return cls + + class CommonFuture: + def __init__(self, *args, **kwargs): + self.calls = collections.defaultdict(lambda: 0) + super().__init__(*args, **kwargs) + + def add_done_callback(self, *args, **kwargs): + self.calls['add_done_callback'] += 1 + return super().add_done_callback(*args, **kwargs) + + class Task(CommonFuture, BaseTask): + pass + + class Future(CommonFuture, BaseFuture): + pass + + def test_subclasses_ctask_cfuture(self): + fut = self.Future(loop=self.loop) + + async def func(): + self.loop.call_soon(lambda: fut.set_result('spam')) + return await fut + + task = self.Task(func(), loop=self.loop) + + result = self.loop.run_until_complete(task) + + self.assertEqual(result, 'spam') + + self.assertEqual( + dict(task.calls), + {'add_done_callback': 1}) + + self.assertEqual( + dict(fut.calls), + {'add_done_callback': 1}) + + # Add patched Task & Future back to the test case + cls.Task = Task + cls.Future = Future + + # Add an extra unit-test + cls.test_subclasses_ctask_cfuture = test_subclasses_ctask_cfuture + + # Disable the "test_task_source_traceback" test + # (the test is hardcoded for a particular call stack, which + # is slightly different for Task subclasses) + cls.test_task_source_traceback = None + + return cls + + +class SetMethodsTest: + + def test_set_result_causes_invalid_state(self): + Future = type(self).Future + self.loop.call_exception_handler = exc_handler = mock.Mock() + + async def foo(): + await asyncio.sleep(0.1) + return 10 + + coro = foo() + task = self.new_task(self.loop, coro) + Future.set_result(task, 'spam') + + self.assertEqual( + self.loop.run_until_complete(task), + 'spam') + + exc_handler.assert_called_once() + exc = exc_handler.call_args[0][0]['exception'] + with self.assertRaisesRegex(asyncio.InvalidStateError, + r'step\(\): already done'): + raise exc + + coro.close() + + def test_set_exception_causes_invalid_state(self): + class MyExc(Exception): + pass + + Future = type(self).Future + self.loop.call_exception_handler = exc_handler = mock.Mock() + + async def foo(): + await asyncio.sleep(0.1) + return 10 + + coro = foo() + task = self.new_task(self.loop, coro) + Future.set_exception(task, MyExc()) + + with self.assertRaises(MyExc): + self.loop.run_until_complete(task) + + exc_handler.assert_called_once() + exc = exc_handler.call_args[0][0]['exception'] + with self.assertRaisesRegex(asyncio.InvalidStateError, + r'step\(\): already done'): + raise exc + + coro.close() + + +@unittest.skipUnless(hasattr(futures, '_CFuture') and + hasattr(tasks, '_CTask'), + 'requires the C _asyncio module') +class CTask_CFuture_Tests(BaseTaskTests, SetMethodsTest, + test_utils.TestCase): + + Task = getattr(tasks, '_CTask', None) + Future = getattr(futures, '_CFuture', None) + all_tasks = getattr(tasks, '_c_all_tasks', None) + current_task = staticmethod(getattr(tasks, '_c_current_task', None)) + + @support.refcount_test + def test_refleaks_in_task___init__(self): + gettotalrefcount = support.get_attribute(sys, 'gettotalrefcount') + async def coro(): + pass + task = self.new_task(self.loop, coro()) + self.loop.run_until_complete(task) + refs_before = gettotalrefcount() + for i in range(100): + task.__init__(coro(), loop=self.loop) + self.loop.run_until_complete(task) + self.assertAlmostEqual(gettotalrefcount() - refs_before, 0, delta=10) + + def test_del__log_destroy_pending_segfault(self): + async def coro(): + pass + task = self.new_task(self.loop, coro()) + self.loop.run_until_complete(task) + with self.assertRaises(AttributeError): + del task._log_destroy_pending + + @unittest.expectedFailure # TODO: RUSTPYTHON; Actual: not called. + def test_log_destroyed_pending_task(self): + return super().test_log_destroyed_pending_task() + + +@unittest.skipUnless(hasattr(futures, '_CFuture') and + hasattr(tasks, '_CTask'), + 'requires the C _asyncio module') +@add_subclass_tests +class CTask_CFuture_SubclassTests(BaseTaskTests, test_utils.TestCase): + + Task = getattr(tasks, '_CTask', None) + Future = getattr(futures, '_CFuture', None) + all_tasks = getattr(tasks, '_c_all_tasks', None) + current_task = staticmethod(getattr(tasks, '_c_current_task', None)) + + @unittest.expectedFailure # TODO: RUSTPYTHON; Actual: not called. + def test_log_destroyed_pending_task(self): + return super().test_log_destroyed_pending_task() + + +@unittest.skipUnless(hasattr(tasks, '_CTask'), + 'requires the C _asyncio module') +@add_subclass_tests +class CTaskSubclass_PyFuture_Tests(BaseTaskTests, test_utils.TestCase): + + Task = getattr(tasks, '_CTask', None) + Future = futures._PyFuture + all_tasks = getattr(tasks, '_c_all_tasks', None) + current_task = staticmethod(getattr(tasks, '_c_current_task', None)) + + @unittest.expectedFailure # TODO: RUSTPYTHON; Actual: not called. + def test_log_destroyed_pending_task(self): + return super().test_log_destroyed_pending_task() + + +@unittest.skipUnless(hasattr(futures, '_CFuture'), + 'requires the C _asyncio module') +@add_subclass_tests +class PyTask_CFutureSubclass_Tests(BaseTaskTests, test_utils.TestCase): + + Future = getattr(futures, '_CFuture', None) + Task = tasks._PyTask + all_tasks = staticmethod(tasks._py_all_tasks) + current_task = staticmethod(tasks._py_current_task) + + @unittest.expectedFailure # TODO: RUSTPYTHON; Actual: not called. + def test_log_destroyed_pending_task(self): + return super().test_log_destroyed_pending_task() + + +@unittest.skipUnless(hasattr(tasks, '_CTask'), + 'requires the C _asyncio module') +class CTask_PyFuture_Tests(BaseTaskTests, test_utils.TestCase): + + Task = getattr(tasks, '_CTask', None) + Future = futures._PyFuture + all_tasks = getattr(tasks, '_c_all_tasks', None) + current_task = staticmethod(getattr(tasks, '_c_current_task', None)) + + @unittest.expectedFailure # TODO: RUSTPYTHON; Actual: not called. + def test_log_destroyed_pending_task(self): + return super().test_log_destroyed_pending_task() + + +@unittest.skipUnless(hasattr(futures, '_CFuture'), + 'requires the C _asyncio module') +class PyTask_CFuture_Tests(BaseTaskTests, test_utils.TestCase): + + Task = tasks._PyTask + Future = getattr(futures, '_CFuture', None) + all_tasks = staticmethod(tasks._py_all_tasks) + current_task = staticmethod(tasks._py_current_task) + + @unittest.expectedFailure # TODO: RUSTPYTHON; Actual: not called. + def test_log_destroyed_pending_task(self): + return super().test_log_destroyed_pending_task() + + +class PyTask_PyFuture_Tests(BaseTaskTests, SetMethodsTest, + test_utils.TestCase): + + Task = tasks._PyTask + Future = futures._PyFuture + all_tasks = staticmethod(tasks._py_all_tasks) + current_task = staticmethod(tasks._py_current_task) + + @unittest.expectedFailure # TODO: RUSTPYTHON; Actual: not called. + def test_log_destroyed_pending_task(self): + return super().test_log_destroyed_pending_task() + + +@add_subclass_tests +class PyTask_PyFuture_SubclassTests(BaseTaskTests, test_utils.TestCase): + Task = tasks._PyTask + Future = futures._PyFuture + all_tasks = staticmethod(tasks._py_all_tasks) + current_task = staticmethod(tasks._py_current_task) + + @unittest.expectedFailure # TODO: RUSTPYTHON; Actual: not called. + def test_log_destroyed_pending_task(self): + return super().test_log_destroyed_pending_task() + +@unittest.skipUnless(hasattr(tasks, '_CTask'), + 'requires the C _asyncio module') +class CTask_Future_Tests(test_utils.TestCase): + + def test_foobar(self): + class Fut(asyncio.Future): + @property + def get_loop(self): + raise AttributeError + + async def coro(): + await fut + return 'spam' + + self.loop = asyncio.new_event_loop() + try: + fut = Fut(loop=self.loop) + self.loop.call_later(0.1, fut.set_result, 1) + task = self.loop.create_task(coro()) + res = self.loop.run_until_complete(task) + finally: + self.loop.close() + + self.assertEqual(res, 'spam') + + +class BaseTaskIntrospectionTests: + _register_task = None + _unregister_task = None + _enter_task = None + _leave_task = None + all_tasks = None + + def test__register_task_1(self): + class TaskLike: + @property + def _loop(self): + return loop + + def done(self): + return False + + task = TaskLike() + loop = mock.Mock() + + self.assertEqual(self.all_tasks(loop), set()) + self._register_task(task) + self.assertEqual(self.all_tasks(loop), {task}) + self._unregister_task(task) + + def test__register_task_2(self): + class TaskLike: + def get_loop(self): + return loop + + def done(self): + return False + + task = TaskLike() + loop = mock.Mock() + + self.assertEqual(self.all_tasks(loop), set()) + self._register_task(task) + self.assertEqual(self.all_tasks(loop), {task}) + self._unregister_task(task) + + def test__register_task_3(self): + class TaskLike: + def get_loop(self): + return loop + + def done(self): + return True + + task = TaskLike() + loop = mock.Mock() + + self.assertEqual(self.all_tasks(loop), set()) + self._register_task(task) + self.assertEqual(self.all_tasks(loop), set()) + self._unregister_task(task) + + def test__enter_task(self): + task = mock.Mock() + loop = mock.Mock() + # _enter_task is called by Task.__step while the loop + # is running, so set the loop as the running loop + # for a more realistic test. + asyncio._set_running_loop(loop) + self.assertIsNone(self.current_task(loop)) + self._enter_task(loop, task) + self.assertIs(self.current_task(loop), task) + self._leave_task(loop, task) + asyncio._set_running_loop(None) + + def test__enter_task_failure(self): + task1 = mock.Mock() + task2 = mock.Mock() + loop = mock.Mock() + asyncio._set_running_loop(loop) + self._enter_task(loop, task1) + with self.assertRaises(RuntimeError): + self._enter_task(loop, task2) + self.assertIs(self.current_task(loop), task1) + self._leave_task(loop, task1) + asyncio._set_running_loop(None) + + def test__leave_task(self): + task = mock.Mock() + loop = mock.Mock() + asyncio._set_running_loop(loop) + self._enter_task(loop, task) + self._leave_task(loop, task) + self.assertIsNone(self.current_task(loop)) + asyncio._set_running_loop(None) + + def test__leave_task_failure1(self): + task1 = mock.Mock() + task2 = mock.Mock() + loop = mock.Mock() + # _leave_task is called by Task.__step while the loop + # is running, so set the loop as the running loop + # for a more realistic test. + asyncio._set_running_loop(loop) + self._enter_task(loop, task1) + with self.assertRaises(RuntimeError): + self._leave_task(loop, task2) + self.assertIs(self.current_task(loop), task1) + self._leave_task(loop, task1) + asyncio._set_running_loop(None) + + def test__leave_task_failure2(self): + task = mock.Mock() + loop = mock.Mock() + asyncio._set_running_loop(loop) + with self.assertRaises(RuntimeError): + self._leave_task(loop, task) + self.assertIsNone(self.current_task(loop)) + asyncio._set_running_loop(None) + + def test__unregister_task(self): + task = mock.Mock() + loop = mock.Mock() + task.get_loop = lambda: loop + self._register_task(task) + self._unregister_task(task) + self.assertEqual(self.all_tasks(loop), set()) + + def test__unregister_task_not_registered(self): + task = mock.Mock() + loop = mock.Mock() + self._unregister_task(task) + self.assertEqual(self.all_tasks(loop), set()) + + +class PyIntrospectionTests(test_utils.TestCase, BaseTaskIntrospectionTests): + _register_task = staticmethod(tasks._py_register_task) + _unregister_task = staticmethod(tasks._py_unregister_task) + _enter_task = staticmethod(tasks._py_enter_task) + _leave_task = staticmethod(tasks._py_leave_task) + all_tasks = staticmethod(tasks._py_all_tasks) + current_task = staticmethod(tasks._py_current_task) + + +@unittest.skipUnless(hasattr(tasks, '_c_register_task'), + 'requires the C _asyncio module') +class CIntrospectionTests(test_utils.TestCase, BaseTaskIntrospectionTests): + if hasattr(tasks, '_c_register_task'): + _register_task = staticmethod(tasks._c_register_task) + _unregister_task = staticmethod(tasks._c_unregister_task) + _enter_task = staticmethod(tasks._c_enter_task) + _leave_task = staticmethod(tasks._c_leave_task) + all_tasks = staticmethod(tasks._c_all_tasks) + current_task = staticmethod(tasks._c_current_task) + else: + _register_task = _unregister_task = _enter_task = _leave_task = None + + +class BaseCurrentLoopTests: + current_task = None + + def setUp(self): + super().setUp() + self.loop = asyncio.new_event_loop() + self.set_event_loop(self.loop) + + def new_task(self, coro): + raise NotImplementedError + + def test_current_task_no_running_loop(self): + self.assertIsNone(self.current_task(loop=self.loop)) + + def test_current_task_no_running_loop_implicit(self): + with self.assertRaisesRegex(RuntimeError, 'no running event loop'): + self.current_task() + + def test_current_task_with_implicit_loop(self): + async def coro(): + self.assertIs(self.current_task(loop=self.loop), task) + + self.assertIs(self.current_task(None), task) + self.assertIs(self.current_task(), task) + + task = self.new_task(coro()) + self.loop.run_until_complete(task) + self.assertIsNone(self.current_task(loop=self.loop)) + + +class PyCurrentLoopTests(BaseCurrentLoopTests, test_utils.TestCase): + current_task = staticmethod(tasks._py_current_task) + + def new_task(self, coro): + return tasks._PyTask(coro, loop=self.loop) + + +@unittest.skipUnless(hasattr(tasks, '_CTask') and + hasattr(tasks, '_c_current_task'), + 'requires the C _asyncio module') +class CCurrentLoopTests(BaseCurrentLoopTests, test_utils.TestCase): + if hasattr(tasks, '_c_current_task'): + current_task = staticmethod(tasks._c_current_task) + else: + current_task = None + + def new_task(self, coro): + return getattr(tasks, '_CTask')(coro, loop=self.loop) + + +class GenericTaskTests(test_utils.TestCase): + + def test_future_subclass(self): + self.assertIsSubclass(asyncio.Task, asyncio.Future) + + @support.cpython_only + def test_asyncio_module_compiled(self): + # Because of circular imports it's easy to make _asyncio + # module non-importable. This is a simple test that will + # fail on systems where C modules were successfully compiled + # (hence the test for _functools etc), but _asyncio somehow didn't. + try: + import _functools # noqa: F401 + import _json # noqa: F401 + import _pickle # noqa: F401 + except ImportError: + self.skipTest('C modules are not available') + else: + try: + import _asyncio # noqa: F401 + except ImportError: + self.fail('_asyncio module is missing') + + +class GatherTestsBase: + + def setUp(self): + super().setUp() + self.one_loop = self.new_test_loop() + self.other_loop = self.new_test_loop() + self.set_event_loop(self.one_loop, cleanup=False) + + def _run_loop(self, loop): + while loop._ready: + test_utils.run_briefly(loop) + + def _check_success(self, **kwargs): + a, b, c = [self.one_loop.create_future() for i in range(3)] + fut = self._gather(*self.wrap_futures(a, b, c), **kwargs) + cb = test_utils.MockCallback() + fut.add_done_callback(cb) + b.set_result(1) + a.set_result(2) + self._run_loop(self.one_loop) + self.assertEqual(cb.called, False) + self.assertFalse(fut.done()) + c.set_result(3) + self._run_loop(self.one_loop) + cb.assert_called_once_with(fut) + self.assertEqual(fut.result(), [2, 1, 3]) + + def test_success(self): + self._check_success() + self._check_success(return_exceptions=False) + + def test_result_exception_success(self): + self._check_success(return_exceptions=True) + + def test_one_exception(self): + a, b, c, d, e = [self.one_loop.create_future() for i in range(5)] + fut = self._gather(*self.wrap_futures(a, b, c, d, e)) + cb = test_utils.MockCallback() + fut.add_done_callback(cb) + exc = ZeroDivisionError() + a.set_result(1) + b.set_exception(exc) + self._run_loop(self.one_loop) + self.assertTrue(fut.done()) + cb.assert_called_once_with(fut) + self.assertIs(fut.exception(), exc) + # Does nothing + c.set_result(3) + d.cancel() + e.set_exception(RuntimeError()) + e.exception() + + def test_return_exceptions(self): + a, b, c, d = [self.one_loop.create_future() for i in range(4)] + fut = self._gather(*self.wrap_futures(a, b, c, d), + return_exceptions=True) + cb = test_utils.MockCallback() + fut.add_done_callback(cb) + exc = ZeroDivisionError() + exc2 = RuntimeError() + b.set_result(1) + c.set_exception(exc) + a.set_result(3) + self._run_loop(self.one_loop) + self.assertFalse(fut.done()) + d.set_exception(exc2) + self._run_loop(self.one_loop) + self.assertTrue(fut.done()) + cb.assert_called_once_with(fut) + self.assertEqual(fut.result(), [3, 1, exc, exc2]) + + def test_env_var_debug(self): + code = '\n'.join(( + 'import asyncio.coroutines', + 'print(asyncio.coroutines._is_debug_mode())')) + + # Test with -E to not fail if the unit test was run with + # PYTHONASYNCIODEBUG set to a non-empty string + sts, stdout, stderr = assert_python_ok('-E', '-c', code) + self.assertEqual(stdout.rstrip(), b'False') + + sts, stdout, stderr = assert_python_ok('-c', code, + PYTHONASYNCIODEBUG='', + PYTHONDEVMODE='') + self.assertEqual(stdout.rstrip(), b'False') + + sts, stdout, stderr = assert_python_ok('-c', code, + PYTHONASYNCIODEBUG='1', + PYTHONDEVMODE='') + self.assertEqual(stdout.rstrip(), b'True') + + sts, stdout, stderr = assert_python_ok('-E', '-c', code, + PYTHONASYNCIODEBUG='1', + PYTHONDEVMODE='') + self.assertEqual(stdout.rstrip(), b'False') + + # -X dev + sts, stdout, stderr = assert_python_ok('-E', '-X', 'dev', + '-c', code) + self.assertEqual(stdout.rstrip(), b'True') + + +class FutureGatherTests(GatherTestsBase, test_utils.TestCase): + + def wrap_futures(self, *futures): + return futures + + def _gather(self, *args, **kwargs): + return asyncio.gather(*args, **kwargs) + + def test_constructor_empty_sequence_without_loop(self): + with self.assertRaisesRegex(RuntimeError, 'no current event loop'): + asyncio.gather() + + def test_constructor_empty_sequence_use_running_loop(self): + async def gather(): + return asyncio.gather() + fut = self.one_loop.run_until_complete(gather()) + self.assertIsInstance(fut, asyncio.Future) + self.assertIs(fut._loop, self.one_loop) + self._run_loop(self.one_loop) + self.assertTrue(fut.done()) + self.assertEqual(fut.result(), []) + + def test_constructor_empty_sequence_use_global_loop(self): + # Deprecated in 3.10, undeprecated in 3.12 + asyncio.set_event_loop(self.one_loop) + self.addCleanup(asyncio.set_event_loop, None) + fut = asyncio.gather() + self.assertIsInstance(fut, asyncio.Future) + self.assertIs(fut._loop, self.one_loop) + self._run_loop(self.one_loop) + self.assertTrue(fut.done()) + self.assertEqual(fut.result(), []) + + def test_constructor_heterogenous_futures(self): + fut1 = self.one_loop.create_future() + fut2 = self.other_loop.create_future() + with self.assertRaises(ValueError): + asyncio.gather(fut1, fut2) + + def test_constructor_homogenous_futures(self): + children = [self.other_loop.create_future() for i in range(3)] + fut = asyncio.gather(*children) + self.assertIs(fut._loop, self.other_loop) + self._run_loop(self.other_loop) + self.assertFalse(fut.done()) + fut = asyncio.gather(*children) + self.assertIs(fut._loop, self.other_loop) + self._run_loop(self.other_loop) + self.assertFalse(fut.done()) + + def test_one_cancellation(self): + a, b, c, d, e = [self.one_loop.create_future() for i in range(5)] + fut = asyncio.gather(a, b, c, d, e) + cb = test_utils.MockCallback() + fut.add_done_callback(cb) + a.set_result(1) + b.cancel() + self._run_loop(self.one_loop) + self.assertTrue(fut.done()) + cb.assert_called_once_with(fut) + self.assertFalse(fut.cancelled()) + self.assertIsInstance(fut.exception(), asyncio.CancelledError) + # Does nothing + c.set_result(3) + d.cancel() + e.set_exception(RuntimeError()) + e.exception() + + def test_result_exception_one_cancellation(self): + a, b, c, d, e, f = [self.one_loop.create_future() + for i in range(6)] + fut = asyncio.gather(a, b, c, d, e, f, return_exceptions=True) + cb = test_utils.MockCallback() + fut.add_done_callback(cb) + a.set_result(1) + zde = ZeroDivisionError() + b.set_exception(zde) + c.cancel() + self._run_loop(self.one_loop) + self.assertFalse(fut.done()) + d.set_result(3) + e.cancel() + rte = RuntimeError() + f.set_exception(rte) + res = self.one_loop.run_until_complete(fut) + self.assertIsInstance(res[2], asyncio.CancelledError) + self.assertIsInstance(res[4], asyncio.CancelledError) + res[2] = res[4] = None + self.assertEqual(res, [1, zde, None, 3, None, rte]) + cb.assert_called_once_with(fut) + + +class CoroutineGatherTests(GatherTestsBase, test_utils.TestCase): + + def wrap_futures(self, *futures): + coros = [] + for fut in futures: + async def coro(fut=fut): + return await fut + coros.append(coro()) + return coros + + def _gather(self, *args, **kwargs): + async def coro(): + return asyncio.gather(*args, **kwargs) + return self.one_loop.run_until_complete(coro()) + + def test_constructor_without_loop(self): + async def coro(): + return 'abc' + gen1 = coro() + self.addCleanup(gen1.close) + gen2 = coro() + self.addCleanup(gen2.close) + with self.assertRaisesRegex(RuntimeError, 'no current event loop'): + asyncio.gather(gen1, gen2) + + def test_constructor_use_running_loop(self): + async def coro(): + return 'abc' + gen1 = coro() + gen2 = coro() + async def gather(): + return asyncio.gather(gen1, gen2) + fut = self.one_loop.run_until_complete(gather()) + self.assertIs(fut._loop, self.one_loop) + self.one_loop.run_until_complete(fut) + + def test_constructor_use_global_loop(self): + # Deprecated in 3.10, undeprecated in 3.12 + async def coro(): + return 'abc' + asyncio.set_event_loop(self.other_loop) + self.addCleanup(asyncio.set_event_loop, None) + gen1 = coro() + gen2 = coro() + fut = asyncio.gather(gen1, gen2) + self.assertIs(fut._loop, self.other_loop) + self.other_loop.run_until_complete(fut) + + def test_duplicate_coroutines(self): + async def coro(s): + return s + c = coro('abc') + fut = self._gather(c, c, coro('def'), c) + self._run_loop(self.one_loop) + self.assertEqual(fut.result(), ['abc', 'abc', 'def', 'abc']) + + def test_cancellation_broadcast(self): + # Cancelling outer() cancels all children. + proof = 0 + waiter = self.one_loop.create_future() + + async def inner(): + nonlocal proof + await waiter + proof += 1 + + child1 = asyncio.ensure_future(inner(), loop=self.one_loop) + child2 = asyncio.ensure_future(inner(), loop=self.one_loop) + gatherer = None + + async def outer(): + nonlocal proof, gatherer + gatherer = asyncio.gather(child1, child2) + await gatherer + proof += 100 + + f = asyncio.ensure_future(outer(), loop=self.one_loop) + test_utils.run_briefly(self.one_loop) + self.assertTrue(f.cancel()) + with self.assertRaises(asyncio.CancelledError): + self.one_loop.run_until_complete(f) + self.assertFalse(gatherer.cancel()) + self.assertTrue(waiter.cancelled()) + self.assertTrue(child1.cancelled()) + self.assertTrue(child2.cancelled()) + test_utils.run_briefly(self.one_loop) + self.assertEqual(proof, 0) + + def test_exception_marking(self): + # Test for the first line marked "Mark exception retrieved." + + async def inner(f): + await f + raise RuntimeError('should not be ignored') + + a = self.one_loop.create_future() + b = self.one_loop.create_future() + + async def outer(): + await asyncio.gather(inner(a), inner(b)) + + f = asyncio.ensure_future(outer(), loop=self.one_loop) + test_utils.run_briefly(self.one_loop) + a.set_result(None) + test_utils.run_briefly(self.one_loop) + b.set_result(None) + test_utils.run_briefly(self.one_loop) + self.assertIsInstance(f.exception(), RuntimeError) + + def test_issue46672(self): + with mock.patch( + 'asyncio.base_events.BaseEventLoop.call_exception_handler', + ): + async def coro(s): + return s + c = coro('abc') + + with self.assertRaises(TypeError): + self._gather(c, {}) + self._run_loop(self.one_loop) + # NameError should not happen: + self.one_loop.call_exception_handler.assert_not_called() + + +class RunCoroutineThreadsafeTests(test_utils.TestCase): + """Test case for asyncio.run_coroutine_threadsafe.""" + + def setUp(self): + super().setUp() + self.loop = asyncio.new_event_loop() + self.set_event_loop(self.loop) # Will cleanup properly + + async def add(self, a, b, fail=False, cancel=False): + """Wait 0.05 second and return a + b.""" + await asyncio.sleep(0.05) + if fail: + raise RuntimeError("Fail!") + if cancel: + asyncio.current_task(self.loop).cancel() + await asyncio.sleep(0) + return a + b + + def target(self, fail=False, cancel=False, timeout=None, + advance_coro=False): + """Run add coroutine in the event loop.""" + coro = self.add(1, 2, fail=fail, cancel=cancel) + future = asyncio.run_coroutine_threadsafe(coro, self.loop) + if advance_coro: + # this is for test_run_coroutine_threadsafe_task_factory_exception; + # otherwise it spills errors and breaks **other** unittests, since + # 'target' is interacting with threads. + + # With this call, `coro` will be advanced. + self.loop.call_soon_threadsafe(coro.send, None) + try: + return future.result(timeout) + finally: + future.done() or future.cancel() + + def test_run_coroutine_threadsafe(self): + """Test coroutine submission from a thread to an event loop.""" + future = self.loop.run_in_executor(None, self.target) + result = self.loop.run_until_complete(future) + self.assertEqual(result, 3) + + def test_run_coroutine_threadsafe_with_exception(self): + """Test coroutine submission from a thread to an event loop + when an exception is raised.""" + future = self.loop.run_in_executor(None, self.target, True) + with self.assertRaises(RuntimeError) as exc_context: + self.loop.run_until_complete(future) + self.assertIn("Fail!", exc_context.exception.args) + + def test_run_coroutine_threadsafe_with_timeout(self): + """Test coroutine submission from a thread to an event loop + when a timeout is raised.""" + callback = lambda: self.target(timeout=0) + future = self.loop.run_in_executor(None, callback) + with self.assertRaises(asyncio.TimeoutError): + self.loop.run_until_complete(future) + test_utils.run_briefly(self.loop) + # Check that there's no pending task (add has been cancelled) + for task in asyncio.all_tasks(self.loop): + self.assertTrue(task.done()) + + def test_run_coroutine_threadsafe_task_cancelled(self): + """Test coroutine submission from a thread to an event loop + when the task is cancelled.""" + callback = lambda: self.target(cancel=True) + future = self.loop.run_in_executor(None, callback) + with self.assertRaises(asyncio.CancelledError): + self.loop.run_until_complete(future) + + def test_run_coroutine_threadsafe_task_factory_exception(self): + """Test coroutine submission from a thread to an event loop + when the task factory raise an exception.""" + + def task_factory(loop, coro): + raise NameError + + run = self.loop.run_in_executor( + None, lambda: self.target(advance_coro=True)) + + # Set exception handler + callback = test_utils.MockCallback() + self.loop.set_exception_handler(callback) + + # Set corrupted task factory + self.addCleanup(self.loop.set_task_factory, + self.loop.get_task_factory()) + self.loop.set_task_factory(task_factory) + + # Run event loop + with self.assertRaises(NameError) as exc_context: + self.loop.run_until_complete(run) + + # Check exceptions + self.assertEqual(len(callback.call_args_list), 1) + (loop, context), kwargs = callback.call_args + self.assertEqual(context['exception'], exc_context.exception) + + +class SleepTests(test_utils.TestCase): + def setUp(self): + super().setUp() + self.loop = asyncio.new_event_loop() + self.set_event_loop(self.loop) + + def tearDown(self): + self.loop.close() + self.loop = None + super().tearDown() + + def test_sleep_zero(self): + result = 0 + + def inc_result(num): + nonlocal result + result += num + + async def coro(): + self.loop.call_soon(inc_result, 1) + self.assertEqual(result, 0) + num = await asyncio.sleep(0, result=10) + self.assertEqual(result, 1) # inc'ed by call_soon + inc_result(num) # num should be 11 + + self.loop.run_until_complete(coro()) + self.assertEqual(result, 11) + + +class CompatibilityTests(test_utils.TestCase): + # Tests for checking a bridge between old-styled coroutines + # and async/await syntax + + def setUp(self): + super().setUp() + self.loop = asyncio.new_event_loop() + self.set_event_loop(self.loop) + + def tearDown(self): + self.loop.close() + self.loop = None + super().tearDown() + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_threads.py b/Lib/test/test_asyncio/test_threads.py new file mode 100644 index 00000000000..8ad5f9b2c9e --- /dev/null +++ b/Lib/test/test_asyncio/test_threads.py @@ -0,0 +1,66 @@ +"""Tests for asyncio/threads.py""" + +import asyncio +import unittest + +from contextvars import ContextVar +from unittest import mock + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +class ToThreadTests(unittest.IsolatedAsyncioTestCase): + async def test_to_thread(self): + result = await asyncio.to_thread(sum, [40, 2]) + self.assertEqual(result, 42) + + async def test_to_thread_exception(self): + def raise_runtime(): + raise RuntimeError("test") + + with self.assertRaisesRegex(RuntimeError, "test"): + await asyncio.to_thread(raise_runtime) + + async def test_to_thread_once(self): + func = mock.Mock() + + await asyncio.to_thread(func) + func.assert_called_once() + + async def test_to_thread_concurrent(self): + calls = [] + def func(): + calls.append(1) + + futs = [] + for _ in range(10): + fut = asyncio.to_thread(func) + futs.append(fut) + await asyncio.gather(*futs) + + self.assertEqual(sum(calls), 10) + + async def test_to_thread_args_kwargs(self): + # Unlike run_in_executor(), to_thread() should directly accept kwargs. + func = mock.Mock() + + await asyncio.to_thread(func, 'test', something=True) + + func.assert_called_once_with('test', something=True) + + async def test_to_thread_contextvars(self): + test_ctx = ContextVar('test_ctx') + + def get_ctx(): + return test_ctx.get() + + test_ctx.set('parrot') + result = await asyncio.to_thread(get_ctx) + + self.assertEqual(result, 'parrot') + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_asyncio/test_timeouts.py b/Lib/test/test_asyncio/test_timeouts.py new file mode 100644 index 00000000000..f60722c48b7 --- /dev/null +++ b/Lib/test/test_asyncio/test_timeouts.py @@ -0,0 +1,411 @@ +"""Tests for asyncio/timeouts.py""" + +import unittest +import time + +import asyncio + +from test.test_asyncio.utils import await_without_task + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + +class TimeoutTests(unittest.IsolatedAsyncioTestCase): + + async def test_timeout_basic(self): + with self.assertRaises(TimeoutError): + async with asyncio.timeout(0.01) as cm: + await asyncio.sleep(10) + self.assertTrue(cm.expired()) + + async def test_timeout_at_basic(self): + loop = asyncio.get_running_loop() + + with self.assertRaises(TimeoutError): + deadline = loop.time() + 0.01 + async with asyncio.timeout_at(deadline) as cm: + await asyncio.sleep(10) + self.assertTrue(cm.expired()) + self.assertEqual(deadline, cm.when()) + + async def test_nested_timeouts(self): + loop = asyncio.get_running_loop() + cancelled = False + with self.assertRaises(TimeoutError): + deadline = loop.time() + 0.01 + async with asyncio.timeout_at(deadline) as cm1: + # Only the topmost context manager should raise TimeoutError + try: + async with asyncio.timeout_at(deadline) as cm2: + await asyncio.sleep(10) + except asyncio.CancelledError: + cancelled = True + raise + self.assertTrue(cancelled) + self.assertTrue(cm1.expired()) + self.assertTrue(cm2.expired()) + + async def test_waiter_cancelled(self): + cancelled = False + with self.assertRaises(TimeoutError): + async with asyncio.timeout(0.01): + try: + await asyncio.sleep(10) + except asyncio.CancelledError: + cancelled = True + raise + self.assertTrue(cancelled) + + async def test_timeout_not_called(self): + loop = asyncio.get_running_loop() + async with asyncio.timeout(10) as cm: + await asyncio.sleep(0.01) + t1 = loop.time() + + self.assertFalse(cm.expired()) + self.assertGreater(cm.when(), t1) + + async def test_timeout_disabled(self): + async with asyncio.timeout(None) as cm: + await asyncio.sleep(0.01) + + self.assertFalse(cm.expired()) + self.assertIsNone(cm.when()) + + async def test_timeout_at_disabled(self): + async with asyncio.timeout_at(None) as cm: + await asyncio.sleep(0.01) + + self.assertFalse(cm.expired()) + self.assertIsNone(cm.when()) + + async def test_timeout_zero(self): + loop = asyncio.get_running_loop() + t0 = loop.time() + with self.assertRaises(TimeoutError): + async with asyncio.timeout(0) as cm: + await asyncio.sleep(10) + t1 = loop.time() + self.assertTrue(cm.expired()) + self.assertTrue(t0 <= cm.when() <= t1) + + async def test_timeout_zero_sleep_zero(self): + loop = asyncio.get_running_loop() + t0 = loop.time() + with self.assertRaises(TimeoutError): + async with asyncio.timeout(0) as cm: + await asyncio.sleep(0) + t1 = loop.time() + self.assertTrue(cm.expired()) + self.assertTrue(t0 <= cm.when() <= t1) + + async def test_timeout_in_the_past_sleep_zero(self): + loop = asyncio.get_running_loop() + t0 = loop.time() + with self.assertRaises(TimeoutError): + async with asyncio.timeout(-11) as cm: + await asyncio.sleep(0) + t1 = loop.time() + self.assertTrue(cm.expired()) + self.assertTrue(t0 >= cm.when() <= t1) + + async def test_foreign_exception_passed(self): + with self.assertRaises(KeyError): + async with asyncio.timeout(0.01) as cm: + raise KeyError + self.assertFalse(cm.expired()) + + async def test_timeout_exception_context(self): + with self.assertRaises(TimeoutError) as cm: + async with asyncio.timeout(0.01): + try: + 1/0 + finally: + await asyncio.sleep(1) + e = cm.exception + # Expect TimeoutError caused by CancelledError raised during handling + # of ZeroDivisionError. + e2 = e.__cause__ + self.assertIsInstance(e2, asyncio.CancelledError) + self.assertIs(e.__context__, e2) + self.assertIsNone(e2.__cause__) + self.assertIsInstance(e2.__context__, ZeroDivisionError) + + async def test_foreign_exception_on_timeout(self): + async def crash(): + try: + await asyncio.sleep(1) + finally: + 1/0 + with self.assertRaises(ZeroDivisionError) as cm: + async with asyncio.timeout(0.01): + await crash() + e = cm.exception + # Expect ZeroDivisionError raised during handling of TimeoutError + # caused by CancelledError. + self.assertIsNone(e.__cause__) + e2 = e.__context__ + self.assertIsInstance(e2, TimeoutError) + e3 = e2.__cause__ + self.assertIsInstance(e3, asyncio.CancelledError) + self.assertIs(e2.__context__, e3) + + async def test_foreign_exception_on_timeout_2(self): + with self.assertRaises(ZeroDivisionError) as cm: + async with asyncio.timeout(0.01): + try: + try: + raise ValueError + finally: + await asyncio.sleep(1) + finally: + try: + raise KeyError + finally: + 1/0 + e = cm.exception + # Expect ZeroDivisionError raised during handling of KeyError + # raised during handling of TimeoutError caused by CancelledError. + self.assertIsNone(e.__cause__) + e2 = e.__context__ + self.assertIsInstance(e2, KeyError) + self.assertIsNone(e2.__cause__) + e3 = e2.__context__ + self.assertIsInstance(e3, TimeoutError) + e4 = e3.__cause__ + self.assertIsInstance(e4, asyncio.CancelledError) + self.assertIsNone(e4.__cause__) + self.assertIsInstance(e4.__context__, ValueError) + self.assertIs(e3.__context__, e4) + + async def test_foreign_cancel_doesnt_timeout_if_not_expired(self): + with self.assertRaises(asyncio.CancelledError): + async with asyncio.timeout(10) as cm: + asyncio.current_task().cancel() + await asyncio.sleep(10) + self.assertFalse(cm.expired()) + + async def test_outer_task_is_not_cancelled(self): + async def outer() -> None: + with self.assertRaises(TimeoutError): + async with asyncio.timeout(0.001): + await asyncio.sleep(10) + + task = asyncio.create_task(outer()) + await task + self.assertFalse(task.cancelled()) + self.assertTrue(task.done()) + + async def test_nested_timeouts_concurrent(self): + with self.assertRaises(TimeoutError): + async with asyncio.timeout(0.002): + with self.assertRaises(TimeoutError): + async with asyncio.timeout(0.1): + # Pretend we crunch some numbers. + time.sleep(0.01) + await asyncio.sleep(1) + + async def test_nested_timeouts_loop_busy(self): + # After the inner timeout is an expensive operation which should + # be stopped by the outer timeout. + loop = asyncio.get_running_loop() + # Disable a message about long running task + loop.slow_callback_duration = 10 + t0 = loop.time() + with self.assertRaises(TimeoutError): + async with asyncio.timeout(0.1): # (1) + with self.assertRaises(TimeoutError): + async with asyncio.timeout(0.01): # (2) + # Pretend the loop is busy for a while. + time.sleep(0.1) + await asyncio.sleep(1) + # TimeoutError was caught by (2) + await asyncio.sleep(10) # This sleep should be interrupted by (1) + t1 = loop.time() + self.assertTrue(t0 <= t1 <= t0 + 1) + + async def test_reschedule(self): + loop = asyncio.get_running_loop() + fut = loop.create_future() + deadline1 = loop.time() + 10 + deadline2 = deadline1 + 20 + + async def f(): + async with asyncio.timeout_at(deadline1) as cm: + fut.set_result(cm) + await asyncio.sleep(50) + + task = asyncio.create_task(f()) + cm = await fut + + self.assertEqual(cm.when(), deadline1) + cm.reschedule(deadline2) + self.assertEqual(cm.when(), deadline2) + cm.reschedule(None) + self.assertIsNone(cm.when()) + + task.cancel() + + with self.assertRaises(asyncio.CancelledError): + await task + self.assertFalse(cm.expired()) + + async def test_repr_active(self): + async with asyncio.timeout(10) as cm: + self.assertRegex(repr(cm), r"") + + async def test_repr_expired(self): + with self.assertRaises(TimeoutError): + async with asyncio.timeout(0.01) as cm: + await asyncio.sleep(10) + self.assertEqual(repr(cm), "") + + async def test_repr_finished(self): + async with asyncio.timeout(10) as cm: + await asyncio.sleep(0) + + self.assertEqual(repr(cm), "") + + async def test_repr_disabled(self): + async with asyncio.timeout(None) as cm: + self.assertEqual(repr(cm), r"") + + async def test_nested_timeout_in_finally(self): + with self.assertRaises(TimeoutError) as cm1: + async with asyncio.timeout(0.01): + try: + await asyncio.sleep(1) + finally: + with self.assertRaises(TimeoutError) as cm2: + async with asyncio.timeout(0.01): + await asyncio.sleep(10) + e1 = cm1.exception + # Expect TimeoutError caused by CancelledError. + e12 = e1.__cause__ + self.assertIsInstance(e12, asyncio.CancelledError) + self.assertIsNone(e12.__cause__) + self.assertIsNone(e12.__context__) + self.assertIs(e1.__context__, e12) + e2 = cm2.exception + # Expect TimeoutError caused by CancelledError raised during + # handling of other CancelledError (which is the same as in + # the above chain). + e22 = e2.__cause__ + self.assertIsInstance(e22, asyncio.CancelledError) + self.assertIsNone(e22.__cause__) + self.assertIs(e22.__context__, e12) + self.assertIs(e2.__context__, e22) + + async def test_timeout_after_cancellation(self): + try: + asyncio.current_task().cancel() + await asyncio.sleep(1) # work which will be cancelled + except asyncio.CancelledError: + pass + finally: + with self.assertRaises(TimeoutError) as cm: + async with asyncio.timeout(0.0): + await asyncio.sleep(1) # some cleanup + + async def test_cancel_in_timeout_after_cancellation(self): + try: + asyncio.current_task().cancel() + await asyncio.sleep(1) # work which will be cancelled + except asyncio.CancelledError: + pass + finally: + with self.assertRaises(asyncio.CancelledError): + async with asyncio.timeout(1.0): + asyncio.current_task().cancel() + await asyncio.sleep(2) # some cleanup + + async def test_timeout_already_entered(self): + async with asyncio.timeout(0.01) as cm: + with self.assertRaisesRegex(RuntimeError, "has already been entered"): + async with cm: + pass + + async def test_timeout_double_enter(self): + async with asyncio.timeout(0.01) as cm: + pass + with self.assertRaisesRegex(RuntimeError, "has already been entered"): + async with cm: + pass + + async def test_timeout_finished(self): + async with asyncio.timeout(0.01) as cm: + pass + with self.assertRaisesRegex(RuntimeError, "finished"): + cm.reschedule(0.02) + + async def test_timeout_expired(self): + with self.assertRaises(TimeoutError): + async with asyncio.timeout(0.01) as cm: + await asyncio.sleep(1) + with self.assertRaisesRegex(RuntimeError, "expired"): + cm.reschedule(0.02) + + async def test_timeout_expiring(self): + async with asyncio.timeout(0.01) as cm: + with self.assertRaises(asyncio.CancelledError): + await asyncio.sleep(1) + with self.assertRaisesRegex(RuntimeError, "expiring"): + cm.reschedule(0.02) + + async def test_timeout_not_entered(self): + cm = asyncio.timeout(0.01) + with self.assertRaisesRegex(RuntimeError, "has not been entered"): + cm.reschedule(0.02) + + async def test_timeout_without_task(self): + cm = asyncio.timeout(0.01) + with self.assertRaisesRegex(RuntimeError, "task"): + await await_without_task(cm.__aenter__()) + with self.assertRaisesRegex(RuntimeError, "has not been entered"): + cm.reschedule(0.02) + + async def test_timeout_taskgroup(self): + async def task(): + try: + await asyncio.sleep(2) # Will be interrupted after 0.01 second + finally: + 1/0 # Crash in cleanup + + with self.assertRaises(ExceptionGroup) as cm: + async with asyncio.timeout(0.01): + async with asyncio.TaskGroup() as tg: + tg.create_task(task()) + try: + raise ValueError + finally: + await asyncio.sleep(1) + eg = cm.exception + # Expect ExceptionGroup raised during handling of TimeoutError caused + # by CancelledError raised during handling of ValueError. + self.assertIsNone(eg.__cause__) + e_1 = eg.__context__ + self.assertIsInstance(e_1, TimeoutError) + e_2 = e_1.__cause__ + self.assertIsInstance(e_2, asyncio.CancelledError) + self.assertIsNone(e_2.__cause__) + self.assertIsInstance(e_2.__context__, ValueError) + self.assertIs(e_1.__context__, e_2) + + self.assertEqual(len(eg.exceptions), 1, eg) + e1 = eg.exceptions[0] + # Expect ZeroDivisionError raised during handling of TimeoutError + # caused by CancelledError (it is a different CancelledError). + self.assertIsInstance(e1, ZeroDivisionError) + self.assertIsNone(e1.__cause__) + e2 = e1.__context__ + self.assertIsInstance(e2, TimeoutError) + e3 = e2.__cause__ + self.assertIsInstance(e3, asyncio.CancelledError) + self.assertIsNone(e3.__context__) + self.assertIsNone(e3.__cause__) + self.assertIs(e2.__context__, e3) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_tools.py b/Lib/test/test_asyncio/test_tools.py new file mode 100644 index 00000000000..34e94830204 --- /dev/null +++ b/Lib/test/test_asyncio/test_tools.py @@ -0,0 +1,1706 @@ +import unittest + +from asyncio import tools + +from collections import namedtuple + +FrameInfo = namedtuple('FrameInfo', ['funcname', 'filename', 'lineno']) +CoroInfo = namedtuple('CoroInfo', ['call_stack', 'task_name']) +TaskInfo = namedtuple('TaskInfo', ['task_id', 'task_name', 'coroutine_stack', 'awaited_by']) +AwaitedInfo = namedtuple('AwaitedInfo', ['thread_id', 'awaited_by']) + + +# mock output of get_all_awaited_by function. +TEST_INPUTS_TREE = [ + [ + # test case containing a task called timer being awaited in two + # different subtasks part of a TaskGroup (root1 and root2) which call + # awaiter functions. + ( + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=2, + task_name="Task-1", + coroutine_stack=[], + awaited_by=[] + ), + TaskInfo( + task_id=3, + task_name="timer", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("awaiter3", "/path/to/app.py", 130), + FrameInfo("awaiter2", "/path/to/app.py", 120), + FrameInfo("awaiter", "/path/to/app.py", 110) + ], + task_name=4 + ), + CoroInfo( + call_stack=[ + FrameInfo("awaiterB3", "/path/to/app.py", 190), + FrameInfo("awaiterB2", "/path/to/app.py", 180), + FrameInfo("awaiterB", "/path/to/app.py", 170) + ], + task_name=5 + ), + CoroInfo( + call_stack=[ + FrameInfo("awaiterB3", "/path/to/app.py", 190), + FrameInfo("awaiterB2", "/path/to/app.py", 180), + FrameInfo("awaiterB", "/path/to/app.py", 170) + ], + task_name=6 + ), + CoroInfo( + call_stack=[ + FrameInfo("awaiter3", "/path/to/app.py", 130), + FrameInfo("awaiter2", "/path/to/app.py", 120), + FrameInfo("awaiter", "/path/to/app.py", 110) + ], + task_name=7 + ) + ] + ), + TaskInfo( + task_id=8, + task_name="root1", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("_aexit", "", 0), + FrameInfo("__aexit__", "", 0), + FrameInfo("main", "", 0) + ], + task_name=2 + ) + ] + ), + TaskInfo( + task_id=9, + task_name="root2", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("_aexit", "", 0), + FrameInfo("__aexit__", "", 0), + FrameInfo("main", "", 0) + ], + task_name=2 + ) + ] + ), + TaskInfo( + task_id=4, + task_name="child1_1", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("_aexit", "", 0), + FrameInfo("__aexit__", "", 0), + FrameInfo("blocho_caller", "", 0), + FrameInfo("bloch", "", 0) + ], + task_name=8 + ) + ] + ), + TaskInfo( + task_id=6, + task_name="child2_1", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("_aexit", "", 0), + FrameInfo("__aexit__", "", 0), + FrameInfo("blocho_caller", "", 0), + FrameInfo("bloch", "", 0) + ], + task_name=8 + ) + ] + ), + TaskInfo( + task_id=7, + task_name="child1_2", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("_aexit", "", 0), + FrameInfo("__aexit__", "", 0), + FrameInfo("blocho_caller", "", 0), + FrameInfo("bloch", "", 0) + ], + task_name=9 + ) + ] + ), + TaskInfo( + task_id=5, + task_name="child2_2", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("_aexit", "", 0), + FrameInfo("__aexit__", "", 0), + FrameInfo("blocho_caller", "", 0), + FrameInfo("bloch", "", 0) + ], + task_name=9 + ) + ] + ) + ] + ), + AwaitedInfo(thread_id=0, awaited_by=[]) + ), + ( + [ + [ + "└── (T) Task-1", + " └── main", + " └── __aexit__", + " └── _aexit", + " ├── (T) root1", + " │ └── bloch", + " │ └── blocho_caller", + " │ └── __aexit__", + " │ └── _aexit", + " │ ├── (T) child1_1", + " │ │ └── awaiter /path/to/app.py:110", + " │ │ └── awaiter2 /path/to/app.py:120", + " │ │ └── awaiter3 /path/to/app.py:130", + " │ │ └── (T) timer", + " │ └── (T) child2_1", + " │ └── awaiterB /path/to/app.py:170", + " │ └── awaiterB2 /path/to/app.py:180", + " │ └── awaiterB3 /path/to/app.py:190", + " │ └── (T) timer", + " └── (T) root2", + " └── bloch", + " └── blocho_caller", + " └── __aexit__", + " └── _aexit", + " ├── (T) child1_2", + " │ └── awaiter /path/to/app.py:110", + " │ └── awaiter2 /path/to/app.py:120", + " │ └── awaiter3 /path/to/app.py:130", + " │ └── (T) timer", + " └── (T) child2_2", + " └── awaiterB /path/to/app.py:170", + " └── awaiterB2 /path/to/app.py:180", + " └── awaiterB3 /path/to/app.py:190", + " └── (T) timer", + ] + ] + ), + ], + [ + # test case containing two roots + ( + AwaitedInfo( + thread_id=9, + awaited_by=[ + TaskInfo( + task_id=5, + task_name="Task-5", + coroutine_stack=[], + awaited_by=[] + ), + TaskInfo( + task_id=6, + task_name="Task-6", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main2", "", 0)], + task_name=5 + ) + ] + ), + TaskInfo( + task_id=7, + task_name="Task-7", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main2", "", 0)], + task_name=5 + ) + ] + ), + TaskInfo( + task_id=8, + task_name="Task-8", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main2", "", 0)], + task_name=5 + ) + ] + ) + ] + ), + AwaitedInfo( + thread_id=10, + awaited_by=[ + TaskInfo( + task_id=1, + task_name="Task-1", + coroutine_stack=[], + awaited_by=[] + ), + TaskInfo( + task_id=2, + task_name="Task-2", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=1 + ) + ] + ), + TaskInfo( + task_id=3, + task_name="Task-3", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=1 + ) + ] + ), + TaskInfo( + task_id=4, + task_name="Task-4", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=1 + ) + ] + ) + ] + ), + AwaitedInfo(thread_id=11, awaited_by=[]), + AwaitedInfo(thread_id=0, awaited_by=[]) + ), + ( + [ + [ + "└── (T) Task-5", + " └── main2", + " ├── (T) Task-6", + " ├── (T) Task-7", + " └── (T) Task-8", + ], + [ + "└── (T) Task-1", + " └── main", + " ├── (T) Task-2", + " ├── (T) Task-3", + " └── (T) Task-4", + ], + ] + ), + ], + [ + # test case containing two roots, one of them without subtasks + ( + [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=2, + task_name="Task-5", + coroutine_stack=[], + awaited_by=[] + ) + ] + ), + AwaitedInfo( + thread_id=3, + awaited_by=[ + TaskInfo( + task_id=4, + task_name="Task-1", + coroutine_stack=[], + awaited_by=[] + ), + TaskInfo( + task_id=5, + task_name="Task-2", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=4 + ) + ] + ), + TaskInfo( + task_id=6, + task_name="Task-3", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=4 + ) + ] + ), + TaskInfo( + task_id=7, + task_name="Task-4", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=4 + ) + ] + ) + ] + ), + AwaitedInfo(thread_id=8, awaited_by=[]), + AwaitedInfo(thread_id=0, awaited_by=[]) + ] + ), + ( + [ + ["└── (T) Task-5"], + [ + "└── (T) Task-1", + " └── main", + " ├── (T) Task-2", + " ├── (T) Task-3", + " └── (T) Task-4", + ], + ] + ), + ], +] + +TEST_INPUTS_CYCLES_TREE = [ + [ + # this test case contains a cycle: two tasks awaiting each other. + ( + [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=2, + task_name="Task-1", + coroutine_stack=[], + awaited_by=[] + ), + TaskInfo( + task_id=3, + task_name="a", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("awaiter2", "", 0)], + task_name=4 + ), + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=2 + ) + ] + ), + TaskInfo( + task_id=4, + task_name="b", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("awaiter", "", 0)], + task_name=3 + ) + ] + ) + ] + ), + AwaitedInfo(thread_id=0, awaited_by=[]) + ] + ), + ([[4, 3, 4]]), + ], + [ + # this test case contains two cycles + ( + [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=2, + task_name="Task-1", + coroutine_stack=[], + awaited_by=[] + ), + TaskInfo( + task_id=3, + task_name="A", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("nested", "", 0), + FrameInfo("nested", "", 0), + FrameInfo("task_b", "", 0) + ], + task_name=4 + ) + ] + ), + TaskInfo( + task_id=4, + task_name="B", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("nested", "", 0), + FrameInfo("nested", "", 0), + FrameInfo("task_c", "", 0) + ], + task_name=5 + ), + CoroInfo( + call_stack=[ + FrameInfo("nested", "", 0), + FrameInfo("nested", "", 0), + FrameInfo("task_a", "", 0) + ], + task_name=3 + ) + ] + ), + TaskInfo( + task_id=5, + task_name="C", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("nested", "", 0), + FrameInfo("nested", "", 0) + ], + task_name=6 + ) + ] + ), + TaskInfo( + task_id=6, + task_name="Task-2", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("nested", "", 0), + FrameInfo("nested", "", 0), + FrameInfo("task_b", "", 0) + ], + task_name=4 + ) + ] + ) + ] + ), + AwaitedInfo(thread_id=0, awaited_by=[]) + ] + ), + ([[4, 3, 4], [4, 6, 5, 4]]), + ], +] + +TEST_INPUTS_TABLE = [ + [ + # test case containing a task called timer being awaited in two + # different subtasks part of a TaskGroup (root1 and root2) which call + # awaiter functions. + ( + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=2, + task_name="Task-1", + coroutine_stack=[], + awaited_by=[] + ), + TaskInfo( + task_id=3, + task_name="timer", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("awaiter3", "", 0), + FrameInfo("awaiter2", "", 0), + FrameInfo("awaiter", "", 0) + ], + task_name=4 + ), + CoroInfo( + call_stack=[ + FrameInfo("awaiter1_3", "", 0), + FrameInfo("awaiter1_2", "", 0), + FrameInfo("awaiter1", "", 0) + ], + task_name=5 + ), + CoroInfo( + call_stack=[ + FrameInfo("awaiter1_3", "", 0), + FrameInfo("awaiter1_2", "", 0), + FrameInfo("awaiter1", "", 0) + ], + task_name=6 + ), + CoroInfo( + call_stack=[ + FrameInfo("awaiter3", "", 0), + FrameInfo("awaiter2", "", 0), + FrameInfo("awaiter", "", 0) + ], + task_name=7 + ) + ] + ), + TaskInfo( + task_id=8, + task_name="root1", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("_aexit", "", 0), + FrameInfo("__aexit__", "", 0), + FrameInfo("main", "", 0) + ], + task_name=2 + ) + ] + ), + TaskInfo( + task_id=9, + task_name="root2", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("_aexit", "", 0), + FrameInfo("__aexit__", "", 0), + FrameInfo("main", "", 0) + ], + task_name=2 + ) + ] + ), + TaskInfo( + task_id=4, + task_name="child1_1", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("_aexit", "", 0), + FrameInfo("__aexit__", "", 0), + FrameInfo("blocho_caller", "", 0), + FrameInfo("bloch", "", 0) + ], + task_name=8 + ) + ] + ), + TaskInfo( + task_id=6, + task_name="child2_1", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("_aexit", "", 0), + FrameInfo("__aexit__", "", 0), + FrameInfo("blocho_caller", "", 0), + FrameInfo("bloch", "", 0) + ], + task_name=8 + ) + ] + ), + TaskInfo( + task_id=7, + task_name="child1_2", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("_aexit", "", 0), + FrameInfo("__aexit__", "", 0), + FrameInfo("blocho_caller", "", 0), + FrameInfo("bloch", "", 0) + ], + task_name=9 + ) + ] + ), + TaskInfo( + task_id=5, + task_name="child2_2", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("_aexit", "", 0), + FrameInfo("__aexit__", "", 0), + FrameInfo("blocho_caller", "", 0), + FrameInfo("bloch", "", 0) + ], + task_name=9 + ) + ] + ) + ] + ), + AwaitedInfo(thread_id=0, awaited_by=[]) + ), + ( + [ + [1, "0x2", "Task-1", "", "", "", "0x0"], + [ + 1, + "0x3", + "timer", + "", + "awaiter3 -> awaiter2 -> awaiter", + "child1_1", + "0x4", + ], + [ + 1, + "0x3", + "timer", + "", + "awaiter1_3 -> awaiter1_2 -> awaiter1", + "child2_2", + "0x5", + ], + [ + 1, + "0x3", + "timer", + "", + "awaiter1_3 -> awaiter1_2 -> awaiter1", + "child2_1", + "0x6", + ], + [ + 1, + "0x3", + "timer", + "", + "awaiter3 -> awaiter2 -> awaiter", + "child1_2", + "0x7", + ], + [ + 1, + "0x8", + "root1", + "", + "_aexit -> __aexit__ -> main", + "Task-1", + "0x2", + ], + [ + 1, + "0x9", + "root2", + "", + "_aexit -> __aexit__ -> main", + "Task-1", + "0x2", + ], + [ + 1, + "0x4", + "child1_1", + "", + "_aexit -> __aexit__ -> blocho_caller -> bloch", + "root1", + "0x8", + ], + [ + 1, + "0x6", + "child2_1", + "", + "_aexit -> __aexit__ -> blocho_caller -> bloch", + "root1", + "0x8", + ], + [ + 1, + "0x7", + "child1_2", + "", + "_aexit -> __aexit__ -> blocho_caller -> bloch", + "root2", + "0x9", + ], + [ + 1, + "0x5", + "child2_2", + "", + "_aexit -> __aexit__ -> blocho_caller -> bloch", + "root2", + "0x9", + ], + ] + ), + ], + [ + # test case containing two roots + ( + AwaitedInfo( + thread_id=9, + awaited_by=[ + TaskInfo( + task_id=5, + task_name="Task-5", + coroutine_stack=[], + awaited_by=[] + ), + TaskInfo( + task_id=6, + task_name="Task-6", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main2", "", 0)], + task_name=5 + ) + ] + ), + TaskInfo( + task_id=7, + task_name="Task-7", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main2", "", 0)], + task_name=5 + ) + ] + ), + TaskInfo( + task_id=8, + task_name="Task-8", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main2", "", 0)], + task_name=5 + ) + ] + ) + ] + ), + AwaitedInfo( + thread_id=10, + awaited_by=[ + TaskInfo( + task_id=1, + task_name="Task-1", + coroutine_stack=[], + awaited_by=[] + ), + TaskInfo( + task_id=2, + task_name="Task-2", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=1 + ) + ] + ), + TaskInfo( + task_id=3, + task_name="Task-3", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=1 + ) + ] + ), + TaskInfo( + task_id=4, + task_name="Task-4", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=1 + ) + ] + ) + ] + ), + AwaitedInfo(thread_id=11, awaited_by=[]), + AwaitedInfo(thread_id=0, awaited_by=[]) + ), + ( + [ + [9, "0x5", "Task-5", "", "", "", "0x0"], + [9, "0x6", "Task-6", "", "main2", "Task-5", "0x5"], + [9, "0x7", "Task-7", "", "main2", "Task-5", "0x5"], + [9, "0x8", "Task-8", "", "main2", "Task-5", "0x5"], + [10, "0x1", "Task-1", "", "", "", "0x0"], + [10, "0x2", "Task-2", "", "main", "Task-1", "0x1"], + [10, "0x3", "Task-3", "", "main", "Task-1", "0x1"], + [10, "0x4", "Task-4", "", "main", "Task-1", "0x1"], + ] + ), + ], + [ + # test case containing two roots, one of them without subtasks + ( + [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=2, + task_name="Task-5", + coroutine_stack=[], + awaited_by=[] + ) + ] + ), + AwaitedInfo( + thread_id=3, + awaited_by=[ + TaskInfo( + task_id=4, + task_name="Task-1", + coroutine_stack=[], + awaited_by=[] + ), + TaskInfo( + task_id=5, + task_name="Task-2", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=4 + ) + ] + ), + TaskInfo( + task_id=6, + task_name="Task-3", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=4 + ) + ] + ), + TaskInfo( + task_id=7, + task_name="Task-4", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=4 + ) + ] + ) + ] + ), + AwaitedInfo(thread_id=8, awaited_by=[]), + AwaitedInfo(thread_id=0, awaited_by=[]) + ] + ), + ( + [ + [1, "0x2", "Task-5", "", "", "", "0x0"], + [3, "0x4", "Task-1", "", "", "", "0x0"], + [3, "0x5", "Task-2", "", "main", "Task-1", "0x4"], + [3, "0x6", "Task-3", "", "main", "Task-1", "0x4"], + [3, "0x7", "Task-4", "", "main", "Task-1", "0x4"], + ] + ), + ], + # CASES WITH CYCLES + [ + # this test case contains a cycle: two tasks awaiting each other. + ( + [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=2, + task_name="Task-1", + coroutine_stack=[], + awaited_by=[] + ), + TaskInfo( + task_id=3, + task_name="a", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("awaiter2", "", 0)], + task_name=4 + ), + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=2 + ) + ] + ), + TaskInfo( + task_id=4, + task_name="b", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("awaiter", "", 0)], + task_name=3 + ) + ] + ) + ] + ), + AwaitedInfo(thread_id=0, awaited_by=[]) + ] + ), + ( + [ + [1, "0x2", "Task-1", "", "", "", "0x0"], + [1, "0x3", "a", "", "awaiter2", "b", "0x4"], + [1, "0x3", "a", "", "main", "Task-1", "0x2"], + [1, "0x4", "b", "", "awaiter", "a", "0x3"], + ] + ), + ], + [ + # this test case contains two cycles + ( + [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=2, + task_name="Task-1", + coroutine_stack=[], + awaited_by=[] + ), + TaskInfo( + task_id=3, + task_name="A", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("nested", "", 0), + FrameInfo("nested", "", 0), + FrameInfo("task_b", "", 0) + ], + task_name=4 + ) + ] + ), + TaskInfo( + task_id=4, + task_name="B", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("nested", "", 0), + FrameInfo("nested", "", 0), + FrameInfo("task_c", "", 0) + ], + task_name=5 + ), + CoroInfo( + call_stack=[ + FrameInfo("nested", "", 0), + FrameInfo("nested", "", 0), + FrameInfo("task_a", "", 0) + ], + task_name=3 + ) + ] + ), + TaskInfo( + task_id=5, + task_name="C", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("nested", "", 0), + FrameInfo("nested", "", 0) + ], + task_name=6 + ) + ] + ), + TaskInfo( + task_id=6, + task_name="Task-2", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("nested", "", 0), + FrameInfo("nested", "", 0), + FrameInfo("task_b", "", 0) + ], + task_name=4 + ) + ] + ) + ] + ), + AwaitedInfo(thread_id=0, awaited_by=[]) + ] + ), + ( + [ + [1, "0x2", "Task-1", "", "", "", "0x0"], + [ + 1, + "0x3", + "A", + "", + "nested -> nested -> task_b", + "B", + "0x4", + ], + [ + 1, + "0x4", + "B", + "", + "nested -> nested -> task_c", + "C", + "0x5", + ], + [ + 1, + "0x4", + "B", + "", + "nested -> nested -> task_a", + "A", + "0x3", + ], + [ + 1, + "0x5", + "C", + "", + "nested -> nested", + "Task-2", + "0x6", + ], + [ + 1, + "0x6", + "Task-2", + "", + "nested -> nested -> task_b", + "B", + "0x4", + ], + ] + ), + ], +] + + +class TestAsyncioToolsTree(unittest.TestCase): + def test_asyncio_utils(self): + for input_, tree in TEST_INPUTS_TREE: + with self.subTest(input_): + result = tools.build_async_tree(input_) + self.assertEqual(result, tree) + + def test_asyncio_utils_cycles(self): + for input_, cycles in TEST_INPUTS_CYCLES_TREE: + with self.subTest(input_): + try: + tools.build_async_tree(input_) + except tools.CycleFoundException as e: + self.assertEqual(e.cycles, cycles) + + +class TestAsyncioToolsTable(unittest.TestCase): + def test_asyncio_utils(self): + for input_, table in TEST_INPUTS_TABLE: + with self.subTest(input_): + result = tools.build_task_table(input_) + self.assertEqual(result, table) + + +class TestAsyncioToolsBasic(unittest.TestCase): + def test_empty_input_tree(self): + """Test build_async_tree with empty input.""" + result = [] + expected_output = [] + self.assertEqual(tools.build_async_tree(result), expected_output) + + def test_empty_input_table(self): + """Test build_task_table with empty input.""" + result = [] + expected_output = [] + self.assertEqual(tools.build_task_table(result), expected_output) + + def test_only_independent_tasks_tree(self): + input_ = [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=10, + task_name="taskA", + coroutine_stack=[], + awaited_by=[] + ), + TaskInfo( + task_id=11, + task_name="taskB", + coroutine_stack=[], + awaited_by=[] + ) + ] + ) + ] + expected = [["└── (T) taskA"], ["└── (T) taskB"]] + result = tools.build_async_tree(input_) + self.assertEqual(sorted(result), sorted(expected)) + + def test_only_independent_tasks_table(self): + input_ = [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=10, + task_name="taskA", + coroutine_stack=[], + awaited_by=[] + ), + TaskInfo( + task_id=11, + task_name="taskB", + coroutine_stack=[], + awaited_by=[] + ) + ] + ) + ] + self.assertEqual( + tools.build_task_table(input_), + [[1, '0xa', 'taskA', '', '', '', '0x0'], [1, '0xb', 'taskB', '', '', '', '0x0']] + ) + + def test_single_task_tree(self): + """Test build_async_tree with a single task and no awaits.""" + result = [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=2, + task_name="Task-1", + coroutine_stack=[], + awaited_by=[] + ) + ] + ) + ] + expected_output = [ + [ + "└── (T) Task-1", + ] + ] + self.assertEqual(tools.build_async_tree(result), expected_output) + + def test_single_task_table(self): + """Test build_task_table with a single task and no awaits.""" + result = [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=2, + task_name="Task-1", + coroutine_stack=[], + awaited_by=[] + ) + ] + ) + ] + expected_output = [[1, '0x2', 'Task-1', '', '', '', '0x0']] + self.assertEqual(tools.build_task_table(result), expected_output) + + def test_cycle_detection(self): + """Test build_async_tree raises CycleFoundException for cyclic input.""" + result = [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=2, + task_name="Task-1", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=3 + ) + ] + ), + TaskInfo( + task_id=3, + task_name="Task-2", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=2 + ) + ] + ) + ] + ) + ] + with self.assertRaises(tools.CycleFoundException) as context: + tools.build_async_tree(result) + self.assertEqual(context.exception.cycles, [[3, 2, 3]]) + + def test_complex_tree(self): + """Test build_async_tree with a more complex tree structure.""" + result = [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=2, + task_name="Task-1", + coroutine_stack=[], + awaited_by=[] + ), + TaskInfo( + task_id=3, + task_name="Task-2", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=2 + ) + ] + ), + TaskInfo( + task_id=4, + task_name="Task-3", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=3 + ) + ] + ) + ] + ) + ] + expected_output = [ + [ + "└── (T) Task-1", + " └── main", + " └── (T) Task-2", + " └── main", + " └── (T) Task-3", + ] + ] + self.assertEqual(tools.build_async_tree(result), expected_output) + + def test_complex_table(self): + """Test build_task_table with a more complex tree structure.""" + result = [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=2, + task_name="Task-1", + coroutine_stack=[], + awaited_by=[] + ), + TaskInfo( + task_id=3, + task_name="Task-2", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=2 + ) + ] + ), + TaskInfo( + task_id=4, + task_name="Task-3", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("main", "", 0)], + task_name=3 + ) + ] + ) + ] + ) + ] + expected_output = [ + [1, '0x2', 'Task-1', '', '', '', '0x0'], + [1, '0x3', 'Task-2', '', 'main', 'Task-1', '0x2'], + [1, '0x4', 'Task-3', '', 'main', 'Task-2', '0x3'] + ] + self.assertEqual(tools.build_task_table(result), expected_output) + + def test_deep_coroutine_chain(self): + input_ = [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=10, + task_name="leaf", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("c1", "", 0), + FrameInfo("c2", "", 0), + FrameInfo("c3", "", 0), + FrameInfo("c4", "", 0), + FrameInfo("c5", "", 0) + ], + task_name=11 + ) + ] + ), + TaskInfo( + task_id=11, + task_name="root", + coroutine_stack=[], + awaited_by=[] + ) + ] + ) + ] + expected = [ + [ + "└── (T) root", + " └── c5", + " └── c4", + " └── c3", + " └── c2", + " └── c1", + " └── (T) leaf", + ] + ] + result = tools.build_async_tree(input_) + self.assertEqual(result, expected) + + def test_multiple_cycles_same_node(self): + input_ = [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=1, + task_name="Task-A", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("call1", "", 0)], + task_name=2 + ) + ] + ), + TaskInfo( + task_id=2, + task_name="Task-B", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("call2", "", 0)], + task_name=3 + ) + ] + ), + TaskInfo( + task_id=3, + task_name="Task-C", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("call3", "", 0)], + task_name=1 + ), + CoroInfo( + call_stack=[FrameInfo("call4", "", 0)], + task_name=2 + ) + ] + ) + ] + ) + ] + with self.assertRaises(tools.CycleFoundException) as ctx: + tools.build_async_tree(input_) + cycles = ctx.exception.cycles + self.assertTrue(any(set(c) == {1, 2, 3} for c in cycles)) + + def test_table_output_format(self): + input_ = [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=1, + task_name="Task-A", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("foo", "", 0)], + task_name=2 + ) + ] + ), + TaskInfo( + task_id=2, + task_name="Task-B", + coroutine_stack=[], + awaited_by=[] + ) + ] + ) + ] + table = tools.build_task_table(input_) + for row in table: + self.assertEqual(len(row), 7) + self.assertIsInstance(row[0], int) # thread ID + self.assertTrue( + isinstance(row[1], str) and row[1].startswith("0x") + ) # hex task ID + self.assertIsInstance(row[2], str) # task name + self.assertIsInstance(row[3], str) # coroutine stack + self.assertIsInstance(row[4], str) # coroutine chain + self.assertIsInstance(row[5], str) # awaiter name + self.assertTrue( + isinstance(row[6], str) and row[6].startswith("0x") + ) # hex awaiter ID + + +class TestAsyncioToolsEdgeCases(unittest.TestCase): + + def test_task_awaits_self(self): + """A task directly awaits itself - should raise a cycle.""" + input_ = [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=1, + task_name="Self-Awaiter", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("loopback", "", 0)], + task_name=1 + ) + ] + ) + ] + ) + ] + with self.assertRaises(tools.CycleFoundException) as ctx: + tools.build_async_tree(input_) + self.assertIn([1, 1], ctx.exception.cycles) + + def test_task_with_missing_awaiter_id(self): + """Awaiter ID not in task list - should not crash, just show 'Unknown'.""" + input_ = [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=1, + task_name="Task-A", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("coro", "", 0)], + task_name=999 + ) + ] + ) + ] + ) + ] + table = tools.build_task_table(input_) + self.assertEqual(len(table), 1) + self.assertEqual(table[0][5], "Unknown") + + def test_duplicate_coroutine_frames(self): + """Same coroutine frame repeated under a parent - should deduplicate.""" + input_ = [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=1, + task_name="Task-1", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("frameA", "", 0)], + task_name=2 + ), + CoroInfo( + call_stack=[FrameInfo("frameA", "", 0)], + task_name=3 + ) + ] + ), + TaskInfo( + task_id=2, + task_name="Task-2", + coroutine_stack=[], + awaited_by=[] + ), + TaskInfo( + task_id=3, + task_name="Task-3", + coroutine_stack=[], + awaited_by=[] + ) + ] + ) + ] + tree = tools.build_async_tree(input_) + # Both children should be under the same coroutine node + flat = "\n".join(tree[0]) + self.assertIn("frameA", flat) + self.assertIn("Task-2", flat) + self.assertIn("Task-1", flat) + + flat = "\n".join(tree[1]) + self.assertIn("frameA", flat) + self.assertIn("Task-3", flat) + self.assertIn("Task-1", flat) + + def test_task_with_no_name(self): + """Task with no name in id2name - should still render with fallback.""" + input_ = [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=1, + task_name="root", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[FrameInfo("f1", "", 0)], + task_name=2 + ) + ] + ), + TaskInfo( + task_id=2, + task_name=None, + coroutine_stack=[], + awaited_by=[] + ) + ] + ) + ] + # If name is None, fallback to string should not crash + tree = tools.build_async_tree(input_) + self.assertIn("(T) None", "\n".join(tree[0])) + + def test_tree_rendering_with_custom_emojis(self): + """Pass custom emojis to the tree renderer.""" + input_ = [ + AwaitedInfo( + thread_id=1, + awaited_by=[ + TaskInfo( + task_id=1, + task_name="MainTask", + coroutine_stack=[], + awaited_by=[ + CoroInfo( + call_stack=[ + FrameInfo("f1", "", 0), + FrameInfo("f2", "", 0) + ], + task_name=2 + ) + ] + ), + TaskInfo( + task_id=2, + task_name="SubTask", + coroutine_stack=[], + awaited_by=[] + ) + ] + ) + ] + tree = tools.build_async_tree(input_, task_emoji="🧵", cor_emoji="🔁") + flat = "\n".join(tree[0]) + self.assertIn("🧵 MainTask", flat) + self.assertIn("🔁 f1", flat) + self.assertIn("🔁 f2", flat) + self.assertIn("🧵 SubTask", flat) diff --git a/Lib/test/test_asyncio/test_transports.py b/Lib/test/test_asyncio/test_transports.py new file mode 100644 index 00000000000..dbb572e2e15 --- /dev/null +++ b/Lib/test/test_asyncio/test_transports.py @@ -0,0 +1,103 @@ +"""Tests for transports.py.""" + +import unittest +from unittest import mock + +import asyncio +from asyncio import transports + + +def tearDownModule(): + # not needed for the test file but added for uniformness with all other + # asyncio test files for the sake of unified cleanup + asyncio.events._set_event_loop_policy(None) + + +class TransportTests(unittest.TestCase): + + def test_ctor_extra_is_none(self): + transport = asyncio.Transport() + self.assertEqual(transport._extra, {}) + + def test_get_extra_info(self): + transport = asyncio.Transport({'extra': 'info'}) + self.assertEqual('info', transport.get_extra_info('extra')) + self.assertIsNone(transport.get_extra_info('unknown')) + + default = object() + self.assertIs(default, transport.get_extra_info('unknown', default)) + + def test_writelines(self): + writer = mock.Mock() + + class MyTransport(asyncio.Transport): + def write(self, data): + writer(data) + + transport = MyTransport() + + transport.writelines([b'line1', + bytearray(b'line2'), + memoryview(b'line3')]) + self.assertEqual(1, writer.call_count) + writer.assert_called_with(b'line1line2line3') + + def test_not_implemented(self): + transport = asyncio.Transport() + + self.assertRaises(NotImplementedError, + transport.set_write_buffer_limits) + self.assertRaises(NotImplementedError, transport.get_write_buffer_size) + self.assertRaises(NotImplementedError, transport.write, 'data') + self.assertRaises(NotImplementedError, transport.write_eof) + self.assertRaises(NotImplementedError, transport.can_write_eof) + self.assertRaises(NotImplementedError, transport.pause_reading) + self.assertRaises(NotImplementedError, transport.resume_reading) + self.assertRaises(NotImplementedError, transport.is_reading) + self.assertRaises(NotImplementedError, transport.close) + self.assertRaises(NotImplementedError, transport.abort) + + def test_dgram_not_implemented(self): + transport = asyncio.DatagramTransport() + + self.assertRaises(NotImplementedError, transport.sendto, 'data') + self.assertRaises(NotImplementedError, transport.abort) + + def test_subprocess_transport_not_implemented(self): + transport = asyncio.SubprocessTransport() + + self.assertRaises(NotImplementedError, transport.get_pid) + self.assertRaises(NotImplementedError, transport.get_returncode) + self.assertRaises(NotImplementedError, transport.get_pipe_transport, 1) + self.assertRaises(NotImplementedError, transport.send_signal, 1) + self.assertRaises(NotImplementedError, transport.terminate) + self.assertRaises(NotImplementedError, transport.kill) + + def test_flowcontrol_mixin_set_write_limits(self): + + class MyTransport(transports._FlowControlMixin, + transports.Transport): + + def get_write_buffer_size(self): + return 512 + + loop = mock.Mock() + transport = MyTransport(loop=loop) + transport._protocol = mock.Mock() + + self.assertFalse(transport._protocol_paused) + + with self.assertRaisesRegex(ValueError, 'high.*must be >= low'): + transport.set_write_buffer_limits(high=0, low=1) + + transport.set_write_buffer_limits(high=1024, low=128) + self.assertFalse(transport._protocol_paused) + self.assertEqual(transport.get_write_buffer_limits(), (128, 1024)) + + transport.set_write_buffer_limits(high=256, low=128) + self.assertTrue(transport._protocol_paused) + self.assertEqual(transport.get_write_buffer_limits(), (128, 256)) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py new file mode 100644 index 00000000000..0faf32f79ea --- /dev/null +++ b/Lib/test/test_asyncio/test_unix_events.py @@ -0,0 +1,1334 @@ +"""Tests for unix_events.py.""" + +import contextlib +import errno +import io +import multiprocessing +from multiprocessing.util import _cleanup_tests as multiprocessing_cleanup_tests +import os +import signal +import socket +import stat +import sys +import time +import unittest +from unittest import mock + +from test import support +from test.support import os_helper +from test.support import socket_helper +from test.support import wait_process +from test.support import hashlib_helper + +if sys.platform == 'win32': + raise unittest.SkipTest('UNIX only') + + +import asyncio +from asyncio import unix_events +from test.test_asyncio import utils as test_utils + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +MOCK_ANY = mock.ANY + + +def EXITCODE(exitcode): + return 32768 + exitcode + + +def SIGNAL(signum): + if not 1 <= signum <= 68: + raise AssertionError(f'invalid signum {signum}') + return 32768 - signum + + +def close_pipe_transport(transport): + # Don't call transport.close() because the event loop and the selector + # are mocked + if transport._pipe is None: + return + transport._pipe.close() + transport._pipe = None + + +@unittest.skipUnless(signal, 'Signals are not supported') +class SelectorEventLoopSignalTests(test_utils.TestCase): + + def setUp(self): + super().setUp() + self.loop = asyncio.SelectorEventLoop() + self.set_event_loop(self.loop) + + def test_check_signal(self): + self.assertRaises( + TypeError, self.loop._check_signal, '1') + self.assertRaises( + ValueError, self.loop._check_signal, signal.NSIG + 1) + + def test_handle_signal_no_handler(self): + self.loop._handle_signal(signal.NSIG + 1) + + def test_handle_signal_cancelled_handler(self): + h = asyncio.Handle(mock.Mock(), (), + loop=mock.Mock()) + h.cancel() + self.loop._signal_handlers[signal.NSIG + 1] = h + self.loop.remove_signal_handler = mock.Mock() + self.loop._handle_signal(signal.NSIG + 1) + self.loop.remove_signal_handler.assert_called_with(signal.NSIG + 1) + + @mock.patch('asyncio.unix_events.signal') + def test_add_signal_handler_setup_error(self, m_signal): + m_signal.NSIG = signal.NSIG + m_signal.valid_signals = signal.valid_signals + m_signal.set_wakeup_fd.side_effect = ValueError + + self.assertRaises( + RuntimeError, + self.loop.add_signal_handler, + signal.SIGINT, lambda: True) + + @mock.patch('asyncio.unix_events.signal') + def test_add_signal_handler_coroutine_error(self, m_signal): + m_signal.NSIG = signal.NSIG + + async def simple_coroutine(): + pass + + # callback must not be a coroutine function + coro_func = simple_coroutine + coro_obj = coro_func() + self.addCleanup(coro_obj.close) + for func in (coro_func, coro_obj): + self.assertRaisesRegex( + TypeError, 'coroutines cannot be used with add_signal_handler', + self.loop.add_signal_handler, + signal.SIGINT, func) + + @mock.patch('asyncio.unix_events.signal') + def test_add_signal_handler(self, m_signal): + m_signal.NSIG = signal.NSIG + m_signal.valid_signals = signal.valid_signals + + cb = lambda: True + self.loop.add_signal_handler(signal.SIGHUP, cb) + h = self.loop._signal_handlers.get(signal.SIGHUP) + self.assertIsInstance(h, asyncio.Handle) + self.assertEqual(h._callback, cb) + + @mock.patch('asyncio.unix_events.signal') + def test_add_signal_handler_install_error(self, m_signal): + m_signal.NSIG = signal.NSIG + m_signal.valid_signals = signal.valid_signals + + def set_wakeup_fd(fd): + if fd == -1: + raise ValueError() + m_signal.set_wakeup_fd = set_wakeup_fd + + class Err(OSError): + errno = errno.EFAULT + m_signal.signal.side_effect = Err + + self.assertRaises( + Err, + self.loop.add_signal_handler, + signal.SIGINT, lambda: True) + + @mock.patch('asyncio.unix_events.signal') + @mock.patch('asyncio.base_events.logger') + def test_add_signal_handler_install_error2(self, m_logging, m_signal): + m_signal.NSIG = signal.NSIG + m_signal.valid_signals = signal.valid_signals + + class Err(OSError): + errno = errno.EINVAL + m_signal.signal.side_effect = Err + + self.loop._signal_handlers[signal.SIGHUP] = lambda: True + self.assertRaises( + RuntimeError, + self.loop.add_signal_handler, + signal.SIGINT, lambda: True) + self.assertFalse(m_logging.info.called) + self.assertEqual(1, m_signal.set_wakeup_fd.call_count) + + @mock.patch('asyncio.unix_events.signal') + @mock.patch('asyncio.base_events.logger') + def test_add_signal_handler_install_error3(self, m_logging, m_signal): + class Err(OSError): + errno = errno.EINVAL + m_signal.signal.side_effect = Err + m_signal.NSIG = signal.NSIG + m_signal.valid_signals = signal.valid_signals + + self.assertRaises( + RuntimeError, + self.loop.add_signal_handler, + signal.SIGINT, lambda: True) + self.assertFalse(m_logging.info.called) + self.assertEqual(2, m_signal.set_wakeup_fd.call_count) + + @mock.patch('asyncio.unix_events.signal') + def test_remove_signal_handler(self, m_signal): + m_signal.NSIG = signal.NSIG + m_signal.valid_signals = signal.valid_signals + + self.loop.add_signal_handler(signal.SIGHUP, lambda: True) + + self.assertTrue( + self.loop.remove_signal_handler(signal.SIGHUP)) + self.assertTrue(m_signal.set_wakeup_fd.called) + self.assertTrue(m_signal.signal.called) + self.assertEqual( + (signal.SIGHUP, m_signal.SIG_DFL), m_signal.signal.call_args[0]) + + @mock.patch('asyncio.unix_events.signal') + def test_remove_signal_handler_2(self, m_signal): + m_signal.NSIG = signal.NSIG + m_signal.SIGINT = signal.SIGINT + m_signal.valid_signals = signal.valid_signals + + self.loop.add_signal_handler(signal.SIGINT, lambda: True) + self.loop._signal_handlers[signal.SIGHUP] = object() + m_signal.set_wakeup_fd.reset_mock() + + self.assertTrue( + self.loop.remove_signal_handler(signal.SIGINT)) + self.assertFalse(m_signal.set_wakeup_fd.called) + self.assertTrue(m_signal.signal.called) + self.assertEqual( + (signal.SIGINT, m_signal.default_int_handler), + m_signal.signal.call_args[0]) + + @mock.patch('asyncio.unix_events.signal') + @mock.patch('asyncio.base_events.logger') + def test_remove_signal_handler_cleanup_error(self, m_logging, m_signal): + m_signal.NSIG = signal.NSIG + m_signal.valid_signals = signal.valid_signals + self.loop.add_signal_handler(signal.SIGHUP, lambda: True) + + m_signal.set_wakeup_fd.side_effect = ValueError + + self.loop.remove_signal_handler(signal.SIGHUP) + self.assertTrue(m_logging.info) + + @mock.patch('asyncio.unix_events.signal') + def test_remove_signal_handler_error(self, m_signal): + m_signal.NSIG = signal.NSIG + m_signal.valid_signals = signal.valid_signals + self.loop.add_signal_handler(signal.SIGHUP, lambda: True) + + m_signal.signal.side_effect = OSError + + self.assertRaises( + OSError, self.loop.remove_signal_handler, signal.SIGHUP) + + @mock.patch('asyncio.unix_events.signal') + def test_remove_signal_handler_error2(self, m_signal): + m_signal.NSIG = signal.NSIG + m_signal.valid_signals = signal.valid_signals + self.loop.add_signal_handler(signal.SIGHUP, lambda: True) + + class Err(OSError): + errno = errno.EINVAL + m_signal.signal.side_effect = Err + + self.assertRaises( + RuntimeError, self.loop.remove_signal_handler, signal.SIGHUP) + + @mock.patch('asyncio.unix_events.signal') + def test_close(self, m_signal): + m_signal.NSIG = signal.NSIG + m_signal.valid_signals = signal.valid_signals + + self.loop.add_signal_handler(signal.SIGHUP, lambda: True) + self.loop.add_signal_handler(signal.SIGCHLD, lambda: True) + + self.assertEqual(len(self.loop._signal_handlers), 2) + + m_signal.set_wakeup_fd.reset_mock() + + self.loop.close() + + self.assertEqual(len(self.loop._signal_handlers), 0) + m_signal.set_wakeup_fd.assert_called_once_with(-1) + + @mock.patch('asyncio.unix_events.sys') + @mock.patch('asyncio.unix_events.signal') + def test_close_on_finalizing(self, m_signal, m_sys): + m_signal.NSIG = signal.NSIG + m_signal.valid_signals = signal.valid_signals + self.loop.add_signal_handler(signal.SIGHUP, lambda: True) + + self.assertEqual(len(self.loop._signal_handlers), 1) + m_sys.is_finalizing.return_value = True + m_signal.signal.reset_mock() + + with self.assertWarnsRegex(ResourceWarning, + "skipping signal handlers removal"): + self.loop.close() + + self.assertEqual(len(self.loop._signal_handlers), 0) + self.assertFalse(m_signal.signal.called) + + +@unittest.skipUnless(hasattr(socket, 'AF_UNIX'), + 'UNIX Sockets are not supported') +class SelectorEventLoopUnixSocketTests(test_utils.TestCase): + + def setUp(self): + super().setUp() + self.loop = asyncio.SelectorEventLoop() + self.set_event_loop(self.loop) + + @socket_helper.skip_unless_bind_unix_socket + def test_create_unix_server_existing_path_sock(self): + with test_utils.unix_socket_path() as path: + sock = socket.socket(socket.AF_UNIX) + sock.bind(path) + sock.listen(1) + sock.close() + + coro = self.loop.create_unix_server(lambda: None, path) + srv = self.loop.run_until_complete(coro) + srv.close() + self.loop.run_until_complete(srv.wait_closed()) + + @socket_helper.skip_unless_bind_unix_socket + def test_create_unix_server_pathlike(self): + with test_utils.unix_socket_path() as path: + path = os_helper.FakePath(path) + srv_coro = self.loop.create_unix_server(lambda: None, path) + srv = self.loop.run_until_complete(srv_coro) + srv.close() + self.loop.run_until_complete(srv.wait_closed()) + + def test_create_unix_connection_pathlike(self): + with test_utils.unix_socket_path() as path: + path = os_helper.FakePath(path) + coro = self.loop.create_unix_connection(lambda: None, path) + with self.assertRaises(FileNotFoundError): + # If path-like object weren't supported, the exception would be + # different. + self.loop.run_until_complete(coro) + + def test_create_unix_server_existing_path_nonsock(self): + path = test_utils.gen_unix_socket_path() + self.addCleanup(os_helper.unlink, path) + # create the file + open(path, "wb").close() + + coro = self.loop.create_unix_server(lambda: None, path) + with self.assertRaisesRegex(OSError, + 'Address.*is already in use'): + self.loop.run_until_complete(coro) + + def test_create_unix_server_ssl_bool(self): + coro = self.loop.create_unix_server(lambda: None, path='spam', + ssl=True) + with self.assertRaisesRegex(TypeError, + 'ssl argument must be an SSLContext'): + self.loop.run_until_complete(coro) + + def test_create_unix_server_nopath_nosock(self): + coro = self.loop.create_unix_server(lambda: None, path=None) + with self.assertRaisesRegex(ValueError, + 'path was not specified, and no sock'): + self.loop.run_until_complete(coro) + + def test_create_unix_server_path_inetsock(self): + sock = socket.socket() + with sock: + coro = self.loop.create_unix_server(lambda: None, path=None, + sock=sock) + with self.assertRaisesRegex(ValueError, + 'A UNIX Domain Stream.*was expected'): + self.loop.run_until_complete(coro) + + def test_create_unix_server_path_dgram(self): + sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) + with sock: + coro = self.loop.create_unix_server(lambda: None, path=None, + sock=sock) + with self.assertRaisesRegex(ValueError, + 'A UNIX Domain Stream.*was expected'): + self.loop.run_until_complete(coro) + + @unittest.skipUnless(hasattr(socket, 'SOCK_NONBLOCK'), + 'no socket.SOCK_NONBLOCK (linux only)') + @socket_helper.skip_unless_bind_unix_socket + def test_create_unix_server_path_stream_bittype(self): + fn = test_utils.gen_unix_socket_path() + self.addCleanup(os_helper.unlink, fn) + + sock = socket.socket(socket.AF_UNIX, + socket.SOCK_STREAM | socket.SOCK_NONBLOCK) + with sock: + sock.bind(fn) + coro = self.loop.create_unix_server(lambda: None, path=None, + sock=sock) + srv = self.loop.run_until_complete(coro) + srv.close() + self.loop.run_until_complete(srv.wait_closed()) + + def test_create_unix_server_ssl_timeout_with_plain_sock(self): + coro = self.loop.create_unix_server(lambda: None, path='spam', + ssl_handshake_timeout=1) + with self.assertRaisesRegex( + ValueError, + 'ssl_handshake_timeout is only meaningful with ssl'): + self.loop.run_until_complete(coro) + + def test_create_unix_connection_path_inetsock(self): + sock = socket.socket() + with sock: + coro = self.loop.create_unix_connection(lambda: None, + sock=sock) + with self.assertRaisesRegex(ValueError, + 'A UNIX Domain Stream.*was expected'): + self.loop.run_until_complete(coro) + + @mock.patch('asyncio.unix_events.socket') + def test_create_unix_server_bind_error(self, m_socket): + # Ensure that the socket is closed on any bind error + sock = mock.Mock() + m_socket.socket.return_value = sock + + sock.bind.side_effect = OSError + coro = self.loop.create_unix_server(lambda: None, path="/test") + with self.assertRaises(OSError): + self.loop.run_until_complete(coro) + self.assertTrue(sock.close.called) + + sock.bind.side_effect = MemoryError + coro = self.loop.create_unix_server(lambda: None, path="/test") + with self.assertRaises(MemoryError): + self.loop.run_until_complete(coro) + self.assertTrue(sock.close.called) + + def test_create_unix_connection_path_sock(self): + coro = self.loop.create_unix_connection( + lambda: None, os.devnull, sock=object()) + with self.assertRaisesRegex(ValueError, 'path and sock can not be'): + self.loop.run_until_complete(coro) + + def test_create_unix_connection_nopath_nosock(self): + coro = self.loop.create_unix_connection( + lambda: None, None) + with self.assertRaisesRegex(ValueError, + 'no path and sock were specified'): + self.loop.run_until_complete(coro) + + def test_create_unix_connection_nossl_serverhost(self): + coro = self.loop.create_unix_connection( + lambda: None, os.devnull, server_hostname='spam') + with self.assertRaisesRegex(ValueError, + 'server_hostname is only meaningful'): + self.loop.run_until_complete(coro) + + def test_create_unix_connection_ssl_noserverhost(self): + coro = self.loop.create_unix_connection( + lambda: None, os.devnull, ssl=True) + + with self.assertRaisesRegex( + ValueError, 'you have to pass server_hostname when using ssl'): + + self.loop.run_until_complete(coro) + + def test_create_unix_connection_ssl_timeout_with_plain_sock(self): + coro = self.loop.create_unix_connection(lambda: None, path='spam', + ssl_handshake_timeout=1) + with self.assertRaisesRegex( + ValueError, + 'ssl_handshake_timeout is only meaningful with ssl'): + self.loop.run_until_complete(coro) + + +@unittest.skipUnless(hasattr(os, 'sendfile'), + 'sendfile is not supported') +class SelectorEventLoopUnixSockSendfileTests(test_utils.TestCase): + DATA = b"12345abcde" * 16 * 1024 # 160 KiB + + class MyProto(asyncio.Protocol): + + def __init__(self, loop): + self.started = False + self.closed = False + self.data = bytearray() + self.fut = loop.create_future() + self.transport = None + self._ready = loop.create_future() + + def connection_made(self, transport): + self.started = True + self.transport = transport + self._ready.set_result(None) + + def data_received(self, data): + self.data.extend(data) + + def connection_lost(self, exc): + self.closed = True + self.fut.set_result(None) + + async def wait_closed(self): + await self.fut + + @classmethod + def setUpClass(cls): + with open(os_helper.TESTFN, 'wb') as fp: + fp.write(cls.DATA) + super().setUpClass() + + @classmethod + def tearDownClass(cls): + os_helper.unlink(os_helper.TESTFN) + super().tearDownClass() + + def setUp(self): + self.loop = asyncio.new_event_loop() + self.set_event_loop(self.loop) + self.file = open(os_helper.TESTFN, 'rb') + self.addCleanup(self.file.close) + super().setUp() + + def make_socket(self, cleanup=True): + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + sock.setblocking(False) + sock.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 1024) + sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 1024) + if cleanup: + self.addCleanup(sock.close) + return sock + + def run_loop(self, coro): + return self.loop.run_until_complete(coro) + + def prepare(self): + sock = self.make_socket() + proto = self.MyProto(self.loop) + port = socket_helper.find_unused_port() + srv_sock = self.make_socket(cleanup=False) + srv_sock.bind((socket_helper.HOST, port)) + server = self.run_loop(self.loop.create_server( + lambda: proto, sock=srv_sock)) + self.run_loop(self.loop.sock_connect(sock, (socket_helper.HOST, port))) + self.run_loop(proto._ready) + + def cleanup(): + proto.transport.close() + self.run_loop(proto.wait_closed()) + + server.close() + self.run_loop(server.wait_closed()) + + self.addCleanup(cleanup) + + return sock, proto + + def test_sock_sendfile_not_available(self): + sock, proto = self.prepare() + with mock.patch('asyncio.unix_events.os', spec=[]): + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, + "os[.]sendfile[(][)] is not available"): + self.run_loop(self.loop._sock_sendfile_native(sock, self.file, + 0, None)) + self.assertEqual(self.file.tell(), 0) + + def test_sock_sendfile_not_a_file(self): + sock, proto = self.prepare() + f = object() + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, + "not a regular file"): + self.run_loop(self.loop._sock_sendfile_native(sock, f, + 0, None)) + self.assertEqual(self.file.tell(), 0) + + def test_sock_sendfile_iobuffer(self): + sock, proto = self.prepare() + f = io.BytesIO() + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, + "not a regular file"): + self.run_loop(self.loop._sock_sendfile_native(sock, f, + 0, None)) + self.assertEqual(self.file.tell(), 0) + + def test_sock_sendfile_not_regular_file(self): + sock, proto = self.prepare() + f = mock.Mock() + f.fileno.return_value = -1 + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, + "not a regular file"): + self.run_loop(self.loop._sock_sendfile_native(sock, f, + 0, None)) + self.assertEqual(self.file.tell(), 0) + + def test_sock_sendfile_cancel1(self): + sock, proto = self.prepare() + + fut = self.loop.create_future() + fileno = self.file.fileno() + self.loop._sock_sendfile_native_impl(fut, None, sock, fileno, + 0, None, len(self.DATA), 0) + fut.cancel() + with contextlib.suppress(asyncio.CancelledError): + self.run_loop(fut) + with self.assertRaises(KeyError): + self.loop._selector.get_key(sock) + + def test_sock_sendfile_cancel2(self): + sock, proto = self.prepare() + + fut = self.loop.create_future() + fileno = self.file.fileno() + self.loop._sock_sendfile_native_impl(fut, None, sock, fileno, + 0, None, len(self.DATA), 0) + fut.cancel() + self.loop._sock_sendfile_native_impl(fut, sock.fileno(), sock, fileno, + 0, None, len(self.DATA), 0) + with self.assertRaises(KeyError): + self.loop._selector.get_key(sock) + + def test_sock_sendfile_blocking_error(self): + sock, proto = self.prepare() + + fileno = self.file.fileno() + fut = mock.Mock() + fut.cancelled.return_value = False + with mock.patch('os.sendfile', side_effect=BlockingIOError()): + self.loop._sock_sendfile_native_impl(fut, None, sock, fileno, + 0, None, len(self.DATA), 0) + key = self.loop._selector.get_key(sock) + self.assertIsNotNone(key) + fut.add_done_callback.assert_called_once_with(mock.ANY) + + def test_sock_sendfile_os_error_first_call(self): + sock, proto = self.prepare() + + fileno = self.file.fileno() + fut = self.loop.create_future() + with mock.patch('os.sendfile', side_effect=OSError()): + self.loop._sock_sendfile_native_impl(fut, None, sock, fileno, + 0, None, len(self.DATA), 0) + with self.assertRaises(KeyError): + self.loop._selector.get_key(sock) + exc = fut.exception() + self.assertIsInstance(exc, asyncio.SendfileNotAvailableError) + self.assertEqual(0, self.file.tell()) + + def test_sock_sendfile_os_error_next_call(self): + sock, proto = self.prepare() + + fileno = self.file.fileno() + fut = self.loop.create_future() + err = OSError() + with mock.patch('os.sendfile', side_effect=err): + self.loop._sock_sendfile_native_impl(fut, sock.fileno(), + sock, fileno, + 1000, None, len(self.DATA), + 1000) + with self.assertRaises(KeyError): + self.loop._selector.get_key(sock) + exc = fut.exception() + self.assertIs(exc, err) + self.assertEqual(1000, self.file.tell()) + + def test_sock_sendfile_exception(self): + sock, proto = self.prepare() + + fileno = self.file.fileno() + fut = self.loop.create_future() + err = asyncio.SendfileNotAvailableError() + with mock.patch('os.sendfile', side_effect=err): + self.loop._sock_sendfile_native_impl(fut, sock.fileno(), + sock, fileno, + 1000, None, len(self.DATA), + 1000) + with self.assertRaises(KeyError): + self.loop._selector.get_key(sock) + exc = fut.exception() + self.assertIs(exc, err) + self.assertEqual(1000, self.file.tell()) + + +class UnixReadPipeTransportTests(test_utils.TestCase): + + def setUp(self): + super().setUp() + self.loop = self.new_test_loop() + self.protocol = test_utils.make_test_protocol(asyncio.Protocol) + self.pipe = mock.Mock(spec_set=io.RawIOBase) + self.pipe.fileno.return_value = 5 + + blocking_patcher = mock.patch('os.set_blocking') + blocking_patcher.start() + self.addCleanup(blocking_patcher.stop) + + fstat_patcher = mock.patch('os.fstat') + m_fstat = fstat_patcher.start() + st = mock.Mock() + st.st_mode = stat.S_IFIFO + m_fstat.return_value = st + self.addCleanup(fstat_patcher.stop) + + def read_pipe_transport(self, waiter=None): + transport = unix_events._UnixReadPipeTransport(self.loop, self.pipe, + self.protocol, + waiter=waiter) + self.addCleanup(close_pipe_transport, transport) + return transport + + def test_ctor(self): + waiter = self.loop.create_future() + tr = self.read_pipe_transport(waiter=waiter) + self.loop.run_until_complete(waiter) + + self.protocol.connection_made.assert_called_with(tr) + self.loop.assert_reader(5, tr._read_ready) + self.assertIsNone(waiter.result()) + + @mock.patch('os.read') + def test__read_ready(self, m_read): + tr = self.read_pipe_transport() + m_read.return_value = b'data' + tr._read_ready() + + m_read.assert_called_with(5, tr.max_size) + self.protocol.data_received.assert_called_with(b'data') + + @mock.patch('os.read') + def test__read_ready_eof(self, m_read): + tr = self.read_pipe_transport() + m_read.return_value = b'' + tr._read_ready() + + m_read.assert_called_with(5, tr.max_size) + self.assertFalse(self.loop.readers) + test_utils.run_briefly(self.loop) + self.protocol.eof_received.assert_called_with() + self.protocol.connection_lost.assert_called_with(None) + + @mock.patch('os.read') + def test__read_ready_blocked(self, m_read): + tr = self.read_pipe_transport() + m_read.side_effect = BlockingIOError + tr._read_ready() + + m_read.assert_called_with(5, tr.max_size) + test_utils.run_briefly(self.loop) + self.assertFalse(self.protocol.data_received.called) + + @mock.patch('asyncio.log.logger.error') + @mock.patch('os.read') + def test__read_ready_error(self, m_read, m_logexc): + tr = self.read_pipe_transport() + err = OSError() + m_read.side_effect = err + tr._close = mock.Mock() + tr._read_ready() + + m_read.assert_called_with(5, tr.max_size) + tr._close.assert_called_with(err) + m_logexc.assert_called_with( + test_utils.MockPattern( + 'Fatal read error on pipe transport' + '\nprotocol:.*\ntransport:.*'), + exc_info=(OSError, MOCK_ANY, MOCK_ANY)) + + @mock.patch('os.read') + def test_pause_reading(self, m_read): + tr = self.read_pipe_transport() + m = mock.Mock() + self.loop.add_reader(5, m) + tr.pause_reading() + self.assertFalse(self.loop.readers) + + @mock.patch('os.read') + def test_resume_reading(self, m_read): + tr = self.read_pipe_transport() + tr.pause_reading() + tr.resume_reading() + self.loop.assert_reader(5, tr._read_ready) + + @mock.patch('os.read') + def test_close(self, m_read): + tr = self.read_pipe_transport() + tr._close = mock.Mock() + tr.close() + tr._close.assert_called_with(None) + + @mock.patch('os.read') + def test_close_already_closing(self, m_read): + tr = self.read_pipe_transport() + tr._closing = True + tr._close = mock.Mock() + tr.close() + self.assertFalse(tr._close.called) + + @mock.patch('os.read') + def test__close(self, m_read): + tr = self.read_pipe_transport() + err = object() + tr._close(err) + self.assertTrue(tr.is_closing()) + self.assertFalse(self.loop.readers) + test_utils.run_briefly(self.loop) + self.protocol.connection_lost.assert_called_with(err) + + def test__call_connection_lost(self): + tr = self.read_pipe_transport() + self.assertIsNotNone(tr._protocol) + self.assertIsNotNone(tr._loop) + + err = None + tr._call_connection_lost(err) + self.protocol.connection_lost.assert_called_with(err) + self.pipe.close.assert_called_with() + + self.assertIsNone(tr._protocol) + self.assertIsNone(tr._loop) + + def test__call_connection_lost_with_err(self): + tr = self.read_pipe_transport() + self.assertIsNotNone(tr._protocol) + self.assertIsNotNone(tr._loop) + + err = OSError() + tr._call_connection_lost(err) + self.protocol.connection_lost.assert_called_with(err) + self.pipe.close.assert_called_with() + + self.assertIsNone(tr._protocol) + self.assertIsNone(tr._loop) + + def test_pause_reading_on_closed_pipe(self): + tr = self.read_pipe_transport() + tr.close() + test_utils.run_briefly(self.loop) + self.assertIsNone(tr._loop) + tr.pause_reading() + + def test_pause_reading_on_paused_pipe(self): + tr = self.read_pipe_transport() + tr.pause_reading() + # the second call should do nothing + tr.pause_reading() + + def test_resume_reading_on_closed_pipe(self): + tr = self.read_pipe_transport() + tr.close() + test_utils.run_briefly(self.loop) + self.assertIsNone(tr._loop) + tr.resume_reading() + + def test_resume_reading_on_paused_pipe(self): + tr = self.read_pipe_transport() + # the pipe is not paused + # resuming should do nothing + tr.resume_reading() + + +class UnixWritePipeTransportTests(test_utils.TestCase): + + def setUp(self): + super().setUp() + self.loop = self.new_test_loop() + self.protocol = test_utils.make_test_protocol(asyncio.BaseProtocol) + self.pipe = mock.Mock(spec_set=io.RawIOBase) + self.pipe.fileno.return_value = 5 + + blocking_patcher = mock.patch('os.set_blocking') + blocking_patcher.start() + self.addCleanup(blocking_patcher.stop) + + fstat_patcher = mock.patch('os.fstat') + m_fstat = fstat_patcher.start() + st = mock.Mock() + st.st_mode = stat.S_IFSOCK + m_fstat.return_value = st + self.addCleanup(fstat_patcher.stop) + + def write_pipe_transport(self, waiter=None): + transport = unix_events._UnixWritePipeTransport(self.loop, self.pipe, + self.protocol, + waiter=waiter) + self.addCleanup(close_pipe_transport, transport) + return transport + + def test_ctor(self): + waiter = self.loop.create_future() + tr = self.write_pipe_transport(waiter=waiter) + self.loop.run_until_complete(waiter) + + self.protocol.connection_made.assert_called_with(tr) + self.loop.assert_reader(5, tr._read_ready) + self.assertEqual(None, waiter.result()) + + def test_can_write_eof(self): + tr = self.write_pipe_transport() + self.assertTrue(tr.can_write_eof()) + + @mock.patch('os.write') + def test_write(self, m_write): + tr = self.write_pipe_transport() + m_write.return_value = 4 + tr.write(b'data') + m_write.assert_called_with(5, b'data') + self.assertFalse(self.loop.writers) + self.assertEqual(bytearray(), tr._buffer) + + @mock.patch('os.write') + def test_write_no_data(self, m_write): + tr = self.write_pipe_transport() + tr.write(b'') + self.assertFalse(m_write.called) + self.assertFalse(self.loop.writers) + self.assertEqual(bytearray(b''), tr._buffer) + + @mock.patch('os.write') + def test_write_partial(self, m_write): + tr = self.write_pipe_transport() + m_write.return_value = 2 + tr.write(b'data') + self.loop.assert_writer(5, tr._write_ready) + self.assertEqual(bytearray(b'ta'), tr._buffer) + + @mock.patch('os.write') + def test_write_buffer(self, m_write): + tr = self.write_pipe_transport() + self.loop.add_writer(5, tr._write_ready) + tr._buffer = bytearray(b'previous') + tr.write(b'data') + self.assertFalse(m_write.called) + self.loop.assert_writer(5, tr._write_ready) + self.assertEqual(bytearray(b'previousdata'), tr._buffer) + + @mock.patch('os.write') + def test_write_again(self, m_write): + tr = self.write_pipe_transport() + m_write.side_effect = BlockingIOError() + tr.write(b'data') + m_write.assert_called_with(5, bytearray(b'data')) + self.loop.assert_writer(5, tr._write_ready) + self.assertEqual(bytearray(b'data'), tr._buffer) + + @mock.patch('asyncio.unix_events.logger') + @mock.patch('os.write') + def test_write_err(self, m_write, m_log): + tr = self.write_pipe_transport() + err = OSError() + m_write.side_effect = err + tr._fatal_error = mock.Mock() + tr.write(b'data') + m_write.assert_called_with(5, b'data') + self.assertFalse(self.loop.writers) + self.assertEqual(bytearray(), tr._buffer) + tr._fatal_error.assert_called_with( + err, + 'Fatal write error on pipe transport') + self.assertEqual(1, tr._conn_lost) + + tr.write(b'data') + self.assertEqual(2, tr._conn_lost) + tr.write(b'data') + tr.write(b'data') + tr.write(b'data') + tr.write(b'data') + # This is a bit overspecified. :-( + m_log.warning.assert_called_with( + 'pipe closed by peer or os.write(pipe, data) raised exception.') + tr.close() + + @mock.patch('os.write') + def test_write_close(self, m_write): + tr = self.write_pipe_transport() + tr._read_ready() # pipe was closed by peer + + tr.write(b'data') + self.assertEqual(tr._conn_lost, 1) + tr.write(b'data') + self.assertEqual(tr._conn_lost, 2) + + def test__read_ready(self): + tr = self.write_pipe_transport() + tr._read_ready() + self.assertFalse(self.loop.readers) + self.assertFalse(self.loop.writers) + self.assertTrue(tr.is_closing()) + test_utils.run_briefly(self.loop) + self.protocol.connection_lost.assert_called_with(None) + + @mock.patch('os.write') + def test__write_ready(self, m_write): + tr = self.write_pipe_transport() + self.loop.add_writer(5, tr._write_ready) + tr._buffer = bytearray(b'data') + m_write.return_value = 4 + tr._write_ready() + self.assertFalse(self.loop.writers) + self.assertEqual(bytearray(), tr._buffer) + + @mock.patch('os.write') + def test__write_ready_partial(self, m_write): + tr = self.write_pipe_transport() + self.loop.add_writer(5, tr._write_ready) + tr._buffer = bytearray(b'data') + m_write.return_value = 3 + tr._write_ready() + self.loop.assert_writer(5, tr._write_ready) + self.assertEqual(bytearray(b'a'), tr._buffer) + + @mock.patch('os.write') + def test__write_ready_again(self, m_write): + tr = self.write_pipe_transport() + self.loop.add_writer(5, tr._write_ready) + tr._buffer = bytearray(b'data') + m_write.side_effect = BlockingIOError() + tr._write_ready() + m_write.assert_called_with(5, bytearray(b'data')) + self.loop.assert_writer(5, tr._write_ready) + self.assertEqual(bytearray(b'data'), tr._buffer) + + @mock.patch('os.write') + def test__write_ready_empty(self, m_write): + tr = self.write_pipe_transport() + self.loop.add_writer(5, tr._write_ready) + tr._buffer = bytearray(b'data') + m_write.return_value = 0 + tr._write_ready() + m_write.assert_called_with(5, bytearray(b'data')) + self.loop.assert_writer(5, tr._write_ready) + self.assertEqual(bytearray(b'data'), tr._buffer) + + @mock.patch('asyncio.log.logger.error') + @mock.patch('os.write') + def test__write_ready_err(self, m_write, m_logexc): + tr = self.write_pipe_transport() + self.loop.add_writer(5, tr._write_ready) + tr._buffer = bytearray(b'data') + m_write.side_effect = err = OSError() + tr._write_ready() + self.assertFalse(self.loop.writers) + self.assertFalse(self.loop.readers) + self.assertEqual(bytearray(), tr._buffer) + self.assertTrue(tr.is_closing()) + m_logexc.assert_not_called() + self.assertEqual(1, tr._conn_lost) + test_utils.run_briefly(self.loop) + self.protocol.connection_lost.assert_called_with(err) + + @mock.patch('os.write') + def test__write_ready_closing(self, m_write): + tr = self.write_pipe_transport() + self.loop.add_writer(5, tr._write_ready) + tr._closing = True + tr._buffer = bytearray(b'data') + m_write.return_value = 4 + tr._write_ready() + self.assertFalse(self.loop.writers) + self.assertFalse(self.loop.readers) + self.assertEqual(bytearray(), tr._buffer) + self.protocol.connection_lost.assert_called_with(None) + self.pipe.close.assert_called_with() + + @mock.patch('os.write') + def test_abort(self, m_write): + tr = self.write_pipe_transport() + self.loop.add_writer(5, tr._write_ready) + self.loop.add_reader(5, tr._read_ready) + tr._buffer = [b'da', b'ta'] + tr.abort() + self.assertFalse(m_write.called) + self.assertFalse(self.loop.readers) + self.assertFalse(self.loop.writers) + self.assertEqual([], tr._buffer) + self.assertTrue(tr.is_closing()) + test_utils.run_briefly(self.loop) + self.protocol.connection_lost.assert_called_with(None) + + def test__call_connection_lost(self): + tr = self.write_pipe_transport() + self.assertIsNotNone(tr._protocol) + self.assertIsNotNone(tr._loop) + + err = None + tr._call_connection_lost(err) + self.protocol.connection_lost.assert_called_with(err) + self.pipe.close.assert_called_with() + + self.assertIsNone(tr._protocol) + self.assertIsNone(tr._loop) + + def test__call_connection_lost_with_err(self): + tr = self.write_pipe_transport() + self.assertIsNotNone(tr._protocol) + self.assertIsNotNone(tr._loop) + + err = OSError() + tr._call_connection_lost(err) + self.protocol.connection_lost.assert_called_with(err) + self.pipe.close.assert_called_with() + + self.assertIsNone(tr._protocol) + self.assertIsNone(tr._loop) + + def test_close(self): + tr = self.write_pipe_transport() + tr.write_eof = mock.Mock() + tr.close() + tr.write_eof.assert_called_with() + + # closing the transport twice must not fail + tr.close() + + def test_close_closing(self): + tr = self.write_pipe_transport() + tr.write_eof = mock.Mock() + tr._closing = True + tr.close() + self.assertFalse(tr.write_eof.called) + + def test_write_eof(self): + tr = self.write_pipe_transport() + tr.write_eof() + self.assertTrue(tr.is_closing()) + self.assertFalse(self.loop.readers) + test_utils.run_briefly(self.loop) + self.protocol.connection_lost.assert_called_with(None) + + def test_write_eof_pending(self): + tr = self.write_pipe_transport() + tr._buffer = [b'data'] + tr.write_eof() + self.assertTrue(tr.is_closing()) + self.assertFalse(self.protocol.connection_lost.called) + + +class TestFunctional(unittest.TestCase): + + def setUp(self): + self.loop = asyncio.new_event_loop() + asyncio.set_event_loop(self.loop) + + def tearDown(self): + self.loop.close() + asyncio.set_event_loop(None) + + def test_add_reader_invalid_argument(self): + def assert_raises(): + return self.assertRaisesRegex(ValueError, r'Invalid file object') + + cb = lambda: None + + with assert_raises(): + self.loop.add_reader(object(), cb) + with assert_raises(): + self.loop.add_writer(object(), cb) + + with assert_raises(): + self.loop.remove_reader(object()) + with assert_raises(): + self.loop.remove_writer(object()) + + def test_add_reader_or_writer_transport_fd(self): + def assert_raises(): + return self.assertRaisesRegex( + RuntimeError, + r'File descriptor .* is used by transport') + + async def runner(): + tr, pr = await self.loop.create_connection( + lambda: asyncio.Protocol(), sock=rsock) + + try: + cb = lambda: None + + with assert_raises(): + self.loop.add_reader(rsock, cb) + with assert_raises(): + self.loop.add_reader(rsock.fileno(), cb) + + with assert_raises(): + self.loop.remove_reader(rsock) + with assert_raises(): + self.loop.remove_reader(rsock.fileno()) + + with assert_raises(): + self.loop.add_writer(rsock, cb) + with assert_raises(): + self.loop.add_writer(rsock.fileno(), cb) + + with assert_raises(): + self.loop.remove_writer(rsock) + with assert_raises(): + self.loop.remove_writer(rsock.fileno()) + + finally: + tr.close() + + rsock, wsock = socket.socketpair() + try: + self.loop.run_until_complete(runner()) + finally: + rsock.close() + wsock.close() + + +# TODO: RUSTPYTHON, fork() segfaults due to stale parking_lot global state +@unittest.skip("TODO: RUSTPYTHON") +@support.requires_fork() +class TestFork(unittest.TestCase): + + def test_fork_not_share_current_task(self): + loop = object() + task = object() + asyncio._set_running_loop(loop) + self.addCleanup(asyncio._set_running_loop, None) + asyncio.tasks._enter_task(loop, task) + self.addCleanup(asyncio.tasks._leave_task, loop, task) + self.assertIs(asyncio.current_task(), task) + r, w = os.pipe() + self.addCleanup(os.close, r) + self.addCleanup(os.close, w) + pid = os.fork() + if pid == 0: + # child + try: + asyncio._set_running_loop(loop) + current_task = asyncio.current_task() + if current_task is None: + os.write(w, b'NO TASK') + else: + os.write(w, b'TASK:' + str(id(current_task)).encode()) + except BaseException as e: + os.write(w, b'ERROR:' + ascii(e).encode()) + finally: + asyncio._set_running_loop(None) + os._exit(0) + else: + # parent + result = os.read(r, 100) + self.assertEqual(result, b'NO TASK') + wait_process(pid, exitcode=0) + + def test_fork_not_share_event_loop(self): + # The forked process should not share the event loop with the parent + loop = object() + asyncio._set_running_loop(loop) + self.assertIs(asyncio.get_running_loop(), loop) + self.addCleanup(asyncio._set_running_loop, None) + r, w = os.pipe() + self.addCleanup(os.close, r) + self.addCleanup(os.close, w) + pid = os.fork() + if pid == 0: + # child + try: + loop = asyncio.get_event_loop() + os.write(w, b'LOOP:' + str(id(loop)).encode()) + except RuntimeError: + os.write(w, b'NO LOOP') + except BaseException as e: + os.write(w, b'ERROR:' + ascii(e).encode()) + finally: + os._exit(0) + else: + # parent + result = os.read(r, 100) + self.assertEqual(result, b'NO LOOP') + wait_process(pid, exitcode=0) + + @hashlib_helper.requires_hashdigest('md5') + @support.skip_if_sanitizer("TSAN doesn't support threads after fork", thread=True) + def test_fork_signal_handling(self): + self.addCleanup(multiprocessing_cleanup_tests) + + # Sending signal to the forked process should not affect the parent + # process + ctx = multiprocessing.get_context('fork') + manager = ctx.Manager() + self.addCleanup(manager.shutdown) + child_started = manager.Event() + child_handled = manager.Event() + parent_handled = manager.Event() + + def child_main(): + def on_sigterm(*args): + child_handled.set() + sys.exit() + + signal.signal(signal.SIGTERM, on_sigterm) + child_started.set() + while True: + time.sleep(1) + + async def main(): + loop = asyncio.get_running_loop() + loop.add_signal_handler(signal.SIGTERM, lambda *args: parent_handled.set()) + + process = ctx.Process(target=child_main) + process.start() + child_started.wait() + os.kill(process.pid, signal.SIGTERM) + process.join(timeout=support.SHORT_TIMEOUT) + + async def func(): + await asyncio.sleep(0.1) + return 42 + + # Test parent's loop is still functional + self.assertEqual(await asyncio.create_task(func()), 42) + + asyncio.run(main()) + + child_handled.wait(timeout=support.SHORT_TIMEOUT) + self.assertFalse(parent_handled.is_set()) + self.assertTrue(child_handled.is_set()) + + @hashlib_helper.requires_hashdigest('md5') + @support.skip_if_sanitizer("TSAN doesn't support threads after fork", thread=True) + def test_fork_asyncio_run(self): + self.addCleanup(multiprocessing_cleanup_tests) + + ctx = multiprocessing.get_context('fork') + manager = ctx.Manager() + self.addCleanup(manager.shutdown) + result = manager.Value('i', 0) + + async def child_main(): + await asyncio.sleep(0.1) + result.value = 42 + + process = ctx.Process(target=lambda: asyncio.run(child_main())) + process.start() + process.join() + + self.assertEqual(result.value, 42) + + @hashlib_helper.requires_hashdigest('md5') + @support.skip_if_sanitizer("TSAN doesn't support threads after fork", thread=True) + def test_fork_asyncio_subprocess(self): + self.addCleanup(multiprocessing_cleanup_tests) + + ctx = multiprocessing.get_context('fork') + manager = ctx.Manager() + self.addCleanup(manager.shutdown) + result = manager.Value('i', 1) + + async def child_main(): + proc = await asyncio.create_subprocess_exec(sys.executable, '-c', 'pass') + result.value = await proc.wait() + + process = ctx.Process(target=lambda: asyncio.run(child_main())) + process.start() + process.join() + + self.assertEqual(result.value, 0) + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_waitfor.py b/Lib/test/test_asyncio/test_waitfor.py new file mode 100644 index 00000000000..dedc6bf69d7 --- /dev/null +++ b/Lib/test/test_asyncio/test_waitfor.py @@ -0,0 +1,353 @@ +import asyncio +import unittest +import time +from test import support + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +# The following value can be used as a very small timeout: +# it passes check "timeout > 0", but has almost +# no effect on the test performance +_EPSILON = 0.0001 + + +class SlowTask: + """ Task will run for this defined time, ignoring cancel requests """ + TASK_TIMEOUT = 0.2 + + def __init__(self): + self.exited = False + + async def run(self): + exitat = time.monotonic() + self.TASK_TIMEOUT + + while True: + tosleep = exitat - time.monotonic() + if tosleep <= 0: + break + + try: + await asyncio.sleep(tosleep) + except asyncio.CancelledError: + pass + + self.exited = True + + +class AsyncioWaitForTest(unittest.IsolatedAsyncioTestCase): + + async def test_asyncio_wait_for_cancelled(self): + t = SlowTask() + + waitfortask = asyncio.create_task( + asyncio.wait_for(t.run(), t.TASK_TIMEOUT * 2)) + await asyncio.sleep(0) + waitfortask.cancel() + await asyncio.wait({waitfortask}) + + self.assertTrue(t.exited) + + async def test_asyncio_wait_for_timeout(self): + t = SlowTask() + + try: + await asyncio.wait_for(t.run(), t.TASK_TIMEOUT / 2) + except asyncio.TimeoutError: + pass + + self.assertTrue(t.exited) + + async def test_wait_for_timeout_less_then_0_or_0_future_done(self): + loop = asyncio.get_running_loop() + + fut = loop.create_future() + fut.set_result('done') + + ret = await asyncio.wait_for(fut, 0) + + self.assertEqual(ret, 'done') + self.assertTrue(fut.done()) + + async def test_wait_for_timeout_less_then_0_or_0_coroutine_do_not_started(self): + foo_started = False + + async def foo(): + nonlocal foo_started + foo_started = True + + with self.assertRaises(asyncio.TimeoutError): + await asyncio.wait_for(foo(), 0) + + self.assertEqual(foo_started, False) + + async def test_wait_for_timeout_less_then_0_or_0(self): + loop = asyncio.get_running_loop() + + for timeout in [0, -1]: + with self.subTest(timeout=timeout): + foo_running = None + started = loop.create_future() + + async def foo(): + nonlocal foo_running + foo_running = True + started.set_result(None) + try: + await asyncio.sleep(10) + finally: + foo_running = False + return 'done' + + fut = asyncio.create_task(foo()) + await started + + with self.assertRaises(asyncio.TimeoutError): + await asyncio.wait_for(fut, timeout) + + self.assertTrue(fut.done()) + # it should have been cancelled due to the timeout + self.assertTrue(fut.cancelled()) + self.assertEqual(foo_running, False) + + async def test_wait_for(self): + foo_running = None + + async def foo(): + nonlocal foo_running + foo_running = True + try: + await asyncio.sleep(support.LONG_TIMEOUT) + finally: + foo_running = False + return 'done' + + fut = asyncio.create_task(foo()) + + with self.assertRaises(asyncio.TimeoutError): + await asyncio.wait_for(fut, 0.1) + self.assertTrue(fut.done()) + # it should have been cancelled due to the timeout + self.assertTrue(fut.cancelled()) + self.assertEqual(foo_running, False) + + async def test_wait_for_blocking(self): + async def coro(): + return 'done' + + res = await asyncio.wait_for(coro(), timeout=None) + self.assertEqual(res, 'done') + + async def test_wait_for_race_condition(self): + loop = asyncio.get_running_loop() + + fut = loop.create_future() + task = asyncio.wait_for(fut, timeout=0.2) + loop.call_soon(fut.set_result, "ok") + res = await task + self.assertEqual(res, "ok") + + async def test_wait_for_cancellation_race_condition(self): + async def inner(): + with self.assertRaises(asyncio.CancelledError): + await asyncio.sleep(1) + return 1 + + result = await asyncio.wait_for(inner(), timeout=.01) + self.assertEqual(result, 1) + + async def test_wait_for_waits_for_task_cancellation(self): + task_done = False + + async def inner(): + nonlocal task_done + try: + await asyncio.sleep(10) + except asyncio.CancelledError: + await asyncio.sleep(_EPSILON) + raise + finally: + task_done = True + + inner_task = asyncio.create_task(inner()) + + with self.assertRaises(asyncio.TimeoutError) as cm: + await asyncio.wait_for(inner_task, timeout=_EPSILON) + + self.assertTrue(task_done) + chained = cm.exception.__context__ + self.assertEqual(type(chained), asyncio.CancelledError) + + async def test_wait_for_waits_for_task_cancellation_w_timeout_0(self): + task_done = False + + async def foo(): + async def inner(): + nonlocal task_done + try: + await asyncio.sleep(10) + except asyncio.CancelledError: + await asyncio.sleep(_EPSILON) + raise + finally: + task_done = True + + inner_task = asyncio.create_task(inner()) + await asyncio.sleep(_EPSILON) + await asyncio.wait_for(inner_task, timeout=0) + + with self.assertRaises(asyncio.TimeoutError) as cm: + await foo() + + self.assertTrue(task_done) + chained = cm.exception.__context__ + self.assertEqual(type(chained), asyncio.CancelledError) + + async def test_wait_for_reraises_exception_during_cancellation(self): + class FooException(Exception): + pass + + async def foo(): + async def inner(): + try: + await asyncio.sleep(0.2) + finally: + raise FooException + + inner_task = asyncio.create_task(inner()) + + await asyncio.wait_for(inner_task, timeout=_EPSILON) + + with self.assertRaises(FooException): + await foo() + + async def _test_cancel_wait_for(self, timeout): + loop = asyncio.get_running_loop() + + async def blocking_coroutine(): + fut = loop.create_future() + # Block: fut result is never set + await fut + + task = asyncio.create_task(blocking_coroutine()) + + wait = asyncio.create_task(asyncio.wait_for(task, timeout)) + loop.call_soon(wait.cancel) + + with self.assertRaises(asyncio.CancelledError): + await wait + + # Python issue #23219: cancelling the wait must also cancel the task + self.assertTrue(task.cancelled()) + + async def test_cancel_blocking_wait_for(self): + await self._test_cancel_wait_for(None) + + async def test_cancel_wait_for(self): + await self._test_cancel_wait_for(60.0) + + async def test_wait_for_cancel_suppressed(self): + # GH-86296: Suppressing CancelledError is discouraged + # but if a task suppresses CancelledError and returns a value, + # `wait_for` should return the value instead of raising CancelledError. + # This is the same behavior as `asyncio.timeout`. + + async def return_42(): + try: + await asyncio.sleep(10) + except asyncio.CancelledError: + return 42 + + res = await asyncio.wait_for(return_42(), timeout=0.1) + self.assertEqual(res, 42) + + + async def test_wait_for_issue86296(self): + # GH-86296: The task should get cancelled and not run to completion. + # inner completes in one cycle of the event loop so it + # completes before the task is cancelled. + + async def inner(): + return 'done' + + inner_task = asyncio.create_task(inner()) + reached_end = False + + async def wait_for_coro(): + await asyncio.wait_for(inner_task, timeout=100) + await asyncio.sleep(1) + nonlocal reached_end + reached_end = True + + task = asyncio.create_task(wait_for_coro()) + self.assertFalse(task.done()) + # Run the task + await asyncio.sleep(0) + task.cancel() + with self.assertRaises(asyncio.CancelledError): + await task + self.assertTrue(inner_task.done()) + self.assertEqual(await inner_task, 'done') + self.assertFalse(reached_end) + + +class WaitForShieldTests(unittest.IsolatedAsyncioTestCase): + + async def test_zero_timeout(self): + # `asyncio.shield` creates a new task which wraps the passed in + # awaitable and shields it from cancellation so with timeout=0 + # the task returned by `asyncio.shield` aka shielded_task gets + # cancelled immediately and the task wrapped by it is scheduled + # to run. + + async def coro(): + await asyncio.sleep(0.01) + return 'done' + + task = asyncio.create_task(coro()) + with self.assertRaises(asyncio.TimeoutError): + shielded_task = asyncio.shield(task) + await asyncio.wait_for(shielded_task, timeout=0) + + # Task is running in background + self.assertFalse(task.done()) + self.assertFalse(task.cancelled()) + self.assertTrue(shielded_task.cancelled()) + + # Wait for the task to complete + await asyncio.sleep(0.1) + self.assertTrue(task.done()) + + + async def test_none_timeout(self): + # With timeout=None the timeout is disabled so it + # runs till completion. + async def coro(): + await asyncio.sleep(0.1) + return 'done' + + task = asyncio.create_task(coro()) + await asyncio.wait_for(asyncio.shield(task), timeout=None) + + self.assertTrue(task.done()) + self.assertEqual(await task, "done") + + async def test_shielded_timeout(self): + # shield prevents the task from being cancelled. + async def coro(): + await asyncio.sleep(0.1) + return 'done' + + task = asyncio.create_task(coro()) + with self.assertRaises(asyncio.TimeoutError): + await asyncio.wait_for(asyncio.shield(task), timeout=0.01) + + self.assertFalse(task.done()) + self.assertFalse(task.cancelled()) + self.assertEqual(await task, "done") + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_windows_events.py b/Lib/test/test_asyncio/test_windows_events.py new file mode 100644 index 00000000000..0af3368627a --- /dev/null +++ b/Lib/test/test_asyncio/test_windows_events.py @@ -0,0 +1,363 @@ +import os +import signal +import socket +import sys +import time +import threading +import unittest +from unittest import mock + +if sys.platform != 'win32': + raise unittest.SkipTest('Windows only') + +import _overlapped +import _winapi + +import asyncio +from asyncio import windows_events +from test.test_asyncio import utils as test_utils + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +class UpperProto(asyncio.Protocol): + def __init__(self): + self.buf = [] + + def connection_made(self, trans): + self.trans = trans + + def data_received(self, data): + self.buf.append(data) + if b'\n' in data: + self.trans.write(b''.join(self.buf).upper()) + self.trans.close() + + +class WindowsEventsTestCase(test_utils.TestCase): + def _unraisablehook(self, unraisable): + # Storing unraisable.object can resurrect an object which is being + # finalized. Storing unraisable.exc_value creates a reference cycle. + self._unraisable = unraisable + print(unraisable) + + def setUp(self): + self._prev_unraisablehook = sys.unraisablehook + self._unraisable = None + sys.unraisablehook = self._unraisablehook + + def tearDown(self): + sys.unraisablehook = self._prev_unraisablehook + self.assertIsNone(self._unraisable) + +class ProactorLoopCtrlC(WindowsEventsTestCase): + + def test_ctrl_c(self): + + def SIGINT_after_delay(): + time.sleep(0.1) + signal.raise_signal(signal.SIGINT) + + thread = threading.Thread(target=SIGINT_after_delay) + loop = asyncio.new_event_loop() + try: + # only start the loop once the event loop is running + loop.call_soon(thread.start) + loop.run_forever() + self.fail("should not fall through 'run_forever'") + except KeyboardInterrupt: + pass + finally: + self.close_loop(loop) + thread.join() + + +class ProactorMultithreading(WindowsEventsTestCase): + def test_run_from_nonmain_thread(self): + finished = False + + async def coro(): + await asyncio.sleep(0) + + def func(): + nonlocal finished + loop = asyncio.new_event_loop() + loop.run_until_complete(coro()) + # close() must not call signal.set_wakeup_fd() + loop.close() + finished = True + + thread = threading.Thread(target=func) + thread.start() + thread.join() + self.assertTrue(finished) + + +class ProactorTests(WindowsEventsTestCase): + + def setUp(self): + super().setUp() + self.loop = asyncio.ProactorEventLoop() + self.set_event_loop(self.loop) + + def test_close(self): + a, b = socket.socketpair() + trans = self.loop._make_socket_transport(a, asyncio.Protocol()) + f = asyncio.ensure_future(self.loop.sock_recv(b, 100), loop=self.loop) + trans.close() + self.loop.run_until_complete(f) + self.assertEqual(f.result(), b'') + b.close() + + def test_double_bind(self): + ADDRESS = r'\\.\pipe\test_double_bind-%s' % os.getpid() + server1 = windows_events.PipeServer(ADDRESS) + with self.assertRaises(PermissionError): + windows_events.PipeServer(ADDRESS) + server1.close() + + def test_pipe(self): + res = self.loop.run_until_complete(self._test_pipe()) + self.assertEqual(res, 'done') + + async def _test_pipe(self): + ADDRESS = r'\\.\pipe\_test_pipe-%s' % os.getpid() + + with self.assertRaises(FileNotFoundError): + await self.loop.create_pipe_connection( + asyncio.Protocol, ADDRESS) + + [server] = await self.loop.start_serving_pipe( + UpperProto, ADDRESS) + self.assertIsInstance(server, windows_events.PipeServer) + + clients = [] + for i in range(5): + stream_reader = asyncio.StreamReader(loop=self.loop) + protocol = asyncio.StreamReaderProtocol(stream_reader, + loop=self.loop) + trans, proto = await self.loop.create_pipe_connection( + lambda: protocol, ADDRESS) + self.assertIsInstance(trans, asyncio.Transport) + self.assertEqual(protocol, proto) + clients.append((stream_reader, trans)) + + for i, (r, w) in enumerate(clients): + w.write('lower-{}\n'.format(i).encode()) + + for i, (r, w) in enumerate(clients): + response = await r.readline() + self.assertEqual(response, 'LOWER-{}\n'.format(i).encode()) + w.close() + + server.close() + + with self.assertRaises(FileNotFoundError): + await self.loop.create_pipe_connection( + asyncio.Protocol, ADDRESS) + + return 'done' + + def test_connect_pipe_cancel(self): + exc = OSError() + exc.winerror = _overlapped.ERROR_PIPE_BUSY + with mock.patch.object(_overlapped, 'ConnectPipe', + side_effect=exc) as connect: + coro = self.loop._proactor.connect_pipe('pipe_address') + task = self.loop.create_task(coro) + + # check that it's possible to cancel connect_pipe() + task.cancel() + with self.assertRaises(asyncio.CancelledError): + self.loop.run_until_complete(task) + + def test_wait_for_handle(self): + event = _overlapped.CreateEvent(None, True, False, None) + self.addCleanup(_winapi.CloseHandle, event) + + # Wait for unset event with 0.5s timeout; + # result should be False at timeout + timeout = 0.5 + fut = self.loop._proactor.wait_for_handle(event, timeout) + start = self.loop.time() + done = self.loop.run_until_complete(fut) + elapsed = self.loop.time() - start + + self.assertEqual(done, False) + self.assertFalse(fut.result()) + self.assertGreaterEqual(elapsed, timeout - test_utils.CLOCK_RES) + + _overlapped.SetEvent(event) + + # Wait for set event; + # result should be True immediately + fut = self.loop._proactor.wait_for_handle(event, 10) + done = self.loop.run_until_complete(fut) + + self.assertEqual(done, True) + self.assertTrue(fut.result()) + + # asyncio issue #195: cancelling a done _WaitHandleFuture + # must not crash + fut.cancel() + + def test_wait_for_handle_cancel(self): + event = _overlapped.CreateEvent(None, True, False, None) + self.addCleanup(_winapi.CloseHandle, event) + + # Wait for unset event with a cancelled future; + # CancelledError should be raised immediately + fut = self.loop._proactor.wait_for_handle(event, 10) + fut.cancel() + with self.assertRaises(asyncio.CancelledError): + self.loop.run_until_complete(fut) + + # asyncio issue #195: cancelling a _WaitHandleFuture twice + # must not crash + fut = self.loop._proactor.wait_for_handle(event) + fut.cancel() + fut.cancel() + + def test_read_self_pipe_restart(self): + # Regression test for https://bugs.python.org/issue39010 + # Previously, restarting a proactor event loop in certain states + # would lead to spurious ConnectionResetErrors being logged. + self.loop.call_exception_handler = mock.Mock() + # Start an operation in another thread so that the self-pipe is used. + # This is theoretically timing-dependent (the task in the executor + # must complete before our start/stop cycles), but in practice it + # seems to work every time. + f = self.loop.run_in_executor(None, lambda: None) + self.loop.stop() + self.loop.run_forever() + self.loop.stop() + self.loop.run_forever() + + # Shut everything down cleanly. This is an important part of the + # test - in issue 39010, the error occurred during loop.close(), + # so we want to close the loop during the test instead of leaving + # it for tearDown. + # + # First wait for f to complete to avoid a "future's result was never + # retrieved" error. + self.loop.run_until_complete(f) + # Now shut down the loop itself (self.close_loop also shuts down the + # loop's default executor). + self.close_loop(self.loop) + self.assertFalse(self.loop.call_exception_handler.called) + + def test_address_argument_type_error(self): + # Regression test for https://github.com/python/cpython/issues/98793 + proactor = self.loop._proactor + sock = socket.socket(type=socket.SOCK_DGRAM) + bad_address = None + with self.assertRaises(TypeError): + proactor.connect(sock, bad_address) + with self.assertRaises(TypeError): + proactor.sendto(sock, b'abc', addr=bad_address) + sock.close() + + def test_client_pipe_stat(self): + res = self.loop.run_until_complete(self._test_client_pipe_stat()) + self.assertEqual(res, 'done') + + async def _test_client_pipe_stat(self): + # Regression test for https://github.com/python/cpython/issues/100573 + ADDRESS = r'\\.\pipe\test_client_pipe_stat-%s' % os.getpid() + + async def probe(): + # See https://github.com/python/cpython/pull/100959#discussion_r1068533658 + h = _overlapped.ConnectPipe(ADDRESS) + try: + _winapi.CloseHandle(_overlapped.ConnectPipe(ADDRESS)) + except OSError as e: + if e.winerror != _overlapped.ERROR_PIPE_BUSY: + raise + finally: + _winapi.CloseHandle(h) + + with self.assertRaises(FileNotFoundError): + await probe() + + [server] = await self.loop.start_serving_pipe(asyncio.Protocol, ADDRESS) + self.assertIsInstance(server, windows_events.PipeServer) + + errors = [] + self.loop.set_exception_handler(lambda _, data: errors.append(data)) + + for i in range(5): + await self.loop.create_task(probe()) + + self.assertEqual(len(errors), 0, errors) + + server.close() + + with self.assertRaises(FileNotFoundError): + await probe() + + return "done" + + def test_loop_restart(self): + # We're fishing for the "RuntimeError: <_overlapped.Overlapped object at XXX> + # still has pending operation at deallocation, the process may crash" error + stop = threading.Event() + def threadMain(): + while not stop.is_set(): + self.loop.call_soon_threadsafe(lambda: None) + time.sleep(0.01) + thr = threading.Thread(target=threadMain) + + # In 10 60-second runs of this test prior to the fix: + # time in seconds until failure: (none), 15.0, 6.4, (none), 7.6, 8.3, 1.7, 22.2, 23.5, 8.3 + # 10 seconds had a 50% failure rate but longer would be more costly + end_time = time.time() + 10 # Run for 10 seconds + self.loop.call_soon(thr.start) + while not self._unraisable: # Stop if we got an unraisable exc + self.loop.stop() + self.loop.run_forever() + if time.time() >= end_time: + break + + stop.set() + thr.join() + + +class WinPolicyTests(WindowsEventsTestCase): + + def test_selector_win_policy(self): + async def main(): + self.assertIsInstance(asyncio.get_running_loop(), asyncio.SelectorEventLoop) + + old_policy = asyncio.events._get_event_loop_policy() + try: + with self.assertWarnsRegex( + DeprecationWarning, + "'asyncio.WindowsSelectorEventLoopPolicy' is deprecated", + ): + asyncio.events._set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) + asyncio.run(main()) + finally: + asyncio.events._set_event_loop_policy(old_policy) + + def test_proactor_win_policy(self): + async def main(): + self.assertIsInstance( + asyncio.get_running_loop(), + asyncio.ProactorEventLoop) + + old_policy = asyncio.events._get_event_loop_policy() + try: + with self.assertWarnsRegex( + DeprecationWarning, + "'asyncio.WindowsProactorEventLoopPolicy' is deprecated", + ): + asyncio.events._set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy()) + asyncio.run(main()) + finally: + asyncio.events._set_event_loop_policy(old_policy) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/test_windows_utils.py b/Lib/test/test_asyncio/test_windows_utils.py new file mode 100644 index 00000000000..97f078ff911 --- /dev/null +++ b/Lib/test/test_asyncio/test_windows_utils.py @@ -0,0 +1,133 @@ +"""Tests for window_utils""" + +import sys +import unittest +import warnings + +if sys.platform != 'win32': + raise unittest.SkipTest('Windows only') + +import _overlapped +import _winapi + +import asyncio +from asyncio import windows_utils +from test import support + + +def tearDownModule(): + asyncio.events._set_event_loop_policy(None) + + +class PipeTests(unittest.TestCase): + + def test_pipe_overlapped(self): + h1, h2 = windows_utils.pipe(overlapped=(True, True)) + try: + ov1 = _overlapped.Overlapped() + self.assertFalse(ov1.pending) + self.assertEqual(ov1.error, 0) + + ov1.ReadFile(h1, 100) + self.assertTrue(ov1.pending) + self.assertEqual(ov1.error, _winapi.ERROR_IO_PENDING) + ERROR_IO_INCOMPLETE = 996 + try: + ov1.getresult() + except OSError as e: + self.assertEqual(e.winerror, ERROR_IO_INCOMPLETE) + else: + raise RuntimeError('expected ERROR_IO_INCOMPLETE') + + ov2 = _overlapped.Overlapped() + self.assertFalse(ov2.pending) + self.assertEqual(ov2.error, 0) + + ov2.WriteFile(h2, b"hello") + self.assertIn(ov2.error, {0, _winapi.ERROR_IO_PENDING}) + + res = _winapi.WaitForMultipleObjects([ov2.event], False, 100) + self.assertEqual(res, _winapi.WAIT_OBJECT_0) + + self.assertFalse(ov1.pending) + self.assertEqual(ov1.error, ERROR_IO_INCOMPLETE) + self.assertFalse(ov2.pending) + self.assertIn(ov2.error, {0, _winapi.ERROR_IO_PENDING}) + self.assertEqual(ov1.getresult(), b"hello") + finally: + _winapi.CloseHandle(h1) + _winapi.CloseHandle(h2) + + def test_pipe_handle(self): + h, _ = windows_utils.pipe(overlapped=(True, True)) + _winapi.CloseHandle(_) + p = windows_utils.PipeHandle(h) + self.assertEqual(p.fileno(), h) + self.assertEqual(p.handle, h) + + # check garbage collection of p closes handle + with warnings.catch_warnings(): + warnings.filterwarnings("ignore", "", ResourceWarning) + del p + support.gc_collect() + try: + _winapi.CloseHandle(h) + except OSError as e: + self.assertEqual(e.winerror, 6) # ERROR_INVALID_HANDLE + else: + raise RuntimeError('expected ERROR_INVALID_HANDLE') + + +class PopenTests(unittest.TestCase): + + def test_popen(self): + command = r"""if 1: + import sys + s = sys.stdin.readline() + sys.stdout.write(s.upper()) + sys.stderr.write('stderr') + """ + msg = b"blah\n" + + p = windows_utils.Popen([sys.executable, '-c', command], + stdin=windows_utils.PIPE, + stdout=windows_utils.PIPE, + stderr=windows_utils.PIPE) + + for f in [p.stdin, p.stdout, p.stderr]: + self.assertIsInstance(f, windows_utils.PipeHandle) + + ovin = _overlapped.Overlapped() + ovout = _overlapped.Overlapped() + overr = _overlapped.Overlapped() + + ovin.WriteFile(p.stdin.handle, msg) + ovout.ReadFile(p.stdout.handle, 100) + overr.ReadFile(p.stderr.handle, 100) + + events = [ovin.event, ovout.event, overr.event] + # Super-long timeout for slow buildbots. + res = _winapi.WaitForMultipleObjects(events, True, + int(support.SHORT_TIMEOUT * 1000)) + self.assertEqual(res, _winapi.WAIT_OBJECT_0) + self.assertFalse(ovout.pending) + self.assertFalse(overr.pending) + self.assertFalse(ovin.pending) + + self.assertEqual(ovin.getresult(), len(msg)) + out = ovout.getresult().rstrip() + err = overr.getresult().rstrip() + + self.assertGreater(len(out), 0) + self.assertGreater(len(err), 0) + # allow for partial reads... + self.assertStartsWith(msg.upper().rstrip(), out) + self.assertStartsWith(b"stderr", err) + + # The context manager calls wait() and closes resources + with p: + pass + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_asyncio/utils.py b/Lib/test/test_asyncio/utils.py new file mode 100644 index 00000000000..a480e16e81b --- /dev/null +++ b/Lib/test/test_asyncio/utils.py @@ -0,0 +1,609 @@ +"""Utilities shared by tests.""" + +import asyncio +import collections +import contextlib +import io +import logging +import os +import re +import selectors +import socket +import socketserver +import sys +import threading +import unittest +import weakref +from ast import literal_eval +from unittest import mock + +from http.server import HTTPServer +from wsgiref.simple_server import WSGIRequestHandler, WSGIServer + +try: + import ssl +except ImportError: # pragma: no cover + ssl = None + +from asyncio import base_events +from asyncio import events +from asyncio import format_helpers +from asyncio import tasks +from asyncio.log import logger +from test import support +from test.support import socket_helper +from test.support import threading_helper + + +# Use the maximum known clock resolution (gh-75191, gh-110088): Windows +# GetTickCount64() has a resolution of 15.6 ms. Use 50 ms to tolerate rounding +# issues. +CLOCK_RES = 0.050 + + +def data_file(*filename): + fullname = os.path.join(support.TEST_HOME_DIR, *filename) + if os.path.isfile(fullname): + return fullname + fullname = os.path.join(os.path.dirname(__file__), '..', *filename) + if os.path.isfile(fullname): + return fullname + raise FileNotFoundError(os.path.join(filename)) + + +ONLYCERT = data_file('certdata', 'ssl_cert.pem') +ONLYKEY = data_file('certdata', 'ssl_key.pem') +SIGNED_CERTFILE = data_file('certdata', 'keycert3.pem') +SIGNING_CA = data_file('certdata', 'pycacert.pem') +with open(data_file('certdata', 'keycert3.pem.reference')) as file: + PEERCERT = literal_eval(file.read()) + +def simple_server_sslcontext(): + server_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) + server_context.load_cert_chain(ONLYCERT, ONLYKEY) + server_context.check_hostname = False + server_context.verify_mode = ssl.CERT_NONE + return server_context + + +def simple_client_sslcontext(*, disable_verify=True): + client_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + client_context.check_hostname = False + if disable_verify: + client_context.verify_mode = ssl.CERT_NONE + return client_context + + +def dummy_ssl_context(): + if ssl is None: + return None + else: + return simple_client_sslcontext(disable_verify=True) + + +def run_briefly(loop): + async def once(): + pass + gen = once() + t = loop.create_task(gen) + # Don't log a warning if the task is not done after run_until_complete(). + # It occurs if the loop is stopped or if a task raises a BaseException. + t._log_destroy_pending = False + try: + loop.run_until_complete(t) + finally: + gen.close() + + +def run_until(loop, pred, timeout=support.SHORT_TIMEOUT): + delay = 0.001 + for _ in support.busy_retry(timeout, error=False): + if pred(): + break + loop.run_until_complete(tasks.sleep(delay)) + delay = max(delay * 2, 1.0) + else: + raise TimeoutError() + + +def run_once(loop): + """Legacy API to run once through the event loop. + + This is the recommended pattern for test code. It will poll the + selector once and run all callbacks scheduled in response to I/O + events. + """ + loop.call_soon(loop.stop) + loop.run_forever() + + +class SilentWSGIRequestHandler(WSGIRequestHandler): + + def get_stderr(self): + return io.StringIO() + + def log_message(self, format, *args): + pass + + +class SilentWSGIServer(WSGIServer): + + request_timeout = support.LOOPBACK_TIMEOUT + + def get_request(self): + request, client_addr = super().get_request() + request.settimeout(self.request_timeout) + return request, client_addr + + def handle_error(self, request, client_address): + pass + + +class SSLWSGIServerMixin: + + def finish_request(self, request, client_address): + # The relative location of our test directory (which + # contains the ssl key and certificate files) differs + # between the stdlib and stand-alone asyncio. + # Prefer our own if we can find it. + context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) + context.load_cert_chain(ONLYCERT, ONLYKEY) + + ssock = context.wrap_socket(request, server_side=True) + try: + self.RequestHandlerClass(ssock, client_address, self) + ssock.close() + except OSError: + # maybe socket has been closed by peer + pass + + +class SSLWSGIServer(SSLWSGIServerMixin, SilentWSGIServer): + pass + + +def _run_test_server(*, address, use_ssl=False, server_cls, server_ssl_cls): + + def loop(environ): + size = int(environ['CONTENT_LENGTH']) + while size: + data = environ['wsgi.input'].read(min(size, 0x10000)) + yield data + size -= len(data) + + def app(environ, start_response): + status = '200 OK' + headers = [('Content-type', 'text/plain')] + start_response(status, headers) + if environ['PATH_INFO'] == '/loop': + return loop(environ) + else: + return [b'Test message'] + + # Run the test WSGI server in a separate thread in order not to + # interfere with event handling in the main thread + server_class = server_ssl_cls if use_ssl else server_cls + httpd = server_class(address, SilentWSGIRequestHandler) + httpd.set_app(app) + httpd.address = httpd.server_address + server_thread = threading.Thread( + target=lambda: httpd.serve_forever(poll_interval=0.05)) + server_thread.start() + try: + yield httpd + finally: + httpd.shutdown() + httpd.server_close() + server_thread.join() + + +if hasattr(socket, 'AF_UNIX'): + + class UnixHTTPServer(socketserver.UnixStreamServer, HTTPServer): + + def server_bind(self): + socketserver.UnixStreamServer.server_bind(self) + self.server_name = '127.0.0.1' + self.server_port = 80 + + + class UnixWSGIServer(UnixHTTPServer, WSGIServer): + + request_timeout = support.LOOPBACK_TIMEOUT + + def server_bind(self): + UnixHTTPServer.server_bind(self) + self.setup_environ() + + def get_request(self): + request, client_addr = super().get_request() + request.settimeout(self.request_timeout) + # Code in the stdlib expects that get_request + # will return a socket and a tuple (host, port). + # However, this isn't true for UNIX sockets, + # as the second return value will be a path; + # hence we return some fake data sufficient + # to get the tests going + return request, ('127.0.0.1', '') + + + class SilentUnixWSGIServer(UnixWSGIServer): + + def handle_error(self, request, client_address): + pass + + + class UnixSSLWSGIServer(SSLWSGIServerMixin, SilentUnixWSGIServer): + pass + + + def gen_unix_socket_path(): + return socket_helper.create_unix_domain_name() + + + @contextlib.contextmanager + def unix_socket_path(): + path = gen_unix_socket_path() + try: + yield path + finally: + try: + os.unlink(path) + except OSError: + pass + + + @contextlib.contextmanager + def run_test_unix_server(*, use_ssl=False): + with unix_socket_path() as path: + yield from _run_test_server(address=path, use_ssl=use_ssl, + server_cls=SilentUnixWSGIServer, + server_ssl_cls=UnixSSLWSGIServer) + + +@contextlib.contextmanager +def run_test_server(*, host='127.0.0.1', port=0, use_ssl=False): + yield from _run_test_server(address=(host, port), use_ssl=use_ssl, + server_cls=SilentWSGIServer, + server_ssl_cls=SSLWSGIServer) + + +def echo_datagrams(sock): + while True: + data, addr = sock.recvfrom(4096) + if data == b'STOP': + sock.close() + break + else: + sock.sendto(data, addr) + + +@contextlib.contextmanager +def run_udp_echo_server(*, host='127.0.0.1', port=0): + addr_info = socket.getaddrinfo(host, port, type=socket.SOCK_DGRAM) + family, type, proto, _, sockaddr = addr_info[0] + sock = socket.socket(family, type, proto) + sock.bind((host, port)) + sockname = sock.getsockname() + thread = threading.Thread(target=lambda: echo_datagrams(sock)) + thread.start() + try: + yield sockname + finally: + # gh-122187: use a separate socket to send the stop message to avoid + # TSan reported race on the same socket. + sock2 = socket.socket(family, type, proto) + sock2.sendto(b'STOP', sockname) + sock2.close() + thread.join() + + +def make_test_protocol(base): + dct = {} + for name in dir(base): + if name.startswith('__') and name.endswith('__'): + # skip magic names + continue + dct[name] = MockCallback(return_value=None) + return type('TestProtocol', (base,) + base.__bases__, dct)() + + +class TestSelector(selectors.BaseSelector): + + def __init__(self): + self.keys = {} + + def register(self, fileobj, events, data=None): + key = selectors.SelectorKey(fileobj, 0, events, data) + self.keys[fileobj] = key + return key + + def unregister(self, fileobj): + return self.keys.pop(fileobj) + + def select(self, timeout): + return [] + + def get_map(self): + return self.keys + + +class TestLoop(base_events.BaseEventLoop): + """Loop for unittests. + + It manages self time directly. + If something scheduled to be executed later then + on next loop iteration after all ready handlers done + generator passed to __init__ is calling. + + Generator should be like this: + + def gen(): + ... + when = yield ... + ... = yield time_advance + + Value returned by yield is absolute time of next scheduled handler. + Value passed to yield is time advance to move loop's time forward. + """ + + def __init__(self, gen=None): + super().__init__() + + if gen is None: + def gen(): + yield + self._check_on_close = False + else: + self._check_on_close = True + + self._gen = gen() + next(self._gen) + self._time = 0 + self._clock_resolution = 1e-9 + self._timers = [] + self._selector = TestSelector() + + self.readers = {} + self.writers = {} + self.reset_counters() + + self._transports = weakref.WeakValueDictionary() + + def time(self): + return self._time + + def advance_time(self, advance): + """Move test time forward.""" + if advance: + self._time += advance + + def close(self): + super().close() + if self._check_on_close: + try: + self._gen.send(0) + except StopIteration: + pass + else: # pragma: no cover + raise AssertionError("Time generator is not finished") + + def _add_reader(self, fd, callback, *args): + self.readers[fd] = events.Handle(callback, args, self, None) + + def _remove_reader(self, fd): + self.remove_reader_count[fd] += 1 + if fd in self.readers: + del self.readers[fd] + return True + else: + return False + + def assert_reader(self, fd, callback, *args): + if fd not in self.readers: + raise AssertionError(f'fd {fd} is not registered') + handle = self.readers[fd] + if handle._callback != callback: + raise AssertionError( + f'unexpected callback: {handle._callback} != {callback}') + if handle._args != args: + raise AssertionError( + f'unexpected callback args: {handle._args} != {args}') + + def assert_no_reader(self, fd): + if fd in self.readers: + raise AssertionError(f'fd {fd} is registered') + + def _add_writer(self, fd, callback, *args): + self.writers[fd] = events.Handle(callback, args, self, None) + + def _remove_writer(self, fd): + self.remove_writer_count[fd] += 1 + if fd in self.writers: + del self.writers[fd] + return True + else: + return False + + def assert_writer(self, fd, callback, *args): + if fd not in self.writers: + raise AssertionError(f'fd {fd} is not registered') + handle = self.writers[fd] + if handle._callback != callback: + raise AssertionError(f'{handle._callback!r} != {callback!r}') + if handle._args != args: + raise AssertionError(f'{handle._args!r} != {args!r}') + + def _ensure_fd_no_transport(self, fd): + if not isinstance(fd, int): + try: + fd = int(fd.fileno()) + except (AttributeError, TypeError, ValueError): + # This code matches selectors._fileobj_to_fd function. + raise ValueError("Invalid file object: " + "{!r}".format(fd)) from None + try: + transport = self._transports[fd] + except KeyError: + pass + else: + raise RuntimeError( + 'File descriptor {!r} is used by transport {!r}'.format( + fd, transport)) + + def add_reader(self, fd, callback, *args): + """Add a reader callback.""" + self._ensure_fd_no_transport(fd) + return self._add_reader(fd, callback, *args) + + def remove_reader(self, fd): + """Remove a reader callback.""" + self._ensure_fd_no_transport(fd) + return self._remove_reader(fd) + + def add_writer(self, fd, callback, *args): + """Add a writer callback..""" + self._ensure_fd_no_transport(fd) + return self._add_writer(fd, callback, *args) + + def remove_writer(self, fd): + """Remove a writer callback.""" + self._ensure_fd_no_transport(fd) + return self._remove_writer(fd) + + def reset_counters(self): + self.remove_reader_count = collections.defaultdict(int) + self.remove_writer_count = collections.defaultdict(int) + + def _run_once(self): + super()._run_once() + for when in self._timers: + advance = self._gen.send(when) + self.advance_time(advance) + self._timers = [] + + def call_at(self, when, callback, *args, context=None): + self._timers.append(when) + return super().call_at(when, callback, *args, context=context) + + def _process_events(self, event_list): + return + + def _write_to_self(self): + pass + + +def MockCallback(**kwargs): + return mock.Mock(spec=['__call__'], **kwargs) + + +class MockPattern(str): + """A regex based str with a fuzzy __eq__. + + Use this helper with 'mock.assert_called_with', or anywhere + where a regex comparison between strings is needed. + + For instance: + mock_call.assert_called_with(MockPattern('spam.*ham')) + """ + def __eq__(self, other): + return bool(re.search(str(self), other, re.S)) + + +class MockInstanceOf: + def __init__(self, type): + self._type = type + + def __eq__(self, other): + return isinstance(other, self._type) + + +def get_function_source(func): + source = format_helpers._get_function_source(func) + if source is None: + raise ValueError("unable to get the source of %r" % (func,)) + return source + + +class TestCase(unittest.TestCase): + @staticmethod + def close_loop(loop): + if loop._default_executor is not None: + if not loop.is_closed(): + loop.run_until_complete(loop.shutdown_default_executor()) + else: + loop._default_executor.shutdown(wait=True) + loop.close() + + def set_event_loop(self, loop, *, cleanup=True): + if loop is None: + raise AssertionError('loop is None') + # ensure that the event loop is passed explicitly in asyncio + events.set_event_loop(None) + if cleanup: + self.addCleanup(self.close_loop, loop) + + def new_test_loop(self, gen=None): + loop = TestLoop(gen) + self.set_event_loop(loop) + return loop + + def setUp(self): + self._thread_cleanup = threading_helper.threading_setup() + + def tearDown(self): + events.set_event_loop(None) + + # Detect CPython bug #23353: ensure that yield/yield-from is not used + # in an except block of a generator + self.assertIsNone(sys.exception()) + + self.doCleanups() + threading_helper.threading_cleanup(*self._thread_cleanup) + support.reap_children() + + +@contextlib.contextmanager +def disable_logger(): + """Context manager to disable asyncio logger. + + For example, it can be used to ignore warnings in debug mode. + """ + old_level = logger.level + try: + logger.setLevel(logging.CRITICAL+1) + yield + finally: + logger.setLevel(old_level) + + +def mock_nonblocking_socket(proto=socket.IPPROTO_TCP, type=socket.SOCK_STREAM, + family=socket.AF_INET): + """Create a mock of a non-blocking socket.""" + sock = mock.MagicMock(socket.socket) + sock.proto = proto + sock.type = type + sock.family = family + sock.gettimeout.return_value = 0.0 + return sock + + +async def await_without_task(coro): + exc = None + def func(): + try: + for _ in coro.__await__(): + pass + except BaseException as err: + nonlocal exc + exc = err + asyncio.get_running_loop().call_soon(func) + await asyncio.sleep(0) + if exc is not None: + raise exc + + +if sys.platform == 'win32': + DefaultEventLoopPolicy = asyncio.windows_events._DefaultEventLoopPolicy +else: + DefaultEventLoopPolicy = asyncio.unix_events._DefaultEventLoopPolicy diff --git a/Lib/test/test_base64.py b/Lib/test/test_base64.py index 409c8c109e8..a6739124571 100644 --- a/Lib/test/test_base64.py +++ b/Lib/test/test_base64.py @@ -1,10 +1,18 @@ -import unittest import base64 import binascii import os +import unittest from array import array +from test.support import cpython_only from test.support import os_helper from test.support import script_helper +from test.support.import_helper import ensure_lazy_imports + + +class LazyImportTest(unittest.TestCase): + @cpython_only + def test_lazy_import(self): + ensure_lazy_imports("base64", {"re", "getopt"}) class LegacyBase64TestCase(unittest.TestCase): @@ -200,18 +208,6 @@ def test_b64decode(self): self.check_other_types(base64.b64decode, b"YWJj", b"abc") self.check_decode_type_errors(base64.b64decode) - # Test with arbitrary alternative characters - tests_altchars = {(b'01a*b$cd', b'*$'): b'\xd3V\xbeo\xf7\x1d', - } - for (data, altchars), res in tests_altchars.items(): - data_str = data.decode('ascii') - altchars_str = altchars.decode('ascii') - - eq(base64.b64decode(data, altchars=altchars), res) - eq(base64.b64decode(data_str, altchars=altchars), res) - eq(base64.b64decode(data, altchars=altchars_str), res) - eq(base64.b64decode(data_str, altchars=altchars_str), res) - # Test standard alphabet for data, res in tests.items(): eq(base64.standard_b64decode(data), res) @@ -232,6 +228,20 @@ def test_b64decode(self): b'\xd3V\xbeo\xf7\x1d') self.check_decode_type_errors(base64.urlsafe_b64decode) + def test_b64decode_altchars(self): + # Test with arbitrary alternative characters + eq = self.assertEqual + res = b'\xd3V\xbeo\xf7\x1d' + for altchars in b'*$', b'+/', b'/+', b'+_', b'-+', b'-/', b'/_': + data = b'01a%cb%ccd' % tuple(altchars) + data_str = data.decode('ascii') + altchars_str = altchars.decode('ascii') + + eq(base64.b64decode(data, altchars=altchars), res) + eq(base64.b64decode(data_str, altchars=altchars), res) + eq(base64.b64decode(data, altchars=altchars_str), res) + eq(base64.b64decode(data_str, altchars=altchars_str), res) + def test_b64decode_padding_error(self): self.assertRaises(binascii.Error, base64.b64decode, b'abc') self.assertRaises(binascii.Error, base64.b64decode, 'abc') @@ -264,9 +274,12 @@ def test_b64decode_invalid_chars(self): base64.b64decode(bstr.decode('ascii'), validate=True) # Normal alphabet characters not discarded when alternative given - res = b'\xFB\xEF\xBE\xFF\xFF\xFF' - self.assertEqual(base64.b64decode(b'++[[//]]', b'[]'), res) - self.assertEqual(base64.urlsafe_b64decode(b'++--//__'), res) + res = b'\xfb\xef\xff' + self.assertEqual(base64.b64decode(b'++//', validate=True), res) + self.assertEqual(base64.b64decode(b'++//', '-_', validate=True), res) + self.assertEqual(base64.b64decode(b'--__', '-_', validate=True), res) + self.assertEqual(base64.urlsafe_b64decode(b'++//'), res) + self.assertEqual(base64.urlsafe_b64decode(b'--__'), res) def test_b32encode(self): eq = self.assertEqual @@ -321,23 +334,33 @@ def test_b32decode_casefold(self): self.assertRaises(binascii.Error, base64.b32decode, b'me======') self.assertRaises(binascii.Error, base64.b32decode, 'me======') + def test_b32decode_map01(self): # Mapping zero and one - eq(base64.b32decode(b'MLO23456'), b'b\xdd\xad\xf3\xbe') - eq(base64.b32decode('MLO23456'), b'b\xdd\xad\xf3\xbe') - - map_tests = {(b'M1023456', b'L'): b'b\xdd\xad\xf3\xbe', - (b'M1023456', b'I'): b'b\x1d\xad\xf3\xbe', - } - for (data, map01), res in map_tests.items(): - data_str = data.decode('ascii') + eq = self.assertEqual + res_L = b'b\xdd\xad\xf3\xbe' + res_I = b'b\x1d\xad\xf3\xbe' + eq(base64.b32decode(b'MLO23456'), res_L) + eq(base64.b32decode('MLO23456'), res_L) + eq(base64.b32decode(b'MIO23456'), res_I) + eq(base64.b32decode('MIO23456'), res_I) + self.assertRaises(binascii.Error, base64.b32decode, b'M1023456') + self.assertRaises(binascii.Error, base64.b32decode, b'M1O23456') + self.assertRaises(binascii.Error, base64.b32decode, b'ML023456') + self.assertRaises(binascii.Error, base64.b32decode, b'MI023456') + + data = b'M1023456' + data_str = data.decode('ascii') + for map01, res in [(b'L', res_L), (b'I', res_I)]: map01_str = map01.decode('ascii') eq(base64.b32decode(data, map01=map01), res) eq(base64.b32decode(data_str, map01=map01), res) eq(base64.b32decode(data, map01=map01_str), res) eq(base64.b32decode(data_str, map01=map01_str), res) - self.assertRaises(binascii.Error, base64.b32decode, data) - self.assertRaises(binascii.Error, base64.b32decode, data_str) + + eq(base64.b32decode(b'M1O23456', map01=map01), res) + eq(base64.b32decode(b'M%c023456' % map01, map01=map01), res) + eq(base64.b32decode(b'M%cO23456' % map01, map01=map01), res) def test_b32decode_error(self): tests = [b'abc', b'ABCDEF==', b'==ABCDEF'] @@ -804,7 +827,7 @@ def test_decode_nonascii_str(self): self.assertRaises(ValueError, f, 'with non-ascii \xcb') def test_ErrorHeritage(self): - self.assertTrue(issubclass(binascii.Error, ValueError)) + self.assertIsSubclass(binascii.Error, ValueError) def test_RFC4648_test_cases(self): # test cases from RFC 4648 section 10 diff --git a/Lib/test/test_baseexception.py b/Lib/test/test_baseexception.py index 63bf538aa53..5870dc7f9da 100644 --- a/Lib/test/test_baseexception.py +++ b/Lib/test/test_baseexception.py @@ -10,13 +10,11 @@ class ExceptionClassTests(unittest.TestCase): inheritance hierarchy)""" def test_builtins_new_style(self): - self.assertTrue(issubclass(Exception, object)) + self.assertIsSubclass(Exception, object) def verify_instance_interface(self, ins): for attr in ("args", "__str__", "__repr__"): - self.assertTrue(hasattr(ins, attr), - "%s missing %s attribute" % - (ins.__class__.__name__, attr)) + self.assertHasAttr(ins, attr) def test_inheritance(self): # Make sure the inheritance hierarchy matches the documentation @@ -65,7 +63,7 @@ def test_inheritance(self): elif last_depth > depth: while superclasses[-1][0] >= depth: superclasses.pop() - self.assertTrue(issubclass(exc, superclasses[-1][1]), + self.assertIsSubclass(exc, superclasses[-1][1], "%s is not a subclass of %s" % (exc.__name__, superclasses[-1][1].__name__)) try: # Some exceptions require arguments; just skip them diff --git a/Lib/test/test_bdb.py b/Lib/test/test_bdb.py index d1c1c786861..c41fb763a16 100644 --- a/Lib/test/test_bdb.py +++ b/Lib/test/test_bdb.py @@ -590,7 +590,7 @@ def fail(self, msg=None): class StateTestCase(BaseTestCase): """Test the step, next, return, until and quit 'set_' methods.""" - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_step(self): self.expect_set = [ ('line', 2, 'tfunc_main'), ('step', ), @@ -601,7 +601,7 @@ def test_step(self): with TracerRun(self) as tracer: tracer.runcall(tfunc_main) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_step_next_on_last_statement(self): for set_type in ('step', 'next'): with self.subTest(set_type=set_type): @@ -616,8 +616,7 @@ def test_step_next_on_last_statement(self): with TracerRun(self) as tracer: tracer.runcall(tfunc_main) - @unittest.skip('TODO: RUSTPYTHON') - # AssertionError: All paired tuples have not been processed, the last one was number 1 [('next',), ('quit',)] + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: All paired tuples have not been processed, the last one was number 1 [('next',), ('quit',)] def test_stepinstr(self): self.expect_set = [ ('line', 2, 'tfunc_main'), ('stepinstr', ), @@ -627,7 +626,7 @@ def test_stepinstr(self): with TracerRun(self) as tracer: tracer.runcall(tfunc_main) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_next(self): self.expect_set = [ ('line', 2, 'tfunc_main'), ('step', ), @@ -639,7 +638,7 @@ def test_next(self): with TracerRun(self) as tracer: tracer.runcall(tfunc_main) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_next_over_import(self): code = """ def main(): @@ -654,7 +653,7 @@ def main(): with TracerRun(self) as tracer: tracer.runcall(tfunc_import) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_next_on_plain_statement(self): # Check that set_next() is equivalent to set_step() on a plain # statement. @@ -667,7 +666,7 @@ def test_next_on_plain_statement(self): with TracerRun(self) as tracer: tracer.runcall(tfunc_main) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_next_in_caller_frame(self): # Check that set_next() in the caller frame causes the tracer # to stop next in the caller frame. @@ -681,7 +680,7 @@ def test_next_in_caller_frame(self): with TracerRun(self) as tracer: tracer.runcall(tfunc_main) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_return(self): self.expect_set = [ ('line', 2, 'tfunc_main'), ('step', ), @@ -694,7 +693,7 @@ def test_return(self): with TracerRun(self) as tracer: tracer.runcall(tfunc_main) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_return_in_caller_frame(self): self.expect_set = [ ('line', 2, 'tfunc_main'), ('step', ), @@ -706,7 +705,7 @@ def test_return_in_caller_frame(self): with TracerRun(self) as tracer: tracer.runcall(tfunc_main) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_until(self): self.expect_set = [ ('line', 2, 'tfunc_main'), ('step', ), @@ -718,7 +717,7 @@ def test_until(self): with TracerRun(self) as tracer: tracer.runcall(tfunc_main) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_until_with_too_large_count(self): self.expect_set = [ ('line', 2, 'tfunc_main'), break_in_func('tfunc_first'), @@ -729,7 +728,7 @@ def test_until_with_too_large_count(self): with TracerRun(self) as tracer: tracer.runcall(tfunc_main) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_until_in_caller_frame(self): self.expect_set = [ ('line', 2, 'tfunc_main'), ('step', ), @@ -741,7 +740,7 @@ def test_until_in_caller_frame(self): with TracerRun(self) as tracer: tracer.runcall(tfunc_main) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs @patch_list(sys.meta_path) def test_skip(self): # Check that tracing is skipped over the import statement in @@ -775,7 +774,7 @@ def test_skip_with_no_name_module(self): bdb = Bdb(skip=['anything*']) self.assertIs(bdb.is_skipped_module(None), False) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_down(self): # Check that set_down() raises BdbError at the newest frame. self.expect_set = [ @@ -784,7 +783,7 @@ def test_down(self): with TracerRun(self) as tracer: self.assertRaises(BdbError, tracer.runcall, tfunc_main) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_up(self): self.expect_set = [ ('line', 2, 'tfunc_main'), ('step', ), @@ -798,7 +797,7 @@ def test_up(self): class BreakpointTestCase(BaseTestCase): """Test the breakpoint set method.""" - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_bp_on_non_existent_module(self): self.expect_set = [ ('line', 2, 'tfunc_import'), ('break', ('/non/existent/module.py', 1)) @@ -806,7 +805,7 @@ def test_bp_on_non_existent_module(self): with TracerRun(self) as tracer: self.assertRaises(BdbError, tracer.runcall, tfunc_import) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_bp_after_last_statement(self): code = """ def main(): @@ -820,7 +819,7 @@ def main(): with TracerRun(self) as tracer: self.assertRaises(BdbError, tracer.runcall, tfunc_import) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_temporary_bp(self): code = """ def func(): @@ -844,7 +843,7 @@ def main(): with TracerRun(self) as tracer: tracer.runcall(tfunc_import) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_disabled_temporary_bp(self): code = """ def func(): @@ -873,7 +872,7 @@ def main(): with TracerRun(self) as tracer: tracer.runcall(tfunc_import) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_bp_condition(self): code = """ def func(a): @@ -894,7 +893,7 @@ def main(): with TracerRun(self) as tracer: tracer.runcall(tfunc_import) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_bp_exception_on_condition_evaluation(self): code = """ def func(a): @@ -914,7 +913,7 @@ def main(): with TracerRun(self) as tracer: tracer.runcall(tfunc_import) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_bp_ignore_count(self): code = """ def func(): @@ -936,7 +935,7 @@ def main(): with TracerRun(self) as tracer: tracer.runcall(tfunc_import) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_ignore_count_on_disabled_bp(self): code = """ def func(): @@ -964,7 +963,7 @@ def main(): with TracerRun(self) as tracer: tracer.runcall(tfunc_import) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_clear_two_bp_on_same_line(self): code = """ def func(): @@ -990,7 +989,7 @@ def main(): with TracerRun(self) as tracer: tracer.runcall(tfunc_import) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_clear_at_no_bp(self): self.expect_set = [ ('line', 2, 'tfunc_import'), ('clear', (__file__, 1)) @@ -1044,7 +1043,7 @@ def test_load_bps_from_previous_Bdb_instance(self): class RunTestCase(BaseTestCase): """Test run, runeval and set_trace.""" - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_run_step(self): # Check that the bdb 'run' method stops at the first line event. code = """ @@ -1057,7 +1056,7 @@ def test_run_step(self): with TracerRun(self) as tracer: tracer.run(compile(textwrap.dedent(code), '', 'exec')) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_runeval_step(self): # Test bdb 'runeval'. code = """ @@ -1080,7 +1079,7 @@ def main(): class IssuesTestCase(BaseTestCase): """Test fixed bdb issues.""" - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_step_at_return_with_no_trace_in_caller(self): # Issue #13183. # Check that the tracer does step into the caller frame when the @@ -1111,7 +1110,7 @@ def func(): with TracerRun(self) as tracer: tracer.runcall(tfunc_import) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_next_until_return_in_generator(self): # Issue #16596. # Check that set_next(), set_until() and set_return() do not treat the @@ -1153,7 +1152,7 @@ def main(): with TracerRun(self) as tracer: tracer.runcall(tfunc_import) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_next_command_in_generator_for_loop(self): # Issue #16596. code = """ @@ -1185,7 +1184,7 @@ def main(): with TracerRun(self) as tracer: tracer.runcall(tfunc_import) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_next_command_in_generator_with_subiterator(self): # Issue #16596. code = """ @@ -1217,7 +1216,7 @@ def main(): with TracerRun(self) as tracer: tracer.runcall(tfunc_import) - @unittest.skip('TODO: RUSTPYTHON; Error in atexit._run_exitfuncs') + @unittest.expectedFailure # TODO: RUSTPYTHON; Error in atexit._run_exitfuncs def test_return_command_in_generator_with_subiterator(self): # Issue #16596. code = """ @@ -1249,8 +1248,7 @@ def main(): with TracerRun(self) as tracer: tracer.runcall(tfunc_import) - @unittest.skip('TODO: RUSTPYTHON') - # AssertionError: All paired tuples have not been processed, the last one was number 1 [('next',)] + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: All paired tuples have not been processed, the last one was number 1 [('next',)] def test_next_to_botframe(self): # gh-125422 # Check that next command won't go to the bottom frame. diff --git a/Lib/test/test_bigmem.py b/Lib/test/test_bigmem.py index aaa9972bc45..8f528812e35 100644 --- a/Lib/test/test_bigmem.py +++ b/Lib/test/test_bigmem.py @@ -638,8 +638,6 @@ def test_encode_utf7(self, size): except MemoryError: pass # acceptable on 32-bit - # TODO: RUSTPYTHON - @unittest.expectedFailure @bigmemtest(size=_4G // 4 + 5, memuse=ascii_char_size + ucs4_char_size + 4) def test_encode_utf32(self, size): try: diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py index 55585d9ef84..1e1114b4a31 100644 --- a/Lib/test/test_builtin.py +++ b/Lib/test/test_builtin.py @@ -2010,21 +2010,18 @@ def test_construct_singletons(self): self.assertRaises(TypeError, tp, 1, 2) self.assertRaises(TypeError, tp, a=1, b=2) - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_warning_notimplemented(self): - # Issue #35712: NotImplemented is a sentinel value that should never + def test_bool_notimplemented(self): + # GH-79893: NotImplemented is a sentinel value that should never # be evaluated in a boolean context (virtually all such use cases # are a result of accidental misuse implementing rich comparison # operations in terms of one another). - # For the time being, it will continue to evaluate as a true value, but - # issue a deprecation warning (with the eventual intent to make it - # a TypeError). - self.assertWarns(DeprecationWarning, bool, NotImplemented) - with self.assertWarns(DeprecationWarning): - self.assertTrue(NotImplemented) - with self.assertWarns(DeprecationWarning): - self.assertFalse(not NotImplemented) + msg = "NotImplemented should not be used in a boolean context" + self.assertRaisesRegex(TypeError, msg, bool, NotImplemented) + with self.assertRaisesRegex(TypeError, msg): + if NotImplemented: + pass + with self.assertRaisesRegex(TypeError, msg): + not NotImplemented class TestBreakpoint(unittest.TestCase): @@ -2271,6 +2268,7 @@ def test_input_tty_non_ascii_unicode_errors(self): # Check stdin/stdout error handler is used when invoking PyOS_Readline() self.check_input_tty("prompté", b"quux\xe9", "ascii") + @unittest.skip('TODO: RUSTPYTHON FAILURE, WORKER BUG') @unittest.expectedFailure # TODO: RUSTPYTHON AssertionError: got 0 lines in pipe but expected 2, child output was: quux def test_input_no_stdout_fileno(self): # Issue #24402: If stdin is the original terminal but stdout.fileno() @@ -2332,8 +2330,6 @@ def test_baddecorator(self): class ShutdownTest(unittest.TestCase): - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_cleanup(self): # Issue #19255: builtins are still available at shutdown code = """if 1: diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py index 8f01f890309..fcef9c0c972 100644 --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -1288,11 +1288,20 @@ class SubBytes(bytes): self.assertNotEqual(id(s), id(1 * s)) self.assertNotEqual(id(s), id(s * 2)) + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_fromhex(self): + return super().test_fromhex() + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_mod(self): + return super().test_mod() + class ByteArrayTest(BaseBytesTest, unittest.TestCase): type2test = bytearray - _testlimitedcapi = import_helper.import_module('_testlimitedcapi') + # XXX: RUSTPYTHON; import_helper.import_module here cause the entire test stopping + _testlimitedcapi = None # import_helper.import_module('_testlimitedcapi') def test_getitem_error(self): b = bytearray(b'python') @@ -1385,6 +1394,7 @@ def by(s): b = by("Hello, world") self.assertEqual(re.findall(br"\w+", b), [by("Hello"), by("world")]) + @unittest.expectedFailure # TODO: RUSTPYTHON def test_setitem(self): def setitem_as_mapping(b, i, val): b[i] = val @@ -1432,6 +1442,7 @@ def do_tests(setitem): with self.subTest("tp_as_sequence"): do_tests(setitem_as_sequence) + @unittest.expectedFailure # TODO: RUSTPYTHON def test_delitem(self): def del_as_mapping(b, i): del b[i] @@ -1618,6 +1629,7 @@ def g(): alloc = b.__alloc__() self.assertGreaterEqual(alloc, len(b)) # NOTE: RUSTPYTHON patched + @unittest.expectedFailure # TODO: RUSTPYTHON def test_extend(self): orig = b'hello' a = bytearray(orig) @@ -1840,6 +1852,7 @@ def test_repeat_after_setslice(self): self.assertEqual(b1, b) self.assertEqual(b3, b'xcxcxc') + @unittest.expectedFailure # TODO: RUSTPYTHON def test_mutating_index(self): # bytearray slice assignment can call into python code # that reallocates the internal buffer @@ -1860,6 +1873,7 @@ def __index__(self): with self.assertRaises(IndexError): self._testlimitedcapi.sequence_setitem(b, 0, Boom()) + @unittest.expectedFailure # TODO: RUSTPYTHON def test_mutating_index_inbounds(self): # gh-91153 continued # Ensure buffer is not broken even if length is correct @@ -1893,6 +1907,14 @@ def __index__(self): self.assertEqual(instance.ba[0], ord("?"), "Assigned bytearray not altered") self.assertEqual(instance.new_ba, bytearray(0x180), "Wrong object altered") + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_fromhex(self): + return super().test_fromhex() + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_mod(self): + return super().test_mod() + class AssortedBytesTest(unittest.TestCase): # @@ -1912,6 +1934,7 @@ def test_bytes_repr(self, f=repr): self.assertEqual(f(b"'\"'"), r"""b'\'"\''""") # '\'"\'' self.assertEqual(f(BytesSubclass(b"abc")), "b'abc'") + @unittest.expectedFailure # TODO: RUSTPYTHON def test_bytearray_repr(self, f=repr): self.assertEqual(f(bytearray()), "bytearray(b'')") self.assertEqual(f(bytearray(b'abc')), "bytearray(b'abc')") @@ -1933,6 +1956,7 @@ def test_bytearray_repr(self, f=repr): def test_bytes_str(self): self.test_bytes_repr(str) + @unittest.expectedFailure # TODO: RUSTPYTHON @check_bytes_warnings def test_bytearray_str(self): self.test_bytearray_repr(str) @@ -2138,7 +2162,6 @@ def test_join(self): s3 = s1.join([b"abcd"]) self.assertIs(type(s3), self.basetype) - @unittest.skip('TODO: RUSTPYTHON; Fails on ByteArraySubclassWithSlotsTest') def test_pickle(self): a = self.type2test(b"abcd") a.x = 10 @@ -2153,7 +2176,6 @@ def test_pickle(self): self.assertEqual(type(a.z), type(b.z)) self.assertFalse(hasattr(b, 'y')) - @unittest.skip('TODO: RUSTPYTHON; Fails on ByteArraySubclassWithSlotsTest') def test_copy(self): a = self.type2test(b"abcd") a.x = 10 @@ -2235,6 +2257,14 @@ class ByteArraySubclassWithSlotsTest(SubclassTest, unittest.TestCase): basetype = bytearray type2test = ByteArraySubclassWithSlots + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_copy(self): + return super().test_copy() + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_pickle(self): + return super().test_pickle() + class BytesSubclassTest(SubclassTest, unittest.TestCase): basetype = bytes type2test = BytesSubclass diff --git a/Lib/test/test_bz2.py b/Lib/test/test_bz2.py index 3617eba8e8d..26b5e79d337 100644 --- a/Lib/test/test_bz2.py +++ b/Lib/test/test_bz2.py @@ -16,7 +16,7 @@ from test.support import import_helper from test.support import threading_helper from test.support.os_helper import unlink, FakePath -import _compression +from compression._common import _streams import sys @@ -126,15 +126,15 @@ def testReadMultiStream(self): def testReadMonkeyMultiStream(self): # Test BZ2File.read() on a multi-stream archive where a stream # boundary coincides with the end of the raw read buffer. - buffer_size = _compression.BUFFER_SIZE - _compression.BUFFER_SIZE = len(self.DATA) + buffer_size = _streams.BUFFER_SIZE + _streams.BUFFER_SIZE = len(self.DATA) try: self.createTempFile(streams=5) with BZ2File(self.filename) as bz2f: self.assertRaises(TypeError, bz2f.read, float()) self.assertEqual(bz2f.read(), self.TEXT * 5) finally: - _compression.BUFFER_SIZE = buffer_size + _streams.BUFFER_SIZE = buffer_size def testReadTrailingJunk(self): self.createTempFile(suffix=self.BAD_DATA) @@ -184,7 +184,7 @@ def testPeek(self): with BZ2File(self.filename) as bz2f: pdata = bz2f.peek() self.assertNotEqual(len(pdata), 0) - self.assertTrue(self.TEXT.startswith(pdata)) + self.assertStartsWith(self.TEXT, pdata) self.assertEqual(bz2f.read(), self.TEXT) def testReadInto(self): @@ -730,8 +730,7 @@ def testOpenBytesFilename(self): self.assertEqual(f.read(), self.DATA) self.assertEqual(f.name, str_filename) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def testOpenPathLikeFilename(self): filename = FakePath(self.filename) with BZ2File(filename, "wb") as f: @@ -744,7 +743,7 @@ def testOpenPathLikeFilename(self): def testDecompressLimited(self): """Decompressed data buffering should be limited""" bomb = bz2.compress(b'\0' * int(2e6), compresslevel=9) - self.assertLess(len(bomb), _compression.BUFFER_SIZE) + self.assertLess(len(bomb), _streams.BUFFER_SIZE) decomp = BZ2File(BytesIO(bomb)) self.assertEqual(decomp.read(1), b'\0') @@ -770,7 +769,7 @@ def testPeekBytesIO(self): with BZ2File(bio) as bz2f: pdata = bz2f.peek() self.assertNotEqual(len(pdata), 0) - self.assertTrue(self.TEXT.startswith(pdata)) + self.assertStartsWith(self.TEXT, pdata) self.assertEqual(bz2f.read(), self.TEXT) def testWriteBytesIO(self): @@ -1190,8 +1189,7 @@ def test_encoding_error_handler(self): as f: self.assertEqual(f.read(), "foobar") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_newline(self): # Test with explicit newline (universal newline mode disabled). text = self.TEXT.decode("ascii") diff --git a/Lib/test/test_calendar.py b/Lib/test/test_calendar.py index df102fe1986..7ade4271b7a 100644 --- a/Lib/test/test_calendar.py +++ b/Lib/test/test_calendar.py @@ -987,10 +987,11 @@ def assertFailure(self, *args): self.assertCLIFails(*args) self.assertCmdFails(*args) + @support.force_not_colorized def test_help(self): stdout = self.run_cmd_ok('-h') self.assertIn(b'usage:', stdout) - self.assertIn(b'calendar.py', stdout) + self.assertIn(b' -m calendar ', stdout) self.assertIn(b'--help', stdout) # special case: stdout but sys.exit() @@ -1097,7 +1098,7 @@ def test_option_type(self): output = run('--type', 'text', '2004') self.assertEqual(output, conv(result_2004_text)) output = run('--type', 'html', '2004') - self.assertEqual(output[:6], b'Calendar for 2004', output) def test_html_output_current_year(self): diff --git a/Lib/test/test_class.py b/Lib/test/test_class.py index 29215f06002..e0756b25a5d 100644 --- a/Lib/test/test_class.py +++ b/Lib/test/test_class.py @@ -1,6 +1,7 @@ "Test the functionality of Python classes implementing operators." import unittest +from test import support from test.support import cpython_only, import_helper, script_helper testmeths = [ @@ -134,6 +135,7 @@ def __%s__(self, *args): AllTests = type("AllTests", (object,), d) del d, statictests, method, method_template +@support.thread_unsafe("callLst is shared between threads") class ClassTests(unittest.TestCase): def setUp(self): callLst[:] = [] @@ -554,7 +556,9 @@ class Custom: self.assertFalse(hasattr(o, "__call__")) self.assertFalse(hasattr(c, "__call__")) - @unittest.skip("TODO: RUSTPYTHON, segmentation fault") + @unittest.skip("TODO: RUSTPYTHON; segmentation fault") + @support.skip_emscripten_stack_overflow() + @support.skip_wasi_stack_overflow() def testSFBug532646(self): # Test for SF bug 532646 @@ -570,8 +574,7 @@ class A: else: self.fail("Failed to raise RecursionError") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def testForExceptionsRaisedInInstanceGetattr2(self): # Tests for exceptions raised in instance_getattr2(). @@ -611,7 +614,6 @@ def assertNotOrderable(self, a, b): with self.assertRaises(TypeError): a >= b - @unittest.skip("TODO: RUSTPYTHON; unstable result") def testHashComparisonOfMethods(self): # Test comparison and hash of methods class A: @@ -689,8 +691,7 @@ class A: with self.assertRaisesRegex(AttributeError, error_msg): del A.x - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def testObjectAttributeAccessErrorMessages(self): class A: pass @@ -740,8 +741,7 @@ def __setattr__(self, name, value) -> None: with self.assertRaisesRegex(AttributeError, error_msg): del B().z - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def testConstructorErrorMessages(self): # bpo-31506: Improves the error message logic for object_new & object_init @@ -846,12 +846,28 @@ def __init__(self, obj): Type(i) self.assertEqual(calls, 100) -try: - from _testinternalcapi import has_inline_values -except ImportError: - has_inline_values = None + def test_specialization_class_call_doesnt_crash(self): + # gh-123185 + + class Foo: + def __init__(self, arg): + pass + + for _ in range(8): + try: + Foo() + except: + pass + + +# from _testinternalcapi import has_inline_values # XXX: RUSTPYTHON + +Py_TPFLAGS_INLINE_VALUES = (1 << 2) +Py_TPFLAGS_MANAGED_DICT = (1 << 4) + +class NoManagedDict: + __slots__ = ('a',) -Py_TPFLAGS_MANAGED_DICT = (1 << 2) class Plain: pass @@ -866,38 +882,55 @@ def __init__(self): self.d = 4 +class VarSizedSubclass(tuple): + pass + + class TestInlineValues(unittest.TestCase): - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_flags(self): - self.assertEqual(Plain.__flags__ & Py_TPFLAGS_MANAGED_DICT, Py_TPFLAGS_MANAGED_DICT) - self.assertEqual(WithAttrs.__flags__ & Py_TPFLAGS_MANAGED_DICT, Py_TPFLAGS_MANAGED_DICT) + @unittest.expectedFailure # TODO: RUSTPYTHON; NameError: name 'has_inline_values' is not defined. + def test_no_flags_for_slots_class(self): + flags = NoManagedDict.__flags__ + self.assertEqual(flags & Py_TPFLAGS_MANAGED_DICT, 0) + self.assertEqual(flags & Py_TPFLAGS_INLINE_VALUES, 0) + self.assertFalse(has_inline_values(NoManagedDict())) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 0 != 4 + def test_both_flags_for_regular_class(self): + for cls in (Plain, WithAttrs): + with self.subTest(cls=cls.__name__): + flags = cls.__flags__ + self.assertEqual(flags & Py_TPFLAGS_MANAGED_DICT, Py_TPFLAGS_MANAGED_DICT) + self.assertEqual(flags & Py_TPFLAGS_INLINE_VALUES, Py_TPFLAGS_INLINE_VALUES) + self.assertTrue(has_inline_values(cls())) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 0 != 4 + def test_managed_dict_only_for_varsized_subclass(self): + flags = VarSizedSubclass.__flags__ + self.assertEqual(flags & Py_TPFLAGS_MANAGED_DICT, Py_TPFLAGS_MANAGED_DICT) + self.assertEqual(flags & Py_TPFLAGS_INLINE_VALUES, 0) + self.assertFalse(has_inline_values(VarSizedSubclass())) + + @unittest.expectedFailure # TODO: RUSTPYTHON def test_has_inline_values(self): c = Plain() self.assertTrue(has_inline_values(c)) del c.__dict__ self.assertFalse(has_inline_values(c)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_instances(self): self.assertTrue(has_inline_values(Plain())) self.assertTrue(has_inline_values(WithAttrs())) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_inspect_dict(self): for cls in (Plain, WithAttrs): c = cls() c.__dict__ self.assertTrue(has_inline_values(c)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_update_dict(self): d = { "e": 5, "f": 6 } for cls in (Plain, WithAttrs): @@ -914,8 +947,7 @@ def check_100(self, obj): for i in range(100): self.assertEqual(getattr(obj, f"a{i}"), i) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_many_attributes(self): class C: pass c = C() @@ -926,8 +958,7 @@ class C: pass c = C() self.assertTrue(has_inline_values(c)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_many_attributes_with_dict(self): class C: pass c = C() @@ -948,8 +979,7 @@ def __init__(self): obj.foo = None # Aborted here self.assertEqual(obj.__dict__, {"foo":None}) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_store_attr_deleted_dict(self): class Foo: pass @@ -959,8 +989,7 @@ class Foo: f.a = 3 self.assertEqual(f.a, 3) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_rematerialize_object_dict(self): # gh-121860: rematerializing an object's managed dictionary after it # had been deleted caused a crash. @@ -979,7 +1008,7 @@ class Bar: pass self.assertIsInstance(f, Bar) self.assertEqual(f.__dict__, {}) - @unittest.skip("TODO: RUSTPYTHON, unexpectedly long runtime") + @unittest.skip("TODO: RUSTPYTHON; unexpectedly long runtime") def test_store_attr_type_cache(self): """Verifies that the type cache doesn't provide a value which is inconsistent from the dict.""" @@ -1030,5 +1059,6 @@ def __init__(self): self.assertFalse(out, msg=out.decode('utf-8')) self.assertFalse(err, msg=err.decode('utf-8')) + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_cmath.py b/Lib/test/test_cmath.py index 44f1b2da638..a96a5780b31 100644 --- a/Lib/test/test_cmath.py +++ b/Lib/test/test_cmath.py @@ -276,7 +276,6 @@ def test_cmath_matches_math(self): self.rAssertAlmostEqual(math.log(v, base), z.real) self.assertEqual(0., z.imag) - @unittest.expectedFailure # TODO: RUSTPYTHON @requires_IEEE_754 def test_specific_values(self): # Some tests need to be skipped on ancient OS X versions. @@ -530,13 +529,11 @@ def testTanhSign(self): # log1p function; If that system function doesn't respect the sign # of zero, then atan and atanh will also have difficulties with # the sign of complex zeros. - @unittest.expectedFailure # TODO: RUSTPYTHON @requires_IEEE_754 def testAtanSign(self): for z in complex_zeros: self.assertComplexesAreIdentical(cmath.atan(z), z) - @unittest.expectedFailure # TODO: RUSTPYTHON @requires_IEEE_754 def testAtanhSign(self): for z in complex_zeros: @@ -583,7 +580,6 @@ def test_complex_near_zero(self): self.assertIsClose(0.001-0.001j, 0.001+0.001j, abs_tol=2e-03) self.assertIsNotClose(0.001-0.001j, 0.001+0.001j, abs_tol=1e-03) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_complex_special(self): self.assertIsNotClose(INF, INF*1j) self.assertIsNotClose(INF*1j, INF) diff --git a/Lib/test/test_cmd.py b/Lib/test/test_cmd.py index 46ec82b7049..dbfec42fc21 100644 --- a/Lib/test/test_cmd.py +++ b/Lib/test/test_cmd.py @@ -11,9 +11,15 @@ import io import textwrap from test import support -from test.support.import_helper import import_module +from test.support.import_helper import ensure_lazy_imports, import_module from test.support.pty_helper import run_pty +class LazyImportTest(unittest.TestCase): + @support.cpython_only + def test_lazy_import(self): + ensure_lazy_imports("cmd", {"inspect", "string"}) + + class samplecmdclass(cmd.Cmd): """ Instance the sampleclass: @@ -289,6 +295,30 @@ def do_tab_completion_test(self, args): self.assertIn(b'ab_completion_test', output) self.assertIn(b'tab completion success', output) + def test_bang_completion_without_do_shell(self): + script = textwrap.dedent(""" + import cmd + class simplecmd(cmd.Cmd): + def completedefault(self, text, line, begidx, endidx): + return ["hello"] + + def default(self, line): + if line.replace(" ", "") == "!hello": + print('tab completion success') + else: + print('tab completion failure') + return True + + simplecmd().cmdloop() + """) + + # '! h' or '!h' and complete 'ello' to 'hello' + for input in [b"! h\t\n", b"!h\t\n"]: + with self.subTest(input=input): + output = run_pty(script, input) + self.assertIn(b'hello', output) + self.assertIn(b'tab completion success', output) + def load_tests(loader, tests, pattern): tests.addTest(doctest.DocTestSuite()) return tests diff --git a/Lib/test/test_code_module.py b/Lib/test/test_code_module.py index 4d5463e6db3..0ed17b390c4 100644 --- a/Lib/test/test_code_module.py +++ b/Lib/test/test_code_module.py @@ -39,19 +39,47 @@ def setUp(self): self.mock_sys() def test_ps1(self): - self.infunc.side_effect = EOFError('Finished') + self.infunc.side_effect = [ + "import code", + "code.sys.ps1", + EOFError('Finished') + ] self.console.interact() - self.assertEqual(self.sysmod.ps1, '>>> ') + output = ''.join(''.join(call[1]) for call in self.stdout.method_calls) + self.assertIn('>>> ', output) + self.assertNotHasAttr(self.sysmod, 'ps1') + + self.infunc.side_effect = [ + "import code", + "code.sys.ps1", + EOFError('Finished') + ] self.sysmod.ps1 = 'custom1> ' self.console.interact() + output = ''.join(''.join(call[1]) for call in self.stdout.method_calls) + self.assertIn('custom1> ', output) self.assertEqual(self.sysmod.ps1, 'custom1> ') def test_ps2(self): - self.infunc.side_effect = EOFError('Finished') + self.infunc.side_effect = [ + "import code", + "code.sys.ps2", + EOFError('Finished') + ] self.console.interact() - self.assertEqual(self.sysmod.ps2, '... ') + output = ''.join(''.join(call[1]) for call in self.stdout.method_calls) + self.assertIn('... ', output) + self.assertNotHasAttr(self.sysmod, 'ps2') + + self.infunc.side_effect = [ + "import code", + "code.sys.ps2", + EOFError('Finished') + ] self.sysmod.ps2 = 'custom2> ' self.console.interact() + output = ''.join(''.join(call[1]) for call in self.stdout.method_calls) + self.assertIn('custom2> ', output) self.assertEqual(self.sysmod.ps2, 'custom2> ') def test_console_stderr(self): @@ -63,9 +91,7 @@ def test_console_stderr(self): else: raise AssertionError("no console stdout") - # TODO: RUSTPYTHON - # AssertionError: Lists differ: [' F[27 chars] x = ?', ' ^', 'SyntaxError: got unexpected token ?'] != [' F[27 chars] x = ?', ' ^', 'SyntaxError: invalid syntax'] - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; + 'SyntaxError: invalid syntax'] def test_syntax_error(self): self.infunc.side_effect = ["def f():", " x = ?", @@ -86,9 +112,7 @@ def test_syntax_error(self): self.assertIsNone(self.sysmod.last_value.__traceback__) self.assertIs(self.sysmod.last_exc, self.sysmod.last_value) - # TODO: RUSTPYTHON - # AssertionError: Lists differ: [' F[15 chars], line 1', ' 1', 'IndentationError: unexpected indentation'] != [' F[15 chars], line 1', ' 1', 'IndentationError: unexpected indent'] - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; - 'IndentationError: unexpected indentation'] def test_indentation_error(self): self.infunc.side_effect = [" 1", EOFError('Finished')] self.console.interact() @@ -105,16 +129,14 @@ def test_indentation_error(self): self.assertIsNone(self.sysmod.last_value.__traceback__) self.assertIs(self.sysmod.last_exc, self.sysmod.last_value) - # TODO: RUSTPYTHON - # AssertionError: False is not true : UnicodeDecodeError: invalid utf-8 sequence of 1 bytes from index 1 - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 'UnicodeDecodeError: invalid utf-8 sequence of 1 bytes from index 1\n\nnow exiti [truncated]... doesn't start with 'UnicodeEncodeError: ' def test_unicode_error(self): self.infunc.side_effect = ["'\ud800'", EOFError('Finished')] self.console.interact() output = ''.join(''.join(call[1]) for call in self.stderr.method_calls) output = output[output.index('(InteractiveConsole)'):] output = output[output.index('\n') + 1:] - self.assertTrue(output.startswith('UnicodeEncodeError: '), output) + self.assertStartsWith(output, 'UnicodeEncodeError: ') self.assertIs(self.sysmod.last_type, UnicodeEncodeError) self.assertIs(type(self.sysmod.last_value), UnicodeEncodeError) self.assertIsNone(self.sysmod.last_traceback) @@ -144,9 +166,7 @@ def test_sysexcepthook(self): ' File "", line 2, in f\n', 'ValueError: BOOM!\n']) - # TODO: RUSTPYTHON - # AssertionError: Lists differ: [' F[35 chars]= ?\n', ' ^\n', 'SyntaxError: got unexpected token ?\n'] != [' F[35 chars]= ?\n', ' ^\n', 'SyntaxError: invalid syntax\n'] - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; + 'SyntaxError: invalid syntax\n'] def test_sysexcepthook_syntax_error(self): self.infunc.side_effect = ["def f():", " x = ?", @@ -170,9 +190,7 @@ def test_sysexcepthook_syntax_error(self): ' ^\n', 'SyntaxError: invalid syntax\n']) - # TODO: RUSTPYTHON - # AssertionError: Lists differ: [' F[21 chars] 1\n', ' 1\n', 'IndentationError: unexpected indentation\n'] != [' F[21 chars] 1\n', ' 1\n', 'IndentationError: unexpected indent\n'] - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; + 'IndentationError: unexpected indent\n'] def test_sysexcepthook_indentation_error(self): self.infunc.side_effect = [" 1", EOFError('Finished')] hook = mock.Mock() @@ -267,7 +285,7 @@ def test_exit_msg(self): self.assertEqual(err_msg, ['write', (expected,), {}]) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: '\nAttributeError\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "", line 1, in \nValueError\n' not found in 'Python on \nType "help", "copyright", "credits" or "license" for more information.\n(InteractiveConsole)\nAttributeError\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "", line 1, in \nValueError: \n\nnow exiting InteractiveConsole...\n' def test_cause_tb(self): self.infunc.side_effect = ["raise ValueError('') from AttributeError", EOFError('Finished')] diff --git a/Lib/test/test_codeccallbacks.py b/Lib/test/test_codeccallbacks.py index 9ca02cea351..763146c94fc 100644 --- a/Lib/test/test_codeccallbacks.py +++ b/Lib/test/test_codeccallbacks.py @@ -281,8 +281,6 @@ def handler2(exc): b"g[<252><223>]" ) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_longstrings(self): # test long strings to check for memory overflow problems errors = [ "strict", "ignore", "replace", "xmlcharrefreplace", @@ -684,8 +682,6 @@ def test_badandgoodsurrogateescapeexceptions(self): ("\udc80", 2) ) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_badandgoodsurrogatepassexceptions(self): surrogatepass_errors = codecs.lookup_error('surrogatepass') # "surrogatepass" complains about a non-exception passed in diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py index 736022599ed..fabf74fd9e8 100644 --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -1,12 +1,15 @@ import codecs import contextlib import copy +import importlib import io import pickle +import os import sys import unittest import encodings from unittest import mock +import warnings from test import support from test.support import os_helper @@ -20,13 +23,12 @@ except ImportError: _testinternalcapi = None -try: - import ctypes -except ImportError: - ctypes = None - SIZEOF_WCHAR_T = -1 -else: - SIZEOF_WCHAR_T = ctypes.sizeof(ctypes.c_wchar) + +def codecs_open_no_warn(*args, **kwargs): + """Call codecs.open(*args, **kwargs) ignoring DeprecationWarning.""" + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + return codecs.open(*args, **kwargs) def coding_checker(self, coder): def check(input, expect): @@ -35,13 +37,13 @@ def check(input, expect): # On small versions of Windows like Windows IoT or Windows Nano Server not all codepages are present def is_code_page_present(cp): - from ctypes import POINTER, WINFUNCTYPE, WinDLL + from ctypes import POINTER, WINFUNCTYPE, WinDLL, Structure from ctypes.wintypes import BOOL, BYTE, WCHAR, UINT, DWORD MAX_LEADBYTES = 12 # 5 ranges, 2 bytes ea., 0 term. MAX_DEFAULTCHAR = 2 # single or double byte MAX_PATH = 260 - class CPINFOEXW(ctypes.Structure): + class CPINFOEXW(Structure): _fields_ = [("MaxCharSize", UINT), ("DefaultChar", BYTE*MAX_DEFAULTCHAR), ("LeadByte", BYTE*MAX_LEADBYTES), @@ -388,7 +390,6 @@ def test_bug1098990_b(self): ill_formed_sequence_replace = "\ufffd" - @unittest.expectedFailure # TODO: RUSTPYTHON def test_lone_surrogates(self): self.assertRaises(UnicodeEncodeError, "\ud800".encode, self.encoding) self.assertEqual("[\uDC80]".encode(self.encoding, "backslashreplace"), @@ -464,7 +465,7 @@ class UTF32Test(ReadTest, unittest.TestCase): b'\x00\x00\x00s\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m' b'\x00\x00\x00s\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m') - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'codecs' has no attribute 'utf_32_ex_decode'. Did you mean: 'utf_16_ex_decode'? def test_only_one_bom(self): _,_,reader,writer = codecs.lookup(self.encoding) # encode some stream @@ -480,7 +481,7 @@ def test_only_one_bom(self): f = reader(s) self.assertEqual(f.read(), "spamspam") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'codecs' has no attribute 'utf_32_ex_decode'. Did you mean: 'utf_16_ex_decode'? def test_badbom(self): s = io.BytesIO(4*b"\xff") f = codecs.getreader(self.encoding)(s) @@ -490,7 +491,7 @@ def test_badbom(self): f = codecs.getreader(self.encoding)(s) self.assertRaises(UnicodeDecodeError, f.read) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'codecs' has no attribute 'utf_32_ex_decode'. Did you mean: 'utf_16_ex_decode'? def test_partial(self): self.check_partial( "\x00\xff\u0100\uffff\U00010000", @@ -522,26 +523,24 @@ def test_partial(self): ] ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_handlers(self): self.assertEqual(('\ufffd', 1), codecs.utf_32_decode(b'\x01', 'replace', True)) self.assertEqual(('', 1), codecs.utf_32_decode(b'\x01', 'ignore', True)) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_errors(self): self.assertRaises(UnicodeDecodeError, codecs.utf_32_decode, b"\xff", "strict", True) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'codecs' has no attribute 'utf_32_ex_decode'. Did you mean: 'utf_16_ex_decode'? def test_decoder_state(self): self.check_state_handling_decode(self.encoding, "spamspam", self.spamle) self.check_state_handling_decode(self.encoding, "spamspam", self.spambe) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_issue8941(self): # Issue #8941: insufficient result allocation when decoding into # surrogate pairs on UCS-2 builds. @@ -552,40 +551,50 @@ def test_issue8941(self): self.assertEqual('\U00010000' * 1024, codecs.utf_32_decode(encoded_be)[0]) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_lone_surrogates(self): + return super().test_lone_surrogates() + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'codecs' has no attribute 'utf_32_ex_decode'. Did you mean: 'utf_16_ex_decode'? def test_bug1098990_a(self): return super().test_bug1098990_a() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'codecs' has no attribute 'utf_32_ex_decode'. Did you mean: 'utf_16_ex_decode'? def test_bug1098990_b(self): return super().test_bug1098990_b() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'codecs' has no attribute 'utf_32_ex_decode'. Did you mean: 'utf_16_ex_decode'? def test_bug1175396(self): return super().test_bug1175396() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'codecs' has no attribute 'utf_32_ex_decode'. Did you mean: 'utf_16_ex_decode'? def test_incremental_surrogatepass(self): return super().test_incremental_surrogatepass() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'codecs' has no attribute 'utf_32_ex_decode'. Did you mean: 'utf_16_ex_decode'? def test_mixed_readline_and_read(self): return super().test_mixed_readline_and_read() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'codecs' has no attribute 'utf_32_ex_decode'. Did you mean: 'utf_16_ex_decode'? def test_readline(self): return super().test_readline() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'codecs' has no attribute 'utf_32_ex_decode'. Did you mean: 'utf_16_ex_decode'? def test_readlinequeue(self): return super().test_readlinequeue() + + + + + + + class UTF32LETest(ReadTest, unittest.TestCase): encoding = "utf-32-le" ill_formed_sequence = b"\x80\xdc\x00\x00" - @unittest.expectedFailure # TODO: RUSTPYTHON def test_partial(self): self.check_partial( "\x00\xff\u0100\uffff\U00010000", @@ -613,16 +622,13 @@ def test_partial(self): ] ) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_simple(self): self.assertEqual("\U00010203".encode(self.encoding), b"\x03\x02\x01\x00") - @unittest.expectedFailure # TODO: RUSTPYTHON def test_errors(self): self.assertRaises(UnicodeDecodeError, codecs.utf_32_le_decode, b"\xff", "strict", True) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_issue8941(self): # Issue #8941: insufficient result allocation when decoding into # surrogate pairs on UCS-2 builds. @@ -630,40 +636,22 @@ def test_issue8941(self): self.assertEqual('\U00010000' * 1024, codecs.utf_32_le_decode(encoded)[0]) - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_bug1098990_a(self): - return super().test_bug1098990_a() + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_lone_surrogates(self): + return super().test_lone_surrogates() + - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_bug1098990_b(self): - return super().test_bug1098990_b() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_bug1175396(self): - return super().test_bug1175396() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_incremental_surrogatepass(self): - return super().test_incremental_surrogatepass() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_mixed_readline_and_read(self): - return super().test_mixed_readline_and_read() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_readline(self): - return super().test_readline() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_readlinequeue(self): - return super().test_readlinequeue() class UTF32BETest(ReadTest, unittest.TestCase): encoding = "utf-32-be" ill_formed_sequence = b"\x00\x00\xdc\x80" - @unittest.expectedFailure # TODO: RUSTPYTHON def test_partial(self): self.check_partial( "\x00\xff\u0100\uffff\U00010000", @@ -691,16 +679,13 @@ def test_partial(self): ] ) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_simple(self): self.assertEqual("\U00010203".encode(self.encoding), b"\x00\x01\x02\x03") - @unittest.expectedFailure # TODO: RUSTPYTHON def test_errors(self): self.assertRaises(UnicodeDecodeError, codecs.utf_32_be_decode, b"\xff", "strict", True) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_issue8941(self): # Issue #8941: insufficient result allocation when decoding into # surrogate pairs on UCS-2 builds. @@ -708,33 +693,16 @@ def test_issue8941(self): self.assertEqual('\U00010000' * 1024, codecs.utf_32_be_decode(encoded)[0]) - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_bug1098990_a(self): - return super().test_bug1098990_a() + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_lone_surrogates(self): + return super().test_lone_surrogates() + - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_bug1098990_b(self): - return super().test_bug1098990_b() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_bug1175396(self): - return super().test_bug1175396() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_incremental_surrogatepass(self): - return super().test_incremental_surrogatepass() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_mixed_readline_and_read(self): - return super().test_mixed_readline_and_read() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_readline(self): - return super().test_readline() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_readlinequeue(self): - return super().test_readlinequeue() class UTF16Test(ReadTest, unittest.TestCase): @@ -771,7 +739,7 @@ def test_badbom(self): f = codecs.getreader(self.encoding)(s) self.assertRaises(UnicodeDecodeError, f.read) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; UnicodeDecodeError: 'utf-16' codec can't decode bytes in position 0-1: unexpected end of data def test_partial(self): self.check_partial( "\x00\xff\u0100\uffff\U00010000", @@ -793,7 +761,7 @@ def test_partial(self): ] ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; IndexError: index out of range def test_handlers(self): self.assertEqual(('\ufffd', 1), codecs.utf_16_decode(b'\x01', 'replace', True)) @@ -821,32 +789,37 @@ def test_bug691291(self): self.addCleanup(os_helper.unlink, os_helper.TESTFN) with open(os_helper.TESTFN, 'wb') as fp: fp.write(s) - with codecs.open(os_helper.TESTFN, 'r', + with codecs_open_no_warn(os_helper.TESTFN, 'r', encoding=self.encoding) as reader: self.assertEqual(reader.read(), s1) def test_invalid_modes(self): for mode in ('U', 'rU', 'r+U'): with self.assertRaises(ValueError) as cm: - codecs.open(os_helper.TESTFN, mode, encoding=self.encoding) + codecs_open_no_warn(os_helper.TESTFN, mode, encoding=self.encoding) self.assertIn('invalid mode', str(cm.exception)) for mode in ('rt', 'wt', 'at', 'r+t'): with self.assertRaises(ValueError) as cm: - codecs.open(os_helper.TESTFN, mode, encoding=self.encoding) + codecs_open_no_warn(os_helper.TESTFN, mode, encoding=self.encoding) self.assertIn("can't have text and binary mode at once", str(cm.exception)) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_lone_surrogates(self): + return super().test_lone_surrogates() + + @unittest.expectedFailure # TODO: RUSTPYTHON; IndexError: index out of range def test_incremental_surrogatepass(self): return super().test_incremental_surrogatepass() + class UTF16LETest(ReadTest, unittest.TestCase): encoding = "utf-16-le" ill_formed_sequence = b"\x80\xdc" - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; UnicodeDecodeError: 'utf-16' codec can't decode bytes in position 0-1: unexpected end of data def test_partial(self): self.check_partial( "\x00\xff\u0100\uffff\U00010000", @@ -866,7 +839,7 @@ def test_partial(self): ] ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_errors(self): tests = [ (b'\xff', '\ufffd'), @@ -888,15 +861,20 @@ def test_nonbmp(self): self.assertEqual(b'\x00\xd8\x03\xde'.decode(self.encoding), "\U00010203") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_lone_surrogates(self): + return super().test_lone_surrogates() + + @unittest.expectedFailure # TODO: RUSTPYTHON; IndexError: index out of range def test_incremental_surrogatepass(self): return super().test_incremental_surrogatepass() + class UTF16BETest(ReadTest, unittest.TestCase): encoding = "utf-16-be" ill_formed_sequence = b"\xdc\x80" - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; UnicodeDecodeError: 'utf-16' codec can't decode bytes in position 0-1: unexpected end of data def test_partial(self): self.check_partial( "\x00\xff\u0100\uffff\U00010000", @@ -916,7 +894,7 @@ def test_partial(self): ] ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_errors(self): tests = [ (b'\xff', '\ufffd'), @@ -938,10 +916,15 @@ def test_nonbmp(self): self.assertEqual(b'\xd8\x00\xde\x03'.decode(self.encoding), "\U00010203") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_lone_surrogates(self): + return super().test_lone_surrogates() + + @unittest.expectedFailure # TODO: RUSTPYTHON; UnicodeDecodeError: 'utf-16' codec can't decode bytes in position 0-1: unexpected end of data def test_incremental_surrogatepass(self): return super().test_incremental_surrogatepass() + class UTF8Test(ReadTest, unittest.TestCase): encoding = "utf-8" ill_formed_sequence = b"\xed\xb2\x80" @@ -987,7 +970,7 @@ def test_decode_error(self): self.assertEqual(data.decode(self.encoding, error_handler), expected) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_lone_surrogates(self): super().test_lone_surrogates() # not sure if this is making sense for @@ -1040,7 +1023,7 @@ def test_incremental_errors(self): class UTF7Test(ReadTest, unittest.TestCase): encoding = "utf-7" - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_ascii(self): # Set D (directly encoded characters) set_d = ('ABCDEFGHIJKLMNOPQRSTUVWXYZ' @@ -1067,7 +1050,7 @@ def test_ascii(self): b'+AAAAAQACAAMABAAFAAYABwAIAAsADAAOAA8AEAARABIAEwAU' b'ABUAFgAXABgAGQAaABsAHAAdAB4AHwBcAH4Afw-') - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: expected at least 5 arguments, got 1 def test_partial(self): self.check_partial( 'a+-b\x00c\x80d\u0100e\U00010000f', @@ -1107,7 +1090,7 @@ def test_partial(self): ] ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_errors(self): tests = [ (b'\xffb', '\ufffdb'), @@ -1138,7 +1121,7 @@ def test_errors(self): raw, 'strict', True) self.assertEqual(raw.decode('utf-7', 'replace'), expected) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_nonbmp(self): self.assertEqual('\U000104A0'.encode(self.encoding), b'+2AHcoA-') self.assertEqual('\ud801\udca0'.encode(self.encoding), b'+2AHcoA-') @@ -1154,7 +1137,7 @@ def test_nonbmp(self): self.assertEqual(b'+IKwgrNgB3KA'.decode(self.encoding), '\u20ac\u20ac\U000104A0') - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_lone_surrogates(self): tests = [ (b'a+2AE-b', 'a\ud801b'), @@ -1175,17 +1158,20 @@ def test_lone_surrogates(self): with self.subTest(raw=raw): self.assertEqual(raw.decode('utf-7', 'replace'), expected) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_bug1175396(self): return super().test_bug1175396() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: expected at least 5 arguments, got 1 + def test_readline(self): + return super().test_readline() + + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: utf_7_decode() takes from 1 to 2 positional arguments but 3 were given def test_incremental_surrogatepass(self): return super().test_incremental_surrogatepass() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_readline(self): - return super().test_readline() + + class UTF16ExTest(unittest.TestCase): @@ -1310,7 +1296,7 @@ def test_raw(self): if b != b'\\': self.assertEqual(decode(b + b'0'), (b + b'0', 2)) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; + (b'[]', 4) def test_escape(self): decode = codecs.escape_decode check = coding_checker(self, decode) @@ -1334,7 +1320,7 @@ def test_escape(self): check(br"[\x41]", b"[A]") check(br"[\x410]", b"[A0]") - @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_warnings(self): decode = codecs.escape_decode check = coding_checker(self, decode) @@ -1342,32 +1328,32 @@ def test_warnings(self): b = bytes([i]) if b not in b'abfnrtvx': with self.assertWarnsRegex(DeprecationWarning, - r"invalid escape sequence '\\%c'" % i): + r'"\\%c" is an invalid escape sequence' % i): check(b"\\" + b, b"\\" + b) with self.assertWarnsRegex(DeprecationWarning, - r"invalid escape sequence '\\%c'" % (i-32)): + r'"\\%c" is an invalid escape sequence' % (i-32)): check(b"\\" + b.upper(), b"\\" + b.upper()) with self.assertWarnsRegex(DeprecationWarning, - r"invalid escape sequence '\\8'"): + r'"\\8" is an invalid escape sequence'): check(br"\8", b"\\8") with self.assertWarns(DeprecationWarning): check(br"\9", b"\\9") with self.assertWarnsRegex(DeprecationWarning, - r"invalid escape sequence '\\\xfa'") as cm: + r'"\\\xfa" is an invalid escape sequence') as cm: check(b"\\\xfa", b"\\\xfa") for i in range(0o400, 0o1000): with self.assertWarnsRegex(DeprecationWarning, - r"invalid octal escape sequence '\\%o'" % i): + r'"\\%o" is an invalid octal escape sequence' % i): check(rb'\%o' % i, bytes([i & 0o377])) with self.assertWarnsRegex(DeprecationWarning, - r"invalid escape sequence '\\z'"): + r'"\\z" is an invalid escape sequence'): self.assertEqual(decode(br'\x\z', 'ignore'), (b'\\z', 4)) with self.assertWarnsRegex(DeprecationWarning, - r"invalid octal escape sequence '\\501'"): + r'"\\501" is an invalid octal escape sequence'): self.assertEqual(decode(br'\x\501', 'ignore'), (b'A', 6)) - @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: not raised by escape_decode + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: not raised by escape_decode def test_errors(self): decode = codecs.escape_decode self.assertRaises(ValueError, decode, br"\x") @@ -1999,9 +1985,9 @@ def test_all(self): def test_open(self): self.addCleanup(os_helper.unlink, os_helper.TESTFN) for mode in ('w', 'r', 'r+', 'w+', 'a', 'a+'): - with self.subTest(mode), \ - codecs.open(os_helper.TESTFN, mode, 'ascii') as file: - self.assertIsInstance(file, codecs.StreamReaderWriter) + with self.subTest(mode), self.assertWarns(DeprecationWarning): + with codecs.open(os_helper.TESTFN, mode, 'ascii') as file: + self.assertIsInstance(file, codecs.StreamReaderWriter) def test_undefined(self): self.assertRaises(UnicodeError, codecs.encode, 'abc', 'undefined') @@ -2018,7 +2004,7 @@ def test_file_closes_if_lookup_error_raised(self): mock_open = mock.mock_open() with mock.patch('builtins.open', mock_open) as file: with self.assertRaises(LookupError): - codecs.open(os_helper.TESTFN, 'wt', 'invalid-encoding') + codecs_open_no_warn(os_helper.TESTFN, 'wt', 'invalid-encoding') file().close.assert_called() @@ -2291,7 +2277,7 @@ def test_basic(self): class BasicUnicodeTest(unittest.TestCase, MixInCheckStateHandling): - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; LookupError: unknown encoding: big5 def test_basics(self): s = "abc123" # all codecs should be able to encode these for encoding in all_unicode_encodings: @@ -2411,7 +2397,7 @@ def test_basics_capi(self): self.assertEqual(decodedresult, s, "encoding=%r" % encoding) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; LookupError: unknown encoding: big5 def test_seek(self): # all codecs should be able to encode these s = "%s\n%s\n" % (100*"abc123", 100*"def456") @@ -2427,7 +2413,7 @@ def test_seek(self): data = reader.read() self.assertEqual(s, data) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; LookupError: unknown encoding: big5 def test_bad_decode_args(self): for encoding in all_unicode_encodings: decoder = codecs.getdecoder(encoding) @@ -2435,7 +2421,7 @@ def test_bad_decode_args(self): if encoding not in ("idna", "punycode"): self.assertRaises(TypeError, decoder, 42) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; LookupError: unknown encoding: big5 def test_bad_encode_args(self): for encoding in all_unicode_encodings: encoder = codecs.getencoder(encoding) @@ -2447,7 +2433,7 @@ def test_encoding_map_type_initialized(self): table_type = type(cp1140.encoding_table) self.assertEqual(table_type, table_type) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; LookupError: unknown encoding: big5 def test_decoder_state(self): # Check that getstate() and setstate() handle the state properly u = "abc123" @@ -2458,7 +2444,7 @@ def test_decoder_state(self): class CharmapTest(unittest.TestCase): - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; IndexError: index out of range def test_decode_with_string_map(self): self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "strict", "abc"), @@ -2514,7 +2500,7 @@ def test_decode_with_string_map(self): ("", len(allbytes)) ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: UnicodeDecodeError not raised by charmap_decode def test_decode_with_int2str_map(self): self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "strict", @@ -2631,7 +2617,7 @@ def test_decode_with_int2str_map(self): b"\x00\x01\x02", "strict", {0: "A", 1: 'Bb', 2: 999999999} ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: character mapping must be in range(65536) def test_decode_with_int2int_map(self): a = ord('a') b = ord('b') @@ -2724,7 +2710,7 @@ def test_streamreaderwriter(self): class TypesTest(unittest.TestCase): - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'codecs' has no attribute 'utf_32_ex_decode'. Did you mean: 'utf_16_ex_decode'? def test_decode_unicode(self): # Most decoders don't accept unicode input decoders = [ @@ -2826,7 +2812,7 @@ def test_escape_decode(self): check(br"\u20ac", "\u20ac") check(br"\U0001d120", "\U0001d120") - @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_decode_warnings(self): decode = codecs.unicode_escape_decode check = coding_checker(self, decode) @@ -2834,30 +2820,30 @@ def test_decode_warnings(self): b = bytes([i]) if b not in b'abfnrtuvx': with self.assertWarnsRegex(DeprecationWarning, - r"invalid escape sequence '\\%c'" % i): + r'"\\%c" is an invalid escape sequence' % i): check(b"\\" + b, "\\" + chr(i)) if b.upper() not in b'UN': with self.assertWarnsRegex(DeprecationWarning, - r"invalid escape sequence '\\%c'" % (i-32)): + r'"\\%c" is an invalid escape sequence' % (i-32)): check(b"\\" + b.upper(), "\\" + chr(i-32)) with self.assertWarnsRegex(DeprecationWarning, - r"invalid escape sequence '\\8'"): + r'"\\8" is an invalid escape sequence'): check(br"\8", "\\8") with self.assertWarns(DeprecationWarning): check(br"\9", "\\9") with self.assertWarnsRegex(DeprecationWarning, - r"invalid escape sequence '\\\xfa'") as cm: + r'"\\\xfa" is an invalid escape sequence') as cm: check(b"\\\xfa", "\\\xfa") for i in range(0o400, 0o1000): with self.assertWarnsRegex(DeprecationWarning, - r"invalid octal escape sequence '\\%o'" % i): + r'"\\%o" is an invalid octal escape sequence' % i): check(rb'\%o' % i, chr(i)) with self.assertWarnsRegex(DeprecationWarning, - r"invalid escape sequence '\\z'"): + r'"\\z" is an invalid escape sequence'): self.assertEqual(decode(br'\x\z', 'ignore'), ('\\z', 4)) with self.assertWarnsRegex(DeprecationWarning, - r"invalid octal escape sequence '\\501'"): + r'"\\501" is an invalid octal escape sequence'): self.assertEqual(decode(br'\x\501', 'ignore'), ('\u0141', 6)) def test_decode_errors(self): @@ -2876,7 +2862,7 @@ def test_decode_errors(self): self.assertEqual(decode(br"\U00110000", "ignore"), ("", 10)) self.assertEqual(decode(br"\U00110000", "replace"), ("\ufffd", 10)) - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: '\x00\t\n\r\\' != '\x00\t\n\r' + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: '\x00\t\n\r\\' != '\x00\t\n\r' def test_partial(self): self.check_partial( "\x00\t\n\r\\\xff\uffff\U00010000", @@ -2916,14 +2902,16 @@ def test_partial(self): ] ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; IndexError: index out of range def test_incremental_surrogatepass(self): return super().test_incremental_surrogatepass() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; UnicodeDecodeError: 'unicodeescape' codec can't decode bytes in position 72-75: truncated \uXXXX escape def test_readline(self): return super().test_readline() + + class RawUnicodeEscapeTest(ReadTest, unittest.TestCase): encoding = "raw-unicode-escape" @@ -2977,7 +2965,7 @@ def test_decode_errors(self): self.assertEqual(decode(br"\U00110000", "ignore"), ("", 10)) self.assertEqual(decode(br"\U00110000", "replace"), ("\ufffd", 10)) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; - \ def test_partial(self): self.check_partial( "\x00\t\n\r\\\xff\uffff\U00010000", @@ -3007,15 +2995,17 @@ def test_partial(self): ] ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; - \ def test_incremental_surrogatepass(self): return super().test_incremental_surrogatepass() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; UnicodeDecodeError: 'rawunicodeescape' codec can't decode bytes in position 72-76: truncated \uXXXX def test_readline(self): return super().test_readline() + + class EscapeEncodeTest(unittest.TestCase): def test_escape_encode(self): @@ -3057,7 +3047,7 @@ def test_ascii(self): self.assertEqual("foo\udc80bar".encode("ascii", "surrogateescape"), b"foo\x80bar") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; Result: FAILURE def test_charmap(self): # bad byte: \xa5 is unmapped in iso-8859-3 self.assertEqual(b"foo\xa5bar".decode("iso-8859-3", "surrogateescape"), @@ -3072,7 +3062,7 @@ def test_latin1(self): class BomTest(unittest.TestCase): - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'codecs' has no attribute 'utf_32_ex_decode'. Did you mean: 'utf_16_ex_decode'? def test_seek0(self): data = "1234567890" tests = ("utf-16", @@ -3084,7 +3074,7 @@ def test_seek0(self): self.addCleanup(os_helper.unlink, os_helper.TESTFN) for encoding in tests: # Check if the BOM is written only once - with codecs.open(os_helper.TESTFN, 'w+', encoding=encoding) as f: + with codecs_open_no_warn(os_helper.TESTFN, 'w+', encoding=encoding) as f: f.write(data) f.write(data) f.seek(0) @@ -3093,7 +3083,7 @@ def test_seek0(self): self.assertEqual(f.read(), data * 2) # Check that the BOM is written after a seek(0) - with codecs.open(os_helper.TESTFN, 'w+', encoding=encoding) as f: + with codecs_open_no_warn(os_helper.TESTFN, 'w+', encoding=encoding) as f: f.write(data[0]) self.assertNotEqual(f.tell(), 0) f.seek(0) @@ -3102,7 +3092,7 @@ def test_seek0(self): self.assertEqual(f.read(), data) # (StreamWriter) Check that the BOM is written after a seek(0) - with codecs.open(os_helper.TESTFN, 'w+', encoding=encoding) as f: + with codecs_open_no_warn(os_helper.TESTFN, 'w+', encoding=encoding) as f: f.writer.write(data[0]) self.assertNotEqual(f.writer.tell(), 0) f.writer.seek(0) @@ -3112,7 +3102,7 @@ def test_seek0(self): # Check that the BOM is not written after a seek() at a position # different than the start - with codecs.open(os_helper.TESTFN, 'w+', encoding=encoding) as f: + with codecs_open_no_warn(os_helper.TESTFN, 'w+', encoding=encoding) as f: f.write(data) f.seek(f.tell()) f.write(data) @@ -3121,7 +3111,7 @@ def test_seek0(self): # (StreamWriter) Check that the BOM is not written after a seek() # at a position different than the start - with codecs.open(os_helper.TESTFN, 'w+', encoding=encoding) as f: + with codecs_open_no_warn(os_helper.TESTFN, 'w+', encoding=encoding) as f: f.writer.write(data) f.writer.seek(f.writer.tell()) f.writer.write(data) @@ -3152,7 +3142,7 @@ def test_seek0(self): bytes_transform_encodings.append("zlib_codec") transform_aliases["zlib_codec"] = ["zip", "zlib"] try: - import bz2 + import bz2 # noqa: F401 except ImportError: pass else: @@ -3251,7 +3241,7 @@ def test_binary_to_text_denylists_text_transforms(self): bad_input.decode("rot_13") self.assertIsNone(failure.exception.__cause__) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'error' object has no attribute '__notes__'. Did you mean: '__ne__'? @unittest.skipUnless(zlib, "Requires zlib support") def test_custom_zlib_error_is_noted(self): # Check zlib codec gives a good error for malformed input @@ -3260,7 +3250,7 @@ def test_custom_zlib_error_is_noted(self): codecs.decode(b"hello", "zlib_codec") self.assertEqual(msg, failure.exception.__notes__[0]) - @unittest.expectedFailure # TODO: RUSTPYTHON; - AttributeError: 'Error' object has no attribute '__notes__' + @unittest.expectedFailure # TODO: RUSTPYTHON; - AttributeError: 'Error' object has no attribute '__notes__' def test_custom_hex_error_is_noted(self): # Check hex codec gives a good error for malformed input import binascii @@ -3278,6 +3268,13 @@ def test_aliases(self): info = codecs.lookup(alias) self.assertEqual(info.name, expected_name) + def test_alias_modules_exist(self): + encodings_dir = os.path.dirname(encodings.__file__) + for value in encodings.aliases.aliases.values(): + codec_mod = f"encodings.{value}" + self.assertIsNotNone(importlib.util.find_spec(codec_mod), + f"Codec module not found: {codec_mod}") + def test_quopri_stateless(self): # Should encode with quotetabs=True encoded = codecs.encode(b"space tab\teol \n", "quopri-codec") @@ -3341,7 +3338,6 @@ def raise_obj(self, *args, **kwds): # Helper to dynamically change the object raised by a test codec raise self.obj_to_raise - @unittest.expectedFailure # TODO: RUSTPYTHON def check_note(self, obj_to_raise, msg, exc_type=RuntimeError): self.obj_to_raise = obj_to_raise self.set_codec(self.raise_obj, self.raise_obj) @@ -3354,55 +3350,55 @@ def check_note(self, obj_to_raise, msg, exc_type=RuntimeError): with self.assertNoted("decoding", exc_type, msg): codecs.decode(b"bytes input", self.codec_name) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'RuntimeError' object has no attribute '__notes__'. Did you mean: '__ne__'? def test_raise_by_type(self): self.check_note(RuntimeError, "") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'RuntimeError' object has no attribute '__notes__'. Did you mean: '__ne__'? def test_raise_by_value(self): msg = "This should be noted" self.check_note(RuntimeError(msg), msg) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'MyRuntimeError' object has no attribute '__notes__'. Did you mean: '__ne__'? def test_raise_grandchild_subclass_exact_size(self): msg = "This should be noted" class MyRuntimeError(RuntimeError): __slots__ = () self.check_note(MyRuntimeError(msg), msg, MyRuntimeError) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'MyRuntimeError' object has no attribute '__notes__'. Did you mean: '__ne__'? def test_raise_subclass_with_weakref_support(self): msg = "This should be noted" class MyRuntimeError(RuntimeError): pass self.check_note(MyRuntimeError(msg), msg, MyRuntimeError) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'CustomInit' object has no attribute '__notes__'. Did you mean: '__ne__'? def test_init_override(self): class CustomInit(RuntimeError): def __init__(self): pass self.check_note(CustomInit, "") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'CustomNew' object has no attribute '__notes__'. Did you mean: '__ne__'? def test_new_override(self): class CustomNew(RuntimeError): def __new__(cls): return super().__new__(cls) self.check_note(CustomNew, "") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'RuntimeError' object has no attribute '__notes__'. Did you mean: '__ne__'? def test_instance_attribute(self): msg = "This should be noted" exc = RuntimeError(msg) exc.attr = 1 self.check_note(exc, "^{}$".format(msg)) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'RuntimeError' object has no attribute '__notes__'. Did you mean: '__ne__'? def test_non_str_arg(self): self.check_note(RuntimeError(1), "1") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'RuntimeError' object has no attribute '__notes__'. Did you mean: '__ne__'? def test_multiple_args(self): msg_re = r"^\('a', 'b', 'c'\)$" self.check_note(RuntimeError('a', 'b', 'c'), msg_re) @@ -3419,7 +3415,7 @@ def test_codec_lookup_failure(self): with self.assertRaisesRegex(LookupError, msg): codecs.decode(b"bytes input", self.codec_name) - @unittest.expectedFailure # TODO: RUSTPYTHON; + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: "^'exception_notes_test' encoder returned 'str' instead of 'bytes'; use codecs.encode\(\) to encode to arbitrary types$" does not match "'exception_notes_test' encoder returned 'str' instead of 'bytes'; use codecs.encode() to encode arbitrary types" def test_unflagged_non_text_codec_handling(self): # The stdlib non-text codecs are now marked so they're # pre-emptively skipped by the text model related methods @@ -3472,7 +3468,11 @@ def test_code_page_name(self): codecs.code_page_decode, self.CP_UTF8, b'\xff', 'strict', True) def check_decode(self, cp, tests): - for raw, errors, expected in tests: + for raw, errors, expected, *rest in tests: + if rest: + altexpected, = rest + else: + altexpected = expected if expected is not None: try: decoded = codecs.code_page_decode(cp, raw, errors, True) @@ -3489,8 +3489,21 @@ def check_decode(self, cp, tests): self.assertRaises(UnicodeDecodeError, codecs.code_page_decode, cp, raw, errors, True) + if altexpected is not None: + decoded = raw.decode(f'cp{cp}', errors) + self.assertEqual(decoded, altexpected, + '%a.decode("cp%s", %r)=%a != %a' + % (raw, cp, errors, decoded, altexpected)) + else: + self.assertRaises(UnicodeDecodeError, + raw.decode, f'cp{cp}', errors) + def check_encode(self, cp, tests): - for text, errors, expected in tests: + for text, errors, expected, *rest in tests: + if rest: + altexpected, = rest + else: + altexpected = expected if expected is not None: try: encoded = codecs.code_page_encode(cp, text, errors) @@ -3501,19 +3514,27 @@ def check_encode(self, cp, tests): '%a.encode("cp%s", %r)=%a != %a' % (text, cp, errors, encoded[0], expected)) self.assertEqual(encoded[1], len(text)) + + encoded = text.encode(f'cp{cp}', errors) + self.assertEqual(encoded, altexpected, + '%a.encode("cp%s", %r)=%a != %a' + % (text, cp, errors, encoded, altexpected)) else: self.assertRaises(UnicodeEncodeError, codecs.code_page_encode, cp, text, errors) + self.assertRaises(UnicodeEncodeError, + text.encode, f'cp{cp}', errors) @unittest.expectedFailure # TODO: RUSTPYTHON def test_cp932(self): self.check_encode(932, ( ('abc', 'strict', b'abc'), ('\uff44\u9a3e', 'strict', b'\x82\x84\xe9\x80'), + ('\uf8f3', 'strict', b'\xff'), # test error handlers ('\xff', 'strict', None), ('[\xff]', 'ignore', b'[]'), - ('[\xff]', 'replace', b'[y]'), + ('[\xff]', 'replace', b'[y]', b'[?]'), ('[\u20ac]', 'replace', b'[?]'), ('[\xff]', 'backslashreplace', b'[\\xff]'), ('[\xff]', 'namereplace', @@ -3527,12 +3548,12 @@ def test_cp932(self): (b'abc', 'strict', 'abc'), (b'\x82\x84\xe9\x80', 'strict', '\uff44\u9a3e'), # invalid bytes - (b'[\xff]', 'strict', None), - (b'[\xff]', 'ignore', '[]'), - (b'[\xff]', 'replace', '[\ufffd]'), - (b'[\xff]', 'backslashreplace', '[\\xff]'), - (b'[\xff]', 'surrogateescape', '[\udcff]'), - (b'[\xff]', 'surrogatepass', None), + (b'[\xff]', 'strict', None, '[\uf8f3]'), + (b'[\xff]', 'ignore', '[]', '[\uf8f3]'), + (b'[\xff]', 'replace', '[\ufffd]', '[\uf8f3]'), + (b'[\xff]', 'backslashreplace', '[\\xff]', '[\uf8f3]'), + (b'[\xff]', 'surrogateescape', '[\udcff]', '[\uf8f3]'), + (b'[\xff]', 'surrogatepass', None, '[\uf8f3]'), (b'\x81\x00abc', 'strict', None), (b'\x81\x00abc', 'ignore', '\x00abc'), (b'\x81\x00abc', 'replace', '\ufffd\x00abc'), @@ -3548,7 +3569,7 @@ def test_cp1252(self): # test error handlers ('\u0141', 'strict', None), ('\u0141', 'ignore', b''), - ('\u0141', 'replace', b'L'), + ('\u0141', 'replace', b'L', b'?'), ('\udc98', 'surrogateescape', b'\x98'), ('\udc98', 'surrogatepass', None), )) @@ -3558,6 +3579,61 @@ def test_cp1252(self): (b'\xff', 'strict', '\xff'), )) + @unittest.expectedFailureIfWindows("TODO: RUSTPYTHON") + def test_cp708(self): + self.check_encode(708, ( + ('abc2%', 'strict', b'abc2%'), + ('\u060c\u0621\u064a', 'strict', b'\xac\xc1\xea'), + ('\u2562\xe7\xa0', 'strict', b'\x86\x87\xff'), + ('\x9a\x9f', 'strict', b'\x9a\x9f'), + ('\u256b', 'strict', b'\xc0'), + # test error handlers + ('[\u0662]', 'strict', None), + ('[\u0662]', 'ignore', b'[]'), + ('[\u0662]', 'replace', b'[?]'), + ('\udca0', 'surrogateescape', b'\xa0'), + ('\udca0', 'surrogatepass', None), + )) + self.check_decode(708, ( + (b'abc2%', 'strict', 'abc2%'), + (b'\xac\xc1\xea', 'strict', '\u060c\u0621\u064a'), + (b'\x86\x87\xff', 'strict', '\u2562\xe7\xa0'), + (b'\x9a\x9f', 'strict', '\x9a\x9f'), + (b'\xc0', 'strict', '\u256b'), + # test error handlers + (b'\xa0', 'strict', None), + (b'[\xa0]', 'ignore', '[]'), + (b'[\xa0]', 'replace', '[\ufffd]'), + (b'[\xa0]', 'backslashreplace', '[\\xa0]'), + (b'[\xa0]', 'surrogateescape', '[\udca0]'), + (b'[\xa0]', 'surrogatepass', None), + )) + + @unittest.expectedFailureIfWindows("TODO: RUSTPYTHON") + def test_cp20106(self): + self.check_encode(20106, ( + ('abc', 'strict', b'abc'), + ('\xa7\xc4\xdf', 'strict', b'@[~'), + # test error handlers + ('@', 'strict', None), + ('@', 'ignore', b''), + ('@', 'replace', b'?'), + ('\udcbf', 'surrogateescape', b'\xbf'), + ('\udcbf', 'surrogatepass', None), + )) + self.check_decode(20106, ( + (b'abc', 'strict', 'abc'), + (b'@[~', 'strict', '\xa7\xc4\xdf'), + (b'\xe1\xfe', 'strict', 'a\xdf'), + # test error handlers + (b'(\xbf)', 'strict', None), + (b'(\xbf)', 'ignore', '()'), + (b'(\xbf)', 'replace', '(\ufffd)'), + (b'(\xbf)', 'backslashreplace', '(\\xbf)'), + (b'(\xbf)', 'surrogateescape', '(\udcbf)'), + (b'(\xbf)', 'surrogatepass', None), + )) + @unittest.expectedFailure # TODO: RUSTPYTHON def test_cp_utf7(self): cp = 65000 @@ -3634,17 +3710,15 @@ def test_incremental(self): False) self.assertEqual(decoded, ('abc', 3)) - def test_mbcs_alias(self): - # Check that looking up our 'default' codepage will return - # mbcs when we don't have a more specific one available - code_page = 99_999 - name = f'cp{code_page}' - with mock.patch('_winapi.GetACP', return_value=code_page): - try: - codec = codecs.lookup(name) - self.assertEqual(codec.name, 'mbcs') - finally: - codecs.unregister(name) + def test_mbcs_code_page(self): + # Check that codec for the current Windows (ANSII) code page is + # always available. + try: + from _winapi import GetACP + except ImportError: + self.skipTest('requires _winapi.GetACP') + cp = GetACP() + codecs.lookup(f'cp{cp}') @support.bigmemtest(size=2**31, memuse=7, dry_run=False) def test_large_input(self, size): @@ -3908,7 +3982,7 @@ def check_decode_strings(self, errors): with self.assertRaises(RuntimeError) as cm: self.decode(encoded, errors) errmsg = str(cm.exception) - self.assertTrue(errmsg.startswith("decode error: "), errmsg) + self.assertStartsWith(errmsg, "decode error: ") else: decoded = self.decode(encoded, errors) self.assertEqual(decoded, expected) @@ -3977,7 +4051,7 @@ def test_rot13_func(self): class CodecNameNormalizationTest(unittest.TestCase): """Test codec name normalization""" - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: Tuples differ: (1, 2, 3, 4) != (None, None, None, None) + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: Tuples differ: (1, 2, 3, 4) != (None, None, None, None) def test_codecs_lookup(self): FOUND = (1, 2, 3, 4) NOT_FOUND = (None, None, None, None) diff --git a/Lib/test/test_collections.py b/Lib/test/test_collections.py index 7b11601fc46..b68305dd7aa 100644 --- a/Lib/test/test_collections.py +++ b/Lib/test/test_collections.py @@ -469,7 +469,6 @@ def test_module_parameter(self): NT = namedtuple('NT', ['x', 'y'], module=collections) self.assertEqual(NT.__module__, collections) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_instance(self): Point = namedtuple('Point', 'x y') p = Point(11, 22) @@ -746,11 +745,11 @@ def validate_isinstance(self, abc, name): C = type('C', (object,), {'__hash__': None}) setattr(C, name, stub) self.assertIsInstance(C(), abc) - self.assertTrue(issubclass(C, abc)) + self.assertIsSubclass(C, abc) C = type('C', (object,), {'__hash__': None}) self.assertNotIsInstance(C(), abc) - self.assertFalse(issubclass(C, abc)) + self.assertNotIsSubclass(C, abc) def validate_comparison(self, instance): ops = ['lt', 'gt', 'le', 'ge', 'ne', 'or', 'and', 'xor', 'sub'] @@ -788,7 +787,6 @@ def _test_gen(): class TestOneTrickPonyABCs(ABCTestCase): - @unittest.expectedFailure # TODO: RUSTPYTHON def test_Awaitable(self): def gen(): yield @@ -817,12 +815,12 @@ def __await__(self): non_samples = [None, int(), gen(), object()] for x in non_samples: self.assertNotIsInstance(x, Awaitable) - self.assertFalse(issubclass(type(x), Awaitable), repr(type(x))) + self.assertNotIsSubclass(type(x), Awaitable) samples = [Bar(), MinimalCoro()] for x in samples: self.assertIsInstance(x, Awaitable) - self.assertTrue(issubclass(type(x), Awaitable)) + self.assertIsSubclass(type(x), Awaitable) c = coro() # Iterable coroutines (generators with CO_ITERABLE_COROUTINE @@ -836,12 +834,11 @@ def __await__(self): class CoroLike: pass Coroutine.register(CoroLike) - self.assertTrue(isinstance(CoroLike(), Awaitable)) - self.assertTrue(issubclass(CoroLike, Awaitable)) + self.assertIsInstance(CoroLike(), Awaitable) + self.assertIsSubclass(CoroLike, Awaitable) CoroLike = None support.gc_collect() # Kill CoroLike to clean-up ABCMeta cache - @unittest.expectedFailure # TODO: RUSTPYTHON def test_Coroutine(self): def gen(): yield @@ -870,12 +867,12 @@ def __await__(self): non_samples = [None, int(), gen(), object(), Bar()] for x in non_samples: self.assertNotIsInstance(x, Coroutine) - self.assertFalse(issubclass(type(x), Coroutine), repr(type(x))) + self.assertNotIsSubclass(type(x), Coroutine) samples = [MinimalCoro()] for x in samples: self.assertIsInstance(x, Awaitable) - self.assertTrue(issubclass(type(x), Awaitable)) + self.assertIsSubclass(type(x), Awaitable) c = coro() # Iterable coroutines (generators with CO_ITERABLE_COROUTINE @@ -896,8 +893,8 @@ def close(self): pass def __await__(self): pass - self.assertTrue(isinstance(CoroLike(), Coroutine)) - self.assertTrue(issubclass(CoroLike, Coroutine)) + self.assertIsInstance(CoroLike(), Coroutine) + self.assertIsSubclass(CoroLike, Coroutine) class CoroLike: def send(self, value): @@ -906,15 +903,15 @@ def close(self): pass def __await__(self): pass - self.assertFalse(isinstance(CoroLike(), Coroutine)) - self.assertFalse(issubclass(CoroLike, Coroutine)) + self.assertNotIsInstance(CoroLike(), Coroutine) + self.assertNotIsSubclass(CoroLike, Coroutine) def test_Hashable(self): # Check some non-hashables non_samples = [bytearray(), list(), set(), dict()] for x in non_samples: self.assertNotIsInstance(x, Hashable) - self.assertFalse(issubclass(type(x), Hashable), repr(type(x))) + self.assertNotIsSubclass(type(x), Hashable) # Check some hashables samples = [None, int(), float(), complex(), @@ -924,14 +921,14 @@ def test_Hashable(self): ] for x in samples: self.assertIsInstance(x, Hashable) - self.assertTrue(issubclass(type(x), Hashable), repr(type(x))) + self.assertIsSubclass(type(x), Hashable) self.assertRaises(TypeError, Hashable) # Check direct subclassing class H(Hashable): def __hash__(self): return super().__hash__() self.assertEqual(hash(H()), 0) - self.assertFalse(issubclass(int, H)) + self.assertNotIsSubclass(int, H) self.validate_abstract_methods(Hashable, '__hash__') self.validate_isinstance(Hashable, '__hash__') @@ -939,13 +936,13 @@ def test_AsyncIterable(self): class AI: def __aiter__(self): return self - self.assertTrue(isinstance(AI(), AsyncIterable)) - self.assertTrue(issubclass(AI, AsyncIterable)) + self.assertIsInstance(AI(), AsyncIterable) + self.assertIsSubclass(AI, AsyncIterable) # Check some non-iterables non_samples = [None, object, []] for x in non_samples: self.assertNotIsInstance(x, AsyncIterable) - self.assertFalse(issubclass(type(x), AsyncIterable), repr(type(x))) + self.assertNotIsSubclass(type(x), AsyncIterable) self.validate_abstract_methods(AsyncIterable, '__aiter__') self.validate_isinstance(AsyncIterable, '__aiter__') @@ -955,13 +952,13 @@ def __aiter__(self): return self async def __anext__(self): raise StopAsyncIteration - self.assertTrue(isinstance(AI(), AsyncIterator)) - self.assertTrue(issubclass(AI, AsyncIterator)) + self.assertIsInstance(AI(), AsyncIterator) + self.assertIsSubclass(AI, AsyncIterator) non_samples = [None, object, []] # Check some non-iterables for x in non_samples: self.assertNotIsInstance(x, AsyncIterator) - self.assertFalse(issubclass(type(x), AsyncIterator), repr(type(x))) + self.assertNotIsSubclass(type(x), AsyncIterator) # Similarly to regular iterators (see issue 10565) class AnextOnly: async def __anext__(self): @@ -974,7 +971,7 @@ def test_Iterable(self): non_samples = [None, 42, 3.14, 1j] for x in non_samples: self.assertNotIsInstance(x, Iterable) - self.assertFalse(issubclass(type(x), Iterable), repr(type(x))) + self.assertNotIsSubclass(type(x), Iterable) # Check some iterables samples = [bytes(), str(), tuple(), list(), set(), frozenset(), dict(), @@ -984,13 +981,13 @@ def test_Iterable(self): ] for x in samples: self.assertIsInstance(x, Iterable) - self.assertTrue(issubclass(type(x), Iterable), repr(type(x))) + self.assertIsSubclass(type(x), Iterable) # Check direct subclassing class I(Iterable): def __iter__(self): return super().__iter__() self.assertEqual(list(I()), []) - self.assertFalse(issubclass(str, I)) + self.assertNotIsSubclass(str, I) self.validate_abstract_methods(Iterable, '__iter__') self.validate_isinstance(Iterable, '__iter__') # Check None blocking @@ -998,22 +995,22 @@ class It: def __iter__(self): return iter([]) class ItBlocked(It): __iter__ = None - self.assertTrue(issubclass(It, Iterable)) - self.assertTrue(isinstance(It(), Iterable)) - self.assertFalse(issubclass(ItBlocked, Iterable)) - self.assertFalse(isinstance(ItBlocked(), Iterable)) + self.assertIsSubclass(It, Iterable) + self.assertIsInstance(It(), Iterable) + self.assertNotIsSubclass(ItBlocked, Iterable) + self.assertNotIsInstance(ItBlocked(), Iterable) def test_Reversible(self): # Check some non-reversibles non_samples = [None, 42, 3.14, 1j, set(), frozenset()] for x in non_samples: self.assertNotIsInstance(x, Reversible) - self.assertFalse(issubclass(type(x), Reversible), repr(type(x))) + self.assertNotIsSubclass(type(x), Reversible) # Check some non-reversible iterables non_reversibles = [_test_gen(), (x for x in []), iter([]), reversed([])] for x in non_reversibles: self.assertNotIsInstance(x, Reversible) - self.assertFalse(issubclass(type(x), Reversible), repr(type(x))) + self.assertNotIsSubclass(type(x), Reversible) # Check some reversible iterables samples = [bytes(), str(), tuple(), list(), OrderedDict(), OrderedDict().keys(), OrderedDict().items(), @@ -1022,11 +1019,11 @@ def test_Reversible(self): dict().keys(), dict().items(), dict().values()] for x in samples: self.assertIsInstance(x, Reversible) - self.assertTrue(issubclass(type(x), Reversible), repr(type(x))) + self.assertIsSubclass(type(x), Reversible) # Check also Mapping, MutableMapping, and Sequence - self.assertTrue(issubclass(Sequence, Reversible), repr(Sequence)) - self.assertFalse(issubclass(Mapping, Reversible), repr(Mapping)) - self.assertFalse(issubclass(MutableMapping, Reversible), repr(MutableMapping)) + self.assertIsSubclass(Sequence, Reversible) + self.assertNotIsSubclass(Mapping, Reversible) + self.assertNotIsSubclass(MutableMapping, Reversible) # Check direct subclassing class R(Reversible): def __iter__(self): @@ -1034,17 +1031,17 @@ def __iter__(self): def __reversed__(self): return iter(list()) self.assertEqual(list(reversed(R())), []) - self.assertFalse(issubclass(float, R)) + self.assertNotIsSubclass(float, R) self.validate_abstract_methods(Reversible, '__reversed__', '__iter__') # Check reversible non-iterable (which is not Reversible) class RevNoIter: def __reversed__(self): return reversed([]) class RevPlusIter(RevNoIter): def __iter__(self): return iter([]) - self.assertFalse(issubclass(RevNoIter, Reversible)) - self.assertFalse(isinstance(RevNoIter(), Reversible)) - self.assertTrue(issubclass(RevPlusIter, Reversible)) - self.assertTrue(isinstance(RevPlusIter(), Reversible)) + self.assertNotIsSubclass(RevNoIter, Reversible) + self.assertNotIsInstance(RevNoIter(), Reversible) + self.assertIsSubclass(RevPlusIter, Reversible) + self.assertIsInstance(RevPlusIter(), Reversible) # Check None blocking class Rev: def __iter__(self): return iter([]) @@ -1053,39 +1050,38 @@ class RevItBlocked(Rev): __iter__ = None class RevRevBlocked(Rev): __reversed__ = None - self.assertTrue(issubclass(Rev, Reversible)) - self.assertTrue(isinstance(Rev(), Reversible)) - self.assertFalse(issubclass(RevItBlocked, Reversible)) - self.assertFalse(isinstance(RevItBlocked(), Reversible)) - self.assertFalse(issubclass(RevRevBlocked, Reversible)) - self.assertFalse(isinstance(RevRevBlocked(), Reversible)) + self.assertIsSubclass(Rev, Reversible) + self.assertIsInstance(Rev(), Reversible) + self.assertNotIsSubclass(RevItBlocked, Reversible) + self.assertNotIsInstance(RevItBlocked(), Reversible) + self.assertNotIsSubclass(RevRevBlocked, Reversible) + self.assertNotIsInstance(RevRevBlocked(), Reversible) def test_Collection(self): # Check some non-collections non_collections = [None, 42, 3.14, 1j, lambda x: 2*x] for x in non_collections: self.assertNotIsInstance(x, Collection) - self.assertFalse(issubclass(type(x), Collection), repr(type(x))) + self.assertNotIsSubclass(type(x), Collection) # Check some non-collection iterables non_col_iterables = [_test_gen(), iter(b''), iter(bytearray()), (x for x in [])] for x in non_col_iterables: self.assertNotIsInstance(x, Collection) - self.assertFalse(issubclass(type(x), Collection), repr(type(x))) + self.assertNotIsSubclass(type(x), Collection) # Check some collections samples = [set(), frozenset(), dict(), bytes(), str(), tuple(), list(), dict().keys(), dict().items(), dict().values()] for x in samples: self.assertIsInstance(x, Collection) - self.assertTrue(issubclass(type(x), Collection), repr(type(x))) + self.assertIsSubclass(type(x), Collection) # Check also Mapping, MutableMapping, etc. - self.assertTrue(issubclass(Sequence, Collection), repr(Sequence)) - self.assertTrue(issubclass(Mapping, Collection), repr(Mapping)) - self.assertTrue(issubclass(MutableMapping, Collection), - repr(MutableMapping)) - self.assertTrue(issubclass(Set, Collection), repr(Set)) - self.assertTrue(issubclass(MutableSet, Collection), repr(MutableSet)) - self.assertTrue(issubclass(Sequence, Collection), repr(MutableSet)) + self.assertIsSubclass(Sequence, Collection) + self.assertIsSubclass(Mapping, Collection) + self.assertIsSubclass(MutableMapping, Collection) + self.assertIsSubclass(Set, Collection) + self.assertIsSubclass(MutableSet, Collection) + self.assertIsSubclass(Sequence, Collection) # Check direct subclassing class Col(Collection): def __iter__(self): @@ -1096,13 +1092,13 @@ def __contains__(self, item): return False class DerCol(Col): pass self.assertEqual(list(iter(Col())), []) - self.assertFalse(issubclass(list, Col)) - self.assertFalse(issubclass(set, Col)) - self.assertFalse(issubclass(float, Col)) + self.assertNotIsSubclass(list, Col) + self.assertNotIsSubclass(set, Col) + self.assertNotIsSubclass(float, Col) self.assertEqual(list(iter(DerCol())), []) - self.assertFalse(issubclass(list, DerCol)) - self.assertFalse(issubclass(set, DerCol)) - self.assertFalse(issubclass(float, DerCol)) + self.assertNotIsSubclass(list, DerCol) + self.assertNotIsSubclass(set, DerCol) + self.assertNotIsSubclass(float, DerCol) self.validate_abstract_methods(Collection, '__len__', '__iter__', '__contains__') # Check sized container non-iterable (which is not Collection) etc. @@ -1115,12 +1111,12 @@ def __contains__(self, item): return False class ColNoCont: def __iter__(self): return iter([]) def __len__(self): return 0 - self.assertFalse(issubclass(ColNoIter, Collection)) - self.assertFalse(isinstance(ColNoIter(), Collection)) - self.assertFalse(issubclass(ColNoSize, Collection)) - self.assertFalse(isinstance(ColNoSize(), Collection)) - self.assertFalse(issubclass(ColNoCont, Collection)) - self.assertFalse(isinstance(ColNoCont(), Collection)) + self.assertNotIsSubclass(ColNoIter, Collection) + self.assertNotIsInstance(ColNoIter(), Collection) + self.assertNotIsSubclass(ColNoSize, Collection) + self.assertNotIsInstance(ColNoSize(), Collection) + self.assertNotIsSubclass(ColNoCont, Collection) + self.assertNotIsInstance(ColNoCont(), Collection) # Check None blocking class SizeBlock: def __iter__(self): return iter([]) @@ -1130,10 +1126,10 @@ class IterBlock: def __len__(self): return 0 def __contains__(self): return True __iter__ = None - self.assertFalse(issubclass(SizeBlock, Collection)) - self.assertFalse(isinstance(SizeBlock(), Collection)) - self.assertFalse(issubclass(IterBlock, Collection)) - self.assertFalse(isinstance(IterBlock(), Collection)) + self.assertNotIsSubclass(SizeBlock, Collection) + self.assertNotIsInstance(SizeBlock(), Collection) + self.assertNotIsSubclass(IterBlock, Collection) + self.assertNotIsInstance(IterBlock(), Collection) # Check None blocking in subclass class ColImpl: def __iter__(self): @@ -1144,15 +1140,15 @@ def __contains__(self, item): return False class NonCol(ColImpl): __contains__ = None - self.assertFalse(issubclass(NonCol, Collection)) - self.assertFalse(isinstance(NonCol(), Collection)) + self.assertNotIsSubclass(NonCol, Collection) + self.assertNotIsInstance(NonCol(), Collection) def test_Iterator(self): non_samples = [None, 42, 3.14, 1j, b"", "", (), [], {}, set()] for x in non_samples: self.assertNotIsInstance(x, Iterator) - self.assertFalse(issubclass(type(x), Iterator), repr(type(x))) + self.assertNotIsSubclass(type(x), Iterator) samples = [iter(bytes()), iter(str()), iter(tuple()), iter(list()), iter(dict()), iter(set()), iter(frozenset()), @@ -1163,7 +1159,7 @@ def test_Iterator(self): ] for x in samples: self.assertIsInstance(x, Iterator) - self.assertTrue(issubclass(type(x), Iterator), repr(type(x))) + self.assertIsSubclass(type(x), Iterator) self.validate_abstract_methods(Iterator, '__next__') # Issue 10565 @@ -1196,7 +1192,7 @@ def throw(self, typ, val=None, tb=None): pass iter(()), iter([]), NonGen1(), NonGen2(), NonGen3()] for x in non_samples: self.assertNotIsInstance(x, Generator) - self.assertFalse(issubclass(type(x), Generator), repr(type(x))) + self.assertNotIsSubclass(type(x), Generator) class Gen: def __iter__(self): return self @@ -1218,7 +1214,7 @@ def gen(): for x in samples: self.assertIsInstance(x, Iterator) self.assertIsInstance(x, Generator) - self.assertTrue(issubclass(type(x), Generator), repr(type(x))) + self.assertIsSubclass(type(x), Generator) self.validate_abstract_methods(Generator, 'send', 'throw') # mixin tests @@ -1267,7 +1263,7 @@ def athrow(self, typ, val=None, tb=None): pass iter(()), iter([]), NonAGen1(), NonAGen2(), NonAGen3()] for x in non_samples: self.assertNotIsInstance(x, AsyncGenerator) - self.assertFalse(issubclass(type(x), AsyncGenerator), repr(type(x))) + self.assertNotIsSubclass(type(x), AsyncGenerator) class Gen: def __aiter__(self): return self @@ -1289,7 +1285,7 @@ async def gen(): for x in samples: self.assertIsInstance(x, AsyncIterator) self.assertIsInstance(x, AsyncGenerator) - self.assertTrue(issubclass(type(x), AsyncGenerator), repr(type(x))) + self.assertIsSubclass(type(x), AsyncGenerator) self.validate_abstract_methods(AsyncGenerator, 'asend', 'athrow') def run_async(coro): @@ -1332,14 +1328,14 @@ def test_Sized(self): ] for x in non_samples: self.assertNotIsInstance(x, Sized) - self.assertFalse(issubclass(type(x), Sized), repr(type(x))) + self.assertNotIsSubclass(type(x), Sized) samples = [bytes(), str(), tuple(), list(), set(), frozenset(), dict(), dict().keys(), dict().items(), dict().values(), ] for x in samples: self.assertIsInstance(x, Sized) - self.assertTrue(issubclass(type(x), Sized), repr(type(x))) + self.assertIsSubclass(type(x), Sized) self.validate_abstract_methods(Sized, '__len__') self.validate_isinstance(Sized, '__len__') @@ -1350,14 +1346,14 @@ def test_Container(self): ] for x in non_samples: self.assertNotIsInstance(x, Container) - self.assertFalse(issubclass(type(x), Container), repr(type(x))) + self.assertNotIsSubclass(type(x), Container) samples = [bytes(), str(), tuple(), list(), set(), frozenset(), dict(), dict().keys(), dict().items(), ] for x in samples: self.assertIsInstance(x, Container) - self.assertTrue(issubclass(type(x), Container), repr(type(x))) + self.assertIsSubclass(type(x), Container) self.validate_abstract_methods(Container, '__contains__') self.validate_isinstance(Container, '__contains__') @@ -1369,7 +1365,7 @@ def test_Callable(self): ] for x in non_samples: self.assertNotIsInstance(x, Callable) - self.assertFalse(issubclass(type(x), Callable), repr(type(x))) + self.assertNotIsSubclass(type(x), Callable) samples = [lambda: None, type, int, object, len, @@ -1377,7 +1373,7 @@ def test_Callable(self): ] for x in samples: self.assertIsInstance(x, Callable) - self.assertTrue(issubclass(type(x), Callable), repr(type(x))) + self.assertIsSubclass(type(x), Callable) self.validate_abstract_methods(Callable, '__call__') self.validate_isinstance(Callable, '__call__') @@ -1385,16 +1381,16 @@ def test_direct_subclassing(self): for B in Hashable, Iterable, Iterator, Reversible, Sized, Container, Callable: class C(B): pass - self.assertTrue(issubclass(C, B)) - self.assertFalse(issubclass(int, C)) + self.assertIsSubclass(C, B) + self.assertNotIsSubclass(int, C) def test_registration(self): for B in Hashable, Iterable, Iterator, Reversible, Sized, Container, Callable: class C: __hash__ = None # Make sure it isn't hashable by default - self.assertFalse(issubclass(C, B), B.__name__) + self.assertNotIsSubclass(C, B) B.register(C) - self.assertTrue(issubclass(C, B)) + self.assertIsSubclass(C, B) class WithSet(MutableSet): @@ -1425,7 +1421,7 @@ class TestCollectionABCs(ABCTestCase): def test_Set(self): for sample in [set, frozenset]: self.assertIsInstance(sample(), Set) - self.assertTrue(issubclass(sample, Set)) + self.assertIsSubclass(sample, Set) self.validate_abstract_methods(Set, '__contains__', '__iter__', '__len__') class MySet(Set): def __contains__(self, x): @@ -1506,9 +1502,9 @@ def __len__(self): def test_MutableSet(self): self.assertIsInstance(set(), MutableSet) - self.assertTrue(issubclass(set, MutableSet)) + self.assertIsSubclass(set, MutableSet) self.assertNotIsInstance(frozenset(), MutableSet) - self.assertFalse(issubclass(frozenset, MutableSet)) + self.assertNotIsSubclass(frozenset, MutableSet) self.validate_abstract_methods(MutableSet, '__contains__', '__iter__', '__len__', 'add', 'discard') @@ -1847,7 +1843,7 @@ def test_Set_hash_matches_frozenset(self): def test_Mapping(self): for sample in [dict]: self.assertIsInstance(sample(), Mapping) - self.assertTrue(issubclass(sample, Mapping)) + self.assertIsSubclass(sample, Mapping) self.validate_abstract_methods(Mapping, '__iter__', '__len__', '__getitem__') class MyMapping(Mapping): def __len__(self): @@ -1862,7 +1858,7 @@ def __iter__(self): def test_MutableMapping(self): for sample in [dict]: self.assertIsInstance(sample(), MutableMapping) - self.assertTrue(issubclass(sample, MutableMapping)) + self.assertIsSubclass(sample, MutableMapping) self.validate_abstract_methods(MutableMapping, '__iter__', '__len__', '__getitem__', '__setitem__', '__delitem__') @@ -1896,12 +1892,12 @@ def test_MutableMapping_subclass(self): def test_Sequence(self): for sample in [tuple, list, bytes, str]: self.assertIsInstance(sample(), Sequence) - self.assertTrue(issubclass(sample, Sequence)) + self.assertIsSubclass(sample, Sequence) self.assertIsInstance(range(10), Sequence) - self.assertTrue(issubclass(range, Sequence)) + self.assertIsSubclass(range, Sequence) self.assertIsInstance(memoryview(b""), Sequence) - self.assertTrue(issubclass(memoryview, Sequence)) - self.assertTrue(issubclass(str, Sequence)) + self.assertIsSubclass(memoryview, Sequence) + self.assertIsSubclass(str, Sequence) self.validate_abstract_methods(Sequence, '__len__', '__getitem__') def test_Sequence_mixins(self): @@ -1965,21 +1961,21 @@ class Z(ByteString, Awaitable): pass def test_Buffer(self): for sample in [bytes, bytearray, memoryview]: self.assertIsInstance(sample(b"x"), Buffer) - self.assertTrue(issubclass(sample, Buffer)) + self.assertIsSubclass(sample, Buffer) for sample in [str, list, tuple]: self.assertNotIsInstance(sample(), Buffer) - self.assertFalse(issubclass(sample, Buffer)) + self.assertNotIsSubclass(sample, Buffer) self.validate_abstract_methods(Buffer, '__buffer__') def test_MutableSequence(self): for sample in [tuple, str, bytes]: self.assertNotIsInstance(sample(), MutableSequence) - self.assertFalse(issubclass(sample, MutableSequence)) + self.assertNotIsSubclass(sample, MutableSequence) for sample in [list, bytearray, deque]: self.assertIsInstance(sample(), MutableSequence) - self.assertTrue(issubclass(sample, MutableSequence)) - self.assertTrue(issubclass(array.array, MutableSequence)) - self.assertFalse(issubclass(str, MutableSequence)) + self.assertIsSubclass(sample, MutableSequence) + self.assertIsSubclass(array.array, MutableSequence) + self.assertNotIsSubclass(str, MutableSequence) self.validate_abstract_methods(MutableSequence, '__len__', '__getitem__', '__setitem__', '__delitem__', 'insert') @@ -2071,8 +2067,8 @@ def test_basics(self): self.assertEqual(c, Counter(a=3, b=2, c=1)) self.assertIsInstance(c, dict) self.assertIsInstance(c, Mapping) - self.assertTrue(issubclass(Counter, dict)) - self.assertTrue(issubclass(Counter, Mapping)) + self.assertIsSubclass(Counter, dict) + self.assertIsSubclass(Counter, Mapping) self.assertEqual(len(c), 3) self.assertEqual(sum(c.values()), 6) self.assertEqual(list(c.values()), [3, 2, 1]) diff --git a/Lib/test/test_compile.py b/Lib/test/test_compile.py index 9f6ad6879c0..337366b4014 100644 --- a/Lib/test/test_compile.py +++ b/Lib/test/test_compile.py @@ -108,7 +108,6 @@ def __getitem__(self, key): exec('z = a', g, d) self.assertEqual(d['z'], 12) - @unittest.skip("TODO: RUSTPYTHON; segmentation fault") def test_extended_arg(self): # default: 1000 * 2.5 = 2500 repetitions repeat = int(sys.getrecursionlimit() * 2.5) @@ -153,8 +152,6 @@ def test_indentation(self): pass""" compile(s, "", "exec") - # TODO: RUSTPYTHON - @unittest.expectedFailure # This test is probably specific to CPython and may not generalize # to other implementations. We are trying to ensure that when # the first line of code starts after 256, correct line numbers @@ -929,8 +926,6 @@ def save_caller_frame(): func(save_caller_frame) self.assertEqual(frame.f_lineno-frame.f_code.co_firstlineno, lastline) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_lineno_after_no_code(self): def no_code1(): "doc string" @@ -1423,8 +1418,6 @@ def test_dict(self): def test_func_args(self): self.check_stack_size("f(" + "x, " * self.N + ")") - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_func_kwargs(self): kwargs = (f'a{i}=x' for i in range(self.N)) self.check_stack_size("f(" + ", ".join(kwargs) + ")") @@ -1434,8 +1427,6 @@ def test_func_kwargs(self): def test_meth_args(self): self.check_stack_size("o.m(" + "x, " * self.N + ")") - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_meth_kwargs(self): kwargs = (f'a{i}=x' for i in range(self.N)) self.check_stack_size("o.m(" + ", ".join(kwargs) + ")") diff --git a/Lib/test/test_compiler_assemble.py b/Lib/test/test_compiler_assemble.py new file mode 100644 index 00000000000..99a11e99d56 --- /dev/null +++ b/Lib/test/test_compiler_assemble.py @@ -0,0 +1,149 @@ +import dis +import io +import textwrap +import types + +from test.support.bytecode_helper import AssemblerTestCase + + +# Tests for the code-object creation stage of the compiler. + +class IsolatedAssembleTests(AssemblerTestCase): + + def complete_metadata(self, metadata, filename="myfile.py"): + if metadata is None: + metadata = {} + for key in ['name', 'qualname']: + metadata.setdefault(key, key) + for key in ['consts']: + metadata.setdefault(key, []) + for key in ['names', 'varnames', 'cellvars', 'freevars', 'fasthidden']: + metadata.setdefault(key, {}) + for key in ['argcount', 'posonlyargcount', 'kwonlyargcount']: + metadata.setdefault(key, 0) + metadata.setdefault('firstlineno', 1) + metadata.setdefault('filename', filename) + return metadata + + def insts_to_code_object(self, insts, metadata): + metadata = self.complete_metadata(metadata) + seq = self.seq_from_insts(insts) + return self.get_code_object(metadata['filename'], seq, metadata) + + def assemble_test(self, insts, metadata, expected): + co = self.insts_to_code_object(insts, metadata) + self.assertIsInstance(co, types.CodeType) + + expected_metadata = {} + for key, value in metadata.items(): + if key == "fasthidden": + # not exposed on code object + continue + if isinstance(value, list): + expected_metadata[key] = tuple(value) + elif isinstance(value, dict): + expected_metadata[key] = tuple(value.keys()) + else: + expected_metadata[key] = value + + for key, value in expected_metadata.items(): + self.assertEqual(getattr(co, "co_" + key), value) + + f = types.FunctionType(co, {}) + for args, res in expected.items(): + self.assertEqual(f(*args), res) + + def test_simple_expr(self): + metadata = { + 'filename' : 'avg.py', + 'name' : 'avg', + 'qualname' : 'stats.avg', + 'consts' : {2 : 0}, + 'argcount' : 2, + 'varnames' : {'x' : 0, 'y' : 1}, + } + + # code for "return (x+y)/2" + insts = [ + ('RESUME', 0), + ('LOAD_FAST', 0, 1), # 'x' + ('LOAD_FAST', 1, 1), # 'y' + ('BINARY_OP', 0, 1), # '+' + ('LOAD_CONST', 0, 1), # 2 + ('BINARY_OP', 11, 1), # '/' + ('RETURN_VALUE', None, 1), + ] + expected = {(3, 4) : 3.5, (-100, 200) : 50, (10, 18) : 14} + self.assemble_test(insts, metadata, expected) + + + def test_expression_with_pseudo_instruction_load_closure(self): + + def mod_two(x): + def inner(): + return x + return inner() % 2 + + inner_code = mod_two.__code__.co_consts[0] + assert isinstance(inner_code, types.CodeType) + + metadata = { + 'filename' : 'mod_two.py', + 'name' : 'mod_two', + 'qualname' : 'nested.mod_two', + 'cellvars' : {'x' : 0}, + 'consts': {None: 0, inner_code: 1, 2: 2}, + 'argcount' : 1, + 'varnames' : {'x' : 0}, + } + + instructions = [ + ('RESUME', 0,), + ('LOAD_CLOSURE', 0, 1), + ('BUILD_TUPLE', 1, 1), + ('LOAD_CONST', 1, 1), + ('MAKE_FUNCTION', None, 2), + ('SET_FUNCTION_ATTRIBUTE', 8, 2), + ('PUSH_NULL', None, 1), + ('CALL', 0, 2), # (lambda: x)() + ('LOAD_CONST', 2, 2), # 2 + ('BINARY_OP', 6, 2), # % + ('RETURN_VALUE', None, 2) + ] + + expected = {(0,): 0, (1,): 1, (2,): 0, (120,): 0, (121,): 1} + self.assemble_test(instructions, metadata, expected) + + + def test_exception_table(self): + metadata = { + 'filename' : 'exc.py', + 'name' : 'exc', + 'consts' : {2 : 0}, + } + + # code for "try: pass\n except: pass" + insts = [ + ('RESUME', 0), + ('SETUP_FINALLY', 4), + ('LOAD_CONST', 0), + ('RETURN_VALUE', None), + ('SETUP_CLEANUP', 10), + ('PUSH_EXC_INFO', None), + ('POP_TOP', None), + ('POP_EXCEPT', None), + ('LOAD_CONST', 0), + ('RETURN_VALUE', None), + ('COPY', 3), + ('POP_EXCEPT', None), + ('RERAISE', 1), + ] + co = self.insts_to_code_object(insts, metadata) + output = io.StringIO() + dis.dis(co, file=output) + exc_table = textwrap.dedent(""" + ExceptionTable: + L1 to L2 -> L2 [0] + L2 to L3 -> L3 [1] lasti + """) + self.assertEndsWith(output.getvalue(), exc_table) diff --git a/Lib/test/test_compiler_codegen.py b/Lib/test/test_compiler_codegen.py new file mode 100644 index 00000000000..d02937c84d9 --- /dev/null +++ b/Lib/test/test_compiler_codegen.py @@ -0,0 +1,163 @@ + +import textwrap +from test.support.bytecode_helper import CodegenTestCase + +# Tests for the code-generation stage of the compiler. +# Examine the un-optimized code generated from the AST. + +class IsolatedCodeGenTests(CodegenTestCase): + + def assertInstructionsMatch_recursive(self, insts, expected_insts): + expected_nested = [i for i in expected_insts if isinstance(i, list)] + expected_insts = [i for i in expected_insts if not isinstance(i, list)] + self.assertInstructionsMatch(insts, expected_insts) + self.assertEqual(len(insts.get_nested()), len(expected_nested)) + for n_insts, n_expected in zip(insts.get_nested(), expected_nested): + self.assertInstructionsMatch_recursive(n_insts, n_expected) + + def codegen_test(self, snippet, expected_insts): + import ast + a = ast.parse(snippet, "my_file.py", "exec") + insts = self.generate_code(a) + self.assertInstructionsMatch_recursive(insts, expected_insts) + + def test_if_expression(self): + snippet = "42 if True else 24" + false_lbl = self.Label() + expected = [ + ('RESUME', 0, 0), + ('ANNOTATIONS_PLACEHOLDER', None), + ('LOAD_CONST', 0, 1), + ('TO_BOOL', 0, 1), + ('POP_JUMP_IF_FALSE', false_lbl := self.Label(), 1), + ('LOAD_CONST', 1, 1), # 42 + ('JUMP_NO_INTERRUPT', exit_lbl := self.Label()), + false_lbl, + ('LOAD_CONST', 2, 1), # 24 + exit_lbl, + ('POP_TOP', None), + ('LOAD_CONST', 1), + ('RETURN_VALUE', None), + ] + self.codegen_test(snippet, expected) + + def test_for_loop(self): + snippet = "for x in l:\n\tprint(x)" + false_lbl = self.Label() + expected = [ + ('RESUME', 0, 0), + ('ANNOTATIONS_PLACEHOLDER', None), + ('LOAD_NAME', 0, 1), + ('GET_ITER', None, 1), + loop_lbl := self.Label(), + ('FOR_ITER', exit_lbl := self.Label(), 1), + ('NOP', None, 1, 1), + ('STORE_NAME', 1, 1), + ('LOAD_NAME', 2, 2), + ('PUSH_NULL', None, 2), + ('LOAD_NAME', 1, 2), + ('CALL', 1, 2), + ('POP_TOP', None), + ('JUMP', loop_lbl), + exit_lbl, + ('END_FOR', None), + ('POP_ITER', None), + ('LOAD_CONST', 0), + ('RETURN_VALUE', None), + ] + self.codegen_test(snippet, expected) + + def test_function(self): + snippet = textwrap.dedent(""" + def f(x): + return x + 42 + """) + expected = [ + # Function definition + ('RESUME', 0), + ('ANNOTATIONS_PLACEHOLDER', None), + ('LOAD_CONST', 0), + ('MAKE_FUNCTION', None), + ('STORE_NAME', 0), + ('LOAD_CONST', 1), + ('RETURN_VALUE', None), + [ + # Function body + ('RESUME', 0), + ('LOAD_FAST', 0), + ('LOAD_CONST', 42), + ('BINARY_OP', 0), + ('RETURN_VALUE', None), + ('LOAD_CONST', 0), + ('RETURN_VALUE', None), + ] + ] + self.codegen_test(snippet, expected) + + def test_nested_functions(self): + snippet = textwrap.dedent(""" + def f(): + def h(): + return 12 + def g(): + x = 1 + y = 2 + z = 3 + u = 4 + return 42 + """) + expected = [ + # Function definition + ('RESUME', 0), + ('ANNOTATIONS_PLACEHOLDER', None), + ('LOAD_CONST', 0), + ('MAKE_FUNCTION', None), + ('STORE_NAME', 0), + ('LOAD_CONST', 1), + ('RETURN_VALUE', None), + [ + # Function body + ('RESUME', 0), + ('LOAD_CONST', 1), + ('MAKE_FUNCTION', None), + ('STORE_FAST', 0), + ('LOAD_CONST', 2), + ('MAKE_FUNCTION', None), + ('STORE_FAST', 1), + ('LOAD_CONST', 0), + ('RETURN_VALUE', None), + [ + ('RESUME', 0), + ('NOP', None), + ('LOAD_CONST', 12), + ('RETURN_VALUE', None), + ('LOAD_CONST', 1), + ('RETURN_VALUE', None), + ], + [ + ('RESUME', 0), + ('LOAD_CONST', 1), + ('STORE_FAST', 0), + ('LOAD_CONST', 2), + ('STORE_FAST', 1), + ('LOAD_CONST', 3), + ('STORE_FAST', 2), + ('LOAD_CONST', 4), + ('STORE_FAST', 3), + ('NOP', None), + ('LOAD_CONST', 42), + ('RETURN_VALUE', None), + ('LOAD_CONST', 0), + ('RETURN_VALUE', None), + ], + ], + ] + self.codegen_test(snippet, expected) + + def test_syntax_error__return_not_in_function(self): + snippet = "return 42" + with self.assertRaisesRegex(SyntaxError, "'return' outside function") as cm: + self.codegen_test(snippet, None) + self.assertIsNone(cm.exception.text) + self.assertEqual(cm.exception.offset, 1) + self.assertEqual(cm.exception.end_offset, 10) diff --git a/Lib/test/test_concurrent_futures/__init__.py b/Lib/test/test_concurrent_futures/__init__.py new file mode 100644 index 00000000000..b38bd38d338 --- /dev/null +++ b/Lib/test/test_concurrent_futures/__init__.py @@ -0,0 +1,18 @@ +import os.path +import unittest +from test import support +from test.support import threading_helper + + +# Adjust if we ever have a platform with processes but not threads. +threading_helper.requires_working_threading(module=True) + + +if support.check_sanitizer(address=True, memory=True): + # gh-90791: Skip the test because it is too slow when Python is built + # with ASAN/MSAN: between 5 and 20 minutes on GitHub Actions. + raise unittest.SkipTest("test too slow on ASAN/MSAN build") + + +def load_tests(*args): + return support.load_package_tests(os.path.dirname(__file__), *args) diff --git a/Lib/test/test_concurrent_futures/executor.py b/Lib/test/test_concurrent_futures/executor.py new file mode 100644 index 00000000000..7442d3bee52 --- /dev/null +++ b/Lib/test/test_concurrent_futures/executor.py @@ -0,0 +1,162 @@ +import threading +import time +import unittest +import weakref +from concurrent import futures +from test import support +from test.support import Py_GIL_DISABLED + + +def mul(x, y): + return x * y + +def capture(*args, **kwargs): + return args, kwargs + + +class MyObject(object): + def my_method(self): + pass + + +def make_dummy_object(_): + return MyObject() + + +# Used in test_swallows_falsey_exceptions +def raiser(exception, msg='std'): + raise exception(msg) + + +class FalseyBoolException(Exception): + def __bool__(self): + return False + + +class FalseyLenException(Exception): + def __len__(self): + return 0 + + +class ExecutorTest: + # Executor.shutdown() and context manager usage is tested by + # ExecutorShutdownTest. + def test_submit(self): + future = self.executor.submit(pow, 2, 8) + self.assertEqual(256, future.result()) + + def test_submit_keyword(self): + future = self.executor.submit(mul, 2, y=8) + self.assertEqual(16, future.result()) + future = self.executor.submit(capture, 1, self=2, fn=3) + self.assertEqual(future.result(), ((1,), {'self': 2, 'fn': 3})) + with self.assertRaises(TypeError): + self.executor.submit(fn=capture, arg=1) + with self.assertRaises(TypeError): + self.executor.submit(arg=1) + + def test_map(self): + self.assertEqual( + list(self.executor.map(pow, range(10), range(10))), + list(map(pow, range(10), range(10)))) + + self.assertEqual( + list(self.executor.map(pow, range(10), range(10), chunksize=3)), + list(map(pow, range(10), range(10)))) + + def test_map_exception(self): + i = self.executor.map(divmod, [1, 1, 1, 1], [2, 3, 0, 5]) + self.assertEqual(i.__next__(), (0, 1)) + self.assertEqual(i.__next__(), (0, 1)) + self.assertRaises(ZeroDivisionError, i.__next__) + + @support.requires_resource('walltime') + def test_map_timeout(self): + results = [] + try: + for i in self.executor.map(time.sleep, + [0, 0, 6], + timeout=5): + results.append(i) + except futures.TimeoutError: + pass + else: + self.fail('expected TimeoutError') + + self.assertEqual([None, None], results) + + def test_shutdown_race_issue12456(self): + # Issue #12456: race condition at shutdown where trying to post a + # sentinel in the call queue blocks (the queue is full while processes + # have exited). + self.executor.map(str, [2] * (self.worker_count + 1)) + self.executor.shutdown() + + @support.cpython_only + def test_no_stale_references(self): + # Issue #16284: check that the executors don't unnecessarily hang onto + # references. + my_object = MyObject() + my_object_collected = threading.Event() + def set_event(): + if Py_GIL_DISABLED: + # gh-117688 Avoid deadlock by setting the event in a + # background thread. The current thread may be in the middle + # of the my_object_collected.wait() call, which holds locks + # needed by my_object_collected.set(). + threading.Thread(target=my_object_collected.set).start() + else: + my_object_collected.set() + my_object_callback = weakref.ref(my_object, lambda obj: set_event()) + # Deliberately discarding the future. + self.executor.submit(my_object.my_method) + del my_object + + if Py_GIL_DISABLED: + # Due to biased reference counting, my_object might only be + # deallocated while the thread that created it runs -- if the + # thread is paused waiting on an event, it may not merge the + # refcount of the queued object. For that reason, we alternate + # between running the GC and waiting for the event. + wait_time = 0 + collected = False + while not collected and wait_time <= support.SHORT_TIMEOUT: + support.gc_collect() + collected = my_object_collected.wait(timeout=1.0) + wait_time += 1.0 + else: + collected = my_object_collected.wait(timeout=support.SHORT_TIMEOUT) + self.assertTrue(collected, + "Stale reference not collected within timeout.") + + def test_max_workers_negative(self): + for number in (0, -1): + with self.assertRaisesRegex(ValueError, + "max_workers must be greater " + "than 0"): + self.executor_type(max_workers=number) + + def test_free_reference(self): + # Issue #14406: Result iterator should not keep an internal + # reference to result objects. + for obj in self.executor.map(make_dummy_object, range(10)): + wr = weakref.ref(obj) + del obj + support.gc_collect() # For PyPy or other GCs. + + for _ in support.sleeping_retry(support.SHORT_TIMEOUT): + if wr() is None: + break + + def test_swallows_falsey_exceptions(self): + # see gh-132063: Prevent exceptions that evaluate as falsey + # from being ignored. + # Recall: `x` is falsey if `len(x)` returns 0 or `bool(x)` returns False. + + msg = 'boolbool' + with self.assertRaisesRegex(FalseyBoolException, msg): + self.executor.submit(raiser, FalseyBoolException, msg).result() + + msg = 'lenlen' + with self.assertRaisesRegex(FalseyLenException, msg): + self.executor.submit(raiser, FalseyLenException, msg).result() diff --git a/Lib/test/test_concurrent_futures/test_as_completed.py b/Lib/test/test_concurrent_futures/test_as_completed.py new file mode 100644 index 00000000000..c90b0021d85 --- /dev/null +++ b/Lib/test/test_concurrent_futures/test_as_completed.py @@ -0,0 +1,118 @@ +import itertools +import time +import unittest +import weakref +from concurrent import futures +from concurrent.futures._base import ( + CANCELLED_AND_NOTIFIED, FINISHED, Future) + +from test import support + +from .util import ( + PENDING_FUTURE, RUNNING_FUTURE, + CANCELLED_AND_NOTIFIED_FUTURE, EXCEPTION_FUTURE, SUCCESSFUL_FUTURE, + create_future, create_executor_tests, setup_module) + + +def mul(x, y): + return x * y + + +class AsCompletedTests: + def test_no_timeout(self): + future1 = self.executor.submit(mul, 2, 21) + future2 = self.executor.submit(mul, 7, 6) + + completed = set(futures.as_completed( + [CANCELLED_AND_NOTIFIED_FUTURE, + EXCEPTION_FUTURE, + SUCCESSFUL_FUTURE, + future1, future2])) + self.assertEqual(set( + [CANCELLED_AND_NOTIFIED_FUTURE, + EXCEPTION_FUTURE, + SUCCESSFUL_FUTURE, + future1, future2]), + completed) + + def test_future_times_out(self): + """Test ``futures.as_completed`` timing out before + completing it's final future.""" + already_completed = {CANCELLED_AND_NOTIFIED_FUTURE, + EXCEPTION_FUTURE, + SUCCESSFUL_FUTURE} + + # Windows clock resolution is around 15.6 ms + short_timeout = 0.100 + for timeout in (0, short_timeout): + with self.subTest(timeout): + + completed_futures = set() + future = self.executor.submit(time.sleep, short_timeout * 10) + + try: + for f in futures.as_completed( + already_completed | {future}, + timeout + ): + completed_futures.add(f) + except futures.TimeoutError: + pass + + # Check that ``future`` wasn't completed. + self.assertEqual(completed_futures, already_completed) + + def test_duplicate_futures(self): + # Issue 20367. Duplicate futures should not raise exceptions or give + # duplicate responses. + # Issue #31641: accept arbitrary iterables. + future1 = self.executor.submit(time.sleep, 2) + completed = [ + f for f in futures.as_completed(itertools.repeat(future1, 3)) + ] + self.assertEqual(len(completed), 1) + + def test_free_reference_yielded_future(self): + # Issue #14406: Generator should not keep references + # to finished futures. + futures_list = [Future() for _ in range(8)] + futures_list.append(create_future(state=CANCELLED_AND_NOTIFIED)) + futures_list.append(create_future(state=FINISHED, result=42)) + + with self.assertRaises(futures.TimeoutError): + for future in futures.as_completed(futures_list, timeout=0): + futures_list.remove(future) + wr = weakref.ref(future) + del future + support.gc_collect() # For PyPy or other GCs. + self.assertIsNone(wr()) + + futures_list[0].set_result("test") + for future in futures.as_completed(futures_list): + futures_list.remove(future) + wr = weakref.ref(future) + del future + support.gc_collect() # For PyPy or other GCs. + self.assertIsNone(wr()) + if futures_list: + futures_list[0].set_result("test") + + def test_correct_timeout_exception_msg(self): + futures_list = [CANCELLED_AND_NOTIFIED_FUTURE, PENDING_FUTURE, + RUNNING_FUTURE, SUCCESSFUL_FUTURE] + + with self.assertRaises(futures.TimeoutError) as cm: + list(futures.as_completed(futures_list, timeout=0)) + + self.assertEqual(str(cm.exception), '2 (of 4) futures unfinished') + + +create_executor_tests(globals(), AsCompletedTests) + + +def setUpModule(): + setup_module() + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_concurrent_futures/test_deadlock.py b/Lib/test/test_concurrent_futures/test_deadlock.py new file mode 100644 index 00000000000..dcc1d68563a --- /dev/null +++ b/Lib/test/test_concurrent_futures/test_deadlock.py @@ -0,0 +1,332 @@ +import contextlib +import queue +import signal +import sys +import time +import unittest +import unittest.mock +from pickle import PicklingError +from concurrent import futures +from concurrent.futures.process import BrokenProcessPool, _ThreadWakeup + +from test import support + +from .util import ( + create_executor_tests, setup_module, + ProcessPoolForkMixin, ProcessPoolForkserverMixin, ProcessPoolSpawnMixin) + + +def _crash(delay=None): + """Induces a segfault.""" + if delay: + time.sleep(delay) + import faulthandler + faulthandler.disable() + faulthandler._sigsegv() + + +def _crash_with_data(data): + """Induces a segfault with dummy data in input.""" + _crash() + + +def _exit(): + """Induces a sys exit with exitcode 1.""" + sys.exit(1) + + +def _raise_error(Err): + """Function that raises an Exception in process.""" + raise Err() + + +def _raise_error_ignore_stderr(Err): + """Function that raises an Exception in process and ignores stderr.""" + import io + sys.stderr = io.StringIO() + raise Err() + + +def _return_instance(cls): + """Function that returns a instance of cls.""" + return cls() + + +class CrashAtPickle(object): + """Bad object that triggers a segfault at pickling time.""" + def __reduce__(self): + _crash() + + +class CrashAtUnpickle(object): + """Bad object that triggers a segfault at unpickling time.""" + def __reduce__(self): + return _crash, () + + +class ExitAtPickle(object): + """Bad object that triggers a process exit at pickling time.""" + def __reduce__(self): + _exit() + + +class ExitAtUnpickle(object): + """Bad object that triggers a process exit at unpickling time.""" + def __reduce__(self): + return _exit, () + + +class ErrorAtPickle(object): + """Bad object that triggers an error at pickling time.""" + def __reduce__(self): + from pickle import PicklingError + raise PicklingError("Error in pickle") + + +class ErrorAtUnpickle(object): + """Bad object that triggers an error at unpickling time.""" + def __reduce__(self): + from pickle import UnpicklingError + return _raise_error_ignore_stderr, (UnpicklingError, ) + + +class ExecutorDeadlockTest: + TIMEOUT = support.LONG_TIMEOUT + + def _fail_on_deadlock(self, executor): + # If we did not recover before TIMEOUT seconds, consider that the + # executor is in a deadlock state and forcefully clean all its + # composants. + import faulthandler + from tempfile import TemporaryFile + with TemporaryFile(mode="w+") as f: + faulthandler.dump_traceback(file=f) + f.seek(0) + tb = f.read() + for p in executor._processes.values(): + p.terminate() + # This should be safe to call executor.shutdown here as all possible + # deadlocks should have been broken. + executor.shutdown(wait=True) + print(f"\nTraceback:\n {tb}", file=sys.__stderr__) + self.fail(f"Executor deadlock:\n\n{tb}") + + + def _check_crash(self, error, func, *args, ignore_stderr=False): + # test for deadlock caused by crashes in a pool + self.executor.shutdown(wait=True) + + executor = self.executor_type( + max_workers=2, mp_context=self.get_context()) + res = executor.submit(func, *args) + + if ignore_stderr: + cm = support.captured_stderr() + else: + cm = contextlib.nullcontext() + + try: + with self.assertRaises(error): + with cm: + res.result(timeout=self.TIMEOUT) + except futures.TimeoutError: + # If we did not recover before TIMEOUT seconds, + # consider that the executor is in a deadlock state + self._fail_on_deadlock(executor) + executor.shutdown(wait=True) + + def test_error_at_task_pickle(self): + # Check problem occurring while pickling a task in + # the task_handler thread + self._check_crash(PicklingError, id, ErrorAtPickle()) + + def test_exit_at_task_unpickle(self): + # Check problem occurring while unpickling a task on workers + self._check_crash(BrokenProcessPool, id, ExitAtUnpickle()) + + def test_error_at_task_unpickle(self): + # gh-109832: Restore stderr overridden by _raise_error_ignore_stderr() + self.addCleanup(setattr, sys, 'stderr', sys.stderr) + + # Check problem occurring while unpickling a task on workers + self._check_crash(BrokenProcessPool, id, ErrorAtUnpickle()) + + def test_crash_at_task_unpickle(self): + # Check problem occurring while unpickling a task on workers + self._check_crash(BrokenProcessPool, id, CrashAtUnpickle()) + + def test_crash_during_func_exec_on_worker(self): + # Check problem occurring during func execution on workers + self._check_crash(BrokenProcessPool, _crash) + + def test_exit_during_func_exec_on_worker(self): + # Check problem occurring during func execution on workers + self._check_crash(SystemExit, _exit) + + def test_error_during_func_exec_on_worker(self): + # Check problem occurring during func execution on workers + self._check_crash(RuntimeError, _raise_error, RuntimeError) + + def test_crash_during_result_pickle_on_worker(self): + # Check problem occurring while pickling a task result + # on workers + self._check_crash(BrokenProcessPool, _return_instance, CrashAtPickle) + + def test_exit_during_result_pickle_on_worker(self): + # Check problem occurring while pickling a task result + # on workers + self._check_crash(SystemExit, _return_instance, ExitAtPickle) + + def test_error_during_result_pickle_on_worker(self): + # Check problem occurring while pickling a task result + # on workers + self._check_crash(PicklingError, _return_instance, ErrorAtPickle) + + def test_error_during_result_unpickle_in_result_handler(self): + # gh-109832: Restore stderr overridden by _raise_error_ignore_stderr() + self.addCleanup(setattr, sys, 'stderr', sys.stderr) + + # Check problem occurring while unpickling a task in + # the result_handler thread + self._check_crash(BrokenProcessPool, + _return_instance, ErrorAtUnpickle, + ignore_stderr=True) + + def test_exit_during_result_unpickle_in_result_handler(self): + # Check problem occurring while unpickling a task in + # the result_handler thread + self._check_crash(BrokenProcessPool, _return_instance, ExitAtUnpickle) + + def test_shutdown_deadlock(self): + # Test that the pool calling shutdown do not cause deadlock + # if a worker fails after the shutdown call. + self.executor.shutdown(wait=True) + with self.executor_type(max_workers=2, + mp_context=self.get_context()) as executor: + self.executor = executor # Allow clean up in fail_on_deadlock + f = executor.submit(_crash, delay=.1) + executor.shutdown(wait=True) + with self.assertRaises(BrokenProcessPool): + f.result() + + def test_shutdown_deadlock_pickle(self): + # Test that the pool calling shutdown with wait=False does not cause + # a deadlock if a task fails at pickle after the shutdown call. + # Reported in bpo-39104. + self.executor.shutdown(wait=True) + with self.executor_type(max_workers=2, + mp_context=self.get_context()) as executor: + self.executor = executor # Allow clean up in fail_on_deadlock + + # Start the executor and get the executor_manager_thread to collect + # the threads and avoid dangling thread that should be cleaned up + # asynchronously. + executor.submit(id, 42).result() + executor_manager = executor._executor_manager_thread + + # Submit a task that fails at pickle and shutdown the executor + # without waiting + f = executor.submit(id, ErrorAtPickle()) + executor.shutdown(wait=False) + with self.assertRaises(PicklingError): + f.result() + + # Make sure the executor is eventually shutdown and do not leave + # dangling threads + executor_manager.join() + + def test_crash_big_data(self): + # Test that there is a clean exception instad of a deadlock when a + # child process crashes while some data is being written into the + # queue. + # https://github.com/python/cpython/issues/94777 + self.executor.shutdown(wait=True) + data = "a" * support.PIPE_MAX_SIZE + with self.executor_type(max_workers=2, + mp_context=self.get_context()) as executor: + self.executor = executor # Allow clean up in fail_on_deadlock + with self.assertRaises(BrokenProcessPool): + list(executor.map(_crash_with_data, [data] * 10)) + + executor.shutdown(wait=True) + + def test_gh105829_should_not_deadlock_if_wakeup_pipe_full(self): + # Issue #105829: The _ExecutorManagerThread wakeup pipe could + # fill up and block. See: https://github.com/python/cpython/issues/105829 + + # Lots of cargo culting while writing this test, apologies if + # something is really stupid... + + self.executor.shutdown(wait=True) + + if not hasattr(signal, 'alarm'): + raise unittest.SkipTest( + "Tested platform does not support the alarm signal") + + def timeout(_signum, _frame): + import faulthandler + faulthandler.dump_traceback() + + raise RuntimeError("timed out while submitting jobs?") + + thread_run = futures.process._ExecutorManagerThread.run + def mock_run(self): + # Delay thread startup so the wakeup pipe can fill up and block + time.sleep(3) + thread_run(self) + + class MockWakeup(_ThreadWakeup): + """Mock wakeup object to force the wakeup to block""" + def __init__(self): + super().__init__() + self._dummy_queue = queue.Queue(maxsize=1) + + def wakeup(self): + self._dummy_queue.put(None, block=True) + super().wakeup() + + def clear(self): + super().clear() + try: + while True: + self._dummy_queue.get_nowait() + except queue.Empty: + pass + + with (unittest.mock.patch.object(futures.process._ExecutorManagerThread, + 'run', mock_run), + unittest.mock.patch('concurrent.futures.process._ThreadWakeup', + MockWakeup)): + with self.executor_type(max_workers=2, + mp_context=self.get_context()) as executor: + self.executor = executor # Allow clean up in fail_on_deadlock + + job_num = 100 + job_data = range(job_num) + + # Need to use sigalarm for timeout detection because + # Executor.submit is not guarded by any timeout (both + # self._work_ids.put(self._queue_count) and + # self._executor_manager_thread_wakeup.wakeup() might + # timeout, maybe more?). In this specific case it was + # the wakeup call that deadlocked on a blocking pipe. + old_handler = signal.signal(signal.SIGALRM, timeout) + try: + signal.alarm(int(self.TIMEOUT)) + self.assertEqual(job_num, len(list(executor.map(int, job_data)))) + finally: + signal.alarm(0) + signal.signal(signal.SIGALRM, old_handler) + + +create_executor_tests(globals(), ExecutorDeadlockTest, + executor_mixins=(ProcessPoolForkMixin, + ProcessPoolForkserverMixin, + ProcessPoolSpawnMixin)) + +def setUpModule(): + setup_module() + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_concurrent_futures/test_future.py b/Lib/test/test_concurrent_futures/test_future.py new file mode 100644 index 00000000000..4066ea1ee4b --- /dev/null +++ b/Lib/test/test_concurrent_futures/test_future.py @@ -0,0 +1,291 @@ +import threading +import time +import unittest +from concurrent import futures +from concurrent.futures._base import ( + PENDING, RUNNING, CANCELLED, CANCELLED_AND_NOTIFIED, FINISHED, Future) + +from test import support + +from .util import ( + PENDING_FUTURE, RUNNING_FUTURE, CANCELLED_FUTURE, + CANCELLED_AND_NOTIFIED_FUTURE, EXCEPTION_FUTURE, SUCCESSFUL_FUTURE, + BaseTestCase, create_future, setup_module) + + +class FutureTests(BaseTestCase): + def test_done_callback_with_result(self): + callback_result = None + def fn(callback_future): + nonlocal callback_result + callback_result = callback_future.result() + + f = Future() + f.add_done_callback(fn) + f.set_result(5) + self.assertEqual(5, callback_result) + + def test_done_callback_with_exception(self): + callback_exception = None + def fn(callback_future): + nonlocal callback_exception + callback_exception = callback_future.exception() + + f = Future() + f.add_done_callback(fn) + f.set_exception(Exception('test')) + self.assertEqual(('test',), callback_exception.args) + + def test_done_callback_with_cancel(self): + was_cancelled = None + def fn(callback_future): + nonlocal was_cancelled + was_cancelled = callback_future.cancelled() + + f = Future() + f.add_done_callback(fn) + self.assertTrue(f.cancel()) + self.assertTrue(was_cancelled) + + def test_done_callback_raises(self): + with support.captured_stderr() as stderr: + raising_was_called = False + fn_was_called = False + + def raising_fn(callback_future): + nonlocal raising_was_called + raising_was_called = True + raise Exception('doh!') + + def fn(callback_future): + nonlocal fn_was_called + fn_was_called = True + + f = Future() + f.add_done_callback(raising_fn) + f.add_done_callback(fn) + f.set_result(5) + self.assertTrue(raising_was_called) + self.assertTrue(fn_was_called) + self.assertIn('Exception: doh!', stderr.getvalue()) + + def test_done_callback_already_successful(self): + callback_result = None + def fn(callback_future): + nonlocal callback_result + callback_result = callback_future.result() + + f = Future() + f.set_result(5) + f.add_done_callback(fn) + self.assertEqual(5, callback_result) + + def test_done_callback_already_failed(self): + callback_exception = None + def fn(callback_future): + nonlocal callback_exception + callback_exception = callback_future.exception() + + f = Future() + f.set_exception(Exception('test')) + f.add_done_callback(fn) + self.assertEqual(('test',), callback_exception.args) + + def test_done_callback_already_cancelled(self): + was_cancelled = None + def fn(callback_future): + nonlocal was_cancelled + was_cancelled = callback_future.cancelled() + + f = Future() + self.assertTrue(f.cancel()) + f.add_done_callback(fn) + self.assertTrue(was_cancelled) + + def test_done_callback_raises_already_succeeded(self): + with support.captured_stderr() as stderr: + def raising_fn(callback_future): + raise Exception('doh!') + + f = Future() + + # Set the result first to simulate a future that runs instantly, + # effectively allowing the callback to be run immediately. + f.set_result(5) + f.add_done_callback(raising_fn) + + self.assertIn('exception calling callback for', stderr.getvalue()) + self.assertIn('doh!', stderr.getvalue()) + + + def test_repr(self): + self.assertRegex(repr(PENDING_FUTURE), + '') + self.assertRegex(repr(RUNNING_FUTURE), + '') + self.assertRegex(repr(CANCELLED_FUTURE), + '') + self.assertRegex(repr(CANCELLED_AND_NOTIFIED_FUTURE), + '') + self.assertRegex( + repr(EXCEPTION_FUTURE), + '') + self.assertRegex( + repr(SUCCESSFUL_FUTURE), + '') + + def test_cancel(self): + f1 = create_future(state=PENDING) + f2 = create_future(state=RUNNING) + f3 = create_future(state=CANCELLED) + f4 = create_future(state=CANCELLED_AND_NOTIFIED) + f5 = create_future(state=FINISHED, exception=OSError()) + f6 = create_future(state=FINISHED, result=5) + + self.assertTrue(f1.cancel()) + self.assertEqual(f1._state, CANCELLED) + + self.assertFalse(f2.cancel()) + self.assertEqual(f2._state, RUNNING) + + self.assertTrue(f3.cancel()) + self.assertEqual(f3._state, CANCELLED) + + self.assertTrue(f4.cancel()) + self.assertEqual(f4._state, CANCELLED_AND_NOTIFIED) + + self.assertFalse(f5.cancel()) + self.assertEqual(f5._state, FINISHED) + + self.assertFalse(f6.cancel()) + self.assertEqual(f6._state, FINISHED) + + def test_cancelled(self): + self.assertFalse(PENDING_FUTURE.cancelled()) + self.assertFalse(RUNNING_FUTURE.cancelled()) + self.assertTrue(CANCELLED_FUTURE.cancelled()) + self.assertTrue(CANCELLED_AND_NOTIFIED_FUTURE.cancelled()) + self.assertFalse(EXCEPTION_FUTURE.cancelled()) + self.assertFalse(SUCCESSFUL_FUTURE.cancelled()) + + def test_done(self): + self.assertFalse(PENDING_FUTURE.done()) + self.assertFalse(RUNNING_FUTURE.done()) + self.assertTrue(CANCELLED_FUTURE.done()) + self.assertTrue(CANCELLED_AND_NOTIFIED_FUTURE.done()) + self.assertTrue(EXCEPTION_FUTURE.done()) + self.assertTrue(SUCCESSFUL_FUTURE.done()) + + def test_running(self): + self.assertFalse(PENDING_FUTURE.running()) + self.assertTrue(RUNNING_FUTURE.running()) + self.assertFalse(CANCELLED_FUTURE.running()) + self.assertFalse(CANCELLED_AND_NOTIFIED_FUTURE.running()) + self.assertFalse(EXCEPTION_FUTURE.running()) + self.assertFalse(SUCCESSFUL_FUTURE.running()) + + def test_result_with_timeout(self): + self.assertRaises(futures.TimeoutError, + PENDING_FUTURE.result, timeout=0) + self.assertRaises(futures.TimeoutError, + RUNNING_FUTURE.result, timeout=0) + self.assertRaises(futures.CancelledError, + CANCELLED_FUTURE.result, timeout=0) + self.assertRaises(futures.CancelledError, + CANCELLED_AND_NOTIFIED_FUTURE.result, timeout=0) + self.assertRaises(OSError, EXCEPTION_FUTURE.result, timeout=0) + self.assertEqual(SUCCESSFUL_FUTURE.result(timeout=0), 42) + + def test_result_with_success(self): + # TODO(brian@sweetapp.com): This test is timing dependent. + def notification(): + # Wait until the main thread is waiting for the result. + time.sleep(1) + f1.set_result(42) + + f1 = create_future(state=PENDING) + t = threading.Thread(target=notification) + t.start() + + self.assertEqual(f1.result(timeout=5), 42) + t.join() + + def test_result_with_cancel(self): + # TODO(brian@sweetapp.com): This test is timing dependent. + def notification(): + # Wait until the main thread is waiting for the result. + time.sleep(1) + f1.cancel() + + f1 = create_future(state=PENDING) + t = threading.Thread(target=notification) + t.start() + + self.assertRaises(futures.CancelledError, + f1.result, timeout=support.SHORT_TIMEOUT) + t.join() + + def test_exception_with_timeout(self): + self.assertRaises(futures.TimeoutError, + PENDING_FUTURE.exception, timeout=0) + self.assertRaises(futures.TimeoutError, + RUNNING_FUTURE.exception, timeout=0) + self.assertRaises(futures.CancelledError, + CANCELLED_FUTURE.exception, timeout=0) + self.assertRaises(futures.CancelledError, + CANCELLED_AND_NOTIFIED_FUTURE.exception, timeout=0) + self.assertTrue(isinstance(EXCEPTION_FUTURE.exception(timeout=0), + OSError)) + self.assertEqual(SUCCESSFUL_FUTURE.exception(timeout=0), None) + + def test_exception_with_success(self): + def notification(): + # Wait until the main thread is waiting for the exception. + time.sleep(1) + with f1._condition: + f1._state = FINISHED + f1._exception = OSError() + f1._condition.notify_all() + + f1 = create_future(state=PENDING) + t = threading.Thread(target=notification) + t.start() + + self.assertTrue(isinstance(f1.exception(timeout=support.SHORT_TIMEOUT), OSError)) + t.join() + + def test_multiple_set_result(self): + f = create_future(state=PENDING) + f.set_result(1) + + with self.assertRaisesRegex( + futures.InvalidStateError, + 'FINISHED: ' + ): + f.set_result(2) + + self.assertTrue(f.done()) + self.assertEqual(f.result(), 1) + + def test_multiple_set_exception(self): + f = create_future(state=PENDING) + e = ValueError() + f.set_exception(e) + + with self.assertRaisesRegex( + futures.InvalidStateError, + 'FINISHED: ' + ): + f.set_exception(Exception()) + + self.assertEqual(f.exception(), e) + + +def setUpModule(): + setup_module() + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_concurrent_futures/test_init.py b/Lib/test/test_concurrent_futures/test_init.py new file mode 100644 index 00000000000..df640929309 --- /dev/null +++ b/Lib/test/test_concurrent_futures/test_init.py @@ -0,0 +1,152 @@ +import contextlib +import logging +import queue +import time +import unittest +import sys +import io +from concurrent.futures._base import BrokenExecutor +from concurrent.futures.process import _check_system_limits + +from logging.handlers import QueueHandler + +from test import support + +from .util import ExecutorMixin, create_executor_tests, setup_module + + +INITIALIZER_STATUS = 'uninitialized' + +def init(x): + global INITIALIZER_STATUS + INITIALIZER_STATUS = x + +def get_init_status(): + return INITIALIZER_STATUS + +def init_fail(log_queue=None): + if log_queue is not None: + logger = logging.getLogger('concurrent.futures') + logger.addHandler(QueueHandler(log_queue)) + logger.setLevel('CRITICAL') + logger.propagate = False + time.sleep(0.1) # let some futures be scheduled + raise ValueError('error in initializer') + + +class InitializerMixin(ExecutorMixin): + worker_count = 2 + + def setUp(self): + global INITIALIZER_STATUS + INITIALIZER_STATUS = 'uninitialized' + self.executor_kwargs = dict(initializer=init, + initargs=('initialized',)) + super().setUp() + + def test_initializer(self): + futures = [self.executor.submit(get_init_status) + for _ in range(self.worker_count)] + + for f in futures: + self.assertEqual(f.result(), 'initialized') + + +class FailingInitializerMixin(ExecutorMixin): + worker_count = 2 + + def setUp(self): + if hasattr(self, "ctx"): + # Pass a queue to redirect the child's logging output + self.mp_context = self.get_context() + self.log_queue = self.mp_context.Queue() + self.executor_kwargs = dict(initializer=init_fail, + initargs=(self.log_queue,)) + else: + # In a thread pool, the child shares our logging setup + # (see _assert_logged()) + self.mp_context = None + self.log_queue = None + self.executor_kwargs = dict(initializer=init_fail) + super().setUp() + + def test_initializer(self): + with self._assert_logged('ValueError: error in initializer'): + try: + future = self.executor.submit(get_init_status) + except BrokenExecutor: + # Perhaps the executor is already broken + pass + else: + with self.assertRaises(BrokenExecutor): + future.result() + + # At some point, the executor should break + for _ in support.sleeping_retry(support.SHORT_TIMEOUT, + "executor not broken"): + if self.executor._broken: + break + + # ... and from this point submit() is guaranteed to fail + with self.assertRaises(BrokenExecutor): + self.executor.submit(get_init_status) + + @contextlib.contextmanager + def _assert_logged(self, msg): + if self.log_queue is not None: + yield + output = [] + try: + while True: + output.append(self.log_queue.get_nowait().getMessage()) + except queue.Empty: + pass + else: + with self.assertLogs('concurrent.futures', 'CRITICAL') as cm: + yield + output = cm.output + self.assertTrue(any(msg in line for line in output), + output) + + +create_executor_tests(globals(), InitializerMixin) +create_executor_tests(globals(), FailingInitializerMixin) + + +@unittest.skipIf(sys.platform == "win32", "Resource Tracker doesn't run on Windows") +class FailingInitializerResourcesTest(unittest.TestCase): + """ + Source: https://github.com/python/cpython/issues/104090 + """ + + def _test(self, test_class): + try: + _check_system_limits() + except NotImplementedError: + self.skipTest("ProcessPoolExecutor unavailable on this system") + + runner = unittest.TextTestRunner(stream=io.StringIO()) + runner.run(test_class('test_initializer')) + + # GH-104090: + # Stop resource tracker manually now, so we can verify there are not leaked resources by checking + # the process exit code + from multiprocessing.resource_tracker import _resource_tracker + _resource_tracker._stop() + + self.assertEqual(_resource_tracker._exitcode, 0) + + def test_spawn(self): + self._test(ProcessPoolSpawnFailingInitializerTest) + + @support.skip_if_sanitizer("TSAN doesn't support threads after fork", thread=True) + def test_forkserver(self): + self._test(ProcessPoolForkserverFailingInitializerTest) + + +def setUpModule(): + setup_module() + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_concurrent_futures/test_process_pool.py b/Lib/test/test_concurrent_futures/test_process_pool.py new file mode 100644 index 00000000000..46dbdedd435 --- /dev/null +++ b/Lib/test/test_concurrent_futures/test_process_pool.py @@ -0,0 +1,233 @@ +import os +import sys +import threading +import time +import unittest +from concurrent import futures +from concurrent.futures.process import BrokenProcessPool + +from test import support +from test.support import hashlib_helper + +from .executor import ExecutorTest, mul +from .util import ( + ProcessPoolForkMixin, ProcessPoolForkserverMixin, ProcessPoolSpawnMixin, + create_executor_tests, setup_module) + + +class EventfulGCObj(): + def __init__(self, mgr): + self.event = mgr.Event() + + def __del__(self): + self.event.set() + + +class ProcessPoolExecutorTest(ExecutorTest): + + @unittest.skipUnless(sys.platform=='win32', 'Windows-only process limit') + def test_max_workers_too_large(self): + with self.assertRaisesRegex(ValueError, + "max_workers must be <= 61"): + futures.ProcessPoolExecutor(max_workers=62) + + def test_killed_child(self): + # When a child process is abruptly terminated, the whole pool gets + # "broken". + futures = [self.executor.submit(time.sleep, 3)] + # Get one of the processes, and terminate (kill) it + p = next(iter(self.executor._processes.values())) + p.terminate() + for fut in futures: + self.assertRaises(BrokenProcessPool, fut.result) + # Submitting other jobs fails as well. + self.assertRaises(BrokenProcessPool, self.executor.submit, pow, 2, 8) + + def test_map_chunksize(self): + def bad_map(): + list(self.executor.map(pow, range(40), range(40), chunksize=-1)) + + ref = list(map(pow, range(40), range(40))) + self.assertEqual( + list(self.executor.map(pow, range(40), range(40), chunksize=6)), + ref) + self.assertEqual( + list(self.executor.map(pow, range(40), range(40), chunksize=50)), + ref) + self.assertEqual( + list(self.executor.map(pow, range(40), range(40), chunksize=40)), + ref) + self.assertRaises(ValueError, bad_map) + + @classmethod + def _test_traceback(cls): + raise RuntimeError(123) # some comment + + def test_traceback(self): + # We want ensure that the traceback from the child process is + # contained in the traceback raised in the main process. + future = self.executor.submit(self._test_traceback) + with self.assertRaises(Exception) as cm: + future.result() + + exc = cm.exception + self.assertIs(type(exc), RuntimeError) + self.assertEqual(exc.args, (123,)) + cause = exc.__cause__ + self.assertIs(type(cause), futures.process._RemoteTraceback) + self.assertIn('raise RuntimeError(123) # some comment', cause.tb) + + with support.captured_stderr() as f1: + try: + raise exc + except RuntimeError: + sys.excepthook(*sys.exc_info()) + self.assertIn('raise RuntimeError(123) # some comment', + f1.getvalue()) + + @unittest.skipIf(sys.platform == 'linux', 'TODO: RUSTPYTHON flaky EOFError') + @hashlib_helper.requires_hashdigest('md5') + def test_ressources_gced_in_workers(self): + # Ensure that argument for a job are correctly gc-ed after the job + # is finished + mgr = self.get_context().Manager() + obj = EventfulGCObj(mgr) + future = self.executor.submit(id, obj) + future.result() + + self.assertTrue(obj.event.wait(timeout=1)) + + # explicitly destroy the object to ensure that EventfulGCObj.__del__() + # is called while manager is still running. + support.gc_collect() + obj = None + support.gc_collect() + + mgr.shutdown() + mgr.join() + + def test_saturation(self): + executor = self.executor + mp_context = self.get_context() + sem = mp_context.Semaphore(0) + job_count = 15 * executor._max_workers + for _ in range(job_count): + executor.submit(sem.acquire) + self.assertEqual(len(executor._processes), executor._max_workers) + for _ in range(job_count): + sem.release() + + @support.requires_gil_enabled("gh-117344: test is flaky without the GIL") + def test_idle_process_reuse_one(self): + executor = self.executor + assert executor._max_workers >= 4 + if self.get_context().get_start_method(allow_none=False) == "fork": + raise unittest.SkipTest("Incompatible with the fork start method.") + executor.submit(mul, 21, 2).result() + executor.submit(mul, 6, 7).result() + executor.submit(mul, 3, 14).result() + self.assertEqual(len(executor._processes), 1) + + def test_idle_process_reuse_multiple(self): + executor = self.executor + assert executor._max_workers <= 5 + if self.get_context().get_start_method(allow_none=False) == "fork": + raise unittest.SkipTest("Incompatible with the fork start method.") + executor.submit(mul, 12, 7).result() + executor.submit(mul, 33, 25) + executor.submit(mul, 25, 26).result() + executor.submit(mul, 18, 29) + executor.submit(mul, 1, 2).result() + executor.submit(mul, 0, 9) + self.assertLessEqual(len(executor._processes), 3) + executor.shutdown() + + def test_max_tasks_per_child(self): + context = self.get_context() + if context.get_start_method(allow_none=False) == "fork": + with self.assertRaises(ValueError): + self.executor_type(1, mp_context=context, max_tasks_per_child=3) + return + # not using self.executor as we need to control construction. + # arguably this could go in another class w/o that mixin. + executor = self.executor_type( + 1, mp_context=context, max_tasks_per_child=3) + f1 = executor.submit(os.getpid) + original_pid = f1.result() + # The worker pid remains the same as the worker could be reused + f2 = executor.submit(os.getpid) + self.assertEqual(f2.result(), original_pid) + self.assertEqual(len(executor._processes), 1) + f3 = executor.submit(os.getpid) + self.assertEqual(f3.result(), original_pid) + + # A new worker is spawned, with a statistically different pid, + # while the previous was reaped. + f4 = executor.submit(os.getpid) + new_pid = f4.result() + self.assertNotEqual(original_pid, new_pid) + self.assertEqual(len(executor._processes), 1) + + executor.shutdown() + + def test_max_tasks_per_child_defaults_to_spawn_context(self): + # not using self.executor as we need to control construction. + # arguably this could go in another class w/o that mixin. + executor = self.executor_type(1, max_tasks_per_child=3) + self.assertEqual(executor._mp_context.get_start_method(), "spawn") + + def test_max_tasks_early_shutdown(self): + context = self.get_context() + if context.get_start_method(allow_none=False) == "fork": + raise unittest.SkipTest("Incompatible with the fork start method.") + # not using self.executor as we need to control construction. + # arguably this could go in another class w/o that mixin. + executor = self.executor_type( + 3, mp_context=context, max_tasks_per_child=1) + futures = [] + for i in range(6): + futures.append(executor.submit(mul, i, i)) + executor.shutdown() + for i, future in enumerate(futures): + self.assertEqual(future.result(), mul(i, i)) + + def test_python_finalization_error(self): + # gh-109047: Catch RuntimeError on thread creation + # during Python finalization. + + context = self.get_context() + + # gh-109047: Mock the threading.start_joinable_thread() function to inject + # RuntimeError: simulate the error raised during Python finalization. + # Block the second creation: create _ExecutorManagerThread, but block + # QueueFeederThread. + orig_start_new_thread = threading._start_joinable_thread + nthread = 0 + def mock_start_new_thread(func, *args, **kwargs): + nonlocal nthread + if nthread >= 1: + raise RuntimeError("can't create new thread at " + "interpreter shutdown") + nthread += 1 + return orig_start_new_thread(func, *args, **kwargs) + + with support.swap_attr(threading, '_start_joinable_thread', + mock_start_new_thread): + executor = self.executor_type(max_workers=2, mp_context=context) + with executor: + with self.assertRaises(BrokenProcessPool): + list(executor.map(mul, [(2, 3)] * 10)) + executor.shutdown() + + +create_executor_tests(globals(), ProcessPoolExecutorTest, + executor_mixins=(ProcessPoolForkMixin, + ProcessPoolForkserverMixin, + ProcessPoolSpawnMixin)) + + +def setUpModule(): + setup_module() + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_concurrent_futures/test_shutdown.py b/Lib/test/test_concurrent_futures/test_shutdown.py new file mode 100644 index 00000000000..820ea6cf253 --- /dev/null +++ b/Lib/test/test_concurrent_futures/test_shutdown.py @@ -0,0 +1,410 @@ +import signal +import sys +import threading +import time +import unittest +from concurrent import futures + +from test import support +from test.support.script_helper import assert_python_ok + +from .util import ( + BaseTestCase, ThreadPoolMixin, ProcessPoolForkMixin, + ProcessPoolForkserverMixin, ProcessPoolSpawnMixin, + create_executor_tests, setup_module) + + +def sleep_and_print(t, msg): + time.sleep(t) + print(msg) + sys.stdout.flush() + + +class ExecutorShutdownTest: + def test_run_after_shutdown(self): + self.executor.shutdown() + self.assertRaises(RuntimeError, + self.executor.submit, + pow, 2, 5) + + @unittest.skip('TODO: RUSTPYTHON; hangs') + def test_interpreter_shutdown(self): + # Test the atexit hook for shutdown of worker threads and processes + rc, out, err = assert_python_ok('-c', """if 1: + from concurrent.futures import {executor_type} + from time import sleep + from test.test_concurrent_futures.test_shutdown import sleep_and_print + if __name__ == "__main__": + context = '{context}' + if context == "": + t = {executor_type}(5) + else: + from multiprocessing import get_context + context = get_context(context) + t = {executor_type}(5, mp_context=context) + t.submit(sleep_and_print, 1.0, "apple") + """.format(executor_type=self.executor_type.__name__, + context=getattr(self, "ctx", ""))) + # Errors in atexit hooks don't change the process exit code, check + # stderr manually. + self.assertFalse(err) + self.assertEqual(out.strip(), b"apple") + + @unittest.skip('TODO: RUSTPYTHON; Hangs') + def test_submit_after_interpreter_shutdown(self): + # Test the atexit hook for shutdown of worker threads and processes + rc, out, err = assert_python_ok('-c', """if 1: + import atexit + @atexit.register + def run_last(): + try: + t.submit(id, None) + except RuntimeError: + print("runtime-error") + raise + from concurrent.futures import {executor_type} + if __name__ == "__main__": + context = '{context}' + if not context: + t = {executor_type}(5) + else: + from multiprocessing import get_context + context = get_context(context) + t = {executor_type}(5, mp_context=context) + t.submit(id, 42).result() + """.format(executor_type=self.executor_type.__name__, + context=getattr(self, "ctx", ""))) + # Errors in atexit hooks don't change the process exit code, check + # stderr manually. + self.assertIn("RuntimeError: cannot schedule new futures", err.decode()) + self.assertEqual(out.strip(), b"runtime-error") + + def test_hang_issue12364(self): + fs = [self.executor.submit(time.sleep, 0.1) for _ in range(50)] + self.executor.shutdown() + for f in fs: + f.result() + + def test_cancel_futures(self): + assert self.worker_count <= 5, "test needs few workers" + fs = [self.executor.submit(time.sleep, .1) for _ in range(50)] + self.executor.shutdown(cancel_futures=True) + # We can't guarantee the exact number of cancellations, but we can + # guarantee that *some* were cancelled. With few workers, many of + # the submitted futures should have been cancelled. + cancelled = [fut for fut in fs if fut.cancelled()] + self.assertGreater(len(cancelled), 20) + + # Ensure the other futures were able to finish. + # Use "not fut.cancelled()" instead of "fut.done()" to include futures + # that may have been left in a pending state. + others = [fut for fut in fs if not fut.cancelled()] + for fut in others: + self.assertTrue(fut.done(), msg=f"{fut._state=}") + self.assertIsNone(fut.exception()) + + # Similar to the number of cancelled futures, we can't guarantee the + # exact number that completed. But, we can guarantee that at least + # one finished. + self.assertGreater(len(others), 0) + + def test_hang_gh83386(self): + """shutdown(wait=False) doesn't hang at exit with running futures. + + See https://github.com/python/cpython/issues/83386. + """ + if self.executor_type == futures.ProcessPoolExecutor: + raise unittest.SkipTest( + "Hangs, see https://github.com/python/cpython/issues/83386") + + rc, out, err = assert_python_ok('-c', """if True: + from concurrent.futures import {executor_type} + from test.test_concurrent_futures.test_shutdown import sleep_and_print + if __name__ == "__main__": + if {context!r}: multiprocessing.set_start_method({context!r}) + t = {executor_type}(max_workers=3) + t.submit(sleep_and_print, 1.0, "apple") + t.shutdown(wait=False) + """.format(executor_type=self.executor_type.__name__, + context=getattr(self, 'ctx', None))) + self.assertFalse(err) + self.assertEqual(out.strip(), b"apple") + + def test_hang_gh94440(self): + """shutdown(wait=True) doesn't hang when a future was submitted and + quickly canceled right before shutdown. + + See https://github.com/python/cpython/issues/94440. + """ + if not hasattr(signal, 'alarm'): + raise unittest.SkipTest( + "Tested platform does not support the alarm signal") + + def timeout(_signum, _frame): + raise RuntimeError("timed out waiting for shutdown") + + kwargs = {} + if getattr(self, 'ctx', None): + kwargs['mp_context'] = self.get_context() + executor = self.executor_type(max_workers=1, **kwargs) + executor.submit(int).result() + old_handler = signal.signal(signal.SIGALRM, timeout) + try: + signal.alarm(5) + executor.submit(int).cancel() + executor.shutdown(wait=True) + finally: + signal.alarm(0) + signal.signal(signal.SIGALRM, old_handler) + + +class ThreadPoolShutdownTest(ThreadPoolMixin, ExecutorShutdownTest, BaseTestCase): + def test_threads_terminate(self): + def acquire_lock(lock): + lock.acquire() + + sem = threading.Semaphore(0) + for i in range(3): + self.executor.submit(acquire_lock, sem) + self.assertEqual(len(self.executor._threads), 3) + for i in range(3): + sem.release() + self.executor.shutdown() + for t in self.executor._threads: + t.join() + + def test_context_manager_shutdown(self): + with futures.ThreadPoolExecutor(max_workers=5) as e: + executor = e + self.assertEqual(list(e.map(abs, range(-5, 5))), + [5, 4, 3, 2, 1, 0, 1, 2, 3, 4]) + + for t in executor._threads: + t.join() + + def test_del_shutdown(self): + executor = futures.ThreadPoolExecutor(max_workers=5) + res = executor.map(abs, range(-5, 5)) + threads = executor._threads + del executor + + for t in threads: + t.join() + + # Make sure the results were all computed before the + # executor got shutdown. + assert all([r == abs(v) for r, v in zip(res, range(-5, 5))]) + + def test_shutdown_no_wait(self): + # Ensure that the executor cleans up the threads when calling + # shutdown with wait=False + executor = futures.ThreadPoolExecutor(max_workers=5) + res = executor.map(abs, range(-5, 5)) + threads = executor._threads + executor.shutdown(wait=False) + for t in threads: + t.join() + + # Make sure the results were all computed before the + # executor got shutdown. + assert all([r == abs(v) for r, v in zip(res, range(-5, 5))]) + + + def test_thread_names_assigned(self): + executor = futures.ThreadPoolExecutor( + max_workers=5, thread_name_prefix='SpecialPool') + executor.map(abs, range(-5, 5)) + threads = executor._threads + del executor + support.gc_collect() # For PyPy or other GCs. + + for t in threads: + self.assertRegex(t.name, r'^SpecialPool_[0-4]$') + t.join() + + def test_thread_names_default(self): + executor = futures.ThreadPoolExecutor(max_workers=5) + executor.map(abs, range(-5, 5)) + threads = executor._threads + del executor + support.gc_collect() # For PyPy or other GCs. + + for t in threads: + # Ensure that our default name is reasonably sane and unique when + # no thread_name_prefix was supplied. + self.assertRegex(t.name, r'ThreadPoolExecutor-\d+_[0-4]$') + t.join() + + def test_cancel_futures_wait_false(self): + # Can only be reliably tested for TPE, since PPE often hangs with + # `wait=False` (even without *cancel_futures*). + rc, out, err = assert_python_ok('-c', """if True: + from concurrent.futures import ThreadPoolExecutor + from test.test_concurrent_futures.test_shutdown import sleep_and_print + if __name__ == "__main__": + t = ThreadPoolExecutor() + t.submit(sleep_and_print, .1, "apple") + t.shutdown(wait=False, cancel_futures=True) + """) + # Errors in atexit hooks don't change the process exit code, check + # stderr manually. + self.assertFalse(err) + # gh-116682: stdout may be empty if shutdown happens before task + # starts executing. + self.assertIn(out.strip(), [b"apple", b""]) + + +class ProcessPoolShutdownTest(ExecutorShutdownTest): + # TODO: RUSTPYTHON - flaky, dict changed size during iteration race condition + @unittest.skip("TODO: RUSTPYTHON - flaky race condition on macOS") + def test_cancel_futures(self): + return super().test_cancel_futures() + + def test_processes_terminate(self): + def acquire_lock(lock): + lock.acquire() + + mp_context = self.get_context() + if mp_context.get_start_method(allow_none=False) == "fork": + # fork pre-spawns, not on demand. + expected_num_processes = self.worker_count + else: + expected_num_processes = 3 + + sem = mp_context.Semaphore(0) + for _ in range(3): + self.executor.submit(acquire_lock, sem) + self.assertEqual(len(self.executor._processes), expected_num_processes) + for _ in range(3): + sem.release() + processes = self.executor._processes + self.executor.shutdown() + + for p in processes.values(): + p.join() + + def test_context_manager_shutdown(self): + with futures.ProcessPoolExecutor( + max_workers=5, mp_context=self.get_context()) as e: + processes = e._processes + self.assertEqual(list(e.map(abs, range(-5, 5))), + [5, 4, 3, 2, 1, 0, 1, 2, 3, 4]) + + for p in processes.values(): + p.join() + + def test_del_shutdown(self): + executor = futures.ProcessPoolExecutor( + max_workers=5, mp_context=self.get_context()) + res = executor.map(abs, range(-5, 5)) + executor_manager_thread = executor._executor_manager_thread + processes = executor._processes + call_queue = executor._call_queue + executor_manager_thread = executor._executor_manager_thread + del executor + support.gc_collect() # For PyPy or other GCs. + + # Make sure that all the executor resources were properly cleaned by + # the shutdown process + executor_manager_thread.join() + for p in processes.values(): + p.join() + call_queue.join_thread() + + # Make sure the results were all computed before the + # executor got shutdown. + assert all([r == abs(v) for r, v in zip(res, range(-5, 5))]) + + def test_shutdown_no_wait(self): + # Ensure that the executor cleans up the processes when calling + # shutdown with wait=False + executor = futures.ProcessPoolExecutor( + max_workers=5, mp_context=self.get_context()) + res = executor.map(abs, range(-5, 5)) + processes = executor._processes + call_queue = executor._call_queue + executor_manager_thread = executor._executor_manager_thread + executor.shutdown(wait=False) + + # Make sure that all the executor resources were properly cleaned by + # the shutdown process + executor_manager_thread.join() + for p in processes.values(): + p.join() + call_queue.join_thread() + + # Make sure the results were all computed before the executor got + # shutdown. + assert all([r == abs(v) for r, v in zip(res, range(-5, 5))]) + + @classmethod + def _failing_task_gh_132969(cls, n): + raise ValueError("failing task") + + @classmethod + def _good_task_gh_132969(cls, n): + time.sleep(0.1 * n) + return n + + def _run_test_issue_gh_132969(self, max_workers): + # max_workers=2 will repro exception + # max_workers=4 will repro exception and then hang + + # Repro conditions + # max_tasks_per_child=1 + # a task ends abnormally + # shutdown(wait=False) is called + start_method = self.get_context().get_start_method() + if (start_method == "fork" or + (start_method == "forkserver" and sys.platform.startswith("win"))): + self.skipTest(f"Skipping test for {start_method = }") + executor = futures.ProcessPoolExecutor( + max_workers=max_workers, + max_tasks_per_child=1, + mp_context=self.get_context()) + f1 = executor.submit(ProcessPoolShutdownTest._good_task_gh_132969, 1) + f2 = executor.submit(ProcessPoolShutdownTest._failing_task_gh_132969, 2) + f3 = executor.submit(ProcessPoolShutdownTest._good_task_gh_132969, 3) + result = 0 + try: + result += f1.result() + result += f2.result() + result += f3.result() + except ValueError: + # stop processing results upon first exception + pass + + # Ensure that the executor cleans up after called + # shutdown with wait=False + executor_manager_thread = executor._executor_manager_thread + executor.shutdown(wait=False) + time.sleep(0.2) + executor_manager_thread.join() + return result + + def test_shutdown_gh_132969_case_1(self): + # gh-132969: test that exception "object of type 'NoneType' has no len()" + # is not raised when shutdown(wait=False) is called. + result = self._run_test_issue_gh_132969(2) + self.assertEqual(result, 1) + + def test_shutdown_gh_132969_case_2(self): + # gh-132969: test that process does not hang and + # exception "object of type 'NoneType' has no len()" is not raised + # when shutdown(wait=False) is called. + result = self._run_test_issue_gh_132969(4) + self.assertEqual(result, 1) + + +create_executor_tests(globals(), ProcessPoolShutdownTest, + executor_mixins=(ProcessPoolForkMixin, + ProcessPoolForkserverMixin, + ProcessPoolSpawnMixin)) + + +def setUpModule(): + setup_module() + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_concurrent_futures/test_thread_pool.py b/Lib/test/test_concurrent_futures/test_thread_pool.py new file mode 100644 index 00000000000..4324241b374 --- /dev/null +++ b/Lib/test/test_concurrent_futures/test_thread_pool.py @@ -0,0 +1,121 @@ +import contextlib +import multiprocessing as mp +import multiprocessing.process +import multiprocessing.util +import os +import threading +import unittest +from concurrent import futures +from test import support + +from .executor import ExecutorTest, mul +from .util import BaseTestCase, ThreadPoolMixin, setup_module + + +class ThreadPoolExecutorTest(ThreadPoolMixin, ExecutorTest, BaseTestCase): + def test_map_submits_without_iteration(self): + """Tests verifying issue 11777.""" + finished = [] + def record_finished(n): + finished.append(n) + + self.executor.map(record_finished, range(10)) + self.executor.shutdown(wait=True) + self.assertCountEqual(finished, range(10)) + + def test_default_workers(self): + executor = self.executor_type() + expected = min(32, (os.process_cpu_count() or 1) + 4) + self.assertEqual(executor._max_workers, expected) + + def test_saturation(self): + executor = self.executor_type(4) + def acquire_lock(lock): + lock.acquire() + + sem = threading.Semaphore(0) + for i in range(15 * executor._max_workers): + executor.submit(acquire_lock, sem) + self.assertEqual(len(executor._threads), executor._max_workers) + for i in range(15 * executor._max_workers): + sem.release() + executor.shutdown(wait=True) + + @support.requires_gil_enabled("gh-117344: test is flaky without the GIL") + def test_idle_thread_reuse(self): + executor = self.executor_type() + executor.submit(mul, 21, 2).result() + executor.submit(mul, 6, 7).result() + executor.submit(mul, 3, 14).result() + self.assertEqual(len(executor._threads), 1) + executor.shutdown(wait=True) + + @support.requires_fork() + @unittest.skipUnless(hasattr(os, 'register_at_fork'), 'need os.register_at_fork') + @support.requires_resource('cpu') + def test_hang_global_shutdown_lock(self): + # bpo-45021: _global_shutdown_lock should be reinitialized in the child + # process, otherwise it will never exit + def submit(pool): + pool.submit(submit, pool) + + with futures.ThreadPoolExecutor(1) as pool: + pool.submit(submit, pool) + + for _ in range(50): + with futures.ProcessPoolExecutor(1, mp_context=mp.get_context('fork')) as workers: + workers.submit(tuple) + + @support.requires_fork() + @unittest.skipUnless(hasattr(os, 'register_at_fork'), 'need os.register_at_fork') + def test_process_fork_from_a_threadpool(self): + # bpo-43944: clear concurrent.futures.thread._threads_queues after fork, + # otherwise child process will try to join parent thread + def fork_process_and_return_exitcode(): + # Ignore the warning about fork with threads. + with self.assertWarnsRegex(DeprecationWarning, + r"use of fork\(\) may lead to deadlocks in the child"): + p = mp.get_context('fork').Process(target=lambda: 1) + p.start() + p.join() + return p.exitcode + + with futures.ThreadPoolExecutor(1) as pool: + process_exitcode = pool.submit(fork_process_and_return_exitcode).result() + + self.assertEqual(process_exitcode, 0) + + def test_executor_map_current_future_cancel(self): + stop_event = threading.Event() + log = [] + + def log_n_wait(ident): + log.append(f"{ident=} started") + try: + stop_event.wait() + finally: + log.append(f"{ident=} stopped") + + with self.executor_type(max_workers=1) as pool: + # submit work to saturate the pool + fut = pool.submit(log_n_wait, ident="first") + try: + with contextlib.closing( + pool.map(log_n_wait, ["second", "third"], timeout=0) + ) as gen: + with self.assertRaises(TimeoutError): + next(gen) + finally: + stop_event.set() + fut.result() + # ident='second' is cancelled as a result of raising a TimeoutError + # ident='third' is cancelled because it remained in the collection of futures + self.assertListEqual(log, ["ident='first' started", "ident='first' stopped"]) + + +def setUpModule(): + setup_module() + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_concurrent_futures/test_wait.py b/Lib/test/test_concurrent_futures/test_wait.py new file mode 100644 index 00000000000..7a7857671a4 --- /dev/null +++ b/Lib/test/test_concurrent_futures/test_wait.py @@ -0,0 +1,219 @@ +import sys +import threading +import unittest +from concurrent import futures +from test import support +from test.support import threading_helper + +from .util import ( + CANCELLED_FUTURE, CANCELLED_AND_NOTIFIED_FUTURE, EXCEPTION_FUTURE, + SUCCESSFUL_FUTURE, + create_executor_tests, setup_module, + BaseTestCase, ThreadPoolMixin, + ProcessPoolForkMixin, ProcessPoolForkserverMixin, ProcessPoolSpawnMixin) + + +def mul(x, y): + return x * y + +def wait_and_raise(e): + e.wait() + raise Exception('this is an exception') + + +class WaitTests: + def test_20369(self): + # See https://bugs.python.org/issue20369 + future = self.executor.submit(mul, 1, 2) + done, not_done = futures.wait([future, future], + return_when=futures.ALL_COMPLETED) + self.assertEqual({future}, done) + self.assertEqual(set(), not_done) + + + def test_first_completed(self): + event = self.create_event() + future1 = self.executor.submit(mul, 21, 2) + future2 = self.executor.submit(event.wait) + + try: + done, not_done = futures.wait( + [CANCELLED_FUTURE, future1, future2], + return_when=futures.FIRST_COMPLETED) + + self.assertEqual(set([future1]), done) + self.assertEqual(set([CANCELLED_FUTURE, future2]), not_done) + finally: + event.set() + future2.result() # wait for job to finish + + def test_first_completed_some_already_completed(self): + event = self.create_event() + future1 = self.executor.submit(event.wait) + + try: + finished, pending = futures.wait( + [CANCELLED_AND_NOTIFIED_FUTURE, SUCCESSFUL_FUTURE, future1], + return_when=futures.FIRST_COMPLETED) + + self.assertEqual( + set([CANCELLED_AND_NOTIFIED_FUTURE, SUCCESSFUL_FUTURE]), + finished) + self.assertEqual(set([future1]), pending) + finally: + event.set() + future1.result() # wait for job to finish + + def test_first_exception(self): + event1 = self.create_event() + event2 = self.create_event() + try: + future1 = self.executor.submit(mul, 2, 21) + future2 = self.executor.submit(wait_and_raise, event1) + future3 = self.executor.submit(event2.wait) + + # Ensure that future1 is completed before future2 finishes + def wait_for_future1(): + future1.result() + event1.set() + + t = threading.Thread(target=wait_for_future1) + t.start() + + finished, pending = futures.wait( + [future1, future2, future3], + return_when=futures.FIRST_EXCEPTION) + + self.assertEqual(set([future1, future2]), finished) + self.assertEqual(set([future3]), pending) + + threading_helper.join_thread(t) + finally: + event1.set() + event2.set() + future3.result() # wait for job to finish + + def test_first_exception_some_already_complete(self): + event = self.create_event() + future1 = self.executor.submit(divmod, 21, 0) + future2 = self.executor.submit(event.wait) + + try: + finished, pending = futures.wait( + [SUCCESSFUL_FUTURE, + CANCELLED_FUTURE, + CANCELLED_AND_NOTIFIED_FUTURE, + future1, future2], + return_when=futures.FIRST_EXCEPTION) + + self.assertEqual(set([SUCCESSFUL_FUTURE, + CANCELLED_AND_NOTIFIED_FUTURE, + future1]), finished) + self.assertEqual(set([CANCELLED_FUTURE, future2]), pending) + finally: + event.set() + future2.result() # wait for job to finish + + def test_first_exception_one_already_failed(self): + event = self.create_event() + future1 = self.executor.submit(event.wait) + + try: + finished, pending = futures.wait( + [EXCEPTION_FUTURE, future1], + return_when=futures.FIRST_EXCEPTION) + + self.assertEqual(set([EXCEPTION_FUTURE]), finished) + self.assertEqual(set([future1]), pending) + finally: + event.set() + future1.result() # wait for job to finish + + def test_all_completed(self): + future1 = self.executor.submit(divmod, 2, 0) + future2 = self.executor.submit(mul, 2, 21) + + finished, pending = futures.wait( + [SUCCESSFUL_FUTURE, + CANCELLED_AND_NOTIFIED_FUTURE, + EXCEPTION_FUTURE, + future1, + future2], + return_when=futures.ALL_COMPLETED) + + self.assertEqual(set([SUCCESSFUL_FUTURE, + CANCELLED_AND_NOTIFIED_FUTURE, + EXCEPTION_FUTURE, + future1, + future2]), finished) + self.assertEqual(set(), pending) + + def test_timeout(self): + short_timeout = 0.050 + + event = self.create_event() + future = self.executor.submit(event.wait) + + try: + finished, pending = futures.wait( + [CANCELLED_AND_NOTIFIED_FUTURE, + EXCEPTION_FUTURE, + SUCCESSFUL_FUTURE, + future], + timeout=short_timeout, + return_when=futures.ALL_COMPLETED) + + self.assertEqual(set([CANCELLED_AND_NOTIFIED_FUTURE, + EXCEPTION_FUTURE, + SUCCESSFUL_FUTURE]), + finished) + self.assertEqual(set([future]), pending) + finally: + event.set() + future.result() # wait for job to finish + + +class ThreadPoolWaitTests(ThreadPoolMixin, WaitTests, BaseTestCase): + + def test_pending_calls_race(self): + # Issue #14406: multi-threaded race condition when waiting on all + # futures. + event = threading.Event() + def future_func(): + event.wait() + oldswitchinterval = sys.getswitchinterval() + support.setswitchinterval(1e-6) + try: + fs = {self.executor.submit(future_func) for i in range(100)} + event.set() + futures.wait(fs, return_when=futures.ALL_COMPLETED) + finally: + sys.setswitchinterval(oldswitchinterval) + + +create_executor_tests(globals(), WaitTests, + executor_mixins=(ProcessPoolForkMixin, + ProcessPoolForkserverMixin, + ProcessPoolSpawnMixin)) + + +def setUpModule(): + setup_module() + +class ProcessPoolForkWaitTest(ProcessPoolForkWaitTest): # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', "TODO: RUSTPYTHON flaky") + def test_first_completed(self): super().test_first_completed() # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', "TODO: RUSTPYTHON Fatal Python error: Segmentation fault") + def test_first_completed_some_already_completed(self): super().test_first_completed_some_already_completed() # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', "TODO: RUSTPYTHON flaky") + def test_first_exception(self): super().test_first_exception() # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', "TODO: RUSTPYTHON flaky") + def test_first_exception_one_already_failed(self): super().test_first_exception_one_already_failed() # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', "TODO: RUSTPYTHON Fatal Python error: Segmentation fault") + def test_first_exception_some_already_complete(self): super().test_first_exception_some_already_complete() # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', "TODO: RUSTPYTHON Fatal Python error: Segmentation fault") + def test_timeout(self): super().test_timeout() # TODO: RUSTPYTHON + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_concurrent_futures/util.py b/Lib/test/test_concurrent_futures/util.py new file mode 100644 index 00000000000..e85ef3b1c91 --- /dev/null +++ b/Lib/test/test_concurrent_futures/util.py @@ -0,0 +1,169 @@ +import multiprocessing +import sys +import threading +import time +import unittest +from concurrent import futures +from concurrent.futures._base import ( + PENDING, RUNNING, CANCELLED, CANCELLED_AND_NOTIFIED, FINISHED, Future, + ) +from concurrent.futures.process import _check_system_limits + +from test import support +from test.support import threading_helper + + +def create_future(state=PENDING, exception=None, result=None): + f = Future() + f._state = state + f._exception = exception + f._result = result + return f + + +PENDING_FUTURE = create_future(state=PENDING) +RUNNING_FUTURE = create_future(state=RUNNING) +CANCELLED_FUTURE = create_future(state=CANCELLED) +CANCELLED_AND_NOTIFIED_FUTURE = create_future(state=CANCELLED_AND_NOTIFIED) +EXCEPTION_FUTURE = create_future(state=FINISHED, exception=OSError()) +SUCCESSFUL_FUTURE = create_future(state=FINISHED, result=42) + + +class BaseTestCase(unittest.TestCase): + def setUp(self): + self._thread_key = threading_helper.threading_setup() + + def tearDown(self): + support.reap_children() + threading_helper.threading_cleanup(*self._thread_key) + + +class ExecutorMixin: + worker_count = 5 + executor_kwargs = {} + + def setUp(self): + super().setUp() + + self.t1 = time.monotonic() + if hasattr(self, "ctx"): + self.executor = self.executor_type( + max_workers=self.worker_count, + mp_context=self.get_context(), + **self.executor_kwargs) + self.manager = self.get_context().Manager() + else: + self.executor = self.executor_type( + max_workers=self.worker_count, + **self.executor_kwargs) + self.manager = None + + def tearDown(self): + self.executor.shutdown(wait=True) + self.executor = None + if self.manager is not None: + self.manager.shutdown() + self.manager = None + + dt = time.monotonic() - self.t1 + if support.verbose: + print("%.2fs" % dt, end=' ') + self.assertLess(dt, 300, "synchronization issue: test lasted too long") + + super().tearDown() + + def get_context(self): + return multiprocessing.get_context(self.ctx) + + +class ThreadPoolMixin(ExecutorMixin): + executor_type = futures.ThreadPoolExecutor + + def create_event(self): + return threading.Event() + + +class ProcessPoolForkMixin(ExecutorMixin): + executor_type = futures.ProcessPoolExecutor + ctx = "fork" + + def get_context(self): + try: + _check_system_limits() + except NotImplementedError: + self.skipTest("ProcessPoolExecutor unavailable on this system") + if sys.platform == "win32": + self.skipTest("require unix system") + if support.check_sanitizer(thread=True): + self.skipTest("TSAN doesn't support threads after fork") + return super().get_context() + + def create_event(self): + return self.manager.Event() + + +class ProcessPoolSpawnMixin(ExecutorMixin): + executor_type = futures.ProcessPoolExecutor + ctx = "spawn" + + def get_context(self): + try: + _check_system_limits() + except NotImplementedError: + self.skipTest("ProcessPoolExecutor unavailable on this system") + return super().get_context() + + def create_event(self): + return self.manager.Event() + + +class ProcessPoolForkserverMixin(ExecutorMixin): + executor_type = futures.ProcessPoolExecutor + ctx = "forkserver" + + def get_context(self): + try: + _check_system_limits() + except NotImplementedError: + self.skipTest("ProcessPoolExecutor unavailable on this system") + if sys.platform == "win32": + self.skipTest("require unix system") + if support.check_sanitizer(thread=True): + self.skipTest("TSAN doesn't support threads after fork") + return super().get_context() + + def create_event(self): + return self.manager.Event() + + +def create_executor_tests(remote_globals, mixin, bases=(BaseTestCase,), + executor_mixins=(ThreadPoolMixin, + ProcessPoolForkMixin, + ProcessPoolForkserverMixin, + ProcessPoolSpawnMixin)): + def strip_mixin(name): + if name.endswith(('Mixin', 'Tests')): + return name[:-5] + elif name.endswith('Test'): + return name[:-4] + else: + return name + + module = remote_globals['__name__'] + for exe in executor_mixins: + name = ("%s%sTest" + % (strip_mixin(exe.__name__), strip_mixin(mixin.__name__))) + cls = type(name, (mixin,) + (exe,) + bases, {'__module__': module}) + remote_globals[name] = cls + + +def setup_module(): + try: + _check_system_limits() + except NotImplementedError: + pass + else: + unittest.addModuleCleanup(multiprocessing.util._cleanup_tests) + + thread_info = threading_helper.threading_setup() + unittest.addModuleCleanup(threading_helper.threading_cleanup, *thread_info) diff --git a/Lib/test/test_configparser.py b/Lib/test/test_configparser.py index d793cc58907..d95b504f826 100644 --- a/Lib/test/test_configparser.py +++ b/Lib/test/test_configparser.py @@ -762,7 +762,7 @@ def test_read_returns_file_list(self): parsed_files = cf.read([], encoding="utf-8") self.assertEqual(parsed_files, []) - @unittest.skip("TODO: RUSTPYTHON, suspected to make CI hang") + # XXX: RUSTPYTHON; This test might cause CI hang def test_read_returns_file_list_with_bytestring_path(self): if self.delimiters[0] != '=': self.skipTest('incompatible format') diff --git a/Lib/test/test_context.py b/Lib/test/test_context.py index 06270e161da..59d2320de85 100644 --- a/Lib/test/test_context.py +++ b/Lib/test/test_context.py @@ -3,6 +3,7 @@ import functools import gc import random +import sys import time import unittest import weakref @@ -216,8 +217,6 @@ def fun(): ctx.run(fun) - # TODO: RUSTPYTHON - @unittest.expectedFailure @isolated_context def test_context_getset_1(self): c = contextvars.ContextVar('c') @@ -316,8 +315,6 @@ def test_context_getset_4(self): with self.assertRaisesRegex(ValueError, 'different Context'): c.reset(tok) - # TODO: RUSTPYTHON - @unittest.expectedFailure @isolated_context def test_context_getset_5(self): c = contextvars.ContextVar('c', default=42) @@ -331,8 +328,6 @@ def fun(): contextvars.copy_context().run(fun) self.assertEqual(c.get(), []) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_context_copy_1(self): ctx1 = contextvars.Context() c = contextvars.ContextVar('c', default=42) @@ -358,9 +353,9 @@ def ctx2_fun(): ctx1.run(ctx1_fun) - @unittest.skip("TODO: RUSTPYTHON; threading is not safe") @isolated_context @threading_helper.requires_working_threading() + @unittest.skipIf(sys.platform == 'darwin', 'TODO: RUSTPYTHON; Flaky on Mac, self.assertEqual(cvar.get(), num + i) AssertionError: 8 != 12') def test_context_threads_1(self): cvar = contextvars.ContextVar('cvar') diff --git a/Lib/test/test_contextlib_async.py b/Lib/test/test_contextlib_async.py index 0cf32abb76f..a224bc68329 100644 --- a/Lib/test/test_contextlib_async.py +++ b/Lib/test/test_contextlib_async.py @@ -120,8 +120,7 @@ async def woohoo(): raise ZeroDivisionError() self.assertEqual(state, [1, 42, 999]) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON async def test_contextmanager_traceback(self): @asynccontextmanager async def f(): @@ -253,8 +252,7 @@ async def woohoo(): raise ZeroDivisionError(999) self.assertEqual(state, [1, 42, 999]) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON async def test_contextmanager_except_stopiter(self): @asynccontextmanager async def woohoo(): @@ -535,11 +533,6 @@ def __exit__(self, *exc_details): ('__aexit__', 'cb_suppress = cb(*exc_details)'), ] - # TODO: RUSTPYTHON - no _asyncio module, pure Python Task adds extra frame - @unittest.expectedFailure - def test_exit_exception_traceback(self): - super().test_exit_exception_traceback() - async def test_async_callback(self): expected = [ ((), {}), @@ -657,8 +650,7 @@ async def __aenter__(self): await stack.enter_async_context(LacksExit()) self.assertFalse(stack._exit_callbacks) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON async def test_async_exit_exception_chaining(self): # Ensure exception chaining matches the reference behaviour async def raise_exc(exc): @@ -690,8 +682,7 @@ async def suppress_exc(*exc_details): self.assertIsInstance(inner_exc, ValueError) self.assertIsInstance(inner_exc.__context__, ZeroDivisionError) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON async def test_async_exit_exception_explicit_none_context(self): # Ensure AsyncExitStack chaining matches actual nested `with` statements # regarding explicit __context__ = None. @@ -738,6 +729,7 @@ class Example(object): pass self.assertIs(stack._exit_callbacks[-1][1], cm) + class TestAsyncNullcontext(unittest.IsolatedAsyncioTestCase): async def test_async_nullcontext(self): class C: diff --git a/Lib/test/test_copy.py b/Lib/test/test_copy.py index 456767bbe0c..a065cc7b071 100644 --- a/Lib/test/test_copy.py +++ b/Lib/test/test_copy.py @@ -207,8 +207,6 @@ def __eq__(self, other): self.assertIsNot(y, x) self.assertEqual(y.foo, x.foo) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_copy_inst_getnewargs_ex(self): class C(int): def __new__(cls, *, foo): @@ -507,8 +505,6 @@ def __eq__(self, other): self.assertEqual(y.foo, x.foo) self.assertIsNot(y.foo, x.foo) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_deepcopy_inst_getnewargs_ex(self): class C(int): def __new__(cls, *, foo): @@ -841,12 +837,15 @@ class C(object): v[x] = y self.assertNotIn(x, u) + def test_copy_weakkeydict(self): self._check_copy_weakdict(weakref.WeakKeyDictionary) + def test_copy_weakvaluedict(self): self._check_copy_weakdict(weakref.WeakValueDictionary) + def test_deepcopy_weakkeydict(self): class C(object): def __init__(self, i): @@ -867,6 +866,7 @@ def __init__(self, i): support.gc_collect() # For PyPy or other GCs. self.assertEqual(len(v), 1) + def test_deepcopy_weakvaluedict(self): class C(object): def __init__(self, i): @@ -938,8 +938,6 @@ def __replace__(self, **changes): self.assertEqual(attrs(copy.replace(a, y=2)), (11, 2, 13)) self.assertEqual(attrs(copy.replace(a, x=1, y=2)), (1, 2, 3)) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_namedtuple(self): from collections import namedtuple from typing import NamedTuple diff --git a/Lib/test/test_coroutines.py b/Lib/test/test_coroutines.py new file mode 100644 index 00000000000..ea17d8d36a1 --- /dev/null +++ b/Lib/test/test_coroutines.py @@ -0,0 +1,2550 @@ +import contextlib +import copy +import inspect +import pickle +import sys +import types +import traceback +import unittest +import warnings +from test import support +from test.support import import_helper +from test.support import warnings_helper +from test.support.script_helper import assert_python_ok +try: + import _testcapi +except ImportError: + _testcapi = None + + +class AsyncYieldFrom: + def __init__(self, obj): + self.obj = obj + + def __await__(self): + yield from self.obj + + +class AsyncYield: + def __init__(self, value): + self.value = value + + def __await__(self): + yield self.value + + +async def asynciter(iterable): + """Convert an iterable to an asynchronous iterator.""" + for x in iterable: + yield x + + +def run_async(coro): + assert coro.__class__ in {types.GeneratorType, types.CoroutineType} + + buffer = [] + result = None + while True: + try: + buffer.append(coro.send(None)) + except StopIteration as ex: + result = ex.args[0] if ex.args else None + break + return buffer, result + + +def run_async__await__(coro): + assert coro.__class__ is types.CoroutineType + aw = coro.__await__() + buffer = [] + result = None + i = 0 + while True: + try: + if i % 2: + buffer.append(next(aw)) + else: + buffer.append(aw.send(None)) + i += 1 + except StopIteration as ex: + result = ex.args[0] if ex.args else None + break + return buffer, result + + +@contextlib.contextmanager +def silence_coro_gc(): + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + yield + support.gc_collect() + + +class AsyncBadSyntaxTest(unittest.TestCase): + + def test_badsyntax_1(self): + samples = [ + """def foo(): + await something() + """, + + """await something()""", + + """async def foo(): + yield from [] + """, + + """async def foo(): + await await fut + """, + + """async def foo(a=await something()): + pass + """, + + """async def foo(a:await something()): + pass + """, + + """async def foo(): + def bar(): + [i async for i in els] + """, + + """async def foo(): + def bar(): + [await i for i in els] + """, + + """async def foo(): + def bar(): + [i for i in els + async for b in els] + """, + + """async def foo(): + def bar(): + [i for i in els + for c in b + async for b in els] + """, + + """async def foo(): + def bar(): + [i for i in els + async for b in els + for c in b] + """, + + """async def foo(): + def bar(): + [[async for i in b] for b in els] + """, + + """async def foo(): + def bar(): + [i for i in els + for b in await els] + """, + + """async def foo(): + def bar(): + [i for i in els + for b in els + if await b] + """, + + """async def foo(): + def bar(): + [i for i in await els] + """, + + """async def foo(): + def bar(): + [i for i in els if await i] + """, + + """def bar(): + [i async for i in els] + """, + + """def bar(): + {i: i async for i in els} + """, + + """def bar(): + {i async for i in els} + """, + + """def bar(): + [await i for i in els] + """, + + """def bar(): + [i for i in els + async for b in els] + """, + + """def bar(): + [i for i in els + for c in b + async for b in els] + """, + + """def bar(): + [i for i in els + async for b in els + for c in b] + """, + + """def bar(): + [i for i in els + for b in await els] + """, + + """def bar(): + [i for i in els + for b in els + if await b] + """, + + """def bar(): + [i for i in await els] + """, + + """def bar(): + [i for i in els if await i] + """, + + """def bar(): + [[i async for i in a] for a in elts] + """, + + """[[i async for i in a] for a in elts] + """, + + """async def foo(): + await + """, + + """async def foo(): + def bar(): pass + await = 1 + """, + + """async def foo(): + + def bar(): pass + await = 1 + """, + + """async def foo(): + def bar(): pass + if 1: + await = 1 + """, + + """def foo(): + async def bar(): pass + if 1: + await a + """, + + """def foo(): + async def bar(): pass + await a + """, + + """def foo(): + def baz(): pass + async def bar(): pass + await a + """, + + """def foo(): + def baz(): pass + # 456 + async def bar(): pass + # 123 + await a + """, + + """async def foo(): + def baz(): pass + # 456 + async def bar(): pass + # 123 + await = 2 + """, + + """def foo(): + + def baz(): pass + + async def bar(): pass + + await a + """, + + """async def foo(): + + def baz(): pass + + async def bar(): pass + + await = 2 + """, + + """async def foo(): + def async(): pass + """, + + """async def foo(): + def await(): pass + """, + + """async def foo(): + def bar(): + await + """, + + """async def foo(): + return lambda async: await + """, + + """async def foo(): + return lambda a: await + """, + + """await a()""", + + """async def foo(a=await b): + pass + """, + + """async def foo(a:await b): + pass + """, + + """def baz(): + async def foo(a=await b): + pass + """, + + """async def foo(async): + pass + """, + + """async def foo(): + def bar(): + def baz(): + async = 1 + """, + + """async def foo(): + def bar(): + def baz(): + pass + async = 1 + """, + + """def foo(): + async def bar(): + + async def baz(): + pass + + def baz(): + 42 + + async = 1 + """, + + """async def foo(): + def bar(): + def baz(): + pass\nawait foo() + """, + + """def foo(): + def bar(): + async def baz(): + pass\nawait foo() + """, + + """async def foo(await): + pass + """, + + """def foo(): + + async def bar(): pass + + await a + """, + + """def foo(): + async def bar(): + pass\nawait a + """, + """def foo(): + async for i in arange(2): + pass + """, + """def foo(): + async with resource: + pass + """, + """async with resource: + pass + """, + """async for i in arange(2): + pass + """, + ] + + for code in samples: + with self.subTest(code=code), self.assertRaises(SyntaxError): + compile(code, "", "exec") + + def test_badsyntax_2(self): + samples = [ + """def foo(): + await = 1 + """, + + """class Bar: + def async(): pass + """, + + """class Bar: + async = 1 + """, + + """class async: + pass + """, + + """class await: + pass + """, + + """import math as await""", + + """def async(): + pass""", + + """def foo(*, await=1): + pass""" + + """async = 1""", + + """print(await=1)""" + ] + + for code in samples: + with self.subTest(code=code), self.assertRaises(SyntaxError): + compile(code, "", "exec") + + def test_badsyntax_3(self): + with self.assertRaises(SyntaxError): + compile("async = 1", "", "exec") + + def test_badsyntax_4(self): + samples = [ + '''def foo(await): + async def foo(): pass + async def foo(): + pass + return await + 1 + ''', + + '''def foo(await): + async def foo(): pass + async def foo(): pass + return await + 1 + ''', + + '''def foo(await): + + async def foo(): pass + + async def foo(): pass + + return await + 1 + ''', + + '''def foo(await): + """spam""" + async def foo(): \ + pass + # 123 + async def foo(): pass + # 456 + return await + 1 + ''', + + '''def foo(await): + def foo(): pass + def foo(): pass + async def bar(): return await_ + await_ = await + try: + bar().send(None) + except StopIteration as ex: + return ex.args[0] + 1 + ''' + ] + + for code in samples: + with self.subTest(code=code), self.assertRaises(SyntaxError): + compile(code, "", "exec") + + +class TokenizerRegrTest(unittest.TestCase): + + def test_oneline_defs(self): + buf = [] + for i in range(500): + buf.append('def i{i}(): return {i}'.format(i=i)) + buf = '\n'.join(buf) + + # Test that 500 consequent, one-line defs is OK + ns = {} + exec(buf, ns, ns) + self.assertEqual(ns['i499'](), 499) + + # Test that 500 consequent, one-line defs *and* + # one 'async def' following them is OK + buf += '\nasync def foo():\n return' + ns = {} + exec(buf, ns, ns) + self.assertEqual(ns['i499'](), 499) + self.assertTrue(inspect.iscoroutinefunction(ns['foo'])) + + +class CoroutineTest(unittest.TestCase): + + def test_gen_1(self): + def gen(): yield + self.assertNotHasAttr(gen, '__await__') + + def test_func_1(self): + async def foo(): + return 10 + + f = foo() + self.assertIsInstance(f, types.CoroutineType) + self.assertTrue(bool(foo.__code__.co_flags & inspect.CO_COROUTINE)) + self.assertFalse(bool(foo.__code__.co_flags & inspect.CO_GENERATOR)) + self.assertTrue(bool(f.cr_code.co_flags & inspect.CO_COROUTINE)) + self.assertFalse(bool(f.cr_code.co_flags & inspect.CO_GENERATOR)) + self.assertEqual(run_async(f), ([], 10)) + + self.assertEqual(run_async__await__(foo()), ([], 10)) + + def bar(): pass + self.assertFalse(bool(bar.__code__.co_flags & inspect.CO_COROUTINE)) + + def test_func_2(self): + async def foo(): + raise StopIteration + + with self.assertRaisesRegex( + RuntimeError, "coroutine raised StopIteration"): + + run_async(foo()) + + def test_func_3(self): + async def foo(): + raise StopIteration + + coro = foo() + self.assertRegex(repr(coro), '^$') + coro.close() + + @unittest.expectedFailure # TODO: RUSTPYTHON; RuntimeError: coroutine raised StopIteration + def test_func_4(self): + async def foo(): + raise StopIteration + coro = foo() + + check = lambda: self.assertRaisesRegex( + TypeError, "'coroutine' object is not iterable") + + with check(): + list(coro) + + with check(): + tuple(coro) + + with check(): + sum(coro) + + with check(): + iter(coro) + + with check(): + for i in coro: + pass + + with check(): + [i for i in coro] + + coro.close() + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: TypeError not raised + def test_func_5(self): + @types.coroutine + def bar(): + yield 1 + + async def foo(): + await bar() + + check = lambda: self.assertRaisesRegex( + TypeError, "'coroutine' object is not iterable") + + coro = foo() + with check(): + for el in coro: + pass + coro.close() + + # the following should pass without an error + for el in bar(): + self.assertEqual(el, 1) + self.assertEqual([el for el in bar()], [1]) + self.assertEqual(tuple(bar()), (1,)) + self.assertEqual(next(iter(bar())), 1) + + def test_func_6(self): + @types.coroutine + def bar(): + yield 1 + yield 2 + + async def foo(): + await bar() + + f = foo() + self.assertEqual(f.send(None), 1) + self.assertEqual(f.send(None), 2) + with self.assertRaises(StopIteration): + f.send(None) + + def test_func_7(self): + async def bar(): + return 10 + coro = bar() + + def foo(): + yield from coro + + with self.assertRaisesRegex( + TypeError, + "cannot 'yield from' a coroutine object in " + "a non-coroutine generator"): + list(foo()) + + coro.close() + + def test_func_8(self): + @types.coroutine + def bar(): + return (yield from coro) + + async def foo(): + return 'spam' + + coro = foo() + self.assertEqual(run_async(bar()), ([], 'spam')) + coro.close() + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: RuntimeWarning not triggered + def test_func_9(self): + async def foo(): + pass + + with self.assertWarnsRegex( + RuntimeWarning, + r"coroutine '.*test_func_9.*foo' was never awaited"): + + foo() + support.gc_collect() + + with self.assertWarnsRegex( + RuntimeWarning, + r"coroutine '.*test_func_9.*foo' was never awaited"): + + with self.assertRaises(TypeError): + # See bpo-32703. + for _ in foo(): + pass + + support.gc_collect() + + def test_func_10(self): + N = 0 + + @types.coroutine + def gen(): + nonlocal N + try: + a = yield + yield (a ** 2) + except ZeroDivisionError: + N += 100 + raise + finally: + N += 1 + + async def foo(): + await gen() + + coro = foo() + aw = coro.__await__() + self.assertIs(aw, iter(aw)) + next(aw) + self.assertEqual(aw.send(10), 100) + + self.assertEqual(N, 0) + aw.close() + self.assertEqual(N, 1) + + coro = foo() + aw = coro.__await__() + next(aw) + with self.assertRaises(ZeroDivisionError): + aw.throw(ZeroDivisionError()) + self.assertEqual(N, 102) + + coro = foo() + aw = coro.__await__() + next(aw) + with self.assertRaises(ZeroDivisionError): + with self.assertWarns(DeprecationWarning): + aw.throw(ZeroDivisionError, ZeroDivisionError(), None) + + def test_func_11(self): + async def func(): pass + coro = func() + # Test that PyCoro_Type and _PyCoroWrapper_Type types were properly + # initialized + self.assertIn('__await__', dir(coro)) + self.assertIn('__iter__', dir(coro.__await__())) + self.assertIn('coroutine_wrapper', repr(coro.__await__())) + coro.close() # avoid RuntimeWarning + + def test_func_12(self): + async def g(): + me.send(None) + await foo + me = g() + with self.assertRaisesRegex(ValueError, + "coroutine already executing"): + me.send(None) + + def test_func_13(self): + async def g(): + pass + + coro = g() + with self.assertRaisesRegex( + TypeError, + "can't send non-None value to a just-started coroutine"): + coro.send('spam') + + coro.close() + + def test_func_14(self): + @types.coroutine + def gen(): + yield + async def coro(): + try: + await gen() + except GeneratorExit: + await gen() + c = coro() + c.send(None) + with self.assertRaisesRegex(RuntimeError, + "coroutine ignored GeneratorExit"): + c.close() + + @unittest.expectedFailure # TODO: RUSTPYTHON; StopIteration + def test_func_15(self): + # See http://bugs.python.org/issue25887 for details + + async def spammer(): + return 'spam' + async def reader(coro): + return await coro + + spammer_coro = spammer() + + with self.assertRaisesRegex(StopIteration, 'spam'): + reader(spammer_coro).send(None) + + with self.assertRaisesRegex(RuntimeError, + 'cannot reuse already awaited coroutine'): + reader(spammer_coro).send(None) + + @unittest.expectedFailure # TODO: RUSTPYTHON; StopIteration + def test_func_16(self): + # See http://bugs.python.org/issue25887 for details + + @types.coroutine + def nop(): + yield + async def send(): + await nop() + return 'spam' + async def read(coro): + await nop() + return await coro + + spammer = send() + + reader = read(spammer) + reader.send(None) + reader.send(None) + with self.assertRaisesRegex(Exception, 'ham'): + reader.throw(Exception('ham')) + + reader = read(spammer) + reader.send(None) + with self.assertRaisesRegex(RuntimeError, + 'cannot reuse already awaited coroutine'): + reader.send(None) + + with self.assertRaisesRegex(RuntimeError, + 'cannot reuse already awaited coroutine'): + reader.throw(Exception('wat')) + + @unittest.expectedFailure # TODO: RUSTPYTHON; StopIteration + def test_func_17(self): + # See http://bugs.python.org/issue25887 for details + + async def coroutine(): + return 'spam' + + coro = coroutine() + with self.assertRaisesRegex(StopIteration, 'spam'): + coro.send(None) + + with self.assertRaisesRegex(RuntimeError, + 'cannot reuse already awaited coroutine'): + coro.send(None) + + with self.assertRaisesRegex(RuntimeError, + 'cannot reuse already awaited coroutine'): + coro.throw(Exception('wat')) + + # Closing a coroutine shouldn't raise any exception even if it's + # already closed/exhausted (similar to generators) + coro.close() + coro.close() + + def test_func_18(self): + # See http://bugs.python.org/issue25887 for details + + async def coroutine(): + return 'spam' + + coro = coroutine() + await_iter = coro.__await__() + it = iter(await_iter) + + with self.assertRaisesRegex(StopIteration, 'spam'): + it.send(None) + + with self.assertRaisesRegex(RuntimeError, + 'cannot reuse already awaited coroutine'): + it.send(None) + + with self.assertRaisesRegex(RuntimeError, + 'cannot reuse already awaited coroutine'): + # Although the iterator protocol requires iterators to + # raise another StopIteration here, we don't want to do + # that. In this particular case, the iterator will raise + # a RuntimeError, so that 'yield from' and 'await' + # expressions will trigger the error, instead of silently + # ignoring the call. + next(it) + + with self.assertRaisesRegex(RuntimeError, + 'cannot reuse already awaited coroutine'): + it.throw(Exception('wat')) + + with self.assertRaisesRegex(RuntimeError, + 'cannot reuse already awaited coroutine'): + it.throw(Exception('wat')) + + # Closing a coroutine shouldn't raise any exception even if it's + # already closed/exhausted (similar to generators) + it.close() + it.close() + + def test_func_19(self): + CHK = 0 + + @types.coroutine + def foo(): + nonlocal CHK + yield + try: + yield + except GeneratorExit: + CHK += 1 + + async def coroutine(): + await foo() + + coro = coroutine() + + coro.send(None) + coro.send(None) + + self.assertEqual(CHK, 0) + coro.close() + self.assertEqual(CHK, 1) + + for _ in range(3): + # Closing a coroutine shouldn't raise any exception even if it's + # already closed/exhausted (similar to generators) + coro.close() + self.assertEqual(CHK, 1) + + def test_coro_wrapper_send_tuple(self): + async def foo(): + return (10,) + + result = run_async__await__(foo()) + self.assertEqual(result, ([], (10,))) + + def test_coro_wrapper_send_stop_iterator(self): + async def foo(): + return StopIteration(10) + + result = run_async__await__(foo()) + self.assertIsInstance(result[1], StopIteration) + self.assertEqual(result[1].value, 10) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'coroutine' object has no attribute 'cr_suspended' + def test_cr_await(self): + @types.coroutine + def a(): + self.assertEqual(inspect.getcoroutinestate(coro_b), inspect.CORO_RUNNING) + self.assertIsNone(coro_b.cr_await) + yield + self.assertEqual(inspect.getcoroutinestate(coro_b), inspect.CORO_RUNNING) + self.assertIsNone(coro_b.cr_await) + + async def c(): + await a() + + async def b(): + self.assertIsNone(coro_b.cr_await) + await c() + self.assertIsNone(coro_b.cr_await) + + coro_b = b() + self.assertEqual(inspect.getcoroutinestate(coro_b), inspect.CORO_CREATED) + self.assertIsNone(coro_b.cr_await) + + coro_b.send(None) + self.assertEqual(inspect.getcoroutinestate(coro_b), inspect.CORO_SUSPENDED) + self.assertEqual(coro_b.cr_await.cr_await.gi_code.co_name, 'a') + + with self.assertRaises(StopIteration): + coro_b.send(None) # complete coroutine + self.assertEqual(inspect.getcoroutinestate(coro_b), inspect.CORO_CLOSED) + self.assertIsNone(coro_b.cr_await) + + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: 'NoneType' object is not iterable + def test_corotype_1(self): + ct = types.CoroutineType + if not support.MISSING_C_DOCSTRINGS: + self.assertIn('into coroutine', ct.send.__doc__) + self.assertIn('inside coroutine', ct.close.__doc__) + self.assertIn('in coroutine', ct.throw.__doc__) + self.assertIn('of the coroutine', ct.__dict__['__name__'].__doc__) + self.assertIn('of the coroutine', ct.__dict__['__qualname__'].__doc__) + self.assertEqual(ct.__name__, 'coroutine') + + async def f(): pass + c = f() + self.assertIn('coroutine object', repr(c)) + c.close() + + def test_await_1(self): + + async def foo(): + await 1 + with self.assertRaisesRegex(TypeError, "'int' object can.t be awaited"): + run_async(foo()) + + def test_await_2(self): + async def foo(): + await [] + with self.assertRaisesRegex(TypeError, "'list' object can.t be awaited"): + run_async(foo()) + + def test_await_3(self): + async def foo(): + await AsyncYieldFrom([1, 2, 3]) + + self.assertEqual(run_async(foo()), ([1, 2, 3], None)) + self.assertEqual(run_async__await__(foo()), ([1, 2, 3], None)) + + def test_await_4(self): + async def bar(): + return 42 + + async def foo(): + return await bar() + + self.assertEqual(run_async(foo()), ([], 42)) + + def test_await_5(self): + class Awaitable: + def __await__(self): + return + + async def foo(): + return (await Awaitable()) + + with self.assertRaisesRegex( + TypeError, "__await__.*returned non-iterator of type"): + + run_async(foo()) + + def test_await_6(self): + class Awaitable: + def __await__(self): + return iter([52]) + + async def foo(): + return (await Awaitable()) + + self.assertEqual(run_async(foo()), ([52], None)) + + def test_await_7(self): + class Awaitable: + def __await__(self): + yield 42 + return 100 + + async def foo(): + return (await Awaitable()) + + self.assertEqual(run_async(foo()), ([42], 100)) + + def test_await_8(self): + class Awaitable: + pass + + async def foo(): return await Awaitable() + + with self.assertRaisesRegex( + TypeError, "'Awaitable' object can't be awaited"): + + run_async(foo()) + + def test_await_9(self): + def wrap(): + return bar + + async def bar(): + return 42 + + async def foo(): + db = {'b': lambda: wrap} + + class DB: + b = wrap + + return (await bar() + await wrap()() + await db['b']()()() + + await bar() * 1000 + await DB.b()()) + + async def foo2(): + return -await bar() + + self.assertEqual(run_async(foo()), ([], 42168)) + self.assertEqual(run_async(foo2()), ([], -42)) + + def test_await_10(self): + async def baz(): + return 42 + + async def bar(): + return baz() + + async def foo(): + return await (await bar()) + + self.assertEqual(run_async(foo()), ([], 42)) + + def test_await_11(self): + def ident(val): + return val + + async def bar(): + return 'spam' + + async def foo(): + return ident(val=await bar()) + + async def foo2(): + return await bar(), 'ham' + + self.assertEqual(run_async(foo2()), ([], ('spam', 'ham'))) + + def test_await_12(self): + async def coro(): + return 'spam' + c = coro() + + class Awaitable: + def __await__(self): + return c + + async def foo(): + return await Awaitable() + + with self.assertRaisesRegex( + TypeError, r"__await__\(\) returned a coroutine"): + run_async(foo()) + + c.close() + + def test_await_13(self): + class Awaitable: + def __await__(self): + return self + + async def foo(): + return await Awaitable() + + with self.assertRaisesRegex( + TypeError, "__await__.*returned non-iterator of type"): + + run_async(foo()) + + def test_await_14(self): + class Wrapper: + # Forces the interpreter to use CoroutineType.__await__ + def __init__(self, coro): + assert coro.__class__ is types.CoroutineType + self.coro = coro + def __await__(self): + return self.coro.__await__() + + class FutureLike: + def __await__(self): + return (yield) + + class Marker(Exception): + pass + + async def coro1(): + try: + return await FutureLike() + except ZeroDivisionError: + raise Marker + async def coro2(): + return await Wrapper(coro1()) + + c = coro2() + c.send(None) + with self.assertRaisesRegex(StopIteration, 'spam'): + c.send('spam') + + c = coro2() + c.send(None) + with self.assertRaises(Marker): + c.throw(ZeroDivisionError) + + def test_await_15(self): + @types.coroutine + def nop(): + yield + + async def coroutine(): + await nop() + + async def waiter(coro): + await coro + + coro = coroutine() + coro.send(None) + + with self.assertRaisesRegex(RuntimeError, + "coroutine is being awaited already"): + waiter(coro).send(None) + + def test_await_16(self): + # See https://bugs.python.org/issue29600 for details. + + async def f(): + return ValueError() + + async def g(): + try: + raise KeyError + except KeyError: + return await f() + + _, result = run_async(g()) + self.assertIsNone(result.__context__) + + def test_await_17(self): + # See https://github.com/python/cpython/issues/131666 for details. + class A: + async def __anext__(self): + raise StopAsyncIteration + def __aiter__(self): + return self + + with contextlib.closing(anext(A(), "a").__await__()) as anext_awaitable: + self.assertRaises(TypeError, anext_awaitable.close, 1) + + def test_with_1(self): + class Manager: + def __init__(self, name): + self.name = name + + async def __aenter__(self): + await AsyncYieldFrom(['enter-1-' + self.name, + 'enter-2-' + self.name]) + return self + + async def __aexit__(self, *args): + await AsyncYieldFrom(['exit-1-' + self.name, + 'exit-2-' + self.name]) + + if self.name == 'B': + return True + + + async def foo(): + async with Manager("A") as a, Manager("B") as b: + await AsyncYieldFrom([('managers', a.name, b.name)]) + 1/0 + + f = foo() + result, _ = run_async(f) + + self.assertEqual( + result, ['enter-1-A', 'enter-2-A', 'enter-1-B', 'enter-2-B', + ('managers', 'A', 'B'), + 'exit-1-B', 'exit-2-B', 'exit-1-A', 'exit-2-A'] + ) + + async def foo(): + async with Manager("A") as a, Manager("C") as c: + await AsyncYieldFrom([('managers', a.name, c.name)]) + 1/0 + + with self.assertRaises(ZeroDivisionError): + run_async(foo()) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: "asynchronous context manager.*__aexit__" does not match "'CM' object does not support the context manager protocol (missed __aexit__ method)" + def test_with_2(self): + class CM: + def __aenter__(self): + pass + + body_executed = None + async def foo(): + nonlocal body_executed + body_executed = False + async with CM(): + body_executed = True + + with self.assertRaisesRegex(TypeError, 'asynchronous context manager.*__aexit__'): + run_async(foo()) + self.assertIs(body_executed, False) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: "asynchronous context manager" does not match "'CM' object does not support the context manager protocol (missed __aenter__ method)" + def test_with_3(self): + class CM: + def __aexit__(self): + pass + + body_executed = None + async def foo(): + nonlocal body_executed + body_executed = False + async with CM(): + body_executed = True + + with self.assertRaisesRegex(TypeError, 'asynchronous context manager'): + run_async(foo()) + self.assertIs(body_executed, False) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: "asynchronous context manager" does not match "'CM' object does not support the context manager protocol (missed __aexit__ method)" + def test_with_4(self): + class CM: + pass + + body_executed = None + async def foo(): + nonlocal body_executed + body_executed = False + async with CM(): + body_executed = True + + with self.assertRaisesRegex(TypeError, 'asynchronous context manager'): + run_async(foo()) + self.assertIs(body_executed, False) + + def test_with_5(self): + # While this test doesn't make a lot of sense, + # it's a regression test for an early bug with opcodes + # generation + + class CM: + async def __aenter__(self): + return self + + async def __aexit__(self, *exc): + pass + + async def func(): + async with CM(): + self.assertEqual((1, ), 1) + + with self.assertRaises(AssertionError): + run_async(func()) + + def test_with_6(self): + class CM: + def __aenter__(self): + return 123 + + def __aexit__(self, *e): + return 456 + + async def foo(): + async with CM(): + pass + + with self.assertRaisesRegex( + TypeError, + "'async with' received an object from __aenter__ " + "that does not implement __await__: int"): + # it's important that __aexit__ wasn't called + run_async(foo()) + + def test_with_7(self): + class CM: + async def __aenter__(self): + return self + + def __aexit__(self, *e): + return 444 + + # Exit with exception + async def foo(): + async with CM(): + 1/0 + + try: + run_async(foo()) + except TypeError as exc: + self.assertRegex( + exc.args[0], + "'async with' received an object from __aexit__ " + "that does not implement __await__: int") + self.assertTrue(exc.__context__ is not None) + self.assertTrue(isinstance(exc.__context__, ZeroDivisionError)) + else: + self.fail('invalid asynchronous context manager did not fail') + + + def test_with_8(self): + CNT = 0 + + class CM: + async def __aenter__(self): + return self + + def __aexit__(self, *e): + return 456 + + # Normal exit + async def foo(): + nonlocal CNT + async with CM(): + CNT += 1 + with self.assertRaisesRegex( + TypeError, + "'async with' received an object from __aexit__ " + "that does not implement __await__: int"): + run_async(foo()) + self.assertEqual(CNT, 1) + + # Exit with 'break' + async def foo(): + nonlocal CNT + for i in range(2): + async with CM(): + CNT += 1 + break + with self.assertRaisesRegex( + TypeError, + "'async with' received an object from __aexit__ " + "that does not implement __await__: int"): + run_async(foo()) + self.assertEqual(CNT, 2) + + # Exit with 'continue' + async def foo(): + nonlocal CNT + for i in range(2): + async with CM(): + CNT += 1 + continue + with self.assertRaisesRegex( + TypeError, + "'async with' received an object from __aexit__ " + "that does not implement __await__: int"): + run_async(foo()) + self.assertEqual(CNT, 3) + + # Exit with 'return' + async def foo(): + nonlocal CNT + async with CM(): + CNT += 1 + return + with self.assertRaisesRegex( + TypeError, + "'async with' received an object from __aexit__ " + "that does not implement __await__: int"): + run_async(foo()) + self.assertEqual(CNT, 4) + + + def test_with_9(self): + CNT = 0 + + class CM: + async def __aenter__(self): + return self + + async def __aexit__(self, *e): + 1/0 + + async def foo(): + nonlocal CNT + async with CM(): + CNT += 1 + + with self.assertRaises(ZeroDivisionError): + run_async(foo()) + + self.assertEqual(CNT, 1) + + def test_with_10(self): + CNT = 0 + + class CM: + async def __aenter__(self): + return self + + async def __aexit__(self, *e): + 1/0 + + async def foo(): + nonlocal CNT + async with CM(): + async with CM(): + raise RuntimeError + + try: + run_async(foo()) + except ZeroDivisionError as exc: + self.assertTrue(exc.__context__ is not None) + self.assertTrue(isinstance(exc.__context__, ZeroDivisionError)) + self.assertTrue(isinstance(exc.__context__.__context__, + RuntimeError)) + else: + self.fail('exception from __aexit__ did not propagate') + + def test_with_11(self): + CNT = 0 + + class CM: + async def __aenter__(self): + raise NotImplementedError + + async def __aexit__(self, *e): + 1/0 + + async def foo(): + nonlocal CNT + async with CM(): + raise RuntimeError + + try: + run_async(foo()) + except NotImplementedError as exc: + self.assertTrue(exc.__context__ is None) + else: + self.fail('exception from __aenter__ did not propagate') + + def test_with_12(self): + CNT = 0 + + class CM: + async def __aenter__(self): + return self + + async def __aexit__(self, *e): + return True + + async def foo(): + nonlocal CNT + async with CM() as cm: + self.assertIs(cm.__class__, CM) + raise RuntimeError + + run_async(foo()) + + def test_with_13(self): + CNT = 0 + + class CM: + async def __aenter__(self): + 1/0 + + async def __aexit__(self, *e): + return True + + async def foo(): + nonlocal CNT + CNT += 1 + async with CM(): + CNT += 1000 + CNT += 10000 + + with self.assertRaises(ZeroDivisionError): + run_async(foo()) + self.assertEqual(CNT, 1) + + def test_for_1(self): + aiter_calls = 0 + + class AsyncIter: + def __init__(self): + self.i = 0 + + def __aiter__(self): + nonlocal aiter_calls + aiter_calls += 1 + return self + + async def __anext__(self): + self.i += 1 + + if not (self.i % 10): + await AsyncYield(self.i * 10) + + if self.i > 100: + raise StopAsyncIteration + + return self.i, self.i + + + buffer = [] + async def test1(): + async for i1, i2 in AsyncIter(): + buffer.append(i1 + i2) + + yielded, _ = run_async(test1()) + # Make sure that __aiter__ was called only once + self.assertEqual(aiter_calls, 1) + self.assertEqual(yielded, [i * 100 for i in range(1, 11)]) + self.assertEqual(buffer, [i*2 for i in range(1, 101)]) + + + buffer = [] + async def test2(): + nonlocal buffer + async for i in AsyncIter(): + buffer.append(i[0]) + if i[0] == 20: + break + else: + buffer.append('what?') + buffer.append('end') + + yielded, _ = run_async(test2()) + # Make sure that __aiter__ was called only once + self.assertEqual(aiter_calls, 2) + self.assertEqual(yielded, [100, 200]) + self.assertEqual(buffer, [i for i in range(1, 21)] + ['end']) + + + buffer = [] + async def test3(): + nonlocal buffer + async for i in AsyncIter(): + if i[0] > 20: + continue + buffer.append(i[0]) + else: + buffer.append('what?') + buffer.append('end') + + yielded, _ = run_async(test3()) + # Make sure that __aiter__ was called only once + self.assertEqual(aiter_calls, 3) + self.assertEqual(yielded, [i * 100 for i in range(1, 11)]) + self.assertEqual(buffer, [i for i in range(1, 21)] + + ['what?', 'end']) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: __aiter__ + def test_for_2(self): + tup = (1, 2, 3) + refs_before = sys.getrefcount(tup) + + async def foo(): + async for i in tup: + print('never going to happen') + + with self.assertRaisesRegex( + TypeError, "async for' requires an object.*__aiter__.*tuple"): + + run_async(foo()) + + self.assertEqual(sys.getrefcount(tup), refs_before) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: "that does not implement __anext__" does not match "'async for' requires an iterator with __anext__ method, got I" + def test_for_3(self): + class I: + def __aiter__(self): + return self + + aiter = I() + refs_before = sys.getrefcount(aiter) + + async def foo(): + async for i in aiter: + print('never going to happen') + + with self.assertRaisesRegex( + TypeError, + r"that does not implement __anext__"): + + run_async(foo()) + + self.assertEqual(sys.getrefcount(aiter), refs_before) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: "async for' received an invalid object.*__anext__.*tuple" does not match "'tuple' object is not an iterator" + def test_for_4(self): + class I: + def __aiter__(self): + return self + + def __anext__(self): + return () + + aiter = I() + refs_before = sys.getrefcount(aiter) + + async def foo(): + async for i in aiter: + print('never going to happen') + + with self.assertRaisesRegex( + TypeError, + "async for' received an invalid object.*__anext__.*tuple"): + + run_async(foo()) + + self.assertEqual(sys.getrefcount(aiter), refs_before) + + def test_for_6(self): + I = 0 + + class Manager: + async def __aenter__(self): + nonlocal I + I += 10000 + + async def __aexit__(self, *args): + nonlocal I + I += 100000 + + class Iterable: + def __init__(self): + self.i = 0 + + def __aiter__(self): + return self + + async def __anext__(self): + if self.i > 10: + raise StopAsyncIteration + self.i += 1 + return self.i + + ############## + + manager = Manager() + iterable = Iterable() + mrefs_before = sys.getrefcount(manager) + irefs_before = sys.getrefcount(iterable) + + async def main(): + nonlocal I + + async with manager: + async for i in iterable: + I += 1 + I += 1000 + + with warnings.catch_warnings(): + warnings.simplefilter("error") + # Test that __aiter__ that returns an asynchronous iterator + # directly does not throw any warnings. + run_async(main()) + self.assertEqual(I, 111011) + + self.assertEqual(sys.getrefcount(manager), mrefs_before) + self.assertEqual(sys.getrefcount(iterable), irefs_before) + + ############## + + async def main(): + nonlocal I + + async with Manager(): + async for i in Iterable(): + I += 1 + I += 1000 + + async with Manager(): + async for i in Iterable(): + I += 1 + I += 1000 + + run_async(main()) + self.assertEqual(I, 333033) + + ############## + + async def main(): + nonlocal I + + async with Manager(): + I += 100 + async for i in Iterable(): + I += 1 + else: + I += 10000000 + I += 1000 + + async with Manager(): + I += 100 + async for i in Iterable(): + I += 1 + else: + I += 10000000 + I += 1000 + + run_async(main()) + self.assertEqual(I, 20555255) + + def test_for_7(self): + CNT = 0 + class AI: + def __aiter__(self): + 1/0 + async def foo(): + nonlocal CNT + async for i in AI(): + CNT += 1 + CNT += 10 + with self.assertRaises(ZeroDivisionError): + run_async(foo()) + self.assertEqual(CNT, 0) + + def test_for_8(self): + CNT = 0 + class AI: + def __aiter__(self): + 1/0 + async def foo(): + nonlocal CNT + async for i in AI(): + CNT += 1 + CNT += 10 + with self.assertRaises(ZeroDivisionError): + with warnings.catch_warnings(): + warnings.simplefilter("error") + # Test that if __aiter__ raises an exception it propagates + # without any kind of warning. + run_async(foo()) + self.assertEqual(CNT, 0) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: "an invalid object from __anext__" does not match "'F' object is not an iterator" + def test_for_11(self): + class F: + def __aiter__(self): + return self + def __anext__(self): + return self + def __await__(self): + 1 / 0 + + async def main(): + async for _ in F(): + pass + + with self.assertRaisesRegex(TypeError, + 'an invalid object from __anext__') as c: + main().send(None) + + err = c.exception + self.assertIsInstance(err.__cause__, ZeroDivisionError) + + def test_for_tuple(self): + class Done(Exception): pass + + class AIter(tuple): + i = 0 + def __aiter__(self): + return self + async def __anext__(self): + if self.i >= len(self): + raise StopAsyncIteration + self.i += 1 + return self[self.i - 1] + + result = [] + async def foo(): + async for i in AIter([42]): + result.append(i) + raise Done + + with self.assertRaises(Done): + foo().send(None) + self.assertEqual(result, [42]) + + def test_for_stop_iteration(self): + class Done(Exception): pass + + class AIter(StopIteration): + i = 0 + def __aiter__(self): + return self + async def __anext__(self): + if self.i: + raise StopAsyncIteration + self.i += 1 + return self.value + + result = [] + async def foo(): + async for i in AIter(42): + result.append(i) + raise Done + + with self.assertRaises(Done): + foo().send(None) + self.assertEqual(result, [42]) + + def test_comp_1(self): + async def f(i): + return i + + async def run_list(): + return [await c for c in [f(1), f(41)]] + + async def run_set(): + return {await c for c in [f(1), f(41)]} + + async def run_dict1(): + return {await c: 'a' for c in [f(1), f(41)]} + + async def run_dict2(): + return {i: await c for i, c in enumerate([f(1), f(41)])} + + self.assertEqual(run_async(run_list()), ([], [1, 41])) + self.assertEqual(run_async(run_set()), ([], {1, 41})) + self.assertEqual(run_async(run_dict1()), ([], {1: 'a', 41: 'a'})) + self.assertEqual(run_async(run_dict2()), ([], {0: 1, 1: 41})) + + def test_comp_2(self): + async def f(i): + return i + + async def run_list(): + return [s for c in [f(''), f('abc'), f(''), f(['de', 'fg'])] + for s in await c] + + self.assertEqual( + run_async(run_list()), + ([], ['a', 'b', 'c', 'de', 'fg'])) + + async def run_set(): + return {d + for c in [f([f([10, 30]), + f([20])])] + for s in await c + for d in await s} + + self.assertEqual( + run_async(run_set()), + ([], {10, 20, 30})) + + async def run_set2(): + return {await s + for c in [f([f(10), f(20)])] + for s in await c} + + self.assertEqual( + run_async(run_set2()), + ([], {10, 20})) + + def test_comp_3(self): + async def f(it): + for i in it: + yield i + + async def run_list(): + return [i + 1 async for i in f([10, 20])] + self.assertEqual( + run_async(run_list()), + ([], [11, 21])) + + async def run_set(): + return {i + 1 async for i in f([10, 20])} + self.assertEqual( + run_async(run_set()), + ([], {11, 21})) + + async def run_dict(): + return {i + 1: i + 2 async for i in f([10, 20])} + self.assertEqual( + run_async(run_dict()), + ([], {11: 12, 21: 22})) + + async def run_gen(): + gen = (i + 1 async for i in f([10, 20])) + return [g + 100 async for g in gen] + self.assertEqual( + run_async(run_gen()), + ([], [111, 121])) + + def test_comp_4(self): + async def f(it): + for i in it: + yield i + + async def run_list(): + return [i + 1 async for i in f([10, 20]) if i > 10] + self.assertEqual( + run_async(run_list()), + ([], [21])) + + async def run_set(): + return {i + 1 async for i in f([10, 20]) if i > 10} + self.assertEqual( + run_async(run_set()), + ([], {21})) + + async def run_dict(): + return {i + 1: i + 2 async for i in f([10, 20]) if i > 10} + self.assertEqual( + run_async(run_dict()), + ([], {21: 22})) + + async def run_gen(): + gen = (i + 1 async for i in f([10, 20]) if i > 10) + return [g + 100 async for g in gen] + self.assertEqual( + run_async(run_gen()), + ([], [121])) + + def test_comp_4_2(self): + async def f(it): + for i in it: + yield i + + async def run_list(): + return [i + 10 async for i in f(range(5)) if 0 < i < 4] + self.assertEqual( + run_async(run_list()), + ([], [11, 12, 13])) + + async def run_set(): + return {i + 10 async for i in f(range(5)) if 0 < i < 4} + self.assertEqual( + run_async(run_set()), + ([], {11, 12, 13})) + + async def run_dict(): + return {i + 10: i + 100 async for i in f(range(5)) if 0 < i < 4} + self.assertEqual( + run_async(run_dict()), + ([], {11: 101, 12: 102, 13: 103})) + + async def run_gen(): + gen = (i + 10 async for i in f(range(5)) if 0 < i < 4) + return [g + 100 async for g in gen] + self.assertEqual( + run_async(run_gen()), + ([], [111, 112, 113])) + + def test_comp_5(self): + async def f(it): + for i in it: + yield i + + async def run_list(): + return [i + 1 for pair in ([10, 20], [30, 40]) if pair[0] > 10 + async for i in f(pair) if i > 30] + self.assertEqual( + run_async(run_list()), + ([], [41])) + + def test_comp_6(self): + async def f(it): + for i in it: + yield i + + async def run_list(): + return [i + 1 async for seq in f([(10, 20), (30,)]) + for i in seq] + + self.assertEqual( + run_async(run_list()), + ([], [11, 21, 31])) + + def test_comp_7(self): + async def f(): + yield 1 + yield 2 + raise Exception('aaa') + + async def run_list(): + return [i async for i in f()] + + with self.assertRaisesRegex(Exception, 'aaa'): + run_async(run_list()) + + def test_comp_8(self): + async def f(): + return [i for i in [1, 2, 3]] + + self.assertEqual( + run_async(f()), + ([], [1, 2, 3])) + + def test_comp_9(self): + async def gen(): + yield 1 + yield 2 + async def f(): + l = [i async for i in gen()] + return [i for i in l] + + self.assertEqual( + run_async(f()), + ([], [1, 2])) + + def test_comp_10(self): + async def f(): + xx = {i for i in [1, 2, 3]} + return {x: x for x in xx} + + self.assertEqual( + run_async(f()), + ([], {1: 1, 2: 2, 3: 3})) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: __aiter__ + def test_nested_comp(self): + async def run_list_inside_list(): + return [[i + j async for i in asynciter([1, 2])] for j in [10, 20]] + self.assertEqual( + run_async(run_list_inside_list()), + ([], [[11, 12], [21, 22]])) + + async def run_set_inside_list(): + return [{i + j async for i in asynciter([1, 2])} for j in [10, 20]] + self.assertEqual( + run_async(run_set_inside_list()), + ([], [{11, 12}, {21, 22}])) + + async def run_list_inside_set(): + return {sum([i async for i in asynciter(range(j))]) for j in [3, 5]} + self.assertEqual( + run_async(run_list_inside_set()), + ([], {3, 10})) + + async def run_dict_inside_dict(): + return {j: {i: i + j async for i in asynciter([1, 2])} for j in [10, 20]} + self.assertEqual( + run_async(run_dict_inside_dict()), + ([], {10: {1: 11, 2: 12}, 20: {1: 21, 2: 22}})) + + async def run_list_inside_gen(): + gen = ([i + j async for i in asynciter([1, 2])] for j in [10, 20]) + return [x async for x in gen] + self.assertEqual( + run_async(run_list_inside_gen()), + ([], [[11, 12], [21, 22]])) + + async def run_gen_inside_list(): + gens = [(i async for i in asynciter(range(j))) for j in [3, 5]] + return [x for g in gens async for x in g] + self.assertEqual( + run_async(run_gen_inside_list()), + ([], [0, 1, 2, 0, 1, 2, 3, 4])) + + async def run_gen_inside_gen(): + gens = ((i async for i in asynciter(range(j))) for j in [3, 5]) + return [x for g in gens async for x in g] + self.assertEqual( + run_async(run_gen_inside_gen()), + ([], [0, 1, 2, 0, 1, 2, 3, 4])) + + async def run_list_inside_list_inside_list(): + return [[[i + j + k async for i in asynciter([1, 2])] + for j in [10, 20]] + for k in [100, 200]] + self.assertEqual( + run_async(run_list_inside_list_inside_list()), + ([], [[[111, 112], [121, 122]], [[211, 212], [221, 222]]])) + + def test_copy(self): + async def func(): pass + coro = func() + with self.assertRaises(TypeError): + copy.copy(coro) + + aw = coro.__await__() + try: + with self.assertRaises(TypeError): + copy.copy(aw) + finally: + aw.close() + + def test_pickle(self): + async def func(): pass + coro = func() + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.assertRaises((TypeError, pickle.PicklingError)): + pickle.dumps(coro, proto) + + aw = coro.__await__() + try: + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.assertRaises((TypeError, pickle.PicklingError)): + pickle.dumps(aw, proto) + finally: + aw.close() + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'NoneType' object has no attribute 'err_msg' + def test_fatal_coro_warning(self): + # Issue 27811 + async def func(): pass + with warnings.catch_warnings(), \ + support.catch_unraisable_exception() as cm: + warnings.filterwarnings("error") + coro = func() + # only store repr() to avoid keeping the coroutine alive + coro_repr = repr(coro) + coro = None + support.gc_collect() + + self.assertEqual(cm.unraisable.err_msg, + f"Exception ignored while finalizing " + f"coroutine {coro_repr}") + self.assertIn("was never awaited", str(cm.unraisable.exc_value)) + + def test_for_assign_raising_stop_async_iteration(self): + class BadTarget: + def __setitem__(self, key, value): + raise StopAsyncIteration(42) + tgt = BadTarget() + async def source(): + yield 10 + + async def run_for(): + with self.assertRaises(StopAsyncIteration) as cm: + async for tgt[0] in source(): + pass + self.assertEqual(cm.exception.args, (42,)) + return 'end' + self.assertEqual(run_async(run_for()), ([], 'end')) + + async def run_list(): + with self.assertRaises(StopAsyncIteration) as cm: + return [0 async for tgt[0] in source()] + self.assertEqual(cm.exception.args, (42,)) + return 'end' + self.assertEqual(run_async(run_list()), ([], 'end')) + + async def run_gen(): + gen = (0 async for tgt[0] in source()) + a = gen.asend(None) + with self.assertRaises(RuntimeError) as cm: + await a + self.assertIsInstance(cm.exception.__cause__, StopAsyncIteration) + self.assertEqual(cm.exception.__cause__.args, (42,)) + return 'end' + self.assertEqual(run_async(run_gen()), ([], 'end')) + + def test_for_assign_raising_stop_async_iteration_2(self): + class BadIterable: + def __iter__(self): + raise StopAsyncIteration(42) + async def badpairs(): + yield BadIterable() + + async def run_for(): + with self.assertRaises(StopAsyncIteration) as cm: + async for i, j in badpairs(): + pass + self.assertEqual(cm.exception.args, (42,)) + return 'end' + self.assertEqual(run_async(run_for()), ([], 'end')) + + async def run_list(): + with self.assertRaises(StopAsyncIteration) as cm: + return [0 async for i, j in badpairs()] + self.assertEqual(cm.exception.args, (42,)) + return 'end' + self.assertEqual(run_async(run_list()), ([], 'end')) + + async def run_gen(): + gen = (0 async for i, j in badpairs()) + a = gen.asend(None) + with self.assertRaises(RuntimeError) as cm: + await a + self.assertIsInstance(cm.exception.__cause__, StopAsyncIteration) + self.assertEqual(cm.exception.__cause__.args, (42,)) + return 'end' + self.assertEqual(run_async(run_gen()), ([], 'end')) + + @unittest.expectedFailure # TODO: RUSTPYTHON; This would crash the interpreter in 3.11a2 + def test_bpo_45813_1(self): + 'This would crash the interpreter in 3.11a2' + async def f(): + pass + with self.assertWarns(RuntimeWarning): + frame = f().cr_frame + frame.clear() + + @unittest.expectedFailure # TODO: RUSTPYTHON; This would crash the interpreter in 3.11a2 + def test_bpo_45813_2(self): + 'This would crash the interpreter in 3.11a2' + async def f(): + pass + gen = f() + with self.assertWarns(RuntimeWarning): + gen.cr_frame.clear() + gen.close() + + def test_cr_frame_after_close(self): + async def f(): + pass + gen = f() + self.assertIsNotNone(gen.cr_frame) + gen.close() + self.assertIsNone(gen.cr_frame) + + def test_stack_in_coroutine_throw(self): + # Regression test for https://github.com/python/cpython/issues/93592 + async def a(): + return await b() + + async def b(): + return await c() + + @types.coroutine + def c(): + try: + # traceback.print_stack() + yield len(traceback.extract_stack()) + except ZeroDivisionError: + # traceback.print_stack() + yield len(traceback.extract_stack()) + + coro = a() + len_send = coro.send(None) + len_throw = coro.throw(ZeroDivisionError) + # before fixing, visible stack from throw would be shorter than from send. + self.assertEqual(len_send, len_throw) + + def test_call_aiter_once_in_comprehension(self): + + class AsyncIterator: + + def __init__(self): + self.val = 0 + + async def __anext__(self): + if self.val == 2: + raise StopAsyncIteration + self.val += 1 + return self.val + + # No __aiter__ method + + class C: + + def __aiter__(self): + return AsyncIterator() + + async def run_listcomp(): + return [i async for i in C()] + + async def run_asyncgen(): + ag = (i async for i in C()) + return [i async for i in ag] + + self.assertEqual(run_async(run_listcomp()), ([], [1, 2])) + self.assertEqual(run_async(run_asyncgen()), ([], [1, 2])) + + +@unittest.skipIf( + support.is_emscripten or support.is_wasi, + "asyncio does not work under Emscripten/WASI yet." +) +class CoroAsyncIOCompatTest(unittest.TestCase): + + def test_asyncio_1(self): + # asyncio cannot be imported when Python is compiled without thread + # support + asyncio = import_helper.import_module('asyncio') + + class MyException(Exception): + pass + + buffer = [] + + class CM: + async def __aenter__(self): + buffer.append(1) + await asyncio.sleep(0.01) + buffer.append(2) + return self + + async def __aexit__(self, exc_type, exc_val, exc_tb): + await asyncio.sleep(0.01) + buffer.append(exc_type.__name__) + + async def f(): + async with CM(): + await asyncio.sleep(0.01) + raise MyException + buffer.append('unreachable') + + loop = asyncio.new_event_loop() + asyncio.set_event_loop(loop) + try: + loop.run_until_complete(f()) + except MyException: + pass + finally: + loop.close() + asyncio.events._set_event_loop_policy(None) + + self.assertEqual(buffer, [1, 2, 'MyException']) + + +class OriginTrackingTest(unittest.TestCase): + def here(self): + info = inspect.getframeinfo(inspect.currentframe().f_back) + return (info.filename, info.lineno) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: None != (('/home/fanninpm/Documents/GitHub/RustPy[74 chars]g'),) + def test_origin_tracking(self): + orig_depth = sys.get_coroutine_origin_tracking_depth() + try: + async def corofn(): + pass + + sys.set_coroutine_origin_tracking_depth(0) + self.assertEqual(sys.get_coroutine_origin_tracking_depth(), 0) + + with contextlib.closing(corofn()) as coro: + self.assertIsNone(coro.cr_origin) + + sys.set_coroutine_origin_tracking_depth(1) + self.assertEqual(sys.get_coroutine_origin_tracking_depth(), 1) + + fname, lineno = self.here() + with contextlib.closing(corofn()) as coro: + self.assertEqual(coro.cr_origin, + ((fname, lineno + 1, "test_origin_tracking"),)) + + sys.set_coroutine_origin_tracking_depth(2) + self.assertEqual(sys.get_coroutine_origin_tracking_depth(), 2) + + def nested(): + return (self.here(), corofn()) + fname, lineno = self.here() + ((nested_fname, nested_lineno), coro) = nested() + with contextlib.closing(coro): + self.assertEqual(coro.cr_origin, + ((nested_fname, nested_lineno, "nested"), + (fname, lineno + 1, "test_origin_tracking"))) + + # Check we handle running out of frames correctly + sys.set_coroutine_origin_tracking_depth(1000) + with contextlib.closing(corofn()) as coro: + self.assertTrue(2 < len(coro.cr_origin) < 1000) + + # We can't set depth negative + with self.assertRaises(ValueError): + sys.set_coroutine_origin_tracking_depth(-1) + # And trying leaves it unchanged + self.assertEqual(sys.get_coroutine_origin_tracking_depth(), 1000) + + finally: + sys.set_coroutine_origin_tracking_depth(orig_depth) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: RuntimeWarning not triggered + def test_origin_tracking_warning(self): + async def corofn(): + pass + + a1_filename, a1_lineno = self.here() + def a1(): + return corofn() # comment in a1 + a1_lineno += 2 + + a2_filename, a2_lineno = self.here() + def a2(): + return a1() # comment in a2 + a2_lineno += 2 + + def check(depth, msg): + sys.set_coroutine_origin_tracking_depth(depth) + with self.assertWarns(RuntimeWarning) as cm: + a2() + support.gc_collect() + self.assertEqual(msg, str(cm.warning)) + + orig_depth = sys.get_coroutine_origin_tracking_depth() + try: + check(0, f"coroutine '{corofn.__qualname__}' was never awaited") + check(1, "".join([ + f"coroutine '{corofn.__qualname__}' was never awaited\n", + "Coroutine created at (most recent call last)\n", + f' File "{a1_filename}", line {a1_lineno}, in a1\n', + " return corofn() # comment in a1", + ])) + check(2, "".join([ + f"coroutine '{corofn.__qualname__}' was never awaited\n", + "Coroutine created at (most recent call last)\n", + f' File "{a2_filename}", line {a2_lineno}, in a2\n', + " return a1() # comment in a2\n", + f' File "{a1_filename}", line {a1_lineno}, in a1\n', + " return corofn() # comment in a1", + ])) + + finally: + sys.set_coroutine_origin_tracking_depth(orig_depth) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'NoneType' object has no attribute 'err_msg' + def test_unawaited_warning_when_module_broken(self): + # Make sure we don't blow up too bad if + # warnings._warn_unawaited_coroutine is broken somehow (e.g. because + # of shutdown problems) + async def corofn(): + pass + + orig_wuc = warnings._warn_unawaited_coroutine + try: + warnings._warn_unawaited_coroutine = lambda coro: 1/0 + with support.catch_unraisable_exception() as cm, \ + warnings_helper.check_warnings( + (r'coroutine .* was never awaited', + RuntimeWarning)): + # only store repr() to avoid keeping the coroutine alive + coro = corofn() + coro_repr = repr(coro) + + # clear reference to the coroutine without awaiting for it + del coro + support.gc_collect() + + self.assertEqual(cm.unraisable.err_msg, + f"Exception ignored while finalizing " + f"coroutine {coro_repr}") + self.assertEqual(cm.unraisable.exc_type, ZeroDivisionError) + + del warnings._warn_unawaited_coroutine + with warnings_helper.check_warnings( + (r'coroutine .* was never awaited', RuntimeWarning)): + corofn() + support.gc_collect() + + finally: + warnings._warn_unawaited_coroutine = orig_wuc + + +class UnawaitedWarningDuringShutdownTest(unittest.TestCase): + # https://bugs.python.org/issue32591#msg310726 + def test_unawaited_warning_during_shutdown(self): + code = ("import asyncio\n" + "async def f(): pass\n" + "async def t(): asyncio.gather(f())\n" + "asyncio.run(t())\n") + assert_python_ok("-c", code) + + code = ("import sys\n" + "async def f(): pass\n" + "sys.coro = f()\n") + assert_python_ok("-c", code) + + code = ("import sys\n" + "async def f(): pass\n" + "sys.corocycle = [f()]\n" + "sys.corocycle.append(sys.corocycle)\n") + assert_python_ok("-c", code) + + +@support.cpython_only +@unittest.skipIf(_testcapi is None, "requires _testcapi") +class CAPITest(unittest.TestCase): + + def test_tp_await_1(self): + from _testcapi import awaitType as at + + async def foo(): + future = at(iter([1])) + return (await future) + + self.assertEqual(foo().send(None), 1) + + def test_tp_await_2(self): + # Test tp_await to __await__ mapping + from _testcapi import awaitType as at + future = at(iter([1])) + self.assertEqual(next(future.__await__()), 1) + + def test_tp_await_3(self): + from _testcapi import awaitType as at + + async def foo(): + future = at(1) + return (await future) + + with self.assertRaisesRegex( + TypeError, "__await__.*returned non-iterator of type 'int'"): + self.assertEqual(foo().send(None), 1) + + +if __name__=="__main__": + unittest.main() diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py index b7f93d1bac9..bf9b1875573 100644 --- a/Lib/test/test_csv.py +++ b/Lib/test/test_csv.py @@ -698,7 +698,6 @@ def test_copy(self): dialect = csv.get_dialect(name) self.assertRaises(TypeError, copy.copy, dialect) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_pickle(self): for name in csv.list_dialects(): dialect = csv.get_dialect(name) diff --git a/Lib/test/test_dataclasses/__init__.py b/Lib/test/test_dataclasses/__init__.py index 3f1e2331bc2..4db73a64e73 100644 --- a/Lib/test/test_dataclasses/__init__.py +++ b/Lib/test/test_dataclasses/__init__.py @@ -2311,6 +2311,7 @@ class C: self.assertDocStrEqual(C.__doc__, "C(x:int=3)") + @unittest.expectedFailure # TODO: RUSTPYTHON def test_docstring_one_field_with_default_none(self): @dataclass class C: diff --git a/Lib/test/test_dbm_sqlite3.py b/Lib/test/test_dbm_sqlite3.py index 39eac7a35ec..f367a98865d 100644 --- a/Lib/test/test_dbm_sqlite3.py +++ b/Lib/test/test_dbm_sqlite3.py @@ -1,12 +1,11 @@ import os import stat import sys -import test.support import unittest from contextlib import closing from functools import partial from pathlib import Path -from test.support import cpython_only, import_helper, os_helper +from test.support import import_helper, os_helper dbm_sqlite3 = import_helper.import_module("dbm.sqlite3") # N.B. The test will fail on some platforms without sqlite3 @@ -44,7 +43,7 @@ def test_uri_substitutions(self): ) for path, normalized in dataset: with self.subTest(path=path, normalized=normalized): - self.assertTrue(_normalize_uri(path).endswith(normalized)) + self.assertEndsWith(_normalize_uri(path), normalized) @unittest.skipUnless(sys.platform == "win32", "requires Windows") def test_uri_windows(self): @@ -63,7 +62,7 @@ def test_uri_windows(self): with self.subTest(path=path, normalized=normalized): if not Path(path).is_absolute(): self.skipTest(f"skipping relative path: {path!r}") - self.assertTrue(_normalize_uri(path).endswith(normalized)) + self.assertEndsWith(_normalize_uri(path), normalized) class ReadOnly(_SQLiteDbmTests): diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py index 01b0c06196c..3017c3337c7 100644 --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -45,6 +45,7 @@ import random import inspect import threading +import contextvars if sys.platform == 'darwin': @@ -753,7 +754,7 @@ def test_explicit_context_create_decimal(self): for v in [-2**63-1, -2**63, -2**31-1, -2**31, 0, 2**31-1, 2**31, 2**63-1, 2**63]: d = nc.create_decimal(v) - self.assertTrue(isinstance(d, Decimal)) + self.assertIsInstance(d, Decimal) self.assertEqual(int(d), v) nc.prec = 3 @@ -813,6 +814,29 @@ def test_explicit_context_create_from_float(self): x = random.expovariate(0.01) * (random.random() * 2.0 - 1.0) self.assertEqual(x, float(nc.create_decimal(x))) # roundtrip + def test_from_number(self, cls=None): + Decimal = self.decimal.Decimal + if cls is None: + cls = Decimal + + def check(arg, expected): + d = cls.from_number(arg) + self.assertIs(type(d), cls) + self.assertEqual(d, expected) + + check(314, Decimal(314)) + check(3.14, Decimal.from_float(3.14)) + check(Decimal('3.14'), Decimal('3.14')) + self.assertRaises(TypeError, cls.from_number, 3+4j) + self.assertRaises(TypeError, cls.from_number, '314') + self.assertRaises(TypeError, cls.from_number, (0, (3, 1, 4), 0)) + self.assertRaises(TypeError, cls.from_number, object()) + + def test_from_number_subclass(self, cls=None): + class DecimalSubclass(self.decimal.Decimal): + pass + self.test_from_number(DecimalSubclass) + def test_unicode_digits(self): Decimal = self.decimal.Decimal @@ -830,9 +854,8 @@ class CExplicitConstructionTest(ExplicitConstructionTest, unittest.TestCase): class PyExplicitConstructionTest(ExplicitConstructionTest, unittest.TestCase): decimal = P - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_unicode_digits(self): # TODO(RUSTPYTHON): Remove this test when it pass + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_unicode_digits(self): return super().test_unicode_digits() class ImplicitConstructionTest: @@ -963,6 +986,7 @@ def test_formatting(self): ('.0f', '0e-2', '0'), ('.0f', '3.14159265', '3'), ('.1f', '3.14159265', '3.1'), + ('.01f', '3.14159265', '3.1'), # leading zero in precision ('.4f', '3.14159265', '3.1416'), ('.6f', '3.14159265', '3.141593'), ('.7f', '3.14159265', '3.1415926'), # round-half-even! @@ -1048,6 +1072,7 @@ def test_formatting(self): ('8,', '123456', ' 123,456'), ('08,', '123456', '0,123,456'), # special case: extra 0 needed ('+08,', '123456', '+123,456'), # but not if there's a sign + ('008,', '123456', '0,123,456'), # leading zero in width (' 08,', '123456', ' 123,456'), ('08,', '-123456', '-123,456'), ('+09,', '123456', '+0,123,456'), @@ -1064,6 +1089,20 @@ def test_formatting(self): (',%', '123.456789', '12,345.6789%'), (',e', '123456', '1.23456e+5'), (',E', '123456', '1.23456E+5'), + # ... with '_' instead + ('_', '1234567', '1_234_567'), + ('07_', '1234.56', '1_234.56'), + ('_', '1.23456789', '1.23456789'), + ('_%', '123.456789', '12_345.6789%'), + # and now for something completely different... + ('.,', '1.23456789', '1.234,567,89'), + ('._', '1.23456789', '1.234_567_89'), + ('.6_f', '12345.23456789', '12345.234_568'), + (',._%', '123.456789', '12,345.678_9%'), + (',._e', '123456', '1.234_56e+5'), + (',.4_e', '123456', '1.234_6e+5'), + (',.3_e', '123456', '1.235e+5'), + (',._E', '123456', '1.234_56E+5'), # negative zero: default behavior ('.1f', '-0', '-0.0'), @@ -1137,6 +1176,10 @@ def test_formatting(self): # bytes format argument self.assertRaises(TypeError, Decimal(1).__format__, b'-020') + # precision or fractional part separator should follow after dot + self.assertRaises(ValueError, format, Decimal(1), '.f') + self.assertRaises(ValueError, format, Decimal(1), '._6f') + def test_negative_zero_format_directed_rounding(self): with self.decimal.localcontext() as ctx: ctx.rounding = ROUND_CEILING @@ -1708,8 +1751,13 @@ def test_threading(self): self.finish1 = threading.Event() self.finish2 = threading.Event() - th1 = threading.Thread(target=thfunc1, args=(self,)) - th2 = threading.Thread(target=thfunc2, args=(self,)) + # This test wants to start threads with an empty context, no matter + # the setting of sys.flags.thread_inherit_context. We pass the + # 'context' argument explicitly with an empty context instance. + th1 = threading.Thread(target=thfunc1, args=(self,), + context=contextvars.Context()) + th2 = threading.Thread(target=thfunc2, args=(self,), + context=contextvars.Context()) th1.start() th2.start() @@ -2573,8 +2621,8 @@ class PythonAPItests: def test_abc(self): Decimal = self.decimal.Decimal - self.assertTrue(issubclass(Decimal, numbers.Number)) - self.assertFalse(issubclass(Decimal, numbers.Real)) + self.assertIsSubclass(Decimal, numbers.Number) + self.assertNotIsSubclass(Decimal, numbers.Real) self.assertIsInstance(Decimal(0), numbers.Number) self.assertNotIsInstance(Decimal(0), numbers.Real) @@ -2673,7 +2721,7 @@ class MyDecimal(Decimal): def __init__(self, _): self.x = 'y' - self.assertTrue(issubclass(MyDecimal, Decimal)) + self.assertIsSubclass(MyDecimal, Decimal) r = MyDecimal.from_float(0.1) self.assertEqual(type(r), MyDecimal) @@ -2891,31 +2939,31 @@ def test_exception_hierarchy(self): Rounded = decimal.Rounded Clamped = decimal.Clamped - self.assertTrue(issubclass(DecimalException, ArithmeticError)) - - self.assertTrue(issubclass(InvalidOperation, DecimalException)) - self.assertTrue(issubclass(FloatOperation, DecimalException)) - self.assertTrue(issubclass(FloatOperation, TypeError)) - self.assertTrue(issubclass(DivisionByZero, DecimalException)) - self.assertTrue(issubclass(DivisionByZero, ZeroDivisionError)) - self.assertTrue(issubclass(Overflow, Rounded)) - self.assertTrue(issubclass(Overflow, Inexact)) - self.assertTrue(issubclass(Overflow, DecimalException)) - self.assertTrue(issubclass(Underflow, Inexact)) - self.assertTrue(issubclass(Underflow, Rounded)) - self.assertTrue(issubclass(Underflow, Subnormal)) - self.assertTrue(issubclass(Underflow, DecimalException)) - - self.assertTrue(issubclass(Subnormal, DecimalException)) - self.assertTrue(issubclass(Inexact, DecimalException)) - self.assertTrue(issubclass(Rounded, DecimalException)) - self.assertTrue(issubclass(Clamped, DecimalException)) - - self.assertTrue(issubclass(decimal.ConversionSyntax, InvalidOperation)) - self.assertTrue(issubclass(decimal.DivisionImpossible, InvalidOperation)) - self.assertTrue(issubclass(decimal.DivisionUndefined, InvalidOperation)) - self.assertTrue(issubclass(decimal.DivisionUndefined, ZeroDivisionError)) - self.assertTrue(issubclass(decimal.InvalidContext, InvalidOperation)) + self.assertIsSubclass(DecimalException, ArithmeticError) + + self.assertIsSubclass(InvalidOperation, DecimalException) + self.assertIsSubclass(FloatOperation, DecimalException) + self.assertIsSubclass(FloatOperation, TypeError) + self.assertIsSubclass(DivisionByZero, DecimalException) + self.assertIsSubclass(DivisionByZero, ZeroDivisionError) + self.assertIsSubclass(Overflow, Rounded) + self.assertIsSubclass(Overflow, Inexact) + self.assertIsSubclass(Overflow, DecimalException) + self.assertIsSubclass(Underflow, Inexact) + self.assertIsSubclass(Underflow, Rounded) + self.assertIsSubclass(Underflow, Subnormal) + self.assertIsSubclass(Underflow, DecimalException) + + self.assertIsSubclass(Subnormal, DecimalException) + self.assertIsSubclass(Inexact, DecimalException) + self.assertIsSubclass(Rounded, DecimalException) + self.assertIsSubclass(Clamped, DecimalException) + + self.assertIsSubclass(decimal.ConversionSyntax, InvalidOperation) + self.assertIsSubclass(decimal.DivisionImpossible, InvalidOperation) + self.assertIsSubclass(decimal.DivisionUndefined, InvalidOperation) + self.assertIsSubclass(decimal.DivisionUndefined, ZeroDivisionError) + self.assertIsSubclass(decimal.InvalidContext, InvalidOperation) @requires_cdecimal class CPythonAPItests(PythonAPItests, unittest.TestCase): @@ -2923,7 +2971,6 @@ class CPythonAPItests(PythonAPItests, unittest.TestCase): class PyPythonAPItests(PythonAPItests, unittest.TestCase): decimal = P - class ContextAPItests: def test_none_args(self): @@ -4371,6 +4418,51 @@ class CContextSubclassing(ContextSubclassing, unittest.TestCase): class PyContextSubclassing(ContextSubclassing, unittest.TestCase): decimal = P +class IEEEContexts: + + def test_ieee_context(self): + # issue 8786: Add support for IEEE 754 contexts to decimal module. + IEEEContext = self.decimal.IEEEContext + + def assert_rest(self, context): + self.assertEqual(context.clamp, 1) + assert_signals(self, context, 'traps', []) + assert_signals(self, context, 'flags', []) + + c = IEEEContext(32) + self.assertEqual(c.prec, 7) + self.assertEqual(c.Emax, 96) + self.assertEqual(c.Emin, -95) + assert_rest(self, c) + + c = IEEEContext(64) + self.assertEqual(c.prec, 16) + self.assertEqual(c.Emax, 384) + self.assertEqual(c.Emin, -383) + assert_rest(self, c) + + c = IEEEContext(128) + self.assertEqual(c.prec, 34) + self.assertEqual(c.Emax, 6144) + self.assertEqual(c.Emin, -6143) + assert_rest(self, c) + + # Invalid values + self.assertRaises(ValueError, IEEEContext, -1) + self.assertRaises(ValueError, IEEEContext, 123) + self.assertRaises(ValueError, IEEEContext, 1024) + + def test_constants(self): + # IEEEContext + IEEE_CONTEXT_MAX_BITS = self.decimal.IEEE_CONTEXT_MAX_BITS + self.assertIn(IEEE_CONTEXT_MAX_BITS, {256, 512}) + +@requires_cdecimal +class CIEEEContexts(IEEEContexts, unittest.TestCase): + decimal = C +class PyIEEEContexts(IEEEContexts, unittest.TestCase): + decimal = P + @skip_if_extra_functionality @requires_cdecimal class CheckAttributes(unittest.TestCase): @@ -4382,6 +4474,7 @@ def test_module_attributes(self): self.assertEqual(C.MAX_EMAX, P.MAX_EMAX) self.assertEqual(C.MIN_EMIN, P.MIN_EMIN) self.assertEqual(C.MIN_ETINY, P.MIN_ETINY) + self.assertEqual(C.IEEE_CONTEXT_MAX_BITS, P.IEEE_CONTEXT_MAX_BITS) self.assertTrue(C.HAVE_THREADS is True or C.HAVE_THREADS is False) self.assertTrue(P.HAVE_THREADS is True or P.HAVE_THREADS is False) @@ -4465,12 +4558,10 @@ def test_implicit_context(self): self.assertIs(Decimal("NaN").fma(7, 1).is_nan(), True) # three arg power self.assertEqual(pow(Decimal(10), 2, 7), 2) + self.assertEqual(pow(10, Decimal(2), 7), 2) if self.decimal == C: - self.assertEqual(pow(10, Decimal(2), 7), 2) self.assertEqual(pow(10, 2, Decimal(7)), 2) else: - # XXX: Three-arg power doesn't use __rpow__. - self.assertRaises(TypeError, pow, 10, Decimal(2), 7) # XXX: There is no special method to dispatch on the # third arg of three-arg power. self.assertRaises(TypeError, pow, 10, 2, Decimal(7)) @@ -4684,10 +4775,6 @@ def tearDown(self): sys.set_int_max_str_digits(self._previous_int_limit) super().tearDown() - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_implicit_context(self): # TODO(RUSTPYTHON): Remove this test when it pass - return super().test_implicit_context() class PyFunctionality(unittest.TestCase): """Extra functionality in decimal.py""" @@ -4872,42 +4959,6 @@ def test_py__round(self): class CFunctionality(unittest.TestCase): """Extra functionality in _decimal""" - @requires_extra_functionality - def test_c_ieee_context(self): - # issue 8786: Add support for IEEE 754 contexts to decimal module. - IEEEContext = C.IEEEContext - DECIMAL32 = C.DECIMAL32 - DECIMAL64 = C.DECIMAL64 - DECIMAL128 = C.DECIMAL128 - - def assert_rest(self, context): - self.assertEqual(context.clamp, 1) - assert_signals(self, context, 'traps', []) - assert_signals(self, context, 'flags', []) - - c = IEEEContext(DECIMAL32) - self.assertEqual(c.prec, 7) - self.assertEqual(c.Emax, 96) - self.assertEqual(c.Emin, -95) - assert_rest(self, c) - - c = IEEEContext(DECIMAL64) - self.assertEqual(c.prec, 16) - self.assertEqual(c.Emax, 384) - self.assertEqual(c.Emin, -383) - assert_rest(self, c) - - c = IEEEContext(DECIMAL128) - self.assertEqual(c.prec, 34) - self.assertEqual(c.Emax, 6144) - self.assertEqual(c.Emin, -6143) - assert_rest(self, c) - - # Invalid values - self.assertRaises(OverflowError, IEEEContext, 2**63) - self.assertRaises(ValueError, IEEEContext, -1) - self.assertRaises(ValueError, IEEEContext, 1024) - @requires_extra_functionality def test_c_context(self): Context = C.Context @@ -4928,12 +4979,6 @@ def test_constants(self): C.DecSubnormal, C.DecUnderflow ) - # IEEEContext - self.assertEqual(C.DECIMAL32, 32) - self.assertEqual(C.DECIMAL64, 64) - self.assertEqual(C.DECIMAL128, 128) - self.assertEqual(C.IEEE_CONTEXT_MAX_BITS, 512) - # Conditions for i, v in enumerate(cond): self.assertEqual(v, 1< at 0x..., file "%s", line %d>) MAKE_FUNCTION - LOAD_FAST 0 (x) + LOAD_FAST_BORROW 0 (x) GET_ITER CALL 0 -%3d LOAD_CONST 2 (1) +%3d LOAD_SMALL_INT 1 %3d BINARY_OP 0 (+) CALL 0 @@ -210,16 +230,17 @@ def bug42562(): dis_bug42562 = """\ RESUME 0 - RETURN_CONST 0 (None) + LOAD_CONST 0 (None) + RETURN_VALUE """ # Extended arg followed by NOP code_bug_45757 = bytes([ - opcode.opmap['EXTENDED_ARG'], 0x01, # EXTENDED_ARG 0x01 - opcode.opmap['NOP'], 0xFF, # NOP 0xFF - opcode.opmap['EXTENDED_ARG'], 0x01, # EXTENDED_ARG 0x01 - opcode.opmap['LOAD_CONST'], 0x29, # LOAD_CONST 0x29 - opcode.opmap['RETURN_VALUE'], 0x00, # RETURN_VALUE 0x00 + opcode.opmap['EXTENDED_ARG'], 0x01, + opcode.opmap['NOP'], 0xFF, + opcode.opmap['EXTENDED_ARG'], 0x01, + opcode.opmap['LOAD_CONST'], 0x29, + opcode.opmap['RETURN_VALUE'], 0x00, ]) dis_bug_45757 = """\ @@ -256,25 +277,27 @@ def wrap_func_w_kwargs(): %3d RESUME 0 %3d LOAD_GLOBAL 1 (func_w_kwargs + NULL) - LOAD_CONST 1 (1) - LOAD_CONST 2 (2) - LOAD_CONST 3 (5) - LOAD_CONST 4 (('c',)) + LOAD_SMALL_INT 1 + LOAD_SMALL_INT 2 + LOAD_SMALL_INT 5 + LOAD_CONST 1 (('c',)) CALL_KW 3 POP_TOP - RETURN_CONST 0 (None) + LOAD_CONST 2 (None) + RETURN_VALUE """ % (wrap_func_w_kwargs.__code__.co_firstlineno, wrap_func_w_kwargs.__code__.co_firstlineno + 1) dis_intrinsic_1_2 = """\ 0 RESUME 0 - 1 LOAD_CONST 0 (0) + 1 LOAD_SMALL_INT 0 LOAD_CONST 1 (('*',)) IMPORT_NAME 0 (math) CALL_INTRINSIC_1 2 (INTRINSIC_IMPORT_STAR) POP_TOP - RETURN_CONST 2 (None) + LOAD_CONST 2 (None) + RETURN_VALUE """ dis_intrinsic_1_5 = """\ @@ -300,7 +323,8 @@ def wrap_func_w_kwargs(): %3d LOAD_GLOBAL 0 (spam) POP_TOP - RETURN_CONST 0 (None) + LOAD_CONST 0 (None) + RETURN_VALUE """ _BIG_LINENO_FORMAT2 = """\ @@ -308,17 +332,20 @@ def wrap_func_w_kwargs(): %4d LOAD_GLOBAL 0 (spam) POP_TOP - RETURN_CONST 0 (None) + LOAD_CONST 0 (None) + RETURN_VALUE """ dis_module_expected_results = """\ Disassembly of f: 4 RESUME 0 - RETURN_CONST 0 (None) + LOAD_CONST 0 (None) + RETURN_VALUE Disassembly of g: 5 RESUME 0 - RETURN_CONST 0 (None) + LOAD_CONST 0 (None) + RETURN_VALUE """ @@ -328,7 +355,7 @@ def wrap_func_w_kwargs(): 0 RESUME 0 1 LOAD_NAME 0 (x) - LOAD_CONST 0 (1) + LOAD_SMALL_INT 1 BINARY_OP 0 (+) RETURN_VALUE """ @@ -339,10 +366,11 @@ def wrap_func_w_kwargs(): 0 RESUME 0 1 LOAD_NAME 0 (x) - LOAD_CONST 0 (1) + LOAD_SMALL_INT 1 BINARY_OP 0 (+) STORE_NAME 0 (x) - RETURN_CONST 1 (None) + LOAD_CONST 1 (None) + RETURN_VALUE """ annot_stmt_str = """\ @@ -354,34 +382,54 @@ def wrap_func_w_kwargs(): # leading newline is for a reason (tests lineno) dis_annot_stmt_str = """\ - 0 RESUME 0 + -- MAKE_CELL 0 (__conditional_annotations__) - 2 SETUP_ANNOTATIONS - LOAD_CONST 0 (1) - STORE_NAME 0 (x) - LOAD_NAME 1 (int) - LOAD_NAME 2 (__annotations__) - LOAD_CONST 1 ('x') - STORE_SUBSCR + 0 RESUME 0 - 3 LOAD_NAME 3 (fun) - PUSH_NULL - LOAD_CONST 0 (1) - CALL 1 - LOAD_NAME 2 (__annotations__) - LOAD_CONST 2 ('y') - STORE_SUBSCR + 2 LOAD_CONST 1 (", line 2>) + MAKE_FUNCTION + STORE_NAME 4 (__annotate__) + BUILD_SET 0 + STORE_NAME 0 (__conditional_annotations__) + LOAD_SMALL_INT 1 + STORE_NAME 1 (x) + LOAD_NAME 0 (__conditional_annotations__) + LOAD_SMALL_INT 0 + SET_ADD 1 + POP_TOP - 4 LOAD_CONST 0 (1) - LOAD_NAME 4 (lst) - LOAD_NAME 3 (fun) - PUSH_NULL - LOAD_CONST 3 (0) - CALL 1 - STORE_SUBSCR - LOAD_NAME 1 (int) - POP_TOP - RETURN_CONST 4 (None) + 3 LOAD_NAME 0 (__conditional_annotations__) + LOAD_SMALL_INT 1 + SET_ADD 1 + POP_TOP + + 4 LOAD_SMALL_INT 1 + LOAD_NAME 2 (lst) + LOAD_NAME 3 (fun) + PUSH_NULL + LOAD_SMALL_INT 0 + CALL 1 + STORE_SUBSCR + LOAD_CONST 2 (None) + RETURN_VALUE +""" + +fn_with_annotate_str = """ +def foo(a: int, b: str) -> str: + return a * b +""" + +dis_fn_with_annotate_str = """\ + 0 RESUME 0 + + 2 LOAD_CONST 0 (", line 2>) + MAKE_FUNCTION + LOAD_CONST 1 (", line 2>) + MAKE_FUNCTION + SET_FUNCTION_ATTRIBUTE 16 (annotate) + STORE_NAME 0 (foo) + LOAD_CONST 2 (None) + RETURN_VALUE """ compound_stmt_str = """\ @@ -393,17 +441,16 @@ def wrap_func_w_kwargs(): dis_compound_stmt_str = """\ 0 RESUME 0 - 1 LOAD_CONST 0 (0) + 1 LOAD_SMALL_INT 0 STORE_NAME 0 (x) - 2 NOP + 2 L1: NOP - 3 L1: LOAD_NAME 0 (x) - LOAD_CONST 1 (1) + 3 LOAD_NAME 0 (x) + LOAD_SMALL_INT 1 BINARY_OP 13 (+=) STORE_NAME 0 (x) - - 2 JUMP_BACKWARD 7 (to L1) + JUMP_BACKWARD 12 (to L1) """ dis_traceback = """\ @@ -411,8 +458,8 @@ def wrap_func_w_kwargs(): %4d NOP -%4d L1: LOAD_CONST 1 (1) - LOAD_CONST 2 (0) +%4d L1: LOAD_SMALL_INT 1 + LOAD_SMALL_INT 0 --> BINARY_OP 11 (/) POP_TOP @@ -423,21 +470,22 @@ def wrap_func_w_kwargs(): %4d LOAD_GLOBAL 0 (Exception) CHECK_EXC_MATCH - POP_JUMP_IF_FALSE 23 (to L7) + POP_JUMP_IF_FALSE 24 (to L7) + NOT_TAKEN STORE_FAST 0 (e) %4d L4: LOAD_FAST 0 (e) LOAD_ATTR 2 (__traceback__) STORE_FAST 1 (tb) L5: POP_EXCEPT - LOAD_CONST 0 (None) + LOAD_CONST 1 (None) STORE_FAST 0 (e) DELETE_FAST 0 (e) %4d LOAD_FAST 1 (tb) RETURN_VALUE - -- L6: LOAD_CONST 0 (None) + -- L6: LOAD_CONST 1 (None) STORE_FAST 0 (e) DELETE_FAST 0 (e) RERAISE 1 @@ -467,20 +515,20 @@ def _fstring(a, b, c, d): dis_fstring = """\ %3d RESUME 0 -%3d LOAD_FAST 0 (a) +%3d LOAD_FAST_BORROW 0 (a) FORMAT_SIMPLE - LOAD_CONST 1 (' ') - LOAD_FAST 1 (b) - LOAD_CONST 2 ('4') + LOAD_CONST 0 (' ') + LOAD_FAST_BORROW 1 (b) + LOAD_CONST 1 ('4') FORMAT_WITH_SPEC - LOAD_CONST 1 (' ') - LOAD_FAST 2 (c) + LOAD_CONST 0 (' ') + LOAD_FAST_BORROW 2 (c) CONVERT_VALUE 2 (repr) FORMAT_SIMPLE - LOAD_CONST 1 (' ') - LOAD_FAST 3 (d) + LOAD_CONST 0 (' ') + LOAD_FAST_BORROW 3 (d) CONVERT_VALUE 2 (repr) - LOAD_CONST 2 ('4') + LOAD_CONST 1 ('4') FORMAT_WITH_SPEC BUILD_STRING 7 RETURN_VALUE @@ -494,43 +542,52 @@ def _with(c): dis_with = """\ %4d RESUME 0 -%4d LOAD_FAST 0 (c) - BEFORE_WITH +%4d LOAD_FAST_BORROW 0 (c) + COPY 1 + LOAD_SPECIAL 1 (__exit__) + SWAP 2 + SWAP 3 + LOAD_SPECIAL 0 (__enter__) + CALL 0 L1: POP_TOP -%4d LOAD_CONST 1 (1) +%4d LOAD_SMALL_INT 1 STORE_FAST 1 (x) -%4d L2: LOAD_CONST 0 (None) - LOAD_CONST 0 (None) - LOAD_CONST 0 (None) - CALL 2 +%4d L2: LOAD_CONST 1 (None) + LOAD_CONST 1 (None) + LOAD_CONST 1 (None) + CALL 3 POP_TOP -%4d LOAD_CONST 2 (2) +%4d LOAD_SMALL_INT 2 STORE_FAST 2 (y) - RETURN_CONST 0 (None) + LOAD_CONST 1 (None) + RETURN_VALUE %4d L3: PUSH_EXC_INFO WITH_EXCEPT_START TO_BOOL - POP_JUMP_IF_TRUE 1 (to L4) + POP_JUMP_IF_TRUE 2 (to L4) + NOT_TAKEN RERAISE 2 L4: POP_TOP L5: POP_EXCEPT POP_TOP POP_TOP + POP_TOP -%4d LOAD_CONST 2 (2) +%4d LOAD_SMALL_INT 2 STORE_FAST 2 (y) - RETURN_CONST 0 (None) + LOAD_CONST 1 (None) + RETURN_VALUE -- L6: COPY 3 POP_EXCEPT RERAISE 1 ExceptionTable: - L1 to L2 -> L3 [1] lasti - L3 to L5 -> L6 [3] lasti + L1 to L2 -> L3 [2] lasti + L3 to L5 -> L6 [4] lasti """ % (_with.__code__.co_firstlineno, _with.__code__.co_firstlineno + 1, _with.__code__.co_firstlineno + 2, @@ -551,7 +608,12 @@ async def _asyncwith(c): L1: RESUME 0 %4d LOAD_FAST 0 (c) - BEFORE_ASYNC_WITH + COPY 1 + LOAD_SPECIAL 3 (__aexit__) + SWAP 2 + SWAP 3 + LOAD_SPECIAL 2 (__aenter__) + CALL 0 GET_AWAITABLE 1 LOAD_CONST 0 (None) L2: SEND 3 (to L5) @@ -561,13 +623,13 @@ async def _asyncwith(c): L5: END_SEND L6: POP_TOP -%4d LOAD_CONST 1 (1) +%4d LOAD_SMALL_INT 1 STORE_FAST 1 (x) %4d L7: LOAD_CONST 0 (None) LOAD_CONST 0 (None) LOAD_CONST 0 (None) - CALL 2 + CALL 3 GET_AWAITABLE 2 LOAD_CONST 0 (None) L8: SEND 3 (to L11) @@ -577,14 +639,15 @@ async def _asyncwith(c): L11: END_SEND POP_TOP -%4d LOAD_CONST 2 (2) +%4d LOAD_SMALL_INT 2 STORE_FAST 2 (y) - RETURN_CONST 0 (None) + LOAD_CONST 0 (None) + RETURN_VALUE %4d L12: CLEANUP_THROW - L13: JUMP_BACKWARD_NO_INTERRUPT 25 (to L5) + L13: JUMP_BACKWARD_NO_INTERRUPT 26 (to L5) L14: CLEANUP_THROW - L15: JUMP_BACKWARD_NO_INTERRUPT 9 (to L11) + L15: JUMP_BACKWARD_NO_INTERRUPT 10 (to L11) L16: PUSH_EXC_INFO WITH_EXCEPT_START GET_AWAITABLE 2 @@ -596,35 +659,39 @@ async def _asyncwith(c): L20: CLEANUP_THROW L21: END_SEND TO_BOOL - POP_JUMP_IF_TRUE 1 (to L22) - RERAISE 2 - L22: POP_TOP - L23: POP_EXCEPT + POP_JUMP_IF_TRUE 2 (to L24) + L22: NOT_TAKEN + L23: RERAISE 2 + L24: POP_TOP + L25: POP_EXCEPT + POP_TOP POP_TOP POP_TOP -%4d LOAD_CONST 2 (2) +%4d LOAD_SMALL_INT 2 STORE_FAST 2 (y) - RETURN_CONST 0 (None) + LOAD_CONST 0 (None) + RETURN_VALUE - -- L24: COPY 3 + -- L26: COPY 3 POP_EXCEPT RERAISE 1 - L25: CALL_INTRINSIC_1 3 (INTRINSIC_STOPITERATION_ERROR) + L27: CALL_INTRINSIC_1 3 (INTRINSIC_STOPITERATION_ERROR) RERAISE 1 ExceptionTable: - L1 to L3 -> L25 [0] lasti - L3 to L4 -> L12 [3] - L4 to L6 -> L25 [0] lasti - L6 to L7 -> L16 [1] lasti - L7 to L9 -> L25 [0] lasti + L1 to L3 -> L27 [0] lasti + L3 to L4 -> L12 [4] + L4 to L6 -> L27 [0] lasti + L6 to L7 -> L16 [2] lasti + L7 to L9 -> L27 [0] lasti L9 to L10 -> L14 [2] - L10 to L13 -> L25 [0] lasti - L14 to L15 -> L25 [0] lasti - L16 to L18 -> L24 [3] lasti - L18 to L19 -> L20 [6] - L19 to L23 -> L24 [3] lasti - L23 to L25 -> L25 [0] lasti + L10 to L13 -> L27 [0] lasti + L14 to L15 -> L27 [0] lasti + L16 to L18 -> L26 [4] lasti + L18 to L19 -> L20 [7] + L19 to L22 -> L26 [4] lasti + L23 to L25 -> L26 [4] lasti + L25 to L27 -> L27 [0] lasti """ % (_asyncwith.__code__.co_firstlineno, _asyncwith.__code__.co_firstlineno + 1, _asyncwith.__code__.co_firstlineno + 2, @@ -652,9 +719,9 @@ def _tryfinallyconst(b): %4d NOP -%4d L1: LOAD_FAST 0 (a) +%4d L1: LOAD_FAST_BORROW 0 (a) -%4d L2: LOAD_FAST 1 (b) +%4d L2: LOAD_FAST_BORROW 1 (b) PUSH_NULL CALL 0 POP_TOP @@ -688,11 +755,12 @@ def _tryfinallyconst(b): %4d NOP -%4d LOAD_FAST 0 (b) +%4d LOAD_FAST_BORROW 0 (b) PUSH_NULL CALL 0 POP_TOP - RETURN_CONST 1 (1) + LOAD_SMALL_INT 1 + RETURN_VALUE -- L1: PUSH_EXC_INFO @@ -735,14 +803,14 @@ def foo(x): %4d RESUME 0 -%4d LOAD_FAST 0 (y) +%4d LOAD_FAST_BORROW 0 (y) BUILD_TUPLE 1 - LOAD_CONST 1 () + LOAD_CONST 0 () MAKE_FUNCTION SET_FUNCTION_ATTRIBUTE 8 (closure) STORE_FAST 1 (foo) -%4d LOAD_FAST 1 (foo) +%4d LOAD_FAST_BORROW 1 (foo) RETURN_VALUE """ % (_h.__code__.co_firstlineno, _h.__code__.co_firstlineno + 1, @@ -759,7 +827,7 @@ def foo(x): %4d RESUME 0 %4d LOAD_GLOBAL 1 (list + NULL) - LOAD_FAST 0 (x) + LOAD_FAST_BORROW 0 (x) BUILD_TUPLE 1 LOAD_CONST 1 ( at 0x..., file "%s", line %d>) MAKE_FUNCTION @@ -786,19 +854,19 @@ def foo(x): POP_TOP L1: RESUME 0 LOAD_FAST 0 (.0) - GET_ITER - L2: FOR_ITER 10 (to L3) + L2: FOR_ITER 14 (to L3) STORE_FAST 1 (z) LOAD_DEREF 2 (x) - LOAD_FAST 1 (z) + LOAD_FAST_BORROW 1 (z) BINARY_OP 0 (+) YIELD_VALUE 0 RESUME 5 POP_TOP - JUMP_BACKWARD 12 (to L2) + JUMP_BACKWARD 16 (to L2) L3: END_FOR - POP_TOP - RETURN_CONST 0 (None) + POP_ITER + LOAD_CONST 0 (None) + RETURN_VALUE -- L4: CALL_INTRINSIC_1 3 (INTRINSIC_STOPITERATION_ERROR) RERAISE 1 @@ -820,7 +888,7 @@ def load_test(x, y=0): %3d LOAD_FAST_LOAD_FAST 1 (x, y) STORE_FAST_STORE_FAST 50 (b, a) -%3d LOAD_FAST_LOAD_FAST 35 (a, b) +%3d LOAD_FAST_BORROW_LOAD_FAST_BORROW 35 (a, b) BUILD_TUPLE 2 RETURN_VALUE """ % (load_test.__code__.co_firstlineno, @@ -835,23 +903,24 @@ def loop_test(): %3d RESUME_CHECK 0 %3d BUILD_LIST 0 - LOAD_CONST 1 ((1, 2, 3)) + LOAD_CONST_MORTAL 2 ((1, 2, 3)) LIST_EXTEND 1 - LOAD_CONST 2 (3) + LOAD_SMALL_INT 3 BINARY_OP 5 (*) GET_ITER L1: FOR_ITER_LIST 14 (to L2) STORE_FAST 0 (i) %3d LOAD_GLOBAL_MODULE 1 (load_test + NULL) - LOAD_FAST 0 (i) + LOAD_FAST_BORROW 0 (i) CALL_PY_GENERAL 1 POP_TOP - JUMP_BACKWARD 16 (to L1) + JUMP_BACKWARD_{: <6} 16 (to L1) %3d L2: END_FOR - POP_TOP - RETURN_CONST 0 (None) + POP_ITER + LOAD_CONST_IMMORTAL 1 (None) + RETURN_VALUE """ % (loop_test.__code__.co_firstlineno, loop_test.__code__.co_firstlineno + 1, loop_test.__code__.co_firstlineno + 2, @@ -863,17 +932,16 @@ def extended_arg_quick(): dis_extended_arg_quick_code = """\ %3d RESUME 0 -%3d LOAD_CONST 1 (Ellipsis) +%3d LOAD_CONST 0 (Ellipsis) EXTENDED_ARG 1 UNPACK_EX 256 POP_TOP STORE_FAST 0 (_) - RETURN_CONST 0 (None) + LOAD_CONST 1 (None) + RETURN_VALUE """% (extended_arg_quick.__code__.co_firstlineno, extended_arg_quick.__code__.co_firstlineno + 1,) -ADAPTIVE_WARMUP_DELAY = 2 - class DisTestBase(unittest.TestCase): "Common utilities for DisTests and TestDisTraceback" @@ -937,31 +1005,112 @@ def test_opname(self): def test_boundaries(self): self.assertEqual(dis.opmap["EXTENDED_ARG"], dis.EXTENDED_ARG) + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 29 not less than or equal to 20 def test_widths(self): long_opcodes = set(['JUMP_BACKWARD_NO_INTERRUPT', - 'INSTRUMENTED_CALL_FUNCTION_EX']) - for opcode, opname in enumerate(dis.opname): + 'LOAD_FAST_BORROW_LOAD_FAST_BORROW', + 'INSTRUMENTED_CALL_FUNCTION_EX', + 'ANNOTATIONS_PLACEHOLDER']) + for op, opname in enumerate(dis.opname): if opname in long_opcodes or opname.startswith("INSTRUMENTED"): continue + if opname in opcode._specialized_opmap: + continue with self.subTest(opname=opname): width = dis._OPNAME_WIDTH - if opcode in dis.hasarg: + if op in dis.hasarg: width += 1 + dis._OPARG_WIDTH self.assertLessEqual(len(opname), width) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_dis(self): self.do_disassembly_test(_f, dis_f) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_dis_with_offsets(self): self.do_disassembly_test(_f, dis_f_with_offsets, show_offsets=True) - @unittest.expectedFailure # TODO: RUSTPYTHON + @requires_debug_ranges() + def test_dis_with_all_positions(self): + def format_instr_positions(instr): + values = tuple('?' if p is None else p for p in instr.positions) + return '%s:%s-%s:%s' % (values[0], values[2], values[1], values[3]) + + instrs = list(dis.get_instructions(_f)) + for instr in instrs: + with self.subTest(instr=instr): + self.assertTrue(all(p is not None for p in instr.positions)) + positions = tuple(map(format_instr_positions, instrs)) + expected = dis_f_with_positions_format % positions + self.do_disassembly_test(_f, expected, show_positions=True) + + @requires_debug_ranges() + def test_dis_with_some_positions(self): + code = ("def f():\n" + " try: pass\n" + " finally:pass") + f = compile(ast.parse(code), "?", "exec").co_consts[0] + + expect = '\n'.join([ + '1:0-1:0 RESUME 0', + '', + '2:3-3:15 NOP', + '', + '3:11-3:15 LOAD_CONST 0 (None)', + '3:11-3:15 RETURN_VALUE', + '', + ' -- L1: PUSH_EXC_INFO', + '', + '3:11-3:15 RERAISE 0', + '', + ' -- L2: COPY 3', + ' -- POP_EXCEPT', + ' -- RERAISE 1', + 'ExceptionTable:', + ' L1 to L2 -> L2 [1] lasti', + '', + ]) + self.do_disassembly_test(f, expect, show_positions=True) + + @requires_debug_ranges() + def test_dis_with_linenos_but_no_columns(self): + code = "def f():\n\tx = 1" + tree = ast.parse(code) + func = tree.body[0] + ass_x = func.body[0].targets[0] + # remove columns information but keep line information + ass_x.col_offset = ass_x.end_col_offset = -1 + f = compile(tree, "?", "exec").co_consts[0] + + expect = '\n'.join([ + '1:0-1:0 RESUME 0', + '', + '2:5-2:6 LOAD_SMALL_INT 1', + '2:?-2:? STORE_FAST 0 (x)', + '2:?-2:? LOAD_CONST 1 (None)', + '2:?-2:? RETURN_VALUE', + '', + ]) + self.do_disassembly_test(f, expect, show_positions=True) + + def test_dis_with_no_positions(self): + def f(): + pass + + f.__code__ = f.__code__.replace(co_linetable=b'') + expect = '\n'.join([ + ' RESUME 0', + ' LOAD_CONST 0 (None)', + ' RETURN_VALUE', + '', + ]) + self.do_disassembly_test(f, expect, show_positions=True) + + @unittest.expectedFailure # TODO: RUSTPYTHON def test_bug_708901(self): self.do_disassembly_test(bug708901, dis_bug708901) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_bug_1333982(self): # This one is checking bytecodes generated for an `assert` statement, # so fails if the tests are run with -O. Skip this test then. @@ -970,38 +1119,35 @@ def test_bug_1333982(self): self.do_disassembly_test(bug1333982, dis_bug1333982) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_bug_42562(self): self.do_disassembly_test(bug42562, dis_bug42562) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_bug_45757(self): # Extended arg followed by NOP self.do_disassembly_test(code_bug_45757, dis_bug_45757) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_bug_46724(self): # Test that negative operargs are handled properly self.do_disassembly_test(bug46724, dis_bug46724) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_kw_names(self): # Test that value is displayed for keyword argument names: self.do_disassembly_test(wrap_func_w_kwargs, dis_kw_names) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_intrinsic_1(self): # Test that argrepr is displayed for CALL_INTRINSIC_1 self.do_disassembly_test("from math import *", dis_intrinsic_1_2) self.do_disassembly_test("+a", dis_intrinsic_1_5) self.do_disassembly_test("(*a,)", dis_intrinsic_1_6) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_intrinsic_2(self): self.assertIn("CALL_INTRINSIC_2 1 (INTRINSIC_PREP_RERAISE_STAR)", self.get_disassembly("try: pass\nexcept* Exception: x")) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_big_linenos(self): def func(count): namespace = {} @@ -1026,51 +1172,49 @@ def func(count): from test import dis_module self.do_disassembly_test(dis_module, dis_module_expected_results) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_disassemble_str(self): self.do_disassembly_test(expr_str, dis_expr_str) self.do_disassembly_test(simple_stmt_str, dis_simple_stmt_str) self.do_disassembly_test(annot_stmt_str, dis_annot_stmt_str) + self.do_disassembly_test(fn_with_annotate_str, dis_fn_with_annotate_str) self.do_disassembly_test(compound_stmt_str, dis_compound_stmt_str) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_disassemble_bytes(self): self.do_disassembly_test(_f.__code__.co_code, dis_f_co_code) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_disassemble_class(self): self.do_disassembly_test(_C, dis_c) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_disassemble_instance_method(self): self.do_disassembly_test(_C(1).__init__, dis_c_instance_method) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_disassemble_instance_method_bytes(self): method_bytecode = _C(1).__init__.__code__.co_code self.do_disassembly_test(method_bytecode, dis_c_instance_method_bytes) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_disassemble_static_method(self): self.do_disassembly_test(_C.sm, dis_c_static_method) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_disassemble_class_method(self): self.do_disassembly_test(_C.cm, dis_c_class_method) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_disassemble_generator(self): gen_func_disas = self.get_disassembly(_g) # Generator function gen_disas = self.get_disassembly(_g(1)) # Generator iterator self.assertEqual(gen_disas, gen_func_disas) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_disassemble_async_generator(self): agen_func_disas = self.get_disassembly(_ag) # Async generator function agen_disas = self.get_disassembly(_ag(1)) # Async generator iterator self.assertEqual(agen_disas, agen_func_disas) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_disassemble_coroutine(self): coro_func_disas = self.get_disassembly(_co) # Coroutine function coro = _co(1) # Coroutine object @@ -1078,24 +1222,23 @@ def test_disassemble_coroutine(self): coro_disas = self.get_disassembly(coro) self.assertEqual(coro_disas, coro_func_disas) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_disassemble_fstring(self): self.do_disassembly_test(_fstring, dis_fstring) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_disassemble_with(self): self.do_disassembly_test(_with, dis_with) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_disassemble_asyncwith(self): self.do_disassembly_test(_asyncwith, dis_asyncwith) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_disassemble_try_finally(self): self.do_disassembly_test(_tryfinally, dis_tryfinally) self.do_disassembly_test(_tryfinallyconst, dis_tryfinallyconst) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_dis_none(self): try: del sys.last_exc @@ -1107,7 +1250,6 @@ def test_dis_none(self): pass self.assertRaises(RuntimeError, dis.dis, None) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_dis_traceback(self): self.maxDiff = None try: @@ -1127,7 +1269,7 @@ def test_dis_traceback(self): def test_dis_object(self): self.assertRaises(TypeError, dis.dis, object()) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_disassemble_recursive(self): def check(expected, **kwargs): dis = self.get_disassembly(_h, **kwargs) @@ -1148,9 +1290,28 @@ def test__try_compile_no_context_exc_on_error(self): except Exception as e: self.assertIsNone(e.__context__) + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: No END_ASYNC_FOR in disassembly of async for + def test_async_for_presentation(self): + + async def afunc(): + async for letter in async_iter1: + l2 + l3 + + disassembly = self.get_disassembly(afunc) + for line in disassembly.split("\n"): + if "END_ASYNC_FOR" in line: + break + else: + self.fail("No END_ASYNC_FOR in disassembly of async for") + self.assertNotIn("to", line) + self.assertIn("from", line) + + @staticmethod - def code_quicken(f, times=ADAPTIVE_WARMUP_DELAY): - for _ in range(times): + def code_quicken(f): + _testinternalcapi = import_helper.import_module("_testinternalcapi") + for _ in range(_testinternalcapi.SPECIALIZATION_THRESHOLD): f() @cpython_only @@ -1160,52 +1321,13 @@ def test_super_instructions(self): got = self.get_disassembly(load_test, adaptive=True) self.do_disassembly_compare(got, dis_load_test_quickened_code) - @cpython_only - @requires_specialization - def test_binary_specialize(self): - binary_op_quicken = """\ - 0 RESUME_CHECK 0 - - 1 LOAD_NAME 0 (a) - LOAD_NAME 1 (b) - %s - RETURN_VALUE -""" - co_int = compile('a + b', "", "eval") - self.code_quicken(lambda: exec(co_int, {}, {'a': 1, 'b': 2})) - got = self.get_disassembly(co_int, adaptive=True) - self.do_disassembly_compare(got, binary_op_quicken % "BINARY_OP_ADD_INT 0 (+)") - - co_unicode = compile('a + b', "", "eval") - self.code_quicken(lambda: exec(co_unicode, {}, {'a': 'a', 'b': 'b'})) - got = self.get_disassembly(co_unicode, adaptive=True) - self.do_disassembly_compare(got, binary_op_quicken % "BINARY_OP_ADD_UNICODE 0 (+)") - - binary_subscr_quicken = """\ - 0 RESUME_CHECK 0 - - 1 LOAD_NAME 0 (a) - LOAD_CONST 0 (0) - %s - RETURN_VALUE -""" - co_list = compile('a[0]', "", "eval") - self.code_quicken(lambda: exec(co_list, {}, {'a': [0]})) - got = self.get_disassembly(co_list, adaptive=True) - self.do_disassembly_compare(got, binary_subscr_quicken % "BINARY_SUBSCR_LIST_INT") - - co_dict = compile('a[0]', "", "eval") - self.code_quicken(lambda: exec(co_dict, {}, {'a': {0: '1'}})) - got = self.get_disassembly(co_dict, adaptive=True) - self.do_disassembly_compare(got, binary_subscr_quicken % "BINARY_SUBSCR_DICT") - @cpython_only @requires_specialization def test_load_attr_specialize(self): load_attr_quicken = """\ 0 RESUME_CHECK 0 - 1 LOAD_CONST 0 ('a') + 1 LOAD_CONST_IMMORTAL 0 ('a') LOAD_ATTR_SLOT 0 (__class__) RETURN_VALUE """ @@ -1222,7 +1344,7 @@ def test_call_specialize(self): 1 LOAD_NAME 0 (str) PUSH_NULL - LOAD_CONST 0 (1) + LOAD_SMALL_INT 1 CALL_STR_1 1 RETURN_VALUE """ @@ -1235,11 +1357,43 @@ def test_call_specialize(self): @requires_specialization def test_loop_quicken(self): # Loop can trigger a quicken where the loop is located - self.code_quicken(loop_test, 4) + self.code_quicken(loop_test) got = self.get_disassembly(loop_test, adaptive=True) - expected = dis_loop_test_quickened_code + jit = sys._jit.is_enabled() + expected = dis_loop_test_quickened_code.format("JIT" if jit else "NO_JIT") self.do_disassembly_compare(got, expected) + @cpython_only + @requires_specialization + def test_loop_with_conditional_at_end_is_quickened(self): + _testinternalcapi = import_helper.import_module("_testinternalcapi") + def for_loop_true(x): + for _ in range(_testinternalcapi.SPECIALIZATION_THRESHOLD): + if x: + pass + + for_loop_true(True) + self.assertIn('FOR_ITER_RANGE', + self.get_disassembly(for_loop_true, adaptive=True)) + + def for_loop_false(x): + for _ in range(_testinternalcapi.SPECIALIZATION_THRESHOLD): + if x: + pass + + for_loop_false(False) + self.assertIn('FOR_ITER_RANGE', + self.get_disassembly(for_loop_false, adaptive=True)) + + def while_loop(): + i = 0 + while i < _testinternalcapi.SPECIALIZATION_THRESHOLD: + i += 1 + + while_loop() + self.assertIn('COMPARE_OP_INT', + self.get_disassembly(while_loop, adaptive=True)) + @cpython_only def test_extended_arg_quick(self): got = self.get_disassembly(extended_arg_quick) @@ -1254,7 +1408,7 @@ def f(): self.code_quicken(f) else: # "copy" the code to un-quicken it: - f.__code__ = f.__code__.replace() + reset_code(f) for instruction in _unroll_caches_as_Instructions(dis.get_instructions( f, show_caches=True, adaptive=adaptive ), show_caches=True): @@ -1321,6 +1475,10 @@ def get_disassembly(self, func, lasti=-1, wrapper=True, **kwargs): dis.disassemble(func, lasti, file=output, **kwargs) return output.getvalue() + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: No END_ASYNC_FOR in disassembly of async for + def test_async_for_presentation(self): + return super().test_async_for_presentation() + if dis.code_info.__doc__ is None: code_info_consts = "0: None" @@ -1335,7 +1493,7 @@ def get_disassembly(self, func, lasti=-1, wrapper=True, **kwargs): Kw-only arguments: 0 Number of locals: 1 Stack size: \\d+ -Flags: OPTIMIZED, NEWLOCALS +Flags: OPTIMIZED, NEWLOCALS, HAS_DOCSTRING Constants: {code_info_consts} Names: @@ -1361,8 +1519,8 @@ def f(c=c): Stack size: \\d+ Flags: OPTIMIZED, NEWLOCALS, VARARGS, VARKEYWORDS, GENERATOR Constants: - 0: None - 1: + 0: + 1: None Variable names: 0: a 1: b @@ -1380,10 +1538,12 @@ def f(c=c): 2: [abedfxyz] 3: [abedfxyz] 4: [abedfxyz] - 5: [abedfxyz]""" + 5: [abedfxyz] + 6: [abedfxyz] + 7: [abedfxyz]""" # NOTE: the order of the cell variables above depends on dictionary order! -co_tricky_nested_f = tricky.__func__.__code__.co_consts[1] +co_tricky_nested_f = tricky.__func__.__code__.co_consts[0] code_info_tricky_nested_f = """\ Filename: (.*) @@ -1447,7 +1607,6 @@ def f(c=c): Flags: 0x0 Constants: 0: 0 - 1: 1 Names: 0: x""" @@ -1467,8 +1626,8 @@ async def async_def(): Stack size: \\d+ Flags: OPTIMIZED, NEWLOCALS, COROUTINE Constants: - 0: None - 1: 1 + 0: 1 + 1: None Names: 0: b 1: c @@ -1487,13 +1646,13 @@ class CodeInfoTests(unittest.TestCase): (async_def, code_info_async_def) ] - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_code_info(self): self.maxDiff = 1000 for x, expected in self.test_pairs: self.assertRegex(dis.code_info(x), expected) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_show_code(self): self.maxDiff = 1000 for x, expected in self.test_pairs: @@ -1555,9 +1714,9 @@ def jumpy(): expected_outer_line = 1 _line_offset = outer.__code__.co_firstlineno - 1 code_object_f = outer.__code__.co_consts[1] -# expected_f_line = code_object_f.co_firstlineno - _line_offset # TODO: RUSTPYTHON; AttributeError: 'int' object has no attribute 'co_firstlineno' -# code_object_inner = code_object_f.co_consts[1] # TODO: RUSTPYTHON; AttributeError: 'int' object has no attribute 'co_consts' -# expected_inner_line = code_object_inner.co_firstlineno - _line_offset # TODO: RUSTPYTHON; NameError: name 'code_object_inner' is not defined +expected_f_line = code_object_f.co_firstlineno - _line_offset +code_object_inner = code_object_f.co_consts[1] +expected_inner_line = code_object_inner.co_firstlineno - _line_offset expected_jumpy_line = 1 # The following lines are useful to regenerate the expected results after @@ -1566,9 +1725,17 @@ def jumpy(): # code_object_inner before rerunning the tests def _stringify_instruction(instr): - # Since line numbers and other offsets change a lot for these - # test cases, ignore them. - return f" {instr._replace(positions=None)!r}," + # Since postions offsets change a lot for these test cases, ignore them. + base = ( + f" make_inst(opname={instr.opname!r}, arg={instr.arg!r}, argval={instr.argval!r}, " + + f"argrepr={instr.argrepr!r}, offset={instr.offset}, start_offset={instr.start_offset}, " + + f"starts_line={instr.starts_line!r}, line_number={instr.line_number}" + ) + if instr.label is not None: + base += f", label={instr.label!r}" + if instr.cache_info: + base += f", cache_info={instr.cache_info!r}" + return base + ")," def _prepare_test_cases(): ignore = io.StringIO() @@ -1606,201 +1773,214 @@ def _prepare_test_cases(): # from test.test_dis import _prepare_test_cases; _prepare_test_cases() -Instruction = dis.Instruction +make_inst = dis.Instruction.make expected_opinfo_outer = [ - Instruction(opname='MAKE_CELL', opcode=94, arg=0, argval='a', argrepr='a', offset=0, start_offset=0, starts_line=True, line_number=None, label=None, positions=None), - Instruction(opname='MAKE_CELL', opcode=94, arg=1, argval='b', argrepr='b', offset=2, start_offset=2, starts_line=False, line_number=None, label=None, positions=None), - Instruction(opname='RESUME', opcode=149, arg=0, argval=0, argrepr='', offset=4, start_offset=4, starts_line=True, line_number=1, label=None, positions=None), - Instruction(opname='LOAD_CONST', opcode=83, arg=5, argval=(3, 4), argrepr='(3, 4)', offset=6, start_offset=6, starts_line=True, line_number=2, label=None, positions=None), - Instruction(opname='LOAD_FAST', opcode=85, arg=0, argval='a', argrepr='a', offset=8, start_offset=8, starts_line=False, line_number=2, label=None, positions=None), - Instruction(opname='LOAD_FAST', opcode=85, arg=1, argval='b', argrepr='b', offset=10, start_offset=10, starts_line=False, line_number=2, label=None, positions=None), - Instruction(opname='BUILD_TUPLE', opcode=52, arg=2, argval=2, argrepr='', offset=12, start_offset=12, starts_line=False, line_number=2, label=None, positions=None), - Instruction(opname='LOAD_CONST', opcode=83, arg=1, argval=code_object_f, argrepr=repr(code_object_f), offset=14, start_offset=14, starts_line=False, line_number=2, label=None, positions=None), - Instruction(opname='MAKE_FUNCTION', opcode=26, arg=None, argval=None, argrepr='', offset=16, start_offset=16, starts_line=False, line_number=2, label=None, positions=None), - Instruction(opname='SET_FUNCTION_ATTRIBUTE', opcode=106, arg=8, argval=8, argrepr='closure', offset=18, start_offset=18, starts_line=False, line_number=2, label=None, positions=None), - Instruction(opname='SET_FUNCTION_ATTRIBUTE', opcode=106, arg=1, argval=1, argrepr='defaults', offset=20, start_offset=20, starts_line=False, line_number=2, label=None, positions=None), - Instruction(opname='STORE_FAST', opcode=110, arg=2, argval='f', argrepr='f', offset=22, start_offset=22, starts_line=False, line_number=2, label=None, positions=None), - Instruction(opname='LOAD_GLOBAL', opcode=91, arg=1, argval='print', argrepr='print + NULL', offset=24, start_offset=24, starts_line=True, line_number=7, label=None, positions=None), - Instruction(opname='LOAD_DEREF', opcode=84, arg=0, argval='a', argrepr='a', offset=34, start_offset=34, starts_line=False, line_number=7, label=None, positions=None), - Instruction(opname='LOAD_DEREF', opcode=84, arg=1, argval='b', argrepr='b', offset=36, start_offset=36, starts_line=False, line_number=7, label=None, positions=None), - Instruction(opname='LOAD_CONST', opcode=83, arg=2, argval='', argrepr="''", offset=38, start_offset=38, starts_line=False, line_number=7, label=None, positions=None), - Instruction(opname='LOAD_CONST', opcode=83, arg=3, argval=1, argrepr='1', offset=40, start_offset=40, starts_line=False, line_number=7, label=None, positions=None), - Instruction(opname='BUILD_LIST', opcode=47, arg=0, argval=0, argrepr='', offset=42, start_offset=42, starts_line=False, line_number=7, label=None, positions=None), - Instruction(opname='BUILD_MAP', opcode=48, arg=0, argval=0, argrepr='', offset=44, start_offset=44, starts_line=False, line_number=7, label=None, positions=None), - Instruction(opname='LOAD_CONST', opcode=83, arg=4, argval='Hello world!', argrepr="'Hello world!'", offset=46, start_offset=46, starts_line=False, line_number=7, label=None, positions=None), - Instruction(opname='CALL', opcode=53, arg=7, argval=7, argrepr='', offset=48, start_offset=48, starts_line=False, line_number=7, label=None, positions=None), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=56, start_offset=56, starts_line=False, line_number=7, label=None, positions=None), - Instruction(opname='LOAD_FAST', opcode=85, arg=2, argval='f', argrepr='f', offset=58, start_offset=58, starts_line=True, line_number=8, label=None, positions=None), - Instruction(opname='RETURN_VALUE', opcode=36, arg=None, argval=None, argrepr='', offset=60, start_offset=60, starts_line=False, line_number=8, label=None, positions=None), + make_inst(opname='MAKE_CELL', arg=0, argval='a', argrepr='a', offset=0, start_offset=0, starts_line=True, line_number=None), + make_inst(opname='MAKE_CELL', arg=1, argval='b', argrepr='b', offset=2, start_offset=2, starts_line=False, line_number=None), + make_inst(opname='RESUME', arg=0, argval=0, argrepr='', offset=4, start_offset=4, starts_line=True, line_number=1), + make_inst(opname='LOAD_CONST', arg=4, argval=(3, 4), argrepr='(3, 4)', offset=6, start_offset=6, starts_line=True, line_number=2), + make_inst(opname='LOAD_FAST_BORROW', arg=0, argval='a', argrepr='a', offset=8, start_offset=8, starts_line=False, line_number=2), + make_inst(opname='LOAD_FAST_BORROW', arg=1, argval='b', argrepr='b', offset=10, start_offset=10, starts_line=False, line_number=2), + make_inst(opname='BUILD_TUPLE', arg=2, argval=2, argrepr='', offset=12, start_offset=12, starts_line=False, line_number=2), + make_inst(opname='LOAD_CONST', arg=1, argval=code_object_f, argrepr=repr(code_object_f), offset=14, start_offset=14, starts_line=False, line_number=2), + make_inst(opname='MAKE_FUNCTION', arg=None, argval=None, argrepr='', offset=16, start_offset=16, starts_line=False, line_number=2), + make_inst(opname='SET_FUNCTION_ATTRIBUTE', arg=8, argval=8, argrepr='closure', offset=18, start_offset=18, starts_line=False, line_number=2), + make_inst(opname='SET_FUNCTION_ATTRIBUTE', arg=1, argval=1, argrepr='defaults', offset=20, start_offset=20, starts_line=False, line_number=2), + make_inst(opname='STORE_FAST', arg=2, argval='f', argrepr='f', offset=22, start_offset=22, starts_line=False, line_number=2), + make_inst(opname='LOAD_GLOBAL', arg=1, argval='print', argrepr='print + NULL', offset=24, start_offset=24, starts_line=True, line_number=7, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + make_inst(opname='LOAD_DEREF', arg=0, argval='a', argrepr='a', offset=34, start_offset=34, starts_line=False, line_number=7), + make_inst(opname='LOAD_DEREF', arg=1, argval='b', argrepr='b', offset=36, start_offset=36, starts_line=False, line_number=7), + make_inst(opname='LOAD_CONST', arg=2, argval='', argrepr="''", offset=38, start_offset=38, starts_line=False, line_number=7), + make_inst(opname='LOAD_SMALL_INT', arg=1, argval=1, argrepr='', offset=40, start_offset=40, starts_line=False, line_number=7), + make_inst(opname='BUILD_LIST', arg=0, argval=0, argrepr='', offset=42, start_offset=42, starts_line=False, line_number=7), + make_inst(opname='BUILD_MAP', arg=0, argval=0, argrepr='', offset=44, start_offset=44, starts_line=False, line_number=7), + make_inst(opname='LOAD_CONST', arg=3, argval='Hello world!', argrepr="'Hello world!'", offset=46, start_offset=46, starts_line=False, line_number=7), + make_inst(opname='CALL', arg=7, argval=7, argrepr='', offset=48, start_offset=48, starts_line=False, line_number=7, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=56, start_offset=56, starts_line=False, line_number=7), + make_inst(opname='LOAD_FAST_BORROW', arg=2, argval='f', argrepr='f', offset=58, start_offset=58, starts_line=True, line_number=8), + make_inst(opname='RETURN_VALUE', arg=None, argval=None, argrepr='', offset=60, start_offset=60, starts_line=False, line_number=8), ] expected_opinfo_f = [ - Instruction(opname='COPY_FREE_VARS', opcode=62, arg=2, argval=2, argrepr='', offset=0, start_offset=0, starts_line=True, line_number=None, label=None, positions=None), - Instruction(opname='MAKE_CELL', opcode=94, arg=0, argval='c', argrepr='c', offset=2, start_offset=2, starts_line=False, line_number=None, label=None, positions=None), - Instruction(opname='MAKE_CELL', opcode=94, arg=1, argval='d', argrepr='d', offset=4, start_offset=4, starts_line=False, line_number=None, label=None, positions=None), - Instruction(opname='RESUME', opcode=149, arg=0, argval=0, argrepr='', offset=6, start_offset=6, starts_line=True, line_number=2, label=None, positions=None), - Instruction(opname='LOAD_CONST', opcode=83, arg=2, argval=(5, 6), argrepr='(5, 6)', offset=8, start_offset=8, starts_line=True, line_number=3, label=None, positions=None), - Instruction(opname='LOAD_FAST', opcode=85, arg=3, argval='a', argrepr='a', offset=10, start_offset=10, starts_line=False, line_number=3, label=None, positions=None), - Instruction(opname='LOAD_FAST', opcode=85, arg=4, argval='b', argrepr='b', offset=12, start_offset=12, starts_line=False, line_number=3, label=None, positions=None), - Instruction(opname='LOAD_FAST', opcode=85, arg=0, argval='c', argrepr='c', offset=14, start_offset=14, starts_line=False, line_number=3, label=None, positions=None), - Instruction(opname='LOAD_FAST', opcode=85, arg=1, argval='d', argrepr='d', offset=16, start_offset=16, starts_line=False, line_number=3, label=None, positions=None), - Instruction(opname='BUILD_TUPLE', opcode=52, arg=4, argval=4, argrepr='', offset=18, start_offset=18, starts_line=False, line_number=3, label=None, positions=None), - #Instruction(opname='LOAD_CONST', opcode=83, arg=1, argval=code_object_inner, argrepr=repr(code_object_inner), offset=20, start_offset=20, starts_line=False, line_number=3, label=None, positions=None), # TODO: RUSTPYTHON; NameError: name 'code_object_inner' is not defined - Instruction(opname='MAKE_FUNCTION', opcode=26, arg=None, argval=None, argrepr='', offset=22, start_offset=22, starts_line=False, line_number=3, label=None, positions=None), - Instruction(opname='SET_FUNCTION_ATTRIBUTE', opcode=106, arg=8, argval=8, argrepr='closure', offset=24, start_offset=24, starts_line=False, line_number=3, label=None, positions=None), - Instruction(opname='SET_FUNCTION_ATTRIBUTE', opcode=106, arg=1, argval=1, argrepr='defaults', offset=26, start_offset=26, starts_line=False, line_number=3, label=None, positions=None), - Instruction(opname='STORE_FAST', opcode=110, arg=2, argval='inner', argrepr='inner', offset=28, start_offset=28, starts_line=False, line_number=3, label=None, positions=None), - Instruction(opname='LOAD_GLOBAL', opcode=91, arg=1, argval='print', argrepr='print + NULL', offset=30, start_offset=30, starts_line=True, line_number=5, label=None, positions=None), - Instruction(opname='LOAD_DEREF', opcode=84, arg=3, argval='a', argrepr='a', offset=40, start_offset=40, starts_line=False, line_number=5, label=None, positions=None), - Instruction(opname='LOAD_DEREF', opcode=84, arg=4, argval='b', argrepr='b', offset=42, start_offset=42, starts_line=False, line_number=5, label=None, positions=None), - Instruction(opname='LOAD_DEREF', opcode=84, arg=0, argval='c', argrepr='c', offset=44, start_offset=44, starts_line=False, line_number=5, label=None, positions=None), - Instruction(opname='LOAD_DEREF', opcode=84, arg=1, argval='d', argrepr='d', offset=46, start_offset=46, starts_line=False, line_number=5, label=None, positions=None), - Instruction(opname='CALL', opcode=53, arg=4, argval=4, argrepr='', offset=48, start_offset=48, starts_line=False, line_number=5, label=None, positions=None), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=56, start_offset=56, starts_line=False, line_number=5, label=None, positions=None), - Instruction(opname='LOAD_FAST', opcode=85, arg=2, argval='inner', argrepr='inner', offset=58, start_offset=58, starts_line=True, line_number=6, label=None, positions=None), - Instruction(opname='RETURN_VALUE', opcode=36, arg=None, argval=None, argrepr='', offset=60, start_offset=60, starts_line=False, line_number=6, label=None, positions=None), + make_inst(opname='COPY_FREE_VARS', arg=2, argval=2, argrepr='', offset=0, start_offset=0, starts_line=True, line_number=None), + make_inst(opname='MAKE_CELL', arg=0, argval='c', argrepr='c', offset=2, start_offset=2, starts_line=False, line_number=None), + make_inst(opname='MAKE_CELL', arg=1, argval='d', argrepr='d', offset=4, start_offset=4, starts_line=False, line_number=None), + make_inst(opname='RESUME', arg=0, argval=0, argrepr='', offset=6, start_offset=6, starts_line=True, line_number=2), + make_inst(opname='LOAD_CONST', arg=2, argval=(5, 6), argrepr='(5, 6)', offset=8, start_offset=8, starts_line=True, line_number=3), + make_inst(opname='LOAD_FAST_BORROW', arg=3, argval='a', argrepr='a', offset=10, start_offset=10, starts_line=False, line_number=3), + make_inst(opname='LOAD_FAST_BORROW', arg=4, argval='b', argrepr='b', offset=12, start_offset=12, starts_line=False, line_number=3), + make_inst(opname='LOAD_FAST_BORROW', arg=0, argval='c', argrepr='c', offset=14, start_offset=14, starts_line=False, line_number=3), + make_inst(opname='LOAD_FAST_BORROW', arg=1, argval='d', argrepr='d', offset=16, start_offset=16, starts_line=False, line_number=3), + make_inst(opname='BUILD_TUPLE', arg=4, argval=4, argrepr='', offset=18, start_offset=18, starts_line=False, line_number=3), + make_inst(opname='LOAD_CONST', arg=1, argval=code_object_inner, argrepr=repr(code_object_inner), offset=20, start_offset=20, starts_line=False, line_number=3), + make_inst(opname='MAKE_FUNCTION', arg=None, argval=None, argrepr='', offset=22, start_offset=22, starts_line=False, line_number=3), + make_inst(opname='SET_FUNCTION_ATTRIBUTE', arg=8, argval=8, argrepr='closure', offset=24, start_offset=24, starts_line=False, line_number=3), + make_inst(opname='SET_FUNCTION_ATTRIBUTE', arg=1, argval=1, argrepr='defaults', offset=26, start_offset=26, starts_line=False, line_number=3), + make_inst(opname='STORE_FAST', arg=2, argval='inner', argrepr='inner', offset=28, start_offset=28, starts_line=False, line_number=3), + make_inst(opname='LOAD_GLOBAL', arg=1, argval='print', argrepr='print + NULL', offset=30, start_offset=30, starts_line=True, line_number=5, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + make_inst(opname='LOAD_DEREF', arg=3, argval='a', argrepr='a', offset=40, start_offset=40, starts_line=False, line_number=5), + make_inst(opname='LOAD_DEREF', arg=4, argval='b', argrepr='b', offset=42, start_offset=42, starts_line=False, line_number=5), + make_inst(opname='LOAD_DEREF', arg=0, argval='c', argrepr='c', offset=44, start_offset=44, starts_line=False, line_number=5), + make_inst(opname='LOAD_DEREF', arg=1, argval='d', argrepr='d', offset=46, start_offset=46, starts_line=False, line_number=5), + make_inst(opname='CALL', arg=4, argval=4, argrepr='', offset=48, start_offset=48, starts_line=False, line_number=5, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=56, start_offset=56, starts_line=False, line_number=5), + make_inst(opname='LOAD_FAST_BORROW', arg=2, argval='inner', argrepr='inner', offset=58, start_offset=58, starts_line=True, line_number=6), + make_inst(opname='RETURN_VALUE', arg=None, argval=None, argrepr='', offset=60, start_offset=60, starts_line=False, line_number=6), ] expected_opinfo_inner = [ - Instruction(opname='COPY_FREE_VARS', opcode=62, arg=4, argval=4, argrepr='', offset=0, start_offset=0, starts_line=True, line_number=None, label=None, positions=None), - Instruction(opname='RESUME', opcode=149, arg=0, argval=0, argrepr='', offset=2, start_offset=2, starts_line=True, line_number=3, label=None, positions=None), - Instruction(opname='LOAD_GLOBAL', opcode=91, arg=1, argval='print', argrepr='print + NULL', offset=4, start_offset=4, starts_line=True, line_number=4, label=None, positions=None), - Instruction(opname='LOAD_DEREF', opcode=84, arg=2, argval='a', argrepr='a', offset=14, start_offset=14, starts_line=False, line_number=4, label=None, positions=None), - Instruction(opname='LOAD_DEREF', opcode=84, arg=3, argval='b', argrepr='b', offset=16, start_offset=16, starts_line=False, line_number=4, label=None, positions=None), - Instruction(opname='LOAD_DEREF', opcode=84, arg=4, argval='c', argrepr='c', offset=18, start_offset=18, starts_line=False, line_number=4, label=None, positions=None), - Instruction(opname='LOAD_DEREF', opcode=84, arg=5, argval='d', argrepr='d', offset=20, start_offset=20, starts_line=False, line_number=4, label=None, positions=None), - Instruction(opname='LOAD_FAST_LOAD_FAST', opcode=88, arg=1, argval=('e', 'f'), argrepr='e, f', offset=22, start_offset=22, starts_line=False, line_number=4, label=None, positions=None), - Instruction(opname='CALL', opcode=53, arg=6, argval=6, argrepr='', offset=24, start_offset=24, starts_line=False, line_number=4, label=None, positions=None), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=32, start_offset=32, starts_line=False, line_number=4, label=None, positions=None), - Instruction(opname='RETURN_CONST', opcode=103, arg=0, argval=None, argrepr='None', offset=34, start_offset=34, starts_line=False, line_number=4, label=None, positions=None), + make_inst(opname='COPY_FREE_VARS', arg=4, argval=4, argrepr='', offset=0, start_offset=0, starts_line=True, line_number=None), + make_inst(opname='RESUME', arg=0, argval=0, argrepr='', offset=2, start_offset=2, starts_line=True, line_number=3), + make_inst(opname='LOAD_GLOBAL', arg=1, argval='print', argrepr='print + NULL', offset=4, start_offset=4, starts_line=True, line_number=4, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + make_inst(opname='LOAD_DEREF', arg=2, argval='a', argrepr='a', offset=14, start_offset=14, starts_line=False, line_number=4), + make_inst(opname='LOAD_DEREF', arg=3, argval='b', argrepr='b', offset=16, start_offset=16, starts_line=False, line_number=4), + make_inst(opname='LOAD_DEREF', arg=4, argval='c', argrepr='c', offset=18, start_offset=18, starts_line=False, line_number=4), + make_inst(opname='LOAD_DEREF', arg=5, argval='d', argrepr='d', offset=20, start_offset=20, starts_line=False, line_number=4), + make_inst(opname='LOAD_FAST_BORROW_LOAD_FAST_BORROW', arg=1, argval=('e', 'f'), argrepr='e, f', offset=22, start_offset=22, starts_line=False, line_number=4), + make_inst(opname='CALL', arg=6, argval=6, argrepr='', offset=24, start_offset=24, starts_line=False, line_number=4, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=32, start_offset=32, starts_line=False, line_number=4), + make_inst(opname='LOAD_CONST', arg=0, argval=None, argrepr='None', offset=34, start_offset=34, starts_line=False, line_number=4), + make_inst(opname='RETURN_VALUE', arg=None, argval=None, argrepr='', offset=36, start_offset=36, starts_line=False, line_number=4), ] expected_opinfo_jumpy = [ - Instruction(opname='RESUME', opcode=149, arg=0, argval=0, argrepr='', offset=0, start_offset=0, starts_line=True, line_number=1, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=91, arg=1, argval='range', argrepr='range + NULL', offset=2, start_offset=2, starts_line=True, line_number=3, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_CONST', opcode=83, arg=1, argval=10, argrepr='10', offset=12, start_offset=12, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=53, arg=1, argval=1, argrepr='', offset=14, start_offset=14, starts_line=False, line_number=3, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='GET_ITER', opcode=19, arg=None, argval=None, argrepr='', offset=22, start_offset=22, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='FOR_ITER', opcode=72, arg=30, argval=88, argrepr='to L4', offset=24, start_offset=24, starts_line=False, line_number=3, label=1, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='STORE_FAST', opcode=110, arg=0, argval='i', argrepr='i', offset=28, start_offset=28, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=91, arg=3, argval='print', argrepr='print + NULL', offset=30, start_offset=30, starts_line=True, line_number=4, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_FAST', opcode=85, arg=0, argval='i', argrepr='i', offset=40, start_offset=40, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=53, arg=1, argval=1, argrepr='', offset=42, start_offset=42, starts_line=False, line_number=4, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=50, start_offset=50, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST', opcode=85, arg=0, argval='i', argrepr='i', offset=52, start_offset=52, starts_line=True, line_number=5, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=83, arg=2, argval=4, argrepr='4', offset=54, start_offset=54, starts_line=False, line_number=5, label=None, positions=None, cache_info=None), - Instruction(opname='COMPARE_OP', opcode=58, arg=18, argval='<', argrepr='bool(<)', offset=56, start_offset=56, starts_line=False, line_number=5, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='POP_JUMP_IF_FALSE', opcode=97, arg=2, argval=68, argrepr='to L2', offset=60, start_offset=60, starts_line=False, line_number=5, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='JUMP_BACKWARD', opcode=77, arg=22, argval=24, argrepr='to L1', offset=64, start_offset=64, starts_line=True, line_number=6, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='LOAD_FAST', opcode=85, arg=0, argval='i', argrepr='i', offset=68, start_offset=68, starts_line=True, line_number=7, label=2, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=83, arg=3, argval=6, argrepr='6', offset=70, start_offset=70, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), - Instruction(opname='COMPARE_OP', opcode=58, arg=148, argval='>', argrepr='bool(>)', offset=72, start_offset=72, starts_line=False, line_number=7, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='POP_JUMP_IF_TRUE', opcode=100, arg=2, argval=84, argrepr='to L3', offset=76, start_offset=76, starts_line=False, line_number=7, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='JUMP_BACKWARD', opcode=77, arg=30, argval=24, argrepr='to L1', offset=80, start_offset=80, starts_line=False, line_number=7, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=84, start_offset=84, starts_line=True, line_number=8, label=3, positions=None, cache_info=None), - Instruction(opname='JUMP_FORWARD', opcode=79, arg=13, argval=114, argrepr='to L5', offset=86, start_offset=86, starts_line=False, line_number=8, label=None, positions=None, cache_info=None), - Instruction(opname='END_FOR', opcode=11, arg=None, argval=None, argrepr='', offset=88, start_offset=88, starts_line=True, line_number=3, label=4, positions=None, cache_info=None), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=90, start_offset=90, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=91, arg=3, argval='print', argrepr='print + NULL', offset=92, start_offset=92, starts_line=True, line_number=10, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_CONST', opcode=83, arg=4, argval='I can haz else clause?', argrepr="'I can haz else clause?'", offset=102, start_offset=102, starts_line=False, line_number=10, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=53, arg=1, argval=1, argrepr='', offset=104, start_offset=104, starts_line=False, line_number=10, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=112, start_offset=112, starts_line=False, line_number=10, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST_CHECK', opcode=87, arg=0, argval='i', argrepr='i', offset=114, start_offset=114, starts_line=True, line_number=11, label=5, positions=None, cache_info=None), - Instruction(opname='TO_BOOL', opcode=40, arg=None, argval=None, argrepr='', offset=116, start_offset=116, starts_line=False, line_number=11, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_JUMP_IF_FALSE', opcode=97, arg=40, argval=208, argrepr='to L9', offset=124, start_offset=124, starts_line=False, line_number=11, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='LOAD_GLOBAL', opcode=91, arg=3, argval='print', argrepr='print + NULL', offset=128, start_offset=128, starts_line=True, line_number=12, label=6, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_FAST', opcode=85, arg=0, argval='i', argrepr='i', offset=138, start_offset=138, starts_line=False, line_number=12, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=53, arg=1, argval=1, argrepr='', offset=140, start_offset=140, starts_line=False, line_number=12, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=148, start_offset=148, starts_line=False, line_number=12, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST', opcode=85, arg=0, argval='i', argrepr='i', offset=150, start_offset=150, starts_line=True, line_number=13, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=83, arg=5, argval=1, argrepr='1', offset=152, start_offset=152, starts_line=False, line_number=13, label=None, positions=None, cache_info=None), - Instruction(opname='BINARY_OP', opcode=45, arg=23, argval=23, argrepr='-=', offset=154, start_offset=154, starts_line=False, line_number=13, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='STORE_FAST', opcode=110, arg=0, argval='i', argrepr='i', offset=158, start_offset=158, starts_line=False, line_number=13, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST', opcode=85, arg=0, argval='i', argrepr='i', offset=160, start_offset=160, starts_line=True, line_number=14, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=83, arg=3, argval=6, argrepr='6', offset=162, start_offset=162, starts_line=False, line_number=14, label=None, positions=None, cache_info=None), - Instruction(opname='COMPARE_OP', opcode=58, arg=148, argval='>', argrepr='bool(>)', offset=164, start_offset=164, starts_line=False, line_number=14, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='POP_JUMP_IF_FALSE', opcode=97, arg=2, argval=176, argrepr='to L7', offset=168, start_offset=168, starts_line=False, line_number=14, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='JUMP_BACKWARD', opcode=77, arg=31, argval=114, argrepr='to L5', offset=172, start_offset=172, starts_line=True, line_number=15, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='LOAD_FAST', opcode=85, arg=0, argval='i', argrepr='i', offset=176, start_offset=176, starts_line=True, line_number=16, label=7, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=83, arg=2, argval=4, argrepr='4', offset=178, start_offset=178, starts_line=False, line_number=16, label=None, positions=None, cache_info=None), - Instruction(opname='COMPARE_OP', opcode=58, arg=18, argval='<', argrepr='bool(<)', offset=180, start_offset=180, starts_line=False, line_number=16, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='POP_JUMP_IF_FALSE', opcode=97, arg=1, argval=190, argrepr='to L8', offset=184, start_offset=184, starts_line=False, line_number=16, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='JUMP_FORWARD', opcode=79, arg=20, argval=230, argrepr='to L10', offset=188, start_offset=188, starts_line=True, line_number=17, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST', opcode=85, arg=0, argval='i', argrepr='i', offset=190, start_offset=190, starts_line=True, line_number=11, label=8, positions=None, cache_info=None), - Instruction(opname='TO_BOOL', opcode=40, arg=None, argval=None, argrepr='', offset=192, start_offset=192, starts_line=False, line_number=11, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_JUMP_IF_FALSE', opcode=97, arg=2, argval=208, argrepr='to L9', offset=200, start_offset=200, starts_line=False, line_number=11, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='JUMP_BACKWARD', opcode=77, arg=40, argval=128, argrepr='to L6', offset=204, start_offset=204, starts_line=False, line_number=11, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='LOAD_GLOBAL', opcode=91, arg=3, argval='print', argrepr='print + NULL', offset=208, start_offset=208, starts_line=True, line_number=19, label=9, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_CONST', opcode=83, arg=6, argval='Who let lolcatz into this test suite?', argrepr="'Who let lolcatz into this test suite?'", offset=218, start_offset=218, starts_line=False, line_number=19, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=53, arg=1, argval=1, argrepr='', offset=220, start_offset=220, starts_line=False, line_number=19, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=228, start_offset=228, starts_line=False, line_number=19, label=None, positions=None, cache_info=None), - Instruction(opname='NOP', opcode=30, arg=None, argval=None, argrepr='', offset=230, start_offset=230, starts_line=True, line_number=20, label=10, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=83, arg=5, argval=1, argrepr='1', offset=232, start_offset=232, starts_line=True, line_number=21, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=83, arg=7, argval=0, argrepr='0', offset=234, start_offset=234, starts_line=False, line_number=21, label=None, positions=None, cache_info=None), - Instruction(opname='BINARY_OP', opcode=45, arg=11, argval=11, argrepr='/', offset=236, start_offset=236, starts_line=False, line_number=21, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=240, start_offset=240, starts_line=False, line_number=21, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST', opcode=85, arg=0, argval='i', argrepr='i', offset=242, start_offset=242, starts_line=True, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='BEFORE_WITH', opcode=2, arg=None, argval=None, argrepr='', offset=244, start_offset=244, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='STORE_FAST', opcode=110, arg=1, argval='dodgy', argrepr='dodgy', offset=246, start_offset=246, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=91, arg=3, argval='print', argrepr='print + NULL', offset=248, start_offset=248, starts_line=True, line_number=26, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_CONST', opcode=83, arg=8, argval='Never reach this', argrepr="'Never reach this'", offset=258, start_offset=258, starts_line=False, line_number=26, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=53, arg=1, argval=1, argrepr='', offset=260, start_offset=260, starts_line=False, line_number=26, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=268, start_offset=268, starts_line=False, line_number=26, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=83, arg=0, argval=None, argrepr='None', offset=270, start_offset=270, starts_line=True, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=83, arg=0, argval=None, argrepr='None', offset=272, start_offset=272, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=83, arg=0, argval=None, argrepr='None', offset=274, start_offset=274, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=53, arg=2, argval=2, argrepr='', offset=276, start_offset=276, starts_line=False, line_number=25, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=284, start_offset=284, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=91, arg=3, argval='print', argrepr='print + NULL', offset=286, start_offset=286, starts_line=True, line_number=28, label=11, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_CONST', opcode=83, arg=10, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=296, start_offset=296, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=53, arg=1, argval=1, argrepr='', offset=298, start_offset=298, starts_line=False, line_number=28, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=306, start_offset=306, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), - Instruction(opname='RETURN_CONST', opcode=103, arg=0, argval=None, argrepr='None', offset=308, start_offset=308, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), - Instruction(opname='PUSH_EXC_INFO', opcode=33, arg=None, argval=None, argrepr='', offset=310, start_offset=310, starts_line=True, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='WITH_EXCEPT_START', opcode=44, arg=None, argval=None, argrepr='', offset=312, start_offset=312, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='TO_BOOL', opcode=40, arg=None, argval=None, argrepr='', offset=314, start_offset=314, starts_line=False, line_number=25, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_JUMP_IF_TRUE', opcode=100, arg=1, argval=328, argrepr='to L12', offset=322, start_offset=322, starts_line=False, line_number=25, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='RERAISE', opcode=102, arg=2, argval=2, argrepr='', offset=326, start_offset=326, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=328, start_offset=328, starts_line=False, line_number=25, label=12, positions=None, cache_info=None), - Instruction(opname='POP_EXCEPT', opcode=31, arg=None, argval=None, argrepr='', offset=330, start_offset=330, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=332, start_offset=332, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=334, start_offset=334, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='JUMP_BACKWARD_NO_INTERRUPT', opcode=78, arg=26, argval=286, argrepr='to L11', offset=336, start_offset=336, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='COPY', opcode=61, arg=3, argval=3, argrepr='', offset=338, start_offset=338, starts_line=True, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='POP_EXCEPT', opcode=31, arg=None, argval=None, argrepr='', offset=340, start_offset=340, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='RERAISE', opcode=102, arg=1, argval=1, argrepr='', offset=342, start_offset=342, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='PUSH_EXC_INFO', opcode=33, arg=None, argval=None, argrepr='', offset=344, start_offset=344, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=91, arg=4, argval='ZeroDivisionError', argrepr='ZeroDivisionError', offset=346, start_offset=346, starts_line=True, line_number=22, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='CHECK_EXC_MATCH', opcode=7, arg=None, argval=None, argrepr='', offset=356, start_offset=356, starts_line=False, line_number=22, label=None, positions=None, cache_info=None), - Instruction(opname='POP_JUMP_IF_FALSE', opcode=97, arg=14, argval=390, argrepr='to L13', offset=358, start_offset=358, starts_line=False, line_number=22, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=362, start_offset=362, starts_line=False, line_number=22, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=91, arg=3, argval='print', argrepr='print + NULL', offset=364, start_offset=364, starts_line=True, line_number=23, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_CONST', opcode=83, arg=9, argval='Here we go, here we go, here we go...', argrepr="'Here we go, here we go, here we go...'", offset=374, start_offset=374, starts_line=False, line_number=23, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=53, arg=1, argval=1, argrepr='', offset=376, start_offset=376, starts_line=False, line_number=23, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=384, start_offset=384, starts_line=False, line_number=23, label=None, positions=None, cache_info=None), - Instruction(opname='POP_EXCEPT', opcode=31, arg=None, argval=None, argrepr='', offset=386, start_offset=386, starts_line=False, line_number=23, label=None, positions=None, cache_info=None), - Instruction(opname='JUMP_BACKWARD_NO_INTERRUPT', opcode=78, arg=52, argval=286, argrepr='to L11', offset=388, start_offset=388, starts_line=False, line_number=23, label=None, positions=None, cache_info=None), - Instruction(opname='RERAISE', opcode=102, arg=0, argval=0, argrepr='', offset=390, start_offset=390, starts_line=True, line_number=22, label=13, positions=None, cache_info=None), - Instruction(opname='COPY', opcode=61, arg=3, argval=3, argrepr='', offset=392, start_offset=392, starts_line=True, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='POP_EXCEPT', opcode=31, arg=None, argval=None, argrepr='', offset=394, start_offset=394, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='RERAISE', opcode=102, arg=1, argval=1, argrepr='', offset=396, start_offset=396, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='PUSH_EXC_INFO', opcode=33, arg=None, argval=None, argrepr='', offset=398, start_offset=398, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=91, arg=3, argval='print', argrepr='print + NULL', offset=400, start_offset=400, starts_line=True, line_number=28, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_CONST', opcode=83, arg=10, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=410, start_offset=410, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=53, arg=1, argval=1, argrepr='', offset=412, start_offset=412, starts_line=False, line_number=28, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=32, arg=None, argval=None, argrepr='', offset=420, start_offset=420, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), - Instruction(opname='RERAISE', opcode=102, arg=0, argval=0, argrepr='', offset=422, start_offset=422, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), - Instruction(opname='COPY', opcode=61, arg=3, argval=3, argrepr='', offset=424, start_offset=424, starts_line=True, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='POP_EXCEPT', opcode=31, arg=None, argval=None, argrepr='', offset=426, start_offset=426, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='RERAISE', opcode=102, arg=1, argval=1, argrepr='', offset=428, start_offset=428, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), + make_inst(opname='RESUME', arg=0, argval=0, argrepr='', offset=0, start_offset=0, starts_line=True, line_number=1), + make_inst(opname='LOAD_GLOBAL', arg=1, argval='range', argrepr='range + NULL', offset=2, start_offset=2, starts_line=True, line_number=3, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + make_inst(opname='LOAD_SMALL_INT', arg=10, argval=10, argrepr='', offset=12, start_offset=12, starts_line=False, line_number=3), + make_inst(opname='CALL', arg=1, argval=1, argrepr='', offset=14, start_offset=14, starts_line=False, line_number=3, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + make_inst(opname='GET_ITER', arg=None, argval=None, argrepr='', offset=22, start_offset=22, starts_line=False, line_number=3), + make_inst(opname='FOR_ITER', arg=32, argval=92, argrepr='to L4', offset=24, start_offset=24, starts_line=False, line_number=3, label=1, cache_info=[('counter', 1, b'\x00\x00')]), + make_inst(opname='STORE_FAST', arg=0, argval='i', argrepr='i', offset=28, start_offset=28, starts_line=False, line_number=3), + make_inst(opname='LOAD_GLOBAL', arg=3, argval='print', argrepr='print + NULL', offset=30, start_offset=30, starts_line=True, line_number=4, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + make_inst(opname='LOAD_FAST_BORROW', arg=0, argval='i', argrepr='i', offset=40, start_offset=40, starts_line=False, line_number=4), + make_inst(opname='CALL', arg=1, argval=1, argrepr='', offset=42, start_offset=42, starts_line=False, line_number=4, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=50, start_offset=50, starts_line=False, line_number=4), + make_inst(opname='LOAD_FAST_BORROW', arg=0, argval='i', argrepr='i', offset=52, start_offset=52, starts_line=True, line_number=5), + make_inst(opname='LOAD_SMALL_INT', arg=4, argval=4, argrepr='', offset=54, start_offset=54, starts_line=False, line_number=5), + make_inst(opname='COMPARE_OP', arg=18, argval='<', argrepr='bool(<)', offset=56, start_offset=56, starts_line=False, line_number=5, cache_info=[('counter', 1, b'\x00\x00')]), + make_inst(opname='POP_JUMP_IF_FALSE', arg=3, argval=70, argrepr='to L2', offset=60, start_offset=60, starts_line=False, line_number=5, cache_info=[('counter', 1, b'\x00\x00')]), + make_inst(opname='NOT_TAKEN', arg=None, argval=None, argrepr='', offset=64, start_offset=64, starts_line=False, line_number=5), + make_inst(opname='JUMP_BACKWARD', arg=23, argval=24, argrepr='to L1', offset=66, start_offset=66, starts_line=True, line_number=6, cache_info=[('counter', 1, b'\x00\x00')]), + make_inst(opname='LOAD_FAST_BORROW', arg=0, argval='i', argrepr='i', offset=70, start_offset=70, starts_line=True, line_number=7, label=2), + make_inst(opname='LOAD_SMALL_INT', arg=6, argval=6, argrepr='', offset=72, start_offset=72, starts_line=False, line_number=7), + make_inst(opname='COMPARE_OP', arg=148, argval='>', argrepr='bool(>)', offset=74, start_offset=74, starts_line=False, line_number=7, cache_info=[('counter', 1, b'\x00\x00')]), + make_inst(opname='POP_JUMP_IF_TRUE', arg=3, argval=88, argrepr='to L3', offset=78, start_offset=78, starts_line=False, line_number=7, cache_info=[('counter', 1, b'\x00\x00')]), + make_inst(opname='NOT_TAKEN', arg=None, argval=None, argrepr='', offset=82, start_offset=82, starts_line=False, line_number=7), + make_inst(opname='JUMP_BACKWARD', arg=32, argval=24, argrepr='to L1', offset=84, start_offset=84, starts_line=False, line_number=7, cache_info=[('counter', 1, b'\x00\x00')]), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=88, start_offset=88, starts_line=True, line_number=8, label=3), + make_inst(opname='JUMP_FORWARD', arg=13, argval=118, argrepr='to L5', offset=90, start_offset=90, starts_line=False, line_number=8), + make_inst(opname='END_FOR', arg=None, argval=None, argrepr='', offset=92, start_offset=92, starts_line=True, line_number=3, label=4), + make_inst(opname='POP_ITER', arg=None, argval=None, argrepr='', offset=94, start_offset=94, starts_line=False, line_number=3), + make_inst(opname='LOAD_GLOBAL', arg=3, argval='print', argrepr='print + NULL', offset=96, start_offset=96, starts_line=True, line_number=10, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + make_inst(opname='LOAD_CONST', arg=1, argval='I can haz else clause?', argrepr="'I can haz else clause?'", offset=106, start_offset=106, starts_line=False, line_number=10), + make_inst(opname='CALL', arg=1, argval=1, argrepr='', offset=108, start_offset=108, starts_line=False, line_number=10, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=116, start_offset=116, starts_line=False, line_number=10), + make_inst(opname='LOAD_FAST_CHECK', arg=0, argval='i', argrepr='i', offset=118, start_offset=118, starts_line=True, line_number=11, label=5), + make_inst(opname='TO_BOOL', arg=None, argval=None, argrepr='', offset=120, start_offset=120, starts_line=False, line_number=11, cache_info=[('counter', 1, b'\x00\x00'), ('version', 2, b'\x00\x00\x00\x00')]), + make_inst(opname='POP_JUMP_IF_FALSE', arg=40, argval=212, argrepr='to L8', offset=128, start_offset=128, starts_line=False, line_number=11, cache_info=[('counter', 1, b'\x00\x00')]), + make_inst(opname='NOT_TAKEN', arg=None, argval=None, argrepr='', offset=132, start_offset=132, starts_line=False, line_number=11), + make_inst(opname='LOAD_GLOBAL', arg=3, argval='print', argrepr='print + NULL', offset=134, start_offset=134, starts_line=True, line_number=12, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + make_inst(opname='LOAD_FAST_BORROW', arg=0, argval='i', argrepr='i', offset=144, start_offset=144, starts_line=False, line_number=12), + make_inst(opname='CALL', arg=1, argval=1, argrepr='', offset=146, start_offset=146, starts_line=False, line_number=12, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=154, start_offset=154, starts_line=False, line_number=12), + make_inst(opname='LOAD_FAST_BORROW', arg=0, argval='i', argrepr='i', offset=156, start_offset=156, starts_line=True, line_number=13), + make_inst(opname='LOAD_SMALL_INT', arg=1, argval=1, argrepr='', offset=158, start_offset=158, starts_line=False, line_number=13), + make_inst(opname='BINARY_OP', arg=23, argval=23, argrepr='-=', offset=160, start_offset=160, starts_line=False, line_number=13, cache_info=[('counter', 1, b'\x00\x00'), ('descr', 4, b'\x00\x00\x00\x00\x00\x00\x00\x00')]), + make_inst(opname='STORE_FAST', arg=0, argval='i', argrepr='i', offset=172, start_offset=172, starts_line=False, line_number=13), + make_inst(opname='LOAD_FAST_BORROW', arg=0, argval='i', argrepr='i', offset=174, start_offset=174, starts_line=True, line_number=14), + make_inst(opname='LOAD_SMALL_INT', arg=6, argval=6, argrepr='', offset=176, start_offset=176, starts_line=False, line_number=14), + make_inst(opname='COMPARE_OP', arg=148, argval='>', argrepr='bool(>)', offset=178, start_offset=178, starts_line=False, line_number=14, cache_info=[('counter', 1, b'\x00\x00')]), + make_inst(opname='POP_JUMP_IF_FALSE', arg=3, argval=192, argrepr='to L6', offset=182, start_offset=182, starts_line=False, line_number=14, cache_info=[('counter', 1, b'\x00\x00')]), + make_inst(opname='NOT_TAKEN', arg=None, argval=None, argrepr='', offset=186, start_offset=186, starts_line=False, line_number=14), + make_inst(opname='JUMP_BACKWARD', arg=37, argval=118, argrepr='to L5', offset=188, start_offset=188, starts_line=True, line_number=15, cache_info=[('counter', 1, b'\x00\x00')]), + make_inst(opname='LOAD_FAST_BORROW', arg=0, argval='i', argrepr='i', offset=192, start_offset=192, starts_line=True, line_number=16, label=6), + make_inst(opname='LOAD_SMALL_INT', arg=4, argval=4, argrepr='', offset=194, start_offset=194, starts_line=False, line_number=16), + make_inst(opname='COMPARE_OP', arg=18, argval='<', argrepr='bool(<)', offset=196, start_offset=196, starts_line=False, line_number=16, cache_info=[('counter', 1, b'\x00\x00')]), + make_inst(opname='POP_JUMP_IF_TRUE', arg=3, argval=210, argrepr='to L7', offset=200, start_offset=200, starts_line=False, line_number=16, cache_info=[('counter', 1, b'\x00\x00')]), + make_inst(opname='NOT_TAKEN', arg=None, argval=None, argrepr='', offset=204, start_offset=204, starts_line=False, line_number=16), + make_inst(opname='JUMP_BACKWARD', arg=46, argval=118, argrepr='to L5', offset=206, start_offset=206, starts_line=False, line_number=16, cache_info=[('counter', 1, b'\x00\x00')]), + make_inst(opname='JUMP_FORWARD', arg=11, argval=234, argrepr='to L9', offset=210, start_offset=210, starts_line=True, line_number=17, label=7), + make_inst(opname='LOAD_GLOBAL', arg=3, argval='print', argrepr='print + NULL', offset=212, start_offset=212, starts_line=True, line_number=19, label=8, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + make_inst(opname='LOAD_CONST', arg=2, argval='Who let lolcatz into this test suite?', argrepr="'Who let lolcatz into this test suite?'", offset=222, start_offset=222, starts_line=False, line_number=19), + make_inst(opname='CALL', arg=1, argval=1, argrepr='', offset=224, start_offset=224, starts_line=False, line_number=19, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=232, start_offset=232, starts_line=False, line_number=19), + make_inst(opname='NOP', arg=None, argval=None, argrepr='', offset=234, start_offset=234, starts_line=True, line_number=20, label=9), + make_inst(opname='LOAD_SMALL_INT', arg=1, argval=1, argrepr='', offset=236, start_offset=236, starts_line=True, line_number=21), + make_inst(opname='LOAD_SMALL_INT', arg=0, argval=0, argrepr='', offset=238, start_offset=238, starts_line=False, line_number=21), + make_inst(opname='BINARY_OP', arg=11, argval=11, argrepr='/', offset=240, start_offset=240, starts_line=False, line_number=21, cache_info=[('counter', 1, b'\x00\x00'), ('descr', 4, b'\x00\x00\x00\x00\x00\x00\x00\x00')]), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=252, start_offset=252, starts_line=False, line_number=21), + make_inst(opname='LOAD_FAST_BORROW', arg=0, argval='i', argrepr='i', offset=254, start_offset=254, starts_line=True, line_number=25), + make_inst(opname='COPY', arg=1, argval=1, argrepr='', offset=256, start_offset=256, starts_line=False, line_number=25), + make_inst(opname='LOAD_SPECIAL', arg=1, argval=1, argrepr='__exit__', offset=258, start_offset=258, starts_line=False, line_number=25), + make_inst(opname='SWAP', arg=2, argval=2, argrepr='', offset=260, start_offset=260, starts_line=False, line_number=25), + make_inst(opname='SWAP', arg=3, argval=3, argrepr='', offset=262, start_offset=262, starts_line=False, line_number=25), + make_inst(opname='LOAD_SPECIAL', arg=0, argval=0, argrepr='__enter__', offset=264, start_offset=264, starts_line=False, line_number=25), + make_inst(opname='CALL', arg=0, argval=0, argrepr='', offset=266, start_offset=266, starts_line=False, line_number=25, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + make_inst(opname='STORE_FAST', arg=1, argval='dodgy', argrepr='dodgy', offset=274, start_offset=274, starts_line=False, line_number=25), + make_inst(opname='LOAD_GLOBAL', arg=3, argval='print', argrepr='print + NULL', offset=276, start_offset=276, starts_line=True, line_number=26, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + make_inst(opname='LOAD_CONST', arg=3, argval='Never reach this', argrepr="'Never reach this'", offset=286, start_offset=286, starts_line=False, line_number=26), + make_inst(opname='CALL', arg=1, argval=1, argrepr='', offset=288, start_offset=288, starts_line=False, line_number=26, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=296, start_offset=296, starts_line=False, line_number=26), + make_inst(opname='LOAD_CONST', arg=4, argval=None, argrepr='None', offset=298, start_offset=298, starts_line=True, line_number=25), + make_inst(opname='LOAD_CONST', arg=4, argval=None, argrepr='None', offset=300, start_offset=300, starts_line=False, line_number=25), + make_inst(opname='LOAD_CONST', arg=4, argval=None, argrepr='None', offset=302, start_offset=302, starts_line=False, line_number=25), + make_inst(opname='CALL', arg=3, argval=3, argrepr='', offset=304, start_offset=304, starts_line=False, line_number=25, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=312, start_offset=312, starts_line=False, line_number=25), + make_inst(opname='LOAD_GLOBAL', arg=3, argval='print', argrepr='print + NULL', offset=314, start_offset=314, starts_line=True, line_number=28, label=10, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + make_inst(opname='LOAD_CONST', arg=6, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=324, start_offset=324, starts_line=False, line_number=28), + make_inst(opname='CALL', arg=1, argval=1, argrepr='', offset=326, start_offset=326, starts_line=False, line_number=28, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=334, start_offset=334, starts_line=False, line_number=28), + make_inst(opname='LOAD_CONST', arg=4, argval=None, argrepr='None', offset=336, start_offset=336, starts_line=False, line_number=28), + make_inst(opname='RETURN_VALUE', arg=None, argval=None, argrepr='', offset=338, start_offset=338, starts_line=False, line_number=28), + make_inst(opname='PUSH_EXC_INFO', arg=None, argval=None, argrepr='', offset=340, start_offset=340, starts_line=True, line_number=25), + make_inst(opname='WITH_EXCEPT_START', arg=None, argval=None, argrepr='', offset=342, start_offset=342, starts_line=False, line_number=25), + make_inst(opname='TO_BOOL', arg=None, argval=None, argrepr='', offset=344, start_offset=344, starts_line=False, line_number=25, cache_info=[('counter', 1, b'\x00\x00'), ('version', 2, b'\x00\x00\x00\x00')]), + make_inst(opname='POP_JUMP_IF_TRUE', arg=2, argval=360, argrepr='to L11', offset=352, start_offset=352, starts_line=False, line_number=25, cache_info=[('counter', 1, b'\x00\x00')]), + make_inst(opname='NOT_TAKEN', arg=None, argval=None, argrepr='', offset=356, start_offset=356, starts_line=False, line_number=25), + make_inst(opname='RERAISE', arg=2, argval=2, argrepr='', offset=358, start_offset=358, starts_line=False, line_number=25), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=360, start_offset=360, starts_line=False, line_number=25, label=11), + make_inst(opname='POP_EXCEPT', arg=None, argval=None, argrepr='', offset=362, start_offset=362, starts_line=False, line_number=25), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=364, start_offset=364, starts_line=False, line_number=25), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=366, start_offset=366, starts_line=False, line_number=25), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=368, start_offset=368, starts_line=False, line_number=25), + make_inst(opname='JUMP_BACKWARD_NO_INTERRUPT', arg=29, argval=314, argrepr='to L10', offset=370, start_offset=370, starts_line=False, line_number=25), + make_inst(opname='COPY', arg=3, argval=3, argrepr='', offset=372, start_offset=372, starts_line=True, line_number=None), + make_inst(opname='POP_EXCEPT', arg=None, argval=None, argrepr='', offset=374, start_offset=374, starts_line=False, line_number=None), + make_inst(opname='RERAISE', arg=1, argval=1, argrepr='', offset=376, start_offset=376, starts_line=False, line_number=None), + make_inst(opname='PUSH_EXC_INFO', arg=None, argval=None, argrepr='', offset=378, start_offset=378, starts_line=False, line_number=None), + make_inst(opname='LOAD_GLOBAL', arg=4, argval='ZeroDivisionError', argrepr='ZeroDivisionError', offset=380, start_offset=380, starts_line=True, line_number=22, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + make_inst(opname='CHECK_EXC_MATCH', arg=None, argval=None, argrepr='', offset=390, start_offset=390, starts_line=False, line_number=22), + make_inst(opname='POP_JUMP_IF_FALSE', arg=15, argval=426, argrepr='to L12', offset=392, start_offset=392, starts_line=False, line_number=22, cache_info=[('counter', 1, b'\x00\x00')]), + make_inst(opname='NOT_TAKEN', arg=None, argval=None, argrepr='', offset=396, start_offset=396, starts_line=False, line_number=22), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=398, start_offset=398, starts_line=False, line_number=22), + make_inst(opname='LOAD_GLOBAL', arg=3, argval='print', argrepr='print + NULL', offset=400, start_offset=400, starts_line=True, line_number=23, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + make_inst(opname='LOAD_CONST', arg=5, argval='Here we go, here we go, here we go...', argrepr="'Here we go, here we go, here we go...'", offset=410, start_offset=410, starts_line=False, line_number=23), + make_inst(opname='CALL', arg=1, argval=1, argrepr='', offset=412, start_offset=412, starts_line=False, line_number=23, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=420, start_offset=420, starts_line=False, line_number=23), + make_inst(opname='POP_EXCEPT', arg=None, argval=None, argrepr='', offset=422, start_offset=422, starts_line=False, line_number=23), + make_inst(opname='JUMP_BACKWARD_NO_INTERRUPT', arg=56, argval=314, argrepr='to L10', offset=424, start_offset=424, starts_line=False, line_number=23), + make_inst(opname='RERAISE', arg=0, argval=0, argrepr='', offset=426, start_offset=426, starts_line=True, line_number=22, label=12), + make_inst(opname='COPY', arg=3, argval=3, argrepr='', offset=428, start_offset=428, starts_line=True, line_number=None), + make_inst(opname='POP_EXCEPT', arg=None, argval=None, argrepr='', offset=430, start_offset=430, starts_line=False, line_number=None), + make_inst(opname='RERAISE', arg=1, argval=1, argrepr='', offset=432, start_offset=432, starts_line=False, line_number=None), + make_inst(opname='PUSH_EXC_INFO', arg=None, argval=None, argrepr='', offset=434, start_offset=434, starts_line=False, line_number=None), + make_inst(opname='LOAD_GLOBAL', arg=3, argval='print', argrepr='print + NULL', offset=436, start_offset=436, starts_line=True, line_number=28, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + make_inst(opname='LOAD_CONST', arg=6, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=446, start_offset=446, starts_line=False, line_number=28), + make_inst(opname='CALL', arg=1, argval=1, argrepr='', offset=448, start_offset=448, starts_line=False, line_number=28, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + make_inst(opname='POP_TOP', arg=None, argval=None, argrepr='', offset=456, start_offset=456, starts_line=False, line_number=28), + make_inst(opname='RERAISE', arg=0, argval=0, argrepr='', offset=458, start_offset=458, starts_line=False, line_number=28), + make_inst(opname='COPY', arg=3, argval=3, argrepr='', offset=460, start_offset=460, starts_line=True, line_number=None), + make_inst(opname='POP_EXCEPT', arg=None, argval=None, argrepr='', offset=462, start_offset=462, starts_line=False, line_number=None), + make_inst(opname='RERAISE', arg=1, argval=1, argrepr='', offset=464, start_offset=464, starts_line=False, line_number=None), ] # One last piece of inspect fodder to check the default line number handling def simple(): pass expected_opinfo_simple = [ - Instruction(opname='RESUME', opcode=149, arg=0, argval=0, argrepr='', offset=0, start_offset=0, starts_line=True, line_number=simple.__code__.co_firstlineno, label=None, positions=None), - Instruction(opname='RETURN_CONST', opcode=103, arg=0, argval=None, argrepr='None', offset=2, start_offset=2, starts_line=False, line_number=simple.__code__.co_firstlineno, label=None), + make_inst(opname='RESUME', arg=0, argval=0, argrepr='', offset=0, start_offset=0, starts_line=True, line_number=simple.__code__.co_firstlineno), + make_inst(opname='LOAD_CONST', arg=0, argval=None, argrepr='None', offset=2, start_offset=2, starts_line=False, line_number=simple.__code__.co_firstlineno), + make_inst(opname='RETURN_VALUE', arg=None, argval=None, argrepr='', offset=4, start_offset=4, starts_line=False, line_number=simple.__code__.co_firstlineno), ] @@ -1823,36 +2003,34 @@ def test_instruction_str(self): for instr in instrs: str(instr) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_default_first_line(self): actual = dis.get_instructions(simple) self.assertInstructionsEqual(list(actual), expected_opinfo_simple) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_first_line_set_to_None(self): actual = dis.get_instructions(simple, first_line=None) self.assertInstructionsEqual(list(actual), expected_opinfo_simple) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_outer(self): actual = dis.get_instructions(outer, first_line=expected_outer_line) self.assertInstructionsEqual(list(actual), expected_opinfo_outer) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_nested(self): with captured_stdout(): f = outer() actual = dis.get_instructions(f, first_line=expected_f_line) self.assertInstructionsEqual(list(actual), expected_opinfo_f) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_doubly_nested(self): with captured_stdout(): inner = outer()() actual = dis.get_instructions(inner, first_line=expected_inner_line) self.assertInstructionsEqual(list(actual), expected_opinfo_inner) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_jumpy(self): actual = dis.get_instructions(jumpy, first_line=expected_jumpy_line) self.assertInstructionsEqual(list(actual), expected_opinfo_jumpy) @@ -1873,6 +2051,7 @@ def test_co_positions(self): (2, 2, 8, 9), (1, 3, 0, 1), (1, 3, 0, 1), + (1, 3, 0, 1), (1, 3, 0, 1) ] self.assertEqual(positions, expected) @@ -1935,12 +2114,12 @@ def roots(a, b, c): self.assertEqual(co_positions, dis_positions) def test_oparg_alias(self): - instruction = Instruction(opname="NOP", opcode=dis.opmap["NOP"], arg=None, argval=None, + instruction = make_inst(opname="NOP", arg=None, argval=None, argrepr='', offset=10, start_offset=10, starts_line=True, line_number=1, label=None, positions=None) self.assertEqual(instruction.arg, instruction.oparg) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_show_caches_with_label(self): def f(x, y, z): if x: @@ -1953,54 +2132,72 @@ def f(x, y, z): dis.dis(f.__code__, file=output, show_caches=True) self.assertIn("L1:", output.getvalue()) + def test_is_op_format(self): + output = io.StringIO() + dis.dis("a is b", file=output, show_caches=True) + self.assertIn("IS_OP 0 (is)", output.getvalue()) + + output = io.StringIO() + dis.dis("a is not b", file=output, show_caches=True) + self.assertIn("IS_OP 1 (is not)", output.getvalue()) + + def test_contains_op_format(self): + output = io.StringIO() + dis.dis("a in b", file=output, show_caches=True) + self.assertIn("CONTAINS_OP 0 (in)", output.getvalue()) + + output = io.StringIO() + dis.dis("a not in b", file=output, show_caches=True) + self.assertIn("CONTAINS_OP 1 (not in)", output.getvalue()) + def test_baseopname_and_baseopcode(self): # Standard instructions - for name, code in dis.opmap.items(): - instruction = Instruction(opname=name, opcode=code, arg=None, argval=None, argrepr='', offset=0, - start_offset=0, starts_line=True, line_number=1, label=None, positions=None) + for name in dis.opmap: + instruction = make_inst(opname=name, arg=None, argval=None, argrepr='', offset=0, + start_offset=0, starts_line=True, line_number=1, label=None, positions=None) baseopname = instruction.baseopname baseopcode = instruction.baseopcode self.assertIsNotNone(baseopname) self.assertIsNotNone(baseopcode) self.assertEqual(name, baseopname) - self.assertEqual(code, baseopcode) + self.assertEqual(instruction.opcode, baseopcode) # Specialized instructions for name in opcode._specialized_opmap: - instruction = Instruction(opname=name, opcode=dis._all_opmap[name], arg=None, argval=None, argrepr='', - offset=0, start_offset=0, starts_line=True, line_number=1, label=None, positions=None) + instruction = make_inst(opname=name, arg=None, argval=None, argrepr='', + offset=0, start_offset=0, starts_line=True, line_number=1, label=None, positions=None) baseopname = instruction.baseopname baseopcode = instruction.baseopcode self.assertIn(name, opcode._specializations[baseopname]) self.assertEqual(opcode.opmap[baseopname], baseopcode) - @unittest.expectedFailure # TODO: RUSTPYTHON - JUMP_BACKWARD/JUMP_FORWARD are placeholders + @unittest.expectedFailure # TODO: RUSTPYTHON; - JUMP_BACKWARD/JUMP_FORWARD are placeholders def test_jump_target(self): # Non-jump instructions should return None - instruction = Instruction(opname="NOP", opcode=dis.opmap["NOP"], arg=None, argval=None, + instruction = make_inst(opname="NOP", arg=None, argval=None, argrepr='', offset=10, start_offset=10, starts_line=True, line_number=1, label=None, positions=None) self.assertIsNone(instruction.jump_target) delta = 100 - instruction = Instruction(opname="JUMP_FORWARD", opcode=dis.opmap["JUMP_FORWARD"], arg=delta, argval=delta, + instruction = make_inst(opname="JUMP_FORWARD", arg=delta, argval=delta, argrepr='', offset=10, start_offset=10, starts_line=True, line_number=1, label=None, positions=None) self.assertEqual(10 + 2 + 100*2, instruction.jump_target) # Test negative deltas - instruction = Instruction(opname="JUMP_BACKWARD", opcode=dis.opmap["JUMP_BACKWARD"], arg=delta, argval=delta, + instruction = make_inst(opname="JUMP_BACKWARD", arg=delta, argval=delta, argrepr='', offset=200, start_offset=200, starts_line=True, line_number=1, label=None, positions=None) self.assertEqual(200 + 2 - 100*2 + 2*1, instruction.jump_target) # Make sure cache entries are handled - instruction = Instruction(opname="SEND", opcode=dis.opmap["SEND"], arg=delta, argval=delta, + instruction = make_inst(opname="SEND", arg=delta, argval=delta, argrepr='', offset=10, start_offset=10, starts_line=True, line_number=1, label=None, positions=None) self.assertEqual(10 + 2 + 1*2 + 100*2, instruction.jump_target) - @unittest.expectedFailure # TODO: RUSTPYTHON - JUMP_BACKWARD is a placeholder + @unittest.expectedFailure # TODO: RUSTPYTHON; - JUMP_BACKWARD is a placeholder def test_argval_argrepr(self): def f(opcode, oparg, offset, *init_args): arg_resolver = dis.ArgResolver(*init_args) @@ -2021,7 +2218,7 @@ def f(opcode, oparg, offset, *init_args): self.assertEqual(f(opcode.opmap["BINARY_OP"], 3, *args), (3, '<<')) self.assertEqual(f(opcode.opmap["CALL_INTRINSIC_1"], 2, *args), (2, 'INTRINSIC_IMPORT_STAR')) - @unittest.expectedFailure # TODO: RUSTPYTHON - JUMP_BACKWARD is a placeholder + @unittest.expectedFailure # TODO: RUSTPYTHON; - JUMP_BACKWARD is a placeholder def test_custom_arg_resolver(self): class MyArgResolver(dis.ArgResolver): def offset_from_jump_arg(self, op, arg, offset): @@ -2141,12 +2338,11 @@ def test_iteration(self): via_generator = list(dis.get_instructions(obj)) self.assertInstructionsEqual(via_object, via_generator) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_explicit_first_line(self): actual = dis.Bytecode(outer, first_line=expected_outer_line) self.assertInstructionsEqual(list(actual), expected_opinfo_outer) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_source_line_in_disassembly(self): # Use the line in the source code actual = dis.Bytecode(simple).dis() @@ -2158,14 +2354,14 @@ def test_source_line_in_disassembly(self): actual = actual.strip().partition(" ")[0] # extract the line no self.assertEqual(actual, "350") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_info(self): self.maxDiff = 1000 for x, expected in CodeInfoTests.test_pairs: b = dis.Bytecode(x) self.assertRegex(b.info(), expected) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_disassembled(self): actual = dis.Bytecode(_f).dis() self.do_disassembly_compare(actual, dis_f) @@ -2177,7 +2373,7 @@ def test_from_traceback(self): self.assertEqual(b.current_offset, tb.tb_lasti) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_from_traceback_dis(self): self.maxDiff = None tb = get_tb() @@ -2193,22 +2389,20 @@ def test_bytecode_co_positions(self): class TestBytecodeTestCase(BytecodeTestCase): def test_assert_not_in_with_op_not_in_bytecode(self): code = compile("a = 1", "", "exec") - self.assertInBytecode(code, "LOAD_CONST", 1) + self.assertInBytecode(code, "LOAD_SMALL_INT", 1) self.assertNotInBytecode(code, "LOAD_NAME") self.assertNotInBytecode(code, "LOAD_NAME", "a") - @unittest.expectedFailure # TODO: RUSTPYTHON def test_assert_not_in_with_arg_not_in_bytecode(self): code = compile("a = 1", "", "exec") - self.assertInBytecode(code, "LOAD_CONST") - self.assertInBytecode(code, "LOAD_CONST", 1) + self.assertInBytecode(code, "LOAD_SMALL_INT") + self.assertInBytecode(code, "LOAD_SMALL_INT", 1) self.assertNotInBytecode(code, "LOAD_CONST", 2) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_assert_not_in_with_arg_in_bytecode(self): code = compile("a = 1", "", "exec") with self.assertRaises(AssertionError): - self.assertNotInBytecode(code, "LOAD_CONST", 1) + self.assertNotInBytecode(code, "LOAD_SMALL_INT", 1) class TestFinderMethods(unittest.TestCase): def test__find_imports(self): @@ -2228,7 +2422,7 @@ def test__find_imports(self): self.assertEqual(len(res), 1) self.assertEqual(res[0], expected) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test__find_store_names(self): cases = [ ("x+y", ()), @@ -2243,7 +2437,7 @@ def test__find_store_names(self): res = tuple(dis._find_store_names(code)) self.assertEqual(res, expected) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_findlabels(self): labels = dis.findlabels(jumpy.__code__.co_code) jumps = [ @@ -2275,7 +2469,7 @@ def setUp(self) -> None: pass return super().setUp() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def get_disassembly(self, tb): output = io.StringIO() with contextlib.redirect_stdout(output): @@ -2286,7 +2480,7 @@ def test_distb_empty(self): with self.assertRaises(RuntimeError): dis.distb() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_distb_last_traceback(self): self.maxDiff = None # We need to have an existing last traceback in `sys`: @@ -2295,7 +2489,7 @@ def test_distb_last_traceback(self): self.do_disassembly_compare(self.get_disassembly(None), dis_traceback) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_distb_explicit_arg(self): self.maxDiff = None tb = get_tb() @@ -2333,8 +2527,8 @@ def _unroll_caches_as_Instructions(instrs, show_caches=False): else: argrepr = "" - yield Instruction("CACHE", CACHE, 0, None, argrepr, offset, offset, - False, None, None, instr.positions) + yield make_inst("CACHE", 0, None, argrepr, offset, offset, + False, None, None, instr.positions) class TestDisCLI(unittest.TestCase): @@ -2374,6 +2568,8 @@ def test_invocation(self): base_flags = [ ('-C', '--show-caches'), ('-O', '--show-offsets'), + ('-P', '--show-positions'), + ('-S', '--specialized'), ] self.set_source(''' @@ -2393,7 +2589,7 @@ def f(): with contextlib.redirect_stderr(io.StringIO()): _ = self.invoke_dis('--unknown') - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_show_cache(self): # test 'python -m dis -C/--show-caches' source = 'print()' @@ -2407,23 +2603,48 @@ def test_show_cache(self): CACHE 0 (func_version: 0) CACHE 0 POP_TOP - RETURN_CONST 0 (None) + LOAD_CONST 0 (None) + RETURN_VALUE ''' for flag in ['-C', '--show-caches']: self.check_output(source, expect, flag) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_show_offsets(self): # test 'python -m dis -O/--show-offsets' source = 'pass' expect = ''' 0 0 RESUME 0 - 1 2 RETURN_CONST 0 (None) + 1 2 LOAD_CONST 0 (None) + 4 RETURN_VALUE ''' for flag in ['-O', '--show-offsets']: self.check_output(source, expect, flag) + def test_show_positions(self): + # test 'python -m dis -P/--show-positions' + source = 'pass' + expect = ''' + 0:0-1:0 RESUME 0 + + 1:0-1:4 LOAD_CONST 0 (None) + 1:0-1:4 RETURN_VALUE + ''' + for flag in ['-P', '--show-positions']: + self.check_output(source, expect, flag) + + def test_specialized_code(self): + # test 'python -m dis -S/--specialized' + source = 'pass' + expect = ''' + 0 RESUME 0 + + 1 LOAD_CONST 0 (None) + RETURN_VALUE + ''' + for flag in ['-S', '--specialized']: + self.check_output(source, expect, flag) + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_docxmlrpc.py b/Lib/test/test_docxmlrpc.py index 3c05412ef57..999533a72a1 100644 --- a/Lib/test/test_docxmlrpc.py +++ b/Lib/test/test_docxmlrpc.py @@ -119,7 +119,7 @@ def test_invalid_get_response(self): response.read() - @unittest.skip('TODO: RUSTPYTHON; http.client.RemoteDisconnected: Remote end closed connection without response') + @unittest.expectedFailure # TODO: RUSTPYTHON; http.client.RemoteDisconnected: Remote end closed connection without response def test_lambda(self): """Test that lambda functionality stays the same. The output produced currently is, I suspect invalid because of the unencoded brackets in the @@ -160,9 +160,9 @@ def test_autolinking(self): b'auto-linked, too:
    \n' b'http://google.com.'), response) + @unittest.expectedFailure # TODO: RUSTPYTHON; http.client.RemoteDisconnected: Remote end closed connection without response @make_request_and_skipIf(sys.flags.optimize >= 2, "Docstrings are omitted with -O2 and above") - @unittest.skip('TODO: RUSTPYTHON; http.client.RemoteDisconnected: Remote end closed connection without response') def test_system_methods(self): """Test the presence of three consecutive system.* methods. @@ -190,7 +190,7 @@ def test_system_methods(self): b'
    \nThis server does NOT support system' b'.methodSignature.'), response) - @unittest.skip('TODO: RUSTPYTHON; http.client.RemoteDisconnected: Remote end closed connection without response') + @unittest.expectedFailure # TODO: RUSTPYTHON; http.client.RemoteDisconnected: Remote end closed connection without response def test_autolink_dotted_methods(self): """Test that selfdot values are made strong automatically in the documentation.""" @@ -200,7 +200,7 @@ def test_autolink_dotted_methods(self): self.assertIn(b"""Try self.add, too.""", response.read()) - @unittest.skip('TODO: RUSTPYTHON; http.client.RemoteDisconnected: Remote end closed connection without response') + @unittest.expectedFailure # TODO: RUSTPYTHON; http.client.RemoteDisconnected: Remote end closed connection without response def test_annotations(self): """ Test that annotations works as expected """ self.client.request("GET", "/") @@ -214,7 +214,7 @@ def test_annotations(self): b'method_annotation(x: bytes)'), response.read()) - @unittest.skip('TODO: RUSTPYTHON; TypeError: HTMLDoc.heading() missing 2 required positional arguments: "fgcol" and "bgcol"') + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: HTMLDoc.heading() missing 2 required positional arguments: "fgcol" and "bgcol" def test_server_title_escape(self): # bpo-38243: Ensure that the server title and documentation # are escaped for HTML. diff --git a/Lib/test/test_email/__init__.py b/Lib/test/test_email/__init__.py new file mode 100644 index 00000000000..455dc48facf --- /dev/null +++ b/Lib/test/test_email/__init__.py @@ -0,0 +1,167 @@ +import os +import unittest +import collections +import email +from email.message import Message +from email._policybase import compat32 +from test.support import load_package_tests +from test.support.testcase import ExtraAssertions +from test.test_email import __file__ as landmark + +# Load all tests in package +def load_tests(*args): + return load_package_tests(os.path.dirname(__file__), *args) + + +# helper code used by a number of test modules. + +def openfile(filename, *args, **kws): + path = os.path.join(os.path.dirname(landmark), 'data', filename) + return open(path, *args, **kws) + + +# Base test class +class TestEmailBase(unittest.TestCase, ExtraAssertions): + + maxDiff = None + # Currently the default policy is compat32. By setting that as the default + # here we make minimal changes in the test_email tests compared to their + # pre-3.3 state. + policy = compat32 + # Likewise, the default message object is Message. + message = Message + + def __init__(self, *args, **kw): + super().__init__(*args, **kw) + self.addTypeEqualityFunc(bytes, self.assertBytesEqual) + + # Backward compatibility to minimize test_email test changes. + ndiffAssertEqual = unittest.TestCase.assertEqual + + def _msgobj(self, filename): + with openfile(filename, encoding="utf-8") as fp: + return email.message_from_file(fp, policy=self.policy) + + def _str_msg(self, string, message=None, policy=None): + if policy is None: + policy = self.policy + if message is None: + message = self.message + return email.message_from_string(string, message, policy=policy) + + def _bytes_msg(self, bytestring, message=None, policy=None): + if policy is None: + policy = self.policy + if message is None: + message = self.message + return email.message_from_bytes(bytestring, message, policy=policy) + + def _make_message(self): + return self.message(policy=self.policy) + + def _bytes_repr(self, b): + return [repr(x) for x in b.splitlines(keepends=True)] + + def assertBytesEqual(self, first, second, msg): + """Our byte strings are really encoded strings; improve diff output""" + self.assertEqual(self._bytes_repr(first), self._bytes_repr(second)) + + def assertDefectsEqual(self, actual, expected): + self.assertEqual(len(actual), len(expected), actual) + for i in range(len(actual)): + self.assertIsInstance(actual[i], expected[i], + 'item {}'.format(i)) + + +def parameterize(cls): + """A test method parameterization class decorator. + + Parameters are specified as the value of a class attribute that ends with + the string '_params'. Call the portion before '_params' the prefix. Then + a method to be parameterized must have the same prefix, the string + '_as_', and an arbitrary suffix. + + The value of the _params attribute may be either a dictionary or a list. + The values in the dictionary and the elements of the list may either be + single values, or a list. If single values, they are turned into single + element tuples. However derived, the resulting sequence is passed via + *args to the parameterized test function. + + In a _params dictionary, the keys become part of the name of the generated + tests. In a _params list, the values in the list are converted into a + string by joining the string values of the elements of the tuple by '_' and + converting any blanks into '_'s, and this become part of the name. + The full name of a generated test is a 'test_' prefix, the portion of the + test function name after the '_as_' separator, plus an '_', plus the name + derived as explained above. + + For example, if we have: + + count_params = range(2) + + def count_as_foo_arg(self, foo): + self.assertEqual(foo+1, myfunc(foo)) + + we will get parameterized test methods named: + test_foo_arg_0 + test_foo_arg_1 + test_foo_arg_2 + + Or we could have: + + example_params = {'foo': ('bar', 1), 'bing': ('bang', 2)} + + def example_as_myfunc_input(self, name, count): + self.assertEqual(name+str(count), myfunc(name, count)) + + and get: + test_myfunc_input_foo + test_myfunc_input_bing + + Note: if and only if the generated test name is a valid identifier can it + be used to select the test individually from the unittest command line. + + The values in the params dict can be a single value, a tuple, or a + dict. If a single value of a tuple, it is passed to the test function + as positional arguments. If a dict, it is a passed via **kw. + + """ + paramdicts = {} + testers = collections.defaultdict(list) + for name, attr in cls.__dict__.items(): + if name.endswith('_params'): + if not hasattr(attr, 'keys'): + d = {} + for x in attr: + if not hasattr(x, '__iter__'): + x = (x,) + n = '_'.join(str(v) for v in x).replace(' ', '_') + d[n] = x + attr = d + paramdicts[name[:-7] + '_as_'] = attr + if '_as_' in name: + testers[name.split('_as_')[0] + '_as_'].append(name) + testfuncs = {} + for name in paramdicts: + if name not in testers: + raise ValueError("No tester found for {}".format(name)) + for name in testers: + if name not in paramdicts: + raise ValueError("No params found for {}".format(name)) + for name, attr in cls.__dict__.items(): + for paramsname, paramsdict in paramdicts.items(): + if name.startswith(paramsname): + testnameroot = 'test_' + name[len(paramsname):] + for paramname, params in paramsdict.items(): + if hasattr(params, 'keys'): + test = (lambda self, name=name, params=params: + getattr(self, name)(**params)) + else: + test = (lambda self, name=name, params=params: + getattr(self, name)(*params)) + testname = testnameroot + '_' + paramname + test.__name__ = testname + testfuncs[testname] = test + for key, value in testfuncs.items(): + setattr(cls, key, value) + return cls diff --git a/Lib/test/test_email/__main__.py b/Lib/test/test_email/__main__.py new file mode 100644 index 00000000000..4b14f773db4 --- /dev/null +++ b/Lib/test/test_email/__main__.py @@ -0,0 +1,4 @@ +from test.test_email import load_tests +import unittest + +unittest.main() diff --git a/Lib/test/test_email/data/msg_01.txt b/Lib/test/test_email/data/msg_01.txt new file mode 100644 index 00000000000..7e33bcf96af --- /dev/null +++ b/Lib/test/test_email/data/msg_01.txt @@ -0,0 +1,19 @@ +Return-Path: +Delivered-To: bbb@zzz.org +Received: by mail.zzz.org (Postfix, from userid 889) + id 27CEAD38CC; Fri, 4 May 2001 14:05:44 -0400 (EDT) +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Transfer-Encoding: 7bit +Message-ID: <15090.61304.110929.45684@aaa.zzz.org> +From: bbb@ddd.com (John X. Doe) +To: bbb@zzz.org +Subject: This is a test message +Date: Fri, 4 May 2001 14:05:44 -0400 + + +Hi, + +Do you like this message? + +-Me diff --git a/Lib/test/test_email/data/msg_02.txt b/Lib/test/test_email/data/msg_02.txt new file mode 100644 index 00000000000..5d0a7e16c82 --- /dev/null +++ b/Lib/test/test_email/data/msg_02.txt @@ -0,0 +1,136 @@ +MIME-version: 1.0 +From: ppp-request@zzz.org +Sender: ppp-admin@zzz.org +To: ppp@zzz.org +Subject: Ppp digest, Vol 1 #2 - 5 msgs +Date: Fri, 20 Apr 2001 20:18:00 -0400 (EDT) +X-Mailer: Mailman v2.0.4 +X-Mailman-Version: 2.0.4 +Content-Type: multipart/mixed; boundary="192.168.1.2.889.32614.987812255.500.21814" + +--192.168.1.2.889.32614.987812255.500.21814 +Content-type: text/plain; charset=us-ascii +Content-description: Masthead (Ppp digest, Vol 1 #2) + +Send Ppp mailing list submissions to + ppp@zzz.org + +To subscribe or unsubscribe via the World Wide Web, visit + http://www.zzz.org/mailman/listinfo/ppp +or, via email, send a message with subject or body 'help' to + ppp-request@zzz.org + +You can reach the person managing the list at + ppp-admin@zzz.org + +When replying, please edit your Subject line so it is more specific +than "Re: Contents of Ppp digest..." + + +--192.168.1.2.889.32614.987812255.500.21814 +Content-type: text/plain; charset=us-ascii +Content-description: Today's Topics (5 msgs) + +Today's Topics: + + 1. testing #1 (Barry A. Warsaw) + 2. testing #2 (Barry A. Warsaw) + 3. testing #3 (Barry A. Warsaw) + 4. testing #4 (Barry A. Warsaw) + 5. testing #5 (Barry A. Warsaw) + +--192.168.1.2.889.32614.987812255.500.21814 +Content-Type: multipart/digest; boundary="__--__--" + +--__--__-- + +Message: 1 +Content-Type: text/plain; charset=us-ascii +Content-Transfer-Encoding: 7bit +Date: Fri, 20 Apr 2001 20:16:13 -0400 +To: ppp@zzz.org +From: barry@digicool.com (Barry A. Warsaw) +Subject: [Ppp] testing #1 +Precedence: bulk + + +hello + + +--__--__-- + +Message: 2 +Date: Fri, 20 Apr 2001 20:16:21 -0400 +Content-Type: text/plain; charset=us-ascii +Content-Transfer-Encoding: 7bit +To: ppp@zzz.org +From: barry@digicool.com (Barry A. Warsaw) +Precedence: bulk + + +hello + + +--__--__-- + +Message: 3 +Date: Fri, 20 Apr 2001 20:16:25 -0400 +Content-Type: text/plain; charset=us-ascii +Content-Transfer-Encoding: 7bit +To: ppp@zzz.org +From: barry@digicool.com (Barry A. Warsaw) +Subject: [Ppp] testing #3 +Precedence: bulk + + +hello + + +--__--__-- + +Message: 4 +Date: Fri, 20 Apr 2001 20:16:28 -0400 +Content-Type: text/plain; charset=us-ascii +Content-Transfer-Encoding: 7bit +To: ppp@zzz.org +From: barry@digicool.com (Barry A. Warsaw) +Subject: [Ppp] testing #4 +Precedence: bulk + + +hello + + +--__--__-- + +Message: 5 +Date: Fri, 20 Apr 2001 20:16:32 -0400 +Content-Type: text/plain; charset=us-ascii +Content-Transfer-Encoding: 7bit +To: ppp@zzz.org +From: barry@digicool.com (Barry A. Warsaw) +Subject: [Ppp] testing #5 +Precedence: bulk + + +hello + + + + +--__--__---- + +--192.168.1.2.889.32614.987812255.500.21814 +Content-type: text/plain; charset=us-ascii +Content-description: Digest Footer + +_______________________________________________ +Ppp mailing list +Ppp@zzz.org +http://www.zzz.org/mailman/listinfo/ppp + + +--192.168.1.2.889.32614.987812255.500.21814-- + +End of Ppp Digest + diff --git a/Lib/test/test_email/data/msg_03.txt b/Lib/test/test_email/data/msg_03.txt new file mode 100644 index 00000000000..c748ebf1176 --- /dev/null +++ b/Lib/test/test_email/data/msg_03.txt @@ -0,0 +1,16 @@ +Return-Path: +Delivered-To: bbb@zzz.org +Received: by mail.zzz.org (Postfix, from userid 889) + id 27CEAD38CC; Fri, 4 May 2001 14:05:44 -0400 (EDT) +Message-ID: <15090.61304.110929.45684@aaa.zzz.org> +From: bbb@ddd.com (John X. Doe) +To: bbb@zzz.org +Subject: This is a test message +Date: Fri, 4 May 2001 14:05:44 -0400 + + +Hi, + +Do you like this message? + +-Me diff --git a/Lib/test/test_email/data/msg_04.txt b/Lib/test/test_email/data/msg_04.txt new file mode 100644 index 00000000000..1f633c4496f --- /dev/null +++ b/Lib/test/test_email/data/msg_04.txt @@ -0,0 +1,37 @@ +Return-Path: +Delivered-To: barry@python.org +Received: by mail.python.org (Postfix, from userid 889) + id C2BF0D37C6; Tue, 11 Sep 2001 00:05:05 -0400 (EDT) +MIME-Version: 1.0 +Content-Type: multipart/mixed; boundary="h90VIIIKmx" +Content-Transfer-Encoding: 7bit +Message-ID: <15261.36209.358846.118674@anthem.python.org> +From: barry@python.org (Barry A. Warsaw) +To: barry@python.org +Subject: a simple multipart +Date: Tue, 11 Sep 2001 00:05:05 -0400 +X-Mailer: VM 6.95 under 21.4 (patch 4) "Artificial Intelligence" XEmacs Lucid +X-Attribution: BAW +X-Oblique-Strategy: Make a door into a window + + +--h90VIIIKmx +Content-Type: text/plain +Content-Disposition: inline; + filename="msg.txt" +Content-Transfer-Encoding: 7bit + +a simple kind of mirror +to reflect upon our own + +--h90VIIIKmx +Content-Type: text/plain +Content-Disposition: inline; + filename="msg.txt" +Content-Transfer-Encoding: 7bit + +a simple kind of mirror +to reflect upon our own + +--h90VIIIKmx-- + diff --git a/Lib/test/test_email/data/msg_05.txt b/Lib/test/test_email/data/msg_05.txt new file mode 100644 index 00000000000..87d5e9cbf8b --- /dev/null +++ b/Lib/test/test_email/data/msg_05.txt @@ -0,0 +1,28 @@ +From: foo +Subject: bar +To: baz +MIME-Version: 1.0 +Content-Type: multipart/report; report-type=delivery-status; + boundary="D1690A7AC1.996856090/mail.example.com" +Message-Id: <20010803162810.0CA8AA7ACC@mail.example.com> + +This is a MIME-encapsulated message. + +--D1690A7AC1.996856090/mail.example.com +Content-Type: text/plain + +Yadda yadda yadda + +--D1690A7AC1.996856090/mail.example.com + +Yadda yadda yadda + +--D1690A7AC1.996856090/mail.example.com +Content-Type: message/rfc822 + +From: nobody@python.org + +Yadda yadda yadda + +--D1690A7AC1.996856090/mail.example.com-- + diff --git a/Lib/test/test_email/data/msg_06.txt b/Lib/test/test_email/data/msg_06.txt new file mode 100644 index 00000000000..f51ac96114b --- /dev/null +++ b/Lib/test/test_email/data/msg_06.txt @@ -0,0 +1,33 @@ +Return-Path: +Delivered-To: barry@python.org +MIME-Version: 1.0 +Content-Type: message/rfc822 +Content-Description: forwarded message +Content-Transfer-Encoding: 7bit +Message-ID: <15265.9482.641338.555352@python.org> +From: barry@python.org (Barry A. Warsaw) +Sender: barry@python.org +To: barry@python.org +Subject: forwarded message from Barry A. Warsaw +Date: Thu, 13 Sep 2001 17:28:42 -0400 +X-Mailer: VM 6.95 under 21.4 (patch 4) "Artificial Intelligence" XEmacs Lucid +X-Attribution: BAW +X-Oblique-Strategy: Be dirty +X-Url: http://barry.wooz.org + +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Return-Path: +Delivered-To: barry@python.org +Message-ID: <15265.9468.713530.98441@python.org> +From: barry@python.org (Barry A. Warsaw) +Sender: barry@python.org +To: barry@python.org +Subject: testing +Date: Thu, 13 Sep 2001 17:28:28 -0400 +X-Mailer: VM 6.95 under 21.4 (patch 4) "Artificial Intelligence" XEmacs Lucid +X-Attribution: BAW +X-Oblique-Strategy: Spectrum analysis +X-Url: http://barry.wooz.org + + diff --git a/Lib/test/test_email/data/msg_07.txt b/Lib/test/test_email/data/msg_07.txt new file mode 100644 index 00000000000..721f3a0d316 --- /dev/null +++ b/Lib/test/test_email/data/msg_07.txt @@ -0,0 +1,83 @@ +MIME-Version: 1.0 +From: Barry +To: Dingus Lovers +Subject: Here is your dingus fish +Date: Fri, 20 Apr 2001 19:35:02 -0400 +Content-Type: multipart/mixed; boundary="BOUNDARY" + +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" + +Hi there, + +This is the dingus fish. + +--BOUNDARY +Content-Type: image/gif; name="dingusfish.gif" +Content-Transfer-Encoding: base64 +content-disposition: attachment; filename="dingusfish.gif" + +R0lGODdhAAEAAfAAAP///wAAACwAAAAAAAEAAQAC/oSPqcvtD6OctNqLs968+w+G4kiW5omm6sq2 +7gvH8kzX9o3n+s73/g8MCofEovGITGICTKbyCV0FDNOo9SqpQqpOrJfXzTQj2vD3TGtqL+NtGQ2f +qTXmxzuOd7WXdcc9DyjU53ewFni4s0fGhdiYaEhGBelICTNoV1j5NUnFcrmUqemjNifJVWpaOqaI +oFq3SspZsSraE7sHq3jr1MZqWvi662vxV4tD+pvKW6aLDOCLyur8PDwbanyDeq0N3DctbQYeLDvR +RY6t95m6UB0d3mwIrV7e2VGNvjjffukeJp4w7F65KecGFsTHQGAygOrgrWs1jt28Rc88KESYcGLA +/obvTkH6p+CinWJiJmIMqXGQwH/y4qk0SYjgQTczT3ajKZGfuI0uJ4kkVI/DT5s3/ejkxI0aT4Y+ +YTYgWbImUaXk9nlLmnSh1qJiJFl0OpUqRK4oOy7NyRQtHWofhoYVxkwWXKUSn0YsS+fUV6lhqfYb +6ayd3Z5qQdG1B7bvQzaJjwUV2lixMUZ7JVsOlfjWVr/3NB/uFvnySBN6Dcb6rGwaRM3wsormw5cC +M9NxWy/bWdufudCvy8bOAjXjVVwta/uO21sE5RHBCzNFXtgq9ORtH4eYjVP4Yryo026nvkFmCeyA +B29efV6ravCMK5JwWd5897Qrx7ll38o6iHDZ/rXPR//feevhF4l7wjUGX3xq1eeRfM4RSJGBIV1D +z1gKPkfWag3mVBVvva1RlX5bAJTPR/2YqNtw/FkIYYEi/pIZiAdpcxpoHtmnYYoZtvhUftzdx5ZX +JSKDW405zkGcZzzGZ6KEv4FI224oDmijlEf+xp6MJK5ojY/ASeVUR+wsKRuJ+XFZ5o7ZeEime8t1 +ouUsU6YjF5ZtUihhkGfCdFQLWQFJ3UXxmElfhQnR+eCdcDbkFZp6vTRmj56ApCihn5QGpaToNZmR +n3NVSpZcQpZ2KEONusaiCsKAug0wkQbJSFO+PTSjneGxOuFjPlUk3ovWvdIerjUg9ZGIOtGq/qeX +eCYrrCX+1UPsgTKGGRSbzd5q156d/gpfbJxe66eD5iQKrXj7RGgruGxs62qebBHUKS32CKluCiqZ +qh+pmehmEb71noAUoe5e9Zm17S7773V10pjrtG4CmuurCV/n6zLK5turWNhqOvFXbjhZrMD0YhKe +wR0zOyuvsh6MWrGoIuzvyWu5y1WIFAqmJselypxXh6dKLNOKEB98L88bS2rkNqqlKzCNJp9c0G0j +Gzh0iRrCbHSXmPR643QS+4rWhgFmnSbSuXCjS0xAOWkU2UdLqyuUNfHSFdUouy3bm5i5GnDM3tG8 +doJ4r5tqu3pPbRSVfvs8uJzeNXhp3n4j/tZ42SwH7eaWUUOjc3qFV9453UHTXZfcLH+OeNs5g36x +lBnHvTm7EbMbLeuaLncao8vWCXimfo1o+843Ak6y4ChNeGntvAYvfLK4ezmoyNIbNCLTCXO9ZV3A +E8/s88RczPzDwI4Ob7XZyl7+9Miban29h+tJZPrE21wgvBphDfrrfPdCTPKJD/y98L1rZwHcV6Jq +Zab0metpuNIX/qAFPoz171WUaUb4HAhBSzHuHfjzHb3kha/2Cctis/ORArVHNYfFyYRH2pYIRzic +isVOfPWD1b6mRTqpCRBozzof6UZVvFXRxWIr3GGrEviGYgyPMfahheiSaLs/9QeFu7oZ/ndSY8DD +ya9x+uPed+7mxN2IzIISBOMLFYWVqC3Pew1T2nFuuCiwZS5/v6II10i4t1OJcUH2U9zxKodHsGGv +Oa+zkvNUYUOa/TCCRutF9MzDwdlUMJADTCGSbDQ5OV4PTamDoPEi6Ecc/RF5RWwkcdSXvSOaDWSn +I9LlvubFTQpuc6JKXLcKeb+xdbKRBnwREemXyjg6ME65aJiOuBgrktzykfPLJBKR9ClMavJ62/Ff +BlNIyod9yX9wcSXexnXFpvkrbXk64xsx5Db7wXKP5fSgsvwIMM/9631VLBfkmtbHRXpqmtei52hG +pUwSlo+BASQoeILDOBgREECxBBh5/iYmNsQ9dIv5+OI++QkqdsJPc3uykz5fkM+OraeekcQF7X4n +B5S67za5U967PmooGQhUXfF7afXyCD7ONdRe17QogYjVx38uLwtrS6nhTnm15LQUnu9E2uK6CNI/ +1HOABj0ESwOjut4FEpFQpdNAm4K2LHnDWHNcmKB2ioKBogysVZtMO2nSxUdZ8Yk2kJc7URioLVI0 +YgmtIwZj4LoeKemgnOnbUdGnzZ4Oa6scqiolBGqS6RgWNLu0RMhcaE6rhhU4hiuqFXPAG8fGwTPW +FKeLMtdVmXLSs5YJGF/YeVm7rREMlY3UYE+yCxbaMXX8y15m5zVHq6GOKDMynzII/jdUHdyVqIy0 +ifX2+r/EgtZcvRzSb72gU9ui87M2VecjKildW/aFqaYhKoryUjfB/g4qtyVuc60xFDGmCxwjW+qu +zjuwl2GkOWn66+3QiiEctvd04OVvcCVzjgT7lrkvjVGKKHmmlDUKowSeikb5kK/mJReuWOxONx+s +ULsl+Lqb0CVn0SrVyJ6wt4t6yTeSCafhPhAf0OXn6L60UMxiLolFAtmN35S2Ob1lZpQ1r/n0Qb5D +oQ1zJiRVDgF8N3Q8TYfbi3DyWCy3lT1nxyBs6FT3S2GOzWRlxwKvlRP0RPJA9SjxEy0UoEnkA+M4 +cnzLMJrBGWLFEaaUb5lvpqbq/loOaU5+DFuHPxo82/OZuM8FXG3oVNZhtWpMpb/0Xu5m/LfLhHZQ +7yuVI0MqZ7NE43imC8jH3IwGZlbPm0xkJYs7+2U48hXTsFSMqgGDvai0kLxyynKNT/waj+q1c1tz +GjOpPBgdCSq3UKZxCSsqFIY+O6JbAWGWcV1pwqLyj5sGqCF1xb1F3varUWqrJv6cN3PrUXzijtfZ +FshpBL3Xwr4GIPvU2N8EjrJgS1zl21rbXQMXeXc5jjFyrhpCzijSv/RQtyPSzHCFMhlME95fHglt +pRsX+dfSQjUeHAlpWzJ5iOo79Ldnaxai6bXTcGO3fp07ri7HLEmXXPlYi8bv/qVxvNcdra6m7Rlb +6JBTb5fd66VhFRjGArh2n7R1rDW4P5NOT9K0I183T2scYkeZ3q/VFyLb09U9ajzXBS8Kgkhc4mBS +kYY9cy3Vy9lUnuNJH8HGIclUilwnBtjUOH0gteGOZ4c/XNrhXLSYDyxfnD8z1pDy7rYRvDolhnbe +UMzxCZUs40s6s7UIvBnLgc0+vKuOkIXeOrDymlp+Zxra4MZLBbVrqD/jTJ597pDmnw5c4+DbyB88 +9Cg9DodYcSuMZT/114pptqc/EuTjRPvH/z5slzI3tluOEBBLqOXLOX+0I5929tO97wkvl/atCz+y +xJrdwteW2FNW/NSmBP+f/maYtVs/bYyBC7Ox3jsYZHL05CIrBa/nS+b3bHfiYm4Ueil1YZZSgAUI +fFZ1dxUmeA2oQRQ3RuGXNGLFV9/XbGFGPV6kfzk1TBBCd+izc7q1H+OHMJwmaBX2IQNYVAKHYepV +SSGCe6CnbYHHETKGNe43EDvFgZr0gB/nVHPHZ80VV1ojOiI3XDvYIkl4ayo4bxQIgrFXWTvBI0nH +VElWMuw2aLUWCRHHf8ymVCHjFlJnOSojfevCYyyyZDH0IcvHhrsnQ5O1OsWzONuVVKIxSxiFZ/tR +fKDAf6xFTnw4O9Qig2VCfW2hJQrmMOuHW0W3dLQmCMO2ccdUd/xyfflH/olTiHZVdGwb8nIwRzSE +J15jFlOJuBZBZ4CiyHyd2IFylFlB+HgHhYabhWOGwYO1ZH/Og1dtQlFMk352CGRSIFTapnWQEUtN +l4zv8S0aaCFDyGCBqDUxZYpxGHX01y/JuH1xhn7TOCnNCI4eKDs5WGX4R425F4vF1o3BJ4vO0otq +I3rimI7jJY1jISqnBxknCIvruF83mF5wN4X7qGLIhR8A2Vg0yFERSIXn9Vv3GHy3Vj/WIkKddlYi +yIMv2I/VMjTLpW7pt05SWIZR0RPyxpB4SIUM9lBPGBl0GC7oSEEwRYLe4pJpZY2P0zbI1n+Oc44w +qY3PUnmF0ixjVpDD/mJ9wpOBGTVgXlaCaZiPcIWK5NiKBIiPdGaQ0TWGvAiG7nMchdZb7Vgf8zNi +MuMyzRdy/lePe9iC4TRx7WhhOQI/QiSVNAmAa2lT/piFbuh7ofJoYSZzrSZ1bvmWw3eN2nKUPVky +uPN5/VRfohRd0VYZoqhKIlU6TXYhJxmPUIloAwc1bPmHEpaZYZORHNlXUJM07hATwHR8MJYqkwWR +WaIezFhxSFlc8/Fq82hEnpeRozg3ULhhr9lAGtVEkCg5ZNRuuVleBPaZadhG0ZgkyPmDOTOKzViM +YgOcpukKqQcbjAWS0IleQ2ROjdh6A+md1qWdBRSX7iSYgFRTtRmBpJioieXJiHfJiMGIR9fJOn8I +MSfXYhspn4ooSa2mSAj4n+8Bmg03fBJZoPOJgsVZRxu1oOMRPXYYjdqjihFaEoZpXBREanuJoRI6 +cibFinq4ngUKh/wQd/H5ofYCZ0HJXR62opZFaAT0iFIZo4DIiUojkjeqKiuoZirKo5Y1a7AWckGa +BkuYoD5lpDK6eUs6CkDqpETwl1EqpfhJpVeKpVl6EgUAADs= + +--BOUNDARY-- diff --git a/Lib/test/test_email/data/msg_08.txt b/Lib/test/test_email/data/msg_08.txt new file mode 100644 index 00000000000..132ce7ada9b --- /dev/null +++ b/Lib/test/test_email/data/msg_08.txt @@ -0,0 +1,24 @@ +MIME-Version: 1.0 +From: Barry Warsaw +To: Dingus Lovers +Subject: Lyrics +Date: Fri, 20 Apr 2001 19:35:02 -0400 +Content-Type: multipart/mixed; boundary="BOUNDARY" + +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" + + +--BOUNDARY +Content-Type: text/html; charset="iso-8859-1" + + +--BOUNDARY +Content-Type: text/plain; charset="iso-8859-2" + + +--BOUNDARY +Content-Type: text/plain; charset="koi8-r" + + +--BOUNDARY-- diff --git a/Lib/test/test_email/data/msg_09.txt b/Lib/test/test_email/data/msg_09.txt new file mode 100644 index 00000000000..0cfa6bab2ba --- /dev/null +++ b/Lib/test/test_email/data/msg_09.txt @@ -0,0 +1,24 @@ +MIME-Version: 1.0 +From: Barry Warsaw +To: Dingus Lovers +Subject: Lyrics +Date: Fri, 20 Apr 2001 19:35:02 -0400 +Content-Type: multipart/mixed; boundary="BOUNDARY" + +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" + + +--BOUNDARY +Content-Type: text/html; charset="iso-8859-1" + + +--BOUNDARY +Content-Type: text/plain + + +--BOUNDARY +Content-Type: text/plain; charset="koi8-r" + + +--BOUNDARY-- diff --git a/Lib/test/test_email/data/msg_10.txt b/Lib/test/test_email/data/msg_10.txt new file mode 100644 index 00000000000..d49e477a818 --- /dev/null +++ b/Lib/test/test_email/data/msg_10.txt @@ -0,0 +1,39 @@ +MIME-Version: 1.0 +From: Barry Warsaw +To: Dingus Lovers +Subject: Lyrics +Date: Fri, 20 Apr 2001 19:35:02 -0400 +Content-Type: multipart/mixed; boundary="BOUNDARY" + +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" +Content-Transfer-Encoding: 7bit + +This is a 7bit encoded message. + +--BOUNDARY +Content-Type: text/html; charset="iso-8859-1" +Content-Transfer-Encoding: Quoted-Printable + +=A1This is a Quoted Printable encoded message! + +--BOUNDARY +Content-Type: text/plain; charset="iso-8859-1" +Content-Transfer-Encoding: Base64 + +VGhpcyBpcyBhIEJhc2U2NCBlbmNvZGVkIG1lc3NhZ2Uu + + +--BOUNDARY +Content-Type: text/plain; charset="iso-8859-1" +Content-Transfer-Encoding: Base64 + +VGhpcyBpcyBhIEJhc2U2NCBlbmNvZGVkIG1lc3NhZ2UuCg== + + +--BOUNDARY +Content-Type: text/plain; charset="iso-8859-1" + +This has no Content-Transfer-Encoding: header. + +--BOUNDARY-- diff --git a/Lib/test/test_email/data/msg_11.txt b/Lib/test/test_email/data/msg_11.txt new file mode 100644 index 00000000000..8f7f1991cbd --- /dev/null +++ b/Lib/test/test_email/data/msg_11.txt @@ -0,0 +1,7 @@ +Content-Type: message/rfc822 +MIME-Version: 1.0 +Subject: The enclosing message + +Subject: An enclosed message + +Here is the body of the message. diff --git a/Lib/test/test_email/data/msg_12.txt b/Lib/test/test_email/data/msg_12.txt new file mode 100644 index 00000000000..b109b985c8b --- /dev/null +++ b/Lib/test/test_email/data/msg_12.txt @@ -0,0 +1,36 @@ +MIME-Version: 1.0 +From: Barry Warsaw +To: Dingus Lovers +Subject: Lyrics +Date: Fri, 20 Apr 2001 19:35:02 -0400 +Content-Type: multipart/mixed; boundary="BOUNDARY" + +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" + + +--BOUNDARY +Content-Type: text/html; charset="iso-8859-1" + + +--BOUNDARY +Content-Type: multipart/mixed; boundary="ANOTHER" + +--ANOTHER +Content-Type: text/plain; charset="iso-8859-2" + + +--ANOTHER +Content-Type: text/plain; charset="iso-8859-3" + +--ANOTHER-- + +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" + + +--BOUNDARY +Content-Type: text/plain; charset="koi8-r" + + +--BOUNDARY-- diff --git a/Lib/test/test_email/data/msg_12a.txt b/Lib/test/test_email/data/msg_12a.txt new file mode 100644 index 00000000000..2092aa0c351 --- /dev/null +++ b/Lib/test/test_email/data/msg_12a.txt @@ -0,0 +1,38 @@ +MIME-Version: 1.0 +From: Barry Warsaw +To: Dingus Lovers +Subject: Lyrics +Date: Fri, 20 Apr 2001 19:35:02 -0400 +Content-Type: multipart/mixed; boundary="BOUNDARY" + +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" + + +--BOUNDARY +Content-Type: text/html; charset="iso-8859-1" + + +--BOUNDARY +Content-Type: multipart/mixed; boundary="ANOTHER" + +--ANOTHER +Content-Type: text/plain; charset="iso-8859-2" + + +--ANOTHER +Content-Type: text/plain; charset="iso-8859-3" + + +--ANOTHER-- + + +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" + + +--BOUNDARY +Content-Type: text/plain; charset="koi8-r" + + +--BOUNDARY-- diff --git a/Lib/test/test_email/data/msg_13.txt b/Lib/test/test_email/data/msg_13.txt new file mode 100644 index 00000000000..8e6d52d5bef --- /dev/null +++ b/Lib/test/test_email/data/msg_13.txt @@ -0,0 +1,94 @@ +MIME-Version: 1.0 +From: Barry +To: Dingus Lovers +Subject: Here is your dingus fish +Date: Fri, 20 Apr 2001 19:35:02 -0400 +Content-Type: multipart/mixed; boundary="OUTER" + +--OUTER +Content-Type: text/plain; charset="us-ascii" + +A text/plain part + +--OUTER +Content-Type: multipart/mixed; boundary=BOUNDARY + + +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" + +Hi there, + +This is the dingus fish. + +--BOUNDARY +Content-Type: image/gif; name="dingusfish.gif" +Content-Transfer-Encoding: base64 +content-disposition: attachment; filename="dingusfish.gif" + +R0lGODdhAAEAAfAAAP///wAAACwAAAAAAAEAAQAC/oSPqcvtD6OctNqLs968+w+G4kiW5omm6sq2 +7gvH8kzX9o3n+s73/g8MCofEovGITGICTKbyCV0FDNOo9SqpQqpOrJfXzTQj2vD3TGtqL+NtGQ2f +qTXmxzuOd7WXdcc9DyjU53ewFni4s0fGhdiYaEhGBelICTNoV1j5NUnFcrmUqemjNifJVWpaOqaI +oFq3SspZsSraE7sHq3jr1MZqWvi662vxV4tD+pvKW6aLDOCLyur8PDwbanyDeq0N3DctbQYeLDvR +RY6t95m6UB0d3mwIrV7e2VGNvjjffukeJp4w7F65KecGFsTHQGAygOrgrWs1jt28Rc88KESYcGLA +/obvTkH6p+CinWJiJmIMqXGQwH/y4qk0SYjgQTczT3ajKZGfuI0uJ4kkVI/DT5s3/ejkxI0aT4Y+ +YTYgWbImUaXk9nlLmnSh1qJiJFl0OpUqRK4oOy7NyRQtHWofhoYVxkwWXKUSn0YsS+fUV6lhqfYb +6ayd3Z5qQdG1B7bvQzaJjwUV2lixMUZ7JVsOlfjWVr/3NB/uFvnySBN6Dcb6rGwaRM3wsormw5cC +M9NxWy/bWdufudCvy8bOAjXjVVwta/uO21sE5RHBCzNFXtgq9ORtH4eYjVP4Yryo026nvkFmCeyA +B29efV6ravCMK5JwWd5897Qrx7ll38o6iHDZ/rXPR//feevhF4l7wjUGX3xq1eeRfM4RSJGBIV1D +z1gKPkfWag3mVBVvva1RlX5bAJTPR/2YqNtw/FkIYYEi/pIZiAdpcxpoHtmnYYoZtvhUftzdx5ZX +JSKDW405zkGcZzzGZ6KEv4FI224oDmijlEf+xp6MJK5ojY/ASeVUR+wsKRuJ+XFZ5o7ZeEime8t1 +ouUsU6YjF5ZtUihhkGfCdFQLWQFJ3UXxmElfhQnR+eCdcDbkFZp6vTRmj56ApCihn5QGpaToNZmR +n3NVSpZcQpZ2KEONusaiCsKAug0wkQbJSFO+PTSjneGxOuFjPlUk3ovWvdIerjUg9ZGIOtGq/qeX +eCYrrCX+1UPsgTKGGRSbzd5q156d/gpfbJxe66eD5iQKrXj7RGgruGxs62qebBHUKS32CKluCiqZ +qh+pmehmEb71noAUoe5e9Zm17S7773V10pjrtG4CmuurCV/n6zLK5turWNhqOvFXbjhZrMD0YhKe +wR0zOyuvsh6MWrGoIuzvyWu5y1WIFAqmJselypxXh6dKLNOKEB98L88bS2rkNqqlKzCNJp9c0G0j +Gzh0iRrCbHSXmPR643QS+4rWhgFmnSbSuXCjS0xAOWkU2UdLqyuUNfHSFdUouy3bm5i5GnDM3tG8 +doJ4r5tqu3pPbRSVfvs8uJzeNXhp3n4j/tZ42SwH7eaWUUOjc3qFV9453UHTXZfcLH+OeNs5g36x +lBnHvTm7EbMbLeuaLncao8vWCXimfo1o+843Ak6y4ChNeGntvAYvfLK4ezmoyNIbNCLTCXO9ZV3A +E8/s88RczPzDwI4Ob7XZyl7+9Miban29h+tJZPrE21wgvBphDfrrfPdCTPKJD/y98L1rZwHcV6Jq +Zab0metpuNIX/qAFPoz171WUaUb4HAhBSzHuHfjzHb3kha/2Cctis/ORArVHNYfFyYRH2pYIRzic +isVOfPWD1b6mRTqpCRBozzof6UZVvFXRxWIr3GGrEviGYgyPMfahheiSaLs/9QeFu7oZ/ndSY8DD +ya9x+uPed+7mxN2IzIISBOMLFYWVqC3Pew1T2nFuuCiwZS5/v6II10i4t1OJcUH2U9zxKodHsGGv +Oa+zkvNUYUOa/TCCRutF9MzDwdlUMJADTCGSbDQ5OV4PTamDoPEi6Ecc/RF5RWwkcdSXvSOaDWSn +I9LlvubFTQpuc6JKXLcKeb+xdbKRBnwREemXyjg6ME65aJiOuBgrktzykfPLJBKR9ClMavJ62/Ff +BlNIyod9yX9wcSXexnXFpvkrbXk64xsx5Db7wXKP5fSgsvwIMM/9631VLBfkmtbHRXpqmtei52hG +pUwSlo+BASQoeILDOBgREECxBBh5/iYmNsQ9dIv5+OI++QkqdsJPc3uykz5fkM+OraeekcQF7X4n +B5S67za5U967PmooGQhUXfF7afXyCD7ONdRe17QogYjVx38uLwtrS6nhTnm15LQUnu9E2uK6CNI/ +1HOABj0ESwOjut4FEpFQpdNAm4K2LHnDWHNcmKB2ioKBogysVZtMO2nSxUdZ8Yk2kJc7URioLVI0 +YgmtIwZj4LoeKemgnOnbUdGnzZ4Oa6scqiolBGqS6RgWNLu0RMhcaE6rhhU4hiuqFXPAG8fGwTPW +FKeLMtdVmXLSs5YJGF/YeVm7rREMlY3UYE+yCxbaMXX8y15m5zVHq6GOKDMynzII/jdUHdyVqIy0 +ifX2+r/EgtZcvRzSb72gU9ui87M2VecjKildW/aFqaYhKoryUjfB/g4qtyVuc60xFDGmCxwjW+qu +zjuwl2GkOWn66+3QiiEctvd04OVvcCVzjgT7lrkvjVGKKHmmlDUKowSeikb5kK/mJReuWOxONx+s +ULsl+Lqb0CVn0SrVyJ6wt4t6yTeSCafhPhAf0OXn6L60UMxiLolFAtmN35S2Ob1lZpQ1r/n0Qb5D +oQ1zJiRVDgF8N3Q8TYfbi3DyWCy3lT1nxyBs6FT3S2GOzWRlxwKvlRP0RPJA9SjxEy0UoEnkA+M4 +cnzLMJrBGWLFEaaUb5lvpqbq/loOaU5+DFuHPxo82/OZuM8FXG3oVNZhtWpMpb/0Xu5m/LfLhHZQ +7yuVI0MqZ7NE43imC8jH3IwGZlbPm0xkJYs7+2U48hXTsFSMqgGDvai0kLxyynKNT/waj+q1c1tz +GjOpPBgdCSq3UKZxCSsqFIY+O6JbAWGWcV1pwqLyj5sGqCF1xb1F3varUWqrJv6cN3PrUXzijtfZ +FshpBL3Xwr4GIPvU2N8EjrJgS1zl21rbXQMXeXc5jjFyrhpCzijSv/RQtyPSzHCFMhlME95fHglt +pRsX+dfSQjUeHAlpWzJ5iOo79Ldnaxai6bXTcGO3fp07ri7HLEmXXPlYi8bv/qVxvNcdra6m7Rlb +6JBTb5fd66VhFRjGArh2n7R1rDW4P5NOT9K0I183T2scYkeZ3q/VFyLb09U9ajzXBS8Kgkhc4mBS +kYY9cy3Vy9lUnuNJH8HGIclUilwnBtjUOH0gteGOZ4c/XNrhXLSYDyxfnD8z1pDy7rYRvDolhnbe +UMzxCZUs40s6s7UIvBnLgc0+vKuOkIXeOrDymlp+Zxra4MZLBbVrqD/jTJ597pDmnw5c4+DbyB88 +9Cg9DodYcSuMZT/114pptqc/EuTjRPvH/z5slzI3tluOEBBLqOXLOX+0I5929tO97wkvl/atCz+y +xJrdwteW2FNW/NSmBP+f/maYtVs/bYyBC7Ox3jsYZHL05CIrBa/nS+b3bHfiYm4Ueil1YZZSgAUI +fFZ1dxUmeA2oQRQ3RuGXNGLFV9/XbGFGPV6kfzk1TBBCd+izc7q1H+OHMJwmaBX2IQNYVAKHYepV +SSGCe6CnbYHHETKGNe43EDvFgZr0gB/nVHPHZ80VV1ojOiI3XDvYIkl4ayo4bxQIgrFXWTvBI0nH +VElWMuw2aLUWCRHHf8ymVCHjFlJnOSojfevCYyyyZDH0IcvHhrsnQ5O1OsWzONuVVKIxSxiFZ/tR +fKDAf6xFTnw4O9Qig2VCfW2hJQrmMOuHW0W3dLQmCMO2ccdUd/xyfflH/olTiHZVdGwb8nIwRzSE +J15jFlOJuBZBZ4CiyHyd2IFylFlB+HgHhYabhWOGwYO1ZH/Og1dtQlFMk352CGRSIFTapnWQEUtN +l4zv8S0aaCFDyGCBqDUxZYpxGHX01y/JuH1xhn7TOCnNCI4eKDs5WGX4R425F4vF1o3BJ4vO0otq +I3rimI7jJY1jISqnBxknCIvruF83mF5wN4X7qGLIhR8A2Vg0yFERSIXn9Vv3GHy3Vj/WIkKddlYi +yIMv2I/VMjTLpW7pt05SWIZR0RPyxpB4SIUM9lBPGBl0GC7oSEEwRYLe4pJpZY2P0zbI1n+Oc44w +qY3PUnmF0ixjVpDD/mJ9wpOBGTVgXlaCaZiPcIWK5NiKBIiPdGaQ0TWGvAiG7nMchdZb7Vgf8zNi +MuMyzRdy/lePe9iC4TRx7WhhOQI/QiSVNAmAa2lT/piFbuh7ofJoYSZzrSZ1bvmWw3eN2nKUPVky +uPN5/VRfohRd0VYZoqhKIlU6TXYhJxmPUIloAwc1bPmHEpaZYZORHNlXUJM07hATwHR8MJYqkwWR +WaIezFhxSFlc8/Fq82hEnpeRozg3ULhhr9lAGtVEkCg5ZNRuuVleBPaZadhG0ZgkyPmDOTOKzViM +YgOcpukKqQcbjAWS0IleQ2ROjdh6A+md1qWdBRSX7iSYgFRTtRmBpJioieXJiHfJiMGIR9fJOn8I +MSfXYhspn4ooSa2mSAj4n+8Bmg03fBJZoPOJgsVZRxu1oOMRPXYYjdqjihFaEoZpXBREanuJoRI6 +cibFinq4ngUKh/wQd/H5ofYCZ0HJXR62opZFaAT0iFIZo4DIiUojkjeqKiuoZirKo5Y1a7AWckGa +BkuYoD5lpDK6eUs6CkDqpETwl1EqpfhJpVeKpVl6EgUAADs= + +--BOUNDARY-- + +--OUTER-- diff --git a/Lib/test/test_email/data/msg_14.txt b/Lib/test/test_email/data/msg_14.txt new file mode 100644 index 00000000000..5d98d2fd145 --- /dev/null +++ b/Lib/test/test_email/data/msg_14.txt @@ -0,0 +1,23 @@ +Return-Path: +Delivered-To: bbb@zzz.org +Received: by mail.zzz.org (Postfix, from userid 889) + id 27CEAD38CC; Fri, 4 May 2001 14:05:44 -0400 (EDT) +MIME-Version: 1.0 +Content-Type: text; charset=us-ascii +Content-Transfer-Encoding: 7bit +Message-ID: <15090.61304.110929.45684@aaa.zzz.org> +From: bbb@ddd.com (John X. Doe) +To: bbb@zzz.org +Subject: This is a test message +Date: Fri, 4 May 2001 14:05:44 -0400 + + +Hi, + +I'm sorry but I'm using a drainbread ISP, which although big and +wealthy can't seem to generate standard compliant email. :( + +This message has a Content-Type: header with no subtype. I hope you +can still read it. + +-Me diff --git a/Lib/test/test_email/data/msg_15.txt b/Lib/test/test_email/data/msg_15.txt new file mode 100644 index 00000000000..0025624e750 --- /dev/null +++ b/Lib/test/test_email/data/msg_15.txt @@ -0,0 +1,52 @@ +Return-Path: +Received: from fepD.post.tele.dk (195.41.46.149) by mail.groupcare.dk (LSMTP for Windows NT v1.1b) with SMTP id <0.0014F8A2@mail.groupcare.dk>; Mon, 30 Apr 2001 12:17:50 +0200 +User-Agent: Microsoft-Outlook-Express-Macintosh-Edition/5.02.2106 +Subject: XX +From: xx@xx.dk +To: XX +Message-ID: +Mime-version: 1.0 +Content-type: multipart/mixed; + boundary="MS_Mac_OE_3071477847_720252_MIME_Part" + +> Denne meddelelse er i MIME-format. Da dit postl + +--MS_Mac_OE_3071477847_720252_MIME_Part +Content-type: multipart/alternative; + boundary="MS_Mac_OE_3071477847_720252_MIME_Part" + + +--MS_Mac_OE_3071477847_720252_MIME_Part +Content-type: text/plain; charset="ISO-8859-1" +Content-transfer-encoding: quoted-printable + +Some removed test. + +--MS_Mac_OE_3071477847_720252_MIME_Part +Content-type: text/html; charset="ISO-8859-1" +Content-transfer-encoding: quoted-printable + + + +Some removed HTML + + +Some removed text. + + + + +--MS_Mac_OE_3071477847_720252_MIME_Part-- + + +--MS_Mac_OE_3071477847_720252_MIME_Part +Content-type: image/gif; name="xx.gif"; + x-mac-creator="6F676C65"; + x-mac-type="47494666" +Content-disposition: attachment +Content-transfer-encoding: base64 + +Some removed base64 encoded chars. + +--MS_Mac_OE_3071477847_720252_MIME_Part-- + diff --git a/Lib/test/test_email/data/msg_16.txt b/Lib/test/test_email/data/msg_16.txt new file mode 100644 index 00000000000..56167e9f5b4 --- /dev/null +++ b/Lib/test/test_email/data/msg_16.txt @@ -0,0 +1,123 @@ +Return-Path: <> +Delivered-To: scr-admin@socal-raves.org +Received: from cougar.noc.ucla.edu (cougar.noc.ucla.edu [169.232.10.18]) + by babylon.socal-raves.org (Postfix) with ESMTP id CCC2C51B84 + for ; Sun, 23 Sep 2001 20:13:54 -0700 (PDT) +Received: from sims-ms-daemon by cougar.noc.ucla.edu + (Sun Internet Mail Server sims.3.5.2000.03.23.18.03.p10) + id <0GK500B01D0B8Y@cougar.noc.ucla.edu> for scr-admin@socal-raves.org; Sun, + 23 Sep 2001 20:14:35 -0700 (PDT) +Received: from cougar.noc.ucla.edu + (Sun Internet Mail Server sims.3.5.2000.03.23.18.03.p10) + id <0GK500B01D0B8X@cougar.noc.ucla.edu>; Sun, 23 Sep 2001 20:14:35 -0700 (PDT) +Date: Sun, 23 Sep 2001 20:14:35 -0700 (PDT) +From: Internet Mail Delivery +Subject: Delivery Notification: Delivery has failed +To: scr-admin@socal-raves.org +Message-id: <0GK500B04D0B8X@cougar.noc.ucla.edu> +MIME-version: 1.0 +Sender: scr-owner@socal-raves.org +Errors-To: scr-owner@socal-raves.org +X-BeenThere: scr@socal-raves.org +X-Mailman-Version: 2.1a3 +Precedence: bulk +List-Help: +List-Post: +List-Subscribe: , + +List-Id: SoCal-Raves +List-Unsubscribe: , + +List-Archive: +Content-Type: multipart/report; boundary="Boundary_(ID_PGS2F2a+z+/jL7hupKgRhA)" + + +--Boundary_(ID_PGS2F2a+z+/jL7hupKgRhA) +Content-type: text/plain; charset=ISO-8859-1 + +This report relates to a message you sent with the following header fields: + + Message-id: <002001c144a6$8752e060$56104586@oxy.edu> + Date: Sun, 23 Sep 2001 20:10:55 -0700 + From: "Ian T. Henry" + To: SoCal Raves + Subject: [scr] yeah for Ians!! + +Your message cannot be delivered to the following recipients: + + Recipient address: jangel1@cougar.noc.ucla.edu + Reason: recipient reached disk quota + + +--Boundary_(ID_PGS2F2a+z+/jL7hupKgRhA) +Content-type: message/DELIVERY-STATUS + +Original-envelope-id: 0GK500B4HD0888@cougar.noc.ucla.edu +Reporting-MTA: dns; cougar.noc.ucla.edu + +Action: failed +Status: 5.0.0 (recipient reached disk quota) +Original-recipient: rfc822;jangel1@cougar.noc.ucla.edu +Final-recipient: rfc822;jangel1@cougar.noc.ucla.edu + +--Boundary_(ID_PGS2F2a+z+/jL7hupKgRhA) +Content-type: MESSAGE/RFC822 + +Return-path: scr-admin@socal-raves.org +Received: from sims-ms-daemon by cougar.noc.ucla.edu + (Sun Internet Mail Server sims.3.5.2000.03.23.18.03.p10) + id <0GK500B01D0B8X@cougar.noc.ucla.edu>; Sun, 23 Sep 2001 20:14:35 -0700 (PDT) +Received: from panther.noc.ucla.edu by cougar.noc.ucla.edu + (Sun Internet Mail Server sims.3.5.2000.03.23.18.03.p10) + with ESMTP id <0GK500B4GD0888@cougar.noc.ucla.edu> for jangel1@sims-ms-daemon; + Sun, 23 Sep 2001 20:14:33 -0700 (PDT) +Received: from babylon.socal-raves.org + (ip-209-85-222-117.dreamhost.com [209.85.222.117]) + by panther.noc.ucla.edu (8.9.1a/8.9.1) with ESMTP id UAA09793 for + ; Sun, 23 Sep 2001 20:14:32 -0700 (PDT) +Received: from babylon (localhost [127.0.0.1]) by babylon.socal-raves.org + (Postfix) with ESMTP id D3B2951B70; Sun, 23 Sep 2001 20:13:47 -0700 (PDT) +Received: by babylon.socal-raves.org (Postfix, from userid 60001) + id A611F51B82; Sun, 23 Sep 2001 20:13:46 -0700 (PDT) +Received: from tiger.cc.oxy.edu (tiger.cc.oxy.edu [134.69.3.112]) + by babylon.socal-raves.org (Postfix) with ESMTP id ADA7351B70 for + ; Sun, 23 Sep 2001 20:13:44 -0700 (PDT) +Received: from ent (n16h86.dhcp.oxy.edu [134.69.16.86]) + by tiger.cc.oxy.edu (8.8.8/8.8.8) with SMTP id UAA08100 for + ; Sun, 23 Sep 2001 20:14:24 -0700 (PDT) +Date: Sun, 23 Sep 2001 20:10:55 -0700 +From: "Ian T. Henry" +Subject: [scr] yeah for Ians!! +Sender: scr-admin@socal-raves.org +To: SoCal Raves +Errors-to: scr-admin@socal-raves.org +Message-id: <002001c144a6$8752e060$56104586@oxy.edu> +MIME-version: 1.0 +X-Mailer: Microsoft Outlook Express 5.50.4522.1200 +Content-type: text/plain; charset=us-ascii +Precedence: bulk +Delivered-to: scr-post@babylon.socal-raves.org +Delivered-to: scr@socal-raves.org +X-Converted-To-Plain-Text: from multipart/alternative by demime 0.98e +X-Converted-To-Plain-Text: Alternative section used was text/plain +X-BeenThere: scr@socal-raves.org +X-Mailman-Version: 2.1a3 +List-Help: +List-Post: +List-Subscribe: , + +List-Id: SoCal-Raves +List-Unsubscribe: , + +List-Archive: + +I always love to find more Ian's that are over 3 years old!! + +Ian +_______________________________________________ +For event info, list questions, or to unsubscribe, see http://www.socal-raves.org/ + + + +--Boundary_(ID_PGS2F2a+z+/jL7hupKgRhA)-- + diff --git a/Lib/test/test_email/data/msg_17.txt b/Lib/test/test_email/data/msg_17.txt new file mode 100644 index 00000000000..8d86e4180dd --- /dev/null +++ b/Lib/test/test_email/data/msg_17.txt @@ -0,0 +1,12 @@ +MIME-Version: 1.0 +From: Barry +To: Dingus Lovers +Subject: Here is your dingus fish +Date: Fri, 20 Apr 2001 19:35:02 -0400 +Content-Type: multipart/mixed; boundary="BOUNDARY" + +Hi there, + +This is the dingus fish. + +[Non-text (image/gif) part of message omitted, filename dingusfish.gif] diff --git a/Lib/test/test_email/data/msg_18.txt b/Lib/test/test_email/data/msg_18.txt new file mode 100644 index 00000000000..f9f4904d366 --- /dev/null +++ b/Lib/test/test_email/data/msg_18.txt @@ -0,0 +1,6 @@ +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit +X-Foobar-Spoink-Defrobnit: wasnipoop; giraffes="very-long-necked-animals"; + spooge="yummy"; hippos="gargantuan"; marshmallows="gooey" + diff --git a/Lib/test/test_email/data/msg_19.txt b/Lib/test/test_email/data/msg_19.txt new file mode 100644 index 00000000000..49bf7fccdd9 --- /dev/null +++ b/Lib/test/test_email/data/msg_19.txt @@ -0,0 +1,43 @@ +Send Ppp mailing list submissions to + ppp@zzz.org + +To subscribe or unsubscribe via the World Wide Web, visit + http://www.zzz.org/mailman/listinfo/ppp +or, via email, send a message with subject or body 'help' to + ppp-request@zzz.org + +You can reach the person managing the list at + ppp-admin@zzz.org + +When replying, please edit your Subject line so it is more specific +than "Re: Contents of Ppp digest..." + +Today's Topics: + + 1. testing #1 (Barry A. Warsaw) + 2. testing #2 (Barry A. Warsaw) + 3. testing #3 (Barry A. Warsaw) + 4. testing #4 (Barry A. Warsaw) + 5. testing #5 (Barry A. Warsaw) + +hello + + +hello + + +hello + + +hello + + +hello + + + +_______________________________________________ +Ppp mailing list +Ppp@zzz.org +http://www.zzz.org/mailman/listinfo/ppp + diff --git a/Lib/test/test_email/data/msg_20.txt b/Lib/test/test_email/data/msg_20.txt new file mode 100644 index 00000000000..1a6a88783ee --- /dev/null +++ b/Lib/test/test_email/data/msg_20.txt @@ -0,0 +1,22 @@ +Return-Path: +Delivered-To: bbb@zzz.org +Received: by mail.zzz.org (Postfix, from userid 889) + id 27CEAD38CC; Fri, 4 May 2001 14:05:44 -0400 (EDT) +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Transfer-Encoding: 7bit +Message-ID: <15090.61304.110929.45684@aaa.zzz.org> +From: bbb@ddd.com (John X. Doe) +To: bbb@zzz.org +Cc: ccc@zzz.org +CC: ddd@zzz.org +cc: eee@zzz.org +Subject: This is a test message +Date: Fri, 4 May 2001 14:05:44 -0400 + + +Hi, + +Do you like this message? + +-Me diff --git a/Lib/test/test_email/data/msg_21.txt b/Lib/test/test_email/data/msg_21.txt new file mode 100644 index 00000000000..23590b255dd --- /dev/null +++ b/Lib/test/test_email/data/msg_21.txt @@ -0,0 +1,20 @@ +From: aperson@dom.ain +To: bperson@dom.ain +Subject: Test +Content-Type: multipart/mixed; boundary="BOUNDARY" + +MIME message +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit + +One +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit + +Two +--BOUNDARY-- +End of MIME message diff --git a/Lib/test/test_email/data/msg_22.txt b/Lib/test/test_email/data/msg_22.txt new file mode 100644 index 00000000000..af9de5fa27b --- /dev/null +++ b/Lib/test/test_email/data/msg_22.txt @@ -0,0 +1,46 @@ +Mime-Version: 1.0 +Message-Id: +Date: Tue, 16 Oct 2001 13:59:25 +0300 +To: a@example.com +From: b@example.com +Content-Type: multipart/mixed; boundary="============_-1208892523==_============" + +--============_-1208892523==_============ +Content-Type: text/plain; charset="us-ascii" ; format="flowed" + +Text text text. +--============_-1208892523==_============ +Content-Id: +Content-Type: image/jpeg; name="wibble.JPG" + ; x-mac-type="4A504547" + ; x-mac-creator="474B4F4E" +Content-Disposition: attachment; filename="wibble.JPG" +Content-Transfer-Encoding: base64 + +/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEB +AQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQH/wAALCAXABIEBAREA +g6bCjjw/pIZSjO6FWFpldjySOmCNrO7DBZibUXhTwtCixw+GtAijVdqxxaPp0aKvmGXa +qrbBQvms0mAMeYS/3iTV1dG0hHaRNK01XblnWxtVdjkHLMIgTyqnk9VB7CrP2KzIINpa +4O7I+zxYO9WV8jZg71Zlb+8rMDkEirAVQFAUAKAFAAAUAYAUDgADgY6DjpRtXj5RxjHA +4wQRj0wQCMdCAewpaKKK/9k= +--============_-1208892523==_============ +Content-Id: +Content-Type: image/jpeg; name="wibble2.JPG" + ; x-mac-type="4A504547" + ; x-mac-creator="474B4F4E" +Content-Disposition: attachment; filename="wibble2.JPG" +Content-Transfer-Encoding: base64 + +/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEB +AQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQH/wAALCAXABJ0BAREA +/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQA +W6NFJJBEkU10kKGTcWMDwxuU+0JHvk8qAtOpNwqSR0n8c3BlDyXHlqsUltHEiTvdXLxR +7vMiGDNJAJWkAMk8ZkCFp5G2oo5W++INrbQtNfTQxJAuXlupz9oS4d5Y1W+E2XlWZJJE +Y7LWYQxTLE1zuMbfBPxw8X2fibVdIbSbI6nLZxX635t9TjtYreWR7WGKJTLJFFKSlozO +0ShxIXM43uC3/9k= +--============_-1208892523==_============ +Content-Type: text/plain; charset="us-ascii" ; format="flowed" + +Text text text. +--============_-1208892523==_============-- + diff --git a/Lib/test/test_email/data/msg_23.txt b/Lib/test/test_email/data/msg_23.txt new file mode 100644 index 00000000000..bb2e8ec36bb --- /dev/null +++ b/Lib/test/test_email/data/msg_23.txt @@ -0,0 +1,8 @@ +From: aperson@dom.ain +Content-Type: multipart/mixed; boundary="BOUNDARY" + +--BOUNDARY +Content-Type: text/plain + +A message part +--BOUNDARY-- diff --git a/Lib/test/test_email/data/msg_24.txt b/Lib/test/test_email/data/msg_24.txt new file mode 100644 index 00000000000..4e52339e86d --- /dev/null +++ b/Lib/test/test_email/data/msg_24.txt @@ -0,0 +1,10 @@ +Content-Type: multipart/mixed; boundary="BOUNDARY" +MIME-Version: 1.0 +Subject: A subject +To: aperson@dom.ain +From: bperson@dom.ain + +--BOUNDARY + + +--BOUNDARY-- diff --git a/Lib/test/test_email/data/msg_25.txt b/Lib/test/test_email/data/msg_25.txt new file mode 100644 index 00000000000..9e35275fe0d --- /dev/null +++ b/Lib/test/test_email/data/msg_25.txt @@ -0,0 +1,117 @@ +From MAILER-DAEMON Fri Apr 06 16:46:09 2001 +Received: from [204.245.199.98] (helo=zinfandel.lacita.com) + by www.linux.org.uk with esmtp (Exim 3.13 #1) + id 14lYR6-0008Iv-00 + for linuxuser-admin@www.linux.org.uk; Fri, 06 Apr 2001 16:46:09 +0100 +Received: from localhost (localhost) by zinfandel.lacita.com (8.7.3/8.6.10-MT4.00) with internal id JAB03225; Fri, 6 Apr 2001 09:23:06 -0800 (GMT-0800) +Date: Fri, 6 Apr 2001 09:23:06 -0800 (GMT-0800) +From: Mail Delivery Subsystem +Subject: Returned mail: Too many hops 19 (17 max): from via [199.164.235.226], to +Message-Id: <200104061723.JAB03225@zinfandel.lacita.com> +To: +To: postmaster@zinfandel.lacita.com +MIME-Version: 1.0 +Content-Type: multipart/report; report-type=delivery-status; + bo +Auto-Submitted: auto-generated (failure) + +This is a MIME-encapsulated message + +--JAB03225.986577786/zinfandel.lacita.com + +The original message was received at Fri, 6 Apr 2001 09:23:03 -0800 (GMT-0800) +from [199.164.235.226] + + ----- The following addresses have delivery notifications ----- + (unrecoverable error) + + ----- Transcript of session follows ----- +554 Too many hops 19 (17 max): from via [199.164.235.226], to + +--JAB03225.986577786/zinfandel.lacita.com +Content-Type: message/delivery-status + +Reporting-MTA: dns; zinfandel.lacita.com +Received-From-MTA: dns; [199.164.235.226] +Arrival-Date: Fri, 6 Apr 2001 09:23:03 -0800 (GMT-0800) + +Final-Recipient: rfc822; scoffman@wellpartner.com +Action: failed +Status: 5.4.6 +Last-Attempt-Date: Fri, 6 Apr 2001 09:23:06 -0800 (GMT-0800) + +--JAB03225.986577786/zinfandel.lacita.com +Content-Type: text/rfc822-headers + +Return-Path: linuxuser-admin@www.linux.org.uk +Received: from ns1.wellpartner.net ([199.164.235.226]) by zinfandel.lacita.com (8.7.3/8.6.10-MT4.00) with ESMTP id JAA03225 for ; Fri, 6 Apr 2001 09:23:03 -0800 (GMT-0800) +Received: from zinfandel.lacita.com ([204.245.199.98]) + by + fo +Received: from ns1.wellpartner.net ([199.164.235.226]) by zinfandel.lacita.com (8.7.3/8.6.10-MT4.00) with ESMTP id JAA03221 for ; Fri, 6 Apr 2001 09:22:18 -0800 (GMT-0800) +Received: from zinfandel.lacita.com ([204.245.199.98]) + by + fo +Received: from ns1.wellpartner.net ([199.164.235.226]) by zinfandel.lacita.com (8.7.3/8.6.10-MT4.00) with ESMTP id JAA03217 for ; Fri, 6 Apr 2001 09:21:37 -0800 (GMT-0800) +Received: from zinfandel.lacita.com ([204.245.199.98]) + by + fo +Received: from ns1.wellpartner.net ([199.164.235.226]) by zinfandel.lacita.com (8.7.3/8.6.10-MT4.00) with ESMTP id JAA03213 for ; Fri, 6 Apr 2001 09:20:56 -0800 (GMT-0800) +Received: from zinfandel.lacita.com ([204.245.199.98]) + by + fo +Received: from ns1.wellpartner.net ([199.164.235.226]) by zinfandel.lacita.com (8.7.3/8.6.10-MT4.00) with ESMTP id JAA03209 for ; Fri, 6 Apr 2001 09:20:15 -0800 (GMT-0800) +Received: from zinfandel.lacita.com ([204.245.199.98]) + by + fo +Received: from ns1.wellpartner.net ([199.164.235.226]) by zinfandel.lacita.com (8.7.3/8.6.10-MT4.00) with ESMTP id JAA03205 for ; Fri, 6 Apr 2001 09:19:33 -0800 (GMT-0800) +Received: from zinfandel.lacita.com ([204.245.199.98]) + by + fo +Received: from ns1.wellpartner.net ([199.164.235.226]) by zinfandel.lacita.com (8.7.3/8.6.10-MT4.00) with ESMTP id JAA03201 for ; Fri, 6 Apr 2001 09:18:52 -0800 (GMT-0800) +Received: from zinfandel.lacita.com ([204.245.199.98]) + by + fo +Received: from ns1.wellpartner.net ([199.164.235.226]) by zinfandel.lacita.com (8.7.3/8.6.10-MT4.00) with ESMTP id JAA03197 for ; Fri, 6 Apr 2001 09:17:54 -0800 (GMT-0800) +Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) + by + fo +Received: from localhost.localdomain + ([ + by + id +Received: from [212.1.130.11] (helo=s1.uklinux.net ident=root) + by + id + fo +Received: from server (ppp-2-22.cvx4.telinco.net [212.1.149.22]) + by + fo +From: Daniel James +Organization: LinuxUser +To: linuxuser@www.linux.org.uk +X-Mailer: KMail [version 1.1.99] +Content-Type: text/plain; + c +MIME-Version: 1.0 +Message-Id: <01040616033903.00962@server> +Content-Transfer-Encoding: 8bit +Subject: [LinuxUser] bulletin no. 45 +Sender: linuxuser-admin@www.linux.org.uk +Errors-To: linuxuser-admin@www.linux.org.uk +X-BeenThere: linuxuser@www.linux.org.uk +X-Mailman-Version: 2.0.3 +Precedence: bulk +List-Help: +List-Post: +List-Subscribe: , + +List-Unsubscribe: , + +Date: Fri, 6 Apr 2001 16:03:39 +0100 + +--JAB03225.986577786/zinfandel.lacita.com-- + + diff --git a/Lib/test/test_email/data/msg_26.txt b/Lib/test/test_email/data/msg_26.txt new file mode 100644 index 00000000000..58efaa9c9a8 --- /dev/null +++ b/Lib/test/test_email/data/msg_26.txt @@ -0,0 +1,46 @@ +Received: from xcar [192.168.0.2] by jeeves.wooster.local + (SMTPD32-7.07 EVAL) id AFF92F0214; Sun, 12 May 2002 08:55:37 +0100 +Date: Sun, 12 May 2002 08:56:15 +0100 +From: Father Time +To: timbo@jeeves.wooster.local +Subject: IMAP file test +Message-ID: <6df65d354b.father.time@rpc.wooster.local> +X-Organization: Home +User-Agent: Messenger-Pro/2.50a (MsgServe/1.50) (RISC-OS/4.02) POPstar/2.03 +MIME-Version: 1.0 +Content-Type: multipart/mixed; boundary="1618492860--2051301190--113853680" +Status: R +X-UIDL: 319998302 + +This message is in MIME format which your mailer apparently does not support. +You either require a newer version of your software which supports MIME, or +a separate MIME decoding utility. Alternatively, ask the sender of this +message to resend it in a different format. + +--1618492860--2051301190--113853680 +Content-Type: text/plain; charset=us-ascii + +Simple email with attachment. + + +--1618492860--2051301190--113853680 +Content-Type: application/riscos; name="clock.bmp,69c"; type=BMP; + load=&fff69c4b; exec=&355dd4d1; access=&03 +Content-Disposition: attachment; filename="clock.bmp" +Content-Transfer-Encoding: base64 + +Qk12AgAAAAAAAHYAAAAoAAAAIAAAACAAAAABAAQAAAAAAAAAAADXDQAA1w0AAAAAAAAA +AAAAAAAAAAAAiAAAiAAAAIiIAIgAAACIAIgAiIgAALu7uwCIiIgAERHdACLuIgAz//8A +zAAAAN0R3QDu7iIA////AAAAAAAAAAAAAAAAAAAAAAAAAAi3AAAAAAAAADeAAAAAAAAA +C3ADMzMzMANwAAAAAAAAAAAHMAAAAANwAAAAAAAAAACAMAd3zPfwAwgAAAAAAAAIAwd/ +f8x/f3AwgAAAAAAAgDB0x/f3//zPAwgAAAAAAAcHfM9////8z/AwAAAAAAiwd/f3//// +////A4AAAAAAcEx/f///////zAMAAAAAiwfM9////3///8zwOAAAAAcHf3////B///// +8DAAAAALB/f3///wd3d3//AwAAAABwTPf//wCQAAD/zAMAAAAAsEx/f///B////8wDAA +AAAHB39////wf/////AwAAAACwf39///8H/////wMAAAAIcHfM9///B////M8DgAAAAA +sHTH///wf///xAMAAAAACHB3f3//8H////cDgAAAAAALB3zH//D//M9wMAAAAAAAgLB0 +z39///xHAwgAAAAAAAgLB3d3RHd3cDCAAAAAAAAAgLAHd0R3cAMIAAAAAAAAgAgLcAAA +AAMwgAgAAAAACDAAAAu7t7cwAAgDgAAAAABzcIAAAAAAAAgDMwAAAAAAN7uwgAAAAAgH +MzMAAAAACH97tzAAAAALu3c3gAAAAAAL+7tzDABAu7f7cAAAAAAACA+3MA7EQAv/sIAA +AAAAAAAIAAAAAAAAAIAAAAAA + +--1618492860--2051301190--113853680-- diff --git a/Lib/test/test_email/data/msg_27.txt b/Lib/test/test_email/data/msg_27.txt new file mode 100644 index 00000000000..d0191769d73 --- /dev/null +++ b/Lib/test/test_email/data/msg_27.txt @@ -0,0 +1,15 @@ +Return-Path: +Received: by mail.dom.ain (Postfix, from userid 889) + id B9D0AD35DB; Tue, 4 Jun 2002 21:46:59 -0400 (EDT) +Message-ID: <15613.28051.707126.569693@dom.ain> +Date: Tue, 4 Jun 2002 21:46:59 -0400 +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Transfer-Encoding: 7bit +Subject: bug demonstration + 12345678911234567892123456789312345678941234567895123456789612345678971234567898112345678911234567892123456789112345678911234567892123456789 + more text +From: aperson@dom.ain (Anne P. Erson) +To: bperson@dom.ain (Barney P. Erson) + +test diff --git a/Lib/test/test_email/data/msg_28.txt b/Lib/test/test_email/data/msg_28.txt new file mode 100644 index 00000000000..1e4824cabb0 --- /dev/null +++ b/Lib/test/test_email/data/msg_28.txt @@ -0,0 +1,25 @@ +From: aperson@dom.ain +MIME-Version: 1.0 +Content-Type: multipart/digest; boundary=BOUNDARY + +--BOUNDARY +Content-Type: message/rfc822 + +Content-Type: text/plain; charset=us-ascii +To: aa@bb.org +From: cc@dd.org +Subject: ee + +message 1 + +--BOUNDARY +Content-Type: message/rfc822 + +Content-Type: text/plain; charset=us-ascii +To: aa@bb.org +From: cc@dd.org +Subject: ee + +message 2 + +--BOUNDARY-- diff --git a/Lib/test/test_email/data/msg_29.txt b/Lib/test/test_email/data/msg_29.txt new file mode 100644 index 00000000000..1fab5616173 --- /dev/null +++ b/Lib/test/test_email/data/msg_29.txt @@ -0,0 +1,22 @@ +Return-Path: +Delivered-To: bbb@zzz.org +Received: by mail.zzz.org (Postfix, from userid 889) + id 27CEAD38CC; Fri, 4 May 2001 14:05:44 -0400 (EDT) +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii; + title*0*="us-ascii'en'This%20is%20even%20more%20"; + title*1*="%2A%2A%2Afun%2A%2A%2A%20"; + title*2="isn't it!" +Content-Transfer-Encoding: 7bit +Message-ID: <15090.61304.110929.45684@aaa.zzz.org> +From: bbb@ddd.com (John X. Doe) +To: bbb@zzz.org +Subject: This is a test message +Date: Fri, 4 May 2001 14:05:44 -0400 + + +Hi, + +Do you like this message? + +-Me diff --git a/Lib/test/test_email/data/msg_30.txt b/Lib/test/test_email/data/msg_30.txt new file mode 100644 index 00000000000..4334bb6e608 --- /dev/null +++ b/Lib/test/test_email/data/msg_30.txt @@ -0,0 +1,23 @@ +From: aperson@dom.ain +MIME-Version: 1.0 +Content-Type: multipart/digest; boundary=BOUNDARY + +--BOUNDARY + +Content-Type: text/plain; charset=us-ascii +To: aa@bb.org +From: cc@dd.org +Subject: ee + +message 1 + +--BOUNDARY + +Content-Type: text/plain; charset=us-ascii +To: aa@bb.org +From: cc@dd.org +Subject: ee + +message 2 + +--BOUNDARY-- diff --git a/Lib/test/test_email/data/msg_31.txt b/Lib/test/test_email/data/msg_31.txt new file mode 100644 index 00000000000..1e58e56cf52 --- /dev/null +++ b/Lib/test/test_email/data/msg_31.txt @@ -0,0 +1,15 @@ +From: aperson@dom.ain +MIME-Version: 1.0 +Content-Type: multipart/mixed; boundary=BOUNDARY_ + +--BOUNDARY +Content-Type: text/plain + +message 1 + +--BOUNDARY +Content-Type: text/plain + +message 2 + +--BOUNDARY-- diff --git a/Lib/test/test_email/data/msg_32.txt b/Lib/test/test_email/data/msg_32.txt new file mode 100644 index 00000000000..07ec5af9a3d --- /dev/null +++ b/Lib/test/test_email/data/msg_32.txt @@ -0,0 +1,14 @@ +Delivered-To: freebsd-isp@freebsd.org +Date: Tue, 26 Sep 2000 12:23:03 -0500 +From: Anne Person +To: Barney Dude +Subject: Re: Limiting Perl CPU Utilization... +Mime-Version: 1.0 +Content-Type: text/plain; charset*=ansi-x3.4-1968''us-ascii +Content-Disposition: inline +User-Agent: Mutt/1.3.8i +Sender: owner-freebsd-isp@FreeBSD.ORG +Precedence: bulk +X-Loop: FreeBSD.org + +Some message. diff --git a/Lib/test/test_email/data/msg_33.txt b/Lib/test/test_email/data/msg_33.txt new file mode 100644 index 00000000000..042787a4fd8 --- /dev/null +++ b/Lib/test/test_email/data/msg_33.txt @@ -0,0 +1,29 @@ +Delivered-To: freebsd-isp@freebsd.org +Date: Wed, 27 Sep 2000 11:11:09 -0500 +From: Anne Person +To: Barney Dude +Subject: Re: Limiting Perl CPU Utilization... +Mime-Version: 1.0 +Content-Type: multipart/signed; micalg*=ansi-x3.4-1968''pgp-md5; + protocol*=ansi-x3.4-1968''application%2Fpgp-signature; + boundary*="ansi-x3.4-1968''EeQfGwPcQSOJBaQU" +Content-Disposition: inline +Sender: owner-freebsd-isp@FreeBSD.ORG +Precedence: bulk +X-Loop: FreeBSD.org + + +--EeQfGwPcQSOJBaQU +Content-Type: text/plain; charset*=ansi-x3.4-1968''us-ascii +Content-Disposition: inline +Content-Transfer-Encoding: quoted-printable + +part 1 + +--EeQfGwPcQSOJBaQU +Content-Type: text/plain +Content-Disposition: inline + +part 2 + +--EeQfGwPcQSOJBaQU-- diff --git a/Lib/test/test_email/data/msg_34.txt b/Lib/test/test_email/data/msg_34.txt new file mode 100644 index 00000000000..055dfea5310 --- /dev/null +++ b/Lib/test/test_email/data/msg_34.txt @@ -0,0 +1,19 @@ +From: aperson@dom.ain +To: bperson@dom.ain +Content-Type: multipart/digest; boundary=XYZ + +--XYZ +Content-Type: text/plain + + +This is a text plain part that is counter to recommended practice in +RFC 2046, $5.1.5, but is not illegal + +--XYZ + +From: cperson@dom.ain +To: dperson@dom.ain + +A submessage + +--XYZ-- diff --git a/Lib/test/test_email/data/msg_35.txt b/Lib/test/test_email/data/msg_35.txt new file mode 100644 index 00000000000..0e2bbcaf718 --- /dev/null +++ b/Lib/test/test_email/data/msg_35.txt @@ -0,0 +1,4 @@ +From: aperson@dom.ain +To: bperson@dom.ain +Subject: here's something interesting +counter to RFC 5322, there's no separating newline here diff --git a/Lib/test/test_email/data/msg_36.txt b/Lib/test/test_email/data/msg_36.txt new file mode 100644 index 00000000000..5632c3062c9 --- /dev/null +++ b/Lib/test/test_email/data/msg_36.txt @@ -0,0 +1,40 @@ +Mime-Version: 1.0 +Content-Type: Multipart/Mixed; Boundary="NextPart" +To: IETF-Announce:; +From: Internet-Drafts@ietf.org +Subject: I-D ACTION:draft-ietf-mboned-mix-00.txt +Date: Tue, 22 Dec 1998 16:55:06 -0500 + +--NextPart + +Blah blah blah + +--NextPart +Content-Type: Multipart/Alternative; Boundary="OtherAccess" + +--OtherAccess +Content-Type: Message/External-body; + access-type="mail-server"; + server="mailserv@ietf.org" + +Content-Type: text/plain +Content-ID: <19981222151406.I-D@ietf.org> + +ENCODING mime +FILE /internet-drafts/draft-ietf-mboned-mix-00.txt + +--OtherAccess +Content-Type: Message/External-body; + name="draft-ietf-mboned-mix-00.txt"; + site="ftp.ietf.org"; + access-type="anon-ftp"; + directory="internet-drafts" + +Content-Type: text/plain +Content-ID: <19981222151406.I-D@ietf.org> + + +--OtherAccess-- + +--NextPart-- + diff --git a/Lib/test/test_email/data/msg_37.txt b/Lib/test/test_email/data/msg_37.txt new file mode 100644 index 00000000000..038d34a1a42 --- /dev/null +++ b/Lib/test/test_email/data/msg_37.txt @@ -0,0 +1,22 @@ +Content-Type: multipart/mixed; boundary=ABCDE + +--ABCDE +Content-Type: text/x-one + +Blah + +--ABCDE +--ABCDE +Content-Type: text/x-two + +Blah + +--ABCDE +--ABCDE +--ABCDE +--ABCDE +Content-Type: text/x-two + +Blah + +--ABCDE-- diff --git a/Lib/test/test_email/data/msg_38.txt b/Lib/test/test_email/data/msg_38.txt new file mode 100644 index 00000000000..006df81cb59 --- /dev/null +++ b/Lib/test/test_email/data/msg_38.txt @@ -0,0 +1,101 @@ +MIME-Version: 1.0 +Content-Type: multipart/mixed; boundary="----- =_aaaaaaaaaa0" + +------- =_aaaaaaaaaa0 +Content-Type: multipart/mixed; boundary="----- =_aaaaaaaaaa1" +Content-ID: <20592.1022586929.1@example.com> + +------- =_aaaaaaaaaa1 +Content-Type: multipart/alternative; boundary="----- =_aaaaaaaaaa2" +Content-ID: <20592.1022586929.2@example.com> + +------- =_aaaaaaaaaa2 +Content-Type: text/plain +Content-ID: <20592.1022586929.3@example.com> +Content-Description: very tricky +Content-Transfer-Encoding: 7bit + + +Unlike the test test_nested-multiples-with-internal-boundary, this +piece of text not only contains the outer boundary tags +------- =_aaaaaaaaaa1 +and +------- =_aaaaaaaaaa0 +but puts them at the start of a line! And, to be even nastier, it +even includes a couple of end tags, such as this one: + +------- =_aaaaaaaaaa1-- + +and this one, which is from a multipart we haven't even seen yet! + +------- =_aaaaaaaaaa4-- + +This will, I'm sure, cause much breakage of MIME parsers. But, as +far as I can tell, it's perfectly legal. I have not yet ever seen +a case of this in the wild, but I've seen *similar* things. + + +------- =_aaaaaaaaaa2 +Content-Type: application/octet-stream +Content-ID: <20592.1022586929.4@example.com> +Content-Description: patch2 +Content-Transfer-Encoding: base64 + +XXX + +------- =_aaaaaaaaaa2-- + +------- =_aaaaaaaaaa1 +Content-Type: multipart/alternative; boundary="----- =_aaaaaaaaaa3" +Content-ID: <20592.1022586929.6@example.com> + +------- =_aaaaaaaaaa3 +Content-Type: application/octet-stream +Content-ID: <20592.1022586929.7@example.com> +Content-Description: patch3 +Content-Transfer-Encoding: base64 + +XXX + +------- =_aaaaaaaaaa3 +Content-Type: application/octet-stream +Content-ID: <20592.1022586929.8@example.com> +Content-Description: patch4 +Content-Transfer-Encoding: base64 + +XXX + +------- =_aaaaaaaaaa3-- + +------- =_aaaaaaaaaa1 +Content-Type: multipart/alternative; boundary="----- =_aaaaaaaaaa4" +Content-ID: <20592.1022586929.10@example.com> + +------- =_aaaaaaaaaa4 +Content-Type: application/octet-stream +Content-ID: <20592.1022586929.11@example.com> +Content-Description: patch5 +Content-Transfer-Encoding: base64 + +XXX + +------- =_aaaaaaaaaa4 +Content-Type: application/octet-stream +Content-ID: <20592.1022586929.12@example.com> +Content-Description: patch6 +Content-Transfer-Encoding: base64 + +XXX + +------- =_aaaaaaaaaa4-- + +------- =_aaaaaaaaaa1-- + +------- =_aaaaaaaaaa0 +Content-Type: text/plain; charset="us-ascii" +Content-ID: <20592.1022586929.15@example.com> + +-- +It's never too late to have a happy childhood. + +------- =_aaaaaaaaaa0-- diff --git a/Lib/test/test_email/data/msg_39.txt b/Lib/test/test_email/data/msg_39.txt new file mode 100644 index 00000000000..124b2691927 --- /dev/null +++ b/Lib/test/test_email/data/msg_39.txt @@ -0,0 +1,83 @@ +MIME-Version: 1.0 +Content-Type: multipart/mixed; boundary="----- =_aaaaaaaaaa0" + +------- =_aaaaaaaaaa0 +Content-Type: multipart/mixed; boundary="----- =_aaaaaaaaaa1" +Content-ID: <20592.1022586929.1@example.com> + +------- =_aaaaaaaaaa1 +Content-Type: multipart/alternative; boundary="----- =_aaaaaaaaaa1" +Content-ID: <20592.1022586929.2@example.com> + +------- =_aaaaaaaaaa1 +Content-Type: application/octet-stream +Content-ID: <20592.1022586929.3@example.com> +Content-Description: patch1 +Content-Transfer-Encoding: base64 + +XXX + +------- =_aaaaaaaaaa1 +Content-Type: application/octet-stream +Content-ID: <20592.1022586929.4@example.com> +Content-Description: patch2 +Content-Transfer-Encoding: base64 + +XXX + +------- =_aaaaaaaaaa1-- + +------- =_aaaaaaaaaa1 +Content-Type: multipart/alternative; boundary="----- =_aaaaaaaaaa1" +Content-ID: <20592.1022586929.6@example.com> + +------- =_aaaaaaaaaa1 +Content-Type: application/octet-stream +Content-ID: <20592.1022586929.7@example.com> +Content-Description: patch3 +Content-Transfer-Encoding: base64 + +XXX + +------- =_aaaaaaaaaa1 +Content-Type: application/octet-stream +Content-ID: <20592.1022586929.8@example.com> +Content-Description: patch4 +Content-Transfer-Encoding: base64 + +XXX + +------- =_aaaaaaaaaa1-- + +------- =_aaaaaaaaaa1 +Content-Type: multipart/alternative; boundary="----- =_aaaaaaaaaa1" +Content-ID: <20592.1022586929.10@example.com> + +------- =_aaaaaaaaaa1 +Content-Type: application/octet-stream +Content-ID: <20592.1022586929.11@example.com> +Content-Description: patch5 +Content-Transfer-Encoding: base64 + +XXX + +------- =_aaaaaaaaaa1 +Content-Type: application/octet-stream +Content-ID: <20592.1022586929.12@example.com> +Content-Description: patch6 +Content-Transfer-Encoding: base64 + +XXX + +------- =_aaaaaaaaaa1-- + +------- =_aaaaaaaaaa1-- + +------- =_aaaaaaaaaa0 +Content-Type: text/plain; charset="us-ascii" +Content-ID: <20592.1022586929.15@example.com> + +-- +It's never too late to have a happy childhood. + +------- =_aaaaaaaaaa0-- diff --git a/Lib/test/test_email/data/msg_40.txt b/Lib/test/test_email/data/msg_40.txt new file mode 100644 index 00000000000..1435fa1e1a0 --- /dev/null +++ b/Lib/test/test_email/data/msg_40.txt @@ -0,0 +1,10 @@ +MIME-Version: 1.0 +Content-Type: text/html; boundary="--961284236552522269" + +----961284236552522269 +Content-Type: text/html; +Content-Transfer-Encoding: 7Bit + + + +----961284236552522269-- diff --git a/Lib/test/test_email/data/msg_41.txt b/Lib/test/test_email/data/msg_41.txt new file mode 100644 index 00000000000..76cdd1cb7f2 --- /dev/null +++ b/Lib/test/test_email/data/msg_41.txt @@ -0,0 +1,8 @@ +From: "Allison Dunlap" +To: yyy@example.com +Subject: 64423 +Date: Sun, 11 Jul 2004 16:09:27 -0300 +MIME-Version: 1.0 +Content-Type: multipart/alternative; + +Blah blah blah diff --git a/Lib/test/test_email/data/msg_42.txt b/Lib/test/test_email/data/msg_42.txt new file mode 100644 index 00000000000..a75f8f4a020 --- /dev/null +++ b/Lib/test/test_email/data/msg_42.txt @@ -0,0 +1,20 @@ +Content-Type: multipart/mixed; boundary="AAA" +From: Mail Delivery Subsystem +To: yyy@example.com + +This is a MIME-encapsulated message + +--AAA + +Stuff + +--AAA +Content-Type: message/rfc822 + +From: webmaster@python.org +To: zzz@example.com +Content-Type: multipart/mixed; boundary="BBB" + +--BBB-- + +--AAA-- diff --git a/Lib/test/test_email/data/msg_43.txt b/Lib/test/test_email/data/msg_43.txt new file mode 100644 index 00000000000..797d12c5688 --- /dev/null +++ b/Lib/test/test_email/data/msg_43.txt @@ -0,0 +1,217 @@ +From SRS0=aO/p=ON=bag.python.org=None@bounce2.pobox.com Fri Nov 26 21:40:36 2004 +X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] + [nil nil nil nil nil nil nil "MAILER DAEMON <>" "MAILER DAEMON <>" nil nil "Banned file: auto__mail.python.bat in mail from you" "^From:" nil nil nil nil "Banned file: auto__mail.python.bat in mail from you" nil nil nil nil nil nil nil] + nil) +MIME-Version: 1.0 +Message-Id: +Content-Type: multipart/report; report-type=delivery-status; + charset=utf-8; + boundary="----------=_1101526904-1956-5" +X-Virus-Scanned: by XS4ALL Virus Scanner +X-UIDL: 4\G!!! +To: +Subject: Banned file: auto__mail.python.bat in mail from you +Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +This is a multi-part message in MIME format... + +------------=_1101526904-1956-5 +Content-Type: text/plain; charset="utf-8" +Content-Disposition: inline +Content-Transfer-Encoding: 7bit + +BANNED FILENAME ALERT + +Your message to: xxxxxxx@dot.ca.gov, xxxxxxxxxxxxx@dot.ca.gov, xxxxxxxxxx@dot.ca.gov, xxxxxxxx@dot.ca.gov, xxxxxxxxxx@dot.ca.gov, xxxxxx@dot.ca.gov, xxxxxxxxxx@dot.ca.gov, xxxxxx@dot.ca.gov, xxxxxx@dot.ca.gov, xxxxxxxxxxxxxxxx@dot.ca.gov, xxxxxxxxxxx@dot.ca.gov, xxxxxxxxxx@dot.ca.gov, xxxxxxxxxx@dot.ca.gov, xxxxxxxxxxxx@dot.ca.gov, xxxxxxxxxxxx@dot.ca.gov, xxxxxxx@dot.ca.gov, xxxxxxxxx@dot.ca.gov, xxxxxxxxxx@dot.ca.gov, xxxxxx@dot.ca.gov, xxx@dot.ca.gov, xxxxxxx@dot.ca.gov, xxxxxxx@dot.ca.gov, xxxxxxxxxxxxxxx@dot.ca.gov, xxxxxxxxxx@dot.ca.gov, xxxxxxx@dot.ca.gov, xxx@dot.ca.gov, xxxxxxxx@dot.ca.gov, xxxxxxxxxxxxx@dot.ca.gov, xxxxxxxxxxxxx@dot.ca.gov, xxxxxxxxxxx@dot.ca.gov, xxxxxxxxx@dot.ca.gov, xxxxxxxxxx@dot.ca.gov, xxxxxxxxxxxx@dot.ca.gov, xxxxxxx@dot.ca.gov, xxxxxxxxxxxxxxx@dot.ca.gov, xxxxxxxxxxxxx@dot.ca.gov, xxxx@dot.ca.gov, xxxxxxxx@dot.ca.gov, xxxxxxxxxx@dot.ca.gov, xxxxxxxxxxxxxxxxxx@dot.ca.gov +was blocked by our Spam Firewall. The email you sent with the following subject has NOT BEEN DELIVERED: + +Subject: Delivery_failure_notice + +An attachment in that mail was of a file type that the Spam Firewall is set to block. + + + +------------=_1101526904-1956-5 +Content-Type: message/delivery-status +Content-Disposition: inline +Content-Transfer-Encoding: 7bit +Content-Description: Delivery error report + +Reporting-MTA: dns; sacspam01.dot.ca.gov +Received-From-MTA: smtp; sacspam01.dot.ca.gov ([127.0.0.1]) +Arrival-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxxxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxxxxxxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxxxxxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +Final-Recipient: rfc822; xxxxxxx@dot.ca.gov +Action: failed +Status: 5.7.1 +Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=01956-02-2 - BANNED: auto__mail.python.bat +Last-Attempt-Date: Fri, 26 Nov 2004 19:41:44 -0800 (PST) + +------------=_1101526904-1956-5 +Content-Type: text/rfc822-headers +Content-Disposition: inline +Content-Transfer-Encoding: 7bit +Content-Description: Undelivered-message headers + +Received: from kgsav.org (ppp-70-242-162-63.dsl.spfdmo.swbell.net [70.242.162.63]) + by sacspam01.dot.ca.gov (Spam Firewall) with SMTP + id A232AD03DE3A; Fri, 26 Nov 2004 19:41:35 -0800 (PST) +From: webmaster@python.org +To: xxxxx@dot.ca.gov +Date: Sat, 27 Nov 2004 03:35:30 UTC +Subject: Delivery_failure_notice +Importance: Normal +X-Priority: 3 (Normal) +X-MSMail-Priority: Normal +Message-ID: +MIME-Version: 1.0 +Content-Type: multipart/mixed; boundary="====67bd2b7a5.f99f7" +Content-Transfer-Encoding: 7bit + +------------=_1101526904-1956-5-- + diff --git a/Lib/test/test_email/data/msg_44.txt b/Lib/test/test_email/data/msg_44.txt new file mode 100644 index 00000000000..15a225287bd --- /dev/null +++ b/Lib/test/test_email/data/msg_44.txt @@ -0,0 +1,33 @@ +Return-Path: +Delivered-To: barry@python.org +Received: by mail.python.org (Postfix, from userid 889) + id C2BF0D37C6; Tue, 11 Sep 2001 00:05:05 -0400 (EDT) +MIME-Version: 1.0 +Content-Type: multipart/mixed; boundary="h90VIIIKmx" +Content-Transfer-Encoding: 7bit +Message-ID: <15261.36209.358846.118674@anthem.python.org> +From: barry@python.org (Barry A. Warsaw) +To: barry@python.org +Subject: a simple multipart +Date: Tue, 11 Sep 2001 00:05:05 -0400 +X-Mailer: VM 6.95 under 21.4 (patch 4) "Artificial Intelligence" XEmacs Lucid +X-Attribution: BAW +X-Oblique-Strategy: Make a door into a window + + +--h90VIIIKmx +Content-Type: text/plain; name="msg.txt" +Content-Transfer-Encoding: 7bit + +a simple kind of mirror +to reflect upon our own + +--h90VIIIKmx +Content-Type: text/plain; name="msg.txt" +Content-Transfer-Encoding: 7bit + +a simple kind of mirror +to reflect upon our own + +--h90VIIIKmx-- + diff --git a/Lib/test/test_email/data/msg_45.txt b/Lib/test/test_email/data/msg_45.txt new file mode 100644 index 00000000000..58fde956e71 --- /dev/null +++ b/Lib/test/test_email/data/msg_45.txt @@ -0,0 +1,33 @@ +From: +To: +Subject: test +X-Long-Line: Some really long line contains a lot of text and thus has to be rewrapped because it is some + really long + line +MIME-Version: 1.0 +Content-Type: multipart/signed; boundary="borderline"; + protocol="application/pgp-signature"; micalg=pgp-sha1 + +This is an OpenPGP/MIME signed message (RFC 2440 and 3156) +--borderline +Content-Type: text/plain +X-Long-Line: Another really long line contains a lot of text and thus has to be rewrapped because it is another + really long + line + +This is the signed contents. + +--borderline +Content-Type: application/pgp-signature; name="signature.asc" +Content-Description: OpenPGP digital signature +Content-Disposition: attachment; filename="signature.asc" + +-----BEGIN PGP SIGNATURE----- +Version: GnuPG v2.0.6 (GNU/Linux) + +iD8DBQFG03voRhp6o4m9dFsRApSZAKCCAN3IkJlVRg6NvAiMHlvvIuMGPQCeLZtj +FGwfnRHFBFO/S4/DKysm0lI= +=t7+s +-----END PGP SIGNATURE----- + +--borderline-- diff --git a/Lib/test/test_email/data/msg_46.txt b/Lib/test/test_email/data/msg_46.txt new file mode 100644 index 00000000000..1e22c4f600a --- /dev/null +++ b/Lib/test/test_email/data/msg_46.txt @@ -0,0 +1,23 @@ +Return-Path: +Delivery-Date: Mon, 08 Feb 2010 14:05:16 +0100 +Received: from example.org (example.org [64.5.53.58]) + by example.net (node=mxbap2) with ESMTP (Nemesis) + id UNIQUE for someone@example.com; Mon, 08 Feb 2010 14:05:16 +0100 +Date: Mon, 01 Feb 2010 12:21:16 +0100 +From: "Sender" +To: +Subject: GroupwiseForwardingTest +Mime-Version: 1.0 +Content-Type: message/rfc822 + +Return-path: +Message-ID: <4B66B890.4070408@teconcept.de> +Date: Mon, 01 Feb 2010 12:18:40 +0100 +From: "Dr. Sender" +MIME-Version: 1.0 +To: "Recipient" +Subject: GroupwiseForwardingTest +Content-Type: text/plain; charset=ISO-8859-15 +Content-Transfer-Encoding: 7bit + +Testing email forwarding with Groupwise 1.2.2010 diff --git a/Lib/test/test_email/data/msg_47.txt b/Lib/test/test_email/data/msg_47.txt new file mode 100644 index 00000000000..bb48b47d96b --- /dev/null +++ b/Lib/test/test_email/data/msg_47.txt @@ -0,0 +1,14 @@ +Date: 01 Jan 2001 00:01+0000 +From: arthur@example.example +MIME-Version: 1.0 +Content-Type: multipart/mixed; boundary=foo + +--foo +Content-Type: text/plain +bar + +--foo +Content-Type: text/html +

    baz

    + +--foo-- \ No newline at end of file diff --git a/Lib/test/test_email/data/python.bmp b/Lib/test/test_email/data/python.bmp new file mode 100644 index 00000000000..675f95191a4 Binary files /dev/null and b/Lib/test/test_email/data/python.bmp differ diff --git a/Lib/test/test_email/data/python.exr b/Lib/test/test_email/data/python.exr new file mode 100644 index 00000000000..773c81ee1fb Binary files /dev/null and b/Lib/test/test_email/data/python.exr differ diff --git a/Lib/test/test_email/data/python.gif b/Lib/test/test_email/data/python.gif new file mode 100644 index 00000000000..efa0be3861d Binary files /dev/null and b/Lib/test/test_email/data/python.gif differ diff --git a/Lib/test/test_email/data/python.jpg b/Lib/test/test_email/data/python.jpg new file mode 100644 index 00000000000..21222c09f5a Binary files /dev/null and b/Lib/test/test_email/data/python.jpg differ diff --git a/Lib/test/test_email/data/python.pbm b/Lib/test/test_email/data/python.pbm new file mode 100644 index 00000000000..1848ba7ff06 --- /dev/null +++ b/Lib/test/test_email/data/python.pbm @@ -0,0 +1,3 @@ +P4 +16 16 +[a_X? \ No newline at end of file diff --git a/Lib/test/test_email/data/python.pgm b/Lib/test/test_email/data/python.pgm new file mode 100644 index 00000000000..8349f2a53a9 Binary files /dev/null and b/Lib/test/test_email/data/python.pgm differ diff --git a/Lib/test/test_email/data/python.png b/Lib/test/test_email/data/python.png new file mode 100644 index 00000000000..1a987f79fcd Binary files /dev/null and b/Lib/test/test_email/data/python.png differ diff --git a/Lib/test/test_email/data/python.ppm b/Lib/test/test_email/data/python.ppm new file mode 100644 index 00000000000..7d9cdb32158 Binary files /dev/null and b/Lib/test/test_email/data/python.ppm differ diff --git a/Lib/test/test_email/data/python.ras b/Lib/test/test_email/data/python.ras new file mode 100644 index 00000000000..130e96f817e Binary files /dev/null and b/Lib/test/test_email/data/python.ras differ diff --git a/Lib/test/test_email/data/python.sgi b/Lib/test/test_email/data/python.sgi new file mode 100644 index 00000000000..ffe9081c7a5 Binary files /dev/null and b/Lib/test/test_email/data/python.sgi differ diff --git a/Lib/test/test_email/data/python.tiff b/Lib/test/test_email/data/python.tiff new file mode 100644 index 00000000000..39d0bfcec02 Binary files /dev/null and b/Lib/test/test_email/data/python.tiff differ diff --git a/Lib/test/test_email/data/python.webp b/Lib/test/test_email/data/python.webp new file mode 100644 index 00000000000..e824ec7fb1c Binary files /dev/null and b/Lib/test/test_email/data/python.webp differ diff --git a/Lib/test/test_email/data/python.xbm b/Lib/test/test_email/data/python.xbm new file mode 100644 index 00000000000..cfbee2e9806 --- /dev/null +++ b/Lib/test/test_email/data/python.xbm @@ -0,0 +1,6 @@ +#define python_width 16 +#define python_height 16 +static char python_bits[] = { + 0xDF, 0xFE, 0x8F, 0xFD, 0x5F, 0xFB, 0xAB, 0xFE, 0xB5, 0x8D, 0xDA, 0x8F, + 0xA5, 0x86, 0xFA, 0x83, 0x1A, 0x80, 0x0D, 0x80, 0x0D, 0x80, 0x0F, 0xE0, + 0x0F, 0xF8, 0x0F, 0xF8, 0x0F, 0xFC, 0xFF, 0xFF, }; diff --git a/Lib/test/test_email/data/sndhdr.aifc b/Lib/test/test_email/data/sndhdr.aifc new file mode 100644 index 00000000000..8aae4e730bd Binary files /dev/null and b/Lib/test/test_email/data/sndhdr.aifc differ diff --git a/Lib/test/test_email/data/sndhdr.aiff b/Lib/test/test_email/data/sndhdr.aiff new file mode 100644 index 00000000000..8c279a762f1 Binary files /dev/null and b/Lib/test/test_email/data/sndhdr.aiff differ diff --git a/Lib/test/test_email/data/sndhdr.au b/Lib/test/test_email/data/sndhdr.au new file mode 100644 index 00000000000..f76b0501b8c Binary files /dev/null and b/Lib/test/test_email/data/sndhdr.au differ diff --git a/Lib/test/test_email/data/sndhdr.wav b/Lib/test/test_email/data/sndhdr.wav new file mode 100644 index 00000000000..0dca36739cd Binary files /dev/null and b/Lib/test/test_email/data/sndhdr.wav differ diff --git a/Lib/test/test_email/test__encoded_words.py b/Lib/test/test_email/test__encoded_words.py new file mode 100644 index 00000000000..dcac7d34e95 --- /dev/null +++ b/Lib/test/test_email/test__encoded_words.py @@ -0,0 +1,208 @@ +import unittest +from email import _encoded_words as _ew +from email import errors +from test.test_email import TestEmailBase + + +class TestDecodeQ(TestEmailBase): + + def _test(self, source, ex_result, ex_defects=[]): + result, defects = _ew.decode_q(source) + self.assertEqual(result, ex_result) + self.assertDefectsEqual(defects, ex_defects) + + def test_no_encoded(self): + self._test(b'foobar', b'foobar') + + def test_spaces(self): + self._test(b'foo=20bar=20', b'foo bar ') + self._test(b'foo_bar_', b'foo bar ') + + def test_run_of_encoded(self): + self._test(b'foo=20=20=21=2Cbar', b'foo !,bar') + + +class TestDecodeB(TestEmailBase): + + def _test(self, source, ex_result, ex_defects=[]): + result, defects = _ew.decode_b(source) + self.assertEqual(result, ex_result) + self.assertDefectsEqual(defects, ex_defects) + + def test_simple(self): + self._test(b'Zm9v', b'foo') + + def test_missing_padding(self): + # 1 missing padding character + self._test(b'dmk', b'vi', [errors.InvalidBase64PaddingDefect]) + # 2 missing padding characters + self._test(b'dg', b'v', [errors.InvalidBase64PaddingDefect]) + + def test_invalid_character(self): + self._test(b'dm\x01k===', b'vi', [errors.InvalidBase64CharactersDefect]) + + def test_invalid_character_and_bad_padding(self): + self._test(b'dm\x01k', b'vi', [errors.InvalidBase64CharactersDefect, + errors.InvalidBase64PaddingDefect]) + + def test_invalid_length(self): + self._test(b'abcde', b'abcde', [errors.InvalidBase64LengthDefect]) + + +class TestDecode(TestEmailBase): + + def test_wrong_format_input_raises(self): + with self.assertRaises(ValueError): + _ew.decode('=?badone?=') + with self.assertRaises(ValueError): + _ew.decode('=?') + with self.assertRaises(ValueError): + _ew.decode('') + with self.assertRaises(KeyError): + _ew.decode('=?utf-8?X?somevalue?=') + + def _test(self, source, result, charset='us-ascii', lang='', defects=[]): + res, char, l, d = _ew.decode(source) + self.assertEqual(res, result) + self.assertEqual(char, charset) + self.assertEqual(l, lang) + self.assertDefectsEqual(d, defects) + + def test_simple_q(self): + self._test('=?us-ascii?q?foo?=', 'foo') + + def test_simple_b(self): + self._test('=?us-ascii?b?dmk=?=', 'vi') + + def test_q_case_ignored(self): + self._test('=?us-ascii?Q?foo?=', 'foo') + + def test_b_case_ignored(self): + self._test('=?us-ascii?B?dmk=?=', 'vi') + + def test_non_trivial_q(self): + self._test('=?latin-1?q?=20F=fcr=20Elise=20?=', ' Für Elise ', 'latin-1') + + def test_q_escaped_bytes_preserved(self): + self._test(b'=?us-ascii?q?=20\xACfoo?='.decode('us-ascii', + 'surrogateescape'), + ' \uDCACfoo', + defects = [errors.UndecodableBytesDefect]) + + def test_b_undecodable_bytes_ignored_with_defect(self): + self._test(b'=?us-ascii?b?dm\xACk?='.decode('us-ascii', + 'surrogateescape'), + 'vi', + defects = [ + errors.InvalidBase64CharactersDefect, + errors.InvalidBase64PaddingDefect]) + + def test_b_invalid_bytes_ignored_with_defect(self): + self._test('=?us-ascii?b?dm\x01k===?=', + 'vi', + defects = [errors.InvalidBase64CharactersDefect]) + + def test_b_invalid_bytes_incorrect_padding(self): + self._test('=?us-ascii?b?dm\x01k?=', + 'vi', + defects = [ + errors.InvalidBase64CharactersDefect, + errors.InvalidBase64PaddingDefect]) + + def test_b_padding_defect(self): + self._test('=?us-ascii?b?dmk?=', + 'vi', + defects = [errors.InvalidBase64PaddingDefect]) + + def test_nonnull_lang(self): + self._test('=?us-ascii*jive?q?test?=', 'test', lang='jive') + + def test_unknown_8bit_charset(self): + self._test('=?unknown-8bit?q?foo=ACbar?=', + b'foo\xacbar'.decode('ascii', 'surrogateescape'), + charset = 'unknown-8bit', + defects = []) + + def test_unknown_charset(self): + self._test('=?foobar?q?foo=ACbar?=', + b'foo\xacbar'.decode('ascii', 'surrogateescape'), + charset = 'foobar', + # XXX Should this be a new Defect instead? + defects = [errors.CharsetError]) + + @unittest.skip("TODO: RUSTPYTHON; str has surrogates") + def test_invalid_character_in_charset(self): + self._test('=?utf-8\udce2\udc80\udc9d?q?foo=ACbar?=', + b'foo\xacbar'.decode('ascii', 'surrogateescape'), + charset = 'utf-8\udce2\udc80\udc9d', + # XXX Should this be a new Defect instead? + defects = [errors.CharsetError]) + + def test_q_nonascii(self): + self._test('=?utf-8?q?=C3=89ric?=', + 'Éric', + charset='utf-8') + + +class TestEncodeQ(TestEmailBase): + + def _test(self, src, expected): + self.assertEqual(_ew.encode_q(src), expected) + + def test_all_safe(self): + self._test(b'foobar', 'foobar') + + def test_spaces(self): + self._test(b'foo bar ', 'foo_bar_') + + def test_run_of_encodables(self): + self._test(b'foo ,,bar', 'foo__=2C=2Cbar') + + +class TestEncodeB(TestEmailBase): + + def test_simple(self): + self.assertEqual(_ew.encode_b(b'foo'), 'Zm9v') + + def test_padding(self): + self.assertEqual(_ew.encode_b(b'vi'), 'dmk=') + + +class TestEncode(TestEmailBase): + + def test_q(self): + self.assertEqual(_ew.encode('foo', 'utf-8', 'q'), '=?utf-8?q?foo?=') + + def test_b(self): + self.assertEqual(_ew.encode('foo', 'utf-8', 'b'), '=?utf-8?b?Zm9v?=') + + def test_auto_q(self): + self.assertEqual(_ew.encode('foo', 'utf-8'), '=?utf-8?q?foo?=') + + def test_auto_q_if_short_mostly_safe(self): + self.assertEqual(_ew.encode('vi.', 'utf-8'), '=?utf-8?q?vi=2E?=') + + def test_auto_b_if_enough_unsafe(self): + self.assertEqual(_ew.encode('.....', 'utf-8'), '=?utf-8?b?Li4uLi4=?=') + + def test_auto_b_if_long_unsafe(self): + self.assertEqual(_ew.encode('vi.vi.vi.vi.vi.', 'utf-8'), + '=?utf-8?b?dmkudmkudmkudmkudmku?=') + + def test_auto_q_if_long_mostly_safe(self): + self.assertEqual(_ew.encode('vi vi vi.vi ', 'utf-8'), + '=?utf-8?q?vi_vi_vi=2Evi_?=') + + def test_utf8_default(self): + self.assertEqual(_ew.encode('foo'), '=?utf-8?q?foo?=') + + def test_lang(self): + self.assertEqual(_ew.encode('foo', lang='jive'), '=?utf-8*jive?q?foo?=') + + def test_unknown_8bit(self): + self.assertEqual(_ew.encode('foo\uDCACbar', charset='unknown-8bit'), + '=?unknown-8bit?q?foo=ACbar?=') + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test__header_value_parser.py b/Lib/test/test_email/test__header_value_parser.py new file mode 100644 index 00000000000..64bc3677e87 --- /dev/null +++ b/Lib/test/test_email/test__header_value_parser.py @@ -0,0 +1,3259 @@ +import string +import unittest +from email import _header_value_parser as parser +from email import errors +from email import policy +from test.test_email import TestEmailBase, parameterize + +class TestTokens(TestEmailBase): + + # EWWhiteSpaceTerminal + + def test_EWWhiteSpaceTerminal(self): + x = parser.EWWhiteSpaceTerminal(' \t', 'fws') + self.assertEqual(x, ' \t') + self.assertEqual(str(x), '') + self.assertEqual(x.value, '') + self.assertEqual(x.token_type, 'fws') + + +class TestParserMixin: + + def _assert_results(self, tl, rest, string, value, defects, remainder, + comments=None): + self.assertEqual(str(tl), string) + self.assertEqual(tl.value, value) + self.assertDefectsEqual(tl.all_defects, defects) + self.assertEqual(rest, remainder) + if comments is not None: + self.assertEqual(tl.comments, comments) + + def _test_get_x(self, method, source, string, value, defects, + remainder, comments=None): + tl, rest = method(source) + self._assert_results(tl, rest, string, value, defects, remainder, + comments=None) + return tl + + def _test_parse_x(self, method, input, string, value, defects, + comments=None): + tl = method(input) + self._assert_results(tl, '', string, value, defects, '', comments) + return tl + + +class TestParser(TestParserMixin, TestEmailBase): + + # _wsp_splitter + + rfc_printable_ascii = bytes(range(33, 127)).decode('ascii') + rfc_atext_chars = (string.ascii_letters + string.digits + + "!#$%&\'*+-/=?^_`{}|~") + rfc_dtext_chars = rfc_printable_ascii.translate(str.maketrans('','',r'\[]')) + + def test__wsp_splitter_one_word(self): + self.assertEqual(parser._wsp_splitter('foo', 1), ['foo']) + + def test__wsp_splitter_two_words(self): + self.assertEqual(parser._wsp_splitter('foo def', 1), + ['foo', ' ', 'def']) + + def test__wsp_splitter_ws_runs(self): + self.assertEqual(parser._wsp_splitter('foo \t def jik', 1), + ['foo', ' \t ', 'def jik']) + + + # get_fws + + def test_get_fws_only(self): + fws = self._test_get_x(parser.get_fws, ' \t ', ' \t ', ' ', [], '') + self.assertEqual(fws.token_type, 'fws') + + def test_get_fws_space(self): + self._test_get_x(parser.get_fws, ' foo', ' ', ' ', [], 'foo') + + def test_get_fws_ws_run(self): + self._test_get_x(parser.get_fws, ' \t foo ', ' \t ', ' ', [], 'foo ') + + # get_encoded_word + + def test_get_encoded_word_missing_start_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_encoded_word('abc') + + def test_get_encoded_word_missing_end_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_encoded_word('=?abc') + + def test_get_encoded_word_missing_middle_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_encoded_word('=?abc?=') + + def test_get_encoded_word_invalid_cte(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_encoded_word('=?utf-8?X?somevalue?=') + + def test_get_encoded_word_valid_ew(self): + self._test_get_x(parser.get_encoded_word, + '=?us-ascii?q?this_is_a_test?= bird', + 'this is a test', + 'this is a test', + [], + ' bird') + + def test_get_encoded_word_internal_spaces(self): + self._test_get_x(parser.get_encoded_word, + '=?us-ascii?q?this is a test?= bird', + 'this is a test', + 'this is a test', + [errors.InvalidHeaderDefect], + ' bird') + + def test_get_encoded_word_gets_first(self): + self._test_get_x(parser.get_encoded_word, + '=?us-ascii?q?first?= =?utf-8?q?second?=', + 'first', + 'first', + [], + ' =?utf-8?q?second?=') + + def test_get_encoded_word_gets_first_even_if_no_space(self): + self._test_get_x(parser.get_encoded_word, + '=?us-ascii?q?first?==?utf-8?q?second?=', + 'first', + 'first', + [errors.InvalidHeaderDefect], + '=?utf-8?q?second?=') + + def test_get_encoded_word_sets_extra_attributes(self): + ew = self._test_get_x(parser.get_encoded_word, + '=?us-ascii*jive?q?first_second?=', + 'first second', + 'first second', + [], + '') + self.assertEqual(ew.charset, 'us-ascii') + self.assertEqual(ew.lang, 'jive') + + def test_get_encoded_word_lang_default_is_blank(self): + ew = self._test_get_x(parser.get_encoded_word, + '=?us-ascii?q?first_second?=', + 'first second', + 'first second', + [], + '') + self.assertEqual(ew.charset, 'us-ascii') + self.assertEqual(ew.lang, '') + + def test_get_encoded_word_non_printable_defect(self): + self._test_get_x(parser.get_encoded_word, + '=?us-ascii?q?first\x02second?=', + 'first\x02second', + 'first\x02second', + [errors.NonPrintableDefect], + '') + + def test_get_encoded_word_leading_internal_space(self): + self._test_get_x(parser.get_encoded_word, + '=?us-ascii?q?=20foo?=', + ' foo', + ' foo', + [], + '') + + def test_get_encoded_word_quopri_utf_escape_follows_cte(self): + # Issue 18044 + self._test_get_x(parser.get_encoded_word, + '=?utf-8?q?=C3=89ric?=', + 'Éric', + 'Éric', + [], + '') + + # get_unstructured + + def _get_unst(self, value): + token = parser.get_unstructured(value) + return token, '' + + def test_get_unstructured_null(self): + self._test_get_x(self._get_unst, '', '', '', [], '') + + def test_get_unstructured_one_word(self): + self._test_get_x(self._get_unst, 'foo', 'foo', 'foo', [], '') + + def test_get_unstructured_normal_phrase(self): + self._test_get_x(self._get_unst, 'foo bar bird', + 'foo bar bird', + 'foo bar bird', + [], + '') + + def test_get_unstructured_normal_phrase_with_whitespace(self): + self._test_get_x(self._get_unst, 'foo \t bar bird', + 'foo \t bar bird', + 'foo bar bird', + [], + '') + + def test_get_unstructured_leading_whitespace(self): + self._test_get_x(self._get_unst, ' foo bar', + ' foo bar', + ' foo bar', + [], + '') + + def test_get_unstructured_trailing_whitespace(self): + self._test_get_x(self._get_unst, 'foo bar ', + 'foo bar ', + 'foo bar ', + [], + '') + + def test_get_unstructured_leading_and_trailing_whitespace(self): + self._test_get_x(self._get_unst, ' foo bar ', + ' foo bar ', + ' foo bar ', + [], + '') + + def test_get_unstructured_one_valid_ew_no_ws(self): + self._test_get_x(self._get_unst, '=?us-ascii?q?bar?=', + 'bar', + 'bar', + [], + '') + + def test_get_unstructured_one_ew_trailing_ws(self): + self._test_get_x(self._get_unst, '=?us-ascii?q?bar?= ', + 'bar ', + 'bar ', + [], + '') + + def test_get_unstructured_one_valid_ew_trailing_text(self): + self._test_get_x(self._get_unst, '=?us-ascii?q?bar?= bird', + 'bar bird', + 'bar bird', + [], + '') + + def test_get_unstructured_phrase_with_ew_in_middle_of_text(self): + self._test_get_x(self._get_unst, 'foo =?us-ascii?q?bar?= bird', + 'foo bar bird', + 'foo bar bird', + [], + '') + + def test_get_unstructured_phrase_with_two_ew(self): + self._test_get_x(self._get_unst, + 'foo =?us-ascii?q?bar?= =?us-ascii?q?bird?=', + 'foo barbird', + 'foo barbird', + [], + '') + + def test_get_unstructured_phrase_with_two_ew_trailing_ws(self): + self._test_get_x(self._get_unst, + 'foo =?us-ascii?q?bar?= =?us-ascii?q?bird?= ', + 'foo barbird ', + 'foo barbird ', + [], + '') + + def test_get_unstructured_phrase_with_ew_with_leading_ws(self): + self._test_get_x(self._get_unst, + ' =?us-ascii?q?bar?=', + ' bar', + ' bar', + [], + '') + + def test_get_unstructured_phrase_with_two_ew_extra_ws(self): + self._test_get_x(self._get_unst, + 'foo =?us-ascii?q?bar?= \t =?us-ascii?q?bird?=', + 'foo barbird', + 'foo barbird', + [], + '') + + def test_get_unstructured_two_ew_extra_ws_trailing_text(self): + self._test_get_x(self._get_unst, + '=?us-ascii?q?test?= =?us-ascii?q?foo?= val', + 'testfoo val', + 'testfoo val', + [], + '') + + def test_get_unstructured_ew_with_internal_ws(self): + self._test_get_x(self._get_unst, + '=?iso-8859-1?q?hello=20world?=', + 'hello world', + 'hello world', + [], + '') + + def test_get_unstructured_ew_with_internal_leading_ws(self): + self._test_get_x(self._get_unst, + ' =?us-ascii?q?=20test?= =?us-ascii?q?=20foo?= val', + ' test foo val', + ' test foo val', + [], + '') + + def test_get_unstructured_invalid_ew(self): + self._test_get_x(self._get_unst, + '=?test val', + '=?test val', + '=?test val', + [], + '') + + def test_get_unstructured_undecodable_bytes(self): + self._test_get_x(self._get_unst, + b'test \xACfoo val'.decode('ascii', 'surrogateescape'), + 'test \uDCACfoo val', + 'test \uDCACfoo val', + [errors.UndecodableBytesDefect], + '') + + def test_get_unstructured_undecodable_bytes_in_EW(self): + self._test_get_x(self._get_unst, + (b'=?us-ascii?q?=20test?= =?us-ascii?q?=20\xACfoo?=' + b' val').decode('ascii', 'surrogateescape'), + ' test \uDCACfoo val', + ' test \uDCACfoo val', + [errors.UndecodableBytesDefect]*2, + '') + + def test_get_unstructured_missing_base64_padding(self): + self._test_get_x(self._get_unst, + '=?utf-8?b?dmk?=', + 'vi', + 'vi', + [errors.InvalidBase64PaddingDefect], + '') + + def test_get_unstructured_invalid_base64_character(self): + self._test_get_x(self._get_unst, + '=?utf-8?b?dm\x01k===?=', + 'vi', + 'vi', + [errors.InvalidBase64CharactersDefect], + '') + + def test_get_unstructured_invalid_base64_character_and_bad_padding(self): + self._test_get_x(self._get_unst, + '=?utf-8?b?dm\x01k?=', + 'vi', + 'vi', + [errors.InvalidBase64CharactersDefect, + errors.InvalidBase64PaddingDefect], + '') + + def test_get_unstructured_invalid_base64_length(self): + # bpo-27397: Return the encoded string since there's no way to decode. + self._test_get_x(self._get_unst, + '=?utf-8?b?abcde?=', + 'abcde', + 'abcde', + [errors.InvalidBase64LengthDefect], + '') + + def test_get_unstructured_no_whitespace_between_ews(self): + self._test_get_x(self._get_unst, + '=?utf-8?q?foo?==?utf-8?q?bar?=', + 'foobar', + 'foobar', + [errors.InvalidHeaderDefect, + errors.InvalidHeaderDefect], + '') + + def test_get_unstructured_ew_without_leading_whitespace(self): + self._test_get_x( + self._get_unst, + 'nowhitespace=?utf-8?q?somevalue?=', + 'nowhitespacesomevalue', + 'nowhitespacesomevalue', + [errors.InvalidHeaderDefect], + '') + + def test_get_unstructured_ew_without_trailing_whitespace(self): + self._test_get_x( + self._get_unst, + '=?utf-8?q?somevalue?=nowhitespace', + 'somevaluenowhitespace', + 'somevaluenowhitespace', + [errors.InvalidHeaderDefect], + '') + + def test_get_unstructured_without_trailing_whitespace_hang_case(self): + self._test_get_x(self._get_unst, + '=?utf-8?q?somevalue?=aa', + 'somevalueaa', + 'somevalueaa', + [errors.InvalidHeaderDefect], + '') + + def test_get_unstructured_invalid_ew2(self): + self._test_get_x(self._get_unst, + '=?utf-8?q?=somevalue?=', + '=?utf-8?q?=somevalue?=', + '=?utf-8?q?=somevalue?=', + [], + '') + + def test_get_unstructured_invalid_ew_cte(self): + self._test_get_x(self._get_unst, + '=?utf-8?X?=somevalue?=', + '=?utf-8?X?=somevalue?=', + '=?utf-8?X?=somevalue?=', + [], + '') + + # get_qp_ctext + + def test_get_qp_ctext_only(self): + ptext = self._test_get_x(parser.get_qp_ctext, + 'foobar', 'foobar', ' ', [], '') + self.assertEqual(ptext.token_type, 'ptext') + + def test_get_qp_ctext_all_printables(self): + with_qp = self.rfc_printable_ascii.replace('\\', '\\\\') + with_qp = with_qp. replace('(', r'\(') + with_qp = with_qp.replace(')', r'\)') + ptext = self._test_get_x(parser.get_qp_ctext, + with_qp, self.rfc_printable_ascii, ' ', [], '') + + def test_get_qp_ctext_two_words_gets_first(self): + self._test_get_x(parser.get_qp_ctext, + 'foo de', 'foo', ' ', [], ' de') + + def test_get_qp_ctext_following_wsp_preserved(self): + self._test_get_x(parser.get_qp_ctext, + 'foo \t\tde', 'foo', ' ', [], ' \t\tde') + + def test_get_qp_ctext_up_to_close_paren_only(self): + self._test_get_x(parser.get_qp_ctext, + 'foo)', 'foo', ' ', [], ')') + + def test_get_qp_ctext_wsp_before_close_paren_preserved(self): + self._test_get_x(parser.get_qp_ctext, + 'foo )', 'foo', ' ', [], ' )') + + def test_get_qp_ctext_close_paren_mid_word(self): + self._test_get_x(parser.get_qp_ctext, + 'foo)bar', 'foo', ' ', [], ')bar') + + def test_get_qp_ctext_up_to_open_paren_only(self): + self._test_get_x(parser.get_qp_ctext, + 'foo(', 'foo', ' ', [], '(') + + def test_get_qp_ctext_wsp_before_open_paren_preserved(self): + self._test_get_x(parser.get_qp_ctext, + 'foo (', 'foo', ' ', [], ' (') + + def test_get_qp_ctext_open_paren_mid_word(self): + self._test_get_x(parser.get_qp_ctext, + 'foo(bar', 'foo', ' ', [], '(bar') + + def test_get_qp_ctext_non_printables(self): + ptext = self._test_get_x(parser.get_qp_ctext, + 'foo\x00bar)', 'foo\x00bar', ' ', + [errors.NonPrintableDefect], ')') + self.assertEqual(ptext.defects[0].non_printables[0], '\x00') + + def test_get_qp_ctext_close_paren_only(self): + self._test_get_x(parser.get_qp_ctext, + ')', '', ' ', [], ')') + + def test_get_qp_ctext_open_paren_only(self): + self._test_get_x(parser.get_qp_ctext, + '(', '', ' ', [], '(') + + def test_get_qp_ctext_no_end_char(self): + self._test_get_x(parser.get_qp_ctext, + '', '', ' ', [], '') + + + # get_qcontent + + def test_get_qcontent_only(self): + ptext = self._test_get_x(parser.get_qcontent, + 'foobar', 'foobar', 'foobar', [], '') + self.assertEqual(ptext.token_type, 'ptext') + + def test_get_qcontent_all_printables(self): + with_qp = self.rfc_printable_ascii.replace('\\', '\\\\') + with_qp = with_qp. replace('"', r'\"') + ptext = self._test_get_x(parser.get_qcontent, with_qp, + self.rfc_printable_ascii, + self.rfc_printable_ascii, [], '') + + def test_get_qcontent_two_words_gets_first(self): + self._test_get_x(parser.get_qcontent, + 'foo de', 'foo', 'foo', [], ' de') + + def test_get_qcontent_following_wsp_preserved(self): + self._test_get_x(parser.get_qcontent, + 'foo \t\tde', 'foo', 'foo', [], ' \t\tde') + + def test_get_qcontent_up_to_dquote_only(self): + self._test_get_x(parser.get_qcontent, + 'foo"', 'foo', 'foo', [], '"') + + def test_get_qcontent_wsp_before_close_paren_preserved(self): + self._test_get_x(parser.get_qcontent, + 'foo "', 'foo', 'foo', [], ' "') + + def test_get_qcontent_close_paren_mid_word(self): + self._test_get_x(parser.get_qcontent, + 'foo"bar', 'foo', 'foo', [], '"bar') + + def test_get_qcontent_non_printables(self): + ptext = self._test_get_x(parser.get_qcontent, + 'foo\x00fg"', 'foo\x00fg', 'foo\x00fg', + [errors.NonPrintableDefect], '"') + self.assertEqual(ptext.defects[0].non_printables[0], '\x00') + + def test_get_qcontent_empty(self): + self._test_get_x(parser.get_qcontent, + '"', '', '', [], '"') + + def test_get_qcontent_no_end_char(self): + self._test_get_x(parser.get_qcontent, + '', '', '', [], '') + + # get_atext + + def test_get_atext_only(self): + atext = self._test_get_x(parser.get_atext, + 'foobar', 'foobar', 'foobar', [], '') + self.assertEqual(atext.token_type, 'atext') + + def test_get_atext_all_atext(self): + atext = self._test_get_x(parser.get_atext, self.rfc_atext_chars, + self.rfc_atext_chars, + self.rfc_atext_chars, [], '') + + def test_get_atext_two_words_gets_first(self): + self._test_get_x(parser.get_atext, + 'foo bar', 'foo', 'foo', [], ' bar') + + def test_get_atext_following_wsp_preserved(self): + self._test_get_x(parser.get_atext, + 'foo \t\tbar', 'foo', 'foo', [], ' \t\tbar') + + def test_get_atext_up_to_special(self): + self._test_get_x(parser.get_atext, + 'foo@bar', 'foo', 'foo', [], '@bar') + + def test_get_atext_non_printables(self): + atext = self._test_get_x(parser.get_atext, + 'foo\x00bar(', 'foo\x00bar', 'foo\x00bar', + [errors.NonPrintableDefect], '(') + self.assertEqual(atext.defects[0].non_printables[0], '\x00') + + # get_bare_quoted_string + + def test_get_bare_quoted_string_only(self): + bqs = self._test_get_x(parser.get_bare_quoted_string, + '"foo"', '"foo"', 'foo', [], '') + self.assertEqual(bqs.token_type, 'bare-quoted-string') + + def test_get_bare_quoted_string_must_start_with_dquote(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_bare_quoted_string('foo"') + with self.assertRaises(errors.HeaderParseError): + parser.get_bare_quoted_string(' "foo"') + + def test_get_bare_quoted_string_only_quotes(self): + self._test_get_x(parser.get_bare_quoted_string, + '""', '""', '', [], '') + + def test_get_bare_quoted_string_missing_endquotes(self): + self._test_get_x(parser.get_bare_quoted_string, + '"', '""', '', [errors.InvalidHeaderDefect], '') + + def test_get_bare_quoted_string_following_wsp_preserved(self): + self._test_get_x(parser.get_bare_quoted_string, + '"foo"\t bar', '"foo"', 'foo', [], '\t bar') + + def test_get_bare_quoted_string_multiple_words(self): + self._test_get_x(parser.get_bare_quoted_string, + '"foo bar moo"', '"foo bar moo"', 'foo bar moo', [], '') + + def test_get_bare_quoted_string_multiple_words_wsp_preserved(self): + self._test_get_x(parser.get_bare_quoted_string, + '" foo moo\t"', '" foo moo\t"', ' foo moo\t', [], '') + + def test_get_bare_quoted_string_end_dquote_mid_word(self): + self._test_get_x(parser.get_bare_quoted_string, + '"foo"bar', '"foo"', 'foo', [], 'bar') + + def test_get_bare_quoted_string_quoted_dquote(self): + self._test_get_x(parser.get_bare_quoted_string, + r'"foo\"in"a', r'"foo\"in"', 'foo"in', [], 'a') + + def test_get_bare_quoted_string_non_printables(self): + self._test_get_x(parser.get_bare_quoted_string, + '"a\x01a"', '"a\x01a"', 'a\x01a', + [errors.NonPrintableDefect], '') + + def test_get_bare_quoted_string_no_end_dquote(self): + self._test_get_x(parser.get_bare_quoted_string, + '"foo', '"foo"', 'foo', + [errors.InvalidHeaderDefect], '') + self._test_get_x(parser.get_bare_quoted_string, + '"foo ', '"foo "', 'foo ', + [errors.InvalidHeaderDefect], '') + + def test_get_bare_quoted_string_empty_quotes(self): + self._test_get_x(parser.get_bare_quoted_string, + '""', '""', '', [], '') + + # Issue 16983: apply postel's law to some bad encoding. + def test_encoded_word_inside_quotes(self): + self._test_get_x(parser.get_bare_quoted_string, + '"=?utf-8?Q?not_really_valid?="', + '"not really valid"', + 'not really valid', + [errors.InvalidHeaderDefect, + errors.InvalidHeaderDefect], + '') + + # get_comment + + def test_get_comment_only(self): + comment = self._test_get_x(parser.get_comment, + '(comment)', '(comment)', ' ', [], '', ['comment']) + self.assertEqual(comment.token_type, 'comment') + + def test_get_comment_must_start_with_paren(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_comment('foo"') + with self.assertRaises(errors.HeaderParseError): + parser.get_comment(' (foo"') + + def test_get_comment_following_wsp_preserved(self): + self._test_get_x(parser.get_comment, + '(comment) \t', '(comment)', ' ', [], ' \t', ['comment']) + + def test_get_comment_multiple_words(self): + self._test_get_x(parser.get_comment, + '(foo bar) \t', '(foo bar)', ' ', [], ' \t', ['foo bar']) + + def test_get_comment_multiple_words_wsp_preserved(self): + self._test_get_x(parser.get_comment, + '( foo bar\t ) \t', '( foo bar\t )', ' ', [], ' \t', + [' foo bar\t ']) + + def test_get_comment_end_paren_mid_word(self): + self._test_get_x(parser.get_comment, + '(foo)bar', '(foo)', ' ', [], 'bar', ['foo']) + + def test_get_comment_quoted_parens(self): + self._test_get_x(parser.get_comment, + r'(foo\) \(\)bar)', r'(foo\) \(\)bar)', ' ', [], '', ['foo) ()bar']) + + def test_get_comment_non_printable(self): + self._test_get_x(parser.get_comment, + '(foo\x7Fbar)', '(foo\x7Fbar)', ' ', + [errors.NonPrintableDefect], '', ['foo\x7Fbar']) + + def test_get_comment_no_end_paren(self): + self._test_get_x(parser.get_comment, + '(foo bar', '(foo bar)', ' ', + [errors.InvalidHeaderDefect], '', ['foo bar']) + self._test_get_x(parser.get_comment, + '(foo bar ', '(foo bar )', ' ', + [errors.InvalidHeaderDefect], '', ['foo bar ']) + + def test_get_comment_nested_comment(self): + comment = self._test_get_x(parser.get_comment, + '(foo(bar))', '(foo(bar))', ' ', [], '', ['foo(bar)']) + self.assertEqual(comment[1].content, 'bar') + + def test_get_comment_nested_comment_wsp(self): + comment = self._test_get_x(parser.get_comment, + '(foo ( bar ) )', '(foo ( bar ) )', ' ', [], '', ['foo ( bar ) ']) + self.assertEqual(comment[2].content, ' bar ') + + def test_get_comment_empty_comment(self): + self._test_get_x(parser.get_comment, + '()', '()', ' ', [], '', ['']) + + def test_get_comment_multiple_nesting(self): + comment = self._test_get_x(parser.get_comment, + '(((((foo)))))', '(((((foo)))))', ' ', [], '', ['((((foo))))']) + for i in range(4, 0, -1): + self.assertEqual(comment[0].content, '('*(i-1)+'foo'+')'*(i-1)) + comment = comment[0] + self.assertEqual(comment.content, 'foo') + + def test_get_comment_missing_end_of_nesting(self): + self._test_get_x(parser.get_comment, + '(((((foo)))', '(((((foo)))))', ' ', + [errors.InvalidHeaderDefect]*2, '', ['((((foo))))']) + + def test_get_comment_qs_in_nested_comment(self): + comment = self._test_get_x(parser.get_comment, + r'(foo (b\)))', r'(foo (b\)))', ' ', [], '', [r'foo (b\))']) + self.assertEqual(comment[2].content, 'b)') + + # get_cfws + + def test_get_cfws_only_ws(self): + cfws = self._test_get_x(parser.get_cfws, + ' \t \t', ' \t \t', ' ', [], '', []) + self.assertEqual(cfws.token_type, 'cfws') + + def test_get_cfws_only_comment(self): + cfws = self._test_get_x(parser.get_cfws, + '(foo)', '(foo)', ' ', [], '', ['foo']) + self.assertEqual(cfws[0].content, 'foo') + + def test_get_cfws_only_mixed(self): + cfws = self._test_get_x(parser.get_cfws, + ' (foo ) ( bar) ', ' (foo ) ( bar) ', ' ', [], '', + ['foo ', ' bar']) + self.assertEqual(cfws[1].content, 'foo ') + self.assertEqual(cfws[3].content, ' bar') + + def test_get_cfws_ends_at_non_leader(self): + cfws = self._test_get_x(parser.get_cfws, + '(foo) bar', '(foo) ', ' ', [], 'bar', ['foo']) + self.assertEqual(cfws[0].content, 'foo') + + def test_get_cfws_ends_at_non_printable(self): + cfws = self._test_get_x(parser.get_cfws, + '(foo) \x07', '(foo) ', ' ', [], '\x07', ['foo']) + self.assertEqual(cfws[0].content, 'foo') + + def test_get_cfws_non_printable_in_comment(self): + cfws = self._test_get_x(parser.get_cfws, + '(foo \x07) "test"', '(foo \x07) ', ' ', + [errors.NonPrintableDefect], '"test"', ['foo \x07']) + self.assertEqual(cfws[0].content, 'foo \x07') + + def test_get_cfws_header_ends_in_comment(self): + cfws = self._test_get_x(parser.get_cfws, + ' (foo ', ' (foo )', ' ', + [errors.InvalidHeaderDefect], '', ['foo ']) + self.assertEqual(cfws[1].content, 'foo ') + + def test_get_cfws_multiple_nested_comments(self): + cfws = self._test_get_x(parser.get_cfws, + '(foo (bar)) ((a)(a))', '(foo (bar)) ((a)(a))', ' ', [], + '', ['foo (bar)', '(a)(a)']) + self.assertEqual(cfws[0].comments, ['foo (bar)']) + self.assertEqual(cfws[2].comments, ['(a)(a)']) + + # get_quoted_string + + def test_get_quoted_string_only(self): + qs = self._test_get_x(parser.get_quoted_string, + '"bob"', '"bob"', 'bob', [], '') + self.assertEqual(qs.token_type, 'quoted-string') + self.assertEqual(qs.quoted_value, '"bob"') + self.assertEqual(qs.content, 'bob') + + def test_get_quoted_string_with_wsp(self): + qs = self._test_get_x(parser.get_quoted_string, + '\t "bob" ', '\t "bob" ', ' bob ', [], '') + self.assertEqual(qs.quoted_value, ' "bob" ') + self.assertEqual(qs.content, 'bob') + + def test_get_quoted_string_with_comments_and_wsp(self): + qs = self._test_get_x(parser.get_quoted_string, + ' (foo) "bob"(bar)', ' (foo) "bob"(bar)', ' bob ', [], '') + self.assertEqual(qs[0][1].content, 'foo') + self.assertEqual(qs[2][0].content, 'bar') + self.assertEqual(qs.content, 'bob') + self.assertEqual(qs.quoted_value, ' "bob" ') + + def test_get_quoted_string_with_multiple_comments(self): + qs = self._test_get_x(parser.get_quoted_string, + ' (foo) (bar) "bob"(bird)', ' (foo) (bar) "bob"(bird)', ' bob ', + [], '') + self.assertEqual(qs[0].comments, ['foo', 'bar']) + self.assertEqual(qs[2].comments, ['bird']) + self.assertEqual(qs.content, 'bob') + self.assertEqual(qs.quoted_value, ' "bob" ') + + def test_get_quoted_string_non_printable_in_comment(self): + qs = self._test_get_x(parser.get_quoted_string, + ' (\x0A) "bob"', ' (\x0A) "bob"', ' bob', + [errors.NonPrintableDefect], '') + self.assertEqual(qs[0].comments, ['\x0A']) + self.assertEqual(qs.content, 'bob') + self.assertEqual(qs.quoted_value, ' "bob"') + + def test_get_quoted_string_non_printable_in_qcontent(self): + qs = self._test_get_x(parser.get_quoted_string, + ' (a) "a\x0B"', ' (a) "a\x0B"', ' a\x0B', + [errors.NonPrintableDefect], '') + self.assertEqual(qs[0].comments, ['a']) + self.assertEqual(qs.content, 'a\x0B') + self.assertEqual(qs.quoted_value, ' "a\x0B"') + + def test_get_quoted_string_internal_ws(self): + qs = self._test_get_x(parser.get_quoted_string, + ' (a) "foo bar "', ' (a) "foo bar "', ' foo bar ', + [], '') + self.assertEqual(qs[0].comments, ['a']) + self.assertEqual(qs.content, 'foo bar ') + self.assertEqual(qs.quoted_value, ' "foo bar "') + + def test_get_quoted_string_header_ends_in_comment(self): + qs = self._test_get_x(parser.get_quoted_string, + ' (a) "bob" (a', ' (a) "bob" (a)', ' bob ', + [errors.InvalidHeaderDefect], '') + self.assertEqual(qs[0].comments, ['a']) + self.assertEqual(qs[2].comments, ['a']) + self.assertEqual(qs.content, 'bob') + self.assertEqual(qs.quoted_value, ' "bob" ') + + def test_get_quoted_string_header_ends_in_qcontent(self): + qs = self._test_get_x(parser.get_quoted_string, + ' (a) "bob', ' (a) "bob"', ' bob', + [errors.InvalidHeaderDefect], '') + self.assertEqual(qs[0].comments, ['a']) + self.assertEqual(qs.content, 'bob') + self.assertEqual(qs.quoted_value, ' "bob"') + + def test_get_quoted_string_cfws_only_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_quoted_string(' (foo) ') + + def test_get_quoted_string_no_quoted_string(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_quoted_string(' (ab) xyz') + + def test_get_quoted_string_qs_ends_at_noncfws(self): + qs = self._test_get_x(parser.get_quoted_string, + '\t "bob" fee', '\t "bob" ', ' bob ', [], 'fee') + self.assertEqual(qs.content, 'bob') + self.assertEqual(qs.quoted_value, ' "bob" ') + + # get_atom + + def test_get_atom_only(self): + atom = self._test_get_x(parser.get_atom, + 'bob', 'bob', 'bob', [], '') + self.assertEqual(atom.token_type, 'atom') + + def test_get_atom_with_wsp(self): + self._test_get_x(parser.get_atom, + '\t bob ', '\t bob ', ' bob ', [], '') + + def test_get_atom_with_comments_and_wsp(self): + atom = self._test_get_x(parser.get_atom, + ' (foo) bob(bar)', ' (foo) bob(bar)', ' bob ', [], '') + self.assertEqual(atom[0][1].content, 'foo') + self.assertEqual(atom[2][0].content, 'bar') + + def test_get_atom_with_multiple_comments(self): + atom = self._test_get_x(parser.get_atom, + ' (foo) (bar) bob(bird)', ' (foo) (bar) bob(bird)', ' bob ', + [], '') + self.assertEqual(atom[0].comments, ['foo', 'bar']) + self.assertEqual(atom[2].comments, ['bird']) + + def test_get_atom_non_printable_in_comment(self): + atom = self._test_get_x(parser.get_atom, + ' (\x0A) bob', ' (\x0A) bob', ' bob', + [errors.NonPrintableDefect], '') + self.assertEqual(atom[0].comments, ['\x0A']) + + def test_get_atom_non_printable_in_atext(self): + atom = self._test_get_x(parser.get_atom, + ' (a) a\x0B', ' (a) a\x0B', ' a\x0B', + [errors.NonPrintableDefect], '') + self.assertEqual(atom[0].comments, ['a']) + + def test_get_atom_header_ends_in_comment(self): + atom = self._test_get_x(parser.get_atom, + ' (a) bob (a', ' (a) bob (a)', ' bob ', + [errors.InvalidHeaderDefect], '') + self.assertEqual(atom[0].comments, ['a']) + self.assertEqual(atom[2].comments, ['a']) + + def test_get_atom_no_atom(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_atom(' (ab) ') + + def test_get_atom_no_atom_before_special(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_atom(' (ab) @') + + def test_get_atom_atom_ends_at_special(self): + atom = self._test_get_x(parser.get_atom, + ' (foo) bob(bar) @bang', ' (foo) bob(bar) ', ' bob ', [], '@bang') + self.assertEqual(atom[0].comments, ['foo']) + self.assertEqual(atom[2].comments, ['bar']) + + def test_get_atom_atom_ends_at_noncfws(self): + self._test_get_x(parser.get_atom, + 'bob fred', 'bob ', 'bob ', [], 'fred') + + def test_get_atom_rfc2047_atom(self): + self._test_get_x(parser.get_atom, + '=?utf-8?q?=20bob?=', ' bob', ' bob', [], '') + + # get_dot_atom_text + + def test_get_dot_atom_text(self): + dot_atom_text = self._test_get_x(parser.get_dot_atom_text, + 'foo.bar.bang', 'foo.bar.bang', 'foo.bar.bang', [], '') + self.assertEqual(dot_atom_text.token_type, 'dot-atom-text') + self.assertEqual(len(dot_atom_text), 5) + + def test_get_dot_atom_text_lone_atom_is_valid(self): + dot_atom_text = self._test_get_x(parser.get_dot_atom_text, + 'foo', 'foo', 'foo', [], '') + + def test_get_dot_atom_text_raises_on_leading_dot(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom_text('.foo.bar') + + def test_get_dot_atom_text_raises_on_trailing_dot(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom_text('foo.bar.') + + def test_get_dot_atom_text_raises_on_leading_non_atext(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom_text(' foo.bar') + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom_text('@foo.bar') + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom_text('"foo.bar"') + + def test_get_dot_atom_text_trailing_text_preserved(self): + dot_atom_text = self._test_get_x(parser.get_dot_atom_text, + 'foo@bar', 'foo', 'foo', [], '@bar') + + def test_get_dot_atom_text_trailing_ws_preserved(self): + dot_atom_text = self._test_get_x(parser.get_dot_atom_text, + 'foo .bar', 'foo', 'foo', [], ' .bar') + + # get_dot_atom + + def test_get_dot_atom_only(self): + dot_atom = self._test_get_x(parser.get_dot_atom, + 'foo.bar.bing', 'foo.bar.bing', 'foo.bar.bing', [], '') + self.assertEqual(dot_atom.token_type, 'dot-atom') + self.assertEqual(len(dot_atom), 1) + + def test_get_dot_atom_with_wsp(self): + self._test_get_x(parser.get_dot_atom, + '\t foo.bar.bing ', '\t foo.bar.bing ', ' foo.bar.bing ', [], '') + + def test_get_dot_atom_with_comments_and_wsp(self): + self._test_get_x(parser.get_dot_atom, + ' (sing) foo.bar.bing (here) ', ' (sing) foo.bar.bing (here) ', + ' foo.bar.bing ', [], '') + + def test_get_dot_atom_space_ends_dot_atom(self): + self._test_get_x(parser.get_dot_atom, + ' (sing) foo.bar .bing (here) ', ' (sing) foo.bar ', + ' foo.bar ', [], '.bing (here) ') + + def test_get_dot_atom_no_atom_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom(' (foo) ') + + def test_get_dot_atom_leading_dot_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom(' (foo) .bar') + + def test_get_dot_atom_two_dots_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom('bar..bang') + + def test_get_dot_atom_trailing_dot_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom(' (foo) bar.bang. foo') + + def test_get_dot_atom_rfc2047_atom(self): + self._test_get_x(parser.get_dot_atom, + '=?utf-8?q?=20bob?=', ' bob', ' bob', [], '') + + # get_word (if this were black box we'd repeat all the qs/atom tests) + + def test_get_word_atom_yields_atom(self): + word = self._test_get_x(parser.get_word, + ' (foo) bar (bang) :ah', ' (foo) bar (bang) ', ' bar ', [], ':ah') + self.assertEqual(word.token_type, 'atom') + self.assertEqual(word[0].token_type, 'cfws') + + def test_get_word_all_CFWS(self): + # bpo-29412: Test that we don't raise IndexError when parsing CFWS only + # token. + with self.assertRaises(errors.HeaderParseError): + parser.get_word('(Recipients list suppressed') + + def test_get_word_qs_yields_qs(self): + word = self._test_get_x(parser.get_word, + '"bar " (bang) ah', '"bar " (bang) ', 'bar ', [], 'ah') + self.assertEqual(word.token_type, 'quoted-string') + self.assertEqual(word[0].token_type, 'bare-quoted-string') + self.assertEqual(word[0].value, 'bar ') + self.assertEqual(word.content, 'bar ') + + def test_get_word_ends_at_dot(self): + self._test_get_x(parser.get_word, + 'foo.', 'foo', 'foo', [], '.') + + # get_phrase + + def test_get_phrase_simple(self): + phrase = self._test_get_x(parser.get_phrase, + '"Fred A. Johnson" is his name, oh.', + '"Fred A. Johnson" is his name', + 'Fred A. Johnson is his name', + [], + ', oh.') + self.assertEqual(phrase.token_type, 'phrase') + + def test_get_phrase_complex(self): + phrase = self._test_get_x(parser.get_phrase, + ' (A) bird (in (my|your)) "hand " is messy\t<>\t', + ' (A) bird (in (my|your)) "hand " is messy\t', + ' bird hand is messy ', + [], + '<>\t') + self.assertEqual(phrase[0][0].comments, ['A']) + self.assertEqual(phrase[0][2].comments, ['in (my|your)']) + + def test_get_phrase_obsolete(self): + phrase = self._test_get_x(parser.get_phrase, + 'Fred A.(weird).O Johnson', + 'Fred A.(weird).O Johnson', + 'Fred A. .O Johnson', + [errors.ObsoleteHeaderDefect]*3, + '') + self.assertEqual(len(phrase), 7) + self.assertEqual(phrase[3].comments, ['weird']) + + def test_get_phrase_pharse_must_start_with_word(self): + phrase = self._test_get_x(parser.get_phrase, + '(even weirder).name', + '(even weirder).name', + ' .name', + [errors.InvalidHeaderDefect] + [errors.ObsoleteHeaderDefect]*2, + '') + self.assertEqual(len(phrase), 3) + self.assertEqual(phrase[0].comments, ['even weirder']) + + def test_get_phrase_ending_with_obsolete(self): + phrase = self._test_get_x(parser.get_phrase, + 'simple phrase.(with trailing comment):boo', + 'simple phrase.(with trailing comment)', + 'simple phrase. ', + [errors.ObsoleteHeaderDefect]*2, + ':boo') + self.assertEqual(len(phrase), 4) + self.assertEqual(phrase[3].comments, ['with trailing comment']) + + def get_phrase_cfws_only_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_phrase(' (foo) ') + + # get_local_part + + def test_get_local_part_simple(self): + local_part = self._test_get_x(parser.get_local_part, + 'dinsdale@python.org', 'dinsdale', 'dinsdale', [], '@python.org') + self.assertEqual(local_part.token_type, 'local-part') + self.assertEqual(local_part.local_part, 'dinsdale') + + def test_get_local_part_with_dot(self): + local_part = self._test_get_x(parser.get_local_part, + 'Fred.A.Johnson@python.org', + 'Fred.A.Johnson', + 'Fred.A.Johnson', + [], + '@python.org') + self.assertEqual(local_part.local_part, 'Fred.A.Johnson') + + def test_get_local_part_with_whitespace(self): + local_part = self._test_get_x(parser.get_local_part, + ' Fred.A.Johnson @python.org', + ' Fred.A.Johnson ', + ' Fred.A.Johnson ', + [], + '@python.org') + self.assertEqual(local_part.local_part, 'Fred.A.Johnson') + + def test_get_local_part_with_cfws(self): + local_part = self._test_get_x(parser.get_local_part, + ' (foo) Fred.A.Johnson (bar (bird)) @python.org', + ' (foo) Fred.A.Johnson (bar (bird)) ', + ' Fred.A.Johnson ', + [], + '@python.org') + self.assertEqual(local_part.local_part, 'Fred.A.Johnson') + self.assertEqual(local_part[0][0].comments, ['foo']) + self.assertEqual(local_part[0][2].comments, ['bar (bird)']) + + def test_get_local_part_simple_quoted(self): + local_part = self._test_get_x(parser.get_local_part, + '"dinsdale"@python.org', '"dinsdale"', '"dinsdale"', [], '@python.org') + self.assertEqual(local_part.token_type, 'local-part') + self.assertEqual(local_part.local_part, 'dinsdale') + + def test_get_local_part_with_quoted_dot(self): + local_part = self._test_get_x(parser.get_local_part, + '"Fred.A.Johnson"@python.org', + '"Fred.A.Johnson"', + '"Fred.A.Johnson"', + [], + '@python.org') + self.assertEqual(local_part.local_part, 'Fred.A.Johnson') + + def test_get_local_part_quoted_with_whitespace(self): + local_part = self._test_get_x(parser.get_local_part, + ' "Fred A. Johnson" @python.org', + ' "Fred A. Johnson" ', + ' "Fred A. Johnson" ', + [], + '@python.org') + self.assertEqual(local_part.local_part, 'Fred A. Johnson') + + def test_get_local_part_quoted_with_cfws(self): + local_part = self._test_get_x(parser.get_local_part, + ' (foo) " Fred A. Johnson " (bar (bird)) @python.org', + ' (foo) " Fred A. Johnson " (bar (bird)) ', + ' " Fred A. Johnson " ', + [], + '@python.org') + self.assertEqual(local_part.local_part, ' Fred A. Johnson ') + self.assertEqual(local_part[0][0].comments, ['foo']) + self.assertEqual(local_part[0][2].comments, ['bar (bird)']) + + + def test_get_local_part_simple_obsolete(self): + local_part = self._test_get_x(parser.get_local_part, + 'Fred. A.Johnson@python.org', + 'Fred. A.Johnson', + 'Fred. A.Johnson', + [errors.ObsoleteHeaderDefect], + '@python.org') + self.assertEqual(local_part.local_part, 'Fred.A.Johnson') + + def test_get_local_part_complex_obsolete_1(self): + local_part = self._test_get_x(parser.get_local_part, + ' (foo )Fred (bar).(bird) A.(sheep)Johnson."and dogs "@python.org', + ' (foo )Fred (bar).(bird) A.(sheep)Johnson."and dogs "', + ' Fred . A. Johnson.and dogs ', + [errors.ObsoleteHeaderDefect], + '@python.org') + self.assertEqual(local_part.local_part, 'Fred.A.Johnson.and dogs ') + + def test_get_local_part_complex_obsolete_invalid(self): + local_part = self._test_get_x(parser.get_local_part, + ' (foo )Fred (bar).(bird) A.(sheep)Johnson "and dogs"@python.org', + ' (foo )Fred (bar).(bird) A.(sheep)Johnson "and dogs"', + ' Fred . A. Johnson and dogs', + [errors.InvalidHeaderDefect]*2, + '@python.org') + self.assertEqual(local_part.local_part, 'Fred.A.Johnson and dogs') + + def test_get_local_part_empty_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_local_part('') + + def test_get_local_part_no_part_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_local_part(' (foo) ') + + def test_get_local_part_special_instead_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_local_part(' (foo) @python.org') + + def test_get_local_part_trailing_dot(self): + local_part = self._test_get_x(parser.get_local_part, + ' borris.@python.org', + ' borris.', + ' borris.', + [errors.InvalidHeaderDefect]*2, + '@python.org') + self.assertEqual(local_part.local_part, 'borris.') + + def test_get_local_part_trailing_dot_with_ws(self): + local_part = self._test_get_x(parser.get_local_part, + ' borris. @python.org', + ' borris. ', + ' borris. ', + [errors.InvalidHeaderDefect]*2, + '@python.org') + self.assertEqual(local_part.local_part, 'borris.') + + def test_get_local_part_leading_dot(self): + local_part = self._test_get_x(parser.get_local_part, + '.borris@python.org', + '.borris', + '.borris', + [errors.InvalidHeaderDefect]*2, + '@python.org') + self.assertEqual(local_part.local_part, '.borris') + + def test_get_local_part_leading_dot_after_ws(self): + local_part = self._test_get_x(parser.get_local_part, + ' .borris@python.org', + ' .borris', + ' .borris', + [errors.InvalidHeaderDefect]*2, + '@python.org') + self.assertEqual(local_part.local_part, '.borris') + + def test_get_local_part_double_dot_raises(self): + local_part = self._test_get_x(parser.get_local_part, + ' borris.(foo).natasha@python.org', + ' borris.(foo).natasha', + ' borris. .natasha', + [errors.InvalidHeaderDefect]*2, + '@python.org') + self.assertEqual(local_part.local_part, 'borris..natasha') + + def test_get_local_part_quoted_strings_in_atom_list(self): + local_part = self._test_get_x(parser.get_local_part, + '""example" example"@example.com', + '""example" example"', + 'example example', + [errors.InvalidHeaderDefect]*3, + '@example.com') + self.assertEqual(local_part.local_part, 'example example') + + def test_get_local_part_valid_and_invalid_qp_in_atom_list(self): + local_part = self._test_get_x(parser.get_local_part, + r'"\\"example\\" example"@example.com', + r'"\\"example\\" example"', + r'\example\\ example', + [errors.InvalidHeaderDefect]*5, + '@example.com') + self.assertEqual(local_part.local_part, r'\example\\ example') + + def test_get_local_part_unicode_defect(self): + # Currently this only happens when parsing unicode, not when parsing + # stuff that was originally binary. + local_part = self._test_get_x(parser.get_local_part, + 'exámple@example.com', + 'exámple', + 'exámple', + [errors.NonASCIILocalPartDefect], + '@example.com') + self.assertEqual(local_part.local_part, 'exámple') + + # get_dtext + + def test_get_dtext_only(self): + dtext = self._test_get_x(parser.get_dtext, + 'foobar', 'foobar', 'foobar', [], '') + self.assertEqual(dtext.token_type, 'ptext') + + def test_get_dtext_all_dtext(self): + dtext = self._test_get_x(parser.get_dtext, self.rfc_dtext_chars, + self.rfc_dtext_chars, + self.rfc_dtext_chars, [], '') + + def test_get_dtext_two_words_gets_first(self): + self._test_get_x(parser.get_dtext, + 'foo bar', 'foo', 'foo', [], ' bar') + + def test_get_dtext_following_wsp_preserved(self): + self._test_get_x(parser.get_dtext, + 'foo \t\tbar', 'foo', 'foo', [], ' \t\tbar') + + def test_get_dtext_non_printables(self): + dtext = self._test_get_x(parser.get_dtext, + 'foo\x00bar]', 'foo\x00bar', 'foo\x00bar', + [errors.NonPrintableDefect], ']') + self.assertEqual(dtext.defects[0].non_printables[0], '\x00') + + def test_get_dtext_with_qp(self): + ptext = self._test_get_x(parser.get_dtext, + r'foo\]\[\\bar\b\e\l\l', + r'foo][\barbell', + r'foo][\barbell', + [errors.ObsoleteHeaderDefect], + '') + + def test_get_dtext_up_to_close_bracket_only(self): + self._test_get_x(parser.get_dtext, + 'foo]', 'foo', 'foo', [], ']') + + def test_get_dtext_wsp_before_close_bracket_preserved(self): + self._test_get_x(parser.get_dtext, + 'foo ]', 'foo', 'foo', [], ' ]') + + def test_get_dtext_close_bracket_mid_word(self): + self._test_get_x(parser.get_dtext, + 'foo]bar', 'foo', 'foo', [], ']bar') + + def test_get_dtext_up_to_open_bracket_only(self): + self._test_get_x(parser.get_dtext, + 'foo[', 'foo', 'foo', [], '[') + + def test_get_dtext_wsp_before_open_bracket_preserved(self): + self._test_get_x(parser.get_dtext, + 'foo [', 'foo', 'foo', [], ' [') + + def test_get_dtext_open_bracket_mid_word(self): + self._test_get_x(parser.get_dtext, + 'foo[bar', 'foo', 'foo', [], '[bar') + + def test_get_dtext_open_bracket_only(self): + self._test_get_x(parser.get_dtext, + '[', '', '', [], '[') + + def test_get_dtext_close_bracket_only(self): + self._test_get_x(parser.get_dtext, + ']', '', '', [], ']') + + def test_get_dtext_empty(self): + self._test_get_x(parser.get_dtext, + '', '', '', [], '') + + # get_domain_literal + + def test_get_domain_literal_only(self): + domain_literal = domain_literal = self._test_get_x(parser.get_domain_literal, + '[127.0.0.1]', + '[127.0.0.1]', + '[127.0.0.1]', + [], + '') + self.assertEqual(domain_literal.token_type, 'domain-literal') + self.assertEqual(domain_literal.domain, '[127.0.0.1]') + self.assertEqual(domain_literal.ip, '127.0.0.1') + + def test_get_domain_literal_with_internal_ws(self): + domain_literal = self._test_get_x(parser.get_domain_literal, + '[ 127.0.0.1\t ]', + '[ 127.0.0.1\t ]', + '[ 127.0.0.1 ]', + [], + '') + self.assertEqual(domain_literal.domain, '[127.0.0.1]') + self.assertEqual(domain_literal.ip, '127.0.0.1') + + def test_get_domain_literal_with_surrounding_cfws(self): + domain_literal = self._test_get_x(parser.get_domain_literal, + '(foo)[ 127.0.0.1] (bar)', + '(foo)[ 127.0.0.1] (bar)', + ' [ 127.0.0.1] ', + [], + '') + self.assertEqual(domain_literal.domain, '[127.0.0.1]') + self.assertEqual(domain_literal.ip, '127.0.0.1') + + def test_get_domain_literal_no_start_char_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_domain_literal('(foo) ') + + def test_get_domain_literal_no_start_char_before_special_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_domain_literal('(foo) @') + + def test_get_domain_literal_bad_dtext_char_before_special_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_domain_literal('(foo) [abc[@') + + # get_domain + + def test_get_domain_regular_domain_only(self): + domain = self._test_get_x(parser.get_domain, + 'example.com', + 'example.com', + 'example.com', + [], + '') + self.assertEqual(domain.token_type, 'domain') + self.assertEqual(domain.domain, 'example.com') + + def test_get_domain_domain_literal_only(self): + domain = self._test_get_x(parser.get_domain, + '[127.0.0.1]', + '[127.0.0.1]', + '[127.0.0.1]', + [], + '') + self.assertEqual(domain.token_type, 'domain') + self.assertEqual(domain.domain, '[127.0.0.1]') + + def test_get_domain_with_cfws(self): + domain = self._test_get_x(parser.get_domain, + '(foo) example.com(bar)\t', + '(foo) example.com(bar)\t', + ' example.com ', + [], + '') + self.assertEqual(domain.domain, 'example.com') + + def test_get_domain_domain_literal_with_cfws(self): + domain = self._test_get_x(parser.get_domain, + '(foo)[127.0.0.1]\t(bar)', + '(foo)[127.0.0.1]\t(bar)', + ' [127.0.0.1] ', + [], + '') + self.assertEqual(domain.domain, '[127.0.0.1]') + + def test_get_domain_domain_with_cfws_ends_at_special(self): + domain = self._test_get_x(parser.get_domain, + '(foo)example.com\t(bar), next', + '(foo)example.com\t(bar)', + ' example.com ', + [], + ', next') + self.assertEqual(domain.domain, 'example.com') + + def test_get_domain_domain_literal_with_cfws_ends_at_special(self): + domain = self._test_get_x(parser.get_domain, + '(foo)[127.0.0.1]\t(bar), next', + '(foo)[127.0.0.1]\t(bar)', + ' [127.0.0.1] ', + [], + ', next') + self.assertEqual(domain.domain, '[127.0.0.1]') + + def test_get_domain_obsolete(self): + domain = self._test_get_x(parser.get_domain, + '(foo) example . (bird)com(bar)\t', + '(foo) example . (bird)com(bar)\t', + ' example . com ', + [errors.ObsoleteHeaderDefect], + '') + self.assertEqual(domain.domain, 'example.com') + + def test_get_domain_empty_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_domain("") + + def test_get_domain_no_non_cfws_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_domain(" (foo)\t") + + def test_get_domain_no_atom_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_domain(" (foo)\t, broken") + + + # get_addr_spec + + def test_get_addr_spec_normal(self): + addr_spec = self._test_get_x(parser.get_addr_spec, + 'dinsdale@example.com', + 'dinsdale@example.com', + 'dinsdale@example.com', + [], + '') + self.assertEqual(addr_spec.token_type, 'addr-spec') + self.assertEqual(addr_spec.local_part, 'dinsdale') + self.assertEqual(addr_spec.domain, 'example.com') + self.assertEqual(addr_spec.addr_spec, 'dinsdale@example.com') + + def test_get_addr_spec_with_doamin_literal(self): + addr_spec = self._test_get_x(parser.get_addr_spec, + 'dinsdale@[127.0.0.1]', + 'dinsdale@[127.0.0.1]', + 'dinsdale@[127.0.0.1]', + [], + '') + self.assertEqual(addr_spec.local_part, 'dinsdale') + self.assertEqual(addr_spec.domain, '[127.0.0.1]') + self.assertEqual(addr_spec.addr_spec, 'dinsdale@[127.0.0.1]') + + def test_get_addr_spec_with_cfws(self): + addr_spec = self._test_get_x(parser.get_addr_spec, + '(foo) dinsdale(bar)@ (bird) example.com (bog)', + '(foo) dinsdale(bar)@ (bird) example.com (bog)', + ' dinsdale@example.com ', + [], + '') + self.assertEqual(addr_spec.local_part, 'dinsdale') + self.assertEqual(addr_spec.domain, 'example.com') + self.assertEqual(addr_spec.addr_spec, 'dinsdale@example.com') + + def test_get_addr_spec_with_qouoted_string_and_cfws(self): + addr_spec = self._test_get_x(parser.get_addr_spec, + '(foo) "roy a bug"(bar)@ (bird) example.com (bog)', + '(foo) "roy a bug"(bar)@ (bird) example.com (bog)', + ' "roy a bug"@example.com ', + [], + '') + self.assertEqual(addr_spec.local_part, 'roy a bug') + self.assertEqual(addr_spec.domain, 'example.com') + self.assertEqual(addr_spec.addr_spec, '"roy a bug"@example.com') + + def test_get_addr_spec_ends_at_special(self): + addr_spec = self._test_get_x(parser.get_addr_spec, + '(foo) "roy a bug"(bar)@ (bird) example.com (bog) , next', + '(foo) "roy a bug"(bar)@ (bird) example.com (bog) ', + ' "roy a bug"@example.com ', + [], + ', next') + self.assertEqual(addr_spec.local_part, 'roy a bug') + self.assertEqual(addr_spec.domain, 'example.com') + self.assertEqual(addr_spec.addr_spec, '"roy a bug"@example.com') + + def test_get_addr_spec_quoted_strings_in_atom_list(self): + addr_spec = self._test_get_x(parser.get_addr_spec, + '""example" example"@example.com', + '""example" example"@example.com', + 'example example@example.com', + [errors.InvalidHeaderDefect]*3, + '') + self.assertEqual(addr_spec.local_part, 'example example') + self.assertEqual(addr_spec.domain, 'example.com') + self.assertEqual(addr_spec.addr_spec, '"example example"@example.com') + + def test_get_addr_spec_dot_atom(self): + addr_spec = self._test_get_x(parser.get_addr_spec, + 'star.a.star@example.com', + 'star.a.star@example.com', + 'star.a.star@example.com', + [], + '') + self.assertEqual(addr_spec.local_part, 'star.a.star') + self.assertEqual(addr_spec.domain, 'example.com') + self.assertEqual(addr_spec.addr_spec, 'star.a.star@example.com') + + def test_get_addr_spec_multiple_domains(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_addr_spec('star@a.star@example.com') + + with self.assertRaises(errors.HeaderParseError): + parser.get_addr_spec('star@a@example.com') + + with self.assertRaises(errors.HeaderParseError): + parser.get_addr_spec('star@172.17.0.1@example.com') + + # get_obs_route + + def test_get_obs_route_simple(self): + obs_route = self._test_get_x(parser.get_obs_route, + '@example.com, @two.example.com:', + '@example.com, @two.example.com:', + '@example.com, @two.example.com:', + [], + '') + self.assertEqual(obs_route.token_type, 'obs-route') + self.assertEqual(obs_route.domains, ['example.com', 'two.example.com']) + + def test_get_obs_route_complex(self): + obs_route = self._test_get_x(parser.get_obs_route, + '(foo),, (blue)@example.com (bar),@two.(foo) example.com (bird):', + '(foo),, (blue)@example.com (bar),@two.(foo) example.com (bird):', + ' ,, @example.com ,@two. example.com :', + [errors.ObsoleteHeaderDefect], # This is the obs-domain + '') + self.assertEqual(obs_route.token_type, 'obs-route') + self.assertEqual(obs_route.domains, ['example.com', 'two.example.com']) + + def test_get_obs_route_no_route_before_end_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_obs_route('(foo) @example.com,') + + def test_get_obs_route_no_route_before_end_raises2(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_obs_route('(foo) @example.com, (foo) ') + + def test_get_obs_route_no_route_before_special_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_obs_route('(foo) [abc],') + + def test_get_obs_route_no_route_before_special_raises2(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_obs_route('(foo) @example.com [abc],') + + def test_get_obs_route_no_domain_after_at_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_obs_route('@') + + def test_get_obs_route_no_domain_after_at_raises2(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_obs_route('@example.com, @') + + # get_angle_addr + + def test_get_angle_addr_simple(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + '', + '', + '', + [], + '') + self.assertEqual(angle_addr.token_type, 'angle-addr') + self.assertEqual(angle_addr.local_part, 'dinsdale') + self.assertEqual(angle_addr.domain, 'example.com') + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, 'dinsdale@example.com') + + def test_get_angle_addr_empty(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + '<>', + '<>', + '<>', + [errors.InvalidHeaderDefect], + '') + self.assertEqual(angle_addr.token_type, 'angle-addr') + self.assertIsNone(angle_addr.local_part) + self.assertIsNone(angle_addr.domain) + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, '<>') + + def test_get_angle_addr_qs_only_quotes(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + '<""@example.com>', + '<""@example.com>', + '<""@example.com>', + [], + '') + self.assertEqual(angle_addr.token_type, 'angle-addr') + self.assertEqual(angle_addr.local_part, '') + self.assertEqual(angle_addr.domain, 'example.com') + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, '""@example.com') + + def test_get_angle_addr_with_cfws(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + ' (foo) (bar)', + ' (foo) (bar)', + ' ', + [], + '') + self.assertEqual(angle_addr.token_type, 'angle-addr') + self.assertEqual(angle_addr.local_part, 'dinsdale') + self.assertEqual(angle_addr.domain, 'example.com') + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, 'dinsdale@example.com') + + def test_get_angle_addr_qs_and_domain_literal(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + '<"Fred Perfect"@[127.0.0.1]>', + '<"Fred Perfect"@[127.0.0.1]>', + '<"Fred Perfect"@[127.0.0.1]>', + [], + '') + self.assertEqual(angle_addr.local_part, 'Fred Perfect') + self.assertEqual(angle_addr.domain, '[127.0.0.1]') + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, '"Fred Perfect"@[127.0.0.1]') + + def test_get_angle_addr_internal_cfws(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + '<(foo) dinsdale@example.com(bar)>', + '<(foo) dinsdale@example.com(bar)>', + '< dinsdale@example.com >', + [], + '') + self.assertEqual(angle_addr.local_part, 'dinsdale') + self.assertEqual(angle_addr.domain, 'example.com') + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, 'dinsdale@example.com') + + def test_get_angle_addr_obs_route(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + '(foo)<@example.com, (bird) @two.example.com: dinsdale@example.com> (bar) ', + '(foo)<@example.com, (bird) @two.example.com: dinsdale@example.com> (bar) ', + ' <@example.com, @two.example.com: dinsdale@example.com> ', + [errors.ObsoleteHeaderDefect], + '') + self.assertEqual(angle_addr.local_part, 'dinsdale') + self.assertEqual(angle_addr.domain, 'example.com') + self.assertEqual(angle_addr.route, ['example.com', 'two.example.com']) + self.assertEqual(angle_addr.addr_spec, 'dinsdale@example.com') + + def test_get_angle_addr_missing_closing_angle(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + '', + '', + [errors.InvalidHeaderDefect], + '') + self.assertEqual(angle_addr.local_part, 'dinsdale') + self.assertEqual(angle_addr.domain, 'example.com') + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, 'dinsdale@example.com') + + def test_get_angle_addr_missing_closing_angle_with_cfws(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + '', + '', + [errors.InvalidHeaderDefect], + '') + self.assertEqual(angle_addr.local_part, 'dinsdale') + self.assertEqual(angle_addr.domain, 'example.com') + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, 'dinsdale@example.com') + + def test_get_angle_addr_ends_at_special(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + ' (foo), next', + ' (foo)', + ' ', + [], + ', next') + self.assertEqual(angle_addr.local_part, 'dinsdale') + self.assertEqual(angle_addr.domain, 'example.com') + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, 'dinsdale@example.com') + + def test_get_angle_addr_empty_raise(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_angle_addr('') + + def test_get_angle_addr_left_angle_only_raise(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_angle_addr('<') + + def test_get_angle_addr_no_angle_raise(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_angle_addr('(foo) ') + + def test_get_angle_addr_no_angle_before_special_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_angle_addr('(foo) , next') + + def test_get_angle_addr_no_angle_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_angle_addr('bar') + + def test_get_angle_addr_special_after_angle_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_angle_addr('(foo) <, bar') + + # get_display_name This is phrase but with a different value. + + def test_get_display_name_simple(self): + display_name = self._test_get_x(parser.get_display_name, + 'Fred A Johnson', + 'Fred A Johnson', + 'Fred A Johnson', + [], + '') + self.assertEqual(display_name.token_type, 'display-name') + self.assertEqual(display_name.display_name, 'Fred A Johnson') + + def test_get_display_name_complex1(self): + display_name = self._test_get_x(parser.get_display_name, + '"Fred A. Johnson" is his name, oh.', + '"Fred A. Johnson" is his name', + '"Fred A. Johnson is his name"', + [], + ', oh.') + self.assertEqual(display_name.token_type, 'display-name') + self.assertEqual(display_name.display_name, 'Fred A. Johnson is his name') + + def test_get_display_name_complex2(self): + display_name = self._test_get_x(parser.get_display_name, + ' (A) bird (in (my|your)) "hand " is messy\t<>\t', + ' (A) bird (in (my|your)) "hand " is messy\t', + ' "bird hand is messy" ', + [], + '<>\t') + self.assertEqual(display_name[0][0].comments, ['A']) + self.assertEqual(display_name[0][2].comments, ['in (my|your)']) + self.assertEqual(display_name.display_name, 'bird hand is messy') + + def test_get_display_name_obsolete(self): + display_name = self._test_get_x(parser.get_display_name, + 'Fred A.(weird).O Johnson', + 'Fred A.(weird).O Johnson', + '"Fred A. .O Johnson"', + [errors.ObsoleteHeaderDefect]*3, + '') + self.assertEqual(len(display_name), 7) + self.assertEqual(display_name[3].comments, ['weird']) + self.assertEqual(display_name.display_name, 'Fred A. .O Johnson') + + def test_get_display_name_pharse_must_start_with_word(self): + display_name = self._test_get_x(parser.get_display_name, + '(even weirder).name', + '(even weirder).name', + ' ".name"', + [errors.InvalidHeaderDefect] + [errors.ObsoleteHeaderDefect]*2, + '') + self.assertEqual(len(display_name), 3) + self.assertEqual(display_name[0].comments, ['even weirder']) + self.assertEqual(display_name.display_name, '.name') + + def test_get_display_name_ending_with_obsolete(self): + display_name = self._test_get_x(parser.get_display_name, + 'simple phrase.(with trailing comment):boo', + 'simple phrase.(with trailing comment)', + '"simple phrase." ', + [errors.ObsoleteHeaderDefect]*2, + ':boo') + self.assertEqual(len(display_name), 4) + self.assertEqual(display_name[3].comments, ['with trailing comment']) + self.assertEqual(display_name.display_name, 'simple phrase.') + + def test_get_display_name_for_invalid_address_field(self): + # bpo-32178: Test that address fields starting with `:` don't cause + # IndexError when parsing the display name. + display_name = self._test_get_x( + parser.get_display_name, + ':Foo ', '', '', [errors.InvalidHeaderDefect], ':Foo ') + self.assertEqual(display_name.value, '') + + # get_name_addr + + def test_get_name_addr_angle_addr_only(self): + name_addr = self._test_get_x(parser.get_name_addr, + '', + '', + '', + [], + '') + self.assertEqual(name_addr.token_type, 'name-addr') + self.assertIsNone(name_addr.display_name) + self.assertEqual(name_addr.local_part, 'dinsdale') + self.assertEqual(name_addr.domain, 'example.com') + self.assertIsNone(name_addr.route) + self.assertEqual(name_addr.addr_spec, 'dinsdale@example.com') + + def test_get_name_addr_atom_name(self): + name_addr = self._test_get_x(parser.get_name_addr, + 'Dinsdale ', + 'Dinsdale ', + 'Dinsdale ', + [], + '') + self.assertEqual(name_addr.token_type, 'name-addr') + self.assertEqual(name_addr.display_name, 'Dinsdale') + self.assertEqual(name_addr.local_part, 'dinsdale') + self.assertEqual(name_addr.domain, 'example.com') + self.assertIsNone(name_addr.route) + self.assertEqual(name_addr.addr_spec, 'dinsdale@example.com') + + def test_get_name_addr_atom_name_with_cfws(self): + name_addr = self._test_get_x(parser.get_name_addr, + '(foo) Dinsdale (bar) (bird)', + '(foo) Dinsdale (bar) (bird)', + ' Dinsdale ', + [], + '') + self.assertEqual(name_addr.display_name, 'Dinsdale') + self.assertEqual(name_addr.local_part, 'dinsdale') + self.assertEqual(name_addr.domain, 'example.com') + self.assertIsNone(name_addr.route) + self.assertEqual(name_addr.addr_spec, 'dinsdale@example.com') + + def test_get_name_addr_name_with_cfws_and_dots(self): + name_addr = self._test_get_x(parser.get_name_addr, + '(foo) Roy.A.Bear (bar) (bird)', + '(foo) Roy.A.Bear (bar) (bird)', + ' "Roy.A.Bear" ', + [errors.ObsoleteHeaderDefect]*2, + '') + self.assertEqual(name_addr.display_name, 'Roy.A.Bear') + self.assertEqual(name_addr.local_part, 'dinsdale') + self.assertEqual(name_addr.domain, 'example.com') + self.assertIsNone(name_addr.route) + self.assertEqual(name_addr.addr_spec, 'dinsdale@example.com') + + def test_get_name_addr_qs_name(self): + name_addr = self._test_get_x(parser.get_name_addr, + '"Roy.A.Bear" ', + '"Roy.A.Bear" ', + '"Roy.A.Bear" ', + [], + '') + self.assertEqual(name_addr.display_name, 'Roy.A.Bear') + self.assertEqual(name_addr.local_part, 'dinsdale') + self.assertEqual(name_addr.domain, 'example.com') + self.assertIsNone(name_addr.route) + self.assertEqual(name_addr.addr_spec, 'dinsdale@example.com') + + def test_get_name_addr_ending_with_dot_without_space(self): + name_addr = self._test_get_x(parser.get_name_addr, + 'John X.', + 'John X.', + '"John X."', + [errors.ObsoleteHeaderDefect], + '') + self.assertEqual(name_addr.display_name, 'John X.') + self.assertEqual(name_addr.local_part, 'jxd') + self.assertEqual(name_addr.domain, 'example.com') + self.assertIsNone(name_addr.route) + self.assertEqual(name_addr.addr_spec, 'jxd@example.com') + + def test_get_name_addr_starting_with_dot(self): + name_addr = self._test_get_x(parser.get_name_addr, + '. Doe ', + '. Doe ', + '". Doe" ', + [errors.InvalidHeaderDefect, errors.ObsoleteHeaderDefect], + '') + self.assertEqual(name_addr.display_name, '. Doe') + self.assertEqual(name_addr.local_part, 'jxd') + self.assertEqual(name_addr.domain, 'example.com') + self.assertIsNone(name_addr.route) + self.assertEqual(name_addr.addr_spec, 'jxd@example.com') + + def test_get_name_addr_with_route(self): + name_addr = self._test_get_x(parser.get_name_addr, + '"Roy.A.Bear" <@two.example.com: dinsdale@example.com>', + '"Roy.A.Bear" <@two.example.com: dinsdale@example.com>', + '"Roy.A.Bear" <@two.example.com: dinsdale@example.com>', + [errors.ObsoleteHeaderDefect], + '') + self.assertEqual(name_addr.display_name, 'Roy.A.Bear') + self.assertEqual(name_addr.local_part, 'dinsdale') + self.assertEqual(name_addr.domain, 'example.com') + self.assertEqual(name_addr.route, ['two.example.com']) + self.assertEqual(name_addr.addr_spec, 'dinsdale@example.com') + + def test_get_name_addr_ends_at_special(self): + name_addr = self._test_get_x(parser.get_name_addr, + '"Roy.A.Bear" , next', + '"Roy.A.Bear" ', + '"Roy.A.Bear" ', + [], + ', next') + self.assertEqual(name_addr.display_name, 'Roy.A.Bear') + self.assertEqual(name_addr.local_part, 'dinsdale') + self.assertEqual(name_addr.domain, 'example.com') + self.assertIsNone(name_addr.route) + self.assertEqual(name_addr.addr_spec, 'dinsdale@example.com') + + def test_get_name_addr_empty_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_name_addr('') + + def test_get_name_addr_no_content_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_name_addr(' (foo) ') + + def test_get_name_addr_no_content_before_special_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_name_addr(' (foo) ,') + + def test_get_name_addr_no_angle_after_display_name_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_name_addr('foo bar') + + # get_mailbox + + def test_get_mailbox_addr_spec_only(self): + mailbox = self._test_get_x(parser.get_mailbox, + 'dinsdale@example.com', + 'dinsdale@example.com', + 'dinsdale@example.com', + [], + '') + self.assertEqual(mailbox.token_type, 'mailbox') + self.assertIsNone(mailbox.display_name) + self.assertEqual(mailbox.local_part, 'dinsdale') + self.assertEqual(mailbox.domain, 'example.com') + self.assertIsNone(mailbox.route) + self.assertEqual(mailbox.addr_spec, 'dinsdale@example.com') + + def test_get_mailbox_angle_addr_only(self): + mailbox = self._test_get_x(parser.get_mailbox, + '', + '', + '', + [], + '') + self.assertEqual(mailbox.token_type, 'mailbox') + self.assertIsNone(mailbox.display_name) + self.assertEqual(mailbox.local_part, 'dinsdale') + self.assertEqual(mailbox.domain, 'example.com') + self.assertIsNone(mailbox.route) + self.assertEqual(mailbox.addr_spec, 'dinsdale@example.com') + + def test_get_mailbox_name_addr(self): + mailbox = self._test_get_x(parser.get_mailbox, + '"Roy A. Bear" ', + '"Roy A. Bear" ', + '"Roy A. Bear" ', + [], + '') + self.assertEqual(mailbox.token_type, 'mailbox') + self.assertEqual(mailbox.display_name, 'Roy A. Bear') + self.assertEqual(mailbox.local_part, 'dinsdale') + self.assertEqual(mailbox.domain, 'example.com') + self.assertIsNone(mailbox.route) + self.assertEqual(mailbox.addr_spec, 'dinsdale@example.com') + + def test_get_mailbox_ends_at_special(self): + mailbox = self._test_get_x(parser.get_mailbox, + '"Roy A. Bear" , rest', + '"Roy A. Bear" ', + '"Roy A. Bear" ', + [], + ', rest') + self.assertEqual(mailbox.token_type, 'mailbox') + self.assertEqual(mailbox.display_name, 'Roy A. Bear') + self.assertEqual(mailbox.local_part, 'dinsdale') + self.assertEqual(mailbox.domain, 'example.com') + self.assertIsNone(mailbox.route) + self.assertEqual(mailbox.addr_spec, 'dinsdale@example.com') + + def test_get_mailbox_quoted_strings_in_atom_list(self): + mailbox = self._test_get_x(parser.get_mailbox, + '""example" example"@example.com', + '""example" example"@example.com', + 'example example@example.com', + [errors.InvalidHeaderDefect]*3, + '') + self.assertEqual(mailbox.local_part, 'example example') + self.assertEqual(mailbox.domain, 'example.com') + self.assertEqual(mailbox.addr_spec, '"example example"@example.com') + + # get_mailbox_list + + def test_get_mailbox_list_single_addr(self): + mailbox_list = self._test_get_x(parser.get_mailbox_list, + 'dinsdale@example.com', + 'dinsdale@example.com', + 'dinsdale@example.com', + [], + '') + self.assertEqual(mailbox_list.token_type, 'mailbox-list') + self.assertEqual(len(mailbox_list.mailboxes), 1) + mailbox = mailbox_list.mailboxes[0] + self.assertIsNone(mailbox.display_name) + self.assertEqual(mailbox.local_part, 'dinsdale') + self.assertEqual(mailbox.domain, 'example.com') + self.assertIsNone(mailbox.route) + self.assertEqual(mailbox.addr_spec, 'dinsdale@example.com') + self.assertEqual(mailbox_list.mailboxes, + mailbox_list.all_mailboxes) + + def test_get_mailbox_list_two_simple_addr(self): + mailbox_list = self._test_get_x(parser.get_mailbox_list, + 'dinsdale@example.com, dinsdale@test.example.com', + 'dinsdale@example.com, dinsdale@test.example.com', + 'dinsdale@example.com, dinsdale@test.example.com', + [], + '') + self.assertEqual(mailbox_list.token_type, 'mailbox-list') + self.assertEqual(len(mailbox_list.mailboxes), 2) + self.assertEqual(mailbox_list.mailboxes[0].addr_spec, + 'dinsdale@example.com') + self.assertEqual(mailbox_list.mailboxes[1].addr_spec, + 'dinsdale@test.example.com') + self.assertEqual(mailbox_list.mailboxes, + mailbox_list.all_mailboxes) + + def test_get_mailbox_list_two_name_addr(self): + mailbox_list = self._test_get_x(parser.get_mailbox_list, + ('"Roy A. Bear" ,' + ' "Fred Flintstone" '), + ('"Roy A. Bear" ,' + ' "Fred Flintstone" '), + ('"Roy A. Bear" ,' + ' "Fred Flintstone" '), + [], + '') + self.assertEqual(len(mailbox_list.mailboxes), 2) + self.assertEqual(mailbox_list.mailboxes[0].addr_spec, + 'dinsdale@example.com') + self.assertEqual(mailbox_list.mailboxes[0].display_name, + 'Roy A. Bear') + self.assertEqual(mailbox_list.mailboxes[1].addr_spec, + 'dinsdale@test.example.com') + self.assertEqual(mailbox_list.mailboxes[1].display_name, + 'Fred Flintstone') + self.assertEqual(mailbox_list.mailboxes, + mailbox_list.all_mailboxes) + + def test_get_mailbox_list_two_complex(self): + mailbox_list = self._test_get_x(parser.get_mailbox_list, + ('(foo) "Roy A. Bear" (bar),' + ' "Fred Flintstone" '), + ('(foo) "Roy A. Bear" (bar),' + ' "Fred Flintstone" '), + (' "Roy A. Bear" ,' + ' "Fred Flintstone" '), + [errors.ObsoleteHeaderDefect], + '') + self.assertEqual(len(mailbox_list.mailboxes), 2) + self.assertEqual(mailbox_list.mailboxes[0].addr_spec, + 'dinsdale@example.com') + self.assertEqual(mailbox_list.mailboxes[0].display_name, + 'Roy A. Bear') + self.assertEqual(mailbox_list.mailboxes[1].addr_spec, + 'dinsdale@test.example.com') + self.assertEqual(mailbox_list.mailboxes[1].display_name, + 'Fred Flintstone') + self.assertEqual(mailbox_list.mailboxes, + mailbox_list.all_mailboxes) + + def test_get_mailbox_list_unparseable_mailbox_null(self): + mailbox_list = self._test_get_x(parser.get_mailbox_list, + ('"Roy A. Bear"[] dinsdale@example.com,' + ' "Fred Flintstone" '), + ('"Roy A. Bear"[] dinsdale@example.com,' + ' "Fred Flintstone" '), + ('"Roy A. Bear"[] dinsdale@example.com,' + ' "Fred Flintstone" '), + [errors.InvalidHeaderDefect, # the 'extra' text after the local part + errors.InvalidHeaderDefect, # the local part with no angle-addr + errors.ObsoleteHeaderDefect, # period in extra text (example.com) + errors.ObsoleteHeaderDefect], # (bird) in valid address. + '') + self.assertEqual(len(mailbox_list.mailboxes), 1) + self.assertEqual(len(mailbox_list.all_mailboxes), 2) + self.assertEqual(mailbox_list.all_mailboxes[0].token_type, + 'invalid-mailbox') + self.assertIsNone(mailbox_list.all_mailboxes[0].display_name) + self.assertEqual(mailbox_list.all_mailboxes[0].local_part, + 'Roy A. Bear') + self.assertIsNone(mailbox_list.all_mailboxes[0].domain) + self.assertEqual(mailbox_list.all_mailboxes[0].addr_spec, + '"Roy A. Bear"') + self.assertIs(mailbox_list.all_mailboxes[1], + mailbox_list.mailboxes[0]) + self.assertEqual(mailbox_list.mailboxes[0].addr_spec, + 'dinsdale@test.example.com') + self.assertEqual(mailbox_list.mailboxes[0].display_name, + 'Fred Flintstone') + + def test_get_mailbox_list_junk_after_valid_address(self): + mailbox_list = self._test_get_x(parser.get_mailbox_list, + ('"Roy A. Bear" @@,' + ' "Fred Flintstone" '), + ('"Roy A. Bear" @@,' + ' "Fred Flintstone" '), + ('"Roy A. Bear" @@,' + ' "Fred Flintstone" '), + [errors.InvalidHeaderDefect], + '') + self.assertEqual(len(mailbox_list.mailboxes), 1) + self.assertEqual(len(mailbox_list.all_mailboxes), 2) + self.assertEqual(mailbox_list.all_mailboxes[0].addr_spec, + 'dinsdale@example.com') + self.assertEqual(mailbox_list.all_mailboxes[0].display_name, + 'Roy A. Bear') + self.assertEqual(mailbox_list.all_mailboxes[0].token_type, + 'invalid-mailbox') + self.assertIs(mailbox_list.all_mailboxes[1], + mailbox_list.mailboxes[0]) + self.assertEqual(mailbox_list.mailboxes[0].addr_spec, + 'dinsdale@test.example.com') + self.assertEqual(mailbox_list.mailboxes[0].display_name, + 'Fred Flintstone') + + def test_get_mailbox_list_empty_list_element(self): + mailbox_list = self._test_get_x(parser.get_mailbox_list, + ('"Roy A. Bear" , (bird),,' + ' "Fred Flintstone" '), + ('"Roy A. Bear" , (bird),,' + ' "Fred Flintstone" '), + ('"Roy A. Bear" , ,,' + ' "Fred Flintstone" '), + [errors.ObsoleteHeaderDefect]*2, + '') + self.assertEqual(len(mailbox_list.mailboxes), 2) + self.assertEqual(mailbox_list.all_mailboxes, + mailbox_list.mailboxes) + self.assertEqual(mailbox_list.all_mailboxes[0].addr_spec, + 'dinsdale@example.com') + self.assertEqual(mailbox_list.all_mailboxes[0].display_name, + 'Roy A. Bear') + self.assertEqual(mailbox_list.mailboxes[1].addr_spec, + 'dinsdale@test.example.com') + self.assertEqual(mailbox_list.mailboxes[1].display_name, + 'Fred Flintstone') + + def test_get_mailbox_list_only_empty_elements(self): + mailbox_list = self._test_get_x(parser.get_mailbox_list, + '(foo),, (bar)', + '(foo),, (bar)', + ' ,, ', + [errors.ObsoleteHeaderDefect]*3, + '') + self.assertEqual(len(mailbox_list.mailboxes), 0) + self.assertEqual(mailbox_list.all_mailboxes, + mailbox_list.mailboxes) + + # get_group_list + + def test_get_group_list_cfws_only(self): + group_list = self._test_get_x(parser.get_group_list, + '(hidden);', + '(hidden)', + ' ', + [], + ';') + self.assertEqual(group_list.token_type, 'group-list') + self.assertEqual(len(group_list.mailboxes), 0) + self.assertEqual(group_list.mailboxes, + group_list.all_mailboxes) + + def test_get_group_list_mailbox_list(self): + group_list = self._test_get_x(parser.get_group_list, + 'dinsdale@example.org, "Fred A. Bear" ', + 'dinsdale@example.org, "Fred A. Bear" ', + 'dinsdale@example.org, "Fred A. Bear" ', + [], + '') + self.assertEqual(group_list.token_type, 'group-list') + self.assertEqual(len(group_list.mailboxes), 2) + self.assertEqual(group_list.mailboxes, + group_list.all_mailboxes) + self.assertEqual(group_list.mailboxes[1].display_name, + 'Fred A. Bear') + + def test_get_group_list_obs_group_list(self): + group_list = self._test_get_x(parser.get_group_list, + ', (foo),,(bar)', + ', (foo),,(bar)', + ', ,, ', + [errors.ObsoleteHeaderDefect], + '') + self.assertEqual(group_list.token_type, 'group-list') + self.assertEqual(len(group_list.mailboxes), 0) + self.assertEqual(group_list.mailboxes, + group_list.all_mailboxes) + + def test_get_group_list_comment_only_invalid(self): + group_list = self._test_get_x(parser.get_group_list, + '(bar)', + '(bar)', + ' ', + [errors.InvalidHeaderDefect], + '') + self.assertEqual(group_list.token_type, 'group-list') + self.assertEqual(len(group_list.mailboxes), 0) + self.assertEqual(group_list.mailboxes, + group_list.all_mailboxes) + + # get_group + + def test_get_group_empty(self): + group = self._test_get_x(parser.get_group, + 'Monty Python:;', + 'Monty Python:;', + 'Monty Python:;', + [], + '') + self.assertEqual(group.token_type, 'group') + self.assertEqual(group.display_name, 'Monty Python') + self.assertEqual(len(group.mailboxes), 0) + self.assertEqual(group.mailboxes, + group.all_mailboxes) + + def test_get_group_null_addr_spec(self): + group = self._test_get_x(parser.get_group, + 'foo: <>;', + 'foo: <>;', + 'foo: <>;', + [errors.InvalidHeaderDefect], + '') + self.assertEqual(group.display_name, 'foo') + self.assertEqual(len(group.mailboxes), 0) + self.assertEqual(len(group.all_mailboxes), 1) + self.assertEqual(group.all_mailboxes[0].value, '<>') + + def test_get_group_cfws_only(self): + group = self._test_get_x(parser.get_group, + 'Monty Python: (hidden);', + 'Monty Python: (hidden);', + 'Monty Python: ;', + [], + '') + self.assertEqual(group.token_type, 'group') + self.assertEqual(group.display_name, 'Monty Python') + self.assertEqual(len(group.mailboxes), 0) + self.assertEqual(group.mailboxes, + group.all_mailboxes) + + def test_get_group_single_mailbox(self): + group = self._test_get_x(parser.get_group, + 'Monty Python: "Fred A. Bear" ;', + 'Monty Python: "Fred A. Bear" ;', + 'Monty Python: "Fred A. Bear" ;', + [], + '') + self.assertEqual(group.token_type, 'group') + self.assertEqual(group.display_name, 'Monty Python') + self.assertEqual(len(group.mailboxes), 1) + self.assertEqual(group.mailboxes, + group.all_mailboxes) + self.assertEqual(group.mailboxes[0].addr_spec, + 'dinsdale@example.com') + + def test_get_group_mixed_list(self): + group = self._test_get_x(parser.get_group, + ('Monty Python: "Fred A. Bear" ,' + '(foo) Roger , x@test.example.com;'), + ('Monty Python: "Fred A. Bear" ,' + '(foo) Roger , x@test.example.com;'), + ('Monty Python: "Fred A. Bear" ,' + ' Roger , x@test.example.com;'), + [], + '') + self.assertEqual(group.token_type, 'group') + self.assertEqual(group.display_name, 'Monty Python') + self.assertEqual(len(group.mailboxes), 3) + self.assertEqual(group.mailboxes, + group.all_mailboxes) + self.assertEqual(group.mailboxes[0].display_name, + 'Fred A. Bear') + self.assertEqual(group.mailboxes[1].display_name, + 'Roger') + self.assertEqual(group.mailboxes[2].local_part, 'x') + + def test_get_group_one_invalid(self): + group = self._test_get_x(parser.get_group, + ('Monty Python: "Fred A. Bear" ,' + '(foo) Roger ping@exampele.com, x@test.example.com;'), + ('Monty Python: "Fred A. Bear" ,' + '(foo) Roger ping@exampele.com, x@test.example.com;'), + ('Monty Python: "Fred A. Bear" ,' + ' Roger ping@exampele.com, x@test.example.com;'), + [errors.InvalidHeaderDefect, # non-angle addr makes local part invalid + errors.InvalidHeaderDefect], # and its not obs-local either: no dots. + '') + self.assertEqual(group.token_type, 'group') + self.assertEqual(group.display_name, 'Monty Python') + self.assertEqual(len(group.mailboxes), 2) + self.assertEqual(len(group.all_mailboxes), 3) + self.assertEqual(group.mailboxes[0].display_name, + 'Fred A. Bear') + self.assertEqual(group.mailboxes[1].local_part, 'x') + self.assertIsNone(group.all_mailboxes[1].display_name) + + def test_get_group_missing_final_semicol(self): + group = self._test_get_x(parser.get_group, + ('Monty Python:"Fred A. Bear" ,' + 'eric@where.test,John '), + ('Monty Python:"Fred A. Bear" ,' + 'eric@where.test,John ;'), + ('Monty Python:"Fred A. Bear" ,' + 'eric@where.test,John ;'), + [errors.InvalidHeaderDefect], + '') + self.assertEqual(group.token_type, 'group') + self.assertEqual(group.display_name, 'Monty Python') + self.assertEqual(len(group.mailboxes), 3) + self.assertEqual(group.mailboxes, + group.all_mailboxes) + self.assertEqual(group.mailboxes[0].addr_spec, + 'dinsdale@example.com') + self.assertEqual(group.mailboxes[0].display_name, + 'Fred A. Bear') + self.assertEqual(group.mailboxes[1].addr_spec, + 'eric@where.test') + self.assertEqual(group.mailboxes[2].display_name, + 'John') + self.assertEqual(group.mailboxes[2].addr_spec, + 'jdoe@test') + # get_address + + def test_get_address_simple(self): + address = self._test_get_x(parser.get_address, + 'dinsdale@example.com', + 'dinsdale@example.com', + 'dinsdale@example.com', + [], + '') + self.assertEqual(address.token_type, 'address') + self.assertEqual(len(address.mailboxes), 1) + self.assertEqual(address.mailboxes, + address.all_mailboxes) + self.assertEqual(address.mailboxes[0].domain, + 'example.com') + self.assertEqual(address[0].token_type, + 'mailbox') + + def test_get_address_complex(self): + address = self._test_get_x(parser.get_address, + '(foo) "Fred A. Bear" <(bird)dinsdale@example.com>', + '(foo) "Fred A. Bear" <(bird)dinsdale@example.com>', + ' "Fred A. Bear" < dinsdale@example.com>', + [], + '') + self.assertEqual(address.token_type, 'address') + self.assertEqual(len(address.mailboxes), 1) + self.assertEqual(address.mailboxes, + address.all_mailboxes) + self.assertEqual(address.mailboxes[0].display_name, + 'Fred A. Bear') + self.assertEqual(address[0].token_type, + 'mailbox') + + def test_get_address_rfc2047_display_name(self): + address = self._test_get_x(parser.get_address, + '=?utf-8?q?=C3=89ric?= ', + 'Éric ', + 'Éric ', + [], + '') + self.assertEqual(address.token_type, 'address') + self.assertEqual(len(address.mailboxes), 1) + self.assertEqual(address.mailboxes, + address.all_mailboxes) + self.assertEqual(address.mailboxes[0].display_name, + 'Éric') + self.assertEqual(address[0].token_type, + 'mailbox') + + def test_get_address_empty_group(self): + address = self._test_get_x(parser.get_address, + 'Monty Python:;', + 'Monty Python:;', + 'Monty Python:;', + [], + '') + self.assertEqual(address.token_type, 'address') + self.assertEqual(len(address.mailboxes), 0) + self.assertEqual(address.mailboxes, + address.all_mailboxes) + self.assertEqual(address[0].token_type, + 'group') + self.assertEqual(address[0].display_name, + 'Monty Python') + + def test_get_address_group(self): + address = self._test_get_x(parser.get_address, + 'Monty Python: x@example.com, y@example.com;', + 'Monty Python: x@example.com, y@example.com;', + 'Monty Python: x@example.com, y@example.com;', + [], + '') + self.assertEqual(address.token_type, 'address') + self.assertEqual(len(address.mailboxes), 2) + self.assertEqual(address.mailboxes, + address.all_mailboxes) + self.assertEqual(address[0].token_type, + 'group') + self.assertEqual(address[0].display_name, + 'Monty Python') + self.assertEqual(address.mailboxes[0].local_part, 'x') + + def test_get_address_quoted_local_part(self): + address = self._test_get_x(parser.get_address, + '"foo bar"@example.com', + '"foo bar"@example.com', + '"foo bar"@example.com', + [], + '') + self.assertEqual(address.token_type, 'address') + self.assertEqual(len(address.mailboxes), 1) + self.assertEqual(address.mailboxes, + address.all_mailboxes) + self.assertEqual(address.mailboxes[0].domain, + 'example.com') + self.assertEqual(address.mailboxes[0].local_part, + 'foo bar') + self.assertEqual(address[0].token_type, 'mailbox') + + def test_get_address_ends_at_special(self): + address = self._test_get_x(parser.get_address, + 'dinsdale@example.com, next', + 'dinsdale@example.com', + 'dinsdale@example.com', + [], + ', next') + self.assertEqual(address.token_type, 'address') + self.assertEqual(len(address.mailboxes), 1) + self.assertEqual(address.mailboxes, + address.all_mailboxes) + self.assertEqual(address.mailboxes[0].domain, + 'example.com') + self.assertEqual(address[0].token_type, 'mailbox') + + def test_get_address_invalid_mailbox_invalid(self): + address = self._test_get_x(parser.get_address, + 'ping example.com, next', + 'ping example.com', + 'ping example.com', + [errors.InvalidHeaderDefect, # addr-spec with no domain + errors.InvalidHeaderDefect, # invalid local-part + errors.InvalidHeaderDefect, # missing .s in local-part + ], + ', next') + self.assertEqual(address.token_type, 'address') + self.assertEqual(len(address.mailboxes), 0) + self.assertEqual(len(address.all_mailboxes), 1) + self.assertIsNone(address.all_mailboxes[0].domain) + self.assertEqual(address.all_mailboxes[0].local_part, 'ping example.com') + self.assertEqual(address[0].token_type, 'invalid-mailbox') + + def test_get_address_quoted_strings_in_atom_list(self): + address = self._test_get_x(parser.get_address, + '""example" example"@example.com', + '""example" example"@example.com', + 'example example@example.com', + [errors.InvalidHeaderDefect]*3, + '') + self.assertEqual(address.all_mailboxes[0].local_part, 'example example') + self.assertEqual(address.all_mailboxes[0].domain, 'example.com') + self.assertEqual(address.all_mailboxes[0].addr_spec, '"example example"@example.com') + + def test_get_address_with_invalid_domain(self): + address = self._test_get_x(parser.get_address, + '', + '', + [errors.InvalidHeaderDefect, # missing trailing '>' on angle-addr + errors.InvalidHeaderDefect, # end of input inside domain-literal + ], + '') + self.assertEqual(address.token_type, 'address') + self.assertEqual(len(address.mailboxes), 0) + self.assertEqual(len(address.all_mailboxes), 1) + self.assertEqual(address.all_mailboxes[0].domain, '[]') + self.assertEqual(address.all_mailboxes[0].local_part, 'T') + self.assertEqual(address.all_mailboxes[0].token_type, 'invalid-mailbox') + self.assertEqual(address[0].token_type, 'invalid-mailbox') + + address = self._test_get_x(parser.get_address, + '!an??:=m==fr2@[C', + '!an??:=m==fr2@[C];', + '!an??:=m==fr2@[C];', + [errors.InvalidHeaderDefect, # end of header in group + errors.InvalidHeaderDefect, # end of input inside domain-literal + ], + '') + self.assertEqual(address.token_type, 'address') + self.assertEqual(len(address.mailboxes), 0) + self.assertEqual(len(address.all_mailboxes), 1) + self.assertEqual(address.all_mailboxes[0].domain, '[C]') + self.assertEqual(address.all_mailboxes[0].local_part, '=m==fr2') + self.assertEqual(address.all_mailboxes[0].token_type, 'invalid-mailbox') + self.assertEqual(address[0].token_type, 'group') + + # get_address_list + + def test_get_address_list_CFWS(self): + address_list = self._test_get_x(parser.get_address_list, + '(Recipient list suppressed)', + '(Recipient list suppressed)', + ' ', + [errors.ObsoleteHeaderDefect], # no content in address list + '') + self.assertEqual(address_list.token_type, 'address-list') + self.assertEqual(len(address_list.mailboxes), 0) + self.assertEqual(address_list.mailboxes, address_list.all_mailboxes) + + def test_get_address_list_mailboxes_simple(self): + address_list = self._test_get_x(parser.get_address_list, + 'dinsdale@example.com', + 'dinsdale@example.com', + 'dinsdale@example.com', + [], + '') + self.assertEqual(address_list.token_type, 'address-list') + self.assertEqual(len(address_list.mailboxes), 1) + self.assertEqual(address_list.mailboxes, + address_list.all_mailboxes) + self.assertEqual([str(x) for x in address_list.mailboxes], + [str(x) for x in address_list.addresses]) + self.assertEqual(address_list.mailboxes[0].domain, 'example.com') + self.assertEqual(address_list[0].token_type, 'address') + self.assertIsNone(address_list[0].display_name) + + def test_get_address_list_mailboxes_two_simple(self): + address_list = self._test_get_x(parser.get_address_list, + 'foo@example.com, "Fred A. Bar" ', + 'foo@example.com, "Fred A. Bar" ', + 'foo@example.com, "Fred A. Bar" ', + [], + '') + self.assertEqual(address_list.token_type, 'address-list') + self.assertEqual(len(address_list.mailboxes), 2) + self.assertEqual(address_list.mailboxes, + address_list.all_mailboxes) + self.assertEqual([str(x) for x in address_list.mailboxes], + [str(x) for x in address_list.addresses]) + self.assertEqual(address_list.mailboxes[0].local_part, 'foo') + self.assertEqual(address_list.mailboxes[1].display_name, "Fred A. Bar") + + def test_get_address_list_mailboxes_complex(self): + address_list = self._test_get_x(parser.get_address_list, + ('"Roy A. Bear" , ' + '(ping) Foo ,' + 'Nobody Is. Special '), + ('"Roy A. Bear" , ' + '(ping) Foo ,' + 'Nobody Is. Special '), + ('"Roy A. Bear" , ' + 'Foo ,' + '"Nobody Is. Special" '), + [errors.ObsoleteHeaderDefect, # period in Is. + errors.ObsoleteHeaderDefect], # cfws in domain + '') + self.assertEqual(address_list.token_type, 'address-list') + self.assertEqual(len(address_list.mailboxes), 3) + self.assertEqual(address_list.mailboxes, + address_list.all_mailboxes) + self.assertEqual([str(x) for x in address_list.mailboxes], + [str(x) for x in address_list.addresses]) + self.assertEqual(address_list.mailboxes[0].domain, 'example.com') + self.assertEqual(address_list.mailboxes[0].token_type, 'mailbox') + self.assertEqual(address_list.addresses[0].token_type, 'address') + self.assertEqual(address_list.mailboxes[1].local_part, 'x') + self.assertEqual(address_list.mailboxes[2].display_name, + 'Nobody Is. Special') + + def test_get_address_list_mailboxes_invalid_addresses(self): + address_list = self._test_get_x(parser.get_address_list, + ('"Roy A. Bear" , ' + '(ping) Foo x@example.com[],' + 'Nobody Is. Special <(bird)example.(bad)com>'), + ('"Roy A. Bear" , ' + '(ping) Foo x@example.com[],' + 'Nobody Is. Special <(bird)example.(bad)com>'), + ('"Roy A. Bear" , ' + 'Foo x@example.com[],' + '"Nobody Is. Special" < example. com>'), + [errors.InvalidHeaderDefect, # invalid address in list + errors.InvalidHeaderDefect, # 'Foo x' local part invalid. + errors.InvalidHeaderDefect, # Missing . in 'Foo x' local part + errors.ObsoleteHeaderDefect, # period in 'Is.' disp-name phrase + errors.InvalidHeaderDefect, # no domain part in addr-spec + errors.ObsoleteHeaderDefect], # addr-spec has comment in it + '') + self.assertEqual(address_list.token_type, 'address-list') + self.assertEqual(len(address_list.mailboxes), 1) + self.assertEqual(len(address_list.all_mailboxes), 3) + self.assertEqual([str(x) for x in address_list.all_mailboxes], + [str(x) for x in address_list.addresses]) + self.assertEqual(address_list.mailboxes[0].domain, 'example.com') + self.assertEqual(address_list.mailboxes[0].token_type, 'mailbox') + self.assertEqual(address_list.addresses[0].token_type, 'address') + self.assertEqual(address_list.addresses[1].token_type, 'address') + self.assertEqual(len(address_list.addresses[0].mailboxes), 1) + self.assertEqual(len(address_list.addresses[1].mailboxes), 0) + self.assertEqual(len(address_list.addresses[1].mailboxes), 0) + self.assertEqual( + address_list.addresses[1].all_mailboxes[0].local_part, 'Foo x') + self.assertEqual( + address_list.addresses[2].all_mailboxes[0].display_name, + "Nobody Is. Special") + + def test_get_address_list_group_empty(self): + address_list = self._test_get_x(parser.get_address_list, + 'Monty Python: ;', + 'Monty Python: ;', + 'Monty Python: ;', + [], + '') + self.assertEqual(address_list.token_type, 'address-list') + self.assertEqual(len(address_list.mailboxes), 0) + self.assertEqual(address_list.mailboxes, + address_list.all_mailboxes) + self.assertEqual(len(address_list.addresses), 1) + self.assertEqual(address_list.addresses[0].token_type, 'address') + self.assertEqual(address_list.addresses[0].display_name, 'Monty Python') + self.assertEqual(len(address_list.addresses[0].mailboxes), 0) + + def test_get_address_list_group_simple(self): + address_list = self._test_get_x(parser.get_address_list, + 'Monty Python: dinsdale@example.com;', + 'Monty Python: dinsdale@example.com;', + 'Monty Python: dinsdale@example.com;', + [], + '') + self.assertEqual(address_list.token_type, 'address-list') + self.assertEqual(len(address_list.mailboxes), 1) + self.assertEqual(address_list.mailboxes, + address_list.all_mailboxes) + self.assertEqual(address_list.mailboxes[0].domain, 'example.com') + self.assertEqual(address_list.addresses[0].display_name, + 'Monty Python') + self.assertEqual(address_list.addresses[0].mailboxes[0].domain, + 'example.com') + + def test_get_address_list_group_and_mailboxes(self): + address_list = self._test_get_x(parser.get_address_list, + ('Monty Python: dinsdale@example.com, "Fred" ;, ' + 'Abe , Bee '), + ('Monty Python: dinsdale@example.com, "Fred" ;, ' + 'Abe , Bee '), + ('Monty Python: dinsdale@example.com, "Fred" ;, ' + 'Abe , Bee '), + [], + '') + self.assertEqual(address_list.token_type, 'address-list') + self.assertEqual(len(address_list.mailboxes), 4) + self.assertEqual(address_list.mailboxes, + address_list.all_mailboxes) + self.assertEqual(len(address_list.addresses), 3) + self.assertEqual(address_list.mailboxes[0].local_part, 'dinsdale') + self.assertEqual(address_list.addresses[0].display_name, + 'Monty Python') + self.assertEqual(address_list.addresses[0].mailboxes[0].domain, + 'example.com') + self.assertEqual(address_list.addresses[0].mailboxes[1].local_part, + 'flint') + self.assertEqual(address_list.addresses[1].mailboxes[0].local_part, + 'x') + self.assertEqual(address_list.addresses[2].mailboxes[0].local_part, + 'y') + self.assertEqual(str(address_list.addresses[1]), + str(address_list.mailboxes[2])) + + def test_invalid_content_disposition(self): + content_disp = self._test_parse_x( + parser.parse_content_disposition_header, + ";attachment", "; attachment", ";attachment", + [errors.InvalidHeaderDefect]*2 + ) + + def test_invalid_content_transfer_encoding(self): + cte = self._test_parse_x( + parser.parse_content_transfer_encoding_header, + ";foo", ";foo", ";foo", [errors.InvalidHeaderDefect]*3 + ) + + # get_msg_id + + def test_get_msg_id_empty(self): + # bpo-38708: Test that HeaderParseError is raised and not IndexError. + with self.assertRaises(errors.HeaderParseError): + parser.get_msg_id('') + + def test_get_msg_id_valid(self): + msg_id = self._test_get_x( + parser.get_msg_id, + "", + "", + "", + [], + '', + ) + self.assertEqual(msg_id.token_type, 'msg-id') + + def test_get_msg_id_obsolete_local(self): + msg_id = self._test_get_x( + parser.get_msg_id, + '<"simeple.local"@example.com>', + '<"simeple.local"@example.com>', + '', + [errors.ObsoleteHeaderDefect], + '', + ) + self.assertEqual(msg_id.token_type, 'msg-id') + + def test_get_msg_id_non_folding_literal_domain(self): + msg_id = self._test_get_x( + parser.get_msg_id, + "", + "", + "", + [], + "", + ) + self.assertEqual(msg_id.token_type, 'msg-id') + + + def test_get_msg_id_obsolete_domain_part(self): + msg_id = self._test_get_x( + parser.get_msg_id, + "", + "", + "", + [errors.ObsoleteHeaderDefect], + "" + ) + + def test_get_msg_id_no_id_right_part(self): + msg_id = self._test_get_x( + parser.get_msg_id, + "", + "", + "", + [errors.InvalidHeaderDefect], + "" + ) + self.assertEqual(msg_id.token_type, 'msg-id') + + def test_get_msg_id_invalid_expected_msg_id_not_found(self): + text = "935-XPB-567:0:45327:9:90305:17843586-40@example.com" + msg_id = parser.parse_message_id(text) + self.assertDefectsEqual( + msg_id.all_defects, + [errors.InvalidHeaderDefect]) + + def test_parse_invalid_message_id(self): + message_id = self._test_parse_x( + parser.parse_message_id, + "935-XPB-567:0:45327:9:90305:17843586-40@example.com", + "935-XPB-567:0:45327:9:90305:17843586-40@example.com", + "935-XPB-567:0:45327:9:90305:17843586-40@example.com", + [errors.InvalidHeaderDefect], + ) + self.assertEqual(message_id.token_type, 'invalid-message-id') + + def test_parse_valid_message_id(self): + message_id = self._test_parse_x( + parser.parse_message_id, + "", + "", + "", + [], + ) + self.assertEqual(message_id.token_type, 'message-id') + + def test_parse_message_id_with_invalid_domain(self): + message_id = self._test_parse_x( + parser.parse_message_id, + "", + "", + [errors.ObsoleteHeaderDefect] + [errors.InvalidHeaderDefect] * 2, + [], + ) + self.assertEqual(message_id.token_type, 'message-id') + self.assertEqual(str(message_id.all_defects[-1]), + "end of input inside domain-literal") + + def test_parse_message_id_with_remaining(self): + message_id = self._test_parse_x( + parser.parse_message_id, + "thensomething", + "", + "", + [errors.InvalidHeaderDefect], + [], + ) + self.assertEqual(message_id.token_type, 'message-id') + self.assertEqual(str(message_id.all_defects[0]), + "Unexpected 'thensomething'") + + def test_get_msg_id_no_angle_start(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_msg_id("msgwithnoankle") + + def test_get_msg_id_no_angle_end(self): + msg_id = self._test_get_x( + parser.get_msg_id, + "", + "", + [errors.InvalidHeaderDefect], + "" + ) + self.assertEqual(msg_id.token_type, 'msg-id') + + def test_get_msg_id_empty_id_left(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_msg_id("<@domain>") + + def test_get_msg_id_empty_id_right(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_msg_id("") + + def test_get_msg_id_no_id_right(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_msg_id("") + + def test_get_msg_id_ws_only_local(self): + msg_id = self._test_get_x( + parser.get_msg_id, + "< @domain>", + "< @domain>", + "< @domain>", + [errors.ObsoleteHeaderDefect], + "" + ) + self.assertEqual(msg_id.token_type, 'msg-id') + + + +@parameterize +class Test_parse_mime_parameters(TestParserMixin, TestEmailBase): + + def mime_parameters_as_value(self, + value, + tl_str, + tl_value, + params, + defects): + mime_parameters = self._test_parse_x(parser.parse_mime_parameters, + value, tl_str, tl_value, defects) + self.assertEqual(mime_parameters.token_type, 'mime-parameters') + self.assertEqual(list(mime_parameters.params), params) + + + mime_parameters_params = { + + 'simple': ( + 'filename="abc.py"', + ' filename="abc.py"', + 'filename=abc.py', + [('filename', 'abc.py')], + []), + + 'multiple_keys': ( + 'filename="abc.py"; xyz=abc', + ' filename="abc.py"; xyz="abc"', + 'filename=abc.py; xyz=abc', + [('filename', 'abc.py'), ('xyz', 'abc')], + []), + + 'split_value': ( + "filename*0*=iso-8859-1''%32%30%31%2E; filename*1*=%74%69%66", + ' filename="201.tif"', + "filename*0*=iso-8859-1''%32%30%31%2E; filename*1*=%74%69%66", + [('filename', '201.tif')], + []), + + # Note that it is undefined what we should do for error recovery when + # there are duplicate parameter names or duplicate parts in a split + # part. We choose to ignore all duplicate parameters after the first + # and to take duplicate or missing rfc 2231 parts in appearance order. + # This is backward compatible with get_param's behavior, but the + # decisions are arbitrary. + + 'duplicate_key': ( + 'filename=abc.gif; filename=def.tiff', + ' filename="abc.gif"', + "filename=abc.gif; filename=def.tiff", + [('filename', 'abc.gif')], + [errors.InvalidHeaderDefect]), + + 'duplicate_key_with_split_value': ( + "filename*0*=iso-8859-1''%32%30%31%2E; filename*1*=%74%69%66;" + " filename=abc.gif", + ' filename="201.tif"', + "filename*0*=iso-8859-1''%32%30%31%2E; filename*1*=%74%69%66;" + " filename=abc.gif", + [('filename', '201.tif')], + [errors.InvalidHeaderDefect]), + + 'duplicate_key_with_split_value_other_order': ( + "filename=abc.gif; " + " filename*0*=iso-8859-1''%32%30%31%2E; filename*1*=%74%69%66", + ' filename="abc.gif"', + "filename=abc.gif;" + " filename*0*=iso-8859-1''%32%30%31%2E; filename*1*=%74%69%66", + [('filename', 'abc.gif')], + [errors.InvalidHeaderDefect]), + + 'duplicate_in_split_value': ( + "filename*0*=iso-8859-1''%32%30%31%2E; filename*1*=%74%69%66;" + " filename*1*=abc.gif", + ' filename="201.tifabc.gif"', + "filename*0*=iso-8859-1''%32%30%31%2E; filename*1*=%74%69%66;" + " filename*1*=abc.gif", + [('filename', '201.tifabc.gif')], + [errors.InvalidHeaderDefect]), + + 'missing_split_value': ( + "filename*0*=iso-8859-1''%32%30%31%2E; filename*3*=%74%69%66;", + ' filename="201.tif"', + "filename*0*=iso-8859-1''%32%30%31%2E; filename*3*=%74%69%66;", + [('filename', '201.tif')], + [errors.InvalidHeaderDefect]), + + 'duplicate_and_missing_split_value': ( + "filename*0*=iso-8859-1''%32%30%31%2E; filename*3*=%74%69%66;" + " filename*3*=abc.gif", + ' filename="201.tifabc.gif"', + "filename*0*=iso-8859-1''%32%30%31%2E; filename*3*=%74%69%66;" + " filename*3*=abc.gif", + [('filename', '201.tifabc.gif')], + [errors.InvalidHeaderDefect]*2), + + # Here we depart from get_param and assume the *0* was missing. + 'duplicate_with_broken_split_value': ( + "filename=abc.gif; " + " filename*2*=iso-8859-1''%32%30%31%2E; filename*3*=%74%69%66", + ' filename="abc.gif201.tif"', + "filename=abc.gif;" + " filename*2*=iso-8859-1''%32%30%31%2E; filename*3*=%74%69%66", + [('filename', 'abc.gif201.tif')], + # Defects are apparent missing *0*, and two 'out of sequence'. + [errors.InvalidHeaderDefect]*3), + + # bpo-37461: Check that we don't go into an infinite loop. + 'extra_dquote': ( + 'r*="\'a\'\\"', + ' r="\\""', + 'r*=\'a\'"', + [('r', '"')], + [errors.InvalidHeaderDefect]*2), + } + +@parameterize +class Test_parse_mime_version(TestParserMixin, TestEmailBase): + + def mime_version_as_value(self, + value, + tl_str, + tl_value, + major, + minor, + defects): + mime_version = self._test_parse_x(parser.parse_mime_version, + value, tl_str, tl_value, defects) + self.assertEqual(mime_version.major, major) + self.assertEqual(mime_version.minor, minor) + + mime_version_params = { + + 'rfc_2045_1': ( + '1.0', + '1.0', + '1.0', + 1, + 0, + []), + + 'RFC_2045_2': ( + '1.0 (produced by MetaSend Vx.x)', + '1.0 (produced by MetaSend Vx.x)', + '1.0 ', + 1, + 0, + []), + + 'RFC_2045_3': ( + '(produced by MetaSend Vx.x) 1.0', + '(produced by MetaSend Vx.x) 1.0', + ' 1.0', + 1, + 0, + []), + + 'RFC_2045_4': ( + '1.(produced by MetaSend Vx.x)0', + '1.(produced by MetaSend Vx.x)0', + '1. 0', + 1, + 0, + []), + + 'empty': ( + '', + '', + '', + None, + None, + [errors.HeaderMissingRequiredValue]), + + } + + + +class TestFolding(TestEmailBase): + + policy = policy.default + + def _test(self, tl, folded, policy=policy): + self.assertEqual(tl.fold(policy=policy), folded, tl.ppstr()) + + def test_simple_unstructured_no_folds(self): + self._test(parser.get_unstructured("This is a test"), + "This is a test\n") + + def test_simple_unstructured_folded(self): + self._test(parser.get_unstructured("This is also a test, but this " + "time there are enough words (and even some " + "symbols) to make it wrap; at least in theory."), + "This is also a test, but this time there are enough " + "words (and even some\n" + " symbols) to make it wrap; at least in theory.\n") + + def test_unstructured_with_unicode_no_folds(self): + self._test(parser.get_unstructured("hübsch kleiner beißt"), + "=?utf-8?q?h=C3=BCbsch_kleiner_bei=C3=9Ft?=\n") + + def test_one_ew_on_each_of_two_wrapped_lines(self): + self._test(parser.get_unstructured("Mein kleiner Kaktus ist sehr " + "hübsch. Es hat viele Stacheln " + "und oft beißt mich."), + "Mein kleiner Kaktus ist sehr =?utf-8?q?h=C3=BCbsch=2E?= " + "Es hat viele Stacheln\n" + " und oft =?utf-8?q?bei=C3=9Ft?= mich.\n") + + def test_ews_combined_before_wrap(self): + self._test(parser.get_unstructured("Mein Kaktus ist hübsch. " + "Es beißt mich. " + "And that's all I'm sayin."), + "Mein Kaktus ist =?utf-8?q?h=C3=BCbsch=2E__Es_bei=C3=9Ft?= " + "mich. And that's\n" + " all I'm sayin.\n") + + def test_unicode_after_unknown_not_combined(self): + self._test(parser.get_unstructured("=?unknown-8bit?q?=A4?=\xa4"), + "=?unknown-8bit?q?=A4?==?utf-8?q?=C2=A4?=\n") + prefix = "0123456789 "*5 + self._test(parser.get_unstructured(prefix + "=?unknown-8bit?q?=A4?=\xa4"), + prefix + "=?unknown-8bit?q?=A4?=\n =?utf-8?q?=C2=A4?=\n") + + def test_ascii_after_unknown_not_combined(self): + self._test(parser.get_unstructured("=?unknown-8bit?q?=A4?=abc"), + "=?unknown-8bit?q?=A4?=abc\n") + prefix = "0123456789 "*5 + self._test(parser.get_unstructured(prefix + "=?unknown-8bit?q?=A4?=abc"), + prefix + "=?unknown-8bit?q?=A4?=\n =?utf-8?q?abc?=\n") + + def test_unknown_after_unicode_not_combined(self): + self._test(parser.get_unstructured("\xa4" + "=?unknown-8bit?q?=A4?="), + "=?utf-8?q?=C2=A4?==?unknown-8bit?q?=A4?=\n") + prefix = "0123456789 "*5 + self._test(parser.get_unstructured(prefix + "\xa4=?unknown-8bit?q?=A4?="), + prefix + "=?utf-8?q?=C2=A4?=\n =?unknown-8bit?q?=A4?=\n") + + def test_unknown_after_ascii_not_combined(self): + self._test(parser.get_unstructured("abc" + "=?unknown-8bit?q?=A4?="), + "abc=?unknown-8bit?q?=A4?=\n") + prefix = "0123456789 "*5 + self._test(parser.get_unstructured(prefix + "abcd=?unknown-8bit?q?=A4?="), + prefix + "abcd\n =?unknown-8bit?q?=A4?=\n") + + def test_unknown_after_unknown(self): + self._test(parser.get_unstructured("=?unknown-8bit?q?=C2?=" + "=?unknown-8bit?q?=A4?="), + "=?unknown-8bit?q?=C2=A4?=\n") + prefix = "0123456789 "*5 + self._test(parser.get_unstructured(prefix + "=?unknown-8bit?q?=C2?=" + "=?unknown-8bit?q?=A4?="), + prefix + "=?unknown-8bit?q?=C2?=\n =?unknown-8bit?q?=A4?=\n") + + # XXX Need test of an encoded word so long that it needs to be wrapped + + def test_simple_address(self): + self._test(parser.get_address_list("abc ")[0], + "abc \n") + + def test_address_list_folding_at_commas(self): + self._test(parser.get_address_list('abc , ' + '"Fred Blunt" , ' + '"J.P.Cool" , ' + '"K<>y" , ' + 'Firesale , ' + '')[0], + 'abc , "Fred Blunt" ,\n' + ' "J.P.Cool" , "K<>y" ,\n' + ' Firesale , \n') + + def test_address_list_with_unicode_names(self): + self._test(parser.get_address_list( + 'Hübsch Kaktus , ' + 'beißt beißt ')[0], + '=?utf-8?q?H=C3=BCbsch?= Kaktus ,\n' + ' =?utf-8?q?bei=C3=9Ft_bei=C3=9Ft?= \n') + + def test_address_list_with_unicode_names_in_quotes(self): + self._test(parser.get_address_list( + '"Hübsch Kaktus" , ' + '"beißt" beißt ')[0], + '=?utf-8?q?H=C3=BCbsch?= Kaktus ,\n' + ' =?utf-8?q?bei=C3=9Ft_bei=C3=9Ft?= \n') + + def test_address_list_with_specials_in_encoded_word(self): + # An encoded-word parsed from a structured header must remain + # encoded when it contains specials. Regression for gh-121284. + policy = self.policy.clone(max_line_length=40) + cases = [ + # (to, folded) + ('=?utf-8?q?A_v=C3=A9ry_long_name_with=2C_comma?= ', + 'A =?utf-8?q?v=C3=A9ry_long_name_with?=\n' + ' =?utf-8?q?=2C?= comma \n'), + ('=?utf-8?q?This_long_name_does_not_need_encoded=2Dword?= ', + 'This long name does not need\n' + ' encoded-word \n'), + ('"A véry long name with, comma" ', + # (This isn't the best fold point, but it's not invalid.) + 'A =?utf-8?q?v=C3=A9ry_long_name_with?=\n' + ' =?utf-8?q?=2C?= comma \n'), + ('"A véry long name containing a, comma" ', + 'A =?utf-8?q?v=C3=A9ry?= long name\n' + ' containing =?utf-8?q?a=2C?= comma\n' + ' \n'), + ] + for (to, folded) in cases: + with self.subTest(to=to): + self._test(parser.get_address_list(to)[0], folded, policy=policy) + + def test_address_list_with_list_separator_after_fold(self): + a = 'x' * 66 + '@example.com' + to = f'{a}, "Hübsch Kaktus" ' + self._test(parser.get_address_list(to)[0], + f'{a},\n =?utf-8?q?H=C3=BCbsch?= Kaktus \n') + + a = '.' * 79 # ('.' is a special, so must be in quoted-string.) + to = f'"{a}" , "Hübsch Kaktus" ' + self._test(parser.get_address_list(to)[0], + f'"{a}"\n' + ' , =?utf-8?q?H=C3=BCbsch?= Kaktus ' + '\n') + + def test_address_list_with_specials_in_long_quoted_string(self): + # Regression for gh-80222. + policy = self.policy.clone(max_line_length=40) + cases = [ + # (to, folded) + ('"Exfiltrator (unclosed comment?" ', + '"Exfiltrator (unclosed\n' + ' comment?" \n'), + ('"Escaped \\" chars \\\\ in quoted-string stay escaped" ', + '"Escaped \\" chars \\\\ in quoted-string\n' + ' stay escaped" \n'), + ('This long display name does not need quotes ', + 'This long display name does not need\n' + ' quotes \n'), + ('"Quotes are not required but are retained here" ', + '"Quotes are not required but are\n' + ' retained here" \n'), + ('"A quoted-string, it can be a valid local-part"@example.com', + '"A quoted-string, it can be a valid\n' + ' local-part"@example.com\n'), + ('"local-part-with-specials@but-no-fws.cannot-fold"@example.com', + '"local-part-with-specials@but-no-fws.cannot-fold"@example.com\n'), + ] + for (to, folded) in cases: + with self.subTest(to=to): + self._test(parser.get_address_list(to)[0], folded, policy=policy) + + # XXX Need tests with comments on various sides of a unicode token, + # and with unicode tokens in the comments. Spaces inside the quotes + # currently don't do the right thing. + + def test_split_at_whitespace_after_header_before_long_token(self): + body = parser.get_unstructured(' ' + 'x'*77) + header = parser.Header([ + parser.HeaderLabel([parser.ValueTerminal('test:', 'atext')]), + parser.CFWSList([parser.WhiteSpaceTerminal(' ', 'fws')]), body]) + self._test(header, 'test: \n ' + 'x'*77 + '\n') + + def test_split_at_whitespace_before_long_token(self): + self._test(parser.get_unstructured('xxx ' + 'y'*77), + 'xxx \n ' + 'y'*77 + '\n') + + def test_overlong_encodeable_is_wrapped(self): + first_token_with_whitespace = 'xxx ' + chrome_leader = '=?utf-8?q?' + len_chrome = len(chrome_leader) + 2 + len_non_y = len_chrome + len(first_token_with_whitespace) + self._test(parser.get_unstructured(first_token_with_whitespace + + 'y'*80), + first_token_with_whitespace + chrome_leader + + 'y'*(78-len_non_y) + '?=\n' + + ' ' + chrome_leader + 'y'*(80-(78-len_non_y)) + '?=\n') + + def test_long_filename_attachment(self): + self._test(parser.parse_content_disposition_header( + 'attachment; filename="TEST_TEST_TEST_TEST' + '_TEST_TEST_TEST_TEST_TEST_TEST_TEST_TEST_TES.txt"'), + "attachment;\n" + " filename*0*=us-ascii''TEST_TEST_TEST_TEST_TEST_TEST" + "_TEST_TEST_TEST_TEST_TEST;\n" + " filename*1*=_TEST_TES.txt\n", + ) + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test_asian_codecs.py b/Lib/test/test_email/test_asian_codecs.py new file mode 100644 index 00000000000..1e0caeeaed0 --- /dev/null +++ b/Lib/test/test_email/test_asian_codecs.py @@ -0,0 +1,81 @@ +# Copyright (C) 2002-2006 Python Software Foundation +# Contact: email-sig@python.org +# email package unit tests for (optional) Asian codecs + +import unittest + +from test.test_email import TestEmailBase +from email.charset import Charset +from email.header import Header, decode_header +from email.message import Message + +# We're compatible with Python 2.3, but it doesn't have the built-in Asian +# codecs, so we have to skip all these tests. +try: + str(b'foo', 'euc-jp') +except LookupError: + raise unittest.SkipTest + + + +class TestEmailAsianCodecs(TestEmailBase): + def test_japanese_codecs(self): + eq = self.ndiffAssertEqual + jcode = "euc-jp" + gcode = "iso-8859-1" + j = Charset(jcode) + g = Charset(gcode) + h = Header("Hello World!") + jhello = str(b'\xa5\xcf\xa5\xed\xa1\xbc\xa5\xef\xa1\xbc' + b'\xa5\xeb\xa5\xc9\xa1\xaa', jcode) + ghello = str(b'Gr\xfc\xdf Gott!', gcode) + h.append(jhello, j) + h.append(ghello, g) + # BAW: This used to -- and maybe should -- fold the two iso-8859-1 + # chunks into a single encoded word. However it doesn't violate the + # standard to have them as two encoded chunks and maybe it's + # reasonable for each .append() call to result in a separate + # encoded word. + eq(h.encode(), """\ +Hello World! =?iso-2022-jp?b?GyRCJU8lbSE8JW8hPCVrJUkhKhsoQg==?= + =?iso-8859-1?q?Gr=FC=DF_Gott!?=""") + eq(decode_header(h.encode()), + [(b'Hello World! ', None), + (b'\x1b$B%O%m!<%o!<%k%I!*\x1b(B', 'iso-2022-jp'), + (b'Gr\xfc\xdf Gott!', gcode)]) + subject_bytes = (b'test-ja \xa4\xd8\xc5\xea\xb9\xc6\xa4\xb5' + b'\xa4\xec\xa4\xbf\xa5\xe1\xa1\xbc\xa5\xeb\xa4\xcf\xbb\xca\xb2' + b'\xf1\xbc\xd4\xa4\xce\xbe\xb5\xc7\xa7\xa4\xf2\xc2\xd4\xa4\xc3' + b'\xa4\xc6\xa4\xa4\xa4\xde\xa4\xb9') + subject = str(subject_bytes, jcode) + h = Header(subject, j, header_name="Subject") + # test a very long header + enc = h.encode() + # TK: splitting point may differ by codec design and/or Header encoding + eq(enc , """\ +=?iso-2022-jp?b?dGVzdC1qYSAbJEIkWEVqOUYkNSRsJD8lYSE8JWskTztKGyhC?= + =?iso-2022-jp?b?GyRCMnE8VCROPjVHJyRyQlQkQyRGJCQkXiQ5GyhC?=""") + # TK: full decode comparison + eq(str(h).encode(jcode), subject_bytes) + + def test_payload_encoding_utf8(self): + jhello = str(b'\xa5\xcf\xa5\xed\xa1\xbc\xa5\xef\xa1\xbc' + b'\xa5\xeb\xa5\xc9\xa1\xaa', 'euc-jp') + msg = Message() + msg.set_payload(jhello, 'utf-8') + ustr = msg.get_payload(decode=True).decode(msg.get_content_charset()) + self.assertEqual(jhello, ustr) + + def test_payload_encoding(self): + jcode = 'euc-jp' + jhello = str(b'\xa5\xcf\xa5\xed\xa1\xbc\xa5\xef\xa1\xbc' + b'\xa5\xeb\xa5\xc9\xa1\xaa', jcode) + msg = Message() + msg.set_payload(jhello, jcode) + ustr = msg.get_payload(decode=True).decode(msg.get_content_charset()) + self.assertEqual(jhello, ustr) + + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test_contentmanager.py b/Lib/test/test_email/test_contentmanager.py new file mode 100644 index 00000000000..dceb54f15e4 --- /dev/null +++ b/Lib/test/test_email/test_contentmanager.py @@ -0,0 +1,836 @@ +import unittest +from test.test_email import TestEmailBase, parameterize +import textwrap +from email import policy +from email.message import EmailMessage +from email.contentmanager import ContentManager, raw_data_manager + + +@parameterize +class TestContentManager(TestEmailBase): + + policy = policy.default + message = EmailMessage + + get_key_params = { + 'full_type': (1, 'text/plain',), + 'maintype_only': (2, 'text',), + 'null_key': (3, '',), + } + + def get_key_as_get_content_key(self, order, key): + def foo_getter(msg, foo=None): + bar = msg['X-Bar-Header'] + return foo, bar + cm = ContentManager() + cm.add_get_handler(key, foo_getter) + m = self._make_message() + m['Content-Type'] = 'text/plain' + m['X-Bar-Header'] = 'foo' + self.assertEqual(cm.get_content(m, foo='bar'), ('bar', 'foo')) + + def get_key_as_get_content_key_order(self, order, key): + def bar_getter(msg): + return msg['X-Bar-Header'] + def foo_getter(msg): + return msg['X-Foo-Header'] + cm = ContentManager() + cm.add_get_handler(key, foo_getter) + for precedence, key in self.get_key_params.values(): + if precedence > order: + cm.add_get_handler(key, bar_getter) + m = self._make_message() + m['Content-Type'] = 'text/plain' + m['X-Bar-Header'] = 'bar' + m['X-Foo-Header'] = 'foo' + self.assertEqual(cm.get_content(m), ('foo')) + + def test_get_content_raises_if_unknown_mimetype_and_no_default(self): + cm = ContentManager() + m = self._make_message() + m['Content-Type'] = 'text/plain' + with self.assertRaisesRegex(KeyError, 'text/plain'): + cm.get_content(m) + + class BaseThing(str): + pass + baseobject_full_path = __name__ + '.' + 'TestContentManager.BaseThing' + class Thing(BaseThing): + pass + testobject_full_path = __name__ + '.' + 'TestContentManager.Thing' + + set_key_params = { + 'type': (0, Thing,), + 'full_path': (1, testobject_full_path,), + 'qualname': (2, 'TestContentManager.Thing',), + 'name': (3, 'Thing',), + 'base_type': (4, BaseThing,), + 'base_full_path': (5, baseobject_full_path,), + 'base_qualname': (6, 'TestContentManager.BaseThing',), + 'base_name': (7, 'BaseThing',), + 'str_type': (8, str,), + 'str_full_path': (9, 'builtins.str',), + 'str_name': (10, 'str',), # str name and qualname are the same + 'null_key': (11, None,), + } + + def set_key_as_set_content_key(self, order, key): + def foo_setter(msg, obj, foo=None): + msg['X-Foo-Header'] = foo + msg.set_payload(obj) + cm = ContentManager() + cm.add_set_handler(key, foo_setter) + m = self._make_message() + msg_obj = self.Thing() + cm.set_content(m, msg_obj, foo='bar') + self.assertEqual(m['X-Foo-Header'], 'bar') + self.assertEqual(m.get_payload(), msg_obj) + + def set_key_as_set_content_key_order(self, order, key): + def foo_setter(msg, obj): + msg['X-FooBar-Header'] = 'foo' + msg.set_payload(obj) + def bar_setter(msg, obj): + msg['X-FooBar-Header'] = 'bar' + cm = ContentManager() + cm.add_set_handler(key, foo_setter) + for precedence, key in self.get_key_params.values(): + if precedence > order: + cm.add_set_handler(key, bar_setter) + m = self._make_message() + msg_obj = self.Thing() + cm.set_content(m, msg_obj) + self.assertEqual(m['X-FooBar-Header'], 'foo') + self.assertEqual(m.get_payload(), msg_obj) + + def test_set_content_raises_if_unknown_type_and_no_default(self): + cm = ContentManager() + m = self._make_message() + msg_obj = self.Thing() + with self.assertRaisesRegex(KeyError, self.testobject_full_path): + cm.set_content(m, msg_obj) + + def test_set_content_raises_if_called_on_multipart(self): + cm = ContentManager() + m = self._make_message() + m['Content-Type'] = 'multipart/foo' + with self.assertRaises(TypeError): + cm.set_content(m, 'test') + + def test_set_content_calls_clear_content(self): + m = self._make_message() + m['Content-Foo'] = 'bar' + m['Content-Type'] = 'text/html' + m['To'] = 'test' + m.set_payload('abc') + cm = ContentManager() + cm.add_set_handler(str, lambda *args, **kw: None) + m.set_content('xyz', content_manager=cm) + self.assertIsNone(m['Content-Foo']) + self.assertIsNone(m['Content-Type']) + self.assertEqual(m['To'], 'test') + self.assertIsNone(m.get_payload()) + + +@parameterize +class TestRawDataManager(TestEmailBase): + # Note: these tests are dependent on the order in which headers are added + # to the message objects by the code. There's no defined ordering in + # RFC5322/MIME, so this makes the tests more fragile than the standards + # require. However, if the header order changes it is best to understand + # *why*, and make sure it isn't a subtle bug in whatever change was + # applied. + + policy = policy.default.clone(max_line_length=60, + content_manager=raw_data_manager) + message = EmailMessage + + def test_get_text_plain(self): + m = self._str_msg(textwrap.dedent("""\ + Content-Type: text/plain + + Basic text. + """)) + self.assertEqual(raw_data_manager.get_content(m), "Basic text.\n") + + def test_get_text_html(self): + m = self._str_msg(textwrap.dedent("""\ + Content-Type: text/html + +

    Basic text.

    + """)) + self.assertEqual(raw_data_manager.get_content(m), + "

    Basic text.

    \n") + + def test_get_text_plain_latin1(self): + m = self._bytes_msg(textwrap.dedent("""\ + Content-Type: text/plain; charset=latin1 + + Basìc tëxt. + """).encode('latin1')) + self.assertEqual(raw_data_manager.get_content(m), "Basìc tëxt.\n") + + def test_get_text_plain_latin1_quoted_printable(self): + m = self._str_msg(textwrap.dedent("""\ + Content-Type: text/plain; charset="latin-1" + Content-Transfer-Encoding: quoted-printable + + Bas=ECc t=EBxt. + """)) + self.assertEqual(raw_data_manager.get_content(m), "Basìc tëxt.\n") + + def test_get_text_plain_utf8_base64(self): + m = self._str_msg(textwrap.dedent("""\ + Content-Type: text/plain; charset="utf8" + Content-Transfer-Encoding: base64 + + QmFzw6xjIHTDq3h0Lgo= + """)) + self.assertEqual(raw_data_manager.get_content(m), "Basìc tëxt.\n") + + def test_get_text_plain_bad_utf8_quoted_printable(self): + m = self._str_msg(textwrap.dedent("""\ + Content-Type: text/plain; charset="utf8" + Content-Transfer-Encoding: quoted-printable + + Bas=c3=acc t=c3=abxt=fd. + """)) + self.assertEqual(raw_data_manager.get_content(m), "Basìc tëxt�.\n") + + def test_get_text_plain_bad_utf8_quoted_printable_ignore_errors(self): + m = self._str_msg(textwrap.dedent("""\ + Content-Type: text/plain; charset="utf8" + Content-Transfer-Encoding: quoted-printable + + Bas=c3=acc t=c3=abxt=fd. + """)) + self.assertEqual(raw_data_manager.get_content(m, errors='ignore'), + "Basìc tëxt.\n") + + def test_get_text_plain_utf8_base64_recoverable_bad_CTE_data(self): + m = self._str_msg(textwrap.dedent("""\ + Content-Type: text/plain; charset="utf8" + Content-Transfer-Encoding: base64 + + QmFzw6xjIHTDq3h0Lgo\xFF= + """)) + self.assertEqual(raw_data_manager.get_content(m, errors='ignore'), + "Basìc tëxt.\n") + + def test_get_text_invalid_keyword(self): + m = self._str_msg(textwrap.dedent("""\ + Content-Type: text/plain + + Basic text. + """)) + with self.assertRaises(TypeError): + raw_data_manager.get_content(m, foo='ignore') + + def test_get_non_text(self): + template = textwrap.dedent("""\ + Content-Type: {} + Content-Transfer-Encoding: base64 + + Ym9ndXMgZGF0YQ== + """) + for maintype in 'audio image video application'.split(): + with self.subTest(maintype=maintype): + m = self._str_msg(template.format(maintype+'/foo')) + self.assertEqual(raw_data_manager.get_content(m), b"bogus data") + + def test_get_non_text_invalid_keyword(self): + m = self._str_msg(textwrap.dedent("""\ + Content-Type: image/jpg + Content-Transfer-Encoding: base64 + + Ym9ndXMgZGF0YQ== + """)) + with self.assertRaises(TypeError): + raw_data_manager.get_content(m, errors='ignore') + + def test_get_raises_on_multipart(self): + m = self._str_msg(textwrap.dedent("""\ + Content-Type: multipart/mixed; boundary="===" + + --=== + --===-- + """)) + with self.assertRaises(KeyError): + raw_data_manager.get_content(m) + + def test_get_message_rfc822_and_external_body(self): + template = textwrap.dedent("""\ + Content-Type: message/{} + + To: foo@example.com + From: bar@example.com + Subject: example + + an example message + """) + for subtype in 'rfc822 external-body'.split(): + with self.subTest(subtype=subtype): + m = self._str_msg(template.format(subtype)) + sub_msg = raw_data_manager.get_content(m) + self.assertIsInstance(sub_msg, self.message) + self.assertEqual(raw_data_manager.get_content(sub_msg), + "an example message\n") + self.assertEqual(sub_msg['to'], 'foo@example.com') + self.assertEqual(sub_msg['from'].addresses[0].username, 'bar') + + def test_get_message_non_rfc822_or_external_body_yields_bytes(self): + m = self._str_msg(textwrap.dedent("""\ + Content-Type: message/partial + + To: foo@example.com + From: bar@example.com + Subject: example + + The real body is in another message. + """)) + self.assertStartsWith(raw_data_manager.get_content(m), b'To: foo@ex') + + def test_set_text_plain(self): + m = self._make_message() + content = "Simple message.\n" + raw_data_manager.set_content(m, content) + self.assertEqual(str(m), textwrap.dedent("""\ + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: 7bit + + Simple message. + """)) + self.assertEqual(m.get_payload(decode=True).decode('utf-8'), content) + self.assertEqual(m.get_content(), content) + + def test_set_text_plain_null(self): + m = self._make_message() + content = '' + raw_data_manager.set_content(m, content) + self.assertEqual(str(m), textwrap.dedent("""\ + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: 7bit + + + """)) + self.assertEqual(m.get_payload(decode=True).decode('utf-8'), '\n') + self.assertEqual(m.get_content(), '\n') + + def test_set_text_html(self): + m = self._make_message() + content = "

    Simple message.

    \n" + raw_data_manager.set_content(m, content, subtype='html') + self.assertEqual(str(m), textwrap.dedent("""\ + Content-Type: text/html; charset="utf-8" + Content-Transfer-Encoding: 7bit + +

    Simple message.

    + """)) + self.assertEqual(m.get_payload(decode=True).decode('utf-8'), content) + self.assertEqual(m.get_content(), content) + + def test_set_text_charset_latin_1(self): + m = self._make_message() + content = "Simple message.\n" + raw_data_manager.set_content(m, content, charset='latin-1') + self.assertEqual(str(m), textwrap.dedent("""\ + Content-Type: text/plain; charset="iso-8859-1" + Content-Transfer-Encoding: 7bit + + Simple message. + """)) + self.assertEqual(m.get_payload(decode=True).decode('utf-8'), content) + self.assertEqual(m.get_content(), content) + + def test_set_text_plain_long_line_heuristics(self): + m = self._make_message() + content = ("Simple but long message that is over 78 characters" + " long to force transfer encoding.\n") + raw_data_manager.set_content(m, content) + self.assertEqual(str(m), textwrap.dedent("""\ + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: quoted-printable + + Simple but long message that is over 78 characters long to = + force transfer encoding. + """)) + self.assertEqual(m.get_payload(decode=True).decode('utf-8'), content) + self.assertEqual(m.get_content(), content) + + def test_set_text_short_line_minimal_non_ascii_heuristics(self): + m = self._make_message() + content = "et là il est monté sur moi et il commence à m'éto.\n" + raw_data_manager.set_content(m, content) + self.assertEqual(bytes(m), textwrap.dedent("""\ + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: 8bit + + et là il est monté sur moi et il commence à m'éto. + """).encode('utf-8')) + self.assertEqual(m.get_payload(decode=True).decode('utf-8'), content) + self.assertEqual(m.get_content(), content) + + def test_set_text_long_line_minimal_non_ascii_heuristics(self): + m = self._make_message() + content = ("j'ai un problème de python. il est sorti de son" + " vivarium. et là il est monté sur moi et il commence" + " à m'éto.\n") + raw_data_manager.set_content(m, content) + self.assertEqual(bytes(m), textwrap.dedent("""\ + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: quoted-printable + + j'ai un probl=C3=A8me de python. il est sorti de son vivari= + um. et l=C3=A0 il est mont=C3=A9 sur moi et il commence = + =C3=A0 m'=C3=A9to. + """).encode('utf-8')) + self.assertEqual(m.get_payload(decode=True).decode('utf-8'), content) + self.assertEqual(m.get_content(), content) + + def test_set_text_11_lines_long_line_minimal_non_ascii_heuristics(self): + m = self._make_message() + content = '\n'*10 + ( + "j'ai un problème de python. il est sorti de son" + " vivarium. et là il est monté sur moi et il commence" + " à m'éto.\n") + raw_data_manager.set_content(m, content) + self.assertEqual(bytes(m), textwrap.dedent("""\ + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: quoted-printable + """ + '\n'*10 + """ + j'ai un probl=C3=A8me de python. il est sorti de son vivari= + um. et l=C3=A0 il est mont=C3=A9 sur moi et il commence = + =C3=A0 m'=C3=A9to. + """).encode('utf-8')) + self.assertEqual(m.get_payload(decode=True).decode('utf-8'), content) + self.assertEqual(m.get_content(), content) + + def test_set_text_maximal_non_ascii_heuristics(self): + m = self._make_message() + content = "áàäéèęöő.\n" + raw_data_manager.set_content(m, content) + self.assertEqual(bytes(m), textwrap.dedent("""\ + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: 8bit + + áàäéèęöő. + """).encode('utf-8')) + self.assertEqual(m.get_payload(decode=True).decode('utf-8'), content) + self.assertEqual(m.get_content(), content) + + def test_set_text_11_lines_maximal_non_ascii_heuristics(self): + m = self._make_message() + content = '\n'*10 + "áàäéèęöő.\n" + raw_data_manager.set_content(m, content) + self.assertEqual(bytes(m), textwrap.dedent("""\ + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: 8bit + """ + '\n'*10 + """ + áàäéèęöő. + """).encode('utf-8')) + self.assertEqual(m.get_payload(decode=True).decode('utf-8'), content) + self.assertEqual(m.get_content(), content) + + def test_set_text_long_line_maximal_non_ascii_heuristics(self): + m = self._make_message() + content = ("áàäéèęöőáàäéèęöőáàäéèęöőáàäéèęöő" + "áàäéèęöőáàäéèęöőáàäéèęöőáàäéèęöő" + "áàäéèęöőáàäéèęöőáàäéèęöőáàäéèęöő.\n") + raw_data_manager.set_content(m, content) + self.assertEqual(bytes(m), textwrap.dedent("""\ + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: base64 + + w6HDoMOkw6nDqMSZw7bFkcOhw6DDpMOpw6jEmcO2xZHDocOgw6TDqcOoxJnD + tsWRw6HDoMOkw6nDqMSZw7bFkcOhw6DDpMOpw6jEmcO2xZHDocOgw6TDqcOo + xJnDtsWRw6HDoMOkw6nDqMSZw7bFkcOhw6DDpMOpw6jEmcO2xZHDocOgw6TD + qcOoxJnDtsWRw6HDoMOkw6nDqMSZw7bFkcOhw6DDpMOpw6jEmcO2xZHDocOg + w6TDqcOoxJnDtsWRLgo= + """).encode('utf-8')) + self.assertEqual(m.get_payload(decode=True).decode('utf-8'), content) + self.assertEqual(m.get_content(), content) + + def test_set_text_11_lines_long_line_maximal_non_ascii_heuristics(self): + # Yes, it chooses "wrong" here. It's a heuristic. So this result + # could change if we come up with a better heuristic. + m = self._make_message() + content = ('\n'*10 + + "áàäéèęöőáàäéèęöőáàäéèęöőáàäéèęöő" + "áàäéèęöőáàäéèęöőáàäéèęöőáàäéèęöő" + "áàäéèęöőáàäéèęöőáàäéèęöőáàäéèęöő.\n") + raw_data_manager.set_content(m, "\n"*10 + + "áàäéèęöőáàäéèęöőáàäéèęöőáàäéèęöő" + "áàäéèęöőáàäéèęöőáàäéèęöőáàäéèęöő" + "áàäéèęöőáàäéèęöőáàäéèęöőáàäéèęöő.\n") + self.assertEqual(bytes(m), textwrap.dedent("""\ + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: quoted-printable + """ + '\n'*10 + """ + =C3=A1=C3=A0=C3=A4=C3=A9=C3=A8=C4=99=C3=B6=C5=91=C3=A1=C3= + =A0=C3=A4=C3=A9=C3=A8=C4=99=C3=B6=C5=91=C3=A1=C3=A0=C3=A4= + =C3=A9=C3=A8=C4=99=C3=B6=C5=91=C3=A1=C3=A0=C3=A4=C3=A9=C3= + =A8=C4=99=C3=B6=C5=91=C3=A1=C3=A0=C3=A4=C3=A9=C3=A8=C4=99= + =C3=B6=C5=91=C3=A1=C3=A0=C3=A4=C3=A9=C3=A8=C4=99=C3=B6=C5= + =91=C3=A1=C3=A0=C3=A4=C3=A9=C3=A8=C4=99=C3=B6=C5=91=C3=A1= + =C3=A0=C3=A4=C3=A9=C3=A8=C4=99=C3=B6=C5=91=C3=A1=C3=A0=C3= + =A4=C3=A9=C3=A8=C4=99=C3=B6=C5=91=C3=A1=C3=A0=C3=A4=C3=A9= + =C3=A8=C4=99=C3=B6=C5=91=C3=A1=C3=A0=C3=A4=C3=A9=C3=A8=C4= + =99=C3=B6=C5=91=C3=A1=C3=A0=C3=A4=C3=A9=C3=A8=C4=99=C3=B6= + =C5=91. + """).encode('utf-8')) + self.assertEqual(m.get_payload(decode=True).decode('utf-8'), content) + self.assertEqual(m.get_content(), content) + + def test_set_text_non_ascii_with_cte_7bit_raises(self): + m = self._make_message() + with self.assertRaises(UnicodeError): + raw_data_manager.set_content(m,"áàäéèęöő.\n", cte='7bit') + + def test_set_text_non_ascii_with_charset_ascii_raises(self): + m = self._make_message() + with self.assertRaises(UnicodeError): + raw_data_manager.set_content(m,"áàäéèęöő.\n", charset='ascii') + + def test_set_text_non_ascii_with_cte_7bit_and_charset_ascii_raises(self): + m = self._make_message() + with self.assertRaises(UnicodeError): + raw_data_manager.set_content(m,"áàäéèęöő.\n", cte='7bit', charset='ascii') + + def test_set_message(self): + m = self._make_message() + m['Subject'] = "Forwarded message" + content = self._make_message() + content['To'] = 'python@vivarium.org' + content['From'] = 'police@monty.org' + content['Subject'] = "get back in your box" + content.set_content("Or face the comfy chair.") + raw_data_manager.set_content(m, content) + self.assertEqual(str(m), textwrap.dedent("""\ + Subject: Forwarded message + Content-Type: message/rfc822 + Content-Transfer-Encoding: 8bit + + To: python@vivarium.org + From: police@monty.org + Subject: get back in your box + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: 7bit + MIME-Version: 1.0 + + Or face the comfy chair. + """)) + payload = m.get_payload(0) + self.assertIsInstance(payload, self.message) + self.assertEqual(str(payload), str(content)) + self.assertIsInstance(m.get_content(), self.message) + self.assertEqual(str(m.get_content()), str(content)) + + def test_set_message_with_non_ascii_and_coercion_to_7bit(self): + m = self._make_message() + m['Subject'] = "Escape report" + content = self._make_message() + content['To'] = 'police@monty.org' + content['From'] = 'victim@monty.org' + content['Subject'] = "Help" + content.set_content("j'ai un problème de python. il est sorti de son" + " vivarium.") + raw_data_manager.set_content(m, content) + self.assertEqual(bytes(m), textwrap.dedent("""\ + Subject: Escape report + Content-Type: message/rfc822 + Content-Transfer-Encoding: 8bit + + To: police@monty.org + From: victim@monty.org + Subject: Help + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: 8bit + MIME-Version: 1.0 + + j'ai un problème de python. il est sorti de son vivarium. + """).encode('utf-8')) + # The choice of base64 for the body encoding is because generator + # doesn't bother with heuristics and uses it unconditionally for utf-8 + # text. + # XXX: the first cte should be 7bit, too...that's a generator bug. + # XXX: the line length in the body also looks like a generator bug. + self.assertEqual(m.as_string(maxheaderlen=self.policy.max_line_length), + textwrap.dedent("""\ + Subject: Escape report + Content-Type: message/rfc822 + Content-Transfer-Encoding: 8bit + + To: police@monty.org + From: victim@monty.org + Subject: Help + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: base64 + MIME-Version: 1.0 + + aidhaSB1biBwcm9ibMOobWUgZGUgcHl0aG9uLiBpbCBlc3Qgc29ydGkgZGUgc29uIHZpdmFyaXVt + Lgo= + """)) + self.assertIsInstance(m.get_content(), self.message) + self.assertEqual(str(m.get_content()), str(content)) + + def test_set_message_invalid_cte_raises(self): + m = self._make_message() + content = self._make_message() + for cte in 'quoted-printable base64'.split(): + for subtype in 'rfc822 external-body'.split(): + with self.subTest(cte=cte, subtype=subtype): + with self.assertRaises(ValueError) as ar: + m.set_content(content, subtype, cte=cte) + exc = str(ar.exception) + self.assertIn(cte, exc) + self.assertIn(subtype, exc) + subtype = 'external-body' + for cte in '8bit binary'.split(): + with self.subTest(cte=cte, subtype=subtype): + with self.assertRaises(ValueError) as ar: + m.set_content(content, subtype, cte=cte) + exc = str(ar.exception) + self.assertIn(cte, exc) + self.assertIn(subtype, exc) + + def test_set_image_jpg(self): + for content in (b"bogus content", + bytearray(b"bogus content"), + memoryview(b"bogus content")): + with self.subTest(content=content): + m = self._make_message() + raw_data_manager.set_content(m, content, 'image', 'jpeg') + self.assertEqual(str(m), textwrap.dedent("""\ + Content-Type: image/jpeg + Content-Transfer-Encoding: base64 + + Ym9ndXMgY29udGVudA== + """)) + self.assertEqual(m.get_payload(decode=True), content) + self.assertEqual(m.get_content(), content) + + def test_set_audio_aif_with_quoted_printable_cte(self): + # Why you would use qp, I don't know, but it is technically supported. + # XXX: the incorrect line length is because binascii.b2a_qp doesn't + # support a line length parameter, but we must use it to get newline + # encoding. + # XXX: what about that lack of tailing newline? Do we actually handle + # that correctly in all cases? That is, if the *source* has an + # unencoded newline, do we add an extra newline to the returned payload + # or not? And can that actually be disambiguated based on the RFC? + m = self._make_message() + content = b'b\xFFgus\tcon\nt\rent ' + b'z'*100 + m.set_content(content, 'audio', 'aif', cte='quoted-printable') + self.assertEqual(bytes(m), textwrap.dedent("""\ + Content-Type: audio/aif + Content-Transfer-Encoding: quoted-printable + MIME-Version: 1.0 + + b=FFgus=09con=0At=0Dent=20zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz= + zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz""").encode('latin-1')) + self.assertEqual(m.get_payload(decode=True), content) + self.assertEqual(m.get_content(), content) + + def test_set_video_mpeg_with_binary_cte(self): + m = self._make_message() + content = b'b\xFFgus\tcon\nt\rent ' + b'z'*100 + m.set_content(content, 'video', 'mpeg', cte='binary') + self.assertEqual(bytes(m), textwrap.dedent("""\ + Content-Type: video/mpeg + Content-Transfer-Encoding: binary + MIME-Version: 1.0 + + """).encode('ascii') + + # XXX: the second \n ought to be a \r, but generator gets it wrong. + # THIS MEANS WE DON'T ACTUALLY SUPPORT THE 'binary' CTE. + b'b\xFFgus\tcon\nt\nent zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz' + + b'zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz') + self.assertEqual(m.get_payload(decode=True), content) + self.assertEqual(m.get_content(), content) + + def test_set_application_octet_stream_with_8bit_cte(self): + # In 8bit mode, universal line end logic applies. It is up to the + # application to make sure the lines are short enough; we don't check. + m = self._make_message() + content = b'b\xFFgus\tcon\nt\rent\n' + b'z'*60 + b'\n' + m.set_content(content, 'application', 'octet-stream', cte='8bit') + self.assertEqual(bytes(m), textwrap.dedent("""\ + Content-Type: application/octet-stream + Content-Transfer-Encoding: 8bit + MIME-Version: 1.0 + + """).encode('ascii') + + b'b\xFFgus\tcon\nt\nent\n' + + b'zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz\n') + self.assertEqual(m.get_payload(decode=True), content) + self.assertEqual(m.get_content(), content) + + def test_set_headers_from_header_objects(self): + m = self._make_message() + content = "Simple message.\n" + header_factory = self.policy.header_factory + raw_data_manager.set_content(m, content, headers=( + header_factory("To", "foo@example.com"), + header_factory("From", "foo@example.com"), + header_factory("Subject", "I'm talking to myself."))) + self.assertEqual(str(m), textwrap.dedent("""\ + Content-Type: text/plain; charset="utf-8" + To: foo@example.com + From: foo@example.com + Subject: I'm talking to myself. + Content-Transfer-Encoding: 7bit + + Simple message. + """)) + + def test_set_headers_from_strings(self): + m = self._make_message() + content = "Simple message.\n" + raw_data_manager.set_content(m, content, headers=( + "X-Foo-Header: foo", + "X-Bar-Header: bar",)) + self.assertEqual(str(m), textwrap.dedent("""\ + Content-Type: text/plain; charset="utf-8" + X-Foo-Header: foo + X-Bar-Header: bar + Content-Transfer-Encoding: 7bit + + Simple message. + """)) + + def test_set_headers_with_invalid_duplicate_string_header_raises(self): + m = self._make_message() + content = "Simple message.\n" + with self.assertRaisesRegex(ValueError, 'Content-Type'): + raw_data_manager.set_content(m, content, headers=( + "Content-Type: foo/bar",) + ) + + def test_set_headers_with_invalid_duplicate_header_header_raises(self): + m = self._make_message() + content = "Simple message.\n" + header_factory = self.policy.header_factory + with self.assertRaisesRegex(ValueError, 'Content-Type'): + raw_data_manager.set_content(m, content, headers=( + header_factory("Content-Type", " foo/bar"),) + ) + + def test_set_headers_with_defective_string_header_raises(self): + m = self._make_message() + content = "Simple message.\n" + with self.assertRaisesRegex(ValueError, 'a@fairly@@invalid@address'): + raw_data_manager.set_content(m, content, headers=( + 'To: a@fairly@@invalid@address',) + ) + print(m['To'].defects) + + def test_set_headers_with_defective_header_header_raises(self): + m = self._make_message() + content = "Simple message.\n" + header_factory = self.policy.header_factory + with self.assertRaisesRegex(ValueError, 'a@fairly@@invalid@address'): + raw_data_manager.set_content(m, content, headers=( + header_factory('To', 'a@fairly@@invalid@address'),) + ) + print(m['To'].defects) + + def test_set_disposition_inline(self): + m = self._make_message() + m.set_content('foo', disposition='inline') + self.assertEqual(m['Content-Disposition'], 'inline') + + def test_set_disposition_attachment(self): + m = self._make_message() + m.set_content('foo', disposition='attachment') + self.assertEqual(m['Content-Disposition'], 'attachment') + + def test_set_disposition_foo(self): + m = self._make_message() + m.set_content('foo', disposition='foo') + self.assertEqual(m['Content-Disposition'], 'foo') + + # XXX: we should have a 'strict' policy mode (beyond raise_on_defect) that + # would cause 'foo' above to raise. + + def test_set_filename(self): + m = self._make_message() + m.set_content('foo', filename='bar.txt') + self.assertEqual(m['Content-Disposition'], + 'attachment; filename="bar.txt"') + + def test_set_filename_and_disposition_inline(self): + m = self._make_message() + m.set_content('foo', disposition='inline', filename='bar.txt') + self.assertEqual(m['Content-Disposition'], 'inline; filename="bar.txt"') + + def test_set_non_ascii_filename(self): + m = self._make_message() + m.set_content('foo', filename='ábárî.txt') + self.assertEqual(bytes(m), textwrap.dedent("""\ + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: 7bit + Content-Disposition: attachment; + filename*=utf-8''%C3%A1b%C3%A1r%C3%AE.txt + MIME-Version: 1.0 + + foo + """).encode('ascii')) + + def test_set_content_bytes_cte_7bit(self): + m = self._make_message() + m.set_content(b'ASCII-only message.\n', + maintype='application', subtype='octet-stream', cte='7bit') + self.assertEqual(str(m), textwrap.dedent("""\ + Content-Type: application/octet-stream + Content-Transfer-Encoding: 7bit + MIME-Version: 1.0 + + ASCII-only message. + """)) + + content_object_params = { + 'text_plain': ('content', ()), + 'text_html': ('content', ('html',)), + 'application_octet_stream': (b'content', + ('application', 'octet_stream')), + 'image_jpeg': (b'content', ('image', 'jpeg')), + 'message_rfc822': (message(), ()), + 'message_external_body': (message(), ('external-body',)), + } + + def content_object_as_header_receiver(self, obj, mimetype): + m = self._make_message() + m.set_content(obj, *mimetype, headers=( + 'To: foo@example.com', + 'From: bar@simple.net')) + self.assertEqual(m['to'], 'foo@example.com') + self.assertEqual(m['from'], 'bar@simple.net') + + def content_object_as_disposition_inline_receiver(self, obj, mimetype): + m = self._make_message() + m.set_content(obj, *mimetype, disposition='inline') + self.assertEqual(m['Content-Disposition'], 'inline') + + def content_object_as_non_ascii_filename_receiver(self, obj, mimetype): + m = self._make_message() + m.set_content(obj, *mimetype, disposition='inline', filename='bár.txt') + self.assertEqual(m['Content-Disposition'], 'inline; filename="bár.txt"') + self.assertEqual(m.get_filename(), "bár.txt") + self.assertEqual(m['Content-Disposition'].params['filename'], "bár.txt") + + def content_object_as_cid_receiver(self, obj, mimetype): + m = self._make_message() + m.set_content(obj, *mimetype, cid='some_random_stuff') + self.assertEqual(m['Content-ID'], 'some_random_stuff') + + def content_object_as_params_receiver(self, obj, mimetype): + m = self._make_message() + params = {'foo': 'bár', 'abc': 'xyz'} + m.set_content(obj, *mimetype, params=params) + if isinstance(obj, str): + params['charset'] = 'utf-8' + self.assertEqual(m['Content-Type'].params, params) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test_defect_handling.py b/Lib/test/test_email/test_defect_handling.py new file mode 100644 index 00000000000..44e76c8ce5e --- /dev/null +++ b/Lib/test/test_email/test_defect_handling.py @@ -0,0 +1,337 @@ +import textwrap +import unittest +import contextlib +from email import policy +from email import errors +from test.test_email import TestEmailBase + + +class TestDefectsBase: + + policy = policy.default + raise_expected = False + + @contextlib.contextmanager + def _raise_point(self, defect): + yield + + def test_same_boundary_inner_outer(self): + source = textwrap.dedent("""\ + Subject: XX + From: xx@xx.dk + To: XX + Mime-version: 1.0 + Content-type: multipart/mixed; + boundary="MS_Mac_OE_3071477847_720252_MIME_Part" + + --MS_Mac_OE_3071477847_720252_MIME_Part + Content-type: multipart/alternative; + boundary="MS_Mac_OE_3071477847_720252_MIME_Part" + + --MS_Mac_OE_3071477847_720252_MIME_Part + Content-type: text/plain; charset="ISO-8859-1" + Content-transfer-encoding: quoted-printable + + text + + --MS_Mac_OE_3071477847_720252_MIME_Part + Content-type: text/html; charset="ISO-8859-1" + Content-transfer-encoding: quoted-printable + + + + --MS_Mac_OE_3071477847_720252_MIME_Part-- + + --MS_Mac_OE_3071477847_720252_MIME_Part + Content-type: image/gif; name="xx.gif"; + Content-disposition: attachment + Content-transfer-encoding: base64 + + Some removed base64 encoded chars. + + --MS_Mac_OE_3071477847_720252_MIME_Part-- + + """) + # XXX better would be to actually detect the duplicate. + with self._raise_point(errors.StartBoundaryNotFoundDefect): + msg = self._str_msg(source) + if self.raise_expected: return + inner = msg.get_payload(0) + self.assertHasAttr(inner, 'defects') + self.assertEqual(len(self.get_defects(inner)), 1) + self.assertIsInstance(self.get_defects(inner)[0], + errors.StartBoundaryNotFoundDefect) + + def test_multipart_no_boundary(self): + source = textwrap.dedent("""\ + Date: Fri, 6 Apr 2001 09:23:06 -0800 (GMT-0800) + From: foobar + Subject: broken mail + MIME-Version: 1.0 + Content-Type: multipart/report; report-type=delivery-status; + + --JAB03225.986577786/zinfandel.lacita.com + + One part + + --JAB03225.986577786/zinfandel.lacita.com + Content-Type: message/delivery-status + + Header: Another part + + --JAB03225.986577786/zinfandel.lacita.com-- + """) + with self._raise_point(errors.NoBoundaryInMultipartDefect): + msg = self._str_msg(source) + if self.raise_expected: return + self.assertIsInstance(msg.get_payload(), str) + self.assertEqual(len(self.get_defects(msg)), 2) + self.assertIsInstance(self.get_defects(msg)[0], + errors.NoBoundaryInMultipartDefect) + self.assertIsInstance(self.get_defects(msg)[1], + errors.MultipartInvariantViolationDefect) + + multipart_msg = textwrap.dedent("""\ + Date: Wed, 14 Nov 2007 12:56:23 GMT + From: foo@bar.invalid + To: foo@bar.invalid + Subject: Content-Transfer-Encoding: base64 and multipart + MIME-Version: 1.0 + Content-Type: multipart/mixed; + boundary="===============3344438784458119861=="{} + + --===============3344438784458119861== + Content-Type: text/plain + + Test message + + --===============3344438784458119861== + Content-Type: application/octet-stream + Content-Transfer-Encoding: base64 + + YWJj + + --===============3344438784458119861==-- + """) + + def test_multipart_invalid_cte(self): + with self._raise_point( + errors.InvalidMultipartContentTransferEncodingDefect): + msg = self._str_msg( + self.multipart_msg.format( + "\nContent-Transfer-Encoding: base64")) + if self.raise_expected: return + self.assertEqual(len(self.get_defects(msg)), 1) + self.assertIsInstance(self.get_defects(msg)[0], + errors.InvalidMultipartContentTransferEncodingDefect) + + def test_multipart_no_cte_no_defect(self): + if self.raise_expected: return + msg = self._str_msg(self.multipart_msg.format('')) + self.assertEqual(len(self.get_defects(msg)), 0) + + def test_multipart_valid_cte_no_defect(self): + if self.raise_expected: return + for cte in ('7bit', '8bit', 'BINary'): + msg = self._str_msg( + self.multipart_msg.format("\nContent-Transfer-Encoding: "+cte)) + self.assertEqual(len(self.get_defects(msg)), 0, "cte="+cte) + + def test_lying_multipart(self): + source = textwrap.dedent("""\ + From: "Allison Dunlap" + To: yyy@example.com + Subject: 64423 + Date: Sun, 11 Jul 2004 16:09:27 -0300 + MIME-Version: 1.0 + Content-Type: multipart/alternative; + + Blah blah blah + """) + with self._raise_point(errors.NoBoundaryInMultipartDefect): + msg = self._str_msg(source) + if self.raise_expected: return + self.assertHasAttr(msg, 'defects') + self.assertEqual(len(self.get_defects(msg)), 2) + self.assertIsInstance(self.get_defects(msg)[0], + errors.NoBoundaryInMultipartDefect) + self.assertIsInstance(self.get_defects(msg)[1], + errors.MultipartInvariantViolationDefect) + + def test_missing_start_boundary(self): + source = textwrap.dedent("""\ + Content-Type: multipart/mixed; boundary="AAA" + From: Mail Delivery Subsystem + To: yyy@example.com + + --AAA + + Stuff + + --AAA + Content-Type: message/rfc822 + + From: webmaster@python.org + To: zzz@example.com + Content-Type: multipart/mixed; boundary="BBB" + + --BBB-- + + --AAA-- + + """) + # The message structure is: + # + # multipart/mixed + # text/plain + # message/rfc822 + # multipart/mixed [*] + # + # [*] This message is missing its start boundary + with self._raise_point(errors.StartBoundaryNotFoundDefect): + outer = self._str_msg(source) + if self.raise_expected: return + bad = outer.get_payload(1).get_payload(0) + self.assertEqual(len(self.get_defects(bad)), 1) + self.assertIsInstance(self.get_defects(bad)[0], + errors.StartBoundaryNotFoundDefect) + + def test_first_line_is_continuation_header(self): + with self._raise_point(errors.FirstHeaderLineIsContinuationDefect): + msg = self._str_msg(' Line 1\nSubject: test\n\nbody') + if self.raise_expected: return + self.assertEqual(msg.keys(), ['Subject']) + self.assertEqual(msg.get_payload(), 'body') + self.assertEqual(len(self.get_defects(msg)), 1) + self.assertDefectsEqual(self.get_defects(msg), + [errors.FirstHeaderLineIsContinuationDefect]) + self.assertEqual(self.get_defects(msg)[0].line, ' Line 1\n') + + def test_missing_header_body_separator(self): + # Our heuristic if we see a line that doesn't look like a header (no + # leading whitespace but no ':') is to assume that the blank line that + # separates the header from the body is missing, and to stop parsing + # headers and start parsing the body. + with self._raise_point(errors.MissingHeaderBodySeparatorDefect): + msg = self._str_msg('Subject: test\nnot a header\nTo: abc\n\nb\n') + if self.raise_expected: return + self.assertEqual(msg.keys(), ['Subject']) + self.assertEqual(msg.get_payload(), 'not a header\nTo: abc\n\nb\n') + self.assertDefectsEqual(self.get_defects(msg), + [errors.MissingHeaderBodySeparatorDefect]) + + def test_bad_padding_in_base64_payload(self): + source = textwrap.dedent("""\ + Subject: test + MIME-Version: 1.0 + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: base64 + + dmk + """) + msg = self._str_msg(source) + with self._raise_point(errors.InvalidBase64PaddingDefect): + payload = msg.get_payload(decode=True) + if self.raise_expected: return + self.assertEqual(payload, b'vi') + self.assertDefectsEqual(self.get_defects(msg), + [errors.InvalidBase64PaddingDefect]) + + def test_invalid_chars_in_base64_payload(self): + source = textwrap.dedent("""\ + Subject: test + MIME-Version: 1.0 + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: base64 + + dm\x01k=== + """) + msg = self._str_msg(source) + with self._raise_point(errors.InvalidBase64CharactersDefect): + payload = msg.get_payload(decode=True) + if self.raise_expected: return + self.assertEqual(payload, b'vi') + self.assertDefectsEqual(self.get_defects(msg), + [errors.InvalidBase64CharactersDefect]) + + def test_invalid_length_of_base64_payload(self): + source = textwrap.dedent("""\ + Subject: test + MIME-Version: 1.0 + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: base64 + + abcde + """) + msg = self._str_msg(source) + with self._raise_point(errors.InvalidBase64LengthDefect): + payload = msg.get_payload(decode=True) + if self.raise_expected: return + self.assertEqual(payload, b'abcde') + self.assertDefectsEqual(self.get_defects(msg), + [errors.InvalidBase64LengthDefect]) + + def test_missing_ending_boundary(self): + source = textwrap.dedent("""\ + To: 1@harrydomain4.com + Subject: Fwd: 1 + MIME-Version: 1.0 + Content-Type: multipart/alternative; + boundary="------------000101020201080900040301" + + --------------000101020201080900040301 + Content-Type: text/plain; charset=ISO-8859-1 + Content-Transfer-Encoding: 7bit + + Alternative 1 + + --------------000101020201080900040301 + Content-Type: text/html; charset=ISO-8859-1 + Content-Transfer-Encoding: 7bit + + Alternative 2 + + """) + with self._raise_point(errors.CloseBoundaryNotFoundDefect): + msg = self._str_msg(source) + if self.raise_expected: return + self.assertEqual(len(msg.get_payload()), 2) + self.assertEqual(msg.get_payload(1).get_payload(), 'Alternative 2\n') + self.assertDefectsEqual(self.get_defects(msg), + [errors.CloseBoundaryNotFoundDefect]) + + +class TestDefectDetection(TestDefectsBase, TestEmailBase): + + def get_defects(self, obj): + return obj.defects + + +class TestDefectCapture(TestDefectsBase, TestEmailBase): + + class CapturePolicy(policy.EmailPolicy): + captured = None + def register_defect(self, obj, defect): + self.captured.append(defect) + + def setUp(self): + self.policy = self.CapturePolicy(captured=list()) + + def get_defects(self, obj): + return self.policy.captured + + +class TestDefectRaising(TestDefectsBase, TestEmailBase): + + policy = TestDefectsBase.policy + policy = policy.clone(raise_on_defect=True) + raise_expected = True + + @contextlib.contextmanager + def _raise_point(self, defect): + with self.assertRaises(defect): + yield + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py new file mode 100644 index 00000000000..2d843c7d723 --- /dev/null +++ b/Lib/test/test_email/test_email.py @@ -0,0 +1,5907 @@ +# Copyright (C) 2001-2010 Python Software Foundation +# Contact: email-sig@python.org +# email package unit tests + +import re +import time +import base64 +import unittest +import textwrap + +from io import StringIO, BytesIO +from itertools import chain +from random import choice +from threading import Thread +from unittest.mock import patch + +import email +import email.policy +import email.utils + +from email.charset import Charset +from email.generator import Generator, DecodedGenerator, BytesGenerator +from email.header import Header, decode_header, make_header +from email.headerregistry import HeaderRegistry +from email.message import Message +from email.mime.application import MIMEApplication +from email.mime.audio import MIMEAudio +from email.mime.base import MIMEBase +from email.mime.image import MIMEImage +from email.mime.message import MIMEMessage +from email.mime.multipart import MIMEMultipart +from email.mime.nonmultipart import MIMENonMultipart +from email.mime.text import MIMEText +from email.parser import Parser, HeaderParser +from email import base64mime +from email import encoders +from email import errors +from email import iterators +from email import quoprimime +from email import utils + +from test import support +from test.support import threading_helper +from test.support.os_helper import unlink +from test.test_email import openfile, TestEmailBase + +# These imports are documented to work, but we are testing them using a +# different path, so we import them here just to make sure they are importable. +from email.parser import FeedParser + +NL = '\n' +EMPTYSTRING = '' +SPACE = ' ' + + +# Test various aspects of the Message class's API +class TestMessageAPI(TestEmailBase): + def test_get_all(self): + eq = self.assertEqual + msg = self._msgobj('msg_20.txt') + eq(msg.get_all('cc'), ['ccc@zzz.org', 'ddd@zzz.org', 'eee@zzz.org']) + eq(msg.get_all('xx', 'n/a'), 'n/a') + + def test_getset_charset(self): + eq = self.assertEqual + msg = Message() + eq(msg.get_charset(), None) + charset = Charset('iso-8859-1') + msg.set_charset(charset) + eq(msg['mime-version'], '1.0') + eq(msg.get_content_type(), 'text/plain') + eq(msg['content-type'], 'text/plain; charset="iso-8859-1"') + eq(msg.get_param('charset'), 'iso-8859-1') + eq(msg['content-transfer-encoding'], 'quoted-printable') + eq(msg.get_charset().input_charset, 'iso-8859-1') + # Remove the charset + msg.set_charset(None) + eq(msg.get_charset(), None) + eq(msg['content-type'], 'text/plain') + # Try adding a charset when there's already MIME headers present + msg = Message() + msg['MIME-Version'] = '2.0' + msg['Content-Type'] = 'text/x-weird' + msg['Content-Transfer-Encoding'] = 'quinted-puntable' + msg.set_charset(charset) + eq(msg['mime-version'], '2.0') + eq(msg['content-type'], 'text/x-weird; charset="iso-8859-1"') + eq(msg['content-transfer-encoding'], 'quinted-puntable') + + def test_set_charset_from_string(self): + eq = self.assertEqual + msg = Message() + msg.set_charset('us-ascii') + eq(msg.get_charset().input_charset, 'us-ascii') + eq(msg['content-type'], 'text/plain; charset="us-ascii"') + + def test_set_payload_with_charset(self): + msg = Message() + charset = Charset('iso-8859-1') + msg.set_payload('This is a string payload', charset) + self.assertEqual(msg.get_charset().input_charset, 'iso-8859-1') + + def test_set_payload_with_8bit_data_and_charset(self): + data = b'\xd0\x90\xd0\x91\xd0\x92' + charset = Charset('utf-8') + msg = Message() + msg.set_payload(data, charset) + self.assertEqual(msg['content-transfer-encoding'], 'base64') + self.assertEqual(msg.get_payload(decode=True), data) + self.assertEqual(msg.get_payload(), '0JDQkdCS\n') + + def test_set_payload_with_non_ascii_and_charset_body_encoding_none(self): + data = b'\xd0\x90\xd0\x91\xd0\x92' + charset = Charset('utf-8') + charset.body_encoding = None # Disable base64 encoding + msg = Message() + msg.set_payload(data.decode('utf-8'), charset) + self.assertEqual(msg['content-transfer-encoding'], '8bit') + self.assertEqual(msg.get_payload(decode=True), data) + + def test_set_payload_with_8bit_data_and_charset_body_encoding_none(self): + data = b'\xd0\x90\xd0\x91\xd0\x92' + charset = Charset('utf-8') + charset.body_encoding = None # Disable base64 encoding + msg = Message() + msg.set_payload(data, charset) + self.assertEqual(msg['content-transfer-encoding'], '8bit') + self.assertEqual(msg.get_payload(decode=True), data) + + def test_set_payload_to_list(self): + msg = Message() + msg.set_payload([]) + self.assertEqual(msg.get_payload(), []) + + def test_attach_when_payload_is_string(self): + msg = Message() + msg['Content-Type'] = 'multipart/mixed' + msg.set_payload('string payload') + sub_msg = MIMEMessage(Message()) + self.assertRaisesRegex(TypeError, "[Aa]ttach.*non-multipart", + msg.attach, sub_msg) + + def test_get_charsets(self): + eq = self.assertEqual + + msg = self._msgobj('msg_08.txt') + charsets = msg.get_charsets() + eq(charsets, [None, 'us-ascii', 'iso-8859-1', 'iso-8859-2', 'koi8-r']) + + msg = self._msgobj('msg_09.txt') + charsets = msg.get_charsets('dingbat') + eq(charsets, ['dingbat', 'us-ascii', 'iso-8859-1', 'dingbat', + 'koi8-r']) + + msg = self._msgobj('msg_12.txt') + charsets = msg.get_charsets() + eq(charsets, [None, 'us-ascii', 'iso-8859-1', None, 'iso-8859-2', + 'iso-8859-3', 'us-ascii', 'koi8-r']) + + def test_get_filename(self): + eq = self.assertEqual + + msg = self._msgobj('msg_04.txt') + filenames = [p.get_filename() for p in msg.get_payload()] + eq(filenames, ['msg.txt', 'msg.txt']) + + msg = self._msgobj('msg_07.txt') + subpart = msg.get_payload(1) + eq(subpart.get_filename(), 'dingusfish.gif') + + def test_get_filename_with_name_parameter(self): + eq = self.assertEqual + + msg = self._msgobj('msg_44.txt') + filenames = [p.get_filename() for p in msg.get_payload()] + eq(filenames, ['msg.txt', 'msg.txt']) + + def test_get_boundary(self): + eq = self.assertEqual + msg = self._msgobj('msg_07.txt') + # No quotes! + eq(msg.get_boundary(), 'BOUNDARY') + + def test_set_boundary(self): + eq = self.assertEqual + # This one has no existing boundary parameter, but the Content-Type: + # header appears fifth. + msg = self._msgobj('msg_01.txt') + msg.set_boundary('BOUNDARY') + header, value = msg.items()[4] + eq(header.lower(), 'content-type') + eq(value, 'text/plain; charset="us-ascii"; boundary="BOUNDARY"') + # This one has a Content-Type: header, with a boundary, stuck in the + # middle of its headers. Make sure the order is preserved; it should + # be fifth. + msg = self._msgobj('msg_04.txt') + msg.set_boundary('BOUNDARY') + header, value = msg.items()[4] + eq(header.lower(), 'content-type') + eq(value, 'multipart/mixed; boundary="BOUNDARY"') + # And this one has no Content-Type: header at all. + msg = self._msgobj('msg_03.txt') + self.assertRaises(errors.HeaderParseError, + msg.set_boundary, 'BOUNDARY') + + def test_make_boundary(self): + msg = MIMEMultipart('form-data') + # Note that when the boundary gets created is an implementation + # detail and might change. + self.assertEqual(msg.items()[0][1], 'multipart/form-data') + # Trigger creation of boundary + msg.as_string() + self.assertStartsWith(msg.items()[0][1], + 'multipart/form-data; boundary="==') + # XXX: there ought to be tests of the uniqueness of the boundary, too. + + def test_message_rfc822_only(self): + # Issue 7970: message/rfc822 not in multipart parsed by + # HeaderParser caused an exception when flattened. + with openfile('msg_46.txt', encoding="utf-8") as fp: + msgdata = fp.read() + parser = HeaderParser() + msg = parser.parsestr(msgdata) + out = StringIO() + gen = Generator(out, True, 0) + gen.flatten(msg, False) + self.assertEqual(out.getvalue(), msgdata) + + def test_byte_message_rfc822_only(self): + # Make sure new bytes header parser also passes this. + with openfile('msg_46.txt', encoding="utf-8") as fp: + msgdata = fp.read().encode('ascii') + parser = email.parser.BytesHeaderParser() + msg = parser.parsebytes(msgdata) + out = BytesIO() + gen = email.generator.BytesGenerator(out) + gen.flatten(msg) + self.assertEqual(out.getvalue(), msgdata) + + def test_get_decoded_payload(self): + eq = self.assertEqual + msg = self._msgobj('msg_10.txt') + # The outer message is a multipart + eq(msg.get_payload(decode=True), None) + # Subpart 1 is 7bit encoded + eq(msg.get_payload(0).get_payload(decode=True), + b'This is a 7bit encoded message.\n') + # Subpart 2 is quopri + eq(msg.get_payload(1).get_payload(decode=True), + b'\xa1This is a Quoted Printable encoded message!\n') + # Subpart 3 is base64 + eq(msg.get_payload(2).get_payload(decode=True), + b'This is a Base64 encoded message.') + # Subpart 4 is base64 with a trailing newline, which + # used to be stripped (issue 7143). + eq(msg.get_payload(3).get_payload(decode=True), + b'This is a Base64 encoded message.\n') + # Subpart 5 has no Content-Transfer-Encoding: header. + eq(msg.get_payload(4).get_payload(decode=True), + b'This has no Content-Transfer-Encoding: header.\n') + + def test_get_decoded_uu_payload(self): + eq = self.assertEqual + msg = Message() + msg.set_payload('begin 666 -\n+:&5L;&\\@=V]R;&0 \n \nend\n') + for cte in ('x-uuencode', 'uuencode', 'uue', 'x-uue'): + msg['content-transfer-encoding'] = cte + eq(msg.get_payload(decode=True), b'hello world') + # Now try some bogus data + msg.set_payload('foo') + eq(msg.get_payload(decode=True), b'foo') + + def test_get_payload_n_raises_on_non_multipart(self): + msg = Message() + self.assertRaises(TypeError, msg.get_payload, 1) + + def test_decoded_generator(self): + eq = self.assertEqual + msg = self._msgobj('msg_07.txt') + with openfile('msg_17.txt', encoding="utf-8") as fp: + text = fp.read() + s = StringIO() + g = DecodedGenerator(s) + g.flatten(msg) + eq(s.getvalue(), text) + + def test__contains__(self): + msg = Message() + msg['From'] = 'Me' + msg['to'] = 'You' + # Check for case insensitivity + self.assertIn('from', msg) + self.assertIn('From', msg) + self.assertIn('FROM', msg) + self.assertIn('to', msg) + self.assertIn('To', msg) + self.assertIn('TO', msg) + + def test_as_string(self): + msg = self._msgobj('msg_01.txt') + with openfile('msg_01.txt', encoding="utf-8") as fp: + text = fp.read() + self.assertEqual(text, str(msg)) + fullrepr = msg.as_string(unixfrom=True) + lines = fullrepr.split('\n') + self.assertStartsWith(lines[0], 'From ') + self.assertEqual(text, NL.join(lines[1:])) + + def test_as_string_policy(self): + msg = self._msgobj('msg_01.txt') + newpolicy = msg.policy.clone(linesep='\r\n') + fullrepr = msg.as_string(policy=newpolicy) + s = StringIO() + g = Generator(s, policy=newpolicy) + g.flatten(msg) + self.assertEqual(fullrepr, s.getvalue()) + + def test_nonascii_as_string_without_cte(self): + m = textwrap.dedent("""\ + MIME-Version: 1.0 + Content-type: text/plain; charset="iso-8859-1" + + Test if non-ascii messages with no Content-Transfer-Encoding set + can be as_string'd: + Föö bär + """) + source = m.encode('iso-8859-1') + expected = textwrap.dedent("""\ + MIME-Version: 1.0 + Content-type: text/plain; charset="iso-8859-1" + Content-Transfer-Encoding: quoted-printable + + Test if non-ascii messages with no Content-Transfer-Encoding set + can be as_string'd: + F=F6=F6 b=E4r + """) + msg = email.message_from_bytes(source) + self.assertEqual(msg.as_string(), expected) + + def test_nonascii_as_string_with_ascii_charset(self): + m = textwrap.dedent("""\ + MIME-Version: 1.0 + Content-type: text/plain; charset="us-ascii" + Content-Transfer-Encoding: 8bit + + Test if non-ascii messages with no Content-Transfer-Encoding set + can be as_string'd: + Föö bär + """) + source = m.encode('iso-8859-1') + expected = source.decode('ascii', 'replace') + msg = email.message_from_bytes(source) + self.assertEqual(msg.as_string(), expected) + + def test_nonascii_as_string_without_content_type_and_cte(self): + m = textwrap.dedent("""\ + MIME-Version: 1.0 + + Test if non-ascii messages with no Content-Type nor + Content-Transfer-Encoding set can be as_string'd: + Föö bär + """) + source = m.encode('iso-8859-1') + expected = source.decode('ascii', 'replace') + msg = email.message_from_bytes(source) + self.assertEqual(msg.as_string(), expected) + + def test_as_bytes(self): + msg = self._msgobj('msg_01.txt') + with openfile('msg_01.txt', encoding="utf-8") as fp: + data = fp.read().encode('ascii') + self.assertEqual(data, bytes(msg)) + fullrepr = msg.as_bytes(unixfrom=True) + lines = fullrepr.split(b'\n') + self.assertStartsWith(lines[0], b'From ') + self.assertEqual(data, b'\n'.join(lines[1:])) + + def test_as_bytes_policy(self): + msg = self._msgobj('msg_01.txt') + newpolicy = msg.policy.clone(linesep='\r\n') + fullrepr = msg.as_bytes(policy=newpolicy) + s = BytesIO() + g = BytesGenerator(s,policy=newpolicy) + g.flatten(msg) + self.assertEqual(fullrepr, s.getvalue()) + + # test_headerregistry.TestContentTypeHeader.bad_params + def test_bad_param(self): + msg = email.message_from_string("Content-Type: blarg; baz; boo\n") + self.assertEqual(msg.get_param('baz'), '') + + def test_continuation_sorting_part_order(self): + msg = email.message_from_string( + "Content-Disposition: attachment; " + "filename*=\"ignored\"; " + "filename*0*=\"utf-8''foo%20\"; " + "filename*1*=\"bar.txt\"\n" + ) + filename = msg.get_filename() + self.assertEqual(filename, 'foo bar.txt') + + def test_sorting_no_continuations(self): + msg = email.message_from_string( + "Content-Disposition: attachment; " + "filename*=\"bar.txt\"; " + ) + filename = msg.get_filename() + self.assertEqual(filename, 'bar.txt') + + def test_missing_filename(self): + msg = email.message_from_string("From: foo\n") + self.assertEqual(msg.get_filename(), None) + + def test_bogus_filename(self): + msg = email.message_from_string( + "Content-Disposition: blarg; filename\n") + self.assertEqual(msg.get_filename(), '') + + def test_missing_boundary(self): + msg = email.message_from_string("From: foo\n") + self.assertEqual(msg.get_boundary(), None) + + def test_get_params(self): + eq = self.assertEqual + msg = email.message_from_string( + 'X-Header: foo=one; bar=two; baz=three\n') + eq(msg.get_params(header='x-header'), + [('foo', 'one'), ('bar', 'two'), ('baz', 'three')]) + msg = email.message_from_string( + 'X-Header: foo; bar=one; baz=two\n') + eq(msg.get_params(header='x-header'), + [('foo', ''), ('bar', 'one'), ('baz', 'two')]) + eq(msg.get_params(), None) + msg = email.message_from_string( + 'X-Header: foo; bar="one"; baz=two\n') + eq(msg.get_params(header='x-header'), + [('foo', ''), ('bar', 'one'), ('baz', 'two')]) + + # test_headerregistry.TestContentTypeHeader.spaces_around_param_equals + def test_get_param_liberal(self): + msg = Message() + msg['Content-Type'] = 'Content-Type: Multipart/mixed; boundary = "CPIMSSMTPC06p5f3tG"' + self.assertEqual(msg.get_param('boundary'), 'CPIMSSMTPC06p5f3tG') + + def test_get_param(self): + eq = self.assertEqual + msg = email.message_from_string( + "X-Header: foo=one; bar=two; baz=three\n") + eq(msg.get_param('bar', header='x-header'), 'two') + eq(msg.get_param('quuz', header='x-header'), None) + eq(msg.get_param('quuz'), None) + msg = email.message_from_string( + 'X-Header: foo; bar="one"; baz=two\n') + eq(msg.get_param('foo', header='x-header'), '') + eq(msg.get_param('bar', header='x-header'), 'one') + eq(msg.get_param('baz', header='x-header'), 'two') + # XXX: We are not RFC-2045 compliant! We cannot parse: + # msg["Content-Type"] = 'text/plain; weird="hey; dolly? [you] @ <\\"home\\">?"' + # msg.get_param("weird") + # yet. + + # test_headerregistry.TestContentTypeHeader.spaces_around_semis + def test_get_param_funky_continuation_lines(self): + msg = self._msgobj('msg_22.txt') + self.assertEqual(msg.get_payload(1).get_param('name'), 'wibble.JPG') + + # test_headerregistry.TestContentTypeHeader.semis_inside_quotes + def test_get_param_with_semis_in_quotes(self): + msg = email.message_from_string( + 'Content-Type: image/pjpeg; name="Jim&&Jill"\n') + self.assertEqual(msg.get_param('name'), 'Jim&&Jill') + self.assertEqual(msg.get_param('name', unquote=False), + '"Jim&&Jill"') + + # test_headerregistry.TestContentTypeHeader.quotes_inside_rfc2231_value + def test_get_param_with_quotes(self): + msg = email.message_from_string( + 'Content-Type: foo; bar*0="baz\\"foobar"; bar*1="\\"baz"') + self.assertEqual(msg.get_param('bar'), 'baz"foobar"baz') + msg = email.message_from_string( + "Content-Type: foo; bar*0=\"baz\\\"foobar\"; bar*1=\"\\\"baz\"") + self.assertEqual(msg.get_param('bar'), 'baz"foobar"baz') + + @unittest.skip('TODO: RUSTPYTHON; Takes a long time to the point of timeouting') + def test_get_param_linear_complexity(self): + # Ensure that email.message._parseparam() is fast. + # See https://github.com/python/cpython/issues/136063. + N = 100_000 + for s, r in [ + ("", ""), + ("foo=bar", "foo=bar"), + (" FOO = bar ", "foo=bar"), + ]: + with self.subTest(s=s, r=r, N=N): + src = f'{s};' * (N - 1) + s + res = email.message._parseparam(src) + self.assertEqual(len(res), N) + self.assertEqual(len(set(res)), 1) + self.assertEqual(res[0], r) + + # This will be considered as a single parameter. + malformed = 's="' + ';' * (N - 1) + res = email.message._parseparam(malformed) + self.assertEqual(res, [malformed]) + + def test_field_containment(self): + msg = email.message_from_string('Header: exists') + self.assertIn('header', msg) + self.assertIn('Header', msg) + self.assertIn('HEADER', msg) + self.assertNotIn('headerx', msg) + + def test_set_param(self): + eq = self.assertEqual + msg = Message() + msg.set_param('charset', 'iso-2022-jp') + eq(msg.get_param('charset'), 'iso-2022-jp') + msg.set_param('importance', 'high value') + eq(msg.get_param('importance'), 'high value') + eq(msg.get_param('importance', unquote=False), '"high value"') + eq(msg.get_params(), [('text/plain', ''), + ('charset', 'iso-2022-jp'), + ('importance', 'high value')]) + eq(msg.get_params(unquote=False), [('text/plain', ''), + ('charset', '"iso-2022-jp"'), + ('importance', '"high value"')]) + msg.set_param('charset', 'iso-9999-xx', header='X-Jimmy') + eq(msg.get_param('charset', header='X-Jimmy'), 'iso-9999-xx') + + def test_del_param(self): + eq = self.assertEqual + msg = self._msgobj('msg_05.txt') + eq(msg.get_params(), + [('multipart/report', ''), ('report-type', 'delivery-status'), + ('boundary', 'D1690A7AC1.996856090/mail.example.com')]) + old_val = msg.get_param("report-type") + msg.del_param("report-type") + eq(msg.get_params(), + [('multipart/report', ''), + ('boundary', 'D1690A7AC1.996856090/mail.example.com')]) + msg.set_param("report-type", old_val) + eq(msg.get_params(), + [('multipart/report', ''), + ('boundary', 'D1690A7AC1.996856090/mail.example.com'), + ('report-type', old_val)]) + + def test_del_param_on_other_header(self): + msg = Message() + msg.add_header('Content-Disposition', 'attachment', filename='bud.gif') + msg.del_param('filename', 'content-disposition') + self.assertEqual(msg['content-disposition'], 'attachment') + + def test_del_param_on_nonexistent_header(self): + msg = Message() + # Deleting param on empty msg should not raise exception. + msg.del_param('filename', 'content-disposition') + + def test_del_nonexistent_param(self): + msg = Message() + msg.add_header('Content-Type', 'text/plain', charset='utf-8') + existing_header = msg['Content-Type'] + msg.del_param('foobar', header='Content-Type') + self.assertEqual(msg['Content-Type'], existing_header) + + def test_set_type(self): + eq = self.assertEqual + msg = Message() + self.assertRaises(ValueError, msg.set_type, 'text') + msg.set_type('text/plain') + eq(msg['content-type'], 'text/plain') + msg.set_param('charset', 'us-ascii') + eq(msg['content-type'], 'text/plain; charset="us-ascii"') + msg.set_type('text/html') + eq(msg['content-type'], 'text/html; charset="us-ascii"') + + def test_set_type_on_other_header(self): + msg = Message() + msg['X-Content-Type'] = 'text/plain' + msg.set_type('application/octet-stream', 'X-Content-Type') + self.assertEqual(msg['x-content-type'], 'application/octet-stream') + + def test_get_content_type_missing(self): + msg = Message() + self.assertEqual(msg.get_content_type(), 'text/plain') + + def test_get_content_type_missing_with_default_type(self): + msg = Message() + msg.set_default_type('message/rfc822') + self.assertEqual(msg.get_content_type(), 'message/rfc822') + + def test_get_content_type_from_message_implicit(self): + msg = self._msgobj('msg_30.txt') + self.assertEqual(msg.get_payload(0).get_content_type(), + 'message/rfc822') + + def test_get_content_type_from_message_explicit(self): + msg = self._msgobj('msg_28.txt') + self.assertEqual(msg.get_payload(0).get_content_type(), + 'message/rfc822') + + def test_get_content_type_from_message_text_plain_implicit(self): + msg = self._msgobj('msg_03.txt') + self.assertEqual(msg.get_content_type(), 'text/plain') + + def test_get_content_type_from_message_text_plain_explicit(self): + msg = self._msgobj('msg_01.txt') + self.assertEqual(msg.get_content_type(), 'text/plain') + + def test_get_content_maintype_missing(self): + msg = Message() + self.assertEqual(msg.get_content_maintype(), 'text') + + def test_get_content_maintype_missing_with_default_type(self): + msg = Message() + msg.set_default_type('message/rfc822') + self.assertEqual(msg.get_content_maintype(), 'message') + + def test_get_content_maintype_from_message_implicit(self): + msg = self._msgobj('msg_30.txt') + self.assertEqual(msg.get_payload(0).get_content_maintype(), 'message') + + def test_get_content_maintype_from_message_explicit(self): + msg = self._msgobj('msg_28.txt') + self.assertEqual(msg.get_payload(0).get_content_maintype(), 'message') + + def test_get_content_maintype_from_message_text_plain_implicit(self): + msg = self._msgobj('msg_03.txt') + self.assertEqual(msg.get_content_maintype(), 'text') + + def test_get_content_maintype_from_message_text_plain_explicit(self): + msg = self._msgobj('msg_01.txt') + self.assertEqual(msg.get_content_maintype(), 'text') + + def test_get_content_subtype_missing(self): + msg = Message() + self.assertEqual(msg.get_content_subtype(), 'plain') + + def test_get_content_subtype_missing_with_default_type(self): + msg = Message() + msg.set_default_type('message/rfc822') + self.assertEqual(msg.get_content_subtype(), 'rfc822') + + def test_get_content_subtype_from_message_implicit(self): + msg = self._msgobj('msg_30.txt') + self.assertEqual(msg.get_payload(0).get_content_subtype(), 'rfc822') + + def test_get_content_subtype_from_message_explicit(self): + msg = self._msgobj('msg_28.txt') + self.assertEqual(msg.get_payload(0).get_content_subtype(), 'rfc822') + + def test_get_content_subtype_from_message_text_plain_implicit(self): + msg = self._msgobj('msg_03.txt') + self.assertEqual(msg.get_content_subtype(), 'plain') + + def test_get_content_subtype_from_message_text_plain_explicit(self): + msg = self._msgobj('msg_01.txt') + self.assertEqual(msg.get_content_subtype(), 'plain') + + def test_get_content_maintype_error(self): + msg = Message() + msg['Content-Type'] = 'no-slash-in-this-string' + self.assertEqual(msg.get_content_maintype(), 'text') + + def test_get_content_subtype_error(self): + msg = Message() + msg['Content-Type'] = 'no-slash-in-this-string' + self.assertEqual(msg.get_content_subtype(), 'plain') + + def test_replace_header(self): + eq = self.assertEqual + msg = Message() + msg.add_header('First', 'One') + msg.add_header('Second', 'Two') + msg.add_header('Third', 'Three') + eq(msg.keys(), ['First', 'Second', 'Third']) + eq(msg.values(), ['One', 'Two', 'Three']) + msg.replace_header('Second', 'Twenty') + eq(msg.keys(), ['First', 'Second', 'Third']) + eq(msg.values(), ['One', 'Twenty', 'Three']) + msg.add_header('First', 'Eleven') + msg.replace_header('First', 'One Hundred') + eq(msg.keys(), ['First', 'Second', 'Third', 'First']) + eq(msg.values(), ['One Hundred', 'Twenty', 'Three', 'Eleven']) + self.assertRaises(KeyError, msg.replace_header, 'Fourth', 'Missing') + + def test_get_content_disposition(self): + msg = Message() + self.assertIsNone(msg.get_content_disposition()) + msg.add_header('Content-Disposition', 'attachment', + filename='random.avi') + self.assertEqual(msg.get_content_disposition(), 'attachment') + msg.replace_header('Content-Disposition', 'inline') + self.assertEqual(msg.get_content_disposition(), 'inline') + msg.replace_header('Content-Disposition', 'InlinE') + self.assertEqual(msg.get_content_disposition(), 'inline') + + # test_defect_handling:test_invalid_chars_in_base64_payload + def test_broken_base64_payload(self): + x = 'AwDp0P7//y6LwKEAcPa/6Q=9' + msg = Message() + msg['content-type'] = 'audio/x-midi' + msg['content-transfer-encoding'] = 'base64' + msg.set_payload(x) + self.assertEqual(msg.get_payload(decode=True), + (b'\x03\x00\xe9\xd0\xfe\xff\xff.\x8b\xc0' + b'\xa1\x00p\xf6\xbf\xe9\x0f')) + self.assertIsInstance(msg.defects[0], + errors.InvalidBase64CharactersDefect) + + def test_broken_unicode_payload(self): + # This test improves coverage but is not a compliance test. + # The behavior in this situation is currently undefined by the API. + x = 'this is a br\xf6ken thing to do' + msg = Message() + msg['content-type'] = 'text/plain' + msg['content-transfer-encoding'] = '8bit' + msg.set_payload(x) + self.assertEqual(msg.get_payload(decode=True), + bytes(x, 'raw-unicode-escape')) + + def test_questionable_bytes_payload(self): + # This test improves coverage but is not a compliance test, + # since it involves poking inside the black box. + x = 'this is a quéstionable thing to do'.encode('utf-8') + msg = Message() + msg['content-type'] = 'text/plain; charset="utf-8"' + msg['content-transfer-encoding'] = '8bit' + msg._payload = x + self.assertEqual(msg.get_payload(decode=True), x) + + # Issue 1078919 + def test_ascii_add_header(self): + msg = Message() + msg.add_header('Content-Disposition', 'attachment', + filename='bud.gif') + self.assertEqual('attachment; filename="bud.gif"', + msg['Content-Disposition']) + + def test_noascii_add_header(self): + msg = Message() + msg.add_header('Content-Disposition', 'attachment', + filename="Fußballer.ppt") + self.assertEqual( + 'attachment; filename*=utf-8\'\'Fu%C3%9Fballer.ppt', + msg['Content-Disposition']) + + def test_nonascii_add_header_via_triple(self): + msg = Message() + msg.add_header('Content-Disposition', 'attachment', + filename=('iso-8859-1', '', 'Fußballer.ppt')) + self.assertEqual( + 'attachment; filename*=iso-8859-1\'\'Fu%DFballer.ppt', + msg['Content-Disposition']) + + def test_ascii_add_header_with_tspecial(self): + msg = Message() + msg.add_header('Content-Disposition', 'attachment', + filename="windows [filename].ppt") + self.assertEqual( + 'attachment; filename="windows [filename].ppt"', + msg['Content-Disposition']) + + def test_nonascii_add_header_with_tspecial(self): + msg = Message() + msg.add_header('Content-Disposition', 'attachment', + filename="Fußballer [filename].ppt") + self.assertEqual( + "attachment; filename*=utf-8''Fu%C3%9Fballer%20%5Bfilename%5D.ppt", + msg['Content-Disposition']) + + def test_binary_quopri_payload(self): + for charset in ('latin-1', 'ascii'): + msg = Message() + msg['content-type'] = 'text/plain; charset=%s' % charset + msg['content-transfer-encoding'] = 'quoted-printable' + msg.set_payload(b'foo=e6=96=87bar') + self.assertEqual( + msg.get_payload(decode=True), + b'foo\xe6\x96\x87bar', + 'get_payload returns wrong result with charset %s.' % charset) + + def test_binary_base64_payload(self): + for charset in ('latin-1', 'ascii'): + msg = Message() + msg['content-type'] = 'text/plain; charset=%s' % charset + msg['content-transfer-encoding'] = 'base64' + msg.set_payload(b'Zm9v5paHYmFy') + self.assertEqual( + msg.get_payload(decode=True), + b'foo\xe6\x96\x87bar', + 'get_payload returns wrong result with charset %s.' % charset) + + def test_binary_uuencode_payload(self): + for charset in ('latin-1', 'ascii'): + for encoding in ('x-uuencode', 'uuencode', 'uue', 'x-uue'): + msg = Message() + msg['content-type'] = 'text/plain; charset=%s' % charset + msg['content-transfer-encoding'] = encoding + msg.set_payload(b"begin 666 -\n)9F]OYI:'8F%R\n \nend\n") + self.assertEqual( + msg.get_payload(decode=True), + b'foo\xe6\x96\x87bar', + str(('get_payload returns wrong result ', + 'with charset {0} and encoding {1}.')).\ + format(charset, encoding)) + + def test_add_header_with_name_only_param(self): + msg = Message() + msg.add_header('Content-Disposition', 'inline', foo_bar=None) + self.assertEqual("inline; foo-bar", msg['Content-Disposition']) + + def test_add_header_with_no_value(self): + msg = Message() + msg.add_header('X-Status', None) + self.assertEqual('', msg['X-Status']) + + # Issue 5871: reject an attempt to embed a header inside a header value + # (header injection attack). + def test_embedded_header_via_Header_rejected(self): + msg = Message() + msg['Dummy'] = Header('dummy\nX-Injected-Header: test') + self.assertRaises(errors.HeaderParseError, msg.as_string) + + def test_embedded_header_via_string_rejected(self): + msg = Message() + msg['Dummy'] = 'dummy\nX-Injected-Header: test' + self.assertRaises(errors.HeaderParseError, msg.as_string) + + def test_unicode_header_defaults_to_utf8_encoding(self): + # Issue 14291 + m = MIMEText('abc\n') + m['Subject'] = 'É test' + self.assertEqual(str(m),textwrap.dedent("""\ + Content-Type: text/plain; charset="us-ascii" + MIME-Version: 1.0 + Content-Transfer-Encoding: 7bit + Subject: =?utf-8?q?=C3=89_test?= + + abc + """)) + + def test_unicode_body_defaults_to_utf8_encoding(self): + # Issue 14291 + m = MIMEText('É testabc\n') + self.assertEqual(str(m),textwrap.dedent("""\ + Content-Type: text/plain; charset="utf-8" + MIME-Version: 1.0 + Content-Transfer-Encoding: base64 + + w4kgdGVzdGFiYwo= + """)) + + def test_string_payload_with_base64_cte(self): + msg = email.message_from_string(textwrap.dedent("""\ + Content-Transfer-Encoding: base64 + + SGVsbG8uIFRlc3Rpbmc= + """), policy=email.policy.default) + self.assertEqual(msg.get_payload(decode=True), b"Hello. Testing") + self.assertDefectsEqual(msg['content-transfer-encoding'].defects, []) + + + +# Test the email.encoders module +class TestEncoders(unittest.TestCase): + + def test_EncodersEncode_base64(self): + with openfile('python.gif', 'rb') as fp: + bindata = fp.read() + mimed = email.mime.image.MIMEImage(bindata) + base64ed = mimed.get_payload() + # the transfer-encoded body lines should all be <=76 characters + lines = base64ed.split('\n') + self.assertLessEqual(max([ len(x) for x in lines ]), 76) + + def test_encode_empty_payload(self): + eq = self.assertEqual + msg = Message() + msg.set_charset('us-ascii') + eq(msg['content-transfer-encoding'], '7bit') + + def test_default_cte(self): + eq = self.assertEqual + # 7bit data and the default us-ascii _charset + msg = MIMEText('hello world') + eq(msg['content-transfer-encoding'], '7bit') + # Similar, but with 8bit data + msg = MIMEText('hello \xf8 world') + eq(msg['content-transfer-encoding'], 'base64') + # And now with a different charset + msg = MIMEText('hello \xf8 world', _charset='iso-8859-1') + eq(msg['content-transfer-encoding'], 'quoted-printable') + + @unittest.expectedFailure # TODO: RUSTPYTHON; LookupError: unknown encoding: iso-2022-jp + def test_encode7or8bit(self): + # Make sure a charset whose input character set is 8bit but + # whose output character set is 7bit gets a transfer-encoding + # of 7bit. + eq = self.assertEqual + msg = MIMEText('文\n', _charset='euc-jp') + eq(msg['content-transfer-encoding'], '7bit') + eq(msg.as_string(), textwrap.dedent("""\ + MIME-Version: 1.0 + Content-Type: text/plain; charset="iso-2022-jp" + Content-Transfer-Encoding: 7bit + + \x1b$BJ8\x1b(B + """)) + + def test_qp_encode_latin1(self): + msg = MIMEText('\xe1\xf6\n', 'text', 'ISO-8859-1') + self.assertEqual(str(msg), textwrap.dedent("""\ + MIME-Version: 1.0 + Content-Type: text/text; charset="iso-8859-1" + Content-Transfer-Encoding: quoted-printable + + =E1=F6 + """)) + + def test_qp_encode_non_latin1(self): + # Issue 16948 + msg = MIMEText('\u017c\n', 'text', 'ISO-8859-2') + self.assertEqual(str(msg), textwrap.dedent("""\ + MIME-Version: 1.0 + Content-Type: text/text; charset="iso-8859-2" + Content-Transfer-Encoding: quoted-printable + + =BF + """)) + + +# Test long header wrapping +class TestLongHeaders(TestEmailBase): + + maxDiff = None + + def test_split_long_continuation(self): + eq = self.ndiffAssertEqual + msg = email.message_from_string("""\ +Subject: bug demonstration +\t12345678911234567892123456789312345678941234567895123456789612345678971234567898112345678911234567892123456789112345678911234567892123456789 +\tmore text + +test +""") + sfp = StringIO() + g = Generator(sfp) + g.flatten(msg) + eq(sfp.getvalue(), """\ +Subject: bug demonstration +\t12345678911234567892123456789312345678941234567895123456789612345678971234567898112345678911234567892123456789112345678911234567892123456789 +\tmore text + +test +""") + + def test_another_long_almost_unsplittable_header(self): + eq = self.ndiffAssertEqual + hstr = """\ +bug demonstration +\t12345678911234567892123456789312345678941234567895123456789612345678971234567898112345678911234567892123456789112345678911234567892123456789 +\tmore text""" + h = Header(hstr, continuation_ws='\t') + eq(h.encode(), """\ +bug demonstration +\t12345678911234567892123456789312345678941234567895123456789612345678971234567898112345678911234567892123456789112345678911234567892123456789 +\tmore text""") + h = Header(hstr.replace('\t', ' ')) + eq(h.encode(), """\ +bug demonstration + 12345678911234567892123456789312345678941234567895123456789612345678971234567898112345678911234567892123456789112345678911234567892123456789 + more text""") + + def test_long_nonstring(self): + eq = self.ndiffAssertEqual + g = Charset("iso-8859-1") + cz = Charset("iso-8859-2") + utf8 = Charset("utf-8") + g_head = (b'Die Mieter treten hier ein werden mit einem Foerderband ' + b'komfortabel den Korridor entlang, an s\xfcdl\xfcndischen ' + b'Wandgem\xe4lden vorbei, gegen die rotierenden Klingen ' + b'bef\xf6rdert. ') + cz_head = (b'Finan\xe8ni metropole se hroutily pod tlakem jejich ' + b'd\xf9vtipu.. ') + utf8_head = ('\u6b63\u78ba\u306b\u8a00\u3046\u3068\u7ffb\u8a33\u306f' + '\u3055\u308c\u3066\u3044\u307e\u305b\u3093\u3002\u4e00' + '\u90e8\u306f\u30c9\u30a4\u30c4\u8a9e\u3067\u3059\u304c' + '\u3001\u3042\u3068\u306f\u3067\u305f\u3089\u3081\u3067' + '\u3059\u3002\u5b9f\u969b\u306b\u306f\u300cWenn ist das ' + 'Nunstuck git und Slotermeyer? Ja! Beiherhund das Oder ' + 'die Flipperwaldt gersput.\u300d\u3068\u8a00\u3063\u3066' + '\u3044\u307e\u3059\u3002') + h = Header(g_head, g, header_name='Subject') + h.append(cz_head, cz) + h.append(utf8_head, utf8) + msg = Message() + msg['Subject'] = h + sfp = StringIO() + g = Generator(sfp) + g.flatten(msg) + eq(sfp.getvalue(), """\ +Subject: =?iso-8859-1?q?Die_Mieter_treten_hier_ein_werden_mit_einem_Foerderb?= + =?iso-8859-1?q?and_komfortabel_den_Korridor_entlang=2C_an_s=FCdl=FCndischen?= + =?iso-8859-1?q?_Wandgem=E4lden_vorbei=2C_gegen_die_rotierenden_Klingen_bef?= + =?iso-8859-1?q?=F6rdert=2E_?= =?iso-8859-2?q?Finan=E8ni_metropole_se_hrouti?= + =?iso-8859-2?q?ly_pod_tlakem_jejich_d=F9vtipu=2E=2E_?= =?utf-8?b?5q2j56K6?= + =?utf-8?b?44Gr6KiA44GG44Go57+76Kiz44Gv44GV44KM44Gm44GE44G+44Gb44KT44CC5LiA?= + =?utf-8?b?6YOo44Gv44OJ44Kk44OE6Kqe44Gn44GZ44GM44CB44GC44Go44Gv44Gn44Gf44KJ?= + =?utf-8?b?44KB44Gn44GZ44CC5a6f6Zqb44Gr44Gv44CMV2VubiBpc3QgZGFzIE51bnN0dWNr?= + =?utf-8?b?IGdpdCB1bmQgU2xvdGVybWV5ZXI/IEphISBCZWloZXJodW5kIGRhcyBPZGVyIGRp?= + =?utf-8?b?ZSBGbGlwcGVyd2FsZHQgZ2Vyc3B1dC7jgI3jgajoqIDjgaPjgabjgYTjgb7jgZk=?= + =?utf-8?b?44CC?= + +""") + eq(h.encode(maxlinelen=76), """\ +=?iso-8859-1?q?Die_Mieter_treten_hier_ein_werden_mit_einem_Foerde?= + =?iso-8859-1?q?rband_komfortabel_den_Korridor_entlang=2C_an_s=FCdl=FCndis?= + =?iso-8859-1?q?chen_Wandgem=E4lden_vorbei=2C_gegen_die_rotierenden_Klinge?= + =?iso-8859-1?q?n_bef=F6rdert=2E_?= =?iso-8859-2?q?Finan=E8ni_metropole_se?= + =?iso-8859-2?q?_hroutily_pod_tlakem_jejich_d=F9vtipu=2E=2E_?= + =?utf-8?b?5q2j56K644Gr6KiA44GG44Go57+76Kiz44Gv44GV44KM44Gm44GE44G+44Gb?= + =?utf-8?b?44KT44CC5LiA6YOo44Gv44OJ44Kk44OE6Kqe44Gn44GZ44GM44CB44GC44Go?= + =?utf-8?b?44Gv44Gn44Gf44KJ44KB44Gn44GZ44CC5a6f6Zqb44Gr44Gv44CMV2VubiBp?= + =?utf-8?b?c3QgZGFzIE51bnN0dWNrIGdpdCB1bmQgU2xvdGVybWV5ZXI/IEphISBCZWlo?= + =?utf-8?b?ZXJodW5kIGRhcyBPZGVyIGRpZSBGbGlwcGVyd2FsZHQgZ2Vyc3B1dC7jgI0=?= + =?utf-8?b?44Go6KiA44Gj44Gm44GE44G+44GZ44CC?=""") + + def test_long_header_encode(self): + eq = self.ndiffAssertEqual + h = Header('wasnipoop; giraffes="very-long-necked-animals"; ' + 'spooge="yummy"; hippos="gargantuan"; marshmallows="gooey"', + header_name='X-Foobar-Spoink-Defrobnit') + eq(h.encode(), '''\ +wasnipoop; giraffes="very-long-necked-animals"; + spooge="yummy"; hippos="gargantuan"; marshmallows="gooey"''') + + def test_long_header_encode_with_tab_continuation_is_just_a_hint(self): + eq = self.ndiffAssertEqual + h = Header('wasnipoop; giraffes="very-long-necked-animals"; ' + 'spooge="yummy"; hippos="gargantuan"; marshmallows="gooey"', + header_name='X-Foobar-Spoink-Defrobnit', + continuation_ws='\t') + eq(h.encode(), '''\ +wasnipoop; giraffes="very-long-necked-animals"; + spooge="yummy"; hippos="gargantuan"; marshmallows="gooey"''') + + def test_long_header_encode_with_tab_continuation(self): + eq = self.ndiffAssertEqual + h = Header('wasnipoop; giraffes="very-long-necked-animals";\t' + 'spooge="yummy"; hippos="gargantuan"; marshmallows="gooey"', + header_name='X-Foobar-Spoink-Defrobnit', + continuation_ws='\t') + eq(h.encode(), '''\ +wasnipoop; giraffes="very-long-necked-animals"; +\tspooge="yummy"; hippos="gargantuan"; marshmallows="gooey"''') + + @unittest.expectedFailure # TODO: RUSTPYTHON; LookupError: unknown encoding: iso-2022-jp + def test_header_encode_with_different_output_charset(self): + h = Header('文', 'euc-jp') + self.assertEqual(h.encode(), "=?iso-2022-jp?b?GyRCSjgbKEI=?=") + + @unittest.expectedFailure # TODO: RUSTPYTHON; LookupError: unknown encoding: euc-jp + def test_long_header_encode_with_different_output_charset(self): + h = Header(b'test-ja \xa4\xd8\xc5\xea\xb9\xc6\xa4\xb5\xa4\xec\xa4' + b'\xbf\xa5\xe1\xa1\xbc\xa5\xeb\xa4\xcf\xbb\xca\xb2\xf1\xbc\xd4' + b'\xa4\xce\xbe\xb5\xc7\xa7\xa4\xf2\xc2\xd4\xa4\xc3\xa4\xc6\xa4' + b'\xa4\xa4\xde\xa4\xb9'.decode('euc-jp'), 'euc-jp') + res = """\ +=?iso-2022-jp?b?dGVzdC1qYSAbJEIkWEVqOUYkNSRsJD8lYSE8JWskTztKMnE8VCROPjUbKEI=?= + =?iso-2022-jp?b?GyRCRyckckJUJEMkRiQkJF4kORsoQg==?=""" + self.assertEqual(h.encode(), res) + + def test_header_splitter(self): + eq = self.ndiffAssertEqual + msg = MIMEText('') + # It'd be great if we could use add_header() here, but that doesn't + # guarantee an order of the parameters. + msg['X-Foobar-Spoink-Defrobnit'] = ( + 'wasnipoop; giraffes="very-long-necked-animals"; ' + 'spooge="yummy"; hippos="gargantuan"; marshmallows="gooey"') + sfp = StringIO() + g = Generator(sfp) + g.flatten(msg) + eq(sfp.getvalue(), '''\ +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit +X-Foobar-Spoink-Defrobnit: wasnipoop; giraffes="very-long-necked-animals"; + spooge="yummy"; hippos="gargantuan"; marshmallows="gooey" + +''') + + def test_no_semis_header_splitter(self): + eq = self.ndiffAssertEqual + msg = Message() + msg['From'] = 'test@dom.ain' + msg['References'] = SPACE.join('<%d@dom.ain>' % i for i in range(10)) + msg.set_payload('Test') + sfp = StringIO() + g = Generator(sfp) + g.flatten(msg) + eq(sfp.getvalue(), """\ +From: test@dom.ain +References: <0@dom.ain> <1@dom.ain> <2@dom.ain> <3@dom.ain> <4@dom.ain> + <5@dom.ain> <6@dom.ain> <7@dom.ain> <8@dom.ain> <9@dom.ain> + +Test""") + + def test_last_split_chunk_does_not_fit(self): + eq = self.ndiffAssertEqual + h = Header('Subject: the first part of this is short, but_the_second' + '_part_does_not_fit_within_maxlinelen_and_thus_should_be_on_a_line' + '_all_by_itself') + eq(h.encode(), """\ +Subject: the first part of this is short, + but_the_second_part_does_not_fit_within_maxlinelen_and_thus_should_be_on_a_line_all_by_itself""") + + def test_splittable_leading_char_followed_by_overlong_unsplittable(self): + eq = self.ndiffAssertEqual + h = Header(', but_the_second' + '_part_does_not_fit_within_maxlinelen_and_thus_should_be_on_a_line' + '_all_by_itself') + eq(h.encode(), """\ +, + but_the_second_part_does_not_fit_within_maxlinelen_and_thus_should_be_on_a_line_all_by_itself""") + + def test_multiple_splittable_leading_char_followed_by_overlong_unsplittable(self): + eq = self.ndiffAssertEqual + h = Header(', , but_the_second' + '_part_does_not_fit_within_maxlinelen_and_thus_should_be_on_a_line' + '_all_by_itself') + eq(h.encode(), """\ +, , + but_the_second_part_does_not_fit_within_maxlinelen_and_thus_should_be_on_a_line_all_by_itself""") + + def test_trailing_splittable_on_overlong_unsplittable(self): + eq = self.ndiffAssertEqual + h = Header('this_part_does_not_fit_within_maxlinelen_and_thus_should_' + 'be_on_a_line_all_by_itself;') + eq(h.encode(), "this_part_does_not_fit_within_maxlinelen_and_thus_should_" + "be_on_a_line_all_by_itself;") + + def test_trailing_splittable_on_overlong_unsplittable_with_leading_splittable(self): + eq = self.ndiffAssertEqual + h = Header('; ' + 'this_part_does_not_fit_within_maxlinelen_and_thus_should_' + 'be_on_a_line_all_by_itself; ') + eq(h.encode(), """\ +; + this_part_does_not_fit_within_maxlinelen_and_thus_should_be_on_a_line_all_by_itself; """) + + def test_long_header_with_multiple_sequential_split_chars(self): + eq = self.ndiffAssertEqual + h = Header('This is a long line that has two whitespaces in a row. ' + 'This used to cause truncation of the header when folded') + eq(h.encode(), """\ +This is a long line that has two whitespaces in a row. This used to cause + truncation of the header when folded""") + + def test_splitter_split_on_punctuation_only_if_fws_with_header(self): + eq = self.ndiffAssertEqual + h = Header('thisverylongheaderhas;semicolons;and,commas,but' + 'they;arenotlegal;fold,points') + eq(h.encode(), "thisverylongheaderhas;semicolons;and,commas,butthey;" + "arenotlegal;fold,points") + + def test_leading_splittable_in_the_middle_just_before_overlong_last_part(self): + eq = self.ndiffAssertEqual + h = Header('this is a test where we need to have more than one line ' + 'before; our final line that is just too big to fit;; ' + 'this_part_does_not_fit_within_maxlinelen_and_thus_should_' + 'be_on_a_line_all_by_itself;') + eq(h.encode(), """\ +this is a test where we need to have more than one line before; + our final line that is just too big to fit;; + this_part_does_not_fit_within_maxlinelen_and_thus_should_be_on_a_line_all_by_itself;""") + + def test_overlong_last_part_followed_by_split_point(self): + eq = self.ndiffAssertEqual + h = Header('this_part_does_not_fit_within_maxlinelen_and_thus_should_' + 'be_on_a_line_all_by_itself ') + eq(h.encode(), "this_part_does_not_fit_within_maxlinelen_and_thus_" + "should_be_on_a_line_all_by_itself ") + + def test_multiline_with_overlong_parts_separated_by_two_split_points(self): + eq = self.ndiffAssertEqual + h = Header('this_is_a__test_where_we_need_to_have_more_than_one_line_' + 'before_our_final_line_; ; ' + 'this_part_does_not_fit_within_maxlinelen_and_thus_should_' + 'be_on_a_line_all_by_itself; ') + eq(h.encode(), """\ +this_is_a__test_where_we_need_to_have_more_than_one_line_before_our_final_line_; + ; + this_part_does_not_fit_within_maxlinelen_and_thus_should_be_on_a_line_all_by_itself; """) + + def test_multiline_with_overlong_last_part_followed_by_split_point(self): + eq = self.ndiffAssertEqual + h = Header('this is a test where we need to have more than one line ' + 'before our final line; ; ' + 'this_part_does_not_fit_within_maxlinelen_and_thus_should_' + 'be_on_a_line_all_by_itself; ') + eq(h.encode(), """\ +this is a test where we need to have more than one line before our final line; + ; + this_part_does_not_fit_within_maxlinelen_and_thus_should_be_on_a_line_all_by_itself; """) + + def test_long_header_with_whitespace_runs(self): + eq = self.ndiffAssertEqual + msg = Message() + msg['From'] = 'test@dom.ain' + msg['References'] = SPACE.join([' '] * 10) + msg.set_payload('Test') + sfp = StringIO() + g = Generator(sfp) + g.flatten(msg) + eq(sfp.getvalue(), """\ +From: test@dom.ain +References: + + \x20\x20 + +Test""") + + def test_long_run_with_semi_header_splitter(self): + eq = self.ndiffAssertEqual + msg = Message() + msg['From'] = 'test@dom.ain' + msg['References'] = SPACE.join([''] * 10) + '; abc' + msg.set_payload('Test') + sfp = StringIO() + g = Generator(sfp) + g.flatten(msg) + eq(sfp.getvalue(), """\ +From: test@dom.ain +References: + + ; abc + +Test""") + + def test_splitter_split_on_punctuation_only_if_fws(self): + eq = self.ndiffAssertEqual + msg = Message() + msg['From'] = 'test@dom.ain' + msg['References'] = ('thisverylongheaderhas;semicolons;and,commas,but' + 'they;arenotlegal;fold,points') + msg.set_payload('Test') + sfp = StringIO() + g = Generator(sfp) + g.flatten(msg) + # XXX the space after the header should not be there. + eq(sfp.getvalue(), """\ +From: test@dom.ain +References:\x20 + thisverylongheaderhas;semicolons;and,commas,butthey;arenotlegal;fold,points + +Test""") + + def test_no_split_long_header(self): + eq = self.ndiffAssertEqual + hstr = 'References: ' + 'x' * 80 + h = Header(hstr) + # These come on two lines because Headers are really field value + # classes and don't really know about their field names. + eq(h.encode(), """\ +References: + xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx""") + h = Header('x' * 80) + eq(h.encode(), 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx') + + def test_splitting_multiple_long_lines(self): + eq = self.ndiffAssertEqual + hstr = """\ +from babylon.socal-raves.org (localhost [127.0.0.1]); by babylon.socal-raves.org (Postfix) with ESMTP id B570E51B81; for ; Sat, 2 Feb 2002 17:00:06 -0800 (PST) +\tfrom babylon.socal-raves.org (localhost [127.0.0.1]); by babylon.socal-raves.org (Postfix) with ESMTP id B570E51B81; for ; Sat, 2 Feb 2002 17:00:06 -0800 (PST) +\tfrom babylon.socal-raves.org (localhost [127.0.0.1]); by babylon.socal-raves.org (Postfix) with ESMTP id B570E51B81; for ; Sat, 2 Feb 2002 17:00:06 -0800 (PST) +""" + h = Header(hstr, continuation_ws='\t') + eq(h.encode(), """\ +from babylon.socal-raves.org (localhost [127.0.0.1]); + by babylon.socal-raves.org (Postfix) with ESMTP id B570E51B81; + for ; + Sat, 2 Feb 2002 17:00:06 -0800 (PST) +\tfrom babylon.socal-raves.org (localhost [127.0.0.1]); + by babylon.socal-raves.org (Postfix) with ESMTP id B570E51B81; + for ; + Sat, 2 Feb 2002 17:00:06 -0800 (PST) +\tfrom babylon.socal-raves.org (localhost [127.0.0.1]); + by babylon.socal-raves.org (Postfix) with ESMTP id B570E51B81; + for ; + Sat, 2 Feb 2002 17:00:06 -0800 (PST)""") + + def test_splitting_first_line_only_is_long(self): + eq = self.ndiffAssertEqual + hstr = """\ +from modemcable093.139-201-24.que.mc.videotron.ca ([24.201.139.93] helo=cthulhu.gerg.ca) +\tby kronos.mems-exchange.org with esmtp (Exim 4.05) +\tid 17k4h5-00034i-00 +\tfor test@mems-exchange.org; Wed, 28 Aug 2002 11:25:20 -0400""" + h = Header(hstr, maxlinelen=78, header_name='Received', + continuation_ws='\t') + eq(h.encode(), """\ +from modemcable093.139-201-24.que.mc.videotron.ca ([24.201.139.93] + helo=cthulhu.gerg.ca) +\tby kronos.mems-exchange.org with esmtp (Exim 4.05) +\tid 17k4h5-00034i-00 +\tfor test@mems-exchange.org; Wed, 28 Aug 2002 11:25:20 -0400""") + + def test_long_8bit_header(self): + eq = self.ndiffAssertEqual + msg = Message() + h = Header('Britische Regierung gibt', 'iso-8859-1', + header_name='Subject') + h.append('gr\xfcnes Licht f\xfcr Offshore-Windkraftprojekte') + eq(h.encode(maxlinelen=76), """\ +=?iso-8859-1?q?Britische_Regierung_gibt_gr=FCnes_Licht_f=FCr_Offs?= + =?iso-8859-1?q?hore-Windkraftprojekte?=""") + msg['Subject'] = h + eq(msg.as_string(maxheaderlen=76), """\ +Subject: =?iso-8859-1?q?Britische_Regierung_gibt_gr=FCnes_Licht_f=FCr_Offs?= + =?iso-8859-1?q?hore-Windkraftprojekte?= + +""") + eq(msg.as_string(maxheaderlen=0), """\ +Subject: =?iso-8859-1?q?Britische_Regierung_gibt_gr=FCnes_Licht_f=FCr_Offshore-Windkraftprojekte?= + +""") + + def test_long_8bit_header_no_charset(self): + eq = self.ndiffAssertEqual + msg = Message() + header_string = ('Britische Regierung gibt gr\xfcnes Licht ' + 'f\xfcr Offshore-Windkraftprojekte ' + '') + msg['Reply-To'] = header_string + eq(msg.as_string(maxheaderlen=78), """\ +Reply-To: =?utf-8?q?Britische_Regierung_gibt_gr=C3=BCnes_Licht_f=C3=BCr_Offs?= + =?utf-8?q?hore-Windkraftprojekte_=3Ca-very-long-address=40example=2Ecom=3E?= + +""") + msg = Message() + msg['Reply-To'] = Header(header_string, + header_name='Reply-To') + eq(msg.as_string(maxheaderlen=78), """\ +Reply-To: =?utf-8?q?Britische_Regierung_gibt_gr=C3=BCnes_Licht_f=C3=BCr_Offs?= + =?utf-8?q?hore-Windkraftprojekte_=3Ca-very-long-address=40example=2Ecom=3E?= + +""") + + def test_long_to_header(self): + eq = self.ndiffAssertEqual + to = ('"Someone Test #A" ,' + ', ' + '"Someone Test #B" , ' + '"Someone Test #C" , ' + '"Someone Test #D" ') + msg = Message() + msg['To'] = to + eq(msg.as_string(maxheaderlen=78), '''\ +To: "Someone Test #A" ,, + "Someone Test #B" , + "Someone Test #C" , + "Someone Test #D" + +''') + + def test_long_line_after_append(self): + eq = self.ndiffAssertEqual + s = 'This is an example of string which has almost the limit of header length.' + h = Header(s) + h.append('Add another line.') + eq(h.encode(maxlinelen=76), """\ +This is an example of string which has almost the limit of header length. + Add another line.""") + + def test_shorter_line_with_append(self): + eq = self.ndiffAssertEqual + s = 'This is a shorter line.' + h = Header(s) + h.append('Add another sentence. (Surprise?)') + eq(h.encode(), + 'This is a shorter line. Add another sentence. (Surprise?)') + + def test_long_field_name(self): + eq = self.ndiffAssertEqual + fn = 'X-Very-Very-Very-Long-Header-Name' + gs = ('Die Mieter treten hier ein werden mit einem Foerderband ' + 'komfortabel den Korridor entlang, an s\xfcdl\xfcndischen ' + 'Wandgem\xe4lden vorbei, gegen die rotierenden Klingen ' + 'bef\xf6rdert. ') + h = Header(gs, 'iso-8859-1', header_name=fn) + # BAW: this seems broken because the first line is too long + eq(h.encode(maxlinelen=76), """\ +=?iso-8859-1?q?Die_Mieter_treten_hier_e?= + =?iso-8859-1?q?in_werden_mit_einem_Foerderband_komfortabel_den_Korridor_e?= + =?iso-8859-1?q?ntlang=2C_an_s=FCdl=FCndischen_Wandgem=E4lden_vorbei=2C_ge?= + =?iso-8859-1?q?gen_die_rotierenden_Klingen_bef=F6rdert=2E_?=""") + + def test_long_received_header(self): + h = ('from FOO.TLD (vizworld.acl.foo.tld [123.452.678.9]) ' + 'by hrothgar.la.mastaler.com (tmda-ofmipd) with ESMTP; ' + 'Wed, 05 Mar 2003 18:10:18 -0700') + msg = Message() + msg['Received-1'] = Header(h, continuation_ws='\t') + msg['Received-2'] = h + # This should be splitting on spaces not semicolons. + self.ndiffAssertEqual(msg.as_string(maxheaderlen=78), """\ +Received-1: from FOO.TLD (vizworld.acl.foo.tld [123.452.678.9]) by + hrothgar.la.mastaler.com (tmda-ofmipd) with ESMTP; + Wed, 05 Mar 2003 18:10:18 -0700 +Received-2: from FOO.TLD (vizworld.acl.foo.tld [123.452.678.9]) by + hrothgar.la.mastaler.com (tmda-ofmipd) with ESMTP; + Wed, 05 Mar 2003 18:10:18 -0700 + +""") + + def test_string_headerinst_eq(self): + h = ('<15975.17901.207240.414604@sgigritzmann1.mathematik.' + 'tu-muenchen.de> (David Bremner\'s message of ' + '"Thu, 6 Mar 2003 13:58:21 +0100")') + msg = Message() + msg['Received-1'] = Header(h, header_name='Received-1', + continuation_ws='\t') + msg['Received-2'] = h + # XXX The space after the ':' should not be there. + self.ndiffAssertEqual(msg.as_string(maxheaderlen=78), """\ +Received-1:\x20 + <15975.17901.207240.414604@sgigritzmann1.mathematik.tu-muenchen.de> (David + Bremner's message of \"Thu, 6 Mar 2003 13:58:21 +0100\") +Received-2:\x20 + <15975.17901.207240.414604@sgigritzmann1.mathematik.tu-muenchen.de> (David + Bremner's message of \"Thu, 6 Mar 2003 13:58:21 +0100\") + +""") + + def test_long_unbreakable_lines_with_continuation(self): + eq = self.ndiffAssertEqual + msg = Message() + t = """\ +iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAGFBMVEUAAAAkHiJeRUIcGBi9 + locQDQ4zJykFBAXJfWDjAAACYUlEQVR4nF2TQY/jIAyFc6lydlG5x8Nyp1Y69wj1PN2I5gzp""" + msg['Face-1'] = t + msg['Face-2'] = Header(t, header_name='Face-2') + msg['Face-3'] = ' ' + t + # XXX This splitting is all wrong. It the first value line should be + # snug against the field name or the space after the header not there. + eq(msg.as_string(maxheaderlen=78), """\ +Face-1:\x20 + iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAGFBMVEUAAAAkHiJeRUIcGBi9 + locQDQ4zJykFBAXJfWDjAAACYUlEQVR4nF2TQY/jIAyFc6lydlG5x8Nyp1Y69wj1PN2I5gzp +Face-2:\x20 + iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAGFBMVEUAAAAkHiJeRUIcGBi9 + locQDQ4zJykFBAXJfWDjAAACYUlEQVR4nF2TQY/jIAyFc6lydlG5x8Nyp1Y69wj1PN2I5gzp +Face-3:\x20 + iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAGFBMVEUAAAAkHiJeRUIcGBi9 + locQDQ4zJykFBAXJfWDjAAACYUlEQVR4nF2TQY/jIAyFc6lydlG5x8Nyp1Y69wj1PN2I5gzp + +""") + + def test_another_long_multiline_header(self): + eq = self.ndiffAssertEqual + m = ('Received: from siimage.com ' + '([172.25.1.3]) by zima.siliconimage.com with ' + 'Microsoft SMTPSVC(5.0.2195.4905); ' + 'Wed, 16 Oct 2002 07:41:11 -0700') + msg = email.message_from_string(m) + eq(msg.as_string(maxheaderlen=78), '''\ +Received: from siimage.com ([172.25.1.3]) by zima.siliconimage.com with + Microsoft SMTPSVC(5.0.2195.4905); Wed, 16 Oct 2002 07:41:11 -0700 + +''') + + def test_long_lines_with_different_header(self): + eq = self.ndiffAssertEqual + h = ('List-Unsubscribe: ' + ',' + ' ') + msg = Message() + msg['List'] = h + msg['List'] = Header(h, header_name='List') + eq(msg.as_string(maxheaderlen=78), """\ +List: List-Unsubscribe: + , + +List: List-Unsubscribe: + , + + +""") + + def test_long_rfc2047_header_with_embedded_fws(self): + h = Header(textwrap.dedent("""\ + We're going to pretend this header is in a non-ascii character set + \tto see if line wrapping with encoded words and embedded + folding white space works"""), + charset='utf-8', + header_name='Test') + self.assertEqual(h.encode()+'\n', textwrap.dedent("""\ + =?utf-8?q?We=27re_going_to_pretend_this_header_is_in_a_non-ascii_chara?= + =?utf-8?q?cter_set?= + =?utf-8?q?_to_see_if_line_wrapping_with_encoded_words_and_embedded?= + =?utf-8?q?_folding_white_space_works?=""")+'\n') + + + +# Test mangling of "From " lines in the body of a message +class TestFromMangling(unittest.TestCase): + def setUp(self): + self.msg = Message() + self.msg['From'] = 'aaa@bbb.org' + self.msg.set_payload("""\ +From the desk of A.A.A.: +Blah blah blah +""") + + def test_mangled_from(self): + s = StringIO() + g = Generator(s, mangle_from_=True) + g.flatten(self.msg) + self.assertEqual(s.getvalue(), """\ +From: aaa@bbb.org + +>From the desk of A.A.A.: +Blah blah blah +""") + + def test_dont_mangle_from(self): + s = StringIO() + g = Generator(s, mangle_from_=False) + g.flatten(self.msg) + self.assertEqual(s.getvalue(), """\ +From: aaa@bbb.org + +From the desk of A.A.A.: +Blah blah blah +""") + + def test_mangle_from_in_preamble_and_epilog(self): + s = StringIO() + g = Generator(s, mangle_from_=True) + msg = email.message_from_string(textwrap.dedent("""\ + From: foo@bar.com + Mime-Version: 1.0 + Content-Type: multipart/mixed; boundary=XXX + + From somewhere unknown + + --XXX + Content-Type: text/plain + + foo + + --XXX-- + + From somewhere unknowable + """)) + g.flatten(msg) + self.assertEqual(len([1 for x in s.getvalue().split('\n') + if x.startswith('>From ')]), 2) + + def test_mangled_from_with_bad_bytes(self): + source = textwrap.dedent("""\ + Content-Type: text/plain; charset="utf-8" + MIME-Version: 1.0 + Content-Transfer-Encoding: 8bit + From: aaa@bbb.org + + """).encode('utf-8') + msg = email.message_from_bytes(source + b'From R\xc3\xb6lli\n') + b = BytesIO() + g = BytesGenerator(b, mangle_from_=True) + g.flatten(msg) + self.assertEqual(b.getvalue(), source + b'>From R\xc3\xb6lli\n') + + def test_multipart_with_bad_bytes_in_cte(self): + # bpo30835 + source = textwrap.dedent("""\ + From: aperson@example.com + Content-Type: multipart/mixed; boundary="1" + Content-Transfer-Encoding: \xc8 + """).encode('utf-8') + msg = email.message_from_bytes(source) + + +# Test the basic MIMEAudio class +class TestMIMEAudio(unittest.TestCase): + def _make_audio(self, ext): + with openfile(f'sndhdr.{ext}', 'rb') as fp: + self._audiodata = fp.read() + self._au = MIMEAudio(self._audiodata) + + def test_guess_minor_type(self): + for ext, subtype in { + 'aifc': 'x-aiff', + 'aiff': 'x-aiff', + 'wav': 'x-wav', + 'au': 'basic', + }.items(): + self._make_audio(ext) + subtype = ext if subtype is None else subtype + self.assertEqual(self._au.get_content_type(), f'audio/{subtype}') + + def test_encoding(self): + self._make_audio('au') + payload = self._au.get_payload() + self.assertEqual(base64.decodebytes(bytes(payload, 'ascii')), + self._audiodata) + + def test_checkSetMinor(self): + self._make_audio('au') + au = MIMEAudio(self._audiodata, 'fish') + self.assertEqual(au.get_content_type(), 'audio/fish') + + def test_add_header(self): + self._make_audio('au') + eq = self.assertEqual + self._au.add_header('Content-Disposition', 'attachment', + filename='sndhdr.au') + eq(self._au['content-disposition'], + 'attachment; filename="sndhdr.au"') + eq(self._au.get_params(header='content-disposition'), + [('attachment', ''), ('filename', 'sndhdr.au')]) + eq(self._au.get_param('filename', header='content-disposition'), + 'sndhdr.au') + missing = [] + eq(self._au.get_param('attachment', header='content-disposition'), '') + self.assertIs(self._au.get_param( + 'foo', failobj=missing, + header='content-disposition'), missing) + # Try some missing stuff + self.assertIs(self._au.get_param('foobar', missing), missing) + self.assertIs(self._au.get_param('attachment', missing, + header='foobar'), missing) + + + +# Test the basic MIMEImage class +class TestMIMEImage(unittest.TestCase): + def _make_image(self, ext): + with openfile(f'python.{ext}', 'rb') as fp: + self._imgdata = fp.read() + self._im = MIMEImage(self._imgdata) + + def test_guess_minor_type(self): + for ext, subtype in { + 'bmp': None, + 'exr': None, + 'gif': None, + 'jpg': 'jpeg', + 'pbm': None, + 'pgm': None, + 'png': None, + 'ppm': None, + 'ras': 'rast', + 'sgi': 'rgb', + 'tiff': None, + 'webp': None, + 'xbm': None, + }.items(): + self._make_image(ext) + subtype = ext if subtype is None else subtype + self.assertEqual(self._im.get_content_type(), f'image/{subtype}') + + def test_encoding(self): + self._make_image('gif') + payload = self._im.get_payload() + self.assertEqual(base64.decodebytes(bytes(payload, 'ascii')), + self._imgdata) + + def test_checkSetMinor(self): + self._make_image('gif') + im = MIMEImage(self._imgdata, 'fish') + self.assertEqual(im.get_content_type(), 'image/fish') + + def test_add_header(self): + self._make_image('gif') + eq = self.assertEqual + self._im.add_header('Content-Disposition', 'attachment', + filename='dingusfish.gif') + eq(self._im['content-disposition'], + 'attachment; filename="dingusfish.gif"') + eq(self._im.get_params(header='content-disposition'), + [('attachment', ''), ('filename', 'dingusfish.gif')]) + eq(self._im.get_param('filename', header='content-disposition'), + 'dingusfish.gif') + missing = [] + eq(self._im.get_param('attachment', header='content-disposition'), '') + self.assertIs(self._im.get_param('foo', failobj=missing, + header='content-disposition'), missing) + # Try some missing stuff + self.assertIs(self._im.get_param('foobar', missing), missing) + self.assertIs(self._im.get_param('attachment', missing, + header='foobar'), missing) + + +# Test the basic MIMEApplication class +class TestMIMEApplication(unittest.TestCase): + def test_headers(self): + eq = self.assertEqual + msg = MIMEApplication(b'\xfa\xfb\xfc\xfd\xfe\xff') + eq(msg.get_content_type(), 'application/octet-stream') + eq(msg['content-transfer-encoding'], 'base64') + + def test_body(self): + eq = self.assertEqual + bytesdata = b'\xfa\xfb\xfc\xfd\xfe\xff' + msg = MIMEApplication(bytesdata) + # whitespace in the cte encoded block is RFC-irrelevant. + eq(msg.get_payload().strip(), '+vv8/f7/') + eq(msg.get_payload(decode=True), bytesdata) + + def test_binary_body_with_encode_7or8bit(self): + # Issue 17171. + bytesdata = b'\xfa\xfb\xfc\xfd\xfe\xff' + msg = MIMEApplication(bytesdata, _encoder=encoders.encode_7or8bit) + # Treated as a string, this will be invalid code points. + self.assertEqual(msg.get_payload(), '\uFFFD' * len(bytesdata)) + self.assertEqual(msg.get_payload(decode=True), bytesdata) + self.assertEqual(msg['Content-Transfer-Encoding'], '8bit') + s = BytesIO() + g = BytesGenerator(s) + g.flatten(msg) + wireform = s.getvalue() + msg2 = email.message_from_bytes(wireform) + self.assertEqual(msg.get_payload(), '\uFFFD' * len(bytesdata)) + self.assertEqual(msg2.get_payload(decode=True), bytesdata) + self.assertEqual(msg2['Content-Transfer-Encoding'], '8bit') + + def test_binary_body_with_encode_noop(self): + # Issue 16564: This does not produce an RFC valid message, since to be + # valid it should have a CTE of binary. But the below works in + # Python2, and is documented as working this way. + bytesdata = b'\xfa\xfb\xfc\xfd\xfe\xff' + msg = MIMEApplication(bytesdata, _encoder=encoders.encode_noop) + # Treated as a string, this will be invalid code points. + self.assertEqual(msg.get_payload(), '\uFFFD' * len(bytesdata)) + self.assertEqual(msg.get_payload(decode=True), bytesdata) + s = BytesIO() + g = BytesGenerator(s) + g.flatten(msg) + wireform = s.getvalue() + msg2 = email.message_from_bytes(wireform) + self.assertEqual(msg.get_payload(), '\uFFFD' * len(bytesdata)) + self.assertEqual(msg2.get_payload(decode=True), bytesdata) + + def test_binary_body_with_unicode_linend_encode_noop(self): + # Issue 19003: This is a variation on #16564. + bytesdata = b'\x0b\xfa\xfb\xfc\xfd\xfe\xff' + msg = MIMEApplication(bytesdata, _encoder=encoders.encode_noop) + self.assertEqual(msg.get_payload(decode=True), bytesdata) + s = BytesIO() + g = BytesGenerator(s) + g.flatten(msg) + wireform = s.getvalue() + msg2 = email.message_from_bytes(wireform) + self.assertEqual(msg2.get_payload(decode=True), bytesdata) + + def test_binary_body_with_encode_quopri(self): + # Issue 14360. + bytesdata = b'\xfa\xfb\xfc\xfd\xfe\xff ' + msg = MIMEApplication(bytesdata, _encoder=encoders.encode_quopri) + self.assertEqual(msg.get_payload(), '=FA=FB=FC=FD=FE=FF=20') + self.assertEqual(msg.get_payload(decode=True), bytesdata) + self.assertEqual(msg['Content-Transfer-Encoding'], 'quoted-printable') + s = BytesIO() + g = BytesGenerator(s) + g.flatten(msg) + wireform = s.getvalue() + msg2 = email.message_from_bytes(wireform) + self.assertEqual(msg.get_payload(), '=FA=FB=FC=FD=FE=FF=20') + self.assertEqual(msg2.get_payload(decode=True), bytesdata) + self.assertEqual(msg2['Content-Transfer-Encoding'], 'quoted-printable') + + def test_binary_body_with_encode_base64(self): + bytesdata = b'\xfa\xfb\xfc\xfd\xfe\xff' + msg = MIMEApplication(bytesdata, _encoder=encoders.encode_base64) + self.assertEqual(msg.get_payload(), '+vv8/f7/\n') + self.assertEqual(msg.get_payload(decode=True), bytesdata) + s = BytesIO() + g = BytesGenerator(s) + g.flatten(msg) + wireform = s.getvalue() + msg2 = email.message_from_bytes(wireform) + self.assertEqual(msg.get_payload(), '+vv8/f7/\n') + self.assertEqual(msg2.get_payload(decode=True), bytesdata) + + +# Test the basic MIMEText class +class TestMIMEText(unittest.TestCase): + def setUp(self): + self._msg = MIMEText('hello there') + + def test_types(self): + eq = self.assertEqual + eq(self._msg.get_content_type(), 'text/plain') + eq(self._msg.get_param('charset'), 'us-ascii') + missing = [] + self.assertIs(self._msg.get_param('foobar', missing), missing) + self.assertIs(self._msg.get_param('charset', missing, header='foobar'), + missing) + + def test_payload(self): + self.assertEqual(self._msg.get_payload(), 'hello there') + self.assertFalse(self._msg.is_multipart()) + + def test_charset(self): + eq = self.assertEqual + msg = MIMEText('hello there', _charset='us-ascii') + eq(msg.get_charset().input_charset, 'us-ascii') + eq(msg['content-type'], 'text/plain; charset="us-ascii"') + # Also accept a Charset instance + charset = Charset('utf-8') + charset.body_encoding = None + msg = MIMEText('hello there', _charset=charset) + eq(msg.get_charset().input_charset, 'utf-8') + eq(msg['content-type'], 'text/plain; charset="utf-8"') + eq(msg.get_payload(), 'hello there') + + def test_7bit_input(self): + eq = self.assertEqual + msg = MIMEText('hello there', _charset='us-ascii') + eq(msg.get_charset().input_charset, 'us-ascii') + eq(msg['content-type'], 'text/plain; charset="us-ascii"') + + def test_7bit_input_no_charset(self): + eq = self.assertEqual + msg = MIMEText('hello there') + eq(msg.get_charset(), 'us-ascii') + eq(msg['content-type'], 'text/plain; charset="us-ascii"') + self.assertIn('hello there', msg.as_string()) + + def test_utf8_input(self): + teststr = '\u043a\u0438\u0440\u0438\u043b\u0438\u0446\u0430' + eq = self.assertEqual + msg = MIMEText(teststr, _charset='utf-8') + eq(msg.get_charset().output_charset, 'utf-8') + eq(msg['content-type'], 'text/plain; charset="utf-8"') + eq(msg.get_payload(decode=True), teststr.encode('utf-8')) + + @unittest.skip("can't fix because of backward compat in email5, " + "will fix in email6") + def test_utf8_input_no_charset(self): + teststr = '\u043a\u0438\u0440\u0438\u043b\u0438\u0446\u0430' + self.assertRaises(UnicodeEncodeError, MIMEText, teststr) + + + +# Test complicated multipart/* messages +class TestMultipart(TestEmailBase): + def setUp(self): + with openfile('python.gif', 'rb') as fp: + data = fp.read() + container = MIMEBase('multipart', 'mixed', boundary='BOUNDARY') + image = MIMEImage(data, name='dingusfish.gif') + image.add_header('content-disposition', 'attachment', + filename='dingusfish.gif') + intro = MIMEText('''\ +Hi there, + +This is the dingus fish. +''') + container.attach(intro) + container.attach(image) + container['From'] = 'Barry ' + container['To'] = 'Dingus Lovers ' + container['Subject'] = 'Here is your dingus fish' + + now = 987809702.54848599 + timetuple = time.localtime(now) + if timetuple[-1] == 0: + tzsecs = time.timezone + else: + tzsecs = time.altzone + if tzsecs > 0: + sign = '-' + else: + sign = '+' + tzoffset = ' %s%04d' % (sign, tzsecs / 36) + container['Date'] = time.strftime( + '%a, %d %b %Y %H:%M:%S', + time.localtime(now)) + tzoffset + self._msg = container + self._im = image + self._txt = intro + + def test_hierarchy(self): + # convenience + eq = self.assertEqual + raises = self.assertRaises + # tests + m = self._msg + self.assertTrue(m.is_multipart()) + eq(m.get_content_type(), 'multipart/mixed') + eq(len(m.get_payload()), 2) + raises(IndexError, m.get_payload, 2) + m0 = m.get_payload(0) + m1 = m.get_payload(1) + self.assertIs(m0, self._txt) + self.assertIs(m1, self._im) + eq(m.get_payload(), [m0, m1]) + self.assertFalse(m0.is_multipart()) + self.assertFalse(m1.is_multipart()) + + def test_empty_multipart_idempotent(self): + text = """\ +Content-Type: multipart/mixed; boundary="BOUNDARY" +MIME-Version: 1.0 +Subject: A subject +To: aperson@dom.ain +From: bperson@dom.ain + + +--BOUNDARY + + +--BOUNDARY-- +""" + msg = Parser().parsestr(text) + self.ndiffAssertEqual(text, msg.as_string()) + + def test_no_parts_in_a_multipart_with_none_epilogue(self): + outer = MIMEBase('multipart', 'mixed') + outer['Subject'] = 'A subject' + outer['To'] = 'aperson@dom.ain' + outer['From'] = 'bperson@dom.ain' + outer.set_boundary('BOUNDARY') + self.ndiffAssertEqual(outer.as_string(), '''\ +Content-Type: multipart/mixed; boundary="BOUNDARY" +MIME-Version: 1.0 +Subject: A subject +To: aperson@dom.ain +From: bperson@dom.ain + +--BOUNDARY + +--BOUNDARY-- +''') + + def test_no_parts_in_a_multipart_with_empty_epilogue(self): + outer = MIMEBase('multipart', 'mixed') + outer['Subject'] = 'A subject' + outer['To'] = 'aperson@dom.ain' + outer['From'] = 'bperson@dom.ain' + outer.preamble = '' + outer.epilogue = '' + outer.set_boundary('BOUNDARY') + self.ndiffAssertEqual(outer.as_string(), '''\ +Content-Type: multipart/mixed; boundary="BOUNDARY" +MIME-Version: 1.0 +Subject: A subject +To: aperson@dom.ain +From: bperson@dom.ain + + +--BOUNDARY + +--BOUNDARY-- +''') + + def test_one_part_in_a_multipart(self): + eq = self.ndiffAssertEqual + outer = MIMEBase('multipart', 'mixed') + outer['Subject'] = 'A subject' + outer['To'] = 'aperson@dom.ain' + outer['From'] = 'bperson@dom.ain' + outer.set_boundary('BOUNDARY') + msg = MIMEText('hello world') + outer.attach(msg) + eq(outer.as_string(), '''\ +Content-Type: multipart/mixed; boundary="BOUNDARY" +MIME-Version: 1.0 +Subject: A subject +To: aperson@dom.ain +From: bperson@dom.ain + +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit + +hello world +--BOUNDARY-- +''') + + def test_seq_parts_in_a_multipart_with_empty_preamble(self): + eq = self.ndiffAssertEqual + outer = MIMEBase('multipart', 'mixed') + outer['Subject'] = 'A subject' + outer['To'] = 'aperson@dom.ain' + outer['From'] = 'bperson@dom.ain' + outer.preamble = '' + msg = MIMEText('hello world') + outer.attach(msg) + outer.set_boundary('BOUNDARY') + eq(outer.as_string(), '''\ +Content-Type: multipart/mixed; boundary="BOUNDARY" +MIME-Version: 1.0 +Subject: A subject +To: aperson@dom.ain +From: bperson@dom.ain + + +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit + +hello world +--BOUNDARY-- +''') + + + def test_seq_parts_in_a_multipart_with_none_preamble(self): + eq = self.ndiffAssertEqual + outer = MIMEBase('multipart', 'mixed') + outer['Subject'] = 'A subject' + outer['To'] = 'aperson@dom.ain' + outer['From'] = 'bperson@dom.ain' + outer.preamble = None + msg = MIMEText('hello world') + outer.attach(msg) + outer.set_boundary('BOUNDARY') + eq(outer.as_string(), '''\ +Content-Type: multipart/mixed; boundary="BOUNDARY" +MIME-Version: 1.0 +Subject: A subject +To: aperson@dom.ain +From: bperson@dom.ain + +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit + +hello world +--BOUNDARY-- +''') + + + def test_seq_parts_in_a_multipart_with_none_epilogue(self): + eq = self.ndiffAssertEqual + outer = MIMEBase('multipart', 'mixed') + outer['Subject'] = 'A subject' + outer['To'] = 'aperson@dom.ain' + outer['From'] = 'bperson@dom.ain' + outer.epilogue = None + msg = MIMEText('hello world') + outer.attach(msg) + outer.set_boundary('BOUNDARY') + eq(outer.as_string(), '''\ +Content-Type: multipart/mixed; boundary="BOUNDARY" +MIME-Version: 1.0 +Subject: A subject +To: aperson@dom.ain +From: bperson@dom.ain + +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit + +hello world +--BOUNDARY-- +''') + + + def test_seq_parts_in_a_multipart_with_empty_epilogue(self): + eq = self.ndiffAssertEqual + outer = MIMEBase('multipart', 'mixed') + outer['Subject'] = 'A subject' + outer['To'] = 'aperson@dom.ain' + outer['From'] = 'bperson@dom.ain' + outer.epilogue = '' + msg = MIMEText('hello world') + outer.attach(msg) + outer.set_boundary('BOUNDARY') + eq(outer.as_string(), '''\ +Content-Type: multipart/mixed; boundary="BOUNDARY" +MIME-Version: 1.0 +Subject: A subject +To: aperson@dom.ain +From: bperson@dom.ain + +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit + +hello world +--BOUNDARY-- +''') + + + def test_seq_parts_in_a_multipart_with_nl_epilogue(self): + eq = self.ndiffAssertEqual + outer = MIMEBase('multipart', 'mixed') + outer['Subject'] = 'A subject' + outer['To'] = 'aperson@dom.ain' + outer['From'] = 'bperson@dom.ain' + outer.epilogue = '\n' + msg = MIMEText('hello world') + outer.attach(msg) + outer.set_boundary('BOUNDARY') + eq(outer.as_string(), '''\ +Content-Type: multipart/mixed; boundary="BOUNDARY" +MIME-Version: 1.0 +Subject: A subject +To: aperson@dom.ain +From: bperson@dom.ain + +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit + +hello world +--BOUNDARY-- + +''') + + def test_message_external_body(self): + eq = self.assertEqual + msg = self._msgobj('msg_36.txt') + eq(len(msg.get_payload()), 2) + msg1 = msg.get_payload(1) + eq(msg1.get_content_type(), 'multipart/alternative') + eq(len(msg1.get_payload()), 2) + for subpart in msg1.get_payload(): + eq(subpart.get_content_type(), 'message/external-body') + eq(len(subpart.get_payload()), 1) + subsubpart = subpart.get_payload(0) + eq(subsubpart.get_content_type(), 'text/plain') + + def test_double_boundary(self): + # msg_37.txt is a multipart that contains two dash-boundary's in a + # row. Our interpretation of RFC 2046 calls for ignoring the second + # and subsequent boundaries. + msg = self._msgobj('msg_37.txt') + self.assertEqual(len(msg.get_payload()), 3) + + def test_nested_inner_contains_outer_boundary(self): + eq = self.ndiffAssertEqual + # msg_38.txt has an inner part that contains outer boundaries. My + # interpretation of RFC 2046 (based on sections 5.1 and 5.1.2) say + # these are illegal and should be interpreted as unterminated inner + # parts. + msg = self._msgobj('msg_38.txt') + sfp = StringIO() + iterators._structure(msg, sfp) + eq(sfp.getvalue(), """\ +multipart/mixed + multipart/mixed + multipart/alternative + text/plain + text/plain + text/plain + text/plain +""") + + def test_nested_with_same_boundary(self): + eq = self.ndiffAssertEqual + # msg 39.txt is similarly evil in that it's got inner parts that use + # the same boundary as outer parts. Again, I believe the way this is + # parsed is closest to the spirit of RFC 2046 + msg = self._msgobj('msg_39.txt') + sfp = StringIO() + iterators._structure(msg, sfp) + eq(sfp.getvalue(), """\ +multipart/mixed + multipart/mixed + multipart/alternative + application/octet-stream + application/octet-stream + text/plain +""") + + def test_boundary_in_non_multipart(self): + msg = self._msgobj('msg_40.txt') + self.assertEqual(msg.as_string(), '''\ +MIME-Version: 1.0 +Content-Type: text/html; boundary="--961284236552522269" + +----961284236552522269 +Content-Type: text/html; +Content-Transfer-Encoding: 7Bit + + + +----961284236552522269-- +''') + + def test_boundary_with_leading_space(self): + eq = self.assertEqual + msg = email.message_from_string('''\ +MIME-Version: 1.0 +Content-Type: multipart/mixed; boundary=" XXXX" + +-- XXXX +Content-Type: text/plain + + +-- XXXX +Content-Type: text/plain + +-- XXXX-- +''') + self.assertTrue(msg.is_multipart()) + eq(msg.get_boundary(), ' XXXX') + eq(len(msg.get_payload()), 2) + + def test_boundary_without_trailing_newline(self): + m = Parser().parsestr("""\ +Content-Type: multipart/mixed; boundary="===============0012394164==" +MIME-Version: 1.0 + +--===============0012394164== +Content-Type: image/file1.jpg +MIME-Version: 1.0 +Content-Transfer-Encoding: base64 + +YXNkZg== +--===============0012394164==--""") + self.assertEqual(m.get_payload(0).get_payload(), 'YXNkZg==') + + def test_mimebase_default_policy(self): + m = MIMEBase('multipart', 'mixed') + self.assertIs(m.policy, email.policy.compat32) + + def test_mimebase_custom_policy(self): + m = MIMEBase('multipart', 'mixed', policy=email.policy.default) + self.assertIs(m.policy, email.policy.default) + +# Test some badly formatted messages +class TestNonConformant(TestEmailBase): + + def test_parse_missing_minor_type(self): + eq = self.assertEqual + msg = self._msgobj('msg_14.txt') + eq(msg.get_content_type(), 'text/plain') + eq(msg.get_content_maintype(), 'text') + eq(msg.get_content_subtype(), 'plain') + + # test_defect_handling + def test_same_boundary_inner_outer(self): + msg = self._msgobj('msg_15.txt') + # XXX We can probably eventually do better + inner = msg.get_payload(0) + self.assertHasAttr(inner, 'defects') + self.assertEqual(len(inner.defects), 1) + self.assertIsInstance(inner.defects[0], + errors.StartBoundaryNotFoundDefect) + + # test_defect_handling + def test_multipart_no_boundary(self): + msg = self._msgobj('msg_25.txt') + self.assertIsInstance(msg.get_payload(), str) + self.assertEqual(len(msg.defects), 2) + self.assertIsInstance(msg.defects[0], + errors.NoBoundaryInMultipartDefect) + self.assertIsInstance(msg.defects[1], + errors.MultipartInvariantViolationDefect) + + multipart_msg = textwrap.dedent("""\ + Date: Wed, 14 Nov 2007 12:56:23 GMT + From: foo@bar.invalid + To: foo@bar.invalid + Subject: Content-Transfer-Encoding: base64 and multipart + MIME-Version: 1.0 + Content-Type: multipart/mixed; + boundary="===============3344438784458119861=="{} + + --===============3344438784458119861== + Content-Type: text/plain + + Test message + + --===============3344438784458119861== + Content-Type: application/octet-stream + Content-Transfer-Encoding: base64 + + YWJj + + --===============3344438784458119861==-- + """) + + # test_defect_handling + def test_multipart_invalid_cte(self): + msg = self._str_msg( + self.multipart_msg.format("\nContent-Transfer-Encoding: base64")) + self.assertEqual(len(msg.defects), 1) + self.assertIsInstance(msg.defects[0], + errors.InvalidMultipartContentTransferEncodingDefect) + + # test_defect_handling + def test_multipart_no_cte_no_defect(self): + msg = self._str_msg(self.multipart_msg.format('')) + self.assertEqual(len(msg.defects), 0) + + # test_defect_handling + def test_multipart_valid_cte_no_defect(self): + for cte in ('7bit', '8bit', 'BINary'): + msg = self._str_msg( + self.multipart_msg.format( + "\nContent-Transfer-Encoding: {}".format(cte))) + self.assertEqual(len(msg.defects), 0) + + # test_headerregistry.TestContentTypeHeader invalid_1 and invalid_2. + def test_invalid_content_type(self): + eq = self.assertEqual + neq = self.ndiffAssertEqual + msg = Message() + # RFC 2045, $5.2 says invalid yields text/plain + msg['Content-Type'] = 'text' + eq(msg.get_content_maintype(), 'text') + eq(msg.get_content_subtype(), 'plain') + eq(msg.get_content_type(), 'text/plain') + # Clear the old value and try something /really/ invalid + del msg['content-type'] + msg['Content-Type'] = 'foo' + eq(msg.get_content_maintype(), 'text') + eq(msg.get_content_subtype(), 'plain') + eq(msg.get_content_type(), 'text/plain') + # Still, make sure that the message is idempotently generated + s = StringIO() + g = Generator(s) + g.flatten(msg) + neq(s.getvalue(), 'Content-Type: foo\n\n') + + def test_no_start_boundary(self): + eq = self.ndiffAssertEqual + msg = self._msgobj('msg_31.txt') + eq(msg.get_payload(), """\ +--BOUNDARY +Content-Type: text/plain + +message 1 + +--BOUNDARY +Content-Type: text/plain + +message 2 + +--BOUNDARY-- +""") + + def test_no_separating_blank_line(self): + eq = self.ndiffAssertEqual + msg = self._msgobj('msg_35.txt') + eq(msg.as_string(), """\ +From: aperson@dom.ain +To: bperson@dom.ain +Subject: here's something interesting + +counter to RFC 5322, there's no separating newline here +""") + + # test_defect_handling + def test_lying_multipart(self): + msg = self._msgobj('msg_41.txt') + self.assertHasAttr(msg, 'defects') + self.assertEqual(len(msg.defects), 2) + self.assertIsInstance(msg.defects[0], + errors.NoBoundaryInMultipartDefect) + self.assertIsInstance(msg.defects[1], + errors.MultipartInvariantViolationDefect) + + # test_defect_handling + def test_missing_start_boundary(self): + outer = self._msgobj('msg_42.txt') + # The message structure is: + # + # multipart/mixed + # text/plain + # message/rfc822 + # multipart/mixed [*] + # + # [*] This message is missing its start boundary + bad = outer.get_payload(1).get_payload(0) + self.assertEqual(len(bad.defects), 1) + self.assertIsInstance(bad.defects[0], + errors.StartBoundaryNotFoundDefect) + + # test_defect_handling + def test_first_line_is_continuation_header(self): + eq = self.assertEqual + m = ' Line 1\nSubject: test\n\nbody' + msg = email.message_from_string(m) + eq(msg.keys(), ['Subject']) + eq(msg.get_payload(), 'body') + eq(len(msg.defects), 1) + self.assertDefectsEqual(msg.defects, + [errors.FirstHeaderLineIsContinuationDefect]) + eq(msg.defects[0].line, ' Line 1\n') + + # test_defect_handling + def test_missing_header_body_separator(self): + # Our heuristic if we see a line that doesn't look like a header (no + # leading whitespace but no ':') is to assume that the blank line that + # separates the header from the body is missing, and to stop parsing + # headers and start parsing the body. + msg = self._str_msg('Subject: test\nnot a header\nTo: abc\n\nb\n') + self.assertEqual(msg.keys(), ['Subject']) + self.assertEqual(msg.get_payload(), 'not a header\nTo: abc\n\nb\n') + self.assertDefectsEqual(msg.defects, + [errors.MissingHeaderBodySeparatorDefect]) + + def test_string_payload_with_extra_space_after_cte(self): + # https://github.com/python/cpython/issues/98188 + cte = "base64 " + msg = email.message_from_string(textwrap.dedent(f"""\ + Content-Transfer-Encoding: {cte} + + SGVsbG8uIFRlc3Rpbmc= + """), policy=email.policy.default) + self.assertEqual(msg.get_payload(decode=True), b"Hello. Testing") + self.assertDefectsEqual(msg['content-transfer-encoding'].defects, []) + + def test_string_payload_with_extra_text_after_cte(self): + msg = email.message_from_string(textwrap.dedent("""\ + Content-Transfer-Encoding: base64 some text + + SGVsbG8uIFRlc3Rpbmc= + """), policy=email.policy.default) + self.assertEqual(msg.get_payload(decode=True), b"Hello. Testing") + cte = msg['content-transfer-encoding'] + self.assertDefectsEqual(cte.defects, [email.errors.InvalidHeaderDefect]) + + def test_string_payload_with_extra_space_after_cte_compat32(self): + cte = "base64 " + msg = email.message_from_string(textwrap.dedent(f"""\ + Content-Transfer-Encoding: {cte} + + SGVsbG8uIFRlc3Rpbmc= + """), policy=email.policy.compat32) + pasted_cte = msg['content-transfer-encoding'] + self.assertEqual(pasted_cte, cte) + self.assertEqual(msg.get_payload(decode=True), b"Hello. Testing") + self.assertDefectsEqual(msg.defects, []) + + + +# Test RFC 2047 header encoding and decoding +class TestRFC2047(TestEmailBase): + def test_rfc2047_multiline(self): + eq = self.assertEqual + s = """Re: =?mac-iceland?q?r=8Aksm=9Arg=8Cs?= baz + foo bar =?mac-iceland?q?r=8Aksm=9Arg=8Cs?=""" + dh = decode_header(s) + eq(dh, [ + (b'Re: ', None), + (b'r\x8aksm\x9arg\x8cs', 'mac-iceland'), + (b' baz foo bar ', None), + (b'r\x8aksm\x9arg\x8cs', 'mac-iceland')]) + header = make_header(dh) + eq(str(header), + 'Re: r\xe4ksm\xf6rg\xe5s baz foo bar r\xe4ksm\xf6rg\xe5s') + self.ndiffAssertEqual(header.encode(maxlinelen=76), """\ +Re: =?mac-iceland?q?r=8Aksm=9Arg=8Cs?= baz foo bar =?mac-iceland?q?r=8Aksm?= + =?mac-iceland?q?=9Arg=8Cs?=""") + + def test_whitespace_keeper_unicode(self): + eq = self.assertEqual + s = '=?ISO-8859-1?Q?Andr=E9?= Pirard ' + dh = decode_header(s) + eq(dh, [(b'Andr\xe9', 'iso-8859-1'), + (b' Pirard ', None)]) + header = str(make_header(dh)) + eq(header, 'Andr\xe9 Pirard ') + + def test_whitespace_keeper_unicode_2(self): + eq = self.assertEqual + s = 'The =?iso-8859-1?b?cXVpY2sgYnJvd24gZm94?= jumped over the =?iso-8859-1?b?bGF6eSBkb2c=?=' + dh = decode_header(s) + eq(dh, [(b'The ', None), (b'quick brown fox', 'iso-8859-1'), + (b' jumped over the ', None), (b'lazy dog', 'iso-8859-1')]) + hu = str(make_header(dh)) + eq(hu, 'The quick brown fox jumped over the lazy dog') + + def test_rfc2047_missing_whitespace(self): + s = 'Sm=?ISO-8859-1?B?9g==?=rg=?ISO-8859-1?B?5Q==?=sbord' + dh = decode_header(s) + self.assertEqual(dh, [(b'Sm', None), (b'\xf6', 'iso-8859-1'), + (b'rg', None), (b'\xe5', 'iso-8859-1'), + (b'sbord', None)]) + + def test_rfc2047_with_whitespace(self): + s = 'Sm =?ISO-8859-1?B?9g==?= rg =?ISO-8859-1?B?5Q==?= sbord' + dh = decode_header(s) + self.assertEqual(dh, [(b'Sm ', None), (b'\xf6', 'iso-8859-1'), + (b' rg ', None), (b'\xe5', 'iso-8859-1'), + (b' sbord', None)]) + + def test_rfc2047_B_bad_padding(self): + s = '=?iso-8859-1?B?%s?=' + data = [ # only test complete bytes + ('dm==', b'v'), ('dm=', b'v'), ('dm', b'v'), + ('dmk=', b'vi'), ('dmk', b'vi') + ] + for q, a in data: + dh = decode_header(s % q) + self.assertEqual(dh, [(a, 'iso-8859-1')]) + + def test_rfc2047_Q_invalid_digits(self): + # issue 10004. + s = '=?iso-8859-1?Q?andr=e9=zz?=' + self.assertEqual(decode_header(s), + [(b'andr\xe9=zz', 'iso-8859-1')]) + + def test_rfc2047_rfc2047_1(self): + # 1st testcase at end of RFC 2047 + s = '(=?ISO-8859-1?Q?a?=)' + self.assertEqual(decode_header(s), + [(b'(', None), (b'a', 'iso-8859-1'), (b')', None)]) + + def test_rfc2047_rfc2047_2(self): + # 2nd testcase at end of RFC 2047 + s = '(=?ISO-8859-1?Q?a?= b)' + self.assertEqual(decode_header(s), + [(b'(', None), (b'a', 'iso-8859-1'), (b' b)', None)]) + + def test_rfc2047_rfc2047_3(self): + # 3rd testcase at end of RFC 2047 + s = '(=?ISO-8859-1?Q?a?= =?ISO-8859-1?Q?b?=)' + self.assertEqual(decode_header(s), + [(b'(', None), (b'ab', 'iso-8859-1'), (b')', None)]) + + def test_rfc2047_rfc2047_4(self): + # 4th testcase at end of RFC 2047 + s = '(=?ISO-8859-1?Q?a?= =?ISO-8859-1?Q?b?=)' + self.assertEqual(decode_header(s), + [(b'(', None), (b'ab', 'iso-8859-1'), (b')', None)]) + + def test_rfc2047_rfc2047_5a(self): + # 5th testcase at end of RFC 2047 newline is \r\n + s = '(=?ISO-8859-1?Q?a?=\r\n =?ISO-8859-1?Q?b?=)' + self.assertEqual(decode_header(s), + [(b'(', None), (b'ab', 'iso-8859-1'), (b')', None)]) + + def test_rfc2047_rfc2047_5b(self): + # 5th testcase at end of RFC 2047 newline is \n + s = '(=?ISO-8859-1?Q?a?=\n =?ISO-8859-1?Q?b?=)' + self.assertEqual(decode_header(s), + [(b'(', None), (b'ab', 'iso-8859-1'), (b')', None)]) + + def test_rfc2047_rfc2047_6(self): + # 6th testcase at end of RFC 2047 + s = '(=?ISO-8859-1?Q?a_b?=)' + self.assertEqual(decode_header(s), + [(b'(', None), (b'a b', 'iso-8859-1'), (b')', None)]) + + def test_rfc2047_rfc2047_7(self): + # 7th testcase at end of RFC 2047 + s = '(=?ISO-8859-1?Q?a?= =?ISO-8859-2?Q?_b?=)' + self.assertEqual(decode_header(s), + [(b'(', None), (b'a', 'iso-8859-1'), (b' b', 'iso-8859-2'), + (b')', None)]) + self.assertEqual(make_header(decode_header(s)).encode(), s.lower()) + self.assertEqual(str(make_header(decode_header(s))), '(a b)') + + def test_multiline_header(self): + s = '=?windows-1252?q?=22M=FCller_T=22?=\r\n ' + self.assertEqual(decode_header(s), + [(b'"M\xfcller T"', 'windows-1252'), + (b'', None)]) + self.assertEqual(make_header(decode_header(s)).encode(), + ''.join(s.splitlines())) + self.assertEqual(str(make_header(decode_header(s))), + '"Müller T" ') + + def test_unencoded_ascii(self): + # bpo-22833/gh-67022: returns [(str, None)] rather than [(bytes, None)] + s = 'header without encoded words' + self.assertEqual(decode_header(s), + [('header without encoded words', None)]) + + def test_unencoded_utf8(self): + # bpo-22833/gh-67022: returns [(str, None)] rather than [(bytes, None)] + s = 'header with unexpected non ASCII caract\xe8res' + self.assertEqual(decode_header(s), + [('header with unexpected non ASCII caract\xe8res', None)]) + + +# Test the MIMEMessage class +class TestMIMEMessage(TestEmailBase): + def setUp(self): + with openfile('msg_11.txt', encoding="utf-8") as fp: + self._text = fp.read() + + def test_type_error(self): + self.assertRaises(TypeError, MIMEMessage, 'a plain string') + + def test_valid_argument(self): + eq = self.assertEqual + subject = 'A sub-message' + m = Message() + m['Subject'] = subject + r = MIMEMessage(m) + eq(r.get_content_type(), 'message/rfc822') + payload = r.get_payload() + self.assertIsInstance(payload, list) + eq(len(payload), 1) + subpart = payload[0] + self.assertIs(subpart, m) + eq(subpart['subject'], subject) + + def test_bad_multipart(self): + msg1 = Message() + msg1['Subject'] = 'subpart 1' + msg2 = Message() + msg2['Subject'] = 'subpart 2' + r = MIMEMessage(msg1) + self.assertRaises(errors.MultipartConversionError, r.attach, msg2) + + def test_generate(self): + # First craft the message to be encapsulated + m = Message() + m['Subject'] = 'An enclosed message' + m.set_payload('Here is the body of the message.\n') + r = MIMEMessage(m) + r['Subject'] = 'The enclosing message' + s = StringIO() + g = Generator(s) + g.flatten(r) + self.assertEqual(s.getvalue(), """\ +Content-Type: message/rfc822 +MIME-Version: 1.0 +Subject: The enclosing message + +Subject: An enclosed message + +Here is the body of the message. +""") + + def test_parse_message_rfc822(self): + eq = self.assertEqual + msg = self._msgobj('msg_11.txt') + eq(msg.get_content_type(), 'message/rfc822') + payload = msg.get_payload() + self.assertIsInstance(payload, list) + eq(len(payload), 1) + submsg = payload[0] + self.assertIsInstance(submsg, Message) + eq(submsg['subject'], 'An enclosed message') + eq(submsg.get_payload(), 'Here is the body of the message.\n') + + def test_dsn(self): + eq = self.assertEqual + # msg 16 is a Delivery Status Notification, see RFC 1894 + msg = self._msgobj('msg_16.txt') + eq(msg.get_content_type(), 'multipart/report') + self.assertTrue(msg.is_multipart()) + eq(len(msg.get_payload()), 3) + # Subpart 1 is a text/plain, human readable section + subpart = msg.get_payload(0) + eq(subpart.get_content_type(), 'text/plain') + eq(subpart.get_payload(), """\ +This report relates to a message you sent with the following header fields: + + Message-id: <002001c144a6$8752e060$56104586@oxy.edu> + Date: Sun, 23 Sep 2001 20:10:55 -0700 + From: "Ian T. Henry" + To: SoCal Raves + Subject: [scr] yeah for Ians!! + +Your message cannot be delivered to the following recipients: + + Recipient address: jangel1@cougar.noc.ucla.edu + Reason: recipient reached disk quota + +""") + # Subpart 2 contains the machine parsable DSN information. It + # consists of two blocks of headers, represented by two nested Message + # objects. + subpart = msg.get_payload(1) + eq(subpart.get_content_type(), 'message/delivery-status') + eq(len(subpart.get_payload()), 2) + # message/delivery-status should treat each block as a bunch of + # headers, i.e. a bunch of Message objects. + dsn1 = subpart.get_payload(0) + self.assertIsInstance(dsn1, Message) + eq(dsn1['original-envelope-id'], '0GK500B4HD0888@cougar.noc.ucla.edu') + eq(dsn1.get_param('dns', header='reporting-mta'), '') + # Try a missing one + eq(dsn1.get_param('nsd', header='reporting-mta'), None) + dsn2 = subpart.get_payload(1) + self.assertIsInstance(dsn2, Message) + eq(dsn2['action'], 'failed') + eq(dsn2.get_params(header='original-recipient'), + [('rfc822', ''), ('jangel1@cougar.noc.ucla.edu', '')]) + eq(dsn2.get_param('rfc822', header='final-recipient'), '') + # Subpart 3 is the original message + subpart = msg.get_payload(2) + eq(subpart.get_content_type(), 'message/rfc822') + payload = subpart.get_payload() + self.assertIsInstance(payload, list) + eq(len(payload), 1) + subsubpart = payload[0] + self.assertIsInstance(subsubpart, Message) + eq(subsubpart.get_content_type(), 'text/plain') + eq(subsubpart['message-id'], + '<002001c144a6$8752e060$56104586@oxy.edu>') + + def test_epilogue(self): + eq = self.ndiffAssertEqual + with openfile('msg_21.txt', encoding="utf-8") as fp: + text = fp.read() + msg = Message() + msg['From'] = 'aperson@dom.ain' + msg['To'] = 'bperson@dom.ain' + msg['Subject'] = 'Test' + msg.preamble = 'MIME message' + msg.epilogue = 'End of MIME message\n' + msg1 = MIMEText('One') + msg2 = MIMEText('Two') + msg.add_header('Content-Type', 'multipart/mixed', boundary='BOUNDARY') + msg.attach(msg1) + msg.attach(msg2) + sfp = StringIO() + g = Generator(sfp) + g.flatten(msg) + eq(sfp.getvalue(), text) + + def test_no_nl_preamble(self): + eq = self.ndiffAssertEqual + msg = Message() + msg['From'] = 'aperson@dom.ain' + msg['To'] = 'bperson@dom.ain' + msg['Subject'] = 'Test' + msg.preamble = 'MIME message' + msg.epilogue = '' + msg1 = MIMEText('One') + msg2 = MIMEText('Two') + msg.add_header('Content-Type', 'multipart/mixed', boundary='BOUNDARY') + msg.attach(msg1) + msg.attach(msg2) + eq(msg.as_string(), """\ +From: aperson@dom.ain +To: bperson@dom.ain +Subject: Test +Content-Type: multipart/mixed; boundary="BOUNDARY" + +MIME message +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit + +One +--BOUNDARY +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit + +Two +--BOUNDARY-- +""") + + def test_default_type(self): + eq = self.assertEqual + with openfile('msg_30.txt', encoding="utf-8") as fp: + msg = email.message_from_file(fp) + container1 = msg.get_payload(0) + eq(container1.get_default_type(), 'message/rfc822') + eq(container1.get_content_type(), 'message/rfc822') + container2 = msg.get_payload(1) + eq(container2.get_default_type(), 'message/rfc822') + eq(container2.get_content_type(), 'message/rfc822') + container1a = container1.get_payload(0) + eq(container1a.get_default_type(), 'text/plain') + eq(container1a.get_content_type(), 'text/plain') + container2a = container2.get_payload(0) + eq(container2a.get_default_type(), 'text/plain') + eq(container2a.get_content_type(), 'text/plain') + + def test_default_type_with_explicit_container_type(self): + eq = self.assertEqual + with openfile('msg_28.txt', encoding="utf-8") as fp: + msg = email.message_from_file(fp) + container1 = msg.get_payload(0) + eq(container1.get_default_type(), 'message/rfc822') + eq(container1.get_content_type(), 'message/rfc822') + container2 = msg.get_payload(1) + eq(container2.get_default_type(), 'message/rfc822') + eq(container2.get_content_type(), 'message/rfc822') + container1a = container1.get_payload(0) + eq(container1a.get_default_type(), 'text/plain') + eq(container1a.get_content_type(), 'text/plain') + container2a = container2.get_payload(0) + eq(container2a.get_default_type(), 'text/plain') + eq(container2a.get_content_type(), 'text/plain') + + def test_default_type_non_parsed(self): + eq = self.assertEqual + neq = self.ndiffAssertEqual + # Set up container + container = MIMEMultipart('digest', 'BOUNDARY') + container.epilogue = '' + # Set up subparts + subpart1a = MIMEText('message 1\n') + subpart2a = MIMEText('message 2\n') + subpart1 = MIMEMessage(subpart1a) + subpart2 = MIMEMessage(subpart2a) + container.attach(subpart1) + container.attach(subpart2) + eq(subpart1.get_content_type(), 'message/rfc822') + eq(subpart1.get_default_type(), 'message/rfc822') + eq(subpart2.get_content_type(), 'message/rfc822') + eq(subpart2.get_default_type(), 'message/rfc822') + neq(container.as_string(0), '''\ +Content-Type: multipart/digest; boundary="BOUNDARY" +MIME-Version: 1.0 + +--BOUNDARY +Content-Type: message/rfc822 +MIME-Version: 1.0 + +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit + +message 1 + +--BOUNDARY +Content-Type: message/rfc822 +MIME-Version: 1.0 + +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit + +message 2 + +--BOUNDARY-- +''') + del subpart1['content-type'] + del subpart1['mime-version'] + del subpart2['content-type'] + del subpart2['mime-version'] + eq(subpart1.get_content_type(), 'message/rfc822') + eq(subpart1.get_default_type(), 'message/rfc822') + eq(subpart2.get_content_type(), 'message/rfc822') + eq(subpart2.get_default_type(), 'message/rfc822') + neq(container.as_string(0), '''\ +Content-Type: multipart/digest; boundary="BOUNDARY" +MIME-Version: 1.0 + +--BOUNDARY + +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit + +message 1 + +--BOUNDARY + +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit + +message 2 + +--BOUNDARY-- +''') + + def test_mime_attachments_in_constructor(self): + eq = self.assertEqual + text1 = MIMEText('') + text2 = MIMEText('') + msg = MIMEMultipart(_subparts=(text1, text2)) + eq(len(msg.get_payload()), 2) + eq(msg.get_payload(0), text1) + eq(msg.get_payload(1), text2) + + def test_default_multipart_constructor(self): + msg = MIMEMultipart() + self.assertTrue(msg.is_multipart()) + + def test_multipart_default_policy(self): + msg = MIMEMultipart() + msg['To'] = 'a@b.com' + msg['To'] = 'c@d.com' + self.assertEqual(msg.get_all('to'), ['a@b.com', 'c@d.com']) + + def test_multipart_custom_policy(self): + msg = MIMEMultipart(policy=email.policy.default) + msg['To'] = 'a@b.com' + with self.assertRaises(ValueError) as cm: + msg['To'] = 'c@d.com' + self.assertEqual(str(cm.exception), + 'There may be at most 1 To headers in a message') + + +# Test the NonMultipart class +class TestNonMultipart(TestEmailBase): + def test_nonmultipart_is_not_multipart(self): + msg = MIMENonMultipart('text', 'plain') + self.assertFalse(msg.is_multipart()) + + def test_attach_raises_exception(self): + msg = Message() + msg['Subject'] = 'subpart 1' + r = MIMENonMultipart('text', 'plain') + self.assertRaises(errors.MultipartConversionError, r.attach, msg) + + +# A general test of parser->model->generator idempotency. IOW, read a message +# in, parse it into a message object tree, then without touching the tree, +# regenerate the plain text. The original text and the transformed text +# should be identical. Note: that we ignore the Unix-From since that may +# contain a changed date. +class TestIdempotent(TestEmailBase): + + linesep = '\n' + + def _msgobj(self, filename): + with openfile(filename, encoding="utf-8") as fp: + data = fp.read() + msg = email.message_from_string(data) + return msg, data + + def _idempotent(self, msg, text, unixfrom=False): + eq = self.ndiffAssertEqual + s = StringIO() + g = Generator(s, maxheaderlen=0) + g.flatten(msg, unixfrom=unixfrom) + eq(text, s.getvalue()) + + def test_parse_text_message(self): + eq = self.assertEqual + msg, text = self._msgobj('msg_01.txt') + eq(msg.get_content_type(), 'text/plain') + eq(msg.get_content_maintype(), 'text') + eq(msg.get_content_subtype(), 'plain') + eq(msg.get_params()[1], ('charset', 'us-ascii')) + eq(msg.get_param('charset'), 'us-ascii') + eq(msg.preamble, None) + eq(msg.epilogue, None) + self._idempotent(msg, text) + + def test_parse_untyped_message(self): + eq = self.assertEqual + msg, text = self._msgobj('msg_03.txt') + eq(msg.get_content_type(), 'text/plain') + eq(msg.get_params(), None) + eq(msg.get_param('charset'), None) + self._idempotent(msg, text) + + def test_simple_multipart(self): + msg, text = self._msgobj('msg_04.txt') + self._idempotent(msg, text) + + def test_MIME_digest(self): + msg, text = self._msgobj('msg_02.txt') + self._idempotent(msg, text) + + def test_long_header(self): + msg, text = self._msgobj('msg_27.txt') + self._idempotent(msg, text) + + def test_MIME_digest_with_part_headers(self): + msg, text = self._msgobj('msg_28.txt') + self._idempotent(msg, text) + + def test_mixed_with_image(self): + msg, text = self._msgobj('msg_06.txt') + self._idempotent(msg, text) + + def test_multipart_report(self): + msg, text = self._msgobj('msg_05.txt') + self._idempotent(msg, text) + + def test_dsn(self): + msg, text = self._msgobj('msg_16.txt') + self._idempotent(msg, text) + + def test_preamble_epilogue(self): + msg, text = self._msgobj('msg_21.txt') + self._idempotent(msg, text) + + def test_multipart_one_part(self): + msg, text = self._msgobj('msg_23.txt') + self._idempotent(msg, text) + + def test_multipart_no_parts(self): + msg, text = self._msgobj('msg_24.txt') + self._idempotent(msg, text) + + def test_no_start_boundary(self): + msg, text = self._msgobj('msg_31.txt') + self._idempotent(msg, text) + + def test_rfc2231_charset(self): + msg, text = self._msgobj('msg_32.txt') + self._idempotent(msg, text) + + def test_more_rfc2231_parameters(self): + msg, text = self._msgobj('msg_33.txt') + self._idempotent(msg, text) + + def test_text_plain_in_a_multipart_digest(self): + msg, text = self._msgobj('msg_34.txt') + self._idempotent(msg, text) + + def test_nested_multipart_mixeds(self): + msg, text = self._msgobj('msg_12a.txt') + self._idempotent(msg, text) + + def test_message_external_body_idempotent(self): + msg, text = self._msgobj('msg_36.txt') + self._idempotent(msg, text) + + def test_message_delivery_status(self): + msg, text = self._msgobj('msg_43.txt') + self._idempotent(msg, text, unixfrom=True) + + def test_message_signed_idempotent(self): + msg, text = self._msgobj('msg_45.txt') + self._idempotent(msg, text) + + def test_content_type(self): + eq = self.assertEqual + # Get a message object and reset the seek pointer for other tests + msg, text = self._msgobj('msg_05.txt') + eq(msg.get_content_type(), 'multipart/report') + # Test the Content-Type: parameters + params = {} + for pk, pv in msg.get_params(): + params[pk] = pv + eq(params['report-type'], 'delivery-status') + eq(params['boundary'], 'D1690A7AC1.996856090/mail.example.com') + eq(msg.preamble, 'This is a MIME-encapsulated message.' + self.linesep) + eq(msg.epilogue, self.linesep) + eq(len(msg.get_payload()), 3) + # Make sure the subparts are what we expect + msg1 = msg.get_payload(0) + eq(msg1.get_content_type(), 'text/plain') + eq(msg1.get_payload(), 'Yadda yadda yadda' + self.linesep) + msg2 = msg.get_payload(1) + eq(msg2.get_content_type(), 'text/plain') + eq(msg2.get_payload(), 'Yadda yadda yadda' + self.linesep) + msg3 = msg.get_payload(2) + eq(msg3.get_content_type(), 'message/rfc822') + self.assertIsInstance(msg3, Message) + payload = msg3.get_payload() + self.assertIsInstance(payload, list) + eq(len(payload), 1) + msg4 = payload[0] + self.assertIsInstance(msg4, Message) + eq(msg4.get_payload(), 'Yadda yadda yadda' + self.linesep) + + def test_parser(self): + eq = self.assertEqual + msg, text = self._msgobj('msg_06.txt') + # Check some of the outer headers + eq(msg.get_content_type(), 'message/rfc822') + # Make sure the payload is a list of exactly one sub-Message, and that + # that submessage has a type of text/plain + payload = msg.get_payload() + self.assertIsInstance(payload, list) + eq(len(payload), 1) + msg1 = payload[0] + self.assertIsInstance(msg1, Message) + eq(msg1.get_content_type(), 'text/plain') + self.assertIsInstance(msg1.get_payload(), str) + eq(msg1.get_payload(), self.linesep) + + + +# Test various other bits of the package's functionality +class TestMiscellaneous(TestEmailBase): + def test_message_from_string(self): + with openfile('msg_01.txt', encoding="utf-8") as fp: + text = fp.read() + msg = email.message_from_string(text) + s = StringIO() + # Don't wrap/continue long headers since we're trying to test + # idempotency. + g = Generator(s, maxheaderlen=0) + g.flatten(msg) + self.assertEqual(text, s.getvalue()) + + def test_message_from_file(self): + with openfile('msg_01.txt', encoding="utf-8") as fp: + text = fp.read() + fp.seek(0) + msg = email.message_from_file(fp) + s = StringIO() + # Don't wrap/continue long headers since we're trying to test + # idempotency. + g = Generator(s, maxheaderlen=0) + g.flatten(msg) + self.assertEqual(text, s.getvalue()) + + def test_message_from_string_with_class(self): + with openfile('msg_01.txt', encoding="utf-8") as fp: + text = fp.read() + + # Create a subclass + class MyMessage(Message): + pass + + msg = email.message_from_string(text, MyMessage) + self.assertIsInstance(msg, MyMessage) + # Try something more complicated + with openfile('msg_02.txt', encoding="utf-8") as fp: + text = fp.read() + msg = email.message_from_string(text, MyMessage) + for subpart in msg.walk(): + self.assertIsInstance(subpart, MyMessage) + + def test_message_from_file_with_class(self): + # Create a subclass + class MyMessage(Message): + pass + + with openfile('msg_01.txt', encoding="utf-8") as fp: + msg = email.message_from_file(fp, MyMessage) + self.assertIsInstance(msg, MyMessage) + # Try something more complicated + with openfile('msg_02.txt', encoding="utf-8") as fp: + msg = email.message_from_file(fp, MyMessage) + for subpart in msg.walk(): + self.assertIsInstance(subpart, MyMessage) + + def test_custom_message_does_not_require_arguments(self): + class MyMessage(Message): + def __init__(self): + super().__init__() + msg = self._str_msg("Subject: test\n\ntest", MyMessage) + self.assertIsInstance(msg, MyMessage) + + def test__all__(self): + module = __import__('email') + self.assertEqual(sorted(module.__all__), [ + 'base64mime', 'charset', 'encoders', 'errors', 'feedparser', + 'generator', 'header', 'iterators', 'message', + 'message_from_binary_file', 'message_from_bytes', + 'message_from_file', 'message_from_string', 'mime', 'parser', + 'quoprimime', 'utils', + ]) + + def test_formatdate(self): + now = time.time() + self.assertEqual(utils.parsedate(utils.formatdate(now))[:6], + time.gmtime(now)[:6]) + + def test_formatdate_localtime(self): + now = time.time() + self.assertEqual( + utils.parsedate(utils.formatdate(now, localtime=True))[:6], + time.localtime(now)[:6]) + + def test_formatdate_usegmt(self): + now = time.time() + self.assertEqual( + utils.formatdate(now, localtime=False), + time.strftime('%a, %d %b %Y %H:%M:%S -0000', time.gmtime(now))) + self.assertEqual( + utils.formatdate(now, localtime=False, usegmt=True), + time.strftime('%a, %d %b %Y %H:%M:%S GMT', time.gmtime(now))) + + # parsedate and parsedate_tz will become deprecated interfaces someday + def test_parsedate_returns_None_for_invalid_strings(self): + # See also test_parsedate_to_datetime_with_invalid_raises_valueerror + # in test_utils. + invalid_dates = [ + '', + ' ', + '0', + 'A Complete Waste of Time', + 'Wed, 3 Apr 2002 12.34.56.78+0800', + '17 June , 2022', + 'Friday, -Nov-82 16:14:55 EST', + 'Friday, Nov--82 16:14:55 EST', + 'Friday, 19-Nov- 16:14:55 EST', + ] + for dtstr in invalid_dates: + with self.subTest(dtstr=dtstr): + self.assertIsNone(utils.parsedate(dtstr)) + self.assertIsNone(utils.parsedate_tz(dtstr)) + # Not a part of the spec but, but this has historically worked: + self.assertIsNone(utils.parsedate(None)) + self.assertIsNone(utils.parsedate_tz(None)) + + def test_parsedate_compact(self): + self.assertEqual(utils.parsedate_tz('Wed, 3 Apr 2002 14:58:26 +0800'), + (2002, 4, 3, 14, 58, 26, 0, 1, -1, 28800)) + # The FWS after the comma is optional + self.assertEqual(utils.parsedate_tz('Wed,3 Apr 2002 14:58:26 +0800'), + (2002, 4, 3, 14, 58, 26, 0, 1, -1, 28800)) + # The comma is optional + self.assertEqual(utils.parsedate_tz('Wed 3 Apr 2002 14:58:26 +0800'), + (2002, 4, 3, 14, 58, 26, 0, 1, -1, 28800)) + + def test_parsedate_no_dayofweek(self): + eq = self.assertEqual + eq(utils.parsedate_tz('5 Feb 2003 13:47:26 -0800'), + (2003, 2, 5, 13, 47, 26, 0, 1, -1, -28800)) + eq(utils.parsedate_tz('February 5, 2003 13:47:26 -0800'), + (2003, 2, 5, 13, 47, 26, 0, 1, -1, -28800)) + + def test_parsedate_no_space_before_positive_offset(self): + self.assertEqual(utils.parsedate_tz('Wed, 3 Apr 2002 14:58:26+0800'), + (2002, 4, 3, 14, 58, 26, 0, 1, -1, 28800)) + + def test_parsedate_no_space_before_negative_offset(self): + # Issue 1155362: we already handled '+' for this case. + self.assertEqual(utils.parsedate_tz('Wed, 3 Apr 2002 14:58:26-0800'), + (2002, 4, 3, 14, 58, 26, 0, 1, -1, -28800)) + + def test_parsedate_accepts_time_with_dots(self): + eq = self.assertEqual + eq(utils.parsedate_tz('5 Feb 2003 13.47.26 -0800'), + (2003, 2, 5, 13, 47, 26, 0, 1, -1, -28800)) + eq(utils.parsedate_tz('5 Feb 2003 13.47 -0800'), + (2003, 2, 5, 13, 47, 0, 0, 1, -1, -28800)) + + def test_parsedate_rfc_850(self): + self.assertEqual(utils.parsedate_tz('Friday, 19-Nov-82 16:14:55 EST'), + (1982, 11, 19, 16, 14, 55, 0, 1, -1, -18000)) + + def test_parsedate_no_seconds(self): + self.assertEqual(utils.parsedate_tz('Wed, 3 Apr 2002 14:58 +0800'), + (2002, 4, 3, 14, 58, 0, 0, 1, -1, 28800)) + + def test_parsedate_dot_time_delimiter(self): + self.assertEqual(utils.parsedate_tz('Wed, 3 Apr 2002 14.58.26 +0800'), + (2002, 4, 3, 14, 58, 26, 0, 1, -1, 28800)) + self.assertEqual(utils.parsedate_tz('Wed, 3 Apr 2002 14.58 +0800'), + (2002, 4, 3, 14, 58, 0, 0, 1, -1, 28800)) + + def test_parsedate_acceptable_to_time_functions(self): + eq = self.assertEqual + timetup = utils.parsedate('5 Feb 2003 13:47:26 -0800') + t = int(time.mktime(timetup)) + eq(time.localtime(t)[:6], timetup[:6]) + eq(int(time.strftime('%Y', timetup)), 2003) + timetup = utils.parsedate_tz('5 Feb 2003 13:47:26 -0800') + t = int(time.mktime(timetup[:9])) + eq(time.localtime(t)[:6], timetup[:6]) + eq(int(time.strftime('%Y', timetup[:9])), 2003) + + def test_mktime_tz(self): + self.assertEqual(utils.mktime_tz((1970, 1, 1, 0, 0, 0, + -1, -1, -1, 0)), 0) + self.assertEqual(utils.mktime_tz((1970, 1, 1, 0, 0, 0, + -1, -1, -1, 1234)), -1234) + + def test_parsedate_y2k(self): + """Test for parsing a date with a two-digit year. + + Parsing a date with a two-digit year should return the correct + four-digit year. RFC 822 allows two-digit years, but RFC 5322 (which + obsoletes RFC 2822, which obsoletes RFC 822) requires four-digit years. + + """ + self.assertEqual(utils.parsedate_tz('25 Feb 03 13:47:26 -0800'), + utils.parsedate_tz('25 Feb 2003 13:47:26 -0800')) + self.assertEqual(utils.parsedate_tz('25 Feb 71 13:47:26 -0800'), + utils.parsedate_tz('25 Feb 1971 13:47:26 -0800')) + + def test_parseaddr_empty(self): + self.assertEqual(utils.parseaddr('<>'), ('', '')) + self.assertEqual(utils.formataddr(utils.parseaddr('<>')), '') + + def test_parseaddr_multiple_domains(self): + self.assertEqual( + utils.parseaddr('a@b@c'), + ('', '') + ) + self.assertEqual( + utils.parseaddr('a@b.c@c'), + ('', '') + ) + self.assertEqual( + utils.parseaddr('a@172.17.0.1@c'), + ('', '') + ) + + def test_noquote_dump(self): + self.assertEqual( + utils.formataddr(('A Silly Person', 'person@dom.ain')), + 'A Silly Person ') + + def test_escape_dump(self): + self.assertEqual( + utils.formataddr(('A (Very) Silly Person', 'person@dom.ain')), + r'"A (Very) Silly Person" ') + self.assertEqual( + utils.parseaddr(r'"A \(Very\) Silly Person" '), + ('A (Very) Silly Person', 'person@dom.ain')) + a = r'A \(Special\) Person' + b = 'person@dom.ain' + self.assertEqual(utils.parseaddr(utils.formataddr((a, b))), (a, b)) + + def test_escape_backslashes(self): + self.assertEqual( + utils.formataddr((r'Arthur \Backslash\ Foobar', 'person@dom.ain')), + r'"Arthur \\Backslash\\ Foobar" ') + a = r'Arthur \Backslash\ Foobar' + b = 'person@dom.ain' + self.assertEqual(utils.parseaddr(utils.formataddr((a, b))), (a, b)) + + def test_quotes_unicode_names(self): + # issue 1690608. email.utils.formataddr() should be RFC 2047 aware. + name = "H\u00e4ns W\u00fcrst" + addr = 'person@dom.ain' + utf8_base64 = "=?utf-8?b?SMOkbnMgV8O8cnN0?= " + latin1_quopri = "=?iso-8859-1?q?H=E4ns_W=FCrst?= " + self.assertEqual(utils.formataddr((name, addr)), utf8_base64) + self.assertEqual(utils.formataddr((name, addr), 'iso-8859-1'), + latin1_quopri) + + def test_accepts_any_charset_like_object(self): + # issue 1690608. email.utils.formataddr() should be RFC 2047 aware. + name = "H\u00e4ns W\u00fcrst" + addr = 'person@dom.ain' + utf8_base64 = "=?utf-8?b?SMOkbnMgV8O8cnN0?= " + foobar = "FOOBAR" + class CharsetMock: + def header_encode(self, string): + return foobar + mock = CharsetMock() + mock_expected = "%s <%s>" % (foobar, addr) + self.assertEqual(utils.formataddr((name, addr), mock), mock_expected) + self.assertEqual(utils.formataddr((name, addr), Charset('utf-8')), + utf8_base64) + + def test_invalid_charset_like_object_raises_error(self): + # issue 1690608. email.utils.formataddr() should be RFC 2047 aware. + name = "H\u00e4ns W\u00fcrst" + addr = 'person@dom.ain' + # An object without a header_encode method: + bad_charset = object() + self.assertRaises(AttributeError, utils.formataddr, (name, addr), + bad_charset) + + def test_unicode_address_raises_error(self): + # issue 1690608. email.utils.formataddr() should be RFC 2047 aware. + addr = 'pers\u00f6n@dom.in' + self.assertRaises(UnicodeError, utils.formataddr, (None, addr)) + self.assertRaises(UnicodeError, utils.formataddr, ("Name", addr)) + + def test_name_with_dot(self): + x = 'John X. Doe ' + y = '"John X. Doe" ' + a, b = ('John X. Doe', 'jxd@example.com') + self.assertEqual(utils.parseaddr(x), (a, b)) + self.assertEqual(utils.parseaddr(y), (a, b)) + # formataddr() quotes the name if there's a dot in it + self.assertEqual(utils.formataddr((a, b)), y) + + def test_parseaddr_preserves_quoted_pairs_in_addresses(self): + # issue 10005. Note that in the third test the second pair of + # backslashes is not actually a quoted pair because it is not inside a + # comment or quoted string: the address being parsed has a quoted + # string containing a quoted backslash, followed by 'example' and two + # backslashes, followed by another quoted string containing a space and + # the word 'example'. parseaddr copies those two backslashes + # literally. Per RFC 5322 this is not technically correct since a \ may + # not appear in an address outside of a quoted string. It is probably + # a sensible Postel interpretation, though. + eq = self.assertEqual + eq(utils.parseaddr('""example" example"@example.com'), + ('', '""example" example"@example.com')) + eq(utils.parseaddr('"\\"example\\" example"@example.com'), + ('', '"\\"example\\" example"@example.com')) + eq(utils.parseaddr('"\\\\"example\\\\" example"@example.com'), + ('', '"\\\\"example\\\\" example"@example.com')) + + def test_parseaddr_preserves_spaces_in_local_part(self): + # issue 9286. A normal RFC 5322 local part should not contain any + # folding white space, but legacy local parts can (they are a sequence + # of atoms, not dotatoms). On the other hand we strip whitespace from + # before the @ and around dots, on the assumption that the whitespace + # around the punctuation is a mistake in what would otherwise be + # an RFC 5322 local part. Leading whitespace is, usual, stripped as well. + self.assertEqual(('', "merwok wok@xample.com"), + utils.parseaddr("merwok wok@xample.com")) + self.assertEqual(('', "merwok wok@xample.com"), + utils.parseaddr("merwok wok@xample.com")) + self.assertEqual(('', "merwok wok@xample.com"), + utils.parseaddr(" merwok wok @xample.com")) + self.assertEqual(('', 'merwok"wok" wok@xample.com'), + utils.parseaddr('merwok"wok" wok@xample.com')) + self.assertEqual(('', 'merwok.wok.wok@xample.com'), + utils.parseaddr('merwok. wok . wok@xample.com')) + + def test_formataddr_does_not_quote_parens_in_quoted_string(self): + addr = ("'foo@example.com' (foo@example.com)", + 'foo@example.com') + addrstr = ('"\'foo@example.com\' ' + '(foo@example.com)" ') + self.assertEqual(utils.parseaddr(addrstr), addr) + self.assertEqual(utils.formataddr(addr), addrstr) + + + def test_multiline_from_comment(self): + x = """\ +Foo +\tBar """ + self.assertEqual(utils.parseaddr(x), ('Foo Bar', 'foo@example.com')) + + def test_quote_dump(self): + self.assertEqual( + utils.formataddr(('A Silly; Person', 'person@dom.ain')), + r'"A Silly; Person" ') + + def test_charset_richcomparisons(self): + eq = self.assertEqual + ne = self.assertNotEqual + cset1 = Charset() + cset2 = Charset() + eq(cset1, 'us-ascii') + eq(cset1, 'US-ASCII') + eq(cset1, 'Us-AsCiI') + eq('us-ascii', cset1) + eq('US-ASCII', cset1) + eq('Us-AsCiI', cset1) + ne(cset1, 'usascii') + ne(cset1, 'USASCII') + ne(cset1, 'UsAsCiI') + ne('usascii', cset1) + ne('USASCII', cset1) + ne('UsAsCiI', cset1) + eq(cset1, cset2) + eq(cset2, cset1) + + def test_getaddresses(self): + eq = self.assertEqual + eq(utils.getaddresses(['aperson@dom.ain (Al Person)', + 'Bud Person ']), + [('Al Person', 'aperson@dom.ain'), + ('Bud Person', 'bperson@dom.ain')]) + + def test_getaddresses_comma_in_name(self): + """GH-106669 regression test.""" + self.assertEqual( + utils.getaddresses( + [ + '"Bud, Person" ', + 'aperson@dom.ain (Al Person)', + '"Mariusz Felisiak" ', + ] + ), + [ + ('Bud, Person', 'bperson@dom.ain'), + ('Al Person', 'aperson@dom.ain'), + ('Mariusz Felisiak', 'to@example.com'), + ], + ) + + def test_parsing_errors(self): + """Test for parsing errors from CVE-2023-27043 and CVE-2019-16056""" + alice = 'alice@example.org' + bob = 'bob@example.com' + empty = ('', '') + + # Test utils.getaddresses() and utils.parseaddr() on malformed email + # addresses: default behavior (strict=True) rejects malformed address, + # and strict=False which tolerates malformed address. + for invalid_separator, expected_non_strict in ( + ('(', [(f'<{bob}>', alice)]), + (')', [('', alice), empty, ('', bob)]), + ('<', [('', alice), empty, ('', bob), empty]), + ('>', [('', alice), empty, ('', bob)]), + ('[', [('', f'{alice}[<{bob}>]')]), + (']', [('', alice), empty, ('', bob)]), + ('@', [empty, empty, ('', bob)]), + (';', [('', alice), empty, ('', bob)]), + (':', [('', alice), ('', bob)]), + ('.', [('', alice + '.'), ('', bob)]), + ('"', [('', alice), ('', f'<{bob}>')]), + ): + address = f'{alice}{invalid_separator}<{bob}>' + with self.subTest(address=address): + self.assertEqual(utils.getaddresses([address]), + [empty]) + self.assertEqual(utils.getaddresses([address], strict=False), + expected_non_strict) + + self.assertEqual(utils.parseaddr([address]), + empty) + self.assertEqual(utils.parseaddr([address], strict=False), + ('', address)) + + # Comma (',') is treated differently depending on strict parameter. + # Comma without quotes. + address = f'{alice},<{bob}>' + self.assertEqual(utils.getaddresses([address]), + [('', alice), ('', bob)]) + self.assertEqual(utils.getaddresses([address], strict=False), + [('', alice), ('', bob)]) + self.assertEqual(utils.parseaddr([address]), + empty) + self.assertEqual(utils.parseaddr([address], strict=False), + ('', address)) + + # Real name between quotes containing comma. + address = '"Alice, alice@example.org" ' + expected_strict = ('Alice, alice@example.org', 'bob@example.com') + self.assertEqual(utils.getaddresses([address]), [expected_strict]) + self.assertEqual(utils.getaddresses([address], strict=False), [expected_strict]) + self.assertEqual(utils.parseaddr([address]), expected_strict) + self.assertEqual(utils.parseaddr([address], strict=False), + ('', address)) + + # Valid parenthesis in comments. + address = 'alice@example.org (Alice)' + expected_strict = ('Alice', 'alice@example.org') + self.assertEqual(utils.getaddresses([address]), [expected_strict]) + self.assertEqual(utils.getaddresses([address], strict=False), [expected_strict]) + self.assertEqual(utils.parseaddr([address]), expected_strict) + self.assertEqual(utils.parseaddr([address], strict=False), + ('', address)) + + # Invalid parenthesis in comments. + address = 'alice@example.org )Alice(' + self.assertEqual(utils.getaddresses([address]), [empty]) + self.assertEqual(utils.getaddresses([address], strict=False), + [('', 'alice@example.org'), ('', ''), ('', 'Alice')]) + self.assertEqual(utils.parseaddr([address]), empty) + self.assertEqual(utils.parseaddr([address], strict=False), + ('', address)) + + # Two addresses with quotes separated by comma. + address = '"Jane Doe" , "John Doe" ' + self.assertEqual(utils.getaddresses([address]), + [('Jane Doe', 'jane@example.net'), + ('John Doe', 'john@example.net')]) + self.assertEqual(utils.getaddresses([address], strict=False), + [('Jane Doe', 'jane@example.net'), + ('John Doe', 'john@example.net')]) + self.assertEqual(utils.parseaddr([address]), empty) + self.assertEqual(utils.parseaddr([address], strict=False), + ('', address)) + + # Test email.utils.supports_strict_parsing attribute + self.assertEqual(email.utils.supports_strict_parsing, True) + + def test_getaddresses_nasty(self): + for addresses, expected in ( + (['"Sürname, Firstname" '], + [('Sürname, Firstname', 'to@example.com')]), + + (['foo: ;'], + [('', '')]), + + (['foo: ;', '"Jason R. Mastaler" '], + [('', ''), ('Jason R. Mastaler', 'jason@dom.ain')]), + + ([r'Pete(A nice \) chap) '], + [('Pete (A nice ) chap his account his host)', 'pete@silly.test')]), + + (['(Empty list)(start)Undisclosed recipients :(nobody(I know))'], + [('', '')]), + + (['Mary <@machine.tld:mary@example.net>, , jdoe@test . example'], + [('Mary', 'mary@example.net'), ('', ''), ('', 'jdoe@test.example')]), + + (['John Doe '], + [('John Doe (comment)', 'jdoe@machine.example')]), + + (['"Mary Smith: Personal Account" '], + [('Mary Smith: Personal Account', 'smith@home.example')]), + + (['Undisclosed recipients:;'], + [('', '')]), + + ([r', "Giant; \"Big\" Box" '], + [('', 'boss@nil.test'), ('Giant; "Big" Box', 'bob@example.net')]), + ): + with self.subTest(addresses=addresses): + self.assertEqual(utils.getaddresses(addresses), + expected) + self.assertEqual(utils.getaddresses(addresses, strict=False), + expected) + + addresses = ['[]*-- =~$'] + self.assertEqual(utils.getaddresses(addresses), + [('', '')]) + self.assertEqual(utils.getaddresses(addresses, strict=False), + [('', ''), ('', ''), ('', '*--')]) + + def test_getaddresses_embedded_comment(self): + """Test proper handling of a nested comment""" + eq = self.assertEqual + addrs = utils.getaddresses(['User ((nested comment)) ']) + eq(addrs[0][1], 'foo@bar.com') + + def test_getaddresses_header_obj(self): + """Test the handling of a Header object.""" + addrs = utils.getaddresses([Header('Al Person ')]) + self.assertEqual(addrs[0][1], 'aperson@dom.ain') + + @threading_helper.requires_working_threading() + @support.requires_resource('cpu') + def test_make_msgid_collisions(self): + # Test make_msgid uniqueness, even with multiple threads + class MsgidsThread(Thread): + def run(self): + # generate msgids for 3 seconds + self.msgids = [] + append = self.msgids.append + make_msgid = utils.make_msgid + clock = time.monotonic + tfin = clock() + 3.0 + while clock() < tfin: + append(make_msgid(domain='testdomain-string')) + + threads = [MsgidsThread() for i in range(5)] + with threading_helper.start_threads(threads): + pass + all_ids = sum([t.msgids for t in threads], []) + self.assertEqual(len(set(all_ids)), len(all_ids)) + + def test_utils_quote_unquote(self): + eq = self.assertEqual + msg = Message() + msg.add_header('content-disposition', 'attachment', + filename='foo\\wacky"name') + eq(msg.get_filename(), 'foo\\wacky"name') + + def test_get_body_encoding_with_bogus_charset(self): + charset = Charset('not a charset') + self.assertEqual(charset.get_body_encoding(), 'base64') + + def test_get_body_encoding_with_uppercase_charset(self): + eq = self.assertEqual + msg = Message() + msg['Content-Type'] = 'text/plain; charset=UTF-8' + eq(msg['content-type'], 'text/plain; charset=UTF-8') + charsets = msg.get_charsets() + eq(len(charsets), 1) + eq(charsets[0], 'utf-8') + charset = Charset(charsets[0]) + eq(charset.get_body_encoding(), 'base64') + msg.set_payload(b'hello world', charset=charset) + eq(msg.get_payload(), 'aGVsbG8gd29ybGQ=\n') + eq(msg.get_payload(decode=True), b'hello world') + eq(msg['content-transfer-encoding'], 'base64') + # Try another one + msg = Message() + msg['Content-Type'] = 'text/plain; charset="US-ASCII"' + charsets = msg.get_charsets() + eq(len(charsets), 1) + eq(charsets[0], 'us-ascii') + charset = Charset(charsets[0]) + eq(charset.get_body_encoding(), encoders.encode_7or8bit) + msg.set_payload('hello world', charset=charset) + eq(msg.get_payload(), 'hello world') + eq(msg['content-transfer-encoding'], '7bit') + + def test_charsets_case_insensitive(self): + lc = Charset('us-ascii') + uc = Charset('US-ASCII') + self.assertEqual(lc.get_body_encoding(), uc.get_body_encoding()) + + def test_partial_falls_inside_message_delivery_status(self): + eq = self.ndiffAssertEqual + # The Parser interface provides chunks of data to FeedParser in 8192 + # byte gulps. SF bug #1076485 found one of those chunks inside + # message/delivery-status header block, which triggered an + # unreadline() of NeedMoreData. + msg = self._msgobj('msg_43.txt') + sfp = StringIO() + iterators._structure(msg, sfp) + eq(sfp.getvalue(), """\ +multipart/report + text/plain + message/delivery-status + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/plain + text/rfc822-headers +""") + + def test_make_msgid_domain(self): + self.assertEqual( + email.utils.make_msgid(domain='testdomain-string')[-19:], + '@testdomain-string>') + + def test_make_msgid_idstring(self): + self.assertEqual( + email.utils.make_msgid(idstring='test-idstring', + domain='testdomain-string')[-33:], + '.test-idstring@testdomain-string>') + + def test_make_msgid_default_domain(self): + with patch('socket.getfqdn') as mock_getfqdn: + mock_getfqdn.return_value = domain = 'pythontest.example.com' + self.assertEndsWith(email.utils.make_msgid(), '@' + domain + '>') + + def test_Generator_linend(self): + # Issue 14645. + with openfile('msg_26.txt', encoding="utf-8", newline='\n') as f: + msgtxt = f.read() + msgtxt_nl = msgtxt.replace('\r\n', '\n') + msg = email.message_from_string(msgtxt) + s = StringIO() + g = email.generator.Generator(s) + g.flatten(msg) + self.assertEqual(s.getvalue(), msgtxt_nl) + + def test_BytesGenerator_linend(self): + # Issue 14645. + with openfile('msg_26.txt', encoding="utf-8", newline='\n') as f: + msgtxt = f.read() + msgtxt_nl = msgtxt.replace('\r\n', '\n') + msg = email.message_from_string(msgtxt_nl) + s = BytesIO() + g = email.generator.BytesGenerator(s) + g.flatten(msg, linesep='\r\n') + self.assertEqual(s.getvalue().decode('ascii'), msgtxt) + + def test_BytesGenerator_linend_with_non_ascii(self): + # Issue 14645. + with openfile('msg_26.txt', 'rb') as f: + msgtxt = f.read() + msgtxt = msgtxt.replace(b'with attachment', b'fo\xf6') + msgtxt_nl = msgtxt.replace(b'\r\n', b'\n') + msg = email.message_from_bytes(msgtxt_nl) + s = BytesIO() + g = email.generator.BytesGenerator(s) + g.flatten(msg, linesep='\r\n') + self.assertEqual(s.getvalue(), msgtxt) + + def test_mime_classes_policy_argument(self): + with openfile('sndhdr.au', 'rb') as fp: + audiodata = fp.read() + with openfile('python.gif', 'rb') as fp: + bindata = fp.read() + classes = [ + (MIMEApplication, ('',)), + (MIMEAudio, (audiodata,)), + (MIMEImage, (bindata,)), + (MIMEMessage, (Message(),)), + (MIMENonMultipart, ('multipart', 'mixed')), + (MIMEText, ('',)), + ] + for cls, constructor in classes: + with self.subTest(cls=cls.__name__, policy='compat32'): + m = cls(*constructor) + self.assertIs(m.policy, email.policy.compat32) + with self.subTest(cls=cls.__name__, policy='default'): + m = cls(*constructor, policy=email.policy.default) + self.assertIs(m.policy, email.policy.default) + + def test_iter_escaped_chars(self): + self.assertEqual(list(utils._iter_escaped_chars(r'a\\b\"c\\"d')), + [(0, 'a'), + (2, '\\\\'), + (3, 'b'), + (5, '\\"'), + (6, 'c'), + (8, '\\\\'), + (9, '"'), + (10, 'd')]) + self.assertEqual(list(utils._iter_escaped_chars('a\\')), + [(0, 'a'), (1, '\\')]) + + def test_strip_quoted_realnames(self): + def check(addr, expected): + self.assertEqual(utils._strip_quoted_realnames(addr), expected) + + check('"Jane Doe" , "John Doe" ', + ' , ') + check(r'"Jane \"Doe\"." ', + ' ') + + # special cases + check(r'before"name"after', 'beforeafter') + check(r'before"name"', 'before') + check(r'b"name"', 'b') # single char + check(r'"name"after', 'after') + check(r'"name"a', 'a') # single char + check(r'"name"', '') + + # no change + for addr in ( + 'Jane Doe , John Doe ', + 'lone " quote', + ): + self.assertEqual(utils._strip_quoted_realnames(addr), addr) + + + def test_check_parenthesis(self): + addr = 'alice@example.net' + self.assertTrue(utils._check_parenthesis(f'{addr} (Alice)')) + self.assertFalse(utils._check_parenthesis(f'{addr} )Alice(')) + self.assertFalse(utils._check_parenthesis(f'{addr} (Alice))')) + self.assertFalse(utils._check_parenthesis(f'{addr} ((Alice)')) + + # Ignore real name between quotes + self.assertTrue(utils._check_parenthesis(f'")Alice((" {addr}')) + + +# Test the iterator/generators +class TestIterators(TestEmailBase): + def test_body_line_iterator(self): + eq = self.assertEqual + neq = self.ndiffAssertEqual + # First a simple non-multipart message + msg = self._msgobj('msg_01.txt') + it = iterators.body_line_iterator(msg) + lines = list(it) + eq(len(lines), 6) + neq(EMPTYSTRING.join(lines), msg.get_payload()) + # Now a more complicated multipart + msg = self._msgobj('msg_02.txt') + it = iterators.body_line_iterator(msg) + lines = list(it) + eq(len(lines), 43) + with openfile('msg_19.txt', encoding="utf-8") as fp: + neq(EMPTYSTRING.join(lines), fp.read()) + + def test_typed_subpart_iterator(self): + eq = self.assertEqual + msg = self._msgobj('msg_04.txt') + it = iterators.typed_subpart_iterator(msg, 'text') + lines = [] + subparts = 0 + for subpart in it: + subparts += 1 + lines.append(subpart.get_payload()) + eq(subparts, 2) + eq(EMPTYSTRING.join(lines), """\ +a simple kind of mirror +to reflect upon our own +a simple kind of mirror +to reflect upon our own +""") + + def test_typed_subpart_iterator_default_type(self): + eq = self.assertEqual + msg = self._msgobj('msg_03.txt') + it = iterators.typed_subpart_iterator(msg, 'text', 'plain') + lines = [] + subparts = 0 + for subpart in it: + subparts += 1 + lines.append(subpart.get_payload()) + eq(subparts, 1) + eq(EMPTYSTRING.join(lines), """\ + +Hi, + +Do you like this message? + +-Me +""") + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_pushCR_LF(self): + '''FeedParser BufferedSubFile.push() assumed it received complete + line endings. A CR ending one push() followed by a LF starting + the next push() added an empty line. + ''' + imt = [ + ("a\r \n", 2), + ("b", 0), + ("c\n", 1), + ("", 0), + ("d\r\n", 1), + ("e\r", 0), + ("\nf", 1), + ("\r\n", 1), + ] + from email.feedparser import BufferedSubFile, NeedMoreData + bsf = BufferedSubFile() + om = [] + nt = 0 + for il, n in imt: + bsf.push(il) + nt += n + n1 = 0 + for ol in iter(bsf.readline, NeedMoreData): + om.append(ol) + n1 += 1 + self.assertEqual(n, n1) + self.assertEqual(len(om), nt) + self.assertEqual(''.join([il for il, n in imt]), ''.join(om)) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_push_random(self): + from email.feedparser import BufferedSubFile, NeedMoreData + + n = 10000 + chunksize = 5 + chars = 'abcd \t\r\n' + + s = ''.join(choice(chars) for i in range(n)) + '\n' + target = s.splitlines(True) + + bsf = BufferedSubFile() + lines = [] + for i in range(0, len(s), chunksize): + chunk = s[i:i+chunksize] + bsf.push(chunk) + lines.extend(iter(bsf.readline, NeedMoreData)) + self.assertEqual(lines, target) + + +class TestFeedParsers(TestEmailBase): + + def parse(self, chunks): + feedparser = FeedParser() + for chunk in chunks: + feedparser.feed(chunk) + return feedparser.close() + + def test_empty_header_name_handled(self): + # Issue 19996 + msg = self.parse("First: val\n: bad\nSecond: val") + self.assertEqual(msg['First'], 'val') + self.assertEqual(msg['Second'], 'val') + + @unittest.expectedFailure # TODO: RUSTPYTHON; Feedparser.feed -> Feedparser._input.push, Feedparser._call_parse -> Feedparser._parse does not keep _input state between calls + def test_newlines(self): + m = self.parse(['a:\nb:\rc:\r\nd:\n']) + self.assertEqual(m.keys(), ['a', 'b', 'c', 'd']) + m = self.parse(['a:\nb:\rc:\r\nd:']) + self.assertEqual(m.keys(), ['a', 'b', 'c', 'd']) + m = self.parse(['a:\rb', 'c:\n']) + self.assertEqual(m.keys(), ['a', 'bc']) + m = self.parse(['a:\r', 'b:\n']) + self.assertEqual(m.keys(), ['a', 'b']) + m = self.parse(['a:\r', '\nb:\n']) + self.assertEqual(m.keys(), ['a', 'b']) + + # Only CR and LF should break header fields + m = self.parse(['a:\x85b:\u2028c:\n']) + self.assertEqual(m.items(), [('a', '\x85b:\u2028c:')]) + m = self.parse(['a:\r', 'b:\x85', 'c:\n']) + self.assertEqual(m.items(), [('a', ''), ('b', '\x85c:')]) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_long_lines(self): + # Expected peak memory use on 32-bit platform: 6*N*M bytes. + M, N = 1000, 20000 + m = self.parse(['a:b\n\n'] + ['x'*M] * N) + self.assertEqual(m.items(), [('a', 'b')]) + self.assertEqual(m.get_payload(), 'x'*M*N) + m = self.parse(['a:b\r\r'] + ['x'*M] * N) + self.assertEqual(m.items(), [('a', 'b')]) + self.assertEqual(m.get_payload(), 'x'*M*N) + m = self.parse(['a:b\r\r'] + ['x'*M+'\x85'] * N) + self.assertEqual(m.items(), [('a', 'b')]) + self.assertEqual(m.get_payload(), ('x'*M+'\x85')*N) + m = self.parse(['a:\r', 'b: '] + ['x'*M] * N) + self.assertEqual(m.items(), [('a', ''), ('b', 'x'*M*N)]) + + +class TestParsers(TestEmailBase): + + def test_header_parser(self): + eq = self.assertEqual + # Parse only the headers of a complex multipart MIME document + with openfile('msg_02.txt', encoding="utf-8") as fp: + msg = HeaderParser().parse(fp) + eq(msg['from'], 'ppp-request@zzz.org') + eq(msg['to'], 'ppp@zzz.org') + eq(msg.get_content_type(), 'multipart/mixed') + self.assertFalse(msg.is_multipart()) + self.assertIsInstance(msg.get_payload(), str) + + def test_bytes_header_parser(self): + eq = self.assertEqual + # Parse only the headers of a complex multipart MIME document + with openfile('msg_02.txt', 'rb') as fp: + msg = email.parser.BytesHeaderParser().parse(fp) + eq(msg['from'], 'ppp-request@zzz.org') + eq(msg['to'], 'ppp@zzz.org') + eq(msg.get_content_type(), 'multipart/mixed') + self.assertFalse(msg.is_multipart()) + self.assertIsInstance(msg.get_payload(), str) + self.assertIsInstance(msg.get_payload(decode=True), bytes) + + def test_header_parser_multipart_is_valid(self): + # Don't flag valid multipart emails as having defects + with openfile('msg_47.txt', encoding="utf-8") as fp: + msgdata = fp.read() + + parser = email.parser.Parser(policy=email.policy.default) + parsed_msg = parser.parsestr(msgdata, headersonly=True) + + self.assertEqual(parsed_msg.defects, []) + + def test_bytes_parser_does_not_close_file(self): + with openfile('msg_02.txt', 'rb') as fp: + email.parser.BytesParser().parse(fp) + self.assertFalse(fp.closed) + + def test_bytes_parser_on_exception_does_not_close_file(self): + with openfile('msg_15.txt', 'rb') as fp: + bytesParser = email.parser.BytesParser + self.assertRaises(email.errors.StartBoundaryNotFoundDefect, + bytesParser(policy=email.policy.strict).parse, + fp) + self.assertFalse(fp.closed) + + def test_parser_does_not_close_file(self): + with openfile('msg_02.txt', encoding="utf-8") as fp: + email.parser.Parser().parse(fp) + self.assertFalse(fp.closed) + + def test_parser_on_exception_does_not_close_file(self): + with openfile('msg_15.txt', encoding="utf-8") as fp: + parser = email.parser.Parser + self.assertRaises(email.errors.StartBoundaryNotFoundDefect, + parser(policy=email.policy.strict).parse, fp) + self.assertFalse(fp.closed) + + def test_whitespace_continuation(self): + eq = self.assertEqual + # This message contains a line after the Subject: header that has only + # whitespace, but it is not empty! + msg = email.message_from_string("""\ +From: aperson@dom.ain +To: bperson@dom.ain +Subject: the next line has a space on it +\x20 +Date: Mon, 8 Apr 2002 15:09:19 -0400 +Message-ID: spam + +Here's the message body +""") + eq(msg['subject'], 'the next line has a space on it\n ') + eq(msg['message-id'], 'spam') + eq(msg.get_payload(), "Here's the message body\n") + + def test_whitespace_continuation_last_header(self): + eq = self.assertEqual + # Like the previous test, but the subject line is the last + # header. + msg = email.message_from_string("""\ +From: aperson@dom.ain +To: bperson@dom.ain +Date: Mon, 8 Apr 2002 15:09:19 -0400 +Message-ID: spam +Subject: the next line has a space on it +\x20 + +Here's the message body +""") + eq(msg['subject'], 'the next line has a space on it\n ') + eq(msg['message-id'], 'spam') + eq(msg.get_payload(), "Here's the message body\n") + + def test_crlf_separation(self): + eq = self.assertEqual + with openfile('msg_26.txt', encoding="utf-8", newline='\n') as fp: + msg = Parser().parse(fp) + eq(len(msg.get_payload()), 2) + part1 = msg.get_payload(0) + eq(part1.get_content_type(), 'text/plain') + eq(part1.get_payload(), 'Simple email with attachment.\r\n\r\n') + part2 = msg.get_payload(1) + eq(part2.get_content_type(), 'application/riscos') + + def test_crlf_flatten(self): + # Using newline='\n' preserves the crlfs in this input file. + with openfile('msg_26.txt', encoding="utf-8", newline='\n') as fp: + text = fp.read() + msg = email.message_from_string(text) + s = StringIO() + g = Generator(s) + g.flatten(msg, linesep='\r\n') + self.assertEqual(s.getvalue(), text) + + maxDiff = None + + def test_multipart_digest_with_extra_mime_headers(self): + eq = self.assertEqual + neq = self.ndiffAssertEqual + with openfile('msg_28.txt', encoding="utf-8") as fp: + msg = email.message_from_file(fp) + # Structure is: + # multipart/digest + # message/rfc822 + # text/plain + # message/rfc822 + # text/plain + eq(msg.is_multipart(), 1) + eq(len(msg.get_payload()), 2) + part1 = msg.get_payload(0) + eq(part1.get_content_type(), 'message/rfc822') + eq(part1.is_multipart(), 1) + eq(len(part1.get_payload()), 1) + part1a = part1.get_payload(0) + eq(part1a.is_multipart(), 0) + eq(part1a.get_content_type(), 'text/plain') + neq(part1a.get_payload(), 'message 1\n') + # next message/rfc822 + part2 = msg.get_payload(1) + eq(part2.get_content_type(), 'message/rfc822') + eq(part2.is_multipart(), 1) + eq(len(part2.get_payload()), 1) + part2a = part2.get_payload(0) + eq(part2a.is_multipart(), 0) + eq(part2a.get_content_type(), 'text/plain') + neq(part2a.get_payload(), 'message 2\n') + + def test_three_lines(self): + # A bug report by Andrew McNamara + lines = ['From: Andrew Person From', 'From']) + eq(msg.get_payload(), 'body') + + def test_rfc2822_space_not_allowed_in_header(self): + eq = self.assertEqual + m = '>From foo@example.com 11:25:53\nFrom: bar\n!"#QUX;~: zoo\n\nbody' + msg = email.message_from_string(m) + eq(len(msg.keys()), 0) + + def test_rfc2822_one_character_header(self): + eq = self.assertEqual + m = 'A: first header\nB: second header\nCC: third header\n\nbody' + msg = email.message_from_string(m) + headers = msg.keys() + headers.sort() + eq(headers, ['A', 'B', 'CC']) + eq(msg.get_payload(), 'body') + + def test_CRLFLF_at_end_of_part(self): + # issue 5610: feedparser should not eat two chars from body part ending + # with "\r\n\n". + m = ( + "From: foo@bar.com\n" + "To: baz\n" + "Mime-Version: 1.0\n" + "Content-Type: multipart/mixed; boundary=BOUNDARY\n" + "\n" + "--BOUNDARY\n" + "Content-Type: text/plain\n" + "\n" + "body ending with CRLF newline\r\n" + "\n" + "--BOUNDARY--\n" + ) + msg = email.message_from_string(m) + self.assertEndsWith(msg.get_payload(0).get_payload(), '\r\n') + + +class Test8BitBytesHandling(TestEmailBase): + # In Python3 all input is string, but that doesn't work if the actual input + # uses an 8bit transfer encoding. To hack around that, in email 5.1 we + # decode byte streams using the surrogateescape error handler, and + # reconvert to binary at appropriate places if we detect surrogates. This + # doesn't allow us to transform headers with 8bit bytes (they get munged), + # but it does allow us to parse and preserve them, and to decode body + # parts that use an 8bit CTE. + + bodytest_msg = textwrap.dedent("""\ + From: foo@bar.com + To: baz + Mime-Version: 1.0 + Content-Type: text/plain; charset={charset} + Content-Transfer-Encoding: {cte} + + {bodyline} + """) + + def test_known_8bit_CTE(self): + m = self.bodytest_msg.format(charset='utf-8', + cte='8bit', + bodyline='pöstal').encode('utf-8') + msg = email.message_from_bytes(m) + self.assertEqual(msg.get_payload(), "pöstal\n") + self.assertEqual(msg.get_payload(decode=True), + "pöstal\n".encode('utf-8')) + + def test_unknown_8bit_CTE(self): + m = self.bodytest_msg.format(charset='notavalidcharset', + cte='8bit', + bodyline='pöstal').encode('utf-8') + msg = email.message_from_bytes(m) + self.assertEqual(msg.get_payload(), "p\uFFFD\uFFFDstal\n") + self.assertEqual(msg.get_payload(decode=True), + "pöstal\n".encode('utf-8')) + + def test_8bit_in_quopri_body(self): + # This is non-RFC compliant data...without 'decode' the library code + # decodes the body using the charset from the headers, and because the + # source byte really is utf-8 this works. This is likely to fail + # against real dirty data (ie: produce mojibake), but the data is + # invalid anyway so it is as good a guess as any. But this means that + # this test just confirms the current behavior; that behavior is not + # necessarily the best possible behavior. With 'decode' it is + # returning the raw bytes, so that test should be of correct behavior, + # or at least produce the same result that email4 did. + m = self.bodytest_msg.format(charset='utf-8', + cte='quoted-printable', + bodyline='p=C3=B6stál').encode('utf-8') + msg = email.message_from_bytes(m) + self.assertEqual(msg.get_payload(), 'p=C3=B6stál\n') + self.assertEqual(msg.get_payload(decode=True), + 'pöstál\n'.encode('utf-8')) + + def test_invalid_8bit_in_non_8bit_cte_uses_replace(self): + # This is similar to the previous test, but proves that if the 8bit + # byte is undecodeable in the specified charset, it gets replaced + # by the unicode 'unknown' character. Again, this may or may not + # be the ideal behavior. Note that if decode=False none of the + # decoders will get involved, so this is the only test we need + # for this behavior. + m = self.bodytest_msg.format(charset='ascii', + cte='quoted-printable', + bodyline='p=C3=B6stál').encode('utf-8') + msg = email.message_from_bytes(m) + self.assertEqual(msg.get_payload(), 'p=C3=B6st\uFFFD\uFFFDl\n') + self.assertEqual(msg.get_payload(decode=True), + 'pöstál\n'.encode('utf-8')) + + # test_defect_handling:test_invalid_chars_in_base64_payload + def test_8bit_in_base64_body(self): + # If we get 8bit bytes in a base64 body, we can just ignore them + # as being outside the base64 alphabet and decode anyway. But + # we register a defect. + m = self.bodytest_msg.format(charset='utf-8', + cte='base64', + bodyline='cMO2c3RhbAá=').encode('utf-8') + msg = email.message_from_bytes(m) + self.assertEqual(msg.get_payload(decode=True), + 'pöstal'.encode('utf-8')) + self.assertIsInstance(msg.defects[0], + errors.InvalidBase64CharactersDefect) + + def test_8bit_in_uuencode_body(self): + # Sticking an 8bit byte in a uuencode block makes it undecodable by + # normal means, so the block is returned undecoded, but as bytes. + m = self.bodytest_msg.format(charset='utf-8', + cte='uuencode', + bodyline='<,.V7bit conversion. + self.assertEqual(out.getvalue(), + self.latin_bin_msg.decode('latin-1')+'\n') + + def test_bytes_feedparser(self): + bfp = email.feedparser.BytesFeedParser() + for i in range(0, len(self.latin_bin_msg), 10): + bfp.feed(self.latin_bin_msg[i:i+10]) + m = bfp.close() + self.assertEqual(str(m), self.latin_bin_msg_as7bit) + + def test_crlf_flatten(self): + with openfile('msg_26.txt', 'rb') as fp: + text = fp.read() + msg = email.message_from_bytes(text) + s = BytesIO() + g = email.generator.BytesGenerator(s) + g.flatten(msg, linesep='\r\n') + self.assertEqual(s.getvalue(), text) + + def test_8bit_multipart(self): + # Issue 11605 + source = textwrap.dedent("""\ + Date: Fri, 18 Mar 2011 17:15:43 +0100 + To: foo@example.com + From: foodwatch-Newsletter + Subject: Aktuelles zu Japan, Klonfleisch und Smiley-System + Message-ID: <76a486bee62b0d200f33dc2ca08220ad@localhost.localdomain> + MIME-Version: 1.0 + Content-Type: multipart/alternative; + boundary="b1_76a486bee62b0d200f33dc2ca08220ad" + + --b1_76a486bee62b0d200f33dc2ca08220ad + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: 8bit + + Guten Tag, , + + mit großer Betroffenheit verfolgen auch wir im foodwatch-Team die + Nachrichten aus Japan. + + + --b1_76a486bee62b0d200f33dc2ca08220ad + Content-Type: text/html; charset="utf-8" + Content-Transfer-Encoding: 8bit + + + + + foodwatch - Newsletter + + +

    mit großer Betroffenheit verfolgen auch wir im foodwatch-Team + die Nachrichten aus Japan.

    + + + --b1_76a486bee62b0d200f33dc2ca08220ad-- + + """).encode('utf-8') + msg = email.message_from_bytes(source) + s = BytesIO() + g = email.generator.BytesGenerator(s) + g.flatten(msg) + self.assertEqual(s.getvalue(), source) + + def test_bytes_generator_b_encoding_linesep(self): + # Issue 14062: b encoding was tacking on an extra \n. + m = Message() + # This has enough non-ascii that it should always end up b encoded. + m['Subject'] = Header('žluťoučký kůň') + s = BytesIO() + g = email.generator.BytesGenerator(s) + g.flatten(m, linesep='\r\n') + self.assertEqual( + s.getvalue(), + b'Subject: =?utf-8?b?xb5sdcWlb3XEjWvDvSBrxa/FiA==?=\r\n\r\n') + + def test_generator_b_encoding_linesep(self): + # Since this broke in ByteGenerator, test Generator for completeness. + m = Message() + # This has enough non-ascii that it should always end up b encoded. + m['Subject'] = Header('žluťoučký kůň') + s = StringIO() + g = email.generator.Generator(s) + g.flatten(m, linesep='\r\n') + self.assertEqual( + s.getvalue(), + 'Subject: =?utf-8?b?xb5sdcWlb3XEjWvDvSBrxa/FiA==?=\r\n\r\n') + + maxDiff = None + + +class BaseTestBytesGeneratorIdempotent: + + maxDiff = None + + def _msgobj(self, filename): + with openfile(filename, 'rb') as fp: + data = fp.read() + data = self.normalize_linesep_regex.sub(self.blinesep, data) + msg = email.message_from_bytes(data) + return msg, data + + def _idempotent(self, msg, data, unixfrom=False): + b = BytesIO() + g = email.generator.BytesGenerator(b, maxheaderlen=0) + g.flatten(msg, unixfrom=unixfrom, linesep=self.linesep) + self.assertEqual(data, b.getvalue()) + + +class TestBytesGeneratorIdempotentNL(BaseTestBytesGeneratorIdempotent, + TestIdempotent): + linesep = '\n' + blinesep = b'\n' + normalize_linesep_regex = re.compile(br'\r\n') + + +class TestBytesGeneratorIdempotentCRLF(BaseTestBytesGeneratorIdempotent, + TestIdempotent): + linesep = '\r\n' + blinesep = b'\r\n' + normalize_linesep_regex = re.compile(br'(? A+B==2', 'A=1,B=A ==> A+B==2') + + def _test_encode(self, body, expected_encoded_body, maxlinelen=None, eol=None): + kwargs = {} + if maxlinelen is None: + # Use body_encode's default. + maxlinelen = 76 + else: + kwargs['maxlinelen'] = maxlinelen + if eol is None: + # Use body_encode's default. + eol = '\n' + else: + kwargs['eol'] = eol + encoded_body = quoprimime.body_encode(body, **kwargs) + self.assertEqual(encoded_body, expected_encoded_body) + if eol == '\n' or eol == '\r\n': + # We know how to split the result back into lines, so maxlinelen + # can be checked. + for line in encoded_body.splitlines(): + self.assertLessEqual(len(line), maxlinelen) + + def test_encode_null(self): + self._test_encode('', '') + + def test_encode_null_lines(self): + self._test_encode('\n\n', '\n\n') + + def test_encode_one_line(self): + self._test_encode('hello\n', 'hello\n') + + def test_encode_one_line_crlf(self): + self._test_encode('hello\r\n', 'hello\n') + + def test_encode_one_line_eol(self): + self._test_encode('hello\n', 'hello\r\n', eol='\r\n') + + def test_encode_one_line_eol_after_non_ascii(self): + # issue 20206; see changeset 0cf700464177 for why the encode/decode. + self._test_encode('hello\u03c5\n'.encode('utf-8').decode('latin1'), + 'hello=CF=85\r\n', eol='\r\n') + + def test_encode_one_space(self): + self._test_encode(' ', '=20') + + def test_encode_one_line_one_space(self): + self._test_encode(' \n', '=20\n') + +# XXX: body_encode() expect strings, but uses ord(char) from these strings +# to index into a 256-entry list. For code points above 255, this will fail. +# Should there be a check for 8-bit only ord() values in body, or at least +# a comment about the expected input? + + def test_encode_two_lines_one_space(self): + self._test_encode(' \n \n', '=20\n=20\n') + + def test_encode_one_word_trailing_spaces(self): + self._test_encode('hello ', 'hello =20') + + def test_encode_one_line_trailing_spaces(self): + self._test_encode('hello \n', 'hello =20\n') + + def test_encode_one_word_trailing_tab(self): + self._test_encode('hello \t', 'hello =09') + + def test_encode_one_line_trailing_tab(self): + self._test_encode('hello \t\n', 'hello =09\n') + + def test_encode_trailing_space_before_maxlinelen(self): + self._test_encode('abcd \n1234', 'abcd =\n\n1234', maxlinelen=6) + + def test_encode_trailing_space_at_maxlinelen(self): + self._test_encode('abcd \n1234', 'abcd=\n=20\n1234', maxlinelen=5) + + def test_encode_trailing_space_beyond_maxlinelen(self): + self._test_encode('abcd \n1234', 'abc=\nd=20\n1234', maxlinelen=4) + + def test_encode_whitespace_lines(self): + self._test_encode(' \n' * 5, '=20\n' * 5) + + def test_encode_quoted_equals(self): + self._test_encode('a = b', 'a =3D b') + + def test_encode_one_long_string(self): + self._test_encode('x' * 100, 'x' * 75 + '=\n' + 'x' * 25) + + def test_encode_one_long_line(self): + self._test_encode('x' * 100 + '\n', 'x' * 75 + '=\n' + 'x' * 25 + '\n') + + def test_encode_one_very_long_line(self): + self._test_encode('x' * 200 + '\n', + 2 * ('x' * 75 + '=\n') + 'x' * 50 + '\n') + + def test_encode_shortest_maxlinelen(self): + self._test_encode('=' * 5, '=3D=\n' * 4 + '=3D', maxlinelen=4) + + def test_encode_maxlinelen_too_small(self): + self.assertRaises(ValueError, self._test_encode, '', '', maxlinelen=3) + + def test_encode(self): + eq = self.assertEqual + eq(quoprimime.body_encode(''), '') + eq(quoprimime.body_encode('hello'), 'hello') + # Test the binary flag + eq(quoprimime.body_encode('hello\r\nworld'), 'hello\nworld') + # Test the maxlinelen arg + eq(quoprimime.body_encode('xxxx ' * 20, maxlinelen=40), """\ +xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx= + xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxx= +x xxxx xxxx xxxx xxxx=20""") + # Test the eol argument + eq(quoprimime.body_encode('xxxx ' * 20, maxlinelen=40, eol='\r\n'), + """\ +xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx=\r + xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxx=\r +x xxxx xxxx xxxx xxxx=20""") + eq(quoprimime.body_encode("""\ +one line + +two line"""), """\ +one line + +two line""") + + + +# Test the Charset class +class TestCharset(unittest.TestCase): + def tearDown(self): + from email import charset as CharsetModule + try: + del CharsetModule.CHARSETS['fake'] + except KeyError: + pass + + def test_codec_encodeable(self): + eq = self.assertEqual + # Make sure us-ascii = no Unicode conversion + c = Charset('us-ascii') + eq(c.header_encode('Hello World!'), 'Hello World!') + # Test 8-bit idempotency with us-ascii + s = '\xa4\xa2\xa4\xa4\xa4\xa6\xa4\xa8\xa4\xaa' + self.assertRaises(UnicodeError, c.header_encode, s) + c = Charset('utf-8') + eq(c.header_encode(s), '=?utf-8?b?wqTCosKkwqTCpMKmwqTCqMKkwqo=?=') + + def test_body_encode(self): + eq = self.assertEqual + # Try a charset with QP body encoding + c = Charset('iso-8859-1') + eq('hello w=F6rld', c.body_encode('hello w\xf6rld')) + # Try a charset with Base64 body encoding + c = Charset('utf-8') + eq('aGVsbG8gd29ybGQ=\n', c.body_encode(b'hello world')) + # Try a charset with None body encoding + c = Charset('us-ascii') + eq('hello world', c.body_encode('hello world')) + # Try the convert argument, where input codec != output codec + c = Charset('euc-jp') + # With apologies to Tokio Kikuchi ;) + # XXX FIXME +## try: +## eq('\x1b$B5FCO;~IW\x1b(B', +## c.body_encode('\xb5\xc6\xc3\xcf\xbb\xfe\xc9\xd7')) +## eq('\xb5\xc6\xc3\xcf\xbb\xfe\xc9\xd7', +## c.body_encode('\xb5\xc6\xc3\xcf\xbb\xfe\xc9\xd7', False)) +## except LookupError: +## # We probably don't have the Japanese codecs installed +## pass + # Testing SF bug #625509, which we have to fake, since there are no + # built-in encodings where the header encoding is QP but the body + # encoding is not. + from email import charset as CharsetModule + CharsetModule.add_charset('fake', CharsetModule.QP, None, 'utf-8') + c = Charset('fake') + eq('hello world', c.body_encode('hello world')) + + def test_unicode_charset_name(self): + charset = Charset('us-ascii') + self.assertEqual(str(charset), 'us-ascii') + self.assertRaises(errors.CharsetError, Charset, 'asc\xffii') + + + +# Test multilingual MIME headers. +class TestHeader(TestEmailBase): + def test_simple(self): + eq = self.ndiffAssertEqual + h = Header('Hello World!') + eq(h.encode(), 'Hello World!') + h.append(' Goodbye World!') + eq(h.encode(), 'Hello World! Goodbye World!') + + def test_simple_surprise(self): + eq = self.ndiffAssertEqual + h = Header('Hello World!') + eq(h.encode(), 'Hello World!') + h.append('Goodbye World!') + eq(h.encode(), 'Hello World! Goodbye World!') + + def test_header_needs_no_decoding(self): + h = 'no decoding needed' + self.assertEqual(decode_header(h), [(h, None)]) + + def test_long(self): + h = Header("I am the very model of a modern Major-General; I've information vegetable, animal, and mineral; I know the kings of England, and I quote the fights historical from Marathon to Waterloo, in order categorical; I'm very well acquainted, too, with matters mathematical; I understand equations, both the simple and quadratical; about binomial theorem I'm teeming with a lot o' news, with many cheerful facts about the square of the hypotenuse.", + maxlinelen=76) + for l in h.encode(splitchars=' ').split('\n '): + self.assertLessEqual(len(l), 76) + + def test_multilingual(self): + eq = self.ndiffAssertEqual + g = Charset("iso-8859-1") + cz = Charset("iso-8859-2") + utf8 = Charset("utf-8") + g_head = (b'Die Mieter treten hier ein werden mit einem ' + b'Foerderband komfortabel den Korridor entlang, ' + b'an s\xfcdl\xfcndischen Wandgem\xe4lden vorbei, ' + b'gegen die rotierenden Klingen bef\xf6rdert. ') + cz_head = (b'Finan\xe8ni metropole se hroutily pod tlakem jejich ' + b'd\xf9vtipu.. ') + utf8_head = ('\u6b63\u78ba\u306b\u8a00\u3046\u3068\u7ffb\u8a33\u306f' + '\u3055\u308c\u3066\u3044\u307e\u305b\u3093\u3002\u4e00' + '\u90e8\u306f\u30c9\u30a4\u30c4\u8a9e\u3067\u3059\u304c' + '\u3001\u3042\u3068\u306f\u3067\u305f\u3089\u3081\u3067' + '\u3059\u3002\u5b9f\u969b\u306b\u306f\u300cWenn ist das ' + 'Nunstuck git und Slotermeyer? Ja! Beiherhund das Oder ' + 'die Flipperwaldt gersput.\u300d\u3068\u8a00\u3063\u3066' + '\u3044\u307e\u3059\u3002') + h = Header(g_head, g) + h.append(cz_head, cz) + h.append(utf8_head, utf8) + enc = h.encode(maxlinelen=76) + eq(enc, """\ +=?iso-8859-1?q?Die_Mieter_treten_hier_ein_werden_mit_einem_Foerderband_kom?= + =?iso-8859-1?q?fortabel_den_Korridor_entlang=2C_an_s=FCdl=FCndischen_Wand?= + =?iso-8859-1?q?gem=E4lden_vorbei=2C_gegen_die_rotierenden_Klingen_bef=F6r?= + =?iso-8859-1?q?dert=2E_?= =?iso-8859-2?q?Finan=E8ni_metropole_se_hroutily?= + =?iso-8859-2?q?_pod_tlakem_jejich_d=F9vtipu=2E=2E_?= =?utf-8?b?5q2j56K6?= + =?utf-8?b?44Gr6KiA44GG44Go57+76Kiz44Gv44GV44KM44Gm44GE44G+44Gb44KT44CC?= + =?utf-8?b?5LiA6YOo44Gv44OJ44Kk44OE6Kqe44Gn44GZ44GM44CB44GC44Go44Gv44Gn?= + =?utf-8?b?44Gf44KJ44KB44Gn44GZ44CC5a6f6Zqb44Gr44Gv44CMV2VubiBpc3QgZGFz?= + =?utf-8?b?IE51bnN0dWNrIGdpdCB1bmQgU2xvdGVybWV5ZXI/IEphISBCZWloZXJodW5k?= + =?utf-8?b?IGRhcyBPZGVyIGRpZSBGbGlwcGVyd2FsZHQgZ2Vyc3B1dC7jgI3jgajoqIA=?= + =?utf-8?b?44Gj44Gm44GE44G+44GZ44CC?=""") + decoded = decode_header(enc) + eq(len(decoded), 3) + eq(decoded[0], (g_head, 'iso-8859-1')) + eq(decoded[1], (cz_head, 'iso-8859-2')) + eq(decoded[2], (utf8_head.encode('utf-8'), 'utf-8')) + ustr = str(h) + eq(ustr, + (b'Die Mieter treten hier ein werden mit einem Foerderband ' + b'komfortabel den Korridor entlang, an s\xc3\xbcdl\xc3\xbcndischen ' + b'Wandgem\xc3\xa4lden vorbei, gegen die rotierenden Klingen ' + b'bef\xc3\xb6rdert. Finan\xc4\x8dni metropole se hroutily pod ' + b'tlakem jejich d\xc5\xafvtipu.. \xe6\xad\xa3\xe7\xa2\xba\xe3\x81' + b'\xab\xe8\xa8\x80\xe3\x81\x86\xe3\x81\xa8\xe7\xbf\xbb\xe8\xa8\xb3' + b'\xe3\x81\xaf\xe3\x81\x95\xe3\x82\x8c\xe3\x81\xa6\xe3\x81\x84\xe3' + b'\x81\xbe\xe3\x81\x9b\xe3\x82\x93\xe3\x80\x82\xe4\xb8\x80\xe9\x83' + b'\xa8\xe3\x81\xaf\xe3\x83\x89\xe3\x82\xa4\xe3\x83\x84\xe8\xaa\x9e' + b'\xe3\x81\xa7\xe3\x81\x99\xe3\x81\x8c\xe3\x80\x81\xe3\x81\x82\xe3' + b'\x81\xa8\xe3\x81\xaf\xe3\x81\xa7\xe3\x81\x9f\xe3\x82\x89\xe3\x82' + b'\x81\xe3\x81\xa7\xe3\x81\x99\xe3\x80\x82\xe5\xae\x9f\xe9\x9a\x9b' + b'\xe3\x81\xab\xe3\x81\xaf\xe3\x80\x8cWenn ist das Nunstuck git ' + b'und Slotermeyer? Ja! Beiherhund das Oder die Flipperwaldt ' + b'gersput.\xe3\x80\x8d\xe3\x81\xa8\xe8\xa8\x80\xe3\x81\xa3\xe3\x81' + b'\xa6\xe3\x81\x84\xe3\x81\xbe\xe3\x81\x99\xe3\x80\x82' + ).decode('utf-8')) + # Test make_header() + newh = make_header(decode_header(enc)) + eq(newh, h) + + def test_empty_header_encode(self): + h = Header() + self.assertEqual(h.encode(), '') + + def test_header_ctor_default_args(self): + eq = self.ndiffAssertEqual + h = Header() + eq(h, '') + h.append('foo', Charset('iso-8859-1')) + eq(h, 'foo') + + def test_explicit_maxlinelen(self): + eq = self.ndiffAssertEqual + hstr = ('A very long line that must get split to something other ' + 'than at the 76th character boundary to test the non-default ' + 'behavior') + h = Header(hstr) + eq(h.encode(), '''\ +A very long line that must get split to something other than at the 76th + character boundary to test the non-default behavior''') + eq(str(h), hstr) + h = Header(hstr, header_name='Subject') + eq(h.encode(), '''\ +A very long line that must get split to something other than at the + 76th character boundary to test the non-default behavior''') + eq(str(h), hstr) + h = Header(hstr, maxlinelen=1024, header_name='Subject') + eq(h.encode(), hstr) + eq(str(h), hstr) + + def test_quopri_splittable(self): + eq = self.ndiffAssertEqual + h = Header(charset='iso-8859-1', maxlinelen=20) + x = 'xxxx ' * 20 + h.append(x) + s = h.encode() + eq(s, """\ +=?iso-8859-1?q?xxx?= + =?iso-8859-1?q?x_?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?_x?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?x_?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?_x?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?x_?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?_x?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?x_?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?_x?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?x_?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?_x?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?x_?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?_x?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?x_?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?_x?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?x_?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?_x?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?x_?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?_x?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?x_?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?xx?= + =?iso-8859-1?q?_?=""") + eq(x, str(make_header(decode_header(s)))) + h = Header(charset='iso-8859-1', maxlinelen=40) + h.append('xxxx ' * 20) + s = h.encode() + eq(s, """\ +=?iso-8859-1?q?xxxx_xxxx_xxxx_xxxx_xxx?= + =?iso-8859-1?q?x_xxxx_xxxx_xxxx_xxxx_?= + =?iso-8859-1?q?xxxx_xxxx_xxxx_xxxx_xx?= + =?iso-8859-1?q?xx_xxxx_xxxx_xxxx_xxxx?= + =?iso-8859-1?q?_xxxx_xxxx_?=""") + eq(x, str(make_header(decode_header(s)))) + + def test_base64_splittable(self): + eq = self.ndiffAssertEqual + h = Header(charset='koi8-r', maxlinelen=20) + x = 'xxxx ' * 20 + h.append(x) + s = h.encode() + eq(s, """\ +=?koi8-r?b?eHh4?= + =?koi8-r?b?eCB4?= + =?koi8-r?b?eHh4?= + =?koi8-r?b?IHh4?= + =?koi8-r?b?eHgg?= + =?koi8-r?b?eHh4?= + =?koi8-r?b?eCB4?= + =?koi8-r?b?eHh4?= + =?koi8-r?b?IHh4?= + =?koi8-r?b?eHgg?= + =?koi8-r?b?eHh4?= + =?koi8-r?b?eCB4?= + =?koi8-r?b?eHh4?= + =?koi8-r?b?IHh4?= + =?koi8-r?b?eHgg?= + =?koi8-r?b?eHh4?= + =?koi8-r?b?eCB4?= + =?koi8-r?b?eHh4?= + =?koi8-r?b?IHh4?= + =?koi8-r?b?eHgg?= + =?koi8-r?b?eHh4?= + =?koi8-r?b?eCB4?= + =?koi8-r?b?eHh4?= + =?koi8-r?b?IHh4?= + =?koi8-r?b?eHgg?= + =?koi8-r?b?eHh4?= + =?koi8-r?b?eCB4?= + =?koi8-r?b?eHh4?= + =?koi8-r?b?IHh4?= + =?koi8-r?b?eHgg?= + =?koi8-r?b?eHh4?= + =?koi8-r?b?eCB4?= + =?koi8-r?b?eHh4?= + =?koi8-r?b?IA==?=""") + eq(x, str(make_header(decode_header(s)))) + h = Header(charset='koi8-r', maxlinelen=40) + h.append(x) + s = h.encode() + eq(s, """\ +=?koi8-r?b?eHh4eCB4eHh4IHh4eHggeHh4?= + =?koi8-r?b?eCB4eHh4IHh4eHggeHh4eCB4?= + =?koi8-r?b?eHh4IHh4eHggeHh4eCB4eHh4?= + =?koi8-r?b?IHh4eHggeHh4eCB4eHh4IHh4?= + =?koi8-r?b?eHggeHh4eCB4eHh4IHh4eHgg?= + =?koi8-r?b?eHh4eCB4eHh4IA==?=""") + eq(x, str(make_header(decode_header(s)))) + + def test_us_ascii_header(self): + eq = self.assertEqual + s = 'hello' + x = decode_header(s) + eq(x, [('hello', None)]) + h = make_header(x) + eq(s, h.encode()) + + def test_string_charset(self): + eq = self.assertEqual + h = Header() + h.append('hello', 'iso-8859-1') + eq(h, 'hello') + +## def test_unicode_error(self): +## raises = self.assertRaises +## raises(UnicodeError, Header, u'[P\xf6stal]', 'us-ascii') +## raises(UnicodeError, Header, '[P\xf6stal]', 'us-ascii') +## h = Header() +## raises(UnicodeError, h.append, u'[P\xf6stal]', 'us-ascii') +## raises(UnicodeError, h.append, '[P\xf6stal]', 'us-ascii') +## raises(UnicodeError, Header, u'\u83ca\u5730\u6642\u592b', 'iso-8859-1') + + def test_utf8_shortest(self): + eq = self.assertEqual + h = Header('p\xf6stal', 'utf-8') + eq(h.encode(), '=?utf-8?q?p=C3=B6stal?=') + h = Header('\u83ca\u5730\u6642\u592b', 'utf-8') + eq(h.encode(), '=?utf-8?b?6I+K5Zyw5pmC5aSr?=') + + def test_bad_8bit_header(self): + raises = self.assertRaises + eq = self.assertEqual + x = b'Ynwp4dUEbay Auction Semiar- No Charge \x96 Earn Big' + raises(UnicodeError, Header, x) + h = Header() + raises(UnicodeError, h.append, x) + e = x.decode('utf-8', 'replace') + eq(str(Header(x, errors='replace')), e) + h.append(x, errors='replace') + eq(str(h), e) + + def test_escaped_8bit_header(self): + x = b'Ynwp4dUEbay Auction Semiar- No Charge \x96 Earn Big' + e = x.decode('ascii', 'surrogateescape') + h = Header(e, charset=email.charset.UNKNOWN8BIT) + self.assertEqual(str(h), + 'Ynwp4dUEbay Auction Semiar- No Charge \uFFFD Earn Big') + self.assertEqual(email.header.decode_header(h), [(x, 'unknown-8bit')]) + + def test_header_handles_binary_unknown8bit(self): + x = b'Ynwp4dUEbay Auction Semiar- No Charge \x96 Earn Big' + h = Header(x, charset=email.charset.UNKNOWN8BIT) + self.assertEqual(str(h), + 'Ynwp4dUEbay Auction Semiar- No Charge \uFFFD Earn Big') + self.assertEqual(email.header.decode_header(h), [(x, 'unknown-8bit')]) + + def test_make_header_handles_binary_unknown8bit(self): + x = b'Ynwp4dUEbay Auction Semiar- No Charge \x96 Earn Big' + h = Header(x, charset=email.charset.UNKNOWN8BIT) + h2 = email.header.make_header(email.header.decode_header(h)) + self.assertEqual(str(h2), + 'Ynwp4dUEbay Auction Semiar- No Charge \uFFFD Earn Big') + self.assertEqual(email.header.decode_header(h2), [(x, 'unknown-8bit')]) + + def test_modify_returned_list_does_not_change_header(self): + h = Header('test') + chunks = email.header.decode_header(h) + chunks.append(('ascii', 'test2')) + self.assertEqual(str(h), 'test') + + def test_encoded_adjacent_nonencoded(self): + eq = self.assertEqual + h = Header() + h.append('hello', 'iso-8859-1') + h.append('world') + s = h.encode() + eq(s, '=?iso-8859-1?q?hello?= world') + h = make_header(decode_header(s)) + eq(h.encode(), s) + + def test_whitespace_keeper(self): + eq = self.assertEqual + s = 'Subject: =?koi8-r?b?8NLP18XSy8EgzsEgxsnOwczYztk=?= =?koi8-r?q?=CA?= zz.' + parts = decode_header(s) + eq(parts, [(b'Subject: ', None), (b'\xf0\xd2\xcf\xd7\xc5\xd2\xcb\xc1 \xce\xc1 \xc6\xc9\xce\xc1\xcc\xd8\xce\xd9\xca', 'koi8-r'), (b' zz.', None)]) + hdr = make_header(parts) + eq(hdr.encode(), + 'Subject: =?koi8-r?b?8NLP18XSy8EgzsEgxsnOwczYztnK?= zz.') + + def test_broken_base64_header(self): + raises = self.assertRaises + s = 'Subject: =?EUC-KR?B?CSixpLDtKSC/7Liuvsax4iC6uLmwMcijIKHaILzSwd/H0SC8+LCjwLsgv7W/+Mj3I ?=' + raises(errors.HeaderParseError, decode_header, s) + + @unittest.expectedFailure # TODO: RUSTPYTHON; LookupError: unknown encoding: iso-2022-jp + def test_shift_jis_charset(self): + h = Header('文', charset='shift_jis') + self.assertEqual(h.encode(), '=?iso-2022-jp?b?GyRCSjgbKEI=?=') + + def test_flatten_header_with_no_value(self): + # Issue 11401 (regression from email 4.x) Note that the space after + # the header doesn't reflect the input, but this is also the way + # email 4.x behaved. At some point it would be nice to fix that. + msg = email.message_from_string("EmptyHeader:") + self.assertEqual(str(msg), "EmptyHeader: \n\n") + + def test_encode_preserves_leading_ws_on_value(self): + msg = Message() + msg['SomeHeader'] = ' value with leading ws' + self.assertEqual(str(msg), "SomeHeader: value with leading ws\n\n") + + def test_whitespace_header(self): + self.assertEqual(Header(' ').encode(), ' ') + + + +# Test RFC 2231 header parameters (en/de)coding +class TestRFC2231(TestEmailBase): + + # test_headerregistry.TestContentTypeHeader.rfc2231_encoded_with_double_quotes + # test_headerregistry.TestContentTypeHeader.rfc2231_single_quote_inside_double_quotes + def test_get_param(self): + eq = self.assertEqual + msg = self._msgobj('msg_29.txt') + eq(msg.get_param('title'), + ('us-ascii', 'en', 'This is even more ***fun*** isn\'t it!')) + eq(msg.get_param('title', unquote=False), + ('us-ascii', 'en', '"This is even more ***fun*** isn\'t it!"')) + + def test_set_param(self): + eq = self.ndiffAssertEqual + msg = Message() + msg.set_param('title', 'This is even more ***fun*** isn\'t it!', + charset='us-ascii') + eq(msg.get_param('title'), + ('us-ascii', '', 'This is even more ***fun*** isn\'t it!')) + msg.set_param('title', 'This is even more ***fun*** isn\'t it!', + charset='us-ascii', language='en') + eq(msg.get_param('title'), + ('us-ascii', 'en', 'This is even more ***fun*** isn\'t it!')) + msg = self._msgobj('msg_01.txt') + msg.set_param('title', 'This is even more ***fun*** isn\'t it!', + charset='us-ascii', language='en') + eq(msg.as_string(maxheaderlen=78), """\ +Return-Path: +Delivered-To: bbb@zzz.org +Received: by mail.zzz.org (Postfix, from userid 889) +\tid 27CEAD38CC; Fri, 4 May 2001 14:05:44 -0400 (EDT) +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit +Message-ID: <15090.61304.110929.45684@aaa.zzz.org> +From: bbb@ddd.com (John X. Doe) +To: bbb@zzz.org +Subject: This is a test message +Date: Fri, 4 May 2001 14:05:44 -0400 +Content-Type: text/plain; charset=us-ascii; + title*=us-ascii'en'This%20is%20even%20more%20%2A%2A%2Afun%2A%2A%2A%20isn%27t%20it%21 + + +Hi, + +Do you like this message? + +-Me +""") + + def test_set_param_requote(self): + msg = Message() + msg.set_param('title', 'foo') + self.assertEqual(msg['content-type'], 'text/plain; title="foo"') + msg.set_param('title', 'bar', requote=False) + self.assertEqual(msg['content-type'], 'text/plain; title=bar') + # tspecial is still quoted. + msg.set_param('title', "(bar)bell", requote=False) + self.assertEqual(msg['content-type'], 'text/plain; title="(bar)bell"') + + def test_del_param(self): + eq = self.ndiffAssertEqual + msg = self._msgobj('msg_01.txt') + msg.set_param('foo', 'bar', charset='us-ascii', language='en') + msg.set_param('title', 'This is even more ***fun*** isn\'t it!', + charset='us-ascii', language='en') + msg.del_param('foo', header='Content-Type') + eq(msg.as_string(maxheaderlen=78), """\ +Return-Path: +Delivered-To: bbb@zzz.org +Received: by mail.zzz.org (Postfix, from userid 889) +\tid 27CEAD38CC; Fri, 4 May 2001 14:05:44 -0400 (EDT) +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit +Message-ID: <15090.61304.110929.45684@aaa.zzz.org> +From: bbb@ddd.com (John X. Doe) +To: bbb@zzz.org +Subject: This is a test message +Date: Fri, 4 May 2001 14:05:44 -0400 +Content-Type: text/plain; charset="us-ascii"; + title*=us-ascii'en'This%20is%20even%20more%20%2A%2A%2Afun%2A%2A%2A%20isn%27t%20it%21 + + +Hi, + +Do you like this message? + +-Me +""") + + # test_headerregistry.TestContentTypeHeader.rfc2231_encoded_charset + # I changed the charset name, though, because the one in the file isn't + # a legal charset name. Should add a test for an illegal charset. + def test_rfc2231_get_content_charset(self): + eq = self.assertEqual + msg = self._msgobj('msg_32.txt') + eq(msg.get_content_charset(), 'us-ascii') + + # test_headerregistry.TestContentTypeHeader.rfc2231_encoded_no_double_quotes + def test_rfc2231_parse_rfc_quoting(self): + m = textwrap.dedent('''\ + Content-Disposition: inline; + \tfilename*0*=''This%20is%20even%20more%20; + \tfilename*1*=%2A%2A%2Afun%2A%2A%2A%20; + \tfilename*2="is it not.pdf" + + ''') + msg = email.message_from_string(m) + self.assertEqual(msg.get_filename(), + 'This is even more ***fun*** is it not.pdf') + self.assertEqual(m, msg.as_string()) + + # test_headerregistry.TestContentTypeHeader.rfc2231_encoded_with_double_quotes + def test_rfc2231_parse_extra_quoting(self): + m = textwrap.dedent('''\ + Content-Disposition: inline; + \tfilename*0*="''This%20is%20even%20more%20"; + \tfilename*1*="%2A%2A%2Afun%2A%2A%2A%20"; + \tfilename*2="is it not.pdf" + + ''') + msg = email.message_from_string(m) + self.assertEqual(msg.get_filename(), + 'This is even more ***fun*** is it not.pdf') + self.assertEqual(m, msg.as_string()) + + # test_headerregistry.TestContentTypeHeader.rfc2231_no_language_or_charset + # but new test uses *0* because otherwise lang/charset is not valid. + # test_headerregistry.TestContentTypeHeader.rfc2231_segmented_normal_values + def test_rfc2231_no_language_or_charset(self): + m = '''\ +Content-Transfer-Encoding: 8bit +Content-Disposition: inline; filename="file____C__DOCUMENTS_20AND_20SETTINGS_FABIEN_LOCAL_20SETTINGS_TEMP_nsmail.htm" +Content-Type: text/html; NAME*0=file____C__DOCUMENTS_20AND_20SETTINGS_FABIEN_LOCAL_20SETTINGS_TEM; NAME*1=P_nsmail.htm + +''' + msg = email.message_from_string(m) + param = msg.get_param('NAME') + self.assertNotIsInstance(param, tuple) + self.assertEqual( + param, + 'file____C__DOCUMENTS_20AND_20SETTINGS_FABIEN_LOCAL_20SETTINGS_TEMP_nsmail.htm') + + # test_headerregistry.TestContentTypeHeader.rfc2231_encoded_no_charset + def test_rfc2231_no_language_or_charset_in_filename(self): + m = '''\ +Content-Disposition: inline; +\tfilename*0*="''This%20is%20even%20more%20"; +\tfilename*1*="%2A%2A%2Afun%2A%2A%2A%20"; +\tfilename*2="is it not.pdf" + +''' + msg = email.message_from_string(m) + self.assertEqual(msg.get_filename(), + 'This is even more ***fun*** is it not.pdf') + + # Duplicate of previous test? + def test_rfc2231_no_language_or_charset_in_filename_encoded(self): + m = '''\ +Content-Disposition: inline; +\tfilename*0*="''This%20is%20even%20more%20"; +\tfilename*1*="%2A%2A%2Afun%2A%2A%2A%20"; +\tfilename*2="is it not.pdf" + +''' + msg = email.message_from_string(m) + self.assertEqual(msg.get_filename(), + 'This is even more ***fun*** is it not.pdf') + + # test_headerregistry.TestContentTypeHeader.rfc2231_partly_encoded, + # but the test below is wrong (the first part should be decoded). + def test_rfc2231_partly_encoded(self): + m = '''\ +Content-Disposition: inline; +\tfilename*0="''This%20is%20even%20more%20"; +\tfilename*1*="%2A%2A%2Afun%2A%2A%2A%20"; +\tfilename*2="is it not.pdf" + +''' + msg = email.message_from_string(m) + self.assertEqual( + msg.get_filename(), + 'This%20is%20even%20more%20***fun*** is it not.pdf') + + def test_rfc2231_partly_nonencoded(self): + m = '''\ +Content-Disposition: inline; +\tfilename*0="This%20is%20even%20more%20"; +\tfilename*1="%2A%2A%2Afun%2A%2A%2A%20"; +\tfilename*2="is it not.pdf" + +''' + msg = email.message_from_string(m) + self.assertEqual( + msg.get_filename(), + 'This%20is%20even%20more%20%2A%2A%2Afun%2A%2A%2A%20is it not.pdf') + + def test_rfc2231_no_language_or_charset_in_boundary(self): + m = '''\ +Content-Type: multipart/alternative; +\tboundary*0*="''This%20is%20even%20more%20"; +\tboundary*1*="%2A%2A%2Afun%2A%2A%2A%20"; +\tboundary*2="is it not.pdf" + +''' + msg = email.message_from_string(m) + self.assertEqual(msg.get_boundary(), + 'This is even more ***fun*** is it not.pdf') + + def test_rfc2231_no_language_or_charset_in_charset(self): + # This is a nonsensical charset value, but tests the code anyway + m = '''\ +Content-Type: text/plain; +\tcharset*0*="This%20is%20even%20more%20"; +\tcharset*1*="%2A%2A%2Afun%2A%2A%2A%20"; +\tcharset*2="is it not.pdf" + +''' + msg = email.message_from_string(m) + self.assertEqual(msg.get_content_charset(), + 'this is even more ***fun*** is it not.pdf') + + # test_headerregistry.TestContentTypeHeader.rfc2231_unknown_charset_treated_as_ascii + def test_rfc2231_bad_encoding_in_filename(self): + m = '''\ +Content-Disposition: inline; +\tfilename*0*="bogus'xx'This%20is%20even%20more%20"; +\tfilename*1*="%2A%2A%2Afun%2A%2A%2A%20"; +\tfilename*2="is it not.pdf" + +''' + msg = email.message_from_string(m) + self.assertEqual(msg.get_filename(), + 'This is even more ***fun*** is it not.pdf') + + def test_rfc2231_bad_encoding_in_charset(self): + m = """\ +Content-Type: text/plain; charset*=bogus''utf-8%E2%80%9D + +""" + msg = email.message_from_string(m) + # This should return None because non-ascii characters in the charset + # are not allowed. + self.assertEqual(msg.get_content_charset(), None) + + def test_rfc2231_bad_character_in_charset(self): + m = """\ +Content-Type: text/plain; charset*=ascii''utf-8%E2%80%9D + +""" + msg = email.message_from_string(m) + # This should return None because non-ascii characters in the charset + # are not allowed. + self.assertEqual(msg.get_content_charset(), None) + + def test_rfc2231_bad_character_in_filename(self): + m = '''\ +Content-Disposition: inline; +\tfilename*0*="ascii'xx'This%20is%20even%20more%20"; +\tfilename*1*="%2A%2A%2Afun%2A%2A%2A%20"; +\tfilename*2*="is it not.pdf%E2" + +''' + msg = email.message_from_string(m) + self.assertEqual(msg.get_filename(), + 'This is even more ***fun*** is it not.pdf\ufffd') + + def test_rfc2231_unknown_encoding(self): + m = """\ +Content-Transfer-Encoding: 8bit +Content-Disposition: inline; filename*=X-UNKNOWN''myfile.txt + +""" + msg = email.message_from_string(m) + self.assertEqual(msg.get_filename(), 'myfile.txt') + + def test_rfc2231_bad_character_in_encoding(self): + m = """\ +Content-Transfer-Encoding: 8bit +Content-Disposition: inline; filename*=utf-8\udce2\udc80\udc9d''myfile.txt + +""" + msg = email.message_from_string(m) + self.assertEqual(msg.get_filename(), 'myfile.txt') + + def test_rfc2231_single_tick_in_filename_extended(self): + eq = self.assertEqual + m = """\ +Content-Type: application/x-foo; +\tname*0*=\"Frank's\"; name*1*=\" Document\" + +""" + msg = email.message_from_string(m) + charset, language, s = msg.get_param('name') + eq(charset, None) + eq(language, None) + eq(s, "Frank's Document") + + # test_headerregistry.TestContentTypeHeader.rfc2231_single_quote_inside_double_quotes + def test_rfc2231_single_tick_in_filename(self): + m = """\ +Content-Type: application/x-foo; name*0=\"Frank's\"; name*1=\" Document\" + +""" + msg = email.message_from_string(m) + param = msg.get_param('name') + self.assertNotIsInstance(param, tuple) + self.assertEqual(param, "Frank's Document") + + def test_rfc2231_missing_tick(self): + m = '''\ +Content-Disposition: inline; +\tfilename*0*="'This%20is%20broken"; +''' + msg = email.message_from_string(m) + self.assertEqual( + msg.get_filename(), + "'This is broken") + + def test_rfc2231_missing_tick_with_encoded_non_ascii(self): + m = '''\ +Content-Disposition: inline; +\tfilename*0*="'This%20is%E2broken"; +''' + msg = email.message_from_string(m) + self.assertEqual( + msg.get_filename(), + "'This is\ufffdbroken") + + # test_headerregistry.TestContentTypeHeader.rfc2231_single_quote_in_value_with_charset_and_lang + def test_rfc2231_tick_attack_extended(self): + eq = self.assertEqual + m = """\ +Content-Type: application/x-foo; +\tname*0*=\"us-ascii'en-us'Frank's\"; name*1*=\" Document\" + +""" + msg = email.message_from_string(m) + charset, language, s = msg.get_param('name') + eq(charset, 'us-ascii') + eq(language, 'en-us') + eq(s, "Frank's Document") + + # test_headerregistry.TestContentTypeHeader.rfc2231_single_quote_in_non_encoded_value + def test_rfc2231_tick_attack(self): + m = """\ +Content-Type: application/x-foo; +\tname*0=\"us-ascii'en-us'Frank's\"; name*1=\" Document\" + +""" + msg = email.message_from_string(m) + param = msg.get_param('name') + self.assertNotIsInstance(param, tuple) + self.assertEqual(param, "us-ascii'en-us'Frank's Document") + + # test_headerregistry.TestContentTypeHeader.rfc2231_single_quotes_inside_quotes + def test_rfc2231_no_extended_values(self): + eq = self.assertEqual + m = """\ +Content-Type: application/x-foo; name=\"Frank's Document\" + +""" + msg = email.message_from_string(m) + eq(msg.get_param('name'), "Frank's Document") + + # test_headerregistry.TestContentTypeHeader.rfc2231_encoded_then_unencoded_segments + def test_rfc2231_encoded_then_unencoded_segments(self): + eq = self.assertEqual + m = """\ +Content-Type: application/x-foo; +\tname*0*=\"us-ascii'en-us'My\"; +\tname*1=\" Document\"; +\tname*2*=\" For You\" + +""" + msg = email.message_from_string(m) + charset, language, s = msg.get_param('name') + eq(charset, 'us-ascii') + eq(language, 'en-us') + eq(s, 'My Document For You') + + # test_headerregistry.TestContentTypeHeader.rfc2231_unencoded_then_encoded_segments + # test_headerregistry.TestContentTypeHeader.rfc2231_quoted_unencoded_then_encoded_segments + def test_rfc2231_unencoded_then_encoded_segments(self): + eq = self.assertEqual + m = """\ +Content-Type: application/x-foo; +\tname*0=\"us-ascii'en-us'My\"; +\tname*1*=\" Document\"; +\tname*2*=\" For You\" + +""" + msg = email.message_from_string(m) + charset, language, s = msg.get_param('name') + eq(charset, 'us-ascii') + eq(language, 'en-us') + eq(s, 'My Document For You') + + def test_should_not_hang_on_invalid_ew_messages(self): + messages = ["""From: user@host.com +To: user@host.com +Bad-Header: + =?us-ascii?Q?LCSwrV11+IB0rSbSker+M9vWR7wEDSuGqmHD89Gt=ea0nJFSaiz4vX3XMJPT4vrE?= + =?us-ascii?Q?xGUZeOnp0o22pLBB7CYLH74Js=wOlK6Tfru2U47qR?= + =?us-ascii?Q?72OfyEY2p2=2FrA9xNFyvH+fBTCmazxwzF8nGkK6D?= + +Hello! +""", """From: ����� �������� +To: "xxx" +Subject: ��� ���������� ����� ����� � ��������� �� ���� +MIME-Version: 1.0 +Content-Type: text/plain; charset="windows-1251"; +Content-Transfer-Encoding: 8bit + +�� ����� � ���� ������ ��� �������� +"""] + for m in messages: + with self.subTest(m=m): + msg = email.message_from_string(m) + + +# Tests to ensure that signed parts of an email are completely preserved, as +# required by RFC1847 section 2.1. Note that these are incomplete, because the +# email package does not currently always preserve the body. See issue 1670765. +class TestSigned(TestEmailBase): + + def _msg_and_obj(self, filename): + with openfile(filename, encoding="utf-8") as fp: + original = fp.read() + msg = email.message_from_string(original) + return original, msg + + def _signed_parts_eq(self, original, result): + # Extract the first mime part of each message + import re + repart = re.compile(r'^--([^\n]+)\n(.*?)\n--\1$', re.S | re.M) + inpart = repart.search(original).group(2) + outpart = repart.search(result).group(2) + self.assertEqual(outpart, inpart) + + def test_long_headers_as_string(self): + original, msg = self._msg_and_obj('msg_45.txt') + result = msg.as_string() + self._signed_parts_eq(original, result) + + def test_long_headers_as_string_maxheaderlen(self): + original, msg = self._msg_and_obj('msg_45.txt') + result = msg.as_string(maxheaderlen=60) + self._signed_parts_eq(original, result) + + def test_long_headers_flatten(self): + original, msg = self._msg_and_obj('msg_45.txt') + fp = StringIO() + Generator(fp).flatten(msg) + result = fp.getvalue() + self._signed_parts_eq(original, result) + +class TestHeaderRegistry(TestEmailBase): + # See issue gh-93010. + def test_HeaderRegistry(self): + reg = HeaderRegistry() + a = reg('Content-Disposition', 'attachment; 0*00="foo"') + self.assertIsInstance(a.defects[0], errors.InvalidHeaderDefect) + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test_generator.py b/Lib/test/test_email/test_generator.py new file mode 100644 index 00000000000..c75a842c335 --- /dev/null +++ b/Lib/test/test_email/test_generator.py @@ -0,0 +1,477 @@ +import io +import textwrap +import unittest +from email import message_from_string, message_from_bytes +from email.message import EmailMessage +from email.generator import Generator, BytesGenerator +from email.headerregistry import Address +from email import policy +import email.errors +from test.test_email import TestEmailBase, parameterize + + +@parameterize +class TestGeneratorBase: + + policy = policy.default + + def msgmaker(self, msg, policy=None): + policy = self.policy if policy is None else policy + return self.msgfunc(msg, policy=policy) + + refold_long_expected = { + 0: textwrap.dedent("""\ + To: whom_it_may_concern@example.com + From: nobody_you_want_to_know@example.com + Subject: We the willing led by the unknowing are doing the + impossible for the ungrateful. We have done so much for so long with so little + we are now qualified to do anything with nothing. + + None + """), + 40: textwrap.dedent("""\ + To: whom_it_may_concern@example.com + From: + nobody_you_want_to_know@example.com + Subject: We the willing led by the + unknowing are doing the impossible for + the ungrateful. We have done so much + for so long with so little we are now + qualified to do anything with nothing. + + None + """), + 20: textwrap.dedent("""\ + To: + whom_it_may_concern@example.com + From: + nobody_you_want_to_know@example.com + Subject: We the + willing led by the + unknowing are doing + the impossible for + the ungrateful. We + have done so much + for so long with so + little we are now + qualified to do + anything with + nothing. + + None + """), + } + refold_long_expected[100] = refold_long_expected[0] + + refold_all_expected = refold_long_expected.copy() + refold_all_expected[0] = ( + "To: whom_it_may_concern@example.com\n" + "From: nobody_you_want_to_know@example.com\n" + "Subject: We the willing led by the unknowing are doing the " + "impossible for the ungrateful. We have done so much for " + "so long with so little we are now qualified to do anything " + "with nothing.\n" + "\n" + "None\n") + refold_all_expected[100] = ( + "To: whom_it_may_concern@example.com\n" + "From: nobody_you_want_to_know@example.com\n" + "Subject: We the willing led by the unknowing are doing the " + "impossible for the ungrateful. We have\n" + " done so much for so long with so little we are now qualified " + "to do anything with nothing.\n" + "\n" + "None\n") + + length_params = [n for n in refold_long_expected] + + def length_as_maxheaderlen_parameter(self, n): + msg = self.msgmaker(self.typ(self.refold_long_expected[0])) + s = self.ioclass() + g = self.genclass(s, maxheaderlen=n, policy=self.policy) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(self.refold_long_expected[n])) + + def length_as_max_line_length_policy(self, n): + msg = self.msgmaker(self.typ(self.refold_long_expected[0])) + s = self.ioclass() + g = self.genclass(s, policy=self.policy.clone(max_line_length=n)) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(self.refold_long_expected[n])) + + def length_as_maxheaderlen_parm_overrides_policy(self, n): + msg = self.msgmaker(self.typ(self.refold_long_expected[0])) + s = self.ioclass() + g = self.genclass(s, maxheaderlen=n, + policy=self.policy.clone(max_line_length=10)) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(self.refold_long_expected[n])) + + def length_as_max_line_length_with_refold_none_does_not_fold(self, n): + msg = self.msgmaker(self.typ(self.refold_long_expected[0])) + s = self.ioclass() + g = self.genclass(s, policy=self.policy.clone(refold_source='none', + max_line_length=n)) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(self.refold_long_expected[0])) + + def length_as_max_line_length_with_refold_all_folds(self, n): + msg = self.msgmaker(self.typ(self.refold_long_expected[0])) + s = self.ioclass() + g = self.genclass(s, policy=self.policy.clone(refold_source='all', + max_line_length=n)) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(self.refold_all_expected[n])) + + def test_crlf_control_via_policy(self): + source = "Subject: test\r\n\r\ntest body\r\n" + expected = source + msg = self.msgmaker(self.typ(source)) + s = self.ioclass() + g = self.genclass(s, policy=policy.SMTP) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(expected)) + + def test_flatten_linesep_overrides_policy(self): + source = "Subject: test\n\ntest body\n" + expected = source + msg = self.msgmaker(self.typ(source)) + s = self.ioclass() + g = self.genclass(s, policy=policy.SMTP) + g.flatten(msg, linesep='\n') + self.assertEqual(s.getvalue(), self.typ(expected)) + + def test_flatten_linesep(self): + source = 'Subject: one\n two\r three\r\n four\r\n\r\ntest body\r\n' + msg = self.msgmaker(self.typ(source)) + self.assertEqual(msg['Subject'], 'one two three four') + + expected = 'Subject: one\n two\n three\n four\n\ntest body\n' + s = self.ioclass() + g = self.genclass(s) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(expected)) + + expected = 'Subject: one two three four\n\ntest body\n' + s = self.ioclass() + g = self.genclass(s, policy=self.policy.clone(refold_source='all')) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(expected)) + + def test_flatten_control_linesep(self): + source = 'Subject: one\v two\f three\x1c four\x1d five\x1e six\r\n\r\ntest body\r\n' + msg = self.msgmaker(self.typ(source)) + self.assertEqual(msg['Subject'], 'one\v two\f three\x1c four\x1d five\x1e six') + + expected = 'Subject: one\v two\f three\x1c four\x1d five\x1e six\n\ntest body\n' + s = self.ioclass() + g = self.genclass(s) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(expected)) + + s = self.ioclass() + g = self.genclass(s, policy=self.policy.clone(refold_source='all')) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(expected)) + + def test_set_mangle_from_via_policy(self): + source = textwrap.dedent("""\ + Subject: test that + from is mangled in the body! + + From time to time I write a rhyme. + """) + variants = ( + (None, True), + (policy.compat32, True), + (policy.default, False), + (policy.default.clone(mangle_from_=True), True), + ) + for p, mangle in variants: + expected = source.replace('From ', '>From ') if mangle else source + with self.subTest(policy=p, mangle_from_=mangle): + msg = self.msgmaker(self.typ(source)) + s = self.ioclass() + g = self.genclass(s, policy=p) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(expected)) + + def test_compat32_max_line_length_does_not_fold_when_none(self): + msg = self.msgmaker(self.typ(self.refold_long_expected[0])) + s = self.ioclass() + g = self.genclass(s, policy=policy.compat32.clone(max_line_length=None)) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(self.refold_long_expected[0])) + + def test_rfc2231_wrapping(self): + # This is pretty much just to make sure we don't have an infinite + # loop; I don't expect anyone to hit this in the field. + msg = self.msgmaker(self.typ(textwrap.dedent("""\ + To: nobody + Content-Disposition: attachment; + filename="afilenamelongenoghtowraphere" + + None + """))) + expected = textwrap.dedent("""\ + To: nobody + Content-Disposition: attachment; + filename*0*=us-ascii''afilename; + filename*1*=longenoghtowraphere + + None + """) + s = self.ioclass() + g = self.genclass(s, policy=self.policy.clone(max_line_length=33)) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(expected)) + + def test_rfc2231_wrapping_switches_to_default_len_if_too_narrow(self): + # This is just to make sure we don't have an infinite loop; I don't + # expect anyone to hit this in the field, so I'm not bothering to make + # the result optimal (the encoding isn't needed). + msg = self.msgmaker(self.typ(textwrap.dedent("""\ + To: nobody + Content-Disposition: attachment; + filename="afilenamelongenoghtowraphere" + + None + """))) + expected = textwrap.dedent("""\ + To: nobody + Content-Disposition: + attachment; + filename*0*=us-ascii''afilenamelongenoghtowraphere + + None + """) + s = self.ioclass() + g = self.genclass(s, policy=self.policy.clone(max_line_length=20)) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(expected)) + + def test_keep_encoded_newlines(self): + msg = self.msgmaker(self.typ(textwrap.dedent("""\ + To: nobody + Subject: Bad subject=?UTF-8?Q?=0A?=Bcc: injection@example.com + + None + """))) + expected = textwrap.dedent("""\ + To: nobody + Subject: Bad subject=?UTF-8?Q?=0A?=Bcc: injection@example.com + + None + """) + s = self.ioclass() + g = self.genclass(s, policy=self.policy.clone(max_line_length=80)) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(expected)) + + def test_keep_long_encoded_newlines(self): + msg = self.msgmaker(self.typ(textwrap.dedent("""\ + To: nobody + Subject: Bad subject=?UTF-8?Q?=0A?=Bcc: injection@example.com + + None + """))) + expected = textwrap.dedent("""\ + To: nobody + Subject: Bad subject + =?utf-8?q?=0A?=Bcc: + injection@example.com + + None + """) + s = self.ioclass() + g = self.genclass(s, policy=self.policy.clone(max_line_length=30)) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(expected)) + + +class TestGenerator(TestGeneratorBase, TestEmailBase): + + msgfunc = staticmethod(message_from_string) + genclass = Generator + ioclass = io.StringIO + typ = str + + def test_flatten_unicode_linesep(self): + source = 'Subject: one\x85 two\u2028 three\u2029 four\r\n\r\ntest body\r\n' + msg = self.msgmaker(self.typ(source)) + self.assertEqual(msg['Subject'], 'one\x85 two\u2028 three\u2029 four') + + expected = 'Subject: =?utf-8?b?b25lwoUgdHdv4oCoIHRocmVl4oCp?= four\n\ntest body\n' + s = self.ioclass() + g = self.genclass(s) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(expected)) + + s = self.ioclass() + g = self.genclass(s, policy=self.policy.clone(refold_source='all')) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(expected)) + + def test_verify_generated_headers(self): + """gh-121650: by default the generator prevents header injection""" + class LiteralHeader(str): + name = 'Header' + def fold(self, **kwargs): + return self + + for text in ( + 'Value\r\nBad Injection\r\n', + 'NoNewLine' + ): + with self.subTest(text=text): + message = message_from_string( + "Header: Value\r\n\r\nBody", + policy=self.policy, + ) + + del message['Header'] + message['Header'] = LiteralHeader(text) + + with self.assertRaises(email.errors.HeaderWriteError): + message.as_string() + + +class TestBytesGenerator(TestGeneratorBase, TestEmailBase): + + msgfunc = staticmethod(message_from_bytes) + genclass = BytesGenerator + ioclass = io.BytesIO + typ = lambda self, x: x.encode('ascii') + + def test_defaults_handle_spaces_between_encoded_words_when_folded(self): + source = ("Уведомление о принятии в работу обращения для" + " подключения услуги") + expected = ('Subject: =?utf-8?b?0KPQstC10LTQvtC80LvQtdC90LjQtSDQviDQv9GA0LjQvdGP0YLQuNC4?=\n' + ' =?utf-8?b?INCyINGA0LDQsdC+0YLRgyDQvtCx0YDQsNGJ0LXQvdC40Y8g0LTQu9GPINC/0L4=?=\n' + ' =?utf-8?b?0LTQutC70Y7Rh9C10L3QuNGPINGD0YHQu9GD0LPQuA==?=\n\n').encode('ascii') + msg = EmailMessage() + msg['Subject'] = source + s = io.BytesIO() + g = BytesGenerator(s) + g.flatten(msg) + self.assertEqual(s.getvalue(), expected) + + def test_defaults_handle_spaces_when_encoded_words_is_folded_in_middle(self): + source = ('A very long long long long long long long long long long long long ' + 'long long long long long long long long long long long súmmäry') + expected = ('Subject: A very long long long long long long long long long long long long\n' + ' long long long long long long long long long long long =?utf-8?q?s=C3=BAmm?=\n' + ' =?utf-8?q?=C3=A4ry?=\n\n').encode('ascii') + msg = EmailMessage() + msg['Subject'] = source + s = io.BytesIO() + g = BytesGenerator(s) + g.flatten(msg) + self.assertEqual(s.getvalue(), expected) + + def test_defaults_handle_spaces_at_start_of_subject(self): + source = " Уведомление" + expected = b"Subject: =?utf-8?b?0KPQstC10LTQvtC80LvQtdC90LjQtQ==?=\n\n" + msg = EmailMessage() + msg['Subject'] = source + s = io.BytesIO() + g = BytesGenerator(s) + g.flatten(msg) + self.assertEqual(s.getvalue(), expected) + + def test_defaults_handle_spaces_at_start_of_continuation_line(self): + source = " ф ффффффффффффффффффф ф ф" + expected = (b"Subject: " + b"=?utf-8?b?0YQg0YTRhNGE0YTRhNGE0YTRhNGE0YTRhNGE0YTRhNGE0YTRhNGE0YQ=?=\n" + b" =?utf-8?b?INGEINGE?=\n\n") + msg = EmailMessage() + msg['Subject'] = source + s = io.BytesIO() + g = BytesGenerator(s) + g.flatten(msg) + self.assertEqual(s.getvalue(), expected) + + def test_cte_type_7bit_handles_unknown_8bit(self): + source = ("Subject: Maintenant je vous présente mon " + "collègue\n\n").encode('utf-8') + expected = ('Subject: Maintenant je vous =?unknown-8bit?q?' + 'pr=C3=A9sente_mon_coll=C3=A8gue?=\n\n').encode('ascii') + msg = message_from_bytes(source) + s = io.BytesIO() + g = BytesGenerator(s, policy=self.policy.clone(cte_type='7bit')) + g.flatten(msg) + self.assertEqual(s.getvalue(), expected) + + def test_cte_type_7bit_transforms_8bit_cte(self): + source = textwrap.dedent("""\ + From: foo@bar.com + To: Dinsdale + Subject: Nudge nudge, wink, wink + Mime-Version: 1.0 + Content-Type: text/plain; charset="latin-1" + Content-Transfer-Encoding: 8bit + + oh là là, know what I mean, know what I mean? + """).encode('latin1') + msg = message_from_bytes(source) + expected = textwrap.dedent("""\ + From: foo@bar.com + To: Dinsdale + Subject: Nudge nudge, wink, wink + Mime-Version: 1.0 + Content-Type: text/plain; charset="iso-8859-1" + Content-Transfer-Encoding: quoted-printable + + oh l=E0 l=E0, know what I mean, know what I mean? + """).encode('ascii') + s = io.BytesIO() + g = BytesGenerator(s, policy=self.policy.clone(cte_type='7bit', + linesep='\n')) + g.flatten(msg) + self.assertEqual(s.getvalue(), expected) + + def test_smtputf8_policy(self): + msg = EmailMessage() + msg['From'] = "Páolo " + msg['To'] = 'Dinsdale' + msg['Subject'] = 'Nudge nudge, wink, wink \u1F609' + msg.set_content("oh là là, know what I mean, know what I mean?") + expected = textwrap.dedent("""\ + From: Páolo + To: Dinsdale + Subject: Nudge nudge, wink, wink \u1F609 + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: 8bit + MIME-Version: 1.0 + + oh là là, know what I mean, know what I mean? + """).encode('utf-8').replace(b'\n', b'\r\n') + s = io.BytesIO() + g = BytesGenerator(s, policy=policy.SMTPUTF8) + g.flatten(msg) + self.assertEqual(s.getvalue(), expected) + + def test_smtp_policy(self): + msg = EmailMessage() + msg["From"] = Address(addr_spec="foo@bar.com", display_name="Páolo") + msg["To"] = Address(addr_spec="bar@foo.com", display_name="Dinsdale") + msg["Subject"] = "Nudge nudge, wink, wink" + msg.set_content("oh boy, know what I mean, know what I mean?") + expected = textwrap.dedent("""\ + From: =?utf-8?q?P=C3=A1olo?= + To: Dinsdale + Subject: Nudge nudge, wink, wink + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: 7bit + MIME-Version: 1.0 + + oh boy, know what I mean, know what I mean? + """).encode().replace(b"\n", b"\r\n") + s = io.BytesIO() + g = BytesGenerator(s, policy=policy.SMTP) + g.flatten(msg) + self.assertEqual(s.getvalue(), expected) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test_headerregistry.py b/Lib/test/test_email/test_headerregistry.py new file mode 100644 index 00000000000..d2c571299bc --- /dev/null +++ b/Lib/test/test_email/test_headerregistry.py @@ -0,0 +1,1822 @@ +import datetime +import textwrap +import unittest +from email import errors +from email import policy +from email.message import Message +from test.test_email import TestEmailBase, parameterize +from email import headerregistry +from email.headerregistry import Address, Group +from email.header import decode_header +from test.support import ALWAYS_EQ + + +DITTO = object() + + +class TestHeaderRegistry(TestEmailBase): + + def test_arbitrary_name_unstructured(self): + factory = headerregistry.HeaderRegistry() + h = factory('foobar', 'test') + self.assertIsInstance(h, headerregistry.BaseHeader) + self.assertIsInstance(h, headerregistry.UnstructuredHeader) + + def test_name_case_ignored(self): + factory = headerregistry.HeaderRegistry() + # Whitebox check that test is valid + self.assertNotIn('Subject', factory.registry) + h = factory('Subject', 'test') + self.assertIsInstance(h, headerregistry.BaseHeader) + self.assertIsInstance(h, headerregistry.UniqueUnstructuredHeader) + + class FooBase: + def __init__(self, *args, **kw): + pass + + def test_override_default_base_class(self): + factory = headerregistry.HeaderRegistry(base_class=self.FooBase) + h = factory('foobar', 'test') + self.assertIsInstance(h, self.FooBase) + self.assertIsInstance(h, headerregistry.UnstructuredHeader) + + class FooDefault: + parse = headerregistry.UnstructuredHeader.parse + + def test_override_default_class(self): + factory = headerregistry.HeaderRegistry(default_class=self.FooDefault) + h = factory('foobar', 'test') + self.assertIsInstance(h, headerregistry.BaseHeader) + self.assertIsInstance(h, self.FooDefault) + + def test_override_default_class_only_overrides_default(self): + factory = headerregistry.HeaderRegistry(default_class=self.FooDefault) + h = factory('subject', 'test') + self.assertIsInstance(h, headerregistry.BaseHeader) + self.assertIsInstance(h, headerregistry.UniqueUnstructuredHeader) + + def test_dont_use_default_map(self): + factory = headerregistry.HeaderRegistry(use_default_map=False) + h = factory('subject', 'test') + self.assertIsInstance(h, headerregistry.BaseHeader) + self.assertIsInstance(h, headerregistry.UnstructuredHeader) + + def test_map_to_type(self): + factory = headerregistry.HeaderRegistry() + h1 = factory('foobar', 'test') + factory.map_to_type('foobar', headerregistry.UniqueUnstructuredHeader) + h2 = factory('foobar', 'test') + self.assertIsInstance(h1, headerregistry.BaseHeader) + self.assertIsInstance(h1, headerregistry.UnstructuredHeader) + self.assertIsInstance(h2, headerregistry.BaseHeader) + self.assertIsInstance(h2, headerregistry.UniqueUnstructuredHeader) + + +class TestHeaderBase(TestEmailBase): + + factory = headerregistry.HeaderRegistry() + + def make_header(self, name, value): + return self.factory(name, value) + + +class TestBaseHeaderFeatures(TestHeaderBase): + + def test_str(self): + h = self.make_header('subject', 'this is a test') + self.assertIsInstance(h, str) + self.assertEqual(h, 'this is a test') + self.assertEqual(str(h), 'this is a test') + + def test_substr(self): + h = self.make_header('subject', 'this is a test') + self.assertEqual(h[5:7], 'is') + + def test_has_name(self): + h = self.make_header('subject', 'this is a test') + self.assertEqual(h.name, 'subject') + + def _test_attr_ro(self, attr): + h = self.make_header('subject', 'this is a test') + with self.assertRaises(AttributeError): + setattr(h, attr, 'foo') + + def test_name_read_only(self): + self._test_attr_ro('name') + + def test_defects_read_only(self): + self._test_attr_ro('defects') + + def test_defects_is_tuple(self): + h = self.make_header('subject', 'this is a test') + self.assertEqual(len(h.defects), 0) + self.assertIsInstance(h.defects, tuple) + # Make sure it is still true when there are defects. + h = self.make_header('date', '') + self.assertEqual(len(h.defects), 1) + self.assertIsInstance(h.defects, tuple) + + # XXX: FIXME + #def test_CR_in_value(self): + # # XXX: this also re-raises the issue of embedded headers, + # # need test and solution for that. + # value = '\r'.join(['this is', ' a test']) + # h = self.make_header('subject', value) + # self.assertEqual(h, value) + # self.assertDefectsEqual(h.defects, [errors.ObsoleteHeaderDefect]) + + +@parameterize +class TestUnstructuredHeader(TestHeaderBase): + + def string_as_value(self, + source, + decoded, + *args): + # TODO: RUSTPYTHON; RustPython currently does not support non-utf8 encoding + if source == '=?gb2312?b?1eLKx9bQzsSy4srUo6E=?=': + raise unittest.SkipTest("TODO: RUSTPYTHON; RustPython currently does not support non-utf8 encoding") + # RUSTPYTHON: End + # ------------------------------------------------------------------ + l = len(args) + defects = args[0] if l>0 else [] + header = 'Subject:' + (' ' if source else '') + folded = header + (args[1] if l>1 else source) + '\n' + h = self.make_header('Subject', source) + self.assertEqual(h, decoded) + self.assertDefectsEqual(h.defects, defects) + self.assertEqual(h.fold(policy=policy.default), folded) + + string_params = { + + 'rfc2047_simple_quopri': ( + '=?utf-8?q?this_is_a_test?=', + 'this is a test', + [], + 'this is a test'), + + 'rfc2047_gb2312_base64': ( + '=?gb2312?b?1eLKx9bQzsSy4srUo6E=?=', + '\u8fd9\u662f\u4e2d\u6587\u6d4b\u8bd5\uff01', + [], + '=?utf-8?b?6L+Z5piv5Lit5paH5rWL6K+V77yB?='), + + 'rfc2047_simple_nonascii_quopri': ( + '=?utf-8?q?=C3=89ric?=', + 'Éric'), + + 'rfc2047_quopri_with_regular_text': ( + 'The =?utf-8?q?=C3=89ric=2C?= Himself', + 'The Éric, Himself'), + + } + + +@parameterize +class TestDateHeader(TestHeaderBase): + + datestring = 'Sun, 23 Sep 2001 20:10:55 -0700' + utcoffset = datetime.timedelta(hours=-7) + tz = datetime.timezone(utcoffset) + dt = datetime.datetime(2001, 9, 23, 20, 10, 55, tzinfo=tz) + + def test_parse_date(self): + h = self.make_header('date', self.datestring) + self.assertEqual(h, self.datestring) + self.assertEqual(h.datetime, self.dt) + self.assertEqual(h.datetime.utcoffset(), self.utcoffset) + self.assertEqual(h.defects, ()) + + def test_set_from_datetime(self): + h = self.make_header('date', self.dt) + self.assertEqual(h, self.datestring) + self.assertEqual(h.datetime, self.dt) + self.assertEqual(h.defects, ()) + + def test_date_header_properties(self): + h = self.make_header('date', self.datestring) + self.assertIsInstance(h, headerregistry.UniqueDateHeader) + self.assertEqual(h.max_count, 1) + self.assertEqual(h.defects, ()) + + def test_resent_date_header_properties(self): + h = self.make_header('resent-date', self.datestring) + self.assertIsInstance(h, headerregistry.DateHeader) + self.assertEqual(h.max_count, None) + self.assertEqual(h.defects, ()) + + def test_no_value_is_defect(self): + h = self.make_header('date', '') + self.assertEqual(len(h.defects), 1) + self.assertIsInstance(h.defects[0], errors.HeaderMissingRequiredValue) + + def test_invalid_date_format(self): + s = 'Not a date header' + h = self.make_header('date', s) + self.assertEqual(h, s) + self.assertIsNone(h.datetime) + self.assertEqual(len(h.defects), 1) + self.assertIsInstance(h.defects[0], errors.InvalidDateDefect) + + def test_invalid_date_value(self): + s = 'Tue, 06 Jun 2017 27:39:33 +0600' + h = self.make_header('date', s) + self.assertEqual(h, s) + self.assertIsNone(h.datetime) + self.assertEqual(len(h.defects), 1) + self.assertIsInstance(h.defects[0], errors.InvalidDateDefect) + + def test_datetime_read_only(self): + h = self.make_header('date', self.datestring) + with self.assertRaises(AttributeError): + h.datetime = 'foo' + + def test_set_date_header_from_datetime(self): + m = Message(policy=policy.default) + m['Date'] = self.dt + self.assertEqual(m['Date'], self.datestring) + self.assertEqual(m['Date'].datetime, self.dt) + + +@parameterize +class TestContentTypeHeader(TestHeaderBase): + + def content_type_as_value(self, + source, + content_type, + maintype, + subtype, + *args): + l = len(args) + parmdict = args[0] if l>0 else {} + defects = args[1] if l>1 else [] + decoded = args[2] if l>2 and args[2] is not DITTO else source + header = 'Content-Type:' + ' ' if source else '' + folded = args[3] if l>3 else header + decoded + '\n' + h = self.make_header('Content-Type', source) + self.assertEqual(h.content_type, content_type) + self.assertEqual(h.maintype, maintype) + self.assertEqual(h.subtype, subtype) + self.assertEqual(h.params, parmdict) + with self.assertRaises(TypeError): + h.params['abc'] = 'xyz' # make sure params is read-only. + self.assertDefectsEqual(h.defects, defects) + self.assertEqual(h, decoded) + self.assertEqual(h.fold(policy=policy.default), folded) + + content_type_params = { + + # Examples from RFC 2045. + + 'RFC_2045_1': ( + 'text/plain; charset=us-ascii (Plain text)', + 'text/plain', + 'text', + 'plain', + {'charset': 'us-ascii'}, + [], + 'text/plain; charset="us-ascii"'), + + 'RFC_2045_2': ( + 'text/plain; charset=us-ascii', + 'text/plain', + 'text', + 'plain', + {'charset': 'us-ascii'}, + [], + 'text/plain; charset="us-ascii"'), + + 'RFC_2045_3': ( + 'text/plain; charset="us-ascii"', + 'text/plain', + 'text', + 'plain', + {'charset': 'us-ascii'}), + + # RFC 2045 5.2 says syntactically invalid values are to be treated as + # text/plain. + + 'no_subtype_in_content_type': ( + 'text/', + 'text/plain', + 'text', + 'plain', + {}, + [errors.InvalidHeaderDefect]), + + 'no_slash_in_content_type': ( + 'foo', + 'text/plain', + 'text', + 'plain', + {}, + [errors.InvalidHeaderDefect]), + + 'junk_text_in_content_type': ( + '', + 'text/plain', + 'text', + 'plain', + {}, + [errors.InvalidHeaderDefect]), + + 'too_many_slashes_in_content_type': ( + 'image/jpeg/foo', + 'text/plain', + 'text', + 'plain', + {}, + [errors.InvalidHeaderDefect]), + + # But unknown names are OK. We could make non-IANA names a defect, but + # by not doing so we make ourselves future proof. The fact that they + # are unknown will be detectable by the fact that they don't appear in + # the mime_registry...and the application is free to extend that list + # to handle them even if the core library doesn't. + + 'unknown_content_type': ( + 'bad/names', + 'bad/names', + 'bad', + 'names'), + + # The content type is case insensitive, and CFWS is ignored. + + 'mixed_case_content_type': ( + 'ImAge/JPeg', + 'image/jpeg', + 'image', + 'jpeg'), + + 'spaces_in_content_type': ( + ' text / plain ', + 'text/plain', + 'text', + 'plain'), + + 'cfws_in_content_type': ( + '(foo) text (bar)/(baz)plain(stuff)', + 'text/plain', + 'text', + 'plain'), + + # test some parameters (more tests could be added for parameters + # associated with other content types, but since parameter parsing is + # generic they would be redundant for the current implementation). + + 'charset_param': ( + 'text/plain; charset="utf-8"', + 'text/plain', + 'text', + 'plain', + {'charset': 'utf-8'}), + + 'capitalized_charset': ( + 'text/plain; charset="US-ASCII"', + 'text/plain', + 'text', + 'plain', + {'charset': 'US-ASCII'}), + + 'unknown_charset': ( + 'text/plain; charset="fOo"', + 'text/plain', + 'text', + 'plain', + {'charset': 'fOo'}), + + 'capitalized_charset_param_name_and_comment': ( + 'text/plain; (interjection) Charset="utf-8"', + 'text/plain', + 'text', + 'plain', + {'charset': 'utf-8'}, + [], + # Should the parameter name be lowercased here? + 'text/plain; Charset="utf-8"'), + + # Since this is pretty much the ur-mimeheader, we'll put all the tests + # that exercise the parameter parsing and formatting here. Note that + # when we refold we may canonicalize, so things like whitespace, + # quoting, and rfc2231 encoding may change from what was in the input + # header. + + 'unquoted_param_value': ( + 'text/plain; title=foo', + 'text/plain', + 'text', + 'plain', + {'title': 'foo'}, + [], + 'text/plain; title="foo"', + ), + + 'param_value_with_tspecials': ( + 'text/plain; title="(bar)foo blue"', + 'text/plain', + 'text', + 'plain', + {'title': '(bar)foo blue'}), + + 'param_with_extra_quoted_whitespace': ( + 'text/plain; title=" a loong way \t home "', + 'text/plain', + 'text', + 'plain', + {'title': ' a loong way \t home '}), + + 'bad_params': ( + 'blarg; baz; boo', + 'text/plain', + 'text', + 'plain', + {'baz': '', 'boo': ''}, + [errors.InvalidHeaderDefect]*3), + + 'spaces_around_param_equals': ( + 'Multipart/mixed; boundary = "CPIMSSMTPC06p5f3tG"', + 'multipart/mixed', + 'multipart', + 'mixed', + {'boundary': 'CPIMSSMTPC06p5f3tG'}, + [], + 'Multipart/mixed; boundary="CPIMSSMTPC06p5f3tG"', + ), + + 'spaces_around_semis': ( + ('image/jpeg; name="wibble.JPG" ; x-mac-type="4A504547" ; ' + 'x-mac-creator="474B4F4E"'), + 'image/jpeg', + 'image', + 'jpeg', + {'name': 'wibble.JPG', + 'x-mac-type': '4A504547', + 'x-mac-creator': '474B4F4E'}, + [], + ('image/jpeg; name="wibble.JPG"; x-mac-type="4A504547"; ' + 'x-mac-creator="474B4F4E"'), + ('Content-Type: image/jpeg; name="wibble.JPG";' + ' x-mac-type="4A504547";\n' + ' x-mac-creator="474B4F4E"\n'), + ), + + 'lots_of_mime_params': ( + ('image/jpeg; name="wibble.JPG"; x-mac-type="4A504547"; ' + 'x-mac-creator="474B4F4E"; x-extrastuff="make it longer"'), + 'image/jpeg', + 'image', + 'jpeg', + {'name': 'wibble.JPG', + 'x-mac-type': '4A504547', + 'x-mac-creator': '474B4F4E', + 'x-extrastuff': 'make it longer'}, + [], + ('image/jpeg; name="wibble.JPG"; x-mac-type="4A504547"; ' + 'x-mac-creator="474B4F4E"; x-extrastuff="make it longer"'), + # In this case the whole of the MimeParameters does *not* fit + # one one line, so we break at a lower syntactic level. + ('Content-Type: image/jpeg; name="wibble.JPG";' + ' x-mac-type="4A504547";\n' + ' x-mac-creator="474B4F4E"; x-extrastuff="make it longer"\n'), + ), + + 'semis_inside_quotes': ( + 'image/jpeg; name="Jim&&Jill"', + 'image/jpeg', + 'image', + 'jpeg', + {'name': 'Jim&&Jill'}), + + 'single_quotes_inside_quotes': ( + 'image/jpeg; name="Jim \'Bob\' Jill"', + 'image/jpeg', + 'image', + 'jpeg', + {'name': "Jim 'Bob' Jill"}), + + 'double_quotes_inside_quotes': ( + r'image/jpeg; name="Jim \"Bob\" Jill"', + 'image/jpeg', + 'image', + 'jpeg', + {'name': 'Jim "Bob" Jill'}, + [], + r'image/jpeg; name="Jim \"Bob\" Jill"'), + + 'non_ascii_in_params': ( + ('foo\xa7/bar; b\xa7r=two; ' + 'baz=thr\xa7e'.encode('latin-1').decode('us-ascii', + 'surrogateescape')), + 'foo\uFFFD/bar', + 'foo\uFFFD', + 'bar', + {'b\uFFFDr': 'two', 'baz': 'thr\uFFFDe'}, + [errors.UndecodableBytesDefect]*3, + 'foo�/bar; b�r="two"; baz="thr�e"', + # XXX Two bugs here: the mime type is not allowed to be an encoded + # word, and we shouldn't be emitting surrogates in the parameter + # names. But I don't know what the behavior should be here, so I'm + # punting for now. In practice this is unlikely to be encountered + # since headers with binary in them only come from a binary source + # and are almost certain to be re-emitted without refolding. + 'Content-Type: =?unknown-8bit?q?foo=A7?=/bar; b\udca7r="two";\n' + " baz*=unknown-8bit''thr%A7e\n", + ), + + # RFC 2231 parameter tests. + + 'rfc2231_segmented_normal_values': ( + 'image/jpeg; name*0="abc"; name*1=".html"', + 'image/jpeg', + 'image', + 'jpeg', + {'name': "abc.html"}, + [], + 'image/jpeg; name="abc.html"'), + + 'quotes_inside_rfc2231_value': ( + r'image/jpeg; bar*0="baz\"foobar"; bar*1="\"baz"', + 'image/jpeg', + 'image', + 'jpeg', + {'bar': 'baz"foobar"baz'}, + [], + r'image/jpeg; bar="baz\"foobar\"baz"'), + + 'non_ascii_rfc2231_value': ( + ('text/plain; charset=us-ascii; ' + "title*=us-ascii'en'This%20is%20" + 'not%20f\xa7n').encode('latin-1').decode('us-ascii', + 'surrogateescape'), + 'text/plain', + 'text', + 'plain', + {'charset': 'us-ascii', 'title': 'This is not f\uFFFDn'}, + [errors.UndecodableBytesDefect], + 'text/plain; charset="us-ascii"; title="This is not f�n"', + 'Content-Type: text/plain; charset="us-ascii";\n' + " title*=unknown-8bit''This%20is%20not%20f%A7n\n", + ), + + 'rfc2231_encoded_charset': ( + 'text/plain; charset*=ansi-x3.4-1968\'\'us-ascii', + 'text/plain', + 'text', + 'plain', + {'charset': 'us-ascii'}, + [], + 'text/plain; charset="us-ascii"'), + + # This follows the RFC: no double quotes around encoded values. + 'rfc2231_encoded_no_double_quotes': ( + ("text/plain;" + "\tname*0*=''This%20is%20;" + "\tname*1*=%2A%2A%2Afun%2A%2A%2A%20;" + '\tname*2="is it not.pdf"'), + 'text/plain', + 'text', + 'plain', + {'name': 'This is ***fun*** is it not.pdf'}, + [], + 'text/plain; name="This is ***fun*** is it not.pdf"', + ), + + # Make sure we also handle it if there are spurious double quotes. + 'rfc2231_encoded_with_double_quotes': ( + ("text/plain;" + '\tname*0*="us-ascii\'\'This%20is%20even%20more%20";' + '\tname*1*="%2A%2A%2Afun%2A%2A%2A%20";' + '\tname*2="is it not.pdf"'), + 'text/plain', + 'text', + 'plain', + {'name': 'This is even more ***fun*** is it not.pdf'}, + [errors.InvalidHeaderDefect]*2, + 'text/plain; name="This is even more ***fun*** is it not.pdf"', + ), + + 'rfc2231_single_quote_inside_double_quotes': ( + ('text/plain; charset=us-ascii;' + '\ttitle*0*="us-ascii\'en\'This%20is%20really%20";' + '\ttitle*1*="%2A%2A%2Afun%2A%2A%2A%20";' + '\ttitle*2="isn\'t it!"'), + 'text/plain', + 'text', + 'plain', + {'charset': 'us-ascii', 'title': "This is really ***fun*** isn't it!"}, + [errors.InvalidHeaderDefect]*2, + ('text/plain; charset="us-ascii"; ' + 'title="This is really ***fun*** isn\'t it!"'), + ('Content-Type: text/plain; charset="us-ascii";\n' + ' title="This is really ***fun*** isn\'t it!"\n'), + ), + + 'rfc2231_single_quote_in_value_with_charset_and_lang': ( + ('application/x-foo;' + "\tname*0*=\"us-ascii'en-us'Frank's\"; name*1*=\" Document\""), + 'application/x-foo', + 'application', + 'x-foo', + {'name': "Frank's Document"}, + [errors.InvalidHeaderDefect]*2, + 'application/x-foo; name="Frank\'s Document"', + ), + + 'rfc2231_single_quote_in_non_encoded_value': ( + ('application/x-foo;' + "\tname*0=\"us-ascii'en-us'Frank's\"; name*1=\" Document\""), + 'application/x-foo', + 'application', + 'x-foo', + {'name': "us-ascii'en-us'Frank's Document"}, + [], + 'application/x-foo; name="us-ascii\'en-us\'Frank\'s Document"', + ), + + 'rfc2231_no_language_or_charset': ( + 'text/plain; NAME*0*=english_is_the_default.html', + 'text/plain', + 'text', + 'plain', + {'name': 'english_is_the_default.html'}, + [errors.InvalidHeaderDefect], + 'text/plain; NAME="english_is_the_default.html"'), + + 'rfc2231_encoded_no_charset': ( + ("text/plain;" + '\tname*0*="\'\'This%20is%20even%20more%20";' + '\tname*1*="%2A%2A%2Afun%2A%2A%2A%20";' + '\tname*2="is it.pdf"'), + 'text/plain', + 'text', + 'plain', + {'name': 'This is even more ***fun*** is it.pdf'}, + [errors.InvalidHeaderDefect]*2, + 'text/plain; name="This is even more ***fun*** is it.pdf"', + ), + + 'rfc2231_partly_encoded': ( + ("text/plain;" + '\tname*0*="\'\'This%20is%20even%20more%20";' + '\tname*1*="%2A%2A%2Afun%2A%2A%2A%20";' + '\tname*2="is it.pdf"'), + 'text/plain', + 'text', + 'plain', + {'name': 'This is even more ***fun*** is it.pdf'}, + [errors.InvalidHeaderDefect]*2, + 'text/plain; name="This is even more ***fun*** is it.pdf"', + ), + + 'rfc2231_partly_encoded_2': ( + ("text/plain;" + '\tname*0*="\'\'This%20is%20even%20more%20";' + '\tname*1="%2A%2A%2Afun%2A%2A%2A%20";' + '\tname*2="is it.pdf"'), + 'text/plain', + 'text', + 'plain', + {'name': 'This is even more %2A%2A%2Afun%2A%2A%2A%20is it.pdf'}, + [errors.InvalidHeaderDefect], + ('text/plain;' + ' name="This is even more %2A%2A%2Afun%2A%2A%2A%20is it.pdf"'), + ('Content-Type: text/plain;\n' + ' name="This is even more %2A%2A%2Afun%2A%2A%2A%20is' + ' it.pdf"\n'), + ), + + 'rfc2231_unknown_charset_treated_as_ascii': ( + "text/plain; name*0*=bogus'xx'ascii_is_the_default", + 'text/plain', + 'text', + 'plain', + {'name': 'ascii_is_the_default'}, + [], + 'text/plain; name="ascii_is_the_default"'), + + 'rfc2231_bad_character_in_charset_parameter_value': ( + "text/plain; charset*=ascii''utf-8%F1%F2%F3", + 'text/plain', + 'text', + 'plain', + {'charset': 'utf-8\uFFFD\uFFFD\uFFFD'}, + [errors.UndecodableBytesDefect], + 'text/plain; charset="utf-8\uFFFD\uFFFD\uFFFD"', + "Content-Type: text/plain;" + " charset*=unknown-8bit''utf-8%F1%F2%F3\n", + ), + + 'rfc2231_utf8_in_supposedly_ascii_charset_parameter_value': ( + "text/plain; charset*=ascii''utf-8%E2%80%9D", + 'text/plain', + 'text', + 'plain', + {'charset': 'utf-8”'}, + [errors.UndecodableBytesDefect], + 'text/plain; charset="utf-8”"', + # XXX Should folding change the charset to utf8? Currently it just + # reproduces the original, which is arguably fine. + "Content-Type: text/plain;" + " charset*=unknown-8bit''utf-8%E2%80%9D\n", + ), + + 'rfc2231_nonascii_in_charset_of_charset_parameter_value': ( + "text/plain; charset*=utf-8”''utf-8%E2%80%9D", + 'text/plain', + 'text', + 'plain', + {'charset': 'utf-8”'}, + [], + 'text/plain; charset="utf-8”"', + "Content-Type: text/plain;" + " charset*=utf-8''utf-8%E2%80%9D\n", + ), + + 'rfc2231_encoded_then_unencoded_segments': ( + ('application/x-foo;' + '\tname*0*="us-ascii\'en-us\'My";' + '\tname*1=" Document";' + '\tname*2=" For You"'), + 'application/x-foo', + 'application', + 'x-foo', + {'name': 'My Document For You'}, + [errors.InvalidHeaderDefect], + 'application/x-foo; name="My Document For You"', + ), + + # My reading of the RFC is that this is an invalid header. The RFC + # says that if charset and language information is given, the first + # segment *must* be encoded. + 'rfc2231_unencoded_then_encoded_segments': ( + ('application/x-foo;' + '\tname*0=us-ascii\'en-us\'My;' + '\tname*1*=" Document";' + '\tname*2*=" For You"'), + 'application/x-foo', + 'application', + 'x-foo', + {'name': 'My Document For You'}, + [errors.InvalidHeaderDefect]*3, + 'application/x-foo; name="My Document For You"', + ), + + # XXX: I would say this one should default to ascii/en for the + # "encoded" segment, since the first segment is not encoded and is + # in double quotes, making the value a valid non-encoded string. The + # old parser decodes this just like the previous case, which may be the + # better Postel rule, but could equally result in borking headers that + # intentionally have quoted quotes in them. We could get this 98% + # right if we treat it as a quoted string *unless* it matches the + # charset'lang'value pattern exactly *and* there is at least one + # encoded segment. Implementing that algorithm will require some + # refactoring, so I haven't done it (yet). + 'rfc2231_quoted_unencoded_then_encoded_segments': ( + ('application/x-foo;' + '\tname*0="us-ascii\'en-us\'My";' + '\tname*1*=" Document";' + '\tname*2*=" For You"'), + 'application/x-foo', + 'application', + 'x-foo', + {'name': "us-ascii'en-us'My Document For You"}, + [errors.InvalidHeaderDefect]*2, + 'application/x-foo; name="us-ascii\'en-us\'My Document For You"', + ), + + # Make sure our folding algorithm produces multiple sections correctly. + # We could mix encoded and non-encoded segments, but we don't, we just + # make them all encoded. It might be worth fixing that, since the + # sections can get used for wrapping ascii text. + 'rfc2231_folded_segments_correctly_formatted': ( + ('application/x-foo;' + '\tname="' + "with spaces"*8 + '"'), + 'application/x-foo', + 'application', + 'x-foo', + {'name': "with spaces"*8}, + [], + 'application/x-foo; name="' + "with spaces"*8 + '"', + "Content-Type: application/x-foo;\n" + " name*0*=us-ascii''with%20spaceswith%20spaceswith%20spaceswith" + "%20spaceswith;\n" + " name*1*=%20spaceswith%20spaceswith%20spaceswith%20spaces\n" + ), + + } + + +@parameterize +class TestContentTransferEncoding(TestHeaderBase): + + def cte_as_value(self, + source, + cte, + *args): + l = len(args) + defects = args[0] if l>0 else [] + decoded = args[1] if l>1 and args[1] is not DITTO else source + header = 'Content-Transfer-Encoding:' + ' ' if source else '' + folded = args[2] if l>2 else header + source + '\n' + h = self.make_header('Content-Transfer-Encoding', source) + self.assertEqual(h.cte, cte) + self.assertDefectsEqual(h.defects, defects) + self.assertEqual(h, decoded) + self.assertEqual(h.fold(policy=policy.default), folded) + + cte_params = { + + 'RFC_2183_1': ( + 'base64', + 'base64',), + + 'no_value': ( + '', + '7bit', + [errors.HeaderMissingRequiredValue], + '', + 'Content-Transfer-Encoding:\n', + ), + + 'junk_after_cte': ( + '7bit and a bunch more', + '7bit', + [errors.InvalidHeaderDefect]), + + 'extra_space_after_cte': ( + 'base64 ', + 'base64', + []), + + } + + +@parameterize +class TestContentDisposition(TestHeaderBase): + + def content_disp_as_value(self, + source, + content_disposition, + *args): + l = len(args) + parmdict = args[0] if l>0 else {} + defects = args[1] if l>1 else [] + decoded = args[2] if l>2 and args[2] is not DITTO else source + header = 'Content-Disposition:' + ' ' if source else '' + folded = args[3] if l>3 else header + source + '\n' + h = self.make_header('Content-Disposition', source) + self.assertEqual(h.content_disposition, content_disposition) + self.assertEqual(h.params, parmdict) + self.assertDefectsEqual(h.defects, defects) + self.assertEqual(h, decoded) + self.assertEqual(h.fold(policy=policy.default), folded) + + content_disp_params = { + + # Examples from RFC 2183. + + 'RFC_2183_1': ( + 'inline', + 'inline',), + + 'RFC_2183_2': ( + ('attachment; filename=genome.jpeg;' + ' modification-date="Wed, 12 Feb 1997 16:29:51 -0500";'), + 'attachment', + {'filename': 'genome.jpeg', + 'modification-date': 'Wed, 12 Feb 1997 16:29:51 -0500'}, + [], + ('attachment; filename="genome.jpeg"; ' + 'modification-date="Wed, 12 Feb 1997 16:29:51 -0500"'), + ('Content-Disposition: attachment; filename="genome.jpeg";\n' + ' modification-date="Wed, 12 Feb 1997 16:29:51 -0500"\n'), + ), + + 'no_value': ( + '', + None, + {}, + [errors.HeaderMissingRequiredValue], + '', + 'Content-Disposition:\n'), + + 'invalid_value': ( + 'ab./k', + 'ab.', + {}, + [errors.InvalidHeaderDefect]), + + 'invalid_value_with_params': ( + 'ab./k; filename="foo"', + 'ab.', + {'filename': 'foo'}, + [errors.InvalidHeaderDefect]), + + 'invalid_parameter_value_with_fws_between_ew': ( + 'attachment; filename="=?UTF-8?Q?Schulbesuchsbest=C3=A4ttigung=2E?=' + ' =?UTF-8?Q?pdf?="', + 'attachment', + {'filename': 'Schulbesuchsbestättigung.pdf'}, + [errors.InvalidHeaderDefect]*3, + ('attachment; filename="Schulbesuchsbestättigung.pdf"'), + ('Content-Disposition: attachment;\n' + ' filename*=utf-8\'\'Schulbesuchsbest%C3%A4ttigung.pdf\n'), + ), + + 'parameter_value_with_fws_between_tokens': ( + 'attachment; filename="File =?utf-8?q?Name?= With Spaces.pdf"', + 'attachment', + {'filename': 'File Name With Spaces.pdf'}, + [errors.InvalidHeaderDefect], + 'attachment; filename="File Name With Spaces.pdf"', + ('Content-Disposition: attachment; filename="File Name With Spaces.pdf"\n'), + ) + } + + +@parameterize +class TestMIMEVersionHeader(TestHeaderBase): + + def version_string_as_MIME_Version(self, + source, + decoded, + version, + major, + minor, + defects): + h = self.make_header('MIME-Version', source) + self.assertEqual(h, decoded) + self.assertEqual(h.version, version) + self.assertEqual(h.major, major) + self.assertEqual(h.minor, minor) + self.assertDefectsEqual(h.defects, defects) + if source: + source = ' ' + source + self.assertEqual(h.fold(policy=policy.default), + 'MIME-Version:' + source + '\n') + + version_string_params = { + + # Examples from the RFC. + + 'RFC_2045_1': ( + '1.0', + '1.0', + '1.0', + 1, + 0, + []), + + 'RFC_2045_2': ( + '1.0 (produced by MetaSend Vx.x)', + '1.0 (produced by MetaSend Vx.x)', + '1.0', + 1, + 0, + []), + + 'RFC_2045_3': ( + '(produced by MetaSend Vx.x) 1.0', + '(produced by MetaSend Vx.x) 1.0', + '1.0', + 1, + 0, + []), + + 'RFC_2045_4': ( + '1.(produced by MetaSend Vx.x)0', + '1.(produced by MetaSend Vx.x)0', + '1.0', + 1, + 0, + []), + + # Other valid values. + + '1_1': ( + '1.1', + '1.1', + '1.1', + 1, + 1, + []), + + '2_1': ( + '2.1', + '2.1', + '2.1', + 2, + 1, + []), + + 'whitespace': ( + '1 .0', + '1 .0', + '1.0', + 1, + 0, + []), + + 'leading_trailing_whitespace_ignored': ( + ' 1.0 ', + ' 1.0 ', + '1.0', + 1, + 0, + []), + + # Recoverable invalid values. We can recover here only because we + # already have a valid value by the time we encounter the garbage. + # Anywhere else, and we don't know where the garbage ends. + + 'non_comment_garbage_after': ( + '1.0 ', + '1.0 ', + '1.0', + 1, + 0, + [errors.InvalidHeaderDefect]), + + # Unrecoverable invalid values. We *could* apply more heuristics to + # get something out of the first two, but doing so is not worth the + # effort. + + 'non_comment_garbage_before': ( + ' 1.0', + ' 1.0', + None, + None, + None, + [errors.InvalidHeaderDefect]), + + 'non_comment_garbage_inside': ( + '1.0', + '1.0', + None, + None, + None, + [errors.InvalidHeaderDefect]), + + 'two_periods': ( + '1..0', + '1..0', + None, + None, + None, + [errors.InvalidHeaderDefect]), + + '2_x': ( + '2.x', + '2.x', + None, # This could be 2, but it seems safer to make it None. + None, + None, + [errors.InvalidHeaderDefect]), + + 'foo': ( + 'foo', + 'foo', + None, + None, + None, + [errors.InvalidHeaderDefect]), + + 'missing': ( + '', + '', + None, + None, + None, + [errors.HeaderMissingRequiredValue]), + + } + + +@parameterize +class TestAddressHeader(TestHeaderBase): + + example_params = { + + 'empty': + ('<>', + [errors.InvalidHeaderDefect], + '<>', + '', + '<>', + '', + '', + None), + + 'address_only': + ('zippy@pinhead.com', + [], + 'zippy@pinhead.com', + '', + 'zippy@pinhead.com', + 'zippy', + 'pinhead.com', + None), + + 'name_and_address': + ('Zaphrod Beblebrux ', + [], + 'Zaphrod Beblebrux ', + 'Zaphrod Beblebrux', + 'zippy@pinhead.com', + 'zippy', + 'pinhead.com', + None), + + 'quoted_local_part': + ('Zaphrod Beblebrux <"foo bar"@pinhead.com>', + [], + 'Zaphrod Beblebrux <"foo bar"@pinhead.com>', + 'Zaphrod Beblebrux', + '"foo bar"@pinhead.com', + 'foo bar', + 'pinhead.com', + None), + + 'quoted_parens_in_name': + (r'"A \(Special\) Person" ', + [], + '"A (Special) Person" ', + 'A (Special) Person', + 'person@dom.ain', + 'person', + 'dom.ain', + None), + + 'quoted_backslashes_in_name': + (r'"Arthur \\Backslash\\ Foobar" ', + [], + r'"Arthur \\Backslash\\ Foobar" ', + r'Arthur \Backslash\ Foobar', + 'person@dom.ain', + 'person', + 'dom.ain', + None), + + 'name_with_dot': + ('John X. Doe ', + [errors.ObsoleteHeaderDefect], + '"John X. Doe" ', + 'John X. Doe', + 'jxd@example.com', + 'jxd', + 'example.com', + None), + + 'quoted_strings_in_local_part': + ('""example" example"@example.com', + [errors.InvalidHeaderDefect]*3, + '"example example"@example.com', + '', + '"example example"@example.com', + 'example example', + 'example.com', + None), + + 'escaped_quoted_strings_in_local_part': + (r'"\"example\" example"@example.com', + [], + r'"\"example\" example"@example.com', + '', + r'"\"example\" example"@example.com', + r'"example" example', + 'example.com', + None), + + 'escaped_escapes_in_local_part': + (r'"\\"example\\" example"@example.com', + [errors.InvalidHeaderDefect]*5, + r'"\\example\\\\ example"@example.com', + '', + r'"\\example\\\\ example"@example.com', + r'\example\\ example', + 'example.com', + None), + + 'spaces_in_unquoted_local_part_collapsed': + ('merwok wok @example.com', + [errors.InvalidHeaderDefect]*2, + '"merwok wok"@example.com', + '', + '"merwok wok"@example.com', + 'merwok wok', + 'example.com', + None), + + 'spaces_around_dots_in_local_part_removed': + ('merwok. wok . wok@example.com', + [errors.ObsoleteHeaderDefect], + 'merwok.wok.wok@example.com', + '', + 'merwok.wok.wok@example.com', + 'merwok.wok.wok', + 'example.com', + None), + + 'rfc2047_atom_is_decoded': + ('=?utf-8?q?=C3=89ric?= ', + [], + 'Éric ', + 'Éric', + 'foo@example.com', + 'foo', + 'example.com', + None), + + 'rfc2047_atom_in_phrase_is_decoded': + ('The =?utf-8?q?=C3=89ric=2C?= Himself ', + [], + '"The Éric, Himself" ', + 'The Éric, Himself', + 'foo@example.com', + 'foo', + 'example.com', + None), + + 'rfc2047_atom_in_quoted_string_is_decoded': + ('"=?utf-8?q?=C3=89ric?=" ', + [errors.InvalidHeaderDefect, + errors.InvalidHeaderDefect], + 'Éric ', + 'Éric', + 'foo@example.com', + 'foo', + 'example.com', + None), + + 'name_ending_with_dot_without_space': + ('John X.', + [errors.ObsoleteHeaderDefect], + '"John X." ', + 'John X.', + 'jxd@example.com', + 'jxd', + 'example.com', + None), + + 'name_starting_with_dot': + ('. Doe ', + [errors.InvalidHeaderDefect, errors.ObsoleteHeaderDefect], + '". Doe" ', + '. Doe', + 'jxd@example.com', + 'jxd', + 'example.com', + None), + + } + + # XXX: Need many more examples, and in particular some with names in + # trailing comments, which aren't currently handled. comments in + # general are not handled yet. + + def example_as_address(self, source, defects, decoded, display_name, + addr_spec, username, domain, comment): + h = self.make_header('sender', source) + self.assertEqual(h, decoded) + self.assertDefectsEqual(h.defects, defects) + a = h.address + self.assertEqual(str(a), decoded) + self.assertEqual(len(h.groups), 1) + self.assertEqual([a], list(h.groups[0].addresses)) + self.assertEqual([a], list(h.addresses)) + self.assertEqual(a.display_name, display_name) + self.assertEqual(a.addr_spec, addr_spec) + self.assertEqual(a.username, username) + self.assertEqual(a.domain, domain) + # XXX: we have no comment support yet. + #self.assertEqual(a.comment, comment) + + def example_as_group(self, source, defects, decoded, display_name, + addr_spec, username, domain, comment): + source = 'foo: {};'.format(source) + gdecoded = 'foo: {};'.format(decoded) if decoded else 'foo:;' + h = self.make_header('to', source) + self.assertEqual(h, gdecoded) + self.assertDefectsEqual(h.defects, defects) + self.assertEqual(h.groups[0].addresses, h.addresses) + self.assertEqual(len(h.groups), 1) + self.assertEqual(len(h.addresses), 1) + a = h.addresses[0] + self.assertEqual(str(a), decoded) + self.assertEqual(a.display_name, display_name) + self.assertEqual(a.addr_spec, addr_spec) + self.assertEqual(a.username, username) + self.assertEqual(a.domain, domain) + + def test_simple_address_list(self): + value = ('Fred , foo@example.com, ' + '"Harry W. Hastings" ') + h = self.make_header('to', value) + self.assertEqual(h, value) + self.assertEqual(len(h.groups), 3) + self.assertEqual(len(h.addresses), 3) + for i in range(3): + self.assertEqual(h.groups[i].addresses[0], h.addresses[i]) + self.assertEqual(str(h.addresses[0]), 'Fred ') + self.assertEqual(str(h.addresses[1]), 'foo@example.com') + self.assertEqual(str(h.addresses[2]), + '"Harry W. Hastings" ') + self.assertEqual(h.addresses[2].display_name, + 'Harry W. Hastings') + + def test_complex_address_list(self): + examples = list(self.example_params.values()) + source = ('dummy list:;, another: (empty);,' + + ', '.join([x[0] for x in examples[:4]]) + ', ' + + r'"A \"list\"": ' + + ', '.join([x[0] for x in examples[4:6]]) + ';,' + + ', '.join([x[0] for x in examples[6:]]) + ) + # XXX: the fact that (empty) disappears here is a potential API design + # bug. We don't currently have a way to preserve comments. + expected = ('dummy list:;, another:;, ' + + ', '.join([x[2] for x in examples[:4]]) + ', ' + + r'"A \"list\"": ' + + ', '.join([x[2] for x in examples[4:6]]) + ';, ' + + ', '.join([x[2] for x in examples[6:]]) + ) + + h = self.make_header('to', source) + self.assertEqual(h.split(','), expected.split(',')) + self.assertEqual(h, expected) + self.assertEqual(len(h.groups), 7 + len(examples) - 6) + self.assertEqual(h.groups[0].display_name, 'dummy list') + self.assertEqual(h.groups[1].display_name, 'another') + self.assertEqual(h.groups[6].display_name, 'A "list"') + self.assertEqual(len(h.addresses), len(examples)) + for i in range(4): + self.assertIsNone(h.groups[i+2].display_name) + self.assertEqual(str(h.groups[i+2].addresses[0]), examples[i][2]) + for i in range(7, 7 + len(examples) - 6): + self.assertIsNone(h.groups[i].display_name) + self.assertEqual(str(h.groups[i].addresses[0]), examples[i-1][2]) + for i in range(len(examples)): + self.assertEqual(str(h.addresses[i]), examples[i][2]) + self.assertEqual(h.addresses[i].addr_spec, examples[i][4]) + + def test_address_read_only(self): + h = self.make_header('sender', 'abc@xyz.com') + with self.assertRaises(AttributeError): + h.address = 'foo' + + def test_addresses_read_only(self): + h = self.make_header('sender', 'abc@xyz.com') + with self.assertRaises(AttributeError): + h.addresses = 'foo' + + def test_groups_read_only(self): + h = self.make_header('sender', 'abc@xyz.com') + with self.assertRaises(AttributeError): + h.groups = 'foo' + + def test_addresses_types(self): + source = 'me ' + h = self.make_header('to', source) + self.assertIsInstance(h.addresses, tuple) + self.assertIsInstance(h.addresses[0], Address) + + def test_groups_types(self): + source = 'me ' + h = self.make_header('to', source) + self.assertIsInstance(h.groups, tuple) + self.assertIsInstance(h.groups[0], Group) + + def test_set_from_Address(self): + h = self.make_header('to', Address('me', 'foo', 'example.com')) + self.assertEqual(h, 'me ') + + def test_set_from_Address_list(self): + h = self.make_header('to', [Address('me', 'foo', 'example.com'), + Address('you', 'bar', 'example.com')]) + self.assertEqual(h, 'me , you ') + + def test_set_from_Address_and_Group_list(self): + h = self.make_header('to', [Address('me', 'foo', 'example.com'), + Group('bing', [Address('fiz', 'z', 'b.com'), + Address('zif', 'f', 'c.com')]), + Address('you', 'bar', 'example.com')]) + self.assertEqual(h, 'me , bing: fiz , ' + 'zif ;, you ') + self.assertEqual(h.fold(policy=policy.default.clone(max_line_length=40)), + 'to: me ,\n' + ' bing: fiz , zif ;,\n' + ' you \n') + + def test_set_from_Group_list(self): + h = self.make_header('to', [Group('bing', [Address('fiz', 'z', 'b.com'), + Address('zif', 'f', 'c.com')])]) + self.assertEqual(h, 'bing: fiz , zif ;') + + +class TestAddressAndGroup(TestEmailBase): + + def _test_attr_ro(self, obj, attr): + with self.assertRaises(AttributeError): + setattr(obj, attr, 'foo') + + def test_address_display_name_ro(self): + self._test_attr_ro(Address('foo', 'bar', 'baz'), 'display_name') + + def test_address_username_ro(self): + self._test_attr_ro(Address('foo', 'bar', 'baz'), 'username') + + def test_address_domain_ro(self): + self._test_attr_ro(Address('foo', 'bar', 'baz'), 'domain') + + def test_group_display_name_ro(self): + self._test_attr_ro(Group('foo'), 'display_name') + + def test_group_addresses_ro(self): + self._test_attr_ro(Group('foo'), 'addresses') + + def test_address_from_username_domain(self): + a = Address('foo', 'bar', 'baz') + self.assertEqual(a.display_name, 'foo') + self.assertEqual(a.username, 'bar') + self.assertEqual(a.domain, 'baz') + self.assertEqual(a.addr_spec, 'bar@baz') + self.assertEqual(str(a), 'foo ') + + def test_address_from_addr_spec(self): + a = Address('foo', addr_spec='bar@baz') + self.assertEqual(a.display_name, 'foo') + self.assertEqual(a.username, 'bar') + self.assertEqual(a.domain, 'baz') + self.assertEqual(a.addr_spec, 'bar@baz') + self.assertEqual(str(a), 'foo ') + + def test_address_with_no_display_name(self): + a = Address(addr_spec='bar@baz') + self.assertEqual(a.display_name, '') + self.assertEqual(a.username, 'bar') + self.assertEqual(a.domain, 'baz') + self.assertEqual(a.addr_spec, 'bar@baz') + self.assertEqual(str(a), 'bar@baz') + + def test_null_address(self): + a = Address() + self.assertEqual(a.display_name, '') + self.assertEqual(a.username, '') + self.assertEqual(a.domain, '') + self.assertEqual(a.addr_spec, '<>') + self.assertEqual(str(a), '<>') + + def test_domain_only(self): + # This isn't really a valid address. + a = Address(domain='buzz') + self.assertEqual(a.display_name, '') + self.assertEqual(a.username, '') + self.assertEqual(a.domain, 'buzz') + self.assertEqual(a.addr_spec, '@buzz') + self.assertEqual(str(a), '@buzz') + + def test_username_only(self): + # This isn't really a valid address. + a = Address(username='buzz') + self.assertEqual(a.display_name, '') + self.assertEqual(a.username, 'buzz') + self.assertEqual(a.domain, '') + self.assertEqual(a.addr_spec, 'buzz') + self.assertEqual(str(a), 'buzz') + + def test_display_name_only(self): + a = Address('buzz') + self.assertEqual(a.display_name, 'buzz') + self.assertEqual(a.username, '') + self.assertEqual(a.domain, '') + self.assertEqual(a.addr_spec, '<>') + self.assertEqual(str(a), 'buzz <>') + + def test_quoting(self): + # Ideally we'd check every special individually, but I'm not up for + # writing that many tests. + a = Address('Sara J.', 'bad name', 'example.com') + self.assertEqual(a.display_name, 'Sara J.') + self.assertEqual(a.username, 'bad name') + self.assertEqual(a.domain, 'example.com') + self.assertEqual(a.addr_spec, '"bad name"@example.com') + self.assertEqual(str(a), '"Sara J." <"bad name"@example.com>') + + def test_il8n(self): + a = Address('Éric', 'wok', 'exàmple.com') + self.assertEqual(a.display_name, 'Éric') + self.assertEqual(a.username, 'wok') + self.assertEqual(a.domain, 'exàmple.com') + self.assertEqual(a.addr_spec, 'wok@exàmple.com') + self.assertEqual(str(a), 'Éric ') + + # XXX: there is an API design issue that needs to be solved here. + #def test_non_ascii_username_raises(self): + # with self.assertRaises(ValueError): + # Address('foo', 'wők', 'example.com') + + def test_crlf_in_constructor_args_raises(self): + cases = ( + dict(display_name='foo\r'), + dict(display_name='foo\n'), + dict(display_name='foo\r\n'), + dict(domain='example.com\r'), + dict(domain='example.com\n'), + dict(domain='example.com\r\n'), + dict(username='wok\r'), + dict(username='wok\n'), + dict(username='wok\r\n'), + dict(addr_spec='wok@example.com\r'), + dict(addr_spec='wok@example.com\n'), + dict(addr_spec='wok@example.com\r\n') + ) + for kwargs in cases: + with self.subTest(kwargs=kwargs), self.assertRaisesRegex(ValueError, "invalid arguments"): + Address(**kwargs) + + def test_non_ascii_username_in_addr_spec_raises(self): + with self.assertRaises(ValueError): + Address('foo', addr_spec='wők@example.com') + + def test_address_addr_spec_and_username_raises(self): + with self.assertRaises(TypeError): + Address('foo', username='bing', addr_spec='bar@baz') + + def test_address_addr_spec_and_domain_raises(self): + with self.assertRaises(TypeError): + Address('foo', domain='bing', addr_spec='bar@baz') + + def test_address_addr_spec_and_username_and_domain_raises(self): + with self.assertRaises(TypeError): + Address('foo', username='bong', domain='bing', addr_spec='bar@baz') + + def test_space_in_addr_spec_username_raises(self): + with self.assertRaises(ValueError): + Address('foo', addr_spec="bad name@example.com") + + def test_bad_addr_sepc_raises(self): + with self.assertRaises(ValueError): + Address('foo', addr_spec="name@ex[]ample.com") + + def test_empty_group(self): + g = Group('foo') + self.assertEqual(g.display_name, 'foo') + self.assertEqual(g.addresses, tuple()) + self.assertEqual(str(g), 'foo:;') + + def test_empty_group_list(self): + g = Group('foo', addresses=[]) + self.assertEqual(g.display_name, 'foo') + self.assertEqual(g.addresses, tuple()) + self.assertEqual(str(g), 'foo:;') + + def test_null_group(self): + g = Group() + self.assertIsNone(g.display_name) + self.assertEqual(g.addresses, tuple()) + self.assertEqual(str(g), 'None:;') + + def test_group_with_addresses(self): + addrs = [Address('b', 'b', 'c'), Address('a', 'b','c')] + g = Group('foo', addrs) + self.assertEqual(g.display_name, 'foo') + self.assertEqual(g.addresses, tuple(addrs)) + self.assertEqual(str(g), 'foo: b , a ;') + + def test_group_with_addresses_no_display_name(self): + addrs = [Address('b', 'b', 'c'), Address('a', 'b','c')] + g = Group(addresses=addrs) + self.assertIsNone(g.display_name) + self.assertEqual(g.addresses, tuple(addrs)) + self.assertEqual(str(g), 'None: b , a ;') + + def test_group_with_one_address_no_display_name(self): + addrs = [Address('b', 'b', 'c')] + g = Group(addresses=addrs) + self.assertIsNone(g.display_name) + self.assertEqual(g.addresses, tuple(addrs)) + self.assertEqual(str(g), 'b ') + + def test_display_name_quoting(self): + g = Group('foo.bar') + self.assertEqual(g.display_name, 'foo.bar') + self.assertEqual(g.addresses, tuple()) + self.assertEqual(str(g), '"foo.bar":;') + + def test_display_name_blanks_not_quoted(self): + g = Group('foo bar') + self.assertEqual(g.display_name, 'foo bar') + self.assertEqual(g.addresses, tuple()) + self.assertEqual(str(g), 'foo bar:;') + + def test_set_message_header_from_address(self): + a = Address('foo', 'bar', 'example.com') + m = Message(policy=policy.default) + m['To'] = a + self.assertEqual(m['to'], 'foo ') + self.assertEqual(m['to'].addresses, (a,)) + + def test_set_message_header_from_group(self): + g = Group('foo bar') + m = Message(policy=policy.default) + m['To'] = g + self.assertEqual(m['to'], 'foo bar:;') + self.assertEqual(m['to'].addresses, g.addresses) + + def test_address_comparison(self): + a = Address('foo', 'bar', 'example.com') + self.assertEqual(Address('foo', 'bar', 'example.com'), a) + self.assertNotEqual(Address('baz', 'bar', 'example.com'), a) + self.assertNotEqual(Address('foo', 'baz', 'example.com'), a) + self.assertNotEqual(Address('foo', 'bar', 'baz'), a) + self.assertFalse(a == object()) + self.assertTrue(a == ALWAYS_EQ) + + def test_group_comparison(self): + a = Address('foo', 'bar', 'example.com') + g = Group('foo bar', [a]) + self.assertEqual(Group('foo bar', (a,)), g) + self.assertNotEqual(Group('baz', [a]), g) + self.assertNotEqual(Group('foo bar', []), g) + self.assertFalse(g == object()) + self.assertTrue(g == ALWAYS_EQ) + + +class TestFolding(TestHeaderBase): + + def test_address_display_names(self): + """Test the folding and encoding of address headers.""" + for name, result in ( + ('Foo Bar, France', '"Foo Bar, France"'), + ('Foo Bar (France)', '"Foo Bar (France)"'), + ('Foo Bar, España', 'Foo =?utf-8?q?Bar=2C_Espa=C3=B1a?='), + ('Foo Bar (España)', 'Foo Bar =?utf-8?b?KEVzcGHDsWEp?='), + ('Foo, Bar España', '=?utf-8?q?Foo=2C_Bar_Espa=C3=B1a?='), + ('Foo, Bar [España]', '=?utf-8?q?Foo=2C_Bar_=5BEspa=C3=B1a=5D?='), + ('Foo Bär, France', 'Foo =?utf-8?q?B=C3=A4r=2C?= France'), + ('Foo Bär ', 'Foo =?utf-8?q?B=C3=A4r_=3CFrance=3E?='), + ( + 'Lôrem ipsum dôlôr sit amet, cônsectetuer adipiscing. ' + 'Suspendisse pôtenti. Aliquam nibh. Suspendisse pôtenti.', + '=?utf-8?q?L=C3=B4rem_ipsum_d=C3=B4l=C3=B4r_sit_amet=2C_c' + '=C3=B4nsectetuer?=\n =?utf-8?q?_adipiscing=2E_Suspendisse' + '_p=C3=B4tenti=2E_Aliquam_nibh=2E?=\n Suspendisse =?utf-8' + '?q?p=C3=B4tenti=2E?=', + ), + ): + h = self.make_header('To', Address(name, addr_spec='a@b.com')) + self.assertEqual(h.fold(policy=policy.default), + 'To: %s \n' % result) + + def test_short_unstructured(self): + h = self.make_header('subject', 'this is a test') + self.assertEqual(h.fold(policy=policy.default), + 'subject: this is a test\n') + + def test_long_unstructured(self): + h = self.make_header('Subject', 'This is a long header ' + 'line that will need to be folded into two lines ' + 'and will demonstrate basic folding') + self.assertEqual(h.fold(policy=policy.default), + 'Subject: This is a long header line that will ' + 'need to be folded into two lines\n' + ' and will demonstrate basic folding\n') + + def test_unstructured_short_max_line_length(self): + h = self.make_header('Subject', 'this is a short header ' + 'that will be folded anyway') + self.assertEqual( + h.fold(policy=policy.default.clone(max_line_length=20)), + textwrap.dedent("""\ + Subject: this is a + short header that + will be folded + anyway + """)) + + def test_fold_unstructured_single_word(self): + h = self.make_header('Subject', 'test') + self.assertEqual(h.fold(policy=policy.default), 'Subject: test\n') + + def test_fold_unstructured_short(self): + h = self.make_header('Subject', 'test test test') + self.assertEqual(h.fold(policy=policy.default), + 'Subject: test test test\n') + + def test_fold_unstructured_with_overlong_word(self): + h = self.make_header('Subject', 'thisisaverylonglineconsistingofa' + 'singlewordthatwontfit') + self.assertEqual( + h.fold(policy=policy.default.clone(max_line_length=20)), + 'Subject: \n' + ' =?utf-8?q?thisisa?=\n' + ' =?utf-8?q?verylon?=\n' + ' =?utf-8?q?glineco?=\n' + ' =?utf-8?q?nsistin?=\n' + ' =?utf-8?q?gofasin?=\n' + ' =?utf-8?q?gleword?=\n' + ' =?utf-8?q?thatwon?=\n' + ' =?utf-8?q?tfit?=\n' + ) + + def test_fold_unstructured_with_two_overlong_words(self): + h = self.make_header('Subject', 'thisisaverylonglineconsistingofa' + 'singlewordthatwontfit plusanotherverylongwordthatwontfit') + self.assertEqual( + h.fold(policy=policy.default.clone(max_line_length=20)), + 'Subject: \n' + ' =?utf-8?q?thisisa?=\n' + ' =?utf-8?q?verylon?=\n' + ' =?utf-8?q?glineco?=\n' + ' =?utf-8?q?nsistin?=\n' + ' =?utf-8?q?gofasin?=\n' + ' =?utf-8?q?gleword?=\n' + ' =?utf-8?q?thatwon?=\n' + ' =?utf-8?q?tfit_pl?=\n' + ' =?utf-8?q?usanoth?=\n' + ' =?utf-8?q?erveryl?=\n' + ' =?utf-8?q?ongword?=\n' + ' =?utf-8?q?thatwon?=\n' + ' =?utf-8?q?tfit?=\n' + ) + + # XXX Need test for when max_line_length is less than the chrome size. + + def test_fold_unstructured_with_slightly_long_word(self): + h = self.make_header('Subject', 'thislongwordislessthanmaxlinelen') + self.assertEqual( + h.fold(policy=policy.default.clone(max_line_length=35)), + 'Subject:\n thislongwordislessthanmaxlinelen\n') + + def test_fold_unstructured_with_commas(self): + # The old wrapper would fold this at the commas. + h = self.make_header('Subject', "This header is intended to " + "demonstrate, in a fairly succinct way, that we now do " + "not give a , special treatment in unstructured headers.") + self.assertEqual( + h.fold(policy=policy.default.clone(max_line_length=60)), + textwrap.dedent("""\ + Subject: This header is intended to demonstrate, in a fairly + succinct way, that we now do not give a , special treatment + in unstructured headers. + """)) + + def test_fold_address_list(self): + h = self.make_header('To', '"Theodore H. Perfect" , ' + '"My address is very long because my name is long" , ' + '"Only A. Friend" ') + self.assertEqual(h.fold(policy=policy.default), textwrap.dedent("""\ + To: "Theodore H. Perfect" , + "My address is very long because my name is long" , + "Only A. Friend" + """)) + + def test_fold_date_header(self): + h = self.make_header('Date', 'Sat, 2 Feb 2002 17:00:06 -0800') + self.assertEqual(h.fold(policy=policy.default), + 'Date: Sat, 02 Feb 2002 17:00:06 -0800\n') + + def test_fold_overlong_words_using_RFC2047(self): + h = self.make_header( + 'X-Report-Abuse', + '') + self.assertEqual( + h.fold(policy=policy.default), + 'X-Report-Abuse: =?utf-8?q?=3Chttps=3A//www=2Emailitapp=2E' + 'com/report=5Fabuse?=\n' + ' =?utf-8?q?=2Ephp=3Fmid=3Dxxx-xxx-xxxx' + 'xxxxxxxxxxxxxxxxxxxx=3D=3D-xxx-xx-xx?=\n' + ' =?utf-8?q?=3E?=\n') + + def test_message_id_header_is_not_folded(self): + h = self.make_header( + 'Message-ID', + '') + self.assertEqual( + h.fold(policy=policy.default.clone(max_line_length=20)), + 'Message-ID: \n') + + # Test message-id isn't folded when id-right is no-fold-literal. + h = self.make_header( + 'Message-ID', + '') + self.assertEqual( + h.fold(policy=policy.default.clone(max_line_length=20)), + 'Message-ID: \n') + + # Test message-id isn't folded when id-right is non-ascii characters. + h = self.make_header('Message-ID', '<ईमेल@wők.com>') + self.assertEqual( + h.fold(policy=policy.default.clone(max_line_length=30)), + 'Message-ID: <ईमेल@wők.com>\n') + + # Test message-id is folded without breaking the msg-id token into + # encoded words, *even* if they don't fit into max_line_length. + h = self.make_header('Message-ID', '<ईमेलfromMessage@wők.com>') + self.assertEqual( + h.fold(policy=policy.default.clone(max_line_length=20)), + 'Message-ID:\n <ईमेलfromMessage@wők.com>\n') + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test_inversion.py b/Lib/test/test_email/test_inversion.py new file mode 100644 index 00000000000..7bd7f2a7206 --- /dev/null +++ b/Lib/test/test_email/test_inversion.py @@ -0,0 +1,78 @@ +"""Test the parser and generator are inverses. + +Note that this is only strictly true if we are parsing RFC valid messages and +producing RFC valid messages. +""" + +import io +import unittest +from email import policy, message_from_bytes +from email.message import EmailMessage +from email.generator import BytesGenerator +from test.test_email import TestEmailBase, parameterize + +# This is like textwrap.dedent for bytes, except that it uses \r\n for the line +# separators on the rebuilt string. +def dedent(bstr): + lines = bstr.splitlines() + if not lines[0].strip(): + raise ValueError("First line must contain text") + stripamt = len(lines[0]) - len(lines[0].lstrip()) + return b'\r\n'.join( + [x[stripamt:] if len(x)>=stripamt else b'' + for x in lines]) + + +@parameterize +class TestInversion(TestEmailBase): + + policy = policy.default + message = EmailMessage + + def msg_as_input(self, msg): + m = message_from_bytes(msg, policy=policy.SMTP) + b = io.BytesIO() + g = BytesGenerator(b) + g.flatten(m) + self.assertEqual(b.getvalue(), msg) + + # XXX: spaces are not preserved correctly here yet in the general case. + msg_params = { + 'header_with_one_space_body': (dedent(b"""\ + From: abc@xyz.com + X-Status:\x20 + Subject: test + + foo + """),), + + 'header_with_invalid_date': (dedent(b"""\ + Date: Tue, 06 Jun 2017 27:39:33 +0600 + From: abc@xyz.com + Subject: timezones + + How do they work even? + """),), + + } + + payload_params = { + 'plain_text': dict(payload='This is a test\n'*20), + 'base64_text': dict(payload=(('xy a'*40+'\n')*5), cte='base64'), + 'qp_text': dict(payload=(('xy a'*40+'\n')*5), cte='quoted-printable'), + } + + def payload_as_body(self, payload, **kw): + msg = self._make_message() + msg['From'] = 'foo' + msg['To'] = 'bar' + msg['Subject'] = 'payload round trip test' + msg.set_content(payload, **kw) + b = bytes(msg) + msg2 = message_from_bytes(b, policy=self.policy) + self.assertEqual(bytes(msg2), b) + self.assertEqual(msg2.get_content(), payload) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test_message.py b/Lib/test/test_email/test_message.py new file mode 100644 index 00000000000..966615dcc1d --- /dev/null +++ b/Lib/test/test_email/test_message.py @@ -0,0 +1,1094 @@ +import textwrap +import unittest +from email import message_from_bytes, message_from_string, policy +from email.message import EmailMessage, MIMEPart +from test.test_email import TestEmailBase, parameterize + + +# Helper. +def first(iterable): + return next(filter(lambda x: x is not None, iterable), None) + + +class Test(TestEmailBase): + + policy = policy.default + + def test_error_on_setitem_if_max_count_exceeded(self): + m = self._str_msg("") + m['To'] = 'abc@xyz' + with self.assertRaises(ValueError): + m['To'] = 'xyz@abc' + + def test_rfc2043_auto_decoded_and_emailmessage_used(self): + m = message_from_string(textwrap.dedent("""\ + Subject: Ayons asperges pour le =?utf-8?q?d=C3=A9jeuner?= + From: =?utf-8?q?Pep=C3=A9?= Le Pew + To: "Penelope Pussycat" <"penelope@example.com"> + MIME-Version: 1.0 + Content-Type: text/plain; charset="utf-8" + + sample text + """), policy=policy.default) + self.assertEqual(m['subject'], "Ayons asperges pour le déjeuner") + self.assertEqual(m['from'], "Pepé Le Pew ") + self.assertIsInstance(m, EmailMessage) + + +@parameterize +class TestEmailMessageBase: + + policy = policy.default + + # The first argument is a triple (related, html, plain) of indices into the + # list returned by 'walk' called on a Message constructed from the third. + # The indices indicate which part should match the corresponding part-type + # when passed to get_body (ie: the "first" part of that type in the + # message). The second argument is a list of indices into the 'walk' list + # of the attachments that should be returned by a call to + # 'iter_attachments'. The third argument is a list of indices into 'walk' + # that should be returned by a call to 'iter_parts'. Note that the first + # item returned by 'walk' is the Message itself. + + message_params = { + + 'empty_message': ( + (None, None, 0), + (), + (), + ""), + + 'non_mime_plain': ( + (None, None, 0), + (), + (), + textwrap.dedent("""\ + To: foo@example.com + + simple text body + """)), + + 'mime_non_text': ( + (None, None, None), + (), + (), + textwrap.dedent("""\ + To: foo@example.com + MIME-Version: 1.0 + Content-Type: image/jpg + + bogus body. + """)), + + 'plain_html_alternative': ( + (None, 2, 1), + (), + (1, 2), + textwrap.dedent("""\ + To: foo@example.com + MIME-Version: 1.0 + Content-Type: multipart/alternative; boundary="===" + + preamble + + --=== + Content-Type: text/plain + + simple body + + --=== + Content-Type: text/html + +

    simple body

    + --===-- + """)), + + 'plain_html_mixed': ( + (None, 2, 1), + (), + (1, 2), + textwrap.dedent("""\ + To: foo@example.com + MIME-Version: 1.0 + Content-Type: multipart/mixed; boundary="===" + + preamble + + --=== + Content-Type: text/plain + + simple body + + --=== + Content-Type: text/html + +

    simple body

    + + --===-- + """)), + + 'plain_html_attachment_mixed': ( + (None, None, 1), + (2,), + (1, 2), + textwrap.dedent("""\ + To: foo@example.com + MIME-Version: 1.0 + Content-Type: multipart/mixed; boundary="===" + + --=== + Content-Type: text/plain + + simple body + + --=== + Content-Type: text/html + Content-Disposition: attachment + +

    simple body

    + + --===-- + """)), + + 'html_text_attachment_mixed': ( + (None, 2, None), + (1,), + (1, 2), + textwrap.dedent("""\ + To: foo@example.com + MIME-Version: 1.0 + Content-Type: multipart/mixed; boundary="===" + + --=== + Content-Type: text/plain + Content-Disposition: AtTaChment + + simple body + + --=== + Content-Type: text/html + +

    simple body

    + + --===-- + """)), + + 'html_text_attachment_inline_mixed': ( + (None, 2, 1), + (), + (1, 2), + textwrap.dedent("""\ + To: foo@example.com + MIME-Version: 1.0 + Content-Type: multipart/mixed; boundary="===" + + --=== + Content-Type: text/plain + Content-Disposition: InLine + + simple body + + --=== + Content-Type: text/html + Content-Disposition: inline + +

    simple body

    + + --===-- + """)), + + # RFC 2387 + 'related': ( + (0, 1, None), + (2,), + (1, 2), + textwrap.dedent("""\ + To: foo@example.com + MIME-Version: 1.0 + Content-Type: multipart/related; boundary="==="; type=text/html + + --=== + Content-Type: text/html + +

    simple body

    + + --=== + Content-Type: image/jpg + Content-ID: + + bogus data + + --===-- + """)), + + # This message structure will probably never be seen in the wild, but + # it proves we distinguish between text parts based on 'start'. The + # content would not, of course, actually work :) + 'related_with_start': ( + (0, 2, None), + (1,), + (1, 2), + textwrap.dedent("""\ + To: foo@example.com + MIME-Version: 1.0 + Content-Type: multipart/related; boundary="==="; type=text/html; + start="" + + --=== + Content-Type: text/html + Content-ID: + + useless text + + --=== + Content-Type: text/html + Content-ID: + +

    simple body

    + + + --===-- + """)), + + + 'mixed_alternative_plain_related': ( + (3, 4, 2), + (6, 7), + (1, 6, 7), + textwrap.dedent("""\ + To: foo@example.com + MIME-Version: 1.0 + Content-Type: multipart/mixed; boundary="===" + + --=== + Content-Type: multipart/alternative; boundary="+++" + + --+++ + Content-Type: text/plain + + simple body + + --+++ + Content-Type: multipart/related; boundary="___" + + --___ + Content-Type: text/html + +

    simple body

    + + --___ + Content-Type: image/jpg + Content-ID: + + bogus jpg body + + --___-- + + --+++-- + + --=== + Content-Type: image/jpg + Content-Disposition: attachment + + bogus jpg body + + --=== + Content-Type: image/jpg + Content-Disposition: AttacHmenT + + another bogus jpg body + + --===-- + """)), + + # This structure suggested by Stephen J. Turnbull...may not exist/be + # supported in the wild, but we want to support it. + 'mixed_related_alternative_plain_html': ( + (1, 4, 3), + (6, 7), + (1, 6, 7), + textwrap.dedent("""\ + To: foo@example.com + MIME-Version: 1.0 + Content-Type: multipart/mixed; boundary="===" + + --=== + Content-Type: multipart/related; boundary="+++" + + --+++ + Content-Type: multipart/alternative; boundary="___" + + --___ + Content-Type: text/plain + + simple body + + --___ + Content-Type: text/html + +

    simple body

    + + --___-- + + --+++ + Content-Type: image/jpg + Content-ID: + + bogus jpg body + + --+++-- + + --=== + Content-Type: image/jpg + Content-Disposition: attachment + + bogus jpg body + + --=== + Content-Type: image/jpg + Content-Disposition: attachment + + another bogus jpg body + + --===-- + """)), + + # Same thing, but proving we only look at the root part, which is the + # first one if there isn't any start parameter. That is, this is a + # broken related. + 'mixed_related_alternative_plain_html_wrong_order': ( + (1, None, None), + (6, 7), + (1, 6, 7), + textwrap.dedent("""\ + To: foo@example.com + MIME-Version: 1.0 + Content-Type: multipart/mixed; boundary="===" + + --=== + Content-Type: multipart/related; boundary="+++" + + --+++ + Content-Type: image/jpg + Content-ID: + + bogus jpg body + + --+++ + Content-Type: multipart/alternative; boundary="___" + + --___ + Content-Type: text/plain + + simple body + + --___ + Content-Type: text/html + +

    simple body

    + + --___-- + + --+++-- + + --=== + Content-Type: image/jpg + Content-Disposition: attachment + + bogus jpg body + + --=== + Content-Type: image/jpg + Content-Disposition: attachment + + another bogus jpg body + + --===-- + """)), + + 'message_rfc822': ( + (None, None, None), + (), + (), + textwrap.dedent("""\ + To: foo@example.com + MIME-Version: 1.0 + Content-Type: message/rfc822 + + To: bar@example.com + From: robot@examp.com + + this is a message body. + """)), + + 'mixed_text_message_rfc822': ( + (None, None, 1), + (2,), + (1, 2), + textwrap.dedent("""\ + To: foo@example.com + MIME-Version: 1.0 + Content-Type: multipart/mixed; boundary="===" + + --=== + Content-Type: text/plain + + Your message has bounced, sir. + + --=== + Content-Type: message/rfc822 + + To: bar@example.com + From: robot@examp.com + + this is a message body. + + --===-- + """)), + + } + + def message_as_get_body(self, body_parts, attachments, parts, msg): + m = self._str_msg(msg) + allparts = list(m.walk()) + expected = [None if n is None else allparts[n] for n in body_parts] + related = 0; html = 1; plain = 2 + self.assertEqual(m.get_body(), first(expected)) + self.assertEqual(m.get_body(preferencelist=( + 'related', 'html', 'plain')), + first(expected)) + self.assertEqual(m.get_body(preferencelist=('related', 'html')), + first(expected[related:html+1])) + self.assertEqual(m.get_body(preferencelist=('related', 'plain')), + first([expected[related], expected[plain]])) + self.assertEqual(m.get_body(preferencelist=('html', 'plain')), + first(expected[html:plain+1])) + self.assertEqual(m.get_body(preferencelist=['related']), + expected[related]) + self.assertEqual(m.get_body(preferencelist=['html']), expected[html]) + self.assertEqual(m.get_body(preferencelist=['plain']), expected[plain]) + self.assertEqual(m.get_body(preferencelist=('plain', 'html')), + first(expected[plain:html-1:-1])) + self.assertEqual(m.get_body(preferencelist=('plain', 'related')), + first([expected[plain], expected[related]])) + self.assertEqual(m.get_body(preferencelist=('html', 'related')), + first(expected[html::-1])) + self.assertEqual(m.get_body(preferencelist=('plain', 'html', 'related')), + first(expected[::-1])) + self.assertEqual(m.get_body(preferencelist=('html', 'plain', 'related')), + first([expected[html], + expected[plain], + expected[related]])) + + def message_as_iter_attachment(self, body_parts, attachments, parts, msg): + m = self._str_msg(msg) + allparts = list(m.walk()) + attachments = [allparts[n] for n in attachments] + self.assertEqual(list(m.iter_attachments()), attachments) + + def message_as_iter_parts(self, body_parts, attachments, parts, msg): + def _is_multipart_msg(msg): + return 'Content-Type: multipart' in msg + + m = self._str_msg(msg) + allparts = list(m.walk()) + parts = [allparts[n] for n in parts] + iter_parts = list(m.iter_parts()) if _is_multipart_msg(msg) else [] + self.assertEqual(iter_parts, parts) + + class _TestContentManager: + def get_content(self, msg, *args, **kw): + return msg, args, kw + def set_content(self, msg, *args, **kw): + self.msg = msg + self.args = args + self.kw = kw + + def test_get_content_with_cm(self): + m = self._str_msg('') + cm = self._TestContentManager() + self.assertEqual(m.get_content(content_manager=cm), (m, (), {})) + msg, args, kw = m.get_content('foo', content_manager=cm, bar=1, k=2) + self.assertEqual(msg, m) + self.assertEqual(args, ('foo',)) + self.assertEqual(kw, dict(bar=1, k=2)) + + def test_get_content_default_cm_comes_from_policy(self): + p = policy.default.clone(content_manager=self._TestContentManager()) + m = self._str_msg('', policy=p) + self.assertEqual(m.get_content(), (m, (), {})) + msg, args, kw = m.get_content('foo', bar=1, k=2) + self.assertEqual(msg, m) + self.assertEqual(args, ('foo',)) + self.assertEqual(kw, dict(bar=1, k=2)) + + def test_set_content_with_cm(self): + m = self._str_msg('') + cm = self._TestContentManager() + m.set_content(content_manager=cm) + self.assertEqual(cm.msg, m) + self.assertEqual(cm.args, ()) + self.assertEqual(cm.kw, {}) + m.set_content('foo', content_manager=cm, bar=1, k=2) + self.assertEqual(cm.msg, m) + self.assertEqual(cm.args, ('foo',)) + self.assertEqual(cm.kw, dict(bar=1, k=2)) + + def test_set_content_default_cm_comes_from_policy(self): + cm = self._TestContentManager() + p = policy.default.clone(content_manager=cm) + m = self._str_msg('', policy=p) + m.set_content() + self.assertEqual(cm.msg, m) + self.assertEqual(cm.args, ()) + self.assertEqual(cm.kw, {}) + m.set_content('foo', bar=1, k=2) + self.assertEqual(cm.msg, m) + self.assertEqual(cm.args, ('foo',)) + self.assertEqual(cm.kw, dict(bar=1, k=2)) + + # outcome is whether xxx_method should raise ValueError error when called + # on multipart/subtype. Blank outcome means it depends on xxx (add + # succeeds, make raises). Note: 'none' means there are content-type + # headers but payload is None...this happening in practice would be very + # unusual, so treating it as if there were content seems reasonable. + # method subtype outcome + subtype_params = ( + ('related', 'no_content', 'succeeds'), + ('related', 'none', 'succeeds'), + ('related', 'plain', 'succeeds'), + ('related', 'related', ''), + ('related', 'alternative', 'raises'), + ('related', 'mixed', 'raises'), + ('alternative', 'no_content', 'succeeds'), + ('alternative', 'none', 'succeeds'), + ('alternative', 'plain', 'succeeds'), + ('alternative', 'related', 'succeeds'), + ('alternative', 'alternative', ''), + ('alternative', 'mixed', 'raises'), + ('mixed', 'no_content', 'succeeds'), + ('mixed', 'none', 'succeeds'), + ('mixed', 'plain', 'succeeds'), + ('mixed', 'related', 'succeeds'), + ('mixed', 'alternative', 'succeeds'), + ('mixed', 'mixed', ''), + ) + + def _make_subtype_test_message(self, subtype): + m = self.message() + payload = None + msg_headers = [ + ('To', 'foo@bar.com'), + ('From', 'bar@foo.com'), + ] + if subtype != 'no_content': + ('content-shadow', 'Logrus'), + msg_headers.append(('X-Random-Header', 'Corwin')) + if subtype == 'text': + payload = '' + msg_headers.append(('Content-Type', 'text/plain')) + m.set_payload('') + elif subtype != 'no_content': + payload = [] + msg_headers.append(('Content-Type', 'multipart/' + subtype)) + msg_headers.append(('X-Trump', 'Random')) + m.set_payload(payload) + for name, value in msg_headers: + m[name] = value + return m, msg_headers, payload + + def _check_disallowed_subtype_raises(self, m, method_name, subtype, method): + with self.assertRaises(ValueError) as ar: + getattr(m, method)() + exc_text = str(ar.exception) + self.assertIn(subtype, exc_text) + self.assertIn(method_name, exc_text) + + def _check_make_multipart(self, m, msg_headers, payload): + count = 0 + for name, value in msg_headers: + if not name.lower().startswith('content-'): + self.assertEqual(m[name], value) + count += 1 + self.assertEqual(len(m), count+1) # +1 for new Content-Type + part = next(m.iter_parts()) + count = 0 + for name, value in msg_headers: + if name.lower().startswith('content-'): + self.assertEqual(part[name], value) + count += 1 + self.assertEqual(len(part), count) + self.assertEqual(part.get_payload(), payload) + + def subtype_as_make(self, method, subtype, outcome): + m, msg_headers, payload = self._make_subtype_test_message(subtype) + make_method = 'make_' + method + if outcome in ('', 'raises'): + self._check_disallowed_subtype_raises(m, method, subtype, make_method) + return + getattr(m, make_method)() + self.assertEqual(m.get_content_maintype(), 'multipart') + self.assertEqual(m.get_content_subtype(), method) + if subtype == 'no_content': + self.assertEqual(len(m.get_payload()), 0) + self.assertEqual(m.items(), + msg_headers + [('Content-Type', + 'multipart/'+method)]) + else: + self.assertEqual(len(m.get_payload()), 1) + self._check_make_multipart(m, msg_headers, payload) + + def subtype_as_make_with_boundary(self, method, subtype, outcome): + # Doing all variation is a bit of overkill... + m = self.message() + if outcome in ('', 'raises'): + m['Content-Type'] = 'multipart/' + subtype + with self.assertRaises(ValueError) as cm: + getattr(m, 'make_' + method)() + return + if subtype == 'plain': + m['Content-Type'] = 'text/plain' + elif subtype != 'no_content': + m['Content-Type'] = 'multipart/' + subtype + getattr(m, 'make_' + method)(boundary="abc") + self.assertTrue(m.is_multipart()) + self.assertEqual(m.get_boundary(), 'abc') + + def test_policy_on_part_made_by_make_comes_from_message(self): + for method in ('make_related', 'make_alternative', 'make_mixed'): + m = self.message(policy=self.policy.clone(content_manager='foo')) + m['Content-Type'] = 'text/plain' + getattr(m, method)() + self.assertEqual(m.get_payload(0).policy.content_manager, 'foo') + + class _TestSetContentManager: + def set_content(self, msg, content, *args, **kw): + msg['Content-Type'] = 'text/plain' + msg.set_payload(content) + + def subtype_as_add(self, method, subtype, outcome): + m, msg_headers, payload = self._make_subtype_test_message(subtype) + cm = self._TestSetContentManager() + add_method = 'add_attachment' if method=='mixed' else 'add_' + method + if outcome == 'raises': + self._check_disallowed_subtype_raises(m, method, subtype, add_method) + return + getattr(m, add_method)('test', content_manager=cm) + self.assertEqual(m.get_content_maintype(), 'multipart') + self.assertEqual(m.get_content_subtype(), method) + if method == subtype or subtype == 'no_content': + self.assertEqual(len(m.get_payload()), 1) + for name, value in msg_headers: + self.assertEqual(m[name], value) + part = m.get_payload()[0] + else: + self.assertEqual(len(m.get_payload()), 2) + self._check_make_multipart(m, msg_headers, payload) + part = m.get_payload()[1] + self.assertEqual(part.get_content_type(), 'text/plain') + self.assertEqual(part.get_payload(), 'test') + if method=='mixed': + self.assertEqual(part['Content-Disposition'], 'attachment') + elif method=='related': + self.assertEqual(part['Content-Disposition'], 'inline') + else: + # Otherwise we don't guess. + self.assertIsNone(part['Content-Disposition']) + + class _TestSetRaisingContentManager: + class CustomError(Exception): + pass + def set_content(self, msg, content, *args, **kw): + raise self.CustomError('test') + + def test_default_content_manager_for_add_comes_from_policy(self): + cm = self._TestSetRaisingContentManager() + m = self.message(policy=self.policy.clone(content_manager=cm)) + for method in ('add_related', 'add_alternative', 'add_attachment'): + with self.assertRaises(self._TestSetRaisingContentManager.CustomError) as ar: + getattr(m, method)('') + self.assertEqual(str(ar.exception), 'test') + + def message_as_clear(self, body_parts, attachments, parts, msg): + m = self._str_msg(msg) + m.clear() + self.assertEqual(len(m), 0) + self.assertEqual(list(m.items()), []) + self.assertIsNone(m.get_payload()) + self.assertEqual(list(m.iter_parts()), []) + + def message_as_clear_content(self, body_parts, attachments, parts, msg): + m = self._str_msg(msg) + expected_headers = [h for h in m.keys() + if not h.lower().startswith('content-')] + m.clear_content() + self.assertEqual(list(m.keys()), expected_headers) + self.assertIsNone(m.get_payload()) + self.assertEqual(list(m.iter_parts()), []) + + def test_is_attachment(self): + m = self._make_message() + self.assertFalse(m.is_attachment()) + m['Content-Disposition'] = 'inline' + self.assertFalse(m.is_attachment()) + m.replace_header('Content-Disposition', 'attachment') + self.assertTrue(m.is_attachment()) + m.replace_header('Content-Disposition', 'AtTachMent') + self.assertTrue(m.is_attachment()) + m.set_param('filename', 'abc.png', 'Content-Disposition') + self.assertTrue(m.is_attachment()) + + def test_iter_attachments_mutation(self): + # We had a bug where iter_attachments was mutating the list. + m = self._make_message() + m.set_content('arbitrary text as main part') + m.add_related('more text as a related part') + m.add_related('yet more text as a second "attachment"') + orig = m.get_payload().copy() + self.assertEqual(len(list(m.iter_attachments())), 2) + self.assertEqual(m.get_payload(), orig) + + get_payload_surrogate_params = { + + 'good_surrogateescape': ( + "String that can be encod\udcc3\udcabd with surrogateescape", + b'String that can be encod\xc3\xabd with surrogateescape' + ), + + 'string_with_utf8': ( + "String with utf-8 charactër", + b'String with utf-8 charact\xebr' + ), + + 'surrogate_and_utf8': ( + "String that cannot be ëncod\udcc3\udcabd with surrogateescape", + b'String that cannot be \xebncod\\udcc3\\udcabd with surrogateescape' + ), + + 'out_of_range_surrogate': ( + "String with \udfff cannot be encoded with surrogateescape", + b'String with \\udfff cannot be encoded with surrogateescape' + ), + } + + def get_payload_surrogate_as_gh_94606(self, msg, expected): + """test for GH issue 94606""" + m = self._str_msg(msg) + payload = m.get_payload(decode=True) + self.assertEqual(expected, payload) + + +class TestEmailMessage(TestEmailMessageBase, TestEmailBase): + message = EmailMessage + + def test_set_content_adds_MIME_Version(self): + m = self._str_msg('') + cm = self._TestContentManager() + self.assertNotIn('MIME-Version', m) + m.set_content(content_manager=cm) + self.assertEqual(m['MIME-Version'], '1.0') + + class _MIME_Version_adding_CM: + def set_content(self, msg, *args, **kw): + msg['MIME-Version'] = '1.0' + + def test_set_content_does_not_duplicate_MIME_Version(self): + m = self._str_msg('') + cm = self._MIME_Version_adding_CM() + self.assertNotIn('MIME-Version', m) + m.set_content(content_manager=cm) + self.assertEqual(m['MIME-Version'], '1.0') + + def test_as_string_uses_max_header_length_by_default(self): + m = self._str_msg('Subject: long line' + ' ab'*50 + '\n\n') + self.assertEqual(len(m.as_string().strip().splitlines()), 3) + + def test_as_string_allows_maxheaderlen(self): + m = self._str_msg('Subject: long line' + ' ab'*50 + '\n\n') + self.assertEqual(len(m.as_string(maxheaderlen=0).strip().splitlines()), + 1) + self.assertEqual(len(m.as_string(maxheaderlen=34).strip().splitlines()), + 6) + + def test_as_string_unixform(self): + m = self._str_msg('test') + m.set_unixfrom('From foo@bar Thu Jan 1 00:00:00 1970') + self.assertEqual(m.as_string(unixfrom=True), + 'From foo@bar Thu Jan 1 00:00:00 1970\n\ntest') + self.assertEqual(m.as_string(unixfrom=False), '\ntest') + + def test_str_defaults_to_policy_max_line_length(self): + m = self._str_msg('Subject: long line' + ' ab'*50 + '\n\n') + self.assertEqual(len(str(m).strip().splitlines()), 3) + + def test_str_defaults_to_utf8(self): + m = EmailMessage() + m['Subject'] = 'unicöde' + self.assertEqual(str(m), 'Subject: unicöde\n\n') + + def test_folding_with_utf8_encoding_1(self): + # bpo-36520 + # + # Fold a line that contains UTF-8 words before + # and after the whitespace fold point, where the + # line length limit is reached within an ASCII + # word. + + m = EmailMessage() + m['Subject'] = 'Hello Wörld! Hello Wörld! ' \ + 'Hello Wörld! Hello Wörld!Hello Wörld!' + self.assertEqual(bytes(m), + b'Subject: Hello =?utf-8?q?W=C3=B6rld!_Hello_W' + b'=C3=B6rld!_Hello_W=C3=B6rld!?=\n' + b' Hello =?utf-8?q?W=C3=B6rld!Hello_W=C3=B6rld!?=\n\n') + + + def test_folding_with_utf8_encoding_2(self): + # bpo-36520 + # + # Fold a line that contains UTF-8 words before + # and after the whitespace fold point, where the + # line length limit is reached at the end of an + # encoded word. + + m = EmailMessage() + m['Subject'] = 'Hello Wörld! Hello Wörld! ' \ + 'Hello Wörlds123! Hello Wörld!Hello Wörld!' + self.assertEqual(bytes(m), + b'Subject: Hello =?utf-8?q?W=C3=B6rld!_Hello_W' + b'=C3=B6rld!_Hello_W=C3=B6rlds123!?=\n' + b' Hello =?utf-8?q?W=C3=B6rld!Hello_W=C3=B6rld!?=\n\n') + + def test_folding_with_utf8_encoding_3(self): + # bpo-36520 + # + # Fold a line that contains UTF-8 words before + # and after the whitespace fold point, where the + # line length limit is reached at the end of the + # first word. + + m = EmailMessage() + m['Subject'] = 'Hello-Wörld!-Hello-Wörld!-Hello-Wörlds123! ' \ + 'Hello Wörld!Hello Wörld!' + self.assertEqual(bytes(m), \ + b'Subject: =?utf-8?q?Hello-W=C3=B6rld!-Hello-W' + b'=C3=B6rld!-Hello-W=C3=B6rlds123!?=\n' + b' Hello =?utf-8?q?W=C3=B6rld!Hello_W=C3=B6rld!?=\n\n') + + def test_folding_with_utf8_encoding_4(self): + # bpo-36520 + # + # Fold a line that contains UTF-8 words before + # and after the fold point, where the first + # word is UTF-8 and the fold point is within + # the word. + + m = EmailMessage() + m['Subject'] = 'Hello-Wörld!-Hello-Wörld!-Hello-Wörlds123!-Hello' \ + ' Wörld!Hello Wörld!' + self.assertEqual(bytes(m), + b'Subject: =?utf-8?q?Hello-W=C3=B6rld!-Hello-W' + b'=C3=B6rld!-Hello-W=C3=B6rlds123!?=\n' + b' =?utf-8?q?-Hello_W=C3=B6rld!Hello_W=C3=B6rld!?=\n\n') + + def test_folding_with_utf8_encoding_5(self): + # bpo-36520 + # + # Fold a line that contains a UTF-8 word after + # the fold point. + + m = EmailMessage() + m['Subject'] = '123456789 123456789 123456789 123456789 123456789' \ + ' 123456789 123456789 Hello Wörld!' + self.assertEqual(bytes(m), + b'Subject: 123456789 123456789 123456789 123456789' + b' 123456789 123456789 123456789\n' + b' Hello =?utf-8?q?W=C3=B6rld!?=\n\n') + + def test_folding_with_utf8_encoding_6(self): + # bpo-36520 + # + # Fold a line that contains a UTF-8 word before + # the fold point and ASCII words after + + m = EmailMessage() + m['Subject'] = '123456789 123456789 123456789 123456789 Hello Wörld!' \ + ' 123456789 123456789 123456789 123456789 123456789' \ + ' 123456789' + self.assertEqual(bytes(m), + b'Subject: 123456789 123456789 123456789 123456789' + b' Hello =?utf-8?q?W=C3=B6rld!?=\n 123456789 ' + b'123456789 123456789 123456789 123456789 ' + b'123456789\n\n') + + def test_folding_with_utf8_encoding_7(self): + # bpo-36520 + # + # Fold a line twice that contains UTF-8 words before + # and after the first fold point, and ASCII words + # after the second fold point. + + m = EmailMessage() + m['Subject'] = '123456789 123456789 Hello Wörld! Hello Wörld! ' \ + '123456789-123456789 123456789 Hello Wörld! 123456789' \ + ' 123456789' + self.assertEqual(bytes(m), + b'Subject: 123456789 123456789 Hello =?utf-8?q?' + b'W=C3=B6rld!_Hello_W=C3=B6rld!?=\n' + b' 123456789-123456789 123456789 Hello ' + b'=?utf-8?q?W=C3=B6rld!?= 123456789\n 123456789\n\n') + + def test_folding_with_utf8_encoding_8(self): + # bpo-36520 + # + # Fold a line twice that contains UTF-8 words before + # the first fold point, and ASCII words after the + # first fold point, and UTF-8 words after the second + # fold point. + + m = EmailMessage() + m['Subject'] = '123456789 123456789 Hello Wörld! Hello Wörld! ' \ + '123456789 123456789 123456789 123456789 123456789 ' \ + '123456789-123456789 123456789 Hello Wörld! 123456789' \ + ' 123456789' + self.assertEqual(bytes(m), + b'Subject: 123456789 123456789 Hello ' + b'=?utf-8?q?W=C3=B6rld!_Hello_W=C3=B6rld!?=\n 123456789 ' + b'123456789 123456789 123456789 123456789 ' + b'123456789-123456789\n 123456789 Hello ' + b'=?utf-8?q?W=C3=B6rld!?= 123456789 123456789\n\n') + + def test_folding_with_short_nospace_1(self): + # bpo-36520 + # + # Fold a line that contains a long whitespace after + # the fold point. + + m = EmailMessage(policy.default) + m['Message-ID'] = '123456789' * 3 + parsed_msg = message_from_bytes(m.as_bytes(), policy=policy.default) + self.assertEqual(parsed_msg['Message-ID'], m['Message-ID']) + + def test_folding_with_long_nospace_default_policy_1(self): + # Fixed: https://github.com/python/cpython/issues/124452 + # + # When the value is too long, it should be converted back + # to its original form without any modifications. + + m = EmailMessage(policy.default) + message = '123456789' * 10 + m['Message-ID'] = message + self.assertEqual(m.as_bytes(), + f'Message-ID:\n {message}\n\n'.encode()) + parsed_msg = message_from_bytes(m.as_bytes(), policy=policy.default) + self.assertEqual(parsed_msg['Message-ID'], m['Message-ID']) + + def test_folding_with_long_nospace_compat32_policy_1(self): + m = EmailMessage(policy.compat32) + message = '123456789' * 10 + m['Message-ID'] = message + parsed_msg = message_from_bytes(m.as_bytes(), policy=policy.default) + self.assertEqual(parsed_msg['Message-ID'], m['Message-ID']) + + def test_folding_with_long_nospace_smtp_policy_1(self): + m = EmailMessage(policy.SMTP) + message = '123456789' * 10 + m['Message-ID'] = message + parsed_msg = message_from_bytes(m.as_bytes(), policy=policy.default) + self.assertEqual(parsed_msg['Message-ID'], m['Message-ID']) + + def test_folding_with_long_nospace_http_policy_1(self): + m = EmailMessage(policy.HTTP) + message = '123456789' * 10 + m['Message-ID'] = message + parsed_msg = message_from_bytes(m.as_bytes(), policy=policy.default) + self.assertEqual(parsed_msg['Message-ID'], m['Message-ID']) + + def test_no_wrapping_max_line_length(self): + # Test that falsey 'max_line_length' are converted to sys.maxsize. + for n in [0, None]: + with self.subTest(max_line_length=n): + self.do_test_no_wrapping_max_line_length(n) + + def do_test_no_wrapping_max_line_length(self, falsey): + self.assertFalse(falsey) + pol = policy.default.clone(max_line_length=falsey) + subj = "S" * 100 + body = "B" * 100 + msg = EmailMessage(policy=pol) + msg["From"] = "a@ex.com" + msg["To"] = "b@ex.com" + msg["Subject"] = subj + msg.set_content(body) + + raw = msg.as_bytes() + self.assertNotIn(b"=\n", raw, + "Found fold indicator; wrapping not disabled") + + parsed = message_from_bytes(raw, policy=policy.default) + self.assertEqual(parsed["Subject"], subj) + parsed_body = parsed.get_body().get_content().rstrip('\n') + self.assertEqual(parsed_body, body) + + def test_get_body_malformed(self): + """test for bpo-42892""" + msg = textwrap.dedent("""\ + Message-ID: <674392CA.4347091@email.au> + Date: Wed, 08 Nov 2017 08:50:22 +0700 + From: Foo Bar + MIME-Version: 1.0 + To: email@email.com + Subject: Python Email + Content-Type: multipart/mixed; + boundary="------------879045806563892972123996" + X-Global-filter:Messagescannedforspamandviruses:passedalltests + + This is a multi-part message in MIME format. + --------------879045806563892972123996 + Content-Type: text/plain; charset=ISO-8859-1; format=flowed + Content-Transfer-Encoding: 7bit + + Your message is ready to be sent with the following file or link + attachments: + XU89 - 08.11.2017 + """) + m = self._str_msg(msg) + # In bpo-42892, this would raise + # AttributeError: 'str' object has no attribute 'is_attachment' + m.get_body() + + def test_get_bytes_payload_with_quoted_printable_encoding(self): + # We use a memoryview to avoid directly changing the private payload + # and to prevent using the dedicated paths for string or bytes objects. + payload = memoryview(b'Some payload') + m = self._make_message() + m.add_header('Content-Transfer-Encoding', 'quoted-printable') + m.set_payload(payload) + self.assertEqual(m.get_payload(decode=True), payload) + + +class TestMIMEPart(TestEmailMessageBase, TestEmailBase): + # Doing the full test run here may seem a bit redundant, since the two + # classes are almost identical. But what if they drift apart? So we do + # the full tests so that any future drift doesn't introduce bugs. + message = MIMEPart + + def test_set_content_does_not_add_MIME_Version(self): + m = self._str_msg('') + cm = self._TestContentManager() + self.assertNotIn('MIME-Version', m) + m.set_content(content_manager=cm) + self.assertNotIn('MIME-Version', m) + + def test_string_payload_with_multipart_content_type(self): + msg = message_from_string(textwrap.dedent("""\ + Content-Type: multipart/mixed; charset="utf-8" + + sample text + """), policy=policy.default) + attachments = msg.iter_attachments() + self.assertEqual(list(attachments), []) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test_parser.py b/Lib/test/test_email/test_parser.py new file mode 100644 index 00000000000..06c86408ab5 --- /dev/null +++ b/Lib/test/test_email/test_parser.py @@ -0,0 +1,110 @@ +import io +import email +import unittest +from email.message import Message, EmailMessage +from email.policy import default +from test.test_email import TestEmailBase + + +class TestCustomMessage(TestEmailBase): + + class MyMessage(Message): + def __init__(self, policy): + self.check_policy = policy + super().__init__() + + MyPolicy = TestEmailBase.policy.clone(linesep='boo') + + def test_custom_message_gets_policy_if_possible_from_string(self): + msg = email.message_from_string("Subject: bogus\n\nmsg\n", + self.MyMessage, + policy=self.MyPolicy) + self.assertIsInstance(msg, self.MyMessage) + self.assertIs(msg.check_policy, self.MyPolicy) + + def test_custom_message_gets_policy_if_possible_from_file(self): + source_file = io.StringIO("Subject: bogus\n\nmsg\n") + msg = email.message_from_file(source_file, + self.MyMessage, + policy=self.MyPolicy) + self.assertIsInstance(msg, self.MyMessage) + self.assertIs(msg.check_policy, self.MyPolicy) + + # XXX add tests for other functions that take Message arg. + + +class TestParserBase: + + def test_only_split_on_cr_lf(self): + # The unicode line splitter splits on unicode linebreaks, which are + # more numerous than allowed by the email RFCs; make sure we are only + # splitting on those two. + for parser in self.parsers: + with self.subTest(parser=parser.__name__): + msg = parser( + "Next-Line: not\x85broken\r\n" + "Null: not\x00broken\r\n" + "Vertical-Tab: not\vbroken\r\n" + "Form-Feed: not\fbroken\r\n" + "File-Separator: not\x1Cbroken\r\n" + "Group-Separator: not\x1Dbroken\r\n" + "Record-Separator: not\x1Ebroken\r\n" + "Line-Separator: not\u2028broken\r\n" + "Paragraph-Separator: not\u2029broken\r\n" + "\r\n", + policy=default, + ) + self.assertEqual(msg.items(), [ + ("Next-Line", "not\x85broken"), + ("Null", "not\x00broken"), + ("Vertical-Tab", "not\vbroken"), + ("Form-Feed", "not\fbroken"), + ("File-Separator", "not\x1Cbroken"), + ("Group-Separator", "not\x1Dbroken"), + ("Record-Separator", "not\x1Ebroken"), + ("Line-Separator", "not\u2028broken"), + ("Paragraph-Separator", "not\u2029broken"), + ]) + self.assertEqual(msg.get_payload(), "") + + class MyMessage(EmailMessage): + pass + + def test_custom_message_factory_on_policy(self): + for parser in self.parsers: + with self.subTest(parser=parser.__name__): + MyPolicy = default.clone(message_factory=self.MyMessage) + msg = parser("To: foo\n\ntest", policy=MyPolicy) + self.assertIsInstance(msg, self.MyMessage) + + def test_factory_arg_overrides_policy(self): + for parser in self.parsers: + with self.subTest(parser=parser.__name__): + MyPolicy = default.clone(message_factory=self.MyMessage) + msg = parser("To: foo\n\ntest", Message, policy=MyPolicy) + self.assertNotIsInstance(msg, self.MyMessage) + self.assertIsInstance(msg, Message) + +# Play some games to get nice output in subTest. This code could be clearer +# if staticmethod supported __name__. + +def message_from_file(s, *args, **kw): + f = io.StringIO(s) + return email.message_from_file(f, *args, **kw) + +class TestParser(TestParserBase, TestEmailBase): + parsers = (email.message_from_string, message_from_file) + +def message_from_bytes(s, *args, **kw): + return email.message_from_bytes(s.encode(), *args, **kw) + +def message_from_binary_file(s, *args, **kw): + f = io.BytesIO(s.encode()) + return email.message_from_binary_file(f, *args, **kw) + +class TestBytesParser(TestParserBase, TestEmailBase): + parsers = (message_from_bytes, message_from_binary_file) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test_pickleable.py b/Lib/test/test_email/test_pickleable.py new file mode 100644 index 00000000000..16b44671146 --- /dev/null +++ b/Lib/test/test_email/test_pickleable.py @@ -0,0 +1,76 @@ +import unittest +import textwrap +import copy +import pickle +import email +import email.message +from email import policy +from email.headerregistry import HeaderRegistry +from test.test_email import TestEmailBase, parameterize + + +@parameterize +class TestPickleCopyHeader(TestEmailBase): + + header_factory = HeaderRegistry() + + unstructured = header_factory('subject', 'this is a test') + + header_params = { + 'subject': ('subject', 'this is a test'), + 'from': ('from', 'frodo@mordor.net'), + 'to': ('to', 'a: k@b.com, y@z.com;, j@f.com'), + 'date': ('date', 'Tue, 29 May 2012 09:24:26 +1000'), + } + + def header_as_deepcopy(self, name, value): + header = self.header_factory(name, value) + h = copy.deepcopy(header) + self.assertEqual(str(h), str(header)) + + def header_as_pickle(self, name, value): + header = self.header_factory(name, value) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + p = pickle.dumps(header, proto) + h = pickle.loads(p) + self.assertEqual(str(h), str(header)) + + +@parameterize +class TestPickleCopyMessage(TestEmailBase): + + # Message objects are a sequence, so we have to make them a one-tuple in + # msg_params so they get passed to the parameterized test method as a + # single argument instead of as a list of headers. + msg_params = {} + + # Note: there will be no custom header objects in the parsed message. + msg_params['parsed'] = (email.message_from_string(textwrap.dedent("""\ + Date: Tue, 29 May 2012 09:24:26 +1000 + From: frodo@mordor.net + To: bilbo@underhill.org + Subject: help + + I think I forgot the ring. + """), policy=policy.default),) + + msg_params['created'] = (email.message.Message(policy=policy.default),) + msg_params['created'][0]['Date'] = 'Tue, 29 May 2012 09:24:26 +1000' + msg_params['created'][0]['From'] = 'frodo@mordor.net' + msg_params['created'][0]['To'] = 'bilbo@underhill.org' + msg_params['created'][0]['Subject'] = 'help' + msg_params['created'][0].set_payload('I think I forgot the ring.') + + def msg_as_deepcopy(self, msg): + msg2 = copy.deepcopy(msg) + self.assertEqual(msg2.as_string(), msg.as_string()) + + def msg_as_pickle(self, msg): + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + p = pickle.dumps(msg, proto) + msg2 = pickle.loads(p) + self.assertEqual(msg2.as_string(), msg.as_string()) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test_policy.py b/Lib/test/test_email/test_policy.py new file mode 100644 index 00000000000..baa35fd68e4 --- /dev/null +++ b/Lib/test/test_email/test_policy.py @@ -0,0 +1,429 @@ +import io +import types +import textwrap +import unittest +import email.errors +import email.policy +import email.parser +import email.generator +import email.message +from email import headerregistry + +def make_defaults(base_defaults, differences): + defaults = base_defaults.copy() + defaults.update(differences) + return defaults + +class PolicyAPITests(unittest.TestCase): + + longMessage = True + + # Base default values. + compat32_defaults = { + 'max_line_length': 78, + 'linesep': '\n', + 'cte_type': '8bit', + 'raise_on_defect': False, + 'mangle_from_': True, + 'message_factory': None, + 'verify_generated_headers': True, + } + # These default values are the ones set on email.policy.default. + # If any of these defaults change, the docs must be updated. + policy_defaults = compat32_defaults.copy() + policy_defaults.update({ + 'utf8': False, + 'raise_on_defect': False, + 'header_factory': email.policy.EmailPolicy.header_factory, + 'refold_source': 'long', + 'content_manager': email.policy.EmailPolicy.content_manager, + 'mangle_from_': False, + 'message_factory': email.message.EmailMessage, + }) + + # For each policy under test, we give here what we expect the defaults to + # be for that policy. The second argument to make defaults is the + # difference between the base defaults and that for the particular policy. + new_policy = email.policy.EmailPolicy() + policies = { + email.policy.compat32: make_defaults(compat32_defaults, {}), + email.policy.default: make_defaults(policy_defaults, {}), + email.policy.SMTP: make_defaults(policy_defaults, + {'linesep': '\r\n'}), + email.policy.SMTPUTF8: make_defaults(policy_defaults, + {'linesep': '\r\n', + 'utf8': True}), + email.policy.HTTP: make_defaults(policy_defaults, + {'linesep': '\r\n', + 'max_line_length': None}), + email.policy.strict: make_defaults(policy_defaults, + {'raise_on_defect': True}), + new_policy: make_defaults(policy_defaults, {}), + } + # Creating a new policy creates a new header factory. There is a test + # later that proves this. + policies[new_policy]['header_factory'] = new_policy.header_factory + + def test_defaults(self): + for policy, expected in self.policies.items(): + for attr, value in expected.items(): + with self.subTest(policy=policy, attr=attr): + self.assertEqual(getattr(policy, attr), value, + ("change {} docs/docstrings if defaults have " + "changed").format(policy)) + + def test_all_attributes_covered(self): + for policy, expected in self.policies.items(): + for attr in dir(policy): + with self.subTest(policy=policy, attr=attr): + if (attr.startswith('_') or + isinstance(getattr(email.policy.EmailPolicy, attr), + types.FunctionType)): + continue + else: + self.assertIn(attr, expected, + "{} is not fully tested".format(attr)) + + def test_abc(self): + with self.assertRaises(TypeError) as cm: + email.policy.Policy() + msg = str(cm.exception) + abstract_methods = ('fold', + 'fold_binary', + 'header_fetch_parse', + 'header_source_parse', + 'header_store_parse') + for method in abstract_methods: + self.assertIn(method, msg) + + def test_policy_is_immutable(self): + for policy, defaults in self.policies.items(): + for attr in defaults: + with self.assertRaisesRegex(AttributeError, attr+".*read-only"): + setattr(policy, attr, None) + with self.assertRaisesRegex(AttributeError, 'no attribute.*foo'): + policy.foo = None + + def test_set_policy_attrs_when_cloned(self): + # None of the attributes has a default value of None, so we set them + # all to None in the clone call and check that it worked. + for policyclass, defaults in self.policies.items(): + testattrdict = {attr: None for attr in defaults} + policy = policyclass.clone(**testattrdict) + for attr in defaults: + self.assertIsNone(getattr(policy, attr)) + + def test_reject_non_policy_keyword_when_called(self): + for policyclass in self.policies: + with self.assertRaises(TypeError): + policyclass(this_keyword_should_not_be_valid=None) + with self.assertRaises(TypeError): + policyclass(newtline=None) + + def test_policy_addition(self): + expected = self.policy_defaults.copy() + p1 = email.policy.default.clone(max_line_length=100) + p2 = email.policy.default.clone(max_line_length=50) + added = p1 + p2 + expected.update(max_line_length=50) + for attr, value in expected.items(): + self.assertEqual(getattr(added, attr), value) + added = p2 + p1 + expected.update(max_line_length=100) + for attr, value in expected.items(): + self.assertEqual(getattr(added, attr), value) + added = added + email.policy.default + for attr, value in expected.items(): + self.assertEqual(getattr(added, attr), value) + + def test_fold_utf8(self): + expected_ascii = 'Subject: =?utf-8?q?=C3=A1?=\n' + expected_utf8 = 'Subject: á\n' + + msg = email.message.EmailMessage() + s = 'á' + msg['Subject'] = s + + p_ascii = email.policy.default.clone() + p_utf8 = email.policy.default.clone(utf8=True) + + self.assertEqual(p_ascii.fold('Subject', msg['Subject']), expected_ascii) + self.assertEqual(p_utf8.fold('Subject', msg['Subject']), expected_utf8) + + self.assertEqual(p_ascii.fold('Subject', s), expected_ascii) + self.assertEqual(p_utf8.fold('Subject', s), expected_utf8) + + def test_fold_zero_max_line_length(self): + expected = 'Subject: =?utf-8?q?=C3=A1?=\n' + + msg = email.message.EmailMessage() + msg['Subject'] = 'á' + + p1 = email.policy.default.clone(max_line_length=0) + p2 = email.policy.default.clone(max_line_length=None) + + self.assertEqual(p1.fold('Subject', msg['Subject']), expected) + self.assertEqual(p2.fold('Subject', msg['Subject']), expected) + + def test_register_defect(self): + class Dummy: + def __init__(self): + self.defects = [] + obj = Dummy() + defect = object() + policy = email.policy.EmailPolicy() + policy.register_defect(obj, defect) + self.assertEqual(obj.defects, [defect]) + defect2 = object() + policy.register_defect(obj, defect2) + self.assertEqual(obj.defects, [defect, defect2]) + + class MyObj: + def __init__(self): + self.defects = [] + + class MyDefect(Exception): + pass + + def test_handle_defect_raises_on_strict(self): + foo = self.MyObj() + defect = self.MyDefect("the telly is broken") + with self.assertRaisesRegex(self.MyDefect, "the telly is broken"): + email.policy.strict.handle_defect(foo, defect) + + def test_handle_defect_registers_defect(self): + foo = self.MyObj() + defect1 = self.MyDefect("one") + email.policy.default.handle_defect(foo, defect1) + self.assertEqual(foo.defects, [defect1]) + defect2 = self.MyDefect("two") + email.policy.default.handle_defect(foo, defect2) + self.assertEqual(foo.defects, [defect1, defect2]) + + class MyPolicy(email.policy.EmailPolicy): + defects = None + def __init__(self, *args, **kw): + super().__init__(*args, defects=[], **kw) + def register_defect(self, obj, defect): + self.defects.append(defect) + + def test_overridden_register_defect_still_raises(self): + foo = self.MyObj() + defect = self.MyDefect("the telly is broken") + with self.assertRaisesRegex(self.MyDefect, "the telly is broken"): + self.MyPolicy(raise_on_defect=True).handle_defect(foo, defect) + + def test_overridden_register_defect_works(self): + foo = self.MyObj() + defect1 = self.MyDefect("one") + my_policy = self.MyPolicy() + my_policy.handle_defect(foo, defect1) + self.assertEqual(my_policy.defects, [defect1]) + self.assertEqual(foo.defects, []) + defect2 = self.MyDefect("two") + my_policy.handle_defect(foo, defect2) + self.assertEqual(my_policy.defects, [defect1, defect2]) + self.assertEqual(foo.defects, []) + + def test_default_header_factory(self): + h = email.policy.default.header_factory('Test', 'test') + self.assertEqual(h.name, 'Test') + self.assertIsInstance(h, headerregistry.UnstructuredHeader) + self.assertIsInstance(h, headerregistry.BaseHeader) + + class Foo: + parse = headerregistry.UnstructuredHeader.parse + + def test_each_Policy_gets_unique_factory(self): + policy1 = email.policy.EmailPolicy() + policy2 = email.policy.EmailPolicy() + policy1.header_factory.map_to_type('foo', self.Foo) + h = policy1.header_factory('foo', 'test') + self.assertIsInstance(h, self.Foo) + self.assertNotIsInstance(h, headerregistry.UnstructuredHeader) + h = policy2.header_factory('foo', 'test') + self.assertNotIsInstance(h, self.Foo) + self.assertIsInstance(h, headerregistry.UnstructuredHeader) + + def test_clone_copies_factory(self): + policy1 = email.policy.EmailPolicy() + policy2 = policy1.clone() + policy1.header_factory.map_to_type('foo', self.Foo) + h = policy1.header_factory('foo', 'test') + self.assertIsInstance(h, self.Foo) + h = policy2.header_factory('foo', 'test') + self.assertIsInstance(h, self.Foo) + + def test_new_factory_overrides_default(self): + mypolicy = email.policy.EmailPolicy() + myfactory = mypolicy.header_factory + newpolicy = mypolicy + email.policy.strict + self.assertEqual(newpolicy.header_factory, myfactory) + newpolicy = email.policy.strict + mypolicy + self.assertEqual(newpolicy.header_factory, myfactory) + + def test_adding_default_policies_preserves_default_factory(self): + newpolicy = email.policy.default + email.policy.strict + self.assertEqual(newpolicy.header_factory, + email.policy.EmailPolicy.header_factory) + self.assertEqual(newpolicy.__dict__, {'raise_on_defect': True}) + + def test_non_ascii_chars_do_not_cause_inf_loop(self): + policy = email.policy.default.clone(max_line_length=20) + actual = policy.fold('Subject', 'ą' * 12) + self.assertEqual( + actual, + 'Subject: \n' + + 12 * ' =?utf-8?q?=C4=85?=\n') + + def test_short_maxlen_error(self): + # RFC 2047 chrome takes up 7 characters, plus the length of the charset + # name, so folding should fail if maxlen is lower than the minimum + # required length for a line. + + # Note: This is only triggered when there is a single word longer than + # max_line_length, hence the 1234567890 at the end of this whimsical + # subject. This is because when we encounter a word longer than + # max_line_length, it is broken down into encoded words to fit + # max_line_length. If the max_line_length isn't large enough to even + # contain the RFC 2047 chrome (`?=?q??=`), we fail. + subject = "Melt away the pounds with this one simple trick! 1234567890" + + for maxlen in [3, 7, 9]: + with self.subTest(maxlen=maxlen): + policy = email.policy.default.clone(max_line_length=maxlen) + with self.assertRaises(email.errors.HeaderParseError): + policy.fold("Subject", subject) + + def test_verify_generated_headers(self): + """Turning protection off allows header injection""" + policy = email.policy.default.clone(verify_generated_headers=False) + for text in ( + 'Header: Value\r\nBad: Injection\r\n', + 'Header: NoNewLine' + ): + with self.subTest(text=text): + message = email.message_from_string( + "Header: Value\r\n\r\nBody", + policy=policy, + ) + class LiteralHeader(str): + name = 'Header' + def fold(self, **kwargs): + return self + + del message['Header'] + message['Header'] = LiteralHeader(text) + + self.assertEqual( + message.as_string(), + f"{text}\nBody", + ) + + # XXX: Need subclassing tests. + # For adding subclassed objects, make sure the usual rules apply (subclass + # wins), but that the order still works (right overrides left). + + +class TestException(Exception): + pass + +class TestPolicyPropagation(unittest.TestCase): + + # The abstract methods are used by the parser but not by the wrapper + # functions that call it, so if the exception gets raised we know that the + # policy was actually propagated all the way to feedparser. + class MyPolicy(email.policy.Policy): + def badmethod(self, *args, **kw): + raise TestException("test") + fold = fold_binary = header_fetch_parser = badmethod + header_source_parse = header_store_parse = badmethod + + def test_message_from_string(self): + with self.assertRaisesRegex(TestException, "^test$"): + email.message_from_string("Subject: test\n\n", + policy=self.MyPolicy) + + def test_message_from_bytes(self): + with self.assertRaisesRegex(TestException, "^test$"): + email.message_from_bytes(b"Subject: test\n\n", + policy=self.MyPolicy) + + def test_message_from_file(self): + f = io.StringIO('Subject: test\n\n') + with self.assertRaisesRegex(TestException, "^test$"): + email.message_from_file(f, policy=self.MyPolicy) + + def test_message_from_binary_file(self): + f = io.BytesIO(b'Subject: test\n\n') + with self.assertRaisesRegex(TestException, "^test$"): + email.message_from_binary_file(f, policy=self.MyPolicy) + + # These are redundant, but we need them for black-box completeness. + + def test_parser(self): + p = email.parser.Parser(policy=self.MyPolicy) + with self.assertRaisesRegex(TestException, "^test$"): + p.parsestr('Subject: test\n\n') + + def test_bytes_parser(self): + p = email.parser.BytesParser(policy=self.MyPolicy) + with self.assertRaisesRegex(TestException, "^test$"): + p.parsebytes(b'Subject: test\n\n') + + # Now that we've established that all the parse methods get the + # policy in to feedparser, we can use message_from_string for + # the rest of the propagation tests. + + def _make_msg(self, source='Subject: test\n\n', policy=None): + self.policy = email.policy.default.clone() if policy is None else policy + return email.message_from_string(source, policy=self.policy) + + def test_parser_propagates_policy_to_message(self): + msg = self._make_msg() + self.assertIs(msg.policy, self.policy) + + def test_parser_propagates_policy_to_sub_messages(self): + msg = self._make_msg(textwrap.dedent("""\ + Subject: mime test + MIME-Version: 1.0 + Content-Type: multipart/mixed, boundary="XXX" + + --XXX + Content-Type: text/plain + + test + --XXX + Content-Type: text/plain + + test2 + --XXX-- + """)) + for part in msg.walk(): + self.assertIs(part.policy, self.policy) + + def test_message_policy_propagates_to_generator(self): + msg = self._make_msg("Subject: test\nTo: foo\n\n", + policy=email.policy.default.clone(linesep='X')) + s = io.StringIO() + g = email.generator.Generator(s) + g.flatten(msg) + self.assertEqual(s.getvalue(), "Subject: testXTo: fooXX") + + def test_message_policy_used_by_as_string(self): + msg = self._make_msg("Subject: test\nTo: foo\n\n", + policy=email.policy.default.clone(linesep='X')) + self.assertEqual(msg.as_string(), "Subject: testXTo: fooXX") + + +class TestConcretePolicies(unittest.TestCase): + + def test_header_store_parse_rejects_newlines(self): + instance = email.policy.EmailPolicy() + self.assertRaises(ValueError, + instance.header_store_parse, + 'From', 'spam\negg@foo.py') + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test_utils.py b/Lib/test/test_email/test_utils.py new file mode 100644 index 00000000000..d04b3909efa --- /dev/null +++ b/Lib/test/test_email/test_utils.py @@ -0,0 +1,186 @@ +import datetime +from email import utils +import test.support +import time +import unittest +import sys +import os.path +import zoneinfo + +class DateTimeTests(unittest.TestCase): + + datestring = 'Sun, 23 Sep 2001 20:10:55' + dateargs = (2001, 9, 23, 20, 10, 55) + offsetstring = ' -0700' + utcoffset = datetime.timedelta(hours=-7) + tz = datetime.timezone(utcoffset) + naive_dt = datetime.datetime(*dateargs) + aware_dt = datetime.datetime(*dateargs, tzinfo=tz) + + def test_naive_datetime(self): + self.assertEqual(utils.format_datetime(self.naive_dt), + self.datestring + ' -0000') + + def test_aware_datetime(self): + self.assertEqual(utils.format_datetime(self.aware_dt), + self.datestring + self.offsetstring) + + def test_usegmt(self): + utc_dt = datetime.datetime(*self.dateargs, + tzinfo=datetime.timezone.utc) + self.assertEqual(utils.format_datetime(utc_dt, usegmt=True), + self.datestring + ' GMT') + + def test_usegmt_with_naive_datetime_raises(self): + with self.assertRaises(ValueError): + utils.format_datetime(self.naive_dt, usegmt=True) + + def test_usegmt_with_non_utc_datetime_raises(self): + with self.assertRaises(ValueError): + utils.format_datetime(self.aware_dt, usegmt=True) + + def test_parsedate_to_datetime(self): + self.assertEqual( + utils.parsedate_to_datetime(self.datestring + self.offsetstring), + self.aware_dt) + + def test_parsedate_to_datetime_naive(self): + self.assertEqual( + utils.parsedate_to_datetime(self.datestring + ' -0000'), + self.naive_dt) + + def test_parsedate_to_datetime_with_invalid_raises_valueerror(self): + # See also test_parsedate_returns_None_for_invalid_strings in test_email. + invalid_dates = [ + '', + ' ', + '0', + 'A Complete Waste of Time', + 'Wed, 3 Apr 2002 12.34.56.78+0800' + 'Tue, 06 Jun 2017 27:39:33 +0600', + 'Tue, 06 Jun 2017 07:39:33 +2600', + 'Tue, 06 Jun 2017 27:39:33', + '17 June , 2022', + 'Friday, -Nov-82 16:14:55 EST', + 'Friday, Nov--82 16:14:55 EST', + 'Friday, 19-Nov- 16:14:55 EST', + ] + for dtstr in invalid_dates: + with self.subTest(dtstr=dtstr): + self.assertRaises(ValueError, utils.parsedate_to_datetime, dtstr) + +class LocaltimeTests(unittest.TestCase): + + def test_localtime_is_tz_aware_daylight_true(self): + test.support.patch(self, time, 'daylight', True) + t = utils.localtime() + self.assertIsNotNone(t.tzinfo) + + def test_localtime_is_tz_aware_daylight_false(self): + test.support.patch(self, time, 'daylight', False) + t = utils.localtime() + self.assertIsNotNone(t.tzinfo) + + def test_localtime_daylight_true_dst_false(self): + test.support.patch(self, time, 'daylight', True) + t0 = datetime.datetime(2012, 3, 12, 1, 1) + t1 = utils.localtime(t0) + t2 = utils.localtime(t1) + self.assertEqual(t1, t2) + + def test_localtime_daylight_false_dst_false(self): + test.support.patch(self, time, 'daylight', False) + t0 = datetime.datetime(2012, 3, 12, 1, 1) + t1 = utils.localtime(t0) + t2 = utils.localtime(t1) + self.assertEqual(t1, t2) + + @test.support.run_with_tz('Europe/Minsk') + def test_localtime_daylight_true_dst_true(self): + test.support.patch(self, time, 'daylight', True) + t0 = datetime.datetime(2012, 3, 12, 1, 1) + t1 = utils.localtime(t0) + t2 = utils.localtime(t1) + self.assertEqual(t1, t2) + + @test.support.run_with_tz('Europe/Minsk') + def test_localtime_daylight_false_dst_true(self): + test.support.patch(self, time, 'daylight', False) + t0 = datetime.datetime(2012, 3, 12, 1, 1) + t1 = utils.localtime(t0) + t2 = utils.localtime(t1) + self.assertEqual(t1, t2) + + @test.support.run_with_tz('EST+05EDT,M3.2.0,M11.1.0') + def test_localtime_epoch_utc_daylight_true(self): + test.support.patch(self, time, 'daylight', True) + t0 = datetime.datetime(1990, 1, 1, tzinfo = datetime.timezone.utc) + t1 = utils.localtime(t0) + t2 = t0 - datetime.timedelta(hours=5) + t2 = t2.replace(tzinfo = datetime.timezone(datetime.timedelta(hours=-5))) + self.assertEqual(t1, t2) + + @test.support.run_with_tz('EST+05EDT,M3.2.0,M11.1.0') + def test_localtime_epoch_utc_daylight_false(self): + test.support.patch(self, time, 'daylight', False) + t0 = datetime.datetime(1990, 1, 1, tzinfo = datetime.timezone.utc) + t1 = utils.localtime(t0) + t2 = t0 - datetime.timedelta(hours=5) + t2 = t2.replace(tzinfo = datetime.timezone(datetime.timedelta(hours=-5))) + self.assertEqual(t1, t2) + + def test_localtime_epoch_notz_daylight_true(self): + test.support.patch(self, time, 'daylight', True) + t0 = datetime.datetime(1990, 1, 1) + t1 = utils.localtime(t0) + t2 = utils.localtime(t0.replace(tzinfo=None)) + self.assertEqual(t1, t2) + + def test_localtime_epoch_notz_daylight_false(self): + test.support.patch(self, time, 'daylight', False) + t0 = datetime.datetime(1990, 1, 1) + t1 = utils.localtime(t0) + t2 = utils.localtime(t0.replace(tzinfo=None)) + self.assertEqual(t1, t2) + + @test.support.run_with_tz('Europe/Kyiv') + def test_variable_tzname(self): + t0 = datetime.datetime(1984, 1, 1, tzinfo=datetime.timezone.utc) + t1 = utils.localtime(t0) + if t1.tzname() in ('Europe', 'UTC'): + self.skipTest("Can't find a Kyiv timezone database") + self.assertEqual(t1.tzname(), 'MSK') + t0 = datetime.datetime(1994, 1, 1, tzinfo=datetime.timezone.utc) + t1 = utils.localtime(t0) + self.assertEqual(t1.tzname(), 'EET') + + def test_isdst_deprecation(self): + with self.assertWarns(DeprecationWarning): + t0 = datetime.datetime(1990, 1, 1) + t1 = utils.localtime(t0, isdst=True) + +# Issue #24836: The timezone files are out of date (pre 2011k) +# on Mac OS X Snow Leopard. +@test.support.requires_mac_ver(10, 7) +class FormatDateTests(unittest.TestCase): + + @test.support.run_with_tz('Europe/Minsk') + def test_formatdate(self): + timeval = time.mktime((2011, 12, 1, 18, 0, 0, 4, 335, 0)) + string = utils.formatdate(timeval, localtime=False, usegmt=False) + self.assertEqual(string, 'Thu, 01 Dec 2011 15:00:00 -0000') + string = utils.formatdate(timeval, localtime=False, usegmt=True) + self.assertEqual(string, 'Thu, 01 Dec 2011 15:00:00 GMT') + + @test.support.run_with_tz('Europe/Minsk') + def test_formatdate_with_localtime(self): + timeval = time.mktime((2011, 1, 1, 18, 0, 0, 6, 1, 0)) + string = utils.formatdate(timeval, localtime=True) + self.assertEqual(string, 'Sat, 01 Jan 2011 18:00:00 +0200') + # Minsk moved from +0200 (with DST) to +0300 (without DST) in 2011 + timeval = time.mktime((2011, 12, 1, 18, 0, 0, 4, 335, 0)) + string = utils.formatdate(timeval, localtime=True) + self.assertEqual(string, 'Thu, 01 Dec 2011 18:00:00 +0300') + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/torture_test.py b/Lib/test/test_email/torture_test.py new file mode 100644 index 00000000000..9cf9362c9b7 --- /dev/null +++ b/Lib/test/test_email/torture_test.py @@ -0,0 +1,127 @@ +# Copyright (C) 2002-2004 Python Software Foundation +# +# A torture test of the email package. This should not be run as part of the +# standard Python test suite since it requires several meg of email messages +# collected in the wild. These source messages are not checked into the +# Python distro, but are available as part of the standalone email package at +# http://sf.net/projects/mimelib + +import sys +import os +import unittest +from io import StringIO + +from test.test_email import TestEmailBase + +import email +from email import __file__ as testfile +from email.iterators import _structure + +def openfile(filename): + from os.path import join, dirname, abspath + path = abspath(join(dirname(testfile), os.pardir, 'moredata', filename)) + return open(path, 'r') + +# Prevent this test from running in the Python distro +def setUpModule(): + try: + openfile('crispin-torture.txt') + except OSError: + raise unittest.SkipTest + + + +class TortureBase(TestEmailBase): + def _msgobj(self, filename): + fp = openfile(filename) + try: + msg = email.message_from_file(fp) + finally: + fp.close() + return msg + + + +class TestCrispinTorture(TortureBase): + # Mark Crispin's torture test from the SquirrelMail project + def test_mondo_message(self): + eq = self.assertEqual + neq = self.ndiffAssertEqual + msg = self._msgobj('crispin-torture.txt') + payload = msg.get_payload() + eq(type(payload), list) + eq(len(payload), 12) + eq(msg.preamble, None) + eq(msg.epilogue, '\n') + # Probably the best way to verify the message is parsed correctly is to + # dump its structure and compare it against the known structure. + fp = StringIO() + _structure(msg, fp=fp) + neq(fp.getvalue(), """\ +multipart/mixed + text/plain + message/rfc822 + multipart/alternative + text/plain + multipart/mixed + text/richtext + application/andrew-inset + message/rfc822 + audio/basic + audio/basic + image/pbm + message/rfc822 + multipart/mixed + multipart/mixed + text/plain + audio/x-sun + multipart/mixed + image/gif + image/gif + application/x-be2 + application/atomicmail + audio/x-sun + message/rfc822 + multipart/mixed + text/plain + image/pgm + text/plain + message/rfc822 + multipart/mixed + text/plain + image/pbm + message/rfc822 + application/postscript + image/gif + message/rfc822 + multipart/mixed + audio/basic + audio/basic + message/rfc822 + multipart/mixed + application/postscript + text/plain + message/rfc822 + multipart/mixed + text/plain + multipart/parallel + image/gif + audio/basic + application/atomicmail + message/rfc822 + audio/x-sun +""") + +def _testclasses(): + mod = sys.modules[__name__] + return [getattr(mod, name) for name in dir(mod) if name.startswith('Test')] + + +def load_tests(loader, tests, pattern): + suite = loader.suiteClass() + for testclass in _testclasses(): + suite.addTest(loader.loadTestsFromTestCase(testclass)) + return suite + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_ensurepip.py b/Lib/test/test_ensurepip.py index a4b36a90d88..6d3c91b0b6d 100644 --- a/Lib/test/test_ensurepip.py +++ b/Lib/test/test_ensurepip.py @@ -6,7 +6,6 @@ import test.support import unittest import unittest.mock -from importlib.resources.abc import Traversable from pathlib import Path import ensurepip diff --git a/Lib/test/test_enum.py b/Lib/test/test_enum.py index 8f4b39b4dbb..0e0cfb96333 100644 --- a/Lib/test/test_enum.py +++ b/Lib/test/test_enum.py @@ -11,7 +11,6 @@ import builtins as bltns from collections import OrderedDict from datetime import date -from functools import partial from enum import Enum, EnumMeta, IntEnum, StrEnum, EnumType, Flag, IntFlag, unique, auto from enum import STRICT, CONFORM, EJECT, KEEP, _simple_enum, _test_simple_enum from enum import verify, UNIQUE, CONTINUOUS, NAMED_FLAGS, ReprEnum @@ -20,7 +19,8 @@ from pickle import dumps, loads, PicklingError, HIGHEST_PROTOCOL from test import support from test.support import ALWAYS_EQ, REPO_ROOT -from test.support import threading_helper +from test.support import threading_helper, cpython_only +from test.support.import_helper import ensure_lazy_imports from datetime import timedelta python_version = sys.version_info[:2] @@ -434,9 +434,9 @@ class Season(self.enum_type): def spam(cls): pass # - self.assertTrue(hasattr(Season, 'spam')) + self.assertHasAttr(Season, 'spam') del Season.spam - self.assertFalse(hasattr(Season, 'spam')) + self.assertNotHasAttr(Season, 'spam') # with self.assertRaises(AttributeError): del Season.SPRING @@ -445,8 +445,6 @@ def spam(cls): with self.assertRaises(AttributeError): del Season.SPRING.name - @unittest.skip('TODO: RUSTPYTHON') - # RuntimeError: Error calling __set_name__ on '_proto_member' instance failed in 'BadSuper' def test_bad_new_super(self): with self.assertRaisesRegex( TypeError, @@ -659,9 +657,6 @@ def __repr__(self): self.assertEqual(str(Generic.item), 'item.test') def test_overridden_str(self): - # TODO: RUSTPYTHON, format(NS.first) does not use __str__ - if self.__class__ in (TestIntFlagFunction, TestIntFlagClass, TestIntEnumFunction, TestIntEnumClass, TestMinimalFloatFunction, TestMinimalFloatClass): - self.skipTest("format(NS.first) does not use __str__") NS = self.NewStrEnum self.assertEqual(str(NS.first), NS.first.name.upper()) self.assertEqual(format(NS.first), NS.first.name.upper()) @@ -1544,22 +1539,6 @@ class Inner(Enum): [Outer.a, Outer.b, Outer.Inner], ) - # TODO: RUSTPYTHON - # AssertionError: FutureWarning not triggered - @unittest.expectedFailure - def test_partial(self): - def func(a, b=5): - return a, b - with self.assertWarnsRegex(FutureWarning, r'partial.*enum\.member') as cm: - class E(Enum): - a = 1 - b = partial(func) - self.assertEqual(cm.filename, __file__) - self.assertIsInstance(E.b, partial) - self.assertEqual(E.b(2), (2, 5)) - with self.assertWarnsRegex(FutureWarning, 'partial'): - self.assertEqual(E.a.b(2), (2, 5)) - def test_enum_with_value_name(self): class Huh(Enum): name = 1 @@ -1769,7 +1748,7 @@ class ThreePart(Enum): self.assertIs(ThreePart((3, 3.0, 'three')), ThreePart.THREE) self.assertIs(ThreePart(3, 3.0, 'three'), ThreePart.THREE) - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: is not + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: is not @reraise_if_not_enum(IntStooges) def test_intenum_from_bytes(self): self.assertIs(IntStooges.from_bytes(b'\x00\x03', 'big'), IntStooges.MOE) @@ -1923,8 +1902,6 @@ def test_wrong_inheritance_order(self): class Wrong(Enum, str): NotHere = 'error before this point' - @unittest.skip('TODO: RUSTPYTHON') - # RuntimeError: Error calling __set_name__ on '_proto_member' instance INVALID in 'RgbColor' def test_raise_custom_error_on_creation(self): class InvalidRgbColorError(ValueError): def __init__(self, r, g, b): @@ -2130,7 +2107,6 @@ class NEI(NamedInt, Enum): test_pickle_dump_load(self.assertIs, NEI.y) test_pickle_dump_load(self.assertIs, NEI) - @unittest.expectedFailure # TODO: RUSTPYTHON; fails on pickle def test_subclasses_with_getnewargs_ex(self): class NamedInt(int): __qualname__ = 'NamedInt' # needed for pickle protocol 4 @@ -2613,8 +2589,6 @@ class Test(Base2): self.assertEqual(Test.flash.flash, 'flashy dynamic') self.assertEqual(Test.flash.value, 1) - @unittest.skip('TODO: RUSTPYTHON') - # RuntimeError: Error calling __set_name__ on '_proto_member' instance grene in 'Color' def test_no_duplicates(self): class UniqueEnum(Enum): def __init__(self, *args): @@ -2679,12 +2653,12 @@ def __new__(cls, value, period): OneDay = day_1 OneWeek = week_1 OneMonth = month_1 - self.assertFalse(hasattr(Period, '_ignore_')) - self.assertFalse(hasattr(Period, 'Period')) - self.assertFalse(hasattr(Period, 'i')) - self.assertTrue(isinstance(Period.day_1, timedelta)) - self.assertTrue(Period.month_1 is Period.day_30) - self.assertTrue(Period.week_4 is Period.day_28) + self.assertNotHasAttr(Period, '_ignore_') + self.assertNotHasAttr(Period, 'Period') + self.assertNotHasAttr(Period, 'i') + self.assertIsInstance(Period.day_1, timedelta) + self.assertIs(Period.month_1, Period.day_30) + self.assertIs(Period.week_4, Period.day_28) def test_nonhash_value(self): class AutoNumberInAList(Enum): @@ -2904,7 +2878,7 @@ class ReformedColor(StrMixin, IntEnum, SomeEnum, AnotherEnum): self.assertEqual(str(ReformedColor.BLUE), 'blue') self.assertEqual(ReformedColor.RED.behavior(), 'booyah') self.assertEqual(ConfusedColor.RED.social(), "what's up?") - self.assertTrue(issubclass(ReformedColor, int)) + self.assertIsSubclass(ReformedColor, int) def test_multiple_inherited_mixin(self): @unique @@ -3000,8 +2974,6 @@ def test_empty_globals(self): local_ls = {} exec(code, global_ns, local_ls) - @unittest.skip('TODO: RUSTPYTHON') - # RuntimeError: Error calling __set_name__ on '_proto_member' instance one in 'FirstFailedStrEnum' def test_strenum(self): class GoodStrEnum(StrEnum): one = '1' @@ -3064,7 +3036,7 @@ class ThirdFailedStrEnum(StrEnum): one = '1' two = b'2', 'ascii', 9 - @unittest.expectedFailure # TODO: RUSTPYTHON; fails on encoding testing : TypeError: Expected type 'str' but 'builtin_function_or_method' found + @unittest.expectedFailure # TODO: RUSTPYTHON; fails on encoding testing : TypeError: Expected type 'str' but 'builtin_function_or_method' found def test_custom_strenum(self): class CustomStrEnum(str, Enum): pass @@ -3126,8 +3098,6 @@ class ThirdFailedStrEnum(CustomStrEnum): one = '1' two = b'2', 'ascii', 9 - @unittest.skip('TODO: RUSTPYTHON') - # RuntimeError: Error calling __set_name__ on '_proto_member' instance key_type in 'Combined' def test_missing_value_error(self): with self.assertRaisesRegex(TypeError, "_value_ not set in __new__"): class Combined(str, Enum): @@ -3414,8 +3384,6 @@ def __new__(cls, c): self.assertEqual(FlagFromChar.a, 158456325028528675187087900672) self.assertEqual(FlagFromChar.a|1, 158456325028528675187087900673) - @unittest.skip('TODO: RUSTPYTHON') - # RuntimeError: Error calling __set_name__ on '_proto_member' instance A in 'MyEnum' def test_init_exception(self): class Base: def __new__(cls, *args): @@ -4004,7 +3972,7 @@ class Color(StrMixin, AllMixin, Flag): self.assertEqual(Color.ALL.value, 7) self.assertEqual(str(Color.BLUE), 'blue') - @unittest.skip('TODO: RUSTPYTHON; flaky test') + @unittest.skip("TODO: RUSTPYTHON; flaky test") @threading_helper.reap_threads @threading_helper.requires_working_threading() def test_unique_composite(self): @@ -4163,7 +4131,6 @@ def test_global_enum_str(self): self.assertEqual(str(NoName.ONE & NoName.TWO), 'NoName(0)') self.assertEqual(str(NoName(0)), 'NoName(0)') - @unittest.expectedFailure # TODO: RUSTPYTHON; format(NewPerm.R) does not use __str__ def test_format(self): Perm = self.Perm self.assertEqual(format(Perm.R, ''), '4') @@ -4526,7 +4493,7 @@ class Color(StrMixin, AllMixin, IntFlag): self.assertEqual(Color.ALL.value, 7) self.assertEqual(str(Color.BLUE), 'blue') - @unittest.skip('TODO: RUSTPYTHON; flaky test') + @unittest.skip("TODO: RUSTPYTHON; flaky test") @threading_helper.reap_threads @threading_helper.requires_working_threading() def test_unique_composite(self): @@ -5075,7 +5042,7 @@ class Color(Enum): MAGENTA = 2 YELLOW = 3 - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; | is a read-only view of the internal mapping. def test_pydoc(self): # indirectly test __objclass__ if StrEnum.__doc__ is None: @@ -5208,7 +5175,6 @@ def test_inspect_signatures(self): ]), ) - @unittest.expectedFailure # TODO: RUSTPYTHON; len is often/always > 256 def test_test_simple_enum(self): @_simple_enum(Enum) class SimpleColor: @@ -5328,6 +5294,10 @@ class MiscTestCase(unittest.TestCase): def test__all__(self): support.check__all__(self, enum, not_exported={'bin', 'show_flag_values'}) + @cpython_only + def test_lazy_import(self): + ensure_lazy_imports("enum", {"functools", "warnings", "inspect", "re"}) + def test_doc_1(self): class Single(Enum): ONE = 1 diff --git a/Lib/test/test_except_star.py b/Lib/test/test_except_star.py index 3e0f8caa9b2..807e7c5a5d6 100644 --- a/Lib/test/test_except_star.py +++ b/Lib/test/test_except_star.py @@ -84,7 +84,8 @@ def test_break_in_except_star(self): if i == 2: break finally: - return 0 + pass + return 0 """) @@ -117,7 +118,8 @@ def test_continue_in_except_star_block_invalid(self): if i == 2: continue finally: - return 0 + pass + return 0 """) def test_return_in_except_star_block_invalid(self): @@ -889,8 +891,7 @@ def test_raise_handle_all_raise_two_unnamed(self): class TestExceptStarExceptionGroupSubclass(ExceptStarTest): - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_except_star_EG_subclass(self): class EG(ExceptionGroup): def __new__(cls, message, excs, code): @@ -1217,4 +1218,4 @@ def test_reraise_unhashable_eg(self): if __name__ == '__main__': - unittest.main() \ No newline at end of file + unittest.main() diff --git a/Lib/test/test_exception_hierarchy.py b/Lib/test/test_exception_hierarchy.py index e8c1c7fd1e7..3472019ea13 100644 --- a/Lib/test/test_exception_hierarchy.py +++ b/Lib/test/test_exception_hierarchy.py @@ -146,8 +146,7 @@ def test_errno_translation(self): self.assertEqual(e.strerror, "File already exists") self.assertEqual(e.filename, "foo.txt") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_blockingioerror(self): args = ("a", "b", "c", "d", "e") for n in range(6): @@ -182,8 +181,7 @@ def test_init_kwdargs(self): self.assertEqual(e.bar, "baz") self.assertEqual(e.args, ("some message",)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_new_overridden(self): e = SubOSErrorWithNew("some message", "baz") self.assertEqual(e.baz, "baz") diff --git a/Lib/test/test_exception_variations.py b/Lib/test/test_exception_variations.py index e103eaf8466..a83a41d2975 100644 --- a/Lib/test/test_exception_variations.py +++ b/Lib/test/test_exception_variations.py @@ -294,8 +294,6 @@ def test_nested_exception_in_finally_with_exception(self): self.assertTrue(hit_except) -# TODO: RUSTPYTHON -''' class ExceptStarTestCases(unittest.TestCase): def test_try_except_else_finally(self): hit_except = False @@ -571,7 +569,7 @@ def test_nested_else_mixed2(self): self.assertFalse(hit_else) self.assertTrue(hit_finally) self.assertTrue(hit_except) -''' + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index ceb94df324e..04af299dea3 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -60,7 +60,7 @@ def raise_catch(self, exc, excname): self.assertEqual(buf1, buf2) self.assertEqual(exc.__name__, excname) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def testRaising(self): self.raise_catch(AttributeError, "AttributeError") self.assertRaises(AttributeError, getattr, sys, "undefined_attribute") @@ -145,7 +145,7 @@ def testRaising(self): self.raise_catch(StopAsyncIteration, "StopAsyncIteration") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def testSyntaxErrorMessage(self): # make sure the right exception message is raised for each of # these code fragments @@ -170,7 +170,7 @@ def ckmsg(src, msg): ckmsg("continue\n", "'continue' not properly in loop") ckmsg("f'{6 0}'", "invalid syntax. Perhaps you forgot a comma?") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def testSyntaxErrorMissingParens(self): def ckmsg(src, msg, exception=SyntaxError): try: @@ -227,14 +227,16 @@ def check(self, src, lineno, offset, end_lineno=None, end_offset=None, encoding= if not isinstance(src, str): src = src.decode(encoding, 'replace') line = src.split('\n')[lineno-1] + if lineno == 1: + line = line.removeprefix('\ufeff') self.assertIn(line, cm.exception.text) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_error_offset_continuation_characters(self): check = self.check check('"\\\n"(1 for c in I,\\\n\\', 2, 2) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def testSyntaxErrorOffset(self): check = self.check check('def fact(x):\n\treturn x!\n', 2, 10) @@ -244,7 +246,9 @@ def testSyntaxErrorOffset(self): check('Python = "\u1e54\xfd\u0163\u0125\xf2\xf1" +', 1, 20) check(b'# -*- coding: cp1251 -*-\nPython = "\xcf\xb3\xf2\xee\xed" +', 2, 19, encoding='cp1251') - check(b'Python = "\xcf\xb3\xf2\xee\xed" +', 1, 10) + check(b'Python = "\xcf\xb3\xf2\xee\xed" +', 1, 12) + check(b'\n\n\nPython = "\xcf\xb3\xf2\xee\xed" +', 4, 12) + check(b'\xef\xbb\xbfPython = "\xcf\xb3\xf2\xee\xed" +', 1, 12) check('x = "a', 1, 5) check('lambda x: x = 2', 1, 1) check('f{a + b + c}', 1, 2) @@ -292,7 +296,7 @@ def baz(): check("pass\npass\npass\n(1+)\npass\npass\npass", 4, 4) check("(1+)", 1, 4) check("[interesting\nfoo()\n", 1, 1) - check(b"\xef\xbb\xbf#coding: utf8\nprint('\xe6\x88\x91')\n", 0, -1) + check(b"\xef\xbb\xbf#coding: utf8\nprint('\xe6\x88\x91')\n", 1, 0) check("""f''' { (123_a) @@ -362,7 +366,7 @@ def test_capi1(): except TypeError as err: co = err.__traceback__.tb_frame.f_code self.assertEqual(co.co_name, "test_capi1") - self.assertTrue(co.co_filename.endswith('test_exceptions.py')) + self.assertEndsWith(co.co_filename, 'test_exceptions.py') else: self.fail("Expected exception") @@ -374,7 +378,7 @@ def test_capi2(): tb = err.__traceback__.tb_next co = tb.tb_frame.f_code self.assertEqual(co.co_name, "__init__") - self.assertTrue(co.co_filename.endswith('test_exceptions.py')) + self.assertEndsWith(co.co_filename, 'test_exceptions.py') co2 = tb.tb_frame.f_back.f_code self.assertEqual(co2.co_name, "test_capi2") else: @@ -428,9 +432,9 @@ def test_WindowsError(self): self.assertEqual(w.filename, None) self.assertEqual(w.filename2, None) + @unittest.expectedFailure # TODO: RUSTPYTHON @unittest.skipUnless(sys.platform == 'win32', 'test specific to Windows') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_windows_message(self): """Should fill in unknown error code in Windows error message""" ctypes = import_module('ctypes') @@ -439,7 +443,7 @@ def test_windows_message(self): with self.assertRaisesRegex(OSError, 'Windows Error 0x%x' % code): ctypes.pythonapi.PyErr_SetFromWindowsErr(code) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def testAttributes(self): # test that exception attributes are happy @@ -605,7 +609,7 @@ def test_invalid_setstate(self): def test_notes(self): for e in [BaseException(1), Exception(2), ValueError(3)]: with self.subTest(e=e): - self.assertFalse(hasattr(e, '__notes__')) + self.assertNotHasAttr(e, '__notes__') e.add_note("My Note") self.assertEqual(e.__notes__, ["My Note"]) @@ -617,7 +621,7 @@ def test_notes(self): self.assertEqual(e.__notes__, ["My Note", "Your Note"]) del e.__notes__ - self.assertFalse(hasattr(e, '__notes__')) + self.assertNotHasAttr(e, '__notes__') e.add_note("Our Note") self.assertEqual(e.__notes__, ["Our Note"]) @@ -658,7 +662,7 @@ def testInvalidTraceback(self): else: self.fail("No exception raised") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_invalid_setattr(self): TE = TypeError exc = Exception() @@ -671,7 +675,7 @@ def test_invalid_setattr(self): msg = "exception context must be None or derive from BaseException" self.assertRaisesRegex(TE, msg, setattr, exc, '__context__', 1) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_invalid_delattr(self): TE = TypeError try: @@ -743,7 +747,7 @@ def __init__(self, fancy_arg): x = DerivedException(fancy_arg=42) self.assertEqual(x.fancy_arg, 42) - @unittest.skipIf(sys.platform == 'win32', 'TODO: RUSTPYTHON; Windows') + @unittest.skipIf(sys.platform == "win32", "TODO: RUSTPYTHON; Windows") @no_tracing def testInfiniteRecursion(self): def f(): @@ -1304,7 +1308,7 @@ def test_context_of_exception_in_else_and_finally(self): self.assertIs(exc, oe) self.assertIs(exc.__context__, ve) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_unicode_change_attributes(self): # See issue 7309. This was a crasher. @@ -1348,7 +1352,7 @@ def test_unicode_errors_no_object(self): for klass in klasses: self.assertEqual(str(klass.__new__(klass)), "") - @unittest.expectedFailure # TODO: RUSTPYTHON; OverflowError: Python int too large to convert to Rust usize + @unittest.expectedFailure # TODO: RUSTPYTHON; OverflowError: Python int too large to convert to Rust usize def test_unicode_error_str_does_not_crash(self): # Test that str(UnicodeError(...)) does not crash. # See https://github.com/python/cpython/issues/123378. @@ -1372,7 +1376,46 @@ def test_unicode_error_str_does_not_crash(self): exc = UnicodeDecodeError('utf-8', encoded, start, end, '') self.assertIsInstance(str(exc), str) - @unittest.skipIf(sys.platform == 'win32', 'TODO: RUSTPYTHON; Windows') + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_unicode_error_evil_str_set_none_object(self): + def side_effect(exc): + exc.object = None + self.do_test_unicode_error_mutate(side_effect) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_unicode_error_evil_str_del_self_object(self): + def side_effect(exc): + del exc.object + self.do_test_unicode_error_mutate(side_effect) + + def do_test_unicode_error_mutate(self, side_effect): + # Test that str(UnicodeError(...)) does not crash when + # side-effects mutate the underlying 'object' attribute. + # See https://github.com/python/cpython/issues/128974. + + class Evil(str): + def __str__(self): + side_effect(exc) + return self + + for reason, encoding in [ + ("reason", Evil("utf-8")), + (Evil("reason"), "utf-8"), + (Evil("reason"), Evil("utf-8")), + ]: + with self.subTest(encoding=encoding, reason=reason): + with self.subTest(UnicodeEncodeError): + exc = UnicodeEncodeError(encoding, "x", 0, 1, reason) + self.assertRaises(TypeError, str, exc) + with self.subTest(UnicodeDecodeError): + exc = UnicodeDecodeError(encoding, b"x", 0, 1, reason) + self.assertRaises(TypeError, str, exc) + + with self.subTest(UnicodeTranslateError): + exc = UnicodeTranslateError("x", 0, 1, Evil("reason")) + self.assertRaises(TypeError, str, exc) + + @unittest.skipIf(sys.platform == "win32", "TODO: RUSTPYTHON; Windows") @no_tracing def test_badisinstance(self): # Bug #2542: if issubclass(e, MyException) raises an exception, @@ -1404,7 +1447,8 @@ def g(): self.assertIsInstance(exc, RecursionError, type(exc)) self.assertIn("maximum recursion depth exceeded", str(exc)) - + @support.skip_wasi_stack_overflow() + @support.skip_emscripten_stack_overflow() @cpython_only @support.requires_resource('cpu') def test_trashcan_recursion(self): @@ -1420,6 +1464,7 @@ def foo(): foo() support.gc_collect() + @support.skip_emscripten_stack_overflow() @cpython_only def test_recursion_normalizing_exception(self): import_module("_testinternalcapi") @@ -1492,11 +1537,12 @@ def test_recursion_normalizing_infinite_exception(self): """ rc, out, err = script_helper.assert_python_failure("-c", code) self.assertEqual(rc, 1) - expected = b'RecursionError: maximum recursion depth exceeded' + expected = b'RecursionError' self.assertTrue(expected in err, msg=f"{expected!r} not found in {err[:3_000]!r}... (truncated)") self.assertIn(b'Done.', out) + @support.skip_emscripten_stack_overflow() def test_recursion_in_except_handler(self): def set_relative_recursion_limit(n): @@ -1602,7 +1648,7 @@ def test_exception_with_doc(self): # test basic usage of PyErr_NewException error1 = _testcapi.make_exception_with_doc("_testcapi.error1") self.assertIs(type(error1), type) - self.assertTrue(issubclass(error1, Exception)) + self.assertIsSubclass(error1, Exception) self.assertIsNone(error1.__doc__) # test with given docstring @@ -1612,21 +1658,21 @@ def test_exception_with_doc(self): # test with explicit base (without docstring) error3 = _testcapi.make_exception_with_doc("_testcapi.error3", base=error2) - self.assertTrue(issubclass(error3, error2)) + self.assertIsSubclass(error3, error2) # test with explicit base tuple class C(object): pass error4 = _testcapi.make_exception_with_doc("_testcapi.error4", doc4, (error3, C)) - self.assertTrue(issubclass(error4, error3)) - self.assertTrue(issubclass(error4, C)) + self.assertIsSubclass(error4, error3) + self.assertIsSubclass(error4, C) self.assertEqual(error4.__doc__, doc4) # test with explicit dictionary error5 = _testcapi.make_exception_with_doc("_testcapi.error5", "", error4, {'a': 1}) - self.assertTrue(issubclass(error5, error4)) + self.assertIsSubclass(error5, error4) self.assertEqual(error5.a, 1) self.assertEqual(error5.__doc__, "") @@ -1654,7 +1700,7 @@ def inner(): gc_collect() # For PyPy or other GCs. self.assertEqual(wr(), None) - @unittest.skipIf(sys.platform == 'win32', 'TODO: RUSTPYTHON; Windows') + @unittest.skipIf(sys.platform == "win32", "TODO: RUSTPYTHON; Windows") @no_tracing def test_recursion_error_cleanup(self): # Same test as above, but with "recursion exceeded" errors @@ -1676,13 +1722,14 @@ def inner(): gc_collect() # For PyPy or other GCs. self.assertEqual(wr(), None) - @unittest.skipIf(sys.platform == 'win32', 'TODO: RUSTPYTHON; error specific to cpython') + @unittest.skipIf(sys.platform == "win32", "TODO: RUSTPYTHON; error specific to cpython") def test_errno_ENOTDIR(self): # Issue #12802: "not a directory" errors are ENOTDIR even on Windows with self.assertRaises(OSError) as cm: os.listdir(__file__) self.assertEqual(cm.exception.errno, errno.ENOTDIR, cm.exception) + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: None != 'Exception ignored while calling dealloca[83 chars]200>' def test_unraisable(self): # Issue #22836: PyErr_WriteUnraisable() should give sensible reports class BrokenDel: @@ -1693,13 +1740,16 @@ def __del__(self): obj = BrokenDel() with support.catch_unraisable_exception() as cm: + obj_repr = repr(type(obj).__del__) del obj gc_collect() # For PyPy or other GCs. - self.assertEqual(cm.unraisable.object, BrokenDel.__del__) + self.assertEqual(cm.unraisable.err_msg, + f"Exception ignored while calling " + f"deallocator {obj_repr}") self.assertIsNotNone(cm.unraisable.exc_traceback) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_unhandled(self): # Check for sensible reporting of unhandled exceptions for exc_type in (ValueError, BrokenStrException): @@ -1719,7 +1769,7 @@ def test_unhandled(self): self.assertIn("", report) else: self.assertIn("test message", report) - self.assertTrue(report.endswith("\n")) + self.assertEndsWith(report, "\n") @cpython_only # Python built with Py_TRACE_REFS fail with a fatal error in @@ -1798,7 +1848,6 @@ def g(): next(i) next(i) - @unittest.expectedFailure # TODO: RUSTPYTHON @unittest.skipUnless(__debug__, "Won't work if __debug__ is False") def test_assert_shadowing(self): # Shadowing AssertionError would cause the assert statement to @@ -1860,6 +1909,30 @@ def test_memory_error_in_subinterp(self): rc, _, err = script_helper.assert_python_ok("-c", code) self.assertIn(b'MemoryError', err) + def test_keyerror_context(self): + # Make sure that _PyErr_SetKeyError() chains exceptions + try: + err1 = None + err2 = None + try: + d = {} + try: + raise ValueError("bug") + except Exception as exc: + err1 = exc + d[1] + except Exception as exc: + err2 = exc + + self.assertIsInstance(err1, ValueError) + self.assertIsInstance(err2, KeyError) + self.assertEqual(err2.__context__, err1) + finally: + # Break any potential reference cycle + exc1 = None + exc2 = None + + @cpython_only # Python built with Py_TRACE_REFS fail with a fatal error in # _PyRefchain_Trace() on memory allocation error. @@ -1873,7 +1946,7 @@ def test_exec_set_nomemory_hang(self): # PyLong_FromLong() from returning cached integers, which # don't require a memory allocation. Prepend some dummy code # to artificially increase the instruction index. - warmup_code = "a = list(range(0, 1))\n" * 20 + warmup_code = "a = list(range(0, 1))\n" * 60 user_input = warmup_code + dedent(""" try: import _testcapi @@ -1988,7 +2061,7 @@ def blech(self): class ImportErrorTests(unittest.TestCase): - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_attributes(self): # Setting 'name' and 'path' should not be a problem. exc = ImportError('test') @@ -2078,7 +2151,7 @@ class AssertionErrorTests(unittest.TestCase): def tearDown(self): unlink(TESTFN) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON @force_not_colorized def test_assertion_error_location(self): cases = [ @@ -2177,7 +2250,7 @@ def test_assertion_error_location(self): result = run_script(source) self.assertEqual(result[-3:], expected) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON @force_not_colorized def test_multiline_not_highlighted(self): cases = [ @@ -2214,7 +2287,7 @@ def test_multiline_not_highlighted(self): class SyntaxErrorTests(unittest.TestCase): maxDiff = None - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON @force_not_colorized def test_range_of_offsets(self): cases = [ @@ -2306,6 +2379,7 @@ def test_range_of_offsets(self): self.assertIn(expected, err.getvalue()) the_exception = exc + @force_not_colorized def test_subclass(self): class MySyntaxError(SyntaxError): pass @@ -2321,7 +2395,7 @@ class MySyntaxError(SyntaxError): ^^^^^ """, err.getvalue()) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_encodings(self): self.addCleanup(unlink, TESTFN) source = ( @@ -2330,7 +2404,7 @@ def test_encodings(self): ) err = run_script(source.encode('cp437')) self.assertEqual(err[-3], ' "┬ó┬ó┬ó┬ó┬ó┬ó" + f(4, x for x in range(1))') - self.assertEqual(err[-2], ' ^^^^^^^^^^^^^^^^^^^') + self.assertEqual(err[-2], ' ^^^') # Check backwards tokenizer errors source = '# -*- coding: ascii -*-\n\n(\n' @@ -2338,14 +2412,14 @@ def test_encodings(self): self.assertEqual(err[-3], ' (') self.assertEqual(err[-2], ' ^') - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_non_utf8(self): # Check non utf-8 characters self.addCleanup(unlink, TESTFN) err = run_script(b"\x89") self.assertIn("SyntaxError: Non-UTF-8 code starting with '\\x89' in file", err[-1]) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_string_source(self): def try_compile(source): with self.assertRaises(SyntaxError) as cm: @@ -2388,7 +2462,7 @@ def try_compile(source): self.assertEqual(exc.offset, 1) self.assertEqual(exc.end_offset, 12) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_file_source(self): self.addCleanup(unlink, TESTFN) err = run_script('return "ä"') @@ -2451,12 +2525,12 @@ def test_attributes_old_constructor(self): self.assertEqual(error, the_exception.text) self.assertEqual("bad bad", the_exception.msg) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_incorrect_constructor(self): args = ("bad.py", 1, 2) self.assertRaises(TypeError, SyntaxError, "bad bad", args) - args = ("bad.py", 1, 2, 4, 5, 6, 7) + args = ("bad.py", 1, 2, 4, 5, 6, 7, 8) self.assertRaises(TypeError, SyntaxError, "bad bad", args) args = ("bad.py", 1, 2, "abcdefg", 1) @@ -2513,7 +2587,7 @@ def in_except(): pass self.lineno_after_raise(in_except, 4) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_lineno_after_other_except(self): def other_except(): try: @@ -2531,7 +2605,7 @@ def in_named_except(): pass self.lineno_after_raise(in_named_except, 4) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_lineno_in_try(self): def in_try(): try: @@ -2570,7 +2644,7 @@ def after_with(): pass self.lineno_after_raise(after_with, 2) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_missing_lineno_shows_as_none(self): def f(): 1/0 diff --git a/Lib/test/test_extcall.py b/Lib/test/test_extcall.py new file mode 100644 index 00000000000..2392bb1d13d --- /dev/null +++ b/Lib/test/test_extcall.py @@ -0,0 +1,563 @@ + +"""Doctest for method/function calls. + +We're going the use these types for extra testing + + >>> from collections import UserList + >>> from collections import UserDict + +We're defining four helper functions + + >>> from test import support + >>> def e(a,b): + ... print(a, b) + + >>> def f(*a, **k): + ... print(a, support.sortdict(k)) + + >>> def g(x, *y, **z): + ... print(x, y, support.sortdict(z)) + + >>> def h(j=1, a=2, h=3): + ... print(j, a, h) + +Argument list examples + + >>> f() + () {} + >>> f(1) + (1,) {} + >>> f(1, 2) + (1, 2) {} + >>> f(1, 2, 3) + (1, 2, 3) {} + >>> f(1, 2, 3, *(4, 5)) + (1, 2, 3, 4, 5) {} + >>> f(1, 2, 3, *[4, 5]) + (1, 2, 3, 4, 5) {} + >>> f(*[1, 2, 3], 4, 5) + (1, 2, 3, 4, 5) {} + >>> f(1, 2, 3, *UserList([4, 5])) + (1, 2, 3, 4, 5) {} + >>> f(1, 2, 3, *[4, 5], *[6, 7]) + (1, 2, 3, 4, 5, 6, 7) {} + >>> f(1, *[2, 3], 4, *[5, 6], 7) + (1, 2, 3, 4, 5, 6, 7) {} + >>> f(*UserList([1, 2]), *UserList([3, 4]), 5, *UserList([6, 7])) + (1, 2, 3, 4, 5, 6, 7) {} + +Here we add keyword arguments + + >>> f(1, 2, 3, **{'a':4, 'b':5}) + (1, 2, 3) {'a': 4, 'b': 5} + >>> f(1, 2, **{'a': -1, 'b': 5}, **{'a': 4, 'c': 6}) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_extcall.f() got multiple values for keyword argument 'a' + >>> f(1, 2, **{'a': -1, 'b': 5}, a=4, c=6) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_extcall.f() got multiple values for keyword argument 'a' + >>> f(1, 2, a=3, **{'a': 4}, **{'a': 5}) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_extcall.f() got multiple values for keyword argument 'a' + >>> f(1, 2, 3, *[4, 5], **{'a':6, 'b':7}) + (1, 2, 3, 4, 5) {'a': 6, 'b': 7} + >>> f(1, 2, 3, x=4, y=5, *(6, 7), **{'a':8, 'b': 9}) + (1, 2, 3, 6, 7) {'a': 8, 'b': 9, 'x': 4, 'y': 5} + >>> f(1, 2, 3, *[4, 5], **{'c': 8}, **{'a':6, 'b':7}) + (1, 2, 3, 4, 5) {'a': 6, 'b': 7, 'c': 8} + >>> f(1, 2, 3, *(4, 5), x=6, y=7, **{'a':8, 'b': 9}) + (1, 2, 3, 4, 5) {'a': 8, 'b': 9, 'x': 6, 'y': 7} + + >>> f(1, 2, 3, **UserDict(a=4, b=5)) + (1, 2, 3) {'a': 4, 'b': 5} + >>> f(1, 2, 3, *(4, 5), **UserDict(a=6, b=7)) + (1, 2, 3, 4, 5) {'a': 6, 'b': 7} + >>> f(1, 2, 3, x=4, y=5, *(6, 7), **UserDict(a=8, b=9)) + (1, 2, 3, 6, 7) {'a': 8, 'b': 9, 'x': 4, 'y': 5} + >>> f(1, 2, 3, *(4, 5), x=6, y=7, **UserDict(a=8, b=9)) + (1, 2, 3, 4, 5) {'a': 8, 'b': 9, 'x': 6, 'y': 7} + +Mix keyword arguments and dict unpacking + + >>> d1 = {'a':1} + + >>> d2 = {'c':3} + + >>> f(b=2, **d1, **d2) + () {'a': 1, 'b': 2, 'c': 3} + + >>> f(**d1, b=2, **d2) + () {'a': 1, 'b': 2, 'c': 3} + + >>> f(**d1, **d2, b=2) + () {'a': 1, 'b': 2, 'c': 3} + + >>> f(**d1, b=2, **d2, d=4) + () {'a': 1, 'b': 2, 'c': 3, 'd': 4} + +Examples with invalid arguments (TypeErrors). We're also testing the function +names in the exception messages. + +Verify clearing of SF bug #733667 + + >>> e(c=4) + Traceback (most recent call last): + ... + TypeError: e() got an unexpected keyword argument 'c' + + >>> g() + Traceback (most recent call last): + ... + TypeError: g() missing 1 required positional argument: 'x' + + >>> g(*()) + Traceback (most recent call last): + ... + TypeError: g() missing 1 required positional argument: 'x' + + >>> g(*(), **{}) + Traceback (most recent call last): + ... + TypeError: g() missing 1 required positional argument: 'x' + + >>> g(1) + 1 () {} + >>> g(1, 2) + 1 (2,) {} + >>> g(1, 2, 3) + 1 (2, 3) {} + >>> g(1, 2, 3, *(4, 5)) + 1 (2, 3, 4, 5) {} + + >>> class Nothing: pass + ... + >>> g(*Nothing()) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_extcall.g() argument after * must be an iterable, not Nothing + + >>> class Nothing: + ... def __len__(self): return 5 + ... + + >>> g(*Nothing()) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_extcall.g() argument after * must be an iterable, not Nothing + + >>> class Nothing(): + ... def __len__(self): return 5 + ... def __getitem__(self, i): + ... if i<3: return i + ... else: raise IndexError(i) + ... + + >>> g(*Nothing()) + 0 (1, 2) {} + + >>> class Nothing: + ... def __init__(self): self.c = 0 + ... def __iter__(self): return self + ... def __next__(self): + ... if self.c == 4: + ... raise StopIteration + ... c = self.c + ... self.c += 1 + ... return c + ... + + >>> g(*Nothing()) + 0 (1, 2, 3) {} + +Check for issue #4806: Does a TypeError in a generator get propagated with the +right error message? (Also check with other iterables.) + + >>> def broken(): raise TypeError("myerror") + ... + + >>> g(*(broken() for i in range(1))) + Traceback (most recent call last): + ... + TypeError: myerror + >>> g(*range(1), *(broken() for i in range(1))) + Traceback (most recent call last): + ... + TypeError: myerror + + >>> class BrokenIterable1: + ... def __iter__(self): + ... raise TypeError('myerror') + ... + >>> g(*BrokenIterable1()) + Traceback (most recent call last): + ... + TypeError: myerror + >>> g(*range(1), *BrokenIterable1()) + Traceback (most recent call last): + ... + TypeError: myerror + + >>> class BrokenIterable2: + ... def __iter__(self): + ... yield 0 + ... raise TypeError('myerror') + ... + >>> g(*BrokenIterable2()) + Traceback (most recent call last): + ... + TypeError: myerror + >>> g(*range(1), *BrokenIterable2()) + Traceback (most recent call last): + ... + TypeError: myerror + + >>> class BrokenSequence: + ... def __getitem__(self, idx): + ... raise TypeError('myerror') + ... + >>> g(*BrokenSequence()) + Traceback (most recent call last): + ... + TypeError: myerror + >>> g(*range(1), *BrokenSequence()) + Traceback (most recent call last): + ... + TypeError: myerror + +Make sure that the function doesn't stomp the dictionary + + >>> d = {'a': 1, 'b': 2, 'c': 3} + >>> d2 = d.copy() + >>> g(1, d=4, **d) + 1 () {'a': 1, 'b': 2, 'c': 3, 'd': 4} + >>> d == d2 + True + +What about willful misconduct? + + >>> def saboteur(**kw): + ... kw['x'] = 'm' + ... return kw + + >>> d = {} + >>> kw = saboteur(a=1, **d) + >>> d + {} + + + >>> g(1, 2, 3, **{'x': 4, 'y': 5}) + Traceback (most recent call last): + ... + TypeError: g() got multiple values for argument 'x' + + >>> f(**{1:2}) + Traceback (most recent call last): + ... + TypeError: keywords must be strings + + >>> h(**{'e': 2}) + Traceback (most recent call last): + ... + TypeError: h() got an unexpected keyword argument 'e' + + >>> h(*h) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_extcall.h() argument after * must be an iterable, not function + + >>> h(1, *h) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: Value after * must be an iterable, not function + + >>> h(*[1], *h) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: Value after * must be an iterable, not function + + >>> dir(*h) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: dir() argument after * must be an iterable, not function + + >>> nothing = None + >>> nothing(*h) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: None argument after * must be an iterable, \ +not function + + >>> h(**h) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_extcall.h() argument after ** must be a mapping, not function + + >>> h(**[]) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_extcall.h() argument after ** must be a mapping, not list + + >>> h(a=1, **h) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_extcall.h() argument after ** must be a mapping, not function + + >>> h(a=1, **[]) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_extcall.h() argument after ** must be a mapping, not list + + >>> h(**{'a': 1}, **h) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_extcall.h() argument after ** must be a mapping, not function + + >>> h(**{'a': 1}, **[]) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_extcall.h() argument after ** must be a mapping, not list + + >>> dir(**h) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: dir() argument after ** must be a mapping, not function + + >>> nothing(**h) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: None argument after ** must be a mapping, \ +not function + + >>> dir(b=1, **{'b': 1}) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: dir() got multiple values for keyword argument 'b' + +Test a kwargs mapping with duplicated keys. + + >>> from collections.abc import Mapping + >>> class MultiDict(Mapping): + ... def __init__(self, items): + ... self._items = items + ... + ... def __iter__(self): + ... return (k for k, v in self._items) + ... + ... def __getitem__(self, key): + ... for k, v in self._items: + ... if k == key: + ... return v + ... raise KeyError(key) + ... + ... def __len__(self): + ... return len(self._items) + ... + ... def keys(self): + ... return [k for k, v in self._items] + ... + ... def values(self): + ... return [v for k, v in self._items] + ... + ... def items(self): + ... return [(k, v) for k, v in self._items] + ... + >>> g(**MultiDict([('x', 1), ('y', 2)])) + 1 () {'y': 2} + + >>> g(**MultiDict([('x', 1), ('x', 2)])) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_extcall.g() got multiple values for keyword argument 'x' + + >>> g(a=3, **MultiDict([('x', 1), ('x', 2)])) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_extcall.g() got multiple values for keyword argument 'x' + + >>> g(**MultiDict([('a', 3)]), **MultiDict([('x', 1), ('x', 2)])) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_extcall.g() got multiple values for keyword argument 'x' + +Call with dict subtype: + + >>> class MyDict(dict): + ... pass + + >>> def s1(**kwargs): + ... return kwargs + >>> def s2(*args, **kwargs): + ... return (args, kwargs) + >>> def s3(*, n, **kwargs): + ... return (n, kwargs) + + >>> md = MyDict({'a': 1, 'b': 2}) + >>> assert s1(**md) == {'a': 1, 'b': 2} + >>> assert s2(*(1, 2), **md) == ((1, 2), {'a': 1, 'b': 2}) + >>> assert s3(**MyDict({'n': 1, 'b': 2})) == (1, {'b': 2}) + >>> s3(**md) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: s3() missing 1 required keyword-only argument: 'n' + +Another helper function + + >>> def f2(*a, **b): + ... return a, b + + + >>> d = {} + >>> for i in range(512): + ... key = 'k%d' % i + ... d[key] = i + >>> a, b = f2(1, *(2,3), **d) + >>> len(a), len(b), b == d + (3, 512, True) + + >>> class Foo: + ... def method(self, arg1, arg2): + ... return arg1+arg2 + + >>> x = Foo() + >>> Foo.method(*(x, 1, 2)) + 3 + >>> Foo.method(x, *(1, 2)) + 3 + >>> Foo.method(*(1, 2, 3)) + 5 + >>> Foo.method(1, *[2, 3]) + 5 + +A PyCFunction that takes only positional parameters should allow an +empty keyword dictionary to pass without a complaint, but raise a +TypeError if te dictionary is not empty + + >>> try: # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + ... silence = id(1, *{}) + ... True + ... except: + ... False + True + + >>> id(1, **{'foo': 1}) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: id() takes no keyword arguments + +A corner case of keyword dictionary items being deleted during +the function call setup. See . + + >>> class Name(str): + ... def __eq__(self, other): + ... try: + ... del x[self] + ... except KeyError: + ... pass + ... return str.__eq__(self, other) + ... def __hash__(self): + ... return str.__hash__(self) + + >>> x = {Name("a"):1, Name("b"):2} + >>> def f(a, b): + ... print(a,b) + >>> f(**x) + 1 2 + +Too many arguments: + + >>> def f(): pass + >>> f(1) + Traceback (most recent call last): + ... + TypeError: f() takes 0 positional arguments but 1 was given + >>> def f(a): pass + >>> f(1, 2) + Traceback (most recent call last): + ... + TypeError: f() takes 1 positional argument but 2 were given + >>> def f(a, b=1): pass + >>> f(1, 2, 3) + Traceback (most recent call last): + ... + TypeError: f() takes from 1 to 2 positional arguments but 3 were given + >>> def f(*, kw): pass + >>> f(1, kw=3) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: f() takes 0 positional arguments but 1 positional argument (and 1 keyword-only argument) were given + >>> def f(*, kw, b): pass + >>> f(1, 2, 3, b=3, kw=3) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: f() takes 0 positional arguments but 3 positional arguments (and 2 keyword-only arguments) were given + >>> def f(a, b=2, *, kw): pass + >>> f(2, 3, 4, kw=4) # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: f() takes from 1 to 2 positional arguments but 3 positional arguments (and 1 keyword-only argument) were given + +Too few and missing arguments: + + >>> def f(a): pass + >>> f() + Traceback (most recent call last): + ... + TypeError: f() missing 1 required positional argument: 'a' + >>> def f(a, b): pass + >>> f() + Traceback (most recent call last): + ... + TypeError: f() missing 2 required positional arguments: 'a' and 'b' + >>> def f(a, b, c): pass + >>> f() + Traceback (most recent call last): + ... + TypeError: f() missing 3 required positional arguments: 'a', 'b', and 'c' + >>> def f(a, b, c, d, e): pass + >>> f() + Traceback (most recent call last): + ... + TypeError: f() missing 5 required positional arguments: 'a', 'b', 'c', 'd', and 'e' + >>> def f(a, b=4, c=5, d=5): pass + >>> f(c=12, b=9) + Traceback (most recent call last): + ... + TypeError: f() missing 1 required positional argument: 'a' + +Same with keyword only args: + + >>> def f(*, w): pass + >>> f() # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: f() missing 1 required keyword-only argument: 'w' + >>> def f(*, a, b, c, d, e): pass + >>> f() # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: f() missing 5 required keyword-only arguments: 'a', 'b', 'c', 'd', and 'e' + +""" + +import doctest +import unittest + +EXPECTED_FAILURE = doctest.register_optionflag('EXPECTED_FAILURE') # TODO: RUSTPYTHON +class CustomOutputChecker(doctest.OutputChecker): # TODO: RUSTPYTHON + def check_output(self, want, got, optionflags): # TODO: RUSTPYTHON + if optionflags & EXPECTED_FAILURE: # TODO: RUSTPYTHON + if want == got: # TODO: RUSTPYTHON + return False # TODO: RUSTPYTHON + return True # TODO: RUSTPYTHON + return super().check_output(want, got, optionflags) # TODO: RUSTPYTHON + +def load_tests(loader, tests, pattern): + tests.addTest(doctest.DocTestSuite(checker=CustomOutputChecker())) # TODO: RUSTPYTHON + return tests + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_fork1.py b/Lib/test/test_fork1.py new file mode 100644 index 00000000000..4f4a5ee0507 --- /dev/null +++ b/Lib/test/test_fork1.py @@ -0,0 +1,103 @@ +"""This test checks for correct fork() behavior. +""" + +import _imp as imp +import os +import signal +import sys +import threading +import time +import unittest + +from test.fork_wait import ForkWait +from test import support + + +# Skip test if fork does not exist. +if not support.has_fork_support: + raise unittest.SkipTest("test module requires working os.fork") + + +class ForkTest(ForkWait): + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: process 44587 exited with code 1, but exit code 42 is expected + def test_threaded_import_lock_fork(self): + """Check fork() in main thread works while a subthread is doing an import""" + import_started = threading.Event() + fake_module_name = "fake test module" + partial_module = "partial" + complete_module = "complete" + def importer(): + imp.acquire_lock() + sys.modules[fake_module_name] = partial_module + import_started.set() + time.sleep(0.01) # Give the other thread time to try and acquire. + sys.modules[fake_module_name] = complete_module + imp.release_lock() + t = threading.Thread(target=importer) + t.start() + import_started.wait() + exitcode = 42 + pid = os.fork() + try: + # PyOS_BeforeFork should have waited for the import to complete + # before forking, so the child can recreate the import lock + # correctly, but also won't see a partially initialised module + if not pid: + m = __import__(fake_module_name) + if m == complete_module: + os._exit(exitcode) + else: + if support.verbose > 1: + print("Child encountered partial module") + os._exit(1) + else: + t.join() + # Exitcode 1 means the child got a partial module (bad.) No + # exitcode (but a hang, which manifests as 'got pid 0') + # means the child deadlocked (also bad.) + self.wait_impl(pid, exitcode=exitcode) + finally: + try: + os.kill(pid, signal.SIGKILL) + except OSError: + pass + + + def test_nested_import_lock_fork(self): + """Check fork() in main thread works while the main thread is doing an import""" + exitcode = 42 + # Issue 9573: this used to trigger RuntimeError in the child process + def fork_with_import_lock(level): + release = 0 + in_child = False + try: + try: + for i in range(level): + imp.acquire_lock() + release += 1 + pid = os.fork() + in_child = not pid + finally: + for i in range(release): + imp.release_lock() + except RuntimeError: + if in_child: + if support.verbose > 1: + print("RuntimeError in child") + os._exit(1) + raise + if in_child: + os._exit(exitcode) + self.wait_impl(pid, exitcode=exitcode) + + # Check this works with various levels of nested + # import in the main thread + for level in range(5): + fork_with_import_lock(level) + + +def tearDownModule(): + support.reap_children() + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_fractions.py b/Lib/test/test_fractions.py index 67c9b98d67f..49fc9c2ba23 100644 --- a/Lib/test/test_fractions.py +++ b/Lib/test/test_fractions.py @@ -1,6 +1,5 @@ """Tests for Lib/fractions.py.""" -import cmath from decimal import Decimal from test.support import requires_IEEE_754, adjust_int_max_str_digits import math @@ -284,6 +283,13 @@ def __repr__(self): class RectComplex(Rect, complex): pass +class Ratio: + def __init__(self, ratio): + self._ratio = ratio + def as_integer_ratio(self): + return self._ratio + + class FractionTest(unittest.TestCase): def assertTypedEquals(self, expected, actual): @@ -355,6 +361,38 @@ def testInitFromDecimal(self): self.assertRaises(OverflowError, F, Decimal('inf')) self.assertRaises(OverflowError, F, Decimal('-inf')) + def testInitFromIntegerRatio(self): + self.assertEqual((7, 3), _components(F(Ratio((7, 3))))) + errmsg = (r"argument should be a string or a Rational instance or " + r"have the as_integer_ratio\(\) method") + # the type also has an "as_integer_ratio" attribute. + self.assertRaisesRegex(TypeError, errmsg, F, Ratio) + # bad ratio + self.assertRaises(TypeError, F, Ratio(7)) + self.assertRaises(ValueError, F, Ratio((7,))) + self.assertRaises(ValueError, F, Ratio((7, 3, 1))) + # only single-argument form + self.assertRaises(TypeError, F, Ratio((3, 7)), 11) + self.assertRaises(TypeError, F, 2, Ratio((-10, 9))) + + # as_integer_ratio not defined in a class + class A: + pass + a = A() + a.as_integer_ratio = lambda: (9, 5) + self.assertEqual((9, 5), _components(F(a))) + + # as_integer_ratio defined in a metaclass + class M(type): + def as_integer_ratio(self): + return (11, 9) + class B(metaclass=M): + pass + self.assertRaisesRegex(TypeError, errmsg, F, B) + self.assertRaisesRegex(TypeError, errmsg, F, B()) + self.assertRaises(TypeError, F.from_number, B) + self.assertRaises(TypeError, F.from_number, B()) + def testFromString(self): self.assertEqual((5, 1), _components(F("5"))) self.assertEqual((5, 1), _components(F("005"))) @@ -548,6 +586,37 @@ def testFromDecimal(self): ValueError, "cannot convert NaN to integer ratio", F.from_decimal, Decimal("snan")) + def testFromNumber(self, cls=F): + def check(arg, numerator, denominator): + f = cls.from_number(arg) + self.assertIs(type(f), cls) + self.assertEqual(f.numerator, numerator) + self.assertEqual(f.denominator, denominator) + + check(10, 10, 1) + check(2.5, 5, 2) + check(Decimal('2.5'), 5, 2) + check(F(22, 7), 22, 7) + check(DummyFraction(22, 7), 22, 7) + check(Rat(22, 7), 22, 7) + check(Ratio((22, 7)), 22, 7) + self.assertRaises(TypeError, cls.from_number, 3+4j) + self.assertRaises(TypeError, cls.from_number, '5/2') + self.assertRaises(TypeError, cls.from_number, []) + self.assertRaises(OverflowError, cls.from_number, float('inf')) + self.assertRaises(OverflowError, cls.from_number, Decimal('inf')) + + # as_integer_ratio not defined in a class + class A: + pass + a = A() + a.as_integer_ratio = lambda: (9, 5) + check(a, 9, 5) + + def testFromNumber_subclass(self): + self.testFromNumber(DummyFraction) + + def test_is_integer(self): self.assertTrue(F(1, 1).is_integer()) self.assertTrue(F(-1, 1).is_integer()) @@ -794,7 +863,7 @@ def testMixedMultiplication(self): self.assertTypedEquals(F(3, 2) * Polar(4, 2), Polar(F(6, 1), 2)) self.assertTypedEquals(F(3, 2) * Polar(4.0, 2), Polar(6.0, 2)) self.assertTypedEquals(F(3, 2) * Rect(4, 3), Rect(F(6, 1), F(9, 2))) - self.assertTypedEquals(F(3, 2) * RectComplex(4, 3), RectComplex(6.0+0j, 4.5+0j)) + self.assertTypedEquals(F(3, 2) * RectComplex(4, 3), RectComplex(6.0, 4.5)) self.assertRaises(TypeError, operator.mul, Polar(4, 2), F(3, 2)) self.assertTypedEquals(Rect(4, 3) * F(3, 2), 6.0 + 4.5j) self.assertEqual(F(3, 2) * SymbolicComplex('X'), SymbolicComplex('3/2 * X')) @@ -1253,6 +1322,8 @@ def test_format_e_presentation_type(self): # Thousands separators (F('1234567.123456'), ',.5e', '1.23457e+06'), (F('123.123456'), '012_.2e', '0_001.23e+02'), + # Thousands separators for fractional part (or for integral too) + (F('1234567.123456'), '.5_e', '1.234_57e+06'), # z flag is legal, but never makes a difference to the output (F(-1, 7**100), 'z.6e', '-3.091690e-85'), ] @@ -1378,6 +1449,12 @@ def test_format_f_presentation_type(self): (F('1234567'), ',.2f', '1,234,567.00'), (F('12345678'), ',.2f', '12,345,678.00'), (F('12345678'), ',f', '12,345,678.000000'), + # Thousands separators for fractional part (or for integral too) + (F('123456.789123123'), '._f', '123456.789_123'), + (F('123456.789123123'), '.7_f', '123456.789_123_1'), + (F('123456.789123123'), '.9_f', '123456.789_123_123'), + (F('123456.789123123'), '.,f', '123456.789,123'), + (F('123456.789123123'), '_.,f', '123_456.789,123'), # Underscore as thousands separator (F(2, 3), '_.2f', '0.67'), (F(2, 3), '_.7f', '0.6666667'), @@ -1446,6 +1523,8 @@ def test_format_f_presentation_type(self): (F(51, 1000), '.1f', '0.1'), (F(149, 1000), '.1f', '0.1'), (F(151, 1000), '.1f', '0.2'), + (F(22, 7), '.02f', '3.14'), # issue gh-130662 + (F(22, 7), '005.02f', '03.14'), ] for fraction, spec, expected in testcases: with self.subTest(fraction=fraction, spec=spec): @@ -1544,17 +1623,16 @@ def test_invalid_formats(self): '=010%', '>00.2f', '>00f', - # Too many zeros - minimum width should not have leading zeros - '006f', - # Leading zeros in precision - '.010f', - '.02f', - '.000f', # Missing precision '.e', '.f', '.g', '.%', + # Thousands separators before precision + '._6e', + '._6f', + '._6g', + '._6%', # Z instead of z for negative zero suppression 'Z.2f' # z flag not supported for general formatting @@ -1591,9 +1669,7 @@ def test_float_format_testfile(self): self.assertEqual(float(format(f, fmt2)), float(rhs)) self.assertEqual(float(format(-f, fmt2)), float('-' + rhs)) - # TODO: RUSTPYTHON - # TypeError: '%' not supported between instances of 'Fraction' and 'complex' - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: '%' not supported between instances of 'Fraction' and 'complex' def test_complex_handling(self): # See issue gh-102840 for more details. @@ -1621,6 +1697,19 @@ def test_complex_handling(self): message % ("divmod()", "complex", "Fraction"), divmod, b, a) + @unittest.expectedFailure # TODO: RUSTPYTHON; Wrong error message + def test_three_argument_pow(self): + message = "unsupported operand type(s) for ** or pow(): '%s', '%s', '%s'" + self.assertRaisesMessage(TypeError, + message % ("Fraction", "int", "int"), + pow, F(3), 4, 5) + self.assertRaisesMessage(TypeError, + message % ("int", "Fraction", "int"), + pow, 3, F(4), 5) + self.assertRaisesMessage(TypeError, + message % ("int", "int", "Fraction"), + pow, 3, 4, F(5)) + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_frozen.py b/Lib/test/test_frozen.py new file mode 100644 index 00000000000..10a6d8535fc --- /dev/null +++ b/Lib/test/test_frozen.py @@ -0,0 +1,58 @@ +"""Basic test of the frozen module (source is in Python/frozen.c).""" + +# The Python/frozen.c source code contains a marshalled Python module +# and therefore depends on the marshal format as well as the bytecode +# format. If those formats have been changed then frozen.c needs to be +# updated. +# +# The test_importlib also tests this module but because those tests +# are much more complicated, it might be unclear why they are failing. +# Invalid marshalled data in frozen.c could case the interpreter to +# crash when __hello__ is imported. + +import importlib.machinery +import sys +import unittest +from test.support import captured_stdout, import_helper + + +class TestFrozen(unittest.TestCase): + def test_frozen(self): + name = '__hello__' + if name in sys.modules: + del sys.modules[name] + with import_helper.frozen_modules(): + import __hello__ + with captured_stdout() as out: + __hello__.main() + self.assertEqual(out.getvalue(), 'Hello world!\n') + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: unexpectedly identical: + def test_frozen_submodule_in_unfrozen_package(self): + with import_helper.CleanImport('__phello__', '__phello__.spam'): + with import_helper.frozen_modules(enabled=False): + import __phello__ + with import_helper.frozen_modules(enabled=True): + import __phello__.spam as spam + self.assertIs(spam, __phello__.spam) + self.assertIsNot(__phello__.__spec__.loader, + importlib.machinery.FrozenImporter) + self.assertIs(spam.__spec__.loader, + importlib.machinery.FrozenImporter) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: unexpectedly identical: + def test_unfrozen_submodule_in_frozen_package(self): + with import_helper.CleanImport('__phello__', '__phello__.spam'): + with import_helper.frozen_modules(enabled=True): + import __phello__ + with import_helper.frozen_modules(enabled=False): + import __phello__.spam as spam + self.assertIs(spam, __phello__.spam) + self.assertIs(__phello__.__spec__.loader, + importlib.machinery.FrozenImporter) + self.assertIsNot(spam.__spec__.loader, + importlib.machinery.FrozenImporter) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_ftplib.py b/Lib/test/test_ftplib.py index 2488f64a218..684f5d438b3 100644 --- a/Lib/test/test_ftplib.py +++ b/Lib/test/test_ftplib.py @@ -971,6 +971,7 @@ def test_data_connection(self): LIST_DATA.encode(self.client.encoding)) self.assertEqual(self.client.voidresp(), "226 transfer complete") + @unittest.skip('TODO: RUSTPYTHON flaky TimeoutError') def test_login(self): # login() is supposed to implicitly secure the control connection self.assertNotIsInstance(self.client.sock, ssl.SSLSocket) @@ -983,6 +984,7 @@ def test_auth_issued_twice(self): self.client.auth() self.assertRaises(ValueError, self.client.auth) + @unittest.skip('TODO: RUSTPYTHON flaky TimeoutError') def test_context(self): self.client.quit() ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py index 047916caf07..07490423116 100644 --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -1,4 +1,5 @@ import abc +from annotationlib import Format, get_annotations import builtins import collections import collections.abc @@ -6,6 +7,7 @@ from itertools import permutations import pickle from random import choice +import re import sys from test import support import threading @@ -19,8 +21,11 @@ import contextlib from inspect import Signature +from test.support import ALWAYS_EQ from test.support import import_helper from test.support import threading_helper +from test.support import cpython_only +from test.support import EqualToForwardRef import functools @@ -60,6 +65,14 @@ def __add__(self, other): class MyDict(dict): pass +class TestImportTime(unittest.TestCase): + + @cpython_only + def test_lazy_import(self): + import_helper.ensure_lazy_imports( + "functools", {"os", "weakref", "typing", "annotationlib", "warnings"} + ) + class TestPartial: @@ -210,6 +223,69 @@ def foo(bar): p2.new_attr = 'spam' self.assertEqual(p2.new_attr, 'spam') + def test_placeholders_trailing_raise(self): + PH = self.module.Placeholder + for args in [(PH,), (0, PH), (0, PH, 1, PH, PH, PH)]: + with self.assertRaises(TypeError): + self.partial(capture, *args) + + def test_placeholders(self): + PH = self.module.Placeholder + # 1 Placeholder + args = (PH, 0) + p = self.partial(capture, *args) + actual_args, actual_kwds = p('x') + self.assertEqual(actual_args, ('x', 0)) + self.assertEqual(actual_kwds, {}) + # 2 Placeholders + args = (PH, 0, PH, 1) + p = self.partial(capture, *args) + with self.assertRaises(TypeError): + p('x') + actual_args, actual_kwds = p('x', 'y') + self.assertEqual(actual_args, ('x', 0, 'y', 1)) + self.assertEqual(actual_kwds, {}) + # Checks via `is` and not `eq` + # thus ALWAYS_EQ isn't treated as Placeholder + p = self.partial(capture, ALWAYS_EQ) + actual_args, actual_kwds = p() + self.assertEqual(len(actual_args), 1) + self.assertIs(actual_args[0], ALWAYS_EQ) + self.assertEqual(actual_kwds, {}) + + def test_placeholders_optimization(self): + PH = self.module.Placeholder + p = self.partial(capture, PH, 0) + p2 = self.partial(p, PH, 1, 2, 3) + self.assertEqual(p2.args, (PH, 0, 1, 2, 3)) + p3 = self.partial(p2, -1, 4) + actual_args, actual_kwds = p3(5) + self.assertEqual(actual_args, (-1, 0, 1, 2, 3, 4, 5)) + self.assertEqual(actual_kwds, {}) + # inner partial has placeholders and outer partial has no args case + p = self.partial(capture, PH, 0) + p2 = self.partial(p) + self.assertEqual(p2.args, (PH, 0)) + self.assertEqual(p2(1), ((1, 0), {})) + + def test_placeholders_kw_restriction(self): + PH = self.module.Placeholder + with self.assertRaisesRegex(TypeError, "Placeholder"): + self.partial(capture, a=PH) + # Passes, as checks via `is` and not `eq` + p = self.partial(capture, a=ALWAYS_EQ) + actual_args, actual_kwds = p() + self.assertEqual(actual_args, ()) + self.assertEqual(len(actual_kwds), 1) + self.assertIs(actual_kwds['a'], ALWAYS_EQ) + + def test_construct_placeholder_singleton(self): + PH = self.module.Placeholder + tp = type(PH) + self.assertIs(tp(), PH) + self.assertRaises(TypeError, tp, 1, 2) + self.assertRaises(TypeError, tp, a=1, b=2) + def test_repr(self): args = (object(), object()) args_repr = ', '.join(repr(a) for a in args) @@ -311,8 +387,26 @@ def test_setstate(self): self.assertEqual(f(2), ((2,), {})) self.assertEqual(f(), ((), {})) + # Set State with placeholders + PH = self.module.Placeholder + f = self.partial(signature) + f.__setstate__((capture, (PH, 1), dict(a=10), dict(attr=[]))) + self.assertEqual(signature(f), (capture, (PH, 1), dict(a=10), dict(attr=[]))) + msg_regex = re.escape("missing positional arguments in 'partial' call; " + "expected at least 1, got 0") + with self.assertRaisesRegex(TypeError, f'^{msg_regex}$') as cm: + f() + self.assertEqual(f(2), ((2, 1), dict(a=10))) + + # Trailing Placeholder error + f = self.partial(signature) + msg_regex = re.escape("trailing Placeholders are not allowed") + with self.assertRaisesRegex(TypeError, f'^{msg_regex}$') as cm: + f.__setstate__((capture, (1, PH), dict(a=10), dict(attr=[]))) + def test_setstate_errors(self): f = self.partial(signature) + self.assertRaises(TypeError, f.__setstate__, (capture, (), {})) self.assertRaises(TypeError, f.__setstate__, (capture, (), {}, {}, None)) self.assertRaises(TypeError, f.__setstate__, [capture, (), {}, None]) @@ -320,6 +414,8 @@ def test_setstate_errors(self): self.assertRaises(TypeError, f.__setstate__, (capture, None, {}, None)) self.assertRaises(TypeError, f.__setstate__, (capture, [], {}, None)) self.assertRaises(TypeError, f.__setstate__, (capture, (), [], None)) + self.assertRaises(TypeError, f.__setstate__, (capture, (), {}, ())) + self.assertRaises(TypeError, f.__setstate__, (capture, (), {}, 'test')) def test_setstate_subclasses(self): f = self.partial(signature) @@ -341,6 +437,8 @@ def test_setstate_subclasses(self): self.assertEqual(r, ((1, 2), {})) self.assertIs(type(r[0]), tuple) + @support.skip_if_sanitizer("thread sanitizer crashes in __tsan::FuncEntry", thread=True) + @support.skip_emscripten_stack_overflow() def test_recursive_pickle(self): with replaced_module('functools', self.module): f = self.partial(capture) @@ -395,7 +493,6 @@ def __getitem__(self, key): f = self.partial(object) self.assertRaises(TypeError, f.__setstate__, BadSequence()) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_partial_as_method(self): class A: meth = self.partial(capture, 1, a=2) @@ -406,9 +503,7 @@ class A: self.assertEqual(A.meth(3, b=4), ((1, 3), {'a': 2, 'b': 4})) self.assertEqual(A.cmeth(3, b=4), ((1, A, 3), {'a': 2, 'b': 4})) self.assertEqual(A.smeth(3, b=4), ((1, 3), {'a': 2, 'b': 4})) - with self.assertWarns(FutureWarning) as w: - self.assertEqual(a.meth(3, b=4), ((1, 3), {'a': 2, 'b': 4})) - self.assertEqual(w.filename, __file__) + self.assertEqual(a.meth(3, b=4), ((1, a, 3), {'a': 2, 'b': 4})) self.assertEqual(a.cmeth(3, b=4), ((1, A, 3), {'a': 2, 'b': 4})) self.assertEqual(a.smeth(3, b=4), ((1, 3), {'a': 2, 'b': 4})) @@ -465,11 +560,18 @@ def __str__(self): self.assertIn('astr', r) self.assertIn("['sth']", r) - def test_repr(self): - return super().test_repr() - - def test_recursive_repr(self): - return super().test_recursive_repr() + def test_placeholders_refcount_smoke(self): + PH = self.module.Placeholder + # sum supports vector call + lst1, start = [], [] + sum_lists = self.partial(sum, PH, start) + for i in range(10): + sum_lists([lst1, lst1]) + # collections.ChainMap initializer does not support vectorcall + map1, map2 = {}, {} + partial_cm = self.partial(collections.ChainMap, PH, map1) + for i in range(10): + partial_cm(map2, map2) class TestPartialPy(TestPartial, unittest.TestCase): @@ -495,6 +597,19 @@ class TestPartialCSubclass(TestPartialC): class TestPartialPySubclass(TestPartialPy): partial = PyPartialSubclass + def test_subclass_optimization(self): + # `partial` input to `partial` subclass + p = py_functools.partial(min, 2) + p2 = self.partial(p, 1) + self.assertIs(p2.func, min) + self.assertEqual(p2(0), 0) + # `partial` subclass input to `partial` subclass + p = self.partial(min, 2) + p2 = self.partial(p, 1) + self.assertIs(p2.func, min) + self.assertEqual(p2(0), 0) + + class TestPartialMethod(unittest.TestCase): class A(object): @@ -564,11 +679,11 @@ def test_bound_method_introspection(self): def test_unbound_method_retrieval(self): obj = self.A - self.assertFalse(hasattr(obj.both, "__self__")) - self.assertFalse(hasattr(obj.nested, "__self__")) - self.assertFalse(hasattr(obj.over_partial, "__self__")) - self.assertFalse(hasattr(obj.static, "__self__")) - self.assertFalse(hasattr(self.a.static, "__self__")) + self.assertNotHasAttr(obj.both, "__self__") + self.assertNotHasAttr(obj.nested, "__self__") + self.assertNotHasAttr(obj.over_partial, "__self__") + self.assertNotHasAttr(obj.static, "__self__") + self.assertNotHasAttr(self.a.static, "__self__") def test_descriptors(self): for obj in [self.A, self.a]: @@ -632,6 +747,20 @@ def f(a, b, /): p = functools.partial(f, 1) self.assertEqual(p(2), f(1, 2)) + def test_subclass_optimization(self): + class PartialMethodSubclass(functools.partialmethod): + pass + # `partialmethod` input to `partialmethod` subclass + p = functools.partialmethod(min, 2) + p2 = PartialMethodSubclass(p, 1) + self.assertIs(p2.func, min) + self.assertEqual(p2.__get__(0)(), 0) + # `partialmethod` subclass input to `partialmethod` subclass + p = PartialMethodSubclass(min, 2) + p2 = PartialMethodSubclass(p, 1) + self.assertIs(p2.func, min) + self.assertEqual(p2.__get__(0)(), 0) + class TestUpdateWrapper(unittest.TestCase): @@ -696,7 +825,7 @@ def wrapper(): self.assertNotEqual(wrapper.__qualname__, f.__qualname__) self.assertEqual(wrapper.__doc__, None) self.assertEqual(wrapper.__annotations__, {}) - self.assertFalse(hasattr(wrapper, 'attr')) + self.assertNotHasAttr(wrapper, 'attr') def test_selective_update(self): def f(): @@ -745,7 +874,7 @@ def wrapper(): pass functools.update_wrapper(wrapper, max) self.assertEqual(wrapper.__name__, 'max') - self.assertTrue(wrapper.__doc__.startswith('max(')) + self.assertStartsWith(wrapper.__doc__, 'max(') self.assertEqual(wrapper.__annotations__, {}) def test_update_type_wrapper(self): @@ -756,6 +885,26 @@ def wrapper(*args): pass self.assertEqual(wrapper.__annotations__, {}) self.assertEqual(wrapper.__type_params__, ()) + def test_update_wrapper_annotations(self): + def inner(x: int): pass + def wrapper(*args): pass + + functools.update_wrapper(wrapper, inner) + self.assertEqual(wrapper.__annotations__, {'x': int}) + self.assertIs(wrapper.__annotate__, inner.__annotate__) + + def with_forward_ref(x: undefined): pass + def wrapper(*args): pass + + functools.update_wrapper(wrapper, with_forward_ref) + + self.assertIs(wrapper.__annotate__, with_forward_ref.__annotate__) + with self.assertRaises(NameError): + wrapper.__annotations__ + + undefined = str + self.assertEqual(wrapper.__annotations__, {'x': undefined}) + class TestWraps(TestUpdateWrapper): @@ -795,7 +944,7 @@ def wrapper(): self.assertEqual(wrapper.__name__, 'wrapper') self.assertNotEqual(wrapper.__qualname__, f.__qualname__) self.assertEqual(wrapper.__doc__, None) - self.assertFalse(hasattr(wrapper, 'attr')) + self.assertNotHasAttr(wrapper, 'attr') def test_selective_update(self): def f(): @@ -897,6 +1046,29 @@ def __getitem__(self, i): d = {"one": 1, "two": 2, "three": 3} self.assertEqual(self.reduce(add, d), "".join(d.keys())) + # test correctness of keyword usage of `initial` in `reduce` + def test_initial_keyword(self): + def add(x, y): + return x + y + self.assertEqual( + self.reduce(add, ['a', 'b', 'c'], ''), + self.reduce(add, ['a', 'b', 'c'], initial=''), + ) + self.assertEqual( + self.reduce(add, [['a', 'c'], [], ['d', 'w']], []), + self.reduce(add, [['a', 'c'], [], ['d', 'w']], initial=[]), + ) + self.assertEqual( + self.reduce(lambda x, y: x*y, range(2,8), 1), + self.reduce(lambda x, y: x*y, range(2,8), initial=1), + ) + self.assertEqual( + self.reduce(lambda x, y: x*y, range(2,21), 1), + self.reduce(lambda x, y: x*y, range(2,21), initial=1), + ) + self.assertRaises(TypeError, self.reduce, add, [0, 1], initial="") + self.assertEqual(self.reduce(42, "", initial="1"), "1") # func is never called with one item + @unittest.skipUnless(c_functools, 'requires the C _functools module') class TestReduceC(TestReduce, unittest.TestCase): @@ -907,6 +1079,12 @@ class TestReduceC(TestReduce, unittest.TestCase): class TestReducePy(TestReduce, unittest.TestCase): reduce = staticmethod(py_functools.reduce) + def test_reduce_with_kwargs(self): + with self.assertWarns(DeprecationWarning): + self.reduce(function=lambda x, y: x + y, sequence=[1, 2, 3, 4, 5], initial=1) + with self.assertWarns(DeprecationWarning): + self.reduce(lambda x, y: x + y, sequence=[1, 2, 3, 4, 5], initial=1) + class TestCmpToKey: @@ -1014,39 +1192,47 @@ def test_disallow_instantiation(self): self, type(c_functools.cmp_to_key(None)) ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; + (mycmp) + def test_cmp_to_signature(self): + return super().test_cmp_to_signature() + + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() got multiple values for argument 'mycmp' + def test_cmp_to_key_arguments(self): + return super().test_cmp_to_key_arguments() + + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() got multiple values for argument 'mycmp' + def test_obj_field(self): + return super().test_obj_field() + + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() takes 1 positional argument but 2 were given def test_bad_cmp(self): return super().test_bad_cmp() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() takes 1 positional argument but 2 were given def test_cmp_to_key(self): return super().test_cmp_to_key() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_cmp_to_key_arguments(self): - return super().test_cmp_to_key_arguments() - - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_cmp_to_signature(self): - return super().test_cmp_to_signature() - - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() takes 1 positional argument but 2 were given def test_hash(self): return super().test_hash() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_obj_field(self): - return super().test_obj_field() - - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() takes 1 positional argument but 2 were given def test_sort_int(self): return super().test_sort_int() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() takes 1 positional argument but 2 were given def test_sort_int_str(self): return super().test_sort_int_str() + + + + + + + + class TestCmpToKeyPy(TestCmpToKey, unittest.TestCase): cmp_to_key = staticmethod(py_functools.cmp_to_key) @@ -1532,6 +1718,7 @@ def f(x): f(0, **{}) self.assertEqual(f.cache_info().hits, 1) + @unittest.expectedFailure # TODO: RUSTPYTHON; Python lru_cache impl doesn't cache hash like C impl def test_lru_hash_only_once(self): # To protect against weird reentrancy bugs and to improve # efficiency when faced with slow __hash__ methods, the @@ -1925,7 +2112,7 @@ def f(): return 1 self.assertEqual(f.cache_parameters(), {'maxsize': 1000, "typed": True}) - @support.suppress_immortalization() + @unittest.expectedFailure # TODO: RUSTPYTHON; GC behavior differs from CPython's refcounting def test_lru_cache_weakrefable(self): @self.module.lru_cache def test_function(x): @@ -1963,8 +2150,38 @@ def orig(a, /, b, c=True): ... self.assertEqual(str(Signature.from_callable(lru.cache_info)), '()') self.assertEqual(str(Signature.from_callable(lru.cache_clear)), '()') + def test_get_annotations(self): + def orig(a: int) -> str: ... + lru = self.module.lru_cache(1)(orig) + + self.assertEqual( + get_annotations(orig), {"a": int, "return": str}, + ) + self.assertEqual( + get_annotations(lru), {"a": int, "return": str}, + ) + + def test_get_annotations_with_forwardref(self): + def orig(a: int) -> nonexistent: ... + lru = self.module.lru_cache(1)(orig) + + self.assertEqual( + get_annotations(orig, format=Format.FORWARDREF), + {"a": int, "return": EqualToForwardRef('nonexistent', owner=orig)}, + ) + self.assertEqual( + get_annotations(lru, format=Format.FORWARDREF), + {"a": int, "return": EqualToForwardRef('nonexistent', owner=lru)}, + ) + with self.assertRaises(NameError): + get_annotations(orig, format=Format.VALUE) + with self.assertRaises(NameError): + get_annotations(lru, format=Format.VALUE) + @support.skip_on_s390x @unittest.skipIf(support.is_wasi, "WASI has limited C stack") + @support.skip_if_sanitizer("requires deep stack", ub=True, thread=True) + @support.skip_emscripten_stack_overflow() def test_lru_recursion(self): @self.module.lru_cache @@ -1973,15 +2190,12 @@ def fib(n): return n return fib(n-1) + fib(n-2) - if not support.Py_DEBUG: - depth = support.get_c_recursion_limit()*2//7 - with support.infinite_recursion(): - fib(depth) + fib(100) if self.module == c_functools: fib.cache_clear() with support.infinite_recursion(): with self.assertRaises(RecursionError): - fib(10000) + fib(support.exceeds_recursion_limit()) @py_functools.lru_cache() @@ -2563,15 +2777,15 @@ def _(self, arg): a.t(0) self.assertEqual(a.arg, "int") aa = A() - self.assertFalse(hasattr(aa, 'arg')) + self.assertNotHasAttr(aa, 'arg') a.t('') self.assertEqual(a.arg, "str") aa = A() - self.assertFalse(hasattr(aa, 'arg')) + self.assertNotHasAttr(aa, 'arg') a.t(0.0) self.assertEqual(a.arg, "base") aa = A() - self.assertFalse(hasattr(aa, 'arg')) + self.assertNotHasAttr(aa, 'arg') def test_staticmethod_register(self): class A: @@ -2806,6 +3020,8 @@ def static_func(arg: int) -> str: A().static_func ): with self.subTest(meth=meth): + self.assertEqual(meth.__module__, __name__) + self.assertEqual(type(meth).__module__, 'functools') self.assertEqual(meth.__qualname__, prefix + meth.__name__) self.assertEqual(meth.__doc__, ('My function docstring' @@ -2820,6 +3036,67 @@ def static_func(arg: int) -> str: self.assertEqual(A.static_func.__name__, 'static_func') self.assertEqual(A().static_func.__name__, 'static_func') + def test_method_repr(self): + class Callable: + def __call__(self, *args): + pass + + class CallableWithName: + __name__ = 'NOQUALNAME' + def __call__(self, *args): + pass + + class A: + @functools.singledispatchmethod + def func(self, arg): + pass + @functools.singledispatchmethod + @classmethod + def cls_func(cls, arg): + pass + @functools.singledispatchmethod + @staticmethod + def static_func(arg): + pass + # No __qualname__, only __name__ + no_qualname = functools.singledispatchmethod(CallableWithName()) + # No __qualname__, no __name__ + no_name = functools.singledispatchmethod(Callable()) + + self.assertEqual(repr(A.__dict__['func']), + f'') + self.assertEqual(repr(A.__dict__['cls_func']), + f'') + self.assertEqual(repr(A.__dict__['static_func']), + f'') + self.assertEqual(repr(A.__dict__['no_qualname']), + f'') + self.assertEqual(repr(A.__dict__['no_name']), + f'') + + self.assertEqual(repr(A.func), + f'') + self.assertEqual(repr(A.cls_func), + f'') + self.assertEqual(repr(A.static_func), + f'') + self.assertEqual(repr(A.no_qualname), + f'') + self.assertEqual(repr(A.no_name), + f'') + + a = A() + self.assertEqual(repr(a.func), + f'') + self.assertEqual(repr(a.cls_func), + f'') + self.assertEqual(repr(a.static_func), + f'') + self.assertEqual(repr(a.no_qualname), + f'') + self.assertEqual(repr(a.no_name), + f'') + def test_double_wrapped_methods(self): def classmethod_friendly_decorator(func): wrapped = func.__func__ @@ -2836,7 +3113,8 @@ def cls_context_manager(cls, arg: int) -> str: try: yield str(arg) finally: - return 'Done' + pass + return 'Done' @classmethod_friendly_decorator @classmethod @@ -2852,7 +3130,8 @@ def cls_context_manager(cls, arg: int) -> str: try: yield str(arg) finally: - return 'Done' + pass + return 'Done' @functools.singledispatchmethod @classmethod_friendly_decorator @@ -2935,16 +3214,16 @@ def i(arg): @i.register(42) def _(arg): return "I annotated with a non-type" - self.assertTrue(str(exc.exception).startswith(msg_prefix + "42")) - self.assertTrue(str(exc.exception).endswith(msg_suffix)) + self.assertStartsWith(str(exc.exception), msg_prefix + "42") + self.assertEndsWith(str(exc.exception), msg_suffix) with self.assertRaises(TypeError) as exc: @i.register def _(arg): return "I forgot to annotate" - self.assertTrue(str(exc.exception).startswith(msg_prefix + + self.assertStartsWith(str(exc.exception), msg_prefix + "._" - )) - self.assertTrue(str(exc.exception).endswith(msg_suffix)) + ) + self.assertEndsWith(str(exc.exception), msg_suffix) with self.assertRaises(TypeError) as exc: @i.register @@ -2954,23 +3233,23 @@ def _(arg: typing.Iterable[str]): # types from `typing`. Instead, annotate with regular types # or ABCs. return "I annotated with a generic collection" - self.assertTrue(str(exc.exception).startswith( + self.assertStartsWith(str(exc.exception), "Invalid annotation for 'arg'." - )) - self.assertTrue(str(exc.exception).endswith( + ) + self.assertEndsWith(str(exc.exception), 'typing.Iterable[str] is not a class.' - )) + ) with self.assertRaises(TypeError) as exc: @i.register def _(arg: typing.Union[int, typing.Iterable[str]]): return "Invalid Union" - self.assertTrue(str(exc.exception).startswith( + self.assertStartsWith(str(exc.exception), "Invalid annotation for 'arg'." - )) - self.assertTrue(str(exc.exception).endswith( - 'typing.Union[int, typing.Iterable[str]] not all arguments are classes.' - )) + ) + self.assertEndsWith(str(exc.exception), + 'int | typing.Iterable[str] not all arguments are classes.' + ) def test_invalid_positional_argument(self): @functools.singledispatch @@ -3117,6 +3396,28 @@ def _(arg: typing.List[float] | bytes): self.assertEqual(f(""), "default") self.assertEqual(f(b""), "default") + def test_forward_reference(self): + @functools.singledispatch + def f(arg, arg2=None): + return "default" + + @f.register + def _(arg: str, arg2: undefined = None): + return "forward reference" + + self.assertEqual(f(1), "default") + self.assertEqual(f(""), "forward reference") + + def test_unresolved_forward_reference(self): + @functools.singledispatch + def f(arg): + return "default" + + with self.assertRaisesRegex(TypeError, "is an unresolved forward reference"): + @f.register + def _(arg: undefined): + return "forward reference" + def test_method_equal_instances(self): # gh-127750: Reference to self was cached class A: @@ -3299,7 +3600,6 @@ class MyClass(metaclass=MyMeta): ): MyClass.prop - @unittest.expectedFailure # TODO: RUSTPYTHON def test_reuse_different_names(self): """Disallow this case because decorated function a would not be cached.""" with self.assertRaises(TypeError) as ctx: diff --git a/Lib/test/test_future_stmt/test_future.py b/Lib/test/test_future_stmt/test_future.py index a7d649b32a6..e57c7227cec 100644 --- a/Lib/test/test_future_stmt/test_future.py +++ b/Lib/test/test_future_stmt/test_future.py @@ -195,7 +195,7 @@ def test_syntactical_future_repl(self): out = kill_python(p) self.assertNotIn(b'SyntaxError: invalid syntax', out) - @unittest.skip('TODO: RUSTPYTHON') + @unittest.expectedFailure # TODO: RUSTPYTHON # SyntaxError: future feature spam is not defined def test_future_dotted_import(self): with self.assertRaises(ImportError): @@ -459,9 +459,6 @@ def test_infinity_numbers(self): self.assertAnnotationEqual("('inf', 1e1000, 'infxxx', 1e1000j)", expected=f"('inf', {inf}, 'infxxx', {infj})") self.assertAnnotationEqual("(1e1000, (1e1000j,))", expected=f"({inf}, ({infj},))") - # TODO: RUSTPYTHON - # AssertionError: SyntaxError not raised - @unittest.expectedFailure def test_annotation_with_complex_target(self): with self.assertRaises(SyntaxError): exec( diff --git a/Lib/test/test_generators.py b/Lib/test/test_generators.py index 853767135aa..5559d58cad4 100644 --- a/Lib/test/test_generators.py +++ b/Lib/test/test_generators.py @@ -176,7 +176,6 @@ def f(): g.send(0) self.assertEqual(next(g), 1) - @unittest.expectedFailure # TODO: RUSTPYTHON; NotImplementedError def test_handle_frame_object_in_creation(self): #Attempt to expose partially constructed frames diff --git a/Lib/test/test_genericclass.py b/Lib/test/test_genericclass.py index aa843f6d9f0..498904dd97f 100644 --- a/Lib/test/test_genericclass.py +++ b/Lib/test/test_genericclass.py @@ -1,5 +1,6 @@ import unittest from test import support +from test.support.import_helper import import_module class TestMROEntry(unittest.TestCase): @@ -98,7 +99,7 @@ def __mro_entries__(self): return () d = C_too_few() with self.assertRaises(TypeError): - class D(d): ... + class E(d): ... def test_mro_entry_errors_2(self): class C_not_callable: @@ -111,7 +112,7 @@ def __mro_entries__(self): return object c = C_not_tuple() with self.assertRaises(TypeError): - class D(c): ... + class E(c): ... def test_mro_entry_metaclass(self): meta_args = [] @@ -227,8 +228,7 @@ def __class_getitem__(cls, one, two): with self.assertRaises(TypeError): C_too_many[int] - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_class_getitem_errors_2(self): class C: def __class_getitem__(cls, item): @@ -279,7 +279,9 @@ def __class_getitem__(cls, item): class CAPITest(unittest.TestCase): def test_c_class(self): - from _testcapi import Generic, GenericAlias + _testcapi = import_module("_testcapi") + Generic = _testcapi.Generic + GenericAlias = _testcapi.GenericAlias self.assertIsInstance(Generic.__class_getitem__(int), GenericAlias) IntGeneric = Generic[int] diff --git a/Lib/test/test_genericpath.py b/Lib/test/test_genericpath.py index 89e4fe1882e..ab580dfad0f 100644 --- a/Lib/test/test_genericpath.py +++ b/Lib/test/test_genericpath.py @@ -7,9 +7,9 @@ import sys import unittest import warnings -from test.support import ( - is_apple, is_emscripten, os_helper, warnings_helper -) +from test import support +from test.support import os_helper +from test.support import warnings_helper from test.support.script_helper import assert_python_ok from test.support.os_helper import FakePath @@ -92,8 +92,8 @@ def test_commonprefix(self): for s1 in testlist: for s2 in testlist: p = commonprefix([s1, s2]) - self.assertTrue(s1.startswith(p)) - self.assertTrue(s2.startswith(p)) + self.assertStartsWith(s1, p) + self.assertStartsWith(s2, p) if s1 != s2: n = len(p) self.assertNotEqual(s1[n:n+1], s2[n:n+1]) @@ -161,7 +161,6 @@ def test_exists(self): self.assertIs(self.pathmodule.lexists(path=filename), True) @unittest.skipUnless(hasattr(os, "pipe"), "requires os.pipe()") - @unittest.skipIf(is_emscripten, "Emscripten pipe fds have no stat") def test_exists_fd(self): r, w = os.pipe() try: @@ -171,8 +170,7 @@ def test_exists_fd(self): os.close(w) self.assertFalse(self.pathmodule.exists(r)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_exists_bool(self): for fd in False, True: with self.assertWarnsRegex(RuntimeWarning, @@ -352,7 +350,6 @@ def test_invalid_paths(self): with self.assertRaisesRegex(ValueError, 'embedded null'): func(b'/tmp\x00abcds') - # Following TestCase is not supposed to be run from test_genericpath. # It is inherited by other test modules (ntpath, posixpath). @@ -449,6 +446,19 @@ def check(value, expected): os.fsencode('$bar%s bar' % nonascii)) check(b'$spam}bar', os.fsencode('%s}bar' % nonascii)) + @support.requires_resource('cpu') + def test_expandvars_large(self): + expandvars = self.pathmodule.expandvars + with os_helper.EnvironmentVarGuard() as env: + env.clear() + env["A"] = "B" + n = 100_000 + self.assertEqual(expandvars('$A'*n), 'B'*n) + self.assertEqual(expandvars('${A}'*n), 'B'*n) + self.assertEqual(expandvars('$A!'*n), 'B!'*n) + self.assertEqual(expandvars('${A}A'*n), 'BA'*n) + self.assertEqual(expandvars('${'*10*n), '${'*10*n) + def test_abspath(self): self.assertIn("foo", self.pathmodule.abspath("foo")) with warnings.catch_warnings(): @@ -506,7 +516,7 @@ def test_nonascii_abspath(self): # directory (when the bytes name is used). and sys.platform not in { "win32", "emscripten", "wasi" - } and not is_apple + } and not support.is_apple ): name = os_helper.TESTFN_UNDECODABLE elif os_helper.TESTFN_NONASCII: diff --git a/Lib/test/test_gettext.py b/Lib/test/test_gettext.py index 0653bb762a9..f4069082969 100644 --- a/Lib/test/test_gettext.py +++ b/Lib/test/test_gettext.py @@ -6,13 +6,12 @@ from functools import partial from test import support -from test.support import os_helper +from test.support import cpython_only, os_helper +from test.support.import_helper import ensure_lazy_imports # TODO: # - Add new tests, for example for "dgettext" -# - Remove dummy tests, for example testing for single and double quotes -# has no sense, it would have if we were testing a parser (i.e. pygettext) # - Tests should have only one assert. GNU_MO_DATA = b'''\ @@ -231,30 +230,6 @@ def test_some_translations_with_context(self): eq(pgettext('my other context', 'nudge nudge'), 'wink wink (in "my other context")') - def test_double_quotes(self): - eq = self.assertEqual - # double quotes - eq(_("albatross"), 'albatross') - eq(_("mullusk"), 'bacon') - eq(_(r"Raymond Luxury Yach-t"), 'Throatwobbler Mangrove') - eq(_(r"nudge nudge"), 'wink wink') - - def test_triple_single_quotes(self): - eq = self.assertEqual - # triple single quotes - eq(_('''albatross'''), 'albatross') - eq(_('''mullusk'''), 'bacon') - eq(_(r'''Raymond Luxury Yach-t'''), 'Throatwobbler Mangrove') - eq(_(r'''nudge nudge'''), 'wink wink') - - def test_triple_double_quotes(self): - eq = self.assertEqual - # triple double quotes - eq(_("""albatross"""), 'albatross') - eq(_("""mullusk"""), 'bacon') - eq(_(r"""Raymond Luxury Yach-t"""), 'Throatwobbler Mangrove') - eq(_(r"""nudge nudge"""), 'wink wink') - def test_multiline_strings(self): eq = self.assertEqual # multiline strings @@ -367,30 +342,6 @@ def test_some_translations_with_context_and_domain(self): eq(gettext.dpgettext('gettext', 'my other context', 'nudge nudge'), 'wink wink (in "my other context")') - def test_double_quotes(self): - eq = self.assertEqual - # double quotes - eq(self._("albatross"), 'albatross') - eq(self._("mullusk"), 'bacon') - eq(self._(r"Raymond Luxury Yach-t"), 'Throatwobbler Mangrove') - eq(self._(r"nudge nudge"), 'wink wink') - - def test_triple_single_quotes(self): - eq = self.assertEqual - # triple single quotes - eq(self._('''albatross'''), 'albatross') - eq(self._('''mullusk'''), 'bacon') - eq(self._(r'''Raymond Luxury Yach-t'''), 'Throatwobbler Mangrove') - eq(self._(r'''nudge nudge'''), 'wink wink') - - def test_triple_double_quotes(self): - eq = self.assertEqual - # triple double quotes - eq(self._("""albatross"""), 'albatross') - eq(self._("""mullusk"""), 'bacon') - eq(self._(r"""Raymond Luxury Yach-t"""), 'Throatwobbler Mangrove') - eq(self._(r"""nudge nudge"""), 'wink wink') - def test_multiline_strings(self): eq = self.assertEqual # multiline strings @@ -434,8 +385,7 @@ def _test_plural_forms(self, ngettext, gettext, x = ngettext(singular, plural, None) self.assertEqual(x, tplural) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_plural_forms(self): self._test_plural_forms( self.ngettext, self.gettext, @@ -446,8 +396,7 @@ def test_plural_forms(self): '%d file deleted', '%d files deleted', '%d file deleted', '%d files deleted') - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_plural_context_forms(self): ngettext = partial(self.npgettext, 'With context') gettext = partial(self.pgettext, 'With context') @@ -460,8 +409,7 @@ def test_plural_context_forms(self): '%d file deleted', '%d files deleted', '%d file deleted', '%d files deleted') - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_plural_wrong_context_forms(self): self._test_plural_forms( partial(self.npgettext, 'Unknown context'), @@ -494,8 +442,7 @@ def setUp(self): self.pgettext = partial(gettext.dpgettext, 'gettext') self.npgettext = partial(gettext.dnpgettext, 'gettext') - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_plural_forms_wrong_domain(self): self._test_plural_forms( partial(gettext.dngettext, 'unknown'), @@ -504,8 +451,7 @@ def test_plural_forms_wrong_domain(self): 'There is %s file', 'There are %s files', numbers_only=False) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_plural_context_forms_wrong_domain(self): self._test_plural_forms( partial(gettext.dnpgettext, 'unknown', 'With context'), @@ -526,8 +472,7 @@ def setUp(self): self.pgettext = t.pgettext self.npgettext = t.npgettext - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_plural_forms_null_translations(self): t = gettext.NullTranslations() self._test_plural_forms( @@ -536,8 +481,7 @@ def test_plural_forms_null_translations(self): 'There is %s file', 'There are %s files', numbers_only=False) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_plural_context_forms_null_translations(self): t = gettext.NullTranslations() self._test_plural_forms( @@ -630,6 +574,7 @@ def test_ar(self): s = ''.join([ str(f(x)) for x in range(200) ]) eq(s, "01233333333444444444444444444444444444444444444444444444444444444444444444444444444444444444444444445553333333344444444444444444444444444444444444444444444444444444444444444444444444444444444444444444") + @support.skip_wasi_stack_overflow() def test_security(self): raises = self.assertRaises # Test for a dangerous expression @@ -994,6 +939,17 @@ def test__all__(self): support.check__all__(self, gettext, not_exported={'c2py', 'ENOENT'}) + @cpython_only + def test_lazy_import(self): + ensure_lazy_imports("gettext", {"re", "warnings", "locale"}) + + +class TranslationFallbackTestCase(unittest.TestCase): + def test_translation_fallback(self): + with os_helper.temp_cwd() as tempdir: + t = gettext.translation('gettext', localedir=tempdir, fallback=True) + self.assertIsInstance(t, gettext.NullTranslations) + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_grammar.py b/Lib/test/test_grammar.py index 323f4ee4c61..aae5c2b1ce3 100644 --- a/Lib/test/test_grammar.py +++ b/Lib/test/test_grammar.py @@ -374,6 +374,7 @@ class F(C, A): self.assertEqual(F.__annotations__, {}) + @unittest.expectedFailure # TODO: RUSTPYTHON def test_var_annot_metaclass_semantics(self): class CMeta(type): @classmethod @@ -403,6 +404,7 @@ def test_var_annot_in_module(self): with self.assertRaises(NameError): ann_module3.D_bad_ann(5) + @unittest.expectedFailure # TODO: RUSTPYTHON def test_var_annot_simple_exec(self): gns = {}; lns= {} exec("'docstring'\n" diff --git a/Lib/test/test_graphlib.py b/Lib/test/test_graphlib.py index 5f38af4024c..66722e0b049 100644 --- a/Lib/test/test_graphlib.py +++ b/Lib/test/test_graphlib.py @@ -140,9 +140,21 @@ def test_calls_before_prepare(self): def test_prepare_multiple_times(self): ts = graphlib.TopologicalSorter() ts.prepare() - with self.assertRaisesRegex(ValueError, r"cannot prepare\(\) more than once"): + ts.prepare() + + def test_prepare_after_pass_out(self): + ts = graphlib.TopologicalSorter({'a': 'bc'}) + ts.prepare() + self.assertEqual(set(ts.get_ready()), {'b', 'c'}) + with self.assertRaisesRegex(ValueError, r"cannot prepare\(\) after starting sort"): ts.prepare() + def test_prepare_cycleerror_each_time(self): + ts = graphlib.TopologicalSorter({'a': 'b', 'b': 'a'}) + for attempt in range(1, 4): + with self.assertRaises(graphlib.CycleError, msg=f"{attempt=}"): + ts.prepare() + def test_invalid_nodes_in_done(self): ts = graphlib.TopologicalSorter() ts.add(1, 2, 3, 4) diff --git a/Lib/test/test_gzip.py b/Lib/test/test_gzip.py index b0d9613cdbd..4a8813c4da1 100644 --- a/Lib/test/test_gzip.py +++ b/Lib/test/test_gzip.py @@ -9,6 +9,7 @@ import struct import sys import unittest +import warnings from subprocess import PIPE, Popen from test.support import catch_unraisable_exception from test.support import import_helper @@ -143,6 +144,38 @@ def test_read1(self): self.assertEqual(f.tell(), nread) self.assertEqual(b''.join(blocks), data1 * 50) + def test_readinto(self): + # 10MB of uncompressible data to ensure multiple reads + large_data = os.urandom(10 * 2**20) + with gzip.GzipFile(self.filename, 'wb') as f: + f.write(large_data) + + buf = bytearray(len(large_data)) + with gzip.GzipFile(self.filename, 'r') as f: + nbytes = f.readinto(buf) + self.assertEqual(nbytes, len(large_data)) + self.assertEqual(buf, large_data) + + def test_readinto1(self): + # 10MB of uncompressible data to ensure multiple reads + large_data = os.urandom(10 * 2**20) + with gzip.GzipFile(self.filename, 'wb') as f: + f.write(large_data) + + nread = 0 + buf = bytearray(len(large_data)) + memview = memoryview(buf) # Simplifies slicing + with gzip.GzipFile(self.filename, 'r') as f: + for count in range(200): + nbytes = f.readinto1(memview[nread:]) + if not nbytes: + break + nread += nbytes + self.assertEqual(f.tell(), nread) + self.assertEqual(buf, large_data) + # readinto1() should require multiple loops + self.assertGreater(count, 1) + @bigmemtest(size=_4G, memuse=1) def test_read_large(self, size): # Read chunk size over UINT_MAX should be supported, despite zlib's @@ -298,13 +331,13 @@ def test_mode(self): def test_1647484(self): for mode in ('wb', 'rb'): with gzip.GzipFile(self.filename, mode) as f: - self.assertTrue(hasattr(f, "name")) + self.assertHasAttr(f, "name") self.assertEqual(f.name, self.filename) def test_paddedfile_getattr(self): self.test_write() with gzip.GzipFile(self.filename, 'rb') as f: - self.assertTrue(hasattr(f.fileobj, "name")) + self.assertHasAttr(f.fileobj, "name") self.assertEqual(f.fileobj.name, self.filename) def test_mtime(self): @@ -312,7 +345,7 @@ def test_mtime(self): with gzip.GzipFile(self.filename, 'w', mtime = mtime) as fWrite: fWrite.write(data1) with gzip.GzipFile(self.filename) as fRead: - self.assertTrue(hasattr(fRead, 'mtime')) + self.assertHasAttr(fRead, 'mtime') self.assertIsNone(fRead.mtime) dataRead = fRead.read() self.assertEqual(dataRead, data1) @@ -427,7 +460,7 @@ def test_zero_padded_file(self): self.assertEqual(d, data1 * 50, "Incorrect data in file") def test_gzip_BadGzipFile_exception(self): - self.assertTrue(issubclass(gzip.BadGzipFile, OSError)) + self.assertIsSubclass(gzip.BadGzipFile, OSError) def test_bad_gzip_file(self): with open(self.filename, 'wb') as file: @@ -715,6 +748,17 @@ def test_compress_mtime(self): f.read(1) # to set mtime attribute self.assertEqual(f.mtime, mtime) + def test_compress_mtime_default(self): + # test for gh-125260 + datac = gzip.compress(data1, mtime=0) + datac2 = gzip.compress(data1) + self.assertEqual(datac, datac2) + datac3 = gzip.compress(data1, mtime=None) + self.assertNotEqual(datac, datac3) + with gzip.GzipFile(fileobj=io.BytesIO(datac3), mode="rb") as f: + f.read(1) # to set mtime attribute + self.assertGreater(f.mtime, 1) + def test_compress_correct_level(self): for mtime in (0, 42): with self.subTest(mtime=mtime): @@ -856,9 +900,10 @@ def test_refloop_unraisable(self): # fileobj would be closed before the GzipFile as the result of a # reference loop. See issue gh-129726 with catch_unraisable_exception() as cm: - gzip.GzipFile(fileobj=io.BytesIO(), mode="w") - gc.collect() - self.assertIsNone(cm.unraisable) + with self.assertWarns(ResourceWarning): + gzip.GzipFile(fileobj=io.BytesIO(), mode="w") + gc.collect() + self.assertIsNone(cm.unraisable) class TestOpen(BaseTest): @@ -991,8 +1036,7 @@ def test_encoding_error_handler(self): as f: self.assertEqual(f.read(), "foobar") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_newline(self): # Test with explicit newline (universal newline mode disabled). uncompressed = data1.decode("ascii") * 50 diff --git a/Lib/test/test_heapq.py b/Lib/test/test_heapq.py index 1aa8e4e2897..d6623fee9bb 100644 --- a/Lib/test/test_heapq.py +++ b/Lib/test/test_heapq.py @@ -13,8 +13,9 @@ # _heapq.nlargest/nsmallest are saved in heapq._nlargest/_smallest when # _heapq is imported, so check them there -func_names = ['heapify', 'heappop', 'heappush', 'heappushpop', 'heapreplace', - '_heappop_max', '_heapreplace_max', '_heapify_max'] +func_names = ['heapify', 'heappop', 'heappush', 'heappushpop', 'heapreplace'] +# Add max-heap variants +func_names += [func + '_max' for func in func_names] class TestModules(TestCase): def test_py_functions(self): @@ -24,7 +25,7 @@ def test_py_functions(self): @skipUnless(c_heapq, 'requires _heapq') def test_c_functions(self): for fname in func_names: - self.assertEqual(getattr(c_heapq, fname).__module__, '_heapq') + self.assertEqual(getattr(c_heapq, fname).__module__, '_heapq', fname) def load_tests(loader, tests, ignore): @@ -74,6 +75,34 @@ def test_push_pop(self): except AttributeError: pass + def test_max_push_pop(self): + # 1) Push 256 random numbers and pop them off, verifying all's OK. + heap = [] + data = [] + self.check_max_invariant(heap) + for i in range(256): + item = random.random() + data.append(item) + self.module.heappush_max(heap, item) + self.check_max_invariant(heap) + results = [] + while heap: + item = self.module.heappop_max(heap) + self.check_max_invariant(heap) + results.append(item) + data_sorted = data[:] + data_sorted.sort(reverse=True) + + self.assertEqual(data_sorted, results) + # 2) Check that the invariant holds for a sorted array + self.check_max_invariant(results) + + self.assertRaises(TypeError, self.module.heappush_max, []) + + exc_types = (AttributeError, TypeError) + self.assertRaises(exc_types, self.module.heappush_max, None, None) + self.assertRaises(exc_types, self.module.heappop_max, None) + def check_invariant(self, heap): # Check the heap invariant. for pos, item in enumerate(heap): @@ -81,6 +110,11 @@ def check_invariant(self, heap): parentpos = (pos-1) >> 1 self.assertTrue(heap[parentpos] <= item) + def check_max_invariant(self, heap): + for pos, item in enumerate(heap[1:], start=1): + parentpos = (pos - 1) >> 1 + self.assertGreaterEqual(heap[parentpos], item) + def test_heapify(self): for size in list(range(30)) + [20000]: heap = [random.random() for dummy in range(size)] @@ -89,6 +123,14 @@ def test_heapify(self): self.assertRaises(TypeError, self.module.heapify, None) + def test_heapify_max(self): + for size in list(range(30)) + [20000]: + heap = [random.random() for dummy in range(size)] + self.module.heapify_max(heap) + self.check_max_invariant(heap) + + self.assertRaises(TypeError, self.module.heapify_max, None) + def test_naive_nbest(self): data = [random.randrange(2000) for i in range(1000)] heap = [] @@ -109,10 +151,7 @@ def heapiter(self, heap): def test_nbest(self): # Less-naive "N-best" algorithm, much faster (if len(data) is big - # enough ) than sorting all of data. However, if we had a max - # heap instead of a min heap, it could go faster still via - # heapify'ing all of data (linear time), then doing 10 heappops - # (10 log-time steps). + # enough ) than sorting all of data. data = [random.randrange(2000) for i in range(1000)] heap = data[:10] self.module.heapify(heap) @@ -125,6 +164,17 @@ def test_nbest(self): self.assertRaises(TypeError, self.module.heapreplace, None, None) self.assertRaises(IndexError, self.module.heapreplace, [], None) + def test_nbest_maxheap(self): + # With a max heap instead of a min heap, the "N-best" algorithm can + # go even faster still via heapify'ing all of data (linear time), then + # doing 10 heappops (10 log-time steps). + data = [random.randrange(2000) for i in range(1000)] + heap = data[:] + self.module.heapify_max(heap) + result = [self.module.heappop_max(heap) for _ in range(10)] + result.reverse() + self.assertEqual(result, sorted(data)[-10:]) + def test_nbest_with_pushpop(self): data = [random.randrange(2000) for i in range(1000)] heap = data[:10] @@ -134,6 +184,62 @@ def test_nbest_with_pushpop(self): self.assertEqual(list(self.heapiter(heap)), sorted(data)[-10:]) self.assertEqual(self.module.heappushpop([], 'x'), 'x') + def test_naive_nworst(self): + # Max-heap variant of "test_naive_nbest" + data = [random.randrange(2000) for i in range(1000)] + heap = [] + for item in data: + self.module.heappush_max(heap, item) + if len(heap) > 10: + self.module.heappop_max(heap) + heap.sort() + expected = sorted(data)[:10] + self.assertEqual(heap, expected) + + def heapiter_max(self, heap): + # An iterator returning a max-heap's elements, largest-first. + try: + while 1: + yield self.module.heappop_max(heap) + except IndexError: + pass + + def test_nworst(self): + # Max-heap variant of "test_nbest" + data = [random.randrange(2000) for i in range(1000)] + heap = data[:10] + self.module.heapify_max(heap) + for item in data[10:]: + if item < heap[0]: # this gets rarer the longer we run + self.module.heapreplace_max(heap, item) + expected = sorted(data, reverse=True)[-10:] + self.assertEqual(list(self.heapiter_max(heap)), expected) + + self.assertRaises(TypeError, self.module.heapreplace_max, None) + self.assertRaises(TypeError, self.module.heapreplace_max, None, None) + self.assertRaises(IndexError, self.module.heapreplace_max, [], None) + + def test_nworst_minheap(self): + # Min-heap variant of "test_nbest_maxheap" + data = [random.randrange(2000) for i in range(1000)] + heap = data[:] + self.module.heapify(heap) + result = [self.module.heappop(heap) for _ in range(10)] + result.reverse() + expected = sorted(data, reverse=True)[-10:] + self.assertEqual(result, expected) + + def test_nworst_with_pushpop(self): + # Max-heap variant of "test_nbest_with_pushpop" + data = [random.randrange(2000) for i in range(1000)] + heap = data[:10] + self.module.heapify_max(heap) + for item in data[10:]: + self.module.heappushpop_max(heap, item) + expected = sorted(data, reverse=True)[-10:] + self.assertEqual(list(self.heapiter_max(heap)), expected) + self.assertEqual(self.module.heappushpop_max([], 'x'), 'x') + def test_heappushpop(self): h = [] x = self.module.heappushpop(h, 10) @@ -153,12 +259,31 @@ def test_heappushpop(self): x = self.module.heappushpop(h, 11) self.assertEqual((h, x), ([11], 10)) + def test_heappushpop_max(self): + h = [] + x = self.module.heappushpop_max(h, 10) + self.assertTupleEqual((h, x), ([], 10)) + + h = [10] + x = self.module.heappushpop_max(h, 10.0) + self.assertTupleEqual((h, x), ([10], 10.0)) + self.assertIsInstance(h[0], int) + self.assertIsInstance(x, float) + + h = [10] + x = self.module.heappushpop_max(h, 11) + self.assertTupleEqual((h, x), ([10], 11)) + + h = [10] + x = self.module.heappushpop_max(h, 9) + self.assertTupleEqual((h, x), ([9], 10)) + def test_heappop_max(self): - # _heapop_max has an optimization for one-item lists which isn't + # heapop_max has an optimization for one-item lists which isn't # covered in other tests, so test that case explicitly here h = [3, 2] - self.assertEqual(self.module._heappop_max(h), 3) - self.assertEqual(self.module._heappop_max(h), 2) + self.assertEqual(self.module.heappop_max(h), 3) + self.assertEqual(self.module.heappop_max(h), 2) def test_heapsort(self): # Exercise everything with repeated heapsort checks @@ -175,6 +300,20 @@ def test_heapsort(self): heap_sorted = [self.module.heappop(heap) for i in range(size)] self.assertEqual(heap_sorted, sorted(data)) + def test_heapsort_max(self): + for trial in range(100): + size = random.randrange(50) + data = [random.randrange(25) for i in range(size)] + if trial & 1: # Half of the time, use heapify_max + heap = data[:] + self.module.heapify_max(heap) + else: # The rest of the time, use heappush_max + heap = [] + for item in data: + self.module.heappush_max(heap, item) + heap_sorted = [self.module.heappop_max(heap) for i in range(size)] + self.assertEqual(heap_sorted, sorted(data, reverse=True)) + def test_merge(self): inputs = [] for i in range(random.randrange(25)): @@ -377,16 +516,20 @@ def __lt__(self, other): class TestErrorHandling: def test_non_sequence(self): - for f in (self.module.heapify, self.module.heappop): + for f in (self.module.heapify, self.module.heappop, + self.module.heapify_max, self.module.heappop_max): self.assertRaises((TypeError, AttributeError), f, 10) for f in (self.module.heappush, self.module.heapreplace, + self.module.heappush_max, self.module.heapreplace_max, self.module.nlargest, self.module.nsmallest): self.assertRaises((TypeError, AttributeError), f, 10, 10) def test_len_only(self): - for f in (self.module.heapify, self.module.heappop): + for f in (self.module.heapify, self.module.heappop, + self.module.heapify_max, self.module.heappop_max): self.assertRaises((TypeError, AttributeError), f, LenOnly()) - for f in (self.module.heappush, self.module.heapreplace): + for f in (self.module.heappush, self.module.heapreplace, + self.module.heappush_max, self.module.heapreplace_max): self.assertRaises((TypeError, AttributeError), f, LenOnly(), 10) for f in (self.module.nlargest, self.module.nsmallest): self.assertRaises(TypeError, f, 2, LenOnly()) @@ -395,7 +538,8 @@ def test_cmp_err(self): seq = [CmpErr(), CmpErr(), CmpErr()] for f in (self.module.heapify, self.module.heappop): self.assertRaises(ZeroDivisionError, f, seq) - for f in (self.module.heappush, self.module.heapreplace): + for f in (self.module.heappush, self.module.heapreplace, + self.module.heappush_max, self.module.heapreplace_max): self.assertRaises(ZeroDivisionError, f, seq, 10) for f in (self.module.nlargest, self.module.nsmallest): self.assertRaises(ZeroDivisionError, f, 2, seq) @@ -403,6 +547,8 @@ def test_cmp_err(self): def test_arg_parsing(self): for f in (self.module.heapify, self.module.heappop, self.module.heappush, self.module.heapreplace, + self.module.heapify_max, self.module.heappop_max, + self.module.heappush_max, self.module.heapreplace_max, self.module.nlargest, self.module.nsmallest): self.assertRaises((TypeError, AttributeError), f, 10) @@ -424,6 +570,10 @@ def test_heappush_mutating_heap(self): # Python version raises IndexError, C version RuntimeError with self.assertRaises((IndexError, RuntimeError)): self.module.heappush(heap, SideEffectLT(5, heap)) + heap = [] + heap.extend(SideEffectLT(i, heap) for i in range(200)) + with self.assertRaises((IndexError, RuntimeError)): + self.module.heappush_max(heap, SideEffectLT(5, heap)) def test_heappop_mutating_heap(self): heap = [] @@ -431,8 +581,12 @@ def test_heappop_mutating_heap(self): # Python version raises IndexError, C version RuntimeError with self.assertRaises((IndexError, RuntimeError)): self.module.heappop(heap) + heap = [] + heap.extend(SideEffectLT(i, heap) for i in range(200)) + with self.assertRaises((IndexError, RuntimeError)): + self.module.heappop_max(heap) - def test_comparison_operator_modifiying_heap(self): + def test_comparison_operator_modifying_heap(self): # See bpo-39421: Strong references need to be taken # when comparing objects as they can alter the heap class EvilClass(int): @@ -444,7 +598,7 @@ def __lt__(self, o): self.module.heappush(heap, EvilClass(0)) self.assertRaises(IndexError, self.module.heappushpop, heap, 1) - def test_comparison_operator_modifiying_heap_two_heaps(self): + def test_comparison_operator_modifying_heap_two_heaps(self): class h(int): def __lt__(self, o): @@ -464,6 +618,17 @@ def __lt__(self, o): self.assertRaises((IndexError, RuntimeError), self.module.heappush, list1, g(1)) self.assertRaises((IndexError, RuntimeError), self.module.heappush, list2, h(1)) + list1, list2 = [], [] + + self.module.heappush_max(list1, h(0)) + self.module.heappush_max(list2, g(0)) + self.module.heappush_max(list1, g(1)) + self.module.heappush_max(list2, h(1)) + + self.assertRaises((IndexError, RuntimeError), self.module.heappush_max, list1, g(1)) + self.assertRaises((IndexError, RuntimeError), self.module.heappush_max, list2, h(1)) + + class TestErrorHandlingPython(TestErrorHandling, TestCase): module = py_heapq diff --git a/Lib/test/test_hmac.py b/Lib/test/test_hmac.py index 1726975e864..30cb02b6718 100644 --- a/Lib/test/test_hmac.py +++ b/Lib/test/test_hmac.py @@ -1,149 +1,429 @@ +"""Test suite for HMAC. + +Python provides three different implementations of HMAC: + +- OpenSSL HMAC using OpenSSL hash functions. +- HACL* HMAC using HACL* hash functions. +- Generic Python HMAC using user-defined hash functions. + +The generic Python HMAC implementation is able to use OpenSSL +callables or names, HACL* named hash functions or arbitrary +objects implementing PEP 247 interface. + +In the two first cases, Python HMAC wraps a C HMAC object (either OpenSSL +or HACL*-based). As a last resort, HMAC is re-implemented in pure Python. +It is however interesting to test the pure Python implementation against +the OpenSSL and HACL* hash functions. +""" + import binascii import functools import hmac import hashlib +import random +import types import unittest -import unittest.mock import warnings - -from test.support import hashlib_helper, check_disallow_instantiation - from _operator import _compare_digest as operator_compare_digest +from test.support import _4G, bigmemtest +from test.support import check_disallow_instantiation +from test.support import hashlib_helper, import_helper +from test.support.hashlib_helper import ( + BuiltinHashFunctionsTrait, + HashFunctionsTrait, + NamedHashFunctionsTrait, + OpenSSLHashFunctionsTrait, +) +from test.support.import_helper import import_fresh_module +from unittest.mock import patch try: - import _hashlib as _hashopenssl - from _hashlib import HMAC as C_HMAC - from _hashlib import hmac_new as c_hmac_new + import _hashlib from _hashlib import compare_digest as openssl_compare_digest except ImportError: - _hashopenssl = None - C_HMAC = None - c_hmac_new = None + _hashlib = None openssl_compare_digest = None try: - import _sha256 as sha256_module + import _sha2 as sha2 except ImportError: - sha256_module = None + sha2 = None -def ignore_warning(func): - @functools.wraps(func) - def wrapper(*args, **kwargs): - with warnings.catch_warnings(): - warnings.filterwarnings("ignore", - category=DeprecationWarning) - return func(*args, **kwargs) - return wrapper +def requires_builtin_sha2(): + return unittest.skipIf(sha2 is None, "requires _sha2") -class TestVectorsTestCase(unittest.TestCase): +class ModuleMixin: + """Mixin with a HMAC module implementation.""" - # TODO: RUSTPYTHON - @unittest.expectedFailure - def assert_hmac_internals( - self, h, digest, hashname, digest_size, block_size - ): - self.assertEqual(h.hexdigest().upper(), digest.upper()) - self.assertEqual(h.digest(), binascii.unhexlify(digest)) + hmac = None + + +class PyModuleMixin(ModuleMixin): + """Pure Python implementation of HMAC. + + The underlying hash functions may be OpenSSL-based or HACL* based, + depending on whether OpenSSL is present or not. + """ + + @classmethod + def setUpClass(cls): + super().setUpClass() + cls.hmac = import_fresh_module('hmac', blocked=['_hashlib', '_hmac']) + + +@hashlib_helper.requires_builtin_hmac() +class BuiltinModuleMixin(ModuleMixin): + """Built-in HACL* implementation of HMAC.""" + + @classmethod + def setUpClass(cls): + super().setUpClass() + cls.hmac = import_fresh_module('_hmac') + + +# Sentinel object used to detect whether a digestmod is given or not. +DIGESTMOD_SENTINEL = object() + + +class CreatorMixin: + """Mixin exposing a method creating a HMAC object.""" + + def hmac_new(self, key, msg=None, digestmod=DIGESTMOD_SENTINEL): + """Create a new HMAC object. + + Implementations should accept arbitrary 'digestmod' as this + method can be used to test which exceptions are being raised. + """ + raise NotImplementedError + + def bind_hmac_new(self, digestmod): + """Return a specialization of hmac_new() with a bound digestmod.""" + return functools.partial(self.hmac_new, digestmod=digestmod) + + +class DigestMixin: + """Mixin exposing a method computing a HMAC digest.""" + + def hmac_digest(self, key, msg=None, digestmod=DIGESTMOD_SENTINEL): + """Compute a HMAC digest. + + Implementations should accept arbitrary 'digestmod' as this + method can be used to test which exceptions are being raised. + """ + raise NotImplementedError + + def bind_hmac_digest(self, digestmod): + """Return a specialization of hmac_digest() with a bound digestmod.""" + return functools.partial(self.hmac_digest, digestmod=digestmod) + + +def _call_newobj_func(new_func, key, msg, digestmod): + if digestmod is DIGESTMOD_SENTINEL: # to test when digestmod is missing + return new_func(key, msg) # expected to raise + # functions creating HMAC objects take a 'digestmod' keyword argument + return new_func(key, msg, digestmod=digestmod) + + +def _call_digest_func(digest_func, key, msg, digestmod): + if digestmod is DIGESTMOD_SENTINEL: # to test when digestmod is missing + return digest_func(key, msg) # expected to raise + # functions directly computing digests take a 'digest' keyword argument + return digest_func(key, msg, digest=digestmod) + + +class ThroughObjectMixin(ModuleMixin, CreatorMixin, DigestMixin): + """Mixin delegating to .HMAC() and .HMAC(...).digest(). + + Both the C implementation and the Python implementation of HMAC should + expose a HMAC class with the same functionalities. + """ + + def hmac_new(self, key, msg=None, digestmod=DIGESTMOD_SENTINEL): + """Create a HMAC object via a module-level class constructor.""" + return _call_newobj_func(self.hmac.HMAC, key, msg, digestmod) + + def hmac_digest(self, key, msg=None, digestmod=DIGESTMOD_SENTINEL): + """Call the digest() method on a HMAC object obtained by hmac_new().""" + return _call_newobj_func(self.hmac_new, key, msg, digestmod).digest() + + +class ThroughModuleAPIMixin(ModuleMixin, CreatorMixin, DigestMixin): + """Mixin delegating to .new() and .digest().""" + + def hmac_new(self, key, msg=None, digestmod=DIGESTMOD_SENTINEL): + """Create a HMAC object via a module-level function.""" + return _call_newobj_func(self.hmac.new, key, msg, digestmod) + + def hmac_digest(self, key, msg=None, digestmod=DIGESTMOD_SENTINEL): + """One-shot HMAC digest computation.""" + return _call_digest_func(self.hmac.digest, key, msg, digestmod) + + +@hashlib_helper.requires_hashlib() +class ThroughOpenSSLAPIMixin(CreatorMixin, DigestMixin): + """Mixin delegating to _hashlib.hmac_new() and _hashlib.hmac_digest().""" + + def hmac_new(self, key, msg=None, digestmod=DIGESTMOD_SENTINEL): + return _call_newobj_func(_hashlib.hmac_new, key, msg, digestmod) + + def hmac_digest(self, key, msg=None, digestmod=DIGESTMOD_SENTINEL): + return _call_digest_func(_hashlib.hmac_digest, key, msg, digestmod) + + +class ThroughBuiltinAPIMixin(BuiltinModuleMixin, CreatorMixin, DigestMixin): + """Mixin delegating to _hmac.new() and _hmac.compute_digest().""" + + def hmac_new(self, key, msg=None, digestmod=DIGESTMOD_SENTINEL): + return _call_newobj_func(self.hmac.new, key, msg, digestmod) + + def hmac_digest(self, key, msg=None, digestmod=DIGESTMOD_SENTINEL): + return _call_digest_func(self.hmac.compute_digest, key, msg, digestmod) + + +class ObjectCheckerMixin: + """Mixin for checking HMAC objects (pure Python, OpenSSL or built-in).""" + + def check_object(self, h, hexdigest, hashname, digest_size, block_size): + """Check a HMAC object 'h' against the given values.""" + self.check_internals(h, hashname, digest_size, block_size) + self.check_hexdigest(h, hexdigest, digest_size) + + def check_internals(self, h, hashname, digest_size, block_size): + """Check the constant attributes of a HMAC object.""" self.assertEqual(h.name, f"hmac-{hashname}") self.assertEqual(h.digest_size, digest_size) self.assertEqual(h.block_size, block_size) - # TODO: RUSTPYTHON - @unittest.expectedFailure + def check_hexdigest(self, h, hexdigest, digest_size): + """Check the HMAC digest of 'h' and its size.""" + self.assertEqual(len(h.digest()), digest_size) + self.assertEqual(h.digest(), binascii.unhexlify(hexdigest)) + self.assertEqual(h.hexdigest().upper(), hexdigest.upper()) + + +class AssertersMixin(CreatorMixin, DigestMixin, ObjectCheckerMixin): + """Mixin class for common tests.""" + + def hmac_new_by_name(self, key, msg=None, *, hashname): + """Alternative implementation of hmac_new(). + + This is typically useful when one needs to test against an HMAC + implementation which only recognizes underlying hash functions + by their name (all HMAC implementations must at least recognize + hash functions by their names but some may use aliases such as + `hashlib.sha1` instead of "sha1"). + + Unlike hmac_new(), this method may assert the type of 'hashname' + as it should only be used in tests that are expected to create + a HMAC object. + """ + self.assertIsInstance(hashname, str) + return self.hmac_new(key, msg, digestmod=hashname) + + def hmac_digest_by_name(self, key, msg=None, *, hashname): + """Alternative implementation of hmac_digest(). + + Unlike hmac_digest(), this method may assert the type of 'hashname' + as it should only be used in tests that are expected to compute a + HMAC digest. + """ + self.assertIsInstance(hashname, str) + return self.hmac_digest(key, msg, digestmod=hashname) + def assert_hmac( - self, key, data, digest, hashfunc, hashname, digest_size, block_size + self, key, msg, hexdigest, hashfunc, hashname, digest_size, block_size ): - h = hmac.HMAC(key, data, digestmod=hashfunc) - self.assert_hmac_internals( - h, digest, hashname, digest_size, block_size + """Check that HMAC(key, msg) == digest. + + The 'hashfunc' and 'hashname' are used as 'digestmod' values, + thereby allowing to test the underlying dispatching mechanism. + + Note that 'hashfunc' may be a string, a callable, or a PEP-257 + module. Note that not all HMAC implementations may recognize the + same set of types for 'hashfunc', but they should always accept + a hash function by its name. + """ + if hashfunc == hashname: + choices = [hashname] + else: + choices = [hashfunc, hashname] + + for digestmod in choices: + with self.subTest(digestmod=digestmod): + self.assert_hmac_new( + key, msg, hexdigest, digestmod, + hashname, digest_size, block_size + ) + self.assert_hmac_hexdigest( + key, msg, hexdigest, digestmod, digest_size + ) + self.assert_hmac_common_cases( + key, msg, hexdigest, digestmod, + hashname, digest_size, block_size + ) + self.assert_hmac_extra_cases( + key, msg, hexdigest, digestmod, + hashname, digest_size, block_size + ) + + self.assert_hmac_new_by_name( + key, msg, hexdigest, hashname, digest_size, block_size + ) + self.assert_hmac_hexdigest_by_name( + key, msg, hexdigest, hashname, digest_size + ) + + def assert_hmac_new( + self, key, msg, hexdigest, digestmod, hashname, digest_size, block_size + ): + """Check that HMAC(key, msg) == digest. + + This test uses the `hmac_new()` method to create HMAC objects. + """ + self.check_hmac_new( + key, msg, hexdigest, hashname, digest_size, block_size, + hmac_new_func=self.hmac_new, + hmac_new_kwds={'digestmod': digestmod}, ) - h = hmac.HMAC(key, data, digestmod=hashname) - self.assert_hmac_internals( - h, digest, hashname, digest_size, block_size + def assert_hmac_new_by_name( + self, key, msg, hexdigest, hashname, digest_size, block_size + ): + """Check that HMAC(key, msg) == digest. + + This test uses the `hmac_new_by_name()` method to create HMAC objects. + """ + self.check_hmac_new( + key, msg, hexdigest, hashname, digest_size, block_size, + hmac_new_func=self.hmac_new_by_name, + hmac_new_kwds={'hashname': hashname}, ) - h = hmac.HMAC(key, digestmod=hashname) - h2 = h.copy() - h2.update(b"test update") - h.update(data) - self.assertEqual(h.hexdigest().upper(), digest.upper()) + def check_hmac_new( + self, key, msg, hexdigest, hashname, digest_size, block_size, + hmac_new_func, hmac_new_kwds=types.MappingProxyType({}), + ): + """Check that HMAC(key, msg) == digest. + + This also tests that using an empty/None initial message and + then calling `h.update(msg)` produces the same result, namely + that HMAC(key, msg) is equivalent to HMAC(key).update(msg). + """ + h = hmac_new_func(key, msg, **hmac_new_kwds) + self.check_object(h, hexdigest, hashname, digest_size, block_size) + + def hmac_new_feed(*args): + h = hmac_new_func(key, *args, **hmac_new_kwds) + h.update(msg) + self.check_hexdigest(h, hexdigest, digest_size) + + with self.subTest('no initial message'): + hmac_new_feed() + with self.subTest('initial message is empty'): + hmac_new_feed(b'') + with self.subTest('initial message is None'): + hmac_new_feed(None) + + def assert_hmac_hexdigest( + self, key, msg, hexdigest, digestmod, digest_size, + ): + """Check a HMAC digest computed by hmac_digest().""" + self.check_hmac_hexdigest( + key, msg, hexdigest, digest_size, + hmac_digest_func=self.hmac_digest, + hmac_digest_kwds={'digestmod': digestmod}, + ) - h = hmac.new(key, data, digestmod=hashname) - self.assert_hmac_internals( - h, digest, hashname, digest_size, block_size + def assert_hmac_hexdigest_by_name( + self, key, msg, hexdigest, hashname, digest_size + ): + """Check a HMAC digest computed by hmac_digest_by_name().""" + self.assertIsInstance(hashname, str) + self.check_hmac_hexdigest( + key, msg, hexdigest, digest_size, + hmac_digest_func=self.hmac_digest_by_name, + hmac_digest_kwds={'hashname': hashname}, ) - h = hmac.new(key, None, digestmod=hashname) - h.update(data) - self.assertEqual(h.hexdigest().upper(), digest.upper()) + def check_hmac_hexdigest( + self, key, msg, hexdigest, digest_size, + hmac_digest_func, hmac_digest_kwds=types.MappingProxyType({}), + ): + """Check and return a HMAC digest computed by hmac_digest_func(). - h = hmac.new(key, digestmod=hashname) - h.update(data) - self.assertEqual(h.hexdigest().upper(), digest.upper()) + This HMAC digest is computed by: - h = hmac.new(key, data, digestmod=hashfunc) - self.assertEqual(h.hexdigest().upper(), digest.upper()) + hmac_digest_func(key, msg, **hmac_digest_kwds) - self.assertEqual( - hmac.digest(key, data, digest=hashname), - binascii.unhexlify(digest) - ) - self.assertEqual( - hmac.digest(key, data, digest=hashfunc), - binascii.unhexlify(digest) - ) + This is typically useful for checking one-shot HMAC functions. + """ + d = hmac_digest_func(key, msg, **hmac_digest_kwds) + self.assertEqual(len(d), digest_size) + self.assertEqual(d, binascii.unhexlify(hexdigest)) + return d - h = hmac.HMAC.__new__(hmac.HMAC) - h._init_old(key, data, digestmod=hashname) - self.assert_hmac_internals( - h, digest, hashname, digest_size, block_size - ) + def assert_hmac_common_cases( + self, key, msg, hexdigest, digestmod, hashname, digest_size, block_size + ): + """Common tests executed by all subclasses.""" + h1 = self.hmac_new_by_name(key, hashname=hashname) + h2 = h1.copy() + h2.update(b"test update should not affect original") + h1.update(msg) + self.check_object(h1, hexdigest, hashname, digest_size, block_size) - if c_hmac_new is not None: - h = c_hmac_new(key, data, digestmod=hashname) - self.assert_hmac_internals( - h, digest, hashname, digest_size, block_size - ) + def assert_hmac_extra_cases( + self, key, msg, hexdigest, digestmod, hashname, digest_size, block_size + ): + """Extra tests that can be added in subclasses.""" - h = c_hmac_new(key, digestmod=hashname) - h2 = h.copy() - h2.update(b"test update") - h.update(data) - self.assertEqual(h.hexdigest().upper(), digest.upper()) - func = getattr(_hashopenssl, f"openssl_{hashname}") - h = c_hmac_new(key, data, digestmod=func) - self.assert_hmac_internals( - h, digest, hashname, digest_size, block_size - ) +class PyAssertersMixin(PyModuleMixin, AssertersMixin): - h = hmac.HMAC.__new__(hmac.HMAC) - h._init_hmac(key, data, digestmod=hashname) - self.assert_hmac_internals( - h, digest, hashname, digest_size, block_size - ) + def assert_hmac_extra_cases( + self, key, msg, hexdigest, digestmod, hashname, digest_size, block_size + ): + h = self.hmac.HMAC.__new__(self.hmac.HMAC) + h._init_old(key, msg, digestmod=digestmod) + self.check_object(h, hexdigest, hashname, digest_size, block_size) - # TODO: RUSTPYTHON - @unittest.expectedFailure - @hashlib_helper.requires_hashdigest('md5', openssl=True) - def test_md5_vectors(self): - # Test the HMAC module against test vectors from the RFC. - def md5test(key, data, digest): - self.assert_hmac( - key, data, digest, - hashfunc=hashlib.md5, - hashname="md5", - digest_size=16, - block_size=64 - ) +class OpenSSLAssertersMixin(ThroughOpenSSLAPIMixin, AssertersMixin): + + def hmac_new_by_name(self, key, msg=None, *, hashname): + self.assertIsInstance(hashname, str) + openssl_func = getattr(_hashlib, f"openssl_{hashname}") + return self.hmac_new(key, msg, digestmod=openssl_func) + + def hmac_digest_by_name(self, key, msg=None, *, hashname): + self.assertIsInstance(hashname, str) + openssl_func = getattr(_hashlib, f"openssl_{hashname}") + return self.hmac_digest(key, msg, digestmod=openssl_func) + + +class BuiltinAssertersMixin(ThroughBuiltinAPIMixin, AssertersMixin): + pass + + +class RFCTestCaseMixin(HashFunctionsTrait, AssertersMixin): + """Test HMAC implementations against RFC 2202/4231 and NIST test vectors. + + - Test vectors for MD5 and SHA-1 are taken from RFC 2202. + - Test vectors for SHA-2 are taken from RFC 4231. + - Test vectors for SHA-3 are NIST's test vectors [1]. + + [1] https://csrc.nist.gov/projects/message-authentication-codes + """ + + def test_md5_rfc2202(self): + def md5test(key, msg, hexdigest): + self.assert_hmac(key, msg, hexdigest, self.md5, "md5", 16, 64) md5test(b"\x0b" * 16, b"Hi There", - "9294727A3638BB1C13F48EF8158BFC9D") + "9294727a3638bb1c13f48ef8158bfc9d") md5test(b"Jefe", b"what do ya want for nothing?", @@ -170,18 +450,9 @@ def md5test(key, data, digest): b"and Larger Than One Block-Size Data"), "6f630fad67cda0ee1fb1f562db3aa53e") - # TODO: RUSTPYTHON - @unittest.expectedFailure - @hashlib_helper.requires_hashdigest('sha1', openssl=True) - def test_sha_vectors(self): - def shatest(key, data, digest): - self.assert_hmac( - key, data, digest, - hashfunc=hashlib.sha1, - hashname="sha1", - digest_size=20, - block_size=64 - ) + def test_sha1_rfc2202(self): + def shatest(key, msg, hexdigest): + self.assert_hmac(key, msg, hexdigest, self.sha1, "sha1", 20, 64) shatest(b"\x0b" * 20, b"Hi There", @@ -212,498 +483,1191 @@ def shatest(key, data, digest): b"and Larger Than One Block-Size Data"), "e8e99d0f45237d786d6bbaa7965c7808bbff1a91") - def _rfc4231_test_cases(self, hashfunc, hash_name, digest_size, block_size): - def hmactest(key, data, hexdigests): - digest = hexdigests[hashfunc] + def test_sha2_224_rfc4231(self): + self._test_sha2_rfc4231(self.sha224, 'sha224', 28, 64) + + def test_sha2_256_rfc4231(self): + self._test_sha2_rfc4231(self.sha256, 'sha256', 32, 64) + + def test_sha2_384_rfc4231(self): + self._test_sha2_rfc4231(self.sha384, 'sha384', 48, 128) + + def test_sha2_512_rfc4231(self): + self._test_sha2_rfc4231(self.sha512, 'sha512', 64, 128) + + def _test_sha2_rfc4231(self, hashfunc, hashname, digest_size, block_size): + def hmactest(key, msg, hexdigests): + hexdigest = hexdigests[hashname] self.assert_hmac( - key, data, digest, + key, msg, hexdigest, hashfunc=hashfunc, - hashname=hash_name, + hashname=hashname, digest_size=digest_size, block_size=block_size ) # 4.2. Test Case 1 - hmactest(key = b'\x0b'*20, - data = b'Hi There', - hexdigests = { - hashlib.sha224: '896fb1128abbdf196832107cd49df33f' - '47b4b1169912ba4f53684b22', - hashlib.sha256: 'b0344c61d8db38535ca8afceaf0bf12b' - '881dc200c9833da726e9376c2e32cff7', - hashlib.sha384: 'afd03944d84895626b0825f4ab46907f' - '15f9dadbe4101ec682aa034c7cebc59c' - 'faea9ea9076ede7f4af152e8b2fa9cb6', - hashlib.sha512: '87aa7cdea5ef619d4ff0b4241a1d6cb0' - '2379f4e2ce4ec2787ad0b30545e17cde' - 'daa833b7d6b8a702038b274eaea3f4e4' - 'be9d914eeb61f1702e696c203a126854', + hmactest(key=b'\x0b' * 20, + msg=b'Hi There', + hexdigests={ + 'sha224': '896fb1128abbdf196832107cd49df33f' + '47b4b1169912ba4f53684b22', + 'sha256': 'b0344c61d8db38535ca8afceaf0bf12b' + '881dc200c9833da726e9376c2e32cff7', + 'sha384': 'afd03944d84895626b0825f4ab46907f' + '15f9dadbe4101ec682aa034c7cebc59c' + 'faea9ea9076ede7f4af152e8b2fa9cb6', + 'sha512': '87aa7cdea5ef619d4ff0b4241a1d6cb0' + '2379f4e2ce4ec2787ad0b30545e17cde' + 'daa833b7d6b8a702038b274eaea3f4e4' + 'be9d914eeb61f1702e696c203a126854', }) # 4.3. Test Case 2 - hmactest(key = b'Jefe', - data = b'what do ya want for nothing?', - hexdigests = { - hashlib.sha224: 'a30e01098bc6dbbf45690f3a7e9e6d0f' - '8bbea2a39e6148008fd05e44', - hashlib.sha256: '5bdcc146bf60754e6a042426089575c7' - '5a003f089d2739839dec58b964ec3843', - hashlib.sha384: 'af45d2e376484031617f78d2b58a6b1b' - '9c7ef464f5a01b47e42ec3736322445e' - '8e2240ca5e69e2c78b3239ecfab21649', - hashlib.sha512: '164b7a7bfcf819e2e395fbe73b56e0a3' - '87bd64222e831fd610270cd7ea250554' - '9758bf75c05a994a6d034f65f8f0e6fd' - 'caeab1a34d4a6b4b636e070a38bce737', + hmactest(key=b'Jefe', + msg=b'what do ya want for nothing?', + hexdigests={ + 'sha224': 'a30e01098bc6dbbf45690f3a7e9e6d0f' + '8bbea2a39e6148008fd05e44', + 'sha256': '5bdcc146bf60754e6a042426089575c7' + '5a003f089d2739839dec58b964ec3843', + 'sha384': 'af45d2e376484031617f78d2b58a6b1b' + '9c7ef464f5a01b47e42ec3736322445e' + '8e2240ca5e69e2c78b3239ecfab21649', + 'sha512': '164b7a7bfcf819e2e395fbe73b56e0a3' + '87bd64222e831fd610270cd7ea250554' + '9758bf75c05a994a6d034f65f8f0e6fd' + 'caeab1a34d4a6b4b636e070a38bce737', }) # 4.4. Test Case 3 - hmactest(key = b'\xaa'*20, - data = b'\xdd'*50, - hexdigests = { - hashlib.sha224: '7fb3cb3588c6c1f6ffa9694d7d6ad264' - '9365b0c1f65d69d1ec8333ea', - hashlib.sha256: '773ea91e36800e46854db8ebd09181a7' - '2959098b3ef8c122d9635514ced565fe', - hashlib.sha384: '88062608d3e6ad8a0aa2ace014c8a86f' - '0aa635d947ac9febe83ef4e55966144b' - '2a5ab39dc13814b94e3ab6e101a34f27', - hashlib.sha512: 'fa73b0089d56a284efb0f0756c890be9' - 'b1b5dbdd8ee81a3655f83e33b2279d39' - 'bf3e848279a722c806b485a47e67c807' - 'b946a337bee8942674278859e13292fb', + hmactest(key=b'\xaa' * 20, + msg=b'\xdd' * 50, + hexdigests={ + 'sha224': '7fb3cb3588c6c1f6ffa9694d7d6ad264' + '9365b0c1f65d69d1ec8333ea', + 'sha256': '773ea91e36800e46854db8ebd09181a7' + '2959098b3ef8c122d9635514ced565fe', + 'sha384': '88062608d3e6ad8a0aa2ace014c8a86f' + '0aa635d947ac9febe83ef4e55966144b' + '2a5ab39dc13814b94e3ab6e101a34f27', + 'sha512': 'fa73b0089d56a284efb0f0756c890be9' + 'b1b5dbdd8ee81a3655f83e33b2279d39' + 'bf3e848279a722c806b485a47e67c807' + 'b946a337bee8942674278859e13292fb', }) # 4.5. Test Case 4 - hmactest(key = bytes(x for x in range(0x01, 0x19+1)), - data = b'\xcd'*50, - hexdigests = { - hashlib.sha224: '6c11506874013cac6a2abc1bb382627c' - 'ec6a90d86efc012de7afec5a', - hashlib.sha256: '82558a389a443c0ea4cc819899f2083a' - '85f0faa3e578f8077a2e3ff46729665b', - hashlib.sha384: '3e8a69b7783c25851933ab6290af6ca7' - '7a9981480850009cc5577c6e1f573b4e' - '6801dd23c4a7d679ccf8a386c674cffb', - hashlib.sha512: 'b0ba465637458c6990e5a8c5f61d4af7' - 'e576d97ff94b872de76f8050361ee3db' - 'a91ca5c11aa25eb4d679275cc5788063' - 'a5f19741120c4f2de2adebeb10a298dd', + hmactest(key=bytes(x for x in range(0x01, 0x19 + 1)), + msg=b'\xcd' * 50, + hexdigests={ + 'sha224': '6c11506874013cac6a2abc1bb382627c' + 'ec6a90d86efc012de7afec5a', + 'sha256': '82558a389a443c0ea4cc819899f2083a' + '85f0faa3e578f8077a2e3ff46729665b', + 'sha384': '3e8a69b7783c25851933ab6290af6ca7' + '7a9981480850009cc5577c6e1f573b4e' + '6801dd23c4a7d679ccf8a386c674cffb', + 'sha512': 'b0ba465637458c6990e5a8c5f61d4af7' + 'e576d97ff94b872de76f8050361ee3db' + 'a91ca5c11aa25eb4d679275cc5788063' + 'a5f19741120c4f2de2adebeb10a298dd', }) # 4.7. Test Case 6 - hmactest(key = b'\xaa'*131, - data = b'Test Using Larger Than Block-Siz' - b'e Key - Hash Key First', - hexdigests = { - hashlib.sha224: '95e9a0db962095adaebe9b2d6f0dbce2' - 'd499f112f2d2b7273fa6870e', - hashlib.sha256: '60e431591ee0b67f0d8a26aacbf5b77f' - '8e0bc6213728c5140546040f0ee37f54', - hashlib.sha384: '4ece084485813e9088d2c63a041bc5b4' - '4f9ef1012a2b588f3cd11f05033ac4c6' - '0c2ef6ab4030fe8296248df163f44952', - hashlib.sha512: '80b24263c7c1a3ebb71493c1dd7be8b4' - '9b46d1f41b4aeec1121b013783f8f352' - '6b56d037e05f2598bd0fd2215d6a1e52' - '95e64f73f63f0aec8b915a985d786598', + hmactest(key=b'\xaa' * 131, + msg=b'Test Using Larger Than Block-Siz' + b'e Key - Hash Key First', + hexdigests={ + 'sha224': '95e9a0db962095adaebe9b2d6f0dbce2' + 'd499f112f2d2b7273fa6870e', + 'sha256': '60e431591ee0b67f0d8a26aacbf5b77f' + '8e0bc6213728c5140546040f0ee37f54', + 'sha384': '4ece084485813e9088d2c63a041bc5b4' + '4f9ef1012a2b588f3cd11f05033ac4c6' + '0c2ef6ab4030fe8296248df163f44952', + 'sha512': '80b24263c7c1a3ebb71493c1dd7be8b4' + '9b46d1f41b4aeec1121b013783f8f352' + '6b56d037e05f2598bd0fd2215d6a1e52' + '95e64f73f63f0aec8b915a985d786598', }) # 4.8. Test Case 7 - hmactest(key = b'\xaa'*131, - data = b'This is a test using a larger th' - b'an block-size key and a larger t' - b'han block-size data. The key nee' - b'ds to be hashed before being use' - b'd by the HMAC algorithm.', - hexdigests = { - hashlib.sha224: '3a854166ac5d9f023f54d517d0b39dbd' - '946770db9c2b95c9f6f565d1', - hashlib.sha256: '9b09ffa71b942fcb27635fbcd5b0e944' - 'bfdc63644f0713938a7f51535c3a35e2', - hashlib.sha384: '6617178e941f020d351e2f254e8fd32c' - '602420feb0b8fb9adccebb82461e99c5' - 'a678cc31e799176d3860e6110c46523e', - hashlib.sha512: 'e37b6a775dc87dbaa4dfa9f96e5e3ffd' - 'debd71f8867289865df5a32d20cdc944' - 'b6022cac3c4982b10d5eeb55c3e4de15' - '134676fb6de0446065c97440fa8c6a58', + hmactest(key=b'\xaa' * 131, + msg=b'This is a test using a larger th' + b'an block-size key and a larger t' + b'han block-size data. The key nee' + b'ds to be hashed before being use' + b'd by the HMAC algorithm.', + hexdigests={ + 'sha224': '3a854166ac5d9f023f54d517d0b39dbd' + '946770db9c2b95c9f6f565d1', + 'sha256': '9b09ffa71b942fcb27635fbcd5b0e944' + 'bfdc63644f0713938a7f51535c3a35e2', + 'sha384': '6617178e941f020d351e2f254e8fd32c' + '602420feb0b8fb9adccebb82461e99c5' + 'a678cc31e799176d3860e6110c46523e', + 'sha512': 'e37b6a775dc87dbaa4dfa9f96e5e3ffd' + 'debd71f8867289865df5a32d20cdc944' + 'b6022cac3c4982b10d5eeb55c3e4de15' + '134676fb6de0446065c97440fa8c6a58', }) - # TODO: RUSTPYTHON - @unittest.expectedFailure - @hashlib_helper.requires_hashdigest('sha224', openssl=True) - def test_sha224_rfc4231(self): - self._rfc4231_test_cases(hashlib.sha224, 'sha224', 28, 64) - - # TODO: RUSTPYTHON - @unittest.expectedFailure - @hashlib_helper.requires_hashdigest('sha256', openssl=True) - def test_sha256_rfc4231(self): - self._rfc4231_test_cases(hashlib.sha256, 'sha256', 32, 64) - - # TODO: RUSTPYTHON - @unittest.expectedFailure - @hashlib_helper.requires_hashdigest('sha384', openssl=True) - def test_sha384_rfc4231(self): - self._rfc4231_test_cases(hashlib.sha384, 'sha384', 48, 128) - - # TODO: RUSTPYTHON - @unittest.expectedFailure - @hashlib_helper.requires_hashdigest('sha512', openssl=True) - def test_sha512_rfc4231(self): - self._rfc4231_test_cases(hashlib.sha512, 'sha512', 64, 128) + def test_sha3_224_nist(self): + for key, msg, hexdigest in [ + ( + bytes(range(28)), + b'Sample message for keylenblocklen', + '078695eecc227c636ad31d063a15dd05a7e819a66ec6d8de1e193e59' + ) + ]: + self.assert_hmac( + key, msg, hexdigest, + hashfunc=self.sha3_224, hashname='sha3_224', + digest_size=28, block_size=144 + ) - @hashlib_helper.requires_hashdigest('sha256') - def test_legacy_block_size_warnings(self): - class MockCrazyHash(object): - """Ain't no block_size attribute here.""" - def __init__(self, *args): - self._x = hashlib.sha256(*args) - self.digest_size = self._x.digest_size - def update(self, v): - self._x.update(v) - def digest(self): - return self._x.digest() + def test_sha3_256_nist(self): + for key, msg, hexdigest in [ + ( + bytes(range(32)), + b'Sample message for keylenblocklen', + '9bcf2c238e235c3ce88404e813bd2f3a' + '97185ac6f238c63d6229a00b07974258' + ) + ]: + self.assert_hmac( + key, msg, hexdigest, + hashfunc=self.sha3_256, hashname='sha3_256', + digest_size=32, block_size=136 + ) - with warnings.catch_warnings(): - warnings.simplefilter('error', RuntimeWarning) - with self.assertRaises(RuntimeWarning): - hmac.HMAC(b'a', b'b', digestmod=MockCrazyHash) - self.fail('Expected warning about missing block_size') + def test_sha3_384_nist(self): + for key, msg, hexdigest in [ + ( + bytes(range(48)), + b'Sample message for keylenblocklen', + 'e5ae4c739f455279368ebf36d4f5354c' + '95aa184c899d3870e460ebc288ef1f94' + '70053f73f7c6da2a71bcaec38ce7d6ac' + ) + ]: + self.assert_hmac( + key, msg, hexdigest, + hashfunc=self.sha3_384, hashname='sha3_384', + digest_size=48, block_size=104 + ) - MockCrazyHash.block_size = 1 - with self.assertRaises(RuntimeWarning): - hmac.HMAC(b'a', b'b', digestmod=MockCrazyHash) - self.fail('Expected warning about small block_size') + def test_sha3_512_nist(self): + for key, msg, hexdigest in [ + ( + bytes(range(64)), + b'Sample message for keylenblocklen', + '5f464f5e5b7848e3885e49b2c385f069' + '4985d0e38966242dc4a5fe3fea4b37d4' + '6b65ceced5dcf59438dd840bab22269f' + '0ba7febdb9fcf74602a35666b2a32915' + ) + ]: + self.assert_hmac( + key, msg, hexdigest, + hashfunc=self.sha3_512, hashname='sha3_512', + digest_size=64, block_size=72 + ) - def test_with_digestmod_no_default(self): - """The digestmod parameter is required as of Python 3.8.""" - with self.assertRaisesRegex(TypeError, r'required.*digestmod'): - key = b"\x0b" * 16 - data = b"Hi There" - hmac.HMAC(key, data, digestmod=None) - with self.assertRaisesRegex(TypeError, r'required.*digestmod'): - hmac.new(key, data) - with self.assertRaisesRegex(TypeError, r'required.*digestmod'): - hmac.HMAC(key, msg=data, digestmod='') - - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_with_fallback(self): - cache = getattr(hashlib, '__builtin_constructor_cache') - try: - cache['foo'] = hashlib.sha256 - hexdigest = hmac.digest(b'key', b'message', 'foo').hex() - expected = '6e9ef29b75fffc5b7abae527d58fdadb2fe42e7219011976917343065f58ed4a' - self.assertEqual(hexdigest, expected) - finally: - cache.pop('foo') +class PurePythonInitHMAC(PyModuleMixin, HashFunctionsTrait): + + @classmethod + def setUpClass(cls): + super().setUpClass() + for meth in ['_init_openssl_hmac', '_init_builtin_hmac']: + fn = getattr(cls.hmac.HMAC, meth) + cm = patch.object(cls.hmac.HMAC, meth, autospec=True, wraps=fn) + cls.enterClassContext(cm) + + @classmethod + def tearDownClass(cls): + cls.hmac.HMAC._init_openssl_hmac.assert_not_called() + cls.hmac.HMAC._init_builtin_hmac.assert_not_called() + # Do not assert that HMAC._init_old() has been called as it's tricky + # to determine whether a test for a specific hash function has been + # executed or not. On regular builds, it will be called but if a + # hash function is not available, it's hard to detect for which + # test we should checj HMAC._init_old() or not. + super().tearDownClass() + + +class PyRFCOpenSSLTestCase(ThroughObjectMixin, + PyAssertersMixin, + OpenSSLHashFunctionsTrait, + RFCTestCaseMixin, + PurePythonInitHMAC, + unittest.TestCase): + """Python implementation of HMAC using hmac.HMAC(). + + The underlying hash functions are OpenSSL-based but + _init_old() is used instead of _init_openssl_hmac(). + """ + + +class PyRFCBuiltinTestCase(ThroughObjectMixin, + PyAssertersMixin, + BuiltinHashFunctionsTrait, + RFCTestCaseMixin, + PurePythonInitHMAC, + unittest.TestCase): + """Python implementation of HMAC using hmac.HMAC(). + + The underlying hash functions are HACL*-based but + _init_old() is used instead of _init_builtin_hmac(). + """ + + +class PyDotNewOpenSSLRFCTestCase(ThroughModuleAPIMixin, + PyAssertersMixin, + OpenSSLHashFunctionsTrait, + RFCTestCaseMixin, + PurePythonInitHMAC, + unittest.TestCase): + """Python implementation of HMAC using hmac.new(). + + The underlying hash functions are OpenSSL-based but + _init_old() is used instead of _init_openssl_hmac(). + """ + + +class PyDotNewBuiltinRFCTestCase(ThroughModuleAPIMixin, + PyAssertersMixin, + BuiltinHashFunctionsTrait, + RFCTestCaseMixin, + PurePythonInitHMAC, + unittest.TestCase): + """Python implementation of HMAC using hmac.new(). + + The underlying hash functions are HACL-based but + _init_old() is used instead of _init_openssl_hmac(). + """ + + +class OpenSSLRFCTestCase(OpenSSLAssertersMixin, + OpenSSLHashFunctionsTrait, + RFCTestCaseMixin, + unittest.TestCase): + """OpenSSL implementation of HMAC. + + The underlying hash functions are also OpenSSL-based. + """ + + @unittest.expectedFailure # TODO: RUSTPYTHON; _hashlib.UnsupportedDigestmodError: unsupported hash type + def test_md5_rfc2202(self): + return super().test_md5_rfc2202() + + @unittest.expectedFailure # TODO: RUSTPYTHON; _hashlib.UnsupportedDigestmodError: unsupported hash type + def test_sha1_rfc2202(self): + return super().test_sha1_rfc2202() + + @unittest.expectedFailure # TODO: RUSTPYTHON; _hashlib.UnsupportedDigestmodError: unsupported hash type + def test_sha2_224_rfc4231(self): + return super().test_sha2_224_rfc4231() + + @unittest.expectedFailure # TODO: RUSTPYTHON; _hashlib.UnsupportedDigestmodError: unsupported hash type + def test_sha2_256_rfc4231(self): + return super().test_sha2_256_rfc4231() + + @unittest.expectedFailure # TODO: RUSTPYTHON; _hashlib.UnsupportedDigestmodError: unsupported hash type + def test_sha2_384_rfc4231(self): + return super().test_sha2_384_rfc4231() + + @unittest.expectedFailure # TODO: RUSTPYTHON; _hashlib.UnsupportedDigestmodError: unsupported hash type + def test_sha2_512_rfc4231(self): + return super().test_sha2_512_rfc4231() + + @unittest.expectedFailure # TODO: RUSTPYTHON; _hashlib.UnsupportedDigestmodError: unsupported hash type + def test_sha3_224_nist(self): + return super().test_sha3_224_nist() + + @unittest.expectedFailure # TODO: RUSTPYTHON; _hashlib.UnsupportedDigestmodError: unsupported hash type + def test_sha3_256_nist(self): + return super().test_sha3_256_nist() + + @unittest.expectedFailure # TODO: RUSTPYTHON; _hashlib.UnsupportedDigestmodError: unsupported hash type + def test_sha3_384_nist(self): + return super().test_sha3_384_nist() -class ConstructorTestCase(unittest.TestCase): + @unittest.expectedFailure # TODO: RUSTPYTHON; _hashlib.UnsupportedDigestmodError: unsupported hash type + def test_sha3_512_nist(self): + return super().test_sha3_512_nist() + + +class BuiltinRFCTestCase(BuiltinAssertersMixin, + NamedHashFunctionsTrait, + RFCTestCaseMixin, + unittest.TestCase): + """Built-in HACL* implementation of HMAC. - expected = ( - "6c845b47f52b3b47f6590c502db7825aad757bf4fadc8fa972f7cd2e76a5bdeb" - ) + The underlying hash functions are also HACL*-based. + """ - # TODO: RUSTPYTHON - @unittest.expectedFailure - @hashlib_helper.requires_hashdigest('sha256') - def test_normal(self): - # Standard constructor call. - try: - hmac.HMAC(b"key", digestmod='sha256') - except Exception: - self.fail("Standard constructor call raised exception.") + def assert_hmac_extra_cases( + self, key, msg, hexdigest, digestmod, hashname, digest_size, block_size + ): + # assert one-shot HMAC at the same time + with self.subTest(key=key, msg=msg, hashname=hashname): + func = getattr(self.hmac, f'compute_{hashname}') + self.assertTrue(callable(func)) + self.check_hmac_hexdigest(key, msg, hexdigest, digest_size, func) + + +class DigestModTestCaseMixin(CreatorMixin, DigestMixin): + """Tests for the 'digestmod' parameter for hmac_new() and hmac_digest().""" + + def assert_raises_missing_digestmod(self): + """A context manager catching errors when a digestmod is missing.""" + return self.assertRaisesRegex(TypeError, + "[M|m]issing.*required.*digestmod") + + def assert_raises_unknown_digestmod(self): + """A context manager catching errors when a digestmod is unknown.""" + return self.assertRaisesRegex(ValueError, "[Uu]nsupported.*") + + def test_constructor_missing_digestmod(self): + catcher = self.assert_raises_missing_digestmod + self.do_test_constructor_missing_digestmod(catcher) + + def test_constructor_unknown_digestmod(self): + catcher = self.assert_raises_unknown_digestmod + self.do_test_constructor_unknown_digestmod(catcher) - @hashlib_helper.requires_hashdigest('sha256') - def test_with_str_key(self): - # Pass a key of type str, which is an error, because it expects a key - # of type bytes - with self.assertRaises(TypeError): - h = hmac.HMAC("key", digestmod='sha256') + def do_test_constructor_missing_digestmod(self, catcher): + for func, args, kwds in self.cases_missing_digestmod_in_constructor(): + with self.subTest(args=args, kwds=kwds), catcher(): + func(*args, **kwds) - @hashlib_helper.requires_hashdigest('sha256') - def test_dot_new_with_str_key(self): - # Pass a key of type str, which is an error, because it expects a key - # of type bytes - with self.assertRaises(TypeError): - h = hmac.new("key", digestmod='sha256') + def do_test_constructor_unknown_digestmod(self, catcher): + for func, args, kwds in self.cases_unknown_digestmod_in_constructor(): + with self.subTest(args=args, kwds=kwds), catcher(): + func(*args, **kwds) - @hashlib_helper.requires_hashdigest('sha256') - def test_withtext(self): - # Constructor call with text. - try: - h = hmac.HMAC(b"key", b"hash this!", digestmod='sha256') - except Exception: - self.fail("Constructor call with text argument raised exception.") - self.assertEqual(h.hexdigest(), self.expected) + def cases_missing_digestmod_in_constructor(self): + raise NotImplementedError - @hashlib_helper.requires_hashdigest('sha256') - def test_with_bytearray(self): - try: - h = hmac.HMAC(bytearray(b"key"), bytearray(b"hash this!"), - digestmod="sha256") - except Exception: - self.fail("Constructor call with bytearray arguments raised exception.") - self.assertEqual(h.hexdigest(), self.expected) + def make_missing_digestmod_cases(self, func, missing_like=()): + """Generate cases for missing digestmod tests. - @hashlib_helper.requires_hashdigest('sha256') - def test_with_memoryview_msg(self): - try: - h = hmac.HMAC(b"key", memoryview(b"hash this!"), digestmod="sha256") - except Exception: - self.fail("Constructor call with memoryview msg raised exception.") - self.assertEqual(h.hexdigest(), self.expected) + Only the Python implementation should consider "falsey" 'digestmod' + values as being equivalent to a missing one. + """ + key, msg = b'unused key', b'unused msg' + choices = [DIGESTMOD_SENTINEL, *missing_like] + return self._invalid_digestmod_cases(func, key, msg, choices) - @hashlib_helper.requires_hashdigest('sha256') - def test_withmodule(self): - # Constructor call with text and digest module. - try: - h = hmac.HMAC(b"key", b"", hashlib.sha256) - except Exception: - self.fail("Constructor call with hashlib.sha256 raised exception.") + def cases_unknown_digestmod_in_constructor(self): + raise NotImplementedError + + def make_unknown_digestmod_cases(self, func, bad_digestmods): + """Generate cases for unknown digestmod tests.""" + key, msg = b'unused key', b'unused msg' + return self._invalid_digestmod_cases(func, key, msg, bad_digestmods) + + def _invalid_digestmod_cases(self, func, key, msg, choices): + cases = [] + for digestmod in choices: + kwargs = {'digestmod': digestmod} + cases.append((func, (key,), kwargs)) + cases.append((func, (key, msg), kwargs)) + cases.append((func, (key,), kwargs | {'msg': msg})) + return cases + + +class ConstructorTestCaseMixin(CreatorMixin, DigestMixin, ObjectCheckerMixin): + """HMAC constructor tests based on HMAC-SHA-2/256.""" + + key = b"key" + msg = b"hash this!" + res = "6c845b47f52b3b47f6590c502db7825aad757bf4fadc8fa972f7cd2e76a5bdeb" + + def do_test_constructor(self, hmac_on_key_and_msg): + self.do_test_constructor_invalid_types(hmac_on_key_and_msg) + self.do_test_constructor_supported_types(hmac_on_key_and_msg) + + def do_test_constructor_invalid_types(self, hmac_on_key_and_msg): + self.assertRaises(TypeError, hmac_on_key_and_msg, 1) + self.assertRaises(TypeError, hmac_on_key_and_msg, "key") + + self.assertRaises(TypeError, hmac_on_key_and_msg, b"key", 1) + self.assertRaises(TypeError, hmac_on_key_and_msg, b"key", "msg") + + def do_test_constructor_supported_types(self, hmac_on_key_and_msg): + for tp_key in [bytes, bytearray]: + for tp_msg in [bytes, bytearray, memoryview]: + with self.subTest(tp_key=tp_key, tp_msg=tp_msg): + h = hmac_on_key_and_msg(tp_key(self.key), tp_msg(self.msg)) + self.assertEqual(h.name, "hmac-sha256") + self.assertEqual(h.hexdigest(), self.res) + + @hashlib_helper.requires_hashdigest("sha256") + def test_constructor(self): + self.do_test_constructor(self.bind_hmac_new("sha256")) + + @hashlib_helper.requires_hashdigest("sha256") + def test_digest(self): + digest = self.hmac_digest(self.key, self.msg, "sha256") + self.assertEqual(digest, binascii.unhexlify(self.res)) + + +class PyConstructorBaseMixin(PyModuleMixin, + DigestModTestCaseMixin, + ConstructorTestCaseMixin): + + def cases_missing_digestmod_in_constructor(self): + func, choices = self.hmac_new, ['', None, False] + return self.make_missing_digestmod_cases(func, choices) + + def cases_unknown_digestmod_in_constructor(self): + func, choices = self.hmac_new, ['unknown'] + return self.make_unknown_digestmod_cases(func, choices) + + @requires_builtin_sha2() + def test_constructor_with_module(self): + self.do_test_constructor(self.bind_hmac_new(sha2.sha256)) + + @requires_builtin_sha2() + def test_digest_with_module(self): + digest = self.hmac_digest(self.key, self.msg, sha2.sha256) + self.assertEqual(digest, binascii.unhexlify(self.res)) + + +class PyConstructorTestCase(ThroughObjectMixin, PyConstructorBaseMixin, + unittest.TestCase): + """Test the hmac.HMAC() pure Python constructor.""" + + +class PyModuleConstructorTestCase(ThroughModuleAPIMixin, PyConstructorBaseMixin, + unittest.TestCase): + """Test the hmac.new() and hmac.digest() functions. + + Note that "self.hmac" is imported by blocking "_hashlib" and "_hmac". + For testing functions in "hmac", extend PyMiscellaneousTests instead. + """ + + def test_hmac_digest_digestmod_parameter(self): + func = self.hmac_digest + + def raiser(): + raise RuntimeError("custom exception") + + with self.assertRaisesRegex(RuntimeError, "custom exception"): + func(b'key', b'msg', raiser) + + with self.assertRaisesRegex(ValueError, 'hash type'): + func(b'key', b'msg', 'unknown') + + with self.assertRaisesRegex(AttributeError, 'new'): + func(b'key', b'msg', 1234) + with self.assertRaisesRegex(AttributeError, 'new'): + func(b'key', b'msg', None) + + +class ExtensionConstructorTestCaseMixin(DigestModTestCaseMixin, + ConstructorTestCaseMixin): + + @property + def obj_type(self): + """The underlying (non-instantiable) C class.""" + raise NotImplementedError + + @property + def exc_type(self): + """The exact exception class raised upon invalid 'digestmod' values.""" + raise NotImplementedError - # TODO: RUSTPYTHON - @unittest.expectedFailure - @unittest.skipUnless(C_HMAC is not None, 'need _hashlib') def test_internal_types(self): - # internal types like _hashlib.C_HMAC are not constructable - check_disallow_instantiation(self, C_HMAC) + # internal C types are immutable and cannot be instantiated + check_disallow_instantiation(self, self.obj_type) with self.assertRaisesRegex(TypeError, "immutable type"): - C_HMAC.value = None - - # TODO: RUSTPYTHON - @unittest.expectedFailure - @unittest.skipUnless(sha256_module is not None, 'need _sha256') - def test_with_sha256_module(self): - h = hmac.HMAC(b"key", b"hash this!", digestmod=sha256_module.sha256) - self.assertEqual(h.hexdigest(), self.expected) - self.assertEqual(h.name, "hmac-sha256") + self.obj_type.value = None - digest = hmac.digest(b"key", b"hash this!", sha256_module.sha256) - self.assertEqual(digest, binascii.unhexlify(self.expected)) + def assert_raises_unknown_digestmod(self): + self.assertIsSubclass(self.exc_type, ValueError) + return self.assertRaises(self.exc_type) + def cases_missing_digestmod_in_constructor(self): + return self.make_missing_digestmod_cases(self.hmac_new) -class SanityTestCase(unittest.TestCase): + def cases_unknown_digestmod_in_constructor(self): + func, choices = self.hmac_new, ['unknown', 1234] + return self.make_unknown_digestmod_cases(func, choices) - # TODO: RUSTPYTHON - @unittest.expectedFailure - @hashlib_helper.requires_hashdigest('sha256') - def test_exercise_all_methods(self): - # Exercising all methods once. - # This must not raise any exceptions - try: - h = hmac.HMAC(b"my secret key", digestmod="sha256") - h.update(b"compute the hash of this text!") - h.digest() - h.hexdigest() - h.copy() - except Exception: - self.fail("Exception raised during normal usage of HMAC class.") +class OpenSSLConstructorTestCase(ThroughOpenSSLAPIMixin, + ExtensionConstructorTestCaseMixin, + unittest.TestCase): -class UpdateTestCase(unittest.TestCase): - @hashlib_helper.requires_hashdigest('sha256') - def test_with_str_update(self): - with self.assertRaises(TypeError): - h = hmac.new(b"key", digestmod='sha256') - h.update("invalid update") + @property + def obj_type(self): + return _hashlib.HMAC + @property + def exc_type(self): + return _hashlib.UnsupportedDigestmodError -class CopyTestCase(unittest.TestCase): + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_hmac_digest_digestmod_parameter(self): + for value in [object, 'unknown', 1234, None]: + with ( + self.subTest(value=value), + self.assert_raises_unknown_digestmod(), + ): + self.hmac_digest(b'key', b'msg', value) - @hashlib_helper.requires_hashdigest('sha256') - def test_attributes_old(self): + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_constructor(self): + return super().test_constructor() + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_constructor_missing_digestmod(self): + return super().test_constructor_missing_digestmod() + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_constructor_unknown_digestmod(self): + return super().test_constructor_unknown_digestmod() + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module '_hashlib' has no attribute 'HMAC'. Did you mean: 'exc_type'? + def test_internal_types(self): + return super().test_internal_types() + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module '_hashlib' has no attribute 'hmac_digest' + def test_digest(self): + return super().test_digest() + + +class BuiltinConstructorTestCase(ThroughBuiltinAPIMixin, + ExtensionConstructorTestCaseMixin, + unittest.TestCase): + + @property + def obj_type(self): + return self.hmac.HMAC + + @property + def exc_type(self): + return self.hmac.UnknownHashError + + def test_hmac_digest_digestmod_parameter(self): + for value in [object, 'unknown', 1234, None]: + with ( + self.subTest(value=value), + self.assert_raises_unknown_digestmod(), + ): + self.hmac_digest(b'key', b'msg', value) + + +class SanityTestCaseMixin(CreatorMixin): + """Sanity checks for HMAC objects and their object interface. + + The tests here use a common digestname and do not check all supported + hash functions. + """ + + # The underlying HMAC class to test. May be in C or in Python. + hmac_class: type + # The underlying hash function name (should be accepted by the HMAC class). + digestname: str + # The expected digest and block sizes (must be hardcoded). + digest_size: int + block_size: int + + def test_methods(self): + h = self.hmac_new(b"my secret key", digestmod=self.digestname) + self.assertIsInstance(h, self.hmac_class) + self.assertIsNone(h.update(b"compute the hash of this text!")) + self.assertIsInstance(h.digest(), bytes) + self.assertIsInstance(h.hexdigest(), str) + self.assertIsInstance(h.copy(), self.hmac_class) + + def test_properties(self): + h = self.hmac_new(b"my secret key", digestmod=self.digestname) + self.assertEqual(h.name, f"hmac-{self.digestname}") + self.assertEqual(h.digest_size, self.digest_size) + self.assertEqual(h.block_size, self.block_size) + + def test_repr(self): + # HMAC object representation may differ across implementations + raise NotImplementedError + + +@hashlib_helper.requires_hashdigest('sha256') +class PySanityTestCase(ThroughObjectMixin, PyModuleMixin, SanityTestCaseMixin, + unittest.TestCase): + + @classmethod + def setUpClass(cls): + super().setUpClass() + cls.hmac_class = cls.hmac.HMAC + cls.digestname = 'sha256' + cls.digest_size = 32 + cls.block_size = 64 + + def test_repr(self): + h = self.hmac_new(b"my secret key", digestmod=self.digestname) + self.assertStartsWith(repr(h), "' + "" + '' + '' + '' + '\u2603' +) + +SAMPLE_RAWTEXT = SAMPLE_RCDATA + '&☺' + + class EventCollector(html.parser.HTMLParser): def __init__(self, *args, autocdata=False, **kw): @@ -97,12 +109,13 @@ def get_events(self): class TestCaseBase(unittest.TestCase): - def get_collector(self): - return EventCollector(convert_charrefs=False) + def get_collector(self, convert_charrefs=False): + return EventCollector(convert_charrefs=convert_charrefs) - def _run_check(self, source, expected_events, collector=None): + def _run_check(self, source, expected_events, + *, collector=None, convert_charrefs=False): if collector is None: - collector = self.get_collector() + collector = self.get_collector(convert_charrefs=convert_charrefs) parser = collector for s in source: parser.feed(s) @@ -116,7 +129,7 @@ def _run_check(self, source, expected_events, collector=None): def _run_check_extra(self, source, events): self._run_check(source, events, - EventCollectorExtra(convert_charrefs=False)) + collector=EventCollectorExtra(convert_charrefs=False)) class HTMLParserTestCase(TestCaseBase): @@ -175,10 +188,87 @@ def test_malformatted_charref(self): ]) def test_unclosed_entityref(self): - self._run_check("&entityref foo", [ - ("entityref", "entityref"), - ("data", " foo"), - ]) + self._run_check('> <', [('entityref', 'gt'), ('data', ' '), ('entityref', 'lt')], + convert_charrefs=False) + self._run_check('> <', [('data', '> <')], convert_charrefs=True) + + self._run_check('&undefined <', + [('entityref', 'undefined'), ('data', ' '), ('entityref', 'lt')], + convert_charrefs=False) + self._run_check('&undefined <', [('data', '&undefined <')], + convert_charrefs=True) + + self._run_check('>undefined <', + [('entityref', 'gtundefined'), ('data', ' '), ('entityref', 'lt')], + convert_charrefs=False) + self._run_check('>undefined <', [('data', '>undefined <')], + convert_charrefs=True) + + self._run_check('& <', [('data', '& '), ('entityref', 'lt')], + convert_charrefs=False) + self._run_check('& <', [('data', '& <')], convert_charrefs=True) + + def test_eof_in_entityref(self): + self._run_check('>', [('entityref', 'gt')], convert_charrefs=False) + self._run_check('>', [('data', '>')], convert_charrefs=True) + + self._run_check('&g', [('entityref', 'g')], convert_charrefs=False) + self._run_check('&g', [('data', '&g')], convert_charrefs=True) + + self._run_check('&undefined', [('entityref', 'undefined')], + convert_charrefs=False) + self._run_check('&undefined', [('data', '&undefined')], + convert_charrefs=True) + + self._run_check('>undefined', [('entityref', 'gtundefined')], + convert_charrefs=False) + self._run_check('>undefined', [('data', '>undefined')], + convert_charrefs=True) + + self._run_check('&', [('data', '&')], convert_charrefs=False) + self._run_check('&', [('data', '&')], convert_charrefs=True) + + def test_unclosed_charref(self): + self._run_check('{ <', [('charref', '123'), ('data', ' '), ('entityref', 'lt')], + convert_charrefs=False) + self._run_check('{ <', [('data', '{ <')], convert_charrefs=True) + self._run_check('« <', [('charref', 'xab'), ('data', ' '), ('entityref', 'lt')], + convert_charrefs=False) + self._run_check('« <', [('data', '\xab <')], convert_charrefs=True) + + self._run_check('� <', + [('charref', '123456789'), ('data', ' '), ('entityref', 'lt')], + convert_charrefs=False) + self._run_check('� <', [('data', '\ufffd <')], + convert_charrefs=True) + self._run_check('� <', + [('charref', 'x123456789'), ('data', ' '), ('entityref', 'lt')], + convert_charrefs=False) + self._run_check('� <', [('data', '\ufffd <')], + convert_charrefs=True) + + self._run_check('&# <', [('data', '&# '), ('entityref', 'lt')], convert_charrefs=False) + self._run_check('&# <', [('data', '&# <')], convert_charrefs=True) + self._run_check('&#x <', [('data', '&#x '), ('entityref', 'lt')], convert_charrefs=False) + self._run_check('&#x <', [('data', '&#x <')], convert_charrefs=True) + + def test_eof_in_charref(self): + self._run_check('{', [('charref', '123')], convert_charrefs=False) + self._run_check('{', [('data', '{')], convert_charrefs=True) + self._run_check('«', [('charref', 'xab')], convert_charrefs=False) + self._run_check('«', [('data', '\xab')], convert_charrefs=True) + + self._run_check('�', [('charref', '123456789')], + convert_charrefs=False) + self._run_check('�', [('data', '\ufffd')], convert_charrefs=True) + self._run_check('�', [('charref', 'x123456789')], + convert_charrefs=False) + self._run_check('�', [('data', '\ufffd')], convert_charrefs=True) + + self._run_check('&#', [('data', '&#')], convert_charrefs=False) + self._run_check('&#', [('data', '&#')], convert_charrefs=True) + self._run_check('&#x', [('data', '&#x')], convert_charrefs=False) + self._run_check('&#x', [('data', '&#x')], convert_charrefs=True) def test_bad_nesting(self): # Strangely, this *is* supposed to test that overlapping @@ -293,30 +383,20 @@ def test_get_starttag_text(self): 'Date().getTime()+\'"><\\/s\'+\'cript>\');\n//]]>'), '\n\n', '', - 'foo = ""', - 'foo = ""', - 'foo = ""', - 'foo = ""', - 'foo = ""', - 'foo = ""', ]) def test_script_content(self, content): s = f'' - self._run_check(s, [("starttag", "script", []), - ("data", content), - ("endtag", "script")]) + self._run_check(s, [ + ("starttag", "script", []), + ("data", content), + ("endtag", "script"), + ]) @support.subTests('content', [ 'a::before { content: ""; }', 'a::before { content: "¬-an-entity-ref;"; }', 'a::before { content: ""; }', 'a::before { content: "\u2603"; }', - 'a::before { content: "< /style>"; }', - 'a::before { content: ""; }', - 'a::before { content: ""; }', - 'a::before { content: ""; }', - 'a::before { content: ""; }', - 'a::before { content: ""; }', ]) def test_style_content(self, content): s = f'' @@ -324,47 +404,59 @@ def test_style_content(self, content): ("data", content), ("endtag", "style")]) - @support.subTests('content', [ - '', - "", - '', - '', - '', - '\u2603', - '< /title>', - '', - '', - '', - '', - '', + @support.subTests('tag', ['title', 'textarea']) + def test_rcdata_content(self, tag): + source = f"<{tag}>{SAMPLE_RCDATA}" + self._run_check(source, [ + ("starttag", tag, []), + ("data", SAMPLE_RCDATA), + ("endtag", tag), ]) - def test_title_content(self, content): - source = f"{content}" + source = f"<{tag}>&" self._run_check(source, [ - ("starttag", "title", []), - ("data", content), - ("endtag", "title"), + ("starttag", tag, []), + ('entityref', 'amp'), + ("endtag", tag), ]) - @support.subTests('content', [ - '', - "", - '', - '', - '', - '\u2603', - '< /textarea>', - '', - '', - '', - '', + @support.subTests('tag', + ['style', 'xmp', 'iframe', 'noembed', 'noframes', 'script']) + def test_rawtext_content(self, tag): + source = f"<{tag}>{SAMPLE_RAWTEXT}" + self._run_check(source, [ + ("starttag", tag, []), + ("data", SAMPLE_RAWTEXT), + ("endtag", tag), + ]) + + def test_noscript_content(self): + source = f"" + # scripting=False -- normal mode + self._run_check(source, [ + ('starttag', 'noscript', []), + ('comment', ' not a comment '), + ('starttag', 'not', [('a', 'start tag')]), + ('unknown decl', 'CDATA[not a cdata'), + ('comment', 'not a bogus comment'), + ('endtag', 'not'), + ('data', '☃'), + ('entityref', 'amp'), + ('charref', '9786'), + ('endtag', 'noscript'), ]) - def test_textarea_content(self, content): - source = f"" + # scripting=True -- RAWTEXT mode self._run_check(source, [ - ("starttag", "textarea", []), + ("starttag", "noscript", []), + ("data", SAMPLE_RAWTEXT), + ("endtag", "noscript"), + ], collector=EventCollector(scripting=True)) + + def test_plaintext_content(self): + content = SAMPLE_RAWTEXT + '' # not closing + source = f"{content}" + self._run_check(source, [ + ("starttag", "plaintext", []), ("data", content), - ("endtag", "textarea"), ]) @support.subTests('endtag', ['script', 'SCRIPT', 'script ', 'script\n', @@ -381,52 +473,65 @@ def test_script_closing_tag(self, endtag): ("endtag", "script")], collector=EventCollectorNoNormalize(convert_charrefs=False)) - @support.subTests('endtag', ['style', 'STYLE', 'style ', 'style\n', - 'style/', 'style foo=bar', 'style foo=">"']) - def test_style_closing_tag(self, endtag): - content = """ - b::before { content: "<!-- not a comment -->"; } - p::before { content: "&not-an-entity-ref;"; } - a::before { content: "<i>"; } - a::after { content: "</i>"; } - """ - s = f'<StyLE>{content}</{endtag}>' - self._run_check(s, [("starttag", "style", []), - ("data", content), - ("endtag", "style")], - collector=EventCollectorNoNormalize(convert_charrefs=False)) - - @support.subTests('endtag', ['title', 'TITLE', 'title ', 'title\n', - 'title/', 'title foo=bar', 'title foo=">"']) - def test_title_closing_tag(self, endtag): - content = "<!-- not a comment --><i>Egg &amp; Spam</i>" - s = f'<TitLe>{content}</{endtag}>' - self._run_check(s, [("starttag", "title", []), - ('data', '<!-- not a comment --><i>Egg & Spam</i>'), - ("endtag", "title")], - collector=EventCollectorNoNormalize(convert_charrefs=True)) - self._run_check(s, [("starttag", "title", []), - ('data', '<!-- not a comment --><i>Egg '), - ('entityref', 'amp'), - ('data', ' Spam</i>'), - ("endtag", "title")], - collector=EventCollectorNoNormalize(convert_charrefs=False)) - - @support.subTests('endtag', ['textarea', 'TEXTAREA', 'textarea ', 'textarea\n', - 'textarea/', 'textarea foo=bar', 'textarea foo=">"']) - def test_textarea_closing_tag(self, endtag): - content = "<!-- not a comment --><i>Egg &amp; Spam</i>" - s = f'<TexTarEa>{content}</{endtag}>' - self._run_check(s, [("starttag", "textarea", []), - ('data', '<!-- not a comment --><i>Egg & Spam</i>'), - ("endtag", "textarea")], - collector=EventCollectorNoNormalize(convert_charrefs=True)) - self._run_check(s, [("starttag", "textarea", []), - ('data', '<!-- not a comment --><i>Egg '), - ('entityref', 'amp'), - ('data', ' Spam</i>'), - ("endtag", "textarea")], - collector=EventCollectorNoNormalize(convert_charrefs=False)) + @support.subTests('tag', [ + 'script', 'style', 'xmp', 'iframe', 'noembed', 'noframes', + 'textarea', 'title', 'noscript', + ]) + def test_closing_tag(self, tag): + for endtag in [tag, tag.upper(), f'{tag} ', f'{tag}\n', + f'{tag}/', f'{tag} foo=bar', f'{tag} foo=">"']: + content = "<!-- not a comment --><i>Spam</i>" + s = f'<{tag.upper()}>{content}</{endtag}>' + self._run_check(s, [ + ("starttag", tag, []), + ('data', content), + ("endtag", tag), + ], collector=EventCollectorNoNormalize(convert_charrefs=False, scripting=True)) + + @support.subTests('tag', [ + 'script', 'style', 'xmp', 'iframe', 'noembed', 'noframes', + 'textarea', 'title', 'noscript', + ]) + def test_invalid_closing_tag(self, tag): + content = ( + f'< /{tag}>' + f'</ {tag}>' + f'</{tag}x>' + f'</{tag}\v>' + f'</{tag}\xa0>' + ) + source = f"<{tag}>{content}</{tag}>" + self._run_check(source, [ + ("starttag", tag, []), + ("data", content), + ("endtag", tag), + ], collector=EventCollector(convert_charrefs=False, scripting=True)) + + @support.subTests('tag,endtag', [ + ('title', 'tıtle'), + ('style', 'ſtyle'), + ('style', 'ſtyle'), + ('style', 'style'), + ('iframe', 'ıframe'), + ('noframes', 'noframeſ'), + ('noscript', 'noſcript'), + ('noscript', 'noscrıpt'), + ('script', 'ſcript'), + ('script', 'scrıpt'), + ]) + def test_invalid_nonascii_closing_tag(self, tag, endtag): + content = f"<br></{endtag}>" + source = f"<{tag}>{content}" + self._run_check(source, [ + ("starttag", tag, []), + ("data", content), + ], collector=EventCollector(convert_charrefs=False, scripting=True)) + source = f"<{tag}>{content}</{tag}>" + self._run_check(source, [ + ("starttag", tag, []), + ("data", content), + ("endtag", tag), + ], collector=EventCollector(convert_charrefs=False, scripting=True)) @support.subTests('tail,end', [ ('', False), @@ -735,20 +840,6 @@ def test_correct_detection_of_start_tags(self): ] self._run_check(html, expected) - def test_EOF_in_charref(self): - # see #17802 - # This test checks that the UnboundLocalError reported in the issue - # is not raised, however I'm not sure the returned values are correct. - # Maybe HTMLParser should use self.unescape for these - data = [ - ('a&', [('data', 'a&')]), - ('a&b', [('data', 'ab')]), - ('a&b ', [('data', 'a'), ('entityref', 'b'), ('data', ' ')]), - ('a&b;', [('data', 'a'), ('entityref', 'b')]), - ] - for html, expected in data: - self._run_check(html, expected) - def test_eof_in_comments(self): data = [ ('<!--', [('comment', '')]), diff --git a/Lib/test/test_http_cookiejar.py b/Lib/test/test_http_cookiejar.py index f4b9dc6a282..51fa4a3d413 100644 --- a/Lib/test/test_http_cookiejar.py +++ b/Lib/test/test_http_cookiejar.py @@ -1,14 +1,16 @@ """Tests for http/cookiejar.py.""" import os +import stat +import sys import re -import test.support +from test import support from test.support import os_helper from test.support import warnings_helper +from test.support.testcase import ExtraAssertions import time import unittest import urllib.request -import pathlib from http.cookiejar import (time2isoz, http2time, iso2time, time2netscape, parse_ns_headers, join_header_words, split_header_words, Cookie, @@ -17,6 +19,7 @@ reach, is_HDN, domain_match, user_domain_match, request_path, request_port, request_host) +mswindows = (sys.platform == "win32") class DateTimeTests(unittest.TestCase): @@ -104,8 +107,7 @@ def test_http2time_formats(self): self.assertEqual(http2time(s.lower()), test_t, s.lower()) self.assertEqual(http2time(s.upper()), test_t, s.upper()) - def test_http2time_garbage(self): - for test in [ + @support.subTests('test', [ '', 'Garbage', 'Mandag 16. September 1996', @@ -120,12 +122,10 @@ def test_http2time_garbage(self): '08-01-3697739', '09 Feb 19942632 22:23:32 GMT', 'Wed, 09 Feb 1994834 22:23:32 GMT', - ]: - self.assertIsNone(http2time(test), - "http2time(%s) is not None\n" - "http2time(test) %s" % (test, http2time(test))) + ]) + def test_http2time_garbage(self, test): + self.assertIsNone(http2time(test)) - @unittest.skip("TODO: RUSTPYTHON, regressed to cubic complexity") def test_http2time_redos_regression_actually_completes(self): # LOOSE_HTTP_DATE_RE was vulnerable to malicious input which caused catastrophic backtracking (REDoS). # If we regress to cubic complexity, this test will take a very long time to succeed. @@ -149,9 +149,7 @@ def parse_date(text): self.assertEqual(parse_date("1994-02-03 19:45:29 +0530"), (1994, 2, 3, 14, 15, 29)) - def test_iso2time_formats(self): - # test iso2time for supported dates. - tests = [ + @support.subTests('s', [ '1994-02-03 00:00:00 -0000', # ISO 8601 format '1994-02-03 00:00:00 +0000', # ISO 8601 format '1994-02-03 00:00:00', # zone is optional @@ -164,16 +162,15 @@ def test_iso2time_formats(self): # A few tests with extra space at various places ' 1994-02-03 ', ' 1994-02-03T00:00:00 ', - ] - + ]) + def test_iso2time_formats(self, s): + # test iso2time for supported dates. test_t = 760233600 # assume broken POSIX counting of seconds - for s in tests: - self.assertEqual(iso2time(s), test_t, s) - self.assertEqual(iso2time(s.lower()), test_t, s.lower()) - self.assertEqual(iso2time(s.upper()), test_t, s.upper()) + self.assertEqual(iso2time(s), test_t, s) + self.assertEqual(iso2time(s.lower()), test_t, s.lower()) + self.assertEqual(iso2time(s.upper()), test_t, s.upper()) - def test_iso2time_garbage(self): - for test in [ + @support.subTests('test', [ '', 'Garbage', 'Thursday, 03-Feb-94 00:00:00 GMT', @@ -186,11 +183,10 @@ def test_iso2time_garbage(self): '01-01-1980 00:00:62', '01-01-1980T00:00:62', '19800101T250000Z', - ]: - self.assertIsNone(iso2time(test), - "iso2time(%r)" % test) + ]) + def test_iso2time_garbage(self, test): + self.assertIsNone(iso2time(test)) - @unittest.skip("TODO, RUSTPYTHON, regressed to quadratic complexity") def test_iso2time_performance_regression(self): # If ISO_DATE_RE regresses to quadratic complexity, this test will take a very long time to succeed. # If fixed, it should complete within a fraction of a second. @@ -200,24 +196,23 @@ def test_iso2time_performance_regression(self): class HeaderTests(unittest.TestCase): - def test_parse_ns_headers(self): - # quotes should be stripped - expected = [[('foo', 'bar'), ('expires', 2209069412), ('version', '0')]] - for hdr in [ + @support.subTests('hdr', [ 'foo=bar; expires=01 Jan 2040 22:23:32 GMT', 'foo=bar; expires="01 Jan 2040 22:23:32 GMT"', - ]: - self.assertEqual(parse_ns_headers([hdr]), expected) - - def test_parse_ns_headers_version(self): - + ]) + def test_parse_ns_headers(self, hdr): # quotes should be stripped - expected = [[('foo', 'bar'), ('version', '1')]] - for hdr in [ + expected = [[('foo', 'bar'), ('expires', 2209069412), ('version', '0')]] + self.assertEqual(parse_ns_headers([hdr]), expected) + + @support.subTests('hdr', [ 'foo=bar; version="1"', 'foo=bar; Version="1"', - ]: - self.assertEqual(parse_ns_headers([hdr]), expected) + ]) + def test_parse_ns_headers_version(self, hdr): + # quotes should be stripped + expected = [[('foo', 'bar'), ('version', '1')]] + self.assertEqual(parse_ns_headers([hdr]), expected) def test_parse_ns_headers_special_names(self): # names such as 'expires' are not special in first name=value pair @@ -233,8 +228,7 @@ def test_join_header_words(self): self.assertEqual(join_header_words([[]]), "") - def test_split_header_words(self): - tests = [ + @support.subTests('arg,expect', [ ("foo", [[("foo", None)]]), ("foo=bar", [[("foo", "bar")]]), (" foo ", [[("foo", None)]]), @@ -251,24 +245,22 @@ def test_split_header_words(self): (r'foo; bar=baz, spam=, foo="\,\;\"", bar= ', [[("foo", None), ("bar", "baz")], [("spam", "")], [("foo", ',;"')], [("bar", "")]]), - ] - - for arg, expect in tests: - try: - result = split_header_words([arg]) - except: - import traceback, io - f = io.StringIO() - traceback.print_exc(None, f) - result = "(error -- traceback follows)\n\n%s" % f.getvalue() - self.assertEqual(result, expect, """ + ]) + def test_split_header_words(self, arg, expect): + try: + result = split_header_words([arg]) + except: + import traceback, io + f = io.StringIO() + traceback.print_exc(None, f) + result = "(error -- traceback follows)\n\n%s" % f.getvalue() + self.assertEqual(result, expect, """ When parsing: '%s' Expected: '%s' Got: '%s' """ % (arg, expect, result)) - def test_roundtrip(self): - tests = [ + @support.subTests('arg,expect', [ ("foo", "foo"), ("foo=bar", "foo=bar"), (" foo ", "foo"), @@ -277,23 +269,35 @@ def test_roundtrip(self): ("foo=bar;bar=baz", "foo=bar; bar=baz"), ('foo bar baz', "foo; bar; baz"), (r'foo="\"" bar="\\"', r'foo="\""; bar="\\"'), + ("föo=bär", 'föo="bär"'), ('foo,,,bar', 'foo, bar'), ('foo=bar,bar=baz', 'foo=bar, bar=baz'), + ("foo=\n", 'foo=""'), + ('foo="\n"', 'foo="\n"'), + ('foo=bar\n', 'foo=bar'), + ('foo="bar\n"', 'foo="bar\n"'), + ('foo=bar\nbaz', 'foo=bar; baz'), + ('foo="bar\nbaz"', 'foo="bar\nbaz"'), ('text/html; charset=iso-8859-1', - 'text/html; charset="iso-8859-1"'), + 'text/html; charset=iso-8859-1'), + + ('text/html; charset="iso-8859/1"', + 'text/html; charset="iso-8859/1"'), ('foo="bar"; port="80,81"; discard, bar=baz', 'foo=bar; port="80,81"; discard, bar=baz'), (r'Basic realm="\"foo\\\\bar\""', - r'Basic; realm="\"foo\\\\bar\""') - ] - - for arg, expect in tests: - input = split_header_words([arg]) - res = join_header_words(input) - self.assertEqual(res, expect, """ + r'Basic; realm="\"foo\\\\bar\""'), + + ('n; foo="foo;_", bar="foo,_"', + 'n; foo="foo;_", bar="foo,_"'), + ]) + def test_roundtrip(self, arg, expect): + input = split_header_words([arg]) + res = join_header_words(input) + self.assertEqual(res, expect, """ When parsing: '%s' Expected: '%s' Got: '%s' @@ -337,9 +341,9 @@ def test_constructor_with_str(self): self.assertEqual(c.filename, filename) def test_constructor_with_path_like(self): - filename = pathlib.Path(os_helper.TESTFN) - c = LWPCookieJar(filename) - self.assertEqual(c.filename, os.fspath(filename)) + filename = os_helper.TESTFN + c = LWPCookieJar(os_helper.FakePath(filename)) + self.assertEqual(c.filename, filename) def test_constructor_with_none(self): c = LWPCookieJar(None) @@ -366,10 +370,63 @@ def test_lwp_valueless_cookie(self): c = LWPCookieJar() c.load(filename, ignore_discard=True) finally: - try: os.unlink(filename) - except OSError: pass + os_helper.unlink(filename) self.assertEqual(c._cookies["www.acme.com"]["/"]["boo"].value, None) + @unittest.skipIf(mswindows, "windows file permissions are incompatible with file modes") + @os_helper.skip_unless_working_chmod + def test_lwp_filepermissions(self): + # Cookie file should only be readable by the creator + filename = os_helper.TESTFN + c = LWPCookieJar() + interact_netscape(c, "http://www.acme.com/", 'boo') + try: + c.save(filename, ignore_discard=True) + st = os.stat(filename) + self.assertEqual(stat.S_IMODE(st.st_mode), 0o600) + finally: + os_helper.unlink(filename) + + @unittest.skipIf(mswindows, "windows file permissions are incompatible with file modes") + @os_helper.skip_unless_working_chmod + def test_mozilla_filepermissions(self): + # Cookie file should only be readable by the creator + filename = os_helper.TESTFN + c = MozillaCookieJar() + interact_netscape(c, "http://www.acme.com/", 'boo') + try: + c.save(filename, ignore_discard=True) + st = os.stat(filename) + self.assertEqual(stat.S_IMODE(st.st_mode), 0o600) + finally: + os_helper.unlink(filename) + + @unittest.skipIf(mswindows, "windows file permissions are incompatible with file modes") + @os_helper.skip_unless_working_chmod + def test_cookie_files_are_truncated(self): + filename = os_helper.TESTFN + for cookiejar_class in (LWPCookieJar, MozillaCookieJar): + c = cookiejar_class(filename) + + req = urllib.request.Request("http://www.acme.com/") + headers = ["Set-Cookie: pll_lang=en; Max-Age=31536000; path=/"] + res = FakeResponse(headers, "http://www.acme.com/") + c.extract_cookies(res, req) + self.assertEqual(len(c), 1) + + try: + # Save the first version with contents: + c.save() + # Now, clear cookies and re-save: + c.clear() + c.save() + # Check that file was truncated: + c.load() + finally: + os_helper.unlink(filename) + + self.assertEqual(len(c), 0) + def test_bad_magic(self): # OSErrors (eg. file doesn't exist) are allowed to propagate filename = os_helper.TESTFN @@ -393,8 +450,7 @@ def test_bad_magic(self): c = cookiejar_class() self.assertRaises(LoadError, c.load, filename) finally: - try: os.unlink(filename) - except OSError: pass + os_helper.unlink(filename) class CookieTests(unittest.TestCase): # XXX @@ -443,14 +499,7 @@ class CookieTests(unittest.TestCase): ## just the 7 special TLD's listed in their spec. And folks rely on ## that... - def test_domain_return_ok(self): - # test optimization: .domain_return_ok() should filter out most - # domains in the CookieJar before we try to access them (because that - # may require disk access -- in particular, with MSIECookieJar) - # This is only a rough check for performance reasons, so it's not too - # critical as long as it's sufficiently liberal. - pol = DefaultCookiePolicy() - for url, domain, ok in [ + @support.subTests('url,domain,ok', [ ("http://foo.bar.com/", "blah.com", False), ("http://foo.bar.com/", "rhubarb.blah.com", False), ("http://foo.bar.com/", "rhubarb.foo.bar.com", False), @@ -470,11 +519,18 @@ def test_domain_return_ok(self): ("http://foo/", ".local", True), ("http://barfoo.com", ".foo.com", False), ("http://barfoo.com", "foo.com", False), - ]: - request = urllib.request.Request(url) - r = pol.domain_return_ok(domain, request) - if ok: self.assertTrue(r) - else: self.assertFalse(r) + ]) + def test_domain_return_ok(self, url, domain, ok): + # test optimization: .domain_return_ok() should filter out most + # domains in the CookieJar before we try to access them (because that + # may require disk access -- in particular, with MSIECookieJar) + # This is only a rough check for performance reasons, so it's not too + # critical as long as it's sufficiently liberal. + pol = DefaultCookiePolicy() + request = urllib.request.Request(url) + r = pol.domain_return_ok(domain, request) + if ok: self.assertTrue(r) + else: self.assertFalse(r) def test_missing_value(self): # missing = sign in Cookie: header is regarded by Mozilla as a missing @@ -490,7 +546,7 @@ def test_missing_value(self): self.assertIsNone(cookie.value) self.assertEqual(cookie.name, '"spam"') self.assertEqual(lwp_cookie_str(cookie), ( - r'"spam"; path="/foo/"; domain="www.acme.com"; ' + r'"spam"; path="/foo/"; domain=www.acme.com; ' 'path_spec; discard; version=0')) old_str = repr(c) c.save(ignore_expires=True, ignore_discard=True) @@ -498,7 +554,7 @@ def test_missing_value(self): c = MozillaCookieJar(filename) c.revert(ignore_expires=True, ignore_discard=True) finally: - os.unlink(c.filename) + os_helper.unlink(c.filename) # cookies unchanged apart from lost info re. whether path was specified self.assertEqual( repr(c), @@ -508,10 +564,7 @@ def test_missing_value(self): self.assertEqual(interact_netscape(c, "http://www.acme.com/foo/"), '"spam"; eggs') - def test_rfc2109_handling(self): - # RFC 2109 cookies are handled as RFC 2965 or Netscape cookies, - # dependent on policy settings - for rfc2109_as_netscape, rfc2965, version in [ + @support.subTests('rfc2109_as_netscape,rfc2965,version', [ # default according to rfc2965 if not explicitly specified (None, False, 0), (None, True, 1), @@ -520,24 +573,27 @@ def test_rfc2109_handling(self): (False, True, 1), (True, False, 0), (True, True, 0), - ]: - policy = DefaultCookiePolicy( - rfc2109_as_netscape=rfc2109_as_netscape, - rfc2965=rfc2965) - c = CookieJar(policy) - interact_netscape(c, "http://www.example.com/", "ni=ni; Version=1") - try: - cookie = c._cookies["www.example.com"]["/"]["ni"] - except KeyError: - self.assertIsNone(version) # didn't expect a stored cookie - else: - self.assertEqual(cookie.version, version) - # 2965 cookies are unaffected - interact_2965(c, "http://www.example.com/", - "foo=bar; Version=1") - if rfc2965: - cookie2965 = c._cookies["www.example.com"]["/"]["foo"] - self.assertEqual(cookie2965.version, 1) + ]) + def test_rfc2109_handling(self, rfc2109_as_netscape, rfc2965, version): + # RFC 2109 cookies are handled as RFC 2965 or Netscape cookies, + # dependent on policy settings + policy = DefaultCookiePolicy( + rfc2109_as_netscape=rfc2109_as_netscape, + rfc2965=rfc2965) + c = CookieJar(policy) + interact_netscape(c, "http://www.example.com/", "ni=ni; Version=1") + try: + cookie = c._cookies["www.example.com"]["/"]["ni"] + except KeyError: + self.assertIsNone(version) # didn't expect a stored cookie + else: + self.assertEqual(cookie.version, version) + # 2965 cookies are unaffected + interact_2965(c, "http://www.example.com/", + "foo=bar; Version=1") + if rfc2965: + cookie2965 = c._cookies["www.example.com"]["/"]["foo"] + self.assertEqual(cookie2965.version, 1) def test_ns_parser(self): c = CookieJar() @@ -598,8 +654,6 @@ def test_ns_parser_special_names(self): self.assertIn('expires', cookies) self.assertIn('version', cookies) - # TODO: RUSTPYTHON; need to update http library to remove warnings - @unittest.expectedFailure def test_expires(self): # if expires is in future, keep cookie... c = CookieJar() @@ -707,8 +761,7 @@ def test_default_path_with_query(self): # Cookie is sent back to the same URI. self.assertEqual(interact_netscape(cj, uri), value) - def test_escape_path(self): - cases = [ + @support.subTests('arg,result', [ # quoted safe ("/foo%2f/bar", "/foo%2F/bar"), ("/foo%2F/bar", "/foo%2F/bar"), @@ -728,9 +781,9 @@ def test_escape_path(self): ("/foo/bar\u00fc", "/foo/bar%C3%BC"), # UTF-8 encoded # unicode ("/foo/bar\uabcd", "/foo/bar%EA%AF%8D"), # UTF-8 encoded - ] - for arg, result in cases: - self.assertEqual(escape_path(arg), result) + ]) + def test_escape_path(self, arg, result): + self.assertEqual(escape_path(arg), result) def test_request_path(self): # with parameters @@ -924,6 +977,48 @@ def test_two_component_domain_ns(self): ## self.assertEqual(len(c), 2) self.assertEqual(len(c), 4) + def test_localhost_domain(self): + c = CookieJar() + + interact_netscape(c, "http://localhost", "foo=bar; domain=localhost;") + + self.assertEqual(len(c), 1) + + def test_localhost_domain_contents(self): + c = CookieJar() + + interact_netscape(c, "http://localhost", "foo=bar; domain=localhost;") + + self.assertEqual(c._cookies[".localhost"]["/"]["foo"].value, "bar") + + def test_localhost_domain_contents_2(self): + c = CookieJar() + + interact_netscape(c, "http://localhost", "foo=bar;") + + self.assertEqual(c._cookies["localhost.local"]["/"]["foo"].value, "bar") + + def test_evil_nonlocal_domain(self): + c = CookieJar() + + interact_netscape(c, "http://evil.com", "foo=bar; domain=.localhost") + + self.assertEqual(len(c), 0) + + def test_evil_local_domain(self): + c = CookieJar() + + interact_netscape(c, "http://localhost", "foo=bar; domain=.evil.com") + + self.assertEqual(len(c), 0) + + def test_evil_local_domain_2(self): + c = CookieJar() + + interact_netscape(c, "http://localhost", "foo=bar; domain=.someother.local") + + self.assertEqual(len(c), 0) + def test_two_component_domain_rfc2965(self): pol = DefaultCookiePolicy(rfc2965=True) c = CookieJar(pol) @@ -1255,11 +1350,11 @@ def test_Cookie_iterator(self): r'port="90,100, 80,8080"; ' r'max-age=100; Comment = "Just kidding! (\"|\\\\) "') - versions = [1, 1, 1, 0, 1] - names = ["bang", "foo", "foo", "spam", "foo"] - domains = [".sol.no", "blah.spam.org", "www.acme.com", - "www.acme.com", "www.acme.com"] - paths = ["/", "/", "/", "/blah", "/blah/"] + versions = [1, 0, 1, 1, 1] + names = ["foo", "spam", "foo", "foo", "bang"] + domains = ["blah.spam.org", "www.acme.com", "www.acme.com", + "www.acme.com", ".sol.no"] + paths = ["/", "/blah", "/blah/", "/", "/"] for i in range(4): i = 0 @@ -1332,7 +1427,7 @@ def cookiejar_from_cookie_headers(headers): self.assertIsNone(cookie.expires) -class LWPCookieTests(unittest.TestCase): +class LWPCookieTests(unittest.TestCase, ExtraAssertions): # Tests taken from libwww-perl, with a few modifications and additions. def test_netscape_example_1(self): @@ -1424,7 +1519,7 @@ def test_netscape_example_1(self): h = req.get_header("Cookie") self.assertIn("PART_NUMBER=ROCKET_LAUNCHER_0001", h) self.assertIn("CUSTOMER=WILE_E_COYOTE", h) - self.assertTrue(h.startswith("SHIPPING=FEDEX;")) + self.assertStartsWith(h, "SHIPPING=FEDEX;") def test_netscape_example_2(self): # Second Example transaction sequence: @@ -1728,8 +1823,7 @@ def test_rejection(self): c = LWPCookieJar(policy=pol) c.load(filename, ignore_discard=True) finally: - try: os.unlink(filename) - except OSError: pass + os_helper.unlink(filename) self.assertEqual(old, repr(c)) @@ -1788,8 +1882,7 @@ def save_and_restore(cj, ignore_discard): DefaultCookiePolicy(rfc2965=True)) new_c.load(ignore_discard=ignore_discard) finally: - try: os.unlink(filename) - except OSError: pass + os_helper.unlink(filename) return new_c new_c = save_and_restore(c, True) diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py index d4a6eefe322..5267d2fe011 100644 --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -1,4 +1,4 @@ -import sys +import enum import errno from http import client, HTTPStatus import io @@ -8,7 +8,6 @@ import re import socket import threading -import warnings import unittest from unittest import mock @@ -17,16 +16,19 @@ from test import support from test.support import os_helper from test.support import socket_helper -from test.support import warnings_helper +from test.support.testcase import ExtraAssertions +support.requires_working_socket(module=True) here = os.path.dirname(__file__) # Self-signed cert file for 'localhost' -CERT_localhost = os.path.join(here, 'certdata/keycert.pem') +CERT_localhost = os.path.join(here, 'certdata', 'keycert.pem') # Self-signed cert file for 'fakehostname' -CERT_fakehostname = os.path.join(here, 'certdata/keycert2.pem') +CERT_fakehostname = os.path.join(here, 'certdata', 'keycert2.pem') # Self-signed cert file for self-signed.pythontest.net -CERT_selfsigned_pythontestdotnet = os.path.join(here, 'certdata/selfsigned_pythontestdotnet.pem') +CERT_selfsigned_pythontestdotnet = os.path.join( + here, 'certdata', 'selfsigned_pythontestdotnet.pem', +) # constants for testing chunked encoding chunked_start = ( @@ -133,7 +135,7 @@ def connect(self): def create_connection(self, *pos, **kw): return FakeSocket(*self.fake_socket_args) -class HeaderTests(TestCase): +class HeaderTests(TestCase, ExtraAssertions): def test_auto_headers(self): # Some headers are added automatically, but should not be added by # .request() if they are explicitly set. @@ -272,7 +274,7 @@ def test_ipv6host_header(self): sock = FakeSocket('') conn.sock = sock conn.request('GET', '/foo') - self.assertTrue(sock.data.startswith(expected)) + self.assertStartsWith(sock.data, expected) expected = b'GET /foo HTTP/1.1\r\nHost: [2001:102A::]\r\n' \ b'Accept-Encoding: identity\r\n\r\n' @@ -280,7 +282,23 @@ def test_ipv6host_header(self): sock = FakeSocket('') conn.sock = sock conn.request('GET', '/foo') - self.assertTrue(sock.data.startswith(expected)) + self.assertStartsWith(sock.data, expected) + + expected = b'GET /foo HTTP/1.1\r\nHost: [fe80::]\r\n' \ + b'Accept-Encoding: identity\r\n\r\n' + conn = client.HTTPConnection('[fe80::%2]') + sock = FakeSocket('') + conn.sock = sock + conn.request('GET', '/foo') + self.assertStartsWith(sock.data, expected) + + expected = b'GET /foo HTTP/1.1\r\nHost: [fe80::]:81\r\n' \ + b'Accept-Encoding: identity\r\n\r\n' + conn = client.HTTPConnection('[fe80::%2]:81') + sock = FakeSocket('') + conn.sock = sock + conn.request('GET', '/foo') + self.assertStartsWith(sock.data, expected) def test_malformed_headers_coped_with(self): # Issue 19996 @@ -318,9 +336,9 @@ def test_parse_all_octets(self): self.assertIsNotNone(resp.getheader('obs-text')) self.assertIn('obs-text', resp.msg) for folded in (resp.getheader('obs-fold'), resp.msg['obs-fold']): - self.assertTrue(folded.startswith('text')) + self.assertStartsWith(folded, 'text') self.assertIn(' folded with space', folded) - self.assertTrue(folded.endswith('folded with tab')) + self.assertEndsWith(folded, 'folded with tab') def test_invalid_headers(self): conn = client.HTTPConnection('example.com') @@ -520,11 +538,203 @@ def _parse_chunked(self, data): return b''.join(body) -class BasicTest(TestCase): +class BasicTest(TestCase, ExtraAssertions): def test_dir_with_added_behavior_on_status(self): # see issue40084 self.assertTrue({'description', 'name', 'phrase', 'value'} <= set(dir(HTTPStatus(404)))) + def test_simple_httpstatus(self): + class CheckedHTTPStatus(enum.IntEnum): + """HTTP status codes and reason phrases + + Status codes from the following RFCs are all observed: + + * RFC 7231: Hypertext Transfer Protocol (HTTP/1.1), obsoletes 2616 + * RFC 6585: Additional HTTP Status Codes + * RFC 3229: Delta encoding in HTTP + * RFC 4918: HTTP Extensions for WebDAV, obsoletes 2518 + * RFC 5842: Binding Extensions to WebDAV + * RFC 7238: Permanent Redirect + * RFC 2295: Transparent Content Negotiation in HTTP + * RFC 2774: An HTTP Extension Framework + * RFC 7725: An HTTP Status Code to Report Legal Obstacles + * RFC 7540: Hypertext Transfer Protocol Version 2 (HTTP/2) + * RFC 2324: Hyper Text Coffee Pot Control Protocol (HTCPCP/1.0) + * RFC 8297: An HTTP Status Code for Indicating Hints + * RFC 8470: Using Early Data in HTTP + """ + def __new__(cls, value, phrase, description=''): + obj = int.__new__(cls, value) + obj._value_ = value + + obj.phrase = phrase + obj.description = description + return obj + + @property + def is_informational(self): + return 100 <= self <= 199 + + @property + def is_success(self): + return 200 <= self <= 299 + + @property + def is_redirection(self): + return 300 <= self <= 399 + + @property + def is_client_error(self): + return 400 <= self <= 499 + + @property + def is_server_error(self): + return 500 <= self <= 599 + + # informational + CONTINUE = 100, 'Continue', 'Request received, please continue' + SWITCHING_PROTOCOLS = (101, 'Switching Protocols', + 'Switching to new protocol; obey Upgrade header') + PROCESSING = 102, 'Processing' + EARLY_HINTS = 103, 'Early Hints' + # success + OK = 200, 'OK', 'Request fulfilled, document follows' + CREATED = 201, 'Created', 'Document created, URL follows' + ACCEPTED = (202, 'Accepted', + 'Request accepted, processing continues off-line') + NON_AUTHORITATIVE_INFORMATION = (203, + 'Non-Authoritative Information', 'Request fulfilled from cache') + NO_CONTENT = 204, 'No Content', 'Request fulfilled, nothing follows' + RESET_CONTENT = 205, 'Reset Content', 'Clear input form for further input' + PARTIAL_CONTENT = 206, 'Partial Content', 'Partial content follows' + MULTI_STATUS = 207, 'Multi-Status' + ALREADY_REPORTED = 208, 'Already Reported' + IM_USED = 226, 'IM Used' + # redirection + MULTIPLE_CHOICES = (300, 'Multiple Choices', + 'Object has several resources -- see URI list') + MOVED_PERMANENTLY = (301, 'Moved Permanently', + 'Object moved permanently -- see URI list') + FOUND = 302, 'Found', 'Object moved temporarily -- see URI list' + SEE_OTHER = 303, 'See Other', 'Object moved -- see Method and URL list' + NOT_MODIFIED = (304, 'Not Modified', + 'Document has not changed since given time') + USE_PROXY = (305, 'Use Proxy', + 'You must use proxy specified in Location to access this resource') + TEMPORARY_REDIRECT = (307, 'Temporary Redirect', + 'Object moved temporarily -- see URI list') + PERMANENT_REDIRECT = (308, 'Permanent Redirect', + 'Object moved permanently -- see URI list') + # client error + BAD_REQUEST = (400, 'Bad Request', + 'Bad request syntax or unsupported method') + UNAUTHORIZED = (401, 'Unauthorized', + 'No permission -- see authorization schemes') + PAYMENT_REQUIRED = (402, 'Payment Required', + 'No payment -- see charging schemes') + FORBIDDEN = (403, 'Forbidden', + 'Request forbidden -- authorization will not help') + NOT_FOUND = (404, 'Not Found', + 'Nothing matches the given URI') + METHOD_NOT_ALLOWED = (405, 'Method Not Allowed', + 'Specified method is invalid for this resource') + NOT_ACCEPTABLE = (406, 'Not Acceptable', + 'URI not available in preferred format') + PROXY_AUTHENTICATION_REQUIRED = (407, + 'Proxy Authentication Required', + 'You must authenticate with this proxy before proceeding') + REQUEST_TIMEOUT = (408, 'Request Timeout', + 'Request timed out; try again later') + CONFLICT = 409, 'Conflict', 'Request conflict' + GONE = (410, 'Gone', + 'URI no longer exists and has been permanently removed') + LENGTH_REQUIRED = (411, 'Length Required', + 'Client must specify Content-Length') + PRECONDITION_FAILED = (412, 'Precondition Failed', + 'Precondition in headers is false') + CONTENT_TOO_LARGE = (413, 'Content Too Large', + 'Content is too large') + REQUEST_ENTITY_TOO_LARGE = CONTENT_TOO_LARGE + URI_TOO_LONG = (414, 'URI Too Long', 'URI is too long') + REQUEST_URI_TOO_LONG = URI_TOO_LONG + UNSUPPORTED_MEDIA_TYPE = (415, 'Unsupported Media Type', + 'Entity body in unsupported format') + RANGE_NOT_SATISFIABLE = (416, + 'Range Not Satisfiable', + 'Cannot satisfy request range') + REQUESTED_RANGE_NOT_SATISFIABLE = RANGE_NOT_SATISFIABLE + EXPECTATION_FAILED = (417, 'Expectation Failed', + 'Expect condition could not be satisfied') + IM_A_TEAPOT = (418, 'I\'m a Teapot', + 'Server refuses to brew coffee because it is a teapot.') + MISDIRECTED_REQUEST = (421, 'Misdirected Request', + 'Server is not able to produce a response') + UNPROCESSABLE_CONTENT = 422, 'Unprocessable Content' + UNPROCESSABLE_ENTITY = UNPROCESSABLE_CONTENT + LOCKED = 423, 'Locked' + FAILED_DEPENDENCY = 424, 'Failed Dependency' + TOO_EARLY = 425, 'Too Early' + UPGRADE_REQUIRED = 426, 'Upgrade Required' + PRECONDITION_REQUIRED = (428, 'Precondition Required', + 'The origin server requires the request to be conditional') + TOO_MANY_REQUESTS = (429, 'Too Many Requests', + 'The user has sent too many requests in ' + 'a given amount of time ("rate limiting")') + REQUEST_HEADER_FIELDS_TOO_LARGE = (431, + 'Request Header Fields Too Large', + 'The server is unwilling to process the request because its header ' + 'fields are too large') + UNAVAILABLE_FOR_LEGAL_REASONS = (451, + 'Unavailable For Legal Reasons', + 'The server is denying access to the ' + 'resource as a consequence of a legal demand') + # server errors + INTERNAL_SERVER_ERROR = (500, 'Internal Server Error', + 'Server got itself in trouble') + NOT_IMPLEMENTED = (501, 'Not Implemented', + 'Server does not support this operation') + BAD_GATEWAY = (502, 'Bad Gateway', + 'Invalid responses from another server/proxy') + SERVICE_UNAVAILABLE = (503, 'Service Unavailable', + 'The server cannot process the request due to a high load') + GATEWAY_TIMEOUT = (504, 'Gateway Timeout', + 'The gateway server did not receive a timely response') + HTTP_VERSION_NOT_SUPPORTED = (505, 'HTTP Version Not Supported', + 'Cannot fulfill request') + VARIANT_ALSO_NEGOTIATES = 506, 'Variant Also Negotiates' + INSUFFICIENT_STORAGE = 507, 'Insufficient Storage' + LOOP_DETECTED = 508, 'Loop Detected' + NOT_EXTENDED = 510, 'Not Extended' + NETWORK_AUTHENTICATION_REQUIRED = (511, + 'Network Authentication Required', + 'The client needs to authenticate to gain network access') + enum._test_simple_enum(CheckedHTTPStatus, HTTPStatus) + + def test_httpstatus_range(self): + """Checks that the statuses are in the 100-599 range""" + + for member in HTTPStatus.__members__.values(): + self.assertGreaterEqual(member, 100) + self.assertLessEqual(member, 599) + + def test_httpstatus_category(self): + """Checks that the statuses belong to the standard categories""" + + categories = ( + ((100, 199), "is_informational"), + ((200, 299), "is_success"), + ((300, 399), "is_redirection"), + ((400, 499), "is_client_error"), + ((500, 599), "is_server_error"), + ) + for member in HTTPStatus.__members__.values(): + for (lower, upper), category in categories: + category_indicator = getattr(member, category) + if lower <= member <= upper: + self.assertTrue(category_indicator) + else: + self.assertFalse(category_indicator) + def test_status_lines(self): # Test HTTP status lines @@ -780,8 +990,7 @@ def test_send_file(self): sock = FakeSocket(body) conn.sock = sock conn.request('GET', '/foo', body) - self.assertTrue(sock.data.startswith(expected), '%r != %r' % - (sock.data[:len(expected)], expected)) + self.assertStartsWith(sock.data, expected) def test_send(self): expected = b'this is a test this is only a test' @@ -872,6 +1081,25 @@ def test_chunked(self): self.assertEqual(resp.read(), expected) resp.close() + # Explicit full read + for n in (-123, -1, None): + with self.subTest('full read', n=n): + sock = FakeSocket(chunked_start + last_chunk + chunked_end) + resp = client.HTTPResponse(sock, method="GET") + resp.begin() + self.assertTrue(resp.chunked) + self.assertEqual(resp.read(n), expected) + resp.close() + + # Read first chunk + with self.subTest('read1(-1)'): + sock = FakeSocket(chunked_start + last_chunk + chunked_end) + resp = client.HTTPResponse(sock, method="GET") + resp.begin() + self.assertTrue(resp.chunked) + self.assertEqual(resp.read1(-1), b"hello worl") + resp.close() + # Various read sizes for n in range(1, 12): sock = FakeSocket(chunked_start + last_chunk + chunked_end) @@ -1227,6 +1455,72 @@ def run_server(): thread.join() self.assertEqual(result, b"proxied data\n") + def test_large_content_length(self): + serv = socket.create_server((HOST, 0)) + self.addCleanup(serv.close) + + def run_server(): + [conn, address] = serv.accept() + with conn: + while conn.recv(1024): + conn.sendall( + b"HTTP/1.1 200 Ok\r\n" + b"Content-Length: %d\r\n" + b"\r\n" % size) + conn.sendall(b'A' * (size//3)) + conn.sendall(b'B' * (size - size//3)) + + thread = threading.Thread(target=run_server) + thread.start() + self.addCleanup(thread.join, 1.0) + + conn = client.HTTPConnection(*serv.getsockname()) + try: + for w in range(15, 27): + size = 1 << w + conn.request("GET", "/") + with conn.getresponse() as response: + self.assertEqual(len(response.read()), size) + finally: + conn.close() + thread.join(1.0) + + def test_large_content_length_truncated(self): + serv = socket.create_server((HOST, 0)) + self.addCleanup(serv.close) + + def run_server(): + while True: + [conn, address] = serv.accept() + with conn: + conn.recv(1024) + if not size: + break + conn.sendall( + b"HTTP/1.1 200 Ok\r\n" + b"Content-Length: %d\r\n" + b"\r\n" + b"Text" % size) + + thread = threading.Thread(target=run_server) + thread.start() + self.addCleanup(thread.join, 1.0) + + conn = client.HTTPConnection(*serv.getsockname()) + try: + for w in range(18, 65): + size = 1 << w + conn.request("GET", "/") + with conn.getresponse() as response: + self.assertRaises(client.IncompleteRead, response.read) + conn.close() + finally: + conn.close() + size = 0 + conn.request("GET", "/") + conn.close() + thread.join(1.0) + def test_putrequest_override_domain_validation(self): """ It should be possible to override the default validation @@ -1266,7 +1560,7 @@ def _encode_request(self, str_url): conn.putrequest('GET', '/☃') -class ExtendedReadTest(TestCase): +class ExtendedReadTest(TestCase, ExtraAssertions): """ Test peek(), read1(), readline() """ @@ -1325,7 +1619,7 @@ def mypeek(n=-1): # then unbounded peek p2 = resp.peek() self.assertGreaterEqual(len(p2), len(p)) - self.assertTrue(p2.startswith(p)) + self.assertStartsWith(p2, p) next = resp.read(len(p2)) self.assertEqual(next, p2) else: @@ -1340,18 +1634,22 @@ def test_readline(self): resp = self.resp self._verify_readline(self.resp.readline, self.lines_expected) - def _verify_readline(self, readline, expected): + def test_readline_without_limit(self): + self._verify_readline(self.resp.readline, self.lines_expected, limit=-1) + + def _verify_readline(self, readline, expected, limit=5): all = [] while True: # short readlines - line = readline(5) + line = readline(limit) if line and line != b"foo": if len(line) < 5: - self.assertTrue(line.endswith(b"\n")) + self.assertEndsWith(line, b"\n") all.append(line) if not line: break self.assertEqual(b"".join(all), expected) + self.assertTrue(self.resp.isclosed()) def test_read1(self): resp = self.resp @@ -1371,6 +1669,7 @@ def test_read1_unbounded(self): break all.append(data) self.assertEqual(b"".join(all), self.lines_expected) + self.assertTrue(resp.isclosed()) def test_read1_bounded(self): resp = self.resp @@ -1382,15 +1681,22 @@ def test_read1_bounded(self): self.assertLessEqual(len(data), 10) all.append(data) self.assertEqual(b"".join(all), self.lines_expected) + self.assertTrue(resp.isclosed()) def test_read1_0(self): self.assertEqual(self.resp.read1(0), b"") + self.assertFalse(self.resp.isclosed()) def test_peek_0(self): p = self.resp.peek(0) self.assertLessEqual(0, len(p)) +class ExtendedReadTestContentLengthKnown(ExtendedReadTest): + _header, _body = ExtendedReadTest.lines.split('\r\n\r\n', 1) + lines = _header + f'\r\nContent-Length: {len(_body)}\r\n\r\n' + _body + + class ExtendedReadTestChunked(ExtendedReadTest): """ Test peek(), read1(), readline() in chunked mode @@ -1447,7 +1753,7 @@ def readline(self, limit): raise -class OfflineTest(TestCase): +class OfflineTest(TestCase, ExtraAssertions): def test_all(self): # Documented objects defined in the module should be in __all__ expected = {"responses"} # Allowlist documented dict() object @@ -1500,13 +1806,17 @@ def test_client_constants(self): 'GONE', 'LENGTH_REQUIRED', 'PRECONDITION_FAILED', + 'CONTENT_TOO_LARGE', 'REQUEST_ENTITY_TOO_LARGE', + 'URI_TOO_LONG', 'REQUEST_URI_TOO_LONG', 'UNSUPPORTED_MEDIA_TYPE', + 'RANGE_NOT_SATISFIABLE', 'REQUESTED_RANGE_NOT_SATISFIABLE', 'EXPECTATION_FAILED', 'IM_A_TEAPOT', 'MISDIRECTED_REQUEST', + 'UNPROCESSABLE_CONTENT', 'UNPROCESSABLE_ENTITY', 'LOCKED', 'FAILED_DEPENDENCY', @@ -1529,7 +1839,7 @@ def test_client_constants(self): ] for const in expected: with self.subTest(constant=const): - self.assertTrue(hasattr(client, const)) + self.assertHasAttr(client, const) class SourceAddressTest(TestCase): @@ -1766,6 +2076,7 @@ def test_networked_good_cert(self): h.close() self.assertIn('nginx', server_string) + @support.requires_resource('walltime') def test_networked_bad_cert(self): # We feed a "CA" cert that is unrelated to the server's cert import ssl @@ -1778,7 +2089,6 @@ def test_networked_bad_cert(self): h.request('GET', '/') self.assertEqual(exc_info.exception.reason, 'CERTIFICATE_VERIFY_FAILED') - @unittest.skipIf(sys.platform == 'darwin', 'Occasionally success on macOS') def test_local_unknown_cert(self): # The custom cert isn't known to the default trust bundle import ssl @@ -1789,7 +2099,7 @@ def test_local_unknown_cert(self): self.assertEqual(exc_info.exception.reason, 'CERTIFICATE_VERIFY_FAILED') def test_local_good_hostname(self): - # The (valid) cert validates the HTTP hostname + # The (valid) cert validates the HTTPS hostname import ssl server = self.make_server(CERT_localhost) context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) @@ -1802,7 +2112,7 @@ def test_local_good_hostname(self): self.assertEqual(resp.status, 404) def test_local_bad_hostname(self): - # The (valid) cert doesn't validate the HTTP hostname + # The (valid) cert doesn't validate the HTTPS hostname import ssl server = self.make_server(CERT_fakehostname) context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) @@ -1810,38 +2120,21 @@ def test_local_bad_hostname(self): h = client.HTTPSConnection('localhost', server.port, context=context) with self.assertRaises(ssl.CertificateError): h.request('GET', '/') - # Same with explicit check_hostname=True - with warnings_helper.check_warnings(('', DeprecationWarning)): - h = client.HTTPSConnection('localhost', server.port, - context=context, check_hostname=True) + + # Same with explicit context.check_hostname=True + context.check_hostname = True + h = client.HTTPSConnection('localhost', server.port, context=context) with self.assertRaises(ssl.CertificateError): h.request('GET', '/') - # With check_hostname=False, the mismatching is ignored - context.check_hostname = False - with warnings_helper.check_warnings(('', DeprecationWarning)): - h = client.HTTPSConnection('localhost', server.port, - context=context, check_hostname=False) - h.request('GET', '/nonexistent') - resp = h.getresponse() - resp.close() - h.close() - self.assertEqual(resp.status, 404) - # The context's check_hostname setting is used if one isn't passed to - # HTTPSConnection. + + # With context.check_hostname=False, the mismatching is ignored context.check_hostname = False h = client.HTTPSConnection('localhost', server.port, context=context) h.request('GET', '/nonexistent') resp = h.getresponse() - self.assertEqual(resp.status, 404) resp.close() h.close() - # Passing check_hostname to HTTPSConnection should override the - # context's setting. - with warnings_helper.check_warnings(('', DeprecationWarning)): - h = client.HTTPSConnection('localhost', server.port, - context=context, check_hostname=True) - with self.assertRaises(ssl.CertificateError): - h.request('GET', '/') + self.assertEqual(resp.status, 404) @unittest.skipIf(not hasattr(client, 'HTTPSConnection'), 'http.client.HTTPSConnection not available') @@ -1877,11 +2170,9 @@ def test_tls13_pha(self): self.assertIs(h._context, context) self.assertFalse(h._context.post_handshake_auth) - with warnings.catch_warnings(): - warnings.filterwarnings('ignore', 'key_file, cert_file and check_hostname are deprecated', - DeprecationWarning) - h = client.HTTPSConnection('localhost', 443, context=context, - cert_file=CERT_localhost) + context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT, cert_file=CERT_localhost) + context.post_handshake_auth = True + h = client.HTTPSConnection('localhost', 443, context=context) self.assertTrue(h._context.post_handshake_auth) @@ -2016,14 +2307,15 @@ def test_getting_header_defaultint(self): header = self.resp.getheader('No-Such-Header',default=42) self.assertEqual(header, 42) -class TunnelTests(TestCase): +class TunnelTests(TestCase, ExtraAssertions): def setUp(self): response_text = ( - 'HTTP/1.0 200 OK\r\n\r\n' # Reply to CONNECT + 'HTTP/1.1 200 OK\r\n\r\n' # Reply to CONNECT 'HTTP/1.1 200 OK\r\n' # Reply to HEAD 'Content-Length: 42\r\n\r\n' ) self.host = 'proxy.com' + self.port = client.HTTP_PORT self.conn = client.HTTPConnection(self.host) self.conn._create_connection = self._create_connection(response_text) @@ -2035,15 +2327,45 @@ def create_connection(address, timeout=None, source_address=None): return FakeSocket(response_text, host=address[0], port=address[1]) return create_connection - def test_set_tunnel_host_port_headers(self): + def test_set_tunnel_host_port_headers_add_host_missing(self): tunnel_host = 'destination.com' tunnel_port = 8888 tunnel_headers = {'User-Agent': 'Mozilla/5.0 (compatible, MSIE 11)'} + tunnel_headers_after = tunnel_headers.copy() + tunnel_headers_after['Host'] = '%s:%d' % (tunnel_host, tunnel_port) self.conn.set_tunnel(tunnel_host, port=tunnel_port, headers=tunnel_headers) self.conn.request('HEAD', '/', '') self.assertEqual(self.conn.sock.host, self.host) - self.assertEqual(self.conn.sock.port, client.HTTP_PORT) + self.assertEqual(self.conn.sock.port, self.port) + self.assertEqual(self.conn._tunnel_host, tunnel_host) + self.assertEqual(self.conn._tunnel_port, tunnel_port) + self.assertEqual(self.conn._tunnel_headers, tunnel_headers_after) + + def test_set_tunnel_host_port_headers_set_host_identical(self): + tunnel_host = 'destination.com' + tunnel_port = 8888 + tunnel_headers = {'User-Agent': 'Mozilla/5.0 (compatible, MSIE 11)', + 'Host': '%s:%d' % (tunnel_host, tunnel_port)} + self.conn.set_tunnel(tunnel_host, port=tunnel_port, + headers=tunnel_headers) + self.conn.request('HEAD', '/', '') + self.assertEqual(self.conn.sock.host, self.host) + self.assertEqual(self.conn.sock.port, self.port) + self.assertEqual(self.conn._tunnel_host, tunnel_host) + self.assertEqual(self.conn._tunnel_port, tunnel_port) + self.assertEqual(self.conn._tunnel_headers, tunnel_headers) + + def test_set_tunnel_host_port_headers_set_host_different(self): + tunnel_host = 'destination.com' + tunnel_port = 8888 + tunnel_headers = {'User-Agent': 'Mozilla/5.0 (compatible, MSIE 11)', + 'Host': '%s:%d' % ('example.com', 4200)} + self.conn.set_tunnel(tunnel_host, port=tunnel_port, + headers=tunnel_headers) + self.conn.request('HEAD', '/', '') + self.assertEqual(self.conn.sock.host, self.host) + self.assertEqual(self.conn.sock.port, self.port) self.assertEqual(self.conn._tunnel_host, tunnel_host) self.assertEqual(self.conn._tunnel_port, tunnel_port) self.assertEqual(self.conn._tunnel_headers, tunnel_headers) @@ -2055,17 +2377,96 @@ def test_disallow_set_tunnel_after_connect(self): 'destination.com') def test_connect_with_tunnel(self): - self.conn.set_tunnel('destination.com') + d = { + b'host': b'destination.com', + b'port': client.HTTP_PORT, + } + self.conn.set_tunnel(d[b'host'].decode('ascii')) + self.conn.request('HEAD', '/', '') + self.assertEqual(self.conn.sock.host, self.host) + self.assertEqual(self.conn.sock.port, self.port) + self.assertIn(b'CONNECT %(host)s:%(port)d HTTP/1.1\r\n' + b'Host: %(host)s:%(port)d\r\n\r\n' % d, + self.conn.sock.data) + self.assertIn(b'HEAD / HTTP/1.1\r\nHost: %(host)s\r\n' % d, + self.conn.sock.data) + + def test_connect_with_tunnel_with_default_port(self): + d = { + b'host': b'destination.com', + b'port': client.HTTP_PORT, + } + self.conn.set_tunnel(d[b'host'].decode('ascii'), port=d[b'port']) + self.conn.request('HEAD', '/', '') + self.assertEqual(self.conn.sock.host, self.host) + self.assertEqual(self.conn.sock.port, self.port) + self.assertIn(b'CONNECT %(host)s:%(port)d HTTP/1.1\r\n' + b'Host: %(host)s:%(port)d\r\n\r\n' % d, + self.conn.sock.data) + self.assertIn(b'HEAD / HTTP/1.1\r\nHost: %(host)s\r\n' % d, + self.conn.sock.data) + + def test_connect_with_tunnel_with_nonstandard_port(self): + d = { + b'host': b'destination.com', + b'port': 8888, + } + self.conn.set_tunnel(d[b'host'].decode('ascii'), port=d[b'port']) + self.conn.request('HEAD', '/', '') + self.assertEqual(self.conn.sock.host, self.host) + self.assertEqual(self.conn.sock.port, self.port) + self.assertIn(b'CONNECT %(host)s:%(port)d HTTP/1.1\r\n' + b'Host: %(host)s:%(port)d\r\n\r\n' % d, + self.conn.sock.data) + self.assertIn(b'HEAD / HTTP/1.1\r\nHost: %(host)s:%(port)d\r\n' % d, + self.conn.sock.data) + + # This request is not RFC-valid, but it's been possible with the library + # for years, so don't break it unexpectedly... This also tests + # case-insensitivity when injecting Host: headers if they're missing. + def test_connect_with_tunnel_with_different_host_header(self): + d = { + b'host': b'destination.com', + b'tunnel_host_header': b'example.com:9876', + b'port': client.HTTP_PORT, + } + self.conn.set_tunnel( + d[b'host'].decode('ascii'), + headers={'HOST': d[b'tunnel_host_header'].decode('ascii')}) + self.conn.request('HEAD', '/', '') + self.assertEqual(self.conn.sock.host, self.host) + self.assertEqual(self.conn.sock.port, self.port) + self.assertIn(b'CONNECT %(host)s:%(port)d HTTP/1.1\r\n' + b'HOST: %(tunnel_host_header)s\r\n\r\n' % d, + self.conn.sock.data) + self.assertIn(b'HEAD / HTTP/1.1\r\nHost: %(host)s\r\n' % d, + self.conn.sock.data) + + def test_connect_with_tunnel_different_host(self): + d = { + b'host': b'destination.com', + b'port': client.HTTP_PORT, + } + self.conn.set_tunnel(d[b'host'].decode('ascii')) + self.conn.request('HEAD', '/', '') + self.assertEqual(self.conn.sock.host, self.host) + self.assertEqual(self.conn.sock.port, self.port) + self.assertIn(b'CONNECT %(host)s:%(port)d HTTP/1.1\r\n' + b'Host: %(host)s:%(port)d\r\n\r\n' % d, + self.conn.sock.data) + self.assertIn(b'HEAD / HTTP/1.1\r\nHost: %(host)s\r\n' % d, + self.conn.sock.data) + + def test_connect_with_tunnel_idna(self): + dest = '\u03b4\u03c0\u03b8.gr' + dest_port = b'%s:%d' % (dest.encode('idna'), client.HTTP_PORT) + expected = b'CONNECT %s HTTP/1.1\r\nHost: %s\r\n\r\n' % ( + dest_port, dest_port) + self.conn.set_tunnel(dest) self.conn.request('HEAD', '/', '') self.assertEqual(self.conn.sock.host, self.host) self.assertEqual(self.conn.sock.port, client.HTTP_PORT) - self.assertIn(b'CONNECT destination.com', self.conn.sock.data) - # issue22095 - self.assertNotIn(b'Host: destination.com:None', self.conn.sock.data) - self.assertIn(b'Host: destination.com', self.conn.sock.data) - - # This test should be removed when CONNECT gets the HTTP/1.1 blessing - self.assertNotIn(b'Host: proxy.com', self.conn.sock.data) + self.assertIn(expected, self.conn.sock.data) def test_tunnel_connect_single_send_connection_setup(self): """Regresstion test for https://bugs.python.org/issue43332.""" @@ -2080,17 +2481,39 @@ def test_tunnel_connect_single_send_connection_setup(self): msg=f'unexpected number of send calls: {mock_send.mock_calls}') proxy_setup_data_sent = mock_send.mock_calls[0][1][0] self.assertIn(b'CONNECT destination.com', proxy_setup_data_sent) - self.assertTrue( - proxy_setup_data_sent.endswith(b'\r\n\r\n'), + self.assertEndsWith(proxy_setup_data_sent, b'\r\n\r\n', msg=f'unexpected proxy data sent {proxy_setup_data_sent!r}') def test_connect_put_request(self): - self.conn.set_tunnel('destination.com') + d = { + b'host': b'destination.com', + b'port': client.HTTP_PORT, + } + self.conn.set_tunnel(d[b'host'].decode('ascii')) + self.conn.request('PUT', '/', '') + self.assertEqual(self.conn.sock.host, self.host) + self.assertEqual(self.conn.sock.port, self.port) + self.assertIn(b'CONNECT %(host)s:%(port)d HTTP/1.1\r\n' + b'Host: %(host)s:%(port)d\r\n\r\n' % d, + self.conn.sock.data) + self.assertIn(b'PUT / HTTP/1.1\r\nHost: %(host)s\r\n' % d, + self.conn.sock.data) + + def test_connect_put_request_ipv6(self): + self.conn.set_tunnel('[1:2:3::4]', 1234) + self.conn.request('PUT', '/', '') + self.assertEqual(self.conn.sock.host, self.host) + self.assertEqual(self.conn.sock.port, client.HTTP_PORT) + self.assertIn(b'CONNECT [1:2:3::4]:1234', self.conn.sock.data) + self.assertIn(b'Host: [1:2:3::4]:1234', self.conn.sock.data) + + def test_connect_put_request_ipv6_port(self): + self.conn.set_tunnel('[1:2:3::4]:1234') self.conn.request('PUT', '/', '') self.assertEqual(self.conn.sock.host, self.host) self.assertEqual(self.conn.sock.port, client.HTTP_PORT) - self.assertIn(b'CONNECT destination.com', self.conn.sock.data) - self.assertIn(b'Host: destination.com', self.conn.sock.data) + self.assertIn(b'CONNECT [1:2:3::4]:1234', self.conn.sock.data) + self.assertIn(b'Host: [1:2:3::4]:1234', self.conn.sock.data) def test_tunnel_debuglog(self): expected_header = 'X-Dummy: 1' @@ -2105,6 +2528,56 @@ def test_tunnel_debuglog(self): lines = output.getvalue().splitlines() self.assertIn('header: {}'.format(expected_header), lines) + def test_proxy_response_headers(self): + expected_header = ('X-Dummy', '1') + response_text = ( + 'HTTP/1.0 200 OK\r\n' + '{0}\r\n\r\n'.format(':'.join(expected_header)) + ) + + self.conn._create_connection = self._create_connection(response_text) + self.conn.set_tunnel('destination.com') + + self.conn.request('PUT', '/', '') + headers = self.conn.get_proxy_response_headers() + self.assertIn(expected_header, headers.items()) + + def test_no_proxy_response_headers(self): + expected_header = ('X-Dummy', '1') + response_text = ( + 'HTTP/1.0 200 OK\r\n' + '{0}\r\n\r\n'.format(':'.join(expected_header)) + ) + + self.conn._create_connection = self._create_connection(response_text) + + self.conn.request('PUT', '/', '') + headers = self.conn.get_proxy_response_headers() + self.assertIsNone(headers) + + def test_tunnel_leak(self): + sock = None + + def _create_connection(address, timeout=None, source_address=None): + nonlocal sock + sock = FakeSocket( + 'HTTP/1.1 404 NOT FOUND\r\n\r\n', + host=address[0], + port=address[1], + ) + return sock + + self.conn._create_connection = _create_connection + self.conn.set_tunnel('destination.com') + exc = None + try: + self.conn.request('HEAD', '/', '') + except OSError as e: + # keeping a reference to exc keeps response alive in the traceback + exc = e + self.assertIsNotNone(exc) + self.assertTrue(sock.file_closed) + if __name__ == '__main__': unittest.main(verbosity=2) diff --git a/Lib/test/test_httpservers.py b/Lib/test/test_httpservers.py index cd689492ca3..f402a34fbdf 100644 --- a/Lib/test/test_httpservers.py +++ b/Lib/test/test_httpservers.py @@ -8,6 +8,7 @@ SimpleHTTPRequestHandler, CGIHTTPRequestHandler from http import server, HTTPStatus +import contextlib import os import socket import sys @@ -26,13 +27,16 @@ import datetime import threading from unittest import mock -from io import BytesIO +from io import BytesIO, StringIO import unittest from test import support -from test.support import os_helper -from test.support import threading_helper +from test.support import ( + is_apple, os_helper, requires_subprocess, threading_helper +) +from test.support.testcase import ExtraAssertions +support.requires_working_socket(module=True) class NoLogRequestHandler: def log_message(self, *args): @@ -64,7 +68,7 @@ def stop(self): self.join() -class BaseTestCase(unittest.TestCase): +class BaseTestCase(unittest.TestCase, ExtraAssertions): def setUp(self): self._threads = threading_helper.threading_setup() os.environ = os_helper.EnvironmentVarGuard() @@ -163,6 +167,27 @@ def test_version_digits(self): res = self.con.getresponse() self.assertEqual(res.status, HTTPStatus.BAD_REQUEST) + def test_version_signs_and_underscores(self): + self.con._http_vsn_str = 'HTTP/-9_9_9.+9_9_9' + self.con.putrequest('GET', '/') + self.con.endheaders() + res = self.con.getresponse() + self.assertEqual(res.status, HTTPStatus.BAD_REQUEST) + + def test_major_version_number_too_long(self): + self.con._http_vsn_str = 'HTTP/909876543210.0' + self.con.putrequest('GET', '/') + self.con.endheaders() + res = self.con.getresponse() + self.assertEqual(res.status, HTTPStatus.BAD_REQUEST) + + def test_minor_version_number_too_long(self): + self.con._http_vsn_str = 'HTTP/1.909876543210' + self.con.putrequest('GET', '/') + self.con.endheaders() + res = self.con.getresponse() + self.assertEqual(res.status, HTTPStatus.BAD_REQUEST) + def test_version_none_get(self): self.con._http_vsn_str = '' self.con.putrequest('GET', '/') @@ -292,6 +317,44 @@ def test_head_via_send_error(self): self.assertEqual(b'', data) +class HTTP09ServerTestCase(BaseTestCase): + + class request_handler(NoLogRequestHandler, BaseHTTPRequestHandler): + """Request handler for HTTP/0.9 server.""" + + def do_GET(self): + self.wfile.write(f'OK: here is {self.path}\r\n'.encode()) + + def setUp(self): + super().setUp() + self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + self.sock = self.enterContext(self.sock) + self.sock.connect((self.HOST, self.PORT)) + + def test_simple_get(self): + self.sock.send(b'GET /index.html\r\n') + res = self.sock.recv(1024) + self.assertEqual(res, b"OK: here is /index.html\r\n") + + def test_invalid_request(self): + self.sock.send(b'POST /index.html\r\n') + res = self.sock.recv(1024) + self.assertIn(b"Bad HTTP/0.9 request type ('POST')", res) + + def test_single_request(self): + self.sock.send(b'GET /foo.html\r\n') + res = self.sock.recv(1024) + self.assertEqual(res, b"OK: here is /foo.html\r\n") + + # Ignore errors if the connection is already closed, + # as this is the expected behavior of HTTP/0.9. + with contextlib.suppress(OSError): + self.sock.send(b'GET /bar.html\r\n') + res = self.sock.recv(1024) + # The server should not process our request. + self.assertEqual(res, b'') + + class RequestHandlerLoggingTestCase(BaseTestCase): class request_handler(BaseHTTPRequestHandler): protocol_version = 'HTTP/1.1' @@ -312,8 +375,7 @@ def test_get(self): self.con.request('GET', '/') self.con.getresponse() - self.assertTrue( - err.getvalue().endswith('"GET / HTTP/1.1" 200 -\n')) + self.assertEndsWith(err.getvalue(), '"GET / HTTP/1.1" 200 -\n') def test_err(self): self.con = http.client.HTTPConnection(self.HOST, self.PORT) @@ -324,8 +386,8 @@ def test_err(self): self.con.getresponse() lines = err.getvalue().split('\n') - self.assertTrue(lines[0].endswith('code 404, message File not found')) - self.assertTrue(lines[1].endswith('"ERROR / HTTP/1.1" 404 -')) + self.assertEndsWith(lines[0], 'code 404, message File not found') + self.assertEndsWith(lines[1], '"ERROR / HTTP/1.1" 404 -') class SimpleHTTPServerTestCase(BaseTestCase): @@ -333,7 +395,7 @@ class request_handler(NoLogRequestHandler, SimpleHTTPRequestHandler): pass def setUp(self): - BaseTestCase.setUp(self) + super().setUp() self.cwd = os.getcwd() basetempdir = tempfile.gettempdir() os.chdir(basetempdir) @@ -361,7 +423,7 @@ def tearDown(self): except: pass finally: - BaseTestCase.tearDown(self) + super().tearDown() def check_status_and_reason(self, response, status, data=None): def close_conn(): @@ -388,34 +450,169 @@ def close_conn(): reader.close() return body - @unittest.skipIf(sys.platform == 'darwin', - 'undecodable name cannot always be decoded on macOS') - @unittest.skipIf(sys.platform == 'win32', - 'undecodable name cannot be decoded on win32') - @unittest.skipUnless(os_helper.TESTFN_UNDECODABLE, - 'need os_helper.TESTFN_UNDECODABLE') - def test_undecodable_filename(self): + def check_list_dir_dirname(self, dirname, quotedname=None): + fullpath = os.path.join(self.tempdir, dirname) + try: + os.mkdir(os.path.join(self.tempdir, dirname)) + except (OSError, UnicodeEncodeError): + self.skipTest(f'Can not create directory {dirname!a} ' + f'on current file system') + + if quotedname is None: + quotedname = urllib.parse.quote(dirname, errors='surrogatepass') + response = self.request(self.base_url + '/' + quotedname + '/') + body = self.check_status_and_reason(response, HTTPStatus.OK) + displaypath = html.escape(f'{self.base_url}/{dirname}/', quote=False) enc = sys.getfilesystemencoding() - filename = os.fsdecode(os_helper.TESTFN_UNDECODABLE) + '.txt' - with open(os.path.join(self.tempdir, filename), 'wb') as f: - f.write(os_helper.TESTFN_UNDECODABLE) + prefix = f'listing for {displaypath}</'.encode(enc, 'surrogateescape') + self.assertIn(prefix + b'title>', body) + self.assertIn(prefix + b'h1>', body) + + def check_list_dir_filename(self, filename): + fullpath = os.path.join(self.tempdir, filename) + content = ascii(fullpath).encode() + (os_helper.TESTFN_UNDECODABLE or b'\xff') + try: + with open(fullpath, 'wb') as f: + f.write(content) + except OSError: + self.skipTest(f'Can not create file {filename!a} ' + f'on current file system') + response = self.request(self.base_url + '/') - if sys.platform == 'darwin': - # On Mac OS the HFS+ filesystem replaces bytes that aren't valid - # UTF-8 into a percent-encoded value. - for name in os.listdir(self.tempdir): - if name != 'test': # Ignore a filename created in setUp(). - filename = name - break body = self.check_status_and_reason(response, HTTPStatus.OK) quotedname = urllib.parse.quote(filename, errors='surrogatepass') - self.assertIn(('href="%s"' % quotedname) - .encode(enc, 'surrogateescape'), body) - self.assertIn(('>%s<' % html.escape(filename, quote=False)) - .encode(enc, 'surrogateescape'), body) + enc = response.headers.get_content_charset() + self.assertIsNotNone(enc) + self.assertIn((f'href="{quotedname}"').encode('ascii'), body) + displayname = html.escape(filename, quote=False) + self.assertIn(f'>{displayname}<'.encode(enc, 'surrogateescape'), body) + response = self.request(self.base_url + '/' + quotedname) - self.check_status_and_reason(response, HTTPStatus.OK, - data=os_helper.TESTFN_UNDECODABLE) + self.check_status_and_reason(response, HTTPStatus.OK, data=content) + + @unittest.skipUnless(os_helper.TESTFN_NONASCII, + 'need os_helper.TESTFN_NONASCII') + def test_list_dir_nonascii_dirname(self): + dirname = os_helper.TESTFN_NONASCII + '.dir' + self.check_list_dir_dirname(dirname) + + @unittest.skipUnless(os_helper.TESTFN_NONASCII, + 'need os_helper.TESTFN_NONASCII') + def test_list_dir_nonascii_filename(self): + filename = os_helper.TESTFN_NONASCII + '.txt' + self.check_list_dir_filename(filename) + + @unittest.skipIf(is_apple, + 'undecodable name cannot always be decoded on Apple platforms') + @unittest.skipIf(sys.platform == 'win32', + 'undecodable name cannot be decoded on win32') + @unittest.skipUnless(os_helper.TESTFN_UNDECODABLE, + 'need os_helper.TESTFN_UNDECODABLE') + def test_list_dir_undecodable_dirname(self): + dirname = os.fsdecode(os_helper.TESTFN_UNDECODABLE) + '.dir' + self.check_list_dir_dirname(dirname) + + @unittest.skipIf(is_apple, + 'undecodable name cannot always be decoded on Apple platforms') + @unittest.skipIf(sys.platform == 'win32', + 'undecodable name cannot be decoded on win32') + @unittest.skipUnless(os_helper.TESTFN_UNDECODABLE, + 'need os_helper.TESTFN_UNDECODABLE') + def test_list_dir_undecodable_filename(self): + filename = os.fsdecode(os_helper.TESTFN_UNDECODABLE) + '.txt' + self.check_list_dir_filename(filename) + + def test_list_dir_undecodable_dirname2(self): + dirname = '\ufffd.dir' + self.check_list_dir_dirname(dirname, quotedname='%ff.dir') + + @unittest.skipUnless(os_helper.TESTFN_UNENCODABLE, + 'need os_helper.TESTFN_UNENCODABLE') + def test_list_dir_unencodable_dirname(self): + dirname = os_helper.TESTFN_UNENCODABLE + '.dir' + self.check_list_dir_dirname(dirname) + + @unittest.skipUnless(os_helper.TESTFN_UNENCODABLE, + 'need os_helper.TESTFN_UNENCODABLE') + def test_list_dir_unencodable_filename(self): + filename = os_helper.TESTFN_UNENCODABLE + '.txt' + self.check_list_dir_filename(filename) + + def test_list_dir_escape_dirname(self): + # Characters that need special treating in URL or HTML. + for name in ('q?', 'f#', '&amp;', '&amp', '<i>', '"dq"', "'sq'", + '%A4', '%E2%82%AC'): + with self.subTest(name=name): + dirname = name + '.dir' + self.check_list_dir_dirname(dirname, + quotedname=urllib.parse.quote(dirname, safe='&<>\'"')) + + def test_list_dir_escape_filename(self): + # Characters that need special treating in URL or HTML. + for name in ('q?', 'f#', '&amp;', '&amp', '<i>', '"dq"', "'sq'", + '%A4', '%E2%82%AC'): + with self.subTest(name=name): + filename = name + '.txt' + self.check_list_dir_filename(filename) + os_helper.unlink(os.path.join(self.tempdir, filename)) + + def test_list_dir_with_query_and_fragment(self): + prefix = f'listing for {self.base_url}/</'.encode('latin1') + response = self.request(self.base_url + '/#123').read() + self.assertIn(prefix + b'title>', response) + self.assertIn(prefix + b'h1>', response) + response = self.request(self.base_url + '/?x=123').read() + self.assertIn(prefix + b'title>', response) + self.assertIn(prefix + b'h1>', response) + + def test_get_dir_redirect_location_domain_injection_bug(self): + """Ensure //evil.co/..%2f../../X does not put //evil.co/ in Location. + + //netloc/ in a Location header is a redirect to a new host. + https://github.com/python/cpython/issues/87389 + + This checks that a path resolving to a directory on our server cannot + resolve into a redirect to another server. + """ + os.mkdir(os.path.join(self.tempdir, 'existing_directory')) + url = f'/python.org/..%2f..%2f..%2f..%2f..%2f../%0a%0d/../{self.tempdir_name}/existing_directory' + expected_location = f'{url}/' # /python.org.../ single slash single prefix, trailing slash + # Canonicalizes to /tmp/tempdir_name/existing_directory which does + # exist and is a dir, triggering the 301 redirect logic. + response = self.request(url) + self.check_status_and_reason(response, HTTPStatus.MOVED_PERMANENTLY) + location = response.getheader('Location') + self.assertEqual(location, expected_location, msg='non-attack failed!') + + # //python.org... multi-slash prefix, no trailing slash + attack_url = f'/{url}' + response = self.request(attack_url) + self.check_status_and_reason(response, HTTPStatus.MOVED_PERMANENTLY) + location = response.getheader('Location') + self.assertNotStartsWith(location, '//') + self.assertEqual(location, expected_location, + msg='Expected Location header to start with a single / and ' + 'end with a / as this is a directory redirect.') + + # ///python.org... triple-slash prefix, no trailing slash + attack3_url = f'//{url}' + response = self.request(attack3_url) + self.check_status_and_reason(response, HTTPStatus.MOVED_PERMANENTLY) + self.assertEqual(response.getheader('Location'), expected_location) + + # If the second word in the http request (Request-URI for the http + # method) is a full URI, we don't worry about it, as that'll be parsed + # and reassembled as a full URI within BaseHTTPRequestHandler.send_head + # so no errant scheme-less //netloc//evil.co/ domain mixup can happen. + attack_scheme_netloc_2slash_url = f'https://pypi.org/{url}' + expected_scheme_netloc_location = f'{attack_scheme_netloc_2slash_url}/' + response = self.request(attack_scheme_netloc_2slash_url) + self.check_status_and_reason(response, HTTPStatus.MOVED_PERMANENTLY) + location = response.getheader('Location') + # We're just ensuring that the scheme and domain make it through, if + # there are or aren't multiple slashes at the start of the path that + # follows that isn't important in this Location: header. + self.assertStartsWith(location, 'https://pypi.org/') def test_get(self): #constructs the path relative to the root directory of the HTTPServer @@ -424,10 +621,19 @@ def test_get(self): # check for trailing "/" which should return 404. See Issue17324 response = self.request(self.base_url + '/test/') self.check_status_and_reason(response, HTTPStatus.NOT_FOUND) + response = self.request(self.base_url + '/test%2f') + self.check_status_and_reason(response, HTTPStatus.NOT_FOUND) + response = self.request(self.base_url + '/test%2F') + self.check_status_and_reason(response, HTTPStatus.NOT_FOUND) response = self.request(self.base_url + '/') self.check_status_and_reason(response, HTTPStatus.OK) + response = self.request(self.base_url + '%2f') + self.check_status_and_reason(response, HTTPStatus.OK) + response = self.request(self.base_url + '%2F') + self.check_status_and_reason(response, HTTPStatus.OK) response = self.request(self.base_url) self.check_status_and_reason(response, HTTPStatus.MOVED_PERMANENTLY) + self.assertEqual(response.getheader("Location"), self.base_url + "/") self.assertEqual(response.getheader("Content-Length"), "0") response = self.request(self.base_url + '/?hi=2') self.check_status_and_reason(response, HTTPStatus.OK) @@ -439,6 +645,9 @@ def test_get(self): self.check_status_and_reason(response, HTTPStatus.NOT_FOUND) response = self.request('/' + 'ThisDoesNotExist' + '/') self.check_status_and_reason(response, HTTPStatus.NOT_FOUND) + os.makedirs(os.path.join(self.tempdir, 'spam', 'index.html')) + response = self.request(self.base_url + '/spam/') + self.check_status_and_reason(response, HTTPStatus.OK) data = b"Dummy index file\r\n" with open(os.path.join(self.tempdir_name, 'index.html'), 'wb') as f: @@ -530,6 +739,8 @@ def test_path_without_leading_slash(self): self.check_status_and_reason(response, HTTPStatus.OK) response = self.request(self.tempdir_name) self.check_status_and_reason(response, HTTPStatus.MOVED_PERMANENTLY) + self.assertEqual(response.getheader("Location"), + self.tempdir_name + "/") response = self.request(self.tempdir_name + '/?hi=2') self.check_status_and_reason(response, HTTPStatus.OK) response = self.request(self.tempdir_name + '?hi=1') @@ -537,27 +748,6 @@ def test_path_without_leading_slash(self): self.assertEqual(response.getheader("Location"), self.tempdir_name + "/?hi=1") - def test_html_escape_filename(self): - filename = '<test&>.txt' - fullpath = os.path.join(self.tempdir, filename) - - try: - open(fullpath, 'wb').close() - except OSError: - raise unittest.SkipTest('Can not create file %s on current file ' - 'system' % filename) - - try: - response = self.request(self.base_url + '/') - body = self.check_status_and_reason(response, HTTPStatus.OK) - enc = response.headers.get_content_charset() - finally: - os.unlink(fullpath) # avoid affecting test_undecodable_filename - - self.assertIsNotNone(enc) - html_text = '>%s<' % html.escape(filename, quote=False) - self.assertIn(html_text.encode(enc), body) - cgi_file1 = """\ #!%s @@ -569,14 +759,19 @@ def test_html_escape_filename(self): cgi_file2 = """\ #!%s -import cgi +import os +import sys +import urllib.parse print("Content-type: text/html") print() -form = cgi.FieldStorage() -print("%%s, %%s, %%s" %% (form.getfirst("spam"), form.getfirst("eggs"), - form.getfirst("bacon"))) +content_length = int(os.environ["CONTENT_LENGTH"]) +query_string = sys.stdin.buffer.read(content_length) +params = {key.decode("utf-8"): val.decode("utf-8") + for key, val in urllib.parse.parse_qsl(query_string)} + +print("%%s, %%s, %%s" %% (params["spam"], params["eggs"], params["bacon"])) """ cgi_file4 = """\ @@ -607,17 +802,40 @@ def test_html_escape_filename(self): print("</pre>") """ -@unittest.skipIf(not hasattr(os, '_exit'), - "TODO: RUSTPYTHON, run_cgi in http/server.py gets stuck as os._exit(127) doesn't currently kill forked processes") +cgi_file7 = """\ +#!%s +import os +import sys + +print("Content-type: text/plain") +print() + +content_length = int(os.environ["CONTENT_LENGTH"]) +body = sys.stdin.buffer.read(content_length) + +print(f"{content_length} {len(body)}") +""" + + @unittest.skipIf(hasattr(os, 'geteuid') and os.geteuid() == 0, "This test can't be run reliably as root (issue #13308).") +@requires_subprocess() class CGIHTTPServerTestCase(BaseTestCase): class request_handler(NoLogRequestHandler, CGIHTTPRequestHandler): - pass + _test_case_self = None # populated by each setUp() method call. + + def __init__(self, *args, **kwargs): + with self._test_case_self.assertWarnsRegex( + DeprecationWarning, + r'http\.server\.CGIHTTPRequestHandler'): + # This context also happens to catch and silence the + # threading DeprecationWarning from os.fork(). + super().__init__(*args, **kwargs) linesep = os.linesep.encode('ascii') def setUp(self): + self.request_handler._test_case_self = self # practical, but yuck. BaseTestCase.setUp(self) self.cwd = os.getcwd() self.parent_dir = tempfile.mkdtemp() @@ -637,12 +855,13 @@ def setUp(self): self.file3_path = None self.file4_path = None self.file5_path = None + self.file6_path = None + self.file7_path = None # The shebang line should be pure ASCII: use symlink if possible. # See issue #7668. self._pythonexe_symlink = None - # TODO: RUSTPYTHON; dl_nt not supported yet - if os_helper.can_symlink() and sys.platform != 'win32': + if os_helper.can_symlink(): self.pythonexe = os.path.join(self.parent_dir, 'python') self._pythonexe_symlink = support.PythonSymlink(self.pythonexe).__enter__() else: @@ -692,9 +911,15 @@ def setUp(self): file6.write(cgi_file6 % self.pythonexe) os.chmod(self.file6_path, 0o777) + self.file7_path = os.path.join(self.cgi_dir, 'file7.py') + with open(self.file7_path, 'w', encoding='utf-8') as file7: + file7.write(cgi_file7 % self.pythonexe) + os.chmod(self.file7_path, 0o777) + os.chdir(self.parent_dir) def tearDown(self): + self.request_handler._test_case_self = None try: os.chdir(self.cwd) if self._pythonexe_symlink: @@ -713,11 +938,16 @@ def tearDown(self): os.remove(self.file5_path) if self.file6_path: os.remove(self.file6_path) + if self.file7_path: + os.remove(self.file7_path) os.rmdir(self.cgi_child_dir) os.rmdir(self.cgi_dir) os.rmdir(self.cgi_dir_in_sub_dir) os.rmdir(self.sub_dir_2) os.rmdir(self.sub_dir_1) + # The 'gmon.out' file can be written in the current working + # directory if C-level code profiling with gprof is enabled. + os_helper.unlink(os.path.join(self.parent_dir, 'gmon.out')) os.rmdir(self.parent_dir) finally: BaseTestCase.tearDown(self) @@ -764,8 +994,7 @@ def test_url_collapse_path(self): msg='path = %r\nGot: %r\nWanted: %r' % (path, actual, expected)) - # TODO: RUSTPYTHON - @unittest.skipIf(sys.platform != 'win32', "TODO: RUSTPYTHON; works only on windows") + @unittest.expectedFailureIf(sys.platform != 'win32', 'TODO: RUSTPYTHON; AssertionError: Tuples differ: (b"", None, 200) != (b"Hello World\n", "text/html", <HTTPStatus.OK: 200>)') def test_headers_and_content(self): res = self.request('/cgi-bin/file1.py') self.assertEqual( @@ -776,9 +1005,7 @@ def test_issue19435(self): res = self.request('///////////nocgi.py/../cgi-bin/nothere.sh') self.assertEqual(res.status, HTTPStatus.NOT_FOUND) - # TODO: RUSTPYTHON - @unittest.skipIf(sys.platform != 'win32', "TODO: RUSTPYTHON; works only on windows") - @unittest.expectedFailure + @unittest.expectedFailureIf(sys.platform != 'win32', 'TODO: RUSTPYTHON; b"" != b"1, python, 123456\n"') def test_post(self): params = urllib.parse.urlencode( {'spam' : 1, 'eggs' : 'python', 'bacon' : 123456}) @@ -787,13 +1014,30 @@ def test_post(self): self.assertEqual(res.read(), b'1, python, 123456' + self.linesep) + @unittest.expectedFailureIf(sys.platform != 'win32', 'TODO: RUSTPYTHON; AssertionError: b"" != b"32768 32768\n"') + def test_large_content_length(self): + for w in range(15, 25): + size = 1 << w + body = b'X' * size + headers = {'Content-Length' : str(size)} + res = self.request('/cgi-bin/file7.py', 'POST', body, headers) + self.assertEqual(res.read(), b'%d %d' % (size, size) + self.linesep) + + @unittest.expectedFailureIf(sys.platform != 'win32', 'TODO: RUSTPYTHON; AssertionError: b"" != b"Hello World\n"') + def test_large_content_length_truncated(self): + with support.swap_attr(self.request_handler, 'timeout', 0.001): + for w in range(18, 65): + size = 1 << w + headers = {'Content-Length' : str(size)} + res = self.request('/cgi-bin/file1.py', 'POST', b'x', headers) + self.assertEqual(res.read(), b'Hello World' + self.linesep) + def test_invaliduri(self): res = self.request('/cgi-bin/invalid') res.read() self.assertEqual(res.status, HTTPStatus.NOT_FOUND) - # TODO: RUSTPYTHON - @unittest.skipIf(sys.platform != 'win32', "TODO: RUSTPYTHON; works only on windows") + @unittest.expectedFailureIf(sys.platform != 'win32', 'TODO: RUSTPYTHON; AssertionError: Tuples differ: (b"Hello World\n", "text/html", <HTTPStatus.OK: 200>) != (b"", None, 200)') def test_authorization(self): headers = {b'Authorization' : b'Basic ' + base64.b64encode(b'username:pass')} @@ -802,8 +1046,7 @@ def test_authorization(self): (b'Hello World' + self.linesep, 'text/html', HTTPStatus.OK), (res.read(), res.getheader('Content-type'), res.status)) - # TODO: RUSTPYTHON - @unittest.skipIf(sys.platform != 'win32', "TODO: RUSTPYTHON; works only on windows") + @unittest.expectedFailureIf(sys.platform != 'win32', 'TODO: RUSTPYTHON; AssertionError: Tuples differ: (b"Hello World\n", "text/html", <HTTPStatus.OK: 200>) != (b"", None, 200)') def test_no_leading_slash(self): # http://bugs.python.org/issue2254 res = self.request('cgi-bin/file1.py') @@ -811,8 +1054,7 @@ def test_no_leading_slash(self): (b'Hello World' + self.linesep, 'text/html', HTTPStatus.OK), (res.read(), res.getheader('Content-type'), res.status)) - # TODO: RUSTPYTHON - @unittest.skipIf(sys.platform != 'win32', "TODO: RUSTPYTHON; works only on windows") + @unittest.expectedFailureIf(sys.platform != 'win32', 'TODO: RUSTPYTHON; ValueError: signal only works in main thread') def test_os_environ_is_not_altered(self): signature = "Test CGI Server" os.environ['SERVER_SOFTWARE'] = signature @@ -822,32 +1064,28 @@ def test_os_environ_is_not_altered(self): (res.read(), res.getheader('Content-type'), res.status)) self.assertEqual(os.environ['SERVER_SOFTWARE'], signature) - # TODO: RUSTPYTHON - @unittest.skipIf(sys.platform != 'win32', "TODO: RUSTPYTHON; works only on windows") + @unittest.expectedFailureIf(sys.platform != 'win32', 'TODO: RUSTPYTHON; ValueError: signal only works in main thread') def test_urlquote_decoding_in_cgi_check(self): res = self.request('/cgi-bin%2ffile1.py') self.assertEqual( (b'Hello World' + self.linesep, 'text/html', HTTPStatus.OK), (res.read(), res.getheader('Content-type'), res.status)) - # TODO: RUSTPYTHON - @unittest.skipIf(sys.platform != 'win32', "TODO: RUSTPYTHON; works only on windows") + @unittest.expectedFailureIf(sys.platform != 'win32', 'TODO: RUSTPYTHON; AssertionError: Tuples differ: (b"Hello World\n", "text/html", <HTTPStatus.OK: 200>) != (b"", None, 200)') def test_nested_cgi_path_issue21323(self): res = self.request('/cgi-bin/child-dir/file3.py') self.assertEqual( (b'Hello World' + self.linesep, 'text/html', HTTPStatus.OK), (res.read(), res.getheader('Content-type'), res.status)) - # TODO: RUSTPYTHON - @unittest.skipIf(sys.platform != 'win32', "TODO: RUSTPYTHON; works only on windows") + @unittest.expectedFailureIf(sys.platform != 'win32', 'TODO: RUSTPYTHON; ValueError: signal only works in main thread') def test_query_with_multiple_question_mark(self): res = self.request('/cgi-bin/file4.py?a=b?c=d') self.assertEqual( (b'a=b?c=d' + self.linesep, 'text/html', HTTPStatus.OK), (res.read(), res.getheader('Content-type'), res.status)) - # TODO: RUSTPYTHON - @unittest.skipIf(sys.platform != 'win32', "TODO: RUSTPYTHON; works only on windows") + @unittest.expectedFailureIf(sys.platform != 'win32', 'TODO: RUSTPYTHON; AssertionError: Tuples differ: (b"k=aa%2F%2Fbb&//q//p//=//a//b//\n", "text/html", <HTTPStatus.OK: 200>) != (b"", None, 200)') def test_query_with_continuous_slashes(self): res = self.request('/cgi-bin/file4.py?k=aa%2F%2Fbb&//q//p//=//a//b//') self.assertEqual( @@ -855,8 +1093,7 @@ def test_query_with_continuous_slashes(self): 'text/html', HTTPStatus.OK), (res.read(), res.getheader('Content-type'), res.status)) - # TODO: RUSTPYTHON - @unittest.skipIf(sys.platform != 'win32', "TODO: RUSTPYTHON; works only on windows") + @unittest.expectedFailureIf(sys.platform != 'win32', 'TODO: RUSTPYTHON; Tuples differ: (b"", None, 200) != (b"Hello World\n", "text/html", <HTTPStatus.OK: 200>)') def test_cgi_path_in_sub_directories(self): try: CGIHTTPRequestHandler.cgi_directories.append('/sub/dir/cgi-bin') @@ -867,8 +1104,7 @@ def test_cgi_path_in_sub_directories(self): finally: CGIHTTPRequestHandler.cgi_directories.remove('/sub/dir/cgi-bin') - # TODO: RUSTPYTHON - @unittest.skipIf(sys.platform != 'win32', "TODO: RUSTPYTHON; works only on windows") + @unittest.expectedFailureIf(sys.platform != 'win32', 'TODO: RUSTPYTHON; AssertionError: b"HTTP_ACCEPT=text/html,text/plain" not found in b""') def test_accept(self): browser_accept = \ 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' @@ -929,7 +1165,7 @@ def numWrites(self): return len(self.datas) -class BaseHTTPRequestHandlerTestCase(unittest.TestCase): +class BaseHTTPRequestHandlerTestCase(unittest.TestCase, ExtraAssertions): """Test the functionality of the BaseHTTPServer. Test the support for the Expect 100-continue header. @@ -960,6 +1196,27 @@ def verify_http_server_response(self, response): match = self.HTTPResponseMatch.search(response) self.assertIsNotNone(match) + def test_unprintable_not_logged(self): + # We call the method from the class directly as our Socketless + # Handler subclass overrode it... nice for everything BUT this test. + self.handler.client_address = ('127.0.0.1', 1337) + log_message = BaseHTTPRequestHandler.log_message + with mock.patch.object(sys, 'stderr', StringIO()) as fake_stderr: + log_message(self.handler, '/foo') + log_message(self.handler, '/\033bar\000\033') + log_message(self.handler, '/spam %s.', 'a') + log_message(self.handler, '/spam %s.', '\033\x7f\x9f\xa0beans') + log_message(self.handler, '"GET /foo\\b"ar\007 HTTP/1.0"') + stderr = fake_stderr.getvalue() + self.assertNotIn('\033', stderr) # non-printable chars are caught. + self.assertNotIn('\000', stderr) # non-printable chars are caught. + lines = stderr.splitlines() + self.assertIn('/foo', lines[0]) + self.assertIn(r'/\x1bbar\x00\x1b', lines[1]) + self.assertIn('/spam a.', lines[2]) + self.assertIn('/spam \\x1b\\x7f\\x9f\xa0beans.', lines[3]) + self.assertIn(r'"GET /foo\\b"ar\x07 HTTP/1.0"', lines[4]) + def test_http_1_1(self): result = self.send_typical_request(b'GET / HTTP/1.1\r\n\r\n') self.verify_http_server_response(result[0]) @@ -996,7 +1253,7 @@ def test_extra_space(self): b'Host: dummy\r\n' b'\r\n' ) - self.assertTrue(result[0].startswith(b'HTTP/1.1 400 ')) + self.assertStartsWith(result[0], b'HTTP/1.1 400 ') self.verify_expected_headers(result[1:result.index(b'\r\n')]) self.assertFalse(self.handler.get_called) @@ -1110,7 +1367,7 @@ def test_request_length(self): # Issue #10714: huge request lines are discarded, to avoid Denial # of Service attacks. result = self.send_typical_request(b'GET ' + b'x' * 65537) - self.assertEqual(result[0], b'HTTP/1.1 414 Request-URI Too Long\r\n') + self.assertEqual(result[0], b'HTTP/1.1 414 URI Too Long\r\n') self.assertFalse(self.handler.get_called) self.assertIsInstance(self.handler.requestline, str) diff --git a/Lib/test/test_imaplib.py b/Lib/test/test_imaplib.py new file mode 100644 index 00000000000..9155a43a06e --- /dev/null +++ b/Lib/test/test_imaplib.py @@ -0,0 +1,1128 @@ +from test import support +from test.support import socket_helper + +from contextlib import contextmanager +import imaplib +import os.path +import socketserver +import time +import calendar +import threading +import re +import socket + +from test.support import verbose, run_with_tz, run_with_locale, cpython_only +from test.support import hashlib_helper +from test.support import threading_helper +import unittest +from unittest import mock +from datetime import datetime, timezone, timedelta +try: + import ssl +except ImportError: + ssl = None + +support.requires_working_socket(module=True) + +CERTFILE = os.path.join(os.path.dirname(__file__) or os.curdir, "certdata", "keycert3.pem") +CAFILE = os.path.join(os.path.dirname(__file__) or os.curdir, "certdata", "pycacert.pem") + + +class TestImaplib(unittest.TestCase): + + def test_Internaldate2tuple(self): + t0 = calendar.timegm((2000, 1, 1, 0, 0, 0, -1, -1, -1)) + tt = imaplib.Internaldate2tuple( + b'25 (INTERNALDATE "01-Jan-2000 00:00:00 +0000")') + self.assertEqual(time.mktime(tt), t0) + tt = imaplib.Internaldate2tuple( + b'25 (INTERNALDATE "01-Jan-2000 11:30:00 +1130")') + self.assertEqual(time.mktime(tt), t0) + tt = imaplib.Internaldate2tuple( + b'25 (INTERNALDATE "31-Dec-1999 12:30:00 -1130")') + self.assertEqual(time.mktime(tt), t0) + + @run_with_tz('MST+07MDT,M4.1.0,M10.5.0') + def test_Internaldate2tuple_issue10941(self): + self.assertNotEqual(imaplib.Internaldate2tuple( + b'25 (INTERNALDATE "02-Apr-2000 02:30:00 +0000")'), + imaplib.Internaldate2tuple( + b'25 (INTERNALDATE "02-Apr-2000 03:30:00 +0000")')) + + def timevalues(self): + return [2000000000, 2000000000.0, time.localtime(2000000000), + (2033, 5, 18, 5, 33, 20, -1, -1, -1), + (2033, 5, 18, 5, 33, 20, -1, -1, 1), + datetime.fromtimestamp(2000000000, + timezone(timedelta(0, 2 * 60 * 60))), + '"18-May-2033 05:33:20 +0200"'] + + @run_with_locale('LC_ALL', 'de_DE', 'fr_FR', '') + # DST rules included to work around quirk where the Gnu C library may not + # otherwise restore the previous time zone + @run_with_tz('STD-1DST,M3.2.0,M11.1.0') + def test_Time2Internaldate(self): + expected = '"18-May-2033 05:33:20 +0200"' + + for t in self.timevalues(): + internal = imaplib.Time2Internaldate(t) + self.assertEqual(internal, expected) + + def test_that_Time2Internaldate_returns_a_result(self): + # Without tzset, we can check only that it successfully + # produces a result, not the correctness of the result itself, + # since the result depends on the timezone the machine is in. + for t in self.timevalues(): + imaplib.Time2Internaldate(t) + + @socket_helper.skip_if_tcp_blackhole + def test_imap4_host_default_value(self): + # Check whether the IMAP4_PORT is truly unavailable. + with socket.socket() as s: + try: + s.connect(('', imaplib.IMAP4_PORT)) + self.skipTest( + "Cannot run the test with local IMAP server running.") + except socket.error: + pass + + # This is the exception that should be raised. + expected_errnos = socket_helper.get_socket_conn_refused_errs() + with self.assertRaises(OSError) as cm: + imaplib.IMAP4() + self.assertIn(cm.exception.errno, expected_errnos) + + +if ssl: + class SecureTCPServer(socketserver.TCPServer): + + def get_request(self): + newsocket, fromaddr = self.socket.accept() + context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) + context.load_cert_chain(CERTFILE) + connstream = context.wrap_socket(newsocket, server_side=True) + return connstream, fromaddr + + IMAP4_SSL = imaplib.IMAP4_SSL + +else: + + class SecureTCPServer: + pass + + IMAP4_SSL = None + + +class SimpleIMAPHandler(socketserver.StreamRequestHandler): + timeout = support.LOOPBACK_TIMEOUT + continuation = None + capabilities = '' + + def setup(self): + super().setup() + self.server.is_selected = False + self.server.logged = None + + def _send(self, message): + if verbose: + print("SENT: %r" % message.strip()) + self.wfile.write(message) + + def _send_line(self, message): + self._send(message + b'\r\n') + + def _send_textline(self, message): + self._send_line(message.encode('ASCII')) + + def _send_tagged(self, tag, code, message): + self._send_textline(' '.join((tag, code, message))) + + def handle(self): + # Send a welcome message. + self._send_textline('* OK IMAP4rev1') + while 1: + # Gather up input until we receive a line terminator or we timeout. + # Accumulate read(1) because it's simpler to handle the differences + # between naked sockets and SSL sockets. + line = b'' + while 1: + try: + part = self.rfile.read(1) + if part == b'': + # Naked sockets return empty strings.. + return + line += part + except OSError: + # ..but SSLSockets raise exceptions. + return + if line.endswith(b'\r\n'): + break + + if verbose: + print('GOT: %r' % line.strip()) + if self.continuation: + try: + self.continuation.send(line) + except StopIteration: + self.continuation = None + continue + splitline = line.decode('ASCII').split() + tag = splitline[0] + cmd = splitline[1] + args = splitline[2:] + + if hasattr(self, 'cmd_' + cmd): + continuation = getattr(self, 'cmd_' + cmd)(tag, args) + if continuation: + self.continuation = continuation + next(continuation) + else: + self._send_tagged(tag, 'BAD', cmd + ' unknown') + + def cmd_CAPABILITY(self, tag, args): + caps = ('IMAP4rev1 ' + self.capabilities + if self.capabilities + else 'IMAP4rev1') + self._send_textline('* CAPABILITY ' + caps) + self._send_tagged(tag, 'OK', 'CAPABILITY completed') + + def cmd_LOGOUT(self, tag, args): + self.server.logged = None + self._send_textline('* BYE IMAP4ref1 Server logging out') + self._send_tagged(tag, 'OK', 'LOGOUT completed') + + def cmd_LOGIN(self, tag, args): + self.server.logged = args[0] + self._send_tagged(tag, 'OK', 'LOGIN completed') + + def cmd_SELECT(self, tag, args): + self.server.is_selected = True + self._send_line(b'* 2 EXISTS') + self._send_tagged(tag, 'OK', '[READ-WRITE] SELECT completed.') + + def cmd_UNSELECT(self, tag, args): + if self.server.is_selected: + self.server.is_selected = False + self._send_tagged(tag, 'OK', 'Returned to authenticated state. (Success)') + else: + self._send_tagged(tag, 'BAD', 'No mailbox selected') + + +class IdleCmdDenyHandler(SimpleIMAPHandler): + capabilities = 'IDLE' + def cmd_IDLE(self, tag, args): + self._send_tagged(tag, 'NO', 'IDLE is not allowed at this time') + + +class IdleCmdHandler(SimpleIMAPHandler): + capabilities = 'IDLE' + def cmd_IDLE(self, tag, args): + # pre-idle-continuation response + self._send_line(b'* 0 EXISTS') + self._send_textline('+ idling') + # simple response + self._send_line(b'* 2 EXISTS') + # complex response: fragmented data due to literal string + self._send_line(b'* 1 FETCH (BODY[HEADER.FIELDS (DATE)] {41}') + self._send(b'Date: Fri, 06 Dec 2024 06:00:00 +0000\r\n\r\n') + self._send_line(b')') + # simple response following a fragmented one + self._send_line(b'* 3 EXISTS') + # response arriving later + time.sleep(1) + self._send_line(b'* 1 RECENT') + r = yield + if r == b'DONE\r\n': + self._send_line(b'* 9 RECENT') + self._send_tagged(tag, 'OK', 'Idle completed') + else: + self._send_tagged(tag, 'BAD', 'Expected DONE') + + +class IdleCmdDelayedPacketHandler(SimpleIMAPHandler): + capabilities = 'IDLE' + def cmd_IDLE(self, tag, args): + self._send_textline('+ idling') + # response line spanning multiple packets, the last one delayed + self._send(b'* 1 EX') + time.sleep(0.2) + self._send(b'IS') + time.sleep(1) + self._send(b'TS\r\n') + r = yield + if r == b'DONE\r\n': + self._send_tagged(tag, 'OK', 'Idle completed') + else: + self._send_tagged(tag, 'BAD', 'Expected DONE') + + +class AuthHandler_CRAM_MD5(SimpleIMAPHandler): + capabilities = 'LOGINDISABLED AUTH=CRAM-MD5' + def cmd_AUTHENTICATE(self, tag, args): + self._send_textline('+ PDE4OTYuNjk3MTcwOTUyQHBvc3RvZmZpY2Uucm' + 'VzdG9uLm1jaS5uZXQ=') + r = yield + if (r == b'dGltIGYxY2E2YmU0NjRiOWVmYT' + b'FjY2E2ZmZkNmNmMmQ5ZjMy\r\n'): + self._send_tagged(tag, 'OK', 'CRAM-MD5 successful') + else: + self._send_tagged(tag, 'NO', 'No access') + + +class NewIMAPTestsMixin: + client = None + + def _setup(self, imap_handler, connect=True): + """ + Sets up imap_handler for tests. imap_handler should inherit from either: + - SimpleIMAPHandler - for testing IMAP commands, + - socketserver.StreamRequestHandler - if raw access to stream is needed. + Returns (client, server). + """ + class TestTCPServer(self.server_class): + def handle_error(self, request, client_address): + """ + End request and raise the error if one occurs. + """ + self.close_request(request) + self.server_close() + raise + + self.addCleanup(self._cleanup) + self.server = self.server_class((socket_helper.HOST, 0), imap_handler) + self.thread = threading.Thread( + name=self._testMethodName+'-server', + target=self.server.serve_forever, + # Short poll interval to make the test finish quickly. + # Time between requests is short enough that we won't wake + # up spuriously too many times. + kwargs={'poll_interval': 0.01}) + self.thread.daemon = True # In case this function raises. + self.thread.start() + + if connect: + self.client = self.imap_class(*self.server.server_address) + + return self.client, self.server + + def _cleanup(self): + """ + Cleans up the test server. This method should not be called manually, + it is added to the cleanup queue in the _setup method already. + """ + # if logout was called already we'd raise an exception trying to + # shutdown the client once again + if self.client is not None and self.client.state != 'LOGOUT': + self.client.shutdown() + # cleanup the server + self.server.shutdown() + self.server.server_close() + threading_helper.join_thread(self.thread) + # Explicitly clear the attribute to prevent dangling thread + self.thread = None + + def test_EOF_without_complete_welcome_message(self): + # http://bugs.python.org/issue5949 + class EOFHandler(socketserver.StreamRequestHandler): + def handle(self): + self.wfile.write(b'* OK') + _, server = self._setup(EOFHandler, connect=False) + self.assertRaises(imaplib.IMAP4.abort, self.imap_class, + *server.server_address) + + def test_line_termination(self): + class BadNewlineHandler(SimpleIMAPHandler): + def cmd_CAPABILITY(self, tag, args): + self._send(b'* CAPABILITY IMAP4rev1 AUTH\n') + self._send_tagged(tag, 'OK', 'CAPABILITY completed') + _, server = self._setup(BadNewlineHandler, connect=False) + self.assertRaises(imaplib.IMAP4.abort, self.imap_class, + *server.server_address) + + def test_enable_raises_error_if_not_AUTH(self): + class EnableHandler(SimpleIMAPHandler): + capabilities = 'AUTH ENABLE UTF8=ACCEPT' + client, _ = self._setup(EnableHandler) + self.assertFalse(client.utf8_enabled) + with self.assertRaisesRegex(imaplib.IMAP4.error, 'ENABLE.*NONAUTH'): + client.enable('foo') + self.assertFalse(client.utf8_enabled) + + def test_enable_raises_error_if_no_capability(self): + client, _ = self._setup(SimpleIMAPHandler) + with self.assertRaisesRegex(imaplib.IMAP4.error, + 'does not support ENABLE'): + client.enable('foo') + + def test_enable_UTF8_raises_error_if_not_supported(self): + client, _ = self._setup(SimpleIMAPHandler) + typ, data = client.login('user', 'pass') + self.assertEqual(typ, 'OK') + with self.assertRaisesRegex(imaplib.IMAP4.error, + 'does not support ENABLE'): + client.enable('UTF8=ACCEPT') + + def test_enable_UTF8_True_append(self): + class UTF8AppendServer(SimpleIMAPHandler): + capabilities = 'ENABLE UTF8=ACCEPT' + def cmd_ENABLE(self, tag, args): + self._send_tagged(tag, 'OK', 'ENABLE successful') + def cmd_AUTHENTICATE(self, tag, args): + self._send_textline('+') + self.server.response = yield + self._send_tagged(tag, 'OK', 'FAKEAUTH successful') + def cmd_APPEND(self, tag, args): + self._send_textline('+') + self.server.response = args + literal = yield + self.server.response.append(literal) + literal = yield + self.server.response.append(literal) + self._send_tagged(tag, 'OK', 'okay') + client, server = self._setup(UTF8AppendServer) + self.assertEqual(client._encoding, 'ascii') + code, _ = client.authenticate('MYAUTH', lambda x: b'fake') + self.assertEqual(code, 'OK') + self.assertEqual(server.response, b'ZmFrZQ==\r\n') # b64 encoded 'fake' + code, _ = client.enable('UTF8=ACCEPT') + self.assertEqual(code, 'OK') + self.assertEqual(client._encoding, 'utf-8') + msg_string = 'Subject: üñí©öðé' + typ, data = client.append( + None, None, None, (msg_string + '\n').encode('utf-8')) + self.assertEqual(typ, 'OK') + self.assertEqual(server.response, + ['INBOX', 'UTF8', + '(~{25}', ('%s\r\n' % msg_string).encode('utf-8'), + b')\r\n' ]) + + def test_search_disallows_charset_in_utf8_mode(self): + class UTF8Server(SimpleIMAPHandler): + capabilities = 'AUTH ENABLE UTF8=ACCEPT' + def cmd_ENABLE(self, tag, args): + self._send_tagged(tag, 'OK', 'ENABLE successful') + def cmd_AUTHENTICATE(self, tag, args): + self._send_textline('+') + self.server.response = yield + self._send_tagged(tag, 'OK', 'FAKEAUTH successful') + client, _ = self._setup(UTF8Server) + typ, _ = client.authenticate('MYAUTH', lambda x: b'fake') + self.assertEqual(typ, 'OK') + typ, _ = client.enable('UTF8=ACCEPT') + self.assertEqual(typ, 'OK') + self.assertTrue(client.utf8_enabled) + with self.assertRaisesRegex(imaplib.IMAP4.error, 'charset.*UTF8'): + client.search('foo', 'bar') + + def test_bad_auth_name(self): + class MyServer(SimpleIMAPHandler): + def cmd_AUTHENTICATE(self, tag, args): + self._send_tagged(tag, 'NO', + 'unrecognized authentication type {}'.format(args[0])) + client, _ = self._setup(MyServer) + with self.assertRaisesRegex(imaplib.IMAP4.error, + 'unrecognized authentication type METHOD'): + client.authenticate('METHOD', lambda: 1) + + def test_invalid_authentication(self): + class MyServer(SimpleIMAPHandler): + def cmd_AUTHENTICATE(self, tag, args): + self._send_textline('+') + self.response = yield + self._send_tagged(tag, 'NO', '[AUTHENTICATIONFAILED] invalid') + client, _ = self._setup(MyServer) + with self.assertRaisesRegex(imaplib.IMAP4.error, + r'\[AUTHENTICATIONFAILED\] invalid'): + client.authenticate('MYAUTH', lambda x: b'fake') + + def test_valid_authentication_bytes(self): + class MyServer(SimpleIMAPHandler): + def cmd_AUTHENTICATE(self, tag, args): + self._send_textline('+') + self.server.response = yield + self._send_tagged(tag, 'OK', 'FAKEAUTH successful') + client, server = self._setup(MyServer) + code, _ = client.authenticate('MYAUTH', lambda x: b'fake') + self.assertEqual(code, 'OK') + self.assertEqual(server.response, b'ZmFrZQ==\r\n') # b64 encoded 'fake' + + def test_valid_authentication_plain_text(self): + class MyServer(SimpleIMAPHandler): + def cmd_AUTHENTICATE(self, tag, args): + self._send_textline('+') + self.server.response = yield + self._send_tagged(tag, 'OK', 'FAKEAUTH successful') + client, server = self._setup(MyServer) + code, _ = client.authenticate('MYAUTH', lambda x: 'fake') + self.assertEqual(code, 'OK') + self.assertEqual(server.response, b'ZmFrZQ==\r\n') # b64 encoded 'fake' + + @hashlib_helper.requires_hashdigest('md5', openssl=True) + def test_login_cram_md5_bytes(self): + client, _ = self._setup(AuthHandler_CRAM_MD5) + self.assertIn('AUTH=CRAM-MD5', client.capabilities) + ret, _ = client.login_cram_md5("tim", b"tanstaaftanstaaf") + self.assertEqual(ret, "OK") + + @hashlib_helper.requires_hashdigest('md5', openssl=True) + def test_login_cram_md5_plain_text(self): + client, _ = self._setup(AuthHandler_CRAM_MD5) + self.assertIn('AUTH=CRAM-MD5', client.capabilities) + ret, _ = client.login_cram_md5("tim", "tanstaaftanstaaf") + self.assertEqual(ret, "OK") + + def test_login_cram_md5_blocked(self): + def side_effect(*a, **kw): + raise ValueError + + client, _ = self._setup(AuthHandler_CRAM_MD5) + self.assertIn('AUTH=CRAM-MD5', client.capabilities) + msg = re.escape("CRAM-MD5 authentication is not supported") + with ( + mock.patch("hmac.HMAC", side_effect=side_effect), + self.assertRaisesRegex(imaplib.IMAP4.error, msg) + ): + client.login_cram_md5("tim", b"tanstaaftanstaaf") + + def test_aborted_authentication(self): + class MyServer(SimpleIMAPHandler): + def cmd_AUTHENTICATE(self, tag, args): + self._send_textline('+') + self.response = yield + if self.response == b'*\r\n': + self._send_tagged( + tag, + 'NO', + '[AUTHENTICATIONFAILED] aborted') + else: + self._send_tagged(tag, 'OK', 'MYAUTH successful') + client, _ = self._setup(MyServer) + with self.assertRaisesRegex(imaplib.IMAP4.error, + r'\[AUTHENTICATIONFAILED\] aborted'): + client.authenticate('MYAUTH', lambda x: None) + + @mock.patch('imaplib._MAXLINE', 10) + def test_linetoolong(self): + class TooLongHandler(SimpleIMAPHandler): + def handle(self): + # send response line longer than the limit set in the next line + self.wfile.write(b'* OK ' + 11 * b'x' + b'\r\n') + _, server = self._setup(TooLongHandler, connect=False) + with self.assertRaisesRegex(imaplib.IMAP4.error, + 'got more than 10 bytes'): + self.imap_class(*server.server_address) + + def test_simple_with_statement(self): + _, server = self._setup(SimpleIMAPHandler, connect=False) + with self.imap_class(*server.server_address): + pass + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'socket' object has no attribute 'timeout'. Did you mean: 'gettimeout'? + def test_imaplib_timeout_test(self): + _, server = self._setup(SimpleIMAPHandler, connect=False) + with self.imap_class(*server.server_address, timeout=None) as client: + self.assertEqual(client.sock.timeout, None) + with self.imap_class(*server.server_address, timeout=support.LOOPBACK_TIMEOUT) as client: + self.assertEqual(client.sock.timeout, support.LOOPBACK_TIMEOUT) + with self.assertRaises(ValueError): + self.imap_class(*server.server_address, timeout=0) + + def test_imaplib_timeout_functionality_test(self): + class TimeoutHandler(SimpleIMAPHandler): + def handle(self): + time.sleep(1) + SimpleIMAPHandler.handle(self) + + _, server = self._setup(TimeoutHandler) + addr = server.server_address[1] + with self.assertRaises(TimeoutError): + client = self.imap_class("localhost", addr, timeout=0.001) + + def test_with_statement(self): + _, server = self._setup(SimpleIMAPHandler, connect=False) + with self.imap_class(*server.server_address) as imap: + imap.login('user', 'pass') + self.assertEqual(server.logged, 'user') + self.assertIsNone(server.logged) + + def test_with_statement_logout(self): + # It is legal to log out explicitly inside the with block + _, server = self._setup(SimpleIMAPHandler, connect=False) + with self.imap_class(*server.server_address) as imap: + imap.login('user', 'pass') + self.assertEqual(server.logged, 'user') + imap.logout() + self.assertIsNone(server.logged) + self.assertIsNone(server.logged) + + # command tests + + def test_idle_capability(self): + client, _ = self._setup(SimpleIMAPHandler) + with self.assertRaisesRegex(imaplib.IMAP4.error, + 'does not support IMAP4 IDLE'): + with client.idle(): + pass + + def test_idle_denied(self): + client, _ = self._setup(IdleCmdDenyHandler) + client.login('user', 'pass') + with self.assertRaises(imaplib.IMAP4.error): + with client.idle() as idler: + pass + + def test_idle_iter(self): + client, _ = self._setup(IdleCmdHandler) + client.login('user', 'pass') + with client.idle() as idler: + # iteration should include response between 'IDLE' & '+ idling' + response = next(idler) + self.assertEqual(response, ('EXISTS', [b'0'])) + # iteration should produce responses + response = next(idler) + self.assertEqual(response, ('EXISTS', [b'2'])) + # fragmented response (with literal string) should arrive whole + expected_fetch_data = [ + (b'1 (BODY[HEADER.FIELDS (DATE)] {41}', + b'Date: Fri, 06 Dec 2024 06:00:00 +0000\r\n\r\n'), + b')'] + typ, data = next(idler) + self.assertEqual(typ, 'FETCH') + self.assertEqual(data, expected_fetch_data) + # response after a fragmented one should arrive separately + response = next(idler) + self.assertEqual(response, ('EXISTS', [b'3'])) + # iteration should have consumed untagged responses + _, data = client.response('EXISTS') + self.assertEqual(data, [None]) + # responses not iterated should be available after idle + _, data = client.response('RECENT') + self.assertEqual(data[0], b'1') + # responses received after 'DONE' should be available after idle + self.assertEqual(data[1], b'9') + + def test_idle_burst(self): + client, _ = self._setup(IdleCmdHandler) + client.login('user', 'pass') + # burst() should yield immediately available responses + with client.idle() as idler: + batch = list(idler.burst()) + self.assertEqual(len(batch), 4) + # burst() should not have consumed later responses + _, data = client.response('RECENT') + self.assertEqual(data, [b'1', b'9']) + + def test_idle_delayed_packet(self): + client, _ = self._setup(IdleCmdDelayedPacketHandler) + client.login('user', 'pass') + # If our readline() implementation fails to preserve line fragments + # when idle timeouts trigger, a response spanning delayed packets + # can be corrupted, leaving the protocol stream in a bad state. + try: + with client.idle(0.5) as idler: + self.assertRaises(StopIteration, next, idler) + except client.abort as err: + self.fail('multi-packet response was corrupted by idle timeout') + + def test_login(self): + client, _ = self._setup(SimpleIMAPHandler) + typ, data = client.login('user', 'pass') + self.assertEqual(typ, 'OK') + self.assertEqual(data[0], b'LOGIN completed') + self.assertEqual(client.state, 'AUTH') + + def test_logout(self): + client, _ = self._setup(SimpleIMAPHandler) + typ, data = client.login('user', 'pass') + self.assertEqual(typ, 'OK') + self.assertEqual(data[0], b'LOGIN completed') + typ, data = client.logout() + self.assertEqual(typ, 'BYE', (typ, data)) + self.assertEqual(data[0], b'IMAP4ref1 Server logging out', (typ, data)) + self.assertEqual(client.state, 'LOGOUT') + + def test_lsub(self): + class LsubCmd(SimpleIMAPHandler): + def cmd_LSUB(self, tag, args): + self._send_textline('* LSUB () "." directoryA') + return self._send_tagged(tag, 'OK', 'LSUB completed') + client, _ = self._setup(LsubCmd) + client.login('user', 'pass') + typ, data = client.lsub() + self.assertEqual(typ, 'OK') + self.assertEqual(data[0], b'() "." directoryA') + + def test_unselect(self): + client, _ = self._setup(SimpleIMAPHandler) + client.login('user', 'pass') + typ, data = client.select() + self.assertEqual(typ, 'OK') + self.assertEqual(data[0], b'2') + + typ, data = client.unselect() + self.assertEqual(typ, 'OK') + self.assertEqual(data[0], b'Returned to authenticated state. (Success)') + self.assertEqual(client.state, 'AUTH') + + # property tests + + def test_file_property_should_not_be_accessed(self): + client, _ = self._setup(SimpleIMAPHandler) + # the 'file' property replaced a private attribute that is now unsafe + with self.assertWarns(RuntimeWarning): + client.file + + +class NewIMAPTests(NewIMAPTestsMixin, unittest.TestCase): + imap_class = imaplib.IMAP4 + server_class = socketserver.TCPServer + + +@unittest.skipUnless(ssl, "SSL not available") +class NewIMAPSSLTests(NewIMAPTestsMixin, unittest.TestCase): + imap_class = IMAP4_SSL + server_class = SecureTCPServer + + def test_ssl_raises(self): + ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + self.assertEqual(ssl_context.verify_mode, ssl.CERT_REQUIRED) + self.assertEqual(ssl_context.check_hostname, True) + ssl_context.load_verify_locations(CAFILE) + + # Allow for flexible libssl error messages. + regex = re.compile(r"""( + IP address mismatch, certificate is not valid for '127.0.0.1' # OpenSSL + | + CERTIFICATE_VERIFY_FAILED # AWS-LC + )""", re.X) + with self.assertRaisesRegex(ssl.CertificateError, regex): + _, server = self._setup(SimpleIMAPHandler, connect=False) + client = self.imap_class(*server.server_address, + ssl_context=ssl_context) + client.shutdown() + + def test_ssl_verified(self): + ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + ssl_context.load_verify_locations(CAFILE) + + _, server = self._setup(SimpleIMAPHandler, connect=False) + client = self.imap_class("localhost", server.server_address[1], + ssl_context=ssl_context) + client.shutdown() + +class ThreadedNetworkedTests(unittest.TestCase): + server_class = socketserver.TCPServer + imap_class = imaplib.IMAP4 + + def make_server(self, addr, hdlr): + + class MyServer(self.server_class): + def handle_error(self, request, client_address): + self.close_request(request) + self.server_close() + raise + + if verbose: + print("creating server") + server = MyServer(addr, hdlr) + self.assertEqual(server.server_address, server.socket.getsockname()) + + if verbose: + print("server created") + print("ADDR =", addr) + print("CLASS =", self.server_class) + print("HDLR =", server.RequestHandlerClass) + + t = threading.Thread( + name='%s serving' % self.server_class, + target=server.serve_forever, + # Short poll interval to make the test finish quickly. + # Time between requests is short enough that we won't wake + # up spuriously too many times. + kwargs={'poll_interval': 0.01}) + t.daemon = True # In case this function raises. + t.start() + if verbose: + print("server running") + return server, t + + def reap_server(self, server, thread): + if verbose: + print("waiting for server") + server.shutdown() + server.server_close() + thread.join() + if verbose: + print("done") + + @contextmanager + def reaped_server(self, hdlr): + server, thread = self.make_server((socket_helper.HOST, 0), hdlr) + try: + yield server + finally: + self.reap_server(server, thread) + + @contextmanager + def reaped_pair(self, hdlr): + with self.reaped_server(hdlr) as server: + client = self.imap_class(*server.server_address) + try: + yield server, client + finally: + client.logout() + + @threading_helper.reap_threads + def test_connect(self): + with self.reaped_server(SimpleIMAPHandler) as server: + client = self.imap_class(*server.server_address) + client.shutdown() + + @threading_helper.reap_threads + def test_bracket_flags(self): + + # This violates RFC 3501, which disallows ']' characters in tag names, + # but imaplib has allowed producing such tags forever, other programs + # also produce them (eg: OtherInbox's Organizer app as of 20140716), + # and Gmail, for example, accepts them and produces them. So we + # support them. See issue #21815. + + class BracketFlagHandler(SimpleIMAPHandler): + + def handle(self): + self.flags = ['Answered', 'Flagged', 'Deleted', 'Seen', 'Draft'] + super().handle() + + def cmd_AUTHENTICATE(self, tag, args): + self._send_textline('+') + self.server.response = yield + self._send_tagged(tag, 'OK', 'FAKEAUTH successful') + + def cmd_SELECT(self, tag, args): + flag_msg = ' \\'.join(self.flags) + self._send_line(('* FLAGS (%s)' % flag_msg).encode('ascii')) + self._send_line(b'* 2 EXISTS') + self._send_line(b'* 0 RECENT') + msg = ('* OK [PERMANENTFLAGS %s \\*)] Flags permitted.' + % flag_msg) + self._send_line(msg.encode('ascii')) + self._send_tagged(tag, 'OK', '[READ-WRITE] SELECT completed.') + + def cmd_STORE(self, tag, args): + new_flags = args[2].strip('(').strip(')').split() + self.flags.extend(new_flags) + flags_msg = '(FLAGS (%s))' % ' \\'.join(self.flags) + msg = '* %s FETCH %s' % (args[0], flags_msg) + self._send_line(msg.encode('ascii')) + self._send_tagged(tag, 'OK', 'STORE completed.') + + with self.reaped_pair(BracketFlagHandler) as (server, client): + code, data = client.authenticate('MYAUTH', lambda x: b'fake') + self.assertEqual(code, 'OK') + self.assertEqual(server.response, b'ZmFrZQ==\r\n') + client.select('test') + typ, [data] = client.store(b'1', "+FLAGS", "[test]") + self.assertIn(b'[test]', data) + client.select('test') + typ, [data] = client.response('PERMANENTFLAGS') + self.assertIn(b'[test]', data) + + @threading_helper.reap_threads + def test_issue5949(self): + + class EOFHandler(socketserver.StreamRequestHandler): + def handle(self): + # EOF without sending a complete welcome message. + self.wfile.write(b'* OK') + + with self.reaped_server(EOFHandler) as server: + self.assertRaises(imaplib.IMAP4.abort, + self.imap_class, *server.server_address) + + @threading_helper.reap_threads + def test_line_termination(self): + + class BadNewlineHandler(SimpleIMAPHandler): + + def cmd_CAPABILITY(self, tag, args): + self._send(b'* CAPABILITY IMAP4rev1 AUTH\n') + self._send_tagged(tag, 'OK', 'CAPABILITY completed') + + with self.reaped_server(BadNewlineHandler) as server: + self.assertRaises(imaplib.IMAP4.abort, + self.imap_class, *server.server_address) + + class UTF8Server(SimpleIMAPHandler): + capabilities = 'AUTH ENABLE UTF8=ACCEPT' + + def cmd_ENABLE(self, tag, args): + self._send_tagged(tag, 'OK', 'ENABLE successful') + + def cmd_AUTHENTICATE(self, tag, args): + self._send_textline('+') + self.server.response = yield + self._send_tagged(tag, 'OK', 'FAKEAUTH successful') + + @threading_helper.reap_threads + def test_enable_raises_error_if_not_AUTH(self): + with self.reaped_pair(self.UTF8Server) as (server, client): + self.assertFalse(client.utf8_enabled) + self.assertRaises(imaplib.IMAP4.error, client.enable, 'foo') + self.assertFalse(client.utf8_enabled) + + # XXX Also need a test that enable after SELECT raises an error. + + @threading_helper.reap_threads + def test_enable_raises_error_if_no_capability(self): + class NoEnableServer(self.UTF8Server): + capabilities = 'AUTH' + with self.reaped_pair(NoEnableServer) as (server, client): + self.assertRaises(imaplib.IMAP4.error, client.enable, 'foo') + + @threading_helper.reap_threads + def test_enable_UTF8_raises_error_if_not_supported(self): + class NonUTF8Server(SimpleIMAPHandler): + pass + with self.assertRaises(imaplib.IMAP4.error): + with self.reaped_pair(NonUTF8Server) as (server, client): + typ, data = client.login('user', 'pass') + self.assertEqual(typ, 'OK') + client.enable('UTF8=ACCEPT') + + @threading_helper.reap_threads + def test_enable_UTF8_True_append(self): + + class UTF8AppendServer(self.UTF8Server): + def cmd_APPEND(self, tag, args): + self._send_textline('+') + self.server.response = args + literal = yield + self.server.response.append(literal) + literal = yield + self.server.response.append(literal) + self._send_tagged(tag, 'OK', 'okay') + + with self.reaped_pair(UTF8AppendServer) as (server, client): + self.assertEqual(client._encoding, 'ascii') + code, _ = client.authenticate('MYAUTH', lambda x: b'fake') + self.assertEqual(code, 'OK') + self.assertEqual(server.response, + b'ZmFrZQ==\r\n') # b64 encoded 'fake' + code, _ = client.enable('UTF8=ACCEPT') + self.assertEqual(code, 'OK') + self.assertEqual(client._encoding, 'utf-8') + msg_string = 'Subject: üñí©öðé' + typ, data = client.append( + None, None, None, (msg_string + '\n').encode('utf-8')) + self.assertEqual(typ, 'OK') + self.assertEqual(server.response, + ['INBOX', 'UTF8', + '(~{25}', ('%s\r\n' % msg_string).encode('utf-8'), + b')\r\n' ]) + + # XXX also need a test that makes sure that the Literal and Untagged_status + # regexes uses unicode in UTF8 mode instead of the default ASCII. + + @threading_helper.reap_threads + def test_search_disallows_charset_in_utf8_mode(self): + with self.reaped_pair(self.UTF8Server) as (server, client): + typ, _ = client.authenticate('MYAUTH', lambda x: b'fake') + self.assertEqual(typ, 'OK') + typ, _ = client.enable('UTF8=ACCEPT') + self.assertEqual(typ, 'OK') + self.assertTrue(client.utf8_enabled) + self.assertRaises(imaplib.IMAP4.error, client.search, 'foo', 'bar') + + @threading_helper.reap_threads + def test_bad_auth_name(self): + + class MyServer(SimpleIMAPHandler): + + def cmd_AUTHENTICATE(self, tag, args): + self._send_tagged(tag, 'NO', 'unrecognized authentication ' + 'type {}'.format(args[0])) + + with self.reaped_pair(MyServer) as (server, client): + with self.assertRaises(imaplib.IMAP4.error): + client.authenticate('METHOD', lambda: 1) + + @threading_helper.reap_threads + def test_invalid_authentication(self): + + class MyServer(SimpleIMAPHandler): + + def cmd_AUTHENTICATE(self, tag, args): + self._send_textline('+') + self.response = yield + self._send_tagged(tag, 'NO', '[AUTHENTICATIONFAILED] invalid') + + with self.reaped_pair(MyServer) as (server, client): + with self.assertRaises(imaplib.IMAP4.error): + code, data = client.authenticate('MYAUTH', lambda x: b'fake') + + @threading_helper.reap_threads + def test_valid_authentication(self): + + class MyServer(SimpleIMAPHandler): + + def cmd_AUTHENTICATE(self, tag, args): + self._send_textline('+') + self.server.response = yield + self._send_tagged(tag, 'OK', 'FAKEAUTH successful') + + with self.reaped_pair(MyServer) as (server, client): + code, data = client.authenticate('MYAUTH', lambda x: b'fake') + self.assertEqual(code, 'OK') + self.assertEqual(server.response, + b'ZmFrZQ==\r\n') # b64 encoded 'fake' + + with self.reaped_pair(MyServer) as (server, client): + code, data = client.authenticate('MYAUTH', lambda x: 'fake') + self.assertEqual(code, 'OK') + self.assertEqual(server.response, + b'ZmFrZQ==\r\n') # b64 encoded 'fake' + + @threading_helper.reap_threads + @hashlib_helper.requires_hashdigest('md5', openssl=True) + def test_login_cram_md5(self): + + class AuthHandler(SimpleIMAPHandler): + + capabilities = 'LOGINDISABLED AUTH=CRAM-MD5' + + def cmd_AUTHENTICATE(self, tag, args): + self._send_textline('+ PDE4OTYuNjk3MTcwOTUyQHBvc3RvZmZpY2Uucm' + 'VzdG9uLm1jaS5uZXQ=') + r = yield + if (r == b'dGltIGYxY2E2YmU0NjRiOWVmYT' + b'FjY2E2ZmZkNmNmMmQ5ZjMy\r\n'): + self._send_tagged(tag, 'OK', 'CRAM-MD5 successful') + else: + self._send_tagged(tag, 'NO', 'No access') + + with self.reaped_pair(AuthHandler) as (server, client): + self.assertTrue('AUTH=CRAM-MD5' in client.capabilities) + ret, data = client.login_cram_md5("tim", "tanstaaftanstaaf") + self.assertEqual(ret, "OK") + + with self.reaped_pair(AuthHandler) as (server, client): + self.assertTrue('AUTH=CRAM-MD5' in client.capabilities) + ret, data = client.login_cram_md5("tim", b"tanstaaftanstaaf") + self.assertEqual(ret, "OK") + + + @threading_helper.reap_threads + def test_aborted_authentication(self): + + class MyServer(SimpleIMAPHandler): + + def cmd_AUTHENTICATE(self, tag, args): + self._send_textline('+') + self.response = yield + + if self.response == b'*\r\n': + self._send_tagged(tag, 'NO', '[AUTHENTICATIONFAILED] aborted') + else: + self._send_tagged(tag, 'OK', 'MYAUTH successful') + + with self.reaped_pair(MyServer) as (server, client): + with self.assertRaises(imaplib.IMAP4.error): + code, data = client.authenticate('MYAUTH', lambda x: None) + + + def test_linetoolong(self): + class TooLongHandler(SimpleIMAPHandler): + def handle(self): + # Send a very long response line + self.wfile.write(b'* OK ' + imaplib._MAXLINE * b'x' + b'\r\n') + + with self.reaped_server(TooLongHandler) as server: + self.assertRaises(imaplib.IMAP4.error, + self.imap_class, *server.server_address) + + def test_truncated_large_literal(self): + size = 0 + class BadHandler(SimpleIMAPHandler): + def handle(self): + self._send_textline('* OK {%d}' % size) + self._send_textline('IMAP4rev1') + + for exponent in range(15, 64): + size = 1 << exponent + with self.subTest(f"size=2e{size}"): + with self.reaped_server(BadHandler) as server: + with self.assertRaises(imaplib.IMAP4.abort): + self.imap_class(*server.server_address) + + @threading_helper.reap_threads + def test_simple_with_statement(self): + # simplest call + with self.reaped_server(SimpleIMAPHandler) as server: + with self.imap_class(*server.server_address): + pass + + @threading_helper.reap_threads + def test_with_statement(self): + with self.reaped_server(SimpleIMAPHandler) as server: + with self.imap_class(*server.server_address) as imap: + imap.login('user', 'pass') + self.assertEqual(server.logged, 'user') + self.assertIsNone(server.logged) + + @threading_helper.reap_threads + def test_with_statement_logout(self): + # what happens if already logout in the block? + with self.reaped_server(SimpleIMAPHandler) as server: + with self.imap_class(*server.server_address) as imap: + imap.login('user', 'pass') + self.assertEqual(server.logged, 'user') + imap.logout() + self.assertIsNone(server.logged) + self.assertIsNone(server.logged) + + @threading_helper.reap_threads + @cpython_only + @unittest.skipUnless(__debug__, "Won't work if __debug__ is False") + def test_dump_ur(self): + # See: http://bugs.python.org/issue26543 + untagged_resp_dict = {'READ-WRITE': [b'']} + + with self.reaped_server(SimpleIMAPHandler) as server: + with self.imap_class(*server.server_address) as imap: + with mock.patch.object(imap, '_mesg') as mock_mesg: + imap._dump_ur(untagged_resp_dict) + mock_mesg.assert_called_with( + "untagged responses dump:READ-WRITE: [b'']" + ) + + +@unittest.skipUnless(ssl, "SSL not available") +class ThreadedNetworkedTestsSSL(ThreadedNetworkedTests): + server_class = SecureTCPServer + imap_class = IMAP4_SSL + + @threading_helper.reap_threads + def test_ssl_verified(self): + ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + ssl_context.load_verify_locations(CAFILE) + + # Allow for flexible libssl error messages. + regex = re.compile(r"""( + IP address mismatch, certificate is not valid for '127.0.0.1' # OpenSSL + | + CERTIFICATE_VERIFY_FAILED # AWS-LC + )""", re.X) + with self.assertRaisesRegex(ssl.CertificateError, regex): + with self.reaped_server(SimpleIMAPHandler) as server: + client = self.imap_class(*server.server_address, + ssl_context=ssl_context) + client.shutdown() + + with self.reaped_server(SimpleIMAPHandler) as server: + client = self.imap_class("localhost", server.server_address[1], + ssl_context=ssl_context) + client.shutdown() + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_import/__init__.py b/Lib/test/test_import/__init__.py index abe8b75e7db..5865dcef1e3 100644 --- a/Lib/test/test_import/__init__.py +++ b/Lib/test/test_import/__init__.py @@ -1385,7 +1385,7 @@ class PycRewritingTests(unittest.TestCase): import sys code_filename = sys._getframe().f_code.co_filename module_filename = __file__ -constant = 1 +constant = 1000 def func(): pass func_filename = func.__code__.co_filename @@ -1455,7 +1455,7 @@ def test_foreign_code(self): code = marshal.load(f) constants = list(code.co_consts) foreign_code = importlib.import_module.__code__ - pos = constants.index(1) + pos = constants.index(1000) constants[pos] = foreign_code code = code.replace(co_consts=tuple(constants)) with open(self.compiled_name, "wb") as f: diff --git a/Lib/test/test_importlib/resources/test_files.py b/Lib/test/test_importlib/resources/test_files.py index 1d04cda1a8f..0e9c5c79a1a 100644 --- a/Lib/test/test_importlib/resources/test_files.py +++ b/Lib/test/test_importlib/resources/test_files.py @@ -52,7 +52,7 @@ class OpenDiskTests(FilesTests, unittest.TestCase): def setUp(self): self.data = data01 - @unittest.expectedFailureIfWindows("TODO: RUSTPYTHON") + @unittest.expectedFailureIfWindows("TODO: RUSTPYTHON, line ending issue") def test_read_bytes(self): super().test_read_bytes() @@ -67,10 +67,11 @@ def setUp(self): self.data = namespacedata01 - @unittest.expectedFailureIfWindows("TODO: RUSTPYTHON") + @unittest.expectedFailureIfWindows("TODO: RUSTPYTHON, line ending issue") def test_read_bytes(self): super().test_read_bytes() + class SiteDir: def setUp(self): self.fixtures = contextlib.ExitStack() diff --git a/Lib/test/test_importlib/resources/test_resource.py b/Lib/test/test_importlib/resources/test_resource.py index 6f75cf57f03..54d32d9b29f 100644 --- a/Lib/test/test_importlib/resources/test_resource.py +++ b/Lib/test/test_importlib/resources/test_resource.py @@ -192,6 +192,11 @@ def test_is_file_failure_does_not_keep_open(self): def test_as_file_does_not_keep_open(self): # pragma: no cover resources.as_file(resources.files('ziptestdata') / 'binary.file') + import os # TODO: RUSTPYTHON see below + @unittest.skipIf( + 'RUSTPYTHON_SKIP_ENV_POLLUTERS' in os.environ, + "TODO: RUSTPYTHON environment pollution when running rustpython -m test --fail-env-changed due to tmpfile leak" + ) def test_entered_path_does_not_keep_open(self): """ Mimic what certifi does on import to make its bundle diff --git a/Lib/test/test_importlib/test_locks.py b/Lib/test/test_importlib/test_locks.py index 17cce741cce..edf0329c753 100644 --- a/Lib/test/test_importlib/test_locks.py +++ b/Lib/test/test_importlib/test_locks.py @@ -29,6 +29,8 @@ class ModuleLockAsRLockTests: test_timeout = None # _release_save() unsupported test_release_save_unacquired = None + # _recursion_count() unsupported + test_recursion_count = None # lock status in repr unsupported test_repr = None test_locked_repr = None @@ -92,7 +94,8 @@ def f(): b.release() if ra: a.release() - lock_tests.Bunch(f, NTHREADS).wait_for_finished() + with lock_tests.Bunch(f, NTHREADS): + pass self.assertEqual(len(results), NTHREADS) return results diff --git a/Lib/test/test_importlib/test_namespace_pkgs.py b/Lib/test/test_importlib/test_namespace_pkgs.py index 65428c3d3ea..97ce4df0848 100644 --- a/Lib/test/test_importlib/test_namespace_pkgs.py +++ b/Lib/test/test_importlib/test_namespace_pkgs.py @@ -288,7 +288,6 @@ def test_project3_succeeds(self): class ZipWithMissingDirectory(NamespacePackageTest): paths = ['missing_directory.zip'] - @unittest.expectedFailure def test_missing_directory(self): # This will fail because missing_directory.zip contains: # Length Date Time Name diff --git a/Lib/test/test_importlib/test_threaded_import.py b/Lib/test/test_importlib/test_threaded_import.py index 3ceb86cbea3..140be1e6619 100644 --- a/Lib/test/test_importlib/test_threaded_import.py +++ b/Lib/test/test_importlib/test_threaded_import.py @@ -136,8 +136,7 @@ def check_parallel_module_init(self, mock_os): if verbose: print("OK.") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.skip("TODO: RUSTPYTHON; flaky") def test_parallel_module_init(self): self.check_parallel_module_init() @@ -249,14 +248,14 @@ def target(): __import__(TESTFN) del sys.modules[TESTFN] - @unittest.skip("TODO: RUSTPYTHON; hang") + @unittest.skip('TODO: RUSTPYTHON; hang; Suspected cause of crashes in Windows CI - PermissionError: [WinError 32] Permission denied: "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\test_python_0cdrhhs_\\test_python_6340æ"') def test_concurrent_futures_circular_import(self): # Regression test for bpo-43515 fn = os.path.join(os.path.dirname(__file__), 'partial', 'cfimport.py') script_helper.assert_python_ok(fn) - @unittest.skip("TODO: RUSTPYTHON - fails on Linux due to multiprocessing issues") + @unittest.skip('TODO: RUSTPYTHON; hang') def test_multiprocessing_pool_circular_import(self): # Regression test for bpo-41567 fn = os.path.join(os.path.dirname(__file__), diff --git a/Lib/test/test_importlib/test_util.py b/Lib/test/test_importlib/test_util.py index bc47159b235..b9ff9a9f9fc 100644 --- a/Lib/test/test_importlib/test_util.py +++ b/Lib/test/test_importlib/test_util.py @@ -636,7 +636,7 @@ def test_magic_number(self): # stakeholders such as OS package maintainers must be notified # in advance. Such exceptional releases will then require an # adjustment to this test case. - EXPECTED_MAGIC_NUMBER = 2997 + EXPECTED_MAGIC_NUMBER = 2996 actual = int.from_bytes(importlib.util.MAGIC_NUMBER[:2], 'little') msg = ( diff --git a/Lib/test/test_inspect/inspect_deferred_annotations.py b/Lib/test/test_inspect/inspect_deferred_annotations.py new file mode 100644 index 00000000000..bb59ef1035b --- /dev/null +++ b/Lib/test/test_inspect/inspect_deferred_annotations.py @@ -0,0 +1,2 @@ +def f(x: undefined): + pass diff --git a/Lib/test/test_inspect/test_inspect.py b/Lib/test/test_inspect/test_inspect.py index b9cfca6df21..13ae2e3cb9c 100644 --- a/Lib/test/test_inspect/test_inspect.py +++ b/Lib/test/test_inspect/test_inspect.py @@ -1,3 +1,4 @@ +from annotationlib import Format, ForwardRef import asyncio import builtins import collections @@ -12,6 +13,8 @@ import os import dis from os.path import normcase +# XXX: RUSTPYTHON +# import _pickle import pickle import shutil import stat @@ -21,45 +24,36 @@ import types import tempfile import textwrap -from typing import Unpack import unicodedata import unittest import unittest.mock import warnings import weakref -# XXX: RUSTPYTHON; skip _pickle tests if _pickle is not available -try: - import _pickle -except ImportError: - _pickle = None - try: from concurrent.futures import ThreadPoolExecutor except ImportError: ThreadPoolExecutor = None -from test.support import cpython_only, import_helper, suppress_immortalization +from test.support import cpython_only, import_helper from test.support import MISSING_C_DOCSTRINGS, ALWAYS_EQ -# XXX: RUSTPYTHON; test.support is not updated yet -from test.support.import_helper import DirsOnSysPath #, ready_to_import +from test.support import run_no_yield_async_fn, EqualToForwardRef +from test.support.import_helper import DirsOnSysPath, ready_to_import from test.support.os_helper import TESTFN, temp_cwd from test.support.script_helper import assert_python_ok, assert_python_failure, kill_python -from test.support import has_subprocess_support, SuppressCrashReport +from test.support import has_subprocess_support from test import support from test.test_inspect import inspect_fodder as mod from test.test_inspect import inspect_fodder2 as mod2 -from test.test_inspect import inspect_stock_annotations from test.test_inspect import inspect_stringized_annotations -from test.test_inspect import inspect_stringized_annotations_2 -from test.test_inspect import inspect_stringized_annotations_pep695 +from test.test_inspect import inspect_deferred_annotations # Functions tested in this suite: # ismodule, isclass, ismethod, isfunction, istraceback, isframe, iscode, -# isbuiltin, isroutine, isgenerator, isgeneratorfunction, getmembers, +# isbuiltin, isroutine, isgenerator, ispackage, isgeneratorfunction, getmembers, # getdoc, getfile, getmodule, getsourcefile, getcomments, getsource, # getclasstree, getargvalues, formatargvalues, currentframe, # stack, trace, ismethoddescriptor, isdatadescriptor, ismethodwrapper @@ -81,11 +75,6 @@ def revise(filename, *args): git = mod.StupidGit() -def tearDownModule(): - if support.has_socket_support: - asyncio.set_event_loop_policy(None) - - def signatures_with_lexicographic_keyword_only_parameters(): """ Yields a whole bunch of functions with only keyword-only parameters, @@ -113,7 +102,7 @@ def unsorted_keyword_only_parameters_fn(*, throw, out, the, baby, with_, class IsTestBase(unittest.TestCase): predicates = set([inspect.isbuiltin, inspect.isclass, inspect.iscode, inspect.isframe, inspect.isfunction, inspect.ismethod, - inspect.ismodule, inspect.istraceback, + inspect.ismodule, inspect.istraceback, inspect.ispackage, inspect.isgenerator, inspect.isgeneratorfunction, inspect.iscoroutine, inspect.iscoroutinefunction, inspect.isasyncgen, inspect.isasyncgenfunction, @@ -129,12 +118,14 @@ def istest(self, predicate, exp): predicate == inspect.iscoroutinefunction) and \ other == inspect.isfunction: continue - self.assertFalse(other(obj), 'not %s(%s)' % (other.__name__, exp)) + if predicate == inspect.ispackage and other == inspect.ismodule: + self.assertTrue(predicate(obj), '%s(%s)' % (predicate.__name__, exp)) + else: + self.assertFalse(other(obj), 'not %s(%s)' % (other.__name__, exp)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test__all__(self): - support.check__all__(self, inspect, not_exported=("modulesbyfile",)) + support.check__all__(self, inspect, not_exported=("modulesbyfile",), extra=("get_annotations",)) def generator_function_example(self): for i in range(2): @@ -185,8 +176,7 @@ def __get__(self, instance, owner): class TestPredicates(IsTestBase): - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_excluding_predicates(self): global tb self.istest(inspect.isbuiltin, 'sys.exit') @@ -236,10 +226,19 @@ def test_excluding_predicates(self): self.assertFalse(inspect.ismethodwrapper(int)) self.assertFalse(inspect.ismethodwrapper(type("AnyClass", (), {}))) + def test_ispackage(self): + self.istest(inspect.ispackage, 'unittest') + self.istest(inspect.ispackage, 'importlib') + self.assertFalse(inspect.ispackage(inspect)) + self.assertFalse(inspect.ispackage(mod)) + self.assertFalse(inspect.ispackage(':)')) + + class FakePackage: + __path__ = None + self.assertFalse(inspect.ispackage(FakePackage())) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_iscoroutine(self): async_gen_coro = async_generator_function_example(1) gen_coro = gen_coroutine_function_example(1) @@ -384,8 +383,6 @@ def do_something_static(): coro.close(); gen_coro.close(); # silence warnings - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_isawaitable(self): def gen(): yield self.assertFalse(inspect.isawaitable(gen())) @@ -443,7 +440,7 @@ def test_isroutine(self): self.assertFalse(inspect.isroutine(int)) self.assertFalse(inspect.isroutine(type('some_class', (), {}))) # partial - self.assertFalse(inspect.isroutine(functools.partial(mod.spam))) + self.assertTrue(inspect.isroutine(functools.partial(mod.spam))) def test_isroutine_singledispatch(self): self.assertTrue(inspect.isroutine(functools.singledispatch(mod.spam))) @@ -484,8 +481,7 @@ class C(object): self.assertIn('a', members) self.assertNotIn('b', members) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_isabstract(self): from abc import ABCMeta, abstractmethod @@ -508,8 +504,7 @@ def foo(self): self.assertFalse(inspect.isabstract(int)) self.assertFalse(inspect.isabstract(5)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_isabstract_during_init_subclass(self): from abc import ABCMeta, abstractmethod isabstract_checks = [] @@ -545,8 +540,7 @@ def test_abuse_done(self): self.istest(inspect.istraceback, 'git.ex.__traceback__') self.istest(inspect.isframe, 'mod.fr') - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_stack(self): self.assertTrue(len(mod.st) >= 5) frame1, frame2, frame3, frame4, *_ = mod.st @@ -575,8 +569,7 @@ def test_stack(self): self.assertIn('inspect.stack()', record.code_context[0]) self.assertEqual(record.index, 0) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_trace(self): self.assertEqual(len(git.tr), 3) frame1, frame2, frame3, = git.tr @@ -599,8 +592,7 @@ def test_frame(self): self.assertEqual(inspect.formatargvalues(args, varargs, varkw, locals), '(x=11, y=14)') - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_previous_frame(self): args, varargs, varkw, locals = inspect.getargvalues(mod.fr.f_back) self.assertEqual(args, ['a', 'b', 'c', 'd', 'e', 'f']) @@ -682,8 +674,6 @@ def test_getfunctions(self): @unittest.skipIf(sys.flags.optimize >= 2, "Docstrings are omitted with -O2 and above") - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_getdoc(self): self.assertEqual(inspect.getdoc(mod), 'A module docstring.') self.assertEqual(inspect.getdoc(mod.StupidGit), @@ -828,12 +818,12 @@ def test_getfile(self): def test_getfile_builtin_module(self): with self.assertRaises(TypeError) as e: inspect.getfile(sys) - self.assertTrue(str(e.exception).startswith('<module')) + self.assertStartsWith(str(e.exception), '<module') def test_getfile_builtin_class(self): with self.assertRaises(TypeError) as e: inspect.getfile(int) - self.assertTrue(str(e.exception).startswith('<class')) + self.assertStartsWith(str(e.exception), '<class') def test_getfile_builtin_function_or_method(self): with self.assertRaises(TypeError) as e_abs: @@ -843,7 +833,6 @@ def test_getfile_builtin_function_or_method(self): inspect.getfile(list.append) self.assertIn('expected, got', str(e_append.exception)) - @suppress_immortalization() def test_getfile_class_without_module(self): class CM(type): @property @@ -902,8 +891,7 @@ def test_getsource_on_generated_class(self): self.assertRaises(OSError, inspect.getsourcelines, A) self.assertIsNone(inspect.getcomments(A)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_getsource_on_class_without_firstlineno(self): __firstlineno__ = 1 class C: @@ -913,8 +901,6 @@ class C: class TestGetsourceStdlib(unittest.TestCase): # Test Python implementations of the stdlib modules - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_getsource_stdlib_collections_abc(self): import collections.abc lines, lineno = inspect.getsourcelines(collections.abc.Sequence) @@ -927,8 +913,7 @@ def test_getsource_stdlib_tomllib(self): self.assertRaises(OSError, inspect.getsource, tomllib.TOMLDecodeError) self.assertRaises(OSError, inspect.getsourcelines, tomllib.TOMLDecodeError) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_getsource_stdlib_abc(self): # Pure Python implementation abc = import_helper.import_fresh_module('abc', blocked=['_abc']) @@ -980,8 +965,7 @@ def test_range_traceback_toplevel_frame(self): class TestDecorators(GetSourceBase): fodderModule = mod2 - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_wrapped_decorator(self): self.assertSourceEqual(mod2.wrapped, 14, 17) @@ -1182,8 +1166,7 @@ def test_nested_class_definition(self): self.assertSourceEqual(mod2.cls183, 183, 188) self.assertSourceEqual(mod2.cls183.cls185, 185, 188) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_class_decorator(self): self.assertSourceEqual(mod2.cls196, 194, 201) self.assertSourceEqual(mod2.cls196.cls200, 198, 201) @@ -1237,17 +1220,13 @@ def f(self): # This is necessary when the test is run multiple times. sys.modules.pop("inspect_actual") - @unittest.skipIf( - support.is_emscripten or support.is_wasi, - "socket.accept is broken" - ) def test_nested_class_definition_inside_async_function(self): - import asyncio - self.addCleanup(asyncio.set_event_loop_policy, None) - self.assertSourceEqual(asyncio.run(mod2.func225()), 226, 227) + run = run_no_yield_async_fn + + self.assertSourceEqual(run(mod2.func225), 226, 227) self.assertSourceEqual(mod2.cls226, 231, 235) self.assertSourceEqual(mod2.cls226.func232, 232, 235) - self.assertSourceEqual(asyncio.run(mod2.cls226().func232()), 233, 234) + self.assertSourceEqual(run(mod2.cls226().func232), 233, 234) def test_class_definition_same_name_diff_methods(self): self.assertSourceEqual(mod2.cls296, 296, 298) @@ -1284,8 +1263,7 @@ def test_class(self): class TestComplexDecorator(GetSourceBase): fodderModule = mod2 - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_parens_in_decorator(self): self.assertSourceEqual(self.fodderModule.complex_decorated, 273, 275) @@ -1397,9 +1375,9 @@ def test(): pass spec = inspect.getfullargspec(test) self.assertEqual(test.__annotations__, spec.annotations) + @unittest.expectedFailure # TODO: RUSTPYTHON; NameError: name '_pickle' is not defined. Did you mean: 'pickle'? Or did you forget to import '_pickle'? @unittest.skipIf(MISSING_C_DOCSTRINGS, "Signature information for builtins requires docstrings") - @unittest.skipIf(_pickle is None, "requires _pickle") def test_getfullargspec_builtin_methods(self): self.assertFullArgSpecEquals(_pickle.Pickler.dump, ['self', 'obj']) @@ -1458,7 +1436,7 @@ def test_getfullargspec_builtin_func_no_signature(self): (dict.__class_getitem__, meth_type_o), ] try: - import _stat + import _stat # noqa: F401 except ImportError: # if the _stat extension is not available, stat.S_IMODE() is # implemented in Python, not in C @@ -1562,8 +1540,7 @@ def m1(self): pass self.assertIn(('md', 'method', A), attrs, 'missing method descriptor') self.assertIn(('dd', 'data', A), attrs, 'missing data descriptor') - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_classify_builtin_types(self): # Simple sanity check that all built-in types can have their # attributes classified. @@ -1818,230 +1795,24 @@ class C(metaclass=M): attrs = [a[0] for a in inspect.getmembers(C)] self.assertNotIn('missing', attrs) - def test_get_annotations_with_stock_annotations(self): - def foo(a:int, b:str): pass - self.assertEqual(inspect.get_annotations(foo), {'a': int, 'b': str}) - - foo.__annotations__ = {'a': 'foo', 'b':'str'} - self.assertEqual(inspect.get_annotations(foo), {'a': 'foo', 'b': 'str'}) - - self.assertEqual(inspect.get_annotations(foo, eval_str=True, locals=locals()), {'a': foo, 'b': str}) - self.assertEqual(inspect.get_annotations(foo, eval_str=True, globals=locals()), {'a': foo, 'b': str}) - - isa = inspect_stock_annotations - self.assertEqual(inspect.get_annotations(isa), {'a': int, 'b': str}) - self.assertEqual(inspect.get_annotations(isa.MyClass), {'a': int, 'b': str}) - self.assertEqual(inspect.get_annotations(isa.function), {'a': int, 'b': str, 'return': isa.MyClass}) - self.assertEqual(inspect.get_annotations(isa.function2), {'a': int, 'b': 'str', 'c': isa.MyClass, 'return': isa.MyClass}) - self.assertEqual(inspect.get_annotations(isa.function3), {'a': 'int', 'b': 'str', 'c': 'MyClass'}) - self.assertEqual(inspect.get_annotations(inspect), {}) # inspect module has no annotations - self.assertEqual(inspect.get_annotations(isa.UnannotatedClass), {}) - self.assertEqual(inspect.get_annotations(isa.unannotated_function), {}) - - self.assertEqual(inspect.get_annotations(isa, eval_str=True), {'a': int, 'b': str}) - self.assertEqual(inspect.get_annotations(isa.MyClass, eval_str=True), {'a': int, 'b': str}) - self.assertEqual(inspect.get_annotations(isa.function, eval_str=True), {'a': int, 'b': str, 'return': isa.MyClass}) - self.assertEqual(inspect.get_annotations(isa.function2, eval_str=True), {'a': int, 'b': str, 'c': isa.MyClass, 'return': isa.MyClass}) - self.assertEqual(inspect.get_annotations(isa.function3, eval_str=True), {'a': int, 'b': str, 'c': isa.MyClass}) - self.assertEqual(inspect.get_annotations(inspect, eval_str=True), {}) - self.assertEqual(inspect.get_annotations(isa.UnannotatedClass, eval_str=True), {}) - self.assertEqual(inspect.get_annotations(isa.unannotated_function, eval_str=True), {}) - - self.assertEqual(inspect.get_annotations(isa, eval_str=False), {'a': int, 'b': str}) - self.assertEqual(inspect.get_annotations(isa.MyClass, eval_str=False), {'a': int, 'b': str}) - self.assertEqual(inspect.get_annotations(isa.function, eval_str=False), {'a': int, 'b': str, 'return': isa.MyClass}) - self.assertEqual(inspect.get_annotations(isa.function2, eval_str=False), {'a': int, 'b': 'str', 'c': isa.MyClass, 'return': isa.MyClass}) - self.assertEqual(inspect.get_annotations(isa.function3, eval_str=False), {'a': 'int', 'b': 'str', 'c': 'MyClass'}) - self.assertEqual(inspect.get_annotations(inspect, eval_str=False), {}) - self.assertEqual(inspect.get_annotations(isa.UnannotatedClass, eval_str=False), {}) - self.assertEqual(inspect.get_annotations(isa.unannotated_function, eval_str=False), {}) - - def times_three(fn): - @functools.wraps(fn) - def wrapper(a, b): - return fn(a*3, b*3) - return wrapper - - wrapped = times_three(isa.function) - self.assertEqual(wrapped(1, 'x'), isa.MyClass(3, 'xxx')) - self.assertIsNot(wrapped.__globals__, isa.function.__globals__) - self.assertEqual(inspect.get_annotations(wrapped), {'a': int, 'b': str, 'return': isa.MyClass}) - self.assertEqual(inspect.get_annotations(wrapped, eval_str=True), {'a': int, 'b': str, 'return': isa.MyClass}) - self.assertEqual(inspect.get_annotations(wrapped, eval_str=False), {'a': int, 'b': str, 'return': isa.MyClass}) - - def test_get_annotations_with_stringized_annotations(self): - isa = inspect_stringized_annotations - self.assertEqual(inspect.get_annotations(isa), {'a': 'int', 'b': 'str'}) - self.assertEqual(inspect.get_annotations(isa.MyClass), {'a': 'int', 'b': 'str'}) - self.assertEqual(inspect.get_annotations(isa.function), {'a': 'int', 'b': 'str', 'return': 'MyClass'}) - self.assertEqual(inspect.get_annotations(isa.function2), {'a': 'int', 'b': "'str'", 'c': 'MyClass', 'return': 'MyClass'}) - self.assertEqual(inspect.get_annotations(isa.function3), {'a': "'int'", 'b': "'str'", 'c': "'MyClass'"}) - self.assertEqual(inspect.get_annotations(isa.UnannotatedClass), {}) - self.assertEqual(inspect.get_annotations(isa.unannotated_function), {}) - - self.assertEqual(inspect.get_annotations(isa, eval_str=True), {'a': int, 'b': str}) - self.assertEqual(inspect.get_annotations(isa.MyClass, eval_str=True), {'a': int, 'b': str}) - self.assertEqual(inspect.get_annotations(isa.function, eval_str=True), {'a': int, 'b': str, 'return': isa.MyClass}) - self.assertEqual(inspect.get_annotations(isa.function2, eval_str=True), {'a': int, 'b': 'str', 'c': isa.MyClass, 'return': isa.MyClass}) - self.assertEqual(inspect.get_annotations(isa.function3, eval_str=True), {'a': 'int', 'b': 'str', 'c': 'MyClass'}) - self.assertEqual(inspect.get_annotations(isa.UnannotatedClass, eval_str=True), {}) - self.assertEqual(inspect.get_annotations(isa.unannotated_function, eval_str=True), {}) - - self.assertEqual(inspect.get_annotations(isa, eval_str=False), {'a': 'int', 'b': 'str'}) - self.assertEqual(inspect.get_annotations(isa.MyClass, eval_str=False), {'a': 'int', 'b': 'str'}) - self.assertEqual(inspect.get_annotations(isa.function, eval_str=False), {'a': 'int', 'b': 'str', 'return': 'MyClass'}) - self.assertEqual(inspect.get_annotations(isa.function2, eval_str=False), {'a': 'int', 'b': "'str'", 'c': 'MyClass', 'return': 'MyClass'}) - self.assertEqual(inspect.get_annotations(isa.function3, eval_str=False), {'a': "'int'", 'b': "'str'", 'c': "'MyClass'"}) - self.assertEqual(inspect.get_annotations(isa.UnannotatedClass, eval_str=False), {}) - self.assertEqual(inspect.get_annotations(isa.unannotated_function, eval_str=False), {}) - - isa2 = inspect_stringized_annotations_2 - self.assertEqual(inspect.get_annotations(isa2), {}) - self.assertEqual(inspect.get_annotations(isa2, eval_str=True), {}) - self.assertEqual(inspect.get_annotations(isa2, eval_str=False), {}) - - def times_three(fn): - @functools.wraps(fn) - def wrapper(a, b): - return fn(a*3, b*3) - return wrapper - - wrapped = times_three(isa.function) - self.assertEqual(wrapped(1, 'x'), isa.MyClass(3, 'xxx')) - self.assertIsNot(wrapped.__globals__, isa.function.__globals__) - self.assertEqual(inspect.get_annotations(wrapped), {'a': 'int', 'b': 'str', 'return': 'MyClass'}) - self.assertEqual(inspect.get_annotations(wrapped, eval_str=True), {'a': int, 'b': str, 'return': isa.MyClass}) - self.assertEqual(inspect.get_annotations(wrapped, eval_str=False), {'a': 'int', 'b': 'str', 'return': 'MyClass'}) - - # test that local namespace lookups work - self.assertEqual(inspect.get_annotations(isa.MyClassWithLocalAnnotations), {'x': 'mytype'}) - self.assertEqual(inspect.get_annotations(isa.MyClassWithLocalAnnotations, eval_str=True), {'x': int}) - - def test_pep695_generic_class_with_future_annotations(self): - ann_module695 = inspect_stringized_annotations_pep695 - A_annotations = inspect.get_annotations(ann_module695.A, eval_str=True) - A_type_params = ann_module695.A.__type_params__ - self.assertIs(A_annotations["x"], A_type_params[0]) - self.assertEqual(A_annotations["y"].__args__[0], Unpack[A_type_params[1]]) - self.assertIs(A_annotations["z"].__args__[0], A_type_params[2]) - - def test_pep695_generic_class_with_future_annotations_and_local_shadowing(self): - B_annotations = inspect.get_annotations( - inspect_stringized_annotations_pep695.B, eval_str=True - ) - self.assertEqual(B_annotations, {"x": int, "y": str, "z": bytes}) - - def test_pep695_generic_class_with_future_annotations_name_clash_with_global_vars(self): - ann_module695 = inspect_stringized_annotations_pep695 - C_annotations = inspect.get_annotations(ann_module695.C, eval_str=True) - self.assertEqual( - set(C_annotations.values()), - set(ann_module695.C.__type_params__) - ) - - def test_pep_695_generic_function_with_future_annotations(self): - ann_module695 = inspect_stringized_annotations_pep695 - generic_func_annotations = inspect.get_annotations( - ann_module695.generic_function, eval_str=True - ) - func_t_params = ann_module695.generic_function.__type_params__ - self.assertEqual( - generic_func_annotations.keys(), {"x", "y", "z", "zz", "return"} - ) - self.assertIs(generic_func_annotations["x"], func_t_params[0]) - self.assertEqual(generic_func_annotations["y"], Unpack[func_t_params[1]]) - self.assertIs(generic_func_annotations["z"].__origin__, func_t_params[2]) - self.assertIs(generic_func_annotations["zz"].__origin__, func_t_params[2]) - - def test_pep_695_generic_function_with_future_annotations_name_clash_with_global_vars(self): - self.assertEqual( - set( - inspect.get_annotations( - inspect_stringized_annotations_pep695.generic_function_2, - eval_str=True - ).values() - ), - set( - inspect_stringized_annotations_pep695.generic_function_2.__type_params__ - ) - ) - - def test_pep_695_generic_method_with_future_annotations(self): - ann_module695 = inspect_stringized_annotations_pep695 - generic_method_annotations = inspect.get_annotations( - ann_module695.D.generic_method, eval_str=True - ) - params = { - param.__name__: param - for param in ann_module695.D.generic_method.__type_params__ - } - self.assertEqual( - generic_method_annotations, - {"x": params["Foo"], "y": params["Bar"], "return": None} - ) - - def test_pep_695_generic_method_with_future_annotations_name_clash_with_global_vars(self): - self.assertEqual( - set( - inspect.get_annotations( - inspect_stringized_annotations_pep695.D.generic_method_2, - eval_str=True - ).values() - ), - set( - inspect_stringized_annotations_pep695.D.generic_method_2.__type_params__ - ) - ) - - def test_pep_695_generic_method_with_future_annotations_name_clash_with_global_and_local_vars(self): - self.assertEqual( - inspect.get_annotations( - inspect_stringized_annotations_pep695.E, eval_str=True - ), - {"x": str}, - ) - - def test_pep_695_generics_with_future_annotations_nested_in_function(self): - results = inspect_stringized_annotations_pep695.nested() - - self.assertEqual( - set(results.F_annotations.values()), - set(results.F.__type_params__) - ) - self.assertEqual( - set(results.F_meth_annotations.values()), - set(results.F.generic_method.__type_params__) - ) - self.assertNotEqual( - set(results.F_meth_annotations.values()), - set(results.F.__type_params__) - ) - self.assertEqual( - set(results.F_meth_annotations.values()).intersection(results.F.__type_params__), - set() - ) - - self.assertEqual(results.G_annotations, {"x": str}) - - self.assertEqual( - set(results.generic_func_annotations.values()), - set(results.generic_func.__type_params__) - ) - class TestFormatAnnotation(unittest.TestCase): def test_typing_replacement(self): from test.typinganndata.ann_module9 import A, ann, ann1 - self.assertEqual(inspect.formatannotation(ann), 'Union[List[str], int]') - self.assertEqual(inspect.formatannotation(ann1), 'Union[List[testModule.typing.A], int]') + self.assertEqual(inspect.formatannotation(ann), 'List[str] | int') + self.assertEqual(inspect.formatannotation(ann1), 'List[testModule.typing.A] | int') self.assertEqual(inspect.formatannotation(A, 'testModule.typing'), 'A') self.assertEqual(inspect.formatannotation(A, 'other'), 'testModule.typing.A') self.assertEqual( inspect.formatannotation(ann1, 'testModule.typing'), - 'Union[List[testModule.typing.A], int]', + 'List[testModule.typing.A] | int', ) + def test_forwardref(self): + fwdref = ForwardRef('fwdref') + self.assertEqual(inspect.formatannotation(fwdref), 'fwdref') + def test_formatannotationrelativeto(self): from test.typinganndata.ann_module9 import A, ann1 @@ -2073,7 +1844,7 @@ class B: ... # Not an instance of "type": self.assertEqual( inspect.formatannotationrelativeto(A)(ann1), - 'Union[List[testModule.typing.A], int]', + 'List[testModule.typing.A] | int', ) @@ -2170,7 +1941,7 @@ def function(): self.assertFalse(inspect.ismethoddescriptor(Owner.static_method)) self.assertFalse(inspect.ismethoddescriptor(function)) self.assertFalse(inspect.ismethoddescriptor(a_lambda)) - self.assertFalse(inspect.ismethoddescriptor(functools.partial(function))) + self.assertTrue(inspect.ismethoddescriptor(functools.partial(function))) def test_descriptor_being_a_class(self): class MethodDescriptorMeta(type): @@ -2252,8 +2023,7 @@ def function(): _global_ref = object() class TestGetClosureVars(unittest.TestCase): - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_name_resolution(self): # Basic test of the 4 different resolution mechanisms def f(nonlocal_ref): @@ -2269,8 +2039,7 @@ def g(local_ref): builtin_vars, unbound_names) self.assertEqual(inspect.getclosurevars(f(_arg)), expected) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_generator_closure(self): def f(nonlocal_ref): def g(local_ref): @@ -2286,8 +2055,7 @@ def g(local_ref): builtin_vars, unbound_names) self.assertEqual(inspect.getclosurevars(f(_arg)), expected) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_method_closure(self): class C: def f(self, nonlocal_ref): @@ -2303,8 +2071,7 @@ def g(local_ref): builtin_vars, unbound_names) self.assertEqual(inspect.getclosurevars(C().f(_arg)), expected) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_attribute_same_name_as_global_var(self): class C: _global_ref = object() @@ -2374,24 +2141,21 @@ def _private_globals(self): exec(code, ns) return ns["f"], ns - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_builtins_fallback(self): f, ns = self._private_globals() ns.pop("__builtins__", None) expected = inspect.ClosureVars({}, {}, {"print":print}, {"path"}) self.assertEqual(inspect.getclosurevars(f), expected) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_builtins_as_dict(self): f, ns = self._private_globals() ns["__builtins__"] = {"path":1} expected = inspect.ClosureVars({}, {}, {"path":1}, {"print"}) self.assertEqual(inspect.getclosurevars(f), expected) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_builtins_as_module(self): f, ns = self._private_globals() ns["__builtins__"] = os @@ -2486,8 +2250,7 @@ def test_varkw_only(self): self.assertEqualCallArgs(f, '**collections.UserDict(a=1, b=2)') self.assertEqualCallArgs(f, 'c=3, **collections.UserDict(a=1, b=2)') - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_keyword_only(self): f = self.makeCallable('a=3, *, c, d=2') self.assertEqualCallArgs(f, 'c=3') @@ -2528,8 +2291,7 @@ def test_multiple_features(self): '(4,[5,6])]), q=0, **collections.UserDict(' 'y=9, z=10)') - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_errors(self): f0 = self.makeCallable('') f1 = self.makeCallable('a, b') @@ -2956,7 +2718,7 @@ def __getattribute__(self, attr): self.assertFalse(test.called) - @suppress_immortalization() + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: <class 'test.test_inspect.test_inspect.TestGetattrStatic.test_cache_does_not_cause_classes_to_persist.<locals>.Foo'> is not None def test_cache_does_not_cause_classes_to_persist(self): # regression test for gh-118013: # check that the internal _shadowed_dict cache does not cause @@ -2984,33 +2746,28 @@ def number_generator(): def _generatorstate(self): return inspect.getgeneratorstate(self.generator) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_created(self): self.assertEqual(self._generatorstate(), inspect.GEN_CREATED) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_suspended(self): next(self.generator) self.assertEqual(self._generatorstate(), inspect.GEN_SUSPENDED) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_closed_after_exhaustion(self): for i in self.generator: pass self.assertEqual(self._generatorstate(), inspect.GEN_CLOSED) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_closed_after_immediate_exception(self): with self.assertRaises(RuntimeError): self.generator.throw(RuntimeError) self.assertEqual(self._generatorstate(), inspect.GEN_CLOSED) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_closed_after_close(self): self.generator.close() self.assertEqual(self._generatorstate(), inspect.GEN_CLOSED) @@ -3039,8 +2796,6 @@ def test_easy_debugging(self): self.assertIn(name, repr(state)) self.assertIn(name, str(state)) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_getgeneratorlocals(self): def each(lst, a=None): b=(1, 2, 3) @@ -3105,19 +2860,16 @@ def tearDown(self): def _coroutinestate(self): return inspect.getcoroutinestate(self.coroutine) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_created(self): self.assertEqual(self._coroutinestate(), inspect.CORO_CREATED) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_suspended(self): self.coroutine.send(None) self.assertEqual(self._coroutinestate(), inspect.CORO_SUSPENDED) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_closed_after_exhaustion(self): while True: try: @@ -3127,15 +2879,13 @@ def test_closed_after_exhaustion(self): self.assertEqual(self._coroutinestate(), inspect.CORO_CLOSED) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_closed_after_immediate_exception(self): with self.assertRaises(RuntimeError): self.coroutine.throw(RuntimeError) self.assertEqual(self._coroutinestate(), inspect.CORO_CLOSED) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_closed_after_close(self): self.coroutine.close() self.assertEqual(self._coroutinestate(), inspect.CORO_CLOSED) @@ -3178,23 +2928,24 @@ async def number_asyncgen(): async def asyncTearDown(self): await self.asyncgen.aclose() + @classmethod + def tearDownClass(cls): + asyncio.events._set_event_loop_policy(None) + def _asyncgenstate(self): return inspect.getasyncgenstate(self.asyncgen) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_created(self): self.assertEqual(self._asyncgenstate(), inspect.AGEN_CREATED) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON async def test_suspended(self): value = await anext(self.asyncgen) self.assertEqual(self._asyncgenstate(), inspect.AGEN_SUSPENDED) self.assertEqual(value, 0) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON async def test_closed_after_exhaustion(self): countdown = 7 with self.assertRaises(StopAsyncIteration): @@ -3203,15 +2954,13 @@ async def test_closed_after_exhaustion(self): self.assertEqual(countdown, 1) self.assertEqual(self._asyncgenstate(), inspect.AGEN_CLOSED) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON async def test_closed_after_immediate_exception(self): with self.assertRaises(RuntimeError): await self.asyncgen.athrow(RuntimeError) self.assertEqual(self._asyncgenstate(), inspect.AGEN_CLOSED) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON async def test_running(self): async def running_check_asyncgen(): for number in range(5): @@ -3234,8 +2983,6 @@ def test_easy_debugging(self): self.assertIn(name, repr(state)) self.assertIn(name, str(state)) - # TODO: RUSTPYTHON - @unittest.expectedFailure async def test_getasyncgenlocals(self): async def each(lst, a=None): b=(1, 2, 3) @@ -3317,7 +3064,7 @@ def test(po, /, pk, pkd=100, *args, ko, kod=10, **kwargs): pass sig = inspect.signature(test) - self.assertTrue(repr(sig).startswith('<Signature')) + self.assertStartsWith(repr(sig), '<Signature') self.assertTrue('(po, /, pk' in repr(sig)) # We need two functions, because it is impossible to represent @@ -3326,7 +3073,7 @@ def test2(pod=42, /): pass sig2 = inspect.signature(test2) - self.assertTrue(repr(sig2).startswith('<Signature')) + self.assertStartsWith(repr(sig2), '<Signature') self.assertTrue('(pod=42, /)' in repr(sig2)) po = sig.parameters['po'] @@ -3382,6 +3129,17 @@ def test2(pod=42, /): with self.assertRaisesRegex(ValueError, 'follows default argument'): S((pkd, pk)) + second_args = args.replace(name="second_args") + with self.assertRaisesRegex(ValueError, 'more than one variadic positional parameter'): + S((args, second_args)) + + with self.assertRaisesRegex(ValueError, 'more than one variadic positional parameter'): + S((args, ko, second_args)) + + second_kwargs = kwargs.replace(name="second_kwargs") + with self.assertRaisesRegex(ValueError, 'more than one variadic keyword parameter'): + S((kwargs, second_kwargs)) + def test_signature_object_pickle(self): def foo(a, b, *, c:1={}, **kw) -> {42:'ham'}: pass foo_partial = functools.partial(foo, a=1) @@ -3633,7 +3391,7 @@ def test_signature_on_builtins_no_signature(self): (dict.__class_getitem__, meth_o), ] try: - import _stat + import _stat # noqa: F401 except ImportError: # if the _stat extension is not available, stat.S_IMODE() is # implemented in Python, not in C @@ -3768,8 +3526,7 @@ def m1d(*args, **kwargs): ('arg2', 1, ..., "positional_or_keyword")), int)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_signature_on_classmethod(self): if not support.MISSING_C_DOCSTRINGS: self.assertEqual(self.signature(classmethod), @@ -3793,8 +3550,7 @@ def foo(cls, arg1, *, arg2=1): ('arg2', 1, ..., "keyword_only")), ...)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_signature_on_staticmethod(self): if not support.MISSING_C_DOCSTRINGS: self.assertEqual(self.signature(staticmethod), @@ -3819,7 +3575,7 @@ def foo(cls, *, arg): ...)) def test_signature_on_partial(self): - from functools import partial + from functools import partial, Placeholder def test(): pass @@ -3874,6 +3630,25 @@ def test(a, b, *, c, d): ('d', ..., ..., "keyword_only")), ...)) + # With Placeholder + self.assertEqual(self.signature(partial(test, Placeholder, 1)), + ((('a', ..., ..., "positional_only"), + ('c', ..., ..., "keyword_only"), + ('d', ..., ..., "keyword_only")), + ...)) + + self.assertEqual(self.signature(partial(test, Placeholder, 1, c=2)), + ((('a', ..., ..., "positional_only"), + ('c', 2, ..., "keyword_only"), + ('d', ..., ..., "keyword_only")), + ...)) + + # Ensure unittest.mock.ANY & similar do not get picked up as a Placeholder + self.assertEqual(self.signature(partial(test, unittest.mock.ANY, 1, c=2)), + ((('c', 2, ..., "keyword_only"), + ('d', ..., ..., "keyword_only")), + ...)) + def test(a, *args, b, **kwargs): pass @@ -3921,6 +3696,15 @@ def test(a, *args, b, **kwargs): ('kwargs', ..., ..., "var_keyword")), ...)) + # With Placeholder + p = partial(test, Placeholder, Placeholder, 1, b=0, test=1) + self.assertEqual(self.signature(p), + ((('a', ..., ..., "positional_only"), + ('args', ..., ..., "var_positional"), + ('b', 0, ..., "keyword_only"), + ('kwargs', ..., ..., "var_keyword")), + ...)) + def test(a, b, c:int) -> 42: pass @@ -4025,6 +3809,34 @@ def foo(a, b, /, c, d, **kwargs): ('kwargs', ..., ..., 'var_keyword')), ...)) + # Positional only With Placeholder + p = partial(foo, Placeholder, 1, c=0, d=1) + self.assertEqual(self.signature(p), + ((('a', ..., ..., "positional_only"), + ('c', 0, ..., "keyword_only"), + ('d', 1, ..., "keyword_only"), + ('kwargs', ..., ..., "var_keyword")), + ...)) + + # Optionals Positional With Placeholder + def foo(a=0, b=1, /, c=2, d=3): + pass + + # Positional + p = partial(foo, Placeholder, 1, c=0, d=1) + self.assertEqual(self.signature(p), + ((('a', ..., ..., "positional_only"), + ('c', 0, ..., "keyword_only"), + ('d', 1, ..., "keyword_only")), + ...)) + + # Positional or Keyword - transformed to positional + p = partial(foo, Placeholder, 1, Placeholder, 1) + self.assertEqual(self.signature(p), + ((('a', ..., ..., "positional_only"), + ('c', ..., ..., "positional_only")), + ...)) + def test_signature_on_partialmethod(self): from functools import partialmethod @@ -4037,18 +3849,32 @@ def test(): inspect.signature(Spam.ham) class Spam: - def test(it, a, *, c) -> 'spam': + def test(it, a, b, *, c) -> 'spam': pass ham = partialmethod(test, c=1) + bar = partialmethod(test, functools.Placeholder, 1, c=1) self.assertEqual(self.signature(Spam.ham, eval_str=False), ((('it', ..., ..., 'positional_or_keyword'), ('a', ..., ..., 'positional_or_keyword'), + ('b', ..., ..., 'positional_or_keyword'), ('c', 1, ..., 'keyword_only')), 'spam')) self.assertEqual(self.signature(Spam().ham, eval_str=False), ((('a', ..., ..., 'positional_or_keyword'), + ('b', ..., ..., 'positional_or_keyword'), + ('c', 1, ..., 'keyword_only')), + 'spam')) + + # With Placeholder + self.assertEqual(self.signature(Spam.bar, eval_str=False), + ((('it', ..., ..., 'positional_only'), + ('a', ..., ..., 'positional_only'), + ('c', 1, ..., 'keyword_only')), + 'spam')) + self.assertEqual(self.signature(Spam().bar, eval_str=False), + ((('a', ..., ..., 'positional_only'), ('c', 1, ..., 'keyword_only')), 'spam')) @@ -4138,8 +3964,7 @@ def wrapped_foo_call(): ('b', ..., ..., "positional_or_keyword")), ...)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_signature_on_class(self): class C: def __init__(self, a): @@ -4205,17 +4030,15 @@ def __init__(self, b): with self.subTest('partial'): class CM(type): - __call__ = functools.partial(lambda x, a: (x, a), 2) + __call__ = functools.partial(lambda x, a, b: (x, a, b), 2) class C(metaclass=CM): - def __init__(self, b): + def __init__(self, c): pass - with self.assertWarns(FutureWarning): - self.assertEqual(C(1), (2, 1)) - with self.assertWarns(FutureWarning): - self.assertEqual(self.signature(C), - ((('a', ..., ..., "positional_or_keyword"),), - ...)) + self.assertEqual(C(1), (2, C, 1)) + self.assertEqual(self.signature(C), + ((('b', ..., ..., "positional_or_keyword"),), + ...)) with self.subTest('partialmethod'): class CM(type): @@ -4398,8 +4221,7 @@ def __init__(self, b): self.assertEqual(self.signature(C.__call__, follow_wrapped=False), varargs_signature) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_signature_on_class_with_wrapped_init(self): class C: @identity_wrapper @@ -4451,14 +4273,12 @@ class C: with self.subTest('partial'): class C: - __init__ = functools.partial(identity_wrapper(lambda x, a: None), 2) + __init__ = functools.partial(identity_wrapper(lambda x, a, b: None), 2) - with self.assertWarns(FutureWarning): - C(1) # does not raise - with self.assertWarns(FutureWarning): - self.assertEqual(self.signature(C), - ((('a', ..., ..., "positional_or_keyword"),), - ...)) + C(1) # does not raise + self.assertEqual(self.signature(C), + ((('b', ..., ..., "positional_or_keyword"),), + ...)) with self.subTest('partialmethod'): class C: @@ -4499,8 +4319,7 @@ def __init__(self, a): self.assertEqual(self.signature(C.__new__, follow_wrapped=False), varargs_signature) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_signature_on_class_with_wrapped_new(self): with self.subTest('FunctionType'): class C: @@ -4589,8 +4408,6 @@ def __new__(cls, a): self.assertEqual(self.signature(C.__new__, follow_wrapped=False), varargs_signature) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_signature_on_class_with_init(self): class C: def __init__(self, b): @@ -4637,14 +4454,12 @@ class C: with self.subTest('partial'): class C: - __init__ = functools.partial(lambda x, a: None, 2) + __init__ = functools.partial(lambda x, a, b: None, 2) - with self.assertWarns(FutureWarning): - C(1) # does not raise - with self.assertWarns(FutureWarning): - self.assertEqual(self.signature(C), - ((('a', ..., ..., "positional_or_keyword"),), - ...)) + C(1) # does not raise + self.assertEqual(self.signature(C), + ((('b', ..., ..., "positional_or_keyword"),), + ...)) with self.subTest('partialmethod'): class C: @@ -4657,8 +4472,7 @@ def _init(self, x, a): ((('a', ..., ..., "positional_or_keyword"),), ...)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_signature_on_class_with_new(self): with self.subTest('FunctionType'): class C: @@ -4819,9 +4633,9 @@ class D(C): pass with self.assertRaisesRegex(ValueError, "callable.*is not supported"): self.assertEqual(inspect.signature(D), None) + @unittest.expectedFailure # TODO: RUSTPYTHON; NameError: name '_pickle' is not defined. Did you mean: 'pickle'? Or did you forget to import '_pickle'? @unittest.skipIf(MISSING_C_DOCSTRINGS, "Signature information for builtins requires docstrings") - @unittest.skipIf(_pickle is None, "requires _pickle") def test_signature_on_builtin_class(self): expected = ('(file, protocol=None, fix_imports=True, ' 'buffer_callback=None)') @@ -4845,8 +4659,7 @@ class P4(P2, metaclass=MetaP): pass self.assertEqual(str(inspect.signature(P4)), '(foo, bar)') - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_signature_on_callable_objects(self): class Foo: def __call__(self, a): @@ -4902,15 +4715,13 @@ class C: with self.subTest('partial'): class C: - __call__ = functools.partial(lambda x, a: (x, a), 2) + __call__ = functools.partial(lambda x, a, b: (x, a, b), 2) c = C() - with self.assertWarns(FutureWarning): - self.assertEqual(c(1), (2, 1)) - with self.assertWarns(FutureWarning): - self.assertEqual(self.signature(c), - ((('a', ..., ..., "positional_or_keyword"),), - ...)) + self.assertEqual(c(1), (2, c, 1)) + self.assertEqual(self.signature(C()), + ((('b', ..., ..., "positional_or_keyword"),), + ...)) with self.subTest('partialmethod'): class C: @@ -5176,6 +4987,11 @@ def foo(a: list[str]) -> Tuple[str, float]: self.assertEqual(str(inspect.signature(foo)), inspect.signature(foo).format()) + def foo(x: undef): + pass + sig = inspect.signature(foo, annotation_format=Format.FORWARDREF) + self.assertEqual(str(sig), '(x: undef)') + def test_signature_str_positional_only(self): P = inspect.Parameter S = inspect.Signature @@ -5265,6 +5081,18 @@ def func( expected_multiline, ) + def test_signature_format_unquote(self): + def func(x: 'int') -> 'str': ... + + self.assertEqual( + inspect.signature(func).format(), + "(x: 'int') -> 'str'" + ) + self.assertEqual( + inspect.signature(func).format(quote_annotation_strings=False), + "(x: int) -> str" + ) + def test_signature_replace_parameters(self): def test(a, b) -> 42: pass @@ -5317,8 +5145,7 @@ def test(): sig = test.__signature__ = inspect.Signature(parameters=(spam_param,)) self.assertEqual(sig, inspect.signature(test)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_signature_on_mangled_parameters(self): class Spam: def foo(self, __p1:1=2, *, __p2:2=3): @@ -5350,9 +5177,9 @@ class foo: pass foo_sig = MySignature.from_callable(foo) self.assertIsInstance(foo_sig, MySignature) + @unittest.expectedFailure # TODO: RUSTPYTHON; NameError: name '_pickle' is not defined. Did you mean: 'pickle'? Or did you forget to import '_pickle'? @unittest.skipIf(MISSING_C_DOCSTRINGS, "Signature information for builtins requires docstrings") - @unittest.skipIf(_pickle is None, "requires _pickle") def test_signature_from_callable_builtin_obj(self): class MySignature(inspect.Signature): pass sig = MySignature.from_callable(_pickle.Pickler) @@ -5500,6 +5327,60 @@ def test_signature_eval_str(self): par('b', PORK, annotation=tuple), ))) + def test_signature_annotation_format(self): + ida = inspect_deferred_annotations + sig = inspect.Signature + par = inspect.Parameter + PORK = inspect.Parameter.POSITIONAL_OR_KEYWORD + for signature_func in (inspect.signature, inspect.Signature.from_callable): + with self.subTest(signature_func=signature_func): + self.assertEqual( + signature_func(ida.f, annotation_format=Format.STRING), + sig([par("x", PORK, annotation="undefined")]) + ) + s1 = signature_func(ida.f, annotation_format=Format.FORWARDREF) + s2 = sig([par("x", PORK, annotation=EqualToForwardRef("undefined", owner=ida.f))]) + #breakpoint() + self.assertEqual( + signature_func(ida.f, annotation_format=Format.FORWARDREF), + sig([par("x", PORK, annotation=EqualToForwardRef("undefined", owner=ida.f))]) + ) + with self.assertRaisesRegex(NameError, "undefined"): + signature_func(ida.f, annotation_format=Format.VALUE) + with self.assertRaisesRegex(NameError, "undefined"): + signature_func(ida.f) + + def test_signature_deferred_annotations(self): + def f(x: undef): + pass + + class C: + x: undef + + def __init__(self, x: undef): + self.x = x + + sig = inspect.signature(f, annotation_format=Format.FORWARDREF) + self.assertEqual(list(sig.parameters), ['x']) + sig = inspect.signature(C, annotation_format=Format.FORWARDREF) + self.assertEqual(list(sig.parameters), ['x']) + + class CallableWrapper: + def __init__(self, func): + self.func = func + self.__annotate__ = func.__annotate__ + + def __call__(self, *args, **kwargs): + return self.func(*args, **kwargs) + + @property + def __annotations__(self): + return self.__annotate__(Format.VALUE) + + cw = CallableWrapper(f) + sig = inspect.signature(cw, annotation_format=Format.FORWARDREF) + self.assertEqual(list(sig.parameters), ['args', 'kwargs']) + def test_signature_none_annotation(self): class funclike: # Has to be callable, and have correct @@ -5525,38 +5406,6 @@ def foo(): pass self.assertEqual(signature_func(foo), inspect.Signature()) self.assertEqual(inspect.get_annotations(foo), {}) - def test_signature_as_str(self): - self.maxDiff = None - class S: - __signature__ = '(a, b=2)' - - self.assertEqual(self.signature(S), - ((('a', ..., ..., 'positional_or_keyword'), - ('b', 2, ..., 'positional_or_keyword')), - ...)) - - def test_signature_as_callable(self): - # __signature__ should be either a staticmethod or a bound classmethod - class S: - @classmethod - def __signature__(cls): - return '(a, b=2)' - - self.assertEqual(self.signature(S), - ((('a', ..., ..., 'positional_or_keyword'), - ('b', 2, ..., 'positional_or_keyword')), - ...)) - - class S: - @staticmethod - def __signature__(): - return '(a, b=2)' - - self.assertEqual(self.signature(S), - ((('a', ..., ..., 'positional_or_keyword'), - ('b', 2, ..., 'positional_or_keyword')), - ...)) - def test_signature_on_derived_classes(self): # gh-105080: Make sure that signatures are consistent on derived classes @@ -5637,7 +5486,7 @@ def test_signature_parameter_object(self): with self.assertRaisesRegex(ValueError, 'cannot have default values'): p.replace(kind=inspect.Parameter.VAR_POSITIONAL) - self.assertTrue(repr(p).startswith('<Parameter')) + self.assertStartsWith(repr(p), '<Parameter') self.assertTrue('"a=42"' in repr(p)) def test_signature_parameter_hashable(self): @@ -6011,7 +5860,7 @@ def test_signature_bind_implicit_arg(self): # Issue #19611: getcallargs should work with comprehensions def make_set(): return set(z * z for z in range(5)) - gencomp_code = make_set.__code__.co_consts[1] + gencomp_code = make_set.__code__.co_consts[0] gencomp_func = types.FunctionType(gencomp_code, {}) iterator = iter(range(5)) @@ -6139,8 +5988,7 @@ def _strip_non_python_syntax(self, input, self.assertEqual(computed_clean_signature, clean_signature) self.assertEqual(computed_self_parameter, self_parameter) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_signature_strip_non_python_syntax(self): self._strip_non_python_syntax( "($module, /, path, mode, *, dir_fd=None, " + @@ -6279,7 +6127,6 @@ def test_builtins_have_signatures(self): 'bytearray': {'count', 'endswith', 'find', 'hex', 'index', 'rfind', 'rindex', 'startswith'}, 'bytes': {'count', 'endswith', 'find', 'hex', 'index', 'rfind', 'rindex', 'startswith'}, 'dict': {'pop'}, - 'int': {'__round__'}, 'memoryview': {'cast', 'hex'}, 'str': {'count', 'endswith', 'find', 'index', 'maketrans', 'rfind', 'rindex', 'startswith'}, } @@ -6343,14 +6190,14 @@ def test_errno_module_has_signatures(self): def test_faulthandler_module_has_signatures(self): import faulthandler - unsupported_signature = {'dump_traceback', 'dump_traceback_later', 'enable'} + unsupported_signature = {'dump_traceback', 'dump_traceback_later', 'enable', 'dump_c_stack'} unsupported_signature |= {name for name in ['register'] if hasattr(faulthandler, name)} self._test_module_has_signatures(faulthandler, unsupported_signature=unsupported_signature) def test_functools_module_has_signatures(self): - no_signature = {'reduce'} - self._test_module_has_signatures(functools, no_signature) + unsupported_signature = {"reduce"} + self._test_module_has_signatures(functools, unsupported_signature=unsupported_signature) def test_gc_module_has_signatures(self): import gc @@ -6384,7 +6231,7 @@ def test_operator_module_has_signatures(self): def test_os_module_has_signatures(self): unsupported_signature = {'chmod', 'utime'} unsupported_signature |= {name for name in - ['get_terminal_size', 'posix_spawn', 'posix_spawnp', + ['get_terminal_size', 'link', 'posix_spawn', 'posix_spawnp', 'register_at_fork', 'startfile'] if hasattr(os, name)} self._test_module_has_signatures(os, unsupported_signature=unsupported_signature) @@ -6445,8 +6292,7 @@ def test_tokenize_module_has_signatures(self): import tokenize self._test_module_has_signatures(tokenize) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_tracemalloc_module_has_signatures(self): import tracemalloc self._test_module_has_signatures(tracemalloc) @@ -6474,8 +6320,7 @@ def test_weakref_module_has_signatures(self): no_signature = {'ReferenceType', 'ref'} self._test_module_has_signatures(weakref, no_signature) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_python_function_override_signature(self): def func(*args, **kwargs): pass @@ -6506,9 +6351,8 @@ def func(*args, **kwargs): with self.assertRaises(ValueError): inspect.signature(func) + @unittest.expectedFailure # TODO: RUSTPYTHON @support.requires_docstrings - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_base_class_have_text_signature(self): # see issue 43118 from test.typinganndata.ann_module7 import BufferedReader @@ -6672,8 +6516,6 @@ def assertInspectEqual(self, path, source): inspected_src.splitlines(True) ) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_getsource_reload(self): # see issue 1218234 with ready_to_import('reload_bug', self.src_before) as (name, path): @@ -6728,9 +6570,8 @@ def run_on_interactive_mode(self, source): raise ValueError("Process didn't exit properly.") return output + @unittest.expectedFailure # TODO: RUSTPYTHON @unittest.skipIf(not has_subprocess_support, "test requires subprocess") - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_getsource(self): output = self.run_on_interactive_mode(textwrap.dedent("""\ def f(): @@ -6748,4 +6589,4 @@ def f(): if __name__ == "__main__": - unittest.main() \ No newline at end of file + unittest.main() diff --git a/Lib/test/test_int.py b/Lib/test/test_int.py index 1ab7a1fb6dd..7f62ac067c4 100644 --- a/Lib/test/test_int.py +++ b/Lib/test/test_int.py @@ -681,7 +681,7 @@ def test_denial_of_service_prevented_int_to_str(self): digits = 78_268 with ( support.adjust_int_max_str_digits(digits), - support.CPUStopwatch() as sw_convert): + support.Stopwatch() as sw_convert): huge_decimal = str(huge_int) self.assertEqual(len(huge_decimal), digits) # Ensuring that we chose a slow enough conversion to measure. @@ -696,7 +696,7 @@ def test_denial_of_service_prevented_int_to_str(self): with support.adjust_int_max_str_digits(int(.995 * digits)): with ( self.assertRaises(ValueError) as err, - support.CPUStopwatch() as sw_fail_huge): + support.Stopwatch() as sw_fail_huge): str(huge_int) self.assertIn('conversion', str(err.exception)) self.assertLessEqual(sw_fail_huge.seconds, sw_convert.seconds/2) @@ -706,7 +706,7 @@ def test_denial_of_service_prevented_int_to_str(self): extra_huge_int = int(f'0x{"c"*500_000}', base=16) # 602060 digits. with ( self.assertRaises(ValueError) as err, - support.CPUStopwatch() as sw_fail_extra_huge): + support.Stopwatch() as sw_fail_extra_huge): # If not limited, 8 seconds said Zen based cloud VM. str(extra_huge_int) self.assertIn('conversion', str(err.exception)) @@ -722,7 +722,7 @@ def test_denial_of_service_prevented_str_to_int(self): huge = '8'*digits with ( support.adjust_int_max_str_digits(digits), - support.CPUStopwatch() as sw_convert): + support.Stopwatch() as sw_convert): int(huge) # Ensuring that we chose a slow enough conversion to measure. # It takes 0.1 seconds on a Zen based cloud VM in an opt build. @@ -734,7 +734,7 @@ def test_denial_of_service_prevented_str_to_int(self): with support.adjust_int_max_str_digits(digits - 1): with ( self.assertRaises(ValueError) as err, - support.CPUStopwatch() as sw_fail_huge): + support.Stopwatch() as sw_fail_huge): int(huge) self.assertIn('conversion', str(err.exception)) self.assertLessEqual(sw_fail_huge.seconds, sw_convert.seconds/2) @@ -744,7 +744,7 @@ def test_denial_of_service_prevented_str_to_int(self): extra_huge = '7'*1_200_000 with ( self.assertRaises(ValueError) as err, - support.CPUStopwatch() as sw_fail_extra_huge): + support.Stopwatch() as sw_fail_extra_huge): # If not limited, 8 seconds in the Zen based cloud VM. int(extra_huge) self.assertIn('conversion', str(err.exception)) diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py index 1e594c49640..5fd011360f0 100644 --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -780,8 +780,8 @@ def test_closefd_attr(self): file = self.open(f.fileno(), "r", encoding="utf-8", closefd=False) self.assertEqual(file.buffer.raw.closefd, False) + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: filter ('', ResourceWarning) did not catch any warning @unittest.skipIf(sys.platform == 'win32', 'TODO: RUSTPYTHON; cyclic GC not supported, causes file locking') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_garbage_collection(self): # FileIO objects are collected, and collecting them flushes # all data to disk. @@ -1803,8 +1803,8 @@ def test_misbehaved_io_read(self): # checking this is not so easy. self.assertRaises(OSError, bufio.read, 10) + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: filter ('', ResourceWarning) did not catch any warning @unittest.skipIf(sys.platform == 'win32', 'TODO: RUSTPYTHON; cyclic GC not supported, causes file locking') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_garbage_collection(self): # C BufferedReader objects are collected. # The Python version has __del__, so it ends into gc.garbage instead @@ -1839,21 +1839,14 @@ def test_bad_readinto_type(self): bufio.readline() self.assertIsInstance(cm.exception.__cause__, TypeError) - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_error_through_destructor(self): - return super().test_error_through_destructor() - - def test_truncate_on_read_only(self): - return super().test_truncate_on_read_only() - - @unittest.skip('TODO: RUSTPYTHON; fallible allocation') - def test_constructor(self): - return super().test_constructor() - - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_pickling_subclass(self): return super().test_pickling_subclass() + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'NoneType' object has no attribute 'exc_type' + def test_error_through_destructor(self): + return super().test_error_through_destructor() + class PyBufferedReaderTest(BufferedReaderTest): tp = pyio.BufferedReader @@ -2168,8 +2161,8 @@ def test_initialization(self): self.assertRaises(ValueError, bufio.__init__, rawio, buffer_size=-1) self.assertRaises(ValueError, bufio.write, b"def") + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: filter ('', ResourceWarning) did not catch any warning @unittest.skipIf(sys.platform == 'win32', 'TODO: RUSTPYTHON; cyclic GC not supported, causes file locking') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_garbage_collection(self): # C BufferedWriter objects are collected, and collecting them flushes # all data to disk. @@ -2192,18 +2185,14 @@ def test_args_error(self): with self.assertRaisesRegex(TypeError, "BufferedWriter"): self.tp(self.BytesIO(), 1024, 1024, 1024) - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_error_through_destructor(self): - return super().test_error_through_destructor() - - @unittest.skip('TODO: RUSTPYTHON; fallible allocation') - def test_constructor(self): - return super().test_constructor() - - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_pickling_subclass(self): return super().test_pickling_subclass() + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'NoneType' object has no attribute 'exc_type' + def test_error_through_destructor(self): + return super().test_error_through_destructor() + class PyBufferedWriterTest(BufferedWriterTest): tp = pyio.BufferedWriter @@ -2680,8 +2669,8 @@ def test_interleaved_readline_write(self): class CBufferedRandomTest(BufferedRandomTest, SizeofTest): tp = io.BufferedRandom + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: filter ('', ResourceWarning) did not catch any warning @unittest.skipIf(sys.platform == 'win32', 'TODO: RUSTPYTHON; cyclic GC not supported, causes file locking') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_garbage_collection(self): CBufferedReaderTest.test_garbage_collection(self) CBufferedWriterTest.test_garbage_collection(self) @@ -2691,18 +2680,14 @@ def test_args_error(self): with self.assertRaisesRegex(TypeError, "BufferedRandom"): self.tp(self.BytesIO(), 1024, 1024, 1024) - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_error_through_destructor(self): - return super().test_error_through_destructor() - - @unittest.skip('TODO: RUSTPYTHON; fallible allocation') - def test_constructor(self): - return super().test_constructor() - - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_pickling_subclass(self): return super().test_pickling_subclass() + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'NoneType' object has no attribute 'exc_type' + def test_error_through_destructor(self): + return super().test_error_through_destructor() + class PyBufferedRandomTest(BufferedRandomTest): tp = pyio.BufferedRandom @@ -2862,6 +2847,7 @@ def setUp(self): def tearDown(self): os_helper.unlink(os_helper.TESTFN) + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: UnicodeEncodeError not raised def test_constructor(self): r = self.BytesIO(b"\xc3\xa9\n\n") b = self.BufferedReader(r, 1000) @@ -3084,6 +3070,7 @@ def test_encoding_errors_writing(self): t.flush() self.assertEqual(b.getvalue(), b"abc?def\n") + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'codecs' has no attribute 'utf_32_ex_decode'. Did you mean: 'utf_16_ex_decode'? def test_newlines(self): input_lines = [ "unix\n", "windows\r\n", "os9\r", "last\n", "nonl" ] @@ -3355,7 +3342,7 @@ def test_seek_and_tell_with_data(data, min_pos=0): finally: StatefulIncrementalDecoder.codecEnabled = 0 - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; LookupError: unknown encoding: euc_jp def test_multibyte_seek_and_tell(self): f = self.open(os_helper.TESTFN, "w", encoding="euc_jp") f.write("AB\n\u3046\u3048\n") @@ -3402,7 +3389,7 @@ def test_seek_with_encoder_state(self): self.assertEqual(f.readline(), "\u00e6\u0300\u0300") f.close() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'codecs' has no attribute 'utf_32_ex_decode'. Did you mean: 'utf_16_ex_decode'? def test_encoded_writes(self): data = "1234567890" tests = ("utf-16", @@ -3541,7 +3528,6 @@ def test_issue2282(self): self.assertEqual(buffer.seekable(), txt.seekable()) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_append_bom(self): # The BOM is not written again when appending to a non-empty file filename = os_helper.TESTFN @@ -3557,7 +3543,6 @@ def test_append_bom(self): with self.open(filename, 'rb') as f: self.assertEqual(f.read(), 'aaaxxx'.encode(charset)) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_seek_bom(self): # Same test, but when seeking manually filename = os_helper.TESTFN @@ -3573,7 +3558,6 @@ def test_seek_bom(self): with self.open(filename, 'rb') as f: self.assertEqual(f.read(), 'bbbzzz'.encode(charset)) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_seek_append_bom(self): # Same test, but first seek to the start and then to the end filename = os_helper.TESTFN @@ -3841,7 +3825,7 @@ def __del__(self): """.format(iomod=iomod, kwargs=kwargs) return assert_python_ok("-c", code) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 'LookupError: unknown encoding: ascii' not found in "Exception ignored in: <function C.__del__ at 0x72d8eea80>\nAttributeError: 'NoneType' object has no attribute 'TextIOWrapper'\n" def test_create_at_shutdown_without_encoding(self): rc, out, err = self._check_create_at_shutdown() if err: @@ -3851,7 +3835,7 @@ def test_create_at_shutdown_without_encoding(self): else: self.assertEqual("ok", out.decode().strip()) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: b"Exception ignored in: <function C.__del__ at 0xbc2a2e940>\nAttributeError: 'NoneType' object has no attribute 'TextIOWrapper'\n" is not false def test_create_at_shutdown_with_encoding(self): rc, out, err = self._check_create_at_shutdown(encoding='utf-8', errors='strict') @@ -4123,7 +4107,7 @@ class CTextIOWrapperTest(TextIOWrapperTest): io = io shutdown_error = "LookupError: unknown encoding: ascii" - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: ValueError not raised by read def test_initialization(self): r = self.BytesIO(b"\xc3\xa9\n\n") b = self.BufferedReader(r, 1000) @@ -4134,8 +4118,8 @@ def test_initialization(self): t = self.TextIOWrapper.__new__(self.TextIOWrapper) self.assertRaises(Exception, repr, t) + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: filter ('', ResourceWarning) did not catch any warning @unittest.skipIf(sys.platform == 'win32', 'TODO: RUSTPYTHON; cyclic GC not supported, causes file locking') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_garbage_collection(self): # C TextIOWrapper objects are collected, and collecting them flushes # all data to disk. @@ -4199,7 +4183,7 @@ def write(self, data): t.write("x"*chunk_size) self.assertEqual([b"abcdef", b"ghi", b"x"*chunk_size], buf._write_stack) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; RuntimeError: reentrant call inside textio def test_issue119506(self): chunk_size = 8192 @@ -4222,87 +4206,74 @@ def write(self, data): self.assertEqual([b"abcdef", b"middle", b"g"*chunk_size], buf._write_stack) - def test_basic_io(self): - return super().test_basic_io() - - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_constructor(self): - return super().test_constructor() + # TODO: RUSTPYTHON; euc_jis_2004 encoding not supported + @unittest.expectedFailure + def test_seek_with_encoder_state(self): + return super().test_seek_with_encoder_state() - def test_detach(self): - return super().test_detach() + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_pickling_subclass(self): + return super().test_pickling_subclass() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_newlines(self): - return super().test_newlines() + @unittest.expectedFailure # TODO: RUSTPYTHON; + + def test_reconfigure_newline(self): + return super().test_reconfigure_newline() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; + ['AAA\nBB\x00B\nCCC\r', 'DDD\r', 'EEE\r', '\nFFF\r', '\nGGG'] def test_newlines_input(self): return super().test_newlines_input() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; + strict + def test_reconfigure_defaults(self): + return super().test_reconfigure_defaults() + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: LookupError not raised def test_non_text_encoding_codecs_are_rejected(self): return super().test_non_text_encoding_codecs_are_rejected() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_reconfigure_defaults(self): - return super().test_reconfigure_defaults() + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: Regex didn't match: "<(_io\\.)?TextIOWrapper name='dummy' mode='r' encoding='utf-8'>" not found in "<_io.TextIOWrapper name='dummy' encoding='utf-8'>" + def test_repr(self): + return super().test_repr() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_reconfigure_encoding_read(self): - return super().test_reconfigure_encoding_read() + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: RuntimeError not raised + def test_recursive_repr(self): + return super().test_recursive_repr() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: UnicodeEncodeError not raised def test_reconfigure_errors(self): return super().test_reconfigure_errors() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_reconfigure_line_buffering(self): - return super().test_reconfigure_line_buffering() + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: UnsupportedOperation not raised + def test_reconfigure_encoding_read(self): + return super().test_reconfigure_encoding_read() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_reconfigure_locale(self): - return super().test_reconfigure_locale() + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: b'' != b'1' + def test_reconfigure_write_through(self): + return super().test_reconfigure_write_through() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_reconfigure_newline(self): - return super().test_reconfigure_newline() + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: b'' != b'AB\nC' + def test_reconfigure_line_buffering(self): + return super().test_reconfigure_line_buffering() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: b'' != b'abc\xe9\n' def test_reconfigure_write(self): return super().test_reconfigure_write() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: b'\xef\xbb\xbfaaa\xef\xbb\xbfxxx' != b'\xef\xbb\xbfaaaxxx' + def test_append_bom(self): + return super().test_append_bom() + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: b'foo\n\xef\xbb\xbf\xc3\xa9\n' != b'foo\n\xc3\xa9\n' def test_reconfigure_write_fromascii(self): return super().test_reconfigure_write_fromascii() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_reconfigure_write_through(self): - return super().test_reconfigure_write_through() - - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'NoneType' object has no attribute 'exc_type' def test_error_through_destructor(self): return super().test_error_through_destructor() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_repr(self): - return super().test_repr() - - def test_uninitialized(self): - return super().test_uninitialized() - - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_recursive_repr(self): - return super().test_recursive_repr() - - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_pickling_subclass(self): - return super().test_pickling_subclass() - - # TODO: RUSTPYTHON; euc_jis_2004 encoding not supported - @unittest.expectedFailure - def test_seek_with_encoder_state(self): - return super().test_seek_with_encoder_state() + @unittest.expectedFailure # TODO: RUSTPYTHON; LookupError: unknown encoding: locale + def test_reconfigure_locale(self): + return super().test_reconfigure_locale() class PyTextIOWrapperTest(TextIOWrapperTest): @@ -4313,10 +4284,6 @@ class PyTextIOWrapperTest(TextIOWrapperTest): def test_constructor(self): return super().test_constructor() - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_newlines(self): - return super().test_newlines() - # TODO: RUSTPYTHON; euc_jis_2004 encoding not supported @unittest.expectedFailure def test_seek_with_encoder_state(self): @@ -4400,7 +4367,7 @@ def _decode_bytewise(s): self.assertEqual(decoder.decode(input), "abc") self.assertEqual(decoder.newlines, None) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'codecs' has no attribute 'utf_32_ex_decode'. Did you mean: 'utf_16_ex_decode'? def test_newline_decoder(self): encodings = ( # None meaning the IncrementalNewlineDecoder takes unicode input @@ -4821,7 +4788,7 @@ def test_check_encoding_warning(self): self.assertTrue( warnings[1].startswith(b"<string>:8: EncodingWarning: ")) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: b'locale' != b'utf-8' def test_text_encoding(self): # PEP 597, bpo-47000. io.text_encoding() returns "locale" or "utf-8" # based on sys.flags.utf8_mode @@ -4889,11 +4856,13 @@ def run(): else: self.assertFalse(err.strip('.!')) + @unittest.expectedFailure # TODO: RUSTPYTHON; without GC+GIL, finalize_modules clears __main__ globals while daemon threads are still running @threading_helper.requires_working_threading() @support.requires_resource('walltime') def test_daemon_threads_shutdown_stdout_deadlock(self): self.check_daemon_threads_shutdown_deadlock('stdout') + @unittest.expectedFailure # TODO: RUSTPYTHON; without GC+GIL, finalize_modules clears __main__ globals while daemon threads are still running @threading_helper.requires_working_threading() @support.requires_resource('walltime') def test_daemon_threads_shutdown_stderr_deadlock(self): @@ -5085,14 +5054,12 @@ def alarm_handler(sig, frame): os.close(w) os.close(r) - @unittest.expectedFailure # TODO: RUSTPYTHON @requires_alarm @support.requires_resource('walltime') def test_interrupted_read_retry_buffered(self): self.check_interrupted_read_retry(lambda x: x.decode('latin1'), mode="rb") - @unittest.expectedFailure # TODO: RUSTPYTHON @requires_alarm @support.requires_resource('walltime') def test_interrupted_read_retry_text(self): @@ -5167,13 +5134,13 @@ def alarm2(sig, frame): if e.errno != errno.EBADF: raise - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.skip("TODO: RUSTPYTHON thread 'main' (103833) panicked at crates/vm/src/stdlib/signal.rs:233:43: RefCell already borrowed") @requires_alarm @support.requires_resource('walltime') def test_interrupted_write_retry_buffered(self): self.check_interrupted_write_retry(b"x", mode="wb") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.skip("TODO: RUSTPYTHON thread 'main' (103833) panicked at crates/vm/src/stdlib/signal.rs:233:43: RefCell already borrowed") @requires_alarm @support.requires_resource('walltime') def test_interrupted_write_retry_text(self): @@ -5183,6 +5150,10 @@ def test_interrupted_write_retry_text(self): class CSignalsTest(SignalsTest): io = io + @unittest.skip("TODO: RUSTPYTHON thread 'main' (103833) panicked at crates/vm/src/stdlib/signal.rs:233:43: RefCell already borrowed") + def test_interrupted_read_retry_buffered(self): # TODO: RUSTPYTHON + return super().test_interrupted_read_retry_buffered() # TODO: RUSTPYTHON + class PySignalsTest(SignalsTest): io = pyio diff --git a/Lib/test/test_ipaddress.py b/Lib/test/test_ipaddress.py index e69e12495ad..8af91e857d8 100644 --- a/Lib/test/test_ipaddress.py +++ b/Lib/test/test_ipaddress.py @@ -12,6 +12,7 @@ import pickle import ipaddress import weakref +from collections.abc import Iterator from test.support import LARGEST, SMALLEST @@ -1472,18 +1473,27 @@ def testGetSupernet4(self): self.ipv6_scoped_network.supernet(new_prefix=62)) def testHosts(self): + hosts = self.ipv4_network.hosts() + self.assertIsInstance(hosts, Iterator) + self.assertEqual(ipaddress.IPv4Address('1.2.3.1'), next(hosts)) hosts = list(self.ipv4_network.hosts()) self.assertEqual(254, len(hosts)) self.assertEqual(ipaddress.IPv4Address('1.2.3.1'), hosts[0]) self.assertEqual(ipaddress.IPv4Address('1.2.3.254'), hosts[-1]) ipv6_network = ipaddress.IPv6Network('2001:658:22a:cafe::/120') + hosts = ipv6_network.hosts() + self.assertIsInstance(hosts, Iterator) + self.assertEqual(ipaddress.IPv6Address('2001:658:22a:cafe::1'), next(hosts)) hosts = list(ipv6_network.hosts()) self.assertEqual(255, len(hosts)) self.assertEqual(ipaddress.IPv6Address('2001:658:22a:cafe::1'), hosts[0]) self.assertEqual(ipaddress.IPv6Address('2001:658:22a:cafe::ff'), hosts[-1]) ipv6_scoped_network = ipaddress.IPv6Network('2001:658:22a:cafe::%scope/120') + hosts = ipv6_scoped_network.hosts() + self.assertIsInstance(hosts, Iterator) + self.assertEqual((ipaddress.IPv6Address('2001:658:22a:cafe::1')), next(hosts)) hosts = list(ipv6_scoped_network.hosts()) self.assertEqual(255, len(hosts)) self.assertEqual(ipaddress.IPv6Address('2001:658:22a:cafe::1'), hosts[0]) @@ -1494,6 +1504,12 @@ def testHosts(self): ipaddress.IPv4Address('2.0.0.1')] str_args = '2.0.0.0/31' tpl_args = ('2.0.0.0', 31) + hosts = ipaddress.ip_network(str_args).hosts() + self.assertIsInstance(hosts, Iterator) + self.assertEqual(next(hosts), addrs[0]) + hosts = ipaddress.ip_network(tpl_args).hosts() + self.assertIsInstance(hosts, Iterator) + self.assertEqual(next(hosts), addrs[0]) self.assertEqual(addrs, list(ipaddress.ip_network(str_args).hosts())) self.assertEqual(addrs, list(ipaddress.ip_network(tpl_args).hosts())) self.assertEqual(list(ipaddress.ip_network(str_args).hosts()), @@ -1503,6 +1519,12 @@ def testHosts(self): addrs = [ipaddress.IPv4Address('1.2.3.4')] str_args = '1.2.3.4/32' tpl_args = ('1.2.3.4', 32) + hosts = ipaddress.ip_network(str_args).hosts() + self.assertIsInstance(hosts, Iterator) + self.assertEqual(next(hosts), addrs[0]) + hosts = ipaddress.ip_network(tpl_args).hosts() + self.assertIsInstance(hosts, Iterator) + self.assertEqual(next(hosts), addrs[0]) self.assertEqual(addrs, list(ipaddress.ip_network(str_args).hosts())) self.assertEqual(addrs, list(ipaddress.ip_network(tpl_args).hosts())) self.assertEqual(list(ipaddress.ip_network(str_args).hosts()), @@ -1512,6 +1534,12 @@ def testHosts(self): ipaddress.IPv6Address('2001:658:22a:cafe::1')] str_args = '2001:658:22a:cafe::/127' tpl_args = ('2001:658:22a:cafe::', 127) + hosts = ipaddress.ip_network(str_args).hosts() + self.assertIsInstance(hosts, Iterator) + self.assertEqual(next(hosts), addrs[0]) + hosts = ipaddress.ip_network(tpl_args).hosts() + self.assertIsInstance(hosts, Iterator) + self.assertEqual(next(hosts), addrs[0]) self.assertEqual(addrs, list(ipaddress.ip_network(str_args).hosts())) self.assertEqual(addrs, list(ipaddress.ip_network(tpl_args).hosts())) self.assertEqual(list(ipaddress.ip_network(str_args).hosts()), @@ -1520,6 +1548,12 @@ def testHosts(self): addrs = [ipaddress.IPv6Address('2001:658:22a:cafe::1'), ] str_args = '2001:658:22a:cafe::1/128' tpl_args = ('2001:658:22a:cafe::1', 128) + hosts = ipaddress.ip_network(str_args).hosts() + self.assertIsInstance(hosts, Iterator) + self.assertEqual(next(hosts), addrs[0]) + hosts = ipaddress.ip_network(tpl_args).hosts() + self.assertIsInstance(hosts, Iterator) + self.assertEqual(next(hosts), addrs[0]) self.assertEqual(addrs, list(ipaddress.ip_network(str_args).hosts())) self.assertEqual(addrs, list(ipaddress.ip_network(tpl_args).hosts())) self.assertEqual(list(ipaddress.ip_network(str_args).hosts()), @@ -2214,12 +2248,18 @@ def testIPv6AddressTooLarge(self): ipaddress.ip_address('ffff::c0a8:ffff%scope')) def testIPVersion(self): + self.assertEqual(ipaddress.IPv4Address.version, 4) + self.assertEqual(ipaddress.IPv6Address.version, 6) + self.assertEqual(self.ipv4_address.version, 4) self.assertEqual(self.ipv6_address.version, 6) self.assertEqual(self.ipv6_scoped_address.version, 6) self.assertEqual(self.ipv6_with_ipv4_part.version, 6) def testMaxPrefixLength(self): + self.assertEqual(ipaddress.IPv4Address.max_prefixlen, 32) + self.assertEqual(ipaddress.IPv6Address.max_prefixlen, 128) + self.assertEqual(self.ipv4_interface.max_prefixlen, 32) self.assertEqual(self.ipv6_interface.max_prefixlen, 128) self.assertEqual(self.ipv6_scoped_interface.max_prefixlen, 128) diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py index 03dadb71f78..e4d44e3d38c 100644 --- a/Lib/test/test_itertools.py +++ b/Lib/test/test_itertools.py @@ -248,6 +248,7 @@ def test_chain_from_iterable(self): self.assertRaises(TypeError, list, chain.from_iterable([2, 3])) self.assertEqual(list(islice(chain.from_iterable(repeat(range(5))), 2)), [0, 1]) + @unittest.expectedFailure # TODO: RUSTPYTHON @pickle_deprecated def test_chain_reducible(self): for oper in [copy.deepcopy] + picklecopiers: @@ -567,6 +568,7 @@ def test_combinatorics(self): self.assertEqual(comb, list(filter(set(perm).__contains__, cwr))) # comb: cwr that is a perm self.assertEqual(comb, sorted(set(cwr) & set(perm))) # comb: both a cwr and a perm + @unittest.expectedFailure # TODO: RUSTPYTHON @pickle_deprecated def test_compress(self): self.assertEqual(list(compress(data='ABCDEF', selectors=[1,0,1,0,1,1])), list('ACEF')) @@ -601,6 +603,7 @@ def test_compress(self): next(testIntermediate) self.assertEqual(list(op(testIntermediate)), list(result2)) + @unittest.expectedFailure # TODO: RUSTPYTHON @pickle_deprecated def test_count(self): self.assertEqual(lzip('abc',count()), [('a', 0), ('b', 1), ('c', 2)]) @@ -1035,6 +1038,7 @@ def test_filter(self): c = filter(isEven, range(6)) self.pickletest(proto, c) + @unittest.expectedFailure # TODO: RUSTPYTHON @pickle_deprecated def test_filterfalse(self): self.assertEqual(list(filterfalse(isEven, range(6))), [1,3,5]) @@ -1142,6 +1146,7 @@ def test_zip_longest_tuple_reuse(self): ids = list(map(id, list(zip_longest('abc', 'def')))) self.assertEqual(len(dict.fromkeys(ids)), len(ids)) + @unittest.expectedFailure # TODO: RUSTPYTHON @pickle_deprecated def test_zip_longest_pickling(self): for proto in range(pickle.HIGHEST_PROTOCOL + 1): @@ -1365,6 +1370,7 @@ def test_product_tuple_reuse(self): self.assertEqual(len(set(map(id, product('abc', 'def')))), 1) self.assertNotEqual(len(set(map(id, list(product('abc', 'def'))))), 1) + @unittest.expectedFailure # TODO: RUSTPYTHON @pickle_deprecated def test_product_pickling(self): # check copy, deepcopy, pickle @@ -1393,6 +1399,7 @@ def test_product_issue_25021(self): p.__setstate__((0, 0, 0x1000)) # will access tuple element 1 if not clamped self.assertRaises(StopIteration, next, p) + @unittest.expectedFailure # TODO: RUSTPYTHON @pickle_deprecated def test_repeat(self): self.assertEqual(list(repeat(object='a', times=3)), ['a', 'a', 'a']) @@ -1458,6 +1465,7 @@ def test_map(self): c = map(tupleize, 'abc', count()) self.pickletest(proto, c) + @unittest.expectedFailure # TODO: RUSTPYTHON @pickle_deprecated def test_starmap(self): self.assertEqual(list(starmap(operator.pow, zip(range(3), range(1,7)))), @@ -1582,6 +1590,7 @@ def __index__(self): self.assertEqual(list(islice(range(100), IntLike(10), IntLike(50), IntLike(5))), list(range(10,50,5))) + @unittest.expectedFailure # TODO: RUSTPYTHON @pickle_deprecated def test_takewhile(self): data = [1, 3, 5, 20, 2, 4, 6, 8] @@ -1807,7 +1816,6 @@ def __next__(self): with self.assertRaisesRegex(RuntimeError, "tee"): next(a) - @unittest.skip("TODO: RUSTPYTHON; , hangs") @threading_helper.requires_working_threading() def test_tee_concurrent(self): start = threading.Event() @@ -1942,6 +1950,7 @@ class TestExamples(unittest.TestCase): def test_accumulate(self): self.assertEqual(list(accumulate([1,2,3,4,5])), [1, 3, 6, 10, 15]) + @unittest.expectedFailure # TODO: RUSTPYTHON @pickle_deprecated def test_accumulate_reducible(self): # check copy, deepcopy, pickle diff --git a/Lib/test/test_json/__init__.py b/Lib/test/test_json/__init__.py index b919af2328f..7091364cddb 100644 --- a/Lib/test/test_json/__init__.py +++ b/Lib/test/test_json/__init__.py @@ -41,8 +41,7 @@ def test_pyjson(self): 'json.encoder') class TestCTest(CTest): - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_cjson(self): self.assertEqual(self.json.scanner.make_scanner.__module__, '_json') self.assertEqual(self.json.decoder.scanstring.__module__, '_json') diff --git a/Lib/test/test_json/test_decode.py b/Lib/test/test_json/test_decode.py index f07f7d55339..b6531b237c0 100644 --- a/Lib/test/test_json/test_decode.py +++ b/Lib/test/test_json/test_decode.py @@ -18,8 +18,7 @@ def test_float(self): self.assertIsInstance(rval, float) self.assertEqual(rval, 1.0) - # TODO: RUSTPYTHON - @unittest.skip("TODO: RUSTPYTHON; called `Result::unwrap()` on an `Err` value: ParseFloatError { kind: Invalid }") + @unittest.skip('TODO: RUSTPYTHON; called `Result::unwrap()` on an `Err` value: ParseFloatError { kind: Invalid }') def test_nonascii_digits_rejected(self): # JSON specifies only ascii digits, see gh-125687 for num in ["1\uff10", "0.\uff10", "0e\uff10"]: @@ -138,11 +137,6 @@ def test_limit_int(self): class TestPyDecode(TestDecode, PyTest): pass class TestCDecode(TestDecode, CTest): - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_keys_reuse(self): - return super().test_keys_reuse() - # TODO: RUSTPYTHON @unittest.expectedFailure def test_limit_int(self): diff --git a/Lib/test/test_json/test_default.py b/Lib/test/test_json/test_default.py index 3ce16684a08..b576947c4f2 100644 --- a/Lib/test/test_json/test_default.py +++ b/Lib/test/test_json/test_default.py @@ -1,4 +1,5 @@ import collections +import unittest # XXX: RUSTPYTHON; importing to be able to skip tests from test.test_json import PyTest, CTest @@ -8,6 +9,26 @@ def test_default(self): self.dumps(type, default=repr), self.dumps(repr(type))) + # TODO: RUSTPYTHON + @unittest.expectedFailure + def test_bad_default(self): + def default(obj): + if obj is NotImplemented: + raise ValueError + if obj is ...: + return NotImplemented + if obj is type: + return collections + return [...] + + with self.assertRaises(ValueError) as cm: + self.dumps(type, default=default) + self.assertEqual(cm.exception.__notes__, + ['when serializing ellipsis object', + 'when serializing list item 0', + 'when serializing module object', + 'when serializing type object']) + def test_ordereddict(self): od = collections.OrderedDict(a=1, b=2, c=3, d=4) od.move_to_end('b') diff --git a/Lib/test/test_json/test_encode_basestring_ascii.py b/Lib/test/test_json/test_encode_basestring_ascii.py index 6a39b72a09d..c90d3e968e5 100644 --- a/Lib/test/test_json/test_encode_basestring_ascii.py +++ b/Lib/test/test_json/test_encode_basestring_ascii.py @@ -8,13 +8,12 @@ ('\u0123\u4567\u89ab\ucdef\uabcd\uef4a', '"\\u0123\\u4567\\u89ab\\ucdef\\uabcd\\uef4a"'), ('controls', '"controls"'), ('\x08\x0c\n\r\t', '"\\b\\f\\n\\r\\t"'), + ('\x00\x1f\x7f', '"\\u0000\\u001f\\u007f"'), ('{"object with 1 member":["array with 1 element"]}', '"{\\"object with 1 member\\":[\\"array with 1 element\\"]}"'), (' s p a c e d ', '" s p a c e d "'), ('\U0001d120', '"\\ud834\\udd20"'), ('\u03b1\u03a9', '"\\u03b1\\u03a9"'), ("`1~!@#$%^&*()_+-={':[,]}|;.</>?", '"`1~!@#$%^&*()_+-={\':[,]}|;.</>?"'), - ('\x08\x0c\n\r\t', '"\\b\\f\\n\\r\\t"'), - ('\u0123\u4567\u89ab\ucdef\uabcd\uef4a', '"\\u0123\\u4567\\u89ab\\ucdef\\uabcd\\uef4a"'), ] class TestEncodeBasestringAscii: diff --git a/Lib/test/test_json/test_fail.py b/Lib/test/test_json/test_fail.py index 7a85665c816..cb88e317b4e 100644 --- a/Lib/test/test_json/test_fail.py +++ b/Lib/test/test_json/test_fail.py @@ -1,7 +1,7 @@ -from test.test_json import PyTest, CTest - import unittest # XXX: RUSTPYTHON; importing to be able to skip tests +from test.test_json import PyTest, CTest + # 2007-10-05 JSONDOCS = [ # https://json.org/JSON_checker/test/fail1.json @@ -102,8 +102,27 @@ def test_non_string_keys_dict(self): def test_not_serializable(self): import sys with self.assertRaisesRegex(TypeError, - 'Object of type module is not JSON serializable'): + 'Object of type module is not JSON serializable') as cm: self.dumps(sys) + self.assertNotHasAttr(cm.exception, '__notes__') + + with self.assertRaises(TypeError) as cm: + self.dumps([1, [2, 3, sys]]) + self.assertEqual(cm.exception.__notes__, + ['when serializing list item 2', + 'when serializing list item 1']) + + with self.assertRaises(TypeError) as cm: + self.dumps((1, (2, 3, sys))) + self.assertEqual(cm.exception.__notes__, + ['when serializing tuple item 2', + 'when serializing tuple item 1']) + + with self.assertRaises(TypeError) as cm: + self.dumps({'a': {'b': sys}}) + self.assertEqual(cm.exception.__notes__, + ["when serializing dict item 'b'", + "when serializing dict item 'a'"]) def test_truncated_input(self): test_cases = [ diff --git a/Lib/test/test_json/test_recursion.py b/Lib/test/test_json/test_recursion.py index 59f6f2c4b19..2a24edef629 100644 --- a/Lib/test/test_json/test_recursion.py +++ b/Lib/test/test_json/test_recursion.py @@ -14,8 +14,8 @@ def test_listrecursion(self): x.append(x) try: self.dumps(x) - except ValueError: - pass + except ValueError as exc: + self.assertEqual(exc.__notes__, ["when serializing list item 0"]) else: self.fail("didn't raise ValueError on list recursion") x = [] @@ -23,8 +23,8 @@ def test_listrecursion(self): x.append(y) try: self.dumps(x) - except ValueError: - pass + except ValueError as exc: + self.assertEqual(exc.__notes__, ["when serializing list item 0"]*2) else: self.fail("didn't raise ValueError on alternating list recursion") y = [] @@ -37,8 +37,8 @@ def test_dictrecursion(self): x["test"] = x try: self.dumps(x) - except ValueError: - pass + except ValueError as exc: + self.assertEqual(exc.__notes__, ["when serializing dict item 'test'"]) else: self.fail("didn't raise ValueError on dict recursion") x = {} @@ -62,31 +62,41 @@ def default(self, o): enc.recurse = True try: enc.encode(JSONTestObject) - except ValueError: - pass + except ValueError as exc: + self.assertEqual(exc.__notes__, + ["when serializing list item 0", + "when serializing type object"]) else: self.fail("didn't raise ValueError on default recursion") + # TODO: RUSTPYTHON - @unittest.skip("TODO: RUSTPYTHON; crashes") + @unittest.skip('TODO: RUSTPYTHON; crashes') + # TODO: RUSTPYHTON; needs to upgrade test.support to 3.14 above + # @support.skip_emscripten_stack_overflow() + # @support.skip_wasi_stack_overflow() def test_highly_nested_objects_decoding(self): + very_deep = 200000 # test that loading highly-nested objects doesn't segfault when C # accelerations are used. See #12017 with self.assertRaises(RecursionError): with support.infinite_recursion(): - self.loads('{"a":' * 100000 + '1' + '}' * 100000) + self.loads('{"a":' * very_deep + '1' + '}' * very_deep) with self.assertRaises(RecursionError): with support.infinite_recursion(): - self.loads('{"a":' * 100000 + '[1]' + '}' * 100000) + self.loads('{"a":' * very_deep + '[1]' + '}' * very_deep) with self.assertRaises(RecursionError): with support.infinite_recursion(): - self.loads('[' * 100000 + '1' + ']' * 100000) + self.loads('[' * very_deep + '1' + ']' * very_deep) + # TODO: RUSTPYHTON; needs to upgrade test.support to 3.14 above + # @support.skip_wasi_stack_overflow() + # @support.skip_emscripten_stack_overflow() @support.requires_resource('cpu') def test_highly_nested_objects_encoding(self): # See #12051 l, d = [], {} - for x in range(100000): + for x in range(200_000): l, d = [l], {'k':d} with self.assertRaises(RecursionError): with support.infinite_recursion(5000): @@ -95,6 +105,9 @@ def test_highly_nested_objects_encoding(self): with support.infinite_recursion(5000): self.dumps(d) + # TODO: RUSTPYHTON; needs to upgrade test.support to 3.14 above + # @support.skip_emscripten_stack_overflow() + # @support.skip_wasi_stack_overflow() def test_endless_recursion(self): # See #12051 class EndlessJSONEncoder(self.json.JSONEncoder): diff --git a/Lib/test/test_json/test_scanstring.py b/Lib/test/test_json/test_scanstring.py index a5c46bb64b4..c7fc30f2235 100644 --- a/Lib/test/test_json/test_scanstring.py +++ b/Lib/test/test_json/test_scanstring.py @@ -3,6 +3,7 @@ import unittest # XXX: RUSTPYTHON; importing to be able to skip tests + class TestScanstring: def test_scanstring(self): scanstring = self.json.decoder.scanstring @@ -143,11 +144,10 @@ def test_bad_escapes(self): with self.assertRaises(self.JSONDecodeError, msg=s): scanstring(s, 1, True) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_overflow(self): with self.assertRaises(OverflowError): - self.json.decoder.scanstring(b"xxx", sys.maxsize+1) + self.json.decoder.scanstring("xxx", sys.maxsize+1) class TestPyScanstring(TestScanstring, PyTest): pass diff --git a/Lib/test/test_json/test_speedups.py b/Lib/test/test_json/test_speedups.py index ada96729123..25b51b307e1 100644 --- a/Lib/test/test_json/test_speedups.py +++ b/Lib/test/test_json/test_speedups.py @@ -40,8 +40,7 @@ def test_make_encoder(self): b"\xCD\x7D\x3D\x4E\x12\x4C\xF9\x79\xD7\x52\xBA\x82\xF2\x27\x4A\x7D\xA0\xCA\x75", None) - # TODO: RUSTPYTHON; TypeError: 'NoneType' object is not callable - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: 'NoneType' object is not callable def test_bad_str_encoder(self): # Issue #31505: There shouldn't be an assertion failure in case # c_make_encoder() receives a bad encoder() argument. @@ -63,8 +62,7 @@ def bad_encoder2(*args): with self.assertRaises(ZeroDivisionError): enc('spam', 4) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_bad_markers_argument_to_encoder(self): # https://bugs.python.org/issue45269 with self.assertRaisesRegex( @@ -74,8 +72,7 @@ def test_bad_markers_argument_to_encoder(self): self.json.encoder.c_make_encoder(1, None, None, None, ': ', ', ', False, False, False) - # TODO: RUSTPYTHON; ZeroDivisionError not raised by test - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; ZeroDivisionError not raised by test def test_bad_bool_args(self): def test(name): self.json.encoder.JSONEncoder(**{name: BadBool()}).encode({'a': 1}) diff --git a/Lib/test/test_json/test_tool.py b/Lib/test/test_json/test_tool.py index 2b63810d539..068a7acb651 100644 --- a/Lib/test/test_json/test_tool.py +++ b/Lib/test/test_json/test_tool.py @@ -6,12 +6,23 @@ import subprocess from test import support -from test.support import os_helper +from test.support import force_not_colorized, os_helper from test.support.script_helper import assert_python_ok +from _colorize import get_theme + +# XXX: RUSTPYTHON; force_colorized not available in test.support +def force_colorized(func): + """Placeholder decorator for RustPython - force_colorized not available.""" + import functools + @functools.wraps(func) + def wrapper(*args, **kwargs): + raise unittest.SkipTest("TODO: RUSTPYTHON; force_colorized not available") + return wrapper + @support.requires_subprocess() -class TestTool(unittest.TestCase): +class TestMain(unittest.TestCase): data = """ [["blorpie"],[ "whoops" ] , [ @@ -19,6 +30,7 @@ class TestTool(unittest.TestCase): "i-vhbjkhnth", {"nifty":87}, {"morefield" :\tfalse,"field" :"yes"} ] """ + module = 'json' expect_without_sort_keys = textwrap.dedent("""\ [ @@ -86,8 +98,9 @@ class TestTool(unittest.TestCase): } """) + @force_not_colorized def test_stdin_stdout(self): - args = sys.executable, '-m', 'json.tool' + args = sys.executable, '-m', self.module process = subprocess.run(args, input=self.data, capture_output=True, text=True, check=True) self.assertEqual(process.stdout, self.expect) self.assertEqual(process.stderr, '') @@ -101,7 +114,8 @@ def _create_infile(self, data=None): def test_infile_stdout(self): infile = self._create_infile() - rc, out, err = assert_python_ok('-m', 'json.tool', infile) + rc, out, err = assert_python_ok('-m', self.module, infile, + PYTHON_COLORS='0') self.assertEqual(rc, 0) self.assertEqual(out.splitlines(), self.expect.encode().splitlines()) self.assertEqual(err, b'') @@ -115,7 +129,8 @@ def test_non_ascii_infile(self): ''').encode() infile = self._create_infile(data) - rc, out, err = assert_python_ok('-m', 'json.tool', infile) + rc, out, err = assert_python_ok('-m', self.module, infile, + PYTHON_COLORS='0') self.assertEqual(rc, 0) self.assertEqual(out.splitlines(), expect.splitlines()) @@ -124,7 +139,8 @@ def test_non_ascii_infile(self): def test_infile_outfile(self): infile = self._create_infile() outfile = os_helper.TESTFN + '.out' - rc, out, err = assert_python_ok('-m', 'json.tool', infile, outfile) + rc, out, err = assert_python_ok('-m', self.module, infile, outfile, + PYTHON_COLORS='0') self.addCleanup(os.remove, outfile) with open(outfile, "r", encoding="utf-8") as fp: self.assertEqual(fp.read(), self.expect) @@ -134,33 +150,38 @@ def test_infile_outfile(self): def test_writing_in_place(self): infile = self._create_infile() - rc, out, err = assert_python_ok('-m', 'json.tool', infile, infile) + rc, out, err = assert_python_ok('-m', self.module, infile, infile, + PYTHON_COLORS='0') with open(infile, "r", encoding="utf-8") as fp: self.assertEqual(fp.read(), self.expect) self.assertEqual(rc, 0) self.assertEqual(out, b'') self.assertEqual(err, b'') + @force_not_colorized def test_jsonlines(self): - args = sys.executable, '-m', 'json.tool', '--json-lines' + args = sys.executable, '-m', self.module, '--json-lines' process = subprocess.run(args, input=self.jsonlines_raw, capture_output=True, text=True, check=True) self.assertEqual(process.stdout, self.jsonlines_expect) self.assertEqual(process.stderr, '') def test_help_flag(self): - rc, out, err = assert_python_ok('-m', 'json.tool', '-h') + rc, out, err = assert_python_ok('-m', self.module, '-h', + PYTHON_COLORS='0') self.assertEqual(rc, 0) - self.assertTrue(out.startswith(b'usage: ')) + self.assertStartsWith(out, b'usage: ') self.assertEqual(err, b'') def test_sort_keys_flag(self): infile = self._create_infile() - rc, out, err = assert_python_ok('-m', 'json.tool', '--sort-keys', infile) + rc, out, err = assert_python_ok('-m', self.module, '--sort-keys', infile, + PYTHON_COLORS='0') self.assertEqual(rc, 0) self.assertEqual(out.splitlines(), self.expect_without_sort_keys.encode().splitlines()) self.assertEqual(err, b'') + @force_not_colorized def test_indent(self): input_ = '[1, 2]' expect = textwrap.dedent('''\ @@ -169,31 +190,34 @@ def test_indent(self): 2 ] ''') - args = sys.executable, '-m', 'json.tool', '--indent', '2' + args = sys.executable, '-m', self.module, '--indent', '2' process = subprocess.run(args, input=input_, capture_output=True, text=True, check=True) self.assertEqual(process.stdout, expect) self.assertEqual(process.stderr, '') + @force_not_colorized def test_no_indent(self): input_ = '[1,\n2]' expect = '[1, 2]\n' - args = sys.executable, '-m', 'json.tool', '--no-indent' + args = sys.executable, '-m', self.module, '--no-indent' process = subprocess.run(args, input=input_, capture_output=True, text=True, check=True) self.assertEqual(process.stdout, expect) self.assertEqual(process.stderr, '') + @force_not_colorized def test_tab(self): input_ = '[1, 2]' expect = '[\n\t1,\n\t2\n]\n' - args = sys.executable, '-m', 'json.tool', '--tab' + args = sys.executable, '-m', self.module, '--tab' process = subprocess.run(args, input=input_, capture_output=True, text=True, check=True) self.assertEqual(process.stdout, expect) self.assertEqual(process.stderr, '') + @force_not_colorized def test_compact(self): input_ = '[ 1 ,\n 2]' expect = '[1,2]\n' - args = sys.executable, '-m', 'json.tool', '--compact' + args = sys.executable, '-m', self.module, '--compact' process = subprocess.run(args, input=input_, capture_output=True, text=True, check=True) self.assertEqual(process.stdout, expect) self.assertEqual(process.stderr, '') @@ -202,7 +226,8 @@ def test_no_ensure_ascii_flag(self): infile = self._create_infile('{"key":"💩"}') outfile = os_helper.TESTFN + '.out' self.addCleanup(os.remove, outfile) - assert_python_ok('-m', 'json.tool', '--no-ensure-ascii', infile, outfile) + assert_python_ok('-m', self.module, '--no-ensure-ascii', infile, + outfile, PYTHON_COLORS='0') with open(outfile, "rb") as f: lines = f.read().splitlines() # asserting utf-8 encoded output file @@ -213,20 +238,99 @@ def test_ensure_ascii_default(self): infile = self._create_infile('{"key":"💩"}') outfile = os_helper.TESTFN + '.out' self.addCleanup(os.remove, outfile) - assert_python_ok('-m', 'json.tool', infile, outfile) + assert_python_ok('-m', self.module, infile, outfile, PYTHON_COLORS='0') with open(outfile, "rb") as f: lines = f.read().splitlines() # asserting an ascii encoded output file expected = [b'{', rb' "key": "\ud83d\udca9"', b"}"] self.assertEqual(lines, expected) + @force_not_colorized @unittest.skipIf(sys.platform =="win32", "The test is failed with ValueError on Windows") def test_broken_pipe_error(self): - cmd = [sys.executable, '-m', 'json.tool'] + cmd = [sys.executable, '-m', self.module] proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE) - # bpo-39828: Closing before json.tool attempts to write into stdout. + # bpo-39828: Closing before json attempts to write into stdout. proc.stdout.close() proc.communicate(b'"{}"') self.assertEqual(proc.returncode, errno.EPIPE) + + @force_colorized + def test_colors(self): + infile = os_helper.TESTFN + self.addCleanup(os.remove, infile) + + t = get_theme().syntax + ob = "{" + cb = "}" + + cases = ( + ('{}', '{}'), + ('[]', '[]'), + ('null', f'{t.keyword}null{t.reset}'), + ('true', f'{t.keyword}true{t.reset}'), + ('false', f'{t.keyword}false{t.reset}'), + ('NaN', f'{t.number}NaN{t.reset}'), + ('Infinity', f'{t.number}Infinity{t.reset}'), + ('-Infinity', f'{t.number}-Infinity{t.reset}'), + ('"foo"', f'{t.string}"foo"{t.reset}'), + (r'" \"foo\" "', f'{t.string}" \\"foo\\" "{t.reset}'), + ('"α"', f'{t.string}"\\u03b1"{t.reset}'), + ('123', f'{t.number}123{t.reset}'), + ('-1.25e+23', f'{t.number}-1.25e+23{t.reset}'), + (r'{"\\": ""}', + f'''\ +{ob} + {t.definition}"\\\\"{t.reset}: {t.string}""{t.reset} +{cb}'''), + (r'{"\\\\": ""}', + f'''\ +{ob} + {t.definition}"\\\\\\\\"{t.reset}: {t.string}""{t.reset} +{cb}'''), + ('''\ +{ + "foo": "bar", + "baz": 1234, + "qux": [true, false, null], + "xyz": [NaN, -Infinity, Infinity] +}''', + f'''\ +{ob} + {t.definition}"foo"{t.reset}: {t.string}"bar"{t.reset}, + {t.definition}"baz"{t.reset}: {t.number}1234{t.reset}, + {t.definition}"qux"{t.reset}: [ + {t.keyword}true{t.reset}, + {t.keyword}false{t.reset}, + {t.keyword}null{t.reset} + ], + {t.definition}"xyz"{t.reset}: [ + {t.number}NaN{t.reset}, + {t.number}-Infinity{t.reset}, + {t.number}Infinity{t.reset} + ] +{cb}'''), + ) + + for input_, expected in cases: + with self.subTest(input=input_): + with open(infile, "w", encoding="utf-8") as fp: + fp.write(input_) + _, stdout_b, _ = assert_python_ok( + '-m', self.module, infile, FORCE_COLOR='1', __isolated='1' + ) + stdout = stdout_b.decode() + stdout = stdout.replace('\r\n', '\n') # normalize line endings + stdout = stdout.strip() + self.assertEqual(stdout, expected) + + +@support.requires_subprocess() +class TestTool(TestMain): + module = 'json.tool' + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_json/test_unicode.py b/Lib/test/test_json/test_unicode.py index 4bdb607e7da..c1fba019ccc 100644 --- a/Lib/test/test_json/test_unicode.py +++ b/Lib/test/test_json/test_unicode.py @@ -34,6 +34,29 @@ def test_encoding7(self): j = self.dumps(u + "\n", ensure_ascii=False) self.assertEqual(j, f'"{u}\\n"') + def test_ascii_non_printable_encode(self): + u = '\b\t\n\f\r\x00\x1f\x7f' + self.assertEqual(self.dumps(u), + '"\\b\\t\\n\\f\\r\\u0000\\u001f\\u007f"') + self.assertEqual(self.dumps(u, ensure_ascii=False), + '"\\b\\t\\n\\f\\r\\u0000\\u001f\x7f"') + + def test_ascii_non_printable_decode(self): + self.assertEqual(self.loads('"\\b\\t\\n\\f\\r"'), + '\b\t\n\f\r') + s = ''.join(map(chr, range(32))) + for c in s: + self.assertRaises(self.JSONDecodeError, self.loads, f'"{c}"') + self.assertEqual(self.loads(f'"{s}"', strict=False), s) + self.assertEqual(self.loads('"\x7f"'), '\x7f') + + def test_escaped_decode(self): + self.assertEqual(self.loads('"\\b\\t\\n\\f\\r"'), '\b\t\n\f\r') + self.assertEqual(self.loads('"\\"\\\\\\/"'), '"\\/') + for c in set(map(chr, range(0x100))) - set('"\\/bfnrt'): + self.assertRaises(self.JSONDecodeError, self.loads, f'"\\{c}"') + self.assertRaises(self.JSONDecodeError, self.loads, f'"\\{c}"', strict=False) + def test_big_unicode_encode(self): u = '\U0001d120' self.assertEqual(self.dumps(u), '"\\ud834\\udd20"') @@ -50,6 +73,18 @@ def test_unicode_decode(self): s = f'"\\u{i:04x}"' self.assertEqual(self.loads(s), u) + def test_single_surrogate_encode(self): + self.assertEqual(self.dumps('\uD83D'), '"\\ud83d"') + self.assertEqual(self.dumps('\uD83D', ensure_ascii=False), '"\ud83d"') + self.assertEqual(self.dumps('\uDC0D'), '"\\udc0d"') + self.assertEqual(self.dumps('\uDC0D', ensure_ascii=False), '"\udc0d"') + + def test_single_surrogate_decode(self): + self.assertEqual(self.loads('"\uD83D"'), '\ud83d') + self.assertEqual(self.loads('"\\uD83D"'), '\ud83d') + self.assertEqual(self.loads('"\udc0d"'), '\udc0d') + self.assertEqual(self.loads('"\\udc0d"'), '\udc0d') + def test_unicode_preservation(self): self.assertEqual(type(self.loads('""')), str) self.assertEqual(type(self.loads('"a"')), str) @@ -59,8 +94,6 @@ def test_bytes_encode(self): self.assertRaises(TypeError, self.dumps, b"hi") self.assertRaises(TypeError, self.dumps, [b"hi"]) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_bytes_decode(self): for encoding, bom in [ ('utf-8', codecs.BOM_UTF8), @@ -104,4 +137,19 @@ def test_object_pairs_hook_with_unicode(self): class TestPyUnicode(TestUnicode, PyTest): pass -class TestCUnicode(TestUnicode, CTest): pass + +class TestCUnicode(TestUnicode, CTest): + # TODO: RUSTPYTHON + @unittest.expectedFailure + def test_ascii_non_printable_encode(self): + return super().test_ascii_non_printable_encode() + + # TODO: RUSTPYTHON + @unittest.skip("TODO: RUSTPYTHON; panics with 'str has surrogates'") + def test_single_surrogate_decode(self): + return super().test_single_surrogate_decode() + + # TODO: RUSTPYTHON + @unittest.skip("TODO: RUSTPYTHON; panics with 'str has surrogates'") + def test_single_surrogate_encode(self): + return super().test_single_surrogate_encode() diff --git a/Lib/test/test_linecache.py b/Lib/test/test_linecache.py index e23e1cc9428..e4aa41ebb43 100644 --- a/Lib/test/test_linecache.py +++ b/Lib/test/test_linecache.py @@ -281,6 +281,19 @@ def test_loader(self): self.assertEqual(linecache.getlines(filename, module_globals), ['source for x.y.z\n']) + def test_frozen(self): + filename = '<frozen fakemodule>' + module_globals = {'__file__': FILENAME} + empty = linecache.getlines(filename) + self.assertEqual(empty, []) + lines = linecache.getlines(filename, module_globals) + self.assertGreater(len(lines), 0) + lines_cached = linecache.getlines(filename) + self.assertEqual(lines, lines_cached) + linecache.clearcache() + empty = linecache.getlines(filename) + self.assertEqual(empty, []) + def test_invalid_names(self): for name, desc in [ ('\x00', 'NUL bytes filename'), diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index 529cb2dc2f1..12b61e76423 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -736,6 +736,7 @@ def remove_loop(fname, tries): @threading_helper.requires_working_threading() @skip_if_asan_fork @skip_if_tsan_fork + @unittest.skip("TODO: RUSTPYTHON; Flaky") def test_post_fork_child_no_deadlock(self): """Ensure child logging locks are not held; bpo-6721 & bpo-36533.""" class _OurHandler(logging.Handler): @@ -1115,7 +1116,6 @@ class SMTPHandlerTest(BaseTest): # bpo-14314, bpo-19665, bpo-34092: don't wait forever TIMEOUT = support.LONG_TIMEOUT - @unittest.skip("TODO: RUSTPYTHON; hangs") def test_basic(self): sockmap = {} server = TestSMTPServer((socket_helper.HOST, 0), self.process_message, 0.001, @@ -2153,7 +2153,6 @@ def handle_request(self, request): request.end_headers() self.handled.set() - @unittest.skip('TODO: RUSTPYTHON; flaky test') def test_output(self): # The log message sent to the HTTPHandler is properly received. logger = logging.getLogger("http") @@ -4058,6 +4057,7 @@ def _mpinit_issue121723(qspec, message_to_log): # log a message (this creates a record put in the queue) logging.getLogger().info(message_to_log) + @unittest.skip('TODO: RUSTPYTHON, flaky EOFError') # TODO: RUSTPYTHON - SemLock not implemented on Windows @unittest.expectedFailureIfWindows("TODO: RUSTPYTHON") @skip_if_tsan_fork diff --git a/Lib/test/test_lzma.py b/Lib/test/test_lzma.py index 1bac61f59e1..1bfc9551ce3 100644 --- a/Lib/test/test_lzma.py +++ b/Lib/test/test_lzma.py @@ -1,4 +1,3 @@ -import _compression import array from io import BytesIO, UnsupportedOperation, DEFAULT_BUFFER_SIZE import os @@ -7,6 +6,7 @@ import sys from test import support import unittest +from compression._common import _streams from test.support import _4G, bigmemtest from test.support.import_helper import import_module @@ -22,8 +22,7 @@ class CompressorDecompressorTestCase(unittest.TestCase): # Test error cases. - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_simple_bad_args(self): self.assertRaises(TypeError, LZMACompressor, []) self.assertRaises(TypeError, LZMACompressor, format=3.45) @@ -64,8 +63,7 @@ def test_simple_bad_args(self): lzd.decompress(empty) self.assertRaises(EOFError, lzd.decompress, b"quux") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_bad_filter_spec(self): self.assertRaises(TypeError, LZMACompressor, filters=[b"wobsite"]) self.assertRaises(ValueError, LZMACompressor, filters=[{"xyzzy": 3}]) @@ -82,8 +80,7 @@ def test_decompressor_after_eof(self): lzd.decompress(COMPRESSED_XZ) self.assertRaises(EOFError, lzd.decompress, b"nyan") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompressor_memlimit(self): lzd = LZMADecompressor(memlimit=1024) self.assertRaises(LZMAError, lzd.decompress, COMPRESSED_XZ) @@ -104,8 +101,7 @@ def _test_decompressor(self, lzd, data, check, unused_data=b""): self.assertTrue(lzd.eof) self.assertEqual(lzd.unused_data, unused_data) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompressor_auto(self): lzd = LZMADecompressor() self._test_decompressor(lzd, COMPRESSED_XZ, lzma.CHECK_CRC64) @@ -113,44 +109,37 @@ def test_decompressor_auto(self): lzd = LZMADecompressor() self._test_decompressor(lzd, COMPRESSED_ALONE, lzma.CHECK_NONE) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompressor_xz(self): lzd = LZMADecompressor(lzma.FORMAT_XZ) self._test_decompressor(lzd, COMPRESSED_XZ, lzma.CHECK_CRC64) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompressor_alone(self): lzd = LZMADecompressor(lzma.FORMAT_ALONE) self._test_decompressor(lzd, COMPRESSED_ALONE, lzma.CHECK_NONE) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompressor_raw_1(self): lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_1) self._test_decompressor(lzd, COMPRESSED_RAW_1, lzma.CHECK_NONE) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompressor_raw_2(self): lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_2) self._test_decompressor(lzd, COMPRESSED_RAW_2, lzma.CHECK_NONE) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompressor_raw_3(self): lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_3) self._test_decompressor(lzd, COMPRESSED_RAW_3, lzma.CHECK_NONE) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompressor_raw_4(self): lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4) self._test_decompressor(lzd, COMPRESSED_RAW_4, lzma.CHECK_NONE) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompressor_chunks(self): lzd = LZMADecompressor() out = [] @@ -163,8 +152,7 @@ def test_decompressor_chunks(self): self.assertTrue(lzd.eof) self.assertEqual(lzd.unused_data, b"") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompressor_chunks_empty(self): lzd = LZMADecompressor() out = [] @@ -180,8 +168,7 @@ def test_decompressor_chunks_empty(self): self.assertTrue(lzd.eof) self.assertEqual(lzd.unused_data, b"") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompressor_chunks_maxsize(self): lzd = LZMADecompressor() max_length = 100 @@ -273,16 +260,14 @@ def test_decompressor_inputbuf_3(self): out.append(lzd.decompress(COMPRESSED_XZ[300:])) self.assertEqual(b''.join(out), INPUT) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompressor_unused_data(self): lzd = LZMADecompressor() extra = b"fooblibar" self._test_decompressor(lzd, COMPRESSED_XZ + extra, lzma.CHECK_CRC64, unused_data=extra) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompressor_bad_input(self): lzd = LZMADecompressor() self.assertRaises(LZMAError, lzd.decompress, COMPRESSED_RAW_1) @@ -296,8 +281,7 @@ def test_decompressor_bad_input(self): lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_1) self.assertRaises(LZMAError, lzd.decompress, COMPRESSED_XZ) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompressor_bug_28275(self): # Test coverage for Issue 28275 lzd = LZMADecompressor() @@ -307,32 +291,28 @@ def test_decompressor_bug_28275(self): # Test that LZMACompressor->LZMADecompressor preserves the input data. - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_roundtrip_xz(self): lzc = LZMACompressor() cdata = lzc.compress(INPUT) + lzc.flush() lzd = LZMADecompressor() self._test_decompressor(lzd, cdata, lzma.CHECK_CRC64) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_roundtrip_alone(self): lzc = LZMACompressor(lzma.FORMAT_ALONE) cdata = lzc.compress(INPUT) + lzc.flush() lzd = LZMADecompressor() self._test_decompressor(lzd, cdata, lzma.CHECK_NONE) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_roundtrip_raw(self): lzc = LZMACompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4) cdata = lzc.compress(INPUT) + lzc.flush() lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4) self._test_decompressor(lzd, cdata, lzma.CHECK_NONE) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_roundtrip_raw_empty(self): lzc = LZMACompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4) cdata = lzc.compress(INPUT) @@ -343,8 +323,7 @@ def test_roundtrip_raw_empty(self): lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4) self._test_decompressor(lzd, cdata, lzma.CHECK_NONE) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_roundtrip_chunks(self): lzc = LZMACompressor() cdata = [] @@ -355,8 +334,7 @@ def test_roundtrip_chunks(self): lzd = LZMADecompressor() self._test_decompressor(lzd, cdata, lzma.CHECK_CRC64) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_roundtrip_empty_chunks(self): lzc = LZMACompressor() cdata = [] @@ -372,8 +350,7 @@ def test_roundtrip_empty_chunks(self): # LZMADecompressor intentionally does not handle concatenated streams. - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompressor_multistream(self): lzd = LZMADecompressor() self._test_decompressor(lzd, COMPRESSED_XZ + COMPRESSED_ALONE, @@ -409,8 +386,6 @@ def test_decompressor_bigmem(self, size): # Pickling raises an exception; there's no way to serialize an lzma_stream. - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_pickle(self): for proto in range(pickle.HIGHEST_PROTOCOL + 1): with self.assertRaises(TypeError): @@ -436,8 +411,7 @@ class CompressDecompressFunctionTestCase(unittest.TestCase): # Test error cases: - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_bad_args(self): self.assertRaises(TypeError, lzma.compress) self.assertRaises(TypeError, lzma.compress, []) @@ -465,24 +439,22 @@ def test_bad_args(self): lzma.decompress(b"", format=lzma.FORMAT_XZ, filters=FILTERS_RAW_1) with self.assertRaises(ValueError): lzma.decompress( - b"", format=lzma.FORMAT_ALONE, filters=FILTERS_RAW_1) + b"", format=lzma.FORMAT_ALONE, filters=FILTERS_RAW_1) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompress_memlimit(self): with self.assertRaises(LZMAError): lzma.decompress(COMPRESSED_XZ, memlimit=1024) with self.assertRaises(LZMAError): lzma.decompress( - COMPRESSED_XZ, format=lzma.FORMAT_XZ, memlimit=1024) + COMPRESSED_XZ, format=lzma.FORMAT_XZ, memlimit=1024) with self.assertRaises(LZMAError): lzma.decompress( - COMPRESSED_ALONE, format=lzma.FORMAT_ALONE, memlimit=1024) + COMPRESSED_ALONE, format=lzma.FORMAT_ALONE, memlimit=1024) # Test LZMADecompressor on known-good input data. - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompress_good_input(self): ddata = lzma.decompress(COMPRESSED_XZ) self.assertEqual(ddata, INPUT) @@ -497,23 +469,22 @@ def test_decompress_good_input(self): self.assertEqual(ddata, INPUT) ddata = lzma.decompress( - COMPRESSED_RAW_1, lzma.FORMAT_RAW, filters=FILTERS_RAW_1) + COMPRESSED_RAW_1, lzma.FORMAT_RAW, filters=FILTERS_RAW_1) self.assertEqual(ddata, INPUT) ddata = lzma.decompress( - COMPRESSED_RAW_2, lzma.FORMAT_RAW, filters=FILTERS_RAW_2) + COMPRESSED_RAW_2, lzma.FORMAT_RAW, filters=FILTERS_RAW_2) self.assertEqual(ddata, INPUT) ddata = lzma.decompress( - COMPRESSED_RAW_3, lzma.FORMAT_RAW, filters=FILTERS_RAW_3) + COMPRESSED_RAW_3, lzma.FORMAT_RAW, filters=FILTERS_RAW_3) self.assertEqual(ddata, INPUT) ddata = lzma.decompress( - COMPRESSED_RAW_4, lzma.FORMAT_RAW, filters=FILTERS_RAW_4) + COMPRESSED_RAW_4, lzma.FORMAT_RAW, filters=FILTERS_RAW_4) self.assertEqual(ddata, INPUT) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompress_incomplete_input(self): self.assertRaises(LZMAError, lzma.decompress, COMPRESSED_XZ[:128]) self.assertRaises(LZMAError, lzma.decompress, COMPRESSED_ALONE[:128]) @@ -526,8 +497,7 @@ def test_decompress_incomplete_input(self): self.assertRaises(LZMAError, lzma.decompress, COMPRESSED_RAW_4[:128], format=lzma.FORMAT_RAW, filters=FILTERS_RAW_4) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompress_bad_input(self): with self.assertRaises(LZMAError): lzma.decompress(COMPRESSED_BOGUS) @@ -543,8 +513,7 @@ def test_decompress_bad_input(self): # Test that compress()->decompress() preserves the input data. - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_roundtrip(self): cdata = lzma.compress(INPUT) ddata = lzma.decompress(cdata) @@ -570,14 +539,12 @@ def test_decompress_multistream(self): # Test robust handling of non-LZMA data following the compressed stream(s). - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompress_trailing_junk(self): ddata = lzma.decompress(COMPRESSED_XZ + COMPRESSED_BOGUS) self.assertEqual(ddata, INPUT) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_decompress_multistream_trailing_junk(self): ddata = lzma.decompress(COMPRESSED_XZ * 3 + COMPRESSED_BOGUS) self.assertEqual(ddata, INPUT * 3) @@ -614,8 +581,7 @@ def test_init(self): self.assertIsInstance(f, LZMAFile) self.assertEqual(f.mode, "wb") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_init_with_PathLike_filename(self): filename = FakePath(TESTFN) with TempFile(filename, COMPRESSED_XZ): @@ -696,8 +662,7 @@ def test_init_bad_mode(self): with self.assertRaises(ValueError): LZMAFile(BytesIO(COMPRESSED_XZ), "rw") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_init_bad_check(self): with self.assertRaises(TypeError): LZMAFile(BytesIO(), "w", check=b"asd") @@ -718,6 +683,7 @@ def test_init_bad_check(self): with self.assertRaises(ValueError): LZMAFile(BytesIO(COMPRESSED_XZ), check=lzma.CHECK_UNKNOWN) + @unittest.expectedFailure # TODO: RUSTPYTHON def test_init_bad_preset(self): with self.assertRaises(TypeError): LZMAFile(BytesIO(), "w", preset=4.39) @@ -725,18 +691,19 @@ def test_init_bad_preset(self): LZMAFile(BytesIO(), "w", preset=10) with self.assertRaises(LZMAError): LZMAFile(BytesIO(), "w", preset=23) - with self.assertRaises(OverflowError): + with self.assertRaises(ValueError): LZMAFile(BytesIO(), "w", preset=-1) - with self.assertRaises(OverflowError): + with self.assertRaises(ValueError): LZMAFile(BytesIO(), "w", preset=-7) + with self.assertRaises(OverflowError): + LZMAFile(BytesIO(), "w", preset=2**1000) with self.assertRaises(TypeError): LZMAFile(BytesIO(), "w", preset="foo") # Cannot specify a preset with mode="r". with self.assertRaises(ValueError): LZMAFile(BytesIO(COMPRESSED_XZ), preset=3) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_init_bad_filter_spec(self): with self.assertRaises(TypeError): LZMAFile(BytesIO(), "w", filters=[b"wobsite"]) @@ -754,8 +721,7 @@ def test_init_bad_filter_spec(self): LZMAFile(BytesIO(), "w", filters=[{"id": lzma.FILTER_X86, "foo": 0}]) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_init_with_preset_and_filters(self): with self.assertRaises(ValueError): LZMAFile(BytesIO(), "w", format=lzma.FORMAT_RAW, @@ -874,8 +840,7 @@ def test_writable(self): f.close() self.assertRaises(ValueError, f.writable) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_read(self): with LZMAFile(BytesIO(COMPRESSED_XZ)) as f: self.assertEqual(f.read(), INPUT) @@ -923,8 +888,7 @@ def test_read_10(self): chunks.append(result) self.assertEqual(b"".join(chunks), INPUT) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_read_multistream(self): with LZMAFile(BytesIO(COMPRESSED_XZ * 5)) as f: self.assertEqual(f.read(), INPUT * 5) @@ -937,22 +901,20 @@ def test_read_multistream(self): def test_read_multistream_buffer_size_aligned(self): # Test the case where a stream boundary coincides with the end # of the raw read buffer. - saved_buffer_size = _compression.BUFFER_SIZE - _compression.BUFFER_SIZE = len(COMPRESSED_XZ) + saved_buffer_size = _streams.BUFFER_SIZE + _streams.BUFFER_SIZE = len(COMPRESSED_XZ) try: with LZMAFile(BytesIO(COMPRESSED_XZ * 5)) as f: self.assertEqual(f.read(), INPUT * 5) finally: - _compression.BUFFER_SIZE = saved_buffer_size + _streams.BUFFER_SIZE = saved_buffer_size - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_read_trailing_junk(self): with LZMAFile(BytesIO(COMPRESSED_XZ + COMPRESSED_BOGUS)) as f: self.assertEqual(f.read(), INPUT) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_read_multistream_trailing_junk(self): with LZMAFile(BytesIO(COMPRESSED_XZ * 5 + COMPRESSED_BOGUS)) as f: self.assertEqual(f.read(), INPUT * 5) @@ -1058,8 +1020,7 @@ def test_read_bad_args(self): with LZMAFile(BytesIO(COMPRESSED_XZ)) as f: self.assertRaises(TypeError, f.read, float()) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_read_bad_data(self): with LZMAFile(BytesIO(COMPRESSED_BOGUS)) as f: self.assertRaises(LZMAError, f.read) @@ -1105,20 +1066,19 @@ def test_peek(self): with LZMAFile(BytesIO(COMPRESSED_XZ)) as f: result = f.peek() self.assertGreater(len(result), 0) - self.assertTrue(INPUT.startswith(result)) + self.assertStartsWith(INPUT, result) self.assertEqual(f.read(), INPUT) with LZMAFile(BytesIO(COMPRESSED_XZ)) as f: result = f.peek(10) self.assertGreater(len(result), 0) - self.assertTrue(INPUT.startswith(result)) + self.assertStartsWith(INPUT, result) self.assertEqual(f.read(), INPUT) def test_peek_bad_args(self): with LZMAFile(BytesIO(), "w") as f: self.assertRaises(ValueError, f.peek) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_iterator(self): with BytesIO(INPUT) as f: lines = f.readlines() @@ -1150,16 +1110,15 @@ def test_readlines(self): def test_decompress_limited(self): """Decompressed data buffering should be limited""" bomb = lzma.compress(b'\0' * int(2e6), preset=6) - self.assertLess(len(bomb), _compression.BUFFER_SIZE) + self.assertLess(len(bomb), _streams.BUFFER_SIZE) decomp = LZMAFile(BytesIO(bomb)) self.assertEqual(decomp.read(1), b'\0') max_decomp = 1 + DEFAULT_BUFFER_SIZE self.assertLessEqual(decomp._buffer.raw.tell(), max_decomp, - "Excessive amount of data was decompressed") + "Excessive amount of data was decompressed") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_write(self): with BytesIO() as dst: with LZMAFile(dst, "w") as f: @@ -1428,8 +1387,7 @@ def test_tell_bad_args(self): f.close() self.assertRaises(ValueError, f.tell) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_issue21872(self): # sometimes decompress data incompletely @@ -1513,8 +1471,7 @@ def test_filename(self): with lzma.open(TESTFN, "rb") as f: self.assertEqual(f.read(), INPUT * 2) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_with_pathlike_filename(self): filename = FakePath(TESTFN) with TempFile(filename): @@ -1541,8 +1498,7 @@ def test_bad_params(self): with self.assertRaises(ValueError): lzma.open(TESTFN, "rb", newline="\n") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_format_and_filters(self): # Test non-default format and filter chain. options = {"format": lzma.FORMAT_RAW, "filters": FILTERS_RAW_1} @@ -1573,8 +1529,7 @@ def test_encoding_error_handler(self): with lzma.open(bio, "rt", encoding="ascii", errors="ignore") as f: self.assertEqual(f.read(), "foobar") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_newline(self): # Test with explicit newline (universal newline mode disabled). text = INPUT.decode("ascii") @@ -1599,8 +1554,7 @@ def test_x_mode(self): class MiscellaneousTestCase(unittest.TestCase): - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_is_check_supported(self): # CHECK_NONE and CHECK_CRC32 should always be supported, # regardless of the options liblzma was compiled with. @@ -1613,8 +1567,7 @@ def test_is_check_supported(self): # This value should not be a valid check ID. self.assertFalse(lzma.is_check_supported(lzma.CHECK_UNKNOWN)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test__encode_filter_properties(self): with self.assertRaises(TypeError): lzma._encode_filter_properties(b"not a dict") @@ -1624,20 +1577,19 @@ def test__encode_filter_properties(self): lzma._encode_filter_properties({"id": lzma.FILTER_LZMA2, "junk": 12}) with self.assertRaises(lzma.LZMAError): lzma._encode_filter_properties({"id": lzma.FILTER_DELTA, - "dist": 9001}) + "dist": 9001}) # Test with parameters used by zipfile module. props = lzma._encode_filter_properties({ - "id": lzma.FILTER_LZMA1, - "pb": 2, - "lp": 0, - "lc": 3, - "dict_size": 8 << 20, - }) + "id": lzma.FILTER_LZMA1, + "pb": 2, + "lp": 0, + "lc": 3, + "dict_size": 8 << 20, + }) self.assertEqual(props, b"]\x00\x00\x80\x00") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test__decode_filter_properties(self): with self.assertRaises(TypeError): lzma._decode_filter_properties(lzma.FILTER_X86, {"should be": bytes}) @@ -1646,7 +1598,7 @@ def test__decode_filter_properties(self): # Test with parameters used by zipfile module. filterspec = lzma._decode_filter_properties( - lzma.FILTER_LZMA1, b"]\x00\x00\x80\x00") + lzma.FILTER_LZMA1, b"]\x00\x00\x80\x00") self.assertEqual(filterspec["id"], lzma.FILTER_LZMA1) self.assertEqual(filterspec["pb"], 2) self.assertEqual(filterspec["lp"], 0) @@ -1661,11 +1613,10 @@ def test__decode_filter_properties(self): filterspec = lzma._decode_filter_properties(f, b"") self.assertEqual(filterspec, {"id": f}) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_filter_properties_roundtrip(self): spec1 = lzma._decode_filter_properties( - lzma.FILTER_LZMA1, b"]\x00\x00\x80\x00") + lzma.FILTER_LZMA1, b"]\x00\x00\x80\x00") reencoded = lzma._encode_filter_properties(spec1) spec2 = lzma._decode_filter_properties(lzma.FILTER_LZMA1, reencoded) self.assertEqual(spec1, spec2) @@ -2194,4 +2145,4 @@ def test_filter_properties_roundtrip(self): if __name__ == "__main__": - unittest.main() \ No newline at end of file + unittest.main() diff --git a/Lib/test/test_marshal.py b/Lib/test/test_marshal.py index 2161e06b2f2..142b45f6a46 100644 --- a/Lib/test/test_marshal.py +++ b/Lib/test/test_marshal.py @@ -35,7 +35,7 @@ def test_ints(self): self.helper(expected) n = n >> 1 - @unittest.skip("TODO: RUSTPYTHON; hang") + @unittest.expectedFailure # TODO: RUSTPYTHON def test_int64(self): # Simulate int marshaling with TYPE_INT64. maxint64 = (1 << 63) - 1 @@ -232,7 +232,7 @@ def check(s): self.assertRaises(ValueError, marshal.loads, s) run_tests(2**20, check) - @unittest.skip("TODO: RUSTPYTHON; segfault") + @unittest.expectedFailure # TODO: RUSTPYTHON; segfault def test_recursion_limit(self): # Create a deeply nested structure. head = last = [] diff --git a/Lib/test/test_math.py b/Lib/test/test_math.py index 1a4d257586b..d14336f8bac 100644 --- a/Lib/test/test_math.py +++ b/Lib/test/test_math.py @@ -573,6 +573,8 @@ def testFloor(self): #self.assertEqual(math.ceil(NINF), NINF) #self.assertTrue(math.isnan(math.floor(NAN))) + class TestFloorIsNone(float): + __floor__ = None class TestFloor: def __floor__(self): return 42 @@ -588,6 +590,7 @@ class TestBadFloor: self.assertEqual(math.floor(FloatLike(41.9)), 41) self.assertRaises(TypeError, math.floor, TestNoFloor()) self.assertRaises(ValueError, math.floor, TestBadFloor()) + self.assertRaises(TypeError, math.floor, TestFloorIsNone(3.5)) t = TestNoFloor() t.__floor__ = lambda *args: args @@ -1125,6 +1128,15 @@ def __index__(self): with self.assertRaises(TypeError): math.isqrt(value) + @support.bigmemtest(2**32, memuse=0.85) + def test_isqrt_huge(self, size): + if size & 1: + size += 1 + v = 1 << size + w = math.isqrt(v) + self.assertEqual(w.bit_length(), size // 2 + 1) + self.assertEqual(w.bit_count(), 1) + def test_lcm(self): lcm = math.lcm self.assertEqual(lcm(0, 0), 0) @@ -1272,6 +1284,13 @@ def testLog10(self): self.assertEqual(math.log(INF), INF) self.assertTrue(math.isnan(math.log10(NAN))) + @support.bigmemtest(2**32, memuse=0.2) + def test_log_huge_integer(self, size): + v = 1 << size + self.assertAlmostEqual(math.log2(v), size) + self.assertAlmostEqual(math.log(v), size * 0.6931471805599453) + self.assertAlmostEqual(math.log10(v), size * 0.3010299956639812) + def testSumProd(self): sumprod = math.sumprod Decimal = decimal.Decimal @@ -1380,7 +1399,6 @@ def test_sumprod_accuracy(self): self.assertEqual(sumprod([True, False] * 10, [0.1] * 20), 1.0) self.assertEqual(sumprod([1.0, 10E100, 1.0, -10E100], [1.0]*4), 2.0) - @unittest.skip("TODO: RUSTPYTHON, Taking a few minutes.") @support.requires_resource('cpu') def test_sumprod_stress(self): sumprod = math.sumprod @@ -2020,7 +2038,6 @@ def test_exceptions(self): else: self.fail("sqrt(-1) didn't raise ValueError") - @unittest.expectedFailure # TODO: RUSTPYTHON @requires_IEEE_754 def test_testfile(self): # Some tests need to be skipped on ancient OS X versions. @@ -2495,6 +2512,46 @@ def test_input_exceptions(self): self.assertRaises(TypeError, math.atan2, 1.0) self.assertRaises(TypeError, math.atan2, 1.0, 2.0, 3.0) + def test_exception_messages(self): + x = -1.1 + with self.assertRaisesRegex(ValueError, + f"expected a nonnegative input, got {x}"): + math.sqrt(x) + with self.assertRaisesRegex(ValueError, + f"expected a positive input, got {x}"): + math.log(x) + with self.assertRaisesRegex(ValueError, + f"expected a positive input, got {x}"): + math.log(123, x) + with self.assertRaisesRegex(ValueError, + f"expected a positive input, got {x}"): + math.log(x, 123) + with self.assertRaisesRegex(ValueError, + f"expected a positive input, got {x}"): + math.log2(x) + with self.assertRaisesRegex(ValueError, + f"expected a positive input, got {x}"): + math.log10(x) + x = decimal.Decimal('-1.1') + with self.assertRaisesRegex(ValueError, + f"expected a positive input, got {x}"): + math.log(x) + x = fractions.Fraction(1, 10**400) + with self.assertRaisesRegex(ValueError, + f"expected a positive input, got {float(x)}"): + math.log(x) + x = -123 + with self.assertRaisesRegex(ValueError, + "expected a positive input$"): + math.log(x) + with self.assertRaisesRegex(ValueError, + f"expected a noninteger or positive integer, got {x}"): + math.gamma(x) + x = 1.0 + with self.assertRaisesRegex(ValueError, + f"expected a number between -1 and 1, got {x}"): + math.atanh(x) + # Custom assertions. def assertIsNaN(self, value): @@ -2724,6 +2781,9 @@ def test_fma_infinities(self): or (sys.platform == "android" and platform.machine() == "x86_64") or support.linked_to_musl(), # gh-131032 f"this platform doesn't implement IEE 754-2008 properly") + # gh-131032: musl is fixed but the fix is not yet released; when the fixed + # version is known change this to: + # or support.linked_to_musl() < (1, <m>, <p>) def test_fma_zero_result(self): nonnegative_finites = [0.0, 1e-300, 2.3, 1e300] diff --git a/Lib/test/test_memoryio.py b/Lib/test/test_memoryio.py index 07d9d38d6e4..b6b95143621 100644 --- a/Lib/test/test_memoryio.py +++ b/Lib/test/test_memoryio.py @@ -745,11 +745,6 @@ def test_init(self): def test_issue5449(self): super().test_issue5449() - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_pickling(self): - super().test_pickling() - def test_read(self): super().test_read() @@ -777,8 +772,6 @@ def test_truncate(self): def test_write(self): super().test_write() - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_getstate(self): memio = self.ioclass() state = memio.__getstate__() @@ -911,11 +904,6 @@ def test_newline_none(self): def test_newlines_property(self): super().test_newlines_property() - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_pickling(self): - super().test_pickling() - def test_read(self): super().test_read() @@ -954,8 +942,6 @@ def test_widechar(self): self.assertEqual(memio.tell(), len(buf) * 2) self.assertEqual(memio.getvalue(), buf + buf) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_getstate(self): memio = self.ioclass() state = memio.__getstate__() @@ -1006,21 +992,11 @@ def test_newline_cr(self): def test_newline_crlf(self): super().test_newline_crlf() - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_newline_default(self): - super().test_newline_default() - # TODO: RUSTPYTHON @unittest.expectedFailure def test_newline_empty(self): super().test_newline_empty() - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_newline_lf(self): - super().test_newline_lf() - # TODO: RUSTPYTHON @unittest.expectedFailure def test_newline_none(self): diff --git a/Lib/test/test_metaclass.py b/Lib/test/test_metaclass.py new file mode 100644 index 00000000000..1707df9075a --- /dev/null +++ b/Lib/test/test_metaclass.py @@ -0,0 +1,321 @@ +import doctest +import unittest + + +doctests = """ + +Basic class construction. + + >>> class C: + ... def meth(self): print("Hello") + ... + >>> C.__class__ is type + True + >>> a = C() + >>> a.__class__ is C + True + >>> a.meth() + Hello + >>> + +Use *args notation for the bases. + + >>> class A: pass + >>> class B: pass + >>> bases = (A, B) + >>> class C(*bases): pass + >>> C.__bases__ == bases + True + >>> + +Use a trivial metaclass. + + >>> class M(type): + ... pass + ... + >>> class C(metaclass=M): + ... def meth(self): print("Hello") + ... + >>> C.__class__ is M + True + >>> a = C() + >>> a.__class__ is C + True + >>> a.meth() + Hello + >>> + +Use **kwds notation for the metaclass keyword. + + >>> kwds = {'metaclass': M} + >>> class C(**kwds): pass + ... + >>> C.__class__ is M + True + >>> a = C() + >>> a.__class__ is C + True + >>> + +Use a metaclass with a __prepare__ static method. + + >>> class M(type): + ... @staticmethod + ... def __prepare__(*args, **kwds): + ... print("Prepare called:", args, kwds) + ... return dict() + ... def __new__(cls, name, bases, namespace, **kwds): + ... print("New called:", kwds) + ... return type.__new__(cls, name, bases, namespace) + ... def __init__(cls, *args, **kwds): + ... pass + ... + >>> class C(metaclass=M): + ... def meth(self): print("Hello") + ... + Prepare called: ('C', ()) {} + New called: {} + >>> + +Also pass another keyword. + + >>> class C(object, metaclass=M, other="haha"): + ... pass + ... + Prepare called: ('C', (<class 'object'>,)) {'other': 'haha'} + New called: {'other': 'haha'} + >>> C.__class__ is M + True + >>> C.__bases__ == (object,) + True + >>> a = C() + >>> a.__class__ is C + True + >>> + +Check that build_class doesn't mutate the kwds dict. + + >>> kwds = {'metaclass': type} + >>> class C(**kwds): pass + ... + >>> kwds == {'metaclass': type} + True + >>> + +Use various combinations of explicit keywords and **kwds. + + >>> bases = (object,) + >>> kwds = {'metaclass': M, 'other': 'haha'} + >>> class C(*bases, **kwds): pass + ... + Prepare called: ('C', (<class 'object'>,)) {'other': 'haha'} + New called: {'other': 'haha'} + >>> C.__class__ is M + True + >>> C.__bases__ == (object,) + True + >>> class B: pass + >>> kwds = {'other': 'haha'} + >>> class C(B, metaclass=M, *bases, **kwds): pass + ... + Prepare called: ('C', (<class 'test.test_metaclass.B'>, <class 'object'>)) {'other': 'haha'} + New called: {'other': 'haha'} + >>> C.__class__ is M + True + >>> C.__bases__ == (B, object) + True + >>> + +Check for duplicate keywords. + + # TODO: RUSTPYTHON + >>> class C(metaclass=type, metaclass=type): pass # doctest: +SKIP + ... + Traceback (most recent call last): + [...] + SyntaxError: keyword argument repeated: metaclass + >>> + +Another way. + + >>> kwds = {'metaclass': type} + + # TODO: RUSTPYTHON + >>> class C(metaclass=type, **kwds): pass # doctest: +SKIP + ... + Traceback (most recent call last): + [...] + TypeError: __build_class__() got multiple values for keyword argument 'metaclass' + >>> + +Use a __prepare__ method that returns an instrumented dict. + + >>> class LoggingDict(dict): + ... def __setitem__(self, key, value): + ... print("d[%r] = %r" % (key, value)) + ... dict.__setitem__(self, key, value) + ... + >>> class Meta(type): + ... @staticmethod + ... def __prepare__(name, bases): + ... return LoggingDict() + ... + + # TODO: RUSTPYTHON + >>> class C(metaclass=Meta): # doctest: +SKIP + ... foo = 2+2 + ... foo = 42 + ... bar = 123 + ... + d['__module__'] = 'test.test_metaclass' + d['__qualname__'] = 'C' + d['__firstlineno__'] = 1 + d['foo'] = 4 + d['foo'] = 42 + d['bar'] = 123 + d['__static_attributes__'] = () + >>> + +Use a metaclass that doesn't derive from type. + + >>> def meta(name, bases, namespace, **kwds): + ... print("meta:", name, bases) + ... print("ns:", sorted(namespace.items())) + ... print("kw:", sorted(kwds.items())) + ... return namespace + ... + + # TODO: RUSTPYTHON + >>> class C(metaclass=meta): # doctest: +SKIP + ... a = 42 + ... b = 24 + ... + meta: C () + ns: [('__firstlineno__', 1), ('__module__', 'test.test_metaclass'), ('__qualname__', 'C'), ('__static_attributes__', ()), ('a', 42), ('b', 24)] + kw: [] + + # TODO: RUSTPYTHON + >>> type(C) is dict # doctest: +SKIP + True + + # TODO: RUSTPYTHON + >>> print(sorted(C.items())) # doctest: +SKIP + [('__firstlineno__', 1), ('__module__', 'test.test_metaclass'), ('__qualname__', 'C'), ('__static_attributes__', ()), ('a', 42), ('b', 24)] + >>> + +And again, with a __prepare__ attribute. + + >>> def prepare(name, bases, **kwds): + ... print("prepare:", name, bases, sorted(kwds.items())) + ... return LoggingDict() + ... + >>> meta.__prepare__ = prepare + + # TODO: RUSTPYTHON + >>> class C(metaclass=meta, other="booh"): # doctest: +SKIP + ... a = 1 + ... a = 2 + ... b = 3 + ... + prepare: C () [('other', 'booh')] + d['__module__'] = 'test.test_metaclass' + d['__qualname__'] = 'C' + d['__firstlineno__'] = 1 + d['a'] = 1 + d['a'] = 2 + d['b'] = 3 + d['__static_attributes__'] = () + meta: C () + ns: [('__firstlineno__', 1), ('__module__', 'test.test_metaclass'), ('__qualname__', 'C'), ('__static_attributes__', ()), ('a', 2), ('b', 3)] + kw: [('other', 'booh')] + >>> + +The default metaclass must define a __prepare__() method. + + >>> type.__prepare__() + {} + >>> + +Make sure it works with subclassing. + + >>> class M(type): + ... @classmethod + ... def __prepare__(cls, *args, **kwds): + ... d = super().__prepare__(*args, **kwds) + ... d["hello"] = 42 + ... return d + ... + >>> class C(metaclass=M): + ... print(hello) + ... + 42 + >>> print(C.hello) + 42 + >>> + +Test failures in looking up the __prepare__ method work. + >>> class ObscureException(Exception): + ... pass + >>> class FailDescr: + ... def __get__(self, instance, owner): + ... raise ObscureException + >>> class Meta(type): + ... __prepare__ = FailDescr() + >>> class X(metaclass=Meta): + ... pass + Traceback (most recent call last): + [...] + test.test_metaclass.ObscureException + +Test setting attributes with a non-base type in mro() (gh-127773). + + >>> class Base: + ... value = 1 + ... + >>> class Meta(type): + ... def mro(cls): + ... return (cls, Base, object) + ... + >>> class WeirdClass(metaclass=Meta): + ... pass + ... + >>> Base.value + 1 + + # TODO: RUSTPYTHON; AttributeError: type object 'WeirdClass' has no attribute 'value' + >>> WeirdClass.value # doctest: +SKIP + 1 + >>> Base.value = 2 + >>> Base.value + 2 + + # TODO: RUSTPYTHON; AttributeError: type object 'WeirdClass' has no attribute 'value' + >>> WeirdClass.value # doctest: +SKIP + 2 + >>> Base.value = 3 + >>> Base.value + 3 + + # TODO: RUSTPYTHON; AttributeError: type object 'WeirdClass' has no attribute 'value' + >>> WeirdClass.value # doctest: +SKIP + 3 + +""" + +import sys + +# Trace function introduces __locals__ which causes various tests to fail. +if hasattr(sys, 'gettrace') and sys.gettrace(): + __test__ = {} +else: + __test__ = {'doctests' : doctests} + +def load_tests(loader, tests, pattern): + tests.addTest(doctest.DocTestSuite()) + return tests + + +if __name__ == "__main__": + # set __name__ to match doctest expectations + __name__ = "test.test_metaclass" + unittest.main() diff --git a/Lib/test/test_mimetypes.py b/Lib/test/test_mimetypes.py index 23092ffd0f3..c1806b1c133 100644 --- a/Lib/test/test_mimetypes.py +++ b/Lib/test/test_mimetypes.py @@ -1,12 +1,18 @@ import io -import locale import mimetypes -import pathlib +import os +import shlex import sys -import unittest - -from test import support +import unittest.mock from platform import win32_edition +from test import support +from test.support import cpython_only, force_not_colorized, os_helper +from test.support.import_helper import ensure_lazy_imports + +try: + import _winapi +except ImportError: + _winapi = None def setUpModule(): @@ -28,15 +34,30 @@ class MimeTypesTestCase(unittest.TestCase): def setUp(self): self.db = mimetypes.MimeTypes() + def test_case_sensitivity(self): + eq = self.assertEqual + eq(self.db.guess_file_type("foobar.html"), ("text/html", None)) + eq(self.db.guess_type("scheme:foobar.html"), ("text/html", None)) + eq(self.db.guess_file_type("foobar.HTML"), ("text/html", None)) + eq(self.db.guess_type("scheme:foobar.HTML"), ("text/html", None)) + eq(self.db.guess_file_type("foobar.tgz"), ("application/x-tar", "gzip")) + eq(self.db.guess_type("scheme:foobar.tgz"), ("application/x-tar", "gzip")) + eq(self.db.guess_file_type("foobar.TGZ"), ("application/x-tar", "gzip")) + eq(self.db.guess_type("scheme:foobar.TGZ"), ("application/x-tar", "gzip")) + eq(self.db.guess_file_type("foobar.tar.Z"), ("application/x-tar", "compress")) + eq(self.db.guess_type("scheme:foobar.tar.Z"), ("application/x-tar", "compress")) + eq(self.db.guess_file_type("foobar.tar.z"), (None, None)) + eq(self.db.guess_type("scheme:foobar.tar.z"), (None, None)) + def test_default_data(self): eq = self.assertEqual - eq(self.db.guess_type("foo.html"), ("text/html", None)) - eq(self.db.guess_type("foo.HTML"), ("text/html", None)) - eq(self.db.guess_type("foo.tgz"), ("application/x-tar", "gzip")) - eq(self.db.guess_type("foo.tar.gz"), ("application/x-tar", "gzip")) - eq(self.db.guess_type("foo.tar.Z"), ("application/x-tar", "compress")) - eq(self.db.guess_type("foo.tar.bz2"), ("application/x-tar", "bzip2")) - eq(self.db.guess_type("foo.tar.xz"), ("application/x-tar", "xz")) + eq(self.db.guess_file_type("foo.html"), ("text/html", None)) + eq(self.db.guess_file_type("foo.HTML"), ("text/html", None)) + eq(self.db.guess_file_type("foo.tgz"), ("application/x-tar", "gzip")) + eq(self.db.guess_file_type("foo.tar.gz"), ("application/x-tar", "gzip")) + eq(self.db.guess_file_type("foo.tar.Z"), ("application/x-tar", "compress")) + eq(self.db.guess_file_type("foo.tar.bz2"), ("application/x-tar", "bzip2")) + eq(self.db.guess_file_type("foo.tar.xz"), ("application/x-tar", "xz")) def test_data_urls(self): eq = self.assertEqual @@ -50,12 +71,10 @@ def test_file_parsing(self): eq = self.assertEqual sio = io.StringIO("x-application/x-unittest pyunit\n") self.db.readfp(sio) - eq(self.db.guess_type("foo.pyunit"), + eq(self.db.guess_file_type("foo.pyunit"), ("x-application/x-unittest", None)) eq(self.db.guess_extension("x-application/x-unittest"), ".pyunit") - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_read_mime_types(self): eq = self.assertEqual @@ -64,32 +83,40 @@ def test_read_mime_types(self): with os_helper.temp_dir() as directory: data = "x-application/x-unittest pyunit\n" - file = pathlib.Path(directory, "sample.mimetype") - file.write_text(data) + file = os.path.join(directory, "sample.mimetype") + with open(file, 'w', encoding="utf-8") as f: + f.write(data) mime_dict = mimetypes.read_mime_types(file) eq(mime_dict[".pyunit"], "x-application/x-unittest") + data = "x-application/x-unittest2 pyunit2\n" + file = os.path.join(directory, "sample2.mimetype") + with open(file, 'w', encoding="utf-8") as f: + f.write(data) + mime_dict = mimetypes.read_mime_types(os_helper.FakePath(file)) + eq(mime_dict[".pyunit2"], "x-application/x-unittest2") + # bpo-41048: read_mime_types should read the rule file with 'utf-8' encoding. # Not with locale encoding. _bootlocale has been imported because io.open(...) # uses it. - with os_helper.temp_dir() as directory: - data = "application/no-mans-land Fran\u00E7ais" - file = pathlib.Path(directory, "sample.mimetype") - file.write_text(data, encoding='utf-8') - import _bootlocale - with support.swap_attr(_bootlocale, 'getpreferredencoding', lambda do_setlocale=True: 'ASCII'): - mime_dict = mimetypes.read_mime_types(file) - eq(mime_dict[".Français"], "application/no-mans-land") + data = "application/no-mans-land Fran\u00E7ais" + filename = "filename" + fp = io.StringIO(data) + with unittest.mock.patch.object(mimetypes, 'open', + return_value=fp) as mock_open: + mime_dict = mimetypes.read_mime_types(filename) + mock_open.assert_called_with(filename, encoding='utf-8') + eq(mime_dict[".Français"], "application/no-mans-land") def test_non_standard_types(self): eq = self.assertEqual # First try strict - eq(self.db.guess_type('foo.xul', strict=True), (None, None)) + eq(self.db.guess_file_type('foo.xul', strict=True), (None, None)) eq(self.db.guess_extension('image/jpg', strict=True), None) # And then non-strict - eq(self.db.guess_type('foo.xul', strict=False), ('text/xul', None)) - eq(self.db.guess_type('foo.XUL', strict=False), ('text/xul', None)) - eq(self.db.guess_type('foo.invalid', strict=False), (None, None)) + eq(self.db.guess_file_type('foo.xul', strict=False), ('text/xul', None)) + eq(self.db.guess_file_type('foo.XUL', strict=False), ('text/xul', None)) + eq(self.db.guess_file_type('foo.invalid', strict=False), (None, None)) eq(self.db.guess_extension('image/jpg', strict=False), '.jpg') eq(self.db.guess_extension('image/JPG', strict=False), '.jpg') @@ -99,37 +126,77 @@ def test_filename_with_url_delimiters(self): # compared to when interpreted as filename because of the semicolon. eq = self.assertEqual gzip_expected = ('application/x-tar', 'gzip') - eq(self.db.guess_type(";1.tar.gz"), gzip_expected) - eq(self.db.guess_type("?1.tar.gz"), gzip_expected) - eq(self.db.guess_type("#1.tar.gz"), gzip_expected) - eq(self.db.guess_type("#1#.tar.gz"), gzip_expected) - eq(self.db.guess_type(";1#.tar.gz"), gzip_expected) - eq(self.db.guess_type(";&1=123;?.tar.gz"), gzip_expected) - eq(self.db.guess_type("?k1=v1&k2=v2.tar.gz"), gzip_expected) + for name in ( + ';1.tar.gz', + '?1.tar.gz', + '#1.tar.gz', + '#1#.tar.gz', + ';1#.tar.gz', + ';&1=123;?.tar.gz', + '?k1=v1&k2=v2.tar.gz', + ): + for prefix in ('', '/', '\\', + 'c:', 'c:/', 'c:\\', 'c:/d/', 'c:\\d\\', + '//share/server/', '\\\\share\\server\\'): + path = prefix + name + with self.subTest(path=path): + eq(self.db.guess_file_type(path), gzip_expected) + eq(self.db.guess_type(path), gzip_expected) + expected = (None, None) if os.name == 'nt' else gzip_expected + for prefix in ('//', '\\\\', '//share/', '\\\\share\\'): + path = prefix + name + with self.subTest(path=path): + eq(self.db.guess_file_type(path), expected) + eq(self.db.guess_type(path), expected) + eq(self.db.guess_file_type(r" \"\`;b&b&c |.tar.gz"), gzip_expected) eq(self.db.guess_type(r" \"\`;b&b&c |.tar.gz"), gzip_expected) + eq(self.db.guess_file_type(r'foo/.tar.gz'), (None, 'gzip')) + eq(self.db.guess_type(r'foo/.tar.gz'), (None, 'gzip')) + expected = (None, 'gzip') if os.name == 'nt' else gzip_expected + eq(self.db.guess_file_type(r'foo\.tar.gz'), expected) + eq(self.db.guess_type(r'foo\.tar.gz'), expected) + eq(self.db.guess_type(r'scheme:foo\.tar.gz'), gzip_expected) + + def test_url(self): + result = self.db.guess_type('http://example.com/host.html') + result = self.db.guess_type('http://host.html') + msg = 'URL only has a host name, not a file' + self.assertSequenceEqual(result, (None, None), msg) + result = self.db.guess_type('http://example.com/host.html') + msg = 'Should be text/html' + self.assertSequenceEqual(result, ('text/html', None), msg) + result = self.db.guess_type('http://example.com/host.html#x.tar') + self.assertSequenceEqual(result, ('text/html', None)) + result = self.db.guess_type('http://example.com/host.html?q=x.tar') + self.assertSequenceEqual(result, ('text/html', None)) + def test_guess_all_types(self): - eq = self.assertEqual - unless = self.assertTrue # First try strict. Use a set here for testing the results because if # test_urllib2 is run before test_mimetypes, global state is modified # such that the 'all' set will have more items in it. - all = set(self.db.guess_all_extensions('text/plain', strict=True)) - unless(all >= set(['.bat', '.c', '.h', '.ksh', '.pl', '.txt'])) + all = self.db.guess_all_extensions('text/plain', strict=True) + self.assertTrue(set(all) >= {'.bat', '.c', '.h', '.ksh', '.pl', '.txt'}) + self.assertEqual(len(set(all)), len(all)) # no duplicates # And now non-strict all = self.db.guess_all_extensions('image/jpg', strict=False) - all.sort() - eq(all, ['.jpg']) + self.assertEqual(all, ['.jpg']) # And now for no hits all = self.db.guess_all_extensions('image/jpg', strict=True) - eq(all, []) + self.assertEqual(all, []) + # And now for type existing in both strict and non-strict mappings. + self.db.add_type('test-type', '.strict-ext') + self.db.add_type('test-type', '.non-strict-ext', strict=False) + all = self.db.guess_all_extensions('test-type', strict=False) + self.assertEqual(all, ['.strict-ext', '.non-strict-ext']) + all = self.db.guess_all_extensions('test-type') + self.assertEqual(all, ['.strict-ext']) + # Test that changing the result list does not affect the global state + all.append('.no-such-ext') + all = self.db.guess_all_extensions('test-type') + self.assertNotIn('.no-such-ext', all) def test_encoding(self): - getpreferredencoding = locale.getpreferredencoding - self.addCleanup(setattr, locale, 'getpreferredencoding', - getpreferredencoding) - locale.getpreferredencoding = lambda: 'ascii' - filename = support.findfile("mime.types") mimes = mimetypes.MimeTypes([filename]) exts = mimes.guess_all_extensions('application/vnd.geocube+xml', @@ -146,29 +213,110 @@ def test_init_reinitializes(self): # Poison should be gone. self.assertEqual(mimetypes.guess_extension('foo/bar'), None) + @unittest.skipIf(sys.platform.startswith("win"), "Non-Windows only") + def test_guess_known_extensions(self): + # Issue 37529 + # The test fails on Windows because Windows adds mime types from the Registry + # and that creates some duplicates. + from mimetypes import types_map + for v in types_map.values(): + self.assertIsNotNone(mimetypes.guess_extension(v)) + def test_preferred_extension(self): def check_extensions(): - self.assertEqual(mimetypes.guess_extension('application/octet-stream'), '.bin') - self.assertEqual(mimetypes.guess_extension('application/postscript'), '.ps') - self.assertEqual(mimetypes.guess_extension('application/vnd.apple.mpegurl'), '.m3u') - self.assertEqual(mimetypes.guess_extension('application/vnd.ms-excel'), '.xls') - self.assertEqual(mimetypes.guess_extension('application/vnd.ms-powerpoint'), '.ppt') - self.assertEqual(mimetypes.guess_extension('application/x-texinfo'), '.texi') - self.assertEqual(mimetypes.guess_extension('application/x-troff'), '.roff') - self.assertEqual(mimetypes.guess_extension('application/xml'), '.xsl') - self.assertEqual(mimetypes.guess_extension('audio/mpeg'), '.mp3') - self.assertEqual(mimetypes.guess_extension('image/jpeg'), '.jpg') - self.assertEqual(mimetypes.guess_extension('image/tiff'), '.tiff') - self.assertEqual(mimetypes.guess_extension('message/rfc822'), '.eml') - self.assertEqual(mimetypes.guess_extension('text/html'), '.html') - self.assertEqual(mimetypes.guess_extension('text/plain'), '.txt') - self.assertEqual(mimetypes.guess_extension('video/mpeg'), '.mpeg') - self.assertEqual(mimetypes.guess_extension('video/quicktime'), '.mov') + for mime_type, ext in ( + ("application/epub+zip", ".epub"), + ("application/octet-stream", ".bin"), + ("application/gzip", ".gz"), + ("application/ogg", ".ogx"), + ("application/postscript", ".ps"), + ("application/vnd.apple.mpegurl", ".m3u"), + ("application/vnd.ms-excel", ".xls"), + ("application/vnd.ms-fontobject", ".eot"), + ("application/vnd.ms-powerpoint", ".ppt"), + ("application/vnd.oasis.opendocument.graphics", ".odg"), + ("application/vnd.oasis.opendocument.presentation", ".odp"), + ("application/vnd.oasis.opendocument.spreadsheet", ".ods"), + ("application/vnd.oasis.opendocument.text", ".odt"), + ("application/vnd.openxmlformats-officedocument.presentationml.presentation", ".pptx"), + ("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", ".xlsx"), + ("application/vnd.openxmlformats-officedocument.wordprocessingml.document", ".docx"), + ("application/vnd.rar", ".rar"), + ("application/x-7z-compressed", ".7z"), + ("application/x-debian-package", ".deb"), + ("application/x-httpd-php", ".php"), + ("application/x-rpm", ".rpm"), + ("application/x-texinfo", ".texi"), + ("application/x-troff", ".roff"), + ("application/xml", ".xsl"), + ("application/yaml", ".yaml"), + ("audio/flac", ".flac"), + ("audio/matroska", ".mka"), + ("audio/mp4", ".m4a"), + ("audio/mpeg", ".mp3"), + ("audio/ogg", ".ogg"), + ("audio/vnd.wave", ".wav"), + ("audio/webm", ".weba"), + ("font/otf", ".otf"), + ("font/ttf", ".ttf"), + ("font/woff", ".woff"), + ("font/woff2", ".woff2"), + ("image/avif", ".avif"), + ("image/emf", ".emf"), + ("image/fits", ".fits"), + ("image/g3fax", ".g3"), + ("image/jp2", ".jp2"), + ("image/jpeg", ".jpg"), + ("image/jpm", ".jpm"), + ("image/t38", ".t38"), + ("image/tiff", ".tiff"), + ("image/tiff-fx", ".tfx"), + ("image/webp", ".webp"), + ("image/wmf", ".wmf"), + ("message/rfc822", ".eml"), + ("model/gltf+json", ".gltf"), + ("model/gltf-binary", ".glb"), + ("model/stl", ".stl"), + ("text/html", ".html"), + ("text/plain", ".txt"), + ("text/rtf", ".rtf"), + ("text/x-rst", ".rst"), + ("video/matroska", ".mkv"), + ("video/matroska-3d", ".mk3d"), + ("video/mpeg", ".mpeg"), + ("video/ogg", ".ogv"), + ("video/quicktime", ".mov"), + ("video/vnd.avi", ".avi"), + ("video/x-m4v", ".m4v"), + ("video/x-ms-wmv", ".wmv"), + ): + with self.subTest(mime_type=mime_type, ext=ext): + self.assertEqual(mimetypes.guess_extension(mime_type), ext) check_extensions() mimetypes.init() check_extensions() + def test_guess_file_type(self): + def check_file_type(): + for mime_type, ext in ( + ("application/yaml", ".yaml"), + ("application/yaml", ".yml"), + ("audio/mpeg", ".mp2"), + ("audio/mpeg", ".mp3"), + ("video/mpeg", ".m1v"), + ("video/mpeg", ".mpe"), + ("video/mpeg", ".mpeg"), + ("video/mpeg", ".mpg"), + ): + with self.subTest(mime_type=mime_type, ext=ext): + result, _ = mimetypes.guess_file_type(f"filename{ext}") + self.assertEqual(result, mime_type) + + check_file_type() + mimetypes.init() + check_file_type() + def test_init_stability(self): mimetypes.init() @@ -189,27 +337,59 @@ def test_init_stability(self): def test_path_like_ob(self): filename = "LICENSE.txt" - filepath = pathlib.Path(filename) - filepath_with_abs_dir = pathlib.Path('/dir/'+filename) - filepath_relative = pathlib.Path('../dir/'+filename) - path_dir = pathlib.Path('./') + filepath = os_helper.FakePath(filename) + filepath_with_abs_dir = os_helper.FakePath('/dir/'+filename) + filepath_relative = os_helper.FakePath('../dir/'+filename) + path_dir = os_helper.FakePath('./') - expected = self.db.guess_type(filename) + expected = self.db.guess_file_type(filename) + self.assertEqual(self.db.guess_file_type(filepath), expected) self.assertEqual(self.db.guess_type(filepath), expected) + self.assertEqual(self.db.guess_file_type( + filepath_with_abs_dir), expected) self.assertEqual(self.db.guess_type( filepath_with_abs_dir), expected) + self.assertEqual(self.db.guess_file_type(filepath_relative), expected) self.assertEqual(self.db.guess_type(filepath_relative), expected) + + self.assertEqual(self.db.guess_file_type(path_dir), (None, None)) self.assertEqual(self.db.guess_type(path_dir), (None, None)) + def test_bytes_path(self): + self.assertEqual(self.db.guess_file_type(b'foo.html'), + self.db.guess_file_type('foo.html')) + self.assertEqual(self.db.guess_file_type(b'foo.tar.gz'), + self.db.guess_file_type('foo.tar.gz')) + self.assertEqual(self.db.guess_file_type(b'foo.tgz'), + self.db.guess_file_type('foo.tgz')) + def test_keywords_args_api(self): + self.assertEqual(self.db.guess_file_type( + path="foo.html", strict=True), ("text/html", None)) self.assertEqual(self.db.guess_type( - url="foo.html", strict=True), ("text/html", None)) + url="scheme:foo.html", strict=True), ("text/html", None)) self.assertEqual(self.db.guess_all_extensions( type='image/jpg', strict=True), []) self.assertEqual(self.db.guess_extension( type='image/jpg', strict=False), '.jpg') + def test_added_types_are_used(self): + mimetypes.add_type('testing/default-type', '') + mime_type, _ = mimetypes.guess_type('') + self.assertEqual(mime_type, 'testing/default-type') + + mime_type, _ = mimetypes.guess_type('test.myext') + self.assertEqual(mime_type, None) + + mimetypes.add_type('testing/type', '.myext') + mime_type, _ = mimetypes.guess_type('test.myext') + self.assertEqual(mime_type, 'testing/type') + + def test_add_type_with_undotted_extension_deprecated(self): + with self.assertWarns(DeprecationWarning): + mimetypes.add_type("testing/type", "undotted") + @unittest.skipUnless(sys.platform.startswith("win"), "Windows only") class Win32MimeTypesTestCase(unittest.TestCase): @@ -236,58 +416,94 @@ def test_registry_parsing(self): eq(self.db.guess_type("image.jpg"), ("image/jpeg", None)) eq(self.db.guess_type("image.png"), ("image/png", None)) + @unittest.skipIf(not hasattr(_winapi, "_mimetypes_read_windows_registry"), + "read_windows_registry accelerator unavailable") + def test_registry_accelerator(self): + from_accel = {} + from_reg = {} + _winapi._mimetypes_read_windows_registry( + lambda v, k: from_accel.setdefault(k, set()).add(v) + ) + mimetypes.MimeTypes._read_windows_registry( + lambda v, k: from_reg.setdefault(k, set()).add(v) + ) + self.assertEqual(list(from_reg), list(from_accel)) + for k in from_reg: + self.assertEqual(from_reg[k], from_accel[k]) + class MiscTestCase(unittest.TestCase): def test__all__(self): support.check__all__(self, mimetypes) - -class MimetypesCliTestCase(unittest.TestCase): - - def mimetypes_cmd(self, *args, **kwargs): - support.patch(self, sys, "argv", [sys.executable, *args]) - with support.captured_stdout() as output: - mimetypes._main() - return output.getvalue().strip() - - def test_help_option(self): - support.patch(self, sys, "argv", [sys.executable, "-h"]) - with support.captured_stdout() as output: - with self.assertRaises(SystemExit) as cm: - mimetypes._main() - - self.assertIn("Usage: mimetypes.py", output.getvalue()) - self.assertEqual(cm.exception.code, 0) - - def test_invalid_option(self): - support.patch(self, sys, "argv", [sys.executable, "--invalid"]) - with support.captured_stdout() as output: - with self.assertRaises(SystemExit) as cm: - mimetypes._main() - - self.assertIn("Usage: mimetypes.py", output.getvalue()) - self.assertEqual(cm.exception.code, 1) - - def test_guess_extension(self): - eq = self.assertEqual - - extension = self.mimetypes_cmd("-l", "-e", "image/jpg") - eq(extension, ".jpg") - - extension = self.mimetypes_cmd("-e", "image/jpg") - eq(extension, "I don't know anything about type image/jpg") - - extension = self.mimetypes_cmd("-e", "image/jpeg") - eq(extension, ".jpg") - - def test_guess_type(self): - eq = self.assertEqual - - type_info = self.mimetypes_cmd("-l", "foo.pic") - eq(type_info, "type: image/pict encoding: None") - - type_info = self.mimetypes_cmd("foo.pic") - eq(type_info, "I don't know anything about type foo.pic") + @cpython_only + def test_lazy_import(self): + ensure_lazy_imports("mimetypes", {"os", "posixpath", "urllib.parse", "argparse"}) + + +class CommandLineTest(unittest.TestCase): + @force_not_colorized + def test_parse_args(self): + args, help_text = mimetypes._parse_args("-h") + self.assertTrue(help_text.startswith("usage: ")) + + args, help_text = mimetypes._parse_args("--invalid") + self.assertTrue(help_text.startswith("usage: ")) + + args, _ = mimetypes._parse_args(shlex.split("-l -e image/jpg")) + self.assertTrue(args.extension) + self.assertTrue(args.lenient) + self.assertEqual(args.type, ["image/jpg"]) + + args, _ = mimetypes._parse_args(shlex.split("-e image/jpg")) + self.assertTrue(args.extension) + self.assertFalse(args.lenient) + self.assertEqual(args.type, ["image/jpg"]) + + args, _ = mimetypes._parse_args(shlex.split("-l foo.webp")) + self.assertFalse(args.extension) + self.assertTrue(args.lenient) + self.assertEqual(args.type, ["foo.webp"]) + + args, _ = mimetypes._parse_args(shlex.split("foo.pic")) + self.assertFalse(args.extension) + self.assertFalse(args.lenient) + self.assertEqual(args.type, ["foo.pic"]) + + def test_multiple_inputs(self): + result = "\n".join(mimetypes._main(shlex.split("foo.pdf foo.png"))) + self.assertEqual( + result, + "type: application/pdf encoding: None\n" + "type: image/png encoding: None" + ) + + def test_multiple_inputs_error(self): + result = "\n".join(mimetypes._main(shlex.split("foo.pdf foo.bar_ext"))) + self.assertEqual( + result, + "type: application/pdf encoding: None\n" + "error: media type unknown for foo.bar_ext" + ) + + + def test_invocation(self): + for command, expected in [ + ("-l -e image/jpg", ".jpg"), + ("-e image/jpeg", ".jpg"), + ("-l foo.webp", "type: image/webp encoding: None"), + ]: + result = "\n".join(mimetypes._main(shlex.split(command))) + self.assertEqual(result, expected) + + def test_invocation_error(self): + for command, expected in [ + ("-e image/jpg", "error: unknown type image/jpg"), + ("foo.bar_ext", "error: media type unknown for foo.bar_ext"), + ]: + with self.subTest(command=command): + result = "\n".join(mimetypes._main(shlex.split(command))) + self.assertEqual(result, expected) if __name__ == "__main__": diff --git a/Lib/test/test_module/__init__.py b/Lib/test/test_module/__init__.py index b599c6d8c8d..59c74fd0d41 100644 --- a/Lib/test/test_module/__init__.py +++ b/Lib/test/test_module/__init__.py @@ -293,8 +293,6 @@ class M(ModuleType): melon = Descr() self.assertRaises(RuntimeError, getattr, M("mymod"), "melon") - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_lazy_create_annotations(self): # module objects lazy create their __annotations__ dict on demand. # the annotations dict is stored in module.__dict__. @@ -334,7 +332,11 @@ def test_annotations_getset_raises(self): del foo.__annotations__ def test_annotations_are_created_correctly(self): - ann_module4 = import_helper.import_fresh_module('test.typinganndata.ann_module4') + ann_module4 = import_helper.import_fresh_module( + 'test.typinganndata.ann_module4', + ) + self.assertFalse("__annotations__" in ann_module4.__dict__) + self.assertEqual(ann_module4.__annotations__, {"a": int, "b": str}) self.assertTrue("__annotations__" in ann_module4.__dict__) del ann_module4.__annotations__ self.assertFalse("__annotations__" in ann_module4.__dict__) diff --git a/Lib/test/test_msvcrt.py b/Lib/test/test_msvcrt.py new file mode 100644 index 00000000000..1c6905bd1ee --- /dev/null +++ b/Lib/test/test_msvcrt.py @@ -0,0 +1,120 @@ +import os +import subprocess +import sys +import unittest +from textwrap import dedent + +from test.support import os_helper, requires_resource +from test.support.os_helper import TESTFN, TESTFN_ASCII + +if sys.platform != "win32": + raise unittest.SkipTest("windows related tests") + +import _winapi +import msvcrt + + +class TestFileOperations(unittest.TestCase): + def test_locking(self): + with open(TESTFN, "w") as f: + self.addCleanup(os_helper.unlink, TESTFN) + + msvcrt.locking(f.fileno(), msvcrt.LK_LOCK, 1) + self.assertRaises(OSError, msvcrt.locking, f.fileno(), msvcrt.LK_NBLCK, 1) + + def test_unlockfile(self): + with open(TESTFN, "w") as f: + self.addCleanup(os_helper.unlink, TESTFN) + + msvcrt.locking(f.fileno(), msvcrt.LK_LOCK, 1) + msvcrt.locking(f.fileno(), msvcrt.LK_UNLCK, 1) + msvcrt.locking(f.fileno(), msvcrt.LK_LOCK, 1) + + def test_setmode(self): + with open(TESTFN, "w") as f: + self.addCleanup(os_helper.unlink, TESTFN) + + msvcrt.setmode(f.fileno(), os.O_BINARY) + msvcrt.setmode(f.fileno(), os.O_TEXT) + + def test_open_osfhandle(self): + h = _winapi.CreateFile(TESTFN_ASCII, _winapi.GENERIC_WRITE, 0, 0, 1, 128, 0) + self.addCleanup(os_helper.unlink, TESTFN_ASCII) + + try: + fd = msvcrt.open_osfhandle(h, os.O_RDONLY) + h = None + os.close(fd) + finally: + if h: + _winapi.CloseHandle(h) + + def test_get_osfhandle(self): + with open(TESTFN, "w") as f: + self.addCleanup(os_helper.unlink, TESTFN) + + msvcrt.get_osfhandle(f.fileno()) + + +c = '\u5b57' # unicode CJK char (meaning 'character') for 'wide-char' tests +c_encoded = b'\x57\x5b' # utf-16-le (which windows internally used) encoded char for this CJK char + + +class TestConsoleIO(unittest.TestCase): + # CREATE_NEW_CONSOLE creates a "popup" window. + @requires_resource('gui') + def run_in_separated_process(self, code): + # Run test in a separated process to avoid stdin conflicts. + # See: gh-110147 + cmd = [sys.executable, '-c', code] + subprocess.run(cmd, check=True, capture_output=True, + creationflags=subprocess.CREATE_NEW_CONSOLE) + + def test_kbhit(self): + code = dedent(''' + import msvcrt + assert msvcrt.kbhit() == 0 + ''') + self.run_in_separated_process(code) + + def test_getch(self): + msvcrt.ungetch(b'c') + self.assertEqual(msvcrt.getch(), b'c') + + def check_getwch(self, funcname): + code = dedent(f''' + import msvcrt + from _testconsole import write_input + with open("CONIN$", "rb", buffering=0) as stdin: + write_input(stdin, {ascii(c_encoded)}) + assert msvcrt.{funcname}() == "{c}" + ''') + self.run_in_separated_process(code) + + def test_getwch(self): + self.check_getwch('getwch') + + def test_getche(self): + msvcrt.ungetch(b'c') + self.assertEqual(msvcrt.getche(), b'c') + + def test_getwche(self): + self.check_getwch('getwche') + + def test_putch(self): + msvcrt.putch(b'c') + + def test_putwch(self): + msvcrt.putwch(c) + + +class TestOther(unittest.TestCase): + def test_heap_min(self): + try: + msvcrt.heapmin() + except OSError: + pass + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_multiprocessing_fork/test_manager.py b/Lib/test/test_multiprocessing_fork/test_manager.py index 9efbb83bbb7..f8d7eddd652 100644 --- a/Lib/test/test_multiprocessing_fork/test_manager.py +++ b/Lib/test/test_multiprocessing_fork/test_manager.py @@ -3,5 +3,22 @@ install_tests_in_module_dict(globals(), 'fork', only_type="manager") +import sys # TODO: RUSTPYTHON +class WithManagerTestCondition(WithManagerTestCondition): # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', 'TODO: RUSTPYTHON, times out') + def test_notify_all(self): super().test_notify_all() # TODO: RUSTPYTHON + +class WithManagerTestQueue(WithManagerTestQueue): # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', 'TODO: RUSTPYTHON, times out') + def test_fork(self): super().test_fork() # TODO: RUSTPYTHON + +local_globs = globals().copy() # TODO: RUSTPYTHON +for name, base in local_globs.items(): # TODO: RUSTPYTHON + if name.startswith('WithManagerTest') and issubclass(base, unittest.TestCase): # TODO: RUSTPYTHON + base = unittest.skipIf( # TODO: RUSTPYTHON + sys.platform == 'linux', # TODO: RUSTPYTHON + 'TODO: RUSTPYTHON flaky BrokenPipeError, flaky ConnectionRefusedError, flaky ConnectionResetError, flaky EOFError' + )(base) # TODO: RUSTPYTHON + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_multiprocessing_fork/test_misc.py b/Lib/test/test_multiprocessing_fork/test_misc.py index 891a494020c..bcf0858258e 100644 --- a/Lib/test/test_multiprocessing_fork/test_misc.py +++ b/Lib/test/test_multiprocessing_fork/test_misc.py @@ -3,5 +3,24 @@ install_tests_in_module_dict(globals(), 'fork', exclude_types=True) +import sys # TODO: RUSTPYTHON +class TestManagerExceptions(TestManagerExceptions): # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', "TODO: RUSTPYTHON flaky") + def test_queue_get(self): super().test_queue_get() # TODO: RUSTPYTHON + +@unittest.skipIf(sys.platform == 'linux', "TODO: RUSTPYTHON flaky") +class TestInitializers(TestInitializers): pass # TODO: RUSTPYTHON + +class TestStartMethod(TestStartMethod): # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', "TODO: RUSTPYTHON flaky") + def test_nested_startmethod(self): super().test_nested_startmethod() # TODO: RUSTPYTHON + +@unittest.skipIf(sys.platform == 'linux', "TODO: RUSTPYTHON flaky") +class TestSyncManagerTypes(TestSyncManagerTypes): pass # TODO: RUSTPYTHON + +class MiscTestCase(MiscTestCase): # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', "TODO: RUSTPYTHON flaky") + def test_forked_thread_not_started(self): super().test_forked_thread_not_started() # TODO: RUSTPYTHON + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_multiprocessing_fork/test_processes.py b/Lib/test/test_multiprocessing_fork/test_processes.py index e64e9afc010..02b7256e41b 100644 --- a/Lib/test/test_multiprocessing_fork/test_processes.py +++ b/Lib/test/test_multiprocessing_fork/test_processes.py @@ -3,5 +3,40 @@ install_tests_in_module_dict(globals(), 'fork', only_type="processes") +import os, sys # TODO: RUSTPYTHON +class WithProcessesTestCondition(WithProcessesTestCondition): # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', 'TODO: RUSTPYTHON flaky timeout') + def test_notify_all(self): super().test_notify_all() # TODO: RUSTPYTHON + +class WithProcessesTestLock(WithProcessesTestLock): # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', 'TODO: RUSTPYTHON flaky BrokenPipeError, flaky ConnectionRefusedError, flaky ConnectionResetError, flaky EOFError') + def test_repr_lock(self): super().test_repr_lock() # TODO: RUSTPYTHON + +class WithProcessesTestManagerRestart(WithProcessesTestManagerRestart): # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', 'TODO: RUSTPYTHON flaky BrokenPipeError, flaky ConnectionRefusedError, flaky ConnectionResetError, flaky EOFError') + def test_rapid_restart(self): super().test_rapid_restart() # TODO: RUSTPYTHON + +class WithProcessesTestProcess(WithProcessesTestProcess): # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', 'TODO: RUSTPYTHON flaky timeout') + def test_args_argument(self): super().test_args_argument() # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', 'TODO: RUSTPYTHON flaky timeout') + def test_process(self): super().test_process() # TODO: RUSTPYTHON + +class WithProcessesTestPoolWorkerLifetime(WithProcessesTestPoolWorkerLifetime): # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', 'TODO: RUSTPYTHON flaky timeout') + def test_pool_worker_lifetime(self): super().test_pool_worker_lifetime() # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', 'TODO: RUSTPYTHON flaky timeout') + def test_pool_worker_lifetime_early_close(self): super().test_pool_worker_lifetime_early_close() # TODO: RUSTPYTHON + +class WithProcessesTestQueue(WithProcessesTestQueue): # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', 'TODO: RUSTPYTHON flaky timeout') + def test_fork(self): super().test_fork() # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', 'TODO: RUSTPYTHON flaky timeout') + def test_get(self): super().test_get() # TODO: RUSTPYTHON + +class WithProcessesTestSharedMemory(WithProcessesTestSharedMemory): # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', 'TODO: RUSTPYTHON flaky BrokenPipeError, flaky ConnectionRefusedError, flaky ConnectionResetError, flaky EOFError') + def test_shared_memory_SharedMemoryManager_basics(self): super().test_shared_memory_SharedMemoryManager_basics() # TODO: RUSTPYTHON + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_multiprocessing_fork/test_threads.py b/Lib/test/test_multiprocessing_fork/test_threads.py index 1670e34cb17..1065ebf7fe4 100644 --- a/Lib/test/test_multiprocessing_fork/test_threads.py +++ b/Lib/test/test_multiprocessing_fork/test_threads.py @@ -3,5 +3,14 @@ install_tests_in_module_dict(globals(), 'fork', only_type="threads") +import os, sys # TODO: RUSTPYTHON +class WithThreadsTestPool(WithThreadsTestPool): # TODO: RUSTPYTHON + @unittest.skip("TODO: RUSTPYTHON; flaky environment pollution when running rustpython -m test --fail-env-changed due to unknown reason") + def test_terminate(self): super().test_terminate() # TODO: RUSTPYTHON + +class WithThreadsTestManagerRestart(WithThreadsTestManagerRestart): # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'linux', 'TODO: RUSTPYTHON flaky flaky BrokenPipeError, flaky ConnectionRefusedError, flaky ConnectionResetError, flaky EOFError') + def test_rapid_restart(self): super().test_rapid_restart() # TODO: RUSTPYTHON + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_multiprocessing_forkserver/test_processes.py b/Lib/test/test_multiprocessing_forkserver/test_processes.py index 360967cf1ae..6f6b8f56837 100644 --- a/Lib/test/test_multiprocessing_forkserver/test_processes.py +++ b/Lib/test/test_multiprocessing_forkserver/test_processes.py @@ -3,5 +3,19 @@ install_tests_in_module_dict(globals(), 'forkserver', only_type="processes") +import os, sys # TODO: RUSTPYTHON +class WithProcessesTestCondition(WithProcessesTestCondition): # TODO: RUSTPYTHON + @unittest.skip('TODO: RUSTPYTHON flaky timeout') + def test_notify(self): super().test_notify() + @unittest.skip('TODO: RUSTPYTHON flaky timeout') + def test_notify_n(self): super().test_notify_n() + +class WithProcessesTestLock(WithProcessesTestLock): # TODO: RUSTPYTHON + @unittest.skipIf( # TODO: RUSTPYTHON + sys.platform == 'linux', # TODO: RUSTPYTHON + 'TODO: RUSTPYTHON flaky BrokenPipeError, flaky ConnectionRefusedError, flaky ConnectionResetError, flaky EOFError' + ) # TODO: RUSTPYTHON + def test_repr_rlock(self): super().test_repr_rlock() # TODO: RUSTPYTHON + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_multiprocessing_spawn/test_processes.py b/Lib/test/test_multiprocessing_spawn/test_processes.py index af764b0d848..21fd6abd655 100644 --- a/Lib/test/test_multiprocessing_spawn/test_processes.py +++ b/Lib/test/test_multiprocessing_spawn/test_processes.py @@ -3,5 +3,17 @@ install_tests_in_module_dict(globals(), 'spawn', only_type="processes") +import os, sys # TODO: RUSTPYTHON +class WithProcessesTestCondition(WithProcessesTestCondition): # TODO: RUSTPYTHON + @unittest.skipIf(sys.platform == 'darwin', 'TODO: RUSTPYTHON flaky timeout') + def test_notify(self): super().test_notify() + +class WithProcessesTestLock(WithProcessesTestLock): # TODO: RUSTPYTHON + @unittest.skipIf( # TODO: RUSTPYTHON + sys.platform == 'linux', # TODO: RUSTPYTHON + 'TODO: RUSTPYTHON flaky BrokenPipeError, flaky ConnectionRefusedError, flaky ConnectionResetError, flaky EOFError' + ) # TODO: RUSTPYTHON + def test_repr_rlock(self): super().test_repr_rlock() # TODO: RUSTPYTHON + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_multiprocessing_spawn/test_threads.py b/Lib/test/test_multiprocessing_spawn/test_threads.py index c1257749b9c..54c52c4188b 100644 --- a/Lib/test/test_multiprocessing_spawn/test_threads.py +++ b/Lib/test/test_multiprocessing_spawn/test_threads.py @@ -3,5 +3,10 @@ install_tests_in_module_dict(globals(), 'spawn', only_type="threads") +import os, sys # TODO: RUSTPYTHON +class WithThreadsTestPool(WithThreadsTestPool): # TODO: RUSTPYTHON + @unittest.skip("TODO: RUSTPYTHON; flaky environment pollution when running rustpython -m test --fail-env-changed due to unknown reason") + def test_terminate(self): super().test_terminate() # TODO: RUSTPYTHON + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py index e1982dfd0bd..9270f325706 100644 --- a/Lib/test/test_ntpath.py +++ b/Lib/test/test_ntpath.py @@ -6,10 +6,9 @@ import sys import unittest import warnings -from ntpath import ALLOW_MISSING from test import support -from test.support import cpython_only, os_helper -from test.support import TestFailed, is_emscripten +from test.support import os_helper +from ntpath import ALLOW_MISSING from test.support.os_helper import FakePath from test import test_genericpath from tempfile import TemporaryFile @@ -59,7 +58,7 @@ def tester(fn, wantResult): fn = fn.replace("\\", "\\\\") gotResult = eval(fn) if wantResult != gotResult and _norm(wantResult) != _norm(gotResult): - raise TestFailed("%s should return: %s but returned: %s" \ + raise support.TestFailed("%s should return: %s but returned: %s" \ %(str(fn), str(wantResult), str(gotResult))) # then with bytes @@ -75,7 +74,7 @@ def tester(fn, wantResult): warnings.simplefilter("ignore", DeprecationWarning) gotResult = eval(fn) if _norm(wantResult) != _norm(gotResult): - raise TestFailed("%s should return: %s but returned: %s" \ + raise support.TestFailed("%s should return: %s but returned: %s" \ %(str(fn), str(wantResult), repr(gotResult))) @@ -1022,6 +1021,19 @@ def check(value, expected): check('%spam%bar', '%sbar' % nonascii) check('%{}%bar'.format(nonascii), 'ham%sbar' % nonascii) + @support.requires_resource('cpu') + def test_expandvars_large(self): + expandvars = ntpath.expandvars + with os_helper.EnvironmentVarGuard() as env: + env.clear() + env["A"] = "B" + n = 100_000 + self.assertEqual(expandvars('%A%'*n), 'B'*n) + self.assertEqual(expandvars('%A%A'*n), 'BA'*n) + self.assertEqual(expandvars("''"*n + '%%'), "''"*n + '%') + self.assertEqual(expandvars("%%"*n), "%"*n) + self.assertEqual(expandvars("$$"*n), "$"*n) + def test_expanduser(self): tester('ntpath.expanduser("test")', 'test') @@ -1229,7 +1241,6 @@ def check_error(paths, expected): self.assertRaises(TypeError, ntpath.commonpath, ['C:\\Foo', b'Foo\\Baz']) self.assertRaises(TypeError, ntpath.commonpath, ['Foo', b'C:\\Foo\\Baz']) - @unittest.skipIf(is_emscripten, "Emscripten cannot fstat unnamed files.") def test_sameopenfile(self): with TemporaryFile() as tf1, TemporaryFile() as tf2: # Make sure the same file is really the same @@ -1440,7 +1451,7 @@ def test_con_device(self): self.assertTrue(os.path.exists(r"\\.\CON")) @unittest.skipIf(sys.platform != 'win32', "Fast paths are only for win32") - @cpython_only + @support.cpython_only def test_fast_paths_in_use(self): # There are fast paths of these functions implemented in posixmodule.c. # Confirm that they are being used, and not the Python fallbacks in diff --git a/Lib/test/test_nturl2path.py b/Lib/test/test_nturl2path.py new file mode 100644 index 00000000000..a6a3422a0f7 --- /dev/null +++ b/Lib/test/test_nturl2path.py @@ -0,0 +1,107 @@ +import unittest + +from test.support import warnings_helper + + +nturl2path = warnings_helper.import_deprecated("nturl2path") + + +class NTURL2PathTest(unittest.TestCase): + """Test pathname2url() and url2pathname()""" + + def test_basic(self): + # Make sure simple tests pass + expected_path = r"parts\of\a\path" + expected_url = "parts/of/a/path" + result = nturl2path.pathname2url(expected_path) + self.assertEqual(expected_url, result, + "pathname2url() failed; %s != %s" % + (result, expected_url)) + result = nturl2path.url2pathname(expected_url) + self.assertEqual(expected_path, result, + "url2pathame() failed; %s != %s" % + (result, expected_path)) + + def test_pathname2url(self): + # Test special prefixes are correctly handled in pathname2url() + fn = nturl2path.pathname2url + self.assertEqual(fn('\\\\?\\C:\\dir'), '///C:/dir') + self.assertEqual(fn('\\\\?\\unc\\server\\share\\dir'), '//server/share/dir') + self.assertEqual(fn("C:"), '///C:') + self.assertEqual(fn("C:\\"), '///C:/') + self.assertEqual(fn('c:\\a\\b.c'), '///c:/a/b.c') + self.assertEqual(fn('C:\\a\\b.c'), '///C:/a/b.c') + self.assertEqual(fn('C:\\a\\b.c\\'), '///C:/a/b.c/') + self.assertEqual(fn('C:\\a\\\\b.c'), '///C:/a//b.c') + self.assertEqual(fn('C:\\a\\b%#c'), '///C:/a/b%25%23c') + self.assertEqual(fn('C:\\a\\b\xe9'), '///C:/a/b%C3%A9') + self.assertEqual(fn('C:\\foo\\bar\\spam.foo'), "///C:/foo/bar/spam.foo") + # NTFS alternate data streams + self.assertEqual(fn('C:\\foo:bar'), '///C:/foo%3Abar') + self.assertEqual(fn('foo:bar'), 'foo%3Abar') + # No drive letter + self.assertEqual(fn("\\folder\\test\\"), '///folder/test/') + self.assertEqual(fn("\\\\folder\\test\\"), '//folder/test/') + self.assertEqual(fn("\\\\\\folder\\test\\"), '///folder/test/') + self.assertEqual(fn('\\\\some\\share\\'), '//some/share/') + self.assertEqual(fn('\\\\some\\share\\a\\b.c'), '//some/share/a/b.c') + self.assertEqual(fn('\\\\some\\share\\a\\b%#c\xe9'), '//some/share/a/b%25%23c%C3%A9') + # Alternate path separator + self.assertEqual(fn('C:/a/b.c'), '///C:/a/b.c') + self.assertEqual(fn('//some/share/a/b.c'), '//some/share/a/b.c') + self.assertEqual(fn('//?/C:/dir'), '///C:/dir') + self.assertEqual(fn('//?/unc/server/share/dir'), '//server/share/dir') + # Round-tripping + urls = ['///C:', + '///folder/test/', + '///C:/foo/bar/spam.foo'] + for url in urls: + self.assertEqual(fn(nturl2path.url2pathname(url)), url) + + def test_url2pathname(self): + fn = nturl2path.url2pathname + self.assertEqual(fn('/'), '\\') + self.assertEqual(fn('/C:/'), 'C:\\') + self.assertEqual(fn("///C|"), 'C:') + self.assertEqual(fn("///C:"), 'C:') + self.assertEqual(fn('///C:/'), 'C:\\') + self.assertEqual(fn('/C|//'), 'C:\\\\') + self.assertEqual(fn('///C|/path'), 'C:\\path') + # No DOS drive + self.assertEqual(fn("///C/test/"), '\\C\\test\\') + self.assertEqual(fn("////C/test/"), '\\\\C\\test\\') + # DOS drive paths + self.assertEqual(fn('c:/path/to/file'), 'c:\\path\\to\\file') + self.assertEqual(fn('C:/path/to/file'), 'C:\\path\\to\\file') + self.assertEqual(fn('C:/path/to/file/'), 'C:\\path\\to\\file\\') + self.assertEqual(fn('C:/path/to//file'), 'C:\\path\\to\\\\file') + self.assertEqual(fn('C|/path/to/file'), 'C:\\path\\to\\file') + self.assertEqual(fn('/C|/path/to/file'), 'C:\\path\\to\\file') + self.assertEqual(fn('///C|/path/to/file'), 'C:\\path\\to\\file') + self.assertEqual(fn("///C|/foo/bar/spam.foo"), 'C:\\foo\\bar\\spam.foo') + # Colons in URI + self.assertEqual(fn('///\u00e8|/'), '\u00e8:\\') + self.assertEqual(fn('//host/share/spam.txt:eggs'), '\\\\host\\share\\spam.txt:eggs') + self.assertEqual(fn('///c:/spam.txt:eggs'), 'c:\\spam.txt:eggs') + # UNC paths + self.assertEqual(fn('//server/path/to/file'), '\\\\server\\path\\to\\file') + self.assertEqual(fn('////server/path/to/file'), '\\\\server\\path\\to\\file') + self.assertEqual(fn('/////server/path/to/file'), '\\\\server\\path\\to\\file') + # Localhost paths + self.assertEqual(fn('//localhost/C:/path/to/file'), 'C:\\path\\to\\file') + self.assertEqual(fn('//localhost/C|/path/to/file'), 'C:\\path\\to\\file') + self.assertEqual(fn('//localhost/path/to/file'), '\\path\\to\\file') + self.assertEqual(fn('//localhost//server/path/to/file'), '\\\\server\\path\\to\\file') + # Percent-encoded forward slashes are preserved for backwards compatibility + self.assertEqual(fn('C:/foo%2fbar'), 'C:\\foo/bar') + self.assertEqual(fn('//server/share/foo%2fbar'), '\\\\server\\share\\foo/bar') + # Round-tripping + paths = ['C:', + r'\C\test\\', + r'C:\foo\bar\spam.foo'] + for path in paths: + self.assertEqual(fn(nturl2path.pathname2url(path)), path) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_opcodes.py b/Lib/test/test_opcodes.py index 72488b2bb6b..f7cc8331b8d 100644 --- a/Lib/test/test_opcodes.py +++ b/Lib/test/test_opcodes.py @@ -39,16 +39,19 @@ class C: pass def test_use_existing_annotations(self): ns = {'__annotations__': {1: 2}} exec('x: int', ns) - self.assertEqual(ns['__annotations__'], {'x': int, 1: 2}) + self.assertEqual(ns['__annotations__'], {1: 2}) def test_do_not_recreate_annotations(self): # Don't rely on the existence of the '__annotations__' global. with support.swap_item(globals(), '__annotations__', {}): - del globals()['__annotations__'] + globals().pop('__annotations__', None) class C: - del __annotations__ - with self.assertRaises(NameError): - x: int + try: + del __annotations__ + except NameError: + pass + x: int + self.assertEqual(C.__annotations__, {"x": int}) def test_raise_class_exceptions(self): diff --git a/Lib/test/test_operator.py b/Lib/test/test_operator.py index 05b7a7462db..f8b36a3c2f1 100644 --- a/Lib/test/test_operator.py +++ b/Lib/test/test_operator.py @@ -347,6 +347,26 @@ def test_is_not(self): self.assertFalse(operator.is_not(a, b)) self.assertTrue(operator.is_not(a,c)) + def test_is_none(self): + operator = self.module + a = 'xyzpdq' + b = '' + c = None + self.assertRaises(TypeError, operator.is_none) + self.assertFalse(operator.is_none(a)) + self.assertFalse(operator.is_none(b)) + self.assertTrue(operator.is_none(c)) + + def test_is_not_none(self): + operator = self.module + a = 'xyzpdq' + b = '' + c = None + self.assertRaises(TypeError, operator.is_not_none) + self.assertTrue(operator.is_not_none(a)) + self.assertTrue(operator.is_not_none(b)) + self.assertFalse(operator.is_not_none(c)) + def test_attrgetter(self): operator = self.module class A: @@ -462,6 +482,8 @@ def bar(self, f=42): return f def baz(*args, **kwds): return kwds['name'], kwds['self'] + def return_arguments(self, *args, **kwds): + return args, kwds a = A() f = operator.methodcaller('foo') self.assertRaises(IndexError, f, a) @@ -478,6 +500,17 @@ def baz(*args, **kwds): f = operator.methodcaller('baz', name='spam', self='eggs') self.assertEqual(f(a), ('spam', 'eggs')) + many_positional_arguments = tuple(range(10)) + many_kw_arguments = dict(zip('abcdefghij', range(10))) + f = operator.methodcaller('return_arguments', *many_positional_arguments) + self.assertEqual(f(a), (many_positional_arguments, {})) + + f = operator.methodcaller('return_arguments', **many_kw_arguments) + self.assertEqual(f(a), ((), many_kw_arguments)) + + f = operator.methodcaller('return_arguments', *many_positional_arguments, **many_kw_arguments) + self.assertEqual(f(a), (many_positional_arguments, many_kw_arguments)) + def test_inplace(self): operator = self.module class C(object): @@ -635,22 +668,20 @@ class PyOperatorTestCase(OperatorTestCase, unittest.TestCase): class COperatorTestCase(OperatorTestCase, unittest.TestCase): module = c_operator - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_attrgetter_signature(self): - super().test_attrgetter_signature() + return super().test_attrgetter_signature() - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_itemgetter_signature(self): - super().test_itemgetter_signature() + return super().test_itemgetter_signature() - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_methodcaller_signature(self): - super().test_methodcaller_signature() + return super().test_methodcaller_signature() +@support.thread_unsafe("swaps global operator module") class OperatorPickleTestCase: def copy(self, obj, proto): with support.swap_item(sys.modules, 'operator', self.module): diff --git a/Lib/test/test_optparse.py b/Lib/test/test_optparse.py index c68214961e3..e476e472780 100644 --- a/Lib/test/test_optparse.py +++ b/Lib/test/test_optparse.py @@ -14,8 +14,9 @@ from io import StringIO from test import support -from test.support import os_helper +from test.support import cpython_only, os_helper from test.support.i18n_helper import TestTranslationsBase, update_translation_snapshots +from test.support.import_helper import ensure_lazy_imports import optparse from optparse import make_option, Option, \ @@ -1655,6 +1656,10 @@ def test__all__(self): not_exported = {'check_builtin', 'AmbiguousOptionError', 'NO_DEFAULT'} support.check__all__(self, optparse, not_exported=not_exported) + @cpython_only + def test_lazy_import(self): + ensure_lazy_imports("optparse", {"textwrap"}) + class TestTranslations(TestTranslationsBase): def test_translations(self): diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py index 4755aef080e..653a05dd011 100644 --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -2350,7 +2350,6 @@ def check_bool(self, f, *args, **kwargs): with self.assertRaises(RuntimeWarning): f(fd, *args, **kwargs) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_fdopen(self): self.check(os.fdopen, encoding="utf-8") self.check_bool(os.fdopen, encoding="utf-8") @@ -2773,7 +2772,6 @@ def test_CTRL_C_EVENT(self): self._kill_with_event(signal.CTRL_C_EVENT, "CTRL_C_EVENT") - @unittest.expectedFailure # TODO: RUSTPYTHON @support.requires_subprocess() def test_CTRL_BREAK_EVENT(self): self._kill_with_event(signal.CTRL_BREAK_EVENT, "CTRL_BREAK_EVENT") diff --git a/Lib/test/test_osx_env.py b/Lib/test/test_osx_env.py new file mode 100644 index 00000000000..80198edcb80 --- /dev/null +++ b/Lib/test/test_osx_env.py @@ -0,0 +1,34 @@ +""" +Test suite for OS X interpreter environment variables. +""" + +from test.support.os_helper import EnvironmentVarGuard +import subprocess +import sys +import sysconfig +import unittest + +@unittest.skipUnless(sys.platform == 'darwin' and + sysconfig.get_config_var('WITH_NEXT_FRAMEWORK'), + 'unnecessary on this platform') +class OSXEnvironmentVariableTestCase(unittest.TestCase): + def _check_sys(self, ev, cond, sv, val = sys.executable + 'dummy'): + with EnvironmentVarGuard() as evg: + subpc = [str(sys.executable), '-c', + 'import sys; sys.exit(2 if "%s" %s %s else 3)' % (val, cond, sv)] + # ensure environment variable does not exist + evg.unset(ev) + # test that test on sys.xxx normally fails + rc = subprocess.call(subpc) + self.assertEqual(rc, 3, "expected %s not %s %s" % (ev, cond, sv)) + # set environ variable + evg.set(ev, val) + # test that sys.xxx has been influenced by the environ value + rc = subprocess.call(subpc) + self.assertEqual(rc, 2, "expected %s %s %s" % (ev, cond, sv)) + + def test_pythonexecutable_sets_sys_executable(self): + self._check_sys('PYTHONEXECUTABLE', '==', 'sys.executable') + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_peepholer.py b/Lib/test/test_peepholer.py new file mode 100644 index 00000000000..c5c10cb5318 --- /dev/null +++ b/Lib/test/test_peepholer.py @@ -0,0 +1,2761 @@ +import dis +import gc +from itertools import combinations, product +import opcode +import sys +import textwrap +import unittest +try: + import _testinternalcapi +except ImportError: + _testinternalcapi = None + +from test import support +from test.support.bytecode_helper import ( + BytecodeTestCase, CfgOptimizationTestCase, CompilationStepTestCase) + + +def compile_pattern_with_fast_locals(pattern): + source = textwrap.dedent( + f""" + def f(x): + match x: + case {pattern}: + pass + """ + ) + namespace = {} + exec(source, namespace) + return namespace["f"].__code__ + + +def count_instr_recursively(f, opname): + count = 0 + for instr in dis.get_instructions(f): + if instr.opname == opname: + count += 1 + if hasattr(f, '__code__'): + f = f.__code__ + for c in f.co_consts: + if hasattr(c, 'co_code'): + count += count_instr_recursively(c, opname) + return count + + +def get_binop_argval(arg): + for i, nb_op in enumerate(opcode._nb_ops): + if arg == nb_op[0]: + return i + assert False, f"{arg} is not a valid BINARY_OP argument." + + +class TestTranforms(BytecodeTestCase): + + def check_jump_targets(self, code): + instructions = list(dis.get_instructions(code)) + targets = {instr.offset: instr for instr in instructions} + for instr in instructions: + if 'JUMP_' not in instr.opname: + continue + tgt = targets[instr.argval] + # jump to unconditional jump + if tgt.opname in ('JUMP_BACKWARD', 'JUMP_FORWARD'): + self.fail(f'{instr.opname} at {instr.offset} ' + f'jumps to {tgt.opname} at {tgt.offset}') + # unconditional jump to RETURN_VALUE + if (instr.opname in ('JUMP_BACKWARD', 'JUMP_FORWARD') and + tgt.opname == 'RETURN_VALUE'): + self.fail(f'{instr.opname} at {instr.offset} ' + f'jumps to {tgt.opname} at {tgt.offset}') + + def check_lnotab(self, code): + "Check that the lnotab byte offsets are sensible." + code = dis._get_code_object(code) + lnotab = list(dis.findlinestarts(code)) + # Don't bother checking if the line info is sensible, because + # most of the line info we can get at comes from lnotab. + min_bytecode = min(t[0] for t in lnotab) + max_bytecode = max(t[0] for t in lnotab) + self.assertGreaterEqual(min_bytecode, 0) + self.assertLess(max_bytecode, len(code.co_code)) + # This could conceivably test more (and probably should, as there + # aren't very many tests of lnotab), if peepholer wasn't scheduled + # to be replaced anyway. + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_unot(self): + # UNARY_NOT POP_JUMP_IF_FALSE --> POP_JUMP_IF_TRUE' + def unot(x): + if not x == 2: + del x + self.assertNotInBytecode(unot, 'UNARY_NOT') + self.assertNotInBytecode(unot, 'POP_JUMP_IF_FALSE') + self.assertInBytecode(unot, 'POP_JUMP_IF_TRUE') + self.check_lnotab(unot) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_elim_inversion_of_is_or_in(self): + for line, cmp_op, invert in ( + ('not a is b', 'IS_OP', 1,), + ('not a is not b', 'IS_OP', 0,), + ('not a in b', 'CONTAINS_OP', 1,), + ('not a not in b', 'CONTAINS_OP', 0,), + ): + with self.subTest(line=line): + code = compile(line, '', 'single') + self.assertInBytecode(code, cmp_op, invert) + self.check_lnotab(code) + + def test_global_as_constant(self): + # LOAD_GLOBAL None/True/False --> LOAD_CONST None/True/False + def f(): + x = None + x = None + return x + def g(): + x = True + return x + def h(): + x = False + return x + + for func, elem in ((f, None), (g, True), (h, False)): + with self.subTest(func=func): + self.assertNotInBytecode(func, 'LOAD_GLOBAL') + self.assertInBytecode(func, 'LOAD_CONST', elem) + self.check_lnotab(func) + + def f(): + 'Adding a docstring made this test fail in Py2.5.0' + return None + + self.assertNotInBytecode(f, 'LOAD_GLOBAL') + self.assertInBytecode(f, 'LOAD_CONST', None) + self.check_lnotab(f) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_while_one(self): + # Skip over: LOAD_CONST trueconst POP_JUMP_IF_FALSE xx + def f(): + while 1: + pass + return list + for elem in ('LOAD_CONST', 'POP_JUMP_IF_FALSE'): + self.assertNotInBytecode(f, elem) + for elem in ('JUMP_BACKWARD',): + self.assertInBytecode(f, elem) + self.check_lnotab(f) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_pack_unpack(self): + for line, elem in ( + ('a, = a,', 'LOAD_CONST',), + ('a, b = a, b', 'SWAP',), + ('a, b, c = a, b, c', 'SWAP',), + ): + with self.subTest(line=line): + code = compile(line,'','single') + self.assertInBytecode(code, elem) + self.assertNotInBytecode(code, 'BUILD_TUPLE') + self.assertNotInBytecode(code, 'UNPACK_SEQUENCE') + self.check_lnotab(code) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_constant_folding_tuples_of_constants(self): + for line, elem in ( + ('a = 1,2,3', (1, 2, 3)), + ('("a","b","c")', ('a', 'b', 'c')), + ('a,b,c,d = 1,2,3,4', (1, 2, 3, 4)), + ('(None, 1, None)', (None, 1, None)), + ('((1, 2), 3, 4)', ((1, 2), 3, 4)), + ): + with self.subTest(line=line): + code = compile(line,'','single') + self.assertInBytecode(code, 'LOAD_CONST', elem) + self.assertNotInBytecode(code, 'BUILD_TUPLE') + self.check_lnotab(code) + + # Long tuples should be folded too. + code = compile(repr(tuple(range(10000))),'','single') + self.assertNotInBytecode(code, 'BUILD_TUPLE') + # One LOAD_CONST for the tuple, one for the None return value + load_consts = [instr for instr in dis.get_instructions(code) + if instr.opname == 'LOAD_CONST'] + self.assertEqual(len(load_consts), 2) + self.check_lnotab(code) + + # Bug 1053819: Tuple of constants misidentified when presented with: + # . . . opcode_with_arg 100 unary_opcode BUILD_TUPLE 1 . . . + # The following would segfault upon compilation + def crater(): + (~[ + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, + ],) + self.check_lnotab(crater) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_constant_folding_lists_of_constants(self): + for line, elem in ( + # in/not in constants with BUILD_LIST should be folded to a tuple: + ('a in [1,2,3]', (1, 2, 3)), + ('a not in ["a","b","c"]', ('a', 'b', 'c')), + ('a in [None, 1, None]', (None, 1, None)), + ('a not in [(1, 2), 3, 4]', ((1, 2), 3, 4)), + ): + with self.subTest(line=line): + code = compile(line, '', 'single') + self.assertInBytecode(code, 'LOAD_CONST', elem) + self.assertNotInBytecode(code, 'BUILD_LIST') + self.check_lnotab(code) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_constant_folding_sets_of_constants(self): + for line, elem in ( + # in/not in constants with BUILD_SET should be folded to a frozenset: + ('a in {1,2,3}', frozenset({1, 2, 3})), + ('a not in {"a","b","c"}', frozenset({'a', 'c', 'b'})), + ('a in {None, 1, None}', frozenset({1, None})), + ('a not in {(1, 2), 3, 4}', frozenset({(1, 2), 3, 4})), + ('a in {1, 2, 3, 3, 2, 1}', frozenset({1, 2, 3})), + ): + with self.subTest(line=line): + code = compile(line, '', 'single') + self.assertNotInBytecode(code, 'BUILD_SET') + self.assertInBytecode(code, 'LOAD_CONST', elem) + self.check_lnotab(code) + + # Ensure that the resulting code actually works: + def f(a): + return a in {1, 2, 3} + + def g(a): + return a not in {1, 2, 3} + + self.assertTrue(f(3)) + self.assertTrue(not f(4)) + self.check_lnotab(f) + + self.assertTrue(not g(3)) + self.assertTrue(g(4)) + self.check_lnotab(g) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_constant_folding_small_int(self): + tests = [ + ('(0, )[0]', 0), + ('(1 + 2, )[0]', 3), + ('(2 + 2 * 2, )[0]', 6), + ('(1, (1 + 2 + 3, ))[1][0]', 6), + ('1 + 2', 3), + ('2 + 2 * 2 // 2 - 2', 2), + ('(255, )[0]', 255), + ('(256, )[0]', None), + ('(1000, )[0]', None), + ('(1 - 2, )[0]', None), + ('255 + 0', 255), + ('255 + 1', None), + ('-1', None), + ('--1', 1), + ('--255', 255), + ('--256', None), + ('~1', None), + ('~~1', 1), + ('~~255', 255), + ('~~256', None), + ('++255', 255), + ('++256', None), + ] + for expr, oparg in tests: + with self.subTest(expr=expr, oparg=oparg): + code = compile(expr, '', 'single') + if oparg is not None: + self.assertInBytecode(code, 'LOAD_SMALL_INT', oparg) + else: + self.assertNotInBytecode(code, 'LOAD_SMALL_INT') + self.check_lnotab(code) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_constant_folding_unaryop(self): + intrinsic_positive = 5 + tests = [ + ('-0', 'UNARY_NEGATIVE', None, True, 'LOAD_SMALL_INT', 0), + ('-0.0', 'UNARY_NEGATIVE', None, True, 'LOAD_CONST', -0.0), + ('-(1.0-1.0)', 'UNARY_NEGATIVE', None, True, 'LOAD_CONST', -0.0), + ('-0.5', 'UNARY_NEGATIVE', None, True, 'LOAD_CONST', -0.5), + ('---1', 'UNARY_NEGATIVE', None, True, 'LOAD_CONST', -1), + ('---""', 'UNARY_NEGATIVE', None, False, None, None), + ('~~~1', 'UNARY_INVERT', None, True, 'LOAD_CONST', -2), + ('~~~""', 'UNARY_INVERT', None, False, None, None), + ('not not True', 'UNARY_NOT', None, True, 'LOAD_CONST', True), + ('not not x', 'UNARY_NOT', None, True, 'LOAD_NAME', 'x'), # this should be optimized regardless of constant or not + ('+++1', 'CALL_INTRINSIC_1', intrinsic_positive, True, 'LOAD_SMALL_INT', 1), + ('---x', 'UNARY_NEGATIVE', None, False, None, None), + ('~~~x', 'UNARY_INVERT', None, False, None, None), + ('+++x', 'CALL_INTRINSIC_1', intrinsic_positive, False, None, None), + ('~True', 'UNARY_INVERT', None, False, None, None), + ] + + for ( + expr, + original_opcode, + original_argval, + is_optimized, + optimized_opcode, + optimized_argval, + ) in tests: + with self.subTest(expr=expr, is_optimized=is_optimized): + code = compile(expr, "", "single") + if is_optimized: + self.assertNotInBytecode(code, original_opcode, argval=original_argval) + self.assertInBytecode(code, optimized_opcode, argval=optimized_argval) + else: + self.assertInBytecode(code, original_opcode, argval=original_argval) + self.check_lnotab(code) + + # Check that -0.0 works after marshaling + def negzero(): + return -(1.0-1.0) + + for instr in dis.get_instructions(negzero): + self.assertNotStartsWith(instr.opname, 'UNARY_') + self.check_lnotab(negzero) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_constant_folding_binop(self): + tests = [ + ('1 + 2', 'NB_ADD', True, 'LOAD_SMALL_INT', 3), + ('1 + 2 + 3', 'NB_ADD', True, 'LOAD_SMALL_INT', 6), + ('1 + ""', 'NB_ADD', False, None, None), + ('1 - 2', 'NB_SUBTRACT', True, 'LOAD_CONST', -1), + ('1 - 2 - 3', 'NB_SUBTRACT', True, 'LOAD_CONST', -4), + ('1 - ""', 'NB_SUBTRACT', False, None, None), + ('2 * 2', 'NB_MULTIPLY', True, 'LOAD_SMALL_INT', 4), + ('2 * 2 * 2', 'NB_MULTIPLY', True, 'LOAD_SMALL_INT', 8), + ('2 / 2', 'NB_TRUE_DIVIDE', True, 'LOAD_CONST', 1.0), + ('2 / 2 / 2', 'NB_TRUE_DIVIDE', True, 'LOAD_CONST', 0.5), + ('2 / ""', 'NB_TRUE_DIVIDE', False, None, None), + ('2 // 2', 'NB_FLOOR_DIVIDE', True, 'LOAD_SMALL_INT', 1), + ('2 // 2 // 2', 'NB_FLOOR_DIVIDE', True, 'LOAD_SMALL_INT', 0), + ('2 // ""', 'NB_FLOOR_DIVIDE', False, None, None), + ('2 % 2', 'NB_REMAINDER', True, 'LOAD_SMALL_INT', 0), + ('2 % 2 % 2', 'NB_REMAINDER', True, 'LOAD_SMALL_INT', 0), + ('2 % ()', 'NB_REMAINDER', False, None, None), + ('2 ** 2', 'NB_POWER', True, 'LOAD_SMALL_INT', 4), + ('2 ** 2 ** 2', 'NB_POWER', True, 'LOAD_SMALL_INT', 16), + ('2 ** ""', 'NB_POWER', False, None, None), + ('2 << 2', 'NB_LSHIFT', True, 'LOAD_SMALL_INT', 8), + ('2 << 2 << 2', 'NB_LSHIFT', True, 'LOAD_SMALL_INT', 32), + ('2 << ""', 'NB_LSHIFT', False, None, None), + ('2 >> 2', 'NB_RSHIFT', True, 'LOAD_SMALL_INT', 0), + ('2 >> 2 >> 2', 'NB_RSHIFT', True, 'LOAD_SMALL_INT', 0), + ('2 >> ""', 'NB_RSHIFT', False, None, None), + ('2 | 2', 'NB_OR', True, 'LOAD_SMALL_INT', 2), + ('2 | 2 | 2', 'NB_OR', True, 'LOAD_SMALL_INT', 2), + ('2 | ""', 'NB_OR', False, None, None), + ('2 & 2', 'NB_AND', True, 'LOAD_SMALL_INT', 2), + ('2 & 2 & 2', 'NB_AND', True, 'LOAD_SMALL_INT', 2), + ('2 & ""', 'NB_AND', False, None, None), + ('2 ^ 2', 'NB_XOR', True, 'LOAD_SMALL_INT', 0), + ('2 ^ 2 ^ 2', 'NB_XOR', True, 'LOAD_SMALL_INT', 2), + ('2 ^ ""', 'NB_XOR', False, None, None), + ('(1, )[0]', 'NB_SUBSCR', True, 'LOAD_SMALL_INT', 1), + ('(1, )[-1]', 'NB_SUBSCR', True, 'LOAD_SMALL_INT', 1), + ('(1 + 2, )[0]', 'NB_SUBSCR', True, 'LOAD_SMALL_INT', 3), + ('(1, (1, 2))[1][1]', 'NB_SUBSCR', True, 'LOAD_SMALL_INT', 2), + ('(1, 2)[2-1]', 'NB_SUBSCR', True, 'LOAD_SMALL_INT', 2), + ('(1, (1, 2))[1][2-1]', 'NB_SUBSCR', True, 'LOAD_SMALL_INT', 2), + ('(1, (1, 2))[1:6][0][2-1]', 'NB_SUBSCR', True, 'LOAD_SMALL_INT', 2), + ('"a"[0]', 'NB_SUBSCR', True, 'LOAD_CONST', 'a'), + ('("a" + "b")[1]', 'NB_SUBSCR', True, 'LOAD_CONST', 'b'), + ('("a" + "b", )[0][1]', 'NB_SUBSCR', True, 'LOAD_CONST', 'b'), + ('("a" * 10)[9]', 'NB_SUBSCR', True, 'LOAD_CONST', 'a'), + ('(1, )[1]', 'NB_SUBSCR', False, None, None), + ('(1, )[-2]', 'NB_SUBSCR', False, None, None), + ('"a"[1]', 'NB_SUBSCR', False, None, None), + ('"a"[-2]', 'NB_SUBSCR', False, None, None), + ('("a" + "b")[2]', 'NB_SUBSCR', False, None, None), + ('("a" + "b", )[0][2]', 'NB_SUBSCR', False, None, None), + ('("a" + "b", )[1][0]', 'NB_SUBSCR', False, None, None), + ('("a" * 10)[10]', 'NB_SUBSCR', False, None, None), + ('(1, (1, 2))[2:6][0][2-1]', 'NB_SUBSCR', False, None, None), + ] + + for ( + expr, + nb_op, + is_optimized, + optimized_opcode, + optimized_argval + ) in tests: + with self.subTest(expr=expr, is_optimized=is_optimized): + code = compile(expr, '', 'single') + nb_op_val = get_binop_argval(nb_op) + if is_optimized: + self.assertNotInBytecode(code, 'BINARY_OP', argval=nb_op_val) + self.assertInBytecode(code, optimized_opcode, argval=optimized_argval) + else: + self.assertInBytecode(code, 'BINARY_OP', argval=nb_op_val) + self.check_lnotab(code) + + # Verify that large sequences do not result from folding + code = compile('"x"*10000', '', 'single') + self.assertInBytecode(code, 'LOAD_CONST', 10000) + self.assertNotIn("x"*10000, code.co_consts) + self.check_lnotab(code) + code = compile('1<<1000', '', 'single') + self.assertInBytecode(code, 'LOAD_CONST', 1000) + self.assertNotIn(1<<1000, code.co_consts) + self.check_lnotab(code) + code = compile('2**1000', '', 'single') + self.assertInBytecode(code, 'LOAD_CONST', 1000) + self.assertNotIn(2**1000, code.co_consts) + self.check_lnotab(code) + + # Test binary subscript on unicode + # valid code get optimized + code = compile('"foo"[0]', '', 'single') + self.assertInBytecode(code, 'LOAD_CONST', 'f') + self.assertNotInBytecode(code, 'BINARY_OP') + self.check_lnotab(code) + code = compile('"\u0061\uffff"[1]', '', 'single') + self.assertInBytecode(code, 'LOAD_CONST', '\uffff') + self.assertNotInBytecode(code,'BINARY_OP') + self.check_lnotab(code) + + # With PEP 393, non-BMP char get optimized + code = compile('"\U00012345"[0]', '', 'single') + self.assertInBytecode(code, 'LOAD_CONST', '\U00012345') + self.assertNotInBytecode(code, 'BINARY_OP') + self.check_lnotab(code) + + # invalid code doesn't get optimized + # out of range + code = compile('"fuu"[10]', '', 'single') + self.assertInBytecode(code, 'BINARY_OP') + self.check_lnotab(code) + + + def test_constant_folding_remove_nop_location(self): + sources = [ + """ + (- + - + - + 1) + """, + + """ + (1 + + + 2 + + + 3) + """, + + """ + (1, + 2, + 3)[0] + """, + + """ + [1, + 2, + 3] + """, + + """ + {1, + 2, + 3} + """, + + """ + 1 in [ + 1, + 2, + 3 + ] + """, + + """ + 1 in { + 1, + 2, + 3 + } + """, + + """ + for _ in [1, + 2, + 3]: + pass + """, + + """ + for _ in [1, + 2, + x]: + pass + """, + + """ + for _ in {1, + 2, + 3}: + pass + """ + ] + + for source in sources: + code = compile(textwrap.dedent(source), '', 'single') + self.assertNotInBytecode(code, 'NOP') + + def test_elim_extra_return(self): + # RETURN LOAD_CONST None RETURN --> RETURN + def f(x): + return x + self.assertNotInBytecode(f, 'LOAD_CONST', None) + returns = [instr for instr in dis.get_instructions(f) + if instr.opname == 'RETURN_VALUE'] + self.assertEqual(len(returns), 1) + self.check_lnotab(f) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_elim_jump_to_return(self): + # JUMP_FORWARD to RETURN --> RETURN + def f(cond, true_value, false_value): + # Intentionally use two-line expression to test issue37213. + return (true_value if cond + else false_value) + self.check_jump_targets(f) + self.assertNotInBytecode(f, 'JUMP_FORWARD') + self.assertNotInBytecode(f, 'JUMP_BACKWARD') + returns = [instr for instr in dis.get_instructions(f) + if instr.opname == 'RETURN_VALUE'] + self.assertEqual(len(returns), 2) + self.check_lnotab(f) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_elim_jump_to_uncond_jump(self): + # POP_JUMP_IF_FALSE to JUMP_FORWARD --> POP_JUMP_IF_FALSE to non-jump + def f(): + if a: + # Intentionally use two-line expression to test issue37213. + if (c + or d): + foo() + else: + baz() + self.check_jump_targets(f) + self.check_lnotab(f) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_elim_jump_to_uncond_jump2(self): + # POP_JUMP_IF_FALSE to JUMP_BACKWARD --> POP_JUMP_IF_FALSE to non-jump + def f(): + while a: + # Intentionally use two-line expression to test issue37213. + if (c + or d): + a = foo() + self.check_jump_targets(f) + self.check_lnotab(f) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_elim_jump_to_uncond_jump3(self): + # Intentionally use two-line expressions to test issue37213. + # POP_JUMP_IF_FALSE to POP_JUMP_IF_FALSE --> POP_JUMP_IF_FALSE to non-jump + def f(a, b, c): + return ((a and b) + and c) + self.check_jump_targets(f) + self.check_lnotab(f) + self.assertEqual(count_instr_recursively(f, 'POP_JUMP_IF_FALSE'), 2) + # POP_JUMP_IF_TRUE to POP_JUMP_IF_TRUE --> POP_JUMP_IF_TRUE to non-jump + def f(a, b, c): + return ((a or b) + or c) + self.check_jump_targets(f) + self.check_lnotab(f) + self.assertEqual(count_instr_recursively(f, 'POP_JUMP_IF_TRUE'), 2) + # JUMP_IF_FALSE_OR_POP to JUMP_IF_TRUE_OR_POP --> POP_JUMP_IF_FALSE to non-jump + def f(a, b, c): + return ((a and b) + or c) + self.check_jump_targets(f) + self.check_lnotab(f) + self.assertEqual(count_instr_recursively(f, 'POP_JUMP_IF_FALSE'), 1) + self.assertEqual(count_instr_recursively(f, 'POP_JUMP_IF_TRUE'), 1) + # POP_JUMP_IF_TRUE to POP_JUMP_IF_FALSE --> POP_JUMP_IF_TRUE to non-jump + def f(a, b, c): + return ((a or b) + and c) + self.check_jump_targets(f) + self.check_lnotab(f) + self.assertEqual(count_instr_recursively(f, 'POP_JUMP_IF_FALSE'), 1) + self.assertEqual(count_instr_recursively(f, 'POP_JUMP_IF_TRUE'), 1) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_elim_jump_to_uncond_jump4(self): + def f(): + for i in range(5): + if i > 3: + print(i) + self.check_jump_targets(f) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_elim_jump_after_return1(self): + # Eliminate dead code: jumps immediately after returns can't be reached + def f(cond1, cond2): + if cond1: return 1 + if cond2: return 2 + while 1: + return 3 + while 1: + if cond1: return 4 + return 5 + return 6 + self.assertNotInBytecode(f, 'JUMP_FORWARD') + self.assertNotInBytecode(f, 'JUMP_BACKWARD') + returns = [instr for instr in dis.get_instructions(f) + if instr.opname == 'RETURN_VALUE'] + self.assertLessEqual(len(returns), 6) + self.check_lnotab(f) + + def test_make_function_doesnt_bail(self): + def f(): + def g()->1+1: + pass + return g + self.assertNotInBytecode(f, 'BINARY_OP') + self.check_lnotab(f) + + def test_in_literal_list(self): + def containtest(): + return x in [a, b] + self.assertEqual(count_instr_recursively(containtest, 'BUILD_LIST'), 0) + self.check_lnotab(containtest) + + def test_iterate_literal_list(self): + def forloop(): + for x in [a, b]: + pass + self.assertEqual(count_instr_recursively(forloop, 'BUILD_LIST'), 0) + self.check_lnotab(forloop) + + def test_condition_with_binop_with_bools(self): + def f(): + if True or False: + return 1 + return 0 + self.assertEqual(f(), 1) + self.check_lnotab(f) + + def test_if_with_if_expression(self): + # Check bpo-37289 + def f(x): + if (True if x else False): + return True + return False + self.assertTrue(f(True)) + self.check_lnotab(f) + + def test_trailing_nops(self): + # Check the lnotab of a function that even after trivial + # optimization has trailing nops, which the lnotab adjustment has to + # handle properly (bpo-38115). + def f(x): + while 1: + return 3 + while 1: + return 5 + return 6 + self.check_lnotab(f) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_assignment_idiom_in_comprehensions(self): + def listcomp(): + return [y for x in a for y in [f(x)]] + self.assertEqual(count_instr_recursively(listcomp, 'FOR_ITER'), 1) + def setcomp(): + return {y for x in a for y in [f(x)]} + self.assertEqual(count_instr_recursively(setcomp, 'FOR_ITER'), 1) + def dictcomp(): + return {y: y for x in a for y in [f(x)]} + self.assertEqual(count_instr_recursively(dictcomp, 'FOR_ITER'), 1) + def genexpr(): + return (y for x in a for y in [f(x)]) + self.assertEqual(count_instr_recursively(genexpr, 'FOR_ITER'), 1) + + @support.requires_resource('cpu') + def test_format_combinations(self): + flags = '-+ #0' + testcases = [ + *product(('', '1234', 'абвг'), 'sra'), + *product((1234, -1234), 'duioxX'), + *product((1234.5678901, -1234.5678901), 'duifegFEG'), + *product((float('inf'), -float('inf')), 'fegFEG'), + ] + width_precs = [ + *product(('', '1', '30'), ('', '.', '.0', '.2')), + ('', '.40'), + ('30', '.40'), + ] + for value, suffix in testcases: + for width, prec in width_precs: + for r in range(len(flags) + 1): + for spec in combinations(flags, r): + fmt = '%' + ''.join(spec) + width + prec + suffix + with self.subTest(fmt=fmt, value=value): + s1 = fmt % value + s2 = eval(f'{fmt!r} % (x,)', {'x': value}) + self.assertEqual(s2, s1, f'{fmt = }') + + def test_format_misc(self): + def format(fmt, *values): + vars = [f'x{i+1}' for i in range(len(values))] + if len(vars) == 1: + args = '(' + vars[0] + ',)' + else: + args = '(' + ', '.join(vars) + ')' + return eval(f'{fmt!r} % {args}', dict(zip(vars, values))) + + self.assertEqual(format('string'), 'string') + self.assertEqual(format('x = %s!', 1234), 'x = 1234!') + self.assertEqual(format('x = %d!', 1234), 'x = 1234!') + self.assertEqual(format('x = %x!', 1234), 'x = 4d2!') + self.assertEqual(format('x = %f!', 1234), 'x = 1234.000000!') + self.assertEqual(format('x = %s!', 1234.0000625), 'x = 1234.0000625!') + self.assertEqual(format('x = %f!', 1234.0000625), 'x = 1234.000063!') + self.assertEqual(format('x = %d!', 1234.0000625), 'x = 1234!') + self.assertEqual(format('x = %s%% %%%%', 1234), 'x = 1234% %%') + self.assertEqual(format('x = %s!', '%% %s'), 'x = %% %s!') + self.assertEqual(format('x = %s, y = %d', 12, 34), 'x = 12, y = 34') + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_format_errors(self): + with self.assertRaisesRegex(TypeError, + 'not enough arguments for format string'): + eval("'%s' % ()") + with self.assertRaisesRegex(TypeError, + 'not all arguments converted during string formatting'): + eval("'%s' % (x, y)", {'x': 1, 'y': 2}) + with self.assertRaisesRegex(ValueError, 'incomplete format'): + eval("'%s%' % (x,)", {'x': 1234}) + with self.assertRaisesRegex(ValueError, 'incomplete format'): + eval("'%s%%%' % (x,)", {'x': 1234}) + with self.assertRaisesRegex(TypeError, + 'not enough arguments for format string'): + eval("'%s%z' % (x,)", {'x': 1234}) + with self.assertRaisesRegex(ValueError, 'unsupported format character'): + eval("'%s%z' % (x, 5)", {'x': 1234}) + with self.assertRaisesRegex(TypeError, 'a real number is required, not str'): + eval("'%d' % (x,)", {'x': '1234'}) + with self.assertRaisesRegex(TypeError, 'an integer is required, not float'): + eval("'%x' % (x,)", {'x': 1234.56}) + with self.assertRaisesRegex(TypeError, 'an integer is required, not str'): + eval("'%x' % (x,)", {'x': '1234'}) + with self.assertRaisesRegex(TypeError, 'must be real number, not str'): + eval("'%f' % (x,)", {'x': '1234'}) + with self.assertRaisesRegex(TypeError, + 'not enough arguments for format string'): + eval("'%s, %s' % (x, *y)", {'x': 1, 'y': []}) + with self.assertRaisesRegex(TypeError, + 'not all arguments converted during string formatting'): + eval("'%s, %s' % (x, *y)", {'x': 1, 'y': [2, 3]}) + + def test_static_swaps_unpack_two(self): + def f(a, b): + a, b = a, b + b, a = a, b + self.assertNotInBytecode(f, "SWAP") + + def test_static_swaps_unpack_three(self): + def f(a, b, c): + a, b, c = a, b, c + a, c, b = a, b, c + b, a, c = a, b, c + b, c, a = a, b, c + c, a, b = a, b, c + c, b, a = a, b, c + self.assertNotInBytecode(f, "SWAP") + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_static_swaps_match_mapping(self): + for a, b, c in product("_a", "_b", "_c"): + pattern = f"{{'a': {a}, 'b': {b}, 'c': {c}}}" + with self.subTest(pattern): + code = compile_pattern_with_fast_locals(pattern) + self.assertNotInBytecode(code, "SWAP") + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_static_swaps_match_class(self): + forms = [ + "C({}, {}, {})", + "C({}, {}, c={})", + "C({}, b={}, c={})", + "C(a={}, b={}, c={})" + ] + for a, b, c in product("_a", "_b", "_c"): + for form in forms: + pattern = form.format(a, b, c) + with self.subTest(pattern): + code = compile_pattern_with_fast_locals(pattern) + self.assertNotInBytecode(code, "SWAP") + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_static_swaps_match_sequence(self): + swaps = {"*_, b, c", "a, *_, c", "a, b, *_"} + forms = ["{}, {}, {}", "{}, {}, *{}", "{}, *{}, {}", "*{}, {}, {}"] + for a, b, c in product("_a", "_b", "_c"): + for form in forms: + pattern = form.format(a, b, c) + with self.subTest(pattern): + code = compile_pattern_with_fast_locals(pattern) + if pattern in swaps: + # If this fails... great! Remove this pattern from swaps + # to prevent regressing on any improvement: + self.assertInBytecode(code, "SWAP") + else: + self.assertNotInBytecode(code, "SWAP") + + +class TestBuglets(unittest.TestCase): + + def test_bug_11510(self): + # folded constant set optimization was commingled with the tuple + # unpacking optimization which would fail if the set had duplicate + # elements so that the set length was unexpected + def f(): + x, y = {1, 1} + return x, y + with self.assertRaises(ValueError): + f() + + def test_bpo_42057(self): + for i in range(10): + try: + raise Exception + except Exception or Exception: + pass + + def test_bpo_45773_pop_jump_if_true(self): + compile("while True or spam: pass", "<test>", "exec") + + def test_bpo_45773_pop_jump_if_false(self): + compile("while True or not spam: pass", "<test>", "exec") + + +class TestMarkingVariablesAsUnKnown(BytecodeTestCase): + + def setUp(self): + self.addCleanup(sys.settrace, sys.gettrace()) + sys.settrace(None) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_load_fast_known_simple(self): + def f(): + x = 1 + y = x + x + self.assertInBytecode(f, 'LOAD_FAST_BORROW_LOAD_FAST_BORROW') + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_load_fast_unknown_simple(self): + def f(): + if condition(): + x = 1 + print(x) + self.assertInBytecode(f, 'LOAD_FAST_CHECK') + self.assertNotInBytecode(f, 'LOAD_FAST') + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_load_fast_unknown_because_del(self): + def f(): + x = 1 + del x + print(x) + self.assertInBytecode(f, 'LOAD_FAST_CHECK') + self.assertNotInBytecode(f, 'LOAD_FAST') + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_load_fast_known_because_parameter(self): + def f1(x): + print(x) + self.assertInBytecode(f1, 'LOAD_FAST_BORROW') + self.assertNotInBytecode(f1, 'LOAD_FAST_CHECK') + + def f2(*, x): + print(x) + self.assertInBytecode(f2, 'LOAD_FAST_BORROW') + self.assertNotInBytecode(f2, 'LOAD_FAST_CHECK') + + def f3(*args): + print(args) + self.assertInBytecode(f3, 'LOAD_FAST_BORROW') + self.assertNotInBytecode(f3, 'LOAD_FAST_CHECK') + + def f4(**kwargs): + print(kwargs) + self.assertInBytecode(f4, 'LOAD_FAST_BORROW') + self.assertNotInBytecode(f4, 'LOAD_FAST_CHECK') + + def f5(x=0): + print(x) + self.assertInBytecode(f5, 'LOAD_FAST_BORROW') + self.assertNotInBytecode(f5, 'LOAD_FAST_CHECK') + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_load_fast_known_because_already_loaded(self): + def f(): + if condition(): + x = 1 + print(x) + print(x) + self.assertInBytecode(f, 'LOAD_FAST_CHECK') + self.assertInBytecode(f, 'LOAD_FAST_BORROW') + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_load_fast_known_multiple_branches(self): + def f(): + if condition(): + x = 1 + else: + x = 2 + print(x) + self.assertInBytecode(f, 'LOAD_FAST_BORROW') + self.assertNotInBytecode(f, 'LOAD_FAST_CHECK') + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_load_fast_unknown_after_error(self): + def f(): + try: + res = 1 / 0 + except ZeroDivisionError: + pass + return res + # LOAD_FAST (known) still occurs in the no-exception branch. + # Assert that it doesn't occur in the LOAD_FAST_CHECK branch. + self.assertInBytecode(f, 'LOAD_FAST_CHECK') + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_load_fast_unknown_after_error_2(self): + def f(): + try: + 1 / 0 + except ZeroDivisionError: + print(a, b, c, d, e, f, g) + a = b = c = d = e = f = g = 1 + self.assertInBytecode(f, 'LOAD_FAST_CHECK') + self.assertNotInBytecode(f, 'LOAD_FAST') + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_load_fast_too_many_locals(self): + # When there get to be too many locals to analyze completely, + # later locals are all converted to LOAD_FAST_CHECK, except + # when a store or prior load occurred in the same basicblock. + def f(): + a00 = a01 = a02 = a03 = a04 = a05 = a06 = a07 = a08 = a09 = 1 + a10 = a11 = a12 = a13 = a14 = a15 = a16 = a17 = a18 = a19 = 1 + a20 = a21 = a22 = a23 = a24 = a25 = a26 = a27 = a28 = a29 = 1 + a30 = a31 = a32 = a33 = a34 = a35 = a36 = a37 = a38 = a39 = 1 + a40 = a41 = a42 = a43 = a44 = a45 = a46 = a47 = a48 = a49 = 1 + a50 = a51 = a52 = a53 = a54 = a55 = a56 = a57 = a58 = a59 = 1 + a60 = a61 = a62 = a63 = a64 = a65 = a66 = a67 = a68 = a69 = 1 + a70 = a71 = a72 = a73 = a74 = a75 = a76 = a77 = a78 = a79 = 1 + del a72, a73 + print(a73) + print(a70, a71, a72, a73) + while True: + print(a00, a01, a62, a63) + print(a64, a65, a78, a79) + + self.assertInBytecode(f, 'LOAD_FAST_BORROW_LOAD_FAST_BORROW', ("a00", "a01")) + self.assertNotInBytecode(f, 'LOAD_FAST_CHECK', "a00") + self.assertNotInBytecode(f, 'LOAD_FAST_CHECK', "a01") + for i in 62, 63: + # First 64 locals: analyze completely + self.assertInBytecode(f, 'LOAD_FAST_BORROW', f"a{i:02}") + self.assertNotInBytecode(f, 'LOAD_FAST_CHECK', f"a{i:02}") + for i in 64, 65, 78, 79: + # Locals >=64 not in the same basicblock + self.assertInBytecode(f, 'LOAD_FAST_CHECK', f"a{i:02}") + self.assertNotInBytecode(f, 'LOAD_FAST', f"a{i:02}") + for i in 70, 71: + # Locals >=64 in the same basicblock + self.assertInBytecode(f, 'LOAD_FAST_BORROW', f"a{i:02}") + self.assertNotInBytecode(f, 'LOAD_FAST_CHECK', f"a{i:02}") + # del statements should invalidate within basicblocks. + self.assertInBytecode(f, 'LOAD_FAST_CHECK', "a72") + self.assertNotInBytecode(f, 'LOAD_FAST', "a72") + # previous checked loads within a basicblock enable unchecked loads + self.assertInBytecode(f, 'LOAD_FAST_CHECK', "a73") + self.assertInBytecode(f, 'LOAD_FAST_BORROW', "a73") + + def test_setting_lineno_no_undefined(self): + code = textwrap.dedent("""\ + def f(): + x = y = 2 + if not x: + return 4 + for i in range(55): + x + 6 + L = 7 + L = 8 + L = 9 + L = 10 + """) + ns = {} + exec(code, ns) + f = ns['f'] + self.assertInBytecode(f, "LOAD_FAST_BORROW") + self.assertNotInBytecode(f, "LOAD_FAST_CHECK") + co_code = f.__code__.co_code + def trace(frame, event, arg): + if event == 'line' and frame.f_lineno == 9: + frame.f_lineno = 3 + sys.settrace(None) + return None + return trace + sys.settrace(trace) + result = f() + self.assertIsNone(result) + self.assertInBytecode(f, "LOAD_FAST_BORROW") + self.assertNotInBytecode(f, "LOAD_FAST_CHECK") + self.assertEqual(f.__code__.co_code, co_code) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_setting_lineno_one_undefined(self): + code = textwrap.dedent("""\ + def f(): + x = y = 2 + if not x: + return 4 + for i in range(55): + x + 6 + del x + L = 8 + L = 9 + L = 10 + """) + ns = {} + exec(code, ns) + f = ns['f'] + self.assertInBytecode(f, "LOAD_FAST_BORROW") + self.assertNotInBytecode(f, "LOAD_FAST_CHECK") + co_code = f.__code__.co_code + def trace(frame, event, arg): + if event == 'line' and frame.f_lineno == 9: + frame.f_lineno = 3 + sys.settrace(None) + return None + return trace + e = r"assigning None to 1 unbound local" + with self.assertWarnsRegex(RuntimeWarning, e): + sys.settrace(trace) + result = f() + self.assertEqual(result, 4) + self.assertInBytecode(f, "LOAD_FAST_BORROW") + self.assertNotInBytecode(f, "LOAD_FAST_CHECK") + self.assertEqual(f.__code__.co_code, co_code) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_setting_lineno_two_undefined(self): + code = textwrap.dedent("""\ + def f(): + x = y = 2 + if not x: + return 4 + for i in range(55): + x + 6 + del x, y + L = 8 + L = 9 + L = 10 + """) + ns = {} + exec(code, ns) + f = ns['f'] + self.assertInBytecode(f, "LOAD_FAST_BORROW") + self.assertNotInBytecode(f, "LOAD_FAST_CHECK") + co_code = f.__code__.co_code + def trace(frame, event, arg): + if event == 'line' and frame.f_lineno == 9: + frame.f_lineno = 3 + sys.settrace(None) + return None + return trace + e = r"assigning None to 2 unbound locals" + with self.assertWarnsRegex(RuntimeWarning, e): + sys.settrace(trace) + result = f() + self.assertEqual(result, 4) + self.assertInBytecode(f, "LOAD_FAST_BORROW") + self.assertNotInBytecode(f, "LOAD_FAST_CHECK") + self.assertEqual(f.__code__.co_code, co_code) + + def make_function_with_no_checks(self): + code = textwrap.dedent("""\ + def f(): + x = 2 + L = 3 + L = 4 + L = 5 + if not L: + x + 7 + y = 2 + """) + ns = {} + exec(code, ns) + f = ns['f'] + self.assertInBytecode(f, "LOAD_FAST_BORROW") + self.assertNotInBytecode(f, "LOAD_FAST_CHECK") + return f + + def test_modifying_local_does_not_add_check(self): + f = self.make_function_with_no_checks() + def trace(frame, event, arg): + if event == 'line' and frame.f_lineno == 4: + frame.f_locals["x"] = 42 + sys.settrace(None) + return None + return trace + sys.settrace(trace) + f() + self.assertInBytecode(f, "LOAD_FAST_BORROW") + self.assertNotInBytecode(f, "LOAD_FAST_CHECK") + + def test_initializing_local_does_not_add_check(self): + f = self.make_function_with_no_checks() + def trace(frame, event, arg): + if event == 'line' and frame.f_lineno == 4: + frame.f_locals["y"] = 42 + sys.settrace(None) + return None + return trace + sys.settrace(trace) + f() + self.assertInBytecode(f, "LOAD_FAST_BORROW") + self.assertNotInBytecode(f, "LOAD_FAST_CHECK") + + +class DirectCfgOptimizerTests(CfgOptimizationTestCase): + + def cfg_optimization_test(self, insts, expected_insts, + consts=None, expected_consts=None, + nlocals=0): + + self.check_instructions(insts) + self.check_instructions(expected_insts) + + if expected_consts is None: + expected_consts = consts + seq = self.seq_from_insts(insts) + opt_insts, opt_consts = self.get_optimized(seq, consts, nlocals) + expected_insts = self.seq_from_insts(expected_insts).get_instructions() + self.assertInstructionsMatch(opt_insts, expected_insts) + self.assertEqual(opt_consts, expected_consts) + + def test_conditional_jump_forward_non_const_condition(self): + insts = [ + ('LOAD_NAME', 1, 11), + ('POP_JUMP_IF_TRUE', lbl := self.Label(), 12), + ('LOAD_CONST', 2, 13), + ('RETURN_VALUE', None, 13), + lbl, + ('LOAD_CONST', 3, 14), + ('RETURN_VALUE', None, 14), + ] + expected_insts = [ + ('LOAD_NAME', 1, 11), + ('POP_JUMP_IF_TRUE', lbl := self.Label(), 12), + ('LOAD_SMALL_INT', 2, 13), + ('RETURN_VALUE', None, 13), + lbl, + ('LOAD_SMALL_INT', 3, 14), + ('RETURN_VALUE', None, 14), + ] + self.cfg_optimization_test(insts, + expected_insts, + consts=[0, 1, 2, 3, 4], + expected_consts=[0]) + + def test_list_exceeding_stack_use_guideline(self): + def f(): + return [ + 0, 1, 2, 3, 4, + 5, 6, 7, 8, 9, + 10, 11, 12, 13, 14, + 15, 16, 17, 18, 19, + 20, 21, 22, 23, 24, + 25, 26, 27, 28, 29, + 30, 31, 32, 33, 34, + 35, 36, 37, 38, 39 + ] + self.assertEqual(f(), list(range(40))) + + def test_set_exceeding_stack_use_guideline(self): + def f(): + return { + 0, 1, 2, 3, 4, + 5, 6, 7, 8, 9, + 10, 11, 12, 13, 14, + 15, 16, 17, 18, 19, + 20, 21, 22, 23, 24, + 25, 26, 27, 28, 29, + 30, 31, 32, 33, 34, + 35, 36, 37, 38, 39 + } + self.assertEqual(f(), frozenset(range(40))) + + def test_nested_const_foldings(self): + # (1, (--2 + ++2 * 2 // 2 - 2, )[0], ~~3, not not True) ==> (1, 2, 3, True) + intrinsic_positive = 5 + before = [ + ('LOAD_SMALL_INT', 1, 0), + ('NOP', None, 0), + ('LOAD_SMALL_INT', 2, 0), + ('UNARY_NEGATIVE', None, 0), + ('NOP', None, 0), + ('UNARY_NEGATIVE', None, 0), + ('NOP', None, 0), + ('NOP', None, 0), + ('LOAD_SMALL_INT', 2, 0), + ('CALL_INTRINSIC_1', intrinsic_positive, 0), + ('NOP', None, 0), + ('CALL_INTRINSIC_1', intrinsic_positive, 0), + ('BINARY_OP', get_binop_argval('NB_MULTIPLY')), + ('LOAD_SMALL_INT', 2, 0), + ('NOP', None, 0), + ('BINARY_OP', get_binop_argval('NB_FLOOR_DIVIDE')), + ('NOP', None, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', get_binop_argval('NB_ADD')), + ('NOP', None, 0), + ('LOAD_SMALL_INT', 2, 0), + ('NOP', None, 0), + ('BINARY_OP', get_binop_argval('NB_SUBTRACT')), + ('NOP', None, 0), + ('BUILD_TUPLE', 1, 0), + ('LOAD_SMALL_INT', 0, 0), + ('BINARY_OP', get_binop_argval('NB_SUBSCR'), 0), + ('NOP', None, 0), + ('LOAD_SMALL_INT', 3, 0), + ('NOP', None, 0), + ('UNARY_INVERT', None, 0), + ('NOP', None, 0), + ('UNARY_INVERT', None, 0), + ('NOP', None, 0), + ('LOAD_SMALL_INT', 3, 0), + ('NOP', None, 0), + ('UNARY_NOT', None, 0), + ('NOP', None, 0), + ('UNARY_NOT', None, 0), + ('NOP', None, 0), + ('BUILD_TUPLE', 4, 0), + ('NOP', None, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_CONST', 1, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[-2, (1, 2, 3, True)]) + + def test_build_empty_tuple(self): + before = [ + ('BUILD_TUPLE', 0, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[()]) + + def test_fold_tuple_of_constants(self): + before = [ + ('NOP', None, 0), + ('LOAD_SMALL_INT', 1, 0), + ('NOP', None, 0), + ('LOAD_SMALL_INT', 2, 0), + ('NOP', None, 0), + ('NOP', None, 0), + ('LOAD_SMALL_INT', 3, 0), + ('NOP', None, 0), + ('BUILD_TUPLE', 3, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[(1, 2, 3)]) + + # not all consts + same = [ + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_NAME', 0, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BUILD_TUPLE', 3, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(same, same, consts=[]) + + def test_fold_constant_intrinsic_list_to_tuple(self): + INTRINSIC_LIST_TO_TUPLE = 6 + + # long tuple + consts = 1000 + before = ( + [('BUILD_LIST', 0, 0)] + + [('LOAD_CONST', 0, 0), ('LIST_APPEND', 1, 0)] * consts + + [('CALL_INTRINSIC_1', INTRINSIC_LIST_TO_TUPLE, 0), ('RETURN_VALUE', None, 0)] + ) + after = [ + ('LOAD_CONST', 1, 0), + ('RETURN_VALUE', None, 0) + ] + result_const = tuple(["test"] * consts) + self.cfg_optimization_test(before, after, consts=["test"], expected_consts=["test", result_const]) + + # empty list + before = [ + ('BUILD_LIST', 0, 0), + ('CALL_INTRINSIC_1', INTRINSIC_LIST_TO_TUPLE, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[()]) + + # multiple BUILD_LIST 0: ([], 1, [], 2) + same = [ + ('BUILD_LIST', 0, 0), + ('BUILD_LIST', 0, 0), + ('LIST_APPEND', 1, 0), + ('LOAD_SMALL_INT', 1, 0), + ('LIST_APPEND', 1, 0), + ('BUILD_LIST', 0, 0), + ('LIST_APPEND', 1, 0), + ('LOAD_SMALL_INT', 2, 0), + ('LIST_APPEND', 1, 0), + ('CALL_INTRINSIC_1', INTRINSIC_LIST_TO_TUPLE, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(same, same, consts=[]) + + # nested folding: (1, 1+1, 3) + before = [ + ('BUILD_LIST', 0, 0), + ('LOAD_SMALL_INT', 1, 0), + ('LIST_APPEND', 1, 0), + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_SMALL_INT', 1, 0), + ('BINARY_OP', get_binop_argval('NB_ADD'), 0), + ('LIST_APPEND', 1, 0), + ('LOAD_SMALL_INT', 3, 0), + ('LIST_APPEND', 1, 0), + ('CALL_INTRINSIC_1', INTRINSIC_LIST_TO_TUPLE, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[(1, 2, 3)]) + + # NOP's in between: (1, 2, 3) + before = [ + ('BUILD_LIST', 0, 0), + ('NOP', None, 0), + ('LOAD_SMALL_INT', 1, 0), + ('NOP', None, 0), + ('NOP', None, 0), + ('LIST_APPEND', 1, 0), + ('NOP', None, 0), + ('LOAD_SMALL_INT', 2, 0), + ('NOP', None, 0), + ('NOP', None, 0), + ('LIST_APPEND', 1, 0), + ('NOP', None, 0), + ('LOAD_SMALL_INT', 3, 0), + ('NOP', None, 0), + ('LIST_APPEND', 1, 0), + ('NOP', None, 0), + ('CALL_INTRINSIC_1', INTRINSIC_LIST_TO_TUPLE, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[(1, 2, 3)]) + + def test_optimize_if_const_list(self): + before = [ + ('NOP', None, 0), + ('LOAD_SMALL_INT', 1, 0), + ('NOP', None, 0), + ('LOAD_SMALL_INT', 2, 0), + ('NOP', None, 0), + ('NOP', None, 0), + ('LOAD_SMALL_INT', 3, 0), + ('NOP', None, 0), + ('BUILD_LIST', 3, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('BUILD_LIST', 0, 0), + ('LOAD_CONST', 0, 0), + ('LIST_EXTEND', 1, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[(1, 2, 3)]) + + # need minimum 3 consts to optimize + same = [ + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BUILD_LIST', 2, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(same, same, consts=[]) + + # not all consts + same = [ + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_NAME', 0, 0), + ('LOAD_SMALL_INT', 3, 0), + ('BUILD_LIST', 3, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(same, same, consts=[]) + + def test_optimize_if_const_set(self): + before = [ + ('NOP', None, 0), + ('LOAD_SMALL_INT', 1, 0), + ('NOP', None, 0), + ('LOAD_SMALL_INT', 2, 0), + ('NOP', None, 0), + ('NOP', None, 0), + ('LOAD_SMALL_INT', 3, 0), + ('NOP', None, 0), + ('BUILD_SET', 3, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('BUILD_SET', 0, 0), + ('LOAD_CONST', 0, 0), + ('SET_UPDATE', 1, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[frozenset({1, 2, 3})]) + + # need minimum 3 consts to optimize + same = [ + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BUILD_SET', 2, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(same, same, consts=[]) + + # not all consts + same = [ + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_NAME', 0, 0), + ('LOAD_SMALL_INT', 3, 0), + ('BUILD_SET', 3, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(same, same, consts=[]) + + def test_optimize_literal_list_for_iter(self): + # for _ in [1, 2]: pass ==> for _ in (1, 2): pass + before = [ + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BUILD_LIST', 2, 0), + ('GET_ITER', None, 0), + start := self.Label(), + ('FOR_ITER', end := self.Label(), 0), + ('STORE_FAST', 0, 0), + ('JUMP', start, 0), + end, + ('END_FOR', None, 0), + ('POP_ITER', None, 0), + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_CONST', 1, 0), + ('GET_ITER', None, 0), + start := self.Label(), + ('FOR_ITER', end := self.Label(), 0), + ('STORE_FAST', 0, 0), + ('JUMP', start, 0), + end, + ('END_FOR', None, 0), + ('POP_ITER', None, 0), + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[None], expected_consts=[None, (1, 2)]) + + # for _ in [1, x]: pass ==> for _ in (1, x): pass + before = [ + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_NAME', 0, 0), + ('BUILD_LIST', 2, 0), + ('GET_ITER', None, 0), + start := self.Label(), + ('FOR_ITER', end := self.Label(), 0), + ('STORE_FAST', 0, 0), + ('JUMP', start, 0), + end, + ('END_FOR', None, 0), + ('POP_ITER', None, 0), + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_NAME', 0, 0), + ('BUILD_TUPLE', 2, 0), + ('GET_ITER', None, 0), + start := self.Label(), + ('FOR_ITER', end := self.Label(), 0), + ('STORE_FAST', 0, 0), + ('JUMP', start, 0), + end, + ('END_FOR', None, 0), + ('POP_ITER', None, 0), + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[None], expected_consts=[None]) + + def test_optimize_literal_set_for_iter(self): + # for _ in {1, 2}: pass ==> for _ in (1, 2): pass + before = [ + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BUILD_SET', 2, 0), + ('GET_ITER', None, 0), + start := self.Label(), + ('FOR_ITER', end := self.Label(), 0), + ('STORE_FAST', 0, 0), + ('JUMP', start, 0), + end, + ('END_FOR', None, 0), + ('POP_ITER', None, 0), + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_CONST', 1, 0), + ('GET_ITER', None, 0), + start := self.Label(), + ('FOR_ITER', end := self.Label(), 0), + ('STORE_FAST', 0, 0), + ('JUMP', start, 0), + end, + ('END_FOR', None, 0), + ('POP_ITER', None, 0), + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[None], expected_consts=[None, frozenset({1, 2})]) + + # non constant literal set is not changed + # for _ in {1, x}: pass ==> for _ in {1, x}: pass + same = [ + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_NAME', 0, 0), + ('BUILD_SET', 2, 0), + ('GET_ITER', None, 0), + start := self.Label(), + ('FOR_ITER', end := self.Label(), 0), + ('STORE_FAST', 0, 0), + ('JUMP', start, 0), + end, + ('END_FOR', None, 0), + ('POP_ITER', None, 0), + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(same, same, consts=[None], expected_consts=[None]) + + def test_optimize_literal_list_contains(self): + # x in [1, 2] ==> x in (1, 2) + before = [ + ('LOAD_NAME', 0, 0), + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BUILD_LIST', 2, 0), + ('CONTAINS_OP', 0, 0), + ('POP_TOP', None, 0), + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_NAME', 0, 0), + ('LOAD_CONST', 1, 0), + ('CONTAINS_OP', 0, 0), + ('POP_TOP', None, 0), + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[None], expected_consts=[None, (1, 2)]) + + # x in [1, y] ==> x in (1, y) + before = [ + ('LOAD_NAME', 0, 0), + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_NAME', 1, 0), + ('BUILD_LIST', 2, 0), + ('CONTAINS_OP', 0, 0), + ('POP_TOP', None, 0), + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_NAME', 0, 0), + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_NAME', 1, 0), + ('BUILD_TUPLE', 2, 0), + ('CONTAINS_OP', 0, 0), + ('POP_TOP', None, 0), + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[None], expected_consts=[None]) + + def test_optimize_literal_set_contains(self): + # x in {1, 2} ==> x in (1, 2) + before = [ + ('LOAD_NAME', 0, 0), + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BUILD_SET', 2, 0), + ('CONTAINS_OP', 0, 0), + ('POP_TOP', None, 0), + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_NAME', 0, 0), + ('LOAD_CONST', 1, 0), + ('CONTAINS_OP', 0, 0), + ('POP_TOP', None, 0), + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[None], expected_consts=[None, frozenset({1, 2})]) + + # non constant literal set is not changed + # x in {1, y} ==> x in {1, y} + same = [ + ('LOAD_NAME', 0, 0), + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_NAME', 1, 0), + ('BUILD_SET', 2, 0), + ('CONTAINS_OP', 0, 0), + ('POP_TOP', None, 0), + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(same, same, consts=[None], expected_consts=[None]) + + def test_optimize_unary_not(self): + # test folding + before = [ + ('LOAD_SMALL_INT', 1, 0), + ('TO_BOOL', None, 0), + ('UNARY_NOT', None, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_CONST', 1, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[True, False]) + + # test cancel out + before = [ + ('LOAD_NAME', 0, 0), + ('TO_BOOL', None, 0), + ('UNARY_NOT', None, 0), + ('UNARY_NOT', None, 0), + ('UNARY_NOT', None, 0), + ('UNARY_NOT', None, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_NAME', 0, 0), + ('TO_BOOL', None, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test eliminate to bool + before = [ + ('LOAD_NAME', 0, 0), + ('TO_BOOL', None, 0), + ('UNARY_NOT', None, 0), + ('TO_BOOL', None, 0), + ('TO_BOOL', None, 0), + ('TO_BOOL', None, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_NAME', 0, 0), + ('TO_BOOL', None, 0), + ('UNARY_NOT', None, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test folding & cancel out + before = [ + ('LOAD_SMALL_INT', 1, 0), + ('TO_BOOL', None, 0), + ('UNARY_NOT', None, 0), + ('UNARY_NOT', None, 0), + ('UNARY_NOT', None, 0), + ('UNARY_NOT', None, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[True]) + + # test folding & eliminate to bool + before = [ + ('LOAD_SMALL_INT', 1, 0), + ('TO_BOOL', None, 0), + ('UNARY_NOT', None, 0), + ('TO_BOOL', None, 0), + ('TO_BOOL', None, 0), + ('TO_BOOL', None, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_CONST', 1, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[True, False]) + + # test cancel out & eliminate to bool (to bool stays as we are not iterating to a fixed point) + before = [ + ('LOAD_NAME', 0, 0), + ('TO_BOOL', None, 0), + ('UNARY_NOT', None, 0), + ('UNARY_NOT', None, 0), + ('UNARY_NOT', None, 0), + ('UNARY_NOT', None, 0), + ('TO_BOOL', None, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_NAME', 0, 0), + ('TO_BOOL', None, 0), + ('TO_BOOL', None, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + is_ = in_ = 0 + isnot = notin = 1 + + # test is/isnot + before = [ + ('LOAD_NAME', 0, 0), + ('LOAD_NAME', 1, 0), + ('IS_OP', is_, 0), + ('UNARY_NOT', None, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_NAME', 0, 0), + ('LOAD_NAME', 1, 0), + ('IS_OP', isnot, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test is/isnot cancel out + before = [ + ('LOAD_NAME', 0, 0), + ('LOAD_NAME', 1, 0), + ('IS_OP', is_, 0), + ('UNARY_NOT', None, 0), + ('UNARY_NOT', None, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_NAME', 0, 0), + ('LOAD_NAME', 1, 0), + ('IS_OP', is_, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test is/isnot eliminate to bool + before = [ + ('LOAD_NAME', 0, 0), + ('LOAD_NAME', 1, 0), + ('IS_OP', is_, 0), + ('UNARY_NOT', None, 0), + ('TO_BOOL', None, 0), + ('TO_BOOL', None, 0), + ('TO_BOOL', None, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_NAME', 0, 0), + ('LOAD_NAME', 1, 0), + ('IS_OP', isnot, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test is/isnot cancel out & eliminate to bool + before = [ + ('LOAD_NAME', 0, 0), + ('LOAD_NAME', 1, 0), + ('IS_OP', is_, 0), + ('UNARY_NOT', None, 0), + ('UNARY_NOT', None, 0), + ('TO_BOOL', None, 0), + ('TO_BOOL', None, 0), + ('TO_BOOL', None, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_NAME', 0, 0), + ('LOAD_NAME', 1, 0), + ('IS_OP', is_, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test in/notin + before = [ + ('LOAD_NAME', 0, 0), + ('LOAD_NAME', 1, 0), + ('CONTAINS_OP', in_, 0), + ('UNARY_NOT', None, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_NAME', 0, 0), + ('LOAD_NAME', 1, 0), + ('CONTAINS_OP', notin, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test in/notin cancel out + before = [ + ('LOAD_NAME', 0, 0), + ('LOAD_NAME', 1, 0), + ('CONTAINS_OP', in_, 0), + ('UNARY_NOT', None, 0), + ('UNARY_NOT', None, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_NAME', 0, 0), + ('LOAD_NAME', 1, 0), + ('CONTAINS_OP', in_, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test is/isnot & eliminate to bool + before = [ + ('LOAD_NAME', 0, 0), + ('LOAD_NAME', 1, 0), + ('CONTAINS_OP', in_, 0), + ('UNARY_NOT', None, 0), + ('TO_BOOL', None, 0), + ('TO_BOOL', None, 0), + ('TO_BOOL', None, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_NAME', 0, 0), + ('LOAD_NAME', 1, 0), + ('CONTAINS_OP', notin, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test in/notin cancel out & eliminate to bool + before = [ + ('LOAD_NAME', 0, 0), + ('LOAD_NAME', 1, 0), + ('CONTAINS_OP', in_, 0), + ('UNARY_NOT', None, 0), + ('UNARY_NOT', None, 0), + ('TO_BOOL', None, 0), + ('RETURN_VALUE', None, 0), + ] + after = [ + ('LOAD_NAME', 0, 0), + ('LOAD_NAME', 1, 0), + ('CONTAINS_OP', in_, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + def test_optimize_if_const_unaryop(self): + # test unary negative + before = [ + ('LOAD_SMALL_INT', 2, 0), + ('UNARY_NEGATIVE', None, 0), + ('UNARY_NEGATIVE', None, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_SMALL_INT', 2, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[-2]) + + # test unary invert + before = [ + ('LOAD_SMALL_INT', 2, 0), + ('UNARY_INVERT', None, 0), + ('UNARY_INVERT', None, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_SMALL_INT', 2, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[-3]) + + # test unary positive + before = [ + ('LOAD_SMALL_INT', 2, 0), + ('CALL_INTRINSIC_1', 5, 0), + ('CALL_INTRINSIC_1', 5, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_SMALL_INT', 2, 0), + ('RETURN_VALUE', None, 0), + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + def test_optimize_if_const_binop(self): + add = get_binop_argval('NB_ADD') + sub = get_binop_argval('NB_SUBTRACT') + mul = get_binop_argval('NB_MULTIPLY') + div = get_binop_argval('NB_TRUE_DIVIDE') + floor = get_binop_argval('NB_FLOOR_DIVIDE') + rem = get_binop_argval('NB_REMAINDER') + pow = get_binop_argval('NB_POWER') + lshift = get_binop_argval('NB_LSHIFT') + rshift = get_binop_argval('NB_RSHIFT') + or_ = get_binop_argval('NB_OR') + and_ = get_binop_argval('NB_AND') + xor = get_binop_argval('NB_XOR') + subscr = get_binop_argval('NB_SUBSCR') + + # test add + before = [ + ('LOAD_SMALL_INT', 2, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', add, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', add, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_SMALL_INT', 6, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test sub + before = [ + ('LOAD_SMALL_INT', 2, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', sub, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', sub, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_CONST', 0, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[-2]) + + # test mul + before = [ + ('LOAD_SMALL_INT', 2, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', mul, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', mul, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_SMALL_INT', 8, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test div + before = [ + ('LOAD_SMALL_INT', 2, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', div, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', div, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_CONST', 1, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[1.0, 0.5]) + + # test floor + before = [ + ('LOAD_SMALL_INT', 2, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', floor, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', floor, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_SMALL_INT', 0, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test rem + before = [ + ('LOAD_SMALL_INT', 2, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', rem, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', rem, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_SMALL_INT', 0, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test pow + before = [ + ('LOAD_SMALL_INT', 2, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', pow, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', pow, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_SMALL_INT', 16, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test lshift + before = [ + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_SMALL_INT', 1, 0), + ('BINARY_OP', lshift, 0), + ('LOAD_SMALL_INT', 1, 0), + ('BINARY_OP', lshift, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_SMALL_INT', 4, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test rshift + before = [ + ('LOAD_SMALL_INT', 4, 0), + ('LOAD_SMALL_INT', 1, 0), + ('BINARY_OP', rshift, 0), + ('LOAD_SMALL_INT', 1, 0), + ('BINARY_OP', rshift, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_SMALL_INT', 1, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test or + before = [ + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', or_, 0), + ('LOAD_SMALL_INT', 4, 0), + ('BINARY_OP', or_, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_SMALL_INT', 7, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test and + before = [ + ('LOAD_SMALL_INT', 1, 0), + ('LOAD_SMALL_INT', 1, 0), + ('BINARY_OP', and_, 0), + ('LOAD_SMALL_INT', 1, 0), + ('BINARY_OP', and_, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_SMALL_INT', 1, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test xor + before = [ + ('LOAD_SMALL_INT', 2, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', xor, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', xor, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_SMALL_INT', 2, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(before, after, consts=[], expected_consts=[]) + + # test subscr + before = [ + ('LOAD_CONST', 0, 0), + ('LOAD_SMALL_INT', 1, 0), + ('BINARY_OP', subscr, 0), + ('LOAD_SMALL_INT', 2, 0), + ('BINARY_OP', subscr, 0), + ('RETURN_VALUE', None, 0) + ] + after = [ + ('LOAD_SMALL_INT', 3, 0), + ('RETURN_VALUE', None, 0) + ] + self.cfg_optimization_test(before, after, consts=[(1, (1, 2, 3))], expected_consts=[(1, (1, 2, 3))]) + + + def test_conditional_jump_forward_const_condition(self): + # The unreachable branch of the jump is removed, the jump + # becomes redundant and is replaced by a NOP (for the lineno) + + insts = [ + ('LOAD_CONST', 3, 11), + ('POP_JUMP_IF_TRUE', lbl := self.Label(), 12), + ('LOAD_CONST', 2, 13), + lbl, + ('LOAD_CONST', 3, 14), + ('RETURN_VALUE', None, 14), + ] + expected_insts = [ + ('NOP', None, 11), + ('NOP', None, 12), + ('LOAD_SMALL_INT', 3, 14), + ('RETURN_VALUE', None, 14), + ] + self.cfg_optimization_test(insts, + expected_insts, + consts=[0, 1, 2, 3, 4], + expected_consts=[0]) + + def test_conditional_jump_backward_non_const_condition(self): + insts = [ + lbl1 := self.Label(), + ('LOAD_NAME', 1, 11), + ('POP_JUMP_IF_TRUE', lbl1, 12), + ('LOAD_NAME', 2, 13), + ('RETURN_VALUE', None, 13), + ] + expected = [ + lbl := self.Label(), + ('LOAD_NAME', 1, 11), + ('POP_JUMP_IF_TRUE', lbl, 12), + ('LOAD_NAME', 2, 13), + ('RETURN_VALUE', None, 13), + ] + self.cfg_optimization_test(insts, expected, consts=list(range(5))) + + def test_conditional_jump_backward_const_condition(self): + # The unreachable branch of the jump is removed + insts = [ + lbl1 := self.Label(), + ('LOAD_CONST', 3, 11), + ('POP_JUMP_IF_TRUE', lbl1, 12), + ('LOAD_CONST', 2, 13), + ('RETURN_VALUE', None, 13), + ] + expected_insts = [ + lbl := self.Label(), + ('NOP', None, 11), + ('JUMP', lbl, 12), + ] + self.cfg_optimization_test(insts, expected_insts, consts=list(range(5))) + + def test_except_handler_label(self): + insts = [ + ('SETUP_FINALLY', handler := self.Label(), 10), + ('POP_BLOCK', None, -1), + ('LOAD_CONST', 1, 11), + ('RETURN_VALUE', None, 11), + handler, + ('LOAD_CONST', 2, 12), + ('RETURN_VALUE', None, 12), + ] + expected_insts = [ + ('SETUP_FINALLY', handler := self.Label(), 10), + ('LOAD_SMALL_INT', 1, 11), + ('RETURN_VALUE', None, 11), + handler, + ('LOAD_SMALL_INT', 2, 12), + ('RETURN_VALUE', None, 12), + ] + self.cfg_optimization_test(insts, expected_insts, consts=list(range(5))) + + def test_no_unsafe_static_swap(self): + # We can't change order of two stores to the same location + insts = [ + ('LOAD_CONST', 0, 1), + ('LOAD_CONST', 1, 2), + ('LOAD_CONST', 2, 3), + ('SWAP', 3, 4), + ('STORE_FAST', 1, 4), + ('STORE_FAST', 1, 4), + ('POP_TOP', None, 4), + ('LOAD_CONST', 0, 5), + ('RETURN_VALUE', None, 5) + ] + expected_insts = [ + ('LOAD_SMALL_INT', 0, 1), + ('LOAD_SMALL_INT', 1, 2), + ('NOP', None, 3), + ('STORE_FAST', 1, 4), + ('POP_TOP', None, 4), + ('LOAD_SMALL_INT', 0, 5), + ('RETURN_VALUE', None, 5) + ] + self.cfg_optimization_test(insts, expected_insts, consts=list(range(3)), nlocals=1) + + def test_dead_store_elimination_in_same_lineno(self): + insts = [ + ('LOAD_CONST', 0, 1), + ('LOAD_CONST', 1, 2), + ('LOAD_CONST', 2, 3), + ('STORE_FAST', 1, 4), + ('STORE_FAST', 1, 4), + ('STORE_FAST', 1, 4), + ('LOAD_CONST', 0, 5), + ('RETURN_VALUE', None, 5) + ] + expected_insts = [ + ('LOAD_SMALL_INT', 0, 1), + ('LOAD_SMALL_INT', 1, 2), + ('NOP', None, 3), + ('POP_TOP', None, 4), + ('STORE_FAST', 1, 4), + ('LOAD_SMALL_INT', 0, 5), + ('RETURN_VALUE', None, 5) + ] + self.cfg_optimization_test(insts, expected_insts, consts=list(range(3)), nlocals=1) + + def test_no_dead_store_elimination_in_different_lineno(self): + insts = [ + ('LOAD_CONST', 0, 1), + ('LOAD_CONST', 1, 2), + ('LOAD_CONST', 2, 3), + ('STORE_FAST', 1, 4), + ('STORE_FAST', 1, 5), + ('STORE_FAST', 1, 6), + ('LOAD_CONST', 0, 5), + ('RETURN_VALUE', None, 5) + ] + expected_insts = [ + ('LOAD_SMALL_INT', 0, 1), + ('LOAD_SMALL_INT', 1, 2), + ('LOAD_SMALL_INT', 2, 3), + ('STORE_FAST', 1, 4), + ('STORE_FAST', 1, 5), + ('STORE_FAST', 1, 6), + ('LOAD_SMALL_INT', 0, 5), + ('RETURN_VALUE', None, 5) + ] + self.cfg_optimization_test(insts, expected_insts, consts=list(range(3)), nlocals=1) + + def test_unconditional_jump_threading(self): + + def get_insts(lno1, lno2, op1, op2): + return [ + lbl2 := self.Label(), + ('LOAD_NAME', 0, 10), + ('POP_TOP', None, 10), + (op1, lbl1 := self.Label(), lno1), + ('LOAD_NAME', 1, 20), + lbl1, + (op2, lbl2, lno2), + ] + + for op1 in ('JUMP', 'JUMP_NO_INTERRUPT'): + for op2 in ('JUMP', 'JUMP_NO_INTERRUPT'): + # different lines + lno1, lno2 = (4, 5) + with self.subTest(lno = (lno1, lno2), ops = (op1, op2)): + insts = get_insts(lno1, lno2, op1, op2) + op = 'JUMP' if 'JUMP' in (op1, op2) else 'JUMP_NO_INTERRUPT' + expected_insts = [ + ('LOAD_NAME', 0, 10), + ('POP_TOP', None, 10), + ('NOP', None, 4), + (op, 0, 5), + ] + self.cfg_optimization_test(insts, expected_insts, consts=list(range(5))) + + # Threading + for lno1, lno2 in [(-1, -1), (-1, 5), (6, -1), (7, 7)]: + with self.subTest(lno = (lno1, lno2), ops = (op1, op2)): + insts = get_insts(lno1, lno2, op1, op2) + lno = lno1 if lno1 != -1 else lno2 + if lno == -1: + lno = 10 # Propagated from the line before + + op = 'JUMP' if 'JUMP' in (op1, op2) else 'JUMP_NO_INTERRUPT' + expected_insts = [ + ('LOAD_NAME', 0, 10), + ('POP_TOP', None, 10), + (op, 0, lno), + ] + self.cfg_optimization_test(insts, expected_insts, consts=list(range(5))) + + def test_list_to_tuple_get_iter(self): + # for _ in (*foo, *bar) -> for _ in [*foo, *bar] + INTRINSIC_LIST_TO_TUPLE = 6 + insts = [ + ("BUILD_LIST", 0, 1), + ("LOAD_FAST", 0, 2), + ("LIST_EXTEND", 1, 3), + ("LOAD_FAST", 1, 4), + ("LIST_EXTEND", 1, 5), + ("CALL_INTRINSIC_1", INTRINSIC_LIST_TO_TUPLE, 6), + ("GET_ITER", None, 7), + top := self.Label(), + ("FOR_ITER", end := self.Label(), 8), + ("STORE_FAST", 2, 9), + ("JUMP", top, 10), + end, + ("END_FOR", None, 11), + ("POP_TOP", None, 12), + ("LOAD_CONST", 0, 13), + ("RETURN_VALUE", None, 14), + ] + expected_insts = [ + ("BUILD_LIST", 0, 1), + ("LOAD_FAST_BORROW", 0, 2), + ("LIST_EXTEND", 1, 3), + ("LOAD_FAST_BORROW", 1, 4), + ("LIST_EXTEND", 1, 5), + ("NOP", None, 6), # ("CALL_INTRINSIC_1", INTRINSIC_LIST_TO_TUPLE, 6), + ("GET_ITER", None, 7), + top := self.Label(), + ("FOR_ITER", end := self.Label(), 8), + ("STORE_FAST", 2, 9), + ("JUMP", top, 10), + end, + ("END_FOR", None, 11), + ("POP_TOP", None, 12), + ("LOAD_CONST", 0, 13), + ("RETURN_VALUE", None, 14), + ] + self.cfg_optimization_test(insts, expected_insts, consts=[None]) + + def test_list_to_tuple_get_iter_is_safe(self): + a, b = [], [] + for item in (*(items := [0, 1, 2, 3]),): + a.append(item) + b.append(items.pop()) + self.assertEqual(a, [0, 1, 2, 3]) + self.assertEqual(b, [3, 2, 1, 0]) + self.assertEqual(items, []) + + +class OptimizeLoadFastTestCase(DirectCfgOptimizerTests): + def make_bb(self, insts): + last_loc = insts[-1][2] + maxconst = 0 + for op, arg, _ in insts: + if op == "LOAD_CONST": + maxconst = max(maxconst, arg) + consts = [None for _ in range(maxconst + 1)] + return insts + [ + ("LOAD_CONST", 0, last_loc + 1), + ("RETURN_VALUE", None, last_loc + 2), + ], consts + + def check(self, insts, expected_insts, consts=None): + insts_bb, insts_consts = self.make_bb(insts) + expected_insts_bb, exp_consts = self.make_bb(expected_insts) + self.cfg_optimization_test(insts_bb, expected_insts_bb, + consts=insts_consts, expected_consts=exp_consts) + + def test_optimized(self): + insts = [ + ("LOAD_FAST", 0, 1), + ("LOAD_FAST", 1, 2), + ("BINARY_OP", 2, 3), + ] + expected = [ + ("LOAD_FAST_BORROW", 0, 1), + ("LOAD_FAST_BORROW", 1, 2), + ("BINARY_OP", 2, 3), + ] + self.check(insts, expected) + + insts = [ + ("LOAD_FAST", 0, 1), + ("LOAD_CONST", 1, 2), + ("SWAP", 2, 3), + ("POP_TOP", None, 4), + ] + expected = [ + ("LOAD_FAST_BORROW", 0, 1), + ("LOAD_CONST", 1, 2), + ("SWAP", 2, 3), + ("POP_TOP", None, 4), + ] + self.check(insts, expected) + + def test_unoptimized_if_unconsumed(self): + insts = [ + ("LOAD_FAST", 0, 1), + ("LOAD_FAST", 1, 2), + ("POP_TOP", None, 3), + ] + expected = [ + ("LOAD_FAST", 0, 1), + ("LOAD_FAST_BORROW", 1, 2), + ("POP_TOP", None, 3), + ] + self.check(insts, expected) + + insts = [ + ("LOAD_FAST", 0, 1), + ("COPY", 1, 2), + ("POP_TOP", None, 3), + ] + expected = [ + ("LOAD_FAST", 0, 1), + ("NOP", None, 2), + ("NOP", None, 3), + ] + self.check(insts, expected) + + def test_unoptimized_if_support_killed(self): + insts = [ + ("LOAD_FAST", 0, 1), + ("LOAD_CONST", 0, 2), + ("STORE_FAST", 0, 3), + ("POP_TOP", None, 4), + ] + self.check(insts, insts) + + insts = [ + ("LOAD_FAST", 0, 1), + ("LOAD_CONST", 0, 2), + ("LOAD_CONST", 0, 3), + ("STORE_FAST_STORE_FAST", ((0 << 4) | 1), 4), + ("POP_TOP", None, 5), + ] + self.check(insts, insts) + + insts = [ + ("LOAD_FAST", 0, 1), + ("DELETE_FAST", 0, 2), + ("POP_TOP", None, 3), + ] + self.check(insts, insts) + + def test_unoptimized_if_aliased(self): + insts = [ + ("LOAD_FAST", 0, 1), + ("STORE_FAST", 1, 2), + ] + self.check(insts, insts) + + insts = [ + ("LOAD_FAST", 0, 1), + ("LOAD_CONST", 0, 3), + ("STORE_FAST_STORE_FAST", ((0 << 4) | 1), 4), + ] + self.check(insts, insts) + + def test_consume_no_inputs(self): + insts = [ + ("LOAD_FAST", 0, 1), + ("GET_LEN", None, 2), + ("STORE_FAST", 1 , 3), + ("STORE_FAST", 2, 4), + ] + self.check(insts, insts) + + def test_consume_some_inputs_no_outputs(self): + insts = [ + ("LOAD_FAST", 0, 1), + ("GET_LEN", None, 2), + ("LIST_APPEND", 0, 3), + ] + self.check(insts, insts) + + def test_check_exc_match(self): + insts = [ + ("LOAD_FAST", 0, 1), + ("LOAD_FAST", 1, 2), + ("CHECK_EXC_MATCH", None, 3) + ] + expected = [ + ("LOAD_FAST", 0, 1), + ("LOAD_FAST_BORROW", 1, 2), + ("CHECK_EXC_MATCH", None, 3) + ] + self.check(insts, expected) + + def test_for_iter(self): + insts = [ + ("LOAD_FAST", 0, 1), + top := self.Label(), + ("FOR_ITER", end := self.Label(), 2), + ("STORE_FAST", 2, 3), + ("JUMP", top, 4), + end, + ("END_FOR", None, 5), + ("POP_TOP", None, 6), + ("LOAD_CONST", 0, 7), + ("RETURN_VALUE", None, 8), + ] + self.cfg_optimization_test(insts, insts, consts=[None]) + + def test_load_attr(self): + insts = [ + ("LOAD_FAST", 0, 1), + ("LOAD_ATTR", 0, 2), + ] + expected = [ + ("LOAD_FAST_BORROW", 0, 1), + ("LOAD_ATTR", 0, 2), + ] + self.check(insts, expected) + + # Method call, leaves self on stack unconsumed + insts = [ + ("LOAD_FAST", 0, 1), + ("LOAD_ATTR", 1, 2), + ] + expected = [ + ("LOAD_FAST", 0, 1), + ("LOAD_ATTR", 1, 2), + ] + self.check(insts, expected) + + def test_super_attr(self): + insts = [ + ("LOAD_FAST", 0, 1), + ("LOAD_FAST", 1, 2), + ("LOAD_FAST", 2, 3), + ("LOAD_SUPER_ATTR", 0, 4), + ] + expected = [ + ("LOAD_FAST_BORROW", 0, 1), + ("LOAD_FAST_BORROW", 1, 2), + ("LOAD_FAST_BORROW", 2, 3), + ("LOAD_SUPER_ATTR", 0, 4), + ] + self.check(insts, expected) + + # Method call, leaves self on stack unconsumed + insts = [ + ("LOAD_FAST", 0, 1), + ("LOAD_FAST", 1, 2), + ("LOAD_FAST", 2, 3), + ("LOAD_SUPER_ATTR", 1, 4), + ] + expected = [ + ("LOAD_FAST_BORROW", 0, 1), + ("LOAD_FAST_BORROW", 1, 2), + ("LOAD_FAST", 2, 3), + ("LOAD_SUPER_ATTR", 1, 4), + ] + self.check(insts, expected) + + def test_send(self): + insts = [ + ("LOAD_FAST", 0, 1), + ("LOAD_FAST", 1, 2), + ("SEND", end := self.Label(), 3), + ("LOAD_CONST", 0, 4), + ("RETURN_VALUE", None, 5), + end, + ("LOAD_CONST", 0, 6), + ("RETURN_VALUE", None, 7) + ] + expected = [ + ("LOAD_FAST", 0, 1), + ("LOAD_FAST_BORROW", 1, 2), + ("SEND", end := self.Label(), 3), + ("LOAD_CONST", 0, 4), + ("RETURN_VALUE", None, 5), + end, + ("LOAD_CONST", 0, 6), + ("RETURN_VALUE", None, 7) + ] + self.cfg_optimization_test(insts, expected, consts=[None]) + + def test_format_simple(self): + # FORMAT_SIMPLE will leave its operand on the stack if it's a unicode + # object. We treat it conservatively and assume that it always leaves + # its operand on the stack. + insts = [ + ("LOAD_FAST", 0, 1), + ("FORMAT_SIMPLE", None, 2), + ("STORE_FAST", 1, 3), + ] + self.check(insts, insts) + + insts = [ + ("LOAD_FAST", 0, 1), + ("FORMAT_SIMPLE", None, 2), + ("POP_TOP", None, 3), + ] + expected = [ + ("LOAD_FAST_BORROW", 0, 1), + ("FORMAT_SIMPLE", None, 2), + ("POP_TOP", None, 3), + ] + self.check(insts, expected) + + def test_set_function_attribute(self): + # SET_FUNCTION_ATTRIBUTE leaves the function on the stack + insts = [ + ("LOAD_CONST", 0, 1), + ("LOAD_FAST", 0, 2), + ("SET_FUNCTION_ATTRIBUTE", 2, 3), + ("STORE_FAST", 1, 4), + ("LOAD_CONST", 0, 5), + ("RETURN_VALUE", None, 6) + ] + self.cfg_optimization_test(insts, insts, consts=[None]) + + insts = [ + ("LOAD_CONST", 0, 1), + ("LOAD_FAST", 0, 2), + ("SET_FUNCTION_ATTRIBUTE", 2, 3), + ("RETURN_VALUE", None, 4) + ] + expected = [ + ("LOAD_CONST", 0, 1), + ("LOAD_FAST_BORROW", 0, 2), + ("SET_FUNCTION_ATTRIBUTE", 2, 3), + ("RETURN_VALUE", None, 4) + ] + self.cfg_optimization_test(insts, expected, consts=[None]) + + def test_get_yield_from_iter(self): + # GET_YIELD_FROM_ITER may leave its operand on the stack + insts = [ + ("LOAD_FAST", 0, 1), + ("GET_YIELD_FROM_ITER", None, 2), + ("LOAD_CONST", 0, 3), + send := self.Label(), + ("SEND", end := self.Label(), 5), + ("YIELD_VALUE", 1, 6), + ("RESUME", 2, 7), + ("JUMP", send, 8), + end, + ("END_SEND", None, 9), + ("LOAD_CONST", 0, 10), + ("RETURN_VALUE", None, 11), + ] + self.cfg_optimization_test(insts, insts, consts=[None]) + + def test_push_exc_info(self): + insts = [ + ("LOAD_FAST", 0, 1), + ("PUSH_EXC_INFO", None, 2), + ] + self.check(insts, insts) + + def test_load_special(self): + # LOAD_SPECIAL may leave self on the stack + insts = [ + ("LOAD_FAST", 0, 1), + ("LOAD_SPECIAL", 0, 2), + ("STORE_FAST", 1, 3), + ] + self.check(insts, insts) + + + def test_del_in_finally(self): + # This loads `obj` onto the stack, executes `del obj`, then returns the + # `obj` from the stack. See gh-133371 for more details. + def create_obj(): + obj = [42] + try: + return obj + finally: + del obj + + obj = create_obj() + # The crash in the linked issue happens while running GC during + # interpreter finalization, so run it here manually. + gc.collect() + self.assertEqual(obj, [42]) + + def test_format_simple_unicode(self): + # Repro from gh-134889 + def f(): + var = f"{1}" + var = f"{var}" + return var + self.assertEqual(f(), "1") + + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_pep646_syntax.py b/Lib/test/test_pep646_syntax.py new file mode 100644 index 00000000000..d9a0aa9a90e --- /dev/null +++ b/Lib/test/test_pep646_syntax.py @@ -0,0 +1,338 @@ +import doctest +import unittest + +doctests = """ + +Setup + + >>> class AClass: + ... def __init__(self): + ... self._setitem_name = None + ... self._setitem_val = None + ... self._delitem_name = None + ... def __setitem__(self, name, val): + ... self._delitem_name = None + ... self._setitem_name = name + ... self._setitem_val = val + ... def __repr__(self): + ... if self._setitem_name is not None: + ... return f"A[{self._setitem_name}]={self._setitem_val}" + ... elif self._delitem_name is not None: + ... return f"delA[{self._delitem_name}]" + ... def __getitem__(self, name): + ... return ParameterisedA(name) + ... def __delitem__(self, name): + ... self._setitem_name = None + ... self._delitem_name = name + ... + >>> class ParameterisedA: + ... def __init__(self, name): + ... self._name = name + ... def __repr__(self): + ... return f"A[{self._name}]" + ... def __iter__(self): + ... for p in self._name: + ... yield p + >>> class B: + ... def __iter__(self): + ... yield StarredB() + ... def __repr__(self): + ... return "B" + >>> class StarredB: + ... def __repr__(self): + ... return "StarredB" + >>> A = AClass() + >>> b = B() + +Slices that are supposed to work, starring our custom B class + + >>> A[*b] + A[(StarredB,)] + >>> A[*b] = 1; A + A[(StarredB,)]=1 + >>> del A[*b]; A + delA[(StarredB,)] + + >>> A[*b, *b] + A[(StarredB, StarredB)] + >>> A[*b, *b] = 1; A + A[(StarredB, StarredB)]=1 + >>> del A[*b, *b]; A + delA[(StarredB, StarredB)] + + >>> A[b, *b] + A[(B, StarredB)] + >>> A[b, *b] = 1; A + A[(B, StarredB)]=1 + >>> del A[b, *b]; A + delA[(B, StarredB)] + + >>> A[*b, b] + A[(StarredB, B)] + >>> A[*b, b] = 1; A + A[(StarredB, B)]=1 + >>> del A[*b, b]; A + delA[(StarredB, B)] + + >>> A[b, b, *b] + A[(B, B, StarredB)] + >>> A[b, b, *b] = 1; A + A[(B, B, StarredB)]=1 + >>> del A[b, b, *b]; A + delA[(B, B, StarredB)] + + >>> A[*b, b, b] + A[(StarredB, B, B)] + >>> A[*b, b, b] = 1; A + A[(StarredB, B, B)]=1 + >>> del A[*b, b, b]; A + delA[(StarredB, B, B)] + + >>> A[b, *b, b] + A[(B, StarredB, B)] + >>> A[b, *b, b] = 1; A + A[(B, StarredB, B)]=1 + >>> del A[b, *b, b]; A + delA[(B, StarredB, B)] + + >>> A[b, b, *b, b] + A[(B, B, StarredB, B)] + >>> A[b, b, *b, b] = 1; A + A[(B, B, StarredB, B)]=1 + >>> del A[b, b, *b, b]; A + delA[(B, B, StarredB, B)] + + >>> A[b, *b, b, b] + A[(B, StarredB, B, B)] + >>> A[b, *b, b, b] = 1; A + A[(B, StarredB, B, B)]=1 + >>> del A[b, *b, b, b]; A + delA[(B, StarredB, B, B)] + + >>> A[A[b, *b, b]] + A[A[(B, StarredB, B)]] + >>> A[A[b, *b, b]] = 1; A + A[A[(B, StarredB, B)]]=1 + >>> del A[A[b, *b, b]]; A + delA[A[(B, StarredB, B)]] + + >>> A[*A[b, *b, b]] + A[(B, StarredB, B)] + >>> A[*A[b, *b, b]] = 1; A + A[(B, StarredB, B)]=1 + >>> del A[*A[b, *b, b]]; A + delA[(B, StarredB, B)] + + >>> A[b, ...] + A[(B, Ellipsis)] + >>> A[b, ...] = 1; A + A[(B, Ellipsis)]=1 + >>> del A[b, ...]; A + delA[(B, Ellipsis)] + + >>> A[*A[b, ...]] + A[(B, Ellipsis)] + >>> A[*A[b, ...]] = 1; A + A[(B, Ellipsis)]=1 + >>> del A[*A[b, ...]]; A + delA[(B, Ellipsis)] + +Slices that are supposed to work, starring a list + + >>> l = [1, 2, 3] + + >>> A[*l] + A[(1, 2, 3)] + >>> A[*l] = 1; A + A[(1, 2, 3)]=1 + >>> del A[*l]; A + delA[(1, 2, 3)] + + >>> A[*l, 4] + A[(1, 2, 3, 4)] + >>> A[*l, 4] = 1; A + A[(1, 2, 3, 4)]=1 + >>> del A[*l, 4]; A + delA[(1, 2, 3, 4)] + + >>> A[0, *l] + A[(0, 1, 2, 3)] + >>> A[0, *l] = 1; A + A[(0, 1, 2, 3)]=1 + >>> del A[0, *l]; A + delA[(0, 1, 2, 3)] + + >>> A[1:2, *l] + A[(slice(1, 2, None), 1, 2, 3)] + >>> A[1:2, *l] = 1; A + A[(slice(1, 2, None), 1, 2, 3)]=1 + >>> del A[1:2, *l]; A + delA[(slice(1, 2, None), 1, 2, 3)] + + >>> repr(A[1:2, *l]) == repr(A[1:2, 1, 2, 3]) + True + +Slices that are supposed to work, starring a tuple + + >>> t = (1, 2, 3) + + >>> A[*t] + A[(1, 2, 3)] + >>> A[*t] = 1; A + A[(1, 2, 3)]=1 + >>> del A[*t]; A + delA[(1, 2, 3)] + + >>> A[*t, 4] + A[(1, 2, 3, 4)] + >>> A[*t, 4] = 1; A + A[(1, 2, 3, 4)]=1 + >>> del A[*t, 4]; A + delA[(1, 2, 3, 4)] + + >>> A[0, *t] + A[(0, 1, 2, 3)] + >>> A[0, *t] = 1; A + A[(0, 1, 2, 3)]=1 + >>> del A[0, *t]; A + delA[(0, 1, 2, 3)] + + >>> A[1:2, *t] + A[(slice(1, 2, None), 1, 2, 3)] + >>> A[1:2, *t] = 1; A + A[(slice(1, 2, None), 1, 2, 3)]=1 + >>> del A[1:2, *t]; A + delA[(slice(1, 2, None), 1, 2, 3)] + + >>> repr(A[1:2, *t]) == repr(A[1:2, 1, 2, 3]) + True + +Starring an expression (rather than a name) in a slice + + >>> def returns_list(): + ... return [1, 2, 3] + + >>> A[returns_list()] + A[[1, 2, 3]] + >>> A[returns_list()] = 1; A + A[[1, 2, 3]]=1 + >>> del A[returns_list()]; A + delA[[1, 2, 3]] + + >>> A[returns_list(), 4] + A[([1, 2, 3], 4)] + >>> A[returns_list(), 4] = 1; A + A[([1, 2, 3], 4)]=1 + >>> del A[returns_list(), 4]; A + delA[([1, 2, 3], 4)] + + >>> A[*returns_list()] + A[(1, 2, 3)] + >>> A[*returns_list()] = 1; A + A[(1, 2, 3)]=1 + >>> del A[*returns_list()]; A + delA[(1, 2, 3)] + + >>> A[*returns_list(), 4] + A[(1, 2, 3, 4)] + >>> A[*returns_list(), 4] = 1; A + A[(1, 2, 3, 4)]=1 + >>> del A[*returns_list(), 4]; A + delA[(1, 2, 3, 4)] + + >>> A[0, *returns_list()] + A[(0, 1, 2, 3)] + >>> A[0, *returns_list()] = 1; A + A[(0, 1, 2, 3)]=1 + >>> del A[0, *returns_list()]; A + delA[(0, 1, 2, 3)] + + >>> A[*returns_list(), *returns_list()] + A[(1, 2, 3, 1, 2, 3)] + >>> A[*returns_list(), *returns_list()] = 1; A + A[(1, 2, 3, 1, 2, 3)]=1 + >>> del A[*returns_list(), *returns_list()]; A + delA[(1, 2, 3, 1, 2, 3)] + +Using both a starred object and a start:stop in a slice +(See also tests in test_syntax confirming that starring *inside* a start:stop +is *not* valid syntax.) + + >>> A[1:2, *b] + A[(slice(1, 2, None), StarredB)] + >>> A[*b, 1:2] + A[(StarredB, slice(1, 2, None))] + >>> A[1:2, *b, 1:2] + A[(slice(1, 2, None), StarredB, slice(1, 2, None))] + >>> A[*b, 1:2, *b] + A[(StarredB, slice(1, 2, None), StarredB)] + + >>> A[1:, *b] + A[(slice(1, None, None), StarredB)] + >>> A[*b, 1:] + A[(StarredB, slice(1, None, None))] + >>> A[1:, *b, 1:] + A[(slice(1, None, None), StarredB, slice(1, None, None))] + >>> A[*b, 1:, *b] + A[(StarredB, slice(1, None, None), StarredB)] + + >>> A[:1, *b] + A[(slice(None, 1, None), StarredB)] + >>> A[*b, :1] + A[(StarredB, slice(None, 1, None))] + >>> A[:1, *b, :1] + A[(slice(None, 1, None), StarredB, slice(None, 1, None))] + >>> A[*b, :1, *b] + A[(StarredB, slice(None, 1, None), StarredB)] + + >>> A[:, *b] + A[(slice(None, None, None), StarredB)] + >>> A[*b, :] + A[(StarredB, slice(None, None, None))] + >>> A[:, *b, :] + A[(slice(None, None, None), StarredB, slice(None, None, None))] + >>> A[*b, :, *b] + A[(StarredB, slice(None, None, None), StarredB)] + +*args annotated as starred expression + + >>> def f1(*args: *b): pass + >>> f1.__annotations__ + {'args': StarredB} + + >>> def f2(*args: *b, arg1): pass + >>> f2.__annotations__ + {'args': StarredB} + + >>> def f3(*args: *b, arg1: int): pass + >>> f3.__annotations__ # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + {'args': StarredB, 'arg1': <class 'int'>} + + >>> def f4(*args: *b, arg1: int = 2): pass + >>> f4.__annotations__ # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + {'args': StarredB, 'arg1': <class 'int'>} + + >>> def f5(*args: *b = (1,)): pass # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE + Traceback (most recent call last): + ... + SyntaxError: invalid syntax +""" + +__test__ = {'doctests' : doctests} + +EXPECTED_FAILURE = doctest.register_optionflag('EXPECTED_FAILURE') # TODO: RUSTPYTHON +class CustomOutputChecker(doctest.OutputChecker): # TODO: RUSTPYTHON + def check_output(self, want, got, optionflags): # TODO: RUSTPYTHON + if optionflags & EXPECTED_FAILURE: # TODO: RUSTPYTHON + if want == got: # TODO: RUSTPYTHON + return False # TODO: RUSTPYTHON + return True # TODO: RUSTPYTHON + return super().check_output(want, got, optionflags) # TODO: RUSTPYTHON + +def load_tests(loader, tests, pattern): + tests.addTest(doctest.DocTestSuite(checker=CustomOutputChecker())) # TODO: RUSTPYTHON + return tests + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_pickle.py b/Lib/test/test_pickle.py index ea51b9d0916..6135cb75d76 100644 --- a/Lib/test/test_pickle.py +++ b/Lib/test/test_pickle.py @@ -97,10 +97,6 @@ def dumps(self, arg, proto=None, **kwargs): def test_picklebuffer_error(self): # TODO(RUSTPYTHON): Remove this test when it passes return super().test_picklebuffer_error() - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_bad_getattr(self): # TODO(RUSTPYTHON): Remove this test when it passes - return super().test_bad_getattr() # TODO: RUSTPYTHON @unittest.expectedFailure @@ -135,61 +131,11 @@ def loads(self, buf, **kwds): def test_c_methods(self): # TODO(RUSTPYTHON): Remove this test when it passes return super().test_c_methods() - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_complex_newobj_ex(self): # TODO(RUSTPYTHON): Remove this test when it passes - return super().test_complex_newobj_ex() - - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_py_methods(self): # TODO(RUSTPYTHON): Remove this test when it passes - return super().test_py_methods() - - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_buffers_error(self): # TODO(RUSTPYTHON): Remove this test when it passes - return super().test_buffers_error() - - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_builtin_functions(self): # TODO(RUSTPYTHON): Remove this test when it passes - return super().test_builtin_functions() - - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_bytearray_memoization(self): # TODO(RUSTPYTHON): Remove this test when it passes - return super().test_bytearray_memoization() - - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_bytes_memoization(self): # TODO(RUSTPYTHON): Remove this test when it passes - return super().test_bytes_memoization() - - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_in_band_buffers(self): # TODO(RUSTPYTHON): Remove this test when it passes - return super().test_in_band_buffers() - - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_oob_buffers(self): # TODO(RUSTPYTHON): Remove this test when it passes - return super().test_oob_buffers() - - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_oob_buffers_writable_to_readonly(self): # TODO(RUSTPYTHON): Remove this test when it passes - return super().test_oob_buffers_writable_to_readonly() - # TODO: RUSTPYTHON @unittest.expectedFailure def test_buffers_error(self): # TODO(RUSTPYTHON): Remove this test when it passes return super().test_buffers_error() - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_builtin_functions(self): # TODO(RUSTPYTHON): Remove this test when it passes - return super().test_builtin_functions() - # TODO: RUSTPYTHON @unittest.expectedFailure def test_bytearray_memoization(self): # TODO(RUSTPYTHON): Remove this test when it passes @@ -215,7 +161,6 @@ def test_oob_buffers(self): # TODO(RUSTPYTHON): Remove this test when it passes def test_oob_buffers_writable_to_readonly(self): # TODO(RUSTPYTHON): Remove this test when it passes return super().test_oob_buffers_writable_to_readonly() - class InMemoryPickleTests(AbstractPickleTests, AbstractUnpickleTests, BigmemPickleTests, unittest.TestCase): @@ -239,11 +184,6 @@ def loads(self, buf, **kwds): def test_c_methods(self): # TODO(RUSTPYTHON): Remove this test when it passes return super().test_c_methods() - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_complex_newobj_ex(self): # TODO(RUSTPYTHON): Remove this test when it passes - return super().test_complex_newobj_ex() - # TODO: RUSTPYTHON @unittest.expectedFailure def test_badly_escaped_string(self): # TODO(RUSTPYTHON): Remove this test when it passes @@ -259,11 +199,6 @@ def test_correctly_quoted_string(self): # TODO(RUSTPYTHON): Remove this test whe def test_load_python2_str_as_bytes(self): # TODO(RUSTPYTHON): Remove this test when it passes return super().test_load_python2_str_as_bytes() - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_py_methods(self): # TODO(RUSTPYTHON): Remove this test when it passes - return super().test_py_methods() - # TODO: RUSTPYTHON @unittest.expectedFailure def test_oob_buffers_writable_to_readonly(self): # TODO(RUSTPYTHON): Remove this test when it passes @@ -274,11 +209,6 @@ def test_oob_buffers_writable_to_readonly(self): # TODO(RUSTPYTHON): Remove this def test_buffers_error(self): # TODO(RUSTPYTHON): Remove this test when it passes return super().test_buffers_error() - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_builtin_functions(self): # TODO(RUSTPYTHON): Remove this test when it passes - return super().test_builtin_functions() - # TODO: RUSTPYTHON @unittest.expectedFailure def test_bytearray_memoization(self): # TODO(RUSTPYTHON): Remove this test when it passes diff --git a/Lib/test/test_pickletools.py b/Lib/test/test_pickletools.py index 492f57cce22..6ec8b348f58 100644 --- a/Lib/test/test_pickletools.py +++ b/Lib/test/test_pickletools.py @@ -67,11 +67,6 @@ def test_optimize_binput_and_memoize(self): def test_buffers_error(self): # TODO(RUSTPYTHON): Remove this test when it passes return super().test_buffers_error() - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_builtin_functions(self): # TODO(RUSTPYTHON): Remove this test when it passes - return super().test_builtin_functions() - # TODO: RUSTPYTHON @unittest.expectedFailure def test_bytearray_memoization(self): # TODO(RUSTPYTHON): Remove this test when it passes @@ -97,16 +92,6 @@ def test_oob_buffers(self): # TODO(RUSTPYTHON): Remove this test when it passes def test_oob_buffers_writable_to_readonly(self): # TODO(RUSTPYTHON): Remove this test when it passes return super().test_oob_buffers_writable_to_readonly() - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_py_methods(self): # TODO(RUSTPYTHON): Remove this test when it passes - return super().test_py_methods() - - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_complex_newobj_ex(self): # TODO(RUSTPYTHON): Remove this test when it passes - return super().test_complex_newobj_ex() - # TODO: RUSTPYTHON @unittest.expectedFailure def test_c_methods(self): # TODO(RUSTPYTHON): Remove this test when it passes diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py index 54302eba4df..3ea5d6d3838 100644 --- a/Lib/test/test_py_compile.py +++ b/Lib/test/test_py_compile.py @@ -109,18 +109,22 @@ def test_cwd(self): self.assertTrue(os.path.exists(self.pyc_path)) self.assertFalse(os.path.exists(self.cache_path)) - import platform - @unittest.expectedFailureIf(sys.platform == "darwin" and int(platform.release().split(".")[0]) < 20, "TODO: RUSTPYTHON") + @unittest.expectedFailureIf( + sys.platform == "darwin" and int( + __import__("platform").release().split(".")[0] + ) < 20, + "TODO: RUSTPYTHON" + ) def test_relative_path(self): py_compile.compile(os.path.relpath(self.source_path), os.path.relpath(self.pyc_path)) self.assertTrue(os.path.exists(self.pyc_path)) self.assertFalse(os.path.exists(self.cache_path)) - @unittest.skipIf(hasattr(os, 'geteuid') and os.geteuid() == 0, - 'non-root user required') + @os_helper.skip_if_dac_override @unittest.skipIf(os.name == 'nt', 'cannot control directory permissions on Windows') + @os_helper.skip_unless_working_chmod def test_exceptions_propagate(self): # Make sure that exceptions raised thanks to issues with writing # bytecode. @@ -133,10 +137,11 @@ def test_exceptions_propagate(self): finally: os.chmod(self.directory, mode.st_mode) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_bad_coding(self): - bad_coding = os.path.join(os.path.dirname(__file__), 'bad_coding2.py') + bad_coding = os.path.join(os.path.dirname(__file__), + 'tokenizedata', + 'bad_coding2.py') with support.captured_stderr(): self.assertIsNone(py_compile.compile(bad_coding, doraise=False)) self.assertFalse(os.path.exists( @@ -198,6 +203,18 @@ def test_invalidation_mode(self): fp.read(), 'test', {}) self.assertEqual(flags, 0b1) + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_quiet(self): + bad_coding = os.path.join(os.path.dirname(__file__), + 'tokenizedata', + 'bad_coding2.py') + with support.captured_stderr() as stderr: + self.assertIsNone(py_compile.compile(bad_coding, doraise=False, quiet=2)) + self.assertIsNone(py_compile.compile(bad_coding, doraise=True, quiet=2)) + self.assertEqual(stderr.getvalue(), '') + with self.assertRaises(py_compile.PyCompileError): + py_compile.compile(bad_coding, doraise=True, quiet=1) + class PyCompileTestsWithSourceEpoch(PyCompileTestsBase, unittest.TestCase, @@ -218,27 +235,31 @@ class PyCompileCLITestCase(unittest.TestCase): def setUp(self): self.directory = tempfile.mkdtemp() self.source_path = os.path.join(self.directory, '_test.py') - self.cache_path = importlib.util.cache_from_source(self.source_path) + self.cache_path = importlib.util.cache_from_source(self.source_path, + optimization='' if __debug__ else 1) with open(self.source_path, 'w') as file: file.write('x = 123\n') def tearDown(self): os_helper.rmtree(self.directory) + @support.requires_subprocess() def pycompilecmd(self, *args, **kwargs): # assert_python_* helpers don't return proc object. We'll just use # subprocess.run() instead of spawn_python() and its friends to test # stdin support of the CLI. + opts = '-m' if __debug__ else '-Om' if args and args[0] == '-' and 'input' in kwargs: - return subprocess.run([sys.executable, '-m', 'py_compile', '-'], + return subprocess.run([sys.executable, opts, 'py_compile', '-'], input=kwargs['input'].encode(), capture_output=True) - return script_helper.assert_python_ok('-m', 'py_compile', *args, **kwargs) + return script_helper.assert_python_ok(opts, 'py_compile', *args, **kwargs) def pycompilecmd_failure(self, *args): return script_helper.assert_python_failure('-m', 'py_compile', *args) def test_stdin(self): + self.assertFalse(os.path.exists(self.cache_path)) result = self.pycompilecmd('-', input=self.source_path) self.assertEqual(result.returncode, 0) self.assertEqual(result.stdout, b'') @@ -253,14 +274,18 @@ def test_with_files(self): self.assertTrue(os.path.exists(self.cache_path)) def test_bad_syntax(self): - bad_syntax = os.path.join(os.path.dirname(__file__), 'badsyntax_3131.py') + bad_syntax = os.path.join(os.path.dirname(__file__), + 'tokenizedata', + 'badsyntax_3131.py') rc, stdout, stderr = self.pycompilecmd_failure(bad_syntax) self.assertEqual(rc, 1) self.assertEqual(stdout, b'') self.assertIn(b'SyntaxError', stderr) def test_bad_syntax_with_quiet(self): - bad_syntax = os.path.join(os.path.dirname(__file__), 'badsyntax_3131.py') + bad_syntax = os.path.join(os.path.dirname(__file__), + 'tokenizedata', + 'badsyntax_3131.py') rc, stdout, stderr = self.pycompilecmd_failure('-q', bad_syntax) self.assertEqual(rc, 1) self.assertEqual(stdout, b'') diff --git a/Lib/test/test_pyclbr.py b/Lib/test/test_pyclbr.py index ad26fe1dba1..9ac4a6efe00 100644 --- a/Lib/test/test_pyclbr.py +++ b/Lib/test/test_pyclbr.py @@ -142,8 +142,6 @@ def defined_in(item, module): if defined_in(item, module): self.assertHaskey(dict, name, ignore) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_easy(self): self.checkModule('pyclbr') # XXX: Metaclasses are not supported @@ -158,8 +156,6 @@ def test_cases(self): # see test.pyclbr_input for the rationale behind the ignored symbols self.checkModule('test.pyclbr_input', ignore=['om', 'f']) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_nested(self): mb = pyclbr # Set arguments for descriptor creation and _creat_tree call. diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 7c5bfba41bf..8b935dc7f12 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -1,12 +1,11 @@ from test.support import (gc_collect, bigmemtest, _2G, cpython_only, captured_stdout, - check_disallow_instantiation, is_emscripten, is_wasi, - warnings_helper, SHORT_TIMEOUT, CPUStopwatch, requires_resource) + check_disallow_instantiation, linked_to_musl, + warnings_helper, SHORT_TIMEOUT, Stopwatch, requires_resource) import locale import re import string import sys -import time import unittest import warnings from re import Scanner @@ -14,7 +13,7 @@ # some platforms lack working multiprocessing try: - import _multiprocessing + import _multiprocessing # noqa: F401 except ImportError: multiprocessing = None else: @@ -621,6 +620,7 @@ def test_re_fullmatch(self): self.assertEqual(re.fullmatch(r"a.*?b", "axxb").span(), (0, 4)) self.assertIsNone(re.fullmatch(r"a+", "ab")) self.assertIsNone(re.fullmatch(r"abc$", "abc\n")) + self.assertIsNone(re.fullmatch(r"abc\z", "abc\n")) self.assertIsNone(re.fullmatch(r"abc\Z", "abc\n")) self.assertIsNone(re.fullmatch(r"(?m)abc$", "abc\n")) self.assertEqual(re.fullmatch(r"ab(?=c)cd", "abcd").span(), (0, 4)) @@ -806,6 +806,8 @@ def test_special_escapes(self): self.assertEqual(re.search(r"\B(b.)\B", "abc bcd bc abxd", re.ASCII).group(1), "bx") self.assertEqual(re.search(r"^abc$", "\nabc\n", re.M).group(0), "abc") + self.assertEqual(re.search(r"^\Aabc\z$", "abc", re.M).group(0), "abc") + self.assertIsNone(re.search(r"^\Aabc\z$", "\nabc\n", re.M)) self.assertEqual(re.search(r"^\Aabc\Z$", "abc", re.M).group(0), "abc") self.assertIsNone(re.search(r"^\Aabc\Z$", "\nabc\n", re.M)) self.assertEqual(re.search(br"\b(b.)\b", @@ -817,6 +819,8 @@ def test_special_escapes(self): self.assertEqual(re.search(br"\B(b.)\B", b"abc bcd bc abxd", re.LOCALE).group(1), b"bx") self.assertEqual(re.search(br"^abc$", b"\nabc\n", re.M).group(0), b"abc") + self.assertEqual(re.search(br"^\Aabc\z$", b"abc", re.M).group(0), b"abc") + self.assertIsNone(re.search(br"^\Aabc\z$", b"\nabc\n", re.M)) self.assertEqual(re.search(br"^\Aabc\Z$", b"abc", re.M).group(0), b"abc") self.assertIsNone(re.search(br"^\Aabc\Z$", b"\nabc\n", re.M)) self.assertEqual(re.search(r"\d\D\w\W\s\S", @@ -840,7 +844,7 @@ def test_other_escapes(self): self.assertEqual(re.match(r"[\^a]+", 'a^').group(), 'a^') self.assertIsNone(re.match(r"[\^a]+", 'b')) re.purge() # for warnings - for c in 'ceghijklmopqyzCEFGHIJKLMNOPQRTVXY': + for c in 'ceghijklmopqyCEFGHIJKLMNOPQRTVXY': with self.subTest(c): self.assertRaises(re.PatternError, re.compile, '\\%c' % c) for c in 'ceghijklmopqyzABCEFGHIJKLMNOPQRTVXYZ': @@ -888,6 +892,8 @@ def test_named_unicode_escapes(self): self.checkPatternError(br'\N{LESS-THAN SIGN}', r'bad escape \N', 0) self.checkPatternError(br'[\N{LESS-THAN SIGN}]', r'bad escape \N', 1) + # TODO: RUSTPYTHON; re.search(r"\B", "") now returns a match in CPython 3.14 + @unittest.expectedFailure def test_word_boundaries(self): # See http://bugs.python.org/issue10713 self.assertEqual(re.search(r"\b(abc)\b", "abc").group(1), "abc") @@ -983,18 +989,15 @@ def test_word_boundaries(self): self.assertIsNone(re.fullmatch(br".+\B", b"abc", re.LOCALE)) self.assertIsNone(re.fullmatch(r".+\B", "ьюя")) self.assertTrue(re.fullmatch(r".+\B", "ьюя", re.ASCII)) - # However, an empty string contains no word boundaries, and also no - # non-boundaries. + # However, an empty string contains no word boundaries. self.assertIsNone(re.search(r"\b", "")) self.assertIsNone(re.search(r"\b", "", re.ASCII)) self.assertIsNone(re.search(br"\b", b"")) self.assertIsNone(re.search(br"\b", b"", re.LOCALE)) - # This one is questionable and different from the perlre behaviour, - # but describes current behavior. - self.assertIsNone(re.search(r"\B", "")) - self.assertIsNone(re.search(r"\B", "", re.ASCII)) - self.assertIsNone(re.search(br"\B", b"")) - self.assertIsNone(re.search(br"\B", b"", re.LOCALE)) + self.assertTrue(re.search(r"\B", "")) + self.assertTrue(re.search(r"\B", "", re.ASCII)) + self.assertTrue(re.search(br"\B", b"")) + self.assertTrue(re.search(br"\B", b"", re.LOCALE)) # A single word-character string has two boundaries, but no # non-boundary gaps. self.assertEqual(len(re.findall(r"\b", "a")), 2) @@ -1423,7 +1426,7 @@ def test_pickling(self): newpat = pickle.loads(pickled) self.assertEqual(newpat, oldpat) # current pickle expects the _compile() reconstructor in re module - from re import _compile + from re import _compile # noqa: F401 @unittest.expectedFailure # TODO: RUSTPYTHON def test_copying(self): @@ -1755,7 +1758,7 @@ def test_bug_6561(self): for x in not_decimal_digits: self.assertIsNone(re.match(r'^\d$', x)) - @unittest.expectedFailure # TODO: RUSTPYTHON a = array.array(typecode)\n ValueError: bad typecode (must be b, B, u, h, H, i, I, l, L, q, Q, f or d) + @unittest.expectedFailure # TODO: RUSTPYTHON; a = array.array(typecode)\n ValueError: bad typecode (must be b, B, u, h, H, i, I, l, L, q, Q, f or d) @warnings_helper.ignore_warnings(category=DeprecationWarning) # gh-80480 array('u') def test_empty_array(self): # SF buf 1647541 @@ -2185,10 +2188,9 @@ def test_bug_20998(self): self.assertEqual(re.fullmatch('[a-c]+', 'ABC', re.I).span(), (0, 3)) @unittest.expectedFailure # TODO: RUSTPYTHON; self.assertTrue(re.match(b'\xc5', b'\xe5', re.L|re.I))\n AssertionError: None is not true - @unittest.skipIf( - is_emscripten or is_wasi, - "musl libc issue on Emscripten/WASI, bpo-46390" - ) + @unittest.skipIf(linked_to_musl(), "musl libc issue, bpo-46390") + @unittest.skipIf(sys.platform.startswith("sunos"), + "test doesn't work on Solaris, gh-91214") def test_locale_caching(self): # Issue #22410 oldlocale = locale.setlocale(locale.LC_CTYPE) @@ -2225,10 +2227,9 @@ def check_en_US_utf8(self): self.assertIsNone(re.match(b'(?Li)\xc5', b'\xe5')) self.assertIsNone(re.match(b'(?Li)\xe5', b'\xc5')) - @unittest.skipIf( - is_emscripten or is_wasi, - "musl libc issue on Emscripten/WASI, bpo-46390" - ) + @unittest.skipIf(linked_to_musl(), "musl libc issue, bpo-46390") + @unittest.skipIf(sys.platform.startswith("sunos"), + "test doesn't work on Solaris, gh-91214") def test_locale_compiled(self): oldlocale = locale.setlocale(locale.LC_CTYPE) self.addCleanup(locale.setlocale, locale.LC_CTYPE, oldlocale) @@ -2487,7 +2488,7 @@ def test_bug_40736(self): @requires_resource('cpu') def test_search_anchor_at_beginning(self): s = 'x'*10**7 - with CPUStopwatch() as stopwatch: + with Stopwatch() as stopwatch: for p in r'\Ay', r'^y': self.assertIsNone(re.search(p, s)) self.assertEqual(re.split(p, s), [s]) @@ -2632,8 +2633,8 @@ def test_findall_atomic_grouping(self): @unittest.expectedFailure # TODO: RUSTPYTHON def test_bug_gh91616(self): - self.assertTrue(re.fullmatch(r'(?s:(?>.*?\.).*)\Z', "a.txt")) # reproducer - self.assertTrue(re.fullmatch(r'(?s:(?=(?P<g0>.*?\.))(?P=g0).*)\Z', "a.txt")) + self.assertTrue(re.fullmatch(r'(?s:(?>.*?\.).*)\z', "a.txt")) # reproducer + self.assertTrue(re.fullmatch(r'(?s:(?=(?P<g0>.*?\.))(?P=g0).*)\z', "a.txt")) def test_bug_gh100061(self): # gh-100061 @@ -2655,7 +2656,7 @@ def test_bug_gh100061(self): self.assertEqual(re.match("(?>(?:ab?c){1,3})", "aca").span(), (0, 2)) self.assertEqual(re.match("(?:ab?c){1,3}+", "aca").span(), (0, 2)) - @unittest.expectedFailure # TODO: RUSTPYTHON; self.assertEqual(re.match('((x)|y|z){3}+', 'xyz').groups(), ('z', 'x'))\n AssertionError: Tuples differ: ('x', 'x') != ('z', 'x') + @unittest.expectedFailure # TODO: RUSTPYTHON; self.assertEqual(re.match('((x)|y|z){3}+', 'xyz').groups(), ('z', 'x'))\n AssertionError: Tuples differ: ('x', 'x') != ('z', 'x') def test_bug_gh101955(self): # Possessive quantifier with nested alternative with capture groups self.assertEqual(re.match('((x)|y|z)*+', 'xyz').groups(), ('z', 'x')) @@ -2893,11 +2894,11 @@ def test_long_pattern(self): pattern = 'Very %spattern' % ('long ' * 1000) r = repr(re.compile(pattern)) self.assertLess(len(r), 300) - self.assertEqual(r[:30], "re.compile('Very long long lon") + self.assertStartsWith(r, "re.compile('Very long long lon") r = repr(re.compile(pattern, re.I)) self.assertLess(len(r), 300) - self.assertEqual(r[:30], "re.compile('Very long long lon") - self.assertEqual(r[-16:], ", re.IGNORECASE)") + self.assertStartsWith(r, "re.compile('Very long long lon") + self.assertEndsWith(r, ", re.IGNORECASE)") def test_flags_repr(self): self.assertEqual(repr(re.I), "re.IGNORECASE") @@ -2977,7 +2978,7 @@ def test_deprecated_modules(self): self.assertEqual(mod.__name__, name) self.assertEqual(mod.__package__, '') for attr in deprecated[name]: - self.assertTrue(hasattr(mod, attr)) + self.assertHasAttr(mod, attr) del sys.modules[name] @cpython_only diff --git a/Lib/test/test_regrtest.py b/Lib/test/test_regrtest.py index d9ae2f35487..ee1d479b884 100644 --- a/Lib/test/test_regrtest.py +++ b/Lib/test/test_regrtest.py @@ -4,29 +4,61 @@ Note: test_regrtest cannot be run twice in parallel. """ +import _colorize import contextlib -import faulthandler +import dataclasses import glob import io +import locale import os.path import platform +import random import re +import shlex +import signal import subprocess import sys import sysconfig import tempfile import textwrap import unittest -from test import libregrtest +import unittest.mock +from xml.etree import ElementTree + from test import support +from test.support import import_helper from test.support import os_helper +from test.libregrtest import cmdline +from test.libregrtest import main +from test.libregrtest import setup from test.libregrtest import utils +from test.libregrtest.filter import get_match_tests, set_match_tests, match_test +from test.libregrtest.result import TestStats +from test.libregrtest.utils import normalize_test_name +if not support.has_subprocess_support: + raise unittest.SkipTest("test module requires subprocess") -Py_DEBUG = hasattr(sys, 'gettotalrefcount') ROOT_DIR = os.path.join(os.path.dirname(__file__), '..', '..') ROOT_DIR = os.path.abspath(os.path.normpath(ROOT_DIR)) LOG_PREFIX = r'[0-9]+:[0-9]+:[0-9]+ (?:load avg: [0-9]+\.[0-9]{2} )?' +RESULT_REGEX = ( + 'passed', + 'failed', + 'skipped', + 'interrupted', + 'env changed', + 'timed out', + 'ran no tests', + 'worker non-zero exit code', +) +RESULT_REGEX = fr'(?:{"|".join(RESULT_REGEX)})' + +EXITCODE_BAD_TEST = 2 +EXITCODE_ENV_CHANGED = 3 +EXITCODE_NO_TESTS_RAN = 4 +EXITCODE_RERUN_FAIL = 5 +EXITCODE_INTERRUPTED = 130 TEST_INTERRUPTED = textwrap.dedent(""" from signal import SIGINT, raise_signal @@ -43,9 +75,13 @@ class ParseArgsTestCase(unittest.TestCase): Test regrtest's argument parsing, function _parse_args(). """ + @staticmethod + def parse_args(args): + return cmdline._parse_args(args) + def checkError(self, args, msg): with support.captured_stderr() as err, self.assertRaises(SystemExit): - libregrtest._parse_args(args) + self.parse_args(args) self.assertIn(msg, err.getvalue()) def test_help(self): @@ -53,94 +89,130 @@ def test_help(self): with self.subTest(opt=opt): with support.captured_stdout() as out, \ self.assertRaises(SystemExit): - libregrtest._parse_args([opt]) + self.parse_args([opt]) self.assertIn('Run Python regression tests.', out.getvalue()) - @unittest.skipUnless(hasattr(faulthandler, 'dump_traceback_later'), - "faulthandler.dump_traceback_later() required") def test_timeout(self): - ns = libregrtest._parse_args(['--timeout', '4.2']) + ns = self.parse_args(['--timeout', '4.2']) self.assertEqual(ns.timeout, 4.2) + + # negative, zero and empty string are treated as "no timeout" + for value in ('-1', '0', ''): + with self.subTest(value=value): + ns = self.parse_args([f'--timeout={value}']) + self.assertEqual(ns.timeout, None) + self.checkError(['--timeout'], 'expected one argument') - self.checkError(['--timeout', 'foo'], 'invalid float value') + self.checkError(['--timeout', 'foo'], 'invalid timeout value:') def test_wait(self): - ns = libregrtest._parse_args(['--wait']) + ns = self.parse_args(['--wait']) self.assertTrue(ns.wait) - def test_worker_args(self): - ns = libregrtest._parse_args(['--worker-args', '[[], {}]']) - self.assertEqual(ns.worker_args, '[[], {}]') - self.checkError(['--worker-args'], 'expected one argument') - def test_start(self): for opt in '-S', '--start': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt, 'foo']) + ns = self.parse_args([opt, 'foo']) self.assertEqual(ns.start, 'foo') self.checkError([opt], 'expected one argument') def test_verbose(self): - ns = libregrtest._parse_args(['-v']) + ns = self.parse_args(['-v']) self.assertEqual(ns.verbose, 1) - ns = libregrtest._parse_args(['-vvv']) + ns = self.parse_args(['-vvv']) self.assertEqual(ns.verbose, 3) - ns = libregrtest._parse_args(['--verbose']) + ns = self.parse_args(['--verbose']) self.assertEqual(ns.verbose, 1) - ns = libregrtest._parse_args(['--verbose'] * 3) + ns = self.parse_args(['--verbose'] * 3) self.assertEqual(ns.verbose, 3) - ns = libregrtest._parse_args([]) + ns = self.parse_args([]) self.assertEqual(ns.verbose, 0) - def test_verbose2(self): - for opt in '-w', '--verbose2': + def test_rerun(self): + for opt in '-w', '--rerun', '--verbose2': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt]) - self.assertTrue(ns.verbose2) + ns = self.parse_args([opt]) + self.assertTrue(ns.rerun) def test_verbose3(self): for opt in '-W', '--verbose3': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt]) + ns = self.parse_args([opt]) self.assertTrue(ns.verbose3) def test_quiet(self): for opt in '-q', '--quiet': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt]) + ns = self.parse_args([opt]) self.assertTrue(ns.quiet) self.assertEqual(ns.verbose, 0) def test_slowest(self): for opt in '-o', '--slowest': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt]) + ns = self.parse_args([opt]) self.assertTrue(ns.print_slow) def test_header(self): - ns = libregrtest._parse_args(['--header']) + ns = self.parse_args(['--header']) self.assertTrue(ns.header) - ns = libregrtest._parse_args(['--verbose']) + ns = self.parse_args(['--verbose']) self.assertTrue(ns.header) def test_randomize(self): - for opt in '-r', '--randomize': + for opt in ('-r', '--randomize'): with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt]) + ns = self.parse_args([opt]) self.assertTrue(ns.randomize) + with os_helper.EnvironmentVarGuard() as env: + # with SOURCE_DATE_EPOCH + env['SOURCE_DATE_EPOCH'] = '1697839080' + ns = self.parse_args(['--randomize']) + regrtest = main.Regrtest(ns) + self.assertFalse(regrtest.randomize) + self.assertIsInstance(regrtest.random_seed, str) + self.assertEqual(regrtest.random_seed, '1697839080') + + # without SOURCE_DATE_EPOCH + del env['SOURCE_DATE_EPOCH'] + ns = self.parse_args(['--randomize']) + regrtest = main.Regrtest(ns) + self.assertTrue(regrtest.randomize) + self.assertIsInstance(regrtest.random_seed, int) + + def test_no_randomize(self): + ns = self.parse_args([]) + self.assertIs(ns.randomize, False) + + ns = self.parse_args(["--randomize"]) + self.assertIs(ns.randomize, True) + + ns = self.parse_args(["--no-randomize"]) + self.assertIs(ns.randomize, False) + + ns = self.parse_args(["--randomize", "--no-randomize"]) + self.assertIs(ns.randomize, False) + + ns = self.parse_args(["--no-randomize", "--randomize"]) + self.assertIs(ns.randomize, False) + def test_randseed(self): - ns = libregrtest._parse_args(['--randseed', '12345']) + ns = self.parse_args(['--randseed', '12345']) self.assertEqual(ns.random_seed, 12345) self.assertTrue(ns.randomize) self.checkError(['--randseed'], 'expected one argument') self.checkError(['--randseed', 'foo'], 'invalid int value') + ns = self.parse_args(['--randseed', '12345', '--no-randomize']) + self.assertEqual(ns.random_seed, 12345) + self.assertFalse(ns.randomize) + def test_fromfile(self): for opt in '-f', '--fromfile': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt, 'foo']) + ns = self.parse_args([opt, 'foo']) self.assertEqual(ns.fromfile, 'foo') self.checkError([opt], 'expected one argument') self.checkError([opt, 'foo', '-s'], "don't go together") @@ -148,46 +220,37 @@ def test_fromfile(self): def test_exclude(self): for opt in '-x', '--exclude': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt]) + ns = self.parse_args([opt]) self.assertTrue(ns.exclude) def test_single(self): for opt in '-s', '--single': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt]) + ns = self.parse_args([opt]) self.assertTrue(ns.single) self.checkError([opt, '-f', 'foo'], "don't go together") - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_ignore(self): - for opt in '-i', '--ignore': + def test_match(self): + for opt in '-m', '--match': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt, 'pattern']) - self.assertEqual(ns.ignore_tests, ['pattern']) + ns = self.parse_args([opt, 'pattern']) + self.assertEqual(ns.match_tests, [('pattern', True)]) self.checkError([opt], 'expected one argument') - self.addCleanup(os_helper.unlink, os_helper.TESTFN) - with open(os_helper.TESTFN, "w") as fp: - print('matchfile1', file=fp) - print('matchfile2', file=fp) - - filename = os.path.abspath(os_helper.TESTFN) - ns = libregrtest._parse_args(['-m', 'match', - '--ignorefile', filename]) - self.assertEqual(ns.ignore_tests, - ['matchfile1', 'matchfile2']) - - def test_match(self): - for opt in '-m', '--match': + for opt in '-i', '--ignore': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt, 'pattern']) - self.assertEqual(ns.match_tests, ['pattern']) + ns = self.parse_args([opt, 'pattern']) + self.assertEqual(ns.match_tests, [('pattern', False)]) self.checkError([opt], 'expected one argument') - ns = libregrtest._parse_args(['-m', 'pattern1', - '-m', 'pattern2']) - self.assertEqual(ns.match_tests, ['pattern1', 'pattern2']) + ns = self.parse_args(['-m', 'pattern1', '-m', 'pattern2']) + self.assertEqual(ns.match_tests, [('pattern1', True), ('pattern2', True)]) + + ns = self.parse_args(['-m', 'pattern1', '-i', 'pattern2']) + self.assertEqual(ns.match_tests, [('pattern1', True), ('pattern2', False)]) + + ns = self.parse_args(['-i', 'pattern1', '-m', 'pattern2']) + self.assertEqual(ns.match_tests, [('pattern1', False), ('pattern2', True)]) self.addCleanup(os_helper.unlink, os_helper.TESTFN) with open(os_helper.TESTFN, "w") as fp: @@ -195,73 +258,76 @@ def test_match(self): print('matchfile2', file=fp) filename = os.path.abspath(os_helper.TESTFN) - ns = libregrtest._parse_args(['-m', 'match', - '--matchfile', filename]) + ns = self.parse_args(['-m', 'match', '--matchfile', filename]) + self.assertEqual(ns.match_tests, + [('match', True), ('matchfile1', True), ('matchfile2', True)]) + + ns = self.parse_args(['-i', 'match', '--ignorefile', filename]) self.assertEqual(ns.match_tests, - ['match', 'matchfile1', 'matchfile2']) + [('match', False), ('matchfile1', False), ('matchfile2', False)]) def test_failfast(self): for opt in '-G', '--failfast': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt, '-v']) + ns = self.parse_args([opt, '-v']) self.assertTrue(ns.failfast) - ns = libregrtest._parse_args([opt, '-W']) + ns = self.parse_args([opt, '-W']) self.assertTrue(ns.failfast) self.checkError([opt], '-G/--failfast needs either -v or -W') def test_use(self): for opt in '-u', '--use': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt, 'gui,network']) + ns = self.parse_args([opt, 'gui,network']) self.assertEqual(ns.use_resources, ['gui', 'network']) - ns = libregrtest._parse_args([opt, 'gui,none,network']) + ns = self.parse_args([opt, 'gui,none,network']) self.assertEqual(ns.use_resources, ['network']) - expected = list(libregrtest.ALL_RESOURCES) + expected = list(cmdline.ALL_RESOURCES) expected.remove('gui') - ns = libregrtest._parse_args([opt, 'all,-gui']) + ns = self.parse_args([opt, 'all,-gui']) self.assertEqual(ns.use_resources, expected) self.checkError([opt], 'expected one argument') self.checkError([opt, 'foo'], 'invalid resource') # all + a resource not part of "all" - ns = libregrtest._parse_args([opt, 'all,tzdata']) + ns = self.parse_args([opt, 'all,tzdata']) self.assertEqual(ns.use_resources, - list(libregrtest.ALL_RESOURCES) + ['tzdata']) + list(cmdline.ALL_RESOURCES) + ['tzdata']) # test another resource which is not part of "all" - ns = libregrtest._parse_args([opt, 'extralargefile']) + ns = self.parse_args([opt, 'extralargefile']) self.assertEqual(ns.use_resources, ['extralargefile']) def test_memlimit(self): for opt in '-M', '--memlimit': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt, '4G']) + ns = self.parse_args([opt, '4G']) self.assertEqual(ns.memlimit, '4G') self.checkError([opt], 'expected one argument') def test_testdir(self): - ns = libregrtest._parse_args(['--testdir', 'foo']) + ns = self.parse_args(['--testdir', 'foo']) self.assertEqual(ns.testdir, os.path.join(os_helper.SAVEDCWD, 'foo')) self.checkError(['--testdir'], 'expected one argument') def test_runleaks(self): for opt in '-L', '--runleaks': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt]) + ns = self.parse_args([opt]) self.assertTrue(ns.runleaks) def test_huntrleaks(self): for opt in '-R', '--huntrleaks': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt, ':']) + ns = self.parse_args([opt, ':']) self.assertEqual(ns.huntrleaks, (5, 4, 'reflog.txt')) - ns = libregrtest._parse_args([opt, '6:']) + ns = self.parse_args([opt, '6:']) self.assertEqual(ns.huntrleaks, (6, 4, 'reflog.txt')) - ns = libregrtest._parse_args([opt, ':3']) + ns = self.parse_args([opt, ':3']) self.assertEqual(ns.huntrleaks, (5, 3, 'reflog.txt')) - ns = libregrtest._parse_args([opt, '6:3:leaks.log']) + ns = self.parse_args([opt, '6:3:leaks.log']) self.assertEqual(ns.huntrleaks, (6, 3, 'leaks.log')) self.checkError([opt], 'expected one argument') self.checkError([opt, '6'], @@ -272,23 +338,33 @@ def test_huntrleaks(self): def test_multiprocess(self): for opt in '-j', '--multiprocess': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt, '2']) + ns = self.parse_args([opt, '2']) self.assertEqual(ns.use_mp, 2) self.checkError([opt], 'expected one argument') self.checkError([opt, 'foo'], 'invalid int value') - self.checkError([opt, '2', '-T'], "don't go together") - self.checkError([opt, '0', '-T'], "don't go together") - def test_coverage(self): + def test_coverage_sequential(self): + for opt in '-T', '--coverage': + with self.subTest(opt=opt): + with support.captured_stderr() as stderr: + ns = self.parse_args([opt]) + self.assertTrue(ns.trace) + self.assertIn( + "collecting coverage without -j is imprecise", + stderr.getvalue(), + ) + + @unittest.skipUnless(support.Py_DEBUG, 'need a debug build') + def test_coverage_mp(self): for opt in '-T', '--coverage': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt]) + ns = self.parse_args([opt, '-j1']) self.assertTrue(ns.trace) def test_coverdir(self): for opt in '-D', '--coverdir': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt, 'foo']) + ns = self.parse_args([opt, 'foo']) self.assertEqual(ns.coverdir, os.path.join(os_helper.SAVEDCWD, 'foo')) self.checkError([opt], 'expected one argument') @@ -296,13 +372,13 @@ def test_coverdir(self): def test_nocoverdir(self): for opt in '-N', '--nocoverdir': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt]) + ns = self.parse_args([opt]) self.assertIsNone(ns.coverdir) def test_threshold(self): for opt in '-t', '--threshold': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt, '1000']) + ns = self.parse_args([opt, '1000']) self.assertEqual(ns.threshold, 1000) self.checkError([opt], 'expected one argument') self.checkError([opt, 'foo'], 'invalid int value') @@ -311,7 +387,7 @@ def test_nowindows(self): for opt in '-n', '--nowindows': with self.subTest(opt=opt): with contextlib.redirect_stderr(io.StringIO()) as stderr: - ns = libregrtest._parse_args([opt]) + ns = self.parse_args([opt]) self.assertTrue(ns.nowindows) err = stderr.getvalue() self.assertIn('the --nowindows (-n) option is deprecated', err) @@ -319,39 +395,39 @@ def test_nowindows(self): def test_forever(self): for opt in '-F', '--forever': with self.subTest(opt=opt): - ns = libregrtest._parse_args([opt]) + ns = self.parse_args([opt]) self.assertTrue(ns.forever) def test_unrecognized_argument(self): self.checkError(['--xxx'], 'usage:') def test_long_option__partial(self): - ns = libregrtest._parse_args(['--qui']) + ns = self.parse_args(['--qui']) self.assertTrue(ns.quiet) self.assertEqual(ns.verbose, 0) def test_two_options(self): - ns = libregrtest._parse_args(['--quiet', '--exclude']) + ns = self.parse_args(['--quiet', '--exclude']) self.assertTrue(ns.quiet) self.assertEqual(ns.verbose, 0) self.assertTrue(ns.exclude) def test_option_with_empty_string_value(self): - ns = libregrtest._parse_args(['--start', '']) + ns = self.parse_args(['--start', '']) self.assertEqual(ns.start, '') def test_arg(self): - ns = libregrtest._parse_args(['foo']) + ns = self.parse_args(['foo']) self.assertEqual(ns.args, ['foo']) def test_option_and_arg(self): - ns = libregrtest._parse_args(['--quiet', 'foo']) + ns = self.parse_args(['--quiet', 'foo']) self.assertTrue(ns.quiet) self.assertEqual(ns.verbose, 0) self.assertEqual(ns.args, ['foo']) def test_arg_option_arg(self): - ns = libregrtest._parse_args(['test_unaryop', '-v', 'test_binop']) + ns = self.parse_args(['test_unaryop', '-v', 'test_binop']) self.assertEqual(ns.verbose, 1) self.assertEqual(ns.args, ['test_unaryop', 'test_binop']) @@ -359,6 +435,118 @@ def test_unknown_option(self): self.checkError(['--unknown-option'], 'unrecognized arguments: --unknown-option') + def create_regrtest(self, args): + ns = cmdline._parse_args(args) + + # Check Regrtest attributes which are more reliable than Namespace + # which has an unclear API + with os_helper.EnvironmentVarGuard() as env: + # Ignore SOURCE_DATE_EPOCH env var if it's set + del env['SOURCE_DATE_EPOCH'] + + regrtest = main.Regrtest(ns) + + return regrtest + + def check_ci_mode(self, args, use_resources, + *, rerun=True, randomize=True, output_on_failure=True): + regrtest = self.create_regrtest(args) + self.assertEqual(regrtest.num_workers, -1) + self.assertEqual(regrtest.want_rerun, rerun) + self.assertEqual(regrtest.fail_rerun, False) + self.assertEqual(regrtest.randomize, randomize) + self.assertIsInstance(regrtest.random_seed, int) + self.assertTrue(regrtest.fail_env_changed) + self.assertTrue(regrtest.print_slowest) + self.assertEqual(regrtest.output_on_failure, output_on_failure) + self.assertEqual(sorted(regrtest.use_resources), sorted(use_resources)) + return regrtest + + def test_fast_ci(self): + args = ['--fast-ci'] + use_resources = sorted(cmdline.ALL_RESOURCES) + use_resources.remove('cpu') + regrtest = self.check_ci_mode(args, use_resources) + self.assertEqual(regrtest.timeout, 10 * 60) + + def test_fast_ci_python_cmd(self): + args = ['--fast-ci', '--python', 'python -X dev'] + use_resources = sorted(cmdline.ALL_RESOURCES) + use_resources.remove('cpu') + regrtest = self.check_ci_mode(args, use_resources, rerun=False) + self.assertEqual(regrtest.timeout, 10 * 60) + self.assertEqual(regrtest.python_cmd, ('python', '-X', 'dev')) + + def test_fast_ci_resource(self): + # it should be possible to override resources individually + args = ['--fast-ci', '-u-network'] + use_resources = sorted(cmdline.ALL_RESOURCES) + use_resources.remove('cpu') + use_resources.remove('network') + self.check_ci_mode(args, use_resources) + + def test_fast_ci_verbose(self): + args = ['--fast-ci', '--verbose'] + use_resources = sorted(cmdline.ALL_RESOURCES) + use_resources.remove('cpu') + regrtest = self.check_ci_mode(args, use_resources, + output_on_failure=False) + self.assertEqual(regrtest.verbose, True) + + def test_slow_ci(self): + args = ['--slow-ci'] + use_resources = sorted(cmdline.ALL_RESOURCES) + regrtest = self.check_ci_mode(args, use_resources) + self.assertEqual(regrtest.timeout, 20 * 60) + + def test_ci_no_randomize(self): + all_resources = set(cmdline.ALL_RESOURCES) + self.check_ci_mode( + ["--slow-ci", "--no-randomize"], all_resources, randomize=False + ) + self.check_ci_mode( + ["--fast-ci", "--no-randomize"], all_resources - {'cpu'}, randomize=False + ) + + def test_dont_add_python_opts(self): + args = ['--dont-add-python-opts'] + ns = cmdline._parse_args(args) + self.assertFalse(ns._add_python_opts) + + def test_bisect(self): + args = ['--bisect'] + regrtest = self.create_regrtest(args) + self.assertTrue(regrtest.want_bisect) + + def test_verbose3_huntrleaks(self): + args = ['-R', '3:10', '--verbose3'] + with support.captured_stderr(): + regrtest = self.create_regrtest(args) + self.assertIsNotNone(regrtest.hunt_refleak) + self.assertEqual(regrtest.hunt_refleak.warmups, 3) + self.assertEqual(regrtest.hunt_refleak.runs, 10) + self.assertFalse(regrtest.output_on_failure) + + def test_single_process(self): + args = ['-j2', '--single-process'] + with support.captured_stderr(): + regrtest = self.create_regrtest(args) + self.assertEqual(regrtest.num_workers, 0) + self.assertTrue(regrtest.single_process) + + args = ['--fast-ci', '--single-process'] + with support.captured_stderr(): + regrtest = self.create_regrtest(args) + self.assertEqual(regrtest.num_workers, 0) + self.assertTrue(regrtest.single_process) + + +@dataclasses.dataclass(slots=True) +class Rerun: + name: str + match: str | None + success: bool + class BaseTestCase(unittest.TestCase): TEST_UNIQUE_ID = 1 @@ -407,41 +595,61 @@ def regex_search(self, regex, output): self.fail("%r not found in %r" % (regex, output)) return match - def check_line(self, output, regex): - regex = re.compile(r'^' + regex, re.MULTILINE) + def check_line(self, output, pattern, full=False, regex=True): + if not regex: + pattern = re.escape(pattern) + if full: + pattern += '\n' + regex = re.compile(r'^' + pattern, re.MULTILINE) self.assertRegex(output, regex) def parse_executed_tests(self, output): - regex = (r'^%s\[ *[0-9]+(?:/ *[0-9]+)*\] (%s)' - % (LOG_PREFIX, self.TESTNAME_REGEX)) + regex = (fr'^{LOG_PREFIX}\[ *[0-9]+(?:/ *[0-9]+)*\] ' + fr'({self.TESTNAME_REGEX}) {RESULT_REGEX}') parser = re.finditer(regex, output, re.MULTILINE) return list(match.group(1) for match in parser) - def check_executed_tests(self, output, tests, skipped=(), failed=(), + def check_executed_tests(self, output, tests, *, stats, + skipped=(), failed=(), env_changed=(), omitted=(), - rerun=(), no_test_ran=(), - randomize=False, interrupted=False, - fail_env_changed=False): + rerun=None, run_no_tests=(), + resource_denied=(), + randomize=False, parallel=False, interrupted=False, + fail_env_changed=False, + forever=False, filtered=False): if isinstance(tests, str): tests = [tests] if isinstance(skipped, str): skipped = [skipped] + if isinstance(resource_denied, str): + resource_denied = [resource_denied] if isinstance(failed, str): failed = [failed] if isinstance(env_changed, str): env_changed = [env_changed] if isinstance(omitted, str): omitted = [omitted] - if isinstance(rerun, str): - rerun = [rerun] - if isinstance(no_test_ran, str): - no_test_ran = [no_test_ran] + if isinstance(run_no_tests, str): + run_no_tests = [run_no_tests] + if isinstance(stats, int): + stats = TestStats(stats) + if parallel: + randomize = True + + rerun_failed = [] + if rerun is not None and not env_changed: + failed = [rerun.name] + if not rerun.success: + rerun_failed.append(rerun.name) executed = self.parse_executed_tests(output) + total_tests = list(tests) + if rerun is not None: + total_tests.append(rerun.name) if randomize: - self.assertEqual(set(executed), set(tests), output) + self.assertEqual(set(executed), set(total_tests), output) else: - self.assertEqual(executed, tests, output) + self.assertEqual(executed, total_tests, output) def plural(count): return 's' if count != 1 else '' @@ -457,12 +665,17 @@ def list_regex(line_format, tests): regex = list_regex('%s test%s skipped', skipped) self.check_line(output, regex) + if resource_denied: + regex = list_regex(r'%s test%s skipped \(resource denied\)', resource_denied) + self.check_line(output, regex) + if failed: regex = list_regex('%s test%s failed', failed) self.check_line(output, regex) if env_changed: - regex = list_regex('%s test%s altered the execution environment', + regex = list_regex(r'%s test%s altered the execution environment ' + r'\(env changed\)', env_changed) self.check_line(output, regex) @@ -470,73 +683,120 @@ def list_regex(line_format, tests): regex = list_regex('%s test%s omitted', omitted) self.check_line(output, regex) - if rerun: - regex = list_regex('%s re-run test%s', rerun) + if rerun is not None: + regex = list_regex('%s re-run test%s', [rerun.name]) self.check_line(output, regex) - regex = LOG_PREFIX + r"Re-running failed tests in verbose mode" + regex = LOG_PREFIX + r"Re-running 1 failed tests in verbose mode" + self.check_line(output, regex) + regex = fr"Re-running {rerun.name} in verbose mode" + if rerun.match: + regex = fr"{regex} \(matching: {rerun.match}\)" self.check_line(output, regex) - for test_name in rerun: - regex = LOG_PREFIX + f"Re-running {test_name} in verbose mode" - self.check_line(output, regex) - if no_test_ran: - regex = list_regex('%s test%s run no tests', no_test_ran) + if run_no_tests: + regex = list_regex('%s test%s run no tests', run_no_tests) self.check_line(output, regex) - good = (len(tests) - len(skipped) - len(failed) - - len(omitted) - len(env_changed) - len(no_test_ran)) + good = (len(tests) - len(skipped) - len(resource_denied) - len(failed) + - len(omitted) - len(env_changed) - len(run_no_tests)) if good: - regex = r'%s test%s OK\.$' % (good, plural(good)) - if not skipped and not failed and good > 1: + regex = r'%s test%s OK\.' % (good, plural(good)) + if not skipped and not failed and (rerun is None or rerun.success) and good > 1: regex = 'All %s' % regex - self.check_line(output, regex) + self.check_line(output, regex, full=True) if interrupted: self.check_line(output, 'Test suite interrupted by signal SIGINT.') - result = [] + # Total tests + text = f'run={stats.tests_run:,}' + if filtered: + text = fr'{text} \(filtered\)' + parts = [text] + if stats.failures: + parts.append(f'failures={stats.failures:,}') + if stats.skipped: + parts.append(f'skipped={stats.skipped:,}') + line = fr'Total tests: {" ".join(parts)}' + self.check_line(output, line, full=True) + + # Total test files + run = len(total_tests) - len(resource_denied) + if rerun is not None: + total_failed = len(rerun_failed) + total_rerun = 1 + else: + total_failed = len(failed) + total_rerun = 0 + if interrupted: + run = 0 + text = f'run={run}' + if not forever: + text = f'{text}/{len(tests)}' + if filtered: + text = fr'{text} \(filtered\)' + report = [text] + for name, ntest in ( + ('failed', total_failed), + ('env_changed', len(env_changed)), + ('skipped', len(skipped)), + ('resource_denied', len(resource_denied)), + ('rerun', total_rerun), + ('run_no_tests', len(run_no_tests)), + ): + if ntest: + report.append(f'{name}={ntest}') + line = fr'Total test files: {" ".join(report)}' + self.check_line(output, line, full=True) + + # Result + state = [] if failed: - result.append('FAILURE') + state.append('FAILURE') elif fail_env_changed and env_changed: - result.append('ENV CHANGED') + state.append('ENV CHANGED') if interrupted: - result.append('INTERRUPTED') - if not any((good, result, failed, interrupted, skipped, + state.append('INTERRUPTED') + if not any((good, failed, interrupted, skipped, env_changed, fail_env_changed)): - result.append("NO TEST RUN") - elif not result: - result.append('SUCCESS') - result = ', '.join(result) - if rerun: - self.check_line(output, 'Tests result: FAILURE') - result = 'FAILURE then %s' % result - - self.check_line(output, 'Tests result: %s' % result) - - def parse_random_seed(self, output): - match = self.regex_search(r'Using random seed ([0-9]+)', output) - randseed = int(match.group(1)) - self.assertTrue(0 <= randseed <= 10000000, randseed) - return randseed + state.append("NO TESTS RAN") + elif not state: + state.append('SUCCESS') + state = ', '.join(state) + if rerun is not None: + new_state = 'SUCCESS' if rerun.success else 'FAILURE' + state = f'{state} then {new_state}' + self.check_line(output, f'Result: {state}', full=True) + + def parse_random_seed(self, output: str) -> str: + match = self.regex_search(r'Using random seed: (.*)', output) + return match.group(1) def run_command(self, args, input=None, exitcode=0, **kw): if not input: input = '' if 'stderr' not in kw: - kw['stderr'] = subprocess.PIPE + kw['stderr'] = subprocess.STDOUT + + env = kw.pop('env', None) + if env is None: + env = dict(os.environ) + env.pop('SOURCE_DATE_EPOCH', None) + proc = subprocess.run(args, - universal_newlines=True, + text=True, input=input, stdout=subprocess.PIPE, + env=env, **kw) if proc.returncode != exitcode: - msg = ("Command %s failed with exit code %s\n" + msg = ("Command %s failed with exit code %s, but exit code %s expected!\n" "\n" "stdout:\n" "---\n" "%s\n" "---\n" - % (str(args), proc.returncode, proc.stdout)) + % (str(args), proc.returncode, exitcode, proc.stdout)) if proc.stderr: msg += ("\n" "stderr:\n" @@ -547,18 +807,24 @@ def run_command(self, args, input=None, exitcode=0, **kw): self.fail(msg) return proc - def run_python(self, args, **kw): - args = [sys.executable, '-X', 'faulthandler', '-I', *args] - proc = self.run_command(args, **kw) + def run_python(self, args, isolated=True, **kw): + extraargs = [] + if 'uops' in sys._xoptions: + # Pass -X uops along + extraargs.extend(['-X', 'uops']) + cmd = [sys.executable, *extraargs, '-X', 'faulthandler'] + if isolated: + cmd.append('-I') + cmd.extend(args) + proc = self.run_command(cmd, **kw) return proc.stdout class CheckActualTests(BaseTestCase): - """ - Check that regrtest appears to find the expected set of tests. - """ - def test_finds_expected_number_of_tests(self): + """ + Check that regrtest appears to find the expected set of tests. + """ args = ['-Wd', '-E', '-bb', '-m', 'test.regrtest', '--list-tests'] output = self.run_python(args) rough_number_of_tests_found = len(output.splitlines()) @@ -578,6 +844,7 @@ def test_finds_expected_number_of_tests(self): f'{", ".join(output.splitlines())}') +@support.force_not_colorized_test_class class ProgramsTestCase(BaseTestCase): """ Test various ways to run the Python test suite. Use options close @@ -595,17 +862,19 @@ def setUp(self): self.python_args = ['-Wd', '-E', '-bb'] self.regrtest_args = ['-uall', '-rwW', '--testdir=%s' % self.tmptestdir] - if hasattr(faulthandler, 'dump_traceback_later'): - self.regrtest_args.extend(('--timeout', '3600', '-j4')) + self.regrtest_args.extend(('--timeout', '3600', '-j4')) if sys.platform == 'win32': self.regrtest_args.append('-n') def check_output(self, output): - self.parse_random_seed(output) - self.check_executed_tests(output, self.tests, randomize=True) + randseed = self.parse_random_seed(output) + self.assertTrue(randseed.isdigit(), randseed) - def run_tests(self, args): - output = self.run_python(args) + self.check_executed_tests(output, self.tests, + randomize=True, stats=len(self.tests)) + + def run_tests(self, args, env=None, isolated=True): + output = self.run_python(args, env=env, isolated=isolated) self.check_output(output) def test_script_regrtest(self): @@ -615,6 +884,7 @@ def test_script_regrtest(self): args = [*self.python_args, script, *self.regrtest_args, *self.tests] self.run_tests(args) + @unittest.skip("TODO: RUSTPYTHON; flaky") def test_module_test(self): # -m test args = [*self.python_args, '-m', 'test', @@ -627,16 +897,14 @@ def test_module_regrtest(self): *self.regrtest_args, *self.tests] self.run_tests(args) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.skip("TODO: RUSTPYTHON; flaky") def test_module_autotest(self): # -m test.autotest args = [*self.python_args, '-m', 'test.autotest', *self.regrtest_args, *self.tests] self.run_tests(args) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.skip("TODO: RUSTPYTHON; flaky") def test_module_from_test_autotest(self): # from test import autotest code = 'from test import autotest' @@ -644,24 +912,18 @@ def test_module_from_test_autotest(self): *self.regrtest_args, *self.tests] self.run_tests(args) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.skip("TODO: RUSTPYTHON; flaky") def test_script_autotest(self): # Lib/test/autotest.py script = os.path.join(self.testdir, 'autotest.py') args = [*self.python_args, script, *self.regrtest_args, *self.tests] self.run_tests(args) - @unittest.skipUnless(sysconfig.is_python_build(), - 'run_tests.py script is not installed') - def test_tools_script_run_tests(self): - # Tools/scripts/run_tests.py - script = os.path.join(ROOT_DIR, 'Tools', 'scripts', 'run_tests.py') - args = [script, *self.regrtest_args, *self.tests] - self.run_tests(args) - def run_batch(self, *args): - proc = self.run_command(args) + proc = self.run_command(args, + # gh-133711: cmd.exe uses the OEM code page + # to display the non-ASCII current directory + errors="backslashreplace") self.check_output(proc.stdout) @unittest.skipUnless(sysconfig.is_python_build(), @@ -673,10 +935,14 @@ def test_tools_buildbot_test(self): test_args = ['--testdir=%s' % self.tmptestdir] if platform.machine() == 'ARM64': test_args.append('-arm64') # ARM 64-bit build + elif platform.machine() == 'ARM': + test_args.append('-arm32') # 32-bit ARM build elif platform.architecture()[0] == '64bit': test_args.append('-x64') # 64-bit build - if not Py_DEBUG: + if not support.Py_DEBUG: test_args.append('+d') # Release build, use python.exe + if sysconfig.get_config_var("Py_GIL_DISABLED"): + test_args.append('--disable-gil') self.run_batch(script, *test_args, *self.tests) @unittest.skipUnless(sys.platform == 'win32', 'Windows only') @@ -688,13 +954,18 @@ def test_pcbuild_rt(self): rt_args = ["-q"] # Quick, don't run tests twice if platform.machine() == 'ARM64': rt_args.append('-arm64') # ARM 64-bit build + elif platform.machine() == 'ARM': + rt_args.append('-arm32') # 32-bit ARM build elif platform.architecture()[0] == '64bit': rt_args.append('-x64') # 64-bit build - if Py_DEBUG: + if support.Py_DEBUG: rt_args.append('-d') # Debug build, use python_d.exe + if sysconfig.get_config_var("Py_GIL_DISABLED"): + rt_args.append('--disable-gil') self.run_batch(script, *rt_args, *self.regrtest_args, *self.tests) +@support.force_not_colorized_test_class class ArgsTestCase(BaseTestCase): """ Test arguments of the Python test suite. @@ -704,6 +975,40 @@ def run_tests(self, *testargs, **kw): cmdargs = ['-m', 'test', '--testdir=%s' % self.tmptestdir, *testargs] return self.run_python(cmdargs, **kw) + def test_success(self): + code = textwrap.dedent(""" + import unittest + + class PassingTests(unittest.TestCase): + def test_test1(self): + pass + + def test_test2(self): + pass + + def test_test3(self): + pass + """) + tests = [self.create_test(f'ok{i}', code=code) for i in range(1, 6)] + + output = self.run_tests(*tests) + self.check_executed_tests(output, tests, + stats=3 * len(tests)) + + def test_skip(self): + code = textwrap.dedent(""" + import unittest + raise unittest.SkipTest("nope") + """) + test_ok = self.create_test('ok') + test_skip = self.create_test('skip', code=code) + tests = [test_ok, test_skip] + + output = self.run_tests(*tests) + self.check_executed_tests(output, tests, + skipped=[test_skip], + stats=1) + def test_failing_test(self): # test a failing test code = textwrap.dedent(""" @@ -717,8 +1022,9 @@ def test_failing(self): test_failing = self.create_test('failing', code=code) tests = [test_ok, test_failing] - output = self.run_tests(*tests, exitcode=2) - self.check_executed_tests(output, tests, failed=test_failing) + output = self.run_tests(*tests, exitcode=EXITCODE_BAD_TEST) + self.check_executed_tests(output, tests, failed=test_failing, + stats=TestStats(2, 1)) def test_resources(self): # test -u command line option @@ -737,17 +1043,19 @@ def test_pass(self): # -u all: 2 resources enabled output = self.run_tests('-u', 'all', *test_names) - self.check_executed_tests(output, test_names) + self.check_executed_tests(output, test_names, stats=2) # -u audio: 1 resource enabled output = self.run_tests('-uaudio', *test_names) self.check_executed_tests(output, test_names, - skipped=tests['network']) + resource_denied=tests['network'], + stats=1) # no option: 0 resources enabled - output = self.run_tests(*test_names) + output = self.run_tests(*test_names, exitcode=EXITCODE_NO_TESTS_RAN) self.check_executed_tests(output, test_names, - skipped=test_names) + resource_denied=test_names, + stats=0) def test_random(self): # test -r and --randseed command line option @@ -758,13 +1066,14 @@ def test_random(self): test = self.create_test('random', code) # first run to get the output with the random seed - output = self.run_tests('-r', test) + output = self.run_tests('-r', test, exitcode=EXITCODE_NO_TESTS_RAN) randseed = self.parse_random_seed(output) match = self.regex_search(r'TESTRANDOM: ([0-9]+)', output) test_random = int(match.group(1)) # try to reproduce with the random seed - output = self.run_tests('-r', '--randseed=%s' % randseed, test) + output = self.run_tests('-r', f'--randseed={randseed}', test, + exitcode=EXITCODE_NO_TESTS_RAN) randseed2 = self.parse_random_seed(output) self.assertEqual(randseed2, randseed) @@ -772,6 +1081,35 @@ def test_random(self): test_random2 = int(match.group(1)) self.assertEqual(test_random2, test_random) + # check that random.seed is used by default + output = self.run_tests(test, exitcode=EXITCODE_NO_TESTS_RAN) + randseed = self.parse_random_seed(output) + self.assertTrue(randseed.isdigit(), randseed) + + # check SOURCE_DATE_EPOCH (integer) + timestamp = '1697839080' + env = dict(os.environ, SOURCE_DATE_EPOCH=timestamp) + output = self.run_tests('-r', test, exitcode=EXITCODE_NO_TESTS_RAN, + env=env) + randseed = self.parse_random_seed(output) + self.assertEqual(randseed, timestamp) + self.check_line(output, 'TESTRANDOM: 520') + + # check SOURCE_DATE_EPOCH (string) + env = dict(os.environ, SOURCE_DATE_EPOCH='XYZ') + output = self.run_tests('-r', test, exitcode=EXITCODE_NO_TESTS_RAN, + env=env) + randseed = self.parse_random_seed(output) + self.assertEqual(randseed, 'XYZ') + self.check_line(output, 'TESTRANDOM: 22') + + # check SOURCE_DATE_EPOCH (empty string): ignore the env var + env = dict(os.environ, SOURCE_DATE_EPOCH='') + output = self.run_tests('-r', test, exitcode=EXITCODE_NO_TESTS_RAN, + env=env) + randseed = self.parse_random_seed(output) + self.assertTrue(randseed.isdigit(), randseed) + def test_fromfile(self): # test --fromfile tests = [self.create_test() for index in range(5)] @@ -794,7 +1132,8 @@ def test_fromfile(self): previous = name output = self.run_tests('--fromfile', filename) - self.check_executed_tests(output, tests) + stats = len(tests) + self.check_executed_tests(output, tests, stats=stats) # test format '[2/7] test_opcodes' with open(filename, "w") as fp: @@ -802,7 +1141,7 @@ def test_fromfile(self): print("[%s/%s] %s" % (index, len(tests), name), file=fp) output = self.run_tests('--fromfile', filename) - self.check_executed_tests(output, tests) + self.check_executed_tests(output, tests, stats=stats) # test format 'test_opcodes' with open(filename, "w") as fp: @@ -810,7 +1149,7 @@ def test_fromfile(self): print(name, file=fp) output = self.run_tests('--fromfile', filename) - self.check_executed_tests(output, tests) + self.check_executed_tests(output, tests, stats=stats) # test format 'Lib/test/test_opcodes.py' with open(filename, "w") as fp: @@ -818,20 +1157,20 @@ def test_fromfile(self): print('Lib/test/%s.py' % name, file=fp) output = self.run_tests('--fromfile', filename) - self.check_executed_tests(output, tests) + self.check_executed_tests(output, tests, stats=stats) def test_interrupted(self): code = TEST_INTERRUPTED test = self.create_test('sigint', code=code) - output = self.run_tests(test, exitcode=130) + output = self.run_tests(test, exitcode=EXITCODE_INTERRUPTED) self.check_executed_tests(output, test, omitted=test, - interrupted=True) + interrupted=True, stats=0) def test_slowest(self): # test --slowest tests = [self.create_test() for index in range(3)] output = self.run_tests("--slowest", *tests) - self.check_executed_tests(output, tests) + self.check_executed_tests(output, tests, stats=len(tests)) regex = ('10 slowest tests:\n' '(?:- %s: .*\n){%s}' % (self.TESTNAME_REGEX, len(tests))) @@ -848,22 +1187,22 @@ def test_slowest_interrupted(self): args = ("--slowest", "-j2", test) else: args = ("--slowest", test) - output = self.run_tests(*args, exitcode=130) + output = self.run_tests(*args, exitcode=EXITCODE_INTERRUPTED) self.check_executed_tests(output, test, - omitted=test, interrupted=True) + omitted=test, interrupted=True, + stats=0) regex = ('10 slowest tests:\n') self.check_line(output, regex) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_coverage(self): # test --coverage test = self.create_test('coverage') output = self.run_tests("--coverage", test) - self.check_executed_tests(output, [test]) + self.check_executed_tests(output, [test], stats=1) regex = (r'lines +cov% +module +\(path\)\n' - r'(?: *[0-9]+ *[0-9]{1,2}% *[^ ]+ +\([^)]+\)+)+') + r'(?: *[0-9]+ *[0-9]{1,2}\.[0-9]% *[^ ]+ +\([^)]+\)+)+') self.check_line(output, regex) def test_wait(self): @@ -890,21 +1229,39 @@ def test_run(self): builtins.__dict__['RUN'] = 1 """) test = self.create_test('forever', code=code) - output = self.run_tests('--forever', test, exitcode=2) - self.check_executed_tests(output, [test]*3, failed=test) - def check_leak(self, code, what): + # --forever + output = self.run_tests('--forever', test, exitcode=EXITCODE_BAD_TEST) + self.check_executed_tests(output, [test]*3, failed=test, + stats=TestStats(3, 1), + forever=True) + + # --forever --rerun + output = self.run_tests('--forever', '--rerun', test, exitcode=0) + self.check_executed_tests(output, [test]*3, + rerun=Rerun(test, + match='test_run', + success=True), + stats=TestStats(4, 1), + forever=True) + + @support.requires_jit_disabled + def check_leak(self, code, what, *, run_workers=False): test = self.create_test('huntrleaks', code=code) filename = 'reflog.txt' self.addCleanup(os_helper.unlink, filename) - output = self.run_tests('--huntrleaks', '3:3:', test, - exitcode=2, + cmd = ['--huntrleaks', '3:3:'] + if run_workers: + cmd.append('-j1') + cmd.append(test) + output = self.run_tests(*cmd, + exitcode=EXITCODE_BAD_TEST, stderr=subprocess.STDOUT) - self.check_executed_tests(output, [test], failed=test) + self.check_executed_tests(output, [test], failed=test, stats=1) - line = 'beginning 6 repetitions\n123456\n......\n' - self.check_line(output, re.escape(line)) + line = r'beginning 6 repetitions. .*\n123:456\n[.0-9X]{3} 111\n' + self.check_line(output, line) line2 = '%s leaked [1, 1, 1] %s, sum=3\n' % (test, what) self.assertIn(line2, output) @@ -913,8 +1270,8 @@ def check_leak(self, code, what): reflog = fp.read() self.assertIn(line2, reflog) - @unittest.skipUnless(Py_DEBUG, 'need a debug build') - def test_huntrleaks(self): + @unittest.skipUnless(support.Py_DEBUG, 'need a debug build') + def check_huntrleaks(self, *, run_workers: bool): # test --huntrleaks code = textwrap.dedent(""" import unittest @@ -925,9 +1282,56 @@ class RefLeakTest(unittest.TestCase): def test_leak(self): GLOBAL_LIST.append(object()) """) - self.check_leak(code, 'references') + self.check_leak(code, 'references', run_workers=run_workers) - @unittest.skipUnless(Py_DEBUG, 'need a debug build') + def test_huntrleaks(self): + self.check_huntrleaks(run_workers=False) + + def test_huntrleaks_mp(self): + self.check_huntrleaks(run_workers=True) + + @unittest.skipUnless(support.Py_DEBUG, 'need a debug build') + def test_huntrleaks_bisect(self): + # test --huntrleaks --bisect + code = textwrap.dedent(""" + import unittest + + GLOBAL_LIST = [] + + class RefLeakTest(unittest.TestCase): + def test1(self): + pass + + def test2(self): + pass + + def test3(self): + GLOBAL_LIST.append(object()) + + def test4(self): + pass + """) + + test = self.create_test('huntrleaks', code=code) + + filename = 'reflog.txt' + self.addCleanup(os_helper.unlink, filename) + cmd = ['--huntrleaks', '3:3:', '--bisect', test] + output = self.run_tests(*cmd, + exitcode=EXITCODE_BAD_TEST, + stderr=subprocess.STDOUT) + + self.assertIn(f"Bisect {test}", output) + self.assertIn(f"Bisect {test}: exit code 0", output) + + # test3 is the one which leaks + self.assertIn("Bisection completed in", output) + self.assertIn( + "Tests (1):\n" + f"* {test}.RefLeakTest.test3\n", + output) + + @unittest.skipUnless(support.Py_DEBUG, 'need a debug build') def test_huntrleaks_fd_leak(self): # test --huntrleaks for file descriptor leak code = textwrap.dedent(""" @@ -981,16 +1385,14 @@ def test_crashed(self): crash_test = self.create_test(name="crash", code=code) tests = [crash_test] - output = self.run_tests("-j2", *tests, exitcode=2) + output = self.run_tests("-j2", *tests, exitcode=EXITCODE_BAD_TEST) self.check_executed_tests(output, tests, failed=crash_test, - randomize=True) + parallel=True, stats=0) def parse_methods(self, output): regex = re.compile("^(test[^ ]+).*ok$", flags=re.MULTILINE) return [match.group(1) for match in regex.finditer(output)] - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_ignorefile(self): code = textwrap.dedent(""" import unittest @@ -1005,8 +1407,6 @@ def test_method3(self): def test_method4(self): pass """) - all_methods = ['test_method1', 'test_method2', - 'test_method3', 'test_method4'] testname = self.create_test(code=code) # only run a subset @@ -1068,8 +1468,6 @@ def test_method4(self): subset = ['test_method1', 'test_method3'] self.assertEqual(methods, subset) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_env_changed(self): code = textwrap.dedent(""" import unittest @@ -1082,52 +1480,265 @@ def test_env_changed(self): # don't fail by default output = self.run_tests(testname) - self.check_executed_tests(output, [testname], env_changed=testname) + self.check_executed_tests(output, [testname], + env_changed=testname, stats=1) # fail with --fail-env-changed - output = self.run_tests("--fail-env-changed", testname, exitcode=3) + output = self.run_tests("--fail-env-changed", testname, + exitcode=EXITCODE_ENV_CHANGED) self.check_executed_tests(output, [testname], env_changed=testname, - fail_env_changed=True) + fail_env_changed=True, stats=1) + + # rerun + output = self.run_tests("--rerun", testname) + self.check_executed_tests(output, [testname], + env_changed=testname, + rerun=Rerun(testname, + match=None, + success=True), + stats=2) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_rerun_fail(self): # FAILURE then FAILURE code = textwrap.dedent(""" import unittest class Tests(unittest.TestCase): - def test_bug(self): - # test always fail + def test_succeed(self): + return + + def test_fail_always(self): + # test that always fails self.fail("bug") """) testname = self.create_test(code=code) - output = self.run_tests("-w", testname, exitcode=2) + output = self.run_tests("--rerun", testname, exitcode=EXITCODE_BAD_TEST) self.check_executed_tests(output, [testname], - failed=testname, rerun=testname) + rerun=Rerun(testname, + "test_fail_always", + success=False), + stats=TestStats(3, 2)) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_rerun_success(self): # FAILURE then SUCCESS - code = textwrap.dedent(""" - import builtins + marker_filename = os.path.abspath("regrtest_marker_filename") + self.addCleanup(os_helper.unlink, marker_filename) + self.assertFalse(os.path.exists(marker_filename)) + + code = textwrap.dedent(f""" + import os.path import unittest + marker_filename = {marker_filename!r} + class Tests(unittest.TestCase): - failed = False + def test_succeed(self): + return def test_fail_once(self): - if not hasattr(builtins, '_test_failed'): - builtins._test_failed = True + if not os.path.exists(marker_filename): + open(marker_filename, "w").close() self.fail("bug") """) testname = self.create_test(code=code) - output = self.run_tests("-w", testname, exitcode=0) + # FAILURE then SUCCESS => exit code 0 + output = self.run_tests("--rerun", testname, exitcode=0) self.check_executed_tests(output, [testname], - rerun=testname) + rerun=Rerun(testname, + match="test_fail_once", + success=True), + stats=TestStats(3, 1)) + os_helper.unlink(marker_filename) + + # with --fail-rerun, exit code EXITCODE_RERUN_FAIL + # on "FAILURE then SUCCESS" state. + output = self.run_tests("--rerun", "--fail-rerun", testname, + exitcode=EXITCODE_RERUN_FAIL) + self.check_executed_tests(output, [testname], + rerun=Rerun(testname, + match="test_fail_once", + success=True), + stats=TestStats(3, 1)) + os_helper.unlink(marker_filename) + + def test_rerun_setup_class_hook_failure(self): + # FAILURE then FAILURE + code = textwrap.dedent(""" + import unittest + + class ExampleTests(unittest.TestCase): + @classmethod + def setUpClass(self): + raise RuntimeError('Fail') + + def test_success(self): + return + """) + testname = self.create_test(code=code) + + output = self.run_tests("--rerun", testname, exitcode=EXITCODE_BAD_TEST) + self.check_executed_tests(output, testname, + failed=[testname], + rerun=Rerun(testname, + match="ExampleTests", + success=False), + stats=0) + + def test_rerun_teardown_class_hook_failure(self): + # FAILURE then FAILURE + code = textwrap.dedent(""" + import unittest + + class ExampleTests(unittest.TestCase): + @classmethod + def tearDownClass(self): + raise RuntimeError('Fail') + + def test_success(self): + return + """) + testname = self.create_test(code=code) + + output = self.run_tests("--rerun", testname, exitcode=EXITCODE_BAD_TEST) + self.check_executed_tests(output, testname, + failed=[testname], + rerun=Rerun(testname, + match="ExampleTests", + success=False), + stats=2) + + def test_rerun_setup_module_hook_failure(self): + # FAILURE then FAILURE + code = textwrap.dedent(""" + import unittest + + def setUpModule(): + raise RuntimeError('Fail') + + class ExampleTests(unittest.TestCase): + def test_success(self): + return + """) + testname = self.create_test(code=code) + + output = self.run_tests("--rerun", testname, exitcode=EXITCODE_BAD_TEST) + self.check_executed_tests(output, testname, + failed=[testname], + rerun=Rerun(testname, + match=None, + success=False), + stats=0) + + def test_rerun_teardown_module_hook_failure(self): + # FAILURE then FAILURE + code = textwrap.dedent(""" + import unittest + + def tearDownModule(): + raise RuntimeError('Fail') + + class ExampleTests(unittest.TestCase): + def test_success(self): + return + """) + testname = self.create_test(code=code) + + output = self.run_tests("--rerun", testname, exitcode=EXITCODE_BAD_TEST) + self.check_executed_tests(output, [testname], + failed=[testname], + rerun=Rerun(testname, + match=None, + success=False), + stats=2) + + def test_rerun_setup_hook_failure(self): + # FAILURE then FAILURE + code = textwrap.dedent(""" + import unittest + + class ExampleTests(unittest.TestCase): + def setUp(self): + raise RuntimeError('Fail') + + def test_success(self): + return + """) + testname = self.create_test(code=code) + + output = self.run_tests("--rerun", testname, exitcode=EXITCODE_BAD_TEST) + self.check_executed_tests(output, testname, + failed=[testname], + rerun=Rerun(testname, + match="test_success", + success=False), + stats=2) + + def test_rerun_teardown_hook_failure(self): + # FAILURE then FAILURE + code = textwrap.dedent(""" + import unittest + + class ExampleTests(unittest.TestCase): + def tearDown(self): + raise RuntimeError('Fail') + + def test_success(self): + return + """) + testname = self.create_test(code=code) + + output = self.run_tests("--rerun", testname, exitcode=EXITCODE_BAD_TEST) + self.check_executed_tests(output, testname, + failed=[testname], + rerun=Rerun(testname, + match="test_success", + success=False), + stats=2) + + def test_rerun_async_setup_hook_failure(self): + # FAILURE then FAILURE + code = textwrap.dedent(""" + import unittest + + class ExampleTests(unittest.IsolatedAsyncioTestCase): + async def asyncSetUp(self): + raise RuntimeError('Fail') + + async def test_success(self): + return + """) + testname = self.create_test(code=code) + + output = self.run_tests("--rerun", testname, exitcode=EXITCODE_BAD_TEST) + self.check_executed_tests(output, testname, + rerun=Rerun(testname, + match="test_success", + success=False), + stats=2) + + def test_rerun_async_teardown_hook_failure(self): + # FAILURE then FAILURE + code = textwrap.dedent(""" + import unittest + + class ExampleTests(unittest.IsolatedAsyncioTestCase): + async def asyncTearDown(self): + raise RuntimeError('Fail') + + async def test_success(self): + return + """) + testname = self.create_test(code=code) + + output = self.run_tests("--rerun", testname, exitcode=EXITCODE_BAD_TEST) + self.check_executed_tests(output, testname, + failed=[testname], + rerun=Rerun(testname, + match="test_success", + success=False), + stats=2) def test_no_tests_ran(self): code = textwrap.dedent(""" @@ -1139,8 +1750,11 @@ def test_bug(self): """) testname = self.create_test(code=code) - output = self.run_tests(testname, "-m", "nosuchtest", exitcode=0) - self.check_executed_tests(output, [testname], no_test_ran=testname) + output = self.run_tests(testname, "-m", "nosuchtest", + exitcode=EXITCODE_NO_TESTS_RAN) + self.check_executed_tests(output, [testname], + run_no_tests=testname, + stats=0, filtered=True) def test_no_tests_ran_skip(self): code = textwrap.dedent(""" @@ -1152,8 +1766,9 @@ def test_skipped(self): """) testname = self.create_test(code=code) - output = self.run_tests(testname, exitcode=0) - self.check_executed_tests(output, [testname]) + output = self.run_tests(testname) + self.check_executed_tests(output, [testname], + stats=TestStats(1, skipped=1)) def test_no_tests_ran_multiple_tests_nonexistent(self): code = textwrap.dedent(""" @@ -1166,9 +1781,11 @@ def test_bug(self): testname = self.create_test(code=code) testname2 = self.create_test(code=code) - output = self.run_tests(testname, testname2, "-m", "nosuchtest", exitcode=0) + output = self.run_tests(testname, testname2, "-m", "nosuchtest", + exitcode=EXITCODE_NO_TESTS_RAN) self.check_executed_tests(output, [testname, testname2], - no_test_ran=[testname, testname2]) + run_no_tests=[testname, testname2], + stats=0, filtered=True) def test_no_test_ran_some_test_exist_some_not(self): code = textwrap.dedent(""" @@ -1191,10 +1808,14 @@ def test_other_bug(self): output = self.run_tests(testname, testname2, "-m", "nosuchtest", "-m", "test_other_bug", exitcode=0) self.check_executed_tests(output, [testname, testname2], - no_test_ran=[testname]) + run_no_tests=[testname], + stats=1, filtered=True) @support.cpython_only - def test_findleaks(self): + def test_uncollectable(self): + # Skip test if _testcapi is missing + import_helper.import_module('_testcapi') + code = textwrap.dedent(r""" import _testcapi import gc @@ -1214,19 +1835,13 @@ def test_garbage(self): """) testname = self.create_test(code=code) - output = self.run_tests("--fail-env-changed", testname, exitcode=3) - self.check_executed_tests(output, [testname], - env_changed=[testname], - fail_env_changed=True) - - # --findleaks is now basically an alias to --fail-env-changed - output = self.run_tests("--findleaks", testname, exitcode=3) + output = self.run_tests("--fail-env-changed", testname, + exitcode=EXITCODE_ENV_CHANGED) self.check_executed_tests(output, [testname], env_changed=[testname], - fail_env_changed=True) + fail_env_changed=True, + stats=1) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_multiprocessing_timeout(self): code = textwrap.dedent(r""" import time @@ -1248,14 +1863,131 @@ def test_sleep(self): """) testname = self.create_test(code=code) - output = self.run_tests("-j2", "--timeout=1.0", testname, exitcode=2) + output = self.run_tests("-j2", "--timeout=1.0", testname, + exitcode=EXITCODE_BAD_TEST) self.check_executed_tests(output, [testname], - failed=testname) + failed=testname, stats=0) self.assertRegex(output, re.compile('%s timed out' % testname, re.MULTILINE)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_unraisable_exc(self): + # --fail-env-changed must catch unraisable exception. + # The exception must be displayed even if sys.stderr is redirected. + code = textwrap.dedent(r""" + import unittest + import weakref + from test.support import captured_stderr + + class MyObject: + pass + + def weakref_callback(obj): + raise Exception("weakref callback bug") + + class Tests(unittest.TestCase): + def test_unraisable_exc(self): + obj = MyObject() + ref = weakref.ref(obj, weakref_callback) + with captured_stderr() as stderr: + # call weakref_callback() which logs + # an unraisable exception + obj = None + self.assertEqual(stderr.getvalue(), '') + """) + testname = self.create_test(code=code) + + output = self.run_tests("--fail-env-changed", "-v", testname, + exitcode=EXITCODE_ENV_CHANGED) + self.check_executed_tests(output, [testname], + env_changed=[testname], + fail_env_changed=True, + stats=1) + self.assertIn("Warning -- Unraisable exception", output) + self.assertIn("Exception: weakref callback bug", output) + + def test_threading_excepthook(self): + # --fail-env-changed must catch uncaught thread exception. + # The exception must be displayed even if sys.stderr is redirected. + code = textwrap.dedent(r""" + import threading + import unittest + from test.support import captured_stderr + + class MyObject: + pass + + def func_bug(): + raise Exception("bug in thread") + + class Tests(unittest.TestCase): + def test_threading_excepthook(self): + with captured_stderr() as stderr: + thread = threading.Thread(target=func_bug) + thread.start() + thread.join() + self.assertEqual(stderr.getvalue(), '') + """) + testname = self.create_test(code=code) + + output = self.run_tests("--fail-env-changed", "-v", testname, + exitcode=EXITCODE_ENV_CHANGED) + self.check_executed_tests(output, [testname], + env_changed=[testname], + fail_env_changed=True, + stats=1) + self.assertIn("Warning -- Uncaught thread exception", output) + self.assertIn("Exception: bug in thread", output) + + def test_print_warning(self): + # bpo-45410: The order of messages must be preserved when -W and + # support.print_warning() are used. + code = textwrap.dedent(r""" + import sys + import unittest + from test import support + + class MyObject: + pass + + def func_bug(): + raise Exception("bug in thread") + + class Tests(unittest.TestCase): + def test_print_warning(self): + print("msg1: stdout") + support.print_warning("msg2: print_warning") + # Fail with ENV CHANGED to see print_warning() log + support.environment_altered = True + """) + testname = self.create_test(code=code) + + # Expect an output like: + # + # test_threading_excepthook (test.test_x.Tests) ... msg1: stdout + # Warning -- msg2: print_warning + # ok + regex = (r"test_print_warning.*msg1: stdout\n" + r"Warning -- msg2: print_warning\n" + r"ok\n") + for option in ("-v", "-W"): + with self.subTest(option=option): + cmd = ["--fail-env-changed", option, testname] + output = self.run_tests(*cmd, exitcode=EXITCODE_ENV_CHANGED) + self.check_executed_tests(output, [testname], + env_changed=[testname], + fail_env_changed=True, + stats=1) + self.assertRegex(output, regex) + + def test_unicode_guard_env(self): + guard = os.environ.get(setup.UNICODE_GUARD_ENV) + self.assertIsNotNone(guard, f"{setup.UNICODE_GUARD_ENV} not set") + if guard.isascii(): + # Skip to signify that the env var value was changed by the user; + # possibly to something ASCII to work around Unicode issues. + self.skipTest("Modified guard") + def test_cleanup(self): dirname = os.path.join(self.tmptestdir, "test_python_123") os.mkdir(dirname) @@ -1271,10 +2003,412 @@ def test_cleanup(self): for name in names: self.assertFalse(os.path.exists(name), name) + @unittest.skip("TODO: RUSTPYTHON; flaky") + @unittest.skipIf(support.is_wasi, + 'checking temp files is not implemented on WASI') + def test_leak_tmp_file(self): + code = textwrap.dedent(r""" + import os.path + import tempfile + import unittest + + class FileTests(unittest.TestCase): + def test_leak_tmp_file(self): + filename = os.path.join(tempfile.gettempdir(), 'mytmpfile') + with open(filename, "wb") as fp: + fp.write(b'content') + """) + testnames = [self.create_test(code=code) for _ in range(3)] + + output = self.run_tests("--fail-env-changed", "-v", "-j2", *testnames, + exitcode=EXITCODE_ENV_CHANGED) + self.check_executed_tests(output, testnames, + env_changed=testnames, + fail_env_changed=True, + parallel=True, + stats=len(testnames)) + for testname in testnames: + self.assertIn(f"Warning -- {testname} leaked temporary " + f"files (1): mytmpfile", + output) + + def test_worker_decode_error(self): + # gh-109425: Use "backslashreplace" error handler to decode stdout. + if sys.platform == 'win32': + encoding = locale.getencoding() + else: + encoding = sys.stdout.encoding + if encoding is None: + encoding = sys.__stdout__.encoding + if encoding is None: + self.skipTest("cannot get regrtest worker encoding") + + nonascii = bytes(ch for ch in range(128, 256)) + corrupted_output = b"nonascii:%s\n" % (nonascii,) + # gh-108989: On Windows, assertion errors are written in UTF-16: when + # decoded each letter is follow by a NUL character. + assertion_failed = 'Assertion failed: tstate_is_alive(tstate)\n' + corrupted_output += assertion_failed.encode('utf-16-le') + try: + corrupted_output.decode(encoding) + except UnicodeDecodeError: + pass + else: + self.skipTest(f"{encoding} can decode non-ASCII bytes") + + expected_line = corrupted_output.decode(encoding, 'backslashreplace') + + code = textwrap.dedent(fr""" + import sys + import unittest + + class Tests(unittest.TestCase): + def test_pass(self): + pass + + # bytes which cannot be decoded from UTF-8 + corrupted_output = {corrupted_output!a} + sys.stdout.buffer.write(corrupted_output) + sys.stdout.buffer.flush() + """) + testname = self.create_test(code=code) + + output = self.run_tests("--fail-env-changed", "-v", "-j1", testname) + self.check_executed_tests(output, [testname], + parallel=True, + stats=1) + self.check_line(output, expected_line, regex=False) + + def test_doctest(self): + code = textwrap.dedent(r''' + import doctest + import sys + from test import support + + def my_function(): + """ + Pass: + + >>> 1 + 1 + 2 + + Failure: + + >>> 2 + 3 + 23 + >>> 1 + 1 + 11 + + Skipped test (ignored): + + >>> id(1.0) # doctest: +SKIP + 7948648 + """ + + def load_tests(loader, tests, pattern): + tests.addTest(doctest.DocTestSuite()) + return tests + ''') + testname = self.create_test(code=code) + + output = self.run_tests("--fail-env-changed", "-v", "-j1", testname, + exitcode=EXITCODE_BAD_TEST) + self.check_executed_tests(output, [testname], + failed=[testname], + parallel=True, + stats=TestStats(1, 1, 0)) + + def _check_random_seed(self, run_workers: bool): + # gh-109276: When -r/--randomize is used, random.seed() is called + # with the same random seed before running each test file. + code = textwrap.dedent(r''' + import random + import unittest + + class RandomSeedTest(unittest.TestCase): + def test_randint(self): + numbers = [random.randint(0, 1000) for _ in range(10)] + print(f"Random numbers: {numbers}") + ''') + tests = [self.create_test(name=f'test_random{i}', code=code) + for i in range(1, 3+1)] + + random_seed = 856_656_202 + cmd = ["--randomize", f"--randseed={random_seed}"] + if run_workers: + # run as many worker processes than the number of tests + cmd.append(f'-j{len(tests)}') + cmd.extend(tests) + output = self.run_tests(*cmd) + + random.seed(random_seed) + # Make the assumption that nothing consume entropy between libregrest + # setup_tests() which calls random.seed() and RandomSeedTest calling + # random.randint(). + numbers = [random.randint(0, 1000) for _ in range(10)] + expected = f"Random numbers: {numbers}" + + regex = r'^Random numbers: .*$' + matches = re.findall(regex, output, flags=re.MULTILINE) + self.assertEqual(matches, [expected] * len(tests)) + + def test_random_seed(self): + self._check_random_seed(run_workers=False) + + @unittest.skip("TODO: RUSTPYTHON; flaky") + def test_random_seed_workers(self): + self._check_random_seed(run_workers=True) + + @unittest.skip("TODO: RUSTPYTHON; flaky") + def test_python_command(self): + code = textwrap.dedent(r""" + import sys + import unittest + + class WorkerTests(unittest.TestCase): + def test_dev_mode(self): + self.assertTrue(sys.flags.dev_mode) + """) + tests = [self.create_test(code=code) for _ in range(3)] + + # Custom Python command: "python -X dev" + python_cmd = [sys.executable, '-X', 'dev'] + # test.libregrtest.cmdline uses shlex.split() to parse the Python + # command line string + python_cmd = shlex.join(python_cmd) + + output = self.run_tests("--python", python_cmd, "-j0", *tests) + self.check_executed_tests(output, tests, + stats=len(tests), parallel=True) + + def test_unload_tests(self): + # Test that unloading test modules does not break tests + # that import from other tests. + # The test execution order matters for this test. + # Both test_regrtest_a and test_regrtest_c which are executed before + # and after test_regrtest_b import a submodule from the test_regrtest_b + # package and use it in testing. test_regrtest_b itself does not import + # that submodule. + # Previously test_regrtest_c failed because test_regrtest_b.util in + # sys.modules was left after test_regrtest_a (making the import + # statement no-op), but new test_regrtest_b without the util attribute + # was imported for test_regrtest_b. + testdir = os.path.join(os.path.dirname(__file__), + 'regrtestdata', 'import_from_tests') + tests = [f'test_regrtest_{name}' for name in ('a', 'b', 'c')] + args = ['-Wd', '-E', '-bb', '-m', 'test', '--testdir=%s' % testdir, *tests] + output = self.run_python(args) + self.check_executed_tests(output, tests, stats=3) + + def check_add_python_opts(self, option): + # --fast-ci and --slow-ci add "-u -W default -bb -E" options to Python + + # Skip test if _testinternalcapi is missing + import_helper.import_module('_testinternalcapi') + + code = textwrap.dedent(r""" + import sys + import unittest + from test import support + try: + from _testcapi import config_get + except ImportError: + config_get = None + + # WASI/WASM buildbots don't use -E option + use_environment = (support.is_emscripten or support.is_wasi) + + class WorkerTests(unittest.TestCase): + @unittest.skipUnless(config_get is None, 'need config_get()') + def test_config(self): + config = config_get() + # -u option + self.assertEqual(config_get('buffered_stdio'), 0) + # -W default option + self.assertTrue(config_get('warnoptions'), ['default']) + # -bb option + self.assertTrue(config_get('bytes_warning'), 2) + # -E option + self.assertTrue(config_get('use_environment'), use_environment) + + def test_python_opts(self): + # -u option + self.assertTrue(sys.__stdout__.write_through) + self.assertTrue(sys.__stderr__.write_through) + + # -W default option + self.assertTrue(sys.warnoptions, ['default']) + + # -bb option + self.assertEqual(sys.flags.bytes_warning, 2) + + # -E option + self.assertEqual(not sys.flags.ignore_environment, + use_environment) + """) + testname = self.create_test(code=code) + + # Use directly subprocess to control the exact command line + cmd = [sys.executable, + "-m", "test", option, + f'--testdir={self.tmptestdir}', + testname] + proc = subprocess.run(cmd, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True) + self.assertEqual(proc.returncode, 0, proc) + + def test_add_python_opts(self): + for opt in ("--fast-ci", "--slow-ci"): + with self.subTest(opt=opt): + self.check_add_python_opts(opt) + + # gh-76319: Raising SIGSEGV on Android may not cause a crash. + @unittest.skipIf(support.is_android, + 'raising SIGSEGV on Android is unreliable') + def test_worker_output_on_failure(self): + # Skip test if faulthandler is missing + import_helper.import_module('faulthandler') + + code = textwrap.dedent(r""" + import faulthandler + import unittest + from test import support + + class CrashTests(unittest.TestCase): + def test_crash(self): + print("just before crash!", flush=True) + + with support.SuppressCrashReport(): + faulthandler._sigsegv(True) + """) + testname = self.create_test(code=code) + + # Sanitizers must not handle SIGSEGV (ex: for test_enable_fd()) + env = dict(os.environ) + option = 'handle_segv=0' + support.set_sanitizer_env_var(env, option) + + output = self.run_tests("-j1", testname, + exitcode=EXITCODE_BAD_TEST, + env=env) + self.check_executed_tests(output, testname, + failed=[testname], + stats=0, parallel=True) + if not support.MS_WINDOWS: + exitcode = -int(signal.SIGSEGV) + self.assertIn(f"Exit code {exitcode} (SIGSEGV)", output) + self.check_line(output, "just before crash!", full=True, regex=False) + + def test_verbose3(self): + code = textwrap.dedent(r""" + import unittest + from test import support + + class VerboseTests(unittest.TestCase): + def test_pass(self): + print("SPAM SPAM SPAM") + """) + testname = self.create_test(code=code) + + # Run sequentially + output = self.run_tests("--verbose3", testname) + self.check_executed_tests(output, testname, stats=1) + self.assertNotIn('SPAM SPAM SPAM', output) + + # -R option needs a debug build + if support.Py_DEBUG: + # Check for reference leaks, run in parallel + output = self.run_tests("-R", "3:3", "-j1", "--verbose3", testname) + self.check_executed_tests(output, testname, stats=1, parallel=True) + self.assertNotIn('SPAM SPAM SPAM', output) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_xml(self): + code = textwrap.dedent(r""" + import unittest + + class VerboseTests(unittest.TestCase): + def test_failed(self): + print("abc \x1b def") + self.fail() + """) + testname = self.create_test(code=code) + + # Run sequentially + filename = os_helper.TESTFN + self.addCleanup(os_helper.unlink, filename) + + output = self.run_tests(testname, "--junit-xml", filename, + exitcode=EXITCODE_BAD_TEST) + self.check_executed_tests(output, testname, + failed=testname, + stats=TestStats(1, 1, 0)) + + # Test generated XML + with open(filename, encoding="utf8") as fp: + content = fp.read() + + testsuite = ElementTree.fromstring(content) + self.assertEqual(int(testsuite.get('tests')), 1) + self.assertEqual(int(testsuite.get('errors')), 0) + self.assertEqual(int(testsuite.get('failures')), 1) + + testcase = testsuite[0][0] + self.assertEqual(testcase.get('status'), 'run') + self.assertEqual(testcase.get('result'), 'completed') + self.assertGreater(float(testcase.get('time')), 0) + for out in testcase.iter('system-out'): + self.assertEqual(out.text, r"abc \x1b def") + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_nonascii(self): + code = textwrap.dedent(r""" + import unittest + + class NonASCIITests(unittest.TestCase): + def test_docstring(self): + '''docstring:\u20ac''' + + def test_subtest(self): + with self.subTest(param='subtest:\u20ac'): + pass + + def test_skip(self): + self.skipTest('skipped:\u20ac') + """) + testname = self.create_test(code=code) + + env = dict(os.environ) + env['PYTHONIOENCODING'] = 'ascii' + + def check(output): + self.check_executed_tests(output, testname, stats=TestStats(3, 0, 1)) + self.assertIn(r'docstring:\u20ac', output) + self.assertIn(r'skipped:\u20ac', output) + + # Run sequentially + output = self.run_tests('-v', testname, env=env, isolated=False) + check(output) + + # Run in parallel + output = self.run_tests('-j1', '-v', testname, env=env, isolated=False) + check(output) + + def test_pgo_exclude(self): + # Get PGO tests + output = self.run_tests('--pgo', '--list-tests') + pgo_tests = output.strip().split() + + # Exclude test_re + output = self.run_tests('--pgo', '--list-tests', '-x', 'test_re') + tests = output.strip().split() + self.assertNotIn('test_re', tests) + self.assertEqual(len(tests), len(pgo_tests) - 1) + class TestUtils(unittest.TestCase): - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_format_duration(self): self.assertEqual(utils.format_duration(0), '0 ms') @@ -1297,6 +2431,219 @@ def test_format_duration(self): self.assertEqual(utils.format_duration(3 * 3600 + 1), '3 hour 1 sec') + def test_normalize_test_name(self): + normalize = normalize_test_name + self.assertEqual(normalize('test_access (test.test_os.FileTests.test_access)'), + 'test_access') + self.assertEqual(normalize('setUpClass (test.test_os.ChownFileTests)', is_error=True), + 'ChownFileTests') + self.assertEqual(normalize('test_success (test.test_bug.ExampleTests.test_success)', is_error=True), + 'test_success') + self.assertIsNone(normalize('setUpModule (test.test_x)', is_error=True)) + self.assertIsNone(normalize('tearDownModule (test.test_module)', is_error=True)) + + def test_format_resources(self): + format_resources = utils.format_resources + ALL_RESOURCES = utils.ALL_RESOURCES + self.assertEqual( + format_resources(("network",)), + 'resources (1): network') + self.assertEqual( + format_resources(("audio", "decimal", "network")), + 'resources (3): audio,decimal,network') + self.assertEqual( + format_resources(ALL_RESOURCES), + 'resources: all') + self.assertEqual( + format_resources(tuple(name for name in ALL_RESOURCES + if name != "cpu")), + 'resources: all,-cpu') + self.assertEqual( + format_resources((*ALL_RESOURCES, "tzdata")), + 'resources: all,tzdata') + + def test_match_test(self): + class Test: + def __init__(self, test_id): + self.test_id = test_id + + def id(self): + return self.test_id + + # Restore patterns once the test completes + patterns = get_match_tests() + self.addCleanup(set_match_tests, patterns) + + test_access = Test('test.test_os.FileTests.test_access') + test_chdir = Test('test.test_os.Win32ErrorTests.test_chdir') + test_copy = Test('test.test_shutil.TestCopy.test_copy') + + # Test acceptance + with support.swap_attr(support, '_test_matchers', ()): + # match all + set_match_tests([]) + self.assertTrue(match_test(test_access)) + self.assertTrue(match_test(test_chdir)) + + # match all using None + set_match_tests(None) + self.assertTrue(match_test(test_access)) + self.assertTrue(match_test(test_chdir)) + + # match the full test identifier + set_match_tests([(test_access.id(), True)]) + self.assertTrue(match_test(test_access)) + self.assertFalse(match_test(test_chdir)) + + # match the module name + set_match_tests([('test_os', True)]) + self.assertTrue(match_test(test_access)) + self.assertTrue(match_test(test_chdir)) + self.assertFalse(match_test(test_copy)) + + # Test '*' pattern + set_match_tests([('test_*', True)]) + self.assertTrue(match_test(test_access)) + self.assertTrue(match_test(test_chdir)) + + # Test case sensitivity + set_match_tests([('filetests', True)]) + self.assertFalse(match_test(test_access)) + set_match_tests([('FileTests', True)]) + self.assertTrue(match_test(test_access)) + + # Test pattern containing '.' and a '*' metacharacter + set_match_tests([('*test_os.*.test_*', True)]) + self.assertTrue(match_test(test_access)) + self.assertTrue(match_test(test_chdir)) + self.assertFalse(match_test(test_copy)) + + # Multiple patterns + set_match_tests([(test_access.id(), True), (test_chdir.id(), True)]) + self.assertTrue(match_test(test_access)) + self.assertTrue(match_test(test_chdir)) + self.assertFalse(match_test(test_copy)) + + set_match_tests([('test_access', True), ('DONTMATCH', True)]) + self.assertTrue(match_test(test_access)) + self.assertFalse(match_test(test_chdir)) + + # Test rejection + with support.swap_attr(support, '_test_matchers', ()): + # match the full test identifier + set_match_tests([(test_access.id(), False)]) + self.assertFalse(match_test(test_access)) + self.assertTrue(match_test(test_chdir)) + + # match the module name + set_match_tests([('test_os', False)]) + self.assertFalse(match_test(test_access)) + self.assertFalse(match_test(test_chdir)) + self.assertTrue(match_test(test_copy)) + + # Test '*' pattern + set_match_tests([('test_*', False)]) + self.assertFalse(match_test(test_access)) + self.assertFalse(match_test(test_chdir)) + + # Test case sensitivity + set_match_tests([('filetests', False)]) + self.assertTrue(match_test(test_access)) + set_match_tests([('FileTests', False)]) + self.assertFalse(match_test(test_access)) + + # Test pattern containing '.' and a '*' metacharacter + set_match_tests([('*test_os.*.test_*', False)]) + self.assertFalse(match_test(test_access)) + self.assertFalse(match_test(test_chdir)) + self.assertTrue(match_test(test_copy)) + + # Multiple patterns + set_match_tests([(test_access.id(), False), (test_chdir.id(), False)]) + self.assertFalse(match_test(test_access)) + self.assertFalse(match_test(test_chdir)) + self.assertTrue(match_test(test_copy)) + + set_match_tests([('test_access', False), ('DONTMATCH', False)]) + self.assertFalse(match_test(test_access)) + self.assertTrue(match_test(test_chdir)) + + # Test mixed filters + with support.swap_attr(support, '_test_matchers', ()): + set_match_tests([('*test_os', False), ('test_access', True)]) + self.assertTrue(match_test(test_access)) + self.assertFalse(match_test(test_chdir)) + self.assertTrue(match_test(test_copy)) + + set_match_tests([('*test_os', True), ('test_access', False)]) + self.assertFalse(match_test(test_access)) + self.assertTrue(match_test(test_chdir)) + self.assertFalse(match_test(test_copy)) + + def test_sanitize_xml(self): + sanitize_xml = utils.sanitize_xml + + # escape invalid XML characters + self.assertEqual(sanitize_xml('abc \x1b\x1f def'), + r'abc \x1b\x1f def') + self.assertEqual(sanitize_xml('nul:\x00, bell:\x07'), + r'nul:\x00, bell:\x07') + self.assertEqual(sanitize_xml('surrogate:\uDC80'), + r'surrogate:\udc80') + self.assertEqual(sanitize_xml('illegal \uFFFE and \uFFFF'), + r'illegal \ufffe and \uffff') + + # no escape for valid XML characters + self.assertEqual(sanitize_xml('a\n\tb'), + 'a\n\tb') + self.assertEqual(sanitize_xml('valid t\xe9xt \u20ac'), + 'valid t\xe9xt \u20ac') + + +from test.libregrtest.results import TestResults + + +class TestColorized(unittest.TestCase): + def test_test_result_get_state(self): + # Arrange + green = _colorize.ANSIColors.GREEN + red = _colorize.ANSIColors.BOLD_RED + reset = _colorize.ANSIColors.RESET + yellow = _colorize.ANSIColors.YELLOW + + good_results = TestResults() + good_results.good = ["good1", "good2"] + bad_results = TestResults() + bad_results.bad = ["bad1", "bad2"] + no_results = TestResults() + no_results.bad = [] + interrupted_results = TestResults() + interrupted_results.interrupted = True + interrupted_worker_bug = TestResults() + interrupted_worker_bug.interrupted = True + interrupted_worker_bug.worker_bug = True + + for results, expected in ( + (good_results, f"{green}SUCCESS{reset}"), + (bad_results, f"{red}FAILURE{reset}"), + (no_results, f"{yellow}NO TESTS RAN{reset}"), + (interrupted_results, f"{yellow}INTERRUPTED{reset}"), + ( + interrupted_worker_bug, + f"{yellow}INTERRUPTED{reset}, {red}WORKER BUG{reset}", + ), + ): + with self.subTest(results=results, expected=expected): + # Act + with unittest.mock.patch( + "_colorize.can_colorize", return_value=True + ): + result = results.get_state(fail_env_changed=False) + + # Assert + self.assertEqual(result, expected) + if __name__ == '__main__': + setup.setup_process() unittest.main() diff --git a/Lib/test/test_reprlib.py b/Lib/test/test_reprlib.py index 738b48f5623..173b49e83d1 100644 --- a/Lib/test/test_reprlib.py +++ b/Lib/test/test_reprlib.py @@ -3,6 +3,7 @@ Nick Mathewson """ +import annotationlib import sys import os import shutil @@ -11,7 +12,7 @@ import unittest import textwrap -from test.support import verbose +from test.support import verbose, EqualToForwardRef from test.support.os_helper import create_empty_file from reprlib import repr as r # Don't shadow builtin repr from reprlib import Repr @@ -149,15 +150,40 @@ def test_frozenset(self): eq(r(frozenset({1, 2, 3, 4, 5, 6})), "frozenset({1, 2, 3, 4, 5, 6})") eq(r(frozenset({1, 2, 3, 4, 5, 6, 7})), "frozenset({1, 2, 3, 4, 5, 6, ...})") + @unittest.expectedFailure # TODO: RUSTPYTHON def test_numbers(self): - eq = self.assertEqual - eq(r(123), repr(123)) - eq(r(123), repr(123)) - eq(r(1.0/3), repr(1.0/3)) - - n = 10**100 - expected = repr(n)[:18] + "..." + repr(n)[-19:] - eq(r(n), expected) + for x in [123, 1.0 / 3]: + self.assertEqual(r(x), repr(x)) + + max_digits = sys.get_int_max_str_digits() + for k in [100, max_digits - 1]: + with self.subTest(f'10 ** {k}', k=k): + n = 10 ** k + expected = repr(n)[:18] + "..." + repr(n)[-19:] + self.assertEqual(r(n), expected) + + def re_msg(n, d): + return (rf'<{n.__class__.__name__} instance with roughly {d} ' + rf'digits \(limit at {max_digits}\) at 0x[a-f0-9]+>') + + k = max_digits + with self.subTest(f'10 ** {k}', k=k): + n = 10 ** k + self.assertRaises(ValueError, repr, n) + self.assertRegex(r(n), re_msg(n, k + 1)) + + for k in [max_digits + 1, 2 * max_digits]: + self.assertGreater(k, 100) + with self.subTest(f'10 ** {k}', k=k): + n = 10 ** k + self.assertRaises(ValueError, repr, n) + self.assertRegex(r(n), re_msg(n, k + 1)) + with self.subTest(f'10 ** {k} - 1', k=k): + n = 10 ** k - 1 + # Here, since math.log10(n) == math.log10(n-1), + # the number of digits of n - 1 is overestimated. + self.assertRaises(ValueError, repr, n) + self.assertRegex(r(n), re_msg(n, k + 1)) def test_instance(self): eq = self.assertEqual @@ -172,24 +198,23 @@ def test_instance(self): eq(r(i3), ("<ClassWithFailingRepr instance at %#x>"%id(i3))) s = r(ClassWithFailingRepr) - self.assertTrue(s.startswith("<class ")) - self.assertTrue(s.endswith(">")) + self.assertStartsWith(s, "<class ") + self.assertEndsWith(s, ">") self.assertIn(s.find("..."), [12, 13]) def test_lambda(self): r = repr(lambda x: x) - self.assertTrue(r.startswith("<function ReprTests.test_lambda.<locals>.<lambda"), r) + self.assertStartsWith(r, "<function ReprTests.test_lambda.<locals>.<lambda") # XXX anonymous functions? see func_repr - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_builtin_function(self): eq = self.assertEqual # Functions eq(repr(hash), '<built-in function hash>') # Methods - self.assertTrue(repr(''.split).startswith( - '<built-in method split of str object at 0x')) + self.assertStartsWith(repr(''.split), + '<built-in method split of str object at 0x') def test_range(self): eq = self.assertEqual @@ -214,8 +239,7 @@ def test_nesting(self): eq(r([[[[[[{}]]]]]]), "[[[[[[{}]]]]]]") eq(r([[[[[[[{}]]]]]]]), "[[[[[[[...]]]]]]]") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_cell(self): def get_cell(): x = 42 @@ -376,20 +400,20 @@ def test_valid_indent(self): 'object': { 1: 'two', b'three': [ - (4.5, 6.7), + (4.5, 6.25), [set((8, 9)), frozenset((10, 11))], ], }, 'tests': ( (dict(indent=None), '''\ - {1: 'two', b'three': [(4.5, 6.7), [{8, 9}, frozenset({10, 11})]]}'''), + {1: 'two', b'three': [(4.5, 6.25), [{8, 9}, frozenset({10, 11})]]}'''), (dict(indent=False), '''\ { 1: 'two', b'three': [ ( 4.5, - 6.7, + 6.25, ), [ { @@ -409,7 +433,7 @@ def test_valid_indent(self): b'three': [ ( 4.5, - 6.7, + 6.25, ), [ { @@ -429,7 +453,7 @@ def test_valid_indent(self): b'three': [ ( 4.5, - 6.7, + 6.25, ), [ { @@ -449,7 +473,7 @@ def test_valid_indent(self): b'three': [ ( 4.5, - 6.7, + 6.25, ), [ { @@ -469,7 +493,7 @@ def test_valid_indent(self): b'three': [ ( 4.5, - 6.7, + 6.25, ), [ { @@ -497,7 +521,7 @@ def test_valid_indent(self): b'three': [ ( 4.5, - 6.7, + 6.25, ), [ { @@ -517,7 +541,7 @@ def test_valid_indent(self): -->b'three': [ -->-->( -->-->-->4.5, - -->-->-->6.7, + -->-->-->6.25, -->-->), -->-->[ -->-->-->{ @@ -537,7 +561,7 @@ def test_valid_indent(self): ....b'three': [ ........( ............4.5, - ............6.7, + ............6.25, ........), ........[ ............{ @@ -733,8 +757,8 @@ class baz: importlib.invalidate_caches() from areallylongpackageandmodulenametotestreprtruncation.areallylongpackageandmodulenametotestreprtruncation import baz ibaz = baz.baz() - self.assertTrue(repr(ibaz).startswith( - "<%s.baz object at 0x" % baz.__name__)) + self.assertStartsWith(repr(ibaz), + "<%s.baz object at 0x" % baz.__name__) def test_method(self): self._check_path_limitations('qux') @@ -747,13 +771,13 @@ def amethod(self): pass from areallylongpackageandmodulenametotestreprtruncation.areallylongpackageandmodulenametotestreprtruncation import qux # Unbound methods first r = repr(qux.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.amethod) - self.assertTrue(r.startswith('<function aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.amethod'), r) + self.assertStartsWith(r, '<function aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.amethod') # Bound method next iqux = qux.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa() r = repr(iqux.amethod) - self.assertTrue(r.startswith( + self.assertStartsWith(r, '<bound method aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.amethod of <%s.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa object at 0x' \ - % (qux.__name__,) ), r) + % (qux.__name__,) ) @unittest.skip('needs a built-in function with a really long name') def test_builtin_function(self): @@ -822,8 +846,7 @@ def __repr__(self): self.assertIs(X.f, X.__repr__.__wrapped__) - # TODO: RUSTPYTHON: AttributeError: 'TypeVar' object has no attribute '__name__' - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'TypeVar' object has no attribute '__name__' def test__type_params__(self): class My: @recursive_repr() @@ -835,5 +858,19 @@ def __repr__[T: str](self, default: T = '') -> str: self.assertEqual(type_params[0].__name__, 'T') self.assertEqual(type_params[0].__bound__, str) + def test_annotations(self): + class My: + @recursive_repr() + def __repr__(self, default: undefined = ...): + return default + + annotations = annotationlib.get_annotations( + My.__repr__, format=annotationlib.Format.FORWARDREF + ) + self.assertEqual( + annotations, + {'default': EqualToForwardRef("undefined", owner=My.__repr__)} + ) + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_robotparser.py b/Lib/test/test_robotparser.py index b0bed431d4b..e33723cc70c 100644 --- a/Lib/test/test_robotparser.py +++ b/Lib/test/test_robotparser.py @@ -16,6 +16,14 @@ class BaseRobotTest: bad = [] site_maps = None + def __init_subclass__(cls): + super().__init_subclass__() + # Remove tests that do nothing. + if not cls.good: + cls.test_good_urls = None + if not cls.bad: + cls.test_bad_urls = None + def setUp(self): lines = io.StringIO(self.robots_txt).readlines() self.parser = urllib.robotparser.RobotFileParser() @@ -231,9 +239,16 @@ class DisallowQueryStringTest(BaseRobotTest, unittest.TestCase): robots_txt = """\ User-agent: * Disallow: /some/path?name=value +Disallow: /another/path? +Disallow: /yet/one/path?name=value&more """ - good = ['/some/path'] - bad = ['/some/path?name=value'] + good = ['/some/path', '/some/path?', + '/some/path%3Fname=value', '/some/path?name%3Dvalue', + '/another/path', '/another/path%3F', + '/yet/one/path?name=value%26more'] + bad = ['/some/path?name=value' + '/another/path?', '/another/path?name=value', + '/yet/one/path?name=value&more'] class UseFirstUserAgentWildcardTest(BaseRobotTest, unittest.TestCase): @@ -249,15 +264,79 @@ class UseFirstUserAgentWildcardTest(BaseRobotTest, unittest.TestCase): bad = ['/some/path'] -class EmptyQueryStringTest(BaseRobotTest, unittest.TestCase): - # normalize the URL first (#17403) +class PercentEncodingTest(BaseRobotTest, unittest.TestCase): robots_txt = """\ User-agent: * -Allow: /some/path? -Disallow: /another/path? - """ - good = ['/some/path?'] - bad = ['/another/path?'] +Disallow: /a1/Z-._~ # unreserved characters +Disallow: /a2/%5A%2D%2E%5F%7E # percent-encoded unreserved characters +Disallow: /u1/%F0%9F%90%8D # percent-encoded ASCII Unicode character +Disallow: /u2/%f0%9f%90%8d +Disallow: /u3/\U0001f40d # raw non-ASCII Unicode character +Disallow: /v1/%F0 # percent-encoded non-ASCII octet +Disallow: /v2/%f0 +Disallow: /v3/\udcf0 # raw non-ASCII octet +Disallow: /p1%xy # raw percent +Disallow: /p2% +Disallow: /p3%25xy # percent-encoded percent +Disallow: /p4%2525xy # double percent-encoded percent +Disallow: /john%20smith # space +Disallow: /john doe +Disallow: /trailingspace%20 +Disallow: /question%3Fq=v # not query +Disallow: /hash%23f # not fragment +Disallow: /dollar%24 +Disallow: /asterisk%2A +Disallow: /sub/dir +Disallow: /slash%2F +Disallow: /query/question?q=%3F +Disallow: /query/raw/question?q=? +Disallow: /query/eq?q%3Dv +Disallow: /query/amp?q=v%26a +""" + good = [ + '/u1/%F0', '/u1/%f0', + '/u2/%F0', '/u2/%f0', + '/u3/%F0', '/u3/%f0', + '/p1%2525xy', '/p2%f0', '/p3%2525xy', '/p4%xy', '/p4%25xy', + '/question?q=v', + '/dollar', '/asterisk', + '/query/eq?q=v', + '/query/amp?q=v&a', + ] + bad = [ + '/a1/Z-._~', '/a1/%5A%2D%2E%5F%7E', + '/a2/Z-._~', '/a2/%5A%2D%2E%5F%7E', + '/u1/%F0%9F%90%8D', '/u1/%f0%9f%90%8d', '/u1/\U0001f40d', + '/u2/%F0%9F%90%8D', '/u2/%f0%9f%90%8d', '/u2/\U0001f40d', + '/u3/%F0%9F%90%8D', '/u3/%f0%9f%90%8d', '/u3/\U0001f40d', + '/v1/%F0', '/v1/%f0', '/v1/\udcf0', '/v1/\U0001f40d', + '/v2/%F0', '/v2/%f0', '/v2/\udcf0', '/v2/\U0001f40d', + '/v3/%F0', '/v3/%f0', '/v3/\udcf0', '/v3/\U0001f40d', + '/p1%xy', '/p1%25xy', + '/p2%', '/p2%25', '/p2%2525', '/p2%xy', + '/p3%xy', '/p3%25xy', + '/p4%2525xy', + '/john%20smith', '/john smith', + '/john%20doe', '/john doe', + '/trailingspace%20', '/trailingspace ', + '/question%3Fq=v', + '/hash#f', '/hash%23f', + '/dollar$', '/dollar%24', + '/asterisk*', '/asterisk%2A', + '/sub/dir', '/sub%2Fdir', + '/slash%2F', '/slash/', + '/query/question?q=?', '/query/question?q=%3F', + '/query/raw/question?q=?', '/query/raw/question?q=%3F', + '/query/eq?q%3Dv', + '/query/amp?q=v%26a', + ] + # other reserved characters + for c in ":/#[]@!$&'()*+,;=": + robots_txt += f'Disallow: /raw{c}\nDisallow: /pc%{ord(c):02X}\n' + bad.append(f'/raw{c}') + bad.append(f'/raw%{ord(c):02X}') + bad.append(f'/pc{c}') + bad.append(f'/pc%{ord(c):02X}') class DefaultEntryTest(BaseRequestRateTest, unittest.TestCase): @@ -299,22 +378,17 @@ def test_string_formatting(self): self.assertEqual(str(self.parser), self.expected_output) -class RobotHandler(BaseHTTPRequestHandler): - - def do_GET(self): - self.send_error(403, "Forbidden access") - - def log_message(self, format, *args): - pass - - -class PasswordProtectedSiteTestCase(unittest.TestCase): +@unittest.skipUnless( + support.has_socket_support, + "Socket server requires working socket." +) +class BaseLocalNetworkTestCase: def setUp(self): # clear _opener global variable self.addCleanup(urllib.request.urlcleanup) - self.server = HTTPServer((socket_helper.HOST, 0), RobotHandler) + self.server = HTTPServer((socket_helper.HOST, 0), self.RobotHandler) self.t = threading.Thread( name='HTTPServer serving', @@ -331,6 +405,57 @@ def tearDown(self): self.t.join() self.server.server_close() + +SAMPLE_ROBOTS_TXT = b'''\ +User-agent: test_robotparser +Disallow: /utf8/\xf0\x9f\x90\x8d +Disallow: /non-utf8/\xf0 +Disallow: //[spam]/path +''' + + +class LocalNetworkTestCase(BaseLocalNetworkTestCase, unittest.TestCase): + class RobotHandler(BaseHTTPRequestHandler): + + def do_GET(self): + self.send_response(200) + self.end_headers() + self.wfile.write(SAMPLE_ROBOTS_TXT) + + def log_message(self, format, *args): + pass + + @threading_helper.reap_threads + def testRead(self): + # Test that reading a weird robots.txt doesn't fail. + addr = self.server.server_address + url = f'http://{socket_helper.HOST}:{addr[1]}' + robots_url = url + '/robots.txt' + parser = urllib.robotparser.RobotFileParser() + parser.set_url(robots_url) + parser.read() + # And it can even interpret the weird paths in some reasonable way. + agent = 'test_robotparser' + self.assertTrue(parser.can_fetch(agent, robots_url)) + self.assertTrue(parser.can_fetch(agent, url + '/utf8/')) + self.assertFalse(parser.can_fetch(agent, url + '/utf8/\U0001f40d')) + self.assertFalse(parser.can_fetch(agent, url + '/utf8/%F0%9F%90%8D')) + self.assertFalse(parser.can_fetch(agent, url + '/utf8/\U0001f40d')) + self.assertTrue(parser.can_fetch(agent, url + '/non-utf8/')) + self.assertFalse(parser.can_fetch(agent, url + '/non-utf8/%F0')) + self.assertFalse(parser.can_fetch(agent, url + '/non-utf8/\U0001f40d')) + self.assertFalse(parser.can_fetch(agent, url + '/%2F[spam]/path')) + + +class PasswordProtectedSiteTestCase(BaseLocalNetworkTestCase, unittest.TestCase): + class RobotHandler(BaseHTTPRequestHandler): + + def do_GET(self): + self.send_error(403, "Forbidden access") + + def log_message(self, format, *args): + pass + @threading_helper.reap_threads def testPasswordProtectedSite(self): addr = self.server.server_address @@ -342,6 +467,7 @@ def testPasswordProtectedSite(self): self.assertFalse(parser.can_fetch("*", robots_url)) +@support.requires_working_socket() class NetworkTestCase(unittest.TestCase): base_url = 'http://www.pythontest.net/' diff --git a/Lib/test/test_runpy.py b/Lib/test/test_runpy.py index c1e255e7af3..56e50391c73 100644 --- a/Lib/test/test_runpy.py +++ b/Lib/test/test_runpy.py @@ -798,7 +798,7 @@ def assertSigInt(self, cmd, *args, **kwargs): # Use -E to ignore PYTHONSAFEPATH cmd = [sys.executable, '-E', *cmd] proc = subprocess.run(cmd, *args, **kwargs, text=True, stderr=subprocess.PIPE) - self.assertTrue(proc.stderr.endswith("\nKeyboardInterrupt\n"), proc.stderr) + self.assertEndsWith(proc.stderr, "\nKeyboardInterrupt\n") self.assertEqual(proc.returncode, self.EXPECTED_CODE) def test_pymain_run_file(self): diff --git a/Lib/test/test_select.py b/Lib/test/test_select.py new file mode 100644 index 00000000000..6ce8cd423f7 --- /dev/null +++ b/Lib/test/test_select.py @@ -0,0 +1,107 @@ +import errno +import select +import subprocess +import sys +import textwrap +import unittest +from test import support + +support.requires_working_socket(module=True) + +@unittest.skipIf((sys.platform[:3]=='win'), + "can't easily test on this system") +class SelectTestCase(unittest.TestCase): + + class Nope: + pass + + class Almost: + def fileno(self): + return 'fileno' + + def test_error_conditions(self): + self.assertRaises(TypeError, select.select, 1, 2, 3) + self.assertRaises(TypeError, select.select, [self.Nope()], [], []) + self.assertRaises(TypeError, select.select, [self.Almost()], [], []) + self.assertRaises(TypeError, select.select, [], [], [], "not a number") + self.assertRaises(ValueError, select.select, [], [], [], -1) + + # Issue #12367: http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/155606 + @unittest.skipIf(sys.platform.startswith('freebsd'), + 'skip because of a FreeBSD bug: kern/155606') + def test_errno(self): + with open(__file__, 'rb') as fp: + fd = fp.fileno() + fp.close() + try: + select.select([fd], [], [], 0) + except OSError as err: + self.assertEqual(err.errno, errno.EBADF) + else: + self.fail("exception not raised") + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: unexpectedly identical: [] + def test_returned_list_identity(self): + # See issue #8329 + r, w, x = select.select([], [], [], 1) + self.assertIsNot(r, w) + self.assertIsNot(r, x) + self.assertIsNot(w, x) + + @support.requires_fork() + def test_select(self): + code = textwrap.dedent(''' + import time + for i in range(10): + print("testing...", flush=True) + time.sleep(0.050) + ''') + cmd = [sys.executable, '-I', '-c', code] + with subprocess.Popen(cmd, stdout=subprocess.PIPE) as proc: + pipe = proc.stdout + for timeout in (0, 1, 2, 4, 8, 16) + (None,)*10: + if support.verbose: + print(f'timeout = {timeout}') + rfd, wfd, xfd = select.select([pipe], [], [], timeout) + self.assertEqual(wfd, []) + self.assertEqual(xfd, []) + if not rfd: + continue + if rfd == [pipe]: + line = pipe.readline() + if support.verbose: + print(repr(line)) + if not line: + if support.verbose: + print('EOF') + break + continue + self.fail('Unexpected return values from select():', + rfd, wfd, xfd) + + # Issue 16230: Crash on select resized list + @unittest.skipIf( + support.is_emscripten, "Emscripten cannot select a fd multiple times." + ) + @unittest.skip("TODO: RUSTPYTHON timed out") + def test_select_mutated(self): + a = [] + class F: + def fileno(self): + del a[-1] + return sys.__stdout__.fileno() + a[:] = [F()] * 10 + self.assertEqual(select.select([], a, []), ([], a[:5], [])) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: TypeError not raised by poll + def test_disallow_instantiation(self): + support.check_disallow_instantiation(self, type(select.poll())) + + if hasattr(select, 'devpoll'): + support.check_disallow_instantiation(self, type(select.devpoll())) + +def tearDownModule(): + support.reap_children() + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_shlex.py b/Lib/test/test_shlex.py index baabccf19f4..7c41432b82f 100644 --- a/Lib/test/test_shlex.py +++ b/Lib/test/test_shlex.py @@ -3,6 +3,8 @@ import shlex import string import unittest +from test.support import cpython_only +from test.support import import_helper # The original test data set was from shellwords, by Hartmut Goebel. @@ -165,14 +167,12 @@ def testSplitNone(self): with self.assertRaises(ValueError): shlex.split(None) - # TODO: RUSTPYTHON; ValueError: Error Retrieving Value - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: Error Retrieving Value def testSplitPosix(self): """Test data splitting with posix parser""" self.splitTest(self.posix_data, comments=True) - # TODO: RUSTPYTHON; ValueError: Error Retrieving Value - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: Error Retrieving Value def testCompat(self): """Test compatibility interface""" for i in range(len(self.data)): @@ -313,8 +313,7 @@ def testEmptyStringHandling(self): s = shlex.shlex("'')abc", punctuation_chars=True) self.assertEqual(list(s), expected) - # TODO: RUSTPYTHON; ValueError: Error Retrieving Value - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: Error Retrieving Value def testUnicodeHandling(self): """Test punctuation_chars and whitespace_split handle unicode.""" ss = "\u2119\u01b4\u2602\u210c\u00f8\u1f24" @@ -334,6 +333,7 @@ def testQuote(self): unsafe = '"`$\\!' + unicode_sample self.assertEqual(shlex.quote(''), "''") + self.assertEqual(shlex.quote(None), "''") self.assertEqual(shlex.quote(safeunquoted), safeunquoted) self.assertEqual(shlex.quote('test file name'), "'test file name'") for u in unsafe: @@ -342,6 +342,8 @@ def testQuote(self): for u in unsafe: self.assertEqual(shlex.quote("test%s'name'" % u), "'test%s'\"'\"'name'\"'\"''" % u) + self.assertRaises(TypeError, shlex.quote, 42) + self.assertRaises(TypeError, shlex.quote, b"abc") def testJoin(self): for split_command, command in [ @@ -354,8 +356,7 @@ def testJoin(self): joined = shlex.join(split_command) self.assertEqual(joined, command) - # TODO: RUSTPYTHON; ValueError: Error Retrieving Value - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: Error Retrieving Value def testJoinRoundtrip(self): all_data = self.data + self.posix_data for command, *split_command in all_data: @@ -371,6 +372,10 @@ def testPunctuationCharsReadOnly(self): with self.assertRaises(AttributeError): shlex_instance.punctuation_chars = False + @cpython_only + def test_lazy_imports(self): + import_helper.ensure_lazy_imports('shlex', {'collections', 're', 'os'}) + # Allow this test to be used with old shlex.py if not getattr(shlex, "split", None): diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py index 73ad81e871b..0e973255621 100644 --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -194,7 +194,6 @@ def test_rmtree_works_on_bytes(self): self.assertIsInstance(victim, bytes) shutil.rmtree(victim) - @unittest.skipIf(sys.platform == 'win32', 'TODO: RUSTPYTHON; flaky') @os_helper.skip_unless_symlink def test_rmtree_fails_on_symlink_onerror(self): tmp = self.mkdtemp() diff --git a/Lib/test/test_signal.py b/Lib/test/test_signal.py index 578bee3172a..5bb7cb5df31 100644 --- a/Lib/test/test_signal.py +++ b/Lib/test/test_signal.py @@ -192,7 +192,7 @@ def test_valid_signals(self): self.assertNotIn(signal.NSIG, s) self.assertLess(len(s), signal.NSIG) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_issue9324(self): # Updated for issue #10003, adding SIGBREAK handler = lambda x, y: None @@ -254,9 +254,6 @@ def test_invalid_socket(self): self.assertRaises((ValueError, OSError), signal.set_wakeup_fd, fd) - # Emscripten does not support fstat on pipes yet. - # https://github.com/emscripten-core/emscripten/issues/16414 - @unittest.skipIf(support.is_emscripten, "Emscripten cannot fstat pipes.") @unittest.skipUnless(hasattr(os, "pipe"), "requires os.pipe()") def test_set_wakeup_fd_result(self): r1, w1 = os.pipe() @@ -275,7 +272,6 @@ def test_set_wakeup_fd_result(self): self.assertEqual(signal.set_wakeup_fd(-1), w2) self.assertEqual(signal.set_wakeup_fd(-1), -1) - @unittest.skipIf(support.is_emscripten, "Emscripten cannot fstat pipes.") @unittest.skipUnless(support.has_socket_support, "needs working sockets.") def test_set_wakeup_fd_socket_result(self): sock1 = socket.socket() @@ -296,7 +292,6 @@ def test_set_wakeup_fd_socket_result(self): # On Windows, files are always blocking and Windows does not provide a # function to test if a socket is in non-blocking mode. @unittest.skipIf(sys.platform == "win32", "tests specific to POSIX") - @unittest.skipIf(support.is_emscripten, "Emscripten cannot fstat pipes.") @unittest.skipUnless(hasattr(os, "pipe"), "requires os.pipe()") def test_set_wakeup_fd_blocking(self): rfd, wfd = os.pipe() @@ -386,7 +381,7 @@ def handler(signum, frame): except ZeroDivisionError: # An ignored exception should have been printed out on stderr err = err.getvalue() - if ('Exception ignored when trying to write to the signal wakeup fd' + if ('Exception ignored while trying to write to the signal wakeup fd' not in err): raise AssertionError(err) if ('OSError: [Errno %d]' % errno.EBADF) not in err: @@ -575,7 +570,7 @@ def handler(signum, frame): signal.raise_signal(signum) err = err.getvalue() - if ('Exception ignored when trying to {action} to the signal wakeup fd' + if ('Exception ignored while trying to {action} to the signal wakeup fd' not in err): raise AssertionError(err) """.format(action=action) @@ -645,7 +640,7 @@ def handler(signum, frame): "buffer" % written) # By default, we get a warning when a signal arrives - msg = ('Exception ignored when trying to {action} ' + msg = ('Exception ignored while trying to {action} ' 'to the signal wakeup fd') signal.set_wakeup_fd(write.fileno()) @@ -1351,7 +1346,7 @@ def handler(signum, frame): # Python handler self.assertEqual(len(sigs), N, "Some signals were lost") - @unittest.skip('TODO: RUSTPYTHON; hang') + @support.requires_gil_enabled("gh-121065: test is flaky on free-threaded build") @unittest.skipIf(is_apple, "crashes due to system bug (FB13453490)") @unittest.skipUnless(hasattr(signal, "SIGUSR1"), "test needs SIGUSR1") @@ -1419,7 +1414,7 @@ def test_sigint(self): with self.assertRaises(KeyboardInterrupt): signal.raise_signal(signal.SIGINT) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON @unittest.skipIf(sys.platform != "win32", "Windows specific test") def test_invalid_argument(self): try: @@ -1443,7 +1438,7 @@ def handler(a, b): signal.raise_signal(signal.SIGINT) self.assertTrue(is_ok) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test__thread_interrupt_main(self): # See https://github.com/python/cpython/issues/102397 code = """if 1: diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py index df279bd9652..56ed457882c 100644 --- a/Lib/test/test_site.py +++ b/Lib/test/test_site.py @@ -8,6 +8,7 @@ import test.support from test import support from test.support.script_helper import assert_python_ok +from test.support import import_helper from test.support import os_helper from test.support import socket_helper from test.support import captured_stderr @@ -308,8 +309,7 @@ def test_getuserbase(self): with EnvironmentVarGuard() as environ: environ['PYTHONUSERBASE'] = 'xoxo' - self.assertTrue(site.getuserbase().startswith('xoxo'), - site.getuserbase()) + self.assertTrue(site.getuserbase().startswith('xoxo')) @unittest.skipUnless(HAS_USER_SITE, 'need user site') def test_getusersitepackages(self): @@ -319,7 +319,7 @@ def test_getusersitepackages(self): # the call sets USER_BASE *and* USER_SITE self.assertEqual(site.USER_SITE, user_site) - self.assertTrue(user_site.startswith(site.USER_BASE), user_site) + self.assertTrue(user_site.startswith(site.USER_BASE)) self.assertEqual(site.USER_BASE, site.getuserbase()) def test_getsitepackages(self): @@ -362,11 +362,10 @@ def test_no_home_directory(self): environ.unset('PYTHONUSERBASE', 'APPDATA') user_base = site.getuserbase() - self.assertTrue(user_base.startswith('~' + os.sep), - user_base) + self.assertTrue(user_base.startswith('~' + os.sep)) user_site = site.getusersitepackages() - self.assertTrue(user_site.startswith(user_base), user_site) + self.assertTrue(user_site.startswith(user_base)) with mock.patch('os.path.isdir', return_value=False) as mock_isdir, \ mock.patch.object(site, 'addsitedir') as mock_addsitedir, \ @@ -515,7 +514,7 @@ def test_sitecustomize_executed(self): # If sitecustomize is available, it should have been imported. if "sitecustomize" not in sys.modules: try: - import sitecustomize + import sitecustomize # noqa: F401 except ImportError: pass else: @@ -578,6 +577,17 @@ def test_license_exists_at_url(self): code = e.code self.assertEqual(code, 200, msg="Can't find " + url) + @support.cpython_only + def test_lazy_imports(self): + import_helper.ensure_lazy_imports("site", [ + "io", + "locale", + "traceback", + "atexit", + "warnings", + "textwrap", + ]) + class StartupImportTests(unittest.TestCase): @@ -843,12 +853,15 @@ def get_excepted_output(self, *args): return 10, None def invoke_command_line(self, *args): - args = ["-m", "site", *args] + cmd_args = [] + if sys.flags.no_user_site: + cmd_args.append("-s") + cmd_args.extend(["-m", "site", *args]) with EnvironmentVarGuard() as env: env["PYTHONUTF8"] = "1" env["PYTHONIOENCODING"] = "utf-8" - proc = spawn_python(*args, text=True, env=env, + proc = spawn_python(*cmd_args, text=True, env=env, encoding='utf-8', errors='replace') output = kill_python(proc) diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py index 08d3dcb5792..e792d4f30a9 100644 --- a/Lib/test/test_socket.py +++ b/Lib/test/test_socket.py @@ -2,8 +2,9 @@ from unittest import mock from test import support from test.support import ( - is_apple, os_helper, refleak_helper, socket_helper, threading_helper + cpython_only, is_apple, os_helper, refleak_helper, socket_helper, threading_helper ) +from test.support.import_helper import ensure_lazy_imports import _thread as thread import array import contextlib @@ -49,9 +50,9 @@ # test unicode string and carriage return MSG = 'Michael Gilfix was here\u1234\r\n'.encode('utf-8') -VMADDR_CID_LOCAL = 1 VSOCKPORT = 1234 AIX = platform.system() == "AIX" +SOLARIS = sys.platform.startswith("sunos") WSL = "microsoft-standard-WSL" in platform.release() try: @@ -258,6 +259,12 @@ def downgrade_malformed_data_warning(): # Size in bytes of the int type SIZEOF_INT = array.array("i").itemsize +class TestLazyImport(unittest.TestCase): + @cpython_only + def test_lazy_import(self): + ensure_lazy_imports("socket", {"array", "selectors"}) + + class SocketTCPTest(unittest.TestCase): def setUp(self): @@ -579,7 +586,7 @@ def clientSetUp(self): cid = get_cid() if cid in (socket.VMADDR_CID_HOST, socket.VMADDR_CID_ANY): # gh-119461: Use the local communication address (loopback) - cid = VMADDR_CID_LOCAL + cid = socket.VMADDR_CID_LOCAL self.cli.connect((cid, VSOCKPORT)) def testStream(self): @@ -904,9 +911,8 @@ def requireSocket(*args): class GeneralModuleTests(unittest.TestCase): + @unittest.expectedFailure # TODO: RUSTPYTHON; gc.is_tracked not implemented @unittest.skipUnless(_socket is not None, 'need _socket module') - # TODO: RUSTPYTHON; gc.is_tracked not implemented - @unittest.expectedFailure def test_socket_type(self): self.assertTrue(gc.is_tracked(_socket.socket)) with self.assertRaisesRegex(TypeError, "immutable"): @@ -969,8 +975,7 @@ def testSocketError(self): with self.assertRaises(OSError, msg=msg % 'socket.gaierror'): raise socket.gaierror - # TODO: RUSTPYTHON; error message format differs - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; error message format differs def testSendtoErrors(self): # Testing that sendto doesn't mask failures. See #10169. s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) @@ -1083,9 +1088,7 @@ def test3542SocketOptions(self): 'IPV6_USE_MIN_MTU', } for opt in opts: - self.assertTrue( - hasattr(socket, opt), f"Missing RFC3542 socket option '{opt}'" - ) + self.assertHasAttr(socket, opt) def testHostnameRes(self): # Testing hostname resolution mechanisms @@ -1154,6 +1157,7 @@ def test_sethostname(self): @unittest.skipUnless(hasattr(socket, 'if_nameindex'), 'socket.if_nameindex() not available.') + @support.skip_android_selinux('if_nameindex') def testInterfaceNameIndex(self): interfaces = socket.if_nameindex() for index, name in interfaces: @@ -1168,14 +1172,16 @@ def testInterfaceNameIndex(self): self.assertIsInstance(_name, str) self.assertEqual(name, _name) + @unittest.expectedFailure # TODO: RUSTPYTHON; OverflowError: Python int too large to convert to Rust u32 @unittest.skipUnless(hasattr(socket, 'if_indextoname'), 'socket.if_indextoname() not available.') + @support.skip_android_selinux('if_indextoname') def testInvalidInterfaceIndexToName(self): with self.assertRaises(OSError) as cm: socket.if_indextoname(0) self.assertIsNotNone(cm.exception.errno) - self.assertRaises(OverflowError, socket.if_indextoname, -1) + self.assertRaises(ValueError, socket.if_indextoname, -1) self.assertRaises(OverflowError, socket.if_indextoname, 2**1000) self.assertRaises(TypeError, socket.if_indextoname, '_DEADBEEF') if hasattr(socket, 'if_nameindex'): @@ -1192,6 +1198,7 @@ def testInvalidInterfaceIndexToName(self): @unittest.skipUnless(hasattr(socket, 'if_nametoindex'), 'socket.if_nametoindex() not available.') + @support.skip_android_selinux('if_nametoindex') def testInvalidInterfaceNameToIndex(self): with self.assertRaises(OSError) as cm: socket.if_nametoindex("_DEADBEEF") @@ -1233,24 +1240,24 @@ def testNtoH(self): self.assertEqual(swapped & mask, mask) self.assertRaises(OverflowError, func, 1<<34) - @support.cpython_only - @unittest.skipIf(_testcapi is None, "requires _testcapi") + @unittest.expectedFailure # TODO: RUSTPYTHON; OverflowError: Python int too large to convert to Rust u16 def testNtoHErrors(self): - import _testcapi s_good_values = [0, 1, 2, 0xffff] l_good_values = s_good_values + [0xffffffff] - l_bad_values = [-1, -2, 1<<32, 1<<1000] - s_bad_values = ( - l_bad_values + - [_testcapi.INT_MIN-1, _testcapi.INT_MAX+1] + - [1 << 16, _testcapi.INT_MAX] - ) + neg_values = [-1, -2, -(1<<15)-1, -(1<<31)-1, -(1<<63)-1, -1<<1000] + l_bad_values = [1<<32, 1<<1000] + s_bad_values = l_bad_values + [1 << 16, (1<<31)-1, 1<<31] for k in s_good_values: socket.ntohs(k) socket.htons(k) for k in l_good_values: socket.ntohl(k) socket.htonl(k) + for k in neg_values: + self.assertRaises(ValueError, socket.ntohs, k) + self.assertRaises(ValueError, socket.htons, k) + self.assertRaises(ValueError, socket.ntohl, k) + self.assertRaises(ValueError, socket.htonl, k) for k in s_bad_values: self.assertRaises(OverflowError, socket.ntohs, k) self.assertRaises(OverflowError, socket.htons, k) @@ -1595,11 +1602,11 @@ def test_getsockaddrarg(self): @unittest.skipUnless(os.name == "nt", "Windows specific") def test_sock_ioctl(self): - self.assertTrue(hasattr(socket.socket, 'ioctl')) - self.assertTrue(hasattr(socket, 'SIO_RCVALL')) - self.assertTrue(hasattr(socket, 'RCVALL_ON')) - self.assertTrue(hasattr(socket, 'RCVALL_OFF')) - self.assertTrue(hasattr(socket, 'SIO_KEEPALIVE_VALS')) + self.assertHasAttr(socket.socket, 'ioctl') + self.assertHasAttr(socket, 'SIO_RCVALL') + self.assertHasAttr(socket, 'RCVALL_ON') + self.assertHasAttr(socket, 'RCVALL_OFF') + self.assertHasAttr(socket, 'SIO_KEEPALIVE_VALS') s = socket.socket() self.addCleanup(s.close) self.assertRaises(ValueError, s.ioctl, -1, None) @@ -1690,8 +1697,11 @@ def testGetaddrinfo(self): # Issue #6697. self.assertRaises(UnicodeEncodeError, socket.getaddrinfo, 'localhost', '\uD800') - # Issue 17269: test workaround for OS X platform bug segfault if hasattr(socket, 'AI_NUMERICSERV'): + self.assertRaises(socket.gaierror, socket.getaddrinfo, "localhost", "http", + flags=socket.AI_NUMERICSERV) + + # Issue 17269: test workaround for OS X platform bug segfault try: # The arguments here are undefined and the call may succeed # or fail. All we care here is that it doesn't segfault. @@ -1927,6 +1937,7 @@ def test_getfqdn_filter_localhost(self): @unittest.skipIf(sys.platform == 'win32', 'does not work on Windows') @unittest.skipIf(AIX, 'Symbolic scope id does not work') @unittest.skipUnless(hasattr(socket, 'if_nameindex'), "test needs socket.if_nameindex()") + @support.skip_android_selinux('if_nameindex') def test_getaddrinfo_ipv6_scopeid_symbolic(self): # Just pick up any network interface (Linux, Mac OS X) (ifindex, test_interface) = socket.if_nameindex()[0] @@ -1960,6 +1971,7 @@ def test_getaddrinfo_ipv6_scopeid_numeric(self): @unittest.skipIf(sys.platform == 'win32', 'does not work on Windows') @unittest.skipIf(AIX, 'Symbolic scope id does not work') @unittest.skipUnless(hasattr(socket, 'if_nameindex'), "test needs socket.if_nameindex()") + @support.skip_android_selinux('if_nameindex') def test_getnameinfo_ipv6_scopeid_symbolic(self): # Just pick up any network interface. (ifindex, test_interface) = socket.if_nameindex()[0] @@ -2367,12 +2379,14 @@ def testCreateISOTPSocket(self): with socket.socket(socket.PF_CAN, socket.SOCK_DGRAM, socket.CAN_ISOTP) as s: pass + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: AF_CAN address must be a tuple (interface,) or (interface, addr) def testTooLongInterfaceName(self): # most systems limit IFNAMSIZ to 16, take 1024 to be sure with socket.socket(socket.PF_CAN, socket.SOCK_DGRAM, socket.CAN_ISOTP) as s: with self.assertRaisesRegex(OSError, 'interface name too long'): s.bind(('x' * 1024, 1, 2)) + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: AF_CAN address must be a tuple (interface,) or (interface, addr) def testBind(self): try: with socket.socket(socket.PF_CAN, socket.SOCK_DGRAM, socket.CAN_ISOTP) as s: @@ -2394,8 +2408,7 @@ def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.interface = "vcan0" - # TODO: RUSTPYTHON - J1939 constants not fully implemented - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; - J1939 constants not fully implemented @unittest.skipUnless(hasattr(socket, "CAN_J1939"), 'socket.CAN_J1939 required for this test.') def testJ1939Constants(self): @@ -2437,8 +2450,7 @@ def testCreateJ1939Socket(self): with socket.socket(socket.PF_CAN, socket.SOCK_DGRAM, socket.CAN_J1939) as s: pass - # TODO: RUSTPYTHON - AF_CAN J1939 address format not fully implemented - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; - AF_CAN J1939 address format not fully implemented def testBind(self): try: with socket.socket(socket.PF_CAN, socket.SOCK_DGRAM, socket.CAN_J1939) as s: @@ -2577,6 +2589,7 @@ def testVSOCKConstants(self): socket.SO_VM_SOCKETS_BUFFER_MAX_SIZE socket.VMADDR_CID_ANY socket.VMADDR_PORT_ANY + socket.VMADDR_CID_LOCAL socket.VMADDR_CID_HOST socket.VM_SOCKETS_INVALID_VERSION socket.IOCTL_VM_SOCKETS_GET_LOCAL_CID @@ -2612,23 +2625,88 @@ def testSocketBufferSize(self): socket.SO_VM_SOCKETS_BUFFER_MIN_SIZE)) -@unittest.skipUnless(HAVE_SOCKET_BLUETOOTH, +@unittest.skipUnless(hasattr(socket, 'AF_BLUETOOTH'), 'Bluetooth sockets required for this test.') class BasicBluetoothTest(unittest.TestCase): + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'socket' has no attribute 'BTPROTO_RFCOMM' def testBluetoothConstants(self): socket.BDADDR_ANY socket.BDADDR_LOCAL socket.AF_BLUETOOTH socket.BTPROTO_RFCOMM + socket.SOL_RFCOMM + + if sys.platform == "win32": + socket.SO_BTH_ENCRYPT + socket.SO_BTH_MTU + socket.SO_BTH_MTU_MAX + socket.SO_BTH_MTU_MIN if sys.platform != "win32": socket.BTPROTO_HCI socket.SOL_HCI socket.BTPROTO_L2CAP + socket.SOL_L2CAP + socket.BTPROTO_SCO + socket.SOL_SCO + socket.HCI_DATA_DIR + + if sys.platform == "linux": + socket.SOL_BLUETOOTH + socket.HCI_DEV_NONE + socket.HCI_CHANNEL_RAW + socket.HCI_CHANNEL_USER + socket.HCI_CHANNEL_MONITOR + socket.HCI_CHANNEL_CONTROL + socket.HCI_CHANNEL_LOGGING + socket.HCI_TIME_STAMP + socket.BT_SECURITY + socket.BT_SECURITY_SDP + socket.BT_FLUSHABLE + socket.BT_POWER + socket.BT_CHANNEL_POLICY + socket.BT_CHANNEL_POLICY_BREDR_ONLY + if hasattr(socket, 'BT_PHY'): + socket.BT_PHY_BR_1M_1SLOT + if hasattr(socket, 'BT_MODE'): + socket.BT_MODE_BASIC + if hasattr(socket, 'BT_VOICE'): + socket.BT_VOICE_TRANSPARENT + socket.BT_VOICE_CVSD_16BIT + socket.L2CAP_LM + socket.L2CAP_LM_MASTER + socket.L2CAP_LM_AUTH + + if sys.platform in ("linux", "freebsd"): + socket.BDADDR_BREDR + socket.BDADDR_LE_PUBLIC + socket.BDADDR_LE_RANDOM + socket.HCI_FILTER + + if sys.platform.startswith(("freebsd", "netbsd", "dragonfly")): + socket.SO_L2CAP_IMTU + socket.SO_L2CAP_FLUSH + socket.SO_RFCOMM_MTU + socket.SO_RFCOMM_FC_INFO + socket.SO_SCO_MTU + + if sys.platform == "freebsd": + socket.SO_SCO_CONNINFO + + if sys.platform.startswith(("netbsd", "dragonfly")): + socket.SO_HCI_EVT_FILTER + socket.SO_HCI_PKT_FILTER + socket.SO_L2CAP_IQOS + socket.SO_L2CAP_LM + socket.L2CAP_LM_AUTH + socket.SO_RFCOMM_LM + socket.RFCOMM_LM_AUTH + socket.SO_SCO_HANDLE - if not sys.platform.startswith("freebsd"): - socket.BTPROTO_SCO +@unittest.skipUnless(HAVE_SOCKET_BLUETOOTH, + 'Bluetooth sockets required for this test.') +class BluetoothTest(unittest.TestCase): def testCreateRfcommSocket(self): with socket.socket(socket.AF_BLUETOOTH, socket.SOCK_STREAM, socket.BTPROTO_RFCOMM) as s: @@ -2644,12 +2722,31 @@ def testCreateHciSocket(self): with socket.socket(socket.AF_BLUETOOTH, socket.SOCK_RAW, socket.BTPROTO_HCI) as s: pass - @unittest.skipIf(sys.platform == "win32" or sys.platform.startswith("freebsd"), - "windows and freebsd do not support SCO sockets") + @unittest.skipIf(sys.platform == "win32", "windows does not support SCO sockets") def testCreateScoSocket(self): with socket.socket(socket.AF_BLUETOOTH, socket.SOCK_SEQPACKET, socket.BTPROTO_SCO) as s: pass + @unittest.skipUnless(HAVE_SOCKET_BLUETOOTH_L2CAP, 'Bluetooth L2CAP sockets required for this test') + def testBindLeAttL2capSocket(self): + BDADDR_LE_PUBLIC = support.get_attribute(socket, 'BDADDR_LE_PUBLIC') + with socket.socket(socket.AF_BLUETOOTH, socket.SOCK_SEQPACKET, socket.BTPROTO_L2CAP) as f: + # ATT is the only CID allowed in userspace by the Linux kernel + CID_ATT = 4 + f.bind((socket.BDADDR_ANY, 0, CID_ATT, BDADDR_LE_PUBLIC)) + addr = f.getsockname() + self.assertEqual(addr, (socket.BDADDR_ANY, 0, CID_ATT, BDADDR_LE_PUBLIC)) + + @unittest.skipUnless(HAVE_SOCKET_BLUETOOTH_L2CAP, 'Bluetooth L2CAP sockets required for this test') + def testBindLePsmL2capSocket(self): + BDADDR_LE_RANDOM = support.get_attribute(socket, 'BDADDR_LE_RANDOM') + with socket.socket(socket.AF_BLUETOOTH, socket.SOCK_SEQPACKET, socket.BTPROTO_L2CAP) as f: + # First user PSM in LE L2CAP + psm = 0x80 + f.bind((socket.BDADDR_ANY, psm, 0, BDADDR_LE_RANDOM)) + addr = f.getsockname() + self.assertEqual(addr, (socket.BDADDR_ANY, psm, 0, BDADDR_LE_RANDOM)) + @unittest.skipUnless(HAVE_SOCKET_BLUETOOTH_L2CAP, 'Bluetooth L2CAP sockets required for this test') def testBindBrEdrL2capSocket(self): with socket.socket(socket.AF_BLUETOOTH, socket.SOCK_SEQPACKET, socket.BTPROTO_L2CAP) as f: @@ -2663,7 +2760,7 @@ def testBindBrEdrL2capSocket(self): def testBadL2capAddr(self): with socket.socket(socket.AF_BLUETOOTH, socket.SOCK_SEQPACKET, socket.BTPROTO_L2CAP) as f: with self.assertRaises(OSError): - f.bind((socket.BDADDR_ANY, 0, 0)) + f.bind((socket.BDADDR_ANY, 0, 0, 0, 0)) with self.assertRaises(OSError): f.bind((socket.BDADDR_ANY,)) with self.assertRaises(OSError): @@ -2710,13 +2807,14 @@ def testBadRfcommAddr(self): @unittest.skipUnless(hasattr(socket, 'BTPROTO_HCI'), 'Bluetooth HCI sockets required for this test') def testBindHciSocket(self): - with socket.socket(socket.AF_BLUETOOTH, socket.SOCK_RAW, socket.BTPROTO_HCI) as s: - if sys.platform.startswith(('netbsd', 'dragonfly', 'freebsd')): + if sys.platform.startswith(('netbsd', 'dragonfly', 'freebsd')): + with socket.socket(socket.AF_BLUETOOTH, socket.SOCK_RAW, socket.BTPROTO_HCI) as s: s.bind(socket.BDADDR_ANY) addr = s.getsockname() self.assertEqual(addr, socket.BDADDR_ANY) - else: - dev = 0 + else: + dev = 0 + with socket.socket(socket.AF_BLUETOOTH, socket.SOCK_RAW, socket.BTPROTO_HCI) as s: try: s.bind((dev,)) except OSError as err: @@ -2726,6 +2824,32 @@ def testBindHciSocket(self): addr = s.getsockname() self.assertEqual(addr, dev) + with (self.subTest('integer'), + socket.socket(socket.AF_BLUETOOTH, socket.SOCK_RAW, socket.BTPROTO_HCI) as s): + s.bind(dev) + addr = s.getsockname() + self.assertEqual(addr, dev) + + with (self.subTest('channel=HCI_CHANNEL_RAW'), + socket.socket(socket.AF_BLUETOOTH, socket.SOCK_RAW, socket.BTPROTO_HCI) as s): + channel = socket.HCI_CHANNEL_RAW + s.bind((dev, channel)) + addr = s.getsockname() + self.assertEqual(addr, dev) + + with (self.subTest('channel=HCI_CHANNEL_USER'), + socket.socket(socket.AF_BLUETOOTH, socket.SOCK_RAW, socket.BTPROTO_HCI) as s): + channel = socket.HCI_CHANNEL_USER + try: + s.bind((dev, channel)) + except OSError as err: + # Needs special permissions. + if err.errno in (errno.EPERM, errno.EBUSY, errno.ERFKILL): + self.skipTest(str(err)) + raise + addr = s.getsockname() + self.assertEqual(addr, (dev, channel)) + @unittest.skipUnless(hasattr(socket, 'BTPROTO_HCI'), 'Bluetooth HCI sockets required for this test') def testBadHciAddr(self): with socket.socket(socket.AF_BLUETOOTH, socket.SOCK_RAW, socket.BTPROTO_HCI) as s: @@ -2749,9 +2873,7 @@ def testBadHciAddr(self): with self.assertRaises(OSError): s.bind(()) with self.assertRaises(OSError): - s.bind((dev, 0)) - with self.assertRaises(OSError): - s.bind(dev) + s.bind((dev, socket.HCI_CHANNEL_RAW, 0, 0)) with self.assertRaises(OSError): s.bind(socket.BDADDR_ANY) with self.assertRaises(OSError): @@ -3782,6 +3904,10 @@ def testCMSG_SPACE(self): # Test CMSG_SPACE() with various valid and invalid values, # checking the assumptions used by sendmsg(). toobig = self.socklen_t_limit - socket.CMSG_SPACE(1) + 1 + if SOLARIS and platform.processor() == "sparc": + # On Solaris SPARC, number of bytes returned by socket.CMSG_SPACE + # increases at different lengths; see gh-91214. + toobig -= 3 values = list(range(257)) + list(range(toobig - 257, toobig)) last = socket.CMSG_SPACE(0) @@ -3928,6 +4054,7 @@ def _testFDPassCMSG_LEN(self): self.createAndSendFDs(1) @unittest.skipIf(is_apple, "skipping, see issue #12958") + @unittest.skipIf(SOLARIS, "skipping, see gh-91214") @unittest.skipIf(AIX, "skipping, see issue #22397") @requireAttrs(socket, "CMSG_SPACE") def testFDPassSeparate(self): @@ -3939,6 +4066,7 @@ def testFDPassSeparate(self): @testFDPassSeparate.client_skip @unittest.skipIf(is_apple, "skipping, see issue #12958") + @unittest.skipIf(SOLARIS, "skipping, see gh-91214") @unittest.skipIf(AIX, "skipping, see issue #22397") def _testFDPassSeparate(self): fd0, fd1 = self.newFDs(2) @@ -3952,6 +4080,7 @@ def _testFDPassSeparate(self): len(MSG)) @unittest.skipIf(is_apple, "skipping, see issue #12958") + @unittest.skipIf(SOLARIS, "skipping, see gh-91214") @unittest.skipIf(AIX, "skipping, see issue #22397") @requireAttrs(socket, "CMSG_SPACE") def testFDPassSeparateMinSpace(self): @@ -3966,6 +4095,7 @@ def testFDPassSeparateMinSpace(self): @testFDPassSeparateMinSpace.client_skip @unittest.skipIf(is_apple, "skipping, see issue #12958") + @unittest.skipIf(SOLARIS, "skipping, see gh-91214") @unittest.skipIf(AIX, "skipping, see issue #22397") def _testFDPassSeparateMinSpace(self): fd0, fd1 = self.newFDs(2) @@ -4998,15 +5128,13 @@ def testInterruptedSendmsgTimeout(self): class TCPCloserTest(ThreadedTCPSocketTest): - def testClose(self): - conn, addr = self.serv.accept() - conn.close() + conn, _ = self.serv.accept() - sd = self.cli - read, write, err = select.select([sd], [], [], 1.0) - self.assertEqual(read, [sd]) - self.assertEqual(sd.recv(1), b'') + read, _, _ = select.select([conn], [], [], support.SHORT_TIMEOUT) + self.assertEqual(read, [conn]) + self.assertEqual(conn.recv(1), b'x') + conn.close() # Calling close() many times should be safe. conn.close() @@ -5014,7 +5142,10 @@ def testClose(self): def _testClose(self): self.cli.connect((HOST, self.port)) - time.sleep(1.0) + self.cli.send(b'x') + read, _, _ = select.select([self.cli], [], [], support.SHORT_TIMEOUT) + self.assertEqual(read, [self.cli]) + self.assertEqual(self.cli.recv(1), b'') class BasicSocketPairTest(SocketPairTest): @@ -5973,10 +6104,10 @@ def testTimeoutZero(self): class TestExceptions(unittest.TestCase): def testExceptionTree(self): - self.assertTrue(issubclass(OSError, Exception)) - self.assertTrue(issubclass(socket.herror, OSError)) - self.assertTrue(issubclass(socket.gaierror, OSError)) - self.assertTrue(issubclass(socket.timeout, OSError)) + self.assertIsSubclass(OSError, Exception) + self.assertIsSubclass(socket.herror, OSError) + self.assertIsSubclass(socket.gaierror, OSError) + self.assertIsSubclass(socket.timeout, OSError) self.assertIs(socket.error, OSError) self.assertIs(socket.timeout, TimeoutError) @@ -6489,8 +6620,7 @@ def remoteProcessServer(cls, q): s2.close() s.close() - # TODO: RUSTPYTHON; multiprocessing.SemLock not implemented - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; multiprocessing.SemLock not implemented def testShare(self): # Transfer the listening server socket to another process # and service it from there. @@ -6842,7 +6972,6 @@ def test_errors(self): @unittest.skipUnless(hasattr(os, "sendfile"), 'os.sendfile() required for this test.') -@unittest.skip("TODO: RUSTPYTHON; os.sendfile count parameter not handled correctly") class SendfileUsingSendfileTest(SendfileUsingSendTest): """ Test the sendfile() implementation of socket.sendfile(). @@ -6850,6 +6979,14 @@ class SendfileUsingSendfileTest(SendfileUsingSendTest): def meth_from_sock(self, sock): return getattr(sock, "_sendfile_use_sendfile") + @unittest.skip("TODO: RUSTPYTHON; os.sendfile count parameter not handled correctly; flaky") + def testCount(self): + return super().testCount() + + @unittest.skip("TODO: RUSTPYTHON; os.sendfile count parameter not handled correctly; flaky") + def testWithTimeout(self): + return super().testWithTimeout() + @unittest.skipUnless(HAVE_SOCKET_ALG, 'AF_ALG required') class LinuxKernelCryptoAPI(unittest.TestCase): @@ -6867,8 +7004,7 @@ def create_alg(self, typ, name): # bpo-31705: On kernel older than 4.5, sendto() failed with ENOKEY, # at least on ppc64le architecture - # TODO: RUSTPYTHON - AF_ALG not fully implemented - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; - AF_ALG not fully implemented @support.requires_linux_version(4, 5) def test_sha256(self): expected = bytes.fromhex("ba7816bf8f01cfea414140de5dae2223b00361a396" @@ -6887,8 +7023,7 @@ def test_sha256(self): op.send(b'') self.assertEqual(op.recv(512), expected) - # TODO: RUSTPYTHON - AF_ALG not fully implemented - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; - AF_ALG not fully implemented def test_hmac_sha1(self): # gh-109396: In FIPS mode, Linux 6.5 requires a key # of at least 112 bits. Use a key of 152 bits. @@ -6904,8 +7039,7 @@ def test_hmac_sha1(self): # Although it should work with 3.19 and newer the test blocks on # Ubuntu 15.10 with Kernel 4.2.0-19. - # TODO: RUSTPYTHON - AF_ALG not fully implemented - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; - AF_ALG not fully implemented @support.requires_linux_version(4, 3) def test_aes_cbc(self): key = bytes.fromhex('06a9214036b8a15b512e03d534120006') @@ -6947,8 +7081,7 @@ def test_aes_cbc(self): self.assertEqual(len(dec), msglen * multiplier) self.assertEqual(dec, msg * multiplier) - # TODO: RUSTPYTHON - AF_ALG not fully implemented - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; - AF_ALG not fully implemented @support.requires_linux_version(4, 9) # see gh-73510 def test_aead_aes_gcm(self): kernel_version = support._get_kernel_version("Linux") @@ -7018,8 +7151,7 @@ def test_aead_aes_gcm(self): res = op.recv(len(msg) - taglen) self.assertEqual(plain, res[assoclen:]) - # TODO: RUSTPYTHON - AF_ALG not fully implemented - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; - AF_ALG not fully implemented @support.requires_linux_version(4, 3) # see test_aes_cbc def test_drbg_pr_sha256(self): # deterministic random bit generator, prediction resistance, sha256 @@ -7072,6 +7204,28 @@ class TestMacOSTCPFlags(unittest.TestCase): def test_tcp_keepalive(self): self.assertTrue(socket.TCP_KEEPALIVE) +@unittest.skipUnless(hasattr(socket, 'TCP_QUICKACK'), 'need socket.TCP_QUICKACK') +class TestQuickackFlag(unittest.TestCase): + def check_set_quickack(self, sock): + # quickack already true by default on some OS distributions + opt = sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_QUICKACK) + if opt: + sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_QUICKACK, 0) + + opt = sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_QUICKACK) + self.assertFalse(opt) + + sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_QUICKACK, 1) + + opt = sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_QUICKACK) + self.assertTrue(opt) + + def test_set_quickack(self): + sock = socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM, + proto=socket.IPPROTO_TCP) + with sock: + self.check_set_quickack(sock) + @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") class TestMSWindowsTCPFlags(unittest.TestCase): @@ -7085,7 +7239,9 @@ class TestMSWindowsTCPFlags(unittest.TestCase): 'TCP_KEEPCNT', # available starting with Windows 10 1709 'TCP_KEEPIDLE', - 'TCP_KEEPINTVL' + 'TCP_KEEPINTVL', + # available starting with Windows 7 / Server 2008 R2 + 'TCP_QUICKACK', } def test_new_tcp_flags(self): @@ -7253,6 +7409,26 @@ def close_fds(fds): self.assertEqual(data, str(index).encode()) +class FreeThreadingTests(unittest.TestCase): + + def test_close_detach_race(self): + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + + def close(): + for _ in range(1000): + s.close() + + def detach(): + for _ in range(1000): + s.detach() + + t1 = threading.Thread(target=close) + t2 = threading.Thread(target=detach) + + with threading_helper.start_threads([t1, t2]): + pass + + def setUpModule(): thread_info = threading_helper.threading_setup() unittest.addModuleCleanup(threading_helper.threading_cleanup, *thread_info) diff --git a/Lib/test/test_sort.py b/Lib/test/test_sort.py index be3d4a8461f..37ce489b79f 100644 --- a/Lib/test/test_sort.py +++ b/Lib/test/test_sort.py @@ -149,7 +149,7 @@ def __lt__(self, other): L = [C() for i in range(50)] self.assertRaises(ValueError, L.sort) - @unittest.skip("TODO: RUSTPYTHON; figure out how to detect sort mutation that doesn't change list length") + @unittest.expectedFailure # TODO: RUSTPYTHON; figure out how to detect sort mutation that doesn't change list length def test_undetected_mutation(self): # Python 2.4a1 did not always detect mutation memorywaster = [] diff --git a/Lib/test/test_sqlite3/__init__.py b/Lib/test/test_sqlite3/__init__.py index d777fca82da..78a1e2078a5 100644 --- a/Lib/test/test_sqlite3/__init__.py +++ b/Lib/test/test_sqlite3/__init__.py @@ -8,8 +8,7 @@ # Implement the unittest "load tests" protocol. def load_tests(*args): + if verbose: + print(f"test_sqlite3: testing with SQLite version {sqlite3.sqlite_version}") pkg_dir = os.path.dirname(__file__) return load_package_tests(pkg_dir, *args) - -if verbose: - print(f"test_sqlite3: testing with SQLite version {sqlite3.sqlite_version}") diff --git a/Lib/test/test_sqlite3/test_backup.py b/Lib/test/test_sqlite3/test_backup.py index fb3a83e3b0e..9d31978b1ad 100644 --- a/Lib/test/test_sqlite3/test_backup.py +++ b/Lib/test/test_sqlite3/test_backup.py @@ -1,6 +1,8 @@ import sqlite3 as sqlite import unittest +from .util import memory_database + class BackupTests(unittest.TestCase): def setUp(self): @@ -32,34 +34,32 @@ def test_bad_target_same_connection(self): self.cx.backup(self.cx) def test_bad_target_closed_connection(self): - bck = sqlite.connect(':memory:') - bck.close() - with self.assertRaises(sqlite.ProgrammingError): - self.cx.backup(bck) + with memory_database() as bck: + bck.close() + with self.assertRaises(sqlite.ProgrammingError): + self.cx.backup(bck) def test_bad_source_closed_connection(self): - bck = sqlite.connect(':memory:') - source = sqlite.connect(":memory:") - source.close() - with self.assertRaises(sqlite.ProgrammingError): - source.backup(bck) + with memory_database() as bck: + source = sqlite.connect(":memory:") + source.close() + with self.assertRaises(sqlite.ProgrammingError): + source.backup(bck) def test_bad_target_in_transaction(self): - bck = sqlite.connect(':memory:') - bck.execute('CREATE TABLE bar (key INTEGER)') - bck.executemany('INSERT INTO bar (key) VALUES (?)', [(3,), (4,)]) - with self.assertRaises(sqlite.OperationalError) as cm: - self.cx.backup(bck) - if sqlite.sqlite_version_info < (3, 8, 8): - self.assertEqual(str(cm.exception), 'target is in transaction') + with memory_database() as bck: + bck.execute('CREATE TABLE bar (key INTEGER)') + bck.executemany('INSERT INTO bar (key) VALUES (?)', [(3,), (4,)]) + with self.assertRaises(sqlite.OperationalError) as cm: + self.cx.backup(bck) def test_keyword_only_args(self): with self.assertRaises(TypeError): - with sqlite.connect(':memory:') as bck: + with memory_database() as bck: self.cx.backup(bck, 1) def test_simple(self): - with sqlite.connect(':memory:') as bck: + with memory_database() as bck: self.cx.backup(bck) self.verify_backup(bck) @@ -69,7 +69,7 @@ def test_progress(self): def progress(status, remaining, total): journal.append(status) - with sqlite.connect(':memory:') as bck: + with memory_database() as bck: self.cx.backup(bck, pages=1, progress=progress) self.verify_backup(bck) @@ -83,7 +83,7 @@ def test_progress_all_pages_at_once_1(self): def progress(status, remaining, total): journal.append(remaining) - with sqlite.connect(':memory:') as bck: + with memory_database() as bck: self.cx.backup(bck, progress=progress) self.verify_backup(bck) @@ -96,18 +96,17 @@ def test_progress_all_pages_at_once_2(self): def progress(status, remaining, total): journal.append(remaining) - with sqlite.connect(':memory:') as bck: + with memory_database() as bck: self.cx.backup(bck, pages=-1, progress=progress) self.verify_backup(bck) self.assertEqual(len(journal), 1) self.assertEqual(journal[0], 0) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_non_callable_progress(self): with self.assertRaises(TypeError) as cm: - with sqlite.connect(':memory:') as bck: + with memory_database() as bck: self.cx.backup(bck, pages=1, progress='bar') self.assertEqual(str(cm.exception), 'progress argument must be a callable') @@ -120,7 +119,7 @@ def progress(status, remaining, total): self.cx.commit() journal.append(remaining) - with sqlite.connect(':memory:') as bck: + with memory_database() as bck: self.cx.backup(bck, pages=1, progress=progress) self.verify_backup(bck) @@ -139,17 +138,17 @@ def progress(status, remaining, total): raise SystemError('nearly out of space') with self.assertRaises(SystemError) as err: - with sqlite.connect(':memory:') as bck: + with memory_database() as bck: self.cx.backup(bck, progress=progress) self.assertEqual(str(err.exception), 'nearly out of space') def test_database_source_name(self): - with sqlite.connect(':memory:') as bck: + with memory_database() as bck: self.cx.backup(bck, name='main') - with sqlite.connect(':memory:') as bck: + with memory_database() as bck: self.cx.backup(bck, name='temp') with self.assertRaises(sqlite.OperationalError) as cm: - with sqlite.connect(':memory:') as bck: + with memory_database() as bck: self.cx.backup(bck, name='non-existing') self.assertIn("unknown database", str(cm.exception)) @@ -157,7 +156,7 @@ def test_database_source_name(self): self.cx.execute('CREATE TABLE attached_db.foo (key INTEGER)') self.cx.executemany('INSERT INTO attached_db.foo (key) VALUES (?)', [(3,), (4,)]) self.cx.commit() - with sqlite.connect(':memory:') as bck: + with memory_database() as bck: self.cx.backup(bck, name='attached_db') self.verify_backup(bck) diff --git a/Lib/test/test_sqlite3/test_cli.py b/Lib/test/test_sqlite3/test_cli.py index 560cd9efc5c..a03d7cbe16b 100644 --- a/Lib/test/test_sqlite3/test_cli.py +++ b/Lib/test/test_sqlite3/test_cli.py @@ -1,52 +1,52 @@ """sqlite3 CLI tests.""" - -import sqlite3 as sqlite -import subprocess -import sys +import sqlite3 import unittest -from test.support import SHORT_TIMEOUT#, requires_subprocess +from sqlite3.__main__ import main as cli from test.support.os_helper import TESTFN, unlink +from test.support import ( + captured_stdout, + captured_stderr, + captured_stdin, + force_not_colorized, +) -# TODO: RUSTPYTHON -#@requires_subprocess() class CommandLineInterface(unittest.TestCase): def _do_test(self, *args, expect_success=True): - with subprocess.Popen( - [sys.executable, "-Xutf8", "-m", "sqlite3", *args], - encoding="utf-8", - bufsize=0, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - ) as proc: - proc.wait() - if expect_success == bool(proc.returncode): - self.fail("".join(proc.stderr)) - stdout = proc.stdout.read() - stderr = proc.stderr.read() - if expect_success: - self.assertEqual(stderr, "") - else: - self.assertEqual(stdout, "") - return stdout, stderr + with ( + captured_stdout() as out, + captured_stderr() as err, + self.assertRaises(SystemExit) as cm + ): + cli(args) + return out.getvalue(), err.getvalue(), cm.exception.code def expect_success(self, *args): - out, _ = self._do_test(*args) + out, err, code = self._do_test(*args) + self.assertEqual(code, 0, + "\n".join([f"Unexpected failure: {args=}", out, err])) + self.assertEqual(err, "") return out def expect_failure(self, *args): - _, err = self._do_test(*args, expect_success=False) + out, err, code = self._do_test(*args, expect_success=False) + self.assertNotEqual(code, 0, + "\n".join([f"Unexpected failure: {args=}", out, err])) + self.assertEqual(out, "") return err + @force_not_colorized def test_cli_help(self): out = self.expect_success("-h") - self.assertIn("usage: python -m sqlite3", out) + self.assertIn("usage: ", out) + self.assertIn(" [-h] [-v] [filename] [sql]", out) + self.assertIn("Python sqlite3 CLI", out) def test_cli_version(self): out = self.expect_success("-v") - self.assertIn(sqlite.sqlite_version, out) + self.assertIn(sqlite3.sqlite_version, out) def test_cli_execute_sql(self): out = self.expect_success(":memory:", "select 1") @@ -69,88 +69,126 @@ def test_cli_on_disk_db(self): self.assertIn("(0,)", out) -# TODO: RUSTPYTHON -#@requires_subprocess() class InteractiveSession(unittest.TestCase): - TIMEOUT = SHORT_TIMEOUT / 10. MEMORY_DB_MSG = "Connected to a transient in-memory database" PS1 = "sqlite> " PS2 = "... " - def start_cli(self, *args): - return subprocess.Popen( - [sys.executable, "-Xutf8", "-m", "sqlite3", *args], - encoding="utf-8", - bufsize=0, - stdin=subprocess.PIPE, - # Note: the banner is printed to stderr, the prompt to stdout. - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - ) - - def expect_success(self, proc): - proc.wait() - if proc.returncode: - self.fail("".join(proc.stderr)) + def run_cli(self, *args, commands=()): + with ( + captured_stdin() as stdin, + captured_stdout() as stdout, + captured_stderr() as stderr, + self.assertRaises(SystemExit) as cm + ): + for cmd in commands: + stdin.write(cmd + "\n") + stdin.seek(0) + cli(args) + + out = stdout.getvalue() + err = stderr.getvalue() + self.assertEqual(cm.exception.code, 0, + f"Unexpected failure: {args=}\n{out}\n{err}") + return out, err def test_interact(self): - with self.start_cli() as proc: - out, err = proc.communicate(timeout=self.TIMEOUT) - self.assertIn(self.MEMORY_DB_MSG, err) - self.assertIn(self.PS1, out) - self.expect_success(proc) + out, err = self.run_cli() + self.assertIn(self.MEMORY_DB_MSG, err) + self.assertIn(self.MEMORY_DB_MSG, err) + self.assertEndsWith(out, self.PS1) + self.assertEqual(out.count(self.PS1), 1) + self.assertEqual(out.count(self.PS2), 0) def test_interact_quit(self): - with self.start_cli() as proc: - out, err = proc.communicate(input=".quit", timeout=self.TIMEOUT) - self.assertIn(self.MEMORY_DB_MSG, err) - self.assertIn(self.PS1, out) - self.expect_success(proc) + out, err = self.run_cli(commands=(".quit",)) + self.assertIn(self.MEMORY_DB_MSG, err) + self.assertEndsWith(out, self.PS1) + self.assertEqual(out.count(self.PS1), 1) + self.assertEqual(out.count(self.PS2), 0) def test_interact_version(self): - with self.start_cli() as proc: - out, err = proc.communicate(input=".version", timeout=self.TIMEOUT) - self.assertIn(self.MEMORY_DB_MSG, err) - self.assertIn(sqlite.sqlite_version, out) - self.expect_success(proc) + out, err = self.run_cli(commands=(".version",)) + self.assertIn(self.MEMORY_DB_MSG, err) + self.assertIn(sqlite3.sqlite_version + "\n", out) + self.assertEndsWith(out, self.PS1) + self.assertEqual(out.count(self.PS1), 2) + self.assertEqual(out.count(self.PS2), 0) + self.assertIn(sqlite3.sqlite_version, out) + + def test_interact_empty_source(self): + out, err = self.run_cli(commands=("", " ")) + self.assertIn(self.MEMORY_DB_MSG, err) + self.assertEndsWith(out, self.PS1) + self.assertEqual(out.count(self.PS1), 3) + self.assertEqual(out.count(self.PS2), 0) + + def test_interact_dot_commands_unknown(self): + out, err = self.run_cli(commands=(".unknown_command", )) + self.assertIn(self.MEMORY_DB_MSG, err) + self.assertEndsWith(out, self.PS1) + self.assertEqual(out.count(self.PS1), 2) + self.assertEqual(out.count(self.PS2), 0) + self.assertIn("Error", err) + # test "unknown_command" is pointed out in the error message + self.assertIn("unknown_command", err) + + def test_interact_dot_commands_empty(self): + out, err = self.run_cli(commands=(".")) + self.assertIn(self.MEMORY_DB_MSG, err) + self.assertEndsWith(out, self.PS1) + self.assertEqual(out.count(self.PS1), 2) + self.assertEqual(out.count(self.PS2), 0) + + def test_interact_dot_commands_with_whitespaces(self): + out, err = self.run_cli(commands=(".version ", ". version")) + self.assertIn(self.MEMORY_DB_MSG, err) + self.assertEqual(out.count(sqlite3.sqlite_version + "\n"), 2) + self.assertEndsWith(out, self.PS1) + self.assertEqual(out.count(self.PS1), 3) + self.assertEqual(out.count(self.PS2), 0) def test_interact_valid_sql(self): - with self.start_cli() as proc: - out, err = proc.communicate(input="select 1;", - timeout=self.TIMEOUT) - self.assertIn(self.MEMORY_DB_MSG, err) - self.assertIn("(1,)", out) - self.expect_success(proc) + out, err = self.run_cli(commands=("SELECT 1;",)) + self.assertIn(self.MEMORY_DB_MSG, err) + self.assertIn("(1,)\n", out) + self.assertEndsWith(out, self.PS1) + self.assertEqual(out.count(self.PS1), 2) + self.assertEqual(out.count(self.PS2), 0) + + def test_interact_incomplete_multiline_sql(self): + out, err = self.run_cli(commands=("SELECT 1",)) + self.assertIn(self.MEMORY_DB_MSG, err) + self.assertEndsWith(out, self.PS2) + self.assertEqual(out.count(self.PS1), 1) + self.assertEqual(out.count(self.PS2), 1) def test_interact_valid_multiline_sql(self): - with self.start_cli() as proc: - out, err = proc.communicate(input="select 1\n;", - timeout=self.TIMEOUT) - self.assertIn(self.MEMORY_DB_MSG, err) - self.assertIn(self.PS2, out) - self.assertIn("(1,)", out) - self.expect_success(proc) + out, err = self.run_cli(commands=("SELECT 1\n;",)) + self.assertIn(self.MEMORY_DB_MSG, err) + self.assertIn(self.PS2, out) + self.assertIn("(1,)\n", out) + self.assertEndsWith(out, self.PS1) + self.assertEqual(out.count(self.PS1), 2) + self.assertEqual(out.count(self.PS2), 1) def test_interact_invalid_sql(self): - with self.start_cli() as proc: - out, err = proc.communicate(input="sel;", timeout=self.TIMEOUT) - self.assertIn(self.MEMORY_DB_MSG, err) - self.assertIn("OperationalError (SQLITE_ERROR)", err) - self.expect_success(proc) + out, err = self.run_cli(commands=("sel;",)) + self.assertIn(self.MEMORY_DB_MSG, err) + self.assertIn("OperationalError (SQLITE_ERROR)", err) + self.assertEndsWith(out, self.PS1) + self.assertEqual(out.count(self.PS1), 2) + self.assertEqual(out.count(self.PS2), 0) def test_interact_on_disk_file(self): self.addCleanup(unlink, TESTFN) - with self.start_cli(TESTFN) as proc: - out, err = proc.communicate(input="create table t(t);", - timeout=self.TIMEOUT) - self.assertIn(TESTFN, err) - self.assertIn(self.PS1, out) - self.expect_success(proc) - with self.start_cli(TESTFN, "select count(t) from t") as proc: - out = proc.stdout.read() - err = proc.stderr.read() - self.assertIn("(0,)", out) - self.expect_success(proc) + + out, err = self.run_cli(TESTFN, commands=("CREATE TABLE t(t);",)) + self.assertIn(TESTFN, err) + self.assertEndsWith(out, self.PS1) + + out, _ = self.run_cli(TESTFN, commands=("SELECT count(t) FROM t;",)) + self.assertIn("(0,)\n", out) if __name__ == "__main__": diff --git a/Lib/test/test_sqlite3/test_dbapi.py b/Lib/test/test_sqlite3/test_dbapi.py index 9a95c489a31..46f098e4655 100644 --- a/Lib/test/test_sqlite3/test_dbapi.py +++ b/Lib/test/test_sqlite3/test_dbapi.py @@ -21,6 +21,7 @@ # 3. This notice may not be removed or altered from any source distribution. import contextlib +import functools import os import sqlite3 as sqlite import subprocess @@ -28,33 +29,18 @@ import threading import unittest import urllib.parse +import warnings from test.support import ( - SHORT_TIMEOUT, check_disallow_instantiation,# requires_subprocess, - #is_emscripten, is_wasi -# TODO: RUSTPYTHON + SHORT_TIMEOUT, check_disallow_instantiation, requires_subprocess ) -from test.support import threading_helper -# TODO: RUSTPYTHON -#from _testcapi import INT_MAX, ULLONG_MAX +from test.support import gc_collect +from test.support import threading_helper, import_helper from os import SEEK_SET, SEEK_CUR, SEEK_END from test.support.os_helper import TESTFN, TESTFN_UNDECODABLE, unlink, temp_dir, FakePath - -# Helper for temporary memory databases -def memory_database(*args, **kwargs): - cx = sqlite.connect(":memory:", *args, **kwargs) - return contextlib.closing(cx) - - -# Temporarily limit a database connection parameter -@contextlib.contextmanager -def cx_limit(cx, category=sqlite.SQLITE_LIMIT_SQL_LENGTH, limit=128): - try: - _prev = cx.setlimit(category, limit) - yield limit - finally: - cx.setlimit(category, _prev) +from .util import memory_database, cx_limit +from .util import MemoryDatabaseMixin class ModuleTests(unittest.TestCase): @@ -62,17 +48,6 @@ def test_api_level(self): self.assertEqual(sqlite.apilevel, "2.0", "apilevel is %s, should be 2.0" % sqlite.apilevel) - def test_deprecated_version(self): - msg = "deprecated and will be removed in Python 3.14" - for attr in "version", "version_info": - with self.subTest(attr=attr): - with self.assertWarnsRegex(DeprecationWarning, msg) as cm: - getattr(sqlite, attr) - self.assertEqual(cm.filename, __file__) - with self.assertWarnsRegex(DeprecationWarning, msg) as cm: - getattr(sqlite.dbapi2, attr) - self.assertEqual(cm.filename, __file__) - def test_thread_safety(self): self.assertIn(sqlite.threadsafety, {0, 1, 3}, "threadsafety is %d, should be 0, 1 or 3" % @@ -84,45 +59,34 @@ def test_param_style(self): sqlite.paramstyle) def test_warning(self): - self.assertTrue(issubclass(sqlite.Warning, Exception), - "Warning is not a subclass of Exception") + self.assertIsSubclass(sqlite.Warning, Exception) def test_error(self): - self.assertTrue(issubclass(sqlite.Error, Exception), - "Error is not a subclass of Exception") + self.assertIsSubclass(sqlite.Error, Exception) def test_interface_error(self): - self.assertTrue(issubclass(sqlite.InterfaceError, sqlite.Error), - "InterfaceError is not a subclass of Error") + self.assertIsSubclass(sqlite.InterfaceError, sqlite.Error) def test_database_error(self): - self.assertTrue(issubclass(sqlite.DatabaseError, sqlite.Error), - "DatabaseError is not a subclass of Error") + self.assertIsSubclass(sqlite.DatabaseError, sqlite.Error) def test_data_error(self): - self.assertTrue(issubclass(sqlite.DataError, sqlite.DatabaseError), - "DataError is not a subclass of DatabaseError") + self.assertIsSubclass(sqlite.DataError, sqlite.DatabaseError) def test_operational_error(self): - self.assertTrue(issubclass(sqlite.OperationalError, sqlite.DatabaseError), - "OperationalError is not a subclass of DatabaseError") + self.assertIsSubclass(sqlite.OperationalError, sqlite.DatabaseError) def test_integrity_error(self): - self.assertTrue(issubclass(sqlite.IntegrityError, sqlite.DatabaseError), - "IntegrityError is not a subclass of DatabaseError") + self.assertIsSubclass(sqlite.IntegrityError, sqlite.DatabaseError) def test_internal_error(self): - self.assertTrue(issubclass(sqlite.InternalError, sqlite.DatabaseError), - "InternalError is not a subclass of DatabaseError") + self.assertIsSubclass(sqlite.InternalError, sqlite.DatabaseError) def test_programming_error(self): - self.assertTrue(issubclass(sqlite.ProgrammingError, sqlite.DatabaseError), - "ProgrammingError is not a subclass of DatabaseError") + self.assertIsSubclass(sqlite.ProgrammingError, sqlite.DatabaseError) def test_not_supported_error(self): - self.assertTrue(issubclass(sqlite.NotSupportedError, - sqlite.DatabaseError), - "NotSupportedError is not a subclass of DatabaseError") + self.assertIsSubclass(sqlite.NotSupportedError, sqlite.DatabaseError) def test_module_constants(self): consts = [ @@ -167,6 +131,7 @@ def test_module_constants(self): "SQLITE_INTERNAL", "SQLITE_INTERRUPT", "SQLITE_IOERR", + "SQLITE_LIMIT_WORKER_THREADS", "SQLITE_LOCKED", "SQLITE_MISMATCH", "SQLITE_MISUSE", @@ -174,6 +139,7 @@ def test_module_constants(self): "SQLITE_NOMEM", "SQLITE_NOTADB", "SQLITE_NOTFOUND", + "SQLITE_NOTICE", "SQLITE_OK", "SQLITE_PERM", "SQLITE_PRAGMA", @@ -181,6 +147,7 @@ def test_module_constants(self): "SQLITE_RANGE", "SQLITE_READ", "SQLITE_READONLY", + "SQLITE_RECURSIVE", "SQLITE_REINDEX", "SQLITE_ROW", "SQLITE_SAVEPOINT", @@ -189,6 +156,7 @@ def test_module_constants(self): "SQLITE_TOOBIG", "SQLITE_TRANSACTION", "SQLITE_UPDATE", + "SQLITE_WARNING", # Run-time limit categories "SQLITE_LIMIT_LENGTH", "SQLITE_LIMIT_SQL_LENGTH", @@ -202,32 +170,43 @@ def test_module_constants(self): "SQLITE_LIMIT_VARIABLE_NUMBER", "SQLITE_LIMIT_TRIGGER_DEPTH", ] - if sqlite.sqlite_version_info >= (3, 7, 17): - consts += ["SQLITE_NOTICE", "SQLITE_WARNING"] - if sqlite.sqlite_version_info >= (3, 8, 3): - consts.append("SQLITE_RECURSIVE") - if sqlite.sqlite_version_info >= (3, 8, 7): - consts.append("SQLITE_LIMIT_WORKER_THREADS") consts += ["PARSE_DECLTYPES", "PARSE_COLNAMES"] # Extended result codes consts += [ "SQLITE_ABORT_ROLLBACK", + "SQLITE_AUTH_USER", "SQLITE_BUSY_RECOVERY", + "SQLITE_BUSY_SNAPSHOT", + "SQLITE_CANTOPEN_CONVPATH", "SQLITE_CANTOPEN_FULLPATH", "SQLITE_CANTOPEN_ISDIR", "SQLITE_CANTOPEN_NOTEMPDIR", + "SQLITE_CONSTRAINT_CHECK", + "SQLITE_CONSTRAINT_COMMITHOOK", + "SQLITE_CONSTRAINT_FOREIGNKEY", + "SQLITE_CONSTRAINT_FUNCTION", + "SQLITE_CONSTRAINT_NOTNULL", + "SQLITE_CONSTRAINT_PRIMARYKEY", + "SQLITE_CONSTRAINT_ROWID", + "SQLITE_CONSTRAINT_TRIGGER", + "SQLITE_CONSTRAINT_UNIQUE", + "SQLITE_CONSTRAINT_VTAB", "SQLITE_CORRUPT_VTAB", "SQLITE_IOERR_ACCESS", + "SQLITE_IOERR_AUTH", "SQLITE_IOERR_BLOCKED", "SQLITE_IOERR_CHECKRESERVEDLOCK", "SQLITE_IOERR_CLOSE", + "SQLITE_IOERR_CONVPATH", "SQLITE_IOERR_DELETE", "SQLITE_IOERR_DELETE_NOENT", "SQLITE_IOERR_DIR_CLOSE", "SQLITE_IOERR_DIR_FSYNC", "SQLITE_IOERR_FSTAT", "SQLITE_IOERR_FSYNC", + "SQLITE_IOERR_GETTEMPPATH", "SQLITE_IOERR_LOCK", + "SQLITE_IOERR_MMAP", "SQLITE_IOERR_NOMEM", "SQLITE_IOERR_RDLOCK", "SQLITE_IOERR_READ", @@ -239,50 +218,18 @@ def test_module_constants(self): "SQLITE_IOERR_SHORT_READ", "SQLITE_IOERR_TRUNCATE", "SQLITE_IOERR_UNLOCK", + "SQLITE_IOERR_VNODE", "SQLITE_IOERR_WRITE", "SQLITE_LOCKED_SHAREDCACHE", + "SQLITE_NOTICE_RECOVER_ROLLBACK", + "SQLITE_NOTICE_RECOVER_WAL", + "SQLITE_OK_LOAD_PERMANENTLY", "SQLITE_READONLY_CANTLOCK", + "SQLITE_READONLY_DBMOVED", "SQLITE_READONLY_RECOVERY", + "SQLITE_READONLY_ROLLBACK", + "SQLITE_WARNING_AUTOINDEX", ] - if sqlite.sqlite_version_info >= (3, 7, 16): - consts += [ - "SQLITE_CONSTRAINT_CHECK", - "SQLITE_CONSTRAINT_COMMITHOOK", - "SQLITE_CONSTRAINT_FOREIGNKEY", - "SQLITE_CONSTRAINT_FUNCTION", - "SQLITE_CONSTRAINT_NOTNULL", - "SQLITE_CONSTRAINT_PRIMARYKEY", - "SQLITE_CONSTRAINT_TRIGGER", - "SQLITE_CONSTRAINT_UNIQUE", - "SQLITE_CONSTRAINT_VTAB", - "SQLITE_READONLY_ROLLBACK", - ] - if sqlite.sqlite_version_info >= (3, 7, 17): - consts += [ - "SQLITE_IOERR_MMAP", - "SQLITE_NOTICE_RECOVER_ROLLBACK", - "SQLITE_NOTICE_RECOVER_WAL", - ] - if sqlite.sqlite_version_info >= (3, 8, 0): - consts += [ - "SQLITE_BUSY_SNAPSHOT", - "SQLITE_IOERR_GETTEMPPATH", - "SQLITE_WARNING_AUTOINDEX", - ] - if sqlite.sqlite_version_info >= (3, 8, 1): - consts += ["SQLITE_CANTOPEN_CONVPATH", "SQLITE_IOERR_CONVPATH"] - if sqlite.sqlite_version_info >= (3, 8, 2): - consts.append("SQLITE_CONSTRAINT_ROWID") - if sqlite.sqlite_version_info >= (3, 8, 3): - consts.append("SQLITE_READONLY_DBMOVED") - if sqlite.sqlite_version_info >= (3, 8, 7): - consts.append("SQLITE_AUTH_USER") - if sqlite.sqlite_version_info >= (3, 9, 0): - consts.append("SQLITE_IOERR_VNODE") - if sqlite.sqlite_version_info >= (3, 10, 0): - consts.append("SQLITE_IOERR_AUTH") - if sqlite.sqlite_version_info >= (3, 14, 1): - consts.append("SQLITE_OK_LOAD_PERMANENTLY") if sqlite.sqlite_version_info >= (3, 21, 0): consts += [ "SQLITE_IOERR_BEGIN_ATOMIC", @@ -316,7 +263,7 @@ def test_module_constants(self): consts.append("SQLITE_IOERR_CORRUPTFS") for const in consts: with self.subTest(const=const): - self.assertTrue(hasattr(sqlite, const)) + self.assertHasAttr(sqlite, const) def test_error_code_on_exception(self): err_msg = "unable to open database file" @@ -330,10 +277,8 @@ def test_error_code_on_exception(self): sqlite.connect(db) e = cm.exception self.assertEqual(e.sqlite_errorcode, err_code) - self.assertTrue(e.sqlite_errorname.startswith("SQLITE_CANTOPEN")) + self.assertStartsWith(e.sqlite_errorname, "SQLITE_CANTOPEN") - @unittest.skipIf(sqlite.sqlite_version_info <= (3, 7, 16), - "Requires SQLite 3.7.16 or newer") def test_extended_error_code_on_exception(self): with memory_database() as con: with con: @@ -347,9 +292,9 @@ def test_extended_error_code_on_exception(self): self.assertEqual(exc.sqlite_errorname, "SQLITE_CONSTRAINT_CHECK") def test_disallow_instantiation(self): - cx = sqlite.connect(":memory:") - check_disallow_instantiation(self, type(cx("select 1"))) - check_disallow_instantiation(self, sqlite.Blob) + with memory_database() as cx: + check_disallow_instantiation(self, type(cx("select 1"))) + check_disallow_instantiation(self, sqlite.Blob) def test_complete_statement(self): self.assertFalse(sqlite.complete_statement("select t")) @@ -363,6 +308,7 @@ def setUp(self): cu = self.cx.cursor() cu.execute("create table test(id integer primary key, name text)") cu.execute("insert into test(name) values (?)", ("foo",)) + cu.close() def tearDown(self): self.cx.close() @@ -418,8 +364,7 @@ def test_use_after_close(self): with self.cx: pass - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_exceptions(self): # Optional DB-API extension. self.assertEqual(self.cx.Warning, sqlite.Warning) @@ -435,28 +380,28 @@ def test_exceptions(self): def test_in_transaction(self): # Can't use db from setUp because we want to test initial state. - cx = sqlite.connect(":memory:") - cu = cx.cursor() - self.assertEqual(cx.in_transaction, False) - cu.execute("create table transactiontest(id integer primary key, name text)") - self.assertEqual(cx.in_transaction, False) - cu.execute("insert into transactiontest(name) values (?)", ("foo",)) - self.assertEqual(cx.in_transaction, True) - cu.execute("select name from transactiontest where name=?", ["foo"]) - row = cu.fetchone() - self.assertEqual(cx.in_transaction, True) - cx.commit() - self.assertEqual(cx.in_transaction, False) - cu.execute("select name from transactiontest where name=?", ["foo"]) - row = cu.fetchone() - self.assertEqual(cx.in_transaction, False) + with memory_database() as cx: + cu = cx.cursor() + self.assertEqual(cx.in_transaction, False) + cu.execute("create table transactiontest(id integer primary key, name text)") + self.assertEqual(cx.in_transaction, False) + cu.execute("insert into transactiontest(name) values (?)", ("foo",)) + self.assertEqual(cx.in_transaction, True) + cu.execute("select name from transactiontest where name=?", ["foo"]) + row = cu.fetchone() + self.assertEqual(cx.in_transaction, True) + cx.commit() + self.assertEqual(cx.in_transaction, False) + cu.execute("select name from transactiontest where name=?", ["foo"]) + row = cu.fetchone() + self.assertEqual(cx.in_transaction, False) + cu.close() def test_in_transaction_ro(self): with self.assertRaises(AttributeError): self.cx.in_transaction = True - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_connection_exceptions(self): exceptions = [ "DataError", @@ -471,14 +416,13 @@ def test_connection_exceptions(self): ] for exc in exceptions: with self.subTest(exc=exc): - self.assertTrue(hasattr(self.cx, exc)) + self.assertHasAttr(self.cx, exc) self.assertIs(getattr(sqlite, exc), getattr(self.cx, exc)) def test_interrupt_on_closed_db(self): - cx = sqlite.connect(":memory:") - cx.close() + self.cx.close() with self.assertRaises(sqlite.ProgrammingError): - cx.interrupt() + self.cx.interrupt() def test_interrupt(self): self.assertIsNone(self.cx.interrupt()) @@ -545,33 +489,30 @@ def test_connection_init_good_isolation_levels(self): cx.isolation_level = level self.assertEqual(cx.isolation_level, level) - # TODO: RUSTPYTHON - # @unittest.expectedFailure - @unittest.skip("TODO: RUSTPYTHON deadlock") def test_connection_reinit(self): - db = ":memory:" - cx = sqlite.connect(db) - cx.text_factory = bytes - cx.row_factory = sqlite.Row - cu = cx.cursor() - cu.execute("create table foo (bar)") - cu.executemany("insert into foo (bar) values (?)", - ((str(v),) for v in range(4))) - cu.execute("select bar from foo") - - rows = [r for r in cu.fetchmany(2)] - self.assertTrue(all(isinstance(r, sqlite.Row) for r in rows)) - self.assertEqual([r[0] for r in rows], [b"0", b"1"]) - - cx.__init__(db) - cx.execute("create table foo (bar)") - cx.executemany("insert into foo (bar) values (?)", - ((v,) for v in ("a", "b", "c", "d"))) - - # This uses the old database, old row factory, but new text factory - rows = [r for r in cu.fetchall()] - self.assertTrue(all(isinstance(r, sqlite.Row) for r in rows)) - self.assertEqual([r[0] for r in rows], ["2", "3"]) + with memory_database() as cx: + cx.text_factory = bytes + cx.row_factory = sqlite.Row + cu = cx.cursor() + cu.execute("create table foo (bar)") + cu.executemany("insert into foo (bar) values (?)", + ((str(v),) for v in range(4))) + cu.execute("select bar from foo") + + rows = [r for r in cu.fetchmany(2)] + self.assertTrue(all(isinstance(r, sqlite.Row) for r in rows)) + self.assertEqual([r[0] for r in rows], [b"0", b"1"]) + + cx.__init__(":memory:") + cx.execute("create table foo (bar)") + cx.executemany("insert into foo (bar) values (?)", + ((v,) for v in ("a", "b", "c", "d"))) + + # This uses the old database, old row factory, but new text factory + rows = [r for r in cu.fetchall()] + self.assertTrue(all(isinstance(r, sqlite.Row) for r in rows)) + self.assertEqual([r[0] for r in rows], ["2", "3"]) + cu.close() def test_connection_bad_reinit(self): cx = sqlite.connect(":memory:") @@ -586,12 +527,64 @@ def test_connection_bad_reinit(self): cx.executemany, "insert into t values(?)", ((v,) for v in range(3))) + @unittest.expectedFailure # TODO: RUSTPYTHON SQLITE_DBCONFIG constants not implemented + def test_connection_config(self): + op = sqlite.SQLITE_DBCONFIG_ENABLE_FKEY + with memory_database() as cx: + with self.assertRaisesRegex(ValueError, "unknown"): + cx.getconfig(-1) + + # Toggle and verify. + old = cx.getconfig(op) + new = not old + cx.setconfig(op, new) + self.assertEqual(cx.getconfig(op), new) + + cx.setconfig(op) # defaults to True + self.assertTrue(cx.getconfig(op)) + + # Check that foreign key support was actually enabled. + with cx: + cx.executescript(""" + create table t(t integer primary key); + create table u(u, foreign key(u) references t(t)); + """) + with self.assertRaisesRegex(sqlite.IntegrityError, "constraint"): + cx.execute("insert into u values(0)") + + @unittest.expectedFailure # TODO: RUSTPYTHON deprecation warning not emitted for positional args + def test_connect_positional_arguments(self): + regex = ( + r"Passing more than 1 positional argument to sqlite3.connect\(\)" + " is deprecated. Parameters 'timeout', 'detect_types', " + "'isolation_level', 'check_same_thread', 'factory', " + "'cached_statements' and 'uri' will become keyword-only " + "parameters in Python 3.15." + ) + with self.assertWarnsRegex(DeprecationWarning, regex) as cm: + cx = sqlite.connect(":memory:", 1.0) + cx.close() + self.assertEqual(cm.filename, __file__) + + @unittest.expectedFailure # TODO: RUSTPYTHON ResourceWarning not emitted + def test_connection_resource_warning(self): + with self.assertWarns(ResourceWarning): + cx = sqlite.connect(":memory:") + del cx + gc_collect() + + @unittest.expectedFailure # TODO: RUSTPYTHON Connection signature inspection not working + def test_connection_signature(self): + from inspect import signature + sig = signature(self.cx) + self.assertEqual(str(sig), "(sql, /)") + -@unittest.skip("TODO: RUSTPYTHON") class UninitialisedConnectionTests(unittest.TestCase): def setUp(self): self.cx = sqlite.Connection.__new__(sqlite.Connection) + @unittest.skip('TODO: RUSTPYTHON') def test_uninit_operations(self): funcs = ( lambda: self.cx.isolation_level, @@ -616,7 +609,6 @@ def test_serialize_deserialize(self): with cx: cx.execute("create table t(t)") data = cx.serialize() - self.assertEqual(len(data), 8192) # Remove test table, verify that it was removed. with cx: @@ -654,6 +646,14 @@ def test_deserialize_corrupt_database(self): class OpenTests(unittest.TestCase): _sql = "create table test(id integer)" + def test_open_with_bytes_path(self): + path = os.fsencode(TESTFN) + self.addCleanup(unlink, path) + self.assertFalse(os.path.exists(path)) + with contextlib.closing(sqlite.connect(path)) as cx: + self.assertTrue(os.path.exists(path)) + cx.execute(self._sql) + def test_open_with_path_like_object(self): """ Checks that we can successfully connect to a database using an object that is PathLike, i.e. has __fspath__(). """ @@ -664,15 +664,21 @@ def test_open_with_path_like_object(self): self.assertTrue(os.path.exists(path)) cx.execute(self._sql) + def get_undecodable_path(self): + path = TESTFN_UNDECODABLE + if not path: + self.skipTest("only works if there are undecodable paths") + try: + open(path, 'wb').close() + except OSError: + self.skipTest(f"can't create file with undecodable path {path!r}") + unlink(path) + return path + @unittest.skipIf(sys.platform == "win32", "skipped on Windows") - @unittest.skipIf(sys.platform == "darwin", "skipped on macOS") - # TODO: RUSTPYTHON - # @unittest.skipIf(is_emscripten or is_wasi, "not supported on Emscripten/WASI") - @unittest.skipUnless(TESTFN_UNDECODABLE, "only works if there are undecodable paths") def test_open_with_undecodable_path(self): - path = TESTFN_UNDECODABLE + path = self.get_undecodable_path() self.addCleanup(unlink, path) - self.assertFalse(os.path.exists(path)) with contextlib.closing(sqlite.connect(path)) as cx: self.assertTrue(os.path.exists(path)) cx.execute(self._sql) @@ -712,21 +718,15 @@ def test_open_uri_readonly(self): cx.execute(self._sql) @unittest.skipIf(sys.platform == "win32", "skipped on Windows") - @unittest.skipIf(sys.platform == "darwin", "skipped on macOS") - # TODO: RUSTPYTHON - # @unittest.skipIf(is_emscripten or is_wasi, "not supported on Emscripten/WASI") - @unittest.skipUnless(TESTFN_UNDECODABLE, "only works if there are undecodable paths") def test_open_undecodable_uri(self): - path = TESTFN_UNDECODABLE + path = self.get_undecodable_path() self.addCleanup(unlink, path) uri = "file:" + urllib.parse.quote(path) - self.assertFalse(os.path.exists(path)) with contextlib.closing(sqlite.connect(uri, uri=True)) as cx: self.assertTrue(os.path.exists(path)) cx.execute(self._sql) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_factory_database_arg(self): def factory(database, *args, **kwargs): nonlocal database_arg @@ -875,6 +875,34 @@ def __getitem__(slf, x): with self.assertRaises(ZeroDivisionError): self.cu.execute("select name from test where name=?", L()) + @unittest.expectedFailure # TODO: RUSTPYTHON mixed named and positional parameters not validated + def test_execute_named_param_and_sequence(self): + dataset = ( + ("select :a", (1,)), + ("select :a, ?, ?", (1, 2, 3)), + ("select ?, :b, ?", (1, 2, 3)), + ("select ?, ?, :c", (1, 2, 3)), + ("select :a, :b, ?", (1, 2, 3)), + ) + msg = "Binding.*is a named parameter" + for query, params in dataset: + with self.subTest(query=query, params=params): + with self.assertRaisesRegex(sqlite.ProgrammingError, msg) as cm: + self.cu.execute(query, params) + + def test_execute_indexed_nameless_params(self): + # See gh-117995: "'?1' is considered a named placeholder" + for query, params, expected in ( + ("select ?1, ?2", (1, 2), (1, 2)), + ("select ?2, ?1", (1, 2), (2, 1)), + ): + with self.subTest(query=query, params=params): + with warnings.catch_warnings(): + warnings.simplefilter("error", DeprecationWarning) + cu = self.cu.execute(query, params) + actual, = cu.fetchall() + self.assertEqual(actual, expected) + def test_execute_too_many_params(self): category = sqlite.SQLITE_LIMIT_VARIABLE_NUMBER msg = "too many SQL variables" @@ -1050,7 +1078,7 @@ def test_array_size(self): # now set to 2 self.cu.arraysize = 2 - # now make the query return 3 rows + # now make the query return 2 rows from a table of 3 rows self.cu.execute("delete from test") self.cu.execute("insert into test(name) values ('A')") self.cu.execute("insert into test(name) values ('B')") @@ -1060,13 +1088,53 @@ def test_array_size(self): self.assertEqual(len(res), 2) + @unittest.expectedFailure # TODO: RUSTPYTHON arraysize validation not implemented + def test_invalid_array_size(self): + UINT32_MAX = (1 << 32) - 1 + setter = functools.partial(setattr, self.cu, 'arraysize') + + self.assertRaises(TypeError, setter, 1.0) + self.assertRaises(ValueError, setter, -3) + self.assertRaises(OverflowError, setter, UINT32_MAX + 1) + + @unittest.expectedFailure # TODO: RUSTPYTHON fetchmany behavior with exhausted cursor differs def test_fetchmany(self): + # no active SQL statement + res = self.cu.fetchmany() + self.assertEqual(res, []) + res = self.cu.fetchmany(1000) + self.assertEqual(res, []) + + # test default parameter + self.cu.execute("select name from test") + res = self.cu.fetchmany() + self.assertEqual(len(res), 1) + + # test when the number of requested rows exceeds the actual count self.cu.execute("select name from test") res = self.cu.fetchmany(100) self.assertEqual(len(res), 1) res = self.cu.fetchmany(100) self.assertEqual(res, []) + # test when size = 0 + self.cu.execute("select name from test") + res = self.cu.fetchmany(0) + self.assertEqual(res, []) + res = self.cu.fetchmany(100) + self.assertEqual(len(res), 1) + res = self.cu.fetchmany(100) + self.assertEqual(res, []) + + @unittest.expectedFailure # TODO: RUSTPYTHON fetchmany size validation not implemented + def test_invalid_fetchmany(self): + UINT32_MAX = (1 << 32) - 1 + fetchmany = self.cu.fetchmany + + self.assertRaises(TypeError, fetchmany, 1.0) + self.assertRaises(ValueError, fetchmany, -3) + self.assertRaises(OverflowError, fetchmany, UINT32_MAX + 1) + def test_fetchmany_kw_arg(self): """Checks if fetchmany works with keyword arguments""" self.cu.execute("select name from test") @@ -1186,12 +1254,9 @@ def test_blob_seek_and_tell(self): self.blob.seek(-10, SEEK_END) self.assertEqual(self.blob.tell(), 40) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_blob_seek_error(self): msg_oor = "offset out of blob range" msg_orig = "'origin' should be os.SEEK_SET, os.SEEK_CUR, or os.SEEK_END" - msg_of = "seek offset results in overflow" dataset = ( (ValueError, msg_oor, lambda: self.blob.seek(1000)), @@ -1203,12 +1268,15 @@ def test_blob_seek_error(self): with self.subTest(exc=exc, msg=msg, fn=fn): self.assertRaisesRegex(exc, msg, fn) + def test_blob_seek_overflow_error(self): # Force overflow errors + msg_of = "seek offset results in overflow" + _testcapi = import_helper.import_module("_testcapi") self.blob.seek(1, SEEK_SET) with self.assertRaisesRegex(OverflowError, msg_of): - self.blob.seek(INT_MAX, SEEK_CUR) + self.blob.seek(_testcapi.INT_MAX, SEEK_CUR) with self.assertRaisesRegex(OverflowError, msg_of): - self.blob.seek(INT_MAX, SEEK_END) + self.blob.seek(_testcapi.INT_MAX, SEEK_END) def test_blob_read(self): buf = self.blob.read() @@ -1362,24 +1430,24 @@ def test_blob_mapping_invalid_index_type(self): with self.assertRaisesRegex(TypeError, msg): self.blob["a"] = b"b" - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_blob_get_item_error(self): dataset = [len(self.blob), 105, -105] for idx in dataset: with self.subTest(idx=idx): with self.assertRaisesRegex(IndexError, "index out of range"): self.blob[idx] - with self.assertRaisesRegex(IndexError, "cannot fit 'int'"): - self.blob[ULLONG_MAX] # Provoke read error self.cx.execute("update test set b='aaaa' where rowid=1") with self.assertRaises(sqlite.OperationalError): self.blob[0] - # TODO: RUSTPYTHON - @unittest.expectedFailure + def test_blob_get_item_error_bigint(self): + _testcapi = import_helper.import_module("_testcapi") + with self.assertRaisesRegex(IndexError, "cannot fit 'int'"): + self.blob[_testcapi.ULLONG_MAX] + + @unittest.expectedFailure # TODO: RUSTPYTHON def test_blob_set_item_error(self): with self.assertRaisesRegex(TypeError, "cannot be interpreted"): self.blob[0] = b"multiple" @@ -1416,7 +1484,7 @@ def test_blob_sequence_not_supported(self): self.blob + self.blob with self.assertRaisesRegex(TypeError, "unsupported operand"): self.blob * 5 - with self.assertRaisesRegex(TypeError, "is not iterable"): + with self.assertRaisesRegex(TypeError, "is not.+iterable"): b"a" in self.blob def test_blob_context_manager(self): @@ -1477,9 +1545,16 @@ def test_blob_closed_db_read(self): "Cannot operate on a closed database", blob.read) + def test_blob_32bit_rowid(self): + # gh-100370: we should not get an OverflowError for 32-bit rowids + with memory_database() as cx: + rowid = 2**32 + cx.execute("create table t(t blob)") + cx.execute("insert into t(rowid, t) values (?, zeroblob(1))", (rowid,)) + cx.blobopen('t', 't', rowid) -# TODO: RUSTPYTHON -# @threading_helper.requires_working_threading() + +@threading_helper.requires_working_threading() class ThreadTests(unittest.TestCase): def setUp(self): self.con = sqlite.connect(":memory:") @@ -1532,8 +1607,7 @@ def test_check_connection_thread(self): with self.subTest(fn=fn): self._run_test(fn) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_check_cursor_thread(self): fns = [ lambda: self.cur.execute("insert into test(name) values('a')"), @@ -1554,12 +1628,12 @@ def run(con, err): except sqlite.Error: err.append("multi-threading not allowed") - con = sqlite.connect(":memory:", check_same_thread=False) - err = [] - t = threading.Thread(target=run, kwargs={"con": con, "err": err}) - t.start() - t.join() - self.assertEqual(len(err), 0, "\n".join(err)) + with memory_database(check_same_thread=False) as con: + err = [] + t = threading.Thread(target=run, kwargs={"con": con, "err": err}) + t.start() + t.join() + self.assertEqual(len(err), 0, "\n".join(err)) class ConstructorTests(unittest.TestCase): @@ -1585,9 +1659,16 @@ def test_binary(self): b = sqlite.Binary(b"\0'") class ExtensionTests(unittest.TestCase): + def setUp(self): + self.con = sqlite.connect(":memory:") + self.cur = self.con.cursor() + + def tearDown(self): + self.cur.close() + self.con.close() + def test_script_string_sql(self): - con = sqlite.connect(":memory:") - cur = con.cursor() + cur = self.cur cur.executescript(""" -- bla bla /* a stupid comment */ @@ -1599,40 +1680,40 @@ def test_script_string_sql(self): self.assertEqual(res, 5) def test_script_syntax_error(self): - con = sqlite.connect(":memory:") - cur = con.cursor() with self.assertRaises(sqlite.OperationalError): - cur.executescript("create table test(x); asdf; create table test2(x)") + self.cur.executescript(""" + CREATE TABLE test(x); + asdf; + CREATE TABLE test2(x) + """) def test_script_error_normal(self): - con = sqlite.connect(":memory:") - cur = con.cursor() with self.assertRaises(sqlite.OperationalError): - cur.executescript("create table test(sadfsadfdsa); select foo from hurz;") + self.cur.executescript(""" + CREATE TABLE test(sadfsadfdsa); + SELECT foo FROM hurz; + """) def test_cursor_executescript_as_bytes(self): - con = sqlite.connect(":memory:") - cur = con.cursor() with self.assertRaises(TypeError): - cur.executescript(b"create table test(foo); insert into test(foo) values (5);") + self.cur.executescript(b""" + CREATE TABLE test(foo); + INSERT INTO test(foo) VALUES (5); + """) def test_cursor_executescript_with_null_characters(self): - con = sqlite.connect(":memory:") - cur = con.cursor() with self.assertRaises(ValueError): - cur.executescript(""" - create table a(i);\0 - insert into a(i) values (5); - """) + self.cur.executescript(""" + CREATE TABLE a(i);\0 + INSERT INTO a(i) VALUES (5); + """) def test_cursor_executescript_with_surrogates(self): - con = sqlite.connect(":memory:") - cur = con.cursor() with self.assertRaises(UnicodeEncodeError): - cur.executescript(""" - create table a(s); - insert into a(s) values ('\ud8ff'); - """) + self.cur.executescript(""" + CREATE TABLE a(s); + INSERT INTO a(s) VALUES ('\ud8ff'); + """) def test_cursor_executescript_too_large_script(self): msg = "query string is too large" @@ -1642,19 +1723,18 @@ def test_cursor_executescript_too_large_script(self): cx.executescript("select 'too large'".ljust(lim+1)) def test_cursor_executescript_tx_control(self): - con = sqlite.connect(":memory:") + con = self.con con.execute("begin") self.assertTrue(con.in_transaction) con.executescript("select 1") self.assertFalse(con.in_transaction) def test_connection_execute(self): - con = sqlite.connect(":memory:") - result = con.execute("select 5").fetchone()[0] + result = self.con.execute("select 5").fetchone()[0] self.assertEqual(result, 5, "Basic test of Connection.execute") def test_connection_executemany(self): - con = sqlite.connect(":memory:") + con = self.con con.execute("create table test(foo)") con.executemany("insert into test(foo) values (?)", [(3,), (4,)]) result = con.execute("select foo from test order by foo").fetchall() @@ -1662,47 +1742,50 @@ def test_connection_executemany(self): self.assertEqual(result[1][0], 4, "Basic test of Connection.executemany") def test_connection_executescript(self): - con = sqlite.connect(":memory:") - con.executescript("create table test(foo); insert into test(foo) values (5);") + con = self.con + con.executescript(""" + CREATE TABLE test(foo); + INSERT INTO test(foo) VALUES (5); + """) result = con.execute("select foo from test").fetchone()[0] self.assertEqual(result, 5, "Basic test of Connection.executescript") + class ClosedConTests(unittest.TestCase): + def check(self, fn, *args, **kwds): + regex = "Cannot operate on a closed database." + with self.assertRaisesRegex(sqlite.ProgrammingError, regex): + fn(*args, **kwds) + + def setUp(self): + self.con = sqlite.connect(":memory:") + self.cur = self.con.cursor() + self.con.close() + + @unittest.expectedFailure # TODO: RUSTPYTHON error message differs for closed connection def test_closed_con_cursor(self): - con = sqlite.connect(":memory:") - con.close() - with self.assertRaises(sqlite.ProgrammingError): - cur = con.cursor() + self.check(self.con.cursor) + @unittest.expectedFailure # TODO: RUSTPYTHON error message differs for closed connection def test_closed_con_commit(self): - con = sqlite.connect(":memory:") - con.close() - with self.assertRaises(sqlite.ProgrammingError): - con.commit() + self.check(self.con.commit) + @unittest.expectedFailure # TODO: RUSTPYTHON error message differs for closed connection def test_closed_con_rollback(self): - con = sqlite.connect(":memory:") - con.close() - with self.assertRaises(sqlite.ProgrammingError): - con.rollback() + self.check(self.con.rollback) + @unittest.expectedFailure # TODO: RUSTPYTHON error message differs for closed connection def test_closed_cur_execute(self): - con = sqlite.connect(":memory:") - cur = con.cursor() - con.close() - with self.assertRaises(sqlite.ProgrammingError): - cur.execute("select 4") + self.check(self.cur.execute, "select 4") + @unittest.expectedFailure # TODO: RUSTPYTHON error message differs for closed connection def test_closed_create_function(self): - con = sqlite.connect(":memory:") - con.close() - def f(x): return 17 - with self.assertRaises(sqlite.ProgrammingError): - con.create_function("foo", 1, f) + def f(x): + return 17 + self.check(self.con.create_function, "foo", 1, f) + @unittest.expectedFailure # TODO: RUSTPYTHON error message differs for closed connection def test_closed_create_aggregate(self): - con = sqlite.connect(":memory:") - con.close() class Agg: def __init__(self): pass @@ -1710,34 +1793,28 @@ def step(self, x): pass def finalize(self): return 17 - with self.assertRaises(sqlite.ProgrammingError): - con.create_aggregate("foo", 1, Agg) + self.check(self.con.create_aggregate, "foo", 1, Agg) + @unittest.expectedFailure # TODO: RUSTPYTHON error message differs for closed connection def test_closed_set_authorizer(self): - con = sqlite.connect(":memory:") - con.close() def authorizer(*args): return sqlite.DENY - with self.assertRaises(sqlite.ProgrammingError): - con.set_authorizer(authorizer) + self.check(self.con.set_authorizer, authorizer) + @unittest.expectedFailure # TODO: RUSTPYTHON error message differs for closed connection def test_closed_set_progress_callback(self): - con = sqlite.connect(":memory:") - con.close() - def progress(): pass - with self.assertRaises(sqlite.ProgrammingError): - con.set_progress_handler(progress, 100) + def progress(): + pass + self.check(self.con.set_progress_handler, progress, 100) + @unittest.expectedFailure # TODO: RUSTPYTHON error message differs for closed connection def test_closed_call(self): - con = sqlite.connect(":memory:") - con.close() - with self.assertRaises(sqlite.ProgrammingError): - con() + self.check(self.con) + -class ClosedCurTests(unittest.TestCase): +class ClosedCurTests(MemoryDatabaseMixin, unittest.TestCase): def test_closed(self): - con = sqlite.connect(":memory:") - cur = con.cursor() + cur = self.cx.cursor() cur.close() for method_name in ("execute", "executemany", "executescript", "fetchall", "fetchmany", "fetchone"): @@ -1846,14 +1923,14 @@ def test_on_conflict_replace(self): self.assertEqual(self.cu.fetchall(), [('Very different data!', 'foo')]) -# TODO: RUSTPYTHON -# @requires_subprocess() +@requires_subprocess() class MultiprocessTests(unittest.TestCase): - CONNECTION_TIMEOUT = SHORT_TIMEOUT / 1000. # Defaults to 30 ms + CONNECTION_TIMEOUT = 0 # Disable the busy timeout. def tearDown(self): unlink(TESTFN) + @unittest.expectedFailure # TODO: RUSTPYTHON multiprocess test fails def test_ctx_mgr_rollback_if_commit_failed(self): # bpo-27334: ctx manager does not rollback if commit fails SCRIPT = f"""if 1: @@ -1919,5 +1996,71 @@ def wait(): self.assertEqual(proc.returncode, 0) +class RowTests(unittest.TestCase): + + def setUp(self): + self.cx = sqlite.connect(":memory:") + self.cx.row_factory = sqlite.Row + + def tearDown(self): + self.cx.close() + + def test_row_keys(self): + cu = self.cx.execute("SELECT 1 as first, 2 as second") + row = cu.fetchone() + self.assertEqual(row.keys(), ["first", "second"]) + + def test_row_length(self): + cu = self.cx.execute("SELECT 1, 2, 3") + row = cu.fetchone() + self.assertEqual(len(row), 3) + + def test_row_getitem(self): + cu = self.cx.execute("SELECT 1 as a, 2 as b") + row = cu.fetchone() + self.assertEqual(row[0], 1) + self.assertEqual(row[1], 2) + self.assertEqual(row["a"], 1) + self.assertEqual(row["b"], 2) + for key in "nokey", 4, 1.2: + with self.subTest(key=key): + with self.assertRaises(IndexError): + row[key] + + def test_row_equality(self): + c1 = self.cx.execute("SELECT 1 as a") + r1 = c1.fetchone() + + c2 = self.cx.execute("SELECT 1 as a") + r2 = c2.fetchone() + + self.assertIsNot(r1, r2) + self.assertEqual(r1, r2) + + c3 = self.cx.execute("SELECT 1 as b") + r3 = c3.fetchone() + + self.assertNotEqual(r1, r3) + + @unittest.expectedFailure # TODO: RUSTPYTHON Row with no description fails + def test_row_no_description(self): + cu = self.cx.cursor() + self.assertIsNone(cu.description) + + row = sqlite.Row(cu, ()) + self.assertEqual(row.keys(), []) + with self.assertRaisesRegex(IndexError, "nokey"): + row["nokey"] + + def test_row_is_a_sequence(self): + from collections.abc import Sequence + + cu = self.cx.execute("SELECT 1") + row = cu.fetchone() + + self.assertIsSubclass(sqlite.Row, Sequence) + self.assertIsInstance(row, Sequence) + + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_sqlite3/test_dump.py b/Lib/test/test_sqlite3/test_dump.py index ec4a11da8b0..74aacc05c2b 100644 --- a/Lib/test/test_sqlite3/test_dump.py +++ b/Lib/test/test_sqlite3/test_dump.py @@ -1,22 +1,18 @@ # Author: Paul Kippes <kippesp@gmail.com> import unittest -import sqlite3 as sqlite -from .test_dbapi import memory_database +from .util import memory_database +from .util import MemoryDatabaseMixin +from .util import requires_virtual_table -# TODO: RUSTPYTHON -@unittest.expectedFailure -class DumpTests(unittest.TestCase): - def setUp(self): - self.cx = sqlite.connect(":memory:") - self.cu = self.cx.cursor() - def tearDown(self): - self.cx.close() +class DumpTests(MemoryDatabaseMixin, unittest.TestCase): + @unittest.expectedFailure # TODO: RUSTPYTHON def test_table_dump(self): expected_sqls = [ + "PRAGMA foreign_keys=OFF;", """CREATE TABLE "index"("index" blob);""" , """INSERT INTO "index" VALUES(X'01');""" @@ -27,7 +23,8 @@ def test_table_dump(self): , "CREATE TABLE t1(id integer primary key, s1 text, " \ "t1_i1 integer not null, i2 integer, unique (s1), " \ - "constraint t1_idx1 unique (i2));" + "constraint t1_idx1 unique (i2), " \ + "constraint t1_i1_idx1 unique (t1_i1));" , "INSERT INTO \"t1\" VALUES(1,'foo',10,20);" , @@ -37,6 +34,9 @@ def test_table_dump(self): "t2_i2 integer, primary key (id)," \ "foreign key(t2_i1) references t1(t1_i1));" , + # Foreign key violation. + "INSERT INTO \"t2\" VALUES(1,2,3);" + , "CREATE TRIGGER trigger_1 update of t1_i1 on t1 " \ "begin " \ "update t2 set t2_i1 = new.t1_i1 where t2_i1 = old.t1_i1; " \ @@ -48,11 +48,87 @@ def test_table_dump(self): [self.cu.execute(s) for s in expected_sqls] i = self.cx.iterdump() actual_sqls = [s for s in i] - expected_sqls = ['BEGIN TRANSACTION;'] + expected_sqls + \ - ['COMMIT;'] + expected_sqls = [ + "PRAGMA foreign_keys=OFF;", + "BEGIN TRANSACTION;", + *expected_sqls[1:], + "COMMIT;", + ] [self.assertEqual(expected_sqls[i], actual_sqls[i]) for i in range(len(expected_sqls))] + @unittest.expectedFailure # TODO: RUSTPYTHON iterdump filter parameter not implemented + def test_table_dump_filter(self): + all_table_sqls = [ + """CREATE TABLE "some_table_2" ("id_1" INTEGER);""", + """INSERT INTO "some_table_2" VALUES(3);""", + """INSERT INTO "some_table_2" VALUES(4);""", + """CREATE TABLE "test_table_1" ("id_2" INTEGER);""", + """INSERT INTO "test_table_1" VALUES(1);""", + """INSERT INTO "test_table_1" VALUES(2);""", + ] + all_views_sqls = [ + """CREATE VIEW "view_1" AS SELECT * FROM "some_table_2";""", + """CREATE VIEW "view_2" AS SELECT * FROM "test_table_1";""", + ] + # Create database structure. + for sql in [*all_table_sqls, *all_views_sqls]: + self.cu.execute(sql) + # %_table_% matches all tables. + dump_sqls = list(self.cx.iterdump(filter="%_table_%")) + self.assertEqual( + dump_sqls, + ["BEGIN TRANSACTION;", *all_table_sqls, "COMMIT;"], + ) + # view_% matches all views. + dump_sqls = list(self.cx.iterdump(filter="view_%")) + self.assertEqual( + dump_sqls, + ["BEGIN TRANSACTION;", *all_views_sqls, "COMMIT;"], + ) + # %_1 matches tables and views with the _1 suffix. + dump_sqls = list(self.cx.iterdump(filter="%_1")) + self.assertEqual( + dump_sqls, + [ + "BEGIN TRANSACTION;", + """CREATE TABLE "test_table_1" ("id_2" INTEGER);""", + """INSERT INTO "test_table_1" VALUES(1);""", + """INSERT INTO "test_table_1" VALUES(2);""", + """CREATE VIEW "view_1" AS SELECT * FROM "some_table_2";""", + "COMMIT;" + ], + ) + # some_% matches some_table_2. + dump_sqls = list(self.cx.iterdump(filter="some_%")) + self.assertEqual( + dump_sqls, + [ + "BEGIN TRANSACTION;", + """CREATE TABLE "some_table_2" ("id_1" INTEGER);""", + """INSERT INTO "some_table_2" VALUES(3);""", + """INSERT INTO "some_table_2" VALUES(4);""", + "COMMIT;" + ], + ) + # Only single object. + dump_sqls = list(self.cx.iterdump(filter="view_2")) + self.assertEqual( + dump_sqls, + [ + "BEGIN TRANSACTION;", + """CREATE VIEW "view_2" AS SELECT * FROM "test_table_1";""", + "COMMIT;" + ], + ) + # % matches all objects. + dump_sqls = list(self.cx.iterdump(filter="%")) + self.assertEqual( + dump_sqls, + ["BEGIN TRANSACTION;", *all_table_sqls, *all_views_sqls, "COMMIT;"], + ) + + @unittest.expectedFailure # TODO: RUSTPYTHON _iterdump not implemented def test_dump_autoincrement(self): expected = [ 'CREATE TABLE "t1" (id integer primary key autoincrement);', @@ -73,6 +149,7 @@ def test_dump_autoincrement(self): actual = [stmt for stmt in self.cx.iterdump()] self.assertEqual(expected, actual) + @unittest.expectedFailure # TODO: RUSTPYTHON _iterdump not implemented def test_dump_autoincrement_create_new_db(self): self.cu.execute("BEGIN TRANSACTION") self.cu.execute("CREATE TABLE t1 (id integer primary key autoincrement)") @@ -98,6 +175,7 @@ def test_dump_autoincrement_create_new_db(self): rows = res.fetchall() self.assertEqual(rows[0][0], seq) + @unittest.expectedFailure # TODO: RUSTPYTHON _iterdump not implemented def test_unorderable_row(self): # iterdump() should be able to cope with unorderable row types (issue #15545) class UnorderableRow: @@ -119,6 +197,44 @@ def __getitem__(self, index): got = list(self.cx.iterdump()) self.assertEqual(expected, got) + @unittest.expectedFailure # TODO: RUSTPYTHON _iterdump not implemented + def test_dump_custom_row_factory(self): + # gh-118221: iterdump should be able to cope with custom row factories. + def dict_factory(cu, row): + fields = [col[0] for col in cu.description] + return dict(zip(fields, row)) + + self.cx.row_factory = dict_factory + CREATE_TABLE = "CREATE TABLE test(t);" + expected = ["BEGIN TRANSACTION;", CREATE_TABLE, "COMMIT;"] + + self.cu.execute(CREATE_TABLE) + actual = list(self.cx.iterdump()) + self.assertEqual(expected, actual) + self.assertEqual(self.cx.row_factory, dict_factory) + + @unittest.expectedFailure # TODO: RUSTPYTHON _iterdump not implemented + @requires_virtual_table("fts4") + def test_dump_virtual_tables(self): + # gh-64662 + expected = [ + "BEGIN TRANSACTION;", + "PRAGMA writable_schema=ON;", + ("INSERT INTO sqlite_master(type,name,tbl_name,rootpage,sql)" + "VALUES('table','test','test',0,'CREATE VIRTUAL TABLE test USING fts4(example)');"), + "CREATE TABLE 'test_content'(docid INTEGER PRIMARY KEY, 'c0example');", + "CREATE TABLE 'test_docsize'(docid INTEGER PRIMARY KEY, size BLOB);", + ("CREATE TABLE 'test_segdir'(level INTEGER,idx INTEGER,start_block INTEGER," + "leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx));"), + "CREATE TABLE 'test_segments'(blockid INTEGER PRIMARY KEY, block BLOB);", + "CREATE TABLE 'test_stat'(id INTEGER PRIMARY KEY, value BLOB);", + "PRAGMA writable_schema=OFF;", + "COMMIT;" + ] + self.cu.execute("CREATE VIRTUAL TABLE test USING fts4(example)") + actual = list(self.cx.iterdump()) + self.assertEqual(expected, actual) + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_sqlite3/test_factory.py b/Lib/test/test_sqlite3/test_factory.py index c13a7481520..d19b98b2056 100644 --- a/Lib/test/test_sqlite3/test_factory.py +++ b/Lib/test/test_sqlite3/test_factory.py @@ -24,6 +24,9 @@ import sqlite3 as sqlite from collections.abc import Sequence +from .util import memory_database +from .util import MemoryDatabaseMixin + def dict_factory(cursor, row): d = {} @@ -37,6 +40,7 @@ def __init__(self, *args, **kwargs): self.row_factory = dict_factory class ConnectionFactoryTests(unittest.TestCase): + @unittest.expectedFailure # TODO: RUSTPYTHON def test_connection_factories(self): class DefectFactory(sqlite.Connection): def __init__(self, *args, **kwargs): @@ -45,13 +49,14 @@ class OkFactory(sqlite.Connection): def __init__(self, *args, **kwargs): sqlite.Connection.__init__(self, *args, **kwargs) - for factory in DefectFactory, OkFactory: - with self.subTest(factory=factory): - con = sqlite.connect(":memory:", factory=factory) - self.assertIsInstance(con, factory) + with memory_database(factory=OkFactory) as con: + self.assertIsInstance(con, OkFactory) + regex = "Base Connection.__init__ not called." + with self.assertRaisesRegex(sqlite.ProgrammingError, regex): + with memory_database(factory=DefectFactory) as con: + self.assertIsInstance(con, DefectFactory) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_connection_factory_relayed_call(self): # gh-95132: keyword args must not be passed as positional args class Factory(sqlite.Connection): @@ -59,29 +64,35 @@ def __init__(self, *args, **kwargs): kwargs["isolation_level"] = None super(Factory, self).__init__(*args, **kwargs) - con = sqlite.connect(":memory:", factory=Factory) - self.assertIsNone(con.isolation_level) - self.assertIsInstance(con, Factory) + with memory_database(factory=Factory) as con: + self.assertIsNone(con.isolation_level) + self.assertIsInstance(con, Factory) + @unittest.expectedFailure # TODO: RUSTPYTHON def test_connection_factory_as_positional_arg(self): class Factory(sqlite.Connection): def __init__(self, *args, **kwargs): super(Factory, self).__init__(*args, **kwargs) - con = sqlite.connect(":memory:", 5.0, 0, None, True, Factory) - self.assertIsNone(con.isolation_level) - self.assertIsInstance(con, Factory) + regex = ( + r"Passing more than 1 positional argument to _sqlite3.Connection\(\) " + r"is deprecated. Parameters 'timeout', 'detect_types', " + r"'isolation_level', 'check_same_thread', 'factory', " + r"'cached_statements' and 'uri' will become keyword-only " + r"parameters in Python 3.15." + ) + with self.assertWarnsRegex(DeprecationWarning, regex) as cm: + with memory_database(5.0, 0, None, True, Factory) as con: + self.assertIsNone(con.isolation_level) + self.assertIsInstance(con, Factory) + self.assertEqual(cm.filename, __file__) -class CursorFactoryTests(unittest.TestCase): - def setUp(self): - self.con = sqlite.connect(":memory:") +class CursorFactoryTests(MemoryDatabaseMixin, unittest.TestCase): def tearDown(self): self.con.close() - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_is_instance(self): cur = self.con.cursor() self.assertIsInstance(cur, sqlite.Cursor) @@ -98,12 +109,10 @@ def test_invalid_factory(self): # invalid callable returning non-cursor self.assertRaises(TypeError, self.con.cursor, lambda con: None) -class RowFactoryTestsBackwardsCompat(unittest.TestCase): - def setUp(self): - self.con = sqlite.connect(":memory:") - # TODO: RUSTPYTHON - @unittest.expectedFailure +class RowFactoryTestsBackwardsCompat(MemoryDatabaseMixin, unittest.TestCase): + + @unittest.expectedFailure # TODO: RUSTPYTHON def test_is_produced_by_factory(self): cur = self.con.cursor(factory=MyCursor) cur.execute("select 4+5 as foo") @@ -111,22 +120,20 @@ def test_is_produced_by_factory(self): self.assertIsInstance(row, dict) cur.close() - def tearDown(self): - self.con.close() -class RowFactoryTests(unittest.TestCase): +class RowFactoryTests(MemoryDatabaseMixin, unittest.TestCase): + def setUp(self): - self.con = sqlite.connect(":memory:") + super().setUp() + self.con.row_factory = sqlite.Row def test_custom_factory(self): self.con.row_factory = lambda cur, row: list(row) row = self.con.execute("select 1, 2").fetchone() self.assertIsInstance(row, list) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_sqlite_row_index(self): - self.con.row_factory = sqlite.Row row = self.con.execute("select 1 as a_1, 2 as b").fetchone() self.assertIsInstance(row, sqlite.Row) @@ -156,10 +163,8 @@ def test_sqlite_row_index(self): with self.assertRaises(IndexError): row[complex()] # index must be int or string - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_sqlite_row_index_unicode(self): - self.con.row_factory = sqlite.Row row = self.con.execute("select 1 as \xff").fetchone() self.assertEqual(row["\xff"], 1) with self.assertRaises(IndexError): @@ -169,7 +174,6 @@ def test_sqlite_row_index_unicode(self): def test_sqlite_row_slice(self): # A sqlite.Row can be sliced like a list. - self.con.row_factory = sqlite.Row row = self.con.execute("select 1, 2, 3, 4").fetchone() self.assertEqual(row[0:0], ()) self.assertEqual(row[0:1], (1,)) @@ -186,30 +190,32 @@ def test_sqlite_row_slice(self): self.assertEqual(row[3:0:-2], (4, 2)) def test_sqlite_row_iter(self): - """Checks if the row object is iterable""" - self.con.row_factory = sqlite.Row + # Checks if the row object is iterable. row = self.con.execute("select 1 as a, 2 as b").fetchone() - for col in row: - pass + + # Is iterable in correct order and produces valid results: + items = [col for col in row] + self.assertEqual(items, [1, 2]) + + # Is iterable the second time: + items = [col for col in row] + self.assertEqual(items, [1, 2]) def test_sqlite_row_as_tuple(self): - """Checks if the row object can be converted to a tuple""" - self.con.row_factory = sqlite.Row + # Checks if the row object can be converted to a tuple. row = self.con.execute("select 1 as a, 2 as b").fetchone() t = tuple(row) self.assertEqual(t, (row['a'], row['b'])) def test_sqlite_row_as_dict(self): - """Checks if the row object can be correctly converted to a dictionary""" - self.con.row_factory = sqlite.Row + # Checks if the row object can be correctly converted to a dictionary. row = self.con.execute("select 1 as a, 2 as b").fetchone() d = dict(row) self.assertEqual(d["a"], row["a"]) self.assertEqual(d["b"], row["b"]) def test_sqlite_row_hash_cmp(self): - """Checks if the row object compares and hashes correctly""" - self.con.row_factory = sqlite.Row + # Checks if the row object compares and hashes correctly. row_1 = self.con.execute("select 1 as a, 2 as b").fetchone() row_2 = self.con.execute("select 1 as a, 2 as b").fetchone() row_3 = self.con.execute("select 1 as a, 3 as b").fetchone() @@ -242,30 +248,29 @@ def test_sqlite_row_hash_cmp(self): self.assertEqual(hash(row_1), hash(row_2)) def test_sqlite_row_as_sequence(self): - """ Checks if the row object can act like a sequence """ - self.con.row_factory = sqlite.Row + # Checks if the row object can act like a sequence. row = self.con.execute("select 1 as a, 2 as b").fetchone() as_tuple = tuple(row) self.assertEqual(list(reversed(row)), list(reversed(as_tuple))) self.assertIsInstance(row, Sequence) + def test_sqlite_row_keys(self): + # Checks if the row object can return a list of columns as strings. + row = self.con.execute("select 1 as a, 2 as b").fetchone() + self.assertEqual(row.keys(), ['a', 'b']) + def test_fake_cursor_class(self): # Issue #24257: Incorrect use of PyObject_IsInstance() caused # segmentation fault. # Issue #27861: Also applies for cursor factory. class FakeCursor(str): __class__ = sqlite.Cursor - self.con.row_factory = sqlite.Row self.assertRaises(TypeError, self.con.cursor, FakeCursor) self.assertRaises(TypeError, sqlite.Row, FakeCursor(), ()) - def tearDown(self): - self.con.close() -class TextFactoryTests(unittest.TestCase): - def setUp(self): - self.con = sqlite.connect(":memory:") +class TextFactoryTests(MemoryDatabaseMixin, unittest.TestCase): def test_unicode(self): austria = "Österreich" @@ -284,17 +289,19 @@ def test_custom(self): austria = "Österreich" row = self.con.execute("select ?", (austria,)).fetchone() self.assertEqual(type(row[0]), str, "type of row[0] must be unicode") - self.assertTrue(row[0].endswith("reich"), "column must contain original data") + self.assertEndsWith(row[0], "reich", "column must contain original data") - def tearDown(self): - self.con.close() class TextFactoryTestsWithEmbeddedZeroBytes(unittest.TestCase): + def setUp(self): self.con = sqlite.connect(":memory:") self.con.execute("create table test (value text)") self.con.execute("insert into test (value) values (?)", ("a\x00b",)) + def tearDown(self): + self.con.close() + def test_string(self): # text_factory defaults to str row = self.con.execute("select value from test").fetchone() @@ -320,9 +327,6 @@ def test_custom(self): self.assertIs(type(row[0]), bytes) self.assertEqual(row[0], b"a\x00b") - def tearDown(self): - self.con.close() - if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_sqlite3/test_hooks.py b/Lib/test/test_sqlite3/test_hooks.py index 21042b9bf10..c47cfab180d 100644 --- a/Lib/test/test_sqlite3/test_hooks.py +++ b/Lib/test/test_sqlite3/test_hooks.py @@ -26,34 +26,31 @@ from test.support.os_helper import TESTFN, unlink -from test.test_sqlite3.test_dbapi import memory_database, cx_limit -from test.test_sqlite3.test_userfunctions import with_tracebacks +from .util import memory_database, cx_limit, with_tracebacks +from .util import MemoryDatabaseMixin -class CollationTests(unittest.TestCase): +class CollationTests(MemoryDatabaseMixin, unittest.TestCase): + def test_create_collation_not_string(self): - con = sqlite.connect(":memory:") with self.assertRaises(TypeError): - con.create_collation(None, lambda x, y: (x > y) - (x < y)) + self.con.create_collation(None, lambda x, y: (x > y) - (x < y)) def test_create_collation_not_callable(self): - con = sqlite.connect(":memory:") with self.assertRaises(TypeError) as cm: - con.create_collation("X", 42) + self.con.create_collation("X", 42) self.assertEqual(str(cm.exception), 'parameter must be callable') def test_create_collation_not_ascii(self): - con = sqlite.connect(":memory:") - con.create_collation("collä", lambda x, y: (x > y) - (x < y)) + self.con.create_collation("collä", lambda x, y: (x > y) - (x < y)) def test_create_collation_bad_upper(self): class BadUpperStr(str): def upper(self): return None - con = sqlite.connect(":memory:") mycoll = lambda x, y: -((x > y) - (x < y)) - con.create_collation(BadUpperStr("mycoll"), mycoll) - result = con.execute(""" + self.con.create_collation(BadUpperStr("mycoll"), mycoll) + result = self.con.execute(""" select x from ( select 'a' as x union @@ -68,8 +65,7 @@ def mycoll(x, y): # reverse order return -((x > y) - (x < y)) - con = sqlite.connect(":memory:") - con.create_collation("mycoll", mycoll) + self.con.create_collation("mycoll", mycoll) sql = """ select x from ( select 'a' as x @@ -79,21 +75,20 @@ def mycoll(x, y): select 'c' as x ) order by x collate mycoll """ - result = con.execute(sql).fetchall() + result = self.con.execute(sql).fetchall() self.assertEqual(result, [('c',), ('b',), ('a',)], msg='the expected order was not returned') - con.create_collation("mycoll", None) + self.con.create_collation("mycoll", None) with self.assertRaises(sqlite.OperationalError) as cm: - result = con.execute(sql).fetchall() + result = self.con.execute(sql).fetchall() self.assertEqual(str(cm.exception), 'no such collation sequence: mycoll') def test_collation_returns_large_integer(self): def mycoll(x, y): # reverse order return -((x > y) - (x < y)) * 2**32 - con = sqlite.connect(":memory:") - con.create_collation("mycoll", mycoll) + self.con.create_collation("mycoll", mycoll) sql = """ select x from ( select 'a' as x @@ -103,7 +98,7 @@ def mycoll(x, y): select 'c' as x ) order by x collate mycoll """ - result = con.execute(sql).fetchall() + result = self.con.execute(sql).fetchall() self.assertEqual(result, [('c',), ('b',), ('a',)], msg="the expected order was not returned") @@ -112,7 +107,7 @@ def test_collation_register_twice(self): Register two different collation functions under the same name. Verify that the last one is actually used. """ - con = sqlite.connect(":memory:") + con = self.con con.create_collation("mycoll", lambda x, y: (x > y) - (x < y)) con.create_collation("mycoll", lambda x, y: -((x > y) - (x < y))) result = con.execute(""" @@ -126,25 +121,26 @@ def test_deregister_collation(self): Register a collation, then deregister it. Make sure an error is raised if we try to use it. """ - con = sqlite.connect(":memory:") + con = self.con con.create_collation("mycoll", lambda x, y: (x > y) - (x < y)) con.create_collation("mycoll", None) with self.assertRaises(sqlite.OperationalError) as cm: con.execute("select 'a' as x union select 'b' as x order by x collate mycoll") self.assertEqual(str(cm.exception), 'no such collation sequence: mycoll') -class ProgressTests(unittest.TestCase): + +class ProgressTests(MemoryDatabaseMixin, unittest.TestCase): + def test_progress_handler_used(self): """ Test that the progress handler is invoked once it is set. """ - con = sqlite.connect(":memory:") progress_calls = [] def progress(): progress_calls.append(None) return 0 - con.set_progress_handler(progress, 1) - con.execute(""" + self.con.set_progress_handler(progress, 1) + self.con.execute(""" create table foo(a, b) """) self.assertTrue(progress_calls) @@ -153,7 +149,7 @@ def test_opcode_count(self): """ Test that the opcode argument is respected. """ - con = sqlite.connect(":memory:") + con = self.con progress_calls = [] def progress(): progress_calls.append(None) @@ -176,11 +172,10 @@ def test_cancel_operation(self): """ Test that returning a non-zero value stops the operation in progress. """ - con = sqlite.connect(":memory:") def progress(): return 1 - con.set_progress_handler(progress, 1) - curs = con.cursor() + self.con.set_progress_handler(progress, 1) + curs = self.con.cursor() self.assertRaises( sqlite.OperationalError, curs.execute, @@ -190,7 +185,7 @@ def test_clear_handler(self): """ Test that setting the progress handler to None clears the previously set handler. """ - con = sqlite.connect(":memory:") + con = self.con action = 0 def progress(): nonlocal action @@ -201,33 +196,47 @@ def progress(): con.execute("select 1 union select 2 union select 3").fetchall() self.assertEqual(action, 0, "progress handler was not cleared") - @with_tracebacks(ZeroDivisionError, name="bad_progress") + @unittest.expectedFailure # TODO: RUSTPYTHON + @with_tracebacks(ZeroDivisionError, msg_regex="bad_progress") def test_error_in_progress_handler(self): - con = sqlite.connect(":memory:") def bad_progress(): 1 / 0 - con.set_progress_handler(bad_progress, 1) + self.con.set_progress_handler(bad_progress, 1) with self.assertRaises(sqlite.OperationalError): - con.execute(""" + self.con.execute(""" create table foo(a, b) """) - @with_tracebacks(ZeroDivisionError, name="bad_progress") + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented + @with_tracebacks(ZeroDivisionError, msg_regex="bad_progress") def test_error_in_progress_handler_result(self): - con = sqlite.connect(":memory:") class BadBool: def __bool__(self): 1 / 0 def bad_progress(): return BadBool() - con.set_progress_handler(bad_progress, 1) + self.con.set_progress_handler(bad_progress, 1) with self.assertRaises(sqlite.OperationalError): - con.execute(""" + self.con.execute(""" create table foo(a, b) """) + @unittest.expectedFailure # TODO: RUSTPYTHON keyword-only arguments not supported for set_progress_handler + def test_progress_handler_keyword_args(self): + regex = ( + r"Passing keyword argument 'progress_handler' to " + r"_sqlite3.Connection.set_progress_handler\(\) is deprecated. " + r"Parameter 'progress_handler' will become positional-only in " + r"Python 3.15." + ) + + with self.assertWarnsRegex(DeprecationWarning, regex) as cm: + self.con.set_progress_handler(progress_handler=lambda: None, n=1) + self.assertEqual(cm.filename, __file__) + + +class TraceCallbackTests(MemoryDatabaseMixin, unittest.TestCase): -class TraceCallbackTests(unittest.TestCase): @contextlib.contextmanager def check_stmt_trace(self, cx, expected): try: @@ -242,12 +251,11 @@ def test_trace_callback_used(self): """ Test that the trace callback is invoked once it is set. """ - con = sqlite.connect(":memory:") traced_statements = [] def trace(statement): traced_statements.append(statement) - con.set_trace_callback(trace) - con.execute("create table foo(a, b)") + self.con.set_trace_callback(trace) + self.con.execute("create table foo(a, b)") self.assertTrue(traced_statements) self.assertTrue(any("create table foo" in stmt for stmt in traced_statements)) @@ -255,7 +263,7 @@ def test_clear_trace_callback(self): """ Test that setting the trace callback to None clears the previously set callback. """ - con = sqlite.connect(":memory:") + con = self.con traced_statements = [] def trace(statement): traced_statements.append(statement) @@ -269,7 +277,7 @@ def test_unicode_content(self): Test that the statement can contain unicode literals. """ unicode_value = '\xf6\xe4\xfc\xd6\xc4\xdc\xdf\u20ac' - con = sqlite.connect(":memory:") + con = self.con traced_statements = [] def trace(statement): traced_statements.append(statement) @@ -317,13 +325,14 @@ def test_trace_expanded_sql(self): cx.execute("create table t(t)") cx.executemany("insert into t values(?)", ((v,) for v in range(3))) + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented @with_tracebacks( sqlite.DataError, regex="Expanded SQL string exceeds the maximum string length" ) def test_trace_too_much_expanded_sql(self): # If the expanded string is too large, we'll fall back to the - # unexpanded SQL statement (for SQLite 3.14.0 and newer). + # unexpanded SQL statement. # The resulting string length is limited by the runtime limit # SQLITE_LIMIT_LENGTH. template = "select 1 as a where a=" @@ -334,8 +343,6 @@ def test_trace_too_much_expanded_sql(self): unexpanded_query = template + "?" expected = [unexpanded_query] - if sqlite.sqlite_version_info < (3, 14, 0): - expected = [] with self.check_stmt_trace(cx, expected): cx.execute(unexpanded_query, (bad_param,)) @@ -343,12 +350,26 @@ def test_trace_too_much_expanded_sql(self): with self.check_stmt_trace(cx, [expanded_query]): cx.execute(unexpanded_query, (ok_param,)) + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented @with_tracebacks(ZeroDivisionError, regex="division by zero") def test_trace_bad_handler(self): with memory_database() as cx: cx.set_trace_callback(lambda stmt: 5/0) cx.execute("select 1") + @unittest.expectedFailure # TODO: RUSTPYTHON keyword-only arguments not supported for set_trace_callback + def test_trace_keyword_args(self): + regex = ( + r"Passing keyword argument 'trace_callback' to " + r"_sqlite3.Connection.set_trace_callback\(\) is deprecated. " + r"Parameter 'trace_callback' will become positional-only in " + r"Python 3.15." + ) + + with self.assertWarnsRegex(DeprecationWarning, regex) as cm: + self.con.set_trace_callback(trace_callback=lambda: None) + self.assertEqual(cm.filename, __file__) + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_sqlite3/test_regression.py b/Lib/test/test_sqlite3/test_regression.py index d746be647c6..0ebd6d5e9da 100644 --- a/Lib/test/test_sqlite3/test_regression.py +++ b/Lib/test/test_sqlite3/test_regression.py @@ -28,15 +28,12 @@ from test import support from unittest.mock import patch -from test.test_sqlite3.test_dbapi import memory_database, cx_limit +from .util import memory_database, cx_limit +from .util import MemoryDatabaseMixin -class RegressionTests(unittest.TestCase): - def setUp(self): - self.con = sqlite.connect(":memory:") - def tearDown(self): - self.con.close() +class RegressionTests(MemoryDatabaseMixin, unittest.TestCase): def test_pragma_user_version(self): # This used to crash pysqlite because this pragma command returns NULL for the column name @@ -45,28 +42,24 @@ def test_pragma_user_version(self): def test_pragma_schema_version(self): # This still crashed pysqlite <= 2.2.1 - con = sqlite.connect(":memory:", detect_types=sqlite.PARSE_COLNAMES) - try: + with memory_database(detect_types=sqlite.PARSE_COLNAMES) as con: cur = self.con.cursor() cur.execute("pragma schema_version") - finally: - cur.close() - con.close() def test_statement_reset(self): # pysqlite 2.1.0 to 2.2.0 have the problem that not all statements are # reset before a rollback, but only those that are still in the # statement cache. The others are not accessible from the connection object. - con = sqlite.connect(":memory:", cached_statements=5) - cursors = [con.cursor() for x in range(5)] - cursors[0].execute("create table test(x)") - for i in range(10): - cursors[0].executemany("insert into test(x) values (?)", [(x,) for x in range(10)]) + with memory_database(cached_statements=5) as con: + cursors = [con.cursor() for x in range(5)] + cursors[0].execute("create table test(x)") + for i in range(10): + cursors[0].executemany("insert into test(x) values (?)", [(x,) for x in range(10)]) - for i in range(5): - cursors[i].execute(" " * i + "select x from test") + for i in range(5): + cursors[i].execute(" " * i + "select x from test") - con.rollback() + con.rollback() def test_column_name_with_spaces(self): cur = self.con.cursor() @@ -81,17 +74,15 @@ def test_statement_finalization_on_close_db(self): # cache when closing the database. statements that were still # referenced in cursors weren't closed and could provoke " # "OperationalError: Unable to close due to unfinalised statements". - con = sqlite.connect(":memory:") cursors = [] # default statement cache size is 100 for i in range(105): - cur = con.cursor() + cur = self.con.cursor() cursors.append(cur) cur.execute("select 1 x union select " + str(i)) - con.close() def test_on_conflict_rollback(self): - con = sqlite.connect(":memory:") + con = self.con con.execute("create table foo(x, unique(x) on conflict rollback)") con.execute("insert into foo(x) values (1)") try: @@ -126,16 +117,16 @@ def test_type_map_usage(self): a statement. This test exhibits the problem. """ SELECT = "select * from foo" - con = sqlite.connect(":memory:",detect_types=sqlite.PARSE_DECLTYPES) - cur = con.cursor() - cur.execute("create table foo(bar timestamp)") - with self.assertWarnsRegex(DeprecationWarning, "adapter"): - cur.execute("insert into foo(bar) values (?)", (datetime.datetime.now(),)) - cur.execute(SELECT) - cur.execute("drop table foo") - cur.execute("create table foo(bar integer)") - cur.execute("insert into foo(bar) values (5)") - cur.execute(SELECT) + with memory_database(detect_types=sqlite.PARSE_DECLTYPES) as con: + cur = con.cursor() + cur.execute("create table foo(bar timestamp)") + with self.assertWarnsRegex(DeprecationWarning, "adapter"): + cur.execute("insert into foo(bar) values (?)", (datetime.datetime.now(),)) + cur.execute(SELECT) + cur.execute("drop table foo") + cur.execute("create table foo(bar integer)") + cur.execute("insert into foo(bar) values (5)") + cur.execute(SELECT) def test_bind_mutating_list(self): # Issue41662: Crash when mutate a list of parameters during iteration. @@ -144,11 +135,11 @@ def __conform__(self, protocol): parameters.clear() return "..." parameters = [X(), 0] - con = sqlite.connect(":memory:",detect_types=sqlite.PARSE_DECLTYPES) - con.execute("create table foo(bar X, baz integer)") - # Should not crash - with self.assertRaises(IndexError): - con.execute("insert into foo(bar, baz) values (?, ?)", parameters) + with memory_database(detect_types=sqlite.PARSE_DECLTYPES) as con: + con.execute("create table foo(bar X, baz integer)") + # Should not crash + with self.assertRaises(IndexError): + con.execute("insert into foo(bar, baz) values (?, ?)", parameters) def test_error_msg_decode_error(self): # When porting the module to Python 3.0, the error message about @@ -173,7 +164,7 @@ def upper(self): def __del__(self): con.isolation_level = "" - con = sqlite.connect(":memory:") + con = self.con con.isolation_level = None for level in "", "DEFERRED", "IMMEDIATE", "EXCLUSIVE": with self.subTest(level=level): @@ -204,8 +195,7 @@ class Cursor(sqlite.Cursor): def __init__(self, con): pass - con = sqlite.connect(":memory:") - cur = Cursor(con) + cur = Cursor(self.con) with self.assertRaises(sqlite.ProgrammingError): cur.execute("select 4+5").fetchall() with self.assertRaisesRegex(sqlite.ProgrammingError, @@ -238,7 +228,9 @@ def test_auto_commit(self): 2.5.3 introduced a regression so that these could no longer be created. """ - con = sqlite.connect(":memory:", isolation_level=None) + with memory_database(isolation_level=None) as con: + self.assertIsNone(con.isolation_level) + self.assertFalse(con.in_transaction) def test_pragma_autocommit(self): """ @@ -266,7 +258,7 @@ def collation_cb(a, b): # Lone surrogate cannot be encoded to the default encoding (utf8) "\uDC80", collation_cb) - @unittest.skip("TODO: RUSTPYTHON deadlock") + @unittest.skip('TODO: RUSTPYTHON; recursive cursor use causes lock contention') def test_recursive_cursor_use(self): """ http://bugs.python.org/issue10811 @@ -274,9 +266,7 @@ def test_recursive_cursor_use(self): Recursively using a cursor, such as when reusing it from a generator led to segfaults. Now we catch recursive cursor usage and raise a ProgrammingError. """ - con = sqlite.connect(":memory:") - - cur = con.cursor() + cur = self.con.cursor() cur.execute("create table a (bar)") cur.execute("create table b (baz)") @@ -296,29 +286,31 @@ def test_convert_timestamp_microsecond_padding(self): since the microsecond string "456" actually represents "456000". """ - con = sqlite.connect(":memory:", detect_types=sqlite.PARSE_DECLTYPES) - cur = con.cursor() - cur.execute("CREATE TABLE t (x TIMESTAMP)") + with memory_database(detect_types=sqlite.PARSE_DECLTYPES) as con: + cur = con.cursor() + cur.execute("CREATE TABLE t (x TIMESTAMP)") - # Microseconds should be 456000 - cur.execute("INSERT INTO t (x) VALUES ('2012-04-04 15:06:00.456')") + # Microseconds should be 456000 + cur.execute("INSERT INTO t (x) VALUES ('2012-04-04 15:06:00.456')") - # Microseconds should be truncated to 123456 - cur.execute("INSERT INTO t (x) VALUES ('2012-04-04 15:06:00.123456789')") + # Microseconds should be truncated to 123456 + cur.execute("INSERT INTO t (x) VALUES ('2012-04-04 15:06:00.123456789')") - cur.execute("SELECT * FROM t") - with self.assertWarnsRegex(DeprecationWarning, "converter"): - values = [x[0] for x in cur.fetchall()] + cur.execute("SELECT * FROM t") + with self.assertWarnsRegex(DeprecationWarning, "converter"): + values = [x[0] for x in cur.fetchall()] - self.assertEqual(values, [ - datetime.datetime(2012, 4, 4, 15, 6, 0, 456000), - datetime.datetime(2012, 4, 4, 15, 6, 0, 123456), - ]) + self.assertEqual(values, [ + datetime.datetime(2012, 4, 4, 15, 6, 0, 456000), + datetime.datetime(2012, 4, 4, 15, 6, 0, 123456), + ]) + @unittest.expectedFailure # TODO: RUSTPYTHON; error message mismatch def test_invalid_isolation_level_type(self): # isolation level is a string, not an integer - self.assertRaises(TypeError, - sqlite.connect, ":memory:", isolation_level=123) + regex = "isolation_level must be str or None" + with self.assertRaisesRegex(TypeError, regex): + memory_database(isolation_level=123).__enter__() def test_null_character(self): @@ -334,7 +326,7 @@ def test_null_character(self): cur.execute, query) def test_surrogates(self): - con = sqlite.connect(":memory:") + con = self.con self.assertRaises(UnicodeEncodeError, con, "select '\ud8ff'") self.assertRaises(UnicodeEncodeError, con, "select '\udcff'") cur = con.cursor() @@ -360,7 +352,7 @@ def test_commit_cursor_reset(self): to return rows multiple times when fetched from cursors after commit. See issues 10513 and 23129 for details. """ - con = sqlite.connect(":memory:") + con = self.con con.executescript(""" create table t(c); create table t2(c); @@ -392,10 +384,9 @@ def test_bpo31770(self): """ def callback(*args): pass - con = sqlite.connect(":memory:") - cur = sqlite.Cursor(con) + cur = sqlite.Cursor(self.con) ref = weakref.ref(cur, callback) - cur.__init__(con) + cur.__init__(self.con) del cur # The interpreter shouldn't crash when ref is collected. del ref @@ -405,8 +396,7 @@ def test_del_isolation_level_segfault(self): with self.assertRaises(AttributeError): del self.con.isolation_level - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_bpo37347(self): class Printer: def log(self, *args): @@ -428,6 +418,7 @@ def test_return_empty_bytestring(self): def test_table_lock_cursor_replace_stmt(self): with memory_database() as con: + con = self.con cur = con.cursor() cur.execute("create table t(t)") cur.executemany("insert into t values(?)", @@ -445,10 +436,11 @@ def test_table_lock_cursor_dealloc(self): con.commit() cur = con.execute("select t from t") del cur + support.gc_collect() con.execute("drop table t") con.commit() - @unittest.skip("TODO: RUSTPYTHON deadlock") + @unittest.skip('TODO: RUSTPYTHON; recursive cursor use causes lock contention') def test_table_lock_cursor_non_readonly_select(self): with memory_database() as con: con.execute("create table t(t)") @@ -461,6 +453,7 @@ def dup(v): con.create_function("dup", 1, dup) cur = con.execute("select dup(t) from t") del cur + support.gc_collect() con.execute("drop table t") con.commit() @@ -476,7 +469,7 @@ def test_executescript_step_through_select(self): self.assertEqual(steps, values) -@unittest.skip("TODO: RUSTPYTHON deadlock") +@unittest.skip('TODO: RUSTPYTHON; recursive cursor use causes lock contention') class RecursiveUseOfCursors(unittest.TestCase): # GH-80254: sqlite3 should not segfault for recursive use of cursors. msg = "Recursive use of cursors not allowed" @@ -496,21 +489,21 @@ def tearDown(self): def test_recursive_cursor_init(self): conv = lambda x: self.cur.__init__(self.con) with patch.dict(sqlite.converters, {"INIT": conv}): - self.cur.execute(f'select x as "x [INIT]", x from test') + self.cur.execute('select x as "x [INIT]", x from test') self.assertRaisesRegex(sqlite.ProgrammingError, self.msg, self.cur.fetchall) def test_recursive_cursor_close(self): conv = lambda x: self.cur.close() with patch.dict(sqlite.converters, {"CLOSE": conv}): - self.cur.execute(f'select x as "x [CLOSE]", x from test') + self.cur.execute('select x as "x [CLOSE]", x from test') self.assertRaisesRegex(sqlite.ProgrammingError, self.msg, self.cur.fetchall) def test_recursive_cursor_iter(self): conv = lambda x, l=[]: self.cur.fetchone() if l else l.append(None) with patch.dict(sqlite.converters, {"ITER": conv}): - self.cur.execute(f'select x as "x [ITER]", x from test') + self.cur.execute('select x as "x [ITER]", x from test') self.assertRaisesRegex(sqlite.ProgrammingError, self.msg, self.cur.fetchall) diff --git a/Lib/test/test_sqlite3/test_transactions.py b/Lib/test/test_sqlite3/test_transactions.py index 9c3d19e79bd..f38d042e598 100644 --- a/Lib/test/test_sqlite3/test_transactions.py +++ b/Lib/test/test_sqlite3/test_transactions.py @@ -22,22 +22,24 @@ import unittest import sqlite3 as sqlite +from contextlib import contextmanager -from test.support import LOOPBACK_TIMEOUT from test.support.os_helper import TESTFN, unlink +from test.support.script_helper import assert_python_ok -from test.test_sqlite3.test_dbapi import memory_database - - -TIMEOUT = LOOPBACK_TIMEOUT / 10 +from .util import memory_database +from .util import MemoryDatabaseMixin +@unittest.skip("TODO: RUSTPYTHON timeout parameter does not accept int type") class TransactionTests(unittest.TestCase): def setUp(self): - self.con1 = sqlite.connect(TESTFN, timeout=TIMEOUT) + # We can disable the busy handlers, since we control + # the order of SQLite C API operations. + self.con1 = sqlite.connect(TESTFN, timeout=0) self.cur1 = self.con1.cursor() - self.con2 = sqlite.connect(TESTFN, timeout=TIMEOUT) + self.con2 = sqlite.connect(TESTFN, timeout=0) self.cur2 = self.con2.cursor() def tearDown(self): @@ -117,10 +119,8 @@ def test_raise_timeout(self): self.cur2.execute("insert into test(i) values (5)") def test_locking(self): - """ - This tests the improved concurrency with pysqlite 2.3.4. You needed - to roll back con2 before you could commit con1. - """ + # This tests the improved concurrency with pysqlite 2.3.4. You needed + # to roll back con2 before you could commit con1. self.cur1.execute("create table test(i)") self.cur1.execute("insert into test(i) values (5)") with self.assertRaises(sqlite.OperationalError): @@ -130,14 +130,14 @@ def test_locking(self): def test_rollback_cursor_consistency(self): """Check that cursors behave correctly after rollback.""" - con = sqlite.connect(":memory:") - cur = con.cursor() - cur.execute("create table test(x)") - cur.execute("insert into test(x) values (5)") - cur.execute("select 1 union select 2 union select 3") + with memory_database() as con: + cur = con.cursor() + cur.execute("create table test(x)") + cur.execute("insert into test(x) values (5)") + cur.execute("select 1 union select 2 union select 3") - con.rollback() - self.assertEqual(cur.fetchall(), [(1,), (2,), (3,)]) + con.rollback() + self.assertEqual(cur.fetchall(), [(1,), (2,), (3,)]) def test_multiple_cursors_and_iternext(self): # gh-94028: statements are cleared and reset in cursor iternext. @@ -216,10 +216,7 @@ def test_no_duplicate_rows_after_rollback_new_query(self): -class SpecialCommandTests(unittest.TestCase): - def setUp(self): - self.con = sqlite.connect(":memory:") - self.cur = self.con.cursor() +class SpecialCommandTests(MemoryDatabaseMixin, unittest.TestCase): def test_drop_table(self): self.cur.execute("create table test(i)") @@ -231,14 +228,8 @@ def test_pragma(self): self.cur.execute("insert into test(i) values (5)") self.cur.execute("pragma count_changes=1") - def tearDown(self): - self.cur.close() - self.con.close() - -class TransactionalDDL(unittest.TestCase): - def setUp(self): - self.con = sqlite.connect(":memory:") +class TransactionalDDL(MemoryDatabaseMixin, unittest.TestCase): def test_ddl_does_not_autostart_transaction(self): # For backwards compatibility reasons, DDL statements should not @@ -266,9 +257,6 @@ def test_transactional_ddl(self): with self.assertRaises(sqlite.OperationalError): self.con.execute("select * from test") - def tearDown(self): - self.con.close() - class IsolationLevelFromInit(unittest.TestCase): CREATE = "create table t(t)" @@ -366,5 +354,183 @@ def test_isolation_level_none(self): self.assertEqual(self.traced, [self.QUERY]) +class AutocommitAttribute(unittest.TestCase): + """Test PEP 249-compliant autocommit behaviour.""" + legacy = sqlite.LEGACY_TRANSACTION_CONTROL + + @contextmanager + def check_stmt_trace(self, cx, expected, reset=True): + try: + traced = [] + cx.set_trace_callback(lambda stmt: traced.append(stmt)) + yield + finally: + self.assertEqual(traced, expected) + if reset: + cx.set_trace_callback(None) + + def test_autocommit_default(self): + with memory_database() as cx: + self.assertEqual(cx.autocommit, + sqlite.LEGACY_TRANSACTION_CONTROL) + + def test_autocommit_setget(self): + dataset = ( + True, + False, + sqlite.LEGACY_TRANSACTION_CONTROL, + ) + for mode in dataset: + with self.subTest(mode=mode): + with memory_database(autocommit=mode) as cx: + self.assertEqual(cx.autocommit, mode) + with memory_database() as cx: + cx.autocommit = mode + self.assertEqual(cx.autocommit, mode) + + @unittest.expectedFailure # TODO: RUSTPYTHON autocommit validation error messages differ + def test_autocommit_setget_invalid(self): + msg = "autocommit must be True, False, or.*LEGACY" + for mode in "a", 12, (), None: + with self.subTest(mode=mode): + with self.assertRaisesRegex(ValueError, msg): + sqlite.connect(":memory:", autocommit=mode) + + @unittest.expectedFailure # TODO: RUSTPYTHON autocommit behavior differs + def test_autocommit_disabled(self): + expected = [ + "SELECT 1", + "COMMIT", + "BEGIN", + "ROLLBACK", + "BEGIN", + ] + with memory_database(autocommit=False) as cx: + self.assertTrue(cx.in_transaction) + with self.check_stmt_trace(cx, expected): + cx.execute("SELECT 1") + cx.commit() + cx.rollback() + + @unittest.expectedFailure # TODO: RUSTPYTHON autocommit behavior differs + def test_autocommit_disabled_implicit_rollback(self): + expected = ["ROLLBACK"] + with memory_database(autocommit=False) as cx: + self.assertTrue(cx.in_transaction) + with self.check_stmt_trace(cx, expected, reset=False): + cx.close() + + def test_autocommit_enabled(self): + expected = ["CREATE TABLE t(t)", "INSERT INTO t VALUES(1)"] + with memory_database(autocommit=True) as cx: + self.assertFalse(cx.in_transaction) + with self.check_stmt_trace(cx, expected): + cx.execute("CREATE TABLE t(t)") + cx.execute("INSERT INTO t VALUES(1)") + self.assertFalse(cx.in_transaction) + + def test_autocommit_enabled_txn_ctl(self): + for op in "commit", "rollback": + with self.subTest(op=op): + with memory_database(autocommit=True) as cx: + meth = getattr(cx, op) + self.assertFalse(cx.in_transaction) + with self.check_stmt_trace(cx, []): + meth() # expect this to pass silently + self.assertFalse(cx.in_transaction) + + @unittest.expectedFailure # TODO: RUSTPYTHON autocommit behavior differs + def test_autocommit_disabled_then_enabled(self): + expected = ["COMMIT"] + with memory_database(autocommit=False) as cx: + self.assertTrue(cx.in_transaction) + with self.check_stmt_trace(cx, expected): + cx.autocommit = True # should commit + self.assertFalse(cx.in_transaction) + + def test_autocommit_enabled_then_disabled(self): + expected = ["BEGIN"] + with memory_database(autocommit=True) as cx: + self.assertFalse(cx.in_transaction) + with self.check_stmt_trace(cx, expected): + cx.autocommit = False # should begin + self.assertTrue(cx.in_transaction) + + def test_autocommit_explicit_then_disabled(self): + expected = ["BEGIN DEFERRED"] + with memory_database(autocommit=True) as cx: + self.assertFalse(cx.in_transaction) + with self.check_stmt_trace(cx, expected): + cx.execute("BEGIN DEFERRED") + cx.autocommit = False # should now be a no-op + self.assertTrue(cx.in_transaction) + + def test_autocommit_enabled_ctx_mgr(self): + with memory_database(autocommit=True) as cx: + # The context manager is a no-op if autocommit=True + with self.check_stmt_trace(cx, []): + with cx: + self.assertFalse(cx.in_transaction) + self.assertFalse(cx.in_transaction) + + @unittest.expectedFailure # TODO: RUSTPYTHON autocommit behavior differs + def test_autocommit_disabled_ctx_mgr(self): + expected = ["COMMIT", "BEGIN"] + with memory_database(autocommit=False) as cx: + with self.check_stmt_trace(cx, expected): + with cx: + self.assertTrue(cx.in_transaction) + self.assertTrue(cx.in_transaction) + + def test_autocommit_compat_ctx_mgr(self): + expected = ["BEGIN ", "INSERT INTO T VALUES(1)", "COMMIT"] + with memory_database(autocommit=self.legacy) as cx: + cx.execute("create table t(t)") + with self.check_stmt_trace(cx, expected): + with cx: + self.assertFalse(cx.in_transaction) + cx.execute("INSERT INTO T VALUES(1)") + self.assertTrue(cx.in_transaction) + self.assertFalse(cx.in_transaction) + + @unittest.expectedFailure # TODO: RUSTPYTHON autocommit behavior differs + def test_autocommit_enabled_executescript(self): + expected = ["BEGIN", "SELECT 1"] + with memory_database(autocommit=True) as cx: + with self.check_stmt_trace(cx, expected): + self.assertFalse(cx.in_transaction) + cx.execute("BEGIN") + cx.executescript("SELECT 1") + self.assertTrue(cx.in_transaction) + + @unittest.expectedFailure # TODO: RUSTPYTHON autocommit behavior differs + def test_autocommit_disabled_executescript(self): + expected = ["SELECT 1"] + with memory_database(autocommit=False) as cx: + with self.check_stmt_trace(cx, expected): + self.assertTrue(cx.in_transaction) + cx.executescript("SELECT 1") + self.assertTrue(cx.in_transaction) + + def test_autocommit_compat_executescript(self): + expected = ["BEGIN", "COMMIT", "SELECT 1"] + with memory_database(autocommit=self.legacy) as cx: + with self.check_stmt_trace(cx, expected): + self.assertFalse(cx.in_transaction) + cx.execute("BEGIN") + cx.executescript("SELECT 1") + self.assertFalse(cx.in_transaction) + + def test_autocommit_disabled_implicit_shutdown(self): + # The implicit ROLLBACK should not call back into Python during + # interpreter tear-down. + code = """if 1: + import sqlite3 + cx = sqlite3.connect(":memory:", autocommit=False) + cx.set_trace_callback(print) + """ + assert_python_ok("-c", code, PYTHONIOENCODING="utf-8") + + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_sqlite3/test_types.py b/Lib/test/test_sqlite3/test_types.py index 62318823510..66d27d21b8d 100644 --- a/Lib/test/test_sqlite3/test_types.py +++ b/Lib/test/test_sqlite3/test_types.py @@ -106,9 +106,9 @@ def test_string_with_surrogates(self): @unittest.skipUnless(sys.maxsize > 2**32, 'requires 64bit platform') @support.bigmemtest(size=2**31, memuse=4, dry_run=False) def test_too_large_string(self, maxsize): - with self.assertRaises(sqlite.InterfaceError): + with self.assertRaises(sqlite.DataError): self.cur.execute("insert into test(s) values (?)", ('x'*(2**31-1),)) - with self.assertRaises(OverflowError): + with self.assertRaises(sqlite.DataError): self.cur.execute("insert into test(s) values (?)", ('x'*(2**31),)) self.cur.execute("select 1 from test") row = self.cur.fetchone() @@ -117,9 +117,9 @@ def test_too_large_string(self, maxsize): @unittest.skipUnless(sys.maxsize > 2**32, 'requires 64bit platform') @support.bigmemtest(size=2**31, memuse=3, dry_run=False) def test_too_large_blob(self, maxsize): - with self.assertRaises(sqlite.InterfaceError): + with self.assertRaises(sqlite.DataError): self.cur.execute("insert into test(s) values (?)", (b'x'*(2**31-1),)) - with self.assertRaises(OverflowError): + with self.assertRaises(sqlite.DataError): self.cur.execute("insert into test(s) values (?)", (b'x'*(2**31),)) self.cur.execute("select 1 from test") row = self.cur.fetchone() @@ -371,7 +371,6 @@ def test_cursor_description_insert(self): self.assertIsNone(self.cur.description) -@unittest.skipIf(sqlite.sqlite_version_info < (3, 8, 3), "CTEs not supported") class CommonTableExpressionTests(unittest.TestCase): def setUp(self): @@ -517,7 +516,7 @@ def test_sqlite_timestamp(self): self.assertEqual(ts, ts2) def test_sql_timestamp(self): - now = datetime.datetime.utcnow() + now = datetime.datetime.now(tz=datetime.UTC) self.cur.execute("insert into test(ts) values (current_timestamp)") self.cur.execute("select ts from test") with self.assertWarnsRegex(DeprecationWarning, "converter"): diff --git a/Lib/test/test_sqlite3/test_userfunctions.py b/Lib/test/test_sqlite3/test_userfunctions.py index e8b98a66a57..3fdde4a26cd 100644 --- a/Lib/test/test_sqlite3/test_userfunctions.py +++ b/Lib/test/test_sqlite3/test_userfunctions.py @@ -21,55 +21,15 @@ # misrepresented as being the original software. # 3. This notice may not be removed or altered from any source distribution. -import contextlib -import functools -import io -import re import sys import unittest import sqlite3 as sqlite from unittest.mock import Mock, patch -from test.support import bigmemtest, catch_unraisable_exception, gc_collect - -from test.test_sqlite3.test_dbapi import cx_limit - - -def with_tracebacks(exc, regex="", name=""): - """Convenience decorator for testing callback tracebacks.""" - def decorator(func): - _regex = re.compile(regex) if regex else None - @functools.wraps(func) - def wrapper(self, *args, **kwargs): - with catch_unraisable_exception() as cm: - # First, run the test with traceback enabled. - with check_tracebacks(self, cm, exc, _regex, name): - func(self, *args, **kwargs) - - # Then run the test with traceback disabled. - func(self, *args, **kwargs) - return wrapper - return decorator - - -@contextlib.contextmanager -def check_tracebacks(self, cm, exc, regex, obj_name): - """Convenience context manager for testing callback tracebacks.""" - sqlite.enable_callback_tracebacks(True) - try: - buf = io.StringIO() - with contextlib.redirect_stderr(buf): - yield - - # TODO: RUSTPYTHON need unraisable exception - # self.assertEqual(cm.unraisable.exc_type, exc) - # if regex: - # msg = str(cm.unraisable.exc_value) - # self.assertIsNotNone(regex.search(msg)) - # if obj_name: - # self.assertEqual(cm.unraisable.object.__name__, obj_name) - finally: - sqlite.enable_callback_tracebacks(False) +from test.support import bigmemtest, gc_collect + +from .util import cx_limit, memory_database +from .util import with_tracebacks def func_returntext(): @@ -196,7 +156,6 @@ def setUp(self): self.con.create_function("returnblob", 0, func_returnblob) self.con.create_function("returnlonglong", 0, func_returnlonglong) self.con.create_function("returnnan", 0, lambda: float("nan")) - self.con.create_function("returntoolargeint", 0, lambda: 1 << 65) self.con.create_function("return_noncont_blob", 0, lambda: memoryview(b"blob")[::2]) self.con.create_function("raiseexception", 0, func_raiseexception) @@ -211,8 +170,9 @@ def setUp(self): def tearDown(self): self.con.close() + @unittest.expectedFailure # TODO: RUSTPYTHON error message differs for invalid num args def test_func_error_on_create(self): - with self.assertRaises(sqlite.OperationalError): + with self.assertRaisesRegex(sqlite.ProgrammingError, "not -100"): self.con.create_function("bla", -100, lambda x: 2*x) def test_func_too_many_args(self): @@ -295,12 +255,8 @@ def test_func_return_nan(self): cur.execute("select returnnan()") self.assertIsNone(cur.fetchone()[0]) - def test_func_return_too_large_int(self): - cur = self.con.cursor() - self.assertRaisesRegex(sqlite.DataError, "string or blob too big", - self.con.execute, "select returntoolargeint()") - - @with_tracebacks(ZeroDivisionError, name="func_raiseexception") + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented + @with_tracebacks(ZeroDivisionError, msg_regex="func_raiseexception") def test_func_exception(self): cur = self.con.cursor() with self.assertRaises(sqlite.OperationalError) as cm: @@ -308,14 +264,16 @@ def test_func_exception(self): cur.fetchone() self.assertEqual(str(cm.exception), 'user-defined function raised exception') - @with_tracebacks(MemoryError, name="func_memoryerror") + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented + @with_tracebacks(MemoryError, msg_regex="func_memoryerror") def test_func_memory_error(self): cur = self.con.cursor() with self.assertRaises(MemoryError): cur.execute("select memoryerror()") cur.fetchone() - @with_tracebacks(OverflowError, name="func_overflowerror") + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented + @with_tracebacks(OverflowError, msg_regex="func_overflowerror") def test_func_overflow_error(self): cur = self.con.cursor() with self.assertRaises(sqlite.DataError): @@ -348,6 +306,7 @@ def test_non_contiguous_blob(self): self.con.execute, "select spam(?)", (memoryview(b"blob")[::2],)) + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented @with_tracebacks(BufferError, regex="buffer.*contiguous") def test_return_non_contiguous_blob(self): with self.assertRaises(sqlite.OperationalError): @@ -388,38 +347,22 @@ def append_result(arg): # Regarding deterministic functions: # # Between 3.8.3 and 3.15.0, deterministic functions were only used to - # optimize inner loops, so for those versions we can only test if the - # sqlite machinery has factored out a call or not. From 3.15.0 and onward, - # deterministic functions were permitted in WHERE clauses of partial - # indices, which allows testing based on syntax, iso. the query optimizer. - @unittest.skipIf(sqlite.sqlite_version_info < (3, 8, 3), "Requires SQLite 3.8.3 or higher") + # optimize inner loops. From 3.15.0 and onward, deterministic functions + # were permitted in WHERE clauses of partial indices, which allows testing + # based on syntax, iso. the query optimizer. def test_func_non_deterministic(self): mock = Mock(return_value=None) self.con.create_function("nondeterministic", 0, mock, deterministic=False) - if sqlite.sqlite_version_info < (3, 15, 0): - self.con.execute("select nondeterministic() = nondeterministic()") - self.assertEqual(mock.call_count, 2) - else: - with self.assertRaises(sqlite.OperationalError): - self.con.execute("create index t on test(t) where nondeterministic() is not null") + with self.assertRaises(sqlite.OperationalError): + self.con.execute("create index t on test(t) where nondeterministic() is not null") - @unittest.skipIf(sqlite.sqlite_version_info < (3, 8, 3), "Requires SQLite 3.8.3 or higher") def test_func_deterministic(self): mock = Mock(return_value=None) self.con.create_function("deterministic", 0, mock, deterministic=True) - if sqlite.sqlite_version_info < (3, 15, 0): - self.con.execute("select deterministic() = deterministic()") - self.assertEqual(mock.call_count, 1) - else: - try: - self.con.execute("create index t on test(t) where deterministic() is not null") - except sqlite.OperationalError: - self.fail("Unexpected failure while creating partial index") - - @unittest.skipIf(sqlite.sqlite_version_info >= (3, 8, 3), "SQLite < 3.8.3 needed") - def test_func_deterministic_not_supported(self): - with self.assertRaises(sqlite.NotSupportedError): - self.con.create_function("deterministic", 0, int, deterministic=True) + try: + self.con.execute("create index t on test(t) where deterministic() is not null") + except sqlite.OperationalError: + self.fail("Unexpected failure while creating partial index") def test_func_deterministic_keyword_only(self): with self.assertRaises(TypeError): @@ -428,29 +371,32 @@ def test_func_deterministic_keyword_only(self): def test_function_destructor_via_gc(self): # See bpo-44304: The destructor of the user function can # crash if is called without the GIL from the gc functions - dest = sqlite.connect(':memory:') def md5sum(t): return - dest.create_function("md5", 1, md5sum) - x = dest("create table lang (name, first_appeared)") - del md5sum, dest + with memory_database() as dest: + dest.create_function("md5", 1, md5sum) + x = dest("create table lang (name, first_appeared)") + del md5sum, dest - y = [x] - y.append(y) + y = [x] + y.append(y) - del x,y - gc_collect() + del x,y + gc_collect() + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented @with_tracebacks(OverflowError) def test_func_return_too_large_int(self): cur = self.con.cursor() + msg = "string or blob too big" for value in 2**63, -2**63-1, 2**64: self.con.create_function("largeint", 0, lambda value=value: value) - with self.assertRaises(sqlite.DataError): + with self.assertRaisesRegex(sqlite.DataError, msg): cur.execute("select largeint()") - @with_tracebacks(UnicodeEncodeError, "surrogates not allowed", "chr") + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented + @with_tracebacks(UnicodeEncodeError, "surrogates not allowed") def test_func_return_text_with_surrogates(self): cur = self.con.cursor() self.con.create_function("pychr", 1, chr) @@ -482,6 +428,30 @@ def test_func_return_illegal_value(self): self.assertRaisesRegex(sqlite.OperationalError, msg, self.con.execute, "select badreturn()") + @unittest.expectedFailure # TODO: RUSTPYTHON deprecation warning not emitted for keyword args + def test_func_keyword_args(self): + regex = ( + r"Passing keyword arguments 'name', 'narg' and 'func' to " + r"_sqlite3.Connection.create_function\(\) is deprecated. " + r"Parameters 'name', 'narg' and 'func' will become " + r"positional-only in Python 3.15." + ) + + def noop(): + return None + + with self.assertWarnsRegex(DeprecationWarning, regex) as cm: + self.con.create_function("noop", 0, func=noop) + self.assertEqual(cm.filename, __file__) + + with self.assertWarnsRegex(DeprecationWarning, regex) as cm: + self.con.create_function("noop", narg=0, func=noop) + self.assertEqual(cm.filename, __file__) + + with self.assertWarnsRegex(DeprecationWarning, regex) as cm: + self.con.create_function(name="noop", narg=0, func=noop) + self.assertEqual(cm.filename, __file__) + class WindowSumInt: def __init__(self): @@ -536,15 +506,20 @@ def setUp(self): """ self.con.create_window_function("sumint", 1, WindowSumInt) + def tearDown(self): + self.cur.close() + self.con.close() + def test_win_sum_int(self): self.cur.execute(self.query % "sumint") self.assertEqual(self.cur.fetchall(), self.expected) + @unittest.expectedFailure # TODO: RUSTPYTHON error message differs for invalid num args def test_win_error_on_create(self): - self.assertRaises(sqlite.ProgrammingError, - self.con.create_window_function, - "shouldfail", -100, WindowSumInt) + with self.assertRaisesRegex(sqlite.ProgrammingError, "not -100"): + self.con.create_window_function("shouldfail", -100, WindowSumInt) + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented @with_tracebacks(BadWindow) def test_win_exception_in_method(self): for meth in "__init__", "step", "value", "inverse": @@ -557,17 +532,19 @@ def test_win_exception_in_method(self): self.cur.execute(self.query % name) self.cur.fetchall() + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented @with_tracebacks(BadWindow) def test_win_exception_in_finalize(self): # Note: SQLite does not (as of version 3.38.0) propagate finalize # callback errors to sqlite3_step(); this implies that OperationalError # is _not_ raised. with patch.object(WindowSumInt, "finalize", side_effect=BadWindow): - name = f"exception_in_finalize" + name = "exception_in_finalize" self.con.create_window_function(name, 1, WindowSumInt) self.cur.execute(self.query % name) self.cur.fetchall() + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented @with_tracebacks(AttributeError) def test_win_missing_method(self): class MissingValue: @@ -599,6 +576,7 @@ def finalize(self): return 42 self.cur.execute(self.query % name) self.cur.fetchall() + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented @with_tracebacks(AttributeError) def test_win_missing_finalize(self): # Note: SQLite does not (as of version 3.38.0) propagate finalize @@ -656,6 +634,7 @@ def setUp(self): """) cur.execute("insert into test(t, i, f, n, b) values (?, ?, ?, ?, ?)", ("foo", 5, 3.14, None, memoryview(b"blob"),)) + cur.close() self.con.create_aggregate("nostep", 1, AggrNoStep) self.con.create_aggregate("nofinalize", 1, AggrNoFinalize) @@ -668,15 +647,15 @@ def setUp(self): self.con.create_aggregate("aggtxt", 1, AggrText) def tearDown(self): - #self.cur.close() - #self.con.close() - pass + self.con.close() + @unittest.expectedFailure # TODO: RUSTPYTHON error message differs for invalid num args def test_aggr_error_on_create(self): - with self.assertRaises(sqlite.OperationalError): + with self.assertRaisesRegex(sqlite.ProgrammingError, "not -100"): self.con.create_function("bla", -100, AggrSum) - @with_tracebacks(AttributeError, name="AggrNoStep") + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented + @with_tracebacks(AttributeError, msg_regex="AggrNoStep") def test_aggr_no_step(self): cur = self.con.cursor() with self.assertRaises(sqlite.OperationalError) as cm: @@ -691,7 +670,8 @@ def test_aggr_no_finalize(self): cur.execute("select nofinalize(t) from test") val = cur.fetchone()[0] - @with_tracebacks(ZeroDivisionError, name="AggrExceptionInInit") + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented + @with_tracebacks(ZeroDivisionError, msg_regex="AggrExceptionInInit") def test_aggr_exception_in_init(self): cur = self.con.cursor() with self.assertRaises(sqlite.OperationalError) as cm: @@ -699,7 +679,8 @@ def test_aggr_exception_in_init(self): val = cur.fetchone()[0] self.assertEqual(str(cm.exception), "user-defined aggregate's '__init__' method raised error") - @with_tracebacks(ZeroDivisionError, name="AggrExceptionInStep") + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented + @with_tracebacks(ZeroDivisionError, msg_regex="AggrExceptionInStep") def test_aggr_exception_in_step(self): cur = self.con.cursor() with self.assertRaises(sqlite.OperationalError) as cm: @@ -707,7 +688,8 @@ def test_aggr_exception_in_step(self): val = cur.fetchone()[0] self.assertEqual(str(cm.exception), "user-defined aggregate's 'step' method raised error") - @with_tracebacks(ZeroDivisionError, name="AggrExceptionInFinalize") + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented + @with_tracebacks(ZeroDivisionError, msg_regex="AggrExceptionInFinalize") def test_aggr_exception_in_finalize(self): cur = self.con.cursor() with self.assertRaises(sqlite.OperationalError) as cm: @@ -772,6 +754,28 @@ def test_aggr_text(self): val = cur.fetchone()[0] self.assertEqual(val, txt) + @unittest.expectedFailure # TODO: RUSTPYTHON keyword-only arguments not supported for create_aggregate + def test_agg_keyword_args(self): + regex = ( + r"Passing keyword arguments 'name', 'n_arg' and 'aggregate_class' to " + r"_sqlite3.Connection.create_aggregate\(\) is deprecated. " + r"Parameters 'name', 'n_arg' and 'aggregate_class' will become " + r"positional-only in Python 3.15." + ) + + with self.assertWarnsRegex(DeprecationWarning, regex) as cm: + self.con.create_aggregate("test", 1, aggregate_class=AggrText) + self.assertEqual(cm.filename, __file__) + + with self.assertWarnsRegex(DeprecationWarning, regex) as cm: + self.con.create_aggregate("test", n_arg=1, aggregate_class=AggrText) + self.assertEqual(cm.filename, __file__) + + with self.assertWarnsRegex(DeprecationWarning, regex) as cm: + self.con.create_aggregate(name="test", n_arg=0, + aggregate_class=AggrText) + self.assertEqual(cm.filename, __file__) + class AuthorizerTests(unittest.TestCase): @staticmethod @@ -783,8 +787,6 @@ def authorizer_cb(action, arg1, arg2, dbname, source): return sqlite.SQLITE_OK def setUp(self): - # TODO: RUSTPYTHON difference 'prohibited' - self.prohibited = 'not authorized' self.con = sqlite.connect(":memory:") self.con.executescript(""" create table t1 (c1, c2); @@ -799,23 +801,38 @@ def setUp(self): self.con.set_authorizer(self.authorizer_cb) def tearDown(self): - pass + self.con.close() + @unittest.expectedFailure # TODO: RUSTPYTHON error message differs def test_table_access(self): with self.assertRaises(sqlite.DatabaseError) as cm: self.con.execute("select * from t2") - self.assertIn(self.prohibited, str(cm.exception)) + self.assertIn('prohibited', str(cm.exception)) + @unittest.expectedFailure # TODO: RUSTPYTHON error message differs def test_column_access(self): with self.assertRaises(sqlite.DatabaseError) as cm: self.con.execute("select c2 from t1") - self.assertIn(self.prohibited, str(cm.exception)) + self.assertIn('prohibited', str(cm.exception)) def test_clear_authorizer(self): self.con.set_authorizer(None) self.con.execute("select * from t2") self.con.execute("select c2 from t1") + @unittest.expectedFailure # TODO: RUSTPYTHON keyword-only arguments not supported for set_authorizer + def test_authorizer_keyword_args(self): + regex = ( + r"Passing keyword argument 'authorizer_callback' to " + r"_sqlite3.Connection.set_authorizer\(\) is deprecated. " + r"Parameter 'authorizer_callback' will become positional-only in " + r"Python 3.15." + ) + + with self.assertWarnsRegex(DeprecationWarning, regex) as cm: + self.con.set_authorizer(authorizer_callback=lambda: None) + self.assertEqual(cm.filename, __file__) + class AuthorizerRaiseExceptionTests(AuthorizerTests): @staticmethod @@ -826,11 +843,13 @@ def authorizer_cb(action, arg1, arg2, dbname, source): raise ValueError return sqlite.SQLITE_OK - @with_tracebacks(ValueError, name="authorizer_cb") + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented + @with_tracebacks(ValueError, msg_regex="authorizer_cb") def test_table_access(self): super().test_table_access() - @with_tracebacks(ValueError, name="authorizer_cb") + @unittest.expectedFailure # TODO: RUSTPYTHON unraisable exception handling not implemented + @with_tracebacks(ValueError, msg_regex="authorizer_cb") def test_column_access(self): super().test_table_access() diff --git a/Lib/test/test_sqlite3/util.py b/Lib/test/test_sqlite3/util.py new file mode 100644 index 00000000000..cccd062160f --- /dev/null +++ b/Lib/test/test_sqlite3/util.py @@ -0,0 +1,89 @@ +import contextlib +import functools +import io +import re +import sqlite3 +import test.support +import unittest + + +# Helper for temporary memory databases +def memory_database(*args, **kwargs): + cx = sqlite3.connect(":memory:", *args, **kwargs) + return contextlib.closing(cx) + + +# Temporarily limit a database connection parameter +@contextlib.contextmanager +def cx_limit(cx, category=sqlite3.SQLITE_LIMIT_SQL_LENGTH, limit=128): + try: + _prev = cx.setlimit(category, limit) + yield limit + finally: + cx.setlimit(category, _prev) + + +def with_tracebacks(exc, regex="", name="", msg_regex=""): + """Convenience decorator for testing callback tracebacks.""" + def decorator(func): + exc_regex = re.compile(regex) if regex else None + _msg_regex = re.compile(msg_regex) if msg_regex else None + @functools.wraps(func) + def wrapper(self, *args, **kwargs): + with test.support.catch_unraisable_exception() as cm: + # First, run the test with traceback enabled. + with check_tracebacks(self, cm, exc, exc_regex, _msg_regex, name): + func(self, *args, **kwargs) + + # Then run the test with traceback disabled. + func(self, *args, **kwargs) + return wrapper + return decorator + + +@contextlib.contextmanager +def check_tracebacks(self, cm, exc, exc_regex, msg_regex, obj_name): + """Convenience context manager for testing callback tracebacks.""" + sqlite3.enable_callback_tracebacks(True) + try: + buf = io.StringIO() + with contextlib.redirect_stderr(buf): + yield + + self.assertEqual(cm.unraisable.exc_type, exc) + if exc_regex: + msg = str(cm.unraisable.exc_value) + self.assertIsNotNone(exc_regex.search(msg), (exc_regex, msg)) + if msg_regex: + msg = cm.unraisable.err_msg + self.assertIsNotNone(msg_regex.search(msg), (msg_regex, msg)) + if obj_name: + self.assertEqual(cm.unraisable.object.__name__, obj_name) + finally: + sqlite3.enable_callback_tracebacks(False) + + +class MemoryDatabaseMixin: + + def setUp(self): + self.con = sqlite3.connect(":memory:") + self.cur = self.con.cursor() + + def tearDown(self): + self.cur.close() + self.con.close() + + @property + def cx(self): + return self.con + + @property + def cu(self): + return self.cur + + +def requires_virtual_table(module): + with memory_database() as cx: + supported = (module,) in list(cx.execute("PRAGMA module_list")) + reason = f"Requires {module!r} virtual table support" + return unittest.skipUnless(supported, reason) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py index 7450cd34143..71b54e286a3 100644 --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -2891,6 +2891,7 @@ def test_echo(self): 'Cannot create a client socket with a PROTOCOL_TLS_SERVER context', str(e.exception)) + @unittest.skip('TODO: RUSTPYTHON flaky') @unittest.skipUnless(support.Py_GIL_DISABLED, "test is only useful if the GIL is disabled") def test_ssl_in_multiple_threads(self): # See GH-124984: OpenSSL is not thread safe. @@ -4595,7 +4596,7 @@ def server_callback(identity): with client_context.wrap_socket(socket.socket()) as s: s.connect((HOST, server.port)) - @unittest.skip("TODO: rustpython") + @unittest.skip("TODO: RUSTPYTHON; Hangs") def test_thread_recv_while_main_thread_sends(self): # GH-137583: Locking was added to calls to send() and recv() on SSL # socket objects. This seemed fine at the surface level because those diff --git a/Lib/test/test_str.py b/Lib/test/test_str.py index 9d43a33cd9e..6b766272a3f 100644 --- a/Lib/test/test_str.py +++ b/Lib/test/test_str.py @@ -565,7 +565,6 @@ def __str__(self): return self.sval self.checkraises(TypeError, ' ', 'join', [1, 2, 3]) self.checkraises(TypeError, ' ', 'join', ['1', '2', 3]) - @unittest.skip('TODO: RUSTPYTHON; oom handling') @unittest.skipIf(sys.maxsize > 2**32, 'needs too much memory on a 64-bit platform') def test_join_overflow(self): @@ -1460,7 +1459,6 @@ def __getitem__(self, key): self.assertRaises(TypeError, '{a}'.format_map, []) self.assertRaises(ZeroDivisionError, '{a}'.format_map, BadMapping()) - @unittest.skip('TODO: RUSTPYTHON; killed for chewing up RAM') def test_format_huge_precision(self): format_string = ".{}f".format(sys.maxsize + 1) with self.assertRaises(ValueError): @@ -2468,7 +2466,6 @@ def test_printable_repr(self): # This test only affects 32-bit platforms because expandtabs can only take # an int as the max value, not a 64-bit C long. If expandtabs is changed # to take a 64-bit long, this test should apply to all platforms. - @unittest.skip('TODO: RUSTPYTHON; oom handling') @unittest.skipIf(sys.maxsize > (1 << 32) or struct.calcsize('P') != 4, 'only applies to 32-bit platforms') def test_expandtabs_overflows_gracefully(self): @@ -2479,7 +2476,7 @@ def test_expandtabs_optimization(self): s = 'abc' self.assertIs(s.expandtabs(), s) - @unittest.skip('TODO: RUSTPYTHON; aborted: memory allocation of 9223372036854775759 bytes failed') + @unittest.expectedFailure # TODO: RUSTPYTHON def test_raiseMemError(self): asciifields = "nnb" compactfields = asciifields + "nP" diff --git a/Lib/test/test_string/__init__.py b/Lib/test/test_string/__init__.py new file mode 100644 index 00000000000..4b16ecc3115 --- /dev/null +++ b/Lib/test/test_string/__init__.py @@ -0,0 +1,5 @@ +import os +from test.support import load_package_tests + +def load_tests(*args): + return load_package_tests(os.path.dirname(__file__), *args) diff --git a/Lib/test/test_string/_support.py b/Lib/test/test_string/_support.py new file mode 100644 index 00000000000..cfead782b7d --- /dev/null +++ b/Lib/test/test_string/_support.py @@ -0,0 +1,67 @@ +import unittest +from string.templatelib import Interpolation + + +class TStringBaseCase: + def assertInterpolationEqual(self, i, exp): + """Test Interpolation equality. + + The *i* argument must be an Interpolation instance. + + The *exp* argument must be a tuple of the form + (value, expression, conversion, format_spec) where the final three + items may be omitted and are assumed to be '', None and '' respectively. + """ + if len(exp) == 4: + actual = (i.value, i.expression, i.conversion, i.format_spec) + self.assertEqual(actual, exp) + elif len(exp) == 3: + self.assertEqual((i.value, i.expression, i.conversion), exp) + self.assertEqual(i.format_spec, "") + elif len(exp) == 2: + self.assertEqual((i.value, i.expression), exp) + self.assertEqual(i.conversion, None) + self.assertEqual(i.format_spec, "") + elif len(exp) == 1: + self.assertEqual((i.value,), exp) + self.assertEqual(i.expression, "") + self.assertEqual(i.conversion, None) + self.assertEqual(i.format_spec, "") + + def assertTStringEqual(self, t, strings, interpolations): + """Test template string literal equality. + + The *strings* argument must be a tuple of strings equal to *t.strings*. + + The *interpolations* argument must be a sequence of tuples which are + compared against *t.interpolations*. Each tuple must match the form + described in the `assertInterpolationEqual` method. + """ + self.assertEqual(t.strings, strings) + self.assertEqual(len(t.interpolations), len(interpolations)) + + for i, exp in zip(t.interpolations, interpolations, strict=True): + self.assertInterpolationEqual(i, exp) + + +def convert(value, conversion): + if conversion == "a": + return ascii(value) + elif conversion == "r": + return repr(value) + elif conversion == "s": + return str(value) + return value + + +def fstring(template): + parts = [] + for item in template: + match item: + case str() as s: + parts.append(s) + case Interpolation(value, _, conversion, format_spec): + value = convert(value, conversion) + value = format(value, format_spec) + parts.append(value) + return "".join(parts) diff --git a/Lib/test/test_string.py b/Lib/test/test_string/test_string.py similarity index 95% rename from Lib/test/test_string.py rename to Lib/test/test_string/test_string.py index 824b89ad517..5394fe4e12c 100644 --- a/Lib/test/test_string.py +++ b/Lib/test/test_string/test_string.py @@ -1,6 +1,15 @@ import unittest import string from string import Template +import types +from test.support import cpython_only +from test.support.import_helper import ensure_lazy_imports + + +class LazyImportTest(unittest.TestCase): + @cpython_only + def test_lazy_import(self): + ensure_lazy_imports("base64", {"re", "collections"}) class ModuleTest(unittest.TestCase): @@ -101,6 +110,24 @@ def test_index_lookup(self): with self.assertRaises(KeyError): fmt.format("{0[2]}{0[0]}", {}) + def test_auto_numbering_lookup(self): + fmt = string.Formatter() + namespace = types.SimpleNamespace(foo=types.SimpleNamespace(bar='baz')) + widths = [None, types.SimpleNamespace(qux=4)] + self.assertEqual( + fmt.format("{.foo.bar:{[1].qux}}", namespace, widths), 'baz ') + + def test_auto_numbering_reenterability(self): + class ReenteringFormatter(string.Formatter): + def format_field(self, value, format_spec): + if format_spec.isdigit() and int(format_spec) > 0: + return self.format('{:{}}!', value, int(format_spec) - 1) + else: + return super().format_field(value, format_spec) + fmt = ReenteringFormatter() + x = types.SimpleNamespace(a='X') + self.assertEqual(fmt.format('{.a:{}}', x, 3), 'X!!!') + def test_override_get_value(self): class NamespaceFormatter(string.Formatter): def __init__(self, namespace={}): diff --git a/Lib/test/test_string/test_templatelib.py b/Lib/test/test_string/test_templatelib.py new file mode 100644 index 00000000000..1c86717155f --- /dev/null +++ b/Lib/test/test_string/test_templatelib.py @@ -0,0 +1,193 @@ +import pickle +import unittest +from collections.abc import Iterator, Iterable +from string.templatelib import Template, Interpolation, convert + +from test.test_string._support import TStringBaseCase, fstring + + +class TestTemplate(unittest.TestCase, TStringBaseCase): + + def test_common(self): + self.assertEqual(type(t'').__name__, 'Template') + self.assertEqual(type(t'').__qualname__, 'Template') + self.assertEqual(type(t'').__module__, 'string.templatelib') + + a = 'a' + i = t'{a}'.interpolations[0] + self.assertEqual(type(i).__name__, 'Interpolation') + self.assertEqual(type(i).__qualname__, 'Interpolation') + self.assertEqual(type(i).__module__, 'string.templatelib') + + def test_final_types(self): + with self.assertRaisesRegex(TypeError, 'is not an acceptable base type'): + class Sub(Template): ... + + with self.assertRaisesRegex(TypeError, 'is not an acceptable base type'): + class Sub(Interpolation): ... + + def test_basic_creation(self): + # Simple t-string creation + t = t'Hello, world' + self.assertIsInstance(t, Template) + self.assertTStringEqual(t, ('Hello, world',), ()) + self.assertEqual(fstring(t), 'Hello, world') + + # Empty t-string + t = t'' + self.assertTStringEqual(t, ('',), ()) + self.assertEqual(fstring(t), '') + + # Multi-line t-string + t = t"""Hello, +world""" + self.assertEqual(t.strings, ('Hello,\nworld',)) + self.assertEqual(len(t.interpolations), 0) + self.assertEqual(fstring(t), 'Hello,\nworld') + + def test_interpolation_creation(self): + i = Interpolation('Maria', 'name', 'a', 'fmt') + self.assertInterpolationEqual(i, ('Maria', 'name', 'a', 'fmt')) + + i = Interpolation('Maria', 'name', 'a') + self.assertInterpolationEqual(i, ('Maria', 'name', 'a')) + + i = Interpolation('Maria', 'name') + self.assertInterpolationEqual(i, ('Maria', 'name')) + + i = Interpolation('Maria') + self.assertInterpolationEqual(i, ('Maria',)) + + def test_creation_interleaving(self): + # Should add strings on either side + t = Template(Interpolation('Maria', 'name', None, '')) + self.assertTStringEqual(t, ('', ''), [('Maria', 'name')]) + self.assertEqual(fstring(t), 'Maria') + + # Should prepend empty string + t = Template(Interpolation('Maria', 'name', None, ''), ' is my name') + self.assertTStringEqual(t, ('', ' is my name'), [('Maria', 'name')]) + self.assertEqual(fstring(t), 'Maria is my name') + + # Should append empty string + t = Template('Hello, ', Interpolation('Maria', 'name', None, '')) + self.assertTStringEqual(t, ('Hello, ', ''), [('Maria', 'name')]) + self.assertEqual(fstring(t), 'Hello, Maria') + + # Should concatenate strings + t = Template('Hello', ', ', Interpolation('Maria', 'name', None, ''), + '!') + self.assertTStringEqual(t, ('Hello, ', '!'), [('Maria', 'name')]) + self.assertEqual(fstring(t), 'Hello, Maria!') + + # Should add strings on either side and in between + t = Template(Interpolation('Maria', 'name', None, ''), + Interpolation('Python', 'language', None, '')) + self.assertTStringEqual( + t, ('', '', ''), [('Maria', 'name'), ('Python', 'language')] + ) + self.assertEqual(fstring(t), 'MariaPython') + + def test_template_values(self): + t = t'Hello, world' + self.assertEqual(t.values, ()) + + name = "Lys" + t = t'Hello, {name}' + self.assertEqual(t.values, ("Lys",)) + + country = "GR" + age = 0 + t = t'Hello, {name}, {age} from {country}' + self.assertEqual(t.values, ("Lys", 0, "GR")) + + def test_pickle_template(self): + user = 'test' + for template in ( + t'', + t"No values", + t'With inter {user}', + t'With ! {user!r}', + t'With format {1 / 0.3:.2f}', + Template(), + Template('a'), + Template(Interpolation('Nikita', 'name', None, '')), + Template('a', Interpolation('Nikita', 'name', 'r', '')), + ): + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto, template=template): + pickled = pickle.dumps(template, protocol=proto) + unpickled = pickle.loads(pickled) + + self.assertEqual(unpickled.values, template.values) + self.assertEqual(fstring(unpickled), fstring(template)) + + def test_pickle_interpolation(self): + for interpolation in ( + Interpolation('Nikita', 'name', None, ''), + Interpolation('Nikita', 'name', 'r', ''), + Interpolation(1/3, 'x', None, '.2f'), + ): + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto, interpolation=interpolation): + pickled = pickle.dumps(interpolation, protocol=proto) + unpickled = pickle.loads(pickled) + + self.assertEqual(unpickled.value, interpolation.value) + self.assertEqual(unpickled.expression, interpolation.expression) + self.assertEqual(unpickled.conversion, interpolation.conversion) + self.assertEqual(unpickled.format_spec, interpolation.format_spec) + + +class TemplateIterTests(unittest.TestCase): + def test_abc(self): + self.assertIsInstance(iter(t''), Iterable) + self.assertIsInstance(iter(t''), Iterator) + + def test_final(self): + TemplateIter = type(iter(t'')) + with self.assertRaisesRegex(TypeError, 'is not an acceptable base type'): + class Sub(TemplateIter): ... + + def test_iter(self): + x = 1 + res = list(iter(t'abc {x} yz')) + + self.assertEqual(res[0], 'abc ') + self.assertIsInstance(res[1], Interpolation) + self.assertEqual(res[1].value, 1) + self.assertEqual(res[1].expression, 'x') + self.assertEqual(res[1].conversion, None) + self.assertEqual(res[1].format_spec, '') + self.assertEqual(res[2], ' yz') + + def test_exhausted(self): + # See https://github.com/python/cpython/issues/134119. + template_iter = iter(t"{1}") + self.assertIsInstance(next(template_iter), Interpolation) + self.assertRaises(StopIteration, next, template_iter) + self.assertRaises(StopIteration, next, template_iter) + + +class TestFunctions(unittest.TestCase): + def test_convert(self): + from fractions import Fraction + + for obj in ('Café', None, 3.14, Fraction(1, 2)): + with self.subTest(f'{obj=}'): + self.assertEqual(convert(obj, None), obj) + self.assertEqual(convert(obj, 's'), str(obj)) + self.assertEqual(convert(obj, 'r'), repr(obj)) + self.assertEqual(convert(obj, 'a'), ascii(obj)) + + # Invalid conversion specifier + with self.assertRaises(ValueError): + convert(obj, 'z') + with self.assertRaises(ValueError): + convert(obj, 1) + with self.assertRaises(ValueError): + convert(obj, object()) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_struct.py b/Lib/test/test_struct.py index ef5602d083b..a03438f1604 100644 --- a/Lib/test/test_struct.py +++ b/Lib/test/test_struct.py @@ -1,22 +1,28 @@ from collections import abc +from itertools import combinations import array import gc import math import operator import unittest +import platform import struct import sys import weakref from test import support -from test.support import import_helper, suppress_immortalization +from test.support import import_helper from test.support.script_helper import assert_python_ok +from test.support.testcase import ComplexesAreIdenticalMixin ISBIGENDIAN = sys.byteorder == "big" integer_codes = 'b', 'B', 'h', 'H', 'i', 'I', 'l', 'L', 'q', 'Q', 'n', 'N' byteorders = '', '@', '=', '<', '>', '!' +INF = float('inf') +NAN = float('nan') + def iter_integer_formats(byteorders=byteorders): for code in integer_codes: for byteorder in byteorders: @@ -33,7 +39,7 @@ def bigendian_to_native(value): else: return string_reverse(value) -class StructTest(unittest.TestCase): +class StructTest(ComplexesAreIdenticalMixin, unittest.TestCase): def test_isbigendian(self): self.assertEqual((struct.pack('=i', 1)[0] == 0), ISBIGENDIAN) @@ -360,8 +366,7 @@ def test_p_code(self): (got,) = struct.unpack(code, got) self.assertEqual(got, expectedback) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_705836(self): # SF bug 705836. "<f" and ">f" had a severe rounding bug, where a carry # from the low-order discarded bits could propagate into the exponent @@ -670,8 +675,7 @@ def test_format_attr(self): s2 = struct.Struct(s.format.encode()) self.assertEqual(s2.format, s.format) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_struct_cleans_up_at_runtime_shutdown(self): code = """if 1: import struct @@ -687,10 +691,9 @@ def __del__(self): rc, stdout, stderr = assert_python_ok("-c", code) self.assertEqual(rc, 0) self.assertEqual(stdout.rstrip(), b"") - self.assertIn(b"Exception ignored in:", stderr) + self.assertIn(b"Exception ignored while calling deallocator", stderr) self.assertIn(b"C.__del__", stderr) - @suppress_immortalization() def test__struct_reference_cycle_cleaned_up(self): # Regression test for python/cpython#94207. @@ -777,8 +780,7 @@ def test_error_propagation(fmt_str): test_error_propagation('N') test_error_propagation('n') - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_struct_subclass_instantiation(self): # Regression test for https://github.com/python/cpython/issues/112358 class MyStruct(struct.Struct): @@ -792,6 +794,34 @@ def test_repr(self): s = struct.Struct('=i2H') self.assertEqual(repr(s), f'Struct({s.format!r})') + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_c_complex_round_trip(self): + values = [complex(*_) for _ in combinations([1, -1, 0.0, -0.0, 2, + -3, INF, -INF, NAN], 2)] + for z in values: + for f in ['F', 'D', '>F', '>D', '<F', '<D']: + with self.subTest(z=z, format=f): + round_trip = struct.unpack(f, struct.pack(f, z))[0] + self.assertComplexesAreIdentical(z, round_trip) + + @unittest.skipIf( + support.is_android or support.is_apple_mobile, + "Subinterpreters are not supported on Android and iOS" + ) + def test_endian_table_init_subinterpreters(self): + # Verify that the _struct extension module can be initialized + # concurrently in subinterpreters (gh-140260). + try: + from concurrent.futures import InterpreterPoolExecutor + except ImportError: + raise unittest.SkipTest("InterpreterPoolExecutor not available") + + code = "import struct" + with InterpreterPoolExecutor(max_workers=5) as executor: + results = executor.map(exec, [code] * 5) + self.assertListEqual(list(results), [None] * 5) + + class UnpackIteratorTest(unittest.TestCase): """ Tests for iterative unpacking (struct.Struct.iter_unpack). @@ -909,10 +939,17 @@ def test_half_float(self): # Check that packing produces a bit pattern representing a quiet NaN: # all exponent bits and the msb of the fraction should all be 1. + if platform.machine().startswith('parisc'): + # HP PA RISC uses 0 for quiet, see: + # https://en.wikipedia.org/wiki/NaN#Encoding + expected = 0x7c + else: + expected = 0x7e + packed = struct.pack('<e', math.nan) - self.assertEqual(packed[1] & 0x7e, 0x7e) + self.assertEqual(packed[1] & 0x7e, expected) packed = struct.pack('<e', -math.nan) - self.assertEqual(packed[1] & 0x7e, 0x7e) + self.assertEqual(packed[1] & 0x7e, expected) # Checks for round-to-even behavior format_bits_float__rounding_list = [ @@ -970,4 +1007,4 @@ def test_half_float(self): if __name__ == '__main__': - unittest.main() \ No newline at end of file + unittest.main() diff --git a/Lib/test/test_subclassinit.py b/Lib/test/test_subclassinit.py index c007476e004..0d32aa509bd 100644 --- a/Lib/test/test_subclassinit.py +++ b/Lib/test/test_subclassinit.py @@ -134,30 +134,28 @@ class Descriptor: def __set_name__(self, owner, name): 1/0 - with self.assertRaises(RuntimeError) as cm: + with self.assertRaises(ZeroDivisionError) as cm: class NotGoingToWork: attr = Descriptor() - exc = cm.exception - self.assertRegex(str(exc), r'\bNotGoingToWork\b') - self.assertRegex(str(exc), r'\battr\b') - self.assertRegex(str(exc), r'\bDescriptor\b') - self.assertIsInstance(exc.__cause__, ZeroDivisionError) + notes = cm.exception.__notes__ + self.assertRegex(str(notes), r'\bNotGoingToWork\b') + self.assertRegex(str(notes), r'\battr\b') + self.assertRegex(str(notes), r'\bDescriptor\b') def test_set_name_wrong(self): class Descriptor: def __set_name__(self): pass - with self.assertRaises(RuntimeError) as cm: + with self.assertRaises(TypeError) as cm: class NotGoingToWork: attr = Descriptor() - exc = cm.exception - self.assertRegex(str(exc), r'\bNotGoingToWork\b') - self.assertRegex(str(exc), r'\battr\b') - self.assertRegex(str(exc), r'\bDescriptor\b') - self.assertIsInstance(exc.__cause__, TypeError) + notes = cm.exception.__notes__ + self.assertRegex(str(notes), r'\bNotGoingToWork\b') + self.assertRegex(str(notes), r'\battr\b') + self.assertRegex(str(notes), r'\bDescriptor\b') def test_set_name_lookup(self): resolved = [] @@ -232,7 +230,7 @@ def __init__(self, name, bases, namespace, otherarg): super().__init__(name, bases, namespace) with self.assertRaises(TypeError): - class MyClass(metaclass=MyMeta, otherarg=1): + class MyClass2(metaclass=MyMeta, otherarg=1): pass class MyMeta(type): @@ -243,10 +241,10 @@ def __init__(self, name, bases, namespace, otherarg): super().__init__(name, bases, namespace) self.otherarg = otherarg - class MyClass(metaclass=MyMeta, otherarg=1): + class MyClass3(metaclass=MyMeta, otherarg=1): pass - self.assertEqual(MyClass.otherarg, 1) + self.assertEqual(MyClass3.otherarg, 1) def test_errors_changed_pep487(self): # These tests failed before Python 3.6, PEP 487 @@ -265,10 +263,10 @@ def __new__(cls, name, bases, namespace, otherarg): self.otherarg = otherarg return self - class MyClass(metaclass=MyMeta, otherarg=1): + class MyClass2(metaclass=MyMeta, otherarg=1): pass - self.assertEqual(MyClass.otherarg, 1) + self.assertEqual(MyClass2.otherarg, 1) def test_type(self): t = type('NewClass', (object,), {}) @@ -281,4 +279,3 @@ def test_type(self): if __name__ == "__main__": unittest.main() - diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index 4d058652723..d95c7857d98 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1291,7 +1291,6 @@ def test_universal_newlines_communicate_stdin_stdout_stderr(self): # to stderr at exit of subprocess. self.assertTrue(stderr.startswith("eline2\neline6\neline7\n")) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_universal_newlines_communicate_encodings(self): # Check that universal newlines mode works for various encodings, # in particular for encodings in the UTF-16 and UTF-32 families. @@ -2274,14 +2273,12 @@ def test_group_error(self): with self.assertRaises(ValueError): subprocess.check_call(ZERO_RETURN_CMD, group=65535) - @unittest.expectedFailure # TODO: RUSTPYTHON; observed gids do not match expected gids @unittest.skipUnless(hasattr(os, 'setgroups'), 'no setgroups() on platform') def test_extra_groups(self): gid = os.getegid() group_list = [65534 if gid != 65534 else 65533] self._test_extra_groups_impl(gid=gid, group_list=group_list) - @unittest.expectedFailure # TODO: RUSTPYTHON @unittest.skipUnless(hasattr(os, 'setgroups'), 'no setgroups() on platform') def test_extra_groups_empty_list(self): self._test_extra_groups_impl(gid=os.getegid(), group_list=[]) @@ -2445,7 +2442,6 @@ def raise_it(): stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, preexec_fn=raise_it) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_preexec_gc_module_failure(self): # This tests the code that disables garbage collection if the child # process will execute any Python. diff --git a/Lib/test/test_support.py b/Lib/test/test_support.py index d66afdc833c..481be2bff92 100644 --- a/Lib/test/test_support.py +++ b/Lib/test/test_support.py @@ -5,6 +5,7 @@ import logging import os import shutil +import signal import socket import stat import subprocess @@ -16,12 +17,13 @@ import warnings from test import support -from test.support import import_helper -from test.support import os_helper -from test.support import script_helper -from test.support import socket_helper -from test.support import warnings_helper -from test.support.testcase import ExtraAssertions +from test.support import ( + import_helper, + os_helper, + script_helper, + socket_helper, + warnings_helper, +) TESTFN = os_helper.TESTFN @@ -51,26 +53,26 @@ def _caplog(): root_logger.removeHandler(handler) -class TestSupport(unittest.TestCase, ExtraAssertions): +class TestSupport(unittest.TestCase): @classmethod def setUpClass(cls): - orig_filter_len = len(warnings.filters) + orig_filter_len = len(warnings._get_filters()) cls._warnings_helper_token = support.ignore_deprecations_from( "test.support.warnings_helper", like=".*used in test_support.*" ) cls._test_support_token = support.ignore_deprecations_from( __name__, like=".*You should NOT be seeing this.*" ) - assert len(warnings.filters) == orig_filter_len + 2 + assert len(warnings._get_filters()) == orig_filter_len + 2 @classmethod def tearDownClass(cls): - orig_filter_len = len(warnings.filters) + orig_filter_len = len(warnings._get_filters()) support.clear_ignored_deprecations( cls._warnings_helper_token, cls._test_support_token, ) - assert len(warnings.filters) == orig_filter_len - 2 + assert len(warnings._get_filters()) == orig_filter_len - 2 def test_ignored_deprecations_are_silent(self): """Test support.ignore_deprecations_from() silences warnings""" @@ -98,7 +100,7 @@ def test_get_original_stdout(self): self.assertEqual(support.get_original_stdout(), sys.stdout) def test_unload(self): - import sched + import sched # noqa: F401 self.assertIn("sched", sys.modules) import_helper.unload("sched") self.assertNotIn("sched", sys.modules) @@ -407,10 +409,10 @@ class Obj: with support.swap_attr(obj, "y", 5) as y: self.assertEqual(obj.y, 5) self.assertIsNone(y) - self.assertFalse(hasattr(obj, 'y')) + self.assertNotHasAttr(obj, 'y') with support.swap_attr(obj, "y", 5): del obj.y - self.assertFalse(hasattr(obj, 'y')) + self.assertNotHasAttr(obj, 'y') def test_swap_item(self): D = {"x":1} @@ -458,6 +460,7 @@ def test_detect_api_mismatch__ignore(self): self.OtherClass, self.RefClass, ignore=ignore) self.assertEqual(set(), missing_items) + @unittest.expectedFailure # TODO: RUSTPYTHON def test_check__all__(self): extra = {'tempdir'} not_exported = {'template'} @@ -469,7 +472,6 @@ def test_check__all__(self): extra = { 'TextTestResult', 'installHandler', - 'IsolatedAsyncioTestCase', } not_exported = {'load_tests', "TestProgram", "BaseTestSuite"} support.check__all__(self, @@ -562,6 +564,7 @@ def test_args_from_interpreter_flags(self): ['-Wignore', '-X', 'dev'], ['-X', 'faulthandler'], ['-X', 'importtime'], + ['-X', 'importtime=2'], ['-X', 'showrefcount'], ['-X', 'tracemalloc'], ['-X', 'tracemalloc=3'], @@ -586,7 +589,6 @@ def test_optim_args_from_interpreter_flags(self): self.check_options(opts, 'optim_args_from_interpreter_flags') @unittest.skipIf(support.is_apple_mobile, "Unstable on Apple Mobile") - @unittest.skipIf(support.is_emscripten, "Unstable in Emscripten") @unittest.skipIf(support.is_wasi, "Unavailable on WASI") def test_fd_count(self): # We cannot test the absolute value of fd_count(): on old Linux kernel @@ -614,17 +616,14 @@ def test_print_warning(self): self.check_print_warning("a\nb", 'Warning -- a\nWarning -- b\n') - # TODO: RUSTPYTHON - strftime extension not fully supported on non-Windows - @unittest.skipUnless(sys.platform == "win32" or support.is_emscripten, - "strftime extension not fully supported on non-Windows") + @unittest.expectedFailureIf(sys.platform != "win32", "TODO: RUSTPYTHON; no has_strftime_extensions yet") def test_has_strftime_extensions(self): - if support.is_emscripten or sys.platform == "win32": + if sys.platform == "win32": self.assertFalse(support.has_strftime_extensions) else: self.assertTrue(support.has_strftime_extensions) - # TODO: RUSTPYTHON - _testinternalcapi module not available - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; - _testinternalcapi module not available def test_get_recursion_depth(self): # test support.get_recursion_depth() code = textwrap.dedent(""" @@ -668,8 +667,7 @@ def test_recursive(depth, limit): """) script_helper.assert_python_ok("-c", code) - # TODO: RUSTPYTHON - stack overflow in debug mode with deep recursion - @unittest.skip("TODO: RUSTPYTHON - causes segfault in debug builds") + @unittest.skip('TODO: RUSTPYTHON; - causes segfault in debug builds') def test_recursion(self): # Test infinite_recursion() and get_recursion_available() functions. def recursive_function(depth): @@ -778,9 +776,31 @@ def test_copy_python_src_ignore(self): self.assertEqual(support.copy_python_src_ignore(path, os.listdir(path)), ignored) + def test_get_signal_name(self): + for exitcode, expected in ( + (-int(signal.SIGINT), 'SIGINT'), + (-int(signal.SIGSEGV), 'SIGSEGV'), + (128 + int(signal.SIGABRT), 'SIGABRT'), + (3221225477, "STATUS_ACCESS_VIOLATION"), + (0xC00000FD, "STATUS_STACK_OVERFLOW"), + ): + self.assertEqual(support.get_signal_name(exitcode), expected, + exitcode) + def test_linked_to_musl(self): linked = support.linked_to_musl() - self.assertIsInstance(linked, bool) + self.assertIsNotNone(linked) + if support.is_wasm32: + self.assertTrue(linked) + # The value is cached, so make sure it returns the same value again. + self.assertIs(linked, support.linked_to_musl()) + # The musl version is either triple or just a major version number. + if linked: + self.assertIsInstance(linked, tuple) + self.assertIn(len(linked), (1, 3)) + for v in linked: + self.assertIsInstance(v, int) + # XXX -follows a list of untested API # make_legacy_pyc diff --git a/Lib/test/test_symtable.py b/Lib/test/test_symtable.py index 8f1a80a5242..59f41848498 100644 --- a/Lib/test/test_symtable.py +++ b/Lib/test/test_symtable.py @@ -2,6 +2,7 @@ Test the API of the symtable module. """ +import re import textwrap import symtable import unittest @@ -202,9 +203,7 @@ class SymtableTest(unittest.TestCase): # XXX: RUSTPYTHON # U = find_block(GenericMine, "U") - - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_type(self): self.assertEqual(self.top.get_type(), "module") self.assertEqual(self.Mine.get_type(), "class") @@ -222,8 +221,7 @@ def test_type(self): self.assertEqual(self.T.get_type(), "type variable") self.assertEqual(self.U.get_type(), "type variable") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_id(self): self.assertGreater(self.top.get_id(), 0) self.assertGreater(self.Mine.get_id(), 0) @@ -256,8 +254,7 @@ def test_lineno(self): self.assertEqual(self.top.get_lineno(), 0) self.assertEqual(self.spam.get_lineno(), 14) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_function_info(self): func = self.spam self.assertEqual(sorted(func.get_parameters()), ["a", "b", "kw", "var"]) @@ -266,8 +263,7 @@ def test_function_info(self): self.assertEqual(sorted(func.get_globals()), ["bar", "glob", "some_assigned_global_var"]) self.assertEqual(self.internal.get_frees(), ("x",)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_globals(self): self.assertTrue(self.spam.lookup("glob").is_global()) self.assertFalse(self.spam.lookup("glob").is_declared_global()) @@ -280,16 +276,14 @@ def test_globals(self): self.assertTrue(self.top.lookup("some_non_assigned_global_var").is_global()) self.assertTrue(self.top.lookup("some_assigned_global_var").is_global()) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_nonlocal(self): self.assertFalse(self.spam.lookup("some_var").is_nonlocal()) self.assertTrue(self.other_internal.lookup("some_var").is_nonlocal()) expected = ("some_var",) self.assertEqual(self.other_internal.get_nonlocals(), expected) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_local(self): self.assertTrue(self.spam.lookup("x").is_local()) self.assertFalse(self.spam.lookup("bar").is_local()) @@ -297,13 +291,11 @@ def test_local(self): self.assertTrue(self.top.lookup("some_non_assigned_global_var").is_local()) self.assertTrue(self.top.lookup("some_assigned_global_var").is_local()) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_free(self): self.assertTrue(self.internal.lookup("x").is_free()) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_referenced(self): self.assertTrue(self.internal.lookup("x").is_referenced()) self.assertTrue(self.spam.lookup("internal").is_referenced()) @@ -320,8 +312,7 @@ def test_symbol_lookup(self): self.assertRaises(KeyError, self.top.lookup, "not_here") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_namespaces(self): self.assertTrue(self.top.lookup("Mine").is_namespace()) self.assertTrue(self.Mine.lookup("a_method").is_namespace()) @@ -346,16 +337,17 @@ def test_assigned(self): self.assertTrue(self.Mine.lookup("a_method").is_assigned()) self.assertFalse(self.internal.lookup("x").is_assigned()) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_annotated(self): st1 = symtable.symtable('def f():\n x: int\n', 'test', 'exec') - st2 = st1.get_children()[0] + st2 = st1.get_children()[1] + self.assertEqual(st2.get_type(), "function") self.assertTrue(st2.lookup('x').is_local()) self.assertTrue(st2.lookup('x').is_annotated()) self.assertFalse(st2.lookup('x').is_global()) st3 = symtable.symtable('def f():\n x = 1\n', 'test', 'exec') - st4 = st3.get_children()[0] + st4 = st3.get_children()[1] + self.assertEqual(st4.get_type(), "function") self.assertTrue(st4.lookup('x').is_local()) self.assertFalse(st4.lookup('x').is_annotated()) @@ -373,8 +365,7 @@ def test_annotated(self): ' x: int', 'test', 'exec') - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_imported(self): self.assertTrue(self.top.lookup("sys").is_imported()) @@ -384,28 +375,34 @@ def test_name(self): self.assertEqual(self.spam.lookup("x").get_name(), "x") self.assertEqual(self.Mine.get_name(), "Mine") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_class_get_methods(self): - self.assertEqual(self.Mine.get_methods(), ('a_method',)) + deprecation_mess = ( + re.escape('symtable.Class.get_methods() is deprecated ' + 'and will be removed in Python 3.16.') + ) + + with self.assertWarnsRegex(DeprecationWarning, deprecation_mess): + self.assertEqual(self.Mine.get_methods(), ('a_method',)) top = symtable.symtable(TEST_COMPLEX_CLASS_CODE, "?", "exec") this = find_block(top, "ComplexClass") - self.assertEqual(this.get_methods(), ( - 'a_method', 'a_method_pep_695', - 'an_async_method', 'an_async_method_pep_695', - 'a_classmethod', 'a_classmethod_pep_695', - 'an_async_classmethod', 'an_async_classmethod_pep_695', - 'a_staticmethod', 'a_staticmethod_pep_695', - 'an_async_staticmethod', 'an_async_staticmethod_pep_695', - 'a_fakemethod', 'a_fakemethod_pep_695', - 'an_async_fakemethod', 'an_async_fakemethod_pep_695', - 'glob_unassigned_meth', 'glob_unassigned_meth_pep_695', - 'glob_unassigned_async_meth', 'glob_unassigned_async_meth_pep_695', - 'glob_assigned_meth', 'glob_assigned_meth_pep_695', - 'glob_assigned_async_meth', 'glob_assigned_async_meth_pep_695', - )) + with self.assertWarnsRegex(DeprecationWarning, deprecation_mess): + self.assertEqual(this.get_methods(), ( + 'a_method', 'a_method_pep_695', + 'an_async_method', 'an_async_method_pep_695', + 'a_classmethod', 'a_classmethod_pep_695', + 'an_async_classmethod', 'an_async_classmethod_pep_695', + 'a_staticmethod', 'a_staticmethod_pep_695', + 'an_async_staticmethod', 'an_async_staticmethod_pep_695', + 'a_fakemethod', 'a_fakemethod_pep_695', + 'an_async_fakemethod', 'an_async_fakemethod_pep_695', + 'glob_unassigned_meth', 'glob_unassigned_meth_pep_695', + 'glob_unassigned_async_meth', 'glob_unassigned_async_meth_pep_695', + 'glob_assigned_meth', 'glob_assigned_meth_pep_695', + 'glob_assigned_async_meth', 'glob_assigned_async_meth_pep_695', + )) # Test generator expressions that are of type TYPE_FUNCTION # but will not be reported by get_methods() since they are @@ -418,7 +415,8 @@ def check_body(body, expected_methods): indented = textwrap.indent(body, ' ' * 4) top = symtable.symtable(f"class A:\n{indented}", "?", "exec") this = find_block(top, "A") - self.assertEqual(this.get_methods(), expected_methods) + with self.assertWarnsRegex(DeprecationWarning, deprecation_mess): + self.assertEqual(this.get_methods(), expected_methods) # statements with 'genexpr' inside it GENEXPRS = ( @@ -459,8 +457,7 @@ def check_body(body, expected_methods): check_body('\n'.join((gen, func)), ('genexpr',)) check_body('\n'.join((func, gen)), ('genexpr',)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_filename_correct(self): ### Bug tickler: SyntaxError file name correct whether error raised ### while parsing or building symbol table. @@ -492,8 +489,7 @@ def test_single(self): def test_exec(self): symbols = symtable.symtable("def f(x): return x", "?", "exec") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_bytes(self): top = symtable.symtable(TEST_CODE.encode('utf8'), "?", "exec") self.assertIsNotNone(find_block(top, "Mine")) @@ -507,8 +503,7 @@ def test_symtable_repr(self): self.assertEqual(str(self.top), "<SymbolTable for module ?>") self.assertEqual(str(self.spam), "<Function SymbolTable for spam in ?>") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_symbol_repr(self): self.assertEqual(repr(self.spam.lookup("glob")), "<symbol 'glob': GLOBAL_IMPLICIT, USE>") @@ -524,19 +519,69 @@ def test_symbol_repr(self): "<symbol 'x': FREE, USE>") self.assertEqual(repr(self.other_internal.lookup("some_var")), "<symbol 'some_var': FREE, USE|DEF_NONLOCAL|DEF_LOCAL>") - - # TODO: RUSTPYTHON - @unittest.expectedFailure + self.assertEqual(repr(self.GenericMine.lookup("T")), + "<symbol 'T': LOCAL, DEF_LOCAL|DEF_TYPE_PARAM>") + + st1 = symtable.symtable("[x for x in [1]]", "?", "exec") + self.assertEqual(repr(st1.lookup("x")), + "<symbol 'x': LOCAL, USE|DEF_LOCAL|DEF_COMP_ITER>") + + st2 = symtable.symtable("[(lambda: x) for x in [1]]", "?", "exec") + self.assertEqual(repr(st2.lookup("x")), + "<symbol 'x': CELL, DEF_LOCAL|DEF_COMP_ITER|DEF_COMP_CELL>") + + st3 = symtable.symtable("def f():\n" + " x = 1\n" + " class A:\n" + " x = 2\n" + " def method():\n" + " return x\n", + "?", "exec") + # child 0 is for __annotate__ + func_f = st3.get_children()[1] + class_A = func_f.get_children()[0] + self.assertEqual(repr(class_A.lookup('x')), + "<symbol 'x': LOCAL, DEF_LOCAL|DEF_FREE_CLASS>") + + @unittest.expectedFailure # TODO: RUSTPYTHON def test_symtable_entry_repr(self): expected = f"<symtable entry top({self.top.get_id()}), line {self.top.get_lineno()}>" self.assertEqual(repr(self.top._table), expected) + def test__symtable_refleak(self): + # Regression test for reference leak in PyUnicode_FSDecoder. + # See https://github.com/python/cpython/issues/139748. + mortal_str = 'this is a mortal string' + # check error path when 'compile_type' AC conversion failed + self.assertRaises(TypeError, symtable.symtable, '', mortal_str, 1) + + +class ComprehensionTests(unittest.TestCase): + def get_identifiers_recursive(self, st, res): + res.extend(st.get_identifiers()) + for ch in st.get_children(): + self.get_identifiers_recursive(ch, res) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 2 != 1 + def test_loopvar_in_only_one_scope(self): + # ensure that the loop variable appears only once in the symtable + comps = [ + "[x for x in [1]]", + "{x for x in [1]}", + "{x:x*x for x in [1]}", + ] + for comp in comps: + with self.subTest(comp=comp): + st = symtable.symtable(comp, "?", "exec") + ids = [] + self.get_identifiers_recursive(st, ids) + self.assertEqual(len([x for x in ids if x == 'x']), 1) + class CommandLineTest(unittest.TestCase): maxDiff = None - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_file(self): filename = os_helper.TESTFN self.addCleanup(os_helper.unlink, filename) diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index 969c2866aac..00c2a9b937b 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -7,19 +7,24 @@ import operator import os import random +import socket import struct import subprocess import sys import sysconfig import test.support +from io import StringIO +from unittest import mock from test import support from test.support import os_helper from test.support.script_helper import assert_python_ok, assert_python_failure +from test.support.socket_helper import find_unused_port from test.support import threading_helper from test.support import import_helper from test.support import force_not_colorized +from test.support import SHORT_TIMEOUT try: - from test.support import interpreters + from concurrent import interpreters except ImportError: interpreters = None import textwrap @@ -52,7 +57,7 @@ def test_original_displayhook(self): dh(None) self.assertEqual(out.getvalue(), "") - self.assertTrue(not hasattr(builtins, "_")) + self.assertNotHasAttr(builtins, "_") # sys.displayhook() requires arguments self.assertRaises(TypeError, dh) @@ -167,7 +172,7 @@ def test_original_excepthook(self): with support.captured_stderr() as err: sys.__excepthook__(*sys.exc_info()) - self.assertTrue(err.getvalue().endswith("ValueError: 42\n")) + self.assertEndsWith(err.getvalue(), "ValueError: 42\n") self.assertRaises(TypeError, sys.__excepthook__) @@ -188,7 +193,7 @@ def test_excepthook_bytes_filename(self): err = err.getvalue() self.assertIn(""" File "b'bytes_filename'", line 123\n""", err) self.assertIn(""" text\n""", err) - self.assertTrue(err.endswith("SyntaxError: msg\n")) + self.assertEndsWith(err, "SyntaxError: msg\n") def test_excepthook(self): with test.support.captured_output("stderr") as stderr: @@ -206,7 +211,7 @@ class SysModuleTest(unittest.TestCase): def tearDown(self): test.support.reap_children() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_exit(self): # call with two arguments self.assertRaises(TypeError, sys.exit, 42, 42) @@ -266,8 +271,7 @@ def check_exit_message(code, expected, **env_vars): rc, out, err = assert_python_failure('-c', code, **env_vars) self.assertEqual(rc, 1) self.assertEqual(out, b'') - self.assertTrue(err.startswith(expected), - "%s doesn't start with %s" % (ascii(err), ascii(expected))) + self.assertStartsWith(err, expected) # test that stderr buffer is flushed before the exit message is written # into stderr @@ -399,36 +403,6 @@ def test_setrecursionlimit_to_depth(self): finally: sys.setrecursionlimit(old_limit) - @unittest.skipUnless(support.Py_GIL_DISABLED, "only meaningful if the GIL is disabled") - @threading_helper.requires_working_threading() - def test_racing_recursion_limit(self): - from threading import Thread - def something_recursive(): - def count(n): - if n > 0: - return count(n - 1) + 1 - return 0 - - count(50) - - def set_recursion_limit(): - for limit in range(100, 200): - sys.setrecursionlimit(limit) - - threads = [] - for _ in range(5): - threads.append(Thread(target=set_recursion_limit)) - - for _ in range(5): - threads.append(Thread(target=something_recursive)) - - with threading_helper.catch_threading_exception() as cm: - with threading_helper.start_threads(threads): - pass - - if cm.exc_value: - raise cm.exc_value - @unittest.expectedFailure # TODO: RUSTPYTHON def test_getwindowsversion(self): # Raise SkipTest if sys doesn't have getwindowsversion attribute @@ -467,7 +441,7 @@ def test_call_tracing(self): @unittest.skipUnless(hasattr(sys, "setdlopenflags"), 'test needs sys.setdlopenflags()') def test_dlopenflags(self): - self.assertTrue(hasattr(sys, "getdlopenflags")) + self.assertHasAttr(sys, "getdlopenflags") self.assertRaises(TypeError, sys.getdlopenflags, 42) oldflags = sys.getdlopenflags() self.assertRaises(TypeError, sys.setdlopenflags) @@ -656,8 +630,7 @@ def g456(): # And the next record must be for g456(). filename, lineno, funcname, sourceline = stack[i+1] self.assertEqual(funcname, "g456") - self.assertTrue((sourceline.startswith("if leave_g.wait(") or - sourceline.startswith("g_raised.set()"))) + self.assertStartsWith(sourceline, ("if leave_g.wait(", "g_raised.set()")) finally: # Reap the spawned thread. leave_g.set() @@ -757,6 +730,8 @@ def test_attributes(self): self.assertIn(sys.float_repr_style, ('short', 'legacy')) if not sys.platform.startswith('win'): self.assertIsInstance(sys.abiflags, str) + else: + self.assertFalse(hasattr(sys, 'abiflags')) def test_thread_info(self): info = sys.thread_info @@ -882,6 +857,7 @@ def test_subinterp_intern_singleton(self): ''')) self.assertTrue(sys._is_interned(s)) + @unittest.expectedFailure # TODO: RUSTPYTHON; needs update for context_aware_warnings def test_sys_flags(self): self.assertTrue(sys.flags) attrs = ("debug", @@ -891,7 +867,7 @@ def test_sys_flags(self): "hash_randomization", "isolated", "dev_mode", "utf8_mode", "warn_default_encoding", "safe_path", "int_max_str_digits") for attr in attrs: - self.assertTrue(hasattr(sys.flags, attr), attr) + self.assertHasAttr(sys.flags, attr) attr_type = bool if attr in ("dev_mode", "safe_path") else int self.assertEqual(type(getattr(sys.flags, attr)), attr_type, attr) self.assertTrue(repr(sys.flags)) @@ -918,9 +894,11 @@ def test_sys_getwindowsversion_no_instantiation(self): @test.support.cpython_only def test_clear_type_cache(self): - sys._clear_type_cache() + with self.assertWarnsRegex(DeprecationWarning, + r"sys\._clear_type_cache\(\) is deprecated.*"): + sys._clear_type_cache() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.skip('TODO: RUSTPYTHON; cp424 encoding not supported, causes panic') @force_not_colorized @support.requires_subprocess() def test_ioencoding(self): @@ -1100,10 +1078,11 @@ def test_implementation(self): levels = {'alpha': 0xA, 'beta': 0xB, 'candidate': 0xC, 'final': 0xF} - self.assertTrue(hasattr(sys.implementation, 'name')) - self.assertTrue(hasattr(sys.implementation, 'version')) - self.assertTrue(hasattr(sys.implementation, 'hexversion')) - self.assertTrue(hasattr(sys.implementation, 'cache_tag')) + self.assertHasAttr(sys.implementation, 'name') + self.assertHasAttr(sys.implementation, 'version') + self.assertHasAttr(sys.implementation, 'hexversion') + self.assertHasAttr(sys.implementation, 'cache_tag') + self.assertHasAttr(sys.implementation, 'supports_isolated_interpreters') version = sys.implementation.version self.assertEqual(version[:2], (version.major, version.minor)) @@ -1117,6 +1096,15 @@ def test_implementation(self): self.assertEqual(sys.implementation.name, sys.implementation.name.lower()) + # https://peps.python.org/pep-0734 + sii = sys.implementation.supports_isolated_interpreters + self.assertIsInstance(sii, bool) + if test.support.check_impl_detail(cpython=True): + if test.support.is_emscripten or test.support.is_wasi: + self.assertFalse(sii) + else: + self.assertTrue(sii) + @test.support.cpython_only def test_debugmallocstats(self): # Test sys._debugmallocstats() @@ -1127,14 +1115,10 @@ def test_debugmallocstats(self): # Output of sys._debugmallocstats() depends on configure flags. # The sysconfig vars are not available on Windows. if sys.platform != "win32": - with_freelists = sysconfig.get_config_var("WITH_FREELISTS") with_pymalloc = sysconfig.get_config_var("WITH_PYMALLOC") - if with_freelists: - self.assertIn(b"free PyDictObjects", err) + self.assertIn(b"free PyDictObjects", err) if with_pymalloc: self.assertIn(b'Small block threshold', err) - if not with_freelists and not with_pymalloc: - self.assertFalse(err) # The function has no parameter self.assertRaises(TypeError, sys._debugmallocstats, True) @@ -1165,18 +1149,20 @@ def test_getallocatedblocks(self): # about the underlying implementation: the function might # return 0 or something greater. self.assertGreaterEqual(a, 0) + gc.collect() + b = sys.getallocatedblocks() + self.assertLessEqual(b, a) try: - # While we could imagine a Python session where the number of - # multiple buffer objects would exceed the sharing of references, - # it is unlikely to happen in a normal test run. - self.assertLess(a, sys.gettotalrefcount()) + # The reported blocks will include immortalized strings, but the + # total ref count will not. This will sanity check that among all + # other objects (those eligible for garbage collection) there + # are more references being tracked than allocated blocks. + interned_immortal = sys.getunicodeinternedsize(_only_immortal=True) + self.assertLess(a - interned_immortal, sys.gettotalrefcount()) except AttributeError: # gettotalrefcount() not available pass gc.collect() - b = sys.getallocatedblocks() - self.assertLessEqual(b, a) - gc.collect() c = sys.getallocatedblocks() self.assertIn(c, range(b - 50, b + 50)) @@ -1186,7 +1172,6 @@ def test_is_gil_enabled(self): else: self.assertTrue(sys._is_gil_enabled()) - @unittest.expectedFailure # TODO: RUSTPYTHON; AtExit.__del__ is not invoked because module destruction is missing. def test_is_finalizing(self): self.assertIs(sys.is_finalizing(), False) # Don't use the atexit module because _Py_Finalizing is only set @@ -1324,7 +1309,6 @@ def test_module_names(self): for name in sys.stdlib_module_names: self.assertIsInstance(name, str) - @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'sys' has no attribute '_stdlib_dir' def test_stdlib_dir(self): os = import_helper.import_fresh_module('os') marker = getattr(os, '__file__', None) @@ -1443,7 +1427,7 @@ def __del__(self): else: self.assertIn("ValueError", report) self.assertIn("del is broken", report) - self.assertTrue(report.endswith("\n")) + self.assertEndsWith(report, "\n") def test_original_unraisablehook_exception_qualname(self): # See bpo-41031, bpo-45083. @@ -1688,15 +1672,19 @@ class C(object): pass # float check(float(0), size('d')) # sys.floatinfo - check(sys.float_info, vsize('') + self.P * len(sys.float_info)) + check(sys.float_info, self.P + vsize('') + self.P * len(sys.float_info)) # frame def func(): return sys._getframe() x = func() - check(x, size('3Pi2c2P7P2ic??2P')) + if support.Py_GIL_DISABLED: + INTERPRETER_FRAME = '9PihcP' + else: + INTERPRETER_FRAME = '9PhcP' + check(x, size('3PiccPPP' + INTERPRETER_FRAME + 'P')) # function def func(): pass - check(func, size('15Pi')) + check(func, size('16Pi')) class c(): @staticmethod def foo(): @@ -1710,7 +1698,7 @@ def bar(cls): check(bar, size('PP')) # generator def get_gen(): yield 1 - check(get_gen(), size('PP4P4c7P2ic??2P')) + check(get_gen(), size('6P4c' + INTERPRETER_FRAME + 'P')) # iterator check(iter('abc'), size('lP')) # callable-iterator @@ -1792,13 +1780,14 @@ def delx(self): del self.__x # super check(super(int), size('3P')) # tuple - check((), vsize('')) - check((1,2,3), vsize('') + 3*self.P) + check((), vsize('') + self.P) + check((1,2,3), vsize('') + self.P + 3*self.P) # type # static type: PyTypeObject fmt = 'P2nPI13Pl4Pn9Pn12PIPc' s = vsize(fmt) check(int, s) + typeid = 'n' if support.Py_GIL_DISABLED else '' # class s = vsize(fmt + # PyTypeObject '4P' # PyAsyncMethods @@ -1806,8 +1795,9 @@ def delx(self): del self.__x '3P' # PyMappingMethods '10P' # PySequenceMethods '2P' # PyBufferProcs - '6P' - '1PIP' # Specializer cache + '7P' + '1PIP' # Specializer cache + + typeid # heap type id (free-threaded only) ) class newstyleclass(object): pass # Separate block for PyDictKeysObject with 8 keys and 5 entries @@ -1912,8 +1902,10 @@ def test_pythontypes(self): # symtable entry # XXX # sys.flags - # FIXME: The +1 will not be necessary once gh-122575 is fixed - check(sys.flags, vsize('') + self.P * (1 + len(sys.flags))) + # FIXME: The +3 is for the 'gil', 'thread_inherit_context' and + # 'context_aware_warnings' flags and will not be necessary once + # gh-122575 is fixed + check(sys.flags, vsize('') + self.P + self.P * (3 + len(sys.flags))) def test_asyncgen_hooks(self): old = sys.get_asyncgen_hooks() @@ -1971,5 +1963,318 @@ def write(self, s): self.assertEqual(out, b"") self.assertEqual(err, b"") +@test.support.support_remote_exec_only +@test.support.cpython_only +class TestRemoteExec(unittest.TestCase): + def tearDown(self): + test.support.reap_children() + + def _run_remote_exec_test(self, script_code, python_args=None, env=None, + prologue='', + script_path=os_helper.TESTFN + '_remote.py'): + # Create the script that will be remotely executed + self.addCleanup(os_helper.unlink, script_path) + + with open(script_path, 'w') as f: + f.write(script_code) + + # Create and run the target process + target = os_helper.TESTFN + '_target.py' + self.addCleanup(os_helper.unlink, target) + + port = find_unused_port() + + with open(target, 'w') as f: + f.write(f''' +import sys +import time +import socket + +# Connect to the test process +sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) +sock.connect(('localhost', {port})) + +{prologue} + +# Signal that the process is ready +sock.sendall(b"ready") + +print("Target process running...") + +# Wait for remote script to be executed +# (the execution will happen as the following +# code is processed as soon as the recv call +# unblocks) +sock.recv(1024) + +# Do a bunch of work to give the remote script time to run +x = 0 +for i in range(100): + x += i + +# Write confirmation back +sock.sendall(b"executed") +sock.close() +''') + + # Start the target process and capture its output + cmd = [sys.executable] + if python_args: + cmd.extend(python_args) + cmd.append(target) + + # Create a socket server to communicate with the target process + server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + server_socket.bind(('localhost', port)) + server_socket.settimeout(SHORT_TIMEOUT) + server_socket.listen(1) + + with subprocess.Popen(cmd, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + env=env, + ) as proc: + client_socket = None + try: + # Accept connection from target process + client_socket, _ = server_socket.accept() + server_socket.close() + + response = client_socket.recv(1024) + self.assertEqual(response, b"ready") + + # Try remote exec on the target process + sys.remote_exec(proc.pid, script_path) + + # Signal script to continue + client_socket.sendall(b"continue") + + # Wait for execution confirmation + response = client_socket.recv(1024) + self.assertEqual(response, b"executed") + + # Return output for test verification + stdout, stderr = proc.communicate(timeout=10.0) + return proc.returncode, stdout, stderr + except PermissionError: + self.skipTest("Insufficient permissions to execute code in remote process") + finally: + if client_socket is not None: + client_socket.close() + proc.kill() + proc.terminate() + proc.wait(timeout=SHORT_TIMEOUT) + + def test_remote_exec(self): + """Test basic remote exec functionality""" + script = 'print("Remote script executed successfully!")' + returncode, stdout, stderr = self._run_remote_exec_test(script) + # self.assertEqual(returncode, 0) + self.assertIn(b"Remote script executed successfully!", stdout) + self.assertEqual(stderr, b"") + + def test_remote_exec_bytes(self): + script = 'print("Remote script executed successfully!")' + script_path = os.fsencode(os_helper.TESTFN) + b'_bytes_remote.py' + returncode, stdout, stderr = self._run_remote_exec_test(script, + script_path=script_path) + self.assertIn(b"Remote script executed successfully!", stdout) + self.assertEqual(stderr, b"") + + @unittest.skipUnless(os_helper.TESTFN_UNDECODABLE, 'requires undecodable path') + @unittest.skipIf(sys.platform == 'darwin', + 'undecodable paths are not supported on macOS') + def test_remote_exec_undecodable(self): + script = 'print("Remote script executed successfully!")' + script_path = os_helper.TESTFN_UNDECODABLE + b'_undecodable_remote.py' + for script_path in [script_path, os.fsdecode(script_path)]: + returncode, stdout, stderr = self._run_remote_exec_test(script, + script_path=script_path) + self.assertIn(b"Remote script executed successfully!", stdout) + self.assertEqual(stderr, b"") + + def test_remote_exec_with_self_process(self): + """Test remote exec with the target process being the same as the test process""" + + code = 'import sys;print("Remote script executed successfully!", file=sys.stderr)' + file = os_helper.TESTFN + '_remote_self.py' + with open(file, 'w') as f: + f.write(code) + self.addCleanup(os_helper.unlink, file) + with mock.patch('sys.stderr', new_callable=StringIO) as mock_stderr: + with mock.patch('sys.stdout', new_callable=StringIO) as mock_stdout: + sys.remote_exec(os.getpid(), os.path.abspath(file)) + print("Done") + self.assertEqual(mock_stderr.getvalue(), "Remote script executed successfully!\n") + self.assertEqual(mock_stdout.getvalue(), "Done\n") + + def test_remote_exec_raises_audit_event(self): + """Test remote exec raises an audit event""" + prologue = '''\ +import sys +def audit_hook(event, arg): + print(f"Audit event: {event}, arg: {arg}".encode("ascii", errors="replace")) +sys.addaudithook(audit_hook) +''' + script = ''' +print("Remote script executed successfully!") +''' + returncode, stdout, stderr = self._run_remote_exec_test(script, prologue=prologue) + self.assertEqual(returncode, 0) + self.assertIn(b"Remote script executed successfully!", stdout) + self.assertIn(b"Audit event: cpython.remote_debugger_script, arg: ", stdout) + self.assertEqual(stderr, b"") + + def test_remote_exec_with_exception(self): + """Test remote exec with an exception raised in the target process + + The exception should be raised in the main thread of the target process + but not crash the target process. + """ + script = ''' +raise Exception("Remote script exception") +''' + returncode, stdout, stderr = self._run_remote_exec_test(script) + self.assertEqual(returncode, 0) + self.assertIn(b"Remote script exception", stderr) + self.assertEqual(stdout.strip(), b"Target process running...") + + def test_new_namespace_for_each_remote_exec(self): + """Test that each remote_exec call gets its own namespace.""" + script = textwrap.dedent( + """ + assert globals() is not __import__("__main__").__dict__ + print("Remote script executed successfully!") + """ + ) + returncode, stdout, stderr = self._run_remote_exec_test(script) + self.assertEqual(returncode, 0) + self.assertEqual(stderr, b"") + self.assertIn(b"Remote script executed successfully", stdout) + + def test_remote_exec_disabled_by_env(self): + """Test remote exec is disabled when PYTHON_DISABLE_REMOTE_DEBUG is set""" + env = os.environ.copy() + env['PYTHON_DISABLE_REMOTE_DEBUG'] = '1' + with self.assertRaisesRegex(RuntimeError, "Remote debugging is not enabled in the remote process"): + self._run_remote_exec_test("print('should not run')", env=env) + + def test_remote_exec_disabled_by_xoption(self): + """Test remote exec is disabled with -Xdisable-remote-debug""" + with self.assertRaisesRegex(RuntimeError, "Remote debugging is not enabled in the remote process"): + self._run_remote_exec_test("print('should not run')", python_args=['-Xdisable-remote-debug']) + + def test_remote_exec_invalid_pid(self): + """Test remote exec with invalid process ID""" + with self.assertRaises(OSError): + sys.remote_exec(99999, "print('should not run')") + + def test_remote_exec_invalid_script(self): + """Test remote exec with invalid script type""" + with self.assertRaises(TypeError): + sys.remote_exec(0, None) + with self.assertRaises(TypeError): + sys.remote_exec(0, 123) + + def test_remote_exec_syntax_error(self): + """Test remote exec with syntax error in script""" + script = ''' +this is invalid python code +''' + returncode, stdout, stderr = self._run_remote_exec_test(script) + self.assertEqual(returncode, 0) + self.assertIn(b"SyntaxError", stderr) + self.assertEqual(stdout.strip(), b"Target process running...") + + def test_remote_exec_invalid_script_path(self): + """Test remote exec with invalid script path""" + with self.assertRaises(OSError): + sys.remote_exec(os.getpid(), "invalid_script_path") + + def test_remote_exec_in_process_without_debug_fails_envvar(self): + """Test remote exec in a process without remote debugging enabled""" + script = os_helper.TESTFN + '_remote.py' + self.addCleanup(os_helper.unlink, script) + with open(script, 'w') as f: + f.write('print("Remote script executed successfully!")') + env = os.environ.copy() + env['PYTHON_DISABLE_REMOTE_DEBUG'] = '1' + + _, out, err = assert_python_failure('-c', f'import os, sys; sys.remote_exec(os.getpid(), "{script}")', **env) + self.assertIn(b"Remote debugging is not enabled", err) + self.assertEqual(out, b"") + + def test_remote_exec_in_process_without_debug_fails_xoption(self): + """Test remote exec in a process without remote debugging enabled""" + script = os_helper.TESTFN + '_remote.py' + self.addCleanup(os_helper.unlink, script) + with open(script, 'w') as f: + f.write('print("Remote script executed successfully!")') + + _, out, err = assert_python_failure('-Xdisable-remote-debug', '-c', f'import os, sys; sys.remote_exec(os.getpid(), "{script}")') + self.assertIn(b"Remote debugging is not enabled", err) + self.assertEqual(out, b"") + +class TestSysJIT(unittest.TestCase): + + def test_jit_is_available(self): + available = sys._jit.is_available() + script = f"import sys; assert sys._jit.is_available() is {available}" + assert_python_ok("-c", script, PYTHON_JIT="0") + assert_python_ok("-c", script, PYTHON_JIT="1") + + def test_jit_is_enabled(self): + available = sys._jit.is_available() + script = "import sys; assert sys._jit.is_enabled() is {enabled}" + assert_python_ok("-c", script.format(enabled=False), PYTHON_JIT="0") + assert_python_ok("-c", script.format(enabled=available), PYTHON_JIT="1") + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_jit_is_active(self): + available = sys._jit.is_available() + script = textwrap.dedent( + """ + import _testcapi + import _testinternalcapi + import sys + + def frame_0_interpreter() -> None: + assert sys._jit.is_active() is False + + def frame_1_interpreter() -> None: + assert sys._jit.is_active() is False + frame_0_interpreter() + assert sys._jit.is_active() is False + + def frame_2_jit(expected: bool) -> None: + # Inlined into the last loop of frame_3_jit: + assert sys._jit.is_active() is expected + # Insert C frame: + _testcapi.pyobject_vectorcall(frame_1_interpreter, None, None) + assert sys._jit.is_active() is expected + + def frame_3_jit() -> None: + # JITs just before the last loop: + for i in range(_testinternalcapi.TIER2_THRESHOLD + 1): + # Careful, doing this in the reverse order breaks tracing: + expected = {enabled} and i == _testinternalcapi.TIER2_THRESHOLD + assert sys._jit.is_active() is expected + frame_2_jit(expected) + assert sys._jit.is_active() is expected + + def frame_4_interpreter() -> None: + assert sys._jit.is_active() is False + frame_3_jit() + assert sys._jit.is_active() is False + + assert sys._jit.is_active() is False + frame_4_interpreter() + assert sys._jit.is_active() is False + """ + ) + assert_python_ok("-c", script.format(enabled=False), PYTHON_JIT="0") + assert_python_ok("-c", script.format(enabled=available), PYTHON_JIT="1") + + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_tabnanny.py b/Lib/test/test_tabnanny.py index aa71166a380..4ac018cc87f 100644 --- a/Lib/test/test_tabnanny.py +++ b/Lib/test/test_tabnanny.py @@ -217,8 +217,7 @@ def test_when_tokenize_tokenerror(self): with self.assertRaises(SystemExit): self.verify_tabnanny_check(file_path, err=err) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; A python source code file eligible for raising `tabnanny.NannyNag`. def test_when_nannynag_error_verbose(self): """A python source code file eligible for raising `tabnanny.NannyNag`. @@ -232,8 +231,7 @@ def test_when_nannynag_error_verbose(self): tabnanny.verbose = 1 self.verify_tabnanny_check(file_path, out=out) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; A python source code file eligible for raising `tabnanny.NannyNag`. def test_when_nannynag_error(self): """A python source code file eligible for raising `tabnanny.NannyNag`.""" with TemporaryPyFile(SOURCE_CODES["nannynag_errored"]) as file_path: @@ -318,8 +316,7 @@ def validate_cmd(self, *args, stdout="", stderr="", partial=False, expect_failur self.assertListEqual(out.splitlines(), stdout.splitlines()) self.assertListEqual(err.splitlines(), stderr.splitlines()) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; Should displays error when errored python file is given. def test_with_errored_file(self): """Should displays error when errored python file is given.""" with TemporaryPyFile(SOURCE_CODES["wrong_indented"]) as file_path: @@ -345,8 +342,7 @@ def test_quiet_flag(self): stdout = f"{file_path}\n" self.validate_cmd("-q", file_path, stdout=stdout) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_verbose_mode(self): """Should display more error information if verbose mode is on.""" with TemporaryPyFile(SOURCE_CODES["nannynag_errored"]) as path: @@ -355,8 +351,7 @@ def test_verbose_mode(self): ).strip() self.validate_cmd("-v", path, stdout=stdout, partial=True) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_double_verbose_mode(self): """Should display detailed error information if double verbose is on.""" with TemporaryPyFile(SOURCE_CODES["nannynag_errored"]) as path: diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py index d29c6918234..92ce7c32a00 100644 --- a/Lib/test/test_tarfile.py +++ b/Lib/test/test_tarfile.py @@ -456,7 +456,6 @@ def test_premature_end_of_archive(self): with self.assertRaisesRegex(tarfile.ReadError, "unexpected end of data"): tar.extractfile(t).read() - @unittest.skip("TODO: RUSTPYTHON, infinite recursion") def test_length_zero_header(self): # bpo-39017 (CVE-2019-20907): reading a zero-length header should fail # with an exception diff --git a/Lib/test/test_termios.py b/Lib/test/test_termios.py new file mode 100644 index 00000000000..e7ebb20b120 --- /dev/null +++ b/Lib/test/test_termios.py @@ -0,0 +1,313 @@ +import errno +import os +import sys +import tempfile +import threading +import unittest +from test import support +from test.support import threading_helper +from test.support.import_helper import import_module + +termios = import_module('termios') + + +@unittest.skipUnless(hasattr(os, 'openpty'), "need os.openpty()") +class TestFunctions(unittest.TestCase): + + def setUp(self): + self.master_fd, self.fd = os.openpty() + self.addCleanup(os.close, self.master_fd) + self.stream = self.enterContext(open(self.fd, 'wb', buffering=0)) + tmp = self.enterContext(tempfile.TemporaryFile(mode='wb', buffering=0)) + self.bad_fd = tmp.fileno() + + def assertRaisesTermiosError(self, err, callable, *args): + # Some versions of Android return EACCES when calling termios functions + # on a regular file. + errs = [err] + if sys.platform == 'android' and err == errno.ENOTTY: + errs.append(errno.EACCES) + + with self.assertRaises(termios.error) as cm: + callable(*args) + self.assertIn(cm.exception.args[0], errs) + + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: Expected type 'int' but 'FileIO' found. + def test_tcgetattr(self): + attrs = termios.tcgetattr(self.fd) + self.assertIsInstance(attrs, list) + self.assertEqual(len(attrs), 7) + for i in range(6): + self.assertIsInstance(attrs[i], int) + iflag, oflag, cflag, lflag, ispeed, ospeed, cc = attrs + self.assertIsInstance(cc, list) + self.assertEqual(len(cc), termios.NCCS) + for i, x in enumerate(cc): + if ((lflag & termios.ICANON) == 0 and + (i == termios.VMIN or i == termios.VTIME)): + self.assertIsInstance(x, int) + else: + self.assertIsInstance(x, bytes) + self.assertEqual(len(x), 1) + self.assertEqual(termios.tcgetattr(self.stream), attrs) + + @unittest.skip("TODO: RUSTPYTHON segfault") + def test_tcgetattr_errors(self): + self.assertRaisesTermiosError(errno.ENOTTY, termios.tcgetattr, self.bad_fd) + self.assertRaises(ValueError, termios.tcgetattr, -1) + self.assertRaises(OverflowError, termios.tcgetattr, 2**1000) + self.assertRaises(TypeError, termios.tcgetattr, object()) + self.assertRaises(TypeError, termios.tcgetattr) + + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: Expected type 'int' but 'FileIO' found. + def test_tcsetattr(self): + attrs = termios.tcgetattr(self.fd) + termios.tcsetattr(self.fd, termios.TCSANOW, attrs) + termios.tcsetattr(self.fd, termios.TCSADRAIN, attrs) + termios.tcsetattr(self.fd, termios.TCSAFLUSH, attrs) + termios.tcsetattr(self.stream, termios.TCSANOW, attrs) + + @unittest.skip("TODO: RUSTPYTHON segfault") + def test_tcsetattr_errors(self): + attrs = termios.tcgetattr(self.fd) + self.assertRaises(TypeError, termios.tcsetattr, self.fd, termios.TCSANOW, tuple(attrs)) + self.assertRaises(TypeError, termios.tcsetattr, self.fd, termios.TCSANOW, attrs[:-1]) + self.assertRaises(TypeError, termios.tcsetattr, self.fd, termios.TCSANOW, attrs + [0]) + for i in range(6): + attrs2 = attrs[:] + attrs2[i] = 2**1000 + self.assertRaises(OverflowError, termios.tcsetattr, self.fd, termios.TCSANOW, attrs2) + attrs2[i] = object() + self.assertRaises(TypeError, termios.tcsetattr, self.fd, termios.TCSANOW, attrs2) + self.assertRaises(TypeError, termios.tcsetattr, self.fd, termios.TCSANOW, attrs[:-1] + [attrs[-1][:-1]]) + self.assertRaises(TypeError, termios.tcsetattr, self.fd, termios.TCSANOW, attrs[:-1] + [attrs[-1] + [b'\0']]) + for i in range(len(attrs[-1])): + attrs2 = attrs[:] + attrs2[-1] = attrs2[-1][:] + attrs2[-1][i] = 2**1000 + self.assertRaises(OverflowError, termios.tcsetattr, self.fd, termios.TCSANOW, attrs2) + attrs2[-1][i] = object() + self.assertRaises(TypeError, termios.tcsetattr, self.fd, termios.TCSANOW, attrs2) + attrs2[-1][i] = b'' + self.assertRaises(TypeError, termios.tcsetattr, self.fd, termios.TCSANOW, attrs2) + attrs2[-1][i] = b'\0\0' + self.assertRaises(TypeError, termios.tcsetattr, self.fd, termios.TCSANOW, attrs2) + self.assertRaises(TypeError, termios.tcsetattr, self.fd, termios.TCSANOW, object()) + self.assertRaises(TypeError, termios.tcsetattr, self.fd, termios.TCSANOW) + self.assertRaisesTermiosError(errno.EINVAL, termios.tcsetattr, self.fd, -1, attrs) + self.assertRaises(OverflowError, termios.tcsetattr, self.fd, 2**1000, attrs) + self.assertRaises(TypeError, termios.tcsetattr, self.fd, object(), attrs) + self.assertRaisesTermiosError(errno.ENOTTY, termios.tcsetattr, self.bad_fd, termios.TCSANOW, attrs) + self.assertRaises(ValueError, termios.tcsetattr, -1, termios.TCSANOW, attrs) + self.assertRaises(OverflowError, termios.tcsetattr, 2**1000, termios.TCSANOW, attrs) + self.assertRaises(TypeError, termios.tcsetattr, object(), termios.TCSANOW, attrs) + self.assertRaises(TypeError, termios.tcsetattr, self.fd, termios.TCSANOW) + + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: Expected type 'int' but 'FileIO' found. + @support.skip_android_selinux('tcsendbreak') + def test_tcsendbreak(self): + try: + termios.tcsendbreak(self.fd, 1) + except termios.error as exc: + if exc.args[0] == errno.ENOTTY and sys.platform.startswith(('freebsd', "netbsd")): + self.skipTest('termios.tcsendbreak() is not supported ' + 'with pseudo-terminals (?) on this platform') + raise + termios.tcsendbreak(self.stream, 1) + + @unittest.skip("TODO: RUSTPYTHON segfault") + @support.skip_android_selinux('tcsendbreak') + def test_tcsendbreak_errors(self): + self.assertRaises(OverflowError, termios.tcsendbreak, self.fd, 2**1000) + self.assertRaises(TypeError, termios.tcsendbreak, self.fd, 0.0) + self.assertRaises(TypeError, termios.tcsendbreak, self.fd, object()) + self.assertRaisesTermiosError(errno.ENOTTY, termios.tcsendbreak, self.bad_fd, 0) + self.assertRaises(ValueError, termios.tcsendbreak, -1, 0) + self.assertRaises(OverflowError, termios.tcsendbreak, 2**1000, 0) + self.assertRaises(TypeError, termios.tcsendbreak, object(), 0) + self.assertRaises(TypeError, termios.tcsendbreak, self.fd) + + @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: Expected type 'int' but 'FileIO' found. + @support.skip_android_selinux('tcdrain') + def test_tcdrain(self): + termios.tcdrain(self.fd) + termios.tcdrain(self.stream) + + @unittest.skip("TODO: RUSTPYTHON segfault") + @support.skip_android_selinux('tcdrain') + def test_tcdrain_errors(self): + self.assertRaisesTermiosError(errno.ENOTTY, termios.tcdrain, self.bad_fd) + self.assertRaises(ValueError, termios.tcdrain, -1) + self.assertRaises(OverflowError, termios.tcdrain, 2**1000) + self.assertRaises(TypeError, termios.tcdrain, object()) + self.assertRaises(TypeError, termios.tcdrain) + + def test_tcflush(self): + termios.tcflush(self.fd, termios.TCIFLUSH) + termios.tcflush(self.fd, termios.TCOFLUSH) + termios.tcflush(self.fd, termios.TCIOFLUSH) + + @unittest.skip("TODO: RUSTPYTHON segfault") + def test_tcflush_errors(self): + self.assertRaisesTermiosError(errno.EINVAL, termios.tcflush, self.fd, -1) + self.assertRaises(OverflowError, termios.tcflush, self.fd, 2**1000) + self.assertRaises(TypeError, termios.tcflush, self.fd, object()) + self.assertRaisesTermiosError(errno.ENOTTY, termios.tcflush, self.bad_fd, termios.TCIFLUSH) + self.assertRaises(ValueError, termios.tcflush, -1, termios.TCIFLUSH) + self.assertRaises(OverflowError, termios.tcflush, 2**1000, termios.TCIFLUSH) + self.assertRaises(TypeError, termios.tcflush, object(), termios.TCIFLUSH) + self.assertRaises(TypeError, termios.tcflush, self.fd) + + def test_tcflush_clear_input_or_output(self): + wfd = self.fd + rfd = self.master_fd + # The data is buffered in the input buffer on Linux, and in + # the output buffer on other platforms. + inbuf = sys.platform in ('linux', 'android') + + os.write(wfd, b'abcdef') + self.assertEqual(os.read(rfd, 2), b'ab') + if inbuf: + # don't flush input + termios.tcflush(rfd, termios.TCOFLUSH) + else: + # don't flush output + termios.tcflush(wfd, termios.TCIFLUSH) + self.assertEqual(os.read(rfd, 2), b'cd') + if inbuf: + # flush input + termios.tcflush(rfd, termios.TCIFLUSH) + else: + # flush output + termios.tcflush(wfd, termios.TCOFLUSH) + os.write(wfd, b'ABCDEF') + self.assertEqual(os.read(rfd, 1024), b'ABCDEF') + + @support.skip_android_selinux('tcflow') + def test_tcflow(self): + termios.tcflow(self.fd, termios.TCOOFF) + termios.tcflow(self.fd, termios.TCOON) + termios.tcflow(self.fd, termios.TCIOFF) + termios.tcflow(self.fd, termios.TCION) + + @unittest.skip("TODO: RUSTPYTHON segfault") + @support.skip_android_selinux('tcflow') + def test_tcflow_errors(self): + self.assertRaisesTermiosError(errno.EINVAL, termios.tcflow, self.fd, -1) + self.assertRaises(OverflowError, termios.tcflow, self.fd, 2**1000) + self.assertRaises(TypeError, termios.tcflow, self.fd, object()) + self.assertRaisesTermiosError(errno.ENOTTY, termios.tcflow, self.bad_fd, termios.TCOON) + self.assertRaises(ValueError, termios.tcflow, -1, termios.TCOON) + self.assertRaises(OverflowError, termios.tcflow, 2**1000, termios.TCOON) + self.assertRaises(TypeError, termios.tcflow, object(), termios.TCOON) + self.assertRaises(TypeError, termios.tcflow, self.fd) + + @support.skip_android_selinux('tcflow') + @unittest.skipUnless(sys.platform in ('linux', 'android'), 'only works on Linux') + def test_tcflow_suspend_and_resume_output(self): + wfd = self.fd + rfd = self.master_fd + write_suspended = threading.Event() + write_finished = threading.Event() + + def writer(): + os.write(wfd, b'abc') + self.assertTrue(write_suspended.wait(support.SHORT_TIMEOUT)) + os.write(wfd, b'def') + write_finished.set() + + with threading_helper.start_threads([threading.Thread(target=writer)]): + self.assertEqual(os.read(rfd, 3), b'abc') + try: + try: + termios.tcflow(wfd, termios.TCOOFF) + finally: + write_suspended.set() + self.assertFalse(write_finished.wait(0.5), + 'output was not suspended') + finally: + termios.tcflow(wfd, termios.TCOON) + self.assertTrue(write_finished.wait(support.SHORT_TIMEOUT), + 'output was not resumed') + self.assertEqual(os.read(rfd, 1024), b'def') + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'termios' has no attribute 'tcgetwinsize' + def test_tcgetwinsize(self): + size = termios.tcgetwinsize(self.fd) + self.assertIsInstance(size, tuple) + self.assertEqual(len(size), 2) + self.assertIsInstance(size[0], int) + self.assertIsInstance(size[1], int) + self.assertEqual(termios.tcgetwinsize(self.stream), size) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'termios' has no attribute 'tcgetwinsize' + def test_tcgetwinsize_errors(self): + self.assertRaisesTermiosError(errno.ENOTTY, termios.tcgetwinsize, self.bad_fd) + self.assertRaises(ValueError, termios.tcgetwinsize, -1) + self.assertRaises(OverflowError, termios.tcgetwinsize, 2**1000) + self.assertRaises(TypeError, termios.tcgetwinsize, object()) + self.assertRaises(TypeError, termios.tcgetwinsize) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'termios' has no attribute 'tcgetwinsize' + def test_tcsetwinsize(self): + size = termios.tcgetwinsize(self.fd) + termios.tcsetwinsize(self.fd, size) + termios.tcsetwinsize(self.fd, list(size)) + termios.tcsetwinsize(self.stream, size) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'termios' has no attribute 'tcgetwinsize' + def test_tcsetwinsize_errors(self): + size = termios.tcgetwinsize(self.fd) + self.assertRaises(TypeError, termios.tcsetwinsize, self.fd, size[:-1]) + self.assertRaises(TypeError, termios.tcsetwinsize, self.fd, size + (0,)) + self.assertRaises(TypeError, termios.tcsetwinsize, self.fd, object()) + self.assertRaises(OverflowError, termios.tcsetwinsize, self.fd, (size[0], 2**1000)) + self.assertRaises(TypeError, termios.tcsetwinsize, self.fd, (size[0], float(size[1]))) + self.assertRaises(TypeError, termios.tcsetwinsize, self.fd, (size[0], object())) + self.assertRaises(OverflowError, termios.tcsetwinsize, self.fd, (2**1000, size[1])) + self.assertRaises(TypeError, termios.tcsetwinsize, self.fd, (float(size[0]), size[1])) + self.assertRaises(TypeError, termios.tcsetwinsize, self.fd, (object(), size[1])) + self.assertRaisesTermiosError(errno.ENOTTY, termios.tcsetwinsize, self.bad_fd, size) + self.assertRaises(ValueError, termios.tcsetwinsize, -1, size) + self.assertRaises(OverflowError, termios.tcsetwinsize, 2**1000, size) + self.assertRaises(TypeError, termios.tcsetwinsize, object(), size) + self.assertRaises(TypeError, termios.tcsetwinsize, self.fd) + + +class TestModule(unittest.TestCase): + def test_constants(self): + self.assertIsInstance(termios.B0, int) + self.assertIsInstance(termios.B38400, int) + self.assertIsInstance(termios.TCSANOW, int) + self.assertIsInstance(termios.TCSADRAIN, int) + self.assertIsInstance(termios.TCSAFLUSH, int) + self.assertIsInstance(termios.TCIFLUSH, int) + self.assertIsInstance(termios.TCOFLUSH, int) + self.assertIsInstance(termios.TCIOFLUSH, int) + self.assertIsInstance(termios.TCOOFF, int) + self.assertIsInstance(termios.TCOON, int) + self.assertIsInstance(termios.TCIOFF, int) + self.assertIsInstance(termios.TCION, int) + self.assertIsInstance(termios.VTIME, int) + self.assertIsInstance(termios.VMIN, int) + self.assertIsInstance(termios.NCCS, int) + self.assertLess(termios.VTIME, termios.NCCS) + self.assertLess(termios.VMIN, termios.NCCS) + + def test_ioctl_constants(self): + # gh-119770: ioctl() constants must be positive + for name in dir(termios): + if not name.startswith('TIO'): + continue + value = getattr(termios, name) + with self.subTest(name=name): + self.assertGreaterEqual(value, 0) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: <class 'termios.error'> is a subclass of <class 'OSError'> + def test_exception(self): + self.assertIsSubclass(termios.error, Exception) + self.assertNotIsSubclass(termios.error, OSError) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_textwrap.py b/Lib/test/test_textwrap.py index dfbc2b93dfc..aca1f427656 100644 --- a/Lib/test/test_textwrap.py +++ b/Lib/test/test_textwrap.py @@ -605,7 +605,7 @@ def test_break_long(self): # bug 1146. Prevent a long word to be wrongly wrapped when the # preceding word is exactly one character shorter than the width self.check_wrap(self.text, 12, - ['Did you say ', + ['Did you say', '"supercalifr', 'agilisticexp', 'ialidocious?', @@ -633,7 +633,7 @@ def test_nobreak_long(self): def test_max_lines_long(self): self.check_wrap(self.text, 12, - ['Did you say ', + ['Did you say', '"supercalifr', 'agilisticexp', '[...]'], @@ -765,10 +765,67 @@ def test_subsequent_indent(self): # of IndentTestCase! class DedentTestCase(unittest.TestCase): + def test_type_error(self): + with self.assertRaisesRegex(TypeError, "expected str object, not"): + dedent(0) + + with self.assertRaisesRegex(TypeError, "expected str object, not"): + dedent(b'') + def assertUnchanged(self, text): """assert that dedent() has no effect on 'text'""" self.assertEqual(text, dedent(text)) + def test_dedent_whitespace(self): + # The empty string. + text = "" + self.assertUnchanged(text) + + # Only spaces. + text = " " + expect = "" + self.assertEqual(expect, dedent(text)) + + # Only tabs. + text = "\t\t\t\t" + expect = "" + self.assertEqual(expect, dedent(text)) + + # A mixture. + text = " \t \t\t \t " + expect = "" + self.assertEqual(expect, dedent(text)) + + # ASCII whitespace. + text = "\f\n\r\t\v " + expect = "\n" + self.assertEqual(expect, dedent(text)) + + # One newline. + text = "\n" + expect = "\n" + self.assertEqual(expect, dedent(text)) + + # Windows-style newlines. + text = "\r\n" * 5 + expect = "\n" * 5 + self.assertEqual(expect, dedent(text)) + + # Whitespace mixture. + text = " \n\t\n \n\t\t\n\n\n " + expect = "\n\n\n\n\n\n" + self.assertEqual(expect, dedent(text)) + + # Lines consisting only of whitespace are always normalised + text = "a\n \n\t\n" + expect = "a\n\n\n" + self.assertEqual(expect, dedent(text)) + + # Whitespace characters on non-empty lines are retained + text = "a\r\n\r\n\r\n" + expect = "a\r\n\n\n" + self.assertEqual(expect, dedent(text)) + def test_dedent_nomargin(self): # No lines indented. text = "Hello there.\nHow are you?\nOh good, I'm glad." diff --git a/Lib/test/test_thread.py b/Lib/test/test_thread.py index f55cf3656ea..4ae8a833b99 100644 --- a/Lib/test/test_thread.py +++ b/Lib/test/test_thread.py @@ -105,7 +105,6 @@ def test_nt_and_posix_stack_size(self): thread.stack_size(0) - @unittest.skip("TODO: RUSTPYTHON, weakref destructors") def test__count(self): # Test the _count() function. orig = thread._count() diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py index 937f5903b5f..e635a6f9734 100644 --- a/Lib/test/test_threading.py +++ b/Lib/test/test_threading.py @@ -3,10 +3,11 @@ """ import test.support -from test.support import threading_helper, requires_subprocess +from test.support import threading_helper, requires_subprocess, requires_gil_enabled from test.support import verbose, cpython_only, os_helper from test.support.import_helper import import_module from test.support.script_helper import assert_python_ok, assert_python_failure +from test.support import force_not_colorized import random import sys @@ -20,11 +21,17 @@ import signal import textwrap import traceback +import warnings from unittest import mock from test import lock_tests from test import support +try: + from test.support import interpreters +except ImportError: + interpreters = None + threading_helper.requires_working_threading(module=True) # Between fork() and exec(), only async-safe functions are allowed (issues @@ -34,6 +41,24 @@ platforms_to_skip = ('netbsd5', 'hp-ux11') +def skip_unless_reliable_fork(test): + if not support.has_fork_support: + return unittest.skip("requires working os.fork()")(test) + if sys.platform in platforms_to_skip: + return unittest.skip("due to known OS bug related to thread+fork")(test) + if support.HAVE_ASAN_FORK_BUG: + return unittest.skip("libasan has a pthread_create() dead lock related to thread+fork")(test) + if support.check_sanitizer(thread=True): + return unittest.skip("TSAN doesn't support threads after fork")(test) + return test + + +def requires_subinterpreters(meth): + """Decorator to skip a test if subinterpreters are not supported.""" + return unittest.skipIf(interpreters is None, + 'subinterpreters required')(meth) + + def restore_default_excepthook(testcase): testcase.addCleanup(setattr, threading, 'excepthook', threading.excepthook) threading.excepthook = threading.__excepthook__ @@ -93,6 +118,7 @@ def tearDown(self): class ThreadTests(BaseTestCase): + maxDiff = 9999 @cpython_only def test_name(self): @@ -148,11 +174,21 @@ def test_args_argument(self): t.start() t.join() - @cpython_only - def test_disallow_instantiation(self): - # Ensure that the type disallows instantiation (bpo-43916) - lock = threading.Lock() - test.support.check_disallow_instantiation(self, type(lock)) + def test_lock_no_args(self): + threading.Lock() # works + self.assertRaises(TypeError, threading.Lock, 1) + self.assertRaises(TypeError, threading.Lock, a=1) + self.assertRaises(TypeError, threading.Lock, 1, 2, a=1, b=2) + + def test_lock_no_subclass(self): + # Intentionally disallow subclasses of threading.Lock because they have + # never been allowed, so why start now just because the type is public? + with self.assertRaises(TypeError): + class MyLock(threading.Lock): pass + + def test_lock_or_none(self): + import types + self.assertIsInstance(threading.Lock | None, types.UnionType) # Create a bunch of threads, let each do some work, wait until all are # done. @@ -204,8 +240,6 @@ def f(): tid = _thread.start_new_thread(f, ()) done.wait() self.assertEqual(ident[0], tid) - # Kill the "immortal" _DummyThread - del threading._active[ident[0]] # run with a small(ish) thread stack size (256 KiB) def test_various_ops_small_stack(self): @@ -233,11 +267,29 @@ def test_various_ops_large_stack(self): def test_foreign_thread(self): # Check that a "foreign" thread can use the threading module. + dummy_thread = None + error = None def f(mutex): - # Calling current_thread() forces an entry for the foreign - # thread to get made in the threading._active map. - threading.current_thread() - mutex.release() + try: + nonlocal dummy_thread + nonlocal error + # Calling current_thread() forces an entry for the foreign + # thread to get made in the threading._active map. + dummy_thread = threading.current_thread() + tid = dummy_thread.ident + self.assertIn(tid, threading._active) + self.assertIsInstance(dummy_thread, threading._DummyThread) + self.assertIs(threading._active.get(tid), dummy_thread) + # gh-29376 + self.assertTrue( + dummy_thread.is_alive(), + 'Expected _DummyThread to be considered alive.' + ) + self.assertIn('_DummyThread', repr(dummy_thread)) + except BaseException as e: + error = e + finally: + mutex.release() mutex = threading.Lock() mutex.acquire() @@ -245,16 +297,29 @@ def f(mutex): tid = _thread.start_new_thread(f, (mutex,)) # Wait for the thread to finish. mutex.acquire() - self.assertIn(tid, threading._active) - self.assertIsInstance(threading._active[tid], threading._DummyThread) - #Issue 29376 - self.assertTrue(threading._active[tid].is_alive()) - self.assertRegex(repr(threading._active[tid]), '_DummyThread') - del threading._active[tid] + if error is not None: + raise error + self.assertEqual(tid, dummy_thread.ident) + # Issue gh-106236: + with self.assertRaises(RuntimeError): + dummy_thread.join() + dummy_thread._started.clear() + with self.assertRaises(RuntimeError): + dummy_thread.is_alive() + # Busy wait for the following condition: after the thread dies, the + # related dummy thread must be removed from threading._active. + timeout = 5 + timeout_at = time.monotonic() + timeout + while time.monotonic() < timeout_at: + if threading._active.get(dummy_thread.ident) is not dummy_thread: + break + time.sleep(.1) + else: + self.fail('It was expected that the created threading._DummyThread was removed from threading._active.') # PyThreadState_SetAsyncExc() is a CPython-only gimmick, not (currently) # exposed at the Python level. This test relies on ctypes to get at it. - @unittest.skip("TODO: RUSTPYTHON; expects @cpython_only") + @cpython_only def test_PyThreadState_SetAsyncExc(self): ctypes = import_module("ctypes") @@ -343,12 +408,13 @@ def run(self): t.join() # else the thread is still running, and we have no way to kill it + @unittest.skip('TODO: RUSTPYTHON; threading._start_new_thread not exposed') def test_limbo_cleanup(self): # Issue 7481: Failure to start thread should cleanup the limbo map. - def fail_new_thread(*args): + def fail_new_thread(*args, **kwargs): raise threading.ThreadError() - _start_new_thread = threading._start_new_thread - threading._start_new_thread = fail_new_thread + _start_joinable_thread = threading._start_joinable_thread + threading._start_joinable_thread = fail_new_thread try: t = threading.Thread(target=lambda: None) self.assertRaises(threading.ThreadError, t.start) @@ -356,13 +422,17 @@ def fail_new_thread(*args): t in threading._limbo, "Failed to cleanup _limbo map on failure of Thread.start().") finally: - threading._start_new_thread = _start_new_thread + threading._start_joinable_thread = _start_joinable_thread - @unittest.skip("TODO: RUSTPYTHON; ctypes.pythonapi is not supported") + @unittest.expectedFailure # TODO: RUSTPYTHON; ctypes.pythonapi is not supported def test_finalize_running_thread(self): # Issue 1402: the PyGILState_Ensure / _Release functions may be called # very late on python exit: on deallocation of a running thread for # example. + if support.check_sanitizer(thread=True): + # the thread running `time.sleep(100)` below will still be alive + # at process exit + self.skipTest("TSAN would report thread leak") import_module("ctypes") rc, out, err = assert_python_failure("-c", """if 1: @@ -395,6 +465,11 @@ def waitingThread(): def test_finalize_with_trace(self): # Issue1733757 # Avoid a deadlock when sys.settrace steps into threading._shutdown + if support.check_sanitizer(thread=True): + # the thread running `time.sleep(2)` below will still be alive + # at process exit + self.skipTest("TSAN would report thread leak") + assert_python_ok("-c", """if 1: import sys, threading @@ -417,8 +492,7 @@ def func(frame, event, arg): sys.settrace(func) """) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_join_nondaemon_on_shutdown(self): # Issue 1722344 # Raising SystemExit skipped threading._shutdown @@ -446,7 +520,7 @@ def test_enumerate_after_join(self): old_interval = sys.getswitchinterval() try: for i in range(1, 100): - sys.setswitchinterval(i * 0.0002) + support.setswitchinterval(i * 0.0002) t = threading.Thread(target=lambda: None) t.start() t.join() @@ -456,6 +530,48 @@ def test_enumerate_after_join(self): finally: sys.setswitchinterval(old_interval) + @support.bigmemtest(size=20, memuse=72*2**20, dry_run=False) + def test_join_from_multiple_threads(self, size): + # Thread.join() should be thread-safe + errors = [] + + def worker(): + time.sleep(0.005) + + def joiner(thread): + try: + thread.join() + except Exception as e: + errors.append(e) + + for N in range(2, 20): + threads = [threading.Thread(target=worker)] + for i in range(N): + threads.append(threading.Thread(target=joiner, + args=(threads[0],))) + for t in threads: + t.start() + time.sleep(0.01) + for t in threads: + t.join() + if errors: + raise errors[0] + + def test_join_with_timeout(self): + lock = _thread.allocate_lock() + lock.acquire() + + def worker(): + lock.acquire() + + thread = threading.Thread(target=worker) + thread.start() + thread.join(timeout=0.01) + assert thread.is_alive() + lock.release() + thread.join() + assert not thread.is_alive() + def test_no_refcycle_through_target(self): class RunSelfFunction(object): def __init__(self, should_raise): @@ -534,40 +650,12 @@ def test_daemon_param(self): t = threading.Thread(daemon=True) self.assertTrue(t.daemon) - @support.requires_fork() - def test_fork_at_exit(self): - # bpo-42350: Calling os.fork() after threading._shutdown() must - # not log an error. - code = textwrap.dedent(""" - import atexit - import os - import sys - from test.support import wait_process - - # Import the threading module to register its "at fork" callback - import threading - - def exit_handler(): - pid = os.fork() - if not pid: - print("child process ok", file=sys.stderr, flush=True) - # child process - else: - wait_process(pid, exitcode=0) - - # exit_handler() will be called after threading._shutdown() - atexit.register(exit_handler) - """) - _, out, err = assert_python_ok("-c", code) - self.assertEqual(out, b'') - self.assertEqual(err.rstrip(), b'child process ok') - - @support.requires_fork() + @skip_unless_reliable_fork def test_dummy_thread_after_fork(self): # Issue #14308: a dummy thread in the active list doesn't mess up # the after-fork mechanism. code = """if 1: - import _thread, threading, os, time + import _thread, threading, os, time, warnings def background_thread(evt): # Creates and registers the _DummyThread instance @@ -579,17 +667,23 @@ def background_thread(evt): _thread.start_new_thread(background_thread, (evt,)) evt.wait() assert threading.active_count() == 2, threading.active_count() - if os.fork() == 0: - assert threading.active_count() == 1, threading.active_count() - os._exit(0) - else: - os.wait() + with warnings.catch_warnings(record=True) as ws: + warnings.filterwarnings( + "always", category=DeprecationWarning) + if os.fork() == 0: + assert threading.active_count() == 1, threading.active_count() + os._exit(0) + else: + assert ws[0].category == DeprecationWarning, ws[0] + assert 'fork' in str(ws[0].message), ws[0] + os.wait() """ _, out, err = assert_python_ok("-c", code) self.assertEqual(out, b'') self.assertEqual(err, b'') - @support.requires_fork() + @unittest.skip('TODO: RUSTPYTHON; flaky') + @skip_unless_reliable_fork def test_is_alive_after_fork(self): # Try hard to trigger #18418: is_alive() could sometimes be True on # threads that vanished after a fork. @@ -602,13 +696,15 @@ def test_is_alive_after_fork(self): for i in range(20): t = threading.Thread(target=lambda: None) t.start() - pid = os.fork() - if pid == 0: - os._exit(11 if t.is_alive() else 10) - else: - t.join() + # Ignore the warning about fork with threads. + with warnings.catch_warnings(category=DeprecationWarning, + action="ignore"): + if (pid := os.fork()) == 0: + os._exit(11 if t.is_alive() else 10) + else: + t.join() - support.wait_process(pid, exitcode=10) + support.wait_process(pid, exitcode=10) def test_main_thread(self): main = threading.main_thread() @@ -623,48 +719,59 @@ def f(): th.start() th.join() - @support.requires_fork() + @skip_unless_reliable_fork @unittest.skipUnless(hasattr(os, 'waitpid'), "test needs os.waitpid()") def test_main_thread_after_fork(self): code = """if 1: import os, threading from test import support + ident = threading.get_ident() pid = os.fork() if pid == 0: + print("current ident", threading.get_ident() == ident) main = threading.main_thread() - print(main.name) - print(main.ident == threading.current_thread().ident) - print(main.ident == threading.get_ident()) + print("main", main.name) + print("main ident", main.ident == ident) + print("current is main", threading.current_thread() is main) else: support.wait_process(pid, exitcode=0) """ _, out, err = assert_python_ok("-c", code) data = out.decode().replace('\r', '') self.assertEqual(err, b"") - self.assertEqual(data, "MainThread\nTrue\nTrue\n") + self.assertEqual(data, + "current ident True\n" + "main MainThread\n" + "main ident True\n" + "current is main True\n") - @unittest.skipIf(sys.platform in platforms_to_skip, "due to known OS bug") - @support.requires_fork() + @skip_unless_reliable_fork @unittest.skipUnless(hasattr(os, 'waitpid'), "test needs os.waitpid()") - @unittest.skipIf(os.name != 'posix', "test needs POSIX semantics") def test_main_thread_after_fork_from_nonmain_thread(self): code = """if 1: - import os, threading, sys + import os, threading, sys, warnings from test import support def func(): - pid = os.fork() - if pid == 0: - main = threading.main_thread() - print(main.name) - print(main.ident == threading.current_thread().ident) - print(main.ident == threading.get_ident()) - # stdout is fully buffered because not a tty, - # we have to flush before exit. - sys.stdout.flush() - else: - support.wait_process(pid, exitcode=0) + ident = threading.get_ident() + with warnings.catch_warnings(record=True) as ws: + warnings.filterwarnings( + "always", category=DeprecationWarning) + pid = os.fork() + if pid == 0: + print("current ident", threading.get_ident() == ident) + main = threading.main_thread() + print("main", main.name, type(main).__name__) + print("main ident", main.ident == ident) + print("current is main", threading.current_thread() is main) + # stdout is fully buffered because not a tty, + # we have to flush before exit. + sys.stdout.flush() + else: + assert ws[0].category == DeprecationWarning, ws[0] + assert 'fork' in str(ws[0].message), ws[0] + support.wait_process(pid, exitcode=0) th = threading.Thread(target=func) th.start() @@ -672,11 +779,82 @@ def func(): """ _, out, err = assert_python_ok("-c", code) data = out.decode().replace('\r', '') - self.assertEqual(err, b"") - self.assertEqual(data, "Thread-1 (func)\nTrue\nTrue\n") + self.assertEqual(err.decode('utf-8'), "") + self.assertEqual(data, + "current ident True\n" + "main Thread-1 (func) Thread\n" + "main ident True\n" + "current is main True\n" + ) + + @skip_unless_reliable_fork + @unittest.skipUnless(hasattr(os, 'waitpid'), "test needs os.waitpid()") + def test_main_thread_after_fork_from_foreign_thread(self, create_dummy=False): + code = """if 1: + import os, threading, sys, traceback, _thread + from test import support - # TODO: RUSTPYTHON - @unittest.expectedFailure + def func(lock): + ident = threading.get_ident() + if %s: + # call current_thread() before fork to allocate DummyThread + current = threading.current_thread() + print("current", current.name, type(current).__name__) + print("ident in _active", ident in threading._active) + # flush before fork, so child won't flush it again + sys.stdout.flush() + pid = os.fork() + if pid == 0: + print("current ident", threading.get_ident() == ident) + main = threading.main_thread() + print("main", main.name, type(main).__name__) + print("main ident", main.ident == ident) + print("current is main", threading.current_thread() is main) + print("_dangling", [t.name for t in list(threading._dangling)]) + # stdout is fully buffered because not a tty, + # we have to flush before exit. + sys.stdout.flush() + try: + threading._shutdown() + os._exit(0) + except: + traceback.print_exc() + sys.stderr.flush() + os._exit(1) + else: + try: + support.wait_process(pid, exitcode=0) + except Exception: + # avoid 'could not acquire lock for + # <_io.BufferedWriter name='<stderr>'> at interpreter shutdown,' + traceback.print_exc() + sys.stderr.flush() + finally: + lock.release() + + join_lock = _thread.allocate_lock() + join_lock.acquire() + th = _thread.start_new_thread(func, (join_lock,)) + join_lock.acquire() + """ % create_dummy + # "DeprecationWarning: This process is multi-threaded, use of fork() + # may lead to deadlocks in the child" + _, out, err = assert_python_ok("-W", "ignore::DeprecationWarning", "-c", code) + data = out.decode().replace('\r', '') + self.assertEqual(err.decode(), "") + self.assertEqual(data, + ("current Dummy-1 _DummyThread\n" if create_dummy else "") + + f"ident in _active {create_dummy!s}\n" + + "current ident True\n" + "main MainThread _MainThread\n" + "main ident True\n" + "current is main True\n" + "_dangling ['MainThread']\n") + + def test_main_thread_after_fork_from_dummy_thread(self, create_dummy=False): + self.test_main_thread_after_fork_from_foreign_thread(create_dummy=True) + + @unittest.expectedFailure # TODO: RUSTPYTHON def test_main_thread_during_shutdown(self): # bpo-31516: current_thread() should still point to the main thread # at shutdown @@ -741,41 +919,6 @@ def f(): rc, out, err = assert_python_ok("-c", code) self.assertEqual(err, b"") - def test_tstate_lock(self): - # Test an implementation detail of Thread objects. - started = _thread.allocate_lock() - finish = _thread.allocate_lock() - started.acquire() - finish.acquire() - def f(): - started.release() - finish.acquire() - time.sleep(0.01) - # The tstate lock is None until the thread is started - t = threading.Thread(target=f) - self.assertIs(t._tstate_lock, None) - t.start() - started.acquire() - self.assertTrue(t.is_alive()) - # The tstate lock can't be acquired when the thread is running - # (or suspended). - tstate_lock = t._tstate_lock - self.assertFalse(tstate_lock.acquire(timeout=0), False) - finish.release() - # When the thread ends, the state_lock can be successfully - # acquired. - self.assertTrue(tstate_lock.acquire(timeout=support.SHORT_TIMEOUT), False) - # But is_alive() is still True: we hold _tstate_lock now, which - # prevents is_alive() from knowing the thread's end-of-life C code - # is done. - self.assertTrue(t.is_alive()) - # Let is_alive() find out the C code is done. - tstate_lock.release() - self.assertFalse(t.is_alive()) - # And verify the thread disposed of _tstate_lock. - self.assertIsNone(t._tstate_lock) - t.join() - def test_repr_stopped(self): # Verify that "stopped" shows up in repr(Thread) appropriately. started = _thread.allocate_lock() @@ -823,6 +966,7 @@ def test_BoundedSemaphore_limit(self): @cpython_only def test_frame_tstate_tracing(self): + _testcapi = import_module("_testcapi") # Issue #14432: Crash when a generator is created in a C thread that is # destroyed while the generator is still used. The issue was that a # generator contains a frame, and the frame kept a reference to the @@ -850,7 +994,6 @@ def callback(): threading.settrace(noop_trace) # Create a generator in a C thread which exits after the call - import _testcapi _testcapi.call_in_temporary_c_thread(callback) # Call the generator in a different Python thread, check that the @@ -874,8 +1017,7 @@ def noop_trace(frame, event, arg): finally: threading.settrace(old_trace) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_gettrace_all_threads(self): def fn(*args): pass old_trace = threading.gettrace() @@ -914,8 +1056,7 @@ def fn(*args): pass finally: threading.setprofile(old_profile) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_getprofile_all_threads(self): def fn(*args): pass old_profile = threading.getprofile() @@ -945,32 +1086,7 @@ def checker(): self.assertEqual(threading.getprofile(), old_profile) self.assertEqual(sys.getprofile(), old_profile) - @cpython_only - def test_shutdown_locks(self): - for daemon in (False, True): - with self.subTest(daemon=daemon): - event = threading.Event() - thread = threading.Thread(target=event.wait, daemon=daemon) - - # Thread.start() must add lock to _shutdown_locks, - # but only for non-daemon thread - thread.start() - tstate_lock = thread._tstate_lock - if not daemon: - self.assertIn(tstate_lock, threading._shutdown_locks) - else: - self.assertNotIn(tstate_lock, threading._shutdown_locks) - - # unblock the thread and join it - event.set() - thread.join() - - # Thread._stop() must remove tstate_lock from _shutdown_locks. - # Daemon threads must never add it to _shutdown_locks. - self.assertNotIn(tstate_lock, threading._shutdown_locks) - - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_locals_at_exit(self): # bpo-19466: thread locals must not be deleted before destructors # are called @@ -1048,6 +1164,164 @@ def import_threading(): self.assertEqual(out, b'') self.assertEqual(err, b'') + # TODO: RUSTPYTHON - __del__ not called during interpreter finalization (no cyclic GC) + @unittest.expectedFailure + def test_start_new_thread_at_finalization(self): + code = """if 1: + import _thread + + def f(): + print("shouldn't be printed") + + class AtFinalization: + def __del__(self): + print("OK") + _thread.start_new_thread(f, ()) + at_finalization = AtFinalization() + """ + _, out, err = assert_python_ok("-c", code) + self.assertEqual(out.strip(), b"OK") + self.assertIn(b"can't create new thread at interpreter shutdown", err) + + def test_start_new_thread_failed(self): + # gh-109746: if Python fails to start newly created thread + # due to failure of underlying PyThread_start_new_thread() call, + # its state should be removed from interpreter' thread states list + # to avoid its double cleanup + try: + from resource import setrlimit, RLIMIT_NPROC + except ImportError as err: + self.skipTest(err) # RLIMIT_NPROC is specific to Linux and BSD + code = """if 1: + import resource + import _thread + + def f(): + print("shouldn't be printed") + + limits = resource.getrlimit(resource.RLIMIT_NPROC) + [_, hard] = limits + resource.setrlimit(resource.RLIMIT_NPROC, (0, hard)) + + try: + handle = _thread.start_joinable_thread(f) + except RuntimeError: + print('ok') + else: + print('!skip!') + handle.join() + """ + _, out, err = assert_python_ok("-u", "-c", code) + out = out.strip() + if b'!skip!' in out: + self.skipTest('RLIMIT_NPROC had no effect; probably superuser') + self.assertEqual(out, b'ok') + self.assertEqual(err, b'') + + + @skip_unless_reliable_fork + @unittest.skipUnless(hasattr(threading, 'get_native_id'), "test needs threading.get_native_id()") + def test_native_id_after_fork(self): + script = """if True: + import threading + import os + from test import support + + parent_thread_native_id = threading.current_thread().native_id + print(parent_thread_native_id, flush=True) + assert parent_thread_native_id == threading.get_native_id() + childpid = os.fork() + if childpid == 0: + print(threading.current_thread().native_id, flush=True) + assert threading.current_thread().native_id == threading.get_native_id() + else: + try: + assert parent_thread_native_id == threading.current_thread().native_id + assert parent_thread_native_id == threading.get_native_id() + finally: + support.wait_process(childpid, exitcode=0) + """ + rc, out, err = assert_python_ok('-c', script) + self.assertEqual(rc, 0) + self.assertEqual(err, b"") + native_ids = out.strip().splitlines() + self.assertEqual(len(native_ids), 2) + self.assertNotEqual(native_ids[0], native_ids[1]) + + @cpython_only + def test_finalize_daemon_thread_hang(self): + if support.check_sanitizer(thread=True, memory=True): + # the thread running `time.sleep(100)` below will still be alive + # at process exit + self.skipTest( + "https://github.com/python/cpython/issues/124878 - Known" + " race condition that TSAN identifies.") + # gh-87135: tests that daemon threads hang during finalization + script = textwrap.dedent(''' + import os + import sys + import threading + import time + import _testcapi + + lock = threading.Lock() + lock.acquire() + thread_started_event = threading.Event() + def thread_func(): + try: + thread_started_event.set() + _testcapi.finalize_thread_hang(lock.acquire) + finally: + # Control must not reach here. + os._exit(2) + + t = threading.Thread(target=thread_func) + t.daemon = True + t.start() + thread_started_event.wait() + # Sleep to ensure daemon thread is blocked on `lock.acquire` + # + # Note: This test is designed so that in the unlikely case that + # `0.1` seconds is not sufficient time for the thread to become + # blocked on `lock.acquire`, the test will still pass, it just + # won't be properly testing the thread behavior during + # finalization. + time.sleep(0.1) + + def run_during_finalization(): + # Wake up daemon thread + lock.release() + # Sleep to give the daemon thread time to crash if it is going + # to. + # + # Note: If due to an exceptionally slow execution this delay is + # insufficient, the test will still pass but will simply be + # ineffective as a test. + time.sleep(0.1) + # If control reaches here, the test succeeded. + os._exit(0) + + # Replace sys.stderr.flush as a way to run code during finalization + orig_flush = sys.stderr.flush + def do_flush(*args, **kwargs): + orig_flush(*args, **kwargs) + if not sys.is_finalizing: + return + sys.stderr.flush = orig_flush + run_during_finalization() + + sys.stderr.flush = do_flush + + # If the follow exit code is retained, `run_during_finalization` + # did not run. + sys.exit(1) + ''') + assert_python_ok("-c", script) + + @unittest.skip('TODO: RUSTPYTHON; Thread._tstate_lock not implemented') + def test_tstate_lock(self): + return super().test_tstate_lock() + class ThreadJoinOnShutdown(BaseTestCase): @@ -1068,8 +1342,6 @@ def joiningfunc(mainthread): data = out.decode().replace('\r', '') self.assertEqual(data, "end of main\nend of thread\n") - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_1_join_on_shutdown(self): # The usual case: on exit, wait for a non-daemon thread script = """if 1: @@ -1082,10 +1354,7 @@ def test_1_join_on_shutdown(self): """ self._run_and_join(script) - @support.requires_fork() - @unittest.skipIf(sys.platform in platforms_to_skip, "due to known OS bug") - # TODO: RUSTPYTHON need to fix test_1_join_on_shutdown then this might work - @unittest.expectedFailure + @skip_unless_reliable_fork def test_2_join_in_forked_process(self): # Like the test above, but from a forked interpreter script = """if 1: @@ -1105,9 +1374,8 @@ def test_2_join_in_forked_process(self): """ self._run_and_join(script) - @support.requires_fork() - @unittest.skipIf(sys.platform in platforms_to_skip, "due to known OS bug") - @unittest.skip("TODO: RUSTPYTHON, flaky test") + @unittest.skip('TODO: RUSTPYTHON; flaky test') + @skip_unless_reliable_fork def test_3_join_in_forked_from_thread(self): # Like the test above, but fork() was called from a worker thread # In the forked process, the main Thread object must be marked as stopped. @@ -1136,10 +1404,16 @@ def worker(): self._run_and_join(script) @unittest.skipIf(sys.platform in platforms_to_skip, "due to known OS bug") - def test_4_daemon_threads(self): + @support.bigmemtest(size=40, memuse=70*2**20, dry_run=False) + def test_4_daemon_threads(self, size): # Check that a daemon thread cannot crash the interpreter on shutdown # by manipulating internal structures that are being disposed of in # the main thread. + if support.check_sanitizer(thread=True): + # some of the threads running `random_io` below will still be alive + # at process exit + self.skipTest("TSAN would report thread leak") + script = """if True: import os import random @@ -1177,8 +1451,33 @@ def main(): rc, out, err = assert_python_ok('-c', script) self.assertFalse(err) - @support.requires_fork() - @unittest.skipIf(sys.platform in platforms_to_skip, "due to known OS bug") + def test_thread_from_thread(self): + script = """if True: + import threading + import time + + def thread2(): + time.sleep(0.05) + print("OK") + + def thread1(): + time.sleep(0.05) + t2 = threading.Thread(target=thread2) + t2.start() + + t = threading.Thread(target=thread1) + t.start() + # do not join() -- the interpreter waits for non-daemon threads to + # finish. + """ + rc, out, err = assert_python_ok('-c', script) + self.assertEqual(err, b"") + self.assertEqual(out.strip(), b"OK") + self.assertEqual(rc, 0) + + # TODO: RUSTPYTHON - parking_lot mutex not fork-safe, child may SIGSEGV + @unittest.skip("TODO: RUSTPYTHON - flaky, parking_lot mutex not fork-safe") + @skip_unless_reliable_fork def test_reinit_tls_after_fork(self): # Issue #13817: fork() would deadlock in a multithreaded program with # the ad-hoc TLS implementation. @@ -1191,19 +1490,20 @@ def do_fork_and_wait(): else: os._exit(50) - # start a bunch of threads that will fork() child processes - threads = [] - for i in range(16): - t = threading.Thread(target=do_fork_and_wait) - threads.append(t) - t.start() + # Ignore the warning about fork with threads. + with warnings.catch_warnings(category=DeprecationWarning, + action="ignore"): + # start a bunch of threads that will fork() child processes + threads = [] + for i in range(16): + t = threading.Thread(target=do_fork_and_wait) + threads.append(t) + t.start() - for t in threads: - t.join() + for t in threads: + t.join() - @support.requires_fork() - # TODO: RUSTPYTHON - @unittest.expectedFailure + @skip_unless_reliable_fork def test_clear_threads_states_after_fork(self): # Issue #17094: check that threads states are cleared after fork() @@ -1214,18 +1514,22 @@ def test_clear_threads_states_after_fork(self): threads.append(t) t.start() - pid = os.fork() - if pid == 0: - # check that threads states have been cleared - if len(sys._current_frames()) == 1: - os._exit(51) - else: - os._exit(52) - else: - support.wait_process(pid, exitcode=51) - - for t in threads: - t.join() + try: + # Ignore the warning about fork with threads. + with warnings.catch_warnings(category=DeprecationWarning, + action="ignore"): + pid = os.fork() + if pid == 0: + # check that threads states have been cleared + if len(sys._current_frames()) == 1: + os._exit(51) + else: + os._exit(52) + else: + support.wait_process(pid, exitcode=51) + finally: + for t in threads: + t.join() class SubinterpThreadingTests(BaseTestCase): @@ -1237,8 +1541,7 @@ def pipe(self): os.set_blocking(r, False) return (r, w) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_threads_join(self): # Non-daemon threads should be joined at subinterpreter shutdown # (issue #18808) @@ -1267,8 +1570,7 @@ def f(): # The thread was joined properly. self.assertEqual(os.read(r, 1), b"x") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_threads_join_2(self): # Same as above, but a delay gets introduced after the thread's # Python code returned but before the thread state is deleted. @@ -1306,8 +1608,47 @@ def f(): # The thread was joined properly. self.assertEqual(os.read(r, 1), b"x") + @requires_subinterpreters + def test_threads_join_with_no_main(self): + r_interp, w_interp = self.pipe() + + INTERP = b'I' + FINI = b'F' + DONE = b'D' + + interp = interpreters.create() + interp.exec(f"""if True: + import os + import threading + import time + + done = False + + def notify_fini(): + global done + done = True + os.write({w_interp}, {FINI!r}) + t.join() + threading._register_atexit(notify_fini) + + def task(): + while not done: + time.sleep(0.1) + os.write({w_interp}, {DONE!r}) + t = threading.Thread(target=task) + t.start() + + os.write({w_interp}, {INTERP!r}) + """) + interp.close() + + self.assertEqual(os.read(r_interp, 1), INTERP) + self.assertEqual(os.read(r_interp, 1), FINI) + self.assertEqual(os.read(r_interp, 1), DONE) + @cpython_only def test_daemon_threads_fatal_error(self): + import_module("_testcapi") subinterp_code = f"""if 1: import os import threading @@ -1329,6 +1670,67 @@ def f(): self.assertIn("Fatal Python error: Py_EndInterpreter: " "not the last thread", err.decode()) + def _check_allowed(self, before_start='', *, + allowed=True, + daemon_allowed=True, + daemon=False, + ): + import_module("_testinternalcapi") + subinterp_code = textwrap.dedent(f""" + import test.support + import threading + def func(): + print('this should not have run!') + t = threading.Thread(target=func, daemon={daemon}) + {before_start} + t.start() + """) + check_multi_interp_extensions = bool(support.Py_GIL_DISABLED) + script = textwrap.dedent(f""" + import test.support + test.support.run_in_subinterp_with_config( + {subinterp_code!r}, + use_main_obmalloc=True, + allow_fork=True, + allow_exec=True, + allow_threads={allowed}, + allow_daemon_threads={daemon_allowed}, + check_multi_interp_extensions={check_multi_interp_extensions}, + own_gil=False, + ) + """) + with test.support.SuppressCrashReport(): + _, _, err = assert_python_ok("-c", script) + return err.decode() + + @cpython_only + def test_threads_not_allowed(self): + err = self._check_allowed( + allowed=False, + daemon_allowed=False, + daemon=False, + ) + self.assertIn('RuntimeError', err) + + @cpython_only + def test_daemon_threads_not_allowed(self): + with self.subTest('via Thread()'): + err = self._check_allowed( + allowed=True, + daemon_allowed=False, + daemon=True, + ) + self.assertIn('RuntimeError', err) + + with self.subTest('via Thread.daemon setter'): + err = self._check_allowed( + 't.daemon = True', + allowed=True, + daemon_allowed=False, + daemon=False, + ) + self.assertIn('RuntimeError', err) + class ThreadingExceptionTests(BaseTestCase): # A RuntimeError should be raised if Thread.start() is called @@ -1357,7 +1759,7 @@ def test_releasing_unacquired_lock(self): lock = threading.Lock() self.assertRaises(RuntimeError, lock.release) - @unittest.skip("TODO: RUSTPYTHON, flaky test") + @unittest.skip('TODO: RUSTPYTHON; flaky test') @requires_subprocess() def test_recursion_limit(self): # Issue 9670 @@ -1367,13 +1769,6 @@ def test_recursion_limit(self): # for threads script = """if True: import threading - # TODO: RUSTPYTHON - # Following lines set the recursion limit to previous default of 512 - # for the execution of this process. Without this, the test runners - # on Github fail. Ideally, at a future point this should be removed. - import os, sys - if os.getenv("CI"): - sys.setrecursionlimit(512) def recurse(): return recurse() @@ -1478,6 +1873,37 @@ def run(): self.assertEqual(out, b'') self.assertNotIn("Unhandled exception", err.decode()) + def test_print_exception_gh_102056(self): + # This used to crash. See gh-102056. + script = r"""if True: + import time + import threading + import _thread + + def f(): + try: + f() + except RecursionError: + f() + + def g(): + try: + raise ValueError() + except* ValueError: + f() + + def h(): + time.sleep(1) + _thread.interrupt_main() + + t = threading.Thread(target=h) + t.start() + g() + t.join() + """ + + assert_python_failure("-c", script) + def test_bare_raise_in_brand_new_thread(self): def bare_raise(): raise @@ -1515,6 +1941,23 @@ def modify_file(): t.start() t.join() + def test_dummy_thread_on_interpreter_shutdown(self): + # GH-130522: When `threading` held a reference to itself and then a + # _DummyThread() object was created, destruction of the dummy thread + # would emit an unraisable exception at shutdown, due to a lock being + # destroyed. + code = """if True: + import sys + import threading + + threading.x = sys.modules[__name__] + x = threading._DummyThread() + """ + rc, out, err = assert_python_ok("-c", code) + self.assertEqual(rc, 0) + self.assertEqual(out, b"") + self.assertEqual(err, b"") + class ThreadRunFail(threading.Thread): def run(self): @@ -1526,6 +1969,7 @@ def setUp(self): restore_default_excepthook(self) super().setUp() + @force_not_colorized def test_excepthook(self): with support.captured_output("stderr") as stderr: thread = ThreadRunFail(name="excepthook thread") @@ -1539,6 +1983,7 @@ def test_excepthook(self): self.assertIn('ValueError: run failed', stderr) @support.cpython_only + @force_not_colorized def test_excepthook_thread_None(self): # threading.excepthook called with thread=None: log the thread # identifier in this case. @@ -1674,32 +2119,51 @@ class PyRLockTests(lock_tests.RLockTests): class CRLockTests(lock_tests.RLockTests): locktype = staticmethod(threading._CRLock) - # TODO: RUSTPYTHON - @unittest.skip("TODO: RUSTPYTHON, flaky test") - def test_different_thread(self): - super().test_different_thread() + def test_signature(self): # gh-102029 + with warnings.catch_warnings(record=True) as warnings_log: + threading.RLock() + self.assertEqual(warnings_log, []) - # TODO: RUSTPYTHON - @unittest.expectedFailure + arg_types = [ + ((1,), {}), + ((), {'a': 1}), + ((1, 2), {'a': 1}), + ] + for args, kwargs in arg_types: + with self.subTest(args=args, kwargs=kwargs): + with self.assertWarns(DeprecationWarning): + threading.RLock(*args, **kwargs) + + # Subtypes with custom `__init__` are allowed (but, not recommended): + class CustomRLock(self.locktype): + def __init__(self, a, *, b) -> None: + super().__init__() + + with warnings.catch_warnings(record=True) as warnings_log: + CustomRLock(1, b=2) + self.assertEqual(warnings_log, []) + + @unittest.expectedFailure # TODO: RUSTPYTHON def test_release_save_unacquired(self): - super().test_release_save_unacquired() + return super().test_release_save_unacquired() + + @unittest.skip('TODO: RUSTPYTHON; flaky test') + def test_different_thread(self): + return super().test_different_thread() class EventTests(lock_tests.EventTests): eventtype = staticmethod(threading.Event) - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_reset_internal_locks(): # TODO: RUSTPYTHON; remove this when done - super().test_reset_internal_locks() - class ConditionAsRLockTests(lock_tests.RLockTests): # Condition uses an RLock by default and exports its API. locktype = staticmethod(threading.Condition) - # TODO: RUSTPYTHON - @unittest.skip("TODO: RUSTPYTHON, flaky test") + def test_recursion_count(self): + self.skipTest("Condition does not expose _recursion_count()") + + @unittest.skip('TODO: RUSTPYTHON; flaky test') def test_different_thread(self): - super().test_different_thread() + return super().test_different_thread() class ConditionTests(lock_tests.ConditionTests): condtype = staticmethod(threading.Condition) @@ -1715,8 +2179,6 @@ class BarrierTests(lock_tests.BarrierTests): class MiscTestCase(unittest.TestCase): - # TODO: RUSTPYTHON - @unittest.expectedFailure def test__all__(self): restore_default_excepthook(self) @@ -1750,7 +2212,8 @@ def check_interrupt_main_noerror(self, signum): # Restore original handler signal.signal(signum, handler) - @unittest.skip("TODO: RUSTPYTHON; flaky") + @unittest.skip('TODO: RUSTPYTHON; flaky') + @requires_gil_enabled("gh-118433: Flaky due to a longstanding bug") def test_interrupt_main_subthread(self): # Calling start_new_thread with a function that executes interrupt_main # should raise KeyboardInterrupt upon completion. @@ -1809,8 +2272,6 @@ def worker(started, cont, interrupted): class AtexitTests(unittest.TestCase): - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_atexit_output(self): rc, out, err = assert_python_ok("-c", """if True: import threading @@ -1839,8 +2300,6 @@ def test_atexit_called_once(self): self.assertFalse(err) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_atexit_after_shutdown(self): # The only way to do this is by registering an atexit within # an atexit, which is intended to raise an exception. diff --git a/Lib/test/test_threadsignals.py b/Lib/test/test_threadsignals.py new file mode 100644 index 00000000000..bf241ada90e --- /dev/null +++ b/Lib/test/test_threadsignals.py @@ -0,0 +1,237 @@ +"""PyUnit testing that threads honor our signal semantics""" + +import unittest +import signal +import os +import sys +from test.support import threading_helper +import _thread as thread +import time + +if (sys.platform[:3] == 'win'): + raise unittest.SkipTest("Can't test signal on %s" % sys.platform) + +process_pid = os.getpid() +signalled_all=thread.allocate_lock() + +USING_PTHREAD_COND = (sys.thread_info.name == 'pthread' + and sys.thread_info.lock == 'mutex+cond') + +def registerSignals(for_usr1, for_usr2, for_alrm): + usr1 = signal.signal(signal.SIGUSR1, for_usr1) + usr2 = signal.signal(signal.SIGUSR2, for_usr2) + alrm = signal.signal(signal.SIGALRM, for_alrm) + return usr1, usr2, alrm + + +# The signal handler. Just note that the signal occurred and +# from who. +def handle_signals(sig,frame): + signal_blackboard[sig]['tripped'] += 1 + signal_blackboard[sig]['tripped_by'] = thread.get_ident() + +# a function that will be spawned as a separate thread. +def send_signals(): + # We use `raise_signal` rather than `kill` because: + # * It verifies that a signal delivered to a background thread still has + # its Python-level handler called on the main thread. + # * It ensures the signal is handled before the thread exits. + signal.raise_signal(signal.SIGUSR1) + signal.raise_signal(signal.SIGUSR2) + signalled_all.release() + + +@threading_helper.requires_working_threading() +class ThreadSignals(unittest.TestCase): + + def test_signals(self): + with threading_helper.wait_threads_exit(): + # Test signal handling semantics of threads. + # We spawn a thread, have the thread send itself two signals, and + # wait for it to finish. Check that we got both signals + # and that they were run by the main thread. + signalled_all.acquire() + self.spawnSignallingThread() + signalled_all.acquire() + + self.assertEqual( signal_blackboard[signal.SIGUSR1]['tripped'], 1) + self.assertEqual( signal_blackboard[signal.SIGUSR1]['tripped_by'], + thread.get_ident()) + self.assertEqual( signal_blackboard[signal.SIGUSR2]['tripped'], 1) + self.assertEqual( signal_blackboard[signal.SIGUSR2]['tripped_by'], + thread.get_ident()) + signalled_all.release() + + def spawnSignallingThread(self): + thread.start_new_thread(send_signals, ()) + + def alarm_interrupt(self, sig, frame): + raise KeyboardInterrupt + + @unittest.skipIf(USING_PTHREAD_COND, + 'POSIX condition variables cannot be interrupted') + @unittest.skipIf(sys.platform.startswith('linux') and + not sys.thread_info.version, + 'Issue 34004: musl does not allow interruption of locks ' + 'by signals.') + # Issue #20564: sem_timedwait() cannot be interrupted on OpenBSD + @unittest.skipIf(sys.platform.startswith('openbsd'), + 'lock cannot be interrupted on OpenBSD') + def test_lock_acquire_interruption(self): + # Mimic receiving a SIGINT (KeyboardInterrupt) with SIGALRM while stuck + # in a deadlock. + # XXX this test can fail when the legacy (non-semaphore) implementation + # of locks is used in thread_pthread.h, see issue #11223. + oldalrm = signal.signal(signal.SIGALRM, self.alarm_interrupt) + try: + lock = thread.allocate_lock() + lock.acquire() + signal.alarm(1) + t1 = time.monotonic() + self.assertRaises(KeyboardInterrupt, lock.acquire, timeout=5) + dt = time.monotonic() - t1 + # Checking that KeyboardInterrupt was raised is not sufficient. + # We want to assert that lock.acquire() was interrupted because + # of the signal, not that the signal handler was called immediately + # after timeout return of lock.acquire() (which can fool assertRaises). + self.assertLess(dt, 3.0) + finally: + signal.alarm(0) + signal.signal(signal.SIGALRM, oldalrm) + + @unittest.skipIf(USING_PTHREAD_COND, + 'POSIX condition variables cannot be interrupted') + @unittest.skipIf(sys.platform.startswith('linux') and + not sys.thread_info.version, + 'Issue 34004: musl does not allow interruption of locks ' + 'by signals.') + # Issue #20564: sem_timedwait() cannot be interrupted on OpenBSD + @unittest.skipIf(sys.platform.startswith('openbsd'), + 'lock cannot be interrupted on OpenBSD') + def test_rlock_acquire_interruption(self): + # Mimic receiving a SIGINT (KeyboardInterrupt) with SIGALRM while stuck + # in a deadlock. + # XXX this test can fail when the legacy (non-semaphore) implementation + # of locks is used in thread_pthread.h, see issue #11223. + oldalrm = signal.signal(signal.SIGALRM, self.alarm_interrupt) + try: + rlock = thread.RLock() + # For reentrant locks, the initial acquisition must be in another + # thread. + def other_thread(): + rlock.acquire() + + with threading_helper.wait_threads_exit(): + thread.start_new_thread(other_thread, ()) + # Wait until we can't acquire it without blocking... + while rlock.acquire(blocking=False): + rlock.release() + time.sleep(0.01) + signal.alarm(1) + t1 = time.monotonic() + self.assertRaises(KeyboardInterrupt, rlock.acquire, timeout=5) + dt = time.monotonic() - t1 + # See rationale above in test_lock_acquire_interruption + self.assertLess(dt, 3.0) + finally: + signal.alarm(0) + signal.signal(signal.SIGALRM, oldalrm) + + def acquire_retries_on_intr(self, lock): + self.sig_recvd = False + def my_handler(signal, frame): + self.sig_recvd = True + + old_handler = signal.signal(signal.SIGUSR1, my_handler) + try: + def other_thread(): + # Acquire the lock in a non-main thread, so this test works for + # RLocks. + lock.acquire() + # Wait until the main thread is blocked in the lock acquire, and + # then wake it up with this. + time.sleep(0.5) + os.kill(process_pid, signal.SIGUSR1) + # Let the main thread take the interrupt, handle it, and retry + # the lock acquisition. Then we'll let it run. + time.sleep(0.5) + lock.release() + + with threading_helper.wait_threads_exit(): + thread.start_new_thread(other_thread, ()) + # Wait until we can't acquire it without blocking... + while lock.acquire(blocking=False): + lock.release() + time.sleep(0.01) + result = lock.acquire() # Block while we receive a signal. + self.assertTrue(self.sig_recvd) + self.assertTrue(result) + finally: + signal.signal(signal.SIGUSR1, old_handler) + + def test_lock_acquire_retries_on_intr(self): + self.acquire_retries_on_intr(thread.allocate_lock()) + + def test_rlock_acquire_retries_on_intr(self): + self.acquire_retries_on_intr(thread.RLock()) + + def test_interrupted_timed_acquire(self): + # Test to make sure we recompute lock acquisition timeouts when we + # receive a signal. Check this by repeatedly interrupting a lock + # acquire in the main thread, and make sure that the lock acquire times + # out after the right amount of time. + # NOTE: this test only behaves as expected if C signals get delivered + # to the main thread. Otherwise lock.acquire() itself doesn't get + # interrupted and the test trivially succeeds. + self.start = None + self.end = None + self.sigs_recvd = 0 + done = thread.allocate_lock() + done.acquire() + lock = thread.allocate_lock() + lock.acquire() + def my_handler(signum, frame): + self.sigs_recvd += 1 + old_handler = signal.signal(signal.SIGUSR1, my_handler) + try: + def timed_acquire(): + self.start = time.monotonic() + lock.acquire(timeout=0.5) + self.end = time.monotonic() + def send_signals(): + for _ in range(40): + time.sleep(0.02) + os.kill(process_pid, signal.SIGUSR1) + done.release() + + with threading_helper.wait_threads_exit(): + # Send the signals from the non-main thread, since the main thread + # is the only one that can process signals. + thread.start_new_thread(send_signals, ()) + timed_acquire() + # Wait for thread to finish + done.acquire() + # This allows for some timing and scheduling imprecision + self.assertLess(self.end - self.start, 2.0) + self.assertGreater(self.end - self.start, 0.3) + # If the signal is received several times before PyErr_CheckSignals() + # is called, the handler will get called less than 40 times. Just + # check it's been called at least once. + self.assertGreater(self.sigs_recvd, 0) + finally: + signal.signal(signal.SIGUSR1, old_handler) + + +def setUpModule(): + global signal_blackboard + + signal_blackboard = { signal.SIGUSR1 : {'tripped': 0, 'tripped_by': 0 }, + signal.SIGUSR2 : {'tripped': 0, 'tripped_by': 0 }, + signal.SIGALRM : {'tripped': 0, 'tripped_by': 0 } } + + oldsigs = registerSignals(handle_signals, handle_signals, handle_signals) + unittest.addModuleCleanup(registerSignals, *oldsigs) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py index 31a2a920d9e..35055e03b8e 100644 --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -53,8 +53,6 @@ class TimeTestCase(unittest.TestCase): def setUp(self): self.t = time.time() - # TODO: RUSTPYTHON, AttributeError: module 'time' has no attribute 'altzone' - @unittest.expectedFailure def test_data_attributes(self): time.altzone time.daylight @@ -622,7 +620,6 @@ def test_large_year(self): self.assertEqual(self.yearstr(12345), '12345') self.assertEqual(self.yearstr(123456789), '123456789') -@unittest.skip("TODO: RUSTPYTHON, ValueError: invalid struct_time parameter") class _TestStrftimeYear: # Issue 13305: For years < 1000, the value is not always diff --git a/Lib/test/test_timeit.py b/Lib/test/test_timeit.py index 72a104fc1a6..2aeebea9f93 100644 --- a/Lib/test/test_timeit.py +++ b/Lib/test/test_timeit.py @@ -222,8 +222,8 @@ def test_repeat_function_zero_iters(self): def assert_exc_string(self, exc_string, expected_exc_name): exc_lines = exc_string.splitlines() self.assertGreater(len(exc_lines), 2) - self.assertTrue(exc_lines[0].startswith('Traceback')) - self.assertTrue(exc_lines[-1].startswith(expected_exc_name)) + self.assertStartsWith(exc_lines[0], 'Traceback') + self.assertStartsWith(exc_lines[-1], expected_exc_name) def test_print_exc(self): s = io.StringIO() @@ -297,9 +297,7 @@ def test_main_negative_reps(self): @unittest.skipIf(sys.flags.optimize >= 2, "need __doc__") def test_main_help(self): s = self.run_main(switches=['-h']) - # Note: It's not clear that the trailing space was intended as part of - # the help text, but since it's there, check for it. - self.assertEqual(s, timeit.__doc__ + ' ') + self.assertEqual(s, timeit.__doc__) def test_main_verbose(self): s = self.run_main(switches=['-v']) diff --git a/Lib/test/test_timeout.py b/Lib/test/test_timeout.py index f40c7ee48b0..70a0175d771 100644 --- a/Lib/test/test_timeout.py +++ b/Lib/test/test_timeout.py @@ -71,7 +71,6 @@ def testTypeCheck(self): self.assertRaises(TypeError, self.sock.settimeout, {}) self.assertRaises(TypeError, self.sock.settimeout, 0j) - @unittest.skip("TODO: RUSTPYTHON; crash") def testRangeCheck(self): # Test range checking by settimeout() self.assertRaises(ValueError, self.sock.settimeout, -1) diff --git a/Lib/test/test_tstring.py b/Lib/test/test_tstring.py new file mode 100644 index 00000000000..e91bf3f8b4e --- /dev/null +++ b/Lib/test/test_tstring.py @@ -0,0 +1,296 @@ +import unittest + +from test.test_string._support import TStringBaseCase, fstring + + +class TestTString(unittest.TestCase, TStringBaseCase): + @unittest.expectedFailure # TODO: RUSTPYTHON; + Template(strings=('Hello',), interpolations=()) + def test_string_representation(self): + # Test __repr__ + t = t"Hello" + self.assertEqual(repr(t), "Template(strings=('Hello',), interpolations=())") + + name = "Python" + t = t"Hello, {name}" + self.assertEqual(repr(t), + "Template(strings=('Hello, ', ''), " + "interpolations=(Interpolation('Python', 'name', None, ''),))" + ) + + def test_interpolation_basics(self): + # Test basic interpolation + name = "Python" + t = t"Hello, {name}" + self.assertTStringEqual(t, ("Hello, ", ""), [(name, "name")]) + self.assertEqual(fstring(t), "Hello, Python") + + # Multiple interpolations + first = "Python" + last = "Developer" + t = t"{first} {last}" + self.assertTStringEqual( + t, ("", " ", ""), [(first, 'first'), (last, 'last')] + ) + self.assertEqual(fstring(t), "Python Developer") + + # Interpolation with expressions + a = 10 + b = 20 + t = t"Sum: {a + b}" + self.assertTStringEqual(t, ("Sum: ", ""), [(a + b, "a + b")]) + self.assertEqual(fstring(t), "Sum: 30") + + # Interpolation with function + def square(x): + return x * x + t = t"Square: {square(5)}" + self.assertTStringEqual( + t, ("Square: ", ""), [(square(5), "square(5)")] + ) + self.assertEqual(fstring(t), "Square: 25") + + # Test attribute access in expressions + class Person: + def __init__(self, name): + self.name = name + + def upper(self): + return self.name.upper() + + person = Person("Alice") + t = t"Name: {person.name}" + self.assertTStringEqual( + t, ("Name: ", ""), [(person.name, "person.name")] + ) + self.assertEqual(fstring(t), "Name: Alice") + + # Test method calls + t = t"Name: {person.upper()}" + self.assertTStringEqual( + t, ("Name: ", ""), [(person.upper(), "person.upper()")] + ) + self.assertEqual(fstring(t), "Name: ALICE") + + # Test dictionary access + data = {"name": "Bob", "age": 30} + t = t"Name: {data['name']}, Age: {data['age']}" + self.assertTStringEqual( + t, ("Name: ", ", Age: ", ""), + [(data["name"], "data['name']"), (data["age"], "data['age']")], + ) + self.assertEqual(fstring(t), "Name: Bob, Age: 30") + + def test_format_specifiers(self): + # Test basic format specifiers + value = 3.14159 + t = t"Pi: {value:.2f}" + self.assertTStringEqual( + t, ("Pi: ", ""), [(value, "value", None, ".2f")] + ) + self.assertEqual(fstring(t), "Pi: 3.14") + + def test_conversions(self): + # Test !s conversion (str) + obj = object() + t = t"Object: {obj!s}" + self.assertTStringEqual(t, ("Object: ", ""), [(obj, "obj", "s")]) + self.assertEqual(fstring(t), f"Object: {str(obj)}") + + # Test !r conversion (repr) + t = t"Data: {obj!r}" + self.assertTStringEqual(t, ("Data: ", ""), [(obj, "obj", "r")]) + self.assertEqual(fstring(t), f"Data: {repr(obj)}") + + # Test !a conversion (ascii) + text = "Café" + t = t"ASCII: {text!a}" + self.assertTStringEqual(t, ("ASCII: ", ""), [(text, "text", "a")]) + self.assertEqual(fstring(t), f"ASCII: {ascii(text)}") + + # Test !z conversion (error) + num = 1 + with self.assertRaises(SyntaxError): + eval("t'{num!z}'") + + @unittest.expectedFailure # TODO: RUSTPYTHON; ? ++++++ + def test_debug_specifier(self): + # Test debug specifier + value = 42 + t = t"Value: {value=}" + self.assertTStringEqual( + t, ("Value: value=", ""), [(value, "value", "r")] + ) + self.assertEqual(fstring(t), "Value: value=42") + + # Test debug specifier with format (conversion default to !r) + t = t"Value: {value=:.2f}" + self.assertTStringEqual( + t, ("Value: value=", ""), [(value, "value", None, ".2f")] + ) + self.assertEqual(fstring(t), "Value: value=42.00") + + # Test debug specifier with conversion + t = t"Value: {value=!s}" + self.assertTStringEqual( + t, ("Value: value=", ""), [(value, "value", "s")] + ) + + # Test white space in debug specifier + t = t"Value: {value = }" + self.assertTStringEqual( + t, ("Value: value = ", ""), [(value, "value", "r")] + ) + self.assertEqual(fstring(t), "Value: value = 42") + + def test_raw_tstrings(self): + path = r"C:\Users" + t = rt"{path}\Documents" + self.assertTStringEqual(t, ("", r"\Documents"), [(path, "path")]) + self.assertEqual(fstring(t), r"C:\Users\Documents") + + # Test alternative prefix + t = tr"{path}\Documents" + self.assertTStringEqual(t, ("", r"\Documents"), [(path, "path")]) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: "can only concatenate string.templatelib.Template \(not "str"\) to string.templatelib.Template" does not match "can only concatenate Template (not 'str') to Template" + def test_template_concatenation(self): + # Test template + template + t1 = t"Hello, " + t2 = t"world" + combined = t1 + t2 + self.assertTStringEqual(combined, ("Hello, world",), ()) + self.assertEqual(fstring(combined), "Hello, world") + + # Test template + string + t1 = t"Hello" + expected_msg = 'can only concatenate string.templatelib.Template ' \ + '\\(not "str"\\) to string.templatelib.Template' + with self.assertRaisesRegex(TypeError, expected_msg): + t1 + ", world" + + # Test template + template with interpolation + name = "Python" + t1 = t"Hello, " + t2 = t"{name}" + combined = t1 + t2 + self.assertTStringEqual(combined, ("Hello, ", ""), [(name, "name")]) + self.assertEqual(fstring(combined), "Hello, Python") + + # Test string + template + expected_msg = 'can only concatenate str ' \ + '\\(not "string.templatelib.Template"\\) to str' + with self.assertRaisesRegex(TypeError, expected_msg): + "Hello, " + t"{name}" + + def test_nested_templates(self): + # Test a template inside another template expression + name = "Python" + inner = t"{name}" + t = t"Language: {inner}" + + t_interp = t.interpolations[0] + self.assertEqual(t.strings, ("Language: ", "")) + self.assertEqual(t_interp.value.strings, ("", "")) + self.assertEqual(t_interp.value.interpolations[0].value, name) + self.assertEqual(t_interp.value.interpolations[0].expression, "name") + self.assertEqual(t_interp.value.interpolations[0].conversion, None) + self.assertEqual(t_interp.value.interpolations[0].format_spec, "") + self.assertEqual(t_interp.expression, "inner") + self.assertEqual(t_interp.conversion, None) + self.assertEqual(t_interp.format_spec, "") + + @unittest.expectedFailure # TODO: RUSTPYTHON multiple instances of AssertionError + def test_syntax_errors(self): + for case, err in ( + ("t'", "unterminated t-string literal"), + ("t'''", "unterminated triple-quoted t-string literal"), + ("t''''", "unterminated triple-quoted t-string literal"), + ("t'{", "'{' was never closed"), + ("t'{'", "t-string: expecting '}'"), + ("t'{a'", "t-string: expecting '}'"), + ("t'}'", "t-string: single '}' is not allowed"), + ("t'{}'", "t-string: valid expression required before '}'"), + ("t'{=x}'", "t-string: valid expression required before '='"), + ("t'{!x}'", "t-string: valid expression required before '!'"), + ("t'{:x}'", "t-string: valid expression required before ':'"), + ("t'{x;y}'", "t-string: expecting '=', or '!', or ':', or '}'"), + ("t'{x=y}'", "t-string: expecting '!', or ':', or '}'"), + ("t'{x!s!}'", "t-string: expecting ':' or '}'"), + ("t'{x!s:'", "t-string: expecting '}', or format specs"), + ("t'{x!}'", "t-string: missing conversion character"), + ("t'{x=!}'", "t-string: missing conversion character"), + ("t'{x!z}'", "t-string: invalid conversion character 'z': " + "expected 's', 'r', or 'a'"), + ("t'{lambda:1}'", "t-string: lambda expressions are not allowed " + "without parentheses"), + ("t'{x:{;}}'", "t-string: expecting a valid expression after '{'"), + ("t'{1:d\n}'", "t-string: newlines are not allowed in format specifiers") + ): + with self.subTest(case), self.assertRaisesRegex(SyntaxError, err): + eval(case) + + def test_runtime_errors(self): + # Test missing variables + with self.assertRaises(NameError): + eval("t'Hello, {name}'") + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_literal_concatenation(self): + # Test concatenation of t-string literals + t = t"Hello, " t"world" + self.assertTStringEqual(t, ("Hello, world",), ()) + self.assertEqual(fstring(t), "Hello, world") + + # Test concatenation with interpolation + name = "Python" + t = t"Hello, " t"{name}" + self.assertTStringEqual(t, ("Hello, ", ""), [(name, "name")]) + self.assertEqual(fstring(t), "Hello, Python") + + # Test disallowed mix of t-string and string/f-string (incl. bytes) + what = 't' + expected_msg = 'cannot mix t-string literals with string or bytes literals' + for case in ( + "t'{what}-string literal' 'str literal'", + "t'{what}-string literal' u'unicode literal'", + "t'{what}-string literal' f'f-string literal'", + "t'{what}-string literal' r'raw string literal'", + "t'{what}-string literal' rf'raw f-string literal'", + "t'{what}-string literal' b'bytes literal'", + "t'{what}-string literal' br'raw bytes literal'", + "'str literal' t'{what}-string literal'", + "u'unicode literal' t'{what}-string literal'", + "f'f-string literal' t'{what}-string literal'", + "r'raw string literal' t'{what}-string literal'", + "rf'raw f-string literal' t'{what}-string literal'", + "b'bytes literal' t'{what}-string literal'", + "br'raw bytes literal' t'{what}-string literal'", + ): + with self.subTest(case): + with self.assertRaisesRegex(SyntaxError, expected_msg): + eval(case) + + def test_triple_quoted(self): + # Test triple-quoted t-strings + t = t""" + Hello, + world + """ + self.assertTStringEqual( + t, ("\n Hello,\n world\n ",), () + ) + self.assertEqual(fstring(t), "\n Hello,\n world\n ") + + # Test triple-quoted with interpolation + name = "Python" + t = t""" + Hello, + {name} + """ + self.assertTStringEqual( + t, ("\n Hello,\n ", "\n "), [(name, "name")] + ) + self.assertEqual(fstring(t), "\n Hello,\n Python\n ") + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_type_comments.py b/Lib/test/test_type_comments.py index 578c138767d..4f41171b2df 100644 --- a/Lib/test/test_type_comments.py +++ b/Lib/test/test_type_comments.py @@ -1,7 +1,6 @@ import ast import sys import unittest -from test import support funcdef = """\ @@ -67,6 +66,14 @@ def foo(): pass """ +parenthesized_withstmt = """\ +with (a as b): # type: int + pass + +with (a, b): # type: int + pass +""" + vardecl = """\ a = 0 # type: int """ @@ -244,8 +251,7 @@ def parse_all(self, source, minver=lowest, maxver=highest, expected_regex=""): def classic_parse(self, source): return ast.parse(source) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'FunctionDef' object has no attribute 'type_comment' def test_funcdef(self): for tree in self.parse_all(funcdef): self.assertEqual(tree.body[0].type_comment, "() -> int") @@ -254,8 +260,7 @@ def test_funcdef(self): self.assertEqual(tree.body[0].type_comment, None) self.assertEqual(tree.body[1].type_comment, None) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: SyntaxError not raised : feature_version=(3, 4) def test_asyncdef(self): for tree in self.parse_all(asyncdef, minver=5): self.assertEqual(tree.body[0].type_comment, "() -> int") @@ -264,75 +269,71 @@ def test_asyncdef(self): self.assertEqual(tree.body[0].type_comment, None) self.assertEqual(tree.body[1].type_comment, None) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_asyncvar(self): - for tree in self.parse_all(asyncvar, maxver=6): - pass + with self.assertRaises(SyntaxError): + self.classic_parse(asyncvar) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: SyntaxError not raised : feature_version=(3, 4) def test_asynccomp(self): for tree in self.parse_all(asynccomp, minver=6): pass - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: SyntaxError not raised : feature_version=(3, 4) def test_matmul(self): for tree in self.parse_all(matmul, minver=5): pass - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_fstring(self): - for tree in self.parse_all(fstring, minver=6): + for tree in self.parse_all(fstring): pass - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: SyntaxError not raised : feature_version=(3, 4) def test_underscorednumber(self): for tree in self.parse_all(underscorednumber, minver=6): pass - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: SyntaxError not raised : feature_version=(3, 4) def test_redundantdef(self): for tree in self.parse_all(redundantdef, maxver=0, expected_regex="^Cannot have two type comments on def"): pass - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'FunctionDef' object has no attribute 'type_comment' def test_nonasciidef(self): for tree in self.parse_all(nonasciidef): self.assertEqual(tree.body[0].type_comment, "() -> àçčéñt") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'For' object has no attribute 'type_comment' def test_forstmt(self): for tree in self.parse_all(forstmt): self.assertEqual(tree.body[0].type_comment, "int") tree = self.classic_parse(forstmt) self.assertEqual(tree.body[0].type_comment, None) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'With' object has no attribute 'type_comment' def test_withstmt(self): for tree in self.parse_all(withstmt): self.assertEqual(tree.body[0].type_comment, "int") tree = self.classic_parse(withstmt) self.assertEqual(tree.body[0].type_comment, None) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'With' object has no attribute 'type_comment' + def test_parenthesized_withstmt(self): + for tree in self.parse_all(parenthesized_withstmt): + self.assertEqual(tree.body[0].type_comment, "int") + self.assertEqual(tree.body[1].type_comment, "int") + tree = self.classic_parse(parenthesized_withstmt) + self.assertEqual(tree.body[0].type_comment, None) + self.assertEqual(tree.body[1].type_comment, None) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: None != 'int' def test_vardecl(self): for tree in self.parse_all(vardecl): self.assertEqual(tree.body[0].type_comment, "int") tree = self.classic_parse(vardecl) self.assertEqual(tree.body[0].type_comment, None) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; + (11, ' whatever')] def test_ignores(self): for tree in self.parse_all(ignores): self.assertEqual( @@ -348,16 +349,15 @@ def test_ignores(self): tree = self.classic_parse(ignores) self.assertEqual(tree.type_ignores, []) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: SyntaxError not raised : feature_version=(3, 4) def test_longargs(self): - for tree in self.parse_all(longargs): + for tree in self.parse_all(longargs, minver=8): for t in tree.body: # The expected args are encoded in the function name todo = set(t.name[1:]) self.assertEqual(len(t.args.args) + len(t.args.posonlyargs), len(todo) - bool(t.args.vararg) - bool(t.args.kwarg)) - self.assertTrue(t.name.startswith('f'), t.name) + self.assertStartsWith(t.name, 'f') for index, c in enumerate(t.name[1:]): todo.remove(c) if c == 'v': @@ -380,8 +380,7 @@ def test_longargs(self): self.assertIsNone(arg.type_comment, "%s(%s:%r)" % (t.name, arg.arg, arg.type_comment)) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; Tests for inappropriately-placed type comments. def test_inappropriate_type_comments(self): """Tests for inappropriately-placed type comments. @@ -406,8 +405,7 @@ def check_both_ways(source): check_both_ways("pass # type: ignorewhatever\n") check_both_ways("pass # type: ignoreé\n") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: mode must be "exec", "eval", "ipython", or "single" def test_func_type_input(self): def parse_func_type_input(source): diff --git a/Lib/test/test_types.py b/Lib/test/test_types.py index b57fdf35fb6..06099b87427 100644 --- a/Lib/test/test_types.py +++ b/Lib/test/test_types.py @@ -1,19 +1,35 @@ # Python test set -- part 6, built-in types -from test.support import run_with_locale, cpython_only +from test.support import ( + run_with_locale, cpython_only, no_rerun, + MISSING_C_DOCSTRINGS, EqualToForwardRef, check_disallow_instantiation, +) +from test.support.script_helper import assert_python_ok +from test.support.import_helper import import_fresh_module + import collections.abc -from collections import namedtuple +from collections import namedtuple, UserDict import copy +# XXX: RUSTPYTHON +try: + import _datetime +except ImportError: + _datetime = None import gc import inspect import pickle import locale import sys +import textwrap import types import unittest.mock import weakref import typing +import unittest # XXX: RUSTPYTHON; importing to be able to skip tests + +c_types = import_fresh_module('types', fresh=['_types']) +py_types = import_fresh_module('types', blocked=['_types']) T = typing.TypeVar("T") @@ -29,6 +45,29 @@ def clear_typing_caches(): class TypesTests(unittest.TestCase): + @unittest.skipUnless(c_types, "TODO: RUSTPYTHON; requires _types module") + def test_names(self): + c_only_names = {'CapsuleType'} + ignored = {'new_class', 'resolve_bases', 'prepare_class', + 'get_original_bases', 'DynamicClassAttribute', 'coroutine'} + + for name in c_types.__all__: + if name not in c_only_names | ignored: + self.assertIs(getattr(c_types, name), getattr(py_types, name)) + + all_names = ignored | { + 'AsyncGeneratorType', 'BuiltinFunctionType', 'BuiltinMethodType', + 'CapsuleType', 'CellType', 'ClassMethodDescriptorType', 'CodeType', + 'CoroutineType', 'EllipsisType', 'FrameType', 'FunctionType', + 'GeneratorType', 'GenericAlias', 'GetSetDescriptorType', + 'LambdaType', 'MappingProxyType', 'MemberDescriptorType', + 'MethodDescriptorType', 'MethodType', 'MethodWrapperType', + 'ModuleType', 'NoneType', 'NotImplementedType', 'SimpleNamespace', + 'TracebackType', 'UnionType', 'WrapperDescriptorType', + } + self.assertEqual(all_names, set(c_types.__all__)) + self.assertEqual(all_names - c_only_names, set(py_types.__all__)) + def test_truth_values(self): if None: self.fail('None is true instead of false') if 0: self.fail('0 is true instead of false') @@ -226,8 +265,8 @@ def test_type_function(self): def test_int__format__(self): def test(i, format_spec, result): # just make sure we have the unified type for integers - assert type(i) == int - assert type(format_spec) == str + self.assertIs(type(i), int) + self.assertIs(type(format_spec), str) self.assertEqual(i.__format__(format_spec), result) test(123456789, 'd', '123456789') @@ -392,8 +431,8 @@ def test(i, format_spec, result): test(123456, "1=20", '11111111111111123456') test(123456, "*=20", '**************123456') - @unittest.expectedFailure - @run_with_locale('LC_NUMERIC', 'en_US.UTF8') + @unittest.expectedFailure # TODO: RUSTPYTHON + @run_with_locale('LC_NUMERIC', 'en_US.UTF8', '') def test_float__format__locale(self): # test locale support for __format__ code 'n' @@ -402,7 +441,8 @@ def test_float__format__locale(self): self.assertEqual(locale.format_string('%g', x, grouping=True), format(x, 'n')) self.assertEqual(locale.format_string('%.10g', x, grouping=True), format(x, '.10n')) - @run_with_locale('LC_NUMERIC', 'en_US.UTF8') + @unittest.expectedFailure # TODO: RUSTPYTHON + @run_with_locale('LC_NUMERIC', 'en_US.UTF8', '') def test_int__format__locale(self): # test locale support for __format__ code 'n' for integers @@ -420,9 +460,6 @@ def test_int__format__locale(self): self.assertEqual(len(format(0, rfmt)), len(format(x, rfmt))) self.assertEqual(len(format(0, lfmt)), len(format(x, lfmt))) self.assertEqual(len(format(0, cfmt)), len(format(x, cfmt))) - - if sys.platform != "darwin": - test_int__format__locale = unittest.expectedFailure(test_int__format__locale) def test_float__format__(self): def test(f, format_spec, result): @@ -489,8 +526,8 @@ def test(f, format_spec, result): # and a number after the decimal. This is tricky, because # a totally empty format specifier means something else. # So, just use a sign flag - test(1e200, '+g', '+1e+200') - test(1e200, '+', '+1e+200') + test(1.25e200, '+g', '+1.25e+200') + test(1.25e200, '+', '+1.25e+200') test(1.1e200, '+g', '+1.1e+200') test(1.1e200, '+', '+1.1e+200') @@ -602,8 +639,9 @@ def test_slot_wrapper_types(self): self.assertIsInstance(object.__lt__, types.WrapperDescriptorType) self.assertIsInstance(int.__lt__, types.WrapperDescriptorType) - # TODO: RUSTPYTHON No signature found in builtin method __get__ of 'method_descriptor' objects. - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; No signature found in builtin method __get__ of 'method_descriptor' objects. + @unittest.skipIf(MISSING_C_DOCSTRINGS, + "Signature information for builtins requires docstrings") def test_dunder_get_signature(self): sig = inspect.signature(object.__init__.__get__) self.assertEqual(list(sig.parameters), ["instance", "owner"]) @@ -627,6 +665,26 @@ def test_method_descriptor_types(self): self.assertIsInstance(int.from_bytes, types.BuiltinMethodType) self.assertIsInstance(int.__new__, types.BuiltinMethodType) + @unittest.expectedFailure # TODO: RUSTPYTHON; ModuleNotFoundError: No module named '_queue' + def test_method_descriptor_crash(self): + # gh-132747: The default __get__() implementation in C was unable + # to handle a second argument of None when called from Python + import _io + import io + import _queue + + to_check = [ + # (method, instance) + (_io._TextIOBase.read, io.StringIO()), + (_queue.SimpleQueue.put, _queue.SimpleQueue()), + (str.capitalize, "nobody expects the spanish inquisition") + ] + + for method, instance in to_check: + with self.subTest(method=method, instance=instance): + bound = method.__get__(instance) + self.assertIsInstance(bound, types.BuiltinMethodType) + def test_ellipsis_type(self): self.assertIsInstance(Ellipsis, types.EllipsisType) @@ -644,6 +702,29 @@ def test_traceback_and_frame_types(self): self.assertIsInstance(exc.__traceback__, types.TracebackType) self.assertIsInstance(exc.__traceback__.tb_frame, types.FrameType) + # XXX: RUSTPYTHON + @unittest.skipUnless(_datetime, "requires _datetime module") + def test_capsule_type(self): + self.assertIsInstance(_datetime.datetime_CAPI, types.CapsuleType) + + def test_call_unbound_crash(self): + # GH-131998: The specialized instruction would get tricked into dereferencing + # a bound "self" that didn't exist if subsequently called unbound. + code = """if True: + + def call(part): + [] + ([] + []) + part.pop() + + for _ in range(3): + call(['a']) + try: + call(list) + except TypeError: + pass + """ + assert_python_ok("-c", code) + class UnionTests(unittest.TestCase): @@ -706,15 +787,54 @@ def test_or_types_operator(self): y = int | bool with self.assertRaises(TypeError): x < y - # Check that we don't crash if typing.Union does not have a tuple in __args__ - y = typing.Union[str, int] - y.__args__ = [str, int] - self.assertEqual(x, y) def test_hash(self): self.assertEqual(hash(int | str), hash(str | int)) self.assertEqual(hash(int | str), hash(typing.Union[int, str])) + def test_union_of_unhashable(self): + class UnhashableMeta(type): + __hash__ = None + + class A(metaclass=UnhashableMeta): ... + class B(metaclass=UnhashableMeta): ... + + self.assertEqual((A | B).__args__, (A, B)) + union1 = A | B + with self.assertRaisesRegex(TypeError, "unhashable type: 'UnhashableMeta'"): + hash(union1) + + union2 = int | B + with self.assertRaisesRegex(TypeError, "unhashable type: 'UnhashableMeta'"): + hash(union2) + + union3 = A | int + with self.assertRaisesRegex(TypeError, "unhashable type: 'UnhashableMeta'"): + hash(union3) + + def test_unhashable_becomes_hashable(self): + is_hashable = False + class UnhashableMeta(type): + def __hash__(self): + if is_hashable: + return 1 + else: + raise TypeError("not hashable") + + class A(metaclass=UnhashableMeta): ... + class B(metaclass=UnhashableMeta): ... + + union = A | B + self.assertEqual(union.__args__, (A, B)) + + with self.assertRaisesRegex(TypeError, "not hashable"): + hash(union) + + is_hashable = True + + with self.assertRaisesRegex(TypeError, "union contains 2 unhashable elements"): + hash(union) + def test_instancecheck_and_subclasscheck(self): for x in (int | str, typing.Union[int, str]): with self.subTest(x=x): @@ -722,15 +842,15 @@ def test_instancecheck_and_subclasscheck(self): self.assertIsInstance(True, x) self.assertIsInstance('a', x) self.assertNotIsInstance(None, x) - self.assertTrue(issubclass(int, x)) - self.assertTrue(issubclass(bool, x)) - self.assertTrue(issubclass(str, x)) - self.assertFalse(issubclass(type(None), x)) + self.assertIsSubclass(int, x) + self.assertIsSubclass(bool, x) + self.assertIsSubclass(str, x) + self.assertNotIsSubclass(type(None), x) for x in (int | None, typing.Union[int, None]): with self.subTest(x=x): self.assertIsInstance(None, x) - self.assertTrue(issubclass(type(None), x)) + self.assertIsSubclass(type(None), x) for x in ( int | collections.abc.Mapping, @@ -739,8 +859,8 @@ def test_instancecheck_and_subclasscheck(self): with self.subTest(x=x): self.assertIsInstance({}, x) self.assertNotIsInstance((), x) - self.assertTrue(issubclass(dict, x)) - self.assertFalse(issubclass(list, x)) + self.assertIsSubclass(dict, x) + self.assertNotIsSubclass(list, x) def test_instancecheck_and_subclasscheck_order(self): T = typing.TypeVar('T') @@ -752,7 +872,7 @@ def test_instancecheck_and_subclasscheck_order(self): for x in will_resolve: with self.subTest(x=x): self.assertIsInstance(1, x) - self.assertTrue(issubclass(int, x)) + self.assertIsSubclass(int, x) wont_resolve = ( T | int, @@ -785,13 +905,13 @@ class BadMeta(type): def __subclasscheck__(cls, sub): 1/0 x = int | BadMeta('A', (), {}) - self.assertTrue(issubclass(int, x)) + self.assertIsSubclass(int, x) self.assertRaises(ZeroDivisionError, issubclass, list, x) def test_or_type_operator_with_TypeVar(self): TV = typing.TypeVar('T') - assert TV | str == typing.Union[TV, str] - assert str | TV == typing.Union[str, TV] + self.assertEqual(TV | str, typing.Union[TV, str]) + self.assertEqual(str | TV, typing.Union[str, TV]) self.assertIs((int | TV)[int], int) self.assertIs((TV | int)[int], int) @@ -895,54 +1015,83 @@ def test_or_type_operator_with_forward(self): ForwardBefore = 'Forward' | T def forward_after(x: ForwardAfter[int]) -> None: ... def forward_before(x: ForwardBefore[int]) -> None: ... - assert typing.get_args(typing.get_type_hints(forward_after)['x']) == (int, Forward) - assert typing.get_args(typing.get_type_hints(forward_before)['x']) == (int, Forward) + self.assertEqual(typing.get_args(typing.get_type_hints(forward_after)['x']), + (int, Forward)) + self.assertEqual(typing.get_args(typing.get_type_hints(forward_before)['x']), + (Forward, int)) def test_or_type_operator_with_Protocol(self): class Proto(typing.Protocol): def meth(self) -> int: ... - assert Proto | str == typing.Union[Proto, str] + self.assertEqual(Proto | str, typing.Union[Proto, str]) def test_or_type_operator_with_Alias(self): - assert list | str == typing.Union[list, str] - assert typing.List | str == typing.Union[typing.List, str] + self.assertEqual(list | str, typing.Union[list, str]) + self.assertEqual(typing.List | str, typing.Union[typing.List, str]) def test_or_type_operator_with_NamedTuple(self): - NT=namedtuple('A', ['B', 'C', 'D']) - assert NT | str == typing.Union[NT,str] + NT = namedtuple('A', ['B', 'C', 'D']) + self.assertEqual(NT | str, typing.Union[NT, str]) def test_or_type_operator_with_TypedDict(self): class Point2D(typing.TypedDict): x: int y: int label: str - assert Point2D | str == typing.Union[Point2D, str] + self.assertEqual(Point2D | str, typing.Union[Point2D, str]) def test_or_type_operator_with_NewType(self): UserId = typing.NewType('UserId', int) - assert UserId | str == typing.Union[UserId, str] + self.assertEqual(UserId | str, typing.Union[UserId, str]) def test_or_type_operator_with_IO(self): - assert typing.IO | str == typing.Union[typing.IO, str] + self.assertEqual(typing.IO | str, typing.Union[typing.IO, str]) def test_or_type_operator_with_SpecialForm(self): - assert typing.Any | str == typing.Union[typing.Any, str] - assert typing.NoReturn | str == typing.Union[typing.NoReturn, str] - assert typing.Optional[int] | str == typing.Union[typing.Optional[int], str] - assert typing.Optional[int] | str == typing.Union[int, str, None] - assert typing.Union[int, bool] | str == typing.Union[int, bool, str] + self.assertEqual(typing.Any | str, typing.Union[typing.Any, str]) + self.assertEqual(typing.NoReturn | str, typing.Union[typing.NoReturn, str]) + self.assertEqual(typing.Optional[int] | str, typing.Union[typing.Optional[int], str]) + self.assertEqual(typing.Optional[int] | str, typing.Union[int, str, None]) + self.assertEqual(typing.Union[int, bool] | str, typing.Union[int, bool, str]) + + def test_or_type_operator_with_Literal(self): + Literal = typing.Literal + self.assertEqual((Literal[1] | Literal[2]).__args__, + (Literal[1], Literal[2])) + + self.assertEqual((Literal[0] | Literal[False]).__args__, + (Literal[0], Literal[False])) + self.assertEqual((Literal[1] | Literal[True]).__args__, + (Literal[1], Literal[True])) + + self.assertEqual(Literal[1] | Literal[1], Literal[1]) + self.assertEqual(Literal['a'] | Literal['a'], Literal['a']) + + import enum + class Ints(enum.IntEnum): + A = 0 + B = 1 + + self.assertEqual(Literal[Ints.A] | Literal[Ints.A], Literal[Ints.A]) + self.assertEqual(Literal[Ints.B] | Literal[Ints.B], Literal[Ints.B]) + + self.assertEqual((Literal[Ints.B] | Literal[Ints.A]).__args__, + (Literal[Ints.B], Literal[Ints.A])) + + self.assertEqual((Literal[0] | Literal[Ints.A]).__args__, + (Literal[0], Literal[Ints.A])) + self.assertEqual((Literal[1] | Literal[Ints.B]).__args__, + (Literal[1], Literal[Ints.B])) def test_or_type_repr(self): - assert repr(int | str) == "int | str" - assert repr((int | str) | list) == "int | str | list" - assert repr(int | (str | list)) == "int | str | list" - assert repr(int | None) == "int | None" - assert repr(int | type(None)) == "int | None" - assert repr(int | typing.GenericAlias(list, int)) == "int | list[int]" - - # TODO: RUSTPYTHON - @unittest.expectedFailure + self.assertEqual(repr(int | str), "int | str") + self.assertEqual(repr((int | str) | list), "int | str | list") + self.assertEqual(repr(int | (str | list)), "int | str | list") + self.assertEqual(repr(int | None), "int | None") + self.assertEqual(repr(int | type(None)), "int | None") + self.assertEqual(repr(int | typing.GenericAlias(list, int)), "int | list[int]") + def test_or_type_operator_with_genericalias(self): a = list[int] b = list[str] @@ -963,9 +1112,14 @@ def __eq__(self, other): return 1 / 0 bt = BadType('bt', (), {}) + bt2 = BadType('bt2', (), {}) # Comparison should fail and errors should propagate out for bad types. + union1 = int | bt + union2 = int | bt2 + with self.assertRaises(ZeroDivisionError): + union1 == union2 with self.assertRaises(ZeroDivisionError): - list[int] | list[bt] + bt | bt2 union_ga = (list[str] | int, collections.abc.Callable[..., str] | int, d | int) @@ -1008,6 +1162,19 @@ def test_or_type_operator_reference_cycle(self): self.assertLessEqual(sys.gettotalrefcount() - before, leeway, msg='Check for union reference leak.') + def test_instantiation(self): + check_disallow_instantiation(self, types.UnionType) + self.assertIs(int, types.UnionType[int]) + self.assertIs(int, types.UnionType[int, int]) + self.assertEqual(int | str, types.UnionType[int, str]) + + for obj in ( + int | typing.ForwardRef("str"), + typing.Union[int, "str"], + ): + self.assertIsInstance(obj, types.UnionType) + self.assertEqual(obj.__args__, (int, EqualToForwardRef("str"))) + class MappingProxyTests(unittest.TestCase): mappingproxy = types.MappingProxyType @@ -1197,8 +1364,7 @@ def test_copy(self): self.assertEqual(view['key1'], 70) self.assertEqual(copy['key1'], 27) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_union(self): mapping = {'a': 0, 'b': 1, 'c': 2} view = self.mappingproxy(mapping) @@ -1215,6 +1381,16 @@ def test_union(self): self.assertDictEqual(mapping, {'a': 0, 'b': 1, 'c': 2}) self.assertDictEqual(other, {'c': 3, 'p': 0}) + def test_hash(self): + class HashableDict(dict): + def __hash__(self): + return 3844817361 + view = self.mappingproxy({'a': 1, 'b': 2}) + self.assertRaises(TypeError, hash, view) + mapping = HashableDict({'a': 1, 'b': 2}) + view = self.mappingproxy(mapping) + self.assertEqual(hash(view), hash(mapping)) + class ClassCreationTests(unittest.TestCase): @@ -1238,7 +1414,7 @@ def test_new_class_basics(self): def test_new_class_subclass(self): C = types.new_class("C", (int,)) - self.assertTrue(issubclass(C, int)) + self.assertIsSubclass(C, int) def test_new_class_meta(self): Meta = self.Meta @@ -1283,7 +1459,7 @@ def func(ns): bases=(int,), kwds=dict(metaclass=Meta, z=2), exec_body=func) - self.assertTrue(issubclass(C, int)) + self.assertIsSubclass(C, int) self.assertIsInstance(C, Meta) self.assertEqual(C.x, 0) self.assertEqual(C.y, 1) @@ -1362,6 +1538,80 @@ class C: pass D = types.new_class('D', (A(), C, B()), {}) self.assertEqual(D.__bases__, (A1, A2, A3, C, B1, B2)) + def test_get_original_bases(self): + T = typing.TypeVar('T') + class A: pass + class B(typing.Generic[T]): pass + class C(B[int]): pass + class D(B[str], float): pass + + self.assertEqual(types.get_original_bases(A), (object,)) + self.assertEqual(types.get_original_bases(B), (typing.Generic[T],)) + self.assertEqual(types.get_original_bases(C), (B[int],)) + self.assertEqual(types.get_original_bases(int), (object,)) + self.assertEqual(types.get_original_bases(D), (B[str], float)) + + class E(list[T]): pass + class F(list[int]): pass + + self.assertEqual(types.get_original_bases(E), (list[T],)) + self.assertEqual(types.get_original_bases(F), (list[int],)) + + class FirstBase(typing.Generic[T]): pass + class SecondBase(typing.Generic[T]): pass + class First(FirstBase[int]): pass + class Second(SecondBase[int]): pass + class G(First, Second): pass + self.assertEqual(types.get_original_bases(G), (First, Second)) + + class First_(typing.Generic[T]): pass + class Second_(typing.Generic[T]): pass + class H(First_, Second_): pass + self.assertEqual(types.get_original_bases(H), (First_, Second_)) + + class ClassBasedNamedTuple(typing.NamedTuple): + x: int + + class GenericNamedTuple(typing.NamedTuple, typing.Generic[T]): + x: T + + CallBasedNamedTuple = typing.NamedTuple("CallBasedNamedTuple", [("x", int)]) + + self.assertIs( + types.get_original_bases(ClassBasedNamedTuple)[0], typing.NamedTuple + ) + self.assertEqual( + types.get_original_bases(GenericNamedTuple), + (typing.NamedTuple, typing.Generic[T]) + ) + self.assertIs( + types.get_original_bases(CallBasedNamedTuple)[0], typing.NamedTuple + ) + + class ClassBasedTypedDict(typing.TypedDict): + x: int + + class GenericTypedDict(typing.TypedDict, typing.Generic[T]): + x: T + + CallBasedTypedDict = typing.TypedDict("CallBasedTypedDict", {"x": int}) + + self.assertIs( + types.get_original_bases(ClassBasedTypedDict)[0], + typing.TypedDict + ) + self.assertEqual( + types.get_original_bases(GenericTypedDict), + (typing.TypedDict, typing.Generic[T]) + ) + self.assertIs( + types.get_original_bases(CallBasedTypedDict)[0], + typing.TypedDict + ) + + with self.assertRaisesRegex(TypeError, "Expected an instance of type"): + types.get_original_bases(object()) + # Many of the following tests are derived from test_descr.py def test_prepare_class(self): # Basic test of metaclass derivation @@ -1622,25 +1872,81 @@ class Model(metaclass=ModelBase): with self.assertRaises(RuntimeWarning): type("SouthPonies", (Model,), {}) + def test_subclass_inherited_slot_update(self): + # gh-132284: Make sure slot update still works after fix. + # Note that after assignment to D.__getitem__ the actual C slot will + # never go back to dict_subscript as it was on class type creation but + # rather be set to slot_mp_subscript, unfortunately there is no way to + # check that here. + + class D(dict): + pass + + d = D({None: None}) + self.assertIs(d[None], None) + D.__getitem__ = lambda self, item: 42 + self.assertEqual(d[None], 42) + D.__getitem__ = dict.__getitem__ + self.assertIs(d[None], None) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: <class 'tuple'> != <class 'test.test_types.ClassCreationTests.test_tu[41 chars]ass'> + def test_tuple_subclass_as_bases(self): + # gh-132176: it used to crash on using + # tuple subclass for as base classes. + class TupleSubclass(tuple): pass + + typ = type("typ", TupleSubclass((int, object)), {}) + self.assertEqual(typ.__bases__, (int, object)) + self.assertEqual(type(typ.__bases__), TupleSubclass) + class SimpleNamespaceTests(unittest.TestCase): def test_constructor(self): - ns1 = types.SimpleNamespace() - ns2 = types.SimpleNamespace(x=1, y=2) - ns3 = types.SimpleNamespace(**dict(x=1, y=2)) + def check(ns, expected): + self.assertEqual(len(ns.__dict__), len(expected)) + self.assertEqual(vars(ns), expected) + # check order + self.assertEqual(list(vars(ns).items()), list(expected.items())) + for name in expected: + self.assertEqual(getattr(ns, name), expected[name]) + + check(types.SimpleNamespace(), {}) + check(types.SimpleNamespace(x=1, y=2), {'x': 1, 'y': 2}) + check(types.SimpleNamespace(**dict(x=1, y=2)), {'x': 1, 'y': 2}) + check(types.SimpleNamespace({'x': 1, 'y': 2}, x=4, z=3), + {'x': 4, 'y': 2, 'z': 3}) + check(types.SimpleNamespace([['x', 1], ['y', 2]], x=4, z=3), + {'x': 4, 'y': 2, 'z': 3}) + check(types.SimpleNamespace(UserDict({'x': 1, 'y': 2}), x=4, z=3), + {'x': 4, 'y': 2, 'z': 3}) + check(types.SimpleNamespace({'x': 1, 'y': 2}), {'x': 1, 'y': 2}) + check(types.SimpleNamespace([['x', 1], ['y', 2]]), {'x': 1, 'y': 2}) + check(types.SimpleNamespace([], x=4, z=3), {'x': 4, 'z': 3}) + check(types.SimpleNamespace({}, x=4, z=3), {'x': 4, 'z': 3}) + check(types.SimpleNamespace([]), {}) + check(types.SimpleNamespace({}), {}) with self.assertRaises(TypeError): - types.SimpleNamespace(1, 2, 3) + types.SimpleNamespace([], []) # too many positional arguments with self.assertRaises(TypeError): - types.SimpleNamespace(**{1: 2}) - - self.assertEqual(len(ns1.__dict__), 0) - self.assertEqual(vars(ns1), {}) - self.assertEqual(len(ns2.__dict__), 2) - self.assertEqual(vars(ns2), {'y': 2, 'x': 1}) - self.assertEqual(len(ns3.__dict__), 2) - self.assertEqual(vars(ns3), {'y': 2, 'x': 1}) + types.SimpleNamespace(1) # not a mapping or iterable + with self.assertRaises(TypeError): + types.SimpleNamespace([1]) # non-iterable + with self.assertRaises(ValueError): + types.SimpleNamespace([['x']]) # not a pair + with self.assertRaises(ValueError): + types.SimpleNamespace([['x', 'y', 'z']]) + with self.assertRaises(TypeError): + types.SimpleNamespace(**{1: 2}) # non-string key + with self.assertRaises(TypeError): + types.SimpleNamespace({1: 2}) + with self.assertRaises(TypeError): + types.SimpleNamespace([[1, 2]]) + with self.assertRaises(TypeError): + types.SimpleNamespace(UserDict({1: 2})) + with self.assertRaises(TypeError): + types.SimpleNamespace([[[], 2]]) # non-hashable key def test_unbound(self): ns1 = vars(types.SimpleNamespace()) @@ -1797,6 +2103,33 @@ def test_pickle(self): self.assertEqual(ns, ns_roundtrip, pname) + def test_replace(self): + ns = types.SimpleNamespace(x=11, y=22) + + ns2 = copy.replace(ns) + self.assertEqual(ns2, ns) + self.assertIsNot(ns2, ns) + self.assertIs(type(ns2), types.SimpleNamespace) + self.assertEqual(vars(ns2), {'x': 11, 'y': 22}) + ns2.x = 3 + self.assertEqual(ns.x, 11) + ns.x = 4 + self.assertEqual(ns2.x, 3) + + self.assertEqual(vars(copy.replace(ns, x=1)), {'x': 1, 'y': 22}) + self.assertEqual(vars(copy.replace(ns, y=2)), {'x': 4, 'y': 2}) + self.assertEqual(vars(copy.replace(ns, x=1, y=2)), {'x': 1, 'y': 2}) + + def test_replace_subclass(self): + class Spam(types.SimpleNamespace): + pass + + spam = Spam(ham=8, eggs=9) + spam2 = copy.replace(spam, ham=5) + + self.assertIs(type(spam2), Spam) + self.assertEqual(vars(spam2), {'ham': 5, 'eggs': 9}) + def test_fake_namespace_compare(self): # Issue #24257: Incorrect use of PyObject_IsInstance() caused # SystemError. @@ -1841,8 +2174,6 @@ def foo(): foo = types.coroutine(foo) self.assertIs(aw, foo()) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_async_def(self): # Test that types.coroutine passes 'async def' coroutines # without modification @@ -2076,7 +2407,7 @@ def foo(): return gen wrapper = foo() wrapper.send(None) with self.assertRaisesRegex(Exception, 'ham'): - wrapper.throw(Exception, Exception('ham')) + wrapper.throw(Exception('ham')) # decorate foo second time foo = types.coroutine(foo) @@ -2099,8 +2430,6 @@ def foo(): foo = types.coroutine(foo) self.assertIs(foo(), gencoro) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_genfunc(self): def gen(): yield self.assertIs(types.coroutine(gen), gen) @@ -2131,5 +2460,125 @@ def coro(): 'close', 'throw'})) +class FunctionTests(unittest.TestCase): + def test_function_type_defaults(self): + def ex(a, /, b, *, c): + return a + b + c + + func = types.FunctionType( + ex.__code__, {}, "func", (1, 2), None, {'c': 3}, + ) + + self.assertEqual(func(), 6) + self.assertEqual(func.__defaults__, (1, 2)) + self.assertEqual(func.__kwdefaults__, {'c': 3}) + + func = types.FunctionType( + ex.__code__, {}, "func", None, None, None, + ) + self.assertEqual(func.__defaults__, None) + self.assertEqual(func.__kwdefaults__, None) + + def test_function_type_wrong_defaults(self): + def ex(a, /, b, *, c): + return a + b + c + + with self.assertRaisesRegex(TypeError, 'arg 4'): + types.FunctionType( + ex.__code__, {}, "func", 1, None, {'c': 3}, + ) + with self.assertRaisesRegex(TypeError, 'arg 6'): + types.FunctionType( + ex.__code__, {}, "func", None, None, 3, + ) + + +@unittest.skip("TODO: RUSTPYTHON; no subinterpreters yet") +class SubinterpreterTests(unittest.TestCase): + + NUMERIC_METHODS = { + '__abs__', + '__add__', + '__bool__', + '__divmod__', + '__float__', + '__floordiv__', + '__index__', + '__int__', + '__lshift__', + '__mod__', + '__mul__', + '__neg__', + '__pos__', + '__pow__', + '__radd__', + '__rdivmod__', + '__rfloordiv__', + '__rlshift__', + '__rmod__', + '__rmul__', + '__rpow__', + '__rrshift__', + '__rshift__', + '__rsub__', + '__rtruediv__', + '__sub__', + '__truediv__', + } + + @classmethod + def setUpClass(cls): + global interpreters + try: + from concurrent import interpreters + except ModuleNotFoundError: + raise unittest.SkipTest('subinterpreters required') + from test.support import channels # noqa: F401 + cls.create_channel = staticmethod(channels.create) + + @cpython_only + @no_rerun('channels (and queues) might have a refleak; see gh-122199') + def test_static_types_inherited_slots(self): + rch, sch = self.create_channel() + + script = textwrap.dedent(""" + import test.support + results = [] + for cls in test.support.iter_builtin_types(): + for attr, _ in test.support.iter_slot_wrappers(cls): + wrapper = getattr(cls, attr) + res = (cls, attr, wrapper) + results.append(res) + results = tuple((repr(c), a, repr(w)) for c, a, w in results) + sch.send_nowait(results) + """) + def collate_results(raw): + results = {} + for cls, attr, wrapper in raw: + key = cls, attr + assert key not in results, (results, key, wrapper) + results[key] = wrapper + return results + + exec(script) + raw = rch.recv_nowait() + main_results = collate_results(raw) + + interp = interpreters.create() + interp.exec('from concurrent import interpreters') + interp.prepare_main(sch=sch) + interp.exec(script) + raw = rch.recv_nowait() + interp_results = collate_results(raw) + + for key, expected in main_results.items(): + cls, attr = key + with self.subTest(cls=cls, slotattr=attr): + actual = interp_results.pop(key) + self.assertEqual(actual, expected) + self.maxDiff = None + self.assertEqual(interp_results, {}) + + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index 3e6c530cecc..3d101c62e12 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -1,3 +1,4 @@ +import annotationlib import contextlib import collections import collections.abc @@ -5,8 +6,10 @@ from functools import lru_cache, wraps, reduce import gc import inspect +import io import itertools import operator +import os import pickle import re import sys @@ -43,13 +46,19 @@ import textwrap import typing import weakref +import warnings import types -from test.support import captured_stderr, cpython_only, infinite_recursion, requires_docstrings, import_helper -from test.support.testcase import ExtraAssertions -from test.typinganndata import ann_module695, mod_generics_cache, _typed_dict_helper +from test.support import ( + captured_stderr, cpython_only, infinite_recursion, requires_docstrings, import_helper, run_code, + EqualToForwardRef, +) +from test.typinganndata import ( + ann_module695, mod_generics_cache, _typed_dict_helper, + ann_module, ann_module2, ann_module3, ann_module5, ann_module6, ann_module8 +) -import unittest # XXX: RUSTPYTHON +import unittest # XXX: RUSTPYTHON; importing to be able to skip tests CANNOT_SUBCLASS_TYPE = 'Cannot subclass special typing classes' @@ -57,7 +66,7 @@ CANNOT_SUBCLASS_INSTANCE = 'Cannot subclass an instance of %s' -class BaseTestCase(TestCase, ExtraAssertions): +class BaseTestCase(TestCase): def clear_caches(self): for f in typing._cleanups: @@ -115,18 +124,18 @@ def test_errors(self): def test_can_subclass(self): class Mock(Any): pass - self.assertTrue(issubclass(Mock, Any)) + self.assertIsSubclass(Mock, Any) self.assertIsInstance(Mock(), Mock) class Something: pass - self.assertFalse(issubclass(Something, Any)) + self.assertNotIsSubclass(Something, Any) self.assertNotIsInstance(Something(), Mock) class MockSomething(Something, Mock): pass - self.assertTrue(issubclass(MockSomething, Any)) - self.assertTrue(issubclass(MockSomething, MockSomething)) - self.assertTrue(issubclass(MockSomething, Something)) - self.assertTrue(issubclass(MockSomething, Mock)) + self.assertIsSubclass(MockSomething, Any) + self.assertIsSubclass(MockSomething, MockSomething) + self.assertIsSubclass(MockSomething, Something) + self.assertIsSubclass(MockSomething, Mock) ms = MockSomething() self.assertIsInstance(ms, MockSomething) self.assertIsInstance(ms, Something) @@ -373,6 +382,7 @@ def test_alias(self): self.assertEqual(get_args(alias_2), (LiteralString,)) self.assertEqual(get_args(alias_3), (LiteralString,)) + class TypeVarTests(BaseTestCase): def test_basic_plain(self): T = TypeVar('T') @@ -467,8 +477,8 @@ def test_or(self): self.assertEqual(X | "x", Union[X, "x"]) self.assertEqual("x" | X, Union["x", X]) # make sure the order is correct - self.assertEqual(get_args(X | "x"), (X, ForwardRef("x"))) - self.assertEqual(get_args("x" | X), (ForwardRef("x"), X)) + self.assertEqual(get_args(X | "x"), (X, EqualToForwardRef("x"))) + self.assertEqual(get_args("x" | X), (EqualToForwardRef("x"), X)) def test_union_constrained(self): A = TypeVar('A', str, bytes) @@ -502,7 +512,7 @@ def test_cannot_instantiate_vars(self): def test_bound_errors(self): with self.assertRaises(TypeError): - TypeVar('X', bound=Union) + TypeVar('X', bound=Optional) with self.assertRaises(TypeError): TypeVar('X', str, float, bound=Employee) with self.assertRaisesRegex(TypeError, @@ -542,7 +552,7 @@ def test_var_substitution(self): def test_bad_var_substitution(self): T = TypeVar('T') bad_args = ( - (), (int, str), Union, + (), (int, str), Optional, Generic, Generic[T], Protocol, Protocol[T], Final, Final[int], ClassVar, ClassVar[int], ) @@ -625,7 +635,7 @@ class TypeParameterDefaultsTests(BaseTestCase): def test_typevar(self): T = TypeVar('T', default=int) self.assertEqual(T.__default__, int) - self.assertTrue(T.has_default()) + self.assertIs(T.has_default(), True) self.assertIsInstance(T, TypeVar) class A(Generic[T]): ... @@ -635,19 +645,19 @@ def test_typevar_none(self): U = TypeVar('U') U_None = TypeVar('U_None', default=None) self.assertIs(U.__default__, NoDefault) - self.assertFalse(U.has_default()) + self.assertIs(U.has_default(), False) self.assertIs(U_None.__default__, None) - self.assertTrue(U_None.has_default()) + self.assertIs(U_None.has_default(), True) class X[T]: ... T, = X.__type_params__ self.assertIs(T.__default__, NoDefault) - self.assertFalse(T.has_default()) + self.assertIs(T.has_default(), False) def test_paramspec(self): P = ParamSpec('P', default=(str, int)) self.assertEqual(P.__default__, (str, int)) - self.assertTrue(P.has_default()) + self.assertIs(P.has_default(), True) self.assertIsInstance(P, ParamSpec) class A(Generic[P]): ... @@ -660,19 +670,19 @@ def test_paramspec_none(self): U = ParamSpec('U') U_None = ParamSpec('U_None', default=None) self.assertIs(U.__default__, NoDefault) - self.assertFalse(U.has_default()) + self.assertIs(U.has_default(), False) self.assertIs(U_None.__default__, None) - self.assertTrue(U_None.has_default()) + self.assertIs(U_None.has_default(), True) class X[**P]: ... P, = X.__type_params__ self.assertIs(P.__default__, NoDefault) - self.assertFalse(P.has_default()) + self.assertIs(P.has_default(), False) def test_typevartuple(self): Ts = TypeVarTuple('Ts', default=Unpack[Tuple[str, int]]) self.assertEqual(Ts.__default__, Unpack[Tuple[str, int]]) - self.assertTrue(Ts.has_default()) + self.assertIs(Ts.has_default(), True) self.assertIsInstance(Ts, TypeVarTuple) class A(Generic[Unpack[Ts]]): ... @@ -754,18 +764,28 @@ class A(Generic[T, P, U]): ... self.assertEqual(A[float, [range]].__args__, (float, (range,), float)) self.assertEqual(A[float, [range], int].__args__, (float, (range,), int)) + def test_paramspec_and_typevar_specialization_2(self): + T = TypeVar("T") + P = ParamSpec('P', default=...) + U = TypeVar("U", default=float) + self.assertEqual(P.__default__, ...) + class A(Generic[T, P, U]): ... + self.assertEqual(A[float].__args__, (float, ..., float)) + self.assertEqual(A[float, [range]].__args__, (float, (range,), float)) + self.assertEqual(A[float, [range], int].__args__, (float, (range,), int)) + def test_typevartuple_none(self): U = TypeVarTuple('U') U_None = TypeVarTuple('U_None', default=None) self.assertIs(U.__default__, NoDefault) - self.assertFalse(U.has_default()) + self.assertIs(U.has_default(), False) self.assertIs(U_None.__default__, None) - self.assertTrue(U_None.has_default()) + self.assertIs(U_None.has_default(), True) class X[**Ts]: ... Ts, = X.__type_params__ self.assertIs(Ts.__default__, NoDefault) - self.assertFalse(Ts.has_default()) + self.assertIs(Ts.has_default(), False) def test_no_default_after_non_default(self): DefaultStrT = TypeVar('DefaultStrT', default=str) @@ -966,7 +986,7 @@ class C(Generic[T]): pass ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_two_parameters(self): T1 = TypeVar('T1') T2 = TypeVar('T2') @@ -1064,7 +1084,7 @@ class C(Generic[T1, T2, T3]): pass eval(expected_str) ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_variadic_parameters(self): T1 = TypeVar('T1') T2 = TypeVar('T2') @@ -1168,7 +1188,6 @@ class C(Generic[*Ts]): pass ) - class UnpackTests(BaseTestCase): def test_accepts_single_type(self): @@ -1999,11 +2018,11 @@ def test_basics(self): self.assertNotEqual(u, Union) def test_union_isinstance(self): - self.assertTrue(isinstance(42, Union[int, str])) - self.assertTrue(isinstance('abc', Union[int, str])) - self.assertFalse(isinstance(3.14, Union[int, str])) - self.assertTrue(isinstance(42, Union[int, list[int]])) - self.assertTrue(isinstance(42, Union[int, Any])) + self.assertIsInstance(42, Union[int, str]) + self.assertIsInstance('abc', Union[int, str]) + self.assertNotIsInstance(3.14, Union[int, str]) + self.assertIsInstance(42, Union[int, list[int]]) + self.assertIsInstance(42, Union[int, Any]) def test_union_isinstance_type_error(self): with self.assertRaises(TypeError): @@ -2020,9 +2039,9 @@ def test_union_isinstance_type_error(self): isinstance(42, Union[Any, str]) def test_optional_isinstance(self): - self.assertTrue(isinstance(42, Optional[int])) - self.assertTrue(isinstance(None, Optional[int])) - self.assertFalse(isinstance('abc', Optional[int])) + self.assertIsInstance(42, Optional[int]) + self.assertIsInstance(None, Optional[int]) + self.assertNotIsInstance('abc', Optional[int]) def test_optional_isinstance_type_error(self): with self.assertRaises(TypeError): @@ -2035,20 +2054,16 @@ def test_optional_isinstance_type_error(self): isinstance(None, Optional[Any]) def test_union_issubclass(self): - self.assertTrue(issubclass(int, Union[int, str])) - self.assertTrue(issubclass(str, Union[int, str])) - self.assertFalse(issubclass(float, Union[int, str])) - self.assertTrue(issubclass(int, Union[int, list[int]])) - self.assertTrue(issubclass(int, Union[int, Any])) - self.assertFalse(issubclass(int, Union[str, Any])) - self.assertTrue(issubclass(int, Union[Any, int])) - self.assertFalse(issubclass(int, Union[Any, str])) + self.assertIsSubclass(int, Union[int, str]) + self.assertIsSubclass(str, Union[int, str]) + self.assertNotIsSubclass(float, Union[int, str]) + self.assertIsSubclass(int, Union[int, list[int]]) + self.assertIsSubclass(int, Union[int, Any]) + self.assertNotIsSubclass(int, Union[str, Any]) + self.assertIsSubclass(int, Union[Any, int]) + self.assertNotIsSubclass(int, Union[Any, str]) def test_union_issubclass_type_error(self): - with self.assertRaises(TypeError): - issubclass(int, Union) - with self.assertRaises(TypeError): - issubclass(Union, int) with self.assertRaises(TypeError): issubclass(Union[int, str], int) with self.assertRaises(TypeError): @@ -2059,12 +2074,12 @@ def test_union_issubclass_type_error(self): issubclass(int, Union[list[int], str]) def test_optional_issubclass(self): - self.assertTrue(issubclass(int, Optional[int])) - self.assertTrue(issubclass(type(None), Optional[int])) - self.assertFalse(issubclass(str, Optional[int])) - self.assertTrue(issubclass(Any, Optional[Any])) - self.assertTrue(issubclass(type(None), Optional[Any])) - self.assertFalse(issubclass(int, Optional[Any])) + self.assertIsSubclass(int, Optional[int]) + self.assertIsSubclass(type(None), Optional[int]) + self.assertNotIsSubclass(str, Optional[int]) + self.assertIsSubclass(Any, Optional[Any]) + self.assertIsSubclass(type(None), Optional[Any]) + self.assertNotIsSubclass(int, Optional[Any]) def test_optional_issubclass_type_error(self): with self.assertRaises(TypeError): @@ -2123,41 +2138,40 @@ class B(metaclass=UnhashableMeta): ... self.assertEqual(Union[A, B].__args__, (A, B)) union1 = Union[A, B] - with self.assertRaises(TypeError): + with self.assertRaisesRegex(TypeError, "unhashable type: 'UnhashableMeta'"): hash(union1) union2 = Union[int, B] - with self.assertRaises(TypeError): + with self.assertRaisesRegex(TypeError, "unhashable type: 'UnhashableMeta'"): hash(union2) union3 = Union[A, int] - with self.assertRaises(TypeError): + with self.assertRaisesRegex(TypeError, "unhashable type: 'UnhashableMeta'"): hash(union3) def test_repr(self): - self.assertEqual(repr(Union), 'typing.Union') u = Union[Employee, int] - self.assertEqual(repr(u), 'typing.Union[%s.Employee, int]' % __name__) + self.assertEqual(repr(u), f'{__name__}.Employee | int') u = Union[int, Employee] - self.assertEqual(repr(u), 'typing.Union[int, %s.Employee]' % __name__) + self.assertEqual(repr(u), f'int | {__name__}.Employee') T = TypeVar('T') u = Union[T, int][int] self.assertEqual(repr(u), repr(int)) u = Union[List[int], int] - self.assertEqual(repr(u), 'typing.Union[typing.List[int], int]') + self.assertEqual(repr(u), 'typing.List[int] | int') u = Union[list[int], dict[str, float]] - self.assertEqual(repr(u), 'typing.Union[list[int], dict[str, float]]') + self.assertEqual(repr(u), 'list[int] | dict[str, float]') u = Union[int | float] - self.assertEqual(repr(u), 'typing.Union[int, float]') + self.assertEqual(repr(u), 'int | float') u = Union[None, str] - self.assertEqual(repr(u), 'typing.Optional[str]') + self.assertEqual(repr(u), 'None | str') u = Union[str, None] - self.assertEqual(repr(u), 'typing.Optional[str]') + self.assertEqual(repr(u), 'str | None') u = Union[None, str, int] - self.assertEqual(repr(u), 'typing.Union[NoneType, str, int]') + self.assertEqual(repr(u), 'None | str | int') u = Optional[str] - self.assertEqual(repr(u), 'typing.Optional[str]') + self.assertEqual(repr(u), 'str | None') def test_dir(self): dir_items = set(dir(Union[str, int])) @@ -2169,14 +2183,11 @@ def test_dir(self): def test_cannot_subclass(self): with self.assertRaisesRegex(TypeError, - r'Cannot subclass typing\.Union'): + r"type 'typing\.Union' is not an acceptable base type"): class C(Union): pass - with self.assertRaisesRegex(TypeError, CANNOT_SUBCLASS_TYPE): - class D(type(Union)): - pass with self.assertRaisesRegex(TypeError, - r'Cannot subclass typing\.Union\[int, str\]'): + r'Cannot subclass int \| str'): class E(Union[int, str]): pass @@ -2192,8 +2203,8 @@ def test_cannot_instantiate(self): type(u)() def test_union_generalization(self): - self.assertFalse(Union[str, typing.Iterable[int]] == str) - self.assertFalse(Union[str, typing.Iterable[int]] == typing.Iterable[int]) + self.assertNotEqual(Union[str, typing.Iterable[int]], str) + self.assertNotEqual(Union[str, typing.Iterable[int]], typing.Iterable[int]) self.assertIn(str, Union[str, typing.Iterable[int]].__args__) self.assertIn(typing.Iterable[int], Union[str, typing.Iterable[int]].__args__) @@ -2222,7 +2233,7 @@ def f(x: u): ... def test_function_repr_union(self): def fun() -> int: ... - self.assertEqual(repr(Union[fun, int]), 'typing.Union[fun, int]') + self.assertEqual(repr(Union[fun, int]), f'{__name__}.{fun.__qualname__} | int') def test_union_str_pattern(self): # Shouldn't crash; see http://bugs.python.org/issue25390 @@ -2270,6 +2281,16 @@ class Ints(enum.IntEnum): self.assertEqual(Union[Literal[1], Literal[Ints.B], Literal[True]].__args__, (Literal[1], Literal[Ints.B], Literal[True])) + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: types.UnionType[int, str] | float != types.UnionType[int, str, float] + def test_allow_non_types_in_or(self): + # gh-140348: Test that using | with a Union object allows things that are + # not allowed by is_unionable(). + U1 = Union[int, str] + self.assertEqual(U1 | float, Union[int, str, float]) + self.assertEqual(U1 | "float", Union[int, str, "float"]) + self.assertEqual(float | U1, Union[float, int, str]) + self.assertEqual("float" | U1, Union["float", int, str]) + class TupleTests(BaseTestCase): @@ -2557,7 +2578,7 @@ def test_concatenate(self): def test_nested_paramspec(self): # Since Callable has some special treatment, we want to be sure - # that substituion works correctly, see gh-103054 + # that substitution works correctly, see gh-103054 Callable = self.Callable P = ParamSpec('P') P2 = ParamSpec('P2') @@ -2609,6 +2630,7 @@ def test_errors(self): with self.assertRaisesRegex(TypeError, "few arguments for"): C1[int] + class TypingCallableTests(BaseCallableTests, BaseTestCase): Callable = typing.Callable @@ -2786,6 +2808,7 @@ class Coordinate(Protocol): x: int y: int + @runtime_checkable class Point(Coordinate, Protocol): label: str @@ -3155,6 +3178,21 @@ def x(self): ... with self.assertRaisesRegex(TypeError, only_classes_allowed): issubclass(1, BadPG) + def test_isinstance_against_superproto_doesnt_affect_subproto_instance(self): + @runtime_checkable + class Base(Protocol): + x: int + + @runtime_checkable + class Child(Base, Protocol): + y: str + + class Capybara: + x = 43 + + self.assertIsInstance(Capybara(), Base) + self.assertNotIsInstance(Capybara(), Child) + def test_implicit_issubclass_between_two_protocols(self): @runtime_checkable class CallableMembersProto(Protocol): @@ -3229,7 +3267,7 @@ def meth2(self, x, y): return True self.assertIsSubclass(NotAProtocolButAnImplicitSubclass2, CallableMembersProto) self.assertIsSubclass(NotAProtocolButAnImplicitSubclass3, CallableMembersProto) - @unittest.skip('TODO: RUSTPYTHON; (no gc)') + @unittest.skip("TODO: RUSTPYTHON; (no gc)") def test_isinstance_checks_not_at_whim_of_gc(self): self.addCleanup(gc.enable) gc.disable() @@ -3848,7 +3886,8 @@ def meth(self): pass acceptable_extra_attrs = { '_is_protocol', '_is_runtime_protocol', '__parameters__', - '__init__', '__annotations__', '__subclasshook__', + '__init__', '__annotations__', '__subclasshook__', '__annotate__', + '__annotations_cache__', '__annotate_func__', } self.assertLessEqual(vars(NonP).keys(), vars(C).keys() | acceptable_extra_attrs) self.assertLessEqual( @@ -4070,8 +4109,8 @@ def test_generic_protocols_repr(self): class P(Protocol[T, S]): pass - self.assertTrue(repr(P[T, S]).endswith('P[~T, ~S]')) - self.assertTrue(repr(P[int, str]).endswith('P[int, str]')) + self.assertEndsWith(repr(P[T, S]), 'P[~T, ~S]') + self.assertEndsWith(repr(P[int, str]), 'P[int, str]') def test_generic_protocols_eq(self): T = TypeVar('T') @@ -4111,12 +4150,12 @@ class PG(Protocol[T]): def meth(self): pass - self.assertTrue(P._is_protocol) - self.assertTrue(PR._is_protocol) - self.assertTrue(PG._is_protocol) - self.assertFalse(P._is_runtime_protocol) - self.assertTrue(PR._is_runtime_protocol) - self.assertTrue(PG[int]._is_protocol) + self.assertIs(P._is_protocol, True) + self.assertIs(PR._is_protocol, True) + self.assertIs(PG._is_protocol, True) + self.assertIs(P._is_runtime_protocol, False) + self.assertIs(PR._is_runtime_protocol, True) + self.assertIs(PG[int]._is_protocol, True) self.assertEqual(typing._get_protocol_attrs(P), {'meth'}) self.assertEqual(typing._get_protocol_attrs(PR), {'x'}) self.assertEqual(frozenset(typing._get_protocol_attrs(PG)), @@ -4172,7 +4211,7 @@ class P(Protocol): Alias2 = typing.Union[P, typing.Iterable] self.assertEqual(Alias, Alias2) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_protocols_pickleable(self): global P, CP # pickle wants to reference the class by name T = TypeVar('T') @@ -4323,11 +4362,50 @@ def __release_buffer__(self, mv: memoryview) -> None: self.assertNotIsSubclass(C, ReleasableBuffer) self.assertNotIsInstance(C(), ReleasableBuffer) + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'io' has no attribute 'Reader' + def test_io_reader_protocol_allowed(self): + @runtime_checkable + class CustomReader(io.Reader[bytes], Protocol): + def close(self): ... + + class A: pass + class B: + def read(self, sz=-1): + return b"" + def close(self): + pass + + self.assertIsSubclass(B, CustomReader) + self.assertIsInstance(B(), CustomReader) + self.assertNotIsSubclass(A, CustomReader) + self.assertNotIsInstance(A(), CustomReader) + + @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'io' has no attribute 'Writer' + def test_io_writer_protocol_allowed(self): + @runtime_checkable + class CustomWriter(io.Writer[bytes], Protocol): + def close(self): ... + + class A: pass + class B: + def write(self, b): + pass + def close(self): + pass + + self.assertIsSubclass(B, CustomWriter) + self.assertIsInstance(B(), CustomWriter) + self.assertNotIsSubclass(A, CustomWriter) + self.assertNotIsInstance(A(), CustomWriter) + def test_builtin_protocol_allowlist(self): with self.assertRaises(TypeError): class CustomProtocol(TestCase, Protocol): pass + class CustomPathLikeProtocol(os.PathLike, Protocol): + pass + class CustomContextManager(typing.ContextManager, Protocol): pass @@ -4541,6 +4619,42 @@ class Commentable(Protocol): ) self.assertIs(type(exc.__cause__), CustomError) + def test_isinstance_with_deferred_evaluation_of_annotations(self): + @runtime_checkable + class P(Protocol): + def meth(self): + ... + + class DeferredClass: + x: undefined + + class DeferredClassImplementingP: + x: undefined | int + + def __init__(self): + self.x = 0 + + def meth(self): + ... + + # override meth with a non-method attribute to make it part of __annotations__ instead of __dict__ + class SubProtocol(P, Protocol): + meth: undefined + + + self.assertIsSubclass(SubProtocol, P) + self.assertNotIsInstance(DeferredClass(), P) + self.assertIsInstance(DeferredClassImplementingP(), P) + + def test_deferred_evaluation_of_annotations(self): + class DeferredProto(Protocol): + x: DoesNotExist + self.assertEqual(get_protocol_members(DeferredProto), {"x"}) + self.assertEqual( + annotationlib.get_annotations(DeferredProto, format=annotationlib.Format.STRING), + {'x': 'DoesNotExist'} + ) + class GenericTests(BaseTestCase): @@ -4588,6 +4702,35 @@ class D(Generic[T]): pass with self.assertRaises(TypeError): D[()] + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_generic_init_subclass_not_called_error(self): + notes = ["Note: this exception may have been caused by " + r"'GenericTests.test_generic_init_subclass_not_called_error.<locals>.Base.__init_subclass__' " + "(or the '__init_subclass__' method on a superclass) not calling 'super().__init_subclass__()'"] + + class Base: + def __init_subclass__(cls) -> None: + # Oops, I forgot super().__init_subclass__()! + pass + + with self.subTest(): + class Sub(Base, Generic[T]): + pass + + with self.assertRaises(AttributeError) as cm: + Sub[int] + + self.assertEqual(cm.exception.__notes__, notes) + + with self.subTest(): + class Sub[U](Base): + pass + + with self.assertRaises(AttributeError) as cm: + Sub[int] + + self.assertEqual(cm.exception.__notes__, notes) + def test_generic_subclass_checks(self): for typ in [list[int], List[int], tuple[int, str], Tuple[int, str], @@ -4659,8 +4802,7 @@ class C(Generic[T]): self.assertNotEqual(Z, Y[int]) self.assertNotEqual(Z, Y[T]) - self.assertTrue(str(Z).endswith( - '.C[typing.Tuple[str, int]]')) + self.assertEndsWith(str(Z), '.C[typing.Tuple[str, int]]') def test_new_repr(self): T = TypeVar('T') @@ -4888,12 +5030,12 @@ class A(Generic[T]): self.assertNotEqual(typing.FrozenSet[A[str]], typing.FrozenSet[mod_generics_cache.B.A[str]]) - self.assertTrue(repr(Tuple[A[str]]).endswith('<locals>.A[str]]')) - self.assertTrue(repr(Tuple[B.A[str]]).endswith('<locals>.B.A[str]]')) - self.assertTrue(repr(Tuple[mod_generics_cache.A[str]]) - .endswith('mod_generics_cache.A[str]]')) - self.assertTrue(repr(Tuple[mod_generics_cache.B.A[str]]) - .endswith('mod_generics_cache.B.A[str]]')) + self.assertEndsWith(repr(Tuple[A[str]]), '<locals>.A[str]]') + self.assertEndsWith(repr(Tuple[B.A[str]]), '<locals>.B.A[str]]') + self.assertEndsWith(repr(Tuple[mod_generics_cache.A[str]]), + 'mod_generics_cache.A[str]]') + self.assertEndsWith(repr(Tuple[mod_generics_cache.B.A[str]]), + 'mod_generics_cache.B.A[str]]') def test_extended_generic_rules_eq(self): T = TypeVar('T') @@ -4914,11 +5056,11 @@ class Derived(Base): ... def test_extended_generic_rules_repr(self): T = TypeVar('T') self.assertEqual(repr(Union[Tuple, Callable]).replace('typing.', ''), - 'Union[Tuple, Callable]') + 'Tuple | Callable') self.assertEqual(repr(Union[Tuple, Tuple[int]]).replace('typing.', ''), - 'Union[Tuple, Tuple[int]]') + 'Tuple | Tuple[int]') self.assertEqual(repr(Callable[..., Optional[T]][int]).replace('typing.', ''), - 'Callable[..., Optional[int]]') + 'Callable[..., int | None]') self.assertEqual(repr(Callable[[], List[T]][int]).replace('typing.', ''), 'Callable[[], List[int]]') @@ -4984,7 +5126,7 @@ class C3: def f(x: X): ... self.assertEqual( get_type_hints(f, globals(), locals()), - {'x': list[list[ForwardRef('X')]]} + {'x': list[list[EqualToForwardRef('X')]]} ) def test_pep695_generic_class_with_future_annotations(self): @@ -5098,9 +5240,9 @@ def __contains__(self, item): with self.assertRaises(TypeError): issubclass(Tuple[int, ...], typing.Iterable) - def test_fail_with_bare_union(self): + def test_fail_with_special_forms(self): with self.assertRaises(TypeError): - List[Union] + List[Final] with self.assertRaises(TypeError): Tuple[Optional] with self.assertRaises(TypeError): @@ -5147,7 +5289,7 @@ def test_all_repr_eq_any(self): self.assertNotEqual(repr(base), '') self.assertEqual(base, base) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_pickle(self): global C # pickle wants to reference the class by name T = TypeVar('T') @@ -5204,10 +5346,12 @@ class Node(Generic[T]): ... Tuple[Any, Any], Node[T], Node[int], Node[Any], typing.Iterable[T], typing.Iterable[Any], typing.Iterable[int], typing.Dict[int, str], typing.Dict[T, Any], ClassVar[int], ClassVar[List[T]], Tuple['T', 'T'], - Union['T', int], List['T'], typing.Mapping['T', int]] - for t in things + [Any]: - self.assertEqual(t, copy(t)) - self.assertEqual(t, deepcopy(t)) + Union['T', int], List['T'], typing.Mapping['T', int], + Union[b"x", b"y"], Any] + for t in things: + with self.subTest(thing=t): + self.assertEqual(t, copy(t)) + self.assertEqual(t, deepcopy(t)) def test_immutability_by_copy_and_pickle(self): # Special forms like Union, Any, etc., generic aliases to containers like List, @@ -5643,8 +5787,6 @@ def test_subclass_special_form(self): for obj in ( ClassVar[int], Final[int], - Union[int, float], - Optional[int], Literal[1, 2], Concatenate[int, ParamSpec("P")], TypeGuard[int], @@ -5676,7 +5818,7 @@ class A: __parameters__ = (T,) # Bare classes should be skipped for a in (List, list): - for b in (A, int, TypeVar, TypeVarTuple, ParamSpec, types.GenericAlias, types.UnionType): + for b in (A, int, TypeVar, TypeVarTuple, ParamSpec, types.GenericAlias, Union): with self.subTest(generic=a, sub=b): with self.assertRaisesRegex(TypeError, '.* is not a generic class'): a[b][str] @@ -5695,7 +5837,7 @@ class A: for s in (int, G, A, List, list, TypeVar, TypeVarTuple, ParamSpec, - types.GenericAlias, types.UnionType): + types.GenericAlias, Union): for t in Tuple, tuple: with self.subTest(tuple=t, sub=s): @@ -5713,7 +5855,7 @@ class A: with self.assertRaises(TypeError): a[int] - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: ".+__typing_subst__.+tuple.+int.*" does not match "'TypeAliasType' object is not subscriptable" + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: ".+__typing_subst__.+tuple.+int.*" does not match "'TypeAliasType' object is not subscriptable" def test_return_non_tuple_while_unpacking(self): # GH-138497: GenericAlias objects didn't ensure that __typing_subst__ actually # returned a tuple @@ -5777,6 +5919,7 @@ def test_no_isinstance(self): with self.assertRaises(TypeError): issubclass(int, ClassVar) + class FinalTests(BaseTestCase): def test_basics(self): @@ -5833,7 +5976,7 @@ def test_final_unmodified(self): def func(x): ... self.assertIs(func, final(func)) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_dunder_final(self): @final def func(): ... @@ -5855,7 +5998,7 @@ def __call__(self, *args, **kwargs): @Wrapper def wrapped(): ... self.assertIsInstance(wrapped, Wrapper) - self.assertIs(False, hasattr(wrapped, "__final__")) + self.assertNotHasAttr(wrapped, "__final__") class Meta(type): @property @@ -5867,7 +6010,7 @@ class WithMeta(metaclass=Meta): ... # Builtin classes throw TypeError if you try to set an # attribute. final(int) - self.assertIs(False, hasattr(int, "__final__")) + self.assertNotHasAttr(int, "__final__") # Make sure it works with common builtin decorators class Methods: @@ -5948,19 +6091,19 @@ def static_method_bad_order(): self.assertEqual(Derived.class_method_good_order(), 42) self.assertIs(True, Derived.class_method_good_order.__override__) self.assertEqual(Derived.class_method_bad_order(), 42) - self.assertIs(False, hasattr(Derived.class_method_bad_order, "__override__")) + self.assertNotHasAttr(Derived.class_method_bad_order, "__override__") self.assertEqual(Derived.static_method_good_order(), 42) self.assertIs(True, Derived.static_method_good_order.__override__) self.assertEqual(Derived.static_method_bad_order(), 42) - self.assertIs(False, hasattr(Derived.static_method_bad_order, "__override__")) + self.assertNotHasAttr(Derived.static_method_bad_order, "__override__") # Base object is not changed: - self.assertIs(False, hasattr(Base.normal_method, "__override__")) - self.assertIs(False, hasattr(Base.class_method_good_order, "__override__")) - self.assertIs(False, hasattr(Base.class_method_bad_order, "__override__")) - self.assertIs(False, hasattr(Base.static_method_good_order, "__override__")) - self.assertIs(False, hasattr(Base.static_method_bad_order, "__override__")) + self.assertNotHasAttr(Base.normal_method, "__override__") + self.assertNotHasAttr(Base.class_method_good_order, "__override__") + self.assertNotHasAttr(Base.class_method_bad_order, "__override__") + self.assertNotHasAttr(Base.static_method_good_order, "__override__") + self.assertNotHasAttr(Base.static_method_bad_order, "__override__") def test_property(self): class Base: @@ -5983,10 +6126,10 @@ def wrong(self) -> int: instance = Child() self.assertEqual(instance.correct, 2) - self.assertTrue(Child.correct.fget.__override__) + self.assertIs(Child.correct.fget.__override__, True) self.assertEqual(instance.wrong, 2) - self.assertFalse(hasattr(Child.wrong, "__override__")) - self.assertFalse(hasattr(Child.wrong.fset, "__override__")) + self.assertNotHasAttr(Child.wrong, "__override__") + self.assertNotHasAttr(Child.wrong.fset, "__override__") def test_silent_failure(self): class CustomProp: @@ -6003,7 +6146,7 @@ def some(self): return 1 self.assertEqual(WithOverride.some, 1) - self.assertFalse(hasattr(WithOverride.some, "__override__")) + self.assertNotHasAttr(WithOverride.some, "__override__") def test_multiple_decorators(self): def with_wraps(f): # similar to `lru_cache` definition @@ -6024,9 +6167,9 @@ def on_bottom(self, a: int) -> int: instance = WithOverride() self.assertEqual(instance.on_top(1), 2) - self.assertTrue(instance.on_top.__override__) + self.assertIs(instance.on_top.__override__, True) self.assertEqual(instance.on_bottom(1), 3) - self.assertTrue(instance.on_bottom.__override__) + self.assertIs(instance.on_bottom.__override__, True) class CastTests(BaseTestCase): @@ -6064,8 +6207,6 @@ def test_errors(self): # We need this to make sure that `@no_type_check` respects `__module__` attr: -from test.typinganndata import ann_module8 - @no_type_check class NoTypeCheck_Outer: Inner = ann_module8.NoTypeCheck_Outer.Inner @@ -6075,474 +6216,168 @@ class NoTypeCheck_WithFunction: NoTypeCheck_function = ann_module8.NoTypeCheck_function -class ForwardRefTests(BaseTestCase): - - def test_basics(self): +class NoTypeCheckTests(BaseTestCase): + def test_no_type_check(self): - class Node(Generic[T]): + @no_type_check + def foo(a: 'whatevers') -> {}: + pass - def __init__(self, label: T): - self.label = label - self.left = self.right = None + th = get_type_hints(foo) + self.assertEqual(th, {}) - def add_both(self, - left: 'Optional[Node[T]]', - right: 'Node[T]' = None, - stuff: int = None, - blah=None): - self.left = left - self.right = right + def test_no_type_check_class(self): - def add_left(self, node: Optional['Node[T]']): - self.add_both(node, None) + @no_type_check + class C: + def foo(a: 'whatevers') -> {}: + pass - def add_right(self, node: 'Node[T]' = None): - self.add_both(None, node) + cth = get_type_hints(C.foo) + self.assertEqual(cth, {}) + ith = get_type_hints(C().foo) + self.assertEqual(ith, {}) - t = Node[int] - both_hints = get_type_hints(t.add_both, globals(), locals()) - self.assertEqual(both_hints['left'], Optional[Node[T]]) - self.assertEqual(both_hints['right'], Node[T]) - self.assertEqual(both_hints['stuff'], int) - self.assertNotIn('blah', both_hints) + def test_no_type_check_no_bases(self): + class C: + def meth(self, x: int): ... + @no_type_check + class D(C): + c = C - left_hints = get_type_hints(t.add_left, globals(), locals()) - self.assertEqual(left_hints['node'], Optional[Node[T]]) + # verify that @no_type_check never affects bases + self.assertEqual(get_type_hints(C.meth), {'x': int}) - right_hints = get_type_hints(t.add_right, globals(), locals()) - self.assertEqual(right_hints['node'], Node[T]) + # and never child classes: + class Child(D): + def foo(self, x: int): ... - def test_forwardref_instance_type_error(self): - fr = typing.ForwardRef('int') - with self.assertRaises(TypeError): - isinstance(42, fr) + self.assertEqual(get_type_hints(Child.foo), {'x': int}) - def test_forwardref_subclass_type_error(self): - fr = typing.ForwardRef('int') - with self.assertRaises(TypeError): - issubclass(int, fr) + def test_no_type_check_nested_types(self): + # See https://bugs.python.org/issue46571 + class Other: + o: int + class B: # Has the same `__name__`` as `A.B` and different `__qualname__` + o: int + @no_type_check + class A: + a: int + class B: + b: int + class C: + c: int + class D: + d: int - def test_forwardref_only_str_arg(self): - with self.assertRaises(TypeError): - typing.ForwardRef(1) # only `str` type is allowed + Other = Other - def test_forward_equality(self): - fr = typing.ForwardRef('int') - self.assertEqual(fr, typing.ForwardRef('int')) - self.assertNotEqual(List['int'], List[int]) - self.assertNotEqual(fr, typing.ForwardRef('int', module=__name__)) - frm = typing.ForwardRef('int', module=__name__) - self.assertEqual(frm, typing.ForwardRef('int', module=__name__)) - self.assertNotEqual(frm, typing.ForwardRef('int', module='__other_name__')) + for klass in [A, A.B, A.B.C, A.D]: + with self.subTest(klass=klass): + self.assertIs(klass.__no_type_check__, True) + self.assertEqual(get_type_hints(klass), {}) - def test_forward_equality_gth(self): - c1 = typing.ForwardRef('C') - c1_gth = typing.ForwardRef('C') - c2 = typing.ForwardRef('C') - c2_gth = typing.ForwardRef('C') + for not_modified in [Other, B]: + with self.subTest(not_modified=not_modified): + with self.assertRaises(AttributeError): + not_modified.__no_type_check__ + self.assertNotEqual(get_type_hints(not_modified), {}) - class C: - pass - def foo(a: c1_gth, b: c2_gth): - pass + def test_no_type_check_class_and_static_methods(self): + @no_type_check + class Some: + @staticmethod + def st(x: int) -> int: ... + @classmethod + def cl(cls, y: int) -> int: ... - self.assertEqual(get_type_hints(foo, globals(), locals()), {'a': C, 'b': C}) - self.assertEqual(c1, c2) - self.assertEqual(c1, c1_gth) - self.assertEqual(c1_gth, c2_gth) - self.assertEqual(List[c1], List[c1_gth]) - self.assertNotEqual(List[c1], List[C]) - self.assertNotEqual(List[c1_gth], List[C]) - self.assertEqual(Union[c1, c1_gth], Union[c1]) - self.assertEqual(Union[c1, c1_gth, int], Union[c1, int]) - - def test_forward_equality_hash(self): - c1 = typing.ForwardRef('int') - c1_gth = typing.ForwardRef('int') - c2 = typing.ForwardRef('int') - c2_gth = typing.ForwardRef('int') - - def foo(a: c1_gth, b: c2_gth): - pass - get_type_hints(foo, globals(), locals()) + self.assertIs(Some.st.__no_type_check__, True) + self.assertEqual(get_type_hints(Some.st), {}) + self.assertIs(Some.cl.__no_type_check__, True) + self.assertEqual(get_type_hints(Some.cl), {}) - self.assertEqual(hash(c1), hash(c2)) - self.assertEqual(hash(c1_gth), hash(c2_gth)) - self.assertEqual(hash(c1), hash(c1_gth)) + def test_no_type_check_other_module(self): + self.assertIs(NoTypeCheck_Outer.__no_type_check__, True) + with self.assertRaises(AttributeError): + ann_module8.NoTypeCheck_Outer.__no_type_check__ + with self.assertRaises(AttributeError): + ann_module8.NoTypeCheck_Outer.Inner.__no_type_check__ - c3 = typing.ForwardRef('int', module=__name__) - c4 = typing.ForwardRef('int', module='__other_name__') + self.assertIs(NoTypeCheck_WithFunction.__no_type_check__, True) + with self.assertRaises(AttributeError): + ann_module8.NoTypeCheck_function.__no_type_check__ - self.assertNotEqual(hash(c3), hash(c1)) - self.assertNotEqual(hash(c3), hash(c1_gth)) - self.assertNotEqual(hash(c3), hash(c4)) - self.assertEqual(hash(c3), hash(typing.ForwardRef('int', module=__name__))) + def test_no_type_check_foreign_functions(self): + # We should not modify this function: + def some(*args: int) -> int: + ... - def test_forward_equality_namespace(self): + @no_type_check class A: - pass - def namespace1(): - a = typing.ForwardRef('A') - def fun(x: a): - pass - get_type_hints(fun, globals(), locals()) - return a + some_alias = some + some_class = classmethod(some) + some_static = staticmethod(some) - def namespace2(): - a = typing.ForwardRef('A') + with self.assertRaises(AttributeError): + some.__no_type_check__ + self.assertEqual(get_type_hints(some), {'args': int, 'return': int}) - class A: - pass - def fun(x: a): - pass + def test_no_type_check_lambda(self): + @no_type_check + class A: + # Corner case: `lambda` is both an assignment and a function: + bar: Callable[[int], int] = lambda arg: arg - get_type_hints(fun, globals(), locals()) - return a + self.assertIs(A.bar.__no_type_check__, True) + self.assertEqual(get_type_hints(A.bar), {}) - self.assertEqual(namespace1(), namespace1()) - self.assertNotEqual(namespace1(), namespace2()) + def test_no_type_check_TypeError(self): + # This simply should not fail with + # `TypeError: can't set attributes of built-in/extension type 'dict'` + no_type_check(dict) - def test_forward_repr(self): - self.assertEqual(repr(List['int']), "typing.List[ForwardRef('int')]") - self.assertEqual(repr(List[ForwardRef('int', module='mod')]), - "typing.List[ForwardRef('int', module='mod')]") + def test_no_type_check_forward_ref_as_string(self): + class C: + foo: typing.ClassVar[int] = 7 + class D: + foo: ClassVar[int] = 7 + class E: + foo: 'typing.ClassVar[int]' = 7 + class F: + foo: 'ClassVar[int]' = 7 - def test_union_forward(self): + expected_result = {'foo': typing.ClassVar[int]} + for clazz in [C, D, E, F]: + self.assertEqual(get_type_hints(clazz), expected_result) - def foo(a: Union['T']): - pass + def test_meta_no_type_check(self): + depr_msg = ( + "'typing.no_type_check_decorator' is deprecated " + "and slated for removal in Python 3.15" + ) + with self.assertWarnsRegex(DeprecationWarning, depr_msg): + @no_type_check_decorator + def magic_decorator(func): + return func - self.assertEqual(get_type_hints(foo, globals(), locals()), - {'a': Union[T]}) + self.assertEqual(magic_decorator.__name__, 'magic_decorator') - def foo(a: tuple[ForwardRef('T')] | int): + @magic_decorator + def foo(a: 'whatevers') -> {}: pass - self.assertEqual(get_type_hints(foo, globals(), locals()), - {'a': tuple[T] | int}) + @magic_decorator + class C: + def foo(a: 'whatevers') -> {}: + pass - def test_tuple_forward(self): - - def foo(a: Tuple['T']): - pass - - self.assertEqual(get_type_hints(foo, globals(), locals()), - {'a': Tuple[T]}) - - def foo(a: tuple[ForwardRef('T')]): - pass - - self.assertEqual(get_type_hints(foo, globals(), locals()), - {'a': tuple[T]}) - - def test_double_forward(self): - def foo(a: 'List[\'int\']'): - pass - self.assertEqual(get_type_hints(foo, globals(), locals()), - {'a': List[int]}) - - def test_forward_recursion_actually(self): - def namespace1(): - a = typing.ForwardRef('A') - A = a - def fun(x: a): pass - - ret = get_type_hints(fun, globals(), locals()) - return a - - def namespace2(): - a = typing.ForwardRef('A') - A = a - def fun(x: a): pass - - ret = get_type_hints(fun, globals(), locals()) - return a - - def cmp(o1, o2): - return o1 == o2 - - with infinite_recursion(25): - r1 = namespace1() - r2 = namespace2() - self.assertIsNot(r1, r2) - self.assertRaises(RecursionError, cmp, r1, r2) - - def test_union_forward_recursion(self): - ValueList = List['Value'] - Value = Union[str, ValueList] - - class C: - foo: List[Value] - class D: - foo: Union[Value, ValueList] - class E: - foo: Union[List[Value], ValueList] - class F: - foo: Union[Value, List[Value], ValueList] - - self.assertEqual(get_type_hints(C, globals(), locals()), get_type_hints(C, globals(), locals())) - self.assertEqual(get_type_hints(C, globals(), locals()), - {'foo': List[Union[str, List[Union[str, List['Value']]]]]}) - self.assertEqual(get_type_hints(D, globals(), locals()), - {'foo': Union[str, List[Union[str, List['Value']]]]}) - self.assertEqual(get_type_hints(E, globals(), locals()), - {'foo': Union[ - List[Union[str, List[Union[str, List['Value']]]]], - List[Union[str, List['Value']]] - ] - }) - self.assertEqual(get_type_hints(F, globals(), locals()), - {'foo': Union[ - str, - List[Union[str, List['Value']]], - List[Union[str, List[Union[str, List['Value']]]]] - ] - }) - - def test_callable_forward(self): - - def foo(a: Callable[['T'], 'T']): - pass - - self.assertEqual(get_type_hints(foo, globals(), locals()), - {'a': Callable[[T], T]}) - - def test_callable_with_ellipsis_forward(self): - - def foo(a: 'Callable[..., T]'): - pass - - self.assertEqual(get_type_hints(foo, globals(), locals()), - {'a': Callable[..., T]}) - - def test_special_forms_forward(self): - - class C: - a: Annotated['ClassVar[int]', (3, 5)] = 4 - b: Annotated['Final[int]', "const"] = 4 - x: 'ClassVar' = 4 - y: 'Final' = 4 - - class CF: - b: List['Final[int]'] = 4 - - self.assertEqual(get_type_hints(C, globals())['a'], ClassVar[int]) - self.assertEqual(get_type_hints(C, globals())['b'], Final[int]) - self.assertEqual(get_type_hints(C, globals())['x'], ClassVar) - self.assertEqual(get_type_hints(C, globals())['y'], Final) - with self.assertRaises(TypeError): - get_type_hints(CF, globals()), - - def test_syntax_error(self): - - with self.assertRaises(SyntaxError): - Generic['/T'] - - def test_delayed_syntax_error(self): - - def foo(a: 'Node[T'): - pass - - with self.assertRaises(SyntaxError): - get_type_hints(foo) - - def test_syntax_error_empty_string(self): - for form in [typing.List, typing.Set, typing.Type, typing.Deque]: - with self.subTest(form=form): - with self.assertRaises(SyntaxError): - form[''] - - def test_name_error(self): - - def foo(a: 'Noode[T]'): - pass - - with self.assertRaises(NameError): - get_type_hints(foo, locals()) - - def test_no_type_check(self): - - @no_type_check - def foo(a: 'whatevers') -> {}: - pass - - th = get_type_hints(foo) - self.assertEqual(th, {}) - - def test_no_type_check_class(self): - - @no_type_check - class C: - def foo(a: 'whatevers') -> {}: - pass - - cth = get_type_hints(C.foo) - self.assertEqual(cth, {}) - ith = get_type_hints(C().foo) - self.assertEqual(ith, {}) - - def test_no_type_check_no_bases(self): - class C: - def meth(self, x: int): ... - @no_type_check - class D(C): - c = C - - # verify that @no_type_check never affects bases - self.assertEqual(get_type_hints(C.meth), {'x': int}) - - # and never child classes: - class Child(D): - def foo(self, x: int): ... - - self.assertEqual(get_type_hints(Child.foo), {'x': int}) - - def test_no_type_check_nested_types(self): - # See https://bugs.python.org/issue46571 - class Other: - o: int - class B: # Has the same `__name__`` as `A.B` and different `__qualname__` - o: int - @no_type_check - class A: - a: int - class B: - b: int - class C: - c: int - class D: - d: int - - Other = Other - - for klass in [A, A.B, A.B.C, A.D]: - with self.subTest(klass=klass): - self.assertTrue(klass.__no_type_check__) - self.assertEqual(get_type_hints(klass), {}) - - for not_modified in [Other, B]: - with self.subTest(not_modified=not_modified): - with self.assertRaises(AttributeError): - not_modified.__no_type_check__ - self.assertNotEqual(get_type_hints(not_modified), {}) - - def test_no_type_check_class_and_static_methods(self): - @no_type_check - class Some: - @staticmethod - def st(x: int) -> int: ... - @classmethod - def cl(cls, y: int) -> int: ... - - self.assertTrue(Some.st.__no_type_check__) - self.assertEqual(get_type_hints(Some.st), {}) - self.assertTrue(Some.cl.__no_type_check__) - self.assertEqual(get_type_hints(Some.cl), {}) - - def test_no_type_check_other_module(self): - self.assertTrue(NoTypeCheck_Outer.__no_type_check__) - with self.assertRaises(AttributeError): - ann_module8.NoTypeCheck_Outer.__no_type_check__ - with self.assertRaises(AttributeError): - ann_module8.NoTypeCheck_Outer.Inner.__no_type_check__ - - self.assertTrue(NoTypeCheck_WithFunction.__no_type_check__) - with self.assertRaises(AttributeError): - ann_module8.NoTypeCheck_function.__no_type_check__ - - def test_no_type_check_foreign_functions(self): - # We should not modify this function: - def some(*args: int) -> int: - ... - - @no_type_check - class A: - some_alias = some - some_class = classmethod(some) - some_static = staticmethod(some) - - with self.assertRaises(AttributeError): - some.__no_type_check__ - self.assertEqual(get_type_hints(some), {'args': int, 'return': int}) - - def test_no_type_check_lambda(self): - @no_type_check - class A: - # Corner case: `lambda` is both an assignment and a function: - bar: Callable[[int], int] = lambda arg: arg - - self.assertTrue(A.bar.__no_type_check__) - self.assertEqual(get_type_hints(A.bar), {}) - - def test_no_type_check_TypeError(self): - # This simply should not fail with - # `TypeError: can't set attributes of built-in/extension type 'dict'` - no_type_check(dict) - - def test_no_type_check_forward_ref_as_string(self): - class C: - foo: typing.ClassVar[int] = 7 - class D: - foo: ClassVar[int] = 7 - class E: - foo: 'typing.ClassVar[int]' = 7 - class F: - foo: 'ClassVar[int]' = 7 - - expected_result = {'foo': typing.ClassVar[int]} - for clazz in [C, D, E, F]: - self.assertEqual(get_type_hints(clazz), expected_result) - - def test_meta_no_type_check(self): - depr_msg = ( - "'typing.no_type_check_decorator' is deprecated " - "and slated for removal in Python 3.15" - ) - with self.assertWarnsRegex(DeprecationWarning, depr_msg): - @no_type_check_decorator - def magic_decorator(func): - return func - - self.assertEqual(magic_decorator.__name__, 'magic_decorator') - - @magic_decorator - def foo(a: 'whatevers') -> {}: - pass - - @magic_decorator - class C: - def foo(a: 'whatevers') -> {}: - pass - - self.assertEqual(foo.__name__, 'foo') - th = get_type_hints(foo) - self.assertEqual(th, {}) - cth = get_type_hints(C.foo) - self.assertEqual(cth, {}) - ith = get_type_hints(C().foo) - self.assertEqual(ith, {}) - - def test_default_globals(self): - code = ("class C:\n" - " def foo(self, a: 'C') -> 'D': pass\n" - "class D:\n" - " def bar(self, b: 'D') -> C: pass\n" - ) - ns = {} - exec(code, ns) - hints = get_type_hints(ns['C'].foo) - self.assertEqual(hints, {'a': ns['C'], 'return': ns['D']}) - - def test_final_forward_ref(self): - self.assertEqual(gth(Loop, globals())['attr'], Final[Loop]) - self.assertNotEqual(gth(Loop, globals())['attr'], Final[int]) - self.assertNotEqual(gth(Loop, globals())['attr'], Final) - - def test_or(self): - X = ForwardRef('X') - # __or__/__ror__ itself - self.assertEqual(X | "x", Union[X, "x"]) - self.assertEqual("x" | X, Union["x", X]) + self.assertEqual(foo.__name__, 'foo') + th = get_type_hints(foo) + self.assertEqual(th, {}) + cth = get_type_hints(C.foo) + self.assertEqual(cth, {}) + ith = get_type_hints(C().foo) + self.assertEqual(ith, {}) class InternalsTests(BaseTestCase): @@ -6580,6 +6415,16 @@ def test_collect_parameters(self): typing._collect_parameters self.assertEqual(cm.filename, __file__) + @cpython_only + def test_lazy_import(self): + import_helper.ensure_lazy_imports("typing", { + "warnings", + "inspect", + "re", + "contextlib", + "annotationlib", + }) + @lru_cache() def cached_func(x, y): @@ -6686,10 +6531,6 @@ def test_overload_registry_repeated(self): self.assertEqual(list(get_overloads(impl)), overloads) -from test.typinganndata import ( - ann_module, ann_module2, ann_module3, ann_module5, ann_module6, -) - T_a = TypeVar('T_a') class AwaitableWrapper(typing.Awaitable[T_a]): @@ -6842,7 +6683,7 @@ def nested(self: 'ForRefExample'): pass -class GetTypeHintTests(BaseTestCase): +class GetTypeHintsTests(BaseTestCase): def test_get_type_hints_from_various_objects(self): # For invalid objects should fail with TypeError (not AttributeError etc). with self.assertRaises(TypeError): @@ -6852,9 +6693,8 @@ def test_get_type_hints_from_various_objects(self): with self.assertRaises(TypeError): gth(None) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_get_type_hints_modules(self): - ann_module_type_hints = {1: 2, 'f': Tuple[int, int], 'x': int, 'y': str, 'u': int | float} + ann_module_type_hints = {'f': Tuple[int, int], 'x': int, 'y': str, 'u': int | float} self.assertEqual(gth(ann_module), ann_module_type_hints) self.assertEqual(gth(ann_module2), {}) self.assertEqual(gth(ann_module3), {}) @@ -6868,11 +6708,12 @@ def test_get_type_hints_modules_forwardref(self): 'default_b': Optional[mod_generics_cache.B]} self.assertEqual(gth(mod_generics_cache), mgc_hints) + @unittest.expectedFailure # TODO: RUSTPYTHON; + {'x': <class 'int'>} def test_get_type_hints_classes(self): self.assertEqual(gth(ann_module.C), # gth will find the right globalns {'y': Optional[ann_module.C]}) self.assertIsInstance(gth(ann_module.j_class), dict) - self.assertEqual(gth(ann_module.M), {'123': 123, 'o': type}) + self.assertEqual(gth(ann_module.M), {'o': type}) self.assertEqual(gth(ann_module.D), {'j': str, 'k': str, 'y': Optional[ann_module.C]}) self.assertEqual(gth(ann_module.Y), {'z': int}) @@ -6903,8 +6744,8 @@ def test_respect_no_type_check(self): class NoTpCheck: class Inn: def __init__(self, x: 'not a type'): ... - self.assertTrue(NoTpCheck.__no_type_check__) - self.assertTrue(NoTpCheck.Inn.__init__.__no_type_check__) + self.assertIs(NoTpCheck.__no_type_check__, True) + self.assertIs(NoTpCheck.Inn.__init__.__no_type_check__, True) self.assertEqual(gth(ann_module2.NTC.meth), {}) class ABase(Generic[T]): def meth(x: int): ... @@ -7050,111 +6891,320 @@ def __iand__(self, other: Const["MySet[T]"]) -> "MySet[T]": {'other': MySet[T], 'return': MySet[T]} ) - def test_get_type_hints_annotated_with_none_default(self): - # See: https://bugs.python.org/issue46195 - def annotated_with_none_default(x: Annotated[int, 'data'] = None): ... - self.assertEqual( - get_type_hints(annotated_with_none_default), - {'x': int}, - ) - self.assertEqual( - get_type_hints(annotated_with_none_default, include_extras=True), - {'x': Annotated[int, 'data']}, - ) + def test_get_type_hints_annotated_with_none_default(self): + # See: https://bugs.python.org/issue46195 + def annotated_with_none_default(x: Annotated[int, 'data'] = None): ... + self.assertEqual( + get_type_hints(annotated_with_none_default), + {'x': int}, + ) + self.assertEqual( + get_type_hints(annotated_with_none_default, include_extras=True), + {'x': Annotated[int, 'data']}, + ) + + def test_get_type_hints_classes_str_annotations(self): + class Foo: + y = str + x: 'y' + # This previously raised an error under PEP 563. + self.assertEqual(get_type_hints(Foo), {'x': str}) + + def test_get_type_hints_bad_module(self): + # bpo-41515 + class BadModule: + pass + BadModule.__module__ = 'bad' # Something not in sys.modules + self.assertNotIn('bad', sys.modules) + self.assertEqual(get_type_hints(BadModule), {}) + + def test_get_type_hints_annotated_bad_module(self): + # See https://bugs.python.org/issue44468 + class BadBase: + foo: tuple + class BadType(BadBase): + bar: list + BadType.__module__ = BadBase.__module__ = 'bad' + self.assertNotIn('bad', sys.modules) + self.assertEqual(get_type_hints(BadType), {'foo': tuple, 'bar': list}) + + def test_forward_ref_and_final(self): + # https://bugs.python.org/issue45166 + hints = get_type_hints(ann_module5) + self.assertEqual(hints, {'name': Final[str]}) + + hints = get_type_hints(ann_module5.MyClass) + self.assertEqual(hints, {'value': Final}) + + def test_top_level_class_var(self): + # This is not meaningful but we don't raise for it. + # https://github.com/python/cpython/issues/133959 + hints = get_type_hints(ann_module6) + self.assertEqual(hints, {'wrong': ClassVar[int]}) + + def test_get_type_hints_typeddict(self): + self.assertEqual(get_type_hints(TotalMovie), {'title': str, 'year': int}) + self.assertEqual(get_type_hints(TotalMovie, include_extras=True), { + 'title': str, + 'year': NotRequired[int], + }) + + self.assertEqual(get_type_hints(AnnotatedMovie), {'title': str, 'year': int}) + self.assertEqual(get_type_hints(AnnotatedMovie, include_extras=True), { + 'title': Annotated[Required[str], "foobar"], + 'year': NotRequired[Annotated[int, 2000]], + }) + + self.assertEqual(get_type_hints(DeeplyAnnotatedMovie), {'title': str, 'year': int}) + self.assertEqual(get_type_hints(DeeplyAnnotatedMovie, include_extras=True), { + 'title': Annotated[Required[str], "foobar", "another level"], + 'year': NotRequired[Annotated[int, 2000]], + }) + + self.assertEqual(get_type_hints(WeirdlyQuotedMovie), {'title': str, 'year': int}) + self.assertEqual(get_type_hints(WeirdlyQuotedMovie, include_extras=True), { + 'title': Annotated[Required[str], "foobar", "another level"], + 'year': NotRequired[Annotated[int, 2000]], + }) + + self.assertEqual(get_type_hints(_typed_dict_helper.VeryAnnotated), {'a': int}) + self.assertEqual(get_type_hints(_typed_dict_helper.VeryAnnotated, include_extras=True), { + 'a': Annotated[Required[int], "a", "b", "c"] + }) + + self.assertEqual(get_type_hints(ChildTotalMovie), {"title": str, "year": int}) + self.assertEqual(get_type_hints(ChildTotalMovie, include_extras=True), { + "title": Required[str], "year": NotRequired[int] + }) + + self.assertEqual(get_type_hints(ChildDeeplyAnnotatedMovie), {"title": str, "year": int}) + self.assertEqual(get_type_hints(ChildDeeplyAnnotatedMovie, include_extras=True), { + "title": Annotated[Required[str], "foobar", "another level"], + "year": NotRequired[Annotated[int, 2000]] + }) + + def test_get_type_hints_collections_abc_callable(self): + # https://github.com/python/cpython/issues/91621 + P = ParamSpec('P') + def f(x: collections.abc.Callable[[int], int]): ... + def g(x: collections.abc.Callable[..., int]): ... + def h(x: collections.abc.Callable[P, int]): ... + + self.assertEqual(get_type_hints(f), {'x': collections.abc.Callable[[int], int]}) + self.assertEqual(get_type_hints(g), {'x': collections.abc.Callable[..., int]}) + self.assertEqual(get_type_hints(h), {'x': collections.abc.Callable[P, int]}) + + def test_get_type_hints_format(self): + class C: + x: undefined + + with self.assertRaises(NameError): + get_type_hints(C) + + with self.assertRaises(NameError): + get_type_hints(C, format=annotationlib.Format.VALUE) + + annos = get_type_hints(C, format=annotationlib.Format.FORWARDREF) + self.assertIsInstance(annos, dict) + self.assertEqual(list(annos), ['x']) + self.assertIsInstance(annos['x'], annotationlib.ForwardRef) + self.assertEqual(annos['x'].__arg__, 'undefined') + + self.assertEqual(get_type_hints(C, format=annotationlib.Format.STRING), + {'x': 'undefined'}) + # Make sure using an int as format also works: + self.assertEqual(get_type_hints(C, format=4), {'x': 'undefined'}) + + def test_get_type_hints_format_function(self): + def func(x: undefined) -> undefined: ... + + # VALUE + with self.assertRaises(NameError): + get_type_hints(func) + with self.assertRaises(NameError): + get_type_hints(func, format=annotationlib.Format.VALUE) + + # FORWARDREF + self.assertEqual( + get_type_hints(func, format=annotationlib.Format.FORWARDREF), + {'x': EqualToForwardRef('undefined', owner=func), + 'return': EqualToForwardRef('undefined', owner=func)}, + ) + + # STRING + self.assertEqual(get_type_hints(func, format=annotationlib.Format.STRING), + {'x': 'undefined', 'return': 'undefined'}) + + def test_callable_with_ellipsis_forward(self): + + def foo(a: 'Callable[..., T]'): + pass + + self.assertEqual(get_type_hints(foo, globals(), locals()), + {'a': Callable[..., T]}) + + def test_special_forms_no_forward(self): + def f(x: ClassVar[int]): + pass + self.assertEqual(get_type_hints(f), {'x': ClassVar[int]}) + + def test_special_forms_forward(self): + + class C: + a: Annotated['ClassVar[int]', (3, 5)] = 4 + b: Annotated['Final[int]', "const"] = 4 + x: 'ClassVar' = 4 + y: 'Final' = 4 + + class CF: + b: List['Final[int]'] = 4 + + self.assertEqual(get_type_hints(C, globals())['a'], ClassVar[int]) + self.assertEqual(get_type_hints(C, globals())['b'], Final[int]) + self.assertEqual(get_type_hints(C, globals())['x'], ClassVar) + self.assertEqual(get_type_hints(C, globals())['y'], Final) + lfi = get_type_hints(CF, globals())['b'] + self.assertIs(get_origin(lfi), list) + self.assertEqual(get_args(lfi), (Final[int],)) + + def test_union_forward_recursion(self): + ValueList = List['Value'] + Value = Union[str, ValueList] + + class C: + foo: List[Value] + class D: + foo: Union[Value, ValueList] + class E: + foo: Union[List[Value], ValueList] + class F: + foo: Union[Value, List[Value], ValueList] + + self.assertEqual(get_type_hints(C, globals(), locals()), get_type_hints(C, globals(), locals())) + self.assertEqual(get_type_hints(C, globals(), locals()), + {'foo': List[Union[str, List[Union[str, List['Value']]]]]}) + self.assertEqual(get_type_hints(D, globals(), locals()), + {'foo': Union[str, List[Union[str, List['Value']]]]}) + self.assertEqual(get_type_hints(E, globals(), locals()), + {'foo': Union[ + List[Union[str, List[Union[str, List['Value']]]]], + List[Union[str, List['Value']]] + ] + }) + self.assertEqual(get_type_hints(F, globals(), locals()), + {'foo': Union[ + str, + List[Union[str, List['Value']]], + List[Union[str, List[Union[str, List['Value']]]]] + ] + }) + + def test_tuple_forward(self): + + def foo(a: Tuple['T']): + pass + + self.assertEqual(get_type_hints(foo, globals(), locals()), + {'a': Tuple[T]}) + + def foo(a: tuple[ForwardRef('T')]): + pass + + self.assertEqual(get_type_hints(foo, globals(), locals()), + {'a': tuple[T]}) + + def test_double_forward(self): + def foo(a: 'List[\'int\']'): + pass + self.assertEqual(get_type_hints(foo, globals(), locals()), + {'a': List[int]}) + + def test_union_forward(self): - def test_get_type_hints_classes_str_annotations(self): - class Foo: - y = str - x: 'y' - # This previously raised an error under PEP 563. - self.assertEqual(get_type_hints(Foo), {'x': str}) + def foo(a: Union['T']): + pass - def test_get_type_hints_bad_module(self): - # bpo-41515 - class BadModule: + self.assertEqual(get_type_hints(foo, globals(), locals()), + {'a': Union[T]}) + + def foo(a: tuple[ForwardRef('T')] | int): pass - BadModule.__module__ = 'bad' # Something not in sys.modules - self.assertNotIn('bad', sys.modules) - self.assertEqual(get_type_hints(BadModule), {}) - def test_get_type_hints_annotated_bad_module(self): - # See https://bugs.python.org/issue44468 - class BadBase: - foo: tuple - class BadType(BadBase): - bar: list - BadType.__module__ = BadBase.__module__ = 'bad' - self.assertNotIn('bad', sys.modules) - self.assertEqual(get_type_hints(BadType), {'foo': tuple, 'bar': list}) + self.assertEqual(get_type_hints(foo, globals(), locals()), + {'a': tuple[T] | int}) - def test_forward_ref_and_final(self): - # https://bugs.python.org/issue45166 - hints = get_type_hints(ann_module5) - self.assertEqual(hints, {'name': Final[str]}) + def test_default_globals(self): + code = ("class C:\n" + " def foo(self, a: 'C') -> 'D': pass\n" + "class D:\n" + " def bar(self, b: 'D') -> C: pass\n" + ) + ns = {} + exec(code, ns) + hints = get_type_hints(ns['C'].foo) + self.assertEqual(hints, {'a': ns['C'], 'return': ns['D']}) - hints = get_type_hints(ann_module5.MyClass) - self.assertEqual(hints, {'value': Final}) + def test_final_forward_ref(self): + gth = get_type_hints + self.assertEqual(gth(Loop, globals())['attr'], Final[Loop]) + self.assertNotEqual(gth(Loop, globals())['attr'], Final[int]) + self.assertNotEqual(gth(Loop, globals())['attr'], Final) - def test_top_level_class_var(self): - # https://bugs.python.org/issue45166 - with self.assertRaisesRegex( - TypeError, - r'typing.ClassVar\[int\] is not valid as type argument', - ): - get_type_hints(ann_module6) + def test_name_error(self): - @unittest.expectedFailure # TODO: RUSTPYTHON - def test_get_type_hints_typeddict(self): - self.assertEqual(get_type_hints(TotalMovie), {'title': str, 'year': int}) - self.assertEqual(get_type_hints(TotalMovie, include_extras=True), { - 'title': str, - 'year': NotRequired[int], - }) + def foo(a: 'Noode[T]'): + pass - self.assertEqual(get_type_hints(AnnotatedMovie), {'title': str, 'year': int}) - self.assertEqual(get_type_hints(AnnotatedMovie, include_extras=True), { - 'title': Annotated[Required[str], "foobar"], - 'year': NotRequired[Annotated[int, 2000]], - }) + with self.assertRaises(NameError): + get_type_hints(foo, locals()) - self.assertEqual(get_type_hints(DeeplyAnnotatedMovie), {'title': str, 'year': int}) - self.assertEqual(get_type_hints(DeeplyAnnotatedMovie, include_extras=True), { - 'title': Annotated[Required[str], "foobar", "another level"], - 'year': NotRequired[Annotated[int, 2000]], - }) + def test_basics(self): - self.assertEqual(get_type_hints(WeirdlyQuotedMovie), {'title': str, 'year': int}) - self.assertEqual(get_type_hints(WeirdlyQuotedMovie, include_extras=True), { - 'title': Annotated[Required[str], "foobar", "another level"], - 'year': NotRequired[Annotated[int, 2000]], - }) + class Node(Generic[T]): - self.assertEqual(get_type_hints(_typed_dict_helper.VeryAnnotated), {'a': int}) - self.assertEqual(get_type_hints(_typed_dict_helper.VeryAnnotated, include_extras=True), { - 'a': Annotated[Required[int], "a", "b", "c"] - }) + def __init__(self, label: T): + self.label = label + self.left = self.right = None - self.assertEqual(get_type_hints(ChildTotalMovie), {"title": str, "year": int}) - self.assertEqual(get_type_hints(ChildTotalMovie, include_extras=True), { - "title": Required[str], "year": NotRequired[int] - }) + def add_both(self, + left: 'Optional[Node[T]]', + right: 'Node[T]' = None, + stuff: int = None, + blah=None): + self.left = left + self.right = right - self.assertEqual(get_type_hints(ChildDeeplyAnnotatedMovie), {"title": str, "year": int}) - self.assertEqual(get_type_hints(ChildDeeplyAnnotatedMovie, include_extras=True), { - "title": Annotated[Required[str], "foobar", "another level"], - "year": NotRequired[Annotated[int, 2000]] - }) + def add_left(self, node: Optional['Node[T]']): + self.add_both(node, None) - def test_get_type_hints_collections_abc_callable(self): - # https://github.com/python/cpython/issues/91621 - P = ParamSpec('P') - def f(x: collections.abc.Callable[[int], int]): ... - def g(x: collections.abc.Callable[..., int]): ... - def h(x: collections.abc.Callable[P, int]): ... + def add_right(self, node: 'Node[T]' = None): + self.add_both(None, node) - self.assertEqual(get_type_hints(f), {'x': collections.abc.Callable[[int], int]}) - self.assertEqual(get_type_hints(g), {'x': collections.abc.Callable[..., int]}) - self.assertEqual(get_type_hints(h), {'x': collections.abc.Callable[P, int]}) + t = Node[int] + both_hints = get_type_hints(t.add_both, globals(), locals()) + self.assertEqual(both_hints['left'], Optional[Node[T]]) + self.assertEqual(both_hints['right'], Node[T]) + self.assertEqual(both_hints['stuff'], int) + self.assertNotIn('blah', both_hints) + + left_hints = get_type_hints(t.add_left, globals(), locals()) + self.assertEqual(left_hints['node'], Optional[Node[T]]) + + right_hints = get_type_hints(t.add_right, globals(), locals()) + self.assertEqual(right_hints['node'], Node[T]) + + def test_stringified_typeddict(self): + ns = run_code( + """ + from __future__ import annotations + from typing import TypedDict + class TD[UniqueT](TypedDict): + a: UniqueT + """ + ) + TD = ns['TD'] + self.assertEqual(TD.__annotations__, {'a': EqualToForwardRef('UniqueT', owner=TD, module=TD.__module__)}) + self.assertEqual(get_type_hints(TD), {'a': TD.__type_params__[0]}) class GetUtilitiesTestCase(TestCase): @@ -7179,7 +7229,7 @@ class C(Generic[T]): pass self.assertIs(get_origin(Callable), collections.abc.Callable) self.assertIs(get_origin(list[int]), list) self.assertIs(get_origin(list), None) - self.assertIs(get_origin(list | str), types.UnionType) + self.assertIs(get_origin(list | str), Union) self.assertIs(get_origin(P.args), P) self.assertIs(get_origin(P.kwargs), P) self.assertIs(get_origin(Required[int]), Required) @@ -7258,6 +7308,125 @@ class C(Generic[T]): pass self.assertEqual(get_args(Unpack[tuple[Unpack[Ts]]]), (tuple[Unpack[Ts]],)) +class EvaluateForwardRefTests(BaseTestCase): + def test_evaluate_forward_ref(self): + int_ref = ForwardRef('int') + self.assertIs(typing.evaluate_forward_ref(int_ref), int) + self.assertIs( + typing.evaluate_forward_ref(int_ref, type_params=()), + int, + ) + self.assertIs( + typing.evaluate_forward_ref(int_ref, format=annotationlib.Format.VALUE), + int, + ) + self.assertIs( + typing.evaluate_forward_ref( + int_ref, format=annotationlib.Format.FORWARDREF, + ), + int, + ) + self.assertEqual( + typing.evaluate_forward_ref( + int_ref, format=annotationlib.Format.STRING, + ), + 'int', + ) + + def test_evaluate_forward_ref_undefined(self): + missing = ForwardRef('missing') + with self.assertRaises(NameError): + typing.evaluate_forward_ref(missing) + self.assertIs( + typing.evaluate_forward_ref( + missing, format=annotationlib.Format.FORWARDREF, + ), + missing, + ) + self.assertEqual( + typing.evaluate_forward_ref( + missing, format=annotationlib.Format.STRING, + ), + "missing", + ) + + def test_evaluate_forward_ref_nested(self): + ref = ForwardRef("int | list['str']") + self.assertEqual( + typing.evaluate_forward_ref(ref), + int | list[str], + ) + self.assertEqual( + typing.evaluate_forward_ref(ref, format=annotationlib.Format.FORWARDREF), + int | list[str], + ) + self.assertEqual( + typing.evaluate_forward_ref(ref, format=annotationlib.Format.STRING), + "int | list['str']", + ) + + why = ForwardRef('"\'str\'"') + self.assertIs(typing.evaluate_forward_ref(why), str) + + def test_evaluate_forward_ref_none(self): + none_ref = ForwardRef('None') + self.assertIs(typing.evaluate_forward_ref(none_ref), None) + + def test_globals(self): + A = "str" + ref = ForwardRef('list[A]') + with self.assertRaises(NameError): + typing.evaluate_forward_ref(ref) + self.assertEqual( + typing.evaluate_forward_ref(ref, globals={'A': A}), + list[str], + ) + + def test_owner(self): + ref = ForwardRef("A") + + with self.assertRaises(NameError): + typing.evaluate_forward_ref(ref) + + # We default to the globals of `owner`, + # so it no longer raises `NameError` + self.assertIs( + typing.evaluate_forward_ref(ref, owner=Loop), A + ) + + def test_inherited_owner(self): + # owner passed to evaluate_forward_ref + ref = ForwardRef("list['A']") + self.assertEqual( + typing.evaluate_forward_ref(ref, owner=Loop), + list[A], + ) + + # owner set on the ForwardRef + ref = ForwardRef("list['A']", owner=Loop) + self.assertEqual( + typing.evaluate_forward_ref(ref), + list[A], + ) + + def test_partial_evaluation(self): + ref = ForwardRef("list[A]") + with self.assertRaises(NameError): + typing.evaluate_forward_ref(ref) + + self.assertEqual( + typing.evaluate_forward_ref(ref, format=annotationlib.Format.FORWARDREF), + list[EqualToForwardRef('A')], + ) + + @unittest.expectedFailure # TODO: RUSTPYTHON; ImportError: cannot import name 'fwdref_module' + def test_with_module(self): + from test.typinganndata import fwdref_module + + typing.evaluate_forward_ref( + fwdref_module.fw,) + + class CollectionsAbcTests(BaseTestCase): def test_hashable(self): @@ -7985,6 +8154,48 @@ class XMethBad2(NamedTuple): def _source(self): return 'no chance for this as well' + def test_annotation_type_check(self): + # These are rejected by _type_check + with self.assertRaises(TypeError): + class X(NamedTuple): + a: Final + with self.assertRaises(TypeError): + class Y(NamedTuple): + a: (1, 2) + + # Conversion by _type_convert + class Z(NamedTuple): + a: None + b: "str" + annos = {'a': type(None), 'b': EqualToForwardRef("str")} + self.assertEqual(Z.__annotations__, annos) + self.assertEqual(Z.__annotate__(annotationlib.Format.VALUE), annos) + self.assertEqual(Z.__annotate__(annotationlib.Format.FORWARDREF), annos) + self.assertEqual(Z.__annotate__(annotationlib.Format.STRING), {"a": "None", "b": "str"}) + + def test_future_annotations(self): + code = """ + from __future__ import annotations + from typing import NamedTuple + class X(NamedTuple): + a: int + b: None + """ + ns = run_code(textwrap.dedent(code)) + X = ns['X'] + self.assertEqual(X.__annotations__, {'a': EqualToForwardRef("int"), 'b': EqualToForwardRef("None")}) + + def test_deferred_annotations(self): + class X(NamedTuple): + y: undefined + + self.assertEqual(X._fields, ('y',)) + with self.assertRaises(NameError): + X.__annotations__ + + undefined = int + self.assertEqual(X.__annotations__, {'y': int}) + def test_multiple_inheritance(self): class A: pass @@ -8231,7 +8442,7 @@ class Bar(NamedTuple): self.assertIsInstance(bar.attr, Vanilla) self.assertEqual(bar.attr.name, "attr") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_setname_raises_the_same_as_on_other_classes(self): class CustomException(BaseException): pass @@ -8286,6 +8497,23 @@ class VeryAnnoying(metaclass=Meta): pass class Foo(NamedTuple): attr = very_annoying + def test_super_explicitly_disallowed(self): + expected_message = ( + "uses of super() and __class__ are unsupported " + "in methods of NamedTuple subclasses" + ) + + with self.assertRaises(TypeError, msg=expected_message): + class ThisWontWork(NamedTuple): + def __repr__(self): + return super().__repr__() + + with self.assertRaises(TypeError, msg=expected_message): + class ThisWontWorkEither(NamedTuple): + @property + def name(self): + return __class__.__name__ + class TypedDictTests(BaseTestCase): def test_basics_functional_syntax(self): @@ -8300,7 +8528,11 @@ def test_basics_functional_syntax(self): self.assertEqual(Emp.__name__, 'Emp') self.assertEqual(Emp.__module__, __name__) self.assertEqual(Emp.__bases__, (dict,)) - self.assertEqual(Emp.__annotations__, {'name': str, 'id': int}) + annos = {'name': str, 'id': int} + self.assertEqual(Emp.__annotations__, annos) + self.assertEqual(Emp.__annotate__(annotationlib.Format.VALUE), annos) + self.assertEqual(Emp.__annotate__(annotationlib.Format.FORWARDREF), annos) + self.assertEqual(Emp.__annotate__(annotationlib.Format.STRING), {'name': 'str', 'id': 'int'}) self.assertEqual(Emp.__total__, True) self.assertEqual(Emp.__required_keys__, {'name', 'id'}) self.assertIsInstance(Emp.__required_keys__, frozenset) @@ -8499,6 +8731,36 @@ class Child(Base1, Base2): self.assertEqual(Child.__required_keys__, frozenset(['a'])) self.assertEqual(Child.__optional_keys__, frozenset()) + def test_inheritance_pep563(self): + def _make_td(future, class_name, annos, base, extra_names=None): + lines = [] + if future: + lines.append('from __future__ import annotations') + lines.append('from typing import TypedDict') + lines.append(f'class {class_name}({base}):') + for name, anno in annos.items(): + lines.append(f' {name}: {anno}') + code = '\n'.join(lines) + ns = run_code(code, extra_names) + return ns[class_name] + + for base_future in (True, False): + for child_future in (True, False): + with self.subTest(base_future=base_future, child_future=child_future): + base = _make_td( + base_future, "Base", {"base": "int"}, "TypedDict" + ) + self.assertIsNotNone(base.__annotate__) + child = _make_td( + child_future, "Child", {"child": "int"}, "Base", {"Base": base} + ) + base_anno = ForwardRef("int", module="builtins", owner=base) if base_future else int + child_anno = ForwardRef("int", module="builtins", owner=child) if child_future else int + self.assertEqual(base.__annotations__, {'base': base_anno}) + self.assertEqual( + child.__annotations__, {'child': child_anno, 'base': base_anno} + ) + def test_required_notrequired_keys(self): self.assertEqual(NontotalMovie.__required_keys__, frozenset({"title"})) @@ -8647,14 +8909,14 @@ class NewGeneric[T](TypedDict): # The TypedDict constructor is not itself a TypedDict self.assertIs(is_typeddict(TypedDict), False) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_get_type_hints(self): self.assertEqual( get_type_hints(Bar), {'a': typing.Optional[int], 'b': int} ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_get_type_hints_generic(self): self.assertEqual( get_type_hints(BarGeneric), @@ -8679,6 +8941,8 @@ class A[T](TypedDict): self.assertEqual(A.__bases__, (Generic, dict)) self.assertEqual(A.__orig_bases__, (TypedDict, Generic[T])) self.assertEqual(A.__mro__, (A, Generic, dict, object)) + self.assertEqual(A.__annotations__, {'a': T}) + self.assertEqual(A.__annotate__(annotationlib.Format.STRING), {'a': 'T'}) self.assertEqual(A.__parameters__, (T,)) self.assertEqual(A[str].__parameters__, ()) self.assertEqual(A[str].__args__, (str,)) @@ -8690,6 +8954,8 @@ class A(TypedDict, Generic[T]): self.assertEqual(A.__bases__, (Generic, dict)) self.assertEqual(A.__orig_bases__, (TypedDict, Generic[T])) self.assertEqual(A.__mro__, (A, Generic, dict, object)) + self.assertEqual(A.__annotations__, {'a': T}) + self.assertEqual(A.__annotate__(annotationlib.Format.STRING), {'a': 'T'}) self.assertEqual(A.__parameters__, (T,)) self.assertEqual(A[str].__parameters__, ()) self.assertEqual(A[str].__args__, (str,)) @@ -8700,6 +8966,8 @@ class A2(Generic[T], TypedDict): self.assertEqual(A2.__bases__, (Generic, dict)) self.assertEqual(A2.__orig_bases__, (Generic[T], TypedDict)) self.assertEqual(A2.__mro__, (A2, Generic, dict, object)) + self.assertEqual(A2.__annotations__, {'a': T}) + self.assertEqual(A2.__annotate__(annotationlib.Format.STRING), {'a': 'T'}) self.assertEqual(A2.__parameters__, (T,)) self.assertEqual(A2[str].__parameters__, ()) self.assertEqual(A2[str].__args__, (str,)) @@ -8710,6 +8978,8 @@ class B(A[KT], total=False): self.assertEqual(B.__bases__, (Generic, dict)) self.assertEqual(B.__orig_bases__, (A[KT],)) self.assertEqual(B.__mro__, (B, Generic, dict, object)) + self.assertEqual(B.__annotations__, {'a': T, 'b': KT}) + self.assertEqual(B.__annotate__(annotationlib.Format.STRING), {'a': 'T', 'b': 'KT'}) self.assertEqual(B.__parameters__, (KT,)) self.assertEqual(B.__total__, False) self.assertEqual(B.__optional_keys__, frozenset(['b'])) @@ -8734,6 +9004,11 @@ class C(B[int]): 'b': KT, 'c': int, }) + self.assertEqual(C.__annotate__(annotationlib.Format.STRING), { + 'a': 'T', + 'b': 'KT', + 'c': 'int', + }) with self.assertRaises(TypeError): C[str] @@ -8753,6 +9028,11 @@ class Point3D(Point2DGeneric[T], Generic[T, KT]): 'b': T, 'c': KT, }) + self.assertEqual(Point3D.__annotate__(annotationlib.Format.STRING), { + 'a': 'T', + 'b': 'T', + 'c': 'KT', + }) self.assertEqual(Point3D[int, str].__origin__, Point3D) with self.assertRaises(TypeError): @@ -8784,10 +9064,15 @@ class WithImplicitAny(B): 'b': KT, 'c': int, }) + self.assertEqual(WithImplicitAny.__annotate__(annotationlib.Format.STRING), { + 'a': 'T', + 'b': 'KT', + 'c': 'int', + }) with self.assertRaises(TypeError): WithImplicitAny[str] - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_non_generic_subscript(self): # For backward compatibility, subscription works # on arbitrary TypedDict types. @@ -8915,7 +9200,7 @@ class Child(Base): self.assertEqual(Child.__readonly_keys__, frozenset()) self.assertEqual(Child.__mutable_keys__, frozenset({'a'})) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_combine_qualifiers(self): class AllTheThings(TypedDict): a: Annotated[Required[ReadOnly[int]], "why not"] @@ -8942,6 +9227,54 @@ class AllTheThings(TypedDict): }, ) + def test_annotations(self): + # _type_check is applied + with self.assertRaisesRegex(TypeError, "Plain typing.Final is not valid as type argument"): + class X(TypedDict): + a: Final + + # _type_convert is applied + class Y(TypedDict): + a: None + b: "int" + fwdref = EqualToForwardRef('int', module=__name__) + self.assertEqual(Y.__annotations__, {'a': type(None), 'b': fwdref}) + self.assertEqual(Y.__annotate__(annotationlib.Format.FORWARDREF), {'a': type(None), 'b': fwdref}) + + # _type_check is also applied later + class Z(TypedDict): + a: undefined + + with self.assertRaises(NameError): + Z.__annotations__ + + undefined = Final + with self.assertRaisesRegex(TypeError, "Plain typing.Final is not valid as type argument"): + Z.__annotations__ + + undefined = None + self.assertEqual(Z.__annotations__, {'a': type(None)}) + + def test_deferred_evaluation(self): + class A(TypedDict): + x: NotRequired[undefined] + y: ReadOnly[undefined] + z: Required[undefined] + + self.assertEqual(A.__required_keys__, frozenset({'y', 'z'})) + self.assertEqual(A.__optional_keys__, frozenset({'x'})) + self.assertEqual(A.__readonly_keys__, frozenset({'y'})) + self.assertEqual(A.__mutable_keys__, frozenset({'x', 'z'})) + + with self.assertRaises(NameError): + A.__annotations__ + + self.assertEqual( + A.__annotate__(annotationlib.Format.STRING), + {'x': 'NotRequired[undefined]', 'y': 'ReadOnly[undefined]', + 'z': 'Required[undefined]'}, + ) + class RequiredTests(BaseTestCase): @@ -9109,7 +9442,7 @@ def test_repr(self): self.assertEqual(repr(Match[str]), 'typing.Match[str]') self.assertEqual(repr(Match[bytes]), 'typing.Match[bytes]') - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_cannot_subclass(self): with self.assertRaisesRegex( TypeError, @@ -9586,6 +9919,19 @@ class B(str): ... self.assertIs(type(field_c2.__metadata__[0]), float) self.assertIs(type(field_c3.__metadata__[0]), bool) + def test_forwardref_partial_evaluation(self): + # Test that Annotated partially evaluates if it contains a ForwardRef + # See: https://github.com/python/cpython/issues/137706 + def f(x: Annotated[undefined, '']): pass + + ann = annotationlib.get_annotations(f, format=annotationlib.Format.FORWARDREF) + + # Test that the attributes are retrievable from the partially evaluated annotation + x_ann = ann['x'] + self.assertIs(get_origin(x_ann), Annotated) + self.assertEqual(x_ann.__origin__, EqualToForwardRef('undefined', owner=f)) + self.assertEqual(x_ann.__metadata__, ('',)) + class TypeAliasTests(BaseTestCase): def test_canonical_usage_with_variable_annotation(self): @@ -10092,6 +10438,7 @@ def test_var_substitution(self): self.assertEqual(C[Concatenate[str, P2]], Concatenate[int, str, P2]) self.assertEqual(C[...], Concatenate[int, ...]) + class TypeGuardTests(BaseTestCase): def test_basics(self): TypeGuard[int] # OK @@ -10283,7 +10630,6 @@ def test_special_attrs(self): typing.ClassVar: 'ClassVar', typing.Concatenate: 'Concatenate', typing.Final: 'Final', - typing.ForwardRef: 'ForwardRef', typing.Literal: 'Literal', typing.NewType: 'NewType', typing.NoReturn: 'NoReturn', @@ -10293,9 +10639,8 @@ def test_special_attrs(self): typing.TypeGuard: 'TypeGuard', typing.TypeIs: 'TypeIs', typing.TypeVar: 'TypeVar', - typing.Union: 'Union', typing.Self: 'Self', - # Subscribed special forms + # Subscripted special forms typing.Annotated[Any, "Annotation"]: 'Annotated', typing.Annotated[int, 'Annotation']: 'Annotated', typing.ClassVar[Any]: 'ClassVar', @@ -10304,13 +10649,12 @@ def test_special_attrs(self): typing.Literal[Any]: 'Literal', typing.Literal[1, 2]: 'Literal', typing.Literal[True, 2]: 'Literal', - typing.Optional[Any]: 'Optional', + typing.Optional[Any]: 'Union', typing.TypeGuard[Any]: 'TypeGuard', typing.TypeIs[Any]: 'TypeIs', typing.Union[Any]: 'Any', typing.Union[int, float]: 'Union', # Incompatible special forms (tested in test_special_attrs2) - # - typing.ForwardRef('set[Any]') # - typing.NewType('TypeName', Any) # - typing.ParamSpec('SpecialAttrsP') # - typing.TypeVar('T') @@ -10324,24 +10668,14 @@ def test_special_attrs(self): for proto in range(pickle.HIGHEST_PROTOCOL + 1): s = pickle.dumps(cls, proto) loaded = pickle.loads(s) - self.assertIs(cls, loaded) + if isinstance(cls, Union): + self.assertEqual(cls, loaded) + else: + self.assertIs(cls, loaded) TypeName = typing.NewType('SpecialAttrsTests.TypeName', Any) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_special_attrs2(self): - # Forward refs provide a different introspection API. __name__ and - # __qualname__ make little sense for forward refs as they can store - # complex typing expressions. - fr = typing.ForwardRef('set[Any]') - self.assertFalse(hasattr(fr, '__name__')) - self.assertFalse(hasattr(fr, '__qualname__')) - self.assertEqual(fr.__module__, 'typing') - # Forward refs are currently unpicklable. - for proto in range(pickle.HIGHEST_PROTOCOL + 1): - with self.assertRaises(TypeError): - pickle.dumps(fr, proto) - self.assertEqual(SpecialAttrsTests.TypeName.__name__, 'TypeName') self.assertEqual( SpecialAttrsTests.TypeName.__qualname__, @@ -10362,7 +10696,7 @@ def test_special_attrs2(self): # to the variable name to which it is assigned". Thus, providing # __qualname__ is unnecessary. self.assertEqual(SpecialAttrsT.__name__, 'SpecialAttrsT') - self.assertFalse(hasattr(SpecialAttrsT, '__qualname__')) + self.assertNotHasAttr(SpecialAttrsT, '__qualname__') self.assertEqual(SpecialAttrsT.__module__, __name__) # Module-level type variables are picklable. for proto in range(pickle.HIGHEST_PROTOCOL + 1): @@ -10371,7 +10705,7 @@ def test_special_attrs2(self): self.assertIs(SpecialAttrsT, loaded) self.assertEqual(SpecialAttrsP.__name__, 'SpecialAttrsP') - self.assertFalse(hasattr(SpecialAttrsP, '__qualname__')) + self.assertNotHasAttr(SpecialAttrsP, '__qualname__') self.assertEqual(SpecialAttrsP.__module__, __name__) # Module-level ParamSpecs are picklable. for proto in range(pickle.HIGHEST_PROTOCOL + 1): @@ -10520,7 +10854,7 @@ def test_no_call(self): with self.assertRaises(TypeError): NoDefault() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_no_attributes(self): with self.assertRaises(AttributeError): NoDefault.foo = 3 @@ -10596,7 +10930,7 @@ class TypeIterationTests(BaseTestCase): Annotated[T, ''], ) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_cannot_iterate(self): expected_error_regex = "object is not iterable" for test_type in self._UNITERABLE_TYPES: @@ -10614,6 +10948,37 @@ def test_is_not_instance_of_iterable(self): self.assertNotIsInstance(type_to_test, collections.abc.Iterable) +class UnionGenericAliasTests(BaseTestCase): + def test_constructor(self): + # Used e.g. in typer, pydantic + with self.assertWarns(DeprecationWarning): + inst = typing._UnionGenericAlias(typing.Union, (int, str)) + self.assertEqual(inst, int | str) + with self.assertWarns(DeprecationWarning): + # name is accepted but ignored + inst = typing._UnionGenericAlias(typing.Union, (int, None), name="Optional") + self.assertEqual(inst, int | None) + + def test_isinstance(self): + # Used e.g. in pydantic + with self.assertWarns(DeprecationWarning): + self.assertTrue(isinstance(Union[int, str], typing._UnionGenericAlias)) + with self.assertWarns(DeprecationWarning): + self.assertFalse(isinstance(int, typing._UnionGenericAlias)) + + def test_eq(self): + # type(t) == _UnionGenericAlias is used in vyos + with self.assertWarns(DeprecationWarning): + self.assertEqual(Union, typing._UnionGenericAlias) + with self.assertWarns(DeprecationWarning): + self.assertEqual(typing._UnionGenericAlias, typing._UnionGenericAlias) + with self.assertWarns(DeprecationWarning): + self.assertNotEqual(int, typing._UnionGenericAlias) + + def test_hashable(self): + self.assertEqual(hash(typing._UnionGenericAlias), hash(Union)) + + def load_tests(loader, tests, pattern): import doctest tests.addTests(doctest.DocTestSuite(typing)) diff --git a/Lib/test/test_unittest/__init__.py b/Lib/test/test_unittest/__init__.py index bc502ef32d2..365f26d6438 100644 --- a/Lib/test/test_unittest/__init__.py +++ b/Lib/test/test_unittest/__init__.py @@ -1,4 +1,5 @@ import os.path + from test.support import load_package_tests diff --git a/Lib/test/test_unittest/__main__.py b/Lib/test/test_unittest/__main__.py index 40a23a297ec..0d53bfab847 100644 --- a/Lib/test/test_unittest/__main__.py +++ b/Lib/test/test_unittest/__main__.py @@ -1,4 +1,5 @@ -from . import load_tests import unittest +from . import load_tests + unittest.main() diff --git a/Lib/test/test_unittest/_test_warnings.py b/Lib/test/test_unittest/_test_warnings.py index 08b846ee47e..d9f41a4144b 100644 --- a/Lib/test/test_unittest/_test_warnings.py +++ b/Lib/test/test_unittest/_test_warnings.py @@ -14,6 +14,7 @@ import unittest import warnings + def warnfun(): warnings.warn('rw', RuntimeWarning) diff --git a/Lib/test/test_unittest/test_assertions.py b/Lib/test/test_unittest/test_assertions.py index 1dec947ea76..3d782573d7b 100644 --- a/Lib/test/test_unittest/test_assertions.py +++ b/Lib/test/test_unittest/test_assertions.py @@ -1,10 +1,11 @@ import datetime +import unittest import warnings import weakref -import unittest -from test.support import gc_collect from itertools import product +from test.support import gc_collect + class Test_Assertions(unittest.TestCase): def test_AlmostEqual(self): diff --git a/Lib/test/test_unittest/test_async_case.py b/Lib/test/test_unittest/test_async_case.py index f77a5888eeb..57228e78f8c 100644 --- a/Lib/test/test_unittest/test_async_case.py +++ b/Lib/test/test_unittest/test_async_case.py @@ -1,7 +1,9 @@ import asyncio import contextvars import unittest + from test import support +from test.support import force_not_colorized support.requires_working_socket(module=True) @@ -11,7 +13,7 @@ class MyException(Exception): def tearDownModule(): - asyncio.set_event_loop_policy(None) + asyncio.events._set_event_loop_policy(None) class TestCM: @@ -48,7 +50,6 @@ def setUp(self): # starting a new event loop self.addCleanup(support.gc_collect) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_full_cycle(self): expected = ['setUp', 'asyncSetUp', @@ -253,6 +254,7 @@ async def on_cleanup(self): test.doCleanups() self.assertEqual(events, ['asyncSetUp', 'test', 'asyncTearDown', 'cleanup']) + @force_not_colorized def test_exception_in_tear_clean_up(self): class Test(unittest.IsolatedAsyncioTestCase): async def asyncSetUp(self): @@ -314,18 +316,21 @@ async def test3(self): self.assertIn('It is deprecated to return a value that is not None', str(w.warning)) self.assertIn('test1', str(w.warning)) self.assertEqual(w.filename, __file__) + self.assertIn("returned 'int'", str(w.warning)) with self.assertWarns(DeprecationWarning) as w: Test('test2').run() self.assertIn('It is deprecated to return a value that is not None', str(w.warning)) self.assertIn('test2', str(w.warning)) self.assertEqual(w.filename, __file__) + self.assertIn("returned 'async_generator'", str(w.warning)) with self.assertWarns(DeprecationWarning) as w: Test('test3').run() self.assertIn('It is deprecated to return a value that is not None', str(w.warning)) self.assertIn('test3', str(w.warning)) self.assertEqual(w.filename, __file__) + self.assertIn(f'returned {Nothing.__name__!r}', str(w.warning)) def test_cleanups_interleave_order(self): events = [] @@ -477,7 +482,7 @@ def test_setup_get_event_loop(self): class TestCase1(unittest.IsolatedAsyncioTestCase): def setUp(self): - asyncio.get_event_loop_policy().get_event_loop() + asyncio.events._get_event_loop_policy().get_event_loop() async def test_demo1(self): pass @@ -486,9 +491,8 @@ async def test_demo1(self): result = test.run() self.assertTrue(result.wasSuccessful()) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_loop_factory(self): - asyncio.set_event_loop_policy(None) + asyncio.events._set_event_loop_policy(None) class TestCase1(unittest.IsolatedAsyncioTestCase): loop_factory = asyncio.EventLoop diff --git a/Lib/test/test_unittest/test_break.py b/Lib/test/test_unittest/test_break.py index 1da98af3e74..8aa20008ac7 100644 --- a/Lib/test/test_unittest/test_break.py +++ b/Lib/test/test_unittest/test_break.py @@ -1,10 +1,10 @@ import gc import io import os -import sys import signal -import weakref +import sys import unittest +import weakref from test import support diff --git a/Lib/test/test_unittest/test_case.py b/Lib/test/test_unittest/test_case.py index 82a442a04e6..6e77040c265 100644 --- a/Lib/test/test_unittest/test_case.py +++ b/Lib/test/test_unittest/test_case.py @@ -1,26 +1,27 @@ import contextlib import difflib -import pprint +import inspect +import logging import pickle +import pprint import re import sys -import logging +import types +import unittest import warnings import weakref -import inspect -import types - +from collections import UserString from copy import deepcopy -from test import support - -import unittest +from test import support +from test.support import captured_stderr, gc_collect from test.test_unittest.support import ( - TestEquality, TestHashing, LoggingResult, LegacyLoggingResult, - ResultWithNoStartTestRunStopTestRun + LegacyLoggingResult, + LoggingResult, + ResultWithNoStartTestRunStopTestRun, + TestEquality, + TestHashing, ) -from test.support import captured_stderr, gc_collect - log_foo = logging.getLogger('foo') log_foobar = logging.getLogger('foo.bar') @@ -54,6 +55,10 @@ def tearDown(self): self.events.append('tearDown') +class List(list): + pass + + class Test_TestCase(unittest.TestCase, TestEquality, TestHashing): ### Set up attributes used by inherited tests @@ -85,7 +90,7 @@ class Test(unittest.TestCase): def runTest(self): raise MyException() def test(self): pass - self.assertEqual(Test().id()[-13:], '.Test.runTest') + self.assertEndsWith(Test().id(), '.Test.runTest') # test that TestCase can be instantiated with no args # primarily for use at the interactive interpreter @@ -106,7 +111,7 @@ class Test(unittest.TestCase): def runTest(self): raise MyException() def test(self): pass - self.assertEqual(Test('test').id()[-10:], '.Test.test') + self.assertEndsWith(Test('test').id(), '.Test.test') # "class TestCase([methodName])" # ... @@ -325,18 +330,40 @@ def test3(self): self.assertIn('It is deprecated to return a value that is not None', str(w.warning)) self.assertIn('test1', str(w.warning)) self.assertEqual(w.filename, __file__) + self.assertIn("returned 'int'", str(w.warning)) with self.assertWarns(DeprecationWarning) as w: Foo('test2').run() self.assertIn('It is deprecated to return a value that is not None', str(w.warning)) self.assertIn('test2', str(w.warning)) self.assertEqual(w.filename, __file__) + self.assertIn("returned 'generator'", str(w.warning)) with self.assertWarns(DeprecationWarning) as w: Foo('test3').run() self.assertIn('It is deprecated to return a value that is not None', str(w.warning)) self.assertIn('test3', str(w.warning)) self.assertEqual(w.filename, __file__) + self.assertIn(f'returned {Nothing.__name__!r}', str(w.warning)) + + def test_deprecation_of_return_val_from_test_async_method(self): + class Foo(unittest.TestCase): + async def test1(self): + return 1 + + with self.assertWarns(DeprecationWarning) as w: + warnings.filterwarnings('ignore', + 'coroutine .* was never awaited', RuntimeWarning) + Foo('test1').run() + support.gc_collect() + self.assertIn('It is deprecated to return a value that is not None', str(w.warning)) + self.assertIn('test1', str(w.warning)) + self.assertEqual(w.filename, __file__) + self.assertIn("returned 'coroutine'", str(w.warning)) + self.assertIn( + 'Maybe you forgot to use IsolatedAsyncioTestCase as the base class?', + str(w.warning), + ) def _check_call_order__subtests(self, result, events, expected_events): class Foo(Test.LoggingTestCase): @@ -678,16 +705,136 @@ def testAssertIsNot(self): self.assertRaises(self.failureException, self.assertIsNot, thing, thing) def testAssertIsInstance(self): - thing = [] + thing = List() self.assertIsInstance(thing, list) - self.assertRaises(self.failureException, self.assertIsInstance, - thing, dict) + self.assertIsInstance(thing, (int, list)) + with self.assertRaises(self.failureException) as cm: + self.assertIsInstance(thing, int) + self.assertEqual(str(cm.exception), + "[] is not an instance of <class 'int'>") + with self.assertRaises(self.failureException) as cm: + self.assertIsInstance(thing, (int, float)) + self.assertEqual(str(cm.exception), + "[] is not an instance of any of (<class 'int'>, <class 'float'>)") + + with self.assertRaises(self.failureException) as cm: + self.assertIsInstance(thing, int, 'ababahalamaha') + self.assertIn('ababahalamaha', str(cm.exception)) + with self.assertRaises(self.failureException) as cm: + self.assertIsInstance(thing, int, msg='ababahalamaha') + self.assertIn('ababahalamaha', str(cm.exception)) def testAssertNotIsInstance(self): - thing = [] - self.assertNotIsInstance(thing, dict) - self.assertRaises(self.failureException, self.assertNotIsInstance, - thing, list) + thing = List() + self.assertNotIsInstance(thing, int) + self.assertNotIsInstance(thing, (int, float)) + with self.assertRaises(self.failureException) as cm: + self.assertNotIsInstance(thing, list) + self.assertEqual(str(cm.exception), + "[] is an instance of <class 'list'>") + with self.assertRaises(self.failureException) as cm: + self.assertNotIsInstance(thing, (int, list)) + self.assertEqual(str(cm.exception), + "[] is an instance of <class 'list'>") + + with self.assertRaises(self.failureException) as cm: + self.assertNotIsInstance(thing, list, 'ababahalamaha') + self.assertIn('ababahalamaha', str(cm.exception)) + with self.assertRaises(self.failureException) as cm: + self.assertNotIsInstance(thing, list, msg='ababahalamaha') + self.assertIn('ababahalamaha', str(cm.exception)) + + def testAssertIsSubclass(self): + self.assertIsSubclass(List, list) + self.assertIsSubclass(List, (int, list)) + with self.assertRaises(self.failureException) as cm: + self.assertIsSubclass(List, int) + self.assertEqual(str(cm.exception), + f"{List!r} is not a subclass of <class 'int'>") + with self.assertRaises(self.failureException) as cm: + self.assertIsSubclass(List, (int, float)) + self.assertEqual(str(cm.exception), + f"{List!r} is not a subclass of any of (<class 'int'>, <class 'float'>)") + with self.assertRaises(self.failureException) as cm: + self.assertIsSubclass(1, int) + self.assertEqual(str(cm.exception), "1 is not a class") + + with self.assertRaises(self.failureException) as cm: + self.assertIsSubclass(List, int, 'ababahalamaha') + self.assertIn('ababahalamaha', str(cm.exception)) + with self.assertRaises(self.failureException) as cm: + self.assertIsSubclass(List, int, msg='ababahalamaha') + self.assertIn('ababahalamaha', str(cm.exception)) + + def testAssertNotIsSubclass(self): + self.assertNotIsSubclass(List, int) + self.assertNotIsSubclass(List, (int, float)) + with self.assertRaises(self.failureException) as cm: + self.assertNotIsSubclass(List, list) + self.assertEqual(str(cm.exception), + f"{List!r} is a subclass of <class 'list'>") + with self.assertRaises(self.failureException) as cm: + self.assertNotIsSubclass(List, (int, list)) + self.assertEqual(str(cm.exception), + f"{List!r} is a subclass of <class 'list'>") + with self.assertRaises(self.failureException) as cm: + self.assertNotIsSubclass(1, int) + self.assertEqual(str(cm.exception), "1 is not a class") + + with self.assertRaises(self.failureException) as cm: + self.assertNotIsSubclass(List, list, 'ababahalamaha') + self.assertIn('ababahalamaha', str(cm.exception)) + with self.assertRaises(self.failureException) as cm: + self.assertNotIsSubclass(List, list, msg='ababahalamaha') + self.assertIn('ababahalamaha', str(cm.exception)) + + def testAssertHasAttr(self): + a = List() + a.x = 1 + self.assertHasAttr(a, 'x') + with self.assertRaises(self.failureException) as cm: + self.assertHasAttr(a, 'y') + self.assertEqual(str(cm.exception), + "'List' object has no attribute 'y'") + with self.assertRaises(self.failureException) as cm: + self.assertHasAttr(List, 'spam') + self.assertEqual(str(cm.exception), + "type object 'List' has no attribute 'spam'") + with self.assertRaises(self.failureException) as cm: + self.assertHasAttr(sys, 'nonexistent') + self.assertEqual(str(cm.exception), + "module 'sys' has no attribute 'nonexistent'") + + with self.assertRaises(self.failureException) as cm: + self.assertHasAttr(a, 'y', 'ababahalamaha') + self.assertIn('ababahalamaha', str(cm.exception)) + with self.assertRaises(self.failureException) as cm: + self.assertHasAttr(a, 'y', msg='ababahalamaha') + self.assertIn('ababahalamaha', str(cm.exception)) + + def testAssertNotHasAttr(self): + a = List() + a.x = 1 + self.assertNotHasAttr(a, 'y') + with self.assertRaises(self.failureException) as cm: + self.assertNotHasAttr(a, 'x') + self.assertEqual(str(cm.exception), + "'List' object has unexpected attribute 'x'") + with self.assertRaises(self.failureException) as cm: + self.assertNotHasAttr(List, 'append') + self.assertEqual(str(cm.exception), + "type object 'List' has unexpected attribute 'append'") + with self.assertRaises(self.failureException) as cm: + self.assertNotHasAttr(sys, 'modules') + self.assertEqual(str(cm.exception), + "module 'sys' has unexpected attribute 'modules'") + + with self.assertRaises(self.failureException) as cm: + self.assertNotHasAttr(a, 'x', 'ababahalamaha') + self.assertIn('ababahalamaha', str(cm.exception)) + with self.assertRaises(self.failureException) as cm: + self.assertNotHasAttr(a, 'x', msg='ababahalamaha') + self.assertIn('ababahalamaha', str(cm.exception)) def testAssertIn(self): animals = {'monkey': 'banana', 'cow': 'grass', 'seal': 'fish'} @@ -1842,6 +1989,186 @@ def testAssertNoLogsYieldsNone(self): pass self.assertIsNone(value) + def testAssertStartsWith(self): + self.assertStartsWith('ababahalamaha', 'ababa') + self.assertStartsWith('ababahalamaha', ('x', 'ababa', 'y')) + self.assertStartsWith(UserString('ababahalamaha'), 'ababa') + self.assertStartsWith(UserString('ababahalamaha'), ('x', 'ababa', 'y')) + self.assertStartsWith(bytearray(b'ababahalamaha'), b'ababa') + self.assertStartsWith(bytearray(b'ababahalamaha'), (b'x', b'ababa', b'y')) + self.assertStartsWith(b'ababahalamaha', bytearray(b'ababa')) + self.assertStartsWith(b'ababahalamaha', + (bytearray(b'x'), bytearray(b'ababa'), bytearray(b'y'))) + + with self.assertRaises(self.failureException) as cm: + self.assertStartsWith('ababahalamaha', 'amaha') + self.assertEqual(str(cm.exception), + "'ababahalamaha' doesn't start with 'amaha'") + with self.assertRaises(self.failureException) as cm: + self.assertStartsWith('ababahalamaha', ('x', 'y')) + self.assertEqual(str(cm.exception), + "'ababahalamaha' doesn't start with any of ('x', 'y')") + + with self.assertRaises(self.failureException) as cm: + self.assertStartsWith(b'ababahalamaha', 'ababa') + self.assertEqual(str(cm.exception), 'Expected str, not bytes') + with self.assertRaises(self.failureException) as cm: + self.assertStartsWith(b'ababahalamaha', ('amaha', 'ababa')) + self.assertEqual(str(cm.exception), 'Expected str, not bytes') + with self.assertRaises(self.failureException) as cm: + self.assertStartsWith([], 'ababa') + self.assertEqual(str(cm.exception), 'Expected str, not list') + with self.assertRaises(self.failureException) as cm: + self.assertStartsWith('ababahalamaha', b'ababa') + self.assertEqual(str(cm.exception), 'Expected bytes, not str') + with self.assertRaises(self.failureException) as cm: + self.assertStartsWith('ababahalamaha', (b'amaha', b'ababa')) + self.assertEqual(str(cm.exception), 'Expected bytes, not str') + with self.assertRaises(TypeError): + self.assertStartsWith('ababahalamaha', ord('a')) + + with self.assertRaises(self.failureException) as cm: + self.assertStartsWith('ababahalamaha', 'amaha', 'abracadabra') + self.assertIn('ababahalamaha', str(cm.exception)) + with self.assertRaises(self.failureException) as cm: + self.assertStartsWith('ababahalamaha', 'amaha', msg='abracadabra') + self.assertIn('ababahalamaha', str(cm.exception)) + + def testAssertNotStartsWith(self): + self.assertNotStartsWith('ababahalamaha', 'amaha') + self.assertNotStartsWith('ababahalamaha', ('x', 'amaha', 'y')) + self.assertNotStartsWith(UserString('ababahalamaha'), 'amaha') + self.assertNotStartsWith(UserString('ababahalamaha'), ('x', 'amaha', 'y')) + self.assertNotStartsWith(bytearray(b'ababahalamaha'), b'amaha') + self.assertNotStartsWith(bytearray(b'ababahalamaha'), (b'x', b'amaha', b'y')) + self.assertNotStartsWith(b'ababahalamaha', bytearray(b'amaha')) + self.assertNotStartsWith(b'ababahalamaha', + (bytearray(b'x'), bytearray(b'amaha'), bytearray(b'y'))) + + with self.assertRaises(self.failureException) as cm: + self.assertNotStartsWith('ababahalamaha', 'ababa') + self.assertEqual(str(cm.exception), + "'ababahalamaha' starts with 'ababa'") + with self.assertRaises(self.failureException) as cm: + self.assertNotStartsWith('ababahalamaha', ('x', 'ababa', 'y')) + self.assertEqual(str(cm.exception), + "'ababahalamaha' starts with 'ababa'") + + with self.assertRaises(self.failureException) as cm: + self.assertNotStartsWith(b'ababahalamaha', 'ababa') + self.assertEqual(str(cm.exception), 'Expected str, not bytes') + with self.assertRaises(self.failureException) as cm: + self.assertNotStartsWith(b'ababahalamaha', ('amaha', 'ababa')) + self.assertEqual(str(cm.exception), 'Expected str, not bytes') + with self.assertRaises(self.failureException) as cm: + self.assertNotStartsWith([], 'ababa') + self.assertEqual(str(cm.exception), 'Expected str, not list') + with self.assertRaises(self.failureException) as cm: + self.assertNotStartsWith('ababahalamaha', b'ababa') + self.assertEqual(str(cm.exception), 'Expected bytes, not str') + with self.assertRaises(self.failureException) as cm: + self.assertNotStartsWith('ababahalamaha', (b'amaha', b'ababa')) + self.assertEqual(str(cm.exception), 'Expected bytes, not str') + with self.assertRaises(TypeError): + self.assertNotStartsWith('ababahalamaha', ord('a')) + + with self.assertRaises(self.failureException) as cm: + self.assertNotStartsWith('ababahalamaha', 'ababa', 'abracadabra') + self.assertIn('ababahalamaha', str(cm.exception)) + with self.assertRaises(self.failureException) as cm: + self.assertNotStartsWith('ababahalamaha', 'ababa', msg='abracadabra') + self.assertIn('ababahalamaha', str(cm.exception)) + + def testAssertEndsWith(self): + self.assertEndsWith('ababahalamaha', 'amaha') + self.assertEndsWith('ababahalamaha', ('x', 'amaha', 'y')) + self.assertEndsWith(UserString('ababahalamaha'), 'amaha') + self.assertEndsWith(UserString('ababahalamaha'), ('x', 'amaha', 'y')) + self.assertEndsWith(bytearray(b'ababahalamaha'), b'amaha') + self.assertEndsWith(bytearray(b'ababahalamaha'), (b'x', b'amaha', b'y')) + self.assertEndsWith(b'ababahalamaha', bytearray(b'amaha')) + self.assertEndsWith(b'ababahalamaha', + (bytearray(b'x'), bytearray(b'amaha'), bytearray(b'y'))) + + with self.assertRaises(self.failureException) as cm: + self.assertEndsWith('ababahalamaha', 'ababa') + self.assertEqual(str(cm.exception), + "'ababahalamaha' doesn't end with 'ababa'") + with self.assertRaises(self.failureException) as cm: + self.assertEndsWith('ababahalamaha', ('x', 'y')) + self.assertEqual(str(cm.exception), + "'ababahalamaha' doesn't end with any of ('x', 'y')") + + with self.assertRaises(self.failureException) as cm: + self.assertEndsWith(b'ababahalamaha', 'amaha') + self.assertEqual(str(cm.exception), 'Expected str, not bytes') + with self.assertRaises(self.failureException) as cm: + self.assertEndsWith(b'ababahalamaha', ('ababa', 'amaha')) + self.assertEqual(str(cm.exception), 'Expected str, not bytes') + with self.assertRaises(self.failureException) as cm: + self.assertEndsWith([], 'amaha') + self.assertEqual(str(cm.exception), 'Expected str, not list') + with self.assertRaises(self.failureException) as cm: + self.assertEndsWith('ababahalamaha', b'amaha') + self.assertEqual(str(cm.exception), 'Expected bytes, not str') + with self.assertRaises(self.failureException) as cm: + self.assertEndsWith('ababahalamaha', (b'ababa', b'amaha')) + self.assertEqual(str(cm.exception), 'Expected bytes, not str') + with self.assertRaises(TypeError): + self.assertEndsWith('ababahalamaha', ord('a')) + + with self.assertRaises(self.failureException) as cm: + self.assertEndsWith('ababahalamaha', 'ababa', 'abracadabra') + self.assertIn('ababahalamaha', str(cm.exception)) + with self.assertRaises(self.failureException) as cm: + self.assertEndsWith('ababahalamaha', 'ababa', msg='abracadabra') + self.assertIn('ababahalamaha', str(cm.exception)) + + def testAssertNotEndsWith(self): + self.assertNotEndsWith('ababahalamaha', 'ababa') + self.assertNotEndsWith('ababahalamaha', ('x', 'ababa', 'y')) + self.assertNotEndsWith(UserString('ababahalamaha'), 'ababa') + self.assertNotEndsWith(UserString('ababahalamaha'), ('x', 'ababa', 'y')) + self.assertNotEndsWith(bytearray(b'ababahalamaha'), b'ababa') + self.assertNotEndsWith(bytearray(b'ababahalamaha'), (b'x', b'ababa', b'y')) + self.assertNotEndsWith(b'ababahalamaha', bytearray(b'ababa')) + self.assertNotEndsWith(b'ababahalamaha', + (bytearray(b'x'), bytearray(b'ababa'), bytearray(b'y'))) + + with self.assertRaises(self.failureException) as cm: + self.assertNotEndsWith('ababahalamaha', 'amaha') + self.assertEqual(str(cm.exception), + "'ababahalamaha' ends with 'amaha'") + with self.assertRaises(self.failureException) as cm: + self.assertNotEndsWith('ababahalamaha', ('x', 'amaha', 'y')) + self.assertEqual(str(cm.exception), + "'ababahalamaha' ends with 'amaha'") + + with self.assertRaises(self.failureException) as cm: + self.assertNotEndsWith(b'ababahalamaha', 'amaha') + self.assertEqual(str(cm.exception), 'Expected str, not bytes') + with self.assertRaises(self.failureException) as cm: + self.assertNotEndsWith(b'ababahalamaha', ('ababa', 'amaha')) + self.assertEqual(str(cm.exception), 'Expected str, not bytes') + with self.assertRaises(self.failureException) as cm: + self.assertNotEndsWith([], 'amaha') + self.assertEqual(str(cm.exception), 'Expected str, not list') + with self.assertRaises(self.failureException) as cm: + self.assertNotEndsWith('ababahalamaha', b'amaha') + self.assertEqual(str(cm.exception), 'Expected bytes, not str') + with self.assertRaises(self.failureException) as cm: + self.assertNotEndsWith('ababahalamaha', (b'ababa', b'amaha')) + self.assertEqual(str(cm.exception), 'Expected bytes, not str') + with self.assertRaises(TypeError): + self.assertNotEndsWith('ababahalamaha', ord('a')) + + with self.assertRaises(self.failureException) as cm: + self.assertNotEndsWith('ababahalamaha', 'amaha', 'abracadabra') + self.assertIn('ababahalamaha', str(cm.exception)) + with self.assertRaises(self.failureException) as cm: + self.assertNotEndsWith('ababahalamaha', 'amaha', msg='abracadabra') + self.assertIn('ababahalamaha', str(cm.exception)) + def testDeprecatedFailMethods(self): """Test that the deprecated fail* methods get removed in 3.12""" deprecated_names = [ diff --git a/Lib/test/test_unittest/test_discovery.py b/Lib/test/test_unittest/test_discovery.py index a44b18406c0..9ed3d04b1f8 100644 --- a/Lib/test/test_unittest/test_discovery.py +++ b/Lib/test/test_unittest/test_discovery.py @@ -1,15 +1,17 @@ import os.path -from os.path import abspath +import pickle import re import sys import types -import pickle -from test import support -from test.support import import_helper - import unittest import unittest.mock +from importlib._bootstrap_external import NamespaceLoader +from os.path import abspath + import test.test_unittest +from test import support +from test.support import import_helper +from test.test_importlib import util as test_util class TestableTestProgram(unittest.TestProgram): @@ -395,7 +397,7 @@ def restore_isdir(): self.addCleanup(restore_isdir) _find_tests_args = [] - def _find_tests(start_dir, pattern): + def _find_tests(start_dir, pattern, namespace=None): _find_tests_args.append((start_dir, pattern)) return ['tests'] loader._find_tests = _find_tests @@ -815,7 +817,7 @@ def test_discovery_from_dotted_path(self): expectedPath = os.path.abspath(os.path.dirname(test.test_unittest.__file__)) self.wasRun = False - def _find_tests(start_dir, pattern): + def _find_tests(start_dir, pattern, namespace=None): self.wasRun = True self.assertEqual(start_dir, expectedPath) return tests @@ -848,6 +850,55 @@ def restore(): 'Can not use builtin modules ' 'as dotted module names') + def test_discovery_from_dotted_namespace_packages(self): + loader = unittest.TestLoader() + + package = types.ModuleType('package') + package.__name__ = "tests" + package.__path__ = ['/a', '/b'] + package.__file__ = None + package.__spec__ = types.SimpleNamespace( + name=package.__name__, + loader=NamespaceLoader(package.__name__, package.__path__, None), + submodule_search_locations=['/a', '/b'] + ) + + def _import(packagename, *args, **kwargs): + sys.modules[packagename] = package + return package + + _find_tests_args = [] + def _find_tests(start_dir, pattern, namespace=None): + _find_tests_args.append((start_dir, pattern)) + return ['%s/tests' % start_dir] + + loader._find_tests = _find_tests + loader.suiteClass = list + + with unittest.mock.patch('builtins.__import__', _import): + # Since loader.discover() can modify sys.path, restore it when done. + with import_helper.DirsOnSysPath(): + # Make sure to remove 'package' from sys.modules when done. + with test_util.uncache('package'): + suite = loader.discover('package') + + self.assertEqual(suite, ['/a/tests', '/b/tests']) + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_discovery_start_dir_is_namespace(self): + """Subdirectory discovery not affected if start_dir is a namespace pkg.""" + loader = unittest.TestLoader() + with ( + import_helper.DirsOnSysPath(os.path.join(os.path.dirname(__file__))), + test_util.uncache('namespace_test_pkg') + ): + suite = loader.discover('namespace_test_pkg') + self.assertEqual( + {list(suite)[0]._tests[0].__module__ for suite in suite._tests if list(suite)}, + # files under namespace_test_pkg.noop not discovered. + {'namespace_test_pkg.test_foo', 'namespace_test_pkg.bar.test_bar'}, + ) + def test_discovery_failed_discovery(self): from test.test_importlib import util diff --git a/Lib/test/test_unittest/test_loader.py b/Lib/test/test_unittest/test_loader.py index 83dd25ca546..0acefccf7f6 100644 --- a/Lib/test/test_unittest/test_loader.py +++ b/Lib/test/test_unittest/test_loader.py @@ -1,9 +1,9 @@ import functools import sys import types - import unittest + class Test_TestLoader(unittest.TestCase): ### Basic object tests @@ -76,7 +76,7 @@ def runTest(self): loader = unittest.TestLoader() # This has to be false for the test to succeed - self.assertFalse('runTest'.startswith(loader.testMethodPrefix)) + self.assertNotStartsWith('runTest', loader.testMethodPrefix) suite = loader.loadTestsFromTestCase(Foo) self.assertIsInstance(suite, loader.suiteClass) diff --git a/Lib/test/test_unittest/test_program.py b/Lib/test/test_unittest/test_program.py index 2e3a7508478..99c5ec48b67 100644 --- a/Lib/test/test_unittest/test_program.py +++ b/Lib/test/test_unittest/test_program.py @@ -1,9 +1,10 @@ import os -import sys import subprocess -from test import support +import sys import unittest + import test.test_unittest +from test import support from test.test_unittest.test_result import BufferedWriter @@ -135,14 +136,14 @@ def test_NonExit(self): argv=["foobar"], testRunner=unittest.TextTestRunner(stream=stream), testLoader=self.TestLoader(self.FooBar)) - self.assertTrue(hasattr(program, 'result')) + self.assertHasAttr(program, 'result') out = stream.getvalue() self.assertIn('\nFAIL: testFail ', out) self.assertIn('\nERROR: testError ', out) self.assertIn('\nUNEXPECTED SUCCESS: testUnexpectedSuccess ', out) expected = ('\n\nFAILED (failures=1, errors=1, skipped=1, ' 'expected failures=1, unexpected successes=1)\n') - self.assertTrue(out.endswith(expected)) + self.assertEndsWith(out, expected) def test_Exit(self): stream = BufferedWriter() @@ -159,7 +160,7 @@ def test_Exit(self): self.assertIn('\nUNEXPECTED SUCCESS: testUnexpectedSuccess ', out) expected = ('\n\nFAILED (failures=1, errors=1, skipped=1, ' 'expected failures=1, unexpected successes=1)\n') - self.assertTrue(out.endswith(expected)) + self.assertEndsWith(out, expected) def test_ExitAsDefault(self): stream = BufferedWriter() @@ -174,7 +175,7 @@ def test_ExitAsDefault(self): self.assertIn('\nUNEXPECTED SUCCESS: testUnexpectedSuccess ', out) expected = ('\n\nFAILED (failures=1, errors=1, skipped=1, ' 'expected failures=1, unexpected successes=1)\n') - self.assertTrue(out.endswith(expected)) + self.assertEndsWith(out, expected) def test_ExitSkippedSuite(self): stream = BufferedWriter() @@ -186,7 +187,7 @@ def test_ExitSkippedSuite(self): self.assertEqual(cm.exception.code, 0) out = stream.getvalue() expected = '\n\nOK (skipped=1)\n' - self.assertTrue(out.endswith(expected)) + self.assertEndsWith(out, expected) def test_ExitEmptySuite(self): stream = BufferedWriter() @@ -506,7 +507,6 @@ def testParseArgsSelectedTestNames(self): self.assertEqual(program.testNamePatterns, ['*foo*', '*bar*', '*pat*']) @unittest.expectedFailureIf(sys.platform != 'win32', 'TODO: RUSTPYTHON') - def testSelectedTestNamesFunctionalTest(self): def run_unittest(args): # Use -E to ignore PYTHONSAFEPATH env var diff --git a/Lib/test/test_unittest/test_result.py b/Lib/test/test_unittest/test_result.py index 4d552d54e9a..c260f90bf03 100644 --- a/Lib/test/test_unittest/test_result.py +++ b/Lib/test/test_unittest/test_result.py @@ -4,8 +4,12 @@ import traceback import unittest from unittest.util import strclass -from test.support import warnings_helper -from test.support import captured_stdout, force_not_colorized_test_class + +from test.support import ( + captured_stdout, + force_not_colorized_test_class, + warnings_helper, +) from test.test_unittest.support import BufferedWriter @@ -13,7 +17,7 @@ class MockTraceback(object): class TracebackException: def __init__(self, *args, **kwargs): self.capture_locals = kwargs.get('capture_locals', False) - def format(self): + def format(self, **kwargs): result = ['A traceback'] if self.capture_locals: result.append('locals') @@ -186,7 +190,7 @@ def test_1(self): test = Foo('test_1') try: test.fail("foo") - except: + except AssertionError: exc_info_tuple = sys.exc_info() result = unittest.TestResult() @@ -214,7 +218,7 @@ def test_1(self): def get_exc_info(): try: test.fail("foo") - except: + except AssertionError: return sys.exc_info() exc_info_tuple = get_exc_info() @@ -241,9 +245,9 @@ def get_exc_info(): try: try: test.fail("foo") - except: + except AssertionError: raise ValueError(42) - except: + except ValueError: return sys.exc_info() exc_info_tuple = get_exc_info() @@ -271,7 +275,7 @@ def get_exc_info(): loop.__cause__ = loop loop.__context__ = loop raise loop - except: + except Exception: return sys.exc_info() exc_info_tuple = get_exc_info() @@ -300,7 +304,7 @@ def get_exc_info(): ex1.__cause__ = ex2 ex2.__context__ = ex1 raise C - except: + except Exception: return sys.exc_info() exc_info_tuple = get_exc_info() @@ -345,7 +349,7 @@ def test_1(self): test = Foo('test_1') try: raise TypeError() - except: + except TypeError: exc_info_tuple = sys.exc_info() result = unittest.TestResult() @@ -454,7 +458,7 @@ def test(result): self.assertTrue(result.failfast) result = runner.run(test) stream.flush() - self.assertTrue(stream.getvalue().endswith('\n\nOK\n')) + self.assertEndsWith(stream.getvalue(), '\n\nOK\n') @force_not_colorized_test_class diff --git a/Lib/test/test_unittest/test_runner.py b/Lib/test/test_unittest/test_runner.py index 790c4d29ca2..b215a3664d1 100644 --- a/Lib/test/test_unittest/test_runner.py +++ b/Lib/test/test_unittest/test_runner.py @@ -1,13 +1,12 @@ import io import os -import sys import pickle import subprocess -from test import support - +import sys import unittest from unittest.case import _Outcome +from test import support from test.test_unittest.support import ( BufferedWriter, LoggingResult, @@ -1297,7 +1296,6 @@ def _makeResult(self): expected = ['startTestRun', 'stopTestRun'] self.assertEqual(events, expected) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_pickle_unpickle(self): # Issue #7197: a TextTestRunner should be (un)pickleable. This is # required by test_multiprocessing under Windows (in verbose mode). diff --git a/Lib/test/test_unittest/test_setups.py b/Lib/test/test_unittest/test_setups.py index 2df703ed934..2468681003b 100644 --- a/Lib/test/test_unittest/test_setups.py +++ b/Lib/test/test_unittest/test_setups.py @@ -1,6 +1,5 @@ import io import sys - import unittest diff --git a/Lib/test/test_unittest/test_skipping.py b/Lib/test/test_unittest/test_skipping.py index f146dcac18e..f5cb860c60b 100644 --- a/Lib/test/test_unittest/test_skipping.py +++ b/Lib/test/test_unittest/test_skipping.py @@ -1,5 +1,6 @@ import unittest +from test.support import force_not_colorized from test.test_unittest.support import LoggingResult @@ -293,6 +294,7 @@ def test_die(self): self.assertFalse(result.unexpectedSuccesses) self.assertTrue(result.wasSuccessful()) + @force_not_colorized def test_expected_failure_and_fail_in_cleanup(self): class Foo(unittest.TestCase): @unittest.expectedFailure @@ -372,6 +374,7 @@ def test_die(self): self.assertEqual(result.unexpectedSuccesses, [test]) self.assertFalse(result.wasSuccessful()) + @force_not_colorized def test_unexpected_success_and_fail_in_cleanup(self): class Foo(unittest.TestCase): @unittest.expectedFailure diff --git a/Lib/test/test_unittest/test_suite.py b/Lib/test/test_unittest/test_suite.py index ca52ee9d9c0..11c8c859f3d 100644 --- a/Lib/test/test_unittest/test_suite.py +++ b/Lib/test/test_unittest/test_suite.py @@ -1,10 +1,9 @@ -import unittest - import gc import sys +import unittest import weakref -from test.test_unittest.support import LoggingResult, TestEquality +from test.test_unittest.support import LoggingResult, TestEquality ### Support code for Test_TestSuite ################################################################ diff --git a/Lib/test/test_unittest/test_util.py b/Lib/test/test_unittest/test_util.py index d590a333930..abadcb96601 100644 --- a/Lib/test/test_unittest/test_util.py +++ b/Lib/test/test_unittest/test_util.py @@ -1,5 +1,9 @@ import unittest -from unittest.util import safe_repr, sorted_list_difference, unorderable_list_difference +from unittest.util import ( + safe_repr, + sorted_list_difference, + unorderable_list_difference, +) class TestUtil(unittest.TestCase): diff --git a/Lib/test/test_unittest/testmock/testasync.py b/Lib/test/test_unittest/testmock/testasync.py index ddc6f0599c8..81d9c9c55fd 100644 --- a/Lib/test/test_unittest/testmock/testasync.py +++ b/Lib/test/test_unittest/testmock/testasync.py @@ -734,7 +734,6 @@ def __aiter__(self): pass async def __anext__(self): pass - @unittest.expectedFailure # TODO: RUSTPYTHON def test_aiter_set_return_value(self): mock_iter = AsyncMock(name="tester") mock_iter.__aiter__.return_value = [1, 2, 3] @@ -760,7 +759,6 @@ def inner_test(mock_type): inner_test(mock_type) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_mock_async_for(self): async def iterate(iterator): accumulator = [] diff --git a/Lib/test/test_unpack_ex.py b/Lib/test/test_unpack_ex.py new file mode 100644 index 00000000000..2cadb9c70ba --- /dev/null +++ b/Lib/test/test_unpack_ex.py @@ -0,0 +1,418 @@ +# Tests for extended unpacking, starred expressions. + +import doctest +import unittest + + +doctests = """ + +Unpack tuple + + >>> t = (1, 2, 3) + >>> a, *b, c = t + >>> a == 1 and b == [2] and c == 3 + True + +Unpack list + + >>> l = [4, 5, 6] + >>> a, *b = l + >>> a == 4 and b == [5, 6] + True + +Unpack implied tuple + + >>> *a, = 7, 8, 9 + >>> a == [7, 8, 9] + True + +Unpack nested implied tuple + + >>> [*[*a]] = [[7,8,9]] + >>> a == [[7,8,9]] + True + +Unpack string... fun! + + >>> a, *b = 'one' + >>> a == 'o' and b == ['n', 'e'] + True + +Unpack long sequence + + >>> a, b, c, *d, e, f, g = range(10) + >>> (a, b, c, d, e, f, g) == (0, 1, 2, [3, 4, 5, 6], 7, 8, 9) + True + +Unpack short sequence + + >>> a, *b, c = (1, 2) + >>> a == 1 and c == 2 and b == [] + True + +Unpack generic sequence + + >>> class Seq: + ... def __getitem__(self, i): + ... if i >= 0 and i < 3: return i + ... raise IndexError + ... + >>> a, *b = Seq() + >>> a == 0 and b == [1, 2] + True + +Unpack in for statement + + >>> for a, *b, c in [(1,2,3), (4,5,6,7)]: + ... print(a, b, c) + ... + 1 [2] 3 + 4 [5, 6] 7 + +Unpack in list + + >>> [a, *b, c] = range(5) + >>> a == 0 and b == [1, 2, 3] and c == 4 + True + +Multiple targets + + >>> a, *b, c = *d, e = range(5) + >>> a == 0 and b == [1, 2, 3] and c == 4 and d == [0, 1, 2, 3] and e == 4 + True + +Assignment unpacking + + >>> a, b, *c = range(5) + >>> a, b, c + (0, 1, [2, 3, 4]) + >>> *a, b, c = a, b, *c + >>> a, b, c + ([0, 1, 2], 3, 4) + +Set display element unpacking + + >>> a = [1, 2, 3] + >>> sorted({1, *a, 0, 4}) + [0, 1, 2, 3, 4] + + >>> {1, *1, 0, 4} + Traceback (most recent call last): + ... + TypeError: 'int' object is not iterable + +Dict display element unpacking + + >>> kwds = {'z': 0, 'w': 12} + >>> sorted({'x': 1, 'y': 2, **kwds}.items()) + [('w', 12), ('x', 1), ('y', 2), ('z', 0)] + + >>> sorted({**{'x': 1}, 'y': 2, **{'z': 3}}.items()) + [('x', 1), ('y', 2), ('z', 3)] + + >>> sorted({**{'x': 1}, 'y': 2, **{'x': 3}}.items()) + [('x', 3), ('y', 2)] + + >>> sorted({**{'x': 1}, **{'x': 3}, 'x': 4}.items()) # TODO: RUSTPYTHON # doctest:+EXPECTED_FAILURE + [('x', 4)] + + >>> {**{}} + {} + + >>> a = {} + >>> {**a}[0] = 1 + >>> a + {} + + >>> {**1} + Traceback (most recent call last): + ... + TypeError: 'int' object is not a mapping + + >>> {**[]} + Traceback (most recent call last): + ... + TypeError: 'list' object is not a mapping + + >>> len(eval("{" + ", ".join("**{{{}: {}}}".format(i, i) + ... for i in range(1000)) + "}")) + 1000 + + >>> {0:1, **{0:2}, 0:3, 0:4} # TODO: RUSTPYTHON # doctest:+EXPECTED_FAILURE + {0: 4} + +List comprehension element unpacking + + >>> a, b, c = [0, 1, 2], 3, 4 + >>> [*a, b, c] + [0, 1, 2, 3, 4] + + >>> l = [a, (3, 4), {5}, {6: None}, (i for i in range(7, 10))] + >>> [*item for item in l] # TODO: RUSTPYTHON # doctest:+EXPECTED_FAILURE + Traceback (most recent call last): + ... + SyntaxError: iterable unpacking cannot be used in comprehension + + >>> [*[0, 1] for i in range(10)] # TODO: RUSTPYTHON # doctest:+EXPECTED_FAILURE + Traceback (most recent call last): + ... + SyntaxError: iterable unpacking cannot be used in comprehension + + >>> [*'a' for i in range(10)] # TODO: RUSTPYTHON # doctest:+EXPECTED_FAILURE + Traceback (most recent call last): + ... + SyntaxError: iterable unpacking cannot be used in comprehension + + >>> [*[] for i in range(10)] # TODO: RUSTPYTHON # doctest:+EXPECTED_FAILURE + Traceback (most recent call last): + ... + SyntaxError: iterable unpacking cannot be used in comprehension + + >>> {**{} for a in [1]} # TODO: RUSTPYTHON # doctest:+EXPECTED_FAILURE + Traceback (most recent call last): + ... + SyntaxError: dict unpacking cannot be used in dict comprehension + +# Pegen is better here. +# Generator expression in function arguments + +# >>> list(*x for x in (range(5) for i in range(3))) +# Traceback (most recent call last): +# ... +# list(*x for x in (range(5) for i in range(3))) +# ^ +# SyntaxError: invalid syntax + + >>> dict(**x for x in [{1:2}]) # TODO: RUSTPYTHON # doctest:+EXPECTED_FAILURE + Traceback (most recent call last): + ... + dict(**x for x in [{1:2}]) + ^ + SyntaxError: invalid syntax + +Iterable argument unpacking + + >>> print(*[1], *[2], 3) + 1 2 3 + +Make sure that they don't corrupt the passed-in dicts. + + >>> def f(x, y): + ... print(x, y) + ... + >>> original_dict = {'x': 1} + >>> f(**original_dict, y=2) + 1 2 + >>> original_dict + {'x': 1} + +Now for some failures + +Make sure the raised errors are right for keyword argument unpackings + + >>> from collections.abc import MutableMapping + >>> class CrazyDict(MutableMapping): + ... def __init__(self): + ... self.d = {} + ... + ... def __iter__(self): + ... for x in self.d.__iter__(): + ... if x == 'c': + ... self.d['z'] = 10 + ... yield x + ... + ... def __getitem__(self, k): + ... return self.d[k] + ... + ... def __len__(self): + ... return len(self.d) + ... + ... def __setitem__(self, k, v): + ... self.d[k] = v + ... + ... def __delitem__(self, k): + ... del self.d[k] + ... + >>> d = CrazyDict() + >>> d.d = {chr(ord('a') + x): x for x in range(5)} + >>> e = {**d} + Traceback (most recent call last): + ... + RuntimeError: dictionary changed size during iteration + + >>> d.d = {chr(ord('a') + x): x for x in range(5)} + >>> def f(**kwargs): print(kwargs) + >>> f(**d) + Traceback (most recent call last): + ... + RuntimeError: dictionary changed size during iteration + +Overridden parameters + + >>> f(x=5, **{'x': 3}, y=2) # TODO: RUSTPYTHON # doctest:+EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_unpack_ex.f() got multiple values for keyword argument 'x' + + >>> f(**{'x': 3}, x=5, y=2) # TODO: RUSTPYTHON # doctest:+EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_unpack_ex.f() got multiple values for keyword argument 'x' + + >>> f(**{'x': 3}, **{'x': 5}, y=2) # TODO: RUSTPYTHON # doctest:+EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_unpack_ex.f() got multiple values for keyword argument 'x' + + >>> f(x=5, **{'x': 3}, **{'x': 2}) # TODO: RUSTPYTHON # doctest:+EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_unpack_ex.f() got multiple values for keyword argument 'x' + + >>> f(**{1: 3}, **{1: 5}) # TODO: RUSTPYTHON # doctest:+EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: test.test_unpack_ex.f() got multiple values for keyword argument '1' + +Unpacking non-sequence + + >>> a, *b = 7 # TODO: RUSTPYTHON # doctest:+EXPECTED_FAILURE + Traceback (most recent call last): + ... + TypeError: cannot unpack non-iterable int object + +Unpacking sequence too short + + >>> a, *b, c, d, e = Seq() + Traceback (most recent call last): + ... + ValueError: not enough values to unpack (expected at least 4, got 3) + +Unpacking sequence too short and target appears last + + >>> a, b, c, d, *e = Seq() + Traceback (most recent call last): + ... + ValueError: not enough values to unpack (expected at least 4, got 3) + +Unpacking a sequence where the test for too long raises a different kind of +error + + >>> class BozoError(Exception): + ... pass + ... + >>> class BadSeq: + ... def __getitem__(self, i): + ... if i >= 0 and i < 3: + ... return i + ... elif i == 3: + ... raise BozoError + ... else: + ... raise IndexError + ... + +Trigger code while not expecting an IndexError (unpack sequence too long, wrong +error) + + >>> a, *b, c, d, e = BadSeq() + Traceback (most recent call last): + ... + test.test_unpack_ex.BozoError + +Now some general starred expressions (all fail). + + >>> a, *b, c, *d, e = range(10) # TODO: RUSTPYTHON # doctest:+ELLIPSIS +EXPECTED_FAILURE + Traceback (most recent call last): + ... + SyntaxError: multiple starred expressions in assignment + + >>> [*b, *c] = range(10) # TODO: RUSTPYTHON # doctest:+ELLIPSIS +EXPECTED_FAILURE + Traceback (most recent call last): + ... + SyntaxError: multiple starred expressions in assignment + + >>> a,*b,*c,*d = range(4) # TODO: RUSTPYTHON # doctest:+ELLIPSIS +EXPECTED_FAILURE + Traceback (most recent call last): + ... + SyntaxError: multiple starred expressions in assignment + + >>> *a = range(10) # doctest:+ELLIPSIS + Traceback (most recent call last): + ... + SyntaxError: starred assignment target must be in a list or tuple + + >>> *a # TODO: RUSTPYTHON # doctest:+ELLIPSIS +EXPECTED_FAILURE + Traceback (most recent call last): + ... + SyntaxError: can't use starred expression here + + >>> *1 # TODO: RUSTPYTHON # doctest:+ELLIPSIS +EXPECTED_FAILURE + Traceback (most recent call last): + ... + SyntaxError: can't use starred expression here + + >>> x = *a # TODO: RUSTPYTHON # doctest:+ELLIPSIS +EXPECTED_FAILURE + Traceback (most recent call last): + ... + SyntaxError: can't use starred expression here + + >>> (*x),y = 1, 2 # TODO: RUSTPYTHON # doctest:+ELLIPSIS +EXPECTED_FAILURE + Traceback (most recent call last): + ... + SyntaxError: cannot use starred expression here + + >>> (((*x))),y = 1, 2 # TODO: RUSTPYTHON # doctest:+ELLIPSIS +EXPECTED_FAILURE + Traceback (most recent call last): + ... + SyntaxError: cannot use starred expression here + + >>> z,(*x),y = 1, 2, 4 # TODO: RUSTPYTHON # doctest:+ELLIPSIS +EXPECTED_FAILURE + Traceback (most recent call last): + ... + SyntaxError: cannot use starred expression here + + >>> z,(*x) = 1, 2 # TODO: RUSTPYTHON # doctest:+ELLIPSIS +EXPECTED_FAILURE + Traceback (most recent call last): + ... + SyntaxError: cannot use starred expression here + + >>> ((*x),y) = 1, 2 # TODO: RUSTPYTHON # doctest:+ELLIPSIS +EXPECTED_FAILURE + Traceback (most recent call last): + ... + SyntaxError: cannot use starred expression here + +Some size constraints (all fail.) + + >>> s = ", ".join("a%d" % i for i in range(1<<8)) + ", *rest = range(1<<8 + 1)" + >>> compile(s, 'test', 'exec') # doctest:+ELLIPSIS + Traceback (most recent call last): + ... + SyntaxError: too many expressions in star-unpacking assignment + + >>> s = ", ".join("a%d" % i for i in range(1<<8 + 1)) + ", *rest = range(1<<8 + 2)" + >>> compile(s, 'test', 'exec') # doctest:+ELLIPSIS + Traceback (most recent call last): + ... + SyntaxError: too many expressions in star-unpacking assignment + +(there is an additional limit, on the number of expressions after the +'*rest', but it's 1<<24 and testing it takes too much memory.) + +""" + +__test__ = {'doctests' : doctests} + +EXPECTED_FAILURE = doctest.register_optionflag('EXPECTED_FAILURE') # TODO: RUSTPYTHON +class CustomOutputChecker(doctest.OutputChecker): # TODO: RUSTPYTHON + def check_output(self, want, got, optionflags): # TODO: RUSTPYTHON + if optionflags & EXPECTED_FAILURE: # TODO: RUSTPYTHON + return not super().check_output(want, got, optionflags) # TODO: RUSTPYTHON + return super().check_output(want, got, optionflags) # TODO: RUSTPYTHON + +def load_tests(loader, tests, pattern): + tests.addTest(doctest.DocTestSuite(checker=CustomOutputChecker())) # TODO: RUSTPYTHON + return tests + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_urllib.py b/Lib/test/test_urllib.py index aee9fb78017..ae524c5ffba 100644 --- a/Lib/test/test_urllib.py +++ b/Lib/test/test_urllib.py @@ -7,13 +7,11 @@ import email.message import io import unittest -from unittest.mock import patch from test import support from test.support import os_helper from test.support import socket_helper -from test.support import warnings_helper -from test.support.testcase import ExtraAssertions import os +import socket try: import ssl except ImportError: @@ -21,7 +19,6 @@ import sys import tempfile -from base64 import b64encode import collections @@ -36,32 +33,6 @@ def hexescape(char): hex_repr = "0%s" % hex_repr return "%" + hex_repr -# Shortcut for testing FancyURLopener -_urlopener = None - - -def urlopen(url, data=None, proxies=None): - """urlopen(url [, data]) -> open file-like object""" - global _urlopener - if proxies is not None: - opener = urllib.request.FancyURLopener(proxies=proxies) - elif not _urlopener: - opener = FancyURLopener() - _urlopener = opener - else: - opener = _urlopener - if data is None: - return opener.open(url) - else: - return opener.open(url, data) - - -def FancyURLopener(): - with warnings_helper.check_warnings( - ('FancyURLopener style of invoking requests is deprecated.', - DeprecationWarning)): - return urllib.request.FancyURLopener() - def fakehttp(fakedata, mock_close=False): class FakeSocket(io.BytesIO): @@ -120,27 +91,7 @@ def unfakehttp(self): http.client.HTTPConnection = self._connection_class -class FakeFTPMixin(object): - def fakeftp(self): - class FakeFtpWrapper(object): - def __init__(self, user, passwd, host, port, dirs, timeout=None, - persistent=True): - pass - - def retrfile(self, file, type): - return io.BytesIO(), 0 - - def close(self): - pass - - self._ftpwrapper_class = urllib.request.ftpwrapper - urllib.request.ftpwrapper = FakeFtpWrapper - - def unfakeftp(self): - urllib.request.ftpwrapper = self._ftpwrapper_class - - -class urlopen_FileTests(unittest.TestCase, ExtraAssertions): +class urlopen_FileTests(unittest.TestCase): """Test urlopen() opening a temporary file. Try to test as much functionality as possible so as to cut down on reliance @@ -159,7 +110,7 @@ def setUp(self): f.close() self.pathname = os_helper.TESTFN self.quoted_pathname = urllib.parse.quote(os.fsencode(self.pathname)) - self.returned_obj = urlopen("file:%s" % self.quoted_pathname) + self.returned_obj = urllib.request.urlopen("file:%s" % self.quoted_pathname) def tearDown(self): """Shut down the open object""" @@ -204,7 +155,7 @@ def test_headers(self): self.assertIsInstance(self.returned_obj.headers, email.message.Message) def test_url(self): - self.assertEqual(self.returned_obj.url, self.quoted_pathname) + self.assertEqual(self.returned_obj.url, "file:" + self.quoted_pathname) def test_status(self): self.assertIsNone(self.returned_obj.status) @@ -213,7 +164,7 @@ def test_info(self): self.assertIsInstance(self.returned_obj.info(), email.message.Message) def test_geturl(self): - self.assertEqual(self.returned_obj.geturl(), self.quoted_pathname) + self.assertEqual(self.returned_obj.geturl(), "file:" + self.quoted_pathname) def test_getcode(self): self.assertIsNone(self.returned_obj.getcode()) @@ -230,6 +181,16 @@ def test_iter(self): def test_relativelocalfile(self): self.assertRaises(ValueError,urllib.request.urlopen,'./' + self.pathname) + def test_remote_authority(self): + # Test for GH-90812. + url = 'file://pythontest.net/foo/bar' + with self.assertRaises(urllib.error.URLError) as e: + urllib.request.urlopen(url) + if os.name == 'nt': + self.assertEqual(e.exception.filename, r'\\pythontest.net\foo\bar') + else: + self.assertEqual(e.exception.reason, 'file:// scheme is supported only on localhost') + class ProxyTests(unittest.TestCase): @@ -338,13 +299,13 @@ def test_getproxies_environment_prefer_lowercase(self): self.assertEqual('http://somewhere:3128', proxies['http']) -class urlopen_HttpTests(unittest.TestCase, FakeHTTPMixin, FakeFTPMixin): +class urlopen_HttpTests(unittest.TestCase, FakeHTTPMixin): """Test urlopen() opening a fake http connection.""" def check_read(self, ver): self.fakehttp(b"HTTP/" + ver + b" 200 OK\r\n\r\nHello!") try: - fp = urlopen("http://python.org/") + fp = urllib.request.urlopen("http://python.org/") self.assertEqual(fp.readline(), b"Hello!") self.assertEqual(fp.readline(), b"") self.assertEqual(fp.geturl(), 'http://python.org/') @@ -365,8 +326,8 @@ def test_url_fragment(self): def test_willclose(self): self.fakehttp(b"HTTP/1.1 200 OK\r\n\r\nHello!") try: - resp = urlopen("http://www.python.org") - self.assertTrue(resp.fp.will_close) + resp = urllib.request.urlopen("http://www.python.org") + self.assertTrue(resp.will_close) finally: self.unfakehttp() @@ -391,9 +352,6 @@ def test_url_path_with_control_char_rejected(self): with self.assertRaisesRegex( InvalidURL, f"contain control.*{escaped_char_repr}"): urllib.request.urlopen(f"https:{schemeless_url}") - # This code path quotes the URL so there is no injection. - resp = urlopen(f"http:{schemeless_url}") - self.assertNotIn(char, resp.geturl()) finally: self.unfakehttp() @@ -415,11 +373,6 @@ def test_url_path_with_newline_header_injection_rejected(self): urllib.request.urlopen(f"http:{schemeless_url}") with self.assertRaisesRegex(InvalidURL, r"contain control.*\\n"): urllib.request.urlopen(f"https:{schemeless_url}") - # This code path quotes the URL so there is no injection. - resp = urlopen(f"http:{schemeless_url}") - self.assertNotIn(' ', resp.geturl()) - self.assertNotIn('\r', resp.geturl()) - self.assertNotIn('\n', resp.geturl()) finally: self.unfakehttp() @@ -434,9 +387,9 @@ def test_url_host_with_control_char_rejected(self): InvalidURL = http.client.InvalidURL with self.assertRaisesRegex( InvalidURL, f"contain control.*{escaped_char_repr}"): - urlopen(f"http:{schemeless_url}") + urllib.request.urlopen(f"http:{schemeless_url}") with self.assertRaisesRegex(InvalidURL, f"contain control.*{escaped_char_repr}"): - urlopen(f"https:{schemeless_url}") + urllib.request.urlopen(f"https:{schemeless_url}") finally: self.unfakehttp() @@ -449,9 +402,9 @@ def test_url_host_with_newline_header_injection_rejected(self): InvalidURL = http.client.InvalidURL with self.assertRaisesRegex( InvalidURL, r"contain control.*\\r"): - urlopen(f"http:{schemeless_url}") + urllib.request.urlopen(f"http:{schemeless_url}") with self.assertRaisesRegex(InvalidURL, r"contain control.*\\n"): - urlopen(f"https:{schemeless_url}") + urllib.request.urlopen(f"https:{schemeless_url}") finally: self.unfakehttp() @@ -501,7 +454,7 @@ def test_invalid_redirect(self): def test_redirect_limit_independent(self): # Ticket #12923: make sure independent requests each use their # own retry limit. - for i in range(FancyURLopener().maxtries): + for i in range(urllib.request.HTTPRedirectHandler.max_redirections): self.fakehttp(b'''HTTP/1.1 302 Found Location: file://guidocomputer.athome.com:/python/license Connection: close @@ -518,89 +471,49 @@ def test_empty_socket(self): # data. (#1680230) self.fakehttp(b'') try: - self.assertRaises(OSError, urlopen, "http://something") + self.assertRaises(OSError, urllib.request.urlopen, "http://something") finally: self.unfakehttp() def test_missing_localfile(self): # Test for #10836 with self.assertRaises(urllib.error.URLError) as e: - urlopen('file://localhost/a/file/which/doesnot/exists.py') + urllib.request.urlopen('file://localhost/a/file/which/doesnot/exists.py') self.assertTrue(e.exception.filename) self.assertTrue(e.exception.reason) def test_file_notexists(self): fd, tmp_file = tempfile.mkstemp() - tmp_fileurl = 'file://localhost/' + tmp_file.replace(os.path.sep, '/') + tmp_file_canon_url = urllib.request.pathname2url(tmp_file, add_scheme=True) + parsed = urllib.parse.urlsplit(tmp_file_canon_url) + tmp_fileurl = parsed._replace(netloc='localhost').geturl() try: self.assertTrue(os.path.exists(tmp_file)) - with urlopen(tmp_fileurl) as fobj: + with urllib.request.urlopen(tmp_fileurl) as fobj: self.assertTrue(fobj) + self.assertEqual(fobj.url, tmp_file_canon_url) finally: os.close(fd) os.unlink(tmp_file) self.assertFalse(os.path.exists(tmp_file)) with self.assertRaises(urllib.error.URLError): - urlopen(tmp_fileurl) + urllib.request.urlopen(tmp_fileurl) def test_ftp_nohost(self): test_ftp_url = 'ftp:///path' with self.assertRaises(urllib.error.URLError) as e: - urlopen(test_ftp_url) + urllib.request.urlopen(test_ftp_url) self.assertFalse(e.exception.filename) self.assertTrue(e.exception.reason) def test_ftp_nonexisting(self): with self.assertRaises(urllib.error.URLError) as e: - urlopen('ftp://localhost/a/file/which/doesnot/exists.py') + urllib.request.urlopen('ftp://localhost/a/file/which/doesnot/exists.py') self.assertFalse(e.exception.filename) self.assertTrue(e.exception.reason) - @patch.object(urllib.request, 'MAXFTPCACHE', 0) - def test_ftp_cache_pruning(self): - self.fakeftp() - try: - urllib.request.ftpcache['test'] = urllib.request.ftpwrapper('user', 'pass', 'localhost', 21, []) - urlopen('ftp://localhost') - finally: - self.unfakeftp() - def test_userpass_inurl(self): - self.fakehttp(b"HTTP/1.0 200 OK\r\n\r\nHello!") - try: - fp = urlopen("http://user:pass@python.org/") - self.assertEqual(fp.readline(), b"Hello!") - self.assertEqual(fp.readline(), b"") - self.assertEqual(fp.geturl(), 'http://user:pass@python.org/') - self.assertEqual(fp.getcode(), 200) - finally: - self.unfakehttp() - - def test_userpass_inurl_w_spaces(self): - self.fakehttp(b"HTTP/1.0 200 OK\r\n\r\nHello!") - try: - userpass = "a b:c d" - url = "http://{}@python.org/".format(userpass) - fakehttp_wrapper = http.client.HTTPConnection - authorization = ("Authorization: Basic %s\r\n" % - b64encode(userpass.encode("ASCII")).decode("ASCII")) - fp = urlopen(url) - # The authorization header must be in place - self.assertIn(authorization, fakehttp_wrapper.buf.decode("UTF-8")) - self.assertEqual(fp.readline(), b"Hello!") - self.assertEqual(fp.readline(), b"") - # the spaces are quoted in URL so no match - self.assertNotEqual(fp.geturl(), url) - self.assertEqual(fp.getcode(), 200) - finally: - self.unfakehttp() - - def test_URLopener_deprecation(self): - with warnings_helper.check_warnings(('',DeprecationWarning)): - urllib.request.URLopener() - - -class urlopen_DataTests(unittest.TestCase, ExtraAssertions): +class urlopen_DataTests(unittest.TestCase): """Test urlopen() opening a data URL.""" def setUp(self): @@ -713,7 +626,7 @@ def tearDown(self): def constructLocalFileUrl(self, filePath): filePath = os.path.abspath(filePath) - return "file://%s" % urllib.request.pathname2url(filePath) + return urllib.request.pathname2url(filePath, add_scheme=True) def createNewTempFile(self, data=b""): """Creates a new temporary file containing the specified data, @@ -1518,6 +1431,32 @@ def test_quoting(self): "url2pathname() failed; %s != %s" % (expect, result)) + def test_pathname2url(self): + # Test cases common to Windows and POSIX. + fn = urllib.request.pathname2url + sep = os.path.sep + self.assertEqual(fn(''), '') + self.assertEqual(fn(sep), '///') + self.assertEqual(fn('a'), 'a') + self.assertEqual(fn(f'a{sep}b.c'), 'a/b.c') + self.assertEqual(fn(f'{sep}a{sep}b.c'), '///a/b.c') + self.assertEqual(fn(f'{sep}a{sep}b%#c'), '///a/b%25%23c') + + def test_pathname2url_add_scheme(self): + sep = os.path.sep + subtests = [ + ('', 'file:'), + (sep, 'file:///'), + ('a', 'file:a'), + (f'a{sep}b.c', 'file:a/b.c'), + (f'{sep}a{sep}b.c', 'file:///a/b.c'), + (f'{sep}a{sep}b%#c', 'file:///a/b%25%23c'), + ] + for path, expected_url in subtests: + with self.subTest(path=path): + self.assertEqual( + urllib.request.pathname2url(path, add_scheme=True), expected_url) + @unittest.skipUnless(sys.platform == 'win32', 'test specific to Windows pathnames.') def test_pathname2url_win(self): @@ -1527,16 +1466,18 @@ def test_pathname2url_win(self): self.assertEqual(fn('\\\\?\\unc\\server\\share\\dir'), '//server/share/dir') self.assertEqual(fn("C:"), '///C:') self.assertEqual(fn("C:\\"), '///C:/') + self.assertEqual(fn('c:\\a\\b.c'), '///c:/a/b.c') self.assertEqual(fn('C:\\a\\b.c'), '///C:/a/b.c') self.assertEqual(fn('C:\\a\\b.c\\'), '///C:/a/b.c/') self.assertEqual(fn('C:\\a\\\\b.c'), '///C:/a//b.c') self.assertEqual(fn('C:\\a\\b%#c'), '///C:/a/b%25%23c') self.assertEqual(fn('C:\\a\\b\xe9'), '///C:/a/b%C3%A9') self.assertEqual(fn('C:\\foo\\bar\\spam.foo'), "///C:/foo/bar/spam.foo") - # Long drive letter - self.assertRaises(IOError, fn, "XX:\\") + # NTFS alternate data streams + self.assertEqual(fn('C:\\foo:bar'), '///C:/foo%3Abar') + self.assertEqual(fn('foo:bar'), 'foo%3Abar') # No drive letter - self.assertEqual(fn("\\folder\\test\\"), '/folder/test/') + self.assertEqual(fn("\\folder\\test\\"), '///folder/test/') self.assertEqual(fn("\\\\folder\\test\\"), '//folder/test/') self.assertEqual(fn("\\\\\\folder\\test\\"), '///folder/test/') self.assertEqual(fn('\\\\some\\share\\'), '//some/share/') @@ -1549,22 +1490,18 @@ def test_pathname2url_win(self): self.assertEqual(fn('//?/unc/server/share/dir'), '//server/share/dir') # Round-tripping urls = ['///C:', - '/folder/test/', + '///folder/test/', '///C:/foo/bar/spam.foo'] for url in urls: self.assertEqual(fn(urllib.request.url2pathname(url)), url) @unittest.skipIf(sys.platform == 'win32', 'test specific to POSIX pathnames') - @unittest.expectedFailure # AssertionError: '//a/b.c' != '////a/b.c' def test_pathname2url_posix(self): fn = urllib.request.pathname2url - self.assertEqual(fn('/'), '/') - self.assertEqual(fn('/a/b.c'), '/a/b.c') self.assertEqual(fn('//a/b.c'), '////a/b.c') self.assertEqual(fn('///a/b.c'), '/////a/b.c') self.assertEqual(fn('////a/b.c'), '//////a/b.c') - self.assertEqual(fn('/a/b%#c'), '/a/b%25%23c') @unittest.skipUnless(os_helper.FS_NONASCII, 'need os_helper.FS_NONASCII') def test_pathname2url_nonascii(self): @@ -1573,11 +1510,90 @@ def test_pathname2url_nonascii(self): url = urllib.parse.quote(os_helper.FS_NONASCII, encoding=encoding, errors=errors) self.assertEqual(urllib.request.pathname2url(os_helper.FS_NONASCII), url) + def test_url2pathname(self): + # Test cases common to Windows and POSIX. + fn = urllib.request.url2pathname + sep = os.path.sep + self.assertEqual(fn(''), '') + self.assertEqual(fn('/'), f'{sep}') + self.assertEqual(fn('///'), f'{sep}') + self.assertEqual(fn('////'), f'{sep}{sep}') + self.assertEqual(fn('foo'), 'foo') + self.assertEqual(fn('foo/bar'), f'foo{sep}bar') + self.assertEqual(fn('/foo/bar'), f'{sep}foo{sep}bar') + self.assertEqual(fn('//localhost/foo/bar'), f'{sep}foo{sep}bar') + self.assertEqual(fn('///foo/bar'), f'{sep}foo{sep}bar') + self.assertEqual(fn('////foo/bar'), f'{sep}{sep}foo{sep}bar') + self.assertEqual(fn('data:blah'), 'data:blah') + self.assertEqual(fn('data://blah'), f'data:{sep}{sep}blah') + self.assertEqual(fn('foo?bar'), 'foo') + self.assertEqual(fn('foo#bar'), 'foo') + self.assertEqual(fn('foo?bar=baz'), 'foo') + self.assertEqual(fn('foo?bar#baz'), 'foo') + self.assertEqual(fn('foo%3Fbar'), 'foo?bar') + self.assertEqual(fn('foo%23bar'), 'foo#bar') + self.assertEqual(fn('foo%3Fbar%3Dbaz'), 'foo?bar=baz') + self.assertEqual(fn('foo%3Fbar%23baz'), 'foo?bar#baz') + + def test_url2pathname_require_scheme(self): + sep = os.path.sep + subtests = [ + ('file:', ''), + ('FILE:', ''), + ('FiLe:', ''), + ('file:/', f'{sep}'), + ('file:///', f'{sep}'), + ('file:////', f'{sep}{sep}'), + ('file:foo', 'foo'), + ('file:foo/bar', f'foo{sep}bar'), + ('file:/foo/bar', f'{sep}foo{sep}bar'), + ('file://localhost/foo/bar', f'{sep}foo{sep}bar'), + ('file:///foo/bar', f'{sep}foo{sep}bar'), + ('file:////foo/bar', f'{sep}{sep}foo{sep}bar'), + ('file:data:blah', 'data:blah'), + ('file:data://blah', f'data:{sep}{sep}blah'), + ] + for url, expected_path in subtests: + with self.subTest(url=url): + self.assertEqual( + urllib.request.url2pathname(url, require_scheme=True), + expected_path) + + def test_url2pathname_require_scheme_errors(self): + subtests = [ + '', + ':', + 'foo', + 'http:foo', + 'localfile:foo', + 'data:foo', + 'data:file:foo', + 'data:file://foo', + ] + for url in subtests: + with self.subTest(url=url): + self.assertRaises( + urllib.error.URLError, + urllib.request.url2pathname, + url, require_scheme=True) + + @unittest.skipIf(support.is_emscripten, "Fixed by https://github.com/emscripten-core/emscripten/pull/24593") + def test_url2pathname_resolve_host(self): + fn = urllib.request.url2pathname + sep = os.path.sep + self.assertEqual(fn('//127.0.0.1/foo/bar', resolve_host=True), f'{sep}foo{sep}bar') + self.assertEqual(fn(f'//{socket.gethostname()}/foo/bar'), f'{sep}foo{sep}bar') + self.assertEqual(fn(f'//{socket.gethostname()}/foo/bar', resolve_host=True), f'{sep}foo{sep}bar') + @unittest.skipUnless(sys.platform == 'win32', 'test specific to Windows pathnames.') def test_url2pathname_win(self): fn = urllib.request.url2pathname self.assertEqual(fn('/C:/'), 'C:\\') + self.assertEqual(fn('//C:'), 'C:') + self.assertEqual(fn('//C:/'), 'C:\\') + self.assertEqual(fn('//C:\\'), 'C:\\') + self.assertEqual(fn('//C:80/'), 'C:80\\') self.assertEqual(fn("///C|"), 'C:') self.assertEqual(fn("///C:"), 'C:') self.assertEqual(fn('///C:/'), 'C:\\') @@ -1587,6 +1603,7 @@ def test_url2pathname_win(self): self.assertEqual(fn("///C/test/"), '\\C\\test\\') self.assertEqual(fn("////C/test/"), '\\\\C\\test\\') # DOS drive paths + self.assertEqual(fn('c:/path/to/file'), 'c:\\path\\to\\file') self.assertEqual(fn('C:/path/to/file'), 'C:\\path\\to\\file') self.assertEqual(fn('C:/path/to/file/'), 'C:\\path\\to\\file\\') self.assertEqual(fn('C:/path/to//file'), 'C:\\path\\to\\\\file') @@ -1594,12 +1611,15 @@ def test_url2pathname_win(self): self.assertEqual(fn('/C|/path/to/file'), 'C:\\path\\to\\file') self.assertEqual(fn('///C|/path/to/file'), 'C:\\path\\to\\file') self.assertEqual(fn("///C|/foo/bar/spam.foo"), 'C:\\foo\\bar\\spam.foo') - # Non-ASCII drive letter - self.assertRaises(IOError, fn, "///\u00e8|/") + # Colons in URI + self.assertEqual(fn('///\u00e8|/'), '\u00e8:\\') + self.assertEqual(fn('//host/share/spam.txt:eggs'), '\\\\host\\share\\spam.txt:eggs') + self.assertEqual(fn('///c:/spam.txt:eggs'), 'c:\\spam.txt:eggs') # UNC paths self.assertEqual(fn('//server/path/to/file'), '\\\\server\\path\\to\\file') self.assertEqual(fn('////server/path/to/file'), '\\\\server\\path\\to\\file') self.assertEqual(fn('/////server/path/to/file'), '\\\\server\\path\\to\\file') + self.assertEqual(fn('//127.0.0.1/path/to/file'), '\\\\127.0.0.1\\path\\to\\file') # Localhost paths self.assertEqual(fn('//localhost/C:/path/to/file'), 'C:\\path\\to\\file') self.assertEqual(fn('//localhost/C|/path/to/file'), 'C:\\path\\to\\file') @@ -1617,14 +1637,14 @@ def test_url2pathname_win(self): @unittest.skipIf(sys.platform == 'win32', 'test specific to POSIX pathnames') - @unittest.expectedFailure # AssertionError: '///foo/bar' != '/foo/bar' def test_url2pathname_posix(self): fn = urllib.request.url2pathname - self.assertEqual(fn('/foo/bar'), '/foo/bar') - self.assertEqual(fn('//foo/bar'), '//foo/bar') - self.assertEqual(fn('///foo/bar'), '/foo/bar') - self.assertEqual(fn('////foo/bar'), '//foo/bar') - self.assertEqual(fn('//localhost/foo/bar'), '/foo/bar') + self.assertRaises(urllib.error.URLError, fn, '//foo/bar') + self.assertRaises(urllib.error.URLError, fn, '//localhost:/foo/bar') + self.assertRaises(urllib.error.URLError, fn, '//:80/foo/bar') + self.assertRaises(urllib.error.URLError, fn, '//:/foo/bar') + self.assertRaises(urllib.error.URLError, fn, '//c:80/foo/bar') + self.assertRaises(urllib.error.URLError, fn, '//127.0.0.1/foo/bar') @unittest.skipUnless(os_helper.FS_NONASCII, 'need os_helper.FS_NONASCII') def test_url2pathname_nonascii(self): @@ -1643,56 +1663,6 @@ def test_thishost(self): self.assertIsInstance(urllib.request.thishost(), tuple) -class URLopener_Tests(FakeHTTPMixin, unittest.TestCase): - """Testcase to test the open method of URLopener class.""" - - def test_quoted_open(self): - class DummyURLopener(urllib.request.URLopener): - def open_spam(self, url): - return url - with warnings_helper.check_warnings( - ('DummyURLopener style of invoking requests is deprecated.', - DeprecationWarning)): - self.assertEqual(DummyURLopener().open( - 'spam://example/ /'),'//example/%20/') - - # test the safe characters are not quoted by urlopen - self.assertEqual(DummyURLopener().open( - "spam://c:|windows%/:=&?~#+!$,;'@()*[]|/path/"), - "//c:|windows%/:=&?~#+!$,;'@()*[]|/path/") - - @warnings_helper.ignore_warnings(category=DeprecationWarning) - def test_urlopener_retrieve_file(self): - with os_helper.temp_dir() as tmpdir: - fd, tmpfile = tempfile.mkstemp(dir=tmpdir) - os.close(fd) - fileurl = "file:" + urllib.request.pathname2url(tmpfile) - filename, _ = urllib.request.URLopener().retrieve(fileurl) - # Some buildbots have TEMP folder that uses a lowercase drive letter. - self.assertEqual(os.path.normcase(filename), os.path.normcase(tmpfile)) - - @warnings_helper.ignore_warnings(category=DeprecationWarning) - def test_urlopener_retrieve_remote(self): - url = "http://www.python.org/file.txt" - self.fakehttp(b"HTTP/1.1 200 OK\r\n\r\nHello!") - self.addCleanup(self.unfakehttp) - filename, _ = urllib.request.URLopener().retrieve(url) - self.assertEqual(os.path.splitext(filename)[1], ".txt") - - @warnings_helper.ignore_warnings(category=DeprecationWarning) - def test_local_file_open(self): - # bpo-35907, CVE-2019-9948: urllib must reject local_file:// scheme - class DummyURLopener(urllib.request.URLopener): - def open_local_file(self, url): - return url - for url in ('local_file://example', 'local-file://example'): - self.assertRaises(OSError, urllib.request.urlopen, url) - self.assertRaises(OSError, urllib.request.URLopener().open, url) - self.assertRaises(OSError, urllib.request.URLopener().retrieve, url) - self.assertRaises(OSError, DummyURLopener().open, url) - self.assertRaises(OSError, DummyURLopener().retrieve, url) - - class RequestTests(unittest.TestCase): """Unit tests for urllib.request.Request.""" diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py index 399c94213a6..7d7f2fa00d3 100644 --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -1,12 +1,14 @@ import unittest from test import support from test.support import os_helper -from test.support import socket_helper +from test.support import requires_subprocess from test.support import warnings_helper from test import test_urllib +from unittest import mock import os import io +import ftplib import socket import array import sys @@ -14,16 +16,20 @@ import subprocess import urllib.request -# The proxy bypass method imported below has logic specific to the OSX -# proxy config data structure but is testable on all platforms. +# The proxy bypass method imported below has logic specific to the +# corresponding system but is testable on all platforms. from urllib.request import (Request, OpenerDirector, HTTPBasicAuthHandler, HTTPPasswordMgrWithPriorAuth, _parse_proxy, + _proxy_bypass_winreg_override, _proxy_bypass_macosx_sysconf, AbstractDigestAuthHandler) -from urllib.parse import urlparse +from urllib.parse import urlsplit import urllib.error import http.client + +support.requires_working_socket(module=True) + # XXX # Request # CacheFTPHandler (hard to write) @@ -38,10 +44,6 @@ def test___all__(self): context = {} exec('from urllib.%s import *' % module, context) del context['__builtins__'] - if module == 'request' and os.name == 'nt': - u, p = context.pop('url2pathname'), context.pop('pathname2url') - self.assertEqual(u.__module__, 'nturl2path') - self.assertEqual(p.__module__, 'nturl2path') for k, v in context.items(): self.assertEqual(v.__module__, 'urllib.%s' % module, "%r is exposed in 'urllib.%s' but defined in %r" % @@ -483,7 +485,18 @@ def build_test_opener(*handler_instances): return opener -class MockHTTPHandler(urllib.request.BaseHandler): +class MockHTTPHandler(urllib.request.HTTPHandler): + # Very simple mock HTTP handler with no special behavior other than using a mock HTTP connection + + def __init__(self, debuglevel=None): + super(MockHTTPHandler, self).__init__(debuglevel=debuglevel) + self.httpconn = MockHTTPClass() + + def http_open(self, req): + return self.do_open(self.httpconn, req) + + +class MockHTTPHandlerRedirect(urllib.request.BaseHandler): # useful for testing redirections and auth # sends supplied headers and code as first response # sends 200 OK as second response @@ -511,16 +524,17 @@ def http_open(self, req): return MockResponse(200, "OK", msg, "", req.get_full_url()) -class MockHTTPSHandler(urllib.request.AbstractHTTPHandler): - # Useful for testing the Proxy-Authorization request by verifying the - # properties of httpcon +if hasattr(http.client, 'HTTPSConnection'): + class MockHTTPSHandler(urllib.request.HTTPSHandler): + # Useful for testing the Proxy-Authorization request by verifying the + # properties of httpcon - def __init__(self, debuglevel=0): - urllib.request.AbstractHTTPHandler.__init__(self, debuglevel=debuglevel) - self.httpconn = MockHTTPClass() + def __init__(self, debuglevel=None, context=None, check_hostname=None): + super(MockHTTPSHandler, self).__init__(debuglevel, context, check_hostname) + self.httpconn = MockHTTPClass() - def https_open(self, req): - return self.do_open(self.httpconn, req) + def https_open(self, req): + return self.do_open(self.httpconn, req) class MockHTTPHandlerCheckAuth(urllib.request.BaseHandler): @@ -700,18 +714,6 @@ def test_processors(self): self.assertIsInstance(args[1], MockResponse) -def sanepathname2url(path): - try: - path.encode("utf-8") - except UnicodeEncodeError: - raise unittest.SkipTest("path is not encodable to utf8") - urlpath = urllib.request.pathname2url(path) - if os.name == "nt" and urlpath.startswith("///"): - urlpath = urlpath[2:] - # XXX don't ask me about the mac... - return urlpath - - class HandlerTests(unittest.TestCase): def test_ftp(self): @@ -738,7 +740,6 @@ def connect_ftp(self, user, passwd, host, port, dirs, self.ftpwrapper = MockFTPWrapper(self.data) return self.ftpwrapper - import ftplib data = "rheum rhaponicum" h = NullFTPHandler(data) h.parent = MockOpener() @@ -761,7 +762,7 @@ def connect_ftp(self, user, passwd, host, port, dirs, ["foo", "bar"], "", None), ("ftp://localhost/baz.gif;type=a", "localhost", ftplib.FTP_PORT, "", "", "A", - [], "baz.gif", None), # XXX really this should guess image/gif + [], "baz.gif", "image/gif"), ]: req = Request(url) req.timeout = None @@ -777,6 +778,29 @@ def connect_ftp(self, user, passwd, host, port, dirs, headers = r.info() self.assertEqual(headers.get("Content-type"), mimetype) self.assertEqual(int(headers["Content-length"]), len(data)) + r.close() + + @support.requires_resource("network") + def test_ftp_error(self): + class ErrorFTPHandler(urllib.request.FTPHandler): + def __init__(self, exception): + self._exception = exception + + def connect_ftp(self, user, passwd, host, port, dirs, + timeout=socket._GLOBAL_DEFAULT_TIMEOUT): + raise self._exception + + exception = ftplib.error_perm( + "500 OOPS: cannot change directory:/nonexistent") + h = ErrorFTPHandler(exception) + urlopen = urllib.request.build_opener(h).open + try: + urlopen("ftp://www.pythontest.net/") + except urllib.error.URLError as raised: + self.assertEqual(raised.reason, + f"ftp error: {exception.args[0]}") + else: + self.fail("Did not raise ftplib exception") def test_file(self): import email.utils @@ -784,19 +808,22 @@ def test_file(self): o = h.parent = MockOpener() TESTFN = os_helper.TESTFN - urlpath = sanepathname2url(os.path.abspath(TESTFN)) towrite = b"hello, world\n" + canonurl = urllib.request.pathname2url(os.path.abspath(TESTFN), add_scheme=True) + parsed = urlsplit(canonurl) + if parsed.netloc: + raise unittest.SkipTest("non-local working directory") urls = [ - "file://localhost%s" % urlpath, - "file://%s" % urlpath, - "file://%s%s" % (socket.gethostbyname('localhost'), urlpath), + canonurl, + parsed._replace(netloc='localhost').geturl(), + parsed._replace(netloc=socket.gethostbyname('localhost')).geturl(), ] try: localaddr = socket.gethostbyname(socket.gethostname()) except socket.gaierror: localaddr = '' if localaddr: - urls.append("file://%s%s" % (localaddr, urlpath)) + urls.append(parsed._replace(netloc=localaddr).geturl()) for url in urls: f = open(TESTFN, "wb") @@ -821,10 +848,10 @@ def test_file(self): self.assertEqual(headers["Content-type"], "text/plain") self.assertEqual(headers["Content-length"], "13") self.assertEqual(headers["Last-modified"], modified) - self.assertEqual(respurl, url) + self.assertEqual(respurl, canonurl) for url in [ - "file://localhost:80%s" % urlpath, + parsed._replace(netloc='localhost:80').geturl(), "file:///file_does_not_exist.txt", "file://not-a-local-host.com//dir/file.txt", "file://%s:80%s/%s" % (socket.gethostbyname('localhost'), @@ -984,6 +1011,7 @@ def test_http_body_fileobj(self): file_obj.close() + @requires_subprocess() def test_http_body_pipe(self): # A file reading from a pipe. # A pipe cannot be seek'ed. There is no way to determine the @@ -1047,12 +1075,37 @@ def test_http_body_array(self): newreq = h.do_request_(req) self.assertEqual(int(newreq.get_header('Content-length')),16) - def test_http_handler_debuglevel(self): + def test_http_handler_global_debuglevel(self): + with mock.patch.object(http.client.HTTPConnection, 'debuglevel', 6): + o = OpenerDirector() + h = MockHTTPHandler() + o.add_handler(h) + o.open("http://www.example.com") + self.assertEqual(h._debuglevel, 6) + + def test_http_handler_local_debuglevel(self): + o = OpenerDirector() + h = MockHTTPHandler(debuglevel=5) + o.add_handler(h) + o.open("http://www.example.com") + self.assertEqual(h._debuglevel, 5) + + @unittest.skipUnless(hasattr(http.client, 'HTTPSConnection'), 'HTTPSConnection required for HTTPS tests.') + def test_https_handler_global_debuglevel(self): + with mock.patch.object(http.client.HTTPSConnection, 'debuglevel', 7): + o = OpenerDirector() + h = MockHTTPSHandler() + o.add_handler(h) + o.open("https://www.example.com") + self.assertEqual(h._debuglevel, 7) + + @unittest.skipUnless(hasattr(http.client, 'HTTPSConnection'), 'HTTPSConnection required for HTTPS tests.') + def test_https_handler_local_debuglevel(self): o = OpenerDirector() - h = MockHTTPSHandler(debuglevel=1) + h = MockHTTPSHandler(debuglevel=4) o.add_handler(h) o.open("https://www.example.com") - self.assertEqual(h._debuglevel, 1) + self.assertEqual(h._debuglevel, 4) def test_http_doubleslash(self): # Checks the presence of any unnecessary double slash in url does not @@ -1096,13 +1149,13 @@ def test_full_url_setter(self): r = Request('http://example.com') for url in urls: r.full_url = url - parsed = urlparse(url) + parsed = urlsplit(url) self.assertEqual(r.get_full_url(), url) # full_url setter uses splittag to split into components. # splittag sets the fragment as None while urlparse sets it to '' self.assertEqual(r.fragment or '', parsed.fragment) - self.assertEqual(urlparse(r.get_full_url()).query, parsed.query) + self.assertEqual(urlsplit(r.get_full_url()).query, parsed.query) def test_full_url_deleter(self): r = Request('http://www.example.com') @@ -1140,15 +1193,15 @@ def test_errors(self): r = MockResponse(200, "OK", {}, "", url) newr = h.http_response(req, r) self.assertIs(r, newr) - self.assertFalse(hasattr(o, "proto")) # o.error not called + self.assertNotHasAttr(o, "proto") # o.error not called r = MockResponse(202, "Accepted", {}, "", url) newr = h.http_response(req, r) self.assertIs(r, newr) - self.assertFalse(hasattr(o, "proto")) # o.error not called + self.assertNotHasAttr(o, "proto") # o.error not called r = MockResponse(206, "Partial content", {}, "", url) newr = h.http_response(req, r) self.assertIs(r, newr) - self.assertFalse(hasattr(o, "proto")) # o.error not called + self.assertNotHasAttr(o, "proto") # o.error not called # anything else calls o.error (and MockOpener returns None, here) r = MockResponse(502, "Bad gateway", {}, "", url) self.assertIsNone(h.http_response(req, r)) @@ -1179,7 +1232,7 @@ def test_redirect(self): o = h.parent = MockOpener() # ordinary redirect behaviour - for code in 301, 302, 303, 307: + for code in 301, 302, 303, 307, 308: for data in None, "blah\nblah\n": method = getattr(h, "http_error_%s" % code) req = Request(from_url, data) @@ -1191,10 +1244,11 @@ def test_redirect(self): try: method(req, MockFile(), code, "Blah", MockHeaders({"location": to_url})) - except urllib.error.HTTPError: - # 307 in response to POST requires user OK - self.assertEqual(code, 307) + except urllib.error.HTTPError as err: + # 307 and 308 in response to POST require user OK + self.assertIn(code, (307, 308)) self.assertIsNotNone(data) + err.close() self.assertEqual(o.req.get_full_url(), to_url) try: self.assertEqual(o.req.get_method(), "GET") @@ -1230,9 +1284,10 @@ def redirect(h, req, url=to_url): while 1: redirect(h, req, "http://example.com/") count = count + 1 - except urllib.error.HTTPError: + except urllib.error.HTTPError as err: # don't stop until max_repeats, because cookies may introduce state self.assertEqual(count, urllib.request.HTTPRedirectHandler.max_repeats) + err.close() # detect endless non-repeating chain of redirects req = Request(from_url, origin_req_host="example.com") @@ -1242,9 +1297,10 @@ def redirect(h, req, url=to_url): while 1: redirect(h, req, "http://example.com/%d" % count) count = count + 1 - except urllib.error.HTTPError: + except urllib.error.HTTPError as err: self.assertEqual(count, urllib.request.HTTPRedirectHandler.max_redirections) + err.close() def test_invalid_redirect(self): from_url = "http://example.com/a.html" @@ -1258,9 +1314,11 @@ def test_invalid_redirect(self): for scheme in invalid_schemes: invalid_url = scheme + '://' + schemeless_url - self.assertRaises(urllib.error.HTTPError, h.http_error_302, + with self.assertRaises(urllib.error.HTTPError) as cm: + h.http_error_302( req, MockFile(), 302, "Security Loophole", MockHeaders({"location": invalid_url})) + cm.exception.close() for scheme in valid_schemes: valid_url = scheme + '://' + schemeless_url @@ -1288,7 +1346,7 @@ def test_cookie_redirect(self): cj = CookieJar() interact_netscape(cj, "http://www.example.com/", "spam=eggs") - hh = MockHTTPHandler(302, "Location: http://www.cracker.com/\r\n\r\n") + hh = MockHTTPHandlerRedirect(302, "Location: http://www.cracker.com/\r\n\r\n") hdeh = urllib.request.HTTPDefaultErrorHandler() hrh = urllib.request.HTTPRedirectHandler() cp = urllib.request.HTTPCookieProcessor(cj) @@ -1298,7 +1356,7 @@ def test_cookie_redirect(self): def test_redirect_fragment(self): redirected_url = 'http://www.example.com/index.html#OK\r\n\r\n' - hh = MockHTTPHandler(302, 'Location: ' + redirected_url) + hh = MockHTTPHandlerRedirect(302, 'Location: ' + redirected_url) hdeh = urllib.request.HTTPDefaultErrorHandler() hrh = urllib.request.HTTPRedirectHandler() o = build_test_opener(hh, hdeh, hrh) @@ -1358,7 +1416,16 @@ def http_open(self, req): response = opener.open('http://example.com/') expected = b'GET ' + result + b' ' request = handler.last_buf - self.assertTrue(request.startswith(expected), repr(request)) + self.assertStartsWith(request, expected) + + def test_redirect_head_request(self): + from_url = "http://example.com/a.html" + to_url = "http://example.com/b.html" + h = urllib.request.HTTPRedirectHandler() + req = Request(from_url, method="HEAD") + fp = MockFile() + new_req = h.redirect_request(req, fp, 302, "Found", {}, to_url) + self.assertEqual(new_req.get_method(), "HEAD") def test_proxy(self): u = "proxy.example.com:3128" @@ -1379,7 +1446,8 @@ def test_proxy(self): [tup[0:2] for tup in o.calls]) def test_proxy_no_proxy(self): - os.environ['no_proxy'] = 'python.org' + env = self.enterContext(os_helper.EnvironmentVarGuard()) + env['no_proxy'] = 'python.org' o = OpenerDirector() ph = urllib.request.ProxyHandler(dict(http="proxy.example.com")) o.add_handler(ph) @@ -1391,10 +1459,10 @@ def test_proxy_no_proxy(self): self.assertEqual(req.host, "www.python.org") o.open(req) self.assertEqual(req.host, "www.python.org") - del os.environ['no_proxy'] def test_proxy_no_proxy_all(self): - os.environ['no_proxy'] = '*' + env = self.enterContext(os_helper.EnvironmentVarGuard()) + env['no_proxy'] = '*' o = OpenerDirector() ph = urllib.request.ProxyHandler(dict(http="proxy.example.com")) o.add_handler(ph) @@ -1402,7 +1470,6 @@ def test_proxy_no_proxy_all(self): self.assertEqual(req.host, "www.python.org") o.open(req) self.assertEqual(req.host, "www.python.org") - del os.environ['no_proxy'] def test_proxy_https(self): o = OpenerDirector() @@ -1420,6 +1487,7 @@ def test_proxy_https(self): self.assertEqual([(handlers[0], "https_open")], [tup[0:2] for tup in o.calls]) + @unittest.skipUnless(hasattr(http.client, 'HTTPSConnection'), 'HTTPSConnection required for HTTPS tests.') def test_proxy_https_proxy_authorization(self): o = OpenerDirector() ph = urllib.request.ProxyHandler(dict(https='proxy.example.com:3128')) @@ -1443,6 +1511,30 @@ def test_proxy_https_proxy_authorization(self): self.assertEqual(req.host, "proxy.example.com:3128") self.assertEqual(req.get_header("Proxy-authorization"), "FooBar") + @unittest.skipUnless(os.name == "nt", "only relevant for Windows") + def test_winreg_proxy_bypass(self): + proxy_override = "www.example.com;*.example.net; 192.168.0.1" + proxy_bypass = _proxy_bypass_winreg_override + for host in ("www.example.com", "www.example.net", "192.168.0.1"): + self.assertTrue(proxy_bypass(host, proxy_override), + "expected bypass of %s to be true" % host) + + for host in ("example.com", "www.example.org", "example.net", + "192.168.0.2"): + self.assertFalse(proxy_bypass(host, proxy_override), + "expected bypass of %s to be False" % host) + + # check intranet address bypass + proxy_override = "example.com; <local>" + self.assertTrue(proxy_bypass("example.com", proxy_override), + "expected bypass of %s to be true" % host) + self.assertFalse(proxy_bypass("example.net", proxy_override), + "expected bypass of %s to be False" % host) + for host in ("test", "localhost"): + self.assertTrue(proxy_bypass(host, proxy_override), + "expect <local> to bypass intranet address '%s'" + % host) + @unittest.skipUnless(sys.platform == 'darwin', "only relevant for OSX") def test_osx_proxy_bypass(self): bypass = { @@ -1483,7 +1575,7 @@ def check_basic_auth(self, headers, realm): password_manager = MockPasswordManager() auth_handler = urllib.request.HTTPBasicAuthHandler(password_manager) body = '\r\n'.join(headers) + '\r\n\r\n' - http_handler = MockHTTPHandler(401, body) + http_handler = MockHTTPHandlerRedirect(401, body) opener.add_handler(auth_handler) opener.add_handler(http_handler) self._test_basic_auth(opener, auth_handler, "Authorization", @@ -1543,7 +1635,7 @@ def test_proxy_basic_auth(self): password_manager = MockPasswordManager() auth_handler = urllib.request.ProxyBasicAuthHandler(password_manager) realm = "ACME Networks" - http_handler = MockHTTPHandler( + http_handler = MockHTTPHandlerRedirect( 407, 'Proxy-Authenticate: Basic realm="%s"\r\n\r\n' % realm) opener.add_handler(auth_handler) opener.add_handler(http_handler) @@ -1555,11 +1647,11 @@ def test_proxy_basic_auth(self): def test_basic_and_digest_auth_handlers(self): # HTTPDigestAuthHandler raised an exception if it couldn't handle a 40* - # response (http://python.org/sf/1479302), where it should instead + # response (https://bugs.python.org/issue1479302), where it should instead # return None to allow another handler (especially # HTTPBasicAuthHandler) to handle the response. - # Also (http://python.org/sf/14797027, RFC 2617 section 1.2), we must + # Also (https://bugs.python.org/issue14797027, RFC 2617 section 1.2), we must # try digest first (since it's the strongest auth scheme), so we record # order of calls here to check digest comes first: class RecordingOpenerDirector(OpenerDirector): @@ -1587,7 +1679,7 @@ def http_error_401(self, *args, **kwds): digest_handler = TestDigestAuthHandler(password_manager) basic_handler = TestBasicAuthHandler(password_manager) realm = "ACME Networks" - http_handler = MockHTTPHandler( + http_handler = MockHTTPHandlerRedirect( 401, 'WWW-Authenticate: Basic realm="%s"\r\n\r\n' % realm) opener.add_handler(basic_handler) opener.add_handler(digest_handler) @@ -1607,7 +1699,7 @@ def test_unsupported_auth_digest_handler(self): opener = OpenerDirector() # While using DigestAuthHandler digest_auth_handler = urllib.request.HTTPDigestAuthHandler(None) - http_handler = MockHTTPHandler( + http_handler = MockHTTPHandlerRedirect( 401, 'WWW-Authenticate: Kerberos\r\n\r\n') opener.add_handler(digest_auth_handler) opener.add_handler(http_handler) @@ -1617,7 +1709,7 @@ def test_unsupported_auth_basic_handler(self): # While using BasicAuthHandler opener = OpenerDirector() basic_auth_handler = urllib.request.HTTPBasicAuthHandler(None) - http_handler = MockHTTPHandler( + http_handler = MockHTTPHandlerRedirect( 401, 'WWW-Authenticate: NTLM\r\n\r\n') opener.add_handler(basic_auth_handler) opener.add_handler(http_handler) @@ -1704,7 +1796,7 @@ def test_basic_prior_auth_send_after_first_success(self): opener = OpenerDirector() opener.add_handler(auth_prior_handler) - http_handler = MockHTTPHandler( + http_handler = MockHTTPHandlerRedirect( 401, 'WWW-Authenticate: Basic realm="%s"\r\n\r\n' % None) opener.add_handler(http_handler) @@ -1814,14 +1906,21 @@ def test_HTTPError_interface(self): url = code = fp = None hdrs = 'Content-Length: 42' err = urllib.error.HTTPError(url, code, msg, hdrs, fp) - self.assertTrue(hasattr(err, 'reason')) + self.assertHasAttr(err, 'reason') self.assertEqual(err.reason, 'something bad happened') - self.assertTrue(hasattr(err, 'headers')) + self.assertHasAttr(err, 'headers') self.assertEqual(err.headers, 'Content-Length: 42') expected_errmsg = 'HTTP Error %s: %s' % (err.code, err.msg) self.assertEqual(str(err), expected_errmsg) expected_errmsg = '<HTTPError %s: %r>' % (err.code, err.msg) self.assertEqual(repr(err), expected_errmsg) + err.close() + + def test_gh_98778(self): + x = urllib.error.HTTPError("url", 405, "METHOD NOT ALLOWED", None, None) + self.assertEqual(getattr(x, "__notes__", ()), ()) + self.assertIsInstance(x.fp.read(), bytes) + x.close() def test_parse_proxy(self): parse_proxy_test_cases = [ @@ -1868,10 +1967,38 @@ def test_parse_proxy(self): self.assertRaises(ValueError, _parse_proxy, 'file:/ftp.example.com'), - def test_unsupported_algorithm(self): - handler = AbstractDigestAuthHandler() + +skip_libssl_fips_mode = unittest.skipIf( + support.is_libssl_fips_mode(), + "conservative skip due to OpenSSL FIPS mode possible algorithm nerfing", +) + + +class TestDigestAuthAlgorithms(unittest.TestCase): + def setUp(self): + self.handler = AbstractDigestAuthHandler() + + @skip_libssl_fips_mode + def test_md5_algorithm(self): + H, KD = self.handler.get_algorithm_impls('MD5') + self.assertEqual(H("foo"), "acbd18db4cc2f85cedef654fccc4a4d8") + self.assertEqual(KD("foo", "bar"), "4e99e8c12de7e01535248d2bac85e732") + + @skip_libssl_fips_mode + def test_sha_algorithm(self): + H, KD = self.handler.get_algorithm_impls('SHA') + self.assertEqual(H("foo"), "0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33") + self.assertEqual(KD("foo", "bar"), "54dcbe67d21d5eb39493d46d89ae1f412d3bd6de") + + @skip_libssl_fips_mode + def test_sha256_algorithm(self): + H, KD = self.handler.get_algorithm_impls('SHA-256') + self.assertEqual(H("foo"), "2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae") + self.assertEqual(KD("foo", "bar"), "a765a8beaa9d561d4c5cbed29d8f4e30870297fdfa9cb7d6e9848a95fec9f937") + + def test_invalid_algorithm(self): with self.assertRaises(ValueError) as exc: - handler.get_algorithm_impls('invalid') + self.handler.get_algorithm_impls('invalid') self.assertEqual( str(exc.exception), "Unsupported digest authentication algorithm 'invalid'" diff --git a/Lib/test/test_urllib2_localnet.py b/Lib/test/test_urllib2_localnet.py index 2c54ef85b4b..d70c0dad7f0 100644 --- a/Lib/test/test_urllib2_localnet.py +++ b/Lib/test/test_urllib2_localnet.py @@ -8,15 +8,17 @@ import unittest import hashlib +from test import support from test.support import hashlib_helper from test.support import threading_helper -from test.support import warnings_helper try: import ssl except ImportError: ssl = None +support.requires_working_socket(module=True) + here = os.path.dirname(__file__) # Self-signed cert file for 'localhost' CERT_localhost = os.path.join(here, 'certdata', 'keycert.pem') @@ -314,7 +316,9 @@ def test_basic_auth_httperror(self): ah = urllib.request.HTTPBasicAuthHandler() ah.add_password(self.REALM, self.server_url, self.USER, self.INCORRECT_PASSWD) urllib.request.install_opener(urllib.request.build_opener(ah)) - self.assertRaises(urllib.error.HTTPError, urllib.request.urlopen, self.server_url) + with self.assertRaises(urllib.error.HTTPError) as cm: + urllib.request.urlopen(self.server_url) + cm.exception.close() @hashlib_helper.requires_hashdigest("md5", openssl=True) @@ -356,23 +360,23 @@ def stop_server(self): self.server.stop() self.server = None - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_proxy_with_bad_password_raises_httperror(self): self.proxy_digest_handler.add_password(self.REALM, self.URL, self.USER, self.PASSWD+"bad") self.digest_auth_handler.set_qop("auth") - self.assertRaises(urllib.error.HTTPError, - self.opener.open, - self.URL) + with self.assertRaises(urllib.error.HTTPError) as cm: + self.opener.open(self.URL) + cm.exception.close() - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_proxy_with_no_password_raises_httperror(self): self.digest_auth_handler.set_qop("auth") - self.assertRaises(urllib.error.HTTPError, - self.opener.open, - self.URL) + with self.assertRaises(urllib.error.HTTPError) as cm: + self.opener.open(self.URL) + cm.exception.close() - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_proxy_qop_auth_works(self): self.proxy_digest_handler.add_password(self.REALM, self.URL, self.USER, self.PASSWD) @@ -381,7 +385,7 @@ def test_proxy_qop_auth_works(self): while result.read(): pass - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_proxy_qop_auth_int_works_or_throws_urlerror(self): self.proxy_digest_handler.add_password(self.REALM, self.URL, self.USER, self.PASSWD) @@ -506,7 +510,7 @@ def start_https_server(self, responses=None, **kwargs): handler.port = server.port return handler - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_redirection(self): expected_response = b"We got here..." responses = [ @@ -520,7 +524,7 @@ def test_redirection(self): self.assertEqual(data, expected_response) self.assertEqual(handler.requests, ["/", "/somewhere_else"]) - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_chunked(self): expected_response = b"hello world" chunked_start = ( @@ -535,7 +539,7 @@ def test_chunked(self): data = self.urlopen("http://localhost:%s/" % handler.port) self.assertEqual(data, expected_response) - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_404(self): expected_response = b"Bad bad bad..." handler = self.start_server([(404, [], expected_response)]) @@ -551,7 +555,7 @@ def test_404(self): self.assertEqual(data, expected_response) self.assertEqual(handler.requests, ["/weeble"]) - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_200(self): expected_response = b"pycon 2008..." handler = self.start_server([(200, [], expected_response)]) @@ -559,7 +563,7 @@ def test_200(self): self.assertEqual(data, expected_response) self.assertEqual(handler.requests, ["/bizarre"]) - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_200_with_parameters(self): expected_response = b"pycon 2008..." handler = self.start_server([(200, [], expected_response)]) @@ -568,41 +572,14 @@ def test_200_with_parameters(self): self.assertEqual(data, expected_response) self.assertEqual(handler.requests, ["/bizarre", b"get=with_feeling"]) - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_https(self): handler = self.start_https_server() context = ssl.create_default_context(cafile=CERT_localhost) data = self.urlopen("https://localhost:%s/bizarre" % handler.port, context=context) self.assertEqual(data, b"we care a bit") - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") - def test_https_with_cafile(self): - handler = self.start_https_server(certfile=CERT_localhost) - with warnings_helper.check_warnings(('', DeprecationWarning)): - # Good cert - data = self.urlopen("https://localhost:%s/bizarre" % handler.port, - cafile=CERT_localhost) - self.assertEqual(data, b"we care a bit") - # Bad cert - with self.assertRaises(urllib.error.URLError) as cm: - self.urlopen("https://localhost:%s/bizarre" % handler.port, - cafile=CERT_fakehostname) - # Good cert, but mismatching hostname - handler = self.start_https_server(certfile=CERT_fakehostname) - with self.assertRaises(urllib.error.URLError) as cm: - self.urlopen("https://localhost:%s/bizarre" % handler.port, - cafile=CERT_fakehostname) - - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") - def test_https_with_cadefault(self): - handler = self.start_https_server(certfile=CERT_localhost) - # Self-signed cert should fail verification with system certificate store - with warnings_helper.check_warnings(('', DeprecationWarning)): - with self.assertRaises(urllib.error.URLError) as cm: - self.urlopen("https://localhost:%s/bizarre" % handler.port, - cadefault=True) - - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_https_sni(self): if ssl is None: self.skipTest("ssl module required") @@ -619,7 +596,7 @@ def cb_sni(ssl_sock, server_name, initial_context): self.urlopen("https://localhost:%s" % handler.port, context=context) self.assertEqual(sni_name, "localhost") - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_sending_headers(self): handler = self.start_server() req = urllib.request.Request("http://localhost:%s/" % handler.port, @@ -628,7 +605,7 @@ def test_sending_headers(self): pass self.assertEqual(handler.headers_received["Range"], "bytes=20-39") - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_sending_headers_camel(self): handler = self.start_server() req = urllib.request.Request("http://localhost:%s/" % handler.port, @@ -638,16 +615,15 @@ def test_sending_headers_camel(self): self.assertIn("X-Some-Header", handler.headers_received.keys()) self.assertNotIn("X-SoMe-hEader", handler.headers_received.keys()) - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_basic(self): handler = self.start_server() with urllib.request.urlopen("http://localhost:%s" % handler.port) as open_url: for attr in ("read", "close", "info", "geturl"): - self.assertTrue(hasattr(open_url, attr), "object returned from " - "urlopen lacks the %s attribute" % attr) + self.assertHasAttr(open_url, attr) self.assertTrue(open_url.read(), "calling 'read' failed") - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_info(self): handler = self.start_server() open_url = urllib.request.urlopen( @@ -659,7 +635,7 @@ def test_info(self): "instance of email.message.Message") self.assertEqual(info_obj.get_content_subtype(), "plain") - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_geturl(self): # Make sure same URL as opened is returned by geturl. handler = self.start_server() @@ -668,7 +644,7 @@ def test_geturl(self): url = open_url.geturl() self.assertEqual(url, "http://localhost:%s" % handler.port) - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_iteration(self): expected_response = b"pycon 2008..." handler = self.start_server([(200, [], expected_response)]) @@ -676,7 +652,7 @@ def test_iteration(self): for line in data: self.assertEqual(line, expected_response) - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_line_iteration(self): lines = [b"We\n", b"got\n", b"here\n", b"verylong " * 8192 + b"\n"] expected_response = b"".join(lines) @@ -689,7 +665,7 @@ def test_line_iteration(self): (index, len(lines[index]), len(line))) self.assertEqual(index + 1, len(lines)) - @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON, ValueError: illegal environment variable name") + @unittest.skipIf(os.name == "nt", "TODO: RUSTPYTHON; ValueError: illegal environment variable name") def test_issue16464(self): # See https://bugs.python.org/issue16464 # and https://bugs.python.org/issue46648 diff --git a/Lib/test/test_urllib2net.py b/Lib/test/test_urllib2net.py index 41f170a6ad5..d015267cefd 100644 --- a/Lib/test/test_urllib2net.py +++ b/Lib/test/test_urllib2net.py @@ -7,7 +7,6 @@ from test.support import os_helper from test.support import socket_helper from test.support import ResourceDenied -from test.test_urllib2 import sanepathname2url from test.support.warnings_helper import check_no_resource_warning import os @@ -192,7 +191,7 @@ def test_file(self): f.write('hi there\n') f.close() urls = [ - 'file:' + sanepathname2url(os.path.abspath(TESTFN)), + urllib.request.pathname2url(os.path.abspath(TESTFN), add_scheme=True), ('file:///nonsensename/etc/passwd', None, urllib.error.URLError), ] diff --git a/Lib/test/test_urllib_response.py b/Lib/test/test_urllib_response.py index 73d2ef0424f..d949fa38bfc 100644 --- a/Lib/test/test_urllib_response.py +++ b/Lib/test/test_urllib_response.py @@ -4,6 +4,11 @@ import tempfile import urllib.response import unittest +from test import support + +if support.is_wasi: + raise unittest.SkipTest("Cannot create socket on WASI") + class TestResponse(unittest.TestCase): @@ -43,6 +48,7 @@ def test_addinfo(self): info = urllib.response.addinfo(self.fp, self.test_headers) self.assertEqual(info.info(), self.test_headers) self.assertEqual(info.headers, self.test_headers) + info.close() def test_addinfourl(self): url = "http://www.python.org" @@ -55,6 +61,7 @@ def test_addinfourl(self): self.assertEqual(infourl.headers, self.test_headers) self.assertEqual(infourl.url, url) self.assertEqual(infourl.status, code) + infourl.close() def tearDown(self): self.sock.close() diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py index 6733fe9c6ea..1a42c35dc49 100644 --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -2,10 +2,10 @@ from test import support from test.support import os_helper from test.support import socket_helper -from test.support.testcase import ExtraAssertions import contextlib import socket +import urllib.error import urllib.parse import urllib.request import os @@ -35,7 +35,7 @@ def testURLread(self): f.read() -class urlopenNetworkTests(unittest.TestCase, ExtraAssertions): +class urlopenNetworkTests(unittest.TestCase): """Tests urllib.request.urlopen using the network. These tests are not exhaustive. Assuming that testing using files does a @@ -101,13 +101,11 @@ def test_getcode(self): # test getcode() with the fancy opener to get 404 error codes URL = self.url + "XXXinvalidXXX" with socket_helper.transient_internet(URL): - with self.assertWarns(DeprecationWarning): - open_url = urllib.request.FancyURLopener().open(URL) - try: - code = open_url.getcode() - finally: - open_url.close() - self.assertEqual(code, 404) + with self.assertRaises(urllib.error.URLError) as e: + with urllib.request.urlopen(URL): + pass + self.assertEqual(e.exception.code, 404) + e.exception.close() @support.requires_resource('walltime') def test_bad_address(self): diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py index af6fe99fb51..5e6a46b3d35 100644 --- a/Lib/test/test_urlparse.py +++ b/Lib/test/test_urlparse.py @@ -2,6 +2,7 @@ import unicodedata import unittest import urllib.parse +from test import support RFC1808_BASE = "http://a/b/c/d;p?q#f" RFC2396_BASE = "http://a/b/c/d;p?q" @@ -19,6 +20,10 @@ ("=a", [('', 'a')]), ("a", [('a', '')]), ("a=", [('a', '')]), + ("a=b=c", [('a', 'b=c')]), + ("a%3Db=c", [('a=b', 'c')]), + ("a=b&c=d", [('a', 'b'), ('c', 'd')]), + ("a=b%26c=d", [('a', 'b&c=d')]), ("&a=b", [('a', 'b')]), ("a=a+b&b=b+c", [('a', 'a b'), ('b', 'b c')]), ("a=1&a=2", [('a', '1'), ('a', '2')]), @@ -29,6 +34,10 @@ (b"=a", [(b'', b'a')]), (b"a", [(b'a', b'')]), (b"a=", [(b'a', b'')]), + (b"a=b=c", [(b'a', b'b=c')]), + (b"a%3Db=c", [(b'a=b', b'c')]), + (b"a=b&c=d", [(b'a', b'b'), (b'c', b'd')]), + (b"a=b%26c=d", [(b'a', b'b&c=d')]), (b"&a=b", [(b'a', b'b')]), (b"a=a+b&b=b+c", [(b'a', b'a b'), (b'b', b'b c')]), (b"a=1&a=2", [(b'a', b'1'), (b'a', b'2')]), @@ -36,6 +45,14 @@ ("a=a+b;b=b+c", [('a', 'a b;b=b c')]), (b";a=b", [(b';a', b'b')]), (b"a=a+b;b=b+c", [(b'a', b'a b;b=b c')]), + + ("\u0141=\xE9", [('\u0141', '\xE9')]), + ("%C5%81=%C3%A9", [('\u0141', '\xE9')]), + ("%81=%A9", [('\ufffd', '\ufffd')]), + (b"\xc5\x81=\xc3\xa9", [(b'\xc5\x81', b'\xc3\xa9')]), + (b"%C5%81=%C3%A9", [(b'\xc5\x81', b'\xc3\xa9')]), + (b"\x81=\xA9", [(b'\x81', b'\xa9')]), + (b"%81=%A9", [(b'\x81', b'\xa9')]), ] # Each parse_qs testcase is a two-tuple that contains @@ -49,6 +66,10 @@ ("=a", {'': ['a']}), ("a", {'a': ['']}), ("a=", {'a': ['']}), + ("a=b=c", {'a': ['b=c']}), + ("a%3Db=c", {'a=b': ['c']}), + ("a=b&c=d", {'a': ['b'], 'c': ['d']}), + ("a=b%26c=d", {'a': ['b&c=d']}), ("&a=b", {'a': ['b']}), ("a=a+b&b=b+c", {'a': ['a b'], 'b': ['b c']}), ("a=1&a=2", {'a': ['1', '2']}), @@ -59,6 +80,10 @@ (b"=a", {b'': [b'a']}), (b"a", {b'a': [b'']}), (b"a=", {b'a': [b'']}), + (b"a=b=c", {b'a': [b'b=c']}), + (b"a%3Db=c", {b'a=b': [b'c']}), + (b"a=b&c=d", {b'a': [b'b'], b'c': [b'd']}), + (b"a=b%26c=d", {b'a': [b'b&c=d']}), (b"&a=b", {b'a': [b'b']}), (b"a=a+b&b=b+c", {b'a': [b'a b'], b'b': [b'b c']}), (b"a=1&a=2", {b'a': [b'1', b'2']}), @@ -66,26 +91,37 @@ ("a=a+b;b=b+c", {'a': ['a b;b=b c']}), (b";a=b", {b';a': [b'b']}), (b"a=a+b;b=b+c", {b'a':[ b'a b;b=b c']}), + (b"a=a%E2%80%99b", {b'a': [b'a\xe2\x80\x99b']}), + + ("\u0141=\xE9", {'\u0141': ['\xE9']}), + ("%C5%81=%C3%A9", {'\u0141': ['\xE9']}), + ("%81=%A9", {'\ufffd': ['\ufffd']}), + (b"\xc5\x81=\xc3\xa9", {b'\xc5\x81': [b'\xc3\xa9']}), + (b"%C5%81=%C3%A9", {b'\xc5\x81': [b'\xc3\xa9']}), + (b"\x81=\xA9", {b'\x81': [b'\xa9']}), + (b"%81=%A9", {b'\x81': [b'\xa9']}), ] class UrlParseTestCase(unittest.TestCase): - def checkRoundtrips(self, url, parsed, split): + def checkRoundtrips(self, url, parsed, split, url2=None): + if url2 is None: + url2 = url result = urllib.parse.urlparse(url) - self.assertEqual(result, parsed) + self.assertSequenceEqual(result, parsed) t = (result.scheme, result.netloc, result.path, result.params, result.query, result.fragment) - self.assertEqual(t, parsed) + self.assertSequenceEqual(t, parsed) # put it back together and it should be the same result2 = urllib.parse.urlunparse(result) - self.assertEqual(result2, url) - self.assertEqual(result2, result.geturl()) + self.assertSequenceEqual(result2, url2) + self.assertSequenceEqual(result2, result.geturl()) # the result of geturl() is a fixpoint; we can always parse it # again to get the same result: result3 = urllib.parse.urlparse(result.geturl()) self.assertEqual(result3.geturl(), result.geturl()) - self.assertEqual(result3, result) + self.assertSequenceEqual(result3, result) self.assertEqual(result3.scheme, result.scheme) self.assertEqual(result3.netloc, result.netloc) self.assertEqual(result3.path, result.path) @@ -99,18 +135,18 @@ def checkRoundtrips(self, url, parsed, split): # check the roundtrip using urlsplit() as well result = urllib.parse.urlsplit(url) - self.assertEqual(result, split) + self.assertSequenceEqual(result, split) t = (result.scheme, result.netloc, result.path, result.query, result.fragment) - self.assertEqual(t, split) + self.assertSequenceEqual(t, split) result2 = urllib.parse.urlunsplit(result) - self.assertEqual(result2, url) - self.assertEqual(result2, result.geturl()) + self.assertSequenceEqual(result2, url2) + self.assertSequenceEqual(result2, result.geturl()) # check the fixpoint property of re-parsing the result of geturl() result3 = urllib.parse.urlsplit(result.geturl()) self.assertEqual(result3.geturl(), result.geturl()) - self.assertEqual(result3, result) + self.assertSequenceEqual(result3, result) self.assertEqual(result3.scheme, result.scheme) self.assertEqual(result3.netloc, result.netloc) self.assertEqual(result3.path, result.path) @@ -121,30 +157,79 @@ def checkRoundtrips(self, url, parsed, split): self.assertEqual(result3.hostname, result.hostname) self.assertEqual(result3.port, result.port) - def test_qsl(self): - for orig, expect in parse_qsl_test_cases: - result = urllib.parse.parse_qsl(orig, keep_blank_values=True) - self.assertEqual(result, expect, "Error parsing %r" % orig) - expect_without_blanks = [v for v in expect if len(v[1])] - result = urllib.parse.parse_qsl(orig, keep_blank_values=False) - self.assertEqual(result, expect_without_blanks, - "Error parsing %r" % orig) - - def test_qs(self): - for orig, expect in parse_qs_test_cases: - result = urllib.parse.parse_qs(orig, keep_blank_values=True) - self.assertEqual(result, expect, "Error parsing %r" % orig) - expect_without_blanks = {v: expect[v] - for v in expect if len(expect[v][0])} - result = urllib.parse.parse_qs(orig, keep_blank_values=False) - self.assertEqual(result, expect_without_blanks, - "Error parsing %r" % orig) - - def test_roundtrips(self): - str_cases = [ + @support.subTests('orig,expect', parse_qsl_test_cases) + def test_qsl(self, orig, expect): + result = urllib.parse.parse_qsl(orig, keep_blank_values=True) + self.assertEqual(result, expect) + expect_without_blanks = [v for v in expect if len(v[1])] + result = urllib.parse.parse_qsl(orig, keep_blank_values=False) + self.assertEqual(result, expect_without_blanks) + + @support.subTests('orig,expect', parse_qs_test_cases) + def test_qs(self, orig, expect): + result = urllib.parse.parse_qs(orig, keep_blank_values=True) + self.assertEqual(result, expect) + expect_without_blanks = {v: expect[v] + for v in expect if len(expect[v][0])} + result = urllib.parse.parse_qs(orig, keep_blank_values=False) + self.assertEqual(result, expect_without_blanks) + + @support.subTests('bytes', (False, True)) + @support.subTests('url,parsed,split', [ + ('path/to/file', + ('', '', 'path/to/file', '', '', ''), + ('', '', 'path/to/file', '', '')), + ('/path/to/file', + ('', '', '/path/to/file', '', '', ''), + ('', '', '/path/to/file', '', '')), + ('//path/to/file', + ('', 'path', '/to/file', '', '', ''), + ('', 'path', '/to/file', '', '')), + ('////path/to/file', + ('', '', '//path/to/file', '', '', ''), + ('', '', '//path/to/file', '', '')), + ('/////path/to/file', + ('', '', '///path/to/file', '', '', ''), + ('', '', '///path/to/file', '', '')), + ('scheme:path/to/file', + ('scheme', '', 'path/to/file', '', '', ''), + ('scheme', '', 'path/to/file', '', '')), + ('scheme:/path/to/file', + ('scheme', '', '/path/to/file', '', '', ''), + ('scheme', '', '/path/to/file', '', '')), + ('scheme://path/to/file', + ('scheme', 'path', '/to/file', '', '', ''), + ('scheme', 'path', '/to/file', '', '')), + ('scheme:////path/to/file', + ('scheme', '', '//path/to/file', '', '', ''), + ('scheme', '', '//path/to/file', '', '')), + ('scheme://///path/to/file', + ('scheme', '', '///path/to/file', '', '', ''), + ('scheme', '', '///path/to/file', '', '')), + ('file:tmp/junk.txt', + ('file', '', 'tmp/junk.txt', '', '', ''), + ('file', '', 'tmp/junk.txt', '', '')), ('file:///tmp/junk.txt', ('file', '', '/tmp/junk.txt', '', '', ''), ('file', '', '/tmp/junk.txt', '', '')), + ('file:////tmp/junk.txt', + ('file', '', '//tmp/junk.txt', '', '', ''), + ('file', '', '//tmp/junk.txt', '', '')), + ('file://///tmp/junk.txt', + ('file', '', '///tmp/junk.txt', '', '', ''), + ('file', '', '///tmp/junk.txt', '', '')), + ('http:tmp/junk.txt', + ('http', '', 'tmp/junk.txt', '', '', ''), + ('http', '', 'tmp/junk.txt', '', '')), + ('http://example.com/tmp/junk.txt', + ('http', 'example.com', '/tmp/junk.txt', '', '', ''), + ('http', 'example.com', '/tmp/junk.txt', '', '')), + ('http:///example.com/tmp/junk.txt', + ('http', '', '/example.com/tmp/junk.txt', '', '', ''), + ('http', '', '/example.com/tmp/junk.txt', '', '')), + ('http:////example.com/tmp/junk.txt', + ('http', '', '//example.com/tmp/junk.txt', '', '', ''), + ('http', '', '//example.com/tmp/junk.txt', '', '')), ('imap://mail.python.org/mbox1', ('imap', 'mail.python.org', '/mbox1', '', '', ''), ('imap', 'mail.python.org', '/mbox1', '', '')), @@ -162,24 +247,68 @@ def test_roundtrips(self): ('svn+ssh', 'svn.zope.org', '/repos/main/ZConfig/trunk/', '', '')), ('git+ssh://git@github.com/user/project.git', - ('git+ssh', 'git@github.com','/user/project.git', - '','',''), - ('git+ssh', 'git@github.com','/user/project.git', - '', '')), - ] - def _encode(t): - return (t[0].encode('ascii'), - tuple(x.encode('ascii') for x in t[1]), - tuple(x.encode('ascii') for x in t[2])) - bytes_cases = [_encode(x) for x in str_cases] - for url, parsed, split in str_cases + bytes_cases: - self.checkRoundtrips(url, parsed, split) - - def test_http_roundtrips(self): - # urllib.parse.urlsplit treats 'http:' as an optimized special case, - # so we test both 'http:' and 'https:' in all the following. - # Three cheers for white box knowledge! - str_cases = [ + ('git+ssh', 'git@github.com','/user/project.git', + '','',''), + ('git+ssh', 'git@github.com','/user/project.git', + '', '')), + ('itms-services://?action=download-manifest&url=https://example.com/app', + ('itms-services', '', '', '', + 'action=download-manifest&url=https://example.com/app', ''), + ('itms-services', '', '', + 'action=download-manifest&url=https://example.com/app', '')), + ('+scheme:path/to/file', + ('', '', '+scheme:path/to/file', '', '', ''), + ('', '', '+scheme:path/to/file', '', '')), + ('sch_me:path/to/file', + ('', '', 'sch_me:path/to/file', '', '', ''), + ('', '', 'sch_me:path/to/file', '', '')), + ('schème:path/to/file', + ('', '', 'schème:path/to/file', '', '', ''), + ('', '', 'schème:path/to/file', '', '')), + ]) + def test_roundtrips(self, bytes, url, parsed, split): + if bytes: + if not url.isascii(): + self.skipTest('non-ASCII bytes') + url = str_encode(url) + parsed = tuple_encode(parsed) + split = tuple_encode(split) + self.checkRoundtrips(url, parsed, split) + + @support.subTests('bytes', (False, True)) + @support.subTests('url,url2,parsed,split', [ + ('///path/to/file', + '/path/to/file', + ('', '', '/path/to/file', '', '', ''), + ('', '', '/path/to/file', '', '')), + ('scheme:///path/to/file', + 'scheme:/path/to/file', + ('scheme', '', '/path/to/file', '', '', ''), + ('scheme', '', '/path/to/file', '', '')), + ('file:/tmp/junk.txt', + 'file:///tmp/junk.txt', + ('file', '', '/tmp/junk.txt', '', '', ''), + ('file', '', '/tmp/junk.txt', '', '')), + ('http:/tmp/junk.txt', + 'http:///tmp/junk.txt', + ('http', '', '/tmp/junk.txt', '', '', ''), + ('http', '', '/tmp/junk.txt', '', '')), + ('https:/tmp/junk.txt', + 'https:///tmp/junk.txt', + ('https', '', '/tmp/junk.txt', '', '', ''), + ('https', '', '/tmp/junk.txt', '', '')), + ]) + def test_roundtrips_normalization(self, bytes, url, url2, parsed, split): + if bytes: + url = str_encode(url) + url2 = str_encode(url2) + parsed = tuple_encode(parsed) + split = tuple_encode(split) + self.checkRoundtrips(url, parsed, split, url2) + + @support.subTests('bytes', (False, True)) + @support.subTests('scheme', ('http', 'https')) + @support.subTests('url,parsed,split', [ ('://www.python.org', ('www.python.org', '', '', '', ''), ('www.python.org', '', '', '')), @@ -195,37 +324,42 @@ def test_http_roundtrips(self): ('://a/b/c/d;p?q#f', ('a', '/b/c/d', 'p', 'q', 'f'), ('a', '/b/c/d;p', 'q', 'f')), - ] - def _encode(t): - return (t[0].encode('ascii'), - tuple(x.encode('ascii') for x in t[1]), - tuple(x.encode('ascii') for x in t[2])) - bytes_cases = [_encode(x) for x in str_cases] - str_schemes = ('http', 'https') - bytes_schemes = (b'http', b'https') - str_tests = str_schemes, str_cases - bytes_tests = bytes_schemes, bytes_cases - for schemes, test_cases in (str_tests, bytes_tests): - for scheme in schemes: - for url, parsed, split in test_cases: - url = scheme + url - parsed = (scheme,) + parsed - split = (scheme,) + split - self.checkRoundtrips(url, parsed, split) - - def checkJoin(self, base, relurl, expected): - str_components = (base, relurl, expected) - self.assertEqual(urllib.parse.urljoin(base, relurl), expected) - bytes_components = baseb, relurlb, expectedb = [ - x.encode('ascii') for x in str_components] - self.assertEqual(urllib.parse.urljoin(baseb, relurlb), expectedb) - - def test_unparse_parse(self): - str_cases = ['Python', './Python','x-newscheme://foo.com/stuff','x://y','x:/y','x:/','/',] - bytes_cases = [x.encode('ascii') for x in str_cases] - for u in str_cases + bytes_cases: - self.assertEqual(urllib.parse.urlunsplit(urllib.parse.urlsplit(u)), u) - self.assertEqual(urllib.parse.urlunparse(urllib.parse.urlparse(u)), u) + ]) + def test_http_roundtrips(self, bytes, scheme, url, parsed, split): + # urllib.parse.urlsplit treats 'http:' as an optimized special case, + # so we test both 'http:' and 'https:' in all the following. + # Three cheers for white box knowledge! + if bytes: + scheme = str_encode(scheme) + url = str_encode(url) + parsed = tuple_encode(parsed) + split = tuple_encode(split) + url = scheme + url + parsed = (scheme,) + parsed + split = (scheme,) + split + self.checkRoundtrips(url, parsed, split) + + def checkJoin(self, base, relurl, expected, *, relroundtrip=True): + with self.subTest(base=base, relurl=relurl): + self.assertEqual(urllib.parse.urljoin(base, relurl), expected) + baseb = base.encode('ascii') + relurlb = relurl.encode('ascii') + expectedb = expected.encode('ascii') + self.assertEqual(urllib.parse.urljoin(baseb, relurlb), expectedb) + + if relroundtrip: + relurl = urllib.parse.urlunsplit(urllib.parse.urlsplit(relurl)) + self.assertEqual(urllib.parse.urljoin(base, relurl), expected) + relurlb = urllib.parse.urlunsplit(urllib.parse.urlsplit(relurlb)) + self.assertEqual(urllib.parse.urljoin(baseb, relurlb), expectedb) + + @support.subTests('bytes', (False, True)) + @support.subTests('u', ['Python', './Python','x-newscheme://foo.com/stuff','x://y','x:/y','x:/','/',]) + def test_unparse_parse(self, bytes, u): + if bytes: + u = str_encode(u) + self.assertEqual(urllib.parse.urlunsplit(urllib.parse.urlsplit(u)), u) + self.assertEqual(urllib.parse.urlunparse(urllib.parse.urlparse(u)), u) def test_RFC1808(self): # "normal" cases from RFC 1808: @@ -384,8 +518,6 @@ def test_RFC3986(self): def test_urljoins(self): self.checkJoin(SIMPLE_BASE, 'g:h','g:h') - self.checkJoin(SIMPLE_BASE, 'http:g','http://a/b/c/g') - self.checkJoin(SIMPLE_BASE, 'http:','http://a/b/c/d') self.checkJoin(SIMPLE_BASE, 'g','http://a/b/c/g') self.checkJoin(SIMPLE_BASE, './g','http://a/b/c/g') self.checkJoin(SIMPLE_BASE, 'g/','http://a/b/c/g/') @@ -406,8 +538,6 @@ def test_urljoins(self): self.checkJoin(SIMPLE_BASE, 'g/./h','http://a/b/c/g/h') self.checkJoin(SIMPLE_BASE, 'g/../h','http://a/b/c/h') self.checkJoin(SIMPLE_BASE, 'http:g','http://a/b/c/g') - self.checkJoin(SIMPLE_BASE, 'http:','http://a/b/c/d') - self.checkJoin(SIMPLE_BASE, 'http:?y','http://a/b/c/d?y') self.checkJoin(SIMPLE_BASE, 'http:g?y','http://a/b/c/g?y') self.checkJoin(SIMPLE_BASE, 'http:g?y/./x','http://a/b/c/g?y/./x') self.checkJoin('http:///', '..','http:///') @@ -437,8 +567,127 @@ def test_urljoins(self): # issue 23703: don't duplicate filename self.checkJoin('a', 'b', 'b') - def test_RFC2732(self): - str_cases = [ + # Test with empty (but defined) components. + self.checkJoin(RFC1808_BASE, '', 'http://a/b/c/d;p?q#f') + self.checkJoin(RFC1808_BASE, '#', 'http://a/b/c/d;p?q#', relroundtrip=False) + self.checkJoin(RFC1808_BASE, '#z', 'http://a/b/c/d;p?q#z') + self.checkJoin(RFC1808_BASE, '?', 'http://a/b/c/d;p?', relroundtrip=False) + self.checkJoin(RFC1808_BASE, '?#z', 'http://a/b/c/d;p?#z', relroundtrip=False) + self.checkJoin(RFC1808_BASE, '?y', 'http://a/b/c/d;p?y') + self.checkJoin(RFC1808_BASE, ';', 'http://a/b/c/;') + self.checkJoin(RFC1808_BASE, ';?y', 'http://a/b/c/;?y') + self.checkJoin(RFC1808_BASE, ';#z', 'http://a/b/c/;#z') + self.checkJoin(RFC1808_BASE, ';x', 'http://a/b/c/;x') + self.checkJoin(RFC1808_BASE, '/w', 'http://a/w') + self.checkJoin(RFC1808_BASE, '//', 'http://a/b/c/d;p?q#f') + self.checkJoin(RFC1808_BASE, '//#z', 'http://a/b/c/d;p?q#z') + self.checkJoin(RFC1808_BASE, '//?y', 'http://a/b/c/d;p?y') + self.checkJoin(RFC1808_BASE, '//;x', 'http://;x') + self.checkJoin(RFC1808_BASE, '///w', 'http://a/w') + self.checkJoin(RFC1808_BASE, '//v', 'http://v') + # For backward compatibility with RFC1630, the scheme name is allowed + # to be present in a relative reference if it is the same as the base + # URI scheme. + self.checkJoin(RFC1808_BASE, 'http:', 'http://a/b/c/d;p?q#f') + self.checkJoin(RFC1808_BASE, 'http:#', 'http://a/b/c/d;p?q#', relroundtrip=False) + self.checkJoin(RFC1808_BASE, 'http:#z', 'http://a/b/c/d;p?q#z') + self.checkJoin(RFC1808_BASE, 'http:?', 'http://a/b/c/d;p?', relroundtrip=False) + self.checkJoin(RFC1808_BASE, 'http:?#z', 'http://a/b/c/d;p?#z', relroundtrip=False) + self.checkJoin(RFC1808_BASE, 'http:?y', 'http://a/b/c/d;p?y') + self.checkJoin(RFC1808_BASE, 'http:;', 'http://a/b/c/;') + self.checkJoin(RFC1808_BASE, 'http:;?y', 'http://a/b/c/;?y') + self.checkJoin(RFC1808_BASE, 'http:;#z', 'http://a/b/c/;#z') + self.checkJoin(RFC1808_BASE, 'http:;x', 'http://a/b/c/;x') + self.checkJoin(RFC1808_BASE, 'http:/w', 'http://a/w') + self.checkJoin(RFC1808_BASE, 'http://', 'http://a/b/c/d;p?q#f') + self.checkJoin(RFC1808_BASE, 'http://#z', 'http://a/b/c/d;p?q#z') + self.checkJoin(RFC1808_BASE, 'http://?y', 'http://a/b/c/d;p?y') + self.checkJoin(RFC1808_BASE, 'http://;x', 'http://;x') + self.checkJoin(RFC1808_BASE, 'http:///w', 'http://a/w') + self.checkJoin(RFC1808_BASE, 'http://v', 'http://v') + # Different scheme is not ignored. + self.checkJoin(RFC1808_BASE, 'https:', 'https:', relroundtrip=False) + self.checkJoin(RFC1808_BASE, 'https:#', 'https:#', relroundtrip=False) + self.checkJoin(RFC1808_BASE, 'https:#z', 'https:#z', relroundtrip=False) + self.checkJoin(RFC1808_BASE, 'https:?', 'https:?', relroundtrip=False) + self.checkJoin(RFC1808_BASE, 'https:?y', 'https:?y', relroundtrip=False) + self.checkJoin(RFC1808_BASE, 'https:;', 'https:;') + self.checkJoin(RFC1808_BASE, 'https:;x', 'https:;x') + + def test_urljoins_relative_base(self): + # According to RFC 3986, Section 5.1, a base URI must conform to + # the absolute-URI syntax rule (Section 4.3). But urljoin() lacks + # a context to establish missed components of the relative base URI. + # It still has to return a sensible result for backwards compatibility. + # The following tests are figments of the imagination and artifacts + # of the current implementation that are not based on any standard. + self.checkJoin('', '', '') + self.checkJoin('', '//', '//', relroundtrip=False) + self.checkJoin('', '//v', '//v') + self.checkJoin('', '//v/w', '//v/w') + self.checkJoin('', '/w', '/w') + self.checkJoin('', '///w', '///w', relroundtrip=False) + self.checkJoin('', 'w', 'w') + + self.checkJoin('//', '', '//') + self.checkJoin('//', '//', '//') + self.checkJoin('//', '//v', '//v') + self.checkJoin('//', '//v/w', '//v/w') + self.checkJoin('//', '/w', '///w') + self.checkJoin('//', '///w', '///w') + self.checkJoin('//', 'w', '///w') + + self.checkJoin('//a', '', '//a') + self.checkJoin('//a', '//', '//a') + self.checkJoin('//a', '//v', '//v') + self.checkJoin('//a', '//v/w', '//v/w') + self.checkJoin('//a', '/w', '//a/w') + self.checkJoin('//a', '///w', '//a/w') + self.checkJoin('//a', 'w', '//a/w') + + for scheme in '', 'http:': + self.checkJoin('http:', scheme + '', 'http:') + self.checkJoin('http:', scheme + '//', 'http:') + self.checkJoin('http:', scheme + '//v', 'http://v') + self.checkJoin('http:', scheme + '//v/w', 'http://v/w') + self.checkJoin('http:', scheme + '/w', 'http:/w') + self.checkJoin('http:', scheme + '///w', 'http:/w') + self.checkJoin('http:', scheme + 'w', 'http:/w') + + self.checkJoin('http://', scheme + '', 'http://') + self.checkJoin('http://', scheme + '//', 'http://') + self.checkJoin('http://', scheme + '//v', 'http://v') + self.checkJoin('http://', scheme + '//v/w', 'http://v/w') + self.checkJoin('http://', scheme + '/w', 'http:///w') + self.checkJoin('http://', scheme + '///w', 'http:///w') + self.checkJoin('http://', scheme + 'w', 'http:///w') + + self.checkJoin('http://a', scheme + '', 'http://a') + self.checkJoin('http://a', scheme + '//', 'http://a') + self.checkJoin('http://a', scheme + '//v', 'http://v') + self.checkJoin('http://a', scheme + '//v/w', 'http://v/w') + self.checkJoin('http://a', scheme + '/w', 'http://a/w') + self.checkJoin('http://a', scheme + '///w', 'http://a/w') + self.checkJoin('http://a', scheme + 'w', 'http://a/w') + + self.checkJoin('/b/c', '', '/b/c') + self.checkJoin('/b/c', '//', '/b/c') + self.checkJoin('/b/c', '//v', '//v') + self.checkJoin('/b/c', '//v/w', '//v/w') + self.checkJoin('/b/c', '/w', '/w') + self.checkJoin('/b/c', '///w', '/w') + self.checkJoin('/b/c', 'w', '/b/w') + + self.checkJoin('///b/c', '', '///b/c') + self.checkJoin('///b/c', '//', '///b/c') + self.checkJoin('///b/c', '//v', '//v') + self.checkJoin('///b/c', '//v/w', '//v/w') + self.checkJoin('///b/c', '/w', '///w') + self.checkJoin('///b/c', '///w', '///w') + self.checkJoin('///b/c', 'w', '///b/w') + + @support.subTests('bytes', (False, True)) + @support.subTests('url,hostname,port', [ ('http://Test.python.org:5432/foo/', 'test.python.org', 5432), ('http://12.34.56.78:5432/foo/', '12.34.56.78', 5432), ('http://[::1]:5432/foo/', '::1', 5432), @@ -469,26 +718,28 @@ def test_RFC2732(self): ('http://[::12.34.56.78]:/foo/', '::12.34.56.78', None), ('http://[::ffff:12.34.56.78]:/foo/', '::ffff:12.34.56.78', None), - ] - def _encode(t): - return t[0].encode('ascii'), t[1].encode('ascii'), t[2] - bytes_cases = [_encode(x) for x in str_cases] - for url, hostname, port in str_cases + bytes_cases: - urlparsed = urllib.parse.urlparse(url) - self.assertEqual((urlparsed.hostname, urlparsed.port) , (hostname, port)) - - str_cases = [ + ]) + def test_RFC2732(self, bytes, url, hostname, port): + if bytes: + url = str_encode(url) + hostname = str_encode(hostname) + urlparsed = urllib.parse.urlparse(url) + self.assertEqual((urlparsed.hostname, urlparsed.port), (hostname, port)) + + @support.subTests('bytes', (False, True)) + @support.subTests('invalid_url', [ 'http://::12.34.56.78]/', 'http://[::1/foo/', 'ftp://[::1/foo/bad]/bad', 'http://[::1/foo/bad]/bad', - 'http://[::ffff:12.34.56.78'] - bytes_cases = [x.encode('ascii') for x in str_cases] - for invalid_url in str_cases + bytes_cases: - self.assertRaises(ValueError, urllib.parse.urlparse, invalid_url) - - def test_urldefrag(self): - str_cases = [ + 'http://[::ffff:12.34.56.78']) + def test_RFC2732_invalid(self, bytes, invalid_url): + if bytes: + invalid_url = str_encode(invalid_url) + self.assertRaises(ValueError, urllib.parse.urlparse, invalid_url) + + @support.subTests('bytes', (False, True)) + @support.subTests('url,defrag,frag', [ ('http://python.org#frag', 'http://python.org', 'frag'), ('http://python.org', 'http://python.org', ''), ('http://python.org/#frag', 'http://python.org/', 'frag'), @@ -499,16 +750,31 @@ def test_urldefrag(self): ('http://python.org/p?q', 'http://python.org/p?q', ''), (RFC1808_BASE, 'http://a/b/c/d;p?q', 'f'), (RFC2396_BASE, 'http://a/b/c/d;p?q', ''), - ] - def _encode(t): - return type(t)(x.encode('ascii') for x in t) - bytes_cases = [_encode(x) for x in str_cases] - for url, defrag, frag in str_cases + bytes_cases: - result = urllib.parse.urldefrag(url) - self.assertEqual(result.geturl(), url) - self.assertEqual(result, (defrag, frag)) - self.assertEqual(result.url, defrag) - self.assertEqual(result.fragment, frag) + ('http://a/b/c;p?q#f', 'http://a/b/c;p?q', 'f'), + ('http://a/b/c;p?q#', 'http://a/b/c;p?q', ''), + ('http://a/b/c;p?q', 'http://a/b/c;p?q', ''), + ('http://a/b/c;p?#f', 'http://a/b/c;p?', 'f'), + ('http://a/b/c;p#f', 'http://a/b/c;p', 'f'), + ('http://a/b/c;?q#f', 'http://a/b/c;?q', 'f'), + ('http://a/b/c?q#f', 'http://a/b/c?q', 'f'), + ('http:///b/c;p?q#f', 'http:///b/c;p?q', 'f'), + ('http:b/c;p?q#f', 'http:b/c;p?q', 'f'), + ('http:;?q#f', 'http:;?q', 'f'), + ('http:?q#f', 'http:?q', 'f'), + ('//a/b/c;p?q#f', '//a/b/c;p?q', 'f'), + ('://a/b/c;p?q#f', '://a/b/c;p?q', 'f'), + ]) + def test_urldefrag(self, bytes, url, defrag, frag): + if bytes: + url = str_encode(url) + defrag = str_encode(defrag) + frag = str_encode(frag) + result = urllib.parse.urldefrag(url) + hash = '#' if isinstance(url, str) else b'#' + self.assertEqual(result.geturl(), url.rstrip(hash)) + self.assertEqual(result, (defrag, frag)) + self.assertEqual(result.url, defrag) + self.assertEqual(result.fragment, frag) def test_urlsplit_scoped_IPv6(self): p = urllib.parse.urlsplit('http://[FE80::822a:a8ff:fe49:470c%tESt]:1234') @@ -649,21 +915,94 @@ def test_urlsplit_remove_unsafe_bytes(self): self.assertEqual(p.scheme, "http") self.assertEqual(p.geturl(), "http://www.python.org/javascript:alert('msg')/?query=something#fragment") - def test_attributes_bad_port(self): + def test_urlsplit_strip_url(self): + noise = bytes(range(0, 0x20 + 1)) + base_url = "http://User:Pass@www.python.org:080/doc/?query=yes#frag" + + url = noise.decode("utf-8") + base_url + p = urllib.parse.urlsplit(url) + self.assertEqual(p.scheme, "http") + self.assertEqual(p.netloc, "User:Pass@www.python.org:080") + self.assertEqual(p.path, "/doc/") + self.assertEqual(p.query, "query=yes") + self.assertEqual(p.fragment, "frag") + self.assertEqual(p.username, "User") + self.assertEqual(p.password, "Pass") + self.assertEqual(p.hostname, "www.python.org") + self.assertEqual(p.port, 80) + self.assertEqual(p.geturl(), base_url) + + url = noise + base_url.encode("utf-8") + p = urllib.parse.urlsplit(url) + self.assertEqual(p.scheme, b"http") + self.assertEqual(p.netloc, b"User:Pass@www.python.org:080") + self.assertEqual(p.path, b"/doc/") + self.assertEqual(p.query, b"query=yes") + self.assertEqual(p.fragment, b"frag") + self.assertEqual(p.username, b"User") + self.assertEqual(p.password, b"Pass") + self.assertEqual(p.hostname, b"www.python.org") + self.assertEqual(p.port, 80) + self.assertEqual(p.geturl(), base_url.encode("utf-8")) + + # Test that trailing space is preserved as some applications rely on + # this within query strings. + query_spaces_url = "https://www.python.org:88/doc/?query= " + p = urllib.parse.urlsplit(noise.decode("utf-8") + query_spaces_url) + self.assertEqual(p.scheme, "https") + self.assertEqual(p.netloc, "www.python.org:88") + self.assertEqual(p.path, "/doc/") + self.assertEqual(p.query, "query= ") + self.assertEqual(p.port, 88) + self.assertEqual(p.geturl(), query_spaces_url) + + p = urllib.parse.urlsplit("www.pypi.org ") + # That "hostname" gets considered a "path" due to the + # trailing space and our existing logic... YUCK... + # and re-assembles via geturl aka unurlsplit into the original. + # django.core.validators.URLValidator (at least through v3.2) relies on + # this, for better or worse, to catch it in a ValidationError via its + # regular expressions. + # Here we test the basic round trip concept of such a trailing space. + self.assertEqual(urllib.parse.urlunsplit(p), "www.pypi.org ") + + # with scheme as cache-key + url = "//www.python.org/" + scheme = noise.decode("utf-8") + "https" + noise.decode("utf-8") + for _ in range(2): + p = urllib.parse.urlsplit(url, scheme=scheme) + self.assertEqual(p.scheme, "https") + self.assertEqual(p.geturl(), "https://www.python.org/") + + @support.subTests('bytes', (False, True)) + @support.subTests('parse', (urllib.parse.urlsplit, urllib.parse.urlparse)) + @support.subTests('port', ("foo", "1.5", "-1", "0x10", "-0", "1_1", " 1", "1 ", "६")) + def test_attributes_bad_port(self, bytes, parse, port): """Check handling of invalid ports.""" - for bytes in (False, True): - for parse in (urllib.parse.urlsplit, urllib.parse.urlparse): - for port in ("foo", "1.5", "-1", "0x10"): - with self.subTest(bytes=bytes, parse=parse, port=port): - netloc = "www.example.net:" + port - url = "http://" + netloc - if bytes: - netloc = netloc.encode("ascii") - url = url.encode("ascii") - p = parse(url) - self.assertEqual(p.netloc, netloc) - with self.assertRaises(ValueError): - p.port + netloc = "www.example.net:" + port + url = "http://" + netloc + "/" + if bytes: + if not (netloc.isascii() and port.isascii()): + self.skipTest('non-ASCII bytes') + netloc = str_encode(netloc) + url = str_encode(url) + p = parse(url) + self.assertEqual(p.netloc, netloc) + with self.assertRaises(ValueError): + p.port + + @support.subTests('bytes', (False, True)) + @support.subTests('parse', (urllib.parse.urlsplit, urllib.parse.urlparse)) + @support.subTests('scheme', (".", "+", "-", "0", "http&", "६http")) + def test_attributes_bad_scheme(self, bytes, parse, scheme): + """Check handling of invalid schemes.""" + url = scheme + "://www.example.net" + if bytes: + if not url.isascii(): + self.skipTest('non-ASCII bytes') + url = url.encode("ascii") + p = parse(url) + self.assertEqual(p.scheme, b"" if bytes else "") def test_attributes_without_netloc(self): # This example is straight from RFC 3261. It looks like it @@ -775,24 +1114,21 @@ def test_anyscheme(self): self.assertEqual(urllib.parse.urlparse(b"x-newscheme://foo.com/stuff?query"), (b'x-newscheme', b'foo.com', b'/stuff', b'', b'query', b'')) - def test_default_scheme(self): + @support.subTests('func', (urllib.parse.urlparse, urllib.parse.urlsplit)) + def test_default_scheme(self, func): # Exercise the scheme parameter of urlparse() and urlsplit() - for func in (urllib.parse.urlparse, urllib.parse.urlsplit): - with self.subTest(function=func): - result = func("http://example.net/", "ftp") - self.assertEqual(result.scheme, "http") - result = func(b"http://example.net/", b"ftp") - self.assertEqual(result.scheme, b"http") - self.assertEqual(func("path", "ftp").scheme, "ftp") - self.assertEqual(func("path", scheme="ftp").scheme, "ftp") - self.assertEqual(func(b"path", scheme=b"ftp").scheme, b"ftp") - self.assertEqual(func("path").scheme, "") - self.assertEqual(func(b"path").scheme, b"") - self.assertEqual(func(b"path", "").scheme, b"") - - def test_parse_fragments(self): - # Exercise the allow_fragments parameter of urlparse() and urlsplit() - tests = ( + result = func("http://example.net/", "ftp") + self.assertEqual(result.scheme, "http") + result = func(b"http://example.net/", b"ftp") + self.assertEqual(result.scheme, b"http") + self.assertEqual(func("path", "ftp").scheme, "ftp") + self.assertEqual(func("path", scheme="ftp").scheme, "ftp") + self.assertEqual(func(b"path", scheme=b"ftp").scheme, b"ftp") + self.assertEqual(func("path").scheme, "") + self.assertEqual(func(b"path").scheme, b"") + self.assertEqual(func(b"path", "").scheme, b"") + + @support.subTests('url,attr,expected_frag', ( ("http:#frag", "path", "frag"), ("//example.net#frag", "path", "frag"), ("index.html#frag", "path", "frag"), @@ -803,25 +1139,24 @@ def test_parse_fragments(self): ("//abc#@frag", "path", "@frag"), ("//abc:80#@frag", "path", "@frag"), ("//abc#@frag:80", "path", "@frag:80"), - ) - for url, attr, expected_frag in tests: - for func in (urllib.parse.urlparse, urllib.parse.urlsplit): - if attr == "params" and func is urllib.parse.urlsplit: - attr = "path" - with self.subTest(url=url, function=func): - result = func(url, allow_fragments=False) - self.assertEqual(result.fragment, "") - self.assertTrue( - getattr(result, attr).endswith("#" + expected_frag)) - self.assertEqual(func(url, "", False).fragment, "") - - result = func(url, allow_fragments=True) - self.assertEqual(result.fragment, expected_frag) - self.assertFalse( - getattr(result, attr).endswith(expected_frag)) - self.assertEqual(func(url, "", True).fragment, - expected_frag) - self.assertEqual(func(url).fragment, expected_frag) + )) + @support.subTests('func', (urllib.parse.urlparse, urllib.parse.urlsplit)) + def test_parse_fragments(self, url, attr, expected_frag, func): + # Exercise the allow_fragments parameter of urlparse() and urlsplit() + if attr == "params" and func is urllib.parse.urlsplit: + attr = "path" + result = func(url, allow_fragments=False) + self.assertEqual(result.fragment, "") + self.assertEndsWith(getattr(result, attr), + "#" + expected_frag) + self.assertEqual(func(url, "", False).fragment, "") + + result = func(url, allow_fragments=True) + self.assertEqual(result.fragment, expected_frag) + self.assertNotEndsWith(getattr(result, attr), expected_frag) + self.assertEqual(func(url, "", True).fragment, + expected_frag) + self.assertEqual(func(url).fragment, expected_frag) def test_mixed_types_rejected(self): # Several functions that process either strings or ASCII encoded bytes @@ -847,7 +1182,14 @@ def test_mixed_types_rejected(self): with self.assertRaisesRegex(TypeError, "Cannot mix str"): urllib.parse.urljoin(b"http://python.org", "http://python.org") - def _check_result_type(self, str_type): + @support.subTests('result_type', [ + urllib.parse.DefragResult, + urllib.parse.SplitResult, + urllib.parse.ParseResult, + ]) + def test_result_pairs(self, result_type): + # Check encoding and decoding between result pairs + str_type = result_type num_args = len(str_type._fields) bytes_type = str_type._encoded_counterpart self.assertIs(bytes_type._decoded_counterpart, str_type) @@ -872,16 +1214,6 @@ def _check_result_type(self, str_type): self.assertEqual(str_result.encode(encoding, errors), bytes_args) self.assertEqual(str_result.encode(encoding, errors), bytes_result) - def test_result_pairs(self): - # Check encoding and decoding between result pairs - result_types = [ - urllib.parse.DefragResult, - urllib.parse.SplitResult, - urllib.parse.ParseResult, - ] - for result_type in result_types: - self._check_result_type(result_type) - def test_parse_qs_encoding(self): result = urllib.parse.parse_qs("key=\u0141%E9", encoding="latin-1") self.assertEqual(result, {'key': ['\u0141\xE9']}) @@ -910,11 +1242,10 @@ def test_parse_qsl_encoding(self): def test_parse_qsl_max_num_fields(self): with self.assertRaises(ValueError): - urllib.parse.parse_qs('&'.join(['a=a']*11), max_num_fields=10) - urllib.parse.parse_qs('&'.join(['a=a']*10), max_num_fields=10) + urllib.parse.parse_qsl('&'.join(['a=a']*11), max_num_fields=10) + urllib.parse.parse_qsl('&'.join(['a=a']*10), max_num_fields=10) - def test_parse_qs_separator(self): - parse_qs_semicolon_cases = [ + @support.subTests('orig,expect', [ (";", {}), (";;", {}), (";a=b", {'a': ['b']}), @@ -925,17 +1256,14 @@ def test_parse_qs_separator(self): (b";a=b", {b'a': [b'b']}), (b"a=a+b;b=b+c", {b'a': [b'a b'], b'b': [b'b c']}), (b"a=1;a=2", {b'a': [b'1', b'2']}), - ] - for orig, expect in parse_qs_semicolon_cases: - with self.subTest(f"Original: {orig!r}, Expected: {expect!r}"): - result = urllib.parse.parse_qs(orig, separator=';') - self.assertEqual(result, expect, "Error parsing %r" % orig) - result_bytes = urllib.parse.parse_qs(orig, separator=b';') - self.assertEqual(result_bytes, expect, "Error parsing %r" % orig) - - - def test_parse_qsl_separator(self): - parse_qsl_semicolon_cases = [ + ]) + def test_parse_qs_separator(self, orig, expect): + result = urllib.parse.parse_qs(orig, separator=';') + self.assertEqual(result, expect) + result_bytes = urllib.parse.parse_qs(orig, separator=b';') + self.assertEqual(result_bytes, expect) + + @support.subTests('orig,expect', [ (";", []), (";;", []), (";a=b", [('a', 'b')]), @@ -946,14 +1274,45 @@ def test_parse_qsl_separator(self): (b";a=b", [(b'a', b'b')]), (b"a=a+b;b=b+c", [(b'a', b'a b'), (b'b', b'b c')]), (b"a=1;a=2", [(b'a', b'1'), (b'a', b'2')]), - ] - for orig, expect in parse_qsl_semicolon_cases: - with self.subTest(f"Original: {orig!r}, Expected: {expect!r}"): - result = urllib.parse.parse_qsl(orig, separator=';') - self.assertEqual(result, expect, "Error parsing %r" % orig) - result_bytes = urllib.parse.parse_qsl(orig, separator=b';') - self.assertEqual(result_bytes, expect, "Error parsing %r" % orig) - + ]) + def test_parse_qsl_separator(self, orig, expect): + result = urllib.parse.parse_qsl(orig, separator=';') + self.assertEqual(result, expect) + result_bytes = urllib.parse.parse_qsl(orig, separator=b';') + self.assertEqual(result_bytes, expect) + + def test_parse_qsl_bytes(self): + self.assertEqual(urllib.parse.parse_qsl(b'a=b'), [(b'a', b'b')]) + self.assertEqual(urllib.parse.parse_qsl(bytearray(b'a=b')), [(b'a', b'b')]) + self.assertEqual(urllib.parse.parse_qsl(memoryview(b'a=b')), [(b'a', b'b')]) + + def test_parse_qsl_false_value(self): + kwargs = dict(keep_blank_values=True, strict_parsing=True) + for x in '', b'', None, memoryview(b''): + self.assertEqual(urllib.parse.parse_qsl(x, **kwargs), []) + self.assertRaises(ValueError, urllib.parse.parse_qsl, x, separator=1) + for x in 0, 0.0, [], {}: + with self.assertWarns(DeprecationWarning) as cm: + self.assertEqual(urllib.parse.parse_qsl(x, **kwargs), []) + self.assertEqual(cm.filename, __file__) + with self.assertWarns(DeprecationWarning) as cm: + self.assertEqual(urllib.parse.parse_qs(x, **kwargs), {}) + self.assertEqual(cm.filename, __file__) + self.assertRaises(ValueError, urllib.parse.parse_qsl, x, separator=1) + + def test_parse_qsl_errors(self): + self.assertRaises(TypeError, urllib.parse.parse_qsl, list(b'a=b')) + self.assertRaises(TypeError, urllib.parse.parse_qsl, iter(b'a=b')) + self.assertRaises(TypeError, urllib.parse.parse_qsl, 1) + self.assertRaises(TypeError, urllib.parse.parse_qsl, object()) + + for separator in '', b'', None, 0, 1, 0.0, 1.5: + with self.assertRaises(ValueError): + urllib.parse.parse_qsl('a=b', separator=separator) + with self.assertRaises(UnicodeEncodeError): + urllib.parse.parse_qsl(b'a=b', separator='\xa6') + with self.assertRaises(UnicodeDecodeError): + urllib.parse.parse_qsl('a=b', separator=b'\xa6') def test_urlencode_sequences(self): # Other tests incidentally urlencode things; test non-covered cases: @@ -985,6 +1344,10 @@ def test_quote_from_bytes(self): self.assertEqual(result, 'archaeological%20arcana') result = urllib.parse.quote_from_bytes(b'') self.assertEqual(result, '') + result = urllib.parse.quote_from_bytes(b'A'*10_000) + self.assertEqual(result, 'A'*10_000) + result = urllib.parse.quote_from_bytes(b'z\x01/ '*253_183) + self.assertEqual(result, 'z%01/%20'*253_183) def test_unquote_to_bytes(self): result = urllib.parse.unquote_to_bytes('abc%20def') @@ -1012,6 +1375,67 @@ def test_issue14072(self): self.assertEqual(p2.scheme, 'tel') self.assertEqual(p2.path, '+31641044153') + def test_invalid_bracketed_hosts(self): + self.assertRaises(ValueError, urllib.parse.urlsplit, 'Scheme://user@[192.0.2.146]/Path?Query') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'Scheme://user@[important.com:8000]/Path?Query') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'Scheme://user@[v123r.IP]/Path?Query') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'Scheme://user@[v12ae]/Path?Query') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'Scheme://user@[v.IP]/Path?Query') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'Scheme://user@[v123.]/Path?Query') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'Scheme://user@[v]/Path?Query') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'Scheme://user@[0439:23af::2309::fae7:1234]/Path?Query') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'Scheme://user@[0439:23af:2309::fae7:1234:2342:438e:192.0.2.146]/Path?Query') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'Scheme://user@]v6a.ip[/Path') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://prefix.[v6a.ip]') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://[v6a.ip].suffix') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://prefix.[v6a.ip]/') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://[v6a.ip].suffix/') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://prefix.[v6a.ip]?') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://[v6a.ip].suffix?') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://prefix.[::1]') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://[::1].suffix') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://prefix.[::1]/') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://[::1].suffix/') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://prefix.[::1]?') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://[::1].suffix?') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://prefix.[::1]:a') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://[::1].suffix:a') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://prefix.[::1]:a1') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://[::1].suffix:a1') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://prefix.[::1]:1a') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://[::1].suffix:1a') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://prefix.[::1]:') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://[::1].suffix:/') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://prefix.[::1]:?') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://user@prefix.[v6a.ip]') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://user@[v6a.ip].suffix') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://[v6a.ip') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://v6a.ip]') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://]v6a.ip[') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://]v6a.ip') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://v6a.ip[') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://prefix.[v6a.ip') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://v6a.ip].suffix') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://prefix]v6a.ip[suffix') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://prefix]v6a.ip') + self.assertRaises(ValueError, urllib.parse.urlsplit, 'scheme://v6a.ip[suffix') + + def test_splitting_bracketed_hosts(self): + p1 = urllib.parse.urlsplit('scheme://user@[v6a.ip]:1234/path?query') + self.assertEqual(p1.hostname, 'v6a.ip') + self.assertEqual(p1.username, 'user') + self.assertEqual(p1.path, '/path') + self.assertEqual(p1.port, 1234) + p2 = urllib.parse.urlsplit('scheme://user@[0439:23af:2309::fae7%test]/path?query') + self.assertEqual(p2.hostname, '0439:23af:2309::fae7%test') + self.assertEqual(p2.username, 'user') + self.assertEqual(p2.path, '/path') + self.assertIs(p2.port, None) + p3 = urllib.parse.urlsplit('scheme://user@[0439:23af:2309::fae7:1234:192.0.2.146%test]/path?query') + self.assertEqual(p3.hostname, '0439:23af:2309::fae7:1234:192.0.2.146%test') + self.assertEqual(p3.username, 'user') + self.assertEqual(p3.path, '/path') + def test_port_casting_failure_message(self): message = "Port could not be cast to integer value as 'oracle'" p1 = urllib.parse.urlparse('http://Server=sde; Service=sde:oracle') @@ -1044,16 +1468,24 @@ def test_telurl_params(self): self.assertEqual(p1.params, 'phone-context=+1-914-555') def test_Quoter_repr(self): - quoter = urllib.parse.Quoter(urllib.parse._ALWAYS_SAFE) + quoter = urllib.parse._Quoter(urllib.parse._ALWAYS_SAFE) self.assertIn('Quoter', repr(quoter)) + def test_clear_cache_for_code_coverage(self): + urllib.parse.clear_cache() + + def test_urllib_parse_getattr_failure(self): + """Test that urllib.parse.__getattr__() fails correctly.""" + with self.assertRaises(AttributeError): + unused = urllib.parse.this_does_not_exist + def test_all(self): expected = [] undocumented = { 'splitattr', 'splithost', 'splitnport', 'splitpasswd', 'splitport', 'splitquery', 'splittag', 'splittype', 'splituser', 'splitvalue', - 'Quoter', 'ResultBase', 'clear_cache', 'to_bytes', 'unwrap', + 'ResultBase', 'clear_cache', 'to_bytes', 'unwrap', } for name in dir(urllib.parse): if name.startswith('_') or name in undocumented: @@ -1063,8 +1495,7 @@ def test_all(self): expected.append(name) self.assertCountEqual(urllib.parse.__all__, expected) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_urlsplit_normalization(self): # Certain characters should never occur in the netloc, # including under normalization. @@ -1073,7 +1504,8 @@ def test_urlsplit_normalization(self): hex_chars = {'{:04X}'.format(ord(c)) for c in illegal_chars} denorm_chars = [ c for c in map(chr, range(128, sys.maxunicode)) - if (hex_chars & set(unicodedata.decomposition(c).split())) + if unicodedata.decomposition(c) + and (hex_chars & set(unicodedata.decomposition(c).split())) and c not in illegal_chars ] # Sanity check that we found at least one such character @@ -1188,6 +1620,7 @@ def test_splitnport(self): self.assertEqual(splitnport('127.0.0.1', 55), ('127.0.0.1', 55)) self.assertEqual(splitnport('parrot:cheese'), ('parrot', None)) self.assertEqual(splitnport('parrot:cheese', 55), ('parrot', None)) + self.assertEqual(splitnport('parrot: +1_0 '), ('parrot', None)) def test_splitquery(self): # Normal cases are exercised by other tests; ensure that we also @@ -1238,15 +1671,15 @@ def test_to_bytes(self): self.assertRaises(UnicodeError, urllib.parse._to_bytes, 'http://www.python.org/medi\u00e6val') - def test_unwrap(self): - for wrapped_url in ('<URL:scheme://host/path>', '<scheme://host/path>', - 'URL:scheme://host/path', 'scheme://host/path'): - url = urllib.parse.unwrap(wrapped_url) - self.assertEqual(url, 'scheme://host/path') + @support.subTests('wrapped_url', + ('<URL:scheme://host/path>', '<scheme://host/path>', + 'URL:scheme://host/path', 'scheme://host/path')) + def test_unwrap(self, wrapped_url): + url = urllib.parse.unwrap(wrapped_url) + self.assertEqual(url, 'scheme://host/path') class DeprecationTest(unittest.TestCase): - def test_splittype_deprecation(self): with self.assertWarns(DeprecationWarning) as cm: urllib.parse.splittype('') @@ -1324,5 +1757,11 @@ def test_to_bytes_deprecation(self): 'urllib.parse.to_bytes() is deprecated as of 3.8') +def str_encode(s): + return s.encode('ascii') + +def tuple_encode(t): + return tuple(str_encode(x) for x in t) + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_uuid.py b/Lib/test/test_uuid.py index ce396aa942b..0e1a723ce3a 100644 --- a/Lib/test/test_uuid.py +++ b/Lib/test/test_uuid.py @@ -1,7 +1,3 @@ -import unittest -from test import support -from test.support import import_helper -from test.support.script_helper import assert_python_ok import builtins import contextlib import copy @@ -9,10 +5,17 @@ import io import os import pickle +import random import sys +import unittest import weakref +from itertools import product from unittest import mock +from test import support +from test.support import import_helper +from test.support.script_helper import assert_python_ok + py_uuid = import_helper.import_fresh_module('uuid', blocked=['_uuid']) c_uuid = import_helper.import_fresh_module('uuid', fresh=['_uuid']) @@ -33,6 +36,47 @@ def get_command_stdout(command, args): class BaseTestUUID: uuid = None + def test_nil_uuid(self): + nil_uuid = self.uuid.NIL + + s = '00000000-0000-0000-0000-000000000000' + i = 0 + self.assertEqual(nil_uuid, self.uuid.UUID(s)) + self.assertEqual(nil_uuid, self.uuid.UUID(int=i)) + self.assertEqual(nil_uuid.int, i) + self.assertEqual(str(nil_uuid), s) + # The Nil UUID falls within the range of the Apollo NCS variant as per + # RFC 9562. + # See https://www.rfc-editor.org/rfc/rfc9562.html#section-5.9-4 + self.assertEqual(nil_uuid.variant, self.uuid.RESERVED_NCS) + # A version field of all zeros is "Unused" in RFC 9562, but the version + # field also only applies to the 10xx variant, i.e. the variant + # specified in RFC 9562. As such, because the Nil UUID falls under a + # different variant, its version is considered undefined. + # See https://www.rfc-editor.org/rfc/rfc9562.html#table2 + self.assertIsNone(nil_uuid.version) + + def test_max_uuid(self): + max_uuid = self.uuid.MAX + + s = 'ffffffff-ffff-ffff-ffff-ffffffffffff' + i = (1 << 128) - 1 + self.assertEqual(max_uuid, self.uuid.UUID(s)) + self.assertEqual(max_uuid, self.uuid.UUID(int=i)) + self.assertEqual(max_uuid.int, i) + self.assertEqual(str(max_uuid), s) + # The Max UUID falls within the range of the "yet-to-be defined" future + # UUID variant as per RFC 9562. + # See https://www.rfc-editor.org/rfc/rfc9562.html#section-5.10-4 + self.assertEqual(max_uuid.variant, self.uuid.RESERVED_FUTURE) + # A version field of all ones is "Reserved for future definition" in + # RFC 9562, but the version field also only applies to the 10xx + # variant, i.e. the variant specified in RFC 9562. As such, because the + # Max UUID falls under a different variant, its version is considered + # undefined. + # See https://www.rfc-editor.org/rfc/rfc9562.html#table2 + self.assertIsNone(max_uuid.version) + def test_safe_uuid_enum(self): class CheckedSafeUUID(enum.Enum): safe = 0 @@ -268,7 +312,7 @@ def test_exceptions(self): # Version number out of range. badvalue(lambda: self.uuid.UUID('00'*16, version=0)) - badvalue(lambda: self.uuid.UUID('00'*16, version=6)) + badvalue(lambda: self.uuid.UUID('00'*16, version=42)) # Integer value out of range. badvalue(lambda: self.uuid.UUID(int=-1)) @@ -682,6 +726,392 @@ def test_uuid5(self): equal(u, self.uuid.UUID(v)) equal(str(u), v) + def test_uuid6(self): + equal = self.assertEqual + u = self.uuid.uuid6() + equal(u.variant, self.uuid.RFC_4122) + equal(u.version, 6) + + fake_nanoseconds = 0x1571_20a1_de1a_c533 + fake_node_value = 0x54e1_acf6_da7f + fake_clock_seq = 0x14c5 + with ( + mock.patch.object(self.uuid, '_last_timestamp_v6', None), + mock.patch.object(self.uuid, 'getnode', return_value=fake_node_value), + mock.patch('time.time_ns', return_value=fake_nanoseconds), + mock.patch('random.getrandbits', return_value=fake_clock_seq) + ): + u = self.uuid.uuid6() + equal(u.variant, self.uuid.RFC_4122) + equal(u.version, 6) + + # 32 (top) | 16 (mid) | 12 (low) == 60 (timestamp) + equal(u.time, 0x1e901fca_7a55_b92) + equal(u.fields[0], 0x1e901fca) # 32 top bits of time + equal(u.fields[1], 0x7a55) # 16 mid bits of time + # 4 bits of version + 12 low bits of time + equal((u.fields[2] >> 12) & 0xf, 6) + equal((u.fields[2] & 0xfff), 0xb92) + # 2 bits of variant + 6 high bits of clock_seq + equal((u.fields[3] >> 6) & 0xf, 2) + equal(u.fields[3] & 0x3f, fake_clock_seq >> 8) + # 8 low bits of clock_seq + equal(u.fields[4], fake_clock_seq & 0xff) + equal(u.fields[5], fake_node_value) + + def test_uuid6_uniqueness(self): + # Test that UUIDv6-generated values are unique. + + # Unlike UUIDv8, only 62 bits can be randomized for UUIDv6. + # In practice, however, it remains unlikely to generate two + # identical UUIDs for the same 60-bit timestamp if neither + # the node ID nor the clock sequence is specified. + uuids = {self.uuid.uuid6() for _ in range(1000)} + self.assertEqual(len(uuids), 1000) + versions = {u.version for u in uuids} + self.assertSetEqual(versions, {6}) + + timestamp = 0x1ec9414c_232a_b00 + fake_nanoseconds = (timestamp - 0x1b21dd21_3814_000) * 100 + + with mock.patch('time.time_ns', return_value=fake_nanoseconds): + def gen(): + with mock.patch.object(self.uuid, '_last_timestamp_v6', None): + return self.uuid.uuid6(node=0, clock_seq=None) + + # By the birthday paradox, sampling N = 1024 UUIDs with identical + # node IDs and timestamps results in duplicates with probability + # close to 1 (not having a duplicate happens with probability of + # order 1E-15) since only the 14-bit clock sequence is randomized. + N = 1024 + uuids = {gen() for _ in range(N)} + self.assertSetEqual({u.node for u in uuids}, {0}) + self.assertSetEqual({u.time for u in uuids}, {timestamp}) + self.assertLess(len(uuids), N, 'collision property does not hold') + + def test_uuid6_node(self): + # Make sure the given node ID appears in the UUID. + # + # Note: when no node ID is specified, the same logic as for UUIDv1 + # is applied to UUIDv6. In particular, there is no need to test that + # getnode() correctly returns positive integers of exactly 48 bits + # since this is done in test_uuid1_eui64(). + self.assertLessEqual(self.uuid.uuid6().node.bit_length(), 48) + + self.assertEqual(self.uuid.uuid6(0).node, 0) + + # tests with explicit values + max_node = 0xffff_ffff_ffff + self.assertEqual(self.uuid.uuid6(max_node).node, max_node) + big_node = 0xE_1234_5678_ABCD # 52-bit node + res_node = 0x0_1234_5678_ABCD # truncated to 48 bits + self.assertEqual(self.uuid.uuid6(big_node).node, res_node) + + # randomized tests + for _ in range(10): + # node with > 48 bits is truncated + for b in [24, 48, 72]: + node = (1 << (b - 1)) | random.getrandbits(b) + with self.subTest(node=node, bitlen=b): + self.assertEqual(node.bit_length(), b) + u = self.uuid.uuid6(node=node) + self.assertEqual(u.node, node & 0xffff_ffff_ffff) + + def test_uuid6_clock_seq(self): + # Make sure the supplied clock sequence appears in the UUID. + # + # For UUIDv6, clock sequence bits are stored from bit 48 to bit 62, + # with the convention that the least significant bit is bit 0 and + # the most significant bit is bit 127. + get_clock_seq = lambda u: (u.int >> 48) & 0x3fff + + u = self.uuid.uuid6() + self.assertLessEqual(get_clock_seq(u).bit_length(), 14) + + # tests with explicit values + big_clock_seq = 0xffff # 16-bit clock sequence + res_clock_seq = 0x3fff # truncated to 14 bits + u = self.uuid.uuid6(clock_seq=big_clock_seq) + self.assertEqual(get_clock_seq(u), res_clock_seq) + + # some randomized tests + for _ in range(10): + # clock_seq with > 14 bits is truncated + for b in [7, 14, 28]: + node = random.getrandbits(48) + clock_seq = (1 << (b - 1)) | random.getrandbits(b) + with self.subTest(node=node, clock_seq=clock_seq, bitlen=b): + self.assertEqual(clock_seq.bit_length(), b) + u = self.uuid.uuid6(node=node, clock_seq=clock_seq) + self.assertEqual(get_clock_seq(u), clock_seq & 0x3fff) + + def test_uuid6_test_vectors(self): + equal = self.assertEqual + # https://www.rfc-editor.org/rfc/rfc9562#name-test-vectors + # (separators are put at the 12th and 28th bits) + timestamp = 0x1ec9414c_232a_b00 + fake_nanoseconds = (timestamp - 0x1b21dd21_3814_000) * 100 + # https://www.rfc-editor.org/rfc/rfc9562#name-example-of-a-uuidv6-value + node = 0x9f6bdeced846 + clock_seq = (3 << 12) | 0x3c8 + + with ( + mock.patch.object(self.uuid, '_last_timestamp_v6', None), + mock.patch('time.time_ns', return_value=fake_nanoseconds) + ): + u = self.uuid.uuid6(node=node, clock_seq=clock_seq) + equal(str(u).upper(), '1EC9414C-232A-6B00-B3C8-9F6BDECED846') + # 32 16 4 12 2 14 48 + # time_hi | time_mid | ver | time_lo | var | clock_seq | node + equal(u.time, timestamp) + equal(u.int & 0xffff_ffff_ffff, node) + equal((u.int >> 48) & 0x3fff, clock_seq) + equal((u.int >> 62) & 0x3, 0b10) + equal((u.int >> 64) & 0xfff, 0xb00) + equal((u.int >> 76) & 0xf, 0x6) + equal((u.int >> 80) & 0xffff, 0x232a) + equal((u.int >> 96) & 0xffff_ffff, 0x1ec9_414c) + + def test_uuid7(self): + equal = self.assertEqual + u = self.uuid.uuid7() + equal(u.variant, self.uuid.RFC_4122) + equal(u.version, 7) + + # 1 Jan 2023 12:34:56.123_456_789 + timestamp_ns = 1672533296_123_456_789 # ns precision + timestamp_ms, _ = divmod(timestamp_ns, 1_000_000) + + for _ in range(100): + counter_hi = random.getrandbits(11) + counter_lo = random.getrandbits(30) + counter = (counter_hi << 30) | counter_lo + + tail = random.getrandbits(32) + # effective number of bits is 32 + 30 + 11 = 73 + random_bits = counter << 32 | tail + + # set all remaining MSB of fake random bits to 1 to ensure that + # the implementation correctly removes them + random_bits = (((1 << 7) - 1) << 73) | random_bits + random_data = random_bits.to_bytes(10) + + with ( + mock.patch.multiple( + self.uuid, + _last_timestamp_v7=None, + _last_counter_v7=0, + ), + mock.patch('time.time_ns', return_value=timestamp_ns), + mock.patch('os.urandom', return_value=random_data) as urand + ): + u = self.uuid.uuid7() + urand.assert_called_once_with(10) + equal(u.variant, self.uuid.RFC_4122) + equal(u.version, 7) + + equal(self.uuid._last_timestamp_v7, timestamp_ms) + equal(self.uuid._last_counter_v7, counter) + + unix_ts_ms = timestamp_ms & 0xffff_ffff_ffff + equal(u.time, unix_ts_ms) + equal((u.int >> 80) & 0xffff_ffff_ffff, unix_ts_ms) + + equal((u.int >> 75) & 1, 0) # check that the MSB is 0 + equal((u.int >> 64) & 0xfff, counter_hi) + equal((u.int >> 32) & 0x3fff_ffff, counter_lo) + equal(u.int & 0xffff_ffff, tail) + + def test_uuid7_uniqueness(self): + # Test that UUIDv7-generated values are unique. + # + # While UUIDv8 has an entropy of 122 bits, those 122 bits may not + # necessarily be sampled from a PRNG. On the other hand, UUIDv7 + # uses os.urandom() as a PRNG which features better randomness. + N = 1000 + uuids = {self.uuid.uuid7() for _ in range(N)} + self.assertEqual(len(uuids), N) + + versions = {u.version for u in uuids} + self.assertSetEqual(versions, {7}) + + def test_uuid7_monotonicity(self): + equal = self.assertEqual + + us = [self.uuid.uuid7() for _ in range(10_000)] + equal(us, sorted(us)) + + with mock.patch.multiple( + self.uuid, + _last_timestamp_v7=0, + _last_counter_v7=0, + ): + # 1 Jan 2023 12:34:56.123_456_789 + timestamp_ns = 1672533296_123_456_789 # ns precision + timestamp_ms, _ = divmod(timestamp_ns, 1_000_000) + + # counter_{hi,lo} are chosen so that "counter + 1" does not overflow + counter_hi = random.getrandbits(11) + counter_lo = random.getrandbits(29) + counter = (counter_hi << 30) | counter_lo + self.assertLess(counter + 1, 0x3ff_ffff_ffff) + + tail = random.getrandbits(32) + random_bits = counter << 32 | tail + random_data = random_bits.to_bytes(10) + + with ( + mock.patch('time.time_ns', return_value=timestamp_ns), + mock.patch('os.urandom', return_value=random_data) as urand + ): + u1 = self.uuid.uuid7() + urand.assert_called_once_with(10) + equal(self.uuid._last_timestamp_v7, timestamp_ms) + equal(self.uuid._last_counter_v7, counter) + equal(u1.time, timestamp_ms) + equal((u1.int >> 64) & 0xfff, counter_hi) + equal((u1.int >> 32) & 0x3fff_ffff, counter_lo) + equal(u1.int & 0xffff_ffff, tail) + + # 1 Jan 2023 12:34:56.123_457_032 (same millisecond but not same ns) + next_timestamp_ns = 1672533296_123_457_032 + next_timestamp_ms, _ = divmod(timestamp_ns, 1_000_000) + equal(timestamp_ms, next_timestamp_ms) + + next_tail_bytes = os.urandom(4) + next_fail = int.from_bytes(next_tail_bytes) + + with ( + mock.patch('time.time_ns', return_value=next_timestamp_ns), + mock.patch('os.urandom', return_value=next_tail_bytes) as urand + ): + u2 = self.uuid.uuid7() + urand.assert_called_once_with(4) + # same milli-second + equal(self.uuid._last_timestamp_v7, timestamp_ms) + # 42-bit counter advanced by 1 + equal(self.uuid._last_counter_v7, counter + 1) + equal(u2.time, timestamp_ms) + equal((u2.int >> 64) & 0xfff, counter_hi) + equal((u2.int >> 32) & 0x3fff_ffff, counter_lo + 1) + equal(u2.int & 0xffff_ffff, next_fail) + + self.assertLess(u1, u2) + + def test_uuid7_timestamp_backwards(self): + equal = self.assertEqual + # 1 Jan 2023 12:34:56.123_456_789 + timestamp_ns = 1672533296_123_456_789 # ns precision + timestamp_ms, _ = divmod(timestamp_ns, 1_000_000) + fake_last_timestamp_v7 = timestamp_ms + 1 + + # counter_{hi,lo} are chosen so that "counter + 1" does not overflow + counter_hi = random.getrandbits(11) + counter_lo = random.getrandbits(29) + counter = (counter_hi << 30) | counter_lo + self.assertLess(counter + 1, 0x3ff_ffff_ffff) + + tail_bytes = os.urandom(4) + tail = int.from_bytes(tail_bytes) + + with ( + mock.patch.multiple( + self.uuid, + _last_timestamp_v7=fake_last_timestamp_v7, + _last_counter_v7=counter, + ), + mock.patch('time.time_ns', return_value=timestamp_ns), + mock.patch('os.urandom', return_value=tail_bytes) as urand + ): + u = self.uuid.uuid7() + urand.assert_called_once_with(4) + equal(u.variant, self.uuid.RFC_4122) + equal(u.version, 7) + equal(self.uuid._last_timestamp_v7, fake_last_timestamp_v7 + 1) + unix_ts_ms = (fake_last_timestamp_v7 + 1) & 0xffff_ffff_ffff + equal(u.time, unix_ts_ms) + equal((u.int >> 80) & 0xffff_ffff_ffff, unix_ts_ms) + # 42-bit counter advanced by 1 + equal(self.uuid._last_counter_v7, counter + 1) + equal((u.int >> 64) & 0xfff, counter_hi) + # 42-bit counter advanced by 1 (counter_hi is untouched) + equal((u.int >> 32) & 0x3fff_ffff, counter_lo + 1) + equal(u.int & 0xffff_ffff, tail) + + def test_uuid7_overflow_counter(self): + equal = self.assertEqual + # 1 Jan 2023 12:34:56.123_456_789 + timestamp_ns = 1672533296_123_456_789 # ns precision + timestamp_ms, _ = divmod(timestamp_ns, 1_000_000) + + new_counter_hi = random.getrandbits(11) + new_counter_lo = random.getrandbits(30) + new_counter = (new_counter_hi << 30) | new_counter_lo + + tail = random.getrandbits(32) + random_bits = (new_counter << 32) | tail + random_data = random_bits.to_bytes(10) + + with ( + mock.patch.multiple( + self.uuid, + _last_timestamp_v7=timestamp_ms, + # same timestamp, but force an overflow on the counter + _last_counter_v7=0x3ff_ffff_ffff, + ), + mock.patch('time.time_ns', return_value=timestamp_ns), + mock.patch('os.urandom', return_value=random_data) as urand + ): + u = self.uuid.uuid7() + urand.assert_called_with(10) + equal(u.variant, self.uuid.RFC_4122) + equal(u.version, 7) + # timestamp advanced due to overflow + equal(self.uuid._last_timestamp_v7, timestamp_ms + 1) + unix_ts_ms = (timestamp_ms + 1) & 0xffff_ffff_ffff + equal(u.time, unix_ts_ms) + equal((u.int >> 80) & 0xffff_ffff_ffff, unix_ts_ms) + # counter overflowed, so we picked a new one + equal(self.uuid._last_counter_v7, new_counter) + equal((u.int >> 64) & 0xfff, new_counter_hi) + equal((u.int >> 32) & 0x3fff_ffff, new_counter_lo) + equal(u.int & 0xffff_ffff, tail) + + def test_uuid8(self): + equal = self.assertEqual + u = self.uuid.uuid8() + + equal(u.variant, self.uuid.RFC_4122) + equal(u.version, 8) + + for (_, hi, mid, lo) in product( + range(10), # repeat 10 times + [None, 0, random.getrandbits(48)], + [None, 0, random.getrandbits(12)], + [None, 0, random.getrandbits(62)], + ): + u = self.uuid.uuid8(hi, mid, lo) + equal(u.variant, self.uuid.RFC_4122) + equal(u.version, 8) + if hi is not None: + equal((u.int >> 80) & 0xffffffffffff, hi) + if mid is not None: + equal((u.int >> 64) & 0xfff, mid) + if lo is not None: + equal(u.int & 0x3fffffffffffffff, lo) + + def test_uuid8_uniqueness(self): + # Test that UUIDv8-generated values are unique (up to a negligible + # probability of failure). There are 122 bits of entropy and assuming + # that the underlying mt-19937-based random generator is sufficiently + # good, it is unlikely to have a collision of two UUIDs. + N = 1000 + uuids = {self.uuid.uuid8() for _ in range(N)} + self.assertEqual(len(uuids), N) + + versions = {u.version for u in uuids} + self.assertSetEqual(versions, {8}) + @support.requires_fork() def testIssue8621(self): # On at least some versions of OSX self.uuid.uuid4 generates @@ -710,6 +1140,23 @@ def test_uuid_weakref(self): weak = weakref.ref(strong) self.assertIs(strong, weak()) + +class CommandLineTestCases: + uuid = None # to be defined in subclasses + + def do_test_standalone_uuid(self, version): + stdout = io.StringIO() + with contextlib.redirect_stdout(stdout): + self.uuid.main() + output = stdout.getvalue().strip() + u = self.uuid.UUID(output) + self.assertEqual(output, str(u)) + self.assertEqual(u.version, version) + + @mock.patch.object(sys, "argv", ["", "-u", "uuid1"]) + def test_cli_uuid1(self): + self.do_test_standalone_uuid(1) + @mock.patch.object(sys, "argv", ["", "-u", "uuid3", "-n", "@dns"]) @mock.patch('sys.stderr', new_callable=io.StringIO) def test_cli_namespace_required_for_uuid3(self, mock_err): @@ -742,6 +1189,20 @@ def test_cli_uuid4_outputted_with_no_args(self): self.assertEqual(output, str(uuid_output)) self.assertEqual(uuid_output.version, 4) + @mock.patch.object(sys, "argv", ["", "-C", "3"]) + def test_cli_uuid4_outputted_with_count(self): + stdout = io.StringIO() + with contextlib.redirect_stdout(stdout): + self.uuid.main() + + output = stdout.getvalue().strip().splitlines() + + # Check that 3 UUIDs in the format of uuid4 have been generated + self.assertEqual(len(output), 3) + for o in output: + uuid_output = self.uuid.UUID(o) + self.assertEqual(uuid_output.version, 4) + @mock.patch.object(sys, "argv", ["", "-u", "uuid3", "-n", "@dns", "-N", "python.org"]) def test_cli_uuid3_ouputted_with_valid_namespace_and_name(self): @@ -770,13 +1231,25 @@ def test_cli_uuid5_ouputted_with_valid_namespace_and_name(self): self.assertEqual(output, str(uuid_output)) self.assertEqual(uuid_output.version, 5) + @mock.patch.object(sys, "argv", ["", "-u", "uuid6"]) + def test_cli_uuid6(self): + self.do_test_standalone_uuid(6) + + @mock.patch.object(sys, "argv", ["", "-u", "uuid7"]) + def test_cli_uuid7(self): + self.do_test_standalone_uuid(7) + + @mock.patch.object(sys, "argv", ["", "-u", "uuid8"]) + def test_cli_uuid8(self): + self.do_test_standalone_uuid(8) + -class TestUUIDWithoutExtModule(BaseTestUUID, unittest.TestCase): +class TestUUIDWithoutExtModule(CommandLineTestCases, BaseTestUUID, unittest.TestCase): uuid = py_uuid @unittest.skipUnless(c_uuid, 'requires the C _uuid module') -class TestUUIDWithExtModule(BaseTestUUID, unittest.TestCase): +class TestUUIDWithExtModule(CommandLineTestCases, BaseTestUUID, unittest.TestCase): uuid = c_uuid def check_has_stable_libuuid_extractable_node(self): diff --git a/Lib/test/test_warnings/__init__.py b/Lib/test/test_warnings/__init__.py index 123d51b77ea..ce1ae9dfa18 100644 --- a/Lib/test/test_warnings/__init__.py +++ b/Lib/test/test_warnings/__init__.py @@ -24,10 +24,13 @@ from warnings import deprecated -py_warnings = import_helper.import_fresh_module('warnings', - blocked=['_warnings']) -c_warnings = import_helper.import_fresh_module('warnings', - fresh=['_warnings']) +py_warnings = import_helper.import_fresh_module('_py_warnings') +py_warnings._set_module(py_warnings) + +c_warnings = import_helper.import_fresh_module( + "warnings", fresh=["_warnings", "_py_warnings"] +) +c_warnings._set_module(c_warnings) @contextmanager def warnings_state(module): @@ -43,15 +46,21 @@ def warnings_state(module): except NameError: pass original_warnings = warning_tests.warnings - original_filters = module.filters - try: + if module._use_context: + saved_context, context = module._new_context() + else: + original_filters = module.filters module.filters = original_filters[:] + try: module.simplefilter("once") warning_tests.warnings = module yield finally: warning_tests.warnings = original_warnings - module.filters = original_filters + if module._use_context: + module._set_context(saved_context) + else: + module.filters = original_filters class TestWarning(Warning): @@ -93,7 +102,7 @@ class PublicAPITests(BaseTest): """ def test_module_all_attribute(self): - self.assertTrue(hasattr(self.module, '__all__')) + self.assertHasAttr(self.module, '__all__') target_api = ["warn", "warn_explicit", "showwarning", "formatwarning", "filterwarnings", "simplefilter", "resetwarnings", "catch_warnings", "deprecated"] @@ -111,14 +120,14 @@ class FilterTests(BaseTest): """Testing the filtering functionality.""" def test_error(self): - with original_warnings.catch_warnings(module=self.module) as w: + with self.module.catch_warnings() as w: self.module.resetwarnings() self.module.filterwarnings("error", category=UserWarning) self.assertRaises(UserWarning, self.module.warn, "FilterTests.test_error") def test_error_after_default(self): - with original_warnings.catch_warnings(module=self.module) as w: + with self.module.catch_warnings() as w: self.module.resetwarnings() message = "FilterTests.test_ignore_after_default" def f(): @@ -136,8 +145,7 @@ def f(): self.assertRaises(UserWarning, f) def test_ignore(self): - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: self.module.resetwarnings() self.module.filterwarnings("ignore", category=UserWarning) self.module.warn("FilterTests.test_ignore", UserWarning) @@ -145,8 +153,7 @@ def test_ignore(self): self.assertEqual(list(__warningregistry__), ['version']) def test_ignore_after_default(self): - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: self.module.resetwarnings() message = "FilterTests.test_ignore_after_default" def f(): @@ -157,44 +164,43 @@ def f(): f() self.assertEqual(len(w), 1) - def test_always(self): - with original_warnings.catch_warnings(record=True, - module=self.module) as w: - self.module.resetwarnings() - self.module.filterwarnings("always", category=UserWarning) - message = "FilterTests.test_always" - def f(): - self.module.warn(message, UserWarning) - f() - self.assertEqual(len(w), 1) - self.assertEqual(w[-1].message.args[0], message) - f() - self.assertEqual(len(w), 2) - self.assertEqual(w[-1].message.args[0], message) + def test_always_and_all(self): + for mode in {"always", "all"}: + with self.module.catch_warnings(record=True) as w: + self.module.resetwarnings() + self.module.filterwarnings(mode, category=UserWarning) + message = "FilterTests.test_always_and_all" + def f(): + self.module.warn(message, UserWarning) + f() + self.assertEqual(len(w), 1) + self.assertEqual(w[-1].message.args[0], message) + f() + self.assertEqual(len(w), 2) + self.assertEqual(w[-1].message.args[0], message) - def test_always_after_default(self): - with original_warnings.catch_warnings(record=True, - module=self.module) as w: - self.module.resetwarnings() - message = "FilterTests.test_always_after_ignore" - def f(): - self.module.warn(message, UserWarning) - f() - self.assertEqual(len(w), 1) - self.assertEqual(w[-1].message.args[0], message) - f() - self.assertEqual(len(w), 1) - self.module.filterwarnings("always", category=UserWarning) - f() - self.assertEqual(len(w), 2) - self.assertEqual(w[-1].message.args[0], message) - f() - self.assertEqual(len(w), 3) - self.assertEqual(w[-1].message.args[0], message) + def test_always_and_all_after_default(self): + for mode in {"always", "all"}: + with self.module.catch_warnings(record=True) as w: + self.module.resetwarnings() + message = "FilterTests.test_always_and_all_after_ignore" + def f(): + self.module.warn(message, UserWarning) + f() + self.assertEqual(len(w), 1) + self.assertEqual(w[-1].message.args[0], message) + f() + self.assertEqual(len(w), 1) + self.module.filterwarnings(mode, category=UserWarning) + f() + self.assertEqual(len(w), 2) + self.assertEqual(w[-1].message.args[0], message) + f() + self.assertEqual(len(w), 3) + self.assertEqual(w[-1].message.args[0], message) def test_default(self): - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: self.module.resetwarnings() self.module.filterwarnings("default", category=UserWarning) message = UserWarning("FilterTests.test_default") @@ -209,8 +215,7 @@ def test_default(self): raise ValueError("loop variant unhandled") def test_module(self): - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: self.module.resetwarnings() self.module.filterwarnings("module", category=UserWarning) message = UserWarning("FilterTests.test_module") @@ -221,8 +226,7 @@ def test_module(self): self.assertEqual(len(w), 0) def test_once(self): - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: self.module.resetwarnings() self.module.filterwarnings("once", category=UserWarning) message = UserWarning("FilterTests.test_once") @@ -237,9 +241,87 @@ def test_once(self): 42) self.assertEqual(len(w), 0) + def test_filter_module(self): + MS_WINDOWS = (sys.platform == 'win32') + with self.module.catch_warnings(record=True) as w: + self.module.simplefilter('error') + self.module.filterwarnings('always', module=r'package\.module\z') + self.module.warn_explicit('msg', UserWarning, 'filename', 42, + module='package.module') + self.assertEqual(len(w), 1) + with self.assertRaises(UserWarning): + self.module.warn_explicit('msg', UserWarning, '/path/to/package/module', 42) + with self.assertRaises(UserWarning): + self.module.warn_explicit('msg', UserWarning, '/path/to/package/module.py', 42) + + with self.module.catch_warnings(record=True) as w: + self.module.simplefilter('error') + self.module.filterwarnings('always', module='package') + self.module.warn_explicit('msg', UserWarning, 'filename', 42, + module='package.module') + self.assertEqual(len(w), 1) + with self.assertRaises(UserWarning): + self.module.warn_explicit('msg', UserWarning, 'filename', 42, + module='other.package.module') + with self.assertRaises(UserWarning): + self.module.warn_explicit('msg', UserWarning, '/path/to/otherpackage/module.py', 42) + + with self.module.catch_warnings(record=True) as w: + self.module.simplefilter('error') + self.module.filterwarnings('always', module=r'/path/to/package/module\z') + self.module.warn_explicit('msg', UserWarning, '/path/to/package/module', 42) + self.assertEqual(len(w), 1) + self.module.warn_explicit('msg', UserWarning, '/path/to/package/module.py', 42) + self.assertEqual(len(w), 2) + with self.assertRaises(UserWarning): + self.module.warn_explicit('msg', UserWarning, '/PATH/TO/PACKAGE/MODULE', 42) + if MS_WINDOWS: + if self.module is py_warnings: + self.module.warn_explicit('msg', UserWarning, r'/path/to/package/module.PY', 42) + self.assertEqual(len(w), 3) + with self.assertRaises(UserWarning): + self.module.warn_explicit('msg', UserWarning, r'/path/to/package/module/__init__.py', 42) + with self.assertRaises(UserWarning): + self.module.warn_explicit('msg', UserWarning, r'/path/to/package/module.pyw', 42) + with self.assertRaises(UserWarning): + self.module.warn_explicit('msg', UserWarning, r'\path\to\package\module', 42) + + with self.module.catch_warnings(record=True) as w: + self.module.simplefilter('error') + self.module.filterwarnings('always', module=r'/path/to/package/__init__\z') + self.module.warn_explicit('msg', UserWarning, '/path/to/package/__init__.py', 42) + self.assertEqual(len(w), 1) + self.module.warn_explicit('msg', UserWarning, '/path/to/package/__init__', 42) + self.assertEqual(len(w), 2) + + if MS_WINDOWS: + with self.module.catch_warnings(record=True) as w: + self.module.simplefilter('error') + self.module.filterwarnings('always', module=r'C:\\path\\to\\package\\module\z') + self.module.warn_explicit('msg', UserWarning, r'C:\path\to\package\module', 42) + self.assertEqual(len(w), 1) + self.module.warn_explicit('msg', UserWarning, r'C:\path\to\package\module.py', 42) + self.assertEqual(len(w), 2) + if self.module is py_warnings: + self.module.warn_explicit('msg', UserWarning, r'C:\path\to\package\module.PY', 42) + self.assertEqual(len(w), 3) + with self.assertRaises(UserWarning): + self.module.warn_explicit('msg', UserWarning, r'C:\path\to\package\module.pyw', 42) + with self.assertRaises(UserWarning): + self.module.warn_explicit('msg', UserWarning, r'C:\PATH\TO\PACKAGE\MODULE', 42) + with self.assertRaises(UserWarning): + self.module.warn_explicit('msg', UserWarning, r'C:/path/to/package/module', 42) + with self.assertRaises(UserWarning): + self.module.warn_explicit('msg', UserWarning, r'C:\path\to\package\module\__init__.py', 42) + + with self.module.catch_warnings(record=True) as w: + self.module.simplefilter('error') + self.module.filterwarnings('always', module=r'<unknown>\z') + self.module.warn_explicit('msg', UserWarning, '', 42) + self.assertEqual(len(w), 1) + def test_module_globals(self): - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: self.module.simplefilter("always", UserWarning) # bpo-33509: module_globals=None must not crash @@ -259,15 +341,14 @@ def test_module_globals(self): self.assertEqual(len(w), 2) def test_inheritance(self): - with original_warnings.catch_warnings(module=self.module) as w: + with self.module.catch_warnings() as w: self.module.resetwarnings() self.module.filterwarnings("error", category=Warning) self.assertRaises(UserWarning, self.module.warn, "FilterTests.test_inheritance", UserWarning) def test_ordering(self): - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: self.module.resetwarnings() self.module.filterwarnings("ignore", category=UserWarning) self.module.filterwarnings("error", category=UserWarning, @@ -282,8 +363,7 @@ def test_ordering(self): def test_filterwarnings(self): # Test filterwarnings(). # Implicitly also tests resetwarnings(). - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: self.module.filterwarnings("error", "", Warning, "", 0) self.assertRaises(UserWarning, self.module.warn, 'convert to error') @@ -307,8 +387,7 @@ def test_filterwarnings(self): self.assertIs(w[-1].category, UserWarning) def test_message_matching(self): - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: self.module.simplefilter("ignore", UserWarning) self.module.filterwarnings("error", "match", UserWarning) self.assertRaises(UserWarning, self.module.warn, "match") @@ -324,54 +403,52 @@ def match(self, a): L[:] = [] L = [("default",X(),UserWarning,X(),0) for i in range(2)] - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: self.module.filters = L self.module.warn_explicit(UserWarning("b"), None, "f.py", 42) self.assertEqual(str(w[-1].message), "b") def test_filterwarnings_duplicate_filters(self): - with original_warnings.catch_warnings(module=self.module): + with self.module.catch_warnings(): self.module.resetwarnings() self.module.filterwarnings("error", category=UserWarning) - self.assertEqual(len(self.module.filters), 1) + self.assertEqual(len(self.module._get_filters()), 1) self.module.filterwarnings("ignore", category=UserWarning) self.module.filterwarnings("error", category=UserWarning) self.assertEqual( - len(self.module.filters), 2, + len(self.module._get_filters()), 2, "filterwarnings inserted duplicate filter" ) self.assertEqual( - self.module.filters[0][0], "error", + self.module._get_filters()[0][0], "error", "filterwarnings did not promote filter to " "the beginning of list" ) def test_simplefilter_duplicate_filters(self): - with original_warnings.catch_warnings(module=self.module): + with self.module.catch_warnings(): self.module.resetwarnings() self.module.simplefilter("error", category=UserWarning) - self.assertEqual(len(self.module.filters), 1) + self.assertEqual(len(self.module._get_filters()), 1) self.module.simplefilter("ignore", category=UserWarning) self.module.simplefilter("error", category=UserWarning) self.assertEqual( - len(self.module.filters), 2, + len(self.module._get_filters()), 2, "simplefilter inserted duplicate filter" ) self.assertEqual( - self.module.filters[0][0], "error", + self.module._get_filters()[0][0], "error", "simplefilter did not promote filter to the beginning of list" ) def test_append_duplicate(self): - with original_warnings.catch_warnings(module=self.module, - record=True) as w: + with self.module.catch_warnings(record=True) as w: self.module.resetwarnings() self.module.simplefilter("ignore") self.module.simplefilter("error", append=True) self.module.simplefilter("ignore", append=True) self.module.warn("test_append_duplicate", category=UserWarning) - self.assertEqual(len(self.module.filters), 2, + self.assertEqual(len(self.module._get_filters()), 2, "simplefilter inserted duplicate filter" ) self.assertEqual(len(w), 0, @@ -401,19 +478,17 @@ def test_argument_validation(self): self.module.simplefilter('ignore', lineno=-1) def test_catchwarnings_with_simplefilter_ignore(self): - with original_warnings.catch_warnings(module=self.module): + with self.module.catch_warnings(module=self.module): self.module.resetwarnings() self.module.simplefilter("error") - with self.module.catch_warnings( - module=self.module, action="ignore" - ): + with self.module.catch_warnings(action="ignore"): self.module.warn("This will be ignored") def test_catchwarnings_with_simplefilter_error(self): - with original_warnings.catch_warnings(module=self.module): + with self.module.catch_warnings(): self.module.resetwarnings() with self.module.catch_warnings( - module=self.module, action="error", category=FutureWarning + action="error", category=FutureWarning ): with support.captured_stderr() as stderr: error_msg = "Other types of warnings are not errors" @@ -435,8 +510,7 @@ class WarnTests(BaseTest): """Test warnings.warn() and warnings.warn_explicit().""" def test_message(self): - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: self.module.simplefilter("once") for i in range(4): text = 'multi %d' %i # Different text on each call. @@ -448,8 +522,7 @@ def test_message(self): def test_warn_nonstandard_types(self): # warn() should handle non-standard types without issue. for ob in (Warning, None, 42): - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: self.module.simplefilter("once") self.module.warn(ob) # Don't directly compare objects since @@ -458,8 +531,7 @@ def test_warn_nonstandard_types(self): def test_filename(self): with warnings_state(self.module): - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: warning_tests.inner("spam1") self.assertEqual(os.path.basename(w[-1].filename), "stacklevel.py") @@ -471,8 +543,7 @@ def test_stacklevel(self): # Test stacklevel argument # make sure all messages are different, so the warning won't be skipped with warnings_state(self.module): - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: warning_tests.inner("spam3", stacklevel=1) self.assertEqual(os.path.basename(w[-1].filename), "stacklevel.py") @@ -494,23 +565,20 @@ def test_stacklevel(self): self.assertEqual(os.path.basename(w[-1].filename), "<sys>") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_stacklevel_import(self): # Issue #24305: With stacklevel=2, module-level warnings should work. import_helper.unload('test.test_warnings.data.import_warning') with warnings_state(self.module): - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: self.module.simplefilter('always') - import test.test_warnings.data.import_warning + import test.test_warnings.data.import_warning # noqa: F401 self.assertEqual(len(w), 1) self.assertEqual(w[0].filename, __file__) def test_skip_file_prefixes(self): with warnings_state(self.module): - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: self.module.simplefilter('always') # Warning never attributed to the data/ package. @@ -533,6 +601,16 @@ def test_skip_file_prefixes(self): warning_tests.package("prefix02", stacklevel=3) self.assertIn("unittest", w[-1].filename) + def test_skip_file_prefixes_file_path(self): + # see: gh-126209 + with warnings_state(self.module): + skipped = warning_tests.__file__ + with self.module.catch_warnings(record=True) as w: + warning_tests.outer("msg", skip_file_prefixes=(skipped,)) + + self.assertEqual(len(w), 1) + self.assertNotEqual(w[-1].filename, skipped) + def test_skip_file_prefixes_type_errors(self): with warnings_state(self.module): warn = warning_tests.warnings.warn @@ -548,23 +626,16 @@ def test_exec_filename(self): codeobj = compile(("import warnings\n" "warnings.warn('hello', UserWarning)"), filename, "exec") - with original_warnings.catch_warnings(record=True) as w: + with self.module.catch_warnings(record=True) as w: self.module.simplefilter("always", category=UserWarning) exec(codeobj) self.assertEqual(w[0].filename, filename) def test_warn_explicit_non_ascii_filename(self): - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: self.module.resetwarnings() self.module.filterwarnings("always", category=UserWarning) - filenames = ["nonascii\xe9\u20ac"] - if not support.is_emscripten: - # JavaScript does not like surrogates. - # Invalid UTF-8 leading byte 0x80 encountered when - # deserializing a UTF-8 string in wasm memory to a JS - # string! - filenames.append("surrogate\udc80") + filenames = ["nonascii\xe9\u20ac", "surrogate\udc80"] for filename in filenames: try: os.fsencode(filename) @@ -625,7 +696,7 @@ class NonWarningSubclass: self.assertIn('category must be a Warning subclass, not ', str(cm.exception)) - with original_warnings.catch_warnings(module=self.module): + with self.module.catch_warnings(): self.module.resetwarnings() self.module.filterwarnings('default') with self.assertWarns(MyWarningClass) as cm: @@ -641,7 +712,7 @@ class NonWarningSubclass: self.assertIsInstance(cm.warning, Warning) def check_module_globals(self, module_globals): - with original_warnings.catch_warnings(module=self.module, record=True) as w: + with self.module.catch_warnings(record=True) as w: self.module.filterwarnings('default') self.module.warn_explicit( 'eggs', UserWarning, 'bar', 1, @@ -654,7 +725,7 @@ def check_module_globals_error(self, module_globals, errmsg, errtype=ValueError) if self.module is py_warnings: self.check_module_globals(module_globals) return - with original_warnings.catch_warnings(module=self.module, record=True) as w: + with self.module.catch_warnings(record=True) as w: self.module.filterwarnings('always') with self.assertRaisesRegex(errtype, re.escape(errmsg)): self.module.warn_explicit( @@ -666,7 +737,7 @@ def check_module_globals_deprecated(self, module_globals, msg): if self.module is py_warnings: self.check_module_globals(module_globals) return - with original_warnings.catch_warnings(module=self.module, record=True) as w: + with self.module.catch_warnings(record=True) as w: self.module.filterwarnings('always') self.module.warn_explicit( 'eggs', UserWarning, 'bar', 1, @@ -734,53 +805,44 @@ def test_gh86298_no_loader_with_spec_loader_okay(self): class CWarnTests(WarnTests, unittest.TestCase): module = c_warnings - # TODO: RUSTPYTHON - @unittest.expectedFailure # As an early adopter, we sanity check the # test.import_helper.import_fresh_module utility function + @unittest.expectedFailure # TODO: RUSTPYTHON def test_accelerated(self): self.assertIsNot(original_warnings, self.module) - self.assertFalse(hasattr(self.module.warn, '__code__')) + self.assertNotHasAttr(self.module.warn, '__code__') - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_gh86298_no_loader_and_spec_is_none(self): - return super().test_gh86298_no_loader_and_spec_is_none() + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_gh86298_loader_and_spec_loader_disagree(self): + return super().test_gh86298_loader_and_spec_loader_disagree() - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_gh86298_loader_is_none_and_spec_is_none(self): return super().test_gh86298_loader_is_none_and_spec_is_none() - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_gh86298_loader_is_none_and_spec_loader_is_none(self): return super().test_gh86298_loader_is_none_and_spec_loader_is_none() - - # TODO: RUSTPYTHON - @unittest.expectedFailure + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_gh86298_no_loader_and_no_spec_loader(self): + return super().test_gh86298_no_loader_and_no_spec_loader() + + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_gh86298_no_loader_and_spec_is_none(self): + return super().test_gh86298_no_loader_and_spec_is_none() + + @unittest.expectedFailure # TODO: RUSTPYTHON def test_gh86298_no_spec(self): return super().test_gh86298_no_spec() - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_gh86298_spec_is_none(self): - return super().test_gh86298_spec_is_none() - - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_gh86298_no_spec_loader(self): return super().test_gh86298_no_spec_loader() - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_gh86298_loader_and_spec_loader_disagree(self): - return super().test_gh86298_loader_and_spec_loader_disagree() - - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_gh86298_no_loader_and_no_spec_loader(self): - return super().test_gh86298_no_loader_and_no_spec_loader() + @unittest.expectedFailure # TODO: RUSTPYTHON + def test_gh86298_spec_is_none(self): + return super().test_gh86298_spec_is_none() class PyWarnTests(WarnTests, unittest.TestCase): module = py_warnings @@ -789,7 +851,7 @@ class PyWarnTests(WarnTests, unittest.TestCase): # test.import_helper.import_fresh_module utility function def test_pure_python(self): self.assertIsNot(original_warnings, self.module) - self.assertTrue(hasattr(self.module.warn, '__code__')) + self.assertHasAttr(self.module.warn, '__code__') class WCmdLineTests(BaseTest): @@ -797,7 +859,7 @@ class WCmdLineTests(BaseTest): def test_improper_input(self): # Uses the private _setoption() function to test the parsing # of command-line warning arguments - with original_warnings.catch_warnings(module=self.module): + with self.module.catch_warnings(): self.assertRaises(self.module._OptionError, self.module._setoption, '1:2:3:4:5:6') self.assertRaises(self.module._OptionError, @@ -816,7 +878,7 @@ def test_improper_input(self): self.assertRaises(UserWarning, self.module.warn, 'convert to error') def test_import_from_module(self): - with original_warnings.catch_warnings(module=self.module): + with self.module.catch_warnings(): self.module._setoption('ignore::Warning') with self.assertRaises(self.module._OptionError): self.module._setoption('ignore::TestWarning') @@ -834,8 +896,6 @@ class CWCmdLineTests(WCmdLineTests, unittest.TestCase): class PyWCmdLineTests(WCmdLineTests, unittest.TestCase): module = py_warnings - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_improper_option(self): # Same as above, but check that the message is printed out when # the interpreter is executed. This also checks that options are @@ -843,8 +903,6 @@ def test_improper_option(self): rc, out, err = assert_python_ok("-Wxxx", "-c", "pass") self.assertIn(b"Invalid -W option ignored: invalid action: 'xxx'", err) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_warnings_bootstrap(self): # Check that the warnings module does get loaded when -W<some option> # is used (see issue #10372 for an example of silent bootstrap failure). @@ -861,11 +919,10 @@ class _WarningsTests(BaseTest, unittest.TestCase): module = c_warnings - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_filter(self): # Everything should function even if 'filters' is not in warnings. - with original_warnings.catch_warnings(module=self.module) as w: + with self.module.catch_warnings() as w: self.module.filterwarnings("error", "", Warning, "", 0) self.assertRaises(UserWarning, self.module.warn, 'convert to error') @@ -873,8 +930,7 @@ def test_filter(self): self.assertRaises(UserWarning, self.module.warn, 'convert to error') - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_onceregistry(self): # Replacing or removing the onceregistry should be okay. global __warningregistry__ @@ -882,8 +938,7 @@ def test_onceregistry(self): try: original_registry = self.module.onceregistry __warningregistry__ = {} - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: self.module.resetwarnings() self.module.filterwarnings("once", category=UserWarning) self.module.warn_explicit(message, UserWarning, "file", 42) @@ -905,15 +960,13 @@ def test_onceregistry(self): finally: self.module.onceregistry = original_registry - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_default_action(self): # Replacing or removing defaultaction should be okay. message = UserWarning("defaultaction test") original = self.module.defaultaction try: - with original_warnings.catch_warnings(record=True, - module=self.module) as w: + with self.module.catch_warnings(record=True) as w: self.module.resetwarnings() registry = {} self.module.warn_explicit(message, UserWarning, "<test>", 42, @@ -946,8 +999,12 @@ def test_default_action(self): def test_showwarning_missing(self): # Test that showwarning() missing is okay. + if self.module._use_context: + # If _use_context is true, the warnings module does not + # override/restore showwarning() + return text = 'del showwarning test' - with original_warnings.catch_warnings(module=self.module): + with self.module.catch_warnings(): self.module.filterwarnings("always", category=UserWarning) del self.module.showwarning with support.captured_output('stderr') as stream: @@ -955,12 +1012,11 @@ def test_showwarning_missing(self): result = stream.getvalue() self.assertIn(text, result) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_showwarnmsg_missing(self): # Test that _showwarnmsg() missing is okay. text = 'del _showwarnmsg test' - with original_warnings.catch_warnings(module=self.module): + with self.module.catch_warnings(): self.module.filterwarnings("always", category=UserWarning) show = self.module._showwarnmsg @@ -974,50 +1030,55 @@ def test_showwarnmsg_missing(self): self.assertIn(text, result) def test_showwarning_not_callable(self): - with original_warnings.catch_warnings(module=self.module): - self.module.filterwarnings("always", category=UserWarning) - self.module.showwarning = print - with support.captured_output('stdout'): - self.module.warn('Warning!') - self.module.showwarning = 23 - self.assertRaises(TypeError, self.module.warn, "Warning!") + orig = self.module.showwarning + try: + with self.module.catch_warnings(): + self.module.filterwarnings("always", category=UserWarning) + self.module.showwarning = print + with support.captured_output('stdout'): + self.module.warn('Warning!') + self.module.showwarning = 23 + self.assertRaises(TypeError, self.module.warn, "Warning!") + finally: + self.module.showwarning = orig def test_show_warning_output(self): # With showwarning() missing, make sure that output is okay. - text = 'test show_warning' - with original_warnings.catch_warnings(module=self.module): - self.module.filterwarnings("always", category=UserWarning) - del self.module.showwarning - with support.captured_output('stderr') as stream: - warning_tests.inner(text) - result = stream.getvalue() - self.assertEqual(result.count('\n'), 2, - "Too many newlines in %r" % result) - first_line, second_line = result.split('\n', 1) - expected_file = os.path.splitext(warning_tests.__file__)[0] + '.py' - first_line_parts = first_line.rsplit(':', 3) - path, line, warning_class, message = first_line_parts - line = int(line) - self.assertEqual(expected_file, path) - self.assertEqual(warning_class, ' ' + UserWarning.__name__) - self.assertEqual(message, ' ' + text) - expected_line = ' ' + linecache.getline(path, line).strip() + '\n' - assert expected_line - self.assertEqual(second_line, expected_line) - - # TODO: RUSTPYTHON - @unittest.expectedFailure + orig = self.module.showwarning + try: + text = 'test show_warning' + with self.module.catch_warnings(): + self.module.filterwarnings("always", category=UserWarning) + del self.module.showwarning + with support.captured_output('stderr') as stream: + warning_tests.inner(text) + result = stream.getvalue() + self.assertEqual(result.count('\n'), 2, + "Too many newlines in %r" % result) + first_line, second_line = result.split('\n', 1) + expected_file = os.path.splitext(warning_tests.__file__)[0] + '.py' + first_line_parts = first_line.rsplit(':', 3) + path, line, warning_class, message = first_line_parts + line = int(line) + self.assertEqual(expected_file, path) + self.assertEqual(warning_class, ' ' + UserWarning.__name__) + self.assertEqual(message, ' ' + text) + expected_line = ' ' + linecache.getline(path, line).strip() + '\n' + assert expected_line + self.assertEqual(second_line, expected_line) + finally: + self.module.showwarning = orig + def test_filename_none(self): # issue #12467: race condition if a warning is emitted at shutdown globals_dict = globals() oldfile = globals_dict['__file__'] try: - catch = original_warnings.catch_warnings(record=True, - module=self.module) + catch = self.module.catch_warnings(record=True) with catch as w: self.module.filterwarnings("always", category=UserWarning) globals_dict['__file__'] = None - original_warnings.warn('test', UserWarning) + self.module.warn('test', UserWarning) self.assertTrue(len(w)) finally: globals_dict['__file__'] = oldfile @@ -1031,8 +1092,7 @@ def test_stderr_none(self): self.assertNotIn(b'Warning!', stderr) self.assertNotIn(b'Error', stderr) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_issue31285(self): # warn_explicit() should neither raise a SystemError nor cause an # assertion failure, in case the return value of get_source() has a @@ -1056,7 +1116,7 @@ def get_source(self, fullname): wmod = self.module - with original_warnings.catch_warnings(module=wmod): + with wmod.catch_warnings(): wmod.filterwarnings('default', category=UserWarning) linecache.clearcache() @@ -1083,7 +1143,7 @@ def test_issue31411(self): # warn_explicit() shouldn't raise a SystemError in case # warnings.onceregistry isn't a dictionary. wmod = self.module - with original_warnings.catch_warnings(module=wmod): + with wmod.catch_warnings(): wmod.filterwarnings('once') with support.swap_attr(wmod, 'onceregistry', None): with self.assertRaises(TypeError): @@ -1094,12 +1154,12 @@ def test_issue31416(self): # warn_explicit() shouldn't cause an assertion failure in case of a # bad warnings.filters or warnings.defaultaction. wmod = self.module - with original_warnings.catch_warnings(module=wmod): - wmod.filters = [(None, None, Warning, None, 0)] + with wmod.catch_warnings(): + wmod._get_filters()[:] = [(None, None, Warning, None, 0)] with self.assertRaises(TypeError): wmod.warn_explicit('foo', Warning, 'bar', 1) - wmod.filters = [] + wmod._get_filters()[:] = [] with support.swap_attr(wmod, 'defaultaction', None), \ self.assertRaises(TypeError): wmod.warn_explicit('foo', Warning, 'bar', 1) @@ -1108,7 +1168,7 @@ def test_issue31416(self): def test_issue31566(self): # warn() shouldn't cause an assertion failure in case of a bad # __name__ global. - with original_warnings.catch_warnings(module=self.module): + with self.module.catch_warnings(): self.module.filterwarnings('error', category=UserWarning) with support.swap_item(globals(), '__name__', b'foo'), \ support.swap_item(globals(), '__file__', None): @@ -1185,8 +1245,7 @@ class CWarningsDisplayTests(WarningsDisplayTests, unittest.TestCase): class PyWarningsDisplayTests(WarningsDisplayTests, unittest.TestCase): module = py_warnings - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_tracemalloc(self): self.addCleanup(os_helper.unlink, os_helper.TESTFN) @@ -1238,16 +1297,18 @@ class CatchWarningTests(BaseTest): """Test catch_warnings().""" def test_catch_warnings_restore(self): + if self.module._use_context: + return # test disabled if using context vars wmod = self.module orig_filters = wmod.filters orig_showwarning = wmod.showwarning # Ensure both showwarning and filters are restored when recording - with wmod.catch_warnings(module=wmod, record=True): + with wmod.catch_warnings(record=True): wmod.filters = wmod.showwarning = object() self.assertIs(wmod.filters, orig_filters) self.assertIs(wmod.showwarning, orig_showwarning) # Same test, but with recording disabled - with wmod.catch_warnings(module=wmod, record=False): + with wmod.catch_warnings(record=False): wmod.filters = wmod.showwarning = object() self.assertIs(wmod.filters, orig_filters) self.assertIs(wmod.showwarning, orig_showwarning) @@ -1255,7 +1316,7 @@ def test_catch_warnings_restore(self): def test_catch_warnings_recording(self): wmod = self.module # Ensure warnings are recorded when requested - with wmod.catch_warnings(module=wmod, record=True) as w: + with wmod.catch_warnings(record=True) as w: self.assertEqual(w, []) self.assertIs(type(w), list) wmod.simplefilter("always") @@ -1269,44 +1330,48 @@ def test_catch_warnings_recording(self): self.assertEqual(w, []) # Ensure warnings are not recorded when not requested orig_showwarning = wmod.showwarning - with wmod.catch_warnings(module=wmod, record=False) as w: + with wmod.catch_warnings(record=False) as w: self.assertIsNone(w) self.assertIs(wmod.showwarning, orig_showwarning) def test_catch_warnings_reentry_guard(self): wmod = self.module # Ensure catch_warnings is protected against incorrect usage - x = wmod.catch_warnings(module=wmod, record=True) + x = wmod.catch_warnings(record=True) self.assertRaises(RuntimeError, x.__exit__) with x: self.assertRaises(RuntimeError, x.__enter__) # Same test, but with recording disabled - x = wmod.catch_warnings(module=wmod, record=False) + x = wmod.catch_warnings(record=False) self.assertRaises(RuntimeError, x.__exit__) with x: self.assertRaises(RuntimeError, x.__enter__) def test_catch_warnings_defaults(self): wmod = self.module - orig_filters = wmod.filters + orig_filters = wmod._get_filters() orig_showwarning = wmod.showwarning # Ensure default behaviour is not to record warnings - with wmod.catch_warnings(module=wmod) as w: + with wmod.catch_warnings() as w: self.assertIsNone(w) self.assertIs(wmod.showwarning, orig_showwarning) - self.assertIsNot(wmod.filters, orig_filters) - self.assertIs(wmod.filters, orig_filters) + self.assertIsNot(wmod._get_filters(), orig_filters) + self.assertIs(wmod._get_filters(), orig_filters) if wmod is sys.modules['warnings']: # Ensure the default module is this one with wmod.catch_warnings() as w: self.assertIsNone(w) self.assertIs(wmod.showwarning, orig_showwarning) - self.assertIsNot(wmod.filters, orig_filters) - self.assertIs(wmod.filters, orig_filters) + self.assertIsNot(wmod._get_filters(), orig_filters) + self.assertIs(wmod._get_filters(), orig_filters) def test_record_override_showwarning_before(self): # Issue #28835: If warnings.showwarning() was overridden, make sure # that catch_warnings(record=True) overrides it again. + if self.module._use_context: + # If _use_context is true, the warnings module does not restore + # showwarning() + return text = "This is a warning" wmod = self.module my_log = [] @@ -1317,7 +1382,7 @@ def my_logger(message, category, filename, lineno, file=None, line=None): # Override warnings.showwarning() before calling catch_warnings() with support.swap_attr(wmod, 'showwarning', my_logger): - with wmod.catch_warnings(module=wmod, record=True) as log: + with wmod.catch_warnings(record=True) as log: self.assertIsNot(wmod.showwarning, my_logger) wmod.simplefilter("always") @@ -1332,6 +1397,10 @@ def my_logger(message, category, filename, lineno, file=None, line=None): def test_record_override_showwarning_inside(self): # Issue #28835: It is possible to override warnings.showwarning() # in the catch_warnings(record=True) context manager. + if self.module._use_context: + # If _use_context is true, the warnings module does not restore + # showwarning() + return text = "This is a warning" wmod = self.module my_log = [] @@ -1340,7 +1409,7 @@ def my_logger(message, category, filename, lineno, file=None, line=None): nonlocal my_log my_log.append(message) - with wmod.catch_warnings(module=wmod, record=True) as log: + with wmod.catch_warnings(record=True) as log: wmod.simplefilter("always") wmod.showwarning = my_logger wmod.warn(text) @@ -1389,8 +1458,7 @@ class PyCatchWarningTests(CatchWarningTests, unittest.TestCase): class EnvironmentVariableTests(BaseTest): - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_single_warning(self): rc, stdout, stderr = assert_python_ok("-c", "import sys; sys.stdout.write(str(sys.warnoptions))", @@ -1398,8 +1466,7 @@ def test_single_warning(self): PYTHONDEVMODE="") self.assertEqual(stdout, b"['ignore::DeprecationWarning']") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_comma_separated_warnings(self): rc, stdout, stderr = assert_python_ok("-c", "import sys; sys.stdout.write(str(sys.warnoptions))", @@ -1408,8 +1475,7 @@ def test_comma_separated_warnings(self): self.assertEqual(stdout, b"['ignore::DeprecationWarning', 'ignore::UnicodeWarning']") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON @force_not_colorized def test_envvar_and_command_line(self): rc, stdout, stderr = assert_python_ok("-Wignore::UnicodeWarning", "-c", @@ -1419,8 +1485,7 @@ def test_envvar_and_command_line(self): self.assertEqual(stdout, b"['ignore::DeprecationWarning', 'ignore::UnicodeWarning']") - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON @force_not_colorized def test_conflicting_envvar_and_command_line(self): rc, stdout, stderr = assert_python_failure("-Werror::DeprecationWarning", "-c", @@ -1462,7 +1527,7 @@ def test_default_filter_configuration(self): code = "import sys; sys.modules.pop('warnings', None); sys.modules['_warnings'] = None; " else: code = "" - code += "import warnings; [print(f) for f in warnings.filters]" + code += "import warnings; [print(f) for f in warnings._get_filters()]" rc, stdout, stderr = assert_python_ok("-c", code, __isolated=True) stdout_lines = [line.strip() for line in stdout.splitlines()] @@ -1470,8 +1535,7 @@ def test_default_filter_configuration(self): self.assertEqual(stdout_lines, expected_output) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON @unittest.skipUnless(sys.getfilesystemencoding() != 'ascii', 'requires non-ascii filesystemencoding') def test_nonascii(self): @@ -1486,17 +1550,25 @@ def test_nonascii(self): class CEnvironmentVariableTests(EnvironmentVariableTests, unittest.TestCase): module = c_warnings - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; Lists differ def test_default_filter_configuration(self): - # XXX: RUSTPYHTON; remove the entire function when fixed - super().test_default_filter_configuration() - + return super().test_default_filter_configuration() class PyEnvironmentVariableTests(EnvironmentVariableTests, unittest.TestCase): module = py_warnings +class LocksTest(unittest.TestCase): + @support.cpython_only + @unittest.skipUnless(c_warnings, 'C module is required') + def test_release_lock_no_lock(self): + with self.assertRaisesRegex( + RuntimeError, + 'cannot release un-acquired lock', + ): + c_warnings._release_lock() + + class _DeprecatedTest(BaseTest, unittest.TestCase): """Test _deprecated().""" @@ -1551,8 +1623,7 @@ def test_issue_8766(self): class FinalizationTest(unittest.TestCase): - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_finalization(self): # Issue #19421: warnings.warn() should not crash # during Python finalization @@ -1570,8 +1641,7 @@ def __del__(self): self.assertEqual(err.decode().rstrip(), '<string>:7: UserWarning: test') - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_late_resource_warning(self): # Issue #21925: Emitting a ResourceWarning late during the Python # shutdown must be logged. @@ -1582,15 +1652,178 @@ def test_late_resource_warning(self): # (_warnings will try to import it) code = "f = open(%a)" % __file__ rc, out, err = assert_python_ok("-Wd", "-c", code) - self.assertTrue(err.startswith(expected), ascii(err)) + self.assertStartsWith(err, expected) # import the warnings module code = "import warnings; f = open(%a)" % __file__ rc, out, err = assert_python_ok("-Wd", "-c", code) - self.assertTrue(err.startswith(expected), ascii(err)) + self.assertStartsWith(err, expected) -class DeprecatedTests(unittest.TestCase): +class AsyncTests(BaseTest): + """Verifies that the catch_warnings() context manager behaves + as expected when used inside async co-routines. This requires + that the context_aware_warnings flag is enabled, so that + the context manager uses a context variable. + """ + + def setUp(self): + super().setUp() + self.module.resetwarnings() + + @unittest.skipIf(not sys.flags.context_aware_warnings, + "requires context aware warnings") + def test_async_context(self): + import asyncio + + # Events to force the execution interleaving we want. + step_a1 = asyncio.Event() + step_a2 = asyncio.Event() + step_b1 = asyncio.Event() + step_b2 = asyncio.Event() + + async def run_a(): + with self.module.catch_warnings(record=True) as w: + await step_a1.wait() + # The warning emitted here should be caught be the enclosing + # context manager. + self.module.warn('run_a warning', UserWarning) + step_b1.set() + await step_a2.wait() + self.assertEqual(len(w), 1) + self.assertEqual(w[0].message.args[0], 'run_a warning') + step_b2.set() + + async def run_b(): + with self.module.catch_warnings(record=True) as w: + step_a1.set() + await step_b1.wait() + # The warning emitted here should be caught be the enclosing + # context manager. + self.module.warn('run_b warning', UserWarning) + step_a2.set() + await step_b2.wait() + self.assertEqual(len(w), 1) + self.assertEqual(w[0].message.args[0], 'run_b warning') + + async def run_tasks(): + await asyncio.gather(run_a(), run_b()) + + asyncio.run(run_tasks()) + + @unittest.skipIf(not sys.flags.context_aware_warnings, + "requires context aware warnings") + def test_async_task_inherit(self): + """Check that a new asyncio task inherits warnings context from the + coroutine that spawns it. + """ + import asyncio + + step1 = asyncio.Event() + step2 = asyncio.Event() + + async def run_child1(): + await step1.wait() + # This should be recorded by the run_parent() catch_warnings + # context. + self.module.warn('child warning', UserWarning) + step2.set() + + async def run_child2(): + # This establishes a new catch_warnings() context. The + # run_child1() task should still be using the context from + # run_parent() if context-aware warnings are enabled. + with self.module.catch_warnings(record=True) as w: + step1.set() + await step2.wait() + + async def run_parent(): + with self.module.catch_warnings(record=True) as w: + await asyncio.gather(run_child1(), run_child2()) + self.assertEqual(len(w), 1) + self.assertEqual(w[0].message.args[0], 'child warning') + + asyncio.run(run_parent()) + + +class CAsyncTests(AsyncTests, unittest.TestCase): + module = c_warnings + + +class PyAsyncTests(AsyncTests, unittest.TestCase): + module = py_warnings + + +class ThreadTests(BaseTest): + """Verifies that the catch_warnings() context manager behaves as + expected when used within threads. This requires that both the + context_aware_warnings flag and thread_inherit_context flags are enabled. + """ + + ENABLE_THREAD_TESTS = (sys.flags.context_aware_warnings and + sys.flags.thread_inherit_context) + + def setUp(self): + super().setUp() + self.module.resetwarnings() + + @unittest.skipIf(not ENABLE_THREAD_TESTS, + "requires thread-safe warnings flags") + def test_threaded_context(self): + import threading + + barrier = threading.Barrier(2, timeout=2) + + def run_a(): + with self.module.catch_warnings(record=True) as w: + barrier.wait() + # The warning emitted here should be caught be the enclosing + # context manager. + self.module.warn('run_a warning', UserWarning) + barrier.wait() + self.assertEqual(len(w), 1) + self.assertEqual(w[0].message.args[0], 'run_a warning') + # Should be caught be the catch_warnings() context manager of run_threads() + self.module.warn('main warning', UserWarning) + + def run_b(): + with self.module.catch_warnings(record=True) as w: + barrier.wait() + # The warning emitted here should be caught be the enclosing + # context manager. + barrier.wait() + self.module.warn('run_b warning', UserWarning) + self.assertEqual(len(w), 1) + self.assertEqual(w[0].message.args[0], 'run_b warning') + # Should be caught be the catch_warnings() context manager of run_threads() + self.module.warn('main warning', UserWarning) + + def run_threads(): + threads = [ + threading.Thread(target=run_a), + threading.Thread(target=run_b), + ] + with self.module.catch_warnings(record=True) as w: + for thread in threads: + thread.start() + for thread in threads: + thread.join() + self.assertEqual(len(w), 2) + self.assertEqual(w[0].message.args[0], 'main warning') + self.assertEqual(w[1].message.args[0], 'main warning') + + run_threads() + + +class CThreadTests(ThreadTests, unittest.TestCase): + module = c_warnings + + +class PyThreadTests(ThreadTests, unittest.TestCase): + module = py_warnings + + +class DeprecatedTests(PyPublicAPITests): def test_dunder_deprecated(self): @deprecated("A will go away soon") class A: @@ -1618,6 +1851,7 @@ def h(x): self.assertEqual(len(overloads), 2) self.assertEqual(overloads[0].__deprecated__, "no more ints") + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_class(self): @deprecated("A will go away soon") class A: @@ -1629,6 +1863,7 @@ class A: with self.assertRaises(TypeError): A(42) + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_class_with_init(self): @deprecated("HasInit will go away soon") class HasInit: @@ -1639,6 +1874,7 @@ def __init__(self, x): instance = HasInit(42) self.assertEqual(instance.x, 42) + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_class_with_new(self): has_new_called = False @@ -1657,6 +1893,7 @@ def __init__(self, x) -> None: self.assertEqual(instance.x, 42) self.assertTrue(has_new_called) + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_class_with_inherited_new(self): new_base_called = False @@ -1678,6 +1915,7 @@ class HasInheritedNew(NewBase): self.assertEqual(instance.x, 42) self.assertTrue(new_base_called) + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_class_with_new_but_no_init(self): new_called = False @@ -1695,6 +1933,7 @@ def __new__(cls, x): self.assertEqual(instance.x, 42) self.assertTrue(new_called) + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_mixin_class(self): @deprecated("Mixin will go away soon") class Mixin: @@ -1711,6 +1950,7 @@ class Child(Base, Mixin): instance = Child(42) self.assertEqual(instance.a, 42) + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_do_not_shadow_user_arguments(self): new_called = False new_called_cls = None @@ -1730,6 +1970,7 @@ class Foo(metaclass=MyMeta, cls='haha'): self.assertTrue(new_called) self.assertEqual(new_called_cls, 'haha') + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_existing_init_subclass(self): @deprecated("C will go away soon") class C: @@ -1746,6 +1987,7 @@ class D(C): self.assertTrue(D.inited) self.assertIsInstance(D(), D) # no deprecation + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_existing_init_subclass_in_base(self): class Base: def __init_subclass__(cls, x) -> None: @@ -1766,6 +2008,27 @@ class D(C, x=3): self.assertEqual(D.inited, 3) + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered + def test_existing_init_subclass_in_sibling_base(self): + @deprecated("A will go away soon") + class A: + pass + class B: + def __init_subclass__(cls, x): + super().__init_subclass__() + cls.inited = x + + with self.assertWarnsRegex(DeprecationWarning, "A will go away soon"): + class C(A, B, x=42): + pass + self.assertEqual(C.inited, 42) + + with self.assertWarnsRegex(DeprecationWarning, "A will go away soon"): + class D(B, A, x=42): + pass + self.assertEqual(D.inited, 42) + + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_init_subclass_has_correct_cls(self): init_subclass_saw = None @@ -1783,6 +2046,7 @@ class C(Base): self.assertIs(init_subclass_saw, C) + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_init_subclass_with_explicit_classmethod(self): init_subclass_saw = None @@ -1801,6 +2065,7 @@ class C(Base): self.assertIs(init_subclass_saw, C) + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_function(self): @deprecated("b will go away soon") def b(): @@ -1809,6 +2074,7 @@ def b(): with self.assertWarnsRegex(DeprecationWarning, "b will go away soon"): b() + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_method(self): class Capybara: @deprecated("x will go away soon") @@ -1819,6 +2085,7 @@ def x(self): with self.assertWarnsRegex(DeprecationWarning, "x will go away soon"): instance.x() + @unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered def test_property(self): class Capybara: @property @@ -1846,6 +2113,7 @@ def no_more_setting(self, value): with self.assertWarnsRegex(DeprecationWarning, "no more setting"): instance.no_more_setting = 42 + @unittest.expectedFailure # TODO: RUSTPYTHON; RuntimeWarning not triggered def test_category(self): @deprecated("c will go away soon", category=RuntimeWarning) def c(): diff --git a/Lib/test/test_warnings/data/stacklevel.py b/Lib/test/test_warnings/data/stacklevel.py index c6dd24733b3..fe36242d3d2 100644 --- a/Lib/test/test_warnings/data/stacklevel.py +++ b/Lib/test/test_warnings/data/stacklevel.py @@ -4,11 +4,13 @@ import warnings from test.test_warnings.data import package_helper -def outer(message, stacklevel=1): - inner(message, stacklevel) -def inner(message, stacklevel=1): - warnings.warn(message, stacklevel=stacklevel) +def outer(message, stacklevel=1, skip_file_prefixes=()): + inner(message, stacklevel, skip_file_prefixes) + +def inner(message, stacklevel=1, skip_file_prefixes=()): + warnings.warn(message, stacklevel=stacklevel, + skip_file_prefixes=skip_file_prefixes) def package(message, *, stacklevel): package_helper.inner_api(message, stacklevel=stacklevel, diff --git a/Lib/test/test_weakref.py b/Lib/test/test_weakref.py index c118987411f..910108406be 100644 --- a/Lib/test/test_weakref.py +++ b/Lib/test/test_weakref.py @@ -13,7 +13,7 @@ import textwrap from test import support -from test.support import script_helper, ALWAYS_EQ, suppress_immortalization +from test.support import script_helper, ALWAYS_EQ from test.support import gc_collect from test.support import import_helper from test.support import threading_helper @@ -289,7 +289,7 @@ def test_ref_reuse(self): self.assertEqual(weakref.getweakrefcount(o), 1, "wrong weak ref count for object after deleting proxy") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_proxy_reuse(self): o = C() proxy1 = weakref.proxy(o) @@ -337,7 +337,7 @@ def __bytes__(self): self.assertIn("__bytes__", dir(weakref.proxy(instance))) self.assertEqual(bytes(weakref.proxy(instance)), b"bytes") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_proxy_index(self): class C: def __index__(self): @@ -346,7 +346,7 @@ def __index__(self): p = weakref.proxy(o) self.assertEqual(operator.index(p), 10) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_proxy_div(self): class C: def __floordiv__(self, other): @@ -359,7 +359,7 @@ def __ifloordiv__(self, other): p //= 5 self.assertEqual(p, 21) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_proxy_matmul(self): class C: def __matmul__(self, other): @@ -383,11 +383,11 @@ def __imatmul__(self, other): # was not honored, and was broken in different ways for # PyWeakref_NewRef() and PyWeakref_NewProxy(). (Two tests.) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_shared_ref_without_callback(self): self.check_shared_without_callback(weakref.ref) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_shared_proxy_without_callback(self): self.check_shared_without_callback(weakref.proxy) @@ -409,7 +409,7 @@ def check_shared_without_callback(self, makeref): p2 = makeref(o) self.assertIs(p1, p2, "callbacks were None, NULL in the C API") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_callable_proxy(self): o = Callable() ref1 = weakref.proxy(o) @@ -439,7 +439,7 @@ def check_proxy(self, o, proxy): self.assertEqual(proxy.foo, 2, "proxy does not reflect attribute modification") del o.foo - self.assertFalse(hasattr(proxy, 'foo'), + self.assertNotHasAttr(proxy, 'foo', "proxy does not reflect attribute removal") proxy.foo = 1 @@ -449,7 +449,7 @@ def check_proxy(self, o, proxy): self.assertEqual(o.foo, 2, "object does not reflect attribute modification via proxy") del proxy.foo - self.assertFalse(hasattr(o, 'foo'), + self.assertNotHasAttr(o, 'foo', "object does not reflect attribute removal via proxy") def test_proxy_deletion(self): @@ -504,7 +504,7 @@ def __iter__(self): # Calls proxy.__next__ self.assertEqual(list(weak_it), [4, 5, 6]) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_proxy_bad_next(self): # bpo-44720: PyIter_Next() shouldn't be called if the reference # isn't an iterator. @@ -594,7 +594,7 @@ def test_getweakrefs(self): self.assertEqual(weakref.getweakrefs(1), [], "list of refs does not match for int") - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_newstyle_number_ops(self): class F(float): pass @@ -668,7 +668,6 @@ class C(object): # deallocation of c2. del c2 - @suppress_immortalization() def test_callback_in_cycle(self): import gc @@ -761,8 +760,7 @@ class D: del c1, c2, C, D gc.collect() - @unittest.expectedFailure # TODO: RUSTPYTHON - @suppress_immortalization() + @unittest.expectedFailure # TODO: RUSTPYTHON def test_callback_in_cycle_resurrection(self): import gc @@ -809,7 +807,7 @@ def C_went_away(ignore): gc.collect() self.assertEqual(alist, []) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_callbacks_on_callback(self): import gc @@ -848,11 +846,9 @@ def cb(self, ignore): gc.collect() self.assertEqual(alist, []) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_gc_during_ref_creation(self): self.check_gc_during_creation(weakref.ref) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_gc_during_proxy_creation(self): self.check_gc_during_creation(weakref.proxy) @@ -901,7 +897,7 @@ def test_init(self): # No exception should be raised here gc.collect() - @suppress_immortalization() + @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: <class 'test.test_weakref.ReferencesTestCase.test_classes.<locals>.A'> != None def test_classes(self): # Check that classes are weakrefable. class A(object): @@ -1003,7 +999,7 @@ def cb(wparent): del root gc.collect() - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_callback_attribute(self): x = Object(1) callback = lambda ref: None @@ -1013,7 +1009,7 @@ def test_callback_attribute(self): ref2 = weakref.ref(x) self.assertIsNone(ref2.__callback__) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_callback_attribute_after_deletion(self): x = Object(1) ref = weakref.ref(x, self.callback) @@ -1065,7 +1061,6 @@ def callback(obj): class SubclassableWeakrefTestCase(TestBase): - @unittest.expectedFailure # TODO: RUSTPYTHON def test_subclass_refs(self): class MyRef(weakref.ref): def __init__(self, ob, callback=None, value=42): @@ -1084,7 +1079,6 @@ def __call__(self): self.assertIsNone(mr()) self.assertTrue(mr.called) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_subclass_refs_dont_replace_standard_refs(self): class MyRef(weakref.ref): pass @@ -1128,7 +1122,7 @@ def meth(self): self.assertEqual(r.slot1, "abc") self.assertEqual(r.slot2, "def") self.assertEqual(r.meth(), "abcdef") - self.assertFalse(hasattr(r, "__dict__")) + self.assertNotHasAttr(r, "__dict__") def test_subclass_refs_with_cycle(self): """Confirm https://bugs.python.org/issue3100 is fixed.""" @@ -1336,11 +1330,11 @@ def check_len_cycles(self, dict_type, cons): self.assertIn(n1, (0, 1)) self.assertEqual(n2, 0) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_weak_keyed_len_cycles(self): self.check_len_cycles(weakref.WeakKeyDictionary, lambda k: (k, 1)) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_weak_valued_len_cycles(self): self.check_len_cycles(weakref.WeakValueDictionary, lambda k: (1, k)) @@ -1368,11 +1362,9 @@ def check_len_race(self, dict_type, cons): self.assertGreaterEqual(n2, 0) self.assertLessEqual(n2, n1) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_weak_keyed_len_race(self): self.check_len_race(weakref.WeakKeyDictionary, lambda k: (k, 1)) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_weak_valued_len_race(self): self.check_len_race(weakref.WeakValueDictionary, lambda k: (1, k)) @@ -1873,7 +1865,7 @@ def test_weak_valued_delitem(self): self.assertEqual(len(d), 1) self.assertEqual(list(d.items()), [('something else', o2)]) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_weak_keyed_bad_delitem(self): d = weakref.WeakKeyDictionary() o = Object('1') @@ -2054,11 +2046,13 @@ def pop_and_collect(lst): raise exc[0] @threading_helper.requires_working_threading() + @support.requires_resource('cpu') def test_threaded_weak_key_dict_copy(self): # Issue #35615: Weakref keys or values getting GC'ed during dict # copying should not result in a crash. self.check_threaded_weak_dict_copy(weakref.WeakKeyDictionary, False) + @unittest.skip("TODO: RUSTPYTHON; occasionally crash (malloc corruption)") @threading_helper.requires_working_threading() @support.requires_resource('cpu') def test_threaded_weak_key_dict_deepcopy(self): @@ -2066,13 +2060,15 @@ def test_threaded_weak_key_dict_deepcopy(self): # copying should not result in a crash. self.check_threaded_weak_dict_copy(weakref.WeakKeyDictionary, True) - @unittest.skip('TODO: RUSTPYTHON; occasionally crash (Exit code -6)') + @unittest.skip("TODO: RUSTPYTHON; occasionally crash (malloc corruption)") @threading_helper.requires_working_threading() + @support.requires_resource('cpu') def test_threaded_weak_value_dict_copy(self): # Issue #35615: Weakref keys or values getting GC'ed during dict # copying should not result in a crash. self.check_threaded_weak_dict_copy(weakref.WeakValueDictionary, False) + @unittest.skip("TODO: RUSTPYTHON; occasionally crash (malloc corruption)") @threading_helper.requires_working_threading() @support.requires_resource('cpu') def test_threaded_weak_value_dict_deepcopy(self): @@ -2257,7 +2253,7 @@ def error(): assert f3.atexit == True assert f4.atexit == True - @unittest.skipIf(sys.platform == 'win32', 'TODO: RUSTPYTHON; Windows') + @unittest.skipIf(sys.platform == "win32", "TODO: RUSTPYTHON; Windows") def test_atexit(self): prog = ('from test.test_weakref import FinalizeTestCase;'+ 'FinalizeTestCase.run_in_child()') @@ -2268,7 +2264,7 @@ def test_atexit(self): class ModuleTestCase(unittest.TestCase): - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON def test_names(self): for name in ('ReferenceType', 'ProxyType', 'CallableProxyType', 'WeakMethod', 'WeakSet', 'WeakKeyDictionary', 'WeakValueDictionary'): @@ -2368,9 +2364,7 @@ def test_names(self): __test__ = {'libreftest' : libreftest} def load_tests(loader, tests, pattern): - # TODO: RUSTPYTHON - # The doctest fails - # tests.addTest(doctest.DocTestSuite()) + tests.addTest(doctest.DocTestSuite()) return tests diff --git a/Lib/test/test_weakset.py b/Lib/test/test_weakset.py index 5e8cacc09dc..af9bbe7cd41 100644 --- a/Lib/test/test_weakset.py +++ b/Lib/test/test_weakset.py @@ -425,8 +425,6 @@ def test_len_cycles(self): self.assertIn(n1, (0, 1)) self.assertEqual(n2, 0) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_len_race(self): # Extended sanity checks for len() in the face of cyclic collection self.addCleanup(gc.set_threshold, *gc.get_threshold()) diff --git a/Lib/test/test_winapi.py b/Lib/test/test_winapi.py index 99b5a0dfd16..e64208330ad 100644 --- a/Lib/test/test_winapi.py +++ b/Lib/test/test_winapi.py @@ -2,10 +2,7 @@ import os import pathlib -import random import re -import threading -import time import unittest from test.support import import_helper, os_helper @@ -158,4 +155,4 @@ def test_namedpipe(self): self.assertEqual((b'', 0), _winapi.PeekNamedPipe(pipe, 8)[:2]) pipe2.write(b'testdata') pipe2.flush() - self.assertEqual((b'testdata', 8), _winapi.PeekNamedPipe(pipe, 8)[:2]) \ No newline at end of file + self.assertEqual((b'testdata', 8), _winapi.PeekNamedPipe(pipe, 8)[:2]) diff --git a/Lib/test/test_wsgiref.py b/Lib/test/test_wsgiref.py index 1a3b4d4b721..d546e3ef219 100644 --- a/Lib/test/test_wsgiref.py +++ b/Lib/test/test_wsgiref.py @@ -134,7 +134,6 @@ def test_environ(self): b"Python test,Python test 2;query=test;/path/" ) - @unittest.expectedFailure # TODO: RUSTPYTHON; http library needs to be updated def test_request_length(self): out, err = run_amock(data=b"GET " + (b"x" * 65537) + b" HTTP/1.0\n\n") self.assertEqual(out.splitlines()[0], diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py index 11c617b5f34..9d6d39307ff 100644 --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -384,7 +384,6 @@ def test_simpleops(self): self.serialize_check(element, '<tag key="value"><subtag /><subtag /></tag>') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_cdata(self): # Test CDATA handling (etc). @@ -407,7 +406,6 @@ def test_file_init(self): self.assertEqual(tree.find("element/../empty-element").tag, 'empty-element') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_path_cache(self): # Check that the path cache behaves sanely. @@ -424,7 +422,6 @@ def test_path_cache(self): for i in range(600): ET.ElementTree(elem).find('./'+str(i)) self.assertLess(len(ElementPath._cache), 500) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_copy(self): # Test copy handling (etc). @@ -888,7 +885,6 @@ def end_ns(self, prefix): ('end-ns', ''), ]) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_initialize_parser_without_target(self): # Explicit None parser = ET.XMLParser(target=None) @@ -936,14 +932,12 @@ def test_children(self): elem.clear() self.assertEqual(list(elem), []) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_writestring(self): elem = ET.XML("<html><body>text</body></html>") self.assertEqual(ET.tostring(elem), b'<html><body>text</body></html>') elem = ET.fromstring("<html><body>text</body></html>") self.assertEqual(ET.tostring(elem), b'<html><body>text</body></html>') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_indent(self): elem = ET.XML("<root></root>") ET.indent(elem) @@ -988,7 +982,6 @@ def test_indent(self): b'</html>' ) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_indent_space(self): elem = ET.XML("<html><body><p>pre<br/>post</p><p>text</p></body></html>") ET.indent(elem, space='\t') @@ -1014,7 +1007,6 @@ def test_indent_space(self): b'</html>' ) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_indent_space_caching(self): elem = ET.XML("<html><body><p>par</p><p>text</p><p><br/></p><p /></body></html>") ET.indent(elem) @@ -1031,7 +1023,6 @@ def test_indent_space_caching(self): len({id(el.tail) for el in elem.iter()}), ) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_indent_level(self): elem = ET.XML("<html><body><p>pre<br/>post</p><p>text</p></body></html>") with self.assertRaises(ValueError): @@ -1064,7 +1055,6 @@ def test_indent_level(self): b' </html>' ) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_tostring_default_namespace(self): elem = ET.XML('<body xmlns="http://effbot.org/ns"><tag/></body>') self.assertEqual( @@ -1076,7 +1066,6 @@ def test_tostring_default_namespace(self): '<body xmlns="http://effbot.org/ns"><tag /></body>' ) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_tostring_default_namespace_different_namespace(self): elem = ET.XML('<body xmlns="http://effbot.org/ns"><tag/></body>') self.assertEqual( @@ -1084,14 +1073,12 @@ def test_tostring_default_namespace_different_namespace(self): '<ns1:body xmlns="foobar" xmlns:ns1="http://effbot.org/ns"><ns1:tag /></ns1:body>' ) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_tostring_default_namespace_original_no_namespace(self): elem = ET.XML('<body><tag/></body>') EXPECTED_MSG = '^cannot use non-qualified names with default_namespace option$' with self.assertRaisesRegex(ValueError, EXPECTED_MSG): ET.tostring(elem, encoding='unicode', default_namespace='foobar') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_tostring_no_xml_declaration(self): elem = ET.XML('<body><tag/></body>') self.assertEqual( @@ -1099,7 +1086,6 @@ def test_tostring_no_xml_declaration(self): '<body><tag /></body>' ) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_tostring_xml_declaration(self): elem = ET.XML('<body><tag/></body>') self.assertEqual( @@ -1107,7 +1093,6 @@ def test_tostring_xml_declaration(self): b"<?xml version='1.0' encoding='utf8'?>\n<body><tag /></body>" ) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_tostring_xml_declaration_unicode_encoding(self): elem = ET.XML('<body><tag/></body>') self.assertEqual( @@ -1115,7 +1100,6 @@ def test_tostring_xml_declaration_unicode_encoding(self): "<?xml version='1.0' encoding='utf-8'?>\n<body><tag /></body>" ) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_tostring_xml_declaration_cases(self): elem = ET.XML('<body><tag>ø</tag></body>') TESTCASES = [ @@ -1160,7 +1144,6 @@ def test_tostring_xml_declaration_cases(self): expected_retval ) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_tostringlist_default_namespace(self): elem = ET.XML('<body xmlns="http://effbot.org/ns"><tag/></body>') self.assertEqual( @@ -1172,7 +1155,6 @@ def test_tostringlist_default_namespace(self): '<body xmlns="http://effbot.org/ns"><tag /></body>' ) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_tostringlist_xml_declaration(self): elem = ET.XML('<body><tag/></body>') self.assertEqual( @@ -1252,7 +1234,6 @@ def bxml(encoding): self.assertRaises(ValueError, ET.XML, xml('undefined').encode('ascii')) self.assertRaises(LookupError, ET.XML, xml('xxx').encode('ascii')) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_methods(self): # Test serialization methods. @@ -1268,7 +1249,6 @@ def test_methods(self): '<html><link><script>1 < 2</script></html>\n') self.assertEqual(serialize(e, method="text"), '1 < 2\n') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_issue18347(self): e = ET.XML('<html><CamelCase>text</CamelCase></html>') self.assertEqual(serialize(e), @@ -1413,7 +1393,6 @@ def test_qname(self): self.assertNotEqual(q1, 'ns:tag') self.assertEqual(q1, '{ns}tag') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_doctype_public(self): # Test PUBLIC doctype. @@ -1496,7 +1475,6 @@ def test_processinginstruction(self): b"<?xml version='1.0' encoding='latin-1'?>\n" b"<?test <testing&>\xe3?>") - @unittest.expectedFailure # TODO: RUSTPYTHON def test_html_empty_elems_serialization(self): # issue 15970 # from http://www.w3.org/TR/html401/index/elements.html @@ -1775,7 +1753,6 @@ def test_events_pi(self): self._feed(parser, "<?pitarget some text ?>\n") self.assert_events(parser, [('pi', (ET.PI, 'pitarget some text '))]) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_events_sequence(self): # Test that events can be some sequence that's not just a tuple or list eventset = {'end', 'start'} @@ -1795,7 +1772,6 @@ def __next__(self): self._feed(parser, "<foo>bar</foo>") self.assert_event_tags(parser, [('start', 'foo'), ('end', 'foo')]) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_unknown_event(self): with self.assertRaises(ValueError): ET.XMLPullParser(events=('start', 'end', 'bogus')) @@ -2210,7 +2186,6 @@ def test_bug_xmltoolkit25(self): self.assertEqual(tree.findtext("tag"), 'text') self.assertEqual(tree.findtext("section/tag"), 'subtext') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_bug_xmltoolkit28(self): # .//tag causes exceptions @@ -2218,7 +2193,6 @@ def test_bug_xmltoolkit28(self): self.assertEqual(summarize_list(tree.findall(".//thead")), []) self.assertEqual(summarize_list(tree.findall(".//tbody")), ['tbody']) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_bug_xmltoolkitX1(self): # dump() doesn't flush the output buffer @@ -2253,7 +2227,6 @@ def test_bug_xmltoolkit39(self): self.assertEqual(ET.tostring(tree, "utf-8"), b'<tag \xc3\xa4ttr="v\xc3\xa4lue" />') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_bug_xmltoolkit54(self): # problems handling internally defined entities @@ -2273,7 +2246,6 @@ def test_bug_xmltoolkit55(self): self.assertEqual(str(cm.exception), 'undefined entity &ldots;: line 1, column 36') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_bug_xmltoolkit60(self): # Handle crash in stream source. @@ -2301,7 +2273,6 @@ def test_bug_xmltoolkit62(self): self.assertEqual(t.find('.//paragraph').text, 'A new cultivar of Begonia plant named \u2018BCT9801BEG\u2019.') - @unittest.expectedFailure # TODO: RUSTPYTHON @unittest.skipIf(sys.gettrace(), "Skips under coverage.") def test_bug_xmltoolkit63(self): # Check reference leak. @@ -2329,7 +2300,6 @@ def test_bug_200708_newline(self): self.assertEqual(ET.tostring(ET.XML(ET.tostring(e))), b'<SomeTag text="def _f():&#10; return 3&#10;" />') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_bug_200708_close(self): # Test default builder. parser = ET.XMLParser() # default @@ -2422,7 +2392,6 @@ def test_bug_1534630(self): e = bob.close() self.assertEqual(serialize(e), '<tag />') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_issue6233(self): e = ET.XML(b"<?xml version='1.0' encoding='utf-8'?>" b'<body>t\xc3\xa3g</body>') @@ -2743,7 +2712,6 @@ def test_pickle(self): self.assertEqual(len(e2), 2) self.assertEqualElements(e, e2) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_pickle_issue18997(self): for proto in range(2, pickle.HIGHEST_PROTOCOL + 1): for dumper, loader in product(self.modules, repeat=2): @@ -3427,7 +3395,6 @@ def test_findall(self): self.assertEqual(summarize_list(e.findall(".//tag[. = 'subtext']")), ['tag', 'tag']) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_test_find_with_ns(self): e = ET.XML(SAMPLE_XML_NS) self.assertEqual(summarize_list(e.findall('tag')), []) @@ -3438,7 +3405,6 @@ def test_test_find_with_ns(self): summarize_list(e.findall(".//{http://effbot.org/ns}tag")), ['{http://effbot.org/ns}tag'] * 3) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_findall_different_nsmaps(self): root = ET.XML(''' <a xmlns:x="X" xmlns:y="Y"> @@ -3456,7 +3422,6 @@ def test_findall_different_nsmaps(self): self.assertEqual(len(root.findall(".//xx:b", namespaces=nsmap)), 2) self.assertEqual(len(root.findall(".//b", namespaces=nsmap)), 1) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_findall_wildcard(self): root = ET.XML(''' <a xmlns:x="X" xmlns:y="Y"> @@ -3501,7 +3466,6 @@ def test_findall_wildcard(self): self.assertEqual(summarize_list(root.findall(".//{}b")), summarize_list(root.findall(".//b"))) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_bad_find(self): e = ET.XML(SAMPLE_XML) with self.assertRaisesRegex(SyntaxError, 'cannot use absolute path'): @@ -3527,7 +3491,6 @@ class ElementIterTest(unittest.TestCase): def _ilist(self, elem, tag=None): return summarize_list(elem.iter(tag)) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_basic(self): doc = ET.XML("<html><body>this is a <i>paragraph</i>.</body>..</html>") self.assertEqual(self._ilist(doc), ['html', 'body', 'i']) @@ -3576,7 +3539,6 @@ def test_corners(self): del a[1] self.assertEqual(self._ilist(a), ['a', 'd']) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_iter_by_tag(self): doc = ET.XML(''' <document> @@ -3641,7 +3603,6 @@ def _check_sample1_element(self, e): self.assertEqual(child.tail, 'tail') self.assertEqual(child.attrib, {}) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_dummy_builder(self): class BaseDummyBuilder: def close(self): @@ -3689,7 +3650,6 @@ def test_treebuilder_pi(self): self.assertEqual(b.pi('target'), (len('target'), None)) self.assertEqual(b.pi('pitarget', ' text '), (len('pitarget'), ' text ')) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_late_tail(self): # Issue #37399: The tail of an ignored comment could overwrite the text before it. class TreeBuilderSubclass(ET.TreeBuilder): @@ -3751,14 +3711,12 @@ class TreeBuilderSubclass(ET.TreeBuilder): self.assertEqual(a[0].tail, 'tail') self.assertEqual(a.text, "text\n") - @unittest.expectedFailure # TODO: RUSTPYTHON def test_treebuilder_elementfactory_none(self): parser = ET.XMLParser(target=ET.TreeBuilder(element_factory=None)) parser.feed(self.sample1) e = parser.close() self._check_sample1_element(e) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_subclass(self): class MyTreeBuilder(ET.TreeBuilder): def foobar(self, x): @@ -3773,7 +3731,6 @@ def foobar(self, x): e = parser.close() self._check_sample1_element(e) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_subclass_comment_pi(self): class MyTreeBuilder(ET.TreeBuilder): def foobar(self, x): @@ -3789,7 +3746,6 @@ def foobar(self, x): e = parser.close() self._check_sample1_element(e) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_element_factory(self): lst = [] def myfactory(tag, attrib): @@ -3813,13 +3769,11 @@ def _check_element_factory_class(self, cls): self.assertIsInstance(e, cls) self._check_sample1_element(e) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_element_factory_subclass(self): class MyElement(ET.Element): pass self._check_element_factory_class(MyElement) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_element_factory_pure_python_subclass(self): # Mimic SimpleTAL's behaviour (issue #16089): both versions of # TreeBuilder should be able to cope with a subclass of the @@ -3851,7 +3805,6 @@ def close(self): ('html', '-//W3C//DTD XHTML 1.0 Transitional//EN', 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd')) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_builder_lookup_errors(self): class RaisingBuilder: def __init__(self, raise_in=None, what=ValueError): @@ -3892,14 +3845,12 @@ def _check_sample_element(self, e): self.assertEqual(e[0].tag, 'line') self.assertEqual(e[0].text, '22') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_constructor_args(self): parser2 = ET.XMLParser(encoding='utf-8', target=ET.TreeBuilder()) parser2.feed(self.sample1) self._check_sample_element(parser2.close()) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_subclass(self): class MyParser(ET.XMLParser): pass @@ -3907,7 +3858,6 @@ class MyParser(ET.XMLParser): parser.feed(self.sample1) self._check_sample_element(parser.close()) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_doctype_warning(self): with warnings.catch_warnings(): warnings.simplefilter('error', DeprecationWarning) @@ -3946,7 +3896,6 @@ def doctype(self, name, pubid, system): ('html', '-//W3C//DTD XHTML 1.0 Transitional//EN', 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd')) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_inherited_doctype(self): '''Ensure that ordinary usage is not deprecated (Issue 19176)''' with warnings.catch_warnings(): @@ -3969,7 +3918,6 @@ def test_parse_string(self): class NamespaceParseTest(unittest.TestCase): - @unittest.expectedFailure # TODO: RUSTPYTHON def test_find_with_namespace(self): nsmap = {'h': 'hello', 'f': 'foo'} doc = ET.fromstring(SAMPLE_XML_NS_ELEMS) @@ -4146,7 +4094,6 @@ def f(): e[:1] = (f() for i in range(2)) class IOTest(unittest.TestCase): - @unittest.expectedFailure # TODO: RUSTPYTHON def test_encoding(self): # Test encoding issues. elem = ET.Element("tag") @@ -4216,7 +4163,6 @@ def test_encoding(self): ("<?xml version='1.0' encoding='%s'?>\n" "<tag key=\"åöö&lt;&gt;\" />" % enc).encode(enc)) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_write_to_filename(self): self.addCleanup(os_helper.unlink, TESTFN) tree = ET.ElementTree(ET.XML('''<site>\xf8</site>''')) @@ -4224,7 +4170,6 @@ def test_write_to_filename(self): with open(TESTFN, 'rb') as f: self.assertEqual(f.read(), b'''<site>&#248;</site>''') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_write_to_filename_with_encoding(self): self.addCleanup(os_helper.unlink, TESTFN) tree = ET.ElementTree(ET.XML('''<site>\xf8</site>''')) @@ -4238,7 +4183,6 @@ def test_write_to_filename_with_encoding(self): b'''<?xml version='1.0' encoding='ISO-8859-1'?>\n''' b'''<site>\xf8</site>''')) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_write_to_filename_as_unicode(self): self.addCleanup(os_helper.unlink, TESTFN) with open(TESTFN, 'w') as f: @@ -4250,7 +4194,6 @@ def test_write_to_filename_as_unicode(self): with open(TESTFN, 'rb') as f: self.assertEqual(f.read(), b"<site>\xc3\xb8</site>") - @unittest.expectedFailure # TODO: RUSTPYTHON def test_write_to_text_file(self): self.addCleanup(os_helper.unlink, TESTFN) tree = ET.ElementTree(ET.XML('''<site>\xf8</site>''')) @@ -4272,7 +4215,6 @@ def test_write_to_text_file(self): with open(TESTFN, 'rb') as f: self.assertEqual(f.read(), b'''<site>\xf8</site>''') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_write_to_binary_file(self): self.addCleanup(os_helper.unlink, TESTFN) tree = ET.ElementTree(ET.XML('''<site>\xf8</site>''')) @@ -4282,7 +4224,6 @@ def test_write_to_binary_file(self): with open(TESTFN, 'rb') as f: self.assertEqual(f.read(), b'''<site>&#248;</site>''') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_write_to_binary_file_with_encoding(self): self.addCleanup(os_helper.unlink, TESTFN) tree = ET.ElementTree(ET.XML('''<site>\xf8</site>''')) @@ -4300,7 +4241,6 @@ def test_write_to_binary_file_with_encoding(self): b'''<?xml version='1.0' encoding='ISO-8859-1'?>\n''' b'''<site>\xf8</site>''') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_write_to_binary_file_with_bom(self): self.addCleanup(os_helper.unlink, TESTFN) tree = ET.ElementTree(ET.XML('''<site>\xf8</site>''')) @@ -4321,28 +4261,24 @@ def test_write_to_binary_file_with_bom(self): '''<?xml version='1.0' encoding='utf-16'?>\n''' '''<site>\xf8</site>'''.encode("utf-16")) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_read_from_stringio(self): tree = ET.ElementTree() stream = io.StringIO('''<?xml version="1.0"?><site></site>''') tree.parse(stream) self.assertEqual(tree.getroot().tag, 'site') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_write_to_stringio(self): tree = ET.ElementTree(ET.XML('''<site>\xf8</site>''')) stream = io.StringIO() tree.write(stream, encoding='unicode') self.assertEqual(stream.getvalue(), '''<site>\xf8</site>''') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_read_from_bytesio(self): tree = ET.ElementTree() raw = io.BytesIO(b'''<?xml version="1.0"?><site></site>''') tree.parse(raw) self.assertEqual(tree.getroot().tag, 'site') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_write_to_bytesio(self): tree = ET.ElementTree(ET.XML('''<site>\xf8</site>''')) raw = io.BytesIO() @@ -4352,7 +4288,6 @@ def test_write_to_bytesio(self): class dummy: pass - @unittest.expectedFailure # TODO: RUSTPYTHON def test_read_from_user_text_reader(self): stream = io.StringIO('''<?xml version="1.0"?><site></site>''') reader = self.dummy() @@ -4361,7 +4296,6 @@ def test_read_from_user_text_reader(self): tree.parse(reader) self.assertEqual(tree.getroot().tag, 'site') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_write_to_user_text_writer(self): tree = ET.ElementTree(ET.XML('''<site>\xf8</site>''')) stream = io.StringIO() @@ -4370,7 +4304,6 @@ def test_write_to_user_text_writer(self): tree.write(writer, encoding='unicode') self.assertEqual(stream.getvalue(), '''<site>\xf8</site>''') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_read_from_user_binary_reader(self): raw = io.BytesIO(b'''<?xml version="1.0"?><site></site>''') reader = self.dummy() @@ -4380,7 +4313,6 @@ def test_read_from_user_binary_reader(self): self.assertEqual(tree.getroot().tag, 'site') tree = ET.ElementTree() - @unittest.expectedFailure # TODO: RUSTPYTHON def test_write_to_user_binary_writer(self): tree = ET.ElementTree(ET.XML('''<site>\xf8</site>''')) raw = io.BytesIO() @@ -4389,7 +4321,6 @@ def test_write_to_user_binary_writer(self): tree.write(writer) self.assertEqual(raw.getvalue(), b'''<site>&#248;</site>''') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_write_to_user_binary_writer_with_bom(self): tree = ET.ElementTree(ET.XML('''<site />''')) raw = io.BytesIO() @@ -4402,7 +4333,6 @@ def test_write_to_user_binary_writer_with_bom(self): '''<?xml version='1.0' encoding='utf-16'?>\n''' '''<site />'''.encode("utf-16")) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_tostringlist_invariant(self): root = ET.fromstring('<tag>foo</tag>') self.assertEqual( @@ -4412,7 +4342,6 @@ def test_tostringlist_invariant(self): ET.tostring(root, 'utf-16'), b''.join(ET.tostringlist(root, 'utf-16'))) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_short_empty_elements(self): root = ET.fromstring('<tag>a<x />b<y></y>c</tag>') self.assertEqual( @@ -4452,7 +4381,6 @@ def test_error_code(self): class KeywordArgsTest(unittest.TestCase): # Test various issues with keyword arguments passed to ET.Element # constructor and methods - @unittest.expectedFailure # TODO: RUSTPYTHON def test_issue14818(self): x = ET.XML("<a>foo</a>") self.assertEqual(x.find('a', None), diff --git a/Lib/test/test_zipfile/_path/test_path.py b/Lib/test/test_zipfile/_path/test_path.py index 2d1c06cd968..5c69c77f7d8 100644 --- a/Lib/test/test_zipfile/_path/test_path.py +++ b/Lib/test/test_zipfile/_path/test_path.py @@ -275,7 +275,8 @@ def test_pathlike_construction(self, alpharep): """ zipfile_ondisk = self.zipfile_ondisk(alpharep) pathlike = FakePath(str(zipfile_ondisk)) - zipfile.Path(pathlike) + root = zipfile.Path(pathlike) + root.root.close() @pass_alpharep def test_traverse_pathlike(self, alpharep): @@ -374,6 +375,7 @@ def test_root_on_disk(self, alpharep): root = zipfile.Path(self.zipfile_ondisk(alpharep)) assert root.name == 'alpharep.zip' == root.filename.name assert root.stem == 'alpharep' == root.filename.stem + root.root.close() @pass_alpharep def test_suffix(self, alpharep): @@ -565,7 +567,7 @@ def test_inheritance(self, alpharep): file = cls(alpharep).joinpath('some dir').parent assert isinstance(file, cls) - @unittest.skipIf(sys.platform == 'win32', "TODO: RUSTPYTHON, fails on Windows") + @unittest.skipIf(sys.platform == 'win32', 'TODO: RUSTPYTHON; fails on Windows') @parameterize( ['alpharep', 'path_type', 'subpath'], itertools.product( @@ -576,11 +578,13 @@ def test_inheritance(self, alpharep): ) def test_pickle(self, alpharep, path_type, subpath): zipfile_ondisk = path_type(str(self.zipfile_ondisk(alpharep))) - - saved_1 = pickle.dumps(zipfile.Path(zipfile_ondisk, at=subpath)) + root = zipfile.Path(zipfile_ondisk, at=subpath) + saved_1 = pickle.dumps(root) + root.root.close() restored_1 = pickle.loads(saved_1) first, *rest = restored_1.iterdir() assert first.read_text(encoding='utf-8').startswith('content of ') + restored_1.root.close() @pass_alpharep def test_extract_orig_with_implied_dirs(self, alpharep): @@ -592,6 +596,7 @@ def test_extract_orig_with_implied_dirs(self, alpharep): # wrap the zipfile for its side effect zipfile.Path(zf) zf.extractall(source_path.parent) + zf.close() @pass_alpharep def test_getinfo_missing(self, alpharep): diff --git a/Lib/test/test_zipfile/test_core.py b/Lib/test/test_zipfile/test_core.py index fa1feef00cd..63413d7b944 100644 --- a/Lib/test/test_zipfile/test_core.py +++ b/Lib/test/test_zipfile/test_core.py @@ -302,26 +302,26 @@ def test_low_compression(self): self.assertEqual(openobj.read(1), b'2') def test_writestr_compression(self): - zipfp = zipfile.ZipFile(TESTFN2, "w") - zipfp.writestr("b.txt", "hello world", compress_type=self.compression) - info = zipfp.getinfo('b.txt') - self.assertEqual(info.compress_type, self.compression) + with zipfile.ZipFile(TESTFN2, "w") as zipfp: + zipfp.writestr("b.txt", "hello world", compress_type=self.compression) + info = zipfp.getinfo('b.txt') + self.assertEqual(info.compress_type, self.compression) def test_writestr_compresslevel(self): - zipfp = zipfile.ZipFile(TESTFN2, "w", compresslevel=1) - zipfp.writestr("a.txt", "hello world", compress_type=self.compression) - zipfp.writestr("b.txt", "hello world", compress_type=self.compression, - compresslevel=2) + with zipfile.ZipFile(TESTFN2, "w", compresslevel=1) as zipfp: + zipfp.writestr("a.txt", "hello world", compress_type=self.compression) + zipfp.writestr("b.txt", "hello world", compress_type=self.compression, + compresslevel=2) - # Compression level follows the constructor. - a_info = zipfp.getinfo('a.txt') - self.assertEqual(a_info.compress_type, self.compression) - self.assertEqual(a_info.compress_level, 1) + # Compression level follows the constructor. + a_info = zipfp.getinfo('a.txt') + self.assertEqual(a_info.compress_type, self.compression) + self.assertEqual(a_info.compress_level, 1) - # Compression level is overridden. - b_info = zipfp.getinfo('b.txt') - self.assertEqual(b_info.compress_type, self.compression) - self.assertEqual(b_info._compresslevel, 2) + # Compression level is overridden. + b_info = zipfp.getinfo('b.txt') + self.assertEqual(b_info.compress_type, self.compression) + self.assertEqual(b_info._compresslevel, 2) def test_read_return_size(self): # Issue #9837: ZipExtFile.read() shouldn't return more bytes @@ -884,6 +884,8 @@ def make_zip64_file( self, file_size_64_set=False, file_size_extra=False, compress_size_64_set=False, compress_size_extra=False, header_offset_64_set=False, header_offset_extra=False, + extensible_data=b'', + end_of_central_dir_size=None, offset_to_end_of_central_dir=None, ): """Generate bytes sequence for a zip with (incomplete) zip64 data. @@ -937,6 +939,12 @@ def make_zip64_file( central_dir_size = struct.pack('<Q', 58 + 8 * len(central_zip64_fields)) offset_to_central_dir = struct.pack('<Q', 50 + 8 * len(local_zip64_fields)) + if end_of_central_dir_size is None: + end_of_central_dir_size = 44 + len(extensible_data) + if offset_to_end_of_central_dir is None: + offset_to_end_of_central_dir = (108 + + 8 * len(local_zip64_fields) + + 8 * len(central_zip64_fields)) local_extra_length = struct.pack("<H", 4 + 8 * len(local_zip64_fields)) central_extra_length = struct.pack("<H", 4 + 8 * len(central_zip64_fields)) @@ -965,14 +973,17 @@ def make_zip64_file( + filename + central_extra # Zip64 end of central directory - + b"PK\x06\x06,\x00\x00\x00\x00\x00\x00\x00-\x00-" - + b"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00" + + b"PK\x06\x06" + + struct.pack('<Q', end_of_central_dir_size) + + b"-\x00-\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00" + b"\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00" + central_dir_size + offset_to_central_dir + + extensible_data # Zip64 end of central directory locator - + b"PK\x06\x07\x00\x00\x00\x00l\x00\x00\x00\x00\x00\x00\x00\x01" - + b"\x00\x00\x00" + + b"PK\x06\x07\x00\x00\x00\x00" + + struct.pack('<Q', offset_to_end_of_central_dir) + + b"\x01\x00\x00\x00" # end of central directory + b"PK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x00:\x00\x00\x002\x00" + b"\x00\x00\x00\x00" @@ -1003,6 +1014,7 @@ def test_bad_zip64_extra(self): with self.assertRaises(zipfile.BadZipFile) as e: zipfile.ZipFile(io.BytesIO(missing_file_size_extra)) self.assertIn('file size', str(e.exception).lower()) + self.assertTrue(zipfile.is_zipfile(io.BytesIO(missing_file_size_extra))) # zip64 file size present, zip64 compress size present, one field in # extra, expecting two, equals missing compress size. @@ -1014,6 +1026,7 @@ def test_bad_zip64_extra(self): with self.assertRaises(zipfile.BadZipFile) as e: zipfile.ZipFile(io.BytesIO(missing_compress_size_extra)) self.assertIn('compress size', str(e.exception).lower()) + self.assertTrue(zipfile.is_zipfile(io.BytesIO(missing_compress_size_extra))) # zip64 compress size present, no fields in extra, expecting one, # equals missing compress size. @@ -1023,6 +1036,7 @@ def test_bad_zip64_extra(self): with self.assertRaises(zipfile.BadZipFile) as e: zipfile.ZipFile(io.BytesIO(missing_compress_size_extra)) self.assertIn('compress size', str(e.exception).lower()) + self.assertTrue(zipfile.is_zipfile(io.BytesIO(missing_compress_size_extra))) # zip64 file size present, zip64 compress size present, zip64 header # offset present, two fields in extra, expecting three, equals missing @@ -1037,6 +1051,7 @@ def test_bad_zip64_extra(self): with self.assertRaises(zipfile.BadZipFile) as e: zipfile.ZipFile(io.BytesIO(missing_header_offset_extra)) self.assertIn('header offset', str(e.exception).lower()) + self.assertTrue(zipfile.is_zipfile(io.BytesIO(missing_header_offset_extra))) # zip64 compress size present, zip64 header offset present, one field # in extra, expecting two, equals missing header offset @@ -1049,6 +1064,7 @@ def test_bad_zip64_extra(self): with self.assertRaises(zipfile.BadZipFile) as e: zipfile.ZipFile(io.BytesIO(missing_header_offset_extra)) self.assertIn('header offset', str(e.exception).lower()) + self.assertTrue(zipfile.is_zipfile(io.BytesIO(missing_header_offset_extra))) # zip64 file size present, zip64 header offset present, one field in # extra, expecting two, equals missing header offset @@ -1061,6 +1077,7 @@ def test_bad_zip64_extra(self): with self.assertRaises(zipfile.BadZipFile) as e: zipfile.ZipFile(io.BytesIO(missing_header_offset_extra)) self.assertIn('header offset', str(e.exception).lower()) + self.assertTrue(zipfile.is_zipfile(io.BytesIO(missing_header_offset_extra))) # zip64 header offset present, no fields in extra, expecting one, # equals missing header offset @@ -1072,6 +1089,63 @@ def test_bad_zip64_extra(self): with self.assertRaises(zipfile.BadZipFile) as e: zipfile.ZipFile(io.BytesIO(missing_header_offset_extra)) self.assertIn('header offset', str(e.exception).lower()) + self.assertTrue(zipfile.is_zipfile(io.BytesIO(missing_header_offset_extra))) + + def test_bad_zip64_end_of_central_dir(self): + zipdata = self.make_zip64_file(end_of_central_dir_size=0) + with self.assertRaisesRegex(zipfile.BadZipFile, 'Corrupt.*record'): + zipfile.ZipFile(io.BytesIO(zipdata)) + self.assertFalse(zipfile.is_zipfile(io.BytesIO(zipdata))) + + zipdata = self.make_zip64_file(end_of_central_dir_size=100) + with self.assertRaisesRegex(zipfile.BadZipFile, 'Corrupt.*record'): + zipfile.ZipFile(io.BytesIO(zipdata)) + self.assertFalse(zipfile.is_zipfile(io.BytesIO(zipdata))) + + zipdata = self.make_zip64_file(offset_to_end_of_central_dir=0) + with self.assertRaisesRegex(zipfile.BadZipFile, 'Corrupt.*record'): + zipfile.ZipFile(io.BytesIO(zipdata)) + self.assertFalse(zipfile.is_zipfile(io.BytesIO(zipdata))) + + zipdata = self.make_zip64_file(offset_to_end_of_central_dir=1000) + with self.assertRaisesRegex(zipfile.BadZipFile, 'Corrupt.*locator'): + zipfile.ZipFile(io.BytesIO(zipdata)) + self.assertFalse(zipfile.is_zipfile(io.BytesIO(zipdata))) + + def test_zip64_end_of_central_dir_record_not_found(self): + zipdata = self.make_zip64_file() + zipdata = zipdata.replace(b"PK\x06\x06", b'\x00'*4) + with self.assertRaisesRegex(zipfile.BadZipFile, 'record not found'): + zipfile.ZipFile(io.BytesIO(zipdata)) + self.assertFalse(zipfile.is_zipfile(io.BytesIO(zipdata))) + + zipdata = self.make_zip64_file( + extensible_data=b'\xca\xfe\x04\x00\x00\x00data') + zipdata = zipdata.replace(b"PK\x06\x06", b'\x00'*4) + with self.assertRaisesRegex(zipfile.BadZipFile, 'record not found'): + zipfile.ZipFile(io.BytesIO(zipdata)) + self.assertFalse(zipfile.is_zipfile(io.BytesIO(zipdata))) + + def test_zip64_extensible_data(self): + # These values are what is set in the make_zip64_file method. + expected_file_size = 8 + expected_compress_size = 8 + expected_header_offset = 0 + expected_content = b"test1234" + + zipdata = self.make_zip64_file( + extensible_data=b'\xca\xfe\x04\x00\x00\x00data') + with zipfile.ZipFile(io.BytesIO(zipdata)) as zf: + zinfo = zf.infolist()[0] + self.assertEqual(zinfo.file_size, expected_file_size) + self.assertEqual(zinfo.compress_size, expected_compress_size) + self.assertEqual(zinfo.header_offset, expected_header_offset) + self.assertEqual(zf.read(zinfo), expected_content) + self.assertTrue(zipfile.is_zipfile(io.BytesIO(zipdata))) + + with self.assertRaisesRegex(zipfile.BadZipFile, 'record not found'): + zipfile.ZipFile(io.BytesIO(b'prepended' + zipdata)) + self.assertFalse(zipfile.is_zipfile(io.BytesIO(b'prepended' + zipdata))) def test_generated_valid_zip64_extra(self): # These values are what is set in the make_zip64_file method. @@ -1357,8 +1431,7 @@ def requiresWriteAccess(self, path): self.skipTest('requires write access to the installed location') unlink(filename) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_write_pyfile(self): self.requiresWriteAccess(os.path.dirname(__file__)) with TemporaryFile() as t, zipfile.PyZipFile(t, "w") as zipfp: @@ -1389,8 +1462,7 @@ def test_write_pyfile(self): self.assertNotIn(bn, zipfp.namelist()) self.assertCompiledIn(bn, zipfp.namelist()) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_write_python_package(self): import email packagedir = os.path.dirname(email.__file__) @@ -1405,8 +1477,7 @@ def test_write_python_package(self): self.assertCompiledIn('email/__init__.py', names) self.assertCompiledIn('email/mime/text.py', names) - # TODO: RUSTPYTHON - AttributeError: module 'os' has no attribute 'supports_effective_ids' - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; - AttributeError: module 'os' has no attribute 'supports_effective_ids' def test_write_filtered_python_package(self): import test packagedir = os.path.dirname(test.__file__) @@ -1437,8 +1508,7 @@ def filter(path): print(reportStr) self.assertTrue('SyntaxError' not in reportStr) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON def test_write_with_optimization(self): import email packagedir = os.path.dirname(email.__file__) @@ -2189,6 +2259,7 @@ def test_empty_zipfile(self): zipf = zipfile.ZipFile(TESTFN, mode="r") except zipfile.BadZipFile: self.fail("Unable to create empty ZIP file in 'w' mode") + zipf.close() zipf = zipfile.ZipFile(TESTFN, mode="a") zipf.close() @@ -2196,6 +2267,7 @@ def test_empty_zipfile(self): zipf = zipfile.ZipFile(TESTFN, mode="r") except: self.fail("Unable to create empty ZIP file in 'a' mode") + zipf.close() def test_open_empty_file(self): # Issue 1710703: Check that opening a file with less than 22 bytes @@ -3020,11 +3092,6 @@ def test_write_after_read(self): self.assertEqual(data1, self.data1) self.assertEqual(data2, self.data2) - # TODO: RUSTPYTHON other tests can impact the file descriptor incrementor - # by leaving file handles unclosed. If there are more than 100 files in - # TESTFN and references to them are left unclosed and ungarbage collected - # in another test, then fileno() will always be too high for this test to - # pass. The solution is to increase the number of files from 100 to 200 def test_many_opens(self): # Verify that read() and open() promptly close the file descriptor, # and don't rely on the garbage collector to free resources. @@ -3348,8 +3415,7 @@ def test_read_zip_with_exe_prepended(self): def test_read_zip64_with_exe_prepended(self): self._test_zip_works(self.exe_zip64) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON @unittest.skipUnless(sys.executable, 'sys.executable required.') @unittest.skipUnless(os.access('/bin/bash', os.X_OK), 'Test relies on #!/bin/bash working.') @@ -3358,8 +3424,7 @@ def test_execute_zip2(self): output = subprocess.check_output([self.exe_zip, sys.executable]) self.assertIn(b'number in executable: 5', output) - # TODO: RUSTPYTHON - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON @unittest.skipUnless(sys.executable, 'sys.executable required.') @unittest.skipUnless(os.access('/bin/bash', os.X_OK), 'Test relies on #!/bin/bash working.') @@ -3369,7 +3434,6 @@ def test_execute_zip64(self): self.assertIn(b'number in executable: 5', output) -# TODO: RUSTPYTHON @unittest.skip("TODO: RUSTPYTHON shift_jis encoding unsupported") class EncodedMetadataTests(unittest.TestCase): file_names = ['\u4e00', '\u4e8c', '\u4e09'] # Han 'one', 'two', 'three' @@ -3377,8 +3441,10 @@ class EncodedMetadataTests(unittest.TestCase): "This is pure ASCII.\n".encode('ascii'), # This is modern Japanese. (UTF-8) "\u3053\u308c\u306f\u73fe\u4ee3\u7684\u65e5\u672c\u8a9e\u3067\u3059\u3002\n".encode('utf-8'), + # TODO RUSTPYTHON + # Uncomment when Shift JIS is supported # This is obsolete Japanese. (Shift JIS) - # "\u3053\u308c\u306f\u53e4\u3044\u65e5\u672c\u8a9e\u3067\u3059\u3002\n".encode('shift_jis'), # TODO: RUSTPYTHON + # "\u3053\u308c\u306f\u53e4\u3044\u65e5\u672c\u8a9e\u3067\u3059\u3002\n".encode('shift_jis'), ] def setUp(self): @@ -3418,11 +3484,13 @@ def _test_read(self, zipfp, expected_names, expected_content): self.assertEqual(info.file_size, len(content)) self.assertEqual(zipfp.read(name), content) + @unittest.expectedFailure # TODO: RUSTPYTHON; def test_read_with_metadata_encoding(self): # Read the ZIP archive with correct metadata_encoding with zipfile.ZipFile(TESTFN, "r", metadata_encoding='shift_jis') as zipfp: self._test_read(zipfp, self.file_names, self.file_content) + @unittest.expectedFailure # TODO: RUSTPYTHON; def test_read_without_metadata_encoding(self): # Read the ZIP archive without metadata_encoding expected_names = [name.encode('shift_jis').decode('cp437') @@ -3430,6 +3498,7 @@ def test_read_without_metadata_encoding(self): with zipfile.ZipFile(TESTFN, "r") as zipfp: self._test_read(zipfp, expected_names, self.file_content) + @unittest.expectedFailure # TODO: RUSTPYTHON; def test_read_with_incorrect_metadata_encoding(self): # Read the ZIP archive with incorrect metadata_encoding expected_names = [name.encode('shift_jis').decode('koi8-u') @@ -3437,6 +3506,7 @@ def test_read_with_incorrect_metadata_encoding(self): with zipfile.ZipFile(TESTFN, "r", metadata_encoding='koi8-u') as zipfp: self._test_read(zipfp, expected_names, self.file_content) + @unittest.expectedFailure # TODO: RUSTPYTHON; def test_read_with_unsuitable_metadata_encoding(self): # Read the ZIP archive with metadata_encoding unsuitable for # decoding metadata @@ -3445,6 +3515,7 @@ def test_read_with_unsuitable_metadata_encoding(self): with self.assertRaises(UnicodeDecodeError): zipfile.ZipFile(TESTFN, "r", metadata_encoding='utf-8') + @unittest.expectedFailure # TODO: RUSTPYTHON; def test_read_after_append(self): newname = '\u56db' # Han 'four' expected_names = [name.encode('shift_jis').decode('cp437') @@ -3471,6 +3542,7 @@ def test_read_after_append(self): else: self.assertEqual(zipfp.read(name), content) + @unittest.expectedFailure # TODO: RUSTPYTHON; def test_write_with_metadata_encoding(self): ZF = zipfile.ZipFile for mode in ("w", "x", "a"): @@ -3478,6 +3550,7 @@ def test_write_with_metadata_encoding(self): "^metadata_encoding is only"): ZF("nonesuch.zip", mode, metadata_encoding="shift_jis") + @unittest.expectedFailure # TODO: RUSTPYTHON; def test_cli_with_metadata_encoding(self): errmsg = "Non-conforming encodings not supported with -c." args = ["--metadata-encoding=shift_jis", "-c", "nonesuch", "nonesuch"] @@ -3497,6 +3570,7 @@ def test_cli_with_metadata_encoding(self): for name in self.file_names: self.assertIn(name, listing) + @unittest.expectedFailure # TODO: RUSTPYTHON; def test_cli_with_metadata_encoding_extract(self): os.mkdir(TESTFN2) self.addCleanup(rmtree, TESTFN2) diff --git a/Lib/test/test_zipimport.py b/Lib/test/test_zipimport.py index b9c1fd6134e..d448e3df5d7 100644 --- a/Lib/test/test_zipimport.py +++ b/Lib/test/test_zipimport.py @@ -1,8 +1,10 @@ import sys import os import marshal +import glob import importlib import importlib.util +import re import struct import time import unittest @@ -50,10 +52,14 @@ def module_path_to_dotted_name(path): TESTMOD = "ziptestmodule" +TESTMOD2 = "ziptestmodule2" +TESTMOD3 = "ziptestmodule3" TESTPACK = "ziptestpackage" TESTPACK2 = "ziptestpackage2" +TESTPACK3 = "ziptestpackage3" TEMP_DIR = os.path.abspath("junk95142") TEMP_ZIP = os.path.abspath("junk95142.zip") +TEST_DATA_DIR = os.path.join(os.path.dirname(__file__), "zipimport_data") pyc_file = importlib.util.cache_from_source(TESTMOD + '.py') pyc_ext = '.pyc' @@ -92,8 +98,10 @@ def makeTree(self, files, dirName=TEMP_DIR): # defined by files under the directory dirName. self.addCleanup(os_helper.rmtree, dirName) - for name, (mtime, data) in files.items(): - path = os.path.join(dirName, name) + for name, data in files.items(): + if isinstance(data, tuple): + mtime, data = data + path = os.path.join(dirName, *name.split('/')) if path[-1] == os.sep: if not os.path.isdir(path): os.makedirs(path) @@ -104,22 +112,18 @@ def makeTree(self, files, dirName=TEMP_DIR): with open(path, 'wb') as fp: fp.write(data) - def makeZip(self, files, zipName=TEMP_ZIP, **kw): + def makeZip(self, files, zipName=TEMP_ZIP, *, + comment=None, file_comment=None, stuff=None, prefix='', **kw): # Create a zip archive based set of modules/packages - # defined by files in the zip file zipName. If the - # key 'stuff' exists in kw it is prepended to the archive. + # defined by files in the zip file zipName. + # If stuff is not None, it is prepended to the archive. self.addCleanup(os_helper.unlink, zipName) - with ZipFile(zipName, "w") as z: - for name, (mtime, data) in files.items(): - zinfo = ZipInfo(name, time.localtime(mtime)) - zinfo.compress_type = self.compression - z.writestr(zinfo, data) - comment = kw.get("comment", None) + with ZipFile(zipName, "w", compression=self.compression) as z: + self.writeZip(z, files, file_comment=file_comment, prefix=prefix) if comment is not None: z.comment = comment - stuff = kw.get("stuff", None) if stuff is not None: # Prepend 'stuff' to the start of the zipfile with open(zipName, "rb") as f: @@ -128,20 +132,47 @@ def makeZip(self, files, zipName=TEMP_ZIP, **kw): f.write(stuff) f.write(data) + def writeZip(self, z, files, *, file_comment=None, prefix=''): + for name, data in files.items(): + if isinstance(data, tuple): + mtime, data = data + else: + mtime = NOW + name = name.replace(os.sep, '/') + zinfo = ZipInfo(prefix + name, time.localtime(mtime)) + zinfo.compress_type = self.compression + if file_comment is not None: + zinfo.comment = file_comment + if data is None: + zinfo.CRC = 0 + z.mkdir(zinfo) + else: + assert name[-1] != '/' + z.writestr(zinfo, data) + + def getZip64Files(self): + # This is the simplest way to make zipfile generate the zip64 EOCD block + return {f"f{n}.py": test_src for n in range(65537)} + def doTest(self, expected_ext, files, *modules, **kw): + if 'prefix' not in kw: + kw['prefix'] = 'pre/fix/' self.makeZip(files, **kw) + self.doTestWithPreBuiltZip(expected_ext, *modules, **kw) - sys.path.insert(0, TEMP_ZIP) + def doTestWithPreBuiltZip(self, expected_ext, *modules, + call=None, prefix='', **kw): + zip_path = os.path.join(TEMP_ZIP, *prefix.split('/')[:-1]) + sys.path.insert(0, zip_path) mod = importlib.import_module(".".join(modules)) - call = kw.get('call') if call is not None: call(mod) if expected_ext: file = mod.get_file() - self.assertEqual(file, os.path.join(TEMP_ZIP, + self.assertEqual(file, os.path.join(zip_path, *modules) + expected_ext) def testAFakeZlib(self): @@ -155,7 +186,8 @@ def testAFakeZlib(self): # zlib.decompress function object, after which the problem being # tested here wouldn't be a problem anymore... # (Hence the 'A' in the test method name: to make it the first - # item in a list sorted by name, like unittest.makeSuite() does.) + # item in a list sorted by name, like + # unittest.TestLoader.getTestCaseNames() does.) # # This test fails on platforms on which the zlib module is # statically linked, but the problem it tests for can't @@ -166,7 +198,7 @@ def testAFakeZlib(self): self.skipTest('zlib is a builtin module') if "zlib" in sys.modules: del sys.modules["zlib"] - files = {"zlib.py": (NOW, test_src)} + files = {"zlib.py": test_src} try: self.doTest(".py", files, "zlib") except ImportError: @@ -177,16 +209,16 @@ def testAFakeZlib(self): self.fail("expected test to raise ImportError") def testPy(self): - files = {TESTMOD + ".py": (NOW, test_src)} + files = {TESTMOD + ".py": test_src} self.doTest(".py", files, TESTMOD) def testPyc(self): - files = {TESTMOD + pyc_ext: (NOW, test_pyc)} + files = {TESTMOD + pyc_ext: test_pyc} self.doTest(pyc_ext, files, TESTMOD) def testBoth(self): - files = {TESTMOD + ".py": (NOW, test_src), - TESTMOD + pyc_ext: (NOW, test_pyc)} + files = {TESTMOD + ".py": test_src, + TESTMOD + pyc_ext: test_pyc} self.doTest(pyc_ext, files, TESTMOD) def testUncheckedHashBasedPyc(self): @@ -219,22 +251,22 @@ def check(mod): self.doTest(None, files, TESTMOD, call=check) def testEmptyPy(self): - files = {TESTMOD + ".py": (NOW, "")} + files = {TESTMOD + ".py": ""} self.doTest(None, files, TESTMOD) def testBadMagic(self): # make pyc magic word invalid, forcing loading from .py badmagic_pyc = bytearray(test_pyc) badmagic_pyc[0] ^= 0x04 # flip an arbitrary bit - files = {TESTMOD + ".py": (NOW, test_src), - TESTMOD + pyc_ext: (NOW, badmagic_pyc)} + files = {TESTMOD + ".py": test_src, + TESTMOD + pyc_ext: badmagic_pyc} self.doTest(".py", files, TESTMOD) def testBadMagic2(self): # make pyc magic word invalid, causing an ImportError badmagic_pyc = bytearray(test_pyc) badmagic_pyc[0] ^= 0x04 # flip an arbitrary bit - files = {TESTMOD + pyc_ext: (NOW, badmagic_pyc)} + files = {TESTMOD + pyc_ext: badmagic_pyc} try: self.doTest(".py", files, TESTMOD) self.fail("This should not be reached") @@ -247,22 +279,22 @@ def testBadMTime(self): # flip the second bit -- not the first as that one isn't stored in the # .py's mtime in the zip archive. badtime_pyc[11] ^= 0x02 - files = {TESTMOD + ".py": (NOW, test_src), - TESTMOD + pyc_ext: (NOW, badtime_pyc)} + files = {TESTMOD + ".py": test_src, + TESTMOD + pyc_ext: badtime_pyc} self.doTest(".py", files, TESTMOD) def test2038MTime(self): # Make sure we can handle mtimes larger than what a 32-bit signed number # can hold. twenty_thirty_eight_pyc = make_pyc(test_co, 2**32 - 1, len(test_src)) - files = {TESTMOD + ".py": (NOW, test_src), - TESTMOD + pyc_ext: (NOW, twenty_thirty_eight_pyc)} + files = {TESTMOD + ".py": test_src, + TESTMOD + pyc_ext: twenty_thirty_eight_pyc} self.doTest(".py", files, TESTMOD) def testPackage(self): packdir = TESTPACK + os.sep - files = {packdir + "__init__" + pyc_ext: (NOW, test_pyc), - packdir + TESTMOD + pyc_ext: (NOW, test_pyc)} + files = {packdir + "__init__" + pyc_ext: test_pyc, + packdir + TESTMOD + pyc_ext: test_pyc} self.doTest(pyc_ext, files, TESTPACK, TESTMOD) def testSubPackage(self): @@ -270,9 +302,9 @@ def testSubPackage(self): # archives. packdir = TESTPACK + os.sep packdir2 = packdir + TESTPACK2 + os.sep - files = {packdir + "__init__" + pyc_ext: (NOW, test_pyc), - packdir2 + "__init__" + pyc_ext: (NOW, test_pyc), - packdir2 + TESTMOD + pyc_ext: (NOW, test_pyc)} + files = {packdir + "__init__" + pyc_ext: test_pyc, + packdir2 + "__init__" + pyc_ext: test_pyc, + packdir2 + TESTMOD + pyc_ext: test_pyc} self.doTest(pyc_ext, files, TESTPACK, TESTPACK2, TESTMOD) def testSubNamespacePackage(self): @@ -281,29 +313,104 @@ def testSubNamespacePackage(self): packdir = TESTPACK + os.sep packdir2 = packdir + TESTPACK2 + os.sep # The first two files are just directory entries (so have no data). - files = {packdir: (NOW, ""), - packdir2: (NOW, ""), - packdir2 + TESTMOD + pyc_ext: (NOW, test_pyc)} + files = {packdir: None, + packdir2: None, + packdir2 + TESTMOD + pyc_ext: test_pyc} self.doTest(pyc_ext, files, TESTPACK, TESTPACK2, TESTMOD) + def testPackageExplicitDirectories(self): + # Test explicit namespace packages with explicit directory entries. + self.addCleanup(os_helper.unlink, TEMP_ZIP) + with ZipFile(TEMP_ZIP, 'w', compression=self.compression) as z: + z.mkdir('a') + z.writestr('a/__init__.py', test_src) + z.mkdir('a/b') + z.writestr('a/b/__init__.py', test_src) + z.mkdir('a/b/c') + z.writestr('a/b/c/__init__.py', test_src) + z.writestr('a/b/c/d.py', test_src) + self._testPackage(initfile='__init__.py') + + def testPackageImplicitDirectories(self): + # Test explicit namespace packages without explicit directory entries. + self.addCleanup(os_helper.unlink, TEMP_ZIP) + with ZipFile(TEMP_ZIP, 'w', compression=self.compression) as z: + z.writestr('a/__init__.py', test_src) + z.writestr('a/b/__init__.py', test_src) + z.writestr('a/b/c/__init__.py', test_src) + z.writestr('a/b/c/d.py', test_src) + self._testPackage(initfile='__init__.py') + + def testNamespacePackageExplicitDirectories(self): + # Test implicit namespace packages with explicit directory entries. + self.addCleanup(os_helper.unlink, TEMP_ZIP) + with ZipFile(TEMP_ZIP, 'w', compression=self.compression) as z: + z.mkdir('a') + z.mkdir('a/b') + z.mkdir('a/b/c') + z.writestr('a/b/c/d.py', test_src) + self._testPackage(initfile=None) + + def testNamespacePackageImplicitDirectories(self): + # Test implicit namespace packages without explicit directory entries. + self.addCleanup(os_helper.unlink, TEMP_ZIP) + with ZipFile(TEMP_ZIP, 'w', compression=self.compression) as z: + z.writestr('a/b/c/d.py', test_src) + self._testPackage(initfile=None) + + def _testPackage(self, initfile): + zi = zipimport.zipimporter(os.path.join(TEMP_ZIP, 'a')) + if initfile is None: + # XXX Should it work? + self.assertRaises(zipimport.ZipImportError, zi.is_package, 'b') + self.assertRaises(zipimport.ZipImportError, zi.get_source, 'b') + self.assertRaises(zipimport.ZipImportError, zi.get_code, 'b') + else: + self.assertTrue(zi.is_package('b')) + self.assertEqual(zi.get_source('b'), test_src) + self.assertEqual(zi.get_code('b').co_filename, + os.path.join(TEMP_ZIP, 'a', 'b', initfile)) + + sys.path.insert(0, TEMP_ZIP) + self.assertNotIn('a', sys.modules) + + mod = importlib.import_module(f'a.b') + self.assertIn('a', sys.modules) + self.assertIs(sys.modules['a.b'], mod) + if initfile is None: + self.assertIsNone(mod.__file__) + else: + self.assertEqual(mod.__file__, + os.path.join(TEMP_ZIP, 'a', 'b', initfile)) + self.assertEqual(len(mod.__path__), 1, mod.__path__) + self.assertEqual(mod.__path__[0], os.path.join(TEMP_ZIP, 'a', 'b')) + + mod2 = importlib.import_module(f'a.b.c.d') + self.assertIn('a.b.c', sys.modules) + self.assertIn('a.b.c.d', sys.modules) + self.assertIs(sys.modules['a.b.c.d'], mod2) + self.assertIs(mod.c.d, mod2) + self.assertEqual(mod2.__file__, + os.path.join(TEMP_ZIP, 'a', 'b', 'c', 'd.py')) + def testMixedNamespacePackage(self): # Test implicit namespace packages spread between a # real filesystem and a zip archive. packdir = TESTPACK + os.sep packdir2 = packdir + TESTPACK2 + os.sep - packdir3 = packdir2 + TESTPACK + '3' + os.sep - files1 = {packdir: (NOW, ""), - packdir + TESTMOD + pyc_ext: (NOW, test_pyc), - packdir2: (NOW, ""), - packdir3: (NOW, ""), - packdir3 + TESTMOD + pyc_ext: (NOW, test_pyc), - packdir2 + TESTMOD + '3' + pyc_ext: (NOW, test_pyc), - packdir2 + TESTMOD + pyc_ext: (NOW, test_pyc)} - files2 = {packdir: (NOW, ""), - packdir + TESTMOD + '2' + pyc_ext: (NOW, test_pyc), - packdir2: (NOW, ""), - packdir2 + TESTMOD + '2' + pyc_ext: (NOW, test_pyc), - packdir2 + TESTMOD + pyc_ext: (NOW, test_pyc)} + packdir3 = packdir2 + TESTPACK3 + os.sep + files1 = {packdir: None, + packdir + TESTMOD + pyc_ext: test_pyc, + packdir2: None, + packdir3: None, + packdir3 + TESTMOD + pyc_ext: test_pyc, + packdir2 + TESTMOD3 + pyc_ext: test_pyc, + packdir2 + TESTMOD + pyc_ext: test_pyc} + files2 = {packdir: None, + packdir + TESTMOD2 + pyc_ext: test_pyc, + packdir2: None, + packdir2 + TESTMOD2 + pyc_ext: test_pyc, + packdir2 + TESTMOD + pyc_ext: test_pyc} zip1 = os.path.abspath("path1.zip") self.makeZip(files1, zip1) @@ -336,8 +443,8 @@ def testMixedNamespacePackage(self): mod = importlib.import_module('.'.join((TESTPACK, TESTMOD))) self.assertEqual("path1.zip", mod.__file__.split(os.sep)[-3]) - # And TESTPACK/(TESTMOD + '2') only exists in path2. - mod = importlib.import_module('.'.join((TESTPACK, TESTMOD + '2'))) + # And TESTPACK/(TESTMOD2) only exists in path2. + mod = importlib.import_module('.'.join((TESTPACK, TESTMOD2))) self.assertEqual(os.path.basename(TEMP_DIR), mod.__file__.split(os.sep)[-3]) @@ -354,13 +461,13 @@ def testMixedNamespacePackage(self): self.assertEqual(os.path.basename(TEMP_DIR), mod.__file__.split(os.sep)[-4]) - # subpkg.TESTMOD + '2' only exists in zip2. - mod = importlib.import_module('.'.join((subpkg, TESTMOD + '2'))) + # subpkg.TESTMOD2 only exists in zip2. + mod = importlib.import_module('.'.join((subpkg, TESTMOD2))) self.assertEqual(os.path.basename(TEMP_DIR), mod.__file__.split(os.sep)[-4]) - # Finally subpkg.TESTMOD + '3' only exists in zip1. - mod = importlib.import_module('.'.join((subpkg, TESTMOD + '3'))) + # Finally subpkg.TESTMOD3 only exists in zip1. + mod = importlib.import_module('.'.join((subpkg, TESTMOD3))) self.assertEqual('path1.zip', mod.__file__.split(os.sep)[-4]) def testNamespacePackage(self): @@ -368,22 +475,22 @@ def testNamespacePackage(self): # archives. packdir = TESTPACK + os.sep packdir2 = packdir + TESTPACK2 + os.sep - packdir3 = packdir2 + TESTPACK + '3' + os.sep - files1 = {packdir: (NOW, ""), - packdir + TESTMOD + pyc_ext: (NOW, test_pyc), - packdir2: (NOW, ""), - packdir3: (NOW, ""), - packdir3 + TESTMOD + pyc_ext: (NOW, test_pyc), - packdir2 + TESTMOD + '3' + pyc_ext: (NOW, test_pyc), - packdir2 + TESTMOD + pyc_ext: (NOW, test_pyc)} + packdir3 = packdir2 + TESTPACK3 + os.sep + files1 = {packdir: None, + packdir + TESTMOD + pyc_ext: test_pyc, + packdir2: None, + packdir3: None, + packdir3 + TESTMOD + pyc_ext: test_pyc, + packdir2 + TESTMOD3 + pyc_ext: test_pyc, + packdir2 + TESTMOD + pyc_ext: test_pyc} zip1 = os.path.abspath("path1.zip") self.makeZip(files1, zip1) - files2 = {packdir: (NOW, ""), - packdir + TESTMOD + '2' + pyc_ext: (NOW, test_pyc), - packdir2: (NOW, ""), - packdir2 + TESTMOD + '2' + pyc_ext: (NOW, test_pyc), - packdir2 + TESTMOD + pyc_ext: (NOW, test_pyc)} + files2 = {packdir: None, + packdir + TESTMOD2 + pyc_ext: test_pyc, + packdir2: None, + packdir2 + TESTMOD2 + pyc_ext: test_pyc, + packdir2 + TESTMOD + pyc_ext: test_pyc} zip2 = os.path.abspath("path2.zip") self.makeZip(files2, zip2) @@ -412,8 +519,8 @@ def testNamespacePackage(self): mod = importlib.import_module('.'.join((TESTPACK, TESTMOD))) self.assertEqual("path1.zip", mod.__file__.split(os.sep)[-3]) - # And TESTPACK/(TESTMOD + '2') only exists in path2. - mod = importlib.import_module('.'.join((TESTPACK, TESTMOD + '2'))) + # And TESTPACK/(TESTMOD2) only exists in path2. + mod = importlib.import_module('.'.join((TESTPACK, TESTMOD2))) self.assertEqual("path2.zip", mod.__file__.split(os.sep)[-3]) # One level deeper... @@ -428,29 +535,22 @@ def testNamespacePackage(self): mod = importlib.import_module('.'.join((subpkg, TESTMOD))) self.assertEqual('path2.zip', mod.__file__.split(os.sep)[-4]) - # subpkg.TESTMOD + '2' only exists in zip2. - mod = importlib.import_module('.'.join((subpkg, TESTMOD + '2'))) + # subpkg.TESTMOD2 only exists in zip2. + mod = importlib.import_module('.'.join((subpkg, TESTMOD2))) self.assertEqual('path2.zip', mod.__file__.split(os.sep)[-4]) - # Finally subpkg.TESTMOD + '3' only exists in zip1. - mod = importlib.import_module('.'.join((subpkg, TESTMOD + '3'))) + # Finally subpkg.TESTMOD3 only exists in zip1. + mod = importlib.import_module('.'.join((subpkg, TESTMOD3))) self.assertEqual('path1.zip', mod.__file__.split(os.sep)[-4]) def testZipImporterMethods(self): packdir = TESTPACK + os.sep packdir2 = packdir + TESTPACK2 + os.sep - files = {packdir + "__init__" + pyc_ext: (NOW, test_pyc), - packdir2 + "__init__" + pyc_ext: (NOW, test_pyc), - packdir2 + TESTMOD + pyc_ext: (NOW, test_pyc), - "spam" + pyc_ext: (NOW, test_pyc)} - - self.addCleanup(os_helper.unlink, TEMP_ZIP) - with ZipFile(TEMP_ZIP, "w") as z: - for name, (mtime, data) in files.items(): - zinfo = ZipInfo(name, time.localtime(mtime)) - zinfo.compress_type = self.compression - zinfo.comment = b"spam" - z.writestr(zinfo, data) + files = {packdir + "__init__" + pyc_ext: test_pyc, + packdir2 + "__init__" + pyc_ext: test_pyc, + packdir2 + TESTMOD + pyc_ext: test_pyc, + "spam" + pyc_ext: test_pyc} + self.makeZip(files, file_comment=b"spam") zi = zipimport.zipimporter(TEMP_ZIP) self.assertEqual(zi.archive, TEMP_ZIP) @@ -459,12 +559,6 @@ def testZipImporterMethods(self): # PEP 302 with warnings.catch_warnings(): warnings.simplefilter("ignore", DeprecationWarning) - find_mod = zi.find_module('spam') - self.assertIsNotNone(find_mod) - self.assertIsInstance(find_mod, zipimport.zipimporter) - self.assertFalse(find_mod.is_package('spam')) - load_mod = find_mod.load_module('spam') - self.assertEqual(find_mod.get_filename('spam'), load_mod.__file__) mod = zi.load_module(TESTPACK) self.assertEqual(zi.get_filename(TESTPACK), mod.__file__) @@ -512,58 +606,70 @@ def testZipImporterMethods(self): def testInvalidateCaches(self): packdir = TESTPACK + os.sep packdir2 = packdir + TESTPACK2 + os.sep - files = {packdir + "__init__" + pyc_ext: (NOW, test_pyc), - packdir2 + "__init__" + pyc_ext: (NOW, test_pyc), - packdir2 + TESTMOD + pyc_ext: (NOW, test_pyc), - "spam" + pyc_ext: (NOW, test_pyc)} - self.addCleanup(os_helper.unlink, TEMP_ZIP) - with ZipFile(TEMP_ZIP, "w") as z: - for name, (mtime, data) in files.items(): - zinfo = ZipInfo(name, time.localtime(mtime)) - zinfo.compress_type = self.compression - zinfo.comment = b"spam" - z.writestr(zinfo, data) + files = {packdir + "__init__" + pyc_ext: test_pyc, + packdir2 + "__init__" + pyc_ext: test_pyc, + packdir2 + TESTMOD + pyc_ext: test_pyc, + "spam" + pyc_ext: test_pyc} + extra_files = [packdir, packdir2] + self.makeZip(files, file_comment=b"spam") zi = zipimport.zipimporter(TEMP_ZIP) - self.assertEqual(zi._files.keys(), files.keys()) + self.assertEqual(sorted(zi._get_files()), sorted([*files, *extra_files])) # Check that the file information remains accurate after reloading zi.invalidate_caches() - self.assertEqual(zi._files.keys(), files.keys()) + self.assertEqual(sorted(zi._get_files()), sorted([*files, *extra_files])) # Add a new file to the ZIP archive - newfile = {"spam2" + pyc_ext: (NOW, test_pyc)} + newfile = {"spam2" + pyc_ext: test_pyc} files.update(newfile) - with ZipFile(TEMP_ZIP, "a") as z: - for name, (mtime, data) in newfile.items(): - zinfo = ZipInfo(name, time.localtime(mtime)) - zinfo.compress_type = self.compression - zinfo.comment = b"spam" - z.writestr(zinfo, data) + with ZipFile(TEMP_ZIP, "a", compression=self.compression) as z: + self.writeZip(z, newfile, file_comment=b"spam") # Check that we can detect the new file after invalidating the cache zi.invalidate_caches() - self.assertEqual(zi._files.keys(), files.keys()) + self.assertEqual(sorted(zi._get_files()), sorted([*files, *extra_files])) spec = zi.find_spec('spam2') self.assertIsNotNone(spec) self.assertIsInstance(spec.loader, zipimport.zipimporter) # Check that the cached data is removed if the file is deleted os.remove(TEMP_ZIP) zi.invalidate_caches() - self.assertFalse(zi._files) + self.assertFalse(zi._get_files()) self.assertIsNone(zipimport._zip_directory_cache.get(zi.archive)) self.assertIsNone(zi.find_spec("name_does_not_matter")) - def testZipImporterMethodsInSubDirectory(self): + def testInvalidateCachesWithMultipleZipimports(self): packdir = TESTPACK + os.sep packdir2 = packdir + TESTPACK2 + os.sep - files = {packdir2 + "__init__" + pyc_ext: (NOW, test_pyc), - packdir2 + TESTMOD + pyc_ext: (NOW, test_pyc)} + files = {packdir + "__init__" + pyc_ext: test_pyc, + packdir2 + "__init__" + pyc_ext: test_pyc, + packdir2 + TESTMOD + pyc_ext: test_pyc, + "spam" + pyc_ext: test_pyc} + extra_files = [packdir, packdir2] + self.makeZip(files, file_comment=b"spam") - self.addCleanup(os_helper.unlink, TEMP_ZIP) - with ZipFile(TEMP_ZIP, "w") as z: - for name, (mtime, data) in files.items(): - zinfo = ZipInfo(name, time.localtime(mtime)) - zinfo.compress_type = self.compression - zinfo.comment = b"eggs" - z.writestr(zinfo, data) + zi = zipimport.zipimporter(TEMP_ZIP) + self.assertEqual(sorted(zi._get_files()), sorted([*files, *extra_files])) + # Zipimporter for the same path. + zi2 = zipimport.zipimporter(TEMP_ZIP) + self.assertEqual(sorted(zi2._get_files()), sorted([*files, *extra_files])) + # Add a new file to the ZIP archive to make the cache wrong. + newfile = {"spam2" + pyc_ext: test_pyc} + files.update(newfile) + with ZipFile(TEMP_ZIP, "a", compression=self.compression) as z: + self.writeZip(z, newfile, file_comment=b"spam") + # Invalidate the cache of the first zipimporter. + zi.invalidate_caches() + # Check that the second zipimporter detects the new file and isn't using a stale cache. + self.assertEqual(sorted(zi2._get_files()), sorted([*files, *extra_files])) + spec = zi2.find_spec('spam2') + self.assertIsNotNone(spec) + self.assertIsInstance(spec.loader, zipimport.zipimporter) + + def testZipImporterMethodsInSubDirectory(self): + packdir = TESTPACK + os.sep + packdir2 = packdir + TESTPACK2 + os.sep + files = {packdir2 + "__init__" + pyc_ext: test_pyc, + packdir2 + TESTMOD + pyc_ext: test_pyc} + self.makeZip(files, file_comment=b"eggs") zi = zipimport.zipimporter(TEMP_ZIP + os.sep + packdir) self.assertEqual(zi.archive, TEMP_ZIP) @@ -585,16 +691,6 @@ def testZipImporterMethodsInSubDirectory(self): pkg_path = TEMP_ZIP + os.sep + packdir + TESTPACK2 zi2 = zipimport.zipimporter(pkg_path) - # PEP 302 - with warnings.catch_warnings(): - warnings.simplefilter("ignore", DeprecationWarning) - find_mod_dotted = zi2.find_module(TESTMOD) - self.assertIsNotNone(find_mod_dotted) - self.assertIsInstance(find_mod_dotted, zipimport.zipimporter) - self.assertFalse(zi2.is_package(TESTMOD)) - load_mod = find_mod_dotted.load_module(TESTMOD) - self.assertEqual( - find_mod_dotted.get_filename(TESTMOD), load_mod.__file__) # PEP 451 spec = zi2.find_spec(TESTMOD) @@ -619,17 +715,33 @@ def testZipImporterMethodsInSubDirectory(self): self.assertIsNone(loader.get_source(mod_name)) self.assertEqual(loader.get_filename(mod_name), mod.__file__) - def testGetData(self): + def testGetDataExplicitDirectories(self): self.addCleanup(os_helper.unlink, TEMP_ZIP) - with ZipFile(TEMP_ZIP, "w") as z: - z.compression = self.compression - name = "testdata.dat" - data = bytes(x for x in range(256)) - z.writestr(name, data) - - zi = zipimport.zipimporter(TEMP_ZIP) - self.assertEqual(data, zi.get_data(name)) - self.assertIn('zipimporter object', repr(zi)) + with ZipFile(TEMP_ZIP, 'w', compression=self.compression) as z: + z.mkdir('a') + z.mkdir('a/b') + z.mkdir('a/b/c') + data = bytes(range(256)) + z.writestr('a/b/c/testdata.dat', data) + self._testGetData() + + def testGetDataImplicitDirectories(self): + self.addCleanup(os_helper.unlink, TEMP_ZIP) + with ZipFile(TEMP_ZIP, 'w', compression=self.compression) as z: + data = bytes(range(256)) + z.writestr('a/b/c/testdata.dat', data) + self._testGetData() + + def _testGetData(self): + zi = zipimport.zipimporter(os.path.join(TEMP_ZIP, 'ignored')) + pathname = os.path.join('a', 'b', 'c', 'testdata.dat') + data = bytes(range(256)) + self.assertEqual(zi.get_data(pathname), data) + self.assertEqual(zi.get_data(os.path.join(TEMP_ZIP, pathname)), data) + self.assertEqual(zi.get_data(os.path.join('a', 'b', '')), b'') + self.assertEqual(zi.get_data(os.path.join(TEMP_ZIP, 'a', 'b', '')), b'') + self.assertRaises(OSError, zi.get_data, os.path.join('a', 'b')) + self.assertRaises(OSError, zi.get_data, os.path.join(TEMP_ZIP, 'a', 'b')) def testImporterAttr(self): src = """if 1: # indent hack @@ -638,9 +750,9 @@ def get_file(): if __loader__.get_data("some.data") != b"some data": raise AssertionError("bad data")\n""" pyc = make_pyc(compile(src, "<???>", "exec"), NOW, len(src)) - files = {TESTMOD + pyc_ext: (NOW, pyc), - "some.data": (NOW, "some data")} - self.doTest(pyc_ext, files, TESTMOD) + files = {TESTMOD + pyc_ext: pyc, + "some.data": "some data"} + self.doTest(pyc_ext, files, TESTMOD, prefix='') def testDefaultOptimizationLevel(self): # zipimport should use the default optimization level (#28131) @@ -648,17 +760,20 @@ def testDefaultOptimizationLevel(self): def test(val): assert(val) return val\n""" - files = {TESTMOD + '.py': (NOW, src)} + files = {TESTMOD + '.py': src} self.makeZip(files) sys.path.insert(0, TEMP_ZIP) mod = importlib.import_module(TESTMOD) self.assertEqual(mod.test(1), 1) - self.assertRaises(AssertionError, mod.test, False) + if __debug__: + self.assertRaises(AssertionError, mod.test, False) + else: + self.assertEqual(mod.test(0), 0) def testImport_WithStuff(self): # try importing from a zipfile which contains additional # stuff at the beginning of the file - files = {TESTMOD + ".py": (NOW, test_src)} + files = {TESTMOD + ".py": test_src} self.doTest(".py", files, TESTMOD, stuff=b"Some Stuff"*31) @@ -666,18 +781,18 @@ def assertModuleSource(self, module): self.assertEqual(inspect.getsource(module), test_src) def testGetSource(self): - files = {TESTMOD + ".py": (NOW, test_src)} + files = {TESTMOD + ".py": test_src} self.doTest(".py", files, TESTMOD, call=self.assertModuleSource) def testGetCompiledSource(self): pyc = make_pyc(compile(test_src, "<???>", "exec"), NOW, len(test_src)) - files = {TESTMOD + ".py": (NOW, test_src), - TESTMOD + pyc_ext: (NOW, pyc)} + files = {TESTMOD + ".py": test_src, + TESTMOD + pyc_ext: pyc} self.doTest(pyc_ext, files, TESTMOD, call=self.assertModuleSource) def runDoctest(self, callback): - files = {TESTMOD + ".py": (NOW, test_src), - "xyz.txt": (NOW, ">>> log.append(True)\n")} + files = {TESTMOD + ".py": test_src, + "xyz.txt": ">>> log.append(True)\n"} self.doTest(".py", files, TESTMOD, call=callback) def doDoctestFile(self, module): @@ -720,56 +835,179 @@ def doTraceback(self, module): s = io.StringIO() print_tb(tb, 1, s) - self.assertTrue(s.getvalue().endswith(raise_src)) + self.assertEndsWith(s.getvalue(), + ' def do_raise(): raise TypeError\n' + '' if support.has_no_debug_ranges() else + ' ^^^^^^^^^^^^^^^\n' + ) else: raise AssertionError("This ought to be impossible") - # TODO: RUSTPYTHON; empty caret lines from equal col/end_col - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; empty caret lines from equal col/end_col def testTraceback(self): - files = {TESTMOD + ".py": (NOW, raise_src)} + files = {TESTMOD + ".py": raise_src} self.doTest(None, files, TESTMOD, call=self.doTraceback) @unittest.skipIf(os_helper.TESTFN_UNENCODABLE is None, "need an unencodable filename") def testUnencodable(self): filename = os_helper.TESTFN_UNENCODABLE + ".zip" - self.addCleanup(os_helper.unlink, filename) - with ZipFile(filename, "w") as z: - zinfo = ZipInfo(TESTMOD + ".py", time.localtime(NOW)) - zinfo.compress_type = self.compression - z.writestr(zinfo, test_src) + self.makeZip({TESTMOD + ".py": test_src}, filename) spec = zipimport.zipimporter(filename).find_spec(TESTMOD) mod = importlib.util.module_from_spec(spec) spec.loader.exec_module(mod) def testBytesPath(self): filename = os_helper.TESTFN + ".zip" - self.addCleanup(os_helper.unlink, filename) - with ZipFile(filename, "w") as z: - zinfo = ZipInfo(TESTMOD + ".py", time.localtime(NOW)) - zinfo.compress_type = self.compression - z.writestr(zinfo, test_src) + self.makeZip({TESTMOD + ".py": test_src}, filename) zipimport.zipimporter(filename) - zipimport.zipimporter(os.fsencode(filename)) + with self.assertRaises(TypeError): + zipimport.zipimporter(os.fsencode(filename)) with self.assertRaises(TypeError): zipimport.zipimporter(bytearray(os.fsencode(filename))) with self.assertRaises(TypeError): zipimport.zipimporter(memoryview(os.fsencode(filename))) def testComment(self): - files = {TESTMOD + ".py": (NOW, test_src)} + files = {TESTMOD + ".py": test_src} self.doTest(".py", files, TESTMOD, comment=b"comment") def testBeginningCruftAndComment(self): - files = {TESTMOD + ".py": (NOW, test_src)} + files = {TESTMOD + ".py": test_src} self.doTest(".py", files, TESTMOD, stuff=b"cruft" * 64, comment=b"hi") def testLargestPossibleComment(self): - files = {TESTMOD + ".py": (NOW, test_src)} + files = {TESTMOD + ".py": test_src} self.doTest(".py", files, TESTMOD, comment=b"c" * ((1 << 16) - 1)) + @support.requires_resource('cpu') + def testZip64(self): + files = self.getZip64Files() + self.doTest(".py", files, "f6") + + @support.requires_resource('cpu') + def testZip64CruftAndComment(self): + files = self.getZip64Files() + self.doTest(".py", files, "f65536", comment=b"c" * ((1 << 16) - 1)) + + @unittest.skip("TODO: RUSTPYTHON; (intermittent success/failures); ValueError: name=\"RustPython/crates/pylib/Lib/test/zipimport_data/sparse-zip64-c0-0x000000000.part\" does not fit expected pattern.") + def testZip64LargeFile(self): + support.requires( + "largefile", + f"test generates files >{0xFFFFFFFF} bytes and takes a long time " + "to run" + ) + + # N.B.: We do a lot of gymnastics below in the ZIP_STORED case to save + # and reconstruct a sparse zip on systems that support sparse files. + # Instead of creating a ~8GB zip file mainly consisting of null bytes + # for every run of the test, we create the zip once and save off the + # non-null portions of the resulting file as data blobs with offsets + # that allow re-creating the zip file sparsely. This drops disk space + # usage to ~9KB for the ZIP_STORED case and drops that test time by ~2 + # orders of magnitude. For the ZIP_DEFLATED case, however, we bite the + # bullet. The resulting zip file is ~8MB of non-null data; so the sparse + # trick doesn't work and would result in that full ~8MB zip data file + # being checked in to source control. + parts_glob = f"sparse-zip64-c{self.compression:d}-0x*.part" + full_parts_glob = os.path.join(TEST_DATA_DIR, parts_glob) + pre_built_zip_parts = glob.glob(full_parts_glob) + + self.addCleanup(os_helper.unlink, TEMP_ZIP) + if not pre_built_zip_parts: + if self.compression != ZIP_STORED: + support.requires( + "cpu", + "test requires a lot of CPU for compression." + ) + self.addCleanup(os_helper.unlink, os_helper.TESTFN) + with open(os_helper.TESTFN, "wb") as f: + f.write(b"data") + f.write(os.linesep.encode()) + f.seek(0xffff_ffff, os.SEEK_CUR) + f.write(os.linesep.encode()) + os.utime(os_helper.TESTFN, (0.0, 0.0)) + with ZipFile( + TEMP_ZIP, + "w", + compression=self.compression, + strict_timestamps=False + ) as z: + z.write(os_helper.TESTFN, "data1") + z.writestr( + ZipInfo("module.py", (1980, 1, 1, 0, 0, 0)), test_src + ) + z.write(os_helper.TESTFN, "data2") + + # This "works" but relies on the zip format having a non-empty + # final page due to the trailing central directory to wind up with + # the correct length file. + def make_sparse_zip_parts(name): + empty_page = b"\0" * 4096 + with open(name, "rb") as f: + part = None + try: + while True: + offset = f.tell() + data = f.read(len(empty_page)) + if not data: + break + if data != empty_page: + if not part: + part_fullname = os.path.join( + TEST_DATA_DIR, + f"sparse-zip64-c{self.compression:d}-" + f"{offset:#011x}.part", + ) + os.makedirs( + os.path.dirname(part_fullname), + exist_ok=True + ) + part = open(part_fullname, "wb") + print("Created", part_fullname) + part.write(data) + else: + if part: + part.close() + part = None + finally: + if part: + part.close() + + if self.compression == ZIP_STORED: + print(f"Creating sparse parts to check in into {TEST_DATA_DIR}:") + make_sparse_zip_parts(TEMP_ZIP) + + else: + def extract_offset(name): + if m := re.search(r"-(0x[0-9a-f]{9})\.part$", name): + return int(m.group(1), base=16) + raise ValueError(f"{name=} does not fit expected pattern.") + offset_parts = [(extract_offset(n), n) for n in pre_built_zip_parts] + with open(TEMP_ZIP, "wb") as f: + for offset, part_fn in sorted(offset_parts): + with open(part_fn, "rb") as part: + f.seek(offset, os.SEEK_SET) + f.write(part.read()) + # Confirm that the reconstructed zip file works and looks right. + with ZipFile(TEMP_ZIP, "r") as z: + self.assertEqual( + z.getinfo("module.py").date_time, (1980, 1, 1, 0, 0, 0) + ) + self.assertEqual( + z.read("module.py"), test_src.encode(), + msg=f"Recreate {full_parts_glob}, unexpected contents." + ) + def assertDataEntry(name): + zinfo = z.getinfo(name) + self.assertEqual(zinfo.date_time, (1980, 1, 1, 0, 0, 0)) + self.assertGreater(zinfo.file_size, 0xffff_ffff) + assertDataEntry("data1") + assertDataEntry("data2") + + self.doTestWithPreBuiltZip(".py", "module") + @support.requires_zlib() class CompressedZipImportTestCase(UncompressedZipImportTestCase): @@ -801,6 +1039,7 @@ def testEmptyFile(self): os_helper.create_empty_file(TESTMOD) self.assertZipFailure(TESTMOD) + @unittest.skipIf(support.is_wasi, "mode 000 not supported.") def testFileUnreadable(self): os_helper.unlink(TESTMOD) fd = os.open(TESTMOD, os.O_CREAT, 000) @@ -844,7 +1083,6 @@ def _testBogusZipFile(self): self.assertRaises(TypeError, z.get_source, None) error = zipimport.ZipImportError - self.assertIsNone(z.find_module('abc')) self.assertIsNone(z.find_spec('abc')) with warnings.catch_warnings(): diff --git a/Lib/test/test_zlib.py b/Lib/test/test_zlib.py index 0a75457ad85..bb1366cb21c 100644 --- a/Lib/test/test_zlib.py +++ b/Lib/test/test_zlib.py @@ -3,7 +3,6 @@ from test.support import import_helper import binascii import copy -import os import pickle import random import sys @@ -13,11 +12,11 @@ zlib = import_helper.import_module('zlib') requires_Compress_copy = unittest.skipUnless( - hasattr(zlib.compressobj(), "copy"), - 'requires Compress.copy()') + hasattr(zlib.compressobj(), "copy"), + 'requires Compress.copy()') requires_Decompress_copy = unittest.skipUnless( - hasattr(zlib.decompressobj(), "copy"), - 'requires Decompress.copy()') + hasattr(zlib.decompressobj(), "copy"), + 'requires Decompress.copy()') def _zlib_runtime_version_tuple(zlib_version=zlib.ZLIB_RUNTIME_VERSION): @@ -154,7 +153,7 @@ def test_badcompressobj(self): self.assertRaises(ValueError, zlib.compressobj, 1, zlib.DEFLATED, 0) # specifying total bits too large causes an error self.assertRaises(ValueError, - zlib.compressobj, 1, zlib.DEFLATED, zlib.MAX_WBITS + 1) + zlib.compressobj, 1, zlib.DEFLATED, zlib.MAX_WBITS + 1) def test_baddecompressobj(self): # verify failure on building decompress object with bad params @@ -242,8 +241,8 @@ def test_incomplete_stream(self): # A useful error message is given x = zlib.compress(HAMLET_SCENE) self.assertRaisesRegex(zlib.error, - "Error -5 while decompressing data: incomplete or truncated stream", - zlib.decompress, x[:-1]) + "Error -5 while decompressing data: incomplete or truncated stream", + zlib.decompress, x[:-1]) # Memory use of the following functions takes into account overallocation @@ -377,7 +376,7 @@ def test_decompinc(self, flush=False, source=None, cx=256, dcx=64): bufs.append(dco.decompress(combuf[i:i+dcx])) self.assertEqual(b'', dco.unconsumed_tail, ######## "(A) uct should be b'': not %d long" % - len(dco.unconsumed_tail)) + len(dco.unconsumed_tail)) self.assertEqual(b'', dco.unused_data) if flush: bufs.append(dco.flush()) @@ -390,7 +389,7 @@ def test_decompinc(self, flush=False, source=None, cx=256, dcx=64): break self.assertEqual(b'', dco.unconsumed_tail, ######## "(B) uct should be b'': not %d long" % - len(dco.unconsumed_tail)) + len(dco.unconsumed_tail)) self.assertEqual(b'', dco.unused_data) self.assertEqual(data, b''.join(bufs)) # Failure means: "decompressobj with init options failed" @@ -419,7 +418,7 @@ def test_decompimax(self, source=None, cx=256, dcx=64): #max_length = 1 + len(cb)//10 chunk = dco.decompress(cb, dcx) self.assertFalse(len(chunk) > dcx, - 'chunk too big (%d>%d)' % (len(chunk), dcx)) + 'chunk too big (%d>%d)' % (len(chunk), dcx)) bufs.append(chunk) cb = dco.unconsumed_tail bufs.append(dco.flush()) @@ -444,7 +443,7 @@ def test_decompressmaxlen(self, flush=False): max_length = 1 + len(cb)//10 chunk = dco.decompress(cb, max_length) self.assertFalse(len(chunk) > max_length, - 'chunk too big (%d>%d)' % (len(chunk),max_length)) + 'chunk too big (%d>%d)' % (len(chunk),max_length)) bufs.append(chunk) cb = dco.unconsumed_tail if flush: @@ -453,7 +452,7 @@ def test_decompressmaxlen(self, flush=False): while chunk: chunk = dco.decompress(b'', max_length) self.assertFalse(len(chunk) > max_length, - 'chunk too big (%d>%d)' % (len(chunk),max_length)) + 'chunk too big (%d>%d)' % (len(chunk),max_length)) bufs.append(chunk) self.assertEqual(data, b''.join(bufs), 'Wrong data retrieved') @@ -490,8 +489,7 @@ def test_clear_unconsumed_tail(self): ddata += dco.decompress(dco.unconsumed_tail) self.assertEqual(dco.unconsumed_tail, b"") - # TODO: RUSTPYTHON: Z_BLOCK support in flate2 - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; Z_BLOCK support in flate2 def test_flushes(self): # Test flush() with the various options, using all the # different levels in order to provide more variations. @@ -633,7 +631,7 @@ def test_decompress_unused_data(self): self.assertEqual(dco.unconsumed_tail, b'') else: data += dco.decompress( - dco.unconsumed_tail + x[i : i + step], maxlen) + dco.unconsumed_tail + x[i : i + step], maxlen) data += dco.flush() self.assertTrue(dco.eof) self.assertEqual(data, source) @@ -747,15 +745,11 @@ def test_baddecompresscopy(self): self.assertRaises(ValueError, copy.copy, d) self.assertRaises(ValueError, copy.deepcopy, d) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_compresspickle(self): for proto in range(pickle.HIGHEST_PROTOCOL + 1): with self.assertRaises((TypeError, pickle.PicklingError)): pickle.dumps(zlib.compressobj(zlib.Z_BEST_COMPRESSION), proto) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_decompresspickle(self): for proto in range(pickle.HIGHEST_PROTOCOL + 1): with self.assertRaises((TypeError, pickle.PicklingError)): @@ -815,8 +809,7 @@ def test_large_unconsumed_tail(self, size): finally: comp = uncomp = data = None - # TODO: RUSTPYTHON: wbits=0 support in flate2 - @unittest.expectedFailure + @unittest.expectedFailure # TODO: RUSTPYTHON; wbits=0 support in flate2 def test_wbits(self): # wbits=0 only supported since zlib v1.2.3.5 supports_wbits_0 = ZLIB_RUNTIME_VERSION_TUPLE >= (1, 2, 3, 5) @@ -945,6 +938,7 @@ def choose_lines(source, number, seed=None, generator=random): Farewell. """ + class ZlibDecompressorTest(unittest.TestCase): # Test adopted from test_bz2.py TEXT = HAMLET_SCENE @@ -1006,8 +1000,6 @@ def testDecompress4G(self, size): compressed = None decompressed = None - # TODO: RUSTPYTHON - @unittest.expectedFailure def testPickle(self): for proto in range(pickle.HIGHEST_PROTOCOL + 1): with self.assertRaises(TypeError): @@ -1021,7 +1013,7 @@ def testDecompressorChunksMaxsize(self): # Feed some input len_ = len(self.BIG_DATA) - 64 out.append(zlibd.decompress(self.BIG_DATA[:len_], - max_length=max_length)) + max_length=max_length)) self.assertFalse(zlibd.needs_input) self.assertEqual(len(out[-1]), max_length) @@ -1032,7 +1024,7 @@ def testDecompressorChunksMaxsize(self): # Retrieve more data while providing more input out.append(zlibd.decompress(self.BIG_DATA[len_:], - max_length=max_length)) + max_length=max_length)) self.assertLessEqual(len(out[-1]), max_length) # Retrieve remaining uncompressed data @@ -1052,7 +1044,7 @@ def test_decompressor_inputbuf_1(self): # Create input buffer and fill it self.assertEqual(zlibd.decompress(self.DATA[:100], - max_length=0), b'') + max_length=0), b'') # Retrieve some results, freeing capacity at beginning # of input buffer @@ -1074,7 +1066,7 @@ def test_decompressor_inputbuf_2(self): # Create input buffer and empty it self.assertEqual(zlibd.decompress(self.DATA[:200], - max_length=0), b'') + max_length=0), b'') out.append(zlibd.decompress(b'')) # Fill buffer with new data @@ -1118,6 +1110,7 @@ def test_refleaks_in___init__(self): zlibd.__init__() self.assertAlmostEqual(gettotalrefcount() - refs_before, 0, delta=10) + class CustomInt: def __index__(self): return 100 diff --git a/Lib/test/test_zoneinfo/test_zoneinfo.py b/Lib/test/test_zoneinfo/test_zoneinfo.py index e05bd046e83..46aa42063a4 100644 --- a/Lib/test/test_zoneinfo/test_zoneinfo.py +++ b/Lib/test/test_zoneinfo/test_zoneinfo.py @@ -1937,8 +1937,6 @@ def test_cache_location(self): self.assertFalse(hasattr(c_zoneinfo.ZoneInfo, "_weak_cache")) self.assertTrue(hasattr(py_zoneinfo.ZoneInfo, "_weak_cache")) - # TODO: RUSTPYTHON - @unittest.expectedFailure def test_gc_tracked(self): import gc diff --git a/Lib/test/test_zstd.py b/Lib/test/test_zstd.py new file mode 100644 index 00000000000..cf618534add --- /dev/null +++ b/Lib/test/test_zstd.py @@ -0,0 +1,2802 @@ +import array +import gc +import io +import pathlib +import random +import re +import os +import unittest +import tempfile +import threading + +from test.support.import_helper import import_module +from test.support import threading_helper +from test.support import _1M +from test.support import Py_GIL_DISABLED + +_zstd = import_module("_zstd") +zstd = import_module("compression.zstd") + +from compression.zstd import ( + open, + compress, + decompress, + ZstdCompressor, + ZstdDecompressor, + ZstdDict, + ZstdError, + zstd_version, + zstd_version_info, + COMPRESSION_LEVEL_DEFAULT, + get_frame_info, + get_frame_size, + finalize_dict, + train_dict, + CompressionParameter, + DecompressionParameter, + Strategy, + ZstdFile, +) + +_1K = 1024 +_130_1K = 130 * _1K +DICT_SIZE1 = 3*_1K + +DAT_130K_D = None +DAT_130K_C = None + +DECOMPRESSED_DAT = None +COMPRESSED_DAT = None + +DECOMPRESSED_100_PLUS_32KB = None +COMPRESSED_100_PLUS_32KB = None + +SKIPPABLE_FRAME = None + +THIS_FILE_BYTES = None +THIS_FILE_STR = None +COMPRESSED_THIS_FILE = None + +COMPRESSED_BOGUS = None + +SAMPLES = None + +TRAINED_DICT = None + +# Cannot be deferred to setup as it is used to check whether or not to skip +# tests +try: + SUPPORT_MULTITHREADING = CompressionParameter.nb_workers.bounds() != (0, 0) +except Exception: + SUPPORT_MULTITHREADING = False + +C_INT_MIN = -(2**31) +C_INT_MAX = (2**31) - 1 + + +def setUpModule(): + # uncompressed size 130KB, more than a zstd block. + # with a frame epilogue, 4 bytes checksum. + global DAT_130K_D + DAT_130K_D = bytes([random.randint(0, 127) for _ in range(130*_1K)]) + + global DAT_130K_C + DAT_130K_C = compress(DAT_130K_D, options={CompressionParameter.checksum_flag:1}) + + global DECOMPRESSED_DAT + DECOMPRESSED_DAT = b'abcdefg123456' * 1000 + + global COMPRESSED_DAT + COMPRESSED_DAT = compress(DECOMPRESSED_DAT) + + global DECOMPRESSED_100_PLUS_32KB + DECOMPRESSED_100_PLUS_32KB = b'a' * (100 + 32*_1K) + + global COMPRESSED_100_PLUS_32KB + COMPRESSED_100_PLUS_32KB = compress(DECOMPRESSED_100_PLUS_32KB) + + global SKIPPABLE_FRAME + SKIPPABLE_FRAME = (0x184D2A50).to_bytes(4, byteorder='little') + \ + (32*_1K).to_bytes(4, byteorder='little') + \ + b'a' * (32*_1K) + + global THIS_FILE_BYTES, THIS_FILE_STR + with io.open(os.path.abspath(__file__), 'rb') as f: + THIS_FILE_BYTES = f.read() + THIS_FILE_BYTES = re.sub(rb'\r?\n', rb'\n', THIS_FILE_BYTES) + THIS_FILE_STR = THIS_FILE_BYTES.decode('utf-8') + + global COMPRESSED_THIS_FILE + COMPRESSED_THIS_FILE = compress(THIS_FILE_BYTES) + + global COMPRESSED_BOGUS + COMPRESSED_BOGUS = DECOMPRESSED_DAT + + # dict data + words = [b'red', b'green', b'yellow', b'black', b'withe', b'blue', + b'lilac', b'purple', b'navy', b'glod', b'silver', b'olive', + b'dog', b'cat', b'tiger', b'lion', b'fish', b'bird'] + lst = [] + for i in range(300): + sample = [b'%s = %d' % (random.choice(words), random.randrange(100)) + for j in range(20)] + sample = b'\n'.join(sample) + + lst.append(sample) + global SAMPLES + SAMPLES = lst + assert len(SAMPLES) > 10 + + global TRAINED_DICT + TRAINED_DICT = train_dict(SAMPLES, 3*_1K) + assert len(TRAINED_DICT.dict_content) <= 3*_1K + + +class FunctionsTestCase(unittest.TestCase): + + def test_version(self): + s = ".".join((str(i) for i in zstd_version_info)) + self.assertEqual(s, zstd_version) + + def test_compressionLevel_values(self): + min, max = CompressionParameter.compression_level.bounds() + self.assertIs(type(COMPRESSION_LEVEL_DEFAULT), int) + self.assertIs(type(min), int) + self.assertIs(type(max), int) + self.assertLess(min, max) + + def test_roundtrip_default(self): + raw_dat = THIS_FILE_BYTES[: len(THIS_FILE_BYTES) // 6] + dat1 = compress(raw_dat) + dat2 = decompress(dat1) + self.assertEqual(dat2, raw_dat) + + def test_roundtrip_level(self): + raw_dat = THIS_FILE_BYTES[: len(THIS_FILE_BYTES) // 6] + level_min, level_max = CompressionParameter.compression_level.bounds() + + for level in range(max(-20, level_min), level_max + 1): + dat1 = compress(raw_dat, level) + dat2 = decompress(dat1) + self.assertEqual(dat2, raw_dat) + + def test_get_frame_info(self): + # no dict + info = get_frame_info(COMPRESSED_100_PLUS_32KB[:20]) + self.assertEqual(info.decompressed_size, 32 * _1K + 100) + self.assertEqual(info.dictionary_id, 0) + + # use dict + dat = compress(b"a" * 345, zstd_dict=TRAINED_DICT) + info = get_frame_info(dat) + self.assertEqual(info.decompressed_size, 345) + self.assertEqual(info.dictionary_id, TRAINED_DICT.dict_id) + + with self.assertRaisesRegex(ZstdError, "not less than the frame header"): + get_frame_info(b"aaaaaaaaaaaaaa") + + def test_get_frame_size(self): + size = get_frame_size(COMPRESSED_100_PLUS_32KB) + self.assertEqual(size, len(COMPRESSED_100_PLUS_32KB)) + + with self.assertRaisesRegex(ZstdError, "not less than this complete frame"): + get_frame_size(b"aaaaaaaaaaaaaa") + + def test_decompress_2x130_1K(self): + decompressed_size = get_frame_info(DAT_130K_C).decompressed_size + self.assertEqual(decompressed_size, _130_1K) + + dat = decompress(DAT_130K_C + DAT_130K_C) + self.assertEqual(len(dat), 2 * _130_1K) + + +class CompressorTestCase(unittest.TestCase): + + def test_simple_compress_bad_args(self): + # ZstdCompressor + self.assertRaises(TypeError, ZstdCompressor, []) + self.assertRaises(TypeError, ZstdCompressor, level=3.14) + self.assertRaises(TypeError, ZstdCompressor, level="abc") + self.assertRaises(TypeError, ZstdCompressor, options=b"abc") + + self.assertRaises(TypeError, ZstdCompressor, zstd_dict=123) + self.assertRaises(TypeError, ZstdCompressor, zstd_dict=b"abcd1234") + self.assertRaises(TypeError, ZstdCompressor, zstd_dict={1: 2, 3: 4}) + + # valid range for compression level is [-(1<<17), 22] + msg = r'illegal compression level {}; the valid range is \[-?\d+, -?\d+\]' + with self.assertRaisesRegex(ValueError, msg.format(C_INT_MAX)): + ZstdCompressor(C_INT_MAX) + with self.assertRaisesRegex(ValueError, msg.format(C_INT_MIN)): + ZstdCompressor(C_INT_MIN) + msg = r'illegal compression level; the valid range is \[-?\d+, -?\d+\]' + with self.assertRaisesRegex(ValueError, msg): + ZstdCompressor(level=-(2**1000)) + with self.assertRaisesRegex(ValueError, msg): + ZstdCompressor(level=2**1000) + + with self.assertRaises(ValueError): + ZstdCompressor(options={CompressionParameter.window_log: 100}) + with self.assertRaises(ValueError): + ZstdCompressor(options={3333: 100}) + + # Method bad arguments + zc = ZstdCompressor() + self.assertRaises(TypeError, zc.compress) + self.assertRaises((TypeError, ValueError), zc.compress, b"foo", b"bar") + self.assertRaises(TypeError, zc.compress, "str") + self.assertRaises((TypeError, ValueError), zc.flush, b"foo") + self.assertRaises(TypeError, zc.flush, b"blah", 1) + + self.assertRaises(ValueError, zc.compress, b'', -1) + self.assertRaises(ValueError, zc.compress, b'', 3) + self.assertRaises(ValueError, zc.flush, zc.CONTINUE) # 0 + self.assertRaises(ValueError, zc.flush, 3) + + zc.compress(b'') + zc.compress(b'', zc.CONTINUE) + zc.compress(b'', zc.FLUSH_BLOCK) + zc.compress(b'', zc.FLUSH_FRAME) + empty = zc.flush() + zc.flush(zc.FLUSH_BLOCK) + zc.flush(zc.FLUSH_FRAME) + + def test_compress_parameters(self): + d = {CompressionParameter.compression_level : 10, + + CompressionParameter.window_log : 12, + CompressionParameter.hash_log : 10, + CompressionParameter.chain_log : 12, + CompressionParameter.search_log : 12, + CompressionParameter.min_match : 4, + CompressionParameter.target_length : 12, + CompressionParameter.strategy : Strategy.lazy, + + CompressionParameter.enable_long_distance_matching : 1, + CompressionParameter.ldm_hash_log : 12, + CompressionParameter.ldm_min_match : 11, + CompressionParameter.ldm_bucket_size_log : 5, + CompressionParameter.ldm_hash_rate_log : 12, + + CompressionParameter.content_size_flag : 1, + CompressionParameter.checksum_flag : 1, + CompressionParameter.dict_id_flag : 0, + + CompressionParameter.nb_workers : 2 if SUPPORT_MULTITHREADING else 0, + CompressionParameter.job_size : 5*_1M if SUPPORT_MULTITHREADING else 0, + CompressionParameter.overlap_log : 9 if SUPPORT_MULTITHREADING else 0, + } + ZstdCompressor(options=d) + + d1 = d.copy() + # larger than signed int + d1[CompressionParameter.ldm_bucket_size_log] = C_INT_MAX + with self.assertRaises(ValueError): + ZstdCompressor(options=d1) + # smaller than signed int + d1[CompressionParameter.ldm_bucket_size_log] = C_INT_MIN + with self.assertRaises(ValueError): + ZstdCompressor(options=d1) + + # out of bounds compression level + level_min, level_max = CompressionParameter.compression_level.bounds() + with self.assertRaises(ValueError): + compress(b'', level_max+1) + with self.assertRaises(ValueError): + compress(b'', level_min-1) + with self.assertRaises(ValueError): + compress(b'', 2**1000) + with self.assertRaises(ValueError): + compress(b'', -(2**1000)) + with self.assertRaises(ValueError): + compress(b'', options={ + CompressionParameter.compression_level: level_max+1}) + with self.assertRaises(ValueError): + compress(b'', options={ + CompressionParameter.compression_level: level_min-1}) + + # zstd lib doesn't support MT compression + if not SUPPORT_MULTITHREADING: + with self.assertRaises(ValueError): + ZstdCompressor(options={CompressionParameter.nb_workers:4}) + with self.assertRaises(ValueError): + ZstdCompressor(options={CompressionParameter.job_size:4}) + with self.assertRaises(ValueError): + ZstdCompressor(options={CompressionParameter.overlap_log:4}) + + # out of bounds error msg + option = {CompressionParameter.window_log:100} + with self.assertRaisesRegex( + ValueError, + "compression parameter 'window_log' received an illegal value 100; " + r'the valid range is \[-?\d+, -?\d+\]', + ): + compress(b'', options=option) + + def test_unknown_compression_parameter(self): + KEY = 100001234 + option = {CompressionParameter.compression_level: 10, + KEY: 200000000} + pattern = rf"invalid compression parameter 'unknown parameter \(key {KEY}\)'" + with self.assertRaisesRegex(ValueError, pattern): + ZstdCompressor(options=option) + + @unittest.skipIf(not SUPPORT_MULTITHREADING, + "zstd build doesn't support multi-threaded compression") + def test_zstd_multithread_compress(self): + size = 40*_1M + b = THIS_FILE_BYTES * (size // len(THIS_FILE_BYTES)) + + options = {CompressionParameter.compression_level : 4, + CompressionParameter.nb_workers : 2} + + # compress() + dat1 = compress(b, options=options) + dat2 = decompress(dat1) + self.assertEqual(dat2, b) + + # ZstdCompressor + c = ZstdCompressor(options=options) + dat1 = c.compress(b, c.CONTINUE) + dat2 = c.compress(b, c.FLUSH_BLOCK) + dat3 = c.compress(b, c.FLUSH_FRAME) + dat4 = decompress(dat1+dat2+dat3) + self.assertEqual(dat4, b * 3) + + # ZstdFile + with ZstdFile(io.BytesIO(), 'w', options=options) as f: + f.write(b) + + def test_compress_flushblock(self): + point = len(THIS_FILE_BYTES) // 2 + + c = ZstdCompressor() + self.assertEqual(c.last_mode, c.FLUSH_FRAME) + dat1 = c.compress(THIS_FILE_BYTES[:point]) + self.assertEqual(c.last_mode, c.CONTINUE) + dat1 += c.compress(THIS_FILE_BYTES[point:], c.FLUSH_BLOCK) + self.assertEqual(c.last_mode, c.FLUSH_BLOCK) + dat2 = c.flush() + pattern = "Compressed data ended before the end-of-stream marker" + with self.assertRaisesRegex(ZstdError, pattern): + decompress(dat1) + + dat3 = decompress(dat1 + dat2) + + self.assertEqual(dat3, THIS_FILE_BYTES) + + def test_compress_flushframe(self): + # test compress & decompress + point = len(THIS_FILE_BYTES) // 2 + + c = ZstdCompressor() + + dat1 = c.compress(THIS_FILE_BYTES[:point]) + self.assertEqual(c.last_mode, c.CONTINUE) + + dat1 += c.compress(THIS_FILE_BYTES[point:], c.FLUSH_FRAME) + self.assertEqual(c.last_mode, c.FLUSH_FRAME) + + nt = get_frame_info(dat1) + self.assertEqual(nt.decompressed_size, None) # no content size + + dat2 = decompress(dat1) + + self.assertEqual(dat2, THIS_FILE_BYTES) + + # single .FLUSH_FRAME mode has content size + c = ZstdCompressor() + dat = c.compress(THIS_FILE_BYTES, mode=c.FLUSH_FRAME) + self.assertEqual(c.last_mode, c.FLUSH_FRAME) + + nt = get_frame_info(dat) + self.assertEqual(nt.decompressed_size, len(THIS_FILE_BYTES)) + + def test_compress_empty(self): + # output empty content frame + self.assertNotEqual(compress(b''), b'') + + c = ZstdCompressor() + self.assertNotEqual(c.compress(b'', c.FLUSH_FRAME), b'') + + def test_set_pledged_input_size(self): + DAT = DECOMPRESSED_100_PLUS_32KB + CHUNK_SIZE = len(DAT) // 3 + + # wrong value + c = ZstdCompressor() + with self.assertRaisesRegex(ValueError, + r'should be a positive int less than \d+'): + c.set_pledged_input_size(-300) + # overflow + with self.assertRaisesRegex(ValueError, + r'should be a positive int less than \d+'): + c.set_pledged_input_size(2**64) + # ZSTD_CONTENTSIZE_ERROR is invalid + with self.assertRaisesRegex(ValueError, + r'should be a positive int less than \d+'): + c.set_pledged_input_size(2**64-2) + # ZSTD_CONTENTSIZE_UNKNOWN should use None + with self.assertRaisesRegex(ValueError, + r'should be a positive int less than \d+'): + c.set_pledged_input_size(2**64-1) + + # check valid values are settable + c.set_pledged_input_size(2**63) + c.set_pledged_input_size(2**64-3) + + # check that zero means empty frame + c = ZstdCompressor(level=1) + c.set_pledged_input_size(0) + c.compress(b'') + dat = c.flush() + ret = get_frame_info(dat) + self.assertEqual(ret.decompressed_size, 0) + + + # wrong mode + c = ZstdCompressor(level=1) + c.compress(b'123456') + self.assertEqual(c.last_mode, c.CONTINUE) + with self.assertRaisesRegex(ValueError, + r'last_mode == FLUSH_FRAME'): + c.set_pledged_input_size(300) + + # None value + c = ZstdCompressor(level=1) + c.set_pledged_input_size(None) + dat = c.compress(DAT) + c.flush() + + ret = get_frame_info(dat) + self.assertEqual(ret.decompressed_size, None) + + # correct value + c = ZstdCompressor(level=1) + c.set_pledged_input_size(len(DAT)) + + chunks = [] + posi = 0 + while posi < len(DAT): + dat = c.compress(DAT[posi:posi+CHUNK_SIZE]) + posi += CHUNK_SIZE + chunks.append(dat) + + dat = c.flush() + chunks.append(dat) + chunks = b''.join(chunks) + + ret = get_frame_info(chunks) + self.assertEqual(ret.decompressed_size, len(DAT)) + self.assertEqual(decompress(chunks), DAT) + + c.set_pledged_input_size(len(DAT)) # the second frame + dat = c.compress(DAT) + c.flush() + + ret = get_frame_info(dat) + self.assertEqual(ret.decompressed_size, len(DAT)) + self.assertEqual(decompress(dat), DAT) + + # not enough data + c = ZstdCompressor(level=1) + c.set_pledged_input_size(len(DAT)+1) + + for start in range(0, len(DAT), CHUNK_SIZE): + end = min(start+CHUNK_SIZE, len(DAT)) + _dat = c.compress(DAT[start:end]) + + with self.assertRaises(ZstdError): + c.flush() + + # too much data + c = ZstdCompressor(level=1) + c.set_pledged_input_size(len(DAT)) + + for start in range(0, len(DAT), CHUNK_SIZE): + end = min(start+CHUNK_SIZE, len(DAT)) + _dat = c.compress(DAT[start:end]) + + with self.assertRaises(ZstdError): + c.compress(b'extra', ZstdCompressor.FLUSH_FRAME) + + # content size not set if content_size_flag == 0 + c = ZstdCompressor(options={CompressionParameter.content_size_flag: 0}) + c.set_pledged_input_size(10) + dat1 = c.compress(b"hello") + dat2 = c.compress(b"world") + dat3 = c.flush() + frame_data = get_frame_info(dat1 + dat2 + dat3) + self.assertIsNone(frame_data.decompressed_size) + + +class DecompressorTestCase(unittest.TestCase): + + def test_simple_decompress_bad_args(self): + # ZstdDecompressor + self.assertRaises(TypeError, ZstdDecompressor, ()) + self.assertRaises(TypeError, ZstdDecompressor, zstd_dict=123) + self.assertRaises(TypeError, ZstdDecompressor, zstd_dict=b'abc') + self.assertRaises(TypeError, ZstdDecompressor, zstd_dict={1:2, 3:4}) + + self.assertRaises(TypeError, ZstdDecompressor, options=123) + self.assertRaises(TypeError, ZstdDecompressor, options='abc') + self.assertRaises(TypeError, ZstdDecompressor, options=b'abc') + + with self.assertRaises(ValueError): + ZstdDecompressor(options={C_INT_MAX: 100}) + with self.assertRaises(ValueError): + ZstdDecompressor(options={C_INT_MIN: 100}) + with self.assertRaises(ValueError): + ZstdDecompressor(options={0: C_INT_MAX}) + with self.assertRaises(OverflowError): + ZstdDecompressor(options={2**1000: 100}) + with self.assertRaises(OverflowError): + ZstdDecompressor(options={-(2**1000): 100}) + with self.assertRaises(OverflowError): + ZstdDecompressor(options={0: -(2**1000)}) + + with self.assertRaises(ValueError): + ZstdDecompressor(options={DecompressionParameter.window_log_max: 100}) + with self.assertRaises(ValueError): + ZstdDecompressor(options={3333: 100}) + + empty = compress(b'') + lzd = ZstdDecompressor() + self.assertRaises(TypeError, lzd.decompress) + self.assertRaises(TypeError, lzd.decompress, b"foo", b"bar") + self.assertRaises(TypeError, lzd.decompress, "str") + lzd.decompress(empty) + + def test_decompress_parameters(self): + d = {DecompressionParameter.window_log_max : 15} + ZstdDecompressor(options=d) + + d1 = d.copy() + # larger than signed int + d1[DecompressionParameter.window_log_max] = 2**1000 + with self.assertRaises(OverflowError): + ZstdDecompressor(None, d1) + # smaller than signed int + d1[DecompressionParameter.window_log_max] = -(2**1000) + with self.assertRaises(OverflowError): + ZstdDecompressor(None, d1) + + d1[DecompressionParameter.window_log_max] = C_INT_MAX + with self.assertRaises(ValueError): + ZstdDecompressor(None, d1) + d1[DecompressionParameter.window_log_max] = C_INT_MIN + with self.assertRaises(ValueError): + ZstdDecompressor(None, d1) + + # out of bounds error msg + options = {DecompressionParameter.window_log_max:100} + with self.assertRaisesRegex( + ValueError, + "decompression parameter 'window_log_max' received an illegal value 100; " + r'the valid range is \[-?\d+, -?\d+\]', + ): + decompress(b'', options=options) + + # out of bounds deecompression parameter + options[DecompressionParameter.window_log_max] = C_INT_MAX + with self.assertRaises(ValueError): + decompress(b'', options=options) + options[DecompressionParameter.window_log_max] = C_INT_MIN + with self.assertRaises(ValueError): + decompress(b'', options=options) + options[DecompressionParameter.window_log_max] = 2**1000 + with self.assertRaises(OverflowError): + decompress(b'', options=options) + options[DecompressionParameter.window_log_max] = -(2**1000) + with self.assertRaises(OverflowError): + decompress(b'', options=options) + + def test_unknown_decompression_parameter(self): + KEY = 100001234 + options = {DecompressionParameter.window_log_max: DecompressionParameter.window_log_max.bounds()[1], + KEY: 200000000} + pattern = rf"invalid decompression parameter 'unknown parameter \(key {KEY}\)'" + with self.assertRaisesRegex(ValueError, pattern): + ZstdDecompressor(options=options) + + def test_decompress_epilogue_flags(self): + # DAT_130K_C has a 4 bytes checksum at frame epilogue + + # full unlimited + d = ZstdDecompressor() + dat = d.decompress(DAT_130K_C) + self.assertEqual(len(dat), _130_1K) + self.assertFalse(d.needs_input) + + with self.assertRaises(EOFError): + dat = d.decompress(b'') + + # full limited + d = ZstdDecompressor() + dat = d.decompress(DAT_130K_C, _130_1K) + self.assertEqual(len(dat), _130_1K) + self.assertFalse(d.needs_input) + + with self.assertRaises(EOFError): + dat = d.decompress(b'', 0) + + # [:-4] unlimited + d = ZstdDecompressor() + dat = d.decompress(DAT_130K_C[:-4]) + self.assertEqual(len(dat), _130_1K) + self.assertTrue(d.needs_input) + + dat = d.decompress(b'') + self.assertEqual(len(dat), 0) + self.assertTrue(d.needs_input) + + # [:-4] limited + d = ZstdDecompressor() + dat = d.decompress(DAT_130K_C[:-4], _130_1K) + self.assertEqual(len(dat), _130_1K) + self.assertFalse(d.needs_input) + + dat = d.decompress(b'', 0) + self.assertEqual(len(dat), 0) + self.assertFalse(d.needs_input) + + # [:-3] unlimited + d = ZstdDecompressor() + dat = d.decompress(DAT_130K_C[:-3]) + self.assertEqual(len(dat), _130_1K) + self.assertTrue(d.needs_input) + + dat = d.decompress(b'') + self.assertEqual(len(dat), 0) + self.assertTrue(d.needs_input) + + # [:-3] limited + d = ZstdDecompressor() + dat = d.decompress(DAT_130K_C[:-3], _130_1K) + self.assertEqual(len(dat), _130_1K) + self.assertFalse(d.needs_input) + + dat = d.decompress(b'', 0) + self.assertEqual(len(dat), 0) + self.assertFalse(d.needs_input) + + # [:-1] unlimited + d = ZstdDecompressor() + dat = d.decompress(DAT_130K_C[:-1]) + self.assertEqual(len(dat), _130_1K) + self.assertTrue(d.needs_input) + + dat = d.decompress(b'') + self.assertEqual(len(dat), 0) + self.assertTrue(d.needs_input) + + # [:-1] limited + d = ZstdDecompressor() + dat = d.decompress(DAT_130K_C[:-1], _130_1K) + self.assertEqual(len(dat), _130_1K) + self.assertFalse(d.needs_input) + + dat = d.decompress(b'', 0) + self.assertEqual(len(dat), 0) + self.assertFalse(d.needs_input) + + def test_decompressor_arg(self): + zd = ZstdDict(b'12345678', is_raw=True) + + with self.assertRaises(TypeError): + d = ZstdDecompressor(zstd_dict={}) + + with self.assertRaises(TypeError): + d = ZstdDecompressor(options=zd) + + ZstdDecompressor() + ZstdDecompressor(zd, {}) + ZstdDecompressor(zstd_dict=zd, options={DecompressionParameter.window_log_max:25}) + + def test_decompressor_1(self): + # empty + d = ZstdDecompressor() + dat = d.decompress(b'') + + self.assertEqual(dat, b'') + self.assertFalse(d.eof) + + # 130_1K full + d = ZstdDecompressor() + dat = d.decompress(DAT_130K_C) + + self.assertEqual(len(dat), _130_1K) + self.assertTrue(d.eof) + self.assertFalse(d.needs_input) + + # 130_1K full, limit output + d = ZstdDecompressor() + dat = d.decompress(DAT_130K_C, _130_1K) + + self.assertEqual(len(dat), _130_1K) + self.assertTrue(d.eof) + self.assertFalse(d.needs_input) + + # 130_1K, without 4 bytes checksum + d = ZstdDecompressor() + dat = d.decompress(DAT_130K_C[:-4]) + + self.assertEqual(len(dat), _130_1K) + self.assertFalse(d.eof) + self.assertTrue(d.needs_input) + + # above, limit output + d = ZstdDecompressor() + dat = d.decompress(DAT_130K_C[:-4], _130_1K) + + self.assertEqual(len(dat), _130_1K) + self.assertFalse(d.eof) + self.assertFalse(d.needs_input) + + # full, unused_data + TRAIL = b'89234893abcd' + d = ZstdDecompressor() + dat = d.decompress(DAT_130K_C + TRAIL, _130_1K) + + self.assertEqual(len(dat), _130_1K) + self.assertTrue(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data, TRAIL) + + def test_decompressor_chunks_read_300(self): + TRAIL = b'89234893abcd' + DAT = DAT_130K_C + TRAIL + d = ZstdDecompressor() + + bi = io.BytesIO(DAT) + lst = [] + while True: + if d.needs_input: + dat = bi.read(300) + if not dat: + break + else: + raise Exception('should not get here') + + ret = d.decompress(dat) + lst.append(ret) + if d.eof: + break + + ret = b''.join(lst) + + self.assertEqual(len(ret), _130_1K) + self.assertTrue(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data + bi.read(), TRAIL) + + def test_decompressor_chunks_read_3(self): + TRAIL = b'89234893' + DAT = DAT_130K_C + TRAIL + d = ZstdDecompressor() + + bi = io.BytesIO(DAT) + lst = [] + while True: + if d.needs_input: + dat = bi.read(3) + if not dat: + break + else: + dat = b'' + + ret = d.decompress(dat, 1) + lst.append(ret) + if d.eof: + break + + ret = b''.join(lst) + + self.assertEqual(len(ret), _130_1K) + self.assertTrue(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data + bi.read(), TRAIL) + + + def test_decompress_empty(self): + with self.assertRaises(ZstdError): + decompress(b'') + + d = ZstdDecompressor() + self.assertEqual(d.decompress(b''), b'') + self.assertFalse(d.eof) + + def test_decompress_empty_content_frame(self): + DAT = compress(b'') + # decompress + self.assertGreaterEqual(len(DAT), 4) + self.assertEqual(decompress(DAT), b'') + + with self.assertRaises(ZstdError): + decompress(DAT[:-1]) + + # ZstdDecompressor + d = ZstdDecompressor() + dat = d.decompress(DAT) + self.assertEqual(dat, b'') + self.assertTrue(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data, b'') + self.assertEqual(d.unused_data, b'') # twice + + d = ZstdDecompressor() + dat = d.decompress(DAT[:-1]) + self.assertEqual(dat, b'') + self.assertFalse(d.eof) + self.assertTrue(d.needs_input) + self.assertEqual(d.unused_data, b'') + self.assertEqual(d.unused_data, b'') # twice + +class DecompressorFlagsTestCase(unittest.TestCase): + + @classmethod + def setUpClass(cls): + options = {CompressionParameter.checksum_flag:1} + c = ZstdCompressor(options=options) + + cls.DECOMPRESSED_42 = b'a'*42 + cls.FRAME_42 = c.compress(cls.DECOMPRESSED_42, c.FLUSH_FRAME) + + cls.DECOMPRESSED_60 = b'a'*60 + cls.FRAME_60 = c.compress(cls.DECOMPRESSED_60, c.FLUSH_FRAME) + + cls.FRAME_42_60 = cls.FRAME_42 + cls.FRAME_60 + cls.DECOMPRESSED_42_60 = cls.DECOMPRESSED_42 + cls.DECOMPRESSED_60 + + cls._130_1K = 130*_1K + + c = ZstdCompressor() + cls.UNKNOWN_FRAME_42 = c.compress(cls.DECOMPRESSED_42) + c.flush() + cls.UNKNOWN_FRAME_60 = c.compress(cls.DECOMPRESSED_60) + c.flush() + cls.UNKNOWN_FRAME_42_60 = cls.UNKNOWN_FRAME_42 + cls.UNKNOWN_FRAME_60 + + cls.TRAIL = b'12345678abcdefg!@#$%^&*()_+|' + + def test_function_decompress(self): + + self.assertEqual(len(decompress(COMPRESSED_100_PLUS_32KB)), 100+32*_1K) + + # 1 frame + self.assertEqual(decompress(self.FRAME_42), self.DECOMPRESSED_42) + + self.assertEqual(decompress(self.UNKNOWN_FRAME_42), self.DECOMPRESSED_42) + + pattern = r"Compressed data ended before the end-of-stream marker" + with self.assertRaisesRegex(ZstdError, pattern): + decompress(self.FRAME_42[:1]) + + with self.assertRaisesRegex(ZstdError, pattern): + decompress(self.FRAME_42[:-4]) + + with self.assertRaisesRegex(ZstdError, pattern): + decompress(self.FRAME_42[:-1]) + + # 2 frames + self.assertEqual(decompress(self.FRAME_42_60), self.DECOMPRESSED_42_60) + + self.assertEqual(decompress(self.UNKNOWN_FRAME_42_60), self.DECOMPRESSED_42_60) + + self.assertEqual(decompress(self.FRAME_42 + self.UNKNOWN_FRAME_60), + self.DECOMPRESSED_42_60) + + self.assertEqual(decompress(self.UNKNOWN_FRAME_42 + self.FRAME_60), + self.DECOMPRESSED_42_60) + + with self.assertRaisesRegex(ZstdError, pattern): + decompress(self.FRAME_42_60[:-4]) + + with self.assertRaisesRegex(ZstdError, pattern): + decompress(self.UNKNOWN_FRAME_42_60[:-1]) + + # 130_1K + self.assertEqual(decompress(DAT_130K_C), DAT_130K_D) + + with self.assertRaisesRegex(ZstdError, pattern): + decompress(DAT_130K_C[:-4]) + + with self.assertRaisesRegex(ZstdError, pattern): + decompress(DAT_130K_C[:-1]) + + # Unknown frame descriptor + with self.assertRaisesRegex(ZstdError, "Unknown frame descriptor"): + decompress(b'aaaaaaaaa') + + with self.assertRaisesRegex(ZstdError, "Unknown frame descriptor"): + decompress(self.FRAME_42 + b'aaaaaaaaa') + + with self.assertRaisesRegex(ZstdError, "Unknown frame descriptor"): + decompress(self.UNKNOWN_FRAME_42_60 + b'aaaaaaaaa') + + # doesn't match checksum + checksum = DAT_130K_C[-4:] + if checksum[0] == 255: + wrong_checksum = bytes([254]) + checksum[1:] + else: + wrong_checksum = bytes([checksum[0]+1]) + checksum[1:] + + dat = DAT_130K_C[:-4] + wrong_checksum + + with self.assertRaisesRegex(ZstdError, "doesn't match checksum"): + decompress(dat) + + def test_function_skippable(self): + self.assertEqual(decompress(SKIPPABLE_FRAME), b'') + self.assertEqual(decompress(SKIPPABLE_FRAME + SKIPPABLE_FRAME), b'') + + # 1 frame + 2 skippable + self.assertEqual(len(decompress(SKIPPABLE_FRAME + SKIPPABLE_FRAME + DAT_130K_C)), + self._130_1K) + + self.assertEqual(len(decompress(DAT_130K_C + SKIPPABLE_FRAME + SKIPPABLE_FRAME)), + self._130_1K) + + self.assertEqual(len(decompress(SKIPPABLE_FRAME + DAT_130K_C + SKIPPABLE_FRAME)), + self._130_1K) + + # unknown size + self.assertEqual(decompress(SKIPPABLE_FRAME + self.UNKNOWN_FRAME_60), + self.DECOMPRESSED_60) + + self.assertEqual(decompress(self.UNKNOWN_FRAME_60 + SKIPPABLE_FRAME), + self.DECOMPRESSED_60) + + # 2 frames + 1 skippable + self.assertEqual(decompress(self.FRAME_42 + SKIPPABLE_FRAME + self.FRAME_60), + self.DECOMPRESSED_42_60) + + self.assertEqual(decompress(SKIPPABLE_FRAME + self.FRAME_42_60), + self.DECOMPRESSED_42_60) + + self.assertEqual(decompress(self.UNKNOWN_FRAME_42_60 + SKIPPABLE_FRAME), + self.DECOMPRESSED_42_60) + + # incomplete + with self.assertRaises(ZstdError): + decompress(SKIPPABLE_FRAME[:1]) + + with self.assertRaises(ZstdError): + decompress(SKIPPABLE_FRAME[:-1]) + + with self.assertRaises(ZstdError): + decompress(self.FRAME_42 + SKIPPABLE_FRAME[:-1]) + + # Unknown frame descriptor + with self.assertRaisesRegex(ZstdError, "Unknown frame descriptor"): + decompress(b'aaaaaaaaa' + SKIPPABLE_FRAME) + + with self.assertRaisesRegex(ZstdError, "Unknown frame descriptor"): + decompress(SKIPPABLE_FRAME + b'aaaaaaaaa') + + with self.assertRaisesRegex(ZstdError, "Unknown frame descriptor"): + decompress(SKIPPABLE_FRAME + SKIPPABLE_FRAME + b'aaaaaaaaa') + + def test_decompressor_1(self): + # empty 1 + d = ZstdDecompressor() + + dat = d.decompress(b'') + self.assertEqual(dat, b'') + self.assertFalse(d.eof) + self.assertTrue(d.needs_input) + self.assertEqual(d.unused_data, b'') + self.assertEqual(d.unused_data, b'') # twice + + dat = d.decompress(b'', 0) + self.assertEqual(dat, b'') + self.assertFalse(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data, b'') + self.assertEqual(d.unused_data, b'') # twice + + dat = d.decompress(COMPRESSED_100_PLUS_32KB + b'a') + self.assertEqual(dat, DECOMPRESSED_100_PLUS_32KB) + self.assertTrue(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data, b'a') + self.assertEqual(d.unused_data, b'a') # twice + + # empty 2 + d = ZstdDecompressor() + + dat = d.decompress(b'', 0) + self.assertEqual(dat, b'') + self.assertFalse(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data, b'') + self.assertEqual(d.unused_data, b'') # twice + + dat = d.decompress(b'') + self.assertEqual(dat, b'') + self.assertFalse(d.eof) + self.assertTrue(d.needs_input) + self.assertEqual(d.unused_data, b'') + self.assertEqual(d.unused_data, b'') # twice + + dat = d.decompress(COMPRESSED_100_PLUS_32KB + b'a') + self.assertEqual(dat, DECOMPRESSED_100_PLUS_32KB) + self.assertTrue(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data, b'a') + self.assertEqual(d.unused_data, b'a') # twice + + # 1 frame + d = ZstdDecompressor() + dat = d.decompress(self.FRAME_42) + + self.assertEqual(dat, self.DECOMPRESSED_42) + self.assertTrue(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data, b'') + self.assertEqual(d.unused_data, b'') # twice + + with self.assertRaises(EOFError): + d.decompress(b'') + + # 1 frame, trail + d = ZstdDecompressor() + dat = d.decompress(self.FRAME_42 + self.TRAIL) + + self.assertEqual(dat, self.DECOMPRESSED_42) + self.assertTrue(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data, self.TRAIL) + self.assertEqual(d.unused_data, self.TRAIL) # twice + + # 1 frame, 32_1K + temp = compress(b'a'*(32*_1K)) + d = ZstdDecompressor() + dat = d.decompress(temp, 32*_1K) + + self.assertEqual(dat, b'a'*(32*_1K)) + self.assertTrue(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data, b'') + self.assertEqual(d.unused_data, b'') # twice + + with self.assertRaises(EOFError): + d.decompress(b'') + + # 1 frame, 32_1K+100, trail + d = ZstdDecompressor() + dat = d.decompress(COMPRESSED_100_PLUS_32KB+self.TRAIL, 100) # 100 bytes + + self.assertEqual(len(dat), 100) + self.assertFalse(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data, b'') + + dat = d.decompress(b'') # 32_1K + + self.assertEqual(len(dat), 32*_1K) + self.assertTrue(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data, self.TRAIL) + self.assertEqual(d.unused_data, self.TRAIL) # twice + + with self.assertRaises(EOFError): + d.decompress(b'') + + # incomplete 1 + d = ZstdDecompressor() + dat = d.decompress(self.FRAME_60[:1]) + + self.assertFalse(d.eof) + self.assertTrue(d.needs_input) + self.assertEqual(d.unused_data, b'') + self.assertEqual(d.unused_data, b'') # twice + + # incomplete 2 + d = ZstdDecompressor() + + dat = d.decompress(self.FRAME_60[:-4]) + self.assertEqual(dat, self.DECOMPRESSED_60) + self.assertFalse(d.eof) + self.assertTrue(d.needs_input) + self.assertEqual(d.unused_data, b'') + self.assertEqual(d.unused_data, b'') # twice + + # incomplete 3 + d = ZstdDecompressor() + + dat = d.decompress(self.FRAME_60[:-1]) + self.assertEqual(dat, self.DECOMPRESSED_60) + self.assertFalse(d.eof) + self.assertTrue(d.needs_input) + self.assertEqual(d.unused_data, b'') + + # incomplete 4 + d = ZstdDecompressor() + + dat = d.decompress(self.FRAME_60[:-4], 60) + self.assertEqual(dat, self.DECOMPRESSED_60) + self.assertFalse(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data, b'') + self.assertEqual(d.unused_data, b'') # twice + + dat = d.decompress(b'') + self.assertEqual(dat, b'') + self.assertFalse(d.eof) + self.assertTrue(d.needs_input) + self.assertEqual(d.unused_data, b'') + self.assertEqual(d.unused_data, b'') # twice + + # Unknown frame descriptor + d = ZstdDecompressor() + with self.assertRaisesRegex(ZstdError, "Unknown frame descriptor"): + d.decompress(b'aaaaaaaaa') + + def test_decompressor_skippable(self): + # 1 skippable + d = ZstdDecompressor() + dat = d.decompress(SKIPPABLE_FRAME) + + self.assertEqual(dat, b'') + self.assertTrue(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data, b'') + self.assertEqual(d.unused_data, b'') # twice + + # 1 skippable, max_length=0 + d = ZstdDecompressor() + dat = d.decompress(SKIPPABLE_FRAME, 0) + + self.assertEqual(dat, b'') + self.assertTrue(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data, b'') + self.assertEqual(d.unused_data, b'') # twice + + # 1 skippable, trail + d = ZstdDecompressor() + dat = d.decompress(SKIPPABLE_FRAME + self.TRAIL) + + self.assertEqual(dat, b'') + self.assertTrue(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data, self.TRAIL) + self.assertEqual(d.unused_data, self.TRAIL) # twice + + # incomplete + d = ZstdDecompressor() + dat = d.decompress(SKIPPABLE_FRAME[:-1]) + + self.assertEqual(dat, b'') + self.assertFalse(d.eof) + self.assertTrue(d.needs_input) + self.assertEqual(d.unused_data, b'') + self.assertEqual(d.unused_data, b'') # twice + + # incomplete + d = ZstdDecompressor() + dat = d.decompress(SKIPPABLE_FRAME[:-1], 0) + + self.assertEqual(dat, b'') + self.assertFalse(d.eof) + self.assertFalse(d.needs_input) + self.assertEqual(d.unused_data, b'') + self.assertEqual(d.unused_data, b'') # twice + + dat = d.decompress(b'') + + self.assertEqual(dat, b'') + self.assertFalse(d.eof) + self.assertTrue(d.needs_input) + self.assertEqual(d.unused_data, b'') + self.assertEqual(d.unused_data, b'') # twice + + + +class ZstdDictTestCase(unittest.TestCase): + + def test_is_raw(self): + # must be passed as a keyword argument + with self.assertRaises(TypeError): + ZstdDict(bytes(8), True) + + # content < 8 + b = b'1234567' + with self.assertRaises(ValueError): + ZstdDict(b) + + # content == 8 + b = b'12345678' + zd = ZstdDict(b, is_raw=True) + self.assertEqual(zd.dict_id, 0) + + temp = compress(b'aaa12345678', level=3, zstd_dict=zd) + self.assertEqual(b'aaa12345678', decompress(temp, zd)) + + # is_raw == False + b = b'12345678abcd' + with self.assertRaises(ValueError): + ZstdDict(b) + + # read only attributes + with self.assertRaises(AttributeError): + zd.dict_content = b + + with self.assertRaises(AttributeError): + zd.dict_id = 10000 + + # ZstdDict arguments + zd = ZstdDict(TRAINED_DICT.dict_content, is_raw=False) + self.assertNotEqual(zd.dict_id, 0) + + zd = ZstdDict(TRAINED_DICT.dict_content, is_raw=True) + self.assertNotEqual(zd.dict_id, 0) # note this assertion + + with self.assertRaises(TypeError): + ZstdDict("12345678abcdef", is_raw=True) + with self.assertRaises(TypeError): + ZstdDict(TRAINED_DICT) + + # invalid parameter + with self.assertRaises(TypeError): + ZstdDict(desk333=345) + + def test_invalid_dict(self): + DICT_MAGIC = 0xEC30A437.to_bytes(4, byteorder='little') + dict_content = DICT_MAGIC + b'abcdefghighlmnopqrstuvwxyz' + + # corrupted + zd = ZstdDict(dict_content, is_raw=False) + with self.assertRaisesRegex(ZstdError, r'ZSTD_CDict.*?content\.$'): + ZstdCompressor(zstd_dict=zd.as_digested_dict) + with self.assertRaisesRegex(ZstdError, r'ZSTD_DDict.*?content\.$'): + ZstdDecompressor(zd) + + # wrong type + with self.assertRaisesRegex(TypeError, r'should be a ZstdDict object'): + ZstdCompressor(zstd_dict=[zd, 1]) + with self.assertRaisesRegex(TypeError, r'should be a ZstdDict object'): + ZstdCompressor(zstd_dict=(zd, 1.0)) + with self.assertRaisesRegex(TypeError, r'should be a ZstdDict object'): + ZstdCompressor(zstd_dict=(zd,)) + with self.assertRaisesRegex(TypeError, r'should be a ZstdDict object'): + ZstdCompressor(zstd_dict=(zd, 1, 2)) + with self.assertRaisesRegex(TypeError, r'should be a ZstdDict object'): + ZstdCompressor(zstd_dict=(zd, -1)) + with self.assertRaisesRegex(TypeError, r'should be a ZstdDict object'): + ZstdCompressor(zstd_dict=(zd, 3)) + with self.assertRaises(OverflowError): + ZstdCompressor(zstd_dict=(zd, 2**1000)) + with self.assertRaises(OverflowError): + ZstdCompressor(zstd_dict=(zd, -2**1000)) + + with self.assertRaisesRegex(TypeError, r'should be a ZstdDict object'): + ZstdDecompressor(zstd_dict=[zd, 1]) + with self.assertRaisesRegex(TypeError, r'should be a ZstdDict object'): + ZstdDecompressor(zstd_dict=(zd, 1.0)) + with self.assertRaisesRegex(TypeError, r'should be a ZstdDict object'): + ZstdDecompressor((zd,)) + with self.assertRaisesRegex(TypeError, r'should be a ZstdDict object'): + ZstdDecompressor((zd, 1, 2)) + with self.assertRaisesRegex(TypeError, r'should be a ZstdDict object'): + ZstdDecompressor((zd, -1)) + with self.assertRaisesRegex(TypeError, r'should be a ZstdDict object'): + ZstdDecompressor((zd, 3)) + with self.assertRaises(OverflowError): + ZstdDecompressor((zd, 2**1000)) + with self.assertRaises(OverflowError): + ZstdDecompressor((zd, -2**1000)) + + def test_train_dict(self): + TRAINED_DICT = train_dict(SAMPLES, DICT_SIZE1) + ZstdDict(TRAINED_DICT.dict_content, is_raw=False) + + self.assertNotEqual(TRAINED_DICT.dict_id, 0) + self.assertGreater(len(TRAINED_DICT.dict_content), 0) + self.assertLessEqual(len(TRAINED_DICT.dict_content), DICT_SIZE1) + self.assertTrue(re.match(r'^<ZstdDict dict_id=\d+ dict_size=\d+>$', str(TRAINED_DICT))) + + # compress/decompress + c = ZstdCompressor(zstd_dict=TRAINED_DICT) + for sample in SAMPLES: + dat1 = compress(sample, zstd_dict=TRAINED_DICT) + dat2 = decompress(dat1, TRAINED_DICT) + self.assertEqual(sample, dat2) + + dat1 = c.compress(sample) + dat1 += c.flush() + dat2 = decompress(dat1, TRAINED_DICT) + self.assertEqual(sample, dat2) + + def test_finalize_dict(self): + DICT_SIZE2 = 200*_1K + C_LEVEL = 6 + + try: + dic2 = finalize_dict(TRAINED_DICT, SAMPLES, DICT_SIZE2, C_LEVEL) + except NotImplementedError: + # < v1.4.5 at compile-time, >= v.1.4.5 at run-time + return + + self.assertNotEqual(dic2.dict_id, 0) + self.assertGreater(len(dic2.dict_content), 0) + self.assertLessEqual(len(dic2.dict_content), DICT_SIZE2) + + # compress/decompress + c = ZstdCompressor(C_LEVEL, zstd_dict=dic2) + for sample in SAMPLES: + dat1 = compress(sample, C_LEVEL, zstd_dict=dic2) + dat2 = decompress(dat1, dic2) + self.assertEqual(sample, dat2) + + dat1 = c.compress(sample) + dat1 += c.flush() + dat2 = decompress(dat1, dic2) + self.assertEqual(sample, dat2) + + # dict mismatch + self.assertNotEqual(TRAINED_DICT.dict_id, dic2.dict_id) + + dat1 = compress(SAMPLES[0], zstd_dict=TRAINED_DICT) + with self.assertRaises(ZstdError): + decompress(dat1, dic2) + + def test_train_dict_arguments(self): + with self.assertRaises(ValueError): + train_dict([], 100*_1K) + + with self.assertRaises(ValueError): + train_dict(SAMPLES, -100) + + with self.assertRaises(ValueError): + train_dict(SAMPLES, 0) + + def test_finalize_dict_arguments(self): + with self.assertRaises(TypeError): + finalize_dict({1:2}, (b'aaa', b'bbb'), 100*_1K, 2) + + with self.assertRaises(ValueError): + finalize_dict(TRAINED_DICT, [], 100*_1K, 2) + + with self.assertRaises(ValueError): + finalize_dict(TRAINED_DICT, SAMPLES, -100, 2) + + with self.assertRaises(ValueError): + finalize_dict(TRAINED_DICT, SAMPLES, 0, 2) + + def test_train_dict_c(self): + # argument wrong type + with self.assertRaises(TypeError): + _zstd.train_dict({}, (), 100) + with self.assertRaises(TypeError): + _zstd.train_dict(bytearray(), (), 100) + with self.assertRaises(TypeError): + _zstd.train_dict(b'', 99, 100) + with self.assertRaises(TypeError): + _zstd.train_dict(b'', [], 100) + with self.assertRaises(TypeError): + _zstd.train_dict(b'', (), 100.1) + with self.assertRaises(TypeError): + _zstd.train_dict(b'', (99.1,), 100) + with self.assertRaises(ValueError): + _zstd.train_dict(b'abc', (4, -1), 100) + with self.assertRaises(ValueError): + _zstd.train_dict(b'abc', (2,), 100) + with self.assertRaises(ValueError): + _zstd.train_dict(b'', (99,), 100) + + # size > size_t + with self.assertRaises(ValueError): + _zstd.train_dict(b'', (2**1000,), 100) + with self.assertRaises(ValueError): + _zstd.train_dict(b'', (-2**1000,), 100) + + # dict_size <= 0 + with self.assertRaises(ValueError): + _zstd.train_dict(b'', (), 0) + with self.assertRaises(ValueError): + _zstd.train_dict(b'', (), -1) + + with self.assertRaises(ZstdError): + _zstd.train_dict(b'', (), 1) + + def test_finalize_dict_c(self): + with self.assertRaises(TypeError): + _zstd.finalize_dict(1, 2, 3, 4, 5) + + # argument wrong type + with self.assertRaises(TypeError): + _zstd.finalize_dict({}, b'', (), 100, 5) + with self.assertRaises(TypeError): + _zstd.finalize_dict(bytearray(TRAINED_DICT.dict_content), b'', (), 100, 5) + with self.assertRaises(TypeError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, {}, (), 100, 5) + with self.assertRaises(TypeError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, bytearray(), (), 100, 5) + with self.assertRaises(TypeError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, b'', 99, 100, 5) + with self.assertRaises(TypeError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, b'', [], 100, 5) + with self.assertRaises(TypeError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, b'', (), 100.1, 5) + with self.assertRaises(TypeError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, b'', (), 100, 5.1) + + with self.assertRaises(ValueError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, b'abc', (4, -1), 100, 5) + with self.assertRaises(ValueError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, b'abc', (2,), 100, 5) + with self.assertRaises(ValueError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, b'', (99,), 100, 5) + + # size > size_t + with self.assertRaises(ValueError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, b'', (2**1000,), 100, 5) + with self.assertRaises(ValueError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, b'', (-2**1000,), 100, 5) + + # dict_size <= 0 + with self.assertRaises(ValueError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, b'', (), 0, 5) + with self.assertRaises(ValueError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, b'', (), -1, 5) + with self.assertRaises(OverflowError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, b'', (), 2**1000, 5) + with self.assertRaises(OverflowError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, b'', (), -2**1000, 5) + + with self.assertRaises(OverflowError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, b'', (), 100, 2**1000) + with self.assertRaises(OverflowError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, b'', (), 100, -2**1000) + + with self.assertRaises(ZstdError): + _zstd.finalize_dict(TRAINED_DICT.dict_content, b'', (), 100, 5) + + def test_train_buffer_protocol_samples(self): + def _nbytes(dat): + if isinstance(dat, (bytes, bytearray)): + return len(dat) + return memoryview(dat).nbytes + + # prepare samples + chunk_lst = [] + wrong_size_lst = [] + correct_size_lst = [] + for _ in range(300): + arr = array.array('Q', [random.randint(0, 20) for i in range(20)]) + chunk_lst.append(arr) + correct_size_lst.append(_nbytes(arr)) + wrong_size_lst.append(len(arr)) + concatenation = b''.join(chunk_lst) + + # wrong size list + with self.assertRaisesRegex(ValueError, + "The samples size tuple doesn't match the concatenation's size"): + _zstd.train_dict(concatenation, tuple(wrong_size_lst), 100*_1K) + + # correct size list + _zstd.train_dict(concatenation, tuple(correct_size_lst), 3*_1K) + + # wrong size list + with self.assertRaisesRegex(ValueError, + "The samples size tuple doesn't match the concatenation's size"): + _zstd.finalize_dict(TRAINED_DICT.dict_content, + concatenation, tuple(wrong_size_lst), 300*_1K, 5) + + # correct size list + _zstd.finalize_dict(TRAINED_DICT.dict_content, + concatenation, tuple(correct_size_lst), 300*_1K, 5) + + def test_as_prefix(self): + # V1 + V1 = THIS_FILE_BYTES + zd = ZstdDict(V1, is_raw=True) + + # V2 + mid = len(V1) // 2 + V2 = V1[:mid] + \ + (b'a' if V1[mid] != int.from_bytes(b'a') else b'b') + \ + V1[mid+1:] + + # compress + dat = compress(V2, zstd_dict=zd.as_prefix) + self.assertEqual(get_frame_info(dat).dictionary_id, 0) + + # decompress + self.assertEqual(decompress(dat, zd.as_prefix), V2) + + # use wrong prefix + zd2 = ZstdDict(SAMPLES[0], is_raw=True) + try: + decompressed = decompress(dat, zd2.as_prefix) + except ZstdError: # expected + pass + else: + self.assertNotEqual(decompressed, V2) + + # read only attribute + with self.assertRaises(AttributeError): + zd.as_prefix = b'1234' + + def test_as_digested_dict(self): + zd = TRAINED_DICT + + # test .as_digested_dict + dat = compress(SAMPLES[0], zstd_dict=zd.as_digested_dict) + self.assertEqual(decompress(dat, zd.as_digested_dict), SAMPLES[0]) + with self.assertRaises(AttributeError): + zd.as_digested_dict = b'1234' + + # test .as_undigested_dict + dat = compress(SAMPLES[0], zstd_dict=zd.as_undigested_dict) + self.assertEqual(decompress(dat, zd.as_undigested_dict), SAMPLES[0]) + with self.assertRaises(AttributeError): + zd.as_undigested_dict = b'1234' + + def test_advanced_compression_parameters(self): + options = {CompressionParameter.compression_level: 6, + CompressionParameter.window_log: 20, + CompressionParameter.enable_long_distance_matching: 1} + + # automatically select + dat = compress(SAMPLES[0], options=options, zstd_dict=TRAINED_DICT) + self.assertEqual(decompress(dat, TRAINED_DICT), SAMPLES[0]) + + # explicitly select + dat = compress(SAMPLES[0], options=options, zstd_dict=TRAINED_DICT.as_digested_dict) + self.assertEqual(decompress(dat, TRAINED_DICT), SAMPLES[0]) + + def test_len(self): + self.assertEqual(len(TRAINED_DICT), len(TRAINED_DICT.dict_content)) + self.assertIn(str(len(TRAINED_DICT)), str(TRAINED_DICT)) + +class FileTestCase(unittest.TestCase): + def setUp(self): + self.DECOMPRESSED_42 = b'a'*42 + self.FRAME_42 = compress(self.DECOMPRESSED_42) + + def test_init(self): + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) as f: + pass + with ZstdFile(io.BytesIO(), "w") as f: + pass + with ZstdFile(io.BytesIO(), "x") as f: + pass + with ZstdFile(io.BytesIO(), "a") as f: + pass + + with ZstdFile(io.BytesIO(), "w", level=12) as f: + pass + with ZstdFile(io.BytesIO(), "w", options={CompressionParameter.checksum_flag:1}) as f: + pass + with ZstdFile(io.BytesIO(), "w", options={}) as f: + pass + with ZstdFile(io.BytesIO(), "w", level=20, zstd_dict=TRAINED_DICT) as f: + pass + + with ZstdFile(io.BytesIO(), "r", options={DecompressionParameter.window_log_max:25}) as f: + pass + with ZstdFile(io.BytesIO(), "r", options={}, zstd_dict=TRAINED_DICT) as f: + pass + + def test_init_with_PathLike_filename(self): + with tempfile.NamedTemporaryFile(delete=False) as tmp_f: + filename = pathlib.Path(tmp_f.name) + + with ZstdFile(filename, "a") as f: + f.write(DECOMPRESSED_100_PLUS_32KB) + with ZstdFile(filename) as f: + self.assertEqual(f.read(), DECOMPRESSED_100_PLUS_32KB) + + with ZstdFile(filename, "a") as f: + f.write(DECOMPRESSED_100_PLUS_32KB) + with ZstdFile(filename) as f: + self.assertEqual(f.read(), DECOMPRESSED_100_PLUS_32KB * 2) + + os.remove(filename) + + def test_init_with_filename(self): + with tempfile.NamedTemporaryFile(delete=False) as tmp_f: + filename = pathlib.Path(tmp_f.name) + + with ZstdFile(filename) as f: + pass + with ZstdFile(filename, "w") as f: + pass + with ZstdFile(filename, "a") as f: + pass + + os.remove(filename) + + def test_init_mode(self): + bi = io.BytesIO() + + with ZstdFile(bi, "r"): + pass + with ZstdFile(bi, "rb"): + pass + with ZstdFile(bi, "w"): + pass + with ZstdFile(bi, "wb"): + pass + with ZstdFile(bi, "a"): + pass + with ZstdFile(bi, "ab"): + pass + + def test_init_with_x_mode(self): + with tempfile.NamedTemporaryFile() as tmp_f: + filename = pathlib.Path(tmp_f.name) + + for mode in ("x", "xb"): + with ZstdFile(filename, mode): + pass + with self.assertRaises(FileExistsError): + with ZstdFile(filename, mode): + pass + os.remove(filename) + + def test_init_bad_mode(self): + with self.assertRaises(ValueError): + ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), (3, "x")) + with self.assertRaises(ValueError): + ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), "") + with self.assertRaises(ValueError): + ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), "xt") + with self.assertRaises(ValueError): + ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), "x+") + with self.assertRaises(ValueError): + ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), "rx") + with self.assertRaises(ValueError): + ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), "wx") + with self.assertRaises(ValueError): + ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), "rt") + with self.assertRaises(ValueError): + ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), "r+") + with self.assertRaises(ValueError): + ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), "wt") + with self.assertRaises(ValueError): + ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), "w+") + with self.assertRaises(ValueError): + ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), "rw") + + with self.assertRaisesRegex(TypeError, + r"not be a CompressionParameter"): + ZstdFile(io.BytesIO(), 'rb', + options={CompressionParameter.compression_level:5}) + with self.assertRaisesRegex(TypeError, + r"not be a DecompressionParameter"): + ZstdFile(io.BytesIO(), 'wb', + options={DecompressionParameter.window_log_max:21}) + + with self.assertRaises(TypeError): + ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), "r", level=12) + + def test_init_bad_check(self): + with self.assertRaises(TypeError): + ZstdFile(io.BytesIO(), "w", level='asd') + # CHECK_UNKNOWN and anything above CHECK_ID_MAX should be invalid. + with self.assertRaises(ValueError): + ZstdFile(io.BytesIO(), "w", options={999:9999}) + with self.assertRaises(ValueError): + ZstdFile(io.BytesIO(), "w", options={CompressionParameter.window_log:99}) + + with self.assertRaises(TypeError): + ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), "r", options=33) + + with self.assertRaises(OverflowError): + ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), + options={DecompressionParameter.window_log_max:2**31}) + + with self.assertRaises(ValueError): + ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), + options={444:333}) + + with self.assertRaises(TypeError): + ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), zstd_dict={1:2}) + + with self.assertRaises(TypeError): + ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), zstd_dict=b'dict123456') + + def test_init_close_fp(self): + # get a temp file name + with tempfile.NamedTemporaryFile(delete=False) as tmp_f: + tmp_f.write(DAT_130K_C) + filename = tmp_f.name + + with self.assertRaises(TypeError): + ZstdFile(filename, options={'a':'b'}) + + # for PyPy + gc.collect() + + os.remove(filename) + + def test_close(self): + with io.BytesIO(COMPRESSED_100_PLUS_32KB) as src: + f = ZstdFile(src) + f.close() + # ZstdFile.close() should not close the underlying file object. + self.assertFalse(src.closed) + # Try closing an already-closed ZstdFile. + f.close() + self.assertFalse(src.closed) + + # Test with a real file on disk, opened directly by ZstdFile. + with tempfile.NamedTemporaryFile(delete=False) as tmp_f: + filename = pathlib.Path(tmp_f.name) + + f = ZstdFile(filename) + fp = f._fp + f.close() + # Here, ZstdFile.close() *should* close the underlying file object. + self.assertTrue(fp.closed) + # Try closing an already-closed ZstdFile. + f.close() + + os.remove(filename) + + def test_closed(self): + f = ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) + try: + self.assertFalse(f.closed) + f.read() + self.assertFalse(f.closed) + finally: + f.close() + self.assertTrue(f.closed) + + f = ZstdFile(io.BytesIO(), "w") + try: + self.assertFalse(f.closed) + finally: + f.close() + self.assertTrue(f.closed) + + def test_fileno(self): + # 1 + f = ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) + try: + self.assertRaises(io.UnsupportedOperation, f.fileno) + finally: + f.close() + self.assertRaises(ValueError, f.fileno) + + # 2 + with tempfile.NamedTemporaryFile(delete=False) as tmp_f: + filename = pathlib.Path(tmp_f.name) + + f = ZstdFile(filename) + try: + self.assertEqual(f.fileno(), f._fp.fileno()) + self.assertIsInstance(f.fileno(), int) + finally: + f.close() + self.assertRaises(ValueError, f.fileno) + + os.remove(filename) + + # 3, no .fileno() method + class C: + def read(self, size=-1): + return b'123' + with ZstdFile(C(), 'rb') as f: + with self.assertRaisesRegex(AttributeError, r'fileno'): + f.fileno() + + def test_name(self): + # 1 + f = ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) + try: + with self.assertRaises(AttributeError): + f.name + finally: + f.close() + with self.assertRaises(ValueError): + f.name + + # 2 + with tempfile.NamedTemporaryFile(delete=False) as tmp_f: + filename = pathlib.Path(tmp_f.name) + + f = ZstdFile(filename) + try: + self.assertEqual(f.name, f._fp.name) + self.assertIsInstance(f.name, str) + finally: + f.close() + with self.assertRaises(ValueError): + f.name + + os.remove(filename) + + # 3, no .filename property + class C: + def read(self, size=-1): + return b'123' + with ZstdFile(C(), 'rb') as f: + with self.assertRaisesRegex(AttributeError, r'name'): + f.name + + def test_seekable(self): + f = ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) + try: + self.assertTrue(f.seekable()) + f.read() + self.assertTrue(f.seekable()) + finally: + f.close() + self.assertRaises(ValueError, f.seekable) + + f = ZstdFile(io.BytesIO(), "w") + try: + self.assertFalse(f.seekable()) + finally: + f.close() + self.assertRaises(ValueError, f.seekable) + + src = io.BytesIO(COMPRESSED_100_PLUS_32KB) + src.seekable = lambda: False + f = ZstdFile(src) + try: + self.assertFalse(f.seekable()) + finally: + f.close() + self.assertRaises(ValueError, f.seekable) + + def test_readable(self): + f = ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) + try: + self.assertTrue(f.readable()) + f.read() + self.assertTrue(f.readable()) + finally: + f.close() + self.assertRaises(ValueError, f.readable) + + f = ZstdFile(io.BytesIO(), "w") + try: + self.assertFalse(f.readable()) + finally: + f.close() + self.assertRaises(ValueError, f.readable) + + def test_writable(self): + f = ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) + try: + self.assertFalse(f.writable()) + f.read() + self.assertFalse(f.writable()) + finally: + f.close() + self.assertRaises(ValueError, f.writable) + + f = ZstdFile(io.BytesIO(), "w") + try: + self.assertTrue(f.writable()) + finally: + f.close() + self.assertRaises(ValueError, f.writable) + + def test_read_0(self): + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) as f: + self.assertEqual(f.read(0), b"") + self.assertEqual(f.read(), DECOMPRESSED_100_PLUS_32KB) + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), + options={DecompressionParameter.window_log_max:20}) as f: + self.assertEqual(f.read(0), b"") + + # empty file + with ZstdFile(io.BytesIO(b'')) as f: + self.assertEqual(f.read(0), b"") + with self.assertRaises(EOFError): + f.read(10) + + with ZstdFile(io.BytesIO(b'')) as f: + with self.assertRaises(EOFError): + f.read(10) + + def test_read_10(self): + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) as f: + chunks = [] + while True: + result = f.read(10) + if not result: + break + self.assertLessEqual(len(result), 10) + chunks.append(result) + self.assertEqual(b"".join(chunks), DECOMPRESSED_100_PLUS_32KB) + + def test_read_multistream(self): + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB * 5)) as f: + self.assertEqual(f.read(), DECOMPRESSED_100_PLUS_32KB * 5) + + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB + SKIPPABLE_FRAME)) as f: + self.assertEqual(f.read(), DECOMPRESSED_100_PLUS_32KB) + + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB + COMPRESSED_DAT)) as f: + self.assertEqual(f.read(), DECOMPRESSED_100_PLUS_32KB + DECOMPRESSED_DAT) + + def test_read_incomplete(self): + with ZstdFile(io.BytesIO(DAT_130K_C[:-200])) as f: + self.assertRaises(EOFError, f.read) + + # Trailing data isn't a valid compressed stream + with ZstdFile(io.BytesIO(self.FRAME_42 + b'12345')) as f: + self.assertRaises(ZstdError, f.read) + + with ZstdFile(io.BytesIO(SKIPPABLE_FRAME + b'12345')) as f: + self.assertRaises(ZstdError, f.read) + + def test_read_truncated(self): + # Drop stream epilogue: 4 bytes checksum + truncated = DAT_130K_C[:-4] + with ZstdFile(io.BytesIO(truncated)) as f: + self.assertRaises(EOFError, f.read) + + with ZstdFile(io.BytesIO(truncated)) as f: + # this is an important test, make sure it doesn't raise EOFError. + self.assertEqual(f.read(130*_1K), DAT_130K_D) + with self.assertRaises(EOFError): + f.read(1) + + # Incomplete header + for i in range(1, 20): + with ZstdFile(io.BytesIO(truncated[:i])) as f: + self.assertRaises(EOFError, f.read, 1) + + def test_read_bad_args(self): + f = ZstdFile(io.BytesIO(COMPRESSED_DAT)) + f.close() + self.assertRaises(ValueError, f.read) + with ZstdFile(io.BytesIO(), "w") as f: + self.assertRaises(ValueError, f.read) + with ZstdFile(io.BytesIO(COMPRESSED_DAT)) as f: + self.assertRaises(TypeError, f.read, float()) + + def test_read_bad_data(self): + with ZstdFile(io.BytesIO(COMPRESSED_BOGUS)) as f: + self.assertRaises(ZstdError, f.read) + + def test_read_exception(self): + class C: + def read(self, size=-1): + raise OSError + with ZstdFile(C()) as f: + with self.assertRaises(OSError): + f.read(10) + + def test_read1(self): + with ZstdFile(io.BytesIO(DAT_130K_C)) as f: + blocks = [] + while True: + result = f.read1() + if not result: + break + blocks.append(result) + self.assertEqual(b"".join(blocks), DAT_130K_D) + self.assertEqual(f.read1(), b"") + + def test_read1_0(self): + with ZstdFile(io.BytesIO(COMPRESSED_DAT)) as f: + self.assertEqual(f.read1(0), b"") + + def test_read1_10(self): + with ZstdFile(io.BytesIO(COMPRESSED_DAT)) as f: + blocks = [] + while True: + result = f.read1(10) + if not result: + break + blocks.append(result) + self.assertEqual(b"".join(blocks), DECOMPRESSED_DAT) + self.assertEqual(f.read1(), b"") + + def test_read1_multistream(self): + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB * 5)) as f: + blocks = [] + while True: + result = f.read1() + if not result: + break + blocks.append(result) + self.assertEqual(b"".join(blocks), DECOMPRESSED_100_PLUS_32KB * 5) + self.assertEqual(f.read1(), b"") + + def test_read1_bad_args(self): + f = ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) + f.close() + self.assertRaises(ValueError, f.read1) + with ZstdFile(io.BytesIO(), "w") as f: + self.assertRaises(ValueError, f.read1) + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) as f: + self.assertRaises(TypeError, f.read1, None) + + def test_readinto(self): + arr = array.array("I", range(100)) + self.assertEqual(len(arr), 100) + self.assertEqual(len(arr) * arr.itemsize, 400) + ba = bytearray(300) + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) as f: + # 0 length output buffer + self.assertEqual(f.readinto(ba[0:0]), 0) + + # use correct length for buffer protocol object + self.assertEqual(f.readinto(arr), 400) + self.assertEqual(arr.tobytes(), DECOMPRESSED_100_PLUS_32KB[:400]) + + # normal readinto + self.assertEqual(f.readinto(ba), 300) + self.assertEqual(ba, DECOMPRESSED_100_PLUS_32KB[400:700]) + + def test_peek(self): + with ZstdFile(io.BytesIO(DAT_130K_C)) as f: + result = f.peek() + self.assertGreater(len(result), 0) + self.assertTrue(DAT_130K_D.startswith(result)) + self.assertEqual(f.read(), DAT_130K_D) + with ZstdFile(io.BytesIO(DAT_130K_C)) as f: + result = f.peek(10) + self.assertGreater(len(result), 0) + self.assertTrue(DAT_130K_D.startswith(result)) + self.assertEqual(f.read(), DAT_130K_D) + + def test_peek_bad_args(self): + with ZstdFile(io.BytesIO(), "w") as f: + self.assertRaises(ValueError, f.peek) + + def test_iterator(self): + with io.BytesIO(THIS_FILE_BYTES) as f: + lines = f.readlines() + compressed = compress(THIS_FILE_BYTES) + + # iter + with ZstdFile(io.BytesIO(compressed)) as f: + self.assertListEqual(list(iter(f)), lines) + + # readline + with ZstdFile(io.BytesIO(compressed)) as f: + for line in lines: + self.assertEqual(f.readline(), line) + self.assertEqual(f.readline(), b'') + self.assertEqual(f.readline(), b'') + + # readlines + with ZstdFile(io.BytesIO(compressed)) as f: + self.assertListEqual(f.readlines(), lines) + + def test_decompress_limited(self): + _ZSTD_DStreamInSize = 128*_1K + 3 + + bomb = compress(b'\0' * int(2e6), level=10) + self.assertLess(len(bomb), _ZSTD_DStreamInSize) + + decomp = ZstdFile(io.BytesIO(bomb)) + self.assertEqual(decomp.read(1), b'\0') + + # BufferedReader uses 128 KiB buffer in __init__.py + max_decomp = 128*_1K + self.assertLessEqual(decomp._buffer.raw.tell(), max_decomp, + "Excessive amount of data was decompressed") + + def test_write(self): + raw_data = THIS_FILE_BYTES[: len(THIS_FILE_BYTES) // 6] + with io.BytesIO() as dst: + with ZstdFile(dst, "w") as f: + f.write(raw_data) + + comp = ZstdCompressor() + expected = comp.compress(raw_data) + comp.flush() + self.assertEqual(dst.getvalue(), expected) + + with io.BytesIO() as dst: + with ZstdFile(dst, "w", level=12) as f: + f.write(raw_data) + + comp = ZstdCompressor(12) + expected = comp.compress(raw_data) + comp.flush() + self.assertEqual(dst.getvalue(), expected) + + with io.BytesIO() as dst: + with ZstdFile(dst, "w", options={CompressionParameter.checksum_flag:1}) as f: + f.write(raw_data) + + comp = ZstdCompressor(options={CompressionParameter.checksum_flag:1}) + expected = comp.compress(raw_data) + comp.flush() + self.assertEqual(dst.getvalue(), expected) + + with io.BytesIO() as dst: + options = {CompressionParameter.compression_level:-5, + CompressionParameter.checksum_flag:1} + with ZstdFile(dst, "w", + options=options) as f: + f.write(raw_data) + + comp = ZstdCompressor(options=options) + expected = comp.compress(raw_data) + comp.flush() + self.assertEqual(dst.getvalue(), expected) + + def test_write_empty_frame(self): + # .FLUSH_FRAME generates an empty content frame + c = ZstdCompressor() + self.assertNotEqual(c.flush(c.FLUSH_FRAME), b'') + self.assertNotEqual(c.flush(c.FLUSH_FRAME), b'') + + # don't generate empty content frame + bo = io.BytesIO() + with ZstdFile(bo, 'w') as f: + pass + self.assertEqual(bo.getvalue(), b'') + + bo = io.BytesIO() + with ZstdFile(bo, 'w') as f: + f.flush(f.FLUSH_FRAME) + self.assertEqual(bo.getvalue(), b'') + + # if .write(b''), generate empty content frame + bo = io.BytesIO() + with ZstdFile(bo, 'w') as f: + f.write(b'') + self.assertNotEqual(bo.getvalue(), b'') + + # has an empty content frame + bo = io.BytesIO() + with ZstdFile(bo, 'w') as f: + f.flush(f.FLUSH_BLOCK) + self.assertNotEqual(bo.getvalue(), b'') + + def test_write_empty_block(self): + # If no internal data, .FLUSH_BLOCK return b''. + c = ZstdCompressor() + self.assertEqual(c.flush(c.FLUSH_BLOCK), b'') + self.assertNotEqual(c.compress(b'123', c.FLUSH_BLOCK), + b'') + self.assertEqual(c.flush(c.FLUSH_BLOCK), b'') + self.assertEqual(c.compress(b''), b'') + self.assertEqual(c.compress(b''), b'') + self.assertEqual(c.flush(c.FLUSH_BLOCK), b'') + + # mode = .last_mode + bo = io.BytesIO() + with ZstdFile(bo, 'w') as f: + f.write(b'123') + f.flush(f.FLUSH_BLOCK) + fp_pos = f._fp.tell() + self.assertNotEqual(fp_pos, 0) + f.flush(f.FLUSH_BLOCK) + self.assertEqual(f._fp.tell(), fp_pos) + + # mode != .last_mode + bo = io.BytesIO() + with ZstdFile(bo, 'w') as f: + f.flush(f.FLUSH_BLOCK) + self.assertEqual(f._fp.tell(), 0) + f.write(b'') + f.flush(f.FLUSH_BLOCK) + self.assertEqual(f._fp.tell(), 0) + + def test_write_101(self): + with io.BytesIO() as dst: + with ZstdFile(dst, "w") as f: + for start in range(0, len(THIS_FILE_BYTES), 101): + f.write(THIS_FILE_BYTES[start:start+101]) + + comp = ZstdCompressor() + expected = comp.compress(THIS_FILE_BYTES) + comp.flush() + self.assertEqual(dst.getvalue(), expected) + + def test_write_append(self): + def comp(data): + comp = ZstdCompressor() + return comp.compress(data) + comp.flush() + + part1 = THIS_FILE_BYTES[:_1K] + part2 = THIS_FILE_BYTES[_1K:1536] + part3 = THIS_FILE_BYTES[1536:] + expected = b"".join(comp(x) for x in (part1, part2, part3)) + with io.BytesIO() as dst: + with ZstdFile(dst, "w") as f: + f.write(part1) + with ZstdFile(dst, "a") as f: + f.write(part2) + with ZstdFile(dst, "a") as f: + f.write(part3) + self.assertEqual(dst.getvalue(), expected) + + def test_write_bad_args(self): + f = ZstdFile(io.BytesIO(), "w") + f.close() + self.assertRaises(ValueError, f.write, b"foo") + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB), "r") as f: + self.assertRaises(ValueError, f.write, b"bar") + with ZstdFile(io.BytesIO(), "w") as f: + self.assertRaises(TypeError, f.write, None) + self.assertRaises(TypeError, f.write, "text") + self.assertRaises(TypeError, f.write, 789) + + def test_writelines(self): + def comp(data): + comp = ZstdCompressor() + return comp.compress(data) + comp.flush() + + with io.BytesIO(THIS_FILE_BYTES) as f: + lines = f.readlines() + with io.BytesIO() as dst: + with ZstdFile(dst, "w") as f: + f.writelines(lines) + expected = comp(THIS_FILE_BYTES) + self.assertEqual(dst.getvalue(), expected) + + def test_seek_forward(self): + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) as f: + f.seek(555) + self.assertEqual(f.read(), DECOMPRESSED_100_PLUS_32KB[555:]) + + def test_seek_forward_across_streams(self): + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB * 2)) as f: + f.seek(len(DECOMPRESSED_100_PLUS_32KB) + 123) + self.assertEqual(f.read(), DECOMPRESSED_100_PLUS_32KB[123:]) + + def test_seek_forward_relative_to_current(self): + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) as f: + f.read(100) + f.seek(1236, 1) + self.assertEqual(f.read(), DECOMPRESSED_100_PLUS_32KB[1336:]) + + def test_seek_forward_relative_to_end(self): + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) as f: + f.seek(-555, 2) + self.assertEqual(f.read(), DECOMPRESSED_100_PLUS_32KB[-555:]) + + def test_seek_backward(self): + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) as f: + f.read(1001) + f.seek(211) + self.assertEqual(f.read(), DECOMPRESSED_100_PLUS_32KB[211:]) + + def test_seek_backward_across_streams(self): + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB * 2)) as f: + f.read(len(DECOMPRESSED_100_PLUS_32KB) + 333) + f.seek(737) + self.assertEqual(f.read(), + DECOMPRESSED_100_PLUS_32KB[737:] + DECOMPRESSED_100_PLUS_32KB) + + def test_seek_backward_relative_to_end(self): + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) as f: + f.seek(-150, 2) + self.assertEqual(f.read(), DECOMPRESSED_100_PLUS_32KB[-150:]) + + def test_seek_past_end(self): + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) as f: + f.seek(len(DECOMPRESSED_100_PLUS_32KB) + 9001) + self.assertEqual(f.tell(), len(DECOMPRESSED_100_PLUS_32KB)) + self.assertEqual(f.read(), b"") + + def test_seek_past_start(self): + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) as f: + f.seek(-88) + self.assertEqual(f.tell(), 0) + self.assertEqual(f.read(), DECOMPRESSED_100_PLUS_32KB) + + def test_seek_bad_args(self): + f = ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) + f.close() + self.assertRaises(ValueError, f.seek, 0) + with ZstdFile(io.BytesIO(), "w") as f: + self.assertRaises(ValueError, f.seek, 0) + with ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) as f: + self.assertRaises(ValueError, f.seek, 0, 3) + # io.BufferedReader raises TypeError instead of ValueError + self.assertRaises((TypeError, ValueError), f.seek, 9, ()) + self.assertRaises(TypeError, f.seek, None) + self.assertRaises(TypeError, f.seek, b"derp") + + def test_seek_not_seekable(self): + class C(io.BytesIO): + def seekable(self): + return False + obj = C(COMPRESSED_100_PLUS_32KB) + with ZstdFile(obj, 'r') as f: + d = f.read(1) + self.assertFalse(f.seekable()) + with self.assertRaisesRegex(io.UnsupportedOperation, + 'File or stream is not seekable'): + f.seek(0) + d += f.read() + self.assertEqual(d, DECOMPRESSED_100_PLUS_32KB) + + def test_tell(self): + with ZstdFile(io.BytesIO(DAT_130K_C)) as f: + pos = 0 + while True: + self.assertEqual(f.tell(), pos) + result = f.read(random.randint(171, 189)) + if not result: + break + pos += len(result) + self.assertEqual(f.tell(), len(DAT_130K_D)) + with ZstdFile(io.BytesIO(), "w") as f: + for pos in range(0, len(DAT_130K_D), 143): + self.assertEqual(f.tell(), pos) + f.write(DAT_130K_D[pos:pos+143]) + self.assertEqual(f.tell(), len(DAT_130K_D)) + + def test_tell_bad_args(self): + f = ZstdFile(io.BytesIO(COMPRESSED_100_PLUS_32KB)) + f.close() + self.assertRaises(ValueError, f.tell) + + def test_file_dict(self): + # default + bi = io.BytesIO() + with ZstdFile(bi, 'w', zstd_dict=TRAINED_DICT) as f: + f.write(SAMPLES[0]) + bi.seek(0) + with ZstdFile(bi, zstd_dict=TRAINED_DICT) as f: + dat = f.read() + self.assertEqual(dat, SAMPLES[0]) + + # .as_(un)digested_dict + bi = io.BytesIO() + with ZstdFile(bi, 'w', zstd_dict=TRAINED_DICT.as_digested_dict) as f: + f.write(SAMPLES[0]) + bi.seek(0) + with ZstdFile(bi, zstd_dict=TRAINED_DICT.as_undigested_dict) as f: + dat = f.read() + self.assertEqual(dat, SAMPLES[0]) + + def test_file_prefix(self): + bi = io.BytesIO() + with ZstdFile(bi, 'w', zstd_dict=TRAINED_DICT.as_prefix) as f: + f.write(SAMPLES[0]) + bi.seek(0) + with ZstdFile(bi, zstd_dict=TRAINED_DICT.as_prefix) as f: + dat = f.read() + self.assertEqual(dat, SAMPLES[0]) + + def test_UnsupportedOperation(self): + # 1 + with ZstdFile(io.BytesIO(), 'r') as f: + with self.assertRaises(io.UnsupportedOperation): + f.write(b'1234') + + # 2 + class T: + def read(self, size): + return b'a' * size + + with self.assertRaises(TypeError): # on creation + with ZstdFile(T(), 'w') as f: + pass + + # 3 + with ZstdFile(io.BytesIO(), 'w') as f: + with self.assertRaises(io.UnsupportedOperation): + f.read(100) + with self.assertRaises(io.UnsupportedOperation): + f.seek(100) + self.assertEqual(f.closed, True) + with self.assertRaises(ValueError): + f.readable() + with self.assertRaises(ValueError): + f.tell() + with self.assertRaises(ValueError): + f.read(100) + + def test_read_readinto_readinto1(self): + lst = [] + with ZstdFile(io.BytesIO(COMPRESSED_THIS_FILE*5)) as f: + while True: + method = random.randint(0, 2) + size = random.randint(0, 300) + + if method == 0: + dat = f.read(size) + if not dat and size: + break + lst.append(dat) + elif method == 1: + ba = bytearray(size) + read_size = f.readinto(ba) + if read_size == 0 and size: + break + lst.append(bytes(ba[:read_size])) + elif method == 2: + ba = bytearray(size) + read_size = f.readinto1(ba) + if read_size == 0 and size: + break + lst.append(bytes(ba[:read_size])) + self.assertEqual(b''.join(lst), THIS_FILE_BYTES*5) + + def test_zstdfile_flush(self): + # closed + f = ZstdFile(io.BytesIO(), 'w') + f.close() + with self.assertRaises(ValueError): + f.flush() + + # read + with ZstdFile(io.BytesIO(), 'r') as f: + # does nothing for read-only stream + f.flush() + + # write + DAT = b'abcd' + bi = io.BytesIO() + with ZstdFile(bi, 'w') as f: + self.assertEqual(f.write(DAT), len(DAT)) + self.assertEqual(f.tell(), len(DAT)) + self.assertEqual(bi.tell(), 0) # not enough for a block + + self.assertEqual(f.flush(), None) + self.assertEqual(f.tell(), len(DAT)) + self.assertGreater(bi.tell(), 0) # flushed + + # write, no .flush() method + class C: + def write(self, b): + return len(b) + with ZstdFile(C(), 'w') as f: + self.assertEqual(f.write(DAT), len(DAT)) + self.assertEqual(f.tell(), len(DAT)) + + self.assertEqual(f.flush(), None) + self.assertEqual(f.tell(), len(DAT)) + + def test_zstdfile_flush_mode(self): + self.assertEqual(ZstdFile.FLUSH_BLOCK, ZstdCompressor.FLUSH_BLOCK) + self.assertEqual(ZstdFile.FLUSH_FRAME, ZstdCompressor.FLUSH_FRAME) + with self.assertRaises(AttributeError): + ZstdFile.CONTINUE + + bo = io.BytesIO() + with ZstdFile(bo, 'w') as f: + # flush block + self.assertEqual(f.write(b'123'), 3) + self.assertIsNone(f.flush(f.FLUSH_BLOCK)) + p1 = bo.tell() + # mode == .last_mode, should return + self.assertIsNone(f.flush()) + p2 = bo.tell() + self.assertEqual(p1, p2) + # flush frame + self.assertEqual(f.write(b'456'), 3) + self.assertIsNone(f.flush(mode=f.FLUSH_FRAME)) + # flush frame + self.assertEqual(f.write(b'789'), 3) + self.assertIsNone(f.flush(f.FLUSH_FRAME)) + p1 = bo.tell() + # mode == .last_mode, should return + self.assertIsNone(f.flush(f.FLUSH_FRAME)) + p2 = bo.tell() + self.assertEqual(p1, p2) + self.assertEqual(decompress(bo.getvalue()), b'123456789') + + bo = io.BytesIO() + with ZstdFile(bo, 'w') as f: + f.write(b'123') + with self.assertRaisesRegex(ValueError, r'\.FLUSH_.*?\.FLUSH_'): + f.flush(ZstdCompressor.CONTINUE) + with self.assertRaises(ValueError): + f.flush(-1) + with self.assertRaises(ValueError): + f.flush(123456) + with self.assertRaises(TypeError): + f.flush(node=ZstdCompressor.CONTINUE) + with self.assertRaises((TypeError, ValueError)): + f.flush('FLUSH_FRAME') + with self.assertRaises(TypeError): + f.flush(b'456', f.FLUSH_BLOCK) + + def test_zstdfile_truncate(self): + with ZstdFile(io.BytesIO(), 'w') as f: + with self.assertRaises(io.UnsupportedOperation): + f.truncate(200) + + def test_zstdfile_iter_issue45475(self): + lines = [l for l in ZstdFile(io.BytesIO(COMPRESSED_THIS_FILE))] + self.assertGreater(len(lines), 0) + + def test_append_new_file(self): + with tempfile.NamedTemporaryFile(delete=True) as tmp_f: + filename = tmp_f.name + + with ZstdFile(filename, 'a') as f: + pass + self.assertTrue(os.path.isfile(filename)) + + os.remove(filename) + +class OpenTestCase(unittest.TestCase): + + def test_binary_modes(self): + with open(io.BytesIO(COMPRESSED_100_PLUS_32KB), "rb") as f: + self.assertEqual(f.read(), DECOMPRESSED_100_PLUS_32KB) + with io.BytesIO() as bio: + with open(bio, "wb") as f: + f.write(DECOMPRESSED_100_PLUS_32KB) + file_data = decompress(bio.getvalue()) + self.assertEqual(file_data, DECOMPRESSED_100_PLUS_32KB) + with open(bio, "ab") as f: + f.write(DECOMPRESSED_100_PLUS_32KB) + file_data = decompress(bio.getvalue()) + self.assertEqual(file_data, DECOMPRESSED_100_PLUS_32KB * 2) + + def test_text_modes(self): + # empty input + with self.assertRaises(EOFError): + with open(io.BytesIO(b''), "rt", encoding="utf-8", newline='\n') as reader: + for _ in reader: + pass + + # read + uncompressed = THIS_FILE_STR.replace(os.linesep, "\n") + with open(io.BytesIO(COMPRESSED_THIS_FILE), "rt", encoding="utf-8") as f: + self.assertEqual(f.read(), uncompressed) + + with io.BytesIO() as bio: + # write + with open(bio, "wt", encoding="utf-8") as f: + f.write(uncompressed) + file_data = decompress(bio.getvalue()).decode("utf-8") + self.assertEqual(file_data.replace(os.linesep, "\n"), uncompressed) + # append + with open(bio, "at", encoding="utf-8") as f: + f.write(uncompressed) + file_data = decompress(bio.getvalue()).decode("utf-8") + self.assertEqual(file_data.replace(os.linesep, "\n"), uncompressed * 2) + + def test_bad_params(self): + with tempfile.NamedTemporaryFile(delete=False) as tmp_f: + TESTFN = pathlib.Path(tmp_f.name) + + with self.assertRaises(ValueError): + open(TESTFN, "") + with self.assertRaises(ValueError): + open(TESTFN, "rbt") + with self.assertRaises(ValueError): + open(TESTFN, "rb", encoding="utf-8") + with self.assertRaises(ValueError): + open(TESTFN, "rb", errors="ignore") + with self.assertRaises(ValueError): + open(TESTFN, "rb", newline="\n") + + os.remove(TESTFN) + + def test_option(self): + options = {DecompressionParameter.window_log_max:25} + with open(io.BytesIO(COMPRESSED_100_PLUS_32KB), "rb", options=options) as f: + self.assertEqual(f.read(), DECOMPRESSED_100_PLUS_32KB) + + options = {CompressionParameter.compression_level:12} + with io.BytesIO() as bio: + with open(bio, "wb", options=options) as f: + f.write(DECOMPRESSED_100_PLUS_32KB) + file_data = decompress(bio.getvalue()) + self.assertEqual(file_data, DECOMPRESSED_100_PLUS_32KB) + + def test_encoding(self): + uncompressed = THIS_FILE_STR.replace(os.linesep, "\n") + + with io.BytesIO() as bio: + with open(bio, "wt", encoding="utf-16-le") as f: + f.write(uncompressed) + file_data = decompress(bio.getvalue()).decode("utf-16-le") + self.assertEqual(file_data.replace(os.linesep, "\n"), uncompressed) + bio.seek(0) + with open(bio, "rt", encoding="utf-16-le") as f: + self.assertEqual(f.read().replace(os.linesep, "\n"), uncompressed) + + def test_encoding_error_handler(self): + with io.BytesIO(compress(b"foo\xffbar")) as bio: + with open(bio, "rt", encoding="ascii", errors="ignore") as f: + self.assertEqual(f.read(), "foobar") + + def test_newline(self): + # Test with explicit newline (universal newline mode disabled). + text = THIS_FILE_STR.replace(os.linesep, "\n") + with io.BytesIO() as bio: + with open(bio, "wt", encoding="utf-8", newline="\n") as f: + f.write(text) + bio.seek(0) + with open(bio, "rt", encoding="utf-8", newline="\r") as f: + self.assertEqual(f.readlines(), [text]) + + def test_x_mode(self): + with tempfile.NamedTemporaryFile(delete=False) as tmp_f: + TESTFN = pathlib.Path(tmp_f.name) + + for mode in ("x", "xb", "xt"): + os.remove(TESTFN) + + if mode == "xt": + encoding = "utf-8" + else: + encoding = None + with open(TESTFN, mode, encoding=encoding): + pass + with self.assertRaises(FileExistsError): + with open(TESTFN, mode): + pass + + os.remove(TESTFN) + + def test_open_dict(self): + # default + bi = io.BytesIO() + with open(bi, 'w', zstd_dict=TRAINED_DICT) as f: + f.write(SAMPLES[0]) + bi.seek(0) + with open(bi, zstd_dict=TRAINED_DICT) as f: + dat = f.read() + self.assertEqual(dat, SAMPLES[0]) + + # .as_(un)digested_dict + bi = io.BytesIO() + with open(bi, 'w', zstd_dict=TRAINED_DICT.as_digested_dict) as f: + f.write(SAMPLES[0]) + bi.seek(0) + with open(bi, zstd_dict=TRAINED_DICT.as_undigested_dict) as f: + dat = f.read() + self.assertEqual(dat, SAMPLES[0]) + + # invalid dictionary + bi = io.BytesIO() + with self.assertRaisesRegex(TypeError, 'zstd_dict'): + open(bi, 'w', zstd_dict={1:2, 2:3}) + + with self.assertRaisesRegex(TypeError, 'zstd_dict'): + open(bi, 'w', zstd_dict=b'1234567890') + + def test_open_prefix(self): + bi = io.BytesIO() + with open(bi, 'w', zstd_dict=TRAINED_DICT.as_prefix) as f: + f.write(SAMPLES[0]) + bi.seek(0) + with open(bi, zstd_dict=TRAINED_DICT.as_prefix) as f: + dat = f.read() + self.assertEqual(dat, SAMPLES[0]) + + def test_buffer_protocol(self): + # don't use len() for buffer protocol objects + arr = array.array("i", range(1000)) + LENGTH = len(arr) * arr.itemsize + + with open(io.BytesIO(), "wb") as f: + self.assertEqual(f.write(arr), LENGTH) + self.assertEqual(f.tell(), LENGTH) + +class FreeThreadingMethodTests(unittest.TestCase): + + @threading_helper.reap_threads + @threading_helper.requires_working_threading() + def test_compress_locking(self): + input = b'a'* (16*_1K) + num_threads = 8 + + # gh-136394: the first output of .compress() includes the frame header + # we run the first .compress() call outside of the threaded portion + # to make the test order-independent + + comp = ZstdCompressor() + parts = [comp.compress(input, ZstdCompressor.FLUSH_BLOCK)] + for _ in range(num_threads): + res = comp.compress(input, ZstdCompressor.FLUSH_BLOCK) + if res: + parts.append(res) + rest1 = comp.flush() + expected = b''.join(parts) + rest1 + + comp = ZstdCompressor() + output = [comp.compress(input, ZstdCompressor.FLUSH_BLOCK)] + def run_method(method, input_data, output_data): + res = method(input_data, ZstdCompressor.FLUSH_BLOCK) + if res: + output_data.append(res) + threads = [] + + for i in range(num_threads): + thread = threading.Thread(target=run_method, args=(comp.compress, input, output)) + + threads.append(thread) + + with threading_helper.start_threads(threads): + pass + + rest2 = comp.flush() + self.assertEqual(rest1, rest2) + actual = b''.join(output) + rest2 + self.assertEqual(expected, actual) + + @threading_helper.reap_threads + @threading_helper.requires_working_threading() + def test_decompress_locking(self): + input = compress(b'a'* (16*_1K)) + num_threads = 8 + # to ensure we decompress over multiple calls, set maxsize + window_size = _1K * 16//num_threads + + decomp = ZstdDecompressor() + parts = [] + for _ in range(num_threads): + res = decomp.decompress(input, window_size) + if res: + parts.append(res) + expected = b''.join(parts) + + comp = ZstdDecompressor() + output = [] + def run_method(method, input_data, output_data): + res = method(input_data, window_size) + if res: + output_data.append(res) + threads = [] + + for i in range(num_threads): + thread = threading.Thread(target=run_method, args=(comp.decompress, input, output)) + + threads.append(thread) + + with threading_helper.start_threads(threads): + pass + + actual = b''.join(output) + self.assertEqual(expected, actual) + + @threading_helper.reap_threads + @threading_helper.requires_working_threading() + def test_compress_shared_dict(self): + num_threads = 8 + + def run_method(b): + level = threading.get_ident() % 4 + # sync threads to increase chance of contention on + # capsule storing dictionary levels + b.wait() + ZstdCompressor(level=level, + zstd_dict=TRAINED_DICT.as_digested_dict) + b.wait() + ZstdCompressor(level=level, + zstd_dict=TRAINED_DICT.as_undigested_dict) + b.wait() + ZstdCompressor(level=level, + zstd_dict=TRAINED_DICT.as_prefix) + threads = [] + + b = threading.Barrier(num_threads) + for i in range(num_threads): + thread = threading.Thread(target=run_method, args=(b,)) + + threads.append(thread) + + with threading_helper.start_threads(threads): + pass + + @threading_helper.reap_threads + @threading_helper.requires_working_threading() + def test_decompress_shared_dict(self): + num_threads = 8 + + def run_method(b): + # sync threads to increase chance of contention on + # decompression dictionary + b.wait() + ZstdDecompressor(zstd_dict=TRAINED_DICT.as_digested_dict) + b.wait() + ZstdDecompressor(zstd_dict=TRAINED_DICT.as_undigested_dict) + b.wait() + ZstdDecompressor(zstd_dict=TRAINED_DICT.as_prefix) + threads = [] + + b = threading.Barrier(num_threads) + for i in range(num_threads): + thread = threading.Thread(target=run_method, args=(b,)) + + threads.append(thread) + + with threading_helper.start_threads(threads): + pass + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/tokenizedata/__init__.py b/Lib/test/tokenizedata/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/Lib/test/tokenizedata/bad_coding.py b/Lib/test/tokenizedata/bad_coding.py new file mode 100644 index 00000000000..971b0a8f3d6 --- /dev/null +++ b/Lib/test/tokenizedata/bad_coding.py @@ -0,0 +1 @@ +# -*- coding: uft-8 -*- diff --git a/Lib/test/tokenizedata/bad_coding2.py b/Lib/test/tokenizedata/bad_coding2.py new file mode 100644 index 00000000000..bb2bb7e1e75 --- /dev/null +++ b/Lib/test/tokenizedata/bad_coding2.py @@ -0,0 +1,2 @@ +#coding: utf8 +print('我') diff --git a/Lib/test/tokenizedata/badsyntax_3131.py b/Lib/test/tokenizedata/badsyntax_3131.py new file mode 100644 index 00000000000..901d3744ca0 --- /dev/null +++ b/Lib/test/tokenizedata/badsyntax_3131.py @@ -0,0 +1,2 @@ +# -*- coding: utf-8 -*- +€ = 2 diff --git a/Lib/test/tokenizedata/badsyntax_pep3120.py b/Lib/test/tokenizedata/badsyntax_pep3120.py new file mode 100644 index 00000000000..d14b4c96ede --- /dev/null +++ b/Lib/test/tokenizedata/badsyntax_pep3120.py @@ -0,0 +1 @@ +print("bse") diff --git a/Lib/test/tokenizedata/coding20731.py b/Lib/test/tokenizedata/coding20731.py new file mode 100644 index 00000000000..b0e227ad110 --- /dev/null +++ b/Lib/test/tokenizedata/coding20731.py @@ -0,0 +1,4 @@ +#coding:latin1 + + + diff --git a/Lib/test/tokenizedata/tokenize_tests-latin1-coding-cookie-and-utf8-bom-sig.txt b/Lib/test/tokenizedata/tokenize_tests-latin1-coding-cookie-and-utf8-bom-sig.txt new file mode 100644 index 00000000000..1b5335b64ed --- /dev/null +++ b/Lib/test/tokenizedata/tokenize_tests-latin1-coding-cookie-and-utf8-bom-sig.txt @@ -0,0 +1,13 @@ +# -*- coding: latin1 -*- +# IMPORTANT: this file has the utf-8 BOM signature '\xef\xbb\xbf' +# at the start of it. Make sure this is preserved if any changes +# are made! Also note that the coding cookie above conflicts with +# the presence of a utf-8 BOM signature -- this is intended. + +# Arbitrary encoded utf-8 text (stolen from test_doctest2.py). +x = 'ЉЊЈЁЂ' +def y(): + """ + And again in a comment. ЉЊЈЁЂ + """ + pass diff --git a/Lib/test/tokenizedata/tokenize_tests-no-coding-cookie-and-utf8-bom-sig-only.txt b/Lib/test/tokenizedata/tokenize_tests-no-coding-cookie-and-utf8-bom-sig-only.txt new file mode 100644 index 00000000000..23fd2168ae5 --- /dev/null +++ b/Lib/test/tokenizedata/tokenize_tests-no-coding-cookie-and-utf8-bom-sig-only.txt @@ -0,0 +1,11 @@ +# IMPORTANT: this file has the utf-8 BOM signature '\xef\xbb\xbf' +# at the start of it. Make sure this is preserved if any changes +# are made! + +# Arbitrary encoded utf-8 text (stolen from test_doctest2.py). +x = 'ЉЊЈЁЂ' +def y(): + """ + And again in a comment. ЉЊЈЁЂ + """ + pass diff --git a/Lib/test/tokenizedata/tokenize_tests-utf8-coding-cookie-and-no-utf8-bom-sig.txt b/Lib/test/tokenizedata/tokenize_tests-utf8-coding-cookie-and-no-utf8-bom-sig.txt new file mode 100644 index 00000000000..04561e48472 --- /dev/null +++ b/Lib/test/tokenizedata/tokenize_tests-utf8-coding-cookie-and-no-utf8-bom-sig.txt @@ -0,0 +1,13 @@ +# -*- coding: utf-8 -*- +# IMPORTANT: unlike the other test_tokenize-*.txt files, this file +# does NOT have the utf-8 BOM signature '\xef\xbb\xbf' at the start +# of it. Make sure this is not added inadvertently by your editor +# if any changes are made to this file! + +# Arbitrary encoded utf-8 text (stolen from test_doctest2.py). +x = 'ЉЊЈЁЂ' +def y(): + """ + And again in a comment. ЉЊЈЁЂ + """ + pass diff --git a/Lib/test/tokenizedata/tokenize_tests-utf8-coding-cookie-and-utf8-bom-sig.txt b/Lib/test/tokenizedata/tokenize_tests-utf8-coding-cookie-and-utf8-bom-sig.txt new file mode 100644 index 00000000000..4b20ff6ad6d --- /dev/null +++ b/Lib/test/tokenizedata/tokenize_tests-utf8-coding-cookie-and-utf8-bom-sig.txt @@ -0,0 +1,12 @@ +# -*- coding: utf-8 -*- +# IMPORTANT: this file has the utf-8 BOM signature '\xef\xbb\xbf' +# at the start of it. Make sure this is preserved if any changes +# are made! + +# Arbitrary encoded utf-8 text (stolen from test_doctest2.py). +x = 'ЉЊЈЁЂ' +def y(): + """ + And again in a comment. ЉЊЈЁЂ + """ + pass diff --git a/Lib/test/tokenizedata/tokenize_tests.txt b/Lib/test/tokenizedata/tokenize_tests.txt new file mode 100644 index 00000000000..c4f5a58a946 --- /dev/null +++ b/Lib/test/tokenizedata/tokenize_tests.txt @@ -0,0 +1,189 @@ +# Tests for the 'tokenize' module. +# Large bits stolen from test_grammar.py. + +# Comments +"#" +#' +#" +#\ + # + # abc +'''# +#''' + +x = 1 # + +# Balancing continuation + +a = (3, 4, + 5, 6) +y = [3, 4, + 5] +z = {'a':5, + 'b':6} +x = (len(repr(y)) + 5*x - a[ + 3 ] + - x + len({ + } + ) + ) + +# Backslash means line continuation: +x = 1 \ ++ 1 + +# Backslash does not means continuation in comments :\ +x = 0 + +# Ordinary integers +0xff != 255 +0o377 != 255 +2147483647 != 0o17777777777 +-2147483647-1 != 0o20000000000 +0o37777777777 != -1 +0xffffffff != -1; 0o37777777777 != -1; -0o1234567 == 0O001234567; 0b10101 == 0B00010101 + +# Long integers +x = 0 +x = 0 +x = 0xffffffffffffffff +x = 0xffffffffffffffff +x = 0o77777777777777777 +x = 0B11101010111111111 +x = 123456789012345678901234567890 +x = 123456789012345678901234567890 + +# Floating-point numbers +x = 3.14 +x = 314. +x = 0.314 +# XXX x = 000.314 +x = .314 +x = 3e14 +x = 3E14 +x = 3e-14 +x = 3e+14 +x = 3.e14 +x = .3e14 +x = 3.1e4 + +# String literals +x = ''; y = ""; +x = '\''; y = "'"; +x = '"'; y = "\""; +x = "doesn't \"shrink\" does it" +y = 'doesn\'t "shrink" does it' +x = "does \"shrink\" doesn't it" +y = 'does "shrink" doesn\'t it' +x = """ +The "quick" +brown fox +jumps over +the 'lazy' dog. +""" +y = '\nThe "quick"\nbrown fox\njumps over\nthe \'lazy\' dog.\n' +y = ''' +The "quick" +brown fox +jumps over +the 'lazy' dog. +'''; +y = "\n\ +The \"quick\"\n\ +brown fox\n\ +jumps over\n\ +the 'lazy' dog.\n\ +"; +y = '\n\ +The \"quick\"\n\ +brown fox\n\ +jumps over\n\ +the \'lazy\' dog.\n\ +'; +x = r'\\' + R'\\' +x = r'\'' + '' +y = r''' +foo bar \\ +baz''' + R''' +foo''' +y = r"""foo +bar \\ baz +""" + R'''spam +''' +x = b'abc' + B'ABC' +y = b"abc" + B"ABC" +x = br'abc' + Br'ABC' + bR'ABC' + BR'ABC' +y = br"abc" + Br"ABC" + bR"ABC" + BR"ABC" +x = rb'abc' + rB'ABC' + Rb'ABC' + RB'ABC' +y = rb"abc" + rB"ABC" + Rb"ABC" + RB"ABC" +x = br'\\' + BR'\\' +x = rb'\\' + RB'\\' +x = br'\'' + '' +x = rb'\'' + '' +y = br''' +foo bar \\ +baz''' + BR''' +foo''' +y = Br"""foo +bar \\ baz +""" + bR'''spam +''' +y = rB"""foo +bar \\ baz +""" + Rb'''spam +''' + +# Indentation +if 1: + x = 2 +if 1: + x = 2 +if 1: + while 0: + if 0: + x = 2 + x = 2 +if 0: + if 2: + while 0: + if 1: + x = 2 + +# Operators + +def d22(a, b, c=1, d=2): pass +def d01v(a=1, *restt, **restd): pass + +(x, y) != ({'a':1}, {'b':2}) + +# comparison +if 1 < 1 > 1 == 1 >= 1 <= 1 != 1 != 1 in 1 not in 1 is 1 is not 1: pass + +# binary +x = 1 & 1 +x = 1 ^ 1 +x = 1 | 1 + +# shift +x = 1 << 1 >> 1 + +# additive +x = 1 - 1 + 1 - 1 + 1 + +# multiplicative +x = 1 / 1 * 1 % 1 + +# unary +x = ~1 ^ 1 & 1 | 1 & 1 ^ -1 +x = -1*1/1 + 1*1 - ---1*1 + +# selector +import sys, time +x = sys.modules['time'].time() + +@staticmethod +def foo(): pass + +@staticmethod +def foo(x:1)->1: pass + diff --git a/Lib/test/typinganndata/ann_module.py b/Lib/test/typinganndata/ann_module.py index 5081e6b5834..e1a1792cb4a 100644 --- a/Lib/test/typinganndata/ann_module.py +++ b/Lib/test/typinganndata/ann_module.py @@ -8,8 +8,6 @@ from typing import Optional from functools import wraps -__annotations__[1] = 2 - class C: x = 5; y: Optional['C'] = None @@ -18,8 +16,6 @@ class C: x: int = 5; y: str = x; f: Tuple[int, int] class M(type): - - __annotations__['123'] = 123 o: type = object (pars): bool = True diff --git a/Lib/test/zipimport_data/sparse-zip64-c0-0x000000000.part b/Lib/test/zipimport_data/sparse-zip64-c0-0x000000000.part new file mode 100644 index 00000000000..c6beae8e255 Binary files /dev/null and b/Lib/test/zipimport_data/sparse-zip64-c0-0x000000000.part differ diff --git a/Lib/test/zipimport_data/sparse-zip64-c0-0x100000000.part b/Lib/test/zipimport_data/sparse-zip64-c0-0x100000000.part new file mode 100644 index 00000000000..74ab03b4648 Binary files /dev/null and b/Lib/test/zipimport_data/sparse-zip64-c0-0x100000000.part differ diff --git a/Lib/test/zipimport_data/sparse-zip64-c0-0x200000000.part b/Lib/test/zipimport_data/sparse-zip64-c0-0x200000000.part new file mode 100644 index 00000000000..9769a404f67 Binary files /dev/null and b/Lib/test/zipimport_data/sparse-zip64-c0-0x200000000.part differ diff --git a/Lib/textwrap.py b/Lib/textwrap.py index 7ca393d1c37..41366fbf443 100644 --- a/Lib/textwrap.py +++ b/Lib/textwrap.py @@ -2,7 +2,7 @@ """ # Copyright (C) 1999-2001 Gregory P. Ward. -# Copyright (C) 2002, 2003 Python Software Foundation. +# Copyright (C) 2002 Python Software Foundation. # Written by Greg Ward <gward@python.net> import re @@ -86,7 +86,7 @@ class TextWrapper: -(?: (?<=%(lt)s{2}-) | (?<=%(lt)s-%(lt)s-)) (?= %(lt)s -? %(lt)s) | # end of word - (?=%(ws)s|\Z) + (?=%(ws)s|\z) | # em-dash (?<=%(wp)s) (?=-{2,}\w) ) @@ -107,7 +107,7 @@ class TextWrapper: sentence_end_re = re.compile(r'[a-z]' # lowercase letter r'[\.\!\?]' # sentence-ending punct. r'[\"\']?' # optional end-of-quote - r'\Z') # end of chunk + r'\z') # end of chunk def __init__(self, width=70, @@ -211,7 +211,7 @@ def _handle_long_word(self, reversed_chunks, cur_line, cur_len, width): # If we're allowed to break long words, then do so: put as much # of the next chunk onto the current line as will fit. - if self.break_long_words: + if self.break_long_words and space_left > 0: end = space_left chunk = reversed_chunks[-1] if self.break_on_hyphens and len(chunk) > space_left: @@ -413,9 +413,6 @@ def shorten(text, width, **kwargs): # -- Loosely related functionality ------------------------------------- -_whitespace_only_re = re.compile('^[ \t]+$', re.MULTILINE) -_leading_whitespace_re = re.compile('(^[ \t]*)(?:[^ \t\n])', re.MULTILINE) - def dedent(text): """Remove any common leading whitespace from every line in `text`. @@ -429,42 +426,22 @@ def dedent(text): Entirely blank lines are normalized to a newline character. """ - # Look for the longest leading string of spaces and tabs common to - # all lines. - margin = None - text = _whitespace_only_re.sub('', text) - indents = _leading_whitespace_re.findall(text) - for indent in indents: - if margin is None: - margin = indent - - # Current line more deeply indented than previous winner: - # no change (previous winner is still on top). - elif indent.startswith(margin): - pass - - # Current line consistent with and no deeper than previous winner: - # it's the new winner. - elif margin.startswith(indent): - margin = indent - - # Find the largest common whitespace between current line and previous - # winner. - else: - for i, (x, y) in enumerate(zip(margin, indent)): - if x != y: - margin = margin[:i] - break + try: + lines = text.split('\n') + except (AttributeError, TypeError): + msg = f'expected str object, not {type(text).__qualname__!r}' + raise TypeError(msg) from None - # sanity check (testing/debugging only) - if 0 and margin: - for line in text.split("\n"): - assert not line or line.startswith(margin), \ - "line = %r, margin = %r" % (line, margin) + # Get length of leading whitespace, inspired by ``os.path.commonprefix()``. + non_blank_lines = [l for l in lines if l and not l.isspace()] + l1 = min(non_blank_lines, default='') + l2 = max(non_blank_lines, default='') + margin = 0 + for margin, c in enumerate(l1): + if c != l2[margin] or c not in ' \t': + break - if margin: - text = re.sub(r'(?m)^' + margin, '', text) - return text + return '\n'.join([l[margin:] if not l.isspace() else '' for l in lines]) def indent(text, prefix, predicate=None): @@ -475,19 +452,20 @@ def indent(text, prefix, predicate=None): it will default to adding 'prefix' to all non-empty lines that do not consist solely of whitespace characters. """ - if predicate is None: - # str.splitlines(True) doesn't produce empty string. - # ''.splitlines(True) => [] - # 'foo\n'.splitlines(True) => ['foo\n'] - # So we can use just `not s.isspace()` here. - predicate = lambda s: not s.isspace() - prefixed_lines = [] - for line in text.splitlines(True): - if predicate(line): - prefixed_lines.append(prefix) - prefixed_lines.append(line) - + if predicate is None: + # str.splitlines(keepends=True) doesn't produce the empty string, + # so we need to use `str.isspace()` rather than a truth test. + # Inlining the predicate leads to a ~30% performance improvement. + for line in text.splitlines(True): + if not line.isspace(): + prefixed_lines.append(prefix) + prefixed_lines.append(line) + else: + for line in text.splitlines(True): + if predicate(line): + prefixed_lines.append(prefix) + prefixed_lines.append(line) return ''.join(prefixed_lines) diff --git a/Lib/threading.py b/Lib/threading.py index 811ea94a126..c03b0b5370c 100644 --- a/Lib/threading.py +++ b/Lib/threading.py @@ -3,11 +3,11 @@ import os as _os import sys as _sys import _thread -import functools +import _contextvars from time import monotonic as _time from _weakrefset import WeakSet -from itertools import islice as _islice, count as _count +from itertools import count as _count try: from _collections import deque as _deque except ImportError: @@ -28,19 +28,30 @@ 'Event', 'Lock', 'RLock', 'Semaphore', 'BoundedSemaphore', 'Thread', 'Barrier', 'BrokenBarrierError', 'Timer', 'ThreadError', 'setprofile', 'settrace', 'local', 'stack_size', - 'excepthook', 'ExceptHookArgs', 'gettrace', 'getprofile'] + 'excepthook', 'ExceptHookArgs', 'gettrace', 'getprofile', + 'setprofile_all_threads','settrace_all_threads'] # Rename some stuff so "from threading import *" is safe -_start_new_thread = _thread.start_new_thread +_start_joinable_thread = _thread.start_joinable_thread +_daemon_threads_allowed = _thread.daemon_threads_allowed _allocate_lock = _thread.allocate_lock -_set_sentinel = _thread._set_sentinel +_LockType = _thread.LockType +_thread_shutdown = _thread._shutdown +_make_thread_handle = _thread._make_thread_handle +_ThreadHandle = _thread._ThreadHandle get_ident = _thread.get_ident +_get_main_thread_ident = _thread._get_main_thread_ident +_is_main_interpreter = _thread._is_main_interpreter try: get_native_id = _thread.get_native_id _HAVE_THREAD_NATIVE_ID = True __all__.append('get_native_id') except AttributeError: _HAVE_THREAD_NATIVE_ID = False +try: + _set_name = _thread.set_name +except AttributeError: + _set_name = None ThreadError = _thread.error try: _CRLock = _thread.RLock @@ -49,6 +60,13 @@ TIMEOUT_MAX = _thread.TIMEOUT_MAX del _thread +# get thread-local implementation, either from the thread +# module, or from the python fallback + +try: + from _thread import _local as local +except ImportError: + from _threading_local import local # Support for profile and trace hooks @@ -60,11 +78,20 @@ def setprofile(func): The func will be passed to sys.setprofile() for each thread, before its run() method is called. - """ global _profile_hook _profile_hook = func +def setprofile_all_threads(func): + """Set a profile function for all threads started from the threading module + and all Python threads that are currently executing. + + The func will be passed to sys.setprofile() for each thread, before its + run() method is called. + """ + setprofile(func) + _sys._setprofileallthreads(func) + def getprofile(): """Get the profiler function as set by threading.setprofile().""" return _profile_hook @@ -74,12 +101,10 @@ def settrace(func): The func will be passed to sys.settrace() for each thread, before its run() method is called. - """ global _trace_hook _trace_hook = func -# TODO: RUSTPYTHON def settrace_all_threads(func): """Set a trace function for all threads started from the threading module and all Python threads that are currently executing. @@ -96,7 +121,7 @@ def gettrace(): # Synchronization classes -Lock = _allocate_lock +Lock = _LockType def RLock(*args, **kwargs): """Factory function that returns a new reentrant lock. @@ -107,6 +132,13 @@ def RLock(*args, **kwargs): acquired it. """ + if args or kwargs: + import warnings + warnings.warn( + 'Passing arguments to RLock is deprecated and will be removed in 3.15', + DeprecationWarning, + stacklevel=2, + ) if _CRLock is None: return _PyRLock(*args, **kwargs) return _CRLock(*args, **kwargs) @@ -133,7 +165,7 @@ def __repr__(self): except KeyError: pass return "<%s %s.%s object owner=%r count=%d at %s>" % ( - "locked" if self._block.locked() else "unlocked", + "locked" if self.locked() else "unlocked", self.__class__.__module__, self.__class__.__qualname__, owner, @@ -210,6 +242,10 @@ def release(self): def __exit__(self, t, v, tb): self.release() + def locked(self): + """Return whether this object is locked.""" + return self._block.locked() + # Internal methods used by condition variables def _acquire_restore(self, state): @@ -229,6 +265,13 @@ def _release_save(self): def _is_owned(self): return self._owner == get_ident() + # Internal method used for reentrancy checks + + def _recursion_count(self): + if self._owner != get_ident(): + return 0 + return self._count + _PyRLock = _RLock @@ -248,24 +291,19 @@ def __init__(self, lock=None): if lock is None: lock = RLock() self._lock = lock - # Export the lock's acquire() and release() methods + # Export the lock's acquire(), release(), and locked() methods self.acquire = lock.acquire self.release = lock.release + self.locked = lock.locked # If the lock defines _release_save() and/or _acquire_restore(), # these override the default implementations (which just call # release() and acquire() on the lock). Ditto for _is_owned(). - try: + if hasattr(lock, '_release_save'): self._release_save = lock._release_save - except AttributeError: - pass - try: + if hasattr(lock, '_acquire_restore'): self._acquire_restore = lock._acquire_restore - except AttributeError: - pass - try: + if hasattr(lock, '_is_owned'): self._is_owned = lock._is_owned - except AttributeError: - pass self._waiters = _deque() def _at_fork_reinit(self): @@ -308,7 +346,7 @@ def wait(self, timeout=None): awakened or timed out, it re-acquires the lock and returns. When the timeout argument is present and not None, it should be a - floating point number specifying a timeout for the operation in seconds + floating-point number specifying a timeout for the operation in seconds (or fractions thereof). When the underlying lock is an RLock, it is not released using its @@ -436,6 +474,11 @@ def __init__(self, value=1): self._cond = Condition(Lock()) self._value = value + def __repr__(self): + cls = self.__class__ + return (f"<{cls.__module__}.{cls.__qualname__} at {id(self):#x}:" + f" value={self._value}>") + def acquire(self, blocking=True, timeout=None): """Acquire a semaphore, decrementing the internal counter by one. @@ -494,8 +537,7 @@ def release(self, n=1): raise ValueError('n must be one or more') with self._cond: self._value += n - for i in range(n): - self._cond.notify() + self._cond.notify(n) def __exit__(self, t, v, tb): self.release() @@ -519,9 +561,14 @@ class BoundedSemaphore(Semaphore): """ def __init__(self, value=1): - Semaphore.__init__(self, value) + super().__init__(value) self._initial_value = value + def __repr__(self): + cls = self.__class__ + return (f"<{cls.__module__}.{cls.__qualname__} at {id(self):#x}:" + f" value={self._value}/{self._initial_value}>") + def release(self, n=1): """Release a semaphore, incrementing the internal counter by one or more. @@ -538,8 +585,7 @@ def release(self, n=1): if self._value + n > self._initial_value: raise ValueError("Semaphore released too many times") self._value += n - for i in range(n): - self._cond.notify() + self._cond.notify(n) class Event: @@ -557,8 +603,13 @@ def __init__(self): self._cond = Condition(Lock()) self._flag = False + def __repr__(self): + cls = self.__class__ + status = 'set' if self._flag else 'unset' + return f"<{cls.__module__}.{cls.__qualname__} at {id(self):#x}: {status}>" + def _at_fork_reinit(self): - # Private method called by Thread._reset_internal_locks() + # Private method called by Thread._after_fork() self._cond._at_fork_reinit() def is_set(self): @@ -568,7 +619,7 @@ def is_set(self): def isSet(self): """Return true if and only if the internal flag is true. - This method is deprecated, use notify_all() instead. + This method is deprecated, use is_set() instead. """ import warnings @@ -605,11 +656,12 @@ def wait(self, timeout=None): the optional timeout occurs. When the timeout argument is present and not None, it should be a - floating point number specifying a timeout for the operation in seconds + floating-point number specifying a timeout for the operation in seconds (or fractions thereof). This method returns the internal flag on exit, so it will always return - True except if a timeout is given and the operation times out. + ``True`` except if a timeout is given and the operation times out, when + it will return ``False``. """ with self._cond: @@ -648,6 +700,8 @@ def __init__(self, parties, action=None, timeout=None): default for all subsequent 'wait()' calls. """ + if parties < 1: + raise ValueError("parties must be >= 1") self._cond = Condition(Lock()) self._action = action self._timeout = timeout @@ -655,6 +709,13 @@ def __init__(self, parties, action=None, timeout=None): self._state = 0 # 0 filling, 1 draining, -1 resetting, -2 broken self._count = 0 + def __repr__(self): + cls = self.__class__ + if self.broken: + return f"<{cls.__module__}.{cls.__qualname__} at {id(self):#x}: broken>" + return (f"<{cls.__module__}.{cls.__qualname__} at {id(self):#x}:" + f" waiters={self.n_waiting}/{self.parties}>") + def wait(self, timeout=None): """Wait for the barrier. @@ -802,25 +863,6 @@ def _newname(name_template): _limbo = {} _dangling = WeakSet() -# Set of Thread._tstate_lock locks of non-daemon threads used by _shutdown() -# to wait until all Python thread states get deleted: -# see Thread._set_tstate_lock(). -_shutdown_locks_lock = _allocate_lock() -_shutdown_locks = set() - -def _maintain_shutdown_locks(): - """ - Drop any shutdown locks that don't correspond to running threads anymore. - - Calling this from time to time avoids an ever-growing _shutdown_locks - set when Thread objects are not joined explicitly. See bpo-37788. - - This must be called with _shutdown_locks_lock acquired. - """ - # If a lock was released, the corresponding thread has exited - to_remove = [lock for lock in _shutdown_locks if not lock.locked()] - _shutdown_locks.difference_update(to_remove) - # Main class for threads @@ -836,7 +878,7 @@ class Thread: _initialized = False def __init__(self, group=None, target=None, name=None, - args=(), kwargs=None, *, daemon=None): + args=(), kwargs=None, *, daemon=None, context=None): """This constructor should always be called with keyword arguments. Arguments are: *group* should be None; reserved for future extension when a ThreadGroup @@ -848,11 +890,19 @@ class is implemented. *name* is the thread name. By default, a unique name is constructed of the form "Thread-N" where N is a small decimal number. - *args* is the argument tuple for the target invocation. Defaults to (). + *args* is a list or tuple of arguments for the target invocation. Defaults to (). *kwargs* is a dictionary of keyword arguments for the target invocation. Defaults to {}. + *context* is the contextvars.Context value to use for the thread. + The default value is None, which means to check + sys.flags.thread_inherit_context. If that flag is true, use a copy + of the context of the caller. If false, use an empty context. To + explicitly start with an empty context, pass a new instance of + contextvars.Context(). To explicitly start with a copy of the current + context, pass the value from contextvars.copy_context(). + If a subclass overrides the constructor, it must make sure to invoke the base class constructor (Thread.__init__()) before doing anything else to the thread. @@ -877,15 +927,17 @@ class is implemented. self._args = args self._kwargs = kwargs if daemon is not None: + if daemon and not _daemon_threads_allowed(): + raise RuntimeError('daemon threads are disabled in this (sub)interpreter') self._daemonic = daemon else: self._daemonic = current_thread().daemon + self._context = context self._ident = None if _HAVE_THREAD_NATIVE_ID: self._native_id = None - self._tstate_lock = None + self._os_thread_handle = _ThreadHandle() self._started = Event() - self._is_stopped = False self._initialized = True # Copy of sys.stderr used by self._invoke_excepthook() self._stderr = _sys.stderr @@ -893,30 +945,26 @@ class is implemented. # For debugging and _after_fork() _dangling.add(self) - def _reset_internal_locks(self, is_alive): - # private! Called by _after_fork() to reset our internal locks as - # they may be in an invalid state leading to a deadlock or crash. + def _after_fork(self, new_ident=None): + # Private! Called by threading._after_fork(). self._started._at_fork_reinit() - if is_alive: - # bpo-42350: If the fork happens when the thread is already stopped - # (ex: after threading._shutdown() has been called), _tstate_lock - # is None. Do nothing in this case. - if self._tstate_lock is not None: - self._tstate_lock._at_fork_reinit() - self._tstate_lock.acquire() + if new_ident is not None: + # This thread is alive. + self._ident = new_ident + assert self._os_thread_handle.ident == new_ident + if _HAVE_THREAD_NATIVE_ID: + self._set_native_id() else: - # The thread isn't alive after fork: it doesn't have a tstate - # anymore. - self._is_stopped = True - self._tstate_lock = None + # Otherwise, the thread is dead, Jim. _PyThread_AfterFork() + # already marked our handle done. + pass def __repr__(self): assert self._initialized, "Thread.__init__() was not called" status = "initial" if self._started.is_set(): status = "started" - self.is_alive() # easy way to get ._is_stopped set when appropriate - if self._is_stopped: + if self._os_thread_handle.is_done(): status = "stopped" if self._daemonic: status += " daemon" @@ -942,13 +990,25 @@ def start(self): with _active_limbo_lock: _limbo[self] = self + + if self._context is None: + # No context provided + if _sys.flags.thread_inherit_context: + # start with a copy of the context of the caller + self._context = _contextvars.copy_context() + else: + # start with an empty context + self._context = _contextvars.Context() + try: - _start_new_thread(self._bootstrap, ()) + # Start joinable thread + _start_joinable_thread(self._bootstrap, handle=self._os_thread_handle, + daemon=self.daemon) except Exception: with _active_limbo_lock: del _limbo[self] raise - self._started.wait() + self._started.wait() # Will set ident and native_id def run(self): """Method representing the thread's activity. @@ -994,25 +1054,20 @@ def _set_ident(self): def _set_native_id(self): self._native_id = get_native_id() - def _set_tstate_lock(self): - """ - Set a lock object which will be released by the interpreter when - the underlying thread state (see pystate.h) gets deleted. - """ - self._tstate_lock = _set_sentinel() - self._tstate_lock.acquire() - - if not self.daemon: - with _shutdown_locks_lock: - _maintain_shutdown_locks() - _shutdown_locks.add(self._tstate_lock) + def _set_os_name(self): + if _set_name is None or not self._name: + return + try: + _set_name(self._name) + except OSError: + pass def _bootstrap_inner(self): try: self._set_ident() - self._set_tstate_lock() if _HAVE_THREAD_NATIVE_ID: self._set_native_id() + self._set_os_name() self._started.set() with _active_limbo_lock: _active[self._ident] = self @@ -1024,44 +1079,11 @@ def _bootstrap_inner(self): _sys.setprofile(_profile_hook) try: - self.run() + self._context.run(self.run) except: self._invoke_excepthook(self) finally: - with _active_limbo_lock: - try: - # We don't call self._delete() because it also - # grabs _active_limbo_lock. - del _active[get_ident()] - except: - pass - - def _stop(self): - # After calling ._stop(), .is_alive() returns False and .join() returns - # immediately. ._tstate_lock must be released before calling ._stop(). - # - # Normal case: C code at the end of the thread's life - # (release_sentinel in _threadmodule.c) releases ._tstate_lock, and - # that's detected by our ._wait_for_tstate_lock(), called by .join() - # and .is_alive(). Any number of threads _may_ call ._stop() - # simultaneously (for example, if multiple threads are blocked in - # .join() calls), and they're not serialized. That's harmless - - # they'll just make redundant rebindings of ._is_stopped and - # ._tstate_lock. Obscure: we rebind ._tstate_lock last so that the - # "assert self._is_stopped" in ._wait_for_tstate_lock() always works - # (the assert is executed only if ._tstate_lock is None). - # - # Special case: _main_thread releases ._tstate_lock via this - # module's _shutdown() function. - lock = self._tstate_lock - if lock is not None: - assert not lock.locked() - self._is_stopped = True - self._tstate_lock = None - if not self.daemon: - with _shutdown_locks_lock: - # Remove our lock and other released locks from _shutdown_locks - _maintain_shutdown_locks() + self._delete() def _delete(self): "Remove current thread from the dict of currently running threads." @@ -1080,7 +1102,7 @@ def join(self, timeout=None): or until the optional timeout occurs. When the timeout argument is present and not None, it should be a - floating point number specifying a timeout for the operation in seconds + floating-point number specifying a timeout for the operation in seconds (or fractions thereof). As join() always returns None, you must call is_alive() after join() to decide whether a timeout happened -- if the thread is still alive, the join() call timed out. @@ -1103,39 +1125,12 @@ def join(self, timeout=None): if self is current_thread(): raise RuntimeError("cannot join current thread") - if timeout is None: - self._wait_for_tstate_lock() - else: - # the behavior of a negative timeout isn't documented, but - # historically .join(timeout=x) for x<0 has acted as if timeout=0 - self._wait_for_tstate_lock(timeout=max(timeout, 0)) - - def _wait_for_tstate_lock(self, block=True, timeout=-1): - # Issue #18808: wait for the thread state to be gone. - # At the end of the thread's life, after all knowledge of the thread - # is removed from C data structures, C code releases our _tstate_lock. - # This method passes its arguments to _tstate_lock.acquire(). - # If the lock is acquired, the C code is done, and self._stop() is - # called. That sets ._is_stopped to True, and ._tstate_lock to None. - lock = self._tstate_lock - if lock is None: - # already determined that the C code is done - assert self._is_stopped - return + # the behavior of a negative timeout isn't documented, but + # historically .join(timeout=x) for x<0 has acted as if timeout=0 + if timeout is not None: + timeout = max(timeout, 0) - try: - if lock.acquire(block, timeout): - lock.release() - self._stop() - except: - if lock.locked(): - # bpo-45274: lock.acquire() acquired the lock, but the function - # was interrupted with an exception before reaching the - # lock.release(). It can happen if a signal handler raises an - # exception, like CTRL+C which raises KeyboardInterrupt. - lock.release() - self._stop() - raise + self._os_thread_handle.join(timeout) @property def name(self): @@ -1152,6 +1147,8 @@ def name(self): def name(self, name): assert self._initialized, "Thread.__init__() not called" self._name = str(name) + if get_ident() == self._ident: + self._set_os_name() @property def ident(self): @@ -1186,10 +1183,7 @@ def is_alive(self): """ assert self._initialized, "Thread.__init__() not called" - if self._is_stopped or not self._started.is_set(): - return False - self._wait_for_tstate_lock(False) - return not self._is_stopped + return self._started.is_set() and not self._os_thread_handle.is_done() @property def daemon(self): @@ -1210,6 +1204,8 @@ def daemon(self): def daemon(self, daemonic): if not self._initialized: raise RuntimeError("Thread.__init__() not called") + if daemonic and not _daemon_threads_allowed(): + raise RuntimeError('daemon threads are disabled in this interpreter') if self._started.is_set(): raise RuntimeError("cannot set daemon status of active thread") self._daemonic = daemonic @@ -1396,19 +1392,45 @@ class _MainThread(Thread): def __init__(self): Thread.__init__(self, name="MainThread", daemon=False) - self._set_tstate_lock() self._started.set() - self._set_ident() + self._ident = _get_main_thread_ident() + self._os_thread_handle = _make_thread_handle(self._ident) if _HAVE_THREAD_NATIVE_ID: self._set_native_id() with _active_limbo_lock: _active[self._ident] = self +# Helper thread-local instance to detect when a _DummyThread +# is collected. Not a part of the public API. +_thread_local_info = local() + + +class _DeleteDummyThreadOnDel: + ''' + Helper class to remove a dummy thread from threading._active on __del__. + ''' + + def __init__(self, dummy_thread): + self._dummy_thread = dummy_thread + self._tident = dummy_thread.ident + # Put the thread on a thread local variable so that when + # the related thread finishes this instance is collected. + # + # Note: no other references to this instance may be created. + # If any client code creates a reference to this instance, + # the related _DummyThread will be kept forever! + _thread_local_info._track_dummy_thread_ref = self + + def __del__(self, _active_limbo_lock=_active_limbo_lock, _active=_active): + with _active_limbo_lock: + if _active.get(self._tident) is self._dummy_thread: + _active.pop(self._tident, None) + + # Dummy thread class to represent threads not started here. -# These aren't garbage collected when they die, nor can they be waited for. -# If they invoke anything in threading.py that calls current_thread(), they -# leave an entry in the _active dict forever after. +# These should be added to `_active` and removed automatically +# when they die, although they can't be waited for. # Their purpose is to return *something* from current_thread(). # They are marked as daemon threads so we won't wait for them # when we exit (conform previous semantics). @@ -1416,24 +1438,31 @@ def __init__(self): class _DummyThread(Thread): def __init__(self): - Thread.__init__(self, name=_newname("Dummy-%d"), daemon=True) - + Thread.__init__(self, name=_newname("Dummy-%d"), + daemon=_daemon_threads_allowed()) self._started.set() self._set_ident() + self._os_thread_handle = _make_thread_handle(self._ident) if _HAVE_THREAD_NATIVE_ID: self._set_native_id() with _active_limbo_lock: _active[self._ident] = self - - def _stop(self): - pass + _DeleteDummyThreadOnDel(self) def is_alive(self): - assert not self._is_stopped and self._started.is_set() - return True + if not self._os_thread_handle.is_done() and self._started.is_set(): + return True + raise RuntimeError("thread is not alive") def join(self, timeout=None): - assert False, "cannot join a dummy thread" + raise RuntimeError("cannot join a dummy thread") + + def _after_fork(self, new_ident=None): + if new_ident is not None: + self.__class__ = _MainThread + self._name = 'MainThread' + self._daemonic = False + Thread._after_fork(self, new_ident=new_ident) # Global API functions @@ -1468,6 +1497,8 @@ def active_count(): enumerate(). """ + # NOTE: if the logic in here ever changes, update Modules/posixmodule.c + # warn_about_fork_with_threads() to match. with _active_limbo_lock: return len(_active) + len(_limbo) @@ -1514,8 +1545,7 @@ def _register_atexit(func, *arg, **kwargs): if _SHUTTING_DOWN: raise RuntimeError("can't register atexit after shutdown") - call = functools.partial(func, *arg, **kwargs) - _threading_atexits.append(call) + _threading_atexits.append(lambda: func(*arg, **kwargs)) from _thread import stack_size @@ -1530,12 +1560,11 @@ def _shutdown(): """ Wait until the Python thread state of all non-daemon threads get deleted. """ - # Obscure: other threads may be waiting to join _main_thread. That's - # dubious, but some code does it. We can't wait for C code to release - # the main thread's tstate_lock - that won't happen until the interpreter - # is nearly dead. So we release it here. Note that just calling _stop() - # isn't enough: other threads may already be waiting on _tstate_lock. - if _main_thread._is_stopped: + # Obscure: other threads may be waiting to join _main_thread. That's + # dubious, but some code does it. We can't wait for it to be marked as done + # normally - that won't happen until the interpreter is nearly dead. So + # mark it done here. + if _main_thread._os_thread_handle.is_done() and _is_main_interpreter(): # _shutdown() was already called return @@ -1547,39 +1576,11 @@ def _shutdown(): for atexit_call in reversed(_threading_atexits): atexit_call() - # Main thread - if _main_thread.ident == get_ident(): - tlock = _main_thread._tstate_lock - # The main thread isn't finished yet, so its thread state lock can't - # have been released. - assert tlock is not None - assert tlock.locked() - tlock.release() - _main_thread._stop() - else: - # bpo-1596321: _shutdown() must be called in the main thread. - # If the threading module was not imported by the main thread, - # _main_thread is the thread which imported the threading module. - # In this case, ignore _main_thread, similar behavior than for threads - # spawned by C libraries or using _thread.start_new_thread(). - pass - - # Join all non-deamon threads - while True: - with _shutdown_locks_lock: - locks = list(_shutdown_locks) - _shutdown_locks.clear() - - if not locks: - break - - for lock in locks: - # mimic Thread.join() - lock.acquire() - lock.release() - - # new threads can be spawned while we were waiting for the other - # threads to complete + if _is_main_interpreter(): + _main_thread._os_thread_handle._set_done() + + # Wait for all non-daemon threads to exit. + _thread_shutdown() def main_thread(): @@ -1588,16 +1589,9 @@ def main_thread(): In normal conditions, the main thread is the thread from which the Python interpreter was started. """ + # XXX Figure this out for subinterpreters. (See gh-75698.) return _main_thread -# get thread-local implementation, either from the thread -# module, or from the python fallback - -try: - from _thread import _local as local -except ImportError: - from _threading_local import local - def _after_fork(): """ @@ -1606,7 +1600,6 @@ def _after_fork(): # Reset _active_limbo_lock, in case we forked while the lock was held # by another (non-forked) thread. http://bugs.python.org/issue874900 global _active_limbo_lock, _main_thread - global _shutdown_locks_lock, _shutdown_locks _active_limbo_lock = RLock() # fork() only copied the current thread; clear references to others. @@ -1622,10 +1615,6 @@ def _after_fork(): _main_thread = current - # reset _shutdown() locks: threads re-register their _tstate_lock below - _shutdown_locks_lock = _allocate_lock() - _shutdown_locks = set() - with _active_limbo_lock: # Dangling thread instances must still have their locks reset, # because someone may join() them. @@ -1635,16 +1624,13 @@ def _after_fork(): # Any lock/condition variable may be currently locked or in an # invalid state, so we reinitialize them. if thread is current: - # There is only one active thread. We reset the ident to - # its new value since it can have changed. - thread._reset_internal_locks(True) + # This is the one and only active thread. ident = get_ident() - thread._ident = ident + thread._after_fork(new_ident=ident) new_active[ident] = thread else: # All the others are already stopped. - thread._reset_internal_locks(False) - thread._stop() + thread._after_fork() _limbo.clear() _active.clear() diff --git a/Lib/timeit.py b/Lib/timeit.py old mode 100755 new mode 100644 index 258dedccd08..e767f018782 --- a/Lib/timeit.py +++ b/Lib/timeit.py @@ -1,5 +1,3 @@ -#! /usr/bin/env python3 - """Tool for measuring execution time of small code snippets. This module avoids a number of common traps for measuring execution @@ -46,7 +44,6 @@ timeit(string, string) -> float repeat(string, string) -> list default_timer() -> float - """ import gc @@ -174,16 +171,14 @@ def timeit(self, number=default_number): the timer function to be used are passed to the constructor. """ it = itertools.repeat(None, number) - # XXX RUSTPYTHON TODO: gc module implementation - # gcold = gc.isenabled() - # gc.disable() - # try: - # timing = self.inner(it, self.timer) - # finally: - # if gcold: - # gc.enable() - # return timing - return self.inner(it, self.timer) + gcold = gc.isenabled() + gc.disable() + try: + timing = self.inner(it, self.timer) + finally: + if gcold: + gc.enable() + return timing def repeat(self, repeat=default_repeat, number=default_number): """Call timeit() a few times. @@ -306,7 +301,7 @@ def main(args=None, *, _wrap_timer=None): precision += 1 verbose += 1 if o in ("-h", "--help"): - print(__doc__, end=' ') + print(__doc__, end="") return 0 setup = "\n".join(setup) or "pass" diff --git a/Lib/token.py b/Lib/token.py index 54d7cdccadc..f61723cc09d 100644 --- a/Lib/token.py +++ b/Lib/token.py @@ -66,12 +66,15 @@ FSTRING_START = 59 FSTRING_MIDDLE = 60 FSTRING_END = 61 -COMMENT = 62 -NL = 63 +TSTRING_START = 62 +TSTRING_MIDDLE = 63 +TSTRING_END = 64 +COMMENT = 65 +NL = 66 # These aren't used by the C tokenizer but are needed for tokenize.py -ERRORTOKEN = 64 -ENCODING = 65 -N_TOKENS = 66 +ERRORTOKEN = 67 +ENCODING = 68 +N_TOKENS = 69 # Special definitions for cooperation with parser NT_OFFSET = 256 @@ -131,11 +134,11 @@ '~': TILDE, } -def ISTERMINAL(x): +def ISTERMINAL(x: int) -> bool: return x < NT_OFFSET -def ISNONTERMINAL(x): +def ISNONTERMINAL(x: int) -> bool: return x >= NT_OFFSET -def ISEOF(x): +def ISEOF(x: int) -> bool: return x == ENDMARKER diff --git a/Lib/types.py b/Lib/types.py index b036a850687..6efac339434 100644 --- a/Lib/types.py +++ b/Lib/types.py @@ -2,65 +2,78 @@ Define names for built-in types that aren't directly accessible as a builtin. """ -import sys - # Iterators in Python aren't a matter of type but of protocol. A large # and changing number of builtin types implement *some* flavor of # iterator. Don't check the type! Use hasattr to check for both # "__iter__" and "__next__" attributes instead. -def _f(): pass -FunctionType = type(_f) -LambdaType = type(lambda: None) # Same as FunctionType -CodeType = type(_f.__code__) -MappingProxyType = type(type.__dict__) -SimpleNamespace = type(sys.implementation) - -def _cell_factory(): - a = 1 - def f(): - nonlocal a - return f.__closure__[0] -CellType = type(_cell_factory()) - -def _g(): - yield 1 -GeneratorType = type(_g()) - -async def _c(): pass -_c = _c() -CoroutineType = type(_c) -_c.close() # Prevent ResourceWarning - -async def _ag(): - yield -_ag = _ag() -AsyncGeneratorType = type(_ag) - -class _C: - def _m(self): pass -MethodType = type(_C()._m) +try: + from _types import * +except ImportError: + import sys + + def _f(): pass + FunctionType = type(_f) + LambdaType = type(lambda: None) # Same as FunctionType + CodeType = type(_f.__code__) + MappingProxyType = type(type.__dict__) + SimpleNamespace = type(sys.implementation) + + def _cell_factory(): + a = 1 + def f(): + nonlocal a + return f.__closure__[0] + CellType = type(_cell_factory()) + + def _g(): + yield 1 + GeneratorType = type(_g()) + + async def _c(): pass + _c = _c() + CoroutineType = type(_c) + _c.close() # Prevent ResourceWarning + + async def _ag(): + yield + _ag = _ag() + AsyncGeneratorType = type(_ag) + + class _C: + def _m(self): pass + MethodType = type(_C()._m) + + BuiltinFunctionType = type(len) + BuiltinMethodType = type([].append) # Same as BuiltinFunctionType + + WrapperDescriptorType = type(object.__init__) + MethodWrapperType = type(object().__str__) + MethodDescriptorType = type(str.join) + ClassMethodDescriptorType = type(dict.__dict__['fromkeys']) + + ModuleType = type(sys) -BuiltinFunctionType = type(len) -BuiltinMethodType = type([].append) # Same as BuiltinFunctionType + try: + raise TypeError + except TypeError as exc: + TracebackType = type(exc.__traceback__) + FrameType = type(exc.__traceback__.tb_frame) -WrapperDescriptorType = type(object.__init__) -MethodWrapperType = type(object().__str__) -MethodDescriptorType = type(str.join) -ClassMethodDescriptorType = type(dict.__dict__['fromkeys']) + GetSetDescriptorType = type(FunctionType.__code__) + MemberDescriptorType = type(FunctionType.__globals__) -ModuleType = type(sys) + GenericAlias = type(list[int]) + UnionType = type(int | str) -try: - raise TypeError -except TypeError as exc: - TracebackType = type(exc.__traceback__) - FrameType = type(exc.__traceback__.tb_frame) + EllipsisType = type(Ellipsis) + NoneType = type(None) + NotImplementedType = type(NotImplemented) -GetSetDescriptorType = type(FunctionType.__code__) -MemberDescriptorType = type(FunctionType.__globals__) + # CapsuleType cannot be accessed from pure Python, + # so there is no fallback definition. -del sys, _f, _g, _C, _c, _ag, _cell_factory # Not for export + del sys, _f, _g, _C, _c, _ag, _cell_factory # Not for export # Provide a PEP 3115 compliant mechanism for class creation @@ -279,8 +292,7 @@ def coroutine(func): if not callable(func): raise TypeError('types.coroutine() expects a callable') - # XXX RUSTPYTHON TODO: iterable coroutine - if (False and func.__class__ is FunctionType and + if (func.__class__ is FunctionType and getattr(func, '__code__', None).__class__ is CodeType): co_flags = func.__code__.co_flags @@ -325,18 +337,4 @@ def wrapped(*args, **kwargs): return wrapped -GenericAlias = type(list[int]) -UnionType = type(int | str) - -EllipsisType = type(Ellipsis) -NoneType = type(None) -NotImplementedType = type(NotImplemented) - -def __getattr__(name): - if name == 'CapsuleType': - import _socket - return type(_socket.CAPI) - raise AttributeError(f"module {__name__!r} has no attribute {name!r}") - -__all__ = [n for n in globals() if n[:1] != '_'] -__all__ += ['CapsuleType'] +__all__ = [n for n in globals() if not n.startswith('_')] # for pydoc diff --git a/Lib/typing.py b/Lib/typing.py index a7397356d65..92b78defd11 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -27,7 +27,7 @@ import operator import sys import types -from types import WrapperDescriptorType, MethodWrapperType, MethodDescriptorType, GenericAlias +from types import GenericAlias from _typing import ( _idfunc, @@ -38,6 +38,7 @@ ParamSpecKwargs, TypeAliasType, Generic, + Union, NoDefault, ) @@ -126,6 +127,7 @@ 'cast', 'clear_overloads', 'dataclass_transform', + 'evaluate_forward_ref', 'final', 'get_args', 'get_origin', @@ -160,17 +162,26 @@ 'Unpack', ] +class _LazyAnnotationLib: + def __getattr__(self, attr): + global _lazy_annotationlib + import annotationlib + _lazy_annotationlib = annotationlib + return getattr(annotationlib, attr) + +_lazy_annotationlib = _LazyAnnotationLib() -def _type_convert(arg, module=None, *, allow_special_forms=False): + +def _type_convert(arg, module=None, *, allow_special_forms=False, owner=None): """For converting None to type(None), and strings to ForwardRef.""" if arg is None: return type(None) if isinstance(arg, str): - return ForwardRef(arg, module=module, is_class=allow_special_forms) + return _make_forward_ref(arg, module=module, is_class=allow_special_forms, owner=owner) return arg -def _type_check(arg, msg, is_argument=True, module=None, *, allow_special_forms=False): +def _type_check(arg, msg, is_argument=True, module=None, *, allow_special_forms=False, owner=None): """Check that the argument is a type, and return it (internal helper). As a special case, accept None and return type(None) instead. Also wrap strings @@ -188,7 +199,7 @@ def _type_check(arg, msg, is_argument=True, module=None, *, allow_special_forms= if is_argument: invalid_generic_forms += (Final,) - arg = _type_convert(arg, module=module, allow_special_forms=allow_special_forms) + arg = _type_convert(arg, module=module, allow_special_forms=allow_special_forms, owner=owner) if (isinstance(arg, _GenericAlias) and arg.__origin__ in invalid_generic_forms): raise TypeError(f"{arg} is not valid as type argument") @@ -240,21 +251,10 @@ def _type_repr(obj): typically enough to uniquely identify a type. For everything else, we fall back on repr(obj). """ - # When changing this function, don't forget about - # `_collections_abc._type_repr`, which does the same thing - # and must be consistent with this one. - if isinstance(obj, type): - if obj.__module__ == 'builtins': - return obj.__qualname__ - return f'{obj.__module__}.{obj.__qualname__}' - if obj is ...: - return '...' - if isinstance(obj, types.FunctionType): - return obj.__name__ if isinstance(obj, tuple): # Special case for `repr` of types with `ParamSpec`: return '[' + ', '.join(_type_repr(t) for t in obj) + ']' - return repr(obj) + return _lazy_annotationlib.type_repr(obj) def _collect_type_parameters(args, *, enforce_default_ordering: bool = True): @@ -356,41 +356,11 @@ def _deduplicate(params, *, unhashable_fallback=False): if not unhashable_fallback: raise # Happens for cases like `Annotated[dict, {'x': IntValidator()}]` - return _deduplicate_unhashable(params) - -def _deduplicate_unhashable(unhashable_params): - new_unhashable = [] - for t in unhashable_params: - if t not in new_unhashable: - new_unhashable.append(t) - return new_unhashable - -def _compare_args_orderless(first_args, second_args): - first_unhashable = _deduplicate_unhashable(first_args) - second_unhashable = _deduplicate_unhashable(second_args) - t = list(second_unhashable) - try: - for elem in first_unhashable: - t.remove(elem) - except ValueError: - return False - return not t - -def _remove_dups_flatten(parameters): - """Internal helper for Union creation and substitution. - - Flatten Unions among parameters, then remove duplicates. - """ - # Flatten out Union[Union[...], ...]. - params = [] - for p in parameters: - if isinstance(p, (_UnionGenericAlias, types.UnionType)): - params.extend(p.__args__) - else: - params.append(p) - - return tuple(_deduplicate(params, unhashable_fallback=True)) - + new_unhashable = [] + for t in params: + if t not in new_unhashable: + new_unhashable.append(t) + return new_unhashable def _flatten_literal_params(parameters): """Internal helper for Literal creation: flatten Literals among parameters.""" @@ -460,7 +430,8 @@ def __repr__(self): _sentinel = _Sentinel() -def _eval_type(t, globalns, localns, type_params=_sentinel, *, recursive_guard=frozenset()): +def _eval_type(t, globalns, localns, type_params=_sentinel, *, recursive_guard=frozenset(), + format=None, owner=None, parent_fwdref=None, prefer_fwd_module=False): """Evaluate all forward references in the given type t. For use of globalns and localns see the docstring for get_type_hints(). @@ -470,12 +441,30 @@ def _eval_type(t, globalns, localns, type_params=_sentinel, *, recursive_guard=f if type_params is _sentinel: _deprecation_warning_for_no_type_params_passed("typing._eval_type") type_params = () - if isinstance(t, ForwardRef): - return t._evaluate(globalns, localns, type_params, recursive_guard=recursive_guard) - if isinstance(t, (_GenericAlias, GenericAlias, types.UnionType)): + if isinstance(t, _lazy_annotationlib.ForwardRef): + # If the forward_ref has __forward_module__ set, evaluate() infers the globals + # from the module, and it will probably pick better than the globals we have here. + # We do this only for calls from get_type_hints() (which opts in through the + # prefer_fwd_module flag), so that the default behavior remains more straightforward. + if prefer_fwd_module and t.__forward_module__ is not None: + globalns = None + # If there are type params on the owner, we need to add them back, because + # annotationlib won't. + if owner_type_params := getattr(owner, "__type_params__", None): + globalns = getattr( + sys.modules.get(t.__forward_module__, None), "__dict__", None + ) + if globalns is not None: + globalns = dict(globalns) + for type_param in owner_type_params: + globalns[type_param.__name__] = type_param + return evaluate_forward_ref(t, globals=globalns, locals=localns, + type_params=type_params, owner=owner, + _recursive_guard=recursive_guard, format=format) + if isinstance(t, (_GenericAlias, GenericAlias, Union)): if isinstance(t, GenericAlias): args = tuple( - ForwardRef(arg) if isinstance(arg, str) else arg + _make_forward_ref(arg, parent_fwdref=parent_fwdref) if isinstance(arg, str) else arg for arg in t.__args__ ) is_unpacked = t.__unpacked__ @@ -488,7 +477,8 @@ def _eval_type(t, globalns, localns, type_params=_sentinel, *, recursive_guard=f ev_args = tuple( _eval_type( - a, globalns, localns, type_params, recursive_guard=recursive_guard + a, globalns, localns, type_params, recursive_guard=recursive_guard, + format=format, owner=owner, prefer_fwd_module=prefer_fwd_module, ) for a in t.__args__ ) @@ -496,7 +486,7 @@ def _eval_type(t, globalns, localns, type_params=_sentinel, *, recursive_guard=f return t if isinstance(t, GenericAlias): return GenericAlias(t.__origin__, ev_args) - if isinstance(t, types.UnionType): + if isinstance(t, Union): return functools.reduce(operator.or_, ev_args) else: return t.copy_with(ev_args) @@ -750,59 +740,6 @@ class FastConnector(Connection): item = _type_check(parameters, f'{self} accepts only single type.', allow_special_forms=True) return _GenericAlias(self, (item,)) -@_SpecialForm -def Union(self, parameters): - """Union type; Union[X, Y] means either X or Y. - - On Python 3.10 and higher, the | operator - can also be used to denote unions; - X | Y means the same thing to the type checker as Union[X, Y]. - - To define a union, use e.g. Union[int, str]. Details: - - The arguments must be types and there must be at least one. - - None as an argument is a special case and is replaced by - type(None). - - Unions of unions are flattened, e.g.:: - - assert Union[Union[int, str], float] == Union[int, str, float] - - - Unions of a single argument vanish, e.g.:: - - assert Union[int] == int # The constructor actually returns int - - - Redundant arguments are skipped, e.g.:: - - assert Union[int, str, int] == Union[int, str] - - - When comparing unions, the argument order is ignored, e.g.:: - - assert Union[int, str] == Union[str, int] - - - You cannot subclass or instantiate a union. - - You can use Optional[X] as a shorthand for Union[X, None]. - """ - if parameters == (): - raise TypeError("Cannot take a Union of no types.") - if not isinstance(parameters, tuple): - parameters = (parameters,) - msg = "Union[arg, ...]: each arg must be a type." - parameters = tuple(_type_check(p, msg) for p in parameters) - parameters = _remove_dups_flatten(parameters) - if len(parameters) == 1: - return parameters[0] - if len(parameters) == 2 and type(None) in parameters: - return _UnionGenericAlias(self, parameters, name="Optional") - return _UnionGenericAlias(self, parameters) - -def _make_union(left, right): - """Used from the C implementation of TypeVar. - - TypeVar.__or__ calls this instead of returning types.UnionType - because we want to allow unions between TypeVars and strings - (forward references). - """ - return Union[left, right] - @_SpecialForm def Optional(self, parameters): """Optional[X] is equivalent to Union[X, None].""" @@ -1012,116 +949,85 @@ def run(arg: Child | Unrelated): return _GenericAlias(self, (item,)) -class ForwardRef(_Final, _root=True): - """Internal wrapper to hold a forward reference.""" - - __slots__ = ('__forward_arg__', '__forward_code__', - '__forward_evaluated__', '__forward_value__', - '__forward_is_argument__', '__forward_is_class__', - '__forward_module__') - - def __init__(self, arg, is_argument=True, module=None, *, is_class=False): - if not isinstance(arg, str): - raise TypeError(f"Forward reference must be a string -- got {arg!r}") +def _make_forward_ref(code, *, parent_fwdref=None, **kwargs): + if parent_fwdref is not None: + if parent_fwdref.__forward_module__ is not None: + kwargs['module'] = parent_fwdref.__forward_module__ + if parent_fwdref.__owner__ is not None: + kwargs['owner'] = parent_fwdref.__owner__ + forward_ref = _lazy_annotationlib.ForwardRef(code, **kwargs) + # For compatibility, eagerly compile the forwardref's code. + forward_ref.__forward_code__ + return forward_ref - # If we do `def f(*args: *Ts)`, then we'll have `arg = '*Ts'`. - # Unfortunately, this isn't a valid expression on its own, so we - # do the unpacking manually. - if arg.startswith('*'): - arg_to_compile = f'({arg},)[0]' # E.g. (*Ts,)[0] or (*tuple[int, int],)[0] - else: - arg_to_compile = arg - try: - code = compile(arg_to_compile, '<string>', 'eval') - except SyntaxError: - raise SyntaxError(f"Forward reference must be an expression -- got {arg!r}") - - self.__forward_arg__ = arg - self.__forward_code__ = code - self.__forward_evaluated__ = False - self.__forward_value__ = None - self.__forward_is_argument__ = is_argument - self.__forward_is_class__ = is_class - self.__forward_module__ = module - - def _evaluate(self, globalns, localns, type_params=_sentinel, *, recursive_guard): - if type_params is _sentinel: - _deprecation_warning_for_no_type_params_passed("typing.ForwardRef._evaluate") - type_params = () - if self.__forward_arg__ in recursive_guard: - return self - if not self.__forward_evaluated__ or localns is not globalns: - if globalns is None and localns is None: - globalns = localns = {} - elif globalns is None: - globalns = localns - elif localns is None: - localns = globalns - if self.__forward_module__ is not None: - globalns = getattr( - sys.modules.get(self.__forward_module__, None), '__dict__', globalns - ) - - # type parameters require some special handling, - # as they exist in their own scope - # but `eval()` does not have a dedicated parameter for that scope. - # For classes, names in type parameter scopes should override - # names in the global scope (which here are called `localns`!), - # but should in turn be overridden by names in the class scope - # (which here are called `globalns`!) - if type_params: - globalns, localns = dict(globalns), dict(localns) - for param in type_params: - param_name = param.__name__ - if not self.__forward_is_class__ or param_name not in globalns: - globalns[param_name] = param - localns.pop(param_name, None) - - type_ = _type_check( - eval(self.__forward_code__, globalns, localns), - "Forward references must evaluate to types.", - is_argument=self.__forward_is_argument__, - allow_special_forms=self.__forward_is_class__, - ) - self.__forward_value__ = _eval_type( - type_, - globalns, - localns, - type_params, - recursive_guard=(recursive_guard | {self.__forward_arg__}), - ) - self.__forward_evaluated__ = True - return self.__forward_value__ - def __eq__(self, other): - if not isinstance(other, ForwardRef): - return NotImplemented - if self.__forward_evaluated__ and other.__forward_evaluated__: - return (self.__forward_arg__ == other.__forward_arg__ and - self.__forward_value__ == other.__forward_value__) - return (self.__forward_arg__ == other.__forward_arg__ and - self.__forward_module__ == other.__forward_module__) - - def __hash__(self): - return hash((self.__forward_arg__, self.__forward_module__)) - - def __or__(self, other): - return Union[self, other] - - def __ror__(self, other): - return Union[other, self] +def evaluate_forward_ref( + forward_ref, + *, + owner=None, + globals=None, + locals=None, + type_params=None, + format=None, + _recursive_guard=frozenset(), +): + """Evaluate a forward reference as a type hint. + + This is similar to calling the ForwardRef.evaluate() method, + but unlike that method, evaluate_forward_ref() also + recursively evaluates forward references nested within the type hint. + + *forward_ref* must be an instance of ForwardRef. *owner*, if given, + should be the object that holds the annotations that the forward reference + derived from, such as a module, class object, or function. It is used to + infer the namespaces to use for looking up names. *globals* and *locals* + can also be explicitly given to provide the global and local namespaces. + *type_params* is a tuple of type parameters that are in scope when + evaluating the forward reference. This parameter should be provided (though + it may be an empty tuple) if *owner* is not given and the forward reference + does not already have an owner set. *format* specifies the format of the + annotation and is a member of the annotationlib.Format enum, defaulting to + VALUE. - def __repr__(self): - if self.__forward_module__ is None: - module_repr = '' - else: - module_repr = f', module={self.__forward_module__!r}' - return f'ForwardRef({self.__forward_arg__!r}{module_repr})' + """ + if format == _lazy_annotationlib.Format.STRING: + return forward_ref.__forward_arg__ + if forward_ref.__forward_arg__ in _recursive_guard: + return forward_ref + + if format is None: + format = _lazy_annotationlib.Format.VALUE + value = forward_ref.evaluate(globals=globals, locals=locals, + type_params=type_params, owner=owner, format=format) + + if (isinstance(value, _lazy_annotationlib.ForwardRef) + and format == _lazy_annotationlib.Format.FORWARDREF): + return value + + if isinstance(value, str): + value = _make_forward_ref(value, module=forward_ref.__forward_module__, + owner=owner or forward_ref.__owner__, + is_argument=forward_ref.__forward_is_argument__, + is_class=forward_ref.__forward_is_class__) + if owner is None: + owner = forward_ref.__owner__ + return _eval_type( + value, + globals, + locals, + type_params, + recursive_guard=_recursive_guard | {forward_ref.__forward_arg__}, + format=format, + owner=owner, + parent_fwdref=forward_ref, + ) def _is_unpacked_typevartuple(x: Any) -> bool: + # Need to check 'is True' here + # See: https://github.com/python/cpython/issues/137706 return ((not isinstance(x, type)) and - getattr(x, '__typing_is_unpacked_typevartuple__', False)) + getattr(x, '__typing_is_unpacked_typevartuple__', False) is True) def _is_typevar_like(x: Any) -> bool: @@ -1191,7 +1097,7 @@ def _paramspec_prepare_subst(self, alias, args): params = alias.__parameters__ i = params.index(self) if i == len(args) and self.has_default(): - args = [*args, self.__default__] + args = (*args, self.__default__) if i >= len(args): raise TypeError(f"Too few arguments for {alias}") # Special case where Z[[int, str, bool]] == Z[int, str, bool] in PEP 612. @@ -1236,14 +1142,26 @@ def _generic_class_getitem(cls, args): f"Parameters to {cls.__name__}[...] must all be unique") else: # Subscripting a regular Generic subclass. - for param in cls.__parameters__: + try: + parameters = cls.__parameters__ + except AttributeError as e: + init_subclass = getattr(cls, '__init_subclass__', None) + if init_subclass not in {None, Generic.__init_subclass__}: + e.add_note( + f"Note: this exception may have been caused by " + f"{init_subclass.__qualname__!r} (or the " + f"'__init_subclass__' method on a superclass) not " + f"calling 'super().__init_subclass__()'" + ) + raise + for param in parameters: prepare = getattr(param, '__typing_prepare_subst__', None) if prepare is not None: args = prepare(cls, args) _check_generic_specialization(cls, args) new_args = [] - for param, new_arg in zip(cls.__parameters__, args): + for param, new_arg in zip(parameters, args): if isinstance(param, TypeVarTuple): new_args.extend(new_arg) else: @@ -1758,45 +1676,41 @@ def __getitem__(self, params): return self.copy_with(params) -class _UnionGenericAlias(_NotIterable, _GenericAlias, _root=True): - def copy_with(self, params): - return Union[params] +class _UnionGenericAliasMeta(type): + def __instancecheck__(self, inst: object) -> bool: + import warnings + warnings._deprecated("_UnionGenericAlias", remove=(3, 17)) + return isinstance(inst, Union) + + def __subclasscheck__(self, inst: type) -> bool: + import warnings + warnings._deprecated("_UnionGenericAlias", remove=(3, 17)) + return issubclass(inst, Union) def __eq__(self, other): - if not isinstance(other, (_UnionGenericAlias, types.UnionType)): - return NotImplemented - try: # fast path - return set(self.__args__) == set(other.__args__) - except TypeError: # not hashable, slow path - return _compare_args_orderless(self.__args__, other.__args__) + import warnings + warnings._deprecated("_UnionGenericAlias", remove=(3, 17)) + if other is _UnionGenericAlias or other is Union: + return True + return NotImplemented def __hash__(self): - return hash(frozenset(self.__args__)) + return hash(Union) - def __repr__(self): - args = self.__args__ - if len(args) == 2: - if args[0] is type(None): - return f'typing.Optional[{_type_repr(args[1])}]' - elif args[1] is type(None): - return f'typing.Optional[{_type_repr(args[0])}]' - return super().__repr__() - def __instancecheck__(self, obj): - for arg in self.__args__: - if isinstance(obj, arg): - return True - return False +class _UnionGenericAlias(metaclass=_UnionGenericAliasMeta): + """Compatibility hack. - def __subclasscheck__(self, cls): - for arg in self.__args__: - if issubclass(cls, arg): - return True - return False + A class named _UnionGenericAlias used to be used to implement + typing.Union. This class exists to serve as a shim to preserve + the meaning of some code that used to use _UnionGenericAlias + directly. - def __reduce__(self): - func, (origin, args) = super().__reduce__() - return func, (Union, args) + """ + def __new__(cls, self_cls, parameters, /, *, name=None): + import warnings + warnings._deprecated("_UnionGenericAlias", remove=(3, 17)) + return Union[parameters] def _value_and_type_iter(parameters): @@ -1918,6 +1832,7 @@ class _TypingEllipsis: '__init__', '__module__', '__new__', '__slots__', '__subclasshook__', '__weakref__', '__class_getitem__', '__match_args__', '__static_attributes__', '__firstlineno__', + '__annotate__', '__annotate_func__', '__annotations_cache__', }) # These special attributes will be not collected as protocol members. @@ -1934,7 +1849,13 @@ def _get_protocol_attrs(cls): for base in cls.__mro__[:-1]: # without object if base.__name__ in {'Protocol', 'Generic'}: continue - annotations = getattr(base, '__annotations__', {}) + try: + annotations = base.__annotations__ + except Exception: + # Only go through annotationlib to handle deferred annotations if we need to + annotations = _lazy_annotationlib.get_annotations( + base, format=_lazy_annotationlib.Format.FORWARDREF + ) for attr in (*base.__dict__, *annotations): if not attr.startswith('_abc_') and attr not in EXCLUDED_ATTRIBUTES: attrs.add(attr) @@ -1987,8 +1908,7 @@ def _allow_reckless_class_checks(depth=2): The abc and functools modules indiscriminately call isinstance() and issubclass() on the whole MRO of a user class, which may contain protocols. """ - # XXX: RUSTPYTHON; https://github.com/python/cpython/pull/136115 - return _caller(depth) in {'abc', '_py_abc', 'functools', None} + return _caller(depth) in {'abc', 'functools', None} _PROTO_ALLOWLIST = { @@ -1998,6 +1918,8 @@ def _allow_reckless_class_checks(depth=2): 'Reversible', 'Buffer', ], 'contextlib': ['AbstractContextManager', 'AbstractAsyncContextManager'], + 'io': ['Reader', 'Writer'], + 'os': ['PathLike'], } @@ -2150,11 +2072,17 @@ def _proto_hook(cls, other): break # ...or in annotations, if it is a sub-protocol. - annotations = getattr(base, '__annotations__', {}) - if (isinstance(annotations, collections.abc.Mapping) and - attr in annotations and - issubclass(other, Generic) and getattr(other, '_is_protocol', False)): - break + if issubclass(other, Generic) and getattr(other, "_is_protocol", False): + # We avoid the slower path through annotationlib here because in most + # cases it should be unnecessary. + try: + annos = base.__annotations__ + except Exception: + annos = _lazy_annotationlib.get_annotations( + base, format=_lazy_annotationlib.Format.FORWARDREF + ) + if attr in annos: + break else: return NotImplemented return True @@ -2217,7 +2145,7 @@ class _AnnotatedAlias(_NotIterable, _GenericAlias, _root=True): """Runtime representation of an annotated type. At its core 'Annotated[t, dec1, dec2, ...]' is an alias for the type 't' - with extra annotations. The alias behaves like a normal typing alias. + with extra metadata. The alias behaves like a normal typing alias. Instantiating is the same as instantiating the underlying type; binding it to types is also the same. @@ -2396,12 +2324,8 @@ def greet(name: str) -> None: return val -_allowed_types = (types.FunctionType, types.BuiltinFunctionType, - types.MethodType, types.ModuleType, - WrapperDescriptorType, MethodWrapperType, MethodDescriptorType) - - -def get_type_hints(obj, globalns=None, localns=None, include_extras=False): +def get_type_hints(obj, globalns=None, localns=None, include_extras=False, + *, format=None): """Return type hints for an object. This is often the same as obj.__annotations__, but it handles @@ -2434,17 +2358,21 @@ def get_type_hints(obj, globalns=None, localns=None, include_extras=False): """ if getattr(obj, '__no_type_check__', None): return {} + Format = _lazy_annotationlib.Format + if format is None: + format = Format.VALUE # Classes require a special treatment. if isinstance(obj, type): hints = {} for base in reversed(obj.__mro__): + ann = _lazy_annotationlib.get_annotations(base, format=format) + if format == Format.STRING: + hints.update(ann) + continue if globalns is None: base_globals = getattr(sys.modules.get(base.__module__, None), '__dict__', {}) else: base_globals = globalns - ann = base.__dict__.get('__annotations__', {}) - if isinstance(ann, types.GetSetDescriptorType): - ann = {} base_locals = dict(vars(base)) if localns is None else localns if localns is None and globalns is None: # This is surprising, but required. Before Python 3.10, @@ -2454,14 +2382,33 @@ def get_type_hints(obj, globalns=None, localns=None, include_extras=False): # *base_globals* first rather than *base_locals*. # This only affects ForwardRefs. base_globals, base_locals = base_locals, base_globals + type_params = base.__type_params__ + base_globals, base_locals = _add_type_params_to_scope( + type_params, base_globals, base_locals, True) for name, value in ann.items(): + if isinstance(value, str): + value = _make_forward_ref(value, is_argument=False, is_class=True) + value = _eval_type(value, base_globals, base_locals, (), + format=format, owner=obj, prefer_fwd_module=True) if value is None: value = type(None) - if isinstance(value, str): - value = ForwardRef(value, is_argument=False, is_class=True) - value = _eval_type(value, base_globals, base_locals, base.__type_params__) hints[name] = value - return hints if include_extras else {k: _strip_annotations(t) for k, t in hints.items()} + if include_extras or format == Format.STRING: + return hints + else: + return {k: _strip_annotations(t) for k, t in hints.items()} + + hints = _lazy_annotationlib.get_annotations(obj, format=format) + if ( + not hints + and not isinstance(obj, types.ModuleType) + and not callable(obj) + and not hasattr(obj, '__annotations__') + and not hasattr(obj, '__annotate__') + ): + raise TypeError(f"{obj!r} is not a module, class, or callable.") + if format == Format.STRING: + return hints if globalns is None: if isinstance(obj, types.ModuleType): @@ -2476,31 +2423,38 @@ def get_type_hints(obj, globalns=None, localns=None, include_extras=False): localns = globalns elif localns is None: localns = globalns - hints = getattr(obj, '__annotations__', None) - if hints is None: - # Return empty annotations for something that _could_ have them. - if isinstance(obj, _allowed_types): - return {} - else: - raise TypeError('{!r} is not a module, class, method, ' - 'or function.'.format(obj)) - hints = dict(hints) type_params = getattr(obj, "__type_params__", ()) + globalns, localns = _add_type_params_to_scope(type_params, globalns, localns, False) for name, value in hints.items(): - if value is None: - value = type(None) if isinstance(value, str): # class-level forward refs were handled above, this must be either # a module-level annotation or a function argument annotation - value = ForwardRef( + value = _make_forward_ref( value, is_argument=not isinstance(obj, types.ModuleType), is_class=False, ) - hints[name] = _eval_type(value, globalns, localns, type_params) + value = _eval_type(value, globalns, localns, (), format=format, owner=obj, prefer_fwd_module=True) + if value is None: + value = type(None) + hints[name] = value return hints if include_extras else {k: _strip_annotations(t) for k, t in hints.items()} +# Add type parameters to the globals and locals scope. This is needed for +# compatibility. +def _add_type_params_to_scope(type_params, globalns, localns, is_class): + if not type_params: + return globalns, localns + globalns = dict(globalns) + localns = dict(localns) + for param in type_params: + if not is_class or param.__name__ not in globalns: + globalns[param.__name__] = param + localns.pop(param.__name__, None) + return globalns, localns + + def _strip_annotations(t): """Strip the annotations from a given type.""" if isinstance(t, _AnnotatedAlias): @@ -2517,7 +2471,7 @@ def _strip_annotations(t): if stripped_args == t.__args__: return t return GenericAlias(t.__origin__, stripped_args) - if isinstance(t, types.UnionType): + if isinstance(t, Union): stripped_args = tuple(_strip_annotations(a) for a in t.__args__) if stripped_args == t.__args__: return t @@ -2551,8 +2505,8 @@ def get_origin(tp): return tp.__origin__ if tp is Generic: return Generic - if isinstance(tp, types.UnionType): - return types.UnionType + if isinstance(tp, Union): + return Union return None @@ -2577,7 +2531,7 @@ def get_args(tp): if _should_unflatten_callable_args(tp, res): res = (list(res[:-1]), res[-1]) return res - if isinstance(tp, types.UnionType): + if isinstance(tp, Union): return tp.__args__ return () @@ -2844,7 +2798,7 @@ class Other(Leaf): # Error reported by type checker Sequence = _alias(collections.abc.Sequence, 1) MutableSequence = _alias(collections.abc.MutableSequence, 1) ByteString = _DeprecatedGenericAlias( - collections.abc.ByteString, 0, removal_version=(3, 14) # Not generic. + collections.abc.ByteString, 0, removal_version=(3, 17) # Not generic. ) # Tuple accepts variable number of parameters. Tuple = _TupleType(tuple, -1, inst=False, name='Tuple') @@ -2977,35 +2931,73 @@ def __round__(self, ndigits: int = 0) -> T: pass -def _make_nmtuple(name, types, module, defaults = ()): - fields = [n for n, t in types] - types = {n: _type_check(t, f"field {n} annotation must be a type") - for n, t in types} +def _make_nmtuple(name, fields, annotate_func, module, defaults = ()): nm_tpl = collections.namedtuple(name, fields, defaults=defaults, module=module) - nm_tpl.__annotations__ = nm_tpl.__new__.__annotations__ = types + nm_tpl.__annotate__ = nm_tpl.__new__.__annotate__ = annotate_func return nm_tpl +def _make_eager_annotate(types): + checked_types = {key: _type_check(val, f"field {key} annotation must be a type") + for key, val in types.items()} + def annotate(format): + match format: + case _lazy_annotationlib.Format.VALUE | _lazy_annotationlib.Format.FORWARDREF: + return checked_types + case _lazy_annotationlib.Format.STRING: + return _lazy_annotationlib.annotations_to_string(types) + case _: + raise NotImplementedError(format) + return annotate + + # attributes prohibited to set in NamedTuple class syntax _prohibited = frozenset({'__new__', '__init__', '__slots__', '__getnewargs__', '_fields', '_field_defaults', '_make', '_replace', '_asdict', '_source'}) -_special = frozenset({'__module__', '__name__', '__annotations__'}) +_special = frozenset({'__module__', '__name__', '__annotations__', '__annotate__', + '__annotate_func__', '__annotations_cache__'}) class NamedTupleMeta(type): def __new__(cls, typename, bases, ns): assert _NamedTuple in bases + if "__classcell__" in ns: + raise TypeError( + "uses of super() and __class__ are unsupported in methods of NamedTuple subclasses") for base in bases: if base is not _NamedTuple and base is not Generic: raise TypeError( 'can only inherit from a NamedTuple type and Generic') bases = tuple(tuple if base is _NamedTuple else base for base in bases) - types = ns.get('__annotations__', {}) + if "__annotations__" in ns: + types = ns["__annotations__"] + field_names = list(types) + annotate = _make_eager_annotate(types) + elif (original_annotate := _lazy_annotationlib.get_annotate_from_class_namespace(ns)) is not None: + types = _lazy_annotationlib.call_annotate_function( + original_annotate, _lazy_annotationlib.Format.FORWARDREF) + field_names = list(types) + + # For backward compatibility, type-check all the types at creation time + for typ in types.values(): + _type_check(typ, "field annotation must be a type") + + def annotate(format): + annos = _lazy_annotationlib.call_annotate_function( + original_annotate, format) + if format != _lazy_annotationlib.Format.STRING: + return {key: _type_check(val, f"field {key} annotation must be a type") + for key, val in annos.items()} + return annos + else: + # Empty NamedTuple + field_names = [] + annotate = lambda format: {} default_names = [] - for field_name in types: + for field_name in field_names: if field_name in ns: default_names.append(field_name) elif default_names: @@ -3013,7 +3005,7 @@ def __new__(cls, typename, bases, ns): f"cannot follow default field" f"{'s' if len(default_names) > 1 else ''} " f"{', '.join(default_names)}") - nm_tpl = _make_nmtuple(typename, types.items(), + nm_tpl = _make_nmtuple(typename, field_names, annotate, defaults=[ns[n] for n in default_names], module=ns['__module__']) nm_tpl.__bases__ = bases @@ -3104,7 +3096,11 @@ class Employee(NamedTuple): import warnings warnings._deprecated(deprecated_thing, message=deprecation_msg, remove=(3, 15)) fields = kwargs.items() - nt = _make_nmtuple(typename, fields, module=_caller()) + types = {n: _type_check(t, f"field {n} annotation must be a type") + for n, t in fields} + field_names = [n for n, _ in fields] + + nt = _make_nmtuple(typename, field_names, _make_eager_annotate(types), module=_caller()) nt.__orig_bases__ = (NamedTuple,) return nt @@ -3158,16 +3154,26 @@ def __new__(cls, name, bases, ns, total=True): else: generic_base = () + ns_annotations = ns.pop('__annotations__', None) + tp_dict = type.__new__(_TypedDictMeta, name, (*generic_base, dict), ns) if not hasattr(tp_dict, '__orig_bases__'): tp_dict.__orig_bases__ = bases - annotations = {} - own_annotations = ns.get('__annotations__', {}) + if ns_annotations is not None: + own_annotate = None + own_annotations = ns_annotations + elif (own_annotate := _lazy_annotationlib.get_annotate_from_class_namespace(ns)) is not None: + own_annotations = _lazy_annotationlib.call_annotate_function( + own_annotate, _lazy_annotationlib.Format.FORWARDREF, owner=tp_dict + ) + else: + own_annotate = None + own_annotations = {} msg = "TypedDict('Name', {f0: t0, f1: t1, ...}); each t must be a type" - own_annotations = { - n: _type_check(tp, msg, module=tp_dict.__module__) + own_checked_annotations = { + n: _type_check(tp, msg, owner=tp_dict, module=tp_dict.__module__) for n, tp in own_annotations.items() } required_keys = set() @@ -3176,8 +3182,6 @@ def __new__(cls, name, bases, ns, total=True): mutable_keys = set() for base in bases: - annotations.update(base.__dict__.get('__annotations__', {})) - base_required = base.__dict__.get('__required_keys__', set()) required_keys |= base_required optional_keys -= base_required @@ -3189,8 +3193,7 @@ def __new__(cls, name, bases, ns, total=True): readonly_keys.update(base.__dict__.get('__readonly_keys__', ())) mutable_keys.update(base.__dict__.get('__mutable_keys__', ())) - annotations.update(own_annotations) - for annotation_key, annotation_type in own_annotations.items(): + for annotation_key, annotation_type in own_checked_annotations.items(): qualifiers = set(_get_typeddict_qualifiers(annotation_type)) if Required in qualifiers: is_required = True @@ -3221,7 +3224,36 @@ def __new__(cls, name, bases, ns, total=True): f"Required keys overlap with optional keys in {name}:" f" {required_keys=}, {optional_keys=}" ) - tp_dict.__annotations__ = annotations + + def __annotate__(format): + annos = {} + for base in bases: + if base is Generic: + continue + base_annotate = base.__annotate__ + if base_annotate is None: + continue + base_annos = _lazy_annotationlib.call_annotate_function( + base_annotate, format, owner=base) + annos.update(base_annos) + if own_annotate is not None: + own = _lazy_annotationlib.call_annotate_function( + own_annotate, format, owner=tp_dict) + if format != _lazy_annotationlib.Format.STRING: + own = { + n: _type_check(tp, msg, module=tp_dict.__module__) + for n, tp in own.items() + } + elif format == _lazy_annotationlib.Format.STRING: + own = _lazy_annotationlib.annotations_to_string(own_annotations) + elif format in (_lazy_annotationlib.Format.FORWARDREF, _lazy_annotationlib.Format.VALUE): + own = own_checked_annotations + else: + raise NotImplementedError(format) + annos.update(own) + return annos + + tp_dict.__annotate__ = __annotate__ tp_dict.__required_keys__ = frozenset(required_keys) tp_dict.__optional_keys__ = frozenset(optional_keys) tp_dict.__readonly_keys__ = frozenset(readonly_keys) @@ -3517,7 +3549,7 @@ def readline(self, limit: int = -1) -> AnyStr: pass @abstractmethod - def readlines(self, hint: int = -1) -> List[AnyStr]: + def readlines(self, hint: int = -1) -> list[AnyStr]: pass @abstractmethod @@ -3533,7 +3565,7 @@ def tell(self) -> int: pass @abstractmethod - def truncate(self, size: int = None) -> int: + def truncate(self, size: int | None = None) -> int: pass @abstractmethod @@ -3545,11 +3577,11 @@ def write(self, s: AnyStr) -> int: pass @abstractmethod - def writelines(self, lines: List[AnyStr]) -> None: + def writelines(self, lines: list[AnyStr]) -> None: pass @abstractmethod - def __enter__(self) -> 'IO[AnyStr]': + def __enter__(self) -> IO[AnyStr]: pass @abstractmethod @@ -3563,11 +3595,11 @@ class BinaryIO(IO[bytes]): __slots__ = () @abstractmethod - def write(self, s: Union[bytes, bytearray]) -> int: + def write(self, s: bytes | bytearray) -> int: pass @abstractmethod - def __enter__(self) -> 'BinaryIO': + def __enter__(self) -> BinaryIO: pass @@ -3588,7 +3620,7 @@ def encoding(self) -> str: @property @abstractmethod - def errors(self) -> Optional[str]: + def errors(self) -> str | None: pass @property @@ -3602,7 +3634,7 @@ def newlines(self) -> Any: pass @abstractmethod - def __enter__(self) -> 'TextIO': + def __enter__(self) -> TextIO: pass @@ -3798,7 +3830,9 @@ def __getattr__(attr): Soft-deprecated objects which are costly to create are only created on-demand here. """ - if attr in {"Pattern", "Match"}: + if attr == "ForwardRef": + obj = _lazy_annotationlib.ForwardRef + elif attr in {"Pattern", "Match"}: import re obj = _alias(getattr(re, attr), 1) elif attr in {"ContextManager", "AsyncContextManager"}: diff --git a/Lib/unittest/__init__.py b/Lib/unittest/__init__.py index 6878c2a8f5a..b049402eed7 100644 --- a/Lib/unittest/__init__.py +++ b/Lib/unittest/__init__.py @@ -27,7 +27,7 @@ def testMultiply(self): http://docs.python.org/library/unittest.html Copyright (c) 1999-2003 Steve Purcell -Copyright (c) 2003-2010 Python Software Foundation +Copyright (c) 2003 Python Software Foundation This module is free software, and you may redistribute it and/or modify it under the same terms as Python itself, so long as this copyright message and disclaimer are retained in their original form. @@ -53,15 +53,30 @@ def testMultiply(self): __unittest = True -from .result import TestResult -from .case import (addModuleCleanup, TestCase, FunctionTestCase, SkipTest, skip, - skipIf, skipUnless, expectedFailure, doModuleCleanups, - enterModuleContext) -from .suite import BaseTestSuite, TestSuite +from .case import ( + FunctionTestCase, + SkipTest, + TestCase, + addModuleCleanup, + doModuleCleanups, + enterModuleContext, + expectedFailure, + skip, + skipIf, + skipUnless, +) from .loader import TestLoader, defaultTestLoader -from .main import TestProgram, main -from .runner import TextTestRunner, TextTestResult -from .signals import installHandler, registerResult, removeResult, removeHandler +from .main import TestProgram, main # noqa: F401 +from .result import TestResult +from .runner import TextTestResult, TextTestRunner +from .signals import ( + installHandler, + registerResult, + removeHandler, + removeResult, +) +from .suite import BaseTestSuite, TestSuite # noqa: F401 + # IsolatedAsyncioTestCase will be imported lazily. diff --git a/Lib/unittest/__main__.py b/Lib/unittest/__main__.py index e5876f569b5..50111190eee 100644 --- a/Lib/unittest/__main__.py +++ b/Lib/unittest/__main__.py @@ -1,6 +1,7 @@ """Main entry point""" import sys + if sys.argv[0].endswith("__main__.py"): import os.path # We change sys.argv[0] to make help message more useful diff --git a/Lib/unittest/_log.py b/Lib/unittest/_log.py index 94868e5bb95..c61abb15745 100644 --- a/Lib/unittest/_log.py +++ b/Lib/unittest/_log.py @@ -1,9 +1,8 @@ -import logging import collections +import logging from .case import _BaseTestCaseContext - _LoggingWatcher = collections.namedtuple("_LoggingWatcher", ["records", "output"]) diff --git a/Lib/unittest/async_case.py b/Lib/unittest/async_case.py index e761ba7e53c..a1c0d6c368c 100644 --- a/Lib/unittest/async_case.py +++ b/Lib/unittest/async_case.py @@ -75,9 +75,17 @@ async def enterAsyncContext(self, cm): enter = cls.__aenter__ exit = cls.__aexit__ except AttributeError: - raise TypeError(f"'{cls.__module__}.{cls.__qualname__}' object does " - f"not support the asynchronous context manager protocol" - ) from None + msg = (f"'{cls.__module__}.{cls.__qualname__}' object does " + "not support the asynchronous context manager protocol") + try: + cls.__enter__ + cls.__exit__ + except AttributeError: + pass + else: + msg += (" but it supports the context manager protocol. " + "Did you mean to use enterContext()?") + raise TypeError(msg) from None result = await enter(cm) self.addAsyncCleanup(exit, cm, None, None, None) return result @@ -91,9 +99,13 @@ def _callSetUp(self): self._callAsync(self.asyncSetUp) def _callTestMethod(self, method): - if self._callMaybeAsync(method) is not None: - warnings.warn(f'It is deprecated to return a value that is not None from a ' - f'test case ({method})', DeprecationWarning, stacklevel=4) + result = self._callMaybeAsync(method) + if result is not None: + msg = ( + f'It is deprecated to return a value that is not None ' + f'from a test case ({method} returned {type(result).__name__!r})', + ) + warnings.warn(msg, DeprecationWarning, stacklevel=4) def _callTearDown(self): self._callAsync(self.asyncTearDown) diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py index 36daa61fa31..b09836d6747 100644 --- a/Lib/unittest/case.py +++ b/Lib/unittest/case.py @@ -1,20 +1,25 @@ """Test case implementation""" -import sys -import functools +import collections +import contextlib import difflib +import functools import pprint import re -import warnings -import collections -import contextlib -import traceback +import sys import time +import traceback import types +import warnings from . import result -from .util import (strclass, safe_repr, _count_diff_all_purpose, - _count_diff_hashable, _common_shorten_repr) +from .util import ( + _common_shorten_repr, + _count_diff_all_purpose, + _count_diff_hashable, + safe_repr, + strclass, +) __unittest = True @@ -111,8 +116,17 @@ def _enter_context(cm, addcleanup): enter = cls.__enter__ exit = cls.__exit__ except AttributeError: - raise TypeError(f"'{cls.__module__}.{cls.__qualname__}' object does " - f"not support the context manager protocol") from None + msg = (f"'{cls.__module__}.{cls.__qualname__}' object does " + "not support the context manager protocol") + try: + cls.__aenter__ + cls.__aexit__ + except AttributeError: + pass + else: + msg += (" but it supports the asynchronous context manager " + "protocol. Did you mean to use enterAsyncContext()?") + raise TypeError(msg) from None result = enter(cm) addcleanup(exit, cm, None, None, None) return result @@ -603,9 +617,18 @@ def _callSetUp(self): self.setUp() def _callTestMethod(self, method): - if method() is not None: - warnings.warn(f'It is deprecated to return a value that is not None from a ' - f'test case ({method})', DeprecationWarning, stacklevel=3) + result = method() + if result is not None: + import inspect + msg = ( + f'It is deprecated to return a value that is not None ' + f'from a test case ({method} returned {type(result).__name__!r})' + ) + if inspect.iscoroutine(result): + msg += ( + '. Maybe you forgot to use IsolatedAsyncioTestCase as the base class?' + ) + warnings.warn(msg, DeprecationWarning, stacklevel=3) def _callTearDown(self): self.tearDown() @@ -1312,13 +1335,71 @@ def assertIsInstance(self, obj, cls, msg=None): """Same as self.assertTrue(isinstance(obj, cls)), with a nicer default message.""" if not isinstance(obj, cls): - standardMsg = '%s is not an instance of %r' % (safe_repr(obj), cls) + if isinstance(cls, tuple): + standardMsg = f'{safe_repr(obj)} is not an instance of any of {cls!r}' + else: + standardMsg = f'{safe_repr(obj)} is not an instance of {cls!r}' self.fail(self._formatMessage(msg, standardMsg)) def assertNotIsInstance(self, obj, cls, msg=None): """Included for symmetry with assertIsInstance.""" if isinstance(obj, cls): - standardMsg = '%s is an instance of %r' % (safe_repr(obj), cls) + if isinstance(cls, tuple): + for x in cls: + if isinstance(obj, x): + cls = x + break + standardMsg = f'{safe_repr(obj)} is an instance of {cls!r}' + self.fail(self._formatMessage(msg, standardMsg)) + + def assertIsSubclass(self, cls, superclass, msg=None): + try: + if issubclass(cls, superclass): + return + except TypeError: + if not isinstance(cls, type): + self.fail(self._formatMessage(msg, f'{cls!r} is not a class')) + raise + if isinstance(superclass, tuple): + standardMsg = f'{cls!r} is not a subclass of any of {superclass!r}' + else: + standardMsg = f'{cls!r} is not a subclass of {superclass!r}' + self.fail(self._formatMessage(msg, standardMsg)) + + def assertNotIsSubclass(self, cls, superclass, msg=None): + try: + if not issubclass(cls, superclass): + return + except TypeError: + if not isinstance(cls, type): + self.fail(self._formatMessage(msg, f'{cls!r} is not a class')) + raise + if isinstance(superclass, tuple): + for x in superclass: + if issubclass(cls, x): + superclass = x + break + standardMsg = f'{cls!r} is a subclass of {superclass!r}' + self.fail(self._formatMessage(msg, standardMsg)) + + def assertHasAttr(self, obj, name, msg=None): + if not hasattr(obj, name): + if isinstance(obj, types.ModuleType): + standardMsg = f'module {obj.__name__!r} has no attribute {name!r}' + elif isinstance(obj, type): + standardMsg = f'type object {obj.__name__!r} has no attribute {name!r}' + else: + standardMsg = f'{type(obj).__name__!r} object has no attribute {name!r}' + self.fail(self._formatMessage(msg, standardMsg)) + + def assertNotHasAttr(self, obj, name, msg=None): + if hasattr(obj, name): + if isinstance(obj, types.ModuleType): + standardMsg = f'module {obj.__name__!r} has unexpected attribute {name!r}' + elif isinstance(obj, type): + standardMsg = f'type object {obj.__name__!r} has unexpected attribute {name!r}' + else: + standardMsg = f'{type(obj).__name__!r} object has unexpected attribute {name!r}' self.fail(self._formatMessage(msg, standardMsg)) def assertRaisesRegex(self, expected_exception, expected_regex, @@ -1382,6 +1463,80 @@ def assertNotRegex(self, text, unexpected_regex, msg=None): msg = self._formatMessage(msg, standardMsg) raise self.failureException(msg) + def _tail_type_check(self, s, tails, msg): + if not isinstance(tails, tuple): + tails = (tails,) + for tail in tails: + if isinstance(tail, str): + if not isinstance(s, str): + self.fail(self._formatMessage(msg, + f'Expected str, not {type(s).__name__}')) + elif isinstance(tail, (bytes, bytearray)): + if not isinstance(s, (bytes, bytearray)): + self.fail(self._formatMessage(msg, + f'Expected bytes, not {type(s).__name__}')) + + def assertStartsWith(self, s, prefix, msg=None): + try: + if s.startswith(prefix): + return + except (AttributeError, TypeError): + self._tail_type_check(s, prefix, msg) + raise + a = safe_repr(s, short=True) + b = safe_repr(prefix) + if isinstance(prefix, tuple): + standardMsg = f"{a} doesn't start with any of {b}" + else: + standardMsg = f"{a} doesn't start with {b}" + self.fail(self._formatMessage(msg, standardMsg)) + + def assertNotStartsWith(self, s, prefix, msg=None): + try: + if not s.startswith(prefix): + return + except (AttributeError, TypeError): + self._tail_type_check(s, prefix, msg) + raise + if isinstance(prefix, tuple): + for x in prefix: + if s.startswith(x): + prefix = x + break + a = safe_repr(s, short=True) + b = safe_repr(prefix) + self.fail(self._formatMessage(msg, f"{a} starts with {b}")) + + def assertEndsWith(self, s, suffix, msg=None): + try: + if s.endswith(suffix): + return + except (AttributeError, TypeError): + self._tail_type_check(s, suffix, msg) + raise + a = safe_repr(s, short=True) + b = safe_repr(suffix) + if isinstance(suffix, tuple): + standardMsg = f"{a} doesn't end with any of {b}" + else: + standardMsg = f"{a} doesn't end with {b}" + self.fail(self._formatMessage(msg, standardMsg)) + + def assertNotEndsWith(self, s, suffix, msg=None): + try: + if not s.endswith(suffix): + return + except (AttributeError, TypeError): + self._tail_type_check(s, suffix, msg) + raise + if isinstance(suffix, tuple): + for x in suffix: + if s.endswith(x): + suffix = x + break + a = safe_repr(s, short=True) + b = safe_repr(suffix) + self.fail(self._formatMessage(msg, f"{a} ends with {b}")) class FunctionTestCase(TestCase): diff --git a/Lib/unittest/loader.py b/Lib/unittest/loader.py index 22797b83a68..fa8d647ad8a 100644 --- a/Lib/unittest/loader.py +++ b/Lib/unittest/loader.py @@ -1,12 +1,11 @@ """Loading unittests.""" +import functools import os import re import sys import traceback import types -import functools - from fnmatch import fnmatch, fnmatchcase from . import case, suite, util @@ -274,6 +273,8 @@ def discover(self, start_dir, pattern='test*.py', top_level_dir=None): self._top_level_dir = top_level_dir is_not_importable = False + is_namespace = False + tests = [] if os.path.isdir(os.path.abspath(start_dir)): start_dir = os.path.abspath(start_dir) if start_dir != top_level_dir: @@ -286,12 +287,25 @@ def discover(self, start_dir, pattern='test*.py', top_level_dir=None): is_not_importable = True else: the_module = sys.modules[start_dir] - top_part = start_dir.split('.')[0] - try: - start_dir = os.path.abspath( - os.path.dirname((the_module.__file__))) - except AttributeError: - if the_module.__name__ in sys.builtin_module_names: + if not hasattr(the_module, "__file__") or the_module.__file__ is None: + # look for namespace packages + try: + spec = the_module.__spec__ + except AttributeError: + spec = None + + if spec and spec.submodule_search_locations is not None: + is_namespace = True + + for path in the_module.__path__: + if (not set_implicit_top and + not path.startswith(top_level_dir)): + continue + self._top_level_dir = \ + (path.split(the_module.__name__ + .replace(".", os.path.sep))[0]) + tests.extend(self._find_tests(path, pattern, namespace=True)) + elif the_module.__name__ in sys.builtin_module_names: # builtin module raise TypeError('Can not use builtin modules ' 'as dotted module names') from None @@ -300,14 +314,27 @@ def discover(self, start_dir, pattern='test*.py', top_level_dir=None): f"don't know how to discover from {the_module!r}" ) from None + else: + top_part = start_dir.split('.')[0] + start_dir = os.path.abspath(os.path.dirname((the_module.__file__))) + if set_implicit_top: - self._top_level_dir = self._get_directory_containing_module(top_part) + if not is_namespace: + if sys.modules[top_part].__file__ is None: + self._top_level_dir = os.path.dirname(the_module.__file__) + if self._top_level_dir not in sys.path: + sys.path.insert(0, self._top_level_dir) + else: + self._top_level_dir = \ + self._get_directory_containing_module(top_part) sys.path.remove(top_level_dir) if is_not_importable: raise ImportError('Start directory is not importable: %r' % start_dir) - tests = list(self._find_tests(start_dir, pattern)) + if not is_namespace: + tests = list(self._find_tests(start_dir, pattern)) + self._top_level_dir = original_top_level_dir return self.suiteClass(tests) @@ -343,7 +370,7 @@ def _match_path(self, path, full_path, pattern): # override this method to use alternative matching strategy return fnmatch(path, pattern) - def _find_tests(self, start_dir, pattern): + def _find_tests(self, start_dir, pattern, namespace=False): """Used by discovery. Yields test suites it loads.""" # Handle the __init__ in this package name = self._get_name_from_path(start_dir) @@ -352,7 +379,8 @@ def _find_tests(self, start_dir, pattern): if name != '.' and name not in self._loading_packages: # name is in self._loading_packages while we have called into # loadTestsFromModule with name. - tests, should_recurse = self._find_test_path(start_dir, pattern) + tests, should_recurse = self._find_test_path( + start_dir, pattern, namespace) if tests is not None: yield tests if not should_recurse: @@ -363,7 +391,8 @@ def _find_tests(self, start_dir, pattern): paths = sorted(os.listdir(start_dir)) for path in paths: full_path = os.path.join(start_dir, path) - tests, should_recurse = self._find_test_path(full_path, pattern) + tests, should_recurse = self._find_test_path( + full_path, pattern, False) if tests is not None: yield tests if should_recurse: @@ -371,11 +400,11 @@ def _find_tests(self, start_dir, pattern): name = self._get_name_from_path(full_path) self._loading_packages.add(name) try: - yield from self._find_tests(full_path, pattern) + yield from self._find_tests(full_path, pattern, False) finally: self._loading_packages.discard(name) - def _find_test_path(self, full_path, pattern): + def _find_test_path(self, full_path, pattern, namespace=False): """Used by discovery. Loads tests from a single file, or a directories' __init__.py when @@ -419,7 +448,8 @@ def _find_test_path(self, full_path, pattern): msg % (mod_name, module_dir, expected_dir)) return self.loadTestsFromModule(module, pattern=pattern), False elif os.path.isdir(full_path): - if not os.path.isfile(os.path.join(full_path, '__init__.py')): + if (not namespace and + not os.path.isfile(os.path.join(full_path, '__init__.py'))): return None, False load_tests = None diff --git a/Lib/unittest/main.py b/Lib/unittest/main.py index a0cd8a9f7ea..1855fccf336 100644 --- a/Lib/unittest/main.py +++ b/Lib/unittest/main.py @@ -1,8 +1,8 @@ """Unittest main program""" -import sys import argparse import os +import sys from . import loader, runner from .signals import installHandler @@ -197,7 +197,7 @@ def _getParentArgParser(self): return parser def _getMainArgParser(self, parent): - parser = argparse.ArgumentParser(parents=[parent]) + parser = argparse.ArgumentParser(parents=[parent], color=True) parser.prog = self.progName parser.print_help = self._print_help @@ -208,7 +208,7 @@ def _getMainArgParser(self, parent): return parser def _getDiscoveryArgParser(self, parent): - parser = argparse.ArgumentParser(parents=[parent]) + parser = argparse.ArgumentParser(parents=[parent], color=True) parser.prog = '%s discover' % self.progName parser.epilog = ('For test discovery all test modules must be ' 'importable from the top level directory of the ' diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py index 6cec61ff35c..1089dcb11f1 100644 --- a/Lib/unittest/mock.py +++ b/Lib/unittest/mock.py @@ -25,19 +25,20 @@ import asyncio +import builtins import contextlib -import io import inspect +import io +import pkgutil import pprint import sys -import builtins -import pkgutil -from asyncio import iscoroutinefunction import threading -from types import CodeType, ModuleType, MethodType -from unittest.util import safe_repr -from functools import wraps, partial +from dataclasses import fields, is_dataclass +from functools import partial, wraps +from inspect import iscoroutinefunction from threading import RLock +from types import CodeType, MethodType, ModuleType +from unittest.util import safe_repr class InvalidSpecError(Exception): @@ -568,6 +569,11 @@ def _mock_add_spec(self, spec, spec_set, _spec_as_instance=False, __dict__['_mock_methods'] = spec __dict__['_spec_asyncs'] = _spec_asyncs + def _mock_extend_spec_methods(self, spec_methods): + methods = self.__dict__.get('_mock_methods') or [] + methods.extend(spec_methods) + self.__dict__['_mock_methods'] = methods + def __get_return_value(self): ret = self._mock_return_value if self._mock_delegate is not None: @@ -1766,7 +1772,7 @@ def patch( the patch is undone. If `new` is omitted, then the target is replaced with an - `AsyncMock if the patched object is an async function or a + `AsyncMock` if the patched object is an async function or a `MagicMock` otherwise. If `patch` is used as a decorator and `new` is omitted, the created mock is passed in as an extra argument to the decorated function. If `patch` is used as a context manager the created @@ -1840,7 +1846,8 @@ def patch( class _patch_dict(object): """ Patch a dictionary, or dictionary like object, and restore the dictionary - to its original state after the test. + to its original state after the test, where the restored dictionary is + a copy of the dictionary as it was before the test. `in_dict` can be a dictionary or a mapping like container. If it is a mapping then it must at least support getting, setting and deleting items @@ -2176,8 +2183,6 @@ def _mock_set_magics(self): if getattr(self, "_mock_methods", None) is not None: these_magics = orig_magics.intersection(self._mock_methods) - - remove_magics = set() remove_magics = orig_magics - these_magics for entry in remove_magics: @@ -2477,7 +2482,7 @@ class AsyncMock(AsyncMockMixin, AsyncMagicMixin, Mock): recognized as an async function, and the result of a call is an awaitable: >>> mock = AsyncMock() - >>> iscoroutinefunction(mock) + >>> inspect.iscoroutinefunction(mock) True >>> inspect.isawaitable(mock()) True @@ -2767,6 +2772,16 @@ def create_autospec(spec, spec_set=False, instance=False, _parent=None, f'[object={spec!r}]') is_async_func = _is_async_func(spec) _kwargs = {'spec': spec} + + entries = [(entry, _missing) for entry in dir(spec)] + if is_type and instance and is_dataclass(spec): + is_dataclass_spec = True + dataclass_fields = fields(spec) + entries.extend((f.name, f.type) for f in dataclass_fields) + dataclass_spec_list = [f.name for f in dataclass_fields] + else: + is_dataclass_spec = False + if spec_set: _kwargs = {'spec_set': spec} elif spec is None: @@ -2802,6 +2817,8 @@ def create_autospec(spec, spec_set=False, instance=False, _parent=None, mock = Klass(parent=_parent, _new_parent=_parent, _new_name=_new_name, name=_name, **_kwargs) + if is_dataclass_spec: + mock._mock_extend_spec_methods(dataclass_spec_list) if isinstance(spec, FunctionTypes): # should only happen at the top level because we don't @@ -2823,7 +2840,7 @@ def create_autospec(spec, spec_set=False, instance=False, _parent=None, _name='()', _parent=mock, wraps=wrapped) - for entry in dir(spec): + for entry, original in entries: if _is_magic(entry): # MagicMock already does the useful magic methods for us continue @@ -2837,10 +2854,11 @@ def create_autospec(spec, spec_set=False, instance=False, _parent=None, # AttributeError on being fetched? # we could be resilient against it, or catch and propagate the # exception when the attribute is fetched from the mock - try: - original = getattr(spec, entry) - except AttributeError: - continue + if original is _missing: + try: + original = getattr(spec, entry) + except AttributeError: + continue child_kwargs = {'spec': original} # Wrap child attributes also. diff --git a/Lib/unittest/result.py b/Lib/unittest/result.py index 3ace0a5b7bf..8eafb3891c9 100644 --- a/Lib/unittest/result.py +++ b/Lib/unittest/result.py @@ -3,9 +3,9 @@ import io import sys import traceback +from functools import wraps from . import util -from functools import wraps __unittest = True @@ -189,7 +189,10 @@ def _exc_info_to_string(self, err, test): tb_e = traceback.TracebackException( exctype, value, tb, capture_locals=self.tb_locals, compact=True) - msgLines = list(tb_e.format()) + from _colorize import can_colorize + + colorize = hasattr(self, "stream") and can_colorize(file=self.stream) + msgLines = list(tb_e.format(colorize=colorize)) if self.buffer: output = sys.stdout.getvalue() diff --git a/Lib/unittest/runner.py b/Lib/unittest/runner.py index 2bcadf0c998..5f22d91aebd 100644 --- a/Lib/unittest/runner.py +++ b/Lib/unittest/runner.py @@ -4,6 +4,8 @@ import time import warnings +from _colorize import get_theme + from . import result from .case import _SubTest from .signals import registerResult @@ -13,18 +15,18 @@ class _WritelnDecorator(object): """Used to decorate file-like objects with a handy 'writeln' method""" - def __init__(self,stream): + def __init__(self, stream): self.stream = stream def __getattr__(self, attr): if attr in ('stream', '__getstate__'): raise AttributeError(attr) - return getattr(self.stream,attr) + return getattr(self.stream, attr) def writeln(self, arg=None): if arg: self.write(arg) - self.write('\n') # text-mode streams translate to \r\n if needed + self.write('\n') # text-mode streams translate to \r\n if needed class TextTestResult(result.TestResult): @@ -43,6 +45,7 @@ def __init__(self, stream, descriptions, verbosity, *, durations=None): self.showAll = verbosity > 1 self.dots = verbosity == 1 self.descriptions = descriptions + self._theme = get_theme(tty_file=stream).unittest self._newline = True self.durations = durations @@ -76,86 +79,100 @@ def _write_status(self, test, status): def addSubTest(self, test, subtest, err): if err is not None: + t = self._theme if self.showAll: if issubclass(err[0], subtest.failureException): - self._write_status(subtest, "FAIL") + self._write_status(subtest, f"{t.fail}FAIL{t.reset}") else: - self._write_status(subtest, "ERROR") + self._write_status(subtest, f"{t.fail}ERROR{t.reset}") elif self.dots: if issubclass(err[0], subtest.failureException): - self.stream.write('F') + self.stream.write(f"{t.fail}F{t.reset}") else: - self.stream.write('E') + self.stream.write(f"{t.fail}E{t.reset}") self.stream.flush() super(TextTestResult, self).addSubTest(test, subtest, err) def addSuccess(self, test): super(TextTestResult, self).addSuccess(test) + t = self._theme if self.showAll: - self._write_status(test, "ok") + self._write_status(test, f"{t.passed}ok{t.reset}") elif self.dots: - self.stream.write('.') + self.stream.write(f"{t.passed}.{t.reset}") self.stream.flush() def addError(self, test, err): super(TextTestResult, self).addError(test, err) + t = self._theme if self.showAll: - self._write_status(test, "ERROR") + self._write_status(test, f"{t.fail}ERROR{t.reset}") elif self.dots: - self.stream.write('E') + self.stream.write(f"{t.fail}E{t.reset}") self.stream.flush() def addFailure(self, test, err): super(TextTestResult, self).addFailure(test, err) + t = self._theme if self.showAll: - self._write_status(test, "FAIL") + self._write_status(test, f"{t.fail}FAIL{t.reset}") elif self.dots: - self.stream.write('F') + self.stream.write(f"{t.fail}F{t.reset}") self.stream.flush() def addSkip(self, test, reason): super(TextTestResult, self).addSkip(test, reason) + t = self._theme if self.showAll: - self._write_status(test, "skipped {0!r}".format(reason)) + self._write_status(test, f"{t.warn}skipped{t.reset} {reason!r}") elif self.dots: - self.stream.write("s") + self.stream.write(f"{t.warn}s{t.reset}") self.stream.flush() def addExpectedFailure(self, test, err): super(TextTestResult, self).addExpectedFailure(test, err) + t = self._theme if self.showAll: - self.stream.writeln("expected failure") + self.stream.writeln(f"{t.warn}expected failure{t.reset}") self.stream.flush() elif self.dots: - self.stream.write("x") + self.stream.write(f"{t.warn}x{t.reset}") self.stream.flush() def addUnexpectedSuccess(self, test): super(TextTestResult, self).addUnexpectedSuccess(test) + t = self._theme if self.showAll: - self.stream.writeln("unexpected success") + self.stream.writeln(f"{t.fail}unexpected success{t.reset}") self.stream.flush() elif self.dots: - self.stream.write("u") + self.stream.write(f"{t.fail}u{t.reset}") self.stream.flush() def printErrors(self): + t = self._theme if self.dots or self.showAll: self.stream.writeln() self.stream.flush() - self.printErrorList('ERROR', self.errors) - self.printErrorList('FAIL', self.failures) - unexpectedSuccesses = getattr(self, 'unexpectedSuccesses', ()) + self.printErrorList(f"{t.fail}ERROR{t.reset}", self.errors) + self.printErrorList(f"{t.fail}FAIL{t.reset}", self.failures) + unexpectedSuccesses = getattr(self, "unexpectedSuccesses", ()) if unexpectedSuccesses: self.stream.writeln(self.separator1) for test in unexpectedSuccesses: - self.stream.writeln(f"UNEXPECTED SUCCESS: {self.getDescription(test)}") + self.stream.writeln( + f"{t.fail}UNEXPECTED SUCCESS{t.fail_info}: " + f"{self.getDescription(test)}{t.reset}" + ) self.stream.flush() def printErrorList(self, flavour, errors): + t = self._theme for test, err in errors: self.stream.writeln(self.separator1) - self.stream.writeln("%s: %s" % (flavour,self.getDescription(test))) + self.stream.writeln( + f"{flavour}{t.fail_info}: {self.getDescription(test)}{t.reset}" + ) self.stream.writeln(self.separator2) self.stream.writeln("%s" % err) self.stream.flush() @@ -232,7 +249,7 @@ def run(self, test): if self.warnings: # if self.warnings is set, use it to filter all the warnings warnings.simplefilter(self.warnings) - startTime = time.perf_counter() + start_time = time.perf_counter() startTestRun = getattr(result, 'startTestRun', None) if startTestRun is not None: startTestRun() @@ -242,8 +259,8 @@ def run(self, test): stopTestRun = getattr(result, 'stopTestRun', None) if stopTestRun is not None: stopTestRun() - stopTime = time.perf_counter() - timeTaken = stopTime - startTime + stop_time = time.perf_counter() + time_taken = stop_time - start_time result.printErrors() if self.durations is not None: self._printDurations(result) @@ -253,10 +270,10 @@ def run(self, test): run = result.testsRun self.stream.writeln("Ran %d test%s in %.3fs" % - (run, run != 1 and "s" or "", timeTaken)) + (run, run != 1 and "s" or "", time_taken)) self.stream.writeln() - expectedFails = unexpectedSuccesses = skipped = 0 + expected_fails = unexpected_successes = skipped = 0 try: results = map(len, (result.expectedFailures, result.unexpectedSuccesses, @@ -264,26 +281,30 @@ def run(self, test): except AttributeError: pass else: - expectedFails, unexpectedSuccesses, skipped = results + expected_fails, unexpected_successes, skipped = results infos = [] + t = get_theme(tty_file=self.stream).unittest + if not result.wasSuccessful(): - self.stream.write("FAILED") + self.stream.write(f"{t.fail_info}FAILED{t.reset}") failed, errored = len(result.failures), len(result.errors) if failed: - infos.append("failures=%d" % failed) + infos.append(f"{t.fail_info}failures={failed}{t.reset}") if errored: - infos.append("errors=%d" % errored) + infos.append(f"{t.fail_info}errors={errored}{t.reset}") elif run == 0 and not skipped: - self.stream.write("NO TESTS RAN") + self.stream.write(f"{t.warn}NO TESTS RAN{t.reset}") else: - self.stream.write("OK") + self.stream.write(f"{t.passed}OK{t.reset}") if skipped: - infos.append("skipped=%d" % skipped) - if expectedFails: - infos.append("expected failures=%d" % expectedFails) - if unexpectedSuccesses: - infos.append("unexpected successes=%d" % unexpectedSuccesses) + infos.append(f"{t.warn}skipped={skipped}{t.reset}") + if expected_fails: + infos.append(f"{t.warn}expected failures={expected_fails}{t.reset}") + if unexpected_successes: + infos.append( + f"{t.fail}unexpected successes={unexpected_successes}{t.reset}" + ) if infos: self.stream.writeln(" (%s)" % (", ".join(infos),)) else: diff --git a/Lib/unittest/signals.py b/Lib/unittest/signals.py index e6a5fc52439..4e654c2c5db 100644 --- a/Lib/unittest/signals.py +++ b/Lib/unittest/signals.py @@ -1,6 +1,5 @@ import signal import weakref - from functools import wraps __unittest = True diff --git a/Lib/unittest/suite.py b/Lib/unittest/suite.py index 6f45b6fe5f6..3c40176f070 100644 --- a/Lib/unittest/suite.py +++ b/Lib/unittest/suite.py @@ -2,8 +2,7 @@ import sys -from . import case -from . import util +from . import case, util __unittest = True diff --git a/Lib/unittest/util.py b/Lib/unittest/util.py index 050eaed0b3f..b81b6a4219b 100644 --- a/Lib/unittest/util.py +++ b/Lib/unittest/util.py @@ -1,6 +1,6 @@ """Various utility functions.""" -from collections import namedtuple, Counter +from collections import Counter, namedtuple from os.path import commonprefix __unittest = True diff --git a/Lib/urllib/error.py b/Lib/urllib/error.py index 8cd901f13f8..a9cd1ecadd6 100644 --- a/Lib/urllib/error.py +++ b/Lib/urllib/error.py @@ -10,7 +10,7 @@ an application may want to handle an exception like a regular response. """ - +import io import urllib.response __all__ = ['URLError', 'HTTPError', 'ContentTooShortError'] @@ -42,12 +42,9 @@ def __init__(self, url, code, msg, hdrs, fp): self.hdrs = hdrs self.fp = fp self.filename = url - # The addinfourl classes depend on fp being a valid file - # object. In some cases, the HTTPError may not have a valid - # file object. If this happens, the simplest workaround is to - # not initialize the base classes. - if fp is not None: - self.__super_init(fp, hdrs, url, code) + if fp is None: + fp = io.BytesIO() + self.__super_init(fp, hdrs, url, code) def __str__(self): return 'HTTP Error %s: %s' % (self.code, self.msg) diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py index b35997bc00c..67d9bbea0d3 100644 --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -25,13 +25,19 @@ scenarios for parsing, and for backward compatibility purposes, some parsing quirks from older RFCs are retained. The testcases in test_urlparse.py provides a good indicator of parsing behavior. + +The WHATWG URL Parser spec should also be considered. We are not compliant with +it either due to existing user code API behavior expectations (Hyrum's Law). +It serves as a useful guide when making changes. """ +from collections import namedtuple +import functools +import math import re -import sys import types -import collections import warnings +import ipaddress __all__ = ["urlparse", "urlunparse", "urljoin", "urldefrag", "urlsplit", "urlunsplit", "urlencode", "parse_qs", @@ -46,18 +52,18 @@ uses_relative = ['', 'ftp', 'http', 'gopher', 'nntp', 'imap', 'wais', 'file', 'https', 'shttp', 'mms', - 'prospero', 'rtsp', 'rtspu', 'sftp', + 'prospero', 'rtsp', 'rtsps', 'rtspu', 'sftp', 'svn', 'svn+ssh', 'ws', 'wss'] uses_netloc = ['', 'ftp', 'http', 'gopher', 'nntp', 'telnet', 'imap', 'wais', 'file', 'mms', 'https', 'shttp', - 'snews', 'prospero', 'rtsp', 'rtspu', 'rsync', + 'snews', 'prospero', 'rtsp', 'rtsps', 'rtspu', 'rsync', 'svn', 'svn+ssh', 'sftp', 'nfs', 'git', 'git+ssh', - 'ws', 'wss'] + 'ws', 'wss', 'itms-services'] uses_params = ['', 'ftp', 'hdl', 'prospero', 'http', 'imap', - 'https', 'shttp', 'rtsp', 'rtspu', 'sip', 'sips', - 'mms', 'sftp', 'tel'] + 'https', 'shttp', 'rtsp', 'rtsps', 'rtspu', 'sip', + 'sips', 'mms', 'sftp', 'tel'] # These are not actually used anymore, but should stay for backwards # compatibility. (They are undocumented, but have a public-looking name.) @@ -66,7 +72,7 @@ 'telnet', 'wais', 'imap', 'snews', 'sip', 'sips'] uses_query = ['', 'http', 'wais', 'imap', 'https', 'shttp', 'mms', - 'gopher', 'rtsp', 'rtspu', 'sip', 'sips'] + 'gopher', 'rtsp', 'rtsps', 'rtspu', 'sip', 'sips'] uses_fragment = ['', 'ftp', 'hdl', 'http', 'gopher', 'news', 'nntp', 'wais', 'https', 'shttp', 'snews', @@ -78,18 +84,17 @@ '0123456789' '+-.') +# Leading and trailing C0 control and space to be stripped per WHATWG spec. +# == "".join([chr(i) for i in range(0, 0x20 + 1)]) +_WHATWG_C0_CONTROL_OR_SPACE = '\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f ' + # Unsafe bytes to be removed per WHATWG spec _UNSAFE_URL_BYTES_TO_REMOVE = ['\t', '\r', '\n'] -# XXX: Consider replacing with functools.lru_cache -MAX_CACHE_SIZE = 20 -_parse_cache = {} - def clear_cache(): - """Clear the parse cache and the quoters cache.""" - _parse_cache.clear() - _safe_quoters.clear() - + """Clear internal performance caches. Undocumented; some tests want it.""" + urlsplit.cache_clear() + _byte_quoter_factory.cache_clear() # Helpers for bytes handling # For 3.2, we deliberately require applications that @@ -171,12 +176,11 @@ def hostname(self): def port(self): port = self._hostinfo[1] if port is not None: - try: - port = int(port, 10) - except ValueError: - message = f'Port could not be cast to integer value as {port!r}' - raise ValueError(message) from None - if not ( 0 <= port <= 65535): + if port.isdigit() and port.isascii(): + port = int(port) + else: + raise ValueError(f"Port could not be cast to integer value as {port!r}") + if not (0 <= port <= 65535): raise ValueError("Port out of range 0-65535") return port @@ -243,13 +247,11 @@ def _hostinfo(self): return hostname, port -from collections import namedtuple - -_DefragResultBase = namedtuple('DefragResult', 'url fragment') +_DefragResultBase = namedtuple('_DefragResultBase', 'url fragment') _SplitResultBase = namedtuple( - 'SplitResult', 'scheme netloc path query fragment') + '_SplitResultBase', 'scheme netloc path query fragment') _ParseResultBase = namedtuple( - 'ParseResult', 'scheme netloc path params query fragment') + '_ParseResultBase', 'scheme netloc path params query fragment') _DefragResultBase.__doc__ = """ DefragResult(url, fragment) @@ -390,20 +392,23 @@ def urlparse(url, scheme='', allow_fragments=True): Note that % escapes are not expanded. """ url, scheme, _coerce_result = _coerce_args(url, scheme) - splitresult = urlsplit(url, scheme, allow_fragments) - scheme, netloc, url, query, fragment = splitresult - if scheme in uses_params and ';' in url: - url, params = _splitparams(url) - else: - params = '' - result = ParseResult(scheme, netloc, url, params, query, fragment) + scheme, netloc, url, params, query, fragment = _urlparse(url, scheme, allow_fragments) + result = ParseResult(scheme or '', netloc or '', url, params or '', query or '', fragment or '') return _coerce_result(result) -def _splitparams(url): +def _urlparse(url, scheme=None, allow_fragments=True): + scheme, netloc, url, query, fragment = _urlsplit(url, scheme, allow_fragments) + if (scheme or '') in uses_params and ';' in url: + url, params = _splitparams(url, allow_none=True) + else: + params = None + return (scheme, netloc, url, params, query, fragment) + +def _splitparams(url, allow_none=False): if '/' in url: i = url.find(';', url.rfind('/')) if i < 0: - return url, '' + return url, None if allow_none else '' else: i = url.find(';') return url[:i], url[i+1:] @@ -434,6 +439,37 @@ def _checknetloc(netloc): raise ValueError("netloc '" + netloc + "' contains invalid " + "characters under NFKC normalization") +def _check_bracketed_netloc(netloc): + # Note that this function must mirror the splitting + # done in NetlocResultMixins._hostinfo(). + hostname_and_port = netloc.rpartition('@')[2] + before_bracket, have_open_br, bracketed = hostname_and_port.partition('[') + if have_open_br: + # No data is allowed before a bracket. + if before_bracket: + raise ValueError("Invalid IPv6 URL") + hostname, _, port = bracketed.partition(']') + # No data is allowed after the bracket but before the port delimiter. + if port and not port.startswith(":"): + raise ValueError("Invalid IPv6 URL") + else: + hostname, _, port = hostname_and_port.partition(':') + _check_bracketed_host(hostname) + +# Valid bracketed hosts are defined in +# https://www.rfc-editor.org/rfc/rfc3986#page-49 and https://url.spec.whatwg.org/ +def _check_bracketed_host(hostname): + if hostname.startswith('v'): + if not re.match(r"\Av[a-fA-F0-9]+\..+\z", hostname): + raise ValueError(f"IPvFuture address is invalid") + else: + ip = ipaddress.ip_address(hostname) # Throws Value Error if not IPv6 or IPv4 + if isinstance(ip, ipaddress.IPv4Address): + raise ValueError(f"An IPv4 address cannot be in brackets") + +# typed=True avoids BytesWarnings being emitted during cache key +# comparison since this API supports both bytes and str input. +@functools.lru_cache(typed=True) def urlsplit(url, scheme='', allow_fragments=True): """Parse a URL into 5 components: <scheme>://<netloc>/<path>?<query>#<fragment> @@ -456,40 +492,43 @@ def urlsplit(url, scheme='', allow_fragments=True): """ url, scheme, _coerce_result = _coerce_args(url, scheme) + scheme, netloc, url, query, fragment = _urlsplit(url, scheme, allow_fragments) + v = SplitResult(scheme or '', netloc or '', url, query or '', fragment or '') + return _coerce_result(v) +def _urlsplit(url, scheme=None, allow_fragments=True): + # Only lstrip url as some applications rely on preserving trailing space. + # (https://url.spec.whatwg.org/#concept-basic-url-parser would strip both) + url = url.lstrip(_WHATWG_C0_CONTROL_OR_SPACE) for b in _UNSAFE_URL_BYTES_TO_REMOVE: url = url.replace(b, "") - scheme = scheme.replace(b, "") + if scheme is not None: + scheme = scheme.strip(_WHATWG_C0_CONTROL_OR_SPACE) + for b in _UNSAFE_URL_BYTES_TO_REMOVE: + scheme = scheme.replace(b, "") allow_fragments = bool(allow_fragments) - key = url, scheme, allow_fragments, type(url), type(scheme) - cached = _parse_cache.get(key, None) - if cached: - return _coerce_result(cached) - if len(_parse_cache) >= MAX_CACHE_SIZE: # avoid runaway growth - clear_cache() - netloc = query = fragment = '' + netloc = query = fragment = None i = url.find(':') - if i > 0: + if i > 0 and url[0].isascii() and url[0].isalpha(): for c in url[:i]: if c not in scheme_chars: break else: scheme, url = url[:i].lower(), url[i+1:] - if url[:2] == '//': netloc, url = _splitnetloc(url, 2) if (('[' in netloc and ']' not in netloc) or (']' in netloc and '[' not in netloc)): raise ValueError("Invalid IPv6 URL") + if '[' in netloc and ']' in netloc: + _check_bracketed_netloc(netloc) if allow_fragments and '#' in url: url, fragment = url.split('#', 1) if '?' in url: url, query = url.split('?', 1) _checknetloc(netloc) - v = SplitResult(scheme, netloc, url, query, fragment) - _parse_cache[key] = v - return _coerce_result(v) + return (scheme, netloc, url, query, fragment) def urlunparse(components): """Put a parsed URL back together again. This may result in a @@ -498,9 +537,15 @@ def urlunparse(components): (the draft states that these are equivalent).""" scheme, netloc, url, params, query, fragment, _coerce_result = ( _coerce_args(*components)) + if not netloc: + if scheme and scheme in uses_netloc and (not url or url[:1] == '/'): + netloc = '' + else: + netloc = None if params: url = "%s;%s" % (url, params) - return _coerce_result(urlunsplit((scheme, netloc, url, query, fragment))) + return _coerce_result(_urlunsplit(scheme or None, netloc, url, + query or None, fragment or None)) def urlunsplit(components): """Combine the elements of a tuple as returned by urlsplit() into a @@ -510,16 +555,27 @@ def urlunsplit(components): empty query; the RFC states that these are equivalent).""" scheme, netloc, url, query, fragment, _coerce_result = ( _coerce_args(*components)) - if netloc or (scheme and scheme in uses_netloc and url[:2] != '//'): + if not netloc: + if scheme and scheme in uses_netloc and (not url or url[:1] == '/'): + netloc = '' + else: + netloc = None + return _coerce_result(_urlunsplit(scheme or None, netloc, url, + query or None, fragment or None)) + +def _urlunsplit(scheme, netloc, url, query, fragment): + if netloc is not None: if url and url[:1] != '/': url = '/' + url - url = '//' + (netloc or '') + url + url = '//' + netloc + url + elif url[:2] == '//': + url = '//' + url if scheme: url = scheme + ':' + url - if query: + if query is not None: url = url + '?' + query - if fragment: + if fragment is not None: url = url + '#' + fragment - return _coerce_result(url) + return url def urljoin(base, url, allow_fragments=True): """Join a base URL and a possibly relative URL to form an absolute @@ -530,26 +586,29 @@ def urljoin(base, url, allow_fragments=True): return base base, url, _coerce_result = _coerce_args(base, url) - bscheme, bnetloc, bpath, bparams, bquery, bfragment = \ - urlparse(base, '', allow_fragments) - scheme, netloc, path, params, query, fragment = \ - urlparse(url, bscheme, allow_fragments) - - if scheme != bscheme or scheme not in uses_relative: + bscheme, bnetloc, bpath, bquery, bfragment = \ + _urlsplit(base, None, allow_fragments) + scheme, netloc, path, query, fragment = \ + _urlsplit(url, None, allow_fragments) + + if scheme is None: + scheme = bscheme + if scheme != bscheme or (scheme and scheme not in uses_relative): return _coerce_result(url) - if scheme in uses_netloc: + if not scheme or scheme in uses_netloc: if netloc: - return _coerce_result(urlunparse((scheme, netloc, path, - params, query, fragment))) + return _coerce_result(_urlunsplit(scheme, netloc, path, + query, fragment)) netloc = bnetloc - if not path and not params: + if not path: path = bpath - params = bparams - if not query: + if query is None: query = bquery - return _coerce_result(urlunparse((scheme, netloc, path, - params, query, fragment))) + if fragment is None: + fragment = bfragment + return _coerce_result(_urlunsplit(scheme, netloc, path, + query, fragment)) base_parts = bpath.split('/') if base_parts[-1] != '': @@ -586,8 +645,8 @@ def urljoin(base, url, allow_fragments=True): # then we need to append the trailing '/' resolved_path.append('') - return _coerce_result(urlunparse((scheme, netloc, '/'.join( - resolved_path) or '/', params, query, fragment))) + return _coerce_result(_urlunsplit(scheme, netloc, '/'.join( + resolved_path) or '/', query, fragment)) def urldefrag(url): @@ -599,18 +658,21 @@ def urldefrag(url): """ url, _coerce_result = _coerce_args(url) if '#' in url: - s, n, p, a, q, frag = urlparse(url) - defrag = urlunparse((s, n, p, a, q, '')) + s, n, p, q, frag = _urlsplit(url) + defrag = _urlunsplit(s, n, p, q, None) else: frag = '' defrag = url - return _coerce_result(DefragResult(defrag, frag)) + return _coerce_result(DefragResult(defrag, frag or '')) _hexdig = '0123456789ABCDEFabcdef' _hextobyte = None def unquote_to_bytes(string): """unquote_to_bytes('abc%20def') -> b'abc def'.""" + return bytes(_unquote_impl(string)) + +def _unquote_impl(string: bytes | bytearray | str) -> bytes | bytearray: # Note: strings are encoded as UTF-8. This is only an issue if it contains # unescaped non-ASCII characters, which URIs should not. if not string: @@ -622,8 +684,8 @@ def unquote_to_bytes(string): bits = string.split(b'%') if len(bits) == 1: return string - res = [bits[0]] - append = res.append + res = bytearray(bits[0]) + append = res.extend # Delay the initialization of the table to not waste memory # if the function is never called global _hextobyte @@ -637,10 +699,20 @@ def unquote_to_bytes(string): except KeyError: append(b'%') append(item) - return b''.join(res) + return res _asciire = re.compile('([\x00-\x7f]+)') +def _generate_unquoted_parts(string, encoding, errors): + previous_match_end = 0 + for ascii_match in _asciire.finditer(string): + start, end = ascii_match.span() + yield string[previous_match_end:start] # Non-ASCII + # The ascii_match[1] group == string[start:end]. + yield _unquote_impl(ascii_match[1]).decode(encoding, errors) + previous_match_end = end + yield string[previous_match_end:] # Non-ASCII tail + def unquote(string, encoding='utf-8', errors='replace'): """Replace %xx escapes by their single-character equivalent. The optional encoding and errors parameters specify how to decode percent-encoded @@ -652,21 +724,16 @@ def unquote(string, encoding='utf-8', errors='replace'): unquote('abc%20def') -> 'abc def'. """ if isinstance(string, bytes): - return unquote_to_bytes(string).decode(encoding, errors) + return _unquote_impl(string).decode(encoding, errors) if '%' not in string: + # Is it a string-like object? string.split return string if encoding is None: encoding = 'utf-8' if errors is None: errors = 'replace' - bits = _asciire.split(string) - res = [bits[0]] - append = res.append - for i in range(1, len(bits), 2): - append(unquote_to_bytes(bits[i]).decode(encoding, errors)) - append(bits[i + 1]) - return ''.join(res) + return ''.join(_generate_unquoted_parts(string, encoding, errors)) def parse_qs(qs, keep_blank_values=False, strict_parsing=False, @@ -702,7 +769,8 @@ def parse_qs(qs, keep_blank_values=False, strict_parsing=False, parsed_result = {} pairs = parse_qsl(qs, keep_blank_values, strict_parsing, encoding=encoding, errors=errors, - max_num_fields=max_num_fields, separator=separator) + max_num_fields=max_num_fields, separator=separator, + _stacklevel=2) for name, value in pairs: if name in parsed_result: parsed_result[name].append(value) @@ -712,7 +780,7 @@ def parse_qs(qs, keep_blank_values=False, strict_parsing=False, def parse_qsl(qs, keep_blank_values=False, strict_parsing=False, - encoding='utf-8', errors='replace', max_num_fields=None, separator='&'): + encoding='utf-8', errors='replace', max_num_fields=None, separator='&', *, _stacklevel=1): """Parse a query given as a string argument. Arguments: @@ -740,11 +808,37 @@ def parse_qsl(qs, keep_blank_values=False, strict_parsing=False, Returns a list, as G-d intended. """ - qs, _coerce_result = _coerce_args(qs) - separator, _ = _coerce_args(separator) - - if not separator or (not isinstance(separator, (str, bytes))): + if not separator or not isinstance(separator, (str, bytes)): raise ValueError("Separator must be of type string or bytes.") + if isinstance(qs, str): + if not isinstance(separator, str): + separator = str(separator, 'ascii') + eq = '=' + def _unquote(s): + return unquote_plus(s, encoding=encoding, errors=errors) + elif qs is None: + return [] + else: + try: + # Use memoryview() to reject integers and iterables, + # acceptable by the bytes constructor. + qs = bytes(memoryview(qs)) + except TypeError: + if not qs: + warnings.warn(f"Accepting {type(qs).__name__} objects with " + f"false value in urllib.parse.parse_qsl() is " + f"deprecated as of 3.14", + DeprecationWarning, stacklevel=_stacklevel + 1) + return [] + raise + if isinstance(separator, str): + separator = bytes(separator, 'ascii') + eq = b'=' + def _unquote(s): + return unquote_to_bytes(s.replace(b'+', b' ')) + + if not qs: + return [] # If max_num_fields is defined then check that the number of fields # is less than max_num_fields. This prevents a memory exhaustion DOS @@ -756,25 +850,14 @@ def parse_qsl(qs, keep_blank_values=False, strict_parsing=False, r = [] for name_value in qs.split(separator): - if not name_value and not strict_parsing: - continue - nv = name_value.split('=', 1) - if len(nv) != 2: - if strict_parsing: + if name_value or strict_parsing: + name, has_eq, value = name_value.partition(eq) + if not has_eq and strict_parsing: raise ValueError("bad query field: %r" % (name_value,)) - # Handle case of a control-name with no equal sign - if keep_blank_values: - nv.append('') - else: - continue - if len(nv[1]) or keep_blank_values: - name = nv[0].replace('+', ' ') - name = unquote(name, encoding=encoding, errors=errors) - name = _coerce_result(name) - value = nv[1].replace('+', ' ') - value = unquote(value, encoding=encoding, errors=errors) - value = _coerce_result(value) - r.append((name, value)) + if value or keep_blank_values: + name = _unquote(name) + value = _unquote(value) + r.append((name, value)) return r def unquote_plus(string, encoding='utf-8', errors='replace'): @@ -791,23 +874,22 @@ def unquote_plus(string, encoding='utf-8', errors='replace'): b'0123456789' b'_.-~') _ALWAYS_SAFE_BYTES = bytes(_ALWAYS_SAFE) -_safe_quoters = {} -class Quoter(collections.defaultdict): - """A mapping from bytes (in range(0,256)) to strings. + +class _Quoter(dict): + """A mapping from bytes numbers (in range(0,256)) to strings. String values are percent-encoded byte values, unless the key < 128, and - in the "safe" set (either the specified safe set, or default set). + in either of the specified safe set, or the always safe set. """ - # Keeps a cache internally, using defaultdict, for efficiency (lookups + # Keeps a cache internally, via __missing__, for efficiency (lookups # of cached keys don't call Python code at all). def __init__(self, safe): """safe: bytes object.""" self.safe = _ALWAYS_SAFE.union(safe) def __repr__(self): - # Without this, will just display as a defaultdict - return "<%s %r>" % (self.__class__.__name__, dict(self)) + return f"<Quoter {dict(self)!r}>" def __missing__(self, b): # Handle a cache miss. Store quoted string in cache and return. @@ -886,6 +968,11 @@ def quote_plus(string, safe='', encoding=None, errors=None): string = quote(string, safe + space, encoding, errors) return string.replace(' ', '+') +# Expectation: A typical program is unlikely to create more than 5 of these. +@functools.lru_cache +def _byte_quoter_factory(safe): + return _Quoter(safe).__getitem__ + def quote_from_bytes(bs, safe='/'): """Like quote(), but accepts a bytes object rather than a str, and does not perform string-to-bytes encoding. It always returns an ASCII string. @@ -899,14 +986,19 @@ def quote_from_bytes(bs, safe='/'): # Normalize 'safe' by converting to bytes and removing non-ASCII chars safe = safe.encode('ascii', 'ignore') else: + # List comprehensions are faster than generator expressions. safe = bytes([c for c in safe if c < 128]) if not bs.rstrip(_ALWAYS_SAFE_BYTES + safe): return bs.decode() - try: - quoter = _safe_quoters[safe] - except KeyError: - _safe_quoters[safe] = quoter = Quoter(safe).__getitem__ - return ''.join([quoter(char) for char in bs]) + quoter = _byte_quoter_factory(safe) + if (bs_len := len(bs)) < 200_000: + return ''.join(map(quoter, bs)) + else: + # This saves memory - https://github.com/python/cpython/issues/95865 + chunk_size = math.isqrt(bs_len) + chunks = [''.join(map(quoter, bs[i:i+chunk_size])) + for i in range(0, bs_len, chunk_size)] + return ''.join(chunks) def urlencode(query, doseq=False, safe='', encoding=None, errors=None, quote_via=quote_plus): @@ -939,10 +1031,9 @@ def urlencode(query, doseq=False, safe='', encoding=None, errors=None, # but that's a minor nit. Since the original implementation # allowed empty dicts that type of behavior probably should be # preserved for consistency - except TypeError: - ty, va, tb = sys.exc_info() + except TypeError as err: raise TypeError("not a valid non-string sequence " - "or mapping object").with_traceback(tb) + "or mapping object") from err l = [] if not doseq: @@ -1125,15 +1216,15 @@ def splitnport(host, defport=-1): def _splitnport(host, defport=-1): """Split host and port, returning numeric port. Return given default port if no ':' found; defaults to -1. - Return numerical port if a valid number are found after ':'. + Return numerical port if a valid number is found after ':'. Return None if ':' but not a valid number.""" host, delim, port = host.rpartition(':') if not delim: host = port elif port: - try: + if port.isdigit() and port.isascii(): nport = int(port) - except ValueError: + else: nport = None return host, nport return host, defport diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py index a0ef60b30de..566b8087aec 100644 --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -11,8 +11,8 @@ Handlers needed to open the requested URL. For example, the HTTPHandler performs HTTP GET and POST requests and deals with non-error returns. The HTTPRedirectHandler automatically deals with -HTTP 301, 302, 303 and 307 redirect errors, and the HTTPDigestAuthHandler -deals with digest authentication. +HTTP 301, 302, 303, 307, and 308 redirect errors, and the +HTTPDigestAuthHandler deals with digest authentication. urlopen(url, data=None) -- Basic usage is the same as original urllib. pass the url and optionally data to post to an HTTP URL, and @@ -83,33 +83,31 @@ import base64 import bisect +import contextlib import email import hashlib import http.client import io import os -import posixpath import re import socket import string import sys import time import tempfile -import contextlib -import warnings from urllib.error import URLError, HTTPError, ContentTooShortError from urllib.parse import ( urlparse, urlsplit, urljoin, unwrap, quote, unquote, _splittype, _splithost, _splitport, _splituser, _splitpasswd, - _splitattr, _splitquery, _splitvalue, _splittag, _to_bytes, + _splitattr, _splitvalue, _splittag, unquote_to_bytes, urlunparse) from urllib.response import addinfourl, addclosehook # check for SSL try: - import ssl + import ssl # noqa: F401 except ImportError: _have_ssl = False else: @@ -129,7 +127,7 @@ 'urlopen', 'install_opener', 'build_opener', 'pathname2url', 'url2pathname', 'getproxies', # Legacy interface - 'urlretrieve', 'urlcleanup', 'URLopener', 'FancyURLopener', + 'urlretrieve', 'urlcleanup', ] # used in User-Agent header sent @@ -137,7 +135,7 @@ _opener = None def urlopen(url, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT, - *, cafile=None, capath=None, cadefault=False, context=None): + *, context=None): '''Open the URL url, which can be either a string or a Request object. *data* must be an object specifying additional data to be sent to @@ -155,14 +153,6 @@ def urlopen(url, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT, If *context* is specified, it must be a ssl.SSLContext instance describing the various SSL options. See HTTPSConnection for more details. - The optional *cafile* and *capath* parameters specify a set of trusted CA - certificates for HTTPS requests. cafile should point to a single file - containing a bundle of CA certificates, whereas capath should point to a - directory of hashed certificate files. More information can be found in - ssl.SSLContext.load_verify_locations(). - - The *cadefault* parameter is ignored. - This function always returns an object which can work as a context manager and has the properties url, headers, and status. @@ -174,8 +164,7 @@ def urlopen(url, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT, the reason phrase returned by the server --- instead of the response headers as it is specified in the documentation for HTTPResponse. - For FTP, file, and data URLs and requests explicitly handled by legacy - URLopener and FancyURLopener classes, this function returns a + For FTP, file, and data URLs, this function returns a urllib.response.addinfourl object. Note that None may be returned if no handler handles the request (though @@ -188,25 +177,7 @@ def urlopen(url, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT, ''' global _opener - if cafile or capath or cadefault: - import warnings - warnings.warn("cafile, capath and cadefault are deprecated, use a " - "custom context instead.", DeprecationWarning, 2) - if context is not None: - raise ValueError( - "You can't pass both context and any of cafile, capath, and " - "cadefault" - ) - if not _have_ssl: - raise ValueError('SSL support not available') - context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH, - cafile=cafile, - capath=capath) - # send ALPN extension to indicate HTTP/1.1 protocol - context.set_alpn_protocols(['http/1.1']) - https_handler = HTTPSHandler(context=context) - opener = build_opener(https_handler) - elif context: + if context: https_handler = HTTPSHandler(context=context) opener = build_opener(https_handler) elif _opener is None: @@ -266,10 +237,7 @@ def urlretrieve(url, filename=None, reporthook=None, data=None): if reporthook: reporthook(blocknum, bs, size) - while True: - block = fp.read(bs) - if not block: - break + while block := fp.read(bs): read += len(block) tfp.write(block) blocknum += 1 @@ -661,7 +629,7 @@ def redirect_request(self, req, fp, code, msg, headers, newurl): but another Handler might. """ m = req.get_method() - if (not (code in (301, 302, 303, 307) and m in ("GET", "HEAD") + if (not (code in (301, 302, 303, 307, 308) and m in ("GET", "HEAD") or code in (301, 302, 303) and m == "POST")): raise HTTPError(req.full_url, code, msg, headers, fp) @@ -680,6 +648,7 @@ def redirect_request(self, req, fp, code, msg, headers, newurl): newheaders = {k: v for k, v in req.headers.items() if k.lower() not in CONTENT_HEADERS} return Request(newurl, + method="HEAD" if m == "HEAD" else "GET", headers=newheaders, origin_req_host=req.origin_req_host, unverifiable=True) @@ -748,7 +717,7 @@ def http_error_302(self, req, fp, code, msg, headers): return self.parent.open(new, timeout=req.timeout) - http_error_301 = http_error_303 = http_error_307 = http_error_302 + http_error_301 = http_error_303 = http_error_307 = http_error_308 = http_error_302 inf_msg = "The HTTP server returned a redirect error that would " \ "lead to an infinite loop.\n" \ @@ -907,9 +876,9 @@ def find_user_password(self, realm, authuri): class HTTPPasswordMgrWithPriorAuth(HTTPPasswordMgrWithDefaultRealm): - def __init__(self, *args, **kwargs): + def __init__(self): self.authenticated = {} - super().__init__(*args, **kwargs) + super().__init__() def add_password(self, realm, uri, user, passwd, is_authenticated=False): self.update_authenticated(uri, is_authenticated) @@ -969,6 +938,7 @@ def _parse_realm(self, header): for mo in AbstractBasicAuthHandler.rx.finditer(header): scheme, quote, realm = mo.groups() if quote not in ['"', "'"]: + import warnings warnings.warn("Basic Auth Realm was unquoted", UserWarning, 3) @@ -1078,7 +1048,7 @@ def http_error_407(self, req, fp, code, msg, headers): class AbstractDigestAuthHandler: - # Digest authentication is specified in RFC 2617. + # Digest authentication is specified in RFC 2617/7616. # XXX The client does not inspect the Authentication-Info header # in a successful response. @@ -1206,11 +1176,14 @@ def get_authorization(self, req, chal): return base def get_algorithm_impls(self, algorithm): + # algorithm names taken from RFC 7616 Section 6.1 # lambdas assume digest modules are imported at the top level if algorithm == 'MD5': H = lambda x: hashlib.md5(x.encode("ascii")).hexdigest() - elif algorithm == 'SHA': + elif algorithm == 'SHA': # non-standard, retained for compatibility. H = lambda x: hashlib.sha1(x.encode("ascii")).hexdigest() + elif algorithm == 'SHA-256': + H = lambda x: hashlib.sha256(x.encode("ascii")).hexdigest() # XXX MD5-sess else: raise ValueError("Unsupported digest authentication " @@ -1255,8 +1228,8 @@ def http_error_407(self, req, fp, code, msg, headers): class AbstractHTTPHandler(BaseHandler): - def __init__(self, debuglevel=0): - self._debuglevel = debuglevel + def __init__(self, debuglevel=None): + self._debuglevel = debuglevel if debuglevel is not None else http.client.HTTPConnection.debuglevel def set_http_debuglevel(self, level): self._debuglevel = level @@ -1382,14 +1355,19 @@ def http_open(self, req): class HTTPSHandler(AbstractHTTPHandler): - def __init__(self, debuglevel=0, context=None, check_hostname=None): + def __init__(self, debuglevel=None, context=None, check_hostname=None): + debuglevel = debuglevel if debuglevel is not None else http.client.HTTPSConnection.debuglevel AbstractHTTPHandler.__init__(self, debuglevel) + if context is None: + http_version = http.client.HTTPSConnection._http_vsn + context = http.client._create_https_context(http_version) + if check_hostname is not None: + context.check_hostname = check_hostname self._context = context - self._check_hostname = check_hostname def https_open(self, req): return self.do_open(http.client.HTTPSConnection, req, - context=self._context, check_hostname=self._check_hostname) + context=self._context) https_request = AbstractHTTPHandler.do_request_ @@ -1472,16 +1450,6 @@ def parse_http_list(s): return [part.strip() for part in res] class FileHandler(BaseHandler): - # Use local file or FTP depending on form of URL - def file_open(self, req): - url = req.selector - if url[:2] == '//' and url[2:3] != '/' and (req.host and - req.host != 'localhost'): - if not req.host in self.get_names(): - raise URLError("file:// scheme is supported only on localhost") - else: - return self.open_local_file(req) - # names for the localhost names = None def get_names(self): @@ -1498,35 +1466,41 @@ def get_names(self): def open_local_file(self, req): import email.utils import mimetypes - host = req.host - filename = req.selector - localfile = url2pathname(filename) + localfile = url2pathname(req.full_url, require_scheme=True, resolve_host=True) try: stats = os.stat(localfile) size = stats.st_size modified = email.utils.formatdate(stats.st_mtime, usegmt=True) - mtype = mimetypes.guess_type(filename)[0] + mtype = mimetypes.guess_file_type(localfile)[0] headers = email.message_from_string( 'Content-type: %s\nContent-length: %d\nLast-modified: %s\n' % (mtype or 'text/plain', size, modified)) - if host: - host, port = _splitport(host) - if not host or \ - (not port and _safe_gethostbyname(host) in self.get_names()): - if host: - origurl = 'file://' + host + filename - else: - origurl = 'file://' + filename - return addinfourl(open(localfile, 'rb'), headers, origurl) + origurl = pathname2url(localfile, add_scheme=True) + return addinfourl(open(localfile, 'rb'), headers, origurl) except OSError as exp: - raise URLError(exp) - raise URLError('file not on local host') + raise URLError(exp, exp.filename) + + file_open = open_local_file -def _safe_gethostbyname(host): +def _is_local_authority(authority, resolve): + # Compare hostnames + if not authority or authority == 'localhost': + return True try: - return socket.gethostbyname(host) - except socket.gaierror: - return None + hostname = socket.gethostname() + except (socket.gaierror, AttributeError): + pass + else: + if authority == hostname: + return True + # Compare IP addresses + if not resolve: + return False + try: + address = socket.gethostbyname(authority) + except (socket.gaierror, AttributeError, UnicodeEncodeError): + return False + return address in FileHandler().get_names() class FTPHandler(BaseHandler): def ftp_open(self, req): @@ -1561,6 +1535,7 @@ def ftp_open(self, req): dirs, file = dirs[:-1], dirs[-1] if dirs and not dirs[0]: dirs = dirs[1:] + fw = None try: fw = self.connect_ftp(user, passwd, host, port, dirs, req.timeout) type = file and 'I' or 'D' @@ -1578,9 +1553,12 @@ def ftp_open(self, req): headers += "Content-length: %d\n" % retrlen headers = email.message_from_string(headers) return addinfourl(fp, headers, req.full_url) - except ftplib.all_errors as exp: - exc = URLError('ftp error: %r' % exp) - raise exc.with_traceback(sys.exc_info()[2]) + except Exception as exp: + if fw is not None and not fw.keepalive: + fw.close() + if isinstance(exp, ftplib.all_errors): + raise URLError(f"ftp error: {exp}") from exp + raise def connect_ftp(self, user, passwd, host, port, dirs, timeout): return ftpwrapper(user, passwd, host, port, dirs, timeout, @@ -1604,14 +1582,15 @@ def setMaxConns(self, m): def connect_ftp(self, user, passwd, host, port, dirs, timeout): key = user, host, port, '/'.join(dirs), timeout - if key in self.cache: - self.timeout[key] = time.time() + self.delay - else: - self.cache[key] = ftpwrapper(user, passwd, host, port, - dirs, timeout) - self.timeout[key] = time.time() + self.delay + conn = self.cache.get(key) + if conn is None or not conn.keepalive: + if conn is not None: + conn.close() + conn = self.cache[key] = ftpwrapper(user, passwd, host, port, + dirs, timeout) + self.timeout[key] = time.time() + self.delay self.check_cache() - return self.cache[key] + return conn def check_cache(self): # first check for old ones @@ -1670,683 +1649,80 @@ def data_open(self, req): return addinfourl(io.BytesIO(data), headers, url) -# Code move from the old urllib module - -MAXFTPCACHE = 10 # Trim the ftp cache beyond this size - -# Helper for non-unix systems -if os.name == 'nt': - from nturl2path import url2pathname, pathname2url -else: - def url2pathname(pathname): - """OS-specific conversion from a relative URL of the 'file' scheme - to a file system path; not recommended for general use.""" - return unquote(pathname) - - def pathname2url(pathname): - """OS-specific conversion from a file system path to a relative URL - of the 'file' scheme; not recommended for general use.""" - return quote(pathname) - - -ftpcache = {} - - -class URLopener: - """Class to open URLs. - This is a class rather than just a subroutine because we may need - more than one set of global protocol-specific options. - Note -- this is a base class for those who don't want the - automatic handling of errors type 302 (relocated) and 401 - (authorization needed).""" - - __tempfiles = None - - version = "Python-urllib/%s" % __version__ - - # Constructor - def __init__(self, proxies=None, **x509): - msg = "%(class)s style of invoking requests is deprecated. " \ - "Use newer urlopen functions/methods" % {'class': self.__class__.__name__} - warnings.warn(msg, DeprecationWarning, stacklevel=3) - if proxies is None: - proxies = getproxies() - assert hasattr(proxies, 'keys'), "proxies must be a mapping" - self.proxies = proxies - self.key_file = x509.get('key_file') - self.cert_file = x509.get('cert_file') - self.addheaders = [('User-Agent', self.version), ('Accept', '*/*')] - self.__tempfiles = [] - self.__unlink = os.unlink # See cleanup() - self.tempcache = None - # Undocumented feature: if you assign {} to tempcache, - # it is used to cache files retrieved with - # self.retrieve(). This is not enabled by default - # since it does not work for changing documents (and I - # haven't got the logic to check expiration headers - # yet). - self.ftpcache = ftpcache - # Undocumented feature: you can use a different - # ftp cache by assigning to the .ftpcache member; - # in case you want logically independent URL openers - # XXX This is not threadsafe. Bah. - - def __del__(self): - self.close() - - def close(self): - self.cleanup() - - def cleanup(self): - # This code sometimes runs when the rest of this module - # has already been deleted, so it can't use any globals - # or import anything. - if self.__tempfiles: - for file in self.__tempfiles: - try: - self.__unlink(file) - except OSError: - pass - del self.__tempfiles[:] - if self.tempcache: - self.tempcache.clear() - - def addheader(self, *args): - """Add a header to be used by the HTTP interface only - e.g. u.addheader('Accept', 'sound/basic')""" - self.addheaders.append(args) - - # External interface - def open(self, fullurl, data=None): - """Use URLopener().open(file) instead of open(file, 'r').""" - fullurl = unwrap(_to_bytes(fullurl)) - fullurl = quote(fullurl, safe="%/:=&?~#+!$,;'@()*[]|") - if self.tempcache and fullurl in self.tempcache: - filename, headers = self.tempcache[fullurl] - fp = open(filename, 'rb') - return addinfourl(fp, headers, fullurl) - urltype, url = _splittype(fullurl) - if not urltype: - urltype = 'file' - if urltype in self.proxies: - proxy = self.proxies[urltype] - urltype, proxyhost = _splittype(proxy) - host, selector = _splithost(proxyhost) - url = (host, fullurl) # Signal special case to open_*() - else: - proxy = None - name = 'open_' + urltype - self.type = urltype - name = name.replace('-', '_') - if not hasattr(self, name) or name == 'open_local_file': - if proxy: - return self.open_unknown_proxy(proxy, fullurl, data) - else: - return self.open_unknown(fullurl, data) - try: - if data is None: - return getattr(self, name)(url) - else: - return getattr(self, name)(url, data) - except (HTTPError, URLError): - raise - except OSError as msg: - raise OSError('socket error', msg).with_traceback(sys.exc_info()[2]) - - def open_unknown(self, fullurl, data=None): - """Overridable interface to open unknown URL type.""" - type, url = _splittype(fullurl) - raise OSError('url error', 'unknown url type', type) - - def open_unknown_proxy(self, proxy, fullurl, data=None): - """Overridable interface to open unknown URL type.""" - type, url = _splittype(fullurl) - raise OSError('url error', 'invalid proxy for %s' % type, proxy) - - # External interface - def retrieve(self, url, filename=None, reporthook=None, data=None): - """retrieve(url) returns (filename, headers) for a local object - or (tempfilename, headers) for a remote object.""" - url = unwrap(_to_bytes(url)) - if self.tempcache and url in self.tempcache: - return self.tempcache[url] - type, url1 = _splittype(url) - if filename is None and (not type or type == 'file'): - try: - fp = self.open_local_file(url1) - hdrs = fp.info() - fp.close() - return url2pathname(_splithost(url1)[1]), hdrs - except OSError: - pass - fp = self.open(url, data) - try: - headers = fp.info() - if filename: - tfp = open(filename, 'wb') - else: - garbage, path = _splittype(url) - garbage, path = _splithost(path or "") - path, garbage = _splitquery(path or "") - path, garbage = _splitattr(path or "") - suffix = os.path.splitext(path)[1] - (fd, filename) = tempfile.mkstemp(suffix) - self.__tempfiles.append(filename) - tfp = os.fdopen(fd, 'wb') - try: - result = filename, headers - if self.tempcache is not None: - self.tempcache[url] = result - bs = 1024*8 - size = -1 - read = 0 - blocknum = 0 - if "content-length" in headers: - size = int(headers["Content-Length"]) - if reporthook: - reporthook(blocknum, bs, size) - while 1: - block = fp.read(bs) - if not block: - break - read += len(block) - tfp.write(block) - blocknum += 1 - if reporthook: - reporthook(blocknum, bs, size) - finally: - tfp.close() - finally: - fp.close() - - # raise exception if actual size does not match content-length header - if size >= 0 and read < size: - raise ContentTooShortError( - "retrieval incomplete: got only %i out of %i bytes" - % (read, size), result) - - return result - - # Each method named open_<type> knows how to open that type of URL - - def _open_generic_http(self, connection_factory, url, data): - """Make an HTTP connection using connection_class. - - This is an internal method that should be called from - open_http() or open_https(). - - Arguments: - - connection_factory should take a host name and return an - HTTPConnection instance. - - url is the url to retrieval or a host, relative-path pair. - - data is payload for a POST request or None. - """ - - user_passwd = None - proxy_passwd= None - if isinstance(url, str): - host, selector = _splithost(url) - if host: - user_passwd, host = _splituser(host) - host = unquote(host) - realhost = host - else: - host, selector = url - # check whether the proxy contains authorization information - proxy_passwd, host = _splituser(host) - # now we proceed with the url we want to obtain - urltype, rest = _splittype(selector) - url = rest - user_passwd = None - if urltype.lower() != 'http': - realhost = None - else: - realhost, rest = _splithost(rest) - if realhost: - user_passwd, realhost = _splituser(realhost) - if user_passwd: - selector = "%s://%s%s" % (urltype, realhost, rest) - if proxy_bypass(realhost): - host = realhost - - if not host: raise OSError('http error', 'no host given') - - if proxy_passwd: - proxy_passwd = unquote(proxy_passwd) - proxy_auth = base64.b64encode(proxy_passwd.encode()).decode('ascii') - else: - proxy_auth = None - - if user_passwd: - user_passwd = unquote(user_passwd) - auth = base64.b64encode(user_passwd.encode()).decode('ascii') - else: - auth = None - http_conn = connection_factory(host) - headers = {} - if proxy_auth: - headers["Proxy-Authorization"] = "Basic %s" % proxy_auth - if auth: - headers["Authorization"] = "Basic %s" % auth - if realhost: - headers["Host"] = realhost - - # Add Connection:close as we don't support persistent connections yet. - # This helps in closing the socket and avoiding ResourceWarning - - headers["Connection"] = "close" - - for header, value in self.addheaders: - headers[header] = value - - if data is not None: - headers["Content-Type"] = "application/x-www-form-urlencoded" - http_conn.request("POST", selector, data, headers) - else: - http_conn.request("GET", selector, headers=headers) - - try: - response = http_conn.getresponse() - except http.client.BadStatusLine: - # something went wrong with the HTTP status line - raise URLError("http protocol error: bad status line") - - # According to RFC 2616, "2xx" code indicates that the client's - # request was successfully received, understood, and accepted. - if 200 <= response.status < 300: - return addinfourl(response, response.msg, "http:" + url, - response.status) - else: - return self.http_error( - url, response.fp, - response.status, response.reason, response.msg, data) - - def open_http(self, url, data=None): - """Use HTTP protocol.""" - return self._open_generic_http(http.client.HTTPConnection, url, data) - - def http_error(self, url, fp, errcode, errmsg, headers, data=None): - """Handle http errors. - - Derived class can override this, or provide specific handlers - named http_error_DDD where DDD is the 3-digit error code.""" - # First check if there's a specific handler for this error - name = 'http_error_%d' % errcode - if hasattr(self, name): - method = getattr(self, name) - if data is None: - result = method(url, fp, errcode, errmsg, headers) - else: - result = method(url, fp, errcode, errmsg, headers, data) - if result: return result - return self.http_error_default(url, fp, errcode, errmsg, headers) - - def http_error_default(self, url, fp, errcode, errmsg, headers): - """Default error handler: close the connection and raise OSError.""" - fp.close() - raise HTTPError(url, errcode, errmsg, headers, None) - - if _have_ssl: - def _https_connection(self, host): - return http.client.HTTPSConnection(host, - key_file=self.key_file, - cert_file=self.cert_file) - - def open_https(self, url, data=None): - """Use HTTPS protocol.""" - return self._open_generic_http(self._https_connection, url, data) - - def open_file(self, url): - """Use local file or FTP depending on form of URL.""" - if not isinstance(url, str): - raise URLError('file error: proxy support for file protocol currently not implemented') - if url[:2] == '//' and url[2:3] != '/' and url[2:12].lower() != 'localhost/': - raise ValueError("file:// scheme is supported only on localhost") - else: - return self.open_local_file(url) - - def open_local_file(self, url): - """Use local file.""" - import email.utils - import mimetypes - host, file = _splithost(url) - localname = url2pathname(file) - try: - stats = os.stat(localname) - except OSError as e: - raise URLError(e.strerror, e.filename) - size = stats.st_size - modified = email.utils.formatdate(stats.st_mtime, usegmt=True) - mtype = mimetypes.guess_type(url)[0] - headers = email.message_from_string( - 'Content-Type: %s\nContent-Length: %d\nLast-modified: %s\n' % - (mtype or 'text/plain', size, modified)) - if not host: - urlfile = file - if file[:1] == '/': - urlfile = 'file://' + file - return addinfourl(open(localname, 'rb'), headers, urlfile) - host, port = _splitport(host) - if (not port - and socket.gethostbyname(host) in ((localhost(),) + thishost())): - urlfile = file - if file[:1] == '/': - urlfile = 'file://' + file - elif file[:2] == './': - raise ValueError("local file url may start with / or file:. Unknown url of type: %s" % url) - return addinfourl(open(localname, 'rb'), headers, urlfile) - raise URLError('local file error: not on local host') - - def open_ftp(self, url): - """Use FTP protocol.""" - if not isinstance(url, str): - raise URLError('ftp error: proxy support for ftp protocol currently not implemented') - import mimetypes - host, path = _splithost(url) - if not host: raise URLError('ftp error: no host given') - host, port = _splitport(host) - user, host = _splituser(host) - if user: user, passwd = _splitpasswd(user) - else: passwd = None - host = unquote(host) - user = unquote(user or '') - passwd = unquote(passwd or '') - host = socket.gethostbyname(host) - if not port: - import ftplib - port = ftplib.FTP_PORT - else: - port = int(port) - path, attrs = _splitattr(path) - path = unquote(path) - dirs = path.split('/') - dirs, file = dirs[:-1], dirs[-1] - if dirs and not dirs[0]: dirs = dirs[1:] - if dirs and not dirs[0]: dirs[0] = '/' - key = user, host, port, '/'.join(dirs) - # XXX thread unsafe! - if len(self.ftpcache) > MAXFTPCACHE: - # Prune the cache, rather arbitrarily - for k in list(self.ftpcache): - if k != key: - v = self.ftpcache[k] - del self.ftpcache[k] - v.close() - try: - if key not in self.ftpcache: - self.ftpcache[key] = \ - ftpwrapper(user, passwd, host, port, dirs) - if not file: type = 'D' - else: type = 'I' - for attr in attrs: - attr, value = _splitvalue(attr) - if attr.lower() == 'type' and \ - value in ('a', 'A', 'i', 'I', 'd', 'D'): - type = value.upper() - (fp, retrlen) = self.ftpcache[key].retrfile(file, type) - mtype = mimetypes.guess_type("ftp:" + url)[0] - headers = "" - if mtype: - headers += "Content-Type: %s\n" % mtype - if retrlen is not None and retrlen >= 0: - headers += "Content-Length: %d\n" % retrlen - headers = email.message_from_string(headers) - return addinfourl(fp, headers, "ftp:" + url) - except ftperrors() as exp: - raise URLError('ftp error %r' % exp).with_traceback(sys.exc_info()[2]) - - def open_data(self, url, data=None): - """Use "data" URL.""" - if not isinstance(url, str): - raise URLError('data error: proxy support for data protocol currently not implemented') - # ignore POSTed data - # - # syntax of data URLs: - # dataurl := "data:" [ mediatype ] [ ";base64" ] "," data - # mediatype := [ type "/" subtype ] *( ";" parameter ) - # data := *urlchar - # parameter := attribute "=" value - try: - [type, data] = url.split(',', 1) - except ValueError: - raise OSError('data error', 'bad data URL') - if not type: - type = 'text/plain;charset=US-ASCII' - semi = type.rfind(';') - if semi >= 0 and '=' not in type[semi:]: - encoding = type[semi+1:] - type = type[:semi] - else: - encoding = '' - msg = [] - msg.append('Date: %s'%time.strftime('%a, %d %b %Y %H:%M:%S GMT', - time.gmtime(time.time()))) - msg.append('Content-type: %s' % type) - if encoding == 'base64': - # XXX is this encoding/decoding ok? - data = base64.decodebytes(data.encode('ascii')).decode('latin-1') - else: - data = unquote(data) - msg.append('Content-Length: %d' % len(data)) - msg.append('') - msg.append(data) - msg = '\n'.join(msg) - headers = email.message_from_string(msg) - f = io.StringIO(msg) - #f.fileno = None # needed for addinfourl - return addinfourl(f, headers, url) - - -class FancyURLopener(URLopener): - """Derived class with handlers for errors we can handle (perhaps).""" - - def __init__(self, *args, **kwargs): - URLopener.__init__(self, *args, **kwargs) - self.auth_cache = {} - self.tries = 0 - self.maxtries = 10 - - def http_error_default(self, url, fp, errcode, errmsg, headers): - """Default error handling -- don't raise an exception.""" - return addinfourl(fp, headers, "http:" + url, errcode) - - def http_error_302(self, url, fp, errcode, errmsg, headers, data=None): - """Error 302 -- relocated (temporarily).""" - self.tries += 1 - try: - if self.maxtries and self.tries >= self.maxtries: - if hasattr(self, "http_error_500"): - meth = self.http_error_500 - else: - meth = self.http_error_default - return meth(url, fp, 500, - "Internal Server Error: Redirect Recursion", - headers) - result = self.redirect_internal(url, fp, errcode, errmsg, - headers, data) - return result - finally: - self.tries = 0 - - def redirect_internal(self, url, fp, errcode, errmsg, headers, data): - if 'location' in headers: - newurl = headers['location'] - elif 'uri' in headers: - newurl = headers['uri'] - else: - return - fp.close() - - # In case the server sent a relative URL, join with original: - newurl = urljoin(self.type + ":" + url, newurl) - - urlparts = urlparse(newurl) - - # For security reasons, we don't allow redirection to anything other - # than http, https and ftp. - - # We are using newer HTTPError with older redirect_internal method - # This older method will get deprecated in 3.3 - - if urlparts.scheme not in ('http', 'https', 'ftp', ''): - raise HTTPError(newurl, errcode, - errmsg + - " Redirection to url '%s' is not allowed." % newurl, - headers, fp) - - return self.open(newurl) +# Code moved from the old urllib module - def http_error_301(self, url, fp, errcode, errmsg, headers, data=None): - """Error 301 -- also relocated (permanently).""" - return self.http_error_302(url, fp, errcode, errmsg, headers, data) +def url2pathname(url, *, require_scheme=False, resolve_host=False): + """Convert the given file URL to a local file system path. - def http_error_303(self, url, fp, errcode, errmsg, headers, data=None): - """Error 303 -- also relocated (essentially identical to 302).""" - return self.http_error_302(url, fp, errcode, errmsg, headers, data) + The 'file:' scheme prefix must be omitted unless *require_scheme* + is set to true. - def http_error_307(self, url, fp, errcode, errmsg, headers, data=None): - """Error 307 -- relocated, but turn POST into error.""" - if data is None: - return self.http_error_302(url, fp, errcode, errmsg, headers, data) - else: - return self.http_error_default(url, fp, errcode, errmsg, headers) - - def http_error_401(self, url, fp, errcode, errmsg, headers, data=None, - retry=False): - """Error 401 -- authentication required. - This function supports Basic authentication only.""" - if 'www-authenticate' not in headers: - URLopener.http_error_default(self, url, fp, - errcode, errmsg, headers) - stuff = headers['www-authenticate'] - match = re.match('[ \t]*([^ \t]+)[ \t]+realm="([^"]*)"', stuff) - if not match: - URLopener.http_error_default(self, url, fp, - errcode, errmsg, headers) - scheme, realm = match.groups() - if scheme.lower() != 'basic': - URLopener.http_error_default(self, url, fp, - errcode, errmsg, headers) - if not retry: - URLopener.http_error_default(self, url, fp, errcode, errmsg, - headers) - name = 'retry_' + self.type + '_basic_auth' - if data is None: - return getattr(self,name)(url, realm) - else: - return getattr(self,name)(url, realm, data) - - def http_error_407(self, url, fp, errcode, errmsg, headers, data=None, - retry=False): - """Error 407 -- proxy authentication required. - This function supports Basic authentication only.""" - if 'proxy-authenticate' not in headers: - URLopener.http_error_default(self, url, fp, - errcode, errmsg, headers) - stuff = headers['proxy-authenticate'] - match = re.match('[ \t]*([^ \t]+)[ \t]+realm="([^"]*)"', stuff) - if not match: - URLopener.http_error_default(self, url, fp, - errcode, errmsg, headers) - scheme, realm = match.groups() - if scheme.lower() != 'basic': - URLopener.http_error_default(self, url, fp, - errcode, errmsg, headers) - if not retry: - URLopener.http_error_default(self, url, fp, errcode, errmsg, - headers) - name = 'retry_proxy_' + self.type + '_basic_auth' - if data is None: - return getattr(self,name)(url, realm) - else: - return getattr(self,name)(url, realm, data) - - def retry_proxy_http_basic_auth(self, url, realm, data=None): - host, selector = _splithost(url) - newurl = 'http://' + host + selector - proxy = self.proxies['http'] - urltype, proxyhost = _splittype(proxy) - proxyhost, proxyselector = _splithost(proxyhost) - i = proxyhost.find('@') + 1 - proxyhost = proxyhost[i:] - user, passwd = self.get_user_passwd(proxyhost, realm, i) - if not (user or passwd): return None - proxyhost = "%s:%s@%s" % (quote(user, safe=''), - quote(passwd, safe=''), proxyhost) - self.proxies['http'] = 'http://' + proxyhost + proxyselector - if data is None: - return self.open(newurl) - else: - return self.open(newurl, data) - - def retry_proxy_https_basic_auth(self, url, realm, data=None): - host, selector = _splithost(url) - newurl = 'https://' + host + selector - proxy = self.proxies['https'] - urltype, proxyhost = _splittype(proxy) - proxyhost, proxyselector = _splithost(proxyhost) - i = proxyhost.find('@') + 1 - proxyhost = proxyhost[i:] - user, passwd = self.get_user_passwd(proxyhost, realm, i) - if not (user or passwd): return None - proxyhost = "%s:%s@%s" % (quote(user, safe=''), - quote(passwd, safe=''), proxyhost) - self.proxies['https'] = 'https://' + proxyhost + proxyselector - if data is None: - return self.open(newurl) - else: - return self.open(newurl, data) - - def retry_http_basic_auth(self, url, realm, data=None): - host, selector = _splithost(url) - i = host.find('@') + 1 - host = host[i:] - user, passwd = self.get_user_passwd(host, realm, i) - if not (user or passwd): return None - host = "%s:%s@%s" % (quote(user, safe=''), - quote(passwd, safe=''), host) - newurl = 'http://' + host + selector - if data is None: - return self.open(newurl) - else: - return self.open(newurl, data) - - def retry_https_basic_auth(self, url, realm, data=None): - host, selector = _splithost(url) - i = host.find('@') + 1 - host = host[i:] - user, passwd = self.get_user_passwd(host, realm, i) - if not (user or passwd): return None - host = "%s:%s@%s" % (quote(user, safe=''), - quote(passwd, safe=''), host) - newurl = 'https://' + host + selector - if data is None: - return self.open(newurl) + The URL authority may be resolved with gethostbyname() if + *resolve_host* is set to true. + """ + if not require_scheme: + url = 'file:' + url + scheme, authority, url = urlsplit(url)[:3] # Discard query and fragment. + if scheme != 'file': + raise URLError("URL is missing a 'file:' scheme") + if os.name == 'nt': + if authority[1:2] == ':': + # e.g. file://c:/file.txt + url = authority + url + elif not _is_local_authority(authority, resolve_host): + # e.g. file://server/share/file.txt + url = '//' + authority + url + elif url[:3] == '///': + # e.g. file://///server/share/file.txt + url = url[1:] else: - return self.open(newurl, data) - - def get_user_passwd(self, host, realm, clear_cache=0): - key = realm + '@' + host.lower() - if key in self.auth_cache: - if clear_cache: - del self.auth_cache[key] - else: - return self.auth_cache[key] - user, passwd = self.prompt_user_passwd(host, realm) - if user or passwd: self.auth_cache[key] = (user, passwd) - return user, passwd - - def prompt_user_passwd(self, host, realm): - """Override this in a GUI environment!""" - import getpass - try: - user = input("Enter username for %s at %s: " % (realm, host)) - passwd = getpass.getpass("Enter password for %s in %s at %s: " % - (user, realm, host)) - return user, passwd - except KeyboardInterrupt: - print() - return None, None + if url[:1] == '/' and url[2:3] in (':', '|'): + # Skip past extra slash before DOS drive in URL path. + url = url[1:] + if url[1:2] == '|': + # Older URLs use a pipe after a drive letter + url = url[:1] + ':' + url[2:] + url = url.replace('/', '\\') + elif not _is_local_authority(authority, resolve_host): + raise URLError("file:// scheme is supported only on localhost") + encoding = sys.getfilesystemencoding() + errors = sys.getfilesystemencodeerrors() + return unquote(url, encoding=encoding, errors=errors) + + +def pathname2url(pathname, *, add_scheme=False): + """Convert the given local file system path to a file URL. + + The 'file:' scheme prefix is omitted unless *add_scheme* + is set to true. + """ + if os.name == 'nt': + pathname = pathname.replace('\\', '/') + encoding = sys.getfilesystemencoding() + errors = sys.getfilesystemencodeerrors() + scheme = 'file:' if add_scheme else '' + drive, root, tail = os.path.splitroot(pathname) + if drive: + # First, clean up some special forms. We are going to sacrifice the + # additional information anyway + if drive[:4] == '//?/': + drive = drive[4:] + if drive[:4].upper() == 'UNC/': + drive = '//' + drive[4:] + if drive[1:] == ':': + # DOS drive specified. Add three slashes to the start, producing + # an authority section with a zero-length authority, and a path + # section starting with a single slash. + drive = '///' + drive + drive = quote(drive, encoding=encoding, errors=errors, safe='/:') + elif root: + # Add explicitly empty authority to absolute path. If the path + # starts with exactly one slash then this change is mostly + # cosmetic, but if it begins with two or more slashes then this + # avoids interpreting the path as a URL authority. + root = '//' + root + tail = quote(tail, encoding=encoding, errors=errors) + return scheme + drive + root + tail # Utility functions @@ -2436,8 +1812,7 @@ def retrfile(self, file, type): conn, retrlen = self.ftp.ntransfercmd(cmd) except ftplib.error_perm as reason: if str(reason)[:3] != '550': - raise URLError('ftp error: %r' % reason).with_traceback( - sys.exc_info()[2]) + raise URLError(f'ftp error: {reason}') from reason if not conn: # Set transfer mode to ASCII! self.ftp.voidcmd('TYPE A') @@ -2464,7 +1839,13 @@ def retrfile(self, file, type): return (ftpobj, retrlen) def endtransfer(self): + if not self.busy: + return self.busy = 0 + try: + self.ftp.voidresp() + except ftperrors(): + pass def close(self): self.keepalive = False @@ -2489,31 +1870,35 @@ def getproxies_environment(): """Return a dictionary of scheme -> proxy server URL mappings. Scan the environment for variables named <scheme>_proxy; - this seems to be the standard convention. If you need a - different way, you can pass a proxies dictionary to the - [Fancy]URLopener constructor. - + this seems to be the standard convention. """ - proxies = {} # in order to prefer lowercase variables, process environment in # two passes: first matches any, second pass matches lowercase only - for name, value in os.environ.items(): - name = name.lower() - if value and name[-6:] == '_proxy': - proxies[name[:-6]] = value + + # select only environment variables which end in (after making lowercase) _proxy + proxies = {} + environment = [] + for name in os.environ: + # fast screen underscore position before more expensive case-folding + if len(name) > 5 and name[-6] == "_" and name[-5:].lower() == "proxy": + value = os.environ[name] + proxy_name = name[:-6].lower() + environment.append((name, value, proxy_name)) + if value: + proxies[proxy_name] = value # CVE-2016-1000110 - If we are running as CGI script, forget HTTP_PROXY # (non-all-lowercase) as it may be set from the web server by a "Proxy:" # header from the client # If "proxy" is lowercase, it will still be used thanks to the next block if 'REQUEST_METHOD' in os.environ: proxies.pop('http', None) - for name, value in os.environ.items(): + for name, value, proxy_name in environment: + # not case-folded, checking here for lower-case env vars only if name[-6:] == '_proxy': - name = name.lower() if value: - proxies[name[:-6]] = value + proxies[proxy_name] = value else: - proxies.pop(name[:-6], None) + proxies.pop(proxy_name, None) return proxies def proxy_bypass_environment(host, proxies=None): @@ -2566,6 +1951,7 @@ def _proxy_bypass_macosx_sysconf(host, proxy_settings): } """ from fnmatch import fnmatch + from ipaddress import AddressValueError, IPv4Address hostonly, port = _splitport(host) @@ -2582,20 +1968,17 @@ def ip2num(ipAddr): return True hostIP = None + try: + hostIP = int(IPv4Address(hostonly)) + except AddressValueError: + pass for value in proxy_settings.get('exceptions', ()): # Items in the list are strings like these: *.local, 169.254/16 if not value: continue m = re.match(r"(\d+(?:\.\d+)*)(/\d+)?", value) - if m is not None: - if hostIP is None: - try: - hostIP = socket.gethostbyname(hostonly) - hostIP = ip2num(hostIP) - except OSError: - continue - + if m is not None and hostIP is not None: base = ip2num(m.group(1)) mask = m.group(2) if mask is None: @@ -2618,6 +2001,31 @@ def ip2num(ipAddr): return False +# Same as _proxy_bypass_macosx_sysconf, testable on all platforms +def _proxy_bypass_winreg_override(host, override): + """Return True if the host should bypass the proxy server. + + The proxy override list is obtained from the Windows + Internet settings proxy override registry value. + + An example of a proxy override value is: + "www.example.com;*.example.net; 192.168.0.1" + """ + from fnmatch import fnmatch + + host, _ = _splitport(host) + proxy_override = override.split(';') + for test in proxy_override: + test = test.strip() + # "<local>" should bypass the proxy server for all intranet addresses + if test == '<local>': + if '.' not in host: + return True + elif fnmatch(host, test): + return True + return False + + if sys.platform == 'darwin': from _scproxy import _get_proxy_settings, _get_proxies @@ -2716,7 +2124,7 @@ def proxy_bypass_registry(host): import winreg except ImportError: # Std modules, so should be around - but you never know! - return 0 + return False try: internetSettings = winreg.OpenKey(winreg.HKEY_CURRENT_USER, r'Software\Microsoft\Windows\CurrentVersion\Internet Settings') @@ -2726,40 +2134,10 @@ def proxy_bypass_registry(host): 'ProxyOverride')[0]) # ^^^^ Returned as Unicode but problems if not converted to ASCII except OSError: - return 0 + return False if not proxyEnable or not proxyOverride: - return 0 - # try to make a host list from name and IP address. - rawHost, port = _splitport(host) - host = [rawHost] - try: - addr = socket.gethostbyname(rawHost) - if addr != rawHost: - host.append(addr) - except OSError: - pass - try: - fqdn = socket.getfqdn(rawHost) - if fqdn != rawHost: - host.append(fqdn) - except OSError: - pass - # make a check value list from the registry entry: replace the - # '<local>' string by the localhost entry and the corresponding - # canonical entry. - proxyOverride = proxyOverride.split(';') - # now check if we match one of the registry values. - for test in proxyOverride: - if test == '<local>': - if '.' not in rawHost: - return 1 - test = test.replace(".", r"\.") # mask dots - test = test.replace("*", r".*") # change glob sequence - test = test.replace("?", r".") # change glob char - for val in host: - if re.match(test, val, re.I): - return 1 - return 0 + return False + return _proxy_bypass_winreg_override(host, proxyOverride) def proxy_bypass(host): """Return True, if host should be bypassed. diff --git a/Lib/urllib/robotparser.py b/Lib/urllib/robotparser.py index c58565e3945..4009fd6b58f 100644 --- a/Lib/urllib/robotparser.py +++ b/Lib/urllib/robotparser.py @@ -11,6 +11,8 @@ """ import collections +import re +import urllib.error import urllib.parse import urllib.request @@ -19,6 +21,19 @@ RequestRate = collections.namedtuple("RequestRate", "requests seconds") +def normalize(path): + unquoted = urllib.parse.unquote(path, errors='surrogateescape') + return urllib.parse.quote(unquoted, errors='surrogateescape') + +def normalize_path(path): + path, sep, query = path.partition('?') + path = normalize(path) + if sep: + query = re.sub(r'[^=&]+', lambda m: normalize(m[0]), query) + path += '?' + query + return path + + class RobotFileParser: """ This class provides a set of methods to read, parse and answer questions about a single robots.txt file. @@ -54,7 +69,7 @@ def modified(self): def set_url(self, url): """Sets the URL referring to a robots.txt file.""" self.url = url - self.host, self.path = urllib.parse.urlparse(url)[1:3] + self.host, self.path = urllib.parse.urlsplit(url)[1:3] def read(self): """Reads the robots.txt URL and feeds it to the parser.""" @@ -65,9 +80,10 @@ def read(self): self.disallow_all = True elif err.code >= 400 and err.code < 500: self.allow_all = True + err.close() else: raw = f.read() - self.parse(raw.decode("utf-8").splitlines()) + self.parse(raw.decode("utf-8", "surrogateescape").splitlines()) def _add_entry(self, entry): if "*" in entry.useragents: @@ -111,7 +127,7 @@ def parse(self, lines): line = line.split(':', 1) if len(line) == 2: line[0] = line[0].strip().lower() - line[1] = urllib.parse.unquote(line[1].strip()) + line[1] = line[1].strip() if line[0] == "user-agent": if state == 2: self._add_entry(entry) @@ -165,10 +181,11 @@ def can_fetch(self, useragent, url): return False # search for given user agent matches # the first match counts - parsed_url = urllib.parse.urlparse(urllib.parse.unquote(url)) - url = urllib.parse.urlunparse(('','',parsed_url.path, - parsed_url.params,parsed_url.query, parsed_url.fragment)) - url = urllib.parse.quote(url) + # TODO: The private API is used in order to preserve an empty query. + # This is temporary until the public API starts supporting this feature. + parsed_url = urllib.parse._urlsplit(url, '') + url = urllib.parse._urlunsplit(None, None, *parsed_url[2:]) + url = normalize_path(url) if not url: url = "/" for entry in self.entries: @@ -211,7 +228,6 @@ def __str__(self): entries = entries + [self.default_entry] return '\n\n'.join(map(str, entries)) - class RuleLine: """A rule line is a single "Allow:" (allowance==True) or "Disallow:" (allowance==False) followed by a path.""" @@ -219,8 +235,7 @@ def __init__(self, path, allowance): if path == '' and not allowance: # an empty value means allow all allowance = True - path = urllib.parse.urlunparse(urllib.parse.urlparse(path)) - self.path = urllib.parse.quote(path) + self.path = normalize_path(path) self.allowance = allowance def applies_to(self, filename): @@ -266,7 +281,7 @@ def applies_to(self, useragent): def allowance(self, filename): """Preconditions: - our agent applies to this entry - - filename is URL decoded""" + - filename is URL encoded""" for line in self.rulelines: if line.applies_to(filename): return line.allowance diff --git a/Lib/uuid.py b/Lib/uuid.py index 55f46eb5106..313f2fc46cb 100644 --- a/Lib/uuid.py +++ b/Lib/uuid.py @@ -1,8 +1,12 @@ -r"""UUID objects (universally unique identifiers) according to RFC 4122. +r"""UUID objects (universally unique identifiers) according to RFC 4122/9562. -This module provides immutable UUID objects (class UUID) and the functions -uuid1(), uuid3(), uuid4(), uuid5() for generating version 1, 3, 4, and 5 -UUIDs as specified in RFC 4122. +This module provides immutable UUID objects (class UUID) and functions for +generating UUIDs corresponding to a specific UUID version as specified in +RFC 4122/9562, e.g., uuid1() for UUID version 1, uuid3() for UUID version 3, +and so on. + +Note that UUID version 2 is deliberately omitted as it is outside the scope +of the RFC. If all you want is a unique ID, you should probably call uuid1() or uuid4(). Note that uuid1() may compromise privacy since it creates a UUID containing @@ -42,10 +46,19 @@ # make a UUID from a 16-byte string >>> uuid.UUID(bytes=x.bytes) UUID('00010203-0405-0607-0809-0a0b0c0d0e0f') + + # get the Nil UUID + >>> uuid.NIL + UUID('00000000-0000-0000-0000-000000000000') + + # get the Max UUID + >>> uuid.MAX + UUID('ffffffff-ffff-ffff-ffff-ffffffffffff') """ import os import sys +import time from enum import Enum, _simple_enum @@ -85,6 +98,19 @@ class SafeUUID: unknown = None +_UINT_128_MAX = (1 << 128) - 1 +# 128-bit mask to clear the variant and version bits of a UUID integral value +_RFC_4122_CLEARFLAGS_MASK = ~((0xf000 << 64) | (0xc000 << 48)) +# RFC 4122 variant bits and version bits to activate on a UUID integral value. +_RFC_4122_VERSION_1_FLAGS = ((1 << 76) | (0x8000 << 48)) +_RFC_4122_VERSION_3_FLAGS = ((3 << 76) | (0x8000 << 48)) +_RFC_4122_VERSION_4_FLAGS = ((4 << 76) | (0x8000 << 48)) +_RFC_4122_VERSION_5_FLAGS = ((5 << 76) | (0x8000 << 48)) +_RFC_4122_VERSION_6_FLAGS = ((6 << 76) | (0x8000 << 48)) +_RFC_4122_VERSION_7_FLAGS = ((7 << 76) | (0x8000 << 48)) +_RFC_4122_VERSION_8_FLAGS = ((8 << 76) | (0x8000 << 48)) + + class UUID: """Instances of the UUID class represent UUIDs as specified in RFC 4122. UUID objects are immutable, hashable, and usable as dictionary keys. @@ -108,7 +134,16 @@ class UUID: fields a tuple of the six integer fields of the UUID, which are also available as six individual attributes - and two derived attributes: + and two derived attributes. Those attributes are not + always relevant to all UUID versions: + + The 'time_*' attributes are only relevant to version 1. + + The 'clock_seq*' and 'node' attributes are only relevant + to versions 1 and 6. + + The 'time' attribute is only relevant to versions 1, 6 + and 7. time_low the first 32 bits of the UUID time_mid the next 16 bits of the UUID @@ -117,19 +152,20 @@ class UUID: clock_seq_low the next 8 bits of the UUID node the last 48 bits of the UUID - time the 60-bit timestamp + time the 60-bit timestamp for UUIDv1/v6, + or the 48-bit timestamp for UUIDv7 clock_seq the 14-bit sequence number hex the UUID as a 32-character hexadecimal string int the UUID as a 128-bit integer - urn the UUID as a URN as specified in RFC 4122 + urn the UUID as a URN as specified in RFC 4122/9562 variant the UUID variant (one of the constants RESERVED_NCS, RFC_4122, RESERVED_MICROSOFT, or RESERVED_FUTURE) - version the UUID version number (1 through 5, meaningful only + version the UUID version number (1 through 8, meaningful only when the variant is RFC_4122) is_safe An enum indicating whether the UUID has been generated in @@ -174,57 +210,69 @@ def __init__(self, hex=None, bytes=None, bytes_le=None, fields=None, if [hex, bytes, bytes_le, fields, int].count(None) != 4: raise TypeError('one of the hex, bytes, bytes_le, fields, ' 'or int arguments must be given') - if hex is not None: + if int is not None: + pass + elif hex is not None: hex = hex.replace('urn:', '').replace('uuid:', '') hex = hex.strip('{}').replace('-', '') if len(hex) != 32: raise ValueError('badly formed hexadecimal UUID string') int = int_(hex, 16) - if bytes_le is not None: + elif bytes_le is not None: if len(bytes_le) != 16: raise ValueError('bytes_le is not a 16-char string') + assert isinstance(bytes_le, bytes_), repr(bytes_le) bytes = (bytes_le[4-1::-1] + bytes_le[6-1:4-1:-1] + bytes_le[8-1:6-1:-1] + bytes_le[8:]) - if bytes is not None: + int = int_.from_bytes(bytes) # big endian + elif bytes is not None: if len(bytes) != 16: raise ValueError('bytes is not a 16-char string') assert isinstance(bytes, bytes_), repr(bytes) int = int_.from_bytes(bytes) # big endian - if fields is not None: + elif fields is not None: if len(fields) != 6: raise ValueError('fields is not a 6-tuple') (time_low, time_mid, time_hi_version, clock_seq_hi_variant, clock_seq_low, node) = fields - if not 0 <= time_low < 1<<32: + if not 0 <= time_low < (1 << 32): raise ValueError('field 1 out of range (need a 32-bit value)') - if not 0 <= time_mid < 1<<16: + if not 0 <= time_mid < (1 << 16): raise ValueError('field 2 out of range (need a 16-bit value)') - if not 0 <= time_hi_version < 1<<16: + if not 0 <= time_hi_version < (1 << 16): raise ValueError('field 3 out of range (need a 16-bit value)') - if not 0 <= clock_seq_hi_variant < 1<<8: + if not 0 <= clock_seq_hi_variant < (1 << 8): raise ValueError('field 4 out of range (need an 8-bit value)') - if not 0 <= clock_seq_low < 1<<8: + if not 0 <= clock_seq_low < (1 << 8): raise ValueError('field 5 out of range (need an 8-bit value)') - if not 0 <= node < 1<<48: + if not 0 <= node < (1 << 48): raise ValueError('field 6 out of range (need a 48-bit value)') clock_seq = (clock_seq_hi_variant << 8) | clock_seq_low int = ((time_low << 96) | (time_mid << 80) | (time_hi_version << 64) | (clock_seq << 48) | node) - if int is not None: - if not 0 <= int < 1<<128: - raise ValueError('int is out of range (need a 128-bit value)') + if not 0 <= int <= _UINT_128_MAX: + raise ValueError('int is out of range (need a 128-bit value)') if version is not None: - if not 1 <= version <= 5: + if not 1 <= version <= 8: raise ValueError('illegal version number') - # Set the variant to RFC 4122. - int &= ~(0xc000 << 48) - int |= 0x8000 << 48 + # clear the variant and the version number bits + int &= _RFC_4122_CLEARFLAGS_MASK + # Set the variant to RFC 4122/9562. + int |= 0x8000_0000_0000_0000 # (0x8000 << 48) # Set the version number. - int &= ~(0xf000 << 64) int |= version << 76 object.__setattr__(self, 'int', int) object.__setattr__(self, 'is_safe', is_safe) + @classmethod + def _from_int(cls, value): + """Create a UUID from an integer *value*. Internal use only.""" + assert 0 <= value <= _UINT_128_MAX, repr(value) + self = object.__new__(cls) + object.__setattr__(self, 'int', value) + object.__setattr__(self, 'is_safe', SafeUUID.unknown) + return self + def __getstate__(self): d = {'int': self.int} if self.is_safe != SafeUUID.unknown: @@ -281,9 +329,8 @@ def __setattr__(self, name, value): raise TypeError('UUID objects are immutable') def __str__(self): - hex = '%032x' % self.int - return '%s-%s-%s-%s-%s' % ( - hex[:8], hex[8:12], hex[12:16], hex[16:20], hex[20:]) + x = self.hex + return f'{x[:8]}-{x[8:12]}-{x[12:16]}-{x[16:20]}-{x[20:]}' @property def bytes(self): @@ -322,8 +369,22 @@ def clock_seq_low(self): @property def time(self): - return (((self.time_hi_version & 0x0fff) << 48) | - (self.time_mid << 32) | self.time_low) + if self.version == 6: + # time_hi (32) | time_mid (16) | ver (4) | time_lo (12) | ... (64) + time_hi = self.int >> 96 + time_lo = (self.int >> 64) & 0x0fff + return time_hi << 28 | (self.time_mid << 12) | time_lo + elif self.version == 7: + # unix_ts_ms (48) | ... (80) + return self.int >> 80 + else: + # time_lo (32) | time_mid (16) | ver (4) | time_hi (12) | ... (64) + # + # For compatibility purposes, we do not warn or raise when the + # version is not 1 (timestamp is irrelevant to other versions). + time_hi = (self.int >> 64) & 0x0fff + time_lo = self.int >> 96 + return time_hi << 48 | (self.time_mid << 32) | time_lo @property def clock_seq(self): @@ -336,7 +397,7 @@ def node(self): @property def hex(self): - return '%032x' % self.int + return self.bytes.hex() @property def urn(self): @@ -355,7 +416,7 @@ def variant(self): @property def version(self): - # The version bits are only meaningful for RFC 4122 UUIDs. + # The version bits are only meaningful for RFC 4122/9562 UUIDs. if self.variant == RFC_4122: return int((self.int >> 76) & 0xf) @@ -374,7 +435,7 @@ def _get_command_stdout(command, *args): # for are actually localized, but in theory some system could do so.) env = dict(os.environ) env['LC_ALL'] = 'C' - # Empty strings will be quoted by popen so we should just ommit it + # Empty strings will be quoted by popen so we should just omit it if args != ('',): command = (executable, *args) else: @@ -572,7 +633,7 @@ def _netstat_getnode(): try: import _uuid _generate_time_safe = getattr(_uuid, "generate_time_safe", None) - _has_stable_extractable_node = getattr(_uuid, "has_stable_extractable_node", False) + _has_stable_extractable_node = _uuid.has_stable_extractable_node _UuidCreate = getattr(_uuid, "UuidCreate", None) except ImportError: _uuid = None @@ -679,7 +740,6 @@ def uuid1(node=None, clock_seq=None): return UUID(bytes=uuid_time, is_safe=is_safe) global _last_timestamp - import time nanoseconds = time.time_ns() # 0x01b21dd213814000 is the number of 100-ns intervals between the # UUID epoch 1582-10-15 00:00:00 and the Unix epoch 1970-01-01 00:00:00. @@ -704,24 +764,171 @@ def uuid3(namespace, name): """Generate a UUID from the MD5 hash of a namespace UUID and a name.""" if isinstance(name, str): name = bytes(name, "utf-8") - from hashlib import md5 - digest = md5( - namespace.bytes + name, - usedforsecurity=False - ).digest() - return UUID(bytes=digest[:16], version=3) + import hashlib + h = hashlib.md5(namespace.bytes + name, usedforsecurity=False) + int_uuid_3 = int.from_bytes(h.digest()) + int_uuid_3 &= _RFC_4122_CLEARFLAGS_MASK + int_uuid_3 |= _RFC_4122_VERSION_3_FLAGS + return UUID._from_int(int_uuid_3) def uuid4(): """Generate a random UUID.""" - return UUID(bytes=os.urandom(16), version=4) + int_uuid_4 = int.from_bytes(os.urandom(16)) + int_uuid_4 &= _RFC_4122_CLEARFLAGS_MASK + int_uuid_4 |= _RFC_4122_VERSION_4_FLAGS + return UUID._from_int(int_uuid_4) def uuid5(namespace, name): """Generate a UUID from the SHA-1 hash of a namespace UUID and a name.""" if isinstance(name, str): name = bytes(name, "utf-8") - from hashlib import sha1 - hash = sha1(namespace.bytes + name).digest() - return UUID(bytes=hash[:16], version=5) + import hashlib + h = hashlib.sha1(namespace.bytes + name, usedforsecurity=False) + int_uuid_5 = int.from_bytes(h.digest()[:16]) + int_uuid_5 &= _RFC_4122_CLEARFLAGS_MASK + int_uuid_5 |= _RFC_4122_VERSION_5_FLAGS + return UUID._from_int(int_uuid_5) + + +_last_timestamp_v6 = None + +def uuid6(node=None, clock_seq=None): + """Similar to :func:`uuid1` but where fields are ordered differently + for improved DB locality. + + More precisely, given a 60-bit timestamp value as specified for UUIDv1, + for UUIDv6 the first 48 most significant bits are stored first, followed + by the 4-bit version (same position), followed by the remaining 12 bits + of the original 60-bit timestamp. + """ + global _last_timestamp_v6 + import time + nanoseconds = time.time_ns() + # 0x01b21dd213814000 is the number of 100-ns intervals between the + # UUID epoch 1582-10-15 00:00:00 and the Unix epoch 1970-01-01 00:00:00. + timestamp = nanoseconds // 100 + 0x01b21dd213814000 + if _last_timestamp_v6 is not None and timestamp <= _last_timestamp_v6: + timestamp = _last_timestamp_v6 + 1 + _last_timestamp_v6 = timestamp + if clock_seq is None: + import random + clock_seq = random.getrandbits(14) # instead of stable storage + time_hi_and_mid = (timestamp >> 12) & 0xffff_ffff_ffff + time_lo = timestamp & 0x0fff # keep 12 bits and clear version bits + clock_s = clock_seq & 0x3fff # keep 14 bits and clear variant bits + if node is None: + node = getnode() + # --- 32 + 16 --- -- 4 -- -- 12 -- -- 2 -- -- 14 --- 48 + # time_hi_and_mid | version | time_lo | variant | clock_seq | node + int_uuid_6 = time_hi_and_mid << 80 + int_uuid_6 |= time_lo << 64 + int_uuid_6 |= clock_s << 48 + int_uuid_6 |= node & 0xffff_ffff_ffff + # by construction, the variant and version bits are already cleared + int_uuid_6 |= _RFC_4122_VERSION_6_FLAGS + return UUID._from_int(int_uuid_6) + + +_last_timestamp_v7 = None +_last_counter_v7 = 0 # 42-bit counter + +def _uuid7_get_counter_and_tail(): + rand = int.from_bytes(os.urandom(10)) + # 42-bit counter with MSB set to 0 + counter = (rand >> 32) & 0x1ff_ffff_ffff + # 32-bit random data + tail = rand & 0xffff_ffff + return counter, tail + + +def uuid7(): + """Generate a UUID from a Unix timestamp in milliseconds and random bits. + + UUIDv7 objects feature monotonicity within a millisecond. + """ + # --- 48 --- -- 4 -- --- 12 --- -- 2 -- --- 30 --- - 32 - + # unix_ts_ms | version | counter_hi | variant | counter_lo | random + # + # 'counter = counter_hi | counter_lo' is a 42-bit counter constructed + # with Method 1 of RFC 9562, §6.2, and its MSB is set to 0. + # + # 'random' is a 32-bit random value regenerated for every new UUID. + # + # If multiple UUIDs are generated within the same millisecond, the LSB + # of 'counter' is incremented by 1. When overflowing, the timestamp is + # advanced and the counter is reset to a random 42-bit integer with MSB + # set to 0. + + global _last_timestamp_v7 + global _last_counter_v7 + + nanoseconds = time.time_ns() + timestamp_ms = nanoseconds // 1_000_000 + + if _last_timestamp_v7 is None or timestamp_ms > _last_timestamp_v7: + counter, tail = _uuid7_get_counter_and_tail() + else: + if timestamp_ms < _last_timestamp_v7: + timestamp_ms = _last_timestamp_v7 + 1 + # advance the 42-bit counter + counter = _last_counter_v7 + 1 + if counter > 0x3ff_ffff_ffff: + # advance the 48-bit timestamp + timestamp_ms += 1 + counter, tail = _uuid7_get_counter_and_tail() + else: + # 32-bit random data + tail = int.from_bytes(os.urandom(4)) + + unix_ts_ms = timestamp_ms & 0xffff_ffff_ffff + counter_msbs = counter >> 30 + # keep 12 counter's MSBs and clear variant bits + counter_hi = counter_msbs & 0x0fff + # keep 30 counter's LSBs and clear version bits + counter_lo = counter & 0x3fff_ffff + # ensure that the tail is always a 32-bit integer (by construction, + # it is already the case, but future interfaces may allow the user + # to specify the random tail) + tail &= 0xffff_ffff + + int_uuid_7 = unix_ts_ms << 80 + int_uuid_7 |= counter_hi << 64 + int_uuid_7 |= counter_lo << 32 + int_uuid_7 |= tail + # by construction, the variant and version bits are already cleared + int_uuid_7 |= _RFC_4122_VERSION_7_FLAGS + res = UUID._from_int(int_uuid_7) + + # defer global update until all computations are done + _last_timestamp_v7 = timestamp_ms + _last_counter_v7 = counter + return res + + +def uuid8(a=None, b=None, c=None): + """Generate a UUID from three custom blocks. + + * 'a' is the first 48-bit chunk of the UUID (octets 0-5); + * 'b' is the mid 12-bit chunk (octets 6-7); + * 'c' is the last 62-bit chunk (octets 8-15). + + When a value is not specified, a pseudo-random value is generated. + """ + if a is None: + import random + a = random.getrandbits(48) + if b is None: + import random + b = random.getrandbits(12) + if c is None: + import random + c = random.getrandbits(62) + int_uuid_8 = (a & 0xffff_ffff_ffff) << 80 + int_uuid_8 |= (b & 0xfff) << 64 + int_uuid_8 |= c & 0x3fff_ffff_ffff_ffff + # by construction, the variant and version bits are already cleared + int_uuid_8 |= _RFC_4122_VERSION_8_FLAGS + return UUID._from_int(int_uuid_8) def main(): @@ -730,7 +937,10 @@ def main(): "uuid1": uuid1, "uuid3": uuid3, "uuid4": uuid4, - "uuid5": uuid5 + "uuid5": uuid5, + "uuid6": uuid6, + "uuid7": uuid7, + "uuid8": uuid8, } uuid_namespace_funcs = ("uuid3", "uuid5") namespaces = { @@ -742,18 +952,24 @@ def main(): import argparse parser = argparse.ArgumentParser( - description="Generates a uuid using the selected uuid function.") - parser.add_argument("-u", "--uuid", choices=uuid_funcs.keys(), default="uuid4", - help="The function to use to generate the uuid. " - "By default uuid4 function is used.") + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + description="Generate a UUID using the selected UUID function.", + color=True, + ) + parser.add_argument("-u", "--uuid", + choices=uuid_funcs.keys(), + default="uuid4", + help="function to generate the UUID") parser.add_argument("-n", "--namespace", - help="The namespace is a UUID, or '@ns' where 'ns' is a " - "well-known predefined UUID addressed by namespace name. " - "Such as @dns, @url, @oid, and @x500. " - "Only required for uuid3/uuid5 functions.") + choices=["any UUID", *namespaces.keys()], + help="uuid3/uuid5 only: " + "a UUID, or a well-known predefined UUID addressed " + "by namespace name") parser.add_argument("-N", "--name", - help="The name used as part of generating the uuid. " - "Only required for uuid3/uuid5 functions.") + help="uuid3/uuid5 only: " + "name used as part of generating the UUID") + parser.add_argument("-C", "--count", metavar="NUM", type=int, default=1, + help="generate NUM fresh UUIDs") args = parser.parse_args() uuid_func = uuid_funcs[args.uuid] @@ -768,9 +984,11 @@ def main(): "Run 'python -m uuid -h' for more information." ) namespace = namespaces[namespace] if namespace in namespaces else UUID(namespace) - print(uuid_func(namespace, name)) + for _ in range(args.count): + print(uuid_func(namespace, name)) else: - print(uuid_func()) + for _ in range(args.count): + print(uuid_func()) # The following standard UUIDs are for use with uuid3() or uuid5(). @@ -780,5 +998,10 @@ def main(): NAMESPACE_OID = UUID('6ba7b812-9dad-11d1-80b4-00c04fd430c8') NAMESPACE_X500 = UUID('6ba7b814-9dad-11d1-80b4-00c04fd430c8') +# RFC 9562 Sections 5.9 and 5.10 define the special Nil and Max UUID formats. + +NIL = UUID('00000000-0000-0000-0000-000000000000') +MAX = UUID('ffffffff-ffff-ffff-ffff-ffffffffffff') + if __name__ == "__main__": main() diff --git a/Lib/venv/scripts/nt/venvlauncher.exe b/Lib/venv/scripts/nt/venvlauncher.exe index 2439c22aa93..c6863b56e57 100644 Binary files a/Lib/venv/scripts/nt/venvlauncher.exe and b/Lib/venv/scripts/nt/venvlauncher.exe differ diff --git a/Lib/venv/scripts/nt/venvlaunchert.exe b/Lib/venv/scripts/nt/venvlaunchert.exe index 99f5f5e9fca..c12a7a869f4 100644 Binary files a/Lib/venv/scripts/nt/venvlaunchert.exe and b/Lib/venv/scripts/nt/venvlaunchert.exe differ diff --git a/Lib/venv/scripts/nt/venvwlauncher.exe b/Lib/venv/scripts/nt/venvwlauncher.exe index 6c43c2e9d93..d0d3733266f 100644 Binary files a/Lib/venv/scripts/nt/venvwlauncher.exe and b/Lib/venv/scripts/nt/venvwlauncher.exe differ diff --git a/Lib/venv/scripts/nt/venvwlaunchert.exe b/Lib/venv/scripts/nt/venvwlaunchert.exe index 74f40deb046..9456a9e9b4a 100644 Binary files a/Lib/venv/scripts/nt/venvwlaunchert.exe and b/Lib/venv/scripts/nt/venvwlaunchert.exe differ diff --git a/Lib/warnings.py b/Lib/warnings.py index f83aaf231ea..6759857d909 100644 --- a/Lib/warnings.py +++ b/Lib/warnings.py @@ -1,735 +1,99 @@ -"""Python part of the warnings subsystem.""" - import sys +__all__ = [ + "warn", + "warn_explicit", + "showwarning", + "formatwarning", + "filterwarnings", + "simplefilter", + "resetwarnings", + "catch_warnings", + "deprecated", +] + +from _py_warnings import ( + WarningMessage, + _DEPRECATED_MSG, + _OptionError, + _add_filter, + _deprecated, + _filters_mutated, + _filters_mutated_lock_held, + _filters_version, + _formatwarning_orig, + _formatwarnmsg, + _formatwarnmsg_impl, + _get_context, + _get_filters, + _getaction, + _getcategory, + _is_filename_to_skip, + _is_internal_filename, + _is_internal_frame, + _lock, + _new_context, + _next_external_frame, + _processoptions, + _set_context, + _set_module, + _setoption, + _setup_defaults, + _showwarning_orig, + _showwarnmsg, + _showwarnmsg_impl, + _use_context, + _warn_unawaited_coroutine, + _warnings_context, + catch_warnings, + defaultaction, + deprecated, + filters, + filterwarnings, + formatwarning, + onceregistry, + resetwarnings, + showwarning, + simplefilter, + warn, + warn_explicit, +) -__all__ = ["warn", "warn_explicit", "showwarning", - "formatwarning", "filterwarnings", "simplefilter", - "resetwarnings", "catch_warnings", "deprecated"] - -def showwarning(message, category, filename, lineno, file=None, line=None): - """Hook to write a warning to a file; replace if you like.""" - msg = WarningMessage(message, category, filename, lineno, file, line) - _showwarnmsg_impl(msg) - -def formatwarning(message, category, filename, lineno, line=None): - """Function to format a warning the standard way.""" - msg = WarningMessage(message, category, filename, lineno, None, line) - return _formatwarnmsg_impl(msg) - -def _showwarnmsg_impl(msg): - file = msg.file - if file is None: - file = sys.stderr - if file is None: - # sys.stderr is None when run with pythonw.exe: - # warnings get lost - return - text = _formatwarnmsg(msg) - try: - file.write(text) - except OSError: - # the file (probably stderr) is invalid - this warning gets lost. - pass - -def _formatwarnmsg_impl(msg): - category = msg.category.__name__ - s = f"{msg.filename}:{msg.lineno}: {category}: {msg.message}\n" - - if msg.line is None: - try: - import linecache - line = linecache.getline(msg.filename, msg.lineno) - except Exception: - # When a warning is logged during Python shutdown, linecache - # and the import machinery don't work anymore - line = None - linecache = None - else: - line = msg.line - if line: - line = line.strip() - s += " %s\n" % line - - if msg.source is not None: - try: - import tracemalloc - # Logging a warning should not raise a new exception: - # catch Exception, not only ImportError and RecursionError. - except Exception: - # don't suggest to enable tracemalloc if it's not available - suggest_tracemalloc = False - tb = None - else: - try: - suggest_tracemalloc = not tracemalloc.is_tracing() - tb = tracemalloc.get_object_traceback(msg.source) - except Exception: - # When a warning is logged during Python shutdown, tracemalloc - # and the import machinery don't work anymore - suggest_tracemalloc = False - tb = None - - if tb is not None: - s += 'Object allocated at (most recent call last):\n' - for frame in tb: - s += (' File "%s", lineno %s\n' - % (frame.filename, frame.lineno)) - - try: - if linecache is not None: - line = linecache.getline(frame.filename, frame.lineno) - else: - line = None - except Exception: - line = None - if line: - line = line.strip() - s += ' %s\n' % line - elif suggest_tracemalloc: - s += (f'{category}: Enable tracemalloc to get the object ' - f'allocation traceback\n') - return s - -# Keep a reference to check if the function was replaced -_showwarning_orig = showwarning - -def _showwarnmsg(msg): - """Hook to write a warning to a file; replace if you like.""" - try: - sw = showwarning - except NameError: - pass - else: - if sw is not _showwarning_orig: - # warnings.showwarning() was replaced - if not callable(sw): - raise TypeError("warnings.showwarning() must be set to a " - "function or method") - - sw(msg.message, msg.category, msg.filename, msg.lineno, - msg.file, msg.line) - return - _showwarnmsg_impl(msg) - -# Keep a reference to check if the function was replaced -_formatwarning_orig = formatwarning - -def _formatwarnmsg(msg): - """Function to format a warning the standard way.""" - try: - fw = formatwarning - except NameError: - pass - else: - if fw is not _formatwarning_orig: - # warnings.formatwarning() was replaced - return fw(msg.message, msg.category, - msg.filename, msg.lineno, msg.line) - return _formatwarnmsg_impl(msg) - -def filterwarnings(action, message="", category=Warning, module="", lineno=0, - append=False): - """Insert an entry into the list of warnings filters (at the front). - - 'action' -- one of "error", "ignore", "always", "default", "module", - or "once" - 'message' -- a regex that the warning message must match - 'category' -- a class that the warning must be a subclass of - 'module' -- a regex that the module name must match - 'lineno' -- an integer line number, 0 matches all warnings - 'append' -- if true, append to the list of filters - """ - if action not in {"error", "ignore", "always", "default", "module", "once"}: - raise ValueError(f"invalid action: {action!r}") - if not isinstance(message, str): - raise TypeError("message must be a string") - if not isinstance(category, type) or not issubclass(category, Warning): - raise TypeError("category must be a Warning subclass") - if not isinstance(module, str): - raise TypeError("module must be a string") - if not isinstance(lineno, int): - raise TypeError("lineno must be an int") - if lineno < 0: - raise ValueError("lineno must be an int >= 0") - - if message or module: - import re - - if message: - message = re.compile(message, re.I) - else: - message = None - if module: - module = re.compile(module) - else: - module = None - - _add_filter(action, message, category, module, lineno, append=append) - -def simplefilter(action, category=Warning, lineno=0, append=False): - """Insert a simple entry into the list of warnings filters (at the front). - - A simple filter matches all modules and messages. - 'action' -- one of "error", "ignore", "always", "default", "module", - or "once" - 'category' -- a class that the warning must be a subclass of - 'lineno' -- an integer line number, 0 matches all warnings - 'append' -- if true, append to the list of filters - """ - if action not in {"error", "ignore", "always", "default", "module", "once"}: - raise ValueError(f"invalid action: {action!r}") - if not isinstance(lineno, int): - raise TypeError("lineno must be an int") - if lineno < 0: - raise ValueError("lineno must be an int >= 0") - _add_filter(action, None, category, None, lineno, append=append) - -def _add_filter(*item, append): - # Remove possible duplicate filters, so new one will be placed - # in correct place. If append=True and duplicate exists, do nothing. - if not append: - try: - filters.remove(item) - except ValueError: - pass - filters.insert(0, item) - else: - if item not in filters: - filters.append(item) - _filters_mutated() - -def resetwarnings(): - """Clear the list of warning filters, so that no filters are active.""" - filters[:] = [] - _filters_mutated() - -class _OptionError(Exception): - """Exception used by option processing helpers.""" - pass - -# Helper to process -W options passed via sys.warnoptions -def _processoptions(args): - for arg in args: - try: - _setoption(arg) - except _OptionError as msg: - print("Invalid -W option ignored:", msg, file=sys.stderr) - -# Helper for _processoptions() -def _setoption(arg): - parts = arg.split(':') - if len(parts) > 5: - raise _OptionError("too many fields (max 5): %r" % (arg,)) - while len(parts) < 5: - parts.append('') - action, message, category, module, lineno = [s.strip() - for s in parts] - action = _getaction(action) - category = _getcategory(category) - if message or module: - import re - if message: - message = re.escape(message) - if module: - module = re.escape(module) + r'\Z' - if lineno: - try: - lineno = int(lineno) - if lineno < 0: - raise ValueError - except (ValueError, OverflowError): - raise _OptionError("invalid lineno %r" % (lineno,)) from None - else: - lineno = 0 - filterwarnings(action, message, category, module, lineno) - -# Helper for _setoption() -def _getaction(action): - if not action: - return "default" - if action == "all": return "always" # Alias - for a in ('default', 'always', 'ignore', 'module', 'once', 'error'): - if a.startswith(action): - return a - raise _OptionError("invalid action: %r" % (action,)) - -# Helper for _setoption() -def _getcategory(category): - if not category: - return Warning - if '.' not in category: - import builtins as m - klass = category - else: - module, _, klass = category.rpartition('.') - try: - m = __import__(module, None, None, [klass]) - except ImportError: - raise _OptionError("invalid module name: %r" % (module,)) from None - try: - cat = getattr(m, klass) - except AttributeError: - raise _OptionError("unknown warning category: %r" % (category,)) from None - if not issubclass(cat, Warning): - raise _OptionError("invalid warning category: %r" % (category,)) - return cat - - -def _is_internal_filename(filename): - return 'importlib' in filename and '_bootstrap' in filename - - -def _is_filename_to_skip(filename, skip_file_prefixes): - return any(filename.startswith(prefix) for prefix in skip_file_prefixes) - - -def _is_internal_frame(frame): - """Signal whether the frame is an internal CPython implementation detail.""" - return _is_internal_filename(frame.f_code.co_filename) - - -def _next_external_frame(frame, skip_file_prefixes): - """Find the next frame that doesn't involve Python or user internals.""" - frame = frame.f_back - while frame is not None and ( - _is_internal_filename(filename := frame.f_code.co_filename) or - _is_filename_to_skip(filename, skip_file_prefixes)): - frame = frame.f_back - return frame - - -# Code typically replaced by _warnings -def warn(message, category=None, stacklevel=1, source=None, - *, skip_file_prefixes=()): - """Issue a warning, or maybe ignore it or raise an exception.""" - # Check if message is already a Warning object - if isinstance(message, Warning): - category = message.__class__ - # Check category argument - if category is None: - category = UserWarning - if not (isinstance(category, type) and issubclass(category, Warning)): - raise TypeError("category must be a Warning subclass, " - "not '{:s}'".format(type(category).__name__)) - if not isinstance(skip_file_prefixes, tuple): - # The C version demands a tuple for implementation performance. - raise TypeError('skip_file_prefixes must be a tuple of strs.') - if skip_file_prefixes: - stacklevel = max(2, stacklevel) - # Get context information - try: - if stacklevel <= 1 or _is_internal_frame(sys._getframe(1)): - # If frame is too small to care or if the warning originated in - # internal code, then do not try to hide any frames. - frame = sys._getframe(stacklevel) - else: - frame = sys._getframe(1) - # Look for one frame less since the above line starts us off. - for x in range(stacklevel-1): - frame = _next_external_frame(frame, skip_file_prefixes) - if frame is None: - raise ValueError - except ValueError: - globals = sys.__dict__ - filename = "<sys>" - lineno = 0 - else: - globals = frame.f_globals - filename = frame.f_code.co_filename - lineno = frame.f_lineno - if '__name__' in globals: - module = globals['__name__'] - else: - module = "<string>" - registry = globals.setdefault("__warningregistry__", {}) - warn_explicit(message, category, filename, lineno, module, registry, - globals, source) - -def warn_explicit(message, category, filename, lineno, - module=None, registry=None, module_globals=None, - source=None): - lineno = int(lineno) - if module is None: - module = filename or "<unknown>" - if module[-3:].lower() == ".py": - module = module[:-3] # XXX What about leading pathname? - if registry is None: - registry = {} - if registry.get('version', 0) != _filters_version: - registry.clear() - registry['version'] = _filters_version - if isinstance(message, Warning): - text = str(message) - category = message.__class__ - else: - text = message - message = category(message) - key = (text, category, lineno) - # Quick test for common case - if registry.get(key): - return - # Search the filters - for item in filters: - action, msg, cat, mod, ln = item - if ((msg is None or msg.match(text)) and - issubclass(category, cat) and - (mod is None or mod.match(module)) and - (ln == 0 or lineno == ln)): - break - else: - action = defaultaction - # Early exit actions - if action == "ignore": - return - - # Prime the linecache for formatting, in case the - # "file" is actually in a zipfile or something. - import linecache - linecache.getlines(filename, module_globals) - - if action == "error": - raise message - # Other actions - if action == "once": - registry[key] = 1 - oncekey = (text, category) - if onceregistry.get(oncekey): - return - onceregistry[oncekey] = 1 - elif action == "always": - pass - elif action == "module": - registry[key] = 1 - altkey = (text, category, 0) - if registry.get(altkey): - return - registry[altkey] = 1 - elif action == "default": - registry[key] = 1 - else: - # Unrecognized actions are errors - raise RuntimeError( - "Unrecognized action (%r) in warnings.filters:\n %s" % - (action, item)) - # Print message and context - msg = WarningMessage(message, category, filename, lineno, source=source) - _showwarnmsg(msg) - - -class WarningMessage(object): - - _WARNING_DETAILS = ("message", "category", "filename", "lineno", "file", - "line", "source") - - def __init__(self, message, category, filename, lineno, file=None, - line=None, source=None): - self.message = message - self.category = category - self.filename = filename - self.lineno = lineno - self.file = file - self.line = line - self.source = source - self._category_name = category.__name__ if category else None - - def __str__(self): - return ("{message : %r, category : %r, filename : %r, lineno : %s, " - "line : %r}" % (self.message, self._category_name, - self.filename, self.lineno, self.line)) - - -class catch_warnings(object): - - """A context manager that copies and restores the warnings filter upon - exiting the context. - - The 'record' argument specifies whether warnings should be captured by a - custom implementation of warnings.showwarning() and be appended to a list - returned by the context manager. Otherwise None is returned by the context - manager. The objects appended to the list are arguments whose attributes - mirror the arguments to showwarning(). - - The 'module' argument is to specify an alternative module to the module - named 'warnings' and imported under that name. This argument is only useful - when testing the warnings module itself. - - If the 'action' argument is not None, the remaining arguments are passed - to warnings.simplefilter() as if it were called immediately on entering the - context. - """ - - def __init__(self, *, record=False, module=None, - action=None, category=Warning, lineno=0, append=False): - """Specify whether to record warnings and if an alternative module - should be used other than sys.modules['warnings']. - - For compatibility with Python 3.0, please consider all arguments to be - keyword-only. - - """ - self._record = record - self._module = sys.modules['warnings'] if module is None else module - self._entered = False - if action is None: - self._filter = None - else: - self._filter = (action, category, lineno, append) - - def __repr__(self): - args = [] - if self._record: - args.append("record=True") - if self._module is not sys.modules['warnings']: - args.append("module=%r" % self._module) - name = type(self).__name__ - return "%s(%s)" % (name, ", ".join(args)) - - def __enter__(self): - if self._entered: - raise RuntimeError("Cannot enter %r twice" % self) - self._entered = True - self._filters = self._module.filters - self._module.filters = self._filters[:] - self._module._filters_mutated() - self._showwarning = self._module.showwarning - self._showwarnmsg_impl = self._module._showwarnmsg_impl - if self._filter is not None: - simplefilter(*self._filter) - if self._record: - log = [] - self._module._showwarnmsg_impl = log.append - # Reset showwarning() to the default implementation to make sure - # that _showwarnmsg() calls _showwarnmsg_impl() - self._module.showwarning = self._module._showwarning_orig - return log - else: - return None - - def __exit__(self, *exc_info): - if not self._entered: - raise RuntimeError("Cannot exit %r without entering first" % self) - self._module.filters = self._filters - self._module._filters_mutated() - self._module.showwarning = self._showwarning - self._module._showwarnmsg_impl = self._showwarnmsg_impl - - -class deprecated: - """Indicate that a class, function or overload is deprecated. - - When this decorator is applied to an object, the type checker - will generate a diagnostic on usage of the deprecated object. - - Usage: - - @deprecated("Use B instead") - class A: - pass - - @deprecated("Use g instead") - def f(): - pass - - @overload - @deprecated("int support is deprecated") - def g(x: int) -> int: ... - @overload - def g(x: str) -> int: ... - - The warning specified by *category* will be emitted at runtime - on use of deprecated objects. For functions, that happens on calls; - for classes, on instantiation and on creation of subclasses. - If the *category* is ``None``, no warning is emitted at runtime. - The *stacklevel* determines where the - warning is emitted. If it is ``1`` (the default), the warning - is emitted at the direct caller of the deprecated object; if it - is higher, it is emitted further up the stack. - Static type checker behavior is not affected by the *category* - and *stacklevel* arguments. - - The deprecation message passed to the decorator is saved in the - ``__deprecated__`` attribute on the decorated object. - If applied to an overload, the decorator - must be after the ``@overload`` decorator for the attribute to - exist on the overload as returned by ``get_overloads()``. - - See PEP 702 for details. - - """ - def __init__( - self, - message: str, - /, - *, - category: type[Warning] | None = DeprecationWarning, - stacklevel: int = 1, - ) -> None: - if not isinstance(message, str): - raise TypeError( - f"Expected an object of type str for 'message', not {type(message).__name__!r}" - ) - self.message = message - self.category = category - self.stacklevel = stacklevel - - def __call__(self, arg, /): - # Make sure the inner functions created below don't - # retain a reference to self. - msg = self.message - category = self.category - stacklevel = self.stacklevel - if category is None: - arg.__deprecated__ = msg - return arg - elif isinstance(arg, type): - import functools - from types import MethodType - - original_new = arg.__new__ - - @functools.wraps(original_new) - def __new__(cls, /, *args, **kwargs): - if cls is arg: - warn(msg, category=category, stacklevel=stacklevel + 1) - if original_new is not object.__new__: - return original_new(cls, *args, **kwargs) - # Mirrors a similar check in object.__new__. - elif cls.__init__ is object.__init__ and (args or kwargs): - raise TypeError(f"{cls.__name__}() takes no arguments") - else: - return original_new(cls) - - arg.__new__ = staticmethod(__new__) - - original_init_subclass = arg.__init_subclass__ - # We need slightly different behavior if __init_subclass__ - # is a bound method (likely if it was implemented in Python) - if isinstance(original_init_subclass, MethodType): - original_init_subclass = original_init_subclass.__func__ - - @functools.wraps(original_init_subclass) - def __init_subclass__(*args, **kwargs): - warn(msg, category=category, stacklevel=stacklevel + 1) - return original_init_subclass(*args, **kwargs) - - arg.__init_subclass__ = classmethod(__init_subclass__) - # Or otherwise, which likely means it's a builtin such as - # object's implementation of __init_subclass__. - else: - @functools.wraps(original_init_subclass) - def __init_subclass__(*args, **kwargs): - warn(msg, category=category, stacklevel=stacklevel + 1) - return original_init_subclass(*args, **kwargs) - - arg.__init_subclass__ = __init_subclass__ - - arg.__deprecated__ = __new__.__deprecated__ = msg - __init_subclass__.__deprecated__ = msg - return arg - elif callable(arg): - import functools - import inspect - - @functools.wraps(arg) - def wrapper(*args, **kwargs): - warn(msg, category=category, stacklevel=stacklevel + 1) - return arg(*args, **kwargs) - - if inspect.iscoroutinefunction(arg): - wrapper = inspect.markcoroutinefunction(wrapper) - - arg.__deprecated__ = wrapper.__deprecated__ = msg - return wrapper - else: - raise TypeError( - "@deprecated decorator with non-None category must be applied to " - f"a class or callable, not {arg!r}" - ) - - -_DEPRECATED_MSG = "{name!r} is deprecated and slated for removal in Python {remove}" - -def _deprecated(name, message=_DEPRECATED_MSG, *, remove, _version=sys.version_info): - """Warn that *name* is deprecated or should be removed. - - RuntimeError is raised if *remove* specifies a major/minor tuple older than - the current Python version or the same version but past the alpha. - - The *message* argument is formatted with *name* and *remove* as a Python - version tuple (e.g. (3, 11)). - - """ - remove_formatted = f"{remove[0]}.{remove[1]}" - if (_version[:2] > remove) or (_version[:2] == remove and _version[3] != "alpha"): - msg = f"{name!r} was slated for removal after Python {remove_formatted} alpha" - raise RuntimeError(msg) - else: - msg = message.format(name=name, remove=remove_formatted) - warn(msg, DeprecationWarning, stacklevel=3) - - -# Private utility function called by _PyErr_WarnUnawaitedCoroutine -def _warn_unawaited_coroutine(coro): - msg_lines = [ - f"coroutine '{coro.__qualname__}' was never awaited\n" - ] - if coro.cr_origin is not None: - import linecache, traceback - def extract(): - for filename, lineno, funcname in reversed(coro.cr_origin): - line = linecache.getline(filename, lineno) - yield (filename, lineno, funcname, line) - msg_lines.append("Coroutine created at (most recent call last)\n") - msg_lines += traceback.format_list(list(extract())) - msg = "".join(msg_lines).rstrip("\n") - # Passing source= here means that if the user happens to have tracemalloc - # enabled and tracking where the coroutine was created, the warning will - # contain that traceback. This does mean that if they have *both* - # coroutine origin tracking *and* tracemalloc enabled, they'll get two - # partially-redundant tracebacks. If we wanted to be clever we could - # probably detect this case and avoid it, but for now we don't bother. - warn(msg, category=RuntimeWarning, stacklevel=2, source=coro) - - -# filters contains a sequence of filter 5-tuples -# The components of the 5-tuple are: -# - an action: error, ignore, always, default, module, or once -# - a compiled regex that must match the warning message -# - a class representing the warning category -# - a compiled regex that must match the module that is being warned -# - a line number for the line being warning, or 0 to mean any line -# If either if the compiled regexs are None, match anything. try: - from _warnings import (filters, _defaultaction, _onceregistry, - warn, warn_explicit, _filters_mutated) - defaultaction = _defaultaction - onceregistry = _onceregistry + # Try to use the C extension, this will replace some parts of the + # _py_warnings implementation imported above. + from _warnings import ( + _acquire_lock, + _defaultaction as defaultaction, + _filters_mutated_lock_held, + _onceregistry as onceregistry, + _release_lock, + _warnings_context, + filters, + warn, + warn_explicit, + ) + _warnings_defaults = True -except ImportError: - filters = [] - defaultaction = "default" - onceregistry = {} - _filters_version = 1 + class _Lock: + def __enter__(self): + _acquire_lock() + return self - def _filters_mutated(): - global _filters_version - _filters_version += 1 + def __exit__(self, *args): + _release_lock() + _lock = _Lock() +except ImportError: _warnings_defaults = False # Module initialization +_set_module(sys.modules[__name__]) _processoptions(sys.warnoptions) if not _warnings_defaults: - # Several warning categories are ignored by default in regular builds - if not hasattr(sys, 'gettotalrefcount'): - filterwarnings("default", category=DeprecationWarning, - module="__main__", append=1) - simplefilter("ignore", category=DeprecationWarning, append=1) - simplefilter("ignore", category=PendingDeprecationWarning, append=1) - simplefilter("ignore", category=ImportWarning, append=1) - simplefilter("ignore", category=ResourceWarning, append=1) + _setup_defaults() del _warnings_defaults +del _setup_defaults diff --git a/Lib/weakref.py b/Lib/weakref.py index 25b70927e29..94e4278143c 100644 --- a/Lib/weakref.py +++ b/Lib/weakref.py @@ -19,7 +19,7 @@ ReferenceType, _remove_dead_weakref) -from _weakrefset import WeakSet, _IterationGuard +from _weakrefset import WeakSet import _collections_abc # Import after _weakref to avoid circular import. import sys @@ -105,34 +105,14 @@ def __init__(self, other=(), /, **kw): def remove(wr, selfref=ref(self), _atomic_removal=_remove_dead_weakref): self = selfref() if self is not None: - if self._iterating: - self._pending_removals.append(wr.key) - else: - # Atomic removal is necessary since this function - # can be called asynchronously by the GC - _atomic_removal(self.data, wr.key) + # Atomic removal is necessary since this function + # can be called asynchronously by the GC + _atomic_removal(self.data, wr.key) self._remove = remove - # A list of keys to be removed - self._pending_removals = [] - self._iterating = set() self.data = {} self.update(other, **kw) - def _commit_removals(self, _atomic_removal=_remove_dead_weakref): - pop = self._pending_removals.pop - d = self.data - # We shouldn't encounter any KeyError, because this method should - # always be called *before* mutating the dict. - while True: - try: - key = pop() - except IndexError: - return - _atomic_removal(d, key) - def __getitem__(self, key): - if self._pending_removals: - self._commit_removals() o = self.data[key]() if o is None: raise KeyError(key) @@ -140,18 +120,12 @@ def __getitem__(self, key): return o def __delitem__(self, key): - if self._pending_removals: - self._commit_removals() del self.data[key] def __len__(self): - if self._pending_removals: - self._commit_removals() return len(self.data) def __contains__(self, key): - if self._pending_removals: - self._commit_removals() try: o = self.data[key]() except KeyError: @@ -162,38 +136,28 @@ def __repr__(self): return "<%s at %#x>" % (self.__class__.__name__, id(self)) def __setitem__(self, key, value): - if self._pending_removals: - self._commit_removals() self.data[key] = KeyedRef(value, self._remove, key) def copy(self): - if self._pending_removals: - self._commit_removals() new = WeakValueDictionary() - with _IterationGuard(self): - for key, wr in self.data.items(): - o = wr() - if o is not None: - new[key] = o + for key, wr in self.data.copy().items(): + o = wr() + if o is not None: + new[key] = o return new __copy__ = copy def __deepcopy__(self, memo): from copy import deepcopy - if self._pending_removals: - self._commit_removals() new = self.__class__() - with _IterationGuard(self): - for key, wr in self.data.items(): - o = wr() - if o is not None: - new[deepcopy(key, memo)] = o + for key, wr in self.data.copy().items(): + o = wr() + if o is not None: + new[deepcopy(key, memo)] = o return new def get(self, key, default=None): - if self._pending_removals: - self._commit_removals() try: wr = self.data[key] except KeyError: @@ -207,21 +171,15 @@ def get(self, key, default=None): return o def items(self): - if self._pending_removals: - self._commit_removals() - with _IterationGuard(self): - for k, wr in self.data.items(): - v = wr() - if v is not None: - yield k, v + for k, wr in self.data.copy().items(): + v = wr() + if v is not None: + yield k, v def keys(self): - if self._pending_removals: - self._commit_removals() - with _IterationGuard(self): - for k, wr in self.data.items(): - if wr() is not None: - yield k + for k, wr in self.data.copy().items(): + if wr() is not None: + yield k __iter__ = keys @@ -235,23 +193,15 @@ def itervaluerefs(self): keep the values around longer than needed. """ - if self._pending_removals: - self._commit_removals() - with _IterationGuard(self): - yield from self.data.values() + yield from self.data.copy().values() def values(self): - if self._pending_removals: - self._commit_removals() - with _IterationGuard(self): - for wr in self.data.values(): - obj = wr() - if obj is not None: - yield obj + for wr in self.data.copy().values(): + obj = wr() + if obj is not None: + yield obj def popitem(self): - if self._pending_removals: - self._commit_removals() while True: key, wr = self.data.popitem() o = wr() @@ -259,8 +209,6 @@ def popitem(self): return key, o def pop(self, key, *args): - if self._pending_removals: - self._commit_removals() try: o = self.data.pop(key)() except KeyError: @@ -279,16 +227,12 @@ def setdefault(self, key, default=None): except KeyError: o = None if o is None: - if self._pending_removals: - self._commit_removals() self.data[key] = KeyedRef(default, self._remove, key) return default else: return o def update(self, other=None, /, **kwargs): - if self._pending_removals: - self._commit_removals() d = self.data if other is not None: if not hasattr(other, "items"): @@ -308,9 +252,7 @@ def valuerefs(self): keep the values around longer than needed. """ - if self._pending_removals: - self._commit_removals() - return list(self.data.values()) + return list(self.data.copy().values()) def __ior__(self, other): self.update(other) @@ -369,57 +311,22 @@ def __init__(self, dict=None): def remove(k, selfref=ref(self)): self = selfref() if self is not None: - if self._iterating: - self._pending_removals.append(k) - else: - try: - del self.data[k] - except KeyError: - pass + try: + del self.data[k] + except KeyError: + pass self._remove = remove - # A list of dead weakrefs (keys to be removed) - self._pending_removals = [] - self._iterating = set() - self._dirty_len = False if dict is not None: self.update(dict) - def _commit_removals(self): - # NOTE: We don't need to call this method before mutating the dict, - # because a dead weakref never compares equal to a live weakref, - # even if they happened to refer to equal objects. - # However, it means keys may already have been removed. - pop = self._pending_removals.pop - d = self.data - while True: - try: - key = pop() - except IndexError: - return - - try: - del d[key] - except KeyError: - pass - - def _scrub_removals(self): - d = self.data - self._pending_removals = [k for k in self._pending_removals if k in d] - self._dirty_len = False - def __delitem__(self, key): - self._dirty_len = True del self.data[ref(key)] def __getitem__(self, key): return self.data[ref(key)] def __len__(self): - if self._dirty_len and self._pending_removals: - # self._pending_removals may still contain keys which were - # explicitly removed, we have to scrub them (see issue #21173). - self._scrub_removals() - return len(self.data) - len(self._pending_removals) + return len(self.data) def __repr__(self): return "<%s at %#x>" % (self.__class__.__name__, id(self)) @@ -429,11 +336,10 @@ def __setitem__(self, key, value): def copy(self): new = WeakKeyDictionary() - with _IterationGuard(self): - for key, value in self.data.items(): - o = key() - if o is not None: - new[o] = value + for key, value in self.data.copy().items(): + o = key() + if o is not None: + new[o] = value return new __copy__ = copy @@ -441,11 +347,10 @@ def copy(self): def __deepcopy__(self, memo): from copy import deepcopy new = self.__class__() - with _IterationGuard(self): - for key, value in self.data.items(): - o = key() - if o is not None: - new[o] = deepcopy(value, memo) + for key, value in self.data.copy().items(): + o = key() + if o is not None: + new[o] = deepcopy(value, memo) return new def get(self, key, default=None): @@ -459,26 +364,23 @@ def __contains__(self, key): return wr in self.data def items(self): - with _IterationGuard(self): - for wr, value in self.data.items(): - key = wr() - if key is not None: - yield key, value + for wr, value in self.data.copy().items(): + key = wr() + if key is not None: + yield key, value def keys(self): - with _IterationGuard(self): - for wr in self.data: - obj = wr() - if obj is not None: - yield obj + for wr in self.data.copy(): + obj = wr() + if obj is not None: + yield obj __iter__ = keys def values(self): - with _IterationGuard(self): - for wr, value in self.data.items(): - if wr() is not None: - yield value + for wr, value in self.data.copy().items(): + if wr() is not None: + yield value def keyrefs(self): """Return a list of weak references to the keys. @@ -493,7 +395,6 @@ def keyrefs(self): return list(self.data) def popitem(self): - self._dirty_len = True while True: key, value = self.data.popitem() o = key() @@ -501,7 +402,6 @@ def popitem(self): return o, value def pop(self, key, *args): - self._dirty_len = True return self.data.pop(ref(key), *args) def setdefault(self, key, default=None): diff --git a/Lib/zipfile/__init__.py b/Lib/zipfile/__init__.py index 05f387a950b..c01f13729e1 100644 --- a/Lib/zipfile/__init__.py +++ b/Lib/zipfile/__init__.py @@ -245,7 +245,7 @@ def is_zipfile(filename): else: with open(filename, "rb") as fp: result = _check_zipfile(fp) - except OSError: + except (OSError, BadZipFile): pass return result @@ -253,16 +253,15 @@ def _EndRecData64(fpin, offset, endrec): """ Read the ZIP64 end-of-archive records and use that to update endrec """ - try: - fpin.seek(offset - sizeEndCentDir64Locator, 2) - except OSError: - # If the seek fails, the file is not large enough to contain a ZIP64 + offset -= sizeEndCentDir64Locator + if offset < 0: + # The file is not large enough to contain a ZIP64 # end-of-archive record, so just return the end record we were given. return endrec - + fpin.seek(offset) data = fpin.read(sizeEndCentDir64Locator) if len(data) != sizeEndCentDir64Locator: - return endrec + raise OSError("Unknown I/O error") sig, diskno, reloff, disks = struct.unpack(structEndArchive64Locator, data) if sig != stringEndArchive64Locator: return endrec @@ -270,16 +269,33 @@ def _EndRecData64(fpin, offset, endrec): if diskno != 0 or disks > 1: raise BadZipFile("zipfiles that span multiple disks are not supported") - # Assume no 'zip64 extensible data' - fpin.seek(offset - sizeEndCentDir64Locator - sizeEndCentDir64, 2) + offset -= sizeEndCentDir64 + if reloff > offset: + raise BadZipFile("Corrupt zip64 end of central directory locator") + # First, check the assumption that there is no prepended data. + fpin.seek(reloff) + extrasz = offset - reloff data = fpin.read(sizeEndCentDir64) if len(data) != sizeEndCentDir64: - return endrec + raise OSError("Unknown I/O error") + if not data.startswith(stringEndArchive64) and reloff != offset: + # Since we already have seen the Zip64 EOCD Locator, it's + # possible we got here because there is prepended data. + # Assume no 'zip64 extensible data' + fpin.seek(offset) + extrasz = 0 + data = fpin.read(sizeEndCentDir64) + if len(data) != sizeEndCentDir64: + raise OSError("Unknown I/O error") + if not data.startswith(stringEndArchive64): + raise BadZipFile("Zip64 end of central directory record not found") + sig, sz, create_version, read_version, disk_num, disk_dir, \ dircount, dircount2, dirsize, diroffset = \ struct.unpack(structEndArchive64, data) - if sig != stringEndArchive64: - return endrec + if (diroffset + dirsize != reloff or + sz + 12 != sizeEndCentDir64 + extrasz): + raise BadZipFile("Corrupt zip64 end of central directory record") # Update the original endrec using data from the ZIP64 record endrec[_ECD_SIGNATURE] = sig @@ -289,6 +305,7 @@ def _EndRecData64(fpin, offset, endrec): endrec[_ECD_ENTRIES_TOTAL] = dircount2 endrec[_ECD_SIZE] = dirsize endrec[_ECD_OFFSET] = diroffset + endrec[_ECD_LOCATION] = offset - extrasz return endrec @@ -322,7 +339,7 @@ def _EndRecData(fpin): endrec.append(filesize - sizeEndCentDir) # Try to read the "Zip64 end of central directory" structure - return _EndRecData64(fpin, -sizeEndCentDir, endrec) + return _EndRecData64(fpin, filesize - sizeEndCentDir, endrec) # Either this is not a ZIP file, or it is a ZIP file with an archive # comment. Search the end of the file for the "end of central directory" @@ -346,8 +363,7 @@ def _EndRecData(fpin): endrec.append(maxCommentStart + start) # Try to read the "Zip64 end of central directory" structure - return _EndRecData64(fpin, maxCommentStart + start - filesize, - endrec) + return _EndRecData64(fpin, maxCommentStart + start, endrec) # Unable to find a valid end of central directory structure return None @@ -1458,9 +1474,6 @@ def _RealGetContents(self): # "concat" is zero, unless zip was concatenated to another file concat = endrec[_ECD_LOCATION] - size_cd - offset_cd - if endrec[_ECD_SIGNATURE] == stringEndArchive64: - # If Zip64 extension structures are present, account for them - concat -= (sizeEndCentDir64 + sizeEndCentDir64Locator) if self.debug > 2: inferred = concat + offset_cd @@ -2082,7 +2095,7 @@ def _write_end_record(self): " would require ZIP64 extensions") zip64endrec = struct.pack( structEndArchive64, stringEndArchive64, - 44, 45, 45, 0, 0, centDirCount, centDirCount, + sizeEndCentDir64 - 12, 45, 45, 0, 0, centDirCount, centDirCount, centDirSize, centDirOffset) self.fp.write(zip64endrec) diff --git a/Lib/zipimport.py b/Lib/zipimport.py index 25eaee9c0f2..444c9dd11d8 100644 --- a/Lib/zipimport.py +++ b/Lib/zipimport.py @@ -1,11 +1,9 @@ """zipimport provides support for importing Python modules from Zip archives. -This module exports three objects: +This module exports two objects: - zipimporter: a class; its constructor takes a path to a Zip archive. - ZipImportError: exception raised by zipimporter objects. It's a subclass of ImportError, so it can be caught as ImportError, too. -- _zip_directory_cache: a dict, mapping archive paths to zip directory - info dicts, as used in zipimporter._files. It is usually not needed to use the zipimport module explicitly; it is used by the builtin import mechanism for sys.path items that are paths @@ -15,14 +13,13 @@ #from importlib import _bootstrap_external #from importlib import _bootstrap # for _verbose_message import _frozen_importlib_external as _bootstrap_external -from _frozen_importlib_external import _unpack_uint16, _unpack_uint32 +from _frozen_importlib_external import _unpack_uint16, _unpack_uint32, _unpack_uint64 import _frozen_importlib as _bootstrap # for _verbose_message import _imp # for check_hash_based_pycs import _io # for open import marshal # for loads import sys # for modules import time # for mktime -import _warnings # For warn() __all__ = ['ZipImportError', 'zipimporter'] @@ -40,8 +37,14 @@ class ZipImportError(ImportError): _module_type = type(sys) END_CENTRAL_DIR_SIZE = 22 -STRING_END_ARCHIVE = b'PK\x05\x06' +END_CENTRAL_DIR_SIZE_64 = 56 +END_CENTRAL_DIR_LOCATOR_SIZE_64 = 20 +STRING_END_ARCHIVE = b'PK\x05\x06' # standard EOCD signature +STRING_END_LOCATOR_64 = b'PK\x06\x07' # Zip64 EOCD Locator signature +STRING_END_ZIP_64 = b'PK\x06\x06' # Zip64 EOCD signature MAX_COMMENT_LEN = (1 << 16) - 1 +MAX_UINT32 = 0xffffffff +ZIP64_EXTRA_TAG = 0x1 class zipimporter(_bootstrap_external._LoaderBasics): """zipimporter(archivepath) -> zipimporter object @@ -63,8 +66,7 @@ class zipimporter(_bootstrap_external._LoaderBasics): # if found, or else read it from the archive. def __init__(self, path): if not isinstance(path, str): - import os - path = os.fsdecode(path) + raise TypeError(f"expected str, not {type(path)!r}") if not path: raise ZipImportError('archive path is empty', path=path) if alt_path_sep: @@ -89,12 +91,8 @@ def __init__(self, path): raise ZipImportError('not a Zip file', path=path) break - try: - files = _zip_directory_cache[path] - except KeyError: - files = _read_directory(path) - _zip_directory_cache[path] = files - self._files = files + if path not in _zip_directory_cache: + _zip_directory_cache[path] = _read_directory(path) self.archive = path # a prefix directory following the ZIP file path. self.prefix = _bootstrap_external._path_join(*prefix[::-1]) @@ -102,64 +100,6 @@ def __init__(self, path): self.prefix += path_sep - # Check whether we can satisfy the import of the module named by - # 'fullname', or whether it could be a portion of a namespace - # package. Return self if we can load it, a string containing the - # full path if it's a possible namespace portion, None if we - # can't load it. - def find_loader(self, fullname, path=None): - """find_loader(fullname, path=None) -> self, str or None. - - Search for a module specified by 'fullname'. 'fullname' must be the - fully qualified (dotted) module name. It returns the zipimporter - instance itself if the module was found, a string containing the - full path name if it's possibly a portion of a namespace package, - or None otherwise. The optional 'path' argument is ignored -- it's - there for compatibility with the importer protocol. - - Deprecated since Python 3.10. Use find_spec() instead. - """ - _warnings.warn("zipimporter.find_loader() is deprecated and slated for " - "removal in Python 3.12; use find_spec() instead", - DeprecationWarning) - mi = _get_module_info(self, fullname) - if mi is not None: - # This is a module or package. - return self, [] - - # Not a module or regular package. See if this is a directory, and - # therefore possibly a portion of a namespace package. - - # We're only interested in the last path component of fullname - # earlier components are recorded in self.prefix. - modpath = _get_module_path(self, fullname) - if _is_dir(self, modpath): - # This is possibly a portion of a namespace - # package. Return the string representing its path, - # without a trailing separator. - return None, [f'{self.archive}{path_sep}{modpath}'] - - return None, [] - - - # Check whether we can satisfy the import of the module named by - # 'fullname'. Return self if we can, None if we can't. - def find_module(self, fullname, path=None): - """find_module(fullname, path=None) -> self or None. - - Search for a module specified by 'fullname'. 'fullname' must be the - fully qualified (dotted) module name. It returns the zipimporter - instance itself if the module was found, or None if it wasn't. - The optional 'path' argument is ignored -- it's there for compatibility - with the importer protocol. - - Deprecated since Python 3.10. Use find_spec() instead. - """ - _warnings.warn("zipimporter.find_module() is deprecated and slated for " - "removal in Python 3.12; use find_spec() instead", - DeprecationWarning) - return self.find_loader(fullname, path)[0] - def find_spec(self, fullname, target=None): """Create a ModuleSpec for the specified module. @@ -211,9 +151,11 @@ def get_data(self, pathname): key = pathname[len(self.archive + path_sep):] try: - toc_entry = self._files[key] + toc_entry = self._get_files()[key] except KeyError: raise OSError(0, '', key) + if toc_entry is None: + return b'' return _get_data(self.archive, toc_entry) @@ -248,7 +190,7 @@ def get_source(self, fullname): fullpath = f'{path}.py' try: - toc_entry = self._files[fullpath] + toc_entry = self._get_files()[fullpath] except KeyError: # we have the module, but no source return None @@ -278,9 +220,11 @@ def load_module(self, fullname): Deprecated since Python 3.10. Use exec_module() instead. """ - msg = ("zipimport.zipimporter.load_module() is deprecated and slated for " - "removal in Python 3.12; use exec_module() instead") - _warnings.warn(msg, DeprecationWarning) + import warnings + warnings._deprecated("zipimport.zipimporter.load_module", + f"{warnings._DEPRECATED_MSG}; " + "use zipimport.zipimporter.exec_module() instead", + remove=(3, 15)) code, ispackage, modpath = _get_module_code(self, fullname) mod = sys.modules.get(fullname) if mod is None or not isinstance(mod, _module_type): @@ -313,28 +257,28 @@ def load_module(self, fullname): def get_resource_reader(self, fullname): - """Return the ResourceReader for a package in a zip file. - - If 'fullname' is a package within the zip file, return the - 'ResourceReader' object for the package. Otherwise return None. - """ - try: - if not self.is_package(fullname): - return None - except ZipImportError: - return None + """Return the ResourceReader for a module in a zip file.""" from importlib.readers import ZipReader + return ZipReader(self, fullname) - def invalidate_caches(self): - """Reload the file data of the archive path.""" + def _get_files(self): + """Return the files within the archive path.""" try: - self._files = _read_directory(self.archive) - _zip_directory_cache[self.archive] = self._files - except ZipImportError: - _zip_directory_cache.pop(self.archive, None) - self._files = {} + files = _zip_directory_cache[self.archive] + except KeyError: + try: + files = _zip_directory_cache[self.archive] = _read_directory(self.archive) + except ZipImportError: + files = {} + + return files + + + def invalidate_caches(self): + """Invalidates the cache of file data of the archive path.""" + _zip_directory_cache.pop(self.archive, None) def __repr__(self): @@ -364,15 +308,15 @@ def _is_dir(self, path): # of a namespace package. We test by seeing if the name, with an # appended path separator, exists. dirpath = path + path_sep - # If dirpath is present in self._files, we have a directory. - return dirpath in self._files + # If dirpath is present in self._get_files(), we have a directory. + return dirpath in self._get_files() # Return some information about a module. def _get_module_info(self, fullname): path = _get_module_path(self, fullname) for suffix, isbytecode, ispackage in _zip_searchorder: fullpath = path + suffix - if fullpath in self._files: + if fullpath in self._get_files(): return ispackage return None @@ -406,16 +350,11 @@ def _read_directory(archive): raise ZipImportError(f"can't open Zip file: {archive!r}", path=archive) with fp: + # GH-87235: On macOS all file descriptors for /dev/fd/N share the same + # file offset, reset the file offset after scanning the zipfile directory + # to not cause problems when some runs 'python3 /dev/fd/9 9<some_script' + start_offset = fp.tell() try: - fp.seek(-END_CENTRAL_DIR_SIZE, 2) - header_position = fp.tell() - buffer = fp.read(END_CENTRAL_DIR_SIZE) - except OSError: - raise ZipImportError(f"can't read Zip file: {archive!r}", path=archive) - if len(buffer) != END_CENTRAL_DIR_SIZE: - raise ZipImportError(f"can't read Zip file: {archive!r}", path=archive) - if buffer[:4] != STRING_END_ARCHIVE: - # Bad: End of Central Dir signature # Check if there's a comment. try: fp.seek(0, 2) @@ -423,98 +362,209 @@ def _read_directory(archive): except OSError: raise ZipImportError(f"can't read Zip file: {archive!r}", path=archive) - max_comment_start = max(file_size - MAX_COMMENT_LEN - - END_CENTRAL_DIR_SIZE, 0) + max_comment_plus_dirs_size = ( + MAX_COMMENT_LEN + END_CENTRAL_DIR_SIZE + + END_CENTRAL_DIR_SIZE_64 + END_CENTRAL_DIR_LOCATOR_SIZE_64) + max_comment_start = max(file_size - max_comment_plus_dirs_size, 0) try: fp.seek(max_comment_start) - data = fp.read() + data = fp.read(max_comment_plus_dirs_size) except OSError: raise ZipImportError(f"can't read Zip file: {archive!r}", path=archive) pos = data.rfind(STRING_END_ARCHIVE) - if pos < 0: + pos64 = data.rfind(STRING_END_ZIP_64) + + if (pos64 >= 0 and pos64+END_CENTRAL_DIR_SIZE_64+END_CENTRAL_DIR_LOCATOR_SIZE_64==pos): + # Zip64 at "correct" offset from standard EOCD + buffer = data[pos64:pos64 + END_CENTRAL_DIR_SIZE_64] + if len(buffer) != END_CENTRAL_DIR_SIZE_64: + raise ZipImportError( + f"corrupt Zip64 file: Expected {END_CENTRAL_DIR_SIZE_64} byte " + f"zip64 central directory, but read {len(buffer)} bytes.", + path=archive) + header_position = file_size - len(data) + pos64 + + central_directory_size = _unpack_uint64(buffer[40:48]) + central_directory_position = _unpack_uint64(buffer[48:56]) + num_entries = _unpack_uint64(buffer[24:32]) + elif pos >= 0: + buffer = data[pos:pos+END_CENTRAL_DIR_SIZE] + if len(buffer) != END_CENTRAL_DIR_SIZE: + raise ZipImportError(f"corrupt Zip file: {archive!r}", + path=archive) + + header_position = file_size - len(data) + pos + + # Buffer now contains a valid EOCD, and header_position gives the + # starting position of it. + central_directory_size = _unpack_uint32(buffer[12:16]) + central_directory_position = _unpack_uint32(buffer[16:20]) + num_entries = _unpack_uint16(buffer[8:10]) + + # N.b. if someday you want to prefer the standard (non-zip64) EOCD, + # you need to adjust position by 76 for arc to be 0. + else: raise ZipImportError(f'not a Zip file: {archive!r}', path=archive) - buffer = data[pos:pos+END_CENTRAL_DIR_SIZE] - if len(buffer) != END_CENTRAL_DIR_SIZE: - raise ZipImportError(f"corrupt Zip file: {archive!r}", - path=archive) - header_position = file_size - len(data) + pos - - header_size = _unpack_uint32(buffer[12:16]) - header_offset = _unpack_uint32(buffer[16:20]) - if header_position < header_size: - raise ZipImportError(f'bad central directory size: {archive!r}', path=archive) - if header_position < header_offset: - raise ZipImportError(f'bad central directory offset: {archive!r}', path=archive) - header_position -= header_size - arc_offset = header_position - header_offset - if arc_offset < 0: - raise ZipImportError(f'bad central directory size or offset: {archive!r}', path=archive) - - files = {} - # Start of Central Directory - count = 0 - try: - fp.seek(header_position) - except OSError: - raise ZipImportError(f"can't read Zip file: {archive!r}", path=archive) - while True: - buffer = fp.read(46) - if len(buffer) < 4: - raise EOFError('EOF read where not expected') - # Start of file header - if buffer[:4] != b'PK\x01\x02': - break # Bad: Central Dir File Header - if len(buffer) != 46: - raise EOFError('EOF read where not expected') - flags = _unpack_uint16(buffer[8:10]) - compress = _unpack_uint16(buffer[10:12]) - time = _unpack_uint16(buffer[12:14]) - date = _unpack_uint16(buffer[14:16]) - crc = _unpack_uint32(buffer[16:20]) - data_size = _unpack_uint32(buffer[20:24]) - file_size = _unpack_uint32(buffer[24:28]) - name_size = _unpack_uint16(buffer[28:30]) - extra_size = _unpack_uint16(buffer[30:32]) - comment_size = _unpack_uint16(buffer[32:34]) - file_offset = _unpack_uint32(buffer[42:46]) - header_size = name_size + extra_size + comment_size - if file_offset > header_offset: - raise ZipImportError(f'bad local header offset: {archive!r}', path=archive) - file_offset += arc_offset + # Buffer now contains a valid EOCD, and header_position gives the + # starting position of it. + # XXX: These are cursory checks but are not as exact or strict as they + # could be. Checking the arc-adjusted value is probably good too. + if header_position < central_directory_size: + raise ZipImportError(f'bad central directory size: {archive!r}', path=archive) + if header_position < central_directory_position: + raise ZipImportError(f'bad central directory offset: {archive!r}', path=archive) + header_position -= central_directory_size + # On just-a-zipfile these values are the same and arc_offset is zero; if + # the file has some bytes prepended, `arc_offset` is the number of such + # bytes. This is used for pex as well as self-extracting .exe. + arc_offset = header_position - central_directory_position + if arc_offset < 0: + raise ZipImportError(f'bad central directory size or offset: {archive!r}', path=archive) + + files = {} + # Start of Central Directory + count = 0 try: - name = fp.read(name_size) - except OSError: - raise ZipImportError(f"can't read Zip file: {archive!r}", path=archive) - if len(name) != name_size: - raise ZipImportError(f"can't read Zip file: {archive!r}", path=archive) - # On Windows, calling fseek to skip over the fields we don't use is - # slower than reading the data because fseek flushes stdio's - # internal buffers. See issue #8745. - try: - if len(fp.read(header_size - name_size)) != header_size - name_size: - raise ZipImportError(f"can't read Zip file: {archive!r}", path=archive) + fp.seek(header_position) except OSError: raise ZipImportError(f"can't read Zip file: {archive!r}", path=archive) + while True: + buffer = fp.read(46) + if len(buffer) < 4: + raise EOFError('EOF read where not expected') + # Start of file header + if buffer[:4] != b'PK\x01\x02': + if count != num_entries: + raise ZipImportError( + f"mismatched num_entries: {count} should be {num_entries} in {archive!r}", + path=archive, + ) + break # Bad: Central Dir File Header + if len(buffer) != 46: + raise EOFError('EOF read where not expected') + flags = _unpack_uint16(buffer[8:10]) + compress = _unpack_uint16(buffer[10:12]) + time = _unpack_uint16(buffer[12:14]) + date = _unpack_uint16(buffer[14:16]) + crc = _unpack_uint32(buffer[16:20]) + data_size = _unpack_uint32(buffer[20:24]) + file_size = _unpack_uint32(buffer[24:28]) + name_size = _unpack_uint16(buffer[28:30]) + extra_size = _unpack_uint16(buffer[30:32]) + comment_size = _unpack_uint16(buffer[32:34]) + file_offset = _unpack_uint32(buffer[42:46]) + header_size = name_size + extra_size + comment_size - if flags & 0x800: - # UTF-8 file names extension - name = name.decode() - else: - # Historical ZIP filename encoding try: - name = name.decode('ascii') - except UnicodeDecodeError: - name = name.decode('latin1').translate(cp437_table) - - name = name.replace('/', path_sep) - path = _bootstrap_external._path_join(archive, name) - t = (path, compress, data_size, file_size, file_offset, time, date, crc) - files[name] = t - count += 1 + name = fp.read(name_size) + except OSError: + raise ZipImportError(f"can't read Zip file: {archive!r}", path=archive) + if len(name) != name_size: + raise ZipImportError(f"can't read Zip file: {archive!r}", path=archive) + # On Windows, calling fseek to skip over the fields we don't use is + # slower than reading the data because fseek flushes stdio's + # internal buffers. See issue #8745. + try: + extra_data_len = header_size - name_size + extra_data = memoryview(fp.read(extra_data_len)) + + if len(extra_data) != extra_data_len: + raise ZipImportError(f"can't read Zip file: {archive!r}", path=archive) + except OSError: + raise ZipImportError(f"can't read Zip file: {archive!r}", path=archive) + + if flags & 0x800: + # UTF-8 file names extension + name = name.decode() + else: + # Historical ZIP filename encoding + try: + name = name.decode('ascii') + except UnicodeDecodeError: + name = name.decode('latin1').translate(cp437_table) + + name = name.replace('/', path_sep) + path = _bootstrap_external._path_join(archive, name) + + # Ordering matches unpacking below. + if ( + file_size == MAX_UINT32 or + data_size == MAX_UINT32 or + file_offset == MAX_UINT32 + ): + # need to decode extra_data looking for a zip64 extra (which might not + # be present) + while extra_data: + if len(extra_data) < 4: + raise ZipImportError(f"can't read header extra: {archive!r}", path=archive) + tag = _unpack_uint16(extra_data[:2]) + size = _unpack_uint16(extra_data[2:4]) + if len(extra_data) < 4 + size: + raise ZipImportError(f"can't read header extra: {archive!r}", path=archive) + if tag == ZIP64_EXTRA_TAG: + if (len(extra_data) - 4) % 8 != 0: + raise ZipImportError(f"can't read header extra: {archive!r}", path=archive) + num_extra_values = (len(extra_data) - 4) // 8 + if num_extra_values > 3: + raise ZipImportError(f"can't read header extra: {archive!r}", path=archive) + import struct + values = list(struct.unpack_from(f"<{min(num_extra_values, 3)}Q", + extra_data, offset=4)) + + # N.b. Here be dragons: the ordering of these is different than + # the header fields, and it's really easy to get it wrong since + # naturally-occurring zips that use all 3 are >4GB + if file_size == MAX_UINT32: + file_size = values.pop(0) + if data_size == MAX_UINT32: + data_size = values.pop(0) + if file_offset == MAX_UINT32: + file_offset = values.pop(0) + + break + + # For a typical zip, this bytes-slicing only happens 2-3 times, on + # small data like timestamps and filesizes. + extra_data = extra_data[4+size:] + else: + _bootstrap._verbose_message( + "zipimport: suspected zip64 but no zip64 extra for {!r}", + path, + ) + # XXX These two statements seem swapped because `central_directory_position` + # is a position within the actual file, but `file_offset` (when compared) is + # as encoded in the entry, not adjusted for this file. + # N.b. this must be after we've potentially read the zip64 extra which can + # change `file_offset`. + if file_offset > central_directory_position: + raise ZipImportError(f'bad local header offset: {archive!r}', path=archive) + file_offset += arc_offset + + t = (path, compress, data_size, file_size, file_offset, time, date, crc) + files[name] = t + count += 1 + finally: + fp.seek(start_offset) _bootstrap._verbose_message('zipimport: found {} names in {!r}', count, archive) + + # Add implicit directories. + count = 0 + for name in list(files): + while True: + i = name.rstrip(path_sep).rfind(path_sep) + if i < 0: + break + name = name[:i + 1] + if name in files: + break + files[name] = None + count += 1 + if count: + _bootstrap._verbose_message('zipimport: added {} implicit directories in {!r}', + count, archive) return files # During bootstrap, we may need to load the encodings @@ -648,7 +698,7 @@ def _unmarshal_code(self, pathname, fullpath, fullname, data): source_bytes = _get_pyc_source(self, fullpath) if source_bytes is not None: source_hash = _imp.source_hash( - _bootstrap_external._RAW_MAGIC_NUMBER, + _imp.pyc_magic_number_token, source_bytes, ) @@ -708,7 +758,7 @@ def _get_mtime_and_size_of_source(self, path): # strip 'c' or 'o' from *.py[co] assert path[-1:] in ('c', 'o') path = path[:-1] - toc_entry = self._files[path] + toc_entry = self._get_files()[path] # fetch the time stamp of the .py file for comparison # with an embedded pyc time stamp time = toc_entry[5] @@ -728,7 +778,7 @@ def _get_pyc_source(self, path): path = path[:-1] try: - toc_entry = self._files[path] + toc_entry = self._get_files()[path] except KeyError: return None else: @@ -744,7 +794,7 @@ def _get_module_code(self, fullname): fullpath = path + suffix _bootstrap._verbose_message('trying {}{}{}', self.archive, path_sep, fullpath, verbosity=2) try: - toc_entry = self._files[fullpath] + toc_entry = self._get_files()[fullpath] except KeyError: pass else: diff --git a/README.md b/README.md index b3ddbe4e7cf..6949c6e66e2 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ # [RustPython](https://rustpython.github.io/) -A Python-3 (CPython >= 3.13.0) Interpreter written in Rust :snake: :scream: +A Python-3 (CPython >= 3.14.0) Interpreter written in Rust :snake: :scream: :metal:. [![Build Status](https://github.com/RustPython/RustPython/workflows/CI/badge.svg)](https://github.com/RustPython/RustPython/actions?query=workflow%3ACI) @@ -245,7 +245,7 @@ To enhance CPython compatibility, try to increase unittest coverage by checking Another approach is to checkout the source code: builtin functions and object methods are often the simplest and easiest way to contribute. -You can also simply run `python -I whats_left.py` to assist in finding any unimplemented +You can also simply run `python -I scripts/whats_left.py` to assist in finding any unimplemented method. ## Compiling to WebAssembly diff --git a/benches/_data/pypi_org__simple__psutil.json b/benches/_data/pypi_org__simple__psutil.json new file mode 100644 index 00000000000..91e2ff6b39e --- /dev/null +++ b/benches/_data/pypi_org__simple__psutil.json @@ -0,0 +1 @@ +{"alternate-locations":[],"files":[{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.1.1.tar.gz","hashes":{"sha256":"25c6caffbf00d8be77489391a784654e99fcbaf2a5278e80f748be4112ee0188"},"provenance":null,"requires-python":null,"size":44485,"upload-time":"2014-02-06T02:06:57.249874Z","url":"https://files.pythonhosted.org/packages/69/e4/7e36e3e6cbc83b76f1c93a63d4c053a03ca99f1c99b106835cb175b5932a/psutil-0.1.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.1.2.tar.gz","hashes":{"sha256":"4a13d7f760b043b263346e48823b1dfd4c202e97b23483e481e5ff696e74509e"},"provenance":null,"requires-python":null,"size":61640,"upload-time":"2014-02-06T02:06:51.674389Z","url":"https://files.pythonhosted.org/packages/6e/51/56198d83577106bf89cb23bffcb273f923aea8d5ffe03e3fce55f830c323/psutil-0.1.2.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.1.3.tar.gz","hashes":{"sha256":"43e327934b4a273da20d3a5797d6abcaab37914f61499d96fcf9e8e1ae75442b"},"provenance":null,"requires-python":null,"size":85749,"upload-time":"2014-02-06T02:06:45.294070Z","url":"https://files.pythonhosted.org/packages/1d/4f/dcfe500fd43e3d6b26d253cb0d7e6e3a7d80224b5059bd50c482aff62eef/psutil-0.1.3.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.0.tar.gz","hashes":{"sha256":"8de8efa92162c94f623297f522e440e818dc7b832f421f9f490324bc7d5d0d92"},"provenance":null,"requires-python":null,"size":129382,"upload-time":"2014-02-06T02:03:03.376283Z","url":"https://files.pythonhosted.org/packages/58/20/3457e441edc1625c6e1dbfcf780d2b22f2e9caa8606c3fd8ce6c48104e87/psutil-0.2.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.0.win-amd64-py2.7.exe","hashes":{"sha256":"4ce48cbbe3c915a9b720ce6465da34ce79465ffde63de8254cc9c8235fef824d"},"provenance":null,"requires-python":null,"size":291186,"upload-time":"2014-02-06T16:48:27.727869Z","url":"https://files.pythonhosted.org/packages/e6/1d/9a90eec0aec7e015d16f3922328336b4f8cd783f782e9ab81146b47ffee3/psutil-0.2.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.0.win-amd64-py3.3.exe","hashes":{"sha256":"eeba1fa1f2455219a05775044f7a76741252ea0bd4c762e7b652d281843c79ff"},"provenance":null,"requires-python":null,"size":289973,"upload-time":"2014-02-06T16:48:36.482056Z","url":"https://files.pythonhosted.org/packages/1d/75/7c67bc2c8304b137a8ff709d21cb2dd7f600bc5ee76ecc88f77ec008e69e/psutil-0.2.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.0.win-amd64-py3.4.exe","hashes":{"sha256":"919dabb5d47dc769a198a5d015c6765ef39b3b920c2ab6d04d05d9aa3ae6bb67"},"provenance":null,"requires-python":null,"size":289890,"upload-time":"2014-02-06T16:48:46.485205Z","url":"https://files.pythonhosted.org/packages/e0/e4/2ec24cecccf111a4dbda892b484bdfd8c00d3da5f99c1a0a79c469c62fb2/psutil-0.2.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.0.win32-py2.5.exe","hashes":{"sha256":"92f51445439f1082dab7733b9150b594b3797f4c96578457768c3e761f86c240"},"provenance":null,"requires-python":null,"size":129555,"upload-time":"2014-02-06T16:47:42.630117Z","url":"https://files.pythonhosted.org/packages/04/4f/478408899102af1d0d877be0b67e302fada1bdd8c8faaa4adc0e19550e53/psutil-0.2.0.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.0.win32-py2.6.exe","hashes":{"sha256":"c3e3dc863834d03485f6c48a735f76ab9a7cff2911d101c8cbd1e553c03f0fae"},"provenance":null,"requires-python":null,"size":262087,"upload-time":"2014-02-06T16:47:50.995531Z","url":"https://files.pythonhosted.org/packages/d7/f9/984326888f6c519cc813a96640cc9a2900dad85e15b9d368fe1f12a9bd11/psutil-0.2.0.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.0.win32-py2.7.exe","hashes":{"sha256":"08c6f16ac00babe2c54067dbc305c0587cb896a8eb64383290cdafdba74ea123"},"provenance":null,"requires-python":null,"size":261605,"upload-time":"2014-02-06T16:47:58.286157Z","url":"https://files.pythonhosted.org/packages/69/5c/be56c645a254ad83a9aa3055564aae543fdf23186aa5ec9c495a3937300b/psutil-0.2.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.0.win32-py3.3.exe","hashes":{"sha256":"e1f7cf0158a94e6b50f9a4b6f4258dec1bd89c9485c65d54b89ba2a09f0d51a3"},"provenance":null,"requires-python":null,"size":256763,"upload-time":"2014-02-06T16:48:06.663656Z","url":"https://files.pythonhosted.org/packages/76/42/d6813ce55af42553b2a3136e5159bd71e544bda5fdaed04c72759514e375/psutil-0.2.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.0.win32-py3.4.exe","hashes":{"sha256":"e49d818a4a49fd1ced40dac31076af622a40407234f717bc6a5c39b3f2335e81"},"provenance":null,"requires-python":null,"size":256718,"upload-time":"2014-02-06T16:48:16.971617Z","url":"https://files.pythonhosted.org/packages/f5/83/45b721a52001c1ba1f196e6b2ba3b12d99cbc03bf3f508f1fe56b4e25c66/psutil-0.2.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.1.tar.gz","hashes":{"sha256":"bd33f5f9e04b7677a932cc90d541ceec0050080d1b053ed39488ef39cb0fb4f4"},"provenance":null,"requires-python":null,"size":144657,"upload-time":"2014-02-06T02:06:37.338565Z","url":"https://files.pythonhosted.org/packages/9b/62/03133a1b4d1439227bc9b27389fc7d1137d111cbb15197094225967e21cf/psutil-0.2.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.1.win-amd64-py2.7.exe","hashes":{"sha256":"ab9d06c0c2a7baadebf66fbc74a2532520b3b6d4a58ba3c1f04311a175ed9e7f"},"provenance":null,"requires-python":null,"size":294651,"upload-time":"2014-02-06T16:46:46.406386Z","url":"https://files.pythonhosted.org/packages/55/3e/e49b7929b9daf9c3dcf31668a43ba669d22bb8606deca52e6f92cb67324d/psutil-0.2.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.1.win-amd64-py3.3.exe","hashes":{"sha256":"076c9914d8aadccd459e3efedc49c10f14d16918d0af5735c22a2bdcc3f90cac"},"provenance":null,"requires-python":null,"size":293431,"upload-time":"2014-02-06T16:46:54.986513Z","url":"https://files.pythonhosted.org/packages/34/f5/f471c56be91d10c9b83d22aa9f6f29006437f2196c89156f00f56f6e4661/psutil-0.2.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.1.win-amd64-py3.4.exe","hashes":{"sha256":"2679f8f9f20e61d85177f3fc6ee10bfbd6c4bd122b3f5774ba36d0035cac4f7a"},"provenance":null,"requires-python":null,"size":293352,"upload-time":"2014-02-06T16:47:04.453988Z","url":"https://files.pythonhosted.org/packages/68/ee/fc85a707394d1b374da61c071dee12299e779d87c09a86c727b6f30ad9a6/psutil-0.2.1.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.1.win32-py2.5.exe","hashes":{"sha256":"a8345934d7585c75d500098491f2a434dbba9e378fd908d76fbeb0bc20b8a3d3"},"provenance":null,"requires-python":null,"size":132806,"upload-time":"2014-02-06T16:46:02.328109Z","url":"https://files.pythonhosted.org/packages/ca/6e/5c0a87e2c1bac5e32471eb84b349f0808f6d6ed49b9118784af9d0e71baa/psutil-0.2.1.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.1.win32-py2.6.exe","hashes":{"sha256":"9b6294c3327625839fdbe92dca4093ab55e422fb348790b2ffc5c6cde163fe5b"},"provenance":null,"requires-python":null,"size":265263,"upload-time":"2014-02-06T16:46:11.006069Z","url":"https://files.pythonhosted.org/packages/e5/13/b358b509d4996df82ef77758c14c0999c443df537eefacc7bb24b6339758/psutil-0.2.1.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.1.win32-py2.7.exe","hashes":{"sha256":"ac4d86abc4c850141cd1e054a64f1fcb6977c4ccbbd65c5bac9bf9c8c5950cb1"},"provenance":null,"requires-python":null,"size":264780,"upload-time":"2014-02-06T16:46:19.079623Z","url":"https://files.pythonhosted.org/packages/a9/1e/8ad399f44f63de6de262fc503b1d3f091f6a9b22feaf3c273ee8bb7a7c38/psutil-0.2.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.1.win32-py3.3.exe","hashes":{"sha256":"60048c47698aa20936b7593636dd88454bae6c436852aef95fb9774b0551d06c"},"provenance":null,"requires-python":null,"size":259950,"upload-time":"2014-02-06T16:46:28.163437Z","url":"https://files.pythonhosted.org/packages/75/f3/296f43c033c0453ebfdf125dc9b8907193e31ffb92256e4b702d16ef8ac1/psutil-0.2.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.2.1.win32-py3.4.exe","hashes":{"sha256":"4aad401c1a45848107d395f97fc9cdb783acf120553ca1761dd77a7d52adb585"},"provenance":null,"requires-python":null,"size":259910,"upload-time":"2014-02-06T16:46:37.546918Z","url":"https://files.pythonhosted.org/packages/ae/8c/751f1e2fdcaa0ea817a8529468dca052b5de55033ee8539b996497b5be0e/psutil-0.2.1.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.3.0.tar.gz","hashes":{"sha256":"d70e16e60a575637f9d75a1005a8987d239700c0955134d4c8666a5aefbe16b8"},"provenance":null,"requires-python":null,"size":153990,"upload-time":"2014-02-06T02:06:31.022178Z","url":"https://files.pythonhosted.org/packages/f2/5c/4e74b08905dab9474a53534507e463fe8eec98b2fc0d29964b0c6a7f959d/psutil-0.3.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.3.0.win-amd64-py2.7.exe","hashes":{"sha256":"a554abcd10534ff32e3cbc4bd80b3c76a73d0bbb50739ee59482f0151703828b"},"provenance":null,"requires-python":null,"size":297305,"upload-time":"2014-02-06T16:45:05.336122Z","url":"https://files.pythonhosted.org/packages/78/91/21944fbddafa1bdf7c0c0ed7d9f4043cd8470e36a1f7a9f6e8b4406a8e39/psutil-0.3.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.3.0.win-amd64-py3.3.exe","hashes":{"sha256":"b33444731c63361ac316657bb305642e562097e870793bc57203cd9a4e4819a4"},"provenance":null,"requires-python":null,"size":296059,"upload-time":"2014-02-06T16:45:13.056233Z","url":"https://files.pythonhosted.org/packages/88/bd/5e9d566e941e165987b8c9bd61583c58242ef4da12b7d9fbb07494580aff/psutil-0.3.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.3.0.win-amd64-py3.4.exe","hashes":{"sha256":"89419f0cab4dd3ef4b32933237e34f58799069930fd25ed7f69deb9c64106c85"},"provenance":null,"requires-python":null,"size":296960,"upload-time":"2014-02-06T16:45:22.608830Z","url":"https://files.pythonhosted.org/packages/1f/06/c827177a80996d00f8f46ec99292e9d03fdf230a6e4fcea25ed9d19f3dcf/psutil-0.3.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.3.0.win32-py2.5.exe","hashes":{"sha256":"515b12e23581730c47e181fa163b457b90b6e848de0637d78b7fa8ca84ec2bc6"},"provenance":null,"requires-python":null,"size":135440,"upload-time":"2014-02-06T16:44:20.300778Z","url":"https://files.pythonhosted.org/packages/e5/55/6a353c646e6d15ee618348ed68cf79800f1275ade127f665779b6be0ed19/psutil-0.3.0.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.3.0.win32-py2.6.exe","hashes":{"sha256":"5bc91423ff25ad8dffb3c9979abdafb1ecd2e9a7ef4b9287bb731e76ea2c4b91"},"provenance":null,"requires-python":null,"size":267782,"upload-time":"2014-02-06T16:44:30.319267Z","url":"https://files.pythonhosted.org/packages/61/04/c447b4d468402bfb10d21995b0eae219c8397c00d9e8a2b209fb032ac1aa/psutil-0.3.0.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.3.0.win32-py2.7.exe","hashes":{"sha256":"4b7fbaca98b72f69561d593ef18b5254a8f98482b561db3d906aac4dd307bbcc"},"provenance":null,"requires-python":null,"size":267308,"upload-time":"2014-02-06T16:44:39.895580Z","url":"https://files.pythonhosted.org/packages/e4/2f/b060e631686e6455ff483ea5b6333382c6dddb5727edbfc9dcca1a8f024c/psutil-0.3.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.3.0.win32-py3.3.exe","hashes":{"sha256":"005217c21551618085e3323d653d75f9ad8f7418bff4fd5109850fba11950d6e"},"provenance":null,"requires-python":null,"size":262514,"upload-time":"2014-02-06T16:44:47.866655Z","url":"https://files.pythonhosted.org/packages/50/34/7bfc452cb2cd0069a81064db51b484ef135125cf1d246d0ff82c65653c81/psutil-0.3.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.3.0.win32-py3.4.exe","hashes":{"sha256":"331ac97b50def25e41bae637642d0131d5c4223b52ab55281d002549632ed5e0"},"provenance":null,"requires-python":null,"size":262479,"upload-time":"2014-02-06T16:44:56.381135Z","url":"https://files.pythonhosted.org/packages/b0/08/1944d2281986237cdaeabfa36195f44dc4c847d5c4fda3523b44c6823e19/psutil-0.3.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.0.tar.gz","hashes":{"sha256":"4b0ecc77d6c503449af3d2f0a41ad4cb8338e173f5d655a8239e41b1a49bc278"},"provenance":null,"requires-python":null,"size":167796,"upload-time":"2014-02-06T02:06:24.041544Z","url":"https://files.pythonhosted.org/packages/88/46/a933ab20c6d9b0ca5704b60307b9e80bdc119b759e89b74a2609b4c10eb6/psutil-0.4.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.0.win-amd64-py2.7.exe","hashes":{"sha256":"9689512e5a32508216bd6cc7cb2fbe938e4a47b483cd345b5bec6dda790a9210"},"provenance":null,"requires-python":null,"size":302844,"upload-time":"2014-02-06T16:43:18.853340Z","url":"https://files.pythonhosted.org/packages/a5/06/3ca1c85b733ceaced144cb211c1fc40a6b3b1c4f7a109bbb8e19fe09eac0/psutil-0.4.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.0.win-amd64-py3.3.exe","hashes":{"sha256":"9ee193dec84c2c91ac8712760eaaae0488dee92db37134dfcddc889ecd4f578d"},"provenance":null,"requires-python":null,"size":301654,"upload-time":"2014-02-06T16:43:28.863005Z","url":"https://files.pythonhosted.org/packages/ac/eb/f6923ea1b46803251f2bb42ad9b8ed90f4b8d04cdb2384c9ea535193e4f7/psutil-0.4.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.0.win-amd64-py3.4.exe","hashes":{"sha256":"94d2d0a24fa62f5655e9e21dd558592bf1d3d515d944eee70e9cd2e2bf2bf244"},"provenance":null,"requires-python":null,"size":302558,"upload-time":"2014-02-06T16:43:38.802980Z","url":"https://files.pythonhosted.org/packages/7c/5b/d9057a158b09c377eab80dae60928f8fe29bdd11635ca725098587981eeb/psutil-0.4.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.0.win32-py2.5.exe","hashes":{"sha256":"fab822fb8968b45695174a2c9b4d40e9ae81126343e24d38b627036579f1ee5e"},"provenance":null,"requires-python":null,"size":140768,"upload-time":"2014-02-06T16:42:35.663297Z","url":"https://files.pythonhosted.org/packages/07/1e/7827d1271248ddef694681747297aeb3f77dafe5c93ff707a625e3947082/psutil-0.4.0.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.0.win32-py2.6.exe","hashes":{"sha256":"ca511f56db8f1586612851a2eedbf80e38cf29f4a98f133410a74da284822747"},"provenance":null,"requires-python":null,"size":273280,"upload-time":"2014-02-06T16:42:44.697290Z","url":"https://files.pythonhosted.org/packages/b4/56/78f182bf9c81a978067ba5447eca2ea70ef2bc9ac94228adfae5428cd108/psutil-0.4.0.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.0.win32-py2.7.exe","hashes":{"sha256":"5b6cbf5aaeab55db1d9657fc75e4b54fa94d17d03ca97c94aa150e1bfdc8fb2e"},"provenance":null,"requires-python":null,"size":272807,"upload-time":"2014-02-06T16:42:52.281617Z","url":"https://files.pythonhosted.org/packages/7d/38/b719c8867699a71a98e92e61fd1b0f12bad997a9de60bf8f6215184692e8/psutil-0.4.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.0.win32-py3.3.exe","hashes":{"sha256":"dfe18b8eb9ee6dbbd49e28f245794c2fc22b56d770038668cb6e112935c49927"},"provenance":null,"requires-python":null,"size":268022,"upload-time":"2014-02-06T16:43:01.185161Z","url":"https://files.pythonhosted.org/packages/f4/2b/13be095a72c83fdbe11d519310cb34ff8e442d72ab52691218a83c216ac4/psutil-0.4.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.0.win32-py3.4.exe","hashes":{"sha256":"5b63e0e83dde30d62206497320c06b27c489ffb261e95ac1267b7a1b8d4ce72d"},"provenance":null,"requires-python":null,"size":267989,"upload-time":"2014-02-06T16:43:09.528893Z","url":"https://files.pythonhosted.org/packages/b4/8e/028b849600447c45c20582d7ddae1e39d31b83799289f54f1b79af664295/psutil-0.4.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.1.tar.gz","hashes":{"sha256":"33002c38f916835c949ae39a84f3f6d09ce01818ed805dfd61f8f3844c395c9d"},"provenance":null,"requires-python":null,"size":171549,"upload-time":"2014-02-06T02:06:17.394021Z","url":"https://files.pythonhosted.org/packages/a0/cd/00550e16a2a0357a9c24946160e60fc947bbe23bc276aff1b4d3e7b90345/psutil-0.4.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.1.win-amd64-py2.7.exe","hashes":{"sha256":"3e408028898b4ad09c207659c1b6cb8e6a74fb39ab3157b761e057a629118fec"},"provenance":null,"requires-python":null,"size":310414,"upload-time":"2014-02-06T16:41:41.640755Z","url":"https://files.pythonhosted.org/packages/06/75/3dc33773b1a1f7055b24245a0c8bec4d5776f1e1125329a13e82449007be/psutil-0.4.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.1.win-amd64-py3.3.exe","hashes":{"sha256":"457cfdecbf63f813bf86e2f736ab3b20447e8d8c2c516dd2a13a7463177f8bc6"},"provenance":null,"requires-python":null,"size":309169,"upload-time":"2014-02-06T16:41:50.926907Z","url":"https://files.pythonhosted.org/packages/6e/9e/6c7f5baf2529d9bba563cdff0b81413b5870b0681739318112adecd99ad0/psutil-0.4.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.1.win-amd64-py3.4.exe","hashes":{"sha256":"fc75e094cf573a7b9977be2a9d3a9d94671bc0ab098d52c391dd63227309cbea"},"provenance":null,"requires-python":null,"size":310087,"upload-time":"2014-02-06T16:42:00.068444Z","url":"https://files.pythonhosted.org/packages/cd/e6/9f963b32ab254b8c2bbedf5ce543cace8b08054924fd26fa233f99863ecc/psutil-0.4.1.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.1.win32-py2.5.exe","hashes":{"sha256":"ef4ed886b4d2d664fab5c369543690cda04e04a3b7496d72e369ef5240a8af6b"},"provenance":null,"requires-python":null,"size":148474,"upload-time":"2014-02-06T16:40:57.772154Z","url":"https://files.pythonhosted.org/packages/65/ad/e7828797558dd5f209694b02ce079cd3b6beacf6a5175f38c1973c688494/psutil-0.4.1.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.1.win32-py2.6.exe","hashes":{"sha256":"aafe9e328ffc173b26c78897c168858da335c85d4a02d666ad68fe2fe14601d1"},"provenance":null,"requires-python":null,"size":280897,"upload-time":"2014-02-06T16:41:07.265101Z","url":"https://files.pythonhosted.org/packages/1e/a1/594bf54e7c4056bc5284023be97f67c930175b3329e086a4ed8966cb067a/psutil-0.4.1.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.1.win32-py2.7.exe","hashes":{"sha256":"cefad502010c78425190498efc54541ae7bbe9eaa73ef4000cf11b032d32a8bb"},"provenance":null,"requires-python":null,"size":280445,"upload-time":"2014-02-06T16:41:15.492817Z","url":"https://files.pythonhosted.org/packages/c6/c4/14807de009a1beab2426b537379a8b05b1d69fef1fde7e23581cc332cdb3/psutil-0.4.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.1.win32-py3.3.exe","hashes":{"sha256":"a35ce124b2dec01d7632627c9370f5202978fce89ae59235cf4d5e05bbd0e02b"},"provenance":null,"requires-python":null,"size":275668,"upload-time":"2014-02-06T16:41:23.498162Z","url":"https://files.pythonhosted.org/packages/38/7d/407f7586bccdeb80f5da82d8ebb98712bfcc7217e3d7c3fc61b3bba893f2/psutil-0.4.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.4.1.win32-py3.4.exe","hashes":{"sha256":"860a039b67ee015a6304e8d05e6dc5c2a447eca331b7d6c9d04f41d7f2fc3a66"},"provenance":null,"requires-python":null,"size":275635,"upload-time":"2014-02-06T16:41:33.515203Z","url":"https://files.pythonhosted.org/packages/c5/b7/d2662ebf114961766c9eb3d9b737cb5541060d35e65b542b0acef470221a/psutil-0.4.1.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.0.tar.gz","hashes":{"sha256":"b10d1c19b9d334bea9f025f3b82b5664d642d26ba32ae0e71e7e20c5a6b4164f"},"provenance":null,"requires-python":null,"size":118411,"upload-time":"2014-02-06T02:06:09.471218Z","url":"https://files.pythonhosted.org/packages/3f/eb/e4115c3ecc189fd345b9a50d521e2ff76990bfb12a91921e93ea6398feef/psutil-0.5.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.0.win-amd64-py2.7.exe","hashes":{"sha256":"5bf7a91fd57be3a63c6b9fa78da63c185b9be909ca49032f3ceda662628be74d"},"provenance":null,"requires-python":null,"size":319335,"upload-time":"2014-02-06T15:53:45.976879Z","url":"https://files.pythonhosted.org/packages/c5/cc/9bb29fdd60e347617c3a09f51a595941ddb4777fc1608068d174ced5bfdd/psutil-0.5.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.0.win-amd64-py3.3.exe","hashes":{"sha256":"0cd9872d0d675b0314d2d15513e8ff3693a417e4ebaa158fc1f8780cc611574c"},"provenance":null,"requires-python":null,"size":317930,"upload-time":"2014-02-06T15:53:57.230392Z","url":"https://files.pythonhosted.org/packages/37/d8/276629523e0477639b33a606f93c2fc2e9c4cafba76b3a4cd1ddc42dd307/psutil-0.5.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.0.win-amd64-py3.4.exe","hashes":{"sha256":"13f71acf1d2fe08db072d890280e985e8414080bcd525c13c66e0e8fbc917710"},"provenance":null,"requires-python":null,"size":318790,"upload-time":"2014-02-06T15:54:08.039630Z","url":"https://files.pythonhosted.org/packages/5f/bc/a026bbb10a4df4cf458b5d6feabc72a0017e6a4f472898280442ff3f9393/psutil-0.5.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.0.win32-py2.5.exe","hashes":{"sha256":"d6839532038b97792d90341465430520e58ea3d0956a1df710b3fdd733dde587"},"provenance":null,"requires-python":null,"size":156923,"upload-time":"2014-02-06T16:39:37.158716Z","url":"https://files.pythonhosted.org/packages/26/83/d1aed38bad84576e48b9ca6501edcf829ca3b48631c823736c36fa2b24de/psutil-0.5.0.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.0.win32-py2.6.exe","hashes":{"sha256":"be79349ecf091470ea4920e45b78914b67f17bec9c23e4bea4ec0ca5fee1feab"},"provenance":null,"requires-python":null,"size":289457,"upload-time":"2014-02-06T15:53:07.782148Z","url":"https://files.pythonhosted.org/packages/4e/b2/bfdeae58283e0fb09e2c6725c01b9429acbd15917c7ead91c96f2df37d05/psutil-0.5.0.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.0.win32-py2.7.exe","hashes":{"sha256":"e4fa8e965eb58a885e65a360f1a7ffc7ed476e9227a2de3726091d7560c694a8"},"provenance":null,"requires-python":null,"size":289002,"upload-time":"2014-02-06T15:53:16.212429Z","url":"https://files.pythonhosted.org/packages/2d/62/ed3c23fb8648a916460e95306d58f5ba42b3b20c68f2a55d1fc6396190a5/psutil-0.5.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.0.win32-py3.3.exe","hashes":{"sha256":"0380ece3eeb6f170ba156c58eab2fa36508600b1c7e10bc9b7c651f721236f38"},"provenance":null,"requires-python":null,"size":284089,"upload-time":"2014-02-06T15:53:25.963338Z","url":"https://files.pythonhosted.org/packages/59/3d/25912eb67f01af306d8ec035041c07daa7f783aafa5f4e1dd90de91c4849/psutil-0.5.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.0.win32-py3.4.exe","hashes":{"sha256":"d6a9e67de81fc0fc58c790e1355ddfbb8a0760963868a538f217ac2325a81581"},"provenance":null,"requires-python":null,"size":284045,"upload-time":"2014-02-06T15:53:35.675392Z","url":"https://files.pythonhosted.org/packages/38/74/e6c5bfb482f011d19bb1366cecf7d4f0b5c9c9b7664ef31e951cc4d0757b/psutil-0.5.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.1.tar.gz","hashes":{"sha256":"f660d244a08373f5e89633650970819a59463b37af1c9d205699fb0d0608986d"},"provenance":null,"requires-python":null,"size":118781,"upload-time":"2014-02-06T02:06:02.078066Z","url":"https://files.pythonhosted.org/packages/aa/06/6ebc13a14c0961d7bb8184da530448d8fc198465eb5ecd1ad36c761807d2/psutil-0.5.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.1.win-amd64-py2.7.exe","hashes":{"sha256":"eff5ed677b6bdb5d75f1de925df774d8029582eea17abc3bbd31b9885d3c649d"},"provenance":null,"requires-python":null,"size":319512,"upload-time":"2014-02-06T16:36:10.664265Z","url":"https://files.pythonhosted.org/packages/8a/26/15751b500afdd0aea141905d5ba5319c38be41e0ca55a374c02827d1a79c/psutil-0.5.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.1.win-amd64-py3.3.exe","hashes":{"sha256":"8a12d8e368bfccfe2074616cae2472dfb1e136a2aaaea5b7dbafdb390166b202"},"provenance":null,"requires-python":null,"size":318245,"upload-time":"2014-02-06T16:36:21.268821Z","url":"https://files.pythonhosted.org/packages/13/9d/49f6fb5e1f75b3d6815bbdbbeb1630f7299d6ff96f85836dbd27989780ae/psutil-0.5.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.1.win-amd64-py3.4.exe","hashes":{"sha256":"9d991aae6b993ac58b540f27b2dff71137db62253f5d7acc5ede1ec845fd6b0b"},"provenance":null,"requires-python":null,"size":319083,"upload-time":"2014-02-06T16:36:31.287414Z","url":"https://files.pythonhosted.org/packages/2c/fa/a0469c75acedccc5006b655434e96a8a6a490725a9ae8208faaf39c043f6/psutil-0.5.1.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.1.win32-py2.5.exe","hashes":{"sha256":"ee32be6667bb69ebefc2fb5e23d115d98e307b5246c01ef50e45cfd8ad824d07"},"provenance":null,"requires-python":null,"size":157149,"upload-time":"2014-02-06T16:35:24.026975Z","url":"https://files.pythonhosted.org/packages/2e/30/9ed6283c7f1a716a3bf76a063b278ace2f89d1a8b2af9d35080dc2a0319b/psutil-0.5.1.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.1.win32-py2.6.exe","hashes":{"sha256":"f434f546d73366bff6283db907105b2607976ff9be67f2b5a7b35533e5f27219"},"provenance":null,"requires-python":null,"size":289555,"upload-time":"2014-02-06T16:35:32.000842Z","url":"https://files.pythonhosted.org/packages/eb/d9/55f5d02e42dc6c73f66c7cfc99bc3bf97cf10e9f259dbea69e123e0ef459/psutil-0.5.1.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.1.win32-py2.7.exe","hashes":{"sha256":"45acf090d65f2f47f3b1444b056cafb1b1ca70a8e266ddf10422734fb0ada897"},"provenance":null,"requires-python":null,"size":289095,"upload-time":"2014-02-06T16:35:40.182115Z","url":"https://files.pythonhosted.org/packages/81/25/cbbb80d1957c28dffb9f20c1eaf349d5642215d7ca15ec5b7015b6a510c8/psutil-0.5.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.1.win32-py3.3.exe","hashes":{"sha256":"9a20a99b6727334c8cc3a10ecc556d2f15a1e92efef97670ed022b86e09df36b"},"provenance":null,"requires-python":null,"size":284290,"upload-time":"2014-02-06T16:35:50.441013Z","url":"https://files.pythonhosted.org/packages/73/a2/ab4f2fa7ba6f8fc51d9cc9a0bb5ec269860a34574d391e39d20c2dca7b56/psutil-0.5.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.5.1.win32-py3.4.exe","hashes":{"sha256":"d547e164013a4d78909cbac0c176d65aa51c06de2fb888c63b738906f5ccc105"},"provenance":null,"requires-python":null,"size":284247,"upload-time":"2014-02-06T16:36:00.155604Z","url":"https://files.pythonhosted.org/packages/7f/8b/c9abd21cdc79b70744e169b98be06f0828687dfcac6468ed87455099f88d/psutil-0.5.1.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.0.tar.gz","hashes":{"sha256":"e918763243371f69bc98b601769df7337e196029dcdb797224f0ede474e17b94"},"provenance":null,"requires-python":null,"size":130803,"upload-time":"2014-02-06T02:05:55.032752Z","url":"https://files.pythonhosted.org/packages/3c/9f/0de622fc62e74f4c154656440c02c8a24a01a997d605a744272eb0d93742/psutil-0.6.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.0.win-amd64-py2.7.exe","hashes":{"sha256":"1792be8fca4b3f88553e1c5cc197e18932c3da9b969ce7f545b741913597d600"},"provenance":null,"requires-python":null,"size":322153,"upload-time":"2014-02-06T16:34:24.006415Z","url":"https://files.pythonhosted.org/packages/5f/fd/fc06aaf69ad438d1354312c728791a1f571b2c59a49b411128911abf647e/psutil-0.6.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.0.win-amd64-py3.3.exe","hashes":{"sha256":"084a1cbeca8e10f0cbba4b9d3f7d3136cb1d03997a868eea334401f82105207a"},"provenance":null,"requires-python":null,"size":320646,"upload-time":"2014-02-06T16:34:33.535927Z","url":"https://files.pythonhosted.org/packages/70/5e/b1d4a1d5238497d48112bc27e3448d05a0acb6c8fcd537565724d916e7c2/psutil-0.6.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.0.win-amd64-py3.4.exe","hashes":{"sha256":"e7233e45ed7505efe46309978edd3e8a1cb4160400c3535512be2e4114615bb9"},"provenance":null,"requires-python":null,"size":321547,"upload-time":"2014-02-06T16:34:44.543150Z","url":"https://files.pythonhosted.org/packages/47/ad/4139c3eaa2ee1da71d071c9b49e8a04ff23f2c71f6fdecc11850b0e7343e/psutil-0.6.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.0.win32-py2.5.exe","hashes":{"sha256":"469628dea02a7d91837e05777cb66bfcdc2e1639cc8607ed93f19679c4e42cef"},"provenance":null,"requires-python":null,"size":160485,"upload-time":"2014-02-06T16:33:39.286987Z","url":"https://files.pythonhosted.org/packages/10/3a/a0136c299b74224be3b0735812a1544e6962d7c61af41adfcc400791c684/psutil-0.6.0.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.0.win32-py2.6.exe","hashes":{"sha256":"a628ad11fefc16d6dd144a8d29a03ac379debcf6c2e4b576e6ee2b2677c15836"},"provenance":null,"requires-python":null,"size":292261,"upload-time":"2014-02-06T16:33:47.696744Z","url":"https://files.pythonhosted.org/packages/2b/79/eec1c63d3ea968cc9754871a1b9d17d50bfc24136ecac415f701b1d240ea/psutil-0.6.0.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.0.win32-py2.7.exe","hashes":{"sha256":"71197523bf8acb49e3e909ea945cc6e77a21693428db2f169e33348988c1bb11"},"provenance":null,"requires-python":null,"size":291811,"upload-time":"2014-02-06T16:33:56.338977Z","url":"https://files.pythonhosted.org/packages/e9/64/fe90d0f1ba2dff8ad2511423720aed3569cfe7b5dbb95ede64af25b77e84/psutil-0.6.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.0.win32-py3.3.exe","hashes":{"sha256":"1d7df98f4532e76685fabbd82e9502c91be9a60e4dfdfd5a19c2152936b7d198"},"provenance":null,"requires-python":null,"size":286963,"upload-time":"2014-02-06T16:34:04.390605Z","url":"https://files.pythonhosted.org/packages/d6/d7/91226f8635d3850917b64e94ca4903910a60739ee7c930ef9678d93638eb/psutil-0.6.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.0.win32-py3.4.exe","hashes":{"sha256":"da568edc3b8ec4be5b8cee54851289eb4c24cbaa8ea165858e9bfc279269bf22"},"provenance":null,"requires-python":null,"size":286905,"upload-time":"2014-02-06T16:34:14.322283Z","url":"https://files.pythonhosted.org/packages/56/67/533833832596e5f8c1fb51d3f94cd110d688579be7af46996cf96ad890e7/psutil-0.6.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.1.tar.gz","hashes":{"sha256":"52eba795281cdd1079f13ded6a851f6d029551ddf552eadc9a2ee3eb26fe994d"},"provenance":null,"requires-python":null,"size":131473,"upload-time":"2014-02-06T02:05:47.971617Z","url":"https://files.pythonhosted.org/packages/4f/e6/989cb0b2f7f0ebe3ab0e7144b78db17387810f1526e98498be96fb755fa9/psutil-0.6.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.1.win-amd64-py2.7.exe","hashes":{"sha256":"566bcec389a77279e10335eb5d0021eb968445720496b70b46d208dbbf8e49a1"},"provenance":null,"requires-python":null,"size":322415,"upload-time":"2014-02-06T16:32:41.593785Z","url":"https://files.pythonhosted.org/packages/9a/bb/cb19a41fa75a2205ac2d337be45183f879aa77e60bbb19387c872c81d451/psutil-0.6.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.1.win-amd64-py3.3.exe","hashes":{"sha256":"fde5e57f9b51057998cce5a030ad481f826832b26cc2a6490d60317c9024724c"},"provenance":null,"requires-python":null,"size":320911,"upload-time":"2014-02-06T16:32:50.930654Z","url":"https://files.pythonhosted.org/packages/7d/d1/9e75848da16a5d165e3918540418f3052e6409357bcce91a7342b22d674f/psutil-0.6.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.1.win-amd64-py3.4.exe","hashes":{"sha256":"402475d1a93306ac025070da9628a0ca0e0d253b0755bf13131318a39aaeea8e"},"provenance":null,"requires-python":null,"size":321810,"upload-time":"2014-02-06T16:33:01.995635Z","url":"https://files.pythonhosted.org/packages/95/82/1f52f8aaa37e07da54da4468c031f59717d3f40cd82b6faaf8f203d84bd7/psutil-0.6.1.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.1.win32-py2.5.exe","hashes":{"sha256":"3f5b8afde564c563cd6fa28ac14ed020d91d654ef1efd6c0e1672282fcde8e65"},"provenance":null,"requires-python":null,"size":160754,"upload-time":"2014-02-06T16:31:55.533884Z","url":"https://files.pythonhosted.org/packages/5b/3f/10449a8a3dfb809fbeae3d853453aafb36376d51f33c46e02b5f0723c620/psutil-0.6.1.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.1.win32-py2.6.exe","hashes":{"sha256":"1c1763ff214042e7be75fd6eb336322a47193342cddfc70b8547f086968e024c"},"provenance":null,"requires-python":null,"size":292530,"upload-time":"2014-02-06T16:32:04.859349Z","url":"https://files.pythonhosted.org/packages/3e/7a/5d53c99ec66ea86548d95acaf98b32f1cbb761cf2e1fc8ee62d0838f8e24/psutil-0.6.1.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.1.win32-py2.7.exe","hashes":{"sha256":"e411f2374994663ad35bc21afe4ff9003e3675e23b07a2a60b57c760da0c5f3f"},"provenance":null,"requires-python":null,"size":292082,"upload-time":"2014-02-06T16:32:13.267526Z","url":"https://files.pythonhosted.org/packages/4a/40/e43e412c55f66b57aee3a418f47b05e7bb92db83df93c0c93b00f11f1357/psutil-0.6.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.1.win32-py3.3.exe","hashes":{"sha256":"03ea1f851920d6ddc286c1940ebd167ed92c1cb7a870f1789510c21c6fa6b7bc"},"provenance":null,"requires-python":null,"size":287227,"upload-time":"2014-02-06T16:32:22.314264Z","url":"https://files.pythonhosted.org/packages/07/12/1ce6196592e01965a375b307a57f68e90488a49a56a647f17ba027382448/psutil-0.6.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.6.1.win32-py3.4.exe","hashes":{"sha256":"095107d63b176fed2f4a44de3a4d19e8aa6291de0c3ee5e398272c9b72bf2254"},"provenance":null,"requires-python":null,"size":287169,"upload-time":"2014-02-06T16:32:32.819904Z","url":"https://files.pythonhosted.org/packages/f4/88/d20f7eefa6b8cc59f9320c52bf561ba52baea2b9b36568b198bca0b3548d/psutil-0.6.1.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.0.tar.gz","hashes":{"sha256":"95089802017ee629b84332deeb367f7f775b42e827ee283d46b7e99d05120d71"},"provenance":null,"requires-python":null,"size":138681,"upload-time":"2014-02-06T02:05:40.485455Z","url":"https://files.pythonhosted.org/packages/0b/8c/aeb6acf5a4610f8d5bb29ade04081d8672c2294c3cefa1f0422c6398bc66/psutil-0.7.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.0.win-amd64-py2.7.exe","hashes":{"sha256":"add60c89c13443617c83116fab34a3c6e4f32ee8b102ff6ffb4cd20ef6f511c3"},"provenance":null,"requires-python":null,"size":325666,"upload-time":"2014-02-06T16:30:51.588803Z","url":"https://files.pythonhosted.org/packages/38/8e/f3023be1a2268b5ee3a20e05ec8997a95c987f6f2b7dadad837674d562e8/psutil-0.7.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.0.win-amd64-py3.3.exe","hashes":{"sha256":"520a43be7fcd21d63c02039cdd7ebfe959b9fa9a865ba9aab6b0194a85184250"},"provenance":null,"requires-python":null,"size":324143,"upload-time":"2014-02-06T16:31:01.226228Z","url":"https://files.pythonhosted.org/packages/7b/03/d6bff7ff8570f3521a1d7949f03741210f11ba5ba807789d37e185a0a780/psutil-0.7.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.0.win-amd64-py3.4.exe","hashes":{"sha256":"0e78c6b441e659d922c5fddd5d5a36a368a61192873a0600ba23171c14dc2a6d"},"provenance":null,"requires-python":null,"size":325001,"upload-time":"2014-02-06T16:31:12.826305Z","url":"https://files.pythonhosted.org/packages/ce/8a/95e7c8b343a92137d82e4635fcb3f5c69c29e2a84ce4d51f2d5d86020b1e/psutil-0.7.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.0.win32-py2.5.exe","hashes":{"sha256":"532b918d8d2df60c20c69c1f764b9b16373ea22c32595fe801e3cc9b1aa5adfb"},"provenance":null,"requires-python":null,"size":163763,"upload-time":"2014-02-06T16:30:02.815062Z","url":"https://files.pythonhosted.org/packages/9e/6f/a946eeb3c6b8848fcf6d21c52884e2639e7d4393b51f330d4555ec177e83/psutil-0.7.0.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.0.win32-py2.6.exe","hashes":{"sha256":"21e1d7880579ef7eea28547c737923eaeee53389d51fa971d9f70bf89e9759ee"},"provenance":null,"requires-python":null,"size":295425,"upload-time":"2014-02-06T16:30:11.405843Z","url":"https://files.pythonhosted.org/packages/14/e9/ac4865772c55a3c09c85d987ac2a0e679c31e897d536f389485019c29450/psutil-0.7.0.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.0.win32-py2.7.exe","hashes":{"sha256":"bc36a7350b89e2ada06691583f4c4c2f99988fe6fa710d4f8e4143bed9f9f8bf"},"provenance":null,"requires-python":null,"size":294933,"upload-time":"2014-02-06T16:30:20.866887Z","url":"https://files.pythonhosted.org/packages/12/1c/053e0d5485a39c055e92b845b6311ab54887a091836a3985369cfffa9e53/psutil-0.7.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.0.win32-py3.3.exe","hashes":{"sha256":"9ce0d7a8cedde996eaae1eda3db9b473d441bd6c797b36a166d7b7a70dd4650e"},"provenance":null,"requires-python":null,"size":290006,"upload-time":"2014-02-06T16:30:29.860768Z","url":"https://files.pythonhosted.org/packages/73/19/8009828f3cfcf529570822e52988b95ac1c25c90ba4fb96ae570b5c25e66/psutil-0.7.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.0.win32-py3.4.exe","hashes":{"sha256":"6fdeca414a470d156ebdf1b6e4e9ca0fd9eb1b53e8d9a0cbb7d4a69d31f68bdc"},"provenance":null,"requires-python":null,"size":289986,"upload-time":"2014-02-06T16:30:39.817575Z","url":"https://files.pythonhosted.org/packages/42/7d/8e159acbf98eed98b778c244287b295f06411ac9e1903d2d358ba48b1d36/psutil-0.7.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.1.tar.gz","hashes":{"sha256":"5236f649318a06dcff8b86947c888d4510abce1783923aa5455b2d62df7204c7"},"provenance":null,"requires-python":null,"size":138525,"upload-time":"2014-02-06T02:05:32.811951Z","url":"https://files.pythonhosted.org/packages/8c/75/1eeb93df943b70c2e18bf412b32d18af3577da831f2bfe8c7d29f8853a67/psutil-0.7.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.1.win-amd64-py2.7.exe","hashes":{"sha256":"47fc6887ec86bd7a422326dfc77e2b43fcbc71e498194f86088bb2268c2372b7"},"provenance":null,"requires-python":null,"size":325663,"upload-time":"2014-02-06T16:29:31.935519Z","url":"https://files.pythonhosted.org/packages/2d/ab/8fde8a7358a21bc2488a5d21a24879ce3cc162d2a99c56f9599c7d759c75/psutil-0.7.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.1.win-amd64-py3.3.exe","hashes":{"sha256":"c00ec6a0e061e31687851c06d542500fd47b0ed415818e0da61c294f2796572b"},"provenance":null,"requires-python":null,"size":324140,"upload-time":"2014-02-06T16:29:41.403650Z","url":"https://files.pythonhosted.org/packages/35/b1/8506f2e78974da833b20b162d70b7e38ac3ab722adf18c03a3e428c9d9c4/psutil-0.7.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.1.win-amd64-py3.4.exe","hashes":{"sha256":"a5238a16cd85d4f3240dfca8ad94cb0bfe8643acb5c21d839a53caa86d8a23ab"},"provenance":null,"requires-python":null,"size":324998,"upload-time":"2014-02-06T16:29:50.929853Z","url":"https://files.pythonhosted.org/packages/cc/29/d00e1011e6db097ffac31c2876fdba51fd1fc251ee472ad6219095739a86/psutil-0.7.1.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.1.win32-py2.5.exe","hashes":{"sha256":"facf21db78120afb79ec51685043a5f84e897a84a7678db4381c0e377a481364"},"provenance":null,"requires-python":null,"size":163764,"upload-time":"2014-02-06T16:28:44.337888Z","url":"https://files.pythonhosted.org/packages/06/d8/a34c090687fa4c09bee5b5a06a762eb1af134d847b907d3e63c58e5d2cba/psutil-0.7.1.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.1.win32-py2.6.exe","hashes":{"sha256":"709d2e851c161f1ffad6bd7ed925ee90229dbd2067fbc26126c11d473075d7b7"},"provenance":null,"requires-python":null,"size":295427,"upload-time":"2014-02-06T16:28:53.964081Z","url":"https://files.pythonhosted.org/packages/4d/8a/d51e550aa0928ab02e4f6b2531453b2e63c8877e30543e342c8a35c90d4d/psutil-0.7.1.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.1.win32-py2.7.exe","hashes":{"sha256":"ee218aa12ee3af563833cf0bf109b15382261a0760a1ce636bf9e26e630f833e"},"provenance":null,"requires-python":null,"size":294930,"upload-time":"2014-02-06T16:29:03.098211Z","url":"https://files.pythonhosted.org/packages/3c/4c/d46a8bc865e58b30dcb160772667abf42f854ec4910ee1100e79961035ce/psutil-0.7.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.1.win32-py3.3.exe","hashes":{"sha256":"fdf22a72f891a45b8a9ebca85b50a055a4ba7a8c05b98f93e1387b8ee88a4562"},"provenance":null,"requires-python":null,"size":290003,"upload-time":"2014-02-06T16:29:12.556491Z","url":"https://files.pythonhosted.org/packages/1a/cf/98fdaf08279cd754414ab6723bbb6bd55d890624b766d64b21c35379d865/psutil-0.7.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-0.7.1.win32-py3.4.exe","hashes":{"sha256":"f675a6346e5bd02469b043afff2db1b8e0b180e8a6d1ca94d44d23f26f60e68f"},"provenance":null,"requires-python":null,"size":289983,"upload-time":"2014-02-06T16:29:21.732021Z","url":"https://files.pythonhosted.org/packages/f4/f9/79ac18809795f53197fb5b6bd452d2b52f62e5bff6a66d9b813077a85eb9/psutil-0.7.1.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.0.tar.gz","hashes":{"sha256":"af776ebeaf2420709b1ab741f664c048dcfc2890e3e7d151ae868fbdc3d47920"},"provenance":null,"requires-python":null,"size":156516,"upload-time":"2014-02-06T02:05:25.754044Z","url":"https://files.pythonhosted.org/packages/ed/7e/dd4062bfbe9b735793a5e63a38d83ba37292c4ada3615fe54403c54e3260/psutil-1.0.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.0.win-amd64-py2.7.exe","hashes":{"sha256":"7868ef2ecc1d847620d99b40f8105b6d5303784abbde34da2388b2c18e73a3d2"},"provenance":null,"requires-python":null,"size":327529,"upload-time":"2014-02-06T16:25:06.308578Z","url":"https://files.pythonhosted.org/packages/11/46/8a6581c14d644f747ea66a12c60ffebebd4f8a77af62192aeb20cc16ae17/psutil-1.0.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.0.win-amd64-py3.3.exe","hashes":{"sha256":"cdced3409a241b9d357b82c72d00ece49602cdc3f694ed988a952761367df560"},"provenance":null,"requires-python":null,"size":326023,"upload-time":"2014-02-06T16:25:15.281576Z","url":"https://files.pythonhosted.org/packages/03/a4/aa6592e676f75705b77df8e319f6aa35cf08a662fb0cfaa34434d273d7c9/psutil-1.0.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.0.win-amd64-py3.4.exe","hashes":{"sha256":"ef6974ac18407fb15430cd7b81aecc2f3666e75ca83fe3a66848c4c5c086e0a3"},"provenance":null,"requires-python":null,"size":326915,"upload-time":"2014-02-06T16:25:27.390929Z","url":"https://files.pythonhosted.org/packages/5b/c1/003f0071d1e2a4dfe12ac2d66ac14f5a5a93049544c11ac9c78d83d85318/psutil-1.0.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.0.win32-py2.5.exe","hashes":{"sha256":"d8213feaa4c560c61514cea4e6a7eec83f6df83694bb97b06d009e919f60c48f"},"provenance":null,"requires-python":null,"size":165576,"upload-time":"2014-02-06T16:24:22.454259Z","url":"https://files.pythonhosted.org/packages/d6/28/5753ba13863c26d8a8fef685690b539a4652f1c4929d13d482eb4d691d99/psutil-1.0.0.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.0.win32-py2.6.exe","hashes":{"sha256":"7571d74d351bba723b28666a13da43c07bb3568543be3f9f7404af2b8c1ecf3a"},"provenance":null,"requires-python":null,"size":297205,"upload-time":"2014-02-06T16:24:31.483662Z","url":"https://files.pythonhosted.org/packages/ad/8c/37599876ea078c7c2db10d746f0e0aa805abcacbab96d1cf5dfb0465c40b/psutil-1.0.0.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.0.win32-py2.7.exe","hashes":{"sha256":"3f311d4f060b83d7cd0068dbad867ad17e699d3c335e494dc31db5380154fdf4"},"provenance":null,"requires-python":null,"size":296787,"upload-time":"2014-02-06T16:24:39.921427Z","url":"https://files.pythonhosted.org/packages/99/4e/8c1f5db3e90dbdbde891c34fd2496805a93aa279433bac716098efad1b4c/psutil-1.0.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.0.win32-py3.3.exe","hashes":{"sha256":"acf6cbc2408869b2843d406a802b6895969a358c762b5ab3e6f5afdd0c918faf"},"provenance":null,"requires-python":null,"size":291856,"upload-time":"2014-02-06T16:24:48.963293Z","url":"https://files.pythonhosted.org/packages/5b/f7/0bad3fc8ff5f226fa933a847b5fb3355562421f0456348747cf18f7137f5/psutil-1.0.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.0.win32-py3.4.exe","hashes":{"sha256":"191c2e6953426c861d7e100d48a5b522307ffc3de7d9e7bc8509aa45fb3de60c"},"provenance":null,"requires-python":null,"size":291810,"upload-time":"2014-02-06T16:24:57.589702Z","url":"https://files.pythonhosted.org/packages/0a/50/18ebd254b6662c98a8310983fff6795c7ac303d25188fd727888ca1c5211/psutil-1.0.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.1.tar.gz","hashes":{"sha256":"ba4c81622434836f6645e8d04e221ca5b22a9bd508c29989407f116b917be5b3"},"provenance":null,"requires-python":null,"size":156516,"upload-time":"2014-02-06T02:05:16.665325Z","url":"https://files.pythonhosted.org/packages/94/50/7c9e94cf6cdbf4e4e41d2e318094c2b0d58d3bb9196017fb6e4897adf277/psutil-1.0.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.1.win-amd64-py2.7.exe","hashes":{"sha256":"2d6705cf53a7474b992d0d5beac4697cdd37f2c24098d4e1b5bb35a83b70b27e"},"provenance":null,"requires-python":null,"size":327529,"upload-time":"2014-02-06T16:20:20.120751Z","url":"https://files.pythonhosted.org/packages/a9/28/ac3c9da11fbe1ae3ddefb2cb6b410c8ad701f39d86489931e87c871cc4d1/psutil-1.0.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.1.win-amd64-py3.3.exe","hashes":{"sha256":"6194670fcbddec715846fa9beca4450b4419423b2d466894da2fa1915fc7b712"},"provenance":null,"requires-python":null,"size":326023,"upload-time":"2014-02-06T16:20:43.401705Z","url":"https://files.pythonhosted.org/packages/97/30/094bb08295e15e05a266fcb15c116912f643fbb4dd8090ec64aca801ea72/psutil-1.0.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.1.win-amd64-py3.4.exe","hashes":{"sha256":"5cf4ddea8dd21b99b90656dafa2d28c32fa9c159ab30580782b65b3b615c62a7"},"provenance":null,"requires-python":null,"size":326915,"upload-time":"2014-02-06T16:20:52.673334Z","url":"https://files.pythonhosted.org/packages/e8/86/e72c22699491573508e85b226f1782eab840b713cd6b8d42655c5329324c/psutil-1.0.1.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.1.win32-py2.5.exe","hashes":{"sha256":"fadf7625c309bc5f6e6d3b9d42836491efabbfda1834db7166775d705db420bf"},"provenance":null,"requires-python":null,"size":165580,"upload-time":"2014-02-06T16:04:23.085098Z","url":"https://files.pythonhosted.org/packages/a5/40/d307abd2ee3015e38cbeeac18c6b0f091cdcac496688f4d2c286145fdc5f/psutil-1.0.1.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.1.win32-py2.6.exe","hashes":{"sha256":"84a257b4adea431473089ea4b798bea98ce64452587ba7b71856e150d88e9213"},"provenance":null,"requires-python":null,"size":297208,"upload-time":"2014-02-06T15:59:31.669293Z","url":"https://files.pythonhosted.org/packages/75/3f/c10ca801aff50ce6ea84f71ce69a32de1b4a1f0439ada94adeccd36e1afd/psutil-1.0.1.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.1.win32-py2.7.exe","hashes":{"sha256":"4b5047bb4055d48f718bbb5a457c9b1c1c06776ca6353cb310b471c38d51324d"},"provenance":null,"requires-python":null,"size":296787,"upload-time":"2014-02-06T15:59:41.621679Z","url":"https://files.pythonhosted.org/packages/15/6c/944089b8dd730314a8eec9faa4d7d383f31490130e9e0534c882173e2eb6/psutil-1.0.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.1.win32-py3.3.exe","hashes":{"sha256":"3553a27b4d0e7552d0d863c017e152a88cb689d2090b39e2bca68db7fc27e8b2"},"provenance":null,"requires-python":null,"size":291856,"upload-time":"2014-02-06T15:59:51.241604Z","url":"https://files.pythonhosted.org/packages/4c/51/eb8b12a0124050492faad7fe3e695cb44e4ec5495a64a8570abf6e7cddc3/psutil-1.0.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.0.1.win32-py3.4.exe","hashes":{"sha256":"d3ab1b882647d66099adefa8ff780c4144bf8df1bb97bc31c55cbd999807cae8"},"provenance":null,"requires-python":null,"size":291810,"upload-time":"2014-02-06T16:00:00.814707Z","url":"https://files.pythonhosted.org/packages/4d/2a/33063e4a674e3be38d77fbc6e5e988f0e0323cd000cb7eb60f9300493631/psutil-1.0.1.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.0.tar.gz","hashes":{"sha256":"31b4b411d3f6960d26dac1f075709ff6d36f60216bf1f2b47fc4dc62a2d0aa6f"},"provenance":null,"requires-python":null,"size":163785,"upload-time":"2013-09-28T09:48:01.746815Z","url":"https://files.pythonhosted.org/packages/f6/71/1f9049fc7936f7f48f0553abd8bf8394f4ad2d9fb63a881b3a653b67abd6/psutil-1.1.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.0.win-amd64-py2.7.exe","hashes":{"sha256":"ef250fbe783c0b31d9574d06324abd78ebf23c75981a8d0f9319d080435aa056"},"provenance":null,"requires-python":null,"size":305459,"upload-time":"2013-09-28T09:55:10.368138Z","url":"https://files.pythonhosted.org/packages/28/ff/e0dcc8e0194817191817ff02926a734a51d24a91e1cfc5c47be8c1e8508c/psutil-1.1.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.0.win-amd64-py3.2.exe","hashes":{"sha256":"a95bf68f2189b7cd3f22558b12a989f368335c905cbd3045ca315514538b3115"},"provenance":null,"requires-python":null,"size":306085,"upload-time":"2013-09-28T09:55:36.198407Z","url":"https://files.pythonhosted.org/packages/a5/be/6f3b8f8ff89eaf480f59bbcd3d18c10ee0206891aa5fb4a25280c031801c/psutil-1.1.0.win-amd64-py3.2.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.0.win32-py2.4.exe","hashes":{"sha256":"bc53be7f65ccb8a530e0c496a8db3ecd78b57b1faa443e43e228df327cf26899"},"provenance":null,"requires-python":null,"size":142269,"upload-time":"2013-09-28T09:51:43.685077Z","url":"https://files.pythonhosted.org/packages/ec/60/0a9a96611c77866a2c4140159a57c2702523ebc5371eaa24e0353d88ac2c/psutil-1.1.0.win32-py2.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.0.win32-py2.5.exe","hashes":{"sha256":"4a587008d6aae91fd1d3fee21141a27b4a2978fff75c80e57a37678a63a1cfcc"},"provenance":null,"requires-python":null,"size":142249,"upload-time":"2013-09-28T09:52:12.373832Z","url":"https://files.pythonhosted.org/packages/b5/16/dd5a768773cb017c4a66d69ec49abdd33df831215418e4c116639ed41afb/psutil-1.1.0.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.0.win32-py2.6.exe","hashes":{"sha256":"be853a268f0f3c3336c9bb648253db85ef2024213b6fc199f84b89643b74b130"},"provenance":null,"requires-python":null,"size":276570,"upload-time":"2013-09-28T09:53:51.099422Z","url":"https://files.pythonhosted.org/packages/cc/e0/adb41ece0bb2c27d75a41d98bd8f9802475d5d488df7d4905229596c0a70/psutil-1.1.0.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.0.win32-py3.2.exe","hashes":{"sha256":"dfdce701cfc7799dc1964debb486907b447fbdb361e06be786a9a5e11a1d5309"},"provenance":null,"requires-python":null,"size":275628,"upload-time":"2013-09-28T09:54:16.430842Z","url":"https://files.pythonhosted.org/packages/8f/c9/231df338b0cef518dbde2f98c1e0bb81142dc7dbec4a39267bb60dde1462/psutil-1.1.0.win32-py3.2.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.0.win32-py3.3.exe","hashes":{"sha256":"b4106508ea96ad24e1102de2a31a99c557ee608abab15d978bfe8d0f626e518c"},"provenance":null,"requires-python":null,"size":270259,"upload-time":"2013-09-28T09:54:43.666687Z","url":"https://files.pythonhosted.org/packages/a7/03/80f426e186dfdb202128c78b421b3ba34de461c32f6ed2c04041fc548d7b/psutil-1.1.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.1.tar.gz","hashes":{"sha256":"a5201e4c2a9b57e9e5d8de92b3a4006d093eedd9b56915b8279f365aaedd0f48"},"provenance":null,"requires-python":null,"size":165467,"upload-time":"2013-10-07T22:38:06.357263Z","url":"https://files.pythonhosted.org/packages/8d/0d/1a4bfc94f9cc783a510b3fc7efd5a2ef39858b3a2b6ab40a094a1ca8a54d/psutil-1.1.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.1.win-amd64-py2.7.exe","hashes":{"sha256":"b8ccaef0a96d7ef40eda7493ffcaa3c5e2d63f531e090c3567e4198b38ca8e33"},"provenance":null,"requires-python":null,"size":305937,"upload-time":"2013-10-07T22:43:47.570605Z","url":"https://files.pythonhosted.org/packages/5a/b8/66489689c8751acd67bdbc8f37534176a14b63f74e74a229c30b230c8a18/psutil-1.1.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.1.win-amd64-py3.2.exe","hashes":{"sha256":"fdabf11b316b0c406709aabb76f6c3cfdeb448c10ed38193786110f13e5c1248"},"provenance":null,"requires-python":null,"size":306563,"upload-time":"2013-10-07T22:44:09.209012Z","url":"https://files.pythonhosted.org/packages/e3/2d/cd5c620f9e5cb090eea573b2dc16574e746529347b6133b2f0b6e686d917/psutil-1.1.1.win-amd64-py3.2.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.1.win32-py2.4.exe","hashes":{"sha256":"74bb1ccf0b28a914bb9d303642c08450ea6c5876851eb71e17f24b3d8672ca7d"},"provenance":null,"requires-python":null,"size":142751,"upload-time":"2013-10-07T22:45:21.036212Z","url":"https://files.pythonhosted.org/packages/28/9a/b83f884add09296894a1223c9c404cb57155c1c4317d318abf8c170e07b5/psutil-1.1.1.win32-py2.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.1.win32-py2.5.exe","hashes":{"sha256":"8e52d5b6af64e3eefbda30e0b16c9f29434ef6f79ea10b6fbd4520a6fbdb2481"},"provenance":null,"requires-python":null,"size":142731,"upload-time":"2013-10-07T22:41:32.428090Z","url":"https://files.pythonhosted.org/packages/8b/80/c41382a4f650f47a37300411169b7b248acf0b0925eb92cb22286362c3df/psutil-1.1.1.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.1.win32-py2.6.exe","hashes":{"sha256":"1e3a17b2a2f2bc138774f9b4b5ff52767f65ffb6349f9e05eb24f243c550ce1b"},"provenance":null,"requires-python":null,"size":277048,"upload-time":"2013-10-07T22:41:54.149373Z","url":"https://files.pythonhosted.org/packages/16/cd/25a3b9af88d130dd1084acab467b30996884219afc0a1e989d2a015ea54b/psutil-1.1.1.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.1.win32-py2.7.exe","hashes":{"sha256":"2e7691ddbde94b1ec7bb5b4cd986b2f763efc6be5b122dd499e156c9802a195b"},"provenance":null,"requires-python":null,"size":276822,"upload-time":"2013-10-07T22:42:18.407145Z","url":"https://files.pythonhosted.org/packages/42/78/eeacb1210abbe15cf06b9810e84afeabae1f9362abe389e8d5ca2c19df43/psutil-1.1.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.1.win32-py3.2.exe","hashes":{"sha256":"01276bd695b5cebc7bea7c97814713b0f3030861e30be88a60d40c9daf3d529c"},"provenance":null,"requires-python":null,"size":276106,"upload-time":"2013-10-07T22:42:41.551135Z","url":"https://files.pythonhosted.org/packages/92/14/90b9a4690f04ef1aab89a97a7b5407708f56785ccc264d9f9ce372feaea4/psutil-1.1.1.win32-py3.2.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.1.win32-py3.3.exe","hashes":{"sha256":"86ce0cfa7a95c7c1bc3e2f5358e446b01fc2be5f0c879c8def21e7ede9dc77de"},"provenance":null,"requires-python":null,"size":270733,"upload-time":"2013-10-07T22:43:07.069809Z","url":"https://files.pythonhosted.org/packages/a7/64/ba4601de7df6130c27f42bcec9f11da4ea905eda26d2f5a41efdb481f377/psutil-1.1.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.2.tar.gz","hashes":{"sha256":"adeb1afcb46327bed6603aa8981dce863f052043a52f003e2742ec7c3739677a"},"provenance":null,"requires-python":null,"size":165709,"upload-time":"2013-10-22T18:13:09.038583Z","url":"https://files.pythonhosted.org/packages/e4/b1/34a4bd75027d08c8db4f6301d6562e333c8d9131dca08b7f76f05aeae00a/psutil-1.1.2.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.2.win-amd64-py2.7.exe","hashes":{"sha256":"4c9e952a7faf50f11fb8bcd6c12c952b063664d9327957c9f6abd498e6ef3bc8"},"provenance":null,"requires-python":null,"size":306157,"upload-time":"2013-10-22T18:16:38.181306Z","url":"https://files.pythonhosted.org/packages/f9/df/437db01296118d668cf654f097ad2b1c341291ba5dc4b5eb80f0a0a40c52/psutil-1.1.2.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.2.win-amd64-py3.2.exe","hashes":{"sha256":"6e8ed376b63b15b09a94eff7998dae1f8506bd174d795e509cc735f875fddabe"},"provenance":null,"requires-python":null,"size":306782,"upload-time":"2013-10-22T18:17:05.663884Z","url":"https://files.pythonhosted.org/packages/c9/d0/1e413f0258d02bf77bc1d94002d041b8853584369e4af039cd5cf89e3270/psutil-1.1.2.win-amd64-py3.2.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.2.win32-py2.4.exe","hashes":{"sha256":"ecc5ab4537259db7254c6f6dc7b7cb5a7f398c322a01612c272a8222696334a8"},"provenance":null,"requires-python":null,"size":142969,"upload-time":"2013-10-22T18:13:39.868851Z","url":"https://files.pythonhosted.org/packages/a7/63/fd5770ec4fe87d30bd836989d314b85662c775a52dbd017747fc69fe8f0e/psutil-1.1.2.win32-py2.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.2.win32-py2.5.exe","hashes":{"sha256":"e05d83d6b53ea24333c8b04d329e28ff11ecad75945f371ff5ce7f785df36aee"},"provenance":null,"requires-python":null,"size":142950,"upload-time":"2013-10-22T18:14:00.511107Z","url":"https://files.pythonhosted.org/packages/8d/e1/22650079452725e44ec790c3e75282f4d341f359b213b2afc7f2ada46930/psutil-1.1.2.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.2.win32-py2.6.exe","hashes":{"sha256":"f06ee6e1508a12afcfed04a4022ded9f872e2a964a62bd86617ece943d89ab01"},"provenance":null,"requires-python":null,"size":277266,"upload-time":"2013-10-22T18:14:33.903116Z","url":"https://files.pythonhosted.org/packages/3e/a4/5177488368f230acd4708a117c0820fb16843521e2a7a492078a2335bb9f/psutil-1.1.2.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.2.win32-py2.7.exe","hashes":{"sha256":"6c5be5538202ed7419911178ded41c65e118104fa634109f528a6d2d3e50a7d0"},"provenance":null,"requires-python":null,"size":277039,"upload-time":"2013-10-22T18:14:56.072979Z","url":"https://files.pythonhosted.org/packages/5e/d6/56f2891f6dd56f950866cc39892e5a56e85331d97c39e2634cfc4014f0df/psutil-1.1.2.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.2.win32-py3.2.exe","hashes":{"sha256":"5cfa97b52fb48dbb8255c120b567be4f06802afa7d2fe71b8fe7c7c4ee53ad88"},"provenance":null,"requires-python":null,"size":276327,"upload-time":"2013-10-22T18:15:25.393310Z","url":"https://files.pythonhosted.org/packages/dd/be/1aea1e7a1a3fb44f4c8d887a1d55e960de283d86875f15457a284268e197/psutil-1.1.2.win32-py3.2.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.2.win32-py3.3.exe","hashes":{"sha256":"f7dc7507fb9d4edb42709b356eb2e4b3da356efa54d83900e4cef59f3adebfbf"},"provenance":null,"requires-python":null,"size":270952,"upload-time":"2013-10-22T18:15:51.874969Z","url":"https://files.pythonhosted.org/packages/f5/7c/1a33b78a66a96e740e197ae55719496ba57bb9cee32f710a5a6affa68cc8/psutil-1.1.2.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.3.tar.gz","hashes":{"sha256":"5e1164086a7ed3b863ebd12315d35086e22252b328401fce901a0862050ef98c"},"provenance":null,"requires-python":null,"size":165550,"upload-time":"2013-11-07T20:47:17.039714Z","url":"https://files.pythonhosted.org/packages/93/fa/1f70b7fcdff77348f4e79d84cc9b568596874ca34940ede78c63d503a095/psutil-1.1.3.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.3.win-amd64-py2.7.exe","hashes":{"sha256":"43cbf089cbe160d193e1658fe4eec4a719430432e866334b25dc3acef73c3e61"},"provenance":null,"requires-python":null,"size":306145,"upload-time":"2013-11-07T21:05:00.986042Z","url":"https://files.pythonhosted.org/packages/03/ec/f05db404504d67a19397e17e64f0276cc610a9dc450eb51ed70436f37c43/psutil-1.1.3.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.3.win32-py2.4.exe","hashes":{"sha256":"ef7445e1b0449af34ff51a9e1d3f030b8794c9fcd2b99d072b2e815d00e8a783"},"provenance":null,"requires-python":null,"size":142958,"upload-time":"2013-11-07T21:16:11.539460Z","url":"https://files.pythonhosted.org/packages/37/c2/5d40dd0a36f0280c1dea0f651cfde79b5e99a0e6fab92273fa3ac41055f0/psutil-1.1.3.win32-py2.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.3.win32-py2.5.exe","hashes":{"sha256":"b7f4103b1058d2ee597f88902ff50f19421b95c18ac9b82ea9e8a00091786131"},"provenance":null,"requires-python":null,"size":142940,"upload-time":"2013-11-07T21:15:50.256574Z","url":"https://files.pythonhosted.org/packages/ee/cb/7d42052f4057c6233ba3f4b7afade92117f58c5d7544ee6ab16e82c515c7/psutil-1.1.3.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.3.win32-py2.6.exe","hashes":{"sha256":"e21b9c5a26ad3328b6f5b629e30944a9b7e56ea6ac0aa051455c8a352e67fbca"},"provenance":null,"requires-python":null,"size":277254,"upload-time":"2013-11-07T20:58:49.794773Z","url":"https://files.pythonhosted.org/packages/b2/7e/c42d752b333c5846d88a8b56bbab23325b247766c100dc6f68c6bc56019d/psutil-1.1.3.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.3.win32-py2.7.exe","hashes":{"sha256":"fc9732a4dea2a3f73f9320472aa54e1cfae45d324bea24bd207d88d3a0a281b0"},"provenance":null,"requires-python":null,"size":277028,"upload-time":"2013-11-07T21:16:56.712839Z","url":"https://files.pythonhosted.org/packages/2b/76/eec917b6f74ea9bd20a55dd8b4b01e69689278235335dbedc2b9be212815/psutil-1.1.3.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.3.win32-py3.2.exe","hashes":{"sha256":"a141d59c03a28bdd32dfce11a5d51b9a63a8e6c6c4245e416eeffcc1c43f1de9"},"provenance":null,"requires-python":null,"size":276315,"upload-time":"2013-11-07T21:02:04.504719Z","url":"https://files.pythonhosted.org/packages/3a/b9/748bbd0c53c74c682051137830048abd0126ae50e3bf4d5854a0188da143/psutil-1.1.3.win32-py3.2.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.1.3.win32-py3.3.exe","hashes":{"sha256":"36e855ea7c872beb04f22d968c55f1ef897df0261ea993ecf5ae6e8216939a55"},"provenance":null,"requires-python":null,"size":270942,"upload-time":"2013-11-07T21:04:24.025560Z","url":"https://files.pythonhosted.org/packages/e6/7b/655e08abdf19d06c8a3fd6fba35932867c80fcff05c97cf909b5d364603c/psutil-1.1.3.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.2.0.tar.gz","hashes":{"sha256":"c3a2b02e92363837499680760b674b3bf3bd03dd9528a5dc41392625a61b162a"},"provenance":null,"requires-python":null,"size":166747,"upload-time":"2013-11-20T20:02:56.854747Z","url":"https://files.pythonhosted.org/packages/77/89/8ee72ae2b5e3c749e43a9c1e95d61eceff022ab7c929cdde8a3b7539a707/psutil-1.2.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.2.0.win-amd64-py2.7.exe","hashes":{"sha256":"2bedcda55618f71bc98d6669bd190e2b51b940e3a3af2512dd77745cb1ebc101"},"provenance":null,"requires-python":null,"size":307021,"upload-time":"2013-11-20T20:08:29.733072Z","url":"https://files.pythonhosted.org/packages/9c/d1/598bcaa9701e06561cb8823a7a04ee44e702b0e327ef0a65bdd97b019613/psutil-1.2.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.2.0.win32-py2.4.exe","hashes":{"sha256":"417695f66562a22ac4f92bf7df1e9dda6264579eb308badd2c3e85df88ab9436"},"provenance":null,"requires-python":null,"size":143831,"upload-time":"2013-11-20T20:18:44.725428Z","url":"https://files.pythonhosted.org/packages/a4/ba/4b54baace7b49b73df74b54815337878137d25fda506e518f2d8dd2472fc/psutil-1.2.0.win32-py2.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.2.0.win32-py2.5.exe","hashes":{"sha256":"40747c59cf92cef26d595a9cefaac9c1a4bc0290abff6004bb092ee2ddec7d7b"},"provenance":null,"requires-python":null,"size":143812,"upload-time":"2013-11-20T20:18:35.170091Z","url":"https://files.pythonhosted.org/packages/88/17/3ba1b45ec63d666c2631f16679adb819564feb775cfd9e774d23e5774a44/psutil-1.2.0.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.2.0.win32-py2.6.exe","hashes":{"sha256":"fb0c3942a48fde9c52828dae4074381556eff8eb5651bdc9c4c107b40dc8318e"},"provenance":null,"requires-python":null,"size":278130,"upload-time":"2013-11-20T20:07:34.211552Z","url":"https://files.pythonhosted.org/packages/ae/4a/e62000dbe462270c30f1e2f2dcc913596f70e264d9a656f881bb9f487283/psutil-1.2.0.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.2.0.win32-py2.7.exe","hashes":{"sha256":"1024eff34ff14f2899186549ca8f8e461403e5700910e52eb013ffffb078cda3"},"provenance":null,"requires-python":null,"size":277905,"upload-time":"2013-11-20T20:18:07.310116Z","url":"https://files.pythonhosted.org/packages/9d/1a/b8e679a7e47229d07e41439a43fc1be48c0d34774e5e35ad6730398485bd/psutil-1.2.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.2.0.win32-py3.2.exe","hashes":{"sha256":"2b588673530d67f45287c31b6dfef02ee4e753aee7e7216448e6dbcb95b482e1"},"provenance":null,"requires-python":null,"size":277189,"upload-time":"2013-11-20T20:05:01.902941Z","url":"https://files.pythonhosted.org/packages/05/10/20a2364e4e69206d66da106288b65bf5bbde0d648aad5bb829f3eb08fabb/psutil-1.2.0.win32-py3.2.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.2.0.win32-py3.3.exe","hashes":{"sha256":"3e23d3b1ef5f20a458092887d1df32f55eaf4fc86f2ed49cc68cf164537128d6"},"provenance":null,"requires-python":null,"size":271857,"upload-time":"2013-11-20T20:05:28.007235Z","url":"https://files.pythonhosted.org/packages/24/39/4630dff3d0fa4896db905a118877fce5b72ced6629ca9958e207f7a0b198/psutil-1.2.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.2.1.tar.gz","hashes":{"sha256":"508e4a44c8253a386a0f86d9c9bd4a1b4cbb2f94e88d49a19c1513653ca66c45"},"provenance":null,"requires-python":null,"size":167397,"upload-time":"2013-11-25T20:06:20.566209Z","url":"https://files.pythonhosted.org/packages/8a/45/3b9dbd7a58482018927f756de098388ee252dd230143ddf486b3017117b1/psutil-1.2.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.2.1.win-amd64-py2.7.exe","hashes":{"sha256":"f3af7b44925554531dff038e0401976a6b92b089ecca51d50be903420d7a262d"},"provenance":null,"requires-python":null,"size":307074,"upload-time":"2013-11-25T20:09:25.387664Z","url":"https://files.pythonhosted.org/packages/40/47/665755b95ad75e6223af96f2d7c04667f663a53dede0315df9832c38b60d/psutil-1.2.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.2.1.win-amd64-py3.3.exe","hashes":{"sha256":"26fd84571ac026861d806a9f64e3bbfd38d619a8195c1edfd31c0a9ee2295b03"},"provenance":null,"requires-python":null,"size":329716,"upload-time":"2014-02-06T00:57:26.931878Z","url":"https://files.pythonhosted.org/packages/d8/65/2e9941492b3d001a87d87b5e5827b1f3cec42e30b7110fa82d24be8c4526/psutil-1.2.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.2.1.win32-py2.4.exe","hashes":{"sha256":"84a4e7e2de1ca6f45359cfd7fd60c807f494385a6d97804bd58759a94b9c5e2d"},"provenance":null,"requires-python":null,"size":143888,"upload-time":"2013-11-25T20:07:25.331363Z","url":"https://files.pythonhosted.org/packages/5c/99/d9147b76eea8c185b6cefbb46de73ae880e5ef0ff36d93cb3f6084e50d59/psutil-1.2.1.win32-py2.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.2.1.win32-py2.5.exe","hashes":{"sha256":"aad7d81607b3ad740fee47b27a9f1434a05fe35dc68abefb1f961f74bae3c3f9"},"provenance":null,"requires-python":null,"size":143868,"upload-time":"2013-11-25T20:07:40.595942Z","url":"https://files.pythonhosted.org/packages/bd/a0/5087d4a5145326a5a07a53ed4f9cd5c09bf5dad4f8d7b9850e6aaa13caa2/psutil-1.2.1.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.2.1.win32-py2.6.exe","hashes":{"sha256":"3ad3a40afd859cf0217a2d643c74be3a12ed2f54ebd4a91c40fa7b13084573c6"},"provenance":null,"requires-python":null,"size":278185,"upload-time":"2013-11-25T20:07:59.926444Z","url":"https://files.pythonhosted.org/packages/c6/e0/67810b602a598488d1f2982451655427effe7c7062184fe036c2b5bc928f/psutil-1.2.1.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.2.1.win32-py2.7.exe","hashes":{"sha256":"02fb79b9e5336ff179c44ce2017308cf46316e19bea70abb8855afd808db2a0f"},"provenance":null,"requires-python":null,"size":277955,"upload-time":"2013-11-25T20:08:20.641779Z","url":"https://files.pythonhosted.org/packages/5b/7f/9334b57597acabaaf2261c93bb9b1f9f02cdfef5c1b1aa808b262f770adb/psutil-1.2.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.2.1.win32-py3.2.exe","hashes":{"sha256":"7e64d065f12e8f941f2dbb2f3df0887b2677fee7b2b4c50ed91e490e094c7273"},"provenance":null,"requires-python":null,"size":277243,"upload-time":"2013-11-25T20:08:40.628203Z","url":"https://files.pythonhosted.org/packages/ae/c5/2842c69c67ae171f219efa8bb11bddc2fcec2cea059721e716fe4d48b50c/psutil-1.2.1.win32-py3.2.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-1.2.1.win32-py3.3.exe","hashes":{"sha256":"1a1c8e6635949a698b6ade3d5d2a8368daff916d8122cf13286c79a52ec8d7a1"},"provenance":null,"requires-python":null,"size":271900,"upload-time":"2013-11-25T20:09:05.464640Z","url":"https://files.pythonhosted.org/packages/86/df/007ca575da6ee7cbb015dc00122028ee0c97fc6a0c9e8bc02333753bfd2f/psutil-1.2.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.0.0.tar.gz","hashes":{"sha256":"38af34b0f40a4f50988a7401b7111ae4468beb5bcce0fbae409504dd3d5f2e8d"},"provenance":null,"requires-python":null,"size":207168,"upload-time":"2014-03-10T11:23:38.510743Z","url":"https://files.pythonhosted.org/packages/9c/2c/d4380234ddc21ecfb03691a982f5f26b03061e165658ac455b61886fe3ff/psutil-2.0.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.0.0.win-amd64-py2.7.exe","hashes":{"sha256":"32f27f9be7c03f60f61dc27af2464cfb1edd64525d05b81478e80dc12913fe3b"},"provenance":null,"requires-python":null,"size":310569,"upload-time":"2014-03-10T11:16:14.555076Z","url":"https://files.pythonhosted.org/packages/c3/10/c0e1b505d7d2b4a7f3294c1b4b2bc2644a4629462d777fe2cdcd57b1debe/psutil-2.0.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.0.0.win-amd64-py3.3.exe","hashes":{"sha256":"219cf2e5832cf68798e522e815d79a7307dea1f6b1d9b2372704f6e7fea085f3"},"provenance":null,"requires-python":null,"size":309024,"upload-time":"2014-03-10T11:16:29.122028Z","url":"https://files.pythonhosted.org/packages/eb/db/ab023b5ce09f314ee58ee4b9e73e85172dd06272501570a34a1afe6115c2/psutil-2.0.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.0.0.win-amd64-py3.4.exe","hashes":{"sha256":"12b9c02e108e887a43e0ed44e3a8e9968e65236d6d0b79c45891471ea2b9e14d"},"provenance":null,"requires-python":null,"size":310130,"upload-time":"2014-03-10T11:16:49.562585Z","url":"https://files.pythonhosted.org/packages/67/94/0dded4aab9c4992bddb311d2ae8fd9a638df5f6039d12a4fe66481f3ea1c/psutil-2.0.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.0.0.win32-py2.4.exe","hashes":{"sha256":"54d8636623a4f676a9a38b0afe3dfad5f1b90f710f2cb6c55e1a0803813d76a5"},"provenance":null,"requires-python":null,"size":143691,"upload-time":"2014-03-10T11:20:46.107824Z","url":"https://files.pythonhosted.org/packages/76/a3/48c0984c0b65be53a9e3e090df0cd5f3e6bddd767c3f8e62cf286be240e1/psutil-2.0.0.win32-py2.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.0.0.win32-py2.5.exe","hashes":{"sha256":"f669fe7e7cab107fb738362366cc9e0ecda532269ac3a9815a28930b474edf0b"},"provenance":null,"requires-python":null,"size":147735,"upload-time":"2014-03-10T11:15:02.487295Z","url":"https://files.pythonhosted.org/packages/f8/ba/346cc719249b9a5281dab059cb8796aff6faf487142f50966fc08330ad79/psutil-2.0.0.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.0.0.win32-py2.6.exe","hashes":{"sha256":"33b580008e0c65073198472da81d0d0c50d2a30e3e82c269713b1e3bdf14c2c6"},"provenance":null,"requires-python":null,"size":280920,"upload-time":"2014-03-10T11:15:17.132174Z","url":"https://files.pythonhosted.org/packages/2f/6f/7326e900c5333d59aa96a574d13321c94a9357ab56b0dd489b8f24ebab78/psutil-2.0.0.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.0.0.win32-py2.7.exe","hashes":{"sha256":"4f9c78cdd57e1c83242096f8343617ae038efd1c23af3864c25992335eabed3f"},"provenance":null,"requires-python":null,"size":280693,"upload-time":"2014-03-10T11:15:31.704968Z","url":"https://files.pythonhosted.org/packages/18/dd/c81485b54894c35fd8b62822563293db7c4dd17a05ea4eade169cf383266/psutil-2.0.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.0.0.win32-py3.3.exe","hashes":{"sha256":"380f8ff680ce3c8fdd7b31a3fe42d697b15a0c8677559691ed90b755051a5acf"},"provenance":null,"requires-python":null,"size":275631,"upload-time":"2014-03-10T11:15:45.735326Z","url":"https://files.pythonhosted.org/packages/77/5a/e87efff3e46862421a9c87847e63ebecd3bb332031305b476399918fea4f/psutil-2.0.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.0.0.win32-py3.4.exe","hashes":{"sha256":"4239b2431c825db2ad98e404a96320f0c78fb1c7d5bdf52a49a00d36bedfa0df"},"provenance":null,"requires-python":null,"size":275638,"upload-time":"2014-03-10T11:16:00.263121Z","url":"https://files.pythonhosted.org/packages/c4/80/35eb7f189482d25e3669871e7dcd295ec38f792dc4670b8635d72b4f949a/psutil-2.0.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.0.tar.gz","hashes":{"sha256":"d1e3ce46736d86164b6d72070f2cbaf86dcf9db03066b7f36a7b302e334a8d01"},"provenance":null,"requires-python":null,"size":211640,"upload-time":"2014-04-08T14:59:37.598513Z","url":"https://files.pythonhosted.org/packages/6c/d1/69431c4fab9b5cecaf28f2f2e0abee21805c5783c47143db5f0f7d42dbec/psutil-2.1.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.0.win-amd64-py2.7.exe","hashes":{"sha256":"9dc35f899946d58baf5b805ebed2f723f75d93f6b6bce212f79581d40f7276db"},"provenance":null,"requires-python":null,"size":312139,"upload-time":"2014-04-08T15:13:59.193442Z","url":"https://files.pythonhosted.org/packages/7a/3c/ce6a447030cdb50cf68a3988337df0c42e52abf45c3adfea3b225760eb70/psutil-2.1.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.0.win-amd64-py3.3.exe","hashes":{"sha256":"46232b7b0eb48c6ec1f12d17b2cc15ec3cc70ed952f2c98edb50aaa405dafa5d"},"provenance":null,"requires-python":null,"size":310591,"upload-time":"2014-04-08T15:14:05.019441Z","url":"https://files.pythonhosted.org/packages/cd/60/7134a7f812ef1eba9373c86b95ce6254f5f58928baba04af161f0066629f/psutil-2.1.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.0.win-amd64-py3.4.exe","hashes":{"sha256":"fa770a2c9e384924df021aed9aa5a6f92db8062a7f700113ba3943e262028454"},"provenance":null,"requires-python":null,"size":311682,"upload-time":"2014-04-08T15:14:24.701683Z","url":"https://files.pythonhosted.org/packages/45/56/3ac0a63799d54cb9b1914214944aed71e49297fb90de92b8d1fe20de3bd8/psutil-2.1.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.0.win32-py2.4.exe","hashes":{"sha256":"15d0d10bbd60e461690188cdd209d6688a1112648589bf291b17cc84e99cb6e7"},"provenance":null,"requires-python":null,"size":145275,"upload-time":"2014-04-08T15:12:04.809516Z","url":"https://files.pythonhosted.org/packages/09/e8/f6e4209b3d6373ea11fa340c2e6e4640a7ee53ef4697c49d610ffdf86674/psutil-2.1.0.win32-py2.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.0.win32-py2.5.exe","hashes":{"sha256":"381ac1027a270c04cf6decdc011a28a4270105d009f213d62daec4b3116b92b6"},"provenance":null,"requires-python":null,"size":149378,"upload-time":"2014-04-08T15:12:25.638187Z","url":"https://files.pythonhosted.org/packages/6d/05/0e7213e6f0dc71490a523a70e6ab5e7cd5140d87dc93a4447da58c440d6b/psutil-2.1.0.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.0.win32-py2.6.exe","hashes":{"sha256":"7eda949fbdf89548b0e52e7d666e096ea66b30b65bcbe04d305e036a24a76d11"},"provenance":null,"requires-python":null,"size":282509,"upload-time":"2014-04-08T15:12:57.055047Z","url":"https://files.pythonhosted.org/packages/f2/88/a856c5ed36d15b8ad74597f380baa29891ec284e7a1be4cb2f91f8453bd8/psutil-2.1.0.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.0.win32-py2.7.exe","hashes":{"sha256":"f1ec06db1b0d27681a1bea4c9f0b33705bc5a00035c32f168da0ea193883bb91"},"provenance":null,"requires-python":null,"size":282317,"upload-time":"2014-04-08T15:13:11.798772Z","url":"https://files.pythonhosted.org/packages/d0/17/ec99e9252dae834f4a012e13c804c86907fcb1cb474f7b1bc767562bfa7b/psutil-2.1.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.0.win32-py3.3.exe","hashes":{"sha256":"feccef3ccf09785c6f81e67223cf2d8736a90c8994623dd75f683dd2bf849235"},"provenance":null,"requires-python":null,"size":277268,"upload-time":"2014-04-08T15:13:26.711424Z","url":"https://files.pythonhosted.org/packages/f0/0a/32abfd9b965c9a433b5011574c904372b954330c170c6f92b637d661ecd2/psutil-2.1.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.0.win32-py3.4.exe","hashes":{"sha256":"4719134be984b0f2ba072ff761f334c2f3dbb8ca6af70ba43d8ca31b7e13c3db"},"provenance":null,"requires-python":null,"size":277279,"upload-time":"2014-04-08T15:13:41.834397Z","url":"https://files.pythonhosted.org/packages/f5/3b/eab6a8d832d805c7a00d0f2398c12a32bea7e8b6eb7d5fbdf869e4bcc9e0/psutil-2.1.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.1.tar.gz","hashes":{"sha256":"bf812a4aa6a41147d0e96e63d826eb7582fda6b54ad8f22534354b7f8ac45593"},"provenance":null,"requires-python":null,"size":216796,"upload-time":"2014-04-30T14:27:01.651394Z","url":"https://files.pythonhosted.org/packages/64/4b/70601d39b8e445265ed148affc49f7bfbd246940637785be5c80e007fa6e/psutil-2.1.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.1.win-amd64-py2.7.exe","hashes":{"sha256":"1b5ee64306535a625b77648e32f1b064c31bf58bfc7a37cde6c7bb3fa4abb6bd"},"provenance":null,"requires-python":null,"size":312763,"upload-time":"2014-04-30T14:31:09.411365Z","url":"https://files.pythonhosted.org/packages/e6/50/df05e0cbfcf20f022756d5e2da32b4f4f37d5bca6f5bd6965b4ef0460e8b/psutil-2.1.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.1.win-amd64-py3.3.exe","hashes":{"sha256":"9e28fe35e1185d59ba55313d90efc122d45b7a2e3fa304869024d26b0a809bc5"},"provenance":null,"requires-python":null,"size":311262,"upload-time":"2014-04-30T14:31:28.733500Z","url":"https://files.pythonhosted.org/packages/a3/a9/b3f114141a49244739321f221f35c300ac7f34ec9e3a352ea70c9fae41f8/psutil-2.1.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.1.win-amd64-py3.4.exe","hashes":{"sha256":"3afa327284a218d6a25b2f3520135337cfa4a47eaea030273ad8eb02894d60fe"},"provenance":null,"requires-python":null,"size":312365,"upload-time":"2014-04-30T14:31:47.981590Z","url":"https://files.pythonhosted.org/packages/c7/67/2aae3f66090c2e06fa60c29a2e554fd2a718949aca53bca78d640212cb34/psutil-2.1.1.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.1.win32-py2.4.exe","hashes":{"sha256":"7c76d99bfaeafbcf66096c69b8fca1f7269603d66cad1b14cd8dd93b14bceeb0"},"provenance":null,"requires-python":null,"size":145848,"upload-time":"2014-04-30T14:37:26.909225Z","url":"https://files.pythonhosted.org/packages/27/4e/c9b4802420d6b5d0a844208c9b8a4e25b3c37305428e40bc1da6f5076891/psutil-2.1.1.win32-py2.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.1.win32-py2.5.exe","hashes":{"sha256":"6aeb358b66cc4367378edadb928c77e82646f36858b2a516298d1917aa6aca25"},"provenance":null,"requires-python":null,"size":149998,"upload-time":"2014-04-30T14:28:49.630619Z","url":"https://files.pythonhosted.org/packages/5d/e9/69dee6454940bb102fbbcaa1e44b46bfefc4c0bf53e5e3835d3de3ebc9ae/psutil-2.1.1.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.1.win32-py2.6.exe","hashes":{"sha256":"a53546550067773920e1f7e6e3f1fad2f2befe45c6ff6c94e22b67f4b54c321a"},"provenance":null,"requires-python":null,"size":283143,"upload-time":"2014-04-30T14:29:06.869445Z","url":"https://files.pythonhosted.org/packages/9f/45/d2dfa6bdd9b64bfd46ad35af774c3819e0d5abf23a99d51adf11984ca658/psutil-2.1.1.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.1.win32-py2.7.exe","hashes":{"sha256":"37ce747f9042c375f62e861d627b5eb7bace24767303f5d0c4c03d17173a551c"},"provenance":null,"requires-python":null,"size":282941,"upload-time":"2014-04-30T14:29:27.059188Z","url":"https://files.pythonhosted.org/packages/78/47/14db8651f9863d301c0673d25fa22b87d13fde2974f94854502886a21fd1/psutil-2.1.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.1.win32-py3.3.exe","hashes":{"sha256":"8f0f88752e1e9bfeced78daf29d90e0028d17f39b805bb0acf70fefe77ba5ccb"},"provenance":null,"requires-python":null,"size":277944,"upload-time":"2014-04-30T14:29:49.405456Z","url":"https://files.pythonhosted.org/packages/f1/63/2fcaa58b101dce55a12f768508a7f0a0028ccc5a90633d86dd4cc0bcdb52/psutil-2.1.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.1.win32-py3.4.exe","hashes":{"sha256":"bc353df13a6ea40651cba82810a38590e2439c158cf6f130cd0876d0dda53118"},"provenance":null,"requires-python":null,"size":279092,"upload-time":"2014-04-30T14:30:08.449338Z","url":"https://files.pythonhosted.org/packages/5f/26/0be35c7f3dc9e78d405ed6be3aa76a9ce97b51e0076db98408a6f2c288fb/psutil-2.1.1.win32-py3.4.exe","yanked":false},{"core-metadata":{"sha256":"32ea7aa3898a3a77f4d45a211e8c96889ec6e5f933ea2ef9e4249d53fc41e4a1"},"data-dist-info-metadata":{"sha256":"32ea7aa3898a3a77f4d45a211e8c96889ec6e5f933ea2ef9e4249d53fc41e4a1"},"filename":"psutil-2.1.2-cp26-none-win32.whl","hashes":{"sha256":"ccb5e28357a4b6c572b97e710a070f349e9b45172315eaed6e0b72e86d333b68"},"provenance":null,"requires-python":null,"size":83869,"upload-time":"2014-09-21T13:47:09.164866Z","url":"https://files.pythonhosted.org/packages/c7/22/811ac7c641191e3b65c053c95eb34f6567bbc5155912630464271ab6f3df/psutil-2.1.2-cp26-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"32ea7aa3898a3a77f4d45a211e8c96889ec6e5f933ea2ef9e4249d53fc41e4a1"},"data-dist-info-metadata":{"sha256":"32ea7aa3898a3a77f4d45a211e8c96889ec6e5f933ea2ef9e4249d53fc41e4a1"},"filename":"psutil-2.1.2-cp27-none-win32.whl","hashes":{"sha256":"41992126c0281c2f5f279a0e8583382a3b840bd0d48262dfb7bc3cb67bdc6587"},"provenance":null,"requires-python":null,"size":83672,"upload-time":"2014-09-21T13:47:14.034296Z","url":"https://files.pythonhosted.org/packages/79/9d/154b179a73695ae605856c2d77ab5da2a66ef4819c3c4f97e4ab297a2902/psutil-2.1.2-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"32ea7aa3898a3a77f4d45a211e8c96889ec6e5f933ea2ef9e4249d53fc41e4a1"},"data-dist-info-metadata":{"sha256":"32ea7aa3898a3a77f4d45a211e8c96889ec6e5f933ea2ef9e4249d53fc41e4a1"},"filename":"psutil-2.1.2-cp27-none-win_amd64.whl","hashes":{"sha256":"79cb57bba4cbeebb7e445d19c531107493458a47d0f4c888ce49fc8dec670c32"},"provenance":null,"requires-python":null,"size":85848,"upload-time":"2014-09-21T13:47:30.044802Z","url":"https://files.pythonhosted.org/packages/00/ae/567c30ff44c263cc78dfe50197184e58b62ca9fcfebad144cd235f5d8d2d/psutil-2.1.2-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"e9eb9bc1a4851e5f154ad139504894781ffa9f0ee16ea77f8c17f4aee9d5aa50"},"data-dist-info-metadata":{"sha256":"e9eb9bc1a4851e5f154ad139504894781ffa9f0ee16ea77f8c17f4aee9d5aa50"},"filename":"psutil-2.1.2-cp33-none-win32.whl","hashes":{"sha256":"c5e6424833620f0d0a70ebc1305260bfad4b5c73989b8a2e8bd01bf39b4b600c"},"provenance":null,"requires-python":null,"size":83742,"upload-time":"2014-09-21T13:47:18.481401Z","url":"https://files.pythonhosted.org/packages/62/4d/be87c318274ade9d6589e8545dd5d5bb4bcc42e92334d88d948cf8daa36b/psutil-2.1.2-cp33-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"e9eb9bc1a4851e5f154ad139504894781ffa9f0ee16ea77f8c17f4aee9d5aa50"},"data-dist-info-metadata":{"sha256":"e9eb9bc1a4851e5f154ad139504894781ffa9f0ee16ea77f8c17f4aee9d5aa50"},"filename":"psutil-2.1.2-cp33-none-win_amd64.whl","hashes":{"sha256":"b4c69627c245025b9209acbffc066ff8ac1237d13e2eddd0461f3149969b7da3"},"provenance":null,"requires-python":null,"size":85822,"upload-time":"2014-09-21T13:47:33.681070Z","url":"https://files.pythonhosted.org/packages/bc/52/ab3c88a0574275ec33d9de0fe7dc9b1a32c573de0e468502876bc1df6f84/psutil-2.1.2-cp33-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"e9eb9bc1a4851e5f154ad139504894781ffa9f0ee16ea77f8c17f4aee9d5aa50"},"data-dist-info-metadata":{"sha256":"e9eb9bc1a4851e5f154ad139504894781ffa9f0ee16ea77f8c17f4aee9d5aa50"},"filename":"psutil-2.1.2-cp34-none-win32.whl","hashes":{"sha256":"97a1d0ddbb028a20feffaf7fa3d0c451004abc74ef7ea1e492d039f8b7278399"},"provenance":null,"requires-python":null,"size":83748,"upload-time":"2014-09-21T13:47:25.367914Z","url":"https://files.pythonhosted.org/packages/ef/4a/c956675314e4a50b319d9894c2ee2d48ce83df4d639d9c2fc06a99415dec/psutil-2.1.2-cp34-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"e9eb9bc1a4851e5f154ad139504894781ffa9f0ee16ea77f8c17f4aee9d5aa50"},"data-dist-info-metadata":{"sha256":"e9eb9bc1a4851e5f154ad139504894781ffa9f0ee16ea77f8c17f4aee9d5aa50"},"filename":"psutil-2.1.2-cp34-none-win_amd64.whl","hashes":{"sha256":"a78226f9236c674d43b206e7da06478cf2bcf10e5aee647f671878ea2434805f"},"provenance":null,"requires-python":null,"size":85789,"upload-time":"2014-09-21T13:47:38.760178Z","url":"https://files.pythonhosted.org/packages/b9/41/68b5fc38b97e037ef55e6475d618c079fe2b5d148f5e3fda795c21d888a7/psutil-2.1.2-cp34-none-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.2.tar.gz","hashes":{"sha256":"897e5163e0669001bf8bcb0557362f14703356336519082a93c38d54e5b392e4"},"provenance":null,"requires-python":null,"size":223595,"upload-time":"2014-09-21T13:50:56.650693Z","url":"https://files.pythonhosted.org/packages/53/6a/8051b913b2f94eb00fd045fe9e14a7182b6e7f088b12c308edd7616a559b/psutil-2.1.2.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.2.win-amd64-py2.7.exe","hashes":{"sha256":"3ed7ef4ab59894e6eb94adae0b656733c91906af169b55443329315322cd63b3"},"provenance":null,"requires-python":null,"size":319335,"upload-time":"2014-09-21T13:48:15.057062Z","url":"https://files.pythonhosted.org/packages/a6/88/cc912d38640ddf3441db1f85f8ff8a87f906056554239a4a211970cf6446/psutil-2.1.2.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.2.win-amd64-py3.3.exe","hashes":{"sha256":"e20f317187a7186e13dc6fdd6960083c227db86fffbb145fe000e8c167a854a6"},"provenance":null,"requires-python":null,"size":317817,"upload-time":"2014-09-21T13:48:20.665047Z","url":"https://files.pythonhosted.org/packages/d2/b4/887323eb2b0b5becb5a7b77ab04167741346fddffe27bc6ae5deeffcc3c1/psutil-2.1.2.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.2.win-amd64-py3.4.exe","hashes":{"sha256":"f7f2a61be7a828656dceffdf0d2fa304a144db6ab5ec4e2b033108f3822df6ff"},"provenance":null,"requires-python":null,"size":317804,"upload-time":"2014-09-21T13:48:33.226186Z","url":"https://files.pythonhosted.org/packages/57/e5/c041b08bea32246f53b7cf27b897e882a695f3ea95eb632b384dd35cf41f/psutil-2.1.2.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.2.win32-py2.4.exe","hashes":{"sha256":"77aac0179fe0e8b39c9b25ea732285d307bb941e715075366dc467b8609ebf09"},"provenance":null,"requires-python":null,"size":155426,"upload-time":"2014-09-21T13:54:03.017765Z","url":"https://files.pythonhosted.org/packages/ce/3f/7c434baec7ca47e65fb1e3bcbc8fe1c9f504d93c750a7fa4f98b16635920/psutil-2.1.2.win32-py2.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.2.win32-py2.5.exe","hashes":{"sha256":"982f4568876d10881f9c90fe992d1d9af353a59b5de7771cc4766d7432aee7ab"},"provenance":null,"requires-python":null,"size":155414,"upload-time":"2014-09-21T13:47:47.970150Z","url":"https://files.pythonhosted.org/packages/46/86/2971b31e2637ddc53adce2d997472cee8d4dec366d5a1e140945b95860d5/psutil-2.1.2.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.2.win32-py2.6.exe","hashes":{"sha256":"f35d91b5fc2b52472c0766fa411ac22f7a5ed8b1ececd96c1486380bdad0bb41"},"provenance":null,"requires-python":null,"size":289714,"upload-time":"2014-09-21T13:47:52.391726Z","url":"https://files.pythonhosted.org/packages/dd/3e/cdc3d4f343e6edc583d0686d1d20d98f9e11de35c51dc7990525ab2ca332/psutil-2.1.2.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.2.win32-py2.7.exe","hashes":{"sha256":"dbd9e4fbe08c0ff6d4a87a4d6dfae16fccd03346f980cffa105941de8801d00b"},"provenance":null,"requires-python":null,"size":289515,"upload-time":"2014-09-21T13:47:57.624310Z","url":"https://files.pythonhosted.org/packages/fb/01/73753147113f6db19734db6e7ac994cee5cce0f0935e12320d7aa1b56a14/psutil-2.1.2.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.2.win32-py3.3.exe","hashes":{"sha256":"73428e7b695e5889f07f1b37c5ec0cc0f49d3808dc40986050f1b13fd7c4c71e"},"provenance":null,"requires-python":null,"size":284506,"upload-time":"2014-09-21T13:48:02.428235Z","url":"https://files.pythonhosted.org/packages/b0/97/ba13c3915aba7776bb0d23819c04255230c46df1f8582752f7e0382c0b67/psutil-2.1.2.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.2.win32-py3.4.exe","hashes":{"sha256":"e2326c7b9f64a0f22891d3b362efcd92389e641023b07bc54567280c0c2e160d"},"provenance":null,"requires-python":null,"size":284538,"upload-time":"2014-09-21T13:48:08.358281Z","url":"https://files.pythonhosted.org/packages/88/bd/843fadc578d62f2888d5a7b4e2a418a28140ac239a5a5039c0d3678df647/psutil-2.1.2.win32-py3.4.exe","yanked":false},{"core-metadata":{"sha256":"69a75d24381600fdb642efa3827cf0c4e9c1c94273f2866e017a7017816167f8"},"data-dist-info-metadata":{"sha256":"69a75d24381600fdb642efa3827cf0c4e9c1c94273f2866e017a7017816167f8"},"filename":"psutil-2.1.3-cp26-none-win32.whl","hashes":{"sha256":"331a3b0b6688f95acebe87c02efca53003e613f1c7892a09063a9d3c0af33656"},"provenance":null,"requires-python":null,"size":83927,"upload-time":"2014-09-26T20:26:46.078909Z","url":"https://files.pythonhosted.org/packages/72/4c/f93448dfe2dec286b6eae91a00c32f1fbf65506e089354c51db76f4efdaf/psutil-2.1.3-cp26-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"69a75d24381600fdb642efa3827cf0c4e9c1c94273f2866e017a7017816167f8"},"data-dist-info-metadata":{"sha256":"69a75d24381600fdb642efa3827cf0c4e9c1c94273f2866e017a7017816167f8"},"filename":"psutil-2.1.3-cp27-none-win32.whl","hashes":{"sha256":"5ee9e17f6e92aaec7c53f6483c6708eec22124060a52ac339d104effcb4af37f"},"provenance":null,"requires-python":null,"size":83731,"upload-time":"2014-09-26T20:26:51.215134Z","url":"https://files.pythonhosted.org/packages/27/c5/a644b5df545c467569c7b8e768717fad6758eab8e2544b7d412d07c30ffe/psutil-2.1.3-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"69a75d24381600fdb642efa3827cf0c4e9c1c94273f2866e017a7017816167f8"},"data-dist-info-metadata":{"sha256":"69a75d24381600fdb642efa3827cf0c4e9c1c94273f2866e017a7017816167f8"},"filename":"psutil-2.1.3-cp27-none-win_amd64.whl","hashes":{"sha256":"47ea728de15c16a7cb476eb4f348be57e9daebfa730b9a4a104e9086da2cf6cd"},"provenance":null,"requires-python":null,"size":85906,"upload-time":"2014-09-26T20:27:07.470817Z","url":"https://files.pythonhosted.org/packages/89/82/1070ecb59d83967ddb2ea58b41ba2b9ffa8f3f686a3f49b8d33dc5684964/psutil-2.1.3-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"6ddf7c489626ecf2cd274a5d66e5785193f3eb6b2d596efff52afc15b251e25f"},"data-dist-info-metadata":{"sha256":"6ddf7c489626ecf2cd274a5d66e5785193f3eb6b2d596efff52afc15b251e25f"},"filename":"psutil-2.1.3-cp33-none-win32.whl","hashes":{"sha256":"ae0aac37af1d7d6821bb11dcffb12bb885b44773858bd40703e9dcb14325369c"},"provenance":null,"requires-python":null,"size":83804,"upload-time":"2014-09-26T20:26:56.233644Z","url":"https://files.pythonhosted.org/packages/b6/fc/7f2ceac523bb5ce9ea93bd85806ebb2d5327f0e430cbb6e70d32e4306e1d/psutil-2.1.3-cp33-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"6ddf7c489626ecf2cd274a5d66e5785193f3eb6b2d596efff52afc15b251e25f"},"data-dist-info-metadata":{"sha256":"6ddf7c489626ecf2cd274a5d66e5785193f3eb6b2d596efff52afc15b251e25f"},"filename":"psutil-2.1.3-cp33-none-win_amd64.whl","hashes":{"sha256":"45ebaf679050a6d64a32a91d901791868a29574f6961629997dcd4661edb4ece"},"provenance":null,"requires-python":null,"size":85884,"upload-time":"2014-09-26T20:27:12.800416Z","url":"https://files.pythonhosted.org/packages/45/33/a50ed3def836cd8fc53ce04d09e50df67a6816fd24a819e8e9c45b93fc74/psutil-2.1.3-cp33-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"6ddf7c489626ecf2cd274a5d66e5785193f3eb6b2d596efff52afc15b251e25f"},"data-dist-info-metadata":{"sha256":"6ddf7c489626ecf2cd274a5d66e5785193f3eb6b2d596efff52afc15b251e25f"},"filename":"psutil-2.1.3-cp34-none-win32.whl","hashes":{"sha256":"0ae7d503fc458181af7dd9a2bbf43c4b5fc7cd6e6797ef4a6d58bb8046026d76"},"provenance":null,"requires-python":null,"size":83804,"upload-time":"2014-09-26T20:27:02.155242Z","url":"https://files.pythonhosted.org/packages/55/e9/c122a9d2528cc36ff4f2824a714ee6d5da5462e4b6725e6b1c7098142c4a/psutil-2.1.3-cp34-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"6ddf7c489626ecf2cd274a5d66e5785193f3eb6b2d596efff52afc15b251e25f"},"data-dist-info-metadata":{"sha256":"6ddf7c489626ecf2cd274a5d66e5785193f3eb6b2d596efff52afc15b251e25f"},"filename":"psutil-2.1.3-cp34-none-win_amd64.whl","hashes":{"sha256":"da0961ba9a6d97c137381cd59b7d330894dd4f7deb5a8291bc3251375fd6d6ec"},"provenance":null,"requires-python":null,"size":85840,"upload-time":"2014-09-26T20:27:18.296315Z","url":"https://files.pythonhosted.org/packages/6f/97/5e01561cde882306c28e462b427f67d549add65f5fca324bf0bbdf831d21/psutil-2.1.3-cp34-none-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.3.tar.gz","hashes":{"sha256":"b434c75f01715777391f10f456002e33d0ca14633f96fdbd9ff9139b42d9452c"},"provenance":null,"requires-python":null,"size":224008,"upload-time":"2014-09-30T18:10:37.283557Z","url":"https://files.pythonhosted.org/packages/fe/a3/7cf43f28bbb52c4d680378f99e900ced201ade5073ee3a7b30e7f09e3c66/psutil-2.1.3.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.3.win-amd64-py2.7.exe","hashes":{"sha256":"449d07df8f8b9700cfae4ee67f0a73e4f96b55697428ae92cab29e33db4c3102"},"provenance":null,"requires-python":null,"size":319620,"upload-time":"2014-09-26T20:24:42.575161Z","url":"https://files.pythonhosted.org/packages/0e/66/bf4346c9ada08acee7e8f87d270fb4d4e6c86a477630adcfa3caa69aa5bb/psutil-2.1.3.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.3.win-amd64-py3.3.exe","hashes":{"sha256":"5b444e5c1f3d3ee7a19a5720c62d6462fe81dd1d1bbb8aa955546ead509b3c4a"},"provenance":null,"requires-python":null,"size":318101,"upload-time":"2014-09-26T20:24:47.973672Z","url":"https://files.pythonhosted.org/packages/bd/a7/888e9fa42c8d475de7b467b9635bb18ed360cb1c261bcf59f5f932d624ea/psutil-2.1.3.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.3.win-amd64-py3.4.exe","hashes":{"sha256":"3d25b4b20aabd360b7eda3dcbf4a14d2b256c2f61a8a569028e1c4b65b4d585a"},"provenance":null,"requires-python":null,"size":318089,"upload-time":"2014-09-26T20:24:55.536742Z","url":"https://files.pythonhosted.org/packages/b1/15/3f657d395213ad90efed423c88f1cb2f3b0a429d12bdea98a28c26de7ff4/psutil-2.1.3.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.3.win32-py2.4.exe","hashes":{"sha256":"36424b4e683d640291a1723f46a3949ebea37e43492f61027c168fb9dfe2055f"},"provenance":null,"requires-python":null,"size":155710,"upload-time":"2014-09-26T20:29:01.901009Z","url":"https://files.pythonhosted.org/packages/31/2e/d5448240fed09e88bbb59de194e1f2d3ba29f936a3231d718e5373736299/psutil-2.1.3.win32-py2.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.3.win32-py2.5.exe","hashes":{"sha256":"baa1803aaa4505fcf48bbd50d038d1f686d9e290defeecde41770d8fe876812b"},"provenance":null,"requires-python":null,"size":155699,"upload-time":"2014-09-26T20:24:13.810499Z","url":"https://files.pythonhosted.org/packages/2a/45/10d7b5057b3f941f803ca8ee690e651d2778e731f15ef3eee83c3f05f82f/psutil-2.1.3.win32-py2.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.3.win32-py2.6.exe","hashes":{"sha256":"55811a125d8a79bed0a65ed855fafc34f7d59cde162542a30cfa518da9e015bc"},"provenance":null,"requires-python":null,"size":289998,"upload-time":"2014-09-26T20:24:18.935804Z","url":"https://files.pythonhosted.org/packages/f3/4d/7f105269ece54ad7344c2a24b42ecb216b2746460f349c0ed1577b5ab8fa/psutil-2.1.3.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.3.win32-py2.7.exe","hashes":{"sha256":"6a1c8f1884f1983f8b74fda5dc89da07a69e3972fc022c3205f4964e1b01d235"},"provenance":null,"requires-python":null,"size":289802,"upload-time":"2014-09-26T20:24:24.112924Z","url":"https://files.pythonhosted.org/packages/93/64/1432ca27dfd7102a6726161b79b15a2997f461d3835867271a6c1e3353f7/psutil-2.1.3.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.3.win32-py3.3.exe","hashes":{"sha256":"8df72fcd30436e78bf8c5c6c796bb6815966511fa0bc8e3065e1aabbe4a2cf3d"},"provenance":null,"requires-python":null,"size":284794,"upload-time":"2014-09-26T20:24:30.110864Z","url":"https://files.pythonhosted.org/packages/0a/bb/d6cc7133624e3532e1d99f6cece35be7bd8d95ff7c82ca28cd502388c225/psutil-2.1.3.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.1.3.win32-py3.4.exe","hashes":{"sha256":"1f353ab0bbe0216e3a1636bfefb70ed366a2b1d95c95689f97e95d603626ef70"},"provenance":null,"requires-python":null,"size":284824,"upload-time":"2014-09-26T20:24:36.842776Z","url":"https://files.pythonhosted.org/packages/4a/66/9b1acf05dba9b9cb012ab3b8ed3d0fd7b01e1620808977fa6f1efe05dee2/psutil-2.1.3.win32-py3.4.exe","yanked":false},{"core-metadata":{"sha256":"30ed17dae0fd30cf429cfd022f7c9e5e8d675d87901927dda94ded45c224eb33"},"data-dist-info-metadata":{"sha256":"30ed17dae0fd30cf429cfd022f7c9e5e8d675d87901927dda94ded45c224eb33"},"filename":"psutil-2.2.0-cp26-none-win32.whl","hashes":{"sha256":"c92d2853ac52d5aaf8c2366435fb74892f7503daf2cf56b785f1ecadbf68e712"},"provenance":null,"requires-python":null,"size":82092,"upload-time":"2015-01-06T15:36:18.243917Z","url":"https://files.pythonhosted.org/packages/3c/65/f5bd2f8b54f14d950596c0c63d7b0f98b6dc779f7c61ac0dd2e9fc7e5e74/psutil-2.2.0-cp26-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"30ed17dae0fd30cf429cfd022f7c9e5e8d675d87901927dda94ded45c224eb33"},"data-dist-info-metadata":{"sha256":"30ed17dae0fd30cf429cfd022f7c9e5e8d675d87901927dda94ded45c224eb33"},"filename":"psutil-2.2.0-cp27-none-win32.whl","hashes":{"sha256":"6ad0e79b95f57a20f0cace08a063b0fc33fd83da1e5e501bc800bc69329a4501"},"provenance":null,"requires-python":null,"size":81891,"upload-time":"2015-01-06T15:36:28.466138Z","url":"https://files.pythonhosted.org/packages/6a/e6/96a4b4976f0eca169715d975b8c669b78b9afca58f9aadf261915694b73e/psutil-2.2.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"30ed17dae0fd30cf429cfd022f7c9e5e8d675d87901927dda94ded45c224eb33"},"data-dist-info-metadata":{"sha256":"30ed17dae0fd30cf429cfd022f7c9e5e8d675d87901927dda94ded45c224eb33"},"filename":"psutil-2.2.0-cp27-none-win_amd64.whl","hashes":{"sha256":"bbe719046986568ed3e84ce66ed617eebc46251ea8193566b3707abdf635afe6"},"provenance":null,"requires-python":null,"size":84072,"upload-time":"2015-01-06T15:36:58.206788Z","url":"https://files.pythonhosted.org/packages/28/28/4e4f94c9778ed16163f6c3dd696ce5e331121ac1600308830a3e98fa979a/psutil-2.2.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ac33f92c3e235fbeaf7e70b46e7b2d1503467bd420fac08e570cd9cfc3d3b738"},"data-dist-info-metadata":{"sha256":"ac33f92c3e235fbeaf7e70b46e7b2d1503467bd420fac08e570cd9cfc3d3b738"},"filename":"psutil-2.2.0-cp33-none-win32.whl","hashes":{"sha256":"73013822a953fc4e3fb269256ec01b261752c590e23851e666201d1bfd32a3a9"},"provenance":null,"requires-python":null,"size":81885,"upload-time":"2015-01-06T15:36:38.776238Z","url":"https://files.pythonhosted.org/packages/66/8d/3143623c2f5bc264197727854040fdc02e3175b4ad991490586db7a512ed/psutil-2.2.0-cp33-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"ac33f92c3e235fbeaf7e70b46e7b2d1503467bd420fac08e570cd9cfc3d3b738"},"data-dist-info-metadata":{"sha256":"ac33f92c3e235fbeaf7e70b46e7b2d1503467bd420fac08e570cd9cfc3d3b738"},"filename":"psutil-2.2.0-cp33-none-win_amd64.whl","hashes":{"sha256":"d664a896feb10ec5bf4c43f4df8b6c7fceeb94677004cc9cf8c9f35b52c0e4fc"},"provenance":null,"requires-python":null,"size":84002,"upload-time":"2015-01-06T15:37:08.775842Z","url":"https://files.pythonhosted.org/packages/bd/08/112807380b8e7b76de8c84f920234a0ebed35e6511271f93f900f691a10c/psutil-2.2.0-cp33-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ac33f92c3e235fbeaf7e70b46e7b2d1503467bd420fac08e570cd9cfc3d3b738"},"data-dist-info-metadata":{"sha256":"ac33f92c3e235fbeaf7e70b46e7b2d1503467bd420fac08e570cd9cfc3d3b738"},"filename":"psutil-2.2.0-cp34-none-win32.whl","hashes":{"sha256":"84299c41b251bef2a8b0812d651f4715209b3c4dfebe4a5df0f103bbdec78221"},"provenance":null,"requires-python":null,"size":81886,"upload-time":"2015-01-06T15:36:49.808417Z","url":"https://files.pythonhosted.org/packages/ef/0e/0cf2fea8f6e2e5ef84797eefc2c5ce561123e2417d7b931a6c54f9f8d413/psutil-2.2.0-cp34-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"ac33f92c3e235fbeaf7e70b46e7b2d1503467bd420fac08e570cd9cfc3d3b738"},"data-dist-info-metadata":{"sha256":"ac33f92c3e235fbeaf7e70b46e7b2d1503467bd420fac08e570cd9cfc3d3b738"},"filename":"psutil-2.2.0-cp34-none-win_amd64.whl","hashes":{"sha256":"3636879fcbde2b0b63db08abd0e3673c2cc72bb14075e46e15f98774b0c78236"},"provenance":null,"requires-python":null,"size":83954,"upload-time":"2015-01-06T15:37:18.558007Z","url":"https://files.pythonhosted.org/packages/82/ed/ce9c4281fd8a944a725c0f9a5d77f32295777ab22f54b4f706a51d59edd3/psutil-2.2.0-cp34-none-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.2.0.tar.gz","hashes":{"sha256":"b15cc9e7cad0991bd1cb806fa90ea85ba3a95d0f1226625ecef993294ad61521"},"provenance":null,"requires-python":null,"size":223676,"upload-time":"2015-01-06T15:38:45.979998Z","url":"https://files.pythonhosted.org/packages/ba/e4/760f64a8a5a5f1b95f3ce17c0d51134952f930caf1218e6ce21902f6c4ab/psutil-2.2.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.2.0.win-amd64-py2.7.exe","hashes":{"sha256":"623baf2a213adc99cbc067e1db04b9e578eb4c38d9535c903e04b6d1ded10cab"},"provenance":null,"requires-python":null,"size":317990,"upload-time":"2015-01-06T15:34:57.187611Z","url":"https://files.pythonhosted.org/packages/30/a8/d1754e9e5492717d1c1afb30970f0677056052859f1af935b48c72c6cd68/psutil-2.2.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.2.0.win-amd64-py3.3.exe","hashes":{"sha256":"fc6d348f603ae8465992a7edbc7c62cc16f0493bdd43aa609dcb7437992cac96"},"provenance":null,"requires-python":null,"size":316424,"upload-time":"2015-01-06T15:35:20.860940Z","url":"https://files.pythonhosted.org/packages/32/f1/e98caa1f6be7ba49dfafa0fbb63f50ffa191d316a1b2b3ec431d97ebf494/psutil-2.2.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.2.0.win-amd64-py3.4.exe","hashes":{"sha256":"5e4508ca6822839f25310bfc14c050e60362dc29ddea6f5eac91de2c7423a471"},"provenance":null,"requires-python":null,"size":316403,"upload-time":"2015-01-06T15:35:42.094623Z","url":"https://files.pythonhosted.org/packages/dd/46/31505418bd950b09d28c6c21be76a4a51d593ec06b412539797080c0aa6b/psutil-2.2.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.2.0.win32-py2.6.exe","hashes":{"sha256":"1f93ccdf415da40f15a84ab6d9d32ddda61bb1b20079dae602356e087f408d28"},"provenance":null,"requires-python":null,"size":288364,"upload-time":"2015-01-06T15:33:25.830661Z","url":"https://files.pythonhosted.org/packages/9c/0d/f360da29c906dafd825edccc219e70dab73370ad2c287fa5baa9f7fa370b/psutil-2.2.0.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.2.0.win32-py2.7.exe","hashes":{"sha256":"83eb1739b7c87a21a65224d77217325e582199bedd0141b29dac81b4e4144c62"},"provenance":null,"requires-python":null,"size":288166,"upload-time":"2015-01-06T15:33:46.994076Z","url":"https://files.pythonhosted.org/packages/7a/30/5bb6644318f2279caf5d334f32df19fe4b2bce11ef418af32124e16f8e98/psutil-2.2.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.2.0.win32-py3.3.exe","hashes":{"sha256":"9db585f37d56381c37738e03aa8beb0b501b26adc7c70660ff182e0473f2cb0a"},"provenance":null,"requires-python":null,"size":283089,"upload-time":"2015-01-06T15:34:11.431221Z","url":"https://files.pythonhosted.org/packages/12/c4/cb75d51edb425ff77e2af5bb32e405ed41beb124ad062fb927bbe135c709/psutil-2.2.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.2.0.win32-py3.4.exe","hashes":{"sha256":"4a0d8a192d1523a3f02a4028bf4ac296f7da92c935464d302618bd639c03a2c6"},"provenance":null,"requires-python":null,"size":283110,"upload-time":"2015-01-06T15:34:33.472476Z","url":"https://files.pythonhosted.org/packages/17/fb/ae589efdfd076d1961e1f858c969d234a63e26f648a79b3fac0409e95c2f/psutil-2.2.0.win32-py3.4.exe","yanked":false},{"core-metadata":{"sha256":"ff83c284ef44f26c82b6ebdf69d521137b95a89dc815e363bc49eedb63a04c98"},"data-dist-info-metadata":{"sha256":"ff83c284ef44f26c82b6ebdf69d521137b95a89dc815e363bc49eedb63a04c98"},"filename":"psutil-2.2.1-cp26-none-win32.whl","hashes":{"sha256":"610f08fc0df646e4997b52f2bf2e40118a560738422d540883297c6f4e38be8c"},"provenance":null,"requires-python":null,"size":82104,"upload-time":"2015-02-02T13:08:03.993201Z","url":"https://files.pythonhosted.org/packages/0a/a7/b6323d30a8dde3f6a139809898e4bdcabcae3129314621aa24eb5ca4468e/psutil-2.2.1-cp26-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"ff83c284ef44f26c82b6ebdf69d521137b95a89dc815e363bc49eedb63a04c98"},"data-dist-info-metadata":{"sha256":"ff83c284ef44f26c82b6ebdf69d521137b95a89dc815e363bc49eedb63a04c98"},"filename":"psutil-2.2.1-cp27-none-win32.whl","hashes":{"sha256":"84ec9e77cab1d9ecf2d93bf43eba255a11b26d480966ee542bc846be1e272ea5"},"provenance":null,"requires-python":null,"size":81907,"upload-time":"2015-02-02T13:08:14.963426Z","url":"https://files.pythonhosted.org/packages/40/b6/a94c6d00ac18f779c72973639b78519de0c488e8e8e8d00b65cf4287bec3/psutil-2.2.1-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"ff83c284ef44f26c82b6ebdf69d521137b95a89dc815e363bc49eedb63a04c98"},"data-dist-info-metadata":{"sha256":"ff83c284ef44f26c82b6ebdf69d521137b95a89dc815e363bc49eedb63a04c98"},"filename":"psutil-2.2.1-cp27-none-win_amd64.whl","hashes":{"sha256":"81a6ab277886f233f230c46073fa6fc97ca2a95a44a14b80c6f6054acb7a644b"},"provenance":null,"requires-python":null,"size":84086,"upload-time":"2015-02-02T13:08:42.917519Z","url":"https://files.pythonhosted.org/packages/1c/ff/6b00405e4eeb3c40d4fc07142ef7bdc3a7a7aa2a1640b65e654bdb7682d1/psutil-2.2.1-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"85bb3fab01138e16635be140adec7d7fd83c5150a93731deb5a4020ec9d569b1"},"data-dist-info-metadata":{"sha256":"85bb3fab01138e16635be140adec7d7fd83c5150a93731deb5a4020ec9d569b1"},"filename":"psutil-2.2.1-cp33-none-win32.whl","hashes":{"sha256":"f5e0d247b09c9460b896ff89098015b7687a2934d4ed6165a5c3a662fad7ab6b"},"provenance":null,"requires-python":null,"size":81896,"upload-time":"2015-02-02T13:08:24.437138Z","url":"https://files.pythonhosted.org/packages/a2/fe/40b7d42057c5e68ff4c733b5689e9ae9d8785f13576326ba44371689eff7/psutil-2.2.1-cp33-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"85bb3fab01138e16635be140adec7d7fd83c5150a93731deb5a4020ec9d569b1"},"data-dist-info-metadata":{"sha256":"85bb3fab01138e16635be140adec7d7fd83c5150a93731deb5a4020ec9d569b1"},"filename":"psutil-2.2.1-cp33-none-win_amd64.whl","hashes":{"sha256":"07f31cbe5fc1183c8efb52bab9a5382ebd001a6e11b5a5df827f69456219d514"},"provenance":null,"requires-python":null,"size":84011,"upload-time":"2015-02-02T13:08:53.420801Z","url":"https://files.pythonhosted.org/packages/fe/8d/ffc94f092a12fc3cb837b9bda93abd88fb12219bcc6684b9bffea5e1f385/psutil-2.2.1-cp33-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"85bb3fab01138e16635be140adec7d7fd83c5150a93731deb5a4020ec9d569b1"},"data-dist-info-metadata":{"sha256":"85bb3fab01138e16635be140adec7d7fd83c5150a93731deb5a4020ec9d569b1"},"filename":"psutil-2.2.1-cp34-none-win32.whl","hashes":{"sha256":"a98b350251df2658c9be6bf0b6ebbab10fe88ccb513079c2c56fb53330530ee6"},"provenance":null,"requires-python":null,"size":81896,"upload-time":"2015-02-02T13:08:34.116542Z","url":"https://files.pythonhosted.org/packages/7f/3a/4bb24092c3de0a44f2fc1afb7bcaf898f1daba67cba5b9188b0ae6527102/psutil-2.2.1-cp34-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"85bb3fab01138e16635be140adec7d7fd83c5150a93731deb5a4020ec9d569b1"},"data-dist-info-metadata":{"sha256":"85bb3fab01138e16635be140adec7d7fd83c5150a93731deb5a4020ec9d569b1"},"filename":"psutil-2.2.1-cp34-none-win_amd64.whl","hashes":{"sha256":"f49d3ca9150f20269eae79d6efac98f0bbe4e5ceac46fe418d881560a04b6d8e"},"provenance":null,"requires-python":null,"size":83963,"upload-time":"2015-02-02T13:09:03.990685Z","url":"https://files.pythonhosted.org/packages/64/f7/c385e9350abbbb082364596fd4b0066fcb0c51ebc103a7ebf6eb9bc837e9/psutil-2.2.1-cp34-none-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.2.1.tar.gz","hashes":{"sha256":"a0e9b96f1946975064724e242ac159f3260db24ffa591c3da0a355361a3a337f"},"provenance":null,"requires-python":null,"size":223688,"upload-time":"2015-02-02T13:10:14.673714Z","url":"https://files.pythonhosted.org/packages/df/47/ee54ef14dd40f8ce831a7581001a5096494dc99fe71586260ca6b531fe86/psutil-2.2.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.2.1.win-amd64-py2.7.exe","hashes":{"sha256":"552eaba2dbc9a49af2da64fc00cedf8847c7c9c2559cc620d9c8855583105764"},"provenance":null,"requires-python":null,"size":317995,"upload-time":"2015-02-02T13:06:42.422390Z","url":"https://files.pythonhosted.org/packages/97/27/e64014166df9efc3d00f987a251938d534d3897d62ef21486559a697a770/psutil-2.2.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.2.1.win-amd64-py3.3.exe","hashes":{"sha256":"d25746746f2680d6174c9eb0e2793cb73304ca200d5ec7334000fdb575cd6577"},"provenance":null,"requires-python":null,"size":316429,"upload-time":"2015-02-02T13:07:07.458895Z","url":"https://files.pythonhosted.org/packages/c2/dd/5fdd5fd6102ae546452a5f32a6f75ec44c11d6b84c188dd33cfe1e2809e1/psutil-2.2.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.2.1.win-amd64-py3.4.exe","hashes":{"sha256":"298ed760a365a846337750c5089adcec2a87014c61132ae39db671982750f35a"},"provenance":null,"requires-python":null,"size":316409,"upload-time":"2015-02-02T13:07:29.047577Z","url":"https://files.pythonhosted.org/packages/d9/59/0f14daf5797c61db2fec25bca49b49aa0a5d57dec57b4e58ae4652dadb95/psutil-2.2.1.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.2.1.win32-py2.6.exe","hashes":{"sha256":"6e49f72a1cd57918b275fdf7dd041618696fc39d5705ac722fdfccaf0b784a93"},"provenance":null,"requires-python":null,"size":288372,"upload-time":"2015-02-02T13:05:10.073479Z","url":"https://files.pythonhosted.org/packages/0c/a2/63967c61fbfc3f60c1b17e652a52d1afeb3c56403990c5a8f2fb801e2aa1/psutil-2.2.1.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.2.1.win32-py2.7.exe","hashes":{"sha256":"9227949290983df92bb3bb57c5b25605ebbc0e61a3f9baa394f752ed7abc914c"},"provenance":null,"requires-python":null,"size":288171,"upload-time":"2015-02-02T13:05:31.604222Z","url":"https://files.pythonhosted.org/packages/46/f3/d50f059d7d297db7335e340333124eac9441c5897bb29223c85ebaa64e7d/psutil-2.2.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.2.1.win32-py3.3.exe","hashes":{"sha256":"f8d94c8228011d1b658c39783596d95244e01950413bfc41fd44e78129ef7075"},"provenance":null,"requires-python":null,"size":283095,"upload-time":"2015-02-02T13:05:54.134208Z","url":"https://files.pythonhosted.org/packages/4a/d6/341767cdc6890adfb19ad60683e66dab3cdde605bc226427c8ec17dbd3f5/psutil-2.2.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-2.2.1.win32-py3.4.exe","hashes":{"sha256":"1e1df017bfa14c0edc87a124b11ca6e7fd8b6b5c9692eb06c1b72077bcca3845"},"provenance":null,"requires-python":null,"size":283115,"upload-time":"2015-02-02T13:06:12.827127Z","url":"https://files.pythonhosted.org/packages/8c/c1/24179e79ab74bde86ee766c4c0d2bb90b98e4e4d14017efeec024caba268/psutil-2.2.1.win32-py3.4.exe","yanked":false},{"core-metadata":{"sha256":"e61eb67de1755dd6d2917f81bb9ee7a0ae6dc5cda2948621b91794f7c5348033"},"data-dist-info-metadata":{"sha256":"e61eb67de1755dd6d2917f81bb9ee7a0ae6dc5cda2948621b91794f7c5348033"},"filename":"psutil-3.0.0-cp27-none-win32.whl","hashes":{"sha256":"9641677de91769127c82aa032b70a8d9aa75369d0a9965aba880ac335b02eccb"},"provenance":null,"requires-python":null,"size":85709,"upload-time":"2015-06-13T19:16:33.857911Z","url":"https://files.pythonhosted.org/packages/bb/cf/f893fa2fb0888384b5d668e22b6e3652c9ce187e749604027a64a6bd6646/psutil-3.0.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"e61eb67de1755dd6d2917f81bb9ee7a0ae6dc5cda2948621b91794f7c5348033"},"data-dist-info-metadata":{"sha256":"e61eb67de1755dd6d2917f81bb9ee7a0ae6dc5cda2948621b91794f7c5348033"},"filename":"psutil-3.0.0-cp27-none-win_amd64.whl","hashes":{"sha256":"11c6b17b37ab1ea04f6a1365bc4598637490819d6b744f3947fd3b8b425366b3"},"provenance":null,"requires-python":null,"size":88043,"upload-time":"2015-06-13T19:16:52.139563Z","url":"https://files.pythonhosted.org/packages/f1/26/a0904455b550f7fd5baa069ca10889825cc22440147813713998cd0676a0/psutil-3.0.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"adc9c5e259a9c63a9ebd7ef296c4a895baa01701f7d8127d1f989e968fa497aa"},"data-dist-info-metadata":{"sha256":"adc9c5e259a9c63a9ebd7ef296c4a895baa01701f7d8127d1f989e968fa497aa"},"filename":"psutil-3.0.0-cp33-none-win32.whl","hashes":{"sha256":"8333d2a496fe5d24171360ab1f78468db7a9691d61fd26bf15ca62336c699bac"},"provenance":null,"requires-python":null,"size":85737,"upload-time":"2015-06-13T19:16:39.871421Z","url":"https://files.pythonhosted.org/packages/ea/45/defb71d5009e178ef13e2474627a1070ae291b2dee64ae00e8f2b1a2aec2/psutil-3.0.0-cp33-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"adc9c5e259a9c63a9ebd7ef296c4a895baa01701f7d8127d1f989e968fa497aa"},"data-dist-info-metadata":{"sha256":"adc9c5e259a9c63a9ebd7ef296c4a895baa01701f7d8127d1f989e968fa497aa"},"filename":"psutil-3.0.0-cp33-none-win_amd64.whl","hashes":{"sha256":"44711a1a683369056d99fd08a05789d217205d72b6dbbe75c0b3743c5e1e3768"},"provenance":null,"requires-python":null,"size":87943,"upload-time":"2015-06-13T19:16:58.439865Z","url":"https://files.pythonhosted.org/packages/3c/0b/648fbaef2a037cc0c05388c0416ba3ca9244c22c4156dba73e4a900540a6/psutil-3.0.0-cp33-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"adc9c5e259a9c63a9ebd7ef296c4a895baa01701f7d8127d1f989e968fa497aa"},"data-dist-info-metadata":{"sha256":"adc9c5e259a9c63a9ebd7ef296c4a895baa01701f7d8127d1f989e968fa497aa"},"filename":"psutil-3.0.0-cp34-none-win32.whl","hashes":{"sha256":"9036f9c57a8c8a571f36c7d3b2c9a15a22dc9a95137fa60191e2eaa9527209d5"},"provenance":null,"requires-python":null,"size":85695,"upload-time":"2015-06-13T19:16:46.286646Z","url":"https://files.pythonhosted.org/packages/6d/ca/1d5dafae4d7396a825531296b6837b45129b63463b4eb221ac3429eb157d/psutil-3.0.0-cp34-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"adc9c5e259a9c63a9ebd7ef296c4a895baa01701f7d8127d1f989e968fa497aa"},"data-dist-info-metadata":{"sha256":"adc9c5e259a9c63a9ebd7ef296c4a895baa01701f7d8127d1f989e968fa497aa"},"filename":"psutil-3.0.0-cp34-none-win_amd64.whl","hashes":{"sha256":"18e53256390b299ff03bd58dc134d070bde4871d8c93baf08faea316e87544c8"},"provenance":null,"requires-python":null,"size":87893,"upload-time":"2015-06-13T19:17:05.116909Z","url":"https://files.pythonhosted.org/packages/59/82/51ab7cecaf03c8cbe7eff3b341f6564c3fa80baaa5a37643096cb526cae2/psutil-3.0.0-cp34-none-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.0.0.tar.gz","hashes":{"sha256":"a43cc84c6a2406e8f23785c68a52b792d95eb91e2b43be40f7c814bf80dc5979"},"provenance":null,"requires-python":null,"size":240872,"upload-time":"2015-06-13T19:14:46.063288Z","url":"https://files.pythonhosted.org/packages/ba/18/c5bb52abb67194aabadfd61286b538ee8856c7cb9c0e92dd64c1b132bf5e/psutil-3.0.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.0.0.win-amd64-py2.7.exe","hashes":{"sha256":"26b2cf4456e039f490f79457100564ff34718f4257b68dee53b42d17493f0187"},"provenance":null,"requires-python":null,"size":323299,"upload-time":"2015-06-13T19:15:53.089846Z","url":"https://files.pythonhosted.org/packages/0c/b1/fa44dd6ed19e8be6f9f05f776e824ce202822f5d6237093c54d6c5102888/psutil-3.0.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.0.0.win-amd64-py3.3.exe","hashes":{"sha256":"32808b29473e788487ee28a65caaab1c9fbd44c2d654ce6d563a89f20e106996"},"provenance":null,"requires-python":null,"size":321705,"upload-time":"2015-06-13T19:16:01.370127Z","url":"https://files.pythonhosted.org/packages/62/4f/89c6ac53e424109639f42033d0e87c4de4ddb610060007c6d5f6081a0fba/psutil-3.0.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.0.0.win-amd64-py3.4.exe","hashes":{"sha256":"37a9869ef17680e768bf32d7556252a04f426e5b1e6fa51d5af16305c8a701b6"},"provenance":null,"requires-python":null,"size":321688,"upload-time":"2015-06-13T19:16:11.540515Z","url":"https://files.pythonhosted.org/packages/d2/67/9c988331d40c445ebc9d29d95534ce3908f3949f6a84d640a6c125bc56e2/psutil-3.0.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.0.0.win32-py2.6.exe","hashes":{"sha256":"f29ee1aafa452cce77667e75f5e0345b678b8c091b288d0118840df30f303bed"},"provenance":null,"requires-python":null,"size":293522,"upload-time":"2015-06-13T19:15:19.130222Z","url":"https://files.pythonhosted.org/packages/10/19/9e2f8a305f5a6c01feee59dfeb38a73dc1ea4f4af0133cef9a36708f4111/psutil-3.0.0.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.0.0.win32-py2.7.exe","hashes":{"sha256":"7cbcf866faa23e5229ecfe44ff562f406b4806caae66cb255dbb327247e540aa"},"provenance":null,"requires-python":null,"size":293319,"upload-time":"2015-06-13T19:15:27.353152Z","url":"https://files.pythonhosted.org/packages/fc/0f/92e595cd2f2a80cead241d45b4ce4961e2515deff101644f3812c75e9bc7/psutil-3.0.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.0.0.win32-py3.3.exe","hashes":{"sha256":"b968771d3db5eef0c17ef6bed2d7756b8ac3cce40e11d5493824863d195dd8e3"},"provenance":null,"requires-python":null,"size":288278,"upload-time":"2015-06-13T19:15:35.880360Z","url":"https://files.pythonhosted.org/packages/fc/29/387c555f9dc38c6bb084a8df8936e607848de9e2e984dbf7eb2a298c1ceb/psutil-3.0.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.0.0.win32-py3.4.exe","hashes":{"sha256":"81ac6f54a27d091c75333f7c90082d028db45a1e78de59d28ca9f38cf6186395"},"provenance":null,"requires-python":null,"size":288253,"upload-time":"2015-06-13T19:15:44.871254Z","url":"https://files.pythonhosted.org/packages/fb/29/25ac80b589c0e56214ac64fdd8216992be162a2b2290f9b88b9a5c517cfd/psutil-3.0.0.win32-py3.4.exe","yanked":false},{"core-metadata":{"sha256":"986bdfc454536c2a940c955834882e01a25de324bf458054673b43d6b47a00ee"},"data-dist-info-metadata":{"sha256":"986bdfc454536c2a940c955834882e01a25de324bf458054673b43d6b47a00ee"},"filename":"psutil-3.0.1-cp27-none-win32.whl","hashes":{"sha256":"bcb8d23121848953ed295f7e3c0875b0164ee98d3245060beb3623c59ff2a1bc"},"provenance":null,"requires-python":null,"size":85793,"upload-time":"2015-06-18T02:36:50.974286Z","url":"https://files.pythonhosted.org/packages/17/05/b2c807b470464fbe3d357734b68199451574fcd75431fd3e5c77be24b6e0/psutil-3.0.1-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"986bdfc454536c2a940c955834882e01a25de324bf458054673b43d6b47a00ee"},"data-dist-info-metadata":{"sha256":"986bdfc454536c2a940c955834882e01a25de324bf458054673b43d6b47a00ee"},"filename":"psutil-3.0.1-cp27-none-win_amd64.whl","hashes":{"sha256":"ba56ec5c052489b7a7015c26ed3f917f2df4ffa9799266e86be41815bc358b80"},"provenance":null,"requires-python":null,"size":88128,"upload-time":"2015-06-18T02:37:08.778852Z","url":"https://files.pythonhosted.org/packages/38/bf/0b743c8a07265f2ecb203f8e60310571dcae33036aa6ba7aa16e2641ac7a/psutil-3.0.1-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"98502550423dd5cf7f5bcfaed4c1d92354cc0353b2b9bf673992e42fa246dce5"},"data-dist-info-metadata":{"sha256":"98502550423dd5cf7f5bcfaed4c1d92354cc0353b2b9bf673992e42fa246dce5"},"filename":"psutil-3.0.1-cp33-none-win32.whl","hashes":{"sha256":"23606e9b42760a8fdeada33281d0c3ce88df220949746e0ddf54d5db7974b4c6"},"provenance":null,"requires-python":null,"size":85819,"upload-time":"2015-06-18T02:36:56.905878Z","url":"https://files.pythonhosted.org/packages/b2/9c/b2c4373b9406eaf33654c4be828d9316ee780f4b3c19d0fa56f55eb64d61/psutil-3.0.1-cp33-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"98502550423dd5cf7f5bcfaed4c1d92354cc0353b2b9bf673992e42fa246dce5"},"data-dist-info-metadata":{"sha256":"98502550423dd5cf7f5bcfaed4c1d92354cc0353b2b9bf673992e42fa246dce5"},"filename":"psutil-3.0.1-cp33-none-win_amd64.whl","hashes":{"sha256":"ae4a7f51f40154d02ab1576e94377171a28cc83fc89c077c152f16ba3dae72f3"},"provenance":null,"requires-python":null,"size":88028,"upload-time":"2015-06-18T02:37:14.406512Z","url":"https://files.pythonhosted.org/packages/7a/59/1a9a10238226dbaed24b8d978e9cc743ac143504b3f4ad8f5e3a169e263a/psutil-3.0.1-cp33-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"98502550423dd5cf7f5bcfaed4c1d92354cc0353b2b9bf673992e42fa246dce5"},"data-dist-info-metadata":{"sha256":"98502550423dd5cf7f5bcfaed4c1d92354cc0353b2b9bf673992e42fa246dce5"},"filename":"psutil-3.0.1-cp34-none-win32.whl","hashes":{"sha256":"b7520db52c7c4e38cdd50bb11be02a826372d8002bf92bbe955823cac32c7f9d"},"provenance":null,"requires-python":null,"size":85783,"upload-time":"2015-06-18T02:37:03.210308Z","url":"https://files.pythonhosted.org/packages/ed/85/1d4d97ce4c9a8c6dd479f5f717694016651170d3731cdecf740db0e4eae3/psutil-3.0.1-cp34-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"98502550423dd5cf7f5bcfaed4c1d92354cc0353b2b9bf673992e42fa246dce5"},"data-dist-info-metadata":{"sha256":"98502550423dd5cf7f5bcfaed4c1d92354cc0353b2b9bf673992e42fa246dce5"},"filename":"psutil-3.0.1-cp34-none-win_amd64.whl","hashes":{"sha256":"1754d4118eaab16f299a37dafaf7d34111e9a8e5ac2a799e2bd9b1a5d9d1122b"},"provenance":null,"requires-python":null,"size":87982,"upload-time":"2015-06-18T02:37:20.490529Z","url":"https://files.pythonhosted.org/packages/1e/e6/0af7e190f74d6f959dfb87f2b56f4711271729952691f843fb91c5e06712/psutil-3.0.1-cp34-none-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.0.1.tar.gz","hashes":{"sha256":"3f213b9ceed3c3068a973e04d7a8b2a29d1076abcb5ef45382517bfc6b808801"},"provenance":null,"requires-python":null,"size":241539,"upload-time":"2015-06-18T02:33:52.119548Z","url":"https://files.pythonhosted.org/packages/aa/5d/cbd3b7227fe7a4c2c77e4031b6c43961563a3ecde2981190e5afe959be51/psutil-3.0.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.0.1.win-amd64-py2.7.exe","hashes":{"sha256":"b0a2ed567b31f71ae2e893768f0da0d51d51d12714471d4b7431e70ff5e36577"},"provenance":null,"requires-python":null,"size":323471,"upload-time":"2015-06-18T02:36:04.537383Z","url":"https://files.pythonhosted.org/packages/9d/a4/815181cacd33f0ce62a3c3aa188af65bd3945888aa2f6c16a925837ca517/psutil-3.0.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.0.1.win-amd64-py3.3.exe","hashes":{"sha256":"99c593cb459b54209cdb4aed4a607fa8b2920fbd4f3c5a9219a0ede114975758"},"provenance":null,"requires-python":null,"size":321879,"upload-time":"2015-06-18T02:36:14.693044Z","url":"https://files.pythonhosted.org/packages/08/b2/278ac09b03db15b9e51c3ae7f678b3fbf050e895a2eccab66d05017bdef9/psutil-3.0.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.0.1.win-amd64-py3.4.exe","hashes":{"sha256":"8f803bfe00a763254581bb6a3d788b3332492fbc67dbf46ad5068b663e44f309"},"provenance":null,"requires-python":null,"size":321861,"upload-time":"2015-06-18T02:36:23.687685Z","url":"https://files.pythonhosted.org/packages/d8/66/d21041db114938ae22d4994ea31f2d8f1353aa61e5f0d0c6c4e185b016a7/psutil-3.0.1.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.0.1.win32-py2.6.exe","hashes":{"sha256":"4a19475c1d6071c685b38f85837f6e6daa2c6ccd4d0132ab840123deb8ea2372"},"provenance":null,"requires-python":null,"size":293700,"upload-time":"2015-06-18T02:35:31.083110Z","url":"https://files.pythonhosted.org/packages/99/67/980b4a9257abaa3f53a60a3441f759301a0aca2922d60e18a05d23fb1b0e/psutil-3.0.1.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.0.1.win32-py2.7.exe","hashes":{"sha256":"363d3dbd610ce7bcf7f13b0a31133ee231d0990e99315142ed37f6ba2c1a84e6"},"provenance":null,"requires-python":null,"size":293492,"upload-time":"2015-06-18T02:35:39.030138Z","url":"https://files.pythonhosted.org/packages/5f/64/b994ed73ab49d5c847d97c47600b539603ebf0d6cfd8d3575b80db7aefb5/psutil-3.0.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.0.1.win32-py3.3.exe","hashes":{"sha256":"fc7cfe1d6919cb67f4144947acfcefc099d7d8299dd88bb4d863e62c44d041b4"},"provenance":null,"requires-python":null,"size":288451,"upload-time":"2015-06-18T02:35:47.145538Z","url":"https://files.pythonhosted.org/packages/1f/11/4871e823ff0d5a302a7f8ece60358d20ba7fc1620c4c9e2d94e826e1ff0e/psutil-3.0.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.0.1.win32-py3.4.exe","hashes":{"sha256":"73091a80ed295f990ab377dfd8cd4f7f00e3abfffb5e500192f1ea9fa58de158"},"provenance":null,"requires-python":null,"size":288427,"upload-time":"2015-06-18T02:35:55.768442Z","url":"https://files.pythonhosted.org/packages/08/af/c78ef8dc09ac61a479968b0c912677d0a0cb138c5ea7813b2a196fa32c53/psutil-3.0.1.win32-py3.4.exe","yanked":false},{"core-metadata":{"sha256":"74e797fa9b89ed58d856034a0b901f68c1d317dd41713af1d4e9bd93e2e19b2c"},"data-dist-info-metadata":{"sha256":"74e797fa9b89ed58d856034a0b901f68c1d317dd41713af1d4e9bd93e2e19b2c"},"filename":"psutil-3.1.0-cp26-none-win32.whl","hashes":{"sha256":"399916a016503c9ae99fd6aafbba4628b86ebdee52d67034e8c3f26e88a3504b"},"provenance":null,"requires-python":null,"size":87753,"upload-time":"2015-07-15T00:41:13.293562Z","url":"https://files.pythonhosted.org/packages/4e/0a/ab710541d02ff3ce4169a0922e4412952332991c4c39a0ea9df20e9279b0/psutil-3.1.0-cp26-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"74e797fa9b89ed58d856034a0b901f68c1d317dd41713af1d4e9bd93e2e19b2c"},"data-dist-info-metadata":{"sha256":"74e797fa9b89ed58d856034a0b901f68c1d317dd41713af1d4e9bd93e2e19b2c"},"filename":"psutil-3.1.0-cp27-none-win32.whl","hashes":{"sha256":"71af7a30eb6ed694a3624330334bd12da28bffdbe813f4b1080fadeb86f5970f"},"provenance":null,"requires-python":null,"size":87559,"upload-time":"2015-07-15T00:41:27.810047Z","url":"https://files.pythonhosted.org/packages/db/87/34b52811b755db94a7dd123e5c0f1d257885535f08f5f185c17810214d55/psutil-3.1.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"74e797fa9b89ed58d856034a0b901f68c1d317dd41713af1d4e9bd93e2e19b2c"},"data-dist-info-metadata":{"sha256":"74e797fa9b89ed58d856034a0b901f68c1d317dd41713af1d4e9bd93e2e19b2c"},"filename":"psutil-3.1.0-cp27-none-win_amd64.whl","hashes":{"sha256":"ada1cf324e6aba0affcb23c6fd959dae9f72de6ec135530788cbf17153d4fd3c"},"provenance":null,"requires-python":null,"size":90071,"upload-time":"2015-07-15T00:42:15.297263Z","url":"https://files.pythonhosted.org/packages/0b/7c/90869233a3e4056ddfdd1040d0e7722d3bb023c74b48bf10c09380c26eae/psutil-3.1.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"e5e4b772973543fffe4b2ffb66ad397284be77c3b6886db90b1fca8245b0d0b1"},"data-dist-info-metadata":{"sha256":"e5e4b772973543fffe4b2ffb66ad397284be77c3b6886db90b1fca8245b0d0b1"},"filename":"psutil-3.1.0-cp33-none-win32.whl","hashes":{"sha256":"0a313ebe14b9e277dfd151f4ad021012fb344dd51248e6de2aa1e7062d678541"},"provenance":null,"requires-python":null,"size":87569,"upload-time":"2015-07-15T00:41:43.765697Z","url":"https://files.pythonhosted.org/packages/f4/7c/56b718693e4c41b32af8bbe39160e8a3ea0ca12d3eece3dbbb8d4c046855/psutil-3.1.0-cp33-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"e5e4b772973543fffe4b2ffb66ad397284be77c3b6886db90b1fca8245b0d0b1"},"data-dist-info-metadata":{"sha256":"e5e4b772973543fffe4b2ffb66ad397284be77c3b6886db90b1fca8245b0d0b1"},"filename":"psutil-3.1.0-cp33-none-win_amd64.whl","hashes":{"sha256":"92a7f420bc97f899b5abab30392c23ba652304aec18415f2d1167da04dae9913"},"provenance":null,"requires-python":null,"size":89918,"upload-time":"2015-07-15T00:42:29.864336Z","url":"https://files.pythonhosted.org/packages/c7/5a/4046949e207b72b93540f3e19d699813fd35e290ccdf48080f332226b912/psutil-3.1.0-cp33-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"e5e4b772973543fffe4b2ffb66ad397284be77c3b6886db90b1fca8245b0d0b1"},"data-dist-info-metadata":{"sha256":"e5e4b772973543fffe4b2ffb66ad397284be77c3b6886db90b1fca8245b0d0b1"},"filename":"psutil-3.1.0-cp34-none-win32.whl","hashes":{"sha256":"8f5a0e859ae6dcc349914fb9ea0acc21cfd82a321d1c1b02d3d92c195f523ccd"},"provenance":null,"requires-python":null,"size":87592,"upload-time":"2015-07-15T00:42:00.678342Z","url":"https://files.pythonhosted.org/packages/09/34/09d53d29318a5fea88bd30d629595805064a0e3776e706eca2d79ceaebac/psutil-3.1.0-cp34-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"e5e4b772973543fffe4b2ffb66ad397284be77c3b6886db90b1fca8245b0d0b1"},"data-dist-info-metadata":{"sha256":"e5e4b772973543fffe4b2ffb66ad397284be77c3b6886db90b1fca8245b0d0b1"},"filename":"psutil-3.1.0-cp34-none-win_amd64.whl","hashes":{"sha256":"dcb4f208ec28fb72b35d1edf49aa51f2cc116b439aa40c4c415cbfe1fee54078"},"provenance":null,"requires-python":null,"size":89877,"upload-time":"2015-07-15T00:42:45.261251Z","url":"https://files.pythonhosted.org/packages/0b/f6/62592864eb064763989aa5830706034f9ad3c6ae2255fb7cae0c66b336a1/psutil-3.1.0-cp34-none-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.1.0.tar.gz","hashes":{"sha256":"4cdfeb2a328b6f8a2937f9b21f513c8aeda96dc076ecafda424f5c401dbad876"},"provenance":null,"requires-python":null,"size":246767,"upload-time":"2015-07-15T00:40:47.134419Z","url":"https://files.pythonhosted.org/packages/ce/d2/ab7f80718b4eafb2e474b8b410274d2c0d65341b963d730e653be9ed0ec8/psutil-3.1.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.1.0.win-amd64-py2.7.exe","hashes":{"sha256":"6570fb3ddde83597e11c062e20ab86210ff84a1fa97e54bc8bda05e4cd34670a"},"provenance":null,"requires-python":null,"size":326179,"upload-time":"2015-07-15T00:42:09.118162Z","url":"https://files.pythonhosted.org/packages/26/f5/c76bf7ef62736913146b0482879705d1d877c9334092a0739f2b3bbae162/psutil-3.1.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.1.0.win-amd64-py3.3.exe","hashes":{"sha256":"578a52f4b108857273d1e32de4d9bebf9b8f842f8c53ab3e242252cfc9bde295"},"provenance":null,"requires-python":null,"size":324540,"upload-time":"2015-07-15T00:42:24.223021Z","url":"https://files.pythonhosted.org/packages/00/51/2dc07e5618adb4a3676ab6c9c1759a3f268eda91808d58f45eb4dfd3d2c5/psutil-3.1.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.1.0.win-amd64-py3.4.exe","hashes":{"sha256":"2aea29ca2a5ea318155fb856f24e6c7563c8741ccfeec862f0fd9af0f2d8ae87"},"provenance":null,"requires-python":null,"size":324490,"upload-time":"2015-07-15T00:42:38.312454Z","url":"https://files.pythonhosted.org/packages/da/76/dcefdcf88fd51becaff6c1ec0ba7566ca654a207fe6d69662723c645e3b0/psutil-3.1.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.1.0.win32-py2.6.exe","hashes":{"sha256":"271f1b6fcb4861d1b0fc7f612b2abaacb36c0f878fcc2908f1cf673337c3472c"},"provenance":null,"requires-python":null,"size":296215,"upload-time":"2015-07-15T00:41:08.168047Z","url":"https://files.pythonhosted.org/packages/3f/c2/f7ec0a70bc58c1918f814001682cca30f4b168f5b46dce913220e625dee6/psutil-3.1.0.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.1.0.win32-py2.7.exe","hashes":{"sha256":"1bde1cea6b7f9bd66202feee289c282b076c00419dc6404db357b05125f4d692"},"provenance":null,"requires-python":null,"size":296024,"upload-time":"2015-07-15T00:41:21.733811Z","url":"https://files.pythonhosted.org/packages/56/2f/c97adcba8f119a23d3580a3a95939c1a37d37d514b304d90912585a85521/psutil-3.1.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.1.0.win32-py3.3.exe","hashes":{"sha256":"667fa795ca5ccde216b769fc8572398598f1e0e2619f78605df1a4c75a475174"},"provenance":null,"requires-python":null,"size":290958,"upload-time":"2015-07-15T00:41:37.038168Z","url":"https://files.pythonhosted.org/packages/37/89/a949b02d66d600c45230a3a622f0d5f491182dbbc96fff9ffbcd869431bd/psutil-3.1.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.1.0.win32-py3.4.exe","hashes":{"sha256":"8e637fe2a23fad4f4ea8ae11402b593e09fc587b86a6ced40b6bc1017be8d978"},"provenance":null,"requires-python":null,"size":290975,"upload-time":"2015-07-15T00:41:53.144204Z","url":"https://files.pythonhosted.org/packages/53/3f/249ff2e418313f30a6dcc4995966725f72b00493848ffddeb72b506e8e50/psutil-3.1.0.win32-py3.4.exe","yanked":false},{"core-metadata":{"sha256":"cb2a855afe0d2280f15e0904acf47929db7a22fcf8be740f65cecf46586d4ca5"},"data-dist-info-metadata":{"sha256":"cb2a855afe0d2280f15e0904acf47929db7a22fcf8be740f65cecf46586d4ca5"},"filename":"psutil-3.1.1-cp26-none-win32.whl","hashes":{"sha256":"13a6377cc8d2859f846058170830127822877e05229c4a43aea893cdcb504d65"},"provenance":null,"requires-python":null,"size":87749,"upload-time":"2015-07-15T12:34:56.810625Z","url":"https://files.pythonhosted.org/packages/e1/9e/721afc99b6fe467b47fa2cad6899acc19b45dee32d30b498dc731b6c09ef/psutil-3.1.1-cp26-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"cb2a855afe0d2280f15e0904acf47929db7a22fcf8be740f65cecf46586d4ca5"},"data-dist-info-metadata":{"sha256":"cb2a855afe0d2280f15e0904acf47929db7a22fcf8be740f65cecf46586d4ca5"},"filename":"psutil-3.1.1-cp27-none-win32.whl","hashes":{"sha256":"5b7228cb69fdaea5aeb901704f5ecd21b7846aa60c2c8d408f22573fcbaa7e6f"},"provenance":null,"requires-python":null,"size":87554,"upload-time":"2015-07-15T12:35:15.647007Z","url":"https://files.pythonhosted.org/packages/07/60/c88366202816ba42b3d8e93e793c14d1ac5e71be30dd53c2d0117c106eec/psutil-3.1.1-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"cb2a855afe0d2280f15e0904acf47929db7a22fcf8be740f65cecf46586d4ca5"},"data-dist-info-metadata":{"sha256":"cb2a855afe0d2280f15e0904acf47929db7a22fcf8be740f65cecf46586d4ca5"},"filename":"psutil-3.1.1-cp27-none-win_amd64.whl","hashes":{"sha256":"da7650e2f3fcf06419d5ad75123e6c68b9bf5ff2a6c91d4c77aaed8e6f444fc4"},"provenance":null,"requires-python":null,"size":90065,"upload-time":"2015-07-15T12:36:14.571710Z","url":"https://files.pythonhosted.org/packages/fa/5b/8834e22cc22b6b0e9c2c68e240ab69754bed7c4c5388fb65abfa716f4a67/psutil-3.1.1-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"19405a3f0d3e28d54384ce148334aeef185226341d9cc2a81c2d2933432a1de3"},"data-dist-info-metadata":{"sha256":"19405a3f0d3e28d54384ce148334aeef185226341d9cc2a81c2d2933432a1de3"},"filename":"psutil-3.1.1-cp33-none-win32.whl","hashes":{"sha256":"f3d68eb44ba49e24a18d6f7934463478294a49152f97fea2eefe1e1e1ee957f3"},"provenance":null,"requires-python":null,"size":87562,"upload-time":"2015-07-15T12:35:34.924344Z","url":"https://files.pythonhosted.org/packages/cb/96/0eb8eb289681364d2cda2a22a7d1abeb0196b321ab95694335dd178a5b35/psutil-3.1.1-cp33-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"19405a3f0d3e28d54384ce148334aeef185226341d9cc2a81c2d2933432a1de3"},"data-dist-info-metadata":{"sha256":"19405a3f0d3e28d54384ce148334aeef185226341d9cc2a81c2d2933432a1de3"},"filename":"psutil-3.1.1-cp33-none-win_amd64.whl","hashes":{"sha256":"f9be0ae975b55a3b5d5a8b769560096d76184b60a56c6e88ff6b7ebecf1bc684"},"provenance":null,"requires-python":null,"size":89916,"upload-time":"2015-07-15T12:36:34.374903Z","url":"https://files.pythonhosted.org/packages/eb/13/a38bc1e0ac6f7c42dddd9c17a206877befb822ba3af9c3b0dab9e85911a6/psutil-3.1.1-cp33-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"19405a3f0d3e28d54384ce148334aeef185226341d9cc2a81c2d2933432a1de3"},"data-dist-info-metadata":{"sha256":"19405a3f0d3e28d54384ce148334aeef185226341d9cc2a81c2d2933432a1de3"},"filename":"psutil-3.1.1-cp34-none-win32.whl","hashes":{"sha256":"8f25aad572bde88d5ee0b3a11a75ff2ae3c8b0a334c4128d6f8eb4fc95172734"},"provenance":null,"requires-python":null,"size":87574,"upload-time":"2015-07-15T12:35:55.507642Z","url":"https://files.pythonhosted.org/packages/02/10/439ec497e3e38a8c493d0c67c56e23d106b85c8b9f616e8df9ec6ce1e606/psutil-3.1.1-cp34-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"19405a3f0d3e28d54384ce148334aeef185226341d9cc2a81c2d2933432a1de3"},"data-dist-info-metadata":{"sha256":"19405a3f0d3e28d54384ce148334aeef185226341d9cc2a81c2d2933432a1de3"},"filename":"psutil-3.1.1-cp34-none-win_amd64.whl","hashes":{"sha256":"4be182c273758dcdbd30827fdeecd889e27cb6a30238798e91bddeebc29cdc4f"},"provenance":null,"requires-python":null,"size":89865,"upload-time":"2015-07-15T12:36:55.016343Z","url":"https://files.pythonhosted.org/packages/ec/2e/8d98579399bc1979904455df182a063dd584b285ee8c141f3c94e7814c47/psutil-3.1.1-cp34-none-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.1.1.tar.gz","hashes":{"sha256":"d3290bd4a027fa0b3a2e2ee87728056fe49d4112640e2b8c2ea4dd94ba0cf057"},"provenance":null,"requires-python":null,"size":247284,"upload-time":"2015-07-15T12:33:47.020532Z","url":"https://files.pythonhosted.org/packages/8d/b3/954de176aa8e3a7782bae52ce938f24726c2c68d0f4c60d159271b6b293d/psutil-3.1.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.1.1.win-amd64-py2.7.exe","hashes":{"sha256":"9c4fd3cc19bbc04eaa7ef3c61e3db26a41ac5e056f770977211d4569d0bf0086"},"provenance":null,"requires-python":null,"size":326261,"upload-time":"2015-07-15T12:36:07.064169Z","url":"https://files.pythonhosted.org/packages/e8/ac/7fb95ccc69ced76d0920e411b2fdfd3d38398c4bce53ec1dae92800df88a/psutil-3.1.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.1.1.win-amd64-py3.3.exe","hashes":{"sha256":"e7cc26f661c9eaa9b32d0543dd7838daea72aad6e9f02fe73715ffd0dcb65170"},"provenance":null,"requires-python":null,"size":324622,"upload-time":"2015-07-15T12:36:27.201153Z","url":"https://files.pythonhosted.org/packages/2f/84/ae41f6bb61d4a93399c621218f99b761171a69a0c9163b9a72db1d53d62a/psutil-3.1.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.1.1.win-amd64-py3.4.exe","hashes":{"sha256":"46cbfd86d6762e63c7df4ab0df889f6f2fffa9b5781ea3fc0431237f2a408382"},"provenance":null,"requires-python":null,"size":324571,"upload-time":"2015-07-15T12:36:47.165677Z","url":"https://files.pythonhosted.org/packages/88/d4/ca15a913ab43222e308774845317544e765718a9e56bd4efe5b3cedf1fbd/psutil-3.1.1.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.1.1.win32-py2.6.exe","hashes":{"sha256":"9efbd578d2f400dfe0ecab123b58d8af105854fdbb6222f841151e010e820b75"},"provenance":null,"requires-python":null,"size":296306,"upload-time":"2015-07-15T12:34:49.442980Z","url":"https://files.pythonhosted.org/packages/0c/f6/e81385c7ec989157eb68688a64a69c5a7477ff93d544893a9e1f251588b1/psutil-3.1.1.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.1.1.win32-py2.7.exe","hashes":{"sha256":"e0065e7cade4ac5ac70411674bc32326dee8d11c44469012a2b5164bf6dea97a"},"provenance":null,"requires-python":null,"size":296106,"upload-time":"2015-07-15T12:35:08.277861Z","url":"https://files.pythonhosted.org/packages/6b/fe/51596968f5a6a0970d9424021989f542d5aa715fe21d1a9c6bbbb0e377a9/psutil-3.1.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.1.1.win32-py3.3.exe","hashes":{"sha256":"c8ab17e07ea4907d2f9129254e82b6765ae08e61f0ce6dc8e2fc1faf145b166c"},"provenance":null,"requires-python":null,"size":291039,"upload-time":"2015-07-15T12:35:27.250487Z","url":"https://files.pythonhosted.org/packages/83/49/ff116fb9981ef04a5aed1c091ace117c214ed752d37be267ea4e2f28efad/psutil-3.1.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.1.1.win32-py3.4.exe","hashes":{"sha256":"3003d8be6e86eb6beb990863a88950f9b9fe53ccaae92edcd8efcd152d7451ea"},"provenance":null,"requires-python":null,"size":291056,"upload-time":"2015-07-15T12:35:47.023807Z","url":"https://files.pythonhosted.org/packages/7b/cd/accf3d7e37006bffe7a569e4fc587eb686d275a19a4e8a37a12930a1e2db/psutil-3.1.1.win32-py3.4.exe","yanked":false},{"core-metadata":{"sha256":"e8c3ed176a6ecb754b5289ef149667a6343dc0380cb33db608d4d556be0650f4"},"data-dist-info-metadata":{"sha256":"e8c3ed176a6ecb754b5289ef149667a6343dc0380cb33db608d4d556be0650f4"},"filename":"psutil-3.2.0-cp27-none-win32.whl","hashes":{"sha256":"1493041336a591f22c77bcb815a399faf9bdac32f79f4de354eda3507a0d6d6b"},"provenance":null,"requires-python":null,"size":88077,"upload-time":"2015-09-02T11:56:33.437905Z","url":"https://files.pythonhosted.org/packages/5f/fc/5f317fd548909b1bbb111d462c072faf8af3938268f3e7dd3ab2f9181461/psutil-3.2.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"e8c3ed176a6ecb754b5289ef149667a6343dc0380cb33db608d4d556be0650f4"},"data-dist-info-metadata":{"sha256":"e8c3ed176a6ecb754b5289ef149667a6343dc0380cb33db608d4d556be0650f4"},"filename":"psutil-3.2.0-cp27-none-win_amd64.whl","hashes":{"sha256":"3744ee760dff697f45731a71e7902514aa043c99800cc8fabeb6bebc9dad973d"},"provenance":null,"requires-python":null,"size":90526,"upload-time":"2015-09-02T11:57:23.745435Z","url":"https://files.pythonhosted.org/packages/84/6c/7efbe64b42748125e7113a90e48c0da9859b7f0363ac85ca5617decbafee/psutil-3.2.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"86ea3a30e605b64b3e8a93353520057387c9224bf456a532eb4f25799e27b050"},"data-dist-info-metadata":{"sha256":"86ea3a30e605b64b3e8a93353520057387c9224bf456a532eb4f25799e27b050"},"filename":"psutil-3.2.0-cp33-none-win32.whl","hashes":{"sha256":"8836f77d2c4ae2935431ca66e445435b87b53b4db637fcceb438b78843239210"},"provenance":null,"requires-python":null,"size":88080,"upload-time":"2015-09-02T11:56:48.988091Z","url":"https://files.pythonhosted.org/packages/64/8e/0a06028a1ac093402885febf2aeb18093f1d28ae2110c7eb10b43e7554c1/psutil-3.2.0-cp33-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"86ea3a30e605b64b3e8a93353520057387c9224bf456a532eb4f25799e27b050"},"data-dist-info-metadata":{"sha256":"86ea3a30e605b64b3e8a93353520057387c9224bf456a532eb4f25799e27b050"},"filename":"psutil-3.2.0-cp33-none-win_amd64.whl","hashes":{"sha256":"b16eb62d9c21efaa2c9ac8a9f8b23bb7a695cb799b597edf4b1289ce8e6973ac"},"provenance":null,"requires-python":null,"size":90372,"upload-time":"2015-09-02T12:03:01.254324Z","url":"https://files.pythonhosted.org/packages/cd/29/a8383040200a3ebe0e985f54f35691cc078a1deb632abb5340d3deb5b7b7/psutil-3.2.0-cp33-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"86ea3a30e605b64b3e8a93353520057387c9224bf456a532eb4f25799e27b050"},"data-dist-info-metadata":{"sha256":"86ea3a30e605b64b3e8a93353520057387c9224bf456a532eb4f25799e27b050"},"filename":"psutil-3.2.0-cp34-none-win32.whl","hashes":{"sha256":"0ac1d68ab3c5a65641cbbb23d19deda466f73226f9d967f91436851995281777"},"provenance":null,"requires-python":null,"size":88096,"upload-time":"2015-09-02T11:57:08.035697Z","url":"https://files.pythonhosted.org/packages/07/c3/76a50982a82c0e9d93d9614a0cd06644c1d3406c9bb80a43f95abdd4ab97/psutil-3.2.0-cp34-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"86ea3a30e605b64b3e8a93353520057387c9224bf456a532eb4f25799e27b050"},"data-dist-info-metadata":{"sha256":"86ea3a30e605b64b3e8a93353520057387c9224bf456a532eb4f25799e27b050"},"filename":"psutil-3.2.0-cp34-none-win_amd64.whl","hashes":{"sha256":"0b26ef262fe2d10185ab562cd0530af7f6d9a6744c631c44e64be94796f4ba2d"},"provenance":null,"requires-python":null,"size":90354,"upload-time":"2015-09-02T12:07:12.010031Z","url":"https://files.pythonhosted.org/packages/58/2d/8b7abb9b6f8956d9a6dfc3b1dffce27efab8c7c0497a6366e7fee444ae53/psutil-3.2.0-cp34-none-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.0.tar.gz","hashes":{"sha256":"06f9d255f8b12a6a04aa2b468ec453c539f54a464d110b3458c32b0152a5c943"},"provenance":null,"requires-python":null,"size":251988,"upload-time":"2015-09-02T11:59:23.849318Z","url":"https://files.pythonhosted.org/packages/1d/3a/d396274e6f086e342dd43401d4012973af98c00b3aabdb5cc4a432df660e/psutil-3.2.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.0.win-amd64-py2.7.exe","hashes":{"sha256":"71fd8712715f8e6acc5bee5719a83a61a396067cf2bfb15b4d8f1f2955648637"},"provenance":null,"requires-python":null,"size":326895,"upload-time":"2015-09-02T11:57:17.538160Z","url":"https://files.pythonhosted.org/packages/e1/49/990073ab7e010965a7d0df5e48181d07c212edd7fafb890feda664ea9b3c/psutil-3.2.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.0.win32-py2.7.exe","hashes":{"sha256":"2a8b5878d4e787d81a1eeddcc09ff28d501a3ceb320c7fffa7e207da5d61d01c"},"provenance":null,"requires-python":null,"size":296802,"upload-time":"2015-09-02T11:56:26.879142Z","url":"https://files.pythonhosted.org/packages/0b/c5/d6ad511c3c17afa9837d08fc26d76e85dc83ebc304c6c7bec3970e74f240/psutil-3.2.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.0.win32-py3.3.exe","hashes":{"sha256":"adfc63ceede4e8f6bf21e4bdf6fc91f70f9612ec2b1bf9ad306828909bb71c52"},"provenance":null,"requires-python":null,"size":291738,"upload-time":"2015-09-02T11:56:41.826609Z","url":"https://files.pythonhosted.org/packages/e3/35/81842c6c4366d19c87d1a1fb4ad4e4d22a18aa9facaea8b6f12ccd4c1212/psutil-3.2.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.0.win32-py3.4.exe","hashes":{"sha256":"4098e0ed7930003ef15feb852e64f73180c17a651c4170fb5573f8c44622d068"},"provenance":null,"requires-python":null,"size":291748,"upload-time":"2015-09-02T11:56:58.479934Z","url":"https://files.pythonhosted.org/packages/e9/e8/5b432a0490328cfff86605b574d2aa31b1fac4e61587dff5a7f76d4cb95e/psutil-3.2.0.win32-py3.4.exe","yanked":false},{"core-metadata":{"sha256":"c06dad4d23110956ffef0ab97d2845cb62aba9d194019b18cf8e24cbd88c0781"},"data-dist-info-metadata":{"sha256":"c06dad4d23110956ffef0ab97d2845cb62aba9d194019b18cf8e24cbd88c0781"},"filename":"psutil-3.2.1-cp26-none-win32.whl","hashes":{"sha256":"a77230ecd6f42d0b549f8eb6aa105f14e4bc5908c754d6e10ff979c900934481"},"provenance":null,"requires-python":null,"size":88306,"upload-time":"2015-09-03T15:37:28.878155Z","url":"https://files.pythonhosted.org/packages/37/46/f348f7728dea66436abdfc9fa14ef017e0148c6bca08a822ee4dd7cb6d75/psutil-3.2.1-cp26-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"c06dad4d23110956ffef0ab97d2845cb62aba9d194019b18cf8e24cbd88c0781"},"data-dist-info-metadata":{"sha256":"c06dad4d23110956ffef0ab97d2845cb62aba9d194019b18cf8e24cbd88c0781"},"filename":"psutil-3.2.1-cp27-none-win32.whl","hashes":{"sha256":"9453b8ceb249d4d9ddc69153729761be340dfef9c99509390e4fb0f1fcbb3853"},"provenance":null,"requires-python":null,"size":88111,"upload-time":"2015-09-03T15:30:11.278302Z","url":"https://files.pythonhosted.org/packages/ba/27/f55ca7d15af50e731e9bbbff9b22fc31a40b786c02f85d173568e5084152/psutil-3.2.1-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"c06dad4d23110956ffef0ab97d2845cb62aba9d194019b18cf8e24cbd88c0781"},"data-dist-info-metadata":{"sha256":"c06dad4d23110956ffef0ab97d2845cb62aba9d194019b18cf8e24cbd88c0781"},"filename":"psutil-3.2.1-cp27-none-win_amd64.whl","hashes":{"sha256":"014714beed46a66370834cebe0bbb53799bddc164f7f0149a4a70e2051f7bc1a"},"provenance":null,"requires-python":null,"size":90561,"upload-time":"2015-09-03T15:35:05.710499Z","url":"https://files.pythonhosted.org/packages/6b/ac/5da840018ce300a258925d4535a55a32b75236d5d777a8de6c3de18e71f3/psutil-3.2.1-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"51cd3e3b33e3b710f799f247a23955f942e6e11d4882ee5e3f3af539eee619c8"},"data-dist-info-metadata":{"sha256":"51cd3e3b33e3b710f799f247a23955f942e6e11d4882ee5e3f3af539eee619c8"},"filename":"psutil-3.2.1-cp33-none-win32.whl","hashes":{"sha256":"5c0daf045fd7d7f105863a5f9508d1698559ebbdfd70d7d8b6fe6fedde575735"},"provenance":null,"requires-python":null,"size":88117,"upload-time":"2015-09-03T15:32:19.746087Z","url":"https://files.pythonhosted.org/packages/c7/96/3ae14e4bf81f18f404bb5285fcda28e50ae6df87e91ad6bf45a9a4f51ac3/psutil-3.2.1-cp33-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"51cd3e3b33e3b710f799f247a23955f942e6e11d4882ee5e3f3af539eee619c8"},"data-dist-info-metadata":{"sha256":"51cd3e3b33e3b710f799f247a23955f942e6e11d4882ee5e3f3af539eee619c8"},"filename":"psutil-3.2.1-cp33-none-win_amd64.whl","hashes":{"sha256":"452622592564cd67f86808c8176720c1443d43e248cfd242d71cff559ed1424c"},"provenance":null,"requires-python":null,"size":90410,"upload-time":"2015-09-03T15:36:13.351055Z","url":"https://files.pythonhosted.org/packages/76/62/fe6f705cb331be5fcc97b268987527dcdb3f3aa104bf830b0ec8bf1e2ad4/psutil-3.2.1-cp33-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"51cd3e3b33e3b710f799f247a23955f942e6e11d4882ee5e3f3af539eee619c8"},"data-dist-info-metadata":{"sha256":"51cd3e3b33e3b710f799f247a23955f942e6e11d4882ee5e3f3af539eee619c8"},"filename":"psutil-3.2.1-cp34-none-win32.whl","hashes":{"sha256":"96379bee09d4c6b4d57d72cb7347dbc51b6847977f2fad01cdfefef3b53e44e3"},"provenance":null,"requires-python":null,"size":88131,"upload-time":"2015-09-03T15:33:36.571683Z","url":"https://files.pythonhosted.org/packages/31/ec/1a54f23c767e27dac09d2372f3522f88ef34f3a0ddd44c16122970259f6f/psutil-3.2.1-cp34-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"51cd3e3b33e3b710f799f247a23955f942e6e11d4882ee5e3f3af539eee619c8"},"data-dist-info-metadata":{"sha256":"51cd3e3b33e3b710f799f247a23955f942e6e11d4882ee5e3f3af539eee619c8"},"filename":"psutil-3.2.1-cp34-none-win_amd64.whl","hashes":{"sha256":"7a7f4f2ed6d2835c48c24a81b251ba4f9b21f6bba2323291f8205c9ecb6f659d"},"provenance":null,"requires-python":null,"size":90387,"upload-time":"2015-09-03T15:36:29.262691Z","url":"https://files.pythonhosted.org/packages/2e/10/f1590ae942a6b8dd2bdeef6088e30e89b30161a264881b14134f3c4a3a0e/psutil-3.2.1-cp34-none-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.1.tar.gz","hashes":{"sha256":"7f6bea8bfe2e5cfffd0f411aa316e837daadced1893b44254bb9a38a654340f7"},"provenance":null,"requires-python":null,"size":251653,"upload-time":"2015-09-03T15:30:34.118573Z","url":"https://files.pythonhosted.org/packages/cd/5f/4fae1036903c01929c48ded6800a8705106ee20f9e39e3f2ad5d1824e210/psutil-3.2.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.1.win-amd64-py2.7.exe","hashes":{"sha256":"e9a8a44f3847a0e20a54d321ed62de4e9cee5bc4e880e25fe88ae20cfa4e32b2"},"provenance":null,"requires-python":null,"size":327122,"upload-time":"2015-09-03T15:34:58.714500Z","url":"https://files.pythonhosted.org/packages/c5/af/4ab069ba93a037a4acf9bb84248daa44204a46687abc6a9f3a82ad8c5ee2/psutil-3.2.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.1.win-amd64-py3.3.exe","hashes":{"sha256":"5a62c38852de4513f1816b9c431a94f02531619c1edc60a2cc163c8754f51c50"},"provenance":null,"requires-python":null,"size":325481,"upload-time":"2015-09-03T15:36:01.424492Z","url":"https://files.pythonhosted.org/packages/32/87/82e449ff9573dde3c78685b64ac3d17b5d19db11e976eab27c4dc5dca942/psutil-3.2.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.1.win-amd64-py3.4.exe","hashes":{"sha256":"da898c0708b99b3892bfb7d5caebb447d14d03c7a655c55e484eb5fcc741c3ca"},"provenance":null,"requires-python":null,"size":325465,"upload-time":"2015-09-03T15:36:22.916532Z","url":"https://files.pythonhosted.org/packages/4b/11/f18b29033a0b383e67f664576eca59fbe8552a9fd97f9f22d6d0ff1c4951/psutil-3.2.1.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.1.win32-py2.6.exe","hashes":{"sha256":"0e5fe3d50f9f8d9a5216cfa23f56890aa0c6a6163869434001f4f2ba463dace5"},"provenance":null,"requires-python":null,"size":297224,"upload-time":"2015-09-03T15:37:22.791161Z","url":"https://files.pythonhosted.org/packages/c0/bb/ed28c191c4d9f27b60d9ea6bd7774b44d78778f0b1fb507ee1e789a490d7/psutil-3.2.1.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.1.win32-py2.7.exe","hashes":{"sha256":"9fb6f11bdd3fdbe1e611ae02b3ad3dff8f70ef6eaa694d13e8ad0906fd7a7261"},"provenance":null,"requires-python":null,"size":297025,"upload-time":"2015-09-03T15:30:14.327211Z","url":"https://files.pythonhosted.org/packages/9a/c1/075598067efefe25f6a2b0cf1b3eb896322597ee64ba097cc6611b89ada7/psutil-3.2.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.1.win32-py3.3.exe","hashes":{"sha256":"605cc7dbe2170e89f2f6709cf1577c8a02f89951fe4a0eb48e72530f605141ca"},"provenance":null,"requires-python":null,"size":291964,"upload-time":"2015-09-03T15:32:12.890526Z","url":"https://files.pythonhosted.org/packages/e0/ce/ff1db37cbdf6b3071e89afe4fced6f73d2cdf9c3a87696e7754d207e0ec5/psutil-3.2.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.1.win32-py3.4.exe","hashes":{"sha256":"0cd127239f527eae6c0f778dd41bb3ced84e6049b919713022c9b72e5f22a1c1"},"provenance":null,"requires-python":null,"size":291976,"upload-time":"2015-09-03T15:33:29.509014Z","url":"https://files.pythonhosted.org/packages/5d/a0/d624d4f5660a476821fe0d920a4c2e995a23151928b15cc3383379228f15/psutil-3.2.1.win32-py3.4.exe","yanked":false},{"core-metadata":{"sha256":"7115f58a06724cf8a31f365d5561fc59f6d68d8e55e20b3d75bf373a45007f7d"},"data-dist-info-metadata":{"sha256":"7115f58a06724cf8a31f365d5561fc59f6d68d8e55e20b3d75bf373a45007f7d"},"filename":"psutil-3.2.2-cp26-none-win32.whl","hashes":{"sha256":"5a8ce70327c0da578a31ebbf0042671ed9be6f4b6b022c02f03302b690074966"},"provenance":null,"requires-python":null,"size":88377,"upload-time":"2015-10-04T16:38:11.705751Z","url":"https://files.pythonhosted.org/packages/4d/af/5b8c2471ea942a4b6ee85706e9279284ae9dc86ee30b6f97db2d84a95433/psutil-3.2.2-cp26-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"7115f58a06724cf8a31f365d5561fc59f6d68d8e55e20b3d75bf373a45007f7d"},"data-dist-info-metadata":{"sha256":"7115f58a06724cf8a31f365d5561fc59f6d68d8e55e20b3d75bf373a45007f7d"},"filename":"psutil-3.2.2-cp27-none-win32.whl","hashes":{"sha256":"6c5809582d3d165511d2319401bd0f6c0e825d7853e49da59027c1fb8aa8f897"},"provenance":null,"requires-python":null,"size":88183,"upload-time":"2015-10-04T16:38:39.878475Z","url":"https://files.pythonhosted.org/packages/63/1e/a510f3f310b5f530336fbc708fb1456bf3e49e3b3d85c31d151b6e389c4f/psutil-3.2.2-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"7115f58a06724cf8a31f365d5561fc59f6d68d8e55e20b3d75bf373a45007f7d"},"data-dist-info-metadata":{"sha256":"7115f58a06724cf8a31f365d5561fc59f6d68d8e55e20b3d75bf373a45007f7d"},"filename":"psutil-3.2.2-cp27-none-win_amd64.whl","hashes":{"sha256":"37f1cc8fc7586cc930ea3737533d6d79c1f761d577fd1bb1bb5798ccd1543b53"},"provenance":null,"requires-python":null,"size":90637,"upload-time":"2015-10-04T16:40:05.379646Z","url":"https://files.pythonhosted.org/packages/e7/b7/f04d64a692159733ed383b4638abd9d3dc4538d4aacb5e193af02a3840a2/psutil-3.2.2-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"93db84a195bf6aa1de88b86ddb7fc98d3600b0ecf0ff4ab69e447ca200c3a4b0"},"data-dist-info-metadata":{"sha256":"93db84a195bf6aa1de88b86ddb7fc98d3600b0ecf0ff4ab69e447ca200c3a4b0"},"filename":"psutil-3.2.2-cp33-none-win32.whl","hashes":{"sha256":"f8fb145f8fa9e223696ff2f99924ea42538f3ad6b9738707292d840acbde528f"},"provenance":null,"requires-python":null,"size":88189,"upload-time":"2015-10-04T16:39:00.887284Z","url":"https://files.pythonhosted.org/packages/77/f3/6b3742040b634692393faf3a81e6c0e40366c22bc338ad3fc62ed21b157a/psutil-3.2.2-cp33-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"93db84a195bf6aa1de88b86ddb7fc98d3600b0ecf0ff4ab69e447ca200c3a4b0"},"data-dist-info-metadata":{"sha256":"93db84a195bf6aa1de88b86ddb7fc98d3600b0ecf0ff4ab69e447ca200c3a4b0"},"filename":"psutil-3.2.2-cp33-none-win_amd64.whl","hashes":{"sha256":"65c78ba625cf9761d5966603838cc959f396bd03536c480db69f8cf37bdf9994"},"provenance":null,"requires-python":null,"size":90471,"upload-time":"2015-10-04T16:40:27.009383Z","url":"https://files.pythonhosted.org/packages/59/33/3ccdbec4ef1452758ba80f711af46736717f63d73786744d6251afb68624/psutil-3.2.2-cp33-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"93db84a195bf6aa1de88b86ddb7fc98d3600b0ecf0ff4ab69e447ca200c3a4b0"},"data-dist-info-metadata":{"sha256":"93db84a195bf6aa1de88b86ddb7fc98d3600b0ecf0ff4ab69e447ca200c3a4b0"},"filename":"psutil-3.2.2-cp34-none-win32.whl","hashes":{"sha256":"3df8d3e32e2b4f7c2ea91014294844670eddb125ba76c24152c0a155a1f73b5b"},"provenance":null,"requires-python":null,"size":88202,"upload-time":"2015-10-04T16:39:39.989654Z","url":"https://files.pythonhosted.org/packages/62/6e/ee3597f32c650f744359e57fd18bcede773dd7465d392dabbb008bc79b48/psutil-3.2.2-cp34-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"93db84a195bf6aa1de88b86ddb7fc98d3600b0ecf0ff4ab69e447ca200c3a4b0"},"data-dist-info-metadata":{"sha256":"93db84a195bf6aa1de88b86ddb7fc98d3600b0ecf0ff4ab69e447ca200c3a4b0"},"filename":"psutil-3.2.2-cp34-none-win_amd64.whl","hashes":{"sha256":"e321d3f029268bc8442a7ff214da43fe91041924898f5e23d88bfda7ecb81acc"},"provenance":null,"requires-python":null,"size":90463,"upload-time":"2015-10-04T16:40:56.120875Z","url":"https://files.pythonhosted.org/packages/ed/fe/f31bb708dfdecfbc59b946ecb9ee3379fe7a8183c37ea6c43d6f4da5117d/psutil-3.2.2-cp34-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"826e3762fc7145422d64b4a51bb0f8ca6b3a05baed01f7ef3c10b391a0a6c865"},"data-dist-info-metadata":{"sha256":"826e3762fc7145422d64b4a51bb0f8ca6b3a05baed01f7ef3c10b391a0a6c865"},"filename":"psutil-3.2.2-cp35-none-win32.whl","hashes":{"sha256":"76c68c9005a2aa983fce440ef98b66e6f200f740f52064a90fdcc30d11771bc2"},"provenance":null,"requires-python":null,"size":90363,"upload-time":"2015-11-06T10:48:59.454097Z","url":"https://files.pythonhosted.org/packages/cb/79/fcedcf009ab9f8c605f2f345b1797b72134ecc6c9c9f786575e34b3471bc/psutil-3.2.2-cp35-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"826e3762fc7145422d64b4a51bb0f8ca6b3a05baed01f7ef3c10b391a0a6c865"},"data-dist-info-metadata":{"sha256":"826e3762fc7145422d64b4a51bb0f8ca6b3a05baed01f7ef3c10b391a0a6c865"},"filename":"psutil-3.2.2-cp35-none-win_amd64.whl","hashes":{"sha256":"d3ac8ad04a509819d7b5d58e453749a3ceb37253267dac6e8856ea7953e22ca0"},"provenance":null,"requires-python":null,"size":93288,"upload-time":"2015-11-06T10:50:14.699381Z","url":"https://files.pythonhosted.org/packages/03/c5/15e44d590afc788228e93cdacf55f98828f326de985242bbf03b3545b129/psutil-3.2.2-cp35-none-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.2.tar.gz","hashes":{"sha256":"f9d848e5bd475ffe7fa3ab1c20d249807e648568af64bb0058412296ec990a0c"},"provenance":null,"requires-python":null,"size":253502,"upload-time":"2015-10-04T16:39:43.138939Z","url":"https://files.pythonhosted.org/packages/dc/b2/ab65a2209b996c891209b8a7444a0c825125fba850efaec07b95bccb3ff5/psutil-3.2.2.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.2.win-amd64-py2.7.exe","hashes":{"sha256":"1fca15005063b401cbf94cebe3c01ef6ba3d86ba563730d5d5d6be962a637cf4"},"provenance":null,"requires-python":null,"size":327220,"upload-time":"2015-10-04T16:39:56.198669Z","url":"https://files.pythonhosted.org/packages/01/7c/47b7ac498c9dd6ac9f9b4489d3bd9cea8158e774592661a7c956a815dc78/psutil-3.2.2.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.2.win-amd64-py3.3.exe","hashes":{"sha256":"01a1f55819019ad13c288c41cb233e17a6ee648baf19591a70f6c2c2295dde6c"},"provenance":null,"requires-python":null,"size":325577,"upload-time":"2015-10-04T16:40:17.960471Z","url":"https://files.pythonhosted.org/packages/5d/89/b3bfca24d038b16af09055912e3a5bc35347ebeb0e6af322959b68dbf237/psutil-3.2.2.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.2.win-amd64-py3.5.exe","hashes":{"sha256":"9afc68c02717fb4416f91b3c2da4c407756f683804383d1499cdc9e1512e7942"},"provenance":null,"requires-python":null,"size":242452,"upload-time":"2015-11-06T10:49:50.288504Z","url":"https://files.pythonhosted.org/packages/1d/0b/9582cadcba005f4eb0207107baeae7eb2382431fea5e4bc0ea7ce683430f/psutil-3.2.2.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.2.win32-py2.6.exe","hashes":{"sha256":"c7c516a83d072d1a375ed3a0b5a1b1b9307ad839011e8d30aa16b0f932f6c481"},"provenance":null,"requires-python":null,"size":297317,"upload-time":"2015-10-04T16:38:04.476936Z","url":"https://files.pythonhosted.org/packages/66/e4/bb85391bb46b607be0578e0a091bc064daeb2d1c2e80aa2dab89260dff00/psutil-3.2.2.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.2.win32-py3.3.exe","hashes":{"sha256":"5168f99f065f6116ad1e8529bd5dd5309815198c6250c9180ff6058c6a3641d9"},"provenance":null,"requires-python":null,"size":292060,"upload-time":"2015-10-04T16:38:46.104177Z","url":"https://files.pythonhosted.org/packages/8e/86/8f1e1c0ffc0530dca5e71777f04fb90833c92d3ffc1d075b8b546874eae5/psutil-3.2.2.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.2.win32-py3.4.exe","hashes":{"sha256":"d5c96aa591ba711dfdadb1ab7a0adf08c5e637644a21437f3a39a9e427aa969b"},"provenance":null,"requires-python":null,"size":292515,"upload-time":"2015-11-06T01:39:51.263901Z","url":"https://files.pythonhosted.org/packages/28/fc/08f1098976de5416cd15967d88e03297569e9a34ca875e7dee38ff9150f0/psutil-3.2.2.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.2.2.win32-py3.5.exe","hashes":{"sha256":"f0e88f94f9822fd34bcd927aba4bb606bcb0ad0dda1543c9333e8175d5c05822"},"provenance":null,"requires-python":null,"size":232367,"upload-time":"2015-11-06T10:48:42.065794Z","url":"https://files.pythonhosted.org/packages/1d/b8/b725f9bd884f75ce141a9e871a79b2bdd1b5f31e814fc4e396d9ff7c98a2/psutil-3.2.2.win32-py3.5.exe","yanked":false},{"core-metadata":{"sha256":"5977c4a46aa367cb1f96ab1a6c70f8c803d8194d6a6440a31450ac202114431f"},"data-dist-info-metadata":{"sha256":"5977c4a46aa367cb1f96ab1a6c70f8c803d8194d6a6440a31450ac202114431f"},"filename":"psutil-3.3.0-cp26-none-win32.whl","hashes":{"sha256":"584f0b29fcc5d523b433cb8918b2fc74d67e30ee0b44a95baf031528f424619f"},"provenance":null,"requires-python":null,"size":90099,"upload-time":"2015-11-25T18:49:50.211423Z","url":"https://files.pythonhosted.org/packages/91/75/c20c3b9f4d3feb3436d607f498744e46dd28b265b8a72509812322198c7c/psutil-3.3.0-cp26-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"5977c4a46aa367cb1f96ab1a6c70f8c803d8194d6a6440a31450ac202114431f"},"data-dist-info-metadata":{"sha256":"5977c4a46aa367cb1f96ab1a6c70f8c803d8194d6a6440a31450ac202114431f"},"filename":"psutil-3.3.0-cp26-none-win_amd64.whl","hashes":{"sha256":"28ca0b6e9d99aa8dc286e8747a4471362b69812a25291de29b6a8d70a1545a0d"},"provenance":null,"requires-python":null,"size":92645,"upload-time":"2015-11-25T18:50:32.375134Z","url":"https://files.pythonhosted.org/packages/6a/d1/0ce316e4346bcae9dd23911366d894eda65875b88ff447ec8f0402ce556b/psutil-3.3.0-cp26-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"5977c4a46aa367cb1f96ab1a6c70f8c803d8194d6a6440a31450ac202114431f"},"data-dist-info-metadata":{"sha256":"5977c4a46aa367cb1f96ab1a6c70f8c803d8194d6a6440a31450ac202114431f"},"filename":"psutil-3.3.0-cp27-none-win32.whl","hashes":{"sha256":"167ad5fff52a672c4ddc1c1a0b25146d6813ebb08a9aab0a3ac45f8a5b669c3b"},"provenance":null,"requires-python":null,"size":90131,"upload-time":"2015-11-25T01:20:43.015358Z","url":"https://files.pythonhosted.org/packages/91/73/1f55b4a19db535759fec5fdbdd0653d7192336557078e3ac9085d7d77cd1/psutil-3.3.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"5977c4a46aa367cb1f96ab1a6c70f8c803d8194d6a6440a31450ac202114431f"},"data-dist-info-metadata":{"sha256":"5977c4a46aa367cb1f96ab1a6c70f8c803d8194d6a6440a31450ac202114431f"},"filename":"psutil-3.3.0-cp27-none-win_amd64.whl","hashes":{"sha256":"e6dea6173a988727bb223d3497349ad5cdef5c0b282eff2d83e5f9065c53f85f"},"provenance":null,"requires-python":null,"size":92586,"upload-time":"2015-11-25T01:22:19.444688Z","url":"https://files.pythonhosted.org/packages/a2/ab/d15a34c6b9090d58601541f8f5564f5b48d01e82f56e07593be969d529e7/psutil-3.3.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"60f0487057f863a2f4cee79fc120f02b60168ae677fac14ba9ccc63127b37514"},"data-dist-info-metadata":{"sha256":"60f0487057f863a2f4cee79fc120f02b60168ae677fac14ba9ccc63127b37514"},"filename":"psutil-3.3.0-cp33-none-win32.whl","hashes":{"sha256":"2af5e0a4aad66049955d0734aa4e3dc8caa17a9eaf8b4c1a27a5f1ee6e40f6fc"},"provenance":null,"requires-python":null,"size":90141,"upload-time":"2015-11-25T01:21:10.374023Z","url":"https://files.pythonhosted.org/packages/c5/1f/5038a2567f5853ea1e0fb55f795c30b339a318717573c5b0c85b8814d733/psutil-3.3.0-cp33-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"60f0487057f863a2f4cee79fc120f02b60168ae677fac14ba9ccc63127b37514"},"data-dist-info-metadata":{"sha256":"60f0487057f863a2f4cee79fc120f02b60168ae677fac14ba9ccc63127b37514"},"filename":"psutil-3.3.0-cp33-none-win_amd64.whl","hashes":{"sha256":"d9884dc0dc2e55e2448e495778dc9899c1c8bf37aeb2f434c1bea74af93c2683"},"provenance":null,"requires-python":null,"size":92432,"upload-time":"2015-11-25T01:22:39.035467Z","url":"https://files.pythonhosted.org/packages/b1/9c/a9cd75c8cfbac44397a2ca76430229c5496b21e0ab93cba5987d80e3f262/psutil-3.3.0-cp33-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"60f0487057f863a2f4cee79fc120f02b60168ae677fac14ba9ccc63127b37514"},"data-dist-info-metadata":{"sha256":"60f0487057f863a2f4cee79fc120f02b60168ae677fac14ba9ccc63127b37514"},"filename":"psutil-3.3.0-cp34-none-win32.whl","hashes":{"sha256":"e27c2fe6dfcc8738be3d2c5a022f785eb72971057e1a9e1e34fba73bce8a71a6"},"provenance":null,"requires-python":null,"size":90151,"upload-time":"2015-11-25T01:21:33.616930Z","url":"https://files.pythonhosted.org/packages/23/57/6a7c3ab4d04d055cada3b5511c40e0e699d8dd5d8217cae6fb68ae61dff6/psutil-3.3.0-cp34-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"60f0487057f863a2f4cee79fc120f02b60168ae677fac14ba9ccc63127b37514"},"data-dist-info-metadata":{"sha256":"60f0487057f863a2f4cee79fc120f02b60168ae677fac14ba9ccc63127b37514"},"filename":"psutil-3.3.0-cp34-none-win_amd64.whl","hashes":{"sha256":"65afd6fecc8f3aed09ee4be63583bc8eb472f06ceaa4fe24c4d1d5a1a3c0e13f"},"provenance":null,"requires-python":null,"size":92416,"upload-time":"2015-11-25T01:23:04.575877Z","url":"https://files.pythonhosted.org/packages/5d/a8/e62ec8105350c1e615ac84b084c7c8799d09e0d1b4530d3e68291dca8976/psutil-3.3.0-cp34-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"60f0487057f863a2f4cee79fc120f02b60168ae677fac14ba9ccc63127b37514"},"data-dist-info-metadata":{"sha256":"60f0487057f863a2f4cee79fc120f02b60168ae677fac14ba9ccc63127b37514"},"filename":"psutil-3.3.0-cp35-none-win32.whl","hashes":{"sha256":"ba1c558fbfcdf94515c2394b1155c1dc56e2bc2a9c17d30349827c9ed8a67e46"},"provenance":null,"requires-python":null,"size":91965,"upload-time":"2015-11-25T01:21:52.452732Z","url":"https://files.pythonhosted.org/packages/6e/6d/cf51e672eef1f1fbf9efce429d5411d4a2f3aa239e079b82531389562cd2/psutil-3.3.0-cp35-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"60f0487057f863a2f4cee79fc120f02b60168ae677fac14ba9ccc63127b37514"},"data-dist-info-metadata":{"sha256":"60f0487057f863a2f4cee79fc120f02b60168ae677fac14ba9ccc63127b37514"},"filename":"psutil-3.3.0-cp35-none-win_amd64.whl","hashes":{"sha256":"ba95ea0022dcb64d36f0c1335c0605fae35bdf3e0fea8d92f5d0f6456a35e55b"},"provenance":null,"requires-python":null,"size":94891,"upload-time":"2015-11-25T01:23:25.107121Z","url":"https://files.pythonhosted.org/packages/90/49/3726db12f0fa7ff8f7e5493cc128ee6b40f5720f7397a4ef01db9e28dd7b/psutil-3.3.0-cp35-none-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.3.0.tar.gz","hashes":{"sha256":"421b6591d16b509aaa8d8c15821d66bb94cb4a8dc4385cad5c51b85d4a096d85"},"provenance":null,"requires-python":null,"size":261983,"upload-time":"2015-11-25T01:20:55.681846Z","url":"https://files.pythonhosted.org/packages/fe/69/c0d8e9b9f8a58cbf71aa4cf7f27c27ee0ab05abe32d9157ec22e223edef4/psutil-3.3.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.3.0.win-amd64-py2.6.exe","hashes":{"sha256":"326b305cbdb6f94dafbfe2c26b11da88b0ab07b8a07f8188ab9d75ff0c6e841a"},"provenance":null,"requires-python":null,"size":329214,"upload-time":"2015-11-25T18:50:19.881553Z","url":"https://files.pythonhosted.org/packages/a9/c1/7642d44312cffaa1b3efc6ac5252b7f1ab1c528903b55d56d7bd46805d92/psutil-3.3.0.win-amd64-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.3.0.win-amd64-py2.7.exe","hashes":{"sha256":"9aede5b2b6fe46b3748ea8e5214443890d1634027bef3d33b7dad16556830278"},"provenance":null,"requires-python":null,"size":329153,"upload-time":"2015-11-25T01:22:03.853435Z","url":"https://files.pythonhosted.org/packages/bb/e0/f8e4e286bf9c075f0e9fb3c0b17cecef04cda5e91f4c54982b91b3baf338/psutil-3.3.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.3.0.win-amd64-py3.3.exe","hashes":{"sha256":"73bed1db894d1aa9c3c7e611d302cdeab7ae8a0dc0eeaf76727878db1ac5cd87"},"provenance":null,"requires-python":null,"size":327512,"upload-time":"2015-11-25T01:22:31.545622Z","url":"https://files.pythonhosted.org/packages/fb/20/9438b78a3155b1eb480a4ea09dab6370f06e0a003cf43c3975743e0c9e8d/psutil-3.3.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.3.0.win-amd64-py3.4.exe","hashes":{"sha256":"935b5dd6d558af512f42501a7c08f41d7aff139af1bb3959daa3abb859234d6c"},"provenance":null,"requires-python":null,"size":327497,"upload-time":"2015-11-25T01:22:51.620885Z","url":"https://files.pythonhosted.org/packages/ae/71/c68af8e9b05144de969da58e1bf5ebfe0859b1c83b827e05ae3116178bb1/psutil-3.3.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.3.0.win-amd64-py3.5.exe","hashes":{"sha256":"4ca0111cf157dcc0f2f69a323c5b5478718d68d45fc9435d84be0ec0f186215b"},"provenance":null,"requires-python":null,"size":243943,"upload-time":"2015-11-25T01:23:16.182734Z","url":"https://files.pythonhosted.org/packages/bd/14/a67db75c827761bf55a50c6ce455cdf0fd7e75d1c7c395b7283359676288/psutil-3.3.0.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.3.0.win32-py2.6.exe","hashes":{"sha256":"b6f13c95398a3fcf0226c4dcfa448560ba5865259cd96ec2810658651e932189"},"provenance":null,"requires-python":null,"size":299025,"upload-time":"2015-11-25T18:49:29.675333Z","url":"https://files.pythonhosted.org/packages/ad/ea/d7c41ad9fab6e89263225c66971f9807a0396925dddf7c20901b637b99e2/psutil-3.3.0.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.3.0.win32-py2.7.exe","hashes":{"sha256":"ee6be30d1635bbdea4c4325d507dc8a0dbbde7e1c198bd62ddb9f43198b9e214"},"provenance":null,"requires-python":null,"size":299056,"upload-time":"2015-11-25T01:20:34.764583Z","url":"https://files.pythonhosted.org/packages/15/f7/a34370848c11d7d7933c0c107763ee470b54a7e48aa90a301919a3ad6757/psutil-3.3.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.3.0.win32-py3.3.exe","hashes":{"sha256":"dfa786858c268d7fbbe1b6175e001ec02738d7cfae0a7ce77bf9b651af676729"},"provenance":null,"requires-python":null,"size":293994,"upload-time":"2015-11-25T01:20:57.318743Z","url":"https://files.pythonhosted.org/packages/71/1e/af5675d52b426857441c29ad88d4fccfd55d300867ad02531d77991ab661/psutil-3.3.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.3.0.win32-py3.4.exe","hashes":{"sha256":"aa77f9de72af9c16cc288cd4a24cf58824388f57d7a81e400c4616457629870e"},"provenance":null,"requires-python":null,"size":294006,"upload-time":"2015-11-25T01:21:23.806352Z","url":"https://files.pythonhosted.org/packages/6d/41/cf5b54535ea052a32a76a8e8e56af817deb95f4ffde49277a52ded29763b/psutil-3.3.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.3.0.win32-py3.5.exe","hashes":{"sha256":"f500093357d04da8140d87932cac2e54ef592a54ca8a743abb2850f60c2c22eb"},"provenance":null,"requires-python":null,"size":233858,"upload-time":"2015-11-25T01:21:44.041656Z","url":"https://files.pythonhosted.org/packages/1e/1d/151535e51338efebe453a28d2f14d4d5b1e1f3ce54ccc63866c96dc7e1bd/psutil-3.3.0.win32-py3.5.exe","yanked":false},{"core-metadata":{"sha256":"5fd1ef4aa7b3f11f37a72bb6da5e9ae73e666b2a5580c3bf58fe259cac90c4e8"},"data-dist-info-metadata":{"sha256":"5fd1ef4aa7b3f11f37a72bb6da5e9ae73e666b2a5580c3bf58fe259cac90c4e8"},"filename":"psutil-3.4.1-cp26-none-win32.whl","hashes":{"sha256":"0b1382db1cf76d53fb1d6e5619b5f3c86126e11a933b200c21ed4fa7fe5037aa"},"provenance":null,"requires-python":null,"size":91763,"upload-time":"2016-01-15T12:34:15.511699Z","url":"https://files.pythonhosted.org/packages/8a/a4/6dfd46e45d06da1a4d42814dbbdcffe3a4fc7f9b655e1c2919ac960512c2/psutil-3.4.1-cp26-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"5fd1ef4aa7b3f11f37a72bb6da5e9ae73e666b2a5580c3bf58fe259cac90c4e8"},"data-dist-info-metadata":{"sha256":"5fd1ef4aa7b3f11f37a72bb6da5e9ae73e666b2a5580c3bf58fe259cac90c4e8"},"filename":"psutil-3.4.1-cp26-none-win_amd64.whl","hashes":{"sha256":"bcf212a926e8cffd3bec2acaeb584bf59a536e569d404bd8ea306f1752fbfc41"},"provenance":null,"requires-python":null,"size":94311,"upload-time":"2016-01-15T12:34:35.252561Z","url":"https://files.pythonhosted.org/packages/85/38/d9882d4e37f4b791bd949a1f45c620e0f2573bb4048eb16d59d469e97ec6/psutil-3.4.1-cp26-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"5fd1ef4aa7b3f11f37a72bb6da5e9ae73e666b2a5580c3bf58fe259cac90c4e8"},"data-dist-info-metadata":{"sha256":"5fd1ef4aa7b3f11f37a72bb6da5e9ae73e666b2a5580c3bf58fe259cac90c4e8"},"filename":"psutil-3.4.1-cp27-none-win32.whl","hashes":{"sha256":"1b8424eaa712fef7da41fc7f391b452e8991a641a54e49c4f46eb72ca2585577"},"provenance":null,"requires-python":null,"size":91570,"upload-time":"2016-01-15T12:25:47.323665Z","url":"https://files.pythonhosted.org/packages/cd/2d/760f774b1325037ea4ef85972f45fc9dee417da33ba225b21a0a8e512f5d/psutil-3.4.1-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"5fd1ef4aa7b3f11f37a72bb6da5e9ae73e666b2a5580c3bf58fe259cac90c4e8"},"data-dist-info-metadata":{"sha256":"5fd1ef4aa7b3f11f37a72bb6da5e9ae73e666b2a5580c3bf58fe259cac90c4e8"},"filename":"psutil-3.4.1-cp27-none-win_amd64.whl","hashes":{"sha256":"46d7429bae3703a0f2980c0299d4d49ada733c7ebd2cfa4e29fa3e31b5b16014"},"provenance":null,"requires-python":null,"size":94024,"upload-time":"2016-01-15T12:29:47.181880Z","url":"https://files.pythonhosted.org/packages/13/06/0104f224dd52bf9e3fb3ef14f6b6b93e9fac72f562842d54445af041f3f0/psutil-3.4.1-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ea21e4b91d38833326191bf60eb600b6764ba491e5b44843eced5c91fcf75a03"},"data-dist-info-metadata":{"sha256":"ea21e4b91d38833326191bf60eb600b6764ba491e5b44843eced5c91fcf75a03"},"filename":"psutil-3.4.1-cp33-none-win32.whl","hashes":{"sha256":"08f4ab9b720310890fa9337321a6e1e8aa525538636526be77e82653588df46b"},"provenance":null,"requires-python":null,"size":91580,"upload-time":"2016-01-15T12:26:59.466185Z","url":"https://files.pythonhosted.org/packages/65/22/f7121341bc75bff65000ecc0c5aad4f2a6d129506c26d5533ade2ca67349/psutil-3.4.1-cp33-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"ea21e4b91d38833326191bf60eb600b6764ba491e5b44843eced5c91fcf75a03"},"data-dist-info-metadata":{"sha256":"ea21e4b91d38833326191bf60eb600b6764ba491e5b44843eced5c91fcf75a03"},"filename":"psutil-3.4.1-cp33-none-win_amd64.whl","hashes":{"sha256":"9e52230373076d0ecdb4aec373afd342c576ab52e11c382e058ed0188181a352"},"provenance":null,"requires-python":null,"size":93869,"upload-time":"2016-01-15T12:30:39.415707Z","url":"https://files.pythonhosted.org/packages/b0/84/a9edadc49ef3dbb89298855ae069b18ec534ca9f79a9294de417b8e46571/psutil-3.4.1-cp33-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ea21e4b91d38833326191bf60eb600b6764ba491e5b44843eced5c91fcf75a03"},"data-dist-info-metadata":{"sha256":"ea21e4b91d38833326191bf60eb600b6764ba491e5b44843eced5c91fcf75a03"},"filename":"psutil-3.4.1-cp34-none-win32.whl","hashes":{"sha256":"e658cd0e0ad7a2971b2eeb6ee4b1a0ad14245003ea47425846bc8c3e892fd567"},"provenance":null,"requires-python":null,"size":91584,"upload-time":"2016-01-15T12:28:02.007581Z","url":"https://files.pythonhosted.org/packages/b6/cd/59a87e4f10181ee228c4edc7d4927e3d62f652cff9f25f95a7e7e9ab3df0/psutil-3.4.1-cp34-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"ea21e4b91d38833326191bf60eb600b6764ba491e5b44843eced5c91fcf75a03"},"data-dist-info-metadata":{"sha256":"ea21e4b91d38833326191bf60eb600b6764ba491e5b44843eced5c91fcf75a03"},"filename":"psutil-3.4.1-cp34-none-win_amd64.whl","hashes":{"sha256":"9c3e1146003df43aec9274be4741371a06896d70d7d590eb882ad59de2c06120"},"provenance":null,"requires-python":null,"size":93858,"upload-time":"2016-01-15T12:31:25.406165Z","url":"https://files.pythonhosted.org/packages/30/06/cf0559d12ca5ded37e6a32b1671be57fad3bade7f24536b943851aa6393e/psutil-3.4.1-cp34-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ea21e4b91d38833326191bf60eb600b6764ba491e5b44843eced5c91fcf75a03"},"data-dist-info-metadata":{"sha256":"ea21e4b91d38833326191bf60eb600b6764ba491e5b44843eced5c91fcf75a03"},"filename":"psutil-3.4.1-cp35-none-win32.whl","hashes":{"sha256":"3d3b2df184a31646a7e66cc48304f900a82c18ab3dc69d2d5f693ea97fca0572"},"provenance":null,"requires-python":null,"size":93406,"upload-time":"2016-01-15T12:28:48.357070Z","url":"https://files.pythonhosted.org/packages/c9/57/6e65ff27fa567cd9a7bfbc0a435e33293451b80865bc3a3a7b13c9bf7799/psutil-3.4.1-cp35-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"ea21e4b91d38833326191bf60eb600b6764ba491e5b44843eced5c91fcf75a03"},"data-dist-info-metadata":{"sha256":"ea21e4b91d38833326191bf60eb600b6764ba491e5b44843eced5c91fcf75a03"},"filename":"psutil-3.4.1-cp35-none-win_amd64.whl","hashes":{"sha256":"820ed01d84ffcda1c613be80c09318d7560dd3505299c65bb99f101963bfc3dd"},"provenance":null,"requires-python":null,"size":96333,"upload-time":"2016-01-15T12:31:50.145506Z","url":"https://files.pythonhosted.org/packages/f0/f6/ccf16168a627d10ffbd80120cd2c521c4c9ecdb4545e402b7deca79f93ac/psutil-3.4.1-cp35-none-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.1.tar.gz","hashes":{"sha256":"c7443659674c87d1f9feecee0dfeea765da02181c58d532e0633337e42180c89"},"provenance":null,"requires-python":null,"size":271657,"upload-time":"2016-01-15T12:22:39.420430Z","url":"https://files.pythonhosted.org/packages/a5/56/c64187a9a6889e622f7ec687254cdb3cc3c706e11bba9244e6ac781ecf38/psutil-3.4.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.1.win-amd64-py2.6.exe","hashes":{"sha256":"e88e43423af984d7f2ecf8babf9d861ff59436794b0fdd2f85e9ea6bf7af6627"},"provenance":null,"requires-python":null,"size":331136,"upload-time":"2016-01-15T12:34:26.229804Z","url":"https://files.pythonhosted.org/packages/f7/90/53adfe2804c9cde062eb5014d88f0d067690fe1457ad2f49a2a553767689/psutil-3.4.1.win-amd64-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.1.win-amd64-py2.7.exe","hashes":{"sha256":"3c08e2c200b222a92a4ecaa8055a48e27e7cfe82d9bf6402b52dd82413a786ed"},"provenance":null,"requires-python":null,"size":330798,"upload-time":"2016-01-15T12:29:26.473954Z","url":"https://files.pythonhosted.org/packages/66/9b/2b58fdab300e5f2a20c3999c485692cfa73cc9d4e50770a19cc871f92743/psutil-3.4.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.1.win-amd64-py3.3.exe","hashes":{"sha256":"d03b8d081a281ebaa2122f259f7c0b3a464b2b98a3c221b9a54bfb0840355a9f"},"provenance":null,"requires-python":null,"size":329156,"upload-time":"2016-01-15T12:30:04.062459Z","url":"https://files.pythonhosted.org/packages/4c/26/695fa5b3578248f424d9a8e5bf2aafc6f706aeb7ec21ee31a5ebc2f79660/psutil-3.4.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.1.win-amd64-py3.4.exe","hashes":{"sha256":"66a243c4b9ad93059be391a18e3f75e015ad70b220df4f7f30f9f578b89f27ad"},"provenance":null,"requires-python":null,"size":329140,"upload-time":"2016-01-15T12:31:00.945171Z","url":"https://files.pythonhosted.org/packages/25/63/54f2ba7cf31bb936b9c2cd7a77fd40a698fb232cd7c95c1ce997295a5954/psutil-3.4.1.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.1.win-amd64-py3.5.exe","hashes":{"sha256":"4ee8641803d68a2e48952951336f9474a8914854da088fca673d67a91da7f9a4"},"provenance":null,"requires-python":null,"size":245586,"upload-time":"2016-01-15T12:31:39.398299Z","url":"https://files.pythonhosted.org/packages/43/54/f2d3b8845105fe5f55d5f0fde36773ab94fe1e35a0f4a5219adc818d586b/psutil-3.4.1.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.1.win32-py2.6.exe","hashes":{"sha256":"a4de0daf0dc7aeff6d45c6a1c782ef30d2b4fc6495196acabcb5cde2fb9b5a74"},"provenance":null,"requires-python":null,"size":300946,"upload-time":"2016-01-15T12:33:59.097771Z","url":"https://files.pythonhosted.org/packages/9e/34/53e1e85df97508df1b3eea711ab2809cc8f01b20e3b2341645673eb5d835/psutil-3.4.1.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.1.win32-py2.7.exe","hashes":{"sha256":"0efcecb6fcc21d83e9d4354754c6b8a8deb47a5fa06ec5d09fcf9799719eeac2"},"provenance":null,"requires-python":null,"size":300700,"upload-time":"2016-01-15T12:25:12.391548Z","url":"https://files.pythonhosted.org/packages/2f/4c/a07a53ff938e3bbc2ba73e4a484af8d1e02054b0bfcaf0f6d30117187d9f/psutil-3.4.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.1.win32-py3.3.exe","hashes":{"sha256":"62d0b529e40262293f39d7455db24f7ee297a1a3fe7f0e3e5923ae8168bd865c"},"provenance":null,"requires-python":null,"size":295638,"upload-time":"2016-01-15T12:26:24.412799Z","url":"https://files.pythonhosted.org/packages/39/6c/03ade7ba131b3952d916ed26c277418234bec0c9a5dfad513b9a5bb51046/psutil-3.4.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.1.win32-py3.4.exe","hashes":{"sha256":"6102294d6150f2c072dbc0166348389e8fa5d14a769ad118b697cda5b31c3381"},"provenance":null,"requires-python":null,"size":295651,"upload-time":"2016-01-15T12:27:36.577070Z","url":"https://files.pythonhosted.org/packages/52/d7/c2e9e0cb21482304e39a7681066c32c50e984f109bcda5929a84af926d70/psutil-3.4.1.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.1.win32-py3.5.exe","hashes":{"sha256":"0ffd99167272bb80c6ecf68f4c3d3176bef0f8c2a68f7e2787cab32413830023"},"provenance":null,"requires-python":null,"size":235502,"upload-time":"2016-01-15T12:28:24.463783Z","url":"https://files.pythonhosted.org/packages/1c/06/4d0ec9a6427db9c3b9885c4d724ca299746519b2ee61b724665d49e352c6/psutil-3.4.1.win32-py3.5.exe","yanked":false},{"core-metadata":{"sha256":"13402cacd1df223d4b15886666e72dc0e55f16091f011fc3f0c83e49b8034134"},"data-dist-info-metadata":{"sha256":"13402cacd1df223d4b15886666e72dc0e55f16091f011fc3f0c83e49b8034134"},"filename":"psutil-3.4.2-cp26-none-win32.whl","hashes":{"sha256":"2ac75c13657ab18eac0014e3f4c80def16978507b30e7719e46042ec93316bb0"},"provenance":null,"requires-python":null,"size":91920,"upload-time":"2016-01-20T16:25:20.303966Z","url":"https://files.pythonhosted.org/packages/be/b7/999dcbee8cc5ae32e64b2d0c9d588a3f5a441a07404772af83e86f3c8bc7/psutil-3.4.2-cp26-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"13402cacd1df223d4b15886666e72dc0e55f16091f011fc3f0c83e49b8034134"},"data-dist-info-metadata":{"sha256":"13402cacd1df223d4b15886666e72dc0e55f16091f011fc3f0c83e49b8034134"},"filename":"psutil-3.4.2-cp26-none-win_amd64.whl","hashes":{"sha256":"162f76140ca09490b9d218840bd641cbd1439245dcc2a9dd41f86224ed19490c"},"provenance":null,"requires-python":null,"size":94467,"upload-time":"2016-01-20T16:27:40.461506Z","url":"https://files.pythonhosted.org/packages/60/97/f9ea4fa7a4914350d15347a6a583c8a185643bb6bd5dc76d13e9d7dfc150/psutil-3.4.2-cp26-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"13402cacd1df223d4b15886666e72dc0e55f16091f011fc3f0c83e49b8034134"},"data-dist-info-metadata":{"sha256":"13402cacd1df223d4b15886666e72dc0e55f16091f011fc3f0c83e49b8034134"},"filename":"psutil-3.4.2-cp27-none-win32.whl","hashes":{"sha256":"b5f4bfdaa6389552501253b13b6022b7e3d715e4dca4b5cc1808f58cca181359"},"provenance":null,"requires-python":null,"size":91721,"upload-time":"2016-01-20T16:25:38.512050Z","url":"https://files.pythonhosted.org/packages/d4/19/4e5c376587076c969784762de8024bb30168a548b402e1b432221c5b97b1/psutil-3.4.2-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"13402cacd1df223d4b15886666e72dc0e55f16091f011fc3f0c83e49b8034134"},"data-dist-info-metadata":{"sha256":"13402cacd1df223d4b15886666e72dc0e55f16091f011fc3f0c83e49b8034134"},"filename":"psutil-3.4.2-cp27-none-win_amd64.whl","hashes":{"sha256":"9267e9bccb5c8b1c5ca872eb1caf88ba0ae47e336eb200be138f51d9f75e1113"},"provenance":null,"requires-python":null,"size":94177,"upload-time":"2016-01-20T16:27:58.779061Z","url":"https://files.pythonhosted.org/packages/d3/19/39f42cdfba58ab593d24f49ffc073c07b9b34ff7d5ba079b975018002e51/psutil-3.4.2-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"23ac4ea4417292cac0684989ffba547b75ac89f3a6ef7965183ec9e389edaa89"},"data-dist-info-metadata":{"sha256":"23ac4ea4417292cac0684989ffba547b75ac89f3a6ef7965183ec9e389edaa89"},"filename":"psutil-3.4.2-cp33-none-win32.whl","hashes":{"sha256":"413800a94815e6bf3e3227823e4d46b06c63bd22ab9e5af112b9220af9a9c9d8"},"provenance":null,"requires-python":null,"size":91738,"upload-time":"2016-01-20T16:25:59.987045Z","url":"https://files.pythonhosted.org/packages/8e/7a/8ba0c9da039b5733edbe17321f0546f08a7066861ffcdb83c31eb061e8f1/psutil-3.4.2-cp33-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"23ac4ea4417292cac0684989ffba547b75ac89f3a6ef7965183ec9e389edaa89"},"data-dist-info-metadata":{"sha256":"23ac4ea4417292cac0684989ffba547b75ac89f3a6ef7965183ec9e389edaa89"},"filename":"psutil-3.4.2-cp33-none-win_amd64.whl","hashes":{"sha256":"45535c18a0f261f90ff1ebb7d74c5e88d582cfb2006e4588498b9c0c9da5acb3"},"provenance":null,"requires-python":null,"size":94024,"upload-time":"2016-01-20T16:28:42.605657Z","url":"https://files.pythonhosted.org/packages/34/a7/94fc00a023ede02c3a7f525c63997a23c22434cbed65738ee3ef8939e084/psutil-3.4.2-cp33-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"23ac4ea4417292cac0684989ffba547b75ac89f3a6ef7965183ec9e389edaa89"},"data-dist-info-metadata":{"sha256":"23ac4ea4417292cac0684989ffba547b75ac89f3a6ef7965183ec9e389edaa89"},"filename":"psutil-3.4.2-cp34-none-win32.whl","hashes":{"sha256":"c9ef9a08254c251858cf747703e6fd75fe6e9549b1e040bb4a501feaf44a5a75"},"provenance":null,"requires-python":null,"size":91749,"upload-time":"2016-01-20T16:26:31.788083Z","url":"https://files.pythonhosted.org/packages/dc/8b/df5d7dcfe8fc5db0c303e518ce12b7117cb70e1cbb29c0396ea6e36fc7a2/psutil-3.4.2-cp34-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"23ac4ea4417292cac0684989ffba547b75ac89f3a6ef7965183ec9e389edaa89"},"data-dist-info-metadata":{"sha256":"23ac4ea4417292cac0684989ffba547b75ac89f3a6ef7965183ec9e389edaa89"},"filename":"psutil-3.4.2-cp34-none-win_amd64.whl","hashes":{"sha256":"0bf8925c3d252178c47bd8f29aff99c57a56f94513354b60069b457ca04bc25b"},"provenance":null,"requires-python":null,"size":94004,"upload-time":"2016-01-20T16:29:27.043903Z","url":"https://files.pythonhosted.org/packages/1c/cf/c9ce0014f43f74b1ce72c004c8f2eda68339cbc19117d9f090ee14afce3e/psutil-3.4.2-cp34-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"23ac4ea4417292cac0684989ffba547b75ac89f3a6ef7965183ec9e389edaa89"},"data-dist-info-metadata":{"sha256":"23ac4ea4417292cac0684989ffba547b75ac89f3a6ef7965183ec9e389edaa89"},"filename":"psutil-3.4.2-cp35-none-win32.whl","hashes":{"sha256":"461d1431a14e4da5e687cfdc2a8576b1f0e3bc658694ab9c6ef2fa1e4c1a4871"},"provenance":null,"requires-python":null,"size":93563,"upload-time":"2016-01-20T16:27:20.576920Z","url":"https://files.pythonhosted.org/packages/de/51/d1ab564dfe98d5fcbccfcab0afdedb269f7266192721e94688ab3956b123/psutil-3.4.2-cp35-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"23ac4ea4417292cac0684989ffba547b75ac89f3a6ef7965183ec9e389edaa89"},"data-dist-info-metadata":{"sha256":"23ac4ea4417292cac0684989ffba547b75ac89f3a6ef7965183ec9e389edaa89"},"filename":"psutil-3.4.2-cp35-none-win_amd64.whl","hashes":{"sha256":"23d4ea79fea3de81daf9460662e49ff718555779b2f5e5e3610648c0a8cafecc"},"provenance":null,"requires-python":null,"size":96484,"upload-time":"2016-01-20T16:29:53.151921Z","url":"https://files.pythonhosted.org/packages/89/d8/dc9ce7b8862ab2d86975dd5199791b0ab2b2168fc2389223a216c2da1d45/psutil-3.4.2-cp35-none-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.2.tar.gz","hashes":{"sha256":"b17fa01aa766daa388362d0eda5c215d77e03a8d37676b68971f37bf3913b725"},"provenance":null,"requires-python":null,"size":274361,"upload-time":"2016-01-20T16:26:46.533423Z","url":"https://files.pythonhosted.org/packages/7b/58/2675697b6831e6ac4b7b7bc4e5dcdb24a2f39f8411186573eb0de16eb6d5/psutil-3.4.2.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.2.win-amd64-py2.6.exe","hashes":{"sha256":"3716cb36373ecfd033c148c8e8e22d815a9b682c87538c5bde2a3faca1a44705"},"provenance":null,"requires-python":null,"size":331279,"upload-time":"2016-01-20T16:27:30.129391Z","url":"https://files.pythonhosted.org/packages/6f/52/26cefb84d714ecdf39f6d4ea62af49af731cf55d8937a252226de41d3fb0/psutil-3.4.2.win-amd64-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.2.win-amd64-py2.7.exe","hashes":{"sha256":"9eba153441fabd6677f9dec95eedbfdcf4fe832a43b91c18f2c15bfd0a12b6c0"},"provenance":null,"requires-python":null,"size":330991,"upload-time":"2016-01-20T16:27:49.025596Z","url":"https://files.pythonhosted.org/packages/1f/85/817b298f6865d7a140897882015096ab25514e113c98ba3896b2e2c0425c/psutil-3.4.2.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.2.win-amd64-py3.3.exe","hashes":{"sha256":"2ffc77ec7452675db45174f77e65bfc9abd5780e696c4dd486fff89e18ef104a"},"provenance":null,"requires-python":null,"size":329349,"upload-time":"2016-01-20T16:28:23.729829Z","url":"https://files.pythonhosted.org/packages/2b/05/ccb8e8dc272a6aa126a6066583102b78df8939aeffd910a3ea28a51d48af/psutil-3.4.2.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.2.win-amd64-py3.4.exe","hashes":{"sha256":"6b576ea8faa312700953de30b92ff49dcd966dcdbf2e039c3655077826b59812"},"provenance":null,"requires-python":null,"size":329334,"upload-time":"2016-01-20T16:29:12.322779Z","url":"https://files.pythonhosted.org/packages/8f/0e/9b3eedad9ea2aa8e51c3ca6aa6485c0ec85bfc73925fbd4d82bbe03ead18/psutil-3.4.2.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.2.win-amd64-py3.5.exe","hashes":{"sha256":"2a987c57ddb06a1e67f75a4dd34d2962f8675c3d60b2104da2d60fdaa378b50f"},"provenance":null,"requires-python":null,"size":245779,"upload-time":"2016-01-20T16:29:45.537968Z","url":"https://files.pythonhosted.org/packages/2f/bb/5483a7a54dfaedcd5bc6d0f9f8beef21d96785589e10d09e246f7092cfe1/psutil-3.4.2.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.2.win32-py2.6.exe","hashes":{"sha256":"87b657f1021ab4155669f77baf9557657a015b0762854702d64ee7cfa19d5ae2"},"provenance":null,"requires-python":null,"size":301089,"upload-time":"2016-01-20T16:25:11.034218Z","url":"https://files.pythonhosted.org/packages/2e/9b/2bb0317a5113b4a3d597a9bcb94cafacb52f15a10631b1f0eb781ceb8e7a/psutil-3.4.2.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.2.win32-py2.7.exe","hashes":{"sha256":"d837d654a78fcc6cf7338fc3c3f025e5a43cf646d4d6cf180f0f3573ef255844"},"provenance":null,"requires-python":null,"size":300894,"upload-time":"2016-01-20T16:25:29.293012Z","url":"https://files.pythonhosted.org/packages/8c/3b/bf4d0698153784231768aa79255f1641efde680c4d178ba327546eba69df/psutil-3.4.2.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.2.win32-py3.3.exe","hashes":{"sha256":"ef756512c86cf24916f47b2209ff5dc69ef4d5ff8b3b0229863aab3537af58a1"},"provenance":null,"requires-python":null,"size":295833,"upload-time":"2016-01-20T16:25:50.417672Z","url":"https://files.pythonhosted.org/packages/50/be/c4911ae27c944e12183d9ba844ff0eee706b5208f92e4929d8120b79448e/psutil-3.4.2.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.2.win32-py3.4.exe","hashes":{"sha256":"13917c61de33518fcdca94a8f1005c4bb0be1f106af773de8c12d6a5a3f349ae"},"provenance":null,"requires-python":null,"size":295845,"upload-time":"2016-01-20T16:26:17.574774Z","url":"https://files.pythonhosted.org/packages/b5/d9/9a15af2703d8a0d7b685511df811fc4930b12d7b85b96573e843a0ba1067/psutil-3.4.2.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-3.4.2.win32-py3.5.exe","hashes":{"sha256":"75c4484f0e1038c1a37ae37fc80f5b59b456e363c696ed889027e79a831853ec"},"provenance":null,"requires-python":null,"size":235696,"upload-time":"2016-01-20T16:27:12.935221Z","url":"https://files.pythonhosted.org/packages/2c/74/46cf554a4a8d7d2367f058e6c90ba26a66e52e818812ec6de53a5013bd87/psutil-3.4.2.win32-py3.5.exe","yanked":false},{"core-metadata":{"sha256":"f4174370ba76a8fc971e342367fb6df5d587165b15b69cf652d5870ad0386329"},"data-dist-info-metadata":{"sha256":"f4174370ba76a8fc971e342367fb6df5d587165b15b69cf652d5870ad0386329"},"filename":"psutil-4.0.0-cp26-none-win32.whl","hashes":{"sha256":"0661261b634f01ec2568136fedf29382f5c94678c34f56b4137b1d019085ca6f"},"provenance":null,"requires-python":null,"size":154479,"upload-time":"2016-02-17T16:42:58.173240Z","url":"https://files.pythonhosted.org/packages/72/e2/24700d1a099dcd824ca7305cf439625a457221459494e677e7649a2f228b/psutil-4.0.0-cp26-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"f4174370ba76a8fc971e342367fb6df5d587165b15b69cf652d5870ad0386329"},"data-dist-info-metadata":{"sha256":"f4174370ba76a8fc971e342367fb6df5d587165b15b69cf652d5870ad0386329"},"filename":"psutil-4.0.0-cp26-none-win_amd64.whl","hashes":{"sha256":"b613a9fb5c3d2b16c65df3121aa369a28caed83391883bc24918cf16c5de495b"},"provenance":null,"requires-python":null,"size":156749,"upload-time":"2016-02-17T16:45:09.428253Z","url":"https://files.pythonhosted.org/packages/62/c0/1adb11a832d5aab8a984702a4f6fc08e5698fa4bbc8dc6eddac1c711c7ab/psutil-4.0.0-cp26-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"f4174370ba76a8fc971e342367fb6df5d587165b15b69cf652d5870ad0386329"},"data-dist-info-metadata":{"sha256":"f4174370ba76a8fc971e342367fb6df5d587165b15b69cf652d5870ad0386329"},"filename":"psutil-4.0.0-cp27-cp27m-win32.whl","hashes":{"sha256":"3eb6bc7e8d92777deb4288178c25455e21109033bb54ec475485b611e92d3b42"},"provenance":null,"requires-python":null,"size":154302,"upload-time":"2016-02-17T16:43:23.863441Z","url":"https://files.pythonhosted.org/packages/e8/c3/542bc833b743e952cbf99017ecb60add0ad3725a82e942b25aa5de523f8c/psutil-4.0.0-cp27-cp27m-win32.whl","yanked":false},{"core-metadata":{"sha256":"f4174370ba76a8fc971e342367fb6df5d587165b15b69cf652d5870ad0386329"},"data-dist-info-metadata":{"sha256":"f4174370ba76a8fc971e342367fb6df5d587165b15b69cf652d5870ad0386329"},"filename":"psutil-4.0.0-cp27-cp27m-win_amd64.whl","hashes":{"sha256":"c94193f38aa3bc35fd5dbcd24653d1f683c88ec8030997d1d56f92207ba7c523"},"provenance":null,"requires-python":null,"size":156487,"upload-time":"2016-02-24T17:23:02.172281Z","url":"https://files.pythonhosted.org/packages/32/1b/5f3cc96374c4eac441e96bb8698556c6c48eacfdcf843093bebcfd8ce56b/psutil-4.0.0-cp27-cp27m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"f4174370ba76a8fc971e342367fb6df5d587165b15b69cf652d5870ad0386329"},"data-dist-info-metadata":{"sha256":"f4174370ba76a8fc971e342367fb6df5d587165b15b69cf652d5870ad0386329"},"filename":"psutil-4.0.0-cp27-none-win_amd64.whl","hashes":{"sha256":"cb969d3c77db8810aba45e8a04e0b2851cd088e338be2430e1ff452f4e06007c"},"provenance":null,"requires-python":null,"size":156486,"upload-time":"2016-02-17T16:45:40.145586Z","url":"https://files.pythonhosted.org/packages/94/37/dc09e24aa80016ddeaff235d2f724d8aac9813b73cc3bf8a7fe3d1878315/psutil-4.0.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"data-dist-info-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"filename":"psutil-4.0.0-cp33-cp33m-win32.whl","hashes":{"sha256":"6a372681382b523bc837ee2eff6a84ded0f85b013b7c29ea6211bc928c7cc656"},"provenance":null,"requires-python":null,"size":154244,"upload-time":"2016-02-24T17:21:40.946801Z","url":"https://files.pythonhosted.org/packages/a0/8a/b9352e0daf69b501296715e0fca1b49d861130eb66156ce3b12aeeb039e4/psutil-4.0.0-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"data-dist-info-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"filename":"psutil-4.0.0-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"2bbb75fc2549965b457f313cbdfb98a00624f25fcb36e075322bb8b8912d83b5"},"provenance":null,"requires-python":null,"size":156393,"upload-time":"2016-02-24T17:23:25.807132Z","url":"https://files.pythonhosted.org/packages/4b/77/0fefa732947da69cba7f2580285eff553fe4a416314234f809901bede361/psutil-4.0.0-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"data-dist-info-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"filename":"psutil-4.0.0-cp33-none-win32.whl","hashes":{"sha256":"d6219c89940d745b614716be7c906660f2108a1d84b8ffc720922596b8306e23"},"provenance":null,"requires-python":null,"size":154245,"upload-time":"2016-02-17T16:43:48.027623Z","url":"https://files.pythonhosted.org/packages/1f/d9/34fb4fab5f1bbfeddc76675b1b5ca00b45ef490e63295af33542cedcd26b/psutil-4.0.0-cp33-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"data-dist-info-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"filename":"psutil-4.0.0-cp33-none-win_amd64.whl","hashes":{"sha256":"a64bb22e264f91a6d80cf8fdd813bd4fdd349dc367b363d517cf8ae1bc2c5db0"},"provenance":null,"requires-python":null,"size":156399,"upload-time":"2016-02-17T16:46:09.673427Z","url":"https://files.pythonhosted.org/packages/97/03/b9485635cb38dfad854754625422a49f434f53f214bff4885580f0fb21e6/psutil-4.0.0-cp33-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"data-dist-info-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"filename":"psutil-4.0.0-cp34-cp34m-win32.whl","hashes":{"sha256":"a521266ac13485772987f00342b53cb230cde98ce91d61154860ba4109fe2ebe"},"provenance":null,"requires-python":null,"size":154269,"upload-time":"2016-02-17T16:44:14.936699Z","url":"https://files.pythonhosted.org/packages/73/32/6399071b097f1251f6fa12770b30a67d5b3c9c0c76e81eacbb6139e1bf6d/psutil-4.0.0-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"data-dist-info-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"filename":"psutil-4.0.0-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"b559b8e8a85cde929e01e94e9635649e8641a88b2d077714933dc7723a967020"},"provenance":null,"requires-python":null,"size":156390,"upload-time":"2016-02-24T17:23:50.194021Z","url":"https://files.pythonhosted.org/packages/71/d7/878b77bad61bd94f4454536e823b6a48cd0af0f23b1506a2c8a49b2578cd/psutil-4.0.0-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"data-dist-info-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"filename":"psutil-4.0.0-cp34-none-win_amd64.whl","hashes":{"sha256":"02d7291f81e78c506ac2b5481aa9dc6d3888195484ac114ac984b37477f60929"},"provenance":null,"requires-python":null,"size":156397,"upload-time":"2016-02-17T16:46:44.626238Z","url":"https://files.pythonhosted.org/packages/ea/22/5f44e6eaa1e82f5a1497f3dfcf045e1998fca36d70de8a370ec96ce0f789/psutil-4.0.0-cp34-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"data-dist-info-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"filename":"psutil-4.0.0-cp35-cp35m-win32.whl","hashes":{"sha256":"7906302696960a6a788bb8fe1165b4ccd0156553b8a2f61640fd45a836d39024"},"provenance":null,"requires-python":null,"size":156177,"upload-time":"2016-02-24T17:22:25.142667Z","url":"https://files.pythonhosted.org/packages/e2/fe/a5ec73e62878cc2d0451b7029f4406647435dd8036ab15d6ed2fd42558bf/psutil-4.0.0-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"data-dist-info-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"filename":"psutil-4.0.0-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"f4214bdb2e96374b4c4a3a818bd8c7867f94571d33b91867b6dfd5f9b328c8ac"},"provenance":null,"requires-python":null,"size":158783,"upload-time":"2016-02-24T17:24:26.439061Z","url":"https://files.pythonhosted.org/packages/1d/a7/9300ad3d4071c191894073a94217ed5c0ca9604c782bdbf083bbedfa9cb1/psutil-4.0.0-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"data-dist-info-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"filename":"psutil-4.0.0-cp35-none-win32.whl","hashes":{"sha256":"4a1631cb8c4de2b6c9b4b16f8800d43de23c683805f7b6a5aec1c268a73df270"},"provenance":null,"requires-python":null,"size":156183,"upload-time":"2016-02-17T16:44:42.458015Z","url":"https://files.pythonhosted.org/packages/14/f0/a2436cb642ecfec0bfb6338e5fa26581d4dbcf1a00f7d9fe99380eb6779f/psutil-4.0.0-cp35-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"data-dist-info-metadata":{"sha256":"d2287fe3ce0e3872765b52193af0ec6f19279df57903b85d13961ca84449ba60"},"filename":"psutil-4.0.0-cp35-none-win_amd64.whl","hashes":{"sha256":"994839b6d99acbf90914fddf2e2817aaffb67ceca5d10134319267e3ffe97258"},"provenance":null,"requires-python":null,"size":158787,"upload-time":"2016-02-17T16:47:18.920483Z","url":"https://files.pythonhosted.org/packages/e8/2a/e215824c785d77119af61802bbb4d16dacc26ec0687709274afa3ac039fa/psutil-4.0.0-cp35-none-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.0.0.tar.gz","hashes":{"sha256":"1a7c672f9ee79c84ff16b8de6f6040080f0e25002ac47f115f4a54aa88e5cfcd"},"provenance":null,"requires-python":null,"size":293800,"upload-time":"2016-02-17T16:41:45.938066Z","url":"https://files.pythonhosted.org/packages/c4/3b/44bcae6c0fc53362bb7325fde25a73b7fd46541b57c89b7556ca81b08e7e/psutil-4.0.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.0.0.win-amd64-py2.6.exe","hashes":{"sha256":"61bf81cbe84e679a5b619e65775b0674b2c463885e49ddab73778198608198c5"},"provenance":null,"requires-python":null,"size":394157,"upload-time":"2016-02-17T16:45:00.666953Z","url":"https://files.pythonhosted.org/packages/86/50/6303a28a4ab5c9b6b9ef74eef70b141d5bd743ab096c58d225b6212fa057/psutil-4.0.0.win-amd64-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.0.0.win-amd64-py2.7.exe","hashes":{"sha256":"23618cbc04e2431d9c4d97f56ab8b4e2e35366c9a9a6e1ef89a3a7287d359864"},"provenance":null,"requires-python":null,"size":393901,"upload-time":"2016-02-17T16:45:21.822412Z","url":"https://files.pythonhosted.org/packages/59/82/93052d6359addea338c528ebd50254806d62bc2b2d1ad1303c49d85162f9/psutil-4.0.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.0.0.win-amd64-py3.3.exe","hashes":{"sha256":"b6f16c71be03495eeb4772c1f3f926213e3ea82ea7779bd1143229e6b419760b"},"provenance":null,"requires-python":null,"size":392318,"upload-time":"2016-02-17T16:45:57.189438Z","url":"https://files.pythonhosted.org/packages/e6/14/9ac37705e0753732c7707b000d1e076daac95ee02f35fd43ce906235ea1f/psutil-4.0.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.0.0.win-amd64-py3.4.exe","hashes":{"sha256":"24b41e436afbcecb07e485e58a52effdbd7b8065ad8a2e4d555b6d88907f19b7"},"provenance":null,"requires-python":null,"size":392315,"upload-time":"2016-02-17T16:46:29.967466Z","url":"https://files.pythonhosted.org/packages/77/a9/ff7c29d2e244f5bdc7654a626cbfcccd401e78df6e9388713f322c7aa7c7/psutil-4.0.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.0.0.win-amd64-py3.5.exe","hashes":{"sha256":"88edc0e7bfa672c245df74b1ac3b59db432cd75e5704beccc268e177ac2ffbbc"},"provenance":null,"requires-python":null,"size":308678,"upload-time":"2016-02-17T16:47:07.280714Z","url":"https://files.pythonhosted.org/packages/cc/1b/863bee07da70fe61cae804333d64242d9001b54288e8ff54e770225bbc0a/psutil-4.0.0.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.0.0.win32-py2.6.exe","hashes":{"sha256":"882a8ac29b63f256f76465d8fcd6e9eaeb9c929acdac26af102da97d66b2b619"},"provenance":null,"requires-python":null,"size":364245,"upload-time":"2016-02-17T16:42:49.267819Z","url":"https://files.pythonhosted.org/packages/41/39/6ea85cb0c748aae2943144118f1696004f5a99c54dab1fc635cffbb0d06c/psutil-4.0.0.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.0.0.win32-py2.7.exe","hashes":{"sha256":"bf3b2e305ca7408df40156c9aa6261c7baaff831441f0c018d0682bd820286f2"},"provenance":null,"requires-python":null,"size":364072,"upload-time":"2016-02-17T16:43:14.467612Z","url":"https://files.pythonhosted.org/packages/8f/0d/3e9cf8abb62d7241531019d78abaa87a19f3fcc017bfb9c2058ba61e8cf1/psutil-4.0.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.0.0.win32-py3.3.exe","hashes":{"sha256":"38d38211bba35c705a007e62b9dcc9be1d222acfcbee812612d4a48f9d8f0230"},"provenance":null,"requires-python":null,"size":358940,"upload-time":"2016-02-17T16:43:38.439349Z","url":"https://files.pythonhosted.org/packages/57/eb/514a71eab624b381473a3df9c3e3a02f5bb15707b12daf02c137271dfd26/psutil-4.0.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.0.0.win32-py3.4.exe","hashes":{"sha256":"bc868653a6502c3a01da32b3a598a8575674975f5586ac0bf9181349588925b9"},"provenance":null,"requires-python":null,"size":358966,"upload-time":"2016-02-17T16:44:02.873748Z","url":"https://files.pythonhosted.org/packages/25/f6/ef4b802658c21d5b79a0e038db941f4b08a7cc2de78df1949ad709542682/psutil-4.0.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.0.0.win32-py3.5.exe","hashes":{"sha256":"38244b0d07d3bece481a6f1c049e6101fdd26f2ee63dadcb63ce993283032fdc"},"provenance":null,"requires-python":null,"size":298915,"upload-time":"2016-02-17T16:44:31.772075Z","url":"https://files.pythonhosted.org/packages/b9/be/b8938c409231dab07d2954ff7b4d1129e725e0e8ab1b016d7f471f3285e9/psutil-4.0.0.win32-py3.5.exe","yanked":false},{"core-metadata":{"sha256":"9a907b7ac326dc42e1ba74d263d9a38c07d0929a2a9f91960d9961923d796b2d"},"data-dist-info-metadata":{"sha256":"9a907b7ac326dc42e1ba74d263d9a38c07d0929a2a9f91960d9961923d796b2d"},"filename":"psutil-4.1.0-cp26-none-win32.whl","hashes":{"sha256":"13aed96ad945db5c6b3d5fbe92be65330a3f2f757a300c7d1578a16efa0ece7f"},"provenance":null,"requires-python":null,"size":159498,"upload-time":"2016-03-12T17:13:58.069685Z","url":"https://files.pythonhosted.org/packages/77/04/d5a92cb5c0e79b84294f6c99b9725806921d1d88032e9d056ca8a7ba31c1/psutil-4.1.0-cp26-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"9a907b7ac326dc42e1ba74d263d9a38c07d0929a2a9f91960d9961923d796b2d"},"data-dist-info-metadata":{"sha256":"9a907b7ac326dc42e1ba74d263d9a38c07d0929a2a9f91960d9961923d796b2d"},"filename":"psutil-4.1.0-cp26-none-win_amd64.whl","hashes":{"sha256":"90b58cf88e80a4af52b79678df474679d231ed22200e6c25605a42ca71708a47"},"provenance":null,"requires-python":null,"size":161924,"upload-time":"2016-03-12T17:17:01.346400Z","url":"https://files.pythonhosted.org/packages/b5/a5/cf96f9f13f9e20bdb4cd2ca1af2ddd74f76fea4bbfb8505c31a5900b38d2/psutil-4.1.0-cp26-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"9a907b7ac326dc42e1ba74d263d9a38c07d0929a2a9f91960d9961923d796b2d"},"data-dist-info-metadata":{"sha256":"9a907b7ac326dc42e1ba74d263d9a38c07d0929a2a9f91960d9961923d796b2d"},"filename":"psutil-4.1.0-cp27-cp27m-win32.whl","hashes":{"sha256":"ac141a44a5c145e9006bc7081c714b2c317077d158b65fe4624c9cbf2b8ac7bf"},"provenance":null,"requires-python":null,"size":159318,"upload-time":"2016-03-12T17:14:48.199436Z","url":"https://files.pythonhosted.org/packages/8a/31/439614cc2ccd6f3ce1d173c0d7c7a9e45be17cd2bf3ae1f8feaaf0a90cee/psutil-4.1.0-cp27-cp27m-win32.whl","yanked":false},{"core-metadata":{"sha256":"9a907b7ac326dc42e1ba74d263d9a38c07d0929a2a9f91960d9961923d796b2d"},"data-dist-info-metadata":{"sha256":"9a907b7ac326dc42e1ba74d263d9a38c07d0929a2a9f91960d9961923d796b2d"},"filename":"psutil-4.1.0-cp27-cp27m-win_amd64.whl","hashes":{"sha256":"3605b6b9f23f3e186b157b03a95e0158559eb74bcef5d51920b8ddb48cc3a7e7"},"provenance":null,"requires-python":null,"size":161579,"upload-time":"2016-03-12T17:17:24.047978Z","url":"https://files.pythonhosted.org/packages/90/97/0a34c0e98bb794f0fc19f0eae13d26fbf39583c768a9a6c614c917135c00/psutil-4.1.0-cp27-cp27m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"28328dafba00a0d87523ceca7f8bb23231019819caf14ec9eeddc3c4c8104338"},"data-dist-info-metadata":{"sha256":"28328dafba00a0d87523ceca7f8bb23231019819caf14ec9eeddc3c4c8104338"},"filename":"psutil-4.1.0-cp33-cp33m-win32.whl","hashes":{"sha256":"5568e21c8eb9de0e56c8a4a38982b725bf42117bca7ac75c7b079e5214aea5c4"},"provenance":null,"requires-python":null,"size":159241,"upload-time":"2016-03-12T17:15:16.355951Z","url":"https://files.pythonhosted.org/packages/bb/de/8e1f8c4ea6035d08e3c87a0cfc8af6f2862da21697c16d1d17311e095117/psutil-4.1.0-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"28328dafba00a0d87523ceca7f8bb23231019819caf14ec9eeddc3c4c8104338"},"data-dist-info-metadata":{"sha256":"28328dafba00a0d87523ceca7f8bb23231019819caf14ec9eeddc3c4c8104338"},"filename":"psutil-4.1.0-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"3d3f8ae20b04b68e65b46dc2eedf15f32925655dacbb11cb7afe56ac562e112a"},"provenance":null,"requires-python":null,"size":161458,"upload-time":"2016-03-12T17:17:51.825112Z","url":"https://files.pythonhosted.org/packages/ac/cf/6241dd597ef4f995ab8e29746c54890c1acbb322484afed05aa8988118e1/psutil-4.1.0-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"28328dafba00a0d87523ceca7f8bb23231019819caf14ec9eeddc3c4c8104338"},"data-dist-info-metadata":{"sha256":"28328dafba00a0d87523ceca7f8bb23231019819caf14ec9eeddc3c4c8104338"},"filename":"psutil-4.1.0-cp34-cp34m-win32.whl","hashes":{"sha256":"faafb81bf7717fa8c44bb0f2e826768f561c0311fd0568090c59c9b253b65238"},"provenance":null,"requires-python":null,"size":159266,"upload-time":"2016-03-12T17:15:48.577644Z","url":"https://files.pythonhosted.org/packages/48/2d/46ba91df965d4f0af5fd4252ac249ff408f6cb966fe1208396933275246f/psutil-4.1.0-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"28328dafba00a0d87523ceca7f8bb23231019819caf14ec9eeddc3c4c8104338"},"data-dist-info-metadata":{"sha256":"28328dafba00a0d87523ceca7f8bb23231019819caf14ec9eeddc3c4c8104338"},"filename":"psutil-4.1.0-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"4ab1ee4152dbb790a37291149b73b1918ab4398c8edb3af7847fa6c884024c93"},"provenance":null,"requires-python":null,"size":161466,"upload-time":"2016-03-12T17:18:23.409579Z","url":"https://files.pythonhosted.org/packages/c0/b1/70868328ddf2cfcde201136bdaf4c9f7fabf868890bc91694fd5fa0fbc19/psutil-4.1.0-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"28328dafba00a0d87523ceca7f8bb23231019819caf14ec9eeddc3c4c8104338"},"data-dist-info-metadata":{"sha256":"28328dafba00a0d87523ceca7f8bb23231019819caf14ec9eeddc3c4c8104338"},"filename":"psutil-4.1.0-cp35-cp35m-win32.whl","hashes":{"sha256":"2fa06b7ba58e870fdaa1427e82ed427a785493c7a998e059e0806b2c48bdbfaf"},"provenance":null,"requires-python":null,"size":161345,"upload-time":"2016-03-12T17:16:33.062364Z","url":"https://files.pythonhosted.org/packages/dc/f7/5d3f84507c057af85bc70da10b51a827766999f221b42cdf9621ca756e80/psutil-4.1.0-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"28328dafba00a0d87523ceca7f8bb23231019819caf14ec9eeddc3c4c8104338"},"data-dist-info-metadata":{"sha256":"28328dafba00a0d87523ceca7f8bb23231019819caf14ec9eeddc3c4c8104338"},"filename":"psutil-4.1.0-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"1ca460fea3822d04f332f5dde144accc5ca4610e5bbce53f15d09cd985f30385"},"provenance":null,"requires-python":null,"size":164147,"upload-time":"2016-03-12T17:18:49.483687Z","url":"https://files.pythonhosted.org/packages/f1/65/040624aab6ca646af0c8b68ac54d08e0a33a672feb9405581cd509741367/psutil-4.1.0-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.1.0.tar.gz","hashes":{"sha256":"c6abebec9c8833baaf1c51dd1b0259246d1d50b9b50e9a4aa66f33b1e98b8d17"},"provenance":null,"requires-python":null,"size":301330,"upload-time":"2016-03-12T17:12:53.032151Z","url":"https://files.pythonhosted.org/packages/71/9b/6b6f630ad4262572839033b69905d415ef152d7701ef40aa98941ba75b38/psutil-4.1.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.1.0.win-amd64-py2.6.exe","hashes":{"sha256":"30a97a9c3ace92001e419a6bb039b2e899c5ab24cab6ad8bc249506475c84a0c"},"provenance":null,"requires-python":null,"size":399511,"upload-time":"2016-03-12T17:16:48.833953Z","url":"https://files.pythonhosted.org/packages/28/bd/c389af84b684d36010a634834f76932ff60f33505c54413c50eceb720a5d/psutil-4.1.0.win-amd64-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.1.0.win-amd64-py2.7.exe","hashes":{"sha256":"3ba9236c38fe85088b9d79bd5871e07f253d225b357ce82db9240e5807f147b6"},"provenance":null,"requires-python":null,"size":399168,"upload-time":"2016-03-12T17:17:14.518227Z","url":"https://files.pythonhosted.org/packages/6c/e8/49ff1b33e9fa6f7ea1232d40bbef9a453424fbf0421a7c53d9ecb9a896e7/psutil-4.1.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.1.0.win-amd64-py3.3.exe","hashes":{"sha256":"99a902d1bf5beb13cca4d7a3dc82efb6eaf40aafe916a5b632d47393313bfcfd"},"provenance":null,"requires-python":null,"size":397557,"upload-time":"2016-03-12T17:17:40.792655Z","url":"https://files.pythonhosted.org/packages/c5/3e/47adda552a79480e2a5d38a2f90144ab4e7ea34eba2b707523cdd1c65fc6/psutil-4.1.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.1.0.win-amd64-py3.4.exe","hashes":{"sha256":"ecf8b83e038acdda9102aebadb1525f89231463772cbe89e3e8a2cf5a5c6065d"},"provenance":null,"requires-python":null,"size":397564,"upload-time":"2016-03-12T17:18:11.514028Z","url":"https://files.pythonhosted.org/packages/52/9f/5874f391a300feead5519872395cbb4d4588eace24962150a03cd9b6ffdf/psutil-4.1.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.1.0.win-amd64-py3.5.exe","hashes":{"sha256":"448db9bf9db5d162f0282af70688e03d67d756b93d5bed94b0790a27a96af75b"},"provenance":null,"requires-python":null,"size":314216,"upload-time":"2016-03-12T17:18:36.745340Z","url":"https://files.pythonhosted.org/packages/32/34/6588580a1775a0741e14946003bf2722c493c7956c7dcc9a40c85dbe19f5/psutil-4.1.0.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.1.0.win32-py2.6.exe","hashes":{"sha256":"11b48d7ec00061960ffbef33a42e919f188e10a6a54c4161692d77a3ac37f1e2"},"provenance":null,"requires-python":null,"size":369441,"upload-time":"2016-03-12T17:13:47.639393Z","url":"https://files.pythonhosted.org/packages/5d/cc/7bf8593a60ed54b47def9a49a60b8bc3517d6fef6ee52a229583a5bc9046/psutil-4.1.0.win32-py2.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.1.0.win32-py2.7.exe","hashes":{"sha256":"c15ddba9f4c278f7d1bbbadc34df89d993171bb328fa1117cbecf68bcc1a01e5"},"provenance":null,"requires-python":null,"size":369263,"upload-time":"2016-03-12T17:14:25.482243Z","url":"https://files.pythonhosted.org/packages/bf/fa/7f0eda490dd5480bb8e6358f9e893fe7f824ec3a5b275b069708272d2260/psutil-4.1.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.1.0.win32-py3.3.exe","hashes":{"sha256":"8f97a836b89b09718b00aedbfd11d8468bbfcb821feadd65e338bd0b972dac54"},"provenance":null,"requires-python":null,"size":364112,"upload-time":"2016-03-12T17:15:06.252090Z","url":"https://files.pythonhosted.org/packages/e8/1c/e7d17500c0f5899f0fd3fd3b6d93c92254aff6de00cabc9c08de5a3803a2/psutil-4.1.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.1.0.win32-py3.4.exe","hashes":{"sha256":"fef37f4e2964043a58b6aa3b3be05ef9b3c7a58feb806d7e49dc91702bac52fa"},"provenance":null,"requires-python":null,"size":364136,"upload-time":"2016-03-12T17:15:29.468205Z","url":"https://files.pythonhosted.org/packages/6d/8e/7acd4079567fc64e1df12be6faeddf0ee19432445210293594444ea980a6/psutil-4.1.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.1.0.win32-py3.5.exe","hashes":{"sha256":"1a9409d2204d397b1ed10528065e45020e0b1c2ee5204bf5a1bc6fdff3f6ab91"},"provenance":null,"requires-python":null,"size":304255,"upload-time":"2016-03-12T17:16:06.024724Z","url":"https://files.pythonhosted.org/packages/5b/30/10f3bef7fa284167a3f9bd3862bcba5cc14fb8509f146a2f696ea5265b93/psutil-4.1.0.win32-py3.5.exe","yanked":false},{"core-metadata":{"sha256":"5eaefa5c21411ed0414953f28eef77176307e672d668daf95d91e18ad3dc36f7"},"data-dist-info-metadata":{"sha256":"5eaefa5c21411ed0414953f28eef77176307e672d668daf95d91e18ad3dc36f7"},"filename":"psutil-4.2.0-cp27-cp27m-win32.whl","hashes":{"sha256":"19f6c8bd30d7827ce4d4bbcfe23fe7158fea3d72f59505850c5afa12985184bb"},"provenance":null,"requires-python":null,"size":165248,"upload-time":"2016-05-15T06:36:58.089018Z","url":"https://files.pythonhosted.org/packages/58/a5/2ccc9f6180ea769005405381f6b0d01fe1268f20cc85877b02c04c27d306/psutil-4.2.0-cp27-cp27m-win32.whl","yanked":false},{"core-metadata":{"sha256":"5eaefa5c21411ed0414953f28eef77176307e672d668daf95d91e18ad3dc36f7"},"data-dist-info-metadata":{"sha256":"5eaefa5c21411ed0414953f28eef77176307e672d668daf95d91e18ad3dc36f7"},"filename":"psutil-4.2.0-cp27-cp27m-win_amd64.whl","hashes":{"sha256":"92bc2351bb4bc7672b3d0e251a449ac2234bbe4fac11f708614bdc0a8ebffe3b"},"provenance":null,"requires-python":null,"size":167782,"upload-time":"2016-05-15T06:37:06.221707Z","url":"https://files.pythonhosted.org/packages/c8/e5/5d0a1b2e182e41888fc4e9f4f657f37f126f9fdcd431b592442311c2db98/psutil-4.2.0-cp27-cp27m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"467235151a7ca9506acf3ca52432d5a215f42e2fe44c043d0a0068e133a957dd"},"data-dist-info-metadata":{"sha256":"467235151a7ca9506acf3ca52432d5a215f42e2fe44c043d0a0068e133a957dd"},"filename":"psutil-4.2.0-cp33-cp33m-win32.whl","hashes":{"sha256":"2e16f792deceb1d33320981aaff7f139561cf6195ee3f1b21256d7f214162517"},"provenance":null,"requires-python":null,"size":165259,"upload-time":"2016-05-15T06:37:12.236981Z","url":"https://files.pythonhosted.org/packages/dd/94/8aeb332d07530b552099eaf207db13d859e09facfa8162892b4f9ef302dd/psutil-4.2.0-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"467235151a7ca9506acf3ca52432d5a215f42e2fe44c043d0a0068e133a957dd"},"data-dist-info-metadata":{"sha256":"467235151a7ca9506acf3ca52432d5a215f42e2fe44c043d0a0068e133a957dd"},"filename":"psutil-4.2.0-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"3c57a6731b3bd4c4af834b0137493a388b76192f5adc2399825015b777e0b02b"},"provenance":null,"requires-python":null,"size":167667,"upload-time":"2016-05-15T06:37:17.736147Z","url":"https://files.pythonhosted.org/packages/d5/6b/c10a228ef2cdbc077171be3b273cd2f49e4f814bf7dc2deb3a464cc126de/psutil-4.2.0-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"467235151a7ca9506acf3ca52432d5a215f42e2fe44c043d0a0068e133a957dd"},"data-dist-info-metadata":{"sha256":"467235151a7ca9506acf3ca52432d5a215f42e2fe44c043d0a0068e133a957dd"},"filename":"psutil-4.2.0-cp34-cp34m-win32.whl","hashes":{"sha256":"0cda72a1efacd2b028c9dbf0731111041e6cf9e7be938162811ab32ab3e88254"},"provenance":null,"requires-python":null,"size":165272,"upload-time":"2016-05-15T06:37:23.488252Z","url":"https://files.pythonhosted.org/packages/35/3e/3db756e014fe3e6e22e35c8394057dcf1eef58076d84fdf83ac00a053182/psutil-4.2.0-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"467235151a7ca9506acf3ca52432d5a215f42e2fe44c043d0a0068e133a957dd"},"data-dist-info-metadata":{"sha256":"467235151a7ca9506acf3ca52432d5a215f42e2fe44c043d0a0068e133a957dd"},"filename":"psutil-4.2.0-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"ce208e1c416e143697a1ee9dd86ae9720c740c11764a1fda88eb28a2ecc0b510"},"provenance":null,"requires-python":null,"size":167627,"upload-time":"2016-05-15T06:37:28.596251Z","url":"https://files.pythonhosted.org/packages/c7/7d/cfb299960cf6923cca782f331b034c09239e9015dedf530dd206177dd6e4/psutil-4.2.0-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"467235151a7ca9506acf3ca52432d5a215f42e2fe44c043d0a0068e133a957dd"},"data-dist-info-metadata":{"sha256":"467235151a7ca9506acf3ca52432d5a215f42e2fe44c043d0a0068e133a957dd"},"filename":"psutil-4.2.0-cp35-cp35m-win32.whl","hashes":{"sha256":"375b0acad448e49c8bc62e036f948af610b4e0cbe2a9a28eebc06357f20f67ea"},"provenance":null,"requires-python":null,"size":167370,"upload-time":"2016-05-15T06:37:34.104073Z","url":"https://files.pythonhosted.org/packages/4d/bc/f49882e8935f147b8922fc8bb0f430fe0e7b0d3231a601cd12e1c0272f77/psutil-4.2.0-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"467235151a7ca9506acf3ca52432d5a215f42e2fe44c043d0a0068e133a957dd"},"data-dist-info-metadata":{"sha256":"467235151a7ca9506acf3ca52432d5a215f42e2fe44c043d0a0068e133a957dd"},"filename":"psutil-4.2.0-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"bd4b535996d06728b50bc7cd8777c402bf7294ad05229c843701bd1e63583c2c"},"provenance":null,"requires-python":null,"size":170689,"upload-time":"2016-05-15T06:37:39.395642Z","url":"https://files.pythonhosted.org/packages/7b/e2/2e1078a38189d51409f50af50b598309a2bd84ebe8ca71b79515da915c82/psutil-4.2.0-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.2.0.tar.gz","hashes":{"sha256":"544f013a0aea7199e07e3efe5627f5d4165179a04c66050b234cc3be2eca1ace"},"provenance":null,"requires-python":null,"size":311767,"upload-time":"2016-05-15T06:35:49.367304Z","url":"https://files.pythonhosted.org/packages/a6/bf/5ce23dc9f50de662af3b4bf54812438c298634224924c4e18b7c3b57a2aa/psutil-4.2.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.2.0.win-amd64-py2.7.exe","hashes":{"sha256":"1329160e09a86029ef4e07f47dbcc39d511c343257a53acf1af429c537caae57"},"provenance":null,"requires-python":null,"size":406095,"upload-time":"2016-05-15T06:37:44.994723Z","url":"https://files.pythonhosted.org/packages/6b/af/9e43a4a4976f1d1291de8be40c848c591c6e48d7e4053a7b26ad88ba750c/psutil-4.2.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.2.0.win-amd64-py3.3.exe","hashes":{"sha256":"12623a1e2e264eac8c899b89d78648e241c12eec754a879453b2e0a4a78b10dd"},"provenance":null,"requires-python":null,"size":404495,"upload-time":"2016-05-15T06:37:50.698273Z","url":"https://files.pythonhosted.org/packages/36/85/64244b1e930aa276205f079ba3e2996e8492bd173af019bbdaee47336a6a/psutil-4.2.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.2.0.win-amd64-py3.4.exe","hashes":{"sha256":"66a4a7793dc543a3c7413cda3187e3ced45acf302f95c4d596ebcfc663c01b40"},"provenance":null,"requires-python":null,"size":404456,"upload-time":"2016-05-15T06:37:56.103096Z","url":"https://files.pythonhosted.org/packages/30/fa/a734058699f351ef90b757e0fd8d67a6145c8272bbed85c498276acacd2e/psutil-4.2.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.2.0.win-amd64-py3.5.exe","hashes":{"sha256":"c2b7aa0a99b06967fb76e83e7e9c7153a2d9a5df073986a99a4e9656cfaabe28"},"provenance":null,"requires-python":null,"size":321489,"upload-time":"2016-05-15T06:38:02.113109Z","url":"https://files.pythonhosted.org/packages/a9/1b/c90c802a6db438aeebac412ac3ecf389022f4fd93abef9c0441358f46a71/psutil-4.2.0.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.2.0.win32-py2.7.exe","hashes":{"sha256":"856480ce003ecd1601bcb83d97e25bfe79f5b08c430ee9f139a5e768173b06ef"},"provenance":null,"requires-python":null,"size":375917,"upload-time":"2016-05-15T06:38:09.945181Z","url":"https://files.pythonhosted.org/packages/e4/63/267b0977027c8a4a2f98a1ffbc2ecc7c0689d12adabee591a1ac99b4c14e/psutil-4.2.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.2.0.win32-py3.3.exe","hashes":{"sha256":"ab83fefffa495813d36300cd3ad3f232cf7c86a5e5a02d8e8ea7ab7dba5a1a90"},"provenance":null,"requires-python":null,"size":370860,"upload-time":"2016-05-15T06:38:16.360218Z","url":"https://files.pythonhosted.org/packages/c0/96/8197557cbebb16be1cfd3c87f1d0972bd2e5b0733b21d0e5d890541634e8/psutil-4.2.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.2.0.win32-py3.4.exe","hashes":{"sha256":"44c9f0e26b93c2cc9437eb88c31df32bd4337c394a959e0c31bf006da6e0f073"},"provenance":null,"requires-python":null,"size":370872,"upload-time":"2016-05-15T06:38:22.312293Z","url":"https://files.pythonhosted.org/packages/22/c9/01646a50e3c52dda4b591aae411e85afc83952107f9906ec8a5806c9fcc0/psutil-4.2.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.2.0.win32-py3.5.exe","hashes":{"sha256":"c0013a6663b794fbe18284e06d4d553a9e2135b5489a2ac6982ad53641966a55"},"provenance":null,"requires-python":null,"size":311007,"upload-time":"2016-05-15T06:38:28.533757Z","url":"https://files.pythonhosted.org/packages/48/6f/2259000fa07a4efcdc843034810eb2f8675ef5b97845912f2265589483f4/psutil-4.2.0.win32-py3.5.exe","yanked":false},{"core-metadata":{"sha256":"e4d7d684395672838b7bb0e9173a41bda97a9a103de6473eecda8beed8de23f6"},"data-dist-info-metadata":{"sha256":"e4d7d684395672838b7bb0e9173a41bda97a9a103de6473eecda8beed8de23f6"},"filename":"psutil-4.3.0-cp27-none-win32.whl","hashes":{"sha256":"99c2ab6c8f0d60e0c86775f8e5844e266af48cc1d9ecd1be209cd407a3e9c9a1"},"provenance":null,"requires-python":null,"size":167400,"upload-time":"2016-06-18T17:56:45.275403Z","url":"https://files.pythonhosted.org/packages/7a/a5/002caeac2ff88526cf0788315bad93be61e66477acd54209fb01cc874745/psutil-4.3.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"e4d7d684395672838b7bb0e9173a41bda97a9a103de6473eecda8beed8de23f6"},"data-dist-info-metadata":{"sha256":"e4d7d684395672838b7bb0e9173a41bda97a9a103de6473eecda8beed8de23f6"},"filename":"psutil-4.3.0-cp27-none-win_amd64.whl","hashes":{"sha256":"a91474d34bf1bc86a0d95e2c198a70723208f9dc9e50258c2060a1bab3796f81"},"provenance":null,"requires-python":null,"size":169895,"upload-time":"2016-06-18T17:56:51.386191Z","url":"https://files.pythonhosted.org/packages/e1/77/fae92fff4ca7092555c7c8fc6e02c5dfc2a9af7e15762b7354d436adeb06/psutil-4.3.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"21e625db3c2ab1770665e4dd336de5c07e12cfa6636d671163b7e4021a1a4341"},"data-dist-info-metadata":{"sha256":"21e625db3c2ab1770665e4dd336de5c07e12cfa6636d671163b7e4021a1a4341"},"filename":"psutil-4.3.0-cp33-cp33m-win32.whl","hashes":{"sha256":"c987f0691c01cbe81813b0c895208c474240c96e26f7d1e945e8dabee5c85437"},"provenance":null,"requires-python":null,"size":167414,"upload-time":"2016-06-18T17:56:57.419709Z","url":"https://files.pythonhosted.org/packages/73/46/224a6a3c05df4e282240826f6bef0bd51da5ca283ffe34ed53f5601fbca1/psutil-4.3.0-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"21e625db3c2ab1770665e4dd336de5c07e12cfa6636d671163b7e4021a1a4341"},"data-dist-info-metadata":{"sha256":"21e625db3c2ab1770665e4dd336de5c07e12cfa6636d671163b7e4021a1a4341"},"filename":"psutil-4.3.0-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"233a943d3e6636d648f05515a4d21b4dda63499d8ca38d6890a57a3f78a9cceb"},"provenance":null,"requires-python":null,"size":169791,"upload-time":"2016-06-18T17:57:03.096187Z","url":"https://files.pythonhosted.org/packages/6b/4d/29b3d73c27cd8f348bb313376dd98a2e97cc8a075862cf483dea4c27e4bf/psutil-4.3.0-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"21e625db3c2ab1770665e4dd336de5c07e12cfa6636d671163b7e4021a1a4341"},"data-dist-info-metadata":{"sha256":"21e625db3c2ab1770665e4dd336de5c07e12cfa6636d671163b7e4021a1a4341"},"filename":"psutil-4.3.0-cp34-cp34m-win32.whl","hashes":{"sha256":"1893fe42b0fb5f11bf84ffe770be5b2e27fb7ec959ba8bf620b704552b738c72"},"provenance":null,"requires-python":null,"size":167396,"upload-time":"2016-06-18T17:57:09.859528Z","url":"https://files.pythonhosted.org/packages/dc/f5/34070d328a578c38131d96c4fe539ebbabf1c31128012755e344c030bb34/psutil-4.3.0-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"21e625db3c2ab1770665e4dd336de5c07e12cfa6636d671163b7e4021a1a4341"},"data-dist-info-metadata":{"sha256":"21e625db3c2ab1770665e4dd336de5c07e12cfa6636d671163b7e4021a1a4341"},"filename":"psutil-4.3.0-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"1345217075dd5bb4fecbf7cb0fe4c0c170e93ec57d48494756f4b617cd21b449"},"provenance":null,"requires-python":null,"size":169759,"upload-time":"2016-06-18T17:57:15.923623Z","url":"https://files.pythonhosted.org/packages/8d/91/7ae0835ae1a4bc1043b565dec9c6468d56c80a4f199472a7d005ddcd48e1/psutil-4.3.0-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"21e625db3c2ab1770665e4dd336de5c07e12cfa6636d671163b7e4021a1a4341"},"data-dist-info-metadata":{"sha256":"21e625db3c2ab1770665e4dd336de5c07e12cfa6636d671163b7e4021a1a4341"},"filename":"psutil-4.3.0-cp35-cp35m-win32.whl","hashes":{"sha256":"c9c4274f5f95a171437c90f65c3e9b71a871753f0a827f930e1b14aa43041eab"},"provenance":null,"requires-python":null,"size":169508,"upload-time":"2016-06-18T17:57:21.072109Z","url":"https://files.pythonhosted.org/packages/e4/40/801cd906da337a5e7a0afaaa1ce5919d9834c50d804d31ee4a1d2120a51c/psutil-4.3.0-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"21e625db3c2ab1770665e4dd336de5c07e12cfa6636d671163b7e4021a1a4341"},"data-dist-info-metadata":{"sha256":"21e625db3c2ab1770665e4dd336de5c07e12cfa6636d671163b7e4021a1a4341"},"filename":"psutil-4.3.0-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"5984ee7b2880abcdaa0819315f69a5f37da963863495c2294392cb3e98141a95"},"provenance":null,"requires-python":null,"size":172803,"upload-time":"2016-06-18T17:57:26.746663Z","url":"https://files.pythonhosted.org/packages/35/54/ddb6e8e583abf2f5a2be52a2223ba4935b382214b696fe334af54fb03dad/psutil-4.3.0-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.0.tar.gz","hashes":{"sha256":"86197ae5978f216d33bfff4383d5cc0b80f079d09cf45a2a406d1abb5d0299f0"},"provenance":null,"requires-python":null,"size":316470,"upload-time":"2016-06-18T17:54:55.929749Z","url":"https://files.pythonhosted.org/packages/22/a8/6ab3f0b3b74a36104785808ec874d24203c6a511ffd2732dd215cf32d689/psutil-4.3.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.0.win-amd64-py2.7.exe","hashes":{"sha256":"1ad0075b6c86c0ea5076149ec39dcecf0c692711c34317d43d73a4d8c4d4ec30"},"provenance":null,"requires-python":null,"size":408495,"upload-time":"2016-06-18T17:57:33.655137Z","url":"https://files.pythonhosted.org/packages/d5/1f/638b17eab913d19203ebd721c4e5c726b57bc50def33ab1ec0070fe2ddc2/psutil-4.3.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.0.win-amd64-py3.3.exe","hashes":{"sha256":"02c042e1f6c68c807de5caf45547971cf02977abf4cd92c8961186af9c91c488"},"provenance":null,"requires-python":null,"size":406900,"upload-time":"2016-06-18T17:57:40.580278Z","url":"https://files.pythonhosted.org/packages/a6/9b/94598ec4041ee2834f7ca21c9f1ff67e8f8cef4376ee9d9b9e3ff6950d05/psutil-4.3.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.0.win-amd64-py3.4.exe","hashes":{"sha256":"2bcfeff969f3718bb31effea752d92511f375547d337687db1bd99ccd85b7ad7"},"provenance":null,"requires-python":null,"size":406871,"upload-time":"2016-06-18T17:57:47.827133Z","url":"https://files.pythonhosted.org/packages/79/2a/abe407e5b594b2a7c0432f5605579d33ce21e0049e3dc2e5c4f37402f760/psutil-4.3.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.0.win-amd64-py3.5.exe","hashes":{"sha256":"34736b91c9785ede9d859a79e28129390f609339014f904c276ad46d0a440730"},"provenance":null,"requires-python":null,"size":323883,"upload-time":"2016-06-18T17:57:54.043353Z","url":"https://files.pythonhosted.org/packages/14/5e/072b19d913b3fc1f86b871c2869d19db3b5fa50c2e9f4980ae8646e189e0/psutil-4.3.0.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.0.win32-py2.7.exe","hashes":{"sha256":"da6ee62fb5ffda188c39aacb0499d401c131046e922ba53fb4b908937e771f94"},"provenance":null,"requires-python":null,"size":378354,"upload-time":"2016-06-18T17:58:00.735354Z","url":"https://files.pythonhosted.org/packages/7e/85/a60111c14eb80aa7a74e1a0086c1a1bbc62282df0c913730e558d16e6a8c/psutil-4.3.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.0.win32-py3.3.exe","hashes":{"sha256":"6f36fa29aa9ca935405a3cebe03cfbc6dde27088f9ad3d9b2d3baa47d4b89914"},"provenance":null,"requires-python":null,"size":373297,"upload-time":"2016-06-18T17:58:08.504160Z","url":"https://files.pythonhosted.org/packages/8e/6e/744b98493947a11625a4c63adb9a148c69c2a6e0cf7ef2afd6212804df1d/psutil-4.3.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.0.win32-py3.4.exe","hashes":{"sha256":"63b75c5374fafdaf3f389229b592f19b88a8c7951d1b973b9113df649ade5cb9"},"provenance":null,"requires-python":null,"size":373279,"upload-time":"2016-06-18T17:58:15.187200Z","url":"https://files.pythonhosted.org/packages/87/ce/a3cfd0e1b7d34fccff488a6a2283e5b947841465f81960bd87eee5f78828/psutil-4.3.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.0.win32-py3.5.exe","hashes":{"sha256":"6d0e1e9bbdab5a7d40b57d8f0841eb9e11c2395ac7ec0ddd396116240d733f46"},"provenance":null,"requires-python":null,"size":313430,"upload-time":"2016-06-18T17:58:22.261628Z","url":"https://files.pythonhosted.org/packages/eb/23/d01e7eccd76a096b0675ab9d07c8fd2de9fd31bc8f4c92f1a150bc612ad4/psutil-4.3.0.win32-py3.5.exe","yanked":false},{"core-metadata":{"sha256":"f2eb91a4c96ace2af993ac8587ddb20d3ed535289e9c5a0fde7a8c3d67ae0e12"},"data-dist-info-metadata":{"sha256":"f2eb91a4c96ace2af993ac8587ddb20d3ed535289e9c5a0fde7a8c3d67ae0e12"},"filename":"psutil-4.3.1-cp27-none-win32.whl","hashes":{"sha256":"b0c5bf0d2a29a6f18ac22e2d24210730dca458c9f961914289c9e027ccb5ae43"},"provenance":null,"requires-python":null,"size":168476,"upload-time":"2016-09-02T13:58:43.429910Z","url":"https://files.pythonhosted.org/packages/2d/70/0b24c7272efbb1d8cac1be1768aabfb8ddb37bdc9ab8a176f6afc7e52b0d/psutil-4.3.1-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"f2eb91a4c96ace2af993ac8587ddb20d3ed535289e9c5a0fde7a8c3d67ae0e12"},"data-dist-info-metadata":{"sha256":"f2eb91a4c96ace2af993ac8587ddb20d3ed535289e9c5a0fde7a8c3d67ae0e12"},"filename":"psutil-4.3.1-cp27-none-win_amd64.whl","hashes":{"sha256":"fc78c29075e623b6ea1c4a1620a120a1534ee05370b76c0ec96f6d161d79e7a1"},"provenance":null,"requires-python":null,"size":170725,"upload-time":"2016-09-02T13:58:47.944701Z","url":"https://files.pythonhosted.org/packages/67/d4/0403e6bd1cf78bd597ac960a3a6ad36cea6c12e3b413c0a1d43361128fb5/psutil-4.3.1-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"2bf14b2dd8f24b0dedf17492f736d534043ed69d60a0cf1d5a8925a8a351061a"},"data-dist-info-metadata":{"sha256":"2bf14b2dd8f24b0dedf17492f736d534043ed69d60a0cf1d5a8925a8a351061a"},"filename":"psutil-4.3.1-cp33-cp33m-win32.whl","hashes":{"sha256":"aa05f44a77ef83773af39446f99e461aa3b6edb7fdabeefdcf06e913d8884d3a"},"provenance":null,"requires-python":null,"size":168384,"upload-time":"2016-09-02T13:58:52.479199Z","url":"https://files.pythonhosted.org/packages/9c/ec/5f3f06012c54de3b4443a6948fc75fe1e348ca3de408b00815b5976b8877/psutil-4.3.1-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"2bf14b2dd8f24b0dedf17492f736d534043ed69d60a0cf1d5a8925a8a351061a"},"data-dist-info-metadata":{"sha256":"2bf14b2dd8f24b0dedf17492f736d534043ed69d60a0cf1d5a8925a8a351061a"},"filename":"psutil-4.3.1-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"6b3882eb16f2f40f1da6208a051800abadb1f82a675d9ef6ca7386e1a208b1ad"},"provenance":null,"requires-python":null,"size":170566,"upload-time":"2016-09-02T13:58:56.870812Z","url":"https://files.pythonhosted.org/packages/de/c6/74f8b4d460d89811b4c8fd426b63530222456bad767fe372ebb4f5f207be/psutil-4.3.1-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"2bf14b2dd8f24b0dedf17492f736d534043ed69d60a0cf1d5a8925a8a351061a"},"data-dist-info-metadata":{"sha256":"2bf14b2dd8f24b0dedf17492f736d534043ed69d60a0cf1d5a8925a8a351061a"},"filename":"psutil-4.3.1-cp34-cp34m-win32.whl","hashes":{"sha256":"cf1be0b16b38f0e2081ff0c81a1a4321c206a824ba6bd51903fdd440abb370b6"},"provenance":null,"requires-python":null,"size":168375,"upload-time":"2016-09-02T13:59:01.079445Z","url":"https://files.pythonhosted.org/packages/75/ff/d02c907869d5e4cc260ce72eb253f5007a7cdf0b47326d19693f8f937eb0/psutil-4.3.1-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"2bf14b2dd8f24b0dedf17492f736d534043ed69d60a0cf1d5a8925a8a351061a"},"data-dist-info-metadata":{"sha256":"2bf14b2dd8f24b0dedf17492f736d534043ed69d60a0cf1d5a8925a8a351061a"},"filename":"psutil-4.3.1-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"afa94bed972722882264a4df06176f6b6e6acc6bcebcc3f1db5428c7271dacba"},"provenance":null,"requires-python":null,"size":170529,"upload-time":"2016-09-02T13:59:05.507453Z","url":"https://files.pythonhosted.org/packages/c9/72/07da416b1dcf258a0cb0587c823e3611392fe29b1fcae6e078b1c254dce5/psutil-4.3.1-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"2bf14b2dd8f24b0dedf17492f736d534043ed69d60a0cf1d5a8925a8a351061a"},"data-dist-info-metadata":{"sha256":"2bf14b2dd8f24b0dedf17492f736d534043ed69d60a0cf1d5a8925a8a351061a"},"filename":"psutil-4.3.1-cp35-cp35m-win32.whl","hashes":{"sha256":"d2254f518624e6b2262f0f878931faa4bdbe8a77d1f8826564bc4576c6a4f85e"},"provenance":null,"requires-python":null,"size":170124,"upload-time":"2016-09-02T13:59:09.837050Z","url":"https://files.pythonhosted.org/packages/cd/b0/07b7083a134c43b58515d59f271734034f8ba06840b1f371eaa6b3ab85b2/psutil-4.3.1-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"2bf14b2dd8f24b0dedf17492f736d534043ed69d60a0cf1d5a8925a8a351061a"},"data-dist-info-metadata":{"sha256":"2bf14b2dd8f24b0dedf17492f736d534043ed69d60a0cf1d5a8925a8a351061a"},"filename":"psutil-4.3.1-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"3b377bc8ba5e62adbc709a90ea07dce2d4addbd6e1cc7acede61ddfa1c66e00a"},"provenance":null,"requires-python":null,"size":173545,"upload-time":"2016-09-02T13:59:14.368908Z","url":"https://files.pythonhosted.org/packages/7e/7e/17c1467158ccac5dc54986a657420fc194686653cfb6feddb6717a60d17f/psutil-4.3.1-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.1.tar.gz","hashes":{"sha256":"38f74182fb9e15cafd0cdf0821098a95cc17301807aed25634a18b66537ba51b"},"provenance":null,"requires-python":null,"size":315878,"upload-time":"2016-09-01T20:56:06.777431Z","url":"https://files.pythonhosted.org/packages/78/cc/f267a1371f229bf16db6a4e604428c3b032b823b83155bd33cef45e49a53/psutil-4.3.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.1.win-amd64-py2.7.exe","hashes":{"sha256":"733210f39e95744da26f2256bc36035fc463b0ae88e91496e97486ba21c63cab"},"provenance":null,"requires-python":null,"size":408715,"upload-time":"2016-09-01T21:31:30.452539Z","url":"https://files.pythonhosted.org/packages/57/44/09ec7b6a3fc1216ecc2182a64f8b6c2eaf8dd0d983fb98ecbf9cecbf54b4/psutil-4.3.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.1.win-amd64-py3.3.exe","hashes":{"sha256":"4690f720054beff4fc66551a6a34512faff328588dca8e2dbed94398b6941112"},"provenance":null,"requires-python":null,"size":407063,"upload-time":"2016-09-01T21:31:37.405818Z","url":"https://files.pythonhosted.org/packages/f2/b5/2921abbf7779d2f7bb210aa819dd2d86ecd004430c59aef9cc52c2d4057b/psutil-4.3.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.1.win-amd64-py3.4.exe","hashes":{"sha256":"fd9b66edb9f8943eda6b39e7bb9bff8b14aa8d785f5b417d7a0bfa53d4781a7a"},"provenance":null,"requires-python":null,"size":407028,"upload-time":"2016-09-01T21:31:44.758391Z","url":"https://files.pythonhosted.org/packages/91/ea/d5fb1ef4615c0febab3a347ffbc98d50177878546c22b3291892f41de8f4/psutil-4.3.1.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.1.win-amd64-py3.5.exe","hashes":{"sha256":"9ab5b62c6571ce545b1c40b9740af81276bd5d94439fd54de07ed59be0ce3f4f"},"provenance":null,"requires-python":null,"size":777648,"upload-time":"2016-09-01T21:31:56.074229Z","url":"https://files.pythonhosted.org/packages/0d/d9/626030b223140d88176b5dda42ea081b843249af11fa71d9e36a465b4b18/psutil-4.3.1.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.1.win32-py2.7.exe","hashes":{"sha256":"ad8857923e9bc5802d5559ab5d70c1abc1a7be8e74e779adde883c5391e2061c"},"provenance":null,"requires-python":null,"size":378819,"upload-time":"2016-09-01T21:30:59.795815Z","url":"https://files.pythonhosted.org/packages/02/99/2cc2981b30b2b1fb5b393c921ad17e4baaa1f95d7de527fe54ee222e5663/psutil-4.3.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.1.win32-py3.3.exe","hashes":{"sha256":"ae20b76cddb3391ea37de5d2aaa1656d6373161bbc8fd868a0ca055194a46e45"},"provenance":null,"requires-python":null,"size":373653,"upload-time":"2016-09-01T21:31:06.076800Z","url":"https://files.pythonhosted.org/packages/06/2f/bbcae933425945d2c9ca3e30e2f35728827e87cedad113808d3f527fe2ea/psutil-4.3.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.1.win32-py3.4.exe","hashes":{"sha256":"0613437cc28b8721de92c582d5baf742dfa6dd824c84b578f8c49a60077e969a"},"provenance":null,"requires-python":null,"size":373650,"upload-time":"2016-09-01T21:31:12.717399Z","url":"https://files.pythonhosted.org/packages/fb/a6/28bca55c499426202341f8224f66fcd69e6577211201b02605be779f44b1/psutil-4.3.1.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.3.1.win32-py3.5.exe","hashes":{"sha256":"c2031732cd0fb7536af491bb8d8119c9263020a52450f9999c884fd49d346b26"},"provenance":null,"requires-python":null,"size":644701,"upload-time":"2016-09-01T21:31:22.237380Z","url":"https://files.pythonhosted.org/packages/f3/c7/f5da6e76f69c97ecfabe684f4d9e179ceaf9f7c1a74d6148480b1a6c41a8/psutil-4.3.1.win32-py3.5.exe","yanked":false},{"core-metadata":{"sha256":"73b13e96cdf0d35ce81d30eb0c2e6b9cbfea41ee45b898f54d9e832e8e3ddd92"},"data-dist-info-metadata":{"sha256":"73b13e96cdf0d35ce81d30eb0c2e6b9cbfea41ee45b898f54d9e832e8e3ddd92"},"filename":"psutil-4.4.0-cp27-none-win32.whl","hashes":{"sha256":"4b907a0bed62a76422eae4e1ed8c8eca25fc21e57a31fc080158b8a300e21dad"},"provenance":null,"requires-python":null,"size":172746,"upload-time":"2016-10-23T14:08:37.739179Z","url":"https://files.pythonhosted.org/packages/8b/00/f7203827a3b2576bf8162ea3ba41e39c3307651218ab08e22e1433d8dc36/psutil-4.4.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"73b13e96cdf0d35ce81d30eb0c2e6b9cbfea41ee45b898f54d9e832e8e3ddd92"},"data-dist-info-metadata":{"sha256":"73b13e96cdf0d35ce81d30eb0c2e6b9cbfea41ee45b898f54d9e832e8e3ddd92"},"filename":"psutil-4.4.0-cp27-none-win_amd64.whl","hashes":{"sha256":"f60ab95f5e65c420743d5dd5285bda2a6bba6712e9380fb9a5903ea539507326"},"provenance":null,"requires-python":null,"size":175102,"upload-time":"2016-10-23T14:08:40.758084Z","url":"https://files.pythonhosted.org/packages/61/5c/d1f89100973829813e485e7bfc2f203e0d11937f958109819b62e8995b51/psutil-4.4.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ee9b7c8ec564a66dc6543830d64d32a37387ea58517feba35aa4eefeb15ce5de"},"data-dist-info-metadata":{"sha256":"ee9b7c8ec564a66dc6543830d64d32a37387ea58517feba35aa4eefeb15ce5de"},"filename":"psutil-4.4.0-cp33-cp33m-win32.whl","hashes":{"sha256":"d45919d8b900a9ae03f3d43c489323842d4051cf7a728169f01a3889f50d24ad"},"provenance":null,"requires-python":null,"size":172612,"upload-time":"2016-10-23T14:08:44.422307Z","url":"https://files.pythonhosted.org/packages/34/16/ecc7366a7d023731b8b051cff64d3bb4bed2efa4947b125a3c9031e3d799/psutil-4.4.0-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"ee9b7c8ec564a66dc6543830d64d32a37387ea58517feba35aa4eefeb15ce5de"},"data-dist-info-metadata":{"sha256":"ee9b7c8ec564a66dc6543830d64d32a37387ea58517feba35aa4eefeb15ce5de"},"filename":"psutil-4.4.0-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"df706e4a8533b43c1a083c2c94e816c7a605487db49e5d49fc64329e0147c0f5"},"provenance":null,"requires-python":null,"size":174980,"upload-time":"2016-10-23T14:08:48.077823Z","url":"https://files.pythonhosted.org/packages/f7/17/6fc596007c243a12cf8c9e81b150f1e7a49c7c85c0544771398f257d5608/psutil-4.4.0-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ee9b7c8ec564a66dc6543830d64d32a37387ea58517feba35aa4eefeb15ce5de"},"data-dist-info-metadata":{"sha256":"ee9b7c8ec564a66dc6543830d64d32a37387ea58517feba35aa4eefeb15ce5de"},"filename":"psutil-4.4.0-cp34-cp34m-win32.whl","hashes":{"sha256":"d119281d253bbcc44b491b7b7e5c38802f0933179af97ab228cfd8d072ad1503"},"provenance":null,"requires-python":null,"size":172623,"upload-time":"2016-10-23T14:08:50.867277Z","url":"https://files.pythonhosted.org/packages/50/8f/e8fcfb1bacd02c743545d58514004978318d8476c8bffeee9fecfc9f1f79/psutil-4.4.0-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"ee9b7c8ec564a66dc6543830d64d32a37387ea58517feba35aa4eefeb15ce5de"},"data-dist-info-metadata":{"sha256":"ee9b7c8ec564a66dc6543830d64d32a37387ea58517feba35aa4eefeb15ce5de"},"filename":"psutil-4.4.0-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"ff8a22a40bf884f52cf8dd86872b5199e7ed58eef1575e837d1d9b668fb2416a"},"provenance":null,"requires-python":null,"size":174974,"upload-time":"2016-10-23T14:08:53.576968Z","url":"https://files.pythonhosted.org/packages/41/8f/31f1fb4bb639ccaff2192b0b2a760118a05fb108e156d4966c81a246071b/psutil-4.4.0-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ee9b7c8ec564a66dc6543830d64d32a37387ea58517feba35aa4eefeb15ce5de"},"data-dist-info-metadata":{"sha256":"ee9b7c8ec564a66dc6543830d64d32a37387ea58517feba35aa4eefeb15ce5de"},"filename":"psutil-4.4.0-cp35-cp35m-win32.whl","hashes":{"sha256":"7b7c850b3afe6c7895cfd3ee630492c4aabe644a6723a583cd56bac0222d7cd8"},"provenance":null,"requires-python":null,"size":174516,"upload-time":"2016-10-23T14:08:56.521588Z","url":"https://files.pythonhosted.org/packages/18/9e/702b1da450a13448f9d96f2d9e2f69eed901c56adb01e173d8307278c883/psutil-4.4.0-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"ee9b7c8ec564a66dc6543830d64d32a37387ea58517feba35aa4eefeb15ce5de"},"data-dist-info-metadata":{"sha256":"ee9b7c8ec564a66dc6543830d64d32a37387ea58517feba35aa4eefeb15ce5de"},"filename":"psutil-4.4.0-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"321d09e39bb4641c98544e51fba598f894d09355a18d14367468723632cb149e"},"provenance":null,"requires-python":null,"size":178046,"upload-time":"2016-10-23T14:08:59.346105Z","url":"https://files.pythonhosted.org/packages/8a/87/a22b95d91fb1a07591ebf373d1c286a16037bf2e321577a083156d7f6a13/psutil-4.4.0-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.0.tar.gz","hashes":{"sha256":"f4da111f473dbf7e813e6610aec1329000536aea5e7d7e73ed20bc42cfda7ecc"},"provenance":null,"requires-python":null,"size":1831734,"upload-time":"2016-10-23T14:09:04.179623Z","url":"https://files.pythonhosted.org/packages/fc/63/af9c6a4f2ab48293f60ec204cc1336f6f17c1cb782ffb0275982ac08d663/psutil-4.4.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.0.win-amd64-py2.7.exe","hashes":{"sha256":"184132916ceb845f12f7cced3a2cf5273097d314f44be8357bdc435a6dee49cd"},"provenance":null,"requires-python":null,"size":413868,"upload-time":"2016-10-23T14:09:10.236158Z","url":"https://files.pythonhosted.org/packages/98/d9/407ee7d17b4e78c45b92639bfbd9b376b6a6022b0c63965946bfd74af6f4/psutil-4.4.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.0.win-amd64-py3.3.exe","hashes":{"sha256":"7bb1c69a062aaff4a1773cca6bbe33b261293318d72246b9519357260b67be1e"},"provenance":null,"requires-python":null,"size":412259,"upload-time":"2016-10-23T14:09:15.072697Z","url":"https://files.pythonhosted.org/packages/3f/cc/b31fbc28247722ae7ece86f8911459e29a08f5075e3d4e794cef20f920b2/psutil-4.4.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.0.win-amd64-py3.4.exe","hashes":{"sha256":"887eca39665d3de362693d66db9c15bf93313cde261de93908452241ee439e56"},"provenance":null,"requires-python":null,"size":412252,"upload-time":"2016-10-23T14:09:19.006281Z","url":"https://files.pythonhosted.org/packages/32/72/88e9e57964144ac3868f817f77e269f987c74fa51e503eb0fe8040523d28/psutil-4.4.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.0.win-amd64-py3.5.exe","hashes":{"sha256":"a289ed4f7be6c535aa4c59c997baf327788d564881c8bf08fee502fab0cdc7cb"},"provenance":null,"requires-python":null,"size":782926,"upload-time":"2016-10-23T14:09:24.493051Z","url":"https://files.pythonhosted.org/packages/45/09/8908f5931a78c90bf2ef7f91a37a752a13d4017347ac2765d0cbb89e82f6/psutil-4.4.0.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.0.win32-py2.7.exe","hashes":{"sha256":"8a18d0527ac339f5501f0bd5471040d5fb83052453eedf13106feaf99ddef6cb"},"provenance":null,"requires-python":null,"size":383869,"upload-time":"2016-10-23T14:09:28.382095Z","url":"https://files.pythonhosted.org/packages/4a/eb/a131c438621822833e3131c4998741c3b68fb6ff67786d25e9398439a640/psutil-4.4.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.0.win32-py3.3.exe","hashes":{"sha256":"8700a0373fd0c95c79ad79bc08e0a7b76cabc0c463658cbe398600c69d57293d"},"provenance":null,"requires-python":null,"size":378663,"upload-time":"2016-10-23T14:09:32.358639Z","url":"https://files.pythonhosted.org/packages/be/c8/fd76b7ba3e1cc61d586c5779d9b0a45d1ad020e1de677eb344d2583bd0f1/psutil-4.4.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.0.win32-py3.4.exe","hashes":{"sha256":"ddd3f76486a366f1dc7adc8a9e8a285ec24b3b213912b8dce0cbbb629f954b8c"},"provenance":null,"requires-python":null,"size":378674,"upload-time":"2016-10-23T14:09:35.849351Z","url":"https://files.pythonhosted.org/packages/ce/0c/efbe5aefdd136bb3e2fc292b0506761a9b04c215d295be07e11263a90923/psutil-4.4.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.0.win32-py3.5.exe","hashes":{"sha256":"28963853709f6e0958edd8d6d7469152098680800deb98a759a67745d65b164b"},"provenance":null,"requires-python":null,"size":649869,"upload-time":"2016-10-23T14:09:40.045794Z","url":"https://files.pythonhosted.org/packages/fa/b6/b97b6d4dcbc87b91db9fd2753a0c3b999b734595a97155a38003be475a6d/psutil-4.4.0.win32-py3.5.exe","yanked":false},{"core-metadata":{"sha256":"b5aa70030cb34744d7b8b2639183dd13ebd01c60f592c3b0c5e6fd6d8086bcd3"},"data-dist-info-metadata":{"sha256":"b5aa70030cb34744d7b8b2639183dd13ebd01c60f592c3b0c5e6fd6d8086bcd3"},"filename":"psutil-4.4.1-cp27-none-win32.whl","hashes":{"sha256":"7e77ec1a9c75a858781c1fb46fe81c999e1ae0e711198b4aaf59e5f5bd373b11"},"provenance":null,"requires-python":null,"size":172150,"upload-time":"2016-10-25T15:16:10.656877Z","url":"https://files.pythonhosted.org/packages/a9/aa/ff4e5602420cda42e9ff12949eae95abf7d6838fc79d7892b29af416f4c2/psutil-4.4.1-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"b5aa70030cb34744d7b8b2639183dd13ebd01c60f592c3b0c5e6fd6d8086bcd3"},"data-dist-info-metadata":{"sha256":"b5aa70030cb34744d7b8b2639183dd13ebd01c60f592c3b0c5e6fd6d8086bcd3"},"filename":"psutil-4.4.1-cp27-none-win_amd64.whl","hashes":{"sha256":"0f7f830db35c1baeb0131b2bba458b77f7db98944b2fedafc34922168e467d09"},"provenance":null,"requires-python":null,"size":174507,"upload-time":"2016-10-25T15:16:16.530462Z","url":"https://files.pythonhosted.org/packages/62/e8/c3d3e4161e29bd86d6b06d16456defcb114fd74693b15e5692df9e2b611e/psutil-4.4.1-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"979b8a1231ad076b4475296c0795b1310188d8e8fda89e2785b4da3efbf4ad52"},"data-dist-info-metadata":{"sha256":"979b8a1231ad076b4475296c0795b1310188d8e8fda89e2785b4da3efbf4ad52"},"filename":"psutil-4.4.1-cp33-cp33m-win32.whl","hashes":{"sha256":"fa4ad0533adef033bbcbac5e20d06e77f9aadf5d9c1596317d1b668f94b01b99"},"provenance":null,"requires-python":null,"size":172015,"upload-time":"2016-10-25T15:16:19.320410Z","url":"https://files.pythonhosted.org/packages/2c/da/0428f61f2c5eda7c5053ae35a2d57b097e4eb15e48f9d222319df7f4cbd3/psutil-4.4.1-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"979b8a1231ad076b4475296c0795b1310188d8e8fda89e2785b4da3efbf4ad52"},"data-dist-info-metadata":{"sha256":"979b8a1231ad076b4475296c0795b1310188d8e8fda89e2785b4da3efbf4ad52"},"filename":"psutil-4.4.1-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"af337d186b07249b86f00a71b4cf6fcfa1964484fe5fb8a7b623f4559c2859c9"},"provenance":null,"requires-python":null,"size":174384,"upload-time":"2016-10-25T15:16:22.634310Z","url":"https://files.pythonhosted.org/packages/da/ce/d36c39da6d387fbfaea68f382319887315e9201e885ef93c418b89209b31/psutil-4.4.1-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"979b8a1231ad076b4475296c0795b1310188d8e8fda89e2785b4da3efbf4ad52"},"data-dist-info-metadata":{"sha256":"979b8a1231ad076b4475296c0795b1310188d8e8fda89e2785b4da3efbf4ad52"},"filename":"psutil-4.4.1-cp34-cp34m-win32.whl","hashes":{"sha256":"4e5cb45c9616dd855c07e538f523c838705ce7c24e021e645cdce4c7894e7209"},"provenance":null,"requires-python":null,"size":172026,"upload-time":"2016-10-25T15:16:28.801885Z","url":"https://files.pythonhosted.org/packages/64/63/49aeca2d1a20f5c5203302a25c825be9262f81d8bf74d7d9e0bf0789b189/psutil-4.4.1-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"979b8a1231ad076b4475296c0795b1310188d8e8fda89e2785b4da3efbf4ad52"},"data-dist-info-metadata":{"sha256":"979b8a1231ad076b4475296c0795b1310188d8e8fda89e2785b4da3efbf4ad52"},"filename":"psutil-4.4.1-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"0876646748f3db5e1678a94ae3f68dcef3bd51e82b34f06109e6a28bcddc266c"},"provenance":null,"requires-python":null,"size":174374,"upload-time":"2016-10-25T15:16:32.462229Z","url":"https://files.pythonhosted.org/packages/e0/b2/48a62d3204a714b6354105ed540d54d1e272d5b230bc35eddc14a1494dd1/psutil-4.4.1-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"979b8a1231ad076b4475296c0795b1310188d8e8fda89e2785b4da3efbf4ad52"},"data-dist-info-metadata":{"sha256":"979b8a1231ad076b4475296c0795b1310188d8e8fda89e2785b4da3efbf4ad52"},"filename":"psutil-4.4.1-cp35-cp35m-win32.whl","hashes":{"sha256":"e4e9033ef5d775ef8a522750688161241e79c7d4669a05784a0a1a8d37dc8c3c"},"provenance":null,"requires-python":null,"size":173920,"upload-time":"2016-10-25T15:16:38.218083Z","url":"https://files.pythonhosted.org/packages/95/e6/d6bad966efeb674bb3357c94f3e89bbd3477a24f2635c3cadc1d0f2aea76/psutil-4.4.1-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"979b8a1231ad076b4475296c0795b1310188d8e8fda89e2785b4da3efbf4ad52"},"data-dist-info-metadata":{"sha256":"979b8a1231ad076b4475296c0795b1310188d8e8fda89e2785b4da3efbf4ad52"},"filename":"psutil-4.4.1-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"d8464100c62932eeeacce2be0a1041f68b3bfcc7be261cf9486c11ee98eaedd2"},"provenance":null,"requires-python":null,"size":177448,"upload-time":"2016-10-25T15:16:42.121191Z","url":"https://files.pythonhosted.org/packages/00/58/6845a2a2f6fbbc56a58f0605744368164bf68889299f134684cc1478baf6/psutil-4.4.1-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.1.tar.gz","hashes":{"sha256":"9da43dbf7c08f5c2a8e5e2c8792f5c438f52435677b1334e9653d23ea028f4f7"},"provenance":null,"requires-python":null,"size":1831794,"upload-time":"2016-10-25T15:16:47.508715Z","url":"https://files.pythonhosted.org/packages/e5/f3/b816daefa9a6757f867f81903f40849dcf0887f588793236b476e6a30ded/psutil-4.4.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.1.win-amd64-py2.7.exe","hashes":{"sha256":"23ac3e7b6784751ceeb4338d54b0e683c955cd22b86acbc089696aeb0717ab75"},"provenance":null,"requires-python":null,"size":408863,"upload-time":"2016-10-25T15:16:52.293269Z","url":"https://files.pythonhosted.org/packages/88/93/3c8434bd64a5f89f6b0cc58e1394e88dc3c272f83f1877afac9ede222c63/psutil-4.4.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.1.win-amd64-py3.3.exe","hashes":{"sha256":"8b97ece77d2dce49dd6adbfc0c9bfb7820be4460d00732bb8cf18b77b9ffb07f"},"provenance":null,"requires-python":null,"size":407253,"upload-time":"2016-10-25T15:16:57.006499Z","url":"https://files.pythonhosted.org/packages/50/21/6b462342ab2f091bed70898894322c107493d69e24a834b9f2ab93a1e876/psutil-4.4.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.1.win-amd64-py3.4.exe","hashes":{"sha256":"a0b5976a5b5a781754b6cebe89f4e21413b04b7f4005f5713f65257bee29be5f"},"provenance":null,"requires-python":null,"size":407244,"upload-time":"2016-10-25T15:17:01.612657Z","url":"https://files.pythonhosted.org/packages/09/94/a7c3c875884adef49f52c19faff68bfd2b04160a5eb0489dbffa60b5c2dc/psutil-4.4.1.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.1.win-amd64-py3.5.exe","hashes":{"sha256":"b0bfe16b9bd095e56b8ec481328ba64094ccd615ee1f789a12a13cfd7bc5e34a"},"provenance":null,"requires-python":null,"size":777920,"upload-time":"2016-10-25T15:17:05.774786Z","url":"https://files.pythonhosted.org/packages/09/bf/3d850244db8e6aa94116db6787edee352692bbb9647e31c056189e6742cc/psutil-4.4.1.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.1.win32-py2.7.exe","hashes":{"sha256":"392877b5f86977b4a747fb61968e3b44df62c1d6d87536f021c4979b601c68f5"},"provenance":null,"requires-python":null,"size":378863,"upload-time":"2016-10-25T15:17:10.258103Z","url":"https://files.pythonhosted.org/packages/ed/f9/ed4cc19c84086149067d03de7cfb9f3213cf02d58b6f75c2ac16f85e0baf/psutil-4.4.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.1.win32-py3.3.exe","hashes":{"sha256":"a571f179f29215f0aa710ec495573a3522f24e8a8ba0be67d094ad6f593fba05"},"provenance":null,"requires-python":null,"size":373657,"upload-time":"2016-10-25T15:17:17.117641Z","url":"https://files.pythonhosted.org/packages/e6/ad/d8ec058821191334cf17db08c32c0308252d7b3c7490417690061dd12096/psutil-4.4.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.1.win32-py3.4.exe","hashes":{"sha256":"c2614dc6194cafd74c147e4b95febe8778430c2ecb91197ad140bf45e3e3ada7"},"provenance":null,"requires-python":null,"size":373668,"upload-time":"2016-10-25T15:17:20.392896Z","url":"https://files.pythonhosted.org/packages/26/15/6d03a2171262a1be565d06c87d584c10daa51e787c7cdd2ff932a433419e/psutil-4.4.1.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.1.win32-py3.5.exe","hashes":{"sha256":"5915da802a1648d0baeeee06b0d1a87eb0e4e20654f65b8428a6f4cc8cd32caf"},"provenance":null,"requires-python":null,"size":644863,"upload-time":"2016-10-25T15:17:24.141533Z","url":"https://files.pythonhosted.org/packages/d1/91/89db11eaa91c1de34d77600c47d657fce7b60e847bcf6457dddfc23b387c/psutil-4.4.1.win32-py3.5.exe","yanked":false},{"core-metadata":{"sha256":"5a3e00c7b872d8d4c7c640b7cfec9ede44850d18f5d050b57037614e2ec90c72"},"data-dist-info-metadata":{"sha256":"5a3e00c7b872d8d4c7c640b7cfec9ede44850d18f5d050b57037614e2ec90c72"},"filename":"psutil-4.4.2-cp27-none-win32.whl","hashes":{"sha256":"15aba78f0262d7839702913f5d2ce1e97c89e31456bb26da1a5f9f7d7fe6d336"},"provenance":null,"requires-python":null,"size":172169,"upload-time":"2016-10-26T11:01:25.916580Z","url":"https://files.pythonhosted.org/packages/d7/da/b7895f01868977c9205bf1c8ff6a88290ec443535084e206b7db7f33918f/psutil-4.4.2-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"5a3e00c7b872d8d4c7c640b7cfec9ede44850d18f5d050b57037614e2ec90c72"},"data-dist-info-metadata":{"sha256":"5a3e00c7b872d8d4c7c640b7cfec9ede44850d18f5d050b57037614e2ec90c72"},"filename":"psutil-4.4.2-cp27-none-win_amd64.whl","hashes":{"sha256":"69e30d789c495b781f7cd47c13ee64452c58abfc7132d6dd1b389af312a78239"},"provenance":null,"requires-python":null,"size":174525,"upload-time":"2016-10-26T11:01:29.001553Z","url":"https://files.pythonhosted.org/packages/0c/76/f50742570195a9a13e26ee3e3e32575a9315df90408056c991af85d23792/psutil-4.4.2-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"34c5de5faa22e5cdae426206704ec3a7d177ec1d34a20e3ad2d072136ac17b84"},"data-dist-info-metadata":{"sha256":"34c5de5faa22e5cdae426206704ec3a7d177ec1d34a20e3ad2d072136ac17b84"},"filename":"psutil-4.4.2-cp33-cp33m-win32.whl","hashes":{"sha256":"e44d6b758a96539e3e02336430d3f85263d43c470c5bad93572e9b6a86c67f76"},"provenance":null,"requires-python":null,"size":172034,"upload-time":"2016-10-26T11:01:31.931360Z","url":"https://files.pythonhosted.org/packages/f1/58/64db332e3a665d0d0260fcfe39ff8609871a1203ed11afcff0e5991db724/psutil-4.4.2-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"34c5de5faa22e5cdae426206704ec3a7d177ec1d34a20e3ad2d072136ac17b84"},"data-dist-info-metadata":{"sha256":"34c5de5faa22e5cdae426206704ec3a7d177ec1d34a20e3ad2d072136ac17b84"},"filename":"psutil-4.4.2-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"c2b0d8d1d8b5669b9884d0dd49ccb4094d163858d672d3d13a3fa817bc8a3197"},"provenance":null,"requires-python":null,"size":174401,"upload-time":"2016-10-26T11:01:35.879864Z","url":"https://files.pythonhosted.org/packages/39/31/4b5a2ab9d20d14fa6c821b7fbbf69ae93708e1a4ba335d4e2ead549b2f59/psutil-4.4.2-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"34c5de5faa22e5cdae426206704ec3a7d177ec1d34a20e3ad2d072136ac17b84"},"data-dist-info-metadata":{"sha256":"34c5de5faa22e5cdae426206704ec3a7d177ec1d34a20e3ad2d072136ac17b84"},"filename":"psutil-4.4.2-cp34-cp34m-win32.whl","hashes":{"sha256":"10fbb631142a3200623f4ab49f8bf82c32b79b8fe179f6056d01da3dfc589da1"},"provenance":null,"requires-python":null,"size":172044,"upload-time":"2016-10-26T11:01:38.950793Z","url":"https://files.pythonhosted.org/packages/67/f6/f034ad76faa7b9bd24d70a663b5dca8d71b6b7ddb4a3ceecd6a0a63afe92/psutil-4.4.2-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"34c5de5faa22e5cdae426206704ec3a7d177ec1d34a20e3ad2d072136ac17b84"},"data-dist-info-metadata":{"sha256":"34c5de5faa22e5cdae426206704ec3a7d177ec1d34a20e3ad2d072136ac17b84"},"filename":"psutil-4.4.2-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"e423dd9cb12256c742d1d56ec38bc7d2a7fa09287c82c41e475e68b9f932c2af"},"provenance":null,"requires-python":null,"size":174393,"upload-time":"2016-10-26T11:01:41.933150Z","url":"https://files.pythonhosted.org/packages/f8/f2/df2d8de978236067d0fce847e3734c6e78bfd57df9db5a2786cc544ca85f/psutil-4.4.2-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"34c5de5faa22e5cdae426206704ec3a7d177ec1d34a20e3ad2d072136ac17b84"},"data-dist-info-metadata":{"sha256":"34c5de5faa22e5cdae426206704ec3a7d177ec1d34a20e3ad2d072136ac17b84"},"filename":"psutil-4.4.2-cp35-cp35m-win32.whl","hashes":{"sha256":"7481f299ae0e966a10cb8dd93a327efd8f51995d9bdc8810dcc65d3b12d856ee"},"provenance":null,"requires-python":null,"size":173938,"upload-time":"2016-10-26T11:01:44.870925Z","url":"https://files.pythonhosted.org/packages/57/2d/0cdc3fdb9853ec1d815e0948104f1c51894f47c8ee2152c019ba754ce678/psutil-4.4.2-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"34c5de5faa22e5cdae426206704ec3a7d177ec1d34a20e3ad2d072136ac17b84"},"data-dist-info-metadata":{"sha256":"34c5de5faa22e5cdae426206704ec3a7d177ec1d34a20e3ad2d072136ac17b84"},"filename":"psutil-4.4.2-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"d96d31d83781c7f3d0df8ccb1cc50650ca84d4722c5070b71ce8f1cc112e02e0"},"provenance":null,"requires-python":null,"size":177465,"upload-time":"2016-10-26T11:01:47.925426Z","url":"https://files.pythonhosted.org/packages/cd/50/3c238d67e025701fa18cc37ed445327efcf14ea4d06e357eef92d1a250bf/psutil-4.4.2-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.2.tar.gz","hashes":{"sha256":"1c37e6428f7fe3aeea607f9249986d9bb933bb98133c7919837fd9aac4996b07"},"provenance":null,"requires-python":null,"size":1832052,"upload-time":"2016-10-26T11:01:53.029413Z","url":"https://files.pythonhosted.org/packages/6c/49/0f784a247868e167389f6ac76b8699b2f3d6f4e8e85685dfec43e58d1ed1/psutil-4.4.2.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.2.win-amd64-py2.7.exe","hashes":{"sha256":"11a20c0328206dce68f8da771461aeaef9c44811e639216fd935837e758632dc"},"provenance":null,"requires-python":null,"size":408884,"upload-time":"2016-10-26T11:01:57.830427Z","url":"https://files.pythonhosted.org/packages/58/bc/ac2a052035e945153f0bdc23d7c169e7e64e0729af6ff96f85d960962def/psutil-4.4.2.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.2.win-amd64-py3.3.exe","hashes":{"sha256":"642194ebefa573de62406883eb33868917bab2cc2e21b68d551248e194dd0b0a"},"provenance":null,"requires-python":null,"size":407275,"upload-time":"2016-10-26T11:02:00.968973Z","url":"https://files.pythonhosted.org/packages/38/b3/c8244a4858de5f9525f176d72cb06584433446a5b9d7272d5eec7a435f81/psutil-4.4.2.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.2.win-amd64-py3.4.exe","hashes":{"sha256":"c02b9fb5f1f3c857938b26a73b1ca92007e8b0b2fd64693b29300fae0ceaf679"},"provenance":null,"requires-python":null,"size":407267,"upload-time":"2016-10-26T11:02:04.291434Z","url":"https://files.pythonhosted.org/packages/cd/4d/8f447ecba1a3d5cae9f3bdd8395b45e2e6d07834a05fa3ce58c766ffcea6/psutil-4.4.2.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.2.win-amd64-py3.5.exe","hashes":{"sha256":"6c40dc16b579f645e1804341322364203d0b21045747e62e360fae843d945e20"},"provenance":null,"requires-python":null,"size":777940,"upload-time":"2016-10-26T11:02:10.124058Z","url":"https://files.pythonhosted.org/packages/73/6e/1c58cbdedcf4e2226f8ca182c947e29f29dba6965fdae0aa8e6a361365ce/psutil-4.4.2.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.2.win32-py2.7.exe","hashes":{"sha256":"c353ecc62e67bf7c7051c087670d49eae9472f1b30bb1623d667b0cd137e8934"},"provenance":null,"requires-python":null,"size":378885,"upload-time":"2016-10-26T11:02:16.586357Z","url":"https://files.pythonhosted.org/packages/37/27/9eba3e29f8291be103a8e5e44c06eefe5357217e9d8d294ff154dc3bff90/psutil-4.4.2.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.2.win32-py3.3.exe","hashes":{"sha256":"7106cb3722235ccb6fe4b18c51f60a548d4b111ec2d209abdcd3998661f4593a"},"provenance":null,"requires-python":null,"size":373678,"upload-time":"2016-10-26T11:02:20.035049Z","url":"https://files.pythonhosted.org/packages/62/25/189e73856fa3172d5e785404eb9cf4561b01094363cc39ec267399e401c4/psutil-4.4.2.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.2.win32-py3.4.exe","hashes":{"sha256":"de1f53fe955dfba562f7791f72517935010a2e88f9caad36917e8c5c03de9051"},"provenance":null,"requires-python":null,"size":373689,"upload-time":"2016-10-26T11:02:23.073976Z","url":"https://files.pythonhosted.org/packages/10/4a/e50ca35162538645c4d98816757763e4aaaac1ccaf055b39265acb8739e1/psutil-4.4.2.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-4.4.2.win32-py3.5.exe","hashes":{"sha256":"2eb123ca86057ed4f31cfc9880e098ee7a8e19c7ec02b068c45e7559ae7539a6"},"provenance":null,"requires-python":null,"size":644886,"upload-time":"2016-10-26T11:02:26.650470Z","url":"https://files.pythonhosted.org/packages/c5/97/018f0580bc3d3d85f1e084e683e904bc33cd58ca96e0ecb5ae6f346d6452/psutil-4.4.2.win32-py3.5.exe","yanked":false},{"core-metadata":{"sha256":"3db8fbdc9c0693e219a15e49b871dc421e85735214638b6e31d68aee5b78c550"},"data-dist-info-metadata":{"sha256":"3db8fbdc9c0693e219a15e49b871dc421e85735214638b6e31d68aee5b78c550"},"filename":"psutil-5.0.0-cp27-none-win32.whl","hashes":{"sha256":"cc2560b527cd88a9bc062ee4bd055c40b9fc107e37db01997422c75a3f94efe9"},"provenance":null,"requires-python":null,"size":175039,"upload-time":"2016-11-06T18:28:50.896397Z","url":"https://files.pythonhosted.org/packages/bb/82/efee0c83eab6ad3dd4a70b2b91013a50776ff6253f9b71d825914e693dfa/psutil-5.0.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"3db8fbdc9c0693e219a15e49b871dc421e85735214638b6e31d68aee5b78c550"},"data-dist-info-metadata":{"sha256":"3db8fbdc9c0693e219a15e49b871dc421e85735214638b6e31d68aee5b78c550"},"filename":"psutil-5.0.0-cp27-none-win_amd64.whl","hashes":{"sha256":"8a6cbc7165a476d08a89ee3078a74d111729cf515fd831db9f635012e56f9759"},"provenance":null,"requires-python":null,"size":177395,"upload-time":"2016-11-06T18:28:54.500906Z","url":"https://files.pythonhosted.org/packages/16/bf/09cb9f5286e5037eba1d5fe347c0f622b325e068757f1072c56f9b130cc3/psutil-5.0.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"48627687813fd477d371c4588c61dc9da3e42f22546da21c80f273e30c587f1b"},"data-dist-info-metadata":{"sha256":"48627687813fd477d371c4588c61dc9da3e42f22546da21c80f273e30c587f1b"},"filename":"psutil-5.0.0-cp33-cp33m-win32.whl","hashes":{"sha256":"9b0f13e325f007a0fd04e9d44cfdb5187c0b3e144f89533324dd9f74c25bd9ec"},"provenance":null,"requires-python":null,"size":174954,"upload-time":"2016-11-06T18:28:57.539707Z","url":"https://files.pythonhosted.org/packages/a2/5d/82b85eb4c24c82064c5c216f52df92f7861ff0a02d3cbcfebfacadc2b28f/psutil-5.0.0-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"48627687813fd477d371c4588c61dc9da3e42f22546da21c80f273e30c587f1b"},"data-dist-info-metadata":{"sha256":"48627687813fd477d371c4588c61dc9da3e42f22546da21c80f273e30c587f1b"},"filename":"psutil-5.0.0-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"ade8924028b2c23cc9ffe4a0737de38c668d50be5ce19495790154f530ce5389"},"provenance":null,"requires-python":null,"size":177229,"upload-time":"2016-11-06T18:29:00.558809Z","url":"https://files.pythonhosted.org/packages/14/5b/00b74d0ada625090dfaba6244860a9ec4df2e79a6c8440dd490e46c65543/psutil-5.0.0-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"48627687813fd477d371c4588c61dc9da3e42f22546da21c80f273e30c587f1b"},"data-dist-info-metadata":{"sha256":"48627687813fd477d371c4588c61dc9da3e42f22546da21c80f273e30c587f1b"},"filename":"psutil-5.0.0-cp34-cp34m-win32.whl","hashes":{"sha256":"af01b73fd66f138e06f804508fd33118823fd2abb89f105ae2b99efa4c8fd1a3"},"provenance":null,"requires-python":null,"size":174957,"upload-time":"2016-11-06T18:29:04.527838Z","url":"https://files.pythonhosted.org/packages/fd/e6/5a08cd23cefadd22b8e74d0bd1c525897c9db4dc1d2f44716dee848b8fe3/psutil-5.0.0-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"48627687813fd477d371c4588c61dc9da3e42f22546da21c80f273e30c587f1b"},"data-dist-info-metadata":{"sha256":"48627687813fd477d371c4588c61dc9da3e42f22546da21c80f273e30c587f1b"},"filename":"psutil-5.0.0-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"846925435e69cc7b802cd7d1a4bd640e180d0db15277c24e196d3a5799bf6760"},"provenance":null,"requires-python":null,"size":177189,"upload-time":"2016-11-06T18:29:08.003063Z","url":"https://files.pythonhosted.org/packages/b4/11/f68bad0d5fc08800006d2355c941e186315987456158a9b06f40e483d4e7/psutil-5.0.0-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"48627687813fd477d371c4588c61dc9da3e42f22546da21c80f273e30c587f1b"},"data-dist-info-metadata":{"sha256":"48627687813fd477d371c4588c61dc9da3e42f22546da21c80f273e30c587f1b"},"filename":"psutil-5.0.0-cp35-cp35m-win32.whl","hashes":{"sha256":"d7885ff254425c64bcc2dbff256ec1367515c15218bfda0fd3d799092437d908"},"provenance":null,"requires-python":null,"size":176812,"upload-time":"2016-11-06T18:29:11.700563Z","url":"https://files.pythonhosted.org/packages/c4/16/84213b90c78e437eff09285138947d12105ea982cb6f8fda4ab2855014e6/psutil-5.0.0-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"48627687813fd477d371c4588c61dc9da3e42f22546da21c80f273e30c587f1b"},"data-dist-info-metadata":{"sha256":"48627687813fd477d371c4588c61dc9da3e42f22546da21c80f273e30c587f1b"},"filename":"psutil-5.0.0-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"e35cb38037973ff05bc52dac4f382d17028104d77f0bb51792de93f359046902"},"provenance":null,"requires-python":null,"size":180171,"upload-time":"2016-11-06T18:29:15.915700Z","url":"https://files.pythonhosted.org/packages/98/ef/40582d2d3e39fdcc202a33ba6aab15f6ccb36cdcd04f6756cc9afe30df30/psutil-5.0.0-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.0.win-amd64-py2.7.exe","hashes":{"sha256":"964e9db2641e3be6a80b5a3135f7a9425f87d8416342b4d967202e4854f3eeba"},"provenance":null,"requires-python":null,"size":411778,"upload-time":"2016-11-06T18:29:32.458135Z","url":"https://files.pythonhosted.org/packages/74/f0/e8964d58e12c7716775157821de2e758fece2581bc9f3b7c333a4de29b90/psutil-5.0.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.0.win-amd64-py3.3.exe","hashes":{"sha256":"d7b933193322523314b0b2d699a43e43a70f43016f73782dfd892bc7ee95ecd1"},"provenance":null,"requires-python":null,"size":410125,"upload-time":"2016-11-06T18:29:37.513499Z","url":"https://files.pythonhosted.org/packages/6e/a9/4438ec3289e3925a638cb079039b748ee4fb7e3ba18a93baed39f4fcd11e/psutil-5.0.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.0.win-amd64-py3.4.exe","hashes":{"sha256":"8709a5057d42d5e55f3940bb1e70370316defb3da03ad869342755b5a2c17c78"},"provenance":null,"requires-python":null,"size":410086,"upload-time":"2016-11-06T18:29:41.067409Z","url":"https://files.pythonhosted.org/packages/db/85/c2e27aab6db3c4d404f0e81762f5a962b7213caed0a789cb5df2990ac489/psutil-5.0.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.0.win-amd64-py3.5.exe","hashes":{"sha256":"c065eaf76a5423341f511e732f1a17e75a55dc4aceee9a321a619a5892aec18f"},"provenance":null,"requires-python":null,"size":780670,"upload-time":"2016-11-06T18:29:45.597507Z","url":"https://files.pythonhosted.org/packages/27/fd/12a4ef4a9940331a5521216e839444dcab0addd3a49fb90c940bdc40e46e/psutil-5.0.0.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.0.win32-py2.7.exe","hashes":{"sha256":"8eca28511d493209f59fe99cebfb8ecc65b8d6691f8a80fa3ab50dbb4994c81b"},"provenance":null,"requires-python":null,"size":381779,"upload-time":"2016-11-06T18:29:49.728094Z","url":"https://files.pythonhosted.org/packages/bc/77/457ddeac355fe88c8d4ee8062ce66ab5dd142f5c9423d1bfd10a568c7884/psutil-5.0.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.0.win32-py3.3.exe","hashes":{"sha256":"7a8e7654789c468d2a6c32508638563741046f138405fea2a4427a9228ac86f4"},"provenance":null,"requires-python":null,"size":376623,"upload-time":"2016-11-06T18:29:53.256127Z","url":"https://files.pythonhosted.org/packages/8d/1a/2f61de7f89602dfd975f961bf057522ff53ef3bdc48610dbdcf306b542df/psutil-5.0.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.0.win32-py3.4.exe","hashes":{"sha256":"4db1f1d8655a63832c9ab85d77c969ab95b740128ca9053b5c108a1e5efe6a7c"},"provenance":null,"requires-python":null,"size":376624,"upload-time":"2016-11-06T18:29:56.669027Z","url":"https://files.pythonhosted.org/packages/c1/3f/d8921d1a8672545a390fd874b77c9a30444b64ab5f8c48e2e9d971f4e98f/psutil-5.0.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.0.win32-py3.5.exe","hashes":{"sha256":"10f05841e3bf7b060b3779367b9cd953a6c97303c08a0e2630a9e461ce124412"},"provenance":null,"requires-python":null,"size":647782,"upload-time":"2016-11-06T18:30:00.475783Z","url":"https://files.pythonhosted.org/packages/ec/53/dc7d9e33e77efd26f7c4a9ba8d1e23ba9ce432077e16a55f1f755970e7f7/psutil-5.0.0.win32-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.0.zip","hashes":{"sha256":"5411e22c63168220f4b8cc42fd05ea96f5b5e65e08b93b675ca50653aea482f8"},"provenance":null,"requires-python":null,"size":374074,"upload-time":"2016-11-06T18:41:47.892123Z","url":"https://files.pythonhosted.org/packages/93/7f/347309562d30c688299727e65f4d76ef34180c406dfb6f2c7b6c8d746e13/psutil-5.0.0.zip","yanked":false},{"core-metadata":{"sha256":"3c16b5cea3c7592513b1491a7696e113d3ba33a4a94d74841ee2340fb4081106"},"data-dist-info-metadata":{"sha256":"3c16b5cea3c7592513b1491a7696e113d3ba33a4a94d74841ee2340fb4081106"},"filename":"psutil-5.0.1-cp27-none-win32.whl","hashes":{"sha256":"1f2379809f2182652fc740faefa511f78b5975e6471e5fa419882dd9e082f245"},"provenance":null,"requires-python":null,"size":176737,"upload-time":"2016-12-21T01:35:18.007609Z","url":"https://files.pythonhosted.org/packages/cd/2d/ecb32ce765c52780768e4078d4d2916bdaa790b313c454f52aebe27d559d/psutil-5.0.1-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"3c16b5cea3c7592513b1491a7696e113d3ba33a4a94d74841ee2340fb4081106"},"data-dist-info-metadata":{"sha256":"3c16b5cea3c7592513b1491a7696e113d3ba33a4a94d74841ee2340fb4081106"},"filename":"psutil-5.0.1-cp27-none-win_amd64.whl","hashes":{"sha256":"86d67da7bfed474b5b0d545b29211f16d622e87c4089de5ea41a3fbcfc4872c7"},"provenance":null,"requires-python":null,"size":179100,"upload-time":"2016-12-21T01:35:20.364405Z","url":"https://files.pythonhosted.org/packages/6b/7e/68835630dcc765452b76b61936f75c1ef71fceae8ebaa7ecb27e42bf3f28/psutil-5.0.1-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"b03798768de1ae02a3387726d3bb1c881c70cdee8bea158f236b08758379017e"},"data-dist-info-metadata":{"sha256":"b03798768de1ae02a3387726d3bb1c881c70cdee8bea158f236b08758379017e"},"filename":"psutil-5.0.1-cp33-cp33m-win32.whl","hashes":{"sha256":"6a95283fd048810811cf971bec5cec3998e1e62e66237ef7a41a42dd0da29f8c"},"provenance":null,"requires-python":null,"size":176612,"upload-time":"2016-12-21T01:35:22.141278Z","url":"https://files.pythonhosted.org/packages/43/46/3fad73a8c581f63f08fb1f93ef5651e939caa5601a50baca7f2af3c54e2b/psutil-5.0.1-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"b03798768de1ae02a3387726d3bb1c881c70cdee8bea158f236b08758379017e"},"data-dist-info-metadata":{"sha256":"b03798768de1ae02a3387726d3bb1c881c70cdee8bea158f236b08758379017e"},"filename":"psutil-5.0.1-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"8f492b531c4321c7c43ef82b60421f4bcf5ded4ba4e13f534c064ad6c2d910ed"},"provenance":null,"requires-python":null,"size":178930,"upload-time":"2016-12-21T01:35:23.896324Z","url":"https://files.pythonhosted.org/packages/b0/d0/6d9e39115aaab0d8d52422b909e365938a6dcebe5fa04fac3a6b70398aee/psutil-5.0.1-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"b03798768de1ae02a3387726d3bb1c881c70cdee8bea158f236b08758379017e"},"data-dist-info-metadata":{"sha256":"b03798768de1ae02a3387726d3bb1c881c70cdee8bea158f236b08758379017e"},"filename":"psutil-5.0.1-cp34-cp34m-win32.whl","hashes":{"sha256":"5cc1b91d4848453b74ad8e63275a19e784ef3acd943c3627134a607b602bc31d"},"provenance":null,"requires-python":null,"size":176623,"upload-time":"2016-12-21T01:35:26.391755Z","url":"https://files.pythonhosted.org/packages/31/21/d836f37c8fde1760e889b0ace67e151c6e56f2ce819b227715a5f291452f/psutil-5.0.1-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"b03798768de1ae02a3387726d3bb1c881c70cdee8bea158f236b08758379017e"},"data-dist-info-metadata":{"sha256":"b03798768de1ae02a3387726d3bb1c881c70cdee8bea158f236b08758379017e"},"filename":"psutil-5.0.1-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"55874a1814faceaa090b7fa7addbf350603fd7042562adaae13eb6e46d3ec907"},"provenance":null,"requires-python":null,"size":178917,"upload-time":"2016-12-21T01:35:28.280103Z","url":"https://files.pythonhosted.org/packages/a1/9c/cb3f68be56ab366dbf4c901002d281e30c50cf951377ca8155c0ad304394/psutil-5.0.1-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"b03798768de1ae02a3387726d3bb1c881c70cdee8bea158f236b08758379017e"},"data-dist-info-metadata":{"sha256":"b03798768de1ae02a3387726d3bb1c881c70cdee8bea158f236b08758379017e"},"filename":"psutil-5.0.1-cp35-cp35m-win32.whl","hashes":{"sha256":"7f2d7b97faa524f75736dfde418680eff332f4be66d6217b67d09c630c90d02e"},"provenance":null,"requires-python":null,"size":178514,"upload-time":"2016-12-21T01:35:30.152277Z","url":"https://files.pythonhosted.org/packages/f7/f1/dd823eab436db1eac1ca2c8364918786584db996dc4d2bef5a3b0ebd7619/psutil-5.0.1-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"b03798768de1ae02a3387726d3bb1c881c70cdee8bea158f236b08758379017e"},"data-dist-info-metadata":{"sha256":"b03798768de1ae02a3387726d3bb1c881c70cdee8bea158f236b08758379017e"},"filename":"psutil-5.0.1-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"bf4072d4d188802505b9229ec00e141083c127bb19a6c5636c62b0daabda4bd5"},"provenance":null,"requires-python":null,"size":181835,"upload-time":"2016-12-21T01:35:32.091445Z","url":"https://files.pythonhosted.org/packages/2a/3a/107a964dc66cb4f22af9e919dde846b73c0b4ff735b10ede680895ed296c/psutil-5.0.1-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"b03798768de1ae02a3387726d3bb1c881c70cdee8bea158f236b08758379017e"},"data-dist-info-metadata":{"sha256":"b03798768de1ae02a3387726d3bb1c881c70cdee8bea158f236b08758379017e"},"filename":"psutil-5.0.1-cp36-cp36m-win32.whl","hashes":{"sha256":"786bbbeb3ea98d82ff5cedc86b640bad97bff435c819f26bddaa388da58d47da"},"provenance":null,"requires-python":null,"size":178650,"upload-time":"2017-01-15T18:27:34.549244Z","url":"https://files.pythonhosted.org/packages/c0/a1/c5d0aa766323b150dce5e51a5360542ccb75c109723d2d95e70d8cb248b8/psutil-5.0.1-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"b03798768de1ae02a3387726d3bb1c881c70cdee8bea158f236b08758379017e"},"data-dist-info-metadata":{"sha256":"b03798768de1ae02a3387726d3bb1c881c70cdee8bea158f236b08758379017e"},"filename":"psutil-5.0.1-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"544c0803760995fe42a2b7050cfd6ed32379b09ce6cd7a1eaf89c221d8669cc3"},"provenance":null,"requires-python":null,"size":181970,"upload-time":"2017-01-15T18:27:39.376642Z","url":"https://files.pythonhosted.org/packages/c0/e4/bf1059c5dd55abf65fe2ac92a0b24a92d09644a9bbc17f1ad2b497ae9d7d/psutil-5.0.1-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.1.tar.gz","hashes":{"sha256":"9d8b7f8353a2b2eb6eb7271d42ec99d0d264a9338a37be46424d56b4e473b39e"},"provenance":null,"requires-python":null,"size":326693,"upload-time":"2016-12-21T01:35:34.145966Z","url":"https://files.pythonhosted.org/packages/d9/c8/8c7a2ab8ec108ba9ab9a4762c5a0d67c283d41b13b5ce46be81fdcae3656/psutil-5.0.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.1.win-amd64-py2.7.exe","hashes":{"sha256":"0c54e3b7cdc0dbb8a19b58c3eb3845a5f9f48d3be2b06ed9aa1e553db8f9db74"},"provenance":null,"requires-python":null,"size":414521,"upload-time":"2016-12-21T01:35:36.513939Z","url":"https://files.pythonhosted.org/packages/3d/90/61f2860cd2c39a4381e415897b7ff94aab21c9aca38690d0c99c1c83b85f/psutil-5.0.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.1.win-amd64-py3.3.exe","hashes":{"sha256":"0fccb19631d555998fc8c98840c83678244f873486d20d5f24ebb0ac8e19d2f1"},"provenance":null,"requires-python":null,"size":412871,"upload-time":"2016-12-21T01:35:38.959549Z","url":"https://files.pythonhosted.org/packages/05/f7/6ba6efe79620bcaa9af41e0a8501e477815edccc0cad4bd379a8e8bea89c/psutil-5.0.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.1.win-amd64-py3.4.exe","hashes":{"sha256":"77fde4936f26080aa14b89d292b3ebefabb80be69ef407352cbad6d2ff6882d4"},"provenance":null,"requires-python":null,"size":412858,"upload-time":"2016-12-21T01:35:42.062765Z","url":"https://files.pythonhosted.org/packages/c7/b3/3b83e69b5d96ed052b45f1fe88882b3976a3a066c0497597d3ce1407f17f/psutil-5.0.1.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.1.win-amd64-py3.5.exe","hashes":{"sha256":"a9125b7bc12127174cf7974444ca2b39a3722f59ead1d985053e7358a3d29acd"},"provenance":null,"requires-python":null,"size":783380,"upload-time":"2016-12-21T01:35:44.660601Z","url":"https://files.pythonhosted.org/packages/4e/cf/d1004fa0617f6d6e39adf48d242f76d5d43ecd497f46add2abb513b6ffcd/psutil-5.0.1.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.1.win-amd64-py3.6.exe","hashes":{"sha256":"9a3f1413b24b6751e97e51284761e1778ec0cd0a456595edad6c2f7c115b3368"},"provenance":null,"requires-python":null,"size":783511,"upload-time":"2017-01-15T18:25:42.785800Z","url":"https://files.pythonhosted.org/packages/a0/94/1f29e03230504dafeb128a4c6e381606b038ce5d4dc9b8731632eb6b0928/psutil-5.0.1.win-amd64-py3.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.1.win32-py2.7.exe","hashes":{"sha256":"671c7b2d3fa8deffb879e9cb6cc00d83d1d990bc81f0c487576b70e811f102bf"},"provenance":null,"requires-python":null,"size":384515,"upload-time":"2016-12-21T01:35:47.208014Z","url":"https://files.pythonhosted.org/packages/2d/7c/85aa8efb36c4fcac4a845b9cfc0744c502620394cdb81db574dd20f7dda0/psutil-5.0.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.1.win32-py3.3.exe","hashes":{"sha256":"603bd19426b59762ad55ba5a0b8237282868addf2d0930e21b4dca79fc188787"},"provenance":null,"requires-python":null,"size":379325,"upload-time":"2016-12-21T01:35:49.642792Z","url":"https://files.pythonhosted.org/packages/1c/06/201f913389b920c24153fbfdc07c7a451653b00688d6c2ba92428b9a2c23/psutil-5.0.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.1.win32-py3.4.exe","hashes":{"sha256":"08e28b4dba54dd6b0d84718c988239c305e58c7f93160496661955de4f6cfe13"},"provenance":null,"requires-python":null,"size":379338,"upload-time":"2016-12-21T01:35:52.592934Z","url":"https://files.pythonhosted.org/packages/58/27/a42b1d12c201880822fd92588321e489c869f4b72eb906f8f79745bee6cb/psutil-5.0.1.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.1.win32-py3.5.exe","hashes":{"sha256":"1e71dd9aa041a7fa55f3bcc78b578e84f31004e6ce9df97229f05c60529fedb1"},"provenance":null,"requires-python":null,"size":650529,"upload-time":"2016-12-21T01:35:55.051616Z","url":"https://files.pythonhosted.org/packages/fb/c6/c60717f25c8ccfddabd57109d03580d24542eedde8d23a673328876b4137/psutil-5.0.1.win32-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.0.1.win32-py3.6.exe","hashes":{"sha256":"c879310328c0c248331dffeea4adbe691fad7b7095cf9c2ac0a4d78a09cd8a17"},"provenance":null,"requires-python":null,"size":650663,"upload-time":"2017-01-15T18:25:51.768008Z","url":"https://files.pythonhosted.org/packages/00/09/04898f2ce604af89d7ca6866632c05a1fb0fbd7b5ea0c7e84a62570c2678/psutil-5.0.1.win32-py3.6.exe","yanked":false},{"core-metadata":{"sha256":"45f08e36846b408dfbda38b60b36dd8b6430d6fbd955f5945dc985a85eac6068"},"data-dist-info-metadata":{"sha256":"45f08e36846b408dfbda38b60b36dd8b6430d6fbd955f5945dc985a85eac6068"},"filename":"psutil-5.1.0-cp27-cp27m-win32.whl","hashes":{"sha256":"85524e46a1c0c7f5274640809deb96c7cdb5133a0eb805bbed0a1f825c8de77e"},"provenance":null,"requires-python":null,"size":183880,"upload-time":"2017-02-01T18:27:27.314031Z","url":"https://files.pythonhosted.org/packages/d9/bf/eae1c0aa2a5a01456e78d5df4c7cefc512854a24f43103034180ff5ea92f/psutil-5.1.0-cp27-cp27m-win32.whl","yanked":false},{"core-metadata":{"sha256":"45f08e36846b408dfbda38b60b36dd8b6430d6fbd955f5945dc985a85eac6068"},"data-dist-info-metadata":{"sha256":"45f08e36846b408dfbda38b60b36dd8b6430d6fbd955f5945dc985a85eac6068"},"filename":"psutil-5.1.0-cp27-cp27m-win_amd64.whl","hashes":{"sha256":"0a64f395295aafe7c22d6cde81076151bf80f26e180e50c00c1cde5857cad224"},"provenance":null,"requires-python":null,"size":186335,"upload-time":"2017-02-01T18:27:35.043588Z","url":"https://files.pythonhosted.org/packages/f6/5d/67120bb4cea2432d333cdf0ab406336ef62122caacb2379e73a88a9cde82/psutil-5.1.0-cp27-cp27m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"f1cc4b1bb4982cfc6e8259dbc0fd96789e31eff9b65d8e5a44401e11d9926c55"},"data-dist-info-metadata":{"sha256":"f1cc4b1bb4982cfc6e8259dbc0fd96789e31eff9b65d8e5a44401e11d9926c55"},"filename":"psutil-5.1.0-cp33-cp33m-win32.whl","hashes":{"sha256":"bd432606d09722220f8f492223e8b69343d30f21e7ed193712632e4fdcf87af2"},"provenance":null,"requires-python":null,"size":183793,"upload-time":"2017-02-01T18:27:43.056910Z","url":"https://files.pythonhosted.org/packages/26/35/ec223413c9301d8eeecf3d319d71a117dbd2847b628b079f3757d5ec693d/psutil-5.1.0-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"f1cc4b1bb4982cfc6e8259dbc0fd96789e31eff9b65d8e5a44401e11d9926c55"},"data-dist-info-metadata":{"sha256":"f1cc4b1bb4982cfc6e8259dbc0fd96789e31eff9b65d8e5a44401e11d9926c55"},"filename":"psutil-5.1.0-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"f98c37506e5abfeb2c857794ccfbce86b376cf6344210ea55f41b0ced5fd8d98"},"provenance":null,"requires-python":null,"size":186185,"upload-time":"2017-02-01T18:27:49.896261Z","url":"https://files.pythonhosted.org/packages/9f/20/bb95f30c99837f3c41a8a595752d9cc1afe595f160d0612eea6e4a905de9/psutil-5.1.0-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"f1cc4b1bb4982cfc6e8259dbc0fd96789e31eff9b65d8e5a44401e11d9926c55"},"data-dist-info-metadata":{"sha256":"f1cc4b1bb4982cfc6e8259dbc0fd96789e31eff9b65d8e5a44401e11d9926c55"},"filename":"psutil-5.1.0-cp34-cp34m-win32.whl","hashes":{"sha256":"8760214ce01f71cd6766527396607cf2b3b41e6f8a34aaa9f716090750fd9925"},"provenance":null,"requires-python":null,"size":183803,"upload-time":"2017-02-01T18:27:56.955786Z","url":"https://files.pythonhosted.org/packages/0b/f2/90a7288b36a18fb522e00116f7f5c657df9dba3a12636d2fc9bcef97f524/psutil-5.1.0-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"f1cc4b1bb4982cfc6e8259dbc0fd96789e31eff9b65d8e5a44401e11d9926c55"},"data-dist-info-metadata":{"sha256":"f1cc4b1bb4982cfc6e8259dbc0fd96789e31eff9b65d8e5a44401e11d9926c55"},"filename":"psutil-5.1.0-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"2309cf46f5db0a78f52e56e06cbb258fe76d3d54cf8ab9fa067b4c26cd722541"},"provenance":null,"requires-python":null,"size":186156,"upload-time":"2017-02-01T18:28:03.790958Z","url":"https://files.pythonhosted.org/packages/51/c6/3757b2bea37edc9889fe85e2ca9b8a2c47b3b758d35f65704f2a5ac5c13e/psutil-5.1.0-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"f1cc4b1bb4982cfc6e8259dbc0fd96789e31eff9b65d8e5a44401e11d9926c55"},"data-dist-info-metadata":{"sha256":"f1cc4b1bb4982cfc6e8259dbc0fd96789e31eff9b65d8e5a44401e11d9926c55"},"filename":"psutil-5.1.0-cp35-cp35m-win32.whl","hashes":{"sha256":"6d89d5f68433afe6755bd26ae3ea4587f395c7613d7be5bc4f0f97b1e299228a"},"provenance":null,"requires-python":null,"size":185696,"upload-time":"2017-02-01T18:28:10.369149Z","url":"https://files.pythonhosted.org/packages/30/9c/322df29c490ceb97d8cbbc663424fad94e900a1921d03cc9c61e36f17444/psutil-5.1.0-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"f1cc4b1bb4982cfc6e8259dbc0fd96789e31eff9b65d8e5a44401e11d9926c55"},"data-dist-info-metadata":{"sha256":"f1cc4b1bb4982cfc6e8259dbc0fd96789e31eff9b65d8e5a44401e11d9926c55"},"filename":"psutil-5.1.0-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"b3c15965b2dfad34ea79494f82007e71b02e1c5352b551d00647fb2be9deabe1"},"provenance":null,"requires-python":null,"size":189130,"upload-time":"2017-02-01T18:28:17.197543Z","url":"https://files.pythonhosted.org/packages/b3/3a/37d6bf7dedf263e1119b03aaee8565182bf2577ed5ecd7975e253e242bd9/psutil-5.1.0-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"f1cc4b1bb4982cfc6e8259dbc0fd96789e31eff9b65d8e5a44401e11d9926c55"},"data-dist-info-metadata":{"sha256":"f1cc4b1bb4982cfc6e8259dbc0fd96789e31eff9b65d8e5a44401e11d9926c55"},"filename":"psutil-5.1.0-cp36-cp36m-win32.whl","hashes":{"sha256":"d32551c242c041b0506d13cfd271db56ba57323283b1f952b858505824f3e82a"},"provenance":null,"requires-python":null,"size":185695,"upload-time":"2017-02-01T18:28:25.059542Z","url":"https://files.pythonhosted.org/packages/42/0b/5ee6feda28c2895b0e8cae4d828d356e13e29bf88bc614065e490bfc8b51/psutil-5.1.0-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"f1cc4b1bb4982cfc6e8259dbc0fd96789e31eff9b65d8e5a44401e11d9926c55"},"data-dist-info-metadata":{"sha256":"f1cc4b1bb4982cfc6e8259dbc0fd96789e31eff9b65d8e5a44401e11d9926c55"},"filename":"psutil-5.1.0-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"3b821bb59911afdba5c0c28b1e6e7511cfb0869dd8827c3ab6916ead508c9155"},"provenance":null,"requires-python":null,"size":189127,"upload-time":"2017-02-01T18:28:32.228516Z","url":"https://files.pythonhosted.org/packages/af/6f/ef0799221597f367171c0bb8b3f1b22865e6b7507ee26e1a6a16f408a058/psutil-5.1.0-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.0.tar.gz","hashes":{"sha256":"7570e1d82345fab3a0adce24baf993adbca4c87a1be2fa6ee79babdaafa817fb"},"provenance":null,"requires-python":null,"size":339603,"upload-time":"2017-02-01T18:29:04.029583Z","url":"https://files.pythonhosted.org/packages/ba/5f/87b151dd53f8790408adf5096fc81c3061313c36a089d9f7ec9e916da0c1/psutil-5.1.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.0.win-amd64-py2.7.exe","hashes":{"sha256":"07aab3a12c00315a144b2a61d384364e64077c08c0a79966f18b744d867f9727"},"provenance":null,"requires-python":null,"size":422485,"upload-time":"2017-02-01T18:24:47.444024Z","url":"https://files.pythonhosted.org/packages/8c/1c/8bf1c2f6f2b4f012449bd9781d55397e85e30487c2879ed35569c7f616de/psutil-5.1.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.0.win-amd64-py3.3.exe","hashes":{"sha256":"b78de627c157b4889e055fd00a635ad0fbe107ae4b3059f419738edbfb15b3f4"},"provenance":null,"requires-python":null,"size":420852,"upload-time":"2017-02-01T18:25:00.582537Z","url":"https://files.pythonhosted.org/packages/de/5a/3041f7d07d9699ef148291721ee167608761e223bc26bbe33483cbe90517/psutil-5.1.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.0.win-amd64-py3.4.exe","hashes":{"sha256":"48fc3945219f6577b61a76f81fbbaa3ce6bdccf0e87650597c88aac38e63904b"},"provenance":null,"requires-python":null,"size":420825,"upload-time":"2017-02-01T18:25:14.404878Z","url":"https://files.pythonhosted.org/packages/d0/83/4e03c76cd202d4a4ead9b468cf2afa611d689b1ae17e4f152b67c05d2a27/psutil-5.1.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.0.win-amd64-py3.5.exe","hashes":{"sha256":"98180376d2f0a3b0d24dbd8ef64da4aac46f98ccc84d2e3ebc3a486b97a75915"},"provenance":null,"requires-python":null,"size":791399,"upload-time":"2017-02-01T18:25:37.374511Z","url":"https://files.pythonhosted.org/packages/ff/86/328005e07308a87f4d86108a51f940fba1fb1c4f60d4676f80fc66a9dd7a/psutil-5.1.0.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.0.win-amd64-py3.6.exe","hashes":{"sha256":"a574d3fb56514ae98f2836a218b0dcf6daeb31b9945cec9b7daa9b619ad3005a"},"provenance":null,"requires-python":null,"size":791396,"upload-time":"2017-02-01T18:26:00.310591Z","url":"https://files.pythonhosted.org/packages/83/86/88c20e9ebb1565598401b23ece21367bd059d569cb0ce797509ebdc429be/psutil-5.1.0.win-amd64-py3.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.0.win32-py2.7.exe","hashes":{"sha256":"b014a8df14f00054762a2ef6f9a449ccde18d9c4de48dc8cb1809343f328f83d"},"provenance":null,"requires-python":null,"size":392388,"upload-time":"2017-02-01T18:26:13.878537Z","url":"https://files.pythonhosted.org/packages/a5/e8/1032177e5af2300b39aa50623e0df4d9a91a9b316803aab8f597e6c855e2/psutil-5.1.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.0.win32-py3.3.exe","hashes":{"sha256":"01e68d047f5023d4cb55b3a2653e296d86f53abc7e63e74c8abeefa627d9307f"},"provenance":null,"requires-python":null,"size":387232,"upload-time":"2017-02-01T18:26:25.910876Z","url":"https://files.pythonhosted.org/packages/cc/b8/24bf07e83d063f552132e7808d7029663466ebed945c532ba05068c65ae3/psutil-5.1.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.0.win32-py3.4.exe","hashes":{"sha256":"bfd279124512ee1b744eb7efa269be2096e0ce7075204daccd6f4b73c359316c"},"provenance":null,"requires-python":null,"size":387243,"upload-time":"2017-02-01T18:26:38.999172Z","url":"https://files.pythonhosted.org/packages/2a/ad/40d39c8911ca0783df9684649038241913fa3119c74030b9d1a02eb28da3/psutil-5.1.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.0.win32-py3.5.exe","hashes":{"sha256":"518356f5384a8996eb4056b6fd01a28bb502476e5d0ab9bd112d8e1dc9ce041a"},"provenance":null,"requires-python":null,"size":658437,"upload-time":"2017-02-01T18:26:58.991042Z","url":"https://files.pythonhosted.org/packages/db/b1/023fa6a995356c4a4ae68ba53e1269d693ad66783ee2c0c4ac1d6e556226/psutil-5.1.0.win32-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.0.win32-py3.6.exe","hashes":{"sha256":"23670901cfa4308cc6c442e08efcd93c6a2adb3bfdbbf51d4c120fc2a1595899"},"provenance":null,"requires-python":null,"size":658436,"upload-time":"2017-02-01T18:27:18.934060Z","url":"https://files.pythonhosted.org/packages/f8/ee/be644cdded3a5761f22cd0255534f04ad89a6fabdfb85f58100d25b6bf1a/psutil-5.1.0.win32-py3.6.exe","yanked":false},{"core-metadata":{"sha256":"b880ef4983d10a0a227170484a19bf585b91f8d2cf0238488b8a0e39337acdad"},"data-dist-info-metadata":{"sha256":"b880ef4983d10a0a227170484a19bf585b91f8d2cf0238488b8a0e39337acdad"},"filename":"psutil-5.1.1-cp27-none-win32.whl","hashes":{"sha256":"b9cc631c31794f8150a034a15448ecbde6c65ab078437eb01e0a71103bced297"},"provenance":null,"requires-python":null,"size":184444,"upload-time":"2017-02-03T12:00:02.925645Z","url":"https://files.pythonhosted.org/packages/e3/6f/3546783d70949aff150555cafb1cdd51cfddbb2fea042ed3d5b930232233/psutil-5.1.1-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"b880ef4983d10a0a227170484a19bf585b91f8d2cf0238488b8a0e39337acdad"},"data-dist-info-metadata":{"sha256":"b880ef4983d10a0a227170484a19bf585b91f8d2cf0238488b8a0e39337acdad"},"filename":"psutil-5.1.1-cp27-none-win_amd64.whl","hashes":{"sha256":"d723be81a8db76ad6d78b04effd33e73813567eb1b1b7669c6926da033cb9774"},"provenance":null,"requires-python":null,"size":186898,"upload-time":"2017-02-03T12:00:10.481992Z","url":"https://files.pythonhosted.org/packages/48/98/9f3277e1d4a6d6fbc501146b9f5e8547c793df038dfa7037816edbf90325/psutil-5.1.1-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"9de4440590e9d06325c73aa9a0f85d5e2beb77f570187d8aa12add76025de520"},"data-dist-info-metadata":{"sha256":"9de4440590e9d06325c73aa9a0f85d5e2beb77f570187d8aa12add76025de520"},"filename":"psutil-5.1.1-cp33-cp33m-win32.whl","hashes":{"sha256":"5fc6d4fe04f014ea68392d1e7ab7103637b9dcbfd7bf3bc6d9d482177bf82777"},"provenance":null,"requires-python":null,"size":184356,"upload-time":"2017-02-03T12:00:18.015691Z","url":"https://files.pythonhosted.org/packages/98/fa/c43ce8992690d3b674259f21c897dab254b3fbbb2758254787e4b0f5aee8/psutil-5.1.1-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"9de4440590e9d06325c73aa9a0f85d5e2beb77f570187d8aa12add76025de520"},"data-dist-info-metadata":{"sha256":"9de4440590e9d06325c73aa9a0f85d5e2beb77f570187d8aa12add76025de520"},"filename":"psutil-5.1.1-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"7d0c6f77ebeb248ee62383340a8bd5a9b067e64618c9056d701eefdccf27f9f4"},"provenance":null,"requires-python":null,"size":186735,"upload-time":"2017-02-03T12:00:25.248665Z","url":"https://files.pythonhosted.org/packages/e1/4f/7d16da2b82d615a7fc57f159e5e7c8776c7b794ca6330a9318ca93eb735b/psutil-5.1.1-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"9de4440590e9d06325c73aa9a0f85d5e2beb77f570187d8aa12add76025de520"},"data-dist-info-metadata":{"sha256":"9de4440590e9d06325c73aa9a0f85d5e2beb77f570187d8aa12add76025de520"},"filename":"psutil-5.1.1-cp34-cp34m-win32.whl","hashes":{"sha256":"d31859dae480bc1a0be48f239bcf3caa26447fae177549a30c4b1a2a2776f299"},"provenance":null,"requires-python":null,"size":184362,"upload-time":"2017-02-03T12:00:31.866390Z","url":"https://files.pythonhosted.org/packages/bc/a8/3f3e69227217d1e7355e6de1980f357f6151ce1543bd520652b47823f93e/psutil-5.1.1-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"9de4440590e9d06325c73aa9a0f85d5e2beb77f570187d8aa12add76025de520"},"data-dist-info-metadata":{"sha256":"9de4440590e9d06325c73aa9a0f85d5e2beb77f570187d8aa12add76025de520"},"filename":"psutil-5.1.1-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"857a3620b12a33ed4169aee959e1640e7323f18dd7726035cc057e1d39639df2"},"provenance":null,"requires-python":null,"size":186740,"upload-time":"2017-02-03T12:00:39.950524Z","url":"https://files.pythonhosted.org/packages/d3/2c/974ced45441b7adf476f1b32ac3f7840f0d7ae295dd4512de21997e2049b/psutil-5.1.1-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"9de4440590e9d06325c73aa9a0f85d5e2beb77f570187d8aa12add76025de520"},"data-dist-info-metadata":{"sha256":"9de4440590e9d06325c73aa9a0f85d5e2beb77f570187d8aa12add76025de520"},"filename":"psutil-5.1.1-cp35-cp35m-win32.whl","hashes":{"sha256":"4ed11e8caff64c452a2eee8c1bea614b717a8e66e97fc88ce272d98a6499cb9a"},"provenance":null,"requires-python":null,"size":186257,"upload-time":"2017-02-03T12:00:46.799733Z","url":"https://files.pythonhosted.org/packages/2f/17/cb34ac2f50ff6499e6b64837a2d34b92696ff1dc14e57933363d94b7301c/psutil-5.1.1-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"9de4440590e9d06325c73aa9a0f85d5e2beb77f570187d8aa12add76025de520"},"data-dist-info-metadata":{"sha256":"9de4440590e9d06325c73aa9a0f85d5e2beb77f570187d8aa12add76025de520"},"filename":"psutil-5.1.1-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"3115d7acbf3cc81345fd7252946a59c3730f7baba546361a535f0a8f00d862c9"},"provenance":null,"requires-python":null,"size":189715,"upload-time":"2017-02-03T12:00:55.165186Z","url":"https://files.pythonhosted.org/packages/09/da/f9f435d53711859d41ee8a0fb87177e8b30b5dbca5bd06dd05b5c4b46ef3/psutil-5.1.1-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"9de4440590e9d06325c73aa9a0f85d5e2beb77f570187d8aa12add76025de520"},"data-dist-info-metadata":{"sha256":"9de4440590e9d06325c73aa9a0f85d5e2beb77f570187d8aa12add76025de520"},"filename":"psutil-5.1.1-cp36-cp36m-win32.whl","hashes":{"sha256":"9f6cb84d0f8e0c993c91d10ef86c637b7f1c1d4d4ca63ec0a73545cc13e9656a"},"provenance":null,"requires-python":null,"size":186256,"upload-time":"2017-02-03T12:01:01.728292Z","url":"https://files.pythonhosted.org/packages/24/19/20a4a33db5d005959458f68816fbb725791ee7843ba4f8a40dfd38e8e840/psutil-5.1.1-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"9de4440590e9d06325c73aa9a0f85d5e2beb77f570187d8aa12add76025de520"},"data-dist-info-metadata":{"sha256":"9de4440590e9d06325c73aa9a0f85d5e2beb77f570187d8aa12add76025de520"},"filename":"psutil-5.1.1-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"35311e26f7138276fa3e7af86bcb8ecbaee945c3549e690a481379075de386ba"},"provenance":null,"requires-python":null,"size":189711,"upload-time":"2017-02-03T12:01:08.476075Z","url":"https://files.pythonhosted.org/packages/6e/2b/44aea26dc3a304285e8eb91af628c9722ba4e1f44f7969cc24fded4aa744/psutil-5.1.1-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.1.tar.gz","hashes":{"sha256":"ece06401d719050a84cca97764ff5b0e41aafe6b6a2ec8a1d0bb89ca5e206d0f"},"provenance":null,"requires-python":null,"size":341006,"upload-time":"2017-02-03T12:01:19.347630Z","url":"https://files.pythonhosted.org/packages/49/ed/2a0b13f890e798b6f1f3625f0e87e5b712471d2c1c625bdcd396d36c56dc/psutil-5.1.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.1.win-amd64-py2.7.exe","hashes":{"sha256":"e75edc462005475da019f82c8a13b215e2e48db8f284d8be14308b611505d185"},"provenance":null,"requires-python":null,"size":423004,"upload-time":"2017-02-03T12:01:32.247678Z","url":"https://files.pythonhosted.org/packages/99/35/96acf109f463ce31cd29ec327d9b3b1c3eb13afbc5635e54c75e1e22c924/psutil-5.1.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.1.win-amd64-py3.3.exe","hashes":{"sha256":"e18274b77e61b3774bcf11a4aa368032fe3bbc21e4217ca903c77541fbe00eef"},"provenance":null,"requires-python":null,"size":421360,"upload-time":"2017-02-03T12:01:46.775449Z","url":"https://files.pythonhosted.org/packages/b7/56/70fc2173d34f0ee73215a0e88e2702933cf1549e3ff3d8f2d757b9e1351c/psutil-5.1.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.1.win-amd64-py3.4.exe","hashes":{"sha256":"76307f8ac94adc87509df43bba28706c27d6c5e4b7429fb658dd5905adae4dc3"},"provenance":null,"requires-python":null,"size":421366,"upload-time":"2017-02-03T12:02:01.021939Z","url":"https://files.pythonhosted.org/packages/f1/9b/961442950e039c65938665dcbe6e72bff1a2dbbd6c6c08c55e3e10db38ed/psutil-5.1.1.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.1.win-amd64-py3.5.exe","hashes":{"sha256":"8101a2b83fa3b93fbf5d5edca7169a4289f34ace2ee25d0f758cec5ae553190f"},"provenance":null,"requires-python":null,"size":791943,"upload-time":"2017-02-03T12:02:23.633303Z","url":"https://files.pythonhosted.org/packages/bb/69/95fdd45b2c1cc4735760b81d826a62433d6726ab6e8e6e2e982fb1264c20/psutil-5.1.1.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.1.win-amd64-py3.6.exe","hashes":{"sha256":"b16b48868a58322edd240cf55a0855e1b6fead3e5f02a41b73503d5c47acf330"},"provenance":null,"requires-python":null,"size":791938,"upload-time":"2017-02-03T12:02:48.386069Z","url":"https://files.pythonhosted.org/packages/15/b4/1ba8944aaae568f8f4acd7fd3ca46a95077f07b005774f212226c99a82d8/psutil-5.1.1.win-amd64-py3.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.1.win32-py2.7.exe","hashes":{"sha256":"12aeeaf5269bc75ed605ffc4979cb95c889989b40d307adf067f04f93f6d3365"},"provenance":null,"requires-python":null,"size":392908,"upload-time":"2017-02-03T12:03:01.259012Z","url":"https://files.pythonhosted.org/packages/d0/38/c274e67564aed7475fba4e0f9eaac3235797226df0b68abce7f3fc96bffa/psutil-5.1.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.1.win32-py3.3.exe","hashes":{"sha256":"2f51c669bd528982fc5397d9c84b8b389a56611c111159b1710570db33fc9750"},"provenance":null,"requires-python":null,"size":387752,"upload-time":"2017-02-03T12:03:13.897990Z","url":"https://files.pythonhosted.org/packages/d9/51/52e109c7884a6fa931882081e94dfd0e1b6adb054d4fc82943e1c9b691d2/psutil-5.1.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.1.win32-py3.4.exe","hashes":{"sha256":"f68ce5d80db909ee655ac8a322cda3abf47f41c134e7cf61b25565f116fce33f"},"provenance":null,"requires-python":null,"size":387758,"upload-time":"2017-02-03T12:03:26.769319Z","url":"https://files.pythonhosted.org/packages/5e/ee/ff6d626da64a799db055bab9f69ff9a50612a645781d14c24078bca418cb/psutil-5.1.1.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.1.win32-py3.5.exe","hashes":{"sha256":"380aa69aa529e4a4e35579d3e0320617e112473240c95b815d3b421c86e2ab6c"},"provenance":null,"requires-python":null,"size":658955,"upload-time":"2017-02-03T12:03:45.921701Z","url":"https://files.pythonhosted.org/packages/62/9b/847c9c6b1c053a15b44ccf809fce54bbd71e466fd8c1b8eecb0f8bbeb2ce/psutil-5.1.1.win32-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.1.win32-py3.6.exe","hashes":{"sha256":"bd3b881faa071a5f6f999d036cfc0d744eed223390fde05ae2a74f0f514f8bd0"},"provenance":null,"requires-python":null,"size":658953,"upload-time":"2017-02-03T12:04:04.770301Z","url":"https://files.pythonhosted.org/packages/80/ef/c23e653bbc1a523217d676fac3ac4050e9fe4d673c84680498c02ff89d23/psutil-5.1.1.win32-py3.6.exe","yanked":false},{"core-metadata":{"sha256":"40c22031c3c1f82a2758dbd44797eade645ac30207e45f42cb9da4fd1dd3808f"},"data-dist-info-metadata":{"sha256":"40c22031c3c1f82a2758dbd44797eade645ac30207e45f42cb9da4fd1dd3808f"},"filename":"psutil-5.1.2-cp27-none-win32.whl","hashes":{"sha256":"caa870244015bb547eeab7377d9fe41c44319fda6862aa56974108a224c04b1a"},"provenance":null,"requires-python":null,"size":185018,"upload-time":"2017-02-03T19:13:24.939112Z","url":"https://files.pythonhosted.org/packages/57/bc/e0a5e35a9b7f407e229db75b24c46bef012609450d7eac6b5e596a604acb/psutil-5.1.2-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"40c22031c3c1f82a2758dbd44797eade645ac30207e45f42cb9da4fd1dd3808f"},"data-dist-info-metadata":{"sha256":"40c22031c3c1f82a2758dbd44797eade645ac30207e45f42cb9da4fd1dd3808f"},"filename":"psutil-5.1.2-cp27-none-win_amd64.whl","hashes":{"sha256":"d60d978f5a9b4bc9bb22c5c4bbeb50043db6fb70e3270c08f51e81357f8ca556"},"provenance":null,"requires-python":null,"size":187473,"upload-time":"2017-02-03T19:13:32.487659Z","url":"https://files.pythonhosted.org/packages/84/bd/49681360ca11a1aeba5e48b80b6bd576695a67fe15baca144c6bfcbc9285/psutil-5.1.2-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"0f6bf0464a251593159b71879d6e05ca05190e53d209a32229b793365d8de056"},"data-dist-info-metadata":{"sha256":"0f6bf0464a251593159b71879d6e05ca05190e53d209a32229b793365d8de056"},"filename":"psutil-5.1.2-cp33-cp33m-win32.whl","hashes":{"sha256":"61b9104157df03790fae3b138afb64ad3ecb663669ee3548e609306e0b35ca61"},"provenance":null,"requires-python":null,"size":184928,"upload-time":"2017-02-03T19:13:39.838419Z","url":"https://files.pythonhosted.org/packages/59/c7/7f31631d96ba58e81ad03b4f7cee234a35c67edc44b2fe5936e41e7d1e9e/psutil-5.1.2-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"0f6bf0464a251593159b71879d6e05ca05190e53d209a32229b793365d8de056"},"data-dist-info-metadata":{"sha256":"0f6bf0464a251593159b71879d6e05ca05190e53d209a32229b793365d8de056"},"filename":"psutil-5.1.2-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"11bfc49cd680dec42c9a7200f8b0cc4d89a9d5dbad7f3b027cfac3a305343a2a"},"provenance":null,"requires-python":null,"size":187306,"upload-time":"2017-02-03T19:13:47.199966Z","url":"https://files.pythonhosted.org/packages/a3/dc/0c87a42dbff252ab68263f88e3bd8ffaa02d766e56f743cc9074bb28c0e1/psutil-5.1.2-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"0f6bf0464a251593159b71879d6e05ca05190e53d209a32229b793365d8de056"},"data-dist-info-metadata":{"sha256":"0f6bf0464a251593159b71879d6e05ca05190e53d209a32229b793365d8de056"},"filename":"psutil-5.1.2-cp34-cp34m-win32.whl","hashes":{"sha256":"75b27bac9e91e9868723d8964e73a157799c26190978029b0ef294ad7727d91d"},"provenance":null,"requires-python":null,"size":184936,"upload-time":"2017-02-03T19:13:54.487932Z","url":"https://files.pythonhosted.org/packages/e6/2c/fa2d3770ea68f7f7577bfcf3fc00a2a68760e7d78b8ccb4a323fb44b27fa/psutil-5.1.2-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"0f6bf0464a251593159b71879d6e05ca05190e53d209a32229b793365d8de056"},"data-dist-info-metadata":{"sha256":"0f6bf0464a251593159b71879d6e05ca05190e53d209a32229b793365d8de056"},"filename":"psutil-5.1.2-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"abcfbf43eb47372e961278ee7da9a4757d59999462225b358f49c8e69a393f32"},"provenance":null,"requires-python":null,"size":187315,"upload-time":"2017-02-03T19:14:02.683013Z","url":"https://files.pythonhosted.org/packages/ed/bf/cc8cdbfc7e10518e2e1141e0724623f52bc2f41e7bdd327f8ed63e0edc01/psutil-5.1.2-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"0f6bf0464a251593159b71879d6e05ca05190e53d209a32229b793365d8de056"},"data-dist-info-metadata":{"sha256":"0f6bf0464a251593159b71879d6e05ca05190e53d209a32229b793365d8de056"},"filename":"psutil-5.1.2-cp35-cp35m-win32.whl","hashes":{"sha256":"3fd346acebeb84d9d351cabc02ea1bee1536fb7e165e7e5ead0e0912ce40cbb1"},"provenance":null,"requires-python":null,"size":186829,"upload-time":"2017-02-03T19:14:10.158362Z","url":"https://files.pythonhosted.org/packages/bf/60/e78e3455085393ab4902ef9b1c82e52d89f7b212582c58b76407a7078ad6/psutil-5.1.2-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"0f6bf0464a251593159b71879d6e05ca05190e53d209a32229b793365d8de056"},"data-dist-info-metadata":{"sha256":"0f6bf0464a251593159b71879d6e05ca05190e53d209a32229b793365d8de056"},"filename":"psutil-5.1.2-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"69644ed20c08bd257039733c71a47d871f5bdd481d63f8408e28f03f491e2a03"},"provenance":null,"requires-python":null,"size":190288,"upload-time":"2017-02-03T19:14:17.352480Z","url":"https://files.pythonhosted.org/packages/55/0a/46eba44208248da5aaeadeeab57aac3ee172df8a7291ad5d62efbed566ee/psutil-5.1.2-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"0f6bf0464a251593159b71879d6e05ca05190e53d209a32229b793365d8de056"},"data-dist-info-metadata":{"sha256":"0f6bf0464a251593159b71879d6e05ca05190e53d209a32229b793365d8de056"},"filename":"psutil-5.1.2-cp36-cp36m-win32.whl","hashes":{"sha256":"006ea083c66aa2a2be18bce84e35d0f3301d4ee3c869cb9d475fecce68470a71"},"provenance":null,"requires-python":null,"size":186826,"upload-time":"2017-02-03T19:14:24.235507Z","url":"https://files.pythonhosted.org/packages/26/e4/f5316795a709397e911eca478d547a3bb51b8dca447b7ed62d328cafa570/psutil-5.1.2-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"0f6bf0464a251593159b71879d6e05ca05190e53d209a32229b793365d8de056"},"data-dist-info-metadata":{"sha256":"0f6bf0464a251593159b71879d6e05ca05190e53d209a32229b793365d8de056"},"filename":"psutil-5.1.2-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"b64fa3a1ec7e74689b093ee6b3a487979157f81915677b9145585a2babe1b9f5"},"provenance":null,"requires-python":null,"size":190282,"upload-time":"2017-02-03T19:14:31.621854Z","url":"https://files.pythonhosted.org/packages/c0/e2/2c758b825b48eb9ae30676499a862b8475efdf683952efbb63f531413489/psutil-5.1.2-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.2.tar.gz","hashes":{"sha256":"43f32b0a392c80cff0f480bd0792763333e46d7062285dd1226b70473c55e8ac"},"provenance":null,"requires-python":null,"size":341325,"upload-time":"2017-02-03T19:14:42.976980Z","url":"https://files.pythonhosted.org/packages/19/2c/41c601cdd5586f601663d6985ff2cf1c5322f1ffd32d67d3001035d9f81d/psutil-5.1.2.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.2.win-amd64-py2.7.exe","hashes":{"sha256":"a44f1735f8464b5cde862d76a78843869da02e1454278a38b1026c9cfa172daf"},"provenance":null,"requires-python":null,"size":423677,"upload-time":"2017-02-03T19:14:56.817144Z","url":"https://files.pythonhosted.org/packages/f2/0e/f26bd5b5e0293f715e18b1de0e46eb3378c8b2a2f54ce67bb48ee47dff44/psutil-5.1.2.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.2.win-amd64-py3.3.exe","hashes":{"sha256":"e8bb29ba0e1526de8932025e07c13f6e26ab43a4cc2861b849ab2daf83ef4c3a"},"provenance":null,"requires-python":null,"size":422033,"upload-time":"2017-02-03T19:15:12.278688Z","url":"https://files.pythonhosted.org/packages/82/d1/e86d5892bfeb5b8439a96b39b2acc892aa54b9df9feb75ef7384673fe883/psutil-5.1.2.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.2.win-amd64-py3.4.exe","hashes":{"sha256":"71cd26331eb0c26ba19d5acb67716741666a581f90bce35cc5cb733eb6bbb087"},"provenance":null,"requires-python":null,"size":422041,"upload-time":"2017-02-03T19:15:27.020838Z","url":"https://files.pythonhosted.org/packages/9f/75/f9232734ab6cdf3d656a4ef65fbe5440948c38db910002a4570b01e233d2/psutil-5.1.2.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.2.win-amd64-py3.5.exe","hashes":{"sha256":"e3ea19ac2c6e1d54cb3de5919040018af2b5a0d846f7f9dc0cc4e2a125725015"},"provenance":null,"requires-python":null,"size":792618,"upload-time":"2017-02-03T19:15:52.787383Z","url":"https://files.pythonhosted.org/packages/e9/3d/cb2444851956cc4c1fe62bc4b266881e07f16b4be3cdc7b5c1ead5d99b76/psutil-5.1.2.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.2.win-amd64-py3.6.exe","hashes":{"sha256":"18ad3e6d3b46dc56eda32674e9da77527bb4ac98503e45c89837fba641a2cc16"},"provenance":null,"requires-python":null,"size":792613,"upload-time":"2017-02-03T19:16:17.350579Z","url":"https://files.pythonhosted.org/packages/36/a5/803a7fdb45924ee69221259184bb2766a1a2819d91e806912f53c6dccc6d/psutil-5.1.2.win-amd64-py3.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.2.win32-py2.7.exe","hashes":{"sha256":"1f8e54923b5d80b880d0dbc2ec5bcf51cbf1db7d54e4d2acdeeb02f42a21735a"},"provenance":null,"requires-python":null,"size":393580,"upload-time":"2017-02-03T19:16:30.238524Z","url":"https://files.pythonhosted.org/packages/b6/89/191e13e1ba569fd143c66fe340e820b909fef8599f78237cc505dd59552b/psutil-5.1.2.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.2.win32-py3.3.exe","hashes":{"sha256":"99a77884670999cf6c589539f1af3af66d8f59d9b8a9697b60398434933f56a8"},"provenance":null,"requires-python":null,"size":388425,"upload-time":"2017-02-03T19:16:43.231178Z","url":"https://files.pythonhosted.org/packages/45/ba/764a926681d90302a54494d59665ea906676b974c2ae0af7b44d5d6d9f24/psutil-5.1.2.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.2.win32-py3.4.exe","hashes":{"sha256":"2a7874d2d2718a4648cfaa4ab731f7de4867230bdcd543bf1c1fda05bc34e068"},"provenance":null,"requires-python":null,"size":388433,"upload-time":"2017-02-03T19:16:58.697184Z","url":"https://files.pythonhosted.org/packages/eb/91/98584bf6f6934c5d02b116634d6e00eddb2ec6b057556d77fdda2fe72ab0/psutil-5.1.2.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.2.win32-py3.5.exe","hashes":{"sha256":"42cbf7db0ce76431676da30e792b80e1228857e50afe859518b999125b5da673"},"provenance":null,"requires-python":null,"size":659629,"upload-time":"2017-02-03T19:17:19.107384Z","url":"https://files.pythonhosted.org/packages/80/b9/b17a77e77be75640594fba23837490a1dc425c9d6a9fc0b4997f5bfce984/psutil-5.1.2.win32-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.2.win32-py3.6.exe","hashes":{"sha256":"cfd6bddd0db750b606454791432023a4260184204f4462b2a4206b7802546ead"},"provenance":null,"requires-python":null,"size":659626,"upload-time":"2017-02-03T19:17:39.500599Z","url":"https://files.pythonhosted.org/packages/8b/bd/af0e5d889818caa35fc8dc5157363c38bdb6fcb7fa1389c5daec24857a63/psutil-5.1.2.win32-py3.6.exe","yanked":false},{"core-metadata":{"sha256":"f3276ad95ad14e59ea78a4da9482d49f8096583e96beedc6b0261259c3d8e39e"},"data-dist-info-metadata":{"sha256":"f3276ad95ad14e59ea78a4da9482d49f8096583e96beedc6b0261259c3d8e39e"},"filename":"psutil-5.1.3-cp27-none-win32.whl","hashes":{"sha256":"359a66879068ce609f8c034b3c575e357a92c033357f398490fc77cf8af46bf7"},"provenance":null,"requires-python":null,"size":185714,"upload-time":"2017-02-07T21:33:17.748014Z","url":"https://files.pythonhosted.org/packages/6f/c0/82c15d73633fdee59b5bd064396038a63a8920359c86460cd01d6ddcedfc/psutil-5.1.3-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"f3276ad95ad14e59ea78a4da9482d49f8096583e96beedc6b0261259c3d8e39e"},"data-dist-info-metadata":{"sha256":"f3276ad95ad14e59ea78a4da9482d49f8096583e96beedc6b0261259c3d8e39e"},"filename":"psutil-5.1.3-cp27-none-win_amd64.whl","hashes":{"sha256":"a0becbbe09bed44f8f5dc3909c7eb383315f932faeb0029abe8d5c737e8dcc7e"},"provenance":null,"requires-python":null,"size":188166,"upload-time":"2017-02-07T21:33:23.089530Z","url":"https://files.pythonhosted.org/packages/80/92/c5136bbade8ba85d0aa2f5d5cbe8af80bac6ea1ef77c5445aa625a8caefb/psutil-5.1.3-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ca962e552341e254e428a5583893983bb026161e7253c100c751493866519e08"},"data-dist-info-metadata":{"sha256":"ca962e552341e254e428a5583893983bb026161e7253c100c751493866519e08"},"filename":"psutil-5.1.3-cp33-cp33m-win32.whl","hashes":{"sha256":"5b2cc379287ded7f9a22521318bf010429234c2864b4146fe518f11729821771"},"provenance":null,"requires-python":null,"size":185622,"upload-time":"2017-02-07T21:33:27.925426Z","url":"https://files.pythonhosted.org/packages/b2/fb/8ad6ff1a9169b35a001ea089ab8657156df1b9b9903e218f1839fba0aae3/psutil-5.1.3-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"ca962e552341e254e428a5583893983bb026161e7253c100c751493866519e08"},"data-dist-info-metadata":{"sha256":"ca962e552341e254e428a5583893983bb026161e7253c100c751493866519e08"},"filename":"psutil-5.1.3-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"d0e88d2e8ac9ede745f589049a74ac1e3e614c4e5eed69e507d58bda8fa3c958"},"provenance":null,"requires-python":null,"size":188003,"upload-time":"2017-02-07T21:33:33.107156Z","url":"https://files.pythonhosted.org/packages/a7/8c/8a8bca010487f008ca026e69e76c53a025905a7bd1759567ec70d85d89a0/psutil-5.1.3-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ca962e552341e254e428a5583893983bb026161e7253c100c751493866519e08"},"data-dist-info-metadata":{"sha256":"ca962e552341e254e428a5583893983bb026161e7253c100c751493866519e08"},"filename":"psutil-5.1.3-cp34-cp34m-win32.whl","hashes":{"sha256":"72b67b988c0a42825a8ca76000fc385dde85652310278cca807db7dfbcba5e7e"},"provenance":null,"requires-python":null,"size":185627,"upload-time":"2017-02-07T21:33:37.199567Z","url":"https://files.pythonhosted.org/packages/81/69/0ea7d353a82df8d8251842c71131f24b8a5cfa6982f7b89e809c37819c0b/psutil-5.1.3-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"ca962e552341e254e428a5583893983bb026161e7253c100c751493866519e08"},"data-dist-info-metadata":{"sha256":"ca962e552341e254e428a5583893983bb026161e7253c100c751493866519e08"},"filename":"psutil-5.1.3-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"474ab9a6abc05fcd7bb5c32cb828f3f9fc54a2cd349d63c94dff0af3b3ba7e64"},"provenance":null,"requires-python":null,"size":188011,"upload-time":"2017-02-07T21:33:41.528046Z","url":"https://files.pythonhosted.org/packages/93/82/2172e0a319e4c1387898b588e4dc9b8f2283fd9eda0a486ebe03ece2bff7/psutil-5.1.3-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ca962e552341e254e428a5583893983bb026161e7253c100c751493866519e08"},"data-dist-info-metadata":{"sha256":"ca962e552341e254e428a5583893983bb026161e7253c100c751493866519e08"},"filename":"psutil-5.1.3-cp35-cp35m-win32.whl","hashes":{"sha256":"92bfc1f1929593ab7793ddce512295336e3e788b86a1bbf32701aa67c5ce27f4"},"provenance":null,"requires-python":null,"size":187522,"upload-time":"2017-02-07T21:33:45.934959Z","url":"https://files.pythonhosted.org/packages/43/74/08fc07b34eeb8e357dbe6bca02b844cf76cef15b36098611a50720bf7786/psutil-5.1.3-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"ca962e552341e254e428a5583893983bb026161e7253c100c751493866519e08"},"data-dist-info-metadata":{"sha256":"ca962e552341e254e428a5583893983bb026161e7253c100c751493866519e08"},"filename":"psutil-5.1.3-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"7be50561ed0060c86385c2ef4dd8a383298f29728eb6e30955ae2ebbd4554e1a"},"provenance":null,"requires-python":null,"size":190983,"upload-time":"2017-02-07T21:33:51.074513Z","url":"https://files.pythonhosted.org/packages/fa/42/42e547764bf65617077b696c4c49bed6109b00696882a196008de5f8917b/psutil-5.1.3-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ca962e552341e254e428a5583893983bb026161e7253c100c751493866519e08"},"data-dist-info-metadata":{"sha256":"ca962e552341e254e428a5583893983bb026161e7253c100c751493866519e08"},"filename":"psutil-5.1.3-cp36-cp36m-win32.whl","hashes":{"sha256":"4de5566d9d8c3695726f9ec3324cd56d3eb363365508ea39854d2ebe5d57b945"},"provenance":null,"requires-python":null,"size":187516,"upload-time":"2017-02-07T21:33:56.040728Z","url":"https://files.pythonhosted.org/packages/c5/5f/71b89c9dede1da356dbbe321ae410786b94e0b516791191c008864844733/psutil-5.1.3-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"ca962e552341e254e428a5583893983bb026161e7253c100c751493866519e08"},"data-dist-info-metadata":{"sha256":"ca962e552341e254e428a5583893983bb026161e7253c100c751493866519e08"},"filename":"psutil-5.1.3-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"678ef7b4e38281ff16dbdac98fc1d0679d46fed3fadd5d4648096fbb6d6b1b95"},"provenance":null,"requires-python":null,"size":190979,"upload-time":"2017-02-07T21:34:01.058372Z","url":"https://files.pythonhosted.org/packages/bc/21/1823e2349b1f6ec526a55d21497e5d627ac26a6c5a3d49a07b4afad45547/psutil-5.1.3-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.3.tar.gz","hashes":{"sha256":"959bd58bdc8152b0a143cb3bd822d4a1b8f7230617b0e3eb2ff6e63812120f2b"},"provenance":null,"requires-python":null,"size":341980,"upload-time":"2017-02-07T21:34:07.636785Z","url":"https://files.pythonhosted.org/packages/78/0a/aa90434c6337dd50d182a81fe4ae4822c953e166a163d1bf5f06abb1ac0b/psutil-5.1.3.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.3.win-amd64-py2.7.exe","hashes":{"sha256":"6c288b7a639f341391ba474d4c8fb495a19220015284b46e7b23f626afafc810"},"provenance":null,"requires-python":null,"size":424369,"upload-time":"2017-02-07T21:34:15.354723Z","url":"https://files.pythonhosted.org/packages/07/bb/aac12b9c56722cf8b6ed0c89eccf1e3db75795576b7e3575001248802c0d/psutil-5.1.3.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.3.win-amd64-py3.3.exe","hashes":{"sha256":"c4b53e0630b83c784f807170ae2d12f1cf1e45e3913f35f9784e5556ba4a0786"},"provenance":null,"requires-python":null,"size":422726,"upload-time":"2017-02-07T21:34:24.113585Z","url":"https://files.pythonhosted.org/packages/1e/5d/4804b1d23e0e8c442876c02438c4f89b3512806b034dea5807ca1ddd6535/psutil-5.1.3.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.3.win-amd64-py3.4.exe","hashes":{"sha256":"57a4e51a0f2fd8f361fcf545eeff54932a29b716ad01e60247d1abaffbc1b954"},"provenance":null,"requires-python":null,"size":422733,"upload-time":"2017-02-07T21:34:32.211085Z","url":"https://files.pythonhosted.org/packages/51/70/34ea430cf5c21540e30b805f0740d81f75c5766c2f68af9207ae18e147dc/psutil-5.1.3.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.3.win-amd64-py3.5.exe","hashes":{"sha256":"4b26f56f09ad206d9fb8b2fa29926a696419b26e2c5d461afe477481cec1105c"},"provenance":null,"requires-python":null,"size":793310,"upload-time":"2017-02-07T21:34:46.367507Z","url":"https://files.pythonhosted.org/packages/a2/8c/004fabd6406879fd1caa1ed4ea7ab850afad6e27e3c5b8e8a4d0d134de3e/psutil-5.1.3.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.3.win-amd64-py3.6.exe","hashes":{"sha256":"94ed102897b8c7103ff51e2b2953caf56bb80c3343523fd3013db3ec91bd8c4b"},"provenance":null,"requires-python":null,"size":793304,"upload-time":"2017-02-07T21:35:00.329688Z","url":"https://files.pythonhosted.org/packages/55/18/6a48a9b9dad56c54236260c1e0e2313497b3af176b053a110ea134b9bb9f/psutil-5.1.3.win-amd64-py3.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.3.win32-py2.7.exe","hashes":{"sha256":"9fcac25e01c0f9f1b6d86c860c6d4da627e458f277f24415f15b1b29cce35f60"},"provenance":null,"requires-python":null,"size":394272,"upload-time":"2017-02-07T21:35:07.791551Z","url":"https://files.pythonhosted.org/packages/33/97/442e6eefe2a12cd00d09721fb24ddf726dd62c1073579a860682919cc640/psutil-5.1.3.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.3.win32-py3.3.exe","hashes":{"sha256":"8349494ee9405a31f4f9d9d3564663c870fed5dd62efd2edfdf64c5841bb838f"},"provenance":null,"requires-python":null,"size":389117,"upload-time":"2017-02-07T21:35:16.679190Z","url":"https://files.pythonhosted.org/packages/ca/46/6d9a5c657298d1363cb37ab0f84eb1fd54639fa4b2729523a68cd6a1b043/psutil-5.1.3.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.3.win32-py3.4.exe","hashes":{"sha256":"c8dc71de8ba61604a5cae5dee5330229dc71538c82ef13458cee838b6c0f6435"},"provenance":null,"requires-python":null,"size":389124,"upload-time":"2017-02-07T21:35:24.590876Z","url":"https://files.pythonhosted.org/packages/5b/c4/7056b6c602ff5be0095fe403617cded940a75a80db49bb51846bc235a0bb/psutil-5.1.3.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.3.win32-py3.5.exe","hashes":{"sha256":"38c1e88f3a8a548d9caa7f56db1cc7d508eda48eb2c4aa484a908bc5d06f87bd"},"provenance":null,"requires-python":null,"size":660318,"upload-time":"2017-02-07T21:35:36.604103Z","url":"https://files.pythonhosted.org/packages/5f/b3/966c2979172a46f9fe42f34ce7321a59102054e26cdf9b26e3d604807953/psutil-5.1.3.win32-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.1.3.win32-py3.6.exe","hashes":{"sha256":"0961ebc2ba4b1c811ef164612d0d963532ad0a9af1755e022a99648a9027b065"},"provenance":null,"requires-python":null,"size":660315,"upload-time":"2017-02-07T21:35:48.440270Z","url":"https://files.pythonhosted.org/packages/be/43/8f0099425146c01c2d77e3ac90b28a7f42d69ccb2af6e161f059db132d99/psutil-5.1.3.win32-py3.6.exe","yanked":false},{"core-metadata":{"sha256":"1b02bec8364ab9ba69161e6031bf88699daaa9da96dcfa6527b452f562d8d646"},"data-dist-info-metadata":{"sha256":"1b02bec8364ab9ba69161e6031bf88699daaa9da96dcfa6527b452f562d8d646"},"filename":"psutil-5.2.0-cp27-none-win32.whl","hashes":{"sha256":"6eb2f6fb976152f320ee48a90ab732d694b2ae0c835260ce4f5af3907584448a"},"provenance":null,"requires-python":null,"size":187506,"upload-time":"2017-03-05T04:50:46.045903Z","url":"https://files.pythonhosted.org/packages/d2/56/56a15e285c7cf0104ed9fc569b2c75a24f97e7ab5c34567956b266d23ba3/psutil-5.2.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"1b02bec8364ab9ba69161e6031bf88699daaa9da96dcfa6527b452f562d8d646"},"data-dist-info-metadata":{"sha256":"1b02bec8364ab9ba69161e6031bf88699daaa9da96dcfa6527b452f562d8d646"},"filename":"psutil-5.2.0-cp27-none-win_amd64.whl","hashes":{"sha256":"35898b80a3f393a7ace8ad5da9a26800676b7fc40628a3a334902b9d0e444c8d"},"provenance":null,"requires-python":null,"size":189973,"upload-time":"2017-03-05T04:50:49.198757Z","url":"https://files.pythonhosted.org/packages/a5/a5/1039829542b856ca4d3d40bb4978fbb679b7f0bb684ece6340ce655aedc9/psutil-5.2.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"49ff4351990c237dc58c60a958925eef7156ce86008b0f5cfef7af0e197da80e"},"data-dist-info-metadata":{"sha256":"49ff4351990c237dc58c60a958925eef7156ce86008b0f5cfef7af0e197da80e"},"filename":"psutil-5.2.0-cp33-cp33m-win32.whl","hashes":{"sha256":"5626533fc459ce1ac4bd017f7a38b99947c039d79175a10a2a6b6246e3a82fc8"},"provenance":null,"requires-python":null,"size":187442,"upload-time":"2017-03-05T04:50:52.517268Z","url":"https://files.pythonhosted.org/packages/8c/1e/7e6ac521b3c393b2f312f1c3795d702f3267dca23d603827d673b8170920/psutil-5.2.0-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"49ff4351990c237dc58c60a958925eef7156ce86008b0f5cfef7af0e197da80e"},"data-dist-info-metadata":{"sha256":"49ff4351990c237dc58c60a958925eef7156ce86008b0f5cfef7af0e197da80e"},"filename":"psutil-5.2.0-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"d34cc4d48245873492e4befc5c58a146f0f6c98038ffa2430e191a6752717c61"},"provenance":null,"requires-python":null,"size":189849,"upload-time":"2017-03-05T04:50:55.188058Z","url":"https://files.pythonhosted.org/packages/b6/d0/6edd271e3ca150104c818ec0f4b2affc447fe79ec1504506cecb2900d391/psutil-5.2.0-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"49ff4351990c237dc58c60a958925eef7156ce86008b0f5cfef7af0e197da80e"},"data-dist-info-metadata":{"sha256":"49ff4351990c237dc58c60a958925eef7156ce86008b0f5cfef7af0e197da80e"},"filename":"psutil-5.2.0-cp34-cp34m-win32.whl","hashes":{"sha256":"5834168071a92037736142616b33691ec4786f8806e28355e74b2e1a037cad4c"},"provenance":null,"requires-python":null,"size":187450,"upload-time":"2017-03-05T04:50:58.524346Z","url":"https://files.pythonhosted.org/packages/89/88/8fb4ce470a2022c33ab3cd16b3f2152f544e264c9db0f2f7159a93e0d2a3/psutil-5.2.0-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"49ff4351990c237dc58c60a958925eef7156ce86008b0f5cfef7af0e197da80e"},"data-dist-info-metadata":{"sha256":"49ff4351990c237dc58c60a958925eef7156ce86008b0f5cfef7af0e197da80e"},"filename":"psutil-5.2.0-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"d29c24bc7c14ecb4e64b3b748814ebe0e3ac049802ea7f129edbfcb068e75c16"},"provenance":null,"requires-python":null,"size":189841,"upload-time":"2017-03-05T04:51:03.287652Z","url":"https://files.pythonhosted.org/packages/43/9b/35cae8c56d3ee2e9a02599fba6a2e1f3fcf3553fe55f70c0ea723f9a9522/psutil-5.2.0-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"49ff4351990c237dc58c60a958925eef7156ce86008b0f5cfef7af0e197da80e"},"data-dist-info-metadata":{"sha256":"49ff4351990c237dc58c60a958925eef7156ce86008b0f5cfef7af0e197da80e"},"filename":"psutil-5.2.0-cp35-cp35m-win32.whl","hashes":{"sha256":"8353692da46bc6024b4001a9ed8849beb863fbb1d022553dd4ed8348745540bb"},"provenance":null,"requires-python":null,"size":189373,"upload-time":"2017-03-05T04:51:07.095988Z","url":"https://files.pythonhosted.org/packages/65/35/fff62f84dc6c165e8a9f7646e2c106bd223a3967a0a3f471979b38b5a5c0/psutil-5.2.0-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"49ff4351990c237dc58c60a958925eef7156ce86008b0f5cfef7af0e197da80e"},"data-dist-info-metadata":{"sha256":"49ff4351990c237dc58c60a958925eef7156ce86008b0f5cfef7af0e197da80e"},"filename":"psutil-5.2.0-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"1e00f5684fb335dacfa750e5e01f83bb79d521eb5f0805b798de0a29a1fb25d4"},"provenance":null,"requires-python":null,"size":192813,"upload-time":"2017-03-05T04:51:10.595180Z","url":"https://files.pythonhosted.org/packages/94/d2/f78b5a0ded0993f4c5127bf17427e4bc10b183dc102a5e469d7f6725ecb9/psutil-5.2.0-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"49ff4351990c237dc58c60a958925eef7156ce86008b0f5cfef7af0e197da80e"},"data-dist-info-metadata":{"sha256":"49ff4351990c237dc58c60a958925eef7156ce86008b0f5cfef7af0e197da80e"},"filename":"psutil-5.2.0-cp36-cp36m-win32.whl","hashes":{"sha256":"55d546333f1423ad219a0798867a9bbf9a90e1912c3336ad275476473624c071"},"provenance":null,"requires-python":null,"size":189376,"upload-time":"2017-03-05T04:51:15.088897Z","url":"https://files.pythonhosted.org/packages/91/8e/bd4f794b9f092d82a5b63b17da95ebd864f544ff62fb70bb1bce0687b013/psutil-5.2.0-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"49ff4351990c237dc58c60a958925eef7156ce86008b0f5cfef7af0e197da80e"},"data-dist-info-metadata":{"sha256":"49ff4351990c237dc58c60a958925eef7156ce86008b0f5cfef7af0e197da80e"},"filename":"psutil-5.2.0-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"548f14e3e21225884904e3ab228a769a73f886a3394399c591ec5f31fedc48ac"},"provenance":null,"requires-python":null,"size":192808,"upload-time":"2017-03-05T04:51:18.018177Z","url":"https://files.pythonhosted.org/packages/75/65/8499f256dc203b94f8a439f52b092742247668365dcb0997aee7349d530d/psutil-5.2.0-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.0.tar.gz","hashes":{"sha256":"2fc91d068faa5613c093335f0e758673ef8c722ad4bfa4aded64c13ae69089eb"},"provenance":null,"requires-python":null,"size":345519,"upload-time":"2017-03-05T04:51:23.230758Z","url":"https://files.pythonhosted.org/packages/3c/2f/f3ab91349c666f009077157b12057e613a3152a46a6c3be883777546b6de/psutil-5.2.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.0.win-amd64-py2.7.exe","hashes":{"sha256":"7cd5dd38e08d74112c68a8f4e9b9e12fac6c6f6270792604c79a9bbd574053fa"},"provenance":null,"requires-python":null,"size":426460,"upload-time":"2017-03-05T04:51:28.726871Z","url":"https://files.pythonhosted.org/packages/ca/6f/6289db524b6aae542fa36d539524e74f25d7f9296aabadb3b5a9f17746e8/psutil-5.2.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.0.win-amd64-py3.3.exe","hashes":{"sha256":"e5f1688d9bfd9e122edd35adcd8a0050430397094d08d95c380bd9c7dae48da3"},"provenance":null,"requires-python":null,"size":424854,"upload-time":"2017-03-05T04:51:34.189840Z","url":"https://files.pythonhosted.org/packages/93/7c/e92a80f5803be3febbcb40807d5d2bfe66dfe20b256c07616599c14ba2aa/psutil-5.2.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.0.win-amd64-py3.4.exe","hashes":{"sha256":"11e684bf163821bd73843ebf9a27b7cb6f8a8325b943954b49b1f622264b6e80"},"provenance":null,"requires-python":null,"size":424846,"upload-time":"2017-03-05T04:51:38.965015Z","url":"https://files.pythonhosted.org/packages/4a/82/5a78b9d40c17dc4d01a06345596ac1e3f7ba590f7329d83e80b817d47f9b/psutil-5.2.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.0.win-amd64-py3.5.exe","hashes":{"sha256":"9eeae9bef0875432b9aea9504ed4f7c72f8ee3e8d7ded63e484a453ee82d4a98"},"provenance":null,"requires-python":null,"size":793372,"upload-time":"2017-03-05T04:51:43.517651Z","url":"https://files.pythonhosted.org/packages/dd/f1/4bf05d2b34198954b5fd6455ebe06dd08cae5357354f6f142ef4321e41ab/psutil-5.2.0.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.0.win-amd64-py3.6.exe","hashes":{"sha256":"461445afc35f98d4b7781ea2f67d0ca4ae6cf36c065ee0f4be61ace59045a2a5"},"provenance":null,"requires-python":null,"size":795416,"upload-time":"2017-03-05T04:51:53.165366Z","url":"https://files.pythonhosted.org/packages/94/59/c54e7f853586561ae42c9bcf1aa0644e8c0298479e4654b4ca36fa9eafe6/psutil-5.2.0.win-amd64-py3.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.0.win32-py2.7.exe","hashes":{"sha256":"8bbbd02eb474045d201f6617d16dc8ee1d9903d5cec94f7f39cce610fc1e924b"},"provenance":null,"requires-python":null,"size":396349,"upload-time":"2017-03-05T04:51:59.453206Z","url":"https://files.pythonhosted.org/packages/21/79/40ea4e11ef6ca2b044d0aeb28d829a716a565b34c7786f779e99c005b80a/psutil-5.2.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.0.win32-py3.3.exe","hashes":{"sha256":"63d4320b0f3498da3551028a6ab9ee1c5aebabe0d23a7c38600c35333953ef6c"},"provenance":null,"requires-python":null,"size":391219,"upload-time":"2017-03-05T04:52:03.289639Z","url":"https://files.pythonhosted.org/packages/2e/9d/def6a3fb8150adfd71889f3e3d48160a1ba6210911baf12ce3ebd294307c/psutil-5.2.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.0.win32-py3.4.exe","hashes":{"sha256":"3c06b0162192db85e04846674a55915fca80f728cf626228a6b31684fc6930da"},"provenance":null,"requires-python":null,"size":391229,"upload-time":"2017-03-05T04:52:10.103932Z","url":"https://files.pythonhosted.org/packages/2c/ae/8616ac1eb00a7770d837b15ebb9ae759c43623c182f32fd43d2e6fed8649/psutil-5.2.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.0.win32-py3.5.exe","hashes":{"sha256":"0ba082468d6b45fb15cc1c4488aaf3ffcf0616a674c46393bf04eccc8d7c2196"},"provenance":null,"requires-python":null,"size":660405,"upload-time":"2017-03-05T04:52:18.130601Z","url":"https://files.pythonhosted.org/packages/f7/b6/c8cb94fd6696414a66021aa2229747d71612551eade262e9ab52eeb54ee2/psutil-5.2.0.win32-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.0.win32-py3.6.exe","hashes":{"sha256":"6ed1cb1c9339493e1f3c379de0155c543a4c8de18224bda894190843f9509cad"},"provenance":null,"requires-python":null,"size":662455,"upload-time":"2017-03-05T04:52:27.540851Z","url":"https://files.pythonhosted.org/packages/5d/9d/8b552e9d4c2a5c3baa00d1baa1468f2a8128acd3eba79ef39e59c182676a/psutil-5.2.0.win32-py3.6.exe","yanked":false},{"core-metadata":{"sha256":"84b8c9735984f276c69b5871ce5635af6381a960dd516077459ef1ded0e0580a"},"data-dist-info-metadata":{"sha256":"84b8c9735984f276c69b5871ce5635af6381a960dd516077459ef1ded0e0580a"},"filename":"psutil-5.2.1-cp27-none-win32.whl","hashes":{"sha256":"4e236c4ec6b0b20171c2477ded7a5b4402e4a877530640f814df839af0a40e30"},"provenance":null,"requires-python":null,"size":187855,"upload-time":"2017-03-24T15:42:20.835636Z","url":"https://files.pythonhosted.org/packages/3d/14/1242a70873873e92732dc35162317df448503a7a32e29c8bdbe30d4fa175/psutil-5.2.1-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"84b8c9735984f276c69b5871ce5635af6381a960dd516077459ef1ded0e0580a"},"data-dist-info-metadata":{"sha256":"84b8c9735984f276c69b5871ce5635af6381a960dd516077459ef1ded0e0580a"},"filename":"psutil-5.2.1-cp27-none-win_amd64.whl","hashes":{"sha256":"e88fe0d0ca5a9623f0d8d6be05a82e33984f27b067f08806bf8a548ba4361b40"},"provenance":null,"requires-python":null,"size":190301,"upload-time":"2017-03-24T15:42:26.623783Z","url":"https://files.pythonhosted.org/packages/4c/03/fffda9f6e1ca56ce989362969b709bf7a7ade16abf7d82661bbec96580f5/psutil-5.2.1-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"187f675b1b9bc62e2ffab1ba395e87ae25e1679d8867bd305462e360a04fd386"},"data-dist-info-metadata":{"sha256":"187f675b1b9bc62e2ffab1ba395e87ae25e1679d8867bd305462e360a04fd386"},"filename":"psutil-5.2.1-cp33-cp33m-win32.whl","hashes":{"sha256":"54275bdbfbd20909d37ed7a2570cf9dd373ac702a89bac4814249cbc10503c03"},"provenance":null,"requires-python":null,"size":187792,"upload-time":"2017-03-24T15:42:30.952153Z","url":"https://files.pythonhosted.org/packages/d7/e0/4fde7667fad4271c06ed5e533a156bd600cdad1b69d8e6f278fe425452d2/psutil-5.2.1-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"187f675b1b9bc62e2ffab1ba395e87ae25e1679d8867bd305462e360a04fd386"},"data-dist-info-metadata":{"sha256":"187f675b1b9bc62e2ffab1ba395e87ae25e1679d8867bd305462e360a04fd386"},"filename":"psutil-5.2.1-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"316c3e334b046dc12b4f0a3dafa1d1c394e38106ac519003694fc8aeb672eafd"},"provenance":null,"requires-python":null,"size":190190,"upload-time":"2017-03-24T15:42:35.956721Z","url":"https://files.pythonhosted.org/packages/a8/c5/63453c20ac576ccb58ee56f88388434380f5e2a729aa08885d2655eb83b7/psutil-5.2.1-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"187f675b1b9bc62e2ffab1ba395e87ae25e1679d8867bd305462e360a04fd386"},"data-dist-info-metadata":{"sha256":"187f675b1b9bc62e2ffab1ba395e87ae25e1679d8867bd305462e360a04fd386"},"filename":"psutil-5.2.1-cp34-cp34m-win32.whl","hashes":{"sha256":"2249c687088145dcce87ecb90221258f9c0e7b7cea830886656cf07351e50e1b"},"provenance":null,"requires-python":null,"size":187785,"upload-time":"2017-03-24T15:42:40.746327Z","url":"https://files.pythonhosted.org/packages/59/8b/8ebb86ae5c0ba81e95bae8263de81038d3d7ee8a050f31b2b58f1a330198/psutil-5.2.1-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"187f675b1b9bc62e2ffab1ba395e87ae25e1679d8867bd305462e360a04fd386"},"data-dist-info-metadata":{"sha256":"187f675b1b9bc62e2ffab1ba395e87ae25e1679d8867bd305462e360a04fd386"},"filename":"psutil-5.2.1-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"f74532c2037fac87b76737798c74102e17f8594ea9de07aa3cb19027a630bdb0"},"provenance":null,"requires-python":null,"size":190213,"upload-time":"2017-03-24T15:42:45.196850Z","url":"https://files.pythonhosted.org/packages/e7/81/c4dd47453864984d1bd5ad0c387efc11aa6791b5abb5b369ebe2e81f7ada/psutil-5.2.1-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"187f675b1b9bc62e2ffab1ba395e87ae25e1679d8867bd305462e360a04fd386"},"data-dist-info-metadata":{"sha256":"187f675b1b9bc62e2ffab1ba395e87ae25e1679d8867bd305462e360a04fd386"},"filename":"psutil-5.2.1-cp35-cp35m-win32.whl","hashes":{"sha256":"c7c8ed864a9ef04d4736a998273e3ba0f95f22300f1e082c13a7c824b514f411"},"provenance":null,"requires-python":null,"size":189737,"upload-time":"2017-03-24T15:42:50.097781Z","url":"https://files.pythonhosted.org/packages/76/3b/2e6b3306dd2927fef9c81fdc29bc450beeb6f4bfe4cddec80260ab042900/psutil-5.2.1-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"187f675b1b9bc62e2ffab1ba395e87ae25e1679d8867bd305462e360a04fd386"},"data-dist-info-metadata":{"sha256":"187f675b1b9bc62e2ffab1ba395e87ae25e1679d8867bd305462e360a04fd386"},"filename":"psutil-5.2.1-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"7a21b9d908a3bf381cc160c157a06bfcea3c6402362b26a2489566914cea9cc5"},"provenance":null,"requires-python":null,"size":193175,"upload-time":"2017-03-24T15:42:55.344665Z","url":"https://files.pythonhosted.org/packages/de/ee/cf9ecf7cea0a984a360bc889bb0bf11335755d5b7d2be9d8399fe5dc01fb/psutil-5.2.1-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"187f675b1b9bc62e2ffab1ba395e87ae25e1679d8867bd305462e360a04fd386"},"data-dist-info-metadata":{"sha256":"187f675b1b9bc62e2ffab1ba395e87ae25e1679d8867bd305462e360a04fd386"},"filename":"psutil-5.2.1-cp36-cp36m-win32.whl","hashes":{"sha256":"d1efbfc743555e7fd366956d8fe39690a3ae87e8e9e9ac06cc80bd7e2ca3059b"},"provenance":null,"requires-python":null,"size":189738,"upload-time":"2017-03-24T15:43:00.448274Z","url":"https://files.pythonhosted.org/packages/15/18/e6b1b4288d885218c845f9a340e236f03352358fc83675b9b8ef96e26227/psutil-5.2.1-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"187f675b1b9bc62e2ffab1ba395e87ae25e1679d8867bd305462e360a04fd386"},"data-dist-info-metadata":{"sha256":"187f675b1b9bc62e2ffab1ba395e87ae25e1679d8867bd305462e360a04fd386"},"filename":"psutil-5.2.1-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"cf40e944f47000375320ce0e712585321ec624a0ef67e8259f522e51bcb35a35"},"provenance":null,"requires-python":null,"size":193175,"upload-time":"2017-03-24T15:43:06.043078Z","url":"https://files.pythonhosted.org/packages/e8/7c/240fd3dfcec8d839a9a48dd2f88ba5f6e687263adc8b2452ed973b66b862/psutil-5.2.1-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.1.tar.gz","hashes":{"sha256":"fe0ea53b302f68fca1c2a3bac289e11344456786141b73391ed4022b412d5455"},"provenance":null,"requires-python":null,"size":347241,"upload-time":"2017-03-24T15:43:12.551784Z","url":"https://files.pythonhosted.org/packages/b8/47/c85fbcd23f40892db6ecc88782beb6ee66d22008c2f9821d777cb1984240/psutil-5.2.1.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.1.win-amd64-py2.7.exe","hashes":{"sha256":"60e9bd558d640eaf9c7a4fbb0627b423b1e58fce95b41b8a24fda9145b753471"},"provenance":null,"requires-python":null,"size":426778,"upload-time":"2017-03-24T15:43:20.982480Z","url":"https://files.pythonhosted.org/packages/88/e8/40e20ea582157c81e55e1765139a5f6e969d8c01e47c016d90946b495531/psutil-5.2.1.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.1.win-amd64-py3.3.exe","hashes":{"sha256":"318cf7bf546a23564fe4f049eae0bf205895a0524120bd549de3e46599a7f265"},"provenance":null,"requires-python":null,"size":425186,"upload-time":"2017-03-24T15:43:29.554448Z","url":"https://files.pythonhosted.org/packages/dd/55/2a74e973eb217fa5006c910a24abbd720efb7720beae9659be14fe96a413/psutil-5.2.1.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.1.win-amd64-py3.4.exe","hashes":{"sha256":"3bcfbe8b8141c8145f1d54c3f9c2c86597508bb7cc2552e333de770a3c9b9368"},"provenance":null,"requires-python":null,"size":425211,"upload-time":"2017-03-24T15:43:39.017569Z","url":"https://files.pythonhosted.org/packages/9d/12/92575d652d33d28e6f8b0f858f3db326db5ffc4c8d55b09ac411b021d86d/psutil-5.2.1.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.1.win-amd64-py3.5.exe","hashes":{"sha256":"979a5804366b47acd0ebf28923ee645e9fc29f4c54cbc44c41d112a1cd36e9ba"},"provenance":null,"requires-python":null,"size":793725,"upload-time":"2017-03-24T15:43:54.421574Z","url":"https://files.pythonhosted.org/packages/7f/58/de0b10442e2f277de4de0ecfae277576a6bfac1a3137fe547a4085dafa32/psutil-5.2.1.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.1.win-amd64-py3.6.exe","hashes":{"sha256":"bf7d2cff21e3262d2b3e33a4b9dc27bbae81e851d694667d68dc7405c67ff31f"},"provenance":null,"requires-python":null,"size":795772,"upload-time":"2017-03-24T15:44:09.051326Z","url":"https://files.pythonhosted.org/packages/33/c0/7094de6644330b8dcdfefb0bae0a00379238588a6cf6cc9cd71c69e0cdce/psutil-5.2.1.win-amd64-py3.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.1.win32-py2.7.exe","hashes":{"sha256":"ad8b603d7cc6d070cf07d39276869683474dace4da51d8050f29893ac2e22baf"},"provenance":null,"requires-python":null,"size":396688,"upload-time":"2017-03-24T15:44:17.148742Z","url":"https://files.pythonhosted.org/packages/a9/7a/5d19102362c28b6a478f9a7f3262f3ca301f8c5fed12e8d0af9e9e82e6a2/psutil-5.2.1.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.1.win32-py3.3.exe","hashes":{"sha256":"71fbaa3649aa8fa92edb1ad2b45de1e9caa7ffc63f448be951d43d6b5c6263b1"},"provenance":null,"requires-python":null,"size":391560,"upload-time":"2017-03-24T15:44:25.069190Z","url":"https://files.pythonhosted.org/packages/0b/01/f2963d84b439b0802c2354d0f777b5ed4bd0c2c11161ba81e7057a0d0523/psutil-5.2.1.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.1.win32-py3.4.exe","hashes":{"sha256":"cab6e8cfab49511f34e7ae40885792d7e655bb107f6f3c89440d5061cb19ad2f"},"provenance":null,"requires-python":null,"size":391554,"upload-time":"2017-03-24T15:44:33.671369Z","url":"https://files.pythonhosted.org/packages/d5/46/b36ff70ba0ba3b92bb5088be595fdb5641ffd982bac8e206e7c4936b2dc5/psutil-5.2.1.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.1.win32-py3.5.exe","hashes":{"sha256":"0e9e3d74f6ee1a6cac503c0bba08563dc3954e723b8392a4c74ce36f46e119ea"},"provenance":null,"requires-python":null,"size":660759,"upload-time":"2017-03-24T15:44:45.335659Z","url":"https://files.pythonhosted.org/packages/62/81/e7431ad75f9d9ae1524ee886c1aff25ec3714058de6568d305de2e0c8373/psutil-5.2.1.win32-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.1.win32-py3.6.exe","hashes":{"sha256":"03e419618c3c715489ca5073cbdac6a0b12da41def69d3e4ee83f18fbb5798e5"},"provenance":null,"requires-python":null,"size":662807,"upload-time":"2017-03-24T15:44:56.446939Z","url":"https://files.pythonhosted.org/packages/77/c8/e256a28a63d06fe028f8837b860b7f6440c6ef9a475fb8c4490e1e08498b/psutil-5.2.1.win32-py3.6.exe","yanked":false},{"core-metadata":{"sha256":"0414f56fbdc05bdf6b0f2659c3f90672e666ae73ba46b3281e07291af1d58219"},"data-dist-info-metadata":{"sha256":"0414f56fbdc05bdf6b0f2659c3f90672e666ae73ba46b3281e07291af1d58219"},"filename":"psutil-5.2.2-cp27-none-win32.whl","hashes":{"sha256":"db473f0d45a56d422502043f3755385fcfd83f5bb0947bc807fcad689230f37f"},"provenance":null,"requires-python":null,"size":187988,"upload-time":"2017-04-10T17:19:35.132698Z","url":"https://files.pythonhosted.org/packages/9c/31/c651e4c475a4d0df9609024a86fcb358a21b7a01872f7c69c7cf501a2896/psutil-5.2.2-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"0414f56fbdc05bdf6b0f2659c3f90672e666ae73ba46b3281e07291af1d58219"},"data-dist-info-metadata":{"sha256":"0414f56fbdc05bdf6b0f2659c3f90672e666ae73ba46b3281e07291af1d58219"},"filename":"psutil-5.2.2-cp27-none-win_amd64.whl","hashes":{"sha256":"dcd9d3131f83480648da40d2c39403657c63a81e56e4e8d8e905bf65c133d59c"},"provenance":null,"requires-python":null,"size":190432,"upload-time":"2017-04-10T17:19:39.867178Z","url":"https://files.pythonhosted.org/packages/6e/5c/15f41041a321ffd4058ae67aade067924489acba6d277e10571b59b3127c/psutil-5.2.2-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"4c73be3d0927068fd96000b5591a3206eb03f1decaf76929c5e5cd06d004e214"},"data-dist-info-metadata":{"sha256":"4c73be3d0927068fd96000b5591a3206eb03f1decaf76929c5e5cd06d004e214"},"filename":"psutil-5.2.2-cp33-cp33m-win32.whl","hashes":{"sha256":"3f79a044db0aae96592ef42be459e37095d0c2cebcae4fd7baf486d37a85a8cd"},"provenance":null,"requires-python":null,"size":187924,"upload-time":"2017-04-10T17:19:44.464637Z","url":"https://files.pythonhosted.org/packages/04/a5/a027a8584208fa2cb6a88e6337b06b11388edf7d39feb0a897c9c2024639/psutil-5.2.2-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"4c73be3d0927068fd96000b5591a3206eb03f1decaf76929c5e5cd06d004e214"},"data-dist-info-metadata":{"sha256":"4c73be3d0927068fd96000b5591a3206eb03f1decaf76929c5e5cd06d004e214"},"filename":"psutil-5.2.2-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"838c66c123cb024bf8c8d2fec902b38c51f75b27988f4487d81383d1d3d8a8ce"},"provenance":null,"requires-python":null,"size":190325,"upload-time":"2017-04-10T17:19:49.563017Z","url":"https://files.pythonhosted.org/packages/bc/95/385e0f7e0299295401d41dd4cb6e568bf50c884af336b92a69d16981f71c/psutil-5.2.2-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"4c73be3d0927068fd96000b5591a3206eb03f1decaf76929c5e5cd06d004e214"},"data-dist-info-metadata":{"sha256":"4c73be3d0927068fd96000b5591a3206eb03f1decaf76929c5e5cd06d004e214"},"filename":"psutil-5.2.2-cp34-cp34m-win32.whl","hashes":{"sha256":"a155875d2fedb614c2cd687fe47953d03a47f76eb39bd5756931b288b685655f"},"provenance":null,"requires-python":null,"size":187918,"upload-time":"2017-04-10T17:19:54.252736Z","url":"https://files.pythonhosted.org/packages/12/ad/aca0f4f146b25fb2b7e9e0735287ba3ebcc02eb2bf84d49916aef730d860/psutil-5.2.2-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"4c73be3d0927068fd96000b5591a3206eb03f1decaf76929c5e5cd06d004e214"},"data-dist-info-metadata":{"sha256":"4c73be3d0927068fd96000b5591a3206eb03f1decaf76929c5e5cd06d004e214"},"filename":"psutil-5.2.2-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"a989876ac0cc7942ef9481b96d3bfc02777dc798d4a7a1b4e8f0f284228f3434"},"provenance":null,"requires-python":null,"size":190345,"upload-time":"2017-04-10T17:19:59.043821Z","url":"https://files.pythonhosted.org/packages/f4/45/6cbf2b7a55375f6aafc68f33581aa143f86ae1be9112546f04d8e9ee34da/psutil-5.2.2-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"4c73be3d0927068fd96000b5591a3206eb03f1decaf76929c5e5cd06d004e214"},"data-dist-info-metadata":{"sha256":"4c73be3d0927068fd96000b5591a3206eb03f1decaf76929c5e5cd06d004e214"},"filename":"psutil-5.2.2-cp35-cp35m-win32.whl","hashes":{"sha256":"32616c5736f1de446e77865305e7f56905c718991f820c8286436adea8192f32"},"provenance":null,"requires-python":null,"size":189869,"upload-time":"2017-04-10T17:20:03.832890Z","url":"https://files.pythonhosted.org/packages/7b/1d/8cef4ee6c1a49b1204dcdca1231ac773e27f2ed0abbbf42deb14aaf2b5cc/psutil-5.2.2-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"4c73be3d0927068fd96000b5591a3206eb03f1decaf76929c5e5cd06d004e214"},"data-dist-info-metadata":{"sha256":"4c73be3d0927068fd96000b5591a3206eb03f1decaf76929c5e5cd06d004e214"},"filename":"psutil-5.2.2-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"50c8ddc3a6d1cda1de6d7aaf1af10896832c6d686fc7d0fe3d01c1eb51e6f521"},"provenance":null,"requires-python":null,"size":193304,"upload-time":"2017-04-10T17:20:09.444679Z","url":"https://files.pythonhosted.org/packages/82/f4/9d4cb35c5e1c84f93718d1851adf0b4147b253111cb89f1996a08d14dba5/psutil-5.2.2-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"4c73be3d0927068fd96000b5591a3206eb03f1decaf76929c5e5cd06d004e214"},"data-dist-info-metadata":{"sha256":"4c73be3d0927068fd96000b5591a3206eb03f1decaf76929c5e5cd06d004e214"},"filename":"psutil-5.2.2-cp36-cp36m-win32.whl","hashes":{"sha256":"e8b65a80e978af9bf10be423442155032c589b7042b4a26edc410dc36819d65e"},"provenance":null,"requires-python":null,"size":189868,"upload-time":"2017-04-10T17:20:16.876191Z","url":"https://files.pythonhosted.org/packages/72/9f/5ff6e45db392bc9dad642dffcca44eeee552289595c087a4f1d245fdb4f9/psutil-5.2.2-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"4c73be3d0927068fd96000b5591a3206eb03f1decaf76929c5e5cd06d004e214"},"data-dist-info-metadata":{"sha256":"4c73be3d0927068fd96000b5591a3206eb03f1decaf76929c5e5cd06d004e214"},"filename":"psutil-5.2.2-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"7a5c0973bd4c1de98d9b225bd4303a0718d31e31d6e2342e825c3e656f7056df"},"provenance":null,"requires-python":null,"size":193305,"upload-time":"2017-04-10T17:20:21.983057Z","url":"https://files.pythonhosted.org/packages/eb/c6/29c695be774c52cca9bb68b94ae4dc866a42ddf29dcd19b7ab4c0d97bdda/psutil-5.2.2-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.2.tar.gz","hashes":{"sha256":"44746540c0fab5b95401520d29eb9ffe84b3b4a235bd1d1971cbe36e1f38dd13"},"provenance":null,"requires-python":null,"size":348413,"upload-time":"2017-04-10T17:20:29.132011Z","url":"https://files.pythonhosted.org/packages/57/93/47a2e3befaf194ccc3d05ffbcba2cdcdd22a231100ef7e4cf63f085c900b/psutil-5.2.2.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.2.win-amd64-py2.7.exe","hashes":{"sha256":"70732850abd11f4d9fa46f0e110af21030e0a6088204f332d335921b36e66305"},"provenance":null,"requires-python":null,"size":427159,"upload-time":"2017-04-10T17:20:36.896101Z","url":"https://files.pythonhosted.org/packages/56/bb/d03fa2260839abbcdd4d323e83c2e91ffaedcb1975b88d5d0bc71c95c1fb/psutil-5.2.2.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.2.win-amd64-py3.3.exe","hashes":{"sha256":"5d2f076788d71d2e1c7276f1e5a1bc255f29c2e80eb8879a9ffc633c5bf69481"},"provenance":null,"requires-python":null,"size":425568,"upload-time":"2017-04-10T17:20:45.676662Z","url":"https://files.pythonhosted.org/packages/79/fd/3d2626e6a9fd4d99859fd8eac7d52ff9850d5e4ea62611a1e3ffe6f3d257/psutil-5.2.2.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.2.win-amd64-py3.4.exe","hashes":{"sha256":"fecda42b274dc618278bd9139e8493c9459d2174376f82b65ba929557f10e880"},"provenance":null,"requires-python":null,"size":425592,"upload-time":"2017-04-10T17:20:54.334109Z","url":"https://files.pythonhosted.org/packages/24/56/637ef0dfac83cd3e51096436faf7ea030f780aff3da98a79b7e7ac98a8bb/psutil-5.2.2.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.2.win-amd64-py3.5.exe","hashes":{"sha256":"92e3500dfaf7a5502ebaf4a7472e2afb9ff0cb36b4e5dc1977b3c774f58332db"},"provenance":null,"requires-python":null,"size":794103,"upload-time":"2017-04-10T17:21:08.107023Z","url":"https://files.pythonhosted.org/packages/09/4d/9cf34797696c0a75fd76606c362ddfbbc0f87d2c19c95ae26e61120241ad/psutil-5.2.2.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.2.win-amd64-py3.6.exe","hashes":{"sha256":"ed09521d49ee177f1205ed9791ad62263feacd2fe1cc20d1d33cf37923f240ea"},"provenance":null,"requires-python":null,"size":796151,"upload-time":"2017-04-10T17:21:22.493104Z","url":"https://files.pythonhosted.org/packages/04/74/5ea4412f31c9652335c03537ff4608ff6a12895fb3a34a187578d693b865/psutil-5.2.2.win-amd64-py3.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.2.win32-py2.7.exe","hashes":{"sha256":"147093b75b8874e55e6b26c540544d40e98845bc4ee74dc6054c881fd2a3eed9"},"provenance":null,"requires-python":null,"size":397069,"upload-time":"2017-04-10T17:21:30.834225Z","url":"https://files.pythonhosted.org/packages/b9/aa/b779310ee8a120b5bb90880d22e7b3869f98f7a30381a71188c6fb7ff4a6/psutil-5.2.2.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.2.win32-py3.3.exe","hashes":{"sha256":"3d3c5c117e55c486a53ef796cc715035bf4f56419cc32dbd124fe26e9289ad1e"},"provenance":null,"requires-python":null,"size":391941,"upload-time":"2017-04-10T17:21:38.680258Z","url":"https://files.pythonhosted.org/packages/e0/3f/fd40d4edbfac1d7996b6ecb3bfa5f923662e21686727eb070490ab6dcca8/psutil-5.2.2.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.2.win32-py3.4.exe","hashes":{"sha256":"0c74c6a494b650966b88da256cab4e507f483c53e85b9b10d3ff9c38f059330b"},"provenance":null,"requires-python":null,"size":391936,"upload-time":"2017-04-10T17:21:46.576288Z","url":"https://files.pythonhosted.org/packages/c7/59/f7aa53e3f72d6dcfce7c60d80e74853a1444a8a4d7fef788441435c645bf/psutil-5.2.2.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.2.win32-py3.5.exe","hashes":{"sha256":"b5583d1c2c858056d39bd148ed25839c4f1b76fec8fb2cb9b564c82997a21266"},"provenance":null,"requires-python":null,"size":661141,"upload-time":"2017-04-10T17:21:57.551743Z","url":"https://files.pythonhosted.org/packages/f2/62/0fc0c5459bcc04171ea3bf5603622db6815ddd92a472db06bcd69612177d/psutil-5.2.2.win32-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.2.2.win32-py3.6.exe","hashes":{"sha256":"1da0aa70d66612588d77daed7784e623aac1fd038681c3acd0e1c76b2b2f0819"},"provenance":null,"requires-python":null,"size":663188,"upload-time":"2017-04-10T17:22:09.360293Z","url":"https://files.pythonhosted.org/packages/a0/7f/494b600e45a8a25a76658a45747f232d38538d1c177f5b80123902d2b8da/psutil-5.2.2.win32-py3.6.exe","yanked":false},{"core-metadata":{"sha256":"d331b301ac1684996cf5629df9922a8709a7bf9243a73151f9aa72cf620ac815"},"data-dist-info-metadata":{"sha256":"d331b301ac1684996cf5629df9922a8709a7bf9243a73151f9aa72cf620ac815"},"filename":"psutil-5.3.0-cp27-none-win32.whl","hashes":{"sha256":"6f8f858cdb79397509ee067ae9d25bee8f4b4902453ac8d155fa1629f03aa39d"},"provenance":null,"requires-python":null,"size":210071,"upload-time":"2017-09-01T10:50:37.295614Z","url":"https://files.pythonhosted.org/packages/db/36/71afb537f3718a00a9e63b75e8534bf14fee38b67fb7cb72eb60a3378162/psutil-5.3.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"d331b301ac1684996cf5629df9922a8709a7bf9243a73151f9aa72cf620ac815"},"data-dist-info-metadata":{"sha256":"d331b301ac1684996cf5629df9922a8709a7bf9243a73151f9aa72cf620ac815"},"filename":"psutil-5.3.0-cp27-none-win_amd64.whl","hashes":{"sha256":"b31d6d19e445b56559abaa21703a6bc4b162aaf9ab99867b6f2bbbdb2c7fce66"},"provenance":null,"requires-python":null,"size":212901,"upload-time":"2017-09-01T10:50:41.892260Z","url":"https://files.pythonhosted.org/packages/c8/2b/9bc89bb0d8a2ac49ab29954017e65a9c28b83b13453418c8166938281458/psutil-5.3.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"47a95c843a21d16ad147a795cf6dc0395740dd91d643d06e100a63f2bb9f4d08"},"data-dist-info-metadata":{"sha256":"47a95c843a21d16ad147a795cf6dc0395740dd91d643d06e100a63f2bb9f4d08"},"filename":"psutil-5.3.0-cp33-cp33m-win32.whl","hashes":{"sha256":"7f1ba5011095e39b3f543e9c87008409dd8a57a3e48ea1022c348244b5af77bf"},"provenance":null,"requires-python":null,"size":209934,"upload-time":"2017-09-01T10:50:46.901965Z","url":"https://files.pythonhosted.org/packages/3c/5b/b020c5f5b6fbe69bfe77072a6d35a6c6d68f7ec7fe8b148223d9365bf8b4/psutil-5.3.0-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"47a95c843a21d16ad147a795cf6dc0395740dd91d643d06e100a63f2bb9f4d08"},"data-dist-info-metadata":{"sha256":"47a95c843a21d16ad147a795cf6dc0395740dd91d643d06e100a63f2bb9f4d08"},"filename":"psutil-5.3.0-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"853f68a85cec0137acf0504d8ca6d40d899e48ecbe931130f593a072a35b812e"},"provenance":null,"requires-python":null,"size":212722,"upload-time":"2017-09-01T10:50:51.125180Z","url":"https://files.pythonhosted.org/packages/ad/f8/4d7f713241f1786097faf48afb51e6cb9f7966d2fc36656098e182056704/psutil-5.3.0-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"70af37a6bd4674f8be60cec24e58ebdb4493432961ecc67d471a1739f1bf5d12"},"data-dist-info-metadata":{"sha256":"70af37a6bd4674f8be60cec24e58ebdb4493432961ecc67d471a1739f1bf5d12"},"filename":"psutil-5.3.0-cp34-cp34m-win32.whl","hashes":{"sha256":"01d9cb9473eee0e7e88319f9a5205a69e6e160b3ab2bd430a05b93bfae1528c2"},"provenance":null,"requires-python":null,"size":209879,"upload-time":"2017-09-01T10:50:55.651560Z","url":"https://files.pythonhosted.org/packages/d6/77/01a752e6d05061decf570acc800cd490766ea4534eccbe8c523b84fe5cc1/psutil-5.3.0-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"70af37a6bd4674f8be60cec24e58ebdb4493432961ecc67d471a1739f1bf5d12"},"data-dist-info-metadata":{"sha256":"70af37a6bd4674f8be60cec24e58ebdb4493432961ecc67d471a1739f1bf5d12"},"filename":"psutil-5.3.0-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"91d37262095c1a0f97a78f5034e10e0108e3fa326c85baa17f8cdd63fa5f81b9"},"provenance":null,"requires-python":null,"size":212614,"upload-time":"2017-09-01T10:50:59.871475Z","url":"https://files.pythonhosted.org/packages/0f/cf/fd1d752d428c5845fed4904e7bcdbb89ea3327aa063247fddcdca319c615/psutil-5.3.0-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"70af37a6bd4674f8be60cec24e58ebdb4493432961ecc67d471a1739f1bf5d12"},"data-dist-info-metadata":{"sha256":"70af37a6bd4674f8be60cec24e58ebdb4493432961ecc67d471a1739f1bf5d12"},"filename":"psutil-5.3.0-cp35-cp35m-win32.whl","hashes":{"sha256":"bd1776dc14b197388d728db72c103c0ebec834690ef1ce138035abf0123e2268"},"provenance":null,"requires-python":null,"size":212062,"upload-time":"2017-09-01T10:51:04.687879Z","url":"https://files.pythonhosted.org/packages/68/05/6d097706fd9cb43eda36a02a5feaee085aeac21f5bc6ea0b557109bc2eca/psutil-5.3.0-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"70af37a6bd4674f8be60cec24e58ebdb4493432961ecc67d471a1739f1bf5d12"},"data-dist-info-metadata":{"sha256":"70af37a6bd4674f8be60cec24e58ebdb4493432961ecc67d471a1739f1bf5d12"},"filename":"psutil-5.3.0-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"7fadb1b1357ef58821b3f1fc2afb6e1601609b0daa3b55c2fabf765e0ea98901"},"provenance":null,"requires-python":null,"size":215734,"upload-time":"2017-09-01T10:51:09.110127Z","url":"https://files.pythonhosted.org/packages/6e/5a/7c688b472ff5b6cb5413acfa5a178b5e8140ffbdf0011b6d0469e97af3b1/psutil-5.3.0-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"70af37a6bd4674f8be60cec24e58ebdb4493432961ecc67d471a1739f1bf5d12"},"data-dist-info-metadata":{"sha256":"70af37a6bd4674f8be60cec24e58ebdb4493432961ecc67d471a1739f1bf5d12"},"filename":"psutil-5.3.0-cp36-cp36m-win32.whl","hashes":{"sha256":"d5f4634a19e7d4692f37d8d67f8418f85f2bc1e2129914ec0e4208bf7838bf63"},"provenance":null,"requires-python":null,"size":212060,"upload-time":"2017-09-01T10:51:14.595884Z","url":"https://files.pythonhosted.org/packages/28/30/4ab277d7e37cd5ee1c47a89d21465c3eec3435b973ca86cd986efdd0aeac/psutil-5.3.0-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"70af37a6bd4674f8be60cec24e58ebdb4493432961ecc67d471a1739f1bf5d12"},"data-dist-info-metadata":{"sha256":"70af37a6bd4674f8be60cec24e58ebdb4493432961ecc67d471a1739f1bf5d12"},"filename":"psutil-5.3.0-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"31505ee459913ef63fa4c1c0d9a11a4da60b5c5ec6a92d6d7f5d12b9653fc61b"},"provenance":null,"requires-python":null,"size":215731,"upload-time":"2017-09-01T10:51:18.736362Z","url":"https://files.pythonhosted.org/packages/c3/e6/98fd6259d8ed834d9a81567d4f41dd3645e12a9aea9d38563efaf245610a/psutil-5.3.0-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.3.0.tar.gz","hashes":{"sha256":"a3940e06e92c84ab6e82b95dad056241beea93c3c9b1d07ddf96485079855185"},"provenance":null,"requires-python":null,"size":397265,"upload-time":"2017-09-01T12:31:14.428985Z","url":"https://files.pythonhosted.org/packages/1c/da/555e3ad3cad30f30bcf0d539cdeae5c8e7ef9e2a6078af645c70aa81e418/psutil-5.3.0.tar.gz","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.3.0.win-amd64-py2.7.exe","hashes":{"sha256":"ba94f021942d6cc27e18dcdccd2c1a0976f0596765ef412316ecb887d4fd3db2"},"provenance":null,"requires-python":null,"size":450426,"upload-time":"2017-09-01T10:51:26.068723Z","url":"https://files.pythonhosted.org/packages/e3/d6/238a22e898d0a3703d5fd486487108bf57d8fab1137bf085d3602d04894a/psutil-5.3.0.win-amd64-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.3.0.win-amd64-py3.3.exe","hashes":{"sha256":"0f2fccf98bc25e8d6d61e24b2cc6350b8dfe8fa7f5251c817e977d8c61146e5d"},"provenance":null,"requires-python":null,"size":448781,"upload-time":"2017-09-01T10:51:33.193333Z","url":"https://files.pythonhosted.org/packages/91/d6/4ccff87a9f93f4837c97c3eb105c4cd1ccd544091658ba1938feca366b59/psutil-5.3.0.win-amd64-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.3.0.win-amd64-py3.4.exe","hashes":{"sha256":"d06f02c53260d16fb445e426410263b2d271cea19136b1bb715cf10b76960359"},"provenance":null,"requires-python":null,"size":448549,"upload-time":"2017-09-01T10:51:40.339200Z","url":"https://files.pythonhosted.org/packages/74/29/91f18e7798a150c922c7da5c2153edecd7af0292332ef4f46e8ebd7184d3/psutil-5.3.0.win-amd64-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.3.0.win-amd64-py3.5.exe","hashes":{"sha256":"724439fb20d083c943a2c62db1aa240fa15fe23644c4d4a1e9f573ffaf0bbddd"},"provenance":null,"requires-python":null,"size":817222,"upload-time":"2017-09-01T10:51:52.173626Z","url":"https://files.pythonhosted.org/packages/41/fe/5081186ce35c0def7db2f8531ccac908b83edf735c2b8b78241633853b34/psutil-5.3.0.win-amd64-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.3.0.win-amd64-py3.6.exe","hashes":{"sha256":"a58708f3f6f74897450babb012cd8067f8911e7c8a1f2991643ec9937a8f6c15"},"provenance":null,"requires-python":null,"size":817218,"upload-time":"2017-09-01T10:52:03.695382Z","url":"https://files.pythonhosted.org/packages/aa/49/2076044f5f8554232eb5f8fb69b4a59c8f96da6d107d6f6a35aea2fb0344/psutil-5.3.0.win-amd64-py3.6.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.3.0.win32-py2.7.exe","hashes":{"sha256":"108dae5ecb68f6e6212bf0553be055a2a0eec210227d8e14c3a26368b118624a"},"provenance":null,"requires-python":null,"size":419952,"upload-time":"2017-09-01T10:52:12.411324Z","url":"https://files.pythonhosted.org/packages/d1/18/7e1a418aff3f65c3072eb765b0e51445df6309be3da25c5a4cb9f1d1d18b/psutil-5.3.0.win32-py2.7.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.3.0.win32-py3.3.exe","hashes":{"sha256":"9832124af1e9ec0f298f17ab11c3bb91164f8068ec9429c39a7f7a0eae637a94"},"provenance":null,"requires-python":null,"size":414764,"upload-time":"2017-09-01T10:52:32.286415Z","url":"https://files.pythonhosted.org/packages/de/46/0ec33721564e235fab9112fecb836eb084d0475ab97d5a6d5c462656e715/psutil-5.3.0.win32-py3.3.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.3.0.win32-py3.4.exe","hashes":{"sha256":"7b8d10e7d72862d1e97caba546b60ce263b3fcecd6176e4c94efebef87ee68d3"},"provenance":null,"requires-python":null,"size":414582,"upload-time":"2017-09-01T10:52:41.243220Z","url":"https://files.pythonhosted.org/packages/86/82/254439b29eea5670633a947ec3f67c5ec33b88322a1a443f121d21dc2714/psutil-5.3.0.win32-py3.4.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.3.0.win32-py3.5.exe","hashes":{"sha256":"ed1f7cbbbf778a6ed98e25d48fdbdc098e66b360427661712610d72c1b4cf5f5"},"provenance":null,"requires-python":null,"size":684021,"upload-time":"2017-09-01T10:52:51.603711Z","url":"https://files.pythonhosted.org/packages/20/9c/2d84e2926e1a89c4d1ea8fc315e05145e6b10af79459727ebb688b22dab8/psutil-5.3.0.win32-py3.5.exe","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.3.0.win32-py3.6.exe","hashes":{"sha256":"3d8d62f3da0b38dbfaf4756a32e18c866530b9066c298da3fc293cfefae22f0a"},"provenance":null,"requires-python":null,"size":684018,"upload-time":"2017-09-01T10:53:02.878374Z","url":"https://files.pythonhosted.org/packages/61/c7/bbcc29ba03d59f1add008245edf0d56d45434b137dde0c0d6b8441ae3be6/psutil-5.3.0.win32-py3.6.exe","yanked":false},{"core-metadata":{"sha256":"efaa364a2342964849b527ba97bfad9c165b3524ee431b8e5ff9d3b6bd3b4047"},"data-dist-info-metadata":{"sha256":"efaa364a2342964849b527ba97bfad9c165b3524ee431b8e5ff9d3b6bd3b4047"},"filename":"psutil-5.3.1-cp27-none-win32.whl","hashes":{"sha256":"7a669b1897b8cdce1cea79defdf3a10fd6e4f0a8e42ac2a971dfe74bc1ce5679"},"provenance":null,"requires-python":null,"size":210168,"upload-time":"2017-09-10T05:26:42.273181Z","url":"https://files.pythonhosted.org/packages/34/90/145ff234428b4bd519c20d460d6d51db7820e8120879ca9cdc88602c57f4/psutil-5.3.1-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"efaa364a2342964849b527ba97bfad9c165b3524ee431b8e5ff9d3b6bd3b4047"},"data-dist-info-metadata":{"sha256":"efaa364a2342964849b527ba97bfad9c165b3524ee431b8e5ff9d3b6bd3b4047"},"filename":"psutil-5.3.1-cp27-none-win_amd64.whl","hashes":{"sha256":"57be53c045f2085e28d5371eedfce804f5e49e7b35fa79bcf63e271046058002"},"provenance":null,"requires-python":null,"size":212998,"upload-time":"2017-09-10T05:27:13.646528Z","url":"https://files.pythonhosted.org/packages/c8/c9/d8cbfc3844e1a3e8b648fcca317ad8589283a7cbbc232c2c5d29cae88352/psutil-5.3.1-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"411e80cfca140c76586e1f0d89c1c6d7b1ff79f31751e6780aee2632e60c3996"},"data-dist-info-metadata":{"sha256":"411e80cfca140c76586e1f0d89c1c6d7b1ff79f31751e6780aee2632e60c3996"},"filename":"psutil-5.3.1-cp33-cp33m-win32.whl","hashes":{"sha256":"27d4c5ff3ab97389a9372d246e1aa27e5f02e4709fede48a0599f89d2873ca88"},"provenance":null,"requires-python":null,"size":210025,"upload-time":"2017-09-10T05:27:19.363106Z","url":"https://files.pythonhosted.org/packages/c4/82/7e412884fcf9ae538b1d96e31688b804377803e34f4e3e86bba869eaa9f7/psutil-5.3.1-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"411e80cfca140c76586e1f0d89c1c6d7b1ff79f31751e6780aee2632e60c3996"},"data-dist-info-metadata":{"sha256":"411e80cfca140c76586e1f0d89c1c6d7b1ff79f31751e6780aee2632e60c3996"},"filename":"psutil-5.3.1-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"72ba7e4c82879b3781ccced1eeb901f07725a36fab66270e7555e484a460760d"},"provenance":null,"requires-python":null,"size":212814,"upload-time":"2017-09-10T05:27:24.067035Z","url":"https://files.pythonhosted.org/packages/8a/1b/e185f211c9a959739c6789872458b78f6424466819afc5c420af09b222af/psutil-5.3.1-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"c340877ca68a4b72ce8f93ad0b6c48c15a2eeb72742c08998e478251efec5a92"},"data-dist-info-metadata":{"sha256":"c340877ca68a4b72ce8f93ad0b6c48c15a2eeb72742c08998e478251efec5a92"},"filename":"psutil-5.3.1-cp34-cp34m-win32.whl","hashes":{"sha256":"fc11c3a52990ec44064cbe026338dedcfff0e0027ca7516416eaa7d4f206c5af"},"provenance":null,"requires-python":null,"size":209976,"upload-time":"2017-09-10T05:27:29.519738Z","url":"https://files.pythonhosted.org/packages/ae/a9/ba609de04d2350878c6c3d641997dd37fa362775bf79aca3e6d542aae89e/psutil-5.3.1-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"c340877ca68a4b72ce8f93ad0b6c48c15a2eeb72742c08998e478251efec5a92"},"data-dist-info-metadata":{"sha256":"c340877ca68a4b72ce8f93ad0b6c48c15a2eeb72742c08998e478251efec5a92"},"filename":"psutil-5.3.1-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"7b1f9856c2fc9503a8a687db85e4f419ad1a10bfcab92ba786a7d43a6aa8cea0"},"provenance":null,"requires-python":null,"size":212710,"upload-time":"2017-09-10T05:27:34.832459Z","url":"https://files.pythonhosted.org/packages/9e/4e/e35f4e9b3f5dfb8eb88be75ccbc6e6a6428443afa4d641ff5e9e29a8991f/psutil-5.3.1-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"c340877ca68a4b72ce8f93ad0b6c48c15a2eeb72742c08998e478251efec5a92"},"data-dist-info-metadata":{"sha256":"c340877ca68a4b72ce8f93ad0b6c48c15a2eeb72742c08998e478251efec5a92"},"filename":"psutil-5.3.1-cp35-cp35m-win32.whl","hashes":{"sha256":"54781e463d9b9aa8c143033ee0d6a3149f9f143e6cc63099a95d4078f433dd56"},"provenance":null,"requires-python":null,"size":212159,"upload-time":"2017-09-10T05:27:39.270103Z","url":"https://files.pythonhosted.org/packages/79/4b/4531d21a7e428f3f25dc1b05be7e2024d9c2d45845ba005193dd9420e6b7/psutil-5.3.1-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"c340877ca68a4b72ce8f93ad0b6c48c15a2eeb72742c08998e478251efec5a92"},"data-dist-info-metadata":{"sha256":"c340877ca68a4b72ce8f93ad0b6c48c15a2eeb72742c08998e478251efec5a92"},"filename":"psutil-5.3.1-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"e9ef8d265298268cad784dfece103ab06bd726512d57fc6ed9f94b55452e4571"},"provenance":null,"requires-python":null,"size":215831,"upload-time":"2017-09-10T05:27:43.869238Z","url":"https://files.pythonhosted.org/packages/aa/a7/1bc0baaea0798c1b29bfb6cec05f18f6a4cbaaf96646818429d998feb2f5/psutil-5.3.1-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"c340877ca68a4b72ce8f93ad0b6c48c15a2eeb72742c08998e478251efec5a92"},"data-dist-info-metadata":{"sha256":"c340877ca68a4b72ce8f93ad0b6c48c15a2eeb72742c08998e478251efec5a92"},"filename":"psutil-5.3.1-cp36-cp36m-win32.whl","hashes":{"sha256":"f5d55618cd5b9270355fb52c0430ff30c4c84c5caf5b1254eec27f80d48e7a12"},"provenance":null,"requires-python":null,"size":212160,"upload-time":"2017-09-10T05:27:51.405015Z","url":"https://files.pythonhosted.org/packages/ba/03/1946dc720fec2083148b1d7b579e789bde85cebad67b22e05d5189871114/psutil-5.3.1-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"c340877ca68a4b72ce8f93ad0b6c48c15a2eeb72742c08998e478251efec5a92"},"data-dist-info-metadata":{"sha256":"c340877ca68a4b72ce8f93ad0b6c48c15a2eeb72742c08998e478251efec5a92"},"filename":"psutil-5.3.1-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"773ba33fe365cb8b0998eedcbe494dc92ce7428998f07dca652a1360a9e2bce8"},"provenance":null,"requires-python":null,"size":215831,"upload-time":"2017-09-10T05:27:56.438641Z","url":"https://files.pythonhosted.org/packages/9b/de/5bd7038f8bb68516eedafc8bba9daf6740011db8afc1bb25cdcc8d654771/psutil-5.3.1-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.3.1.tar.gz","hashes":{"sha256":"12dd9c8abbad15f055e9579130035b38617020ce176f4a498b7870e6321ffa67"},"provenance":null,"requires-python":null,"size":397075,"upload-time":"2017-09-10T05:28:34.436466Z","url":"https://files.pythonhosted.org/packages/d3/0a/74dcbb162554909b208e5dbe9f4e7278d78cc27470993e05177005e627d0/psutil-5.3.1.tar.gz","yanked":false},{"core-metadata":{"sha256":"dc4ac37cc269cb56f6fd4292649312b61d28d4970d9dce30eebae6899076846a"},"data-dist-info-metadata":{"sha256":"dc4ac37cc269cb56f6fd4292649312b61d28d4970d9dce30eebae6899076846a"},"filename":"psutil-5.4.0-cp27-none-win32.whl","hashes":{"sha256":"8121039d2280275ac82f99a0a48110450cbf5b356a11c842c8f5cdeafdf105e1"},"provenance":null,"requires-python":null,"size":218177,"upload-time":"2017-10-12T07:26:59.429237Z","url":"https://files.pythonhosted.org/packages/4c/89/08a536124b4ee1fd850982f59ab7268a359e4160a6bcc1d473f6971fbdd4/psutil-5.4.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"dc4ac37cc269cb56f6fd4292649312b61d28d4970d9dce30eebae6899076846a"},"data-dist-info-metadata":{"sha256":"dc4ac37cc269cb56f6fd4292649312b61d28d4970d9dce30eebae6899076846a"},"filename":"psutil-5.4.0-cp27-none-win_amd64.whl","hashes":{"sha256":"fcd93acb2602d01b86e0cfa4c2db689a81badae98d9c572348c94f1b2ea4b30d"},"provenance":null,"requires-python":null,"size":220988,"upload-time":"2017-10-12T07:27:04.404602Z","url":"https://files.pythonhosted.org/packages/c9/3d/0cd95044d1245166ddf24144e1b3e7a6b6a81933de3ff48c1851664109fc/psutil-5.4.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"b165c036e28aeff5aef41800231918509ab9366f5817be37e7123de4bd7ce3a2"},"data-dist-info-metadata":{"sha256":"b165c036e28aeff5aef41800231918509ab9366f5817be37e7123de4bd7ce3a2"},"filename":"psutil-5.4.0-cp33-cp33m-win32.whl","hashes":{"sha256":"60a58bfdda1fc6e86ecf95c6eef71252d9049694df9aa0a16c2841a425fc9deb"},"provenance":null,"requires-python":null,"size":218040,"upload-time":"2017-10-12T07:27:13.087638Z","url":"https://files.pythonhosted.org/packages/e5/1a/0f27898ad585d924e768b0c7029c7d7ac429994a3a032419c51f1c1f3e41/psutil-5.4.0-cp33-cp33m-win32.whl","yanked":false},{"core-metadata":{"sha256":"b165c036e28aeff5aef41800231918509ab9366f5817be37e7123de4bd7ce3a2"},"data-dist-info-metadata":{"sha256":"b165c036e28aeff5aef41800231918509ab9366f5817be37e7123de4bd7ce3a2"},"filename":"psutil-5.4.0-cp33-cp33m-win_amd64.whl","hashes":{"sha256":"9956b370243005d5561a94efa44b0cddb826d1f14d21958003925b008d3b9eb1"},"provenance":null,"requires-python":null,"size":220796,"upload-time":"2017-10-12T07:27:19.356245Z","url":"https://files.pythonhosted.org/packages/19/8e/b42236e04fbd03ffa2a08a393c8800fd7d51b11823c0e18723f0780d9b6a/psutil-5.4.0-cp33-cp33m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"eed08ae034f087bf660376ef37f75441a8f3d124c69a4be6089c1e755219a000"},"data-dist-info-metadata":{"sha256":"eed08ae034f087bf660376ef37f75441a8f3d124c69a4be6089c1e755219a000"},"filename":"psutil-5.4.0-cp34-cp34m-win32.whl","hashes":{"sha256":"104fec73d9ed573351f3efbf1b7ee19eb3b4097e2b3d9ff26b1ac5bca52b6f9e"},"provenance":null,"requires-python":null,"size":217969,"upload-time":"2017-10-12T07:27:24.486998Z","url":"https://files.pythonhosted.org/packages/11/42/0711b59b7f2f2f7de7912d30bc599950011d25401bda8a3330f878ff1e56/psutil-5.4.0-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"eed08ae034f087bf660376ef37f75441a8f3d124c69a4be6089c1e755219a000"},"data-dist-info-metadata":{"sha256":"eed08ae034f087bf660376ef37f75441a8f3d124c69a4be6089c1e755219a000"},"filename":"psutil-5.4.0-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"64b2814b30452d854d5f7a7c9c0d77423388b44eb2a8bcab3b84feeceaba8ffb"},"provenance":null,"requires-python":null,"size":220696,"upload-time":"2017-10-12T07:27:29.623184Z","url":"https://files.pythonhosted.org/packages/e6/67/e19ccc78646810cbc432c429e2993315cf50a126379b83b0f8b3eb9172b9/psutil-5.4.0-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"eed08ae034f087bf660376ef37f75441a8f3d124c69a4be6089c1e755219a000"},"data-dist-info-metadata":{"sha256":"eed08ae034f087bf660376ef37f75441a8f3d124c69a4be6089c1e755219a000"},"filename":"psutil-5.4.0-cp35-cp35m-win32.whl","hashes":{"sha256":"1a89ba967d4b9a3d5f19ea2c63b09e5ffb3a81de3116ead7bfb67b9c308e8dba"},"provenance":null,"requires-python":null,"size":220150,"upload-time":"2017-10-12T07:27:34.351564Z","url":"https://files.pythonhosted.org/packages/cc/c1/47edb3fccbb1354499297702ca55aec41ff7aab67b4df69aba9db4e52a7c/psutil-5.4.0-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"eed08ae034f087bf660376ef37f75441a8f3d124c69a4be6089c1e755219a000"},"data-dist-info-metadata":{"sha256":"eed08ae034f087bf660376ef37f75441a8f3d124c69a4be6089c1e755219a000"},"filename":"psutil-5.4.0-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"c8362902d4d94640c61960c18d8aa1af074822a549c3d4f137be3aa62c17f4b9"},"provenance":null,"requires-python":null,"size":223827,"upload-time":"2017-10-12T07:27:46.430105Z","url":"https://files.pythonhosted.org/packages/a9/e4/d399d060bbf40c1480ce2bfbc429edbd42769b048c4c7fdc51a49d74c9cf/psutil-5.4.0-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"eed08ae034f087bf660376ef37f75441a8f3d124c69a4be6089c1e755219a000"},"data-dist-info-metadata":{"sha256":"eed08ae034f087bf660376ef37f75441a8f3d124c69a4be6089c1e755219a000"},"filename":"psutil-5.4.0-cp36-cp36m-win32.whl","hashes":{"sha256":"04ed7548dfe61ab2561a94ac848a5b79239bb23e9015596ffd0644efd22461ba"},"provenance":null,"requires-python":null,"size":220151,"upload-time":"2017-10-12T07:27:51.153048Z","url":"https://files.pythonhosted.org/packages/2b/46/88fdd6206f01b04547bb89a6d2ff7eec380e5c59bda56ee03a0759ed397c/psutil-5.4.0-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"eed08ae034f087bf660376ef37f75441a8f3d124c69a4be6089c1e755219a000"},"data-dist-info-metadata":{"sha256":"eed08ae034f087bf660376ef37f75441a8f3d124c69a4be6089c1e755219a000"},"filename":"psutil-5.4.0-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"346555603ea903a5524bb37a49b0e0f90960c6c9973ebf794ae0802a4aa875eb"},"provenance":null,"requires-python":null,"size":223825,"upload-time":"2017-10-12T07:27:57.513929Z","url":"https://files.pythonhosted.org/packages/34/e2/8421ca5b99209fa43e48d2f76a21d5d86d7346838053abf7a4d2aa37255e/psutil-5.4.0-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.4.0.tar.gz","hashes":{"sha256":"8e6397ec24a2ec09751447d9f169486b68b37ac7a8d794dca003ace4efaafc6a"},"provenance":null,"requires-python":null,"size":406945,"upload-time":"2017-10-12T07:22:51.321681Z","url":"https://files.pythonhosted.org/packages/8d/96/1fc6468be91521192861966c40bd73fdf8b065eae6d82dd0f870b9825a65/psutil-5.4.0.tar.gz","yanked":false},{"core-metadata":{"sha256":"e0b722d181bbe7aba5016d4d14c510f34b0ceccccc534c1324af4b5bb619046f"},"data-dist-info-metadata":{"sha256":"e0b722d181bbe7aba5016d4d14c510f34b0ceccccc534c1324af4b5bb619046f"},"filename":"psutil-5.4.1-cp27-none-win32.whl","hashes":{"sha256":"7ef26ebe728ac821de17df23820e6ffcfd37c409fc865380e4d5ae1388f274a1"},"provenance":null,"requires-python":null,"size":218627,"upload-time":"2017-11-08T13:50:32.260924Z","url":"https://files.pythonhosted.org/packages/dd/dd/1811a99faefd2b5947b4e68bd70767b525fdbad65481a4bd2ee7e6408749/psutil-5.4.1-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"e0b722d181bbe7aba5016d4d14c510f34b0ceccccc534c1324af4b5bb619046f"},"data-dist-info-metadata":{"sha256":"e0b722d181bbe7aba5016d4d14c510f34b0ceccccc534c1324af4b5bb619046f"},"filename":"psutil-5.4.1-cp27-none-win_amd64.whl","hashes":{"sha256":"692dc72817d157aae522231dd334ea2524c6b07d844db0e7a2d6897820083427"},"provenance":null,"requires-python":null,"size":221450,"upload-time":"2017-11-08T13:50:37.542045Z","url":"https://files.pythonhosted.org/packages/51/c1/eec6a42a9f5fcd564c3fe3c6435c2c00e4e951a37f3ea3d324b04503ca6f/psutil-5.4.1-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"bea16c7d13fd463723ac0b38f421db313ae19443becf2ea91b0a77f6f3d8d8f2"},"data-dist-info-metadata":{"sha256":"bea16c7d13fd463723ac0b38f421db313ae19443becf2ea91b0a77f6f3d8d8f2"},"filename":"psutil-5.4.1-cp34-cp34m-win32.whl","hashes":{"sha256":"92342777d46e4630cf17d437412dc7fce0a8561217e074d36a35eb911ffd570e"},"provenance":null,"requires-python":null,"size":218421,"upload-time":"2017-11-08T13:50:42.728570Z","url":"https://files.pythonhosted.org/packages/e1/bd/d34935cd39f893d6e9ed46df5245aa71b29d2408b98f23755f234d517f80/psutil-5.4.1-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"bea16c7d13fd463723ac0b38f421db313ae19443becf2ea91b0a77f6f3d8d8f2"},"data-dist-info-metadata":{"sha256":"bea16c7d13fd463723ac0b38f421db313ae19443becf2ea91b0a77f6f3d8d8f2"},"filename":"psutil-5.4.1-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"f8f2f47a987c32ed3ca2068f3dfa9060dc9ff6cbed023d627d3f27060f4e59c4"},"provenance":null,"requires-python":null,"size":221144,"upload-time":"2017-11-08T13:50:47.928399Z","url":"https://files.pythonhosted.org/packages/24/ae/dcb7394e75b23b71f46742ffdab21d864a7ae74124b7930e5ea4f47b9049/psutil-5.4.1-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"bea16c7d13fd463723ac0b38f421db313ae19443becf2ea91b0a77f6f3d8d8f2"},"data-dist-info-metadata":{"sha256":"bea16c7d13fd463723ac0b38f421db313ae19443becf2ea91b0a77f6f3d8d8f2"},"filename":"psutil-5.4.1-cp35-cp35m-win32.whl","hashes":{"sha256":"1fce45549618d1930afefe322834ba91758331725bfdaec73ba6abcc83f6dc11"},"provenance":null,"requires-python":null,"size":220611,"upload-time":"2017-11-08T13:50:52.754745Z","url":"https://files.pythonhosted.org/packages/e8/58/c60fbf66c58d1e4b18902d601a294cb6ee993f3d051b44fdf397b6166852/psutil-5.4.1-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"bea16c7d13fd463723ac0b38f421db313ae19443becf2ea91b0a77f6f3d8d8f2"},"data-dist-info-metadata":{"sha256":"bea16c7d13fd463723ac0b38f421db313ae19443becf2ea91b0a77f6f3d8d8f2"},"filename":"psutil-5.4.1-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"f8a88553b2b5916f3bd814a91942215822a1dabae6db033cbb019095d6a24bc2"},"provenance":null,"requires-python":null,"size":224288,"upload-time":"2017-11-08T13:50:58.405900Z","url":"https://files.pythonhosted.org/packages/4f/f8/0e4e80114bc58267199c932f9227d09a00dea952b37400f76aa2a3bb9492/psutil-5.4.1-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"bea16c7d13fd463723ac0b38f421db313ae19443becf2ea91b0a77f6f3d8d8f2"},"data-dist-info-metadata":{"sha256":"bea16c7d13fd463723ac0b38f421db313ae19443becf2ea91b0a77f6f3d8d8f2"},"filename":"psutil-5.4.1-cp36-cp36m-win32.whl","hashes":{"sha256":"4139f76baa59142b907dd581d7ff3506a5163cb8ef69e8e92060df330bbf5788"},"provenance":null,"requires-python":null,"size":220610,"upload-time":"2017-11-08T13:51:03.361290Z","url":"https://files.pythonhosted.org/packages/8e/16/02eb53ea087776d9f219973e7b52c7d729929a2727c15894842f9b3629e6/psutil-5.4.1-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"bea16c7d13fd463723ac0b38f421db313ae19443becf2ea91b0a77f6f3d8d8f2"},"data-dist-info-metadata":{"sha256":"bea16c7d13fd463723ac0b38f421db313ae19443becf2ea91b0a77f6f3d8d8f2"},"filename":"psutil-5.4.1-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"d61bc04401ce938576e4c6ec201e812ed4114bfb9712202b87003619116c90c6"},"provenance":null,"requires-python":null,"size":224287,"upload-time":"2017-11-08T13:51:07.991086Z","url":"https://files.pythonhosted.org/packages/4a/89/0610a20ab3d1546fbe288001fb79ec4818ef6de29f89259c39daea85984f/psutil-5.4.1-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.4.1.tar.gz","hashes":{"sha256":"42e2de159e3c987435cb3b47d6f37035db190a1499f3af714ba7af5c379b6ba2"},"provenance":null,"requires-python":null,"size":408489,"upload-time":"2017-11-08T13:51:15.716367Z","url":"https://files.pythonhosted.org/packages/fe/17/0f0bf5792b2dfe6003efc5175c76225f7d3426f88e2bf8d360cfab870cd8/psutil-5.4.1.tar.gz","yanked":false},{"core-metadata":{"sha256":"3e4ef4cfed824c06d6acaff8f28af76203e0c1238133c71c6de75c654e9c89b1"},"data-dist-info-metadata":{"sha256":"3e4ef4cfed824c06d6acaff8f28af76203e0c1238133c71c6de75c654e9c89b1"},"filename":"psutil-5.4.2-cp27-none-win32.whl","hashes":{"sha256":"2fbbc7dce43c5240b9dc6d56302d57412f1c5a0d665d1f04eb05a6b7279f4e9b"},"provenance":null,"requires-python":null,"size":221114,"upload-time":"2017-12-07T12:03:17.419322Z","url":"https://files.pythonhosted.org/packages/e6/e2/09d55a6e899cf1a7b6a22d0cc2a75d45553df5b63d7c9f2eb2553c7207bc/psutil-5.4.2-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"3e4ef4cfed824c06d6acaff8f28af76203e0c1238133c71c6de75c654e9c89b1"},"data-dist-info-metadata":{"sha256":"3e4ef4cfed824c06d6acaff8f28af76203e0c1238133c71c6de75c654e9c89b1"},"filename":"psutil-5.4.2-cp27-none-win_amd64.whl","hashes":{"sha256":"259ec8578d19643179eb2377348c63b650b51ba40f58f2620a3d9732b8a0b557"},"provenance":null,"requires-python":null,"size":223996,"upload-time":"2017-12-07T12:03:22.758210Z","url":"https://files.pythonhosted.org/packages/47/fc/e2199322f422e4bb6e25808a335132235fb0f3fabb0ea71ec3442719fdaf/psutil-5.4.2-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"aee7fd8368b69d2c7b666317dd0c8e73e7af18ef863dabadfb44e8f46ffef9d5"},"data-dist-info-metadata":{"sha256":"aee7fd8368b69d2c7b666317dd0c8e73e7af18ef863dabadfb44e8f46ffef9d5"},"filename":"psutil-5.4.2-cp34-cp34m-win32.whl","hashes":{"sha256":"d3808be8241433db17fa955566c3b8be61dac8ba8f221dcbb202a9daba918db5"},"provenance":null,"requires-python":null,"size":220935,"upload-time":"2017-12-07T12:03:27.058880Z","url":"https://files.pythonhosted.org/packages/d5/50/044ad4b47bf0e992a11ae7cb060e8c7dd52eb6983c2c4b6fdd5314fcc3b2/psutil-5.4.2-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"aee7fd8368b69d2c7b666317dd0c8e73e7af18ef863dabadfb44e8f46ffef9d5"},"data-dist-info-metadata":{"sha256":"aee7fd8368b69d2c7b666317dd0c8e73e7af18ef863dabadfb44e8f46ffef9d5"},"filename":"psutil-5.4.2-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"449747f638c221f8ce6ca3548aefef13339aa05b453cc1f233f4d6c31c206198"},"provenance":null,"requires-python":null,"size":223749,"upload-time":"2017-12-07T12:03:31.671187Z","url":"https://files.pythonhosted.org/packages/b5/77/6f8b4c7c6e9e8fa60b0e0627853ba7182f1a2459ad0b557fe3257bbe014b/psutil-5.4.2-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"aee7fd8368b69d2c7b666317dd0c8e73e7af18ef863dabadfb44e8f46ffef9d5"},"data-dist-info-metadata":{"sha256":"aee7fd8368b69d2c7b666317dd0c8e73e7af18ef863dabadfb44e8f46ffef9d5"},"filename":"psutil-5.4.2-cp35-cp35m-win32.whl","hashes":{"sha256":"f6c2d54abd59ed8691882de7fd6b248f5808a567885f20f50b3b4b9eedaebb1f"},"provenance":null,"requires-python":null,"size":223260,"upload-time":"2017-12-07T12:03:36.139696Z","url":"https://files.pythonhosted.org/packages/ad/fa/ac5b6bc2b25817509fc2e44f910f39b3385d1575262f7802b9c113ab786a/psutil-5.4.2-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"aee7fd8368b69d2c7b666317dd0c8e73e7af18ef863dabadfb44e8f46ffef9d5"},"data-dist-info-metadata":{"sha256":"aee7fd8368b69d2c7b666317dd0c8e73e7af18ef863dabadfb44e8f46ffef9d5"},"filename":"psutil-5.4.2-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"e3d00d8fc3d4217f05d07af45390f072c04cb7c7dddd70b86b728e5fbe485c81"},"provenance":null,"requires-python":null,"size":226950,"upload-time":"2017-12-07T12:03:40.889125Z","url":"https://files.pythonhosted.org/packages/0e/5c/67b33328d4f307608006c3630838b272719125b36e5456a4dd8f4e76eca9/psutil-5.4.2-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"aee7fd8368b69d2c7b666317dd0c8e73e7af18ef863dabadfb44e8f46ffef9d5"},"data-dist-info-metadata":{"sha256":"aee7fd8368b69d2c7b666317dd0c8e73e7af18ef863dabadfb44e8f46ffef9d5"},"filename":"psutil-5.4.2-cp36-cp36m-win32.whl","hashes":{"sha256":"3473d6abad9d6ec7b8a97f4dc55f0b3483ecf470d85f08f5e23c1c07592b914f"},"provenance":null,"requires-python":null,"size":223259,"upload-time":"2017-12-07T12:03:45.502272Z","url":"https://files.pythonhosted.org/packages/59/75/97862069d3fa20f1f2897fecd5b6822a3c06ee2af1161f1beb320cc4f5f8/psutil-5.4.2-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"aee7fd8368b69d2c7b666317dd0c8e73e7af18ef863dabadfb44e8f46ffef9d5"},"data-dist-info-metadata":{"sha256":"aee7fd8368b69d2c7b666317dd0c8e73e7af18ef863dabadfb44e8f46ffef9d5"},"filename":"psutil-5.4.2-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"7dc6c3bbb5d28487f791f195d6abfdef295d34c44ce6cb5f2d178613fb3338ab"},"provenance":null,"requires-python":null,"size":226952,"upload-time":"2017-12-07T12:03:51.257182Z","url":"https://files.pythonhosted.org/packages/8f/9a/5dea138e49addd68d8e65d08103dd28428f1ea0b8d4e8beef9b24f069a16/psutil-5.4.2-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.4.2.tar.gz","hashes":{"sha256":"00a1f9ff8d1e035fba7bfdd6977fa8ea7937afdb4477339e5df3dba78194fe11"},"provenance":null,"requires-python":null,"size":411888,"upload-time":"2017-12-07T12:03:58.583601Z","url":"https://files.pythonhosted.org/packages/54/24/aa854703715fa161110daa001afce75d21d1840e9ab5eb28708d6a5058b0/psutil-5.4.2.tar.gz","yanked":false},{"core-metadata":{"sha256":"9de2e14fb34c7d6a6d1b3692c6e7830d8a950ce4740bd70dbc4f2900ad780b67"},"data-dist-info-metadata":{"sha256":"9de2e14fb34c7d6a6d1b3692c6e7830d8a950ce4740bd70dbc4f2900ad780b67"},"filename":"psutil-5.4.3-cp27-none-win32.whl","hashes":{"sha256":"82a06785db8eeb637b349006cc28a92e40cd190fefae9875246d18d0de7ccac8"},"provenance":null,"requires-python":null,"size":220984,"upload-time":"2018-01-01T20:33:16.515515Z","url":"https://files.pythonhosted.org/packages/e5/cc/6dd427e738a8db6d0b66525856da43d2ef12c4c19269863927f7cf0e2aaf/psutil-5.4.3-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"9de2e14fb34c7d6a6d1b3692c6e7830d8a950ce4740bd70dbc4f2900ad780b67"},"data-dist-info-metadata":{"sha256":"9de2e14fb34c7d6a6d1b3692c6e7830d8a950ce4740bd70dbc4f2900ad780b67"},"filename":"psutil-5.4.3-cp27-none-win_amd64.whl","hashes":{"sha256":"4152ae231709e3e8b80e26b6da20dc965a1a589959c48af1ed024eca6473f60d"},"provenance":null,"requires-python":null,"size":223871,"upload-time":"2018-01-01T20:33:24.887595Z","url":"https://files.pythonhosted.org/packages/b9/e4/6867765edcab8d12a52c84c9b0af492ecb99f8cc565ad552341bcf73ebd9/psutil-5.4.3-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"7a6c999b9c5b0661b0db22bb84f2ef4f1cf396a3f1ed6311aad005f3bee574e8"},"data-dist-info-metadata":{"sha256":"7a6c999b9c5b0661b0db22bb84f2ef4f1cf396a3f1ed6311aad005f3bee574e8"},"filename":"psutil-5.4.3-cp34-cp34m-win32.whl","hashes":{"sha256":"230eeb3aeb077814f3a2cd036ddb6e0f571960d327298cc914c02385c3e02a63"},"provenance":null,"requires-python":null,"size":220823,"upload-time":"2018-01-01T20:33:31.019909Z","url":"https://files.pythonhosted.org/packages/5b/fc/745a864190a4221cdb984e666f4218e98d9a53a64b4dcf2eb7a71c1bf693/psutil-5.4.3-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"7a6c999b9c5b0661b0db22bb84f2ef4f1cf396a3f1ed6311aad005f3bee574e8"},"data-dist-info-metadata":{"sha256":"7a6c999b9c5b0661b0db22bb84f2ef4f1cf396a3f1ed6311aad005f3bee574e8"},"filename":"psutil-5.4.3-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"a3286556d4d2f341108db65d8e20d0cd3fcb9a91741cb5eb496832d7daf2a97c"},"provenance":null,"requires-python":null,"size":223605,"upload-time":"2018-01-01T20:33:38.392793Z","url":"https://files.pythonhosted.org/packages/b0/25/414738d5e8e75418e560a36651d1e1b09c9df05440a2a808d999a5548b1e/psutil-5.4.3-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"7a6c999b9c5b0661b0db22bb84f2ef4f1cf396a3f1ed6311aad005f3bee574e8"},"data-dist-info-metadata":{"sha256":"7a6c999b9c5b0661b0db22bb84f2ef4f1cf396a3f1ed6311aad005f3bee574e8"},"filename":"psutil-5.4.3-cp35-cp35m-win32.whl","hashes":{"sha256":"94d4e63189f2593960e73acaaf96be235dd8a455fe2bcb37d8ad6f0e87f61556"},"provenance":null,"requires-python":null,"size":223167,"upload-time":"2018-01-01T20:33:43.796884Z","url":"https://files.pythonhosted.org/packages/e9/80/8da216f42050220f37f7133d2accfcd001a1bd0f31d7cdb8660acb46b8fe/psutil-5.4.3-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"7a6c999b9c5b0661b0db22bb84f2ef4f1cf396a3f1ed6311aad005f3bee574e8"},"data-dist-info-metadata":{"sha256":"7a6c999b9c5b0661b0db22bb84f2ef4f1cf396a3f1ed6311aad005f3bee574e8"},"filename":"psutil-5.4.3-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"c91eee73eea00df5e62c741b380b7e5b6fdd553891bee5669817a3a38d036f13"},"provenance":null,"requires-python":null,"size":226810,"upload-time":"2018-01-01T20:33:50.884798Z","url":"https://files.pythonhosted.org/packages/23/34/b3de39502c2c34899f9e7ae3c8d1050c9317997ab1fe6c647e7a789571a8/psutil-5.4.3-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"7a6c999b9c5b0661b0db22bb84f2ef4f1cf396a3f1ed6311aad005f3bee574e8"},"data-dist-info-metadata":{"sha256":"7a6c999b9c5b0661b0db22bb84f2ef4f1cf396a3f1ed6311aad005f3bee574e8"},"filename":"psutil-5.4.3-cp36-cp36m-win32.whl","hashes":{"sha256":"779ec7e7621758ca11a8d99a1064996454b3570154277cc21342a01148a49c28"},"provenance":null,"requires-python":null,"size":223168,"upload-time":"2018-01-01T20:33:56.458784Z","url":"https://files.pythonhosted.org/packages/da/c1/caadba7c64f72118b02f019c60ad85a5668ddf0a32836230b71692b0cbfa/psutil-5.4.3-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"7a6c999b9c5b0661b0db22bb84f2ef4f1cf396a3f1ed6311aad005f3bee574e8"},"data-dist-info-metadata":{"sha256":"7a6c999b9c5b0661b0db22bb84f2ef4f1cf396a3f1ed6311aad005f3bee574e8"},"filename":"psutil-5.4.3-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"8a15d773203a1277e57b1d11a7ccdf70804744ef4a9518a87ab8436995c31a4b"},"provenance":null,"requires-python":null,"size":226804,"upload-time":"2018-01-01T20:34:03.130620Z","url":"https://files.pythonhosted.org/packages/71/80/90799d3dc6e33e650ee03f96fa18157faed885593eabea3a6560ebff7de0/psutil-5.4.3-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.4.3.tar.gz","hashes":{"sha256":"e2467e9312c2fa191687b89ff4bc2ad8843be4af6fb4dc95a7cc5f7d7a327b18"},"provenance":null,"requires-python":null,"size":412550,"upload-time":"2018-01-01T20:34:13.285899Z","url":"https://files.pythonhosted.org/packages/e2/e1/600326635f97fee89bf8426fef14c5c29f4849c79f68fd79f433d8c1bd96/psutil-5.4.3.tar.gz","yanked":false},{"core-metadata":{"sha256":"485b43e92c234e987a6b5a9be22665d46ab26940b08a2edad8583aa4743c0439"},"data-dist-info-metadata":{"sha256":"485b43e92c234e987a6b5a9be22665d46ab26940b08a2edad8583aa4743c0439"},"filename":"psutil-5.4.4-cp27-none-win32.whl","hashes":{"sha256":"8f208867d41eb3b6de416df098a9a28d08d40b432467d821b8ef5bb589a394ce"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":216647,"upload-time":"2018-04-13T09:10:24.220147Z","url":"https://files.pythonhosted.org/packages/e8/cd/dbf537e32de1c9f06a0069bf0ef13c8707f653e9af2b0ea0ed7040b73083/psutil-5.4.4-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"485b43e92c234e987a6b5a9be22665d46ab26940b08a2edad8583aa4743c0439"},"data-dist-info-metadata":{"sha256":"485b43e92c234e987a6b5a9be22665d46ab26940b08a2edad8583aa4743c0439"},"filename":"psutil-5.4.4-cp27-none-win_amd64.whl","hashes":{"sha256":"77b5e310de17085346ef2c4c21b64d5e39616ab4559b8ef6fea9f6f2ab0de66f"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":219668,"upload-time":"2018-04-13T09:10:29.328807Z","url":"https://files.pythonhosted.org/packages/7f/37/538bb4275d8a26ec369944878a681000e827e72cab9b4f27f4b1b5932446/psutil-5.4.4-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"893713cd1f34926704c2e5f844b554a79ad42ce1c20046651dad28613eb2d0bc"},"data-dist-info-metadata":{"sha256":"893713cd1f34926704c2e5f844b554a79ad42ce1c20046651dad28613eb2d0bc"},"filename":"psutil-5.4.4-cp34-cp34m-win32.whl","hashes":{"sha256":"fec0e59dacbe91db7e063f038301f49da7e9361732fc31d28338ecaa4719520e"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":216642,"upload-time":"2018-04-13T09:10:33.842941Z","url":"https://files.pythonhosted.org/packages/92/e9/c9c4ec1a0ac55ee1514c1a249d017dd2f7a89e727d236ed6862b493de154/psutil-5.4.4-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"893713cd1f34926704c2e5f844b554a79ad42ce1c20046651dad28613eb2d0bc"},"data-dist-info-metadata":{"sha256":"893713cd1f34926704c2e5f844b554a79ad42ce1c20046651dad28613eb2d0bc"},"filename":"psutil-5.4.4-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"1268fb6959cd8d761c30e13e79908ae73ba5a69c3c3a5d09a7a27278446f9800"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":219513,"upload-time":"2018-04-13T09:10:38.041575Z","url":"https://files.pythonhosted.org/packages/83/49/c903f446d28bfb6e92fa08b710f68cd5d17cc2ccfc4a13fe607f8b20f6dd/psutil-5.4.4-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"893713cd1f34926704c2e5f844b554a79ad42ce1c20046651dad28613eb2d0bc"},"data-dist-info-metadata":{"sha256":"893713cd1f34926704c2e5f844b554a79ad42ce1c20046651dad28613eb2d0bc"},"filename":"psutil-5.4.4-cp35-cp35m-win32.whl","hashes":{"sha256":"7eb2d80ef79d90474a03eead13b32e541d1fdeb47468cf04c881f0a7392ddbc5"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":219220,"upload-time":"2018-04-13T09:10:42.685382Z","url":"https://files.pythonhosted.org/packages/bb/63/c9b3e9ff8d23409896a7e9e7356a730fdfb5a45a1edc0e6d4ca5ce655f29/psutil-5.4.4-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"893713cd1f34926704c2e5f844b554a79ad42ce1c20046651dad28613eb2d0bc"},"data-dist-info-metadata":{"sha256":"893713cd1f34926704c2e5f844b554a79ad42ce1c20046651dad28613eb2d0bc"},"filename":"psutil-5.4.4-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"69f1db4d13f362ce11a6246b20c752c31b87a6fd77452170fd03c26a8a20a4f2"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":222757,"upload-time":"2018-04-13T09:10:47.950981Z","url":"https://files.pythonhosted.org/packages/72/22/a5ce34af1285679e02d7fd701ff6389f579a17e623dd89236ea1873ce12b/psutil-5.4.4-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"893713cd1f34926704c2e5f844b554a79ad42ce1c20046651dad28613eb2d0bc"},"data-dist-info-metadata":{"sha256":"893713cd1f34926704c2e5f844b554a79ad42ce1c20046651dad28613eb2d0bc"},"filename":"psutil-5.4.4-cp36-cp36m-win32.whl","hashes":{"sha256":"6eb59bcfd48eade8889bae67a16e0d8c7b18af0732ba64dead61206fd7cb4e45"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":219224,"upload-time":"2018-04-13T09:10:52.291473Z","url":"https://files.pythonhosted.org/packages/3c/6d/5e9a3683d4532997525aa20d1d9ce0ca1201271d30aad9e5f18c34459478/psutil-5.4.4-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"893713cd1f34926704c2e5f844b554a79ad42ce1c20046651dad28613eb2d0bc"},"data-dist-info-metadata":{"sha256":"893713cd1f34926704c2e5f844b554a79ad42ce1c20046651dad28613eb2d0bc"},"filename":"psutil-5.4.4-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"a4af5d4fcf6022886a30fb3b4fff71ff25f645865a68506680d43a3e634764af"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":222759,"upload-time":"2018-04-13T09:10:56.584621Z","url":"https://files.pythonhosted.org/packages/b6/61/eeeab30fa737b8b95b790d3eb8f49ebedeb783e43aef2d8d851687592d6c/psutil-5.4.4-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.4.4.tar.gz","hashes":{"sha256":"5959e33e0fc69742dd22e88bfc7789a1f2e1fc2297794b543119e10cdac8dfb1"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":417890,"upload-time":"2018-04-13T09:11:03.199870Z","url":"https://files.pythonhosted.org/packages/35/35/7da482448cd9ee3555faa9c5e541e37b18a849fb961e55d6fda6ca936ddb/psutil-5.4.4.tar.gz","yanked":false},{"core-metadata":{"sha256":"c8af77aa4cab9fad5f3310c13c633fb6bb0f5a44421d0e8662a9f16b38ca9aec"},"data-dist-info-metadata":{"sha256":"c8af77aa4cab9fad5f3310c13c633fb6bb0f5a44421d0e8662a9f16b38ca9aec"},"filename":"psutil-5.4.5-cp27-none-win32.whl","hashes":{"sha256":"33384065f0014351fa70187548e3e95952c4df4bc5c38648bd0e647d21eaaf01"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":216630,"upload-time":"2018-04-13T17:59:43.195295Z","url":"https://files.pythonhosted.org/packages/63/9f/529a599db3057602114a30aa5e3641e78bce6c6e195adb75309c9286cb88/psutil-5.4.5-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"c8af77aa4cab9fad5f3310c13c633fb6bb0f5a44421d0e8662a9f16b38ca9aec"},"data-dist-info-metadata":{"sha256":"c8af77aa4cab9fad5f3310c13c633fb6bb0f5a44421d0e8662a9f16b38ca9aec"},"filename":"psutil-5.4.5-cp27-none-win_amd64.whl","hashes":{"sha256":"f24cd52bafa06917935fe1b68c5a45593abe1f3097dc35b2dfc4718236795890"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":219650,"upload-time":"2018-04-13T17:59:53.389911Z","url":"https://files.pythonhosted.org/packages/b6/ca/2d23b37e9b30908174d2cb596f60f06b3858856a2e595c931f7d4d640c03/psutil-5.4.5-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"4fb804f1abb5eb53785449da8e1201c9af34b09face243251213fa79878f9142"},"data-dist-info-metadata":{"sha256":"4fb804f1abb5eb53785449da8e1201c9af34b09face243251213fa79878f9142"},"filename":"psutil-5.4.5-cp34-cp34m-win32.whl","hashes":{"sha256":"99029b6af386b22882f0b6d537ffed5a9c3d5ff31782974aeaa1d683262d8543"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":216589,"upload-time":"2018-04-13T18:00:04.166075Z","url":"https://files.pythonhosted.org/packages/bf/bc/f687dfa4679aad782fb78c43fed2626cb0157567a5b06790997e5aa0f166/psutil-5.4.5-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"4fb804f1abb5eb53785449da8e1201c9af34b09face243251213fa79878f9142"},"data-dist-info-metadata":{"sha256":"4fb804f1abb5eb53785449da8e1201c9af34b09face243251213fa79878f9142"},"filename":"psutil-5.4.5-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"51e12aa74509832443862373a2655052b20c83cad7322f49d217452500b9a405"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":219462,"upload-time":"2018-04-13T18:00:14.637120Z","url":"https://files.pythonhosted.org/packages/32/b5/545953316dd9cb053e4c7d4f70d88aba5362dbbe58422ca6bbec1bbf8956/psutil-5.4.5-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"4fb804f1abb5eb53785449da8e1201c9af34b09face243251213fa79878f9142"},"data-dist-info-metadata":{"sha256":"4fb804f1abb5eb53785449da8e1201c9af34b09face243251213fa79878f9142"},"filename":"psutil-5.4.5-cp35-cp35m-win32.whl","hashes":{"sha256":"325c334596ad2d8a178d0e7b4eecc91748096a87489b3701ee16986173000aaa"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":219172,"upload-time":"2018-04-13T18:00:28.230677Z","url":"https://files.pythonhosted.org/packages/00/ed/fdf2930c41e76e3a8bc59bf998062ee5ad0c393170a7d2c273dd3b259794/psutil-5.4.5-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"4fb804f1abb5eb53785449da8e1201c9af34b09face243251213fa79878f9142"},"data-dist-info-metadata":{"sha256":"4fb804f1abb5eb53785449da8e1201c9af34b09face243251213fa79878f9142"},"filename":"psutil-5.4.5-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"52a91ba928a5e86e0249b4932d6e36972a72d1ad8dcc5b7f753a2ae14825a4ba"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":222706,"upload-time":"2018-04-13T18:00:38.180085Z","url":"https://files.pythonhosted.org/packages/c6/bf/09b13c17f54f0004ccb43cc1c2d36bab2eb75f471564b7856749dcaf62c3/psutil-5.4.5-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"4fb804f1abb5eb53785449da8e1201c9af34b09face243251213fa79878f9142"},"data-dist-info-metadata":{"sha256":"4fb804f1abb5eb53785449da8e1201c9af34b09face243251213fa79878f9142"},"filename":"psutil-5.4.5-cp36-cp36m-win32.whl","hashes":{"sha256":"b10703a109cc9225cd588c207f7f93480a420ade35c13515ea8f20063b42a392"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":219171,"upload-time":"2018-04-13T18:00:52.610707Z","url":"https://files.pythonhosted.org/packages/3c/ae/34952007b4d64f88a03510866b9cd90207e391f6b2b59b6301ad96fa0fb5/psutil-5.4.5-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"4fb804f1abb5eb53785449da8e1201c9af34b09face243251213fa79878f9142"},"data-dist-info-metadata":{"sha256":"4fb804f1abb5eb53785449da8e1201c9af34b09face243251213fa79878f9142"},"filename":"psutil-5.4.5-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"ddba952ed256151844d82fb13c8fb1019fe11ecaeacbd659d67ba5661ae73d0d"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":222710,"upload-time":"2018-04-13T18:01:02.457641Z","url":"https://files.pythonhosted.org/packages/4c/bb/303f15f4a47b96ff0ae5025d89b330e2be314085c418c0b726877476e937/psutil-5.4.5-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.4.5.tar.gz","hashes":{"sha256":"ebe293be36bb24b95cdefc5131635496e88b17fabbcf1e4bc9b5c01f5e489cfe"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":418003,"upload-time":"2018-04-13T18:01:19.381491Z","url":"https://files.pythonhosted.org/packages/14/a2/8ac7dda36eac03950ec2668ab1b466314403031c83a95c5efc81d2acf163/psutil-5.4.5.tar.gz","yanked":false},{"core-metadata":{"sha256":"aa807ba097a31687e416722680ba225449cb087946d4db7e62db8c1659a43a5d"},"data-dist-info-metadata":{"sha256":"aa807ba097a31687e416722680ba225449cb087946d4db7e62db8c1659a43a5d"},"filename":"psutil-5.4.6-cp27-none-win32.whl","hashes":{"sha256":"319e12f6bae4d4d988fbff3bed792953fa3b44c791f085b0a1a230f755671ef7"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":216468,"upload-time":"2018-06-07T15:39:23.967375Z","url":"https://files.pythonhosted.org/packages/b4/00/82c9fb4ffca22f2f6d0d883469584cd0cff71a604a19809015045b1fbab6/psutil-5.4.6-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"aa807ba097a31687e416722680ba225449cb087946d4db7e62db8c1659a43a5d"},"data-dist-info-metadata":{"sha256":"aa807ba097a31687e416722680ba225449cb087946d4db7e62db8c1659a43a5d"},"filename":"psutil-5.4.6-cp27-none-win_amd64.whl","hashes":{"sha256":"7789885a72aa3075d28d028236eb3f2b84d908f81d38ad41769a6ddc2fd81b7c"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":219532,"upload-time":"2018-06-07T15:39:26.297791Z","url":"https://files.pythonhosted.org/packages/c2/23/22df1d36dc8ae002e9f646f9ed06b4f6bfbc7a22b67804c3a497be21d002/psutil-5.4.6-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"b7bbbf54e7767a4dcfada805607a533828d7e74794bf64ea1ae76700fc24d495"},"data-dist-info-metadata":{"sha256":"b7bbbf54e7767a4dcfada805607a533828d7e74794bf64ea1ae76700fc24d495"},"filename":"psutil-5.4.6-cp34-cp34m-win32.whl","hashes":{"sha256":"0ff2b16e9045d01edb1dd10d7fbcc184012e37f6cd38029e959f2be9c6223f50"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":216464,"upload-time":"2018-06-07T15:39:28.563215Z","url":"https://files.pythonhosted.org/packages/f9/fa/966988e350306e1a1a9024e77ad5f118cbfe11318e8bdfc258c3d5a1c68b/psutil-5.4.6-cp34-cp34m-win32.whl","yanked":false},{"core-metadata":{"sha256":"b7bbbf54e7767a4dcfada805607a533828d7e74794bf64ea1ae76700fc24d495"},"data-dist-info-metadata":{"sha256":"b7bbbf54e7767a4dcfada805607a533828d7e74794bf64ea1ae76700fc24d495"},"filename":"psutil-5.4.6-cp34-cp34m-win_amd64.whl","hashes":{"sha256":"dc85fad15ef98103ecc047a0d81b55bbf5fe1b03313b96e883acc2e2fa87ed5c"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":219331,"upload-time":"2018-06-07T15:39:31.250304Z","url":"https://files.pythonhosted.org/packages/a5/0d/40b552c2c089523df1f7ab5a0249fbf90cb2e80e89177b0189e41e367adc/psutil-5.4.6-cp34-cp34m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"b7bbbf54e7767a4dcfada805607a533828d7e74794bf64ea1ae76700fc24d495"},"data-dist-info-metadata":{"sha256":"b7bbbf54e7767a4dcfada805607a533828d7e74794bf64ea1ae76700fc24d495"},"filename":"psutil-5.4.6-cp35-cp35m-win32.whl","hashes":{"sha256":"7f4616bcb44a6afda930cfc40215e5e9fa7c6896e683b287c771c937712fbe2f"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":219052,"upload-time":"2018-06-07T15:39:33.732713Z","url":"https://files.pythonhosted.org/packages/00/4d/194ec701de80c704f679bf78495c054994cc403884ffec816787813c4fde/psutil-5.4.6-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"b7bbbf54e7767a4dcfada805607a533828d7e74794bf64ea1ae76700fc24d495"},"data-dist-info-metadata":{"sha256":"b7bbbf54e7767a4dcfada805607a533828d7e74794bf64ea1ae76700fc24d495"},"filename":"psutil-5.4.6-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"529ae235896efb99a6f77653a7138273ab701ec9f0343a1f5030945108dee3c4"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":222533,"upload-time":"2018-06-07T15:39:36.224840Z","url":"https://files.pythonhosted.org/packages/e3/21/f82f270326e098f211bcc36cbb2ae7100732dcad03bd324e6af8c9d7e407/psutil-5.4.6-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"b7bbbf54e7767a4dcfada805607a533828d7e74794bf64ea1ae76700fc24d495"},"data-dist-info-metadata":{"sha256":"b7bbbf54e7767a4dcfada805607a533828d7e74794bf64ea1ae76700fc24d495"},"filename":"psutil-5.4.6-cp36-cp36m-win32.whl","hashes":{"sha256":"254adb6a27c888f141d2a6032ae231d8ed4fc5f7583b4c825e5f7d7c78d26d2e"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":219048,"upload-time":"2018-06-07T15:39:38.273913Z","url":"https://files.pythonhosted.org/packages/d6/e0/0f1b4f61246c4e2b540898b1ca0fa51ee2f52f0366956974f1039e00ed67/psutil-5.4.6-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"b7bbbf54e7767a4dcfada805607a533828d7e74794bf64ea1ae76700fc24d495"},"data-dist-info-metadata":{"sha256":"b7bbbf54e7767a4dcfada805607a533828d7e74794bf64ea1ae76700fc24d495"},"filename":"psutil-5.4.6-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"a9b85b335b40a528a8e2a6b549592138de8429c6296e7361892958956e6a73cf"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":222534,"upload-time":"2018-06-07T15:39:40.675791Z","url":"https://files.pythonhosted.org/packages/36/4b/80d9eb5d39ec4b4d8aec8b098b5097a7291de20bbbe6c2ab233b9d8fe245/psutil-5.4.6-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"b7bbbf54e7767a4dcfada805607a533828d7e74794bf64ea1ae76700fc24d495"},"data-dist-info-metadata":{"sha256":"b7bbbf54e7767a4dcfada805607a533828d7e74794bf64ea1ae76700fc24d495"},"filename":"psutil-5.4.6-cp37-cp37m-win32.whl","hashes":{"sha256":"6d981b4d863b20c8ceed98b8ac3d1ca7f96d28707a80845d360fa69c8fc2c44b"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":219860,"upload-time":"2018-06-28T22:47:57.979619Z","url":"https://files.pythonhosted.org/packages/7a/62/28923c44954b6cf8aee637f3a2f30e0e1ff39ec0f74a4f98069d37f00751/psutil-5.4.6-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"b7bbbf54e7767a4dcfada805607a533828d7e74794bf64ea1ae76700fc24d495"},"data-dist-info-metadata":{"sha256":"b7bbbf54e7767a4dcfada805607a533828d7e74794bf64ea1ae76700fc24d495"},"filename":"psutil-5.4.6-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"7fdb3d02bfd68f508e6745021311a4a4dbfec53fca03721474e985f310e249ba"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":224296,"upload-time":"2018-06-28T22:48:01.565801Z","url":"https://files.pythonhosted.org/packages/b1/be/78f9d786bddc190c4b394a01531741a11b95f1522cf2759958f13b46407f/psutil-5.4.6-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.4.6.tar.gz","hashes":{"sha256":"686e5a35fe4c0acc25f3466c32e716f2d498aaae7b7edc03e2305b682226bcf6"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":418059,"upload-time":"2018-06-07T15:39:42.364107Z","url":"https://files.pythonhosted.org/packages/51/9e/0f8f5423ce28c9109807024f7bdde776ed0b1161de20b408875de7e030c3/psutil-5.4.6.tar.gz","yanked":false},{"core-metadata":{"sha256":"94ef062f36609f64cc686f1930bf277790b85ced443769e242d41eb7a2588bec"},"data-dist-info-metadata":{"sha256":"94ef062f36609f64cc686f1930bf277790b85ced443769e242d41eb7a2588bec"},"filename":"psutil-5.4.7-cp27-none-win32.whl","hashes":{"sha256":"b34611280a2d0697f1c499e15e936d88109170194b390599c98bab8072a71f05"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":217777,"upload-time":"2018-08-14T21:01:10.586697Z","url":"https://files.pythonhosted.org/packages/b7/e9/bedbdfecef9d708489cfcd8b9aeada8d8f014fc14644c7129c7177e80d32/psutil-5.4.7-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"94ef062f36609f64cc686f1930bf277790b85ced443769e242d41eb7a2588bec"},"data-dist-info-metadata":{"sha256":"94ef062f36609f64cc686f1930bf277790b85ced443769e242d41eb7a2588bec"},"filename":"psutil-5.4.7-cp27-none-win_amd64.whl","hashes":{"sha256":"a890c3e490493f21da2817ffc92822693bc0d6bcac9999caa04ffce8dd4e7132"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":220989,"upload-time":"2018-08-14T21:01:13.273922Z","url":"https://files.pythonhosted.org/packages/50/6a/34525bc4e6e153bf6e849a4c4e936742b365f6819c0462cebfa4f082a3c4/psutil-5.4.7-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"77861491100106f950300f1104d6fcb309d103d34e379a69c8780e9edefa6710"},"data-dist-info-metadata":{"sha256":"77861491100106f950300f1104d6fcb309d103d34e379a69c8780e9edefa6710"},"filename":"psutil-5.4.7-cp35-cp35m-win32.whl","hashes":{"sha256":"1914bacbd2fc2af8f795daa44b9d2e0649a147460cfd21b1a70a124472f66d40"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":220344,"upload-time":"2018-08-14T21:01:15.659359Z","url":"https://files.pythonhosted.org/packages/36/32/5b10d7c3940c64fe92620c368ede8a10016d51aa36079a5cd69944da5a74/psutil-5.4.7-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"77861491100106f950300f1104d6fcb309d103d34e379a69c8780e9edefa6710"},"data-dist-info-metadata":{"sha256":"77861491100106f950300f1104d6fcb309d103d34e379a69c8780e9edefa6710"},"filename":"psutil-5.4.7-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"d081707ef0081920533db30200a2d30d5c0ea9cf6afa7cf8881ae4516cc69c48"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":224081,"upload-time":"2018-08-14T21:01:18.052521Z","url":"https://files.pythonhosted.org/packages/ef/fc/8dc7731df7de2f4c65378a7147ecc977221093eee90d9777ca501c2790c5/psutil-5.4.7-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"77861491100106f950300f1104d6fcb309d103d34e379a69c8780e9edefa6710"},"data-dist-info-metadata":{"sha256":"77861491100106f950300f1104d6fcb309d103d34e379a69c8780e9edefa6710"},"filename":"psutil-5.4.7-cp36-cp36m-win32.whl","hashes":{"sha256":"0d8da7333549a998556c18eb2af3ce902c28d66ceb947505c008f91e9f988abd"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":220342,"upload-time":"2018-08-14T21:01:20.870660Z","url":"https://files.pythonhosted.org/packages/20/6e/a9a0f84bc3efe970b4c8688b7e7f14ee8342497de8a88cffd35bb485cdcc/psutil-5.4.7-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"77861491100106f950300f1104d6fcb309d103d34e379a69c8780e9edefa6710"},"data-dist-info-metadata":{"sha256":"77861491100106f950300f1104d6fcb309d103d34e379a69c8780e9edefa6710"},"filename":"psutil-5.4.7-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"cea2557ee6a9faa2c100947637ded68414e12b851633c4ce26e0311b2a2ed539"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":224085,"upload-time":"2018-08-14T21:01:22.891149Z","url":"https://files.pythonhosted.org/packages/97/9e/c056abafcf0fc7ca5bddbc21ad1bb7c67889e16c088b9759f00b95fefcb4/psutil-5.4.7-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"77861491100106f950300f1104d6fcb309d103d34e379a69c8780e9edefa6710"},"data-dist-info-metadata":{"sha256":"77861491100106f950300f1104d6fcb309d103d34e379a69c8780e9edefa6710"},"filename":"psutil-5.4.7-cp37-cp37m-win32.whl","hashes":{"sha256":"215d61a901e67b1a35e14c6aedef317f7fa7e6075a20c150fd11bd2c906d2c83"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":220347,"upload-time":"2018-08-14T21:01:25.157286Z","url":"https://files.pythonhosted.org/packages/2e/33/5cef36162d94cf0adce428729adeb18b8548ff060781854f3aca71e6b0f0/psutil-5.4.7-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"77861491100106f950300f1104d6fcb309d103d34e379a69c8780e9edefa6710"},"data-dist-info-metadata":{"sha256":"77861491100106f950300f1104d6fcb309d103d34e379a69c8780e9edefa6710"},"filename":"psutil-5.4.7-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"51057c03aea251ad6667c2bba259bc7ed3210222d3a74152c84e3ab06e1da0ba"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":224087,"upload-time":"2018-08-14T21:01:27.563142Z","url":"https://files.pythonhosted.org/packages/bb/15/aa3d11ae8bf04b7683224f7d3b8f2dd4d3f8a918dcce59bb1f987fca9c6e/psutil-5.4.7-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.4.7.tar.gz","hashes":{"sha256":"5b6322b167a5ba0c5463b4d30dfd379cd4ce245a1162ebf8fc7ab5c5ffae4f3b"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":420300,"upload-time":"2018-08-14T21:01:30.063470Z","url":"https://files.pythonhosted.org/packages/7d/9a/1e93d41708f8ed2b564395edfa3389f0fd6d567597401c2e5e2775118d8b/psutil-5.4.7.tar.gz","yanked":false},{"core-metadata":{"sha256":"000e5f1178f3b76020f7866397e41f07c1f4564da730a45c24894e3aebebb7cb"},"data-dist-info-metadata":{"sha256":"000e5f1178f3b76020f7866397e41f07c1f4564da730a45c24894e3aebebb7cb"},"filename":"psutil-5.4.8-cp27-none-win32.whl","hashes":{"sha256":"809c9cef0402e3e48b5a1dddc390a8a6ff58b15362ea5714494073fa46c3d293"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":220106,"upload-time":"2018-10-30T09:57:34.685390Z","url":"https://files.pythonhosted.org/packages/5a/3f/3f0920df352dae7f824e0e612ff02591378f78405d6c7663dcac023005c4/psutil-5.4.8-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"000e5f1178f3b76020f7866397e41f07c1f4564da730a45c24894e3aebebb7cb"},"data-dist-info-metadata":{"sha256":"000e5f1178f3b76020f7866397e41f07c1f4564da730a45c24894e3aebebb7cb"},"filename":"psutil-5.4.8-cp27-none-win_amd64.whl","hashes":{"sha256":"3b7a4daf4223dae171a67a89314ac5ca0738e94064a78d99cfd751c55d05f315"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":223347,"upload-time":"2018-10-30T09:57:37.065973Z","url":"https://files.pythonhosted.org/packages/0f/fb/6aecd2c8c9d0ac83d789eaf9f9ec052dd61dd5aea2b47ffa4704175d7a2a/psutil-5.4.8-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"edcd0deabcf9a4ae02acef42bf31c36db6ad085b36a3cc549cfded47de1f1e8f"},"data-dist-info-metadata":{"sha256":"edcd0deabcf9a4ae02acef42bf31c36db6ad085b36a3cc549cfded47de1f1e8f"},"filename":"psutil-5.4.8-cp35-cp35m-win32.whl","hashes":{"sha256":"bbffac64cfd01c6bcf90eb1bedc6c80501c4dae8aef4ad6d6dd49f8f05f6fc5a"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":222713,"upload-time":"2018-10-30T09:57:39.214229Z","url":"https://files.pythonhosted.org/packages/46/2e/ce4ec4b60decc23e0e4d148b6f44c7ddd06ba0ab207dfaee21958bd669df/psutil-5.4.8-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"edcd0deabcf9a4ae02acef42bf31c36db6ad085b36a3cc549cfded47de1f1e8f"},"data-dist-info-metadata":{"sha256":"edcd0deabcf9a4ae02acef42bf31c36db6ad085b36a3cc549cfded47de1f1e8f"},"filename":"psutil-5.4.8-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"b4d1b735bf5b120813f4c89db8ac22d89162c558cbd7fdd298866125fe906219"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":226441,"upload-time":"2018-10-30T09:57:41.411069Z","url":"https://files.pythonhosted.org/packages/7f/28/5ccdb98eff12e7741cc2a6d9dcfdd5d9e06f6d363c2c019d5bfa0e0c1282/psutil-5.4.8-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"edcd0deabcf9a4ae02acef42bf31c36db6ad085b36a3cc549cfded47de1f1e8f"},"data-dist-info-metadata":{"sha256":"edcd0deabcf9a4ae02acef42bf31c36db6ad085b36a3cc549cfded47de1f1e8f"},"filename":"psutil-5.4.8-cp36-cp36m-win32.whl","hashes":{"sha256":"3e19be3441134445347af3767fa7770137d472a484070840eee6653b94ac5576"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":222711,"upload-time":"2018-10-30T09:57:43.901154Z","url":"https://files.pythonhosted.org/packages/b5/31/8ac896ca77a6aa75ee900698f96ddce46e96bb2484a92457c359a4e4bae6/psutil-5.4.8-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"edcd0deabcf9a4ae02acef42bf31c36db6ad085b36a3cc549cfded47de1f1e8f"},"data-dist-info-metadata":{"sha256":"edcd0deabcf9a4ae02acef42bf31c36db6ad085b36a3cc549cfded47de1f1e8f"},"filename":"psutil-5.4.8-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"1c19957883e0b93d081d41687089ad630e370e26dc49fd9df6951d6c891c4736"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":226440,"upload-time":"2018-10-30T09:57:46.331527Z","url":"https://files.pythonhosted.org/packages/3b/15/62d1eeb4c015e20295e0197f7de0202bd9e5bcb5529b9503932decde2505/psutil-5.4.8-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"edcd0deabcf9a4ae02acef42bf31c36db6ad085b36a3cc549cfded47de1f1e8f"},"data-dist-info-metadata":{"sha256":"edcd0deabcf9a4ae02acef42bf31c36db6ad085b36a3cc549cfded47de1f1e8f"},"filename":"psutil-5.4.8-cp37-cp37m-win32.whl","hashes":{"sha256":"bfcea4f189177b2d2ce4a34b03c4ac32c5b4c22e21f5b093d9d315e6e253cd81"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":222709,"upload-time":"2018-10-30T09:57:48.860424Z","url":"https://files.pythonhosted.org/packages/21/1e/fe6731e5f03ddf2e57d5b307f25bba294262bc88e27a0fbefdb3515d1727/psutil-5.4.8-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"edcd0deabcf9a4ae02acef42bf31c36db6ad085b36a3cc549cfded47de1f1e8f"},"data-dist-info-metadata":{"sha256":"edcd0deabcf9a4ae02acef42bf31c36db6ad085b36a3cc549cfded47de1f1e8f"},"filename":"psutil-5.4.8-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"1c71b9716790e202a00ab0931a6d1e25db1aa1198bcacaea2f5329f75d257fff"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":226447,"upload-time":"2018-10-30T09:57:50.676641Z","url":"https://files.pythonhosted.org/packages/50/00/ae52663b879333aa5c65fc9a87ddc24169f8fdd1831762a1ba9c9be7740d/psutil-5.4.8-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.4.8.tar.gz","hashes":{"sha256":"6e265c8f3da00b015d24b842bfeb111f856b13d24f2c57036582568dc650d6c3"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":422742,"upload-time":"2018-10-30T09:57:52.770639Z","url":"https://files.pythonhosted.org/packages/e3/58/0eae6e4466e5abf779d7e2b71fac7fba5f59e00ea36ddb3ed690419ccb0f/psutil-5.4.8.tar.gz","yanked":false},{"core-metadata":{"sha256":"1d8c3b2819c1de19ccc23f8de068eab529322b6df33673da0ec45a41f785bdcc"},"data-dist-info-metadata":{"sha256":"1d8c3b2819c1de19ccc23f8de068eab529322b6df33673da0ec45a41f785bdcc"},"filename":"psutil-5.5.0-cp27-none-win32.whl","hashes":{"sha256":"96f3fdb4ef7467854d46ad5a7e28eb4c6dc6d455d751ddf9640cd6d52bdb03d7"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":221314,"upload-time":"2019-01-23T18:23:33.526811Z","url":"https://files.pythonhosted.org/packages/fa/53/53f8c4f1af6f81b169ce76bfd0f56698bc1705da498a47d2ce701f7d7fe3/psutil-5.5.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"1d8c3b2819c1de19ccc23f8de068eab529322b6df33673da0ec45a41f785bdcc"},"data-dist-info-metadata":{"sha256":"1d8c3b2819c1de19ccc23f8de068eab529322b6df33673da0ec45a41f785bdcc"},"filename":"psutil-5.5.0-cp27-none-win_amd64.whl","hashes":{"sha256":"d23f7025bac9b3e38adc6bd032cdaac648ac0074d18e36950a04af35458342e8"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":224546,"upload-time":"2019-01-23T18:23:36.443393Z","url":"https://files.pythonhosted.org/packages/86/96/a7bfcc3aebedd7112ff353204901db6a1a0c1f3555b2788c68842bb78005/psutil-5.5.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"053bd08cfadac2a889dd4c4d2e10bd70c7d11e50fa66c23c145303639bb537b1"},"data-dist-info-metadata":{"sha256":"053bd08cfadac2a889dd4c4d2e10bd70c7d11e50fa66c23c145303639bb537b1"},"filename":"psutil-5.5.0-cp35-cp35m-win32.whl","hashes":{"sha256":"04d2071100aaad59f9bcbb801be2125d53b2e03b1517d9fed90b45eea51d297e"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":224084,"upload-time":"2019-01-23T18:23:39.132334Z","url":"https://files.pythonhosted.org/packages/c6/ca/f5d3841ca35e3e3607ed64fe61d2c392054692f05f35e807335299a7952b/psutil-5.5.0-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"053bd08cfadac2a889dd4c4d2e10bd70c7d11e50fa66c23c145303639bb537b1"},"data-dist-info-metadata":{"sha256":"053bd08cfadac2a889dd4c4d2e10bd70c7d11e50fa66c23c145303639bb537b1"},"filename":"psutil-5.5.0-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"d0c4230d60376aee0757d934020b14899f6020cd70ef8d2cb4f228b6ffc43e8f"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":227536,"upload-time":"2019-01-23T18:23:42.027389Z","url":"https://files.pythonhosted.org/packages/86/0d/a13c15ddccd8e2ccabe63f6d4f139f5a94150b758026e030310e87dded80/psutil-5.5.0-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"053bd08cfadac2a889dd4c4d2e10bd70c7d11e50fa66c23c145303639bb537b1"},"data-dist-info-metadata":{"sha256":"053bd08cfadac2a889dd4c4d2e10bd70c7d11e50fa66c23c145303639bb537b1"},"filename":"psutil-5.5.0-cp36-cp36m-win32.whl","hashes":{"sha256":"3ac48568f5b85fee44cd8002a15a7733deca056a191d313dbf24c11519c0c4a8"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":224085,"upload-time":"2019-01-23T18:23:44.450649Z","url":"https://files.pythonhosted.org/packages/45/00/7cdd50ded02e18e50667e2f76ceb645ecdfce59deb39422485f198c7be37/psutil-5.5.0-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"053bd08cfadac2a889dd4c4d2e10bd70c7d11e50fa66c23c145303639bb537b1"},"data-dist-info-metadata":{"sha256":"053bd08cfadac2a889dd4c4d2e10bd70c7d11e50fa66c23c145303639bb537b1"},"filename":"psutil-5.5.0-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"f0fcb7d3006dd4d9ccf3ccd0595d44c6abbfd433ec31b6ca177300ee3f19e54e"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":227537,"upload-time":"2019-01-23T18:23:46.937101Z","url":"https://files.pythonhosted.org/packages/48/d1/c9105512328c7f9800c51992b912df6f945eac696dfcd850f719541f67f3/psutil-5.5.0-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"053bd08cfadac2a889dd4c4d2e10bd70c7d11e50fa66c23c145303639bb537b1"},"data-dist-info-metadata":{"sha256":"053bd08cfadac2a889dd4c4d2e10bd70c7d11e50fa66c23c145303639bb537b1"},"filename":"psutil-5.5.0-cp37-cp37m-win32.whl","hashes":{"sha256":"c8ee08ad1b716911c86f12dc753eb1879006224fd51509f077987bb6493be615"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":224078,"upload-time":"2019-01-23T18:23:49.451606Z","url":"https://files.pythonhosted.org/packages/3f/14/5adcad73f22ae0c8fba8b054c0bb7c33c906121b588cc6cbdd686f098947/psutil-5.5.0-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"053bd08cfadac2a889dd4c4d2e10bd70c7d11e50fa66c23c145303639bb537b1"},"data-dist-info-metadata":{"sha256":"053bd08cfadac2a889dd4c4d2e10bd70c7d11e50fa66c23c145303639bb537b1"},"filename":"psutil-5.5.0-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"b755be689d6fc8ebc401e1d5ce5bac867e35788f10229e166338484eead51b12"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":227542,"upload-time":"2019-01-23T18:23:52.035332Z","url":"https://files.pythonhosted.org/packages/38/f1/a822d2b3d973c1ddd9d8a81d269e36987bab20e7bb28ecaa55aef66e8df5/psutil-5.5.0-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.5.0.tar.gz","hashes":{"sha256":"1aba93430050270750d046a179c5f3d6e1f5f8b96c20399ba38c596b28fc4d37"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":425058,"upload-time":"2019-01-23T18:23:54.951599Z","url":"https://files.pythonhosted.org/packages/6e/a0/833bcbcede5141cc5615e50c7cc5b960ce93d9c9b885fbe3b7d36e48a2d4/psutil-5.5.0.tar.gz","yanked":false},{"core-metadata":{"sha256":"0fa640ac67268845bdf0ad87325deaf6caae5372bf3703f282c5340f9af1f8c2"},"data-dist-info-metadata":{"sha256":"0fa640ac67268845bdf0ad87325deaf6caae5372bf3703f282c5340f9af1f8c2"},"filename":"psutil-5.5.1-cp27-none-win32.whl","hashes":{"sha256":"77c231b4dff8c1c329a4cd1c22b96c8976c597017ff5b09993cd148d6a94500c"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":221914,"upload-time":"2019-02-15T19:34:04.528250Z","url":"https://files.pythonhosted.org/packages/cc/cd/64aaf20c945662260026a128a08e46b93a49953224c0dccfdc6f37495d45/psutil-5.5.1-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"0fa640ac67268845bdf0ad87325deaf6caae5372bf3703f282c5340f9af1f8c2"},"data-dist-info-metadata":{"sha256":"0fa640ac67268845bdf0ad87325deaf6caae5372bf3703f282c5340f9af1f8c2"},"filename":"psutil-5.5.1-cp27-none-win_amd64.whl","hashes":{"sha256":"5ce6b5eb0267233459f4d3980c205828482f450999b8f5b684d9629fea98782a"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":225325,"upload-time":"2019-02-15T19:34:07.193210Z","url":"https://files.pythonhosted.org/packages/08/92/97b011d665ade1caf05dd02a3af4ede751c7b80f34812bc81479ec867d85/psutil-5.5.1-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"2aea42bc7128a735bd3dd530e3776b5c46cfb57362fa41b02ed50e6275148b66"},"data-dist-info-metadata":{"sha256":"2aea42bc7128a735bd3dd530e3776b5c46cfb57362fa41b02ed50e6275148b66"},"filename":"psutil-5.5.1-cp35-cp35m-win32.whl","hashes":{"sha256":"a013b4250ccbddc9d22feca0f986a1afc71717ad026c0f2109bbffd007351191"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":224822,"upload-time":"2019-02-15T19:34:09.528347Z","url":"https://files.pythonhosted.org/packages/6f/05/70f033e35cd34bc23a08793eaf713347c615674585ccfc7628b40eac4094/psutil-5.5.1-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"2aea42bc7128a735bd3dd530e3776b5c46cfb57362fa41b02ed50e6275148b66"},"data-dist-info-metadata":{"sha256":"2aea42bc7128a735bd3dd530e3776b5c46cfb57362fa41b02ed50e6275148b66"},"filename":"psutil-5.5.1-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"ef3e5e02b3c5d1df366abe7b4820400d5c427579668ad4465ff189d28ded5ebd"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":228328,"upload-time":"2019-02-15T19:34:12.035704Z","url":"https://files.pythonhosted.org/packages/8e/8d/1854ed30b9f69f4b2cc04ef4364ae8a52ad2988a3223bf6314d2d47f0f04/psutil-5.5.1-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"2aea42bc7128a735bd3dd530e3776b5c46cfb57362fa41b02ed50e6275148b66"},"data-dist-info-metadata":{"sha256":"2aea42bc7128a735bd3dd530e3776b5c46cfb57362fa41b02ed50e6275148b66"},"filename":"psutil-5.5.1-cp36-cp36m-win32.whl","hashes":{"sha256":"ad43b83119eeea6d5751023298cd331637e542cbd332196464799e25a5519f8f"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":224831,"upload-time":"2019-02-15T19:34:15.031823Z","url":"https://files.pythonhosted.org/packages/e8/c1/32fe16cb90192a9413f3ba303021047c158cbdde5aabd7e26ace8b54f69e/psutil-5.5.1-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"2aea42bc7128a735bd3dd530e3776b5c46cfb57362fa41b02ed50e6275148b66"},"data-dist-info-metadata":{"sha256":"2aea42bc7128a735bd3dd530e3776b5c46cfb57362fa41b02ed50e6275148b66"},"filename":"psutil-5.5.1-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"ec1ef313530a9457e48d25e3fdb1723dfa636008bf1b970027462d46f2555d59"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":228328,"upload-time":"2019-02-15T19:34:17.494390Z","url":"https://files.pythonhosted.org/packages/3e/6e/c0af4900f18811f09b93064588e53f3997abc051ae43f717d1ba610de3b7/psutil-5.5.1-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"2aea42bc7128a735bd3dd530e3776b5c46cfb57362fa41b02ed50e6275148b66"},"data-dist-info-metadata":{"sha256":"2aea42bc7128a735bd3dd530e3776b5c46cfb57362fa41b02ed50e6275148b66"},"filename":"psutil-5.5.1-cp37-cp37m-win32.whl","hashes":{"sha256":"c177777c787d247d02dae6c855330f9ed3e1abf8ca1744c26dd5ff968949999a"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":224832,"upload-time":"2019-02-15T19:34:19.685175Z","url":"https://files.pythonhosted.org/packages/00/e6/561fed27453add44af41a52e13e1dfca4d1e35705d698769edea6292339a/psutil-5.5.1-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"2aea42bc7128a735bd3dd530e3776b5c46cfb57362fa41b02ed50e6275148b66"},"data-dist-info-metadata":{"sha256":"2aea42bc7128a735bd3dd530e3776b5c46cfb57362fa41b02ed50e6275148b66"},"filename":"psutil-5.5.1-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"8846ab0be0cdccd6cc92ecd1246a16e2f2e49f53bd73e522c3a75ac291e1b51d"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":228327,"upload-time":"2019-02-15T19:34:22.218113Z","url":"https://files.pythonhosted.org/packages/3d/22/ed4fa46c5bfd95b4dc57d6544c3fe6568abe398aef3990f6011777f1a3f3/psutil-5.5.1-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.5.1.tar.gz","hashes":{"sha256":"72cebfaa422b7978a1d3632b65ff734a34c6b34f4578b68a5c204d633756b810"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":426750,"upload-time":"2019-02-15T19:34:24.841527Z","url":"https://files.pythonhosted.org/packages/c7/01/7c30b247cdc5ba29623faa5c8cf1f1bbf7e041783c340414b0ed7e067c64/psutil-5.5.1.tar.gz","yanked":false},{"core-metadata":{"sha256":"254b7bff05e4525511dae40a6fde55a8c45855ad1f6f2e1bc7dc4a50d7c0caf6"},"data-dist-info-metadata":{"sha256":"254b7bff05e4525511dae40a6fde55a8c45855ad1f6f2e1bc7dc4a50d7c0caf6"},"filename":"psutil-5.6.0-cp27-none-win32.whl","hashes":{"sha256":"1020a37214c4138e34962881372b40f390582b5c8245680c04349c2afb785a25"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":222534,"upload-time":"2019-03-05T12:00:59.141291Z","url":"https://files.pythonhosted.org/packages/04/f5/11b1c93a8882615fdaf6222aaf9d3197f250ab3036d7ecf6b6c8594ddf61/psutil-5.6.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"254b7bff05e4525511dae40a6fde55a8c45855ad1f6f2e1bc7dc4a50d7c0caf6"},"data-dist-info-metadata":{"sha256":"254b7bff05e4525511dae40a6fde55a8c45855ad1f6f2e1bc7dc4a50d7c0caf6"},"filename":"psutil-5.6.0-cp27-none-win_amd64.whl","hashes":{"sha256":"d9cdc2e82aeb82200fff3640f375fac39d88b1bed27ce08377cd7fb0e3621cb7"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":225803,"upload-time":"2019-03-05T12:01:01.831664Z","url":"https://files.pythonhosted.org/packages/11/88/ed94a7c091fb6ad8fbf545f0f20e140c47286712d6d85dd8cfc40b34fe72/psutil-5.6.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"4587b1b69c3d6666cbea0b1ac5ab97e537ce98c5c4986439eec64421d1b2baeb"},"data-dist-info-metadata":{"sha256":"4587b1b69c3d6666cbea0b1ac5ab97e537ce98c5c4986439eec64421d1b2baeb"},"filename":"psutil-5.6.0-cp35-cp35m-win32.whl","hashes":{"sha256":"c4a2f42abee709ed97b4498c21aa608ac31fc1f7cc8aa60ebdcd3c80757a038d"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":226153,"upload-time":"2019-03-05T12:01:04.695139Z","url":"https://files.pythonhosted.org/packages/ab/5c/dacbb4b6623dc390939e1785a818e6e816eb42c8c96c1f2d30a2fc22a773/psutil-5.6.0-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"4587b1b69c3d6666cbea0b1ac5ab97e537ce98c5c4986439eec64421d1b2baeb"},"data-dist-info-metadata":{"sha256":"4587b1b69c3d6666cbea0b1ac5ab97e537ce98c5c4986439eec64421d1b2baeb"},"filename":"psutil-5.6.0-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"722dc0dcce5272f3c5c41609fdc2c8f0ee3f976550c2d2f2057e26ba760be9c0"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":230078,"upload-time":"2019-03-05T12:01:07.429607Z","url":"https://files.pythonhosted.org/packages/41/27/9e6d39ae822387ced9a77da904e24e8796c1fa55c5f637cd35221b170980/psutil-5.6.0-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"4587b1b69c3d6666cbea0b1ac5ab97e537ce98c5c4986439eec64421d1b2baeb"},"data-dist-info-metadata":{"sha256":"4587b1b69c3d6666cbea0b1ac5ab97e537ce98c5c4986439eec64421d1b2baeb"},"filename":"psutil-5.6.0-cp36-cp36m-win32.whl","hashes":{"sha256":"1c8e6444ca1cee9a60a1a35913b8409722f7474616e0e21004e4ffadba59964b"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":226154,"upload-time":"2019-03-05T12:01:10.226359Z","url":"https://files.pythonhosted.org/packages/fb/d0/14b83939ed41c5a80dca6fa072b1a99cd576e33810f691e73a08a7c045b4/psutil-5.6.0-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"4587b1b69c3d6666cbea0b1ac5ab97e537ce98c5c4986439eec64421d1b2baeb"},"data-dist-info-metadata":{"sha256":"4587b1b69c3d6666cbea0b1ac5ab97e537ce98c5c4986439eec64421d1b2baeb"},"filename":"psutil-5.6.0-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"151c9858c268a1523e16fab33e3bc3bae8a0e57b57cf7fcad85fb409cbac6baf"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":230083,"upload-time":"2019-03-05T12:01:12.681476Z","url":"https://files.pythonhosted.org/packages/ce/3a/ff53c0ee59a864c3614fcaea45f4246e670934ea0d30b632c6e3905533c9/psutil-5.6.0-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"4587b1b69c3d6666cbea0b1ac5ab97e537ce98c5c4986439eec64421d1b2baeb"},"data-dist-info-metadata":{"sha256":"4587b1b69c3d6666cbea0b1ac5ab97e537ce98c5c4986439eec64421d1b2baeb"},"filename":"psutil-5.6.0-cp37-cp37m-win32.whl","hashes":{"sha256":"86f61a1438c026c980a4c3e2dd88a5774a3a0f00d6d0954d6c5cf8d1921b804e"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":226154,"upload-time":"2019-03-05T12:01:15.084497Z","url":"https://files.pythonhosted.org/packages/17/15/bf444a07aae2200f7b4e786c73bfe959201cb9ab63f737be12d21b5f252e/psutil-5.6.0-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"4587b1b69c3d6666cbea0b1ac5ab97e537ce98c5c4986439eec64421d1b2baeb"},"data-dist-info-metadata":{"sha256":"4587b1b69c3d6666cbea0b1ac5ab97e537ce98c5c4986439eec64421d1b2baeb"},"filename":"psutil-5.6.0-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"da6676a484adec2fdd3e1ce1b70799881ffcb958e40208dd4c5beba0011f3589"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":230081,"upload-time":"2019-03-05T12:01:17.936041Z","url":"https://files.pythonhosted.org/packages/88/fd/a32491e77b37ffc00faf79c864975cfd720ae0ac867d93c90640d44adf43/psutil-5.6.0-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.6.0.tar.gz","hashes":{"sha256":"dca71c08335fbfc6929438fe3a502f169ba96dd20e50b3544053d6be5cb19d82"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":426596,"upload-time":"2019-03-05T12:01:20.833448Z","url":"https://files.pythonhosted.org/packages/79/e6/a4e3c92fe19d386dcc6149dbf0b76f1c93c5491ae9d9ecf866f6769b45a4/psutil-5.6.0.tar.gz","yanked":false},{"core-metadata":{"sha256":"36d6733a6cb331fb0a7acb6d901557574835b038d157bc2fc11404d520fe4b92"},"data-dist-info-metadata":{"sha256":"36d6733a6cb331fb0a7acb6d901557574835b038d157bc2fc11404d520fe4b92"},"filename":"psutil-5.6.1-cp27-none-win32.whl","hashes":{"sha256":"23e9cd90db94fbced5151eaaf9033ae9667c033dffe9e709da761c20138d25b6"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":223000,"upload-time":"2019-03-11T17:24:48.368476Z","url":"https://files.pythonhosted.org/packages/9d/f2/a84e650af7f04940709466384e94a0894cfe736e7cf43f48fb3bfb01be1b/psutil-5.6.1-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"36d6733a6cb331fb0a7acb6d901557574835b038d157bc2fc11404d520fe4b92"},"data-dist-info-metadata":{"sha256":"36d6733a6cb331fb0a7acb6d901557574835b038d157bc2fc11404d520fe4b92"},"filename":"psutil-5.6.1-cp27-none-win_amd64.whl","hashes":{"sha256":"e1494d20ffe7891d07d8cb9a8b306c1a38d48b13575265d090fc08910c56d474"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":226302,"upload-time":"2019-03-11T17:24:51.027456Z","url":"https://files.pythonhosted.org/packages/46/51/4007e6188b6d6b68c8b4195d6755bd76585cffd4d286d9f1815ff9f1af01/psutil-5.6.1-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ea349a2dffe50e05b14983a2cf8238d799954fa7f79adc264520315279fa3b54"},"data-dist-info-metadata":{"sha256":"ea349a2dffe50e05b14983a2cf8238d799954fa7f79adc264520315279fa3b54"},"filename":"psutil-5.6.1-cp35-cp35m-win32.whl","hashes":{"sha256":"ec4b4b638b84d42fc48139f9352f6c6587ee1018d55253542ee28db7480cc653"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":226675,"upload-time":"2019-03-11T17:24:53.625833Z","url":"https://files.pythonhosted.org/packages/55/f4/2dc147b66111116f5cb6a85fe42519b2cbfdbf6138562f8b0427bc754fc8/psutil-5.6.1-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"ea349a2dffe50e05b14983a2cf8238d799954fa7f79adc264520315279fa3b54"},"data-dist-info-metadata":{"sha256":"ea349a2dffe50e05b14983a2cf8238d799954fa7f79adc264520315279fa3b54"},"filename":"psutil-5.6.1-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"c1fd45931889dc1812ba61a517630d126f6185f688eac1693171c6524901b7de"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":230613,"upload-time":"2019-03-11T17:24:56.387303Z","url":"https://files.pythonhosted.org/packages/af/dd/f3bdafb37af7bf417f984db5381a5a27899f93fd2f87d5fd10fb6f3f4087/psutil-5.6.1-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ea349a2dffe50e05b14983a2cf8238d799954fa7f79adc264520315279fa3b54"},"data-dist-info-metadata":{"sha256":"ea349a2dffe50e05b14983a2cf8238d799954fa7f79adc264520315279fa3b54"},"filename":"psutil-5.6.1-cp36-cp36m-win32.whl","hashes":{"sha256":"d463a142298112426ebd57351b45c39adb41341b91f033aa903fa4c6f76abecc"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":226672,"upload-time":"2019-03-11T17:24:58.629155Z","url":"https://files.pythonhosted.org/packages/d5/cc/e3fe388fe3c987973e929cb17cceb33bcd2ff7f8b754cd354826fbb7dfe7/psutil-5.6.1-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"ea349a2dffe50e05b14983a2cf8238d799954fa7f79adc264520315279fa3b54"},"data-dist-info-metadata":{"sha256":"ea349a2dffe50e05b14983a2cf8238d799954fa7f79adc264520315279fa3b54"},"filename":"psutil-5.6.1-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"9c3a768486194b4592c7ae9374faa55b37b9877fd9746fb4028cb0ac38fd4c60"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":230621,"upload-time":"2019-03-11T17:25:01.330880Z","url":"https://files.pythonhosted.org/packages/16/6a/cc5ba8d7e3ada0d4621d493dbdcb43ed38f3549642916a14c9e070add21a/psutil-5.6.1-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ea349a2dffe50e05b14983a2cf8238d799954fa7f79adc264520315279fa3b54"},"data-dist-info-metadata":{"sha256":"ea349a2dffe50e05b14983a2cf8238d799954fa7f79adc264520315279fa3b54"},"filename":"psutil-5.6.1-cp37-cp37m-win32.whl","hashes":{"sha256":"27858d688a58cbfdd4434e1c40f6c79eb5014b709e725c180488ccdf2f721729"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":226673,"upload-time":"2019-03-11T17:25:03.865746Z","url":"https://files.pythonhosted.org/packages/a8/8e/3e04eefe955ed94f93c3cde5dc1f1ccd99e2b9a56697fa804ea9f54f7baa/psutil-5.6.1-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"ea349a2dffe50e05b14983a2cf8238d799954fa7f79adc264520315279fa3b54"},"data-dist-info-metadata":{"sha256":"ea349a2dffe50e05b14983a2cf8238d799954fa7f79adc264520315279fa3b54"},"filename":"psutil-5.6.1-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"354601a1d1a1322ae5920ba397c58d06c29728a15113598d1a8158647aaa5385"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":230621,"upload-time":"2019-03-11T17:25:06.996226Z","url":"https://files.pythonhosted.org/packages/6a/48/dbcda6d136da319e8bee8196e6c52ff7febf56bd241435cf6a516341a4b1/psutil-5.6.1-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.6.1.tar.gz","hashes":{"sha256":"fa0a570e0a30b9dd618bffbece590ae15726b47f9f1eaf7518dfb35f4d7dcd21"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":427472,"upload-time":"2019-03-11T17:25:09.802408Z","url":"https://files.pythonhosted.org/packages/2f/b8/11ec5006d2ec2998cb68349b8d1317c24c284cf918ecd6729739388e4c56/psutil-5.6.1.tar.gz","yanked":false},{"core-metadata":{"sha256":"34922aac8bd27ca8268d8103c8b443cb013543dbfb0b3b7cf9f18d802b4fc760"},"data-dist-info-metadata":{"sha256":"34922aac8bd27ca8268d8103c8b443cb013543dbfb0b3b7cf9f18d802b4fc760"},"filename":"psutil-5.6.2-cp27-none-win32.whl","hashes":{"sha256":"76fb0956d6d50e68e3f22e7cc983acf4e243dc0fcc32fd693d398cb21c928802"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":226692,"upload-time":"2019-04-26T02:43:03.614779Z","url":"https://files.pythonhosted.org/packages/56/8f/1bfb7e563e413f110ee06ac0cf12bb4b27f78e9cf277892d97ba08de4eac/psutil-5.6.2-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"34922aac8bd27ca8268d8103c8b443cb013543dbfb0b3b7cf9f18d802b4fc760"},"data-dist-info-metadata":{"sha256":"34922aac8bd27ca8268d8103c8b443cb013543dbfb0b3b7cf9f18d802b4fc760"},"filename":"psutil-5.6.2-cp27-none-win_amd64.whl","hashes":{"sha256":"753c5988edc07da00dafd6d3d279d41f98c62cd4d3a548c4d05741a023b0c2e7"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":230225,"upload-time":"2019-04-26T02:43:08.318806Z","url":"https://files.pythonhosted.org/packages/91/3f/2ae9cd04b2ccc5340838383ba638839a498d2613936b7830079f77de2bf1/psutil-5.6.2-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"11c017c8a57096625072309b7badf4a1f677690026dbfc728de5f3dfd4ae2936"},"data-dist-info-metadata":{"sha256":"11c017c8a57096625072309b7badf4a1f677690026dbfc728de5f3dfd4ae2936"},"filename":"psutil-5.6.2-cp35-cp35m-win32.whl","hashes":{"sha256":"a4c62319ec6bf2b3570487dd72d471307ae5495ce3802c1be81b8a22e438b4bc"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":230751,"upload-time":"2019-04-26T02:43:12.014098Z","url":"https://files.pythonhosted.org/packages/ec/39/33a7d6c4e347ffff7784ec4967c04d6319886c5a77c208d29a0980045bc8/psutil-5.6.2-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"11c017c8a57096625072309b7badf4a1f677690026dbfc728de5f3dfd4ae2936"},"data-dist-info-metadata":{"sha256":"11c017c8a57096625072309b7badf4a1f677690026dbfc728de5f3dfd4ae2936"},"filename":"psutil-5.6.2-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"ef342cb7d9b60e6100364f50c57fa3a77d02ff8665d5b956746ac01901247ac4"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":234724,"upload-time":"2019-04-26T02:43:15.850786Z","url":"https://files.pythonhosted.org/packages/a9/43/87e610adacc3f8e51d61c27a9d48db2c0e7fbac783764fefca4d4ec71dbe/psutil-5.6.2-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"11c017c8a57096625072309b7badf4a1f677690026dbfc728de5f3dfd4ae2936"},"data-dist-info-metadata":{"sha256":"11c017c8a57096625072309b7badf4a1f677690026dbfc728de5f3dfd4ae2936"},"filename":"psutil-5.6.2-cp36-cp36m-win32.whl","hashes":{"sha256":"acba1df9da3983ec3c9c963adaaf530fcb4be0cd400a8294f1ecc2db56499ddd"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":230749,"upload-time":"2019-04-26T02:43:19.804035Z","url":"https://files.pythonhosted.org/packages/23/2a/0f14e964507adc1732ade6eea3abd92afe34305614515ab856cbccca489a/psutil-5.6.2-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"11c017c8a57096625072309b7badf4a1f677690026dbfc728de5f3dfd4ae2936"},"data-dist-info-metadata":{"sha256":"11c017c8a57096625072309b7badf4a1f677690026dbfc728de5f3dfd4ae2936"},"filename":"psutil-5.6.2-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"206eb909aa8878101d0eca07f4b31889c748f34ed6820a12eb3168c7aa17478e"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":234728,"upload-time":"2019-04-26T02:43:23.354778Z","url":"https://files.pythonhosted.org/packages/62/b0/54effe77128bdd8b62ca10edf38c22dbe5594ad8b34ce31836011949ac0a/psutil-5.6.2-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"11c017c8a57096625072309b7badf4a1f677690026dbfc728de5f3dfd4ae2936"},"data-dist-info-metadata":{"sha256":"11c017c8a57096625072309b7badf4a1f677690026dbfc728de5f3dfd4ae2936"},"filename":"psutil-5.6.2-cp37-cp37m-win32.whl","hashes":{"sha256":"649f7ffc02114dced8fbd08afcd021af75f5f5b2311bc0e69e53e8f100fe296f"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":230750,"upload-time":"2019-04-26T02:43:26.831308Z","url":"https://files.pythonhosted.org/packages/86/90/ea3d046bb26aebcbf25fba32cf9ec7d0954ea6b8e4e9d9c87a31633dd96b/psutil-5.6.2-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"11c017c8a57096625072309b7badf4a1f677690026dbfc728de5f3dfd4ae2936"},"data-dist-info-metadata":{"sha256":"11c017c8a57096625072309b7badf4a1f677690026dbfc728de5f3dfd4ae2936"},"filename":"psutil-5.6.2-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"6ebf2b9c996bb8c7198b385bade468ac8068ad8b78c54a58ff288cd5f61992c7"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":234731,"upload-time":"2019-04-26T02:43:30.698781Z","url":"https://files.pythonhosted.org/packages/98/3c/65f28f78848730c18dfa3ff3a107e3911b6d51d1442cfce8db53356179c3/psutil-5.6.2-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.6.2.tar.gz","hashes":{"sha256":"828e1c3ca6756c54ac00f1427fdac8b12e21b8a068c3bb9b631a1734cada25ed"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":432907,"upload-time":"2019-04-26T02:43:34.455685Z","url":"https://files.pythonhosted.org/packages/c6/c1/beed5e4eaa1345901b595048fab1c85aee647ea0fc02d9e8bf9aceb81078/psutil-5.6.2.tar.gz","yanked":false},{"core-metadata":{"sha256":"b5acb81452f4830e88ff6014732a59e7fc4839457f7950deab90c8eed0d4e6b8"},"data-dist-info-metadata":{"sha256":"b5acb81452f4830e88ff6014732a59e7fc4839457f7950deab90c8eed0d4e6b8"},"filename":"psutil-5.6.3-cp27-none-win32.whl","hashes":{"sha256":"d5350cb66690915d60f8b233180f1e49938756fb2d501c93c44f8fb5b970cc63"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":226847,"upload-time":"2019-06-11T04:24:07.412816Z","url":"https://files.pythonhosted.org/packages/da/76/7e445566f2c4363691a98f16df0072c6ba92c7c29b4410ef2df6514c9861/psutil-5.6.3-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"b5acb81452f4830e88ff6014732a59e7fc4839457f7950deab90c8eed0d4e6b8"},"data-dist-info-metadata":{"sha256":"b5acb81452f4830e88ff6014732a59e7fc4839457f7950deab90c8eed0d4e6b8"},"filename":"psutil-5.6.3-cp27-none-win_amd64.whl","hashes":{"sha256":"b6e08f965a305cd84c2d07409bc16fbef4417d67b70c53b299116c5b895e3f45"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":230406,"upload-time":"2019-06-11T04:24:10.852056Z","url":"https://files.pythonhosted.org/packages/72/75/43047d7df3ea2af2bcd072e63420b8fa240b729c052295f8c4b964335d36/psutil-5.6.3-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"2df22107b0d2ddaa4d080521f6dd4f4139500bf928eeee6cbbfd64b2abb197fb"},"data-dist-info-metadata":{"sha256":"2df22107b0d2ddaa4d080521f6dd4f4139500bf928eeee6cbbfd64b2abb197fb"},"filename":"psutil-5.6.3-cp35-cp35m-win32.whl","hashes":{"sha256":"cf49178021075d47c61c03c0229ac0c60d5e2830f8cab19e2d88e579b18cdb76"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":230912,"upload-time":"2019-06-11T04:24:14.533883Z","url":"https://files.pythonhosted.org/packages/c2/f2/d99eaeefb2c0b1f7f9aca679db17f4072d4cc362f40f809157d4e2d273dd/psutil-5.6.3-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"2df22107b0d2ddaa4d080521f6dd4f4139500bf928eeee6cbbfd64b2abb197fb"},"data-dist-info-metadata":{"sha256":"2df22107b0d2ddaa4d080521f6dd4f4139500bf928eeee6cbbfd64b2abb197fb"},"filename":"psutil-5.6.3-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"bc96d437dfbb8865fc8828cf363450001cb04056bbdcdd6fc152c436c8a74c61"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":234942,"upload-time":"2019-06-11T04:24:18.128893Z","url":"https://files.pythonhosted.org/packages/90/86/d5ae0eb79cab6acc00d3640a45243e3e0602dc2f7abca29fc2fe6b4819ca/psutil-5.6.3-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"2df22107b0d2ddaa4d080521f6dd4f4139500bf928eeee6cbbfd64b2abb197fb"},"data-dist-info-metadata":{"sha256":"2df22107b0d2ddaa4d080521f6dd4f4139500bf928eeee6cbbfd64b2abb197fb"},"filename":"psutil-5.6.3-cp36-cp36m-win32.whl","hashes":{"sha256":"eba238cf1989dfff7d483c029acb0ac4fcbfc15de295d682901f0e2497e6781a"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":230914,"upload-time":"2019-06-11T04:24:21.735106Z","url":"https://files.pythonhosted.org/packages/28/0c/e41fd3020662487cf92f0c47b09285d7e53c42ee56cdc3ddb03a147cfa5d/psutil-5.6.3-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"2df22107b0d2ddaa4d080521f6dd4f4139500bf928eeee6cbbfd64b2abb197fb"},"data-dist-info-metadata":{"sha256":"2df22107b0d2ddaa4d080521f6dd4f4139500bf928eeee6cbbfd64b2abb197fb"},"filename":"psutil-5.6.3-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"954f782608bfef9ae9f78e660e065bd8ffcfaea780f9f2c8a133bb7cb9e826d7"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":234945,"upload-time":"2019-06-11T04:24:26.649579Z","url":"https://files.pythonhosted.org/packages/86/91/f15a3aae2af13f008ed95e02292d1a2e84615ff42b7203357c1c0bbe0651/psutil-5.6.3-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"2df22107b0d2ddaa4d080521f6dd4f4139500bf928eeee6cbbfd64b2abb197fb"},"data-dist-info-metadata":{"sha256":"2df22107b0d2ddaa4d080521f6dd4f4139500bf928eeee6cbbfd64b2abb197fb"},"filename":"psutil-5.6.3-cp37-cp37m-win32.whl","hashes":{"sha256":"503e4b20fa9d3342bcf58191bbc20a4a5ef79ca7df8972e6197cc14c5513e73d"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":230914,"upload-time":"2019-06-11T04:24:30.881936Z","url":"https://files.pythonhosted.org/packages/3b/92/2a7fb18054ac12483fb72f281c285d21642ca3d29fc6a06f0e44d4b36d83/psutil-5.6.3-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"2df22107b0d2ddaa4d080521f6dd4f4139500bf928eeee6cbbfd64b2abb197fb"},"data-dist-info-metadata":{"sha256":"2df22107b0d2ddaa4d080521f6dd4f4139500bf928eeee6cbbfd64b2abb197fb"},"filename":"psutil-5.6.3-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"028a1ec3c6197eadd11e7b46e8cc2f0720dc18ac6d7aabdb8e8c0d6c9704f000"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":234949,"upload-time":"2019-06-11T04:24:34.722512Z","url":"https://files.pythonhosted.org/packages/7c/58/f5d68ddca37480d8557b8566a20bf6108d7e1c6c9b9208ee0786e0cd012b/psutil-5.6.3-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"991ea9c126c59fb12521c83a1676786ad9a7ea2af4f3985b1e8086af3eda4156"},"data-dist-info-metadata":{"sha256":"991ea9c126c59fb12521c83a1676786ad9a7ea2af4f3985b1e8086af3eda4156"},"filename":"psutil-5.6.3-cp38-cp38-win_amd64.whl","hashes":{"sha256":"12542c3642909f4cd1928a2fba59e16fa27e47cbeea60928ebb62a8cbd1ce123"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238041,"upload-time":"2019-10-24T10:11:31.818649Z","url":"https://files.pythonhosted.org/packages/a7/7f/0761489b5467af4e97ae3c5a25f24f8662d69a25692d85490c8ea5d52e45/psutil-5.6.3-cp38-cp38-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.6.3.tar.gz","hashes":{"sha256":"863a85c1c0a5103a12c05a35e59d336e1d665747e531256e061213e2e90f63f3"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":435374,"upload-time":"2019-06-11T04:24:39.293855Z","url":"https://files.pythonhosted.org/packages/1c/ca/5b8c1fe032a458c2c4bcbe509d1401dca9dda35c7fc46b36bb81c2834740/psutil-5.6.3.tar.gz","yanked":false},{"core-metadata":{"sha256":"2625785be98a15c8fa27587a95cae17a6673efa1e3bc2154856b9af05aaac32a"},"data-dist-info-metadata":{"sha256":"2625785be98a15c8fa27587a95cae17a6673efa1e3bc2154856b9af05aaac32a"},"filename":"psutil-5.6.4-cp27-none-win32.whl","hashes":{"sha256":"75d50d1138b2476a11dca33ab1ad2b78707d428418b581966ccedac768358f72"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":228290,"upload-time":"2019-11-04T08:38:28.711265Z","url":"https://files.pythonhosted.org/packages/8a/63/0273163f0197ff2d8f026ae0f9c7cac371f8819f6451e6cfd2e1e7132b90/psutil-5.6.4-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"2625785be98a15c8fa27587a95cae17a6673efa1e3bc2154856b9af05aaac32a"},"data-dist-info-metadata":{"sha256":"2625785be98a15c8fa27587a95cae17a6673efa1e3bc2154856b9af05aaac32a"},"filename":"psutil-5.6.4-cp27-none-win_amd64.whl","hashes":{"sha256":"0ff1f630ee0df7c048ef53e50196437d2c9cebab8ccca0e3078d9300c4b7da47"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":231797,"upload-time":"2019-11-04T08:38:33.155488Z","url":"https://files.pythonhosted.org/packages/db/d0/17a47a1876cf408d0dd679f78ba18a206228fdb95928137dd85538a51298/psutil-5.6.4-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"f16d89990a0e49264d1393eb1ee021563470e55f511d8518008f85c41d8ff3b0"},"data-dist-info-metadata":{"sha256":"f16d89990a0e49264d1393eb1ee021563470e55f511d8518008f85c41d8ff3b0"},"filename":"psutil-5.6.4-cp35-cp35m-win32.whl","hashes":{"sha256":"10175ea15b7e4a1bf1a0863da7e17042862b3ea3e7d24285c96fa4cc65ab9788"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":232325,"upload-time":"2019-11-04T08:38:37.308918Z","url":"https://files.pythonhosted.org/packages/cb/22/e2ce6539342fb0ebd65fb9e2a0a168585eb5cae9bd5dca7977735f8d430e/psutil-5.6.4-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"f16d89990a0e49264d1393eb1ee021563470e55f511d8518008f85c41d8ff3b0"},"data-dist-info-metadata":{"sha256":"f16d89990a0e49264d1393eb1ee021563470e55f511d8518008f85c41d8ff3b0"},"filename":"psutil-5.6.4-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"f6b66a5663700b71bac3d8ecf6533a1550a679823e63b2c92dc4c3c8c244c52e"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236342,"upload-time":"2019-11-04T08:38:41.583480Z","url":"https://files.pythonhosted.org/packages/32/9a/beaf5df1f8b38d3ec7ec737f873515813586aa26926738896957c3af0a9d/psutil-5.6.4-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"f16d89990a0e49264d1393eb1ee021563470e55f511d8518008f85c41d8ff3b0"},"data-dist-info-metadata":{"sha256":"f16d89990a0e49264d1393eb1ee021563470e55f511d8518008f85c41d8ff3b0"},"filename":"psutil-5.6.4-cp36-cp36m-win32.whl","hashes":{"sha256":"4f637dd25d3bce4879d0b4032d13f4120ba18ed2d028e85d911d429f447c251c"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":232330,"upload-time":"2019-11-04T08:38:46.049782Z","url":"https://files.pythonhosted.org/packages/74/3c/61d46eccf9fca27fc11f30d27af11c3886609d5eb52475926a1ad12d6ebd/psutil-5.6.4-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"f16d89990a0e49264d1393eb1ee021563470e55f511d8518008f85c41d8ff3b0"},"data-dist-info-metadata":{"sha256":"f16d89990a0e49264d1393eb1ee021563470e55f511d8518008f85c41d8ff3b0"},"filename":"psutil-5.6.4-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"43f0d7536a98c20a538242ce2bd8c64dbc1f6c396e97f2bdceb496d7583b9b80"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236347,"upload-time":"2019-11-04T08:38:50.265170Z","url":"https://files.pythonhosted.org/packages/2c/59/beae1392ad5188b419709d3e04641cbe93e36184b7b8686af825a2232b2b/psutil-5.6.4-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"f16d89990a0e49264d1393eb1ee021563470e55f511d8518008f85c41d8ff3b0"},"data-dist-info-metadata":{"sha256":"f16d89990a0e49264d1393eb1ee021563470e55f511d8518008f85c41d8ff3b0"},"filename":"psutil-5.6.4-cp37-cp37m-win32.whl","hashes":{"sha256":"f0ec1a3ea56503f4facc1dca364cf3dd66dc39169c4603000d3d34270e05fbb3"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":232327,"upload-time":"2019-11-04T08:38:54.615037Z","url":"https://files.pythonhosted.org/packages/e6/2e/e0c14d0fa46afb15b1467daa792f143e675034f8c544d03a2b7365d4926c/psutil-5.6.4-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"f16d89990a0e49264d1393eb1ee021563470e55f511d8518008f85c41d8ff3b0"},"data-dist-info-metadata":{"sha256":"f16d89990a0e49264d1393eb1ee021563470e55f511d8518008f85c41d8ff3b0"},"filename":"psutil-5.6.4-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"512e77ac987105e2d7aa2386d9f260434ad8b71e41484f8d84bfecd4ae3764ca"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236347,"upload-time":"2019-11-04T08:38:58.188759Z","url":"https://files.pythonhosted.org/packages/a3/59/fe32a3ec677990a5ed6b1a44dc2e372c9ee40693890814e8b6d96c290c4d/psutil-5.6.4-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"f16d89990a0e49264d1393eb1ee021563470e55f511d8518008f85c41d8ff3b0"},"data-dist-info-metadata":{"sha256":"f16d89990a0e49264d1393eb1ee021563470e55f511d8518008f85c41d8ff3b0"},"filename":"psutil-5.6.4-cp38-cp38-win32.whl","hashes":{"sha256":"41d645f100c6b4c995ff342ef7d79a936f3f48e9a816d7d655c69b352460341d"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":234629,"upload-time":"2019-11-04T08:46:40.809419Z","url":"https://files.pythonhosted.org/packages/50/0c/7b2fa0baedd36147beb820f662844b1079e168c06e9fe62d78a4d2666375/psutil-5.6.4-cp38-cp38-win32.whl","yanked":false},{"core-metadata":{"sha256":"f16d89990a0e49264d1393eb1ee021563470e55f511d8518008f85c41d8ff3b0"},"data-dist-info-metadata":{"sha256":"f16d89990a0e49264d1393eb1ee021563470e55f511d8518008f85c41d8ff3b0"},"filename":"psutil-5.6.4-cp38-cp38-win_amd64.whl","hashes":{"sha256":"fb58e87c29ec0fb99937b95c5d473bb786d263aaa767d017a6bd4ad52d694e79"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":239057,"upload-time":"2019-11-04T08:46:45.192939Z","url":"https://files.pythonhosted.org/packages/e3/28/fa3a597ed798a5a2cb9c9e53c2d8b6bc1310c4a147c2aaa65c5d3afdb6ed/psutil-5.6.4-cp38-cp38-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.6.4.tar.gz","hashes":{"sha256":"512e854d68f8b42f79b2c7864d997b39125baff9bcff00028ce43543867de7c4"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":447564,"upload-time":"2019-11-04T08:39:02.419203Z","url":"https://files.pythonhosted.org/packages/47/ea/d3b6d6fd0b4a6c12984df652525f394e68c8678d2b05075219144eb3a1cf/psutil-5.6.4.tar.gz","yanked":false},{"core-metadata":{"sha256":"3ccf04696d9a6c2c81c586788345860eb9fc45feada6313ab8d40865c7a9ec78"},"data-dist-info-metadata":{"sha256":"3ccf04696d9a6c2c81c586788345860eb9fc45feada6313ab8d40865c7a9ec78"},"filename":"psutil-5.6.5-cp27-none-win32.whl","hashes":{"sha256":"145e0f3ab9138165f9e156c307100905fd5d9b7227504b8a9d3417351052dc3d"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":228289,"upload-time":"2019-11-06T10:07:27.672551Z","url":"https://files.pythonhosted.org/packages/97/52/9a44db00d4400814384d574f3a2c2b588259c212e457f542c2589dbdac58/psutil-5.6.5-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"3ccf04696d9a6c2c81c586788345860eb9fc45feada6313ab8d40865c7a9ec78"},"data-dist-info-metadata":{"sha256":"3ccf04696d9a6c2c81c586788345860eb9fc45feada6313ab8d40865c7a9ec78"},"filename":"psutil-5.6.5-cp27-none-win_amd64.whl","hashes":{"sha256":"3feea46fbd634a93437b718518d15b5dd49599dfb59a30c739e201cc79bb759d"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":231796,"upload-time":"2019-11-06T10:07:31.973662Z","url":"https://files.pythonhosted.org/packages/c4/35/4422a966d304faa401d55dc45caf342722a14c0a9b56d57ecf208f9bb6a3/psutil-5.6.5-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"98b3da0b296c70a30e4e371a72ba5d3205dc455f09fe824464a824ad9b4fd6ad"},"data-dist-info-metadata":{"sha256":"98b3da0b296c70a30e4e371a72ba5d3205dc455f09fe824464a824ad9b4fd6ad"},"filename":"psutil-5.6.5-cp35-cp35m-win32.whl","hashes":{"sha256":"348ad4179938c965a27d29cbda4a81a1b2c778ecd330a221aadc7bd33681afbd"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":232325,"upload-time":"2019-11-06T10:07:36.463607Z","url":"https://files.pythonhosted.org/packages/4c/2f/93ac7f065f8c4c361e422dffff21dea1803cb40f2449e6ea52800caddf9f/psutil-5.6.5-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"98b3da0b296c70a30e4e371a72ba5d3205dc455f09fe824464a824ad9b4fd6ad"},"data-dist-info-metadata":{"sha256":"98b3da0b296c70a30e4e371a72ba5d3205dc455f09fe824464a824ad9b4fd6ad"},"filename":"psutil-5.6.5-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"474e10a92eeb4100c276d4cc67687adeb9d280bbca01031a3e41fb35dfc1d131"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236340,"upload-time":"2019-11-06T10:07:41.723247Z","url":"https://files.pythonhosted.org/packages/9f/49/fddf3b6138a5d47668e2daf98ac9a6eeea4c28e3eb68b56580f18e8fe6af/psutil-5.6.5-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"98b3da0b296c70a30e4e371a72ba5d3205dc455f09fe824464a824ad9b4fd6ad"},"data-dist-info-metadata":{"sha256":"98b3da0b296c70a30e4e371a72ba5d3205dc455f09fe824464a824ad9b4fd6ad"},"filename":"psutil-5.6.5-cp36-cp36m-win32.whl","hashes":{"sha256":"e3f5f9278867e95970854e92d0f5fe53af742a7fc4f2eba986943345bcaed05d"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":232327,"upload-time":"2019-11-06T10:07:45.647021Z","url":"https://files.pythonhosted.org/packages/ae/5d/11bb7fb7cc004bdf1325c0b40827e67e479ababe47c553ee871494353acf/psutil-5.6.5-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"98b3da0b296c70a30e4e371a72ba5d3205dc455f09fe824464a824ad9b4fd6ad"},"data-dist-info-metadata":{"sha256":"98b3da0b296c70a30e4e371a72ba5d3205dc455f09fe824464a824ad9b4fd6ad"},"filename":"psutil-5.6.5-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"dfb8c5c78579c226841908b539c2374da54da648ee5a837a731aa6a105a54c00"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236344,"upload-time":"2019-11-06T10:07:49.683064Z","url":"https://files.pythonhosted.org/packages/c6/18/221e8a5084585c6a2550894fb0617dc4691b5216f4aa6bb82c330aa5d99c/psutil-5.6.5-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"98b3da0b296c70a30e4e371a72ba5d3205dc455f09fe824464a824ad9b4fd6ad"},"data-dist-info-metadata":{"sha256":"98b3da0b296c70a30e4e371a72ba5d3205dc455f09fe824464a824ad9b4fd6ad"},"filename":"psutil-5.6.5-cp37-cp37m-win32.whl","hashes":{"sha256":"021d361439586a0fd8e64f8392eb7da27135db980f249329f1a347b9de99c695"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":232327,"upload-time":"2019-11-06T10:07:53.427002Z","url":"https://files.pythonhosted.org/packages/31/f8/d612f18fed1422a016bb641d3ce1922904a2aa0cc3472ce4abf2716cf542/psutil-5.6.5-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"98b3da0b296c70a30e4e371a72ba5d3205dc455f09fe824464a824ad9b4fd6ad"},"data-dist-info-metadata":{"sha256":"98b3da0b296c70a30e4e371a72ba5d3205dc455f09fe824464a824ad9b4fd6ad"},"filename":"psutil-5.6.5-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"e9649bb8fc5cea1f7723af53e4212056a6f984ee31784c10632607f472dec5ee"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236346,"upload-time":"2019-11-06T10:07:57.218779Z","url":"https://files.pythonhosted.org/packages/97/c7/9b18c3b429c987796d74647c61030c7029518ac223d1a664951417781fe4/psutil-5.6.5-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"98b3da0b296c70a30e4e371a72ba5d3205dc455f09fe824464a824ad9b4fd6ad"},"data-dist-info-metadata":{"sha256":"98b3da0b296c70a30e4e371a72ba5d3205dc455f09fe824464a824ad9b4fd6ad"},"filename":"psutil-5.6.5-cp38-cp38-win32.whl","hashes":{"sha256":"47aeb4280e80f27878caae4b572b29f0ec7967554b701ba33cd3720b17ba1b07"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":234627,"upload-time":"2019-11-06T10:08:50.811061Z","url":"https://files.pythonhosted.org/packages/03/94/e4ee514cfbc4cca176fcc6b4b1118a724848b570941e90f0b98a9bd234e1/psutil-5.6.5-cp38-cp38-win32.whl","yanked":false},{"core-metadata":{"sha256":"98b3da0b296c70a30e4e371a72ba5d3205dc455f09fe824464a824ad9b4fd6ad"},"data-dist-info-metadata":{"sha256":"98b3da0b296c70a30e4e371a72ba5d3205dc455f09fe824464a824ad9b4fd6ad"},"filename":"psutil-5.6.5-cp38-cp38-win_amd64.whl","hashes":{"sha256":"73a7e002781bc42fd014dfebb3fc0e45f8d92a4fb9da18baea6fb279fbc1d966"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":239056,"upload-time":"2019-11-06T10:08:55.334494Z","url":"https://files.pythonhosted.org/packages/cd/3b/de8a1f1692de2f16716a108a63366aa66692e5a087b6e0458eef9739e652/psutil-5.6.5-cp38-cp38-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.6.5.tar.gz","hashes":{"sha256":"d051532ac944f1be0179e0506f6889833cf96e466262523e57a871de65a15147"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":447489,"upload-time":"2019-11-06T10:08:01.639000Z","url":"https://files.pythonhosted.org/packages/03/9a/95c4b3d0424426e5fd94b5302ff74cea44d5d4f53466e1228ac8e73e14b4/psutil-5.6.5.tar.gz","yanked":false},{"core-metadata":{"sha256":"0ee5221d5d9af774f9490843a768567a29b9f7ace20754cf0822d80eb298fdd1"},"data-dist-info-metadata":{"sha256":"0ee5221d5d9af774f9490843a768567a29b9f7ace20754cf0822d80eb298fdd1"},"filename":"psutil-5.6.6-cp27-none-win32.whl","hashes":{"sha256":"06660136ab88762309775fd47290d7da14094422d915f0466e0adf8e4b22214e"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":228564,"upload-time":"2019-11-25T12:30:48.823010Z","url":"https://files.pythonhosted.org/packages/c9/c2/4d703fcb5aacd4cb8d472d3c1d0e7d8c21e17f37b515016a7ee34ff3f4a0/psutil-5.6.6-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"0ee5221d5d9af774f9490843a768567a29b9f7ace20754cf0822d80eb298fdd1"},"data-dist-info-metadata":{"sha256":"0ee5221d5d9af774f9490843a768567a29b9f7ace20754cf0822d80eb298fdd1"},"filename":"psutil-5.6.6-cp27-none-win_amd64.whl","hashes":{"sha256":"f21a7bb4b207e4e7c60b3c40ffa89d790997619f04bbecec9db8e3696122bc78"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":232093,"upload-time":"2019-11-25T12:30:53.306595Z","url":"https://files.pythonhosted.org/packages/76/4d/6452c4791f9d95b48cca084b7cc6aa8a72b2f4c8d3d8bd38e7f3abfaf364/psutil-5.6.6-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"29b50729d8b20274ad25b9f22deb812c81cdc56e61cba32bbf7c5423b9c1ef5d"},"data-dist-info-metadata":{"sha256":"29b50729d8b20274ad25b9f22deb812c81cdc56e61cba32bbf7c5423b9c1ef5d"},"filename":"psutil-5.6.6-cp35-cp35m-win32.whl","hashes":{"sha256":"5e8dbf31871b0072bcba8d1f2861c0ec6c84c78f13c723bb6e981bce51b58f12"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":232818,"upload-time":"2019-11-25T12:30:58.414546Z","url":"https://files.pythonhosted.org/packages/f3/b3/e585b9b0c5a40e6a778e32e8d3040ab2363433990417202bf6cc0261ed77/psutil-5.6.6-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"29b50729d8b20274ad25b9f22deb812c81cdc56e61cba32bbf7c5423b9c1ef5d"},"data-dist-info-metadata":{"sha256":"29b50729d8b20274ad25b9f22deb812c81cdc56e61cba32bbf7c5423b9c1ef5d"},"filename":"psutil-5.6.6-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"724390895cff80add7a1c4e7e0a04d9c94f3ee61423a2dcafd83784fabbd1ee9"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236751,"upload-time":"2019-11-25T12:31:03.059024Z","url":"https://files.pythonhosted.org/packages/73/38/a8ebf6dc6ada2257591284be45a52dcd19479163b8d3575186333a79a18e/psutil-5.6.6-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"29b50729d8b20274ad25b9f22deb812c81cdc56e61cba32bbf7c5423b9c1ef5d"},"data-dist-info-metadata":{"sha256":"29b50729d8b20274ad25b9f22deb812c81cdc56e61cba32bbf7c5423b9c1ef5d"},"filename":"psutil-5.6.6-cp36-cp36m-win32.whl","hashes":{"sha256":"6d81b9714791ef9a3a00b2ca846ee547fc5e53d259e2a6258c3d2054928039ff"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":232821,"upload-time":"2019-11-25T12:31:08.161481Z","url":"https://files.pythonhosted.org/packages/5b/04/2223b4fe61d3e5962c08ce5062b09633fcfdd8c3bb08c31b76306f748431/psutil-5.6.6-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"29b50729d8b20274ad25b9f22deb812c81cdc56e61cba32bbf7c5423b9c1ef5d"},"data-dist-info-metadata":{"sha256":"29b50729d8b20274ad25b9f22deb812c81cdc56e61cba32bbf7c5423b9c1ef5d"},"filename":"psutil-5.6.6-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"3004361c6b93dbad71330d992c1ae409cb8314a6041a0b67507cc882357f583e"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236759,"upload-time":"2019-11-25T12:31:13.077581Z","url":"https://files.pythonhosted.org/packages/33/6c/6eb959bca82064d42e725dddd3aeeb39d9bed34eed7b513880bcfd8a3d59/psutil-5.6.6-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"29b50729d8b20274ad25b9f22deb812c81cdc56e61cba32bbf7c5423b9c1ef5d"},"data-dist-info-metadata":{"sha256":"29b50729d8b20274ad25b9f22deb812c81cdc56e61cba32bbf7c5423b9c1ef5d"},"filename":"psutil-5.6.6-cp37-cp37m-win32.whl","hashes":{"sha256":"0fc7a5619b47f74331add476fbc6022d7ca801c22865c7069ec0867920858963"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":232820,"upload-time":"2019-11-25T12:31:17.411233Z","url":"https://files.pythonhosted.org/packages/76/46/3a8dc20eb9d7d2c44178e71c2412dcc2c6476ab71c05a2cf2f77247c6e53/psutil-5.6.6-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"29b50729d8b20274ad25b9f22deb812c81cdc56e61cba32bbf7c5423b9c1ef5d"},"data-dist-info-metadata":{"sha256":"29b50729d8b20274ad25b9f22deb812c81cdc56e61cba32bbf7c5423b9c1ef5d"},"filename":"psutil-5.6.6-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"f60042bef7dc50a78c06334ca8e25580455948ba2fa98f240d034a4fed9141a5"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236757,"upload-time":"2019-11-25T12:31:21.816790Z","url":"https://files.pythonhosted.org/packages/40/48/5debf9783077ac71f0d715c1171fde9ef287909f61672ed9a3e5fdca63cc/psutil-5.6.6-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"29b50729d8b20274ad25b9f22deb812c81cdc56e61cba32bbf7c5423b9c1ef5d"},"data-dist-info-metadata":{"sha256":"29b50729d8b20274ad25b9f22deb812c81cdc56e61cba32bbf7c5423b9c1ef5d"},"filename":"psutil-5.6.6-cp38-cp38-win32.whl","hashes":{"sha256":"0c11adde31011a286197630ba2671e34651f004cc418d30ae06d2033a43c9e20"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":233147,"upload-time":"2019-11-25T12:31:26.294340Z","url":"https://files.pythonhosted.org/packages/c6/10/035c432a15bec90d51f3625d4b70b7d12f1b363062d0d8815213229f69ca/psutil-5.6.6-cp38-cp38-win32.whl","yanked":false},{"core-metadata":{"sha256":"29b50729d8b20274ad25b9f22deb812c81cdc56e61cba32bbf7c5423b9c1ef5d"},"data-dist-info-metadata":{"sha256":"29b50729d8b20274ad25b9f22deb812c81cdc56e61cba32bbf7c5423b9c1ef5d"},"filename":"psutil-5.6.6-cp38-cp38-win_amd64.whl","hashes":{"sha256":"0c211eec4185725847cb6c28409646c7cfa56fdb531014b35f97b5dc7fe04ff9"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":237165,"upload-time":"2019-11-25T12:31:30.835876Z","url":"https://files.pythonhosted.org/packages/95/81/fb02ea8de73eca26cfa347f5ce81a7963b9dee6e038e0a7389ccbc971093/psutil-5.6.6-cp38-cp38-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.6.6.tar.gz","hashes":{"sha256":"ad21281f7bd6c57578dd53913d2d44218e9e29fd25128d10ff7819ef16fa46e7"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":447805,"upload-time":"2019-11-25T12:31:36.347718Z","url":"https://files.pythonhosted.org/packages/5f/dc/edf6758183afc7591a16bd4b8a44d8eea80aca1327ea60161dd3bad9ad22/psutil-5.6.6.tar.gz","yanked":false},{"core-metadata":{"sha256":"943b5895bc5955d1ab23e61bd0a4a6c4a4be1be625fd6bb204033b5d93574bf6"},"data-dist-info-metadata":{"sha256":"943b5895bc5955d1ab23e61bd0a4a6c4a4be1be625fd6bb204033b5d93574bf6"},"filename":"psutil-5.6.7-cp27-none-win32.whl","hashes":{"sha256":"1b1575240ca9a90b437e5a40db662acd87bbf181f6aa02f0204978737b913c6b"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":228449,"upload-time":"2019-11-26T07:25:43.864472Z","url":"https://files.pythonhosted.org/packages/13/a7/626f257d22168c954fd3ad69760c02bdec27c0648a62f6ea5060c4d40672/psutil-5.6.7-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"943b5895bc5955d1ab23e61bd0a4a6c4a4be1be625fd6bb204033b5d93574bf6"},"data-dist-info-metadata":{"sha256":"943b5895bc5955d1ab23e61bd0a4a6c4a4be1be625fd6bb204033b5d93574bf6"},"filename":"psutil-5.6.7-cp27-none-win_amd64.whl","hashes":{"sha256":"28f771129bfee9fc6b63d83a15d857663bbdcae3828e1cb926e91320a9b5b5cd"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":231969,"upload-time":"2019-11-26T07:25:50.426501Z","url":"https://files.pythonhosted.org/packages/52/44/e1e1954da522ea8640e035c8b101c116a9f8a0e94e04e108e56911064de5/psutil-5.6.7-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"15f7352602bc6ef07ec1f8cc5816d4cee0171e750a5d38106a7277f703882211"},"data-dist-info-metadata":{"sha256":"15f7352602bc6ef07ec1f8cc5816d4cee0171e750a5d38106a7277f703882211"},"filename":"psutil-5.6.7-cp35-cp35m-win32.whl","hashes":{"sha256":"21231ef1c1a89728e29b98a885b8e0a8e00d09018f6da5cdc1f43f988471a995"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":232644,"upload-time":"2019-11-26T07:25:56.230501Z","url":"https://files.pythonhosted.org/packages/d8/39/bf74da6282d9521fe3987b2d67f581b3464e635e6cb56660d0315d1bf1ed/psutil-5.6.7-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"15f7352602bc6ef07ec1f8cc5816d4cee0171e750a5d38106a7277f703882211"},"data-dist-info-metadata":{"sha256":"15f7352602bc6ef07ec1f8cc5816d4cee0171e750a5d38106a7277f703882211"},"filename":"psutil-5.6.7-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"b74b43fecce384a57094a83d2778cdfc2e2d9a6afaadd1ebecb2e75e0d34e10d"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236538,"upload-time":"2019-11-26T07:26:01.396655Z","url":"https://files.pythonhosted.org/packages/b3/84/0a899c6fac13aedfb4734413a2357a504c4f21cbf8acd7ad4caa6712d8cf/psutil-5.6.7-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"15f7352602bc6ef07ec1f8cc5816d4cee0171e750a5d38106a7277f703882211"},"data-dist-info-metadata":{"sha256":"15f7352602bc6ef07ec1f8cc5816d4cee0171e750a5d38106a7277f703882211"},"filename":"psutil-5.6.7-cp36-cp36m-win32.whl","hashes":{"sha256":"e85f727ffb21539849e6012f47b12f6dd4c44965e56591d8dec6e8bc9ab96f4a"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":232649,"upload-time":"2019-11-26T07:26:06.727306Z","url":"https://files.pythonhosted.org/packages/7c/d7/be2b607abfab4a98f04dd2155d6a7a40a666618d69c079897f09ce776a34/psutil-5.6.7-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"15f7352602bc6ef07ec1f8cc5816d4cee0171e750a5d38106a7277f703882211"},"data-dist-info-metadata":{"sha256":"15f7352602bc6ef07ec1f8cc5816d4cee0171e750a5d38106a7277f703882211"},"filename":"psutil-5.6.7-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"b560f5cd86cf8df7bcd258a851ca1ad98f0d5b8b98748e877a0aec4e9032b465"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236541,"upload-time":"2019-11-26T07:26:11.229459Z","url":"https://files.pythonhosted.org/packages/78/e6/ce8a91afd605f254342f1294790f2a77c76202386d6927eb5ff0e36e4449/psutil-5.6.7-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"15f7352602bc6ef07ec1f8cc5816d4cee0171e750a5d38106a7277f703882211"},"data-dist-info-metadata":{"sha256":"15f7352602bc6ef07ec1f8cc5816d4cee0171e750a5d38106a7277f703882211"},"filename":"psutil-5.6.7-cp37-cp37m-win32.whl","hashes":{"sha256":"094f899ac3ef72422b7e00411b4ed174e3c5a2e04c267db6643937ddba67a05b"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":232645,"upload-time":"2019-11-26T07:26:15.814275Z","url":"https://files.pythonhosted.org/packages/c4/e1/80c7840db569ad5b1b60987893e066a5536779c0d2402363cbf1230613a2/psutil-5.6.7-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"15f7352602bc6ef07ec1f8cc5816d4cee0171e750a5d38106a7277f703882211"},"data-dist-info-metadata":{"sha256":"15f7352602bc6ef07ec1f8cc5816d4cee0171e750a5d38106a7277f703882211"},"filename":"psutil-5.6.7-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"fd2e09bb593ad9bdd7429e779699d2d47c1268cbde4dda95fcd1bd17544a0217"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236539,"upload-time":"2019-11-26T07:26:20.217732Z","url":"https://files.pythonhosted.org/packages/9d/84/0a2006cc263e9f5b6dfbb2301fbcce5558f0d6d17d0c11c7c6749a45c79e/psutil-5.6.7-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"15f7352602bc6ef07ec1f8cc5816d4cee0171e750a5d38106a7277f703882211"},"data-dist-info-metadata":{"sha256":"15f7352602bc6ef07ec1f8cc5816d4cee0171e750a5d38106a7277f703882211"},"filename":"psutil-5.6.7-cp38-cp38-win32.whl","hashes":{"sha256":"70387772f84fa5c3bb6a106915a2445e20ac8f9821c5914d7cbde148f4d7ff73"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":232983,"upload-time":"2019-11-26T07:26:24.615122Z","url":"https://files.pythonhosted.org/packages/55/9d/9a6df5f730a1e2a3938fad0ccf541b30fad34706128b43ed3f965eaf7550/psutil-5.6.7-cp38-cp38-win32.whl","yanked":false},{"core-metadata":{"sha256":"15f7352602bc6ef07ec1f8cc5816d4cee0171e750a5d38106a7277f703882211"},"data-dist-info-metadata":{"sha256":"15f7352602bc6ef07ec1f8cc5816d4cee0171e750a5d38106a7277f703882211"},"filename":"psutil-5.6.7-cp38-cp38-win_amd64.whl","hashes":{"sha256":"10b7f75cc8bd676cfc6fa40cd7d5c25b3f45a0e06d43becd7c2d2871cbb5e806"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236965,"upload-time":"2019-11-26T07:26:28.827408Z","url":"https://files.pythonhosted.org/packages/8a/fa/b573850e912d6ffdad4aef3f5f705f94a64d098a83eec15d1cd3e1223f5e/psutil-5.6.7-cp38-cp38-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.6.7.tar.gz","hashes":{"sha256":"ffad8eb2ac614518bbe3c0b8eb9dffdb3a8d2e3a7d5da51c5b974fb723a5c5aa"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":448321,"upload-time":"2019-11-26T07:26:34.515073Z","url":"https://files.pythonhosted.org/packages/73/93/4f8213fbe66fc20cb904f35e6e04e20b47b85bee39845cc66a0bcf5ccdcb/psutil-5.6.7.tar.gz","yanked":false},{"core-metadata":{"sha256":"4665f6ffe5781ebc50d69efc5363ace07e50d1916f179a1cd2e51352df7eb11a"},"data-dist-info-metadata":{"sha256":"4665f6ffe5781ebc50d69efc5363ace07e50d1916f179a1cd2e51352df7eb11a"},"filename":"psutil-5.7.0-cp27-none-win32.whl","hashes":{"sha256":"298af2f14b635c3c7118fd9183843f4e73e681bb6f01e12284d4d70d48a60953"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":227641,"upload-time":"2020-02-18T18:02:31.085536Z","url":"https://files.pythonhosted.org/packages/0b/6b/f613593812c5f379c6d609bf5eca36a409812f508e13c704acd25712a73e/psutil-5.7.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"4665f6ffe5781ebc50d69efc5363ace07e50d1916f179a1cd2e51352df7eb11a"},"data-dist-info-metadata":{"sha256":"4665f6ffe5781ebc50d69efc5363ace07e50d1916f179a1cd2e51352df7eb11a"},"filename":"psutil-5.7.0-cp27-none-win_amd64.whl","hashes":{"sha256":"75e22717d4dbc7ca529ec5063000b2b294fc9a367f9c9ede1f65846c7955fd38"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":230741,"upload-time":"2020-02-18T18:02:35.453910Z","url":"https://files.pythonhosted.org/packages/79/b1/377fa0f28630d855cb6b5bfb2ee4c1bf0df3bc2603c691ceefce59a95181/psutil-5.7.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"9743d37444bb906e26ad0d76922bd72314c8385e19a68cb8a763a3e70859b867"},"data-dist-info-metadata":{"sha256":"9743d37444bb906e26ad0d76922bd72314c8385e19a68cb8a763a3e70859b867"},"filename":"psutil-5.7.0-cp35-cp35m-win32.whl","hashes":{"sha256":"f344ca230dd8e8d5eee16827596f1c22ec0876127c28e800d7ae20ed44c4b310"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":231151,"upload-time":"2020-02-18T18:02:39.410543Z","url":"https://files.pythonhosted.org/packages/74/e6/4a0ef10b1a4ca43954cd8fd9eac02cc8606f9d2a5a66859a283f5f95452b/psutil-5.7.0-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"9743d37444bb906e26ad0d76922bd72314c8385e19a68cb8a763a3e70859b867"},"data-dist-info-metadata":{"sha256":"9743d37444bb906e26ad0d76922bd72314c8385e19a68cb8a763a3e70859b867"},"filename":"psutil-5.7.0-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"e2d0c5b07c6fe5a87fa27b7855017edb0d52ee73b71e6ee368fae268605cc3f5"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":235435,"upload-time":"2020-02-18T18:02:43.014979Z","url":"https://files.pythonhosted.org/packages/65/c2/0aeb9f0cc7e4be2807aa052b3fd017e59439ed6d830b461f8ecb35b2f367/psutil-5.7.0-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"9743d37444bb906e26ad0d76922bd72314c8385e19a68cb8a763a3e70859b867"},"data-dist-info-metadata":{"sha256":"9743d37444bb906e26ad0d76922bd72314c8385e19a68cb8a763a3e70859b867"},"filename":"psutil-5.7.0-cp36-cp36m-win32.whl","hashes":{"sha256":"a02f4ac50d4a23253b68233b07e7cdb567bd025b982d5cf0ee78296990c22d9e"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":231156,"upload-time":"2020-02-18T18:02:46.432233Z","url":"https://files.pythonhosted.org/packages/c9/37/b94930ae428b2d67d505aecc5ba84c53a0b75479a8a87cd35cc9a2c6eb7e/psutil-5.7.0-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"9743d37444bb906e26ad0d76922bd72314c8385e19a68cb8a763a3e70859b867"},"data-dist-info-metadata":{"sha256":"9743d37444bb906e26ad0d76922bd72314c8385e19a68cb8a763a3e70859b867"},"filename":"psutil-5.7.0-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"1413f4158eb50e110777c4f15d7c759521703bd6beb58926f1d562da40180058"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":235439,"upload-time":"2020-02-18T18:02:49.890797Z","url":"https://files.pythonhosted.org/packages/4f/3c/205850b172a14a8b9fdc9b1e84a2c055d6b9aea226431da7685bea644f04/psutil-5.7.0-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"9743d37444bb906e26ad0d76922bd72314c8385e19a68cb8a763a3e70859b867"},"data-dist-info-metadata":{"sha256":"9743d37444bb906e26ad0d76922bd72314c8385e19a68cb8a763a3e70859b867"},"filename":"psutil-5.7.0-cp37-cp37m-win32.whl","hashes":{"sha256":"d008ddc00c6906ec80040d26dc2d3e3962109e40ad07fd8a12d0284ce5e0e4f8"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":231153,"upload-time":"2020-02-18T18:02:53.653943Z","url":"https://files.pythonhosted.org/packages/54/25/7825fefd62635f7ca556c8e0d44369ce4674aa2ca0eca50b8ae4ff49954b/psutil-5.7.0-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"9743d37444bb906e26ad0d76922bd72314c8385e19a68cb8a763a3e70859b867"},"data-dist-info-metadata":{"sha256":"9743d37444bb906e26ad0d76922bd72314c8385e19a68cb8a763a3e70859b867"},"filename":"psutil-5.7.0-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"73f35ab66c6c7a9ce82ba44b1e9b1050be2a80cd4dcc3352cc108656b115c74f"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":235439,"upload-time":"2020-02-18T18:02:57.139440Z","url":"https://files.pythonhosted.org/packages/86/f7/385040b90dd190edc28908c4a26af99b00ae37564ee5f5c4526dc1d80c27/psutil-5.7.0-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"9743d37444bb906e26ad0d76922bd72314c8385e19a68cb8a763a3e70859b867"},"data-dist-info-metadata":{"sha256":"9743d37444bb906e26ad0d76922bd72314c8385e19a68cb8a763a3e70859b867"},"filename":"psutil-5.7.0-cp38-cp38-win32.whl","hashes":{"sha256":"60b86f327c198561f101a92be1995f9ae0399736b6eced8f24af41ec64fb88d4"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":231789,"upload-time":"2020-02-18T18:03:00.808478Z","url":"https://files.pythonhosted.org/packages/63/d5/f34a9433a0299d944605fb5a970306a89e076f5412164179dc59ebf70fa9/psutil-5.7.0-cp38-cp38-win32.whl","yanked":false},{"core-metadata":{"sha256":"9743d37444bb906e26ad0d76922bd72314c8385e19a68cb8a763a3e70859b867"},"data-dist-info-metadata":{"sha256":"9743d37444bb906e26ad0d76922bd72314c8385e19a68cb8a763a3e70859b867"},"filename":"psutil-5.7.0-cp38-cp38-win_amd64.whl","hashes":{"sha256":"d84029b190c8a66a946e28b4d3934d2ca1528ec94764b180f7d6ea57b0e75e26"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":235952,"upload-time":"2020-02-18T18:03:04.130414Z","url":"https://files.pythonhosted.org/packages/86/fe/9f1d1f8c1c8138d42fc0e7c06ca5004e01f38e86e61342374d8e0fa919e4/psutil-5.7.0-cp38-cp38-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.7.0.tar.gz","hashes":{"sha256":"685ec16ca14d079455892f25bd124df26ff9137664af445563c1bd36629b5e0e"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":449628,"upload-time":"2020-02-18T18:03:07.566242Z","url":"https://files.pythonhosted.org/packages/c4/b8/3512f0e93e0db23a71d82485ba256071ebef99b227351f0f5540f744af41/psutil-5.7.0.tar.gz","yanked":false},{"core-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"data-dist-info-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"filename":"psutil-5.7.1-cp27-cp27m-macosx_10_9_x86_64.whl","hashes":{"sha256":"2dfb7b5638ffaa33602a86b39cca60cded2324dabbe2617b1b5e65250e448769"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":233646,"upload-time":"2020-07-15T11:14:11.635539Z","url":"https://files.pythonhosted.org/packages/79/e4/cbaa3ecc458c2dd8da64073de983473543b8b6ef4ca21159cea9069d53dd/psutil-5.7.1-cp27-cp27m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"data-dist-info-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"filename":"psutil-5.7.1-cp35-cp35m-macosx_10_9_x86_64.whl","hashes":{"sha256":"36c5e6882caf3d385c6c3a0d2f3b302b4cc337c808ea589d9a8c563b545beb8b"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":233862,"upload-time":"2020-07-15T11:14:27.607855Z","url":"https://files.pythonhosted.org/packages/4d/d9/48c3d16c1dfbbf528bd69254b5a604c9f6860f12169b1b73a5005723c6bf/psutil-5.7.1-cp35-cp35m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"data-dist-info-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"filename":"psutil-5.7.1-cp35-cp35m-win32.whl","hashes":{"sha256":"3c5ffd00bc1ee809350dca97613985d387a7e13dff61d62fc1bdf4dc10892ddd"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238370,"upload-time":"2020-07-15T11:14:34.746888Z","url":"https://files.pythonhosted.org/packages/d0/e2/d4cdadda6a9fba79026ab628fc2b4da5e2e48dcfc6beada0a39363732ba1/psutil-5.7.1-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"data-dist-info-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"filename":"psutil-5.7.1-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"4975c33aebe7de191d745ee3c545e907edd14d65c850a0b185c05024aa77cbcd"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242862,"upload-time":"2020-07-15T11:14:37.084757Z","url":"https://files.pythonhosted.org/packages/a0/6b/cdb41805a6bb62c051cfbb1b65a9cb40767e0144b3d40fdd7082d8271701/psutil-5.7.1-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"data-dist-info-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"filename":"psutil-5.7.1-cp36-cp36m-macosx_10_9_x86_64.whl","hashes":{"sha256":"436a6e99098eba14b54a149f921c9d4e1df729f02645876af0c828396d36c46a"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":233861,"upload-time":"2020-07-15T11:14:39.242780Z","url":"https://files.pythonhosted.org/packages/1f/fb/097aeed40c361225cb69d6d04202421f2c172d7e42753130d1b619f68956/psutil-5.7.1-cp36-cp36m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"data-dist-info-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"filename":"psutil-5.7.1-cp36-cp36m-win32.whl","hashes":{"sha256":"630ceda48c16b24ffd981fe06ae1a43684af1a3a837d6a3496a1be3dd3c7d332"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238378,"upload-time":"2020-07-15T11:14:45.606142Z","url":"https://files.pythonhosted.org/packages/ae/49/cba9353fd9946eac95031c85763daaf7904d3c3e8b0b4f2801199586b413/psutil-5.7.1-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"data-dist-info-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"filename":"psutil-5.7.1-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"d3bb7f65199595a72a3ec53e4d05c159857ab832fadaae9d85e68db467d2d191"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242864,"upload-time":"2020-07-15T11:14:47.861971Z","url":"https://files.pythonhosted.org/packages/7b/38/f27fc6a30f81be1ee657bd4c355c2dc03a5fbb49f304a37c79c0bed05821/psutil-5.7.1-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"data-dist-info-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"filename":"psutil-5.7.1-cp37-cp37m-macosx_10_9_x86_64.whl","hashes":{"sha256":"66d085317599684f70d995dd4a770894f518fb34d027d7f742b579bf47732858"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":233863,"upload-time":"2020-07-15T11:14:49.991128Z","url":"https://files.pythonhosted.org/packages/23/cb/410a516385c8cd69f090f98c8014636c51d124c96e4d6ab51e1bb2d04232/psutil-5.7.1-cp37-cp37m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"data-dist-info-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"filename":"psutil-5.7.1-cp37-cp37m-win32.whl","hashes":{"sha256":"fb442b912fe28d80e0f966adcc3df4e394fbb7ef7575ae21fd171aeb06c8b0df"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238369,"upload-time":"2020-07-15T11:14:57.383010Z","url":"https://files.pythonhosted.org/packages/59/44/e9cfa470dd2790b5475ceb590949842a5f2feb52445e898576b721033f04/psutil-5.7.1-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"data-dist-info-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"filename":"psutil-5.7.1-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"425d6c95ca3ece7ff4da7e67af2954b8eb56b0f15743b237dc84ad975f51c2a4"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242865,"upload-time":"2020-07-15T11:14:59.839217Z","url":"https://files.pythonhosted.org/packages/06/76/b4607e0eaf36369ad86f7ac73bde19aeaf32c82fb22675cb8f8dd975c692/psutil-5.7.1-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"data-dist-info-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"filename":"psutil-5.7.1-cp38-cp38-macosx_10_9_x86_64.whl","hashes":{"sha256":"f2817a763c33c19fdefbb832c790bc85b3de90b51fb69dae43097a9885be0332"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":234129,"upload-time":"2020-07-15T11:15:01.982258Z","url":"https://files.pythonhosted.org/packages/ba/f7/64bf7fd7a12a40c50408b7d90cdf3addc28071e5463af1dbb7f3884a32d2/psutil-5.7.1-cp38-cp38-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"data-dist-info-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"filename":"psutil-5.7.1-cp38-cp38-win32.whl","hashes":{"sha256":"3cf43d2265ee03fcf70f0f574487ed19435c92a330e15a3e773144811c1275f0"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":239004,"upload-time":"2020-07-15T11:15:08.125783Z","url":"https://files.pythonhosted.org/packages/7e/64/c3bd24d53f6056ded095e8d147c0ca269bb6d858aea903561bd660d67035/psutil-5.7.1-cp38-cp38-win32.whl","yanked":false},{"core-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"data-dist-info-metadata":{"sha256":"5cfd93088b3a9182d5543595d138ec1f3a9f0fee62e4e1dcbc3332900a0871ea"},"filename":"psutil-5.7.1-cp38-cp38-win_amd64.whl","hashes":{"sha256":"006b720a67881037c8b02b1de012a39a2f007bd2b1b244b58fabef8eff0ad6d2"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":243418,"upload-time":"2020-07-15T11:15:10.044651Z","url":"https://files.pythonhosted.org/packages/ff/5a/1e990cf86f47721225143ed4a903a226685fa1ba0b43500f52d71500b1be/psutil-5.7.1-cp38-cp38-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"621e2f3b573aa563717ee39c3f052b4cd3cda5e1f25e5973fa77e727433c6ce1"},"data-dist-info-metadata":{"sha256":"621e2f3b573aa563717ee39c3f052b4cd3cda5e1f25e5973fa77e727433c6ce1"},"filename":"psutil-5.7.1-pp27-pypy_73-macosx_10_9_x86_64.whl","hashes":{"sha256":"0c9187ec0c314a128362c3409afea2b80c6d6d2c2cb1d661fe20631a2ff8ad77"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":232076,"upload-time":"2020-07-15T11:15:12.408818Z","url":"https://files.pythonhosted.org/packages/80/fd/d91ff7582513d093097678eedd141a4879698da26e6163fbae16905aa75b/psutil-5.7.1-pp27-pypy_73-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.7.1.tar.gz","hashes":{"sha256":"4ef6845b35e152e6937d4f28388c2440ca89a0089ced0a30a116fa3ceefdfa3a"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":460148,"upload-time":"2020-07-15T11:15:21.985314Z","url":"https://files.pythonhosted.org/packages/2c/5c/cb95a715fb635e1ca858ffb8c50a523a16e2dc06aa3e207ab73cb93516af/psutil-5.7.1.tar.gz","yanked":false},{"core-metadata":{"sha256":"91f2a99bfe758ed3b22b8fc235d7bfa5b885bb5cdd7bd1a38cccff1e9756d183"},"data-dist-info-metadata":{"sha256":"91f2a99bfe758ed3b22b8fc235d7bfa5b885bb5cdd7bd1a38cccff1e9756d183"},"filename":"psutil-5.7.2-cp27-none-win32.whl","hashes":{"sha256":"f2018461733b23f308c298653c8903d32aaad7873d25e1d228765e91ae42c3f2"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":234782,"upload-time":"2020-07-15T13:19:05.321836Z","url":"https://files.pythonhosted.org/packages/d0/da/d7da0365f690e7555f6dda34bcb5bde10266379c9a23ee6a0735c3a7fdfd/psutil-5.7.2-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"91f2a99bfe758ed3b22b8fc235d7bfa5b885bb5cdd7bd1a38cccff1e9756d183"},"data-dist-info-metadata":{"sha256":"91f2a99bfe758ed3b22b8fc235d7bfa5b885bb5cdd7bd1a38cccff1e9756d183"},"filename":"psutil-5.7.2-cp27-none-win_amd64.whl","hashes":{"sha256":"66c18ca7680a31bf16ee22b1d21b6397869dda8059dbdb57d9f27efa6615f195"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238050,"upload-time":"2020-07-15T13:19:07.922423Z","url":"https://files.pythonhosted.org/packages/7d/9d/30a053a06d598cee4bdbc6ba69df44ced9e6d2ebb16e2de401a2a3bc6d63/psutil-5.7.2-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"4ece4267ea2940759bedb71b133199d6c5b33ccc17f4c3060d08d56f5697ca45"},"data-dist-info-metadata":{"sha256":"4ece4267ea2940759bedb71b133199d6c5b33ccc17f4c3060d08d56f5697ca45"},"filename":"psutil-5.7.2-cp35-cp35m-win32.whl","hashes":{"sha256":"5e9d0f26d4194479a13d5f4b3798260c20cecf9ac9a461e718eb59ea520a360c"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238373,"upload-time":"2020-07-15T13:19:10.146779Z","url":"https://files.pythonhosted.org/packages/df/27/e5cf14b0894b4f06c23dc4f58288c60b17e71d8bef9af463f0b32ee46773/psutil-5.7.2-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"4ece4267ea2940759bedb71b133199d6c5b33ccc17f4c3060d08d56f5697ca45"},"data-dist-info-metadata":{"sha256":"4ece4267ea2940759bedb71b133199d6c5b33ccc17f4c3060d08d56f5697ca45"},"filename":"psutil-5.7.2-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"4080869ed93cce662905b029a1770fe89c98787e543fa7347f075ade761b19d6"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242862,"upload-time":"2020-07-15T13:19:12.906781Z","url":"https://files.pythonhosted.org/packages/0a/4c/d31d58992314664e69bda6d575c1fd47b86ed5d67e00e300fc909040a9aa/psutil-5.7.2-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"4ece4267ea2940759bedb71b133199d6c5b33ccc17f4c3060d08d56f5697ca45"},"data-dist-info-metadata":{"sha256":"4ece4267ea2940759bedb71b133199d6c5b33ccc17f4c3060d08d56f5697ca45"},"filename":"psutil-5.7.2-cp36-cp36m-win32.whl","hashes":{"sha256":"d8a82162f23c53b8525cf5f14a355f5d1eea86fa8edde27287dd3a98399e4fdf"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238380,"upload-time":"2020-07-15T13:19:15.247084Z","url":"https://files.pythonhosted.org/packages/57/c5/0aa3b1513b914a417db7ee149b60579a139111f81f79f5f1d38ae440cebf/psutil-5.7.2-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"4ece4267ea2940759bedb71b133199d6c5b33ccc17f4c3060d08d56f5697ca45"},"data-dist-info-metadata":{"sha256":"4ece4267ea2940759bedb71b133199d6c5b33ccc17f4c3060d08d56f5697ca45"},"filename":"psutil-5.7.2-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"0ee3c36428f160d2d8fce3c583a0353e848abb7de9732c50cf3356dd49ad63f8"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242866,"upload-time":"2020-07-15T13:19:17.782781Z","url":"https://files.pythonhosted.org/packages/22/f8/7be159475303a508347efc82c0d5858b1786fe73fc2a6b21d82891791920/psutil-5.7.2-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"4ece4267ea2940759bedb71b133199d6c5b33ccc17f4c3060d08d56f5697ca45"},"data-dist-info-metadata":{"sha256":"4ece4267ea2940759bedb71b133199d6c5b33ccc17f4c3060d08d56f5697ca45"},"filename":"psutil-5.7.2-cp37-cp37m-win32.whl","hashes":{"sha256":"ff1977ba1a5f71f89166d5145c3da1cea89a0fdb044075a12c720ee9123ec818"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238375,"upload-time":"2020-07-15T13:19:20.591814Z","url":"https://files.pythonhosted.org/packages/56/de/6f07749f275d0ba7f9b985cd6a4526e2fa47ad63b5179948c6650117f7d9/psutil-5.7.2-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"4ece4267ea2940759bedb71b133199d6c5b33ccc17f4c3060d08d56f5697ca45"},"data-dist-info-metadata":{"sha256":"4ece4267ea2940759bedb71b133199d6c5b33ccc17f4c3060d08d56f5697ca45"},"filename":"psutil-5.7.2-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"a5b120bb3c0c71dfe27551f9da2f3209a8257a178ed6c628a819037a8df487f1"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242863,"upload-time":"2020-07-15T13:19:23.022907Z","url":"https://files.pythonhosted.org/packages/f8/9b/1d7df5e1747e047abef4ec877d895b642f3a796ab8bd2e0f682516740dfe/psutil-5.7.2-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"4ece4267ea2940759bedb71b133199d6c5b33ccc17f4c3060d08d56f5697ca45"},"data-dist-info-metadata":{"sha256":"4ece4267ea2940759bedb71b133199d6c5b33ccc17f4c3060d08d56f5697ca45"},"filename":"psutil-5.7.2-cp38-cp38-win32.whl","hashes":{"sha256":"10512b46c95b02842c225f58fa00385c08fa00c68bac7da2d9a58ebe2c517498"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":239003,"upload-time":"2020-07-15T13:19:25.770960Z","url":"https://files.pythonhosted.org/packages/da/d6/f66bbdbc8831a5cc78ba0e9bf69d924e68eac1a7b4191de93cf4e3643c54/psutil-5.7.2-cp38-cp38-win32.whl","yanked":false},{"core-metadata":{"sha256":"4ece4267ea2940759bedb71b133199d6c5b33ccc17f4c3060d08d56f5697ca45"},"data-dist-info-metadata":{"sha256":"4ece4267ea2940759bedb71b133199d6c5b33ccc17f4c3060d08d56f5697ca45"},"filename":"psutil-5.7.2-cp38-cp38-win_amd64.whl","hashes":{"sha256":"68d36986ded5dac7c2dcd42f2682af1db80d4bce3faa126a6145c1637e1b559f"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":243419,"upload-time":"2020-07-15T13:19:28.240707Z","url":"https://files.pythonhosted.org/packages/6c/e6/f963547a36a96f74244cbe5e4046a02f140e3b7cbc5e5176035b38e2deb2/psutil-5.7.2-cp38-cp38-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.7.2.tar.gz","hashes":{"sha256":"90990af1c3c67195c44c9a889184f84f5b2320dce3ee3acbd054e3ba0b4a7beb"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":460198,"upload-time":"2020-07-15T13:19:30.438785Z","url":"https://files.pythonhosted.org/packages/aa/3e/d18f2c04cf2b528e18515999b0c8e698c136db78f62df34eee89cee205f1/psutil-5.7.2.tar.gz","yanked":false},{"core-metadata":{"sha256":"6032661358cee4f792940fdac32ebc65bab2fa2463828785b8a8f0fb6c2c210b"},"data-dist-info-metadata":{"sha256":"6032661358cee4f792940fdac32ebc65bab2fa2463828785b8a8f0fb6c2c210b"},"filename":"psutil-5.7.3-cp27-none-win32.whl","hashes":{"sha256":"1cd6a0c9fb35ece2ccf2d1dd733c1e165b342604c67454fd56a4c12e0a106787"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":235225,"upload-time":"2020-10-24T14:00:54.139761Z","url":"https://files.pythonhosted.org/packages/1d/a2/b732590561ef9d7dbc078ed0e2635e282115604a478911fef97ddaa3ad43/psutil-5.7.3-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"6032661358cee4f792940fdac32ebc65bab2fa2463828785b8a8f0fb6c2c210b"},"data-dist-info-metadata":{"sha256":"6032661358cee4f792940fdac32ebc65bab2fa2463828785b8a8f0fb6c2c210b"},"filename":"psutil-5.7.3-cp27-none-win_amd64.whl","hashes":{"sha256":"e02c31b2990dcd2431f4524b93491941df39f99619b0d312dfe1d4d530b08b4b"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238499,"upload-time":"2020-10-24T14:00:57.724543Z","url":"https://files.pythonhosted.org/packages/b4/4c/c14a9485957b00c20f70e208a03663e81ddc8dafdf5137fee2d50aa1ee5e/psutil-5.7.3-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"9f3f41a734ced0a9775d53996e940b0d2411f0f5105b0cdcac5e8e3ffd8da0e9"},"data-dist-info-metadata":{"sha256":"9f3f41a734ced0a9775d53996e940b0d2411f0f5105b0cdcac5e8e3ffd8da0e9"},"filename":"psutil-5.7.3-cp35-cp35m-win32.whl","hashes":{"sha256":"56c85120fa173a5d2ad1d15a0c6e0ae62b388bfb956bb036ac231fbdaf9e4c22"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238813,"upload-time":"2020-10-24T15:02:21.678008Z","url":"https://files.pythonhosted.org/packages/e0/ba/a7c1096470e11c449019690ee9e7fd3adca1a4b9cfa6e5a13b60db3187b4/psutil-5.7.3-cp35-cp35m-win32.whl","yanked":false},{"core-metadata":{"sha256":"9f3f41a734ced0a9775d53996e940b0d2411f0f5105b0cdcac5e8e3ffd8da0e9"},"data-dist-info-metadata":{"sha256":"9f3f41a734ced0a9775d53996e940b0d2411f0f5105b0cdcac5e8e3ffd8da0e9"},"filename":"psutil-5.7.3-cp35-cp35m-win_amd64.whl","hashes":{"sha256":"fa38ac15dbf161ab1e941ff4ce39abd64b53fec5ddf60c23290daed2bc7d1157"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":243298,"upload-time":"2020-10-24T15:02:24.823245Z","url":"https://files.pythonhosted.org/packages/3f/7c/98aada1208462c841788712383f4288b3c31e45504570a818c0c303d78e7/psutil-5.7.3-cp35-cp35m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"9f3f41a734ced0a9775d53996e940b0d2411f0f5105b0cdcac5e8e3ffd8da0e9"},"data-dist-info-metadata":{"sha256":"9f3f41a734ced0a9775d53996e940b0d2411f0f5105b0cdcac5e8e3ffd8da0e9"},"filename":"psutil-5.7.3-cp36-cp36m-win32.whl","hashes":{"sha256":"01bc82813fbc3ea304914581954979e637bcc7084e59ac904d870d6eb8bb2bc7"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238821,"upload-time":"2020-10-24T15:02:27.844759Z","url":"https://files.pythonhosted.org/packages/44/a8/ebfcbb4967e74a27049ea6e13b3027ae05c0cb73d1a2b71c2f0519c6d5f2/psutil-5.7.3-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"9f3f41a734ced0a9775d53996e940b0d2411f0f5105b0cdcac5e8e3ffd8da0e9"},"data-dist-info-metadata":{"sha256":"9f3f41a734ced0a9775d53996e940b0d2411f0f5105b0cdcac5e8e3ffd8da0e9"},"filename":"psutil-5.7.3-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"6a3e1fd2800ca45083d976b5478a2402dd62afdfb719b30ca46cd28bb25a2eb4"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":243300,"upload-time":"2020-10-24T15:02:31.552663Z","url":"https://files.pythonhosted.org/packages/9a/c3/3b0023b46fc038eff02fbb69a0e6e50d15a7dce25e717d8469e8eaa837a7/psutil-5.7.3-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"9f3f41a734ced0a9775d53996e940b0d2411f0f5105b0cdcac5e8e3ffd8da0e9"},"data-dist-info-metadata":{"sha256":"9f3f41a734ced0a9775d53996e940b0d2411f0f5105b0cdcac5e8e3ffd8da0e9"},"filename":"psutil-5.7.3-cp37-cp37m-win32.whl","hashes":{"sha256":"fbcac492cb082fa38d88587d75feb90785d05d7e12d4565cbf1ecc727aff71b7"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238814,"upload-time":"2020-10-24T15:02:34.703606Z","url":"https://files.pythonhosted.org/packages/e1/f0/d4f58ddf077d970440b82b92e909e8e9b2f50e39a2dc2aa716b1e2fde5ef/psutil-5.7.3-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"9f3f41a734ced0a9775d53996e940b0d2411f0f5105b0cdcac5e8e3ffd8da0e9"},"data-dist-info-metadata":{"sha256":"9f3f41a734ced0a9775d53996e940b0d2411f0f5105b0cdcac5e8e3ffd8da0e9"},"filename":"psutil-5.7.3-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"5d9106ff5ec2712e2f659ebbd112967f44e7d33f40ba40530c485cc5904360b8"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":243302,"upload-time":"2020-10-24T15:02:38.016752Z","url":"https://files.pythonhosted.org/packages/f1/a0/094a6e32185bd1288a4681d91ebe362d5b41aa64413bbbd96ed547051f17/psutil-5.7.3-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"9f3f41a734ced0a9775d53996e940b0d2411f0f5105b0cdcac5e8e3ffd8da0e9"},"data-dist-info-metadata":{"sha256":"9f3f41a734ced0a9775d53996e940b0d2411f0f5105b0cdcac5e8e3ffd8da0e9"},"filename":"psutil-5.7.3-cp38-cp38-win32.whl","hashes":{"sha256":"ade6af32eb80a536eff162d799e31b7ef92ddcda707c27bbd077238065018df4"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":239426,"upload-time":"2020-10-24T15:02:41.250786Z","url":"https://files.pythonhosted.org/packages/bd/95/394485321a128f5ddb23f0a559f940309280f57bd6117580868fb2d5a246/psutil-5.7.3-cp38-cp38-win32.whl","yanked":false},{"core-metadata":{"sha256":"9f3f41a734ced0a9775d53996e940b0d2411f0f5105b0cdcac5e8e3ffd8da0e9"},"data-dist-info-metadata":{"sha256":"9f3f41a734ced0a9775d53996e940b0d2411f0f5105b0cdcac5e8e3ffd8da0e9"},"filename":"psutil-5.7.3-cp38-cp38-win_amd64.whl","hashes":{"sha256":"2cb55ef9591b03ef0104bedf67cc4edb38a3edf015cf8cf24007b99cb8497542"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":243849,"upload-time":"2020-10-24T15:02:44.162783Z","url":"https://files.pythonhosted.org/packages/df/64/8d7b55ac87e67398ffc260d43a5fb327f1e230b09758b7d8caaecf917dd6/psutil-5.7.3-cp38-cp38-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.7.3.tar.gz","hashes":{"sha256":"af73f7bcebdc538eda9cc81d19db1db7bf26f103f91081d780bbacfcb620dee2"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":465556,"upload-time":"2020-10-24T14:02:15.604830Z","url":"https://files.pythonhosted.org/packages/33/e0/82d459af36bda999f82c7ea86c67610591cf5556168f48fd6509e5fa154d/psutil-5.7.3.tar.gz","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp27-cp27m-macosx_10_9_x86_64.whl","hashes":{"sha256":"0066a82f7b1b37d334e68697faba68e5ad5e858279fd6351c8ca6024e8d6ba64"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":235772,"upload-time":"2020-12-19T01:19:27.017908Z","url":"https://files.pythonhosted.org/packages/f5/7f/a2559a514bdeb2a33e4bf3dc3d2bb17d5acded718893869a82536130cfb3/psutil-5.8.0-cp27-cp27m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp27-cp27m-manylinux2010_i686.whl","hashes":{"sha256":"0ae6f386d8d297177fd288be6e8d1afc05966878704dad9847719650e44fc49c"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":284461,"upload-time":"2020-12-19T01:19:30.158384Z","url":"https://files.pythonhosted.org/packages/19/2c/9f1bad783faee4e9704868f381913e68dbb69f0de3fcdc71ee7071c47847/psutil-5.8.0-cp27-cp27m-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp27-cp27m-manylinux2010_x86_64.whl","hashes":{"sha256":"12d844996d6c2b1d3881cfa6fa201fd635971869a9da945cf6756105af73d2df"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":287768,"upload-time":"2020-12-19T01:19:33.394809Z","url":"https://files.pythonhosted.org/packages/82/0a/eddb9a51ba5055cc7c242da07c1643a6b146070740c5eb5540277a0f01f4/psutil-5.8.0-cp27-cp27m-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp27-cp27mu-manylinux2010_i686.whl","hashes":{"sha256":"02b8292609b1f7fcb34173b25e48d0da8667bc85f81d7476584d889c6e0f2131"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":284476,"upload-time":"2020-12-19T01:19:36.479406Z","url":"https://files.pythonhosted.org/packages/15/28/47c28171fd7eeb83df74f78ccac090211f4a49408f376eb8e78a7bb47dc0/psutil-5.8.0-cp27-cp27mu-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp27-cp27mu-manylinux2010_x86_64.whl","hashes":{"sha256":"6ffe81843131ee0ffa02c317186ed1e759a145267d54fdef1bc4ea5f5931ab60"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":287754,"upload-time":"2020-12-19T01:19:39.219693Z","url":"https://files.pythonhosted.org/packages/26/ef/461e9eec56fba7fa66692c4af00cbd6547b788a7ca818d9b8b5f1951f228/psutil-5.8.0-cp27-cp27mu-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"91c5c4b49dc8b3365ce65ca99e7cee1cf8d862d4379fcc384b1711d4600dd869"},"data-dist-info-metadata":{"sha256":"91c5c4b49dc8b3365ce65ca99e7cee1cf8d862d4379fcc384b1711d4600dd869"},"filename":"psutil-5.8.0-cp27-none-win32.whl","hashes":{"sha256":"ea313bb02e5e25224e518e4352af4bf5e062755160f77e4b1767dd5ccb65f876"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236561,"upload-time":"2020-12-19T01:19:41.916204Z","url":"https://files.pythonhosted.org/packages/a8/b3/6a21c5b7e4f600bd6eaaecd4a5e76230fa34876e48cbc87b2cef0ab91c0a/psutil-5.8.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"91c5c4b49dc8b3365ce65ca99e7cee1cf8d862d4379fcc384b1711d4600dd869"},"data-dist-info-metadata":{"sha256":"91c5c4b49dc8b3365ce65ca99e7cee1cf8d862d4379fcc384b1711d4600dd869"},"filename":"psutil-5.8.0-cp27-none-win_amd64.whl","hashes":{"sha256":"5da29e394bdedd9144c7331192e20c1f79283fb03b06e6abd3a8ae45ffecee65"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":239900,"upload-time":"2020-12-19T01:19:45.046259Z","url":"https://files.pythonhosted.org/packages/b2/3d/01ef1f4bf71413078bf2ce2aae04d47bc132cfede58738183a9de41aa122/psutil-5.8.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp36-cp36m-macosx_10_9_x86_64.whl","hashes":{"sha256":"74fb2557d1430fff18ff0d72613c5ca30c45cdbfcddd6a5773e9fc1fe9364be8"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236013,"upload-time":"2020-12-19T01:19:47.734642Z","url":"https://files.pythonhosted.org/packages/30/81/37ebe0ba2840b76681072e786bae3319cade8a6861029d0ae885c274fa0b/psutil-5.8.0-cp36-cp36m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp36-cp36m-manylinux2010_i686.whl","hashes":{"sha256":"74f2d0be88db96ada78756cb3a3e1b107ce8ab79f65aa885f76d7664e56928f6"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":289247,"upload-time":"2020-12-19T01:19:50.429849Z","url":"https://files.pythonhosted.org/packages/2e/7c/13a6c3f068aa39ffafd99ae159c1a345521e7dd0074ccadb917e5670dbdc/psutil-5.8.0-cp36-cp36m-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp36-cp36m-manylinux2010_x86_64.whl","hashes":{"sha256":"99de3e8739258b3c3e8669cb9757c9a861b2a25ad0955f8e53ac662d66de61ac"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":291875,"upload-time":"2020-12-19T01:19:53.450222Z","url":"https://files.pythonhosted.org/packages/da/82/56cd16a4c5f53e3e5dd7b2c30d5c803e124f218ebb644ca9c30bc907eadd/psutil-5.8.0-cp36-cp36m-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp36-cp36m-win32.whl","hashes":{"sha256":"36b3b6c9e2a34b7d7fbae330a85bf72c30b1c827a4366a07443fc4b6270449e2"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":240337,"upload-time":"2020-12-19T01:19:56.748668Z","url":"https://files.pythonhosted.org/packages/19/29/f7a38ee30083f2caa14cc77a6d34c4d5cfd1a69641e87bf1b3d6ba90d0ba/psutil-5.8.0-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"52de075468cd394ac98c66f9ca33b2f54ae1d9bff1ef6b67a212ee8f639ec06d"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":244835,"upload-time":"2020-12-19T01:19:59.102544Z","url":"https://files.pythonhosted.org/packages/44/ed/49d75a29007727d44937ed4d233f116be346bc4657a83b5a9e2f423bca57/psutil-5.8.0-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp37-cp37m-macosx_10_9_x86_64.whl","hashes":{"sha256":"c6a5fd10ce6b6344e616cf01cc5b849fa8103fbb5ba507b6b2dee4c11e84c935"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236013,"upload-time":"2020-12-19T01:20:02.260367Z","url":"https://files.pythonhosted.org/packages/fe/19/83ab423a7b69cafe4078dea751acdff7377e4b59c71e3718125ba3c341f9/psutil-5.8.0-cp37-cp37m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp37-cp37m-manylinux2010_i686.whl","hashes":{"sha256":"61f05864b42fedc0771d6d8e49c35f07efd209ade09a5afe6a5059e7bb7bf83d"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":290299,"upload-time":"2020-12-19T01:20:04.653817Z","url":"https://files.pythonhosted.org/packages/cc/5f/2a1967092086acc647962168d0e6fd1c22e14a973f03e3ffb1e2f0da5de9/psutil-5.8.0-cp37-cp37m-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp37-cp37m-manylinux2010_x86_64.whl","hashes":{"sha256":"0dd4465a039d343925cdc29023bb6960ccf4e74a65ad53e768403746a9207023"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":296329,"upload-time":"2020-12-19T01:20:07.174230Z","url":"https://files.pythonhosted.org/packages/84/da/f7efdcf012b51506938553dbe302aecc22f3f43abd5cffa8320e8e0588d5/psutil-5.8.0-cp37-cp37m-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp37-cp37m-win32.whl","hashes":{"sha256":"1bff0d07e76114ec24ee32e7f7f8d0c4b0514b3fae93e3d2aaafd65d22502394"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":240337,"upload-time":"2020-12-19T01:20:10.138302Z","url":"https://files.pythonhosted.org/packages/18/c9/1db6aa0d28831f60408a6aab9d108c2edbd5a9ed11e5957a91d9d8023898/psutil-5.8.0-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"fcc01e900c1d7bee2a37e5d6e4f9194760a93597c97fee89c4ae51701de03563"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":244834,"upload-time":"2020-12-19T01:20:12.840200Z","url":"https://files.pythonhosted.org/packages/71/ce/35107e81e7eae55c847313f872d4258a71d2640fa04f57c5520fc81473ce/psutil-5.8.0-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp38-cp38-macosx_10_9_x86_64.whl","hashes":{"sha256":"6223d07a1ae93f86451d0198a0c361032c4c93ebd4bf6d25e2fb3edfad9571ef"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236297,"upload-time":"2020-12-19T01:20:15.489742Z","url":"https://files.pythonhosted.org/packages/10/d6/c5c19e40bb05e2cb5f053f480dfe47e9543a8322f1a5985d7352bf689611/psutil-5.8.0-cp38-cp38-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp38-cp38-manylinux2010_i686.whl","hashes":{"sha256":"d225cd8319aa1d3c85bf195c4e07d17d3cd68636b8fc97e6cf198f782f99af28"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":293809,"upload-time":"2020-12-19T01:20:18.539919Z","url":"https://files.pythonhosted.org/packages/e9/d6/7d0bcf272923f6b3433e22effd31860b63ab580d65fb2d8f5cb443a9e6fc/psutil-5.8.0-cp38-cp38-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp38-cp38-manylinux2010_x86_64.whl","hashes":{"sha256":"28ff7c95293ae74bf1ca1a79e8805fcde005c18a122ca983abf676ea3466362b"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":296040,"upload-time":"2020-12-19T01:20:21.423284Z","url":"https://files.pythonhosted.org/packages/3b/c2/78109a12da9febb2f965abf29da6f81b0a3f2b89a7b59d88b759e68dc6db/psutil-5.8.0-cp38-cp38-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp38-cp38-win32.whl","hashes":{"sha256":"ce8b867423291cb65cfc6d9c4955ee9bfc1e21fe03bb50e177f2b957f1c2469d"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":240950,"upload-time":"2020-12-19T01:20:24.359373Z","url":"https://files.pythonhosted.org/packages/87/be/6511e1341c203608fe2553249216c40b92cd8a72d8b35fa3c1decee9a616/psutil-5.8.0-cp38-cp38-win32.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp38-cp38-win_amd64.whl","hashes":{"sha256":"90f31c34d25b1b3ed6c40cdd34ff122b1887a825297c017e4cbd6796dd8b672d"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":245386,"upload-time":"2020-12-19T01:20:26.799194Z","url":"https://files.pythonhosted.org/packages/8e/5c/c4b32c2024daeac35e126b90a1ff7a0209ef8b32675d1d50e55d58e78c81/psutil-5.8.0-cp38-cp38-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp39-cp39-macosx_10_9_x86_64.whl","hashes":{"sha256":"6323d5d845c2785efb20aded4726636546b26d3b577aded22492908f7c1bdda7"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":236274,"upload-time":"2020-12-19T01:20:29.615503Z","url":"https://files.pythonhosted.org/packages/12/80/8d09c345f19af2b29a309f8f9284e3ba1ae1ebd9438419080c14630f743a/psutil-5.8.0-cp39-cp39-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp39-cp39-manylinux2010_i686.whl","hashes":{"sha256":"245b5509968ac0bd179287d91210cd3f37add77dad385ef238b275bad35fa1c4"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":291032,"upload-time":"2020-12-19T01:20:32.422979Z","url":"https://files.pythonhosted.org/packages/b6/2f/118e23a8f4e59d2c4ffe03a921cc72f364966e25548dc6c5a3011a334dc5/psutil-5.8.0-cp39-cp39-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp39-cp39-manylinux2010_x86_64.whl","hashes":{"sha256":"90d4091c2d30ddd0a03e0b97e6a33a48628469b99585e2ad6bf21f17423b112b"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":293491,"upload-time":"2020-12-19T01:20:34.915141Z","url":"https://files.pythonhosted.org/packages/91/4d/033cc02ae3a47197d0ced818814e4bb8d9d29ebed4f1eb55badedec160f7/psutil-5.8.0-cp39-cp39-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp39-cp39-win32.whl","hashes":{"sha256":"ea372bcc129394485824ae3e3ddabe67dc0b118d262c568b4d2602a7070afdb0"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":241508,"upload-time":"2020-12-19T01:20:37.527554Z","url":"https://files.pythonhosted.org/packages/a7/13/7285b74e061da21dfc4f15c8307eb2da1d2137367502d6598f03f4a5b5e7/psutil-5.8.0-cp39-cp39-win32.whl","yanked":false},{"core-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"data-dist-info-metadata":{"sha256":"069863fda59a3742371c9f60ad268b0b04c760a595c7d1a1de557d590bf205e6"},"filename":"psutil-5.8.0-cp39-cp39-win_amd64.whl","hashes":{"sha256":"f4634b033faf0d968bb9220dd1c793b897ab7f1189956e1aa9eae752527127d3"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":246139,"upload-time":"2020-12-19T01:20:39.890196Z","url":"https://files.pythonhosted.org/packages/21/71/33cb528381c443df1ee25cbb451da975421bddb5099b11e7f2eb3fc90d6d/psutil-5.8.0-cp39-cp39-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.8.0.tar.gz","hashes":{"sha256":"0c9ccb99ab76025f2f0bbecf341d4656e9c1351db8cc8a03ccd62e318ab4b5c6"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":470886,"upload-time":"2020-12-19T01:20:42.916847Z","url":"https://files.pythonhosted.org/packages/e1/b0/7276de53321c12981717490516b7e612364f2cb372ee8901bd4a66a000d7/psutil-5.8.0.tar.gz","yanked":false},{"core-metadata":{"sha256":"146a681ce7fa3f640f9204e5391309605d72b0f4ddaaf9927df9b73319a4490e"},"data-dist-info-metadata":{"sha256":"146a681ce7fa3f640f9204e5391309605d72b0f4ddaaf9927df9b73319a4490e"},"filename":"psutil-5.9.0-cp27-cp27m-manylinux2010_i686.whl","hashes":{"sha256":"55ce319452e3d139e25d6c3f85a1acf12d1607ddedea5e35fb47a552c051161b"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":285348,"upload-time":"2021-12-29T21:26:19.591062Z","url":"https://files.pythonhosted.org/packages/c9/62/5cfcb69c256d469236d4bddeb7ad4ee6a8b37d604dcfc82b7c938fd8ee37/psutil-5.9.0-cp27-cp27m-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"146a681ce7fa3f640f9204e5391309605d72b0f4ddaaf9927df9b73319a4490e"},"data-dist-info-metadata":{"sha256":"146a681ce7fa3f640f9204e5391309605d72b0f4ddaaf9927df9b73319a4490e"},"filename":"psutil-5.9.0-cp27-cp27m-manylinux2010_x86_64.whl","hashes":{"sha256":"7336292a13a80eb93c21f36bde4328aa748a04b68c13d01dfddd67fc13fd0618"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":288125,"upload-time":"2021-12-29T21:26:24.644701Z","url":"https://files.pythonhosted.org/packages/eb/0d/c19872c9121208bbbb4335bb13a4a2f2b95661fd69d24f26e32f94e5a8a1/psutil-5.9.0-cp27-cp27m-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"146a681ce7fa3f640f9204e5391309605d72b0f4ddaaf9927df9b73319a4490e"},"data-dist-info-metadata":{"sha256":"146a681ce7fa3f640f9204e5391309605d72b0f4ddaaf9927df9b73319a4490e"},"filename":"psutil-5.9.0-cp27-cp27mu-manylinux2010_i686.whl","hashes":{"sha256":"cb8d10461c1ceee0c25a64f2dd54872b70b89c26419e147a05a10b753ad36ec2"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":285364,"upload-time":"2021-12-29T21:26:28.242468Z","url":"https://files.pythonhosted.org/packages/1a/3e/ff287d01bca130b72cf53a9b20bbc31bf566d503ee63adf8c7dcfd9315e2/psutil-5.9.0-cp27-cp27mu-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"146a681ce7fa3f640f9204e5391309605d72b0f4ddaaf9927df9b73319a4490e"},"data-dist-info-metadata":{"sha256":"146a681ce7fa3f640f9204e5391309605d72b0f4ddaaf9927df9b73319a4490e"},"filename":"psutil-5.9.0-cp27-cp27mu-manylinux2010_x86_64.whl","hashes":{"sha256":"7641300de73e4909e5d148e90cc3142fb890079e1525a840cf0dfd39195239fd"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":288110,"upload-time":"2021-12-29T21:26:31.963660Z","url":"https://files.pythonhosted.org/packages/d8/49/fbce284331d482703decdc8dec9bfd910fa00a3acd5b974e8efa8c30104a/psutil-5.9.0-cp27-cp27mu-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"e84069ba77055def96d865c3845efa434ef6204d61f576071bac882eb98e785c"},"data-dist-info-metadata":{"sha256":"e84069ba77055def96d865c3845efa434ef6204d61f576071bac882eb98e785c"},"filename":"psutil-5.9.0-cp27-none-win32.whl","hashes":{"sha256":"ea42d747c5f71b5ccaa6897b216a7dadb9f52c72a0fe2b872ef7d3e1eacf3ba3"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":239056,"upload-time":"2021-12-29T21:26:35.738857Z","url":"https://files.pythonhosted.org/packages/ab/d7/a8b076603943ebce7872ca7d4e012f6dcdc33e86eabb117921a6fe6e1f8a/psutil-5.9.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"e84069ba77055def96d865c3845efa434ef6204d61f576071bac882eb98e785c"},"data-dist-info-metadata":{"sha256":"e84069ba77055def96d865c3845efa434ef6204d61f576071bac882eb98e785c"},"filename":"psutil-5.9.0-cp27-none-win_amd64.whl","hashes":{"sha256":"ef216cc9feb60634bda2f341a9559ac594e2eeaadd0ba187a4c2eb5b5d40b91c"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242517,"upload-time":"2021-12-29T21:26:39.603955Z","url":"https://files.pythonhosted.org/packages/2d/7a/ee32fa2c5712fa0bc6a9f376ffe9d2e1dc856e2e011d2bab4e12293dcd88/psutil-5.9.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp310-cp310-macosx_10_9_x86_64.whl","hashes":{"sha256":"90a58b9fcae2dbfe4ba852b57bd4a1dded6b990a33d6428c7614b7d48eccb492"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238624,"upload-time":"2021-12-29T21:26:42.964503Z","url":"https://files.pythonhosted.org/packages/89/48/2c6f566d35a38fb9f882e51d75425a6f1d097cb946e05b6aff98d450a151/psutil-5.9.0-cp310-cp310-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"ff0d41f8b3e9ebb6b6110057e40019a432e96aae2008951121ba4e56040b84f3"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":279343,"upload-time":"2021-12-29T21:26:46.859457Z","url":"https://files.pythonhosted.org/packages/11/46/e790221e8281af5163517a17a20c88b10a75a5642d9c5106a868f2879edd/psutil-5.9.0-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"742c34fff804f34f62659279ed5c5b723bb0195e9d7bd9907591de9f8f6558e2"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":281400,"upload-time":"2021-12-29T21:26:51.801761Z","url":"https://files.pythonhosted.org/packages/6f/8a/d1810472a4950a31df385eafbc9bd20cde971814ff6533021dc565bf14ae/psutil-5.9.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp310-cp310-win32.whl","hashes":{"sha256":"8293942e4ce0c5689821f65ce6522ce4786d02af57f13c0195b40e1edb1db61d"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":241383,"upload-time":"2021-12-29T21:26:55.364799Z","url":"https://files.pythonhosted.org/packages/61/93/4251cfa58e5bbd7f92e1bfb965a0c41376cbcbc83c524a8b60d2678f0edd/psutil-5.9.0-cp310-cp310-win32.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp310-cp310-win_amd64.whl","hashes":{"sha256":"9b51917c1af3fa35a3f2dabd7ba96a2a4f19df3dec911da73875e1edaf22a40b"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":245540,"upload-time":"2021-12-29T21:26:59.088538Z","url":"https://files.pythonhosted.org/packages/9f/c9/7fb339d6a04db3b4ab94671536d11e03b23c056d1604e50e564075a96cd8/psutil-5.9.0-cp310-cp310-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp36-cp36m-macosx_10_9_x86_64.whl","hashes":{"sha256":"e9805fed4f2a81de98ae5fe38b75a74c6e6ad2df8a5c479594c7629a1fe35f56"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238331,"upload-time":"2022-01-07T14:28:07.566595Z","url":"https://files.pythonhosted.org/packages/48/cb/6841d4f39b5711652a93359748879f2977ede55c1020f69d038891073592/psutil-5.9.0-cp36-cp36m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"c51f1af02334e4b516ec221ee26b8fdf105032418ca5a5ab9737e8c87dafe203"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":277267,"upload-time":"2022-01-07T14:28:12.318473Z","url":"https://files.pythonhosted.org/packages/1f/2d/e6640979580db1b51220d3165e256a1d0a31847944a3e2622800a737fe86/psutil-5.9.0-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"32acf55cb9a8cbfb29167cd005951df81b567099295291bcfd1027365b36591d"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":279702,"upload-time":"2022-01-07T14:28:16.706587Z","url":"https://files.pythonhosted.org/packages/64/87/461555057b080e1996427098a6c51c64a8a9025ec18571dabfe5be07eeec/psutil-5.9.0-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp36-cp36m-win32.whl","hashes":{"sha256":"e5c783d0b1ad6ca8a5d3e7b680468c9c926b804be83a3a8e95141b05c39c9f64"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":243193,"upload-time":"2022-01-07T14:28:20.695929Z","url":"https://files.pythonhosted.org/packages/2b/a3/24d36239a7bfa30b0eb4302b045417796e9f2c7c21b296d2405735e8949e/psutil-5.9.0-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"d62a2796e08dd024b8179bd441cb714e0f81226c352c802fca0fd3f89eeacd94"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":247862,"upload-time":"2022-01-07T14:28:23.774803Z","url":"https://files.pythonhosted.org/packages/14/c9/f0bccd60a25197d63a02688ea8f5c5cd5a8b1baf1a7d6bf493d3291132d2/psutil-5.9.0-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp37-cp37m-macosx_10_9_x86_64.whl","hashes":{"sha256":"3d00a664e31921009a84367266b35ba0aac04a2a6cad09c550a89041034d19a0"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238334,"upload-time":"2021-12-29T21:27:03.461413Z","url":"https://files.pythonhosted.org/packages/70/40/0a6ca5641f7574b6ea38cdb561c30065659734755a1779db67b56e225f84/psutil-5.9.0-cp37-cp37m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"7779be4025c540d1d65a2de3f30caeacc49ae7a2152108adeaf42c7534a115ce"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":278077,"upload-time":"2021-12-29T21:27:07.864433Z","url":"https://files.pythonhosted.org/packages/6b/c0/0f233f87e816c20e5489bca749798255a464282cdd5911d62bb8344c4b5a/psutil-5.9.0-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"072664401ae6e7c1bfb878c65d7282d4b4391f1bc9a56d5e03b5a490403271b5"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":280736,"upload-time":"2021-12-29T21:27:11.252493Z","url":"https://files.pythonhosted.org/packages/60/f9/b78291ed21146ece2417bd1ba715564c6d3bdf2f1e9297ed67709bb36eeb/psutil-5.9.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp37-cp37m-win32.whl","hashes":{"sha256":"df2c8bd48fb83a8408c8390b143c6a6fa10cb1a674ca664954de193fdcab36a9"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":241810,"upload-time":"2021-12-29T21:27:14.020326Z","url":"https://files.pythonhosted.org/packages/47/3f/0475146306d02270243e55cad8167d5185c8918933953c90eda846d72ff3/psutil-5.9.0-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"1d7b433519b9a38192dfda962dd8f44446668c009833e1429a52424624f408b4"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":246380,"upload-time":"2021-12-29T21:27:17.444889Z","url":"https://files.pythonhosted.org/packages/7c/d6/4ade7cebfe04710a89e2dc5638f712f09dc5e402a8fea95c3d16dc7f64bf/psutil-5.9.0-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp38-cp38-macosx_10_9_x86_64.whl","hashes":{"sha256":"c3400cae15bdb449d518545cbd5b649117de54e3596ded84aacabfbb3297ead2"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238620,"upload-time":"2021-12-29T21:27:21.799788Z","url":"https://files.pythonhosted.org/packages/89/8e/2a8814f903bc06471621f6e0cd3fc1a7085868656106f31aacf2f844eea2/psutil-5.9.0-cp38-cp38-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"b2237f35c4bbae932ee98902a08050a27821f8f6dfa880a47195e5993af4702d"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":281246,"upload-time":"2021-12-29T21:27:24.591382Z","url":"https://files.pythonhosted.org/packages/4c/95/3c0858c62ec02106cf5f3e79d74223264a6269a16996f31d5ab43abcec86/psutil-5.9.0-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"1070a9b287846a21a5d572d6dddd369517510b68710fca56b0e9e02fd24bed9a"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":283823,"upload-time":"2021-12-29T21:27:27.809766Z","url":"https://files.pythonhosted.org/packages/0a/66/b2188d8e738ee52206a4ee804907f6eab5bcc9fc0e8486e7ab973a8323b7/psutil-5.9.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp38-cp38-win32.whl","hashes":{"sha256":"76cebf84aac1d6da5b63df11fe0d377b46b7b500d892284068bacccf12f20666"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242160,"upload-time":"2021-12-29T21:27:31.512894Z","url":"https://files.pythonhosted.org/packages/d0/cf/7a86fc08f821d66c528939f155079df7d0945678fc474c6a6455c909f6eb/psutil-5.9.0-cp38-cp38-win32.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp38-cp38-win_amd64.whl","hashes":{"sha256":"3151a58f0fbd8942ba94f7c31c7e6b310d2989f4da74fcbf28b934374e9bf841"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":246488,"upload-time":"2021-12-29T21:27:36.422809Z","url":"https://files.pythonhosted.org/packages/62/d4/72fc44dfd9939851bd672e94e43d12848a98b1d2c3f6f794d54a220fe4a7/psutil-5.9.0-cp38-cp38-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp39-cp39-macosx_10_9_x86_64.whl","hashes":{"sha256":"539e429da49c5d27d5a58e3563886057f8fc3868a5547b4f1876d9c0f007bccf"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238613,"upload-time":"2021-12-29T21:27:39.813718Z","url":"https://files.pythonhosted.org/packages/48/6a/c6e88a5584544033dbb8318c380e7e1e3796e5ac336577eb91dc75bdecd7/psutil-5.9.0-cp39-cp39-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"58c7d923dc209225600aec73aa2c4ae8ea33b1ab31bc11ef8a5933b027476f07"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":278536,"upload-time":"2021-12-29T21:27:43.204962Z","url":"https://files.pythonhosted.org/packages/f7/b1/82e95f6368dbde6b7e54ea6b18cf8ac3958223540d0bcbde23ba7be19478/psutil-5.9.0-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"3611e87eea393f779a35b192b46a164b1d01167c9d323dda9b1e527ea69d697d"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":280415,"upload-time":"2021-12-29T21:27:47.616041Z","url":"https://files.pythonhosted.org/packages/c4/35/7cec9647be077784d20913404f914fffd8fe6dfd0673e29f7bd822ac1331/psutil-5.9.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp39-cp39-win32.whl","hashes":{"sha256":"4e2fb92e3aeae3ec3b7b66c528981fd327fb93fd906a77215200404444ec1845"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":241428,"upload-time":"2021-12-29T21:27:52.616449Z","url":"https://files.pythonhosted.org/packages/5a/c6/923aed22f6c9c5197998fa6907c983e884975a0ae3430ccd8514f5fd0d6a/psutil-5.9.0-cp39-cp39-win32.whl","yanked":false},{"core-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"data-dist-info-metadata":{"sha256":"ad269c4c58fadbf100c6116eb3ebed5d77878f0c43b9314d18addc011b2b2b80"},"filename":"psutil-5.9.0-cp39-cp39-win_amd64.whl","hashes":{"sha256":"7d190ee2eaef7831163f254dc58f6d2e2a22e27382b936aab51c835fc080c3d3"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":245606,"upload-time":"2021-12-29T21:27:56.202709Z","url":"https://files.pythonhosted.org/packages/9e/9e/3a48f15a1539505e2f3058a709eee56acfb379f2b0ff409d6291099e2a7e/psutil-5.9.0-cp39-cp39-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.9.0.tar.gz","hashes":{"sha256":"869842dbd66bb80c3217158e629d6fceaecc3a3166d3d1faee515b05dd26ca25"},"provenance":null,"requires-python":">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":478322,"upload-time":"2021-12-29T21:27:59.163343Z","url":"https://files.pythonhosted.org/packages/47/b6/ea8a7728f096a597f0032564e8013b705aa992a0990becd773dcc4d7b4a7/psutil-5.9.0.tar.gz","yanked":false},{"core-metadata":{"sha256":"ac1d788a80e2f59751f3eb37a186124618aa8bafbbe6cd2ad03985f508e49a67"},"data-dist-info-metadata":{"sha256":"ac1d788a80e2f59751f3eb37a186124618aa8bafbbe6cd2ad03985f508e49a67"},"filename":"psutil-5.9.1-cp27-cp27m-manylinux2010_i686.whl","hashes":{"sha256":"799759d809c31aab5fe4579e50addf84565e71c1dc9f1c31258f159ff70d3f87"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":286186,"upload-time":"2022-05-20T20:09:47.558524Z","url":"https://files.pythonhosted.org/packages/77/06/f9fd79449440d7217d6bf2c90998d540e125cfeffe39d214a328dadc46f4/psutil-5.9.1-cp27-cp27m-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"ac1d788a80e2f59751f3eb37a186124618aa8bafbbe6cd2ad03985f508e49a67"},"data-dist-info-metadata":{"sha256":"ac1d788a80e2f59751f3eb37a186124618aa8bafbbe6cd2ad03985f508e49a67"},"filename":"psutil-5.9.1-cp27-cp27m-manylinux2010_x86_64.whl","hashes":{"sha256":"9272167b5f5fbfe16945be3db475b3ce8d792386907e673a209da686176552af"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":288858,"upload-time":"2022-05-20T20:09:52.956913Z","url":"https://files.pythonhosted.org/packages/cf/29/ad704a45960bfb52ef8bf0beb9c41c09ce92d61c40333f03e9a03f246c22/psutil-5.9.1-cp27-cp27m-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"e6c8d739b32cc31942a1726ba4e33d58fe5d46a1400a2526e858888d58324393"},"data-dist-info-metadata":{"sha256":"e6c8d739b32cc31942a1726ba4e33d58fe5d46a1400a2526e858888d58324393"},"filename":"psutil-5.9.1-cp27-cp27m-win32.whl","hashes":{"sha256":"0904727e0b0a038830b019551cf3204dd48ef5c6868adc776e06e93d615fc5fc"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":239479,"upload-time":"2022-05-20T20:09:57.166474Z","url":"https://files.pythonhosted.org/packages/7e/8d/e0a66123fa98e309597815de518b47a7a6c571a8f886fc8d4db2331fd2ab/psutil-5.9.1-cp27-cp27m-win32.whl","yanked":false},{"core-metadata":{"sha256":"e6c8d739b32cc31942a1726ba4e33d58fe5d46a1400a2526e858888d58324393"},"data-dist-info-metadata":{"sha256":"e6c8d739b32cc31942a1726ba4e33d58fe5d46a1400a2526e858888d58324393"},"filename":"psutil-5.9.1-cp27-cp27m-win_amd64.whl","hashes":{"sha256":"e7e10454cb1ab62cc6ce776e1c135a64045a11ec4c6d254d3f7689c16eb3efd2"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242812,"upload-time":"2022-05-20T20:10:01.868774Z","url":"https://files.pythonhosted.org/packages/1b/53/8f0772df0a6d593bc2fcdf12f4f790bab5c4f6a77bb61a8ddaad2cbba7f8/psutil-5.9.1-cp27-cp27m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"ac1d788a80e2f59751f3eb37a186124618aa8bafbbe6cd2ad03985f508e49a67"},"data-dist-info-metadata":{"sha256":"ac1d788a80e2f59751f3eb37a186124618aa8bafbbe6cd2ad03985f508e49a67"},"filename":"psutil-5.9.1-cp27-cp27mu-manylinux2010_i686.whl","hashes":{"sha256":"56960b9e8edcca1456f8c86a196f0c3d8e3e361320071c93378d41445ffd28b0"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":286184,"upload-time":"2022-05-20T20:10:05.575313Z","url":"https://files.pythonhosted.org/packages/2d/56/54b4ed8102ce5a2f5367b4e766c1873c18f9c32cde321435d0e0ee2abcc5/psutil-5.9.1-cp27-cp27mu-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"ac1d788a80e2f59751f3eb37a186124618aa8bafbbe6cd2ad03985f508e49a67"},"data-dist-info-metadata":{"sha256":"ac1d788a80e2f59751f3eb37a186124618aa8bafbbe6cd2ad03985f508e49a67"},"filename":"psutil-5.9.1-cp27-cp27mu-manylinux2010_x86_64.whl","hashes":{"sha256":"44d1826150d49ffd62035785a9e2c56afcea66e55b43b8b630d7706276e87f22"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":288863,"upload-time":"2022-05-20T20:10:09.032899Z","url":"https://files.pythonhosted.org/packages/2c/9d/dc329b7da284677ea843f3ff4b35b8ab3b96b65a58a544b3c3f86d9d032f/psutil-5.9.1-cp27-cp27mu-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"data-dist-info-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"filename":"psutil-5.9.1-cp310-cp310-macosx_10_9_x86_64.whl","hashes":{"sha256":"c7be9d7f5b0d206f0bbc3794b8e16fb7dbc53ec9e40bbe8787c6f2d38efcf6c9"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":239266,"upload-time":"2022-05-20T20:10:12.541628Z","url":"https://files.pythonhosted.org/packages/d1/16/6239e76ab5d990dc7866bc22a80585f73421588d63b42884d607f5f815e2/psutil-5.9.1-cp310-cp310-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"data-dist-info-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"filename":"psutil-5.9.1-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"abd9246e4cdd5b554a2ddd97c157e292ac11ef3e7af25ac56b08b455c829dca8"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":280127,"upload-time":"2022-05-20T20:10:16.449769Z","url":"https://files.pythonhosted.org/packages/14/06/39d7e963a6a8bbf26519de208593cdb0ddfe22918b8989f4b2363d4ab49f/psutil-5.9.1-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"data-dist-info-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"filename":"psutil-5.9.1-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"29a442e25fab1f4d05e2655bb1b8ab6887981838d22effa2396d584b740194de"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":282049,"upload-time":"2022-05-20T20:10:19.914788Z","url":"https://files.pythonhosted.org/packages/6d/c6/6a4e46802e8690d50ba6a56c7f79ac283e703fcfa0fdae8e41909c8cef1f/psutil-5.9.1-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"b48f9beb61d032244f4c239867a736641369484a967d3ac29eeab577ce127229"},"data-dist-info-metadata":{"sha256":"b48f9beb61d032244f4c239867a736641369484a967d3ac29eeab577ce127229"},"filename":"psutil-5.9.1-cp310-cp310-win32.whl","hashes":{"sha256":"20b27771b077dcaa0de1de3ad52d22538fe101f9946d6dc7869e6f694f079329"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":241699,"upload-time":"2022-05-20T20:10:23.278390Z","url":"https://files.pythonhosted.org/packages/26/b4/a58cf15ea649faa92c54f00c627aef1d50b9f1abf207485f10c967a50c95/psutil-5.9.1-cp310-cp310-win32.whl","yanked":false},{"core-metadata":{"sha256":"b48f9beb61d032244f4c239867a736641369484a967d3ac29eeab577ce127229"},"data-dist-info-metadata":{"sha256":"b48f9beb61d032244f4c239867a736641369484a967d3ac29eeab577ce127229"},"filename":"psutil-5.9.1-cp310-cp310-win_amd64.whl","hashes":{"sha256":"58678bbadae12e0db55186dc58f2888839228ac9f41cc7848853539b70490021"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":245843,"upload-time":"2022-05-20T20:10:26.856840Z","url":"https://files.pythonhosted.org/packages/c0/5a/2ac88d5265b711c8aa4e786825b38d5d0b1e5ecbdd0ce78e9b04a820d247/psutil-5.9.1-cp310-cp310-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"data-dist-info-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"filename":"psutil-5.9.1-cp36-cp36m-macosx_10_9_x86_64.whl","hashes":{"sha256":"3a76ad658641172d9c6e593de6fe248ddde825b5866464c3b2ee26c35da9d237"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":239007,"upload-time":"2022-05-20T20:10:30.191299Z","url":"https://files.pythonhosted.org/packages/65/1d/6a112f146faee6292a6c3ee2a7f24a8e572697adb7e1c5de3d8508f647cc/psutil-5.9.1-cp36-cp36m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"data-dist-info-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"filename":"psutil-5.9.1-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"a6a11e48cb93a5fa606306493f439b4aa7c56cb03fc9ace7f6bfa21aaf07c453"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":278074,"upload-time":"2022-05-20T20:10:33.843171Z","url":"https://files.pythonhosted.org/packages/7e/52/a02dc53e26714a339c8b4972d8e3f268e4db8905f5d1a3a100f1e40b6fa7/psutil-5.9.1-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"data-dist-info-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"filename":"psutil-5.9.1-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"068935df39055bf27a29824b95c801c7a5130f118b806eee663cad28dca97685"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":280402,"upload-time":"2022-05-20T20:10:38.665061Z","url":"https://files.pythonhosted.org/packages/6b/76/a8cb69ed3566877dcbccf408f5f9d6055227ad4fed694e88809fa8506b0b/psutil-5.9.1-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"data-dist-info-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"filename":"psutil-5.9.1-cp36-cp36m-win32.whl","hashes":{"sha256":"0f15a19a05f39a09327345bc279c1ba4a8cfb0172cc0d3c7f7d16c813b2e7d36"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":243501,"upload-time":"2022-05-20T20:10:42.164047Z","url":"https://files.pythonhosted.org/packages/85/4d/78173e3dffb74c5fa87914908f143473d0b8b9183f9d275333679a4e4649/psutil-5.9.1-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"data-dist-info-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"filename":"psutil-5.9.1-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"db417f0865f90bdc07fa30e1aadc69b6f4cad7f86324b02aa842034efe8d8c4d"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":248156,"upload-time":"2022-05-20T20:10:45.327961Z","url":"https://files.pythonhosted.org/packages/73/1a/d78f2f2de2aad6628415d2a48917cabc2c7fb0c3a31c7cdf187cffa4eb36/psutil-5.9.1-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"data-dist-info-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"filename":"psutil-5.9.1-cp37-cp37m-macosx_10_9_x86_64.whl","hashes":{"sha256":"91c7ff2a40c373d0cc9121d54bc5f31c4fa09c346528e6a08d1845bce5771ffc"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238960,"upload-time":"2022-05-20T20:10:48.678116Z","url":"https://files.pythonhosted.org/packages/d6/ef/fd4dc9085e3879c3af63fe60667dd3b71adf50d030b5549315f4a619271b/psutil-5.9.1-cp37-cp37m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"data-dist-info-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"filename":"psutil-5.9.1-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"fea896b54f3a4ae6f790ac1d017101252c93f6fe075d0e7571543510f11d2676"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":278553,"upload-time":"2022-05-20T20:10:52.019471Z","url":"https://files.pythonhosted.org/packages/97/f6/0180e58dd1359da7d6fbc27d04dac6fb500dc758b6f4b65407608bb13170/psutil-5.9.1-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"data-dist-info-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"filename":"psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"3054e923204b8e9c23a55b23b6df73a8089ae1d075cb0bf711d3e9da1724ded4"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":281356,"upload-time":"2022-05-20T20:10:55.963864Z","url":"https://files.pythonhosted.org/packages/13/71/c25adbd9b33a2e27edbe1fc84b3111a5ad97611885d7abcbdd8d1f2bb7ca/psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"b48f9beb61d032244f4c239867a736641369484a967d3ac29eeab577ce127229"},"data-dist-info-metadata":{"sha256":"b48f9beb61d032244f4c239867a736641369484a967d3ac29eeab577ce127229"},"filename":"psutil-5.9.1-cp37-cp37m-win32.whl","hashes":{"sha256":"d2d006286fbcb60f0b391741f520862e9b69f4019b4d738a2a45728c7e952f1b"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242105,"upload-time":"2022-05-20T20:10:59.100543Z","url":"https://files.pythonhosted.org/packages/2a/32/136cd5bf55728ea64a22b1d817890e35fc17314c46a24ee3268b65f9076f/psutil-5.9.1-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"b48f9beb61d032244f4c239867a736641369484a967d3ac29eeab577ce127229"},"data-dist-info-metadata":{"sha256":"b48f9beb61d032244f4c239867a736641369484a967d3ac29eeab577ce127229"},"filename":"psutil-5.9.1-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"b14ee12da9338f5e5b3a3ef7ca58b3cba30f5b66f7662159762932e6d0b8f680"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":246671,"upload-time":"2022-05-20T20:11:02.513602Z","url":"https://files.pythonhosted.org/packages/df/88/427f3959855fcb3ab04891e00c026a246892feb11b20433db814b7a24405/psutil-5.9.1-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"data-dist-info-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"filename":"psutil-5.9.1-cp38-cp38-macosx_10_9_x86_64.whl","hashes":{"sha256":"19f36c16012ba9cfc742604df189f2f28d2720e23ff7d1e81602dbe066be9fd1"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":239250,"upload-time":"2022-05-20T20:11:06.292977Z","url":"https://files.pythonhosted.org/packages/46/80/1de3a9bac336b5c8e4f7b0ff2e80c85ba237f18f2703be68884ee6798432/psutil-5.9.1-cp38-cp38-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"data-dist-info-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"filename":"psutil-5.9.1-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"944c4b4b82dc4a1b805329c980f270f170fdc9945464223f2ec8e57563139cf4"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":281678,"upload-time":"2022-05-20T20:11:09.990719Z","url":"https://files.pythonhosted.org/packages/fd/ba/c5a3f46f351ab609cc0be6a563e492900c57e3d5c9bda0b79b84d8c3eae9/psutil-5.9.1-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"data-dist-info-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"filename":"psutil-5.9.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"4b6750a73a9c4a4e689490ccb862d53c7b976a2a35c4e1846d049dcc3f17d83b"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":284665,"upload-time":"2022-05-20T20:11:13.584309Z","url":"https://files.pythonhosted.org/packages/9d/41/d5f2db2ab7f5dff2fa795993a0cd6fa8a8f39ca197c3a86857875333ec10/psutil-5.9.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"b48f9beb61d032244f4c239867a736641369484a967d3ac29eeab577ce127229"},"data-dist-info-metadata":{"sha256":"b48f9beb61d032244f4c239867a736641369484a967d3ac29eeab577ce127229"},"filename":"psutil-5.9.1-cp38-cp38-win32.whl","hashes":{"sha256":"a8746bfe4e8f659528c5c7e9af5090c5a7d252f32b2e859c584ef7d8efb1e689"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242461,"upload-time":"2022-05-20T20:11:16.942263Z","url":"https://files.pythonhosted.org/packages/41/ec/5fd3e9388d0ed1edfdeae71799df374f4a117932646a63413fa95a121e9f/psutil-5.9.1-cp38-cp38-win32.whl","yanked":false},{"core-metadata":{"sha256":"b48f9beb61d032244f4c239867a736641369484a967d3ac29eeab577ce127229"},"data-dist-info-metadata":{"sha256":"b48f9beb61d032244f4c239867a736641369484a967d3ac29eeab577ce127229"},"filename":"psutil-5.9.1-cp38-cp38-win_amd64.whl","hashes":{"sha256":"79c9108d9aa7fa6fba6e668b61b82facc067a6b81517cab34d07a84aa89f3df0"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":246804,"upload-time":"2022-05-20T20:11:20.322459Z","url":"https://files.pythonhosted.org/packages/b2/ad/65e2b2b97677f98d718388dc11b2a9d7f177ebbae5eef72547a32bc28911/psutil-5.9.1-cp38-cp38-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"data-dist-info-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"filename":"psutil-5.9.1-cp39-cp39-macosx_10_9_x86_64.whl","hashes":{"sha256":"28976df6c64ddd6320d281128817f32c29b539a52bdae5e192537bc338a9ec81"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":239248,"upload-time":"2022-05-20T20:11:23.352312Z","url":"https://files.pythonhosted.org/packages/9f/ca/84ce3e48b3ca2f0f74314d89929b3a523220f3f4a8dff395d6ef74dadef3/psutil-5.9.1-cp39-cp39-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"data-dist-info-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"filename":"psutil-5.9.1-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"b88f75005586131276634027f4219d06e0561292be8bd6bc7f2f00bdabd63c4e"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":279320,"upload-time":"2022-05-20T20:11:27.310813Z","url":"https://files.pythonhosted.org/packages/a9/97/b7e3532d97d527349701d2143c3f868733b94e2db6f531b07811b698f549/psutil-5.9.1-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"data-dist-info-metadata":{"sha256":"1f39ef2570a850c4cbdb8430a13b995343050494e85d316030a206bd7da4c63b"},"filename":"psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"645bd4f7bb5b8633803e0b6746ff1628724668681a434482546887d22c7a9537"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":281065,"upload-time":"2022-05-20T20:11:30.617653Z","url":"https://files.pythonhosted.org/packages/62/1f/f14225bda76417ab9bd808ff21d5cd59d5435a9796ca09b34d4cb0edcd88/psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"b48f9beb61d032244f4c239867a736641369484a967d3ac29eeab577ce127229"},"data-dist-info-metadata":{"sha256":"b48f9beb61d032244f4c239867a736641369484a967d3ac29eeab577ce127229"},"filename":"psutil-5.9.1-cp39-cp39-win32.whl","hashes":{"sha256":"32c52611756096ae91f5d1499fe6c53b86f4a9ada147ee42db4991ba1520e574"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":241748,"upload-time":"2022-05-20T20:11:34.374569Z","url":"https://files.pythonhosted.org/packages/b1/d2/c5374a784567c1e42ee8a589b1b42e2bd6e14c7be3c234d84360ab3a0a39/psutil-5.9.1-cp39-cp39-win32.whl","yanked":false},{"core-metadata":{"sha256":"b48f9beb61d032244f4c239867a736641369484a967d3ac29eeab577ce127229"},"data-dist-info-metadata":{"sha256":"b48f9beb61d032244f4c239867a736641369484a967d3ac29eeab577ce127229"},"filename":"psutil-5.9.1-cp39-cp39-win_amd64.whl","hashes":{"sha256":"f65f9a46d984b8cd9b3750c2bdb419b2996895b005aefa6cbaba9a143b1ce2c5"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":245883,"upload-time":"2022-05-20T20:11:37.840887Z","url":"https://files.pythonhosted.org/packages/e0/ac/fd6f098969d49f046083ac032e6788d9f861903596fb9555a02bf50a1238/psutil-5.9.1-cp39-cp39-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.9.1.tar.gz","hashes":{"sha256":"57f1819b5d9e95cdfb0c881a8a5b7d542ed0b7c522d575706a80bedc848c8954"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":479090,"upload-time":"2022-05-20T20:11:41.043143Z","url":"https://files.pythonhosted.org/packages/d6/de/0999ea2562b96d7165812606b18f7169307b60cd378bc29cf3673322c7e9/psutil-5.9.1.tar.gz","yanked":false},{"core-metadata":{"sha256":"fcf26ce4e0d8482a82643f948df7370a2afab20a048f7563ee3fb420705fcf07"},"data-dist-info-metadata":{"sha256":"fcf26ce4e0d8482a82643f948df7370a2afab20a048f7563ee3fb420705fcf07"},"filename":"psutil-5.9.2-cp27-cp27m-manylinux2010_i686.whl","hashes":{"sha256":"8f024fbb26c8daf5d70287bb3edfafa22283c255287cf523c5d81721e8e5d82c"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":285739,"upload-time":"2022-09-04T20:15:28.668835Z","url":"https://files.pythonhosted.org/packages/37/a4/cb10e4c0faa3091de22eb78fa1c332566e60b9b59001bef326a4c1070417/psutil-5.9.2-cp27-cp27m-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"fcf26ce4e0d8482a82643f948df7370a2afab20a048f7563ee3fb420705fcf07"},"data-dist-info-metadata":{"sha256":"fcf26ce4e0d8482a82643f948df7370a2afab20a048f7563ee3fb420705fcf07"},"filename":"psutil-5.9.2-cp27-cp27m-manylinux2010_x86_64.whl","hashes":{"sha256":"b2f248ffc346f4f4f0d747ee1947963613216b06688be0be2e393986fe20dbbb"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":289461,"upload-time":"2022-09-04T20:15:32.802519Z","url":"https://files.pythonhosted.org/packages/93/40/58dfcab15435b6fedf5385bc7e88a4c162cc6af0056f5d9d97f5ebfd7fa0/psutil-5.9.2-cp27-cp27m-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"5f8d323cf6b24ee1e72553da21c2f9124f45da854ffd5befab575ceb9733c773"},"data-dist-info-metadata":{"sha256":"5f8d323cf6b24ee1e72553da21c2f9124f45da854ffd5befab575ceb9733c773"},"filename":"psutil-5.9.2-cp27-cp27m-win32.whl","hashes":{"sha256":"b1928b9bf478d31fdffdb57101d18f9b70ed4e9b0e41af751851813547b2a9ab"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":240036,"upload-time":"2022-09-04T20:15:36.698093Z","url":"https://files.pythonhosted.org/packages/42/eb/83470960f2c13a026b07051456ad834f5fea0c80e8cb83fc65005f5f18d5/psutil-5.9.2-cp27-cp27m-win32.whl","yanked":false},{"core-metadata":{"sha256":"5f8d323cf6b24ee1e72553da21c2f9124f45da854ffd5befab575ceb9733c773"},"data-dist-info-metadata":{"sha256":"5f8d323cf6b24ee1e72553da21c2f9124f45da854ffd5befab575ceb9733c773"},"filename":"psutil-5.9.2-cp27-cp27m-win_amd64.whl","hashes":{"sha256":"404f4816c16a2fcc4eaa36d7eb49a66df2d083e829d3e39ee8759a411dbc9ecf"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":243378,"upload-time":"2022-09-04T20:15:40.746927Z","url":"https://files.pythonhosted.org/packages/d1/5b/b9d6ac192d3108e1dc7875ab1579b7f65eb7bf0ef799dadd3f3798d0af2e/psutil-5.9.2-cp27-cp27m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"fcf26ce4e0d8482a82643f948df7370a2afab20a048f7563ee3fb420705fcf07"},"data-dist-info-metadata":{"sha256":"fcf26ce4e0d8482a82643f948df7370a2afab20a048f7563ee3fb420705fcf07"},"filename":"psutil-5.9.2-cp27-cp27mu-manylinux2010_i686.whl","hashes":{"sha256":"94e621c6a4ddb2573d4d30cba074f6d1aa0186645917df42c811c473dd22b339"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":285740,"upload-time":"2022-09-04T20:15:44.420732Z","url":"https://files.pythonhosted.org/packages/b6/96/ddf877440f2686eb17933531507fe4822ff1ed76d85df4a093a605b91db8/psutil-5.9.2-cp27-cp27mu-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"fcf26ce4e0d8482a82643f948df7370a2afab20a048f7563ee3fb420705fcf07"},"data-dist-info-metadata":{"sha256":"fcf26ce4e0d8482a82643f948df7370a2afab20a048f7563ee3fb420705fcf07"},"filename":"psutil-5.9.2-cp27-cp27mu-manylinux2010_x86_64.whl","hashes":{"sha256":"256098b4f6ffea6441eb54ab3eb64db9ecef18f6a80d7ba91549195d55420f84"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":289451,"upload-time":"2022-09-04T20:15:47.573981Z","url":"https://files.pythonhosted.org/packages/d7/df/ff5c766b50350f2a4555d5068127d372bb26201a2a5eeda9efc8dbf570b4/psutil-5.9.2-cp27-cp27mu-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp310-cp310-macosx_10_9_x86_64.whl","hashes":{"sha256":"614337922702e9be37a39954d67fdb9e855981624d8011a9927b8f2d3c9625d9"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":239273,"upload-time":"2022-09-04T20:15:52.215984Z","url":"https://files.pythonhosted.org/packages/04/5d/d52473097582db5d3094bc34acf9874de726327a3166426e22ed0806de6a/psutil-5.9.2-cp310-cp310-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"39ec06dc6c934fb53df10c1672e299145ce609ff0611b569e75a88f313634969"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":280571,"upload-time":"2022-09-04T20:15:55.608887Z","url":"https://files.pythonhosted.org/packages/47/2b/bd12c4f2d1bd3024fe7c5d8388f8a5627cc02fbe11d62bd451aff356415d/psutil-5.9.2-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"e3ac2c0375ef498e74b9b4ec56df3c88be43fe56cac465627572dbfb21c4be34"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":282777,"upload-time":"2022-09-04T20:15:59.840541Z","url":"https://files.pythonhosted.org/packages/4c/85/7a112fb6a8c598a6f5d079228bbc03ae84c472397be79c075e7514b6ed36/psutil-5.9.2-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp310-cp310-win32.whl","hashes":{"sha256":"e4c4a7636ffc47b7141864f1c5e7d649f42c54e49da2dd3cceb1c5f5d29bfc85"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":241713,"upload-time":"2022-09-05T14:16:44.931542Z","url":"https://files.pythonhosted.org/packages/39/07/5cbcf3322031fcf8dcbfa431b1c145f193c96b18964ef374a88d6a83f2c9/psutil-5.9.2-cp310-cp310-win32.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp310-cp310-win_amd64.whl","hashes":{"sha256":"f4cb67215c10d4657e320037109939b1c1d2fd70ca3d76301992f89fe2edb1f1"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":245853,"upload-time":"2022-09-05T14:16:48.388734Z","url":"https://files.pythonhosted.org/packages/ae/9c/d29dd82d5fda2c6c6d959d57101c78ddbac8325defe94e1b9f983e7cfff3/psutil-5.9.2-cp310-cp310-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"c9979ffe6f0cae3b896ebf06f4e0fa63198d968c57d0df7bb3405936e7b9eb5f"},"data-dist-info-metadata":{"sha256":"c9979ffe6f0cae3b896ebf06f4e0fa63198d968c57d0df7bb3405936e7b9eb5f"},"filename":"psutil-5.9.2-cp36-cp36m-macosx_10_9_x86_64.whl","hashes":{"sha256":"dc9bda7d5ced744622f157cc8d8bdd51735dafcecff807e928ff26bdb0ff097d"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238998,"upload-time":"2022-09-04T20:16:03.484887Z","url":"https://files.pythonhosted.org/packages/df/aa/8268eee572fb9bdf3486d384e3973ad9d635403841c6e7f2af7781e5525b/psutil-5.9.2-cp36-cp36m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"c9979ffe6f0cae3b896ebf06f4e0fa63198d968c57d0df7bb3405936e7b9eb5f"},"data-dist-info-metadata":{"sha256":"c9979ffe6f0cae3b896ebf06f4e0fa63198d968c57d0df7bb3405936e7b9eb5f"},"filename":"psutil-5.9.2-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"d75291912b945a7351d45df682f9644540d564d62115d4a20d45fa17dc2d48f8"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":278226,"upload-time":"2022-09-04T20:16:07.134877Z","url":"https://files.pythonhosted.org/packages/f0/43/bcb92221f5dd45e155337aae37e412fe02a3e5d99e936156a4dcff89fa55/psutil-5.9.2-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"c9979ffe6f0cae3b896ebf06f4e0fa63198d968c57d0df7bb3405936e7b9eb5f"},"data-dist-info-metadata":{"sha256":"c9979ffe6f0cae3b896ebf06f4e0fa63198d968c57d0df7bb3405936e7b9eb5f"},"filename":"psutil-5.9.2-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"b4018d5f9b6651f9896c7a7c2c9f4652e4eea53f10751c4e7d08a9093ab587ec"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":280489,"upload-time":"2022-09-04T20:16:10.463113Z","url":"https://files.pythonhosted.org/packages/a4/eb/d841d5bc526641aad65373b0a4850e98284580df967daff5288779090ea3/psutil-5.9.2-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"c9979ffe6f0cae3b896ebf06f4e0fa63198d968c57d0df7bb3405936e7b9eb5f"},"data-dist-info-metadata":{"sha256":"c9979ffe6f0cae3b896ebf06f4e0fa63198d968c57d0df7bb3405936e7b9eb5f"},"filename":"psutil-5.9.2-cp36-cp36m-win32.whl","hashes":{"sha256":"f40ba362fefc11d6bea4403f070078d60053ed422255bd838cd86a40674364c9"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":243511,"upload-time":"2022-09-05T14:16:51.624881Z","url":"https://files.pythonhosted.org/packages/54/5f/3619e7d22ded096fa6dbd329fc057bfcf53e998b1e2c1ecc07a4155175b1/psutil-5.9.2-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"c9979ffe6f0cae3b896ebf06f4e0fa63198d968c57d0df7bb3405936e7b9eb5f"},"data-dist-info-metadata":{"sha256":"c9979ffe6f0cae3b896ebf06f4e0fa63198d968c57d0df7bb3405936e7b9eb5f"},"filename":"psutil-5.9.2-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"9770c1d25aee91417eba7869139d629d6328a9422ce1cdd112bd56377ca98444"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":248164,"upload-time":"2022-09-05T14:16:55.091982Z","url":"https://files.pythonhosted.org/packages/53/ac/7c4ff994b1ea7d46a84932f0c8d49e28e36a668173975876353f4ea38588/psutil-5.9.2-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp37-cp37m-macosx_10_9_x86_64.whl","hashes":{"sha256":"42638876b7f5ef43cef8dcf640d3401b27a51ee3fa137cb2aa2e72e188414c32"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":238970,"upload-time":"2022-09-04T20:16:14.127339Z","url":"https://files.pythonhosted.org/packages/55/c5/fd2c45a0845e7bae07c8112ed67c21163742cc116732ac2702d9139a9a92/psutil-5.9.2-cp37-cp37m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"91aa0dac0c64688667b4285fa29354acfb3e834e1fd98b535b9986c883c2ce1d"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":279452,"upload-time":"2022-09-04T20:16:18.609472Z","url":"https://files.pythonhosted.org/packages/89/cf/b228a7554eda5e72fd8c33b89c628a86336e5cdbd62fe8b8d2a61a099b2d/psutil-5.9.2-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"4fb54941aac044a61db9d8eb56fc5bee207db3bc58645d657249030e15ba3727"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":281281,"upload-time":"2022-09-04T20:16:22.476894Z","url":"https://files.pythonhosted.org/packages/3d/73/d8c87b5612c58d1e6c6d91997c1590771d34e4ee27d9c11eb1e64ecbf365/psutil-5.9.2-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp37-cp37m-win32.whl","hashes":{"sha256":"7cbb795dcd8ed8fd238bc9e9f64ab188f3f4096d2e811b5a82da53d164b84c3f"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242115,"upload-time":"2022-09-05T14:16:58.770138Z","url":"https://files.pythonhosted.org/packages/98/42/62470fae4e1e9c0f4336acf74af9d4a6d5c6b5788c8435ec387e987a7ebe/psutil-5.9.2-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"5d39e3a2d5c40efa977c9a8dd4f679763c43c6c255b1340a56489955dbca767c"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":246680,"upload-time":"2022-09-05T14:17:03.836879Z","url":"https://files.pythonhosted.org/packages/5e/a2/4025f29069010f118eba4bcd681167d547525d40d2c45029db2f64606f86/psutil-5.9.2-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp38-cp38-macosx_10_9_x86_64.whl","hashes":{"sha256":"fd331866628d18223a4265371fd255774affd86244fc307ef66eaf00de0633d5"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":239258,"upload-time":"2022-09-04T20:16:26.490949Z","url":"https://files.pythonhosted.org/packages/2b/52/c69f5d0acc4bbd3cf44178f025e498666d2eebc216f5f5725d9142244365/psutil-5.9.2-cp38-cp38-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"b315febaebae813326296872fdb4be92ad3ce10d1d742a6b0c49fb619481ed0b"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":282623,"upload-time":"2022-09-04T20:16:29.707162Z","url":"https://files.pythonhosted.org/packages/c4/02/5fc4419f47f141ec0dd28db36fb8bcf1eb6e9df332690617b052c8bec76d/psutil-5.9.2-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"f7929a516125f62399d6e8e026129c8835f6c5a3aab88c3fff1a05ee8feb840d"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":284813,"upload-time":"2022-09-04T20:16:33.456488Z","url":"https://files.pythonhosted.org/packages/79/61/a8d6d649996494672d8a86fe8be6c81b2880ee30881709d84435f2505b47/psutil-5.9.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp38-cp38-win32.whl","hashes":{"sha256":"561dec454853846d1dd0247b44c2e66a0a0c490f937086930ec4b8f83bf44f06"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242471,"upload-time":"2022-09-05T14:17:07.677347Z","url":"https://files.pythonhosted.org/packages/6f/8d/41c402ae33b1ce3f8e37a0dec691d753cbe66e6784e7fd26ed0cd16d99ab/psutil-5.9.2-cp38-cp38-win32.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp38-cp38-win_amd64.whl","hashes":{"sha256":"67b33f27fc0427483b61563a16c90d9f3b547eeb7af0ef1b9fe024cdc9b3a6ea"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":246816,"upload-time":"2022-09-05T14:17:11.540699Z","url":"https://files.pythonhosted.org/packages/29/07/a35c4127942cce6899d447cb54f9926d33cf1800a37c09192dd9b5a08744/psutil-5.9.2-cp38-cp38-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp39-cp39-macosx_10_9_x86_64.whl","hashes":{"sha256":"b3591616fa07b15050b2f87e1cdefd06a554382e72866fcc0ab2be9d116486c8"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":239256,"upload-time":"2022-09-04T20:16:37.181485Z","url":"https://files.pythonhosted.org/packages/65/74/0ad485d753b2f0d00ee4ec933da1e169bc4c8f4f58db88132e886efed14b/psutil-5.9.2-cp39-cp39-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"14b29f581b5edab1f133563272a6011925401804d52d603c5c606936b49c8b97"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":279700,"upload-time":"2022-09-04T20:16:40.816901Z","url":"https://files.pythonhosted.org/packages/bb/df/0819b9aed416b0dedf668cc6b3f291899c276cb2b566c4aa0dc212a03d55/psutil-5.9.2-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"4642fd93785a29353d6917a23e2ac6177308ef5e8be5cc17008d885cb9f70f12"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":281896,"upload-time":"2022-09-04T20:16:45.773716Z","url":"https://files.pythonhosted.org/packages/b3/61/54822666fbbdd4ae1825f7a0b0cf8925a96fac1f778b4a0d5c9c066cf4b2/psutil-5.9.2-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp39-cp39-win32.whl","hashes":{"sha256":"ed29ea0b9a372c5188cdb2ad39f937900a10fb5478dc077283bf86eeac678ef1"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":241759,"upload-time":"2022-09-05T14:17:15.034289Z","url":"https://files.pythonhosted.org/packages/67/cf/f620f740da5bb5895b441248e08b0cd167fb545ecaa3e74ea06f3551975e/psutil-5.9.2-cp39-cp39-win32.whl","yanked":false},{"core-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"data-dist-info-metadata":{"sha256":"9318f57df027edece05c45114c189721cfbe0a56aa2ada92fe2b21340be7af15"},"filename":"psutil-5.9.2-cp39-cp39-win_amd64.whl","hashes":{"sha256":"68b35cbff92d1f7103d8f1db77c977e72f49fcefae3d3d2b91c76b0e7aef48b8"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":245892,"upload-time":"2022-09-05T14:17:18.376384Z","url":"https://files.pythonhosted.org/packages/10/cf/7595896a7487937c171f53bae2eeb0adcc1690ebeef684ac180a77910639/psutil-5.9.2-cp39-cp39-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.9.2.tar.gz","hashes":{"sha256":"feb861a10b6c3bb00701063b37e4afc754f8217f0f09c42280586bd6ac712b5c"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":479757,"upload-time":"2022-09-04T20:16:49.093359Z","url":"https://files.pythonhosted.org/packages/8f/57/828ac1f70badc691a716e77bfae258ef5db76bb7830109bf4bcf882de020/psutil-5.9.2.tar.gz","yanked":false},{"core-metadata":{"sha256":"eecee58b65701e8da473f7e6142a96da3fac24d305218d11c887ca28bcd6cd61"},"data-dist-info-metadata":{"sha256":"eecee58b65701e8da473f7e6142a96da3fac24d305218d11c887ca28bcd6cd61"},"filename":"psutil-5.9.3-cp27-cp27m-macosx_10_9_x86_64.whl","hashes":{"sha256":"b4a247cd3feaae39bb6085fcebf35b3b8ecd9b022db796d89c8f05067ca28e71"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":241919,"upload-time":"2022-10-18T20:12:47.255096Z","url":"https://files.pythonhosted.org/packages/74/42/6268344958236744962c711664de259598fe2005e5818c7d6bc77ae12690/psutil-5.9.3-cp27-cp27m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"eecee58b65701e8da473f7e6142a96da3fac24d305218d11c887ca28bcd6cd61"},"data-dist-info-metadata":{"sha256":"eecee58b65701e8da473f7e6142a96da3fac24d305218d11c887ca28bcd6cd61"},"filename":"psutil-5.9.3-cp27-cp27m-manylinux2010_i686.whl","hashes":{"sha256":"5fa88e3d5d0b480602553d362c4b33a63e0c40bfea7312a7bf78799e01e0810b"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":293266,"upload-time":"2022-10-18T20:12:51.216028Z","url":"https://files.pythonhosted.org/packages/79/6a/7bb45dddeb348cdb9d91d7bc78e903026870ef7f257c35de250392719cf8/psutil-5.9.3-cp27-cp27m-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"eecee58b65701e8da473f7e6142a96da3fac24d305218d11c887ca28bcd6cd61"},"data-dist-info-metadata":{"sha256":"eecee58b65701e8da473f7e6142a96da3fac24d305218d11c887ca28bcd6cd61"},"filename":"psutil-5.9.3-cp27-cp27m-manylinux2010_x86_64.whl","hashes":{"sha256":"767ef4fa33acda16703725c0473a91e1832d296c37c63896c7153ba81698f1ab"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":297585,"upload-time":"2022-10-18T20:12:54.920684Z","url":"https://files.pythonhosted.org/packages/02/c7/d5a6106cf31cc58f4a8a9d88b1ab8405b645b02c482353dd59f5ef19926f/psutil-5.9.3-cp27-cp27m-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"3a494c9ef6c9e0802bc0a2f0d176ca6b72ed9e8ba229b930c23d38f5f92be462"},"data-dist-info-metadata":{"sha256":"3a494c9ef6c9e0802bc0a2f0d176ca6b72ed9e8ba229b930c23d38f5f92be462"},"filename":"psutil-5.9.3-cp27-cp27m-win32.whl","hashes":{"sha256":"9a4af6ed1094f867834f5f07acd1250605a0874169a5fcadbcec864aec2496a6"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":240677,"upload-time":"2022-10-18T20:12:58.145019Z","url":"https://files.pythonhosted.org/packages/62/0a/27aa8d95995fe97a944939f8fff7183f151814a1052b76d125812bed4800/psutil-5.9.3-cp27-cp27m-win32.whl","yanked":false},{"core-metadata":{"sha256":"3a494c9ef6c9e0802bc0a2f0d176ca6b72ed9e8ba229b930c23d38f5f92be462"},"data-dist-info-metadata":{"sha256":"3a494c9ef6c9e0802bc0a2f0d176ca6b72ed9e8ba229b930c23d38f5f92be462"},"filename":"psutil-5.9.3-cp27-cp27m-win_amd64.whl","hashes":{"sha256":"fa5e32c7d9b60b2528108ade2929b115167fe98d59f89555574715054f50fa31"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":244019,"upload-time":"2022-10-18T20:13:01.999625Z","url":"https://files.pythonhosted.org/packages/2b/0a/36951d279e1d716ab264b04e8ddb12e0c08cc1c7cbd44f2d22c84dc61e33/psutil-5.9.3-cp27-cp27m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"eecee58b65701e8da473f7e6142a96da3fac24d305218d11c887ca28bcd6cd61"},"data-dist-info-metadata":{"sha256":"eecee58b65701e8da473f7e6142a96da3fac24d305218d11c887ca28bcd6cd61"},"filename":"psutil-5.9.3-cp27-cp27mu-manylinux2010_i686.whl","hashes":{"sha256":"fe79b4ad4836e3da6c4650cb85a663b3a51aef22e1a829c384e18fae87e5e727"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":293262,"upload-time":"2022-10-18T20:13:05.579750Z","url":"https://files.pythonhosted.org/packages/5b/9c/5412473100e3213d970c8b9291371816e57f1e4a74296b2e3b8a5c8ebb47/psutil-5.9.3-cp27-cp27mu-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"eecee58b65701e8da473f7e6142a96da3fac24d305218d11c887ca28bcd6cd61"},"data-dist-info-metadata":{"sha256":"eecee58b65701e8da473f7e6142a96da3fac24d305218d11c887ca28bcd6cd61"},"filename":"psutil-5.9.3-cp27-cp27mu-manylinux2010_x86_64.whl","hashes":{"sha256":"db8e62016add2235cc87fb7ea000ede9e4ca0aa1f221b40cef049d02d5d2593d"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":297584,"upload-time":"2022-10-18T20:13:09.701506Z","url":"https://files.pythonhosted.org/packages/8f/5a/e9e98bb3ade26bc7847d5722d0e4a6d437621fa8fc02269d9cba78f6f241/psutil-5.9.3-cp27-cp27mu-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"data-dist-info-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"filename":"psutil-5.9.3-cp310-cp310-macosx_10_9_x86_64.whl","hashes":{"sha256":"941a6c2c591da455d760121b44097781bc970be40e0e43081b9139da485ad5b7"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242575,"upload-time":"2022-10-18T20:13:13.343655Z","url":"https://files.pythonhosted.org/packages/95/90/822c926e170e8a5769ff11edb92ac59dd523df505b5d56cad0ef3f15c325/psutil-5.9.3-cp310-cp310-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"data-dist-info-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"filename":"psutil-5.9.3-cp310-cp310-macosx_11_0_arm64.whl","hashes":{"sha256":"71b1206e7909792d16933a0d2c1c7f04ae196186c51ba8567abae1d041f06dcb"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":243240,"upload-time":"2022-10-18T20:13:17.345095Z","url":"https://files.pythonhosted.org/packages/42/9e/243aa51c3d71355913dafc27c5cb7ffdbe9a42c939a5aace526906bfc721/psutil-5.9.3-cp310-cp310-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"data-dist-info-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"filename":"psutil-5.9.3-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"f57d63a2b5beaf797b87024d018772439f9d3103a395627b77d17a8d72009543"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":289285,"upload-time":"2022-10-18T20:13:20.535587Z","url":"https://files.pythonhosted.org/packages/f7/b0/6925fbfac4c342cb2f8bad1571b48e12802ac8031e1d4453a31e9a12b64d/psutil-5.9.3-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"data-dist-info-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"filename":"psutil-5.9.3-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"e7507f6c7b0262d3e7b0eeda15045bf5881f4ada70473b87bc7b7c93b992a7d7"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":292339,"upload-time":"2022-10-18T20:13:26.635784Z","url":"https://files.pythonhosted.org/packages/ed/2c/483ed7332d74b3fef0f5ba13c192d33f21fe95df5468a7ca040f02bd7af9/psutil-5.9.3-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"ddacf95a106563420486b913e49d19570b225ee95ad876d657e72d3df9c09536"},"data-dist-info-metadata":{"sha256":"ddacf95a106563420486b913e49d19570b225ee95ad876d657e72d3df9c09536"},"filename":"psutil-5.9.3-cp310-cp310-win32.whl","hashes":{"sha256":"1b540599481c73408f6b392cdffef5b01e8ff7a2ac8caae0a91b8222e88e8f1e"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242906,"upload-time":"2022-10-18T20:13:30.051727Z","url":"https://files.pythonhosted.org/packages/55/07/94730401200098b1119dc9f5d3a271e3bf865b31bfa64a2b58a0bbd9d222/psutil-5.9.3-cp310-cp310-win32.whl","yanked":false},{"core-metadata":{"sha256":"ddacf95a106563420486b913e49d19570b225ee95ad876d657e72d3df9c09536"},"data-dist-info-metadata":{"sha256":"ddacf95a106563420486b913e49d19570b225ee95ad876d657e72d3df9c09536"},"filename":"psutil-5.9.3-cp310-cp310-win_amd64.whl","hashes":{"sha256":"547ebb02031fdada635452250ff39942db8310b5c4a8102dfe9384ee5791e650"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":247045,"upload-time":"2022-10-18T20:13:34.024950Z","url":"https://files.pythonhosted.org/packages/37/c0/8a102d4ce45dbc5d04932b52327c4385b88023635e57af9d457ca5ea6bb3/psutil-5.9.3-cp310-cp310-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"0815ae21fa21e764d04c42f06a34a9f89e9b720a9d80f565b7943a08cf8e10ad"},"data-dist-info-metadata":{"sha256":"0815ae21fa21e764d04c42f06a34a9f89e9b720a9d80f565b7943a08cf8e10ad"},"filename":"psutil-5.9.3-cp36-cp36m-macosx_10_9_x86_64.whl","hashes":{"sha256":"d8c3cc6bb76492133474e130a12351a325336c01c96a24aae731abf5a47fe088"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242147,"upload-time":"2022-10-18T20:13:37.172733Z","url":"https://files.pythonhosted.org/packages/61/f2/74908ddbe57863007e3b3a76f39b509bbab9892d0949f1e9d5a888f8ec60/psutil-5.9.3-cp36-cp36m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"0815ae21fa21e764d04c42f06a34a9f89e9b720a9d80f565b7943a08cf8e10ad"},"data-dist-info-metadata":{"sha256":"0815ae21fa21e764d04c42f06a34a9f89e9b720a9d80f565b7943a08cf8e10ad"},"filename":"psutil-5.9.3-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"07d880053c6461c9b89cd5d4808f3b8336665fa3acdefd6777662c5ed73a851a"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":286758,"upload-time":"2022-10-18T20:13:40.932526Z","url":"https://files.pythonhosted.org/packages/30/2f/696c4459864385cc5c63a21f30584dfd99d2130c21c8b3084ffbaa0edd82/psutil-5.9.3-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"0815ae21fa21e764d04c42f06a34a9f89e9b720a9d80f565b7943a08cf8e10ad"},"data-dist-info-metadata":{"sha256":"0815ae21fa21e764d04c42f06a34a9f89e9b720a9d80f565b7943a08cf8e10ad"},"filename":"psutil-5.9.3-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"5e8b50241dd3c2ed498507f87a6602825073c07f3b7e9560c58411c14fe1e1c9"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":290363,"upload-time":"2022-10-18T20:13:43.847649Z","url":"https://files.pythonhosted.org/packages/2c/80/2f3072492a7f14faf4f4565dd26fe1baf4b3fd28557f1427b6708064a622/psutil-5.9.3-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"2ef63858bce55082c5fb7e0b7dabc2fd37fc333286a4d4e5b64df61afb5235de"},"data-dist-info-metadata":{"sha256":"2ef63858bce55082c5fb7e0b7dabc2fd37fc333286a4d4e5b64df61afb5235de"},"filename":"psutil-5.9.3-cp36-cp36m-win32.whl","hashes":{"sha256":"828c9dc9478b34ab96be75c81942d8df0c2bb49edbb481f597314d92b6441d89"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":244701,"upload-time":"2022-10-18T20:13:47.431806Z","url":"https://files.pythonhosted.org/packages/db/e3/10363d747d900f89f7920b8e4060b42cd862b580a69a2b9c9788c4de9035/psutil-5.9.3-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"2ef63858bce55082c5fb7e0b7dabc2fd37fc333286a4d4e5b64df61afb5235de"},"data-dist-info-metadata":{"sha256":"2ef63858bce55082c5fb7e0b7dabc2fd37fc333286a4d4e5b64df61afb5235de"},"filename":"psutil-5.9.3-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"ed15edb14f52925869250b1375f0ff58ca5c4fa8adefe4883cfb0737d32f5c02"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":249356,"upload-time":"2022-10-18T20:13:51.794050Z","url":"https://files.pythonhosted.org/packages/ac/cc/092ca7ae0c5f270bb14720cd8ac86a3fafda25fae31d08d2465eed4498b3/psutil-5.9.3-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"data-dist-info-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"filename":"psutil-5.9.3-cp37-cp37m-macosx_10_9_x86_64.whl","hashes":{"sha256":"d266cd05bd4a95ca1c2b9b5aac50d249cf7c94a542f47e0b22928ddf8b80d1ef"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242154,"upload-time":"2022-10-18T20:14:12.425822Z","url":"https://files.pythonhosted.org/packages/16/51/d431f7db3a3a44d9c03ec1681835a5de52d2f0bb7e28f29ecd806ccc46ec/psutil-5.9.3-cp37-cp37m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"data-dist-info-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"filename":"psutil-5.9.3-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"7e4939ff75149b67aef77980409f156f0082fa36accc475d45c705bb00c6c16a"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":287383,"upload-time":"2022-10-18T20:14:15.962130Z","url":"https://files.pythonhosted.org/packages/94/b0/cd3be14dc74a6f262b1de296841a5141a794cc485d4e3af5c1c0ffc9b886/psutil-5.9.3-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"data-dist-info-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"filename":"psutil-5.9.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"68fa227c32240c52982cb931801c5707a7f96dd8927f9102d6c7771ea1ff5698"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":291396,"upload-time":"2022-10-18T20:14:19.855473Z","url":"https://files.pythonhosted.org/packages/5e/86/856aa554ec7eb843fb006ef125cf4543ee9058cb39ad09d131dd820c71f7/psutil-5.9.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"ddacf95a106563420486b913e49d19570b225ee95ad876d657e72d3df9c09536"},"data-dist-info-metadata":{"sha256":"ddacf95a106563420486b913e49d19570b225ee95ad876d657e72d3df9c09536"},"filename":"psutil-5.9.3-cp37-cp37m-win32.whl","hashes":{"sha256":"beb57d8a1ca0ae0eb3d08ccaceb77e1a6d93606f0e1754f0d60a6ebd5c288837"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":243306,"upload-time":"2022-10-18T20:14:23.302434Z","url":"https://files.pythonhosted.org/packages/ab/10/547feeec01275dd544a389ba05ecb3c316015d4b402cc7b440ca2d98ebcd/psutil-5.9.3-cp37-cp37m-win32.whl","yanked":false},{"core-metadata":{"sha256":"ddacf95a106563420486b913e49d19570b225ee95ad876d657e72d3df9c09536"},"data-dist-info-metadata":{"sha256":"ddacf95a106563420486b913e49d19570b225ee95ad876d657e72d3df9c09536"},"filename":"psutil-5.9.3-cp37-cp37m-win_amd64.whl","hashes":{"sha256":"12500d761ac091f2426567f19f95fd3f15a197d96befb44a5c1e3cbe6db5752c"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":247870,"upload-time":"2022-10-18T20:14:26.971453Z","url":"https://files.pythonhosted.org/packages/0c/f1/50e71c11ef14c592686dfc60e2b42a381fe57af2d22713e66a72c07cf9d1/psutil-5.9.3-cp37-cp37m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"data-dist-info-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"filename":"psutil-5.9.3-cp38-cp38-macosx_10_9_x86_64.whl","hashes":{"sha256":"ba38cf9984d5462b506e239cf4bc24e84ead4b1d71a3be35e66dad0d13ded7c1"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242611,"upload-time":"2022-10-18T20:14:30.842925Z","url":"https://files.pythonhosted.org/packages/01/d6/9ca99b416dddf4a49855a9ebf4af3a2db9526e94e9693da169fa5ed61788/psutil-5.9.3-cp38-cp38-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"data-dist-info-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"filename":"psutil-5.9.3-cp38-cp38-macosx_11_0_arm64.whl","hashes":{"sha256":"46907fa62acaac364fff0b8a9da7b360265d217e4fdeaca0a2397a6883dffba2"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":243235,"upload-time":"2022-10-18T20:14:34.363643Z","url":"https://files.pythonhosted.org/packages/2c/ce/daf28e50305fdbba0754ba58ab0346ec6cfa41293110412f4c6bf74738bb/psutil-5.9.3-cp38-cp38-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"data-dist-info-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"filename":"psutil-5.9.3-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"a04a1836894c8279e5e0a0127c0db8e198ca133d28be8a2a72b4db16f6cf99c1"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":291620,"upload-time":"2022-10-18T20:14:38.179697Z","url":"https://files.pythonhosted.org/packages/b9/cf/56278ae450741b6390491aecaa5f6152ff491bf00544799830e98340ff48/psutil-5.9.3-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"data-dist-info-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"filename":"psutil-5.9.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"8a4e07611997acf178ad13b842377e3d8e9d0a5bac43ece9bfc22a96735d9a4f"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":295598,"upload-time":"2022-10-18T20:14:41.691453Z","url":"https://files.pythonhosted.org/packages/af/5d/9c03a47af929fc12699fcf5174313744eef33a7b9e106e8111f57427b7d7/psutil-5.9.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"ddacf95a106563420486b913e49d19570b225ee95ad876d657e72d3df9c09536"},"data-dist-info-metadata":{"sha256":"ddacf95a106563420486b913e49d19570b225ee95ad876d657e72d3df9c09536"},"filename":"psutil-5.9.3-cp38-cp38-win32.whl","hashes":{"sha256":"6ced1ad823ecfa7d3ce26fe8aa4996e2e53fb49b7fed8ad81c80958501ec0619"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":243663,"upload-time":"2022-10-18T20:14:45.084651Z","url":"https://files.pythonhosted.org/packages/69/3d/e1a12f505eb0171912b94e4689453639bb0deeb70ab4eddbc7b9266f819e/psutil-5.9.3-cp38-cp38-win32.whl","yanked":false},{"core-metadata":{"sha256":"ddacf95a106563420486b913e49d19570b225ee95ad876d657e72d3df9c09536"},"data-dist-info-metadata":{"sha256":"ddacf95a106563420486b913e49d19570b225ee95ad876d657e72d3df9c09536"},"filename":"psutil-5.9.3-cp38-cp38-win_amd64.whl","hashes":{"sha256":"35feafe232d1aaf35d51bd42790cbccb882456f9f18cdc411532902370d660df"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":248005,"upload-time":"2022-10-18T20:14:48.240035Z","url":"https://files.pythonhosted.org/packages/69/cf/47a028bbb4589fdc0494bc60f134c73e319ec78c86c37e2dc66fd118e4db/psutil-5.9.3-cp38-cp38-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"data-dist-info-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"filename":"psutil-5.9.3-cp39-cp39-macosx_10_9_x86_64.whl","hashes":{"sha256":"538fcf6ae856b5e12d13d7da25ad67f02113c96f5989e6ad44422cb5994ca7fc"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242570,"upload-time":"2022-10-18T20:14:51.709705Z","url":"https://files.pythonhosted.org/packages/e5/64/ced1461fd5ebc944d90f9e471149991893bd7ede05b5a88069c1953738dc/psutil-5.9.3-cp39-cp39-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"data-dist-info-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"filename":"psutil-5.9.3-cp39-cp39-macosx_11_0_arm64.whl","hashes":{"sha256":"a3d81165b8474087bb90ec4f333a638ccfd1d69d34a9b4a1a7eaac06648f9fbe"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":243242,"upload-time":"2022-10-18T20:14:54.641746Z","url":"https://files.pythonhosted.org/packages/ac/55/c108e74f22905382aeeef56110bd6c4b89b5fc64944d21cb83acb66faa4c/psutil-5.9.3-cp39-cp39-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"data-dist-info-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"filename":"psutil-5.9.3-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"3a7826e68b0cf4ce2c1ee385d64eab7d70e3133171376cac53d7c1790357ec8f"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":288533,"upload-time":"2022-10-18T20:14:57.902650Z","url":"https://files.pythonhosted.org/packages/db/6f/2441388c48306f9b9d561080c6ba652b4ebd1199faac237069ec8983c8ef/psutil-5.9.3-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"data-dist-info-metadata":{"sha256":"235e00601e80ff0d0cb06ce872f02a457343187ed6fdfb79f6f9f1302b209409"},"filename":"psutil-5.9.3-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"9ec296f565191f89c48f33d9544d8d82b0d2af7dd7d2d4e6319f27a818f8d1cc"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":291401,"upload-time":"2022-10-18T20:15:01.503982Z","url":"https://files.pythonhosted.org/packages/03/47/15604dd812b1b860e81cabaf8c930474c549773389170cd03a093ecf54b6/psutil-5.9.3-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"ddacf95a106563420486b913e49d19570b225ee95ad876d657e72d3df9c09536"},"data-dist-info-metadata":{"sha256":"ddacf95a106563420486b913e49d19570b225ee95ad876d657e72d3df9c09536"},"filename":"psutil-5.9.3-cp39-cp39-win32.whl","hashes":{"sha256":"9ec95df684583b5596c82bb380c53a603bb051cf019d5c849c47e117c5064395"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242954,"upload-time":"2022-10-18T20:15:04.992990Z","url":"https://files.pythonhosted.org/packages/2f/5e/c74dab9858ca67a68a543ad8fefac2aec107383c171019b45ba9ac5223c1/psutil-5.9.3-cp39-cp39-win32.whl","yanked":false},{"core-metadata":{"sha256":"ddacf95a106563420486b913e49d19570b225ee95ad876d657e72d3df9c09536"},"data-dist-info-metadata":{"sha256":"ddacf95a106563420486b913e49d19570b225ee95ad876d657e72d3df9c09536"},"filename":"psutil-5.9.3-cp39-cp39-win_amd64.whl","hashes":{"sha256":"4bd4854f0c83aa84a5a40d3b5d0eb1f3c128f4146371e03baed4589fe4f3c931"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":247081,"upload-time":"2022-10-18T20:15:08.131670Z","url":"https://files.pythonhosted.org/packages/34/31/9aa19bf0fb0cecae904c9e1ac400c5704d935252515da605aa08fca2be86/psutil-5.9.3-cp39-cp39-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.9.3.tar.gz","hashes":{"sha256":"7ccfcdfea4fc4b0a02ca2c31de7fcd186beb9cff8207800e14ab66f79c773af6"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":483579,"upload-time":"2022-10-18T20:15:11.635566Z","url":"https://files.pythonhosted.org/packages/de/eb/1c01a34c86ee3b058c556e407ce5b07cb7d186ebe47b3e69d6f152ca5cc5/psutil-5.9.3.tar.gz","yanked":false},{"core-metadata":{"sha256":"2123949a577021ace561405ef01228f83e4c6d7ccd4ec38ded6fd271e4243960"},"data-dist-info-metadata":{"sha256":"2123949a577021ace561405ef01228f83e4c6d7ccd4ec38ded6fd271e4243960"},"filename":"psutil-5.9.4-cp27-cp27m-macosx_10_9_x86_64.whl","hashes":{"sha256":"c1ca331af862803a42677c120aff8a814a804e09832f166f226bfd22b56feee8"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242993,"upload-time":"2022-11-07T18:44:23.237667Z","url":"https://files.pythonhosted.org/packages/60/f8/b92fecd5297edcecda825a04dfde7cb0a2ecd178eb976cb5a7956e375c6a/psutil-5.9.4-cp27-cp27m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"2123949a577021ace561405ef01228f83e4c6d7ccd4ec38ded6fd271e4243960"},"data-dist-info-metadata":{"sha256":"2123949a577021ace561405ef01228f83e4c6d7ccd4ec38ded6fd271e4243960"},"filename":"psutil-5.9.4-cp27-cp27m-manylinux2010_i686.whl","hashes":{"sha256":"68908971daf802203f3d37e78d3f8831b6d1014864d7a85937941bb35f09aefe"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":294126,"upload-time":"2022-11-07T18:44:28.809923Z","url":"https://files.pythonhosted.org/packages/8e/6b/9a3a5471b74d92dc85bfd71a7f7a55e013b258d86b4c3826ace9d49f7b8c/psutil-5.9.4-cp27-cp27m-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"2123949a577021ace561405ef01228f83e4c6d7ccd4ec38ded6fd271e4243960"},"data-dist-info-metadata":{"sha256":"2123949a577021ace561405ef01228f83e4c6d7ccd4ec38ded6fd271e4243960"},"filename":"psutil-5.9.4-cp27-cp27m-manylinux2010_x86_64.whl","hashes":{"sha256":"3ff89f9b835100a825b14c2808a106b6fdcc4b15483141482a12c725e7f78549"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":298394,"upload-time":"2022-11-07T18:44:34.503099Z","url":"https://files.pythonhosted.org/packages/1d/80/e1502ba4ff65390bd17b4612010762075f64f5a0e7c28e889c4820bd95a9/psutil-5.9.4-cp27-cp27m-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"8fc2f52ee81f3c31e5cf1d7031e3122056d1ec9de55bbc714417091f2e0c5d34"},"data-dist-info-metadata":{"sha256":"8fc2f52ee81f3c31e5cf1d7031e3122056d1ec9de55bbc714417091f2e0c5d34"},"filename":"psutil-5.9.4-cp27-cp27m-win32.whl","hashes":{"sha256":"852dd5d9f8a47169fe62fd4a971aa07859476c2ba22c2254d4a1baa4e10b95ad"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":242001,"upload-time":"2022-11-07T18:44:39.844844Z","url":"https://files.pythonhosted.org/packages/53/ae/536719016fe9399187dbf52cdc65aef942f82b75924495918a2f701bcb77/psutil-5.9.4-cp27-cp27m-win32.whl","yanked":false},{"core-metadata":{"sha256":"8fc2f52ee81f3c31e5cf1d7031e3122056d1ec9de55bbc714417091f2e0c5d34"},"data-dist-info-metadata":{"sha256":"8fc2f52ee81f3c31e5cf1d7031e3122056d1ec9de55bbc714417091f2e0c5d34"},"filename":"psutil-5.9.4-cp27-cp27m-win_amd64.whl","hashes":{"sha256":"9120cd39dca5c5e1c54b59a41d205023d436799b1c8c4d3ff71af18535728e94"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":245334,"upload-time":"2022-11-07T18:44:44.877461Z","url":"https://files.pythonhosted.org/packages/99/9c/7a5761f9d2e79e6f781db5b25eeb9e74c2dc533bc52ee4749cb055a32ce9/psutil-5.9.4-cp27-cp27m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"2123949a577021ace561405ef01228f83e4c6d7ccd4ec38ded6fd271e4243960"},"data-dist-info-metadata":{"sha256":"2123949a577021ace561405ef01228f83e4c6d7ccd4ec38ded6fd271e4243960"},"filename":"psutil-5.9.4-cp27-cp27mu-manylinux2010_i686.whl","hashes":{"sha256":"6b92c532979bafc2df23ddc785ed116fced1f492ad90a6830cf24f4d1ea27d24"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":294138,"upload-time":"2022-11-07T18:44:49.025560Z","url":"https://files.pythonhosted.org/packages/ec/be/b8df2071eda861e65a1b2cec35770bb1f4523737e84a10aa41c53e39e9bc/psutil-5.9.4-cp27-cp27mu-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"2123949a577021ace561405ef01228f83e4c6d7ccd4ec38ded6fd271e4243960"},"data-dist-info-metadata":{"sha256":"2123949a577021ace561405ef01228f83e4c6d7ccd4ec38ded6fd271e4243960"},"filename":"psutil-5.9.4-cp27-cp27mu-manylinux2010_x86_64.whl","hashes":{"sha256":"efeae04f9516907be44904cc7ce08defb6b665128992a56957abc9b61dca94b7"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":298409,"upload-time":"2022-11-07T18:44:56.505271Z","url":"https://files.pythonhosted.org/packages/89/a8/dd2f0866a7e87de751fb5f7c6eca99cbb953c81be76e1814ab3c8c3b0908/psutil-5.9.4-cp27-cp27mu-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"fa7c88f96167219220d13bcc9fd94ae254e582adb547fdbb756d356aa9a18739"},"data-dist-info-metadata":{"sha256":"fa7c88f96167219220d13bcc9fd94ae254e582adb547fdbb756d356aa9a18739"},"filename":"psutil-5.9.4-cp36-abi3-macosx_10_9_x86_64.whl","hashes":{"sha256":"54d5b184728298f2ca8567bf83c422b706200bcbbfafdc06718264f9393cfeb7"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":243468,"upload-time":"2022-11-07T18:45:00.474371Z","url":"https://files.pythonhosted.org/packages/a5/73/35cea01aad1baf901c915dc95ea33a2f271c8ff8cf2f1c73b7f591f1bdf1/psutil-5.9.4-cp36-abi3-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"fa7c88f96167219220d13bcc9fd94ae254e582adb547fdbb756d356aa9a18739"},"data-dist-info-metadata":{"sha256":"fa7c88f96167219220d13bcc9fd94ae254e582adb547fdbb756d356aa9a18739"},"filename":"psutil-5.9.4-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"16653106f3b59386ffe10e0bad3bb6299e169d5327d3f187614b1cb8f24cf2e1"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":277515,"upload-time":"2022-11-07T18:45:05.428752Z","url":"https://files.pythonhosted.org/packages/5a/37/ef88eed265d93bc28c681316f68762c5e04167519e5627a0187c8878b409/psutil-5.9.4-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"fa7c88f96167219220d13bcc9fd94ae254e582adb547fdbb756d356aa9a18739"},"data-dist-info-metadata":{"sha256":"fa7c88f96167219220d13bcc9fd94ae254e582adb547fdbb756d356aa9a18739"},"filename":"psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"54c0d3d8e0078b7666984e11b12b88af2db11d11249a8ac8920dd5ef68a66e08"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":280218,"upload-time":"2022-11-07T18:45:11.831087Z","url":"https://files.pythonhosted.org/packages/6e/c8/784968329c1c67c28cce91991ef9af8a8913aa5a3399a6a8954b1380572f/psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"bca9c896f97136a7e608898e18955f0413786b5391cd3140eec51917f648c864"},"data-dist-info-metadata":{"sha256":"bca9c896f97136a7e608898e18955f0413786b5391cd3140eec51917f648c864"},"filename":"psutil-5.9.4-cp36-abi3-win32.whl","hashes":{"sha256":"149555f59a69b33f056ba1c4eb22bb7bf24332ce631c44a319cec09f876aaeff"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":247217,"upload-time":"2022-11-08T11:50:03.989781Z","url":"https://files.pythonhosted.org/packages/3e/af/fe14b984e8b0f778d502d387b789d846cb2fcc3989f63be942741266d8c8/psutil-5.9.4-cp36-abi3-win32.whl","yanked":false},{"core-metadata":{"sha256":"bca9c896f97136a7e608898e18955f0413786b5391cd3140eec51917f648c864"},"data-dist-info-metadata":{"sha256":"bca9c896f97136a7e608898e18955f0413786b5391cd3140eec51917f648c864"},"filename":"psutil-5.9.4-cp36-abi3-win_amd64.whl","hashes":{"sha256":"fd8522436a6ada7b4aad6638662966de0d61d241cb821239b2ae7013d41a43d4"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":252462,"upload-time":"2022-11-08T11:50:07.829565Z","url":"https://files.pythonhosted.org/packages/25/6e/ba97809175c90cbdcd33b470e466ebf0854d15d1506e605cc0ddd284d5b6/psutil-5.9.4-cp36-abi3-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"3c3c1d22a00986e9736611ca6121b0926aba1c13b75f0b3509aebf5f95b0d409"},"data-dist-info-metadata":{"sha256":"3c3c1d22a00986e9736611ca6121b0926aba1c13b75f0b3509aebf5f95b0d409"},"filename":"psutil-5.9.4-cp38-abi3-macosx_11_0_arm64.whl","hashes":{"sha256":"6001c809253a29599bc0dfd5179d9f8a5779f9dffea1da0f13c53ee568115e1e"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":244234,"upload-time":"2022-11-07T19:53:20.872251Z","url":"https://files.pythonhosted.org/packages/79/26/f026804298b933b11640cc2d15155a545805df732e5ead3a2ad7cf45a38b/psutil-5.9.4-cp38-abi3-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.9.4.tar.gz","hashes":{"sha256":"3d7f9739eb435d4b1338944abe23f49584bde5395f27487d2ee25ad9a8774a62"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":485825,"upload-time":"2022-11-07T19:53:36.245577Z","url":"https://files.pythonhosted.org/packages/3d/7d/d05864a69e452f003c0d77e728e155a89a2a26b09e64860ddd70ad64fb26/psutil-5.9.4.tar.gz","yanked":false},{"core-metadata":{"sha256":"950f52c16a7bc656305be0171e1d567927034839d8c5af80880070db4ba278be"},"data-dist-info-metadata":{"sha256":"950f52c16a7bc656305be0171e1d567927034839d8c5af80880070db4ba278be"},"filename":"psutil-5.9.5-cp27-cp27m-macosx_10_9_x86_64.whl","hashes":{"sha256":"be8929ce4313f9f8146caad4272f6abb8bf99fc6cf59344a3167ecd74f4f203f"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":244852,"upload-time":"2023-04-17T18:24:26.646150Z","url":"https://files.pythonhosted.org/packages/3b/e4/fee119c206545fd37be1e5fa4eeb0c729a52ec2ade4f728ae1fd1acb2a3a/psutil-5.9.5-cp27-cp27m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"950f52c16a7bc656305be0171e1d567927034839d8c5af80880070db4ba278be"},"data-dist-info-metadata":{"sha256":"950f52c16a7bc656305be0171e1d567927034839d8c5af80880070db4ba278be"},"filename":"psutil-5.9.5-cp27-cp27m-manylinux2010_i686.whl","hashes":{"sha256":"ab8ed1a1d77c95453db1ae00a3f9c50227ebd955437bcf2a574ba8adbf6a74d5"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":296014,"upload-time":"2023-04-17T18:24:31.346064Z","url":"https://files.pythonhosted.org/packages/8d/24/ed6b6506f187def39887a91a68e58336eff4cf3e3d5a163ded58bee98624/psutil-5.9.5-cp27-cp27m-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"950f52c16a7bc656305be0171e1d567927034839d8c5af80880070db4ba278be"},"data-dist-info-metadata":{"sha256":"950f52c16a7bc656305be0171e1d567927034839d8c5af80880070db4ba278be"},"filename":"psutil-5.9.5-cp27-cp27m-manylinux2010_x86_64.whl","hashes":{"sha256":"4aef137f3345082a3d3232187aeb4ac4ef959ba3d7c10c33dd73763fbc063da4"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":300274,"upload-time":"2023-04-17T18:24:35.244779Z","url":"https://files.pythonhosted.org/packages/89/fa/ab117fa86195050802207639f5daee857791daaabe9a996935b5b77dbe10/psutil-5.9.5-cp27-cp27m-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"950f52c16a7bc656305be0171e1d567927034839d8c5af80880070db4ba278be"},"data-dist-info-metadata":{"sha256":"950f52c16a7bc656305be0171e1d567927034839d8c5af80880070db4ba278be"},"filename":"psutil-5.9.5-cp27-cp27mu-manylinux2010_i686.whl","hashes":{"sha256":"ea8518d152174e1249c4f2a1c89e3e6065941df2fa13a1ab45327716a23c2b48"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":295979,"upload-time":"2023-04-17T18:24:38.850555Z","url":"https://files.pythonhosted.org/packages/99/f5/ec768e107445f18baa907509aaa0562a4d148a602bd97e8114d79bd6c84d/psutil-5.9.5-cp27-cp27mu-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"950f52c16a7bc656305be0171e1d567927034839d8c5af80880070db4ba278be"},"data-dist-info-metadata":{"sha256":"950f52c16a7bc656305be0171e1d567927034839d8c5af80880070db4ba278be"},"filename":"psutil-5.9.5-cp27-cp27mu-manylinux2010_x86_64.whl","hashes":{"sha256":"acf2aef9391710afded549ff602b5887d7a2349831ae4c26be7c807c0a39fac4"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":300257,"upload-time":"2023-04-17T18:24:42.928863Z","url":"https://files.pythonhosted.org/packages/5f/da/de9d2342db0b7a96863ef84ab94ef1022eec78ece05aac253cddc494e1a7/psutil-5.9.5-cp27-cp27mu-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"b8327a43c6d94aba442f903852217084687e619e3d853297d6743f1cd1a4fada"},"data-dist-info-metadata":{"sha256":"b8327a43c6d94aba442f903852217084687e619e3d853297d6743f1cd1a4fada"},"filename":"psutil-5.9.5-cp27-none-win32.whl","hashes":{"sha256":"5b9b8cb93f507e8dbaf22af6a2fd0ccbe8244bf30b1baad6b3954e935157ae3f"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":243956,"upload-time":"2023-04-17T18:24:46.410452Z","url":"https://files.pythonhosted.org/packages/cf/e3/6af6ec0cbe72f63e9a16d8b53590489e40ed0ff0c99b6a6f05d6af3bb80e/psutil-5.9.5-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"b8327a43c6d94aba442f903852217084687e619e3d853297d6743f1cd1a4fada"},"data-dist-info-metadata":{"sha256":"b8327a43c6d94aba442f903852217084687e619e3d853297d6743f1cd1a4fada"},"filename":"psutil-5.9.5-cp27-none-win_amd64.whl","hashes":{"sha256":"8c5f7c5a052d1d567db4ddd231a9d27a74e8e4a9c3f44b1032762bd7b9fdcd42"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":247338,"upload-time":"2023-04-17T18:24:49.089552Z","url":"https://files.pythonhosted.org/packages/26/f2/dcd8a3cc9c9b1fcd7576a54e3603ce4d1f85672f2687a44050340f7d47b0/psutil-5.9.5-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"612932ce0d3ae556043e5e9a609d33777fe6fb88556dd92d0e07cb53270d1db6"},"data-dist-info-metadata":{"sha256":"612932ce0d3ae556043e5e9a609d33777fe6fb88556dd92d0e07cb53270d1db6"},"filename":"psutil-5.9.5-cp36-abi3-macosx_10_9_x86_64.whl","hashes":{"sha256":"3c6f686f4225553615612f6d9bc21f1c0e305f75d7d8454f9b46e901778e7217"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":245316,"upload-time":"2023-04-17T18:24:52.864585Z","url":"https://files.pythonhosted.org/packages/9a/76/c0195c3443a725c24b3a479f57636dec89efe53d19d435d1752c5188f7de/psutil-5.9.5-cp36-abi3-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"612932ce0d3ae556043e5e9a609d33777fe6fb88556dd92d0e07cb53270d1db6"},"data-dist-info-metadata":{"sha256":"612932ce0d3ae556043e5e9a609d33777fe6fb88556dd92d0e07cb53270d1db6"},"filename":"psutil-5.9.5-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"7a7dd9997128a0d928ed4fb2c2d57e5102bb6089027939f3b722f3a210f9a8da"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":279398,"upload-time":"2023-04-17T18:24:56.977087Z","url":"https://files.pythonhosted.org/packages/e5/2e/56db2b45508ad484b3f22888b3e1adaaf09b8766eaa058ed0e4486c1abae/psutil-5.9.5-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"612932ce0d3ae556043e5e9a609d33777fe6fb88556dd92d0e07cb53270d1db6"},"data-dist-info-metadata":{"sha256":"612932ce0d3ae556043e5e9a609d33777fe6fb88556dd92d0e07cb53270d1db6"},"filename":"psutil-5.9.5-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"89518112647f1276b03ca97b65cc7f64ca587b1eb0278383017c2a0dcc26cbe4"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":282082,"upload-time":"2023-04-17T18:25:00.863664Z","url":"https://files.pythonhosted.org/packages/af/4d/389441079ecef400e2551a3933224885a7bde6b8a4810091d628cdd75afe/psutil-5.9.5-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"612932ce0d3ae556043e5e9a609d33777fe6fb88556dd92d0e07cb53270d1db6"},"data-dist-info-metadata":{"sha256":"612932ce0d3ae556043e5e9a609d33777fe6fb88556dd92d0e07cb53270d1db6"},"filename":"psutil-5.9.5-cp36-abi3-win32.whl","hashes":{"sha256":"104a5cc0e31baa2bcf67900be36acde157756b9c44017b86b2c049f11957887d"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":249834,"upload-time":"2023-04-17T18:25:05.571829Z","url":"https://files.pythonhosted.org/packages/fa/e0/e91277b1cabf5c3f2995c22314553f1be68b17444260101f365c5a5b6ba1/psutil-5.9.5-cp36-abi3-win32.whl","yanked":false},{"core-metadata":{"sha256":"612932ce0d3ae556043e5e9a609d33777fe6fb88556dd92d0e07cb53270d1db6"},"data-dist-info-metadata":{"sha256":"612932ce0d3ae556043e5e9a609d33777fe6fb88556dd92d0e07cb53270d1db6"},"filename":"psutil-5.9.5-cp36-abi3-win_amd64.whl","hashes":{"sha256":"b258c0c1c9d145a1d5ceffab1134441c4c5113b2417fafff7315a917a026c3c9"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":255148,"upload-time":"2023-04-17T18:25:09.779955Z","url":"https://files.pythonhosted.org/packages/86/f3/23e4e4e7ec7855d506ed928756b04735c246b14d9f778ed7ffaae18d8043/psutil-5.9.5-cp36-abi3-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"2ae643bfca9fa3b942bf775226368a8ef859018ea312be94e137a8511ba0da07"},"data-dist-info-metadata":{"sha256":"2ae643bfca9fa3b942bf775226368a8ef859018ea312be94e137a8511ba0da07"},"filename":"psutil-5.9.5-cp38-abi3-macosx_11_0_arm64.whl","hashes":{"sha256":"c607bb3b57dc779d55e1554846352b4e358c10fff3abf3514a7a6601beebdb30"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":246094,"upload-time":"2023-04-17T18:25:14.584295Z","url":"https://files.pythonhosted.org/packages/ed/98/2624954f83489ab13fde2b544baa337d5578c07eee304d320d9ba56e1b1f/psutil-5.9.5-cp38-abi3-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.9.5.tar.gz","hashes":{"sha256":"5410638e4df39c54d957fc51ce03048acd8e6d60abc0f5107af51e5fb566eb3c"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*","size":493489,"upload-time":"2023-04-17T18:25:18.787463Z","url":"https://files.pythonhosted.org/packages/d6/0f/96b7309212a926c1448366e9ce69b081ea79d63265bde33f11cc9cfc2c07/psutil-5.9.5.tar.gz","yanked":false},{"core-metadata":{"sha256":"03f47dbf7a696dbe30acaf142fc193530fe197d70ea954e60b2a6279946aca36"},"data-dist-info-metadata":{"sha256":"03f47dbf7a696dbe30acaf142fc193530fe197d70ea954e60b2a6279946aca36"},"filename":"psutil-5.9.6-cp27-cp27m-macosx_10_9_x86_64.whl","hashes":{"sha256":"fb8a697f11b0f5994550555fcfe3e69799e5b060c8ecf9e2f75c69302cc35c0d"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":245665,"upload-time":"2023-10-15T09:08:50.362285Z","url":"https://files.pythonhosted.org/packages/84/d6/7e23b2b208db3953f630934bc0e9c1736a0a831a781acf8c5891c27b29cf/psutil-5.9.6-cp27-cp27m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"03f47dbf7a696dbe30acaf142fc193530fe197d70ea954e60b2a6279946aca36"},"data-dist-info-metadata":{"sha256":"03f47dbf7a696dbe30acaf142fc193530fe197d70ea954e60b2a6279946aca36"},"filename":"psutil-5.9.6-cp27-cp27m-manylinux2010_i686.whl","hashes":{"sha256":"91ecd2d9c00db9817a4b4192107cf6954addb5d9d67a969a4f436dbc9200f88c"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":297167,"upload-time":"2023-10-15T09:08:52.370112Z","url":"https://files.pythonhosted.org/packages/d2/76/f154e5169756f3d18da160359a404f49f476756809ef21a79afdd0d5b552/psutil-5.9.6-cp27-cp27m-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"03f47dbf7a696dbe30acaf142fc193530fe197d70ea954e60b2a6279946aca36"},"data-dist-info-metadata":{"sha256":"03f47dbf7a696dbe30acaf142fc193530fe197d70ea954e60b2a6279946aca36"},"filename":"psutil-5.9.6-cp27-cp27m-manylinux2010_x86_64.whl","hashes":{"sha256":"10e8c17b4f898d64b121149afb136c53ea8b68c7531155147867b7b1ac9e7e28"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":301338,"upload-time":"2023-10-15T09:08:55.187829Z","url":"https://files.pythonhosted.org/packages/35/e8/5cc0e149ec32a91d459fbe51d0ce3c2dd7f8d67bc1400803ff810247d6dc/psutil-5.9.6-cp27-cp27m-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"03f47dbf7a696dbe30acaf142fc193530fe197d70ea954e60b2a6279946aca36"},"data-dist-info-metadata":{"sha256":"03f47dbf7a696dbe30acaf142fc193530fe197d70ea954e60b2a6279946aca36"},"filename":"psutil-5.9.6-cp27-cp27mu-manylinux2010_i686.whl","hashes":{"sha256":"18cd22c5db486f33998f37e2bb054cc62fd06646995285e02a51b1e08da97017"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":297167,"upload-time":"2023-10-15T09:08:57.863175Z","url":"https://files.pythonhosted.org/packages/8d/f7/074071fa91dab747c8d1fe2eb74da439b3712248d6b254ba0136ada8694f/psutil-5.9.6-cp27-cp27mu-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"03f47dbf7a696dbe30acaf142fc193530fe197d70ea954e60b2a6279946aca36"},"data-dist-info-metadata":{"sha256":"03f47dbf7a696dbe30acaf142fc193530fe197d70ea954e60b2a6279946aca36"},"filename":"psutil-5.9.6-cp27-cp27mu-manylinux2010_x86_64.whl","hashes":{"sha256":"ca2780f5e038379e520281e4c032dddd086906ddff9ef0d1b9dcf00710e5071c"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":301350,"upload-time":"2023-10-15T09:09:00.411072Z","url":"https://files.pythonhosted.org/packages/4a/65/557545149422a7845248641c1c35a0c8ea940c838896320f774072e16523/psutil-5.9.6-cp27-cp27mu-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"cbf967d735616b8f7384d1cb095719fdf508b6e794f91eda3c559b9eff000943"},"data-dist-info-metadata":{"sha256":"cbf967d735616b8f7384d1cb095719fdf508b6e794f91eda3c559b9eff000943"},"filename":"psutil-5.9.6-cp27-none-win32.whl","hashes":{"sha256":"70cb3beb98bc3fd5ac9ac617a327af7e7f826373ee64c80efd4eb2856e5051e9"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":244900,"upload-time":"2023-10-15T09:09:03.051464Z","url":"https://files.pythonhosted.org/packages/b8/23/d5d9e20c4ae7374abe1f826c69ecf2ab52f93827ca2b92c2c51f9aeb9226/psutil-5.9.6-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"cbf967d735616b8f7384d1cb095719fdf508b6e794f91eda3c559b9eff000943"},"data-dist-info-metadata":{"sha256":"cbf967d735616b8f7384d1cb095719fdf508b6e794f91eda3c559b9eff000943"},"filename":"psutil-5.9.6-cp27-none-win_amd64.whl","hashes":{"sha256":"51dc3d54607c73148f63732c727856f5febec1c7c336f8f41fcbd6315cce76ac"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":248205,"upload-time":"2023-10-15T09:09:05.456917Z","url":"https://files.pythonhosted.org/packages/7a/5e/db765b94cb620c04aaea0cb03d8b589905e50ec278130d25646eead8dff0/psutil-5.9.6-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"2e9d8110f5906489070a50d523dc29d42ccde7cff56677d3b49eae7ac8f23006"},"data-dist-info-metadata":{"sha256":"2e9d8110f5906489070a50d523dc29d42ccde7cff56677d3b49eae7ac8f23006"},"filename":"psutil-5.9.6-cp36-abi3-macosx_10_9_x86_64.whl","hashes":{"sha256":"c69596f9fc2f8acd574a12d5f8b7b1ba3765a641ea5d60fb4736bf3c08a8214a"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":246101,"upload-time":"2023-10-15T09:09:08.012635Z","url":"https://files.pythonhosted.org/packages/f8/36/35b12441ba1bc6684c9215191f955415196ca57ca85d88e313bec7f2cf8e/psutil-5.9.6-cp36-abi3-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"2e9d8110f5906489070a50d523dc29d42ccde7cff56677d3b49eae7ac8f23006"},"data-dist-info-metadata":{"sha256":"2e9d8110f5906489070a50d523dc29d42ccde7cff56677d3b49eae7ac8f23006"},"filename":"psutil-5.9.6-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"92e0cc43c524834af53e9d3369245e6cc3b130e78e26100d1f63cdb0abeb3d3c"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":280854,"upload-time":"2023-10-15T09:09:09.832401Z","url":"https://files.pythonhosted.org/packages/61/c8/e684dea1912943347922ab5c05efc94b4ff3d7470038e8afbe3941ef9efe/psutil-5.9.6-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"2e9d8110f5906489070a50d523dc29d42ccde7cff56677d3b49eae7ac8f23006"},"data-dist-info-metadata":{"sha256":"2e9d8110f5906489070a50d523dc29d42ccde7cff56677d3b49eae7ac8f23006"},"filename":"psutil-5.9.6-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"748c9dd2583ed86347ed65d0035f45fa8c851e8d90354c122ab72319b5f366f4"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":283614,"upload-time":"2023-10-15T09:09:12.314910Z","url":"https://files.pythonhosted.org/packages/19/06/4e3fa3c1b79271e933c5ddbad3a48aa2c3d5f592a0fb7c037f3e0f619f4d/psutil-5.9.6-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"2e9d8110f5906489070a50d523dc29d42ccde7cff56677d3b49eae7ac8f23006"},"data-dist-info-metadata":{"sha256":"2e9d8110f5906489070a50d523dc29d42ccde7cff56677d3b49eae7ac8f23006"},"filename":"psutil-5.9.6-cp36-cp36m-win32.whl","hashes":{"sha256":"3ebf2158c16cc69db777e3c7decb3c0f43a7af94a60d72e87b2823aebac3d602"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":250445,"upload-time":"2023-10-15T09:09:14.942143Z","url":"https://files.pythonhosted.org/packages/3f/63/d4a8dace1756b9c84b94683aa80ed0ba8fc7a4421904933b472d59268976/psutil-5.9.6-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"2e9d8110f5906489070a50d523dc29d42ccde7cff56677d3b49eae7ac8f23006"},"data-dist-info-metadata":{"sha256":"2e9d8110f5906489070a50d523dc29d42ccde7cff56677d3b49eae7ac8f23006"},"filename":"psutil-5.9.6-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"ff18b8d1a784b810df0b0fff3bcb50ab941c3b8e2c8de5726f9c71c601c611aa"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":255893,"upload-time":"2023-10-15T09:09:17.467925Z","url":"https://files.pythonhosted.org/packages/ad/00/c87d449746f8962eb9203554b46ab7dcf243be236dcf007372902791b374/psutil-5.9.6-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"3eb79e7ee359462d9b616150575b67510adf8a297f3b4c2b93aeb95daef17fb8"},"data-dist-info-metadata":{"sha256":"3eb79e7ee359462d9b616150575b67510adf8a297f3b4c2b93aeb95daef17fb8"},"filename":"psutil-5.9.6-cp37-abi3-win32.whl","hashes":{"sha256":"a6f01f03bf1843280f4ad16f4bde26b817847b4c1a0db59bf6419807bc5ce05c"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":248489,"upload-time":"2023-10-15T09:09:19.912001Z","url":"https://files.pythonhosted.org/packages/06/ac/f31a0faf98267e63fc6ed046ad2aca68bd79521380026e92fd4921c869aa/psutil-5.9.6-cp37-abi3-win32.whl","yanked":false},{"core-metadata":{"sha256":"3eb79e7ee359462d9b616150575b67510adf8a297f3b4c2b93aeb95daef17fb8"},"data-dist-info-metadata":{"sha256":"3eb79e7ee359462d9b616150575b67510adf8a297f3b4c2b93aeb95daef17fb8"},"filename":"psutil-5.9.6-cp37-abi3-win_amd64.whl","hashes":{"sha256":"6e5fb8dc711a514da83098bc5234264e551ad980cec5f85dabf4d38ed6f15e9a"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":252327,"upload-time":"2023-10-15T09:09:32.052033Z","url":"https://files.pythonhosted.org/packages/c5/b2/699c50fe0b0402a1ccb64ad71313bcb740e735008dd3ab9abeddbe148e45/psutil-5.9.6-cp37-abi3-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"7298b90db12439da117e2817e35cbbb00312edeb4f885274cd1135a533903d6c"},"data-dist-info-metadata":{"sha256":"7298b90db12439da117e2817e35cbbb00312edeb4f885274cd1135a533903d6c"},"filename":"psutil-5.9.6-cp38-abi3-macosx_11_0_arm64.whl","hashes":{"sha256":"daecbcbd29b289aac14ece28eca6a3e60aa361754cf6da3dfb20d4d32b6c7f57"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":246859,"upload-time":"2023-10-15T09:09:34.494297Z","url":"https://files.pythonhosted.org/packages/9e/cb/e4b83c27eea66bc255effc967053f6fce7c14906dd9b43a348ead9f0cfea/psutil-5.9.6-cp38-abi3-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.9.6.tar.gz","hashes":{"sha256":"e4b92ddcd7dd4cdd3f900180ea1e104932c7bce234fb88976e2a3b296441225a"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":496866,"upload-time":"2023-10-15T09:08:46.623978Z","url":"https://files.pythonhosted.org/packages/2d/01/beb7331fc6c8d1c49dd051e3611379bfe379e915c808e1301506027fce9d/psutil-5.9.6.tar.gz","yanked":false},{"core-metadata":{"sha256":"bee12356aebb79b07f3adec05e4a7e1cad1a4e76f3cbf8f7617b97d7598ed4e8"},"data-dist-info-metadata":{"sha256":"bee12356aebb79b07f3adec05e4a7e1cad1a4e76f3cbf8f7617b97d7598ed4e8"},"filename":"psutil-5.9.7-cp27-cp27m-macosx_10_9_x86_64.whl","hashes":{"sha256":"0bd41bf2d1463dfa535942b2a8f0e958acf6607ac0be52265ab31f7923bcd5e6"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":245542,"upload-time":"2023-12-17T11:25:25.875219Z","url":"https://files.pythonhosted.org/packages/3e/16/c86fcf73f02bd0a3d49b0dcabc8ebd4020647be2ea40ff668f717587af97/psutil-5.9.7-cp27-cp27m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"bee12356aebb79b07f3adec05e4a7e1cad1a4e76f3cbf8f7617b97d7598ed4e8"},"data-dist-info-metadata":{"sha256":"bee12356aebb79b07f3adec05e4a7e1cad1a4e76f3cbf8f7617b97d7598ed4e8"},"filename":"psutil-5.9.7-cp27-cp27m-manylinux2010_i686.whl","hashes":{"sha256":"5794944462509e49d4d458f4dbfb92c47539e7d8d15c796f141f474010084056"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":312042,"upload-time":"2023-12-17T11:25:29.439835Z","url":"https://files.pythonhosted.org/packages/93/fc/e45a8e9b2acd54fe80ededa2f7b19de21e776f64e00437417c16c3e139d9/psutil-5.9.7-cp27-cp27m-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"bee12356aebb79b07f3adec05e4a7e1cad1a4e76f3cbf8f7617b97d7598ed4e8"},"data-dist-info-metadata":{"sha256":"bee12356aebb79b07f3adec05e4a7e1cad1a4e76f3cbf8f7617b97d7598ed4e8"},"filename":"psutil-5.9.7-cp27-cp27m-manylinux2010_x86_64.whl","hashes":{"sha256":"fe361f743cb3389b8efda21980d93eb55c1f1e3898269bc9a2a1d0bb7b1f6508"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":312681,"upload-time":"2023-12-17T11:25:33.024609Z","url":"https://files.pythonhosted.org/packages/d7/43/dd7034a3a3a900e95b9dcf47ee710680cfd11a224ab18b31c34370da36a8/psutil-5.9.7-cp27-cp27m-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"bee12356aebb79b07f3adec05e4a7e1cad1a4e76f3cbf8f7617b97d7598ed4e8"},"data-dist-info-metadata":{"sha256":"bee12356aebb79b07f3adec05e4a7e1cad1a4e76f3cbf8f7617b97d7598ed4e8"},"filename":"psutil-5.9.7-cp27-cp27mu-manylinux2010_i686.whl","hashes":{"sha256":"e469990e28f1ad738f65a42dcfc17adaed9d0f325d55047593cb9033a0ab63df"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":312066,"upload-time":"2023-12-17T11:25:37.010306Z","url":"https://files.pythonhosted.org/packages/ff/ea/a47eecddcd97d65b496ac655c9f9ba8af270c203d5ea1630273cfc5ec740/psutil-5.9.7-cp27-cp27mu-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"bee12356aebb79b07f3adec05e4a7e1cad1a4e76f3cbf8f7617b97d7598ed4e8"},"data-dist-info-metadata":{"sha256":"bee12356aebb79b07f3adec05e4a7e1cad1a4e76f3cbf8f7617b97d7598ed4e8"},"filename":"psutil-5.9.7-cp27-cp27mu-manylinux2010_x86_64.whl","hashes":{"sha256":"3c4747a3e2ead1589e647e64aad601981f01b68f9398ddf94d01e3dc0d1e57c7"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":312684,"upload-time":"2023-12-17T11:25:39.891899Z","url":"https://files.pythonhosted.org/packages/cd/ee/d946d0b758120e724d9cdd9607c304ff1eedb9380bf60597c295dc7def6b/psutil-5.9.7-cp27-cp27mu-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"e17b0155d6125da908d40f364ab8665d24296d45f194128d7d4fee06adf24366"},"data-dist-info-metadata":{"sha256":"e17b0155d6125da908d40f364ab8665d24296d45f194128d7d4fee06adf24366"},"filename":"psutil-5.9.7-cp27-none-win32.whl","hashes":{"sha256":"1d4bc4a0148fdd7fd8f38e0498639ae128e64538faa507df25a20f8f7fb2341c"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":244785,"upload-time":"2023-12-17T11:25:42.761256Z","url":"https://files.pythonhosted.org/packages/98/c5/6773a3f1c384ac4863665e167cd4da72433b3020580c0b7c6a7b497e11e2/psutil-5.9.7-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"e17b0155d6125da908d40f364ab8665d24296d45f194128d7d4fee06adf24366"},"data-dist-info-metadata":{"sha256":"e17b0155d6125da908d40f364ab8665d24296d45f194128d7d4fee06adf24366"},"filename":"psutil-5.9.7-cp27-none-win_amd64.whl","hashes":{"sha256":"4c03362e280d06bbbfcd52f29acd79c733e0af33d707c54255d21029b8b32ba6"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":248087,"upload-time":"2023-12-17T11:25:45.316036Z","url":"https://files.pythonhosted.org/packages/2d/91/40ac017db38c9f7f325385dd0dab1be3d4c65e3291100e74d5d7b6a213e8/psutil-5.9.7-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"75f8cc7702054674e6edcb7c6c070053f879e044c73942a152279c8d35fec61c"},"data-dist-info-metadata":{"sha256":"75f8cc7702054674e6edcb7c6c070053f879e044c73942a152279c8d35fec61c"},"filename":"psutil-5.9.7-cp36-abi3-macosx_10_9_x86_64.whl","hashes":{"sha256":"ea36cc62e69a13ec52b2f625c27527f6e4479bca2b340b7a452af55b34fcbe2e"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":245972,"upload-time":"2023-12-17T11:25:48.202730Z","url":"https://files.pythonhosted.org/packages/6c/63/86a4ccc640b4ee1193800f57bbd20b766853c0cdbdbb248a27cdfafe6cbf/psutil-5.9.7-cp36-abi3-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"75f8cc7702054674e6edcb7c6c070053f879e044c73942a152279c8d35fec61c"},"data-dist-info-metadata":{"sha256":"75f8cc7702054674e6edcb7c6c070053f879e044c73942a152279c8d35fec61c"},"filename":"psutil-5.9.7-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"1132704b876e58d277168cd729d64750633d5ff0183acf5b3c986b8466cd0284"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":282514,"upload-time":"2023-12-17T11:25:51.371460Z","url":"https://files.pythonhosted.org/packages/58/80/cc6666b3968646f2d94de66bbc63d701d501f4aa04de43dd7d1f5dc477dd/psutil-5.9.7-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"75f8cc7702054674e6edcb7c6c070053f879e044c73942a152279c8d35fec61c"},"data-dist-info-metadata":{"sha256":"75f8cc7702054674e6edcb7c6c070053f879e044c73942a152279c8d35fec61c"},"filename":"psutil-5.9.7-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"fe8b7f07948f1304497ce4f4684881250cd859b16d06a1dc4d7941eeb6233bfe"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":285469,"upload-time":"2023-12-17T11:25:54.250669Z","url":"https://files.pythonhosted.org/packages/be/fa/f1f626620e3b47e6237dcc64cb8cc1472f139e99422e5b9fa5bbcf457f48/psutil-5.9.7-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"75f8cc7702054674e6edcb7c6c070053f879e044c73942a152279c8d35fec61c"},"data-dist-info-metadata":{"sha256":"75f8cc7702054674e6edcb7c6c070053f879e044c73942a152279c8d35fec61c"},"filename":"psutil-5.9.7-cp36-cp36m-win32.whl","hashes":{"sha256":"b27f8fdb190c8c03914f908a4555159327d7481dac2f01008d483137ef3311a9"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":250357,"upload-time":"2023-12-17T12:38:23.681291Z","url":"https://files.pythonhosted.org/packages/63/16/11dfb52cdccd561da711ee2c127b4c0bd2baf4736d10828c707694f31b90/psutil-5.9.7-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"75f8cc7702054674e6edcb7c6c070053f879e044c73942a152279c8d35fec61c"},"data-dist-info-metadata":{"sha256":"75f8cc7702054674e6edcb7c6c070053f879e044c73942a152279c8d35fec61c"},"filename":"psutil-5.9.7-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"44969859757f4d8f2a9bd5b76eba8c3099a2c8cf3992ff62144061e39ba8568e"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":255808,"upload-time":"2023-12-17T12:38:34.170079Z","url":"https://files.pythonhosted.org/packages/0e/88/9b74b25c63b91ff0403a1b89e258238380b4a88e4116cbae4eaadbb4c17a/psutil-5.9.7-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"8eee389ea25ebe96edb28822645ab1709ede8c32315d59c0c91214e32d607b8e"},"data-dist-info-metadata":{"sha256":"8eee389ea25ebe96edb28822645ab1709ede8c32315d59c0c91214e32d607b8e"},"filename":"psutil-5.9.7-cp37-abi3-win32.whl","hashes":{"sha256":"c727ca5a9b2dd5193b8644b9f0c883d54f1248310023b5ad3e92036c5e2ada68"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":248406,"upload-time":"2023-12-17T12:38:50.326952Z","url":"https://files.pythonhosted.org/packages/7c/b8/dc6ebfc030b47cccc5f5229eeb15e64142b4782796c3ce169ccd60b4d511/psutil-5.9.7-cp37-abi3-win32.whl","yanked":false},{"core-metadata":{"sha256":"8eee389ea25ebe96edb28822645ab1709ede8c32315d59c0c91214e32d607b8e"},"data-dist-info-metadata":{"sha256":"8eee389ea25ebe96edb28822645ab1709ede8c32315d59c0c91214e32d607b8e"},"filename":"psutil-5.9.7-cp37-abi3-win_amd64.whl","hashes":{"sha256":"f37f87e4d73b79e6c5e749440c3113b81d1ee7d26f21c19c47371ddea834f414"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":252245,"upload-time":"2023-12-17T12:39:00.686632Z","url":"https://files.pythonhosted.org/packages/50/28/92b74d95dd991c837813ffac0c79a581a3d129eb0fa7c1dd616d9901e0f3/psutil-5.9.7-cp37-abi3-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"c6272489f71a5b9474dff03296e3f8e100ef6f8ed990f7e4ef444ac6e6a3d6fb"},"data-dist-info-metadata":{"sha256":"c6272489f71a5b9474dff03296e3f8e100ef6f8ed990f7e4ef444ac6e6a3d6fb"},"filename":"psutil-5.9.7-cp38-abi3-macosx_11_0_arm64.whl","hashes":{"sha256":"032f4f2c909818c86cea4fe2cc407f1c0f0cde8e6c6d702b28b8ce0c0d143340"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":246739,"upload-time":"2023-12-17T11:25:57.305436Z","url":"https://files.pythonhosted.org/packages/ba/8a/000d0e80156f0b96c55bda6c60f5ed6543d7b5e893ccab83117e50de1400/psutil-5.9.7-cp38-abi3-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.9.7.tar.gz","hashes":{"sha256":"3f02134e82cfb5d089fddf20bb2e03fd5cd52395321d1c8458a9e58500ff417c"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":498429,"upload-time":"2023-12-17T11:25:21.220127Z","url":"https://files.pythonhosted.org/packages/a0/d0/c9ae661a302931735237791f04cb7086ac244377f78692ba3b3eae3a9619/psutil-5.9.7.tar.gz","yanked":false},{"core-metadata":{"sha256":"8da6995843b9daa47e0867912881271ea684ac8d0bcb3aee77791084ee8c0be4"},"data-dist-info-metadata":{"sha256":"8da6995843b9daa47e0867912881271ea684ac8d0bcb3aee77791084ee8c0be4"},"filename":"psutil-5.9.8-cp27-cp27m-macosx_10_9_x86_64.whl","hashes":{"sha256":"26bd09967ae00920df88e0352a91cff1a78f8d69b3ecabbfe733610c0af486c8"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":248274,"upload-time":"2024-01-19T20:47:14.006890Z","url":"https://files.pythonhosted.org/packages/15/9a/c3e2922e2d672bafd37cf3b9681097c350463cdcf0e286e907ddd6cfb014/psutil-5.9.8-cp27-cp27m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"8da6995843b9daa47e0867912881271ea684ac8d0bcb3aee77791084ee8c0be4"},"data-dist-info-metadata":{"sha256":"8da6995843b9daa47e0867912881271ea684ac8d0bcb3aee77791084ee8c0be4"},"filename":"psutil-5.9.8-cp27-cp27m-manylinux2010_i686.whl","hashes":{"sha256":"05806de88103b25903dff19bb6692bd2e714ccf9e668d050d144012055cbca73"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":314796,"upload-time":"2024-01-19T20:47:17.872998Z","url":"https://files.pythonhosted.org/packages/62/e6/6d62285989d53a83def28ea49b46d3e00462d1273c7c47d9678ee28a0a39/psutil-5.9.8-cp27-cp27m-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"8da6995843b9daa47e0867912881271ea684ac8d0bcb3aee77791084ee8c0be4"},"data-dist-info-metadata":{"sha256":"8da6995843b9daa47e0867912881271ea684ac8d0bcb3aee77791084ee8c0be4"},"filename":"psutil-5.9.8-cp27-cp27m-manylinux2010_x86_64.whl","hashes":{"sha256":"611052c4bc70432ec770d5d54f64206aa7203a101ec273a0cd82418c86503bb7"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":315422,"upload-time":"2024-01-19T20:47:21.442877Z","url":"https://files.pythonhosted.org/packages/38/ba/41815f353f79374c1ad82aba998c666c7209793daf12f4799cfaa7302f29/psutil-5.9.8-cp27-cp27m-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"8da6995843b9daa47e0867912881271ea684ac8d0bcb3aee77791084ee8c0be4"},"data-dist-info-metadata":{"sha256":"8da6995843b9daa47e0867912881271ea684ac8d0bcb3aee77791084ee8c0be4"},"filename":"psutil-5.9.8-cp27-cp27mu-manylinux2010_i686.whl","hashes":{"sha256":"50187900d73c1381ba1454cf40308c2bf6f34268518b3f36a9b663ca87e65e36"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":314802,"upload-time":"2024-01-19T20:47:24.219052Z","url":"https://files.pythonhosted.org/packages/a8/2f/ad80cc502c452e1f207307a7d53533505ca47c503ec6e9f7e2c9fbb367e8/psutil-5.9.8-cp27-cp27mu-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"8da6995843b9daa47e0867912881271ea684ac8d0bcb3aee77791084ee8c0be4"},"data-dist-info-metadata":{"sha256":"8da6995843b9daa47e0867912881271ea684ac8d0bcb3aee77791084ee8c0be4"},"filename":"psutil-5.9.8-cp27-cp27mu-manylinux2010_x86_64.whl","hashes":{"sha256":"02615ed8c5ea222323408ceba16c60e99c3f91639b07da6373fb7e6539abc56d"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":315420,"upload-time":"2024-01-19T20:47:26.828052Z","url":"https://files.pythonhosted.org/packages/e4/c3/357a292dee683282f7a46b752a76c5d56c78bf8f5d9def0ca0d39073344a/psutil-5.9.8-cp27-cp27mu-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"bcd64b1f8b81a5a0ea3cff661abf5d6e3a937f442bb982b9bdcddbfd8608aa9c"},"data-dist-info-metadata":{"sha256":"bcd64b1f8b81a5a0ea3cff661abf5d6e3a937f442bb982b9bdcddbfd8608aa9c"},"filename":"psutil-5.9.8-cp27-none-win32.whl","hashes":{"sha256":"36f435891adb138ed3c9e58c6af3e2e6ca9ac2f365efe1f9cfef2794e6c93b4e"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":248660,"upload-time":"2024-01-19T20:47:29.706532Z","url":"https://files.pythonhosted.org/packages/fe/5f/c26deb822fd3daf8fde4bdb658bf87d9ab1ffd3fca483816e89a9a9a9084/psutil-5.9.8-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"bcd64b1f8b81a5a0ea3cff661abf5d6e3a937f442bb982b9bdcddbfd8608aa9c"},"data-dist-info-metadata":{"sha256":"bcd64b1f8b81a5a0ea3cff661abf5d6e3a937f442bb982b9bdcddbfd8608aa9c"},"filename":"psutil-5.9.8-cp27-none-win_amd64.whl","hashes":{"sha256":"bd1184ceb3f87651a67b2708d4c3338e9b10c5df903f2e3776b62303b26cb631"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":251966,"upload-time":"2024-01-19T20:47:33.134054Z","url":"https://files.pythonhosted.org/packages/32/1d/cf66073d74d6146187e2d0081a7616df4437214afa294ee4f16f80a2f96a/psutil-5.9.8-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"19f679f9f89daead325ce0a0c8de1fba70a554793a47fcdf273fb714ac742a35"},"data-dist-info-metadata":{"sha256":"19f679f9f89daead325ce0a0c8de1fba70a554793a47fcdf273fb714ac742a35"},"filename":"psutil-5.9.8-cp36-abi3-macosx_10_9_x86_64.whl","hashes":{"sha256":"aee678c8720623dc456fa20659af736241f575d79429a0e5e9cf88ae0605cc81"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":248702,"upload-time":"2024-01-19T20:47:36.303498Z","url":"https://files.pythonhosted.org/packages/e7/e3/07ae864a636d70a8a6f58da27cb1179192f1140d5d1da10886ade9405797/psutil-5.9.8-cp36-abi3-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"19f679f9f89daead325ce0a0c8de1fba70a554793a47fcdf273fb714ac742a35"},"data-dist-info-metadata":{"sha256":"19f679f9f89daead325ce0a0c8de1fba70a554793a47fcdf273fb714ac742a35"},"filename":"psutil-5.9.8-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"8cb6403ce6d8e047495a701dc7c5bd788add903f8986d523e3e20b98b733e421"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":285242,"upload-time":"2024-01-19T20:47:39.650099Z","url":"https://files.pythonhosted.org/packages/b3/bd/28c5f553667116b2598b9cc55908ec435cb7f77a34f2bff3e3ca765b0f78/psutil-5.9.8-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"19f679f9f89daead325ce0a0c8de1fba70a554793a47fcdf273fb714ac742a35"},"data-dist-info-metadata":{"sha256":"19f679f9f89daead325ce0a0c8de1fba70a554793a47fcdf273fb714ac742a35"},"filename":"psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"d06016f7f8625a1825ba3732081d77c94589dca78b7a3fc072194851e88461a4"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":288191,"upload-time":"2024-01-19T20:47:43.078208Z","url":"https://files.pythonhosted.org/packages/c5/4f/0e22aaa246f96d6ac87fe5ebb9c5a693fbe8877f537a1022527c47ca43c5/psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"19f679f9f89daead325ce0a0c8de1fba70a554793a47fcdf273fb714ac742a35"},"data-dist-info-metadata":{"sha256":"19f679f9f89daead325ce0a0c8de1fba70a554793a47fcdf273fb714ac742a35"},"filename":"psutil-5.9.8-cp36-cp36m-win32.whl","hashes":{"sha256":"7d79560ad97af658a0f6adfef8b834b53f64746d45b403f225b85c5c2c140eee"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":253203,"upload-time":"2024-01-19T20:47:46.133427Z","url":"https://files.pythonhosted.org/packages/dd/9e/85c3bd5b466d96c091bbd6339881e99106adb43d5d60bde32ac181ab6fef/psutil-5.9.8-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"19f679f9f89daead325ce0a0c8de1fba70a554793a47fcdf273fb714ac742a35"},"data-dist-info-metadata":{"sha256":"19f679f9f89daead325ce0a0c8de1fba70a554793a47fcdf273fb714ac742a35"},"filename":"psutil-5.9.8-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"27cc40c3493bb10de1be4b3f07cae4c010ce715290a5be22b98493509c6299e2"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":258655,"upload-time":"2024-01-19T20:47:48.804624Z","url":"https://files.pythonhosted.org/packages/0b/58/bcffb5ab03ec558e565d2871c01215dde74e11f583fb71e7d2b107200caa/psutil-5.9.8-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"949401d7571a13b3b43062a6c13b80a2a1361c3da4af20751f844e6ee3750021"},"data-dist-info-metadata":{"sha256":"949401d7571a13b3b43062a6c13b80a2a1361c3da4af20751f844e6ee3750021"},"filename":"psutil-5.9.8-cp37-abi3-win32.whl","hashes":{"sha256":"bc56c2a1b0d15aa3eaa5a60c9f3f8e3e565303b465dbf57a1b730e7a2b9844e0"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":251252,"upload-time":"2024-01-19T20:47:52.880124Z","url":"https://files.pythonhosted.org/packages/6e/f5/2aa3a4acdc1e5940b59d421742356f133185667dd190b166dbcfcf5d7b43/psutil-5.9.8-cp37-abi3-win32.whl","yanked":false},{"core-metadata":{"sha256":"949401d7571a13b3b43062a6c13b80a2a1361c3da4af20751f844e6ee3750021"},"data-dist-info-metadata":{"sha256":"949401d7571a13b3b43062a6c13b80a2a1361c3da4af20751f844e6ee3750021"},"filename":"psutil-5.9.8-cp37-abi3-win_amd64.whl","hashes":{"sha256":"8db4c1b57507eef143a15a6884ca10f7c73876cdf5d51e713151c1236a0e68cf"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":255090,"upload-time":"2024-01-19T20:47:56.019799Z","url":"https://files.pythonhosted.org/packages/93/52/3e39d26feae7df0aa0fd510b14012c3678b36ed068f7d78b8d8784d61f0e/psutil-5.9.8-cp37-abi3-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"1526b4912f0f04a9c6523b41c128f4d1c6c666fbbdccb62c6705d7e5747c95cc"},"data-dist-info-metadata":{"sha256":"1526b4912f0f04a9c6523b41c128f4d1c6c666fbbdccb62c6705d7e5747c95cc"},"filename":"psutil-5.9.8-cp38-abi3-macosx_11_0_arm64.whl","hashes":{"sha256":"d16bbddf0693323b8c6123dd804100241da461e41d6e332fb0ba6058f630f8c8"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":249898,"upload-time":"2024-01-19T20:47:59.238740Z","url":"https://files.pythonhosted.org/packages/05/33/2d74d588408caedd065c2497bdb5ef83ce6082db01289a1e1147f6639802/psutil-5.9.8-cp38-abi3-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-5.9.8.tar.gz","hashes":{"sha256":"6be126e3225486dff286a8fb9a06246a5253f4c7c53b475ea5f5ac934e64194c"},"provenance":null,"requires-python":">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*","size":503247,"upload-time":"2024-01-19T20:47:09.517227Z","url":"https://files.pythonhosted.org/packages/90/c7/6dc0a455d111f68ee43f27793971cf03fe29b6ef972042549db29eec39a2/psutil-5.9.8.tar.gz","yanked":false},{"core-metadata":{"sha256":"c8301b547ede1d12db293b91c2eda41064442de769d8466cfdb34279587226a9"},"data-dist-info-metadata":{"sha256":"c8301b547ede1d12db293b91c2eda41064442de769d8466cfdb34279587226a9"},"filename":"psutil-6.0.0-cp27-cp27m-macosx_10_9_x86_64.whl","hashes":{"sha256":"a021da3e881cd935e64a3d0a20983bda0bb4cf80e4f74fa9bfcb1bc5785360c6"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":250527,"upload-time":"2024-06-18T21:40:17.061973Z","url":"https://files.pythonhosted.org/packages/13/e5/35ebd7169008752be5561cafdba3f1634be98193b85fe3d22e883f9fe2e1/psutil-6.0.0-cp27-cp27m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"c8301b547ede1d12db293b91c2eda41064442de769d8466cfdb34279587226a9"},"data-dist-info-metadata":{"sha256":"c8301b547ede1d12db293b91c2eda41064442de769d8466cfdb34279587226a9"},"filename":"psutil-6.0.0-cp27-cp27m-manylinux2010_i686.whl","hashes":{"sha256":"1287c2b95f1c0a364d23bc6f2ea2365a8d4d9b726a3be7294296ff7ba97c17f0"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":316838,"upload-time":"2024-06-18T21:40:32.679396Z","url":"https://files.pythonhosted.org/packages/92/a7/083388ef0964a6d74df51c677b3d761e0866d823d37e3a8823551c0d375d/psutil-6.0.0-cp27-cp27m-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"c8301b547ede1d12db293b91c2eda41064442de769d8466cfdb34279587226a9"},"data-dist-info-metadata":{"sha256":"c8301b547ede1d12db293b91c2eda41064442de769d8466cfdb34279587226a9"},"filename":"psutil-6.0.0-cp27-cp27m-manylinux2010_x86_64.whl","hashes":{"sha256":"a9a3dbfb4de4f18174528d87cc352d1f788b7496991cca33c6996f40c9e3c92c"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":317493,"upload-time":"2024-06-18T21:40:41.710402Z","url":"https://files.pythonhosted.org/packages/52/2f/44b7005f306ea8bfd24aa662b5d0ba6ea1daf29dbd0b6c7bbcd3606373ad/psutil-6.0.0-cp27-cp27m-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"c8301b547ede1d12db293b91c2eda41064442de769d8466cfdb34279587226a9"},"data-dist-info-metadata":{"sha256":"c8301b547ede1d12db293b91c2eda41064442de769d8466cfdb34279587226a9"},"filename":"psutil-6.0.0-cp27-cp27mu-manylinux2010_i686.whl","hashes":{"sha256":"6ec7588fb3ddaec7344a825afe298db83fe01bfaaab39155fa84cf1c0d6b13c3"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":316855,"upload-time":"2024-06-18T21:40:47.752750Z","url":"https://files.pythonhosted.org/packages/81/c9/8cb36769b6636d817be3414ebbb27a9ab3fbe6d13835d00f31e77e1fccce/psutil-6.0.0-cp27-cp27mu-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"c8301b547ede1d12db293b91c2eda41064442de769d8466cfdb34279587226a9"},"data-dist-info-metadata":{"sha256":"c8301b547ede1d12db293b91c2eda41064442de769d8466cfdb34279587226a9"},"filename":"psutil-6.0.0-cp27-cp27mu-manylinux2010_x86_64.whl","hashes":{"sha256":"1e7c870afcb7d91fdea2b37c24aeb08f98b6d67257a5cb0a8bc3ac68d0f1a68c"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":317519,"upload-time":"2024-06-18T21:40:53.708954Z","url":"https://files.pythonhosted.org/packages/14/c0/024ac5369ca160e9ed45ed09247d9d779c460017fbd9aa801fd6eb0f060c/psutil-6.0.0-cp27-cp27mu-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"30df484959758f1144d54b7788b34cc3dde563aa4d6853a6e549f2f022b23c5e"},"data-dist-info-metadata":{"sha256":"30df484959758f1144d54b7788b34cc3dde563aa4d6853a6e549f2f022b23c5e"},"filename":"psutil-6.0.0-cp27-none-win32.whl","hashes":{"sha256":"02b69001f44cc73c1c5279d02b30a817e339ceb258ad75997325e0e6169d8b35"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":249766,"upload-time":"2024-06-18T21:40:58.381272Z","url":"https://files.pythonhosted.org/packages/c5/66/78c9c3020f573c58101dc43a44f6855d01bbbd747e24da2f0c4491200ea3/psutil-6.0.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"30df484959758f1144d54b7788b34cc3dde563aa4d6853a6e549f2f022b23c5e"},"data-dist-info-metadata":{"sha256":"30df484959758f1144d54b7788b34cc3dde563aa4d6853a6e549f2f022b23c5e"},"filename":"psutil-6.0.0-cp27-none-win_amd64.whl","hashes":{"sha256":"21f1fb635deccd510f69f485b87433460a603919b45e2a324ad65b0cc74f8fb1"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":253024,"upload-time":"2024-06-18T21:41:04.548455Z","url":"https://files.pythonhosted.org/packages/e1/3f/2403aa9558bea4d3854b0e5e567bc3dd8e9fbc1fc4453c0aa9aafeb75467/psutil-6.0.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"cce2895e87556b6152233398a38cd33b9b0712906393b6834b1448aec0a505ca"},"data-dist-info-metadata":{"sha256":"cce2895e87556b6152233398a38cd33b9b0712906393b6834b1448aec0a505ca"},"filename":"psutil-6.0.0-cp36-abi3-macosx_10_9_x86_64.whl","hashes":{"sha256":"c588a7e9b1173b6e866756dde596fd4cad94f9399daf99ad8c3258b3cb2b47a0"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":250961,"upload-time":"2024-06-18T21:41:11.662513Z","url":"https://files.pythonhosted.org/packages/0b/37/f8da2fbd29690b3557cca414c1949f92162981920699cd62095a984983bf/psutil-6.0.0-cp36-abi3-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"cce2895e87556b6152233398a38cd33b9b0712906393b6834b1448aec0a505ca"},"data-dist-info-metadata":{"sha256":"cce2895e87556b6152233398a38cd33b9b0712906393b6834b1448aec0a505ca"},"filename":"psutil-6.0.0-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"6ed2440ada7ef7d0d608f20ad89a04ec47d2d3ab7190896cd62ca5fc4fe08bf0"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":287478,"upload-time":"2024-06-18T21:41:16.180526Z","url":"https://files.pythonhosted.org/packages/35/56/72f86175e81c656a01c4401cd3b1c923f891b31fbcebe98985894176d7c9/psutil-6.0.0-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"cce2895e87556b6152233398a38cd33b9b0712906393b6834b1448aec0a505ca"},"data-dist-info-metadata":{"sha256":"cce2895e87556b6152233398a38cd33b9b0712906393b6834b1448aec0a505ca"},"filename":"psutil-6.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"5fd9a97c8e94059b0ef54a7d4baf13b405011176c3b6ff257c247cae0d560ecd"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":290455,"upload-time":"2024-06-18T21:41:29.048203Z","url":"https://files.pythonhosted.org/packages/19/74/f59e7e0d392bc1070e9a70e2f9190d652487ac115bb16e2eff6b22ad1d24/psutil-6.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"cce2895e87556b6152233398a38cd33b9b0712906393b6834b1448aec0a505ca"},"data-dist-info-metadata":{"sha256":"cce2895e87556b6152233398a38cd33b9b0712906393b6834b1448aec0a505ca"},"filename":"psutil-6.0.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl","hashes":{"sha256":"e2e8d0054fc88153ca0544f5c4d554d42e33df2e009c4ff42284ac9ebdef4132"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":292046,"upload-time":"2024-06-18T21:41:33.530555Z","url":"https://files.pythonhosted.org/packages/cd/5f/60038e277ff0a9cc8f0c9ea3d0c5eb6ee1d2470ea3f9389d776432888e47/psutil-6.0.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"cce2895e87556b6152233398a38cd33b9b0712906393b6834b1448aec0a505ca"},"data-dist-info-metadata":{"sha256":"cce2895e87556b6152233398a38cd33b9b0712906393b6834b1448aec0a505ca"},"filename":"psutil-6.0.0-cp36-cp36m-win32.whl","hashes":{"sha256":"fc8c9510cde0146432bbdb433322861ee8c3efbf8589865c8bf8d21cb30c4d14"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":255537,"upload-time":"2024-06-18T21:41:38.034852Z","url":"https://files.pythonhosted.org/packages/cd/ff/39c38910cdb8f02fc9965afb520967a1e9307d53d14879dddd0a4f41f6f8/psutil-6.0.0-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"cce2895e87556b6152233398a38cd33b9b0712906393b6834b1448aec0a505ca"},"data-dist-info-metadata":{"sha256":"cce2895e87556b6152233398a38cd33b9b0712906393b6834b1448aec0a505ca"},"filename":"psutil-6.0.0-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"34859b8d8f423b86e4385ff3665d3f4d94be3cdf48221fbe476e883514fdb71c"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":260973,"upload-time":"2024-06-18T21:41:41.566213Z","url":"https://files.pythonhosted.org/packages/08/88/16dd53af4a84e719e27a5ad7db040231415d8caeb48f019bacafbb4d0002/psutil-6.0.0-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"9e9a9bbc68eaa4d9f9716cef6b0d5f90d4d57007c2008e7301c292c7714a6abc"},"data-dist-info-metadata":{"sha256":"9e9a9bbc68eaa4d9f9716cef6b0d5f90d4d57007c2008e7301c292c7714a6abc"},"filename":"psutil-6.0.0-cp37-abi3-win32.whl","hashes":{"sha256":"a495580d6bae27291324fe60cea0b5a7c23fa36a7cd35035a16d93bdcf076b9d"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":253560,"upload-time":"2024-06-18T21:41:46.067057Z","url":"https://files.pythonhosted.org/packages/8b/20/2ff69ad9c35c3df1858ac4e094f20bd2374d33c8643cf41da8fd7cdcb78b/psutil-6.0.0-cp37-abi3-win32.whl","yanked":false},{"core-metadata":{"sha256":"9e9a9bbc68eaa4d9f9716cef6b0d5f90d4d57007c2008e7301c292c7714a6abc"},"data-dist-info-metadata":{"sha256":"9e9a9bbc68eaa4d9f9716cef6b0d5f90d4d57007c2008e7301c292c7714a6abc"},"filename":"psutil-6.0.0-cp37-abi3-win_amd64.whl","hashes":{"sha256":"33ea5e1c975250a720b3a6609c490db40dae5d83a4eb315170c4fe0d8b1f34b3"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":257399,"upload-time":"2024-06-18T21:41:52.100137Z","url":"https://files.pythonhosted.org/packages/73/44/561092313ae925f3acfaace6f9ddc4f6a9c748704317bad9c8c8f8a36a79/psutil-6.0.0-cp37-abi3-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"c0710bf31d8e9160c90747acceb7692596b32d632efd784e22990fc01fad42dd"},"data-dist-info-metadata":{"sha256":"c0710bf31d8e9160c90747acceb7692596b32d632efd784e22990fc01fad42dd"},"filename":"psutil-6.0.0-cp38-abi3-macosx_11_0_arm64.whl","hashes":{"sha256":"ffe7fc9b6b36beadc8c322f84e1caff51e8703b88eee1da46d1e3a6ae11b4fd0"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":251988,"upload-time":"2024-06-18T21:41:57.337231Z","url":"https://files.pythonhosted.org/packages/7c/06/63872a64c312a24fb9b4af123ee7007a306617da63ff13bcc1432386ead7/psutil-6.0.0-cp38-abi3-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-6.0.0.tar.gz","hashes":{"sha256":"8faae4f310b6d969fa26ca0545338b21f73c6b15db7c4a8d934a5482faa818f2"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":508067,"upload-time":"2024-06-18T21:40:10.559591Z","url":"https://files.pythonhosted.org/packages/18/c7/8c6872f7372eb6a6b2e4708b88419fb46b857f7a2e1892966b851cc79fc9/psutil-6.0.0.tar.gz","yanked":false},{"core-metadata":{"sha256":"c1b2df70b2192969ac8cec0847acb78d7ddd669a788780aad80d1133c89c4a43"},"data-dist-info-metadata":{"sha256":"c1b2df70b2192969ac8cec0847acb78d7ddd669a788780aad80d1133c89c4a43"},"filename":"psutil-6.1.0-cp27-cp27m-macosx_10_9_x86_64.whl","hashes":{"sha256":"ff34df86226c0227c52f38b919213157588a678d049688eded74c76c8ba4a5d0"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":247385,"upload-time":"2024-10-17T21:31:49.162372Z","url":"https://files.pythonhosted.org/packages/cd/8e/87b51bedb52f0fa02a6c9399702912a5059b24c7242fa8ea4fd027cb5238/psutil-6.1.0-cp27-cp27m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"c1b2df70b2192969ac8cec0847acb78d7ddd669a788780aad80d1133c89c4a43"},"data-dist-info-metadata":{"sha256":"c1b2df70b2192969ac8cec0847acb78d7ddd669a788780aad80d1133c89c4a43"},"filename":"psutil-6.1.0-cp27-cp27m-manylinux2010_i686.whl","hashes":{"sha256":"c0e0c00aa18ca2d3b2b991643b799a15fc8f0563d2ebb6040f64ce8dc027b942"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":312067,"upload-time":"2024-10-17T21:31:51.572046Z","url":"https://files.pythonhosted.org/packages/2c/56/99304ecbf1f25a2aa336c66e43a8f9462de70d089d3fbb487991dfd96b37/psutil-6.1.0-cp27-cp27m-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"c1b2df70b2192969ac8cec0847acb78d7ddd669a788780aad80d1133c89c4a43"},"data-dist-info-metadata":{"sha256":"c1b2df70b2192969ac8cec0847acb78d7ddd669a788780aad80d1133c89c4a43"},"filename":"psutil-6.1.0-cp27-cp27m-manylinux2010_x86_64.whl","hashes":{"sha256":"000d1d1ebd634b4efb383f4034437384e44a6d455260aaee2eca1e9c1b55f047"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":312345,"upload-time":"2024-10-17T21:31:53.950865Z","url":"https://files.pythonhosted.org/packages/24/87/7c1eeb2fd86a8eb792b15438a3d25eda05c970924df3457669b50e0c022b/psutil-6.1.0-cp27-cp27m-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"c1b2df70b2192969ac8cec0847acb78d7ddd669a788780aad80d1133c89c4a43"},"data-dist-info-metadata":{"sha256":"c1b2df70b2192969ac8cec0847acb78d7ddd669a788780aad80d1133c89c4a43"},"filename":"psutil-6.1.0-cp27-cp27mu-manylinux2010_i686.whl","hashes":{"sha256":"5cd2bcdc75b452ba2e10f0e8ecc0b57b827dd5d7aaffbc6821b2a9a242823a76"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":312058,"upload-time":"2024-10-17T21:31:55.843636Z","url":"https://files.pythonhosted.org/packages/aa/fe/c94a914040c74b2bbe2ddb2c82b3f9a74d8a40401bb1239b0e949331c957/psutil-6.1.0-cp27-cp27mu-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"c1b2df70b2192969ac8cec0847acb78d7ddd669a788780aad80d1133c89c4a43"},"data-dist-info-metadata":{"sha256":"c1b2df70b2192969ac8cec0847acb78d7ddd669a788780aad80d1133c89c4a43"},"filename":"psutil-6.1.0-cp27-cp27mu-manylinux2010_x86_64.whl","hashes":{"sha256":"045f00a43c737f960d273a83973b2511430d61f283a44c96bf13a6e829ba8fdc"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":312330,"upload-time":"2024-10-17T21:31:57.551704Z","url":"https://files.pythonhosted.org/packages/44/8c/624823d5a5a9ec8635d63b273c3ab1554a4fcc3513f4d0236ff9706f1025/psutil-6.1.0-cp27-cp27mu-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"6449e5b027e55d1dc345bb3b5089a60a968abd4f30577107504c292168ff86ea"},"data-dist-info-metadata":{"sha256":"6449e5b027e55d1dc345bb3b5089a60a968abd4f30577107504c292168ff86ea"},"filename":"psutil-6.1.0-cp27-none-win32.whl","hashes":{"sha256":"9118f27452b70bb1d9ab3198c1f626c2499384935aaf55388211ad982611407e"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":246648,"upload-time":"2024-10-17T21:31:59.369185Z","url":"https://files.pythonhosted.org/packages/da/2b/f4dea5d993d9cd22ad958eea828a41d5d225556123d372f02547c29c4f97/psutil-6.1.0-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"6449e5b027e55d1dc345bb3b5089a60a968abd4f30577107504c292168ff86ea"},"data-dist-info-metadata":{"sha256":"6449e5b027e55d1dc345bb3b5089a60a968abd4f30577107504c292168ff86ea"},"filename":"psutil-6.1.0-cp27-none-win_amd64.whl","hashes":{"sha256":"a8506f6119cff7015678e2bce904a4da21025cc70ad283a53b099e7620061d85"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":249905,"upload-time":"2024-10-17T21:32:01.974050Z","url":"https://files.pythonhosted.org/packages/9f/14/4aa97a7f2e0ac33a050d990ab31686d651ae4ef8c86661fef067f00437b9/psutil-6.1.0-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"f6ba356ca54ff4137597ca1c9c55cca6c6ccfd672a0c976cc370711d0a2652e8"},"data-dist-info-metadata":{"sha256":"f6ba356ca54ff4137597ca1c9c55cca6c6ccfd672a0c976cc370711d0a2652e8"},"filename":"psutil-6.1.0-cp36-abi3-macosx_10_9_x86_64.whl","hashes":{"sha256":"6e2dcd475ce8b80522e51d923d10c7871e45f20918e027ab682f94f1c6351688"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":247762,"upload-time":"2024-10-17T21:32:05.991637Z","url":"https://files.pythonhosted.org/packages/01/9e/8be43078a171381953cfee33c07c0d628594b5dbfc5157847b85022c2c1b/psutil-6.1.0-cp36-abi3-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"31aa42573bd48a390acf1e517f08ce9f05a4954542dc8afb9a8805655df7aa18"},"data-dist-info-metadata":{"sha256":"31aa42573bd48a390acf1e517f08ce9f05a4954542dc8afb9a8805655df7aa18"},"filename":"psutil-6.1.0-cp36-abi3-macosx_11_0_arm64.whl","hashes":{"sha256":"0895b8414afafc526712c498bd9de2b063deaac4021a3b3c34566283464aff8e"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":248777,"upload-time":"2024-10-17T21:32:07.872442Z","url":"https://files.pythonhosted.org/packages/1d/cb/313e80644ea407f04f6602a9e23096540d9dc1878755f3952ea8d3d104be/psutil-6.1.0-cp36-abi3-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"f6ba356ca54ff4137597ca1c9c55cca6c6ccfd672a0c976cc370711d0a2652e8"},"data-dist-info-metadata":{"sha256":"f6ba356ca54ff4137597ca1c9c55cca6c6ccfd672a0c976cc370711d0a2652e8"},"filename":"psutil-6.1.0-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"9dcbfce5d89f1d1f2546a2090f4fcf87c7f669d1d90aacb7d7582addece9fb38"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":284259,"upload-time":"2024-10-17T21:32:10.177301Z","url":"https://files.pythonhosted.org/packages/65/8e/bcbe2025c587b5d703369b6a75b65d41d1367553da6e3f788aff91eaf5bd/psutil-6.1.0-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"f6ba356ca54ff4137597ca1c9c55cca6c6ccfd672a0c976cc370711d0a2652e8"},"data-dist-info-metadata":{"sha256":"f6ba356ca54ff4137597ca1c9c55cca6c6ccfd672a0c976cc370711d0a2652e8"},"filename":"psutil-6.1.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"498c6979f9c6637ebc3a73b3f87f9eb1ec24e1ce53a7c5173b8508981614a90b"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":287255,"upload-time":"2024-10-17T21:32:11.964687Z","url":"https://files.pythonhosted.org/packages/58/4d/8245e6f76a93c98aab285a43ea71ff1b171bcd90c9d238bf81f7021fb233/psutil-6.1.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"f6ba356ca54ff4137597ca1c9c55cca6c6ccfd672a0c976cc370711d0a2652e8"},"data-dist-info-metadata":{"sha256":"f6ba356ca54ff4137597ca1c9c55cca6c6ccfd672a0c976cc370711d0a2652e8"},"filename":"psutil-6.1.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl","hashes":{"sha256":"d905186d647b16755a800e7263d43df08b790d709d575105d419f8b6ef65423a"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":288804,"upload-time":"2024-10-17T21:32:13.785068Z","url":"https://files.pythonhosted.org/packages/27/c2/d034856ac47e3b3cdfa9720d0e113902e615f4190d5d1bdb8df4b2015fb2/psutil-6.1.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"4a33e7577c0dd3577291121b5beb777de2767238e7d66ad068c92fd37d3f3d6a"},"data-dist-info-metadata":{"sha256":"4a33e7577c0dd3577291121b5beb777de2767238e7d66ad068c92fd37d3f3d6a"},"filename":"psutil-6.1.0-cp36-cp36m-win32.whl","hashes":{"sha256":"6d3fbbc8d23fcdcb500d2c9f94e07b1342df8ed71b948a2649b5cb060a7c94ca"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":252360,"upload-time":"2024-10-17T21:32:16.434334Z","url":"https://files.pythonhosted.org/packages/43/39/414d7b67f4df35bb9c373d0fb9a75dd40b223d9bd6d02ebdc7658fd461a3/psutil-6.1.0-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"4a33e7577c0dd3577291121b5beb777de2767238e7d66ad068c92fd37d3f3d6a"},"data-dist-info-metadata":{"sha256":"4a33e7577c0dd3577291121b5beb777de2767238e7d66ad068c92fd37d3f3d6a"},"filename":"psutil-6.1.0-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"1209036fbd0421afde505a4879dee3b2fd7b1e14fee81c0069807adcbbcca747"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":257797,"upload-time":"2024-10-17T21:32:18.946615Z","url":"https://files.pythonhosted.org/packages/ca/da/ef86c99e33be4aa888570e79350caca8c4819b62f84a6d9274c88c40e331/psutil-6.1.0-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"7fdb0cc933f96b13edc5fc4b2255851b3a0222e29777763dd064aaeacaed9cb6"},"data-dist-info-metadata":{"sha256":"7fdb0cc933f96b13edc5fc4b2255851b3a0222e29777763dd064aaeacaed9cb6"},"filename":"psutil-6.1.0-cp37-abi3-win32.whl","hashes":{"sha256":"1ad45a1f5d0b608253b11508f80940985d1d0c8f6111b5cb637533a0e6ddc13e"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":250386,"upload-time":"2024-10-17T21:32:21.399329Z","url":"https://files.pythonhosted.org/packages/ea/55/5389ed243c878725feffc0d6a3bc5ef6764312b6fc7c081faaa2cfa7ef37/psutil-6.1.0-cp37-abi3-win32.whl","yanked":false},{"core-metadata":{"sha256":"7fdb0cc933f96b13edc5fc4b2255851b3a0222e29777763dd064aaeacaed9cb6"},"data-dist-info-metadata":{"sha256":"7fdb0cc933f96b13edc5fc4b2255851b3a0222e29777763dd064aaeacaed9cb6"},"filename":"psutil-6.1.0-cp37-abi3-win_amd64.whl","hashes":{"sha256":"a8fb3752b491d246034fa4d279ff076501588ce8cbcdbb62c32fd7a377d996be"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":254228,"upload-time":"2024-10-17T21:32:23.880601Z","url":"https://files.pythonhosted.org/packages/11/91/87fa6f060e649b1e1a7b19a4f5869709fbf750b7c8c262ee776ec32f3028/psutil-6.1.0-cp37-abi3-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-6.1.0.tar.gz","hashes":{"sha256":"353815f59a7f64cdaca1c0307ee13558a0512f6db064e92fe833784f08539c7a"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":508565,"upload-time":"2024-10-17T21:31:45.680545Z","url":"https://files.pythonhosted.org/packages/26/10/2a30b13c61e7cf937f4adf90710776b7918ed0a9c434e2c38224732af310/psutil-6.1.0.tar.gz","yanked":false},{"core-metadata":{"sha256":"b63844c7465e7a965b315aaeaab5c82b781f31476c21c62e1c9a135bceff2115"},"data-dist-info-metadata":{"sha256":"b63844c7465e7a965b315aaeaab5c82b781f31476c21c62e1c9a135bceff2115"},"filename":"psutil-6.1.1-cp27-cp27m-macosx_10_9_x86_64.whl","hashes":{"sha256":"9ccc4316f24409159897799b83004cb1e24f9819b0dcf9c0b68bdcb6cefee6a8"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":247226,"upload-time":"2024-12-19T18:21:25.276122Z","url":"https://files.pythonhosted.org/packages/09/ea/f8844afff4c8c11d1d0586b737d8d579fd7cb13f1fa3eea599c71877b526/psutil-6.1.1-cp27-cp27m-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"b63844c7465e7a965b315aaeaab5c82b781f31476c21c62e1c9a135bceff2115"},"data-dist-info-metadata":{"sha256":"b63844c7465e7a965b315aaeaab5c82b781f31476c21c62e1c9a135bceff2115"},"filename":"psutil-6.1.1-cp27-cp27m-manylinux2010_i686.whl","hashes":{"sha256":"ca9609c77ea3b8481ab005da74ed894035936223422dc591d6772b147421f777"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":312292,"upload-time":"2024-12-19T18:21:30.930117Z","url":"https://files.pythonhosted.org/packages/51/f8/e376f9410beb915bbf64cb4ae8ce5cf2d03e9a661a2519ebc6a63045a1ca/psutil-6.1.1-cp27-cp27m-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"b63844c7465e7a965b315aaeaab5c82b781f31476c21c62e1c9a135bceff2115"},"data-dist-info-metadata":{"sha256":"b63844c7465e7a965b315aaeaab5c82b781f31476c21c62e1c9a135bceff2115"},"filename":"psutil-6.1.1-cp27-cp27m-manylinux2010_x86_64.whl","hashes":{"sha256":"8df0178ba8a9e5bc84fed9cfa61d54601b371fbec5c8eebad27575f1e105c0d4"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":312542,"upload-time":"2024-12-19T18:21:34.735400Z","url":"https://files.pythonhosted.org/packages/a7/3a/069d6c1e4a7af3cdb162c9ba0737ff9baed1d05cbab6f082f49e3b9ab0a5/psutil-6.1.1-cp27-cp27m-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"b63844c7465e7a965b315aaeaab5c82b781f31476c21c62e1c9a135bceff2115"},"data-dist-info-metadata":{"sha256":"b63844c7465e7a965b315aaeaab5c82b781f31476c21c62e1c9a135bceff2115"},"filename":"psutil-6.1.1-cp27-cp27mu-manylinux2010_i686.whl","hashes":{"sha256":"1924e659d6c19c647e763e78670a05dbb7feaf44a0e9c94bf9e14dfc6ba50468"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":312279,"upload-time":"2024-12-19T18:21:37.897094Z","url":"https://files.pythonhosted.org/packages/81/d5/ee5de2cb8d0c938bb07dcccd4ff7e950359bd6ddbd2fe3118552f863bb52/psutil-6.1.1-cp27-cp27mu-manylinux2010_i686.whl","yanked":false},{"core-metadata":{"sha256":"b63844c7465e7a965b315aaeaab5c82b781f31476c21c62e1c9a135bceff2115"},"data-dist-info-metadata":{"sha256":"b63844c7465e7a965b315aaeaab5c82b781f31476c21c62e1c9a135bceff2115"},"filename":"psutil-6.1.1-cp27-cp27mu-manylinux2010_x86_64.whl","hashes":{"sha256":"018aeae2af92d943fdf1da6b58665124897cfc94faa2ca92098838f83e1b1bca"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":312521,"upload-time":"2024-12-19T18:21:40.651860Z","url":"https://files.pythonhosted.org/packages/37/98/443eff82762b3f2c6a4bd0cdf3bc5c9f62245376c5486b39ee194e920794/psutil-6.1.1-cp27-cp27mu-manylinux2010_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"55f120e7e066d74e328c2e0d559aa667c611e9e63f5a1bc5cae3d034db0be644"},"data-dist-info-metadata":{"sha256":"55f120e7e066d74e328c2e0d559aa667c611e9e63f5a1bc5cae3d034db0be644"},"filename":"psutil-6.1.1-cp27-none-win32.whl","hashes":{"sha256":"6d4281f5bbca041e2292be3380ec56a9413b790579b8e593b1784499d0005dac"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":246855,"upload-time":"2024-12-19T18:54:12.657947Z","url":"https://files.pythonhosted.org/packages/d2/d4/8095b53c4950f44dc99b8d983b796f405ae1f58d80978fcc0421491b4201/psutil-6.1.1-cp27-none-win32.whl","yanked":false},{"core-metadata":{"sha256":"55f120e7e066d74e328c2e0d559aa667c611e9e63f5a1bc5cae3d034db0be644"},"data-dist-info-metadata":{"sha256":"55f120e7e066d74e328c2e0d559aa667c611e9e63f5a1bc5cae3d034db0be644"},"filename":"psutil-6.1.1-cp27-none-win_amd64.whl","hashes":{"sha256":"c777eb75bb33c47377c9af68f30e9f11bc78e0f07fbf907be4a5d70b2fe5f030"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":250110,"upload-time":"2024-12-19T18:54:16.635901Z","url":"https://files.pythonhosted.org/packages/b1/63/0b6425ea4f2375988209a9934c90d6079cc7537847ed58a28fbe30f4277e/psutil-6.1.1-cp27-none-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"06627f791c3970710891bde5c145df9bac4b643a1529e21d25a8c1c679aca404"},"data-dist-info-metadata":{"sha256":"06627f791c3970710891bde5c145df9bac4b643a1529e21d25a8c1c679aca404"},"filename":"psutil-6.1.1-cp36-abi3-macosx_10_9_x86_64.whl","hashes":{"sha256":"fc0ed7fe2231a444fc219b9c42d0376e0a9a1a72f16c5cfa0f68d19f1a0663e8"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":247511,"upload-time":"2024-12-19T18:21:45.163741Z","url":"https://files.pythonhosted.org/packages/61/99/ca79d302be46f7bdd8321089762dd4476ee725fce16fc2b2e1dbba8cac17/psutil-6.1.1-cp36-abi3-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"f53aa914fe0e63af2de701b5d7b1c28a494b1852f4eddac6bec1c1d8eecc00d7"},"data-dist-info-metadata":{"sha256":"f53aa914fe0e63af2de701b5d7b1c28a494b1852f4eddac6bec1c1d8eecc00d7"},"filename":"psutil-6.1.1-cp36-abi3-macosx_11_0_arm64.whl","hashes":{"sha256":"0bdd4eab935276290ad3cb718e9809412895ca6b5b334f5a9111ee6d9aff9377"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":248985,"upload-time":"2024-12-19T18:21:49.254078Z","url":"https://files.pythonhosted.org/packages/0b/6b/73dbde0dd38f3782905d4587049b9be64d76671042fdcaf60e2430c6796d/psutil-6.1.1-cp36-abi3-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"06627f791c3970710891bde5c145df9bac4b643a1529e21d25a8c1c679aca404"},"data-dist-info-metadata":{"sha256":"06627f791c3970710891bde5c145df9bac4b643a1529e21d25a8c1c679aca404"},"filename":"psutil-6.1.1-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"b6e06c20c05fe95a3d7302d74e7097756d4ba1247975ad6905441ae1b5b66003"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":284488,"upload-time":"2024-12-19T18:21:51.638630Z","url":"https://files.pythonhosted.org/packages/17/38/c319d31a1d3f88c5b79c68b3116c129e5133f1822157dd6da34043e32ed6/psutil-6.1.1-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"06627f791c3970710891bde5c145df9bac4b643a1529e21d25a8c1c679aca404"},"data-dist-info-metadata":{"sha256":"06627f791c3970710891bde5c145df9bac4b643a1529e21d25a8c1c679aca404"},"filename":"psutil-6.1.1-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"97f7cb9921fbec4904f522d972f0c0e1f4fabbdd4e0287813b21215074a0f160"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":287477,"upload-time":"2024-12-19T18:21:55.306984Z","url":"https://files.pythonhosted.org/packages/9c/39/0f88a830a1c8a3aba27fededc642da37613c57cbff143412e3536f89784f/psutil-6.1.1-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"06627f791c3970710891bde5c145df9bac4b643a1529e21d25a8c1c679aca404"},"data-dist-info-metadata":{"sha256":"06627f791c3970710891bde5c145df9bac4b643a1529e21d25a8c1c679aca404"},"filename":"psutil-6.1.1-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl","hashes":{"sha256":"33431e84fee02bc84ea36d9e2c4a6d395d479c9dd9bba2376c1f6ee8f3a4e0b3"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":289017,"upload-time":"2024-12-19T18:21:57.875754Z","url":"https://files.pythonhosted.org/packages/47/da/99f4345d4ddf2845cb5b5bd0d93d554e84542d116934fde07a0c50bd4e9f/psutil-6.1.1-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"4899522d2eb9cfd3ec5a1a377c43fb70f9e3963a8989dd7f84c20d6d00083470"},"data-dist-info-metadata":{"sha256":"4899522d2eb9cfd3ec5a1a377c43fb70f9e3963a8989dd7f84c20d6d00083470"},"filename":"psutil-6.1.1-cp36-cp36m-win32.whl","hashes":{"sha256":"384636b1a64b47814437d1173be1427a7c83681b17a450bfc309a1953e329603"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":252576,"upload-time":"2024-12-19T18:22:01.852822Z","url":"https://files.pythonhosted.org/packages/8e/1f/1aebe4dd5914ccba6f7d6cc6d11fb79f6f23f95b858a7f631446bdc5d67f/psutil-6.1.1-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"4899522d2eb9cfd3ec5a1a377c43fb70f9e3963a8989dd7f84c20d6d00083470"},"data-dist-info-metadata":{"sha256":"4899522d2eb9cfd3ec5a1a377c43fb70f9e3963a8989dd7f84c20d6d00083470"},"filename":"psutil-6.1.1-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"8be07491f6ebe1a693f17d4f11e69d0dc1811fa082736500f649f79df7735303"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":258012,"upload-time":"2024-12-19T18:22:04.204308Z","url":"https://files.pythonhosted.org/packages/f4/de/fb4561e59611c19a2d7377c2b2534d11274b8a7df9bb7b7e7f1de5be3641/psutil-6.1.1-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"492e1282d630b4482b89e0c91cd388cea83f8efa851c6c2ffc46a6875c62b52d"},"data-dist-info-metadata":{"sha256":"492e1282d630b4482b89e0c91cd388cea83f8efa851c6c2ffc46a6875c62b52d"},"filename":"psutil-6.1.1-cp37-abi3-win32.whl","hashes":{"sha256":"eaa912e0b11848c4d9279a93d7e2783df352b082f40111e078388701fd479e53"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":250602,"upload-time":"2024-12-19T18:22:08.808295Z","url":"https://files.pythonhosted.org/packages/38/53/bd755c2896f4461fd4f36fa6a6dcb66a88a9e4b9fd4e5b66a77cf9d4a584/psutil-6.1.1-cp37-abi3-win32.whl","yanked":false},{"core-metadata":{"sha256":"492e1282d630b4482b89e0c91cd388cea83f8efa851c6c2ffc46a6875c62b52d"},"data-dist-info-metadata":{"sha256":"492e1282d630b4482b89e0c91cd388cea83f8efa851c6c2ffc46a6875c62b52d"},"filename":"psutil-6.1.1-cp37-abi3-win_amd64.whl","hashes":{"sha256":"f35cfccb065fff93529d2afb4a2e89e363fe63ca1e4a5da22b603a85833c2649"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":254444,"upload-time":"2024-12-19T18:22:11.335598Z","url":"https://files.pythonhosted.org/packages/7b/d7/7831438e6c3ebbfa6e01a927127a6cb42ad3ab844247f3c5b96bea25d73d/psutil-6.1.1-cp37-abi3-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-6.1.1.tar.gz","hashes":{"sha256":"cf8496728c18f2d0b45198f06895be52f36611711746b7f30c464b422b50e2f5"},"provenance":null,"requires-python":"!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7","size":508502,"upload-time":"2024-12-19T18:21:20.568966Z","url":"https://files.pythonhosted.org/packages/1f/5a/07871137bb752428aa4b659f910b399ba6f291156bdea939be3e96cae7cb/psutil-6.1.1.tar.gz","yanked":false},{"core-metadata":{"sha256":"0453e4553a618524d8001e2f64a83b1c24419826cab87bf6b734e6723d383c89"},"data-dist-info-metadata":{"sha256":"0453e4553a618524d8001e2f64a83b1c24419826cab87bf6b734e6723d383c89"},"filename":"psutil-7.0.0-cp36-abi3-macosx_10_9_x86_64.whl","hashes":{"sha256":"101d71dc322e3cffd7cea0650b09b3d08b8e7c4109dd6809fe452dfd00e58b25"},"provenance":null,"requires-python":">=3.6","size":238051,"upload-time":"2025-02-13T21:54:12.360451Z","url":"https://files.pythonhosted.org/packages/ed/e6/2d26234410f8b8abdbf891c9da62bee396583f713fb9f3325a4760875d22/psutil-7.0.0-cp36-abi3-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"65f64bcd26d7284f4f07568c343ced6641bf8235c8c5a09a4986dd2ec5f68de7"},"data-dist-info-metadata":{"sha256":"65f64bcd26d7284f4f07568c343ced6641bf8235c8c5a09a4986dd2ec5f68de7"},"filename":"psutil-7.0.0-cp36-abi3-macosx_11_0_arm64.whl","hashes":{"sha256":"39db632f6bb862eeccf56660871433e111b6ea58f2caea825571951d4b6aa3da"},"provenance":null,"requires-python":">=3.6","size":239535,"upload-time":"2025-02-13T21:54:16.070769Z","url":"https://files.pythonhosted.org/packages/04/8b/30f930733afe425e3cbfc0e1468a30a18942350c1a8816acfade80c005c4/psutil-7.0.0-cp36-abi3-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"0453e4553a618524d8001e2f64a83b1c24419826cab87bf6b734e6723d383c89"},"data-dist-info-metadata":{"sha256":"0453e4553a618524d8001e2f64a83b1c24419826cab87bf6b734e6723d383c89"},"filename":"psutil-7.0.0-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"1fcee592b4c6f146991ca55919ea3d1f8926497a713ed7faaf8225e174581e91"},"provenance":null,"requires-python":">=3.6","size":275004,"upload-time":"2025-02-13T21:54:18.662603Z","url":"https://files.pythonhosted.org/packages/2a/ed/d362e84620dd22876b55389248e522338ed1bf134a5edd3b8231d7207f6d/psutil-7.0.0-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"0453e4553a618524d8001e2f64a83b1c24419826cab87bf6b734e6723d383c89"},"data-dist-info-metadata":{"sha256":"0453e4553a618524d8001e2f64a83b1c24419826cab87bf6b734e6723d383c89"},"filename":"psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"4b1388a4f6875d7e2aff5c4ca1cc16c545ed41dd8bb596cefea80111db353a34"},"provenance":null,"requires-python":">=3.6","size":277986,"upload-time":"2025-02-13T21:54:21.811145Z","url":"https://files.pythonhosted.org/packages/bf/b9/b0eb3f3cbcb734d930fdf839431606844a825b23eaf9a6ab371edac8162c/psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"0453e4553a618524d8001e2f64a83b1c24419826cab87bf6b734e6723d383c89"},"data-dist-info-metadata":{"sha256":"0453e4553a618524d8001e2f64a83b1c24419826cab87bf6b734e6723d383c89"},"filename":"psutil-7.0.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl","hashes":{"sha256":"a5f098451abc2828f7dc6b58d44b532b22f2088f4999a937557b603ce72b1993"},"provenance":null,"requires-python":">=3.6","size":279544,"upload-time":"2025-02-13T21:54:24.680762Z","url":"https://files.pythonhosted.org/packages/eb/a2/709e0fe2f093556c17fbafda93ac032257242cabcc7ff3369e2cb76a97aa/psutil-7.0.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"1326531887a042b91e9db0d6e74bd66dde0a4fd27151ed4b8baeca1fad635542"},"data-dist-info-metadata":{"sha256":"1326531887a042b91e9db0d6e74bd66dde0a4fd27151ed4b8baeca1fad635542"},"filename":"psutil-7.0.0-cp36-cp36m-win32.whl","hashes":{"sha256":"84df4eb63e16849689f76b1ffcb36db7b8de703d1bc1fe41773db487621b6c17"},"provenance":null,"requires-python":">=3.6","size":243024,"upload-time":"2025-02-13T21:54:27.767214Z","url":"https://files.pythonhosted.org/packages/98/04/9e7b8afdad85824dec17de92c121d0fb1907ded624f486b86cd5e8189ebe/psutil-7.0.0-cp36-cp36m-win32.whl","yanked":false},{"core-metadata":{"sha256":"1326531887a042b91e9db0d6e74bd66dde0a4fd27151ed4b8baeca1fad635542"},"data-dist-info-metadata":{"sha256":"1326531887a042b91e9db0d6e74bd66dde0a4fd27151ed4b8baeca1fad635542"},"filename":"psutil-7.0.0-cp36-cp36m-win_amd64.whl","hashes":{"sha256":"1e744154a6580bc968a0195fd25e80432d3afec619daf145b9e5ba16cc1d688e"},"provenance":null,"requires-python":">=3.6","size":248462,"upload-time":"2025-02-13T21:54:31.148496Z","url":"https://files.pythonhosted.org/packages/25/9b/43f2c5f7794a3eba3fc0bb47020d1da44d43ff41c95637c5d760c3ef33eb/psutil-7.0.0-cp36-cp36m-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"8c4198dfca297dfee074ee46394207f317965c25dc1af8ad38862b954795a7c1"},"data-dist-info-metadata":{"sha256":"8c4198dfca297dfee074ee46394207f317965c25dc1af8ad38862b954795a7c1"},"filename":"psutil-7.0.0-cp37-abi3-win32.whl","hashes":{"sha256":"ba3fcef7523064a6c9da440fc4d6bd07da93ac726b5733c29027d7dc95b39d99"},"provenance":null,"requires-python":">=3.6","size":241053,"upload-time":"2025-02-13T21:54:34.310916Z","url":"https://files.pythonhosted.org/packages/50/e6/eecf58810b9d12e6427369784efe814a1eec0f492084ce8eb8f4d89d6d61/psutil-7.0.0-cp37-abi3-win32.whl","yanked":false},{"core-metadata":{"sha256":"8c4198dfca297dfee074ee46394207f317965c25dc1af8ad38862b954795a7c1"},"data-dist-info-metadata":{"sha256":"8c4198dfca297dfee074ee46394207f317965c25dc1af8ad38862b954795a7c1"},"filename":"psutil-7.0.0-cp37-abi3-win_amd64.whl","hashes":{"sha256":"4cf3d4eb1aa9b348dec30105c55cd9b7d4629285735a102beb4441e38db90553"},"provenance":null,"requires-python":">=3.6","size":244885,"upload-time":"2025-02-13T21:54:37.486453Z","url":"https://files.pythonhosted.org/packages/50/1b/6921afe68c74868b4c9fa424dad3be35b095e16687989ebbb50ce4fceb7c/psutil-7.0.0-cp37-abi3-win_amd64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-7.0.0.tar.gz","hashes":{"sha256":"7be9c3eba38beccb6495ea33afd982a44074b78f28c434a1f51cc07fd315c456"},"provenance":null,"requires-python":">=3.6","size":497003,"upload-time":"2025-02-13T21:54:07.946974Z","url":"https://files.pythonhosted.org/packages/2a/80/336820c1ad9286a4ded7e845b2eccfcb27851ab8ac6abece774a6ff4d3de/psutil-7.0.0.tar.gz","yanked":false},{"core-metadata":{"sha256":"ac00a0bcd2510cbd355745e1742a41ef2eadd59546c28788d6122a57d329c6b6"},"data-dist-info-metadata":{"sha256":"ac00a0bcd2510cbd355745e1742a41ef2eadd59546c28788d6122a57d329c6b6"},"filename":"psutil-7.1.0-cp36-abi3-macosx_10_9_x86_64.whl","hashes":{"sha256":"76168cef4397494250e9f4e73eb3752b146de1dd950040b29186d0cce1d5ca13"},"provenance":null,"requires-python":">=3.6","size":245242,"upload-time":"2025-09-17T20:14:56.126572Z","url":"https://files.pythonhosted.org/packages/46/62/ce4051019ee20ce0ed74432dd73a5bb087a6704284a470bb8adff69a0932/psutil-7.1.0-cp36-abi3-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"ac00a0bcd2510cbd355745e1742a41ef2eadd59546c28788d6122a57d329c6b6"},"data-dist-info-metadata":{"sha256":"ac00a0bcd2510cbd355745e1742a41ef2eadd59546c28788d6122a57d329c6b6"},"filename":"psutil-7.1.0-cp36-abi3-macosx_11_0_arm64.whl","hashes":{"sha256":"5d007560c8c372efdff9e4579c2846d71de737e4605f611437255e81efcca2c5"},"provenance":null,"requires-python":">=3.6","size":246682,"upload-time":"2025-09-17T20:14:58.250040Z","url":"https://files.pythonhosted.org/packages/38/61/f76959fba841bf5b61123fbf4b650886dc4094c6858008b5bf73d9057216/psutil-7.1.0-cp36-abi3-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"ac00a0bcd2510cbd355745e1742a41ef2eadd59546c28788d6122a57d329c6b6"},"data-dist-info-metadata":{"sha256":"ac00a0bcd2510cbd355745e1742a41ef2eadd59546c28788d6122a57d329c6b6"},"filename":"psutil-7.1.0-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"22e4454970b32472ce7deaa45d045b34d3648ce478e26a04c7e858a0a6e75ff3"},"provenance":null,"requires-python":">=3.6","size":287994,"upload-time":"2025-09-17T20:14:59.901485Z","url":"https://files.pythonhosted.org/packages/88/7a/37c99d2e77ec30d63398ffa6a660450b8a62517cabe44b3e9bae97696e8d/psutil-7.1.0-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"ac00a0bcd2510cbd355745e1742a41ef2eadd59546c28788d6122a57d329c6b6"},"data-dist-info-metadata":{"sha256":"ac00a0bcd2510cbd355745e1742a41ef2eadd59546c28788d6122a57d329c6b6"},"filename":"psutil-7.1.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"8c70e113920d51e89f212dd7be06219a9b88014e63a4cec69b684c327bc474e3"},"provenance":null,"requires-python":">=3.6","size":291163,"upload-time":"2025-09-17T20:15:01.481447Z","url":"https://files.pythonhosted.org/packages/9d/de/04c8c61232f7244aa0a4b9a9fbd63a89d5aeaf94b2fc9d1d16e2faa5cbb0/psutil-7.1.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"ac00a0bcd2510cbd355745e1742a41ef2eadd59546c28788d6122a57d329c6b6"},"data-dist-info-metadata":{"sha256":"ac00a0bcd2510cbd355745e1742a41ef2eadd59546c28788d6122a57d329c6b6"},"filename":"psutil-7.1.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl","hashes":{"sha256":"7d4a113425c037300de3ac8b331637293da9be9713855c4fc9d2d97436d7259d"},"provenance":null,"requires-python":">=3.6","size":293625,"upload-time":"2025-09-17T20:15:04.492789Z","url":"https://files.pythonhosted.org/packages/f4/58/c4f976234bf6d4737bc8c02a81192f045c307b72cf39c9e5c5a2d78927f6/psutil-7.1.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"24dfa33010cf11743d32bccef7c6bf186df383e24c75c8f904a21dc21061932a"},"data-dist-info-metadata":{"sha256":"24dfa33010cf11743d32bccef7c6bf186df383e24c75c8f904a21dc21061932a"},"filename":"psutil-7.1.0-cp37-abi3-win32.whl","hashes":{"sha256":"09ad740870c8d219ed8daae0ad3b726d3bf9a028a198e7f3080f6a1888b99bca"},"provenance":null,"requires-python":">=3.6","size":244812,"upload-time":"2025-09-17T20:15:07.462276Z","url":"https://files.pythonhosted.org/packages/79/87/157c8e7959ec39ced1b11cc93c730c4fb7f9d408569a6c59dbd92ceb35db/psutil-7.1.0-cp37-abi3-win32.whl","yanked":false},{"core-metadata":{"sha256":"24dfa33010cf11743d32bccef7c6bf186df383e24c75c8f904a21dc21061932a"},"data-dist-info-metadata":{"sha256":"24dfa33010cf11743d32bccef7c6bf186df383e24c75c8f904a21dc21061932a"},"filename":"psutil-7.1.0-cp37-abi3-win_amd64.whl","hashes":{"sha256":"57f5e987c36d3146c0dd2528cd42151cf96cd359b9d67cfff836995cc5df9a3d"},"provenance":null,"requires-python":">=3.6","size":247965,"upload-time":"2025-09-17T20:15:09.673366Z","url":"https://files.pythonhosted.org/packages/bf/e9/b44c4f697276a7a95b8e94d0e320a7bf7f3318521b23de69035540b39838/psutil-7.1.0-cp37-abi3-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"24a51389a00af9d96fada7f4a79aed176d3a74f0ae399a5305d12ce9784f2635"},"data-dist-info-metadata":{"sha256":"24a51389a00af9d96fada7f4a79aed176d3a74f0ae399a5305d12ce9784f2635"},"filename":"psutil-7.1.0-cp37-abi3-win_arm64.whl","hashes":{"sha256":"6937cb68133e7c97b6cc9649a570c9a18ba0efebed46d8c5dae4c07fa1b67a07"},"provenance":null,"requires-python":">=3.6","size":244971,"upload-time":"2025-09-17T20:15:12.262753Z","url":"https://files.pythonhosted.org/packages/26/65/1070a6e3c036f39142c2820c4b52e9243246fcfc3f96239ac84472ba361e/psutil-7.1.0-cp37-abi3-win_arm64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-7.1.0.tar.gz","hashes":{"sha256":"655708b3c069387c8b77b072fc429a57d0e214221d01c0a772df7dfedcb3bcd2"},"provenance":null,"requires-python":">=3.6","size":497660,"upload-time":"2025-09-17T20:14:52.902036Z","url":"https://files.pythonhosted.org/packages/b3/31/4723d756b59344b643542936e37a31d1d3204bcdc42a7daa8ee9eb06fb50/psutil-7.1.0.tar.gz","yanked":false},{"core-metadata":{"sha256":"d94db19e208334cc22bb81d2d0102f837a1101c98a0d350fa3d257181ae1ef09"},"data-dist-info-metadata":{"sha256":"d94db19e208334cc22bb81d2d0102f837a1101c98a0d350fa3d257181ae1ef09"},"filename":"psutil-7.1.1-cp36-abi3-macosx_10_9_x86_64.whl","hashes":{"sha256":"8fa59d7b1f01f0337f12cd10dbd76e4312a4d3c730a4fedcbdd4e5447a8b8460"},"provenance":null,"requires-python":">=3.6","size":244221,"upload-time":"2025-10-19T15:44:03.145914Z","url":"https://files.pythonhosted.org/packages/51/30/f97f8fb1f9ecfbeae4b5ca738dcae66ab28323b5cfbc96cb5565f3754056/psutil-7.1.1-cp36-abi3-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"d94db19e208334cc22bb81d2d0102f837a1101c98a0d350fa3d257181ae1ef09"},"data-dist-info-metadata":{"sha256":"d94db19e208334cc22bb81d2d0102f837a1101c98a0d350fa3d257181ae1ef09"},"filename":"psutil-7.1.1-cp36-abi3-macosx_11_0_arm64.whl","hashes":{"sha256":"2a95104eae85d088891716db676f780c1404fc15d47fde48a46a5d61e8f5ad2c"},"provenance":null,"requires-python":">=3.6","size":245660,"upload-time":"2025-10-19T15:44:05.657308Z","url":"https://files.pythonhosted.org/packages/7b/98/b8d1f61ebf35f4dbdbaabadf9208282d8adc820562f0257e5e6e79e67bf2/psutil-7.1.1-cp36-abi3-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"d94db19e208334cc22bb81d2d0102f837a1101c98a0d350fa3d257181ae1ef09"},"data-dist-info-metadata":{"sha256":"d94db19e208334cc22bb81d2d0102f837a1101c98a0d350fa3d257181ae1ef09"},"filename":"psutil-7.1.1-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","hashes":{"sha256":"98629cd8567acefcc45afe2f4ba1e9290f579eacf490a917967decce4b74ee9b"},"provenance":null,"requires-python":">=3.6","size":286963,"upload-time":"2025-10-19T15:44:08.877299Z","url":"https://files.pythonhosted.org/packages/f0/4a/b8015d7357fefdfe34bc4a3db48a107bae4bad0b94fb6eb0613f09a08ada/psutil-7.1.1-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl","yanked":false},{"core-metadata":{"sha256":"d94db19e208334cc22bb81d2d0102f837a1101c98a0d350fa3d257181ae1ef09"},"data-dist-info-metadata":{"sha256":"d94db19e208334cc22bb81d2d0102f837a1101c98a0d350fa3d257181ae1ef09"},"filename":"psutil-7.1.1-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","hashes":{"sha256":"92ebc58030fb054fa0f26c3206ef01c31c29d67aee1367e3483c16665c25c8d2"},"provenance":null,"requires-python":">=3.6","size":290118,"upload-time":"2025-10-19T15:44:11.897889Z","url":"https://files.pythonhosted.org/packages/3d/3c/b56076bb35303d0733fc47b110a1c9cce081a05ae2e886575a3587c1ee76/psutil-7.1.1-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"d94db19e208334cc22bb81d2d0102f837a1101c98a0d350fa3d257181ae1ef09"},"data-dist-info-metadata":{"sha256":"d94db19e208334cc22bb81d2d0102f837a1101c98a0d350fa3d257181ae1ef09"},"filename":"psutil-7.1.1-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl","hashes":{"sha256":"146a704f224fb2ded2be3da5ac67fc32b9ea90c45b51676f9114a6ac45616967"},"provenance":null,"requires-python":">=3.6","size":292587,"upload-time":"2025-10-19T15:44:14.670833Z","url":"https://files.pythonhosted.org/packages/dc/af/c13d360c0adc6f6218bf9e2873480393d0f729c8dd0507d171f53061c0d3/psutil-7.1.1-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"b46a0b2c74ca7acef22d45a6a1b88ac5c5a55793782e1b66e08e8ac56f33ddb1"},"data-dist-info-metadata":{"sha256":"b46a0b2c74ca7acef22d45a6a1b88ac5c5a55793782e1b66e08e8ac56f33ddb1"},"filename":"psutil-7.1.1-cp37-abi3-win32.whl","hashes":{"sha256":"295c4025b5cd880f7445e4379e6826f7307e3d488947bf9834e865e7847dc5f7"},"provenance":null,"requires-python":">=3.6","size":243772,"upload-time":"2025-10-19T15:44:16.938205Z","url":"https://files.pythonhosted.org/packages/90/2d/c933e7071ba60c7862813f2c7108ec4cf8304f1c79660efeefd0de982258/psutil-7.1.1-cp37-abi3-win32.whl","yanked":false},{"core-metadata":{"sha256":"b46a0b2c74ca7acef22d45a6a1b88ac5c5a55793782e1b66e08e8ac56f33ddb1"},"data-dist-info-metadata":{"sha256":"b46a0b2c74ca7acef22d45a6a1b88ac5c5a55793782e1b66e08e8ac56f33ddb1"},"filename":"psutil-7.1.1-cp37-abi3-win_amd64.whl","hashes":{"sha256":"9b4f17c5f65e44f69bd3a3406071a47b79df45cf2236d1f717970afcb526bcd3"},"provenance":null,"requires-python":">=3.6","size":246936,"upload-time":"2025-10-19T15:44:18.663465Z","url":"https://files.pythonhosted.org/packages/be/f3/11fd213fff15427bc2853552138760c720fd65032d99edfb161910d04127/psutil-7.1.1-cp37-abi3-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"41f2736f936aaaa81486aad04717acc54669b33ae5ad93df72d7a7b26aa50fa7"},"data-dist-info-metadata":{"sha256":"41f2736f936aaaa81486aad04717acc54669b33ae5ad93df72d7a7b26aa50fa7"},"filename":"psutil-7.1.1-cp37-abi3-win_arm64.whl","hashes":{"sha256":"5457cf741ca13da54624126cd5d333871b454ab133999a9a103fb097a7d7d21a"},"provenance":null,"requires-python":">=3.6","size":243944,"upload-time":"2025-10-19T15:44:20.666512Z","url":"https://files.pythonhosted.org/packages/0a/8d/8a9a45c8b655851f216c1d44f68e3533dc8d2c752ccd0f61f1aa73be4893/psutil-7.1.1-cp37-abi3-win_arm64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-7.1.1.tar.gz","hashes":{"sha256":"092b6350145007389c1cfe5716050f02030a05219d90057ea867d18fe8d372fc"},"provenance":null,"requires-python":">=3.6","size":487067,"upload-time":"2025-10-19T15:43:59.373160Z","url":"https://files.pythonhosted.org/packages/89/fc/889242351a932d6183eec5df1fc6539b6f36b6a88444f1e63f18668253aa/psutil-7.1.1.tar.gz","yanked":false},{"core-metadata":{"sha256":"e7b235bb0ba92037ddba6bcb227d0db55f64a7d54b2302a64ee7d8ba78b81c44"},"data-dist-info-metadata":{"sha256":"e7b235bb0ba92037ddba6bcb227d0db55f64a7d54b2302a64ee7d8ba78b81c44"},"filename":"psutil-7.1.2-cp313-cp313t-macosx_10_13_x86_64.whl","hashes":{"sha256":"0cc5c6889b9871f231ed5455a9a02149e388fffcb30b607fb7a8896a6d95f22e"},"provenance":null,"requires-python":">=3.6","size":238575,"upload-time":"2025-10-25T10:46:38.728747Z","url":"https://files.pythonhosted.org/packages/b8/d9/b56cc9f883140ac10021a8c9b0f4e16eed1ba675c22513cdcbce3ba64014/psutil-7.1.2-cp313-cp313t-macosx_10_13_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"e7b235bb0ba92037ddba6bcb227d0db55f64a7d54b2302a64ee7d8ba78b81c44"},"data-dist-info-metadata":{"sha256":"e7b235bb0ba92037ddba6bcb227d0db55f64a7d54b2302a64ee7d8ba78b81c44"},"filename":"psutil-7.1.2-cp313-cp313t-macosx_11_0_arm64.whl","hashes":{"sha256":"8e9e77a977208d84aa363a4a12e0f72189d58bbf4e46b49aae29a2c6e93ef206"},"provenance":null,"requires-python":">=3.6","size":239297,"upload-time":"2025-10-25T10:46:41.347184Z","url":"https://files.pythonhosted.org/packages/36/eb/28d22de383888deb252c818622196e709da98816e296ef95afda33f1c0a2/psutil-7.1.2-cp313-cp313t-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"e7b235bb0ba92037ddba6bcb227d0db55f64a7d54b2302a64ee7d8ba78b81c44"},"data-dist-info-metadata":{"sha256":"e7b235bb0ba92037ddba6bcb227d0db55f64a7d54b2302a64ee7d8ba78b81c44"},"filename":"psutil-7.1.2-cp313-cp313t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","hashes":{"sha256":"7d9623a5e4164d2220ecceb071f4b333b3c78866141e8887c072129185f41278"},"provenance":null,"requires-python":">=3.6","size":280420,"upload-time":"2025-10-25T10:46:44.122205Z","url":"https://files.pythonhosted.org/packages/89/5d/220039e2f28cc129626e54d63892ab05c0d56a29818bfe7268dcb5008932/psutil-7.1.2-cp313-cp313t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"e7b235bb0ba92037ddba6bcb227d0db55f64a7d54b2302a64ee7d8ba78b81c44"},"data-dist-info-metadata":{"sha256":"e7b235bb0ba92037ddba6bcb227d0db55f64a7d54b2302a64ee7d8ba78b81c44"},"filename":"psutil-7.1.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","hashes":{"sha256":"364b1c10fe4ed59c89ec49e5f1a70da353b27986fa8233b4b999df4742a5ee2f"},"provenance":null,"requires-python":">=3.6","size":283049,"upload-time":"2025-10-25T10:46:47.095973Z","url":"https://files.pythonhosted.org/packages/ba/7a/286f0e1c167445b2ef4a6cbdfc8c59fdb45a5a493788950cf8467201dc73/psutil-7.1.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"b27a63f31491a49efc9ff48050f5fdcca4606a487fac91e6cc46853031038b01"},"data-dist-info-metadata":{"sha256":"b27a63f31491a49efc9ff48050f5fdcca4606a487fac91e6cc46853031038b01"},"filename":"psutil-7.1.2-cp313-cp313t-win_amd64.whl","hashes":{"sha256":"f101ef84de7e05d41310e3ccbdd65a6dd1d9eed85e8aaf0758405d022308e204"},"provenance":null,"requires-python":">=3.6","size":248713,"upload-time":"2025-10-25T10:46:49.573269Z","url":"https://files.pythonhosted.org/packages/aa/cc/7eb93260794a42e39b976f3a4dde89725800b9f573b014fac142002a5c98/psutil-7.1.2-cp313-cp313t-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"b27a63f31491a49efc9ff48050f5fdcca4606a487fac91e6cc46853031038b01"},"data-dist-info-metadata":{"sha256":"b27a63f31491a49efc9ff48050f5fdcca4606a487fac91e6cc46853031038b01"},"filename":"psutil-7.1.2-cp313-cp313t-win_arm64.whl","hashes":{"sha256":"20c00824048a95de67f00afedc7b08b282aa08638585b0206a9fb51f28f1a165"},"provenance":null,"requires-python":">=3.6","size":244644,"upload-time":"2025-10-25T10:46:51.924062Z","url":"https://files.pythonhosted.org/packages/ab/1a/0681a92b53366e01f0a099f5237d0c8a2f79d322ac589cccde5e30c8a4e2/psutil-7.1.2-cp313-cp313t-win_arm64.whl","yanked":false},{"core-metadata":{"sha256":"e7b235bb0ba92037ddba6bcb227d0db55f64a7d54b2302a64ee7d8ba78b81c44"},"data-dist-info-metadata":{"sha256":"e7b235bb0ba92037ddba6bcb227d0db55f64a7d54b2302a64ee7d8ba78b81c44"},"filename":"psutil-7.1.2-cp314-cp314t-macosx_10_15_x86_64.whl","hashes":{"sha256":"e09cfe92aa8e22b1ec5e2d394820cf86c5dff6367ac3242366485dfa874d43bc"},"provenance":null,"requires-python":">=3.6","size":238640,"upload-time":"2025-10-25T10:46:54.089898Z","url":"https://files.pythonhosted.org/packages/56/9e/f1c5c746b4ed5320952acd3002d3962fe36f30524c00ea79fdf954cc6779/psutil-7.1.2-cp314-cp314t-macosx_10_15_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"e7b235bb0ba92037ddba6bcb227d0db55f64a7d54b2302a64ee7d8ba78b81c44"},"data-dist-info-metadata":{"sha256":"e7b235bb0ba92037ddba6bcb227d0db55f64a7d54b2302a64ee7d8ba78b81c44"},"filename":"psutil-7.1.2-cp314-cp314t-macosx_11_0_arm64.whl","hashes":{"sha256":"fa6342cf859c48b19df3e4aa170e4cfb64aadc50b11e06bb569c6c777b089c9e"},"provenance":null,"requires-python":">=3.6","size":239303,"upload-time":"2025-10-25T10:46:56.932071Z","url":"https://files.pythonhosted.org/packages/32/ee/fd26216a735395cc25c3899634e34aeb41fb1f3dbb44acc67d9e594be562/psutil-7.1.2-cp314-cp314t-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"e7b235bb0ba92037ddba6bcb227d0db55f64a7d54b2302a64ee7d8ba78b81c44"},"data-dist-info-metadata":{"sha256":"e7b235bb0ba92037ddba6bcb227d0db55f64a7d54b2302a64ee7d8ba78b81c44"},"filename":"psutil-7.1.2-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","hashes":{"sha256":"625977443498ee7d6c1e63e93bacca893fd759a66c5f635d05e05811d23fb5ee"},"provenance":null,"requires-python":">=3.6","size":281717,"upload-time":"2025-10-25T10:46:59.116890Z","url":"https://files.pythonhosted.org/packages/3c/cd/7d96eaec4ef7742b845a9ce2759a2769ecce4ab7a99133da24abacbc9e41/psutil-7.1.2-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"e7b235bb0ba92037ddba6bcb227d0db55f64a7d54b2302a64ee7d8ba78b81c44"},"data-dist-info-metadata":{"sha256":"e7b235bb0ba92037ddba6bcb227d0db55f64a7d54b2302a64ee7d8ba78b81c44"},"filename":"psutil-7.1.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","hashes":{"sha256":"4a24bcd7b7f2918d934af0fb91859f621b873d6aa81267575e3655cd387572a7"},"provenance":null,"requires-python":">=3.6","size":284575,"upload-time":"2025-10-25T10:47:00.944625Z","url":"https://files.pythonhosted.org/packages/bc/1a/7f0b84bdb067d35fe7fade5fff888408688caf989806ce2d6dae08c72dd5/psutil-7.1.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"b27a63f31491a49efc9ff48050f5fdcca4606a487fac91e6cc46853031038b01"},"data-dist-info-metadata":{"sha256":"b27a63f31491a49efc9ff48050f5fdcca4606a487fac91e6cc46853031038b01"},"filename":"psutil-7.1.2-cp314-cp314t-win_amd64.whl","hashes":{"sha256":"329f05610da6380982e6078b9d0881d9ab1e9a7eb7c02d833bfb7340aa634e31"},"provenance":null,"requires-python":">=3.6","size":249491,"upload-time":"2025-10-25T10:47:03.174087Z","url":"https://files.pythonhosted.org/packages/de/05/7820ef8f7b275268917e0c750eada5834581206d9024ca88edce93c4b762/psutil-7.1.2-cp314-cp314t-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"b27a63f31491a49efc9ff48050f5fdcca4606a487fac91e6cc46853031038b01"},"data-dist-info-metadata":{"sha256":"b27a63f31491a49efc9ff48050f5fdcca4606a487fac91e6cc46853031038b01"},"filename":"psutil-7.1.2-cp314-cp314t-win_arm64.whl","hashes":{"sha256":"7b04c29e3c0c888e83ed4762b70f31e65c42673ea956cefa8ced0e31e185f582"},"provenance":null,"requires-python":">=3.6","size":244880,"upload-time":"2025-10-25T10:47:05.228789Z","url":"https://files.pythonhosted.org/packages/db/9a/58de399c7cb58489f08498459ff096cd76b3f1ddc4f224ec2c5ef729c7d0/psutil-7.1.2-cp314-cp314t-win_arm64.whl","yanked":false},{"core-metadata":{"sha256":"9060d8b9832fb047375c7eeac0d3ca41668a9f56cea96f12e30ce72cc4acea8b"},"data-dist-info-metadata":{"sha256":"9060d8b9832fb047375c7eeac0d3ca41668a9f56cea96f12e30ce72cc4acea8b"},"filename":"psutil-7.1.2-cp36-abi3-macosx_10_9_x86_64.whl","hashes":{"sha256":"c9ba5c19f2d46203ee8c152c7b01df6eec87d883cfd8ee1af2ef2727f6b0f814"},"provenance":null,"requires-python":">=3.6","size":237244,"upload-time":"2025-10-25T10:47:07.086631Z","url":"https://files.pythonhosted.org/packages/ae/89/b9f8d47ddbc52d7301fc868e8224e5f44ed3c7f55e6d0f54ecaf5dd9ff5e/psutil-7.1.2-cp36-abi3-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"9060d8b9832fb047375c7eeac0d3ca41668a9f56cea96f12e30ce72cc4acea8b"},"data-dist-info-metadata":{"sha256":"9060d8b9832fb047375c7eeac0d3ca41668a9f56cea96f12e30ce72cc4acea8b"},"filename":"psutil-7.1.2-cp36-abi3-macosx_11_0_arm64.whl","hashes":{"sha256":"2a486030d2fe81bec023f703d3d155f4823a10a47c36784c84f1cc7f8d39bedb"},"provenance":null,"requires-python":">=3.6","size":238101,"upload-time":"2025-10-25T10:47:09.523680Z","url":"https://files.pythonhosted.org/packages/c8/7a/8628c2f6b240680a67d73d8742bb9ff39b1820a693740e43096d5dcb01e5/psutil-7.1.2-cp36-abi3-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"9060d8b9832fb047375c7eeac0d3ca41668a9f56cea96f12e30ce72cc4acea8b"},"data-dist-info-metadata":{"sha256":"9060d8b9832fb047375c7eeac0d3ca41668a9f56cea96f12e30ce72cc4acea8b"},"filename":"psutil-7.1.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","hashes":{"sha256":"3efd8fc791492e7808a51cb2b94889db7578bfaea22df931424f874468e389e3"},"provenance":null,"requires-python":">=3.6","size":258675,"upload-time":"2025-10-25T10:47:11.082823Z","url":"https://files.pythonhosted.org/packages/30/28/5e27f4d5a0e347f8e3cc16cd7d35533dbce086c95807f1f0e9cd77e26c10/psutil-7.1.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"9060d8b9832fb047375c7eeac0d3ca41668a9f56cea96f12e30ce72cc4acea8b"},"data-dist-info-metadata":{"sha256":"9060d8b9832fb047375c7eeac0d3ca41668a9f56cea96f12e30ce72cc4acea8b"},"filename":"psutil-7.1.2-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","hashes":{"sha256":"e2aeb9b64f481b8eabfc633bd39e0016d4d8bbcd590d984af764d80bf0851b8a"},"provenance":null,"requires-python":">=3.6","size":260203,"upload-time":"2025-10-25T10:47:13.226564Z","url":"https://files.pythonhosted.org/packages/e5/5c/79cf60c9acf36d087f0db0f82066fca4a780e97e5b3a2e4c38209c03d170/psutil-7.1.2-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"4e0958cd1551f981a3d0063faf8314b741cd97cb43d1e58934800cb4da537f64"},"data-dist-info-metadata":{"sha256":"4e0958cd1551f981a3d0063faf8314b741cd97cb43d1e58934800cb4da537f64"},"filename":"psutil-7.1.2-cp37-abi3-win_amd64.whl","hashes":{"sha256":"8e17852114c4e7996fe9da4745c2bdef001ebbf2f260dec406290e66628bdb91"},"provenance":null,"requires-python":">=3.6","size":246714,"upload-time":"2025-10-25T10:47:15.093571Z","url":"https://files.pythonhosted.org/packages/f7/03/0a464404c51685dcb9329fdd660b1721e076ccd7b3d97dee066bcc9ffb15/psutil-7.1.2-cp37-abi3-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"b27a63f31491a49efc9ff48050f5fdcca4606a487fac91e6cc46853031038b01"},"data-dist-info-metadata":{"sha256":"b27a63f31491a49efc9ff48050f5fdcca4606a487fac91e6cc46853031038b01"},"filename":"psutil-7.1.2-cp37-abi3-win_arm64.whl","hashes":{"sha256":"3e988455e61c240cc879cb62a008c2699231bf3e3d061d7fce4234463fd2abb4"},"provenance":null,"requires-python":">=3.6","size":243742,"upload-time":"2025-10-25T10:47:17.302139Z","url":"https://files.pythonhosted.org/packages/6a/32/97ca2090f2f1b45b01b6aa7ae161cfe50671de097311975ca6eea3e7aabc/psutil-7.1.2-cp37-abi3-win_arm64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-7.1.2.tar.gz","hashes":{"sha256":"aa225cdde1335ff9684708ee8c72650f6598d5ed2114b9a7c5802030b1785018"},"provenance":null,"requires-python":">=3.6","size":487424,"upload-time":"2025-10-25T10:46:34.931002Z","url":"https://files.pythonhosted.org/packages/cd/ec/7b8e6b9b1d22708138630ef34c53ab2b61032c04f16adfdbb96791c8c70c/psutil-7.1.2.tar.gz","yanked":false},{"core-metadata":{"sha256":"fee2408ed83d5d231935e659a072f72e41c592bc880496074a506e9d6b3eada7"},"data-dist-info-metadata":{"sha256":"fee2408ed83d5d231935e659a072f72e41c592bc880496074a506e9d6b3eada7"},"filename":"psutil-7.1.3-cp313-cp313t-macosx_10_13_x86_64.whl","hashes":{"sha256":"0005da714eee687b4b8decd3d6cc7c6db36215c9e74e5ad2264b90c3df7d92dc"},"provenance":null,"requires-python":">=3.6","size":239751,"upload-time":"2025-11-02T12:25:58.161404Z","url":"https://files.pythonhosted.org/packages/bd/93/0c49e776b8734fef56ec9c5c57f923922f2cf0497d62e0f419465f28f3d0/psutil-7.1.3-cp313-cp313t-macosx_10_13_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"fee2408ed83d5d231935e659a072f72e41c592bc880496074a506e9d6b3eada7"},"data-dist-info-metadata":{"sha256":"fee2408ed83d5d231935e659a072f72e41c592bc880496074a506e9d6b3eada7"},"filename":"psutil-7.1.3-cp313-cp313t-macosx_11_0_arm64.whl","hashes":{"sha256":"19644c85dcb987e35eeeaefdc3915d059dac7bd1167cdcdbf27e0ce2df0c08c0"},"provenance":null,"requires-python":">=3.6","size":240368,"upload-time":"2025-11-02T12:26:00.491685Z","url":"https://files.pythonhosted.org/packages/6f/8d/b31e39c769e70780f007969815195a55c81a63efebdd4dbe9e7a113adb2f/psutil-7.1.3-cp313-cp313t-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"fee2408ed83d5d231935e659a072f72e41c592bc880496074a506e9d6b3eada7"},"data-dist-info-metadata":{"sha256":"fee2408ed83d5d231935e659a072f72e41c592bc880496074a506e9d6b3eada7"},"filename":"psutil-7.1.3-cp313-cp313t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","hashes":{"sha256":"95ef04cf2e5ba0ab9eaafc4a11eaae91b44f4ef5541acd2ee91d9108d00d59a7"},"provenance":null,"requires-python":">=3.6","size":287134,"upload-time":"2025-11-02T12:26:02.613574Z","url":"https://files.pythonhosted.org/packages/62/61/23fd4acc3c9eebbf6b6c78bcd89e5d020cfde4acf0a9233e9d4e3fa698b4/psutil-7.1.3-cp313-cp313t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"fee2408ed83d5d231935e659a072f72e41c592bc880496074a506e9d6b3eada7"},"data-dist-info-metadata":{"sha256":"fee2408ed83d5d231935e659a072f72e41c592bc880496074a506e9d6b3eada7"},"filename":"psutil-7.1.3-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","hashes":{"sha256":"1068c303be3a72f8e18e412c5b2a8f6d31750fb152f9cb106b54090296c9d251"},"provenance":null,"requires-python":">=3.6","size":289904,"upload-time":"2025-11-02T12:26:05.207933Z","url":"https://files.pythonhosted.org/packages/30/1c/f921a009ea9ceb51aa355cb0cc118f68d354db36eae18174bab63affb3e6/psutil-7.1.3-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"a690b24adf2968fc63d76d10c2e8052107d0af31da1588424d365c1ff08c9fd1"},"data-dist-info-metadata":{"sha256":"a690b24adf2968fc63d76d10c2e8052107d0af31da1588424d365c1ff08c9fd1"},"filename":"psutil-7.1.3-cp313-cp313t-win_amd64.whl","hashes":{"sha256":"18349c5c24b06ac5612c0428ec2a0331c26443d259e2a0144a9b24b4395b58fa"},"provenance":null,"requires-python":">=3.6","size":249642,"upload-time":"2025-11-02T12:26:07.447774Z","url":"https://files.pythonhosted.org/packages/a6/82/62d68066e13e46a5116df187d319d1724b3f437ddd0f958756fc052677f4/psutil-7.1.3-cp313-cp313t-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"a690b24adf2968fc63d76d10c2e8052107d0af31da1588424d365c1ff08c9fd1"},"data-dist-info-metadata":{"sha256":"a690b24adf2968fc63d76d10c2e8052107d0af31da1588424d365c1ff08c9fd1"},"filename":"psutil-7.1.3-cp313-cp313t-win_arm64.whl","hashes":{"sha256":"c525ffa774fe4496282fb0b1187725793de3e7c6b29e41562733cae9ada151ee"},"provenance":null,"requires-python":">=3.6","size":245518,"upload-time":"2025-11-02T12:26:09.719155Z","url":"https://files.pythonhosted.org/packages/df/ad/c1cd5fe965c14a0392112f68362cfceb5230819dbb5b1888950d18a11d9f/psutil-7.1.3-cp313-cp313t-win_arm64.whl","yanked":false},{"core-metadata":{"sha256":"fee2408ed83d5d231935e659a072f72e41c592bc880496074a506e9d6b3eada7"},"data-dist-info-metadata":{"sha256":"fee2408ed83d5d231935e659a072f72e41c592bc880496074a506e9d6b3eada7"},"filename":"psutil-7.1.3-cp314-cp314t-macosx_10_15_x86_64.whl","hashes":{"sha256":"b403da1df4d6d43973dc004d19cee3b848e998ae3154cc8097d139b77156c353"},"provenance":null,"requires-python":">=3.6","size":239843,"upload-time":"2025-11-02T12:26:11.968073Z","url":"https://files.pythonhosted.org/packages/2e/bb/6670bded3e3236eb4287c7bcdc167e9fae6e1e9286e437f7111caed2f909/psutil-7.1.3-cp314-cp314t-macosx_10_15_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"fee2408ed83d5d231935e659a072f72e41c592bc880496074a506e9d6b3eada7"},"data-dist-info-metadata":{"sha256":"fee2408ed83d5d231935e659a072f72e41c592bc880496074a506e9d6b3eada7"},"filename":"psutil-7.1.3-cp314-cp314t-macosx_11_0_arm64.whl","hashes":{"sha256":"ad81425efc5e75da3f39b3e636293360ad8d0b49bed7df824c79764fb4ba9b8b"},"provenance":null,"requires-python":">=3.6","size":240369,"upload-time":"2025-11-02T12:26:14.358801Z","url":"https://files.pythonhosted.org/packages/b8/66/853d50e75a38c9a7370ddbeefabdd3d3116b9c31ef94dc92c6729bc36bec/psutil-7.1.3-cp314-cp314t-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"fee2408ed83d5d231935e659a072f72e41c592bc880496074a506e9d6b3eada7"},"data-dist-info-metadata":{"sha256":"fee2408ed83d5d231935e659a072f72e41c592bc880496074a506e9d6b3eada7"},"filename":"psutil-7.1.3-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","hashes":{"sha256":"8f33a3702e167783a9213db10ad29650ebf383946e91bc77f28a5eb083496bc9"},"provenance":null,"requires-python":">=3.6","size":288210,"upload-time":"2025-11-02T12:26:16.699739Z","url":"https://files.pythonhosted.org/packages/41/bd/313aba97cb5bfb26916dc29cf0646cbe4dd6a89ca69e8c6edce654876d39/psutil-7.1.3-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"fee2408ed83d5d231935e659a072f72e41c592bc880496074a506e9d6b3eada7"},"data-dist-info-metadata":{"sha256":"fee2408ed83d5d231935e659a072f72e41c592bc880496074a506e9d6b3eada7"},"filename":"psutil-7.1.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","hashes":{"sha256":"fac9cd332c67f4422504297889da5ab7e05fd11e3c4392140f7370f4208ded1f"},"provenance":null,"requires-python":">=3.6","size":291182,"upload-time":"2025-11-02T12:26:18.848963Z","url":"https://files.pythonhosted.org/packages/c2/fa/76e3c06e760927a0cfb5705eb38164254de34e9bd86db656d4dbaa228b04/psutil-7.1.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"a690b24adf2968fc63d76d10c2e8052107d0af31da1588424d365c1ff08c9fd1"},"data-dist-info-metadata":{"sha256":"a690b24adf2968fc63d76d10c2e8052107d0af31da1588424d365c1ff08c9fd1"},"filename":"psutil-7.1.3-cp314-cp314t-win_amd64.whl","hashes":{"sha256":"3792983e23b69843aea49c8f5b8f115572c5ab64c153bada5270086a2123c7e7"},"provenance":null,"requires-python":">=3.6","size":250466,"upload-time":"2025-11-02T12:26:21.183069Z","url":"https://files.pythonhosted.org/packages/0f/1d/5774a91607035ee5078b8fd747686ebec28a962f178712de100d00b78a32/psutil-7.1.3-cp314-cp314t-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"a690b24adf2968fc63d76d10c2e8052107d0af31da1588424d365c1ff08c9fd1"},"data-dist-info-metadata":{"sha256":"a690b24adf2968fc63d76d10c2e8052107d0af31da1588424d365c1ff08c9fd1"},"filename":"psutil-7.1.3-cp314-cp314t-win_arm64.whl","hashes":{"sha256":"31d77fcedb7529f27bb3a0472bea9334349f9a04160e8e6e5020f22c59893264"},"provenance":null,"requires-python":">=3.6","size":245756,"upload-time":"2025-11-02T12:26:23.148427Z","url":"https://files.pythonhosted.org/packages/00/ca/e426584bacb43a5cb1ac91fae1937f478cd8fbe5e4ff96574e698a2c77cd/psutil-7.1.3-cp314-cp314t-win_arm64.whl","yanked":false},{"core-metadata":{"sha256":"3578dc4ee5f43e542d29839741b9e3d8760c3c0e5d8c388ec2b0b980bd8c66a1"},"data-dist-info-metadata":{"sha256":"3578dc4ee5f43e542d29839741b9e3d8760c3c0e5d8c388ec2b0b980bd8c66a1"},"filename":"psutil-7.1.3-cp36-abi3-macosx_10_9_x86_64.whl","hashes":{"sha256":"2bdbcd0e58ca14996a42adf3621a6244f1bb2e2e528886959c72cf1e326677ab"},"provenance":null,"requires-python":">=3.6","size":238359,"upload-time":"2025-11-02T12:26:25.284599Z","url":"https://files.pythonhosted.org/packages/ef/94/46b9154a800253e7ecff5aaacdf8ebf43db99de4a2dfa18575b02548654e/psutil-7.1.3-cp36-abi3-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"3578dc4ee5f43e542d29839741b9e3d8760c3c0e5d8c388ec2b0b980bd8c66a1"},"data-dist-info-metadata":{"sha256":"3578dc4ee5f43e542d29839741b9e3d8760c3c0e5d8c388ec2b0b980bd8c66a1"},"filename":"psutil-7.1.3-cp36-abi3-macosx_11_0_arm64.whl","hashes":{"sha256":"bc31fa00f1fbc3c3802141eede66f3a2d51d89716a194bf2cd6fc68310a19880"},"provenance":null,"requires-python":">=3.6","size":239171,"upload-time":"2025-11-02T12:26:27.230751Z","url":"https://files.pythonhosted.org/packages/68/3a/9f93cff5c025029a36d9a92fef47220ab4692ee7f2be0fba9f92813d0cb8/psutil-7.1.3-cp36-abi3-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"3578dc4ee5f43e542d29839741b9e3d8760c3c0e5d8c388ec2b0b980bd8c66a1"},"data-dist-info-metadata":{"sha256":"3578dc4ee5f43e542d29839741b9e3d8760c3c0e5d8c388ec2b0b980bd8c66a1"},"filename":"psutil-7.1.3-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","hashes":{"sha256":"3bb428f9f05c1225a558f53e30ccbad9930b11c3fc206836242de1091d3e7dd3"},"provenance":null,"requires-python":">=3.6","size":263261,"upload-time":"2025-11-02T12:26:29.480632Z","url":"https://files.pythonhosted.org/packages/ce/b1/5f49af514f76431ba4eea935b8ad3725cdeb397e9245ab919dbc1d1dc20f/psutil-7.1.3-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"3578dc4ee5f43e542d29839741b9e3d8760c3c0e5d8c388ec2b0b980bd8c66a1"},"data-dist-info-metadata":{"sha256":"3578dc4ee5f43e542d29839741b9e3d8760c3c0e5d8c388ec2b0b980bd8c66a1"},"filename":"psutil-7.1.3-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","hashes":{"sha256":"56d974e02ca2c8eb4812c3f76c30e28836fffc311d55d979f1465c1feeb2b68b"},"provenance":null,"requires-python":">=3.6","size":264635,"upload-time":"2025-11-02T12:26:31.740762Z","url":"https://files.pythonhosted.org/packages/e0/95/992c8816a74016eb095e73585d747e0a8ea21a061ed3689474fabb29a395/psutil-7.1.3-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"89db7acafbe66cd8b6a060ae993afda9c8f50aa4342c34c92a284be2c8c8534f"},"data-dist-info-metadata":{"sha256":"89db7acafbe66cd8b6a060ae993afda9c8f50aa4342c34c92a284be2c8c8534f"},"filename":"psutil-7.1.3-cp37-abi3-win_amd64.whl","hashes":{"sha256":"f39c2c19fe824b47484b96f9692932248a54c43799a84282cfe58d05a6449efd"},"provenance":null,"requires-python":">=3.6","size":247633,"upload-time":"2025-11-02T12:26:33.887174Z","url":"https://files.pythonhosted.org/packages/55/4c/c3ed1a622b6ae2fd3c945a366e64eb35247a31e4db16cf5095e269e8eb3c/psutil-7.1.3-cp37-abi3-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"a690b24adf2968fc63d76d10c2e8052107d0af31da1588424d365c1ff08c9fd1"},"data-dist-info-metadata":{"sha256":"a690b24adf2968fc63d76d10c2e8052107d0af31da1588424d365c1ff08c9fd1"},"filename":"psutil-7.1.3-cp37-abi3-win_arm64.whl","hashes":{"sha256":"bd0d69cee829226a761e92f28140bec9a5ee9d5b4fb4b0cc589068dbfff559b1"},"provenance":null,"requires-python":">=3.6","size":244608,"upload-time":"2025-11-02T12:26:36.136434Z","url":"https://files.pythonhosted.org/packages/c9/ad/33b2ccec09bf96c2b2ef3f9a6f66baac8253d7565d8839e024a6b905d45d/psutil-7.1.3-cp37-abi3-win_arm64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-7.1.3.tar.gz","hashes":{"sha256":"6c86281738d77335af7aec228328e944b30930899ea760ecf33a4dba66be5e74"},"provenance":null,"requires-python":">=3.6","size":489059,"upload-time":"2025-11-02T12:25:54.619294Z","url":"https://files.pythonhosted.org/packages/e1/88/bdd0a41e5857d5d703287598cbf08dad90aed56774ea52ae071bae9071b6/psutil-7.1.3.tar.gz","yanked":false},{"core-metadata":{"sha256":"2170dd4480c057291a293d121c10bf5228540dd58ab43945d29672ec02a0a23a"},"data-dist-info-metadata":{"sha256":"2170dd4480c057291a293d121c10bf5228540dd58ab43945d29672ec02a0a23a"},"filename":"psutil-7.2.0-cp313-cp313t-macosx_10_13_x86_64.whl","hashes":{"sha256":"c31e927555539132a00380c971816ea43d089bf4bd5f3e918ed8c16776d68474"},"provenance":null,"requires-python":">=3.6","size":129593,"upload-time":"2025-12-23T20:26:28.019569Z","url":"https://files.pythonhosted.org/packages/a8/8e/b35aae6ed19bc4e2286cac4832e4d522fcf00571867b0a85a3f77ef96a80/psutil-7.2.0-cp313-cp313t-macosx_10_13_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"2170dd4480c057291a293d121c10bf5228540dd58ab43945d29672ec02a0a23a"},"data-dist-info-metadata":{"sha256":"2170dd4480c057291a293d121c10bf5228540dd58ab43945d29672ec02a0a23a"},"filename":"psutil-7.2.0-cp313-cp313t-macosx_11_0_arm64.whl","hashes":{"sha256":"db8e44e766cef86dea47d9a1fa535d38dc76449e5878a92f33683b7dba5bfcb2"},"provenance":null,"requires-python":">=3.6","size":130104,"upload-time":"2025-12-23T20:26:30.270800Z","url":"https://files.pythonhosted.org/packages/61/a2/773d17d74e122bbffe08b97f73f2d4a01ef53fb03b98e61b8e4f64a9c6b9/psutil-7.2.0-cp313-cp313t-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"2170dd4480c057291a293d121c10bf5228540dd58ab43945d29672ec02a0a23a"},"data-dist-info-metadata":{"sha256":"2170dd4480c057291a293d121c10bf5228540dd58ab43945d29672ec02a0a23a"},"filename":"psutil-7.2.0-cp313-cp313t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","hashes":{"sha256":"85ef849ac92169dedc59a7ac2fb565f47b3468fbe1524bf748746bc21afb94c7"},"provenance":null,"requires-python":">=3.6","size":180579,"upload-time":"2025-12-23T20:26:32.628357Z","url":"https://files.pythonhosted.org/packages/0d/e3/d3a9b3f4bd231abbd70a988beb2e3edd15306051bccbfc4472bd34a56e01/psutil-7.2.0-cp313-cp313t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"2170dd4480c057291a293d121c10bf5228540dd58ab43945d29672ec02a0a23a"},"data-dist-info-metadata":{"sha256":"2170dd4480c057291a293d121c10bf5228540dd58ab43945d29672ec02a0a23a"},"filename":"psutil-7.2.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","hashes":{"sha256":"26782bdbae2f5c14ce9ebe8ad2411dc2ca870495e0cd90f8910ede7fa5e27117"},"provenance":null,"requires-python":">=3.6","size":183171,"upload-time":"2025-12-23T20:26:34.972726Z","url":"https://files.pythonhosted.org/packages/66/f8/6c73044424aabe1b7824d4d4504029d406648286d8fe7ba8c4682e0d3042/psutil-7.2.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"795be0019027992887b63578f61759602599118117e81710fb28c5b7fb5023d6"},"data-dist-info-metadata":{"sha256":"795be0019027992887b63578f61759602599118117e81710fb28c5b7fb5023d6"},"filename":"psutil-7.2.0-cp313-cp313t-win_amd64.whl","hashes":{"sha256":"b7665f612d3b38a583391b95969667a53aaf6c5706dc27a602c9a4874fbf09e4"},"provenance":null,"requires-python":">=3.6","size":139055,"upload-time":"2025-12-23T20:26:36.848832Z","url":"https://files.pythonhosted.org/packages/48/7d/76d7a863340885d41826562225a566683e653ee6c9ba03c9f3856afa7d80/psutil-7.2.0-cp313-cp313t-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"795be0019027992887b63578f61759602599118117e81710fb28c5b7fb5023d6"},"data-dist-info-metadata":{"sha256":"795be0019027992887b63578f61759602599118117e81710fb28c5b7fb5023d6"},"filename":"psutil-7.2.0-cp313-cp313t-win_arm64.whl","hashes":{"sha256":"4413373c174520ae28a24a8974ad8ce6b21f060d27dde94e25f8c73a7effe57a"},"provenance":null,"requires-python":">=3.6","size":134737,"upload-time":"2025-12-23T20:26:38.784066Z","url":"https://files.pythonhosted.org/packages/a0/48/200054ada0ae4872c8a71db54f3eb6a9af4101680ee6830d373b7fda526b/psutil-7.2.0-cp313-cp313t-win_arm64.whl","yanked":false},{"core-metadata":{"sha256":"2170dd4480c057291a293d121c10bf5228540dd58ab43945d29672ec02a0a23a"},"data-dist-info-metadata":{"sha256":"2170dd4480c057291a293d121c10bf5228540dd58ab43945d29672ec02a0a23a"},"filename":"psutil-7.2.0-cp314-cp314t-macosx_10_15_x86_64.whl","hashes":{"sha256":"2f2f53fd114e7946dfba3afb98c9b7c7f376009447360ca15bfb73f2066f84c7"},"provenance":null,"requires-python":">=3.6","size":129692,"upload-time":"2025-12-23T20:26:40.623176Z","url":"https://files.pythonhosted.org/packages/44/86/98da45dff471b93ef5ce5bcaefa00e3038295a7880a77cf74018243d37fb/psutil-7.2.0-cp314-cp314t-macosx_10_15_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"2170dd4480c057291a293d121c10bf5228540dd58ab43945d29672ec02a0a23a"},"data-dist-info-metadata":{"sha256":"2170dd4480c057291a293d121c10bf5228540dd58ab43945d29672ec02a0a23a"},"filename":"psutil-7.2.0-cp314-cp314t-macosx_11_0_arm64.whl","hashes":{"sha256":"e65c41d7e60068f60ce43b31a3a7fc90deb0dfd34ffc824a2574c2e5279b377e"},"provenance":null,"requires-python":">=3.6","size":130110,"upload-time":"2025-12-23T20:26:42.569228Z","url":"https://files.pythonhosted.org/packages/50/ee/10eae91ba4ad071c92db3c178ba861f30406342de9f0ddbe6d51fd741236/psutil-7.2.0-cp314-cp314t-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"2170dd4480c057291a293d121c10bf5228540dd58ab43945d29672ec02a0a23a"},"data-dist-info-metadata":{"sha256":"2170dd4480c057291a293d121c10bf5228540dd58ab43945d29672ec02a0a23a"},"filename":"psutil-7.2.0-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","hashes":{"sha256":"cc66d21366850a4261412ce994ae9976bba9852dafb4f2fa60db68ed17ff5281"},"provenance":null,"requires-python":">=3.6","size":181487,"upload-time":"2025-12-23T20:26:44.633020Z","url":"https://files.pythonhosted.org/packages/87/3a/2b2897443d56fedbbc34ac68a0dc7d55faa05d555372a2f989109052f86d/psutil-7.2.0-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"2170dd4480c057291a293d121c10bf5228540dd58ab43945d29672ec02a0a23a"},"data-dist-info-metadata":{"sha256":"2170dd4480c057291a293d121c10bf5228540dd58ab43945d29672ec02a0a23a"},"filename":"psutil-7.2.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","hashes":{"sha256":"e025d67b42b8f22b096d5d20f5171de0e0fefb2f0ce983a13c5a1b5ed9872706"},"provenance":null,"requires-python":">=3.6","size":184320,"upload-time":"2025-12-23T20:26:46.830615Z","url":"https://files.pythonhosted.org/packages/11/66/44308428f7333db42c5ea7390c52af1b38f59b80b80c437291f58b5dfdad/psutil-7.2.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"795be0019027992887b63578f61759602599118117e81710fb28c5b7fb5023d6"},"data-dist-info-metadata":{"sha256":"795be0019027992887b63578f61759602599118117e81710fb28c5b7fb5023d6"},"filename":"psutil-7.2.0-cp314-cp314t-win_amd64.whl","hashes":{"sha256":"45f6b91f7ad63414d6454fd609e5e3556d0e1038d5d9c75a1368513bdf763f57"},"provenance":null,"requires-python":">=3.6","size":140372,"upload-time":"2025-12-23T20:26:49.334377Z","url":"https://files.pythonhosted.org/packages/18/28/d2feadc7f18e501c5ce687c377db7dca924585418fd694272b8e488ea99f/psutil-7.2.0-cp314-cp314t-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"795be0019027992887b63578f61759602599118117e81710fb28c5b7fb5023d6"},"data-dist-info-metadata":{"sha256":"795be0019027992887b63578f61759602599118117e81710fb28c5b7fb5023d6"},"filename":"psutil-7.2.0-cp314-cp314t-win_arm64.whl","hashes":{"sha256":"87b18a19574139d60a546e88b5f5b9cbad598e26cdc790d204ab95d7024f03ee"},"provenance":null,"requires-python":">=3.6","size":135400,"upload-time":"2025-12-23T20:26:51.585604Z","url":"https://files.pythonhosted.org/packages/b2/1d/48381f5fd0425aa054c4ee3de24f50de3d6c347019f3aec75f357377d447/psutil-7.2.0-cp314-cp314t-win_arm64.whl","yanked":false},{"core-metadata":{"sha256":"78cbedc39a964b930e33a3c9bd2f10aaa72c827cfcce30c4074dffd544c24a58"},"data-dist-info-metadata":{"sha256":"78cbedc39a964b930e33a3c9bd2f10aaa72c827cfcce30c4074dffd544c24a58"},"filename":"psutil-7.2.0-cp36-abi3-macosx_10_9_x86_64.whl","hashes":{"sha256":"977a2fcd132d15cb05b32b2d85b98d087cad039b0ce435731670ba74da9e6133"},"provenance":null,"requires-python":">=3.6","size":128116,"upload-time":"2025-12-23T20:26:53.516520Z","url":"https://files.pythonhosted.org/packages/40/c5/a49160bf3e165b7b93a60579a353cf5d939d7f878fe5fd369110f1d18043/psutil-7.2.0-cp36-abi3-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"78cbedc39a964b930e33a3c9bd2f10aaa72c827cfcce30c4074dffd544c24a58"},"data-dist-info-metadata":{"sha256":"78cbedc39a964b930e33a3c9bd2f10aaa72c827cfcce30c4074dffd544c24a58"},"filename":"psutil-7.2.0-cp36-abi3-macosx_11_0_arm64.whl","hashes":{"sha256":"24151011c21fadd94214d7139d7c6c54569290d7e553989bdf0eab73b13beb8c"},"provenance":null,"requires-python":">=3.6","size":128925,"upload-time":"2025-12-23T20:26:55.573324Z","url":"https://files.pythonhosted.org/packages/10/a1/c75feb480f60cd768fb6ed00ac362a16a33e5076ec8475a22d8162fb2659/psutil-7.2.0-cp36-abi3-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"78cbedc39a964b930e33a3c9bd2f10aaa72c827cfcce30c4074dffd544c24a58"},"data-dist-info-metadata":{"sha256":"78cbedc39a964b930e33a3c9bd2f10aaa72c827cfcce30c4074dffd544c24a58"},"filename":"psutil-7.2.0-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","hashes":{"sha256":"91f211ba9279e7c61d9d8f84b713cfc38fa161cb0597d5cb3f1ca742f6848254"},"provenance":null,"requires-python":">=3.6","size":154666,"upload-time":"2025-12-23T20:26:57.312776Z","url":"https://files.pythonhosted.org/packages/12/ff/e93136587c00a543f4bc768b157fac2c47cd77b180d4f4e5c6efb6ea53a2/psutil-7.2.0-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"78cbedc39a964b930e33a3c9bd2f10aaa72c827cfcce30c4074dffd544c24a58"},"data-dist-info-metadata":{"sha256":"78cbedc39a964b930e33a3c9bd2f10aaa72c827cfcce30c4074dffd544c24a58"},"filename":"psutil-7.2.0-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","hashes":{"sha256":"f37415188b7ea98faf90fed51131181646c59098b077550246e2e092e127418b"},"provenance":null,"requires-python":">=3.6","size":156109,"upload-time":"2025-12-23T20:26:58.851353Z","url":"https://files.pythonhosted.org/packages/b8/dd/4c2de9c3827c892599d277a69d2224136800870a8a88a80981de905de28d/psutil-7.2.0-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"78cbedc39a964b930e33a3c9bd2f10aaa72c827cfcce30c4074dffd544c24a58"},"data-dist-info-metadata":{"sha256":"78cbedc39a964b930e33a3c9bd2f10aaa72c827cfcce30c4074dffd544c24a58"},"filename":"psutil-7.2.0-cp36-abi3-musllinux_1_2_aarch64.whl","hashes":{"sha256":"0d12c7ce6ed1128cd81fd54606afa054ac7dbb9773469ebb58cf2f171c49f2ac"},"provenance":null,"requires-python":">=3.6","size":148081,"upload-time":"2025-12-23T20:27:01.318270Z","url":"https://files.pythonhosted.org/packages/81/3f/090943c682d3629968dd0b04826ddcbc760ee1379021dbe316e2ddfcd01b/psutil-7.2.0-cp36-abi3-musllinux_1_2_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"78cbedc39a964b930e33a3c9bd2f10aaa72c827cfcce30c4074dffd544c24a58"},"data-dist-info-metadata":{"sha256":"78cbedc39a964b930e33a3c9bd2f10aaa72c827cfcce30c4074dffd544c24a58"},"filename":"psutil-7.2.0-cp36-abi3-musllinux_1_2_x86_64.whl","hashes":{"sha256":"ca0faef7976530940dcd39bc5382d0d0d5eb023b186a4901ca341bd8d8684151"},"provenance":null,"requires-python":">=3.6","size":147376,"upload-time":"2025-12-23T20:27:03.347816Z","url":"https://files.pythonhosted.org/packages/c4/88/c39648ebb8ec182d0364af53cdefe6eddb5f3872ba718b5855a8ff65d6d4/psutil-7.2.0-cp36-abi3-musllinux_1_2_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"758019d458979edff64acde45968217765d5828ba125593a164a985cdc0754bd"},"data-dist-info-metadata":{"sha256":"758019d458979edff64acde45968217765d5828ba125593a164a985cdc0754bd"},"filename":"psutil-7.2.0-cp37-abi3-win_amd64.whl","hashes":{"sha256":"abdb74137ca232d20250e9ad471f58d500e7743bc8253ba0bfbf26e570c0e437"},"provenance":null,"requires-python":">=3.6","size":136910,"upload-time":"2025-12-23T20:27:05.289344Z","url":"https://files.pythonhosted.org/packages/01/a2/5b39e08bd9b27476bc7cce7e21c71a481ad60b81ffac49baf02687a50d7f/psutil-7.2.0-cp37-abi3-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"795be0019027992887b63578f61759602599118117e81710fb28c5b7fb5023d6"},"data-dist-info-metadata":{"sha256":"795be0019027992887b63578f61759602599118117e81710fb28c5b7fb5023d6"},"filename":"psutil-7.2.0-cp37-abi3-win_arm64.whl","hashes":{"sha256":"284e71038b3139e7ab3834b63b3eb5aa5565fcd61a681ec746ef9a0a8c457fd2"},"provenance":null,"requires-python":">=3.6","size":133807,"upload-time":"2025-12-23T20:27:06.825130Z","url":"https://files.pythonhosted.org/packages/59/54/53839db1258c1eaeb4ded57ff202144ebc75b23facc05a74fd98d338b0c6/psutil-7.2.0-cp37-abi3-win_arm64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-7.2.0.tar.gz","hashes":{"sha256":"2e4f8e1552f77d14dc96fb0f6240c5b34a37081c0889f0853b3b29a496e5ef64"},"provenance":null,"requires-python":">=3.6","size":489863,"upload-time":"2025-12-23T20:26:24.616214Z","url":"https://files.pythonhosted.org/packages/be/7c/31d1c3ceb1260301f87565f50689dc6da3db427ece1e1e012af22abca54e/psutil-7.2.0.tar.gz","yanked":false},{"core-metadata":{"sha256":"23740d4f8fcd640f5d191b7b232f6a393e1df6bf023302f619c701f995dff02d"},"data-dist-info-metadata":{"sha256":"23740d4f8fcd640f5d191b7b232f6a393e1df6bf023302f619c701f995dff02d"},"filename":"psutil-7.2.1-cp313-cp313t-macosx_10_13_x86_64.whl","hashes":{"sha256":"ba9f33bb525b14c3ea563b2fd521a84d2fa214ec59e3e6a2858f78d0844dd60d"},"provenance":null,"requires-python":">=3.6","size":129624,"upload-time":"2025-12-29T08:26:04.255517Z","url":"https://files.pythonhosted.org/packages/77/8e/f0c242053a368c2aa89584ecd1b054a18683f13d6e5a318fc9ec36582c94/psutil-7.2.1-cp313-cp313t-macosx_10_13_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"23740d4f8fcd640f5d191b7b232f6a393e1df6bf023302f619c701f995dff02d"},"data-dist-info-metadata":{"sha256":"23740d4f8fcd640f5d191b7b232f6a393e1df6bf023302f619c701f995dff02d"},"filename":"psutil-7.2.1-cp313-cp313t-macosx_11_0_arm64.whl","hashes":{"sha256":"81442dac7abfc2f4f4385ea9e12ddf5a796721c0f6133260687fec5c3780fa49"},"provenance":null,"requires-python":">=3.6","size":130132,"upload-time":"2025-12-29T08:26:06.228150Z","url":"https://files.pythonhosted.org/packages/26/97/a58a4968f8990617decee234258a2b4fc7cd9e35668387646c1963e69f26/psutil-7.2.1-cp313-cp313t-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"23740d4f8fcd640f5d191b7b232f6a393e1df6bf023302f619c701f995dff02d"},"data-dist-info-metadata":{"sha256":"23740d4f8fcd640f5d191b7b232f6a393e1df6bf023302f619c701f995dff02d"},"filename":"psutil-7.2.1-cp313-cp313t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","hashes":{"sha256":"ea46c0d060491051d39f0d2cff4f98d5c72b288289f57a21556cc7d504db37fc"},"provenance":null,"requires-python":">=3.6","size":180612,"upload-time":"2025-12-29T08:26:08.276538Z","url":"https://files.pythonhosted.org/packages/db/6d/ed44901e830739af5f72a85fa7ec5ff1edea7f81bfbf4875e409007149bd/psutil-7.2.1-cp313-cp313t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"23740d4f8fcd640f5d191b7b232f6a393e1df6bf023302f619c701f995dff02d"},"data-dist-info-metadata":{"sha256":"23740d4f8fcd640f5d191b7b232f6a393e1df6bf023302f619c701f995dff02d"},"filename":"psutil-7.2.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","hashes":{"sha256":"35630d5af80d5d0d49cfc4d64c1c13838baf6717a13effb35869a5919b854cdf"},"provenance":null,"requires-python":">=3.6","size":183201,"upload-time":"2025-12-29T08:26:10.622093Z","url":"https://files.pythonhosted.org/packages/c7/65/b628f8459bca4efbfae50d4bf3feaab803de9a160b9d5f3bd9295a33f0c2/psutil-7.2.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"08f1da19e657e59de5e9a94e4295fa686697c5c4e76447bd048138c2227e8293"},"data-dist-info-metadata":{"sha256":"08f1da19e657e59de5e9a94e4295fa686697c5c4e76447bd048138c2227e8293"},"filename":"psutil-7.2.1-cp313-cp313t-win_amd64.whl","hashes":{"sha256":"923f8653416604e356073e6e0bccbe7c09990acef442def2f5640dd0faa9689f"},"provenance":null,"requires-python":">=3.6","size":139081,"upload-time":"2025-12-29T08:26:12.483257Z","url":"https://files.pythonhosted.org/packages/fb/23/851cadc9764edcc18f0effe7d0bf69f727d4cf2442deb4a9f78d4e4f30f2/psutil-7.2.1-cp313-cp313t-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"08f1da19e657e59de5e9a94e4295fa686697c5c4e76447bd048138c2227e8293"},"data-dist-info-metadata":{"sha256":"08f1da19e657e59de5e9a94e4295fa686697c5c4e76447bd048138c2227e8293"},"filename":"psutil-7.2.1-cp313-cp313t-win_arm64.whl","hashes":{"sha256":"cfbe6b40ca48019a51827f20d830887b3107a74a79b01ceb8cc8de4ccb17b672"},"provenance":null,"requires-python":">=3.6","size":134767,"upload-time":"2025-12-29T08:26:14.528248Z","url":"https://files.pythonhosted.org/packages/59/82/d63e8494ec5758029f31c6cb06d7d161175d8281e91d011a4a441c8a43b5/psutil-7.2.1-cp313-cp313t-win_arm64.whl","yanked":false},{"core-metadata":{"sha256":"23740d4f8fcd640f5d191b7b232f6a393e1df6bf023302f619c701f995dff02d"},"data-dist-info-metadata":{"sha256":"23740d4f8fcd640f5d191b7b232f6a393e1df6bf023302f619c701f995dff02d"},"filename":"psutil-7.2.1-cp314-cp314t-macosx_10_15_x86_64.whl","hashes":{"sha256":"494c513ccc53225ae23eec7fe6e1482f1b8a44674241b54561f755a898650679"},"provenance":null,"requires-python":">=3.6","size":129716,"upload-time":"2025-12-29T08:26:16.017500Z","url":"https://files.pythonhosted.org/packages/05/c2/5fb764bd61e40e1fe756a44bd4c21827228394c17414ade348e28f83cd79/psutil-7.2.1-cp314-cp314t-macosx_10_15_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"23740d4f8fcd640f5d191b7b232f6a393e1df6bf023302f619c701f995dff02d"},"data-dist-info-metadata":{"sha256":"23740d4f8fcd640f5d191b7b232f6a393e1df6bf023302f619c701f995dff02d"},"filename":"psutil-7.2.1-cp314-cp314t-macosx_11_0_arm64.whl","hashes":{"sha256":"3fce5f92c22b00cdefd1645aa58ab4877a01679e901555067b1bd77039aa589f"},"provenance":null,"requires-python":">=3.6","size":130133,"upload-time":"2025-12-29T08:26:18.009017Z","url":"https://files.pythonhosted.org/packages/c9/d2/935039c20e06f615d9ca6ca0ab756cf8408a19d298ffaa08666bc18dc805/psutil-7.2.1-cp314-cp314t-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"23740d4f8fcd640f5d191b7b232f6a393e1df6bf023302f619c701f995dff02d"},"data-dist-info-metadata":{"sha256":"23740d4f8fcd640f5d191b7b232f6a393e1df6bf023302f619c701f995dff02d"},"filename":"psutil-7.2.1-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","hashes":{"sha256":"93f3f7b0bb07711b49626e7940d6fe52aa9940ad86e8f7e74842e73189712129"},"provenance":null,"requires-python":">=3.6","size":181518,"upload-time":"2025-12-29T08:26:20.241501Z","url":"https://files.pythonhosted.org/packages/77/69/19f1eb0e01d24c2b3eacbc2f78d3b5add8a89bf0bb69465bc8d563cc33de/psutil-7.2.1-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"23740d4f8fcd640f5d191b7b232f6a393e1df6bf023302f619c701f995dff02d"},"data-dist-info-metadata":{"sha256":"23740d4f8fcd640f5d191b7b232f6a393e1df6bf023302f619c701f995dff02d"},"filename":"psutil-7.2.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","hashes":{"sha256":"d34d2ca888208eea2b5c68186841336a7f5e0b990edec929be909353a202768a"},"provenance":null,"requires-python":">=3.6","size":184348,"upload-time":"2025-12-29T08:26:22.215886Z","url":"https://files.pythonhosted.org/packages/e1/6d/7e18b1b4fa13ad370787626c95887b027656ad4829c156bb6569d02f3262/psutil-7.2.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"08f1da19e657e59de5e9a94e4295fa686697c5c4e76447bd048138c2227e8293"},"data-dist-info-metadata":{"sha256":"08f1da19e657e59de5e9a94e4295fa686697c5c4e76447bd048138c2227e8293"},"filename":"psutil-7.2.1-cp314-cp314t-win_amd64.whl","hashes":{"sha256":"2ceae842a78d1603753561132d5ad1b2f8a7979cb0c283f5b52fb4e6e14b1a79"},"provenance":null,"requires-python":">=3.6","size":140400,"upload-time":"2025-12-29T08:26:23.993885Z","url":"https://files.pythonhosted.org/packages/98/60/1672114392dd879586d60dd97896325df47d9a130ac7401318005aab28ec/psutil-7.2.1-cp314-cp314t-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"08f1da19e657e59de5e9a94e4295fa686697c5c4e76447bd048138c2227e8293"},"data-dist-info-metadata":{"sha256":"08f1da19e657e59de5e9a94e4295fa686697c5c4e76447bd048138c2227e8293"},"filename":"psutil-7.2.1-cp314-cp314t-win_arm64.whl","hashes":{"sha256":"08a2f175e48a898c8eb8eace45ce01777f4785bc744c90aa2cc7f2fa5462a266"},"provenance":null,"requires-python":">=3.6","size":135430,"upload-time":"2025-12-29T08:26:25.999852Z","url":"https://files.pythonhosted.org/packages/fb/7b/d0e9d4513c46e46897b46bcfc410d51fc65735837ea57a25170f298326e6/psutil-7.2.1-cp314-cp314t-win_arm64.whl","yanked":false},{"core-metadata":{"sha256":"c340b0dbe2145d45dac650326d0f34f3ee664206d3bcd32785393b3476525944"},"data-dist-info-metadata":{"sha256":"c340b0dbe2145d45dac650326d0f34f3ee664206d3bcd32785393b3476525944"},"filename":"psutil-7.2.1-cp36-abi3-macosx_10_9_x86_64.whl","hashes":{"sha256":"b2e953fcfaedcfbc952b44744f22d16575d3aa78eb4f51ae74165b4e96e55f42"},"provenance":null,"requires-python":">=3.6","size":128137,"upload-time":"2025-12-29T08:26:27.759659Z","url":"https://files.pythonhosted.org/packages/c5/cf/5180eb8c8bdf6a503c6919f1da28328bd1e6b3b1b5b9d5b01ae64f019616/psutil-7.2.1-cp36-abi3-macosx_10_9_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"c340b0dbe2145d45dac650326d0f34f3ee664206d3bcd32785393b3476525944"},"data-dist-info-metadata":{"sha256":"c340b0dbe2145d45dac650326d0f34f3ee664206d3bcd32785393b3476525944"},"filename":"psutil-7.2.1-cp36-abi3-macosx_11_0_arm64.whl","hashes":{"sha256":"05cc68dbb8c174828624062e73078e7e35406f4ca2d0866c272c2410d8ef06d1"},"provenance":null,"requires-python":">=3.6","size":128947,"upload-time":"2025-12-29T08:26:29.548034Z","url":"https://files.pythonhosted.org/packages/c5/2c/78e4a789306a92ade5000da4f5de3255202c534acdadc3aac7b5458fadef/psutil-7.2.1-cp36-abi3-macosx_11_0_arm64.whl","yanked":false},{"core-metadata":{"sha256":"c340b0dbe2145d45dac650326d0f34f3ee664206d3bcd32785393b3476525944"},"data-dist-info-metadata":{"sha256":"c340b0dbe2145d45dac650326d0f34f3ee664206d3bcd32785393b3476525944"},"filename":"psutil-7.2.1-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","hashes":{"sha256":"5e38404ca2bb30ed7267a46c02f06ff842e92da3bb8c5bfdadbd35a5722314d8"},"provenance":null,"requires-python":">=3.6","size":154694,"upload-time":"2025-12-29T08:26:32.147774Z","url":"https://files.pythonhosted.org/packages/29/f8/40e01c350ad9a2b3cb4e6adbcc8a83b17ee50dd5792102b6142385937db5/psutil-7.2.1-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"c340b0dbe2145d45dac650326d0f34f3ee664206d3bcd32785393b3476525944"},"data-dist-info-metadata":{"sha256":"c340b0dbe2145d45dac650326d0f34f3ee664206d3bcd32785393b3476525944"},"filename":"psutil-7.2.1-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","hashes":{"sha256":"ab2b98c9fc19f13f59628d94df5cc4cc4844bc572467d113a8b517d634e362c6"},"provenance":null,"requires-python":">=3.6","size":156136,"upload-time":"2025-12-29T08:26:34.079911Z","url":"https://files.pythonhosted.org/packages/06/e4/b751cdf839c011a9714a783f120e6a86b7494eb70044d7d81a25a5cd295f/psutil-7.2.1-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"c340b0dbe2145d45dac650326d0f34f3ee664206d3bcd32785393b3476525944"},"data-dist-info-metadata":{"sha256":"c340b0dbe2145d45dac650326d0f34f3ee664206d3bcd32785393b3476525944"},"filename":"psutil-7.2.1-cp36-abi3-musllinux_1_2_aarch64.whl","hashes":{"sha256":"f78baafb38436d5a128f837fab2d92c276dfb48af01a240b861ae02b2413ada8"},"provenance":null,"requires-python":">=3.6","size":148108,"upload-time":"2025-12-29T08:26:36.225796Z","url":"https://files.pythonhosted.org/packages/44/ad/bbf6595a8134ee1e94a4487af3f132cef7fce43aef4a93b49912a48c3af7/psutil-7.2.1-cp36-abi3-musllinux_1_2_aarch64.whl","yanked":false},{"core-metadata":{"sha256":"c340b0dbe2145d45dac650326d0f34f3ee664206d3bcd32785393b3476525944"},"data-dist-info-metadata":{"sha256":"c340b0dbe2145d45dac650326d0f34f3ee664206d3bcd32785393b3476525944"},"filename":"psutil-7.2.1-cp36-abi3-musllinux_1_2_x86_64.whl","hashes":{"sha256":"99a4cd17a5fdd1f3d014396502daa70b5ec21bf4ffe38393e152f8e449757d67"},"provenance":null,"requires-python":">=3.6","size":147402,"upload-time":"2025-12-29T08:26:39.210698Z","url":"https://files.pythonhosted.org/packages/1c/15/dd6fd869753ce82ff64dcbc18356093471a5a5adf4f77ed1f805d473d859/psutil-7.2.1-cp36-abi3-musllinux_1_2_x86_64.whl","yanked":false},{"core-metadata":{"sha256":"54955c981ab8a8b2b87efc900b6916c908a315660b0c3c5d205c910565bb3261"},"data-dist-info-metadata":{"sha256":"54955c981ab8a8b2b87efc900b6916c908a315660b0c3c5d205c910565bb3261"},"filename":"psutil-7.2.1-cp37-abi3-win_amd64.whl","hashes":{"sha256":"b1b0671619343aa71c20ff9767eced0483e4fc9e1f489d50923738caf6a03c17"},"provenance":null,"requires-python":">=3.6","size":136938,"upload-time":"2025-12-29T08:26:41.036253Z","url":"https://files.pythonhosted.org/packages/34/68/d9317542e3f2b180c4306e3f45d3c922d7e86d8ce39f941bb9e2e9d8599e/psutil-7.2.1-cp37-abi3-win_amd64.whl","yanked":false},{"core-metadata":{"sha256":"08f1da19e657e59de5e9a94e4295fa686697c5c4e76447bd048138c2227e8293"},"data-dist-info-metadata":{"sha256":"08f1da19e657e59de5e9a94e4295fa686697c5c4e76447bd048138c2227e8293"},"filename":"psutil-7.2.1-cp37-abi3-win_arm64.whl","hashes":{"sha256":"0d67c1822c355aa6f7314d92018fb4268a76668a536f133599b91edd48759442"},"provenance":null,"requires-python":">=3.6","size":133836,"upload-time":"2025-12-29T08:26:43.086974Z","url":"https://files.pythonhosted.org/packages/3e/73/2ce007f4198c80fcf2cb24c169884f833fe93fbc03d55d302627b094ee91/psutil-7.2.1-cp37-abi3-win_arm64.whl","yanked":false},{"core-metadata":false,"data-dist-info-metadata":false,"filename":"psutil-7.2.1.tar.gz","hashes":{"sha256":"f7583aec590485b43ca601dd9cea0dcd65bd7bb21d30ef4ddbf4ea6b5ed1bdd3"},"provenance":null,"requires-python":">=3.6","size":490253,"upload-time":"2025-12-29T08:26:00.169622Z","url":"https://files.pythonhosted.org/packages/73/cb/09e5184fb5fc0358d110fc3ca7f6b1d033800734d34cac10f4136cfac10e/psutil-7.2.1.tar.gz","yanked":false}],"meta":{"_last-serial":33241970,"api-version":"1.4"},"name":"psutil","project-status":{"status":"active"},"versions":["0.1.1","0.1.2","0.1.3","0.2.0","0.2.1","0.3.0","0.4.0","0.4.1","0.5.0","0.5.1","0.6.0","0.6.1","0.7.0","0.7.1","1.0.0","1.0.1","1.1.0","1.1.1","1.1.2","1.1.3","1.2.0","1.2.1","2.0.0","2.1.0","2.1.1","2.1.2","2.1.3","2.2.0","2.2.1","3.0.0","3.0.1","3.1.0","3.1.1","3.2.0","3.2.1","3.2.2","3.3.0","3.4.1","3.4.2","4.0.0","4.1.0","4.2.0","4.3.0","4.3.1","4.4.0","4.4.1","4.4.2","5.0.0","5.0.1","5.1.0","5.1.1","5.1.2","5.1.3","5.2.0","5.2.1","5.2.2","5.3.0","5.3.1","5.4.0","5.4.1","5.4.2","5.4.3","5.4.4","5.4.5","5.4.6","5.4.7","5.4.8","5.5.0","5.5.1","5.6.0","5.6.1","5.6.2","5.6.3","5.6.4","5.6.5","5.6.6","5.6.7","5.7.0","5.7.1","5.7.2","5.7.3","5.8.0","5.9.0","5.9.1","5.9.2","5.9.3","5.9.4","5.9.5","5.9.6","5.9.7","5.9.8","6.0.0","6.1.0","6.1.1","7.0.0","7.1.0","7.1.1","7.1.2","7.1.3","7.2.0","7.2.1"]} diff --git a/benches/benchmarks/json_loads.py b/benches/benchmarks/json_loads.py new file mode 100644 index 00000000000..f67f14d2d9e --- /dev/null +++ b/benches/benchmarks/json_loads.py @@ -0,0 +1,7 @@ +import json + +with open('benches/_data/pypi_org__simple__psutil.json') as f: + data = f.read() + + +loaded = json.loads(data) diff --git a/benches/execution.rs b/benches/execution.rs index c2239b59d12..2816f0e8201 100644 --- a/benches/execution.rs +++ b/benches/execution.rs @@ -24,7 +24,9 @@ fn bench_rustpython_code(b: &mut Bencher, name: &str, source: &str) { settings.path_list.push("Lib/".to_string()); settings.write_bytecode = false; settings.user_site_directory = false; - Interpreter::without_stdlib(settings).enter(|vm| { + let builder = Interpreter::builder(settings); + let defs = rustpython_stdlib::stdlib_module_defs(&builder.ctx); + builder.add_native_modules(&defs).build().enter(|vm| { // Note: bench_cpython is both compiling and executing the code. // As such we compile the code in the benchmark loop as well. b.iter(|| { diff --git a/benches/microbenchmarks.rs b/benches/microbenchmarks.rs index 98993b41543..ba5dcd6c2ec 100644 --- a/benches/microbenchmarks.rs +++ b/benches/microbenchmarks.rs @@ -113,12 +113,10 @@ fn bench_rustpython_code(group: &mut BenchmarkGroup<WallTime>, bench: &MicroBenc settings.write_bytecode = false; settings.user_site_directory = false; - Interpreter::with_init(settings, |vm| { - for (name, init) in rustpython_stdlib::get_module_inits() { - vm.add_native_module(name, init); - } - }) - .enter(|vm| { + let builder = Interpreter::builder(settings); + let defs = rustpython_stdlib::stdlib_module_defs(&builder.ctx); + let interp = builder.add_native_modules(&defs).build(); + interp.enter(|vm| { let setup_code = vm .compile(&bench.setup, Mode::Exec, bench.name.to_owned()) .expect("Error compiling setup code"); diff --git a/crates/codegen/src/compile.rs b/crates/codegen/src/compile.rs index 66eb785e962..9b59d9da8c7 100644 --- a/crates/codegen/src/compile.rs +++ b/crates/codegen/src/compile.rs @@ -21,29 +21,65 @@ use itertools::Itertools; use malachite_bigint::BigInt; use num_complex::Complex; use num_traits::{Num, ToPrimitive}; -use ruff_python_ast::{ - Alias, Arguments, BoolOp, CmpOp, Comprehension, ConversionFlag, DebugText, Decorator, DictItem, - ExceptHandler, ExceptHandlerExceptHandler, Expr, ExprAttribute, ExprBoolOp, ExprContext, - ExprFString, ExprList, ExprName, ExprSlice, ExprStarred, ExprSubscript, ExprTuple, ExprUnaryOp, - FString, FStringFlags, FStringPart, Identifier, Int, InterpolatedStringElement, - InterpolatedStringElements, Keyword, MatchCase, ModExpression, ModModule, Operator, Parameters, - Pattern, PatternMatchAs, PatternMatchClass, PatternMatchMapping, PatternMatchOr, - PatternMatchSequence, PatternMatchSingleton, PatternMatchStar, PatternMatchValue, Singleton, - Stmt, StmtExpr, TypeParam, TypeParamParamSpec, TypeParamTypeVar, TypeParamTypeVarTuple, - TypeParams, UnaryOp, WithItem, - visitor::{Visitor, walk_expr}, -}; +use ruff_python_ast as ast; use ruff_text_size::{Ranged, TextRange}; +use std::collections::HashSet; + use rustpython_compiler_core::{ Mode, OneIndexed, PositionEncoding, SourceFile, SourceLocation, bytecode::{ - self, Arg as OpArgMarker, BinaryOperator, BuildSliceArgCount, CodeObject, - ComparisonOperator, ConstantData, ConvertValueOparg, Instruction, Invert, OpArg, OpArgType, - UnpackExArgs, + self, AnyInstruction, Arg as OpArgMarker, BinaryOperator, BuildSliceArgCount, CodeObject, + ComparisonOperator, ConstantData, ConvertValueOparg, Instruction, IntrinsicFunction1, + Invert, OpArg, OpArgType, PseudoInstruction, SpecialMethod, UnpackExArgs, + encode_load_attr_arg, encode_load_super_attr_arg, }, }; use rustpython_wtf8::Wtf8Buf; -use std::collections::HashSet; + +/// Extension trait for `ast::Expr` to add constant checking methods +trait ExprExt { + /// Check if an expression is a constant literal + fn is_constant(&self) -> bool; + + /// Check if a slice expression has all constant elements + fn is_constant_slice(&self) -> bool; + + /// Check if we should use BINARY_SLICE/STORE_SLICE optimization + fn should_use_slice_optimization(&self) -> bool; +} + +impl ExprExt for ast::Expr { + fn is_constant(&self) -> bool { + matches!( + self, + ast::Expr::NumberLiteral(_) + | ast::Expr::StringLiteral(_) + | ast::Expr::BytesLiteral(_) + | ast::Expr::NoneLiteral(_) + | ast::Expr::BooleanLiteral(_) + | ast::Expr::EllipsisLiteral(_) + ) + } + + fn is_constant_slice(&self) -> bool { + match self { + ast::Expr::Slice(s) => { + let lower_const = + s.lower.is_none() || s.lower.as_deref().is_some_and(|e| e.is_constant()); + let upper_const = + s.upper.is_none() || s.upper.as_deref().is_some_and(|e| e.is_constant()); + let step_const = + s.step.is_none() || s.step.as_deref().is_some_and(|e| e.is_constant()); + lower_const && upper_const && step_const + } + _ => false, + } + } + + fn should_use_slice_optimization(&self) -> bool { + !self.is_constant_slice() && matches!(self, ast::Expr::Slice(s) if s.step.is_none()) + } +} const MAXBLOCKS: usize = 20; @@ -70,20 +106,28 @@ pub enum FBlockType { pub enum FBlockDatum { None, /// For FinallyTry: stores the finally body statements to compile during unwind - FinallyBody(Vec<Stmt>), + FinallyBody(Vec<ast::Stmt>), /// For HandlerCleanup: stores the exception variable name (e.g., "e" in "except X as e") ExceptionName(String), } +/// Type of super() call optimization detected by can_optimize_super_call() +#[derive(Debug, Clone)] +enum SuperCallType<'a> { + /// super(class, self) - explicit 2-argument form + TwoArg { + class_arg: &'a ast::Expr, + self_arg: &'a ast::Expr, + }, + /// super() - implicit 0-argument form (uses __class__ cell) + ZeroArg, +} + #[derive(Debug, Clone)] pub struct FBlockInfo { pub fb_type: FBlockType, pub fb_block: BlockIdx, pub fb_exit: BlockIdx, - // For Python 3.11+ exception table generation - pub fb_handler: Option<BlockIdx>, // Exception handler block - pub fb_stack_depth: u32, // Stack depth at block entry - pub fb_preserve_lasti: bool, // Whether to preserve lasti (for SETUP_CLEANUP) // additional data for fblock unwinding pub fb_datum: FBlockDatum, } @@ -97,20 +141,6 @@ enum NameUsage { Store, Delete, } - -enum CallType { - Positional { nargs: u32 }, - Keyword { nargs: u32 }, - Ex { has_kwargs: bool }, -} - -fn is_forbidden_name(name: &str) -> bool { - // See https://docs.python.org/3/library/constants.html#built-in-constants - const BUILTIN_CONSTANTS: &[&str] = &["__debug__"]; - - BUILTIN_CONSTANTS.contains(&name) -} - /// Main structure holding the state of compilation. struct Compiler { code_stack: Vec<ir::CodeInfo>, @@ -125,13 +155,14 @@ struct Compiler { in_annotation: bool, } +#[derive(Clone, Copy)] enum DoneWithFuture { No, DoneWithDoc, Yes, } -#[derive(Debug, Clone)] +#[derive(Clone, Copy, Debug)] pub struct CompileOpts { /// How optimized the bytecode output should be; any optimize > 0 does /// not emit assert statements @@ -179,7 +210,7 @@ enum ComprehensionType { Dict, } -fn validate_duplicate_params(params: &Parameters) -> Result<(), CodegenErrorType> { +fn validate_duplicate_params(params: &ast::Parameters) -> Result<(), CodegenErrorType> { let mut seen_params = HashSet::new(); for param in params { let param_name = param.name().as_str(); @@ -212,7 +243,7 @@ pub fn compile_top( /// Compile a standard Python program to bytecode pub fn compile_program( - ast: &ModModule, + ast: &ast::ModModule, source_file: SourceFile, opts: CompileOpts, ) -> CompileResult<CodeObject> { @@ -227,7 +258,7 @@ pub fn compile_program( /// Compile a Python program to bytecode for the context of a REPL pub fn compile_program_single( - ast: &ModModule, + ast: &ast::ModModule, source_file: SourceFile, opts: CompileOpts, ) -> CompileResult<CodeObject> { @@ -241,7 +272,7 @@ pub fn compile_program_single( } pub fn compile_block_expression( - ast: &ModModule, + ast: &ast::ModModule, source_file: SourceFile, opts: CompileOpts, ) -> CompileResult<CodeObject> { @@ -255,7 +286,7 @@ pub fn compile_block_expression( } pub fn compile_expression( - ast: &ModExpression, + ast: &ast::ModExpression, source_file: SourceFile, opts: CompileOpts, ) -> CompileResult<CodeObject> { @@ -268,17 +299,24 @@ pub fn compile_expression( } macro_rules! emit { - ($c:expr, Instruction::$op:ident { $arg:ident$(,)? }$(,)?) => { - $c.emit_arg($arg, |x| Instruction::$op { $arg: x }) + // Struct variant with single identifier (e.g., Foo::A { arg }) + ($c:expr, $enum:ident :: $op:ident { $arg:ident $(,)? } $(,)?) => { + $c.emit_arg($arg, |x| $enum::$op { $arg: x }) }; - ($c:expr, Instruction::$op:ident { $arg:ident : $arg_val:expr $(,)? }$(,)?) => { - $c.emit_arg($arg_val, |x| Instruction::$op { $arg: x }) + + // Struct variant with explicit value (e.g., Foo::A { arg: 42 }) + ($c:expr, $enum:ident :: $op:ident { $arg:ident : $arg_val:expr $(,)? } $(,)?) => { + $c.emit_arg($arg_val, |x| $enum::$op { $arg: x }) }; - ($c:expr, Instruction::$op:ident( $arg_val:expr $(,)? )$(,)?) => { - $c.emit_arg($arg_val, Instruction::$op) + + // Tuple variant (e.g., Foo::B(42)) + ($c:expr, $enum:ident :: $op:ident($arg_val:expr $(,)? ) $(,)?) => { + $c.emit_arg($arg_val, $enum::$op) }; - ($c:expr, Instruction::$op:ident$(,)?) => { - $c.emit_no_arg(Instruction::$op) + + // No-arg variant (e.g., Foo::C) + ($c:expr, $enum:ident :: $op:ident $(,)?) => { + $c.emit_no_arg($enum::$op) }; } @@ -383,7 +421,7 @@ enum CollectionType { impl Compiler { fn new(opts: CompileOpts, source_file: SourceFile, code_name: String) -> Self { let module_code = ir::CodeInfo { - flags: bytecode::CodeFlags::NEW_LOCALS, + flags: bytecode::CodeFlags::NEWLOCALS, source_path: source_file.name().to_owned(), private: None, blocks: vec![ir::Block::default()], @@ -406,6 +444,8 @@ impl Compiler { in_inlined_comp: false, fblock: Vec::with_capacity(MAXBLOCKS), symbol_table_index: 0, // Module is always the first symbol table + in_conditional_block: 0, + next_conditional_annotation_index: 0, }; Self { code_stack: vec![module_code], @@ -426,46 +466,32 @@ impl Compiler { } } - /// Check if the slice is a two-element slice (no step) - // = is_two_element_slice - const fn is_two_element_slice(slice: &Expr) -> bool { - matches!(slice, Expr::Slice(s) if s.step.is_none()) - } - - /// Compile a slice expression - // = compiler_slice - fn compile_slice(&mut self, s: &ExprSlice) -> CompileResult<BuildSliceArgCount> { - // Compile lower + /// Compile just start and stop of a slice (for BINARY_SLICE/STORE_SLICE) + // = codegen_slice_two_parts + fn compile_slice_two_parts(&mut self, s: &ast::ExprSlice) -> CompileResult<()> { + // Compile lower (or None) if let Some(lower) = &s.lower { self.compile_expression(lower)?; } else { self.emit_load_const(ConstantData::None); } - // Compile upper + // Compile upper (or None) if let Some(upper) = &s.upper { self.compile_expression(upper)?; } else { self.emit_load_const(ConstantData::None); } - Ok(match &s.step { - Some(step) => { - // Compile step if present - self.compile_expression(step)?; - BuildSliceArgCount::Three - } - None => BuildSliceArgCount::Two, - }) + Ok(()) } - /// Compile a subscript expression // = compiler_subscript fn compile_subscript( &mut self, - value: &Expr, - slice: &Expr, - ctx: ExprContext, + value: &ast::Expr, + slice: &ast::Expr, + ctx: ast::ExprContext, ) -> CompileResult<()> { // 1. Check subscripter and index for Load context // 2. VISIT value @@ -473,7 +499,7 @@ impl Compiler { // 4. Otherwise VISIT slice and emit appropriate instruction // For Load context, some checks are skipped for now - // if ctx == ExprContext::Load { + // if ctx == ast::ExprContext::Load { // check_subscripter(value); // check_index(value, slice); // } @@ -481,20 +507,20 @@ impl Compiler { // VISIT(c, expr, e->v.Subscript.value) self.compile_expression(value)?; - // Handle two-element slice (for Load/Store, not Del) - if Self::is_two_element_slice(slice) && !matches!(ctx, ExprContext::Del) { - let argc = match slice { - Expr::Slice(s) => self.compile_slice(s)?, - _ => unreachable!("is_two_element_slice should only return true for Expr::Slice"), + // Handle two-element non-constant slice with BINARY_SLICE/STORE_SLICE + if slice.should_use_slice_optimization() && !matches!(ctx, ast::ExprContext::Del) { + match slice { + ast::Expr::Slice(s) => self.compile_slice_two_parts(s)?, + _ => unreachable!( + "should_use_slice_optimization should only return true for ast::Expr::Slice" + ), }; match ctx { - ExprContext::Load => { - emit!(self, Instruction::BuildSlice { argc }); - emit!(self, Instruction::Subscript); + ast::ExprContext::Load => { + emit!(self, Instruction::BinarySlice); } - ExprContext::Store => { - emit!(self, Instruction::BuildSlice { argc }); - emit!(self, Instruction::StoreSubscript); + ast::ExprContext::Store => { + emit!(self, Instruction::StoreSlice); } _ => unreachable!(), } @@ -504,10 +530,15 @@ impl Compiler { // Emit appropriate instruction based on context match ctx { - ExprContext::Load => emit!(self, Instruction::Subscript), - ExprContext::Store => emit!(self, Instruction::StoreSubscript), - ExprContext::Del => emit!(self, Instruction::DeleteSubscript), - ExprContext::Invalid => { + ast::ExprContext::Load => emit!( + self, + Instruction::BinaryOp { + op: BinaryOperator::Subscr + } + ), + ast::ExprContext::Store => emit!(self, Instruction::StoreSubscr), + ast::ExprContext::Del => emit!(self, Instruction::DeleteSubscr), + ast::ExprContext::Invalid => { return Err(self.error(CodegenErrorType::SyntaxError( "Invalid expression context".to_owned(), ))); @@ -520,7 +551,7 @@ impl Compiler { /// Helper function for compiling tuples/lists/sets with starred expressions /// - /// Parameters: + /// ast::Parameters: /// - elts: The elements to compile /// - pushed: Number of items already on the stack /// - collection_type: What type of collection to build (tuple, list, set) @@ -528,42 +559,119 @@ impl Compiler { // = starunpack_helper in compile.c fn starunpack_helper( &mut self, - elts: &[Expr], + elts: &[ast::Expr], pushed: u32, collection_type: CollectionType, ) -> CompileResult<()> { - // Use RustPython's existing approach with BuildXFromTuples - let (size, unpack) = self.gather_elements(pushed, elts)?; + let n = elts.len().to_u32(); + let seen_star = elts.iter().any(|e| matches!(e, ast::Expr::Starred(_))); + + // Determine collection size threshold for optimization + let big = match collection_type { + CollectionType::Set => n > 8, + _ => n > 4, + }; - if unpack { - // Has starred elements + // If no stars and not too big, compile all elements and build once + if !seen_star && !big { + for elt in elts { + self.compile_expression(elt)?; + } + let total_size = n + pushed; match collection_type { + CollectionType::List => { + emit!(self, Instruction::BuildList { size: total_size }); + } + CollectionType::Set => { + emit!(self, Instruction::BuildSet { size: total_size }); + } CollectionType::Tuple => { - if size > 1 || pushed > 0 { - emit!(self, Instruction::BuildTupleFromTuples { size }); + emit!(self, Instruction::BuildTuple { size: total_size }); + } + } + return Ok(()); + } + + // Has stars or too big: use streaming approach + let mut sequence_built = false; + let mut i = 0u32; + + for elt in elts.iter() { + if let ast::Expr::Starred(ast::ExprStarred { value, .. }) = elt { + // When we hit first star, build sequence with elements so far + if !sequence_built { + match collection_type { + CollectionType::List => { + emit!(self, Instruction::BuildList { size: i + pushed }); + } + CollectionType::Set => { + emit!(self, Instruction::BuildSet { size: i + pushed }); + } + CollectionType::Tuple => { + emit!(self, Instruction::BuildList { size: i + pushed }); + } } - // If size == 1 and pushed == 0, the single tuple is already on the stack + sequence_built = true; } - CollectionType::List => { - emit!(self, Instruction::BuildListFromTuples { size }); + + // Compile the starred expression and extend + self.compile_expression(value)?; + match collection_type { + CollectionType::List => { + emit!(self, Instruction::ListExtend { i: 0 }); + } + CollectionType::Set => { + emit!(self, Instruction::SetUpdate { i: 0 }); + } + CollectionType::Tuple => { + emit!(self, Instruction::ListExtend { i: 0 }); + } } - CollectionType::Set => { - emit!(self, Instruction::BuildSetFromTuples { size }); + } else { + // Non-starred element + self.compile_expression(elt)?; + + if sequence_built { + // Sequence already exists, append to it + match collection_type { + CollectionType::List => { + emit!(self, Instruction::ListAppend { i: 0 }); + } + CollectionType::Set => { + emit!(self, Instruction::SetAdd { i: 0 }); + } + CollectionType::Tuple => { + emit!(self, Instruction::ListAppend { i: 0 }); + } + } + } else { + // Still collecting elements before first star + i += 1; } } - } else { - // No starred elements + } + + // If we never built sequence (all non-starred), build it now + if !sequence_built { match collection_type { - CollectionType::Tuple => { - emit!(self, Instruction::BuildTuple { size }); - } CollectionType::List => { - emit!(self, Instruction::BuildList { size }); + emit!(self, Instruction::BuildList { size: i + pushed }); } CollectionType::Set => { - emit!(self, Instruction::BuildSet { size }); + emit!(self, Instruction::BuildSet { size: i + pushed }); + } + CollectionType::Tuple => { + emit!(self, Instruction::BuildTuple { size: i + pushed }); } } + } else if collection_type == CollectionType::Tuple { + // For tuples, convert the list to tuple + emit!( + self, + Instruction::CallIntrinsic1 { + func: IntrinsicFunction1::ListToTuple + } + ); } Ok(()) @@ -637,25 +745,85 @@ impl Compiler { } /// Push the next symbol table on to the stack - fn push_symbol_table(&mut self) -> &SymbolTable { + fn push_symbol_table(&mut self) -> CompileResult<&SymbolTable> { // Look up the next table contained in the scope of the current table let current_table = self .symbol_table_stack .last_mut() .expect("no current symbol table"); - if current_table.sub_tables.is_empty() { - panic!( - "push_symbol_table: no sub_tables available in {} (type: {:?})", - current_table.name, current_table.typ - ); + if current_table.next_sub_table >= current_table.sub_tables.len() { + let name = current_table.name.clone(); + let typ = current_table.typ; + return Err(self.error(CodegenErrorType::SyntaxError(format!( + "no symbol table available in {} (type: {:?})", + name, typ + )))); } - let table = current_table.sub_tables.remove(0); + let idx = current_table.next_sub_table; + current_table.next_sub_table += 1; + let table = current_table.sub_tables[idx].clone(); // Push the next table onto the stack self.symbol_table_stack.push(table); - self.current_symbol_table() + Ok(self.current_symbol_table()) + } + + /// Push the annotation symbol table from the next sub_table's annotation_block + /// The annotation_block is stored in the function's scope, which is the next sub_table + /// Returns true if annotation_block exists, false otherwise + fn push_annotation_symbol_table(&mut self) -> bool { + let current_table = self + .symbol_table_stack + .last_mut() + .expect("no current symbol table"); + + // The annotation_block is in the next sub_table (function scope) + let next_idx = current_table.next_sub_table; + if next_idx >= current_table.sub_tables.len() { + return false; + } + + let next_table = &mut current_table.sub_tables[next_idx]; + if let Some(annotation_block) = next_table.annotation_block.take() { + self.symbol_table_stack.push(*annotation_block); + true + } else { + false + } + } + + /// Push the annotation symbol table for module/class level annotations + /// This takes annotation_block from the current symbol table (not sub_tables) + fn push_current_annotation_symbol_table(&mut self) -> bool { + let current_table = self + .symbol_table_stack + .last_mut() + .expect("no current symbol table"); + + // For modules/classes, annotation_block is directly in the current table + if let Some(annotation_block) = current_table.annotation_block.take() { + self.symbol_table_stack.push(*annotation_block); + true + } else { + false + } + } + + /// Pop the annotation symbol table and restore it to the function scope's annotation_block + fn pop_annotation_symbol_table(&mut self) { + let annotation_table = self.symbol_table_stack.pop().expect("compiler bug"); + let current_table = self + .symbol_table_stack + .last_mut() + .expect("no current symbol table"); + + // Restore to the next sub_table (function scope) where it came from + let next_idx = current_table.next_sub_table; + if next_idx < current_table.sub_tables.len() { + current_table.sub_tables[next_idx].annotation_block = Some(Box::new(annotation_table)); + } } /// Pop the current symbol table off the stack @@ -663,6 +831,151 @@ impl Compiler { self.symbol_table_stack.pop().expect("compiler bug") } + /// Check if a super() call can be optimized + /// Returns Some(SuperCallType) if optimization is possible, None otherwise + fn can_optimize_super_call<'a>( + &self, + value: &'a ast::Expr, + attr: &str, + ) -> Option<SuperCallType<'a>> { + // 1. value must be a Call expression + let ast::Expr::Call(ast::ExprCall { + func, arguments, .. + }) = value + else { + return None; + }; + + // 2. func must be Name("super") + let ast::Expr::Name(ast::ExprName { id, .. }) = func.as_ref() else { + return None; + }; + if id.as_str() != "super" { + return None; + } + + // 3. attr must not be "__class__" + if attr == "__class__" { + return None; + } + + // 4. No keyword arguments + if !arguments.keywords.is_empty() { + return None; + } + + // 5. Must be inside a function (not at module level or class body) + if !self.ctx.in_func() { + return None; + } + + // 6. "super" must be GlobalImplicit (not redefined locally or at module level) + let table = self.current_symbol_table(); + if let Some(symbol) = table.lookup("super") + && symbol.scope != SymbolScope::GlobalImplicit + { + return None; + } + // Also check top-level scope to detect module-level shadowing. + // Only block if super is actually *bound* at module level (not just used). + if let Some(top_table) = self.symbol_table_stack.first() + && let Some(sym) = top_table.lookup("super") + && sym.scope != SymbolScope::GlobalImplicit + { + return None; + } + + // 7. Check argument pattern + let args = &arguments.args; + + // No starred expressions allowed + if args.iter().any(|arg| matches!(arg, ast::Expr::Starred(_))) { + return None; + } + + match args.len() { + 2 => { + // 2-arg: super(class, self) + Some(SuperCallType::TwoArg { + class_arg: &args[0], + self_arg: &args[1], + }) + } + 0 => { + // 0-arg: super() - need __class__ cell and first parameter + // Enclosing function should have at least one positional argument + let info = self.code_stack.last()?; + if info.metadata.argcount == 0 && info.metadata.posonlyargcount == 0 { + return None; + } + + // Check if __class__ is available as a cell/free variable + // The scope must be Free (from enclosing class) or have FREE_CLASS flag + if let Some(symbol) = table.lookup("__class__") { + if symbol.scope != SymbolScope::Free + && !symbol.flags.contains(SymbolFlags::FREE_CLASS) + { + return None; + } + } else { + // __class__ not in symbol table, optimization not possible + return None; + } + + Some(SuperCallType::ZeroArg) + } + _ => None, // 1 or 3+ args - not optimizable + } + } + + /// Load arguments for super() optimization onto the stack + /// Stack result: [global_super, class, self] + fn load_args_for_super(&mut self, super_type: &SuperCallType<'_>) -> CompileResult<()> { + // 1. Load global super + self.compile_name("super", NameUsage::Load)?; + + match super_type { + SuperCallType::TwoArg { + class_arg, + self_arg, + } => { + // 2-arg: load provided arguments + self.compile_expression(class_arg)?; + self.compile_expression(self_arg)?; + } + SuperCallType::ZeroArg => { + // 0-arg: load __class__ cell and first parameter + // Load __class__ from cell/free variable + let scope = self.get_ref_type("__class__").map_err(|e| self.error(e))?; + let idx = match scope { + SymbolScope::Cell => self.get_cell_var_index("__class__")?, + SymbolScope::Free => self.get_free_var_index("__class__")?, + _ => { + return Err(self.error(CodegenErrorType::SyntaxError( + "super(): __class__ cell not found".to_owned(), + ))); + } + }; + self.emit_arg(idx, Instruction::LoadDeref); + + // Load first parameter (typically 'self'). + // Safety: can_optimize_super_call() ensures argcount > 0, and + // parameters are always added to varnames first (see symboltable.rs). + let first_param = { + let info = self.code_stack.last().unwrap(); + info.metadata.varnames.first().cloned() + }; + let first_param = first_param.ok_or_else(|| { + self.error(CodegenErrorType::SyntaxError( + "super(): no arguments and no first parameter".to_owned(), + )) + })?; + self.compile_name(&first_param, NameUsage::Load)?; + } + } + Ok(()) + } + /// Check if this is an inlined comprehension context (PEP 709) /// Currently disabled - always returns false to avoid stack issues fn is_inlined_comprehension_context(&self, _comprehension_type: ComprehensionType) -> bool { @@ -680,12 +993,6 @@ impl Compiler { key: usize, // In RustPython, we use the index in symbol_table_stack as key lineno: u32, ) -> CompileResult<()> { - // Create location - let location = SourceLocation { - line: OneIndexed::new(lineno as usize).unwrap_or(OneIndexed::MIN), - character_offset: OneIndexed::MIN, - }; - // Allocate a new compiler unit // In Rust, we'll create the structure directly @@ -731,6 +1038,12 @@ impl Compiler { cellvar_cache.insert("__classdict__".to_string()); } + // Handle implicit __conditional_annotations__ cell if needed + // Only for class scope - module scope uses NAME operations, not DEREF + if ste.has_conditional_annotations && scope_type == CompilerScope::Class { + cellvar_cache.insert("__conditional_annotations__".to_string()); + } + // Build freevars using dictbytype (FREE scope, offset by cellvars size) let mut freevar_cache = IndexSet::default(); let mut free_names: Vec<_> = ste @@ -751,21 +1064,27 @@ impl Compiler { CompilerScope::Module => (bytecode::CodeFlags::empty(), 0, 0, 0), CompilerScope::Class => (bytecode::CodeFlags::empty(), 0, 0, 0), CompilerScope::Function | CompilerScope::AsyncFunction | CompilerScope::Lambda => ( - bytecode::CodeFlags::NEW_LOCALS | bytecode::CodeFlags::IS_OPTIMIZED, + bytecode::CodeFlags::NEWLOCALS | bytecode::CodeFlags::OPTIMIZED, 0, // Will be set later in enter_function 0, // Will be set later in enter_function 0, // Will be set later in enter_function ), CompilerScope::Comprehension => ( - bytecode::CodeFlags::NEW_LOCALS | bytecode::CodeFlags::IS_OPTIMIZED, + bytecode::CodeFlags::NEWLOCALS | bytecode::CodeFlags::OPTIMIZED, 0, 1, // comprehensions take one argument (.0) 0, ), CompilerScope::TypeParams => ( - bytecode::CodeFlags::NEW_LOCALS | bytecode::CodeFlags::IS_OPTIMIZED, + bytecode::CodeFlags::NEWLOCALS | bytecode::CodeFlags::OPTIMIZED, + 0, + 0, 0, + ), + CompilerScope::Annotation => ( + bytecode::CodeFlags::NEWLOCALS | bytecode::CodeFlags::OPTIMIZED, 0, + 1, // annotation scope takes one argument (format) 0, ), }; @@ -806,6 +1125,8 @@ impl Compiler { in_inlined_comp: false, fblock: Vec::with_capacity(MAXBLOCKS), symbol_table_index: key, + in_conditional_block: 0, + next_conditional_annotation_index: 0, }; // Push the old compiler unit on the stack (like PyCapsule) @@ -817,33 +1138,50 @@ impl Compiler { self.set_qualname(); } - // Emit RESUME instruction - let _resume_loc = if scope_type == CompilerScope::Module { - // Module scope starts with lineno 0 - SourceLocation { - line: OneIndexed::MIN, - character_offset: OneIndexed::MIN, - } - } else { - location - }; + // Emit RESUME (handles async preamble and module lineno 0) + // CPython: LOCATION(lineno, lineno, 0, 0), then loc.lineno = 0 for module + self.emit_resume_for_scope(scope_type, lineno); - // Set the source range for the RESUME instruction - // For now, just use an empty range at the beginning - self.current_source_range = TextRange::default(); - emit!( - self, - Instruction::Resume { - arg: bytecode::ResumeType::AtFuncStart as u32 - } - ); + Ok(()) + } - if scope_type == CompilerScope::Module { - // This would be loc.lineno = -1 in CPython - // We handle this differently in RustPython + /// Emit RESUME instruction with proper handling for async preamble and module lineno. + /// codegen_enter_scope equivalent for RESUME emission. + fn emit_resume_for_scope(&mut self, scope_type: CompilerScope, lineno: u32) { + // For async functions/coroutines, emit RETURN_GENERATOR + POP_TOP before RESUME + if scope_type == CompilerScope::AsyncFunction { + emit!(self, Instruction::ReturnGenerator); + emit!(self, Instruction::PopTop); } - Ok(()) + // CPython: LOCATION(lineno, lineno, 0, 0) + // Module scope: loc.lineno = 0 (before the first line) + let lineno_override = if scope_type == CompilerScope::Module { + Some(0) + } else { + None + }; + + // Use lineno for location (col = 0 as in CPython) + let location = SourceLocation { + line: OneIndexed::new(lineno as usize).unwrap_or(OneIndexed::MIN), + character_offset: OneIndexed::MIN, // col = 0 + }; + let end_location = location; // end_lineno = lineno, end_col = 0 + let except_handler = None; + + self.current_block().instructions.push(ir::InstructionInfo { + instr: Instruction::Resume { + arg: OpArgMarker::marker(), + } + .into(), + arg: OpArg(bytecode::ResumeType::AtFuncStart as u32), + target: BlockIdx::NULL, + location, + end_location, + except_handler, + lineno_override, + }); } fn push_output( @@ -853,9 +1191,9 @@ impl Compiler { arg_count: u32, kwonlyarg_count: u32, obj_name: String, - ) { + ) -> CompileResult<()> { // First push the symbol table - let table = self.push_symbol_table(); + let table = self.push_symbol_table()?; let scope_type = table.typ; // The key is the current position in the symbol table stack @@ -865,11 +1203,7 @@ impl Compiler { let lineno = self.get_source_line_number().get(); // Call enter_scope which does most of the work - if let Err(e) = self.enter_scope(&obj_name, scope_type, key, lineno.to_u32()) { - // In the current implementation, push_output doesn't return an error, - // so we panic here. This maintains the same behavior. - panic!("enter_scope failed: {e:?}"); - } + self.enter_scope(&obj_name, scope_type, key, lineno.to_u32())?; // Override the values that push_output sets explicitly // enter_scope sets default values based on scope_type, but push_output @@ -880,6 +1214,7 @@ impl Compiler { info.metadata.posonlyargcount = posonlyarg_count; info.metadata.kwonlyargcount = kwonlyarg_count; } + Ok(()) } // compiler_exit_scope @@ -887,7 +1222,7 @@ impl Compiler { let _table = self.pop_symbol_table(); // Various scopes can have sub_tables: - // - TypeParams scope can have sub_tables (the function body's symbol table) + // - ast::TypeParams scope can have sub_tables (the function body's symbol table) // - Module scope can have sub_tables (for TypeAlias scopes, nested functions, classes) // - Function scope can have sub_tables (for nested functions, classes) // - Class scope can have sub_tables (for nested classes, methods) @@ -898,6 +1233,85 @@ impl Compiler { unwrap_internal(self, stack_top.finalize_code(&self.opts)) } + /// Exit annotation scope - similar to exit_scope but restores annotation_block to parent + fn exit_annotation_scope(&mut self) -> CodeObject { + self.pop_annotation_symbol_table(); + + let pop = self.code_stack.pop(); + let stack_top = compiler_unwrap_option(self, pop); + unwrap_internal(self, stack_top.finalize_code(&self.opts)) + } + + /// Enter annotation scope using the symbol table's annotation_block + /// Returns false if no annotation_block exists + fn enter_annotation_scope(&mut self, func_name: &str) -> CompileResult<bool> { + if !self.push_annotation_symbol_table() { + return Ok(false); + } + + let key = self.symbol_table_stack.len() - 1; + let lineno = self.get_source_line_number().get(); + let annotate_name = format!("<annotate of {func_name}>"); + + self.enter_scope( + &annotate_name, + CompilerScope::Annotation, + key, + lineno.to_u32(), + )?; + + // Override arg_count since enter_scope sets it to 1 but we need the varnames + // setup to be correct too + self.current_code_info() + .metadata + .varnames + .insert("format".to_owned()); + + // Emit format validation: if format > VALUE_WITH_FAKE_GLOBALS: raise NotImplementedError + // VALUE_WITH_FAKE_GLOBALS = 2 (from annotationlib.Format) + self.emit_format_validation()?; + + Ok(true) + } + + /// Emit format parameter validation for annotation scope + /// if format > VALUE_WITH_FAKE_GLOBALS (2): raise NotImplementedError + fn emit_format_validation(&mut self) -> CompileResult<()> { + use bytecode::ComparisonOperator::Greater; + + // Load format parameter (first local variable, index 0) + emit!(self, Instruction::LoadFast(0)); + + // Load VALUE_WITH_FAKE_GLOBALS constant (2) + self.emit_load_const(ConstantData::Integer { value: 2.into() }); + + // Compare: format > 2 + emit!(self, Instruction::CompareOp { op: Greater }); + + // Jump to body if format <= 2 (comparison is false) + let body_block = self.new_block(); + emit!(self, Instruction::PopJumpIfFalse { target: body_block }); + + // Raise NotImplementedError + emit!( + self, + Instruction::LoadCommonConstant { + idx: bytecode::CommonConstant::NotImplementedError + } + ); + emit!( + self, + Instruction::RaiseVarargs { + kind: bytecode::RaiseKind::Raise + } + ); + + // Body label - continue with annotation evaluation + self.switch_to_block(body_block); + + Ok(()) + } + /// Push a new fblock // = compiler_push_fblock fn push_fblock( @@ -906,48 +1320,15 @@ impl Compiler { fb_block: BlockIdx, fb_exit: BlockIdx, ) -> CompileResult<()> { - self.push_fblock_full( - fb_type, - fb_block, - fb_exit, - None, - 0, - false, - FBlockDatum::None, - ) - } - - /// Push an fblock with exception handler info - fn push_fblock_with_handler( - &mut self, - fb_type: FBlockType, - fb_block: BlockIdx, - fb_exit: BlockIdx, - fb_handler: Option<BlockIdx>, - fb_stack_depth: u32, - fb_preserve_lasti: bool, - ) -> CompileResult<()> { - self.push_fblock_full( - fb_type, - fb_block, - fb_exit, - fb_handler, - fb_stack_depth, - fb_preserve_lasti, - FBlockDatum::None, - ) + self.push_fblock_full(fb_type, fb_block, fb_exit, FBlockDatum::None) } /// Push an fblock with all parameters including fb_datum - #[allow(clippy::too_many_arguments)] fn push_fblock_full( &mut self, fb_type: FBlockType, fb_block: BlockIdx, fb_exit: BlockIdx, - fb_handler: Option<BlockIdx>, - fb_stack_depth: u32, - fb_preserve_lasti: bool, fb_datum: FBlockDatum, ) -> CompileResult<()> { let code = self.current_code_info(); @@ -960,9 +1341,6 @@ impl Compiler { fb_type, fb_block, fb_exit, - fb_handler, - fb_stack_depth, - fb_preserve_lasti, fb_datum, }); Ok(()) @@ -994,11 +1372,11 @@ impl Compiler { if preserve_tos { emit!(self, Instruction::Swap { index: 2 }); } - emit!(self, Instruction::PopTop); + emit!(self, Instruction::PopIter); } FBlockType::TryExcept => { - // No POP_BLOCK with exception table, just pop fblock + emit!(self, PseudoInstruction::PopBlock); } FBlockType::FinallyTry => { @@ -1009,41 +1387,43 @@ impl Compiler { } FBlockType::FinallyEnd => { - // Stack when in FinallyEnd: [..., prev_exc, exc] or - // [..., prev_exc, exc, return_value] if preserve_tos - // Note: No lasti here - it's only pushed for cleanup handler exceptions - // We need to pop: exc, prev_exc (via PopException) + // codegen_unwind_fblock(FINALLY_END) if preserve_tos { emit!(self, Instruction::Swap { index: 2 }); } - emit!(self, Instruction::PopTop); // exc + emit!(self, Instruction::PopTop); // exc_value if preserve_tos { emit!(self, Instruction::Swap { index: 2 }); } - emit!(self, Instruction::PopException); // prev_exc is restored + emit!(self, PseudoInstruction::PopBlock); + emit!(self, Instruction::PopExcept); } FBlockType::With | FBlockType::AsyncWith => { // Stack when entering: [..., __exit__, return_value (if preserve_tos)] // Need to call __exit__(None, None, None) - emit!(self, Instruction::PopBlock); + emit!(self, PseudoInstruction::PopBlock); // If preserving return value, swap it below __exit__ if preserve_tos { emit!(self, Instruction::Swap { index: 2 }); } + // Stack after swap: [..., return_value, __exit__] or [..., __exit__] - // Call __exit__(None, None, None) - compiler_call_exit_with_nones - // Stack: [..., __exit__] or [..., return_value, __exit__] + // Call __exit__(None, None, None) + // Call protocol: [callable, self_or_null, arg1, arg2, arg3] + emit!(self, Instruction::PushNull); + // Stack: [..., __exit__, NULL] self.emit_load_const(ConstantData::None); self.emit_load_const(ConstantData::None); self.emit_load_const(ConstantData::None); - emit!(self, Instruction::CallFunctionPositional { nargs: 3 }); + // Stack: [..., __exit__, NULL, None, None, None] + emit!(self, Instruction::Call { nargs: 3 }); // For async with, await the result if matches!(info.fb_type, FBlockType::AsyncWith) { - emit!(self, Instruction::GetAwaitable); + emit!(self, Instruction::GetAwaitable { arg: 2 }); self.emit_load_const(ConstantData::None); self.compile_yield_from_sequence(true)?; } @@ -1053,10 +1433,17 @@ impl Compiler { } FBlockType::HandlerCleanup => { + // codegen_unwind_fblock(HANDLER_CLEANUP) + if let FBlockDatum::ExceptionName(_) = info.fb_datum { + // Named handler: PopBlock for inner SETUP_CLEANUP + emit!(self, PseudoInstruction::PopBlock); + } if preserve_tos { emit!(self, Instruction::Swap { index: 2 }); } - emit!(self, Instruction::PopException); + // PopBlock for outer SETUP_CLEANUP (ExceptionHandler) + emit!(self, PseudoInstruction::PopBlock); + emit!(self, Instruction::PopExcept); // If there's an exception name, clean it up if let FBlockDatum::ExceptionName(ref name) = info.fb_datum { @@ -1129,46 +1516,20 @@ impl Compiler { self.unwind_fblock(&fblock_info, preserve_tos)?; } UnwindInfo::FinallyTry { body, fblock_idx } => { + // codegen_unwind_fblock(FINALLY_TRY) + emit!(self, PseudoInstruction::PopBlock); + // Temporarily remove the FinallyTry fblock so nested return/break/continue // in the finally body won't see it again let code = self.current_code_info(); let saved_fblock = code.fblock.remove(fblock_idx); // Push PopValue fblock if preserving tos - // IMPORTANT: When preserving TOS (return value), we need to update the - // exception handler's stack_depth to account for the return value on stack. - // Otherwise, if an exception occurs during the finally body, the stack - // will be unwound to the wrong depth and the return value will be lost. if preserve_tos { - // Get the handler info from the saved fblock (or current handler) - // and create a new handler with stack_depth + 1 - let (handler, stack_depth, preserve_lasti) = - if let Some(handler) = saved_fblock.fb_handler { - ( - Some(handler), - saved_fblock.fb_stack_depth + 1, // +1 for return value - saved_fblock.fb_preserve_lasti, - ) - } else { - // No handler in saved_fblock, check current handler - if let Some(current_handler) = self.current_except_handler() { - ( - Some(current_handler.handler_block), - current_handler.stack_depth + 1, // +1 for return value - current_handler.preserve_lasti, - ) - } else { - (None, 1, false) // No handler, but still track the return value - } - }; - - self.push_fblock_with_handler( + self.push_fblock( FBlockType::PopValue, saved_fblock.fb_block, saved_fblock.fb_block, - handler, - stack_depth, - preserve_lasti, )?; } @@ -1188,33 +1549,13 @@ impl Compiler { Ok(()) } - /// Get the current exception handler from fblock stack - fn current_except_handler(&self) -> Option<ir::ExceptHandlerInfo> { - let code = self.code_stack.last()?; - // Walk fblock stack from top to find the nearest exception handler - for fblock in code.fblock.iter().rev() { - if let Some(handler) = fblock.fb_handler { - return Some(ir::ExceptHandlerInfo { - handler_block: handler, - stack_depth: fblock.fb_stack_depth, - preserve_lasti: fblock.fb_preserve_lasti, - }); - } - } - None - } - // could take impl Into<Cow<str>>, but everything is borrowed from ast structs; we never // actually have a `String` to pass fn name(&mut self, name: &str) -> bytecode::NameIdx { self._name_inner(name, |i| &mut i.metadata.names) } fn varname(&mut self, name: &str) -> CompileResult<bytecode::NameIdx> { - if Self::is_forbidden_arg_name(name) { - return Err(self.error(CodegenErrorType::SyntaxError(format!( - "cannot assign to {name}", - )))); - } + // Note: __debug__ checks are now handled in symboltable phase Ok(self._name_inner(name, |i| &mut i.metadata.varnames)) } fn _name_inner( @@ -1252,7 +1593,7 @@ impl Compiler { let mut parent_idx = stack_size - 2; let mut parent = &self.code_stack[parent_idx]; - // If parent is TypeParams scope, look at grandparent + // If parent is ast::TypeParams scope, look at grandparent // Check if parent is a type params scope by name pattern if parent.metadata.name.starts_with("<generic parameters of ") { if stack_size == 2 { @@ -1299,7 +1640,7 @@ impl Compiler { let parent_obj_name = &parent.metadata.name; // Determine if parent is a function-like scope - let is_function_parent = parent.flags.contains(bytecode::CodeFlags::IS_OPTIMIZED) + let is_function_parent = parent.flags.contains(bytecode::CodeFlags::OPTIMIZED) && !parent_obj_name.starts_with("<") // Not a special scope like <lambda>, <listcomp>, etc. && parent_obj_name != "<module>"; // Not the module scope @@ -1325,12 +1666,16 @@ impl Compiler { fn compile_program( &mut self, - body: &ModModule, + body: &ast::ModModule, symbol_table: SymbolTable, ) -> CompileResult<()> { let size_before = self.code_stack.len(); + // Set future_annotations from symbol table (detected during symbol table scan) + self.future_annotations = symbol_table.future_annotations; self.symbol_table_stack.push(symbol_table); + self.emit_resume_for_scope(CompilerScope::Module, 1); + let (doc, statements) = split_doc(&body.body, &self.opts); if let Some(value) = doc { self.emit_load_const(ConstantData::Str { @@ -1340,10 +1685,24 @@ impl Compiler { emit!(self, Instruction::StoreGlobal(doc)) } + // Handle annotations based on future_annotations flag if Self::find_ann(statements) { - emit!(self, Instruction::SetupAnnotation); + if self.future_annotations { + // PEP 563: Initialize __annotations__ dict + emit!(self, Instruction::SetupAnnotations); + } else { + // PEP 649: Generate __annotate__ function FIRST (before statements) + self.compile_module_annotate(statements)?; + + // PEP 649: Initialize __conditional_annotations__ set after __annotate__ + if self.current_symbol_table().has_conditional_annotations { + emit!(self, Instruction::BuildSet { size: 0 }); + self.store_name("__conditional_annotations__")?; + } + } } + // Compile all statements self.compile_statements(statements)?; assert_eq!(self.code_stack.len(), size_before); @@ -1355,18 +1714,35 @@ impl Compiler { fn compile_program_single( &mut self, - body: &[Stmt], + body: &[ast::Stmt], symbol_table: SymbolTable, ) -> CompileResult<()> { + // Set future_annotations from symbol table (detected during symbol table scan) + self.future_annotations = symbol_table.future_annotations; self.symbol_table_stack.push(symbol_table); + self.emit_resume_for_scope(CompilerScope::Module, 1); + + // Handle annotations based on future_annotations flag if Self::find_ann(body) { - emit!(self, Instruction::SetupAnnotation); + if self.future_annotations { + // PEP 563: Initialize __annotations__ dict + emit!(self, Instruction::SetupAnnotations); + } else { + // PEP 649: Generate __annotate__ function FIRST (before statements) + self.compile_module_annotate(body)?; + + // PEP 649: Initialize __conditional_annotations__ set after __annotate__ + if self.current_symbol_table().has_conditional_annotations { + emit!(self, Instruction::BuildSet { size: 0 }); + self.store_name("__conditional_annotations__")?; + } + } } if let Some((last, body)) = body.split_last() { for statement in body { - if let Stmt::Expr(StmtExpr { value, .. }) = &statement { + if let ast::Stmt::Expr(ast::StmtExpr { value, .. }) = &statement { self.compile_expression(value)?; emit!( self, @@ -1381,9 +1757,9 @@ impl Compiler { } } - if let Stmt::Expr(StmtExpr { value, .. }) = &last { + if let ast::Stmt::Expr(ast::StmtExpr { value, .. }) = &last { self.compile_expression(value)?; - emit!(self, Instruction::CopyItem { index: 1_u32 }); + emit!(self, Instruction::Copy { index: 1_u32 }); emit!( self, Instruction::CallIntrinsic1 { @@ -1406,22 +1782,23 @@ impl Compiler { fn compile_block_expr( &mut self, - body: &[Stmt], + body: &[ast::Stmt], symbol_table: SymbolTable, ) -> CompileResult<()> { self.symbol_table_stack.push(symbol_table); + self.emit_resume_for_scope(CompilerScope::Module, 1); self.compile_statements(body)?; if let Some(last_statement) = body.last() { match last_statement { - Stmt::Expr(_) => { + ast::Stmt::Expr(_) => { self.current_block().instructions.pop(); // pop Instruction::PopTop } - Stmt::FunctionDef(_) | Stmt::ClassDef(_) => { + ast::Stmt::FunctionDef(_) | ast::Stmt::ClassDef(_) => { let pop_instructions = self.current_block().instructions.pop(); let store_inst = compiler_unwrap_option(self, pop_instructions); // pop Instruction::Store - emit!(self, Instruction::CopyItem { index: 1_u32 }); + emit!(self, Instruction::Copy { index: 1_u32 }); self.current_block().instructions.push(store_inst); } _ => self.emit_load_const(ConstantData::None), @@ -1435,16 +1812,18 @@ impl Compiler { // Compile statement in eval mode: fn compile_eval( &mut self, - expression: &ModExpression, + expression: &ast::ModExpression, symbol_table: SymbolTable, ) -> CompileResult<()> { self.symbol_table_stack.push(symbol_table); + self.emit_resume_for_scope(CompilerScope::Module, 1); + self.compile_expression(&expression.body)?; self.emit_return_value(); Ok(()) } - fn compile_statements(&mut self, statements: &[Stmt]) -> CompileResult<()> { + fn compile_statements(&mut self, statements: &[ast::Stmt]) -> CompileResult<()> { for statement in statements { self.compile_statement(statement)? } @@ -1468,15 +1847,6 @@ impl Compiler { symboltable::mangle_name(private, name) } - fn check_forbidden_name(&mut self, name: &str, usage: NameUsage) -> CompileResult<()> { - let msg = match usage { - NameUsage::Store if is_forbidden_name(name) => "cannot assign to", - NameUsage::Delete if is_forbidden_name(name) => "cannot delete", - _ => return Ok(()), - }; - Err(self.error(CodegenErrorType::SyntaxError(format!("{msg} {name}")))) - } - // = compiler_nameop fn compile_name(&mut self, name: &str, usage: NameUsage) -> CompileResult<()> { enum NameOp { @@ -1484,10 +1854,10 @@ impl Compiler { Global, Deref, Name, + DictOrGlobals, // PEP 649: can_see_class_scope } let name = self.mangle(name); - self.check_forbidden_name(&name, usage)?; // Special handling for __debug__ if NameUsage::Load == usage && name == "__debug__" { @@ -1500,16 +1870,18 @@ impl Compiler { // Determine the operation type based on symbol scope let is_function_like = self.ctx.in_func(); - // Look up the symbol, handling TypeParams scope specially - let (symbol_scope, _is_typeparams) = { + // Look up the symbol, handling ast::TypeParams and Annotation scopes specially + let (symbol_scope, can_see_class_scope) = { let current_table = self.current_symbol_table(); let is_typeparams = current_table.typ == CompilerScope::TypeParams; + let is_annotation = current_table.typ == CompilerScope::Annotation; + let can_see_class = current_table.can_see_class_scope; // First try to find in current table let symbol = current_table.lookup(name.as_ref()); - // If not found and we're in TypeParams scope, try parent scope - let symbol = if symbol.is_none() && is_typeparams { + // If not found and we're in ast::TypeParams or Annotation scope, try parent scope + let symbol = if symbol.is_none() && (is_typeparams || is_annotation) { self.symbol_table_stack .get(self.symbol_table_stack.len() - 2) // Try to get parent index .expect("Symbol has no parent! This is a compiler bug.") @@ -1518,14 +1890,46 @@ impl Compiler { symbol }; - (symbol.map(|s| s.scope), is_typeparams) + (symbol.map(|s| s.scope), can_see_class) }; - let actual_scope = symbol_scope.ok_or_else(|| { - self.error(CodegenErrorType::SyntaxError(format!( - "The symbol '{name}' must be present in the symbol table" - ))) - })?; + // Special handling for class scope implicit cell variables + // These are treated as Cell even if not explicitly marked in symbol table + // Only for LOAD operations - explicit stores like `__class__ = property(...)` + // should use STORE_NAME to store in class namespace dict + let symbol_scope = { + let current_table = self.current_symbol_table(); + if current_table.typ == CompilerScope::Class + && usage == NameUsage::Load + && (name == "__class__" + || name == "__classdict__" + || name == "__conditional_annotations__") + { + Some(SymbolScope::Cell) + } else { + symbol_scope + } + }; + + // In annotation or type params scope, missing symbols are treated as global implicit + // This allows referencing global names like Union, Optional, etc. that are imported + // at module level but not explicitly bound in the function scope + let actual_scope = match symbol_scope { + Some(scope) => scope, + None => { + let current_table = self.current_symbol_table(); + if matches!( + current_table.typ, + CompilerScope::Annotation | CompilerScope::TypeParams + ) { + SymbolScope::GlobalImplicit + } else { + return Err(self.error(CodegenErrorType::SyntaxError(format!( + "the symbol '{name}' must be present in the symbol table" + )))); + } + } + }; // Determine operation type based on scope let op_type = match actual_scope { @@ -1539,7 +1943,11 @@ impl Compiler { } } SymbolScope::GlobalImplicit => { - if is_function_like { + // PEP 649: In annotation scope with class visibility, use DictOrGlobals + // to check classdict first before globals + if can_see_class_scope { + NameOp::DictOrGlobals + } else if is_function_like { NameOp::Global } else { NameOp::Name @@ -1560,9 +1968,15 @@ impl Compiler { let op = match usage { NameUsage::Load => { - // Special case for class scope + // ClassBlock (not inlined comp): LOAD_LOCALS first, then LOAD_FROM_DICT_OR_DEREF if self.ctx.in_class && !self.ctx.in_func() { - Instruction::LoadClassDeref + emit!(self, Instruction::LoadLocals); + Instruction::LoadFromDictOrDeref + // can_see_class_scope: LOAD_DEREF(__classdict__) first + } else if can_see_class_scope { + let classdict_idx = self.get_free_var_index("__classdict__")?; + self.emit_arg(classdict_idx, Instruction::LoadDeref); + Instruction::LoadFromDictOrDeref } else { Instruction::LoadDeref } @@ -1593,33 +2007,51 @@ impl Compiler { NameOp::Name => { let idx = self.get_global_name_index(&name); let op = match usage { - NameUsage::Load => Instruction::LoadNameAny, - NameUsage::Store => Instruction::StoreLocal, - NameUsage::Delete => Instruction::DeleteLocal, + NameUsage::Load => Instruction::LoadName, + NameUsage::Store => Instruction::StoreName, + NameUsage::Delete => Instruction::DeleteName, }; self.emit_arg(idx, op); } + NameOp::DictOrGlobals => { + // PEP 649: First check classdict (from __classdict__ freevar), then globals + let idx = self.get_global_name_index(&name); + match usage { + NameUsage::Load => { + // Load __classdict__ first (it's a free variable in annotation scope) + let classdict_idx = self.get_free_var_index("__classdict__")?; + self.emit_arg(classdict_idx, Instruction::LoadDeref); + self.emit_arg(idx, Instruction::LoadFromDictOrGlobals); + } + // Store/Delete in annotation scope should use Name ops + NameUsage::Store => { + self.emit_arg(idx, Instruction::StoreName); + } + NameUsage::Delete => { + self.emit_arg(idx, Instruction::DeleteName); + } + } + } } Ok(()) } - fn compile_statement(&mut self, statement: &Stmt) -> CompileResult<()> { - use ruff_python_ast::*; + fn compile_statement(&mut self, statement: &ast::Stmt) -> CompileResult<()> { trace!("Compiling {statement:?}"); self.set_source_range(statement.range()); match &statement { // we do this here because `from __future__` still executes that `from` statement at runtime, // we still need to compile the ImportFrom down below - Stmt::ImportFrom(StmtImportFrom { module, names, .. }) + ast::Stmt::ImportFrom(ast::StmtImportFrom { module, names, .. }) if module.as_ref().map(|id| id.as_str()) == Some("__future__") => { self.compile_future_features(names)? } // ignore module-level doc comments - Stmt::Expr(StmtExpr { value, .. }) - if matches!(&**value, Expr::StringLiteral(..)) + ast::Stmt::Expr(ast::StmtExpr { value, .. }) + if matches!(&**value, ast::Expr::StringLiteral(..)) && matches!(self.done_with_future_stmts, DoneWithFuture::No) => { self.done_with_future_stmts = DoneWithFuture::DoneWithDoc @@ -1629,7 +2061,7 @@ impl Compiler { } match &statement { - Stmt::Import(StmtImport { names, .. }) => { + ast::Stmt::Import(ast::StmtImport { names, .. }) => { // import a, b, c as d for name in names { let name = &name; @@ -1642,7 +2074,7 @@ impl Compiler { if let Some(alias) = &name.asname { for part in name.name.split('.').skip(1) { let idx = self.name(part); - emit!(self, Instruction::LoadAttr { idx }); + self.emit_load_attr(idx); } self.store_name(alias.as_str())? } else { @@ -1650,7 +2082,7 @@ impl Compiler { } } } - Stmt::ImportFrom(StmtImportFrom { + ast::Stmt::ImportFrom(ast::StmtImportFrom { level, module, names, @@ -1695,6 +2127,7 @@ impl Compiler { func: bytecode::IntrinsicFunction1::ImportStar } ); + emit!(self, Instruction::PopTop); } else { // from mod import a, b as c @@ -1716,21 +2149,22 @@ impl Compiler { emit!(self, Instruction::PopTop); } } - Stmt::Expr(StmtExpr { value, .. }) => { + ast::Stmt::Expr(ast::StmtExpr { value, .. }) => { self.compile_expression(value)?; // Pop result of stack, since we not use it: emit!(self, Instruction::PopTop); } - Stmt::Global(_) | Stmt::Nonlocal(_) => { + ast::Stmt::Global(_) | ast::Stmt::Nonlocal(_) => { // Handled during symbol table construction. } - Stmt::If(StmtIf { + ast::Stmt::If(ast::StmtIf { test, body, elif_else_clauses, .. }) => { + self.enter_conditional_block(); match elif_else_clauses.as_slice() { // Only if [] => { @@ -1748,7 +2182,7 @@ impl Compiler { self.compile_statements(body)?; emit!( self, - Instruction::Jump { + PseudoInstruction::Jump { target: after_block } ); @@ -1764,7 +2198,7 @@ impl Compiler { self.compile_statements(&clause.body)?; emit!( self, - Instruction::Jump { + PseudoInstruction::Jump { target: after_block } ); @@ -1778,17 +2212,18 @@ impl Compiler { self.switch_to_block(after_block); } } + self.leave_conditional_block(); } - Stmt::While(StmtWhile { + ast::Stmt::While(ast::StmtWhile { test, body, orelse, .. }) => self.compile_while(test, body, orelse)?, - Stmt::With(StmtWith { + ast::Stmt::With(ast::StmtWith { items, body, is_async, .. }) => self.compile_with(items, body, *is_async)?, - Stmt::For(StmtFor { + ast::Stmt::For(ast::StmtFor { target, iter, body, @@ -1796,8 +2231,10 @@ impl Compiler { is_async, .. }) => self.compile_for(target, iter, body, orelse, *is_async)?, - Stmt::Match(StmtMatch { subject, cases, .. }) => self.compile_match(subject, cases)?, - Stmt::Raise(StmtRaise { + ast::Stmt::Match(ast::StmtMatch { subject, cases, .. }) => { + self.compile_match(subject, cases)? + } + ast::Stmt::Raise(ast::StmtRaise { exc, cause, range, .. }) => { let kind = match exc { @@ -1814,9 +2251,13 @@ impl Compiler { None => bytecode::RaiseKind::BareRaise, }; self.set_source_range(*range); - emit!(self, Instruction::Raise { kind }); + emit!(self, Instruction::RaiseVarargs { kind }); + // Start a new block so dead code after raise doesn't + // corrupt the except stack in label_exception_targets + let dead = self.new_block(); + self.switch_to_block(dead); } - Stmt::Try(StmtTry { + ast::Stmt::Try(ast::StmtTry { body, handlers, orelse, @@ -1824,13 +2265,15 @@ impl Compiler { is_star, .. }) => { + self.enter_conditional_block(); if *is_star { self.compile_try_star_except(body, handlers, orelse, finalbody)? } else { self.compile_try_statement(body, handlers, orelse, finalbody)? } + self.leave_conditional_block(); } - Stmt::FunctionDef(StmtFunctionDef { + ast::Stmt::FunctionDef(ast::StmtFunctionDef { name, parameters, body, @@ -1852,7 +2295,7 @@ impl Compiler { type_params.as_deref(), )? } - Stmt::ClassDef(StmtClassDef { + ast::Stmt::ClassDef(ast::StmtClassDef { name, body, decorator_list, @@ -1866,26 +2309,31 @@ impl Compiler { type_params.as_deref(), arguments.as_deref(), )?, - Stmt::Assert(StmtAssert { test, msg, .. }) => { + ast::Stmt::Assert(ast::StmtAssert { test, msg, .. }) => { // if some flag, ignore all assert statements! if self.opts.optimize == 0 { let after_block = self.new_block(); self.compile_jump_if(test, true, after_block)?; - let assertion_error = self.name("AssertionError"); - emit!(self, Instruction::LoadGlobal(assertion_error)); + emit!( + self, + Instruction::LoadCommonConstant { + idx: bytecode::CommonConstant::AssertionError + } + ); + emit!(self, Instruction::PushNull); match msg { Some(e) => { self.compile_expression(e)?; - emit!(self, Instruction::CallFunctionPositional { nargs: 1 }); + emit!(self, Instruction::Call { nargs: 1 }); } None => { - emit!(self, Instruction::CallFunctionPositional { nargs: 0 }); + emit!(self, Instruction::Call { nargs: 0 }); } } emit!( self, - Instruction::Raise { + Instruction::RaiseVarargs { kind: bytecode::RaiseKind::Raise, } ); @@ -1893,15 +2341,19 @@ impl Compiler { self.switch_to_block(after_block); } } - Stmt::Break(_) => { + ast::Stmt::Break(_) => { // Unwind fblock stack until we find a loop, emitting cleanup for each fblock self.compile_break_continue(statement.range(), true)?; + let dead = self.new_block(); + self.switch_to_block(dead); } - Stmt::Continue(_) => { + ast::Stmt::Continue(_) => { // Unwind fblock stack until we find a loop, emitting cleanup for each fblock self.compile_break_continue(statement.range(), false)?; + let dead = self.new_block(); + self.switch_to_block(dead); } - Stmt::Return(StmtReturn { value, .. }) => { + ast::Stmt::Return(ast::StmtReturn { value, .. }) => { if !self.ctx.in_func() { return Err( self.error_ranged(CodegenErrorType::InvalidReturn, statement.range()) @@ -1914,7 +2366,7 @@ impl Compiler { && self .current_code_info() .flags - .contains(bytecode::CodeFlags::IS_GENERATOR) + .contains(bytecode::CodeFlags::GENERATOR) { return Err(self.error_ranged( CodegenErrorType::AsyncReturnValue, @@ -1932,35 +2384,38 @@ impl Compiler { self.emit_return_const(ConstantData::None); } } + let dead = self.new_block(); + self.switch_to_block(dead); } - Stmt::Assign(StmtAssign { targets, value, .. }) => { + ast::Stmt::Assign(ast::StmtAssign { targets, value, .. }) => { self.compile_expression(value)?; for (i, target) in targets.iter().enumerate() { if i + 1 != targets.len() { - emit!(self, Instruction::CopyItem { index: 1_u32 }); + emit!(self, Instruction::Copy { index: 1_u32 }); } self.compile_store(target)?; } } - Stmt::AugAssign(StmtAugAssign { + ast::Stmt::AugAssign(ast::StmtAugAssign { target, op, value, .. }) => self.compile_augassign(target, op, value)?, - Stmt::AnnAssign(StmtAnnAssign { + ast::Stmt::AnnAssign(ast::StmtAnnAssign { target, annotation, value, + simple, .. - }) => self.compile_annotated_assign(target, annotation, value.as_deref())?, - Stmt::Delete(StmtDelete { targets, .. }) => { + }) => self.compile_annotated_assign(target, annotation, value.as_deref(), *simple)?, + ast::Stmt::Delete(ast::StmtDelete { targets, .. }) => { for target in targets { self.compile_delete(target)?; } } - Stmt::Pass(_) => { + ast::Stmt::Pass(_) => { // No need to emit any code here :) } - Stmt::TypeAlias(StmtTypeAlias { + ast::Stmt::TypeAlias(ast::StmtTypeAlias { name, type_params, value, @@ -1984,7 +2439,7 @@ impl Compiler { if let Some(type_params) = type_params { // For TypeAlias, we need to use push_symbol_table to properly handle the TypeAlias scope - self.push_symbol_table(); + self.push_symbol_table()?; // Compile type params and push to stack self.compile_type_params(type_params)?; @@ -2016,32 +2471,33 @@ impl Compiler { ); self.store_name(&name_string)?; } - Stmt::IpyEscapeCommand(_) => todo!(), + ast::Stmt::IpyEscapeCommand(_) => todo!(), } Ok(()) } - fn compile_delete(&mut self, expression: &Expr) -> CompileResult<()> { - use ruff_python_ast::*; + fn compile_delete(&mut self, expression: &ast::Expr) -> CompileResult<()> { match &expression { - Expr::Name(ExprName { id, .. }) => self.compile_name(id.as_str(), NameUsage::Delete)?, - Expr::Attribute(ExprAttribute { value, attr, .. }) => { - self.check_forbidden_name(attr.as_str(), NameUsage::Delete)?; + ast::Expr::Name(ast::ExprName { id, .. }) => { + self.compile_name(id.as_str(), NameUsage::Delete)? + } + ast::Expr::Attribute(ast::ExprAttribute { value, attr, .. }) => { self.compile_expression(value)?; let idx = self.name(attr.as_str()); emit!(self, Instruction::DeleteAttr { idx }); } - Expr::Subscript(ExprSubscript { + ast::Expr::Subscript(ast::ExprSubscript { value, slice, ctx, .. }) => { self.compile_subscript(value, slice, *ctx)?; } - Expr::Tuple(ExprTuple { elts, .. }) | Expr::List(ExprList { elts, .. }) => { + ast::Expr::Tuple(ast::ExprTuple { elts, .. }) + | ast::Expr::List(ast::ExprList { elts, .. }) => { for element in elts { self.compile_delete(element)?; } } - Expr::BinOp(_) | Expr::UnaryOp(_) => { + ast::Expr::BinOp(_) | ast::Expr::UnaryOp(_) => { return Err(self.error(CodegenErrorType::Delete("expression"))); } _ => return Err(self.error(CodegenErrorType::Delete(expression.python_name()))), @@ -2049,7 +2505,7 @@ impl Compiler { Ok(()) } - fn enter_function(&mut self, name: &str, parameters: &Parameters) -> CompileResult<()> { + fn enter_function(&mut self, name: &str, parameters: &ast::Parameters) -> CompileResult<()> { // TODO: partition_in_place let mut kw_without_defaults = vec![]; let mut kw_with_defaults = vec![]; @@ -2062,12 +2518,12 @@ impl Compiler { } self.push_output( - bytecode::CodeFlags::NEW_LOCALS | bytecode::CodeFlags::IS_OPTIMIZED, + bytecode::CodeFlags::NEWLOCALS | bytecode::CodeFlags::OPTIMIZED, parameters.posonlyargs.len().to_u32(), (parameters.posonlyargs.len() + parameters.args.len()).to_u32(), parameters.kwonlyargs.len().to_u32(), name.to_owned(), - ); + )?; let args_iter = core::iter::empty() .chain(&parameters.posonlyargs) @@ -2080,51 +2536,57 @@ impl Compiler { } if let Some(name) = parameters.vararg.as_deref() { - self.current_code_info().flags |= bytecode::CodeFlags::HAS_VARARGS; + self.current_code_info().flags |= bytecode::CodeFlags::VARARGS; self.varname(name.name.as_str())?; } if let Some(name) = parameters.kwarg.as_deref() { - self.current_code_info().flags |= bytecode::CodeFlags::HAS_VARKEYWORDS; + self.current_code_info().flags |= bytecode::CodeFlags::VARKEYWORDS; self.varname(name.name.as_str())?; } Ok(()) } - fn prepare_decorators(&mut self, decorator_list: &[Decorator]) -> CompileResult<()> { + /// Push decorators onto the stack in source order. + /// For @dec1 @dec2 def foo(): stack becomes [dec1, NULL, dec2, NULL] + fn prepare_decorators(&mut self, decorator_list: &[ast::Decorator]) -> CompileResult<()> { for decorator in decorator_list { self.compile_expression(&decorator.expression)?; + emit!(self, Instruction::PushNull); } Ok(()) } - fn apply_decorators(&mut self, decorator_list: &[Decorator]) { - // Apply decorators: + /// Apply decorators in reverse order (LIFO from stack). + /// Stack [dec1, NULL, dec2, NULL, func] -> dec2(func) -> dec1(dec2(func)) + /// The forward loop works because each Call pops from TOS, naturally + /// applying decorators bottom-up (innermost first). + fn apply_decorators(&mut self, decorator_list: &[ast::Decorator]) { for _ in decorator_list { - emit!(self, Instruction::CallFunctionPositional { nargs: 1 }); + emit!(self, Instruction::Call { nargs: 1 }); } } /// Compile type parameter bound or default in a separate scope and return closure fn compile_type_param_bound_or_default( &mut self, - expr: &Expr, + expr: &ast::Expr, name: &str, allow_starred: bool, ) -> CompileResult<()> { // Push the next symbol table onto the stack - self.push_symbol_table(); + self.push_symbol_table()?; // Get the current symbol table let key = self.symbol_table_stack.len() - 1; - let lineno = expr.range().start().to_u32(); + let lineno = self.get_source_line_number().get().to_u32(); // Enter scope with the type parameter name self.enter_scope(name, CompilerScope::TypeParams, key, lineno)?; // Compile the expression - if allow_starred && matches!(expr, Expr::Starred(_)) { - if let Expr::Starred(starred) = expr { + if allow_starred && matches!(expr, ast::Expr::Starred(_)) { + if let ast::Expr::Starred(starred) = expr { self.compile_expression(&starred.value)?; emit!(self, Instruction::UnpackSequence { size: 1 }); } @@ -2141,20 +2603,21 @@ impl Compiler { // Create type params function with closure self.make_closure(code, bytecode::MakeFunctionFlags::empty())?; + emit!(self, Instruction::PushNull); // Call the function immediately - emit!(self, Instruction::CallFunctionPositional { nargs: 0 }); + emit!(self, Instruction::Call { nargs: 0 }); Ok(()) } /// Store each type parameter so it is accessible to the current scope, and leave a tuple of /// all the type parameters on the stack. Handles default values per PEP 695. - fn compile_type_params(&mut self, type_params: &TypeParams) -> CompileResult<()> { + fn compile_type_params(&mut self, type_params: &ast::TypeParams) -> CompileResult<()> { // First, compile each type parameter and store it for type_param in &type_params.type_params { match type_param { - TypeParam::TypeVar(TypeParamTypeVar { + ast::TypeParam::TypeVar(ast::TypeParamTypeVar { name, bound, default, @@ -2198,10 +2661,10 @@ impl Compiler { ); } - emit!(self, Instruction::CopyItem { index: 1_u32 }); + emit!(self, Instruction::Copy { index: 1_u32 }); self.store_name(name.as_ref())?; } - TypeParam::ParamSpec(TypeParamParamSpec { name, default, .. }) => { + ast::TypeParam::ParamSpec(ast::TypeParamParamSpec { name, default, .. }) => { self.emit_load_const(ConstantData::Str { value: name.as_str().into(), }); @@ -2223,10 +2686,12 @@ impl Compiler { ); } - emit!(self, Instruction::CopyItem { index: 1_u32 }); + emit!(self, Instruction::Copy { index: 1_u32 }); self.store_name(name.as_ref())?; } - TypeParam::TypeVarTuple(TypeParamTypeVarTuple { name, default, .. }) => { + ast::TypeParam::TypeVarTuple(ast::TypeParamTypeVarTuple { + name, default, .. + }) => { self.emit_load_const(ConstantData::Str { value: name.as_str().into(), }); @@ -2249,7 +2714,7 @@ impl Compiler { ); } - emit!(self, Instruction::CopyItem { index: 1_u32 }); + emit!(self, Instruction::Copy { index: 1_u32 }); self.store_name(name.as_ref())?; } }; @@ -2265,10 +2730,10 @@ impl Compiler { fn compile_try_statement( &mut self, - body: &[Stmt], - handlers: &[ExceptHandler], - orelse: &[Stmt], - finalbody: &[Stmt], + body: &[ast::Stmt], + handlers: &[ast::ExceptHandler], + orelse: &[ast::Stmt], + finalbody: &[ast::Stmt], ) -> CompileResult<()> { let handler_block = self.new_block(); let finally_block = self.new_block(); @@ -2290,25 +2755,24 @@ impl Compiler { // Normal path jumps here to skip exception path blocks let end_block = self.new_block(); - // Calculate the stack depth at this point (for exception table) - // SETUP_FINALLY captures current stack depth - let current_depth = self.handler_stack_depth(); - // Setup a finally block if we have a finally statement. // Push fblock with handler info for exception table generation // IMPORTANT: handler goes to finally_except_block (exception path), not finally_block if !finalbody.is_empty() { - // No SetupFinally emit - exception table handles this - // Store finally body in fb_datum for unwind_fblock to compile inline // SETUP_FINALLY doesn't push lasti for try body handler // Exception table: L1 to L2 -> L4 [1] (no lasti) + let setup_target = finally_except_block.unwrap_or(finally_block); + emit!( + self, + PseudoInstruction::SetupFinally { + target: setup_target + } + ); + // Store finally body in fb_datum for unwind_fblock to compile inline self.push_fblock_full( FBlockType::FinallyTry, finally_block, finally_block, - finally_except_block, // Exception path goes to finally_except_block - current_depth, - false, // No lasti for first finally handler FBlockDatum::FinallyBody(finalbody.to_vec()), // Clone finally body for unwind )?; } @@ -2324,6 +2788,7 @@ impl Compiler { // Pop FinallyTry fblock BEFORE compiling orelse/finally (normal path) // This prevents exception table from covering the normal path if !finalbody.is_empty() { + emit!(self, PseudoInstruction::PopBlock); self.pop_fblock(FBlockType::FinallyTry); } @@ -2332,13 +2797,8 @@ impl Compiler { // Snapshot sub_tables before first finally compilation // This allows us to restore them for the second compilation (exception path) - let sub_tables_snapshot = if !finalbody.is_empty() && finally_except_block.is_some() { - Some( - self.symbol_table_stack - .last() - .map(|t| t.sub_tables.clone()) - .unwrap_or_default(), - ) + let sub_table_cursor = if !finalbody.is_empty() && finally_except_block.is_some() { + self.symbol_table_stack.last().map(|t| t.next_sub_table) } else { None }; @@ -2349,56 +2809,58 @@ impl Compiler { } // Jump to end (skip exception path blocks) - emit!(self, Instruction::Jump { target: end_block }); + emit!(self, PseudoInstruction::Jump { target: end_block }); if let Some(finally_except) = finally_except_block { // Restore sub_tables for exception path compilation - if let Some(snapshot) = sub_tables_snapshot + if let Some(cursor) = sub_table_cursor && let Some(current_table) = self.symbol_table_stack.last_mut() { - current_table.sub_tables = snapshot; + current_table.next_sub_table = cursor; } self.switch_to_block(finally_except); - // PUSH_EXC_INFO first, THEN push FinallyEnd fblock - // Stack after unwind (no lasti): [exc] (depth = current_depth + 1) - // Stack after PUSH_EXC_INFO: [prev_exc, exc] (depth = current_depth + 2) + // SETUP_CLEANUP before PUSH_EXC_INFO + if let Some(cleanup) = finally_cleanup_block { + emit!(self, PseudoInstruction::SetupCleanup { target: cleanup }); + } emit!(self, Instruction::PushExcInfo); if let Some(cleanup) = finally_cleanup_block { - // FinallyEnd fblock must be pushed AFTER PUSH_EXC_INFO - // Depth = current_depth + 1 (only prev_exc remains after RERAISE pops exc) - // Exception table: L4 to L5 -> L6 [2] lasti (cleanup handler DOES push lasti) - self.push_fblock_with_handler( - FBlockType::FinallyEnd, - cleanup, - cleanup, - Some(cleanup), - current_depth + 1, - true, // Cleanup handler pushes lasti - )?; + self.push_fblock(FBlockType::FinallyEnd, cleanup, cleanup)?; } self.compile_statements(finalbody)?; - // RERAISE 0 is emitted BEFORE pop_fblock - // This ensures RERAISE goes to cleanup block (FinallyEnd handler) - // which then properly restores prev_exc before going to outer handler + + // Pop FinallyEnd fblock BEFORE emitting RERAISE + // This ensures RERAISE routes to outer exception handler, not cleanup block + // Cleanup block is only for new exceptions raised during finally body execution + if finally_cleanup_block.is_some() { + emit!(self, PseudoInstruction::PopBlock); + self.pop_fblock(FBlockType::FinallyEnd); + } + + // Restore prev_exc as current exception before RERAISE + // Stack: [prev_exc, exc] -> COPY 2 -> [prev_exc, exc, prev_exc] + // POP_EXCEPT pops prev_exc and sets exc_info->exc_value = prev_exc + // Stack after POP_EXCEPT: [prev_exc, exc] + emit!(self, Instruction::Copy { index: 2_u32 }); + emit!(self, Instruction::PopExcept); + + // RERAISE 0: re-raise the original exception to outer handler emit!( self, - Instruction::Raise { + Instruction::RaiseVarargs { kind: bytecode::RaiseKind::ReraiseFromStack } ); - if finally_cleanup_block.is_some() { - self.pop_fblock(FBlockType::FinallyEnd); - } } if let Some(cleanup) = finally_cleanup_block { self.switch_to_block(cleanup); - emit!(self, Instruction::CopyItem { index: 3_u32 }); - emit!(self, Instruction::PopException); + emit!(self, Instruction::Copy { index: 3_u32 }); + emit!(self, Instruction::PopExcept); emit!( self, - Instruction::Raise { + Instruction::RaiseVarargs { kind: bytecode::RaiseKind::ReraiseFromStack } ); @@ -2409,20 +2871,17 @@ impl Compiler { } // try: - // Push fblock with handler info for exception table generation - // No SetupExcept emit - exception table handles this - self.push_fblock_with_handler( - FBlockType::TryExcept, - handler_block, - handler_block, - Some(handler_block), - current_depth, // stack depth for exception handler - false, // no lasti for except - )?; + emit!( + self, + PseudoInstruction::SetupFinally { + target: handler_block + } + ); + self.push_fblock(FBlockType::TryExcept, handler_block, handler_block)?; self.compile_statements(body)?; + emit!(self, PseudoInstruction::PopBlock); self.pop_fblock(FBlockType::TryExcept); - // No PopBlock emit - exception table handles this - emit!(self, Instruction::Jump { target: else_block }); + emit!(self, PseudoInstruction::Jump { target: else_block }); // except handlers: self.switch_to_block(handler_block); @@ -2433,33 +2892,42 @@ impl Compiler { // After PUSH_EXC_INFO, stack is [prev_exc, exc] // depth=1 means keep prev_exc on stack when routing to cleanup let cleanup_block = self.new_block(); - self.push_fblock_with_handler( - FBlockType::ExceptionHandler, - cleanup_block, - cleanup_block, - Some(cleanup_block), - current_depth + 1, // After PUSH_EXC_INFO: [prev_exc] stays on stack - true, // preserve_lasti for cleanup - )?; + emit!( + self, + PseudoInstruction::SetupCleanup { + target: cleanup_block + } + ); + self.push_fblock(FBlockType::ExceptionHandler, cleanup_block, cleanup_block)?; // Exception is on top of stack now, pushed by unwind_blocks - // PUSH_EXC_INFO transforms [exc] -> [prev_exc, exc] for PopException + // PUSH_EXC_INFO transforms [exc] -> [prev_exc, exc] for PopExcept emit!(self, Instruction::PushExcInfo); for handler in handlers { - let ExceptHandler::ExceptHandler(ExceptHandlerExceptHandler { - type_, name, body, .. + let ast::ExceptHandler::ExceptHandler(ast::ExceptHandlerExceptHandler { + type_, + name, + body, + .. }) = &handler; let next_handler = self.new_block(); // If we gave a typ, // check if this handler can handle the exception: if let Some(exc_type) = type_ { - // Duplicate exception for test: - emit!(self, Instruction::CopyItem { index: 1_u32 }); - // Check exception type: + // Stack: [prev_exc, exc] self.compile_expression(exc_type)?; - emit!(self, Instruction::JumpIfNotExcMatch(next_handler)); + // Stack: [prev_exc, exc, type] + emit!(self, Instruction::CheckExcMatch); + // Stack: [prev_exc, exc, bool] + emit!( + self, + Instruction::PopJumpIfFalse { + target: next_handler + } + ); + // Stack: [prev_exc, exc] // We have a match, store in name (except x as y) if let Some(alias) = name { @@ -2478,16 +2946,17 @@ impl Compiler { let handler_cleanup_block = if name.is_some() { // SETUP_CLEANUP(cleanup_end) for named handler let cleanup_end = self.new_block(); - // Stack at handler entry: [prev_exc, exc] - // depth = 1 (prev_exc on stack after exception is popped) - let handler_depth = current_depth + 1; - self.push_fblock_with_handler( + emit!( + self, + PseudoInstruction::SetupCleanup { + target: cleanup_end + } + ); + self.push_fblock_full( FBlockType::HandlerCleanup, cleanup_end, cleanup_end, - Some(cleanup_end), - handler_depth, - true, // preserve_lasti for RERAISE + FBlockDatum::ExceptionName(name.as_ref().unwrap().as_str().to_owned()), )?; Some(cleanup_end) } else { @@ -2500,12 +2969,16 @@ impl Compiler { self.compile_statements(body)?; self.pop_fblock(FBlockType::HandlerCleanup); + // PopBlock for inner SETUP_CLEANUP (named handler only) + if handler_cleanup_block.is_some() { + emit!(self, PseudoInstruction::PopBlock); + } // Create a block for normal path continuation (after handler body succeeds) let handler_normal_exit = self.new_block(); emit!( self, - Instruction::Jump { + PseudoInstruction::Jump { target: handler_normal_exit, } ); @@ -2528,7 +3001,7 @@ impl Compiler { // which does COPY 3; POP_EXCEPT; RERAISE emit!( self, - Instruction::Raise { + Instruction::RaiseVarargs { kind: bytecode::RaiseKind::ReraiseFromStack, } ); @@ -2537,11 +3010,11 @@ impl Compiler { // Switch to normal exit block - this is where handler body success continues self.switch_to_block(handler_normal_exit); + // PopBlock for outer SETUP_CLEANUP (ExceptionHandler) + emit!(self, PseudoInstruction::PopBlock); // Now pop ExceptionHandler - the normal path continues from here - // POP_BLOCK (HandlerCleanup) then POP_BLOCK (SETUP_CLEANUP) - // followed by POP_EXCEPT self.pop_fblock(FBlockType::ExceptionHandler); - emit!(self, Instruction::PopException); + emit!(self, Instruction::PopExcept); // Delete the exception variable if it was bound (normal path) if let Some(alias) = name { @@ -2551,24 +3024,24 @@ impl Compiler { self.compile_name(alias.as_str(), NameUsage::Delete)?; } + // Pop FinallyTry block before jumping to finally body. + // The else_block path also pops this; both paths must agree + // on the except stack when entering finally_block. + if !finalbody.is_empty() { + emit!(self, PseudoInstruction::PopBlock); + } + // Jump to finally block emit!( self, - Instruction::Jump { + PseudoInstruction::Jump { target: finally_block, } ); // Re-push ExceptionHandler for next handler in the loop // This will be popped at the end of handlers loop or when matched - self.push_fblock_with_handler( - FBlockType::ExceptionHandler, - cleanup_block, - cleanup_block, - Some(cleanup_block), - current_depth + 1, // After PUSH_EXC_INFO: [prev_exc] stays on stack - true, // preserve_lasti for cleanup - )?; + self.push_fblock(FBlockType::ExceptionHandler, cleanup_block, cleanup_block)?; // Emit a new label for the next handler self.switch_to_block(next_handler); @@ -2581,7 +3054,7 @@ impl Compiler { // NOTE: We emit RERAISE 0 BEFORE popping fblock so it is within cleanup handler scope emit!( self, - Instruction::Raise { + Instruction::RaiseVarargs { kind: bytecode::RaiseKind::ReraiseFromStack, } ); @@ -2596,11 +3069,11 @@ impl Compiler { // POP_EXCEPT: pop prev_exc from stack and restore -> [prev_exc, lasti, exc] // RERAISE 1: reraise with lasti self.switch_to_block(cleanup_block); - emit!(self, Instruction::CopyItem { index: 3_u32 }); - emit!(self, Instruction::PopException); + emit!(self, Instruction::Copy { index: 3_u32 }); + emit!(self, Instruction::PopExcept); emit!( self, - Instruction::Raise { + Instruction::RaiseVarargs { kind: bytecode::RaiseKind::ReraiseFromStack, } ); @@ -2612,18 +3085,13 @@ impl Compiler { // Pop the FinallyTry fblock before jumping to finally if !finalbody.is_empty() { - // No PopBlock/EnterFinally emit - exception table handles this + emit!(self, PseudoInstruction::PopBlock); self.pop_fblock(FBlockType::FinallyTry); } // Snapshot sub_tables before first finally compilation (for double compilation issue) - let sub_tables_snapshot = if !finalbody.is_empty() && finally_except_block.is_some() { - Some( - self.symbol_table_stack - .last() - .map(|t| t.sub_tables.clone()) - .unwrap_or_default(), - ) + let sub_table_cursor = if !finalbody.is_empty() && finally_except_block.is_some() { + self.symbol_table_stack.last().map(|t| t.next_sub_table) } else { None }; @@ -2634,7 +3102,7 @@ impl Compiler { self.compile_statements(finalbody)?; // Jump to end_block to skip exception path blocks // This prevents fall-through to finally_except_block - emit!(self, Instruction::Jump { target: end_block }); + emit!(self, PseudoInstruction::Jump { target: end_block }); } // finally (exception path) @@ -2642,52 +3110,50 @@ impl Compiler { // Stack at entry: [lasti, exc] (from exception table with preserve_lasti=true) if let Some(finally_except) = finally_except_block { // Restore sub_tables for exception path compilation - if let Some(snapshot) = sub_tables_snapshot + if let Some(cursor) = sub_table_cursor && let Some(current_table) = self.symbol_table_stack.last_mut() { - current_table.sub_tables = snapshot; + current_table.next_sub_table = cursor; } self.switch_to_block(finally_except); // SETUP_CLEANUP for finally body // Exceptions during finally body need to go to cleanup block - // Stack at entry: [lasti, exc] (lasti from exception table, exc pushed) - // After PUSH_EXC_INFO: [lasti, prev_exc, exc] - // So depth should account for lasti being on stack if let Some(cleanup) = finally_cleanup_block { - self.push_fblock_with_handler( - FBlockType::FinallyEnd, - cleanup, - cleanup, - Some(cleanup), - current_depth + 1, // [lasti] on stack before PUSH_EXC_INFO - true, - )?; + emit!(self, PseudoInstruction::SetupCleanup { target: cleanup }); } - - // PUSH_EXC_INFO: [lasti, exc] -> [lasti, prev_exc, exc] - // Sets exc as current VM exception, saves prev_exc for restoration emit!(self, Instruction::PushExcInfo); + if let Some(cleanup) = finally_cleanup_block { + self.push_fblock(FBlockType::FinallyEnd, cleanup, cleanup)?; + } // Run finally body self.compile_statements(finalbody)?; - // RERAISE 0 is emitted BEFORE pop_fblock - // This ensures RERAISE goes to cleanup block (FinallyEnd handler) - // which then properly restores prev_exc before going to outer handler - // RERAISE 0: reraise the exception on TOS + // Pop FinallyEnd fblock BEFORE emitting RERAISE + // This ensures RERAISE routes to outer exception handler, not cleanup block + // Cleanup block is only for new exceptions raised during finally body execution + if finally_cleanup_block.is_some() { + emit!(self, PseudoInstruction::PopBlock); + self.pop_fblock(FBlockType::FinallyEnd); + } + + // Restore prev_exc as current exception before RERAISE + // Stack: [lasti, prev_exc, exc] -> COPY 2 -> [lasti, prev_exc, exc, prev_exc] + // POP_EXCEPT pops prev_exc and sets exc_info->exc_value = prev_exc + // Stack after POP_EXCEPT: [lasti, prev_exc, exc] + emit!(self, Instruction::Copy { index: 2_u32 }); + emit!(self, Instruction::PopExcept); + + // RERAISE 0: re-raise the original exception to outer handler // Stack: [lasti, prev_exc, exc] - exception is on top emit!( self, - Instruction::Raise { + Instruction::RaiseVarargs { kind: bytecode::RaiseKind::ReraiseFromStack, } ); - - if finally_cleanup_block.is_some() { - self.pop_fblock(FBlockType::FinallyEnd); - } } // finally cleanup block @@ -2696,13 +3162,13 @@ impl Compiler { if let Some(cleanup) = finally_cleanup_block { self.switch_to_block(cleanup); // COPY 3: copy the exception from position 3 - emit!(self, Instruction::CopyItem { index: 3_u32 }); + emit!(self, Instruction::Copy { index: 3_u32 }); // POP_EXCEPT: restore prev_exc as current exception - emit!(self, Instruction::PopException); + emit!(self, Instruction::PopExcept); // RERAISE 1: reraise with lasti from stack emit!( self, - Instruction::Raise { + Instruction::RaiseVarargs { kind: bytecode::RaiseKind::ReraiseFromStack, } ); @@ -2717,10 +3183,10 @@ impl Compiler { fn compile_try_star_except( &mut self, - body: &[Stmt], - handlers: &[ExceptHandler], - orelse: &[Stmt], - finalbody: &[Stmt], + body: &[ast::Stmt], + handlers: &[ast::ExceptHandler], + orelse: &[ast::Stmt], + finalbody: &[ast::Stmt], ) -> CompileResult<()> { // compiler_try_star_except // Stack layout during handler processing: [prev_exc, orig, list, rest] @@ -2730,37 +3196,41 @@ impl Compiler { let end_block = self.new_block(); let reraise_star_block = self.new_block(); let reraise_block = self.new_block(); - let _cleanup_block = self.new_block(); - - // Calculate the stack depth at this point (for exception table) - let current_depth = self.handler_stack_depth(); + let finally_cleanup_block = if !finalbody.is_empty() { + Some(self.new_block()) + } else { + None + }; + let exit_block = self.new_block(); // Push fblock with handler info for exception table generation if !finalbody.is_empty() { - // No SetupFinally emit - exception table handles this - self.push_fblock_with_handler( + emit!( + self, + PseudoInstruction::SetupFinally { + target: finally_block + } + ); + self.push_fblock_full( FBlockType::FinallyTry, finally_block, finally_block, - Some(finally_block), - current_depth, // stack depth for exception handler - true, // preserve lasti for finally + FBlockDatum::FinallyBody(finalbody.to_vec()), )?; } // SETUP_FINALLY for try body - // Push fblock with handler info for exception table generation - self.push_fblock_with_handler( - FBlockType::TryExcept, - handler_block, - handler_block, - Some(handler_block), - current_depth, // stack depth for exception handler - false, // no lasti for except - )?; + emit!( + self, + PseudoInstruction::SetupFinally { + target: handler_block + } + ); + self.push_fblock(FBlockType::TryExcept, handler_block, handler_block)?; self.compile_statements(body)?; + emit!(self, PseudoInstruction::PopBlock); self.pop_fblock(FBlockType::TryExcept); - emit!(self, Instruction::Jump { target: else_block }); + emit!(self, PseudoInstruction::Jump { target: else_block }); // Exception handler entry self.switch_to_block(handler_block); @@ -2775,30 +3245,41 @@ impl Compiler { let eg_dummy2 = self.new_block(); self.push_fblock(FBlockType::ExceptionGroupHandler, eg_dummy1, eg_dummy2)?; + // Initialize handler stack before the loop + // BUILD_LIST 0 + COPY 2 to set up [prev_exc, orig, list, rest] + emit!(self, Instruction::BuildList { size: 0 }); + // Stack: [prev_exc, exc, []] + emit!(self, Instruction::Copy { index: 2 }); + // Stack: [prev_exc, orig, list, rest] + let n = handlers.len(); + if n == 0 { + // Empty handlers (invalid AST) - append rest to list and proceed + // Stack: [prev_exc, orig, list, rest] + emit!(self, Instruction::ListAppend { i: 0 }); + // Stack: [prev_exc, orig, list] + emit!( + self, + PseudoInstruction::Jump { + target: reraise_star_block + } + ); + } for (i, handler) in handlers.iter().enumerate() { - let ExceptHandler::ExceptHandler(ExceptHandlerExceptHandler { - type_, name, body, .. + let ast::ExceptHandler::ExceptHandler(ast::ExceptHandlerExceptHandler { + type_, + name, + body, + .. }) = handler; let no_match_block = self.new_block(); let next_block = self.new_block(); - // first handler creates list and copies exc - if i == 0 { - // ADDOP_I(c, loc, BUILD_LIST, 0); - emit!(self, Instruction::BuildList { size: 0 }); - // Stack: [prev_exc, exc, []] - // ADDOP_I(c, loc, COPY, 2); - emit!(self, Instruction::CopyItem { index: 2 }); - // Stack: [prev_exc, exc, [], exc_copy] - // Now stack is: [prev_exc, orig, list, rest] - } - // Compile exception type if let Some(exc_type) = type_ { // Check for unparenthesized tuple - if let Expr::Tuple(ExprTuple { elts, range, .. }) = exc_type.as_ref() + if let ast::Expr::Tuple(ast::ExprTuple { elts, range, .. }) = exc_type.as_ref() && let Some(first) = elts.first() && range.start().to_u32() == first.range().start().to_u32() { @@ -2820,25 +3301,19 @@ impl Compiler { // ADDOP_I(c, loc, COPY, 1); // ADDOP_JUMP(c, loc, POP_JUMP_IF_NONE, no_match); - emit!(self, Instruction::CopyItem { index: 1 }); - self.emit_load_const(ConstantData::None); - emit!(self, Instruction::IsOp(bytecode::Invert::No)); // is None? + emit!(self, Instruction::Copy { index: 1 }); emit!( self, - Instruction::PopJumpIfTrue { + Instruction::PopJumpIfNone { target: no_match_block } ); // Handler matched // Stack: [prev_exc, orig, list, new_rest, match] + // Note: CheckEgMatch already sets the matched exception as current exception let handler_except_block = self.new_block(); - // Set matched exception as current exception (for __context__ in handler body) - // This ensures that exceptions raised in the handler get the matched part - // as their __context__, not the original full exception group - emit!(self, Instruction::SetExcInfo); - // Store match to name or pop if let Some(alias) = name { self.store_name(alias.as_str())?; @@ -2848,21 +3323,28 @@ impl Compiler { // Stack: [prev_exc, orig, list, new_rest] // HANDLER_CLEANUP fblock for handler body - // Stack depth: prev_exc(1) + orig(1) + list(1) + new_rest(1) = 4 - let eg_handler_depth = self.handler_stack_depth() + 4; - self.push_fblock_with_handler( + emit!( + self, + PseudoInstruction::SetupCleanup { + target: handler_except_block + } + ); + self.push_fblock_full( FBlockType::HandlerCleanup, next_block, end_block, - Some(handler_except_block), - eg_handler_depth, - true, // preserve lasti + if let Some(alias) = name { + FBlockDatum::ExceptionName(alias.as_str().to_owned()) + } else { + FBlockDatum::None + }, )?; // Execute handler body self.compile_statements(body)?; // Handler body completed normally + emit!(self, PseudoInstruction::PopBlock); self.pop_fblock(FBlockType::HandlerCleanup); // Cleanup name binding @@ -2873,7 +3355,7 @@ impl Compiler { } // Jump to next handler - emit!(self, Instruction::Jump { target: next_block }); + emit!(self, PseudoInstruction::Jump { target: next_block }); // Handler raised an exception (cleanup_end label) self.switch_to_block(handler_except_block); @@ -2901,7 +3383,7 @@ impl Compiler { // JUMP except_with_error // We directly JUMP to next_block since no_match_block falls through to it - emit!(self, Instruction::Jump { target: next_block }); + emit!(self, PseudoInstruction::Jump { target: next_block }); // No match - pop match (None) self.switch_to_block(no_match_block); @@ -2925,7 +3407,7 @@ impl Compiler { // Stack: [prev_exc, orig, list] emit!( self, - Instruction::Jump { + PseudoInstruction::Jump { target: reraise_star_block } ); @@ -2950,15 +3432,13 @@ impl Compiler { // Stack: [prev_exc, result] // COPY 1 - emit!(self, Instruction::CopyItem { index: 1 }); + emit!(self, Instruction::Copy { index: 1 }); // Stack: [prev_exc, result, result] // POP_JUMP_IF_NOT_NONE reraise - self.emit_load_const(ConstantData::None); - emit!(self, Instruction::IsOp(bytecode::Invert::Yes)); // is not None? emit!( self, - Instruction::PopJumpIfTrue { + Instruction::PopJumpIfNotNone { target: reraise_block } ); @@ -2971,14 +3451,15 @@ impl Compiler { // POP_BLOCK - no-op for us with exception tables (fblocks handle this) // POP_EXCEPT - restore previous exception context - emit!(self, Instruction::PopException); + emit!(self, Instruction::PopExcept); // Stack: [] if !finalbody.is_empty() { + emit!(self, PseudoInstruction::PopBlock); self.pop_fblock(FBlockType::FinallyTry); } - emit!(self, Instruction::Jump { target: end_block }); + emit!(self, PseudoInstruction::Jump { target: end_block }); // Reraise the result self.switch_to_block(reraise_block); @@ -2990,7 +3471,7 @@ impl Compiler { // Stack: [result, prev_exc] // POP_EXCEPT - emit!(self, Instruction::PopException); + emit!(self, Instruction::PopExcept); // Stack: [result] // RERAISE 0 @@ -3002,13 +3483,17 @@ impl Compiler { // that branches from try body success (where FinallyTry is still active). // We need to re-push FinallyTry to reflect the correct fblock state for else path. if !finalbody.is_empty() { - self.push_fblock_with_handler( + emit!( + self, + PseudoInstruction::SetupFinally { + target: finally_block + } + ); + self.push_fblock_full( FBlockType::FinallyTry, finally_block, finally_block, - Some(finally_block), - current_depth, - true, + FBlockDatum::FinallyBody(finalbody.to_vec()), )?; } self.switch_to_block(else_block); @@ -3016,30 +3501,76 @@ impl Compiler { if !finalbody.is_empty() { // Pop the FinallyTry fblock we just pushed for the else path + emit!(self, PseudoInstruction::PopBlock); self.pop_fblock(FBlockType::FinallyTry); } - emit!(self, Instruction::Jump { target: end_block }); + emit!(self, PseudoInstruction::Jump { target: end_block }); self.switch_to_block(end_block); if !finalbody.is_empty() { + // Snapshot sub_tables before first finally compilation + let sub_table_cursor = self.symbol_table_stack.last().map(|t| t.next_sub_table); + + // Compile finally body inline for normal path + self.compile_statements(finalbody)?; + emit!(self, PseudoInstruction::Jump { target: exit_block }); + + // Restore sub_tables for exception path compilation + if let Some(cursor) = sub_table_cursor + && let Some(current_table) = self.symbol_table_stack.last_mut() + { + current_table.next_sub_table = cursor; + } + + // Exception handler path self.switch_to_block(finally_block); + emit!(self, Instruction::PushExcInfo); + + if let Some(cleanup) = finally_cleanup_block { + emit!(self, PseudoInstruction::SetupCleanup { target: cleanup }); + self.push_fblock(FBlockType::FinallyEnd, cleanup, cleanup)?; + } + self.compile_statements(finalbody)?; - // No EndFinally emit - exception table handles this + + if finally_cleanup_block.is_some() { + emit!(self, PseudoInstruction::PopBlock); + self.pop_fblock(FBlockType::FinallyEnd); + } + + emit!(self, Instruction::Copy { index: 2_u32 }); + emit!(self, Instruction::PopExcept); + emit!( + self, + Instruction::RaiseVarargs { + kind: bytecode::RaiseKind::ReraiseFromStack + } + ); + + if let Some(cleanup) = finally_cleanup_block { + self.switch_to_block(cleanup); + emit!(self, Instruction::Copy { index: 3_u32 }); + emit!(self, Instruction::PopExcept); + emit!( + self, + Instruction::RaiseVarargs { + kind: bytecode::RaiseKind::ReraiseFromStack + } + ); + } } - Ok(()) - } + self.switch_to_block(exit_block); - fn is_forbidden_arg_name(name: &str) -> bool { - is_forbidden_name(name) + Ok(()) } /// Compile default arguments // = compiler_default_arguments fn compile_default_arguments( &mut self, - parameters: &Parameters, + parameters: &ast::Parameters, ) -> CompileResult<bytecode::MakeFunctionFlags> { let mut funcflags = bytecode::MakeFunctionFlags::empty(); @@ -3097,8 +3628,8 @@ impl Compiler { fn compile_function_body( &mut self, name: &str, - parameters: &Parameters, - body: &[Stmt], + parameters: &ast::Parameters, + body: &[ast::Stmt], is_async: bool, funcflags: bytecode::MakeFunctionFlags, ) -> CompileResult<()> { @@ -3106,7 +3637,7 @@ impl Compiler { self.enter_function(name, parameters)?; self.current_code_info() .flags - .set(bytecode::CodeFlags::IS_COROUTINE, is_async); + .set(bytecode::CodeFlags::COROUTINE, is_async); // Set up context let prev_ctx = self.ctx; @@ -3125,19 +3656,27 @@ impl Compiler { // Set qualname self.set_qualname(); - // Handle docstring + // Handle docstring - store in co_consts[0] if present let (doc_str, body) = split_doc(body, &self.opts); - self.current_code_info() - .metadata - .consts - .insert_full(ConstantData::None); + if let Some(doc) = &doc_str { + // Docstring present: store in co_consts[0] and set HAS_DOCSTRING flag + self.current_code_info() + .metadata + .consts + .insert_full(ConstantData::Str { + value: doc.to_string().into(), + }); + self.current_code_info().flags |= bytecode::CodeFlags::HAS_DOCSTRING; + } + // If no docstring, don't add None to co_consts + // Note: RETURN_GENERATOR + POP_TOP for async functions is emitted in enter_scope() // Compile body statements self.compile_statements(body)?; // Emit None at end if needed match body.last() { - Some(Stmt::Return(_)) => {} + Some(ast::Stmt::Return(_)) => {} _ => { self.emit_return_const(ConstantData::None); } @@ -3150,30 +3689,41 @@ impl Compiler { // Create function object with closure self.make_closure(code, funcflags)?; - // Handle docstring if present - if let Some(doc) = doc_str { - emit!(self, Instruction::CopyItem { index: 1_u32 }); - self.emit_load_const(ConstantData::Str { - value: doc.to_string().into(), - }); - emit!(self, Instruction::Swap { index: 2 }); - let doc_attr = self.name("__doc__"); - emit!(self, Instruction::StoreAttr { idx: doc_attr }); - } + // Note: docstring is now retrieved from co_consts[0] by the VM + // when HAS_DOCSTRING flag is set, so no runtime __doc__ assignment needed Ok(()) } - /// Compile function annotations - // = compiler_visit_annotations - fn visit_annotations( + /// Compile function annotations as a closure (PEP 649) + /// Returns true if an __annotate__ closure was created + /// Uses symbol table's annotation_block for proper scoping. + fn compile_annotations_closure( &mut self, - parameters: &Parameters, - returns: Option<&Expr>, - ) -> CompileResult<u32> { - let mut num_annotations = 0; + func_name: &str, + parameters: &ast::Parameters, + returns: Option<&ast::Expr>, + ) -> CompileResult<bool> { + // Try to enter annotation scope - returns false if no annotation_block exists + if !self.enter_annotation_scope(func_name)? { + return Ok(false); + } + + // Count annotations + let parameters_iter = core::iter::empty() + .chain(&parameters.posonlyargs) + .chain(&parameters.args) + .chain(&parameters.kwonlyargs) + .map(|x| &x.parameter) + .chain(parameters.vararg.as_deref()) + .chain(parameters.kwarg.as_deref()); + + let num_annotations: u32 = + u32::try_from(parameters_iter.filter(|p| p.annotation.is_some()).count()) + .expect("too many annotations") + + if returns.is_some() { 1 } else { 0 }; - // Handle parameter annotations + // Compile annotations inside the annotation scope let parameters_iter = core::iter::empty() .chain(&parameters.posonlyargs) .chain(&parameters.args) @@ -3188,20 +3738,238 @@ impl Compiler { value: self.mangle(param.name.as_str()).into_owned().into(), }); self.compile_annotation(annotation)?; - num_annotations += 1; } } - // Handle return annotation last + // Handle return annotation if let Some(annotation) = returns { self.emit_load_const(ConstantData::Str { value: "return".into(), }); self.compile_annotation(annotation)?; - num_annotations += 1; } - Ok(num_annotations) + // Build the map and return it + emit!( + self, + Instruction::BuildMap { + size: num_annotations, + } + ); + emit!(self, Instruction::ReturnValue); + + // Exit the annotation scope and get the code object + let annotate_code = self.exit_annotation_scope(); + + // Make a closure from the code object + self.make_closure(annotate_code, bytecode::MakeFunctionFlags::empty())?; + + Ok(true) + } + + /// Collect simple annotations from module body in AST order (including nested blocks) + /// Returns list of (name, annotation_expr) pairs + /// This must match the order that annotations are compiled to ensure + /// conditional_annotation_index stays in sync with __annotate__ enumeration. + fn collect_simple_annotations(body: &[ast::Stmt]) -> Vec<(&str, &ast::Expr)> { + fn walk<'a>(stmts: &'a [ast::Stmt], out: &mut Vec<(&'a str, &'a ast::Expr)>) { + for stmt in stmts { + match stmt { + ast::Stmt::AnnAssign(ast::StmtAnnAssign { + target, + annotation, + simple, + .. + }) if *simple && matches!(target.as_ref(), ast::Expr::Name(_)) => { + if let ast::Expr::Name(ast::ExprName { id, .. }) = target.as_ref() { + out.push((id.as_str(), annotation.as_ref())); + } + } + ast::Stmt::If(ast::StmtIf { + body, + elif_else_clauses, + .. + }) => { + walk(body, out); + for clause in elif_else_clauses { + walk(&clause.body, out); + } + } + ast::Stmt::For(ast::StmtFor { body, orelse, .. }) + | ast::Stmt::While(ast::StmtWhile { body, orelse, .. }) => { + walk(body, out); + walk(orelse, out); + } + ast::Stmt::With(ast::StmtWith { body, .. }) => walk(body, out), + ast::Stmt::Try(ast::StmtTry { + body, + handlers, + orelse, + finalbody, + .. + }) => { + walk(body, out); + for handler in handlers { + let ast::ExceptHandler::ExceptHandler( + ast::ExceptHandlerExceptHandler { body, .. }, + ) = handler; + walk(body, out); + } + walk(orelse, out); + walk(finalbody, out); + } + ast::Stmt::Match(ast::StmtMatch { cases, .. }) => { + for case in cases { + walk(&case.body, out); + } + } + _ => {} + } + } + } + let mut annotations = Vec::new(); + walk(body, &mut annotations); + annotations + } + + /// Compile module-level __annotate__ function (PEP 649) + /// Returns true if __annotate__ was created and stored + fn compile_module_annotate(&mut self, body: &[ast::Stmt]) -> CompileResult<bool> { + // Collect simple annotations from module body first + let annotations = Self::collect_simple_annotations(body); + + if annotations.is_empty() { + return Ok(false); + } + + // Check if we have conditional annotations + let has_conditional = self.current_symbol_table().has_conditional_annotations; + + // Get parent scope type and name BEFORE pushing annotation symbol table + let parent_scope_type = self.current_symbol_table().typ; + let parent_name = self + .symbol_table_stack + .last() + .map(|t| t.name.as_str()) + .unwrap_or("module") + .to_owned(); + let scope_name = format!("<annotate of {parent_name}>"); + + // Try to push annotation symbol table from current scope + if !self.push_current_annotation_symbol_table() { + return Ok(false); + } + + // Enter annotation scope for code generation + let key = self.symbol_table_stack.len() - 1; + let lineno = self.get_source_line_number().get(); + self.enter_scope(&scope_name, CompilerScope::Annotation, key, lineno.to_u32())?; + + // Add 'format' parameter to varnames + self.current_code_info() + .metadata + .varnames + .insert("format".to_owned()); + + // Emit format validation: if format > VALUE_WITH_FAKE_GLOBALS: raise NotImplementedError + self.emit_format_validation()?; + + if has_conditional { + // PEP 649: Build dict incrementally, checking conditional annotations + // Start with empty dict + emit!(self, Instruction::BuildMap { size: 0 }); + + // Process each annotation + for (idx, (name, annotation)) in annotations.iter().enumerate() { + // Check if index is in __conditional_annotations__ + let not_set_block = self.new_block(); + + // LOAD_CONST index + self.emit_load_const(ConstantData::Integer { value: idx.into() }); + // Load __conditional_annotations__ from appropriate scope + // Class scope: LoadDeref (freevars), Module scope: LoadGlobal + if parent_scope_type == CompilerScope::Class { + let idx = self.get_free_var_index("__conditional_annotations__")?; + emit!(self, Instruction::LoadDeref(idx)); + } else { + let cond_annotations_name = self.name("__conditional_annotations__"); + emit!(self, Instruction::LoadGlobal(cond_annotations_name)); + } + // CONTAINS_OP (in) + emit!(self, Instruction::ContainsOp(bytecode::Invert::No)); + // POP_JUMP_IF_FALSE not_set + emit!( + self, + Instruction::PopJumpIfFalse { + target: not_set_block + } + ); + + // Annotation value + self.compile_annotation(annotation)?; + // COPY dict to TOS + emit!(self, Instruction::Copy { index: 2 }); + // LOAD_CONST name + self.emit_load_const(ConstantData::Str { + value: self.mangle(name).into_owned().into(), + }); + // STORE_SUBSCR - dict[name] = value + emit!(self, Instruction::StoreSubscr); + + // not_set label + self.switch_to_block(not_set_block); + } + + // Return the dict + emit!(self, Instruction::ReturnValue); + } else { + // No conditional annotations - use simple BuildMap + let num_annotations = u32::try_from(annotations.len()).expect("too many annotations"); + + // Compile annotations inside the annotation scope + for (name, annotation) in annotations { + self.emit_load_const(ConstantData::Str { + value: self.mangle(name).into_owned().into(), + }); + self.compile_annotation(annotation)?; + } + + // Build the map and return it + emit!( + self, + Instruction::BuildMap { + size: num_annotations, + } + ); + emit!(self, Instruction::ReturnValue); + } + + // Exit annotation scope - pop symbol table, restore to parent's annotation_block, and get code + let annotation_table = self.pop_symbol_table(); + // Restore annotation_block to module's symbol table + self.symbol_table_stack + .last_mut() + .expect("no module symbol table") + .annotation_block = Some(Box::new(annotation_table)); + // Exit code scope + let pop = self.code_stack.pop(); + let annotate_code = unwrap_internal( + self, + compiler_unwrap_option(self, pop).finalize_code(&self.opts), + ); + + // Make a closure from the code object + self.make_closure(annotate_code, bytecode::MakeFunctionFlags::empty())?; + + // Store as __annotate_func__ for classes, __annotate__ for modules + let name = if parent_scope_type == CompilerScope::Class { + "__annotate_func__" + } else { + "__annotate__" + }; + self.store_name(name)?; + + Ok(true) } // = compiler_function @@ -3209,12 +3977,12 @@ impl Compiler { fn compile_function_def( &mut self, name: &str, - parameters: &Parameters, - body: &[Stmt], - decorator_list: &[Decorator], - returns: Option<&Expr>, // TODO: use type hint somehow.. + parameters: &ast::Parameters, + body: &[ast::Stmt], + decorator_list: &[ast::Decorator], + returns: Option<&ast::Expr>, // TODO: use type hint somehow.. is_async: bool, - type_params: Option<&TypeParams>, + type_params: Option<&ast::TypeParams>, ) -> CompileResult<()> { self.prepare_decorators(decorator_list)?; @@ -3233,20 +4001,15 @@ impl Compiler { num_typeparam_args += 1; } - // SWAP if we have both - if num_typeparam_args == 2 { - emit!(self, Instruction::Swap { index: 2 }); - } - // Enter type params scope let type_params_name = format!("<generic parameters of {name}>"); self.push_output( - bytecode::CodeFlags::IS_OPTIMIZED | bytecode::CodeFlags::NEW_LOCALS, + bytecode::CodeFlags::OPTIMIZED | bytecode::CodeFlags::NEWLOCALS, 0, num_typeparam_args as u32, 0, type_params_name, - ); + )?; // Add parameter names to varnames for the type params scope // These will be passed as arguments when the closure is called @@ -3273,18 +4036,12 @@ impl Compiler { } } - // Compile annotations - let mut annotations_flag = bytecode::MakeFunctionFlags::empty(); - let num_annotations = self.visit_annotations(parameters, returns)?; - if num_annotations > 0 { - annotations_flag = bytecode::MakeFunctionFlags::ANNOTATIONS; - emit!( - self, - Instruction::BuildMap { - size: num_annotations, - } - ); - } + // Compile annotations as closure (PEP 649) + let annotations_flag = if self.compile_annotations_closure(name, parameters, returns)? { + bytecode::MakeFunctionFlags::ANNOTATE + } else { + bytecode::MakeFunctionFlags::empty() + }; // Compile function body let final_funcflags = funcflags | annotations_flag; @@ -3316,23 +4073,40 @@ impl Compiler { // Make closure for type params code self.make_closure(type_params_code, bytecode::MakeFunctionFlags::empty())?; - // Call the closure + // Call the type params closure with defaults/kwdefaults as arguments. + // Call protocol: [callable, self_or_null, arg1, ..., argN] + // We need to reorder: [args..., closure] -> [closure, NULL, args...] + // Using Swap operations to move closure down and insert NULL. + // Note: num_typeparam_args is at most 2 (defaults tuple, kwdefaults dict). if num_typeparam_args > 0 { - emit!( - self, - Instruction::Swap { - index: (num_typeparam_args + 1) as u32 + match num_typeparam_args { + 1 => { + // Stack: [arg1, closure] + emit!(self, Instruction::Swap { index: 2 }); // [closure, arg1] + emit!(self, Instruction::PushNull); // [closure, arg1, NULL] + emit!(self, Instruction::Swap { index: 2 }); // [closure, NULL, arg1] } - ); + 2 => { + // Stack: [arg1, arg2, closure] + emit!(self, Instruction::Swap { index: 3 }); // [closure, arg2, arg1] + emit!(self, Instruction::Swap { index: 2 }); // [closure, arg1, arg2] + emit!(self, Instruction::PushNull); // [closure, arg1, arg2, NULL] + emit!(self, Instruction::Swap { index: 3 }); // [closure, NULL, arg2, arg1] + emit!(self, Instruction::Swap { index: 2 }); // [closure, NULL, arg1, arg2] + } + _ => unreachable!("only defaults and kwdefaults are supported"), + } emit!( self, - Instruction::CallFunctionPositional { + Instruction::Call { nargs: num_typeparam_args as u32 } ); } else { - // No arguments, just call the closure - emit!(self, Instruction::CallFunctionPositional { nargs: 0 }); + // Stack: [closure] + emit!(self, Instruction::PushNull); + // Stack: [closure, NULL] + emit!(self, Instruction::Call { nargs: 0 }); } } @@ -3348,12 +4122,18 @@ impl Compiler { /// Determines if a variable should be CELL or FREE type // = get_ref_type fn get_ref_type(&self, name: &str) -> Result<SymbolScope, CodegenErrorType> { - // Special handling for __class__ and __classdict__ in class scope - if self.ctx.in_class && (name == "__class__" || name == "__classdict__") { + let table = self.symbol_table_stack.last().unwrap(); + + // Special handling for __class__, __classdict__, and __conditional_annotations__ in class scope + // This should only apply when we're actually IN a class body, + // not when we're in a method nested inside a class. + if table.typ == CompilerScope::Class + && (name == "__class__" + || name == "__classdict__" + || name == "__conditional_annotations__") + { return Ok(SymbolScope::Cell); } - - let table = self.symbol_table_stack.last().unwrap(); match table.lookup(name) { Some(symbol) => match symbol.scope { SymbolScope::Cell => Ok(SymbolScope::Cell), @@ -3431,7 +4211,7 @@ impl Compiler { } }; - emit!(self, Instruction::LoadClosure(idx.to_u32())); + emit!(self, PseudoInstruction::LoadClosure(idx.to_u32())); } // Build tuple of closure variables @@ -3474,6 +4254,16 @@ impl Compiler { ); } + // Set __annotate__ closure if present (PEP 649) + if flags.contains(bytecode::MakeFunctionFlags::ANNOTATE) { + emit!( + self, + Instruction::SetFunctionAttribute { + attr: bytecode::MakeFunctionFlags::ANNOTATE + } + ); + } + // Set kwdefaults if present if flags.contains(bytecode::MakeFunctionFlags::KW_ONLY_DEFAULTS) { emit!( @@ -3508,15 +4298,14 @@ impl Compiler { } // Python/compile.c find_ann - fn find_ann(body: &[Stmt]) -> bool { - use ruff_python_ast::*; + fn find_ann(body: &[ast::Stmt]) -> bool { for statement in body { let res = match &statement { - Stmt::AnnAssign(_) => true, - Stmt::For(StmtFor { body, orelse, .. }) => { + ast::Stmt::AnnAssign(_) => true, + ast::Stmt::For(ast::StmtFor { body, orelse, .. }) => { Self::find_ann(body) || Self::find_ann(orelse) } - Stmt::If(StmtIf { + ast::Stmt::If(ast::StmtIf { body, elif_else_clauses, .. @@ -3524,11 +4313,11 @@ impl Compiler { Self::find_ann(body) || elif_else_clauses.iter().any(|x| Self::find_ann(&x.body)) } - Stmt::While(StmtWhile { body, orelse, .. }) => { + ast::Stmt::While(ast::StmtWhile { body, orelse, .. }) => { Self::find_ann(body) || Self::find_ann(orelse) } - Stmt::With(StmtWith { body, .. }) => Self::find_ann(body), - Stmt::Try(StmtTry { + ast::Stmt::With(ast::StmtWith { body, .. }) => Self::find_ann(body), + ast::Stmt::Try(ast::StmtTry { body, orelse, finalbody, @@ -3548,13 +4337,13 @@ impl Compiler { fn compile_class_body( &mut self, name: &str, - body: &[Stmt], - type_params: Option<&TypeParams>, + body: &[ast::Stmt], + type_params: Option<&ast::TypeParams>, firstlineno: u32, ) -> CompileResult<CodeObject> { // 1. Enter class scope let key = self.symbol_table_stack.len(); - self.push_symbol_table(); + self.push_symbol_table()?; self.enter_scope(name, CompilerScope::Class, key, firstlineno)?; // Set qualname using the new method @@ -3570,20 +4359,20 @@ impl Compiler { let dunder_name = self.name("__name__"); emit!(self, Instruction::LoadGlobal(dunder_name)); let dunder_module = self.name("__module__"); - emit!(self, Instruction::StoreLocal(dunder_module)); + emit!(self, Instruction::StoreName(dunder_module)); // Store __qualname__ self.emit_load_const(ConstantData::Str { value: qualname.into(), }); let qualname_name = self.name("__qualname__"); - emit!(self, Instruction::StoreLocal(qualname_name)); + emit!(self, Instruction::StoreName(qualname_name)); // Store __doc__ only if there's an explicit docstring if let Some(doc) = doc_str { self.emit_load_const(ConstantData::Str { value: doc.into() }); let doc_name = self.name("__doc__"); - emit!(self, Instruction::StoreLocal(doc_name)); + emit!(self, Instruction::StoreName(doc_name)); } // Store __firstlineno__ (new in Python 3.12+) @@ -3591,22 +4380,41 @@ impl Compiler { value: BigInt::from(firstlineno), }); let firstlineno_name = self.name("__firstlineno__"); - emit!(self, Instruction::StoreLocal(firstlineno_name)); + emit!(self, Instruction::StoreName(firstlineno_name)); // Set __type_params__ if we have type parameters if type_params.is_some() { // Load .type_params from enclosing scope let dot_type_params = self.name(".type_params"); - emit!(self, Instruction::LoadNameAny(dot_type_params)); + emit!(self, Instruction::LoadName(dot_type_params)); // Store as __type_params__ let dunder_type_params = self.name("__type_params__"); - emit!(self, Instruction::StoreLocal(dunder_type_params)); + emit!(self, Instruction::StoreName(dunder_type_params)); + } + + // PEP 649: Initialize __classdict__ cell for class annotation scope + if self.current_symbol_table().needs_classdict { + emit!(self, Instruction::LoadLocals); + let classdict_idx = self.get_cell_var_index("__classdict__")?; + emit!(self, Instruction::StoreDeref(classdict_idx)); } - // Setup annotations if needed + // Handle class annotations based on future_annotations flag if Self::find_ann(body) { - emit!(self, Instruction::SetupAnnotation); + if self.future_annotations { + // PEP 563: Initialize __annotations__ dict for class + emit!(self, Instruction::SetupAnnotations); + } else { + // PEP 649: Initialize __conditional_annotations__ set if needed for class + if self.current_symbol_table().has_conditional_annotations { + emit!(self, Instruction::BuildSet { size: 0 }); + self.store_name("__conditional_annotations__")?; + } + + // PEP 649: Generate __annotate__ function for class annotations + self.compile_module_annotate(body)?; + } } // 3. Compile the class body @@ -3623,10 +4431,10 @@ impl Compiler { .position(|var| *var == "__class__"); if let Some(classcell_idx) = classcell_idx { - emit!(self, Instruction::LoadClosure(classcell_idx.to_u32())); - emit!(self, Instruction::CopyItem { index: 1_u32 }); + emit!(self, PseudoInstruction::LoadClosure(classcell_idx.to_u32())); + emit!(self, Instruction::Copy { index: 1_u32 }); let classcell = self.name("__classcell__"); - emit!(self, Instruction::StoreLocal(classcell)); + emit!(self, Instruction::StoreName(classcell)); } else { self.emit_load_const(ConstantData::None); } @@ -3641,10 +4449,10 @@ impl Compiler { fn compile_class_def( &mut self, name: &str, - body: &[Stmt], - decorator_list: &[Decorator], - type_params: Option<&TypeParams>, - arguments: Option<&Arguments>, + body: &[ast::Stmt], + decorator_list: &[ast::Decorator], + type_params: Option<&ast::TypeParams>, + arguments: Option<&ast::Arguments>, ) -> CompileResult<()> { self.prepare_decorators(decorator_list)?; @@ -3655,12 +4463,12 @@ impl Compiler { if is_generic { let type_params_name = format!("<generic parameters of {name}>"); self.push_output( - bytecode::CodeFlags::IS_OPTIMIZED | bytecode::CodeFlags::NEW_LOCALS, + bytecode::CodeFlags::OPTIMIZED | bytecode::CodeFlags::NEWLOCALS, 0, 0, 0, type_params_name, - ); + )?; // Set private name for name mangling self.code_stack.last_mut().unwrap().private = Some(name.to_owned()); @@ -3668,7 +4476,7 @@ impl Compiler { // Compile type parameters and store as .type_params self.compile_type_params(type_params.unwrap())?; let dot_type_params = self.name(".type_params"); - emit!(self, Instruction::StoreLocal(dot_type_params)); + emit!(self, Instruction::StoreName(dot_type_params)); } // Step 2: Compile class body (always done, whether generic or not) @@ -3689,64 +4497,125 @@ impl Compiler { let dot_generic_base = self.name(".generic_base"); // Create .generic_base - emit!(self, Instruction::LoadNameAny(dot_type_params)); + emit!(self, Instruction::LoadName(dot_type_params)); emit!( self, Instruction::CallIntrinsic1 { func: bytecode::IntrinsicFunction1::SubscriptGeneric } ); - emit!(self, Instruction::StoreLocal(dot_generic_base)); + emit!(self, Instruction::StoreName(dot_generic_base)); // Generate class creation code emit!(self, Instruction::LoadBuildClass); + emit!(self, Instruction::PushNull); // Set up the class function with type params let mut func_flags = bytecode::MakeFunctionFlags::empty(); - emit!(self, Instruction::LoadNameAny(dot_type_params)); + emit!(self, Instruction::LoadName(dot_type_params)); func_flags |= bytecode::MakeFunctionFlags::TYPE_PARAMS; // Create class function with closure self.make_closure(class_code, func_flags)?; self.emit_load_const(ConstantData::Str { value: name.into() }); - // Compile original bases - let base_count = if let Some(arguments) = arguments { - for arg in &arguments.args { - self.compile_expression(arg)?; + // Compile bases and call __build_class__ + // Check for starred bases or **kwargs + let has_starred = arguments.is_some_and(|args| { + args.args + .iter() + .any(|arg| matches!(arg, ast::Expr::Starred(_))) + }); + let has_double_star = + arguments.is_some_and(|args| args.keywords.iter().any(|kw| kw.arg.is_none())); + + if has_starred || has_double_star { + // Use CallFunctionEx for *bases or **kwargs + // Stack has: [__build_class__, NULL, class_func, name] + // Need to build: args tuple = (class_func, name, *bases, .generic_base) + + // Build a list starting with class_func and name (2 elements already on stack) + emit!(self, Instruction::BuildList { size: 2 }); + + // Add bases to the list + if let Some(arguments) = arguments { + for arg in &arguments.args { + if let ast::Expr::Starred(ast::ExprStarred { value, .. }) = arg { + // Starred: compile and extend + self.compile_expression(value)?; + emit!(self, Instruction::ListExtend { i: 0 }); + } else { + // Non-starred: compile and append + self.compile_expression(arg)?; + emit!(self, Instruction::ListAppend { i: 0 }); + } + } + } + + // Add .generic_base as final element + emit!(self, Instruction::LoadName(dot_generic_base)); + emit!(self, Instruction::ListAppend { i: 0 }); + + // Convert list to tuple + emit!( + self, + Instruction::CallIntrinsic1 { + func: IntrinsicFunction1::ListToTuple + } + ); + + // Build kwargs if needed + if arguments.is_some_and(|args| !args.keywords.is_empty()) { + self.compile_keywords(&arguments.unwrap().keywords)?; + } else { + emit!(self, Instruction::PushNull); } - arguments.args.len() + emit!(self, Instruction::CallFunctionEx); } else { - 0 - }; + // Simple case: no starred bases, no **kwargs + // Compile bases normally + let base_count = if let Some(arguments) = arguments { + for arg in &arguments.args { + self.compile_expression(arg)?; + } + arguments.args.len() + } else { + 0 + }; - // Load .generic_base as the last base - emit!(self, Instruction::LoadNameAny(dot_generic_base)); + // Load .generic_base as the last base + emit!(self, Instruction::LoadName(dot_generic_base)); - let nargs = 2 + u32::try_from(base_count).expect("too many base classes") + 1; // function, name, bases..., generic_base + let nargs = 2 + u32::try_from(base_count).expect("too many base classes") + 1; - // Handle keyword arguments - if let Some(arguments) = arguments - && !arguments.keywords.is_empty() - { - for keyword in &arguments.keywords { - if let Some(name) = &keyword.arg { - self.emit_load_const(ConstantData::Str { + // Handle keyword arguments (no **kwargs here) + if let Some(arguments) = arguments + && !arguments.keywords.is_empty() + { + let mut kwarg_names = vec![]; + for keyword in &arguments.keywords { + let name = keyword.arg.as_ref().expect( + "keyword argument name must be set (no **kwargs in this branch)", + ); + kwarg_names.push(ConstantData::Str { value: name.as_str().into(), }); + self.compile_expression(&keyword.value)?; } - self.compile_expression(&keyword.value)?; + self.emit_load_const(ConstantData::Tuple { + elements: kwarg_names, + }); + emit!( + self, + Instruction::CallKw { + nargs: nargs + + u32::try_from(arguments.keywords.len()) + .expect("too many keyword arguments") + } + ); + } else { + emit!(self, Instruction::Call { nargs }); } - emit!( - self, - Instruction::CallFunctionKeyword { - nargs: nargs - + u32::try_from(arguments.keywords.len()) - .expect("too many keyword arguments") - } - ); - } else { - emit!(self, Instruction::CallFunctionPositional { nargs }); } // Return the created class @@ -3757,21 +4626,22 @@ impl Compiler { // Execute the type params function self.make_closure(type_params_code, bytecode::MakeFunctionFlags::empty())?; - emit!(self, Instruction::CallFunctionPositional { nargs: 0 }); + emit!(self, Instruction::PushNull); + emit!(self, Instruction::Call { nargs: 0 }); } else { // Non-generic class: standard path emit!(self, Instruction::LoadBuildClass); + emit!(self, Instruction::PushNull); // Create class function with closure self.make_closure(class_code, bytecode::MakeFunctionFlags::empty())?; self.emit_load_const(ConstantData::Str { value: name.into() }); - let call = if let Some(arguments) = arguments { - self.compile_call_inner(2, arguments)? + if let Some(arguments) = arguments { + self.codegen_call_helper(2, arguments)?; } else { - CallType::Positional { nargs: 2 } - }; - self.compile_normal_call(call); + emit!(self, Instruction::Call { nargs: 2 }); + } } // Step 4: Apply decorators and store (common to both paths) @@ -3779,7 +4649,14 @@ impl Compiler { self.store_name(name) } - fn compile_while(&mut self, test: &Expr, body: &[Stmt], orelse: &[Stmt]) -> CompileResult<()> { + fn compile_while( + &mut self, + test: &ast::Expr, + body: &[ast::Stmt], + orelse: &[ast::Stmt], + ) -> CompileResult<()> { + self.enter_conditional_block(); + let while_block = self.new_block(); let else_block = self.new_block(); let after_block = self.new_block(); @@ -3797,7 +4674,7 @@ impl Compiler { self.ctx.loop_data = was_in_loop; emit!( self, - Instruction::Jump { + PseudoInstruction::Jump { target: while_block, } ); @@ -3808,15 +4685,19 @@ impl Compiler { // Note: PopBlock is no longer emitted for loops self.compile_statements(orelse)?; self.switch_to_block(after_block); + + self.leave_conditional_block(); Ok(()) } fn compile_with( &mut self, - items: &[WithItem], - body: &[Stmt], + items: &[ast::WithItem], + body: &[ast::Stmt], is_async: bool, ) -> CompileResult<()> { + self.enter_conditional_block(); + // Python 3.12+ style with statement: // // BEFORE_WITH # TOS: ctx_mgr -> [__exit__, __enter__ result] @@ -3853,28 +4734,67 @@ impl Compiler { let exc_handler_block = self.new_block(); let after_block = self.new_block(); - // Compile context expression and BEFORE_WITH + // Compile context expression and load __enter__/__exit__ methods self.compile_expression(&item.context_expr)?; self.set_source_range(with_range); + // Stack: [cm] + emit!(self, Instruction::Copy { index: 1_u32 }); // [cm, cm] + if is_async { if self.ctx.func != FunctionContext::AsyncFunction { return Err(self.error(CodegenErrorType::InvalidAsyncWith)); } - emit!(self, Instruction::BeforeAsyncWith); - emit!(self, Instruction::GetAwaitable); + // Load __aexit__ and __aenter__, then call __aenter__ + emit!( + self, + Instruction::LoadSpecial { + method: SpecialMethod::AExit + } + ); // [cm, bound_aexit] + emit!(self, Instruction::Swap { index: 2_u32 }); // [bound_aexit, cm] + emit!( + self, + Instruction::LoadSpecial { + method: SpecialMethod::AEnter + } + ); // [bound_aexit, bound_aenter] + // bound_aenter is already bound, call with NULL self_or_null + emit!(self, Instruction::PushNull); // [bound_aexit, bound_aenter, NULL] + emit!(self, Instruction::Call { nargs: 0 }); // [bound_aexit, awaitable] + emit!(self, Instruction::GetAwaitable { arg: 1 }); self.emit_load_const(ConstantData::None); self.compile_yield_from_sequence(true)?; } else { - emit!(self, Instruction::BeforeWith); + // Load __exit__ and __enter__, then call __enter__ + emit!( + self, + Instruction::LoadSpecial { + method: SpecialMethod::Exit + } + ); // [cm, bound_exit] + emit!(self, Instruction::Swap { index: 2_u32 }); // [bound_exit, cm] + emit!( + self, + Instruction::LoadSpecial { + method: SpecialMethod::Enter + } + ); // [bound_exit, bound_enter] + // bound_enter is already bound, call with NULL self_or_null + emit!(self, Instruction::PushNull); // [bound_exit, bound_enter, NULL] + emit!(self, Instruction::Call { nargs: 0 }); // [bound_exit, enter_result] } // Stack: [..., __exit__, enter_result] // Push fblock for exception table - handler goes to exc_handler_block // preserve_lasti=true for with statements - // Use handler_stack_depth() to include all items on stack (for loops, etc.) - let with_depth = self.handler_stack_depth() + 1; // +1 for current __exit__ - self.push_fblock_with_handler( + emit!( + self, + PseudoInstruction::SetupWith { + target: exc_handler_block + } + ); + self.push_fblock( if is_async { FBlockType::AsyncWith } else { @@ -3882,9 +4802,6 @@ impl Compiler { }, exc_handler_block, // block start (will become exit target after store) after_block, - Some(exc_handler_block), - with_depth, - true, // preserve_lasti=true )?; // Store or pop the enter result @@ -3911,6 +4828,7 @@ impl Compiler { } // Pop fblock before normal exit + emit!(self, PseudoInstruction::PopBlock); self.pop_fblock(if is_async { FBlockType::AsyncWith } else { @@ -3921,19 +4839,20 @@ impl Compiler { // Stack: [..., __exit__] // Call __exit__(None, None, None) self.set_source_range(with_range); + emit!(self, Instruction::PushNull); self.emit_load_const(ConstantData::None); self.emit_load_const(ConstantData::None); self.emit_load_const(ConstantData::None); - emit!(self, Instruction::CallFunctionPositional { nargs: 3 }); + emit!(self, Instruction::Call { nargs: 3 }); if is_async { - emit!(self, Instruction::GetAwaitable); + emit!(self, Instruction::GetAwaitable { arg: 2 }); self.emit_load_const(ConstantData::None); self.compile_yield_from_sequence(true)?; } emit!(self, Instruction::PopTop); // Pop __exit__ result emit!( self, - Instruction::Jump { + PseudoInstruction::Jump { target: after_block } ); @@ -3947,20 +4866,13 @@ impl Compiler { let cleanup_block = self.new_block(); let suppress_block = self.new_block(); - // Push nested fblock for cleanup handler - // Stack at exc_handler_block entry: [..., __exit__, lasti, exc] - // After PUSH_EXC_INFO: [..., __exit__, lasti, prev_exc, exc] - // If exception in __exit__, cleanup handler entry: [..., __exit__, lasti, prev_exc, lasti2, exc2] - // cleanup_depth should be: with_depth + 2 (lasti + prev_exc) - let cleanup_depth = with_depth + 2; - self.push_fblock_with_handler( - FBlockType::ExceptionHandler, - exc_handler_block, - after_block, - Some(cleanup_block), - cleanup_depth, - true, // preserve_lasti=true - )?; + emit!( + self, + PseudoInstruction::SetupCleanup { + target: cleanup_block + } + ); + self.push_fblock(FBlockType::ExceptionHandler, exc_handler_block, after_block)?; // PUSH_EXC_INFO: [exc] -> [prev_exc, exc] emit!(self, Instruction::PushExcInfo); @@ -3971,7 +4883,7 @@ impl Compiler { emit!(self, Instruction::WithExceptStart); if is_async { - emit!(self, Instruction::GetAwaitable); + emit!(self, Instruction::GetAwaitable { arg: 2 }); self.emit_load_const(ConstantData::None); self.compile_yield_from_sequence(true)?; } @@ -3989,6 +4901,7 @@ impl Compiler { // handler points to the outer handler (try-except), not cleanup_block. // This is critical: when RERAISE propagates the exception, the exception // table should route it to the outer try-except, not back to cleanup. + emit!(self, PseudoInstruction::PopBlock); self.pop_fblock(FBlockType::ExceptionHandler); // Not suppressed: RERAISE 2 @@ -4000,12 +4913,12 @@ impl Compiler { // Need to pop: True, exc, prev_exc, __exit__ self.switch_to_block(suppress_block); emit!(self, Instruction::PopTop); // pop True (TO_BOOL result) - emit!(self, Instruction::PopException); // pop exc and restore prev_exc + emit!(self, Instruction::PopExcept); // pop exc and restore prev_exc emit!(self, Instruction::PopTop); // pop __exit__ emit!(self, Instruction::PopTop); // pop lasti emit!( self, - Instruction::Jump { + PseudoInstruction::Jump { target: after_block } ); @@ -4021,24 +4934,27 @@ impl Compiler { // to be in the exception table for these instructions. // If we cleared fblock, exceptions here would propagate uncaught. self.switch_to_block(cleanup_block); - emit!(self, Instruction::CopyItem { index: 3 }); - emit!(self, Instruction::PopException); + emit!(self, Instruction::Copy { index: 3 }); + emit!(self, Instruction::PopExcept); emit!(self, Instruction::Reraise { depth: 1 }); // ===== After block ===== self.switch_to_block(after_block); + self.leave_conditional_block(); Ok(()) } fn compile_for( &mut self, - target: &Expr, - iter: &Expr, - body: &[Stmt], - orelse: &[Stmt], + target: &ast::Expr, + iter: &ast::Expr, + body: &[ast::Stmt], + orelse: &[ast::Stmt], is_async: bool, ) -> CompileResult<()> { + self.enter_conditional_block(); + // Start loop let for_block = self.new_block(); let else_block = self.new_block(); @@ -4055,25 +4971,19 @@ impl Compiler { self.switch_to_block(for_block); - // Push fblock for async for loop with exception handler info - // Note: SetupExcept is no longer emitted (exception table handles StopAsyncIteration) - // Stack at this point: [..., async_iterator] - // We need handler_stack_depth() + 1 to keep parent items + async_iterator on stack when exception occurs - let async_for_depth = self.handler_stack_depth() + 1; - self.push_fblock_with_handler( - FBlockType::ForLoop, - for_block, - after_block, - Some(else_block), // Handler for StopAsyncIteration - async_for_depth, // stack depth: keep async_iterator and parent items - false, // no lasti needed - )?; + // codegen_async_for: push fblock BEFORE SETUP_FINALLY + self.push_fblock(FBlockType::ForLoop, for_block, after_block)?; + // SETUP_FINALLY to guard the __anext__ call + emit!(self, PseudoInstruction::SetupFinally { target: else_block }); emit!(self, Instruction::GetANext); self.emit_load_const(ConstantData::None); self.compile_yield_from_sequence(true)?; + // POP_BLOCK for SETUP_FINALLY - only GetANext/yield_from are protected + emit!(self, PseudoInstruction::PopBlock); + + // Success block for __anext__ self.compile_store(target)?; - // Note: PopBlock is no longer emitted (exception table handles this) } else { // Retrieve Iterator emit!(self, Instruction::GetIter); @@ -4092,20 +5002,27 @@ impl Compiler { let was_in_loop = self.ctx.loop_data.replace((for_block, after_block)); self.compile_statements(body)?; self.ctx.loop_data = was_in_loop; - emit!(self, Instruction::Jump { target: for_block }); + emit!(self, PseudoInstruction::Jump { target: for_block }); self.switch_to_block(else_block); - // Pop fblock + // Except block for __anext__ / end of sync for + // No PopBlock here - for async, POP_BLOCK is already in for_block self.pop_fblock(FBlockType::ForLoop); if is_async { emit!(self, Instruction::EndAsyncFor); + } else { + // END_FOR + POP_ITER pattern (CPython 3.14) + // FOR_ITER jumps to END_FOR, but VM skips it (+1) to reach POP_ITER + emit!(self, Instruction::EndFor); + emit!(self, Instruction::PopIter); } self.compile_statements(orelse)?; self.switch_to_block(after_block); + self.leave_conditional_block(); Ok(()) } @@ -4151,7 +5068,7 @@ impl Compiler { JumpOp::Jump => { emit!( self, - Instruction::Jump { + PseudoInstruction::Jump { target: pc.fail_pop[pops] } ); @@ -4218,7 +5135,7 @@ impl Compiler { /// to the list of captured names. fn pattern_helper_store_name( &mut self, - n: Option<&Identifier>, + n: Option<&ast::Identifier>, pc: &mut PatternContext, ) -> CompileResult<()> { match n { @@ -4252,7 +5169,7 @@ impl Compiler { } } - fn pattern_unpack_helper(&mut self, elts: &[Pattern]) -> CompileResult<()> { + fn pattern_unpack_helper(&mut self, elts: &[ast::Pattern]) -> CompileResult<()> { let n = elts.len(); let mut seen_star = false; for (i, elt) in elts.iter().enumerate() { @@ -4288,7 +5205,7 @@ impl Compiler { fn pattern_helper_sequence_unpack( &mut self, - patterns: &[Pattern], + patterns: &[ast::Pattern], _star: Option<usize>, pc: &mut PatternContext, ) -> CompileResult<()> { @@ -4308,7 +5225,7 @@ impl Compiler { fn pattern_helper_sequence_subscr( &mut self, - patterns: &[Pattern], + patterns: &[ast::Pattern], star: usize, pc: &mut PatternContext, ) -> CompileResult<()> { @@ -4324,7 +5241,7 @@ impl Compiler { continue; } // Duplicate the subject. - emit!(self, Instruction::CopyItem { index: 1_u32 }); + emit!(self, Instruction::Copy { index: 1_u32 }); if i < star { // For indices before the star, use a nonnegative index equal to i. self.emit_load_const(ConstantData::Integer { value: i.into() }); @@ -4344,7 +5261,12 @@ impl Compiler { ); } // Use BINARY_OP/NB_SUBSCR to extract the element. - emit!(self, Instruction::BinarySubscript); + emit!( + self, + Instruction::BinaryOp { + op: BinaryOperator::Subscr + } + ); // Compile the subpattern in irrefutable mode. self.compile_pattern_subpattern(pattern, pc)?; } @@ -4356,7 +5278,7 @@ impl Compiler { fn compile_pattern_subpattern( &mut self, - p: &Pattern, + p: &ast::Pattern, pc: &mut PatternContext, ) -> CompileResult<()> { // Save the current allow_irrefutable state. @@ -4372,7 +5294,7 @@ impl Compiler { fn compile_pattern_as( &mut self, - p: &PatternMatchAs, + p: &ast::PatternMatchAs, pc: &mut PatternContext, ) -> CompileResult<()> { // If there is no sub-pattern, then it's an irrefutable match. @@ -4397,7 +5319,7 @@ impl Compiler { // Otherwise, there is a sub-pattern. Duplicate the object on top of the stack. pc.on_top += 1; - emit!(self, Instruction::CopyItem { index: 1_u32 }); + emit!(self, Instruction::Copy { index: 1_u32 }); // Compile the sub-pattern. self.compile_pattern(p.pattern.as_ref().unwrap(), pc)?; // After success, decrement the on_top counter. @@ -4409,7 +5331,7 @@ impl Compiler { fn compile_pattern_star( &mut self, - p: &PatternMatchStar, + p: &ast::PatternMatchStar, pc: &mut PatternContext, ) -> CompileResult<()> { self.pattern_helper_store_name(p.name.as_ref(), pc)?; @@ -4420,8 +5342,8 @@ impl Compiler { /// and not duplicated. fn validate_kwd_attrs( &mut self, - attrs: &[Identifier], - _patterns: &[Pattern], + attrs: &[ast::Identifier], + _patterns: &[ast::Pattern], ) -> CompileResult<()> { let n_attrs = attrs.len(); for i in 0..n_attrs { @@ -4444,7 +5366,7 @@ impl Compiler { fn compile_pattern_class( &mut self, - p: &PatternMatchClass, + p: &ast::PatternMatchClass, pc: &mut PatternContext, ) -> CompileResult<()> { // Extract components from the MatchClass pattern. @@ -4498,7 +5420,7 @@ impl Compiler { // 2. Emit MATCH_CLASS with nargs. emit!(self, Instruction::MatchClass(u32::try_from(nargs).unwrap())); // 3. Duplicate the top of the stack. - emit!(self, Instruction::CopyItem { index: 1_u32 }); + emit!(self, Instruction::Copy { index: 1_u32 }); // 4. Load None. self.emit_load_const(ConstantData::None); // 5. Compare with IS_OP 1. @@ -4523,7 +5445,7 @@ impl Compiler { for subpattern in patterns.iter().chain(kwd_patterns.iter()) { // Check if this is a true wildcard (underscore pattern without name binding) let is_true_wildcard = match subpattern { - Pattern::MatchAs(match_as) => { + ast::Pattern::MatchAs(match_as) => { // Only consider it wildcard if both pattern and name are None (i.e., "_") match_as.pattern.is_none() && match_as.name.is_none() } @@ -4546,7 +5468,7 @@ impl Compiler { fn compile_pattern_mapping( &mut self, - p: &PatternMatchMapping, + p: &ast::PatternMatchMapping, pc: &mut PatternContext, ) -> CompileResult<()> { let mapping = p; @@ -4597,7 +5519,7 @@ impl Compiler { // Stack: [subject, len, size] emit!( self, - Instruction::CompareOperation { + Instruction::CompareOp { op: ComparisonOperator::GreaterOrEqual } ); @@ -4619,14 +5541,14 @@ impl Compiler { // Validate and compile keys let mut seen = HashSet::new(); for key in keys { - let is_attribute = matches!(key, Expr::Attribute(_)); + let is_attribute = matches!(key, ast::Expr::Attribute(_)); let is_literal = matches!( key, - Expr::NumberLiteral(_) - | Expr::StringLiteral(_) - | Expr::BytesLiteral(_) - | Expr::BooleanLiteral(_) - | Expr::NoneLiteral(_) + ast::Expr::NumberLiteral(_) + | ast::Expr::StringLiteral(_) + | ast::Expr::BytesLiteral(_) + | ast::Expr::BooleanLiteral(_) + | ast::Expr::NoneLiteral(_) ); let key_repr = if is_literal { UnparseExpr::new(key, &self.source_file).to_string() @@ -4663,7 +5585,7 @@ impl Compiler { pc.on_top += 2; // subject and keys_tuple are underneath // Check if match succeeded - emit!(self, Instruction::CopyItem { index: 1_u32 }); + emit!(self, Instruction::Copy { index: 1_u32 }); // Stack: [subject, keys_tuple, values_tuple, values_tuple_copy] // Check if copy is None (consumes the copy like POP_JUMP_IF_NONE) @@ -4716,14 +5638,14 @@ impl Compiler { // Copy rest_dict which is at position (1 + remaining) from TOS emit!( self, - Instruction::CopyItem { + Instruction::Copy { index: 1 + remaining } ); // Stack: [rest_dict, k1, ..., kn, rest_dict] emit!(self, Instruction::Swap { index: 2 }); // Stack: [rest_dict, k1, ..., kn-1, rest_dict, kn] - emit!(self, Instruction::DeleteSubscript); + emit!(self, Instruction::DeleteSubscr); // Stack: [rest_dict, k1, ..., kn-1] (removed kn from rest_dict) remaining -= 1; } @@ -4749,7 +5671,7 @@ impl Compiler { fn compile_pattern_or( &mut self, - p: &PatternMatchOr, + p: &ast::PatternMatchOr, pc: &mut PatternContext, ) -> CompileResult<()> { // Ensure the pattern is a MatchOr. @@ -4773,7 +5695,7 @@ impl Compiler { pc.fail_pop.clear(); pc.on_top = 0; // Emit a COPY(1) instruction before compiling the alternative. - emit!(self, Instruction::CopyItem { index: 1_u32 }); + emit!(self, Instruction::Copy { index: 1_u32 }); self.compile_pattern(alt, pc)?; let n_stores = pc.stores.len(); @@ -4817,7 +5739,7 @@ impl Compiler { } } // Emit a jump to the common end label and reset any failure jump targets. - emit!(self, Instruction::Jump { target: end }); + emit!(self, PseudoInstruction::Jump { target: end }); self.emit_and_reset_fail_pop(pc)?; } @@ -4857,11 +5779,11 @@ impl Compiler { fn compile_pattern_sequence( &mut self, - p: &PatternMatchSequence, + p: &ast::PatternMatchSequence, pc: &mut PatternContext, ) -> CompileResult<()> { // Ensure the pattern is a MatchSequence. - let patterns = &p.patterns; // a slice of Pattern + let patterns = &p.patterns; // a slice of ast::Pattern let size = patterns.len(); let mut star: Option<usize> = None; let mut only_wildcard = true; @@ -4901,7 +5823,7 @@ impl Compiler { self.emit_load_const(ConstantData::Integer { value: size.into() }); emit!( self, - Instruction::CompareOperation { + Instruction::CompareOp { op: ComparisonOperator::Equal } ); @@ -4914,7 +5836,7 @@ impl Compiler { }); emit!( self, - Instruction::CompareOperation { + Instruction::CompareOp { op: ComparisonOperator::GreaterOrEqual } ); @@ -4924,7 +5846,7 @@ impl Compiler { // Whatever comes next should consume the subject. pc.on_top -= 1; if only_wildcard { - // Patterns like: [] / [_] / [_, _] / [*_] / [_, *_] / [_, _, *_] / etc. + // ast::Patterns like: [] / [_] / [_, _] / [*_] / [_, *_] / [_, _, *_] / etc. emit!(self, Instruction::PopTop); } else if star_wildcard { self.pattern_helper_sequence_subscr(patterns, star.unwrap(), pc)?; @@ -4936,14 +5858,14 @@ impl Compiler { fn compile_pattern_value( &mut self, - p: &PatternMatchValue, + p: &ast::PatternMatchValue, pc: &mut PatternContext, ) -> CompileResult<()> { // TODO: ensure literal or attribute lookup self.compile_expression(&p.value)?; emit!( self, - Instruction::CompareOperation { + Instruction::CompareOp { op: bytecode::ComparisonOperator::Equal } ); @@ -4954,14 +5876,14 @@ impl Compiler { fn compile_pattern_singleton( &mut self, - p: &PatternMatchSingleton, + p: &ast::PatternMatchSingleton, pc: &mut PatternContext, ) -> CompileResult<()> { // Load the singleton constant value. self.emit_load_const(match p.value { - Singleton::None => ConstantData::None, - Singleton::False => ConstantData::Boolean { value: false }, - Singleton::True => ConstantData::Boolean { value: true }, + ast::Singleton::None => ConstantData::None, + ast::Singleton::False => ConstantData::Boolean { value: false }, + ast::Singleton::True => ConstantData::Boolean { value: true }, }); // Compare using the "Is" operator. emit!(self, Instruction::IsOp(Invert::No)); @@ -4972,32 +5894,32 @@ impl Compiler { fn compile_pattern( &mut self, - pattern_type: &Pattern, + pattern_type: &ast::Pattern, pattern_context: &mut PatternContext, ) -> CompileResult<()> { match &pattern_type { - Pattern::MatchValue(pattern_type) => { + ast::Pattern::MatchValue(pattern_type) => { self.compile_pattern_value(pattern_type, pattern_context) } - Pattern::MatchSingleton(pattern_type) => { + ast::Pattern::MatchSingleton(pattern_type) => { self.compile_pattern_singleton(pattern_type, pattern_context) } - Pattern::MatchSequence(pattern_type) => { + ast::Pattern::MatchSequence(pattern_type) => { self.compile_pattern_sequence(pattern_type, pattern_context) } - Pattern::MatchMapping(pattern_type) => { + ast::Pattern::MatchMapping(pattern_type) => { self.compile_pattern_mapping(pattern_type, pattern_context) } - Pattern::MatchClass(pattern_type) => { + ast::Pattern::MatchClass(pattern_type) => { self.compile_pattern_class(pattern_type, pattern_context) } - Pattern::MatchStar(pattern_type) => { + ast::Pattern::MatchStar(pattern_type) => { self.compile_pattern_star(pattern_type, pattern_context) } - Pattern::MatchAs(pattern_type) => { + ast::Pattern::MatchAs(pattern_type) => { self.compile_pattern_as(pattern_type, pattern_context) } - Pattern::MatchOr(pattern_type) => { + ast::Pattern::MatchOr(pattern_type) => { self.compile_pattern_or(pattern_type, pattern_context) } } @@ -5005,8 +5927,8 @@ impl Compiler { fn compile_match_inner( &mut self, - subject: &Expr, - cases: &[MatchCase], + subject: &ast::Expr, + cases: &[ast::MatchCase], pattern_context: &mut PatternContext, ) -> CompileResult<()> { self.compile_expression(subject)?; @@ -5020,7 +5942,7 @@ impl Compiler { for (i, m) in cases.iter().enumerate().take(case_count) { // Only copy the subject if not on the last case if i != case_count - 1 { - emit!(self, Instruction::CopyItem { index: 1_u32 }); + emit!(self, Instruction::Copy { index: 1_u32 }); } pattern_context.stores = Vec::with_capacity(1); @@ -5053,7 +5975,7 @@ impl Compiler { } self.compile_statements(&m.body)?; - emit!(self, Instruction::Jump { target: end }); + emit!(self, PseudoInstruction::Jump { target: end }); self.emit_and_reset_fail_pop(pattern_context)?; } @@ -5067,7 +5989,7 @@ impl Compiler { if let Some(ref guard) = m.guard { // Compile guard and jump to end if false self.compile_expression(guard)?; - emit!(self, Instruction::CopyItem { index: 1_u32 }); + emit!(self, Instruction::Copy { index: 1_u32 }); emit!(self, Instruction::PopJumpIfFalse { target: end }); emit!(self, Instruction::PopTop); } @@ -5077,28 +5999,34 @@ impl Compiler { Ok(()) } - fn compile_match(&mut self, subject: &Expr, cases: &[MatchCase]) -> CompileResult<()> { + fn compile_match( + &mut self, + subject: &ast::Expr, + cases: &[ast::MatchCase], + ) -> CompileResult<()> { + self.enter_conditional_block(); let mut pattern_context = PatternContext::new(); self.compile_match_inner(subject, cases, &mut pattern_context)?; + self.leave_conditional_block(); Ok(()) } /// [CPython `compiler_addcompare`](https://github.com/python/cpython/blob/627894459a84be3488a1789919679c997056a03c/Python/compile.c#L2880-L2924) - fn compile_addcompare(&mut self, op: &CmpOp) { + fn compile_addcompare(&mut self, op: &ast::CmpOp) { use bytecode::ComparisonOperator::*; match op { - CmpOp::Eq => emit!(self, Instruction::CompareOperation { op: Equal }), - CmpOp::NotEq => emit!(self, Instruction::CompareOperation { op: NotEqual }), - CmpOp::Lt => emit!(self, Instruction::CompareOperation { op: Less }), - CmpOp::LtE => emit!(self, Instruction::CompareOperation { op: LessOrEqual }), - CmpOp::Gt => emit!(self, Instruction::CompareOperation { op: Greater }), - CmpOp::GtE => { - emit!(self, Instruction::CompareOperation { op: GreaterOrEqual }) + ast::CmpOp::Eq => emit!(self, Instruction::CompareOp { op: Equal }), + ast::CmpOp::NotEq => emit!(self, Instruction::CompareOp { op: NotEqual }), + ast::CmpOp::Lt => emit!(self, Instruction::CompareOp { op: Less }), + ast::CmpOp::LtE => emit!(self, Instruction::CompareOp { op: LessOrEqual }), + ast::CmpOp::Gt => emit!(self, Instruction::CompareOp { op: Greater }), + ast::CmpOp::GtE => { + emit!(self, Instruction::CompareOp { op: GreaterOrEqual }) } - CmpOp::In => emit!(self, Instruction::ContainsOp(Invert::No)), - CmpOp::NotIn => emit!(self, Instruction::ContainsOp(Invert::Yes)), - CmpOp::Is => emit!(self, Instruction::IsOp(Invert::No)), - CmpOp::IsNot => emit!(self, Instruction::IsOp(Invert::Yes)), + ast::CmpOp::In => emit!(self, Instruction::ContainsOp(Invert::No)), + ast::CmpOp::NotIn => emit!(self, Instruction::ContainsOp(Invert::Yes)), + ast::CmpOp::Is => emit!(self, Instruction::IsOp(Invert::No)), + ast::CmpOp::IsNot => emit!(self, Instruction::IsOp(Invert::Yes)), } } @@ -5122,9 +6050,9 @@ impl Compiler { /// - [CPython `compiler_compare`](https://github.com/python/cpython/blob/627894459a84be3488a1789919679c997056a03c/Python/compile.c#L4678-L4717) fn compile_compare( &mut self, - left: &Expr, - ops: &[CmpOp], - comparators: &[Expr], + left: &ast::Expr, + ops: &[ast::CmpOp], + comparators: &[ast::Expr], ) -> CompileResult<()> { let (last_op, mid_ops) = ops.split_last().unwrap(); let (last_comparator, mid_comparators) = comparators.split_last().unwrap(); @@ -5147,26 +6075,21 @@ impl Compiler { // store rhs for the next comparison in chain emit!(self, Instruction::Swap { index: 2 }); - emit!(self, Instruction::CopyItem { index: 2 }); + emit!(self, Instruction::Copy { index: 2 }); self.compile_addcompare(op); // if comparison result is false, we break with this value; if true, try the next one. - /* - emit!(self, Instruction::CopyItem { index: 1 }); - // emit!(self, Instruction::ToBool); // TODO: Uncomment this + emit!(self, Instruction::Copy { index: 1 }); emit!(self, Instruction::PopJumpIfFalse { target: cleanup }); emit!(self, Instruction::PopTop); - */ - - emit!(self, Instruction::JumpIfFalseOrPop { target: cleanup }); } self.compile_expression(last_comparator)?; self.compile_addcompare(last_op); let end = self.new_block(); - emit!(self, Instruction::Jump { target: end }); + emit!(self, PseudoInstruction::Jump { target: end }); // early exit left us with stack: `rhs, comparison_result`. We need to clean up rhs. self.switch_to_block(cleanup); @@ -5177,7 +6100,7 @@ impl Compiler { Ok(()) } - fn compile_annotation(&mut self, annotation: &Expr) -> CompileResult<()> { + fn compile_annotation(&mut self, annotation: &ast::Expr) -> CompileResult<()> { if self.future_annotations { self.emit_load_const(ConstantData::Str { value: UnparseExpr::new(annotation, &self.source_file) @@ -5190,7 +6113,7 @@ impl Compiler { // Special handling for starred annotations (*Ts -> Unpack[Ts]) let result = match annotation { - Expr::Starred(ExprStarred { value, .. }) => { + ast::Expr::Starred(ast::ExprStarred { value, .. }) => { // *args: *Ts (where Ts is a TypeVarTuple). // Do [annotation_value] = [*Ts]. self.compile_expression(value)?; @@ -5208,59 +6131,89 @@ impl Compiler { fn compile_annotated_assign( &mut self, - target: &Expr, - annotation: &Expr, - value: Option<&Expr>, + target: &ast::Expr, + annotation: &ast::Expr, + value: Option<&ast::Expr>, + simple: bool, ) -> CompileResult<()> { + // Perform the actual assignment first if let Some(value) = value { self.compile_expression(value)?; self.compile_store(target)?; } - // Annotations are only evaluated in a module or class. - if self.ctx.in_func() { - return Ok(()); - } - - // Compile annotation: - self.compile_annotation(annotation)?; - - if let Expr::Name(ExprName { id, .. }) = &target { - // Store as dict entry in __annotations__ dict: - let annotations = self.name("__annotations__"); - emit!(self, Instruction::LoadNameAny(annotations)); - self.emit_load_const(ConstantData::Str { - value: self.mangle(id.as_str()).into_owned().into(), - }); - emit!(self, Instruction::StoreSubscript); - } else { - // Drop annotation if not assigned to simple identifier. - emit!(self, Instruction::PopTop); + // If we have a simple name in module or class scope, store annotation + if simple + && !self.ctx.in_func() + && let ast::Expr::Name(ast::ExprName { id, .. }) = target + { + if self.future_annotations { + // PEP 563: Store stringified annotation directly to __annotations__ + // Compile annotation as string + self.compile_annotation(annotation)?; + // Load __annotations__ + let annotations_name = self.name("__annotations__"); + emit!(self, Instruction::LoadName(annotations_name)); + // Load the variable name + self.emit_load_const(ConstantData::Str { + value: self.mangle(id.as_str()).into_owned().into(), + }); + // Store: __annotations__[name] = annotation + emit!(self, Instruction::StoreSubscr); + } else { + // PEP 649: Handle conditional annotations + if self.current_symbol_table().has_conditional_annotations { + // Allocate an index for every annotation when has_conditional_annotations + // This keeps indices aligned with compile_module_annotate's enumeration + let code_info = self.current_code_info(); + let annotation_index = code_info.next_conditional_annotation_index; + code_info.next_conditional_annotation_index += 1; + + // Determine if this annotation is conditional + // Module and Class scopes both need all annotations tracked + let scope_type = self.current_symbol_table().typ; + let in_conditional_block = self.current_code_info().in_conditional_block > 0; + let is_conditional = + matches!(scope_type, CompilerScope::Module | CompilerScope::Class) + || in_conditional_block; + + // Only add to __conditional_annotations__ set if actually conditional + if is_conditional { + let cond_annotations_name = self.name("__conditional_annotations__"); + emit!(self, Instruction::LoadName(cond_annotations_name)); + self.emit_load_const(ConstantData::Integer { + value: annotation_index.into(), + }); + emit!(self, Instruction::SetAdd { i: 0_u32 }); + emit!(self, Instruction::PopTop); + } + } + } } Ok(()) } - fn compile_store(&mut self, target: &Expr) -> CompileResult<()> { + fn compile_store(&mut self, target: &ast::Expr) -> CompileResult<()> { match &target { - Expr::Name(ExprName { id, .. }) => self.store_name(id.as_str())?, - Expr::Subscript(ExprSubscript { + ast::Expr::Name(ast::ExprName { id, .. }) => self.store_name(id.as_str())?, + ast::Expr::Subscript(ast::ExprSubscript { value, slice, ctx, .. }) => { self.compile_subscript(value, slice, *ctx)?; } - Expr::Attribute(ExprAttribute { value, attr, .. }) => { - self.check_forbidden_name(attr.as_str(), NameUsage::Store)?; + ast::Expr::Attribute(ast::ExprAttribute { value, attr, .. }) => { self.compile_expression(value)?; let idx = self.name(attr.as_str()); emit!(self, Instruction::StoreAttr { idx }); } - Expr::List(ExprList { elts, .. }) | Expr::Tuple(ExprTuple { elts, .. }) => { + ast::Expr::List(ast::ExprList { elts, .. }) + | ast::Expr::Tuple(ast::ExprTuple { elts, .. }) => { let mut seen_star = false; // Scan for star args: for (i, element) in elts.iter().enumerate() { - if let Expr::Starred(_) = &element { + if let ast::Expr::Starred(_) = &element { if seen_star { return Err(self.error(CodegenErrorType::MultipleStarArgs)); } else { @@ -5290,7 +6243,7 @@ impl Compiler { } for element in elts { - if let Expr::Starred(ExprStarred { value, .. }) = &element { + if let ast::Expr::Starred(ast::ExprStarred { value, .. }) = &element { self.compile_store(value)?; } else { self.compile_store(element)?; @@ -5299,7 +6252,7 @@ impl Compiler { } _ => { return Err(self.error(match target { - Expr::Starred(_) => CodegenErrorType::SyntaxError( + ast::Expr::Starred(_) => CodegenErrorType::SyntaxError( "starred assignment target must be in a list or tuple".to_owned(), ), _ => CodegenErrorType::Assign(target.python_name()), @@ -5312,9 +6265,9 @@ impl Compiler { fn compile_augassign( &mut self, - target: &Expr, - op: &Operator, - value: &Expr, + target: &ast::Expr, + op: &ast::Operator, + value: &ast::Expr, ) -> CompileResult<()> { enum AugAssignKind<'a> { Name { id: &'a str }, @@ -5323,12 +6276,12 @@ impl Compiler { } let kind = match &target { - Expr::Name(ExprName { id, .. }) => { + ast::Expr::Name(ast::ExprName { id, .. }) => { let id = id.as_str(); self.compile_name(id, NameUsage::Load)?; AugAssignKind::Name { id } } - Expr::Subscript(ExprSubscript { + ast::Expr::Subscript(ast::ExprSubscript { value, slice, ctx: _, @@ -5338,18 +6291,22 @@ impl Compiler { // But we can't use compile_subscript directly because we need DUP_TOP2 self.compile_expression(value)?; self.compile_expression(slice)?; - emit!(self, Instruction::CopyItem { index: 2_u32 }); - emit!(self, Instruction::CopyItem { index: 2_u32 }); - emit!(self, Instruction::Subscript); + emit!(self, Instruction::Copy { index: 2_u32 }); + emit!(self, Instruction::Copy { index: 2_u32 }); + emit!( + self, + Instruction::BinaryOp { + op: BinaryOperator::Subscr + } + ); AugAssignKind::Subscript } - Expr::Attribute(ExprAttribute { value, attr, .. }) => { + ast::Expr::Attribute(ast::ExprAttribute { value, attr, .. }) => { let attr = attr.as_str(); - self.check_forbidden_name(attr, NameUsage::Store)?; self.compile_expression(value)?; - emit!(self, Instruction::CopyItem { index: 1_u32 }); + emit!(self, Instruction::Copy { index: 1_u32 }); let idx = self.name(attr); - emit!(self, Instruction::LoadAttr { idx }); + self.emit_load_attr(idx); AugAssignKind::Attr { idx } } _ => { @@ -5369,7 +6326,7 @@ impl Compiler { // stack: CONTAINER SLICE RESULT emit!(self, Instruction::Swap { index: 3 }); emit!(self, Instruction::Swap { index: 2 }); - emit!(self, Instruction::StoreSubscript); + emit!(self, Instruction::StoreSubscr); } AugAssignKind::Attr { idx } => { // stack: CONTAINER RESULT @@ -5381,21 +6338,21 @@ impl Compiler { Ok(()) } - fn compile_op(&mut self, op: &Operator, inplace: bool) { + fn compile_op(&mut self, op: &ast::Operator, inplace: bool) { let bin_op = match op { - Operator::Add => BinaryOperator::Add, - Operator::Sub => BinaryOperator::Subtract, - Operator::Mult => BinaryOperator::Multiply, - Operator::MatMult => BinaryOperator::MatrixMultiply, - Operator::Div => BinaryOperator::TrueDivide, - Operator::FloorDiv => BinaryOperator::FloorDivide, - Operator::Mod => BinaryOperator::Remainder, - Operator::Pow => BinaryOperator::Power, - Operator::LShift => BinaryOperator::Lshift, - Operator::RShift => BinaryOperator::Rshift, - Operator::BitOr => BinaryOperator::Or, - Operator::BitXor => BinaryOperator::Xor, - Operator::BitAnd => BinaryOperator::And, + ast::Operator::Add => BinaryOperator::Add, + ast::Operator::Sub => BinaryOperator::Subtract, + ast::Operator::Mult => BinaryOperator::Multiply, + ast::Operator::MatMult => BinaryOperator::MatrixMultiply, + ast::Operator::Div => BinaryOperator::TrueDivide, + ast::Operator::FloorDiv => BinaryOperator::FloorDivide, + ast::Operator::Mod => BinaryOperator::Remainder, + ast::Operator::Pow => BinaryOperator::Power, + ast::Operator::LShift => BinaryOperator::Lshift, + ast::Operator::RShift => BinaryOperator::Rshift, + ast::Operator::BitOr => BinaryOperator::Or, + ast::Operator::BitXor => BinaryOperator::Xor, + ast::Operator::BitAnd => BinaryOperator::And, }; let op = if inplace { bin_op.as_inplace() } else { bin_op }; @@ -5412,15 +6369,15 @@ impl Compiler { /// (indicated by the condition parameter). fn compile_jump_if( &mut self, - expression: &Expr, + expression: &ast::Expr, condition: bool, target_block: BlockIdx, ) -> CompileResult<()> { // Compile expression for test, and jump to label if false match &expression { - Expr::BoolOp(ExprBoolOp { op, values, .. }) => { + ast::Expr::BoolOp(ast::ExprBoolOp { op, values, .. }) => { match op { - BoolOp::And => { + ast::BoolOp::And => { if condition { // If all values are true. let end_block = self.new_block(); @@ -5441,7 +6398,7 @@ impl Compiler { } } } - BoolOp::Or => { + ast::BoolOp::Or => { if condition { // If any of the values is true. for value in values { @@ -5464,8 +6421,8 @@ impl Compiler { } } } - Expr::UnaryOp(ExprUnaryOp { - op: UnaryOp::Not, + ast::Expr::UnaryOp(ast::ExprUnaryOp { + op: ast::UnaryOp::Not, operand, .. }) => { @@ -5496,7 +6453,7 @@ impl Compiler { /// Compile a boolean operation as an expression. /// This means, that the last value remains on the stack. - fn compile_bool_op(&mut self, op: &BoolOp, values: &[Expr]) -> CompileResult<()> { + fn compile_bool_op(&mut self, op: &ast::BoolOp, values: &[ast::Expr]) -> CompileResult<()> { let after_block = self.new_block(); let (last_value, values) = values.split_last().unwrap(); @@ -5504,9 +6461,9 @@ impl Compiler { for value in values { self.compile_expression(value)?; - emit!(self, Instruction::CopyItem { index: 1_u32 }); + emit!(self, Instruction::Copy { index: 1_u32 }); match op { - BoolOp::And => { + ast::BoolOp::And => { emit!( self, Instruction::PopJumpIfFalse { @@ -5514,7 +6471,7 @@ impl Compiler { } ); } - BoolOp::Or => { + ast::BoolOp::Or => { emit!( self, Instruction::PopJumpIfTrue { @@ -5533,7 +6490,7 @@ impl Compiler { Ok(()) } - fn compile_dict(&mut self, items: &[DictItem]) -> CompileResult<()> { + fn compile_dict(&mut self, items: &[ast::DictItem]) -> CompileResult<()> { // FIXME: correct order to build map, etc d = {**a, 'key': 2} should override // 'key' in dict a let mut size = 0; @@ -5577,22 +6534,18 @@ impl Compiler { emit!(self, Instruction::Send { target: exit_block }); // SETUP_FINALLY fail - set up exception handler for YIELD_VALUE - // Stack at this point: [receiver, yielded_value] - // handler_depth = base + 2 (receiver + yielded_value) - let handler_depth = self.handler_stack_depth() + 2; - self.push_fblock_with_handler( + emit!(self, PseudoInstruction::SetupFinally { target: fail_block }); + self.push_fblock( FBlockType::TryExcept, // Use TryExcept for exception handler send_block, exit_block, - Some(fail_block), - handler_depth, - false, // no lasti needed )?; // YIELD_VALUE with arg=1 (yield-from/await mode - not wrapped for async gen) emit!(self, Instruction::YieldValue { arg: 1 }); - // POP_BLOCK (implicit - pop fblock before RESUME) + // POP_BLOCK before RESUME + emit!(self, PseudoInstruction::PopBlock); self.pop_fblock(FBlockType::TryExcept); // RESUME @@ -5607,8 +6560,11 @@ impl Compiler { } ); - // JUMP_NO_INTERRUPT send (regular JUMP in RustPython) - emit!(self, Instruction::Jump { target: send_block }); + // JUMP_BACKWARD_NO_INTERRUPT send + emit!( + self, + PseudoInstruction::JumpNoInterrupt { target: send_block } + ); // fail: CLEANUP_THROW // Stack when exception: [receiver, yielded_value, exc] @@ -5627,18 +6583,19 @@ impl Compiler { Ok(()) } - fn compile_expression(&mut self, expression: &Expr) -> CompileResult<()> { - use ruff_python_ast::*; + fn compile_expression(&mut self, expression: &ast::Expr) -> CompileResult<()> { trace!("Compiling {expression:?}"); let range = expression.range(); self.set_source_range(range); match &expression { - Expr::Call(ExprCall { + ast::Expr::Call(ast::ExprCall { func, arguments, .. }) => self.compile_call(func, arguments)?, - Expr::BoolOp(ExprBoolOp { op, values, .. }) => self.compile_bool_op(op, values)?, - Expr::BinOp(ExprBinOp { + ast::Expr::BoolOp(ast::ExprBoolOp { op, values, .. }) => { + self.compile_bool_op(op, values)? + } + ast::Expr::BinOp(ast::ExprBinOp { left, op, right, .. }) => { self.compile_expression(left)?; @@ -5647,36 +6604,53 @@ impl Compiler { // Perform operation: self.compile_op(op, false); } - Expr::Subscript(ExprSubscript { + ast::Expr::Subscript(ast::ExprSubscript { value, slice, ctx, .. }) => { self.compile_subscript(value, slice, *ctx)?; } - Expr::UnaryOp(ExprUnaryOp { op, operand, .. }) => { + ast::Expr::UnaryOp(ast::ExprUnaryOp { op, operand, .. }) => { self.compile_expression(operand)?; // Perform operation: match op { - UnaryOp::UAdd => emit!( + ast::UnaryOp::UAdd => emit!( self, Instruction::CallIntrinsic1 { func: bytecode::IntrinsicFunction1::UnaryPositive } ), - UnaryOp::USub => emit!(self, Instruction::UnaryNegative), - UnaryOp::Not => { + ast::UnaryOp::USub => emit!(self, Instruction::UnaryNegative), + ast::UnaryOp::Not => { emit!(self, Instruction::ToBool); emit!(self, Instruction::UnaryNot); } - UnaryOp::Invert => emit!(self, Instruction::UnaryInvert), + ast::UnaryOp::Invert => emit!(self, Instruction::UnaryInvert), }; } - Expr::Attribute(ExprAttribute { value, attr, .. }) => { - self.compile_expression(value)?; - let idx = self.name(attr.as_str()); - emit!(self, Instruction::LoadAttr { idx }); + ast::Expr::Attribute(ast::ExprAttribute { value, attr, .. }) => { + // Check for super() attribute access optimization + if let Some(super_type) = self.can_optimize_super_call(value, attr.as_str()) { + // super().attr or super(cls, self).attr optimization + // Stack: [global_super, class, self] → LOAD_SUPER_ATTR → [attr] + self.load_args_for_super(&super_type)?; + let idx = self.name(attr.as_str()); + match super_type { + SuperCallType::TwoArg { .. } => { + self.emit_load_super_attr(idx); + } + SuperCallType::ZeroArg => { + self.emit_load_zero_super_attr(idx); + } + } + } else { + // Normal attribute access + self.compile_expression(value)?; + let idx = self.name(attr.as_str()); + self.emit_load_attr(idx); + } } - Expr::Compare(ExprCompare { + ast::Expr::Compare(ast::ExprCompare { left, ops, comparators, @@ -5684,25 +6658,25 @@ impl Compiler { }) => { self.compile_compare(left, ops, comparators)?; } - // Expr::Constant(ExprConstant { value, .. }) => { + // ast::Expr::Constant(ExprConstant { value, .. }) => { // self.emit_load_const(compile_constant(value)); // } - Expr::List(ExprList { elts, .. }) => { + ast::Expr::List(ast::ExprList { elts, .. }) => { self.starunpack_helper(elts, 0, CollectionType::List)?; } - Expr::Tuple(ExprTuple { elts, .. }) => { + ast::Expr::Tuple(ast::ExprTuple { elts, .. }) => { self.starunpack_helper(elts, 0, CollectionType::Tuple)?; } - Expr::Set(ExprSet { elts, .. }) => { + ast::Expr::Set(ast::ExprSet { elts, .. }) => { self.starunpack_helper(elts, 0, CollectionType::Set)?; } - Expr::Dict(ExprDict { items, .. }) => { + ast::Expr::Dict(ast::ExprDict { items, .. }) => { self.compile_dict(items)?; } - Expr::Slice(ExprSlice { + ast::Expr::Slice(ast::ExprSlice { lower, upper, step, .. }) => { - let mut compile_bound = |bound: Option<&Expr>| match bound { + let mut compile_bound = |bound: Option<&ast::Expr>| match bound { Some(exp) => self.compile_expression(exp), None => { self.emit_load_const(ConstantData::None); @@ -5720,7 +6694,7 @@ impl Compiler { }; emit!(self, Instruction::BuildSlice { argc }); } - Expr::Yield(ExprYield { value, .. }) => { + ast::Expr::Yield(ast::ExprYield { value, .. }) => { if !self.ctx.in_func() { return Err(self.error(CodegenErrorType::InvalidYield)); } @@ -5738,16 +6712,16 @@ impl Compiler { } ); } - Expr::Await(ExprAwait { value, .. }) => { + ast::Expr::Await(ast::ExprAwait { value, .. }) => { if self.ctx.func != FunctionContext::AsyncFunction { return Err(self.error(CodegenErrorType::InvalidAwait)); } self.compile_expression(value)?; - emit!(self, Instruction::GetAwaitable); + emit!(self, Instruction::GetAwaitable { arg: 0 }); self.emit_load_const(ConstantData::None); self.compile_yield_from_sequence(true)?; } - Expr::YieldFrom(ExprYieldFrom { value, .. }) => { + ast::Expr::YieldFrom(ast::ExprYieldFrom { value, .. }) => { match self.ctx.func { FunctionContext::NoFunction => { return Err(self.error(CodegenErrorType::InvalidYieldFrom)); @@ -5763,11 +6737,11 @@ impl Compiler { self.emit_load_const(ConstantData::None); self.compile_yield_from_sequence(false)?; } - Expr::Name(ExprName { id, .. }) => self.load_name(id.as_str())?, - Expr::Lambda(ExprLambda { + ast::Expr::Name(ast::ExprName { id, .. }) => self.load_name(id.as_str())?, + ast::Expr::Lambda(ast::ExprLambda { parameters, body, .. }) => { - let default_params = Parameters::default(); + let default_params = ast::Parameters::default(); let params = parameters.as_deref().unwrap_or(&default_params); validate_duplicate_params(params).map_err(|e| self.error(e))?; @@ -5835,10 +6809,7 @@ impl Compiler { in_async_scope: false, }; - self.current_code_info() - .metadata - .consts - .insert_full(ConstantData::None); + // Lambda cannot have docstrings, so no None is added to co_consts self.compile_expression(body)?; self.emit_return_value(); @@ -5849,14 +6820,17 @@ impl Compiler { self.ctx = prev_ctx; } - Expr::ListComp(ExprListComp { + ast::Expr::ListComp(ast::ExprListComp { elt, generators, .. }) => { self.compile_comprehension( "<listcomp>", - Some(Instruction::BuildList { - size: OpArgMarker::marker(), - }), + Some( + Instruction::BuildList { + size: OpArgMarker::marker(), + } + .into(), + ), generators, &|compiler| { compiler.compile_comprehension_element(elt)?; @@ -5872,14 +6846,17 @@ impl Compiler { Self::contains_await(elt) || Self::generators_contain_await(generators), )?; } - Expr::SetComp(ExprSetComp { + ast::Expr::SetComp(ast::ExprSetComp { elt, generators, .. }) => { self.compile_comprehension( "<setcomp>", - Some(Instruction::BuildSet { - size: OpArgMarker::marker(), - }), + Some( + Instruction::BuildSet { + size: OpArgMarker::marker(), + } + .into(), + ), generators, &|compiler| { compiler.compile_comprehension_element(elt)?; @@ -5895,7 +6872,7 @@ impl Compiler { Self::contains_await(elt) || Self::generators_contain_await(generators), )?; } - Expr::DictComp(ExprDictComp { + ast::Expr::DictComp(ast::ExprDictComp { key, value, generators, @@ -5903,9 +6880,12 @@ impl Compiler { }) => { self.compile_comprehension( "<dictcomp>", - Some(Instruction::BuildMap { - size: OpArgMarker::marker(), - }), + Some( + Instruction::BuildMap { + size: OpArgMarker::marker(), + } + .into(), + ), generators, &|compiler| { // changed evaluation order for Py38 named expression PEP 572 @@ -5927,7 +6907,7 @@ impl Compiler { || Self::generators_contain_await(generators), )?; } - Expr::Generator(ExprGenerator { + ast::Expr::Generator(ast::ExprGenerator { elt, generators, .. }) => { // Check if element or generators contain async content @@ -5961,7 +6941,7 @@ impl Compiler { element_contains_await, )?; } - Expr::Starred(ExprStarred { value, .. }) => { + ast::Expr::Starred(ast::ExprStarred { value, .. }) => { if self.in_annotation { // In annotation context, starred expressions are allowed (PEP 646) // For now, just compile the inner value without wrapping with Unpack @@ -5971,7 +6951,7 @@ impl Compiler { return Err(self.error(CodegenErrorType::InvalidStarExpr)); } } - Expr::If(ExprIf { + ast::Expr::If(ast::ExprIf { test, body, orelse, .. }) => { let else_block = self.new_block(); @@ -5982,7 +6962,7 @@ impl Compiler { self.compile_expression(body)?; emit!( self, - Instruction::Jump { + PseudoInstruction::Jump { target: after_block, } ); @@ -5995,23 +6975,23 @@ impl Compiler { self.switch_to_block(after_block); } - Expr::Named(ExprNamed { + ast::Expr::Named(ast::ExprNamed { target, value, node_index: _, range: _, }) => { self.compile_expression(value)?; - emit!(self, Instruction::CopyItem { index: 1_u32 }); + emit!(self, Instruction::Copy { index: 1_u32 }); self.compile_store(target)?; } - Expr::FString(fstring) => { + ast::Expr::FString(fstring) => { self.compile_expr_fstring(fstring)?; } - Expr::TString(_) => { - return Err(self.error(CodegenErrorType::NotImplementedYet)); + ast::Expr::TString(tstring) => { + self.compile_expr_tstring(tstring)?; } - Expr::StringLiteral(string) => { + ast::Expr::StringLiteral(string) => { let value = string.value.to_str(); if value.contains(char::REPLACEMENT_CHARACTER) { let value = string @@ -6030,42 +7010,42 @@ impl Compiler { }); } } - Expr::BytesLiteral(bytes) => { + ast::Expr::BytesLiteral(bytes) => { let iter = bytes.value.iter().flat_map(|x| x.iter().copied()); let v: Vec<u8> = iter.collect(); self.emit_load_const(ConstantData::Bytes { value: v }); } - Expr::NumberLiteral(number) => match &number.value { - Number::Int(int) => { + ast::Expr::NumberLiteral(number) => match &number.value { + ast::Number::Int(int) => { let value = ruff_int_to_bigint(int).map_err(|e| self.error(e))?; self.emit_load_const(ConstantData::Integer { value }); } - Number::Float(float) => { + ast::Number::Float(float) => { self.emit_load_const(ConstantData::Float { value: *float }); } - Number::Complex { real, imag } => { + ast::Number::Complex { real, imag } => { self.emit_load_const(ConstantData::Complex { value: Complex::new(*real, *imag), }); } }, - Expr::BooleanLiteral(b) => { + ast::Expr::BooleanLiteral(b) => { self.emit_load_const(ConstantData::Boolean { value: b.value }); } - Expr::NoneLiteral(_) => { + ast::Expr::NoneLiteral(_) => { self.emit_load_const(ConstantData::None); } - Expr::EllipsisLiteral(_) => { + ast::Expr::EllipsisLiteral(_) => { self.emit_load_const(ConstantData::Ellipsis); } - Expr::IpyEscapeCommand(_) => { + ast::Expr::IpyEscapeCommand(_) => { panic!("unexpected ipy escape command"); } } Ok(()) } - fn compile_keywords(&mut self, keywords: &[Keyword]) -> CompileResult<()> { + fn compile_keywords(&mut self, keywords: &[ast::Keyword]) -> CompileResult<()> { let mut size = 0; let groupby = keywords.iter().chunk_by(|e| e.arg.is_none()); for (is_unpacking, sub_keywords) in &groupby { @@ -6090,153 +7070,227 @@ impl Compiler { } } if size > 1 { - emit!(self, Instruction::BuildMapForCall { size }); + // Merge all dicts: first dict is accumulator, merge rest into it + for _ in 1..size { + emit!(self, Instruction::DictMerge { index: 1 }); + } } Ok(()) } - fn compile_call(&mut self, func: &Expr, args: &Arguments) -> CompileResult<()> { - let method = if let Expr::Attribute(ExprAttribute { value, attr, .. }) = &func { - self.compile_expression(value)?; - let idx = self.name(attr.as_str()); - emit!(self, Instruction::LoadMethod { idx }); - true + fn compile_call(&mut self, func: &ast::Expr, args: &ast::Arguments) -> CompileResult<()> { + // Method call: obj → LOAD_ATTR_METHOD → [method, self_or_null] → args → CALL + // Regular call: func → PUSH_NULL → args → CALL + if let ast::Expr::Attribute(ast::ExprAttribute { value, attr, .. }) = &func { + // Check for super() method call optimization + if let Some(super_type) = self.can_optimize_super_call(value, attr.as_str()) { + // super().method() or super(cls, self).method() optimization + // Stack: [global_super, class, self] → LOAD_SUPER_METHOD → [method, self] + self.load_args_for_super(&super_type)?; + let idx = self.name(attr.as_str()); + match super_type { + SuperCallType::TwoArg { .. } => { + self.emit_load_super_method(idx); + } + SuperCallType::ZeroArg => { + self.emit_load_zero_super_method(idx); + } + } + self.codegen_call_helper(0, args)?; + } else { + // Normal method call: compile object, then LOAD_ATTR with method flag + // LOAD_ATTR(method=1) pushes [method, self_or_null] on stack + self.compile_expression(value)?; + let idx = self.name(attr.as_str()); + self.emit_load_attr_method(idx); + self.codegen_call_helper(0, args)?; + } } else { + // Regular call: push func, then NULL for self_or_null slot + // Stack layout: [func, NULL, args...] - same as method call [func, self, args...] self.compile_expression(func)?; - false - }; - let call = self.compile_call_inner(0, args)?; - if method { - self.compile_method_call(call) - } else { - self.compile_normal_call(call) + emit!(self, Instruction::PushNull); + self.codegen_call_helper(0, args)?; } Ok(()) } - fn compile_normal_call(&mut self, ty: CallType) { - match ty { - CallType::Positional { nargs } => { - emit!(self, Instruction::CallFunctionPositional { nargs }) - } - CallType::Keyword { nargs } => emit!(self, Instruction::CallFunctionKeyword { nargs }), - CallType::Ex { has_kwargs } => emit!(self, Instruction::CallFunctionEx { has_kwargs }), + /// Compile subkwargs: emit key-value pairs for BUILD_MAP + fn codegen_subkwargs( + &mut self, + keywords: &[ast::Keyword], + begin: usize, + end: usize, + ) -> CompileResult<()> { + let n = end - begin; + assert!(n > 0); + + // For large kwargs, use BUILD_MAP(0) + MAP_ADD to avoid stack overflow + let big = n * 2 > 8; // STACK_USE_GUIDELINE approximation + + if big { + emit!(self, Instruction::BuildMap { size: 0 }); } - } - fn compile_method_call(&mut self, ty: CallType) { - match ty { - CallType::Positional { nargs } => { - emit!(self, Instruction::CallMethodPositional { nargs }) + + for kw in &keywords[begin..end] { + // Key first, then value - this is critical! + self.emit_load_const(ConstantData::Str { + value: kw.arg.as_ref().unwrap().as_str().into(), + }); + self.compile_expression(&kw.value)?; + + if big { + emit!(self, Instruction::MapAdd { i: 0 }); } - CallType::Keyword { nargs } => emit!(self, Instruction::CallMethodKeyword { nargs }), - CallType::Ex { has_kwargs } => emit!(self, Instruction::CallMethodEx { has_kwargs }), } + + if !big { + emit!(self, Instruction::BuildMap { size: n.to_u32() }); + } + + Ok(()) } - fn compile_call_inner( + /// Compile call arguments and emit the appropriate CALL instruction. + fn codegen_call_helper( &mut self, additional_positional: u32, - arguments: &Arguments, - ) -> CompileResult<CallType> { - let count = u32::try_from(arguments.len()).unwrap() + additional_positional; + arguments: &ast::Arguments, + ) -> CompileResult<()> { + let nelts = arguments.args.len(); + let nkwelts = arguments.keywords.len(); - // Normal arguments: - let (size, unpack) = self.gather_elements(additional_positional, &arguments.args)?; + // Check if we have starred args or **kwargs + let has_starred = arguments + .args + .iter() + .any(|arg| matches!(arg, ast::Expr::Starred(_))); let has_double_star = arguments.keywords.iter().any(|k| k.arg.is_none()); - for keyword in &arguments.keywords { - if let Some(name) = &keyword.arg { - self.check_forbidden_name(name.as_str(), NameUsage::Store)?; - } - } + // Check if exceeds stack guideline + let too_big = nelts + nkwelts * 2 > 8; - let call = if unpack || has_double_star { - // Create a tuple with positional args: - if unpack { - emit!(self, Instruction::BuildTupleFromTuples { size }); - } else { - emit!(self, Instruction::BuildTuple { size }); + if !has_starred && !has_double_star && !too_big { + // Simple call path: no * or ** args + for arg in &arguments.args { + self.compile_expression(arg)?; } - // Create an optional map with kw-args: - let has_kwargs = !arguments.keywords.is_empty(); - if has_kwargs { - self.compile_keywords(&arguments.keywords)?; - } - CallType::Ex { has_kwargs } - } else if !arguments.keywords.is_empty() { - let mut kwarg_names = vec![]; - for keyword in &arguments.keywords { - if let Some(name) = &keyword.arg { + if nkwelts > 0 { + // Compile keyword values and build kwnames tuple + let mut kwarg_names = Vec::with_capacity(nkwelts); + for keyword in &arguments.keywords { kwarg_names.push(ConstantData::Str { - value: name.as_str().into(), + value: keyword.arg.as_ref().unwrap().as_str().into(), }); - } else { - // This means **kwargs! - panic!("name must be set"); + self.compile_expression(&keyword.value)?; } - self.compile_expression(&keyword.value)?; - } - self.emit_load_const(ConstantData::Tuple { - elements: kwarg_names, - }); - CallType::Keyword { nargs: count } + self.emit_load_const(ConstantData::Tuple { + elements: kwarg_names, + }); + + let nargs = additional_positional + nelts.to_u32() + nkwelts.to_u32(); + emit!(self, Instruction::CallKw { nargs }); + } else { + let nargs = additional_positional + nelts.to_u32(); + emit!(self, Instruction::Call { nargs }); + } } else { - CallType::Positional { nargs: count } - }; + // ex_call path: has * or ** args - Ok(call) - } + // Compile positional arguments + if additional_positional == 0 + && nelts == 1 + && matches!(arguments.args[0], ast::Expr::Starred(_)) + { + // Special case: single starred arg + // Even in this case, we need to ensure it's a tuple for CallFunctionEx + if let ast::Expr::Starred(ast::ExprStarred { value, .. }) = &arguments.args[0] { + // Check if the value is already a tuple expression + if matches!(value.as_ref(), ast::Expr::Tuple(_)) { + // Tuple literals can be used directly + self.compile_expression(value)?; + } else { + // For all other cases (including variables that might be lists), + // build a list and convert to tuple to ensure correct type + emit!(self, Instruction::BuildList { size: 0 }); + self.compile_expression(value)?; + emit!(self, Instruction::ListExtend { i: 0 }); + emit!( + self, + Instruction::CallIntrinsic1 { + func: IntrinsicFunction1::ListToTuple + } + ); + } + } + } else { + // Use starunpack_helper to build a list, then convert to tuple + self.starunpack_helper( + &arguments.args, + additional_positional, + CollectionType::List, + )?; + emit!( + self, + Instruction::CallIntrinsic1 { + func: IntrinsicFunction1::ListToTuple + } + ); + } - // Given a vector of expr / star expr generate code which gives either - // a list of expressions on the stack, or a list of tuples. - fn gather_elements(&mut self, before: u32, elements: &[Expr]) -> CompileResult<(u32, bool)> { - // First determine if we have starred elements: - let has_stars = elements.iter().any(|e| matches!(e, Expr::Starred(_))); + // Compile keyword arguments + if nkwelts > 0 { + let mut have_dict = false; + let mut nseen = 0usize; - let size = if has_stars { - let mut size = 0; - let mut iter = elements.iter().peekable(); - let mut run_size = before; + for (i, keyword) in arguments.keywords.iter().enumerate() { + if keyword.arg.is_none() { + // **kwargs unpacking + if nseen > 0 { + // Pack up preceding keywords using codegen_subkwargs + self.codegen_subkwargs(&arguments.keywords, i - nseen, i)?; + if have_dict { + emit!(self, Instruction::DictMerge { index: 1 }); + } + have_dict = true; + nseen = 0; + } - loop { - if iter.peek().is_none_or(|e| matches!(e, Expr::Starred(_))) { - emit!(self, Instruction::BuildTuple { size: run_size }); - run_size = 0; - size += 1; - } + if !have_dict { + emit!(self, Instruction::BuildMap { size: 0 }); + have_dict = true; + } - match iter.next() { - Some(Expr::Starred(ExprStarred { value, .. })) => { - self.compile_expression(value)?; - // We need to collect each unpacked element into a - // tuple, since any side-effects during the conversion - // should be made visible before evaluating remaining - // expressions. - emit!(self, Instruction::BuildTupleFromIter); - size += 1; + self.compile_expression(&keyword.value)?; + emit!(self, Instruction::DictMerge { index: 1 }); + } else { + nseen += 1; } - Some(element) => { - self.compile_expression(element)?; - run_size += 1; + } + + // Pack up any trailing keyword arguments + if nseen > 0 { + self.codegen_subkwargs(&arguments.keywords, nkwelts - nseen, nkwelts)?; + if have_dict { + emit!(self, Instruction::DictMerge { index: 1 }); } - None => break, + have_dict = true; } - } - size - } else { - for element in elements { - self.compile_expression(element)?; + assert!(have_dict); + } else { + emit!(self, Instruction::PushNull); } - before + elements.len().to_u32() - }; - Ok((size, has_stars)) + emit!(self, Instruction::CallFunctionEx); + } + + Ok(()) } - fn compile_comprehension_element(&mut self, element: &Expr) -> CompileResult<()> { + fn compile_comprehension_element(&mut self, element: &ast::Expr) -> CompileResult<()> { self.compile_expression(element).map_err(|e| { if let CodegenErrorType::InvalidStarExpr = e.error { self.error(CodegenErrorType::SyntaxError( @@ -6251,8 +7305,8 @@ impl Compiler { fn compile_comprehension( &mut self, name: &str, - init_collection: Option<Instruction>, - generators: &[Comprehension], + init_collection: Option<AnyInstruction>, + generators: &[ast::Comprehension], compile_element: &dyn Fn(&mut Self) -> CompileResult<()>, comprehension_type: ComprehensionType, element_contains_await: bool, @@ -6314,15 +7368,15 @@ impl Compiler { in_async_scope: prev_ctx.in_async_scope || is_async, }; - let flags = bytecode::CodeFlags::NEW_LOCALS | bytecode::CodeFlags::IS_OPTIMIZED; + let flags = bytecode::CodeFlags::NEWLOCALS | bytecode::CodeFlags::OPTIMIZED; let flags = if is_async { - flags | bytecode::CodeFlags::IS_COROUTINE + flags | bytecode::CodeFlags::COROUTINE } else { flags }; // Create magnificent function <listcomp>: - self.push_output(flags, 1, 1, 0, name.to_owned()); + self.push_output(flags, 1, 1, 0, name.to_owned())?; // Mark that we're in an inlined comprehension self.current_code_info().in_inlined_comp = true; @@ -6361,23 +7415,25 @@ impl Compiler { loop_labels.push((loop_block, after_block, generator.is_async)); self.switch_to_block(loop_block); if generator.is_async { + emit!( + self, + PseudoInstruction::SetupFinally { + target: after_block + } + ); emit!(self, Instruction::GetANext); - - let current_depth = (init_collection.is_some() as u32) - + u32::try_from(loop_labels.len()).unwrap() - + 1; - self.push_fblock_with_handler( + self.push_fblock( FBlockType::AsyncComprehensionGenerator, loop_block, after_block, - Some(after_block), - current_depth, - false, )?; self.emit_load_const(ConstantData::None); self.compile_yield_from_sequence(true)?; - self.compile_store(&generator.target)?; + // POP_BLOCK before store: only __anext__/yield_from are + // protected by SetupFinally targeting END_ASYNC_FOR. + emit!(self, PseudoInstruction::PopBlock); self.pop_fblock(FBlockType::AsyncComprehensionGenerator); + self.compile_store(&generator.target)?; } else { emit!( self, @@ -6397,12 +7453,17 @@ impl Compiler { compile_element(self)?; for (loop_block, after_block, is_async) in loop_labels.iter().rev().copied() { - emit!(self, Instruction::Jump { target: loop_block }); + emit!(self, PseudoInstruction::Jump { target: loop_block }); self.switch_to_block(after_block); if is_async { + // EndAsyncFor pops both the exception and the aiter + // (handler depth is before GetANext, so aiter is at handler depth) emit!(self, Instruction::EndAsyncFor); - emit!(self, Instruction::PopTop); + } else { + // END_FOR + POP_ITER pattern (CPython 3.14) + emit!(self, Instruction::EndFor); + emit!(self, Instruction::PopIter); } } @@ -6418,6 +7479,7 @@ impl Compiler { // Create comprehension function with closure self.make_closure(code, bytecode::MakeFunctionFlags::empty())?; + emit!(self, Instruction::PushNull); // Evaluate iterated item: self.compile_expression(&generators[0].iter)?; @@ -6431,9 +7493,9 @@ impl Compiler { }; // Call just created <listcomp> function: - emit!(self, Instruction::CallFunctionPositional { nargs: 1 }); + emit!(self, Instruction::Call { nargs: 1 }); if is_async_list_set_dict_comprehension { - emit!(self, Instruction::GetAwaitable); + emit!(self, Instruction::GetAwaitable { arg: 0 }); self.emit_load_const(ConstantData::None); self.compile_yield_from_sequence(true)?; } @@ -6442,25 +7504,25 @@ impl Compiler { } /// Collect variable names from an assignment target expression - fn collect_target_names(&self, target: &Expr, names: &mut Vec<String>) { + fn collect_target_names(&self, target: &ast::Expr, names: &mut Vec<String>) { match target { - Expr::Name(name) => { + ast::Expr::Name(name) => { let name_str = name.id.to_string(); if !names.contains(&name_str) { names.push(name_str); } } - Expr::Tuple(tuple) => { + ast::Expr::Tuple(tuple) => { for elt in &tuple.elts { self.collect_target_names(elt, names); } } - Expr::List(list) => { + ast::Expr::List(list) => { for elt in &list.elts { self.collect_target_names(elt, names); } } - Expr::Starred(starred) => { + ast::Expr::Starred(starred) => { self.collect_target_names(&starred.value, names); } _ => { @@ -6473,8 +7535,8 @@ impl Compiler { /// This generates bytecode inline without creating a new code object fn compile_inlined_comprehension( &mut self, - init_collection: Option<Instruction>, - generators: &[Comprehension], + init_collection: Option<AnyInstruction>, + generators: &[ast::Comprehension], compile_element: &dyn Fn(&mut Self) -> CompileResult<()>, _has_an_async_gen: bool, ) -> CompileResult<()> { @@ -6533,20 +7595,13 @@ impl Compiler { let end_block = self.new_block(); if !pushed_locals.is_empty() { - // Calculate stack depth for exception handler - // Stack: [saved_locals..., collection?, iterator] - let depth = self.handler_stack_depth() - + u32::try_from(pushed_locals.len()).unwrap() - + init_collection.is_some() as u32 - + 1; - self.push_fblock_with_handler( - FBlockType::TryExcept, - cleanup_block, - end_block, - Some(cleanup_block), - depth, - false, - )?; + emit!( + self, + PseudoInstruction::SetupFinally { + target: cleanup_block + } + ); + self.push_fblock(FBlockType::TryExcept, cleanup_block, end_block)?; } // Step 5: Compile the comprehension loop(s) @@ -6594,21 +7649,26 @@ impl Compiler { // Step 7: Close all loops for (loop_block, after_block, is_async) in loop_labels.iter().rev().copied() { - emit!(self, Instruction::Jump { target: loop_block }); + emit!(self, PseudoInstruction::Jump { target: loop_block }); self.switch_to_block(after_block); if is_async { emit!(self, Instruction::EndAsyncFor); + // Pop the iterator + emit!(self, Instruction::PopTop); + } else { + // END_FOR + POP_ITER pattern (CPython 3.14) + emit!(self, Instruction::EndFor); + emit!(self, Instruction::PopIter); } - // Pop the iterator - emit!(self, Instruction::PopTop); } // Step 8: Clean up - restore saved locals if !pushed_locals.is_empty() { + emit!(self, PseudoInstruction::PopBlock); self.pop_fblock(FBlockType::TryExcept); // Normal path: jump past cleanup - emit!(self, Instruction::Jump { target: end_block }); + emit!(self, PseudoInstruction::Jump { target: end_block }); // Exception cleanup path self.switch_to_block(cleanup_block); @@ -6631,7 +7691,7 @@ impl Compiler { // Re-raise the exception emit!( self, - Instruction::Raise { + Instruction::RaiseVarargs { kind: bytecode::RaiseKind::ReraiseFromStack } ); @@ -6659,7 +7719,7 @@ impl Compiler { Ok(()) } - fn compile_future_features(&mut self, features: &[Alias]) -> Result<(), CodegenError> { + fn compile_future_features(&mut self, features: &[ast::Alias]) -> Result<(), CodegenError> { if let DoneWithFuture::Yes = self.done_with_future_stmts { return Err(self.error(CodegenErrorType::InvalidFuturePlacement)); } @@ -6681,30 +7741,31 @@ impl Compiler { } // Low level helper functions: - fn _emit(&mut self, instr: Instruction, arg: OpArg, target: BlockIdx) { + fn _emit<I: Into<AnyInstruction>>(&mut self, instr: I, arg: OpArg, target: BlockIdx) { let range = self.current_source_range; let source = self.source_file.to_source_code(); let location = source.source_location(range.start(), PositionEncoding::Utf8); let end_location = source.source_location(range.end(), PositionEncoding::Utf8); - let except_handler = self.current_except_handler(); + let except_handler = None; self.current_block().instructions.push(ir::InstructionInfo { - instr, + instr: instr.into(), arg, target, location, end_location, except_handler, + lineno_override: None, }); } - fn emit_no_arg(&mut self, ins: Instruction) { + fn emit_no_arg<I: Into<AnyInstruction>>(&mut self, ins: I) { self._emit(ins, OpArg::null(), BlockIdx::NULL) } - fn emit_arg<A: OpArgType, T: EmitArg<A>>( + fn emit_arg<A: OpArgType, T: EmitArg<A>, I: Into<AnyInstruction>>( &mut self, arg: T, - f: impl FnOnce(OpArgMarker<A>) -> Instruction, + f: impl FnOnce(OpArgMarker<A>) -> I, ) { let (op, arg, target) = arg.emit(f); self._emit(op, arg, target) @@ -6723,17 +7784,53 @@ impl Compiler { } fn emit_return_const(&mut self, constant: ConstantData) { - let idx = self.arg_constant(constant); - self.emit_arg(idx, |idx| Instruction::ReturnConst { idx }) + self.emit_load_const(constant); + emit!(self, Instruction::ReturnValue) + } + + /// Emit LOAD_ATTR for attribute access (method=false). + /// Encodes: (name_idx << 1) | 0 + fn emit_load_attr(&mut self, name_idx: u32) { + let encoded = encode_load_attr_arg(name_idx, false); + self.emit_arg(encoded, |arg| Instruction::LoadAttr { idx: arg }) + } + + /// Emit LOAD_ATTR with method flag set (for method calls). + /// Encodes: (name_idx << 1) | 1 + fn emit_load_attr_method(&mut self, name_idx: u32) { + let encoded = encode_load_attr_arg(name_idx, true); + self.emit_arg(encoded, |arg| Instruction::LoadAttr { idx: arg }) + } + + /// Emit LOAD_SUPER_ATTR for 2-arg super().attr access. + /// Encodes: (name_idx << 2) | 0b10 (method=0, class=1) + fn emit_load_super_attr(&mut self, name_idx: u32) { + let encoded = encode_load_super_attr_arg(name_idx, false, true); + self.emit_arg(encoded, |arg| Instruction::LoadSuperAttr { arg }) + } + + /// Emit LOAD_SUPER_ATTR for 2-arg super().method() call. + /// Encodes: (name_idx << 2) | 0b11 (method=1, class=1) + fn emit_load_super_method(&mut self, name_idx: u32) { + let encoded = encode_load_super_attr_arg(name_idx, true, true); + self.emit_arg(encoded, |arg| Instruction::LoadSuperAttr { arg }) + } + + /// Emit LOAD_SUPER_ATTR for 0-arg super().attr access. + /// Encodes: (name_idx << 2) | 0b00 (method=0, class=0) + fn emit_load_zero_super_attr(&mut self, name_idx: u32) { + let encoded = encode_load_super_attr_arg(name_idx, false, false); + self.emit_arg(encoded, |arg| Instruction::LoadSuperAttr { arg }) + } + + /// Emit LOAD_SUPER_ATTR for 0-arg super().method() call. + /// Encodes: (name_idx << 2) | 0b01 (method=1, class=0) + fn emit_load_zero_super_method(&mut self, name_idx: u32) { + let encoded = encode_load_super_attr_arg(name_idx, true, false); + self.emit_arg(encoded, |arg| Instruction::LoadSuperAttr { arg }) } fn emit_return_value(&mut self) { - if let Some(inst) = self.current_block().instructions.last_mut() - && let Instruction::LoadConst { idx } = inst.instr - { - inst.instr = Instruction::ReturnConst { idx }; - return; - } emit!(self, Instruction::ReturnValue) } @@ -6741,6 +7838,19 @@ impl Compiler { self.code_stack.last_mut().expect("no code on stack") } + /// Enter a conditional block (if/for/while/match/try/with) + /// PEP 649: Track conditional annotation context + fn enter_conditional_block(&mut self) { + self.current_code_info().in_conditional_block += 1; + } + + /// Leave a conditional block + fn leave_conditional_block(&mut self) { + let code_info = self.current_code_info(); + debug_assert!(code_info.in_conditional_block > 0); + code_info.in_conditional_block -= 1; + } + /// Compile break or continue statement with proper fblock cleanup. /// compiler_break, compiler_continue /// This handles unwinding through With blocks and exception handlers. @@ -6797,7 +7907,10 @@ impl Compiler { With { is_async: bool, }, - HandlerCleanup, + HandlerCleanup { + name: Option<String>, + }, + TryExcept, FinallyTry { body: Vec<ruff_python_ast::Stmt>, fblock_idx: usize, @@ -6818,7 +7931,14 @@ impl Compiler { unwind_actions.push(UnwindAction::With { is_async: true }); } FBlockType::HandlerCleanup => { - unwind_actions.push(UnwindAction::HandlerCleanup); + let name = match &code.fblock[i].fb_datum { + FBlockDatum::ExceptionName(name) => Some(name.clone()), + _ => None, + }; + unwind_actions.push(UnwindAction::HandlerCleanup { name }); + } + FBlockType::TryExcept => { + unwind_actions.push(UnwindAction::TryExcept); } FBlockType::FinallyTry => { // Need to execute finally body before break/continue @@ -6846,24 +7966,46 @@ impl Compiler { for action in unwind_actions { match action { UnwindAction::With { is_async } => { + // codegen_unwind_fblock(WITH/ASYNC_WITH) + emit!(self, PseudoInstruction::PopBlock); // compiler_call_exit_with_nones + emit!(self, Instruction::PushNull); self.emit_load_const(ConstantData::None); self.emit_load_const(ConstantData::None); self.emit_load_const(ConstantData::None); - emit!(self, Instruction::CallFunctionPositional { nargs: 3 }); + emit!(self, Instruction::Call { nargs: 3 }); if is_async { - emit!(self, Instruction::GetAwaitable); + emit!(self, Instruction::GetAwaitable { arg: 2 }); self.emit_load_const(ConstantData::None); self.compile_yield_from_sequence(true)?; } emit!(self, Instruction::PopTop); } - UnwindAction::HandlerCleanup => { - emit!(self, Instruction::PopException); + UnwindAction::HandlerCleanup { ref name } => { + // codegen_unwind_fblock(HANDLER_CLEANUP) + if name.is_some() { + // Named handler: PopBlock for inner SETUP_CLEANUP + emit!(self, PseudoInstruction::PopBlock); + } + // PopBlock for outer SETUP_CLEANUP (ExceptionHandler) + emit!(self, PseudoInstruction::PopBlock); + emit!(self, Instruction::PopExcept); + if let Some(name) = name { + self.emit_load_const(ConstantData::None); + self.store_name(name)?; + self.compile_name(name, NameUsage::Delete)?; + } + } + UnwindAction::TryExcept => { + // codegen_unwind_fblock(TRY_EXCEPT) + emit!(self, PseudoInstruction::PopBlock); } UnwindAction::FinallyTry { body, fblock_idx } => { + // codegen_unwind_fblock(FINALLY_TRY) + emit!(self, PseudoInstruction::PopBlock); + // compile finally body inline // Temporarily pop the FinallyTry fblock so nested break/continue // in the finally body won't see it again. @@ -6878,11 +8020,10 @@ impl Compiler { code.fblock.insert(fblock_idx, saved_fblock); } UnwindAction::FinallyEnd => { - // Stack when in FinallyEnd: [..., prev_exc, exc] - // Note: No lasti here - it's only pushed for cleanup handler exceptions - // We need to pop: exc, prev_exc (via PopException) - emit!(self, Instruction::PopTop); // exc - emit!(self, Instruction::PopException); // prev_exc is restored + // codegen_unwind_fblock(FINALLY_END) + emit!(self, Instruction::PopTop); // exc_value + emit!(self, PseudoInstruction::PopBlock); + emit!(self, Instruction::PopExcept); } UnwindAction::PopValue => { // Pop the return value - continue/break cancels the pending return @@ -6893,52 +8034,16 @@ impl Compiler { // For break in a for loop, pop the iterator if is_break && is_for_loop { - emit!(self, Instruction::PopTop); + emit!(self, Instruction::PopIter); } // Jump to target - if is_break { - emit!(self, Instruction::Break { target: exit_block }); - } else { - emit!(self, Instruction::Continue { target: loop_block }); - } + let target = if is_break { exit_block } else { loop_block }; + emit!(self, PseudoInstruction::Jump { target }); Ok(()) } - /// Calculate the current exception handler stack depth. - /// CPython calculates this based on the SETUP_FINALLY/SETUP_CLEANUP stack depth. - fn handler_stack_depth(&self) -> u32 { - let code = match self.code_stack.last() { - Some(c) => c, - None => return 0, - }; - let mut depth = 0u32; - for fblock in &code.fblock { - match fblock.fb_type { - FBlockType::ForLoop => depth += 1, - FBlockType::With | FBlockType::AsyncWith => depth += 1, - // HandlerCleanup does NOT add to stack depth - it only tracks - // cleanup code for named exception handlers. The stack item - // (prev_exc) is already counted by ExceptionHandler. - // FBlockType::HandlerCleanup => depth += 1, - // inside exception handler, prev_exc is on stack - FBlockType::ExceptionHandler => depth += 1, - // ExceptionGroupHandler: inside except* handler path - // Stack has [prev_exc, orig, list, rest] - add 4 for these - FBlockType::ExceptionGroupHandler => depth += 4, - // FinallyEnd: inside finally exception path - // Stack has [prev_exc, exc] - add 2 for these (no lasti at this level) - FBlockType::FinallyEnd => depth += 2, - // PopValue: preserving a return value on stack during inline finally - // The return value adds 1 to the stack depth - FBlockType::PopValue => depth += 1, - _ => {} - } - } - depth - } - fn current_block(&mut self) -> &mut ir::Block { let info = self.current_code_info(); &mut info.blocks[info.current_block] @@ -6981,7 +8086,7 @@ impl Compiler { } fn mark_generator(&mut self) { - self.current_code_info().flags |= bytecode::CodeFlags::IS_GENERATOR + self.current_code_info().flags |= bytecode::CodeFlags::GENERATOR } /// Whether the expression contains an await expression and @@ -6993,26 +8098,28 @@ impl Compiler { /// async for: ... /// ``` /// are statements, so we won't check for them here - fn contains_await(expression: &Expr) -> bool { + fn contains_await(expression: &ast::Expr) -> bool { + use ast::visitor::Visitor; + #[derive(Default)] struct AwaitVisitor { found: bool, } - impl Visitor<'_> for AwaitVisitor { - fn visit_expr(&mut self, expr: &Expr) { + impl ast::visitor::Visitor<'_> for AwaitVisitor { + fn visit_expr(&mut self, expr: &ast::Expr) { if self.found { return; } match expr { - Expr::Await(_) => self.found = true, + ast::Expr::Await(_) => self.found = true, // Note: We do NOT check for async comprehensions here. // Async list/set/dict comprehensions are handled by compile_comprehension // which already awaits the result. A generator expression containing // an async comprehension as its element does NOT become an async generator, // because the async comprehension is awaited when evaluating the element. - _ => walk_expr(self, expr), + _ => ast::visitor::walk_expr(self, expr), } } } @@ -7024,7 +8131,7 @@ impl Compiler { /// Check if any of the generators (except the first one's iter) contains an await expression. /// The first generator's iter is evaluated outside the comprehension scope. - fn generators_contain_await(generators: &[Comprehension]) -> bool { + fn generators_contain_await(generators: &[ast::Comprehension]) -> bool { for (i, generator) in generators.iter().enumerate() { // First generator's iter is evaluated outside the comprehension if i > 0 && Self::contains_await(&generator.iter) { @@ -7040,7 +8147,7 @@ impl Compiler { false } - fn compile_expr_fstring(&mut self, fstring: &ExprFString) -> CompileResult<()> { + fn compile_expr_fstring(&mut self, fstring: &ast::ExprFString) -> CompileResult<()> { let fstring = &fstring.value; for part in fstring { self.compile_fstring_part(part)?; @@ -7057,9 +8164,9 @@ impl Compiler { Ok(()) } - fn compile_fstring_part(&mut self, part: &FStringPart) -> CompileResult<()> { + fn compile_fstring_part(&mut self, part: &ast::FStringPart) -> CompileResult<()> { match part { - FStringPart::Literal(string) => { + ast::FStringPart::Literal(string) => { if string.value.contains(char::REPLACEMENT_CHARACTER) { // might have a surrogate literal; should reparse to be sure let source = self.source_file.slice(string.range); @@ -7075,24 +8182,24 @@ impl Compiler { } Ok(()) } - FStringPart::FString(fstring) => self.compile_fstring(fstring), + ast::FStringPart::FString(fstring) => self.compile_fstring(fstring), } } - fn compile_fstring(&mut self, fstring: &FString) -> CompileResult<()> { + fn compile_fstring(&mut self, fstring: &ast::FString) -> CompileResult<()> { self.compile_fstring_elements(fstring.flags, &fstring.elements) } fn compile_fstring_elements( &mut self, - flags: FStringFlags, - fstring_elements: &InterpolatedStringElements, + flags: ast::FStringFlags, + fstring_elements: &ast::InterpolatedStringElements, ) -> CompileResult<()> { let mut element_count = 0; for element in fstring_elements { element_count += 1; match element { - InterpolatedStringElement::Literal(string) => { + ast::InterpolatedStringElement::Literal(string) => { if string.value.contains(char::REPLACEMENT_CHARACTER) { // might have a surrogate literal; should reparse to be sure let source = self.source_file.slice(string.range); @@ -7109,15 +8216,15 @@ impl Compiler { }); } } - InterpolatedStringElement::Interpolation(fstring_expr) => { + ast::InterpolatedStringElement::Interpolation(fstring_expr) => { let mut conversion = match fstring_expr.conversion { - ConversionFlag::None => ConvertValueOparg::None, - ConversionFlag::Str => ConvertValueOparg::Str, - ConversionFlag::Repr => ConvertValueOparg::Repr, - ConversionFlag::Ascii => ConvertValueOparg::Ascii, + ast::ConversionFlag::None => ConvertValueOparg::None, + ast::ConversionFlag::Str => ConvertValueOparg::Str, + ast::ConversionFlag::Repr => ConvertValueOparg::Repr, + ast::ConversionFlag::Ascii => ConvertValueOparg::Ascii, }; - if let Some(DebugText { leading, trailing }) = &fstring_expr.debug_text { + if let Some(ast::DebugText { leading, trailing }) = &fstring_expr.debug_text { let range = fstring_expr.expression.range(); let source = self.source_file.slice(range); let text = [leading, source, trailing].concat(); @@ -7176,26 +8283,139 @@ impl Compiler { Ok(()) } + + fn compile_expr_tstring(&mut self, expr_tstring: &ast::ExprTString) -> CompileResult<()> { + // ast::TStringValue can contain multiple ast::TString parts (implicit concatenation) + // Each ast::TString part should be compiled and the results merged into a single Template + let tstring_value = &expr_tstring.value; + + // Collect all strings and compile all interpolations + let mut all_strings: Vec<Wtf8Buf> = Vec::new(); + let mut current_string = Wtf8Buf::new(); + let mut interp_count: u32 = 0; + + for tstring in tstring_value.iter() { + self.compile_tstring_into( + tstring, + &mut all_strings, + &mut current_string, + &mut interp_count, + )?; + } + + // Add trailing string + all_strings.push(std::mem::take(&mut current_string)); + + // Now build the Template: + // Stack currently has all interpolations from compile_tstring_into calls + + // 1. Build interpolations tuple from the interpolations on the stack + emit!(self, Instruction::BuildTuple { size: interp_count }); + + // 2. Load all string parts + let string_count: u32 = all_strings + .len() + .try_into() + .expect("t-string string count overflowed"); + for s in &all_strings { + self.emit_load_const(ConstantData::Str { value: s.clone() }); + } + + // 3. Build strings tuple + emit!(self, Instruction::BuildTuple { size: string_count }); + + // 4. Swap so strings is below interpolations: [interps, strings] -> [strings, interps] + emit!(self, Instruction::Swap { index: 2 }); + + // 5. Build the Template + emit!(self, Instruction::BuildTemplate); + + Ok(()) + } + + fn compile_tstring_into( + &mut self, + tstring: &ast::TString, + strings: &mut Vec<Wtf8Buf>, + current_string: &mut Wtf8Buf, + interp_count: &mut u32, + ) -> CompileResult<()> { + for element in &tstring.elements { + match element { + ast::InterpolatedStringElement::Literal(lit) => { + // Accumulate literal parts into current_string + current_string.push_str(&lit.value); + } + ast::InterpolatedStringElement::Interpolation(interp) => { + // Finish current string segment + strings.push(std::mem::take(current_string)); + + // Compile the interpolation value + self.compile_expression(&interp.expression)?; + + // Load the expression source string + let expr_range = interp.expression.range(); + let expr_source = self.source_file.slice(expr_range); + self.emit_load_const(ConstantData::Str { + value: expr_source.to_string().into(), + }); + + // Determine conversion code + let conversion: u32 = match interp.conversion { + ast::ConversionFlag::None => 0, + ast::ConversionFlag::Str => 1, + ast::ConversionFlag::Repr => 2, + ast::ConversionFlag::Ascii => 3, + }; + + // Handle format_spec + let has_format_spec = interp.format_spec.is_some(); + if let Some(format_spec) = &interp.format_spec { + // Compile format_spec as a string using fstring element compilation + // Use default ast::FStringFlags since format_spec syntax is independent of t-string flags + self.compile_fstring_elements( + ast::FStringFlags::empty(), + &format_spec.elements, + )?; + } + + // Emit BUILD_INTERPOLATION + // oparg encoding: (conversion << 2) | has_format_spec + let oparg = (conversion << 2) | (has_format_spec as u32); + emit!(self, Instruction::BuildInterpolation { oparg }); + + *interp_count += 1; + } + } + } + + Ok(()) + } } trait EmitArg<Arg: OpArgType> { - fn emit( + fn emit<I: Into<AnyInstruction>>( self, - f: impl FnOnce(OpArgMarker<Arg>) -> Instruction, - ) -> (Instruction, OpArg, BlockIdx); + f: impl FnOnce(OpArgMarker<Arg>) -> I, + ) -> (AnyInstruction, OpArg, BlockIdx); } + impl<T: OpArgType> EmitArg<T> for T { - fn emit(self, f: impl FnOnce(OpArgMarker<T>) -> Instruction) -> (Instruction, OpArg, BlockIdx) { + fn emit<I: Into<AnyInstruction>>( + self, + f: impl FnOnce(OpArgMarker<T>) -> I, + ) -> (AnyInstruction, OpArg, BlockIdx) { let (marker, arg) = OpArgMarker::new(self); - (f(marker), arg, BlockIdx::NULL) + (f(marker).into(), arg, BlockIdx::NULL) } } + impl EmitArg<bytecode::Label> for BlockIdx { - fn emit( + fn emit<I: Into<AnyInstruction>>( self, - f: impl FnOnce(OpArgMarker<bytecode::Label>) -> Instruction, - ) -> (Instruction, OpArg, BlockIdx) { - (f(OpArgMarker::marker()), OpArg::null(), self) + f: impl FnOnce(OpArgMarker<bytecode::Label>) -> I, + ) -> (AnyInstruction, OpArg, BlockIdx) { + (f(OpArgMarker::marker()).into(), OpArg::null(), self) } } @@ -7264,12 +8484,12 @@ fn expandtabs(input: &str, tab_size: usize) -> String { expanded_str } -fn split_doc<'a>(body: &'a [Stmt], opts: &CompileOpts) -> (Option<String>, &'a [Stmt]) { - if let Some((Stmt::Expr(expr), body_rest)) = body.split_first() { +fn split_doc<'a>(body: &'a [ast::Stmt], opts: &CompileOpts) -> (Option<String>, &'a [ast::Stmt]) { + if let Some((ast::Stmt::Expr(expr), body_rest)) = body.split_first() { let doc_comment = match &*expr.value { - Expr::StringLiteral(value) => Some(&value.value), + ast::Expr::StringLiteral(value) => Some(&value.value), // f-strings are not allowed in Python doc comments. - Expr::FString(_) => None, + ast::Expr::FString(_) => None, _ => None, }; if let Some(doc) = doc_comment { @@ -7283,7 +8503,7 @@ fn split_doc<'a>(body: &'a [Stmt], opts: &CompileOpts) -> (Option<String>, &'a [ (None, body) } -pub fn ruff_int_to_bigint(int: &Int) -> Result<BigInt, CodegenErrorType> { +pub fn ruff_int_to_bigint(int: &ast::Int) -> Result<BigInt, CodegenErrorType> { if let Some(small) = int.as_u64() { Ok(BigInt::from(small)) } else { @@ -7293,7 +8513,7 @@ pub fn ruff_int_to_bigint(int: &Int) -> Result<BigInt, CodegenErrorType> { /// Converts a `ruff` ast integer into a `BigInt`. /// Unlike small integers, big integers may be stored in one of four possible radix representations. -fn parse_big_integer(int: &Int) -> Result<BigInt, CodegenErrorType> { +fn parse_big_integer(int: &ast::Int) -> Result<BigInt, CodegenErrorType> { // TODO: Improve ruff API // Can we avoid this copy? let s = format!("{int}"); @@ -7336,35 +8556,34 @@ impl ToU32 for usize { #[cfg(test)] mod ruff_tests { use super::*; - use ruff_python_ast::name::Name; - use ruff_python_ast::*; + use ast::name::Name; /// Test if the compiler can correctly identify fstrings containing an `await` expression. #[test] fn test_fstring_contains_await() { let range = TextRange::default(); - let flags = FStringFlags::empty(); + let flags = ast::FStringFlags::empty(); // f'{x}' - let expr_x = Expr::Name(ExprName { - node_index: AtomicNodeIndex::NONE, + let expr_x = ast::Expr::Name(ast::ExprName { + node_index: ast::AtomicNodeIndex::NONE, range, id: Name::new("x"), - ctx: ExprContext::Load, + ctx: ast::ExprContext::Load, }); - let not_present = &Expr::FString(ExprFString { - node_index: AtomicNodeIndex::NONE, + let not_present = &ast::Expr::FString(ast::ExprFString { + node_index: ast::AtomicNodeIndex::NONE, range, - value: FStringValue::single(FString { - node_index: AtomicNodeIndex::NONE, + value: ast::FStringValue::single(ast::FString { + node_index: ast::AtomicNodeIndex::NONE, range, - elements: vec![InterpolatedStringElement::Interpolation( - InterpolatedElement { - node_index: AtomicNodeIndex::NONE, + elements: vec![ast::InterpolatedStringElement::Interpolation( + ast::InterpolatedElement { + node_index: ast::AtomicNodeIndex::NONE, range, expression: Box::new(expr_x), debug_text: None, - conversion: ConversionFlag::None, + conversion: ast::ConversionFlag::None, format_spec: None, }, )] @@ -7375,29 +8594,29 @@ mod ruff_tests { assert!(!Compiler::contains_await(not_present)); // f'{await x}' - let expr_await_x = Expr::Await(ExprAwait { - node_index: AtomicNodeIndex::NONE, + let expr_await_x = ast::Expr::Await(ast::ExprAwait { + node_index: ast::AtomicNodeIndex::NONE, range, - value: Box::new(Expr::Name(ExprName { - node_index: AtomicNodeIndex::NONE, + value: Box::new(ast::Expr::Name(ast::ExprName { + node_index: ast::AtomicNodeIndex::NONE, range, id: Name::new("x"), - ctx: ExprContext::Load, + ctx: ast::ExprContext::Load, })), }); - let present = &Expr::FString(ExprFString { - node_index: AtomicNodeIndex::NONE, + let present = &ast::Expr::FString(ast::ExprFString { + node_index: ast::AtomicNodeIndex::NONE, range, - value: FStringValue::single(FString { - node_index: AtomicNodeIndex::NONE, + value: ast::FStringValue::single(ast::FString { + node_index: ast::AtomicNodeIndex::NONE, range, - elements: vec![InterpolatedStringElement::Interpolation( - InterpolatedElement { - node_index: AtomicNodeIndex::NONE, + elements: vec![ast::InterpolatedStringElement::Interpolation( + ast::InterpolatedElement { + node_index: ast::AtomicNodeIndex::NONE, range, expression: Box::new(expr_await_x), debug_text: None, - conversion: ConversionFlag::None, + conversion: ast::ConversionFlag::None, format_spec: None, }, )] @@ -7408,45 +8627,45 @@ mod ruff_tests { assert!(Compiler::contains_await(present)); // f'{x:{await y}}' - let expr_x = Expr::Name(ExprName { - node_index: AtomicNodeIndex::NONE, + let expr_x = ast::Expr::Name(ast::ExprName { + node_index: ast::AtomicNodeIndex::NONE, range, id: Name::new("x"), - ctx: ExprContext::Load, + ctx: ast::ExprContext::Load, }); - let expr_await_y = Expr::Await(ExprAwait { - node_index: AtomicNodeIndex::NONE, + let expr_await_y = ast::Expr::Await(ast::ExprAwait { + node_index: ast::AtomicNodeIndex::NONE, range, - value: Box::new(Expr::Name(ExprName { - node_index: AtomicNodeIndex::NONE, + value: Box::new(ast::Expr::Name(ast::ExprName { + node_index: ast::AtomicNodeIndex::NONE, range, id: Name::new("y"), - ctx: ExprContext::Load, + ctx: ast::ExprContext::Load, })), }); - let present = &Expr::FString(ExprFString { - node_index: AtomicNodeIndex::NONE, + let present = &ast::Expr::FString(ast::ExprFString { + node_index: ast::AtomicNodeIndex::NONE, range, - value: FStringValue::single(FString { - node_index: AtomicNodeIndex::NONE, + value: ast::FStringValue::single(ast::FString { + node_index: ast::AtomicNodeIndex::NONE, range, - elements: vec![InterpolatedStringElement::Interpolation( - InterpolatedElement { - node_index: AtomicNodeIndex::NONE, + elements: vec![ast::InterpolatedStringElement::Interpolation( + ast::InterpolatedElement { + node_index: ast::AtomicNodeIndex::NONE, range, expression: Box::new(expr_x), debug_text: None, - conversion: ConversionFlag::None, - format_spec: Some(Box::new(InterpolatedStringFormatSpec { - node_index: AtomicNodeIndex::NONE, + conversion: ast::ConversionFlag::None, + format_spec: Some(Box::new(ast::InterpolatedStringFormatSpec { + node_index: ast::AtomicNodeIndex::NONE, range, - elements: vec![InterpolatedStringElement::Interpolation( - InterpolatedElement { - node_index: AtomicNodeIndex::NONE, + elements: vec![ast::InterpolatedStringElement::Interpolation( + ast::InterpolatedElement { + node_index: ast::AtomicNodeIndex::NONE, range, expression: Box::new(expr_await_y), debug_text: None, - conversion: ConversionFlag::None, + conversion: ast::ConversionFlag::None, format_spec: None, }, )] diff --git a/crates/codegen/src/error.rs b/crates/codegen/src/error.rs index 5bd7c7ecaf0..9f1dcc27058 100644 --- a/crates/codegen/src/error.rs +++ b/crates/codegen/src/error.rs @@ -3,7 +3,7 @@ use core::fmt::Display; use rustpython_compiler_core::SourceLocation; use thiserror::Error; -#[derive(Debug)] +#[derive(Clone, Copy, Debug)] pub enum PatternUnreachableReason { NameCapture, Wildcard, diff --git a/crates/codegen/src/ir.rs b/crates/codegen/src/ir.rs index 6cc22b438f5..de5781ab647 100644 --- a/crates/codegen/src/ir.rs +++ b/crates/codegen/src/ir.rs @@ -1,16 +1,28 @@ use core::ops; use crate::{IndexMap, IndexSet, error::InternalError}; +use malachite_bigint::BigInt; +use num_traits::ToPrimitive; + use rustpython_compiler_core::{ OneIndexed, SourceLocation, bytecode::{ - CodeFlags, CodeObject, CodeUnit, CodeUnits, ConstantData, ExceptionTableEntry, - InstrDisplayContext, Instruction, Label, OpArg, PyCodeLocationInfoKind, - encode_exception_table, + AnyInstruction, Arg, CodeFlags, CodeObject, CodeUnit, CodeUnits, ConstantData, + ExceptionTableEntry, InstrDisplayContext, Instruction, InstructionMetadata, Label, OpArg, + PseudoInstruction, PyCodeLocationInfoKind, encode_exception_table, }, varint::{write_signed_varint, write_varint}, }; +/// Location info for linetable generation (allows line 0 for RESUME) +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +struct LineTableLocation { + line: i32, + end_line: i32, + col: i32, + end_col: i32, +} + /// Metadata for a code unit // = _PyCompile_CodeUnitMetadata #[derive(Clone, Debug)] @@ -83,18 +95,20 @@ impl ops::IndexMut<BlockIdx> for Vec<Block> { } } -#[derive(Debug, Clone)] +#[derive(Clone, Copy, Debug)] pub struct InstructionInfo { - pub instr: Instruction, + pub instr: AnyInstruction, pub arg: OpArg, pub target: BlockIdx, pub location: SourceLocation, pub end_location: SourceLocation, pub except_handler: Option<ExceptHandlerInfo>, + /// Override line number for linetable (e.g., line 0 for module RESUME) + pub lineno_override: Option<i32>, } -/// Exception handler information for an instruction -#[derive(Debug, Clone)] +/// Exception handler information for an instruction. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct ExceptHandlerInfo { /// Block to jump to when exception occurs pub handler_block: BlockIdx, @@ -111,6 +125,13 @@ pub struct ExceptHandlerInfo { pub struct Block { pub instructions: Vec<InstructionInfo>, pub next: BlockIdx, + // Post-codegen analysis fields (set by label_exception_targets) + /// Whether this block is an exception handler target (b_except_handler) + pub except_handler: bool, + /// Whether to preserve lasti for this handler block (b_preserve_lasti) + pub preserve_lasti: bool, + /// Stack depth at block entry, set by stack depth analysis + pub start_depth: Option<u32>, } impl Default for Block { @@ -118,6 +139,9 @@ impl Default for Block { Self { instructions: Vec::new(), next: BlockIdx::NULL, + except_handler: false, + preserve_lasti: false, + start_depth: None, } } } @@ -143,6 +167,14 @@ pub struct CodeInfo { // Reference to the symbol table for this scope pub symbol_table_index: usize, + + // PEP 649: Track nesting depth inside conditional blocks (if/for/while/etc.) + // u_in_conditional_block + pub in_conditional_block: u32, + + // PEP 649: Next index for conditional annotation tracking + // u_next_conditional_annotation_index + pub next_conditional_annotation_index: u32, } impl CodeInfo { @@ -150,10 +182,30 @@ impl CodeInfo { mut self, opts: &crate::compile::CompileOpts, ) -> crate::InternalResult<CodeObject> { + // Always fold tuple constants + self.fold_tuple_constants(); + // Python only applies LOAD_SMALL_INT conversion to module-level code + // (not inside functions). Module code lacks OPTIMIZED flag. + // Note: RustPython incorrectly sets NEWLOCALS on modules, so only check OPTIMIZED + let is_module_level = !self.flags.contains(CodeFlags::OPTIMIZED); + if is_module_level { + self.convert_to_load_small_int(); + } + self.remove_unused_consts(); + self.remove_nops(); + if opts.optimize > 0 { self.dce(); + self.peephole_optimize(); } + // Always apply LOAD_FAST_BORROW optimization + self.optimize_load_fast_borrow(); + + // Post-codegen CFG analysis passes (flowgraph.c pipeline) + mark_except_handlers(&mut self.blocks); + label_exception_targets(&mut self.blocks); + let max_stackdepth = self.max_stackdepth()?; let cell2arg = self.cell2arg(); @@ -169,6 +221,8 @@ impl CodeInfo { in_inlined_comp: _, fblock: _, symbol_table_index: _, + in_conditional_block: _, + next_conditional_annotation_index: _, } = self; let CodeUnitMetadata { @@ -188,6 +242,18 @@ impl CodeInfo { let mut instructions = Vec::new(); let mut locations = Vec::new(); + let mut linetable_locations: Vec<LineTableLocation> = Vec::new(); + + // Convert pseudo ops and remove resulting NOPs + convert_pseudo_ops(&mut blocks, varname_cache.len() as u32); + for block in blocks + .iter_mut() + .filter(|b| b.next != BlockIdx::NULL || !b.instructions.is_empty()) + { + block + .instructions + .retain(|ins| !matches!(ins.instr.real(), Some(Instruction::Nop))); + } let mut block_to_offset = vec![Label(0); blocks.len()]; // block_to_index: maps block idx to instruction index (for exception table) @@ -213,23 +279,64 @@ impl CodeInfo { let mut next_block = BlockIdx(0); while next_block != BlockIdx::NULL { let block = &mut blocks[next_block]; + // Track current instruction offset for jump direction resolution + let mut current_offset = block_to_offset[next_block.idx()].0; for info in &mut block.instructions { - let (op, arg, target) = (info.instr, &mut info.arg, info.target); + let target = info.target; if target != BlockIdx::NULL { let new_arg = OpArg(block_to_offset[target.idx()].0); - recompile_extended_arg |= new_arg.instr_size() != arg.instr_size(); - *arg = new_arg; + recompile_extended_arg |= new_arg.instr_size() != info.arg.instr_size(); + info.arg = new_arg; } - let (extras, lo_arg) = arg.split(); + + // Convert JUMP pseudo to real instructions (direction depends on offset) + let op = match info.instr { + AnyInstruction::Pseudo(PseudoInstruction::Jump { .. }) + if target != BlockIdx::NULL => + { + let target_offset = block_to_offset[target.idx()].0; + if target_offset > current_offset { + Instruction::JumpForward { + target: Arg::marker(), + } + } else { + Instruction::JumpBackward { + target: Arg::marker(), + } + } + } + AnyInstruction::Pseudo(PseudoInstruction::JumpNoInterrupt { .. }) + if target != BlockIdx::NULL => + { + // JumpNoInterrupt is always backward (used in yield-from/await loops) + Instruction::JumpBackwardNoInterrupt { + target: Arg::marker(), + } + } + other => other.expect_real(), + }; + + let (extras, lo_arg) = info.arg.split(); locations.extend(core::iter::repeat_n( (info.location, info.end_location), - arg.instr_size(), + info.arg.instr_size(), )); + // Collect linetable locations with lineno_override support + let lt_loc = LineTableLocation { + line: info + .lineno_override + .unwrap_or_else(|| info.location.line.get() as i32), + end_line: info.end_location.line.get() as i32, + col: info.location.character_offset.to_zero_indexed() as i32, + end_col: info.end_location.character_offset.to_zero_indexed() as i32, + }; + linetable_locations.extend(core::iter::repeat_n(lt_loc, info.arg.instr_size())); instructions.extend( extras .map(|byte| CodeUnit::new(Instruction::ExtendedArg, byte)) .chain([CodeUnit { op, arg: lo_arg }]), ); + current_offset += info.arg.instr_size() as u32; } next_block = block.next; } @@ -239,12 +346,13 @@ impl CodeInfo { } instructions.clear(); - locations.clear() + locations.clear(); + linetable_locations.clear(); } - // Generate linetable from locations + // Generate linetable from linetable_locations (supports line 0 for RESUME) let linetable = generate_linetable( - &locations, + &linetable_locations, first_line_number.get() as i32, opts.debug_ranges, ); @@ -283,8 +391,8 @@ impl CodeInfo { let total_args = self.metadata.argcount + self.metadata.kwonlyargcount - + self.flags.contains(CodeFlags::HAS_VARARGS) as u32 - + self.flags.contains(CodeFlags::HAS_VARKEYWORDS) as u32; + + self.flags.contains(CodeFlags::VARARGS) as u32 + + self.flags.contains(CodeFlags::VARKEYWORDS) as u32; let mut found_cellarg = false; let cell2arg = self @@ -311,7 +419,7 @@ impl CodeInfo { for block in &mut self.blocks { let mut last_instr = None; for (i, ins) in block.instructions.iter().enumerate() { - if ins.instr.unconditional_branch() { + if ins.instr.is_scope_exit() || ins.instr.is_unconditional_jump() { last_instr = Some(i); break; } @@ -322,7 +430,346 @@ impl CodeInfo { } } - fn max_stackdepth(&self) -> crate::InternalResult<u32> { + /// Constant folding: fold LOAD_CONST/LOAD_SMALL_INT + BUILD_TUPLE into LOAD_CONST tuple + /// fold_tuple_of_constants + fn fold_tuple_constants(&mut self) { + for block in &mut self.blocks { + let mut i = 0; + while i < block.instructions.len() { + let instr = &block.instructions[i]; + // Look for BUILD_TUPLE + let Some(Instruction::BuildTuple { .. }) = instr.instr.real() else { + i += 1; + continue; + }; + + let tuple_size = instr.arg.0 as usize; + if tuple_size == 0 || i < tuple_size { + i += 1; + continue; + } + + // Check if all preceding instructions are constant-loading + let start_idx = i - tuple_size; + let mut elements = Vec::with_capacity(tuple_size); + let mut all_const = true; + + for j in start_idx..i { + let load_instr = &block.instructions[j]; + match load_instr.instr.real() { + Some(Instruction::LoadConst { .. }) => { + let const_idx = load_instr.arg.0 as usize; + if let Some(constant) = + self.metadata.consts.get_index(const_idx).cloned() + { + elements.push(constant); + } else { + all_const = false; + break; + } + } + Some(Instruction::LoadSmallInt { .. }) => { + // arg is the i32 value stored as u32 (two's complement) + let value = load_instr.arg.0 as i32; + elements.push(ConstantData::Integer { + value: BigInt::from(value), + }); + } + _ => { + all_const = false; + break; + } + } + } + + if !all_const { + i += 1; + continue; + } + + // Note: The first small int is added to co_consts during compilation + // (in compile_default_arguments). + // We don't need to add it here again. + + // Create tuple constant and add to consts + let tuple_const = ConstantData::Tuple { elements }; + let (const_idx, _) = self.metadata.consts.insert_full(tuple_const); + + // Replace preceding LOAD instructions with NOP + for j in start_idx..i { + block.instructions[j].instr = Instruction::Nop.into(); + } + + // Replace BUILD_TUPLE with LOAD_CONST + block.instructions[i].instr = Instruction::LoadConst { idx: Arg::marker() }.into(); + block.instructions[i].arg = OpArg(const_idx as u32); + + i += 1; + } + } + } + + /// Peephole optimization: combine consecutive instructions into super-instructions + fn peephole_optimize(&mut self) { + for block in &mut self.blocks { + let mut i = 0; + while i + 1 < block.instructions.len() { + let combined = { + let curr = &block.instructions[i]; + let next = &block.instructions[i + 1]; + + // Only combine if both are real instructions (not pseudo) + let (Some(curr_instr), Some(next_instr)) = + (curr.instr.real(), next.instr.real()) + else { + i += 1; + continue; + }; + + match (curr_instr, next_instr) { + // LoadFast + LoadFast -> LoadFastLoadFast (if both indices < 16) + (Instruction::LoadFast(_), Instruction::LoadFast(_)) => { + let idx1 = curr.arg.0; + let idx2 = next.arg.0; + if idx1 < 16 && idx2 < 16 { + let packed = (idx1 << 4) | idx2; + Some(( + Instruction::LoadFastLoadFast { arg: Arg::marker() }, + OpArg(packed), + )) + } else { + None + } + } + // StoreFast + StoreFast -> StoreFastStoreFast (if both indices < 16) + (Instruction::StoreFast(_), Instruction::StoreFast(_)) => { + let idx1 = curr.arg.0; + let idx2 = next.arg.0; + if idx1 < 16 && idx2 < 16 { + let packed = (idx1 << 4) | idx2; + Some(( + Instruction::StoreFastStoreFast { arg: Arg::marker() }, + OpArg(packed), + )) + } else { + None + } + } + _ => None, + } + }; + + if let Some((new_instr, new_arg)) = combined { + // Combine: keep first instruction's location, replace with combined instruction + block.instructions[i].instr = new_instr.into(); + block.instructions[i].arg = new_arg; + // Remove the second instruction + block.instructions.remove(i + 1); + // Don't increment i - check if we can combine again with the next instruction + } else { + i += 1; + } + } + } + } + + /// Convert LOAD_CONST for small integers to LOAD_SMALL_INT + /// maybe_instr_make_load_smallint + fn convert_to_load_small_int(&mut self) { + for block in &mut self.blocks { + for instr in &mut block.instructions { + // Check if it's a LOAD_CONST instruction + let Some(Instruction::LoadConst { .. }) = instr.instr.real() else { + continue; + }; + + // Get the constant value + let const_idx = instr.arg.0 as usize; + let Some(constant) = self.metadata.consts.get_index(const_idx) else { + continue; + }; + + // Check if it's a small integer + let ConstantData::Integer { value } = constant else { + continue; + }; + + // Check if it's in small int range: -5 to 256 (_PY_IS_SMALL_INT) + if let Some(small) = value.to_i32().filter(|v| (-5..=256).contains(v)) { + // Convert LOAD_CONST to LOAD_SMALL_INT + instr.instr = Instruction::LoadSmallInt { idx: Arg::marker() }.into(); + // The arg is the i32 value stored as u32 (two's complement) + instr.arg = OpArg(small as u32); + } + } + } + } + + /// Remove constants that are no longer referenced by LOAD_CONST instructions. + /// remove_unused_consts + fn remove_unused_consts(&mut self) { + let nconsts = self.metadata.consts.len(); + if nconsts == 0 { + return; + } + + // Mark used constants + // The first constant (index 0) is always kept (may be docstring) + let mut used = vec![false; nconsts]; + used[0] = true; + + for block in &self.blocks { + for instr in &block.instructions { + if let Some(Instruction::LoadConst { .. }) = instr.instr.real() { + let idx = instr.arg.0 as usize; + if idx < nconsts { + used[idx] = true; + } + } + } + } + + // Check if any constants can be removed + let n_used: usize = used.iter().filter(|&&u| u).count(); + if n_used == nconsts { + return; // Nothing to remove + } + + // Build old_to_new index mapping + let mut old_to_new = vec![0usize; nconsts]; + let mut new_idx = 0usize; + for (old_idx, &is_used) in used.iter().enumerate() { + if is_used { + old_to_new[old_idx] = new_idx; + new_idx += 1; + } + } + + // Build new consts list + let old_consts: Vec<_> = self.metadata.consts.iter().cloned().collect(); + self.metadata.consts.clear(); + for (old_idx, constant) in old_consts.into_iter().enumerate() { + if used[old_idx] { + self.metadata.consts.insert(constant); + } + } + + // Update LOAD_CONST instruction arguments + for block in &mut self.blocks { + for instr in &mut block.instructions { + if let Some(Instruction::LoadConst { .. }) = instr.instr.real() { + let old_idx = instr.arg.0 as usize; + if old_idx < nconsts { + instr.arg = OpArg(old_to_new[old_idx] as u32); + } + } + } + } + } + + /// Remove NOP instructions from all blocks + fn remove_nops(&mut self) { + for block in &mut self.blocks { + block + .instructions + .retain(|ins| !matches!(ins.instr.real(), Some(Instruction::Nop))); + } + } + + /// Optimize LOAD_FAST to LOAD_FAST_BORROW where safe. + /// + /// A LOAD_FAST can be converted to LOAD_FAST_BORROW if its value is + /// consumed within the same basic block (not passed to another block). + /// This is a reference counting optimization in CPython; in RustPython + /// we implement it for bytecode compatibility. + fn optimize_load_fast_borrow(&mut self) { + // NOT_LOCAL marker: instruction didn't come from a LOAD_FAST + const NOT_LOCAL: usize = usize::MAX; + + for block in &mut self.blocks { + if block.instructions.is_empty() { + continue; + } + + // Track which instructions' outputs are still on stack at block end + // For each instruction, we track if its pushed value(s) are unconsumed + let mut unconsumed = vec![false; block.instructions.len()]; + + // Simulate stack: each entry is the instruction index that pushed it + // (or NOT_LOCAL if not from LOAD_FAST/LOAD_FAST_LOAD_FAST). + // + // CPython (flowgraph.c optimize_load_fast) pre-fills the stack with + // dummy refs for values inherited from predecessor blocks. We take + // the simpler approach of aborting the optimisation for the whole + // block on stack underflow. + let mut stack: Vec<usize> = Vec::new(); + let mut underflow = false; + + for (i, info) in block.instructions.iter().enumerate() { + let Some(instr) = info.instr.real() else { + continue; + }; + + let stack_effect_info = instr.stack_effect_info(info.arg.into()); + let (pushes, pops) = (stack_effect_info.pushed(), stack_effect_info.popped()); + + // Pop values from stack + for _ in 0..pops { + if stack.pop().is_none() { + // Stack underflow — block receives values from a predecessor. + // Abort optimisation for the entire block. + underflow = true; + break; + } + } + if underflow { + break; + } + + // Push values to stack with source instruction index + let source = match instr { + Instruction::LoadFast(_) | Instruction::LoadFastLoadFast { .. } => i, + _ => NOT_LOCAL, + }; + for _ in 0..pushes { + stack.push(source); + } + } + + if underflow { + continue; + } + + // Mark instructions whose values remain on stack at block end + for &src in &stack { + if src != NOT_LOCAL { + unconsumed[src] = true; + } + } + + // Convert LOAD_FAST to LOAD_FAST_BORROW where value is fully consumed + for (i, info) in block.instructions.iter_mut().enumerate() { + if unconsumed[i] { + continue; + } + let Some(instr) = info.instr.real() else { + continue; + }; + match instr { + Instruction::LoadFast(_) => { + info.instr = Instruction::LoadFastBorrow(Arg::marker()).into(); + } + Instruction::LoadFastLoadFast { .. } => { + info.instr = + Instruction::LoadFastBorrowLoadFastBorrow { arg: Arg::marker() }.into(); + } + _ => {} + } + } + } + } + + fn max_stackdepth(&mut self) -> crate::InternalResult<u32> { let mut maxdepth = 0u32; let mut stack = Vec::with_capacity(self.blocks.len()); let mut start_depths = vec![u32::MAX; self.blocks.len()]; @@ -349,7 +796,7 @@ impl CodeInfo { let block = &self.blocks[block_idx]; for ins in &block.instructions { let instr = &ins.instr; - let effect = instr.stack_effect(ins.arg, false); + let effect = instr.stack_effect(ins.arg.into()); if DEBUG { let display_arg = if ins.target == BlockIdx::NULL { ins.arg @@ -374,50 +821,43 @@ impl CodeInfo { } // Process target blocks for branching instructions if ins.target != BlockIdx::NULL { - let effect = instr.stack_effect(ins.arg, true); - let target_depth = depth.checked_add_signed(effect).ok_or({ - if effect < 0 { - InternalError::StackUnderflow - } else { - InternalError::StackOverflow + if instr.is_block_push() { + // SETUP_* pseudo ops: target is a handler block. + // Handler entry depth uses the jump-path stack effect: + // SETUP_FINALLY: +1 (pushes exc) + // SETUP_CLEANUP: +2 (pushes lasti + exc) + // SETUP_WITH: +1 (pops __enter__ result, pushes lasti + exc) + let handler_effect: u32 = match instr.pseudo() { + Some(PseudoInstruction::SetupCleanup { .. }) => 2, + _ => 1, // SetupFinally and SetupWith + }; + let handler_depth = depth + handler_effect; + if handler_depth > maxdepth { + maxdepth = handler_depth; } - })?; - if target_depth > maxdepth { - maxdepth = target_depth - } - stackdepth_push(&mut stack, &mut start_depths, ins.target, target_depth); - } - // Process exception handler blocks - // When exception occurs, stack is unwound to handler.stack_depth, then: - // - If preserve_lasti: push lasti (+1) - // - Push exception (+1) - // - Handler block starts with PUSH_EXC_INFO as its first instruction - // So the starting depth for the handler block (BEFORE PUSH_EXC_INFO) is: - // handler.stack_depth + preserve_lasti + 1 (exc) - // PUSH_EXC_INFO will then add +1 when the block is processed - if let Some(ref handler) = ins.except_handler { - let handler_depth = handler.stack_depth + 1 + (handler.preserve_lasti as u32); // +1 for exception, +1 for lasti if preserve_lasti - if DEBUG { - eprintln!( - " HANDLER: block={} depth={} (base={} lasti={})", - handler.handler_block.0, - handler_depth, - handler.stack_depth, - handler.preserve_lasti - ); - } - if handler_depth > maxdepth { - maxdepth = handler_depth; + stackdepth_push(&mut stack, &mut start_depths, ins.target, handler_depth); + } else { + // SEND jumps to END_SEND with receiver still on stack. + // END_SEND performs the receiver pop. + let jump_effect = match instr.real() { + Some(Instruction::Send { .. }) => 0i32, + _ => effect, + }; + let target_depth = depth.checked_add_signed(jump_effect).ok_or({ + if jump_effect < 0 { + InternalError::StackUnderflow + } else { + InternalError::StackOverflow + } + })?; + if target_depth > maxdepth { + maxdepth = target_depth + } + stackdepth_push(&mut stack, &mut start_depths, ins.target, target_depth); } - stackdepth_push( - &mut stack, - &mut start_depths, - handler.handler_block, - handler_depth, - ); } depth = new_depth; - if instr.unconditional_branch() { + if instr.is_scope_exit() || instr.is_unconditional_jump() { continue 'process_blocks; } } @@ -429,6 +869,25 @@ impl CodeInfo { if DEBUG { eprintln!("DONE: {maxdepth}"); } + + // Fix up handler stack_depth in ExceptHandlerInfo using start_depths + // computed above: depth = start_depth - 1 - preserve_lasti + for block in self.blocks.iter_mut() { + for ins in &mut block.instructions { + if let Some(ref mut handler) = ins.except_handler { + let h_start = start_depths[handler.handler_block.idx()]; + if h_start != u32::MAX { + let adjustment = 1 + handler.preserve_lasti as u32; + debug_assert!( + h_start >= adjustment, + "handler start depth {h_start} too shallow for adjustment {adjustment}" + ); + handler.stack_depth = h_start.saturating_sub(adjustment); + } + } + } + } + Ok(maxdepth) } } @@ -482,7 +941,7 @@ fn iter_blocks(blocks: &[Block]) -> impl Iterator<Item = (BlockIdx, &Block)> + ' /// Generate Python 3.11+ format linetable from source locations fn generate_linetable( - locations: &[(SourceLocation, SourceLocation)], + locations: &[LineTableLocation], first_line: i32, debug_ranges: bool, ) -> Box<[u8]> { @@ -497,7 +956,7 @@ fn generate_linetable( let mut i = 0; while i < locations.len() { - let (loc, end_loc) = &locations[i]; + let loc = &locations[i]; // Count consecutive instructions with the same location let mut length = 1; @@ -510,8 +969,8 @@ fn generate_linetable( let entry_length = length.min(8); // Get line information - let line = loc.line.get() as i32; - let end_line = end_loc.line.get() as i32; + let line = loc.line; + let end_line = loc.end_line; let line_delta = line - prev_line; let end_line_delta = end_line - line; @@ -531,8 +990,8 @@ fn generate_linetable( } // Get column information (only when debug_ranges is enabled) - let col = loc.character_offset.to_zero_indexed() as i32; - let end_col = end_loc.character_offset.to_zero_indexed() as i32; + let col = loc.col; + let end_col = loc.end_col; // Choose the appropriate encoding based on line delta and column info if line_delta == 0 && end_line_delta == 0 { @@ -614,13 +1073,13 @@ fn generate_exception_table(blocks: &[Block], block_to_index: &[u32]) -> Box<[u8 // instr_size includes EXTENDED_ARG instructions let instr_size = instr.arg.instr_size() as u32; - match (&current_entry, &instr.except_handler) { + match (&current_entry, instr.except_handler) { // No current entry, no handler - nothing to do (None, None) => {} // No current entry, handler starts - begin new entry (None, Some(handler)) => { - current_entry = Some((handler.clone(), instr_index)); + current_entry = Some((handler, instr_index)); } // Current entry exists, same handler - continue @@ -639,7 +1098,7 @@ fn generate_exception_table(blocks: &[Block], block_to_index: &[u32]) -> Box<[u8 curr_handler.stack_depth as u16, curr_handler.preserve_lasti, )); - current_entry = Some((handler.clone(), instr_index)); + current_entry = Some((handler, instr_index)); } // Current entry exists, no handler - finish current entry @@ -674,3 +1133,197 @@ fn generate_exception_table(blocks: &[Block], block_to_index: &[u32]) -> Box<[u8 encode_exception_table(&entries) } + +/// Mark exception handler target blocks. +/// flowgraph.c mark_except_handlers +pub(crate) fn mark_except_handlers(blocks: &mut [Block]) { + // Reset handler flags + for block in blocks.iter_mut() { + block.except_handler = false; + block.preserve_lasti = false; + } + // Mark target blocks of SETUP_* as except handlers + let targets: Vec<usize> = blocks + .iter() + .flat_map(|b| b.instructions.iter()) + .filter(|i| i.instr.is_block_push() && i.target != BlockIdx::NULL) + .map(|i| i.target.idx()) + .collect(); + for idx in targets { + blocks[idx].except_handler = true; + } +} + +/// Label exception targets: walk CFG with except stack, set per-instruction +/// handler info and block preserve_lasti flag. Converts POP_BLOCK to NOP. +/// flowgraph.c label_exception_targets + push_except_block +pub(crate) fn label_exception_targets(blocks: &mut [Block]) { + #[derive(Clone)] + struct ExceptEntry { + handler_block: BlockIdx, + preserve_lasti: bool, + } + + let num_blocks = blocks.len(); + if num_blocks == 0 { + return; + } + + let mut visited = vec![false; num_blocks]; + let mut block_stacks: Vec<Option<Vec<ExceptEntry>>> = vec![None; num_blocks]; + + // Entry block + visited[0] = true; + block_stacks[0] = Some(Vec::new()); + + let mut todo = vec![BlockIdx(0)]; + + while let Some(block_idx) = todo.pop() { + let bi = block_idx.idx(); + let mut stack = block_stacks[bi].take().unwrap_or_default(); + let mut last_yield_except_depth: i32 = -1; + + let instr_count = blocks[bi].instructions.len(); + for i in 0..instr_count { + // Read all needed fields (each temporary borrow ends immediately) + let target = blocks[bi].instructions[i].target; + let arg = blocks[bi].instructions[i].arg; + let is_push = blocks[bi].instructions[i].instr.is_block_push(); + let is_pop = blocks[bi].instructions[i].instr.is_pop_block(); + + if is_push { + // Determine preserve_lasti from instruction type (push_except_block) + let preserve_lasti = matches!( + blocks[bi].instructions[i].instr.pseudo(), + Some( + PseudoInstruction::SetupWith { .. } + | PseudoInstruction::SetupCleanup { .. } + ) + ); + + // Set preserve_lasti on handler block + if preserve_lasti && target != BlockIdx::NULL { + blocks[target.idx()].preserve_lasti = true; + } + + // Propagate except stack to handler block if not visited + if target != BlockIdx::NULL && !visited[target.idx()] { + visited[target.idx()] = true; + block_stacks[target.idx()] = Some(stack.clone()); + todo.push(target); + } + + // Push handler onto except stack + stack.push(ExceptEntry { + handler_block: target, + preserve_lasti, + }); + } else if is_pop { + debug_assert!( + !stack.is_empty(), + "POP_BLOCK with empty except stack at block {bi} instruction {i}" + ); + stack.pop(); + // POP_BLOCK → NOP + blocks[bi].instructions[i].instr = Instruction::Nop.into(); + } else { + // Set except_handler for this instruction from except stack top + // stack_depth placeholder: filled by fixup_handler_depths + let handler_info = stack.last().map(|e| ExceptHandlerInfo { + handler_block: e.handler_block, + stack_depth: 0, + preserve_lasti: e.preserve_lasti, + }); + blocks[bi].instructions[i].except_handler = handler_info; + + // Track YIELD_VALUE except stack depth + if matches!( + blocks[bi].instructions[i].instr.real(), + Some(Instruction::YieldValue { .. }) + ) { + last_yield_except_depth = stack.len() as i32; + } + + // Set RESUME DEPTH1 flag based on last yield's except depth + if matches!( + blocks[bi].instructions[i].instr.real(), + Some(Instruction::Resume { .. }) + ) { + const RESUME_AT_FUNC_START: u32 = 0; + const RESUME_OPARG_LOCATION_MASK: u32 = 0x3; + const RESUME_OPARG_DEPTH1_MASK: u32 = 0x4; + + if (arg.0 & RESUME_OPARG_LOCATION_MASK) != RESUME_AT_FUNC_START { + if last_yield_except_depth == 1 { + blocks[bi].instructions[i].arg = + OpArg(arg.0 | RESUME_OPARG_DEPTH1_MASK); + } + last_yield_except_depth = -1; + } + } + + // For jump instructions, propagate except stack to target + if target != BlockIdx::NULL && !visited[target.idx()] { + visited[target.idx()] = true; + block_stacks[target.idx()] = Some(stack.clone()); + todo.push(target); + } + } + } + + // Propagate to fallthrough block (block.next) + let next = blocks[bi].next; + if next != BlockIdx::NULL && !visited[next.idx()] { + let has_fallthrough = blocks[bi] + .instructions + .last() + .map(|ins| !ins.instr.is_scope_exit() && !ins.instr.is_unconditional_jump()) + .unwrap_or(true); // Empty block falls through + if has_fallthrough { + visited[next.idx()] = true; + block_stacks[next.idx()] = Some(stack); + todo.push(next); + } + } + } +} + +/// Convert remaining pseudo ops to real instructions or NOP. +/// flowgraph.c convert_pseudo_ops +pub(crate) fn convert_pseudo_ops(blocks: &mut [Block], varnames_len: u32) { + for block in blocks.iter_mut() { + for info in &mut block.instructions { + let Some(pseudo) = info.instr.pseudo() else { + continue; + }; + match pseudo { + // Block push pseudo ops → NOP + PseudoInstruction::SetupCleanup { .. } + | PseudoInstruction::SetupFinally { .. } + | PseudoInstruction::SetupWith { .. } => { + info.instr = Instruction::Nop.into(); + } + // PopBlock in reachable blocks is converted to NOP by + // label_exception_targets. Dead blocks may still have them. + PseudoInstruction::PopBlock => { + info.instr = Instruction::Nop.into(); + } + // LOAD_CLOSURE → LOAD_FAST (with varnames offset) + PseudoInstruction::LoadClosure(idx) => { + let new_idx = varnames_len + idx.get(info.arg); + info.arg = OpArg(new_idx); + info.instr = Instruction::LoadFast(Arg::marker()).into(); + } + // Jump pseudo ops are resolved during block linearization + PseudoInstruction::Jump { .. } | PseudoInstruction::JumpNoInterrupt { .. } => {} + // These should have been resolved earlier + PseudoInstruction::AnnotationsPlaceholder + | PseudoInstruction::JumpIfFalse { .. } + | PseudoInstruction::JumpIfTrue { .. } + | PseudoInstruction::StoreFastMaybeNull(_) => { + unreachable!("Unexpected pseudo instruction in convert_pseudo_ops: {pseudo:?}") + } + } + } + } +} diff --git a/crates/codegen/src/lib.rs b/crates/codegen/src/lib.rs index 34d3870ae91..9dd7384170a 100644 --- a/crates/codegen/src/lib.rs +++ b/crates/codegen/src/lib.rs @@ -18,7 +18,7 @@ pub mod symboltable; mod unparse; pub use compile::CompileOpts; -use ruff_python_ast::Expr; +use ruff_python_ast as ast; pub(crate) use compile::InternalResult; @@ -27,7 +27,7 @@ pub trait ToPythonName { fn python_name(&self) -> &'static str; } -impl ToPythonName for Expr { +impl ToPythonName for ast::Expr { fn python_name(&self) -> &'static str { match self { Self::BoolOp { .. } | Self::BinOp { .. } | Self::UnaryOp { .. } => "operator", diff --git a/crates/codegen/src/snapshots/rustpython_codegen__compile__tests__if_ands.snap b/crates/codegen/src/snapshots/rustpython_codegen__compile__tests__if_ands.snap index 4c9c29887ee..26583f5da0f 100644 --- a/crates/codegen/src/snapshots/rustpython_codegen__compile__tests__if_ands.snap +++ b/crates/codegen/src/snapshots/rustpython_codegen__compile__tests__if_ands.snap @@ -2,11 +2,13 @@ source: crates/codegen/src/compile.rs expression: "compile_exec(\"\\\nif True and False and False:\n pass\n\")" --- - 1 0 LOAD_CONST (True) - 1 POP_JUMP_IF_FALSE (6) - 2 LOAD_CONST (False) - 3 POP_JUMP_IF_FALSE (6) - 4 LOAD_CONST (False) - 5 POP_JUMP_IF_FALSE (6) + 1 0 RESUME (0) + 1 LOAD_CONST (True) + 2 POP_JUMP_IF_FALSE (7) + 3 LOAD_CONST (False) + 4 POP_JUMP_IF_FALSE (7) + 5 LOAD_CONST (False) + 6 POP_JUMP_IF_FALSE (7) - 2 >> 6 RETURN_CONST (None) + 2 >> 7 LOAD_CONST (None) + 8 RETURN_VALUE diff --git a/crates/codegen/src/snapshots/rustpython_codegen__compile__tests__if_mixed.snap b/crates/codegen/src/snapshots/rustpython_codegen__compile__tests__if_mixed.snap index a93479df96e..21976b257ef 100644 --- a/crates/codegen/src/snapshots/rustpython_codegen__compile__tests__if_mixed.snap +++ b/crates/codegen/src/snapshots/rustpython_codegen__compile__tests__if_mixed.snap @@ -2,13 +2,15 @@ source: crates/codegen/src/compile.rs expression: "compile_exec(\"\\\nif (True and False) or (False and True):\n pass\n\")" --- - 1 0 LOAD_CONST (True) - 1 POP_JUMP_IF_FALSE (4) - 2 LOAD_CONST (False) - 3 POP_JUMP_IF_TRUE (8) - >> 4 LOAD_CONST (False) - 5 POP_JUMP_IF_FALSE (8) - 6 LOAD_CONST (True) - 7 POP_JUMP_IF_FALSE (8) + 1 0 RESUME (0) + 1 LOAD_CONST (True) + 2 POP_JUMP_IF_FALSE (5) + 3 LOAD_CONST (False) + 4 POP_JUMP_IF_TRUE (9) + >> 5 LOAD_CONST (False) + 6 POP_JUMP_IF_FALSE (9) + 7 LOAD_CONST (True) + 8 POP_JUMP_IF_FALSE (9) - 2 >> 8 RETURN_CONST (None) + 2 >> 9 LOAD_CONST (None) + 10 RETURN_VALUE diff --git a/crates/codegen/src/snapshots/rustpython_codegen__compile__tests__if_ors.snap b/crates/codegen/src/snapshots/rustpython_codegen__compile__tests__if_ors.snap index 37a862cc65a..e1d41377db7 100644 --- a/crates/codegen/src/snapshots/rustpython_codegen__compile__tests__if_ors.snap +++ b/crates/codegen/src/snapshots/rustpython_codegen__compile__tests__if_ors.snap @@ -2,11 +2,13 @@ source: crates/codegen/src/compile.rs expression: "compile_exec(\"\\\nif True or False or False:\n pass\n\")" --- - 1 0 LOAD_CONST (True) - 1 POP_JUMP_IF_TRUE (6) - 2 LOAD_CONST (False) - 3 POP_JUMP_IF_TRUE (6) - 4 LOAD_CONST (False) - 5 POP_JUMP_IF_FALSE (6) + 1 0 RESUME (0) + 1 LOAD_CONST (True) + 2 POP_JUMP_IF_TRUE (7) + 3 LOAD_CONST (False) + 4 POP_JUMP_IF_TRUE (7) + 5 LOAD_CONST (False) + 6 POP_JUMP_IF_FALSE (7) - 2 >> 6 RETURN_CONST (None) + 2 >> 7 LOAD_CONST (None) + 8 RETURN_VALUE diff --git a/crates/codegen/src/snapshots/rustpython_codegen__compile__tests__nested_double_async_with.snap b/crates/codegen/src/snapshots/rustpython_codegen__compile__tests__nested_double_async_with.snap index 679c5ebade4..2a482f3ecb9 100644 --- a/crates/codegen/src/snapshots/rustpython_codegen__compile__tests__nested_double_async_with.snap +++ b/crates/codegen/src/snapshots/rustpython_codegen__compile__tests__nested_double_async_with.snap @@ -2,140 +2,164 @@ source: crates/codegen/src/compile.rs expression: "compile_exec(\"\\\nasync def test():\n for stop_exc in (StopIteration('spam'), StopAsyncIteration('ham')):\n with self.subTest(type=type(stop_exc)):\n try:\n async with egg():\n raise stop_exc\n except Exception as ex:\n self.assertIs(ex, stop_exc)\n else:\n self.fail(f'{stop_exc} was suppressed')\n\")" --- - 3 0 LOAD_CONST (<code object test at ??? file "source_path", line 1>): 1 0 RESUME (0) + 1 0 RESUME (0) - 2 1 LOAD_GLOBAL (0, StopIteration) - 2 LOAD_CONST ("spam") - 3 CALL (1) - 4 LOAD_GLOBAL (1, StopAsyncIteration) - 5 LOAD_CONST ("ham") + 3 1 LOAD_CONST (<code object test at ??? file "source_path", line 1>): 1 0 RETURN_GENERATOR + 1 POP_TOP + 2 RESUME (0) + + 2 3 LOAD_GLOBAL (0, StopIteration) + 4 PUSH_NULL + 5 LOAD_CONST ("spam") 6 CALL (1) - 7 BUILD_TUPLE (2) - 8 GET_ITER - >> 9 FOR_ITER (123) - 10 STORE_FAST (0, stop_exc) + 7 LOAD_GLOBAL (1, StopAsyncIteration) + 8 PUSH_NULL + 9 LOAD_CONST ("ham") + 10 CALL (1) + 11 BUILD_TUPLE (2) + 12 GET_ITER + >> 13 FOR_ITER (141) + 14 STORE_FAST (0, stop_exc) - 3 11 LOAD_GLOBAL (2, self) - 12 LOAD_METHOD (3, subTest) - 13 LOAD_GLOBAL (4, type) - 14 LOAD_FAST (0, stop_exc) - 15 CALL (1) - 16 LOAD_CONST (("type")) - 17 CALL_METHOD_KW (1) - 18 BEFORE_WITH - 19 POP_TOP + 3 15 LOAD_GLOBAL (2, self) + 16 LOAD_ATTR (7, subTest, method=true) + 17 LOAD_GLOBAL (4, type) + 18 PUSH_NULL + 19 LOAD_FAST (0, stop_exc) + 20 CALL (1) + 21 LOAD_CONST (("type")) + 22 CALL_KW (1) + 23 COPY (1) + 24 LOAD_SPECIAL (__exit__) + 25 SWAP (2) + 26 LOAD_SPECIAL (__enter__) + 27 PUSH_NULL + 28 CALL (0) + 29 POP_TOP - 5 20 LOAD_GLOBAL (5, egg) - 21 CALL (0) - 22 BEFORE_ASYNC_WITH - 23 GET_AWAITABLE - 24 LOAD_CONST (None) - >> 25 SEND (30) - 26 YIELD_VALUE (1) - 27 RESUME (3) - 28 JUMP (25) - 29 CLEANUP_THROW - >> 30 END_SEND - 31 POP_TOP + 5 30 LOAD_GLOBAL (5, egg) + 31 PUSH_NULL + 32 CALL (0) + 33 COPY (1) + 34 LOAD_SPECIAL (__aexit__) + 35 SWAP (2) + 36 LOAD_SPECIAL (__aenter__) + 37 PUSH_NULL + 38 CALL (0) + 39 GET_AWAITABLE (1) + 40 LOAD_CONST (None) + >> 41 SEND (46) + 42 YIELD_VALUE (1) + 43 RESUME (3) + 44 JUMP_BACKWARD_NO_INTERRUPT(41) + 45 CLEANUP_THROW + >> 46 END_SEND + 47 POP_TOP - 6 32 LOAD_FAST (0, stop_exc) - 33 RAISE_VARARGS (Raise) + 6 48 LOAD_FAST (0, stop_exc) + 49 RAISE_VARARGS (Raise) - 5 34 LOAD_CONST (None) - 35 LOAD_CONST (None) - 36 LOAD_CONST (None) - 37 CALL (3) - 38 GET_AWAITABLE - 39 LOAD_CONST (None) - >> 40 SEND (45) - 41 YIELD_VALUE (1) - 42 RESUME (3) - 43 JUMP (40) - 44 CLEANUP_THROW - >> 45 END_SEND - 46 POP_TOP - 47 JUMP (69) - 48 PUSH_EXC_INFO - 49 WITH_EXCEPT_START - 50 GET_AWAITABLE + 5 50 PUSH_NULL 51 LOAD_CONST (None) - >> 52 SEND (57) - 53 YIELD_VALUE (1) - 54 RESUME (3) - 55 JUMP (52) - 56 CLEANUP_THROW - >> 57 END_SEND - 58 TO_BOOL - 59 POP_JUMP_IF_TRUE (61) - 60 RERAISE (2) - >> 61 POP_TOP - 62 POP_EXCEPT + 52 LOAD_CONST (None) + 53 LOAD_CONST (None) + 54 CALL (3) + 55 GET_AWAITABLE (2) + 56 LOAD_CONST (None) + >> 57 SEND (62) + 58 YIELD_VALUE (1) + 59 RESUME (3) + 60 JUMP_BACKWARD_NO_INTERRUPT(57) + 61 CLEANUP_THROW + >> 62 END_SEND 63 POP_TOP - 64 POP_TOP - 65 JUMP (69) - 66 COPY (3) - 67 POP_EXCEPT - 68 RERAISE (1) - >> 69 JUMP (95) - 70 PUSH_EXC_INFO - 71 COPY (1) - - 7 72 LOAD_GLOBAL (6, Exception) - 73 JUMP_IF_NOT_EXC_MATCH(91) - 74 STORE_FAST (1, ex) - - 8 75 LOAD_GLOBAL (2, self) - 76 LOAD_METHOD (7, assertIs) - 77 LOAD_FAST (1, ex) - 78 LOAD_FAST (0, stop_exc) - 79 CALL_METHOD (2) + 64 JUMP_FORWARD (86) + 65 PUSH_EXC_INFO + 66 WITH_EXCEPT_START + 67 GET_AWAITABLE (2) + 68 LOAD_CONST (None) + >> 69 SEND (74) + 70 YIELD_VALUE (1) + 71 RESUME (3) + 72 JUMP_BACKWARD_NO_INTERRUPT(69) + 73 CLEANUP_THROW + >> 74 END_SEND + 75 TO_BOOL + 76 POP_JUMP_IF_TRUE (78) + 77 RERAISE (2) + >> 78 POP_TOP + 79 POP_EXCEPT 80 POP_TOP - 81 JUMP (86) - 82 LOAD_CONST (None) - 83 STORE_FAST (1, ex) - 84 DELETE_FAST (1, ex) - 85 RAISE_VARARGS (ReraiseFromStack) - >> 86 POP_EXCEPT - 87 LOAD_CONST (None) - 88 STORE_FAST (1, ex) - 89 DELETE_FAST (1, ex) - 90 JUMP (103) - >> 91 RAISE_VARARGS (ReraiseFromStack) - 92 COPY (3) - 93 POP_EXCEPT - 94 RAISE_VARARGS (ReraiseFromStack) + 81 POP_TOP + 82 JUMP_FORWARD (86) + 83 COPY (3) + 84 POP_EXCEPT + 85 RERAISE (1) + >> 86 JUMP_FORWARD (112) + 87 PUSH_EXC_INFO - 10 >> 95 LOAD_GLOBAL (2, self) - 96 LOAD_METHOD (8, fail) - 97 LOAD_FAST (0, stop_exc) - 98 FORMAT_SIMPLE - 99 LOAD_CONST (" was suppressed") - 100 BUILD_STRING (2) - 101 CALL_METHOD (1) - 102 POP_TOP + 7 88 LOAD_GLOBAL (6, Exception) + 89 CHECK_EXC_MATCH + 90 POP_JUMP_IF_FALSE (108) + 91 STORE_FAST (1, ex) - 3 >> 103 LOAD_CONST (None) + 8 92 LOAD_GLOBAL (2, self) + 93 LOAD_ATTR (15, assertIs, method=true) + 94 LOAD_FAST (1, ex) + 95 LOAD_FAST (0, stop_exc) + 96 CALL (2) + 97 POP_TOP + 98 JUMP_FORWARD (103) + 99 LOAD_CONST (None) + 100 STORE_FAST (1, ex) + 101 DELETE_FAST (1, ex) + 102 RAISE_VARARGS (ReraiseFromStack) + >> 103 POP_EXCEPT 104 LOAD_CONST (None) - 105 LOAD_CONST (None) - 106 CALL (3) - 107 POP_TOP - 108 JUMP (122) - 109 PUSH_EXC_INFO - 110 WITH_EXCEPT_START - 111 TO_BOOL - 112 POP_JUMP_IF_TRUE (114) - 113 RERAISE (2) - >> 114 POP_TOP - 115 POP_EXCEPT - 116 POP_TOP - 117 POP_TOP - 118 JUMP (122) - 119 COPY (3) - 120 POP_EXCEPT - 121 RERAISE (1) - >> 122 JUMP (9) - >> 123 RETURN_CONST (None) + 105 STORE_FAST (1, ex) + 106 DELETE_FAST (1, ex) + 107 JUMP_FORWARD (120) + >> 108 RAISE_VARARGS (ReraiseFromStack) + 109 COPY (3) + 110 POP_EXCEPT + 111 RAISE_VARARGS (ReraiseFromStack) + + 10 >> 112 LOAD_GLOBAL (2, self) + 113 LOAD_ATTR (17, fail, method=true) + 114 LOAD_FAST_BORROW (0, stop_exc) + 115 FORMAT_SIMPLE + 116 LOAD_CONST (" was suppressed") + 117 BUILD_STRING (2) + 118 CALL (1) + 119 POP_TOP + + 3 >> 120 PUSH_NULL + 121 LOAD_CONST (None) + 122 LOAD_CONST (None) + 123 LOAD_CONST (None) + 124 CALL (3) + 125 POP_TOP + 126 JUMP_FORWARD (140) + 127 PUSH_EXC_INFO + 128 WITH_EXCEPT_START + 129 TO_BOOL + 130 POP_JUMP_IF_TRUE (132) + 131 RERAISE (2) + >> 132 POP_TOP + 133 POP_EXCEPT + 134 POP_TOP + 135 POP_TOP + 136 JUMP_FORWARD (140) + 137 COPY (3) + 138 POP_EXCEPT + 139 RERAISE (1) + >> 140 JUMP_BACKWARD (13) + >> 141 END_FOR + 142 POP_ITER + 143 LOAD_CONST (None) + 144 RETURN_VALUE - 1 MAKE_FUNCTION - 2 STORE_NAME (0, test) - 3 RETURN_CONST (None) + 2 MAKE_FUNCTION + 3 STORE_NAME (0, test) + 4 LOAD_CONST (None) + 5 RETURN_VALUE diff --git a/crates/codegen/src/string_parser.rs b/crates/codegen/src/string_parser.rs index 175e75c1a26..7e1558d2b17 100644 --- a/crates/codegen/src/string_parser.rs +++ b/crates/codegen/src/string_parser.rs @@ -7,7 +7,7 @@ use core::convert::Infallible; -use ruff_python_ast::{AnyStringFlags, StringFlags}; +use ruff_python_ast::{self as ast, StringFlags as _}; use rustpython_wtf8::{CodePoint, Wtf8, Wtf8Buf}; // use ruff_python_parser::{LexicalError, LexicalErrorType}; @@ -24,11 +24,11 @@ struct StringParser { /// Current position of the parser in the source. cursor: usize, /// Flags that can be used to query information about the string. - flags: AnyStringFlags, + flags: ast::AnyStringFlags, } impl StringParser { - const fn new(source: Box<str>, flags: AnyStringFlags) -> Self { + const fn new(source: Box<str>, flags: ast::AnyStringFlags) -> Self { Self { source, cursor: 0, @@ -272,7 +272,7 @@ impl StringParser { } } -pub(crate) fn parse_string_literal(source: &str, flags: AnyStringFlags) -> Box<Wtf8> { +pub(crate) fn parse_string_literal(source: &str, flags: ast::AnyStringFlags) -> Box<Wtf8> { let source = &source[flags.opener_len().to_usize()..]; let source = &source[..source.len() - flags.quote_len().to_usize()]; StringParser::new(source.into(), flags) @@ -280,7 +280,10 @@ pub(crate) fn parse_string_literal(source: &str, flags: AnyStringFlags) -> Box<W .unwrap_or_else(|x| match x {}) } -pub(crate) fn parse_fstring_literal_element(source: Box<str>, flags: AnyStringFlags) -> Box<Wtf8> { +pub(crate) fn parse_fstring_literal_element( + source: Box<str>, + flags: ast::AnyStringFlags, +) -> Box<Wtf8> { StringParser::new(source, flags) .parse_fstring_middle() .unwrap_or_else(|x| match x {}) diff --git a/crates/codegen/src/symboltable.rs b/crates/codegen/src/symboltable.rs index fe4976bb086..2b243262b3c 100644 --- a/crates/codegen/src/symboltable.rs +++ b/crates/codegen/src/symboltable.rs @@ -13,14 +13,10 @@ use crate::{ }; use alloc::{borrow::Cow, fmt}; use bitflags::bitflags; -use ruff_python_ast::{ - self as ast, Comprehension, Decorator, Expr, Identifier, ModExpression, ModModule, Parameter, - ParameterWithDefault, Parameters, Pattern, PatternMatchAs, PatternMatchClass, - PatternMatchMapping, PatternMatchOr, PatternMatchSequence, PatternMatchStar, PatternMatchValue, - Stmt, TypeParam, TypeParamParamSpec, TypeParamTypeVar, TypeParamTypeVarTuple, TypeParams, -}; +use ruff_python_ast as ast; use ruff_text_size::{Ranged, TextRange}; use rustpython_compiler_core::{PositionEncoding, SourceFile, SourceLocation}; +use std::collections::HashSet; /// Captures all symbols in the current scope, and has a list of sub-scopes in this scope. #[derive(Clone)] @@ -44,6 +40,9 @@ pub struct SymbolTable { /// AST nodes. pub sub_tables: Vec<SymbolTable>, + /// Cursor pointing to the next sub-table to consume during compilation. + pub next_sub_table: usize, + /// Variable names in definition order (parameters first, then locals) pub varnames: Vec<String>, @@ -59,6 +58,17 @@ pub struct SymbolTable { /// Whether this comprehension scope should be inlined (PEP 709) /// True for list/set/dict comprehensions in non-generator expressions pub comp_inlined: bool, + + /// PEP 649: Reference to annotation scope for this block + /// Annotations are compiled as a separate `__annotate__` function + pub annotation_block: Option<Box<SymbolTable>>, + + /// PEP 649: Whether this scope has conditional annotations + /// (annotations inside if/for/while/etc. blocks or at module level) + pub has_conditional_annotations: bool, + + /// Whether `from __future__ import annotations` is active + pub future_annotations: bool, } impl SymbolTable { @@ -70,21 +80,31 @@ impl SymbolTable { is_nested, symbols: IndexMap::default(), sub_tables: vec![], + next_sub_table: 0, varnames: Vec::new(), needs_class_closure: false, needs_classdict: false, can_see_class_scope: false, comp_inlined: false, + annotation_block: None, + has_conditional_annotations: false, + future_annotations: false, } } - pub fn scan_program(program: &ModModule, source_file: SourceFile) -> SymbolTableResult<Self> { + pub fn scan_program( + program: &ast::ModModule, + source_file: SourceFile, + ) -> SymbolTableResult<Self> { let mut builder = SymbolTableBuilder::new(source_file); builder.scan_statements(program.body.as_ref())?; builder.finish() } - pub fn scan_expr(expr: &ModExpression, source_file: SourceFile) -> SymbolTableResult<Self> { + pub fn scan_expr( + expr: &ast::ModExpression, + source_file: SourceFile, + ) -> SymbolTableResult<Self> { let mut builder = SymbolTableBuilder::new(source_file); builder.scan_expression(expr.body.as_ref(), ExpressionContext::Load)?; builder.finish() @@ -104,6 +124,8 @@ pub enum CompilerScope { Lambda, Comprehension, TypeParams, + /// PEP 649: Annotation scope for deferred evaluation + Annotation, } impl fmt::Display for CompilerScope { @@ -116,9 +138,8 @@ impl fmt::Display for CompilerScope { Self::Lambda => write!(f, "lambda"), Self::Comprehension => write!(f, "comprehension"), Self::TypeParams => write!(f, "type parameter"), + Self::Annotation => write!(f, "annotation"), // TODO missing types from the C implementation - // if self._table.type == _symtable.TYPE_ANNOTATION: - // return "annotation" // if self._table.type == _symtable.TYPE_TYPE_VAR_BOUND: // return "TypeVar bound" // if self._table.type == _symtable.TYPE_TYPE_ALIAS: @@ -240,30 +261,36 @@ impl core::fmt::Debug for SymbolTable { */ fn analyze_symbol_table(symbol_table: &mut SymbolTable) -> SymbolTableResult { let mut analyzer = SymbolTableAnalyzer::default(); - analyzer.analyze_symbol_table(symbol_table) + // Discard the newfree set at the top level - it's only needed for propagation + // Pass None for class_entry at top level + let _newfree = analyzer.analyze_symbol_table(symbol_table, None)?; + Ok(()) } /* Drop __class__ and __classdict__ from free variables in class scope and set the appropriate flags. Equivalent to CPython's drop_class_free(). See: https://github.com/python/cpython/blob/main/Python/symtable.c#L884 + + This function removes __class__ and __classdict__ from the + `newfree` set (which contains free variables collected from all child scopes) + and sets the corresponding flags on the class's symbol table entry. */ -fn drop_class_free(symbol_table: &mut SymbolTable) { - // Check if __class__ is used as a free variable - if let Some(class_symbol) = symbol_table.symbols.get("__class__") - && class_symbol.scope == SymbolScope::Free - { +fn drop_class_free(symbol_table: &mut SymbolTable, newfree: &mut HashSet<String>) { + // Check if __class__ is in the free variables collected from children + // If found, it means a child scope (method) references __class__ + if newfree.remove("__class__") { symbol_table.needs_class_closure = true; - // Note: In CPython, the symbol is removed from the free set, - // but in RustPython we handle this differently during code generation } - // Check if __classdict__ is used as a free variable - if let Some(classdict_symbol) = symbol_table.symbols.get("__classdict__") - && classdict_symbol.scope == SymbolScope::Free - { + // Check if __classdict__ is in the free variables collected from children + if newfree.remove("__classdict__") { symbol_table.needs_classdict = true; - // Note: In CPython, the symbol is removed from the free set, - // but in RustPython we handle this differently during code generation + } + + // Check if __conditional_annotations__ is in the free variables collected from children + // Remove it from free set - it's handled specially in class scope + if newfree.remove("__conditional_annotations__") { + symbol_table.has_conditional_annotations = true; } } @@ -333,16 +360,56 @@ struct SymbolTableAnalyzer { } impl SymbolTableAnalyzer { - fn analyze_symbol_table(&mut self, symbol_table: &mut SymbolTable) -> SymbolTableResult { + /// Analyze a symbol table and return the set of free variables. + /// See symtable.c analyze_block(). + /// class_entry: PEP 649 - enclosing class symbols for annotation scopes + fn analyze_symbol_table( + &mut self, + symbol_table: &mut SymbolTable, + class_entry: Option<&SymbolMap>, + ) -> SymbolTableResult<HashSet<String>> { let symbols = core::mem::take(&mut symbol_table.symbols); let sub_tables = &mut *symbol_table.sub_tables; + // Collect free variables from all child scopes + let mut newfree = HashSet::new(); + + let annotation_block = &mut symbol_table.annotation_block; + + // PEP 649: Determine class_entry to pass to children + // If current scope is a class with annotation block that can_see_class_scope, + // we need to pass class symbols to the annotation scope + let is_class = symbol_table.typ == CompilerScope::Class; + + // Clone class symbols if needed for annotation scope (to avoid borrow conflict) + let class_symbols_for_ann = if is_class + && annotation_block + .as_ref() + .is_some_and(|b| b.can_see_class_scope) + { + Some(symbols.clone()) + } else { + None + }; + let mut info = (symbols, symbol_table.typ); self.tables.with_append(&mut info, |list| { let inner_scope = unsafe { &mut *(list as *mut _ as *mut Self) }; - // Analyze sub scopes: + // Analyze sub scopes and collect their free variables for sub_table in sub_tables.iter_mut() { - inner_scope.analyze_symbol_table(sub_table)?; + // Sub-scopes (functions, nested classes) don't inherit class_entry + let child_free = inner_scope.analyze_symbol_table(sub_table, None)?; + // Propagate child's free variables to this scope + newfree.extend(child_free); + } + // PEP 649: Analyze annotation block if present + if let Some(annotation_table) = annotation_block { + // Pass class symbols to annotation scope if can_see_class_scope + let ann_class_entry = class_symbols_for_ann.as_ref().or(class_entry); + let child_free = + inner_scope.analyze_symbol_table(annotation_table, ann_class_entry)?; + // Propagate annotation's free variables to this scope + newfree.extend(child_free); } Ok(()) })?; @@ -380,17 +447,25 @@ impl SymbolTableAnalyzer { } } - // Analyze symbols: + // Analyze symbols in current scope for symbol in symbol_table.symbols.values_mut() { - self.analyze_symbol(symbol, symbol_table.typ, sub_tables)?; + self.analyze_symbol(symbol, symbol_table.typ, sub_tables, class_entry)?; + + // Collect free variables from this scope + // These will be propagated to the parent scope + if symbol.scope == SymbolScope::Free || symbol.flags.contains(SymbolFlags::FREE_CLASS) { + newfree.insert(symbol.name.clone()); + } } - // Handle class-specific implicit cells (like CPython) + // Handle class-specific implicit cells + // This removes __class__ and __classdict__ from newfree if present + // and sets the corresponding flags on the symbol table if symbol_table.typ == CompilerScope::Class { - drop_class_free(symbol_table); + drop_class_free(symbol_table, &mut newfree); } - Ok(()) + Ok(newfree) } fn analyze_symbol( @@ -398,6 +473,7 @@ impl SymbolTableAnalyzer { symbol: &mut Symbol, st_typ: CompilerScope, sub_tables: &[SymbolTable], + class_entry: Option<&SymbolMap>, ) -> SymbolTableResult { if symbol .flags @@ -417,7 +493,8 @@ impl SymbolTableAnalyzer { // check if the name is already defined in any outer scope // therefore if scope_depth < 2 - || self.found_in_outer_scope(&symbol.name) != Some(SymbolScope::Free) + || self.found_in_outer_scope(&symbol.name, st_typ) + != Some(SymbolScope::Free) { return Err(SymbolTableError { error: format!("no binding for nonlocal '{}' found", symbol.name), @@ -443,11 +520,23 @@ impl SymbolTableAnalyzer { // all is well } SymbolScope::Unknown => { + // PEP 649: Check class_entry first (like analyze_name) + // If name is bound in enclosing class, mark as GlobalImplicit + if let Some(class_symbols) = class_entry + && let Some(class_sym) = class_symbols.get(&symbol.name) + { + // DEF_BOUND && !DEF_NONLOCAL -> GLOBAL_IMPLICIT + if class_sym.is_bound() && class_sym.scope != SymbolScope::Free { + symbol.scope = SymbolScope::GlobalImplicit; + return Ok(()); + } + } + // Try hard to figure out what the scope of this symbol is. let scope = if symbol.is_bound() { self.found_in_inner_scope(sub_tables, &symbol.name, st_typ) .unwrap_or(SymbolScope::Local) - } else if let Some(scope) = self.found_in_outer_scope(&symbol.name) { + } else if let Some(scope) = self.found_in_outer_scope(&symbol.name, st_typ) { scope } else if self.tables.is_empty() { // Don't make assumptions when we don't know. @@ -463,14 +552,44 @@ impl SymbolTableAnalyzer { Ok(()) } - fn found_in_outer_scope(&mut self, name: &str) -> Option<SymbolScope> { + fn found_in_outer_scope(&mut self, name: &str, st_typ: CompilerScope) -> Option<SymbolScope> { let mut decl_depth = None; for (i, (symbols, typ)) in self.tables.iter().rev().enumerate() { if matches!(typ, CompilerScope::Module) - || matches!(typ, CompilerScope::Class if name != "__class__") + || matches!(typ, CompilerScope::Class if name != "__class__" && name != "__classdict__" && name != "__conditional_annotations__") { continue; } + + // PEP 649: Annotation scope is conceptually a sibling of the function, + // not a child. Skip the immediate parent function scope when looking + // for outer variables from annotation scope. + if st_typ == CompilerScope::Annotation + && i == 0 + && matches!( + typ, + CompilerScope::Function | CompilerScope::AsyncFunction | CompilerScope::Lambda + ) + { + continue; + } + + // __class__ and __classdict__ are implicitly declared in class scope + // This handles the case where nested scopes reference them + if (name == "__class__" || name == "__classdict__") + && matches!(typ, CompilerScope::Class) + { + decl_depth = Some(i); + break; + } + + // __conditional_annotations__ is implicitly declared in class scope + // for classes with conditional annotations + if name == "__conditional_annotations__" && matches!(typ, CompilerScope::Class) { + decl_depth = Some(i); + break; + } + if let Some(sym) = symbols.get(name) { match sym.scope { SymbolScope::GlobalExplicit => return Some(SymbolScope::GlobalExplicit), @@ -627,12 +746,19 @@ impl SymbolTableAnalyzer { location: None, }); } + CompilerScope::Annotation => { + // Named expression is not allowed in annotation scope + return Err(SymbolTableError { + error: "named expression cannot be used within an annotation".to_string(), + location: None, + }); + } } Ok(()) } } -#[derive(Debug, Clone)] +#[derive(Clone, Copy, Debug)] enum SymbolUsage { Global, Nonlocal, @@ -655,6 +781,8 @@ struct SymbolTableBuilder { source_file: SourceFile, // Current scope's varnames being collected (temporary storage) current_varnames: Vec<String>, + // Stack to preserve parent varnames when entering nested scopes + varnames_stack: Vec<Vec<String>>, // Track if we're inside an iterable definition expression (for nested comprehensions) in_iter_def_exp: bool, // Track if we're inside an annotation (yield/await/named expr not allowed) @@ -665,6 +793,8 @@ struct SymbolTableBuilder { in_comp_inner_loop_target: bool, // Scope info for error messages (e.g., "a TypeVar bound") scope_info: Option<&'static str>, + // PEP 649: Track if we're inside a conditional block (if/for/while/etc.) + in_conditional_block: bool, } /// Enum to indicate in what mode an expression @@ -688,11 +818,13 @@ impl SymbolTableBuilder { future_annotations: false, source_file, current_varnames: Vec::new(), + varnames_stack: Vec::new(), in_iter_def_exp: false, in_annotation: false, in_type_alias: false, in_comp_inner_loop_target: false, scope_info: None, + in_conditional_block: false, }; this.enter_scope("top", CompilerScope::Module, 0); this @@ -703,6 +835,8 @@ impl SymbolTableBuilder { let mut symbol_table = self.tables.pop().unwrap(); // Save varnames for the top-level module scope symbol_table.varnames = self.current_varnames; + // Propagate future_annotations to the symbol table + symbol_table.future_annotations = self.future_annotations; analyze_symbol_table(&mut symbol_table)?; Ok(symbol_table) } @@ -711,12 +845,19 @@ impl SymbolTableBuilder { let is_nested = self .tables .last() - .map(|table| table.is_nested || table.typ == CompilerScope::Function) + .map(|table| { + table.is_nested + || matches!( + table.typ, + CompilerScope::Function | CompilerScope::AsyncFunction + ) + }) .unwrap_or(false); let table = SymbolTable::new(name.to_owned(), typ, line_number, is_nested); self.tables.push(table); - // Clear current_varnames for the new scope - self.current_varnames.clear(); + // Save parent's varnames and start fresh for the new scope + self.varnames_stack + .push(core::mem::take(&mut self.current_varnames)); } fn enter_type_param_block(&mut self, name: &str, line_number: u32) -> SymbolTableResult { @@ -750,6 +891,86 @@ impl SymbolTableBuilder { // Save the collected varnames to the symbol table table.varnames = core::mem::take(&mut self.current_varnames); self.tables.last_mut().unwrap().sub_tables.push(table); + // Restore parent's varnames + self.current_varnames = self.varnames_stack.pop().unwrap_or_default(); + } + + /// Enter annotation scope (PEP 649) + /// Creates or reuses the annotation block for the current scope + fn enter_annotation_scope(&mut self, line_number: u32) { + let current = self.tables.last_mut().unwrap(); + let can_see_class_scope = current.typ == CompilerScope::Class; + let has_conditional = current.has_conditional_annotations; + + // Create annotation block if not exists + if current.annotation_block.is_none() { + let mut annotation_table = SymbolTable::new( + "__annotate__".to_owned(), + CompilerScope::Annotation, + line_number, + true, // is_nested + ); + // Annotation scope in class can see class scope + annotation_table.can_see_class_scope = can_see_class_scope; + // Add 'format' parameter + annotation_table.varnames.push("format".to_owned()); + current.annotation_block = Some(Box::new(annotation_table)); + } + + // Take the annotation block and push to stack for processing + let annotation_table = current.annotation_block.take().unwrap(); + self.tables.push(*annotation_table); + // Save parent's varnames and seed with existing annotation varnames (e.g., "format") + self.varnames_stack + .push(core::mem::take(&mut self.current_varnames)); + self.current_varnames = self.tables.last().unwrap().varnames.clone(); + + if can_see_class_scope && !self.future_annotations { + self.add_classdict_freevar(); + // Also add __conditional_annotations__ as free var if parent has conditional annotations + if has_conditional { + self.add_conditional_annotations_freevar(); + } + } + } + + /// Leave annotation scope (PEP 649) + /// Stores the annotation block back to parent instead of sub_tables + fn leave_annotation_scope(&mut self) { + let mut table = self.tables.pop().unwrap(); + // Save the collected varnames to the symbol table + table.varnames = core::mem::take(&mut self.current_varnames); + // Store back to parent's annotation_block (not sub_tables) + let parent = self.tables.last_mut().unwrap(); + parent.annotation_block = Some(Box::new(table)); + // Restore parent's varnames + self.current_varnames = self.varnames_stack.pop().unwrap_or_default(); + } + + fn add_classdict_freevar(&mut self) { + let table = self.tables.last_mut().unwrap(); + let name = "__classdict__"; + let symbol = table + .symbols + .entry(name.to_owned()) + .or_insert_with(|| Symbol::new(name)); + symbol.scope = SymbolScope::Free; + symbol + .flags + .insert(SymbolFlags::REFERENCED | SymbolFlags::FREE_CLASS); + } + + fn add_conditional_annotations_freevar(&mut self) { + let table = self.tables.last_mut().unwrap(); + let name = "__conditional_annotations__"; + let symbol = table + .symbols + .entry(name.to_owned()) + .or_insert_with(|| Symbol::new(name)); + symbol.scope = SymbolScope::Free; + symbol + .flags + .insert(SymbolFlags::REFERENCED | SymbolFlags::FREE_CLASS); } fn line_index_start(&self, range: TextRange) -> u32 { @@ -759,21 +980,27 @@ impl SymbolTableBuilder { .get() as _ } - fn scan_statements(&mut self, statements: &[Stmt]) -> SymbolTableResult { + fn scan_statements(&mut self, statements: &[ast::Stmt]) -> SymbolTableResult { for statement in statements { self.scan_statement(statement)?; } Ok(()) } - fn scan_parameters(&mut self, parameters: &[ParameterWithDefault]) -> SymbolTableResult { + fn scan_parameters(&mut self, parameters: &[ast::ParameterWithDefault]) -> SymbolTableResult { for parameter in parameters { self.scan_parameter(&parameter.parameter)?; } Ok(()) } - fn scan_parameter(&mut self, parameter: &Parameter) -> SymbolTableResult { + fn scan_parameter(&mut self, parameter: &ast::Parameter) -> SymbolTableResult { + self.check_name( + parameter.name.as_str(), + ExpressionContext::Store, + parameter.name.range, + )?; + let usage = if parameter.annotation.is_some() { SymbolUsage::AnnotationParameter } else { @@ -799,20 +1026,63 @@ impl SymbolTableBuilder { self.register_ident(&parameter.name, usage) } - fn scan_annotation(&mut self, annotation: &Expr) -> SymbolTableResult { + fn scan_annotation(&mut self, annotation: &ast::Expr) -> SymbolTableResult { + let current_scope = self.tables.last().map(|t| t.typ); + + // PEP 649: Check if this is a conditional annotation + // Module-level: always conditional (module may be partially executed) + // Class-level: conditional only when inside if/for/while/etc. + if !self.future_annotations { + let is_conditional = matches!(current_scope, Some(CompilerScope::Module)) + || (matches!(current_scope, Some(CompilerScope::Class)) + && self.in_conditional_block); + + if is_conditional && !self.tables.last().unwrap().has_conditional_annotations { + self.tables.last_mut().unwrap().has_conditional_annotations = true; + // Register __conditional_annotations__ symbol in the scope (USE flag, not DEF) + self.register_name( + "__conditional_annotations__", + SymbolUsage::Used, + annotation.range(), + )?; + } + } + + // Create annotation scope for deferred evaluation + let line_number = self.line_index_start(annotation.range()); + self.enter_annotation_scope(line_number); + if self.future_annotations { - Ok(()) - } else { + // PEP 563: annotations are stringified at compile time + // Don't scan expression - symbols would fail to resolve + // Just create the annotation_block structure + self.leave_annotation_scope(); + return Ok(()); + } + + // PEP 649: scan expression for symbol references + // Class annotations are evaluated in class locals (not module globals) + let was_in_annotation = self.in_annotation; + self.in_annotation = true; + let result = self.scan_expression(annotation, ExpressionContext::Load); + self.in_annotation = was_in_annotation; + + self.leave_annotation_scope(); + + // Module scope: re-scan to register symbols (builtins like str, int) + // Class scope: do NOT re-scan to preserve class-local symbol resolution + if matches!(current_scope, Some(CompilerScope::Module)) { let was_in_annotation = self.in_annotation; self.in_annotation = true; - let result = self.scan_expression(annotation, ExpressionContext::Load); + let _ = self.scan_expression(annotation, ExpressionContext::Load); self.in_annotation = was_in_annotation; - result } + + result } - fn scan_statement(&mut self, statement: &Stmt) -> SymbolTableResult { - use ruff_python_ast::*; + fn scan_statement(&mut self, statement: &ast::Stmt) -> SymbolTableResult { + use ast::*; if let Stmt::ImportFrom(StmtImportFrom { module, names, .. }) = &statement && module.as_ref().map(|id| id.as_str()) == Some("__future__") { @@ -839,13 +1109,28 @@ impl SymbolTableBuilder { type_params, returns, range, + is_async, .. }) => { self.scan_decorators(decorator_list, ExpressionContext::Load)?; self.register_ident(name, SymbolUsage::Assigned)?; - if let Some(expression) = returns { - self.scan_annotation(expression)?; - } + + // Save the parent's annotation_block before scanning function annotations, + // so function annotations don't interfere with parent scope annotations. + // This applies to both class scope (methods) and module scope (top-level functions). + let parent_scope_typ = self.tables.last().map(|t| t.typ); + let should_save_annotation_block = matches!( + parent_scope_typ, + Some(CompilerScope::Class) | Some(CompilerScope::Module) + ); + let saved_annotation_block = if should_save_annotation_block { + self.tables.last_mut().unwrap().annotation_block.take() + } else { + None + }; + + // For generic functions, enter type_param block FIRST so that + // annotation scopes are nested inside and can see type parameters. if let Some(type_params) = type_params { self.enter_type_param_block( &format!("<generic parameters of {}>", name.as_str()), @@ -853,16 +1138,29 @@ impl SymbolTableBuilder { )?; self.scan_type_params(type_params)?; } + let has_return_annotation = if let Some(expression) = returns { + self.scan_annotation(expression)?; + true + } else { + false + }; self.enter_scope_with_parameters( name.as_str(), parameters, self.line_index_start(*range), + has_return_annotation, + *is_async, )?; self.scan_statements(body)?; self.leave_scope(); if type_params.is_some() { self.leave_scope(); } + + // Restore parent's annotation_block after processing the function + if let Some(block) = saved_annotation_block { + self.tables.last_mut().unwrap().annotation_block = Some(block); + } } Stmt::ClassDef(StmtClassDef { name, @@ -885,6 +1183,10 @@ impl SymbolTableBuilder { CompilerScope::Class, self.line_index_start(*range), ); + // Reset in_conditional_block for new class scope + // (each scope has its own conditional context) + let saved_in_conditional = self.in_conditional_block; + self.in_conditional_block = false; let prev_class = self.class_name.replace(name.to_string()); self.register_name("__module__", SymbolUsage::Assigned, *range)?; self.register_name("__qualname__", SymbolUsage::Assigned, *range)?; @@ -892,6 +1194,7 @@ impl SymbolTableBuilder { self.register_name("__class__", SymbolUsage::Assigned, *range)?; self.scan_statements(body)?; self.leave_scope(); + self.in_conditional_block = saved_in_conditional; self.class_name = prev_class; if let Some(arguments) = arguments { self.scan_expressions(&arguments.args, ExpressionContext::Load)?; @@ -915,6 +1218,9 @@ impl SymbolTableBuilder { .. }) => { self.scan_expression(test, ExpressionContext::Load)?; + // PEP 649: Track conditional block for annotations + let saved_in_conditional_block = self.in_conditional_block; + self.in_conditional_block = true; self.scan_statements(body)?; for elif in elif_else_clauses { if let Some(test) = &elif.test { @@ -922,6 +1228,7 @@ impl SymbolTableBuilder { } self.scan_statements(&elif.body)?; } + self.in_conditional_block = saved_in_conditional_block; } Stmt::For(StmtFor { target, @@ -932,15 +1239,23 @@ impl SymbolTableBuilder { }) => { self.scan_expression(target, ExpressionContext::Store)?; self.scan_expression(iter, ExpressionContext::Load)?; + // PEP 649: Track conditional block for annotations + let saved_in_conditional_block = self.in_conditional_block; + self.in_conditional_block = true; self.scan_statements(body)?; self.scan_statements(orelse)?; + self.in_conditional_block = saved_in_conditional_block; } Stmt::While(StmtWhile { test, body, orelse, .. }) => { self.scan_expression(test, ExpressionContext::Load)?; + // PEP 649: Track conditional block for annotations + let saved_in_conditional_block = self.in_conditional_block; + self.in_conditional_block = true; self.scan_statements(body)?; self.scan_statements(orelse)?; + self.in_conditional_block = saved_in_conditional_block; } Stmt::Break(_) | Stmt::Continue(_) | Stmt::Pass(_) => { // No symbols here. @@ -950,6 +1265,7 @@ impl SymbolTableBuilder { for name in names { if let Some(alias) = &name.asname { // `import my_module as my_alias` + self.check_name(alias.as_str(), ExpressionContext::Store, alias.range)?; self.register_ident(alias, SymbolUsage::Imported)?; } else if name.name.as_str() == "*" { // Star imports are only allowed at module level @@ -964,12 +1280,10 @@ impl SymbolTableBuilder { } // Don't register star imports as symbols } else { - // `import module` - self.register_name( - name.name.split('.').next().unwrap(), - SymbolUsage::Imported, - name.name.range, - )?; + // `import module` or `from x import name` + let imported_name = name.name.split('.').next().unwrap(); + self.check_name(imported_name, ExpressionContext::Store, name.name.range)?; + self.register_name(imported_name, SymbolUsage::Imported, name.name.range)?; } } } @@ -1006,7 +1320,26 @@ impl SymbolTableBuilder { // https://github.com/python/cpython/blob/main/Python/symtable.c#L1233 match &**target { Expr::Name(ast::ExprName { id, .. }) if *simple => { - self.register_name(id.as_str(), SymbolUsage::AnnotationAssigned, *range)?; + let id_str = id.as_str(); + + self.check_name(id_str, ExpressionContext::Store, *range)?; + + self.register_name(id_str, SymbolUsage::AnnotationAssigned, *range)?; + // PEP 649: Register annotate function in module/class scope + let current_scope = self.tables.last().map(|t| t.typ); + match current_scope { + Some(CompilerScope::Module) => { + self.register_name("__annotate__", SymbolUsage::Assigned, *range)?; + } + Some(CompilerScope::Class) => { + self.register_name( + "__annotate_func__", + SymbolUsage::Assigned, + *range, + )?; + } + _ => {} + } } _ => { self.scan_expression(target, ExpressionContext::Store)?; @@ -1024,7 +1357,11 @@ impl SymbolTableBuilder { self.scan_expression(expression, ExpressionContext::Store)?; } } + // PEP 649: Track conditional block for annotations + let saved_in_conditional_block = self.in_conditional_block; + self.in_conditional_block = true; self.scan_statements(body)?; + self.in_conditional_block = saved_in_conditional_block; } Stmt::Try(StmtTry { body, @@ -1033,6 +1370,9 @@ impl SymbolTableBuilder { finalbody, .. }) => { + // PEP 649: Track conditional block for annotations + let saved_in_conditional_block = self.in_conditional_block; + self.in_conditional_block = true; self.scan_statements(body)?; for handler in handlers { let ExceptHandler::ExceptHandler(ast::ExceptHandlerExceptHandler { @@ -1051,9 +1391,13 @@ impl SymbolTableBuilder { } self.scan_statements(orelse)?; self.scan_statements(finalbody)?; + self.in_conditional_block = saved_in_conditional_block; } Stmt::Match(StmtMatch { subject, cases, .. }) => { self.scan_expression(subject, ExpressionContext::Load)?; + // PEP 649: Track conditional block for annotations + let saved_in_conditional_block = self.in_conditional_block; + self.in_conditional_block = true; for case in cases { self.scan_pattern(&case.pattern)?; if let Some(guard) = &case.guard { @@ -1061,6 +1405,7 @@ impl SymbolTableBuilder { } self.scan_statements(&case.body)?; } + self.in_conditional_block = saved_in_conditional_block; } Stmt::Raise(StmtRaise { exc, cause, .. }) => { if let Some(expression) = exc { @@ -1099,7 +1444,7 @@ impl SymbolTableBuilder { fn scan_decorators( &mut self, - decorators: &[Decorator], + decorators: &[ast::Decorator], context: ExpressionContext, ) -> SymbolTableResult { for decorator in decorators { @@ -1110,7 +1455,7 @@ impl SymbolTableBuilder { fn scan_expressions( &mut self, - expressions: &[Expr], + expressions: &[ast::Expr], context: ExpressionContext, ) -> SymbolTableResult { for expression in expressions { @@ -1121,10 +1466,10 @@ impl SymbolTableBuilder { fn scan_expression( &mut self, - expression: &Expr, + expression: &ast::Expr, context: ExpressionContext, ) -> SymbolTableResult { - use ruff_python_ast::*; + use ast::*; // Check for expressions not allowed in certain contexts // (type parameters, annotations, type aliases, TypeVar bounds/defaults) @@ -1196,8 +1541,9 @@ impl SymbolTableBuilder { self.scan_expression(slice, ExpressionContext::Load)?; } Expr::Attribute(ExprAttribute { - value, range: _, .. + value, attr, range, .. }) => { + self.check_name(attr.as_str(), context, *range)?; self.scan_expression(value, ExpressionContext::Load)?; } Expr::Dict(ExprDict { @@ -1341,11 +1687,17 @@ impl SymbolTableBuilder { self.scan_expressions(&arguments.args, ExpressionContext::Load)?; for keyword in &arguments.keywords { + if let Some(arg) = &keyword.arg { + self.check_name(arg.as_str(), ExpressionContext::Store, keyword.range)?; + } self.scan_expression(&keyword.value, ExpressionContext::Load)?; } } Expr::Name(ExprName { id, range, .. }) => { let id = id.as_str(); + + self.check_name(id, context, *range)?; + // Determine the contextual usage of this symbol: match context { ExpressionContext::Delete => { @@ -1365,7 +1717,10 @@ impl SymbolTableBuilder { // Interesting stuff about the __class__ variable: // https://docs.python.org/3/reference/datamodel.html?highlight=__class__#creating-the-class-object if context == ExpressionContext::Load - && self.tables.last().unwrap().typ == CompilerScope::Function + && matches!( + self.tables.last().unwrap().typ, + CompilerScope::Function | CompilerScope::AsyncFunction + ) && id == "super" { self.register_name("__class__", SymbolUsage::Used, *range)?; @@ -1382,6 +1737,8 @@ impl SymbolTableBuilder { "lambda", parameters, self.line_index_start(expression.range()), + false, // lambdas have no return annotation + false, // lambdas are never async )?; } else { self.enter_scope( @@ -1411,14 +1768,19 @@ impl SymbolTableBuilder { } } Expr::TString(tstring) => { - return Err(SymbolTableError { - error: "not yet implemented".into(), - location: Some( - self.source_file - .to_source_code() - .source_location(tstring.range.start(), PositionEncoding::Utf8), - ), - }); + // Scan t-string interpolation expressions (similar to f-strings) + for expr in tstring + .value + .elements() + .filter_map(|x| x.as_interpolation()) + { + self.scan_expression(&expr.expression, ExpressionContext::Load)?; + if let Some(format_spec) = &expr.format_spec { + for element in format_spec.elements.interpolations() { + self.scan_expression(&element.expression, ExpressionContext::Load)? + } + } + } } // Constants Expr::StringLiteral(_) @@ -1463,6 +1825,7 @@ impl SymbolTableBuilder { // propagate inner names. if let Expr::Name(ExprName { id, .. }) = &**target { let id = id.as_str(); + self.check_name(id, ExpressionContext::Store, *range)?; let table = self.tables.last().unwrap(); if table.typ == CompilerScope::Comprehension { self.register_name( @@ -1487,9 +1850,9 @@ impl SymbolTableBuilder { fn scan_comprehension( &mut self, scope_name: &str, - elt1: &Expr, - elt2: Option<&Expr>, - generators: &[Comprehension], + elt1: &ast::Expr, + elt2: Option<&ast::Expr>, + generators: &[ast::Comprehension], range: TextRange, is_generator: bool, ) -> SymbolTableResult { @@ -1556,7 +1919,7 @@ impl SymbolTableBuilder { // = symtable_visit_type_param_bound_or_default fn scan_type_param_bound_or_default( &mut self, - expr: &Expr, + expr: &ast::Expr, scope_name: &str, scope_info: &'static str, ) -> SymbolTableResult { @@ -1582,14 +1945,14 @@ impl SymbolTableBuilder { result } - fn scan_type_params(&mut self, type_params: &TypeParams) -> SymbolTableResult { + fn scan_type_params(&mut self, type_params: &ast::TypeParams) -> SymbolTableResult { // Check for duplicate type parameter names let mut seen_names: std::collections::HashSet<&str> = std::collections::HashSet::new(); for type_param in &type_params.type_params { let (name, range) = match type_param { - TypeParam::TypeVar(tv) => (tv.name.as_str(), tv.range), - TypeParam::ParamSpec(ps) => (ps.name.as_str(), ps.range), - TypeParam::TypeVarTuple(tvt) => (tvt.name.as_str(), tvt.range), + ast::TypeParam::TypeVar(tv) => (tv.name.as_str(), tv.range), + ast::TypeParam::ParamSpec(ps) => (ps.name.as_str(), ps.range), + ast::TypeParam::TypeVarTuple(tvt) => (tvt.name.as_str(), tvt.range), }; if !seen_names.insert(name) { return Err(SymbolTableError { @@ -1609,7 +1972,7 @@ impl SymbolTableBuilder { // First register all type parameters for type_param in &type_params.type_params { match type_param { - TypeParam::TypeVar(TypeParamTypeVar { + ast::TypeParam::TypeVar(ast::TypeParamTypeVar { name, bound, range: type_var_range, @@ -1641,7 +2004,7 @@ impl SymbolTableBuilder { )?; } } - TypeParam::ParamSpec(TypeParamParamSpec { + ast::TypeParam::ParamSpec(ast::TypeParamParamSpec { name, range: param_spec_range, default, @@ -1659,7 +2022,7 @@ impl SymbolTableBuilder { )?; } } - TypeParam::TypeVarTuple(TypeParamTypeVarTuple { + ast::TypeParam::TypeVarTuple(ast::TypeParamTypeVarTuple { name, range: type_var_tuple_range, default, @@ -1682,22 +2045,24 @@ impl SymbolTableBuilder { Ok(()) } - fn scan_patterns(&mut self, patterns: &[Pattern]) -> SymbolTableResult { + fn scan_patterns(&mut self, patterns: &[ast::Pattern]) -> SymbolTableResult { for pattern in patterns { self.scan_pattern(pattern)?; } Ok(()) } - fn scan_pattern(&mut self, pattern: &Pattern) -> SymbolTableResult { - use Pattern::*; + fn scan_pattern(&mut self, pattern: &ast::Pattern) -> SymbolTableResult { + use ast::Pattern::*; match pattern { - MatchValue(PatternMatchValue { value, .. }) => { + MatchValue(ast::PatternMatchValue { value, .. }) => { self.scan_expression(value, ExpressionContext::Load)? } MatchSingleton(_) => {} - MatchSequence(PatternMatchSequence { patterns, .. }) => self.scan_patterns(patterns)?, - MatchMapping(PatternMatchMapping { + MatchSequence(ast::PatternMatchSequence { patterns, .. }) => { + self.scan_patterns(patterns)? + } + MatchMapping(ast::PatternMatchMapping { keys, patterns, rest, @@ -1709,19 +2074,19 @@ impl SymbolTableBuilder { self.register_ident(rest, SymbolUsage::Assigned)?; } } - MatchClass(PatternMatchClass { cls, arguments, .. }) => { + MatchClass(ast::PatternMatchClass { cls, arguments, .. }) => { self.scan_expression(cls, ExpressionContext::Load)?; self.scan_patterns(&arguments.patterns)?; for kw in &arguments.keywords { self.scan_pattern(&kw.pattern)?; } } - MatchStar(PatternMatchStar { name, .. }) => { + MatchStar(ast::PatternMatchStar { name, .. }) => { if let Some(name) = name { self.register_ident(name, SymbolUsage::Assigned)?; } } - MatchAs(PatternMatchAs { pattern, name, .. }) => { + MatchAs(ast::PatternMatchAs { pattern, name, .. }) => { if let Some(pattern) = pattern { self.scan_pattern(pattern)?; } @@ -1729,7 +2094,7 @@ impl SymbolTableBuilder { self.register_ident(name, SymbolUsage::Assigned)?; } } - MatchOr(PatternMatchOr { patterns, .. }) => self.scan_patterns(patterns)?, + MatchOr(ast::PatternMatchOr { patterns, .. }) => self.scan_patterns(patterns)?, } Ok(()) } @@ -1737,8 +2102,10 @@ impl SymbolTableBuilder { fn enter_scope_with_parameters( &mut self, name: &str, - parameters: &Parameters, + parameters: &ast::Parameters, line_number: u32, + has_return_annotation: bool, + is_async: bool, ) -> SymbolTableResult { // Evaluate eventual default parameters: for default in parameters @@ -1776,7 +2143,44 @@ impl SymbolTableBuilder { self.scan_annotation(annotation)?; } - self.enter_scope(name, CompilerScope::Function, line_number); + // Check if this function has any annotations (parameter or return) + let has_param_annotations = parameters + .posonlyargs + .iter() + .chain(parameters.args.iter()) + .chain(parameters.kwonlyargs.iter()) + .any(|p| p.parameter.annotation.is_some()) + || parameters + .vararg + .as_ref() + .is_some_and(|p| p.annotation.is_some()) + || parameters + .kwarg + .as_ref() + .is_some_and(|p| p.annotation.is_some()); + + let has_any_annotations = has_param_annotations || has_return_annotation; + + // Take annotation_block if this function has any annotations. + // When in class scope, the class's annotation_block was saved before scanning + // function annotations, so the current annotation_block belongs to this function. + let annotation_block = if has_any_annotations { + self.tables.last_mut().unwrap().annotation_block.take() + } else { + None + }; + + let scope_type = if is_async { + CompilerScope::AsyncFunction + } else { + CompilerScope::Function + }; + self.enter_scope(name, scope_type, line_number); + + // Move annotation_block to function scope only if we have one + if let Some(block) = annotation_block { + self.tables.last_mut().unwrap().annotation_block = Some(block); + } // Fill scope with parameter names: self.scan_parameters(&parameters.posonlyargs)?; @@ -1791,10 +2195,41 @@ impl SymbolTableBuilder { Ok(()) } - fn register_ident(&mut self, ident: &Identifier, role: SymbolUsage) -> SymbolTableResult { + fn register_ident(&mut self, ident: &ast::Identifier, role: SymbolUsage) -> SymbolTableResult { self.register_name(ident.as_str(), role, ident.range) } + fn check_name( + &self, + name: &str, + context: ExpressionContext, + range: TextRange, + ) -> SymbolTableResult { + if name == "__debug__" { + let location = Some( + self.source_file + .to_source_code() + .source_location(range.start(), PositionEncoding::Utf8), + ); + match context { + ExpressionContext::Store | ExpressionContext::Iter => { + return Err(SymbolTableError { + error: "cannot assign to __debug__".to_owned(), + location, + }); + } + ExpressionContext::Delete => { + return Err(SymbolTableError { + error: "cannot delete __debug__".to_owned(), + location, + }); + } + _ => {} + } + } + Ok(()) + } + fn register_name( &mut self, name: &str, @@ -1806,6 +2241,9 @@ impl SymbolTableBuilder { .to_source_code() .source_location(range.start(), PositionEncoding::Utf8); let location = Some(location); + + // Note: __debug__ checks are handled by check_name function, so no check needed here. + let scope_depth = self.tables.len(); let table = self.tables.last_mut().unwrap(); diff --git a/crates/codegen/src/unparse.rs b/crates/codegen/src/unparse.rs index 7b26d229187..eef2128587a 100644 --- a/crates/codegen/src/unparse.rs +++ b/crates/codegen/src/unparse.rs @@ -1,9 +1,6 @@ use alloc::fmt; use core::fmt::Display as _; -use ruff_python_ast::{ - self as ruff, Arguments, BoolOp, Comprehension, ConversionFlag, Expr, Identifier, Operator, - Parameter, ParameterWithDefault, Parameters, -}; +use ruff_python_ast as ast; use ruff_text_size::Ranged; use rustpython_compiler_core::SourceFile; use rustpython_literal::escape::{AsciiEscape, UnicodeEscape}; @@ -40,7 +37,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { self.f.write_str(s) } - fn p_id(&mut self, s: &Identifier) -> fmt::Result { + fn p_id(&mut self, s: &ast::Identifier) -> fmt::Result { self.f.write_str(s.as_str()) } @@ -59,7 +56,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { self.f.write_fmt(f) } - fn unparse_expr(&mut self, ast: &Expr, level: u8) -> fmt::Result { + fn unparse_expr(&mut self, ast: &ast::Expr, level: u8) -> fmt::Result { macro_rules! op_prec { ($op_ty:ident, $x:expr, $enu:path, $($var:ident($op:literal, $prec:ident)),*$(,)?) => { match $x { @@ -83,13 +80,13 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { }}; } match &ast { - Expr::BoolOp(ruff::ExprBoolOp { + ast::Expr::BoolOp(ast::ExprBoolOp { op, values, node_index: _, range: _range, }) => { - let (op, prec) = op_prec!(bin, op, BoolOp, And("and", AND), Or("or", OR)); + let (op, prec) = op_prec!(bin, op, ast::BoolOp, And("and", AND), Or("or", OR)); group_if!(prec, { let mut first = true; for val in values { @@ -98,7 +95,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { } }) } - Expr::Named(ruff::ExprNamed { + ast::Expr::Named(ast::ExprNamed { target, value, node_index: _, @@ -110,18 +107,18 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { self.unparse_expr(value, precedence::ATOM)?; }) } - Expr::BinOp(ruff::ExprBinOp { + ast::Expr::BinOp(ast::ExprBinOp { left, op, right, node_index: _, range: _range, }) => { - let right_associative = matches!(op, Operator::Pow); + let right_associative = matches!(op, ast::Operator::Pow); let (op, prec) = op_prec!( bin, op, - Operator, + ast::Operator, Add("+", ARITH), Sub("-", ARITH), Mult("*", TERM), @@ -142,7 +139,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { self.unparse_expr(right, prec + !right_associative as u8)?; }) } - Expr::UnaryOp(ruff::ExprUnaryOp { + ast::Expr::UnaryOp(ast::ExprUnaryOp { op, operand, node_index: _, @@ -151,7 +148,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { let (op, prec) = op_prec!( un, op, - ruff::UnaryOp, + ast::UnaryOp, Invert("~", FACTOR), Not("not ", NOT), UAdd("+", FACTOR), @@ -162,7 +159,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { self.unparse_expr(operand, prec)?; }) } - Expr::Lambda(ruff::ExprLambda { + ast::Expr::Lambda(ast::ExprLambda { parameters, body, node_index: _, @@ -178,7 +175,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { write!(self, ": {}", UnparseExpr::new(body, self.source))?; }) } - Expr::If(ruff::ExprIf { + ast::Expr::If(ast::ExprIf { test, body, orelse, @@ -193,7 +190,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { self.unparse_expr(orelse, precedence::TEST)?; }) } - Expr::Dict(ruff::ExprDict { + ast::Expr::Dict(ast::ExprDict { items, node_index: _, range: _range, @@ -211,7 +208,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { } self.p("}")?; } - Expr::Set(ruff::ExprSet { + ast::Expr::Set(ast::ExprSet { elts, node_index: _, range: _range, @@ -224,7 +221,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { } self.p("}")?; } - Expr::ListComp(ruff::ExprListComp { + ast::Expr::ListComp(ast::ExprListComp { elt, generators, node_index: _, @@ -235,7 +232,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { self.unparse_comp(generators)?; self.p("]")?; } - Expr::SetComp(ruff::ExprSetComp { + ast::Expr::SetComp(ast::ExprSetComp { elt, generators, node_index: _, @@ -246,7 +243,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { self.unparse_comp(generators)?; self.p("}")?; } - Expr::DictComp(ruff::ExprDictComp { + ast::Expr::DictComp(ast::ExprDictComp { key, value, generators, @@ -260,7 +257,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { self.unparse_comp(generators)?; self.p("}")?; } - Expr::Generator(ruff::ExprGenerator { + ast::Expr::Generator(ast::ExprGenerator { parenthesized: _, elt, generators, @@ -272,7 +269,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { self.unparse_comp(generators)?; self.p(")")?; } - Expr::Await(ruff::ExprAwait { + ast::Expr::Await(ast::ExprAwait { value, node_index: _, range: _range, @@ -282,7 +279,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { self.unparse_expr(value, precedence::ATOM)?; }) } - Expr::Yield(ruff::ExprYield { + ast::Expr::Yield(ast::ExprYield { value, node_index: _, range: _range, @@ -293,7 +290,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { self.p("(yield)")?; } } - Expr::YieldFrom(ruff::ExprYieldFrom { + ast::Expr::YieldFrom(ast::ExprYieldFrom { value, node_index: _, range: _range, @@ -304,7 +301,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { UnparseExpr::new(value, self.source) )?; } - Expr::Compare(ruff::ExprCompare { + ast::Expr::Compare(ast::ExprCompare { left, ops, comparators, @@ -322,9 +319,9 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { } }) } - Expr::Call(ruff::ExprCall { + ast::Expr::Call(ast::ExprCall { func, - arguments: Arguments { args, keywords, .. }, + arguments: ast::Arguments { args, keywords, .. }, node_index: _, range: _range, }) => { @@ -332,7 +329,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { self.p("(")?; if let ( [ - Expr::Generator(ruff::ExprGenerator { + ast::Expr::Generator(ast::ExprGenerator { elt, generators, node_index: _, @@ -365,9 +362,9 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { } self.p(")")?; } - Expr::FString(ruff::ExprFString { value, .. }) => self.unparse_fstring(value)?, - Expr::TString(_) => self.p("t\"\"")?, - Expr::StringLiteral(ruff::ExprStringLiteral { value, .. }) => { + ast::Expr::FString(ast::ExprFString { value, .. }) => self.unparse_fstring(value)?, + ast::Expr::TString(_) => self.p("t\"\"")?, + ast::Expr::StringLiteral(ast::ExprStringLiteral { value, .. }) => { if value.is_unicode() { self.p("u")? } @@ -375,12 +372,12 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { .str_repr() .fmt(self.f)? } - Expr::BytesLiteral(ruff::ExprBytesLiteral { value, .. }) => { + ast::Expr::BytesLiteral(ast::ExprBytesLiteral { value, .. }) => { AsciiEscape::new_repr(&value.bytes().collect::<Vec<_>>()) .bytes_repr() .fmt(self.f)? } - Expr::NumberLiteral(ruff::ExprNumberLiteral { value, .. }) => { + ast::Expr::NumberLiteral(ast::ExprNumberLiteral { value, .. }) => { #[allow(clippy::correctness, clippy::assertions_on_constants)] const { assert!(f64::MAX_10_EXP == 308) @@ -388,28 +385,28 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { let inf_str = "1e309"; match value { - ruff::Number::Int(int) => int.fmt(self.f)?, - &ruff::Number::Float(fp) => { + ast::Number::Int(int) => int.fmt(self.f)?, + &ast::Number::Float(fp) => { if fp.is_infinite() { self.p(inf_str)? } else { self.p(&rustpython_literal::float::to_string(fp))? } } - &ruff::Number::Complex { real, imag } => self + &ast::Number::Complex { real, imag } => self .p(&rustpython_literal::complex::to_string(real, imag) .replace("inf", inf_str))?, } } - Expr::BooleanLiteral(ruff::ExprBooleanLiteral { value, .. }) => { + ast::Expr::BooleanLiteral(ast::ExprBooleanLiteral { value, .. }) => { self.p(if *value { "True" } else { "False" })? } - Expr::NoneLiteral(ruff::ExprNoneLiteral { .. }) => self.p("None")?, - Expr::EllipsisLiteral(ruff::ExprEllipsisLiteral { .. }) => self.p("...")?, - Expr::Attribute(ruff::ExprAttribute { value, attr, .. }) => { + ast::Expr::NoneLiteral(ast::ExprNoneLiteral { .. }) => self.p("None")?, + ast::Expr::EllipsisLiteral(ast::ExprEllipsisLiteral { .. }) => self.p("...")?, + ast::Expr::Attribute(ast::ExprAttribute { value, attr, .. }) => { self.unparse_expr(value, precedence::ATOM)?; - let period = if let Expr::NumberLiteral(ruff::ExprNumberLiteral { - value: ruff::Number::Int(_), + let period = if let ast::Expr::NumberLiteral(ast::ExprNumberLiteral { + value: ast::Number::Int(_), .. }) = value.as_ref() { @@ -420,19 +417,19 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { self.p(period)?; self.p_id(attr)?; } - Expr::Subscript(ruff::ExprSubscript { value, slice, .. }) => { + ast::Expr::Subscript(ast::ExprSubscript { value, slice, .. }) => { self.unparse_expr(value, precedence::ATOM)?; let lvl = precedence::TUPLE; self.p("[")?; self.unparse_expr(slice, lvl)?; self.p("]")?; } - Expr::Starred(ruff::ExprStarred { value, .. }) => { + ast::Expr::Starred(ast::ExprStarred { value, .. }) => { self.p("*")?; self.unparse_expr(value, precedence::EXPR)?; } - Expr::Name(ruff::ExprName { id, .. }) => self.p(id.as_str())?, - Expr::List(ruff::ExprList { elts, .. }) => { + ast::Expr::Name(ast::ExprName { id, .. }) => self.p(id.as_str())?, + ast::Expr::List(ast::ExprList { elts, .. }) => { self.p("[")?; let mut first = true; for elt in elts { @@ -441,7 +438,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { } self.p("]")?; } - Expr::Tuple(ruff::ExprTuple { elts, .. }) => { + ast::Expr::Tuple(ast::ExprTuple { elts, .. }) => { if elts.is_empty() { self.p("()")?; } else { @@ -455,7 +452,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { }) } } - Expr::Slice(ruff::ExprSlice { + ast::Expr::Slice(ast::ExprSlice { lower, upper, step, @@ -474,12 +471,12 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { self.unparse_expr(step, precedence::TEST)?; } } - Expr::IpyEscapeCommand(_) => {} + ast::Expr::IpyEscapeCommand(_) => {} } Ok(()) } - fn unparse_arguments(&mut self, args: &Parameters) -> fmt::Result { + fn unparse_arguments(&mut self, args: &ast::Parameters) -> fmt::Result { let mut first = true; for (i, arg) in args.posonlyargs.iter().chain(&args.args).enumerate() { self.p_delim(&mut first, ", ")?; @@ -504,7 +501,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { } Ok(()) } - fn unparse_function_arg(&mut self, arg: &ParameterWithDefault) -> fmt::Result { + fn unparse_function_arg(&mut self, arg: &ast::ParameterWithDefault) -> fmt::Result { self.unparse_arg(&arg.parameter)?; if let Some(default) = &arg.default { write!(self, "={}", UnparseExpr::new(default, self.source))?; @@ -512,7 +509,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { Ok(()) } - fn unparse_arg(&mut self, arg: &Parameter) -> fmt::Result { + fn unparse_arg(&mut self, arg: &ast::Parameter) -> fmt::Result { self.p_id(&arg.name)?; if let Some(ann) = &arg.annotation { write!(self, ": {}", UnparseExpr::new(ann, self.source))?; @@ -520,7 +517,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { Ok(()) } - fn unparse_comp(&mut self, generators: &[Comprehension]) -> fmt::Result { + fn unparse_comp(&mut self, generators: &[ast::Comprehension]) -> fmt::Result { for comp in generators { self.p(if comp.is_async { " async for " @@ -538,10 +535,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { Ok(()) } - fn unparse_fstring_body( - &mut self, - elements: &[ruff::InterpolatedStringElement], - ) -> fmt::Result { + fn unparse_fstring_body(&mut self, elements: &[ast::InterpolatedStringElement]) -> fmt::Result { for elem in elements { self.unparse_fstring_elem(elem)?; } @@ -550,15 +544,15 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { fn unparse_formatted( &mut self, - val: &Expr, - debug_text: Option<&ruff::DebugText>, - conversion: ConversionFlag, - spec: Option<&ruff::InterpolatedStringFormatSpec>, + val: &ast::Expr, + debug_text: Option<&ast::DebugText>, + conversion: ast::ConversionFlag, + spec: Option<&ast::InterpolatedStringFormatSpec>, ) -> fmt::Result { - let buffered = to_string_fmt(|f| { - Unparser::new(f, self.source).unparse_expr(val, precedence::TEST + 1) - }); - if let Some(ruff::DebugText { leading, trailing }) = debug_text { + let buffered = + fmt::from_fn(|f| Unparser::new(f, self.source).unparse_expr(val, precedence::TEST + 1)) + .to_string(); + if let Some(ast::DebugText { leading, trailing }) = debug_text { self.p(leading)?; self.p(self.source.slice(val.range()))?; self.p(trailing)?; @@ -573,7 +567,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { self.p(&buffered)?; drop(buffered); - if conversion != ConversionFlag::None { + if conversion != ast::ConversionFlag::None { self.p("!")?; let buf = &[conversion as u8]; let c = core::str::from_utf8(buf).unwrap(); @@ -590,9 +584,9 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { Ok(()) } - fn unparse_fstring_elem(&mut self, elem: &ruff::InterpolatedStringElement) -> fmt::Result { + fn unparse_fstring_elem(&mut self, elem: &ast::InterpolatedStringElement) -> fmt::Result { match elem { - ruff::InterpolatedStringElement::Interpolation(ruff::InterpolatedElement { + ast::InterpolatedStringElement::Interpolation(ast::InterpolatedElement { expression, debug_text, conversion, @@ -604,7 +598,7 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { *conversion, format_spec.as_deref(), ), - ruff::InterpolatedStringElement::Literal(ruff::InterpolatedStringLiteralElement { + ast::InterpolatedStringElement::Literal(ast::InterpolatedStringLiteralElement { value, .. }) => self.unparse_fstring_str(value), @@ -616,16 +610,17 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { self.p(&s) } - fn unparse_fstring(&mut self, value: &ruff::FStringValue) -> fmt::Result { + fn unparse_fstring(&mut self, value: &ast::FStringValue) -> fmt::Result { self.p("f")?; - let body = to_string_fmt(|f| { + let body = fmt::from_fn(|f| { value.iter().try_for_each(|part| match part { - ruff::FStringPart::Literal(lit) => f.write_str(lit), - ruff::FStringPart::FString(ruff::FString { elements, .. }) => { + ast::FStringPart::Literal(lit) => f.write_str(lit), + ast::FStringPart::FString(ast::FString { elements, .. }) => { Unparser::new(f, self.source).unparse_fstring_body(elements) } }) - }); + }) + .to_string(); // .unparse_fstring_body(elements)); UnicodeEscape::new_repr(body.as_str().as_ref()) .str_repr() @@ -634,12 +629,12 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> { } pub struct UnparseExpr<'a> { - expr: &'a Expr, + expr: &'a ast::Expr, source: &'a SourceFile, } impl<'a> UnparseExpr<'a> { - pub const fn new(expr: &'a Expr, source: &'a SourceFile) -> Self { + pub const fn new(expr: &'a ast::Expr, source: &'a SourceFile) -> Self { Self { expr, source } } } @@ -649,14 +644,3 @@ impl fmt::Display for UnparseExpr<'_> { Unparser::new(f, self.source).unparse_expr(self.expr, precedence::TEST) } } - -fn to_string_fmt(f: impl FnOnce(&mut fmt::Formatter<'_>) -> fmt::Result) -> String { - use core::cell::Cell; - struct Fmt<F>(Cell<Option<F>>); - impl<F: FnOnce(&mut fmt::Formatter<'_>) -> fmt::Result> fmt::Display for Fmt<F> { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - self.0.take().unwrap()(f) - } - } - Fmt(Cell::new(Some(f))).to_string() -} diff --git a/crates/common/src/cformat.rs b/crates/common/src/cformat.rs index 24332396fdb..7b9609e90ae 100644 --- a/crates/common/src/cformat.rs +++ b/crates/common/src/cformat.rs @@ -14,7 +14,7 @@ use rustpython_literal::{float, format::Case}; use crate::wtf8::{CodePoint, Wtf8, Wtf8Buf}; -#[derive(Debug, PartialEq)] +#[derive(Clone, Copy, Debug, PartialEq)] pub enum CFormatErrorType { UnmatchedKeyParentheses, MissingModuloSign, @@ -27,7 +27,7 @@ pub enum CFormatErrorType { // also contains how many chars the parsing function consumed pub type ParsingError = (CFormatErrorType, usize); -#[derive(Debug, PartialEq)] +#[derive(Clone, Copy, Debug, PartialEq)] pub struct CFormatError { pub typ: CFormatErrorType, // FIXME pub index: usize, diff --git a/crates/common/src/crt_fd.rs b/crates/common/src/crt_fd.rs index 1902a362e32..7c93a30bb65 100644 --- a/crates/common/src/crt_fd.rs +++ b/crates/common/src/crt_fd.rs @@ -5,8 +5,10 @@ use alloc::fmt; use core::cmp; use std::{ffi, io}; +#[cfg(unix)] +use std::os::fd::AsFd; #[cfg(not(windows))] -use std::os::fd::{AsFd, AsRawFd, BorrowedFd, FromRawFd, IntoRawFd, OwnedFd, RawFd}; +use std::os::fd::{AsRawFd, BorrowedFd, FromRawFd, IntoRawFd, OwnedFd, RawFd}; #[cfg(windows)] use std::os::windows::io::BorrowedHandle; diff --git a/crates/common/src/encodings.rs b/crates/common/src/encodings.rs index d54581eb9ea..c2f139b6bb9 100644 --- a/crates/common/src/encodings.rs +++ b/crates/common/src/encodings.rs @@ -262,6 +262,7 @@ pub mod errors { use super::*; use core::fmt::Write; + #[derive(Clone, Copy)] pub struct Strict; impl<Ctx: EncodeContext> EncodeErrorHandler<Ctx> for Strict { @@ -286,6 +287,7 @@ pub mod errors { } } + #[derive(Clone, Copy)] pub struct Ignore; impl<Ctx: EncodeContext> EncodeErrorHandler<Ctx> for Ignore { @@ -310,6 +312,7 @@ pub mod errors { } } + #[derive(Clone, Copy)] pub struct Replace; impl<Ctx: EncodeContext> EncodeErrorHandler<Ctx> for Replace { @@ -338,6 +341,7 @@ pub mod errors { } } + #[derive(Clone, Copy)] pub struct XmlCharRefReplace; impl<Ctx: EncodeContext> EncodeErrorHandler<Ctx> for XmlCharRefReplace { @@ -358,6 +362,7 @@ pub mod errors { } } + #[derive(Clone, Copy)] pub struct BackslashReplace; impl<Ctx: EncodeContext> EncodeErrorHandler<Ctx> for BackslashReplace { @@ -394,6 +399,7 @@ pub mod errors { } } + #[derive(Clone, Copy)] pub struct NameReplace; impl<Ctx: EncodeContext> EncodeErrorHandler<Ctx> for NameReplace { @@ -422,6 +428,7 @@ pub mod errors { } } + #[derive(Clone, Copy)] pub struct SurrogateEscape; impl<Ctx: EncodeContext> EncodeErrorHandler<Ctx> for SurrogateEscape { diff --git a/crates/common/src/fileutils.rs b/crates/common/src/fileutils.rs index 9ed5e77afbb..af9e5ad2d96 100644 --- a/crates/common/src/fileutils.rs +++ b/crates/common/src/fileutils.rs @@ -46,7 +46,7 @@ pub mod windows { pub const SECS_BETWEEN_EPOCHS: i64 = 11644473600; // Seconds between 1.1.1601 and 1.1.1970 - #[derive(Default)] + #[derive(Clone, Copy, Default)] pub struct StatStruct { pub st_dev: libc::c_ulong, pub st_ino: u64, @@ -256,6 +256,7 @@ pub mod windows { m as _ } + #[derive(Clone, Copy)] #[repr(C)] pub struct FILE_STAT_BASIC_INFORMATION { pub FileId: i64, @@ -275,8 +276,9 @@ pub mod windows { pub FileId128: [u64; 2], } - #[repr(C)] #[allow(dead_code)] + #[derive(Clone, Copy)] + #[repr(C)] pub enum FILE_INFO_BY_NAME_CLASS { FileStatByNameInfo, FileStatLxByNameInfo, @@ -303,7 +305,8 @@ pub mod windows { let GetFileInformationByName = GET_FILE_INFORMATION_BY_NAME .get_or_init(|| { - let library_name = OsString::from("api-ms-win-core-file-l2-1-4").to_wide_with_nul(); + let library_name = + OsString::from("api-ms-win-core-file-l2-1-4.dll").to_wide_with_nul(); let module = unsafe { LoadLibraryW(library_name.as_ptr()) }; if module.is_null() { return None; diff --git a/crates/common/src/format.rs b/crates/common/src/format.rs index 1afee519aef..2842bd0a3d4 100644 --- a/crates/common/src/format.rs +++ b/crates/common/src/format.rs @@ -110,7 +110,7 @@ impl FormatParse for FormatSign { } } -#[derive(Debug, PartialEq)] +#[derive(Clone, Copy, Debug, PartialEq)] pub enum FormatGrouping { Comma, Underscore, @@ -136,7 +136,7 @@ impl From<&FormatGrouping> for char { } } -#[derive(Debug, PartialEq)] +#[derive(Clone, Copy, Debug, PartialEq)] pub enum FormatType { String, Binary, @@ -199,7 +199,7 @@ impl FormatParse for FormatType { } } -#[derive(Debug, PartialEq)] +#[derive(Clone, Copy, Debug, PartialEq)] pub struct FormatSpec { conversion: Option<FormatConversion>, fill: Option<CodePoint>, @@ -845,7 +845,7 @@ impl Deref for AsciiStr<'_> { } } -#[derive(Debug, PartialEq)] +#[derive(Clone, Copy, Debug, PartialEq)] pub enum FormatSpecError { DecimalDigitsTooMany, PrecisionTooBig, @@ -862,7 +862,7 @@ pub enum FormatSpecError { NotImplemented(char, &'static str), } -#[derive(Debug, PartialEq)] +#[derive(Clone, Copy, Debug, PartialEq)] pub enum FormatParseError { UnmatchedBracket, MissingStartBracket, diff --git a/crates/common/src/hash.rs b/crates/common/src/hash.rs index 40c428d89e3..f8f3783d224 100644 --- a/crates/common/src/hash.rs +++ b/crates/common/src/hash.rs @@ -25,6 +25,7 @@ pub const SEED_BITS: usize = core::mem::size_of::<u64>() * 2 * 8; // pub const CUTOFF: usize = 7; +#[derive(Clone, Copy)] pub struct HashSecret { k0: u64, k1: u64, diff --git a/crates/common/src/int.rs b/crates/common/src/int.rs index 57696e21fe7..9cfe2e0d738 100644 --- a/crates/common/src/int.rs +++ b/crates/common/src/int.rs @@ -29,7 +29,7 @@ pub fn float_to_ratio(value: f64) -> Option<(BigInt, BigInt)> { }) } -#[derive(Debug, Eq, PartialEq)] +#[derive(Copy, Clone, Debug, Eq, PartialEq)] pub enum BytesToIntError { InvalidLiteral { base: u32 }, InvalidBase, diff --git a/crates/common/src/lib.rs b/crates/common/src/lib.rs index 0181562d043..d52ed063867 100644 --- a/crates/common/src/lib.rs +++ b/crates/common/src/lib.rs @@ -1,7 +1,5 @@ //! A crate to hold types and functions common to all rustpython components. -#![cfg_attr(all(target_os = "wasi", target_env = "p2"), feature(wasip2))] - extern crate alloc; #[macro_use] diff --git a/crates/common/src/linked_list.rs b/crates/common/src/linked_list.rs index fb2b1260346..48cdb4feb95 100644 --- a/crates/common/src/linked_list.rs +++ b/crates/common/src/linked_list.rs @@ -333,7 +333,7 @@ impl<T> Pointers<T> { } } - const fn get_prev(&self) -> Option<NonNull<T>> { + pub const fn get_prev(&self) -> Option<NonNull<T>> { // SAFETY: prev is the first field in PointersInner, which is #[repr(C)]. unsafe { let inner = self.inner.get(); @@ -341,7 +341,7 @@ impl<T> Pointers<T> { ptr::read(prev) } } - const fn get_next(&self) -> Option<NonNull<T>> { + pub const fn get_next(&self) -> Option<NonNull<T>> { // SAFETY: next is the second field in PointersInner, which is #[repr(C)]. unsafe { let inner = self.inner.get(); @@ -351,7 +351,7 @@ impl<T> Pointers<T> { } } - const fn set_prev(&mut self, value: Option<NonNull<T>>) { + pub const fn set_prev(&mut self, value: Option<NonNull<T>>) { // SAFETY: prev is the first field in PointersInner, which is #[repr(C)]. unsafe { let inner = self.inner.get(); @@ -359,7 +359,7 @@ impl<T> Pointers<T> { ptr::write(prev, value); } } - const fn set_next(&mut self, value: Option<NonNull<T>>) { + pub const fn set_next(&mut self, value: Option<NonNull<T>>) { // SAFETY: next is the second field in PointersInner, which is #[repr(C)]. unsafe { let inner = self.inner.get(); diff --git a/crates/common/src/lock/cell_lock.rs b/crates/common/src/lock/cell_lock.rs index 73d722a8fdb..0e045c5950b 100644 --- a/crates/common/src/lock/cell_lock.rs +++ b/crates/common/src/lock/cell_lock.rs @@ -201,10 +201,14 @@ fn deadlock(lock_kind: &str, ty: &str) -> ! { panic!("deadlock: tried to {lock_kind}lock a Cell{ty} twice") } +#[derive(Clone, Copy)] pub struct SingleThreadId(()); + unsafe impl GetThreadId for SingleThreadId { const INIT: Self = Self(()); + fn nonzero_thread_id(&self) -> NonZero<usize> { - NonZero::new(1).unwrap() + // Safety: This is constant. + unsafe { NonZero::new_unchecked(1) } } } diff --git a/crates/common/src/lock/thread_mutex.rs b/crates/common/src/lock/thread_mutex.rs index 67ffc89245d..ec10ab1f181 100644 --- a/crates/common/src/lock/thread_mutex.rs +++ b/crates/common/src/lock/thread_mutex.rs @@ -121,19 +121,23 @@ impl<R: RawMutex, G: GetThreadId, T: ?Sized> ThreadMutex<R, G, T> { } } } -// Whether ThreadMutex::try_lock failed because the mutex was already locked on another thread or -// on the current thread + +#[derive(Clone, Copy)] pub enum TryLockThreadError { + /// Failed to lock because mutex was already locked on another thread. Other, + /// Failed to lock because mutex was already locked on current thread. Current, } struct LockedPlaceholder(&'static str); + impl fmt::Debug for LockedPlaceholder { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { f.write_str(self.0) } } + impl<R: RawMutex, G: GetThreadId, T: ?Sized + fmt::Debug> fmt::Debug for ThreadMutex<R, G, T> { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { match self.try_lock() { diff --git a/crates/common/src/os.rs b/crates/common/src/os.rs index 1ce25988d28..ef8547289a2 100644 --- a/crates/common/src/os.rs +++ b/crates/common/src/os.rs @@ -94,9 +94,61 @@ pub fn bytes_as_os_str(b: &[u8]) -> Result<&std::ffi::OsStr, Utf8Error> { #[cfg(unix)] pub use std::os::unix::ffi; -#[cfg(target_os = "wasi")] + +// WASIp1 uses stable std::os::wasi::ffi +#[cfg(all(target_os = "wasi", not(target_env = "p2")))] pub use std::os::wasi::ffi; +// WASIp2: std::os::wasip2::ffi is unstable, so we provide a stable implementation +// leveraging WASI's UTF-8 string guarantee +#[cfg(all(target_os = "wasi", target_env = "p2"))] +pub mod ffi { + use std::ffi::{OsStr, OsString}; + + pub trait OsStrExt: sealed::Sealed { + fn as_bytes(&self) -> &[u8]; + fn from_bytes(slice: &[u8]) -> &Self; + } + + impl OsStrExt for OsStr { + fn as_bytes(&self) -> &[u8] { + // WASI strings are guaranteed to be UTF-8 + self.to_str().expect("wasip2 strings are UTF-8").as_bytes() + } + + fn from_bytes(slice: &[u8]) -> &OsStr { + // WASI strings are guaranteed to be UTF-8 + OsStr::new(std::str::from_utf8(slice).expect("wasip2 strings are UTF-8")) + } + } + + pub trait OsStringExt: sealed::Sealed { + fn from_vec(vec: Vec<u8>) -> Self; + fn into_vec(self) -> Vec<u8>; + } + + impl OsStringExt for OsString { + fn from_vec(vec: Vec<u8>) -> OsString { + // WASI strings are guaranteed to be UTF-8 + OsString::from(String::from_utf8(vec).expect("wasip2 strings are UTF-8")) + } + + fn into_vec(self) -> Vec<u8> { + // WASI strings are guaranteed to be UTF-8 + self.to_str() + .expect("wasip2 strings are UTF-8") + .as_bytes() + .to_vec() + } + } + + mod sealed { + pub trait Sealed {} + impl Sealed for std::ffi::OsStr {} + impl Sealed for std::ffi::OsString {} + } +} + #[cfg(windows)] pub fn errno_to_winerror(errno: i32) -> i32 { use libc::*; @@ -131,7 +183,10 @@ pub fn winerror_to_errno(winerror: i32) -> i32 { use libc::*; use windows_sys::Win32::{ Foundation::*, - Networking::WinSock::{WSAEACCES, WSAEBADF, WSAEFAULT, WSAEINTR, WSAEINVAL, WSAEMFILE}, + Networking::WinSock::{ + WSAEACCES, WSAEBADF, WSAECONNABORTED, WSAECONNREFUSED, WSAECONNRESET, WSAEFAULT, + WSAEINTR, WSAEINVAL, WSAEMFILE, + }, }; // Unwrap FACILITY_WIN32 HRESULT errors. // if ((winerror & 0xFFFF0000) == 0x80070000) { @@ -218,6 +273,11 @@ pub fn winerror_to_errno(winerror: i32) -> i32 { ERROR_BROKEN_PIPE | ERROR_NO_DATA => EPIPE, ERROR_DIR_NOT_EMPTY => ENOTEMPTY, ERROR_NO_UNICODE_TRANSLATION => EILSEQ, + // Connection-related Windows error codes - map to Winsock error codes + // which Python uses on Windows (errno.ECONNREFUSED = 10061, etc.) + ERROR_CONNECTION_REFUSED => WSAECONNREFUSED, + ERROR_CONNECTION_ABORTED => WSAECONNABORTED, + ERROR_NETNAME_DELETED => WSAECONNRESET, ERROR_INVALID_FUNCTION | ERROR_INVALID_ACCESS | ERROR_INVALID_DATA diff --git a/crates/common/src/refcount.rs b/crates/common/src/refcount.rs index a5fbfa8fc36..c7038667099 100644 --- a/crates/common/src/refcount.rs +++ b/crates/common/src/refcount.rs @@ -40,6 +40,16 @@ impl RefCount { } } + #[inline] + pub fn inc_by(&self, n: usize) { + debug_assert!(n <= Self::MASK); + let old_size = self.strong.fetch_add(n, Relaxed); + + if old_size & Self::MASK > Self::MASK - n { + std::process::abort(); + } + } + /// Returns true if successful #[inline] pub fn safe_inc(&self) -> bool { diff --git a/crates/common/src/str.rs b/crates/common/src/str.rs index 155012ed21f..a250c5865d3 100644 --- a/crates/common/src/str.rs +++ b/crates/common/src/str.rs @@ -449,6 +449,7 @@ pub fn to_ascii(value: &str) -> AsciiString { unsafe { AsciiString::from_ascii_unchecked(ascii) } } +#[derive(Clone, Copy)] pub struct UnicodeEscapeCodepoint(pub CodePoint); impl fmt::Display for UnicodeEscapeCodepoint { diff --git a/crates/compiler-core/src/bytecode.rs b/crates/compiler-core/src/bytecode.rs index dc61e3b9b5a..13884dc8a73 100644 --- a/crates/compiler-core/src/bytecode.rs +++ b/crates/compiler-core/src/bytecode.rs @@ -8,15 +8,31 @@ use crate::{ }; use alloc::{collections::BTreeSet, fmt, vec::Vec}; use bitflags::bitflags; -use core::{hash, marker::PhantomData, mem, num::NonZeroU8, ops::Deref}; +use core::{hash, mem, ops::Deref}; use itertools::Itertools; use malachite_bigint::BigInt; use num_complex::Complex64; use rustpython_wtf8::{Wtf8, Wtf8Buf}; +pub use crate::bytecode::{ + instruction::{ + AnyInstruction, Arg, Instruction, InstructionMetadata, PseudoInstruction, StackEffect, + decode_load_attr_arg, decode_load_super_attr_arg, encode_load_attr_arg, + encode_load_super_attr_arg, + }, + oparg::{ + BinaryOperator, BuildSliceArgCount, CommonConstant, ComparisonOperator, ConvertValueOparg, + IntrinsicFunction1, IntrinsicFunction2, Invert, Label, MakeFunctionFlags, NameIdx, OpArg, + OpArgByte, OpArgState, OpArgType, RaiseKind, ResumeType, SpecialMethod, UnpackExArgs, + }, +}; + +mod instruction; +mod oparg; + /// Exception table entry for zero-cost exception handling /// Format: (start, size, target, depth<<1|lasti) -#[derive(Clone, Debug, PartialEq, Eq)] +#[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct ExceptionTableEntry { /// Start instruction offset (inclusive) pub start: u32, @@ -31,7 +47,7 @@ pub struct ExceptionTableEntry { } impl ExceptionTableEntry { - pub fn new(start: u32, end: u32, target: u32, depth: u16, push_lasti: bool) -> Self { + pub const fn new(start: u32, end: u32, target: u32, depth: u16, push_lasti: bool) -> Self { Self { start, end, @@ -84,88 +100,6 @@ pub fn find_exception_handler(table: &[u8], offset: u32) -> Option<ExceptionTabl None } -/// Oparg values for [`Instruction::ConvertValue`]. -/// -/// ## See also -/// -/// - [CPython FVC_* flags](https://github.com/python/cpython/blob/8183fa5e3f78ca6ab862de7fb8b14f3d929421e0/Include/ceval.h#L129-L132) -#[repr(u8)] -#[derive(Copy, Clone, Debug, Hash, PartialEq, Eq)] -pub enum ConvertValueOparg { - /// No conversion. - /// - /// ```python - /// f"{x}" - /// f"{x:4}" - /// ``` - None = 0, - /// Converts by calling `str(<value>)`. - /// - /// ```python - /// f"{x!s}" - /// f"{x!s:2}" - /// ``` - Str = 1, - /// Converts by calling `repr(<value>)`. - /// - /// ```python - /// f"{x!r}" - /// f"{x!r:2}" - /// ``` - Repr = 2, - /// Converts by calling `ascii(<value>)`. - /// - /// ```python - /// f"{x!a}" - /// f"{x!a:2}" - /// ``` - Ascii = 3, -} - -impl fmt::Display for ConvertValueOparg { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - let out = match self { - Self::Str => "1 (str)", - Self::Repr => "2 (repr)", - Self::Ascii => "3 (ascii)", - // We should never reach this. `FVC_NONE` are being handled by `Instruction::FormatSimple` - Self::None => "", - }; - - write!(f, "{out}") - } -} - -impl OpArgType for ConvertValueOparg { - #[inline] - fn from_op_arg(x: u32) -> Option<Self> { - Some(match x { - // Ruff `ConversionFlag::None` is `-1i8`, - // when its converted to `u8` its value is `u8::MAX` - 0 | 255 => Self::None, - 1 => Self::Str, - 2 => Self::Repr, - 3 => Self::Ascii, - _ => return None, - }) - } - - #[inline] - fn to_op_arg(self) -> u32 { - self as u32 - } -} - -/// Resume type for the RESUME instruction -#[derive(Copy, Clone, Debug, Hash, PartialEq, Eq)] -#[repr(u32)] -pub enum ResumeType { - AtFuncStart = 0, - AfterYield = 1, - AfterYieldFrom = 2, - AfterAwait = 3, -} - /// CPython 3.11+ linetable location info codes #[derive(Copy, Clone, Debug, PartialEq, Eq)] #[repr(u8)] @@ -356,781 +290,17 @@ pub struct CodeObject<C: Constant = ConstantData> { bitflags! { #[derive(Copy, Clone, Debug, PartialEq)] - pub struct CodeFlags: u16 { - const NEW_LOCALS = 0x01; - const IS_GENERATOR = 0x02; - const IS_COROUTINE = 0x04; - const HAS_VARARGS = 0x08; - const HAS_VARKEYWORDS = 0x10; - const IS_OPTIMIZED = 0x20; - } -} - -impl CodeFlags { - pub const NAME_MAPPING: &'static [(&'static str, Self)] = &[ - ("GENERATOR", Self::IS_GENERATOR), - ("COROUTINE", Self::IS_COROUTINE), - ( - "ASYNC_GENERATOR", - Self::from_bits_truncate(Self::IS_GENERATOR.bits() | Self::IS_COROUTINE.bits()), - ), - ("VARARGS", Self::HAS_VARARGS), - ("VARKEYWORDS", Self::HAS_VARKEYWORDS), - ]; -} - -/// an opcode argument that may be extended by a prior ExtendedArg -#[derive(Copy, Clone, PartialEq, Eq)] -#[repr(transparent)] -pub struct OpArgByte(pub u8); - -impl OpArgByte { - pub const fn null() -> Self { - Self(0) - } -} - -impl From<u8> for OpArgByte { - fn from(raw: u8) -> Self { - Self(raw) - } -} - -impl fmt::Debug for OpArgByte { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - self.0.fmt(f) - } -} - -/// a full 32-bit op_arg, including any possible ExtendedArg extension -#[derive(Copy, Clone, Debug)] -#[repr(transparent)] -pub struct OpArg(pub u32); - -impl OpArg { - pub const fn null() -> Self { - Self(0) - } - - /// Returns how many CodeUnits a instruction with this op_arg will be encoded as - #[inline] - pub const fn instr_size(self) -> usize { - (self.0 > 0xff) as usize + (self.0 > 0xff_ff) as usize + (self.0 > 0xff_ff_ff) as usize + 1 - } - - /// returns the arg split into any necessary ExtendedArg components (in big-endian order) and - /// the arg for the real opcode itself - #[inline(always)] - pub fn split(self) -> (impl ExactSizeIterator<Item = OpArgByte>, OpArgByte) { - let mut it = self - .0 - .to_le_bytes() - .map(OpArgByte) - .into_iter() - .take(self.instr_size()); - let lo = it.next().unwrap(); - (it.rev(), lo) - } -} - -impl From<u32> for OpArg { - fn from(raw: u32) -> Self { - Self(raw) - } -} - -#[derive(Default, Copy, Clone)] -#[repr(transparent)] -pub struct OpArgState { - state: u32, -} - -impl OpArgState { - #[inline(always)] - pub fn get(&mut self, ins: CodeUnit) -> (Instruction, OpArg) { - let arg = self.extend(ins.arg); - if ins.op != Instruction::ExtendedArg { - self.reset(); - } - (ins.op, arg) - } - - #[inline(always)] - pub fn extend(&mut self, arg: OpArgByte) -> OpArg { - self.state = (self.state << 8) | u32::from(arg.0); - OpArg(self.state) - } - - #[inline(always)] - pub const fn reset(&mut self) { - self.state = 0 - } -} - -pub trait OpArgType: Copy { - fn from_op_arg(x: u32) -> Option<Self>; - - fn to_op_arg(self) -> u32; -} - -impl OpArgType for u32 { - #[inline(always)] - fn from_op_arg(x: u32) -> Option<Self> { - Some(x) - } - - #[inline(always)] - fn to_op_arg(self) -> u32 { - self - } -} - -impl OpArgType for bool { - #[inline(always)] - fn from_op_arg(x: u32) -> Option<Self> { - Some(x != 0) - } - - #[inline(always)] - fn to_op_arg(self) -> u32 { - self as u32 - } -} - -macro_rules! op_arg_enum_impl { - (enum $name:ident { $($(#[$var_attr:meta])* $var:ident = $value:literal,)* }) => { - impl OpArgType for $name { - fn to_op_arg(self) -> u32 { - self as u32 - } - - fn from_op_arg(x: u32) -> Option<Self> { - Some(match u8::try_from(x).ok()? { - $($value => Self::$var,)* - _ => return None, - }) - } - } - }; -} - -macro_rules! op_arg_enum { - ($(#[$attr:meta])* $vis:vis enum $name:ident { $($(#[$var_attr:meta])* $var:ident = $value:literal,)* }) => { - $(#[$attr])* - $vis enum $name { - $($(#[$var_attr])* $var = $value,)* - } - - op_arg_enum_impl!(enum $name { - $($(#[$var_attr])* $var = $value,)* - }); - }; -} - -#[derive(Copy, Clone)] -pub struct Arg<T: OpArgType>(PhantomData<T>); - -impl<T: OpArgType> Arg<T> { - #[inline] - pub const fn marker() -> Self { - Self(PhantomData) - } - - #[inline] - pub fn new(arg: T) -> (Self, OpArg) { - (Self(PhantomData), OpArg(arg.to_op_arg())) - } - - #[inline] - pub fn new_single(arg: T) -> (Self, OpArgByte) - where - T: Into<u8>, - { - (Self(PhantomData), OpArgByte(arg.into())) - } - - #[inline(always)] - pub fn get(self, arg: OpArg) -> T { - self.try_get(arg).unwrap() - } - - #[inline(always)] - pub fn try_get(self, arg: OpArg) -> Option<T> { - T::from_op_arg(arg.0) - } - - /// # Safety - /// T::from_op_arg(self) must succeed - #[inline(always)] - pub unsafe fn get_unchecked(self, arg: OpArg) -> T { - // SAFETY: requirements forwarded from caller - unsafe { T::from_op_arg(arg.0).unwrap_unchecked() } - } -} - -impl<T: OpArgType> PartialEq for Arg<T> { - fn eq(&self, _: &Self) -> bool { - true - } -} - -impl<T: OpArgType> Eq for Arg<T> {} - -impl<T: OpArgType> fmt::Debug for Arg<T> { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - write!(f, "Arg<{}>", core::any::type_name::<T>()) - } -} - -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Ord, PartialOrd)] -#[repr(transparent)] -// XXX: if you add a new instruction that stores a Label, make sure to add it in -// Instruction::label_arg -pub struct Label(pub u32); - -impl OpArgType for Label { - #[inline(always)] - fn from_op_arg(x: u32) -> Option<Self> { - Some(Self(x)) - } - - #[inline(always)] - fn to_op_arg(self) -> u32 { - self.0 - } -} - -impl fmt::Display for Label { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - self.0.fmt(f) - } -} - -op_arg_enum!( - /// The kind of Raise that occurred. - #[derive(Copy, Clone, Debug, PartialEq, Eq)] - #[repr(u8)] - pub enum RaiseKind { - /// Bare `raise` statement with no arguments. - /// Gets the current exception from VM state (topmost_exception). - /// Maps to RAISE_VARARGS with oparg=0. - BareRaise = 0, - /// `raise exc` - exception is on the stack. - /// Maps to RAISE_VARARGS with oparg=1. - Raise = 1, - /// `raise exc from cause` - exception and cause are on the stack. - /// Maps to RAISE_VARARGS with oparg=2. - RaiseCause = 2, - /// Reraise exception from the stack top. - /// Used in exception handler cleanup blocks (finally, except). - /// Gets exception from stack, not from VM state. - /// Maps to the RERAISE opcode. - ReraiseFromStack = 3, - } -); - -op_arg_enum!( - /// Intrinsic function for CALL_INTRINSIC_1 - #[derive(Copy, Clone, Debug, PartialEq, Eq)] - #[repr(u8)] - pub enum IntrinsicFunction1 { - // Invalid = 0, - Print = 1, - /// Import * operation - ImportStar = 2, - // StopIterationError = 3, - // AsyncGenWrap = 4, - UnaryPositive = 5, - /// Convert list to tuple - ListToTuple = 6, - /// Type parameter related - TypeVar = 7, - ParamSpec = 8, - TypeVarTuple = 9, - /// Generic subscript for PEP 695 - SubscriptGeneric = 10, - TypeAlias = 11, - } -); - -op_arg_enum!( - /// Intrinsic function for CALL_INTRINSIC_2 - #[derive(Copy, Clone, Debug, PartialEq, Eq)] - #[repr(u8)] - pub enum IntrinsicFunction2 { - PrepReraiseStar = 1, - TypeVarWithBound = 2, - TypeVarWithConstraint = 3, - SetFunctionTypeParams = 4, - /// Set default value for type parameter (PEP 695) - SetTypeparamDefault = 5, - } -); - -pub type NameIdx = u32; - -/// A Single bytecode instruction. -/// Instructions are ordered to match CPython 3.13 opcode numbers exactly. -/// HAVE_ARGUMENT = 44: opcodes 0-43 have no argument, 44+ have arguments. -#[derive(Debug, Copy, Clone, PartialEq, Eq)] -#[repr(u8)] -pub enum Instruction { - // ==================== No-argument instructions (opcode < 44) ==================== - // 0: CACHE - placeholder for inline cache (not executed) - Cache, - // 1: BEFORE_ASYNC_WITH - BeforeAsyncWith, - // 2: BEFORE_WITH - BeforeWith, - // 3: Reserved (BINARY_OP_INPLACE_ADD_UNICODE in CPython) - Reserved3, - // 4: BINARY_SLICE - not implemented, placeholder - BinarySlice, - // 5: BINARY_SUBSCR - BinarySubscript, - // 6: CHECK_EG_MATCH - CheckEgMatch, - // 7: CHECK_EXC_MATCH - CheckExcMatch, - // 8: CLEANUP_THROW - CleanupThrow, - // 9: DELETE_SUBSCR - DeleteSubscript, - // 10: END_ASYNC_FOR - EndAsyncFor, - // 11: END_FOR - not implemented, placeholder - EndFor, - // 12: END_SEND - EndSend, - // 13: EXIT_INIT_CHECK - not implemented, placeholder - ExitInitCheck, - // 14: FORMAT_SIMPLE - FormatSimple, - // 15: FORMAT_WITH_SPEC - FormatWithSpec, - // 16: GET_AITER - GetAIter, - // 17: RESERVED - Reserved17, - // 18: GET_ANEXT - GetANext, - // 19: GET_ITER - GetIter, - // 20: GET_LEN - GetLen, - // 21: GET_YIELD_FROM_ITER - not implemented, placeholder - GetYieldFromIter, - // 22: INTERPRETER_EXIT - not implemented, placeholder - InterpreterExit, - // 23: LOAD_ASSERTION_ERROR - not implemented, placeholder - LoadAssertionError, - // 24: LOAD_BUILD_CLASS - LoadBuildClass, - // 25: LOAD_LOCALS - not implemented, placeholder - LoadLocals, - // 26: MAKE_FUNCTION - MakeFunction, - // 27: MATCH_KEYS - MatchKeys, - // 28: MATCH_MAPPING - MatchMapping, - // 29: MATCH_SEQUENCE - MatchSequence, - // 30: NOP - Nop, - // 31: POP_EXCEPT - PopException, - // 32: POP_TOP - PopTop, - // 33: PUSH_EXC_INFO - PushExcInfo, - // 34: PUSH_NULL - not implemented, placeholder - PushNull, - // 35: RETURN_GENERATOR - not implemented, placeholder - ReturnGenerator, - // 36: RETURN_VALUE - ReturnValue, - // 37: SETUP_ANNOTATIONS - SetupAnnotation, - // 38: STORE_SLICE - not implemented, placeholder - StoreSlice, - // 39: STORE_SUBSCR - StoreSubscript, - // 40: TO_BOOL - ToBool, - // 41: UNARY_INVERT - UnaryInvert, - // 42: UNARY_NEGATIVE - UnaryNegative, - // 43: UNARY_NOT - UnaryNot, - // ==================== With-argument instructions (opcode >= 44) ==================== - // 44: WITH_EXCEPT_START - WithExceptStart, - // 45: BINARY_OP - BinaryOp { - op: Arg<BinaryOperator>, - }, - // 46: BUILD_CONST_KEY_MAP - not implemented, placeholder - BuildConstKeyMap { - size: Arg<u32>, - }, - // 47: BUILD_LIST - BuildList { - size: Arg<u32>, - }, - // 48: BUILD_MAP - BuildMap { - size: Arg<u32>, - }, - // 49: BUILD_SET - BuildSet { - size: Arg<u32>, - }, - // 50: BUILD_SLICE - BuildSlice { - argc: Arg<BuildSliceArgCount>, - }, - // 51: BUILD_STRING - BuildString { - size: Arg<u32>, - }, - // 52: BUILD_TUPLE - BuildTuple { - size: Arg<u32>, - }, - // 53: CALL - CallFunctionPositional { - nargs: Arg<u32>, - }, - // 54: CALL_FUNCTION_EX - CallFunctionEx { - has_kwargs: Arg<bool>, - }, - // 55: CALL_INTRINSIC_1 - CallIntrinsic1 { - func: Arg<IntrinsicFunction1>, - }, - // 56: CALL_INTRINSIC_2 - CallIntrinsic2 { - func: Arg<IntrinsicFunction2>, - }, - // 57: CALL_KW - CallFunctionKeyword { - nargs: Arg<u32>, - }, - // 58: COMPARE_OP - CompareOperation { - op: Arg<ComparisonOperator>, - }, - // 59: CONTAINS_OP - ContainsOp(Arg<Invert>), - // 60: CONVERT_VALUE - ConvertValue { - oparg: Arg<ConvertValueOparg>, - }, - // 61: COPY - CopyItem { - index: Arg<u32>, - }, - // 62: COPY_FREE_VARS - not implemented, placeholder - CopyFreeVars { - count: Arg<u32>, - }, - // 63: DELETE_ATTR - DeleteAttr { - idx: Arg<NameIdx>, - }, - // 64: DELETE_DEREF - DeleteDeref(Arg<NameIdx>), - // 65: DELETE_FAST - DeleteFast(Arg<NameIdx>), - // 66: DELETE_GLOBAL - DeleteGlobal(Arg<NameIdx>), - // 67: DELETE_NAME - DeleteLocal(Arg<NameIdx>), - // 68: DICT_MERGE - not implemented, placeholder - DictMerge { - index: Arg<u32>, - }, - // 69: DICT_UPDATE - DictUpdate { - index: Arg<u32>, - }, - // 70: ENTER_EXECUTOR - not implemented, placeholder - EnterExecutor { - index: Arg<u32>, - }, - // 71: EXTENDED_ARG - ExtendedArg, - // 72: FOR_ITER - ForIter { - target: Arg<Label>, - }, - // 73: GET_AWAITABLE - GetAwaitable, - // 74: IMPORT_FROM - ImportFrom { - idx: Arg<NameIdx>, - }, - // 75: IMPORT_NAME - ImportName { - idx: Arg<NameIdx>, - }, - // 76: IS_OP - IsOp(Arg<Invert>), - // 77: JUMP_BACKWARD - not implemented, placeholder - JumpBackward { - target: Arg<Label>, - }, - // 78: JUMP_BACKWARD_NO_INTERRUPT - not implemented, placeholder - JumpBackwardNoInterrupt { - target: Arg<Label>, - }, - // 79: JUMP_FORWARD - not implemented, placeholder - JumpForward { - target: Arg<Label>, - }, - // 80: LIST_APPEND - ListAppend { - i: Arg<u32>, - }, - // 81: LIST_EXTEND - not implemented, placeholder - ListExtend { - i: Arg<u32>, - }, - // 82: LOAD_ATTR - LoadAttr { - idx: Arg<NameIdx>, - }, - // 83: LOAD_CONST - LoadConst { - idx: Arg<u32>, - }, - // 84: LOAD_DEREF - LoadDeref(Arg<NameIdx>), - // 85: LOAD_FAST - LoadFast(Arg<NameIdx>), - // 86: LOAD_FAST_AND_CLEAR - LoadFastAndClear(Arg<NameIdx>), - // 87: LOAD_FAST_CHECK - not implemented, placeholder - LoadFastCheck(Arg<NameIdx>), - // 88: LOAD_FAST_LOAD_FAST - not implemented, placeholder - LoadFastLoadFast { - arg: Arg<u32>, - }, - // 89: LOAD_FROM_DICT_OR_DEREF - not implemented, placeholder - LoadFromDictOrDeref(Arg<NameIdx>), - // 90: LOAD_FROM_DICT_OR_GLOBALS - not implemented, placeholder - LoadFromDictOrGlobals(Arg<NameIdx>), - // 91: LOAD_GLOBAL - LoadGlobal(Arg<NameIdx>), - // 92: LOAD_NAME - LoadNameAny(Arg<NameIdx>), - // 93: LOAD_SUPER_ATTR - not implemented, placeholder - LoadSuperAttr { - arg: Arg<u32>, - }, - // 94: MAKE_CELL - not implemented, placeholder - MakeCell(Arg<NameIdx>), - // 95: MAP_ADD - MapAdd { - i: Arg<u32>, - }, - // 96: MATCH_CLASS - MatchClass(Arg<u32>), - // 97: POP_JUMP_IF_FALSE - PopJumpIfFalse { - target: Arg<Label>, - }, - // 98: POP_JUMP_IF_NONE - not implemented, placeholder - PopJumpIfNone { - target: Arg<Label>, - }, - // 99: POP_JUMP_IF_NOT_NONE - not implemented, placeholder - PopJumpIfNotNone { - target: Arg<Label>, - }, - // 100: POP_JUMP_IF_TRUE - PopJumpIfTrue { - target: Arg<Label>, - }, - // 101: RAISE_VARARGS - Raise { - kind: Arg<RaiseKind>, - }, - // 102: RERAISE - Reraise { - depth: Arg<u32>, - }, - // 103: RETURN_CONST - ReturnConst { - idx: Arg<u32>, - }, - // 104: SEND - Send { - target: Arg<Label>, - }, - // 105: SET_ADD - SetAdd { - i: Arg<u32>, - }, - // 106: SET_FUNCTION_ATTRIBUTE - SetFunctionAttribute { - attr: Arg<MakeFunctionFlags>, - }, - // 107: SET_UPDATE - not implemented, placeholder - SetUpdate { - i: Arg<u32>, - }, - // 108: STORE_ATTR - StoreAttr { - idx: Arg<NameIdx>, - }, - // 109: STORE_DEREF - StoreDeref(Arg<NameIdx>), - // 110: STORE_FAST - StoreFast(Arg<NameIdx>), - // 111: STORE_FAST_LOAD_FAST - StoreFastLoadFast { - store_idx: Arg<NameIdx>, - load_idx: Arg<NameIdx>, - }, - // 112: STORE_FAST_STORE_FAST - not implemented, placeholder - StoreFastStoreFast { - arg: Arg<u32>, - }, - // 113: STORE_GLOBAL - StoreGlobal(Arg<NameIdx>), - // 114: STORE_NAME - StoreLocal(Arg<NameIdx>), - // 115: SWAP - Swap { - index: Arg<u32>, - }, - // 116: UNPACK_EX - UnpackEx { - args: Arg<UnpackExArgs>, - }, - // 117: UNPACK_SEQUENCE - UnpackSequence { - size: Arg<u32>, - }, - // 118: YIELD_VALUE - YieldValue { - arg: Arg<u32>, - }, - // ==================== RustPython-only instructions (119+) ==================== - // 119: BREAK - Break { - target: Arg<Label>, - }, - // 120: BUILD_LIST_FROM_TUPLES - BuildListFromTuples { - size: Arg<u32>, - }, - // 121: BUILD_MAP_FOR_CALL - BuildMapForCall { - size: Arg<u32>, - }, - // 122: BUILD_SET_FROM_TUPLES - BuildSetFromTuples { - size: Arg<u32>, - }, - // 123: BUILD_TUPLE_FROM_ITER - BuildTupleFromIter, - // 124: BUILD_TUPLE_FROM_TUPLES - BuildTupleFromTuples { - size: Arg<u32>, - }, - // 125: CALL_METHOD - CallMethodPositional { - nargs: Arg<u32>, - }, - // 126: CALL_METHOD_KW - CallMethodKeyword { - nargs: Arg<u32>, - }, - // 127: CALL_METHOD_EX - CallMethodEx { - has_kwargs: Arg<bool>, - }, - // 128: CONTINUE - Continue { - target: Arg<Label>, - }, - // 129: JUMP (CPython uses pseudo-op 256) - Jump { - target: Arg<Label>, - }, - // 130: JUMP_IF_FALSE_OR_POP - JumpIfFalseOrPop { - target: Arg<Label>, - }, - // 131: JUMP_IF_TRUE_OR_POP - JumpIfTrueOrPop { - target: Arg<Label>, - }, - // 132: JUMP_IF_NOT_EXC_MATCH - JumpIfNotExcMatch(Arg<Label>), - // 133: LOAD_CLASSDEREF - LoadClassDeref(Arg<NameIdx>), - // 134: LOAD_CLOSURE (CPython uses pseudo-op 258) - LoadClosure(Arg<NameIdx>), - // 135: LOAD_METHOD (CPython uses pseudo-op 259) - LoadMethod { - idx: Arg<NameIdx>, - }, - // 136: POP_BLOCK (CPython uses pseudo-op 263) - PopBlock, - // 137: REVERSE - Reverse { - amount: Arg<u32>, - }, - // 138: SET_EXC_INFO - SetExcInfo, - // 139: SUBSCRIPT - Subscript, - // 140-148: Reserved (padding to keep RESUME at 149) - Reserved140, - Reserved141, - Reserved142, - Reserved143, - Reserved144, - Reserved145, - Reserved146, - Reserved147, - Reserved148, - // 149: RESUME - Resume { - arg: Arg<u32>, - }, - // If you add a new instruction here, be sure to keep LAST_INSTRUCTION updated -} - -// This must be kept up to date to avoid marshaling errors -const LAST_INSTRUCTION: Instruction = Instruction::Resume { arg: Arg::marker() }; - -const _: () = assert!(mem::size_of::<Instruction>() == 1); - -impl From<Instruction> for u8 { - #[inline] - fn from(ins: Instruction) -> Self { - // SAFETY: there's no padding bits - unsafe { core::mem::transmute::<Instruction, Self>(ins) } - } -} - -impl TryFrom<u8> for Instruction { - type Error = MarshalError; - - #[inline] - fn try_from(value: u8) -> Result<Self, MarshalError> { - if value <= u8::from(LAST_INSTRUCTION) { - Ok(unsafe { core::mem::transmute::<u8, Self>(value) }) - } else { - Err(MarshalError::InvalidBytecode) - } + pub struct CodeFlags: u32 { + const OPTIMIZED = 0x0001; + const NEWLOCALS = 0x0002; + const VARARGS = 0x0004; + const VARKEYWORDS = 0x0008; + const GENERATOR = 0x0020; + const COROUTINE = 0x0080; + const ITERABLE_COROUTINE = 0x0100; + /// If a code object represents a function and has a docstring, + /// this bit is set and the first item in co_consts is the docstring. + const HAS_DOCSTRING = 0x4000000; } } @@ -1201,31 +371,6 @@ impl Deref for CodeUnits { } } -use self::Instruction::*; - -bitflags! { - #[derive(Copy, Clone, Debug, PartialEq)] - pub struct MakeFunctionFlags: u8 { - const CLOSURE = 0x01; - const ANNOTATIONS = 0x02; - const KW_ONLY_DEFAULTS = 0x04; - const DEFAULTS = 0x08; - const TYPE_PARAMS = 0x10; - } -} - -impl OpArgType for MakeFunctionFlags { - #[inline(always)] - fn from_op_arg(x: u32) -> Option<Self> { - Self::from_bits(x as u8) - } - - #[inline(always)] - fn to_op_arg(self) -> u32 { - self.bits().into() - } -} - /// A Constant (which usually encapsulates data within it) /// /// # Examples @@ -1381,245 +526,6 @@ impl<C: Constant> BorrowedConstant<'_, C> { } } -op_arg_enum!( - /// The possible comparison operators - #[derive(Debug, Copy, Clone, PartialEq, Eq)] - #[repr(u8)] - pub enum ComparisonOperator { - // be intentional with bits so that we can do eval_ord with just a bitwise and - // bits: | Equal | Greater | Less | - Less = 0b001, - Greater = 0b010, - NotEqual = 0b011, - Equal = 0b100, - LessOrEqual = 0b101, - GreaterOrEqual = 0b110, - } -); - -op_arg_enum!( - /// The possible Binary operators - /// - /// # Examples - /// - /// ```rust - /// use rustpython_compiler_core::bytecode::{Arg, BinaryOperator, Instruction}; - /// let (op, _) = Arg::new(BinaryOperator::Add); - /// let instruction = Instruction::BinaryOp { op }; - /// ``` - /// - /// See also: - /// - [_PyEval_BinaryOps](https://github.com/python/cpython/blob/8183fa5e3f78ca6ab862de7fb8b14f3d929421e0/Python/ceval.c#L316-L343) - #[repr(u8)] - #[derive(Clone, Copy, Debug, Eq, PartialEq)] - pub enum BinaryOperator { - /// `+` - Add = 0, - /// `&` - And = 1, - /// `//` - FloorDivide = 2, - /// `<<` - Lshift = 3, - /// `@` - MatrixMultiply = 4, - /// `*` - Multiply = 5, - /// `%` - Remainder = 6, - /// `|` - Or = 7, - /// `**` - Power = 8, - /// `>>` - Rshift = 9, - /// `-` - Subtract = 10, - /// `/` - TrueDivide = 11, - /// `^` - Xor = 12, - /// `+=` - InplaceAdd = 13, - /// `&=` - InplaceAnd = 14, - /// `//=` - InplaceFloorDivide = 15, - /// `<<=` - InplaceLshift = 16, - /// `@=` - InplaceMatrixMultiply = 17, - /// `*=` - InplaceMultiply = 18, - /// `%=` - InplaceRemainder = 19, - /// `|=` - InplaceOr = 20, - /// `**=` - InplacePower = 21, - /// `>>=` - InplaceRshift = 22, - /// `-=` - InplaceSubtract = 23, - /// `/=` - InplaceTrueDivide = 24, - /// `^=` - InplaceXor = 25, - } -); - -impl BinaryOperator { - /// Get the "inplace" version of the operator. - /// This has no effect if `self` is already an "inplace" operator. - /// - /// # Example - /// ```rust - /// use rustpython_compiler_core::bytecode::BinaryOperator; - /// - /// assert_eq!(BinaryOperator::Power.as_inplace(), BinaryOperator::InplacePower); - /// - /// assert_eq!(BinaryOperator::InplaceSubtract.as_inplace(), BinaryOperator::InplaceSubtract); - /// ``` - #[must_use] - pub const fn as_inplace(self) -> Self { - match self { - Self::Add => Self::InplaceAdd, - Self::And => Self::InplaceAnd, - Self::FloorDivide => Self::InplaceFloorDivide, - Self::Lshift => Self::InplaceLshift, - Self::MatrixMultiply => Self::InplaceMatrixMultiply, - Self::Multiply => Self::InplaceMultiply, - Self::Remainder => Self::InplaceRemainder, - Self::Or => Self::InplaceOr, - Self::Power => Self::InplacePower, - Self::Rshift => Self::InplaceRshift, - Self::Subtract => Self::InplaceSubtract, - Self::TrueDivide => Self::InplaceTrueDivide, - Self::Xor => Self::InplaceXor, - _ => self, - } - } -} - -impl fmt::Display for BinaryOperator { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - let op = match self { - Self::Add => "+", - Self::And => "&", - Self::FloorDivide => "//", - Self::Lshift => "<<", - Self::MatrixMultiply => "@", - Self::Multiply => "*", - Self::Remainder => "%", - Self::Or => "|", - Self::Power => "**", - Self::Rshift => ">>", - Self::Subtract => "-", - Self::TrueDivide => "/", - Self::Xor => "^", - Self::InplaceAdd => "+=", - Self::InplaceAnd => "&=", - Self::InplaceFloorDivide => "//=", - Self::InplaceLshift => "<<=", - Self::InplaceMatrixMultiply => "@=", - Self::InplaceMultiply => "*=", - Self::InplaceRemainder => "%=", - Self::InplaceOr => "|=", - Self::InplacePower => "**=", - Self::InplaceRshift => ">>=", - Self::InplaceSubtract => "-=", - Self::InplaceTrueDivide => "/=", - Self::InplaceXor => "^=", - }; - write!(f, "{op}") - } -} - -op_arg_enum!( - /// Whether or not to invert the operation. - #[repr(u8)] - #[derive(Debug, Copy, Clone, PartialEq, Eq)] - pub enum Invert { - /// ```py - /// foo is bar - /// x in lst - /// ``` - No = 0, - /// ```py - /// foo is not bar - /// x not in lst - /// ``` - Yes = 1, - } -); - -/// Specifies if a slice is built with either 2 or 3 arguments. -#[derive(Clone, Copy, Debug, Eq, PartialEq)] -pub enum BuildSliceArgCount { - /// ```py - /// x[5:10] - /// ``` - Two, - /// ```py - /// x[5:10:2] - /// ``` - Three, -} - -impl OpArgType for BuildSliceArgCount { - #[inline(always)] - fn from_op_arg(x: u32) -> Option<Self> { - Some(match x { - 2 => Self::Two, - 3 => Self::Three, - _ => return None, - }) - } - - #[inline(always)] - fn to_op_arg(self) -> u32 { - u32::from(self.argc().get()) - } -} - -impl BuildSliceArgCount { - /// Get the numeric value of `Self`. - #[must_use] - pub const fn argc(self) -> NonZeroU8 { - let inner = match self { - Self::Two => 2, - Self::Three => 3, - }; - // Safety: `inner` can be either 2 or 3. - unsafe { NonZeroU8::new_unchecked(inner) } - } -} - -#[derive(Copy, Clone)] -pub struct UnpackExArgs { - pub before: u8, - pub after: u8, -} - -impl OpArgType for UnpackExArgs { - #[inline(always)] - fn from_op_arg(x: u32) -> Option<Self> { - let [before, after, ..] = x.to_le_bytes(); - Some(Self { before, after }) - } - - #[inline(always)] - fn to_op_arg(self) -> u32 { - u32::from_le_bytes([self.before, self.after, 0, 0]) - } -} - -impl fmt::Display for UnpackExArgs { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - write!(f, "before: {}, after: {}", self.before, self.after) - } -} - /* Maintain a stack of blocks on the VM. pub enum BlockType { @@ -1665,14 +571,14 @@ impl<C: Constant> CodeObject<C> { let args = &self.varnames[..nargs]; let kwonlyargs = &self.varnames[nargs..varargs_pos]; - let vararg = if self.flags.contains(CodeFlags::HAS_VARARGS) { + let vararg = if self.flags.contains(CodeFlags::VARARGS) { let vararg = &self.varnames[varargs_pos]; varargs_pos += 1; Some(vararg) } else { None }; - let varkwarg = if self.flags.contains(CodeFlags::HAS_VARKEYWORDS) { + let varkwarg = if self.flags.contains(CodeFlags::VARKEYWORDS) { Some(&self.varnames[varargs_pos]) } else { None @@ -1843,458 +749,6 @@ impl<C: Constant> fmt::Display for CodeObject<C> { } } -impl Instruction { - /// Gets the label stored inside this instruction, if it exists - #[inline] - pub const fn label_arg(&self) -> Option<Arg<Label>> { - match self { - Jump { target: l } - | JumpIfNotExcMatch(l) - | PopJumpIfTrue { target: l } - | PopJumpIfFalse { target: l } - | JumpIfTrueOrPop { target: l } - | JumpIfFalseOrPop { target: l } - | ForIter { target: l } - | Break { target: l } - | Continue { target: l } - | Send { target: l } - // Dummy jump instructions (placeholders) - | JumpBackward { target: l } - | JumpBackwardNoInterrupt { target: l } - | JumpForward { target: l } - | PopJumpIfNone { target: l } - | PopJumpIfNotNone { target: l } => Some(*l), - _ => None, - } - } - - /// Whether this is an unconditional branching - /// - /// # Examples - /// - /// ``` - /// use rustpython_compiler_core::bytecode::{Arg, Instruction}; - /// let jump_inst = Instruction::Jump { target: Arg::marker() }; - /// assert!(jump_inst.unconditional_branch()) - /// ``` - pub const fn unconditional_branch(&self) -> bool { - matches!( - self, - Jump { .. } - | Continue { .. } - | Break { .. } - | ReturnValue - | ReturnConst { .. } - | Raise { .. } - | Reraise { .. } - ) - } - - /// What effect this instruction has on the stack - /// - /// # Examples - /// - /// ``` - /// use rustpython_compiler_core::bytecode::{Arg, Instruction, Label}; - /// let (target, jump_arg) = Arg::new(Label(0xF)); - /// let jump_instruction = Instruction::Jump { target }; - /// assert_eq!(jump_instruction.stack_effect(jump_arg, true), 0); - /// ``` - /// - pub fn stack_effect(&self, arg: OpArg, jump: bool) -> i32 { - match self { - // Dummy/placeholder instructions (never executed) - Cache | Reserved3 | Reserved17 | Reserved140 | Reserved141 | Reserved142 - | Reserved143 | Reserved144 | Reserved145 | Reserved146 | Reserved147 | Reserved148 => { - 0 - } - BinarySlice | EndFor | ExitInitCheck | GetYieldFromIter | InterpreterExit - | LoadAssertionError | LoadLocals | PushNull | ReturnGenerator | StoreSlice => 0, - BuildConstKeyMap { .. } - | CopyFreeVars { .. } - | DictMerge { .. } - | EnterExecutor { .. } - | ListExtend { .. } - | LoadFastCheck(_) - | LoadFastLoadFast { .. } - | LoadFromDictOrDeref(_) - | LoadFromDictOrGlobals(_) - | LoadSuperAttr { .. } - | MakeCell(_) - | SetUpdate { .. } - | StoreFastStoreFast { .. } => 0, - JumpBackward { .. } | JumpBackwardNoInterrupt { .. } | JumpForward { .. } => 0, - PopJumpIfNone { .. } | PopJumpIfNotNone { .. } => -1, - - Nop => 0, - ImportName { .. } => -1, - ImportFrom { .. } => 1, - LoadFast(_) | LoadFastAndClear(_) | LoadNameAny(_) | LoadGlobal(_) | LoadDeref(_) - | LoadClassDeref(_) => 1, - StoreFast(_) | StoreLocal(_) | StoreGlobal(_) | StoreDeref(_) => -1, - StoreFastLoadFast { .. } => 0, // pop 1, push 1 - DeleteFast(_) | DeleteLocal(_) | DeleteGlobal(_) | DeleteDeref(_) => 0, - LoadClosure(_) => 1, - Subscript => -1, - StoreSubscript => -3, - DeleteSubscript => -2, - LoadAttr { .. } => 0, - StoreAttr { .. } => -2, - DeleteAttr { .. } => -1, - LoadConst { .. } => 1, - BinaryOp { .. } | CompareOperation { .. } => -1, - BinarySubscript => -1, - CopyItem { .. } => 1, - PopTop => -1, - Swap { .. } => 0, - ToBool => 0, - GetIter => 0, - GetLen => 1, - CallIntrinsic1 { .. } => 0, // Takes 1, pushes 1 - CallIntrinsic2 { .. } => -1, // Takes 2, pushes 1 - Continue { .. } => 0, - Break { .. } => 0, - Jump { .. } => 0, - PopJumpIfTrue { .. } | PopJumpIfFalse { .. } => -1, - JumpIfTrueOrPop { .. } | JumpIfFalseOrPop { .. } => { - if jump { - 0 - } else { - -1 - } - } - MakeFunction => { - // CPython 3.13 style: MakeFunction only pops code object - -1 + 1 // pop code, push function - } - SetFunctionAttribute { .. } => { - // pops attribute value and function, pushes function back - -2 + 1 - } - CallFunctionPositional { nargs } => -(nargs.get(arg) as i32) - 1 + 1, - CallMethodPositional { nargs } => -(nargs.get(arg) as i32) - 3 + 1, - CallFunctionKeyword { nargs } => -1 - (nargs.get(arg) as i32) - 1 + 1, - CallMethodKeyword { nargs } => -1 - (nargs.get(arg) as i32) - 3 + 1, - CallFunctionEx { has_kwargs } => -1 - (has_kwargs.get(arg) as i32) - 1 + 1, - CallMethodEx { has_kwargs } => -1 - (has_kwargs.get(arg) as i32) - 3 + 1, - CheckEgMatch => 0, // pops 2 (exc, type), pushes 2 (rest, match) - ConvertValue { .. } => 0, - FormatSimple => 0, - FormatWithSpec => -1, - LoadMethod { .. } => -1 + 3, - ForIter { .. } => { - if jump { - -1 - } else { - 1 - } - } - IsOp(_) | ContainsOp(_) => -1, - JumpIfNotExcMatch(_) => -2, - ReturnValue => -1, - ReturnConst { .. } => 0, - Resume { .. } => 0, - YieldValue { .. } => 0, - // SEND: (receiver, val) -> (receiver, retval) - no change, both paths keep same depth - Send { .. } => 0, - // END_SEND: (receiver, value) -> (value) - EndSend => -1, - // CLEANUP_THROW: (sub_iter, last_sent_val, exc) -> (None, value) = 3 pop, 2 push = -1 - CleanupThrow => -1, - SetExcInfo => 0, - PushExcInfo => 1, // [exc] -> [prev_exc, exc] - CheckExcMatch => 0, // [exc, type] -> [exc, bool] (pops type, pushes bool) - Reraise { .. } => 0, // Exception raised, stack effect doesn't matter - SetupAnnotation => 0, - BeforeWith => 1, // push __exit__, then replace ctx_mgr with __enter__ result - WithExceptStart => 1, // push __exit__ result - PopBlock => 0, - Raise { kind } => { - // Stack effects for different raise kinds: - // - Reraise (0): gets from VM state, no stack pop - // - Raise (1): pops 1 exception - // - RaiseCause (2): pops 2 (exception + cause) - // - ReraiseFromStack (3): pops 1 exception from stack - match kind.get(arg) { - RaiseKind::BareRaise => 0, - RaiseKind::Raise => -1, - RaiseKind::RaiseCause => -2, - RaiseKind::ReraiseFromStack => -1, - } - } - BuildString { size } - | BuildTuple { size, .. } - | BuildTupleFromTuples { size, .. } - | BuildList { size, .. } - | BuildListFromTuples { size, .. } - | BuildSet { size, .. } - | BuildSetFromTuples { size, .. } => -(size.get(arg) as i32) + 1, - BuildTupleFromIter => 0, - BuildMap { size } => { - let nargs = size.get(arg) * 2; - -(nargs as i32) + 1 - } - BuildMapForCall { size } => { - let nargs = size.get(arg); - -(nargs as i32) + 1 - } - DictUpdate { .. } => -1, - BuildSlice { argc } => { - // push 1 - // pops either 2/3 - // Default to Two (2 args) if arg is invalid - 1 - (argc - .try_get(arg) - .unwrap_or(BuildSliceArgCount::Two) - .argc() - .get() as i32) - } - ListAppend { .. } | SetAdd { .. } => -1, - MapAdd { .. } => -2, - LoadBuildClass => 1, - UnpackSequence { size } => -1 + size.get(arg) as i32, - UnpackEx { args } => { - let UnpackExArgs { before, after } = args.get(arg); - -1 + before as i32 + 1 + after as i32 - } - PopException => 0, - Reverse { .. } => 0, - GetAwaitable => 0, - BeforeAsyncWith => 1, - GetAIter => 0, - GetANext => 1, - EndAsyncFor => -2, // pops (awaitable, exc) from stack - MatchMapping | MatchSequence => 1, // Push bool result - MatchKeys => 1, // Pop 2 (subject, keys), push 3 (subject, keys_or_none, values_or_none) - MatchClass(_) => -2, - ExtendedArg => 0, - UnaryInvert => 0, - UnaryNegative => 0, - UnaryNot => 0, - } - } - - pub fn display<'a>( - &'a self, - arg: OpArg, - ctx: &'a impl InstrDisplayContext, - ) -> impl fmt::Display + 'a { - struct FmtFn<F>(F); - impl<F: Fn(&mut fmt::Formatter<'_>) -> fmt::Result> fmt::Display for FmtFn<F> { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - (self.0)(f) - } - } - FmtFn(move |f: &mut fmt::Formatter<'_>| self.fmt_dis(arg, f, ctx, false, 0, 0)) - } - - #[allow(clippy::too_many_arguments)] - fn fmt_dis( - &self, - arg: OpArg, - f: &mut fmt::Formatter<'_>, - ctx: &impl InstrDisplayContext, - expand_code_objects: bool, - pad: usize, - level: usize, - ) -> fmt::Result { - macro_rules! w { - ($variant:ident) => { - write!(f, stringify!($variant)) - }; - ($variant:ident, $map:ident = $arg_marker:expr) => {{ - let arg = $arg_marker.get(arg); - write!(f, "{:pad$}({}, {})", stringify!($variant), arg, $map(arg)) - }}; - ($variant:ident, $arg_marker:expr) => { - write!(f, "{:pad$}({})", stringify!($variant), $arg_marker.get(arg)) - }; - ($variant:ident, ?$arg_marker:expr) => { - write!( - f, - "{:pad$}({:?})", - stringify!($variant), - $arg_marker.get(arg) - ) - }; - } - - let varname = |i: u32| ctx.get_varname(i as usize); - let name = |i: u32| ctx.get_name(i as usize); - let cell_name = |i: u32| ctx.get_cell_name(i as usize); - - let fmt_const = - |op: &str, arg: OpArg, f: &mut fmt::Formatter<'_>, idx: &Arg<u32>| -> fmt::Result { - let value = ctx.get_constant(idx.get(arg) as usize); - match value.borrow_constant() { - BorrowedConstant::Code { code } if expand_code_objects => { - write!(f, "{op:pad$}({code:?}):")?; - code.display_inner(f, true, level + 1)?; - Ok(()) - } - c => { - write!(f, "{op:pad$}(")?; - c.fmt_display(f)?; - write!(f, ")") - } - } - }; - - match self { - // Dummy/placeholder instructions - Cache => w!(CACHE), - Reserved3 | Reserved17 | Reserved140 | Reserved141 | Reserved142 | Reserved143 - | Reserved144 | Reserved145 | Reserved146 | Reserved147 | Reserved148 => w!(RESERVED), - BinarySlice => w!(BINARY_SLICE), - EndFor => w!(END_FOR), - ExitInitCheck => w!(EXIT_INIT_CHECK), - GetYieldFromIter => w!(GET_YIELD_FROM_ITER), - InterpreterExit => w!(INTERPRETER_EXIT), - LoadAssertionError => w!(LOAD_ASSERTION_ERROR), - LoadLocals => w!(LOAD_LOCALS), - PushNull => w!(PUSH_NULL), - ReturnGenerator => w!(RETURN_GENERATOR), - StoreSlice => w!(STORE_SLICE), - UnaryInvert => w!(UNARY_INVERT), - UnaryNegative => w!(UNARY_NEGATIVE), - UnaryNot => w!(UNARY_NOT), - BuildConstKeyMap { size } => w!(BUILD_CONST_KEY_MAP, size), - CopyFreeVars { count } => w!(COPY_FREE_VARS, count), - DictMerge { index } => w!(DICT_MERGE, index), - EnterExecutor { index } => w!(ENTER_EXECUTOR, index), - JumpBackward { target } => w!(JUMP_BACKWARD, target), - JumpBackwardNoInterrupt { target } => w!(JUMP_BACKWARD_NO_INTERRUPT, target), - JumpForward { target } => w!(JUMP_FORWARD, target), - ListExtend { i } => w!(LIST_EXTEND, i), - LoadFastCheck(idx) => w!(LOAD_FAST_CHECK, varname = idx), - LoadFastLoadFast { arg } => w!(LOAD_FAST_LOAD_FAST, arg), - LoadFromDictOrDeref(idx) => w!(LOAD_FROM_DICT_OR_DEREF, cell_name = idx), - LoadFromDictOrGlobals(idx) => w!(LOAD_FROM_DICT_OR_GLOBALS, name = idx), - LoadSuperAttr { arg } => w!(LOAD_SUPER_ATTR, arg), - MakeCell(idx) => w!(MAKE_CELL, cell_name = idx), - PopJumpIfNone { target } => w!(POP_JUMP_IF_NONE, target), - PopJumpIfNotNone { target } => w!(POP_JUMP_IF_NOT_NONE, target), - SetUpdate { i } => w!(SET_UPDATE, i), - StoreFastStoreFast { arg } => w!(STORE_FAST_STORE_FAST, arg), - - // Real instructions - BeforeAsyncWith => w!(BEFORE_ASYNC_WITH), - BeforeWith => w!(BEFORE_WITH), - BinaryOp { op } => write!(f, "{:pad$}({})", "BINARY_OP", op.get(arg)), - BinarySubscript => w!(BINARY_SUBSCR), - Break { target } => w!(BREAK, target), - BuildList { size } => w!(BUILD_LIST, size), - BuildListFromTuples { size } => w!(BUILD_LIST_FROM_TUPLES, size), - BuildMap { size } => w!(BUILD_MAP, size), - BuildMapForCall { size } => w!(BUILD_MAP_FOR_CALL, size), - BuildSet { size } => w!(BUILD_SET, size), - BuildSetFromTuples { size } => w!(BUILD_SET_FROM_TUPLES, size), - BuildSlice { argc } => w!(BUILD_SLICE, ?argc), - BuildString { size } => w!(BUILD_STRING, size), - BuildTuple { size } => w!(BUILD_TUPLE, size), - BuildTupleFromIter => w!(BUILD_TUPLE_FROM_ITER), - BuildTupleFromTuples { size } => w!(BUILD_TUPLE_FROM_TUPLES, size), - CallFunctionEx { has_kwargs } => w!(CALL_FUNCTION_EX, has_kwargs), - CallFunctionKeyword { nargs } => w!(CALL_KW, nargs), - CallFunctionPositional { nargs } => w!(CALL, nargs), - CallIntrinsic1 { func } => w!(CALL_INTRINSIC_1, ?func), - CallIntrinsic2 { func } => w!(CALL_INTRINSIC_2, ?func), - CallMethodEx { has_kwargs } => w!(CALL_METHOD_EX, has_kwargs), - CallMethodKeyword { nargs } => w!(CALL_METHOD_KW, nargs), - CallMethodPositional { nargs } => w!(CALL_METHOD, nargs), - CheckEgMatch => w!(CHECK_EG_MATCH), - CheckExcMatch => w!(CHECK_EXC_MATCH), - CleanupThrow => w!(CLEANUP_THROW), - CompareOperation { op } => w!(COMPARE_OP, ?op), - ContainsOp(inv) => w!(CONTAINS_OP, ?inv), - Continue { target } => w!(CONTINUE, target), - ConvertValue { oparg } => write!(f, "{:pad$}{}", "CONVERT_VALUE", oparg.get(arg)), - CopyItem { index } => w!(COPY, index), - DeleteAttr { idx } => w!(DELETE_ATTR, name = idx), - DeleteDeref(idx) => w!(DELETE_DEREF, cell_name = idx), - DeleteFast(idx) => w!(DELETE_FAST, varname = idx), - DeleteGlobal(idx) => w!(DELETE_GLOBAL, name = idx), - DeleteLocal(idx) => w!(DELETE_NAME, name = idx), - DeleteSubscript => w!(DELETE_SUBSCR), - DictUpdate { index } => w!(DICT_UPDATE, index), - EndAsyncFor => w!(END_ASYNC_FOR), - EndSend => w!(END_SEND), - ExtendedArg => w!(EXTENDED_ARG, Arg::<u32>::marker()), - ForIter { target } => w!(FOR_ITER, target), - FormatSimple => w!(FORMAT_SIMPLE), - FormatWithSpec => w!(FORMAT_WITH_SPEC), - GetAIter => w!(GET_AITER), - GetANext => w!(GET_ANEXT), - GetAwaitable => w!(GET_AWAITABLE), - GetIter => w!(GET_ITER), - GetLen => w!(GET_LEN), - ImportFrom { idx } => w!(IMPORT_FROM, name = idx), - ImportName { idx } => w!(IMPORT_NAME, name = idx), - IsOp(inv) => w!(IS_OP, ?inv), - Jump { target } => w!(JUMP, target), - JumpIfFalseOrPop { target } => w!(JUMP_IF_FALSE_OR_POP, target), - JumpIfNotExcMatch(target) => w!(JUMP_IF_NOT_EXC_MATCH, target), - JumpIfTrueOrPop { target } => w!(JUMP_IF_TRUE_OR_POP, target), - ListAppend { i } => w!(LIST_APPEND, i), - LoadAttr { idx } => w!(LOAD_ATTR, name = idx), - LoadBuildClass => w!(LOAD_BUILD_CLASS), - LoadClassDeref(idx) => w!(LOAD_CLASSDEREF, cell_name = idx), - LoadClosure(i) => w!(LOAD_CLOSURE, cell_name = i), - LoadConst { idx } => fmt_const("LOAD_CONST", arg, f, idx), - LoadDeref(idx) => w!(LOAD_DEREF, cell_name = idx), - LoadFast(idx) => w!(LOAD_FAST, varname = idx), - LoadFastAndClear(idx) => w!(LOAD_FAST_AND_CLEAR, varname = idx), - LoadGlobal(idx) => w!(LOAD_GLOBAL, name = idx), - LoadMethod { idx } => w!(LOAD_METHOD, name = idx), - LoadNameAny(idx) => w!(LOAD_NAME, name = idx), - MakeFunction => w!(MAKE_FUNCTION), - MapAdd { i } => w!(MAP_ADD, i), - MatchClass(arg) => w!(MATCH_CLASS, arg), - MatchKeys => w!(MATCH_KEYS), - MatchMapping => w!(MATCH_MAPPING), - MatchSequence => w!(MATCH_SEQUENCE), - Nop => w!(NOP), - PopBlock => w!(POP_BLOCK), - PopException => w!(POP_EXCEPT), - PopJumpIfFalse { target } => w!(POP_JUMP_IF_FALSE, target), - PopJumpIfTrue { target } => w!(POP_JUMP_IF_TRUE, target), - PopTop => w!(POP_TOP), - PushExcInfo => w!(PUSH_EXC_INFO), - Raise { kind } => w!(RAISE_VARARGS, ?kind), - Reraise { depth } => w!(RERAISE, depth), - Resume { arg } => w!(RESUME, arg), - ReturnConst { idx } => fmt_const("RETURN_CONST", arg, f, idx), - ReturnValue => w!(RETURN_VALUE), - Reverse { amount } => w!(REVERSE, amount), - Send { target } => w!(SEND, target), - SetAdd { i } => w!(SET_ADD, i), - SetExcInfo => w!(SET_EXC_INFO), - SetFunctionAttribute { attr } => w!(SET_FUNCTION_ATTRIBUTE, ?attr), - SetupAnnotation => w!(SETUP_ANNOTATIONS), - StoreAttr { idx } => w!(STORE_ATTR, name = idx), - StoreDeref(idx) => w!(STORE_DEREF, cell_name = idx), - StoreFast(idx) => w!(STORE_FAST, varname = idx), - StoreFastLoadFast { - store_idx, - load_idx, - } => { - write!(f, "STORE_FAST_LOAD_FAST")?; - write!(f, " ({}, {})", store_idx.get(arg), load_idx.get(arg)) - } - StoreGlobal(idx) => w!(STORE_GLOBAL, name = idx), - StoreLocal(idx) => w!(STORE_NAME, name = idx), - StoreSubscript => w!(STORE_SUBSCR), - Subscript => w!(SUBSCRIPT), - Swap { index } => w!(SWAP, index), - ToBool => w!(TO_BOOL), - UnpackEx { args } => w!(UNPACK_EX, args), - UnpackSequence { size } => w!(UNPACK_SEQUENCE, size), - WithExceptStart => w!(WITH_EXCEPT_START), - YieldValue { arg } => w!(YIELD_VALUE, arg), - } - } -} - pub trait InstrDisplayContext { type Constant: Constant; diff --git a/crates/compiler-core/src/bytecode/instruction.rs b/crates/compiler-core/src/bytecode/instruction.rs new file mode 100644 index 00000000000..ee36220b538 --- /dev/null +++ b/crates/compiler-core/src/bytecode/instruction.rs @@ -0,0 +1,1321 @@ +use core::{fmt, marker::PhantomData, mem}; + +use crate::{ + bytecode::{ + BorrowedConstant, Constant, InstrDisplayContext, + oparg::{ + BinaryOperator, BuildSliceArgCount, CommonConstant, ComparisonOperator, + ConvertValueOparg, IntrinsicFunction1, IntrinsicFunction2, Invert, Label, + MakeFunctionFlags, NameIdx, OpArg, OpArgByte, OpArgType, RaiseKind, SpecialMethod, + UnpackExArgs, + }, + }, + marshal::MarshalError, +}; + +/// A Single bytecode instruction that are executed by the VM. +/// +/// Currently aligned with CPython 3.14. +/// +/// ## See also +/// - [CPython opcode IDs](https://github.com/python/cpython/blob/v3.14.2/Include/opcode_ids.h) +#[derive(Clone, Copy, Debug)] +#[repr(u8)] +pub enum Instruction { + // No-argument instructions (opcode < HAVE_ARGUMENT=44) + Cache = 0, + BinarySlice = 1, + BuildTemplate = 2, + BinaryOpInplaceAddUnicode = 3, + CallFunctionEx = 4, + CheckEgMatch = 5, + CheckExcMatch = 6, + CleanupThrow = 7, + DeleteSubscr = 8, + EndFor = 9, + EndSend = 10, + ExitInitCheck = 11, // Placeholder + FormatSimple = 12, + FormatWithSpec = 13, + GetAIter = 14, + GetANext = 15, + GetIter = 16, + Reserved = 17, + GetLen = 18, + GetYieldFromIter = 19, + InterpreterExit = 20, // Placeholder + LoadBuildClass = 21, + LoadLocals = 22, + MakeFunction = 23, + MatchKeys = 24, + MatchMapping = 25, + MatchSequence = 26, + Nop = 27, + NotTaken = 28, + PopExcept = 29, + PopIter = 30, + PopTop = 31, + PushExcInfo = 32, + PushNull = 33, + ReturnGenerator = 34, + ReturnValue = 35, + SetupAnnotations = 36, + StoreSlice = 37, + StoreSubscr = 38, + ToBool = 39, + UnaryInvert = 40, + UnaryNegative = 41, + UnaryNot = 42, + WithExceptStart = 43, + // CPython 3.14 opcodes with arguments (44-120) + BinaryOp { + op: Arg<BinaryOperator>, + } = 44, + /// Build an Interpolation from value, expression string, and optional format_spec on stack. + /// + /// oparg encoding: (conversion << 2) | has_format_spec + /// - has_format_spec (bit 0): if 1, format_spec is on stack + /// - conversion (bits 2+): 0=None, 1=Str, 2=Repr, 3=Ascii + /// + /// Stack: [value, expression_str, format_spec?] -> [interpolation] + BuildInterpolation { + oparg: Arg<u32>, + } = 45, + BuildList { + size: Arg<u32>, + } = 46, + BuildMap { + size: Arg<u32>, + } = 47, + BuildSet { + size: Arg<u32>, + } = 48, + BuildSlice { + argc: Arg<BuildSliceArgCount>, + } = 49, + BuildString { + size: Arg<u32>, + } = 50, + BuildTuple { + size: Arg<u32>, + } = 51, + Call { + nargs: Arg<u32>, + } = 52, + CallIntrinsic1 { + func: Arg<IntrinsicFunction1>, + } = 53, + CallIntrinsic2 { + func: Arg<IntrinsicFunction2>, + } = 54, + CallKw { + nargs: Arg<u32>, + } = 55, + CompareOp { + op: Arg<ComparisonOperator>, + } = 56, + ContainsOp(Arg<Invert>) = 57, + ConvertValue { + oparg: Arg<ConvertValueOparg>, + } = 58, + Copy { + index: Arg<u32>, + } = 59, + CopyFreeVars { + count: Arg<u32>, + } = 60, + DeleteAttr { + idx: Arg<NameIdx>, + } = 61, + DeleteDeref(Arg<NameIdx>) = 62, + DeleteFast(Arg<NameIdx>) = 63, + DeleteGlobal(Arg<NameIdx>) = 64, + DeleteName(Arg<NameIdx>) = 65, + DictMerge { + index: Arg<u32>, + } = 66, + DictUpdate { + index: Arg<u32>, + } = 67, + EndAsyncFor = 68, + ExtendedArg = 69, + ForIter { + target: Arg<Label>, + } = 70, + GetAwaitable { + arg: Arg<u32>, + } = 71, + ImportFrom { + idx: Arg<NameIdx>, + } = 72, + ImportName { + idx: Arg<NameIdx>, + } = 73, + IsOp(Arg<Invert>) = 74, + JumpBackward { + target: Arg<Label>, + } = 75, + JumpBackwardNoInterrupt { + target: Arg<Label>, + } = 76, // Placeholder + JumpForward { + target: Arg<Label>, + } = 77, + ListAppend { + i: Arg<u32>, + } = 78, + ListExtend { + i: Arg<u32>, + } = 79, + LoadAttr { + idx: Arg<NameIdx>, + } = 80, + LoadCommonConstant { + idx: Arg<CommonConstant>, + } = 81, + LoadConst { + idx: Arg<u32>, + } = 82, + LoadDeref(Arg<NameIdx>) = 83, + LoadFast(Arg<NameIdx>) = 84, + LoadFastAndClear(Arg<NameIdx>) = 85, + LoadFastBorrow(Arg<NameIdx>) = 86, + LoadFastBorrowLoadFastBorrow { + arg: Arg<u32>, + } = 87, + LoadFastCheck(Arg<NameIdx>) = 88, + LoadFastLoadFast { + arg: Arg<u32>, + } = 89, + LoadFromDictOrDeref(Arg<NameIdx>) = 90, + LoadFromDictOrGlobals(Arg<NameIdx>) = 91, + LoadGlobal(Arg<NameIdx>) = 92, + LoadName(Arg<NameIdx>) = 93, + LoadSmallInt { + idx: Arg<u32>, + } = 94, + LoadSpecial { + method: Arg<SpecialMethod>, + } = 95, + LoadSuperAttr { + arg: Arg<u32>, + } = 96, + MakeCell(Arg<NameIdx>) = 97, + MapAdd { + i: Arg<u32>, + } = 98, + MatchClass(Arg<u32>) = 99, + PopJumpIfFalse { + target: Arg<Label>, + } = 100, + PopJumpIfNone { + target: Arg<Label>, + } = 101, + PopJumpIfNotNone { + target: Arg<Label>, + } = 102, + PopJumpIfTrue { + target: Arg<Label>, + } = 103, + RaiseVarargs { + kind: Arg<RaiseKind>, + } = 104, + Reraise { + depth: Arg<u32>, + } = 105, + Send { + target: Arg<Label>, + } = 106, + SetAdd { + i: Arg<u32>, + } = 107, + SetFunctionAttribute { + attr: Arg<MakeFunctionFlags>, + } = 108, + SetUpdate { + i: Arg<u32>, + } = 109, + StoreAttr { + idx: Arg<NameIdx>, + } = 110, + StoreDeref(Arg<NameIdx>) = 111, + StoreFast(Arg<NameIdx>) = 112, + StoreFastLoadFast { + store_idx: Arg<NameIdx>, + load_idx: Arg<NameIdx>, + } = 113, + StoreFastStoreFast { + arg: Arg<u32>, + } = 114, + StoreGlobal(Arg<NameIdx>) = 115, + StoreName(Arg<NameIdx>) = 116, + Swap { + index: Arg<u32>, + } = 117, + UnpackEx { + args: Arg<UnpackExArgs>, + } = 118, + UnpackSequence { + size: Arg<u32>, + } = 119, + YieldValue { + arg: Arg<u32>, + } = 120, + // CPython 3.14 RESUME (128) + Resume { + arg: Arg<u32>, + } = 128, + // CPython 3.14 specialized opcodes (129-211) + BinaryOpAddFloat = 129, // Placeholder + BinaryOpAddInt = 130, // Placeholder + BinaryOpAddUnicode = 131, // Placeholder + BinaryOpExtend = 132, // Placeholder + BinaryOpMultiplyFloat = 133, // Placeholder + BinaryOpMultiplyInt = 134, // Placeholder + BinaryOpSubscrDict = 135, // Placeholder + BinaryOpSubscrGetitem = 136, // Placeholder + BinaryOpSubscrListInt = 137, // Placeholder + BinaryOpSubscrListSlice = 138, // Placeholder + BinaryOpSubscrStrInt = 139, // Placeholder + BinaryOpSubscrTupleInt = 140, // Placeholder + BinaryOpSubtractFloat = 141, // Placeholder + BinaryOpSubtractInt = 142, // Placeholder + CallAllocAndEnterInit = 143, // Placeholder + CallBoundMethodExactArgs = 144, // Placeholder + CallBoundMethodGeneral = 145, // Placeholder + CallBuiltinClass = 146, // Placeholder + CallBuiltinFast = 147, // Placeholder + CallBuiltinFastWithKeywords = 148, // Placeholder + CallBuiltinO = 149, // Placeholder + CallIsinstance = 150, // Placeholder + CallKwBoundMethod = 151, // Placeholder + CallKwNonPy = 152, // Placeholder + CallKwPy = 153, // Placeholder + CallLen = 154, // Placeholder + CallListAppend = 155, // Placeholder + CallMethodDescriptorFast = 156, // Placeholder + CallMethodDescriptorFastWithKeywords = 157, // Placeholder + CallMethodDescriptorNoargs = 158, // Placeholder + CallMethodDescriptorO = 159, // Placeholder + CallNonPyGeneral = 160, // Placeholder + CallPyExactArgs = 161, // Placeholder + CallPyGeneral = 162, // Placeholder + CallStr1 = 163, // Placeholder + CallTuple1 = 164, // Placeholder + CallType1 = 165, // Placeholder + CompareOpFloat = 166, // Placeholder + CompareOpInt = 167, // Placeholder + CompareOpStr = 168, // Placeholder + ContainsOpDict = 169, // Placeholder + ContainsOpSet = 170, // Placeholder + ForIterGen = 171, // Placeholder + ForIterList = 172, // Placeholder + ForIterRange = 173, // Placeholder + ForIterTuple = 174, // Placeholder + JumpBackwardJit = 175, // Placeholder + JumpBackwardNoJit = 176, // Placeholder + LoadAttrClass = 177, // Placeholder + LoadAttrClassWithMetaclassCheck = 178, // Placeholder + LoadAttrGetattributeOverridden = 179, // Placeholder + LoadAttrInstanceValue = 180, // Placeholder + LoadAttrMethodLazyDict = 181, // Placeholder + LoadAttrMethodNoDict = 182, // Placeholder + LoadAttrMethodWithValues = 183, // Placeholder + LoadAttrModule = 184, // Placeholder + LoadAttrNondescriptorNoDict = 185, // Placeholder + LoadAttrNondescriptorWithValues = 186, // Placeholder + LoadAttrProperty = 187, // Placeholder + LoadAttrSlot = 188, // Placeholder + LoadAttrWithHint = 189, // Placeholder + LoadConstImmortal = 190, // Placeholder + LoadConstMortal = 191, // Placeholder + LoadGlobalBuiltin = 192, // Placeholder + LoadGlobalModule = 193, // Placeholder + LoadSuperAttrAttr = 194, // Placeholder + LoadSuperAttrMethod = 195, // Placeholder + ResumeCheck = 196, // Placeholder + SendGen = 197, // Placeholder + StoreAttrInstanceValue = 198, // Placeholder + StoreAttrSlot = 199, // Placeholder + StoreAttrWithHint = 200, // Placeholder + StoreSubscrDict = 201, // Placeholder + StoreSubscrListInt = 202, // Placeholder + ToBoolAlwaysTrue = 203, // Placeholder + ToBoolBool = 204, // Placeholder + ToBoolInt = 205, // Placeholder + ToBoolList = 206, // Placeholder + ToBoolNone = 207, // Placeholder + ToBoolStr = 208, // Placeholder + UnpackSequenceList = 209, // Placeholder + UnpackSequenceTuple = 210, // Placeholder + UnpackSequenceTwoTuple = 211, // Placeholder + // CPython 3.14 instrumented opcodes (234-254) + InstrumentedEndFor = 234, // Placeholder + InstrumentedPopIter = 235, // Placeholder + InstrumentedEndSend = 236, // Placeholder + InstrumentedForIter = 237, // Placeholder + InstrumentedInstruction = 238, // Placeholder + InstrumentedJumpForward = 239, // Placeholder + InstrumentedNotTaken = 240, + InstrumentedPopJumpIfTrue = 241, // Placeholder + InstrumentedPopJumpIfFalse = 242, // Placeholder + InstrumentedPopJumpIfNone = 243, // Placeholder + InstrumentedPopJumpIfNotNone = 244, // Placeholder + InstrumentedResume = 245, // Placeholder + InstrumentedReturnValue = 246, // Placeholder + InstrumentedYieldValue = 247, // Placeholder + InstrumentedEndAsyncFor = 248, // Placeholder + InstrumentedLoadSuperAttr = 249, // Placeholder + InstrumentedCall = 250, // Placeholder + InstrumentedCallKw = 251, // Placeholder + InstrumentedCallFunctionEx = 252, // Placeholder + InstrumentedJumpBackward = 253, // Placeholder + InstrumentedLine = 254, // Placeholder + EnterExecutor = 255, // Placeholder +} + +const _: () = assert!(mem::size_of::<Instruction>() == 1); + +impl From<Instruction> for u8 { + #[inline] + fn from(ins: Instruction) -> Self { + // SAFETY: there's no padding bits + unsafe { mem::transmute::<Instruction, Self>(ins) } + } +} + +impl TryFrom<u8> for Instruction { + type Error = MarshalError; + + #[inline] + fn try_from(value: u8) -> Result<Self, MarshalError> { + // CPython-compatible opcodes (0-120) + let cpython_start = u8::from(Self::Cache); + let cpython_end = u8::from(Self::YieldValue { arg: Arg::marker() }); + + // Resume has a non-contiguous opcode (128) + let resume_id = u8::from(Self::Resume { arg: Arg::marker() }); + let enter_executor_id = u8::from(Self::EnterExecutor); + + let specialized_start = u8::from(Self::BinaryOpAddFloat); + let specialized_end = u8::from(Self::UnpackSequenceTwoTuple); + + let instrumented_start = u8::from(Self::InstrumentedEndFor); + let instrumented_end = u8::from(Self::InstrumentedLine); + + // No RustPython-only opcodes anymore - all opcodes match CPython 3.14 + let custom_ops: &[u8] = &[]; + + if (cpython_start..=cpython_end).contains(&value) + || value == resume_id + || value == enter_executor_id + || custom_ops.contains(&value) + || (specialized_start..=specialized_end).contains(&value) + || (instrumented_start..=instrumented_end).contains(&value) + { + Ok(unsafe { mem::transmute::<u8, Self>(value) }) + } else { + Err(Self::Error::InvalidBytecode) + } + } +} + +impl InstructionMetadata for Instruction { + #[inline] + fn label_arg(&self) -> Option<Arg<Label>> { + match self { + Self::JumpBackward { target: l } + | Self::JumpBackwardNoInterrupt { target: l } + | Self::JumpForward { target: l } + | Self::PopJumpIfTrue { target: l } + | Self::PopJumpIfFalse { target: l } + | Self::PopJumpIfNone { target: l } + | Self::PopJumpIfNotNone { target: l } + | Self::ForIter { target: l } + | Self::Send { target: l } => Some(*l), + _ => None, + } + } + + fn is_unconditional_jump(&self) -> bool { + matches!( + self, + Self::JumpForward { .. } + | Self::JumpBackward { .. } + | Self::JumpBackwardNoInterrupt { .. } + ) + } + + fn is_scope_exit(&self) -> bool { + matches!( + self, + Self::ReturnValue | Self::RaiseVarargs { .. } | Self::Reraise { .. } + ) + } + + fn stack_effect_info(&self, oparg: u32) -> StackEffect { + // Reason for converting oparg to i32 is because of expressions like `1 + (oparg -1)` + // that causes underflow errors. + let oparg = i32::try_from(oparg).expect("oparg does not fit in an `i32`"); + + // NOTE: Please don't "simplify" expressions here (i.e. `1 + (oparg - 1)`) + // as it will be harder to see diff with what CPython auto-generates + let (pushed, popped) = match self { + Self::BinaryOp { .. } => (1, 2), + Self::BinaryOpAddFloat => (1, 2), + Self::BinaryOpAddInt => (1, 2), + Self::BinaryOpAddUnicode => (1, 2), + Self::BinaryOpExtend => (1, 2), + Self::BinaryOpInplaceAddUnicode => (0, 2), + Self::BinaryOpMultiplyFloat => (1, 2), + Self::BinaryOpMultiplyInt => (1, 2), + Self::BinaryOpSubscrDict => (1, 2), + Self::BinaryOpSubscrGetitem => (0, 2), + Self::BinaryOpSubscrListInt => (1, 2), + Self::BinaryOpSubscrListSlice => (1, 2), + Self::BinaryOpSubscrStrInt => (1, 2), + Self::BinaryOpSubscrTupleInt => (1, 2), + Self::BinaryOpSubtractFloat => (1, 2), + Self::BinaryOpSubtractInt => (1, 2), + Self::BinarySlice { .. } => (1, 3), + Self::BuildInterpolation { .. } => (1, 2 + (oparg & 1)), + Self::BuildList { .. } => (1, oparg), + Self::BuildMap { .. } => (1, oparg * 2), + Self::BuildSet { .. } => (1, oparg), + Self::BuildSlice { .. } => (1, oparg), + Self::BuildString { .. } => (1, oparg), + Self::BuildTemplate { .. } => (1, 2), + Self::BuildTuple { .. } => (1, oparg), + Self::Cache => (0, 0), + Self::Call { .. } => (1, 2 + oparg), + Self::CallAllocAndEnterInit => (0, 2 + oparg), + Self::CallBoundMethodExactArgs => (0, 2 + oparg), + Self::CallBoundMethodGeneral => (0, 2 + oparg), + Self::CallBuiltinClass => (1, 2 + oparg), + Self::CallBuiltinFast => (1, 2 + oparg), + Self::CallBuiltinFastWithKeywords => (1, 2 + oparg), + Self::CallBuiltinO => (1, 2 + oparg), + Self::CallFunctionEx => (1, 4), + Self::CallIntrinsic1 { .. } => (1, 1), + Self::CallIntrinsic2 { .. } => (1, 2), + Self::CallIsinstance => (1, 2 + oparg), + Self::CallKw { .. } => (1, 3 + oparg), + Self::CallKwBoundMethod => (0, 3 + oparg), + Self::CallKwNonPy => (1, 3 + oparg), + Self::CallKwPy => (0, 3 + oparg), + Self::CallLen => (1, 3), + Self::CallListAppend => (0, 3), + Self::CallMethodDescriptorFast => (1, 2 + oparg), + Self::CallMethodDescriptorFastWithKeywords => (1, 2 + oparg), + Self::CallMethodDescriptorNoargs => (1, 2 + oparg), + Self::CallMethodDescriptorO => (1, 2 + oparg), + Self::CallNonPyGeneral => (1, 2 + oparg), + Self::CallPyExactArgs => (0, 2 + oparg), + Self::CallPyGeneral => (0, 2 + oparg), + Self::CallStr1 => (1, 3), + Self::CallTuple1 => (1, 3), + Self::CallType1 => (1, 3), + Self::CheckEgMatch => (2, 2), + Self::CheckExcMatch => (2, 2), + Self::CleanupThrow => (2, 3), + Self::CompareOp { .. } => (1, 2), + Self::CompareOpFloat => (1, 2), + Self::CompareOpInt => (1, 2), + Self::CompareOpStr => (1, 2), + Self::ContainsOp(_) => (1, 2), + Self::ContainsOpDict => (1, 2), + Self::ContainsOpSet => (1, 2), + Self::ConvertValue { .. } => (1, 1), + Self::Copy { .. } => (2 + (oparg - 1), 1 + (oparg - 1)), + Self::CopyFreeVars { .. } => (0, 0), + Self::DeleteAttr { .. } => (0, 1), + Self::DeleteDeref(_) => (0, 0), + Self::DeleteFast(_) => (0, 0), + Self::DeleteGlobal(_) => (0, 0), + Self::DeleteName(_) => (0, 0), + Self::DeleteSubscr => (0, 2), + Self::DictMerge { .. } => (4 + (oparg - 1), 5 + (oparg - 1)), + Self::DictUpdate { .. } => (1 + (oparg - 1), 2 + (oparg - 1)), + Self::EndAsyncFor => (0, 2), + Self::EndFor => (0, 1), + Self::EndSend => (1, 2), + Self::EnterExecutor => (0, 0), + Self::ExitInitCheck => (0, 1), + Self::ExtendedArg => (0, 0), + Self::ForIter { .. } => (2, 1), + Self::ForIterGen => (1, 1), + Self::ForIterList => (2, 1), + Self::ForIterRange => (2, 1), + Self::ForIterTuple => (2, 1), + Self::FormatSimple => (1, 1), + Self::FormatWithSpec => (1, 2), + Self::GetAIter => (1, 1), + Self::GetANext => (2, 1), + Self::GetAwaitable { .. } => (1, 1), + Self::GetIter => (1, 1), + Self::GetLen => (2, 1), + Self::GetYieldFromIter => (1, 1), + Self::ImportFrom { .. } => (2, 1), + Self::ImportName { .. } => (1, 2), + Self::InstrumentedCall => (1, 2 + oparg), + Self::InstrumentedCallFunctionEx => (1, 4), + Self::InstrumentedCallKw => (1, 3 + oparg), + Self::InstrumentedEndAsyncFor => (0, 2), + Self::InstrumentedEndFor => (1, 2), + Self::InstrumentedEndSend => (1, 2), + Self::InstrumentedForIter => (2, 1), + Self::InstrumentedInstruction => (0, 0), + Self::InstrumentedJumpBackward => (0, 0), + Self::InstrumentedJumpForward => (0, 0), + Self::InstrumentedLine => (0, 0), + Self::InstrumentedLoadSuperAttr => (1 + (oparg & 1), 3), + Self::InstrumentedNotTaken => (0, 0), + Self::InstrumentedPopIter => (0, 1), + Self::InstrumentedPopJumpIfFalse => (0, 1), + Self::InstrumentedPopJumpIfNone => (0, 1), + Self::InstrumentedPopJumpIfNotNone => (0, 1), + Self::InstrumentedPopJumpIfTrue => (0, 1), + Self::InstrumentedResume => (0, 0), + Self::InstrumentedReturnValue => (1, 1), + Self::InstrumentedYieldValue => (1, 1), + Self::InterpreterExit => (0, 1), + Self::IsOp(_) => (1, 2), + Self::JumpBackward { .. } => (0, 0), + Self::JumpBackwardJit => (0, 0), + Self::JumpBackwardNoInterrupt { .. } => (0, 0), + Self::JumpBackwardNoJit => (0, 0), + Self::JumpForward { .. } => (0, 0), + Self::ListAppend { .. } => (1 + (oparg - 1), 2 + (oparg - 1)), + Self::ListExtend { .. } => (1 + (oparg - 1), 2 + (oparg - 1)), + Self::LoadAttr { .. } => (1 + (oparg & 1), 1), + Self::LoadAttrClass => (1 + (oparg & 1), 1), + Self::LoadAttrClassWithMetaclassCheck => (1 + (oparg & 1), 1), + Self::LoadAttrGetattributeOverridden => (1, 1), + Self::LoadAttrInstanceValue => (1 + (oparg & 1), 1), + Self::LoadAttrMethodLazyDict => (2, 1), + Self::LoadAttrMethodNoDict => (2, 1), + Self::LoadAttrMethodWithValues => (2, 1), + Self::LoadAttrModule => (1 + (oparg & 1), 1), + Self::LoadAttrNondescriptorNoDict => (1, 1), + Self::LoadAttrNondescriptorWithValues => (1, 1), + Self::LoadAttrProperty => (0, 1), + Self::LoadAttrSlot => (1 + (oparg & 1), 1), + Self::LoadAttrWithHint => (1 + (oparg & 1), 1), + Self::LoadBuildClass => (1, 0), + Self::LoadCommonConstant { .. } => (1, 0), + Self::LoadConst { .. } => (1, 0), + Self::LoadConstImmortal => (1, 0), + Self::LoadConstMortal => (1, 0), + Self::LoadDeref(_) => (1, 0), + Self::LoadFast(_) => (1, 0), + Self::LoadFastAndClear(_) => (1, 0), + Self::LoadFastBorrow(_) => (1, 0), + Self::LoadFastBorrowLoadFastBorrow { .. } => (2, 0), + Self::LoadFastCheck(_) => (1, 0), + Self::LoadFastLoadFast { .. } => (2, 0), + Self::LoadFromDictOrDeref(_) => (1, 1), + Self::LoadFromDictOrGlobals(_) => (1, 1), + Self::LoadGlobal(_) => ( + 1, // TODO: Differs from CPython `1 + (oparg & 1)` + 0, + ), + Self::LoadGlobalBuiltin => (1 + (oparg & 1), 0), + Self::LoadGlobalModule => (1 + (oparg & 1), 0), + Self::LoadLocals => (1, 0), + Self::LoadName(_) => (1, 0), + Self::LoadSmallInt { .. } => (1, 0), + Self::LoadSpecial { .. } => (1, 1), + Self::LoadSuperAttr { .. } => (1 + (oparg & 1), 3), + Self::LoadSuperAttrAttr => (1, 3), + Self::LoadSuperAttrMethod => (2, 3), + Self::MakeCell(_) => (0, 0), + Self::MakeFunction { .. } => (1, 1), + Self::MapAdd { .. } => (1 + (oparg - 1), 3 + (oparg - 1)), + Self::MatchClass { .. } => (1, 3), + Self::MatchKeys { .. } => (3, 2), + Self::MatchMapping => (2, 1), + Self::MatchSequence => (2, 1), + Self::Nop => (0, 0), + Self::NotTaken => (0, 0), + Self::PopExcept => (0, 1), + Self::PopIter => (0, 1), + Self::PopJumpIfFalse { .. } => (0, 1), + Self::PopJumpIfNone { .. } => (0, 1), + Self::PopJumpIfNotNone { .. } => (0, 1), + Self::PopJumpIfTrue { .. } => (0, 1), + Self::PopTop => (0, 1), + Self::PushExcInfo => (2, 1), + Self::PushNull => (1, 0), + Self::RaiseVarargs { kind } => ( + 0, + // TODO: Differs from CPython: `oparg` + match kind.get((oparg as u32).into()) { + RaiseKind::BareRaise => 0, + RaiseKind::Raise => 1, + RaiseKind::RaiseCause => 2, + RaiseKind::ReraiseFromStack => 1, + }, + ), + Self::Reraise { .. } => ( + 1 + oparg, // TODO: Differs from CPython: `oparg` + 1 + oparg, + ), + Self::Reserved => (0, 0), + Self::Resume { .. } => (0, 0), + Self::ResumeCheck => (0, 0), + Self::ReturnGenerator => (1, 0), + Self::ReturnValue => ( + 0, // TODO: Differs from CPython: `1` + 1, + ), + Self::Send { .. } => (2, 2), + Self::SendGen => (1, 2), + Self::SetAdd { .. } => (1 + (oparg - 1), 2 + (oparg - 1)), + Self::SetFunctionAttribute { .. } => (1, 2), + Self::SetUpdate { .. } => (1 + (oparg - 1), 2 + (oparg - 1)), + Self::SetupAnnotations => (0, 0), + Self::StoreAttr { .. } => (0, 2), + Self::StoreAttrInstanceValue => (0, 2), + Self::StoreAttrSlot => (0, 2), + Self::StoreAttrWithHint => (0, 2), + Self::StoreDeref(_) => (0, 1), + Self::StoreFast(_) => (0, 1), + Self::StoreFastLoadFast { .. } => (1, 1), + Self::StoreFastStoreFast { .. } => (0, 2), + Self::StoreGlobal(_) => (0, 1), + Self::StoreName(_) => (0, 1), + Self::StoreSlice => (0, 4), + Self::StoreSubscr => (0, 3), + Self::StoreSubscrDict => (0, 3), + Self::StoreSubscrListInt => (0, 3), + Self::Swap { .. } => (2 + (oparg - 2), 2 + (oparg - 2)), + Self::ToBool => (1, 1), + Self::ToBoolAlwaysTrue => (1, 1), + Self::ToBoolBool => (1, 1), + Self::ToBoolInt => (1, 1), + Self::ToBoolList => (1, 1), + Self::ToBoolNone => (1, 1), + Self::ToBoolStr => (1, 1), + Self::UnaryInvert => (1, 1), + Self::UnaryNegative => (1, 1), + Self::UnaryNot => (1, 1), + Self::UnpackEx { .. } => (1 + (oparg & 0xFF) + (oparg >> 8), 1), + Self::UnpackSequence { .. } => (oparg, 1), + Self::UnpackSequenceList => (oparg, 1), + Self::UnpackSequenceTuple => (oparg, 1), + Self::UnpackSequenceTwoTuple => (2, 1), + Self::WithExceptStart => (6, 5), + Self::YieldValue { .. } => (1, 1), + }; + + debug_assert!((0..=i32::MAX).contains(&pushed)); + debug_assert!((0..=i32::MAX).contains(&popped)); + + StackEffect::new(pushed as u32, popped as u32) + } + + #[allow(clippy::too_many_arguments)] + fn fmt_dis( + &self, + arg: OpArg, + f: &mut fmt::Formatter<'_>, + ctx: &impl InstrDisplayContext, + expand_code_objects: bool, + pad: usize, + level: usize, + ) -> fmt::Result { + macro_rules! w { + ($variant:ident) => { + write!(f, stringify!($variant)) + }; + ($variant:ident, $map:ident = $arg_marker:expr) => {{ + let arg = $arg_marker.get(arg); + write!(f, "{:pad$}({}, {})", stringify!($variant), arg, $map(arg)) + }}; + ($variant:ident, $arg_marker:expr) => { + write!(f, "{:pad$}({})", stringify!($variant), $arg_marker.get(arg)) + }; + ($variant:ident, ?$arg_marker:expr) => { + write!( + f, + "{:pad$}({:?})", + stringify!($variant), + $arg_marker.get(arg) + ) + }; + } + + let varname = |i: u32| ctx.get_varname(i as usize); + let name = |i: u32| ctx.get_name(i as usize); + let cell_name = |i: u32| ctx.get_cell_name(i as usize); + + let fmt_const = + |op: &str, arg: OpArg, f: &mut fmt::Formatter<'_>, idx: &Arg<u32>| -> fmt::Result { + let value = ctx.get_constant(idx.get(arg) as usize); + match value.borrow_constant() { + BorrowedConstant::Code { code } if expand_code_objects => { + write!(f, "{op:pad$}({code:?}):")?; + code.display_inner(f, true, level + 1)?; + Ok(()) + } + c => { + write!(f, "{op:pad$}(")?; + c.fmt_display(f)?; + write!(f, ")") + } + } + }; + + match self { + Self::BinaryOp { op } => write!(f, "{:pad$}({})", "BINARY_OP", op.get(arg)), + Self::BuildList { size } => w!(BUILD_LIST, size), + Self::BuildMap { size } => w!(BUILD_MAP, size), + Self::BuildSet { size } => w!(BUILD_SET, size), + Self::BuildSlice { argc } => w!(BUILD_SLICE, ?argc), + Self::BuildString { size } => w!(BUILD_STRING, size), + Self::BuildTuple { size } => w!(BUILD_TUPLE, size), + Self::Call { nargs } => w!(CALL, nargs), + Self::CallFunctionEx => w!(CALL_FUNCTION_EX), + Self::CallKw { nargs } => w!(CALL_KW, nargs), + Self::CallIntrinsic1 { func } => w!(CALL_INTRINSIC_1, ?func), + Self::CallIntrinsic2 { func } => w!(CALL_INTRINSIC_2, ?func), + Self::CheckEgMatch => w!(CHECK_EG_MATCH), + Self::CheckExcMatch => w!(CHECK_EXC_MATCH), + Self::CleanupThrow => w!(CLEANUP_THROW), + Self::CompareOp { op } => w!(COMPARE_OP, ?op), + Self::ContainsOp(inv) => w!(CONTAINS_OP, ?inv), + Self::ConvertValue { oparg } => write!(f, "{:pad$}{}", "CONVERT_VALUE", oparg.get(arg)), + Self::Copy { index } => w!(COPY, index), + Self::DeleteAttr { idx } => w!(DELETE_ATTR, name = idx), + Self::DeleteDeref(idx) => w!(DELETE_DEREF, cell_name = idx), + Self::DeleteFast(idx) => w!(DELETE_FAST, varname = idx), + Self::DeleteGlobal(idx) => w!(DELETE_GLOBAL, name = idx), + Self::DeleteName(idx) => w!(DELETE_NAME, name = idx), + Self::DeleteSubscr => w!(DELETE_SUBSCR), + Self::DictMerge { index } => w!(DICT_MERGE, index), + Self::DictUpdate { index } => w!(DICT_UPDATE, index), + Self::EndAsyncFor => w!(END_ASYNC_FOR), + Self::EndSend => w!(END_SEND), + Self::ExtendedArg => w!(EXTENDED_ARG, Arg::<u32>::marker()), + Self::ForIter { target } => w!(FOR_ITER, target), + Self::FormatSimple => w!(FORMAT_SIMPLE), + Self::FormatWithSpec => w!(FORMAT_WITH_SPEC), + Self::GetAIter => w!(GET_AITER), + Self::GetANext => w!(GET_ANEXT), + Self::GetAwaitable { arg } => w!(GET_AWAITABLE, arg), + Self::Reserved => w!(RESERVED), + Self::GetIter => w!(GET_ITER), + Self::GetLen => w!(GET_LEN), + Self::ImportFrom { idx } => w!(IMPORT_FROM, name = idx), + Self::ImportName { idx } => w!(IMPORT_NAME, name = idx), + Self::IsOp(inv) => w!(IS_OP, ?inv), + Self::JumpBackward { target } => w!(JUMP_BACKWARD, target), + Self::JumpBackwardNoInterrupt { target } => w!(JUMP_BACKWARD_NO_INTERRUPT, target), + Self::JumpForward { target } => w!(JUMP_FORWARD, target), + Self::ListAppend { i } => w!(LIST_APPEND, i), + Self::ListExtend { i } => w!(LIST_EXTEND, i), + Self::LoadAttr { idx } => { + let encoded = idx.get(arg); + let (name_idx, is_method) = decode_load_attr_arg(encoded); + let attr_name = name(name_idx); + if is_method { + write!( + f, + "{:pad$}({}, {}, method=true)", + "LOAD_ATTR", encoded, attr_name + ) + } else { + write!(f, "{:pad$}({}, {})", "LOAD_ATTR", encoded, attr_name) + } + } + Self::LoadBuildClass => w!(LOAD_BUILD_CLASS), + Self::LoadFromDictOrDeref(i) => w!(LOAD_FROM_DICT_OR_DEREF, cell_name = i), + Self::LoadConst { idx } => fmt_const("LOAD_CONST", arg, f, idx), + Self::LoadSmallInt { idx } => w!(LOAD_SMALL_INT, idx), + Self::LoadDeref(idx) => w!(LOAD_DEREF, cell_name = idx), + Self::LoadFast(idx) => w!(LOAD_FAST, varname = idx), + Self::LoadFastAndClear(idx) => w!(LOAD_FAST_AND_CLEAR, varname = idx), + Self::LoadFastBorrow(idx) => w!(LOAD_FAST_BORROW, varname = idx), + Self::LoadFastCheck(idx) => w!(LOAD_FAST_CHECK, varname = idx), + Self::LoadFastLoadFast { arg: packed } => { + let oparg = packed.get(arg); + let idx1 = oparg >> 4; + let idx2 = oparg & 15; + let name1 = varname(idx1); + let name2 = varname(idx2); + write!(f, "{:pad$}({}, {})", "LOAD_FAST_LOAD_FAST", name1, name2) + } + Self::LoadFastBorrowLoadFastBorrow { arg: packed } => { + let oparg = packed.get(arg); + let idx1 = oparg >> 4; + let idx2 = oparg & 15; + let name1 = varname(idx1); + let name2 = varname(idx2); + write!( + f, + "{:pad$}({}, {})", + "LOAD_FAST_BORROW_LOAD_FAST_BORROW", name1, name2 + ) + } + Self::LoadFromDictOrGlobals(idx) => w!(LOAD_FROM_DICT_OR_GLOBALS, name = idx), + Self::LoadGlobal(idx) => w!(LOAD_GLOBAL, name = idx), + Self::LoadName(idx) => w!(LOAD_NAME, name = idx), + Self::LoadSpecial { method } => w!(LOAD_SPECIAL, method), + Self::LoadSuperAttr { arg: idx } => { + let encoded = idx.get(arg); + let (name_idx, load_method, has_class) = decode_load_super_attr_arg(encoded); + let attr_name = name(name_idx); + write!( + f, + "{:pad$}({}, {}, method={}, class={})", + "LOAD_SUPER_ATTR", encoded, attr_name, load_method, has_class + ) + } + Self::MakeFunction => w!(MAKE_FUNCTION), + Self::MapAdd { i } => w!(MAP_ADD, i), + Self::MatchClass(arg) => w!(MATCH_CLASS, arg), + Self::MatchKeys => w!(MATCH_KEYS), + Self::MatchMapping => w!(MATCH_MAPPING), + Self::MatchSequence => w!(MATCH_SEQUENCE), + Self::Nop => w!(NOP), + Self::PopExcept => w!(POP_EXCEPT), + Self::PopJumpIfFalse { target } => w!(POP_JUMP_IF_FALSE, target), + Self::PopJumpIfTrue { target } => w!(POP_JUMP_IF_TRUE, target), + Self::PopTop => w!(POP_TOP), + Self::EndFor => w!(END_FOR), + Self::PopIter => w!(POP_ITER), + Self::PushExcInfo => w!(PUSH_EXC_INFO), + Self::PushNull => w!(PUSH_NULL), + Self::RaiseVarargs { kind } => w!(RAISE_VARARGS, ?kind), + Self::Reraise { depth } => w!(RERAISE, depth), + Self::Resume { arg } => w!(RESUME, arg), + Self::ReturnValue => w!(RETURN_VALUE), + Self::ReturnGenerator => w!(RETURN_GENERATOR), + Self::Send { target } => w!(SEND, target), + Self::SetAdd { i } => w!(SET_ADD, i), + Self::SetFunctionAttribute { attr } => w!(SET_FUNCTION_ATTRIBUTE, ?attr), + Self::SetupAnnotations => w!(SETUP_ANNOTATIONS), + Self::SetUpdate { i } => w!(SET_UPDATE, i), + Self::StoreAttr { idx } => w!(STORE_ATTR, name = idx), + Self::StoreDeref(idx) => w!(STORE_DEREF, cell_name = idx), + Self::StoreFast(idx) => w!(STORE_FAST, varname = idx), + Self::StoreFastLoadFast { + store_idx, + load_idx, + } => { + write!(f, "STORE_FAST_LOAD_FAST")?; + write!(f, " ({}, {})", store_idx.get(arg), load_idx.get(arg)) + } + Self::StoreGlobal(idx) => w!(STORE_GLOBAL, name = idx), + Self::StoreName(idx) => w!(STORE_NAME, name = idx), + Self::StoreSubscr => w!(STORE_SUBSCR), + Self::Swap { index } => w!(SWAP, index), + Self::ToBool => w!(TO_BOOL), + Self::UnpackEx { args } => w!(UNPACK_EX, args), + Self::UnpackSequence { size } => w!(UNPACK_SEQUENCE, size), + Self::WithExceptStart => w!(WITH_EXCEPT_START), + Self::UnaryInvert => w!(UNARY_INVERT), + Self::UnaryNegative => w!(UNARY_NEGATIVE), + Self::UnaryNot => w!(UNARY_NOT), + Self::YieldValue { arg } => w!(YIELD_VALUE, arg), + Self::GetYieldFromIter => w!(GET_YIELD_FROM_ITER), + Self::BuildTemplate => w!(BUILD_TEMPLATE), + Self::BuildInterpolation { oparg } => w!(BUILD_INTERPOLATION, oparg), + _ => w!(RUSTPYTHON_PLACEHOLDER), + } + } +} + +/// Instructions used by the compiler. They are not executed by the VM. +/// +/// CPython 3.14.2 aligned (256-266). +#[derive(Clone, Copy, Debug)] +#[repr(u16)] +pub enum PseudoInstruction { + // CPython 3.14.2 pseudo instructions (256-266) + AnnotationsPlaceholder = 256, + Jump { target: Arg<Label> } = 257, + JumpIfFalse { target: Arg<Label> } = 258, + JumpIfTrue { target: Arg<Label> } = 259, + JumpNoInterrupt { target: Arg<Label> } = 260, + LoadClosure(Arg<NameIdx>) = 261, + PopBlock = 262, + SetupCleanup { target: Arg<Label> } = 263, + SetupFinally { target: Arg<Label> } = 264, + SetupWith { target: Arg<Label> } = 265, + StoreFastMaybeNull(Arg<NameIdx>) = 266, +} + +const _: () = assert!(mem::size_of::<PseudoInstruction>() == 2); + +impl From<PseudoInstruction> for u16 { + #[inline] + fn from(ins: PseudoInstruction) -> Self { + // SAFETY: there's no padding bits + unsafe { mem::transmute::<PseudoInstruction, Self>(ins) } + } +} + +impl TryFrom<u16> for PseudoInstruction { + type Error = MarshalError; + + #[inline] + fn try_from(value: u16) -> Result<Self, MarshalError> { + let start = u16::from(Self::AnnotationsPlaceholder); + let end = u16::from(Self::StoreFastMaybeNull(Arg::marker())); + + if (start..=end).contains(&value) { + Ok(unsafe { mem::transmute::<u16, Self>(value) }) + } else { + Err(Self::Error::InvalidBytecode) + } + } +} + +impl PseudoInstruction { + /// Returns true if this is a block push pseudo instruction + /// (SETUP_FINALLY, SETUP_CLEANUP, or SETUP_WITH). + pub fn is_block_push(&self) -> bool { + matches!( + self, + Self::SetupCleanup { .. } | Self::SetupFinally { .. } | Self::SetupWith { .. } + ) + } +} + +impl InstructionMetadata for PseudoInstruction { + fn label_arg(&self) -> Option<Arg<Label>> { + match self { + Self::Jump { target: l } + | Self::JumpIfFalse { target: l } + | Self::JumpIfTrue { target: l } + | Self::JumpNoInterrupt { target: l } + | Self::SetupCleanup { target: l } + | Self::SetupFinally { target: l } + | Self::SetupWith { target: l } => Some(*l), + _ => None, + } + } + + fn is_scope_exit(&self) -> bool { + false + } + + fn stack_effect_info(&self, _oparg: u32) -> StackEffect { + // Reason for converting oparg to i32 is because of expressions like `1 + (oparg -1)` + // that causes underflow errors. + let _oparg = i32::try_from(_oparg).expect("oparg does not fit in an `i32`"); + + // NOTE: Please don't "simplify" expressions here (i.e. `1 + (oparg - 1)`) + // as it will be harder to see diff with what CPython auto-generates + let (pushed, popped) = match self { + Self::AnnotationsPlaceholder => (0, 0), + Self::Jump { .. } => (0, 0), + Self::JumpIfFalse { .. } => (1, 1), + Self::JumpIfTrue { .. } => (1, 1), + Self::JumpNoInterrupt { .. } => (0, 0), + Self::LoadClosure(_) => (1, 0), + Self::PopBlock => (0, 0), + // Normal path effect is 0 (these are NOPs on fall-through). + // Handler entry effects are computed directly in max_stackdepth(). + Self::SetupCleanup { .. } => (0, 0), + Self::SetupFinally { .. } => (0, 0), + Self::SetupWith { .. } => (0, 0), + Self::StoreFastMaybeNull(_) => (0, 1), + }; + + debug_assert!((0..=i32::MAX).contains(&pushed)); + debug_assert!((0..=i32::MAX).contains(&popped)); + + StackEffect::new(pushed as u32, popped as u32) + } + + fn is_unconditional_jump(&self) -> bool { + matches!(self, Self::Jump { .. } | Self::JumpNoInterrupt { .. }) + } + + fn fmt_dis( + &self, + _arg: OpArg, + _f: &mut fmt::Formatter<'_>, + _ctx: &impl InstrDisplayContext, + _expand_code_objects: bool, + _pad: usize, + _level: usize, + ) -> fmt::Result { + unimplemented!() + } +} + +#[derive(Clone, Copy, Debug)] +pub enum AnyInstruction { + Real(Instruction), + Pseudo(PseudoInstruction), +} + +impl From<Instruction> for AnyInstruction { + fn from(value: Instruction) -> Self { + Self::Real(value) + } +} + +impl From<PseudoInstruction> for AnyInstruction { + fn from(value: PseudoInstruction) -> Self { + Self::Pseudo(value) + } +} + +impl TryFrom<u8> for AnyInstruction { + type Error = MarshalError; + + fn try_from(value: u8) -> Result<Self, Self::Error> { + Ok(Instruction::try_from(value)?.into()) + } +} + +impl TryFrom<u16> for AnyInstruction { + type Error = MarshalError; + + fn try_from(value: u16) -> Result<Self, Self::Error> { + match u8::try_from(value) { + Ok(v) => v.try_into(), + Err(_) => Ok(PseudoInstruction::try_from(value)?.into()), + } + } +} + +macro_rules! inst_either { + (fn $name:ident ( &self $(, $arg:ident : $arg_ty:ty )* ) -> $ret:ty ) => { + fn $name(&self $(, $arg : $arg_ty )* ) -> $ret { + match self { + Self::Real(op) => op.$name($($arg),*), + Self::Pseudo(op) => op.$name($($arg),*), + } + } + }; +} + +impl InstructionMetadata for AnyInstruction { + inst_either!(fn label_arg(&self) -> Option<Arg<Label>>); + + inst_either!(fn is_unconditional_jump(&self) -> bool); + + inst_either!(fn is_scope_exit(&self) -> bool); + + inst_either!(fn stack_effect(&self, oparg: u32) -> i32); + + inst_either!(fn stack_effect_info(&self, oparg: u32) -> StackEffect); + + inst_either!(fn fmt_dis( + &self, + arg: OpArg, + f: &mut fmt::Formatter<'_>, + ctx: &impl InstrDisplayContext, + expand_code_objects: bool, + pad: usize, + level: usize + ) -> fmt::Result); +} + +impl AnyInstruction { + /// Gets the inner value of [`Self::Real`]. + pub const fn real(self) -> Option<Instruction> { + match self { + Self::Real(ins) => Some(ins), + _ => None, + } + } + + /// Gets the inner value of [`Self::Pseudo`]. + pub const fn pseudo(self) -> Option<PseudoInstruction> { + match self { + Self::Pseudo(ins) => Some(ins), + _ => None, + } + } + + /// Same as [`Self::real`] but panics if wasn't called on [`Self::Real`]. + /// + /// # Panics + /// + /// If was called on something else other than [`Self::Real`]. + pub const fn expect_real(self) -> Instruction { + self.real() + .expect("Expected Instruction::Real, found Instruction::Pseudo") + } + + /// Same as [`Self::pseudo`] but panics if wasn't called on [`Self::Pseudo`]. + /// + /// # Panics + /// + /// If was called on something else other than [`Self::Pseudo`]. + pub const fn expect_pseudo(self) -> PseudoInstruction { + self.pseudo() + .expect("Expected Instruction::Pseudo, found Instruction::Real") + } + + /// Returns true if this is a block push pseudo instruction + /// (SETUP_FINALLY, SETUP_CLEANUP, or SETUP_WITH). + pub fn is_block_push(&self) -> bool { + matches!(self, Self::Pseudo(p) if p.is_block_push()) + } + + /// Returns true if this is a POP_BLOCK pseudo instruction. + pub fn is_pop_block(&self) -> bool { + matches!(self, Self::Pseudo(PseudoInstruction::PopBlock)) + } +} + +/// What effect the instruction has on the stack. +#[derive(Clone, Copy)] +pub struct StackEffect { + /// How many items the instruction is pushing on the stack. + pushed: u32, + /// How many items the instruction is popping from the stack. + popped: u32, +} + +impl StackEffect { + /// Creates a new [`Self`]. + pub const fn new(pushed: u32, popped: u32) -> Self { + Self { pushed, popped } + } + + /// Get the calculated stack effect as [`i32`]. + pub fn effect(self) -> i32 { + self.into() + } + + /// Get the pushed count. + pub const fn pushed(self) -> u32 { + self.pushed + } + + /// Get the popped count. + pub const fn popped(self) -> u32 { + self.popped + } +} + +impl From<StackEffect> for i32 { + fn from(effect: StackEffect) -> Self { + (effect.pushed() as i32) - (effect.popped() as i32) + } +} + +pub trait InstructionMetadata { + /// Gets the label stored inside this instruction, if it exists. + fn label_arg(&self) -> Option<Arg<Label>>; + + fn is_scope_exit(&self) -> bool; + + fn is_unconditional_jump(&self) -> bool; + + /// Stack effect info for how many items are pushed/popped from the stack, + /// for this instruction. + fn stack_effect_info(&self, oparg: u32) -> StackEffect; + + /// Stack effect of [`Self::stack_effect_info`]. + fn stack_effect(&self, oparg: u32) -> i32 { + self.stack_effect_info(oparg).effect() + } + + #[allow(clippy::too_many_arguments)] + fn fmt_dis( + &self, + arg: OpArg, + f: &mut fmt::Formatter<'_>, + ctx: &impl InstrDisplayContext, + expand_code_objects: bool, + pad: usize, + level: usize, + ) -> fmt::Result; + + fn display(&self, arg: OpArg, ctx: &impl InstrDisplayContext) -> impl fmt::Display { + fmt::from_fn(move |f| self.fmt_dis(arg, f, ctx, false, 0, 0)) + } +} + +#[derive(Copy, Clone)] +pub struct Arg<T: OpArgType>(PhantomData<T>); + +impl<T: OpArgType> Arg<T> { + #[inline] + pub const fn marker() -> Self { + Self(PhantomData) + } + + #[inline] + pub fn new(arg: T) -> (Self, OpArg) { + (Self(PhantomData), OpArg(arg.to_op_arg())) + } + + #[inline] + pub fn new_single(arg: T) -> (Self, OpArgByte) + where + T: Into<u8>, + { + (Self(PhantomData), OpArgByte(arg.into())) + } + + #[inline(always)] + pub fn get(self, arg: OpArg) -> T { + self.try_get(arg).unwrap() + } + + #[inline(always)] + pub fn try_get(self, arg: OpArg) -> Result<T, MarshalError> { + T::from_op_arg(arg.0) + } + + /// # Safety + /// T::from_op_arg(self) must succeed + #[inline(always)] + pub unsafe fn get_unchecked(self, arg: OpArg) -> T { + // SAFETY: requirements forwarded from caller + unsafe { T::from_op_arg(arg.0).unwrap_unchecked() } + } +} + +impl<T: OpArgType> PartialEq for Arg<T> { + fn eq(&self, _: &Self) -> bool { + true + } +} + +impl<T: OpArgType> Eq for Arg<T> {} + +impl<T: OpArgType> fmt::Debug for Arg<T> { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "Arg<{}>", core::any::type_name::<T>()) + } +} + +/// Encode LOAD_ATTR oparg: bit 0 = method flag, bits 1+ = name index. +#[inline] +pub const fn encode_load_attr_arg(name_idx: u32, is_method: bool) -> u32 { + (name_idx << 1) | (is_method as u32) +} + +/// Decode LOAD_ATTR oparg: returns (name_idx, is_method). +#[inline] +pub const fn decode_load_attr_arg(oparg: u32) -> (u32, bool) { + let is_method = (oparg & 1) == 1; + let name_idx = oparg >> 1; + (name_idx, is_method) +} + +/// Encode LOAD_SUPER_ATTR oparg: bit 0 = load_method, bit 1 = has_class, bits 2+ = name index. +#[inline] +pub const fn encode_load_super_attr_arg(name_idx: u32, load_method: bool, has_class: bool) -> u32 { + (name_idx << 2) | ((has_class as u32) << 1) | (load_method as u32) +} + +/// Decode LOAD_SUPER_ATTR oparg: returns (name_idx, load_method, has_class). +#[inline] +pub const fn decode_load_super_attr_arg(oparg: u32) -> (u32, bool, bool) { + let load_method = (oparg & 1) == 1; + let has_class = (oparg & 2) == 2; + let name_idx = oparg >> 2; + (name_idx, load_method, has_class) +} diff --git a/crates/compiler-core/src/bytecode/oparg.rs b/crates/compiler-core/src/bytecode/oparg.rs new file mode 100644 index 00000000000..724fd6fcd10 --- /dev/null +++ b/crates/compiler-core/src/bytecode/oparg.rs @@ -0,0 +1,657 @@ +use bitflags::bitflags; + +use core::{fmt, num::NonZeroU8}; + +use crate::{ + bytecode::{CodeUnit, instruction::Instruction}, + marshal::MarshalError, +}; + +pub trait OpArgType: Copy { + fn from_op_arg(x: u32) -> Result<Self, MarshalError>; + + fn to_op_arg(self) -> u32; +} + +/// Opcode argument that may be extended by a prior ExtendedArg. +#[derive(Copy, Clone, PartialEq, Eq)] +#[repr(transparent)] +pub struct OpArgByte(pub u8); + +impl OpArgByte { + pub const fn null() -> Self { + Self(0) + } +} + +impl From<u8> for OpArgByte { + fn from(raw: u8) -> Self { + Self(raw) + } +} + +impl fmt::Debug for OpArgByte { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + self.0.fmt(f) + } +} + +/// Full 32-bit op_arg, including any possible ExtendedArg extension. +#[derive(Copy, Clone, Debug)] +#[repr(transparent)] +pub struct OpArg(pub u32); + +impl OpArg { + pub const fn null() -> Self { + Self(0) + } + + /// Returns how many CodeUnits a instruction with this op_arg will be encoded as + #[inline] + pub const fn instr_size(self) -> usize { + (self.0 > 0xff) as usize + (self.0 > 0xff_ff) as usize + (self.0 > 0xff_ff_ff) as usize + 1 + } + + /// returns the arg split into any necessary ExtendedArg components (in big-endian order) and + /// the arg for the real opcode itself + #[inline(always)] + pub fn split(self) -> (impl ExactSizeIterator<Item = OpArgByte>, OpArgByte) { + let mut it = self + .0 + .to_le_bytes() + .map(OpArgByte) + .into_iter() + .take(self.instr_size()); + let lo = it.next().unwrap(); + (it.rev(), lo) + } +} + +impl From<u32> for OpArg { + fn from(raw: u32) -> Self { + Self(raw) + } +} + +impl From<OpArg> for u32 { + fn from(value: OpArg) -> Self { + value.0 + } +} + +#[derive(Default, Copy, Clone)] +#[repr(transparent)] +pub struct OpArgState { + state: u32, +} + +impl OpArgState { + #[inline(always)] + pub fn get(&mut self, ins: CodeUnit) -> (Instruction, OpArg) { + let arg = self.extend(ins.arg); + if !matches!(ins.op, Instruction::ExtendedArg) { + self.reset(); + } + (ins.op, arg) + } + + #[inline(always)] + pub fn extend(&mut self, arg: OpArgByte) -> OpArg { + self.state = (self.state << 8) | u32::from(arg.0); + OpArg(self.state) + } + + #[inline(always)] + pub const fn reset(&mut self) { + self.state = 0 + } +} + +/// Oparg values for [`Instruction::ConvertValue`]. +/// +/// ## See also +/// +/// - [CPython FVC_* flags](https://github.com/python/cpython/blob/8183fa5e3f78ca6ab862de7fb8b14f3d929421e0/Include/ceval.h#L129-L132) +#[repr(u8)] +#[derive(Copy, Clone, Debug, Hash, PartialEq, Eq)] +pub enum ConvertValueOparg { + /// No conversion. + /// + /// ```python + /// f"{x}" + /// f"{x:4}" + /// ``` + None = 0, + /// Converts by calling `str(<value>)`. + /// + /// ```python + /// f"{x!s}" + /// f"{x!s:2}" + /// ``` + Str = 1, + /// Converts by calling `repr(<value>)`. + /// + /// ```python + /// f"{x!r}" + /// f"{x!r:2}" + /// ``` + Repr = 2, + /// Converts by calling `ascii(<value>)`. + /// + /// ```python + /// f"{x!a}" + /// f"{x!a:2}" + /// ``` + Ascii = 3, +} + +impl fmt::Display for ConvertValueOparg { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let out = match self { + Self::Str => "1 (str)", + Self::Repr => "2 (repr)", + Self::Ascii => "3 (ascii)", + // We should never reach this. `FVC_NONE` are being handled by `Instruction::FormatSimple` + Self::None => "", + }; + + write!(f, "{out}") + } +} + +impl OpArgType for ConvertValueOparg { + #[inline] + fn from_op_arg(x: u32) -> Result<Self, MarshalError> { + Ok(match x { + // Ruff `ConversionFlag::None` is `-1i8`, + // when its converted to `u8` its value is `u8::MAX` + 0 | 255 => Self::None, + 1 => Self::Str, + 2 => Self::Repr, + 3 => Self::Ascii, + _ => return Err(MarshalError::InvalidBytecode), + }) + } + + #[inline] + fn to_op_arg(self) -> u32 { + self as u32 + } +} + +/// Resume type for the RESUME instruction +#[derive(Copy, Clone, Debug, Hash, PartialEq, Eq)] +#[repr(u32)] +pub enum ResumeType { + AtFuncStart = 0, + AfterYield = 1, + AfterYieldFrom = 2, + AfterAwait = 3, +} + +impl OpArgType for u32 { + #[inline(always)] + fn from_op_arg(x: u32) -> Result<Self, MarshalError> { + Ok(x) + } + + #[inline(always)] + fn to_op_arg(self) -> u32 { + self + } +} + +impl OpArgType for bool { + #[inline(always)] + fn from_op_arg(x: u32) -> Result<Self, MarshalError> { + Ok(x != 0) + } + + #[inline(always)] + fn to_op_arg(self) -> u32 { + self as u32 + } +} + +macro_rules! op_arg_enum_impl { + (enum $name:ident { $($(#[$var_attr:meta])* $var:ident = $value:literal,)* }) => { + impl OpArgType for $name { + fn to_op_arg(self) -> u32 { + self as u32 + } + + fn from_op_arg(x: u32) -> Result<Self, MarshalError> { + Ok(match u8::try_from(x).map_err(|_| MarshalError::InvalidBytecode)? { + $($value => Self::$var,)* + _ => return Err(MarshalError::InvalidBytecode), + }) + } + } + }; +} + +macro_rules! op_arg_enum { + ($(#[$attr:meta])* $vis:vis enum $name:ident { $($(#[$var_attr:meta])* $var:ident = $value:literal,)* }) => { + $(#[$attr])* + $vis enum $name { + $($(#[$var_attr])* $var = $value,)* + } + + op_arg_enum_impl!(enum $name { + $($(#[$var_attr])* $var = $value,)* + }); + }; +} + +pub type NameIdx = u32; + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Ord, PartialOrd)] +#[repr(transparent)] +pub struct Label(pub u32); + +impl OpArgType for Label { + #[inline(always)] + fn from_op_arg(x: u32) -> Result<Self, MarshalError> { + Ok(Self(x)) + } + + #[inline(always)] + fn to_op_arg(self) -> u32 { + self.0 + } +} + +impl fmt::Display for Label { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + self.0.fmt(f) + } +} + +op_arg_enum!( + /// The kind of Raise that occurred. + #[derive(Copy, Clone, Debug, PartialEq, Eq)] + #[repr(u8)] + pub enum RaiseKind { + /// Bare `raise` statement with no arguments. + /// Gets the current exception from VM state (topmost_exception). + /// Maps to RAISE_VARARGS with oparg=0. + BareRaise = 0, + /// `raise exc` - exception is on the stack. + /// Maps to RAISE_VARARGS with oparg=1. + Raise = 1, + /// `raise exc from cause` - exception and cause are on the stack. + /// Maps to RAISE_VARARGS with oparg=2. + RaiseCause = 2, + /// Reraise exception from the stack top. + /// Used in exception handler cleanup blocks (finally, except). + /// Gets exception from stack, not from VM state. + /// Maps to the RERAISE opcode. + ReraiseFromStack = 3, + } +); + +op_arg_enum!( + /// Intrinsic function for CALL_INTRINSIC_1 + #[derive(Copy, Clone, Debug, PartialEq, Eq)] + #[repr(u8)] + pub enum IntrinsicFunction1 { + // Invalid = 0, + Print = 1, + /// Import * operation + ImportStar = 2, + /// Convert StopIteration to RuntimeError in async context + StopIterationError = 3, + AsyncGenWrap = 4, + UnaryPositive = 5, + /// Convert list to tuple + ListToTuple = 6, + /// Type parameter related + TypeVar = 7, + ParamSpec = 8, + TypeVarTuple = 9, + /// Generic subscript for PEP 695 + SubscriptGeneric = 10, + TypeAlias = 11, + } +); + +op_arg_enum!( + /// Intrinsic function for CALL_INTRINSIC_2 + #[derive(Copy, Clone, Debug, PartialEq, Eq)] + #[repr(u8)] + pub enum IntrinsicFunction2 { + PrepReraiseStar = 1, + TypeVarWithBound = 2, + TypeVarWithConstraint = 3, + SetFunctionTypeParams = 4, + /// Set default value for type parameter (PEP 695) + SetTypeparamDefault = 5, + } +); + +bitflags! { + #[derive(Copy, Clone, Debug, PartialEq)] + pub struct MakeFunctionFlags: u8 { + const CLOSURE = 0x01; + const ANNOTATIONS = 0x02; + const KW_ONLY_DEFAULTS = 0x04; + const DEFAULTS = 0x08; + const TYPE_PARAMS = 0x10; + /// PEP 649: __annotate__ function closure (instead of __annotations__ dict) + const ANNOTATE = 0x20; + } +} + +impl OpArgType for MakeFunctionFlags { + #[inline(always)] + fn from_op_arg(x: u32) -> Result<Self, MarshalError> { + Self::from_bits(x as u8).ok_or(MarshalError::InvalidBytecode) + } + + #[inline(always)] + fn to_op_arg(self) -> u32 { + self.bits().into() + } +} + +op_arg_enum!( + /// The possible comparison operators + #[derive(Debug, Copy, Clone, PartialEq, Eq)] + #[repr(u8)] + pub enum ComparisonOperator { + // be intentional with bits so that we can do eval_ord with just a bitwise and + // bits: | Equal | Greater | Less | + Less = 0b001, + Greater = 0b010, + NotEqual = 0b011, + Equal = 0b100, + LessOrEqual = 0b101, + GreaterOrEqual = 0b110, + } +); + +op_arg_enum!( + /// The possible Binary operators + /// + /// # Examples + /// + /// ```rust + /// use rustpython_compiler_core::bytecode::{Arg, BinaryOperator, Instruction}; + /// let (op, _) = Arg::new(BinaryOperator::Add); + /// let instruction = Instruction::BinaryOp { op }; + /// ``` + /// + /// See also: + /// - [_PyEval_BinaryOps](https://github.com/python/cpython/blob/8183fa5e3f78ca6ab862de7fb8b14f3d929421e0/Python/ceval.c#L316-L343) + #[repr(u8)] + #[derive(Clone, Copy, Debug, Eq, PartialEq)] + pub enum BinaryOperator { + /// `+` + Add = 0, + /// `&` + And = 1, + /// `//` + FloorDivide = 2, + /// `<<` + Lshift = 3, + /// `@` + MatrixMultiply = 4, + /// `*` + Multiply = 5, + /// `%` + Remainder = 6, + /// `|` + Or = 7, + /// `**` + Power = 8, + /// `>>` + Rshift = 9, + /// `-` + Subtract = 10, + /// `/` + TrueDivide = 11, + /// `^` + Xor = 12, + /// `+=` + InplaceAdd = 13, + /// `&=` + InplaceAnd = 14, + /// `//=` + InplaceFloorDivide = 15, + /// `<<=` + InplaceLshift = 16, + /// `@=` + InplaceMatrixMultiply = 17, + /// `*=` + InplaceMultiply = 18, + /// `%=` + InplaceRemainder = 19, + /// `|=` + InplaceOr = 20, + /// `**=` + InplacePower = 21, + /// `>>=` + InplaceRshift = 22, + /// `-=` + InplaceSubtract = 23, + /// `/=` + InplaceTrueDivide = 24, + /// `^=` + InplaceXor = 25, + /// `[]` subscript + Subscr = 26, + } +); + +impl BinaryOperator { + /// Get the "inplace" version of the operator. + /// This has no effect if `self` is already an "inplace" operator. + /// + /// # Example + /// ```rust + /// use rustpython_compiler_core::bytecode::BinaryOperator; + /// + /// assert_eq!(BinaryOperator::Power.as_inplace(), BinaryOperator::InplacePower); + /// + /// assert_eq!(BinaryOperator::InplaceSubtract.as_inplace(), BinaryOperator::InplaceSubtract); + /// ``` + #[must_use] + pub const fn as_inplace(self) -> Self { + match self { + Self::Add => Self::InplaceAdd, + Self::And => Self::InplaceAnd, + Self::FloorDivide => Self::InplaceFloorDivide, + Self::Lshift => Self::InplaceLshift, + Self::MatrixMultiply => Self::InplaceMatrixMultiply, + Self::Multiply => Self::InplaceMultiply, + Self::Remainder => Self::InplaceRemainder, + Self::Or => Self::InplaceOr, + Self::Power => Self::InplacePower, + Self::Rshift => Self::InplaceRshift, + Self::Subtract => Self::InplaceSubtract, + Self::TrueDivide => Self::InplaceTrueDivide, + Self::Xor => Self::InplaceXor, + _ => self, + } + } +} + +impl fmt::Display for BinaryOperator { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let op = match self { + Self::Add => "+", + Self::And => "&", + Self::FloorDivide => "//", + Self::Lshift => "<<", + Self::MatrixMultiply => "@", + Self::Multiply => "*", + Self::Remainder => "%", + Self::Or => "|", + Self::Power => "**", + Self::Rshift => ">>", + Self::Subtract => "-", + Self::TrueDivide => "/", + Self::Xor => "^", + Self::InplaceAdd => "+=", + Self::InplaceAnd => "&=", + Self::InplaceFloorDivide => "//=", + Self::InplaceLshift => "<<=", + Self::InplaceMatrixMultiply => "@=", + Self::InplaceMultiply => "*=", + Self::InplaceRemainder => "%=", + Self::InplaceOr => "|=", + Self::InplacePower => "**=", + Self::InplaceRshift => ">>=", + Self::InplaceSubtract => "-=", + Self::InplaceTrueDivide => "/=", + Self::InplaceXor => "^=", + Self::Subscr => "[]", + }; + write!(f, "{op}") + } +} + +op_arg_enum!( + /// Whether or not to invert the operation. + #[repr(u8)] + #[derive(Debug, Copy, Clone, PartialEq, Eq)] + pub enum Invert { + /// ```py + /// foo is bar + /// x in lst + /// ``` + No = 0, + /// ```py + /// foo is not bar + /// x not in lst + /// ``` + Yes = 1, + } +); + +op_arg_enum!( + /// Special method for LOAD_SPECIAL opcode (context managers). + #[repr(u8)] + #[derive(Debug, Copy, Clone, PartialEq, Eq)] + pub enum SpecialMethod { + /// `__enter__` for sync context manager + Enter = 0, + /// `__exit__` for sync context manager + Exit = 1, + /// `__aenter__` for async context manager + AEnter = 2, + /// `__aexit__` for async context manager + AExit = 3, + } +); + +impl fmt::Display for SpecialMethod { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let method_name = match self { + Self::Enter => "__enter__", + Self::Exit => "__exit__", + Self::AEnter => "__aenter__", + Self::AExit => "__aexit__", + }; + write!(f, "{method_name}") + } +} + +op_arg_enum!( + /// Common constants for LOAD_COMMON_CONSTANT opcode. + /// pycore_opcode_utils.h CONSTANT_* + #[repr(u8)] + #[derive(Debug, Copy, Clone, PartialEq, Eq)] + pub enum CommonConstant { + /// `AssertionError` exception type + AssertionError = 0, + /// `NotImplementedError` exception type + NotImplementedError = 1, + /// Built-in `tuple` type + BuiltinTuple = 2, + /// Built-in `all` function + BuiltinAll = 3, + /// Built-in `any` function + BuiltinAny = 4, + } +); + +impl fmt::Display for CommonConstant { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let name = match self { + Self::AssertionError => "AssertionError", + Self::NotImplementedError => "NotImplementedError", + Self::BuiltinTuple => "tuple", + Self::BuiltinAll => "all", + Self::BuiltinAny => "any", + }; + write!(f, "{name}") + } +} + +/// Specifies if a slice is built with either 2 or 3 arguments. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum BuildSliceArgCount { + /// ```py + /// x[5:10] + /// ``` + Two, + /// ```py + /// x[5:10:2] + /// ``` + Three, +} + +impl OpArgType for BuildSliceArgCount { + #[inline(always)] + fn from_op_arg(x: u32) -> Result<Self, MarshalError> { + Ok(match x { + 2 => Self::Two, + 3 => Self::Three, + _ => return Err(MarshalError::InvalidBytecode), + }) + } + + #[inline(always)] + fn to_op_arg(self) -> u32 { + u32::from(self.argc().get()) + } +} + +impl BuildSliceArgCount { + /// Get the numeric value of `Self`. + #[must_use] + pub const fn argc(self) -> NonZeroU8 { + let inner = match self { + Self::Two => 2, + Self::Three => 3, + }; + // Safety: `inner` can be either 2 or 3. + unsafe { NonZeroU8::new_unchecked(inner) } + } +} + +#[derive(Copy, Clone)] +pub struct UnpackExArgs { + pub before: u8, + pub after: u8, +} + +impl OpArgType for UnpackExArgs { + #[inline(always)] + fn from_op_arg(x: u32) -> Result<Self, MarshalError> { + let [before, after, ..] = x.to_le_bytes(); + Ok(Self { before, after }) + } + + #[inline(always)] + fn to_op_arg(self) -> u32 { + u32::from_le_bytes([self.before, self.after, 0, 0]) + } +} + +impl fmt::Display for UnpackExArgs { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "before: {}, after: {}", self.before, self.after) + } +} diff --git a/crates/compiler-core/src/marshal.rs b/crates/compiler-core/src/marshal.rs index 5b528fe7e50..decb25d5283 100644 --- a/crates/compiler-core/src/marshal.rs +++ b/crates/compiler-core/src/marshal.rs @@ -4,9 +4,9 @@ use malachite_bigint::{BigInt, Sign}; use num_complex::Complex64; use rustpython_wtf8::Wtf8; -pub const FORMAT_VERSION: u32 = 4; +pub const FORMAT_VERSION: u32 = 5; -#[derive(Debug)] +#[derive(Clone, Copy, Debug)] pub enum MarshalError { /// Unexpected End Of File Eof, @@ -42,6 +42,7 @@ impl core::error::Error for MarshalError {} type Result<T, E = MarshalError> = core::result::Result<T, E>; +#[derive(Clone, Copy)] #[repr(u8)] enum Type { // Null = b'0', @@ -65,6 +66,7 @@ enum Type { // Unknown = b'?', Set = b'<', FrozenSet = b'>', + Slice = b':', // Added in version 5 Ascii = b'a', // AsciiInterned = b'A', // SmallTuple = b')', @@ -101,6 +103,7 @@ impl TryFrom<u8> for Type { // b'?' => Unknown, b'<' => Set, b'>' => FrozenSet, + b':' => Slice, b'a' => Ascii, // b'A' => AsciiInterned, // b')' => SmallTuple, @@ -202,7 +205,7 @@ pub fn deserialize_code<R: Read, Bag: ConstantBag>( }) .collect::<Result<Box<[(SourceLocation, SourceLocation)]>>>()?; - let flags = CodeFlags::from_bits_truncate(rdr.read_u16()?); + let flags = CodeFlags::from_bits_truncate(rdr.read_u32()?); let posonlyarg_count = rdr.read_u32()?; let arg_count = rdr.read_u32()?; @@ -466,6 +469,12 @@ pub fn deserialize_value<R: Read, Bag: MarshalBag>(rdr: &mut R, bag: Bag) -> Res bag.make_bytes(value) } Type::Code => bag.make_code(deserialize_code(rdr, bag.constant_bag())?), + Type::Slice => { + // Slice constants are not yet supported in RustPython + // This would require adding a Slice variant to ConstantData enum + // For now, return an error if we encounter a slice in marshal data + return Err(MarshalError::BadType); + } }; Ok(value) } @@ -660,7 +669,7 @@ pub fn serialize_code<W: Write, C: Constant>(buf: &mut W, code: &CodeObject<C>) buf.write_u32(end.character_offset.to_zero_indexed() as _); } - buf.write_u16(code.flags.bits()); + buf.write_u32(code.flags.bits()); buf.write_u32(code.posonlyarg_count); buf.write_u32(code.arg_count); diff --git a/crates/compiler-core/src/mode.rs b/crates/compiler-core/src/mode.rs index f2b19d677be..181ea4fdfe7 100644 --- a/crates/compiler-core/src/mode.rs +++ b/crates/compiler-core/src/mode.rs @@ -22,7 +22,7 @@ impl core::str::FromStr for Mode { } /// Returned when a given mode is not valid. -#[derive(Debug)] +#[derive(Clone, Copy, Debug)] pub struct ModeParseError; impl core::fmt::Display for ModeParseError { diff --git a/crates/derive-impl/src/compile_bytecode.rs b/crates/derive-impl/src/compile_bytecode.rs index 23c90690dad..16984139fcd 100644 --- a/crates/derive-impl/src/compile_bytecode.rs +++ b/crates/derive-impl/src/compile_bytecode.rs @@ -4,7 +4,7 @@ //! // either: //! source = "python_source_code", //! // or -//! file = "file/path/relative/to/$CARGO_MANIFEST_DIR", +//! file = "file/path/relative/to/this/file", //! //! // the mode to compile the code in //! mode = "exec", // or "eval" or "single" @@ -17,10 +17,9 @@ use crate::Diagnostic; use proc_macro2::{Span, TokenStream}; use quote::quote; use rustpython_compiler_core::{Mode, bytecode::CodeObject, frozen}; -use std::sync::LazyLock; use std::{ collections::HashMap, - env, fs, + fs, path::{Path, PathBuf}, }; use syn::{ @@ -29,17 +28,13 @@ use syn::{ spanned::Spanned, }; -static CARGO_MANIFEST_DIR: LazyLock<PathBuf> = LazyLock::new(|| { - PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").expect("CARGO_MANIFEST_DIR is not present")) -}); - enum CompilationSourceKind { /// Source is a File (Path) - File(PathBuf), + File { base: PathBuf, rel_path: PathBuf }, /// Direct Raw source code SourceCode(String), /// Source is a directory - Dir(PathBuf), + Dir { base: PathBuf, rel_path: PathBuf }, } struct CompiledModule { @@ -85,12 +80,9 @@ impl CompilationSource { compiler: &dyn Compiler, ) -> Result<HashMap<String, CompiledModule>, Diagnostic> { match &self.kind { - CompilationSourceKind::Dir(rel_path) => self.compile_dir( - &CARGO_MANIFEST_DIR.join(rel_path), - String::new(), - mode, - compiler, - ), + CompilationSourceKind::Dir { base, rel_path } => { + self.compile_dir(base, &base.join(rel_path), String::new(), mode, compiler) + } _ => Ok(hashmap! { module_name.clone() => CompiledModule { code: self.compile_single(mode, module_name, compiler)?, @@ -107,8 +99,8 @@ impl CompilationSource { compiler: &dyn Compiler, ) -> Result<CodeObject, Diagnostic> { match &self.kind { - CompilationSourceKind::File(rel_path) => { - let path = CARGO_MANIFEST_DIR.join(rel_path); + CompilationSourceKind::File { base, rel_path } => { + let path = base.join(rel_path); let source = fs::read_to_string(&path).map_err(|err| { Diagnostic::spans_error( self.span, @@ -124,7 +116,7 @@ impl CompilationSource { compiler, || "string literal", ), - CompilationSourceKind::Dir(_) => { + CompilationSourceKind::Dir { .. } => { unreachable!("Can't use compile_single with directory source") } } @@ -132,6 +124,7 @@ impl CompilationSource { fn compile_dir( &self, + base: &Path, path: &Path, parent: String, mode: Mode, @@ -160,6 +153,7 @@ impl CompilationSource { })?; if path.is_dir() { code_map.extend(self.compile_dir( + base, &path, if parent.is_empty() { file_name.to_string() @@ -188,10 +182,7 @@ impl CompilationSource { ) })?; self.compile_string(&source, mode, module_name.clone(), compiler, || { - path.strip_prefix(&*CARGO_MANIFEST_DIR) - .ok() - .unwrap_or(&path) - .display() + path.strip_prefix(base).ok().unwrap_or(&path).display() }) }; let code = compile_path(&path).or_else(|e| { @@ -257,6 +248,16 @@ impl PyCompileArgs { .get_ident() .ok_or_else(|| meta.error("unknown arg"))?; let check_str = || meta.value()?.call(parse_str); + let str_path = || { + let s = check_str()?; + let mut base_path = s + .span() + .unwrap() + .local_file() + .ok_or_else(|| err_span!(s, "filepath literal has no span information"))?; + base_path.pop(); + Ok::<_, syn::Error>((base_path, PathBuf::from(s.value()))) + }; if ident == "mode" { let s = check_str()?; match s.value().parse() { @@ -274,9 +275,9 @@ impl PyCompileArgs { }); } else if ident == "file" { assert_source_empty(&source)?; - let path = check_str()?.value().into(); + let (base, rel_path) = str_path()?; source = Some(CompilationSource { - kind: CompilationSourceKind::File(path), + kind: CompilationSourceKind::File { base, rel_path }, span: (ident.span(), meta.input.cursor().span()), }); } else if ident == "dir" { @@ -285,9 +286,9 @@ impl PyCompileArgs { } assert_source_empty(&source)?; - let path = check_str()?.value().into(); + let (base, rel_path) = str_path()?; source = Some(CompilationSource { - kind: CompilationSourceKind::Dir(path), + kind: CompilationSourceKind::Dir { base, rel_path }, span: (ident.span(), meta.input.cursor().span()), }); } else if ident == "crate_name" { diff --git a/crates/derive-impl/src/from_args.rs b/crates/derive-impl/src/from_args.rs index 667f887e81c..9f2d0460fb0 100644 --- a/crates/derive-impl/src/from_args.rs +++ b/crates/derive-impl/src/from_args.rs @@ -37,6 +37,7 @@ struct ArgAttribute { name: Option<String>, kind: ParameterKind, default: Option<DefaultValue>, + error_msg: Option<String>, } impl ArgAttribute { @@ -63,6 +64,7 @@ impl ArgAttribute { name: None, kind, default: None, + error_msg: None, }); return Ok(()); }; @@ -94,6 +96,12 @@ impl ArgAttribute { } let val = meta.value()?.parse::<syn::LitStr>()?; self.name = Some(val.value()) + } else if meta.path.is_ident("error_msg") { + if self.error_msg.is_some() { + return Err(meta.error("already have an error_msg")); + } + let val = meta.value()?.parse::<syn::LitStr>()?; + self.error_msg = Some(val.value()) } else { return Err(meta.error("Unrecognized pyarg attribute")); } @@ -146,8 +154,15 @@ fn generate_field((i, field): (usize, &Field)) -> Result<TokenStream> { .or(name_string) .ok_or_else(|| err_span!(field, "field in tuple struct must have name attribute"))?; - let middle = quote! { - .map(|x| ::rustpython_vm::convert::TryFromObject::try_from_object(vm, x)).transpose()? + let middle = if let Some(error_msg) = &attr.error_msg { + quote! { + .map(|x| ::rustpython_vm::convert::TryFromObject::try_from_object(vm, x) + .map_err(|_| vm.new_type_error(#error_msg))).transpose()? + } + } else { + quote! { + .map(|x| ::rustpython_vm::convert::TryFromObject::try_from_object(vm, x)).transpose()? + } }; let ending = if let Some(default) = attr.default { diff --git a/crates/derive-impl/src/pyclass.rs b/crates/derive-impl/src/pyclass.rs index 028d3d7c292..f88fa059817 100644 --- a/crates/derive-impl/src/pyclass.rs +++ b/crates/derive-impl/src/pyclass.rs @@ -574,51 +574,80 @@ pub(crate) fn impl_pyclass(attr: PunctuatedNestedMeta, item: Item) -> Result<Tok )?; const ALLOWED_TRAVERSE_OPTS: &[&str] = &["manual"]; - // try to know if it have a `#[pyclass(trace)]` exist on this struct - // TODO(discord9): rethink on auto detect `#[Derive(PyTrace)]` - - // 1. no `traverse` at all: generate a dummy try_traverse - // 2. `traverse = "manual"`: generate a try_traverse, but not #[derive(Traverse)] - // 3. `traverse`: generate a try_traverse, and #[derive(Traverse)] - let (maybe_trace_code, derive_trace) = { - if class_meta.inner()._has_key("traverse")? { - let maybe_trace_code = quote! { - impl ::rustpython_vm::object::MaybeTraverse for #ident { - const IS_TRACE: bool = true; - fn try_traverse(&self, tracer_fn: &mut ::rustpython_vm::object::TraverseFn) { - ::rustpython_vm::object::Traverse::traverse(self, tracer_fn); - } + // Generate MaybeTraverse impl with both traverse and clear support + // + // For traverse: + // 1. no `traverse` at all: HAS_TRAVERSE = false, try_traverse does nothing + // 2. `traverse = "manual"`: HAS_TRAVERSE = true, but no #[derive(Traverse)] + // 3. `traverse`: HAS_TRAVERSE = true, and #[derive(Traverse)] + // + // For clear (tp_clear): + // 1. no `clear`: HAS_CLEAR = HAS_TRAVERSE (default: same as traverse) + // 2. `clear` or `clear = true`: HAS_CLEAR = true, try_clear calls Traverse::clear + // 3. `clear = false`: HAS_CLEAR = false (rare: traverse without clear) + let has_traverse = class_meta.inner()._has_key("traverse")?; + let has_clear = if class_meta.inner()._has_key("clear")? { + // If clear attribute is present, use its value + class_meta.inner()._bool("clear")? + } else { + // If clear attribute is absent, default to same as traverse + has_traverse + }; + + let derive_trace = if has_traverse { + // _optional_str returns Err when key exists without value (e.g., `traverse` vs `traverse = "manual"`) + // We want to derive Traverse in that case, so we handle Err as Ok(None) + let value = class_meta.inner()._optional_str("traverse").ok().flatten(); + if let Some(s) = value { + if !ALLOWED_TRAVERSE_OPTS.contains(&s.as_str()) { + bail_span!( + item, + "traverse attribute only accept {ALLOWED_TRAVERSE_OPTS:?} as value or no value at all", + ); + } + assert_eq!(s, "manual"); + quote! {} + } else { + quote! {#[derive(Traverse)]} + } + } else { + quote! {} + }; + + let maybe_traverse_code = { + let try_traverse_body = if has_traverse { + quote! { + ::rustpython_vm::object::Traverse::traverse(self, tracer_fn); + } + } else { + quote! { + // do nothing + } + }; + + let try_clear_body = if has_clear { + quote! { + ::rustpython_vm::object::Traverse::clear(self, out); + } + } else { + quote! { + // do nothing + } + }; + + quote! { + impl ::rustpython_vm::object::MaybeTraverse for #ident { + const HAS_TRAVERSE: bool = #has_traverse; + const HAS_CLEAR: bool = #has_clear; + + fn try_traverse(&self, tracer_fn: &mut ::rustpython_vm::object::TraverseFn) { + #try_traverse_body } - }; - // if the key `traverse` exist but not as key-value, _optional_str return Err(...) - // so we need to check if it is Ok(Some(...)) - let value = class_meta.inner()._optional_str("traverse"); - let derive_trace = if let Ok(Some(s)) = value { - if !ALLOWED_TRAVERSE_OPTS.contains(&s.as_str()) { - bail_span!( - item, - "traverse attribute only accept {ALLOWED_TRAVERSE_OPTS:?} as value or no value at all", - ); + + fn try_clear(&mut self, out: &mut ::std::vec::Vec<::rustpython_vm::PyObjectRef>) { + #try_clear_body } - assert_eq!(s, "manual"); - quote! {} - } else { - quote! {#[derive(Traverse)]} - }; - (maybe_trace_code, derive_trace) - } else { - ( - // a dummy impl, which do nothing - // #attrs - quote! { - impl ::rustpython_vm::object::MaybeTraverse for #ident { - fn try_traverse(&self, tracer_fn: &mut ::rustpython_vm::object::TraverseFn) { - // do nothing - } - } - }, - quote! {}, - ) + } } }; @@ -635,13 +664,10 @@ pub(crate) fn impl_pyclass(attr: PunctuatedNestedMeta, item: Item) -> Result<Tok quote! { // static_assertions::const_assert!(std::mem::size_of::<#base_type>() <= std::mem::size_of::<#ident>()); impl ::rustpython_vm::PyPayload for #ident { - #[inline] - fn payload_type_id() -> ::std::any::TypeId { - <#base_type as ::rustpython_vm::PyPayload>::payload_type_id() - } + const PAYLOAD_TYPE_ID: ::core::any::TypeId = <#base_type as ::rustpython_vm::PyPayload>::PAYLOAD_TYPE_ID; #[inline] - fn validate_downcastable_from(obj: &::rustpython_vm::PyObject) -> bool { + unsafe fn validate_downcastable_from(obj: &::rustpython_vm::PyObject) -> bool { <Self as ::rustpython_vm::class::PyClassDef>::BASICSIZE <= obj.class().slots.basicsize && obj.class().fast_issubclass(<Self as ::rustpython_vm::class::StaticType>::static_type()) } @@ -678,7 +704,7 @@ pub(crate) fn impl_pyclass(attr: PunctuatedNestedMeta, item: Item) -> Result<Tok let ret = quote! { #derive_trace #item - #maybe_trace_code + #maybe_traverse_code #class_def #impl_payload #empty_impl @@ -781,36 +807,10 @@ pub(crate) fn impl_pyexception_impl(attr: PunctuatedNestedMeta, item: Item) -> R } }; - // We need this method, because of how `CPython` copies `__init__` - // from `BaseException` in `SimpleExtendsException` macro. - // See: `(initproc)BaseException_init` - // spell-checker:ignore initproc - let slot_init = if with_contains(&with_items, "Initializer") { - quote!() - } else { - with_items.push(Ident::new("Initializer", Span::call_site())); - quote! { - impl ::rustpython_vm::types::Initializer for #self_ty { - type Args = ::rustpython_vm::function::FuncArgs; - - fn slot_init( - zelf: ::rustpython_vm::PyObjectRef, - args: ::rustpython_vm::function::FuncArgs, - vm: &::rustpython_vm::VirtualMachine, - ) -> ::rustpython_vm::PyResult<()> { - <Self as ::rustpython_vm::class::PyClassDef>::Base::slot_init(zelf, args, vm) - } - - fn init( - _zelf: ::rustpython_vm::PyRef<Self>, - _args: Self::Args, - _vm: &::rustpython_vm::VirtualMachine - ) -> ::rustpython_vm::PyResult<()> { - unreachable!("slot_init is defined") - } - } - } - }; + // SimpleExtendsException: inherits BaseException_init from the base class via MRO. + // Only exceptions that explicitly specify `with(Initializer)` will have + // their own __init__ in __dict__. + let slot_init = quote!(); let extra_attrs_tokens = if extra_attrs.is_empty() { quote!() @@ -1207,13 +1207,18 @@ where let item_meta = MemberItemMeta::from_attr(ident.clone(), &item_attr)?; let (py_name, member_item_kind) = item_meta.member_name()?; - let member_kind = match item_meta.member_kind()? { - Some(s) => match s.as_str() { - "bool" => MemberKind::Bool, - _ => unreachable!(), - }, - _ => MemberKind::ObjectEx, - }; + let member_kind = item_meta.member_kind()?; + if let Some(ref s) = member_kind { + match s.as_str() { + "bool" | "object" => {} + other => { + return Err(self.new_syn_error( + args.item.span(), + &format!("unknown member type '{other}'"), + )); + } + } + } // Add #[allow(non_snake_case)] for setter methods if matches!(member_item_kind, MemberItemKind::Set) { @@ -1393,37 +1398,47 @@ impl ToTokens for GetSetNursery { } } +/// Member kind as string, matching `rustpython_vm::builtins::descriptor::MemberKind` variants. +/// None means ObjectEx (default). Valid values: "bool", "object". +type MemberKindStr = Option<String>; + #[derive(Default)] -#[allow(clippy::type_complexity)] struct MemberNursery { - map: HashMap<(String, MemberKind), (Option<Ident>, Option<Ident>)>, + map: HashMap<String, MemberNurseryEntry>, validated: bool, } +struct MemberNurseryEntry { + kind: MemberKindStr, + getter: Option<Ident>, + setter: Option<Ident>, +} + enum MemberItemKind { Get, Set, } -#[derive(Eq, PartialEq, Hash)] -enum MemberKind { - Bool, - ObjectEx, -} - impl MemberNursery { fn add_item( &mut self, name: String, kind: MemberItemKind, - member_kind: MemberKind, + member_kind: MemberKindStr, item_ident: Ident, ) -> Result<()> { assert!(!self.validated, "new item is not allowed after validation"); - let entry = self.map.entry((name.clone(), member_kind)).or_default(); + let entry = self + .map + .entry(name.clone()) + .or_insert_with(|| MemberNurseryEntry { + kind: member_kind, + getter: None, + setter: None, + }); let func = match kind { - MemberItemKind::Get => &mut entry.0, - MemberItemKind::Set => &mut entry.1, + MemberItemKind::Get => &mut entry.getter, + MemberItemKind::Set => &mut entry.setter, }; if func.is_some() { bail_span!(item_ident, "Multiple member accessors with name '{}'", name); @@ -1434,10 +1449,10 @@ impl MemberNursery { fn validate(&mut self) -> Result<()> { let mut errors = Vec::new(); - for ((name, _), (getter, setter)) in &self.map { - if getter.is_none() { + for (name, entry) in &self.map { + if entry.getter.is_none() { errors.push(err_span!( - setter.as_ref().unwrap(), + entry.setter.as_ref().unwrap(), "Member '{}' is missing a getter", name )); @@ -1452,30 +1467,31 @@ impl MemberNursery { impl ToTokens for MemberNursery { fn to_tokens(&self, tokens: &mut TokenStream) { assert!(self.validated, "Call `validate()` before token generation"); - let properties = self - .map - .iter() - .map(|((name, member_kind), (getter, setter))| { - let setter = match setter { - Some(setter) => quote_spanned! { setter.span() => Some(Self::#setter)}, - None => quote! { None }, - }; - let member_kind = match member_kind { - MemberKind::Bool => { - quote!(::rustpython_vm::builtins::descriptor::MemberKind::Bool) - } - MemberKind::ObjectEx => { - quote!(::rustpython_vm::builtins::descriptor::MemberKind::ObjectEx) - } - }; - quote_spanned! { getter.span() => - class.set_str_attr( - #name, - ctx.new_member(#name, #member_kind, Self::#getter, #setter, class), - ctx, - ); + let properties = self.map.iter().map(|(name, entry)| { + let setter = match &entry.setter { + Some(setter) => quote_spanned! { setter.span() => Some(Self::#setter)}, + None => quote! { None }, + }; + let member_kind = match entry.kind.as_deref() { + Some("bool") => { + quote!(::rustpython_vm::builtins::descriptor::MemberKind::Bool) } - }); + Some("object") => { + quote!(::rustpython_vm::builtins::descriptor::MemberKind::Object) + } + _ => { + quote!(::rustpython_vm::builtins::descriptor::MemberKind::ObjectEx) + } + }; + let getter = entry.getter.as_ref().unwrap(); + quote_spanned! { getter.span() => + class.set_str_attr( + #name, + ctx.new_member(#name, #member_kind, Self::#getter, #setter, class), + ctx, + ); + } + }); tokens.extend(properties); } } diff --git a/crates/derive-impl/src/pymodule.rs b/crates/derive-impl/src/pymodule.rs index 3689ac97fd8..278aab37dbc 100644 --- a/crates/derive-impl/src/pymodule.rs +++ b/crates/derive-impl/src/pymodule.rs @@ -58,7 +58,7 @@ struct ModuleContext { name: String, function_items: FunctionNursery, attribute_items: ItemNursery, - has_extend_module: bool, // TODO: check if `fn extend_module` exists + has_module_exec: bool, errors: Vec<syn::Error>, } @@ -82,6 +82,12 @@ pub fn impl_pymodule(attr: PunctuatedNestedMeta, module_item: Item) -> Result<To // collect to context for item in items.iter_mut() { + // Check if module_exec function is already defined + if let Item::Fn(func) = item + && func.sig.ident == "module_exec" + { + context.has_module_exec = true; + } if matches!(item, Item::Impl(_) | Item::Trait(_)) { // #[pyclass] implementations continue; @@ -123,7 +129,7 @@ pub fn impl_pymodule(attr: PunctuatedNestedMeta, module_item: Item) -> Result<To pub(crate) const DOC: Option<&'static str> = #doc; }, parse_quote! { - pub(crate) fn __module_def( + pub(crate) fn module_def( ctx: &::rustpython_vm::Context, ) -> &'static ::rustpython_vm::builtins::PyModuleDef { DEF.get_or_init(|| { @@ -133,29 +139,17 @@ pub fn impl_pymodule(attr: PunctuatedNestedMeta, module_item: Item) -> Result<To methods: METHOD_DEFS, slots: Default::default(), }; - def.slots.exec = Some(extend_module); + def.slots.exec = Some(module_exec); def }) } }, - parse_quote! { - #[allow(dead_code)] - pub(crate) fn make_module( - vm: &::rustpython_vm::VirtualMachine - ) -> ::rustpython_vm::PyRef<::rustpython_vm::builtins::PyModule> { - use ::rustpython_vm::PyPayload; - let module = ::rustpython_vm::builtins::PyModule::from_def(__module_def(&vm.ctx)).into_ref(&vm.ctx); - __init_dict(vm, &module); - extend_module(vm, &module).unwrap(); - module - } - }, ]); } - if !is_submodule && !context.has_extend_module { + if !is_submodule && !context.has_module_exec { items.push(parse_quote! { - pub(crate) fn extend_module(vm: &::rustpython_vm::VirtualMachine, module: &::rustpython_vm::Py<::rustpython_vm::builtins::PyModule>) -> ::rustpython_vm::PyResult<()> { - __extend_module(vm, module); + pub(crate) fn module_exec(vm: &::rustpython_vm::VirtualMachine, module: &::rustpython_vm::Py<::rustpython_vm::builtins::PyModule>) -> ::rustpython_vm::PyResult<()> { + __module_exec(vm, module); Ok(()) } }); @@ -192,11 +186,10 @@ pub fn impl_pymodule(attr: PunctuatedNestedMeta, module_item: Item) -> Result<To } }, parse_quote! { - pub(crate) fn __extend_module( + pub(crate) fn __module_exec( vm: &::rustpython_vm::VirtualMachine, module: &::rustpython_vm::Py<::rustpython_vm::builtins::PyModule>, ) { - module.__init_methods(vm).unwrap(); __init_attributes(vm, module); } }, diff --git a/crates/derive-impl/src/pystructseq.rs b/crates/derive-impl/src/pystructseq.rs index 32b603fe478..ccff85fae79 100644 --- a/crates/derive-impl/src/pystructseq.rs +++ b/crates/derive-impl/src/pystructseq.rs @@ -592,13 +592,10 @@ pub(crate) fn impl_pystruct_sequence( // Subtype uses base type's payload_type_id impl ::rustpython_vm::PyPayload for #pytype_ident { - #[inline] - fn payload_type_id() -> ::std::any::TypeId { - <::rustpython_vm::builtins::PyTuple as ::rustpython_vm::PyPayload>::payload_type_id() - } + const PAYLOAD_TYPE_ID: ::core::any::TypeId = <::rustpython_vm::builtins::PyTuple as ::rustpython_vm::PyPayload>::PAYLOAD_TYPE_ID; #[inline] - fn validate_downcastable_from(obj: &::rustpython_vm::PyObject) -> bool { + unsafe fn validate_downcastable_from(obj: &::rustpython_vm::PyObject) -> bool { obj.class().fast_issubclass(<Self as ::rustpython_vm::class::StaticType>::static_type()) } @@ -609,9 +606,16 @@ pub(crate) fn impl_pystruct_sequence( // MaybeTraverse - delegate to inner PyTuple impl ::rustpython_vm::object::MaybeTraverse for #pytype_ident { + const HAS_TRAVERSE: bool = true; + const HAS_CLEAR: bool = true; + fn try_traverse(&self, traverse_fn: &mut ::rustpython_vm::object::TraverseFn<'_>) { self.0.try_traverse(traverse_fn) } + + fn try_clear(&mut self, out: &mut ::std::vec::Vec<::rustpython_vm::PyObjectRef>) { + self.0.try_clear(out) + } } // PySubclass for proper inheritance diff --git a/crates/derive-impl/src/util.rs b/crates/derive-impl/src/util.rs index 6be1fcdf7ad..b09ad9c93fe 100644 --- a/crates/derive-impl/src/util.rs +++ b/crates/derive-impl/src/util.rs @@ -372,6 +372,7 @@ impl ItemMeta for ClassItemMeta { "ctx", "impl", "traverse", + "clear", // tp_clear ]; fn from_inner(inner: ItemMetaInner) -> Self { diff --git a/crates/derive/src/lib.rs b/crates/derive/src/lib.rs index 5a3ff84c63a..1183b75b714 100644 --- a/crates/derive/src/lib.rs +++ b/crates/derive/src/lib.rs @@ -143,8 +143,8 @@ pub fn pyexception(attr: TokenStream, item: TokenStream) -> TokenStream { } /// This attribute must be applied to an inline module. -/// It defines a Python module in the form a `make_module` function in the module; -/// this has to be used in a `get_module_inits` to properly register the module. +/// It defines a Python module in the form of a `module_def` function in the module; +/// this has to be used in a `add_native_module` to properly register the module. /// Additionally, this macro defines 'MODULE_NAME' and 'DOC' in the module. /// # Arguments /// - `name`: the name of the python module, diff --git a/crates/jit/Cargo.toml b/crates/jit/Cargo.toml index fde10b7c006..e3e89636181 100644 --- a/crates/jit/Cargo.toml +++ b/crates/jit/Cargo.toml @@ -17,12 +17,13 @@ num-traits = { workspace = true } thiserror = { workspace = true } libffi = { workspace = true } -cranelift = "0.127" -cranelift-jit = "0.127" -cranelift-module = "0.127" +cranelift = "0.128" +cranelift-jit = "0.128" +cranelift-module = "0.128" [dev-dependencies] rustpython-derive = { workspace = true } +rustpython-wtf8 = { workspace = true } approx = "0.5.1" diff --git a/crates/jit/src/instructions.rs b/crates/jit/src/instructions.rs index d6e3a07e111..19931038fe0 100644 --- a/crates/jit/src/instructions.rs +++ b/crates/jit/src/instructions.rs @@ -27,6 +27,7 @@ enum JitValue { Float(Value), Bool(Value), None, + Null, Tuple(Vec<JitValue>), FuncRef(FuncRef), } @@ -45,14 +46,14 @@ impl JitValue { JitValue::Int(_) => Some(JitType::Int), JitValue::Float(_) => Some(JitType::Float), JitValue::Bool(_) => Some(JitType::Bool), - JitValue::None | JitValue::Tuple(_) | JitValue::FuncRef(_) => None, + JitValue::None | JitValue::Null | JitValue::Tuple(_) | JitValue::FuncRef(_) => None, } } fn into_value(self) -> Option<Value> { match self { JitValue::Int(val) | JitValue::Float(val) | JitValue::Bool(val) => Some(val), - JitValue::None | JitValue::Tuple(_) | JitValue::FuncRef(_) => None, + JitValue::None | JitValue::Null | JitValue::Tuple(_) | JitValue::FuncRef(_) => None, } } } @@ -139,7 +140,9 @@ impl<'a, 'b> FunctionCompiler<'a, 'b> { } JitValue::Bool(val) => Ok(val), JitValue::None => Ok(self.builder.ins().iconst(types::I8, 0)), - JitValue::Tuple(_) | JitValue::FuncRef(_) => Err(JitCompileError::NotSupported), + JitValue::Null | JitValue::Tuple(_) | JitValue::FuncRef(_) => { + Err(JitCompileError::NotSupported) + } } } @@ -208,8 +211,9 @@ impl<'a, 'b> FunctionCompiler<'a, 'b> { // If that was an unconditional branch or return, mark future instructions unreachable match instruction { Instruction::ReturnValue - | Instruction::ReturnConst { .. } - | Instruction::Jump { .. } => { + | Instruction::JumpBackward { .. } + | Instruction::JumpBackwardNoInterrupt { .. } + | Instruction::JumpForward { .. } => { in_unreachable_code = true; } _ => {} @@ -454,7 +458,7 @@ impl<'a, 'b> FunctionCompiler<'a, 'b> { self.stack.push(JitValue::Tuple(elements)); Ok(()) } - Instruction::CallFunctionPositional { nargs } => { + Instruction::Call { nargs } => { let nargs = nargs.get(arg); let mut args = Vec::new(); @@ -463,6 +467,12 @@ impl<'a, 'b> FunctionCompiler<'a, 'b> { args.push(arg.into_value().unwrap()); } + // Pop self_or_null (should be Null for JIT-compiled recursive calls) + let self_or_null = self.stack.pop().ok_or(JitCompileError::BadBytecode)?; + if !matches!(self_or_null, JitValue::Null) { + return Err(JitCompileError::NotSupported); + } + match self.stack.pop().ok_or(JitCompileError::BadBytecode)? { JitValue::FuncRef(reference) => { let call = self.builder.ins().call(reference, &args); @@ -474,6 +484,10 @@ impl<'a, 'b> FunctionCompiler<'a, 'b> { _ => Err(JitCompileError::BadBytecode), } } + Instruction::PushNull => { + self.stack.push(JitValue::Null); + Ok(()) + } Instruction::CallIntrinsic1 { func } => { match func.get(arg) { IntrinsicFunction1::UnaryPositive => { @@ -489,7 +503,7 @@ impl<'a, 'b> FunctionCompiler<'a, 'b> { _ => Err(JitCompileError::NotSupported), } } - Instruction::CompareOperation { op, .. } => { + Instruction::CompareOp { op, .. } => { let op = op.get(arg); // the rhs is popped off first let b = self.stack.pop().ok_or(JitCompileError::BadBytecode)?; @@ -545,7 +559,9 @@ impl<'a, 'b> FunctionCompiler<'a, 'b> { } Instruction::ExtendedArg => Ok(()), - Instruction::Jump { target } => { + Instruction::JumpBackward { target } + | Instruction::JumpBackwardNoInterrupt { target } + | Instruction::JumpForward { target } => { let target_block = self.get_or_create_block(target.get(arg)); self.builder.ins().jump(target_block, &[]); Ok(()) @@ -556,7 +572,13 @@ impl<'a, 'b> FunctionCompiler<'a, 'b> { self.stack.push(val); Ok(()) } - Instruction::LoadFast(idx) => { + Instruction::LoadSmallInt { idx } => { + let small_int = idx.get(arg) as i64; + let val = self.builder.ins().iconst(types::I64, small_int); + self.stack.push(JitValue::Int(val)); + Ok(()) + } + Instruction::LoadFast(idx) | Instruction::LoadFastBorrow(idx) => { let local = self.variables[idx.get(arg) as usize] .as_ref() .ok_or(JitCompileError::BadBytecode)?; @@ -566,6 +588,22 @@ impl<'a, 'b> FunctionCompiler<'a, 'b> { )); Ok(()) } + Instruction::LoadFastLoadFast { arg: packed } + | Instruction::LoadFastBorrowLoadFastBorrow { arg: packed } => { + let oparg = packed.get(arg); + let idx1 = oparg >> 4; + let idx2 = oparg & 0xF; + for idx in [idx1, idx2] { + let local = self.variables[idx as usize] + .as_ref() + .ok_or(JitCompileError::BadBytecode)?; + self.stack.push(JitValue::from_type_and_value( + local.ty.clone(), + self.builder.use_var(local.var), + )); + } + Ok(()) + } Instruction::LoadGlobal(idx) => { let name = &bytecode.names[idx.get(arg) as usize]; @@ -577,10 +615,6 @@ impl<'a, 'b> FunctionCompiler<'a, 'b> { } } Instruction::Nop => Ok(()), - Instruction::PopBlock => { - // TODO: block support - Ok(()) - } Instruction::PopJumpIfFalse { target } => { let cond = self.stack.pop().ok_or(JitCompileError::BadBytecode)?; let val = self.boolean_val(cond)?; @@ -615,11 +649,6 @@ impl<'a, 'b> FunctionCompiler<'a, 'b> { // TODO: Implement the resume instruction Ok(()) } - Instruction::ReturnConst { idx } => { - let val = self - .prepare_const(bytecode.constants[idx.get(arg) as usize].borrow_constant())?; - self.return_value(val) - } Instruction::ReturnValue => { let val = self.stack.pop().ok_or(JitCompileError::BadBytecode)?; self.return_value(val) diff --git a/crates/jit/tests/common.rs b/crates/jit/tests/common.rs index 17c280ec3c9..a862d9eef69 100644 --- a/crates/jit/tests/common.rs +++ b/crates/jit/tests/common.rs @@ -3,19 +3,20 @@ use rustpython_compiler_core::bytecode::{ CodeObject, ConstantData, Instruction, OpArg, OpArgState, }; use rustpython_jit::{CompiledCode, JitType}; +use rustpython_wtf8::{Wtf8, Wtf8Buf}; use std::collections::HashMap; #[derive(Debug, Clone)] pub struct Function { code: Box<CodeObject>, - annotations: HashMap<String, StackValue>, + annotations: HashMap<Wtf8Buf, StackValue>, } impl Function { pub fn compile(self) -> CompiledCode { let mut arg_types = Vec::new(); for arg in self.code.arg_names().args { - let arg_type = match self.annotations.get(arg) { + let arg_type = match self.annotations.get(AsRef::<Wtf8>::as_ref(arg.as_str())) { Some(StackValue::String(annotation)) => match annotation.as_str() { "int" => JitType::Int, "float" => JitType::Float, @@ -27,7 +28,7 @@ impl Function { arg_types.push(arg_type); } - let ret_type = match self.annotations.get("return") { + let ret_type = match self.annotations.get(AsRef::<Wtf8>::as_ref("return")) { Some(StackValue::String(annotation)) => match annotation.as_str() { "int" => Some(JitType::Int), "float" => Some(JitType::Float), @@ -45,7 +46,7 @@ impl Function { enum StackValue { String(String), None, - Map(HashMap<String, StackValue>), + Map(HashMap<Wtf8Buf, StackValue>), Code(Box<CodeObject>), Function(Function), } @@ -63,6 +64,97 @@ impl From<ConstantData> for StackValue { } } +/// Extract annotations from an annotate function's bytecode. +/// The annotate function uses BUILD_MAP with key-value pairs loaded before it. +/// Keys are parameter names (from LOAD_CONST), values are type names (from LOAD_NAME/LOAD_GLOBAL). +fn extract_annotations_from_annotate_code(code: &CodeObject) -> HashMap<Wtf8Buf, StackValue> { + let mut annotations = HashMap::new(); + let mut stack: Vec<(bool, usize)> = Vec::new(); // (is_const, index) + let mut op_arg_state = OpArgState::default(); + + for &word in code.instructions.iter() { + let (instruction, arg) = op_arg_state.get(word); + + match instruction { + Instruction::LoadConst { idx } => { + stack.push((true, idx.get(arg) as usize)); + } + Instruction::LoadName(idx) | Instruction::LoadGlobal(idx) => { + stack.push((false, idx.get(arg) as usize)); + } + Instruction::BuildMap { size, .. } => { + let count = size.get(arg) as usize; + // Stack has key-value pairs in order: k1, v1, k2, v2, ... + // So we need count * 2 items from the stack + let start = stack.len().saturating_sub(count * 2); + let pairs: Vec<_> = stack.drain(start..).collect(); + + for chunk in pairs.chunks(2) { + if chunk.len() == 2 { + let (key_is_const, key_idx) = chunk[0]; + let (val_is_const, val_idx) = chunk[1]; + + // Key should be a const string (parameter name) + if key_is_const + && let ConstantData::Str { value } = &code.constants[key_idx] + { + let param_name = value; + // Value can be a name (type ref) or a const string (forward ref) + let type_name = if val_is_const { + match code.constants.get(val_idx) { + Some(ConstantData::Str { value }) => value + .as_str() + .map(|s| s.to_owned()) + .unwrap_or_else(|_| value.to_string_lossy().into_owned()), + Some(other) => panic!( + "Unsupported annotation const for '{:?}' at idx {}: {:?}", + param_name, val_idx, other + ), + None => panic!( + "Annotation const idx out of bounds for '{:?}': {} (len={})", + param_name, + val_idx, + code.constants.len() + ), + } + } else { + match code.names.get(val_idx) { + Some(name) => name.clone(), + None => panic!( + "Annotation name idx out of bounds for '{:?}': {} (len={})", + param_name, + val_idx, + code.names.len() + ), + } + }; + annotations.insert(param_name.clone(), StackValue::String(type_name)); + } + } + } + // Return after processing BUILD_MAP - we got our annotations + return annotations; + } + Instruction::Resume { .. } + | Instruction::LoadFast(_) + | Instruction::CompareOp { .. } + | Instruction::ExtendedArg => { + // Ignore these instructions for annotation extraction + } + Instruction::ReturnValue => { + // End of function - return what we have + return annotations; + } + _ => { + // For other instructions, clear the stack tracking as we don't understand the effect + stack.clear(); + } + } + } + + annotations +} + pub struct StackMachine { stack: Vec<StackValue>, locals: HashMap<String, StackValue>, @@ -92,14 +184,17 @@ impl StackMachine { names: &[String], ) -> ControlFlow<()> { match instruction { + Instruction::Resume { .. } => { + // No-op for JIT tests - just marks function entry point + } Instruction::LoadConst { idx } => { let idx = idx.get(arg); self.stack.push(constants[idx as usize].clone().into()) } - Instruction::LoadNameAny(idx) => self + Instruction::LoadName(idx) => self .stack .push(StackValue::String(names[idx.get(arg) as usize].clone())), - Instruction::StoreLocal(idx) => { + Instruction::StoreName(idx) => { let idx = idx.get(arg); self.locals .insert(names[idx as usize].clone(), self.stack.pop().unwrap()); @@ -114,7 +209,7 @@ impl StackMachine { for _ in 0..size.get(arg) { let value = self.stack.pop().unwrap(); let name = if let Some(StackValue::String(name)) = self.stack.pop() { - name + Wtf8Buf::from(name) } else { unimplemented!("no string keys isn't yet supported in py_function!") }; @@ -143,13 +238,31 @@ impl StackMachine { }; let attr_value = self.stack.pop().expect("Expected attribute value on stack"); - // For now, we only handle ANNOTATIONS flag in JIT tests - if attr - .get(arg) + let flags = attr.get(arg); + + // Handle ANNOTATE flag (PEP 649 style - Python 3.14+) + // The attr_value is a function that returns annotations when called + if flags.contains(rustpython_compiler_core::bytecode::MakeFunctionFlags::ANNOTATE) { + if let StackValue::Function(annotate_func) = attr_value { + // Parse the annotate function's bytecode to extract annotations + // The pattern is: LOAD_CONST (key), LOAD_NAME (value), ... BUILD_MAP + let annotate_code = &annotate_func.code; + let annotations = extract_annotations_from_annotate_code(annotate_code); + + let updated_func = Function { + code: func.code, + annotations, + }; + self.stack.push(StackValue::Function(updated_func)); + } else { + panic!("Expected annotate function for ANNOTATE flag"); + } + } + // Handle old ANNOTATIONS flag (Python 3.12 style) + else if flags .contains(rustpython_compiler_core::bytecode::MakeFunctionFlags::ANNOTATIONS) { if let StackValue::Map(annotations) = attr_value { - // Update function's annotations let updated_func = Function { code: func.code, annotations, @@ -160,15 +273,9 @@ impl StackMachine { } } else { // For other attributes, just push the function back unchanged - // (since JIT tests mainly care about type annotations) self.stack.push(StackValue::Function(func)); } } - Instruction::ReturnConst { idx } => { - let idx = idx.get(arg); - self.stack.push(constants[idx as usize].clone().into()); - return ControlFlow::Break(()); - } Instruction::ReturnValue => return ControlFlow::Break(()), Instruction::ExtendedArg => {} _ => unimplemented!( diff --git a/crates/pylib/build.rs b/crates/pylib/build.rs index 9b135690f82..f96ef9b477c 100644 --- a/crates/pylib/build.rs +++ b/crates/pylib/build.rs @@ -11,15 +11,25 @@ fn main() { process_python_libs("./Lib/**/*"); } - if cfg!(windows) - && let Ok(real_path) = std::fs::read_to_string("Lib") - { - let canonicalized_path = std::fs::canonicalize(real_path) - .expect("failed to resolve RUSTPYTHONPATH during build time"); - // Strip the extended path prefix (\\?\) that canonicalize adds on Windows - let path_str = canonicalized_path.to_str().unwrap(); - let path_str = path_str.strip_prefix(r"\\?\").unwrap_or(path_str); - println!("cargo:rustc-env=win_lib_path={path_str}"); + if cfg!(windows) { + // On Windows, the Lib entry can be either: + // 1. A text file containing the relative path (git without symlink support) + // 2. A proper symlink (git with symlink support) + // We handle both cases to resolve to the actual Lib directory. + let lib_path = if let Ok(real_path) = std::fs::read_to_string("Lib") { + // Case 1: Text file containing relative path + std::path::PathBuf::from(real_path.trim()) + } else { + // Case 2: Symlink or directory - canonicalize directly + std::path::PathBuf::from("Lib") + }; + + if let Ok(canonicalized_path) = std::fs::canonicalize(&lib_path) { + // Strip the extended path prefix (\\?\) that canonicalize adds on Windows + let path_str = canonicalized_path.to_str().unwrap(); + let path_str = path_str.strip_prefix(r"\\?\").unwrap_or(path_str); + println!("cargo:rustc-env=win_lib_path={path_str}"); + } } } diff --git a/crates/pylib/src/lib.rs b/crates/pylib/src/lib.rs index f8a47ba67da..afdff75b725 100644 --- a/crates/pylib/src/lib.rs +++ b/crates/pylib/src/lib.rs @@ -11,4 +11,4 @@ pub const LIB_PATH: &str = match option_env!("win_lib_path") { #[cfg(feature = "freeze-stdlib")] pub const FROZEN_STDLIB: &rustpython_compiler_core::frozen::FrozenLib = - rustpython_derive::py_freeze!(dir = "./Lib", crate_name = "rustpython_compiler_core"); + rustpython_derive::py_freeze!(dir = "../Lib", crate_name = "rustpython_compiler_core"); diff --git a/crates/sre_engine/src/constants.rs b/crates/sre_engine/src/constants.rs index d90c08cb374..b38ecb109b8 100644 --- a/crates/sre_engine/src/constants.rs +++ b/crates/sre_engine/src/constants.rs @@ -14,9 +14,10 @@ use bitflags::bitflags; pub const SRE_MAGIC: usize = 20230612; -#[derive(num_enum::TryFromPrimitive, Debug, PartialEq, Eq)] -#[repr(u32)] + #[allow(non_camel_case_types, clippy::upper_case_acronyms)] +#[derive(num_enum::TryFromPrimitive, Copy, Clone, Debug, PartialEq, Eq)] +#[repr(u32)] pub enum SreOpcode { FAILURE = 0, SUCCESS = 1, @@ -63,9 +64,9 @@ pub enum SreOpcode { RANGE_UNI_IGNORE = 42, } -#[derive(num_enum::TryFromPrimitive, Debug, PartialEq, Eq)] -#[repr(u32)] #[allow(non_camel_case_types, clippy::upper_case_acronyms)] +#[derive(num_enum::TryFromPrimitive, Clone, Copy, Debug, PartialEq, Eq)] +#[repr(u32)] pub enum SreAtCode { BEGINNING = 0, BEGINNING_LINE = 1, @@ -81,9 +82,9 @@ pub enum SreAtCode { UNI_NON_BOUNDARY = 11, } -#[derive(num_enum::TryFromPrimitive, Debug)] -#[repr(u32)] #[allow(non_camel_case_types, clippy::upper_case_acronyms)] +#[derive(num_enum::TryFromPrimitive, Clone, Copy, Debug)] +#[repr(u32)] pub enum SreCatCode { DIGIT = 0, NOT_DIGIT = 1, @@ -120,6 +121,7 @@ bitflags! { } bitflags! { + #[derive(Clone, Copy)] pub struct SreInfo: u32 { const PREFIX = 1; const LITERAL = 2; diff --git a/crates/stdlib/Cargo.toml b/crates/stdlib/Cargo.toml index e943470a3af..3770fc42644 100644 --- a/crates/stdlib/Cargo.toml +++ b/crates/stdlib/Cargo.toml @@ -22,6 +22,7 @@ ssl-rustls-fips = ["ssl-rustls", "aws-lc-rs/fips"] ssl-openssl = ["ssl", "openssl", "openssl-sys", "foreign-types-shared", "openssl-probe"] ssl-vendor = ["ssl-openssl", "openssl/vendored"] tkinter = ["dep:tk-sys", "dep:tcl-sys", "dep:widestring"] +flame-it = ["flame"] [dependencies] # rustpython crates @@ -33,6 +34,7 @@ ahash = { workspace = true } ascii = { workspace = true } cfg-if = { workspace = true } crossbeam-utils = { workspace = true } +flame = { workspace = true, optional = true } hex = { workspace = true } itertools = { workspace = true } indexmap = { workspace = true } @@ -83,7 +85,7 @@ unicode-bidi-mirroring = { workspace = true } # compression adler32 = "1.2.0" crc32fast = "1.3.2" -flate2 = { version = "<=1.1.5", default-features = false, features = ["zlib-rs"] } +flate2 = { version = "<=1.1.8", default-features = false, features = ["zlib-rs"] } libz-sys = { package = "libz-rs-sys", version = "0.5" } bzip2 = "0.6" @@ -115,11 +117,11 @@ dns-lookup = "3.0" # OpenSSL dependencies (optional, for ssl-openssl feature) openssl = { version = "0.10.72", optional = true } openssl-sys = { version = "0.9.110", optional = true } -openssl-probe = { version = "0.1.5", optional = true } +openssl-probe = { version = "0.2.1", optional = true } foreign-types-shared = { version = "0.1.1", optional = true } # Rustls dependencies (optional, for ssl-rustls feature) -rustls = { version = "0.23.35", default-features = false, features = ["std", "tls12", "aws_lc_rs"], optional = true } +rustls = { version = "0.23.36", default-features = false, features = ["std", "tls12", "aws_lc_rs"], optional = true } rustls-native-certs = { version = "0.8", optional = true } rustls-pemfile = { version = "2.2", optional = true } rustls-platform-verifier = { version = "0.6", optional = true } diff --git a/crates/stdlib/src/_asyncio.rs b/crates/stdlib/src/_asyncio.rs new file mode 100644 index 00000000000..6e7a8c6e0e5 --- /dev/null +++ b/crates/stdlib/src/_asyncio.rs @@ -0,0 +1,2798 @@ +//! _asyncio module - provides native asyncio support +//! +//! This module provides native implementations of Future and Task classes, + +pub(crate) use _asyncio::module_def; + +#[pymodule] +pub(crate) mod _asyncio { + use crate::{ + common::lock::PyRwLock, + vm::{ + AsObject, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, + builtins::{ + PyBaseException, PyBaseExceptionRef, PyDict, PyDictRef, PyGenericAlias, PyList, + PyListRef, PyModule, PySet, PyTuple, PyType, PyTypeRef, + }, + extend_module, + function::{FuncArgs, KwArgs, OptionalArg, OptionalOption, PySetterValue}, + protocol::PyIterReturn, + recursion::ReprGuard, + types::{ + Callable, Constructor, Destructor, Initializer, IterNext, Iterable, Representable, + SelfIter, + }, + warn, + }, + }; + use crossbeam_utils::atomic::AtomicCell; + use std::sync::atomic::{AtomicBool, AtomicI32, AtomicU64, Ordering}; + + pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { + __module_exec(vm, module); + + // Initialize module-level state + let weakref_module = vm.import("weakref", 0)?; + let weak_set_class = vm + .get_attribute_opt(weakref_module, vm.ctx.intern_str("WeakSet"))? + .ok_or_else(|| vm.new_attribute_error("WeakSet not found"))?; + let scheduled_tasks = weak_set_class.call((), vm)?; + let eager_tasks = PySet::default().into_ref(&vm.ctx); + let current_tasks = PyDict::default().into_ref(&vm.ctx); + + extend_module!(vm, module, { + "_scheduled_tasks" => scheduled_tasks, + "_eager_tasks" => eager_tasks, + "_current_tasks" => current_tasks, + }); + + // Register fork handler to clear task state in child process + #[cfg(unix)] + { + let on_fork = vm + .get_attribute_opt(module.to_owned().into(), vm.ctx.intern_str("_on_fork"))? + .expect("_on_fork not found in _asyncio module"); + vm.state.after_forkers_child.lock().push(on_fork); + } + + Ok(()) + } + + #[derive(FromArgs)] + struct AddDoneCallbackArgs { + #[pyarg(positional)] + func: PyObjectRef, + #[pyarg(named, optional)] + context: OptionalOption<PyObjectRef>, + } + + #[derive(FromArgs)] + struct CancelArgs { + #[pyarg(any, optional)] + msg: OptionalOption<PyObjectRef>, + } + + #[derive(FromArgs)] + struct LoopArg { + #[pyarg(any, name = "loop", optional)] + loop_: OptionalOption<PyObjectRef>, + } + + #[derive(FromArgs)] + struct GetStackArgs { + #[pyarg(named, optional)] + limit: OptionalOption<PyObjectRef>, + } + + #[derive(FromArgs)] + struct PrintStackArgs { + #[pyarg(named, optional)] + limit: OptionalOption<PyObjectRef>, + #[pyarg(named, optional)] + file: OptionalOption<PyObjectRef>, + } + + #[derive(Clone, Copy, PartialEq, Eq, Debug)] + enum FutureState { + Pending, + Cancelled, + Finished, + } + + impl FutureState { + fn as_str(&self) -> &'static str { + match self { + FutureState::Pending => "PENDING", + FutureState::Cancelled => "CANCELLED", + FutureState::Finished => "FINISHED", + } + } + } + + /// asyncio.Future implementation + #[pyattr] + #[pyclass(name = "Future", module = "_asyncio", traverse)] + #[derive(Debug, PyPayload)] + #[repr(C)] // Required for inheritance - ensures base field is at offset 0 in subclasses + struct PyFuture { + fut_loop: PyRwLock<Option<PyObjectRef>>, + fut_callback0: PyRwLock<Option<PyObjectRef>>, + fut_context0: PyRwLock<Option<PyObjectRef>>, + fut_callbacks: PyRwLock<Option<PyObjectRef>>, + fut_exception: PyRwLock<Option<PyObjectRef>>, + fut_exception_tb: PyRwLock<Option<PyObjectRef>>, + fut_result: PyRwLock<Option<PyObjectRef>>, + fut_source_tb: PyRwLock<Option<PyObjectRef>>, + fut_cancel_msg: PyRwLock<Option<PyObjectRef>>, + fut_cancelled_exc: PyRwLock<Option<PyObjectRef>>, + fut_awaited_by: PyRwLock<Option<PyObjectRef>>, + #[pytraverse(skip)] + fut_state: AtomicCell<FutureState>, + #[pytraverse(skip)] + fut_awaited_by_is_set: AtomicBool, + #[pytraverse(skip)] + fut_log_tb: AtomicBool, + #[pytraverse(skip)] + fut_blocking: AtomicBool, + } + + impl Constructor for PyFuture { + type Args = FuncArgs; + + fn py_new(_cls: &Py<PyType>, _args: Self::Args, _vm: &VirtualMachine) -> PyResult<Self> { + Ok(PyFuture::new_empty()) + } + } + + impl Initializer for PyFuture { + type Args = FuncArgs; + + fn init(zelf: PyRef<Self>, args: Self::Args, vm: &VirtualMachine) -> PyResult<()> { + // Future does not accept positional arguments + if !args.args.is_empty() { + return Err(vm.new_type_error("Future() takes no positional arguments".to_string())); + } + // Extract only 'loop' keyword argument + let loop_ = args.kwargs.get("loop").cloned(); + PyFuture::py_init(&zelf, loop_, vm) + } + } + + #[pyclass( + flags(BASETYPE, HAS_DICT), + with(Constructor, Initializer, Destructor, Representable, Iterable) + )] + impl PyFuture { + fn new_empty() -> Self { + Self { + fut_loop: PyRwLock::new(None), + fut_callback0: PyRwLock::new(None), + fut_context0: PyRwLock::new(None), + fut_callbacks: PyRwLock::new(None), + fut_exception: PyRwLock::new(None), + fut_exception_tb: PyRwLock::new(None), + fut_result: PyRwLock::new(None), + fut_source_tb: PyRwLock::new(None), + fut_cancel_msg: PyRwLock::new(None), + fut_cancelled_exc: PyRwLock::new(None), + fut_awaited_by: PyRwLock::new(None), + fut_state: AtomicCell::new(FutureState::Pending), + fut_awaited_by_is_set: AtomicBool::new(false), + fut_log_tb: AtomicBool::new(false), + fut_blocking: AtomicBool::new(false), + } + } + + fn py_init( + zelf: &PyRef<Self>, + loop_: Option<PyObjectRef>, + vm: &VirtualMachine, + ) -> PyResult<()> { + // Get the event loop + let loop_obj = match loop_ { + Some(l) if !vm.is_none(&l) => l, + _ => get_event_loop(vm)?, + }; + *zelf.fut_loop.write() = Some(loop_obj.clone()); + + // Check if loop has get_debug method and call it + if let Ok(Some(get_debug)) = + vm.get_attribute_opt(loop_obj.clone(), vm.ctx.intern_str("get_debug")) + && let Ok(debug) = get_debug.call((), vm) + && debug.try_to_bool(vm).unwrap_or(false) + { + // Get source traceback + if let Ok(tb_module) = vm.import("traceback", 0) + && let Ok(Some(extract_stack)) = + vm.get_attribute_opt(tb_module, vm.ctx.intern_str("extract_stack")) + && let Ok(tb) = extract_stack.call((), vm) + { + *zelf.fut_source_tb.write() = Some(tb); + } + } + + Ok(()) + } + + #[pymethod] + fn result(&self, vm: &VirtualMachine) -> PyResult { + match self.fut_state.load() { + FutureState::Pending => Err(new_invalid_state_error(vm, "Result is not ready.")), + FutureState::Cancelled => { + let exc = self.make_cancelled_error_impl(vm); + Err(exc) + } + FutureState::Finished => { + self.fut_log_tb.store(false, Ordering::Relaxed); + if let Some(exc) = self.fut_exception.read().clone() { + let exc: PyBaseExceptionRef = exc.downcast().unwrap(); + // Restore the original traceback to prevent traceback accumulation + if let Some(tb) = self.fut_exception_tb.read().clone() { + let _ = exc.set___traceback__(tb, vm); + } + Err(exc) + } else { + Ok(self + .fut_result + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none())) + } + } + } + } + + #[pymethod] + fn exception(&self, vm: &VirtualMachine) -> PyResult { + match self.fut_state.load() { + FutureState::Pending => Err(new_invalid_state_error(vm, "Exception is not set.")), + FutureState::Cancelled => { + let exc = self.make_cancelled_error_impl(vm); + Err(exc) + } + FutureState::Finished => { + self.fut_log_tb.store(false, Ordering::Relaxed); + Ok(self + .fut_exception + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none())) + } + } + } + + #[pymethod] + fn set_result(zelf: PyRef<Self>, result: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { + if zelf.fut_loop.read().is_none() { + return Err(vm.new_runtime_error("Future object is not initialized.".to_string())); + } + if zelf.fut_state.load() != FutureState::Pending { + return Err(new_invalid_state_error(vm, "invalid state")); + } + *zelf.fut_result.write() = Some(result); + zelf.fut_state.store(FutureState::Finished); + Self::schedule_callbacks(&zelf, vm)?; + Ok(()) + } + + #[pymethod] + fn set_exception( + zelf: PyRef<Self>, + exception: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult<()> { + if zelf.fut_loop.read().is_none() { + return Err(vm.new_runtime_error("Future object is not initialized.".to_string())); + } + if zelf.fut_state.load() != FutureState::Pending { + return Err(new_invalid_state_error(vm, "invalid state")); + } + + // Normalize the exception + let exc = if exception.fast_isinstance(vm.ctx.types.type_type) { + exception.call((), vm)? + } else { + exception + }; + + if !exc.fast_isinstance(vm.ctx.exceptions.base_exception_type) { + return Err(vm.new_type_error(format!( + "exception must be a BaseException, not {}", + exc.class().name() + ))); + } + + // Wrap StopIteration in RuntimeError + let exc = if exc.fast_isinstance(vm.ctx.exceptions.stop_iteration) { + let msg = "StopIteration interacts badly with generators and cannot be raised into a Future"; + let runtime_err = vm.new_runtime_error(msg.to_string()); + // Set cause and context to the original StopIteration + let stop_iter: PyRef<PyBaseException> = exc.downcast().unwrap(); + runtime_err.set___cause__(Some(stop_iter.clone())); + runtime_err.set___context__(Some(stop_iter)); + runtime_err.into() + } else { + exc + }; + + // Save the original traceback for later restoration + if let Ok(exc_ref) = exc.clone().downcast::<PyBaseException>() { + let tb = exc_ref.__traceback__().map(|tb| tb.into()); + *zelf.fut_exception_tb.write() = tb; + } + + *zelf.fut_exception.write() = Some(exc); + zelf.fut_state.store(FutureState::Finished); + zelf.fut_log_tb.store(true, Ordering::Relaxed); + Self::schedule_callbacks(&zelf, vm)?; + Ok(()) + } + + #[pymethod] + fn add_done_callback( + zelf: PyRef<Self>, + args: AddDoneCallbackArgs, + vm: &VirtualMachine, + ) -> PyResult<()> { + if zelf.fut_loop.read().is_none() { + return Err(vm.new_runtime_error("Future object is not initialized.".to_string())); + } + let ctx = match args.context.flatten() { + Some(c) => c, + None => get_copy_context(vm)?, + }; + + if zelf.fut_state.load() != FutureState::Pending { + Self::call_soon_with_context(&zelf, args.func, Some(ctx), vm)?; + } else if zelf.fut_callback0.read().is_none() { + *zelf.fut_callback0.write() = Some(args.func); + *zelf.fut_context0.write() = Some(ctx); + } else { + let tuple = vm.ctx.new_tuple(vec![args.func, ctx]); + let mut callbacks = zelf.fut_callbacks.write(); + if callbacks.is_none() { + *callbacks = Some(vm.ctx.new_list(vec![tuple.into()]).into()); + } else { + let list = callbacks.as_ref().unwrap(); + vm.call_method(list, "append", (tuple,))?; + } + } + Ok(()) + } + + #[pymethod] + fn remove_done_callback(&self, func: PyObjectRef, vm: &VirtualMachine) -> PyResult<usize> { + if self.fut_loop.read().is_none() { + return Err(vm.new_runtime_error("Future object is not initialized.".to_string())); + } + let mut cleared_callback0 = 0usize; + + // Check fut_callback0 first + // Clone to release lock before comparison (which may run Python code) + let cb0 = self.fut_callback0.read().clone(); + if let Some(cb0) = cb0 { + let cmp = vm.identical_or_equal(&cb0, &func)?; + if cmp { + *self.fut_callback0.write() = None; + *self.fut_context0.write() = None; + cleared_callback0 = 1; + } + } + + // Check if fut_callbacks exists + let callbacks = self.fut_callbacks.read().clone(); + let callbacks = match callbacks { + Some(c) => c, + None => return Ok(cleared_callback0), + }; + + let list: PyListRef = callbacks.downcast().unwrap(); + let len = list.borrow_vec().len(); + + if len == 0 { + *self.fut_callbacks.write() = None; + return Ok(cleared_callback0); + } + + // Special case for single callback + if len == 1 { + let item = list.borrow_vec().first().cloned(); + if let Some(item) = item { + let tuple: &PyTuple = item.downcast_ref().unwrap(); + let cb = tuple.first().unwrap().clone(); + let cmp = vm.identical_or_equal(&cb, &func)?; + if cmp { + *self.fut_callbacks.write() = None; + return Ok(1 + cleared_callback0); + } + } + return Ok(cleared_callback0); + } + + // Multiple callbacks - iterate with index, checking validity each time + // to handle evil comparisons + let mut new_callbacks = Vec::with_capacity(len); + let mut i = 0usize; + let mut removed = 0usize; + + loop { + // Re-check fut_callbacks on each iteration (evil code may have cleared it) + let callbacks = self.fut_callbacks.read().clone(); + let callbacks = match callbacks { + Some(c) => c, + None => break, + }; + let list: PyListRef = callbacks.downcast().unwrap(); + let current_len = list.borrow_vec().len(); + if i >= current_len { + break; + } + + // Get item and release lock before comparison + let item = list.borrow_vec().get(i).cloned(); + let item = match item { + Some(item) => item, + None => break, + }; + + let tuple: &PyTuple = item.downcast_ref().unwrap(); + let cb = tuple.first().unwrap().clone(); + let cmp = vm.identical_or_equal(&cb, &func)?; + + if !cmp { + new_callbacks.push(item); + } else { + removed += 1; + } + i += 1; + } + + // Update fut_callbacks with filtered list + if new_callbacks.is_empty() { + *self.fut_callbacks.write() = None; + } else { + *self.fut_callbacks.write() = Some(vm.ctx.new_list(new_callbacks).into()); + } + + Ok(removed + cleared_callback0) + } + + #[pymethod] + fn cancel(zelf: PyRef<Self>, args: CancelArgs, vm: &VirtualMachine) -> PyResult<bool> { + if zelf.fut_loop.read().is_none() { + return Err(vm.new_runtime_error("Future object is not initialized.".to_string())); + } + if zelf.fut_state.load() != FutureState::Pending { + // Clear log_tb even when cancel fails + zelf.fut_log_tb.store(false, Ordering::Relaxed); + return Ok(false); + } + + *zelf.fut_cancel_msg.write() = args.msg.flatten(); + zelf.fut_state.store(FutureState::Cancelled); + Self::schedule_callbacks(&zelf, vm)?; + Ok(true) + } + + #[pymethod] + fn cancelled(&self) -> bool { + self.fut_state.load() == FutureState::Cancelled + } + + #[pymethod] + fn done(&self) -> bool { + self.fut_state.load() != FutureState::Pending + } + + #[pymethod] + fn get_loop(&self, vm: &VirtualMachine) -> PyResult { + self.fut_loop + .read() + .clone() + .ok_or_else(|| vm.new_runtime_error("Future object is not initialized.")) + } + + #[pymethod] + fn _make_cancelled_error(&self, vm: &VirtualMachine) -> PyBaseExceptionRef { + self.make_cancelled_error_impl(vm) + } + + fn make_cancelled_error_impl(&self, vm: &VirtualMachine) -> PyBaseExceptionRef { + if let Some(exc) = self.fut_cancelled_exc.read().clone() + && let Ok(exc) = exc.downcast::<PyBaseException>() + { + return exc; + } + + let msg = self.fut_cancel_msg.read().clone(); + let args = if let Some(m) = msg { vec![m] } else { vec![] }; + + let exc = match get_cancelled_error_type(vm) { + Ok(cancelled_error) => vm.new_exception(cancelled_error, args), + Err(_) => vm.new_runtime_error("cancelled"), + }; + *self.fut_cancelled_exc.write() = Some(exc.clone().into()); + exc + } + + fn schedule_callbacks(zelf: &PyRef<Self>, vm: &VirtualMachine) -> PyResult<()> { + // Collect all callbacks first to avoid holding locks during callback execution + // This prevents deadlock when callbacks access the future's properties + let mut callbacks_to_call: Vec<(PyObjectRef, Option<PyObjectRef>)> = Vec::new(); + + // Take callback0 - release lock before collecting from list + let cb0 = zelf.fut_callback0.write().take(); + let ctx0 = zelf.fut_context0.write().take(); + if let Some(cb) = cb0 { + callbacks_to_call.push((cb, ctx0)); + } + + // Take callbacks list and collect items + let callbacks_list = zelf.fut_callbacks.write().take(); + if let Some(callbacks) = callbacks_list + && let Ok(list) = callbacks.downcast::<PyList>() + { + // Clone the items while holding the list lock, then release + let items: Vec<_> = list.borrow_vec().iter().cloned().collect(); + for item in items { + if let Some(tuple) = item.downcast_ref::<PyTuple>() + && let (Some(cb), Some(ctx)) = (tuple.first(), tuple.get(1)) + { + callbacks_to_call.push((cb.clone(), Some(ctx.clone()))); + } + } + } + + // Now call all callbacks without holding any locks + for (cb, ctx) in callbacks_to_call { + Self::call_soon_with_context(zelf, cb, ctx, vm)?; + } + + Ok(()) + } + + fn call_soon_with_context( + zelf: &PyRef<Self>, + callback: PyObjectRef, + context: Option<PyObjectRef>, + vm: &VirtualMachine, + ) -> PyResult<()> { + let loop_obj = zelf.fut_loop.read().clone(); + if let Some(loop_obj) = loop_obj { + // call_soon(callback, *args, context=context) + // callback receives the future as its argument + let future_arg: PyObjectRef = zelf.clone().into(); + let args = if let Some(ctx) = context { + FuncArgs::new( + vec![callback, future_arg], + KwArgs::new([("context".to_owned(), ctx)].into_iter().collect()), + ) + } else { + FuncArgs::new(vec![callback, future_arg], KwArgs::default()) + }; + vm.call_method(&loop_obj, "call_soon", args)?; + } + Ok(()) + } + + // Properties + #[pygetset] + fn _state(&self) -> &'static str { + self.fut_state.load().as_str() + } + + #[pygetset] + fn _asyncio_future_blocking(&self) -> bool { + self.fut_blocking.load(Ordering::Relaxed) + } + + #[pygetset(setter)] + fn set__asyncio_future_blocking( + &self, + value: PySetterValue<bool>, + vm: &VirtualMachine, + ) -> PyResult<()> { + match value { + PySetterValue::Assign(v) => { + self.fut_blocking.store(v, Ordering::Relaxed); + Ok(()) + } + PySetterValue::Delete => { + Err(vm.new_attribute_error("cannot delete attribute".to_string())) + } + } + } + + #[pygetset] + fn _loop(&self, vm: &VirtualMachine) -> PyObjectRef { + self.fut_loop + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()) + } + + #[pygetset] + fn _callbacks(&self, vm: &VirtualMachine) -> PyResult<PyObjectRef> { + let mut result = Vec::new(); + + if let Some(cb0) = self.fut_callback0.read().clone() { + let ctx0 = self + .fut_context0 + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()); + result.push(vm.ctx.new_tuple(vec![cb0, ctx0]).into()); + } + + if let Some(callbacks) = self.fut_callbacks.read().clone() { + let list: PyListRef = callbacks.downcast().unwrap(); + for item in list.borrow_vec().iter() { + result.push(item.clone()); + } + } + + // Return None if no callbacks + if result.is_empty() { + Ok(vm.ctx.none()) + } else { + Ok(vm.ctx.new_list(result).into()) + } + } + + #[pygetset] + fn _result(&self, vm: &VirtualMachine) -> PyObjectRef { + self.fut_result + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()) + } + + #[pygetset] + fn _exception(&self, vm: &VirtualMachine) -> PyObjectRef { + self.fut_exception + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()) + } + + #[pygetset] + fn _log_traceback(&self) -> bool { + self.fut_log_tb.load(Ordering::Relaxed) + } + + #[pygetset(setter)] + fn set__log_traceback( + &self, + value: PySetterValue<bool>, + vm: &VirtualMachine, + ) -> PyResult<()> { + match value { + PySetterValue::Assign(v) => { + if v { + return Err(vm.new_value_error( + "_log_traceback can only be set to False".to_string(), + )); + } + self.fut_log_tb.store(false, Ordering::Relaxed); + Ok(()) + } + PySetterValue::Delete => { + Err(vm.new_attribute_error("cannot delete attribute".to_string())) + } + } + } + + #[pygetset] + fn _source_traceback(&self, vm: &VirtualMachine) -> PyObjectRef { + self.fut_source_tb + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()) + } + + #[pygetset] + fn _cancel_message(&self, vm: &VirtualMachine) -> PyObjectRef { + self.fut_cancel_msg + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()) + } + + #[pygetset(setter)] + fn set__cancel_message(&self, value: PySetterValue) { + match value { + PySetterValue::Assign(v) => *self.fut_cancel_msg.write() = Some(v), + PySetterValue::Delete => *self.fut_cancel_msg.write() = None, + } + } + + #[pygetset] + fn _asyncio_awaited_by(&self, vm: &VirtualMachine) -> PyResult<PyObjectRef> { + let awaited_by = self.fut_awaited_by.read().clone(); + match awaited_by { + None => Ok(vm.ctx.none()), + Some(obj) => { + if self.fut_awaited_by_is_set.load(Ordering::Relaxed) { + // Already a Set + Ok(obj) + } else { + // Single object - create a Set for the return value + let new_set = PySet::default().into_ref(&vm.ctx); + new_set.add(obj, vm)?; + Ok(new_set.into()) + } + } + } + } + + /// Add waiter to fut_awaited_by with single-object optimization + fn awaited_by_add(&self, waiter: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { + let mut awaited_by = self.fut_awaited_by.write(); + if awaited_by.is_none() { + // First waiter - store directly + *awaited_by = Some(waiter); + return Ok(()); + } + + if self.fut_awaited_by_is_set.load(Ordering::Relaxed) { + // Already a Set - add to it + let set = awaited_by.as_ref().unwrap(); + vm.call_method(set, "add", (waiter,))?; + } else { + // Single object - convert to Set + let existing = awaited_by.take().unwrap(); + let new_set = PySet::default().into_ref(&vm.ctx); + new_set.add(existing, vm)?; + new_set.add(waiter, vm)?; + *awaited_by = Some(new_set.into()); + self.fut_awaited_by_is_set.store(true, Ordering::Relaxed); + } + Ok(()) + } + + /// Discard waiter from fut_awaited_by with single-object optimization + fn awaited_by_discard(&self, waiter: &PyObject, vm: &VirtualMachine) -> PyResult<()> { + let mut awaited_by = self.fut_awaited_by.write(); + if awaited_by.is_none() { + return Ok(()); + } + + let obj = awaited_by.as_ref().unwrap(); + if !self.fut_awaited_by_is_set.load(Ordering::Relaxed) { + // Single object - check if it matches + if obj.is(waiter) { + *awaited_by = None; + } + } else { + // It's a Set - use discard + vm.call_method(obj, "discard", (waiter.to_owned(),))?; + } + Ok(()) + } + + #[pymethod] + fn __iter__(zelf: PyRef<Self>, vm: &VirtualMachine) -> PyResult<PyFutureIter> { + Self::__await__(zelf, vm) + } + + #[pymethod] + fn __await__(zelf: PyRef<Self>, _vm: &VirtualMachine) -> PyResult<PyFutureIter> { + Ok(PyFutureIter { + future: PyRwLock::new(Some(zelf.into())), + }) + } + + #[pyclassmethod] + fn __class_getitem__( + cls: PyTypeRef, + args: PyObjectRef, + vm: &VirtualMachine, + ) -> PyGenericAlias { + PyGenericAlias::from_args(cls, args, vm) + } + } + + impl Destructor for PyFuture { + fn del(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<()> { + // Check if we should log the traceback + // Don't log if log_tb is false or if the future was cancelled + if !zelf.fut_log_tb.load(Ordering::Relaxed) { + return Ok(()); + } + + if zelf.fut_state.load() == FutureState::Cancelled { + return Ok(()); + } + + let exc = zelf.fut_exception.read().clone(); + let exc = match exc { + Some(e) => e, + None => return Ok(()), + }; + + let loop_obj = zelf.fut_loop.read().clone(); + let loop_obj = match loop_obj { + Some(l) => l, + None => return Ok(()), + }; + + // Create context dict for call_exception_handler + let context = PyDict::default().into_ref(&vm.ctx); + let class_name = zelf.class().name().to_string(); + let message = format!("{} exception was never retrieved", class_name); + context.set_item( + vm.ctx.intern_str("message"), + vm.ctx.new_str(message).into(), + vm, + )?; + context.set_item(vm.ctx.intern_str("exception"), exc, vm)?; + context.set_item(vm.ctx.intern_str("future"), zelf.to_owned().into(), vm)?; + + if let Some(tb) = zelf.fut_source_tb.read().clone() { + context.set_item(vm.ctx.intern_str("source_traceback"), tb, vm)?; + } + + // Call loop.call_exception_handler(context) + let _ = vm.call_method(&loop_obj, "call_exception_handler", (context,)); + Ok(()) + } + } + + impl Representable for PyFuture { + fn repr_str(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<String> { + let class_name = zelf.class().name().to_string(); + if let Some(_guard) = ReprGuard::enter(vm, zelf.as_object()) { + let info = get_future_repr_info(zelf.as_object(), vm)?; + Ok(format!("<{} {}>", class_name, info)) + } else { + Ok(format!("<{} ...>", class_name)) + } + } + } + + impl Iterable for PyFuture { + fn iter(zelf: PyRef<Self>, _vm: &VirtualMachine) -> PyResult { + Ok(PyFutureIter { + future: PyRwLock::new(Some(zelf.into())), + } + .into_pyobject(_vm)) + } + } + + fn get_future_repr_info(future: &PyObject, vm: &VirtualMachine) -> PyResult<String> { + // Try to use asyncio.base_futures._future_repr_info + // Import from sys.modules if available, otherwise try regular import + let sys_modules = vm.sys_module.get_attr("modules", vm)?; + let module = + if let Ok(m) = sys_modules.get_item(&*vm.ctx.new_str("asyncio.base_futures"), vm) { + m + } else { + // vm.import returns the top-level module, get base_futures submodule + match vm + .import("asyncio.base_futures", 0) + .and_then(|asyncio| asyncio.get_attr(vm.ctx.intern_str("base_futures"), vm)) + { + Ok(m) => m, + Err(_) => return get_future_repr_info_fallback(future, vm), + } + }; + + let func = match vm.get_attribute_opt(module, vm.ctx.intern_str("_future_repr_info")) { + Ok(Some(f)) => f, + _ => return get_future_repr_info_fallback(future, vm), + }; + + let info = match func.call((future.to_owned(),), vm) { + Ok(i) => i, + Err(_) => return get_future_repr_info_fallback(future, vm), + }; + + let list: PyListRef = match info.downcast() { + Ok(l) => l, + Err(_) => return get_future_repr_info_fallback(future, vm), + }; + + let parts: Vec<String> = list + .borrow_vec() + .iter() + .filter_map(|x: &PyObjectRef| x.str(vm).ok().map(|s| s.as_str().to_string())) + .collect(); + Ok(parts.join(" ")) + } + + fn get_future_repr_info_fallback(future: &PyObject, vm: &VirtualMachine) -> PyResult<String> { + // Fallback: build repr from properties directly + if let Ok(Some(state)) = + vm.get_attribute_opt(future.to_owned(), vm.ctx.intern_str("_state")) + { + let state_str = state + .str(vm) + .map(|s| s.as_str().to_lowercase()) + .unwrap_or_else(|_| "unknown".to_string()); + return Ok(state_str); + } + Ok("state=unknown".to_string()) + } + + fn get_task_repr_info(task: &PyObject, vm: &VirtualMachine) -> PyResult<String> { + // vm.import returns the top-level module, get base_tasks submodule + match vm + .import("asyncio.base_tasks", 0) + .and_then(|asyncio| asyncio.get_attr(vm.ctx.intern_str("base_tasks"), vm)) + { + Ok(base_tasks) => { + match vm.get_attribute_opt(base_tasks, vm.ctx.intern_str("_task_repr_info")) { + Ok(Some(func)) => { + let info: PyObjectRef = func.call((task.to_owned(),), vm)?; + let list: PyListRef = info.downcast().map_err(|_| { + vm.new_type_error("_task_repr_info should return a list") + })?; + let parts: Vec<String> = list + .borrow_vec() + .iter() + .map(|x: &PyObjectRef| x.str(vm).map(|s| s.as_str().to_string())) + .collect::<PyResult<Vec<_>>>()?; + Ok(parts.join(" ")) + } + _ => get_future_repr_info(task, vm), + } + } + Err(_) => get_future_repr_info(task, vm), + } + } + + #[pyattr] + #[pyclass(name = "FutureIter", module = "_asyncio", traverse)] + #[derive(Debug, PyPayload)] + struct PyFutureIter { + future: PyRwLock<Option<PyObjectRef>>, + } + + #[pyclass(with(IterNext, Iterable))] + impl PyFutureIter { + #[pymethod] + fn send(&self, _value: PyObjectRef, vm: &VirtualMachine) -> PyResult { + let future = self.future.read().clone(); + let future = match future { + Some(f) => f, + None => return Err(vm.new_stop_iteration(None)), + }; + + // Try to get blocking flag (check Task first since it inherits from Future) + let blocking = if let Some(task) = future.downcast_ref::<PyTask>() { + task.base.fut_blocking.load(Ordering::Relaxed) + } else if let Some(fut) = future.downcast_ref::<PyFuture>() { + fut.fut_blocking.load(Ordering::Relaxed) + } else { + // For non-native futures, check the attribute + vm.get_attribute_opt( + future.clone(), + vm.ctx.intern_str("_asyncio_future_blocking"), + )? + .map(|v| v.try_to_bool(vm)) + .transpose()? + .unwrap_or(false) + }; + + // Check if future is done + let done = vm.call_method(&future, "done", ())?; + if done.try_to_bool(vm)? { + *self.future.write() = None; + let result = vm.call_method(&future, "result", ())?; + return Err(vm.new_stop_iteration(Some(result))); + } + + // If still pending and blocking is already set, raise RuntimeError + // This means await wasn't used with future + if blocking { + return Err(vm.new_runtime_error("await wasn't used with future")); + } + + // First call: set blocking flag and yield the future (check Task first) + if let Some(task) = future.downcast_ref::<PyTask>() { + task.base.fut_blocking.store(true, Ordering::Relaxed); + } else if let Some(fut) = future.downcast_ref::<PyFuture>() { + fut.fut_blocking.store(true, Ordering::Relaxed); + } else { + future.set_attr( + vm.ctx.intern_str("_asyncio_future_blocking"), + vm.ctx.true_value.clone(), + vm, + )?; + } + Ok(future) + } + + #[pymethod] + fn throw( + &self, + exc_type: PyObjectRef, + exc_val: OptionalArg, + exc_tb: OptionalArg, + vm: &VirtualMachine, + ) -> PyResult { + // Warn about deprecated (type, val, tb) signature + if exc_val.is_present() || exc_tb.is_present() { + warn::warn( + vm.ctx.new_str( + "the (type, val, tb) signature of throw() is deprecated, \ + use throw(val) instead", + ), + Some(vm.ctx.exceptions.deprecation_warning.to_owned()), + 1, + None, + vm, + )?; + } + + *self.future.write() = None; + + // Validate tb if present + if let OptionalArg::Present(ref tb) = exc_tb + && !vm.is_none(tb) + && !tb.fast_isinstance(vm.ctx.types.traceback_type) + { + return Err(vm.new_type_error(format!( + "throw() third argument must be a traceback object, not '{}'", + tb.class().name() + ))); + } + + let exc = if exc_type.fast_isinstance(vm.ctx.types.type_type) { + // exc_type is a class + let exc_class: PyTypeRef = exc_type.clone().downcast().unwrap(); + // Must be a subclass of BaseException + if !exc_class.fast_issubclass(vm.ctx.exceptions.base_exception_type) { + return Err(vm.new_type_error( + "exceptions must be classes or instances deriving from BaseException, not type".to_string() + )); + } + + let val = exc_val.unwrap_or_none(vm); + if vm.is_none(&val) { + exc_type.call((), vm)? + } else if val.fast_isinstance(&exc_class) { + val + } else { + exc_type.call((val,), vm)? + } + } else if exc_type.fast_isinstance(vm.ctx.exceptions.base_exception_type) { + // exc_type is an exception instance + if let OptionalArg::Present(ref val) = exc_val + && !vm.is_none(val) + { + return Err(vm.new_type_error( + "instance exception may not have a separate value".to_string(), + )); + } + exc_type + } else { + // exc_type is neither a class nor an exception instance + return Err(vm.new_type_error(format!( + "exceptions must be classes or instances deriving from BaseException, not {}", + exc_type.class().name() + ))); + }; + + if let OptionalArg::Present(tb) = exc_tb + && !vm.is_none(&tb) + { + exc.set_attr(vm.ctx.intern_str("__traceback__"), tb, vm)?; + } + + Err(exc.downcast().unwrap()) + } + + #[pymethod] + fn close(&self) { + *self.future.write() = None; + } + } + + impl SelfIter for PyFutureIter {} + impl IterNext for PyFutureIter { + fn next(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<PyIterReturn> { + PyIterReturn::from_pyresult(zelf.send(vm.ctx.none(), vm), vm) + } + } + + #[pyattr] + #[pyclass(name = "Task", module = "_asyncio", base = PyFuture, traverse)] + #[derive(Debug)] + #[repr(C)] + struct PyTask { + // Base class (must be first field for inheritance) + base: PyFuture, + // Task-specific fields + task_coro: PyRwLock<Option<PyObjectRef>>, + task_fut_waiter: PyRwLock<Option<PyObjectRef>>, + task_name: PyRwLock<Option<PyObjectRef>>, + task_context: PyRwLock<Option<PyObjectRef>>, + #[pytraverse(skip)] + task_must_cancel: AtomicBool, + #[pytraverse(skip)] + task_num_cancels_requested: AtomicI32, + #[pytraverse(skip)] + task_log_destroy_pending: AtomicBool, + } + + #[derive(FromArgs)] + struct TaskInitArgs { + #[pyarg(positional)] + coro: PyObjectRef, + #[pyarg(named, name = "loop", optional)] + loop_: OptionalOption<PyObjectRef>, + #[pyarg(named, optional)] + name: OptionalOption<PyObjectRef>, + #[pyarg(named, optional)] + context: OptionalOption<PyObjectRef>, + #[pyarg(named, optional)] + eager_start: OptionalOption<bool>, + } + + static TASK_NAME_COUNTER: AtomicU64 = AtomicU64::new(0); + + impl Constructor for PyTask { + type Args = FuncArgs; + + fn py_new(_cls: &Py<PyType>, _args: Self::Args, _vm: &VirtualMachine) -> PyResult<Self> { + Ok(Self { + base: PyFuture::new_empty(), + task_coro: PyRwLock::new(None), + task_fut_waiter: PyRwLock::new(None), + task_name: PyRwLock::new(None), + task_context: PyRwLock::new(None), + task_must_cancel: AtomicBool::new(false), + task_num_cancels_requested: AtomicI32::new(0), + task_log_destroy_pending: AtomicBool::new(true), + }) + } + } + + impl Initializer for PyTask { + type Args = TaskInitArgs; + + fn init(zelf: PyRef<Self>, args: Self::Args, vm: &VirtualMachine) -> PyResult<()> { + PyTask::py_init(&zelf, args, vm) + } + } + + #[pyclass( + flags(BASETYPE, HAS_DICT), + with(Constructor, Initializer, Destructor, Representable, Iterable) + )] + impl PyTask { + fn py_init(zelf: &PyRef<Self>, args: TaskInitArgs, vm: &VirtualMachine) -> PyResult<()> { + // Validate coroutine + if !is_coroutine(args.coro.clone(), vm)? { + return Err(vm.new_type_error(format!( + "a coroutine was expected, got {}", + args.coro.repr(vm)? + ))); + } + + // Get the event loop + let loop_obj = match args.loop_.flatten() { + Some(l) => l, + None => get_running_loop(vm) + .map_err(|_| vm.new_runtime_error("no current event loop"))?, + }; + *zelf.base.fut_loop.write() = Some(loop_obj.clone()); + + // Check if loop has get_debug method and capture source traceback if enabled + if let Ok(Some(get_debug)) = + vm.get_attribute_opt(loop_obj.clone(), vm.ctx.intern_str("get_debug")) + && let Ok(debug) = get_debug.call((), vm) + && debug.try_to_bool(vm).unwrap_or(false) + { + // Get source traceback + if let Ok(tb_module) = vm.import("traceback", 0) + && let Ok(Some(extract_stack)) = + vm.get_attribute_opt(tb_module, vm.ctx.intern_str("extract_stack")) + && let Ok(tb) = extract_stack.call((), vm) + { + *zelf.base.fut_source_tb.write() = Some(tb); + } + } + + // Get or create context + let context = match args.context.flatten() { + Some(c) => c, + None => get_copy_context(vm)?, + }; + *zelf.task_context.write() = Some(context); + + // Set coroutine + *zelf.task_coro.write() = Some(args.coro); + + // Set task name + let name = match args.name.flatten() { + Some(n) => { + if !n.fast_isinstance(vm.ctx.types.str_type) { + n.str(vm)?.into() + } else { + n + } + } + None => { + let counter = TASK_NAME_COUNTER.fetch_add(1, Ordering::SeqCst); + vm.ctx.new_str(format!("Task-{}", counter + 1)).into() + } + }; + *zelf.task_name.write() = Some(name); + + let eager_start = args.eager_start.flatten().unwrap_or(false); + + // Check if we should do eager start: only if the loop is running + let do_eager_start = if eager_start { + let is_running = vm.call_method(&loop_obj, "is_running", ())?; + is_running.is_true(vm)? + } else { + false + }; + + if do_eager_start { + // Eager start: run first step synchronously (loop is already running) + task_eager_start(zelf, vm)?; + } else { + // Non-eager or loop not running: schedule the first step + _register_task(zelf.clone().into(), vm)?; + let task_obj: PyObjectRef = zelf.clone().into(); + let step_wrapper = TaskStepMethWrapper::new(task_obj).into_ref(&vm.ctx); + vm.call_method(&loop_obj, "call_soon", (step_wrapper,))?; + } + + Ok(()) + } + + // Future methods delegation + #[pymethod] + fn result(&self, vm: &VirtualMachine) -> PyResult { + match self.base.fut_state.load() { + FutureState::Pending => Err(new_invalid_state_error(vm, "Result is not ready.")), + FutureState::Cancelled => Err(self.make_cancelled_error_impl(vm)), + FutureState::Finished => { + self.base.fut_log_tb.store(false, Ordering::Relaxed); + if let Some(exc) = self.base.fut_exception.read().clone() { + let exc: PyBaseExceptionRef = exc.downcast().unwrap(); + // Restore the original traceback to prevent traceback accumulation + if let Some(tb) = self.base.fut_exception_tb.read().clone() { + let _ = exc.set___traceback__(tb, vm); + } + Err(exc) + } else { + Ok(self + .base + .fut_result + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none())) + } + } + } + } + + #[pymethod] + fn exception(&self, vm: &VirtualMachine) -> PyResult { + match self.base.fut_state.load() { + FutureState::Pending => Err(new_invalid_state_error(vm, "Exception is not set.")), + FutureState::Cancelled => Err(self.make_cancelled_error_impl(vm)), + FutureState::Finished => { + self.base.fut_log_tb.store(false, Ordering::Relaxed); + Ok(self + .base + .fut_exception + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none())) + } + } + } + + #[pymethod] + fn set_result( + _zelf: PyObjectRef, + _result: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult<()> { + Err(vm.new_runtime_error("Task does not support set_result operation")) + } + + #[pymethod] + fn set_exception(&self, _exception: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { + Err(vm.new_runtime_error("Task does not support set_exception operation")) + } + + fn make_cancelled_error_impl(&self, vm: &VirtualMachine) -> PyBaseExceptionRef { + if let Some(exc) = self.base.fut_cancelled_exc.read().clone() + && let Ok(exc) = exc.downcast::<PyBaseException>() + { + return exc; + } + + let msg = self.base.fut_cancel_msg.read().clone(); + let args = if let Some(m) = msg { vec![m] } else { vec![] }; + + let exc = match get_cancelled_error_type(vm) { + Ok(cancelled_error) => vm.new_exception(cancelled_error, args), + Err(_) => vm.new_runtime_error("cancelled"), + }; + *self.base.fut_cancelled_exc.write() = Some(exc.clone().into()); + exc + } + + #[pymethod] + fn add_done_callback( + zelf: PyRef<Self>, + args: AddDoneCallbackArgs, + vm: &VirtualMachine, + ) -> PyResult<()> { + if zelf.base.fut_loop.read().is_none() { + return Err(vm.new_runtime_error("Future object is not initialized.".to_string())); + } + let ctx = match args.context.flatten() { + Some(c) => c, + None => get_copy_context(vm)?, + }; + + if zelf.base.fut_state.load() != FutureState::Pending { + Self::call_soon_with_context(&zelf, args.func, Some(ctx), vm)?; + } else if zelf.base.fut_callback0.read().is_none() { + *zelf.base.fut_callback0.write() = Some(args.func); + *zelf.base.fut_context0.write() = Some(ctx); + } else { + let tuple = vm.ctx.new_tuple(vec![args.func, ctx]); + let mut callbacks = zelf.base.fut_callbacks.write(); + if callbacks.is_none() { + *callbacks = Some(vm.ctx.new_list(vec![tuple.into()]).into()); + } else { + let list = callbacks.as_ref().unwrap(); + vm.call_method(list, "append", (tuple,))?; + } + } + Ok(()) + } + + #[pymethod] + fn remove_done_callback(&self, func: PyObjectRef, vm: &VirtualMachine) -> PyResult<usize> { + if self.base.fut_loop.read().is_none() { + return Err(vm.new_runtime_error("Future object is not initialized.".to_string())); + } + let mut cleared_callback0 = 0usize; + + // Check fut_callback0 first + // Clone to release lock before comparison (which may run Python code) + let cb0 = self.base.fut_callback0.read().clone(); + if let Some(cb0) = cb0 { + let cmp = vm.identical_or_equal(&cb0, &func)?; + if cmp { + *self.base.fut_callback0.write() = None; + *self.base.fut_context0.write() = None; + cleared_callback0 = 1; + } + } + + // Check if fut_callbacks exists + let callbacks = self.base.fut_callbacks.read().clone(); + let callbacks = match callbacks { + Some(c) => c, + None => return Ok(cleared_callback0), + }; + + let list: PyListRef = callbacks.downcast().unwrap(); + let len = list.borrow_vec().len(); + + if len == 0 { + *self.base.fut_callbacks.write() = None; + return Ok(cleared_callback0); + } + + // Special case for single callback + if len == 1 { + let item = list.borrow_vec().first().cloned(); + if let Some(item) = item { + let tuple: &PyTuple = item.downcast_ref().unwrap(); + let cb = tuple.first().unwrap().clone(); + let cmp = vm.identical_or_equal(&cb, &func)?; + if cmp { + *self.base.fut_callbacks.write() = None; + return Ok(1 + cleared_callback0); + } + } + return Ok(cleared_callback0); + } + + // Multiple callbacks - iterate with index, checking validity each time + // to handle evil comparisons + let mut new_callbacks = Vec::with_capacity(len); + let mut i = 0usize; + let mut removed = 0usize; + + loop { + // Re-check fut_callbacks on each iteration (evil code may have cleared it) + let callbacks = self.base.fut_callbacks.read().clone(); + let callbacks = match callbacks { + Some(c) => c, + None => break, + }; + let list: PyListRef = callbacks.downcast().unwrap(); + let current_len = list.borrow_vec().len(); + if i >= current_len { + break; + } + + // Get item and release lock before comparison + let item = list.borrow_vec().get(i).cloned(); + let item = match item { + Some(item) => item, + None => break, + }; + + let tuple: &PyTuple = item.downcast_ref().unwrap(); + let cb = tuple.first().unwrap().clone(); + let cmp = vm.identical_or_equal(&cb, &func)?; + + if !cmp { + new_callbacks.push(item); + } else { + removed += 1; + } + i += 1; + } + + // Update fut_callbacks with filtered list + if new_callbacks.is_empty() { + *self.base.fut_callbacks.write() = None; + } else { + *self.base.fut_callbacks.write() = Some(vm.ctx.new_list(new_callbacks).into()); + } + + Ok(removed + cleared_callback0) + } + + fn schedule_callbacks(zelf: &PyRef<Self>, vm: &VirtualMachine) -> PyResult<()> { + // Collect all callbacks first to avoid holding locks during callback execution + // This prevents deadlock when callbacks access the future's properties + let mut callbacks_to_call: Vec<(PyObjectRef, Option<PyObjectRef>)> = Vec::new(); + + // Take callback0 - release lock before collecting from list + let cb0 = zelf.base.fut_callback0.write().take(); + let ctx0 = zelf.base.fut_context0.write().take(); + if let Some(cb) = cb0 { + callbacks_to_call.push((cb, ctx0)); + } + + // Take callbacks list and collect items + let callbacks_list = zelf.base.fut_callbacks.write().take(); + if let Some(callbacks) = callbacks_list + && let Ok(list) = callbacks.downcast::<PyList>() + { + // Clone the items while holding the list lock, then release + let items: Vec<_> = list.borrow_vec().iter().cloned().collect(); + for item in items { + if let Some(tuple) = item.downcast_ref::<PyTuple>() + && let (Some(cb), Some(ctx)) = (tuple.first(), tuple.get(1)) + { + callbacks_to_call.push((cb.clone(), Some(ctx.clone()))); + } + } + } + + // Now call all callbacks without holding any locks + for (cb, ctx) in callbacks_to_call { + Self::call_soon_with_context(zelf, cb, ctx, vm)?; + } + + Ok(()) + } + + fn call_soon_with_context( + zelf: &PyRef<Self>, + callback: PyObjectRef, + context: Option<PyObjectRef>, + vm: &VirtualMachine, + ) -> PyResult<()> { + let loop_obj = zelf.base.fut_loop.read().clone(); + if let Some(loop_obj) = loop_obj { + // call_soon(callback, *args, context=context) + // callback receives the task as its argument + let task_arg: PyObjectRef = zelf.clone().into(); + let args = if let Some(ctx) = context { + FuncArgs::new( + vec![callback, task_arg], + KwArgs::new([("context".to_owned(), ctx)].into_iter().collect()), + ) + } else { + FuncArgs::new(vec![callback, task_arg], KwArgs::default()) + }; + vm.call_method(&loop_obj, "call_soon", args)?; + } + Ok(()) + } + + #[pymethod] + fn cancel(&self, args: CancelArgs, vm: &VirtualMachine) -> PyResult<bool> { + if self.base.fut_state.load() != FutureState::Pending { + // Clear log_tb even when cancel fails (task is already done) + self.base.fut_log_tb.store(false, Ordering::Relaxed); + return Ok(false); + } + + self.task_num_cancels_requested + .fetch_add(1, Ordering::SeqCst); + + let msg_value = args.msg.flatten(); + + if let Some(fut_waiter) = self.task_fut_waiter.read().clone() { + // Call cancel with msg=msg keyword argument + let cancel_args = if let Some(ref m) = msg_value { + FuncArgs::new( + vec![], + KwArgs::new([("msg".to_owned(), m.clone())].into_iter().collect()), + ) + } else { + FuncArgs::new(vec![], KwArgs::default()) + }; + let cancel_result = vm.call_method(&fut_waiter, "cancel", cancel_args)?; + if cancel_result.try_to_bool(vm)? { + return Ok(true); + } + } + + self.task_must_cancel.store(true, Ordering::Relaxed); + *self.base.fut_cancel_msg.write() = msg_value; + Ok(true) + } + + #[pymethod] + fn cancelled(&self) -> bool { + self.base.fut_state.load() == FutureState::Cancelled + } + + #[pymethod] + fn done(&self) -> bool { + self.base.fut_state.load() != FutureState::Pending + } + + #[pymethod] + fn cancelling(&self) -> i32 { + self.task_num_cancels_requested.load(Ordering::SeqCst) + } + + #[pymethod] + fn uncancel(&self) -> i32 { + let prev = self + .task_num_cancels_requested + .fetch_sub(1, Ordering::SeqCst); + if prev <= 0 { + self.task_num_cancels_requested.store(0, Ordering::SeqCst); + 0 + } else { + let new_val = prev - 1; + // When cancelling count reaches 0, reset _must_cancel + if new_val == 0 { + self.task_must_cancel.store(false, Ordering::SeqCst); + } + new_val + } + } + + #[pymethod] + fn get_coro(&self, vm: &VirtualMachine) -> PyObjectRef { + self.task_coro + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()) + } + + #[pymethod] + fn get_context(&self, vm: &VirtualMachine) -> PyObjectRef { + self.task_context + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()) + } + + #[pymethod] + fn get_name(&self, vm: &VirtualMachine) -> PyObjectRef { + self.task_name + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()) + } + + #[pymethod] + fn set_name(&self, name: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { + let name = if !name.fast_isinstance(vm.ctx.types.str_type) { + name.str(vm)?.into() + } else { + name + }; + *self.task_name.write() = Some(name); + Ok(()) + } + + #[pymethod] + fn get_loop(&self, vm: &VirtualMachine) -> PyResult { + self.base + .fut_loop + .read() + .clone() + .ok_or_else(|| vm.new_runtime_error("Task object is not initialized.")) + } + + #[pymethod] + fn get_stack(zelf: PyRef<Self>, args: GetStackArgs, vm: &VirtualMachine) -> PyResult { + let limit = args.limit.flatten().unwrap_or_else(|| vm.ctx.none()); + // vm.import returns the top-level module, get base_tasks submodule + let asyncio = vm.import("asyncio.base_tasks", 0)?; + let base_tasks = asyncio.get_attr(vm.ctx.intern_str("base_tasks"), vm)?; + let get_stack_func = base_tasks.get_attr(vm.ctx.intern_str("_task_get_stack"), vm)?; + get_stack_func.call((zelf, limit), vm) + } + + #[pymethod] + fn print_stack( + zelf: PyRef<Self>, + args: PrintStackArgs, + vm: &VirtualMachine, + ) -> PyResult<()> { + let limit = args.limit.flatten().unwrap_or_else(|| vm.ctx.none()); + let file = args.file.flatten().unwrap_or_else(|| vm.ctx.none()); + // vm.import returns the top-level module, get base_tasks submodule + let asyncio = vm.import("asyncio.base_tasks", 0)?; + let base_tasks = asyncio.get_attr(vm.ctx.intern_str("base_tasks"), vm)?; + let print_stack_func = + base_tasks.get_attr(vm.ctx.intern_str("_task_print_stack"), vm)?; + print_stack_func.call((zelf, limit, file), vm)?; + Ok(()) + } + + #[pymethod] + fn _make_cancelled_error(&self, vm: &VirtualMachine) -> PyBaseExceptionRef { + self.make_cancelled_error_impl(vm) + } + + // Properties + #[pygetset] + fn _state(&self) -> &'static str { + self.base.fut_state.load().as_str() + } + + #[pygetset] + fn _asyncio_future_blocking(&self) -> bool { + self.base.fut_blocking.load(Ordering::Relaxed) + } + + #[pygetset(setter)] + fn set__asyncio_future_blocking( + &self, + value: PySetterValue<bool>, + vm: &VirtualMachine, + ) -> PyResult<()> { + match value { + PySetterValue::Assign(v) => { + self.base.fut_blocking.store(v, Ordering::Relaxed); + Ok(()) + } + PySetterValue::Delete => { + Err(vm.new_attribute_error("cannot delete attribute".to_string())) + } + } + } + + #[pygetset] + fn _loop(&self, vm: &VirtualMachine) -> PyObjectRef { + self.base + .fut_loop + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()) + } + + #[pygetset] + fn _log_destroy_pending(&self) -> bool { + self.task_log_destroy_pending.load(Ordering::Relaxed) + } + + #[pygetset(setter)] + fn set__log_destroy_pending( + &self, + value: PySetterValue<bool>, + vm: &VirtualMachine, + ) -> PyResult<()> { + match value { + PySetterValue::Assign(v) => { + self.task_log_destroy_pending.store(v, Ordering::Relaxed); + Ok(()) + } + PySetterValue::Delete => { + Err(vm.new_attribute_error("can't delete _log_destroy_pending".to_owned())) + } + } + } + + #[pygetset] + fn _log_traceback(&self) -> bool { + self.base.fut_log_tb.load(Ordering::Relaxed) + } + + #[pygetset(setter)] + fn set__log_traceback( + &self, + value: PySetterValue<bool>, + vm: &VirtualMachine, + ) -> PyResult<()> { + match value { + PySetterValue::Assign(v) => { + if v { + return Err(vm.new_value_error( + "_log_traceback can only be set to False".to_string(), + )); + } + self.base.fut_log_tb.store(false, Ordering::Relaxed); + Ok(()) + } + PySetterValue::Delete => { + Err(vm.new_attribute_error("cannot delete attribute".to_string())) + } + } + } + + #[pygetset] + fn _must_cancel(&self) -> bool { + self.task_must_cancel.load(Ordering::Relaxed) + } + + #[pygetset] + fn _coro(&self, vm: &VirtualMachine) -> PyObjectRef { + self.task_coro + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()) + } + + #[pygetset] + fn _fut_waiter(&self, vm: &VirtualMachine) -> PyObjectRef { + self.task_fut_waiter + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()) + } + + #[pygetset] + fn _source_traceback(&self, vm: &VirtualMachine) -> PyObjectRef { + self.base + .fut_source_tb + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()) + } + + #[pygetset] + fn _result(&self, vm: &VirtualMachine) -> PyObjectRef { + self.base + .fut_result + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()) + } + + #[pygetset] + fn _exception(&self, vm: &VirtualMachine) -> PyObjectRef { + self.base + .fut_exception + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()) + } + + #[pygetset] + fn _cancel_message(&self, vm: &VirtualMachine) -> PyObjectRef { + self.base + .fut_cancel_msg + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()) + } + + #[pygetset(setter)] + fn set__cancel_message(&self, value: PySetterValue) { + match value { + PySetterValue::Assign(v) => *self.base.fut_cancel_msg.write() = Some(v), + PySetterValue::Delete => *self.base.fut_cancel_msg.write() = None, + } + } + + #[pygetset] + fn _callbacks(&self, vm: &VirtualMachine) -> PyObjectRef { + let mut result: Vec<PyObjectRef> = Vec::new(); + if let Some(cb) = self.base.fut_callback0.read().clone() { + let ctx = self + .base + .fut_context0 + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()); + result.push(vm.ctx.new_tuple(vec![cb, ctx]).into()); + } + if let Some(callbacks) = self.base.fut_callbacks.read().clone() + && let Ok(list) = callbacks.downcast::<PyList>() + { + for item in list.borrow_vec().iter() { + result.push(item.clone()); + } + } + // Return None if no callbacks + if result.is_empty() { + vm.ctx.none() + } else { + vm.ctx.new_list(result).into() + } + } + + #[pymethod] + fn __iter__(zelf: PyRef<Self>, _vm: &VirtualMachine) -> PyResult<PyFutureIter> { + Ok(PyFutureIter { + future: PyRwLock::new(Some(zelf.into())), + }) + } + + #[pymethod] + fn __await__(zelf: PyRef<Self>, _vm: &VirtualMachine) -> PyResult<PyFutureIter> { + Ok(PyFutureIter { + future: PyRwLock::new(Some(zelf.into())), + }) + } + + #[pyclassmethod] + fn __class_getitem__( + cls: PyTypeRef, + args: PyObjectRef, + vm: &VirtualMachine, + ) -> PyGenericAlias { + PyGenericAlias::from_args(cls, args, vm) + } + } + + impl Destructor for PyTask { + fn del(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<()> { + let loop_obj = zelf.base.fut_loop.read().clone(); + + // Check if task is pending and log_destroy_pending is True + if zelf.base.fut_state.load() == FutureState::Pending + && zelf.task_log_destroy_pending.load(Ordering::Relaxed) + { + if let Some(loop_obj) = loop_obj.clone() { + let context = PyDict::default().into_ref(&vm.ctx); + let task_repr = zelf + .as_object() + .repr(vm) + .unwrap_or_else(|_| vm.ctx.new_str("<Task>")); + let message = + format!("Task was destroyed but it is pending!\ntask: {}", task_repr); + context.set_item( + vm.ctx.intern_str("message"), + vm.ctx.new_str(message).into(), + vm, + )?; + context.set_item(vm.ctx.intern_str("task"), zelf.to_owned().into(), vm)?; + + if let Some(tb) = zelf.base.fut_source_tb.read().clone() { + context.set_item(vm.ctx.intern_str("source_traceback"), tb, vm)?; + } + + let _ = vm.call_method(&loop_obj, "call_exception_handler", (context,)); + } + return Ok(()); + } + + // Check if we should log the traceback for exception + if !zelf.base.fut_log_tb.load(Ordering::Relaxed) { + return Ok(()); + } + + let exc = zelf.base.fut_exception.read().clone(); + let exc = match exc { + Some(e) => e, + None => return Ok(()), + }; + + let loop_obj = match loop_obj { + Some(l) => l, + None => return Ok(()), + }; + + // Create context dict for call_exception_handler + let context = PyDict::default().into_ref(&vm.ctx); + let class_name = zelf.class().name().to_string(); + let message = format!("{} exception was never retrieved", class_name); + context.set_item( + vm.ctx.intern_str("message"), + vm.ctx.new_str(message).into(), + vm, + )?; + context.set_item(vm.ctx.intern_str("exception"), exc, vm)?; + context.set_item(vm.ctx.intern_str("future"), zelf.to_owned().into(), vm)?; + + if let Some(tb) = zelf.base.fut_source_tb.read().clone() { + context.set_item(vm.ctx.intern_str("source_traceback"), tb, vm)?; + } + + // Call loop.call_exception_handler(context) + let _ = vm.call_method(&loop_obj, "call_exception_handler", (context,)); + Ok(()) + } + } + + impl Representable for PyTask { + fn repr_str(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<String> { + let class_name = zelf.class().name().to_string(); + + if let Some(_guard) = ReprGuard::enter(vm, zelf.as_object()) { + // Try to use _task_repr_info if available + if let Ok(info) = get_task_repr_info(zelf.as_object(), vm) + && info != "state=unknown" + { + return Ok(format!("<{} {}>", class_name, info)); + } + + // Fallback: build repr from task properties directly + let state = zelf.base.fut_state.load().as_str().to_lowercase(); + let name = zelf + .task_name + .read() + .as_ref() + .and_then(|n| n.str(vm).ok()) + .map(|s| s.as_str().to_string()) + .unwrap_or_else(|| "?".to_string()); + let coro_repr = zelf + .task_coro + .read() + .as_ref() + .and_then(|c| c.repr(vm).ok()) + .map(|s| s.as_str().to_string()) + .unwrap_or_else(|| "?".to_string()); + + Ok(format!( + "<{} {} name='{}' coro={}>", + class_name, state, name, coro_repr + )) + } else { + Ok(format!("<{} ...>", class_name)) + } + } + } + + impl Iterable for PyTask { + fn iter(zelf: PyRef<Self>, _vm: &VirtualMachine) -> PyResult { + Ok(PyFutureIter { + future: PyRwLock::new(Some(zelf.into())), + } + .into_pyobject(_vm)) + } + } + + /// Eager start: run first step synchronously + fn task_eager_start(zelf: &PyRef<PyTask>, vm: &VirtualMachine) -> PyResult<()> { + let loop_obj = zelf.base.fut_loop.read().clone(); + let loop_obj = match loop_obj { + Some(l) => l, + None => return Err(vm.new_runtime_error("Task has no loop")), + }; + + // Register task before running step + let task_obj: PyObjectRef = zelf.clone().into(); + _register_task(task_obj.clone(), vm)?; + + // Register as eager task + _register_eager_task(task_obj.clone(), vm)?; + + // Swap current task - save previous task + let prev_task = _swap_current_task(loop_obj.clone(), task_obj.clone(), vm)?; + + // Get coro and context + let coro = zelf.task_coro.read().clone(); + let context = zelf.task_context.read().clone(); + + // Run the first step with context (using context.run(callable, *args)) + let step_result = if let Some(ctx) = context { + // Call context.run(coro.send, None) + let coro_ref = match coro { + Some(c) => c, + None => { + let _ = _swap_current_task(loop_obj.clone(), prev_task, vm); + _unregister_eager_task(task_obj.clone(), vm)?; + return Ok(()); + } + }; + let send_method = coro_ref.get_attr(vm.ctx.intern_str("send"), vm)?; + vm.call_method(&ctx, "run", (send_method, vm.ctx.none())) + } else { + // Run without context + match coro { + Some(c) => vm.call_method(&c, "send", (vm.ctx.none(),)), + None => { + let _ = _swap_current_task(loop_obj.clone(), prev_task, vm); + _unregister_eager_task(task_obj.clone(), vm)?; + return Ok(()); + } + } + }; + + // Restore previous task + let _ = _swap_current_task(loop_obj.clone(), prev_task, vm); + + // Unregister from eager tasks + _unregister_eager_task(task_obj.clone(), vm)?; + + // Handle the result + match step_result { + Ok(result) => { + task_step_handle_result(zelf, result, vm)?; + } + Err(e) => { + task_step_handle_exception(zelf, e, vm)?; + } + } + + // If task is no longer pending, clear the coroutine + if zelf.base.fut_state.load() != FutureState::Pending { + *zelf.task_coro.write() = None; + } + + Ok(()) + } + + /// Task step implementation + fn task_step_impl( + task: &PyObjectRef, + exc: Option<PyObjectRef>, + vm: &VirtualMachine, + ) -> PyResult { + let task_ref: PyRef<PyTask> = task + .clone() + .downcast() + .map_err(|_| vm.new_type_error("task_step called with non-Task object"))?; + + if task_ref.base.fut_state.load() != FutureState::Pending { + // Task is already done - report InvalidStateError via exception handler + let loop_obj = task_ref.base.fut_loop.read().clone(); + if let Some(loop_obj) = loop_obj { + let exc = new_invalid_state_error(vm, "step(): already done"); + let context = vm.ctx.new_dict(); + context.set_item("message", vm.new_pyobj("step(): already done"), vm)?; + context.set_item("exception", exc.clone().into(), vm)?; + context.set_item("task", task.clone(), vm)?; + let _ = vm.call_method(&loop_obj, "call_exception_handler", (context,)); + } + return Ok(vm.ctx.none()); + } + + *task_ref.task_fut_waiter.write() = None; + + let coro = task_ref.task_coro.read().clone(); + let coro = match coro { + Some(c) => c, + None => return Ok(vm.ctx.none()), + }; + + // Get event loop for enter/leave task + let loop_obj = task_ref.base.fut_loop.read().clone(); + let loop_obj = match loop_obj { + Some(l) => l, + None => return Ok(vm.ctx.none()), + }; + + // Get task context + let context = task_ref.task_context.read().clone(); + + // Enter task - register as current task + _enter_task(loop_obj.clone(), task.clone(), vm)?; + + // Determine the exception to throw (if any) + // If task_must_cancel is set and exc is None or not CancelledError, create CancelledError + let exc_to_throw = if task_ref.task_must_cancel.load(Ordering::Relaxed) { + task_ref.task_must_cancel.store(false, Ordering::Relaxed); + if let Some(ref e) = exc { + if is_cancelled_error_obj(e, vm) { + exc.clone() + } else { + Some(task_ref.make_cancelled_error_impl(vm).into()) + } + } else { + Some(task_ref.make_cancelled_error_impl(vm).into()) + } + } else { + exc + }; + + // Run coroutine step within task's context + let result = if let Some(ctx) = context { + // Use context.run(callable, *args) to run within the task's context + if let Some(ref exc_obj) = exc_to_throw { + let throw_method = coro.get_attr(vm.ctx.intern_str("throw"), vm)?; + vm.call_method(&ctx, "run", (throw_method, exc_obj.clone())) + } else { + let send_method = coro.get_attr(vm.ctx.intern_str("send"), vm)?; + vm.call_method(&ctx, "run", (send_method, vm.ctx.none())) + } + } else { + // Fallback: run without context + if let Some(ref exc_obj) = exc_to_throw { + vm.call_method(&coro, "throw", (exc_obj.clone(),)) + } else { + vm.call_method(&coro, "send", (vm.ctx.none(),)) + } + }; + + // Leave task - unregister as current task (must happen even on error) + let _ = _leave_task(loop_obj, task.clone(), vm); + + match result { + Ok(result) => { + task_step_handle_result(&task_ref, result, vm)?; + } + Err(e) => { + task_step_handle_exception(&task_ref, e, vm)?; + } + } + + Ok(vm.ctx.none()) + } + + fn task_step_handle_result( + task: &PyRef<PyTask>, + result: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult<()> { + // Check if task awaits on itself + let task_obj: PyObjectRef = task.clone().into(); + if result.is(&task_obj) { + let msg = format!( + "Task cannot await on itself: {}", + task_obj.repr(vm)?.as_str() + ); + task.base.fut_state.store(FutureState::Finished); + *task.base.fut_exception.write() = Some(vm.new_runtime_error(msg).into()); + PyTask::schedule_callbacks(task, vm)?; + _unregister_task(task_obj, vm)?; + return Ok(()); + } + + let blocking = vm + .get_attribute_opt( + result.clone(), + vm.ctx.intern_str("_asyncio_future_blocking"), + )? + .and_then(|v| v.try_to_bool(vm).ok()) + .unwrap_or(false); + + if blocking { + result.set_attr( + vm.ctx.intern_str("_asyncio_future_blocking"), + vm.ctx.new_bool(false), + vm, + )?; + + // Get the future's loop, similar to get_future_loop: + // 1. If it's our native Future/Task, access fut_loop directly (check Task first) + // 2. Otherwise try get_loop(), falling back to _loop on AttributeError + let fut_loop = if let Ok(task) = result.clone().downcast::<PyTask>() { + task.base + .fut_loop + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none()) + } else if let Ok(fut) = result.clone().downcast::<PyFuture>() { + fut.fut_loop.read().clone().unwrap_or_else(|| vm.ctx.none()) + } else { + // Try get_loop(), fall back to _loop on AttributeError + match vm.call_method(&result, "get_loop", ()) { + Ok(loop_obj) => loop_obj, + Err(e) if e.fast_isinstance(vm.ctx.exceptions.attribute_error) => { + result.get_attr(vm.ctx.intern_str("_loop"), vm)? + } + Err(e) => return Err(e), + } + }; + let task_loop = task.base.fut_loop.read().clone(); + if let Some(task_loop) = task_loop + && !fut_loop.is(&task_loop) + { + let task_repr = task + .as_object() + .repr(vm) + .unwrap_or_else(|_| vm.ctx.new_str("<Task>")); + let result_repr = result + .repr(vm) + .unwrap_or_else(|_| vm.ctx.new_str("<Future>")); + let msg = format!( + "Task {} got Future {} attached to a different loop", + task_repr, result_repr + ); + task.base.fut_state.store(FutureState::Finished); + *task.base.fut_exception.write() = Some(vm.new_runtime_error(msg).into()); + PyTask::schedule_callbacks(task, vm)?; + _unregister_task(task.clone().into(), vm)?; + return Ok(()); + } + + *task.task_fut_waiter.write() = Some(result.clone()); + + let task_obj: PyObjectRef = task.clone().into(); + let wakeup_wrapper = TaskWakeupMethWrapper::new(task_obj.clone()).into_ref(&vm.ctx); + vm.call_method(&result, "add_done_callback", (wakeup_wrapper,))?; + + // Track awaited_by relationship for introspection + future_add_to_awaited_by(result.clone(), task_obj, vm)?; + + // If task_must_cancel is set, cancel the awaited future immediately + // This propagates the cancellation through the future chain + if task.task_must_cancel.load(Ordering::Relaxed) { + let cancel_msg = task.base.fut_cancel_msg.read().clone(); + let cancel_args = if let Some(ref m) = cancel_msg { + FuncArgs::new( + vec![], + KwArgs::new([("msg".to_owned(), m.clone())].into_iter().collect()), + ) + } else { + FuncArgs::new(vec![], KwArgs::default()) + }; + let cancel_result = vm.call_method(&result, "cancel", cancel_args)?; + if cancel_result.try_to_bool(vm).unwrap_or(false) { + task.task_must_cancel.store(false, Ordering::Relaxed); + } + } + } else if vm.is_none(&result) { + let loop_obj = task.base.fut_loop.read().clone(); + if let Some(loop_obj) = loop_obj { + let task_obj: PyObjectRef = task.clone().into(); + let step_wrapper = TaskStepMethWrapper::new(task_obj).into_ref(&vm.ctx); + vm.call_method(&loop_obj, "call_soon", (step_wrapper,))?; + } + } else { + let msg = format!("Task got bad yield: {}", result.repr(vm)?.as_str()); + task.base.fut_state.store(FutureState::Finished); + *task.base.fut_exception.write() = Some(vm.new_runtime_error(msg).into()); + PyTask::schedule_callbacks(task, vm)?; + _unregister_task(task.clone().into(), vm)?; + } + + Ok(()) + } + + fn task_step_handle_exception( + task: &PyRef<PyTask>, + exc: PyBaseExceptionRef, + vm: &VirtualMachine, + ) -> PyResult<()> { + // Check for KeyboardInterrupt or SystemExit - these should be re-raised + let should_reraise = exc.fast_isinstance(vm.ctx.exceptions.keyboard_interrupt) + || exc.fast_isinstance(vm.ctx.exceptions.system_exit); + + if exc.fast_isinstance(vm.ctx.exceptions.stop_iteration) { + // Check if task was cancelled while running + if task.task_must_cancel.load(Ordering::Relaxed) { + // Task was cancelled - treat as cancelled instead of result + task.task_must_cancel.store(false, Ordering::Relaxed); + let cancelled_exc = task.base.make_cancelled_error_impl(vm); + task.base.fut_state.store(FutureState::Cancelled); + *task.base.fut_cancelled_exc.write() = Some(cancelled_exc.into()); + } else { + let result = exc.get_arg(0).unwrap_or_else(|| vm.ctx.none()); + task.base.fut_state.store(FutureState::Finished); + *task.base.fut_result.write() = Some(result); + } + PyTask::schedule_callbacks(task, vm)?; + _unregister_task(task.clone().into(), vm)?; + } else if is_cancelled_error(&exc, vm) { + task.base.fut_state.store(FutureState::Cancelled); + *task.base.fut_cancelled_exc.write() = Some(exc.clone().into()); + PyTask::schedule_callbacks(task, vm)?; + _unregister_task(task.clone().into(), vm)?; + } else { + task.base.fut_state.store(FutureState::Finished); + // Save the original traceback for later restoration + let tb = exc.__traceback__().map(|tb| tb.into()); + *task.base.fut_exception_tb.write() = tb; + *task.base.fut_exception.write() = Some(exc.clone().into()); + task.base.fut_log_tb.store(true, Ordering::Relaxed); + PyTask::schedule_callbacks(task, vm)?; + _unregister_task(task.clone().into(), vm)?; + } + + // Re-raise KeyboardInterrupt and SystemExit after storing in task + if should_reraise { + return Err(exc); + } + + Ok(()) + } + + fn task_wakeup_impl(task: &PyObjectRef, fut: &PyObjectRef, vm: &VirtualMachine) -> PyResult { + let task_ref: PyRef<PyTask> = task + .clone() + .downcast() + .map_err(|_| vm.new_type_error("task_wakeup called with non-Task object"))?; + + // Remove awaited_by relationship before resuming + future_discard_from_awaited_by(fut.clone(), task.clone(), vm)?; + + *task_ref.task_fut_waiter.write() = None; + + // Call result() on the awaited future to get either result or exception + // If result() raises an exception (like CancelledError), pass it to task_step + let exc = match vm.call_method(fut, "result", ()) { + Ok(_) => None, + Err(e) => Some(e.into()), + }; + + // Call task_step directly instead of using call_soon + // This allows the awaiting task to continue in the same event loop iteration + task_step_impl(task, exc, vm) + } + + // Module Functions + + fn get_all_tasks_set(vm: &VirtualMachine) -> PyResult<PyObjectRef> { + // Use the module-level _scheduled_tasks WeakSet + let asyncio_module = vm.import("_asyncio", 0)?; + vm.get_attribute_opt(asyncio_module, vm.ctx.intern_str("_scheduled_tasks"))? + .ok_or_else(|| vm.new_attribute_error("_scheduled_tasks not found")) + } + + fn get_eager_tasks_set(vm: &VirtualMachine) -> PyResult<PyObjectRef> { + // Use the module-level _eager_tasks Set + let asyncio_module = vm.import("_asyncio", 0)?; + vm.get_attribute_opt(asyncio_module, vm.ctx.intern_str("_eager_tasks"))? + .ok_or_else(|| vm.new_attribute_error("_eager_tasks not found")) + } + + fn get_current_tasks_dict(vm: &VirtualMachine) -> PyResult<PyObjectRef> { + // Use the module-level _current_tasks Dict + let asyncio_module = vm.import("_asyncio", 0)?; + vm.get_attribute_opt(asyncio_module, vm.ctx.intern_str("_current_tasks"))? + .ok_or_else(|| vm.new_attribute_error("_current_tasks not found")) + } + + #[pyfunction] + fn _get_running_loop(vm: &VirtualMachine) -> PyObjectRef { + vm.asyncio_running_loop + .borrow() + .clone() + .unwrap_or_else(|| vm.ctx.none()) + } + + #[pyfunction] + fn _set_running_loop(loop_: OptionalOption<PyObjectRef>, vm: &VirtualMachine) { + *vm.asyncio_running_loop.borrow_mut() = loop_.flatten(); + } + + #[pyfunction] + fn get_running_loop(vm: &VirtualMachine) -> PyResult { + vm.asyncio_running_loop + .borrow() + .clone() + .ok_or_else(|| vm.new_runtime_error("no running event loop")) + } + + #[pyfunction] + fn get_event_loop(vm: &VirtualMachine) -> PyResult { + if let Some(loop_) = vm.asyncio_running_loop.borrow().clone() { + return Ok(loop_); + } + + let asyncio_events = vm.import("asyncio.events", 0)?; + let get_event_loop_policy = vm + .get_attribute_opt(asyncio_events, vm.ctx.intern_str("get_event_loop_policy"))? + .ok_or_else(|| vm.new_attribute_error("get_event_loop_policy"))?; + let policy = get_event_loop_policy.call((), vm)?; + let get_event_loop = vm + .get_attribute_opt(policy, vm.ctx.intern_str("get_event_loop"))? + .ok_or_else(|| vm.new_attribute_error("get_event_loop"))?; + get_event_loop.call((), vm) + } + + #[pyfunction] + fn current_task(args: LoopArg, vm: &VirtualMachine) -> PyResult { + let loop_obj = match args.loop_.flatten() { + Some(l) if !vm.is_none(&l) => l, + _ => { + // When loop is None or not provided, use the running loop + match vm.asyncio_running_loop.borrow().clone() { + Some(l) => l, + None => return Err(vm.new_runtime_error("no running event loop")), + } + } + }; + + // Fast path: if the loop is the current thread's running loop, + // return the per-thread running task directly + let is_current_loop = vm + .asyncio_running_loop + .borrow() + .as_ref() + .is_some_and(|rl| rl.is(&loop_obj)); + + if is_current_loop { + return Ok(vm + .asyncio_running_task + .borrow() + .clone() + .unwrap_or_else(|| vm.ctx.none())); + } + + // Slow path: look up in the module-level dict for cross-thread queries + let current_tasks = get_current_tasks_dict(vm)?; + let dict: PyDictRef = current_tasks.downcast().unwrap(); + + match dict.get_item(&*loop_obj, vm) { + Ok(task) => Ok(task), + Err(_) => Ok(vm.ctx.none()), + } + } + + #[pyfunction] + fn all_tasks(args: LoopArg, vm: &VirtualMachine) -> PyResult { + let loop_obj = match args.loop_.flatten() { + Some(l) if !vm.is_none(&l) => l, + _ => get_running_loop(vm)?, + }; + + let all_tasks_set = get_all_tasks_set(vm)?; + let result_set = PySet::default().into_ref(&vm.ctx); + + let iter = vm.call_method(&all_tasks_set, "__iter__", ())?; + loop { + match vm.call_method(&iter, "__next__", ()) { + Ok(task) => { + // Try get_loop() method first, fallback to _loop property + let task_loop = if let Ok(l) = vm.call_method(&task, "get_loop", ()) { + Some(l) + } else if let Ok(Some(l)) = + vm.get_attribute_opt(task.clone(), vm.ctx.intern_str("_loop")) + { + Some(l) + } else { + None + }; + + if let Some(task_loop) = task_loop + && task_loop.is(&loop_obj) + && let Ok(done) = vm.call_method(&task, "done", ()) + && !done.try_to_bool(vm).unwrap_or(true) + { + result_set.add(task, vm)?; + } + } + Err(e) if e.fast_isinstance(vm.ctx.exceptions.stop_iteration) => break, + Err(e) => return Err(e), + } + } + + Ok(result_set.into()) + } + + #[pyfunction] + fn _register_task(task: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { + let all_tasks_set = get_all_tasks_set(vm)?; + vm.call_method(&all_tasks_set, "add", (task,))?; + Ok(()) + } + + #[pyfunction] + fn _unregister_task(task: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { + let all_tasks_set = get_all_tasks_set(vm)?; + vm.call_method(&all_tasks_set, "discard", (task,))?; + Ok(()) + } + + #[pyfunction] + fn _register_eager_task(task: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { + let eager_tasks_set = get_eager_tasks_set(vm)?; + vm.call_method(&eager_tasks_set, "add", (task,))?; + Ok(()) + } + + #[pyfunction] + fn _unregister_eager_task(task: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { + let eager_tasks_set = get_eager_tasks_set(vm)?; + vm.call_method(&eager_tasks_set, "discard", (task,))?; + Ok(()) + } + + #[pyfunction] + fn _enter_task(loop_: PyObjectRef, task: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { + // Per-thread check, matching CPython's ts->asyncio_running_task + { + let running_task = vm.asyncio_running_task.borrow(); + if running_task.is_some() { + return Err(vm.new_runtime_error(format!( + "Cannot enter into task {:?} while another task {:?} is being executed.", + task, + running_task.as_ref().unwrap() + ))); + } + } + + *vm.asyncio_running_task.borrow_mut() = Some(task.clone()); + + // Also update the module-level dict for cross-thread queries + if let Ok(current_tasks) = get_current_tasks_dict(vm) + && let Ok(dict) = current_tasks.downcast::<rustpython_vm::builtins::PyDict>() + { + let _ = dict.set_item(&*loop_, task, vm); + } + Ok(()) + } + + #[pyfunction] + fn _leave_task(loop_: PyObjectRef, task: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { + // Per-thread check, matching CPython's ts->asyncio_running_task + { + let running_task = vm.asyncio_running_task.borrow(); + match running_task.as_ref() { + None => { + return Err(vm.new_runtime_error( + "_leave_task: task is not the current task".to_owned(), + )); + } + Some(current) if !current.is(&task) => { + return Err(vm.new_runtime_error( + "_leave_task: task is not the current task".to_owned(), + )); + } + _ => {} + } + } + + *vm.asyncio_running_task.borrow_mut() = None; + + // Also update the module-level dict + if let Ok(current_tasks) = get_current_tasks_dict(vm) + && let Ok(dict) = current_tasks.downcast::<rustpython_vm::builtins::PyDict>() + { + let _ = dict.del_item(&*loop_, vm); + } + Ok(()) + } + + #[pyfunction] + fn _swap_current_task(loop_: PyObjectRef, task: PyObjectRef, vm: &VirtualMachine) -> PyResult { + // Per-thread swap, matching CPython's swap_current_task + let prev = vm + .asyncio_running_task + .borrow() + .clone() + .unwrap_or_else(|| vm.ctx.none()); + + if vm.is_none(&task) { + *vm.asyncio_running_task.borrow_mut() = None; + } else { + *vm.asyncio_running_task.borrow_mut() = Some(task.clone()); + } + + // Also update the module-level dict for cross-thread queries + if let Ok(current_tasks) = get_current_tasks_dict(vm) + && let Ok(dict) = current_tasks.downcast::<rustpython_vm::builtins::PyDict>() + { + if vm.is_none(&task) { + let _ = dict.del_item(&*loop_, vm); + } else { + let _ = dict.set_item(&*loop_, task, vm); + } + } + + Ok(prev) + } + + /// Reset task state after fork in child process. + #[pyfunction] + fn _on_fork(vm: &VirtualMachine) -> PyResult<()> { + // Clear current_tasks dict so child process doesn't inherit parent's tasks + if let Ok(current_tasks) = get_current_tasks_dict(vm) { + vm.call_method(&current_tasks, "clear", ())?; + } + // Clear the running loop and task + *vm.asyncio_running_loop.borrow_mut() = None; + *vm.asyncio_running_task.borrow_mut() = None; + Ok(()) + } + + #[pyfunction] + fn future_add_to_awaited_by( + fut: PyObjectRef, + waiter: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult<()> { + // Only operate on native Future/Task objects (including subclasses). + // Non-native objects are silently ignored. + if let Some(task) = fut.downcast_ref::<PyTask>() { + return task.base.awaited_by_add(waiter, vm); + } + if let Some(future) = fut.downcast_ref::<PyFuture>() { + return future.awaited_by_add(waiter, vm); + } + Ok(()) + } + + #[pyfunction] + fn future_discard_from_awaited_by( + fut: PyObjectRef, + waiter: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult<()> { + // Only operate on native Future/Task objects (including subclasses). + // Non-native objects are silently ignored. + if let Some(task) = fut.downcast_ref::<PyTask>() { + return task.base.awaited_by_discard(&waiter, vm); + } + if let Some(future) = fut.downcast_ref::<PyFuture>() { + return future.awaited_by_discard(&waiter, vm); + } + Ok(()) + } + + // TaskStepMethWrapper - wrapper for task step callback with proper repr + + #[pyattr] + #[pyclass(name, traverse)] + #[derive(Debug, PyPayload)] + struct TaskStepMethWrapper { + task: PyRwLock<Option<PyObjectRef>>, + } + + #[pyclass(with(Callable, Representable))] + impl TaskStepMethWrapper { + fn new(task: PyObjectRef) -> Self { + Self { + task: PyRwLock::new(Some(task)), + } + } + + // __self__ property returns the task, used by _format_handle in base_events.py + #[pygetset] + fn __self__(&self, vm: &VirtualMachine) -> PyObjectRef { + self.task.read().clone().unwrap_or_else(|| vm.ctx.none()) + } + + #[pygetset] + fn __qualname__(&self, vm: &VirtualMachine) -> PyResult<Option<PyObjectRef>> { + match self.task.read().as_ref() { + Some(t) => vm.get_attribute_opt(t.clone(), vm.ctx.intern_str("__qualname__")), + None => Ok(None), + } + } + } + + impl Callable for TaskStepMethWrapper { + type Args = (); + fn call(zelf: &Py<Self>, _args: Self::Args, vm: &VirtualMachine) -> PyResult { + let task = zelf.task.read().clone(); + match task { + Some(t) => task_step_impl(&t, None, vm), + None => Ok(vm.ctx.none()), + } + } + } + + impl Representable for TaskStepMethWrapper { + fn repr_str(zelf: &Py<Self>, _vm: &VirtualMachine) -> PyResult<String> { + Ok(format!( + "<{} object at {:#x}>", + zelf.class().name(), + zelf.get_id() + )) + } + } + + /// TaskWakeupMethWrapper - wrapper for task wakeup callback with proper repr + #[pyattr] + #[pyclass(name, traverse)] + #[derive(Debug, PyPayload)] + struct TaskWakeupMethWrapper { + task: PyRwLock<Option<PyObjectRef>>, + } + + #[pyclass(with(Callable, Representable))] + impl TaskWakeupMethWrapper { + fn new(task: PyObjectRef) -> Self { + Self { + task: PyRwLock::new(Some(task)), + } + } + + #[pygetset] + fn __qualname__(&self, vm: &VirtualMachine) -> PyResult<Option<PyObjectRef>> { + match self.task.read().as_ref() { + Some(t) => vm.get_attribute_opt(t.clone(), vm.ctx.intern_str("__qualname__")), + None => Ok(None), + } + } + } + + impl Callable for TaskWakeupMethWrapper { + type Args = (PyObjectRef,); + fn call(zelf: &Py<Self>, args: Self::Args, vm: &VirtualMachine) -> PyResult { + let task = zelf.task.read().clone(); + match task { + Some(t) => task_wakeup_impl(&t, &args.0, vm), + None => Ok(vm.ctx.none()), + } + } + } + + impl Representable for TaskWakeupMethWrapper { + fn repr_str(zelf: &Py<Self>, _vm: &VirtualMachine) -> PyResult<String> { + Ok(format!( + "<{} object at {:#x}>", + zelf.class().name(), + zelf.get_id() + )) + } + } + + fn is_coroutine(obj: PyObjectRef, vm: &VirtualMachine) -> PyResult<bool> { + if obj.class().is(vm.ctx.types.coroutine_type) { + return Ok(true); + } + + let asyncio_coroutines = vm.import("asyncio.coroutines", 0)?; + if let Some(iscoroutine) = + vm.get_attribute_opt(asyncio_coroutines, vm.ctx.intern_str("iscoroutine"))? + { + let result = iscoroutine.call((obj,), vm)?; + result.try_to_bool(vm) + } else { + Ok(false) + } + } + + fn new_invalid_state_error(vm: &VirtualMachine, msg: &str) -> PyBaseExceptionRef { + match vm.import("asyncio.exceptions", 0) { + Ok(module) => { + match vm.get_attribute_opt(module, vm.ctx.intern_str("InvalidStateError")) { + Ok(Some(exc_type)) => match exc_type.call((msg,), vm) { + Ok(exc) => exc.downcast().unwrap(), + Err(_) => vm.new_runtime_error(msg.to_string()), + }, + _ => vm.new_runtime_error(msg.to_string()), + } + } + Err(_) => vm.new_runtime_error(msg.to_string()), + } + } + + fn get_copy_context(vm: &VirtualMachine) -> PyResult<PyObjectRef> { + let contextvars = vm.import("contextvars", 0)?; + let copy_context = vm + .get_attribute_opt(contextvars, vm.ctx.intern_str("copy_context"))? + .ok_or_else(|| vm.new_attribute_error("copy_context not found"))?; + copy_context.call((), vm) + } + + fn get_cancelled_error_type(vm: &VirtualMachine) -> PyResult<PyTypeRef> { + let module = vm.import("asyncio.exceptions", 0)?; + let exc_type = vm + .get_attribute_opt(module, vm.ctx.intern_str("CancelledError"))? + .ok_or_else(|| vm.new_attribute_error("CancelledError not found"))?; + exc_type + .downcast() + .map_err(|_| vm.new_type_error("CancelledError is not a type".to_string())) + } + + fn is_cancelled_error(exc: &PyBaseExceptionRef, vm: &VirtualMachine) -> bool { + match get_cancelled_error_type(vm) { + Ok(cancelled_error) => exc.fast_isinstance(&cancelled_error), + Err(_) => false, + } + } + + fn is_cancelled_error_obj(obj: &PyObjectRef, vm: &VirtualMachine) -> bool { + match get_cancelled_error_type(vm) { + Ok(cancelled_error) => obj.fast_isinstance(&cancelled_error), + Err(_) => false, + } + } +} diff --git a/crates/stdlib/src/opcode.rs b/crates/stdlib/src/_opcode.rs similarity index 71% rename from crates/stdlib/src/opcode.rs rename to crates/stdlib/src/_opcode.rs index cf6502181a7..f2b447e78b6 100644 --- a/crates/stdlib/src/opcode.rs +++ b/crates/stdlib/src/_opcode.rs @@ -1,33 +1,47 @@ -pub(crate) use opcode::make_module; +pub(crate) use _opcode::module_def; #[pymodule] -mod opcode { +mod _opcode { use crate::vm::{ AsObject, PyObjectRef, PyResult, VirtualMachine, builtins::{PyInt, PyIntRef}, - bytecode::Instruction, + bytecode::{AnyInstruction, Instruction, InstructionMetadata, PseudoInstruction}, }; use core::ops::Deref; - struct Opcode(Instruction); + #[derive(Clone, Copy)] + struct Opcode(AnyInstruction); impl Deref for Opcode { - type Target = Instruction; + type Target = AnyInstruction; fn deref(&self) -> &Self::Target { &self.0 } } + impl TryFrom<i32> for Opcode { + type Error = (); + + fn try_from(value: i32) -> Result<Self, Self::Error> { + Ok(Self( + u16::try_from(value) + .map_err(|_| ())? + .try_into() + .map_err(|_| ())?, + )) + } + } + impl Opcode { - // https://github.com/python/cpython/blob/bcee1c322115c581da27600f2ae55e5439c027eb/Include/opcode_ids.h#L238 - const HAVE_ARGUMENT: i32 = 44; + // https://github.com/python/cpython/blob/v3.14.2/Include/opcode_ids.h#L252 + const HAVE_ARGUMENT: i32 = 43; pub fn try_from_pyint(raw: PyIntRef, vm: &VirtualMachine) -> PyResult<Self> { let instruction = raw - .try_to_primitive::<u8>(vm) + .try_to_primitive::<u16>(vm) .and_then(|v| { - Instruction::try_from(v).map_err(|_| { + AnyInstruction::try_from(v).map_err(|_| { vm.new_exception_empty(vm.ctx.exceptions.value_error.to_owned()) }) }) @@ -36,13 +50,14 @@ mod opcode { Ok(Self(instruction)) } - /// Check if opcode is valid (can be converted to an Instruction) + const fn inner(self) -> AnyInstruction { + self.0 + } + + /// Check if opcode is valid (can be converted to an AnyInstruction) #[must_use] pub fn is_valid(opcode: i32) -> bool { - if !(0..=255).contains(&opcode) { - return false; - } - Instruction::try_from(opcode as u8).is_ok() + Self::try_from(opcode).is_ok() } /// Check if instruction has an argument @@ -54,81 +69,74 @@ mod opcode { /// Check if instruction uses co_consts #[must_use] pub fn has_const(opcode: i32) -> bool { - Self::is_valid(opcode) - && matches!( - Instruction::try_from(opcode as u8), - Ok(Instruction::LoadConst { .. } | Instruction::ReturnConst { .. }) - ) + matches!( + Self::try_from(opcode).map(|op| op.inner()), + Ok(AnyInstruction::Real(Instruction::LoadConst { .. })) + ) } /// Check if instruction uses co_names #[must_use] pub fn has_name(opcode: i32) -> bool { - Self::is_valid(opcode) - && matches!( - Instruction::try_from(opcode as u8), - Ok(Instruction::DeleteAttr { .. } + matches!( + Self::try_from(opcode).map(|op| op.inner()), + Ok(AnyInstruction::Real( + Instruction::DeleteAttr { .. } | Instruction::DeleteGlobal(_) - | Instruction::DeleteLocal(_) + | Instruction::DeleteName(_) | Instruction::ImportFrom { .. } | Instruction::ImportName { .. } | Instruction::LoadAttr { .. } | Instruction::LoadGlobal(_) - | Instruction::LoadMethod { .. } - | Instruction::LoadNameAny(_) + | Instruction::LoadName(_) | Instruction::StoreAttr { .. } | Instruction::StoreGlobal(_) - | Instruction::StoreLocal(_)) - ) + | Instruction::StoreName(_) + )) + ) } /// Check if instruction is a jump #[must_use] pub fn has_jump(opcode: i32) -> bool { - Self::is_valid(opcode) - && matches!( - Instruction::try_from(opcode as u8), - Ok(Instruction::Break { .. } - | Instruction::Continue { .. } - | Instruction::ForIter { .. } - | Instruction::JumpIfFalseOrPop { .. } - | Instruction::JumpIfNotExcMatch(_) - | Instruction::JumpIfTrueOrPop { .. } - | Instruction::Jump { .. } + matches!( + Self::try_from(opcode).map(|op| op.inner()), + Ok(AnyInstruction::Real( + Instruction::ForIter { .. } | Instruction::PopJumpIfFalse { .. } | Instruction::PopJumpIfTrue { .. } - | Instruction::PopJumpIfNone { .. } - | Instruction::PopJumpIfNotNone { .. } - | Instruction::Send { .. }) - ) + | Instruction::Send { .. } + ) | AnyInstruction::Pseudo(PseudoInstruction::Jump { .. })) + ) } /// Check if instruction uses co_freevars/co_cellvars #[must_use] pub fn has_free(opcode: i32) -> bool { - Self::is_valid(opcode) - && matches!( - Instruction::try_from(opcode as u8), - Ok(Instruction::DeleteDeref(_) - | Instruction::LoadClassDeref(_) - | Instruction::LoadClosure(_) + matches!( + Self::try_from(opcode).map(|op| op.inner()), + Ok(AnyInstruction::Real( + Instruction::DeleteDeref(_) + | Instruction::LoadFromDictOrDeref(_) | Instruction::LoadDeref(_) - | Instruction::StoreDeref(_)) - ) + | Instruction::StoreDeref(_) + )) + ) } /// Check if instruction uses co_varnames (local variables) #[must_use] pub fn has_local(opcode: i32) -> bool { - Self::is_valid(opcode) - && matches!( - Instruction::try_from(opcode as u8), - Ok(Instruction::DeleteFast(_) + matches!( + Self::try_from(opcode).map(|op| op.inner()), + Ok(AnyInstruction::Real( + Instruction::DeleteFast(_) | Instruction::LoadFast(_) | Instruction::LoadFastAndClear(_) | Instruction::StoreFast(_) - | Instruction::StoreFastLoadFast { .. }) - ) + | Instruction::StoreFastLoadFast { .. } + )) + ) } /// Check if instruction has exception info @@ -140,8 +148,11 @@ mod opcode { } } + // prepare specialization #[pyattr] const ENABLE_SPECIALIZATION: i8 = 1; + #[pyattr] + const ENABLE_SPECIALIZATION_FT: i8 = 1; #[derive(FromArgs)] struct StackEffectArgs { @@ -186,7 +197,8 @@ mod opcode { let opcode = Opcode::try_from_pyint(args.opcode, vm)?; - Ok(opcode.stack_effect(oparg.into(), jump)) + let _ = jump; // Python API accepts jump but it's not used + Ok(opcode.stack_effect(oparg)) } #[pyfunction] @@ -294,6 +306,7 @@ mod opcode { ("NB_INPLACE_SUBTRACT", "-="), ("NB_INPLACE_TRUE_DIVIDE", "/="), ("NB_INPLACE_XOR", "^="), + ("NB_SUBSCR", "[]"), ] .into_iter() .map(|(a, b)| { @@ -305,8 +318,19 @@ mod opcode { } #[pyfunction] - fn get_executor(_code: PyObjectRef, vm: &VirtualMachine) -> PyResult<PyObjectRef> { - // TODO + fn get_special_method_names(vm: &VirtualMachine) -> Vec<PyObjectRef> { + ["__enter__", "__exit__", "__aenter__", "__aexit__"] + .into_iter() + .map(|x| vm.ctx.new_str(x).into()) + .collect() + } + + #[pyfunction] + fn get_executor( + _code: PyObjectRef, + _offset: i32, + vm: &VirtualMachine, + ) -> PyResult<PyObjectRef> { Ok(vm.ctx.none()) } diff --git a/crates/stdlib/src/_remote_debugging.rs b/crates/stdlib/src/_remote_debugging.rs new file mode 100644 index 00000000000..57aa9876a01 --- /dev/null +++ b/crates/stdlib/src/_remote_debugging.rs @@ -0,0 +1,107 @@ +pub(crate) use _remote_debugging::module_def; + +#[pymodule] +mod _remote_debugging { + use crate::vm::{ + Py, PyObjectRef, PyResult, VirtualMachine, + builtins::PyType, + function::FuncArgs, + types::{Constructor, PyStructSequence}, + }; + + #[pystruct_sequence_data] + struct FrameInfoData { + filename: String, + lineno: i64, + funcname: String, + } + + #[pyattr] + #[pystruct_sequence( + name = "FrameInfo", + module = "_remote_debugging", + data = "FrameInfoData" + )] + struct FrameInfo; + + #[pyclass(with(PyStructSequence))] + impl FrameInfo {} + + #[pystruct_sequence_data] + struct TaskInfoData { + task_id: PyObjectRef, + task_name: PyObjectRef, + coroutine_stack: PyObjectRef, + awaited_by: PyObjectRef, + } + + #[pyattr] + #[pystruct_sequence(name = "TaskInfo", module = "_remote_debugging", data = "TaskInfoData")] + struct TaskInfo; + + #[pyclass(with(PyStructSequence))] + impl TaskInfo {} + + #[pystruct_sequence_data] + struct CoroInfoData { + call_stack: PyObjectRef, + task_name: PyObjectRef, + } + + #[pyattr] + #[pystruct_sequence(name = "CoroInfo", module = "_remote_debugging", data = "CoroInfoData")] + struct CoroInfo; + + #[pyclass(with(PyStructSequence))] + impl CoroInfo {} + + #[pystruct_sequence_data] + struct ThreadInfoData { + thread_id: PyObjectRef, + frame_info: PyObjectRef, + } + + #[pyattr] + #[pystruct_sequence( + name = "ThreadInfo", + module = "_remote_debugging", + data = "ThreadInfoData" + )] + struct ThreadInfo; + + #[pyclass(with(PyStructSequence))] + impl ThreadInfo {} + + #[pystruct_sequence_data] + struct AwaitedInfoData { + thread_id: PyObjectRef, + awaited_by: PyObjectRef, + } + + #[pyattr] + #[pystruct_sequence( + name = "AwaitedInfo", + module = "_remote_debugging", + data = "AwaitedInfoData" + )] + struct AwaitedInfo; + + #[pyclass(with(PyStructSequence))] + impl AwaitedInfo {} + + #[pyattr] + #[pyclass(name = "RemoteUnwinder", module = "_remote_debugging")] + #[derive(Debug, PyPayload)] + struct RemoteUnwinder {} + + impl Constructor for RemoteUnwinder { + type Args = FuncArgs; + + fn py_new(_cls: &Py<PyType>, _args: Self::Args, vm: &VirtualMachine) -> PyResult<Self> { + Err(vm.new_not_implemented_error("_remote_debugging is not available".to_owned())) + } + } + + #[pyclass(with(Constructor))] + impl RemoteUnwinder {} +} diff --git a/crates/stdlib/src/sqlite.rs b/crates/stdlib/src/_sqlite3.rs similarity index 98% rename from crates/stdlib/src/sqlite.rs rename to crates/stdlib/src/_sqlite3.rs index 7e0392b1f30..825fff06b75 100644 --- a/crates/stdlib/src/sqlite.rs +++ b/crates/stdlib/src/_sqlite3.rs @@ -8,18 +8,10 @@ // spell-checker:ignore cantlock commithook foreignkey notnull primarykey gettemppath autoindex convpath // spell-checker:ignore dbmoved vnode nbytes -use rustpython_vm::{AsObject, PyRef, VirtualMachine, builtins::PyModule}; - -// pub(crate) use _sqlite::make_module; -pub(crate) fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - // TODO: sqlite version check - let module = _sqlite::make_module(vm); - _sqlite::setup_module(module.as_object(), vm); - module -} +pub(crate) use _sqlite3::module_def; #[pymodule] -mod _sqlite { +mod _sqlite3 { use libsqlite3_sys::{ SQLITE_BLOB, SQLITE_DETERMINISTIC, SQLITE_FLOAT, SQLITE_INTEGER, SQLITE_NULL, SQLITE_OPEN_CREATE, SQLITE_OPEN_READWRITE, SQLITE_OPEN_URI, SQLITE_TEXT, SQLITE_TRACE_STMT, @@ -58,8 +50,8 @@ mod _sqlite { TryFromBorrowedObject, VirtualMachine, atomic_func, builtins::{ PyBaseException, PyBaseExceptionRef, PyByteArray, PyBytes, PyDict, PyDictRef, PyFloat, - PyInt, PyIntRef, PySlice, PyStr, PyStrRef, PyTuple, PyTupleRef, PyType, PyTypeRef, - PyUtf8Str, PyUtf8StrRef, + PyInt, PyIntRef, PyModule, PySlice, PyStr, PyStrRef, PyTuple, PyTupleRef, PyType, + PyTypeRef, PyUtf8Str, PyUtf8StrRef, }, convert::IntoObject, function::{ @@ -425,6 +417,12 @@ mod _sqlite { name: PyStrRef, } + #[derive(FromArgs)] + struct CursorArgs { + #[pyarg(any, default)] + factory: OptionalArg<PyObjectRef>, + } + struct CallbackData { obj: NonNull<PyObject>, vm: *const VirtualMachine, @@ -846,26 +844,26 @@ mod _sqlite { .expect("enable traceback not initialize") } - pub(super) fn setup_module(module: &PyObject, vm: &VirtualMachine) { + pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { + __module_exec(vm, module); + for (name, code) in ERROR_CODES { let name = vm.ctx.intern_str(*name); let code = vm.new_pyobj(*code); - module.set_attr(name, code, vm).unwrap(); + module.set_attr(name, code, vm)?; } - setup_module_exceptions(module, vm); + setup_module_exceptions(module.as_object(), vm); let _ = CONVERTERS.set(vm.ctx.new_dict()); let _ = ADAPTERS.set(vm.ctx.new_dict()); let _ = USER_FUNCTION_EXCEPTION.set(PyAtomicRef::from(None)); let _ = ENABLE_TRACEBACK.set(Radium::new(false)); - module - .set_attr("converters", converters().to_owned(), vm) - .unwrap(); - module - .set_attr("adapters", adapters().to_owned(), vm) - .unwrap(); + module.set_attr("converters", converters().to_owned(), vm)?; + module.set_attr("adapters", adapters().to_owned(), vm)?; + + Ok(()) } #[pyattr] @@ -1023,22 +1021,29 @@ mod _sqlite { #[pymethod] fn cursor( zelf: PyRef<Self>, - factory: OptionalArg<ArgCallable>, + args: CursorArgs, vm: &VirtualMachine, - ) -> PyResult<PyRef<Cursor>> { + ) -> PyResult<PyObjectRef> { zelf.db_lock(vm).map(drop)?; - let cursor = if let OptionalArg::Present(factory) = factory { - let cursor = factory.invoke((zelf.clone(),), vm)?; - let cursor = cursor.downcast::<Cursor>().map_err(|x| { - vm.new_type_error(format!("factory must return a cursor, not {}", x.class())) - })?; - let _ = unsafe { cursor.row_factory.swap(zelf.row_factory.to_owned()) }; - cursor - } else { - let row_factory = zelf.row_factory.to_owned(); - Cursor::new(zelf, row_factory, vm).into_ref(&vm.ctx) + let factory = match args.factory { + OptionalArg::Present(f) => f, + OptionalArg::Missing => Cursor::class(&vm.ctx).to_owned().into(), }; + + let cursor = factory.call((zelf.clone(),), vm)?; + + if !cursor.class().fast_issubclass(Cursor::class(&vm.ctx)) { + return Err(vm.new_type_error(format!( + "factory must return a cursor, not {}", + cursor.class() + ))); + } + + if let Some(cursor_ref) = cursor.downcast_ref::<Cursor>() { + let _ = unsafe { cursor_ref.row_factory.swap(zelf.row_factory.to_owned()) }; + } + Ok(cursor) } diff --git a/crates/stdlib/src/array.rs b/crates/stdlib/src/array.rs index 463027d9baa..3f33f23d78d 100644 --- a/crates/stdlib/src/array.rs +++ b/crates/stdlib/src/array.rs @@ -1,33 +1,6 @@ // spell-checker:ignore typecode tofile tolist fromfile -use rustpython_vm::{PyRef, VirtualMachine, builtins::PyModule}; - -pub(crate) fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - let module = array::make_module(vm); - - let array = module - .get_attr("array", vm) - .expect("Expect array has array type."); - - let collections_abc = vm - .import("collections.abc", 0) - .expect("Expect collections exist."); - let abc = collections_abc - .get_attr("abc", vm) - .expect("Expect collections has abc submodule."); - let mutable_sequence = abc - .get_attr("MutableSequence", vm) - .expect("Expect collections.abc has MutableSequence type."); - - let register = &mutable_sequence - .get_attr("register", vm) - .expect("Expect collections.abc.MutableSequence has register method."); - register - .call((array,), vm) - .expect("Expect collections.abc.MutableSequence.register(array.array) not fail."); - - module -} +pub(crate) use array::module_def; #[pymodule(name = "array")] mod array { @@ -1658,4 +1631,25 @@ mod array { }; PyArray::from(array).into_ref_with_type(vm, cls) } + + // Register array.array as collections.abc.MutableSequence + pub(crate) fn module_exec( + vm: &VirtualMachine, + module: &Py<crate::vm::builtins::PyModule>, + ) -> PyResult<()> { + __module_exec(vm, module); + + let array_type = module + .get_attr("array", vm) + .expect("array module has array type"); + + // vm.import returns the top-level module, so we need to get abc submodule + let collections_abc = vm.import("collections.abc", 0)?; + let abc = collections_abc.get_attr("abc", vm)?; + let mutable_sequence = abc.get_attr("MutableSequence", vm)?; + let register = mutable_sequence.get_attr("register", vm)?; + register.call((array_type,), vm)?; + + Ok(()) + } } diff --git a/crates/stdlib/src/binascii.rs b/crates/stdlib/src/binascii.rs index 671d1d9e253..ee55d482e4c 100644 --- a/crates/stdlib/src/binascii.rs +++ b/crates/stdlib/src/binascii.rs @@ -1,7 +1,7 @@ // spell-checker:ignore hexlify unhexlify uuencodes CRCTAB rlecode rledecode pub(super) use decl::crc32; -pub(crate) use decl::make_module; +pub(crate) use decl::module_def; use rustpython_vm::{VirtualMachine, builtins::PyBaseExceptionRef, convert::ToPyException}; const PAD: u8 = 61u8; diff --git a/crates/stdlib/src/bisect.rs b/crates/stdlib/src/bisect.rs index e712d0275d2..69b6e8aee46 100644 --- a/crates/stdlib/src/bisect.rs +++ b/crates/stdlib/src/bisect.rs @@ -1,4 +1,4 @@ -pub(crate) use _bisect::make_module; +pub(crate) use _bisect::module_def; #[pymodule] mod _bisect { diff --git a/crates/stdlib/src/blake2.rs b/crates/stdlib/src/blake2.rs index 4209c966e86..5d2e991b929 100644 --- a/crates/stdlib/src/blake2.rs +++ b/crates/stdlib/src/blake2.rs @@ -1,6 +1,6 @@ // spell-checker:ignore usedforsecurity HASHXOF -pub(crate) use _blake2::make_module; +pub(crate) use _blake2::module_def; #[pymodule] mod _blake2 { diff --git a/crates/stdlib/src/bz2.rs b/crates/stdlib/src/bz2.rs index 93142e92a68..575f33c4b8f 100644 --- a/crates/stdlib/src/bz2.rs +++ b/crates/stdlib/src/bz2.rs @@ -1,6 +1,6 @@ // spell-checker:ignore compresslevel -pub(crate) use _bz2::make_module; +pub(crate) use _bz2::module_def; #[pymodule] mod _bz2 { diff --git a/crates/stdlib/src/cmath.rs b/crates/stdlib/src/cmath.rs index 6ce471195cc..e7ea317d212 100644 --- a/crates/stdlib/src/cmath.rs +++ b/crates/stdlib/src/cmath.rs @@ -1,6 +1,5 @@ -// TODO: Keep track of rust-num/num-complex/issues/2. A common trait could help with duplication -// that exists between cmath and math. -pub(crate) use cmath::make_module; +pub(crate) use cmath::module_def; + #[pymodule] mod cmath { use crate::vm::{ @@ -9,137 +8,141 @@ mod cmath { }; use num_complex::Complex64; + use crate::math::pymath_exception; + // Constants - #[pyattr] - use core::f64::consts::{E as e, PI as pi, TAU as tau}; + #[pyattr(name = "e")] + const E: f64 = pymath::cmath::E; + #[pyattr(name = "pi")] + const PI: f64 = pymath::cmath::PI; + #[pyattr(name = "tau")] + const TAU: f64 = pymath::cmath::TAU; #[pyattr(name = "inf")] - const INF: f64 = f64::INFINITY; + const INF: f64 = pymath::cmath::INF; #[pyattr(name = "nan")] - const NAN: f64 = f64::NAN; + const NAN: f64 = pymath::cmath::NAN; #[pyattr(name = "infj")] - const INFJ: Complex64 = Complex64::new(0., f64::INFINITY); + const INFJ: Complex64 = pymath::cmath::INFJ; #[pyattr(name = "nanj")] - const NANJ: Complex64 = Complex64::new(0., f64::NAN); + const NANJ: Complex64 = pymath::cmath::NANJ; #[pyfunction] - fn phase(z: ArgIntoComplex) -> f64 { - z.into_complex().arg() + fn phase(z: ArgIntoComplex, vm: &VirtualMachine) -> PyResult<f64> { + pymath::cmath::phase(z.into_complex()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn polar(x: ArgIntoComplex) -> (f64, f64) { - x.into_complex().to_polar() + fn polar(x: ArgIntoComplex, vm: &VirtualMachine) -> PyResult<(f64, f64)> { + pymath::cmath::polar(x.into_complex()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn rect(r: ArgIntoFloat, phi: ArgIntoFloat) -> Complex64 { - Complex64::from_polar(r.into_float(), phi.into_float()) + fn rect(r: ArgIntoFloat, phi: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<Complex64> { + pymath::cmath::rect(r.into_float(), phi.into_float()) + .map_err(|err| pymath_exception(err, vm)) } #[pyfunction] fn isinf(z: ArgIntoComplex) -> bool { - let Complex64 { re, im } = z.into_complex(); - re.is_infinite() || im.is_infinite() + pymath::cmath::isinf(z.into_complex()) } #[pyfunction] fn isfinite(z: ArgIntoComplex) -> bool { - z.into_complex().is_finite() + pymath::cmath::isfinite(z.into_complex()) } #[pyfunction] fn isnan(z: ArgIntoComplex) -> bool { - z.into_complex().is_nan() + pymath::cmath::isnan(z.into_complex()) } #[pyfunction] fn exp(z: ArgIntoComplex, vm: &VirtualMachine) -> PyResult<Complex64> { - let z = z.into_complex(); - result_or_overflow(z, z.exp(), vm) + pymath::cmath::exp(z.into_complex()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn sqrt(z: ArgIntoComplex) -> Complex64 { - z.into_complex().sqrt() + fn sqrt(z: ArgIntoComplex, vm: &VirtualMachine) -> PyResult<Complex64> { + pymath::cmath::sqrt(z.into_complex()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn sin(z: ArgIntoComplex) -> Complex64 { - z.into_complex().sin() + fn sin(z: ArgIntoComplex, vm: &VirtualMachine) -> PyResult<Complex64> { + pymath::cmath::sin(z.into_complex()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn asin(z: ArgIntoComplex) -> Complex64 { - z.into_complex().asin() + fn asin(z: ArgIntoComplex, vm: &VirtualMachine) -> PyResult<Complex64> { + pymath::cmath::asin(z.into_complex()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn cos(z: ArgIntoComplex) -> Complex64 { - z.into_complex().cos() + fn cos(z: ArgIntoComplex, vm: &VirtualMachine) -> PyResult<Complex64> { + pymath::cmath::cos(z.into_complex()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn acos(z: ArgIntoComplex) -> Complex64 { - z.into_complex().acos() + fn acos(z: ArgIntoComplex, vm: &VirtualMachine) -> PyResult<Complex64> { + pymath::cmath::acos(z.into_complex()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn log(z: ArgIntoComplex, base: OptionalArg<ArgIntoComplex>) -> Complex64 { - // TODO: Complex64.log with a negative base yields wrong results. - // Issue is with num_complex::Complex64 implementation of log - // which returns NaN when base is negative. - // log10(z) / log10(base) yields correct results but division - // doesn't handle pos/neg zero nicely. (i.e log(1, 0.5)) - z.into_complex().log( - base.into_option() - .map(|base| base.into_complex().re) - .unwrap_or(core::f64::consts::E), + fn log( + z: ArgIntoComplex, + base: OptionalArg<ArgIntoComplex>, + vm: &VirtualMachine, + ) -> PyResult<Complex64> { + pymath::cmath::log( + z.into_complex(), + base.into_option().map(|b| b.into_complex()), ) + .map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn log10(z: ArgIntoComplex) -> Complex64 { - z.into_complex().log(10.0) + fn log10(z: ArgIntoComplex, vm: &VirtualMachine) -> PyResult<Complex64> { + pymath::cmath::log10(z.into_complex()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn acosh(z: ArgIntoComplex) -> Complex64 { - z.into_complex().acosh() + fn acosh(z: ArgIntoComplex, vm: &VirtualMachine) -> PyResult<Complex64> { + pymath::cmath::acosh(z.into_complex()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn atan(z: ArgIntoComplex) -> Complex64 { - z.into_complex().atan() + fn atan(z: ArgIntoComplex, vm: &VirtualMachine) -> PyResult<Complex64> { + pymath::cmath::atan(z.into_complex()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn atanh(z: ArgIntoComplex) -> Complex64 { - z.into_complex().atanh() + fn atanh(z: ArgIntoComplex, vm: &VirtualMachine) -> PyResult<Complex64> { + pymath::cmath::atanh(z.into_complex()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn tan(z: ArgIntoComplex) -> Complex64 { - z.into_complex().tan() + fn tan(z: ArgIntoComplex, vm: &VirtualMachine) -> PyResult<Complex64> { + pymath::cmath::tan(z.into_complex()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn tanh(z: ArgIntoComplex) -> Complex64 { - z.into_complex().tanh() + fn tanh(z: ArgIntoComplex, vm: &VirtualMachine) -> PyResult<Complex64> { + pymath::cmath::tanh(z.into_complex()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn sinh(z: ArgIntoComplex) -> Complex64 { - z.into_complex().sinh() + fn sinh(z: ArgIntoComplex, vm: &VirtualMachine) -> PyResult<Complex64> { + pymath::cmath::sinh(z.into_complex()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn cosh(z: ArgIntoComplex) -> Complex64 { - z.into_complex().cosh() + fn cosh(z: ArgIntoComplex, vm: &VirtualMachine) -> PyResult<Complex64> { + pymath::cmath::cosh(z.into_complex()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn asinh(z: ArgIntoComplex) -> Complex64 { - z.into_complex().asinh() + fn asinh(z: ArgIntoComplex, vm: &VirtualMachine) -> PyResult<Complex64> { + pymath::cmath::asinh(z.into_complex()).map_err(|err| pymath_exception(err, vm)) } #[derive(FromArgs)] @@ -158,52 +161,10 @@ mod cmath { fn isclose(args: IsCloseArgs, vm: &VirtualMachine) -> PyResult<bool> { let a = args.a.into_complex(); let b = args.b.into_complex(); - let rel_tol = args.rel_tol.map_or(1e-09, |v| v.into_float()); - let abs_tol = args.abs_tol.map_or(0.0, |v| v.into_float()); - - if rel_tol < 0.0 || abs_tol < 0.0 { - return Err(vm.new_value_error("tolerances must be non-negative")); - } - - if a == b { - /* short circuit exact equality -- needed to catch two infinities of - the same sign. And perhaps speeds things up a bit sometimes. - */ - return Ok(true); - } - - /* This catches the case of two infinities of opposite sign, or - one infinity and one finite number. Two infinities of opposite - sign would otherwise have an infinite relative tolerance. - Two infinities of the same sign are caught by the equality check - above. - */ - if a.is_infinite() || b.is_infinite() { - return Ok(false); - } + let rel_tol = args.rel_tol.into_option().map(|v| v.into_float()); + let abs_tol = args.abs_tol.into_option().map(|v| v.into_float()); - let diff = c_abs(b - a); - - Ok(diff <= (rel_tol * c_abs(b)) || (diff <= (rel_tol * c_abs(a))) || diff <= abs_tol) - } - - #[inline] - fn c_abs(Complex64 { re, im }: Complex64) -> f64 { - re.hypot(im) - } - - #[inline] - fn result_or_overflow( - value: Complex64, - result: Complex64, - vm: &VirtualMachine, - ) -> PyResult<Complex64> { - if !result.is_finite() && value.is_finite() { - // CPython doesn't return `inf` when called with finite - // values, it raises OverflowError instead. - Err(vm.new_overflow_error("math range error")) - } else { - Ok(result) - } + pymath::cmath::isclose(a, b, rel_tol, abs_tol) + .map_err(|_| vm.new_value_error("tolerances must be non-negative")) } } diff --git a/crates/stdlib/src/contextvars.rs b/crates/stdlib/src/contextvars.rs index 329342fe6dc..700a94692d5 100644 --- a/crates/stdlib/src/contextvars.rs +++ b/crates/stdlib/src/contextvars.rs @@ -1,20 +1,9 @@ -use crate::vm::{PyRef, VirtualMachine, builtins::PyModule, class::StaticType}; +pub(crate) use _contextvars::module_def; + +use crate::vm::PyRef; use _contextvars::PyContext; use core::cell::RefCell; -pub(crate) fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - let module = _contextvars::make_module(vm); - let token_type = module.get_attr("Token", vm).unwrap(); - token_type - .set_attr( - "MISSING", - _contextvars::ContextTokenMissing::static_type().to_owned(), - vm, - ) - .unwrap(); - module -} - thread_local! { // TODO: Vec doesn't seem to match copy behavior static CONTEXTS: RefCell<Vec<PyRef<PyContext>>> = RefCell::default(); @@ -24,12 +13,12 @@ thread_local! { mod _contextvars { use crate::vm::{ AsObject, Py, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, atomic_func, - builtins::{PyGenericAlias, PyStrRef, PyType, PyTypeRef}, + builtins::{PyGenericAlias, PyList, PyStrRef, PyType, PyTypeRef}, class::StaticType, common::hash::PyHash, function::{ArgCallable, FuncArgs, OptionalArg}, protocol::{PyMappingMethods, PySequenceMethods}, - types::{AsMapping, AsSequence, Constructor, Hashable, Representable}, + types::{AsMapping, AsSequence, Constructor, Hashable, Iterable, Representable}, }; use core::{ cell::{Cell, RefCell, UnsafeCell}, @@ -163,7 +152,7 @@ mod _contextvars { } } - #[pyclass(with(Constructor, AsMapping, AsSequence))] + #[pyclass(with(Constructor, AsMapping, AsSequence, Iterable))] impl PyContext { #[pymethod] fn run( @@ -179,11 +168,15 @@ mod _contextvars { } #[pymethod] - fn copy(&self) -> Self { + fn copy(&self, vm: &VirtualMachine) -> Self { + // Deep copy the vars - clone the underlying Hamt data, not just the PyRef + let vars_copy = HamtObject { + hamt: RefCell::new(self.inner.vars.hamt.borrow().clone()), + }; Self { inner: ContextInner { idx: Cell::new(usize::MAX), - vars: self.inner.vars.clone(), + vars: vars_copy.into_ref(&vm.ctx), entered: Cell::new(false), }, } @@ -205,11 +198,6 @@ mod _contextvars { self.borrow_vars().len() } - #[pymethod] - fn __iter__(&self) -> PyResult { - unimplemented!("Context.__iter__ is currently under construction") - } - #[pymethod] fn get( &self, @@ -238,6 +226,15 @@ mod _contextvars { let vars = zelf.borrow_vars(); vars.values().map(|value| value.to_owned()).collect() } + + // TODO: wrong return type + #[pymethod] + fn items(zelf: PyRef<Self>, vm: &VirtualMachine) -> Vec<PyObjectRef> { + let vars = zelf.borrow_vars(); + vars.iter() + .map(|(k, v)| vm.ctx.new_tuple(vec![k.clone().into(), v.clone()]).into()) + .collect() + } } impl Constructor for PyContext { @@ -281,6 +278,15 @@ mod _contextvars { } } + impl Iterable for PyContext { + fn iter(zelf: PyRef<Self>, vm: &VirtualMachine) -> PyResult { + let vars = zelf.borrow_vars(); + let keys: Vec<PyObjectRef> = vars.keys().map(|k| k.clone().into()).collect(); + let list = vm.ctx.new_list(keys); + <PyList as Iterable>::iter(list, vm) + } + } + #[pyattr] #[pyclass(name, traverse)] #[derive(PyPayload)] @@ -574,6 +580,22 @@ mod _contextvars { ) -> PyGenericAlias { PyGenericAlias::from_args(cls, args, vm) } + + #[pymethod] + fn __enter__(zelf: PyRef<Self>) -> PyRef<Self> { + zelf + } + + #[pymethod] + fn __exit__( + zelf: &Py<Self>, + _ty: PyObjectRef, + _val: PyObjectRef, + _tb: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult<()> { + ContextVar::reset(&zelf.var, zelf.to_owned(), vm) + } } impl Constructor for ContextToken { @@ -612,6 +634,19 @@ mod _contextvars { #[pyfunction] fn copy_context(vm: &VirtualMachine) -> PyContext { - PyContext::current(vm).copy() + PyContext::current(vm).copy(vm) + } + + // Set Token.MISSING attribute + pub(crate) fn module_exec( + vm: &VirtualMachine, + module: &Py<crate::vm::builtins::PyModule>, + ) -> PyResult<()> { + __module_exec(vm, module); + + let token_type = module.get_attr("Token", vm)?; + token_type.set_attr("MISSING", ContextTokenMissing::static_type().to_owned(), vm)?; + + Ok(()) } } diff --git a/crates/stdlib/src/csv.rs b/crates/stdlib/src/csv.rs index 4f6cbd76828..7c89ebbba66 100644 --- a/crates/stdlib/src/csv.rs +++ b/crates/stdlib/src/csv.rs @@ -1,4 +1,4 @@ -pub(crate) use _csv::make_module; +pub(crate) use _csv::module_def; #[pymodule] mod _csv { diff --git a/crates/stdlib/src/dis.rs b/crates/stdlib/src/dis.rs deleted file mode 100644 index 341137f91f4..00000000000 --- a/crates/stdlib/src/dis.rs +++ /dev/null @@ -1,58 +0,0 @@ -pub(crate) use decl::make_module; - -#[pymodule(name = "dis")] -mod decl { - use crate::vm::{ - PyObjectRef, PyRef, PyResult, TryFromObject, VirtualMachine, - builtins::{PyCode, PyDictRef, PyStrRef}, - bytecode::CodeFlags, - }; - - #[pyfunction] - fn dis(obj: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { - let co = if let Ok(co) = obj.get_attr("__code__", vm) { - // Method or function: - PyRef::try_from_object(vm, co)? - } else if let Ok(co_str) = PyStrRef::try_from_object(vm, obj.clone()) { - #[cfg(not(feature = "compiler"))] - { - let _ = co_str; - return Err( - vm.new_runtime_error("dis.dis() with str argument requires `compiler` feature") - ); - } - #[cfg(feature = "compiler")] - { - vm.compile( - co_str.as_str(), - crate::vm::compiler::Mode::Exec, - "<dis>".to_owned(), - ) - .map_err(|err| vm.new_syntax_error(&err, Some(co_str.as_str())))? - } - } else { - PyRef::try_from_object(vm, obj)? - }; - disassemble(co) - } - - #[pyfunction] - fn disassemble(co: PyRef<PyCode>) -> PyResult<()> { - print!("{}", &co.code); - Ok(()) - } - - #[pyattr(name = "COMPILER_FLAG_NAMES")] - fn compiler_flag_names(vm: &VirtualMachine) -> PyDictRef { - let dict = vm.ctx.new_dict(); - for (name, flag) in CodeFlags::NAME_MAPPING { - dict.set_item( - &*vm.new_pyobj(flag.bits()), - vm.ctx.new_str(*name).into(), - vm, - ) - .unwrap(); - } - dict - } -} diff --git a/crates/stdlib/src/faulthandler.rs b/crates/stdlib/src/faulthandler.rs index eba5643b866..6a2a0933404 100644 --- a/crates/stdlib/src/faulthandler.rs +++ b/crates/stdlib/src/faulthandler.rs @@ -1,11 +1,13 @@ -pub(crate) use decl::make_module; +pub(crate) use decl::module_def; #[allow(static_mut_refs)] // TODO: group code only with static mut refs #[pymodule(name = "faulthandler")] mod decl { use crate::vm::{ - PyObjectRef, PyResult, VirtualMachine, builtins::PyFloat, frame::Frame, - function::OptionalArg, py_io::Write, + PyObjectRef, PyResult, VirtualMachine, + frame::Frame, + function::{ArgIntoFloat, OptionalArg}, + py_io::Write, }; use alloc::sync::Arc; use core::sync::atomic::{AtomicBool, AtomicI32, Ordering}; @@ -551,7 +553,7 @@ mod decl { } let mut action: libc::sigaction = core::mem::zeroed(); - action.sa_sigaction = faulthandler_fatal_error as libc::sighandler_t; + action.sa_sigaction = faulthandler_fatal_error as *const () as libc::sighandler_t; // SA_NODEFER flag action.sa_flags = libc::SA_NODEFER; @@ -581,7 +583,7 @@ mod decl { handler.previous = libc::signal( handler.signum, - faulthandler_fatal_error as libc::sighandler_t, + faulthandler_fatal_error as *const () as libc::sighandler_t, ); // SIG_ERR is -1 as sighandler_t (which is usize on Windows) @@ -762,8 +764,8 @@ mod decl { #[derive(FromArgs)] #[allow(unused)] struct DumpTracebackLaterArgs { - #[pyarg(positional)] - timeout: PyObjectRef, + #[pyarg(positional, error_msg = "timeout must be a number (int or float)")] + timeout: ArgIntoFloat, #[pyarg(any, default = false)] repeat: bool, #[pyarg(any, default)] @@ -774,18 +776,7 @@ mod decl { #[pyfunction] fn dump_traceback_later(args: DumpTracebackLaterArgs, vm: &VirtualMachine) -> PyResult<()> { - use num_traits::ToPrimitive; - // Convert timeout to f64 (accepting int or float) - let timeout: f64 = if let Some(float) = args.timeout.downcast_ref::<PyFloat>() { - float.to_f64() - } else if let Some(int) = args.timeout.try_index_opt(vm).transpose()? { - int.as_bigint() - .to_i64() - .ok_or_else(|| vm.new_overflow_error("timeout value is too large".to_owned()))? - as f64 - } else { - return Err(vm.new_type_error("timeout must be a number (int or float)".to_owned())); - }; + let timeout: f64 = args.timeout.into_float(); if timeout <= 0.0 { return Err(vm.new_value_error("timeout must be greater than 0".to_owned())); @@ -946,7 +937,10 @@ mod decl { libc::signal(signum, user.previous); libc::raise(signum); // Re-register our handler - libc::signal(signum, faulthandler_user_signal as libc::sighandler_t); + libc::signal( + signum, + faulthandler_user_signal as *const () as libc::sighandler_t, + ); } } } @@ -997,7 +991,10 @@ mod decl { let previous = if !user_signals::is_enabled(signum) { // Install signal handler let prev = unsafe { - libc::signal(args.signum, faulthandler_user_signal as libc::sighandler_t) + libc::signal( + args.signum, + faulthandler_user_signal as *const () as libc::sighandler_t, + ) }; if prev == libc::SIG_ERR { return Err(vm.new_os_error(format!( diff --git a/crates/stdlib/src/fcntl.rs b/crates/stdlib/src/fcntl.rs index 477c1f54210..407a2dfd6b3 100644 --- a/crates/stdlib/src/fcntl.rs +++ b/crates/stdlib/src/fcntl.rs @@ -1,6 +1,6 @@ // spell-checker:disable -pub(crate) use fcntl::make_module; +pub(crate) use fcntl::module_def; #[pymodule] mod fcntl { diff --git a/crates/stdlib/src/gc.rs b/crates/stdlib/src/gc.rs deleted file mode 100644 index 5fc96a302f7..00000000000 --- a/crates/stdlib/src/gc.rs +++ /dev/null @@ -1,76 +0,0 @@ -pub(crate) use gc::make_module; - -#[pymodule] -mod gc { - use crate::vm::{PyResult, VirtualMachine, function::FuncArgs}; - - #[pyfunction] - fn collect(_args: FuncArgs, _vm: &VirtualMachine) -> i32 { - 0 - } - - #[pyfunction] - fn isenabled(_args: FuncArgs, _vm: &VirtualMachine) -> bool { - false - } - - #[pyfunction] - fn enable(_args: FuncArgs, vm: &VirtualMachine) -> PyResult { - Err(vm.new_not_implemented_error("")) - } - - #[pyfunction] - fn disable(_args: FuncArgs, vm: &VirtualMachine) -> PyResult { - Err(vm.new_not_implemented_error("")) - } - - #[pyfunction] - fn get_count(_args: FuncArgs, vm: &VirtualMachine) -> PyResult { - Err(vm.new_not_implemented_error("")) - } - - #[pyfunction] - fn get_debug(_args: FuncArgs, vm: &VirtualMachine) -> PyResult { - Err(vm.new_not_implemented_error("")) - } - - #[pyfunction] - fn get_objects(_args: FuncArgs, vm: &VirtualMachine) -> PyResult { - Err(vm.new_not_implemented_error("")) - } - - #[pyfunction] - fn get_referents(_args: FuncArgs, vm: &VirtualMachine) -> PyResult { - Err(vm.new_not_implemented_error("")) - } - - #[pyfunction] - fn get_referrers(_args: FuncArgs, vm: &VirtualMachine) -> PyResult { - Err(vm.new_not_implemented_error("")) - } - - #[pyfunction] - fn get_stats(_args: FuncArgs, vm: &VirtualMachine) -> PyResult { - Err(vm.new_not_implemented_error("")) - } - - #[pyfunction] - fn get_threshold(_args: FuncArgs, vm: &VirtualMachine) -> PyResult { - Err(vm.new_not_implemented_error("")) - } - - #[pyfunction] - fn is_tracked(_args: FuncArgs, vm: &VirtualMachine) -> PyResult { - Err(vm.new_not_implemented_error("")) - } - - #[pyfunction] - fn set_debug(_args: FuncArgs, vm: &VirtualMachine) -> PyResult { - Err(vm.new_not_implemented_error("")) - } - - #[pyfunction] - fn set_threshold(_args: FuncArgs, vm: &VirtualMachine) -> PyResult { - Err(vm.new_not_implemented_error("")) - } -} diff --git a/crates/stdlib/src/grp.rs b/crates/stdlib/src/grp.rs index 9f7e4195509..c1a52eee62e 100644 --- a/crates/stdlib/src/grp.rs +++ b/crates/stdlib/src/grp.rs @@ -1,5 +1,5 @@ // spell-checker:disable -pub(crate) use grp::make_module; +pub(crate) use grp::module_def; #[pymodule] mod grp { diff --git a/crates/stdlib/src/hashlib.rs b/crates/stdlib/src/hashlib.rs index 2da47ceb740..c9902eff60a 100644 --- a/crates/stdlib/src/hashlib.rs +++ b/crates/stdlib/src/hashlib.rs @@ -1,6 +1,6 @@ // spell-checker:ignore usedforsecurity HASHXOF -pub(crate) use _hashlib::make_module; +pub(crate) use _hashlib::module_def; #[pymodule] pub mod _hashlib { @@ -11,7 +11,7 @@ pub mod _hashlib { class::StaticType, convert::ToPyObject, function::{ArgBytesLike, ArgStrOrBytesLike, FuncArgs, OptionalArg}, - types::{Constructor, Initializer, Representable}, + types::{Constructor, Representable}, }; use blake2::{Blake2b512, Blake2s256}; use digest::{DynDigest, core_api::BlockSizeUser}; diff --git a/crates/stdlib/src/json.rs b/crates/stdlib/src/json.rs index a3fd7972126..3baeba629c8 100644 --- a/crates/stdlib/src/json.rs +++ b/crates/stdlib/src/json.rs @@ -1,4 +1,4 @@ -pub(crate) use _json::make_module; +pub(crate) use _json::module_def; mod machinery; #[pymodule] @@ -7,7 +7,7 @@ mod _json { use crate::vm::{ AsObject, Py, PyObjectRef, PyPayload, PyResult, VirtualMachine, builtins::{PyBaseExceptionRef, PyStrRef, PyType}, - convert::{ToPyObject, ToPyResult}, + convert::ToPyResult, function::{IntoFuncArgs, OptionalArg}, protocol::PyIterReturn, types::{Callable, Constructor}, @@ -15,6 +15,29 @@ mod _json { use core::str::FromStr; use malachite_bigint::BigInt; use rustpython_common::wtf8::Wtf8Buf; + use std::collections::HashMap; + + /// Skip JSON whitespace characters (space, tab, newline, carriage return). + /// Works with a byte slice and returns the number of bytes skipped. + /// Since all JSON whitespace chars are ASCII, bytes == chars. + #[inline] + fn skip_whitespace(bytes: &[u8]) -> usize { + flame_guard!("_json::skip_whitespace"); + let mut count = 0; + for &b in bytes { + match b { + b' ' | b'\t' | b'\n' | b'\r' => count += 1, + _ => break, + } + } + count + } + + /// Check if a byte slice starts with a given ASCII pattern. + #[inline] + fn starts_with_bytes(bytes: &[u8], pattern: &[u8]) -> bool { + bytes.len() >= pattern.len() && &bytes[..pattern.len()] == pattern + } #[pyattr(name = "make_scanner")] #[pyclass(name = "Scanner", traverse)] @@ -68,57 +91,64 @@ mod _json { impl JsonScanner { fn parse( &self, - s: &str, pystr: PyStrRef, - idx: usize, + char_idx: usize, + byte_idx: usize, scan_once: PyObjectRef, vm: &VirtualMachine, ) -> PyResult<PyIterReturn> { - let c = match s.chars().next() { - Some(c) => c, + flame_guard!("JsonScanner::parse"); + let bytes = pystr.as_str().as_bytes(); + let wtf8 = pystr.as_wtf8(); + + let first_byte = match bytes.get(byte_idx) { + Some(&b) => b, None => { return Ok(PyIterReturn::StopIteration(Some( - vm.ctx.new_int(idx).into(), + vm.ctx.new_int(char_idx).into(), ))); } }; - let next_idx = idx + c.len_utf8(); - match c { - '"' => { - return scanstring(pystr, next_idx, OptionalArg::Present(self.strict), vm) - .map(|x| PyIterReturn::Return(x.to_pyobject(vm))); + + match first_byte { + b'"' => { + // Parse string - pass slice starting after the quote + let (wtf8_result, chars_consumed, _bytes_consumed) = + machinery::scanstring(&wtf8[byte_idx + 1..], char_idx + 1, self.strict) + .map_err(|e| py_decode_error(e, pystr.clone(), vm))?; + let end_char_idx = char_idx + 1 + chars_consumed; + return Ok(PyIterReturn::Return( + vm.new_tuple((wtf8_result, end_char_idx)).into(), + )); } - '{' => { - // TODO: parse the object in rust - let parse_obj = self.ctx.get_attr("parse_object", vm)?; - let result = parse_obj.call( - ( - (pystr, next_idx), - self.strict, - scan_once, - self.object_hook.clone(), - self.object_pairs_hook.clone(), - ), - vm, - ); - return PyIterReturn::from_pyresult(result, vm); + b'{' => { + // Parse object in Rust + let mut memo = HashMap::new(); + return self + .parse_object(pystr, char_idx + 1, byte_idx + 1, &scan_once, &mut memo, vm) + .map(|(obj, end_char, _end_byte)| { + PyIterReturn::Return(vm.new_tuple((obj, end_char)).into()) + }); } - '[' => { - // TODO: parse the array in rust - let parse_array = self.ctx.get_attr("parse_array", vm)?; - return PyIterReturn::from_pyresult( - parse_array.call(((pystr, next_idx), scan_once), vm), - vm, - ); + b'[' => { + // Parse array in Rust + let mut memo = HashMap::new(); + return self + .parse_array(pystr, char_idx + 1, byte_idx + 1, &scan_once, &mut memo, vm) + .map(|(obj, end_char, _end_byte)| { + PyIterReturn::Return(vm.new_tuple((obj, end_char)).into()) + }); } _ => {} } + let s = &pystr.as_str()[byte_idx..]; + macro_rules! parse_const { ($s:literal, $val:expr) => { if s.starts_with($s) { return Ok(PyIterReturn::Return( - vm.new_tuple(($val, idx + $s.len())).into(), + vm.new_tuple(($val, char_idx + $s.len())).into(), )); } }; @@ -129,15 +159,20 @@ mod _json { parse_const!("false", false); if let Some((res, len)) = self.parse_number(s, vm) { - return Ok(PyIterReturn::Return(vm.new_tuple((res?, idx + len)).into())); + return Ok(PyIterReturn::Return( + vm.new_tuple((res?, char_idx + len)).into(), + )); } macro_rules! parse_constant { ($s:literal) => { if s.starts_with($s) { return Ok(PyIterReturn::Return( - vm.new_tuple((self.parse_constant.call(($s,), vm)?, idx + $s.len())) - .into(), + vm.new_tuple(( + self.parse_constant.call(($s,), vm)?, + char_idx + $s.len(), + )) + .into(), )); } }; @@ -148,11 +183,12 @@ mod _json { parse_constant!("-Infinity"); Ok(PyIterReturn::StopIteration(Some( - vm.ctx.new_int(idx).into(), + vm.ctx.new_int(char_idx).into(), ))) } fn parse_number(&self, s: &str, vm: &VirtualMachine) -> Option<(PyResult, usize)> { + flame_guard!("JsonScanner::parse_number"); let mut has_neg = false; let mut has_decimal = false; let mut has_exponent = false; @@ -187,32 +223,448 @@ mod _json { }; Some((ret, buf.len())) } + + /// Parse a JSON object starting after the opening '{'. + /// Returns (parsed_object, end_char_index, end_byte_index). + fn parse_object( + &self, + pystr: PyStrRef, + start_char_idx: usize, + start_byte_idx: usize, + scan_once: &PyObjectRef, + memo: &mut HashMap<String, PyStrRef>, + vm: &VirtualMachine, + ) -> PyResult<(PyObjectRef, usize, usize)> { + flame_guard!("JsonScanner::parse_object"); + + let bytes = pystr.as_str().as_bytes(); + let wtf8 = pystr.as_wtf8(); + let mut char_idx = start_char_idx; + let mut byte_idx = start_byte_idx; + + // Skip initial whitespace + let ws = skip_whitespace(&bytes[byte_idx..]); + char_idx += ws; + byte_idx += ws; + + // Check for empty object + match bytes.get(byte_idx) { + Some(b'}') => { + return self.finalize_object(vec![], char_idx + 1, byte_idx + 1, vm); + } + Some(b'"') => { + // Continue to parse first key + } + _ => { + return Err(self.make_decode_error( + "Expecting property name enclosed in double quotes", + pystr, + char_idx, + vm, + )); + } + } + + let mut pairs: Vec<(PyObjectRef, PyObjectRef)> = Vec::new(); + + loop { + // We're now at '"', skip it + char_idx += 1; + byte_idx += 1; + + // Parse key string using scanstring with byte slice + let (key_wtf8, chars_consumed, bytes_consumed) = + machinery::scanstring(&wtf8[byte_idx..], char_idx, self.strict) + .map_err(|e| py_decode_error(e, pystr.clone(), vm))?; + + char_idx += chars_consumed; + byte_idx += bytes_consumed; + + // Key memoization - reuse existing key strings + let key_str = key_wtf8.to_string(); + let key: PyObjectRef = match memo.get(&key_str) { + Some(cached) => cached.clone().into(), + None => { + let py_key = vm.ctx.new_str(key_str.clone()); + memo.insert(key_str, py_key.clone()); + py_key.into() + } + }; + + // Skip whitespace after key + let ws = skip_whitespace(&bytes[byte_idx..]); + char_idx += ws; + byte_idx += ws; + + // Expect ':' delimiter + match bytes.get(byte_idx) { + Some(b':') => { + char_idx += 1; + byte_idx += 1; + } + _ => { + return Err(self.make_decode_error( + "Expecting ':' delimiter", + pystr, + char_idx, + vm, + )); + } + } + + // Skip whitespace after ':' + let ws = skip_whitespace(&bytes[byte_idx..]); + char_idx += ws; + byte_idx += ws; + + // Parse value recursively + let (value, value_char_end, value_byte_end) = + self.call_scan_once(scan_once, pystr.clone(), char_idx, byte_idx, memo, vm)?; + + pairs.push((key, value)); + char_idx = value_char_end; + byte_idx = value_byte_end; + + // Skip whitespace after value + let ws = skip_whitespace(&bytes[byte_idx..]); + char_idx += ws; + byte_idx += ws; + + // Check for ',' or '}' + match bytes.get(byte_idx) { + Some(b'}') => { + char_idx += 1; + byte_idx += 1; + break; + } + Some(b',') => { + let comma_char_idx = char_idx; + char_idx += 1; + byte_idx += 1; + + // Skip whitespace after comma + let ws = skip_whitespace(&bytes[byte_idx..]); + char_idx += ws; + byte_idx += ws; + + // Next must be '"' + match bytes.get(byte_idx) { + Some(b'"') => { + // Continue to next key-value pair + } + Some(b'}') => { + // Trailing comma before end of object + return Err(self.make_decode_error( + "Illegal trailing comma before end of object", + pystr, + comma_char_idx, + vm, + )); + } + _ => { + return Err(self.make_decode_error( + "Expecting property name enclosed in double quotes", + pystr, + char_idx, + vm, + )); + } + } + } + _ => { + return Err(self.make_decode_error( + "Expecting ',' delimiter", + pystr, + char_idx, + vm, + )); + } + } + } + + self.finalize_object(pairs, char_idx, byte_idx, vm) + } + + /// Parse a JSON array starting after the opening '['. + /// Returns (parsed_array, end_char_index, end_byte_index). + fn parse_array( + &self, + pystr: PyStrRef, + start_char_idx: usize, + start_byte_idx: usize, + scan_once: &PyObjectRef, + memo: &mut HashMap<String, PyStrRef>, + vm: &VirtualMachine, + ) -> PyResult<(PyObjectRef, usize, usize)> { + flame_guard!("JsonScanner::parse_array"); + + let bytes = pystr.as_str().as_bytes(); + let mut char_idx = start_char_idx; + let mut byte_idx = start_byte_idx; + + // Skip initial whitespace + let ws = skip_whitespace(&bytes[byte_idx..]); + char_idx += ws; + byte_idx += ws; + + // Check for empty array + if bytes.get(byte_idx) == Some(&b']') { + return Ok((vm.ctx.new_list(vec![]).into(), char_idx + 1, byte_idx + 1)); + } + + let mut values: Vec<PyObjectRef> = Vec::new(); + + loop { + // Parse value + let (value, value_char_end, value_byte_end) = + self.call_scan_once(scan_once, pystr.clone(), char_idx, byte_idx, memo, vm)?; + + values.push(value); + char_idx = value_char_end; + byte_idx = value_byte_end; + + // Skip whitespace after value + let ws = skip_whitespace(&bytes[byte_idx..]); + char_idx += ws; + byte_idx += ws; + + match bytes.get(byte_idx) { + Some(b']') => { + char_idx += 1; + byte_idx += 1; + break; + } + Some(b',') => { + let comma_char_idx = char_idx; + char_idx += 1; + byte_idx += 1; + + // Skip whitespace after comma + let ws = skip_whitespace(&bytes[byte_idx..]); + char_idx += ws; + byte_idx += ws; + + // Check for trailing comma + if bytes.get(byte_idx) == Some(&b']') { + return Err(self.make_decode_error( + "Illegal trailing comma before end of array", + pystr, + comma_char_idx, + vm, + )); + } + } + _ => { + return Err(self.make_decode_error( + "Expecting ',' delimiter", + pystr, + char_idx, + vm, + )); + } + } + } + + Ok((vm.ctx.new_list(values).into(), char_idx, byte_idx)) + } + + /// Finalize object construction with hooks. + fn finalize_object( + &self, + pairs: Vec<(PyObjectRef, PyObjectRef)>, + end_char_idx: usize, + end_byte_idx: usize, + vm: &VirtualMachine, + ) -> PyResult<(PyObjectRef, usize, usize)> { + let result = if let Some(ref pairs_hook) = self.object_pairs_hook { + // object_pairs_hook takes priority - pass list of tuples + let pairs_list: Vec<PyObjectRef> = pairs + .into_iter() + .map(|(k, v)| vm.new_tuple((k, v)).into()) + .collect(); + pairs_hook.call((vm.ctx.new_list(pairs_list),), vm)? + } else { + // Build a dict from pairs + let dict = vm.ctx.new_dict(); + for (key, value) in pairs { + dict.set_item(&*key, value, vm)?; + } + + // Apply object_hook if present + let dict_obj: PyObjectRef = dict.into(); + if let Some(ref hook) = self.object_hook { + hook.call((dict_obj,), vm)? + } else { + dict_obj + } + }; + + Ok((result, end_char_idx, end_byte_idx)) + } + + /// Call scan_once and handle the result. + /// Returns (value, end_char_idx, end_byte_idx). + fn call_scan_once( + &self, + scan_once: &PyObjectRef, + pystr: PyStrRef, + char_idx: usize, + byte_idx: usize, + memo: &mut HashMap<String, PyStrRef>, + vm: &VirtualMachine, + ) -> PyResult<(PyObjectRef, usize, usize)> { + let s = pystr.as_str(); + let bytes = s.as_bytes(); + let wtf8 = pystr.as_wtf8(); + + let first_byte = match bytes.get(byte_idx) { + Some(&b) => b, + None => return Err(self.make_decode_error("Expecting value", pystr, char_idx, vm)), + }; + + match first_byte { + b'"' => { + // String - pass slice starting after the quote + let (wtf8_result, chars_consumed, bytes_consumed) = + machinery::scanstring(&wtf8[byte_idx + 1..], char_idx + 1, self.strict) + .map_err(|e| py_decode_error(e, pystr.clone(), vm))?; + let py_str = vm.ctx.new_str(wtf8_result.to_string()); + Ok(( + py_str.into(), + char_idx + 1 + chars_consumed, + byte_idx + 1 + bytes_consumed, + )) + } + b'{' => { + // Object + self.parse_object(pystr, char_idx + 1, byte_idx + 1, scan_once, memo, vm) + } + b'[' => { + // Array + self.parse_array(pystr, char_idx + 1, byte_idx + 1, scan_once, memo, vm) + } + b'n' if starts_with_bytes(&bytes[byte_idx..], b"null") => { + // null + Ok((vm.ctx.none(), char_idx + 4, byte_idx + 4)) + } + b't' if starts_with_bytes(&bytes[byte_idx..], b"true") => { + // true + Ok((vm.ctx.new_bool(true).into(), char_idx + 4, byte_idx + 4)) + } + b'f' if starts_with_bytes(&bytes[byte_idx..], b"false") => { + // false + Ok((vm.ctx.new_bool(false).into(), char_idx + 5, byte_idx + 5)) + } + b'N' if starts_with_bytes(&bytes[byte_idx..], b"NaN") => { + // NaN + let result = self.parse_constant.call(("NaN",), vm)?; + Ok((result, char_idx + 3, byte_idx + 3)) + } + b'I' if starts_with_bytes(&bytes[byte_idx..], b"Infinity") => { + // Infinity + let result = self.parse_constant.call(("Infinity",), vm)?; + Ok((result, char_idx + 8, byte_idx + 8)) + } + b'-' => { + // -Infinity or negative number + if starts_with_bytes(&bytes[byte_idx..], b"-Infinity") { + let result = self.parse_constant.call(("-Infinity",), vm)?; + return Ok((result, char_idx + 9, byte_idx + 9)); + } + // Negative number - numbers are ASCII so len == bytes + if let Some((result, len)) = self.parse_number(&s[byte_idx..], vm) { + return Ok((result?, char_idx + len, byte_idx + len)); + } + Err(self.make_decode_error("Expecting value", pystr, char_idx, vm)) + } + b'0'..=b'9' => { + // Positive number - numbers are ASCII so len == bytes + if let Some((result, len)) = self.parse_number(&s[byte_idx..], vm) { + return Ok((result?, char_idx + len, byte_idx + len)); + } + Err(self.make_decode_error("Expecting value", pystr, char_idx, vm)) + } + _ => { + // Fall back to scan_once for unrecognized input + // Note: This path requires char_idx for Python compatibility + let result = scan_once.call((pystr.clone(), char_idx as isize), vm); + + match result { + Ok(tuple) => { + use crate::vm::builtins::PyTupleRef; + let tuple: PyTupleRef = tuple.try_into_value(vm)?; + if tuple.len() != 2 { + return Err(vm.new_value_error("scan_once must return 2-tuple")); + } + let value = tuple.as_slice()[0].clone(); + let end_char_idx: isize = tuple.as_slice()[1].try_to_value(vm)?; + // For fallback, we need to calculate byte_idx from char_idx + // This is expensive but fallback should be rare + let end_byte_idx = s + .char_indices() + .nth(end_char_idx as usize) + .map(|(i, _)| i) + .unwrap_or(s.len()); + Ok((value, end_char_idx as usize, end_byte_idx)) + } + Err(err) if err.fast_isinstance(vm.ctx.exceptions.stop_iteration) => { + Err(self.make_decode_error("Expecting value", pystr, char_idx, vm)) + } + Err(err) => Err(err), + } + } + } + } + + /// Create a decode error. + fn make_decode_error( + &self, + msg: &str, + s: PyStrRef, + pos: usize, + vm: &VirtualMachine, + ) -> PyBaseExceptionRef { + let err = machinery::DecodeError::new(msg, pos); + py_decode_error(err, s, vm) + } } impl Callable for JsonScanner { type Args = (PyStrRef, isize); - fn call(zelf: &Py<Self>, (pystr, idx): Self::Args, vm: &VirtualMachine) -> PyResult { - if idx < 0 { + fn call(zelf: &Py<Self>, (pystr, char_idx): Self::Args, vm: &VirtualMachine) -> PyResult { + if char_idx < 0 { return Err(vm.new_value_error("idx cannot be negative")); } - let idx = idx as usize; - let mut chars = pystr.as_str().chars(); - if idx > 0 && chars.nth(idx - 1).is_none() { - PyIterReturn::StopIteration(Some(vm.ctx.new_int(idx).into())).to_pyresult(vm) + let char_idx = char_idx as usize; + let s = pystr.as_str(); + + // Calculate byte index from char index (O(char_idx) but only at entry point) + let byte_idx = if char_idx == 0 { + 0 } else { - zelf.parse( - chars.as_str(), - pystr.clone(), - idx, - zelf.to_owned().into(), - vm, - ) - .and_then(|x| x.to_pyresult(vm)) - } + match s.char_indices().nth(char_idx) { + Some((byte_i, _)) => byte_i, + None => { + // char_idx is beyond the string length + return PyIterReturn::StopIteration(Some(vm.ctx.new_int(char_idx).into())) + .to_pyresult(vm); + } + } + }; + + zelf.parse( + pystr.clone(), + char_idx, + byte_idx, + zelf.to_owned().into(), + vm, + ) + .and_then(|x| x.to_pyresult(vm)) } } fn encode_string(s: &str, ascii_only: bool) -> String { + flame_guard!("_json::encode_string"); let mut buf = Vec::<u8>::with_capacity(s.len() + 2); machinery::write_json_string(s, ascii_only, &mut buf) // SAFETY: writing to a vec can't fail @@ -253,7 +705,29 @@ mod _json { strict: OptionalArg<bool>, vm: &VirtualMachine, ) -> PyResult<(Wtf8Buf, usize)> { - machinery::scanstring(s.as_wtf8(), end, strict.unwrap_or(true)) - .map_err(|e| py_decode_error(e, s, vm)) + flame_guard!("_json::scanstring"); + let wtf8 = s.as_wtf8(); + + // Convert char index `end` to byte index + let byte_idx = if end == 0 { + 0 + } else { + wtf8.code_point_indices() + .nth(end) + .map(|(i, _)| i) + .ok_or_else(|| { + py_decode_error( + machinery::DecodeError::new("Unterminated string starting at", end - 1), + s.clone(), + vm, + ) + })? + }; + + let (result, chars_consumed, _bytes_consumed) = + machinery::scanstring(&wtf8[byte_idx..], end, strict.unwrap_or(true)) + .map_err(|e| py_decode_error(e, s, vm))?; + + Ok((result, end + chars_consumed)) } } diff --git a/crates/stdlib/src/json/machinery.rs b/crates/stdlib/src/json/machinery.rs index 57b8ae441f7..f33a135ab20 100644 --- a/crates/stdlib/src/json/machinery.rs +++ b/crates/stdlib/src/json/machinery.rs @@ -30,6 +30,7 @@ use std::io; use itertools::Itertools; +use memchr::memchr2; use rustpython_common::wtf8::{CodePoint, Wtf8, Wtf8Buf}; static ESCAPE_CHARS: [&str; 0x20] = [ @@ -108,7 +109,7 @@ pub struct DecodeError { pub pos: usize, } impl DecodeError { - fn new(msg: impl Into<String>, pos: usize) -> Self { + pub fn new(msg: impl Into<String>, pos: usize) -> Self { let msg = msg.into(); Self { msg, pos } } @@ -126,24 +127,63 @@ impl StrOrChar<'_> { } } } +/// Scan a JSON string starting right after the opening quote. +/// +/// # Arguments +/// * `s` - The string slice starting at the first character after the opening `"` +/// * `char_offset` - The character index where this slice starts (for error messages) +/// * `strict` - Whether to reject control characters +/// +/// # Returns +/// * `Ok((result, chars_consumed, bytes_consumed))` - The decoded string and how much was consumed +/// * `Err(DecodeError)` - If the string is malformed pub fn scanstring<'a>( s: &'a Wtf8, - end: usize, + char_offset: usize, strict: bool, -) -> Result<(Wtf8Buf, usize), DecodeError> { +) -> Result<(Wtf8Buf, usize, usize), DecodeError> { + flame_guard!("machinery::scanstring"); + let unterminated_err = || DecodeError::new("Unterminated string starting at", char_offset - 1); + + let bytes = s.as_bytes(); + + // Fast path: use memchr to find " or \ quickly + if let Some(pos) = memchr2(b'"', b'\\', bytes) + && bytes[pos] == b'"' + { + let content_bytes = &bytes[..pos]; + + // In strict mode, check for control characters (0x00-0x1F) + let has_control_char = strict && content_bytes.iter().any(|&b| b < 0x20); + + if !has_control_char { + flame_guard!("machinery::scanstring::fast_path"); + let result_slice = &s[..pos]; + let char_count = result_slice.code_points().count(); + let mut out = Wtf8Buf::with_capacity(pos); + out.push_wtf8(result_slice); + // +1 for the closing quote + return Ok((out, char_count + 1, pos + 1)); + } + } + + // Slow path: chunk-based parsing for strings with escapes or control chars + flame_guard!("machinery::scanstring::slow_path"); let mut chunks: Vec<StrOrChar<'a>> = Vec::new(); let mut output_len = 0usize; let mut push_chunk = |chunk: StrOrChar<'a>| { output_len += chunk.len(); chunks.push(chunk); }; - let unterminated_err = || DecodeError::new("Unterminated string starting at", end - 1); - let mut chars = s.code_point_indices().enumerate().skip(end).peekable(); - let &(_, (mut chunk_start, _)) = chars.peek().ok_or_else(unterminated_err)?; + + let mut chars = s.code_point_indices().enumerate().peekable(); + let mut chunk_start: usize = 0; + while let Some((char_i, (byte_i, c))) = chars.next() { match c.to_char_lossy() { '"' => { push_chunk(StrOrChar::Str(&s[chunk_start..byte_i])); + flame_guard!("machinery::scanstring::assemble_chunks"); let mut out = Wtf8Buf::with_capacity(output_len); for x in chunks { match x { @@ -151,11 +191,12 @@ pub fn scanstring<'a>( StrOrChar::Char(c) => out.push(c), } } - return Ok((out, char_i + 1)); + // +1 for the closing quote + return Ok((out, char_i + 1, byte_i + 1)); } '\\' => { push_chunk(StrOrChar::Str(&s[chunk_start..byte_i])); - let (_, (_, c)) = chars.next().ok_or_else(unterminated_err)?; + let (next_char_i, (_, c)) = chars.next().ok_or_else(unterminated_err)?; let esc = match c.to_char_lossy() { '"' => "\"", '\\' => "\\", @@ -166,20 +207,21 @@ pub fn scanstring<'a>( 'r' => "\r", 't' => "\t", 'u' => { - let mut uni = decode_unicode(&mut chars, char_i)?; + let mut uni = decode_unicode(&mut chars, char_offset + char_i)?; chunk_start = byte_i + 6; if let Some(lead) = uni.to_lead_surrogate() { // uni is a surrogate -- try to find its pair let mut chars2 = chars.clone(); - if let Some(((pos2, _), (_, _))) = chars2 + if let Some(((_, (byte_pos2, _)), (_, _))) = chars2 .next_tuple() .filter(|((_, (_, c1)), (_, (_, c2)))| *c1 == '\\' && *c2 == 'u') { - let uni2 = decode_unicode(&mut chars2, pos2)?; + let uni2 = + decode_unicode(&mut chars2, char_offset + next_char_i + 1)?; if let Some(trail) = uni2.to_trail_surrogate() { // ok, we found what we were looking for -- \uXXXX\uXXXX, both surrogates uni = lead.merge(trail).into(); - chunk_start = pos2 + 6; + chunk_start = byte_pos2 + 6; chars = chars2; } } @@ -188,7 +230,10 @@ pub fn scanstring<'a>( continue; } _ => { - return Err(DecodeError::new(format!("Invalid \\escape: {c:?}"), char_i)); + return Err(DecodeError::new( + format!("Invalid \\escape: {c:?}"), + char_offset + char_i, + )); } }; chunk_start = byte_i + 2; @@ -197,7 +242,7 @@ pub fn scanstring<'a>( '\x00'..='\x1f' if strict => { return Err(DecodeError::new( format!("Invalid control character {c:?} at"), - char_i, + char_offset + char_i, )); } _ => {} @@ -211,12 +256,13 @@ fn decode_unicode<I>(it: &mut I, pos: usize) -> Result<CodePoint, DecodeError> where I: Iterator<Item = (usize, (usize, CodePoint))>, { + flame_guard!("machinery::decode_unicode"); let err = || DecodeError::new("Invalid \\uXXXX escape", pos); - let mut uni = 0; - for x in (0..4).rev() { + let mut uni = 0u16; + for _ in 0..4 { let (_, (_, c)) = it.next().ok_or_else(err)?; let d = c.to_char().and_then(|c| c.to_digit(16)).ok_or_else(err)? as u16; - uni += d * 16u16.pow(x); + uni = (uni << 4) | d; } Ok(uni.into()) } diff --git a/crates/stdlib/src/lib.rs b/crates/stdlib/src/lib.rs index 6b7796c8bad..02b7407446f 100644 --- a/crates/stdlib/src/lib.rs +++ b/crates/stdlib/src/lib.rs @@ -2,23 +2,24 @@ // how `mod` works, but we want this sometimes for pymodule declarations #![allow(clippy::module_inception)] -#![cfg_attr(all(target_os = "wasi", target_env = "p2"), feature(wasip2))] #[macro_use] extern crate rustpython_derive; extern crate alloc; +#[macro_use] +pub(crate) mod macros; + +mod _asyncio; +mod _remote_debugging; pub mod array; mod binascii; mod bisect; +mod bz2; mod cmath; +mod compression; // internal module mod contextvars; mod csv; -mod dis; -mod gc; - -mod bz2; -mod compression; // internal module #[cfg(not(any(target_os = "android", target_arch = "wasm32")))] mod lzma; mod zlib; @@ -36,10 +37,10 @@ mod json; #[cfg(not(any(target_os = "ios", target_arch = "wasm32")))] mod locale; +mod _opcode; mod math; #[cfg(any(unix, windows))] mod mmap; -mod opcode; mod pyexpat; mod pystruct; mod random; @@ -63,6 +64,11 @@ mod posixshmem; #[cfg(unix)] mod posixsubprocess; // libc is missing constants on redox +#[cfg(all( + feature = "sqlite", + not(any(target_os = "android", target_arch = "wasm32")) +))] +mod _sqlite3; #[cfg(all(unix, not(any(target_os = "android", target_os = "redox"))))] mod grp; #[cfg(windows)] @@ -73,11 +79,6 @@ mod resource; mod scproxy; #[cfg(any(unix, windows, target_os = "wasi"))] mod select; -#[cfg(all( - feature = "sqlite", - not(any(target_os = "android", target_arch = "wasm32")) -))] -mod sqlite; #[cfg(all(not(target_arch = "wasm32"), feature = "ssl-openssl"))] mod openssl; @@ -103,132 +104,88 @@ mod tkinter; use rustpython_common as common; use rustpython_vm as vm; -use crate::vm::{builtins, stdlib::StdlibInitFunc}; -use alloc::borrow::Cow; +use crate::vm::{Context, builtins}; -pub fn get_module_inits() -> impl Iterator<Item = (Cow<'static, str>, StdlibInitFunc)> { - macro_rules! modules { - { - $( - #[cfg($cfg:meta)] - { $( $key:expr => $val:expr),* $(,)? } - )* - } => {{ - [ - $( - $(#[cfg($cfg)] (Cow::<'static, str>::from($key), Box::new($val) as StdlibInitFunc),)* - )* - ] - .into_iter() - }}; - } - modules! { - #[cfg(all())] - { - "array" => array::make_module, - "binascii" => binascii::make_module, - "_bisect" => bisect::make_module, - "_bz2" => bz2::make_module, - "cmath" => cmath::make_module, - "_contextvars" => contextvars::make_module, - "_csv" => csv::make_module, - "_dis" => dis::make_module, - "faulthandler" => faulthandler::make_module, - "gc" => gc::make_module, - "_hashlib" => hashlib::make_module, - "_sha1" => sha1::make_module, - "_sha3" => sha3::make_module, - "_sha256" => sha256::make_module, - "_sha512" => sha512::make_module, - "_md5" => md5::make_module, - "_blake2" => blake2::make_module, - "_json" => json::make_module, - "math" => math::make_module, - "pyexpat" => pyexpat::make_module, - "_opcode" => opcode::make_module, - "_random" => random::make_module, - "_statistics" => statistics::make_module, - "_struct" => pystruct::make_module, - "unicodedata" => unicodedata::make_module, - "zlib" => zlib::make_module, - "_statistics" => statistics::make_module, - "_suggestions" => suggestions::make_module, - // crate::vm::sysmodule::sysconfigdata_name() => sysconfigdata::make_module, - } +/// Returns module definitions for multi-phase init modules. +/// These modules are added to sys.modules BEFORE their exec function runs, +/// allowing safe circular imports. +pub fn stdlib_module_defs(ctx: &Context) -> Vec<&'static builtins::PyModuleDef> { + vec![ + _asyncio::module_def(ctx), + _opcode::module_def(ctx), + _remote_debugging::module_def(ctx), + array::module_def(ctx), + binascii::module_def(ctx), + bisect::module_def(ctx), + blake2::module_def(ctx), + bz2::module_def(ctx), + cmath::module_def(ctx), + contextvars::module_def(ctx), + csv::module_def(ctx), + faulthandler::module_def(ctx), #[cfg(any(unix, target_os = "wasi"))] - { - "fcntl" => fcntl::make_module, - } - #[cfg(any(unix, windows, target_os = "wasi"))] - { - "select" => select::make_module, - } - #[cfg(not(target_arch = "wasm32"))] - { - "_multiprocessing" => multiprocessing::make_module, - "_socket" => socket::make_module, - } + fcntl::module_def(ctx), + #[cfg(all(unix, not(any(target_os = "android", target_os = "redox"))))] + grp::module_def(ctx), + hashlib::module_def(ctx), + json::module_def(ctx), + #[cfg(not(any(target_os = "ios", target_arch = "wasm32")))] + locale::module_def(ctx), #[cfg(not(any(target_os = "android", target_arch = "wasm32")))] - { - "_lzma" => lzma::make_module, - } - #[cfg(all(feature = "sqlite", not(any(target_os = "android", target_arch = "wasm32"))))] - { - "_sqlite3" => sqlite::make_module, - } - #[cfg(all(not(target_arch = "wasm32"), feature = "ssl-rustls"))] - { - "_ssl" => ssl::make_module, - } + lzma::module_def(ctx), + math::module_def(ctx), + md5::module_def(ctx), + #[cfg(any(unix, windows))] + mmap::module_def(ctx), + #[cfg(not(target_arch = "wasm32"))] + multiprocessing::module_def(ctx), #[cfg(all(not(target_arch = "wasm32"), feature = "ssl-openssl"))] - { - "_ssl" => openssl::make_module, - } + openssl::module_def(ctx), #[cfg(windows)] - { - "_overlapped" => overlapped::make_module, - } - // Unix-only + overlapped::module_def(ctx), #[cfg(unix)] - { - "_posixsubprocess" => posixsubprocess::make_module, - } + posixsubprocess::module_def(ctx), #[cfg(all(unix, not(target_os = "redox"), not(target_os = "android")))] - { - "_posixshmem" => posixshmem::make_module, - } - #[cfg(any(unix, windows))] - { - "mmap" => mmap::make_module, - } + posixshmem::module_def(ctx), + pyexpat::module_def(ctx), + pystruct::module_def(ctx), + random::module_def(ctx), #[cfg(all(unix, not(target_os = "redox")))] - { - "syslog" => syslog::make_module, - "resource" => resource::make_module, - } - #[cfg(all(unix, not(any(target_os = "ios", target_os = "redox"))))] - { - "termios" => termios::make_module, - } - #[cfg(all(unix, not(any(target_os = "android", target_os = "redox"))))] - { - "grp" => grp::make_module, - } + resource::module_def(ctx), #[cfg(target_os = "macos")] - { - "_scproxy" => scproxy::make_module, - } - #[cfg(not(any(target_os = "android", target_os = "ios", target_os = "windows", target_arch = "wasm32", target_os = "redox")))] - { - "_uuid" => uuid::make_module, - } - #[cfg(not(any(target_os = "ios", target_arch = "wasm32")))] - { - "_locale" => locale::make_module, - } + scproxy::module_def(ctx), + #[cfg(any(unix, windows, target_os = "wasi"))] + select::module_def(ctx), + sha1::module_def(ctx), + sha256::module_def(ctx), + sha3::module_def(ctx), + sha512::module_def(ctx), + #[cfg(not(target_arch = "wasm32"))] + socket::module_def(ctx), + #[cfg(all( + feature = "sqlite", + not(any(target_os = "android", target_arch = "wasm32")) + ))] + _sqlite3::module_def(ctx), + #[cfg(all(not(target_arch = "wasm32"), feature = "ssl-rustls"))] + ssl::module_def(ctx), + statistics::module_def(ctx), + suggestions::module_def(ctx), + #[cfg(all(unix, not(target_os = "redox")))] + syslog::module_def(ctx), + #[cfg(all(unix, not(any(target_os = "ios", target_os = "redox"))))] + termios::module_def(ctx), #[cfg(feature = "tkinter")] - { - "_tkinter" => tkinter::make_module, - } - } + tkinter::module_def(ctx), + unicodedata::module_def(ctx), + #[cfg(not(any( + target_os = "android", + target_os = "ios", + target_os = "windows", + target_arch = "wasm32", + target_os = "redox" + )))] + uuid::module_def(ctx), + zlib::module_def(ctx), + ] } diff --git a/crates/stdlib/src/locale.rs b/crates/stdlib/src/locale.rs index 51cf352590b..496325b5038 100644 --- a/crates/stdlib/src/locale.rs +++ b/crates/stdlib/src/locale.rs @@ -1,6 +1,6 @@ // spell-checker:ignore abday abmon yesexpr noexpr CRNCYSTR RADIXCHAR AMPM THOUSEP -pub(crate) use _locale::make_module; +pub(crate) use _locale::module_def; #[cfg(windows)] #[repr(C)] @@ -101,10 +101,42 @@ mod _locale { unsafe fn pystr_from_raw_cstr(vm: &VirtualMachine, raw_ptr: *const libc::c_char) -> PyResult { let slice = unsafe { CStr::from_ptr(raw_ptr) }; - let string = slice - .to_str() - .expect("localeconv always return decodable string"); - Ok(vm.new_pyobj(string)) + + // Fast path: ASCII/UTF-8 + if let Ok(s) = slice.to_str() { + return Ok(vm.new_pyobj(s)); + } + + // On Windows, locale strings use the ANSI code page encoding + #[cfg(windows)] + { + use windows_sys::Win32::Globalization::{CP_ACP, MultiByteToWideChar}; + let bytes = slice.to_bytes(); + unsafe { + let len = MultiByteToWideChar( + CP_ACP, + 0, + bytes.as_ptr(), + bytes.len() as i32, + ptr::null_mut(), + 0, + ); + if len > 0 { + let mut wide = vec![0u16; len as usize]; + MultiByteToWideChar( + CP_ACP, + 0, + bytes.as_ptr(), + bytes.len() as i32, + wide.as_mut_ptr(), + len, + ); + return Ok(vm.new_pyobj(String::from_utf16_lossy(&wide))); + } + } + } + + Ok(vm.new_pyobj(String::from_utf8_lossy(slice.to_bytes()).into_owned())) } #[pyattr(name = "Error", once)] diff --git a/crates/stdlib/src/lzma.rs b/crates/stdlib/src/lzma.rs index b18ac3ee69a..0189c9d2ed0 100644 --- a/crates/stdlib/src/lzma.rs +++ b/crates/stdlib/src/lzma.rs @@ -1,6 +1,6 @@ // spell-checker:ignore ARMTHUMB -pub(crate) use _lzma::make_module; +pub(crate) use _lzma::module_def; #[pymodule] mod _lzma { diff --git a/crates/stdlib/src/macros.rs b/crates/stdlib/src/macros.rs new file mode 100644 index 00000000000..385f4b1c4ab --- /dev/null +++ b/crates/stdlib/src/macros.rs @@ -0,0 +1,7 @@ +#[macro_export] +macro_rules! flame_guard { + ($name:expr) => { + #[cfg(feature = "flame-it")] + let _guard = ::flame::start_guard($name); + }; +} diff --git a/crates/stdlib/src/math.rs b/crates/stdlib/src/math.rs index fb8945c74f7..238c7fd4928 100644 --- a/crates/stdlib/src/math.rs +++ b/crates/stdlib/src/math.rs @@ -1,70 +1,48 @@ -pub(crate) use math::make_module; +pub(crate) use math::module_def; -use crate::{builtins::PyBaseExceptionRef, vm::VirtualMachine}; +use crate::vm::{VirtualMachine, builtins::PyBaseExceptionRef}; #[pymodule] mod math { use crate::vm::{ - PyObject, PyObjectRef, PyRef, PyResult, VirtualMachine, + AsObject, PyObject, PyObjectRef, PyRef, PyResult, VirtualMachine, builtins::{PyFloat, PyInt, PyIntRef, PyStrInterned, try_bigint_to_f64, try_f64_to_bigint}, function::{ArgIndex, ArgIntoFloat, ArgIterable, Either, OptionalArg, PosArgs}, identifier, }; - use core::cmp::Ordering; - use itertools::Itertools; use malachite_bigint::BigInt; - use num_traits::{One, Signed, ToPrimitive, Zero}; - use rustpython_common::{float_ops, int::true_div}; + use num_traits::{Signed, ToPrimitive}; + + use super::{float_repr, pymath_exception}; // Constants #[pyattr] use core::f64::consts::{E as e, PI as pi, TAU as tau}; - use super::pymath_error_to_exception; #[pyattr(name = "inf")] const INF: f64 = f64::INFINITY; #[pyattr(name = "nan")] const NAN: f64 = f64::NAN; - // Helper macro: - macro_rules! call_math_func { - ( $fun:ident, $name:ident, $vm:ident ) => {{ - let value = $name.into_float(); - let result = value.$fun(); - result_or_overflow(value, result, $vm) - }}; - } - - #[inline] - fn result_or_overflow(value: f64, result: f64, vm: &VirtualMachine) -> PyResult<f64> { - if !result.is_finite() && value.is_finite() { - // CPython doesn't return `inf` when called with finite - // values, it raises OverflowError instead. - Err(vm.new_overflow_error("math range error")) - } else { - Ok(result) - } - } - // Number theory functions: #[pyfunction] fn fabs(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - call_math_func!(abs, x, vm) + pymath::math::fabs(x.into_float()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] fn isfinite(x: ArgIntoFloat) -> bool { - x.into_float().is_finite() + pymath::math::isfinite(x.into_float()) } #[pyfunction] fn isinf(x: ArgIntoFloat) -> bool { - x.into_float().is_infinite() + pymath::math::isinf(x.into_float()) } #[pyfunction] fn isnan(x: ArgIntoFloat) -> bool { - x.into_float().is_nan() + pymath::math::isnan(x.into_float()) } #[derive(FromArgs)] @@ -83,420 +61,286 @@ mod math { fn isclose(args: IsCloseArgs, vm: &VirtualMachine) -> PyResult<bool> { let a = args.a.into_float(); let b = args.b.into_float(); - let rel_tol = args.rel_tol.map_or(1e-09, |v| v.into_float()); - let abs_tol = args.abs_tol.map_or(0.0, |v| v.into_float()); - - if rel_tol < 0.0 || abs_tol < 0.0 { - return Err(vm.new_value_error("tolerances must be non-negative")); - } - - if a == b { - /* short circuit exact equality -- needed to catch two infinities of - the same sign. And perhaps speeds things up a bit sometimes. - */ - return Ok(true); - } + let rel_tol = args.rel_tol.into_option().map(|v| v.into_float()); + let abs_tol = args.abs_tol.into_option().map(|v| v.into_float()); - /* This catches the case of two infinities of opposite sign, or - one infinity and one finite number. Two infinities of opposite - sign would otherwise have an infinite relative tolerance. - Two infinities of the same sign are caught by the equality check - above. - */ - - if a.is_infinite() || b.is_infinite() { - return Ok(false); - } - - let diff = (b - a).abs(); - - Ok((diff <= (rel_tol * b).abs()) || (diff <= (rel_tol * a).abs()) || (diff <= abs_tol)) + pymath::math::isclose(a, b, rel_tol, abs_tol) + .map_err(|_| vm.new_value_error("tolerances must be non-negative")) } #[pyfunction] - fn copysign(x: ArgIntoFloat, y: ArgIntoFloat) -> f64 { - x.into_float().copysign(y.into_float()) + fn copysign(x: ArgIntoFloat, y: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { + pymath::math::copysign(x.into_float(), y.into_float()) + .map_err(|err| pymath_exception(err, vm)) } // Power and logarithmic functions: #[pyfunction] fn exp(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - call_math_func!(exp, x, vm) + pymath::math::exp(x.into_float()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] fn exp2(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - call_math_func!(exp2, x, vm) + pymath::math::exp2(x.into_float()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] fn expm1(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - call_math_func!(exp_m1, x, vm) + pymath::math::expm1(x.into_float()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] fn log(x: PyObjectRef, base: OptionalArg<ArgIntoFloat>, vm: &VirtualMachine) -> PyResult<f64> { - let base: f64 = base.map(Into::into).unwrap_or(core::f64::consts::E); - if base.is_sign_negative() { - return Err(vm.new_value_error("math domain error")); + let base = base.into_option().map(|v| v.into_float()); + // Check base first for proper error messages + if let Some(b) = base { + if b <= 0.0 { + return Err(vm.new_value_error(format!( + "expected a positive input, got {}", + super::float_repr(b) + ))); + } + if b == 1.0 { + return Err(vm.new_value_error("math domain error".to_owned())); + } + } + // Handle BigInt specially for large values (only for actual int type, not float) + if let Some(i) = x.downcast_ref::<PyInt>() { + return pymath::math::log_bigint(i.as_bigint(), base).map_err(|err| match err { + pymath::Error::EDOM => vm.new_value_error("expected a positive input".to_owned()), + _ => pymath_exception(err, vm), + }); } - log2(x, vm).map(|log_x| log_x / base.log2()) + let val = x.try_float(vm)?.to_f64(); + pymath::math::log(val, base).map_err(|err| match err { + pymath::Error::EDOM => vm.new_value_error(format!( + "expected a positive input, got {}", + super::float_repr(val) + )), + _ => pymath_exception(err, vm), + }) } #[pyfunction] fn log1p(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - let x = x.into_float(); - if x.is_nan() || x > -1.0_f64 { - Ok(x.ln_1p()) - } else { - Err(vm.new_value_error("math domain error")) - } - } - - /// Generates the base-2 logarithm of a BigInt `x` - fn int_log2(x: &BigInt) -> f64 { - // log2(x) = log2(2^n * 2^-n * x) = n + log2(x/2^n) - // If we set 2^n to be the greatest power of 2 below x, then x/2^n is in [1, 2), and can - // thus be converted into a float. - let n = x.bits() as u32 - 1; - let frac = true_div(x, &BigInt::from(2).pow(n)); - f64::from(n) + frac.log2() + pymath::math::log1p(x.into_float()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] fn log2(x: PyObjectRef, vm: &VirtualMachine) -> PyResult<f64> { - match x.try_float(vm) { - Ok(x) => { - let x = x.to_f64(); - if x.is_nan() || x > 0.0_f64 { - Ok(x.log2()) - } else { - Err(vm.new_value_error("math domain error")) - } - } - Err(float_err) => { - if let Ok(x) = x.try_int(vm) { - let x = x.as_bigint(); - if x.is_positive() { - Ok(int_log2(x)) - } else { - Err(vm.new_value_error("math domain error")) - } - } else { - // Return the float error, as it will be more intuitive to users - Err(float_err) - } - } + // Handle BigInt specially for large values (only for actual int type, not float) + if let Some(i) = x.downcast_ref::<PyInt>() { + return pymath::math::log2_bigint(i.as_bigint()).map_err(|err| match err { + pymath::Error::EDOM => vm.new_value_error("expected a positive input".to_owned()), + _ => pymath_exception(err, vm), + }); } + let val = x.try_float(vm)?.to_f64(); + pymath::math::log2(val).map_err(|err| match err { + pymath::Error::EDOM => vm.new_value_error(format!( + "expected a positive input, got {}", + super::float_repr(val) + )), + _ => pymath_exception(err, vm), + }) } #[pyfunction] fn log10(x: PyObjectRef, vm: &VirtualMachine) -> PyResult<f64> { - log2(x, vm).map(|log_x| log_x / 10f64.log2()) - } - - #[pyfunction] - fn pow(x: ArgIntoFloat, y: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - let x = x.into_float(); - let y = y.into_float(); - - if x < 0.0 && x.is_finite() && y.fract() != 0.0 && y.is_finite() - || x == 0.0 && y < 0.0 && y != f64::NEG_INFINITY - { - return Err(vm.new_value_error("math domain error")); - } - - let value = x.powf(y); - - if x.is_finite() && y.is_finite() && value.is_infinite() { - return Err(vm.new_overflow_error("math range error")); + // Handle BigInt specially for large values (only for actual int type, not float) + if let Some(i) = x.downcast_ref::<PyInt>() { + return pymath::math::log10_bigint(i.as_bigint()).map_err(|err| match err { + pymath::Error::EDOM => vm.new_value_error("expected a positive input".to_owned()), + _ => pymath_exception(err, vm), + }); } - - Ok(value) + let val = x.try_float(vm)?.to_f64(); + pymath::math::log10(val).map_err(|err| match err { + pymath::Error::EDOM => vm.new_value_error(format!( + "expected a positive input, got {}", + super::float_repr(val) + )), + _ => pymath_exception(err, vm), + }) } #[pyfunction] - fn sqrt(value: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - let value = value.into_float(); - if value.is_nan() { - return Ok(value); - } - if value.is_sign_negative() { - if value.is_zero() { - return Ok(-0.0f64); - } - return Err(vm.new_value_error("math domain error")); - } - Ok(value.sqrt()) + fn pow(x: ArgIntoFloat, y: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { + pymath::math::pow(x.into_float(), y.into_float()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn isqrt(x: ArgIndex, vm: &VirtualMachine) -> PyResult<BigInt> { - let x = x.into_int_ref(); - let value = x.as_bigint(); - - if value.is_negative() { - return Err(vm.new_value_error("isqrt() argument must be nonnegative")); - } - Ok(value.sqrt()) + fn sqrt(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { + let val = x.into_float(); + pymath::math::sqrt(val).map_err(|err| match err { + pymath::Error::EDOM => vm.new_value_error(format!( + "expected a nonnegative input, got {}", + super::float_repr(val) + )), + _ => pymath_exception(err, vm), + }) } // Trigonometric functions: #[pyfunction] fn acos(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - let x = x.into_float(); - if x.is_nan() || (-1.0_f64..=1.0_f64).contains(&x) { - Ok(x.acos()) - } else { - Err(vm.new_value_error("math domain error")) - } + let val = x.into_float(); + pymath::math::acos(val).map_err(|err| match err { + pymath::Error::EDOM => vm.new_value_error(format!( + "expected a number in range from -1 up to 1, got {}", + float_repr(val) + )), + _ => pymath_exception(err, vm), + }) } #[pyfunction] fn asin(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - let x = x.into_float(); - if x.is_nan() || (-1.0_f64..=1.0_f64).contains(&x) { - Ok(x.asin()) - } else { - Err(vm.new_value_error("math domain error")) - } + let val = x.into_float(); + pymath::math::asin(val).map_err(|err| match err { + pymath::Error::EDOM => vm.new_value_error(format!( + "expected a number in range from -1 up to 1, got {}", + float_repr(val) + )), + _ => pymath_exception(err, vm), + }) } #[pyfunction] fn atan(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - call_math_func!(atan, x, vm) + pymath::math::atan(x.into_float()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn atan2(y: ArgIntoFloat, x: ArgIntoFloat) -> f64 { - y.into_float().atan2(x.into()) + fn atan2(y: ArgIntoFloat, x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { + pymath::math::atan2(y.into_float(), x.into_float()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] fn cos(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - let x = x.into_float(); - if x.is_infinite() { - return Err(vm.new_value_error("math domain error")); - } - result_or_overflow(x, x.cos(), vm) + let val = x.into_float(); + pymath::math::cos(val).map_err(|err| match err { + pymath::Error::EDOM => { + vm.new_value_error(format!("expected a finite input, got {}", float_repr(val))) + } + _ => pymath_exception(err, vm), + }) } #[pyfunction] fn hypot(coordinates: PosArgs<ArgIntoFloat>) -> f64 { - let mut coordinates = ArgIntoFloat::vec_into_f64(coordinates.into_vec()); - let mut max = 0.0; - let mut has_nan = false; - for f in &mut coordinates { - *f = f.abs(); - if f.is_nan() { - has_nan = true; - } else if *f > max { - max = *f - } - } - // inf takes precedence over nan - if max.is_infinite() { - return max; - } - if has_nan { - return f64::NAN; - } - coordinates.sort_unstable_by(|x, y| x.total_cmp(y).reverse()); - vector_norm(&coordinates) - } - - /// Implementation of accurate hypotenuse algorithm from Borges 2019. - /// See https://arxiv.org/abs/1904.09481. - /// This assumes that its arguments are positive finite and have been scaled to avoid overflow - /// and underflow. - fn accurate_hypot(max: f64, min: f64) -> f64 { - if min <= max * (f64::EPSILON / 2.0).sqrt() { - return max; - } - let hypot = max.mul_add(max, min * min).sqrt(); - let hypot_sq = hypot * hypot; - let max_sq = max * max; - let correction = (-min).mul_add(min, hypot_sq - max_sq) + hypot.mul_add(hypot, -hypot_sq) - - max.mul_add(max, -max_sq); - hypot - correction / (2.0 * hypot) - } - - /// Calculates the norm of the vector given by `v`. - /// `v` is assumed to be a list of non-negative finite floats, sorted in descending order. - fn vector_norm(v: &[f64]) -> f64 { - // Drop zeros from the vector. - let zero_count = v.iter().rev().cloned().take_while(|x| *x == 0.0).count(); - let v = &v[..v.len() - zero_count]; - if v.is_empty() { - return 0.0; - } - if v.len() == 1 { - return v[0]; - } - // Calculate scaling to avoid overflow / underflow. - let max = *v.first().unwrap(); - let min = *v.last().unwrap(); - let scale = if max > (f64::MAX / v.len() as f64).sqrt() { - max - } else if min < f64::MIN_POSITIVE.sqrt() { - // ^ This can be an `else if`, because if the max is near f64::MAX and the min is near - // f64::MIN_POSITIVE, then the min is relatively unimportant and will be effectively - // ignored. - min - } else { - 1.0 - }; - let mut norm = v - .iter() - .copied() - .map(|x| x / scale) - .reduce(accurate_hypot) - .unwrap_or_default(); - if v.len() > 2 { - // For larger lists of numbers, we can accumulate a rounding error, so a correction is - // needed, similar to that in `accurate_hypot()`. - // First, we estimate [sum of squares - norm^2], then we add the first-order - // approximation of the square root of that to `norm`. - let correction = v - .iter() - .copied() - .map(|x| (x / scale).powi(2)) - .chain(core::iter::once(-norm * norm)) - // Pairwise summation of floats gives less rounding error than a naive sum. - .tree_reduce(core::ops::Add::add) - .expect("expected at least 1 element"); - norm = norm + correction / (2.0 * norm); - } - norm * scale + let coords = ArgIntoFloat::vec_into_f64(coordinates.into_vec()); + pymath::math::hypot(&coords) } #[pyfunction] fn dist(p: Vec<ArgIntoFloat>, q: Vec<ArgIntoFloat>, vm: &VirtualMachine) -> PyResult<f64> { - let mut max = 0.0; - let mut has_nan = false; - let p = ArgIntoFloat::vec_into_f64(p); let q = ArgIntoFloat::vec_into_f64(q); - let mut diffs = vec![]; - if p.len() != q.len() { return Err(vm.new_value_error("both points must have the same number of dimensions")); } - - for i in 0..p.len() { - let px = p[i]; - let qx = q[i]; - - let x = (px - qx).abs(); - if x.is_nan() { - has_nan = true; - } - - diffs.push(x); - if x > max { - max = x; - } - } - - if max.is_infinite() { - return Ok(max); - } - if has_nan { - return Ok(f64::NAN); - } - diffs.sort_unstable_by(|x, y| x.total_cmp(y).reverse()); - Ok(vector_norm(&diffs)) + Ok(pymath::math::dist(&p, &q)) } #[pyfunction] fn sin(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - let x = x.into_float(); - if x.is_infinite() { - return Err(vm.new_value_error("math domain error")); - } - result_or_overflow(x, x.sin(), vm) + let val = x.into_float(); + pymath::math::sin(val).map_err(|err| match err { + pymath::Error::EDOM => { + vm.new_value_error(format!("expected a finite input, got {}", float_repr(val))) + } + _ => pymath_exception(err, vm), + }) } #[pyfunction] fn tan(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - let x = x.into_float(); - if x.is_infinite() { - return Err(vm.new_value_error("math domain error")); - } - result_or_overflow(x, x.tan(), vm) + let val = x.into_float(); + pymath::math::tan(val).map_err(|err| match err { + pymath::Error::EDOM => { + vm.new_value_error(format!("expected a finite input, got {}", float_repr(val))) + } + _ => pymath_exception(err, vm), + }) } #[pyfunction] fn degrees(x: ArgIntoFloat) -> f64 { - x.into_float() * (180.0 / core::f64::consts::PI) + pymath::math::degrees(x.into_float()) } #[pyfunction] fn radians(x: ArgIntoFloat) -> f64 { - x.into_float() * (core::f64::consts::PI / 180.0) + pymath::math::radians(x.into_float()) } // Hyperbolic functions: #[pyfunction] fn acosh(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - let x = x.into_float(); - if x.is_sign_negative() || x.is_zero() { - Err(vm.new_value_error("math domain error")) - } else { - Ok(x.acosh()) - } + pymath::math::acosh(x.into_float()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] fn asinh(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - call_math_func!(asinh, x, vm) + pymath::math::asinh(x.into_float()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] fn atanh(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - let x = x.into_float(); - if x >= 1.0_f64 || x <= -1.0_f64 { - Err(vm.new_value_error("math domain error")) - } else { - Ok(x.atanh()) - } + let val = x.into_float(); + pymath::math::atanh(val).map_err(|err| match err { + pymath::Error::EDOM => vm.new_value_error(format!( + "expected a number between -1 and 1, got {}", + super::float_repr(val) + )), + _ => pymath_exception(err, vm), + }) } #[pyfunction] fn cosh(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - call_math_func!(cosh, x, vm) + pymath::math::cosh(x.into_float()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] fn sinh(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - call_math_func!(sinh, x, vm) + pymath::math::sinh(x.into_float()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] fn tanh(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - call_math_func!(tanh, x, vm) + pymath::math::tanh(x.into_float()).map_err(|err| pymath_exception(err, vm)) } // Special functions: #[pyfunction] - fn erf(x: ArgIntoFloat) -> f64 { - pymath::erf(x.into()) + fn erf(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { + pymath::math::erf(x.into_float()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn erfc(x: ArgIntoFloat) -> f64 { - pymath::erfc(x.into()) + fn erfc(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { + pymath::math::erfc(x.into_float()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] fn gamma(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - pymath::gamma(x.into()).map_err(|err| pymath_error_to_exception(err, vm)) + let val = x.into_float(); + pymath::math::gamma(val).map_err(|err| match err { + pymath::Error::EDOM => vm.new_value_error(format!( + "expected a noninteger or positive integer, got {}", + super::float_repr(val) + )), + _ => pymath_exception(err, vm), + }) } #[pyfunction] fn lgamma(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - pymath::lgamma(x.into()).map_err(|err| pymath_error_to_exception(err, vm)) + pymath::math::lgamma(x.into_float()).map_err(|err| pymath_exception(err, vm)) } fn try_magic_method( @@ -521,37 +365,43 @@ mod math { #[pyfunction] fn ceil(x: PyObjectRef, vm: &VirtualMachine) -> PyResult { - let result_or_err = try_magic_method(identifier!(vm, __ceil__), vm, &x); - if result_or_err.is_err() - && let Some(v) = x.try_float_opt(vm) - { + // Only call __ceil__ if the class defines it - if it exists but is not callable, + // the error should be propagated (not fall back to float conversion) + if x.class().has_attr(identifier!(vm, __ceil__)) { + return try_magic_method(identifier!(vm, __ceil__), vm, &x); + } + // __ceil__ not defined - fall back to float conversion + if let Some(v) = x.try_float_opt(vm) { let v = try_f64_to_bigint(v?.to_f64().ceil(), vm)?; return Ok(vm.ctx.new_int(v).into()); } - result_or_err + Err(vm.new_type_error(format!( + "type '{}' doesn't define '__ceil__' method", + x.class().name(), + ))) } #[pyfunction] fn floor(x: PyObjectRef, vm: &VirtualMachine) -> PyResult { - let result_or_err = try_magic_method(identifier!(vm, __floor__), vm, &x); - if result_or_err.is_err() - && let Some(v) = x.try_float_opt(vm) - { + // Only call __floor__ if the class defines it - if it exists but is not callable, + // the error should be propagated (not fall back to float conversion) + if x.class().has_attr(identifier!(vm, __floor__)) { + return try_magic_method(identifier!(vm, __floor__), vm, &x); + } + // __floor__ not defined - fall back to float conversion + if let Some(v) = x.try_float_opt(vm) { let v = try_f64_to_bigint(v?.to_f64().floor(), vm)?; return Ok(vm.ctx.new_int(v).into()); } - result_or_err + Err(vm.new_type_error(format!( + "type '{}' doesn't define '__floor__' method", + x.class().name(), + ))) } #[pyfunction] fn frexp(x: ArgIntoFloat) -> (f64, i32) { - let value: f64 = x.into(); - if value.is_finite() { - let (m, exp) = float_ops::decompose_float(value); - (m * value.signum(), exp) - } else { - (value, 0) - } + pymath::math::frexp(x.into_float()) } #[pyfunction] @@ -564,315 +414,24 @@ mod math { Either::A(f) => f.to_f64(), Either::B(z) => try_bigint_to_f64(z.as_bigint(), vm)?, }; - - if value == 0_f64 || !value.is_finite() { - // NaNs, zeros and infinities are returned unchanged - return Ok(value); - } - - // Using IEEE 754 bit manipulation to handle large exponents correctly. - // Direct multiplication would overflow for large i values, especially when computing - // the largest finite float (i=1024, x<1.0). By directly modifying the exponent bits, - // we avoid intermediate overflow to infinity. - - // Scale subnormals to normal range first, then adjust exponent. - let (mant, exp0) = if value.abs() < f64::MIN_POSITIVE { - let scaled = value * (1u64 << 54) as f64; // multiply by 2^54 - let (mant_scaled, exp_scaled) = float_ops::decompose_float(scaled); - (mant_scaled, exp_scaled - 54) // adjust exponent back - } else { - float_ops::decompose_float(value) - }; - - let i_big = i.as_bigint(); - let overflow_bound = BigInt::from(1024_i32 - exp0); // i > 1024 - exp0 => overflow - if i_big > &overflow_bound { - return Err(vm.new_overflow_error("math range error")); - } - if i_big == &overflow_bound && mant == 1.0 { - return Err(vm.new_overflow_error("math range error")); - } - let underflow_bound = BigInt::from(-1074_i32 - exp0); // i < -1074 - exp0 => 0.0 with sign - if i_big < &underflow_bound { - return Ok(0.0f64.copysign(value)); - } - - let i_small: i32 = i_big - .to_i32() - .expect("exponent within [-1074-exp0, 1024-exp0] must fit in i32"); - let exp = exp0 + i_small; - - const SIGN_MASK: u64 = 0x8000_0000_0000_0000; - const FRAC_MASK: u64 = 0x000F_FFFF_FFFF_FFFF; - let sign_bit: u64 = if value.is_sign_negative() { - SIGN_MASK - } else { - 0 - }; - let mant_bits = mant.to_bits() & FRAC_MASK; - if exp >= -1021 { - let e_bits = (1022_i32 + exp) as u64; - let result_bits = sign_bit | (e_bits << 52) | mant_bits; - return Ok(f64::from_bits(result_bits)); - } - - let full_mant: u64 = (1u64 << 52) | mant_bits; - let shift: u32 = (-exp - 1021) as u32; - let frac_shifted = full_mant >> shift; - let lost_bits = full_mant & ((1u64 << shift) - 1); - - let half = 1u64 << (shift - 1); - let frac = if (lost_bits > half) || (lost_bits == half && (frac_shifted & 1) == 1) { - frac_shifted + 1 - } else { - frac_shifted - }; - - let result_bits = if frac >= (1u64 << 52) { - sign_bit | (1u64 << 52) - } else { - sign_bit | frac - }; - Ok(f64::from_bits(result_bits)) - } - - fn math_perf_arb_len_int_op<F>(args: PosArgs<ArgIndex>, op: F, default: BigInt) -> BigInt - where - F: Fn(&BigInt, &PyInt) -> BigInt, - { - let arg_vec = args.into_vec(); - - if arg_vec.is_empty() { - return default; - } else if arg_vec.len() == 1 { - return op(arg_vec[0].as_ref().as_bigint(), arg_vec[0].as_ref()); - } - - let mut res = arg_vec[0].as_ref().as_bigint().clone(); - for num in &arg_vec[1..] { - res = op(&res, num.as_ref()) - } - res + pymath::math::ldexp_bigint(value, i.as_bigint()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] - fn gcd(args: PosArgs<ArgIndex>) -> BigInt { - use num_integer::Integer; - math_perf_arb_len_int_op(args, |x, y| x.gcd(y.as_bigint()), BigInt::zero()) - } - - #[pyfunction] - fn lcm(args: PosArgs<ArgIndex>) -> BigInt { - use num_integer::Integer; - math_perf_arb_len_int_op(args, |x, y| x.lcm(y.as_bigint()), BigInt::one()) - } - - #[pyfunction] - fn cbrt(x: ArgIntoFloat) -> f64 { - x.into_float().cbrt() + fn cbrt(x: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { + pymath::math::cbrt(x.into_float()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] fn fsum(seq: ArgIterable<ArgIntoFloat>, vm: &VirtualMachine) -> PyResult<f64> { - let mut partials = Vec::with_capacity(32); - let mut special_sum = 0.0; - let mut inf_sum = 0.0; - - for obj in seq.iter(vm)? { - let mut x = obj?.into_float(); - - let xsave = x; - let mut i = 0; - // This inner loop applies `hi`/`lo` summation to each - // partial so that the list of partial sums remains exact. - for j in 0..partials.len() { - let mut y: f64 = partials[j]; - if x.abs() < y.abs() { - core::mem::swap(&mut x, &mut y); - } - // Rounded `x+y` is stored in `hi` with round-off stored in - // `lo`. Together `hi+lo` are exactly equal to `x+y`. - let hi = x + y; - let lo = y - (hi - x); - if lo != 0.0 { - partials[i] = lo; - i += 1; - } - x = hi; - } - - partials.truncate(i); - if x != 0.0 { - if !x.is_finite() { - // a non-finite x could arise either as - // a result of intermediate overflow, or - // as a result of a nan or inf in the - // summands - if xsave.is_finite() { - return Err(vm.new_overflow_error("intermediate overflow in fsum")); - } - if xsave.is_infinite() { - inf_sum += xsave; - } - special_sum += xsave; - // reset partials - partials.clear(); - } else { - partials.push(x); - } - } - } - if special_sum != 0.0 { - return if inf_sum.is_nan() { - Err(vm.new_value_error("-inf + inf in fsum")) - } else { - Ok(special_sum) - }; - } - - let mut n = partials.len(); - if n > 0 { - n -= 1; - let mut hi = partials[n]; - - let mut lo = 0.0; - while n > 0 { - let x = hi; - - n -= 1; - let y = partials[n]; - - hi = x + y; - lo = y - (hi - x); - if lo != 0.0 { - break; - } - } - if n > 0 && ((lo < 0.0 && partials[n - 1] < 0.0) || (lo > 0.0 && partials[n - 1] > 0.0)) - { - let y = lo + lo; - let x = hi + y; - - // Make half-even rounding work across multiple partials. - // Needed so that sum([1e-16, 1, 1e16]) will round-up the last - // digit to two instead of down to zero (the 1e-16 makes the 1 - // slightly closer to two). With a potential 1 ULP rounding - // error fixed-up, math.fsum() can guarantee commutativity. - if y == x - hi { - hi = x; - } - } - - Ok(hi) - } else { - Ok(0.0) - } - } - - #[pyfunction] - fn factorial(x: PyIntRef, vm: &VirtualMachine) -> PyResult<BigInt> { - let value = x.as_bigint(); - let one = BigInt::one(); - if value.is_negative() { - return Err(vm.new_value_error("factorial() not defined for negative values")); - } else if *value <= one { - return Ok(one); - } - // start from 2, since we know that value > 1 and 1*2=2 - let mut current = one + 1; - let mut product = BigInt::from(2u8); - while current < *value { - current += 1; - product *= &current; - } - Ok(product) - } - - #[pyfunction] - fn perm( - n: ArgIndex, - k: OptionalArg<Option<ArgIndex>>, - vm: &VirtualMachine, - ) -> PyResult<BigInt> { - let n = n.into_int_ref(); - let n = n.as_bigint(); - let k_ref; - let v = match k.flatten() { - Some(k) => { - k_ref = k.into_int_ref(); - k_ref.as_bigint() - } - None => n, - }; - - if n.is_negative() || v.is_negative() { - return Err(vm.new_value_error("perm() not defined for negative values")); - } - if v > n { - return Ok(BigInt::zero()); - } - let mut result = BigInt::one(); - let mut current = n.clone(); - let tmp = n - v; - while current > tmp { - result *= &current; - current -= 1; - } - Ok(result) - } - - #[pyfunction] - fn comb(n: ArgIndex, k: ArgIndex, vm: &VirtualMachine) -> PyResult<BigInt> { - let k = k.into_int_ref(); - let mut k = k.as_bigint(); - let n = n.into_int_ref(); - let n = n.as_bigint(); - let one = BigInt::one(); - let zero = BigInt::zero(); - - if n.is_negative() || k.is_negative() { - return Err(vm.new_value_error("comb() not defined for negative values")); - } - - let temp = n - k; - if temp.is_negative() { - return Ok(zero); - } - - if temp < *k { - k = &temp - } - - if k.is_zero() { - return Ok(one); - } - - let mut result = n.clone(); - let mut factor = n.clone(); - let mut current = one; - while current < *k { - factor -= 1; - current += 1; - - result *= &factor; - result /= &current; - } - - Ok(result) + let values: Result<Vec<f64>, _> = + seq.iter(vm)?.map(|r| r.map(|v| v.into_float())).collect(); + pymath::math::fsum(values?).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] fn modf(x: ArgIntoFloat) -> (f64, f64) { - let x = x.into_float(); - if !x.is_finite() { - if x.is_infinite() { - return (0.0_f64.copysign(x), x); - } else if x.is_nan() { - return (x, x); - } - } - - (x.fract(), x.trunc()) + pymath::math::modf(x.into_float()) } #[derive(FromArgs)] @@ -887,85 +446,36 @@ mod math { #[pyfunction] fn nextafter(arg: NextAfterArgs, vm: &VirtualMachine) -> PyResult<f64> { - let steps: Option<i64> = arg - .steps - .map(|v| v.into_int_ref().try_to_primitive(vm)) - .transpose()? - .into_option(); - let x: f64 = arg.x.into(); - let y: f64 = arg.y.into(); - match steps { + let x = arg.x.into_float(); + let y = arg.y.into_float(); + + let steps = match arg.steps.into_option() { Some(steps) => { + let steps: i64 = steps.into_int_ref().try_to_primitive(vm)?; if steps < 0 { return Err(vm.new_value_error("steps must be a non-negative integer")); } - Ok(float_ops::nextafter_with_steps(x, y, steps as u64)) + Some(steps as u64) } - None => Ok(float_ops::nextafter(x, y)), - } + None => None, + }; + Ok(pymath::math::nextafter(x, y, steps)) } #[pyfunction] fn ulp(x: ArgIntoFloat) -> f64 { - float_ops::ulp(x.into()) - } - - fn fmod(x: f64, y: f64) -> f64 { - if y.is_infinite() && x.is_finite() { - return x; - } - - x % y + pymath::math::ulp(x.into_float()) } #[pyfunction(name = "fmod")] fn py_fmod(x: ArgIntoFloat, y: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - let x = x.into_float(); - let y = y.into_float(); - - let r = fmod(x, y); - - if r.is_nan() && !x.is_nan() && !y.is_nan() { - return Err(vm.new_value_error("math domain error")); - } - - Ok(r) + pymath::math::fmod(x.into_float(), y.into_float()).map_err(|err| pymath_exception(err, vm)) } #[pyfunction] fn remainder(x: ArgIntoFloat, y: ArgIntoFloat, vm: &VirtualMachine) -> PyResult<f64> { - let x = x.into_float(); - let y = y.into_float(); - - if x.is_finite() && y.is_finite() { - if y == 0.0 { - return Err(vm.new_value_error("math domain error")); - } - - let abs_x = x.abs(); - let abs_y = y.abs(); - let modulus = abs_x % abs_y; - - let c = abs_y - modulus; - let r = match modulus.partial_cmp(&c) { - Some(Ordering::Less) => modulus, - Some(Ordering::Greater) => -c, - _ => modulus - 2.0 * fmod(0.5 * (abs_x - modulus), abs_y), - }; - - return Ok(1.0_f64.copysign(x) * r); - } - if x.is_infinite() && !y.is_nan() { - return Err(vm.new_value_error("math domain error")); - } - if x.is_nan() || y.is_nan() { - return Ok(f64::NAN); - } - if y.is_infinite() { - Ok(x) - } else { - Err(vm.new_value_error("math domain error")) - } + pymath::math::remainder(x.into_float(), y.into_float()) + .map_err(|err| pymath_exception(err, vm)) } #[derive(FromArgs)] @@ -978,15 +488,118 @@ mod math { #[pyfunction] fn prod(args: ProdArgs, vm: &VirtualMachine) -> PyResult<PyObjectRef> { + use crate::vm::builtins::PyInt; + let iter = args.iterable; + let start = args.start; + + // Check if start is provided and what type it is (exact types only, not subclasses) + let (mut obj_result, start_is_int, start_is_float) = match &start { + OptionalArg::Present(s) => { + let is_int = s.class().is(vm.ctx.types.int_type); + let is_float = s.class().is(vm.ctx.types.float_type); + (Some(s.clone()), is_int, is_float) + } + OptionalArg::Missing => (None, true, false), // Default is int 1 + }; + + let mut item_iter = iter.iter(vm)?; + + // Integer fast path + if start_is_int && !start_is_float { + let mut int_result: i64 = match &start { + OptionalArg::Present(s) => { + if let Some(i) = s.downcast_ref::<PyInt>() { + match i.as_bigint().try_into() { + Ok(v) => v, + Err(_) => { + // Start overflows i64, fall through to generic path + obj_result = Some(s.clone()); + i64::MAX // Will be ignored + } + } + } else { + 1 + } + } + OptionalArg::Missing => 1, + }; - let mut result = args.start.unwrap_or_else(|| vm.new_pyobj(1)); + if obj_result.is_none() { + loop { + let item = match item_iter.next() { + Some(r) => r?, + None => return Ok(vm.ctx.new_int(int_result).into()), + }; + + // Only use fast path for exact int type (not subclasses) + if item.class().is(vm.ctx.types.int_type) + && let Some(int_item) = item.downcast_ref::<PyInt>() + && let Ok(b) = int_item.as_bigint().try_into() as Result<i64, _> + && let Some(product) = int_result.checked_mul(b) + { + int_result = product; + continue; + } - // TODO: CPython has optimized implementation for this - // refer: https://github.com/python/cpython/blob/main/Modules/mathmodule.c#L3093-L3193 - for obj in iter.iter(vm)? { - let obj = obj?; - result = vm._mul(&result, &obj)?; + // Overflow or non-int: restore to PyObject and continue + obj_result = Some(vm.ctx.new_int(int_result).into()); + let temp = vm._mul(obj_result.as_ref().unwrap(), &item)?; + obj_result = Some(temp); + break; + } + } + } + + // Float fast path + let obj_float = obj_result + .as_ref() + .and_then(|obj| obj.clone().downcast::<PyFloat>().ok()); + if obj_float.is_some() || start_is_float { + let mut flt_result: f64 = if let Some(ref f) = obj_float { + f.to_f64() + } else if start_is_float && let OptionalArg::Present(s) = &start { + s.downcast_ref::<PyFloat>() + .map(|f| f.to_f64()) + .unwrap_or(1.0) + } else { + 1.0 + }; + + loop { + let item = match item_iter.next() { + Some(r) => r?, + None => return Ok(vm.ctx.new_float(flt_result).into()), + }; + + // Only use fast path for exact float/int types (not subclasses) + if item.class().is(vm.ctx.types.float_type) + && let Some(f) = item.downcast_ref::<PyFloat>() + { + flt_result *= f.to_f64(); + continue; + } + if item.class().is(vm.ctx.types.int_type) + && let Some(i) = item.downcast_ref::<PyInt>() + && let Ok(v) = i.as_bigint().try_into() as Result<i64, _> + { + flt_result *= v as f64; + continue; + } + + // Non-exact-float/int: restore and continue with generic path + obj_result = Some(vm.ctx.new_float(flt_result).into()); + let temp = vm._mul(obj_result.as_ref().unwrap(), &item)?; + obj_result = Some(temp); + break; + } + } + + // Generic path for remaining items + let mut result = obj_result.unwrap_or_else(|| vm.ctx.new_int(1).into()); + for item in item_iter { + let item = item?; + result = vm._mul(&result, &item)?; } Ok(result) @@ -998,29 +611,145 @@ mod math { q: ArgIterable<PyObjectRef>, vm: &VirtualMachine, ) -> PyResult<PyObjectRef> { + use crate::vm::builtins::PyInt; + let mut p_iter = p.iter(vm)?; let mut q_iter = q.iter(vm)?; - // We cannot just create a float because the iterator may contain - // anything as long as it supports __add__ and __mul__. - let mut result = vm.new_pyobj(0); + + // Fast path state + let mut int_path_enabled = true; + let mut int_total: i64 = 0; + let mut int_total_in_use = false; + let mut flt_p_values: Vec<f64> = Vec::new(); + let mut flt_q_values: Vec<f64> = Vec::new(); + + // Fallback accumulator for generic Python path + let mut obj_total: Option<PyObjectRef> = None; + loop { let m_p = p_iter.next(); let m_q = q_iter.next(); - match (m_p, m_q) { - (Some(r_p), Some(r_q)) => { - let p = r_p?; - let q = r_q?; - let tmp = vm._mul(&p, &q)?; - result = vm._add(&result, &tmp)?; + + let (p_i, q_i, finished) = match (m_p, m_q) { + (Some(r_p), Some(r_q)) => (Some(r_p?), Some(r_q?), false), + (None, None) => (None, None, true), + _ => return Err(vm.new_value_error("Inputs are not the same length")), + }; + + // Integer fast path (only for exact int types, not subclasses) + if int_path_enabled { + if !finished { + let (p_i, q_i) = (p_i.as_ref().unwrap(), q_i.as_ref().unwrap()); + if p_i.class().is(vm.ctx.types.int_type) + && q_i.class().is(vm.ctx.types.int_type) + && let (Some(p_int), Some(q_int)) = + (p_i.downcast_ref::<PyInt>(), q_i.downcast_ref::<PyInt>()) + && let (Ok(p_val), Ok(q_val)) = ( + p_int.as_bigint().try_into() as Result<i64, _>, + q_int.as_bigint().try_into() as Result<i64, _>, + ) + && let Some(prod) = p_val.checked_mul(q_val) + && let Some(new_total) = int_total.checked_add(prod) + { + int_total = new_total; + int_total_in_use = true; + continue; + } } - (None, None) => break, - _ => { - return Err(vm.new_value_error("Inputs are not the same length")); + // Finalize int path + int_path_enabled = false; + if int_total_in_use { + let int_obj: PyObjectRef = vm.ctx.new_int(int_total).into(); + obj_total = Some(match obj_total { + Some(total) => vm._add(&total, &int_obj)?, + None => int_obj, + }); + int_total = 0; + int_total_in_use = false; } } + + // Float fast path - only when at least one value is exact float type + // (not subclasses, to preserve custom __mul__/__add__ behavior) + { + if !finished { + let (p_i, q_i) = (p_i.as_ref().unwrap(), q_i.as_ref().unwrap()); + + let p_is_exact_float = p_i.class().is(vm.ctx.types.float_type); + let q_is_exact_float = q_i.class().is(vm.ctx.types.float_type); + let p_is_exact_int = p_i.class().is(vm.ctx.types.int_type); + let q_is_exact_int = q_i.class().is(vm.ctx.types.int_type); + let p_is_exact_numeric = p_is_exact_float || p_is_exact_int; + let q_is_exact_numeric = q_is_exact_float || q_is_exact_int; + let has_exact_float = p_is_exact_float || q_is_exact_float; + + // Only use float path if at least one is exact float and both are exact int/float + if has_exact_float && p_is_exact_numeric && q_is_exact_numeric { + let p_flt = if let Some(f) = p_i.downcast_ref::<PyFloat>() { + Some(f.to_f64()) + } else if let Some(i) = p_i.downcast_ref::<PyInt>() { + // PyLong_AsDouble fails for integers too large for f64 + try_bigint_to_f64(i.as_bigint(), vm).ok() + } else { + None + }; + + let q_flt = if let Some(f) = q_i.downcast_ref::<PyFloat>() { + Some(f.to_f64()) + } else if let Some(i) = q_i.downcast_ref::<PyInt>() { + // PyLong_AsDouble fails for integers too large for f64 + try_bigint_to_f64(i.as_bigint(), vm).ok() + } else { + None + }; + + if let (Some(p_val), Some(q_val)) = (p_flt, q_flt) { + flt_p_values.push(p_val); + flt_q_values.push(q_val); + continue; + } + } + } + // Finalize float path + if !flt_p_values.is_empty() { + let flt_result = pymath::math::sumprod(&flt_p_values, &flt_q_values); + let flt_obj: PyObjectRef = vm.ctx.new_float(flt_result).into(); + obj_total = Some(match obj_total { + Some(total) => vm._add(&total, &flt_obj)?, + None => flt_obj, + }); + flt_p_values.clear(); + flt_q_values.clear(); + } + } + + if finished { + break; + } + + // Generic Python path + let (p_i, q_i) = (p_i.unwrap(), q_i.unwrap()); + + // Collect current + remaining elements + let p_remaining: Result<Vec<PyObjectRef>, _> = + std::iter::once(Ok(p_i)).chain(p_iter).collect(); + let q_remaining: Result<Vec<PyObjectRef>, _> = + std::iter::once(Ok(q_i)).chain(q_iter).collect(); + let (p_vec, q_vec) = (p_remaining?, q_remaining?); + + if p_vec.len() != q_vec.len() { + return Err(vm.new_value_error("Inputs are not the same length")); + } + + let mut total = obj_total.unwrap_or_else(|| vm.ctx.new_int(0).into()); + for (p_item, q_item) in p_vec.into_iter().zip(q_vec) { + let prod = vm._mul(&p_item, &q_item)?; + total = vm._add(&total, &prod)?; + } + return Ok(total); } - Ok(result) + Ok(obj_total.unwrap_or_else(|| vm.ctx.new_int(0).into())) } #[pyfunction] @@ -1030,30 +759,202 @@ mod math { z: ArgIntoFloat, vm: &VirtualMachine, ) -> PyResult<f64> { - let x = x.into_float(); - let y = y.into_float(); - let z = z.into_float(); - let result = x.mul_add(y, z); + pymath::math::fma(x.into_float(), y.into_float(), z.into_float()).map_err(|err| match err { + pymath::Error::EDOM => vm.new_value_error("invalid operation in fma"), + pymath::Error::ERANGE => vm.new_overflow_error("overflow in fma"), + }) + } + + // Integer functions: + + #[pyfunction] + fn isqrt(x: ArgIndex, vm: &VirtualMachine) -> PyResult<BigInt> { + let value = x.into_int_ref(); + pymath::math::integer::isqrt(value.as_bigint()) + .map_err(|_| vm.new_value_error("isqrt() argument must be nonnegative")) + } - if result.is_finite() { - return Ok(result); + #[pyfunction] + fn gcd(args: PosArgs<ArgIndex>) -> BigInt { + let ints: Vec<_> = args + .into_vec() + .into_iter() + .map(|x| x.into_int_ref()) + .collect(); + let refs: Vec<_> = ints.iter().map(|x| x.as_bigint()).collect(); + pymath::math::integer::gcd(&refs) + } + + #[pyfunction] + fn lcm(args: PosArgs<ArgIndex>) -> BigInt { + let ints: Vec<_> = args + .into_vec() + .into_iter() + .map(|x| x.into_int_ref()) + .collect(); + let refs: Vec<_> = ints.iter().map(|x| x.as_bigint()).collect(); + pymath::math::integer::lcm(&refs) + } + + #[pyfunction] + fn factorial(x: PyIntRef, vm: &VirtualMachine) -> PyResult<BigInt> { + // Check for negative before overflow - negative values are always invalid + if x.as_bigint().is_negative() { + return Err(vm.new_value_error("factorial() not defined for negative values")); } + let n: i64 = x.try_to_primitive(vm).map_err(|_| { + vm.new_overflow_error("factorial() argument should not exceed 9223372036854775807") + })?; + pymath::math::integer::factorial(n) + .map(|r| r.into()) + .map_err(|_| vm.new_value_error("factorial() not defined for negative values")) + } + + #[pyfunction] + fn perm( + n: ArgIndex, + k: OptionalArg<Option<ArgIndex>>, + vm: &VirtualMachine, + ) -> PyResult<BigInt> { + let n_int = n.into_int_ref(); + let n_big = n_int.as_bigint(); - if result.is_nan() { - if !x.is_nan() && !y.is_nan() && !z.is_nan() { - return Err(vm.new_value_error("invalid operation in fma")); + if n_big.is_negative() { + return Err(vm.new_value_error("n must be a non-negative integer")); + } + + // k = None means k = n (factorial) + let k_int = k.flatten().map(|k| k.into_int_ref()); + let k_big: Option<&BigInt> = k_int.as_ref().map(|k| k.as_bigint()); + + if let Some(k_val) = k_big { + if k_val.is_negative() { + return Err(vm.new_value_error("k must be a non-negative integer")); + } + if k_val > n_big { + return Ok(BigInt::from(0u8)); } - } else if x.is_finite() && y.is_finite() && z.is_finite() { - return Err(vm.new_overflow_error("overflow in fma")); } - Ok(result) + // Convert k to u64 (required by pymath) + let ki: u64 = match k_big { + None => match n_big.to_u64() { + Some(n) => n, + None => { + return Err(vm.new_overflow_error(format!("n must not exceed {}", u64::MAX))); + } + }, + Some(k_val) => match k_val.to_u64() { + Some(k) => k, + None => { + return Err(vm.new_overflow_error(format!("k must not exceed {}", u64::MAX))); + } + }, + }; + + // Fast path: n fits in i64 + if let Some(ni) = n_big.to_i64() + && ni >= 0 + && ki > 1 + { + let result = pymath::math::integer::perm(ni, Some(ki as i64)) + .map_err(|_| vm.new_value_error("perm() error"))?; + return Ok(result.into()); + } + + // BigInt path: use perm_bigint + let result = pymath::math::perm_bigint(n_big, ki); + Ok(result.into()) + } + + #[pyfunction] + fn comb(n: ArgIndex, k: ArgIndex, vm: &VirtualMachine) -> PyResult<BigInt> { + let n_int = n.into_int_ref(); + let n_big = n_int.as_bigint(); + let k_int = k.into_int_ref(); + let k_big = k_int.as_bigint(); + + if n_big.is_negative() { + return Err(vm.new_value_error("n must be a non-negative integer")); + } + if k_big.is_negative() { + return Err(vm.new_value_error("k must be a non-negative integer")); + } + + // Fast path: n fits in i64 + if let Some(ni) = n_big.to_i64() + && ni >= 0 + { + // k overflow or k > n means result is 0 + let ki = match k_big.to_i64() { + Some(k) if k >= 0 && k <= ni => k, + _ => return Ok(BigInt::from(0u8)), + }; + // Apply symmetry: use min(k, n-k) + let ki = ki.min(ni - ki); + if ki > 1 { + let result = pymath::math::integer::comb(ni, ki) + .map_err(|_| vm.new_value_error("comb() error"))?; + return Ok(result.into()); + } + // ki <= 1 cases + if ki == 0 { + return Ok(BigInt::from(1u8)); + } + return Ok(n_big.clone()); // ki == 1 + } + + // BigInt path: n doesn't fit in i64 + // Apply symmetry: k = min(k, n - k) + let n_minus_k = n_big - k_big; + if n_minus_k.is_negative() { + return Ok(BigInt::from(0u8)); + } + let effective_k = if &n_minus_k < k_big { + &n_minus_k + } else { + k_big + }; + + // k must fit in u64 + let ki: u64 = match effective_k.to_u64() { + Some(k) => k, + None => { + return Err( + vm.new_overflow_error(format!("min(n - k, k) must not exceed {}", u64::MAX)) + ); + } + }; + + let result = pymath::math::comb_bigint(n_big, ki); + Ok(result.into()) } } -fn pymath_error_to_exception(err: pymath::Error, vm: &VirtualMachine) -> PyBaseExceptionRef { +pub(crate) fn pymath_exception(err: pymath::Error, vm: &VirtualMachine) -> PyBaseExceptionRef { match err { pymath::Error::EDOM => vm.new_value_error("math domain error"), pymath::Error::ERANGE => vm.new_overflow_error("math range error"), } } + +/// Format a float in Python style (ensures trailing .0 for integers). +fn float_repr(value: f64) -> String { + if value.is_nan() { + "nan".to_owned() + } else if value.is_infinite() { + if value.is_sign_positive() { + "inf".to_owned() + } else { + "-inf".to_owned() + } + } else { + let s = format!("{}", value); + // If no decimal point and not in scientific notation, add .0 + if !s.contains('.') && !s.contains('e') && !s.contains('E') { + format!("{}.0", s) + } else { + s + } + } +} diff --git a/crates/stdlib/src/md5.rs b/crates/stdlib/src/md5.rs index dca48242bbd..11923b165ed 100644 --- a/crates/stdlib/src/md5.rs +++ b/crates/stdlib/src/md5.rs @@ -1,4 +1,4 @@ -pub(crate) use _md5::make_module; +pub(crate) use _md5::module_def; #[pymodule] mod _md5 { diff --git a/crates/stdlib/src/mmap.rs b/crates/stdlib/src/mmap.rs index d3895521096..625b6db5f6c 100644 --- a/crates/stdlib/src/mmap.rs +++ b/crates/stdlib/src/mmap.rs @@ -1,6 +1,6 @@ // spell-checker:disable //! mmap module -pub(crate) use mmap::make_module; +pub(crate) use mmap::module_def; #[pymodule] mod mmap { diff --git a/crates/stdlib/src/multiprocessing.rs b/crates/stdlib/src/multiprocessing.rs index 21b7bfa85c7..f3a7831defe 100644 --- a/crates/stdlib/src/multiprocessing.rs +++ b/crates/stdlib/src/multiprocessing.rs @@ -1,4 +1,4 @@ -pub(crate) use _multiprocessing::make_module; +pub(crate) use _multiprocessing::module_def; #[cfg(windows)] #[pymodule] @@ -547,6 +547,13 @@ mod _multiprocessing { self.last_tid.store(0, Ordering::Release); } + /// SemLock objects cannot be pickled directly. + /// Use multiprocessing.synchronize.SemLock wrapper which handles pickling. + #[pymethod] + fn __reduce__(&self, vm: &VirtualMachine) -> PyResult { + Err(vm.new_type_error("cannot pickle 'SemLock' object".to_owned())) + } + /// Num of `acquire()`s minus num of `release()`s for this process. // _multiprocessing_SemLock__count_impl #[pymethod] diff --git a/crates/stdlib/src/openssl.rs b/crates/stdlib/src/openssl.rs index 38103a9ab05..29f32b46386 100644 --- a/crates/stdlib/src/openssl.rs +++ b/crates/stdlib/src/openssl.rs @@ -23,31 +23,25 @@ cfg_if::cfg_if! { } } -use crate::vm::{PyRef, VirtualMachine, builtins::PyModule}; -use openssl_probe::ProbeResult; +pub(crate) use _ssl::module_def; -pub(crate) fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - // if openssl is vendored, it doesn't know the locations - // of system certificates - cache the probe result now. - #[cfg(openssl_vendored)] - LazyLock::force(&PROBE); - _ssl::make_module(vm) -} +use openssl_probe::ProbeResult; +use std::sync::LazyLock; // define our own copy of ProbeResult so we can handle the vendor case // easily, without having to have a bunch of cfgs cfg_if::cfg_if! { if #[cfg(openssl_vendored)] { - use std::sync::LazyLock; static PROBE: LazyLock<ProbeResult> = LazyLock::new(openssl_probe::probe); - fn probe() -> &'static ProbeResult { &PROBE } } else { - fn probe() -> &'static ProbeResult { - &ProbeResult { cert_file: None, cert_dir: None } - } + static PROBE: LazyLock<ProbeResult> = LazyLock::new(|| ProbeResult { cert_file: None, cert_dir: vec![] }); } } +fn probe() -> &'static ProbeResult { + &PROBE +} + #[allow(non_upper_case_globals)] #[pymodule(with(cert::ssl_cert, ssl_error::ssl_error, ossl101, ossl111, windows))] mod _ssl { @@ -66,8 +60,8 @@ mod _ssl { vm::{ AsObject, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, builtins::{ - PyBaseException, PyBaseExceptionRef, PyBytesRef, PyListRef, PyStrRef, PyType, - PyWeak, + PyBaseException, PyBaseExceptionRef, PyBytesRef, PyListRef, PyModule, PyStrRef, + PyType, PyWeak, }, class_or_notimplemented, convert::ToPyException, @@ -103,6 +97,16 @@ mod _ssl { // Import certificate types from parent module use super::cert::{self, cert_to_certificate, cert_to_py}; + pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { + // if openssl is vendored, it doesn't know the locations + // of system certificates - cache the probe result now. + #[cfg(openssl_vendored)] + std::sync::LazyLock::force(&super::PROBE); + + __module_exec(vm, module); + Ok(()) + } + // Re-export PySSLCertificate to make it available in the _ssl module // It will be automatically exposed to Python via #[pyclass] #[allow(unused_imports)] @@ -446,7 +450,7 @@ mod _ssl { }); let cert_dir = probe .cert_dir - .as_ref() + .first() .map(PathBuf::from) .unwrap_or_else(|| { path_from_cstr(unsafe { CStr::from_ptr(sys::X509_get_default_cert_dir()) }) @@ -2801,49 +2805,120 @@ mod _ssl { let stream = self.connection.read(); let ssl_ptr = stream.ssl().as_ptr(); - // Perform SSL shutdown - may need to be called twice: - // 1st call: sends close-notify, returns 0 - // 2nd call: reads peer's close-notify, returns 1 - let mut ret = unsafe { sys::SSL_shutdown(ssl_ptr) }; - - // If ret == 0, try once more to complete the bidirectional shutdown - // This handles the case where peer's close-notify is already available - if ret == 0 { - ret = unsafe { sys::SSL_shutdown(ssl_ptr) }; + // BIO mode: just try shutdown once and raise SSLWantReadError if needed + if stream.is_bio() { + let ret = unsafe { sys::SSL_shutdown(ssl_ptr) }; + if ret < 0 { + let err = unsafe { sys::SSL_get_error(ssl_ptr, ret) }; + if err == sys::SSL_ERROR_WANT_READ { + return Err(create_ssl_want_read_error(vm).upcast()); + } else if err == sys::SSL_ERROR_WANT_WRITE { + return Err(create_ssl_want_write_error(vm).upcast()); + } else { + return Err(new_ssl_error( + vm, + format!("SSL shutdown failed: error code {}", err), + )); + } + } else if ret == 0 { + // Sent close-notify, waiting for peer's - raise SSLWantReadError + return Err(create_ssl_want_read_error(vm).upcast()); + } + return Ok(None); } - if ret < 0 { - // Error occurred + // Socket mode: loop with select to wait for peer's close-notify + let socket_stream = stream.get_ref().expect("get_ref() failed for socket mode"); + let deadline = socket_stream.timeout_deadline(); + + // Track how many times we've seen ret == 0 (max 2 tries) + let mut zeros = 0; + + loop { + let ret = unsafe { sys::SSL_shutdown(ssl_ptr) }; + + // ret > 0: complete shutdown + if ret > 0 { + break; + } + + // ret == 0: sent our close-notify, need to receive peer's + if ret == 0 { + zeros += 1; + if zeros > 1 { + // Already tried twice, break out (legacy behavior) + break; + } + // Wait briefly for peer's close_notify before retrying + match socket_stream.select(SslNeeds::Read, &deadline) { + SelectRet::TimedOut => { + return Err(vm.new_exception_msg( + vm.ctx.exceptions.timeout_error.to_owned(), + "The read operation timed out".to_owned(), + )); + } + SelectRet::Closed => { + return Err(socket_closed_error(vm)); + } + SelectRet::Nonblocking => { + // Non-blocking socket: return SSLWantReadError + return Err(create_ssl_want_read_error(vm).upcast()); + } + SelectRet::Ok => { + // Data available, continue to retry + } + } + continue; + } + + // ret < 0: error or would-block let err = unsafe { sys::SSL_get_error(ssl_ptr, ret) }; - if err == sys::SSL_ERROR_WANT_READ { - return Err(create_ssl_want_read_error(vm).upcast()); + let needs = if err == sys::SSL_ERROR_WANT_READ { + SslNeeds::Read } else if err == sys::SSL_ERROR_WANT_WRITE { - return Err(create_ssl_want_write_error(vm).upcast()); + SslNeeds::Write } else { + // Real error return Err(new_ssl_error( vm, format!("SSL shutdown failed: error code {}", err), )); - } - } else if ret == 0 { - // Still waiting for peer's close-notify after retry - // In BIO mode, raise SSLWantReadError - if stream.is_bio() { - return Err(create_ssl_want_read_error(vm).upcast()); - } - } + }; - // BIO mode doesn't have an underlying socket to return - if stream.is_bio() { - return Ok(None); + // Wait on the socket + match socket_stream.select(needs, &deadline) { + SelectRet::TimedOut => { + let msg = if err == sys::SSL_ERROR_WANT_READ { + "The read operation timed out" + } else { + "The write operation timed out" + }; + return Err(vm.new_exception_msg( + vm.ctx.exceptions.timeout_error.to_owned(), + msg.to_owned(), + )); + } + SelectRet::Closed => { + return Err(socket_closed_error(vm)); + } + SelectRet::Nonblocking => { + // Non-blocking socket, raise SSLWantReadError/SSLWantWriteError + if err == sys::SSL_ERROR_WANT_READ { + return Err(create_ssl_want_read_error(vm).upcast()); + } else { + return Err(create_ssl_want_write_error(vm).upcast()); + } + } + SelectRet::Ok => { + // Socket is ready, retry shutdown + continue; + } + } } - // Return the underlying socket for socket mode - let socket = stream - .get_ref() - .expect("unwrap() called on bio mode; should only be called in socket mode"); - Ok(Some(socket.0.clone())) + // Return the underlying socket + Ok(Some(socket_stream.0.clone())) } #[cfg(osslconf = "OPENSSL_NO_COMP")] diff --git a/crates/stdlib/src/overlapped.rs b/crates/stdlib/src/overlapped.rs index 664aa3f392c..eb2a968c042 100644 --- a/crates/stdlib/src/overlapped.rs +++ b/crates/stdlib/src/overlapped.rs @@ -1,6 +1,6 @@ // spell-checker:disable -pub(crate) use _overlapped::make_module; +pub(crate) use _overlapped::module_def; #[allow(non_snake_case)] #[pymodule] @@ -8,17 +8,17 @@ mod _overlapped { // straight-forward port of Modules/overlapped.c use crate::vm::{ - AsObject, Py, PyObjectRef, PyPayload, PyResult, VirtualMachine, + Py, PyObjectRef, PyPayload, PyResult, VirtualMachine, builtins::{PyBaseExceptionRef, PyBytesRef, PyType}, common::lock::PyMutex, convert::{ToPyException, ToPyObject}, function::OptionalArg, protocol::PyBuffer, - types::Constructor, + types::{Constructor, Destructor}, }; use windows_sys::Win32::{ Foundation::{self, GetLastError, HANDLE}, - Networking::WinSock::{AF_INET, AF_INET6, SOCKADDR_IN, SOCKADDR_IN6}, + Networking::WinSock::{AF_INET, AF_INET6, SOCKADDR, SOCKADDR_IN, SOCKADDR_IN6}, System::IO::OVERLAPPED, }; @@ -41,6 +41,95 @@ mod _overlapped { #[pyattr] const NULL: isize = 0; + // Function pointers for Winsock extension functions + static ACCEPT_EX: std::sync::OnceLock<usize> = std::sync::OnceLock::new(); + static CONNECT_EX: std::sync::OnceLock<usize> = std::sync::OnceLock::new(); + static DISCONNECT_EX: std::sync::OnceLock<usize> = std::sync::OnceLock::new(); + static TRANSMIT_FILE: std::sync::OnceLock<usize> = std::sync::OnceLock::new(); + + fn initialize_winsock_extensions(vm: &VirtualMachine) -> PyResult<()> { + use windows_sys::Win32::Networking::WinSock::{ + IPPROTO_TCP, SIO_GET_EXTENSION_FUNCTION_POINTER, SOCK_STREAM, SOCKET_ERROR, WSAIoctl, + closesocket, socket, + }; + + // GUIDs for extension functions + const WSAID_ACCEPTEX: windows_sys::core::GUID = windows_sys::core::GUID { + data1: 0xb5367df1, + data2: 0xcbac, + data3: 0x11cf, + data4: [0x95, 0xca, 0x00, 0x80, 0x5f, 0x48, 0xa1, 0x92], + }; + const WSAID_CONNECTEX: windows_sys::core::GUID = windows_sys::core::GUID { + data1: 0x25a207b9, + data2: 0xddf3, + data3: 0x4660, + data4: [0x8e, 0xe9, 0x76, 0xe5, 0x8c, 0x74, 0x06, 0x3e], + }; + const WSAID_DISCONNECTEX: windows_sys::core::GUID = windows_sys::core::GUID { + data1: 0x7fda2e11, + data2: 0x8630, + data3: 0x436f, + data4: [0xa0, 0x31, 0xf5, 0x36, 0xa6, 0xee, 0xc1, 0x57], + }; + const WSAID_TRANSMITFILE: windows_sys::core::GUID = windows_sys::core::GUID { + data1: 0xb5367df0, + data2: 0xcbac, + data3: 0x11cf, + data4: [0x95, 0xca, 0x00, 0x80, 0x5f, 0x48, 0xa1, 0x92], + }; + + // Check all four locks to prevent partial initialization + if ACCEPT_EX.get().is_some() + && CONNECT_EX.get().is_some() + && DISCONNECT_EX.get().is_some() + && TRANSMIT_FILE.get().is_some() + { + return Ok(()); + } + + let s = unsafe { socket(AF_INET as i32, SOCK_STREAM, IPPROTO_TCP) }; + if s == windows_sys::Win32::Networking::WinSock::INVALID_SOCKET { + return Err( + vm.new_os_error("Failed to create socket for WSA extension init".to_owned()) + ); + } + + let mut dw_bytes: u32 = 0; + + macro_rules! get_extension { + ($guid:expr, $lock:expr) => {{ + let mut func_ptr: usize = 0; + let ret = unsafe { + WSAIoctl( + s, + SIO_GET_EXTENSION_FUNCTION_POINTER, + &$guid as *const _ as *const _, + std::mem::size_of_val(&$guid) as u32, + &mut func_ptr as *mut _ as *mut _, + std::mem::size_of::<usize>() as u32, + &mut dw_bytes, + std::ptr::null_mut(), + None, + ) + }; + if ret == SOCKET_ERROR { + unsafe { closesocket(s) }; + return Err(vm.new_os_error("Failed to get WSA extension function".to_owned())); + } + let _ = $lock.set(func_ptr); + }}; + } + + get_extension!(WSAID_ACCEPTEX, ACCEPT_EX); + get_extension!(WSAID_CONNECTEX, CONNECT_EX); + get_extension!(WSAID_DISCONNECTEX, DISCONNECT_EX); + get_extension!(WSAID_TRANSMITFILE, TRANSMIT_FILE); + + unsafe { closesocket(s) }; + Ok(()) + } + #[pyattr] #[pyclass(name)] #[derive(PyPayload)] @@ -62,7 +151,6 @@ mod _overlapped { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { let zelf = self.inner.lock(); f.debug_struct("Overlapped") - // .field("overlapped", &(self.overlapped as *const _ as usize)) .field("handle", &zelf.handle) .field("error", &zelf.error) .field("data", &zelf.data) @@ -70,67 +158,75 @@ mod _overlapped { } } - #[allow(dead_code)] // TODO: remove when done #[derive(Debug)] enum OverlappedData { None, NotStarted, Read(PyBytesRef), + // Fields below store buffers that must be kept alive during async operations + #[allow(dead_code)] ReadInto(PyBuffer), + #[allow(dead_code)] Write(PyBuffer), - Accept(PyObjectRef), - Connect, + #[allow(dead_code)] + Accept(PyBytesRef), + Connect(Vec<u8>), // Store address bytes to keep them alive during async operation Disconnect, ConnectNamedPipe, + #[allow(dead_code)] // Reserved for named pipe support WaitNamedPipeAndConnect, TransmitFile, ReadFrom(OverlappedReadFrom), - WriteTo(PyBuffer), + WriteTo(OverlappedWriteTo), // Store address bytes for WSASendTo ReadFromInto(OverlappedReadFromInto), } struct OverlappedReadFrom { - // A (buffer, (host, port)) tuple - result: PyObjectRef, // The actual read buffer - allocated_buffer: PyObjectRef, - #[allow(dead_code)] - address: SOCKADDR_IN6, // TODO: remove when done - address_length: libc::c_int, + allocated_buffer: PyBytesRef, + address: SOCKADDR_IN6, + address_length: i32, } impl core::fmt::Debug for OverlappedReadFrom { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { f.debug_struct("OverlappedReadFrom") - .field("result", &self.result) .field("allocated_buffer", &self.allocated_buffer) - // .field("address", &self.address) .field("address_length", &self.address_length) .finish() } } struct OverlappedReadFromInto { - // A (number of bytes read, (host, port)) tuple - result: PyObjectRef, /* Buffer passed by the user */ user_buffer: PyBuffer, - #[allow(dead_code)] - address: SOCKADDR_IN6, // TODO: remove when done - address_length: libc::c_int, + address: SOCKADDR_IN6, + address_length: i32, } impl core::fmt::Debug for OverlappedReadFromInto { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { f.debug_struct("OverlappedReadFromInto") - .field("result", &self.result) .field("user_buffer", &self.user_buffer) - // .field("address", &self.address) .field("address_length", &self.address_length) .finish() } } + struct OverlappedWriteTo { + buf: PyBuffer, + address: Vec<u8>, // Keep address alive during async operation + } + + impl core::fmt::Debug for OverlappedWriteTo { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + f.debug_struct("OverlappedWriteTo") + .field("buf", &self.buf) + .field("address", &self.address.len()) + .finish() + } + } + fn mark_as_completed(ov: &mut OVERLAPPED) { ov.Internal = 0; if !ov.hEvent.is_null() { @@ -139,29 +235,111 @@ mod _overlapped { } fn from_windows_err(err: u32, vm: &VirtualMachine) -> PyBaseExceptionRef { - use Foundation::{ERROR_CONNECTION_ABORTED, ERROR_CONNECTION_REFUSED}; debug_assert_ne!(err, 0, "call errno_err instead"); - let exc = match err { - ERROR_CONNECTION_REFUSED => vm.ctx.exceptions.connection_refused_error, - ERROR_CONNECTION_ABORTED => vm.ctx.exceptions.connection_aborted_error, - err => return std::io::Error::from_raw_os_error(err as i32).to_pyexception(vm), - }; - // TODO: set errno and winerror - vm.new_exception_empty(exc.to_owned()) + std::io::Error::from_raw_os_error(err as i32).to_pyexception(vm) } fn HasOverlappedIoCompleted(overlapped: &OVERLAPPED) -> bool { overlapped.Internal != (Foundation::STATUS_PENDING as usize) } - /// Parse a SOCKADDR_IN6 (which can also hold IPv4 addresses) to a Python address tuple - fn unparse_address( - addr: &SOCKADDR_IN6, - _addr_len: libc::c_int, - vm: &VirtualMachine, - ) -> PyObjectRef { - use crate::vm::convert::ToPyObject; + /// Parse a Python address tuple to SOCKADDR + fn parse_address(addr_obj: &PyObjectRef, vm: &VirtualMachine) -> PyResult<(Vec<u8>, i32)> { + use crate::vm::builtins::PyTuple; + use windows_sys::Win32::Networking::WinSock::WSAStringToAddressW; + + let tuple = addr_obj + .downcast_ref::<PyTuple>() + .ok_or_else(|| vm.new_type_error("address must be a tuple".to_owned()))?; + + let tuple_len = tuple.len(); + + if tuple_len == 2 { + // IPv4: (host, port) + let host: String = tuple[0].try_to_value(vm)?; + let port: u16 = tuple[1].try_to_value(vm)?; + + let mut addr: SOCKADDR_IN = unsafe { std::mem::zeroed() }; + addr.sin_family = AF_INET; + addr.sin_port = port.to_be(); + + // Convert host string to address + let host_wide: Vec<u16> = host.encode_utf16().chain(std::iter::once(0)).collect(); + let mut addr_len = std::mem::size_of::<SOCKADDR_IN>() as i32; + + let ret = unsafe { + WSAStringToAddressW( + host_wide.as_ptr(), + AF_INET as i32, + std::ptr::null(), + &mut addr as *mut _ as *mut SOCKADDR, + &mut addr_len, + ) + }; + + if ret != 0 { + return Err(vm.new_os_error(format!("Invalid IPv4 address: {}", host))); + } + + // Restore port (WSAStringToAddressW overwrites it) + addr.sin_port = port.to_be(); + + let bytes = unsafe { + std::slice::from_raw_parts( + &addr as *const _ as *const u8, + std::mem::size_of::<SOCKADDR_IN>(), + ) + }; + Ok((bytes.to_vec(), std::mem::size_of::<SOCKADDR_IN>() as i32)) + } else if tuple_len == 4 { + // IPv6: (host, port, flowinfo, scope_id) + let host: String = tuple[0].try_to_value(vm)?; + let port: u16 = tuple[1].try_to_value(vm)?; + let flowinfo: u32 = tuple[2].try_to_value(vm)?; + let scope_id: u32 = tuple[3].try_to_value(vm)?; + + let mut addr: SOCKADDR_IN6 = unsafe { std::mem::zeroed() }; + addr.sin6_family = AF_INET6; + addr.sin6_port = port.to_be(); + addr.sin6_flowinfo = flowinfo; + addr.Anonymous.sin6_scope_id = scope_id; + + let host_wide: Vec<u16> = host.encode_utf16().chain(std::iter::once(0)).collect(); + let mut addr_len = std::mem::size_of::<SOCKADDR_IN6>() as i32; + + let ret = unsafe { + WSAStringToAddressW( + host_wide.as_ptr(), + AF_INET6 as i32, + std::ptr::null(), + &mut addr as *mut _ as *mut SOCKADDR, + &mut addr_len, + ) + }; + + if ret != 0 { + return Err(vm.new_os_error(format!("Invalid IPv6 address: {}", host))); + } + + // Restore fields that WSAStringToAddressW might overwrite + addr.sin6_port = port.to_be(); + addr.sin6_flowinfo = flowinfo; + addr.Anonymous.sin6_scope_id = scope_id; + + let bytes = unsafe { + std::slice::from_raw_parts( + &addr as *const _ as *const u8, + std::mem::size_of::<SOCKADDR_IN6>(), + ) + }; + Ok((bytes.to_vec(), std::mem::size_of::<SOCKADDR_IN6>() as i32)) + } else { + Err(vm.new_value_error("address tuple must have 2 or 4 elements".to_owned())) + } + } + /// Parse a SOCKADDR_IN6 (which can also hold IPv4 addresses) to a Python address tuple + fn unparse_address(addr: &SOCKADDR_IN6, _addr_len: i32, vm: &VirtualMachine) -> PyObjectRef { unsafe { let family = addr.sin6_family; if family == AF_INET { @@ -199,7 +377,7 @@ mod _overlapped { } } - #[pyclass(with(Constructor))] + #[pyclass(with(Constructor, Destructor))] impl Overlapped { #[pygetset] fn address(&self, _vm: &VirtualMachine) -> usize { @@ -214,73 +392,16 @@ mod _overlapped { && !matches!(inner.data, OverlappedData::NotStarted) } - fn WSARecv_inner( - inner: &mut OverlappedInner, - handle: isize, - buf: &[u8], - mut flags: u32, - vm: &VirtualMachine, - ) -> PyResult { - use windows_sys::Win32::Foundation::{ - ERROR_BROKEN_PIPE, ERROR_IO_PENDING, ERROR_MORE_DATA, ERROR_SUCCESS, - }; - - let wsabuf = windows_sys::Win32::Networking::WinSock::WSABUF { - buf: buf.as_ptr() as *mut _, - len: buf.len() as _, - }; - let mut n_read: u32 = 0; - // TODO: optimization with MaybeUninit - let ret = unsafe { - windows_sys::Win32::Networking::WinSock::WSARecv( - handle as _, - &wsabuf, - 1, - &mut n_read, - &mut flags, - &mut inner.overlapped, - None, - ) - }; - let err = if ret < 0 { - unsafe { windows_sys::Win32::Networking::WinSock::WSAGetLastError() as u32 } - } else { - Foundation::ERROR_SUCCESS - }; - inner.error = err; - match err { - ERROR_BROKEN_PIPE => { - mark_as_completed(&mut inner.overlapped); - Err(from_windows_err(err, vm)) - } - ERROR_SUCCESS | ERROR_MORE_DATA | ERROR_IO_PENDING => Ok(vm.ctx.none()), - _ => Err(from_windows_err(err, vm)), - } + #[pygetset] + fn error(&self, _vm: &VirtualMachine) -> u32 { + let inner = self.inner.lock(); + inner.error } - #[pymethod] - fn WSARecv( - zelf: &Py<Self>, - handle: isize, - size: u32, - flags: u32, - vm: &VirtualMachine, - ) -> PyResult { - let mut inner = zelf.inner.lock(); - if !matches!(inner.data, OverlappedData::None) { - return Err(vm.new_value_error("operation already attempted")); - } - - #[cfg(target_pointer_width = "32")] - let size = core::cmp::min(size, std::isize::MAX as _); - - let buf = vec![0u8; std::cmp::max(size, 1) as usize]; - let buf = vm.ctx.new_bytes(buf); - inner.handle = handle as _; - - let r = Self::WSARecv_inner(&mut inner, handle as _, buf.as_bytes(), flags, vm); - inner.data = OverlappedData::Read(buf); - r + #[pygetset] + fn event(&self, _vm: &VirtualMachine) -> isize { + let inner = self.inner.lock(); + inner.overlapped.hEvent as isize } #[pymethod] @@ -311,7 +432,6 @@ mod _overlapped { use windows_sys::Win32::Foundation::{ ERROR_BROKEN_PIPE, ERROR_IO_PENDING, ERROR_MORE_DATA, ERROR_SUCCESS, }; - use windows_sys::Win32::System::IO::GetOverlappedResult; let mut inner = zelf.inner.lock(); let wait = wait.unwrap_or(false); @@ -327,7 +447,7 @@ mod _overlapped { // Get the result let mut transferred: u32 = 0; let ret = unsafe { - GetOverlappedResult( + windows_sys::Win32::System::IO::GetOverlappedResult( inner.handle, &inner.overlapped, &mut transferred, @@ -349,16 +469,7 @@ mod _overlapped { // For read operations, broken pipe is acceptable match &inner.data { OverlappedData::Read(_) | OverlappedData::ReadInto(_) => {} - OverlappedData::ReadFrom(rf) - if rf.result.is(&vm.ctx.none()) - || rf.allocated_buffer.is(&vm.ctx.none()) => - { - return Err(from_windows_err(err, vm)); - } OverlappedData::ReadFrom(_) => {} - OverlappedData::ReadFromInto(rfi) if rfi.result.is(&vm.ctx.none()) => { - return Err(from_windows_err(err, vm)); - } OverlappedData::ReadFromInto(_) => {} _ => return Err(from_windows_err(err, vm)), } @@ -372,7 +483,6 @@ mod _overlapped { // Return result based on operation type match &inner.data { OverlappedData::Read(buf) => { - // Resize buffer to actual bytes read let bytes = buf.as_bytes(); let result = if transferred as usize != bytes.len() { vm.ctx.new_bytes(bytes[..transferred as usize].to_vec()) @@ -381,45 +491,22 @@ mod _overlapped { }; Ok(result.into()) } - OverlappedData::ReadInto(_) => { - // Return number of bytes read + OverlappedData::ReadInto(_) => Ok(vm.ctx.new_int(transferred).into()), + OverlappedData::Write(_) | OverlappedData::WriteTo(_) => { Ok(vm.ctx.new_int(transferred).into()) } - OverlappedData::Write(_) => { - // Return number of bytes written - Ok(vm.ctx.new_int(transferred).into()) - } - OverlappedData::Accept(_) => { - // Return None for accept - Ok(vm.ctx.none()) - } - OverlappedData::Connect => { - // Return None for connect - Ok(vm.ctx.none()) - } - OverlappedData::Disconnect => { - // Return None for disconnect - Ok(vm.ctx.none()) - } - OverlappedData::ConnectNamedPipe => { - // Return None for connect named pipe - Ok(vm.ctx.none()) - } - OverlappedData::WaitNamedPipeAndConnect => { - // Return None - Ok(vm.ctx.none()) - } + OverlappedData::Accept(_) => Ok(vm.ctx.none()), + OverlappedData::Connect(_) => Ok(vm.ctx.none()), + OverlappedData::Disconnect => Ok(vm.ctx.none()), + OverlappedData::ConnectNamedPipe => Ok(vm.ctx.none()), + OverlappedData::WaitNamedPipeAndConnect => Ok(vm.ctx.none()), + OverlappedData::TransmitFile => Ok(vm.ctx.none()), OverlappedData::ReadFrom(rf) => { - // Return (resized_buffer, (host, port)) tuple - let buf = rf - .allocated_buffer - .downcast_ref::<crate::vm::builtins::PyBytes>() - .unwrap(); - let bytes = buf.as_bytes(); + let bytes = rf.allocated_buffer.as_bytes(); let resized_buf = if transferred as usize != bytes.len() { vm.ctx.new_bytes(bytes[..transferred as usize].to_vec()) } else { - buf.to_owned() + rf.allocated_buffer.clone() }; let addr_tuple = unparse_address(&rf.address, rf.address_length, vm); Ok(vm @@ -428,7 +515,6 @@ mod _overlapped { .into()) } OverlappedData::ReadFromInto(rfi) => { - // Return (transferred, (host, port)) tuple let addr_tuple = unparse_address(&rfi.address, rfi.address_length, vm); Ok(vm .ctx @@ -438,108 +524,1342 @@ mod _overlapped { _ => Ok(vm.ctx.none()), } } - } - impl Constructor for Overlapped { - type Args = (isize,); + // ReadFile + #[pymethod] + fn ReadFile(zelf: &Py<Self>, handle: isize, size: u32, vm: &VirtualMachine) -> PyResult { + use windows_sys::Win32::Foundation::{ + ERROR_BROKEN_PIPE, ERROR_IO_PENDING, ERROR_MORE_DATA, ERROR_SUCCESS, + }; + use windows_sys::Win32::Storage::FileSystem::ReadFile; - fn py_new( - _cls: &Py<PyType>, - (mut event,): Self::Args, - vm: &VirtualMachine, - ) -> PyResult<Self> { - if event == INVALID_HANDLE_VALUE { - event = unsafe { - windows_sys::Win32::System::Threading::CreateEventA( - core::ptr::null(), - Foundation::TRUE, - Foundation::FALSE, - core::ptr::null(), - ) as isize - }; - if event == NULL { - return Err(vm.new_last_os_error()); + let mut inner = zelf.inner.lock(); + if !matches!(inner.data, OverlappedData::None) { + return Err(vm.new_value_error("operation already attempted".to_owned())); + } + + #[cfg(target_pointer_width = "32")] + let size = core::cmp::min(size, isize::MAX as u32); + + let buf = vec![0u8; std::cmp::max(size, 1) as usize]; + let buf = vm.ctx.new_bytes(buf); + inner.handle = handle as HANDLE; + inner.data = OverlappedData::Read(buf.clone()); + + let mut nread: u32 = 0; + let ret = unsafe { + ReadFile( + handle as HANDLE, + buf.as_bytes().as_ptr() as *mut _, + size, + &mut nread, + &mut inner.overlapped, + ) + }; + + let err = if ret != 0 { + ERROR_SUCCESS + } else { + unsafe { GetLastError() } + }; + inner.error = err; + + match err { + ERROR_BROKEN_PIPE => { + mark_as_completed(&mut inner.overlapped); + Err(from_windows_err(err, vm)) + } + ERROR_SUCCESS | ERROR_MORE_DATA | ERROR_IO_PENDING => Ok(vm.ctx.none()), + _ => { + inner.data = OverlappedData::NotStarted; + Err(from_windows_err(err, vm)) } } + } - let mut overlapped: OVERLAPPED = unsafe { std::mem::zeroed() }; - if event != NULL { - overlapped.hEvent = event as _; + // ReadFileInto + #[pymethod] + fn ReadFileInto( + zelf: &Py<Self>, + handle: isize, + buf: PyBuffer, + vm: &VirtualMachine, + ) -> PyResult { + use windows_sys::Win32::Foundation::{ + ERROR_BROKEN_PIPE, ERROR_IO_PENDING, ERROR_MORE_DATA, ERROR_SUCCESS, + }; + use windows_sys::Win32::Storage::FileSystem::ReadFile; + + let mut inner = zelf.inner.lock(); + if !matches!(inner.data, OverlappedData::None) { + return Err(vm.new_value_error("operation already attempted".to_owned())); } - let inner = OverlappedInner { - overlapped, - handle: NULL as _, - error: 0, - data: OverlappedData::None, + + inner.handle = handle as HANDLE; + let buf_len = buf.desc.len; + + // For async read, buffer must be contiguous - we can't use a temporary copy + // because Windows writes data directly to the buffer after this call returns + let Some(contiguous) = buf.as_contiguous_mut() else { + return Err(vm.new_buffer_error("buffer is not contiguous".to_owned())); }; - Ok(Overlapped { - inner: PyMutex::new(inner), - }) - } - } - unsafe fn u64_to_handle(raw_ptr_value: u64) -> HANDLE { - raw_ptr_value as HANDLE - } + inner.data = OverlappedData::ReadInto(buf.clone()); - #[pyfunction] - fn CreateIoCompletionPort( - handle: isize, - port: isize, - key: usize, - concurrency: u32, - vm: &VirtualMachine, - ) -> PyResult<isize> { - let r = unsafe { - windows_sys::Win32::System::IO::CreateIoCompletionPort( - handle as _, - port as _, - key, - concurrency, - ) as isize - }; - if r as usize == 0 { - return Err(vm.new_last_os_error()); - } - Ok(r) - } + let mut nread: u32 = 0; + let ret = unsafe { + ReadFile( + handle as HANDLE, + contiguous.as_ptr() as *mut _, + buf_len as u32, + &mut nread, + &mut inner.overlapped, + ) + }; - #[pyfunction] - fn GetQueuedCompletionStatus(port: isize, msecs: u32, vm: &VirtualMachine) -> PyResult { - let mut bytes_transferred = 0; - let mut completion_key = 0; - let mut overlapped: *mut OVERLAPPED = std::ptr::null_mut(); - let ret = unsafe { - windows_sys::Win32::System::IO::GetQueuedCompletionStatus( - port as _, - &mut bytes_transferred, - &mut completion_key, - &mut overlapped, - msecs, - ) - }; - let err = if ret != 0 { - Foundation::ERROR_SUCCESS - } else { - unsafe { Foundation::GetLastError() } - }; - if overlapped.is_null() { - if err == Foundation::WAIT_TIMEOUT { - return Ok(vm.ctx.none()); + let err = if ret != 0 { + ERROR_SUCCESS } else { - return Err(vm.new_last_os_error()); + unsafe { GetLastError() } + }; + inner.error = err; + + match err { + ERROR_BROKEN_PIPE => { + mark_as_completed(&mut inner.overlapped); + Err(from_windows_err(err, vm)) + } + ERROR_SUCCESS | ERROR_MORE_DATA | ERROR_IO_PENDING => Ok(vm.ctx.none()), + _ => { + inner.data = OverlappedData::NotStarted; + Err(from_windows_err(err, vm)) + } } } - let value = vm.ctx.new_tuple(vec![ - err.to_pyobject(vm), - completion_key.to_pyobject(vm), + // WSARecv + #[pymethod] + fn WSARecv( + zelf: &Py<Self>, + handle: isize, + size: u32, + flags: OptionalArg<u32>, + vm: &VirtualMachine, + ) -> PyResult { + use windows_sys::Win32::Foundation::{ + ERROR_BROKEN_PIPE, ERROR_IO_PENDING, ERROR_MORE_DATA, ERROR_SUCCESS, + }; + use windows_sys::Win32::Networking::WinSock::{WSABUF, WSAGetLastError, WSARecv}; + + let mut inner = zelf.inner.lock(); + if !matches!(inner.data, OverlappedData::None) { + return Err(vm.new_value_error("operation already attempted".to_owned())); + } + + let mut flags = flags.unwrap_or(0); + + #[cfg(target_pointer_width = "32")] + let size = core::cmp::min(size, isize::MAX as u32); + + let buf = vec![0u8; std::cmp::max(size, 1) as usize]; + let buf = vm.ctx.new_bytes(buf); + inner.handle = handle as HANDLE; + inner.data = OverlappedData::Read(buf.clone()); + + let wsabuf = WSABUF { + buf: buf.as_bytes().as_ptr() as *mut _, + len: size, + }; + let mut nread: u32 = 0; + + let ret = unsafe { + WSARecv( + handle as _, + &wsabuf, + 1, + &mut nread, + &mut flags, + &mut inner.overlapped, + None, + ) + }; + + let err = if ret < 0 { + unsafe { WSAGetLastError() as u32 } + } else { + ERROR_SUCCESS + }; + inner.error = err; + + match err { + ERROR_BROKEN_PIPE => { + mark_as_completed(&mut inner.overlapped); + Err(from_windows_err(err, vm)) + } + ERROR_SUCCESS | ERROR_MORE_DATA | ERROR_IO_PENDING => Ok(vm.ctx.none()), + _ => { + inner.data = OverlappedData::NotStarted; + Err(from_windows_err(err, vm)) + } + } + } + + // WSARecvInto + #[pymethod] + fn WSARecvInto( + zelf: &Py<Self>, + handle: isize, + buf: PyBuffer, + flags: u32, + vm: &VirtualMachine, + ) -> PyResult { + use windows_sys::Win32::Foundation::{ + ERROR_BROKEN_PIPE, ERROR_IO_PENDING, ERROR_MORE_DATA, ERROR_SUCCESS, + }; + use windows_sys::Win32::Networking::WinSock::{WSABUF, WSAGetLastError, WSARecv}; + + let mut inner = zelf.inner.lock(); + if !matches!(inner.data, OverlappedData::None) { + return Err(vm.new_value_error("operation already attempted".to_owned())); + } + + let mut flags = flags; + inner.handle = handle as HANDLE; + let buf_len = buf.desc.len; + + let Some(contiguous) = buf.as_contiguous_mut() else { + return Err(vm.new_buffer_error("buffer is not contiguous".to_owned())); + }; + + inner.data = OverlappedData::ReadInto(buf.clone()); + + let wsabuf = WSABUF { + buf: contiguous.as_ptr() as *mut _, + len: buf_len as u32, + }; + let mut nread: u32 = 0; + + let ret = unsafe { + WSARecv( + handle as _, + &wsabuf, + 1, + &mut nread, + &mut flags, + &mut inner.overlapped, + None, + ) + }; + + let err = if ret < 0 { + unsafe { WSAGetLastError() as u32 } + } else { + ERROR_SUCCESS + }; + inner.error = err; + + match err { + ERROR_BROKEN_PIPE => { + mark_as_completed(&mut inner.overlapped); + Err(from_windows_err(err, vm)) + } + ERROR_SUCCESS | ERROR_MORE_DATA | ERROR_IO_PENDING => Ok(vm.ctx.none()), + _ => { + inner.data = OverlappedData::NotStarted; + Err(from_windows_err(err, vm)) + } + } + } + + // WriteFile + #[pymethod] + fn WriteFile( + zelf: &Py<Self>, + handle: isize, + buf: PyBuffer, + vm: &VirtualMachine, + ) -> PyResult { + use windows_sys::Win32::Foundation::{ERROR_IO_PENDING, ERROR_SUCCESS}; + use windows_sys::Win32::Storage::FileSystem::WriteFile; + + let mut inner = zelf.inner.lock(); + if !matches!(inner.data, OverlappedData::None) { + return Err(vm.new_value_error("operation already attempted".to_owned())); + } + + inner.handle = handle as HANDLE; + let buf_len = buf.desc.len; + + // For async write, buffer must be contiguous - we can't use a temporary copy + // because Windows reads from the buffer after this call returns + let Some(contiguous) = buf.as_contiguous() else { + return Err(vm.new_buffer_error("buffer is not contiguous".to_owned())); + }; + + inner.data = OverlappedData::Write(buf.clone()); + + let mut written: u32 = 0; + let ret = unsafe { + WriteFile( + handle as HANDLE, + contiguous.as_ptr() as *const _, + buf_len as u32, + &mut written, + &mut inner.overlapped, + ) + }; + + let err = if ret != 0 { + ERROR_SUCCESS + } else { + unsafe { GetLastError() } + }; + inner.error = err; + + match err { + ERROR_SUCCESS | ERROR_IO_PENDING => Ok(vm.ctx.none()), + _ => { + inner.data = OverlappedData::NotStarted; + Err(from_windows_err(err, vm)) + } + } + } + + // WSASend + #[pymethod] + fn WSASend( + zelf: &Py<Self>, + handle: isize, + buf: PyBuffer, + flags: u32, + vm: &VirtualMachine, + ) -> PyResult { + use windows_sys::Win32::Foundation::{ERROR_IO_PENDING, ERROR_SUCCESS}; + use windows_sys::Win32::Networking::WinSock::{WSABUF, WSAGetLastError, WSASend}; + + let mut inner = zelf.inner.lock(); + if !matches!(inner.data, OverlappedData::None) { + return Err(vm.new_value_error("operation already attempted".to_owned())); + } + + inner.handle = handle as HANDLE; + let buf_len = buf.desc.len; + + let Some(contiguous) = buf.as_contiguous() else { + return Err(vm.new_buffer_error("buffer is not contiguous".to_owned())); + }; + + inner.data = OverlappedData::Write(buf.clone()); + + let wsabuf = WSABUF { + buf: contiguous.as_ptr() as *mut _, + len: buf_len as u32, + }; + let mut written: u32 = 0; + + let ret = unsafe { + WSASend( + handle as _, + &wsabuf, + 1, + &mut written, + flags, + &mut inner.overlapped, + None, + ) + }; + + let err = if ret < 0 { + unsafe { WSAGetLastError() as u32 } + } else { + ERROR_SUCCESS + }; + inner.error = err; + + match err { + ERROR_SUCCESS | ERROR_IO_PENDING => Ok(vm.ctx.none()), + _ => { + inner.data = OverlappedData::NotStarted; + Err(from_windows_err(err, vm)) + } + } + } + + // AcceptEx + #[pymethod] + fn AcceptEx( + zelf: &Py<Self>, + listen_socket: isize, + accept_socket: isize, + vm: &VirtualMachine, + ) -> PyResult { + use windows_sys::Win32::Foundation::{ERROR_IO_PENDING, ERROR_SUCCESS}; + use windows_sys::Win32::Networking::WinSock::WSAGetLastError; + + initialize_winsock_extensions(vm)?; + + let mut inner = zelf.inner.lock(); + if !matches!(inner.data, OverlappedData::None) { + return Err(vm.new_value_error("operation already attempted".to_owned())); + } + + // Buffer size: local address + remote address + let size = std::mem::size_of::<SOCKADDR_IN6>() + 16; + let buf = vec![0u8; size * 2]; + let buf = vm.ctx.new_bytes(buf); + + inner.handle = listen_socket as HANDLE; + inner.data = OverlappedData::Accept(buf.clone()); + + let mut bytes_received: u32 = 0; + + type AcceptExFn = unsafe extern "system" fn( + sListenSocket: usize, + sAcceptSocket: usize, + lpOutputBuffer: *mut core::ffi::c_void, + dwReceiveDataLength: u32, + dwLocalAddressLength: u32, + dwRemoteAddressLength: u32, + lpdwBytesReceived: *mut u32, + lpOverlapped: *mut OVERLAPPED, + ) -> i32; + + let accept_ex: AcceptExFn = unsafe { std::mem::transmute(*ACCEPT_EX.get().unwrap()) }; + + let ret = unsafe { + accept_ex( + listen_socket as _, + accept_socket as _, + buf.as_bytes().as_ptr() as *mut _, + 0, + size as u32, + size as u32, + &mut bytes_received, + &mut inner.overlapped, + ) + }; + + let err = if ret != 0 { + ERROR_SUCCESS + } else { + unsafe { WSAGetLastError() as u32 } + }; + inner.error = err; + + match err { + ERROR_SUCCESS | ERROR_IO_PENDING => Ok(vm.ctx.none()), + _ => { + inner.data = OverlappedData::NotStarted; + Err(from_windows_err(err, vm)) + } + } + } + + // ConnectEx + #[pymethod] + fn ConnectEx( + zelf: &Py<Self>, + socket: isize, + address: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult { + use windows_sys::Win32::Foundation::{ERROR_IO_PENDING, ERROR_SUCCESS}; + use windows_sys::Win32::Networking::WinSock::WSAGetLastError; + + initialize_winsock_extensions(vm)?; + + let mut inner = zelf.inner.lock(); + if !matches!(inner.data, OverlappedData::None) { + return Err(vm.new_value_error("operation already attempted".to_owned())); + } + + let (addr_bytes, addr_len) = parse_address(&address, vm)?; + + inner.handle = socket as HANDLE; + // Store addr_bytes in OverlappedData to keep it alive during async operation + inner.data = OverlappedData::Connect(addr_bytes); + + type ConnectExFn = unsafe extern "system" fn( + s: usize, + name: *const SOCKADDR, + namelen: i32, + lpSendBuffer: *const core::ffi::c_void, + dwSendDataLength: u32, + lpdwBytesSent: *mut u32, + lpOverlapped: *mut OVERLAPPED, + ) -> i32; + + let connect_ex: ConnectExFn = + unsafe { std::mem::transmute(*CONNECT_EX.get().unwrap()) }; + + // Get pointer to the stored address data + let addr_ptr = match &inner.data { + OverlappedData::Connect(bytes) => bytes.as_ptr(), + _ => unreachable!(), + }; + + let ret = unsafe { + connect_ex( + socket as _, + addr_ptr as *const SOCKADDR, + addr_len, + std::ptr::null(), + 0, + std::ptr::null_mut(), + &mut inner.overlapped, + ) + }; + + let err = if ret != 0 { + ERROR_SUCCESS + } else { + unsafe { WSAGetLastError() as u32 } + }; + inner.error = err; + + match err { + ERROR_SUCCESS | ERROR_IO_PENDING => Ok(vm.ctx.none()), + _ => { + inner.data = OverlappedData::NotStarted; + Err(from_windows_err(err, vm)) + } + } + } + + // DisconnectEx + #[pymethod] + fn DisconnectEx( + zelf: &Py<Self>, + socket: isize, + flags: u32, + vm: &VirtualMachine, + ) -> PyResult { + use windows_sys::Win32::Foundation::{ERROR_IO_PENDING, ERROR_SUCCESS}; + use windows_sys::Win32::Networking::WinSock::WSAGetLastError; + + initialize_winsock_extensions(vm)?; + + let mut inner = zelf.inner.lock(); + if !matches!(inner.data, OverlappedData::None) { + return Err(vm.new_value_error("operation already attempted".to_owned())); + } + + inner.handle = socket as HANDLE; + inner.data = OverlappedData::Disconnect; + + type DisconnectExFn = unsafe extern "system" fn( + s: usize, + lpOverlapped: *mut OVERLAPPED, + dwFlags: u32, + dwReserved: u32, + ) -> i32; + + let disconnect_ex: DisconnectExFn = + unsafe { std::mem::transmute(*DISCONNECT_EX.get().unwrap()) }; + + let ret = unsafe { disconnect_ex(socket as _, &mut inner.overlapped, flags, 0) }; + + let err = if ret != 0 { + ERROR_SUCCESS + } else { + unsafe { WSAGetLastError() as u32 } + }; + inner.error = err; + + match err { + ERROR_SUCCESS | ERROR_IO_PENDING => Ok(vm.ctx.none()), + _ => { + inner.data = OverlappedData::NotStarted; + Err(from_windows_err(err, vm)) + } + } + } + + // TransmitFile + #[allow(clippy::too_many_arguments)] + #[pymethod] + fn TransmitFile( + zelf: &Py<Self>, + socket: isize, + file: isize, + offset: u32, + offset_high: u32, + count_to_write: u32, + count_per_send: u32, + flags: u32, + vm: &VirtualMachine, + ) -> PyResult { + use windows_sys::Win32::Foundation::{ERROR_IO_PENDING, ERROR_SUCCESS}; + use windows_sys::Win32::Networking::WinSock::WSAGetLastError; + + initialize_winsock_extensions(vm)?; + + let mut inner = zelf.inner.lock(); + if !matches!(inner.data, OverlappedData::None) { + return Err(vm.new_value_error("operation already attempted".to_owned())); + } + + inner.handle = socket as HANDLE; + inner.data = OverlappedData::TransmitFile; + inner.overlapped.Anonymous.Anonymous.Offset = offset; + inner.overlapped.Anonymous.Anonymous.OffsetHigh = offset_high; + + type TransmitFileFn = unsafe extern "system" fn( + hSocket: usize, + hFile: HANDLE, + nNumberOfBytesToWrite: u32, + nNumberOfBytesPerSend: u32, + lpOverlapped: *mut OVERLAPPED, + lpTransmitBuffers: *const core::ffi::c_void, + dwReserved: u32, + ) -> i32; + + let transmit_file: TransmitFileFn = + unsafe { std::mem::transmute(*TRANSMIT_FILE.get().unwrap()) }; + + let ret = unsafe { + transmit_file( + socket as _, + file as HANDLE, + count_to_write, + count_per_send, + &mut inner.overlapped, + std::ptr::null(), + flags, + ) + }; + + let err = if ret != 0 { + ERROR_SUCCESS + } else { + unsafe { WSAGetLastError() as u32 } + }; + inner.error = err; + + match err { + ERROR_SUCCESS | ERROR_IO_PENDING => Ok(vm.ctx.none()), + _ => { + inner.data = OverlappedData::NotStarted; + Err(from_windows_err(err, vm)) + } + } + } + + // ConnectNamedPipe + #[pymethod] + fn ConnectNamedPipe(zelf: &Py<Self>, pipe: isize, vm: &VirtualMachine) -> PyResult<bool> { + use windows_sys::Win32::Foundation::{ + ERROR_IO_PENDING, ERROR_PIPE_CONNECTED, ERROR_SUCCESS, + }; + use windows_sys::Win32::System::Pipes::ConnectNamedPipe; + + let mut inner = zelf.inner.lock(); + if !matches!(inner.data, OverlappedData::None) { + return Err(vm.new_value_error("operation already attempted".to_owned())); + } + + inner.handle = pipe as HANDLE; + inner.data = OverlappedData::ConnectNamedPipe; + + let ret = unsafe { ConnectNamedPipe(pipe as HANDLE, &mut inner.overlapped) }; + + let err = if ret != 0 { + ERROR_SUCCESS + } else { + unsafe { GetLastError() } + }; + inner.error = err; + + match err { + ERROR_PIPE_CONNECTED => { + mark_as_completed(&mut inner.overlapped); + Ok(true) + } + ERROR_SUCCESS | ERROR_IO_PENDING => Ok(false), + _ => { + inner.data = OverlappedData::NotStarted; + Err(from_windows_err(err, vm)) + } + } + } + + // WSASendTo + #[pymethod] + fn WSASendTo( + zelf: &Py<Self>, + handle: isize, + buf: PyBuffer, + flags: u32, + address: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult { + use windows_sys::Win32::Foundation::{ERROR_IO_PENDING, ERROR_SUCCESS}; + use windows_sys::Win32::Networking::WinSock::{WSABUF, WSAGetLastError, WSASendTo}; + + let mut inner = zelf.inner.lock(); + if !matches!(inner.data, OverlappedData::None) { + return Err(vm.new_value_error("operation already attempted".to_owned())); + } + + let (addr_bytes, addr_len) = parse_address(&address, vm)?; + + inner.handle = handle as HANDLE; + let buf_len = buf.desc.len; + + let Some(contiguous) = buf.as_contiguous() else { + return Err(vm.new_buffer_error("buffer is not contiguous".to_owned())); + }; + + // Store both buffer and address in OverlappedData to keep them alive + inner.data = OverlappedData::WriteTo(OverlappedWriteTo { + buf: buf.clone(), + address: addr_bytes, + }); + + let wsabuf = WSABUF { + buf: contiguous.as_ptr() as *mut _, + len: buf_len as u32, + }; + let mut written: u32 = 0; + + // Get pointer to the stored address data + let addr_ptr = match &inner.data { + OverlappedData::WriteTo(wt) => wt.address.as_ptr(), + _ => unreachable!(), + }; + + let ret = unsafe { + WSASendTo( + handle as _, + &wsabuf, + 1, + &mut written, + flags, + addr_ptr as *const SOCKADDR, + addr_len, + &mut inner.overlapped, + None, + ) + }; + + let err = if ret < 0 { + unsafe { WSAGetLastError() as u32 } + } else { + ERROR_SUCCESS + }; + inner.error = err; + + match err { + ERROR_SUCCESS | ERROR_IO_PENDING => Ok(vm.ctx.none()), + _ => { + inner.data = OverlappedData::NotStarted; + Err(from_windows_err(err, vm)) + } + } + } + + // WSARecvFrom + #[pymethod] + fn WSARecvFrom( + zelf: &Py<Self>, + handle: isize, + size: u32, + flags: OptionalArg<u32>, + vm: &VirtualMachine, + ) -> PyResult { + use windows_sys::Win32::Foundation::{ + ERROR_BROKEN_PIPE, ERROR_IO_PENDING, ERROR_MORE_DATA, ERROR_SUCCESS, + }; + use windows_sys::Win32::Networking::WinSock::{WSABUF, WSAGetLastError, WSARecvFrom}; + + let mut inner = zelf.inner.lock(); + if !matches!(inner.data, OverlappedData::None) { + return Err(vm.new_value_error("operation already attempted".to_owned())); + } + + let mut flags = flags.unwrap_or(0); + + #[cfg(target_pointer_width = "32")] + let size = core::cmp::min(size, isize::MAX as u32); + + let buf = vec![0u8; std::cmp::max(size, 1) as usize]; + let buf = vm.ctx.new_bytes(buf); + inner.handle = handle as HANDLE; + + let address: SOCKADDR_IN6 = unsafe { std::mem::zeroed() }; + let address_length = std::mem::size_of::<SOCKADDR_IN6>() as i32; + + inner.data = OverlappedData::ReadFrom(OverlappedReadFrom { + allocated_buffer: buf.clone(), + address, + address_length, + }); + + let wsabuf = WSABUF { + buf: buf.as_bytes().as_ptr() as *mut _, + len: size, + }; + let mut nread: u32 = 0; + + // Get mutable reference to address in inner.data + let (addr_ptr, addr_len_ptr) = match &mut inner.data { + OverlappedData::ReadFrom(rf) => ( + &mut rf.address as *mut SOCKADDR_IN6, + &mut rf.address_length as *mut i32, + ), + _ => unreachable!(), + }; + + let ret = unsafe { + WSARecvFrom( + handle as _, + &wsabuf, + 1, + &mut nread, + &mut flags, + addr_ptr as *mut SOCKADDR, + addr_len_ptr, + &mut inner.overlapped, + None, + ) + }; + + let err = if ret < 0 { + unsafe { WSAGetLastError() as u32 } + } else { + ERROR_SUCCESS + }; + inner.error = err; + + match err { + ERROR_BROKEN_PIPE => { + mark_as_completed(&mut inner.overlapped); + Err(from_windows_err(err, vm)) + } + ERROR_SUCCESS | ERROR_MORE_DATA | ERROR_IO_PENDING => Ok(vm.ctx.none()), + _ => { + inner.data = OverlappedData::NotStarted; + Err(from_windows_err(err, vm)) + } + } + } + + // WSARecvFromInto + #[pymethod] + fn WSARecvFromInto( + zelf: &Py<Self>, + handle: isize, + buf: PyBuffer, + size: u32, + flags: OptionalArg<u32>, + vm: &VirtualMachine, + ) -> PyResult { + use windows_sys::Win32::Foundation::{ + ERROR_BROKEN_PIPE, ERROR_IO_PENDING, ERROR_MORE_DATA, ERROR_SUCCESS, + }; + use windows_sys::Win32::Networking::WinSock::{WSABUF, WSAGetLastError, WSARecvFrom}; + + let mut inner = zelf.inner.lock(); + if !matches!(inner.data, OverlappedData::None) { + return Err(vm.new_value_error("operation already attempted".to_owned())); + } + + let mut flags = flags.unwrap_or(0); + inner.handle = handle as HANDLE; + + let Some(contiguous) = buf.as_contiguous_mut() else { + return Err(vm.new_buffer_error("buffer is not contiguous".to_owned())); + }; + + // Validate size against buffer length to prevent buffer overflow + let buf_len = buf.desc.len as u32; + if size > buf_len { + return Err(vm.new_value_error("size exceeds buffer length".to_owned())); + } + + let address: SOCKADDR_IN6 = unsafe { std::mem::zeroed() }; + let address_length = std::mem::size_of::<SOCKADDR_IN6>() as i32; + + inner.data = OverlappedData::ReadFromInto(OverlappedReadFromInto { + user_buffer: buf.clone(), + address, + address_length, + }); + + let wsabuf = WSABUF { + buf: contiguous.as_ptr() as *mut _, + len: size, + }; + let mut nread: u32 = 0; + + // Get mutable reference to address in inner.data + let (addr_ptr, addr_len_ptr) = match &mut inner.data { + OverlappedData::ReadFromInto(rfi) => ( + &mut rfi.address as *mut SOCKADDR_IN6, + &mut rfi.address_length as *mut i32, + ), + _ => unreachable!(), + }; + + let ret = unsafe { + WSARecvFrom( + handle as _, + &wsabuf, + 1, + &mut nread, + &mut flags, + addr_ptr as *mut SOCKADDR, + addr_len_ptr, + &mut inner.overlapped, + None, + ) + }; + + let err = if ret < 0 { + unsafe { WSAGetLastError() as u32 } + } else { + ERROR_SUCCESS + }; + inner.error = err; + + match err { + ERROR_BROKEN_PIPE => { + mark_as_completed(&mut inner.overlapped); + Err(from_windows_err(err, vm)) + } + ERROR_SUCCESS | ERROR_MORE_DATA | ERROR_IO_PENDING => Ok(vm.ctx.none()), + _ => { + inner.data = OverlappedData::NotStarted; + Err(from_windows_err(err, vm)) + } + } + } + } + + impl Constructor for Overlapped { + type Args = (OptionalArg<isize>,); + + fn py_new(_cls: &Py<PyType>, (event,): Self::Args, vm: &VirtualMachine) -> PyResult<Self> { + let mut event = event.unwrap_or(INVALID_HANDLE_VALUE); + + if event == INVALID_HANDLE_VALUE { + event = unsafe { + windows_sys::Win32::System::Threading::CreateEventA( + core::ptr::null(), + Foundation::TRUE, + Foundation::FALSE, + core::ptr::null(), + ) as isize + }; + if event == NULL { + return Err(vm.new_last_os_error()); + } + } + + let mut overlapped: OVERLAPPED = unsafe { std::mem::zeroed() }; + if event != NULL { + overlapped.hEvent = event as HANDLE; + } + let inner = OverlappedInner { + overlapped, + handle: NULL as HANDLE, + error: 0, + data: OverlappedData::None, + }; + Ok(Overlapped { + inner: PyMutex::new(inner), + }) + } + } + + impl Destructor for Overlapped { + fn del(zelf: &Py<Self>, _vm: &VirtualMachine) -> PyResult<()> { + use windows_sys::Win32::System::IO::{CancelIoEx, GetOverlappedResult}; + + let mut inner = zelf.inner.lock(); + let olderr = unsafe { GetLastError() }; + + // Cancel pending I/O and wait for completion + if !HasOverlappedIoCompleted(&inner.overlapped) + && !matches!( + inner.data, + OverlappedData::None | OverlappedData::NotStarted + ) + { + let cancelled = unsafe { CancelIoEx(inner.handle, &inner.overlapped) } != 0; + + if cancelled { + // Wait for the cancellation to complete + let mut transferred: u32 = 0; + unsafe { + GetOverlappedResult( + inner.handle, + &inner.overlapped, + &mut transferred, + 1, // bWait = TRUE + ) + }; + } + } + + // Close the event handle + if !inner.overlapped.hEvent.is_null() { + unsafe { + Foundation::CloseHandle(inner.overlapped.hEvent); + } + inner.overlapped.hEvent = std::ptr::null_mut(); + } + + // Restore last error + unsafe { Foundation::SetLastError(olderr) }; + + Ok(()) + } + } + + #[pyfunction] + fn ConnectPipe(address: String, vm: &VirtualMachine) -> PyResult<isize> { + use windows_sys::Win32::Storage::FileSystem::{ + CreateFileW, FILE_FLAG_OVERLAPPED, FILE_GENERIC_READ, FILE_GENERIC_WRITE, OPEN_EXISTING, + }; + + let address_wide: Vec<u16> = address.encode_utf16().chain(std::iter::once(0)).collect(); + + let handle = unsafe { + CreateFileW( + address_wide.as_ptr(), + FILE_GENERIC_READ | FILE_GENERIC_WRITE, + 0, + std::ptr::null(), + OPEN_EXISTING, + FILE_FLAG_OVERLAPPED, + std::ptr::null_mut(), + ) + }; + + if handle == windows_sys::Win32::Foundation::INVALID_HANDLE_VALUE { + return Err(vm.new_last_os_error()); + } + + Ok(handle as isize) + } + + #[pyfunction] + fn CreateIoCompletionPort( + handle: isize, + port: isize, + key: usize, + concurrency: u32, + vm: &VirtualMachine, + ) -> PyResult<isize> { + let r = unsafe { + windows_sys::Win32::System::IO::CreateIoCompletionPort( + handle as HANDLE, + port as HANDLE, + key, + concurrency, + ) as isize + }; + if r == 0 { + return Err(vm.new_last_os_error()); + } + Ok(r) + } + + #[pyfunction] + fn GetQueuedCompletionStatus(port: isize, msecs: u32, vm: &VirtualMachine) -> PyResult { + let mut bytes_transferred = 0; + let mut completion_key = 0; + let mut overlapped: *mut OVERLAPPED = std::ptr::null_mut(); + let ret = unsafe { + windows_sys::Win32::System::IO::GetQueuedCompletionStatus( + port as HANDLE, + &mut bytes_transferred, + &mut completion_key, + &mut overlapped, + msecs, + ) + }; + let err = if ret != 0 { + Foundation::ERROR_SUCCESS + } else { + unsafe { GetLastError() } + }; + if overlapped.is_null() { + if err == Foundation::WAIT_TIMEOUT { + return Ok(vm.ctx.none()); + } else { + return Err(vm.new_last_os_error()); + } + } + + let value = vm.ctx.new_tuple(vec![ + err.to_pyobject(vm), bytes_transferred.to_pyobject(vm), + completion_key.to_pyobject(vm), (overlapped as usize).to_pyobject(vm), ]); Ok(value.into()) } + #[pyfunction] + fn PostQueuedCompletionStatus( + port: isize, + bytes: u32, + key: usize, + address: usize, + vm: &VirtualMachine, + ) -> PyResult<()> { + let ret = unsafe { + windows_sys::Win32::System::IO::PostQueuedCompletionStatus( + port as HANDLE, + bytes, + key, + address as *mut OVERLAPPED, + ) + }; + if ret == 0 { + return Err(vm.new_last_os_error()); + } + Ok(()) + } + + // Registry to track callback data for proper cleanup + // Uses Arc for reference counting to prevent use-after-free when callback + // and UnregisterWait race - the data stays alive until both are done + static WAIT_CALLBACK_REGISTRY: std::sync::OnceLock< + std::sync::Mutex<std::collections::HashMap<isize, std::sync::Arc<PostCallbackData>>>, + > = std::sync::OnceLock::new(); + + fn wait_callback_registry() + -> &'static std::sync::Mutex<std::collections::HashMap<isize, std::sync::Arc<PostCallbackData>>> + { + WAIT_CALLBACK_REGISTRY + .get_or_init(|| std::sync::Mutex::new(std::collections::HashMap::new())) + } + + // Callback data for RegisterWaitWithQueue + // Uses Arc to ensure the data stays alive while callback is executing + struct PostCallbackData { + completion_port: HANDLE, + overlapped: *mut OVERLAPPED, + } + + // SAFETY: The pointers are handles/addresses passed from Python and are + // only used to call Windows APIs. They are not dereferenced as Rust pointers. + unsafe impl Send for PostCallbackData {} + unsafe impl Sync for PostCallbackData {} + + unsafe extern "system" fn post_to_queue_callback( + parameter: *mut core::ffi::c_void, + timer_or_wait_fired: bool, + ) { + // Reconstruct Arc from raw pointer - this gives us ownership of one reference + // The Arc prevents use-after-free since we own a reference count + let data = unsafe { std::sync::Arc::from_raw(parameter as *const PostCallbackData) }; + + unsafe { + let _ = windows_sys::Win32::System::IO::PostQueuedCompletionStatus( + data.completion_port, + if timer_or_wait_fired { 1 } else { 0 }, + 0, + data.overlapped, + ); + } + // Arc is dropped here, decrementing refcount + // Memory is freed only when all references (callback + registry) are gone + } + + #[pyfunction] + fn RegisterWaitWithQueue( + object: isize, + completion_port: isize, + overlapped: usize, + timeout: u32, + vm: &VirtualMachine, + ) -> PyResult<isize> { + use windows_sys::Win32::System::Threading::{ + RegisterWaitForSingleObject, WT_EXECUTEINWAITTHREAD, WT_EXECUTEONLYONCE, + }; + + let data = std::sync::Arc::new(PostCallbackData { + completion_port: completion_port as HANDLE, + overlapped: overlapped as *mut OVERLAPPED, + }); + + // Create raw pointer for the callback - this increments refcount + let data_ptr = std::sync::Arc::into_raw(data.clone()); + + let mut new_wait_object: HANDLE = std::ptr::null_mut(); + let ret = unsafe { + RegisterWaitForSingleObject( + &mut new_wait_object, + object as HANDLE, + Some(post_to_queue_callback), + data_ptr as *mut _, + timeout, + WT_EXECUTEINWAITTHREAD | WT_EXECUTEONLYONCE, + ) + }; + + if ret == 0 { + // Registration failed - reconstruct Arc to drop the extra reference + unsafe { + let _ = std::sync::Arc::from_raw(data_ptr); + } + return Err(vm.new_last_os_error()); + } + + // Store in registry for cleanup tracking + let wait_handle = new_wait_object as isize; + if let Ok(mut registry) = wait_callback_registry().lock() { + registry.insert(wait_handle, data); + } + + Ok(wait_handle) + } + + // Helper to cleanup callback data when unregistering + // Just removes from registry - Arc ensures memory stays alive if callback is running + fn cleanup_wait_callback_data(wait_handle: isize) { + if let Ok(mut registry) = wait_callback_registry().lock() { + // Removing from registry drops one Arc reference + // If callback already ran, this frees the memory + // If callback is still pending/running, it holds the other reference + registry.remove(&wait_handle); + } + } + + #[pyfunction] + fn UnregisterWait(wait_handle: isize, vm: &VirtualMachine) -> PyResult<()> { + use windows_sys::Win32::System::Threading::UnregisterWait; + + let ret = unsafe { UnregisterWait(wait_handle as HANDLE) }; + // Cleanup callback data regardless of UnregisterWait result + // (callback may have already fired, or may never fire) + cleanup_wait_callback_data(wait_handle); + if ret == 0 { + return Err(vm.new_last_os_error()); + } + Ok(()) + } + + #[pyfunction] + fn UnregisterWaitEx(wait_handle: isize, event: isize, vm: &VirtualMachine) -> PyResult<()> { + use windows_sys::Win32::System::Threading::UnregisterWaitEx; + + let ret = unsafe { UnregisterWaitEx(wait_handle as HANDLE, event as HANDLE) }; + // Cleanup callback data regardless of UnregisterWaitEx result + cleanup_wait_callback_data(wait_handle); + if ret == 0 { + return Err(vm.new_last_os_error()); + } + Ok(()) + } + + #[pyfunction] + fn BindLocal(socket: isize, family: i32, vm: &VirtualMachine) -> PyResult<()> { + use windows_sys::Win32::Networking::WinSock::{INADDR_ANY, SOCKET_ERROR, bind}; + + let ret = if family == AF_INET as i32 { + let mut addr: SOCKADDR_IN = unsafe { std::mem::zeroed() }; + addr.sin_family = AF_INET; + addr.sin_port = 0; + addr.sin_addr.S_un.S_addr = INADDR_ANY; + unsafe { + bind( + socket as _, + &addr as *const _ as *const SOCKADDR, + std::mem::size_of::<SOCKADDR_IN>() as i32, + ) + } + } else if family == AF_INET6 as i32 { + // in6addr_any is all zeros, which we have from zeroed() + let mut addr: SOCKADDR_IN6 = unsafe { std::mem::zeroed() }; + addr.sin6_family = AF_INET6; + addr.sin6_port = 0; + unsafe { + bind( + socket as _, + &addr as *const _ as *const SOCKADDR, + std::mem::size_of::<SOCKADDR_IN6>() as i32, + ) + } + } else { + return Err(vm.new_value_error("family must be AF_INET or AF_INET6".to_owned())); + }; + + if ret == SOCKET_ERROR { + return Err(vm.new_last_os_error()); + } + Ok(()) + } + + #[pyfunction] + fn FormatMessage(error_code: u32, _vm: &VirtualMachine) -> PyResult<String> { + use windows_sys::Win32::Foundation::LocalFree; + use windows_sys::Win32::System::Diagnostics::Debug::{ + FORMAT_MESSAGE_ALLOCATE_BUFFER, FORMAT_MESSAGE_FROM_SYSTEM, + FORMAT_MESSAGE_IGNORE_INSERTS, FormatMessageW, + }; + + // LANG_NEUTRAL = 0, SUBLANG_DEFAULT = 1 + const LANG_NEUTRAL: u32 = 0; + const SUBLANG_DEFAULT: u32 = 1; + + let mut buffer: *mut u16 = std::ptr::null_mut(); + + let len = unsafe { + FormatMessageW( + FORMAT_MESSAGE_ALLOCATE_BUFFER + | FORMAT_MESSAGE_FROM_SYSTEM + | FORMAT_MESSAGE_IGNORE_INSERTS, + std::ptr::null(), + error_code, + (SUBLANG_DEFAULT << 10) | LANG_NEUTRAL, + &mut buffer as *mut _ as *mut u16, + 0, + std::ptr::null(), + ) + }; + + if len == 0 || buffer.is_null() { + return Ok(format!("unknown error code {}", error_code)); + } + + // Convert to Rust string, trimming trailing whitespace + let slice = unsafe { std::slice::from_raw_parts(buffer, len as usize) }; + let msg = String::from_utf16_lossy(slice).trim_end().to_string(); + + unsafe { LocalFree(buffer as *mut _) }; + + Ok(msg) + } + + #[pyfunction] + fn WSAConnect(socket: isize, address: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { + use windows_sys::Win32::Networking::WinSock::{SOCKET_ERROR, WSAConnect}; + + let (addr_bytes, addr_len) = parse_address(&address, vm)?; + + let ret = unsafe { + WSAConnect( + socket as _, + addr_bytes.as_ptr() as *const SOCKADDR, + addr_len, + std::ptr::null(), + std::ptr::null_mut(), + std::ptr::null(), + std::ptr::null(), + ) + }; + + if ret == SOCKET_ERROR { + return Err(vm.new_last_os_error()); + } + Ok(()) + } + #[pyfunction] fn CreateEvent( event_attributes: PyObjectRef, @@ -549,22 +1869,22 @@ mod _overlapped { vm: &VirtualMachine, ) -> PyResult<isize> { if !vm.is_none(&event_attributes) { - return Err(vm.new_value_error("EventAttributes must be None")); + return Err(vm.new_value_error("EventAttributes must be None".to_owned())); } - let name = match name { - Some(name) => { - let name = widestring::WideCString::from_str(&name).unwrap(); - name.as_ptr() - } - None => core::ptr::null(), - }; + let name_wide: Option<Vec<u16>> = + name.map(|n| n.encode_utf16().chain(std::iter::once(0)).collect()); + let name_ptr = name_wide + .as_ref() + .map(|n| n.as_ptr()) + .unwrap_or(std::ptr::null()); + let event = unsafe { windows_sys::Win32::System::Threading::CreateEventW( - core::ptr::null(), - manual_reset as _, - initial_state as _, - name, + std::ptr::null(), + if manual_reset { 1 } else { 0 }, + if initial_state { 1 } else { 0 }, + name_ptr, ) as isize }; if event == NULL { @@ -574,8 +1894,8 @@ mod _overlapped { } #[pyfunction] - fn SetEvent(handle: u64, vm: &VirtualMachine) -> PyResult<()> { - let ret = unsafe { windows_sys::Win32::System::Threading::SetEvent(u64_to_handle(handle)) }; + fn SetEvent(handle: isize, vm: &VirtualMachine) -> PyResult<()> { + let ret = unsafe { windows_sys::Win32::System::Threading::SetEvent(handle as HANDLE) }; if ret == 0 { return Err(vm.new_last_os_error()); } @@ -583,9 +1903,8 @@ mod _overlapped { } #[pyfunction] - fn ResetEvent(handle: u64, vm: &VirtualMachine) -> PyResult<()> { - let ret = - unsafe { windows_sys::Win32::System::Threading::ResetEvent(u64_to_handle(handle)) }; + fn ResetEvent(handle: isize, vm: &VirtualMachine) -> PyResult<()> { + let ret = unsafe { windows_sys::Win32::System::Threading::ResetEvent(handle as HANDLE) }; if ret == 0 { return Err(vm.new_last_os_error()); } diff --git a/crates/stdlib/src/posixshmem.rs b/crates/stdlib/src/posixshmem.rs index a52866b7985..2a142d8b6f3 100644 --- a/crates/stdlib/src/posixshmem.rs +++ b/crates/stdlib/src/posixshmem.rs @@ -1,5 +1,5 @@ #[cfg(all(unix, not(target_os = "redox"), not(target_os = "android")))] -pub(crate) use _posixshmem::make_module; +pub(crate) use _posixshmem::module_def; #[cfg(all(unix, not(target_os = "redox"), not(target_os = "android")))] #[pymodule] diff --git a/crates/stdlib/src/posixsubprocess.rs b/crates/stdlib/src/posixsubprocess.rs index a6badc081d2..3d5e4c356d9 100644 --- a/crates/stdlib/src/posixsubprocess.rs +++ b/crates/stdlib/src/posixsubprocess.rs @@ -22,7 +22,7 @@ use alloc::ffi::CString; use core::{convert::Infallible as Never, ffi::CStr, marker::PhantomData, ops::Deref}; -pub(crate) use _posixsubprocess::make_module; +pub(crate) use _posixsubprocess::module_def; #[pymodule] mod _posixsubprocess { diff --git a/crates/stdlib/src/pyexpat.rs b/crates/stdlib/src/pyexpat.rs index 89cf690770d..7d603c72ed5 100644 --- a/crates/stdlib/src/pyexpat.rs +++ b/crates/stdlib/src/pyexpat.rs @@ -2,18 +2,7 @@ // spell-checker: ignore libexpat -use crate::vm::{PyRef, VirtualMachine, builtins::PyModule, extend_module}; - -pub fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - let module = _pyexpat::make_module(vm); - - extend_module!(vm, &module, { - "errors" => _errors::make_module(vm), - "model" => _model::make_module(vm), - }); - - module -} +pub(crate) use _pyexpat::module_def; macro_rules! create_property { ($ctx: expr, $attributes: expr, $name: expr, $class: expr, $element: ident) => { @@ -52,14 +41,30 @@ macro_rules! create_bool_property { mod _pyexpat { use crate::vm::{ Context, Py, PyObjectRef, PyPayload, PyRef, PyResult, TryFromObject, VirtualMachine, - builtins::{PyBytesRef, PyStr, PyStrRef, PyType}, - function::ArgBytesLike, - function::{Either, IntoFuncArgs, OptionalArg}, + builtins::{PyBytesRef, PyException, PyModule, PyStr, PyStrRef, PyType}, + extend_module, + function::{ArgBytesLike, Either, IntoFuncArgs, OptionalArg}, + types::Constructor, }; use rustpython_common::lock::PyRwLock; use std::io::Cursor; use xml::reader::XmlEvent; + pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { + __module_exec(vm, module); + + // Add submodules + let model = super::_model::module_def(&vm.ctx).create_module(vm)?; + let errors = super::_errors::module_def(&vm.ctx).create_module(vm)?; + + extend_module!(vm, module, { + "model" => model, + "errors" => errors, + }); + + Ok(()) + } + type MutableObject = PyRwLock<PyObjectRef>; #[pyattr(name = "version_info")] @@ -415,6 +420,17 @@ mod _pyexpat { PyExpatLikeXmlParser::new(ns_sep, args.intern, vm) } + + // TODO: Tie this exception to the module's state. + #[pyattr] + #[pyattr(name = "error")] + #[pyexception(name = "ExpatError", base = PyException)] + #[derive(Debug)] + #[repr(transparent)] + pub struct PyExpatError(PyException); + + #[pyexception] + impl PyExpatError {} } #[pymodule(name = "model")] diff --git a/crates/stdlib/src/pystruct.rs b/crates/stdlib/src/pystruct.rs index 8801f0d705e..d3be417edb3 100644 --- a/crates/stdlib/src/pystruct.rs +++ b/crates/stdlib/src/pystruct.rs @@ -5,7 +5,7 @@ //! Use this rust module to do byte packing: //! <https://docs.rs/byteorder/1.2.6/byteorder/> -pub(crate) use _struct::make_module; +pub(crate) use _struct::module_def; #[pymodule] pub(crate) mod _struct { diff --git a/crates/stdlib/src/random.rs b/crates/stdlib/src/random.rs index be31d3011d7..35e6473d8f6 100644 --- a/crates/stdlib/src/random.rs +++ b/crates/stdlib/src/random.rs @@ -1,6 +1,6 @@ //! Random module. -pub(crate) use _random::make_module; +pub(crate) use _random::module_def; #[pymodule] mod _random { diff --git a/crates/stdlib/src/re.rs b/crates/stdlib/src/re.rs index 5af45671522..fdb14d427fc 100644 --- a/crates/stdlib/src/re.rs +++ b/crates/stdlib/src/re.rs @@ -1,4 +1,4 @@ -pub(crate) use re::make_module; +pub(crate) use re::module_def; #[pymodule] mod re { diff --git a/crates/stdlib/src/resource.rs b/crates/stdlib/src/resource.rs index e6df75e4b01..34c8161e0cd 100644 --- a/crates/stdlib/src/resource.rs +++ b/crates/stdlib/src/resource.rs @@ -1,6 +1,6 @@ // spell-checker:disable -pub(crate) use resource::make_module; +pub(crate) use resource::module_def; #[pymodule] mod resource { diff --git a/crates/stdlib/src/scproxy.rs b/crates/stdlib/src/scproxy.rs index b105dc4f2fb..09e7cdc6046 100644 --- a/crates/stdlib/src/scproxy.rs +++ b/crates/stdlib/src/scproxy.rs @@ -1,4 +1,4 @@ -pub(crate) use _scproxy::make_module; +pub(crate) use _scproxy::module_def; #[pymodule] mod _scproxy { diff --git a/crates/stdlib/src/select.rs b/crates/stdlib/src/select.rs index 3c2f5e63c7c..bc8aded5478 100644 --- a/crates/stdlib/src/select.rs +++ b/crates/stdlib/src/select.rs @@ -1,25 +1,13 @@ // spell-checker:disable +pub(crate) use decl::module_def; + use crate::vm::{ - PyObject, PyObjectRef, PyRef, PyResult, TryFromObject, VirtualMachine, builtins::PyListRef, - builtins::PyModule, + PyObject, PyObjectRef, PyResult, TryFromObject, VirtualMachine, builtins::PyListRef, }; use core::mem; use std::io; -pub(crate) fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - #[cfg(windows)] - crate::vm::windows::init_winsock(); - - #[cfg(unix)] - { - use crate::vm::class::PyClassImpl; - decl::poll::PyPoll::make_class(&vm.ctx); - } - - decl::make_module(vm) -} - #[cfg(unix)] mod platform { pub use libc::{FD_ISSET, FD_SET, FD_SETSIZE, FD_ZERO, fd_set, select, timeval}; @@ -73,7 +61,7 @@ mod platform { #[cfg(target_os = "wasi")] mod platform { pub use libc::{FD_SETSIZE, timeval}; - pub use std::os::wasi::io::RawFd; + pub use std::os::fd::RawFd; pub fn check_err(x: i32) -> bool { x < 0 @@ -221,13 +209,27 @@ fn sec_to_timeval(sec: f64) -> timeval { mod decl { use super::*; use crate::vm::{ - PyObjectRef, PyResult, VirtualMachine, - builtins::PyTypeRef, + Py, PyObjectRef, PyResult, VirtualMachine, + builtins::{PyModule, PyTypeRef}, convert::ToPyException, function::{Either, OptionalOption}, stdlib::time, }; + pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { + #[cfg(windows)] + crate::vm::windows::init_winsock(); + + #[cfg(unix)] + { + use crate::vm::class::PyClassImpl; + poll::PyPoll::make_class(&vm.ctx); + } + + __module_exec(vm, module); + Ok(()) + } + #[pyattr] fn error(vm: &VirtualMachine) -> PyTypeRef { vm.ctx.exceptions.os_error.to_owned() @@ -545,7 +547,7 @@ mod decl { pub(super) mod epoll { use super::*; use crate::vm::{ - Py, PyPayload, + Py, PyPayload, PyRef, builtins::PyType, common::lock::{PyRwLock, PyRwLockReadGuard}, convert::{IntoPyException, ToPyObject}, diff --git a/crates/stdlib/src/sha1.rs b/crates/stdlib/src/sha1.rs index 04845bb76b5..28d4bcbbee1 100644 --- a/crates/stdlib/src/sha1.rs +++ b/crates/stdlib/src/sha1.rs @@ -1,4 +1,4 @@ -pub(crate) use _sha1::make_module; +pub(crate) use _sha1::module_def; #[pymodule] mod _sha1 { diff --git a/crates/stdlib/src/sha256.rs b/crates/stdlib/src/sha256.rs index 5d031968aeb..bf4a39373ea 100644 --- a/crates/stdlib/src/sha256.rs +++ b/crates/stdlib/src/sha256.rs @@ -1,14 +1,7 @@ -use crate::vm::{PyRef, VirtualMachine, builtins::PyModule}; - -pub(crate) fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - let _ = vm.import("_hashlib", 0); - _sha256::make_module(vm) -} - #[pymodule] mod _sha256 { use crate::hashlib::_hashlib::{HashArgs, local_sha224, local_sha256}; - use crate::vm::{PyPayload, PyResult, VirtualMachine}; + use crate::vm::{Py, PyPayload, PyResult, VirtualMachine, builtins::PyModule}; #[pyfunction] fn sha224(args: HashArgs, vm: &VirtualMachine) -> PyResult { @@ -19,4 +12,12 @@ mod _sha256 { fn sha256(args: HashArgs, vm: &VirtualMachine) -> PyResult { Ok(local_sha256(args).into_pyobject(vm)) } + + pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { + let _ = vm.import("_hashlib", 0); + __module_exec(vm, module); + Ok(()) + } } + +pub(crate) use _sha256::module_def; diff --git a/crates/stdlib/src/sha3.rs b/crates/stdlib/src/sha3.rs index 07b61d9aed2..e7342d0dd69 100644 --- a/crates/stdlib/src/sha3.rs +++ b/crates/stdlib/src/sha3.rs @@ -1,4 +1,4 @@ -pub(crate) use _sha3::make_module; +pub(crate) use _sha3::module_def; #[pymodule] mod _sha3 { diff --git a/crates/stdlib/src/sha512.rs b/crates/stdlib/src/sha512.rs index baf63fdacf0..303c0067daa 100644 --- a/crates/stdlib/src/sha512.rs +++ b/crates/stdlib/src/sha512.rs @@ -1,14 +1,7 @@ -use crate::vm::{PyRef, VirtualMachine, builtins::PyModule}; - -pub(crate) fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - let _ = vm.import("_hashlib", 0); - _sha512::make_module(vm) -} - #[pymodule] mod _sha512 { use crate::hashlib::_hashlib::{HashArgs, local_sha384, local_sha512}; - use crate::vm::{PyPayload, PyResult, VirtualMachine}; + use crate::vm::{Py, PyPayload, PyResult, VirtualMachine, builtins::PyModule}; #[pyfunction] fn sha384(args: HashArgs, vm: &VirtualMachine) -> PyResult { @@ -19,4 +12,12 @@ mod _sha512 { fn sha512(args: HashArgs, vm: &VirtualMachine) -> PyResult { Ok(local_sha512(args).into_pyobject(vm)) } + + pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { + let _ = vm.import("_hashlib", 0); + __module_exec(vm, module); + Ok(()) + } } + +pub(crate) use _sha512::module_def; diff --git a/crates/stdlib/src/socket.rs b/crates/stdlib/src/socket.rs index ea5f39cb40d..6b33fe52e37 100644 --- a/crates/stdlib/src/socket.rs +++ b/crates/stdlib/src/socket.rs @@ -1,27 +1,35 @@ // spell-checker:disable -use crate::vm::{PyRef, VirtualMachine, builtins::PyModule}; +pub(crate) use _socket::module_def; + #[cfg(feature = "ssl")] pub(super) use _socket::{PySocket, SelectKind, sock_select, timeout_error_msg}; -pub fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - #[cfg(windows)] - crate::vm::windows::init_winsock(); - _socket::make_module(vm) -} - #[pymodule] mod _socket { use crate::common::lock::{PyMappedRwLockReadGuard, PyRwLock, PyRwLockReadGuard}; use crate::vm::{ AsObject, Py, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, - builtins::{PyBaseExceptionRef, PyListRef, PyOSError, PyStrRef, PyTupleRef, PyTypeRef}, + builtins::{ + PyBaseExceptionRef, PyListRef, PyModule, PyOSError, PyStrRef, PyTupleRef, PyTypeRef, + }, common::os::ErrorExt, convert::{IntoPyException, ToPyObject, TryFromBorrowedObject, TryFromObject}, - function::{ArgBytesLike, ArgMemoryBuffer, Either, FsPath, OptionalArg, OptionalOption}, + function::{ + ArgBytesLike, ArgMemoryBuffer, ArgStrOrBytesLike, Either, FsPath, OptionalArg, + OptionalOption, + }, types::{Constructor, DefaultConstructor, Initializer, Representable}, utils::ToCString, }; + + pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { + #[cfg(windows)] + crate::vm::windows::init_winsock(); + + __module_exec(vm, module); + Ok(()) + } use core::{ mem::MaybeUninit, net::{Ipv4Addr, Ipv6Addr, SocketAddr}, @@ -62,10 +70,10 @@ mod _socket { NI_MAXHOST, NI_MAXSERV, NI_NAMEREQD, NI_NOFQDN, NI_NUMERICHOST, NI_NUMERICSERV, RCVALL_IPLEVEL, RCVALL_OFF, RCVALL_ON, RCVALL_SOCKETLEVELONLY, SD_BOTH as SHUT_RDWR, SD_RECEIVE as SHUT_RD, SD_SEND as SHUT_WR, SIO_KEEPALIVE_VALS, SIO_LOOPBACK_FAST_PATH, - SIO_RCVALL, SO_BROADCAST, SO_ERROR, SO_LINGER, SO_OOBINLINE, SO_REUSEADDR, SO_TYPE, - SO_USELOOPBACK, SOCK_DGRAM, SOCK_RAW, SOCK_RDM, SOCK_SEQPACKET, SOCK_STREAM, - SOL_SOCKET, SOMAXCONN, TCP_NODELAY, WSAEBADF, WSAECONNRESET, WSAENOTSOCK, - WSAEWOULDBLOCK, + SIO_RCVALL, SO_BROADCAST, SO_ERROR, SO_KEEPALIVE, SO_LINGER, SO_OOBINLINE, SO_RCVBUF, + SO_REUSEADDR, SO_SNDBUF, SO_TYPE, SO_USELOOPBACK, SOCK_DGRAM, SOCK_RAW, SOCK_RDM, + SOCK_SEQPACKET, SOCK_STREAM, SOL_SOCKET, SOMAXCONN, TCP_NODELAY, WSAEBADF, + WSAECONNRESET, WSAENOTSOCK, WSAEWOULDBLOCK, }; pub use windows_sys::Win32::Networking::WinSock::{ INVALID_SOCKET, SOCKET_ERROR, WSA_FLAG_OVERLAPPED, WSADuplicateSocketW, @@ -102,8 +110,8 @@ mod _socket { IPPROTO_ICMPV6, IPPROTO_IP, IPPROTO_IPV6, IPPROTO_TCP, IPPROTO_TCP as SOL_TCP, IPPROTO_UDP, MSG_CTRUNC, MSG_DONTROUTE, MSG_OOB, MSG_PEEK, MSG_TRUNC, MSG_WAITALL, NI_DGRAM, NI_MAXHOST, NI_NAMEREQD, NI_NOFQDN, NI_NUMERICHOST, NI_NUMERICSERV, SHUT_RD, SHUT_RDWR, SHUT_WR, - SO_BROADCAST, SO_ERROR, SO_LINGER, SO_OOBINLINE, SO_REUSEADDR, SO_TYPE, SOCK_DGRAM, - SOCK_STREAM, SOL_SOCKET, TCP_NODELAY, + SO_BROADCAST, SO_ERROR, SO_KEEPALIVE, SO_LINGER, SO_OOBINLINE, SO_RCVBUF, SO_REUSEADDR, + SO_SNDBUF, SO_TYPE, SOCK_DGRAM, SOCK_STREAM, SOL_SOCKET, TCP_NODELAY, }; #[cfg(not(target_os = "redox"))] @@ -295,8 +303,8 @@ mod _socket { #[cfg(any(unix, target_os = "android"))] #[pyattr] use c::{ - EAI_SYSTEM, MSG_EOR, SO_ACCEPTCONN, SO_DEBUG, SO_DONTROUTE, SO_KEEPALIVE, SO_RCVBUF, - SO_RCVLOWAT, SO_RCVTIMEO, SO_SNDBUF, SO_SNDLOWAT, SO_SNDTIMEO, + EAI_SYSTEM, MSG_EOR, SO_ACCEPTCONN, SO_DEBUG, SO_DONTROUTE, SO_RCVLOWAT, SO_RCVTIMEO, + SO_SNDLOWAT, SO_SNDTIMEO, }; #[cfg(any(target_os = "android", target_os = "linux"))] @@ -2778,9 +2786,9 @@ mod _socket { #[derive(FromArgs)] struct GAIOptions { #[pyarg(positional)] - host: Option<PyStrRef>, + host: Option<ArgStrOrBytesLike>, #[pyarg(positional)] - port: Option<Either<PyStrRef, i32>>, + port: Option<Either<ArgStrOrBytesLike, i32>>, #[pyarg(positional, default = c::AF_UNSPEC)] family: i32, @@ -2804,9 +2812,9 @@ mod _socket { flags: opts.flags, }; - // Encode host using IDNA encoding + // Encode host: str uses IDNA encoding, bytes must be valid UTF-8 let host_encoded: Option<String> = match opts.host.as_ref() { - Some(s) => { + Some(ArgStrOrBytesLike::Str(s)) => { let encoded = vm.state .codec_registry @@ -2815,19 +2823,43 @@ mod _socket { .map_err(|_| vm.new_runtime_error("idna output is not utf8".to_owned()))?; Some(host_str.to_owned()) } + Some(ArgStrOrBytesLike::Buf(b)) => { + let bytes = b.borrow_buf(); + let host_str = core::str::from_utf8(&bytes).map_err(|_| { + vm.new_unicode_decode_error("host bytes is not utf8".to_owned()) + })?; + Some(host_str.to_owned()) + } None => None, }; let host = host_encoded.as_deref(); - // Encode port using UTF-8 - let port: Option<alloc::borrow::Cow<'_, str>> = match opts.port.as_ref() { - Some(Either::A(s)) => Some(alloc::borrow::Cow::Borrowed(s.to_str().ok_or_else( - || vm.new_unicode_encode_error("surrogates not allowed".to_owned()), - )?)), - Some(Either::B(i)) => Some(alloc::borrow::Cow::Owned(i.to_string())), + // Encode port: str/bytes as service name, int as port number + let port_encoded: Option<String> = match opts.port.as_ref() { + Some(Either::A(sb)) => { + let port_str = match sb { + ArgStrOrBytesLike::Str(s) => { + // For str, check for surrogates and raise UnicodeEncodeError if found + s.to_str() + .ok_or_else(|| vm.new_unicode_encode_error("surrogates not allowed"))? + .to_owned() + } + ArgStrOrBytesLike::Buf(b) => { + // For bytes, check if it's valid UTF-8 + let bytes = b.borrow_buf(); + core::str::from_utf8(&bytes) + .map_err(|_| { + vm.new_unicode_decode_error("port is not utf8".to_owned()) + })? + .to_owned() + } + }; + Some(port_str) + } + Some(Either::B(i)) => Some(i.to_string()), None => None, }; - let port = port.as_ref().map(|p| p.as_ref()); + let port = port_encoded.as_deref(); let addrs = dns_lookup::getaddrinfo(host, port, Some(hints)) .map_err(|err| convert_socket_error(vm, err, SocketError::GaiError))?; diff --git a/crates/stdlib/src/ssl.rs b/crates/stdlib/src/ssl.rs index 5a219baaf7f..5bb6c259e66 100644 --- a/crates/stdlib/src/ssl.rs +++ b/crates/stdlib/src/ssl.rs @@ -1,4 +1,4 @@ -// spell-checker: ignore ssleof aesccm aesgcm getblocking setblocking ENDTLS +// spell-checker: ignore ssleof aesccm aesgcm capath getblocking setblocking ENDTLS TLSEXT //! Pure Rust SSL/TLS implementation using rustls //! @@ -25,7 +25,7 @@ mod compat; // SSL exception types (shared with openssl backend) mod error; -pub(crate) use _ssl::make_module; +pub(crate) use _ssl::module_def; #[allow(non_snake_case)] #[allow(non_upper_case_globals)] @@ -42,7 +42,9 @@ mod _ssl { VirtualMachine, builtins::{PyBaseExceptionRef, PyBytesRef, PyListRef, PyStrRef, PyType, PyTypeRef}, convert::IntoPyException, - function::{ArgBytesLike, ArgMemoryBuffer, FuncArgs, OptionalArg, PyComparisonValue}, + function::{ + ArgBytesLike, ArgMemoryBuffer, Either, FuncArgs, OptionalArg, PyComparisonValue, + }, stdlib::warnings, types::{Comparable, Constructor, Hashable, PyComparisonOp, Representable}, }, @@ -50,7 +52,8 @@ mod _ssl { // Import error types used in this module (others are exposed via pymodule(with(...))) use super::error::{ - PySSLEOFError, PySSLError, create_ssl_want_read_error, create_ssl_want_write_error, + PySSLError, create_ssl_eof_error, create_ssl_want_read_error, create_ssl_want_write_error, + create_ssl_zero_return_error, }; use alloc::sync::Arc; use core::{ @@ -821,20 +824,20 @@ mod _ssl { #[derive(FromArgs)] struct LoadVerifyLocationsArgs { - #[pyarg(any, optional)] - cafile: OptionalArg<Option<PyObjectRef>>, - #[pyarg(any, optional)] - capath: OptionalArg<Option<PyObjectRef>>, - #[pyarg(any, optional)] - cadata: OptionalArg<PyObjectRef>, + #[pyarg(any, optional, error_msg = "path should be a str or bytes")] + cafile: OptionalArg<Option<Either<PyStrRef, ArgBytesLike>>>, + #[pyarg(any, optional, error_msg = "path should be a str or bytes")] + capath: OptionalArg<Option<Either<PyStrRef, ArgBytesLike>>>, + #[pyarg(any, optional, error_msg = "cadata should be a str or bytes")] + cadata: OptionalArg<Option<Either<PyStrRef, ArgBytesLike>>>, } #[derive(FromArgs)] struct LoadCertChainArgs { - #[pyarg(any)] - certfile: PyObjectRef, - #[pyarg(any, optional)] - keyfile: OptionalArg<Option<PyObjectRef>>, + #[pyarg(any, error_msg = "path should be a str or bytes")] + certfile: Either<PyStrRef, ArgBytesLike>, + #[pyarg(any, optional, error_msg = "path should be a str or bytes")] + keyfile: OptionalArg<Option<Either<PyStrRef, ArgBytesLike>>>, #[pyarg(any, optional)] password: OptionalArg<PyObjectRef>, } @@ -1229,7 +1232,7 @@ mod _ssl { // Check that at least one argument is provided let has_cafile = matches!(&args.cafile, OptionalArg::Present(Some(_))); let has_capath = matches!(&args.capath, OptionalArg::Present(Some(_))); - let has_cadata = matches!(&args.cadata, OptionalArg::Present(obj) if !vm.is_none(obj)); + let has_cadata = matches!(&args.cadata, OptionalArg::Present(Some(_))); if !has_cafile && !has_capath && !has_cadata { return Err( @@ -1250,10 +1253,8 @@ mod _ssl { None }; - let cadata_parsed = if let OptionalArg::Present(ref cadata_obj) = args.cadata - && !vm.is_none(cadata_obj) - { - let is_string = PyStrRef::try_from_object(vm, cadata_obj.clone()).is_ok(); + let cadata_parsed = if let OptionalArg::Present(Some(ref cadata_obj)) = args.cadata { + let is_string = matches!(cadata_obj, Either::A(_)); let data_vec = self.parse_cadata_arg(cadata_obj, vm)?; Some((data_vec, is_string)) } else { @@ -1903,6 +1904,7 @@ mod _ssl { client_hello_buffer: PyMutex::new(None), shutdown_state: PyMutex::new(ShutdownState::NotStarted), pending_tls_output: PyMutex::new(Vec::new()), + write_buffered_len: PyMutex::new(0), deferred_cert_error: Arc::new(ParkingRwLock::new(None)), }; @@ -1974,6 +1976,7 @@ mod _ssl { client_hello_buffer: PyMutex::new(None), shutdown_state: PyMutex::new(ShutdownState::NotStarted), pending_tls_output: PyMutex::new(Vec::new()), + write_buffered_len: PyMutex::new(0), deferred_cert_error: Arc::new(ParkingRwLock::new(None)), }; @@ -1987,14 +1990,14 @@ mod _ssl { // Helper functions (private): /// Parse path argument (str or bytes) to string - fn parse_path_arg(arg: &PyObject, vm: &VirtualMachine) -> PyResult<String> { - if let Ok(s) = PyStrRef::try_from_object(vm, arg.to_owned()) { - Ok(s.as_str().to_owned()) - } else if let Ok(b) = ArgBytesLike::try_from_object(vm, arg.to_owned()) { - String::from_utf8(b.borrow_buf().to_vec()) - .map_err(|_| vm.new_value_error("path contains invalid UTF-8".to_owned())) - } else { - Err(vm.new_type_error("path should be a str or bytes".to_owned())) + fn parse_path_arg( + arg: &Either<PyStrRef, ArgBytesLike>, + vm: &VirtualMachine, + ) -> PyResult<String> { + match arg { + Either::A(s) => Ok(s.as_str().to_owned()), + Either::B(b) => String::from_utf8(b.borrow_buf().to_vec()) + .map_err(|_| vm.new_value_error("path contains invalid UTF-8".to_owned())), } } @@ -2165,13 +2168,14 @@ mod _ssl { } /// Helper: Parse cadata argument (str or bytes) - fn parse_cadata_arg(&self, arg: &PyObject, vm: &VirtualMachine) -> PyResult<Vec<u8>> { - if let Ok(s) = PyStrRef::try_from_object(vm, arg.to_owned()) { - Ok(s.as_str().as_bytes().to_vec()) - } else if let Ok(b) = ArgBytesLike::try_from_object(vm, arg.to_owned()) { - Ok(b.borrow_buf().to_vec()) - } else { - Err(vm.new_type_error("cadata should be a str or bytes".to_owned())) + fn parse_cadata_arg( + &self, + arg: &Either<PyStrRef, ArgBytesLike>, + _vm: &VirtualMachine, + ) -> PyResult<Vec<u8>> { + match arg { + Either::A(s) => Ok(s.as_str().as_bytes().to_vec()), + Either::B(b) => Ok(b.borrow_buf().to_vec()), } } @@ -2345,6 +2349,10 @@ mod _ssl { // but the socket cannot accept all the data immediately #[pytraverse(skip)] pub(crate) pending_tls_output: PyMutex<Vec<u8>>, + // Tracks bytes already buffered in rustls for the current write operation + // Prevents duplicate writes when retrying after WantWrite/WantRead + #[pytraverse(skip)] + pub(crate) write_buffered_len: PyMutex<usize>, // Deferred client certificate verification error (for TLS 1.3) // Stores error message if client cert verification failed during handshake // Error is raised on first I/O operation after handshake @@ -2604,6 +2612,36 @@ mod _ssl { Ok(timed_out) } + // Internal implementation with explicit timeout override + pub(crate) fn sock_wait_for_io_with_timeout( + &self, + kind: SelectKind, + timeout: Option<std::time::Duration>, + vm: &VirtualMachine, + ) -> PyResult<bool> { + if self.is_bio_mode() { + // BIO mode doesn't use select + return Ok(false); + } + + if let Some(t) = timeout + && t.is_zero() + { + // Non-blocking mode - don't use select + return Ok(false); + } + + let py_socket: PyRef<PySocket> = self.sock.clone().try_into_value(vm)?; + let socket = py_socket + .sock() + .map_err(|e| vm.new_os_error(format!("Failed to get socket: {e}")))?; + + let timed_out = sock_select(&socket, kind, timeout) + .map_err(|e| vm.new_os_error(format!("select failed: {e}")))?; + + Ok(timed_out) + } + // SNI (Server Name Indication) Helper Methods: // These methods support the server-side handshake SNI callback mechanism @@ -2652,7 +2690,31 @@ mod _ssl { }; let initial_context: PyObjectRef = self.context.read().clone().into(); - let result = callback.call((ssl_sock, server_name_py, initial_context), vm)?; + // catches exceptions from the callback and reports them as unraisable + let result = match callback.call((ssl_sock, server_name_py, initial_context), vm) { + Ok(result) => result, + Err(exc) => { + vm.run_unraisable( + exc, + Some("in ssl servername callback".to_owned()), + callback.clone(), + ); + // Return SSL error like SSL_TLSEXT_ERR_ALERT_FATAL + let ssl_exc: PyBaseExceptionRef = vm + .new_os_subtype_error( + PySSLError::class(&vm.ctx).to_owned(), + None, + "SNI callback raised exception", + ) + .upcast(); + let _ = ssl_exc.as_object().set_attr( + "reason", + vm.ctx.new_str("TLSV1_ALERT_INTERNAL_ERROR"), + vm, + ); + return Err(ssl_exc); + } + }; // Check return value type (must be None or integer) if !vm.is_none(&result) { @@ -2724,6 +2786,16 @@ mod _ssl { recv_method.call((self.sock.clone(), vm.ctx.new_int(size)), vm) } + /// Peek at socket data without consuming it (MSG_PEEK). + /// Used during TLS shutdown to avoid consuming post-TLS cleartext data. + pub(crate) fn sock_peek(&self, size: usize, vm: &VirtualMachine) -> PyResult<PyObjectRef> { + let socket_mod = vm.import("socket", 0)?; + let socket_class = socket_mod.get_attr("socket", vm)?; + let recv_method = socket_class.get_attr("recv", vm)?; + let msg_peek = socket_mod.get_attr("MSG_PEEK", vm)?; + recv_method.call((self.sock.clone(), vm.ctx.new_int(size), msg_peek), vm) + } + /// Socket send - just sends data, caller must handle pending flush /// Use flush_pending_tls_output before this if ordering is important pub(crate) fn sock_send(&self, data: &[u8], vm: &VirtualMachine) -> PyResult<PyObjectRef> { @@ -2759,6 +2831,7 @@ mod _ssl { let is_non_blocking = socket_timeout.map(|t| t.is_zero()).unwrap_or(false); let mut sent_total = 0; + while sent_total < pending.len() { // Calculate timeout: use deadline if provided, otherwise use socket timeout let timeout_to_use = if let Some(dl) = deadline { @@ -2786,6 +2859,9 @@ mod _ssl { if timed_out { // Keep unsent data in pending buffer *pending = pending[sent_total..].to_vec(); + if is_non_blocking { + return Err(create_ssl_want_write_error(vm).upcast()); + } return Err( timeout_error_msg(vm, "The write operation timed out".to_string()).upcast(), ); @@ -2800,6 +2876,7 @@ mod _ssl { *pending = pending[sent_total..].to_vec(); return Err(create_ssl_want_write_error(vm).upcast()); } + // Socket said ready but sent 0 bytes - retry continue; } sent_total += sent; @@ -2892,6 +2969,9 @@ mod _ssl { pub(crate) fn blocking_flush_all_pending(&self, vm: &VirtualMachine) -> PyResult<()> { // Get socket timeout to respect during flush let timeout = self.get_socket_timeout(vm)?; + if timeout.map(|t| t.is_zero()).unwrap_or(false) { + return self.flush_pending_tls_output(vm, None); + } loop { let pending_data = { @@ -2924,8 +3004,7 @@ mod _ssl { let mut pending = self.pending_tls_output.lock(); pending.drain(..sent); } - // If sent == 0, socket wasn't ready despite select() saying so - // Continue loop to retry - this avoids infinite loops + // If sent == 0, loop will retry with sock_select } Err(e) => { if is_blocking_io_error(&e, vm) { @@ -3436,9 +3515,9 @@ mod _ssl { } // Check if connection has been shut down - // After unwrap()/shutdown(), read operations should fail with SSLError + // Only block after shutdown is COMPLETED, not during shutdown process let shutdown_state = *self.shutdown_state.lock(); - if shutdown_state != ShutdownState::NotStarted { + if shutdown_state == ShutdownState::Completed { return Err(vm .new_os_subtype_error( PySSLError::class(&vm.ctx).to_owned(), @@ -3491,18 +3570,74 @@ mod _ssl { return_data(buf, &buffer, vm) } Err(crate::ssl::compat::SslError::Eof) => { + // If plaintext is still buffered, return it before EOF. + let pending = { + let mut conn_guard = self.connection.lock(); + let conn = match conn_guard.as_mut() { + Some(conn) => conn, + None => return Err(create_ssl_eof_error(vm).upcast()), + }; + use std::io::BufRead; + let mut reader = conn.reader(); + reader.fill_buf().map(|buf| buf.len()).unwrap_or(0) + }; + if pending > 0 { + let mut buf = vec![0u8; pending.min(len)]; + let read_retry = { + let mut conn_guard = self.connection.lock(); + let conn = conn_guard + .as_mut() + .ok_or_else(|| vm.new_value_error("Connection not established"))?; + crate::ssl::compat::ssl_read(conn, &mut buf, self, vm) + }; + if let Ok(n) = read_retry { + buf.truncate(n); + return return_data(buf, &buffer, vm); + } + } // EOF occurred in violation of protocol (unexpected closure) - Err(vm - .new_os_subtype_error( - PySSLEOFError::class(&vm.ctx).to_owned(), - None, - "EOF occurred in violation of protocol", - ) - .upcast()) + Err(create_ssl_eof_error(vm).upcast()) } Err(crate::ssl::compat::SslError::ZeroReturn) => { - // Clean closure with close_notify - return empty data - return_data(vec![], &buffer, vm) + // If plaintext is still buffered, return it before clean EOF. + let pending = { + let mut conn_guard = self.connection.lock(); + let conn = match conn_guard.as_mut() { + Some(conn) => conn, + None => return Err(create_ssl_zero_return_error(vm).upcast()), + }; + use std::io::BufRead; + let mut reader = conn.reader(); + reader.fill_buf().map(|buf| buf.len()).unwrap_or(0) + }; + if pending > 0 { + let mut buf = vec![0u8; pending.min(len)]; + let read_retry = { + let mut conn_guard = self.connection.lock(); + let conn = conn_guard + .as_mut() + .ok_or_else(|| vm.new_value_error("Connection not established"))?; + crate::ssl::compat::ssl_read(conn, &mut buf, self, vm) + }; + if let Ok(n) = read_retry { + buf.truncate(n); + return return_data(buf, &buffer, vm); + } + } + // Clean closure with close_notify + // CPython behavior depends on whether we've sent our close_notify: + // - If we've already sent close_notify (unwrap was called): raise SSLZeroReturnError + // - If we haven't sent close_notify yet: return empty bytes + let our_shutdown_state = *self.shutdown_state.lock(); + if our_shutdown_state == ShutdownState::SentCloseNotify + || our_shutdown_state == ShutdownState::Completed + { + // We already sent close_notify, now receiving peer's → SSLZeroReturnError + Err(create_ssl_zero_return_error(vm).upcast()) + } else { + // We haven't sent close_notify yet → return empty bytes + return_data(vec![], &buffer, vm) + } } Err(crate::ssl::compat::SslError::WantRead) => { // Non-blocking mode: would block @@ -3556,21 +3691,18 @@ mod _ssl { let data_bytes = data.borrow_buf(); let data_len = data_bytes.len(); - // return 0 immediately for empty write if data_len == 0 { return Ok(0); } - // Ensure handshake is done - if not, complete it first - // This matches OpenSSL behavior where SSL_write() auto-completes handshake + // Ensure handshake is done (SSL_write auto-completes handshake) if !*self.handshake_done.lock() { self.do_handshake(vm)?; } - // Check if connection has been shut down - // After unwrap()/shutdown(), write operations should fail with SSLError - let shutdown_state = *self.shutdown_state.lock(); - if shutdown_state != ShutdownState::NotStarted { + // Check shutdown state + // Only block after shutdown is COMPLETED, not during shutdown process + if *self.shutdown_state.lock() == ShutdownState::Completed { return Err(vm .new_os_subtype_error( PySSLError::class(&vm.ctx).to_owned(), @@ -3580,76 +3712,32 @@ mod _ssl { .upcast()); } - { + // Call ssl_write (matches CPython's SSL_write_ex loop) + let result = { let mut conn_guard = self.connection.lock(); let conn = conn_guard .as_mut() .ok_or_else(|| vm.new_value_error("Connection not established"))?; - let is_bio = self.is_bio_mode(); - let data: &[u8] = data_bytes.as_ref(); + crate::ssl::compat::ssl_write(conn, data_bytes.as_ref(), self, vm) + }; - // CRITICAL: Flush any pending TLS data before writing new data - // This ensures TLS 1.3 Finished message reaches server before application data - // Without this, server may not be ready to process our data - if !is_bio { - self.flush_pending_tls_output(vm, None)?; + match result { + Ok(n) => { + self.check_deferred_cert_error(vm)?; + Ok(n) } - - // Write data in chunks to avoid filling the internal TLS buffer - // rustls has a limited internal buffer, so we need to flush periodically - const CHUNK_SIZE: usize = 16384; // 16KB chunks (typical TLS record size) - let mut written = 0; - - while written < data.len() { - let chunk_end = core::cmp::min(written + CHUNK_SIZE, data.len()); - let chunk = &data[written..chunk_end]; - - // Write chunk to TLS layer - { - let mut writer = conn.writer(); - use std::io::Write; - writer - .write_all(chunk) - .map_err(|e| vm.new_os_error(format!("Write failed: {e}")))?; - // Flush to ensure data is converted to TLS records - writer - .flush() - .map_err(|e| vm.new_os_error(format!("Flush failed: {e}")))?; - } - - written = chunk_end; - - // Flush TLS data to socket after each chunk - if conn.wants_write() { - if is_bio { - self.write_pending_tls(conn, vm)?; - } else { - // Socket mode: flush all pending TLS data - // First, try to send any previously pending data - self.flush_pending_tls_output(vm, None)?; - - while conn.wants_write() { - let mut buf = Vec::new(); - conn.write_tls(&mut buf).map_err(|e| { - vm.new_os_error(format!("TLS write failed: {e}")) - })?; - - if !buf.is_empty() { - // Try to send TLS data, saving unsent bytes to pending buffer - self.send_tls_output(buf, vm)?; - } - } - } - } + Err(crate::ssl::compat::SslError::WantRead) => { + Err(create_ssl_want_read_error(vm).upcast()) + } + Err(crate::ssl::compat::SslError::WantWrite) => { + Err(create_ssl_want_write_error(vm).upcast()) } + Err(crate::ssl::compat::SslError::Timeout(msg)) => { + Err(timeout_error_msg(vm, msg).upcast()) + } + Err(e) => Err(e.into_py_err(vm)), } - - // Check for deferred certificate verification errors (TLS 1.3) - // Must be checked AFTER write completes, as the error may be set during I/O - self.check_deferred_cert_error(vm)?; - - Ok(data_len) } #[pymethod] @@ -3989,6 +4077,10 @@ mod _ssl { // Write close_notify to outgoing buffer/BIO self.write_pending_tls(conn, vm)?; + // Ensure close_notify and any pending TLS data are flushed + if !is_bio { + self.flush_pending_tls_output(vm, None)?; + } // Update state *self.shutdown_state.lock() = ShutdownState::SentCloseNotify; @@ -4037,43 +4129,115 @@ mod _ssl { peer_closed = true; } } else if let Some(timeout) = timeout_mode { - // All socket modes (blocking, timeout, non-blocking): - // Return immediately after sending our close_notify. - // - // This matches CPython/OpenSSL behavior where SSL_shutdown() - // returns after sending close_notify, allowing the app to - // close the socket without waiting for peer's close_notify. - // - // Waiting for peer's close_notify can cause deadlock with - // asyncore-based servers where both sides wait for the other's - // close_notify before closing the connection. - - // Ensure all pending TLS data is sent before returning - // This prevents data loss when rustls drains its buffer - // but the socket couldn't accept all data immediately - drop(conn_guard); - - // Respect socket timeout settings for flushing pending TLS data match timeout { Some(0.0) => { - // Non-blocking: best-effort flush, ignore errors - // to avoid deadlock with asyncore-based servers + // Non-blocking: return immediately after sending close_notify. + // Don't wait for peer's close_notify to avoid blocking. + drop(conn_guard); + // Best-effort flush; WouldBlock is expected in non-blocking mode. + // Other errors indicate close_notify may not have been sent, + // but we still complete shutdown to avoid inconsistent state. let _ = self.flush_pending_tls_output(vm, None); + *self.shutdown_state.lock() = ShutdownState::Completed; + *self.connection.lock() = None; + return Ok(self.sock.clone()); } - Some(_t) => { - // Timeout mode: use flush with socket's timeout - // Errors (including timeout) are propagated to caller - self.flush_pending_tls_output(vm, None)?; - } - None => { - // Blocking mode: wait until all pending data is sent - self.blocking_flush_all_pending(vm)?; + _ => { + // Blocking or timeout mode: wait for peer's close_notify. + // This is proper TLS shutdown - we should receive peer's + // close_notify before closing the connection. + drop(conn_guard); + + // Flush our close_notify first + if timeout.is_none() { + self.blocking_flush_all_pending(vm)?; + } else { + self.flush_pending_tls_output(vm, None)?; + } + + // Calculate deadline for timeout mode + let deadline = timeout.map(|t| { + std::time::Instant::now() + std::time::Duration::from_secs_f64(t) + }); + + // Wait for peer's close_notify + loop { + // Re-acquire connection lock for each iteration + let mut conn_guard = self.connection.lock(); + let conn = match conn_guard.as_mut() { + Some(c) => c, + None => break, // Connection already closed + }; + + // Check if peer already sent close_notify + if self.check_peer_closed(conn, vm)? { + break; + } + + drop(conn_guard); + + // Check timeout + let remaining_timeout = if let Some(dl) = deadline { + let now = std::time::Instant::now(); + if now >= dl { + // Timeout reached - raise TimeoutError + return Err(vm.new_exception_msg( + vm.ctx.exceptions.timeout_error.to_owned(), + "The read operation timed out".to_owned(), + )); + } + Some(dl - now) + } else { + None // Blocking mode: no timeout + }; + + // Wait for socket to be readable + let timed_out = self.sock_wait_for_io_with_timeout( + SelectKind::Read, + remaining_timeout, + vm, + )?; + + if timed_out { + // Timeout waiting for peer's close_notify + // Raise TimeoutError + return Err(vm.new_exception_msg( + vm.ctx.exceptions.timeout_error.to_owned(), + "The read operation timed out".to_owned(), + )); + } + + // Try to read data from socket + let mut conn_guard = self.connection.lock(); + let conn = match conn_guard.as_mut() { + Some(c) => c, + None => break, + }; + + // Read and process any incoming TLS data + match self.try_read_close_notify(conn, vm) { + Ok(closed) => { + if closed { + break; + } + // Check again after processing + if self.check_peer_closed(conn, vm)? { + break; + } + } + Err(_) => { + // Socket error - peer likely closed connection + break; + } + } + } + + // Shutdown complete + *self.shutdown_state.lock() = ShutdownState::Completed; + *self.connection.lock() = None; + return Ok(self.sock.clone()); } } - - *self.shutdown_state.lock() = ShutdownState::Completed; - *self.connection.lock() = None; - return Ok(self.sock.clone()); } // Step 3: Check again if peer has sent close_notify (non-blocking/BIO mode only) @@ -4133,45 +4297,118 @@ mod _ssl { conn: &mut TlsConnection, vm: &VirtualMachine, ) -> PyResult<bool> { - // Try to read incoming data + // In socket mode, peek first to avoid consuming post-TLS cleartext + // data. During STARTTLS, after close_notify exchange, the socket + // transitions to cleartext. Without peeking, sock_recv may consume + // cleartext data meant for the application after unwrap(). + if self.incoming_bio.is_none() { + return self.try_read_close_notify_socket(conn, vm); + } + + // BIO mode: read from incoming BIO match self.sock_recv(SSL3_RT_MAX_PLAIN_LENGTH, vm) { Ok(bytes_obj) => { let bytes = ArgBytesLike::try_from_object(vm, bytes_obj)?; let data = bytes.borrow_buf(); if data.is_empty() { - // Empty read could mean EOF or just "no data yet" in BIO mode if let Some(ref bio) = self.incoming_bio { // BIO mode: check if EOF was signaled via write_eof() let bio_obj: PyObjectRef = bio.clone().into(); let eof_attr = bio_obj.get_attr("eof", vm)?; let is_eof = eof_attr.try_to_bool(vm)?; if !is_eof { - // No EOF signaled, just no data available yet return Ok(false); } } - // Socket mode or BIO with EOF: peer closed connection - // This is "ragged EOF" - peer closed without close_notify return Ok(true); } - // Feed data to TLS connection let data_slice: &[u8] = data.as_ref(); let mut cursor = std::io::Cursor::new(data_slice); let _ = conn.read_tls(&mut cursor); + let _ = conn.process_new_packets(); + Ok(false) + } + Err(e) => { + if is_blocking_io_error(&e, vm) { + return Ok(false); + } + Ok(true) + } + } + } - // Process packets + /// Socket-mode close_notify reader that respects TLS record boundaries. + /// Uses MSG_PEEK to inspect data before consuming, preventing accidental + /// consumption of post-TLS cleartext data during STARTTLS transitions. + /// + /// Equivalent to OpenSSL's `SSL_set_read_ahead(ssl, 0)` — rustls has no + /// such knob, so we enforce record-level reads manually via peek. + fn try_read_close_notify_socket( + &self, + conn: &mut TlsConnection, + vm: &VirtualMachine, + ) -> PyResult<bool> { + // Peek at the first 5 bytes (TLS record header size) + let peeked_obj = match self.sock_peek(5, vm) { + Ok(obj) => obj, + Err(e) => { + if is_blocking_io_error(&e, vm) { + return Ok(false); + } + return Ok(true); + } + }; + + let peeked = ArgBytesLike::try_from_object(vm, peeked_obj)?; + let peek_data = peeked.borrow_buf(); + + if peek_data.is_empty() { + return Ok(true); // EOF + } + + // TLS record content types: ChangeCipherSpec(20), Alert(21), + // Handshake(22), ApplicationData(23) + let content_type = peek_data[0]; + if !(20..=23).contains(&content_type) { + // Not a TLS record - post-TLS cleartext data. + // Peer has completed TLS shutdown; don't consume this data. + return Ok(true); + } + + // Determine how many bytes to read for exactly one TLS record + let recv_size = if peek_data.len() >= 5 { + let record_length = u16::from_be_bytes([peek_data[3], peek_data[4]]) as usize; + 5 + record_length + } else { + // Partial header available - read just these bytes for now + peek_data.len() + }; + + drop(peek_data); + drop(peeked); + + // Now consume exactly one TLS record from the socket + match self.sock_recv(recv_size, vm) { + Ok(bytes_obj) => { + let bytes = ArgBytesLike::try_from_object(vm, bytes_obj)?; + let data = bytes.borrow_buf(); + + if data.is_empty() { + return Ok(true); + } + + let data_slice: &[u8] = data.as_ref(); + let mut cursor = std::io::Cursor::new(data_slice); + let _ = conn.read_tls(&mut cursor); let _ = conn.process_new_packets(); Ok(false) } Err(e) => { - // BlockingIOError means no data yet if is_blocking_io_error(&e, vm) { return Ok(false); } - // Connection reset, EOF, or other error means peer closed - // ECONNRESET, EPIPE, broken pipe, etc. Ok(true) } } @@ -4332,6 +4569,20 @@ mod _ssl { } } + // Clean up SSL socket resources on drop + impl Drop for PySSLSocket { + fn drop(&mut self) { + // Only clear connection state. + // Do NOT clear pending_tls_output - it may contain data that hasn't + // been flushed to the socket yet. SSLSocket._real_close() in Python + // doesn't call shutdown(), so when the socket is closed, pending TLS + // data would be lost if we clear it here. + // All fields (Vec, primitives) are automatically freed when the + // struct is dropped, so explicit clearing is unnecessary. + let _ = self.connection.lock().take(); + } + } + // MemoryBIO - provides in-memory buffer for SSL/TLS I/O #[pyattr] #[pyclass(name = "MemoryBIO", module = "ssl")] diff --git a/crates/stdlib/src/ssl/compat.rs b/crates/stdlib/src/ssl/compat.rs index 00e2c8d3c32..5bf2cd8b60f 100644 --- a/crates/stdlib/src/ssl/compat.rs +++ b/crates/stdlib/src/ssl/compat.rs @@ -36,8 +36,8 @@ use super::_ssl::PySSLSocket; // Import error types and helper functions from error module use super::error::{ - PySSLCertVerificationError, PySSLError, create_ssl_eof_error, create_ssl_want_read_error, - create_ssl_want_write_error, create_ssl_zero_return_error, + PySSLCertVerificationError, PySSLError, create_ssl_eof_error, create_ssl_syscall_error, + create_ssl_want_read_error, create_ssl_want_write_error, create_ssl_zero_return_error, }; // SSL Verification Flags @@ -553,8 +553,8 @@ impl SslError { SslError::WantWrite => create_ssl_want_write_error(vm).upcast(), SslError::Timeout(msg) => timeout_error_msg(vm, msg).upcast(), SslError::Syscall(msg) => { - // Create SSLError with library=None for syscall errors during SSL operations - Self::create_ssl_error_with_reason(vm, None, &msg, msg.clone()) + // SSLSyscallError with errno=SSL_ERROR_SYSCALL (5) + create_ssl_syscall_error(vm, msg).upcast() } SslError::Ssl(msg) => vm .new_os_subtype_error( @@ -1039,6 +1039,36 @@ fn send_all_bytes( return Err(SslError::Timeout("The operation timed out".to_string())); } + // Wait for socket to be writable before sending + let timed_out = if let Some(dl) = deadline { + let now = std::time::Instant::now(); + if now >= dl { + socket + .pending_tls_output + .lock() + .extend_from_slice(&buf[sent_total..]); + return Err(SslError::Timeout( + "The write operation timed out".to_string(), + )); + } + socket + .sock_wait_for_io_with_timeout(SelectKind::Write, Some(dl - now), vm) + .map_err(SslError::Py)? + } else { + socket + .sock_wait_for_io_impl(SelectKind::Write, vm) + .map_err(SslError::Py)? + }; + if timed_out { + socket + .pending_tls_output + .lock() + .extend_from_slice(&buf[sent_total..]); + return Err(SslError::Timeout( + "The write operation timed out".to_string(), + )); + } + match socket.sock_send(&buf[sent_total..], vm) { Ok(result) => { let sent: usize = result @@ -1237,8 +1267,7 @@ fn handle_handshake_complete( } } else if conn.wants_write() { // Send all pending data (e.g., TLS 1.3 NewSessionTicket) to socket - // Best-effort: WantWrite means socket buffer full, pending data will be - // sent in subsequent read/write calls. Don't fail handshake for this. + // Must drain ALL rustls buffer - don't break on WantWrite while conn.wants_write() { let tls_data = ssl_write_tls_records(conn)?; if tls_data.is_empty() { @@ -1246,7 +1275,13 @@ fn handle_handshake_complete( } match send_all_bytes(socket, tls_data, vm, None) { Ok(()) => {} - Err(SslError::WantWrite) => break, + Err(SslError::WantWrite) => { + // Socket buffer full, data saved to pending_tls_output + // Flush pending and continue draining rustls buffer + socket + .blocking_flush_all_pending(vm) + .map_err(SslError::Py)?; + } Err(e) => return Err(e), } } @@ -1256,6 +1291,7 @@ fn handle_handshake_complete( // TLS 1.3 Finished must reach server before handshake is considered complete // Without this, server may not process application data if !socket.is_bio_mode() { + // Flush pending_tls_output to ensure all TLS data reaches the server socket .blocking_flush_all_pending(vm) .map_err(SslError::Py)?; @@ -1437,9 +1473,17 @@ pub(super) fn ssl_do_handshake( } } - // If we exit the loop without completing handshake, return error - // Check rustls state to provide better error message + // If we exit the loop without completing handshake, return appropriate error if conn.is_handshaking() { + // For non-blocking sockets, return WantRead/WantWrite to signal caller + // should retry when socket is ready. This matches OpenSSL behavior. + if conn.wants_write() { + return Err(SslError::WantWrite); + } + if conn.wants_read() { + return Err(SslError::WantRead); + } + // Neither wants_read nor wants_write - this is a real error Err(SslError::Syscall(format!( "SSL handshake failed: incomplete after {iteration_count} iterations", ))) @@ -1508,6 +1552,11 @@ pub(super) fn ssl_read( // Try to read plaintext from rustls buffer if let Some(n) = try_read_plaintext(conn, buf)? { + if n == 0 { + // EOF from TLS - close_notify received + // Return ZeroReturn so Python raises SSLZeroReturnError + return Err(SslError::ZeroReturn); + } return Ok(n); } @@ -1575,6 +1624,14 @@ pub(super) fn ssl_read( if let Some(t) = timeout && t.is_zero() { + // Non-blocking socket: check if peer has closed before returning WantRead + // If close_notify was received, we should return ZeroReturn (EOF), not WantRead + // This is critical for asyncore-based applications that rely on recv() returning + // 0 or raising SSL_ERROR_ZERO_RETURN to detect connection close. + let io_state = conn.process_new_packets().map_err(SslError::from_rustls)?; + if io_state.peer_has_closed() { + return Err(SslError::ZeroReturn); + } // Non-blocking socket: return immediately return Err(SslError::WantRead); } @@ -1599,7 +1656,13 @@ pub(super) fn ssl_read( .unwrap_or(0); if bytes_read == 0 { - // No more data available - connection might be closed + // No more data available - check if this is clean shutdown or unexpected EOF + // If close_notify was already received, return ZeroReturn (clean closure) + // Otherwise, return Eof (unexpected EOF) + let io_state = conn.process_new_packets().map_err(SslError::from_rustls)?; + if io_state.peer_has_closed() { + return Err(SslError::ZeroReturn); + } return Err(SslError::Eof); } @@ -1642,6 +1705,187 @@ pub(super) fn ssl_read( } } +/// Equivalent to OpenSSL's SSL_write() +/// +/// Writes application data to TLS connection. +/// Automatically handles TLS record I/O as needed. +/// +/// = SSL_write_ex() +pub(super) fn ssl_write( + conn: &mut TlsConnection, + data: &[u8], + socket: &PySSLSocket, + vm: &VirtualMachine, +) -> SslResult<usize> { + if data.is_empty() { + return Ok(0); + } + + let is_bio = socket.is_bio_mode(); + + // Get socket timeout and calculate deadline (= _PyDeadline_Init) + let deadline = if !is_bio { + match socket.get_socket_timeout(vm).map_err(SslError::Py)? { + Some(timeout) if !timeout.is_zero() => Some(std::time::Instant::now() + timeout), + _ => None, + } + } else { + None + }; + + // Flush any pending TLS output before writing new data + if !is_bio { + socket + .flush_pending_tls_output(vm, deadline) + .map_err(SslError::Py)?; + } + + // Check if we already have data buffered from a previous retry + // (prevents duplicate writes when retrying after WantWrite/WantRead) + let already_buffered = *socket.write_buffered_len.lock(); + + // Only write plaintext if not already buffered + // Track how much we wrote for partial write handling + let mut bytes_written_to_rustls = 0usize; + + if already_buffered == 0 { + // Write plaintext to rustls (= SSL_write_ex internal buffer write) + bytes_written_to_rustls = { + let mut writer = conn.writer(); + use std::io::Write; + // Use write() instead of write_all() to support partial writes. + // In BIO mode (asyncio), when the internal buffer is full, + // we want to write as much as possible and return that count, + // rather than failing completely. + match writer.write(data) { + Ok(0) if !data.is_empty() => { + // Buffer is full and nothing could be written. + // In BIO mode, return WantWrite so the caller can + // drain the outgoing BIO and retry. + if is_bio { + return Err(SslError::WantWrite); + } + return Err(SslError::Syscall("Write failed: buffer full".to_string())); + } + Ok(n) => n, + Err(e) => { + if is_bio { + // In BIO mode, treat write errors as WantWrite + return Err(SslError::WantWrite); + } + return Err(SslError::Syscall(format!("Write failed: {e}"))); + } + } + }; + // Mark data as buffered (only the portion we actually wrote) + *socket.write_buffered_len.lock() = bytes_written_to_rustls; + } else if already_buffered != data.len() { + // Caller is retrying with different data - this is a protocol error + // Clear the buffer state and return an SSL error (bad write retry) + *socket.write_buffered_len.lock() = 0; + return Err(SslError::Ssl("bad write retry".to_string())); + } + // else: already_buffered == data.len(), this is a valid retry + + // Loop to send TLS records, handling WANT_READ/WANT_WRITE + // Matches CPython's do-while loop on SSL_ERROR_WANT_READ/WANT_WRITE + loop { + // Check deadline + if let Some(dl) = deadline + && std::time::Instant::now() >= dl + { + return Err(SslError::Timeout( + "The write operation timed out".to_string(), + )); + } + + // Check if rustls has TLS data to send + if !conn.wants_write() { + // All TLS data sent successfully + break; + } + + // Get TLS records from rustls + let tls_data = ssl_write_tls_records(conn)?; + if tls_data.is_empty() { + break; + } + + // Send TLS data to socket + match send_all_bytes(socket, tls_data, vm, deadline) { + Ok(()) => { + // Successfully sent, continue loop to check for more data + } + Err(SslError::WantWrite) => { + // Non-blocking socket would block - return WANT_WRITE + // If we had a partial write to rustls, return partial success + // instead of error to match OpenSSL partial-write semantics + if bytes_written_to_rustls > 0 && bytes_written_to_rustls < data.len() { + *socket.write_buffered_len.lock() = 0; + return Ok(bytes_written_to_rustls); + } + // Keep write_buffered_len set so we don't re-buffer on retry + return Err(SslError::WantWrite); + } + Err(SslError::WantRead) => { + // Need to read before write can complete (e.g., renegotiation) + if is_bio { + // If we had a partial write to rustls, return partial success + if bytes_written_to_rustls > 0 && bytes_written_to_rustls < data.len() { + *socket.write_buffered_len.lock() = 0; + return Ok(bytes_written_to_rustls); + } + // Keep write_buffered_len set so we don't re-buffer on retry + return Err(SslError::WantRead); + } + // For socket mode, try to read TLS data + let recv_result = socket.sock_recv(4096, vm).map_err(SslError::Py)?; + ssl_read_tls_records(conn, recv_result, false, vm)?; + conn.process_new_packets().map_err(SslError::from_rustls)?; + // Continue loop + } + Err(e @ SslError::Timeout(_)) => { + // If we had a partial write to rustls, return partial success + if bytes_written_to_rustls > 0 && bytes_written_to_rustls < data.len() { + *socket.write_buffered_len.lock() = 0; + return Ok(bytes_written_to_rustls); + } + // Preserve buffered state so retry doesn't duplicate data + // (send_all_bytes saved unsent TLS bytes to pending_tls_output) + return Err(e); + } + Err(e) => { + // Clear buffer state on error + *socket.write_buffered_len.lock() = 0; + return Err(e); + } + } + } + + // Final flush to ensure all data is sent + if !is_bio { + socket + .flush_pending_tls_output(vm, deadline) + .map_err(SslError::Py)?; + } + + // Determine how many bytes we actually wrote + let actual_written = if bytes_written_to_rustls > 0 { + // Fresh write: return what we wrote to rustls + bytes_written_to_rustls + } else if already_buffered > 0 { + // Retry of previous write: return the full buffered amount + already_buffered + } else { + data.len() + }; + + // Write completed successfully - clear buffer state + *socket.write_buffered_len.lock() = 0; + + Ok(actual_written) +} + // Helper functions (private-ish, used by public SSL functions) /// Write TLS records from rustls to socket @@ -1678,26 +1922,24 @@ fn ssl_read_tls_records( // 1. Clean shutdown: received TLS close_notify → return ZeroReturn (0 bytes) // 2. Unexpected EOF: no close_notify → return Eof (SSLEOFError) // - // SSL_ERROR_ZERO_RETURN vs SSL_ERROR_SYSCALL(errno=0) logic + // SSL_ERROR_ZERO_RETURN vs SSL_ERROR_EOF logic // CPython checks SSL_get_shutdown() & SSL_RECEIVED_SHUTDOWN // // Process any buffered TLS records (may contain close_notify) - let _ = conn.process_new_packets(); - - // IMPORTANT: CPython's default behavior (suppress_ragged_eofs=True) - // treats empty recv() as clean shutdown, returning 0 bytes instead of raising SSLEOFError. - // - // This is necessary for HTTP/1.0 servers that: - // 1. Send response without Content-Length header - // 2. Signal end-of-response by closing connection (TCP FIN) - // 3. Don't send TLS close_notify before TCP close - // - // While this could theoretically allow truncation attacks, - // it's the standard behavior for compatibility with real-world servers. - // Python only raises SSLEOFError when suppress_ragged_eofs=False is explicitly set. - // - // TODO: Implement suppress_ragged_eofs parameter if needed for strict security mode. - return Err(SslError::ZeroReturn); + match conn.process_new_packets() { + Ok(io_state) => { + if io_state.peer_has_closed() { + // Received close_notify - normal SSL closure (SSL_ERROR_ZERO_RETURN) + return Err(SslError::ZeroReturn); + } else { + // No close_notify - ragged EOF (SSL_ERROR_EOF → SSLEOFError) + // CPython raises SSLEOFError here, which SSLSocket.read() handles + // based on suppress_ragged_eofs setting + return Err(SslError::Eof); + } + } + Err(e) => return Err(SslError::from_rustls(e)), + } } } @@ -1810,6 +2052,9 @@ fn ssl_ensure_data_available( let data = match socket.sock_recv(2048, vm) { Ok(data) => data, Err(e) => { + if is_blocking_io_error(&e, vm) { + return Err(SslError::WantRead); + } // Before returning socket error, check if rustls already has a queued TLS alert // This mirrors CPython/OpenSSL behavior: SSL errors take precedence over socket errors // On Windows, TCP RST may arrive before we read the alert, but rustls may have diff --git a/crates/stdlib/src/ssl/error.rs b/crates/stdlib/src/ssl/error.rs index d77910f6aa1..cbc59e0e8f6 100644 --- a/crates/stdlib/src/ssl/error.rs +++ b/crates/stdlib/src/ssl/error.rs @@ -7,7 +7,7 @@ pub(crate) mod ssl_error { use crate::vm::{ Py, PyPayload, PyRef, PyResult, VirtualMachine, builtins::{PyBaseException, PyOSError, PyStrRef}, - types::{Constructor, Initializer}, + types::Constructor, }; // Error type constants - exposed as pyattr and available for internal use @@ -132,4 +132,15 @@ pub(crate) mod ssl_error { "TLS/SSL connection has been closed (EOF)", ) } + + pub fn create_ssl_syscall_error( + vm: &VirtualMachine, + msg: impl Into<String>, + ) -> PyRef<PyOSError> { + vm.new_os_subtype_error( + PySSLSyscallError::class(&vm.ctx).to_owned(), + Some(SSL_ERROR_SYSCALL), + msg.into(), + ) + } } diff --git a/crates/stdlib/src/statistics.rs b/crates/stdlib/src/statistics.rs index 8be2447ffd6..641d8864ed3 100644 --- a/crates/stdlib/src/statistics.rs +++ b/crates/stdlib/src/statistics.rs @@ -1,4 +1,4 @@ -pub(crate) use _statistics::make_module; +pub(crate) use _statistics::module_def; #[pymodule] mod _statistics { diff --git a/crates/stdlib/src/suggestions.rs b/crates/stdlib/src/suggestions.rs index e49e9dd4a4b..e0667dfb553 100644 --- a/crates/stdlib/src/suggestions.rs +++ b/crates/stdlib/src/suggestions.rs @@ -1,4 +1,4 @@ -pub(crate) use _suggestions::make_module; +pub(crate) use _suggestions::module_def; #[pymodule] mod _suggestions { diff --git a/crates/stdlib/src/syslog.rs b/crates/stdlib/src/syslog.rs index b52d1415692..8f446a8e161 100644 --- a/crates/stdlib/src/syslog.rs +++ b/crates/stdlib/src/syslog.rs @@ -1,6 +1,6 @@ // spell-checker:ignore logoption openlog setlogmask upto NDELAY ODELAY -pub(crate) use syslog::make_module; +pub(crate) use syslog::module_def; #[pymodule(name = "syslog")] mod syslog { diff --git a/crates/stdlib/src/termios.rs b/crates/stdlib/src/termios.rs index a9ae1375c6f..de402724434 100644 --- a/crates/stdlib/src/termios.rs +++ b/crates/stdlib/src/termios.rs @@ -1,6 +1,6 @@ // spell-checker:disable -pub(crate) use self::termios::make_module; +pub(crate) use self::termios::module_def; #[pymodule] mod termios { @@ -261,13 +261,12 @@ mod termios { } fn termios_error(err: std::io::Error, vm: &VirtualMachine) -> PyBaseExceptionRef { - vm.new_exception( + vm.new_os_subtype_error( error_type(vm), - vec![ - err.posix_errno().to_pyobject(vm), - vm.ctx.new_str(err.to_string()).into(), - ], + Some(err.posix_errno()), + vm.ctx.new_str(err.to_string()), ) + .upcast() } #[pyattr(name = "error", once)] diff --git a/crates/stdlib/src/tkinter.rs b/crates/stdlib/src/tkinter.rs index 6b7c4387d6c..b258002c129 100644 --- a/crates/stdlib/src/tkinter.rs +++ b/crates/stdlib/src/tkinter.rs @@ -1,6 +1,6 @@ // spell-checker:ignore createcommand -pub(crate) use self::_tkinter::make_module; +pub(crate) use self::_tkinter::module_def; #[pymodule] mod _tkinter { diff --git a/crates/stdlib/src/unicodedata.rs b/crates/stdlib/src/unicodedata.rs index 68d9a17e575..a859e39df10 100644 --- a/crates/stdlib/src/unicodedata.rs +++ b/crates/stdlib/src/unicodedata.rs @@ -4,35 +4,12 @@ // spell-checker:ignore nfkc unistr unidata +pub(crate) use unicodedata::module_def; + use crate::vm::{ - PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, builtins::PyModule, - builtins::PyStr, convert::TryFromBorrowedObject, + PyObject, PyResult, VirtualMachine, builtins::PyStr, convert::TryFromBorrowedObject, }; -pub fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - let module = unicodedata::make_module(vm); - - let ucd: PyObjectRef = unicodedata::Ucd::new(unic_ucd_age::UNICODE_VERSION) - .into_ref(&vm.ctx) - .into(); - - for attr in [ - "category", - "lookup", - "name", - "bidirectional", - "east_asian_width", - "normalize", - "mirrored", - ] { - crate::vm::extend_module!(vm, &module, { - attr => ucd.get_attr(attr, vm).unwrap(), - }); - } - - module -} - enum NormalizeForm { Nfc, Nfkc, @@ -60,7 +37,8 @@ impl<'a> TryFromBorrowedObject<'a> for NormalizeForm { #[pymodule] mod unicodedata { use crate::vm::{ - PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, builtins::PyStrRef, + Py, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, + builtins::{PyModule, PyStrRef}, function::OptionalArg, }; use itertools::Itertools; @@ -73,6 +51,27 @@ mod unicodedata { use unic_ucd_category::GeneralCategory; use unicode_bidi_mirroring::is_mirroring; + pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { + __module_exec(vm, module); + + // Add UCD methods as module-level functions + let ucd: PyObjectRef = Ucd::new(UNICODE_VERSION).into_ref(&vm.ctx).into(); + + for attr in [ + "category", + "lookup", + "name", + "bidirectional", + "east_asian_width", + "normalize", + "mirrored", + ] { + module.set_attr(attr, ucd.get_attr(attr, vm)?, vm)?; + } + + Ok(()) + } + #[pyattr] #[pyclass(name = "UCD")] #[derive(Debug, PyPayload)] diff --git a/crates/stdlib/src/uuid.rs b/crates/stdlib/src/uuid.rs index 3f75db402c8..3bc01610d43 100644 --- a/crates/stdlib/src/uuid.rs +++ b/crates/stdlib/src/uuid.rs @@ -1,4 +1,4 @@ -pub(crate) use _uuid::make_module; +pub(crate) use _uuid::module_def; #[pymodule] mod _uuid { diff --git a/crates/stdlib/src/zlib.rs b/crates/stdlib/src/zlib.rs index 632543c5c64..40269f12bbf 100644 --- a/crates/stdlib/src/zlib.rs +++ b/crates/stdlib/src/zlib.rs @@ -1,6 +1,6 @@ // spell-checker:ignore compressobj decompressobj zdict chunksize zlibmodule miniz chunker -pub(crate) use zlib::make_module; +pub(crate) use zlib::module_def; #[pymodule] mod zlib { diff --git a/crates/venvlauncher/src/main.rs b/crates/venvlauncher/src/main.rs index fe147ce7ff3..7087e791e37 100644 --- a/crates/venvlauncher/src/main.rs +++ b/crates/venvlauncher/src/main.rs @@ -114,12 +114,12 @@ mod tests { let cfg_path = temp_dir.join("test_pyvenv.cfg"); let mut file = fs::File::create(&cfg_path).unwrap(); - writeln!(file, "home = C:\\Python313").unwrap(); + writeln!(file, "home = C:\\Python314").unwrap(); writeln!(file, "include-system-site-packages = false").unwrap(); - writeln!(file, "version = 3.13.0").unwrap(); + writeln!(file, "version = 3.14.0").unwrap(); let home = read_home(&cfg_path).unwrap(); - assert_eq!(home, "C:\\Python313"); + assert_eq!(home, "C:\\Python314"); fs::remove_file(&cfg_path).unwrap(); } diff --git a/crates/vm/Cargo.toml b/crates/vm/Cargo.toml index b74aba41145..6e05c7cde48 100644 --- a/crates/vm/Cargo.toml +++ b/crates/vm/Cargo.toml @@ -10,7 +10,7 @@ repository.workspace = true license.workspace = true [features] -default = ["compiler", "wasmbind", "stdio"] +default = ["compiler", "wasmbind", "stdio", "gc"] stdio = [] importlib = [] encodings = ["importlib"] @@ -19,6 +19,7 @@ flame-it = ["flame", "flamer"] freeze-stdlib = ["encodings"] jit = ["rustpython-jit"] threading = ["rustpython-common/threading"] +gc = [] compiler = ["parser", "codegen", "rustpython-compiler"] ast = ["ruff_python_ast", "ruff_text_size"] codegen = ["rustpython-codegen", "ast"] @@ -77,6 +78,7 @@ memchr = { workspace = true } caseless = "0.2.2" flamer = { version = "0.5", optional = true } half = "2" +psm = "0.1" optional = { workspace = true } result-like = "0.5.0" timsort = "0.1.2" diff --git a/crates/vm/Lib/python_builtins/__reducelib.py b/crates/vm/Lib/python_builtins/__reducelib.py deleted file mode 100644 index 0067cd0a818..00000000000 --- a/crates/vm/Lib/python_builtins/__reducelib.py +++ /dev/null @@ -1,86 +0,0 @@ -# Modified from code from the PyPy project: -# https://bitbucket.org/pypy/pypy/src/default/pypy/objspace/std/objectobject.py - -# The MIT License - -# Permission is hereby granted, free of charge, to any person -# obtaining a copy of this software and associated documentation -# files (the "Software"), to deal in the Software without -# restriction, including without limitation the rights to use, -# copy, modify, merge, publish, distribute, sublicense, and/or -# sell copies of the Software, and to permit persons to whom the -# Software is furnished to do so, subject to the following conditions: - -# The above copyright notice and this permission notice shall be included -# in all copies or substantial portions of the Software. - -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS -# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL -# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -# DEALINGS IN THE SOFTWARE. - -import copyreg - - -def _abstract_method_error(typ): - methods = ", ".join(sorted(typ.__abstractmethods__)) - err = "Can't instantiate abstract class %s with abstract methods %s" - raise TypeError(err % (typ.__name__, methods)) - - -def reduce_2(obj): - cls = obj.__class__ - - try: - getnewargs = obj.__getnewargs__ - except AttributeError: - args = () - else: - args = getnewargs() - if not isinstance(args, tuple): - raise TypeError("__getnewargs__ should return a tuple") - - try: - getstate = obj.__getstate__ - except AttributeError: - state = getattr(obj, "__dict__", None) - names = slotnames(cls) # not checking for list - if names is not None: - slots = {} - for name in names: - try: - value = getattr(obj, name) - except AttributeError: - pass - else: - slots[name] = value - if slots: - state = state, slots - else: - state = getstate() - - listitems = iter(obj) if isinstance(obj, list) else None - dictitems = iter(obj.items()) if isinstance(obj, dict) else None - - newobj = copyreg.__newobj__ - - args2 = (cls,) + args - return newobj, args2, state, listitems, dictitems - - -def slotnames(cls): - if not isinstance(cls, type): - return None - - try: - return cls.__dict__["__slotnames__"] - except KeyError: - pass - - slotnames = copyreg._slotnames(cls) - if not isinstance(slotnames, list) and slotnames is not None: - raise TypeError("copyreg._slotnames didn't return a list or None") - return slotnames diff --git a/crates/vm/src/builtins/asyncgenerator.rs b/crates/vm/src/builtins/asyncgenerator.rs index e16cbc6d18d..7523714a3d0 100644 --- a/crates/vm/src/builtins/asyncgenerator.rs +++ b/crates/vm/src/builtins/asyncgenerator.rs @@ -1,4 +1,4 @@ -use super::{PyCode, PyGenericAlias, PyStrRef, PyType, PyTypeRef}; +use super::{PyCode, PyGenerator, PyGenericAlias, PyStrRef, PyType, PyTypeRef}; use crate::{ AsObject, Context, Py, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, builtins::PyBaseExceptionRef, @@ -7,13 +7,14 @@ use crate::{ coroutine::{Coro, warn_deprecated_throw_signature}, frame::FrameRef, function::OptionalArg, + object::{Traverse, TraverseFn}, protocol::PyIterReturn, types::{Destructor, IterNext, Iterable, Representable, SelfIter}, }; use crossbeam_utils::atomic::AtomicCell; -#[pyclass(name = "async_generator", module = false)] +#[pyclass(name = "async_generator", module = false, traverse = "manual")] #[derive(Debug)] pub struct PyAsyncGen { inner: Coro, @@ -23,6 +24,13 @@ pub struct PyAsyncGen { // ag_origin_or_finalizer - stores the finalizer callback ag_finalizer: PyMutex<Option<PyObjectRef>>, } + +unsafe impl Traverse for PyAsyncGen { + fn traverse(&self, tracer_fn: &mut TraverseFn<'_>) { + self.inner.traverse(tracer_fn); + self.ag_finalizer.traverse(tracer_fn); + } +} type PyAsyncGenRef = PyRef<PyAsyncGen>; impl PyPayload for PyAsyncGen { @@ -57,14 +65,14 @@ impl PyAsyncGen { zelf.ag_hooks_inited.store(true); - // Get and store finalizer from thread-local storage - let finalizer = crate::vm::thread::ASYNC_GEN_FINALIZER.with_borrow(|f| f.as_ref().cloned()); + // Get and store finalizer from VM + let finalizer = vm.async_gen_finalizer.borrow().clone(); if let Some(finalizer) = finalizer { *zelf.ag_finalizer.lock() = Some(finalizer); } // Call firstiter hook - let firstiter = crate::vm::thread::ASYNC_GEN_FIRSTITER.with_borrow(|f| f.as_ref().cloned()); + let firstiter = vm.async_gen_firstiter.borrow().clone(); if let Some(firstiter) = firstiter { let obj: PyObjectRef = zelf.to_owned().into(); firstiter.call((obj,), vm)?; @@ -79,9 +87,14 @@ impl PyAsyncGen { if let Some(finalizer) = finalizer && !zelf.inner.closed.load() { - // Ignore any errors (PyErr_WriteUnraisable) + // Create a strong reference for the finalizer call. + // This keeps the object alive during the finalizer execution. let obj: PyObjectRef = zelf.to_owned().into(); - let _ = finalizer.call((obj,), vm); + + // Call the finalizer. Any exceptions are handled as unraisable. + if let Err(e) = finalizer.call((obj,), vm) { + vm.run_unraisable(e, Some("async generator finalizer".to_owned()), finalizer); + } } } @@ -110,8 +123,12 @@ impl PyAsyncGen { self.inner.frame().yield_from_target() } #[pygetset] - fn ag_frame(&self, _vm: &VirtualMachine) -> FrameRef { - self.inner.frame() + fn ag_frame(&self, _vm: &VirtualMachine) -> Option<FrameRef> { + if self.inner.closed() { + None + } else { + Some(self.inner.frame()) + } } #[pygetset] fn ag_running(&self, _vm: &VirtualMachine) -> bool { @@ -163,6 +180,7 @@ impl PyRef<PyAsyncGen> { exc_tb: OptionalArg, vm: &VirtualMachine, ) -> PyResult<PyAsyncGenAThrow> { + warn_deprecated_throw_signature(&exc_val, &exc_tb, vm)?; PyAsyncGen::init_hooks(&self, vm)?; Ok(PyAsyncGenAThrow { ag: self, @@ -199,9 +217,20 @@ impl Representable for PyAsyncGen { } } -#[pyclass(module = false, name = "async_generator_wrapped_value")] +#[pyclass( + module = false, + name = "async_generator_wrapped_value", + traverse = "manual" +)] #[derive(Debug)] pub(crate) struct PyAsyncGenWrappedValue(pub PyObjectRef); + +unsafe impl Traverse for PyAsyncGenWrappedValue { + fn traverse(&self, tracer_fn: &mut TraverseFn<'_>) { + self.0.traverse(tracer_fn); + } +} + impl PyPayload for PyAsyncGenWrappedValue { #[inline] fn class(ctx: &Context) -> &'static Py<PyType> { @@ -244,7 +273,7 @@ enum AwaitableState { Closed, } -#[pyclass(module = false, name = "async_generator_asend")] +#[pyclass(module = false, name = "async_generator_asend", traverse = "manual")] #[derive(Debug)] pub(crate) struct PyAsyncGenASend { ag: PyAsyncGenRef, @@ -252,6 +281,13 @@ pub(crate) struct PyAsyncGenASend { value: PyObjectRef, } +unsafe impl Traverse for PyAsyncGenASend { + fn traverse(&self, tracer_fn: &mut TraverseFn<'_>) { + self.ag.traverse(tracer_fn); + self.value.traverse(tracer_fn); + } +} + impl PyPayload for PyAsyncGenASend { #[inline] fn class(ctx: &Context) -> &'static Py<PyType> { @@ -293,7 +329,7 @@ impl PyAsyncGenASend { let res = self.ag.inner.send(self.ag.as_object(), val, vm); let res = PyAsyncGenWrappedValue::unbox(&self.ag, res, vm); if res.is_err() { - self.close(); + self.set_closed(); } res } @@ -306,8 +342,23 @@ impl PyAsyncGenASend { exc_tb: OptionalArg, vm: &VirtualMachine, ) -> PyResult { - if let AwaitableState::Closed = self.state.load() { - return Err(vm.new_runtime_error("cannot reuse already awaited __anext__()/asend()")); + match self.state.load() { + AwaitableState::Closed => { + return Err( + vm.new_runtime_error("cannot reuse already awaited __anext__()/asend()") + ); + } + AwaitableState::Init => { + if self.ag.running_async.load() { + self.state.store(AwaitableState::Closed); + return Err( + vm.new_runtime_error("anext(): asynchronous generator is already running") + ); + } + self.ag.running_async.store(true); + self.state.store(AwaitableState::Iter); + } + AwaitableState::Iter => {} } warn_deprecated_throw_signature(&exc_val, &exc_tb, vm)?; @@ -320,13 +371,36 @@ impl PyAsyncGenASend { ); let res = PyAsyncGenWrappedValue::unbox(&self.ag, res, vm); if res.is_err() { - self.close(); + self.set_closed(); } res } #[pymethod] - fn close(&self) { + fn close(&self, vm: &VirtualMachine) -> PyResult<()> { + if matches!(self.state.load(), AwaitableState::Closed) { + return Ok(()); + } + let result = self.throw( + vm.ctx.exceptions.generator_exit.to_owned().into(), + OptionalArg::Missing, + OptionalArg::Missing, + vm, + ); + match result { + Ok(_) => Err(vm.new_runtime_error("coroutine ignored GeneratorExit")), + Err(e) + if e.fast_isinstance(vm.ctx.exceptions.stop_iteration) + || e.fast_isinstance(vm.ctx.exceptions.stop_async_iteration) + || e.fast_isinstance(vm.ctx.exceptions.generator_exit) => + { + Ok(()) + } + Err(e) => Err(e), + } + } + + fn set_closed(&self) { self.state.store(AwaitableState::Closed); } } @@ -338,7 +412,7 @@ impl IterNext for PyAsyncGenASend { } } -#[pyclass(module = false, name = "async_generator_athrow")] +#[pyclass(module = false, name = "async_generator_athrow", traverse = "manual")] #[derive(Debug)] pub(crate) struct PyAsyncGenAThrow { ag: PyAsyncGenRef, @@ -347,6 +421,13 @@ pub(crate) struct PyAsyncGenAThrow { value: (PyObjectRef, PyObjectRef, PyObjectRef), } +unsafe impl Traverse for PyAsyncGenAThrow { + fn traverse(&self, tracer_fn: &mut TraverseFn<'_>) { + self.ag.traverse(tracer_fn); + self.value.traverse(tracer_fn); + } +} + impl PyPayload for PyAsyncGenAThrow { #[inline] fn class(ctx: &Context) -> &'static Py<PyType> { @@ -430,6 +511,30 @@ impl PyAsyncGenAThrow { exc_tb: OptionalArg, vm: &VirtualMachine, ) -> PyResult { + match self.state.load() { + AwaitableState::Closed => { + return Err(vm.new_runtime_error("cannot reuse already awaited aclose()/athrow()")); + } + AwaitableState::Init => { + if self.ag.running_async.load() { + self.state.store(AwaitableState::Closed); + let msg = if self.aclose { + "aclose(): asynchronous generator is already running" + } else { + "athrow(): asynchronous generator is already running" + }; + return Err(vm.new_runtime_error(msg.to_owned())); + } + if self.ag.inner.closed() { + self.state.store(AwaitableState::Closed); + return Err(vm.new_stop_iteration(None)); + } + self.ag.running_async.store(true); + self.state.store(AwaitableState::Iter); + } + AwaitableState::Iter => {} + } + warn_deprecated_throw_signature(&exc_val, &exc_tb, vm)?; let ret = self.ag.inner.throw( self.ag.as_object(), @@ -451,8 +556,27 @@ impl PyAsyncGenAThrow { } #[pymethod] - fn close(&self) { - self.state.store(AwaitableState::Closed); + fn close(&self, vm: &VirtualMachine) -> PyResult<()> { + if matches!(self.state.load(), AwaitableState::Closed) { + return Ok(()); + } + let result = self.throw( + vm.ctx.exceptions.generator_exit.to_owned().into(), + OptionalArg::Missing, + OptionalArg::Missing, + vm, + ); + match result { + Ok(_) => Err(vm.new_runtime_error("coroutine ignored GeneratorExit")), + Err(e) + if e.fast_isinstance(vm.ctx.exceptions.stop_iteration) + || e.fast_isinstance(vm.ctx.exceptions.stop_async_iteration) + || e.fast_isinstance(vm.ctx.exceptions.generator_exit) => + { + Ok(()) + } + Err(e) => Err(e), + } } fn ignored_close(&self, res: &PyResult<PyIterReturn>) -> bool { @@ -463,11 +587,13 @@ impl PyAsyncGenAThrow { } fn yield_close(&self, vm: &VirtualMachine) -> PyBaseExceptionRef { self.ag.running_async.store(false); + self.ag.inner.closed.store(true); self.state.store(AwaitableState::Closed); vm.new_runtime_error("async generator ignored GeneratorExit") } fn check_error(&self, exc: PyBaseExceptionRef, vm: &VirtualMachine) -> PyBaseExceptionRef { self.ag.running_async.store(false); + self.ag.inner.closed.store(true); self.state.store(AwaitableState::Closed); if self.aclose && (exc.fast_isinstance(vm.ctx.exceptions.stop_async_iteration) @@ -489,7 +615,7 @@ impl IterNext for PyAsyncGenAThrow { /// Awaitable wrapper for anext() builtin with default value. /// When StopAsyncIteration is raised, it converts it to StopIteration(default). -#[pyclass(module = false, name = "anext_awaitable")] +#[pyclass(module = false, name = "anext_awaitable", traverse = "manual")] #[derive(Debug)] pub struct PyAnextAwaitable { wrapped: PyObjectRef, @@ -497,6 +623,13 @@ pub struct PyAnextAwaitable { state: AtomicCell<AwaitableState>, } +unsafe impl Traverse for PyAnextAwaitable { + fn traverse(&self, tracer_fn: &mut TraverseFn<'_>) { + self.wrapped.traverse(tracer_fn); + self.default_value.traverse(tracer_fn); + } +} + impl PyPayload for PyAnextAwaitable { #[inline] fn class(ctx: &Context) -> &'static Py<PyType> { @@ -545,13 +678,35 @@ impl PyAnextAwaitable { let awaitable = if wrapped.class().is(vm.ctx.types.coroutine_type) { // Coroutine - get __await__ later wrapped.clone() + } else if let Some(generator) = wrapped.downcast_ref::<PyGenerator>() { + // Generator with CO_ITERABLE_COROUTINE flag can be awaited + // (e.g., generators decorated with @types.coroutine) + if generator + .as_coro() + .frame() + .code + .flags + .contains(crate::bytecode::CodeFlags::ITERABLE_COROUTINE) + { + // Return the generator itself as the iterator + return Ok(wrapped.clone()); + } + // Fall through: try to get __await__ method for generator subclasses + if let Some(await_method) = vm.get_method(wrapped.clone(), identifier!(vm, __await__)) { + await_method?.call((), vm)? + } else { + return Err(vm.new_type_error(format!( + "'{}' object can't be awaited", + wrapped.class().name() + ))); + } } else { // Try to get __await__ method if let Some(await_method) = vm.get_method(wrapped.clone(), identifier!(vm, __await__)) { await_method?.call((), vm)? } else { return Err(vm.new_type_error(format!( - "object {} can't be used in 'await' expression", + "'{}' object can't be awaited", wrapped.class().name() ))); } @@ -647,16 +802,24 @@ impl IterNext for PyAnextAwaitable { /// _PyGen_Finalize for async generators impl Destructor for PyAsyncGen { fn del(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<()> { - // Generator isn't paused, so no need to close + // Generator is already closed, nothing to do if zelf.inner.closed.load() { return Ok(()); } + // Call the async generator finalizer hook if set. Self::call_finalizer(zelf, vm); + Ok(()) } } +impl Drop for PyAsyncGen { + fn drop(&mut self) { + self.inner.frame().clear_generator(); + } +} + pub fn init(ctx: &Context) { PyAsyncGen::extend_class(ctx, ctx.types.async_generator); PyAsyncGenASend::extend_class(ctx, ctx.types.async_generator_asend); diff --git a/crates/vm/src/builtins/bytearray.rs b/crates/vm/src/builtins/bytearray.rs index 0f9dce7230b..5a3aee59072 100644 --- a/crates/vm/src/builtins/bytearray.rs +++ b/crates/vm/src/builtins/bytearray.rs @@ -322,8 +322,8 @@ impl PyByteArray { } #[pyclassmethod] - fn fromhex(cls: PyTypeRef, string: PyStrRef, vm: &VirtualMachine) -> PyResult { - let bytes = PyBytesInner::fromhex(string.as_bytes(), vm)?; + fn fromhex(cls: PyTypeRef, string: PyObjectRef, vm: &VirtualMachine) -> PyResult { + let bytes = PyBytesInner::fromhex_object(string, vm)?; let bytes = vm.ctx.new_bytes(bytes); let args = vec![bytes.into()].into(); PyType::call(&cls, args, vm) diff --git a/crates/vm/src/builtins/bytes.rs b/crates/vm/src/builtins/bytes.rs index 01e67358aae..64e00cdc919 100644 --- a/crates/vm/src/builtins/bytes.rs +++ b/crates/vm/src/builtins/bytes.rs @@ -316,8 +316,8 @@ impl PyBytes { } #[pyclassmethod] - fn fromhex(cls: PyTypeRef, string: PyStrRef, vm: &VirtualMachine) -> PyResult { - let bytes = PyBytesInner::fromhex(string.as_bytes(), vm)?; + fn fromhex(cls: PyTypeRef, string: PyObjectRef, vm: &VirtualMachine) -> PyResult { + let bytes = PyBytesInner::fromhex_object(string, vm)?; let bytes = vm.ctx.new_bytes(bytes).into(); PyType::call(&cls, vec![bytes].into(), vm) } diff --git a/crates/vm/src/builtins/classmethod.rs b/crates/vm/src/builtins/classmethod.rs index 5b7f9218658..d2f1377be04 100644 --- a/crates/vm/src/builtins/classmethod.rs +++ b/crates/vm/src/builtins/classmethod.rs @@ -3,7 +3,7 @@ use crate::{ AsObject, Context, Py, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, class::PyClassImpl, common::lock::PyMutex, - function::FuncArgs, + function::{FuncArgs, PySetterValue}, types::{Constructor, GetDescriptor, Initializer, Representable}, }; @@ -158,6 +158,27 @@ impl PyClassMethod { self.callable.lock().get_attr("__annotations__", vm) } + #[pygetset(setter)] + fn set___annotations__(&self, value: PySetterValue, vm: &VirtualMachine) -> PyResult<()> { + match value { + PySetterValue::Assign(v) => self.callable.lock().set_attr("__annotations__", v, vm), + PySetterValue::Delete => Ok(()), // Silently ignore delete like CPython + } + } + + #[pygetset] + fn __annotate__(&self, vm: &VirtualMachine) -> PyResult { + self.callable.lock().get_attr("__annotate__", vm) + } + + #[pygetset(setter)] + fn set___annotate__(&self, value: PySetterValue, vm: &VirtualMachine) -> PyResult<()> { + match value { + PySetterValue::Assign(v) => self.callable.lock().set_attr("__annotate__", v, vm), + PySetterValue::Delete => Ok(()), // Silently ignore delete like CPython + } + } + #[pygetset] fn __isabstractmethod__(&self, vm: &VirtualMachine) -> PyObjectRef { match vm.get_attribute_opt(self.callable.lock().clone(), "__isabstractmethod__") { diff --git a/crates/vm/src/builtins/code.rs b/crates/vm/src/builtins/code.rs index 85816aabb7c..932100db94f 100644 --- a/crates/vm/src/builtins/code.rs +++ b/crates/vm/src/builtins/code.rs @@ -152,7 +152,7 @@ pub struct ReplaceArgs { #[pyarg(named, optional)] co_names: OptionalArg<Vec<PyObjectRef>>, #[pyarg(named, optional)] - co_flags: OptionalArg<u16>, + co_flags: OptionalArg<u32>, #[pyarg(named, optional)] co_varnames: OptionalArg<Vec<PyObjectRef>>, #[pyarg(named, optional)] @@ -411,7 +411,7 @@ pub struct PyCodeNewArgs { kwonlyargcount: u32, nlocals: u32, stacksize: u32, - flags: u16, + flags: u32, co_code: PyBytesRef, consts: PyTupleRef, names: PyTupleRef, @@ -628,7 +628,7 @@ impl PyCode { } #[pygetset] - const fn co_flags(&self) -> u16 { + const fn co_flags(&self) -> u32 { self.code.flags.bits() } @@ -650,6 +650,12 @@ impl PyCode { vm.ctx.new_bytes(bytes.to_vec()) } + #[pygetset] + pub fn _co_code_adaptive(&self, vm: &VirtualMachine) -> crate::builtins::PyBytesRef { + // RustPython doesn't have adaptive/specialized bytecode, so return regular co_code + self.co_code(vm) + } + #[pygetset] pub fn co_freevars(&self, vm: &VirtualMachine) -> PyTupleRef { let names = self diff --git a/crates/vm/src/builtins/complex.rs b/crates/vm/src/builtins/complex.rs index 78729b2f5c0..dd68061557d 100644 --- a/crates/vm/src/builtins/complex.rs +++ b/crates/vm/src/builtins/complex.rs @@ -276,9 +276,13 @@ impl PyComplex { } #[pymethod] - fn __format__(&self, spec: PyStrRef, vm: &VirtualMachine) -> PyResult<String> { + fn __format__(zelf: &Py<Self>, spec: PyStrRef, vm: &VirtualMachine) -> PyResult<String> { + // Empty format spec: equivalent to str(self) + if spec.is_empty() { + return Ok(zelf.as_object().str(vm)?.as_str().to_owned()); + } FormatSpec::parse(spec.as_str()) - .and_then(|format_spec| format_spec.format_complex(&self.value)) + .and_then(|format_spec| format_spec.format_complex(&zelf.value)) .map_err(|err| err.into_pyexception(vm)) } } diff --git a/crates/vm/src/builtins/coroutine.rs b/crates/vm/src/builtins/coroutine.rs index 9e8d5d534f1..bca00f84367 100644 --- a/crates/vm/src/builtins/coroutine.rs +++ b/crates/vm/src/builtins/coroutine.rs @@ -5,18 +5,25 @@ use crate::{ coroutine::{Coro, warn_deprecated_throw_signature}, frame::FrameRef, function::OptionalArg, + object::{Traverse, TraverseFn}, protocol::PyIterReturn, types::{IterNext, Iterable, Representable, SelfIter}, }; use crossbeam_utils::atomic::AtomicCell; -#[pyclass(module = false, name = "coroutine")] +#[pyclass(module = false, name = "coroutine", traverse = "manual")] #[derive(Debug)] // PyCoro_Type in CPython pub struct PyCoroutine { inner: Coro, } +unsafe impl Traverse for PyCoroutine { + fn traverse(&self, tracer_fn: &mut TraverseFn<'_>) { + self.inner.traverse(tracer_fn); + } +} + impl PyPayload for PyCoroutine { #[inline] fn class(ctx: &Context) -> &'static Py<PyType> { @@ -69,8 +76,12 @@ impl PyCoroutine { self.inner.frame().yield_from_target() } #[pygetset] - fn cr_frame(&self, _vm: &VirtualMachine) -> FrameRef { - self.inner.frame() + fn cr_frame(&self, _vm: &VirtualMachine) -> Option<FrameRef> { + if self.inner.closed() { + None + } else { + Some(self.inner.frame()) + } } #[pygetset] fn cr_running(&self, _vm: &VirtualMachine) -> bool { @@ -138,7 +149,7 @@ impl IterNext for PyCoroutine { } } -#[pyclass(module = false, name = "coroutine_wrapper")] +#[pyclass(module = false, name = "coroutine_wrapper", traverse = "manual")] #[derive(Debug)] // PyCoroWrapper_Type in CPython pub struct PyCoroutineWrapper { @@ -146,6 +157,12 @@ pub struct PyCoroutineWrapper { closed: AtomicCell<bool>, } +unsafe impl Traverse for PyCoroutineWrapper { + fn traverse(&self, tracer_fn: &mut TraverseFn<'_>) { + self.coro.traverse(tracer_fn); + } +} + impl PyPayload for PyCoroutineWrapper { #[inline] fn class(ctx: &Context) -> &'static Py<PyType> { @@ -205,6 +222,12 @@ impl IterNext for PyCoroutineWrapper { } } +impl Drop for PyCoroutine { + fn drop(&mut self) { + self.inner.frame().clear_generator(); + } +} + pub fn init(ctx: &Context) { PyCoroutine::extend_class(ctx, ctx.types.coroutine_type); PyCoroutineWrapper::extend_class(ctx, ctx.types.coroutine_wrapper_type); diff --git a/crates/vm/src/builtins/descriptor.rs b/crates/vm/src/builtins/descriptor.rs index 89dafdd14b7..e1b92746a0f 100644 --- a/crates/vm/src/builtins/descriptor.rs +++ b/crates/vm/src/builtins/descriptor.rs @@ -170,6 +170,7 @@ impl Representable for PyMethodDescriptor { #[derive(Debug)] pub enum MemberKind { + Object = 6, Bool = 14, ObjectEx = 16, } @@ -253,11 +254,20 @@ fn calculate_qualname(descr: &PyDescriptorOwned, vm: &VirtualMachine) -> PyResul } } -#[pyclass( - with(GetDescriptor, Representable), - flags(BASETYPE, DISALLOW_INSTANTIATION) -)] +#[pyclass(with(GetDescriptor, Representable), flags(DISALLOW_INSTANTIATION))] impl PyMemberDescriptor { + #[pymember] + fn __objclass__(vm: &VirtualMachine, zelf: PyObjectRef) -> PyResult { + let zelf: &Py<Self> = zelf.try_to_value(vm)?; + Ok(zelf.common.typ.clone().into()) + } + + #[pymember] + fn __name__(vm: &VirtualMachine, zelf: PyObjectRef) -> PyResult { + let zelf: &Py<Self> = zelf.try_to_value(vm)?; + Ok(zelf.common.name.to_owned().into()) + } + #[pygetset] fn __doc__(&self) -> Option<String> { self.member.doc.to_owned() @@ -276,6 +286,23 @@ impl PyMemberDescriptor { }) } + #[pymethod] + fn __reduce__(&self, vm: &VirtualMachine) -> PyResult { + let builtins_getattr = vm.builtins.get_attr("getattr", vm)?; + Ok(vm + .ctx + .new_tuple(vec![ + builtins_getattr, + vm.ctx + .new_tuple(vec![ + self.common.typ.clone().into(), + vm.ctx.new_str(self.common.name.as_str()).into(), + ]) + .into(), + ]) + .into()) + } + #[pyslot] fn descr_set( zelf: &PyObject, @@ -306,6 +333,7 @@ fn get_slot_from_object( vm: &VirtualMachine, ) -> PyResult { let slot = match member.kind { + MemberKind::Object => obj.get_slot(offset).unwrap_or_else(|| vm.ctx.none()), MemberKind::Bool => obj .get_slot(offset) .unwrap_or_else(|| vm.ctx.new_bool(false).into()), @@ -325,25 +353,38 @@ fn set_slot_at_object( vm: &VirtualMachine, ) -> PyResult<()> { match member.kind { + MemberKind::Object => match value { + PySetterValue::Assign(v) => { + obj.set_slot(offset, Some(v)); + } + PySetterValue::Delete => { + obj.set_slot(offset, None); + } + }, MemberKind::Bool => { match value { PySetterValue::Assign(v) => { if !v.class().is(vm.ctx.types.bool_type) { return Err(vm.new_type_error("attribute value type must be bool")); } - obj.set_slot(offset, Some(v)) } - PySetterValue::Delete => obj.set_slot(offset, None), - }; - } - MemberKind::ObjectEx => { - let value = match value { - PySetterValue::Assign(v) => Some(v), - PySetterValue::Delete => None, + PySetterValue::Delete => { + return Err(vm.new_type_error("can't delete numeric/char attribute".to_owned())); + } }; - obj.set_slot(offset, value); } + MemberKind::ObjectEx => match value { + PySetterValue::Assign(v) => { + obj.set_slot(offset, Some(v)); + } + PySetterValue::Delete => { + if obj.get_slot(offset).is_none() { + return Err(vm.new_attribute_error(member.name.clone())); + } + obj.set_slot(offset, None); + } + }, } Ok(()) @@ -364,26 +405,23 @@ impl GetDescriptor for PyMemberDescriptor { fn descr_get( zelf: PyObjectRef, obj: Option<PyObjectRef>, - cls: Option<PyObjectRef>, + _cls: Option<PyObjectRef>, vm: &VirtualMachine, ) -> PyResult { let descr = Self::_as_pyref(&zelf, vm)?; match obj { - Some(x) => descr.member.get(x, vm), - None => { - // When accessed from class (not instance), for __doc__ member descriptor, - // return the class's docstring if available - // When accessed from class (not instance), check if the class has - // an attribute with the same name as this member descriptor - if let Some(cls) = cls - && let Ok(cls_type) = cls.downcast::<PyType>() - && let Some(interned) = vm.ctx.interned_str(descr.member.name.as_str()) - && let Some(attr) = cls_type.attributes.read().get(&interned) - { - return Ok(attr.clone()); + Some(x) => { + if !x.class().fast_issubclass(&descr.common.typ) { + return Err(vm.new_type_error(format!( + "descriptor '{}' for '{}' objects doesn't apply to a '{}' object", + descr.common.name, + descr.common.typ.name(), + x.class().name() + ))); } - Ok(zelf) + descr.member.get(x, vm) } + None => Ok(zelf), } } } diff --git a/crates/vm/src/builtins/dict.rs b/crates/vm/src/builtins/dict.rs index 43f7100e963..fcb51c2ca0e 100644 --- a/crates/vm/src/builtins/dict.rs +++ b/crates/vm/src/builtins/dict.rs @@ -2,6 +2,7 @@ use super::{ IterStatus, PositionIterInternal, PyBaseExceptionRef, PyGenericAlias, PyMappingProxy, PySet, PyStr, PyStrRef, PyTupleRef, PyType, PyTypeRef, set::PySetInner, }; +use crate::object::{Traverse, TraverseFn}; use crate::{ AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyRefExact, PyResult, TryFromObject, atomic_func, @@ -29,13 +30,28 @@ use std::sync::LazyLock; pub type DictContentType = dict_inner::Dict; -#[pyclass(module = false, name = "dict", unhashable = true, traverse)] +#[pyclass(module = false, name = "dict", unhashable = true, traverse = "manual")] #[derive(Default)] pub struct PyDict { entries: DictContentType, } pub type PyDictRef = PyRef<PyDict>; +// SAFETY: Traverse properly visits all owned PyObjectRefs +unsafe impl Traverse for PyDict { + fn traverse(&self, traverse_fn: &mut TraverseFn<'_>) { + self.entries.traverse(traverse_fn); + } + + fn clear(&mut self, out: &mut Vec<PyObjectRef>) { + // Pop all entries and collect both keys and values + for (key, value) in self.entries.drain_entries() { + out.push(key); + out.push(value); + } + } +} + impl fmt::Debug for PyDict { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { // TODO: implement more detailed, non-recursive Debug formatter @@ -62,6 +78,24 @@ impl PyDict { &self.entries } + /// Returns all keys as a Vec, atomically under a single read lock. + /// Thread-safe: prevents "dictionary changed size during iteration" errors. + pub fn keys_vec(&self) -> Vec<PyObjectRef> { + self.entries.keys() + } + + /// Returns all values as a Vec, atomically under a single read lock. + /// Thread-safe: prevents "dictionary changed size during iteration" errors. + pub fn values_vec(&self) -> Vec<PyObjectRef> { + self.entries.values() + } + + /// Returns all items as a Vec, atomically under a single read lock. + /// Thread-safe: prevents "dictionary changed size during iteration" errors. + pub fn items_vec(&self) -> Vec<(PyObjectRef, PyObjectRef)> { + self.entries.items() + } + // Used in update and ior. pub(crate) fn merge_object(&self, other: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { let casted: Result<PyRefExact<Self>, _> = other.downcast_exact(vm); diff --git a/crates/vm/src/builtins/float.rs b/crates/vm/src/builtins/float.rs index f101a3aa8e3..230d288b790 100644 --- a/crates/vm/src/builtins/float.rs +++ b/crates/vm/src/builtins/float.rs @@ -116,7 +116,7 @@ fn inner_divmod(v1: f64, v2: f64, vm: &VirtualMachine) -> PyResult<(f64, f64)> { pub fn float_pow(v1: f64, v2: f64, vm: &VirtualMachine) -> PyResult { if v1.is_zero() && v2.is_sign_negative() { - let msg = "0.0 cannot be raised to a negative power"; + let msg = "zero to a negative power"; Err(vm.new_zero_division_error(msg.to_owned())) } else if v1.is_sign_negative() && (v2.floor() - v2).abs() > f64::EPSILON { let v1 = Complex64::new(v1, 0.); @@ -214,9 +214,13 @@ fn float_from_string(val: PyObjectRef, vm: &VirtualMachine) -> PyResult<f64> { )] impl PyFloat { #[pymethod] - fn __format__(&self, spec: PyStrRef, vm: &VirtualMachine) -> PyResult<String> { + fn __format__(zelf: &Py<Self>, spec: PyStrRef, vm: &VirtualMachine) -> PyResult<String> { + // Empty format spec: equivalent to str(self) + if spec.is_empty() { + return Ok(zelf.as_object().str(vm)?.as_str().to_owned()); + } FormatSpec::parse(spec.as_str()) - .and_then(|format_spec| format_spec.format_float(self.value)) + .and_then(|format_spec| format_spec.format_float(zelf.value)) .map_err(|err| err.into_pyexception(vm)) } diff --git a/crates/vm/src/builtins/frame.rs b/crates/vm/src/builtins/frame.rs index 28c4e751476..94ffffafc39 100644 --- a/crates/vm/src/builtins/frame.rs +++ b/crates/vm/src/builtins/frame.rs @@ -115,10 +115,47 @@ impl Frame { PySetterValue::Delete => Err(vm.new_type_error("can't delete numeric/char attribute")), } } + + #[pymember(type = "bool")] + fn f_trace_opcodes(vm: &VirtualMachine, zelf: PyObjectRef) -> PyResult { + let zelf: FrameRef = zelf.downcast().unwrap_or_else(|_| unreachable!()); + let trace_opcodes = zelf.trace_opcodes.lock(); + Ok(vm.ctx.new_bool(*trace_opcodes).into()) + } + + #[pymember(type = "bool", setter)] + fn set_f_trace_opcodes( + vm: &VirtualMachine, + zelf: PyObjectRef, + value: PySetterValue, + ) -> PyResult<()> { + match value { + PySetterValue::Assign(value) => { + let zelf: FrameRef = zelf.downcast().unwrap_or_else(|_| unreachable!()); + + let value: PyIntRef = value + .downcast() + .map_err(|_| vm.new_type_error("attribute value type must be bool"))?; + + let mut trace_opcodes = zelf.trace_opcodes.lock(); + *trace_opcodes = !value.as_bigint().is_zero(); + + // TODO: Implement the equivalent of _PyEval_SetOpcodeTrace() + + Ok(()) + } + PySetterValue::Delete => Err(vm.new_type_error("can't delete numeric/char attribute")), + } + } } #[pyclass] impl Py<Frame> { + #[pygetset] + fn f_generator(&self) -> Option<PyObjectRef> { + self.generator.to_owned() + } + #[pygetset] pub fn f_back(&self, vm: &VirtualMachine) -> Option<PyRef<Frame>> { // TODO: actually store f_back inside Frame struct diff --git a/crates/vm/src/builtins/function.rs b/crates/vm/src/builtins/function.rs index 3e5048e133f..163b484a8b0 100644 --- a/crates/vm/src/builtins/function.rs +++ b/crates/vm/src/builtins/function.rs @@ -2,8 +2,8 @@ mod jit; use super::{ - PyAsyncGen, PyCode, PyCoroutine, PyDictRef, PyGenerator, PyStr, PyStrRef, PyTuple, PyTupleRef, - PyType, + PyAsyncGen, PyCode, PyCoroutine, PyDictRef, PyGenerator, PyModule, PyStr, PyStrRef, PyTuple, + PyTupleRef, PyType, }; #[cfg(feature = "jit")] use crate::common::lock::OnceCell; @@ -36,7 +36,8 @@ pub struct PyFunction { name: PyMutex<PyStrRef>, qualname: PyMutex<PyStrRef>, type_params: PyMutex<PyTupleRef>, - annotations: PyMutex<PyDictRef>, + annotations: PyMutex<Option<PyDictRef>>, + annotate: PyMutex<Option<PyObjectRef>>, module: PyMutex<PyObjectRef>, doc: PyMutex<PyObjectRef>, #[cfg(feature = "jit")] @@ -50,6 +51,70 @@ unsafe impl Traverse for PyFunction { closure.as_untyped().traverse(tracer_fn); } self.defaults_and_kwdefaults.traverse(tracer_fn); + // Traverse additional fields that may contain references + self.type_params.lock().traverse(tracer_fn); + self.annotations.lock().traverse(tracer_fn); + self.module.lock().traverse(tracer_fn); + self.doc.lock().traverse(tracer_fn); + } + + fn clear(&mut self, out: &mut Vec<crate::PyObjectRef>) { + // Pop closure if present (equivalent to Py_CLEAR(func_closure)) + if let Some(closure) = self.closure.take() { + out.push(closure.into()); + } + + // Pop defaults and kwdefaults + if let Some(mut guard) = self.defaults_and_kwdefaults.try_lock() { + if let Some(defaults) = guard.0.take() { + out.push(defaults.into()); + } + if let Some(kwdefaults) = guard.1.take() { + out.push(kwdefaults.into()); + } + } + + // Clear annotations and annotate (Py_CLEAR) + if let Some(mut guard) = self.annotations.try_lock() + && let Some(annotations) = guard.take() + { + out.push(annotations.into()); + } + if let Some(mut guard) = self.annotate.try_lock() + && let Some(annotate) = guard.take() + { + out.push(annotate); + } + + // Clear module, doc, and type_params (Py_CLEAR) + if let Some(mut guard) = self.module.try_lock() { + let old_module = + std::mem::replace(&mut *guard, Context::genesis().none.to_owned().into()); + out.push(old_module); + } + if let Some(mut guard) = self.doc.try_lock() { + let old_doc = std::mem::replace(&mut *guard, Context::genesis().none.to_owned().into()); + out.push(old_doc); + } + if let Some(mut guard) = self.type_params.try_lock() { + let old_type_params = + std::mem::replace(&mut *guard, Context::genesis().empty_tuple.to_owned()); + out.push(old_type_params.into()); + } + + // Replace name and qualname with empty string to break potential str subclass cycles + // name and qualname could be str subclasses, so they could have reference cycles + if let Some(mut guard) = self.name.try_lock() { + let old_name = std::mem::replace(&mut *guard, Context::genesis().empty_str.to_owned()); + out.push(old_name.into()); + } + if let Some(mut guard) = self.qualname.try_lock() { + let old_qualname = + std::mem::replace(&mut *guard, Context::genesis().empty_str.to_owned()); + out.push(old_qualname.into()); + } + + // Note: globals, builtins, code are NOT cleared (required to be non-NULL) } } @@ -67,9 +132,26 @@ impl PyFunction { if let Some(frame) = vm.current_frame() { frame.builtins.clone().into() } else { - vm.builtins.clone().into() + vm.builtins.dict().into() } }); + // If builtins is a module, use its __dict__ instead + let builtins = if let Some(module) = builtins.downcast_ref::<PyModule>() { + module.dict().into() + } else { + builtins + }; + + // Get docstring from co_consts[0] if HAS_DOCSTRING flag is set + let doc = if code.code.flags.contains(bytecode::CodeFlags::HAS_DOCSTRING) { + code.code + .constants + .first() + .map(|c| c.as_object().to_owned()) + .unwrap_or_else(|| vm.ctx.none()) + } else { + vm.ctx.none() + }; let qualname = vm.ctx.new_str(code.qualname.as_str()); let func = Self { @@ -81,9 +163,10 @@ impl PyFunction { name, qualname: PyMutex::new(qualname), type_params: PyMutex::new(vm.ctx.empty_tuple.clone()), - annotations: PyMutex::new(vm.ctx.new_dict()), + annotations: PyMutex::new(None), + annotate: PyMutex::new(None), module: PyMutex::new(module), - doc: PyMutex::new(vm.ctx.none()), + doc: PyMutex::new(doc), #[cfg(feature = "jit")] jitted_code: OnceCell::new(), }; @@ -124,7 +207,7 @@ impl PyFunction { let mut vararg_offset = total_args; // Pack other positional arguments in to *args: - if code.flags.contains(bytecode::CodeFlags::HAS_VARARGS) { + if code.flags.contains(bytecode::CodeFlags::VARARGS) { let vararg_value = vm.ctx.new_tuple(args_iter.collect()); fastlocals[vararg_offset] = Some(vararg_value.into()); vararg_offset += 1; @@ -155,7 +238,7 @@ impl PyFunction { } // Do we support `**kwargs` ? - let kwargs = if code.flags.contains(bytecode::CodeFlags::HAS_VARKEYWORDS) { + let kwargs = if code.flags.contains(bytecode::CodeFlags::VARKEYWORDS) { let d = vm.ctx.new_dict(); fastlocals[vararg_offset] = Some(d.clone().into()); Some(d) @@ -358,7 +441,7 @@ impl PyFunction { ))); } }; - *self.annotations.lock() = annotations; + *self.annotations.lock() = Some(annotations); } else if attr == bytecode::MakeFunctionFlags::CLOSURE { // For closure, we need special handling // The closure tuple contains cell objects @@ -382,6 +465,12 @@ impl PyFunction { )) })?; *self.type_params.lock() = type_params; + } else if attr == bytecode::MakeFunctionFlags::ANNOTATE { + // PEP 649: Store the __annotate__ function closure + if !attr_value.is_callable() { + return Err(vm.new_type_error("__annotate__ must be callable".to_owned())); + } + *self.annotate.lock() = Some(attr_value); } else { unreachable!("This is a compiler bug"); } @@ -414,7 +503,7 @@ impl Py<PyFunction> { let code = self.code.lock().clone(); - let locals = if code.flags.contains(bytecode::CodeFlags::NEW_LOCALS) { + let locals = if code.flags.contains(bytecode::CodeFlags::NEWLOCALS) { ArgMapping::from_dict_exact(vm.ctx.new_dict()) } else if let Some(locals) = locals { locals @@ -436,17 +525,26 @@ impl Py<PyFunction> { self.fill_locals_from_args(&frame, func_args, vm)?; // If we have a generator, create a new generator - let is_gen = code.flags.contains(bytecode::CodeFlags::IS_GENERATOR); - let is_coro = code.flags.contains(bytecode::CodeFlags::IS_COROUTINE); + let is_gen = code.flags.contains(bytecode::CodeFlags::GENERATOR); + let is_coro = code.flags.contains(bytecode::CodeFlags::COROUTINE); match (is_gen, is_coro) { (true, false) => { - Ok(PyGenerator::new(frame, self.__name__(), self.__qualname__()).into_pyobject(vm)) + let obj = PyGenerator::new(frame.clone(), self.__name__(), self.__qualname__()) + .into_pyobject(vm); + frame.set_generator(&obj); + Ok(obj) } (false, true) => { - Ok(PyCoroutine::new(frame, self.__name__(), self.__qualname__()).into_pyobject(vm)) + let obj = PyCoroutine::new(frame.clone(), self.__name__(), self.__qualname__()) + .into_pyobject(vm); + frame.set_generator(&obj); + Ok(obj) } (true, true) => { - Ok(PyAsyncGen::new(frame, self.__name__(), self.__qualname__()).into_pyobject(vm)) + let obj = PyAsyncGen::new(frame.clone(), self.__name__(), self.__qualname__()) + .into_pyobject(vm); + frame.set_generator(&obj); + Ok(obj) } (false, false) => vm.run_frame(frame), } @@ -568,13 +666,103 @@ impl PyFunction { } #[pygetset] - fn __annotations__(&self) -> PyDictRef { - self.annotations.lock().clone() + fn __annotations__(&self, vm: &VirtualMachine) -> PyResult<PyDictRef> { + // First check if we have cached annotations + { + let annotations = self.annotations.lock(); + if let Some(ref ann) = *annotations { + return Ok(ann.clone()); + } + } + + // Check for callable __annotate__ and clone it before calling + let annotate_fn = { + let annotate = self.annotate.lock(); + if let Some(ref func) = *annotate + && func.is_callable() + { + Some(func.clone()) + } else { + None + } + }; + + // Release locks before calling __annotate__ to avoid deadlock + if let Some(annotate_fn) = annotate_fn { + let one = vm.ctx.new_int(1); + let ann_dict = annotate_fn.call((one,), vm)?; + let ann_dict = ann_dict + .downcast::<crate::builtins::PyDict>() + .map_err(|obj| { + vm.new_type_error(format!( + "__annotate__ returned non-dict of type '{}'", + obj.class().name() + )) + })?; + + // Cache the result + *self.annotations.lock() = Some(ann_dict.clone()); + return Ok(ann_dict); + } + + // No __annotate__ or not callable, create empty dict + let new_dict = vm.ctx.new_dict(); + *self.annotations.lock() = Some(new_dict.clone()); + Ok(new_dict) + } + + #[pygetset(setter)] + fn set___annotations__( + &self, + value: PySetterValue<Option<PyObjectRef>>, + vm: &VirtualMachine, + ) -> PyResult<()> { + let annotations = match value { + PySetterValue::Assign(Some(value)) => { + let annotations = value.downcast::<crate::builtins::PyDict>().map_err(|_| { + vm.new_type_error("__annotations__ must be set to a dict object") + })?; + Some(annotations) + } + PySetterValue::Assign(None) | PySetterValue::Delete => None, + }; + *self.annotations.lock() = annotations; + + // Clear __annotate__ when __annotations__ is set + *self.annotate.lock() = None; + Ok(()) + } + + #[pygetset] + fn __annotate__(&self, vm: &VirtualMachine) -> PyObjectRef { + self.annotate + .lock() + .clone() + .unwrap_or_else(|| vm.ctx.none()) } #[pygetset(setter)] - fn set___annotations__(&self, annotations: PyDictRef) { - *self.annotations.lock() = annotations + fn set___annotate__( + &self, + value: PySetterValue<Option<PyObjectRef>>, + vm: &VirtualMachine, + ) -> PyResult<()> { + let annotate = match value { + PySetterValue::Assign(Some(value)) => { + if !value.is_callable() { + return Err(vm.new_type_error("__annotate__ must be callable or None")); + } + // Clear cached __annotations__ when __annotate__ is set + *self.annotations.lock() = None; + Some(value) + } + PySetterValue::Assign(None) => None, + PySetterValue::Delete => { + return Err(vm.new_type_error("__annotate__ cannot be deleted")); + } + }; + *self.annotate.lock() = annotate; + Ok(()) } #[pygetset] @@ -676,14 +864,14 @@ pub struct PyFunctionNewArgs { code: PyRef<PyCode>, #[pyarg(positional)] globals: PyDictRef, - #[pyarg(any, optional)] + #[pyarg(any, optional, error_msg = "arg 3 (name) must be None or string")] name: OptionalArg<PyStrRef>, - #[pyarg(any, optional)] - defaults: OptionalArg<PyTupleRef>, - #[pyarg(any, optional)] - closure: OptionalArg<PyTupleRef>, - #[pyarg(any, optional)] - kwdefaults: OptionalArg<PyDictRef>, + #[pyarg(any, optional, error_msg = "arg 4 (defaults) must be None or tuple")] + argdefs: Option<PyTupleRef>, + #[pyarg(any, optional, error_msg = "arg 5 (closure) must be None or tuple")] + closure: Option<PyTupleRef>, + #[pyarg(any, optional, error_msg = "arg 6 (kwdefaults) must be None or dict")] + kwdefaults: Option<PyDictRef>, } impl Constructor for PyFunction { @@ -691,7 +879,7 @@ impl Constructor for PyFunction { fn py_new(_cls: &Py<PyType>, args: Self::Args, vm: &VirtualMachine) -> PyResult<Self> { // Handle closure - must be a tuple of cells - let closure = if let Some(closure_tuple) = args.closure.into_option() { + let closure = if let Some(closure_tuple) = args.closure { // Check that closure length matches code's free variables if closure_tuple.len() != args.code.freevars.len() { return Err(vm.new_value_error(format!( @@ -722,10 +910,10 @@ impl Constructor for PyFunction { if let Some(closure_tuple) = closure { func.closure = Some(closure_tuple); } - if let Some(defaults) = args.defaults.into_option() { - func.defaults_and_kwdefaults.lock().0 = Some(defaults); + if let Some(argdefs) = args.argdefs { + func.defaults_and_kwdefaults.lock().0 = Some(argdefs); } - if let Some(kwdefaults) = args.kwdefaults.into_option() { + if let Some(kwdefaults) = args.kwdefaults { func.defaults_and_kwdefaults.lock().1 = Some(kwdefaults); } diff --git a/crates/vm/src/builtins/function/jit.rs b/crates/vm/src/builtins/function/jit.rs index de0a528b734..a28335900da 100644 --- a/crates/vm/src/builtins/function/jit.rs +++ b/crates/vm/src/builtins/function/jit.rs @@ -72,7 +72,7 @@ pub fn get_jit_arg_types(func: &Py<PyFunction>, vm: &VirtualMachine) -> PyResult if code .flags - .intersects(CodeFlags::HAS_VARARGS | CodeFlags::HAS_VARKEYWORDS) + .intersects(CodeFlags::VARARGS | CodeFlags::VARKEYWORDS) { return Err(new_jit_error( "Can't jit functions with variable number of arguments".to_owned(), diff --git a/crates/vm/src/builtins/generator.rs b/crates/vm/src/builtins/generator.rs index 9a1e737500b..f4deb8cc7a2 100644 --- a/crates/vm/src/builtins/generator.rs +++ b/crates/vm/src/builtins/generator.rs @@ -9,16 +9,23 @@ use crate::{ coroutine::{Coro, warn_deprecated_throw_signature}, frame::FrameRef, function::OptionalArg, + object::{Traverse, TraverseFn}, protocol::PyIterReturn, types::{IterNext, Iterable, Representable, SelfIter}, }; -#[pyclass(module = false, name = "generator")] +#[pyclass(module = false, name = "generator", traverse = "manual")] #[derive(Debug)] pub struct PyGenerator { inner: Coro, } +unsafe impl Traverse for PyGenerator { + fn traverse(&self, tracer_fn: &mut TraverseFn<'_>) { + self.inner.traverse(tracer_fn); + } +} + impl PyPayload for PyGenerator { #[inline] fn class(ctx: &Context) -> &'static Py<PyType> { @@ -59,8 +66,12 @@ impl PyGenerator { } #[pygetset] - fn gi_frame(&self, _vm: &VirtualMachine) -> FrameRef { - self.inner.frame() + fn gi_frame(&self, _vm: &VirtualMachine) -> Option<FrameRef> { + if self.inner.closed() { + None + } else { + Some(self.inner.frame()) + } } #[pygetset] @@ -129,6 +140,12 @@ impl IterNext for PyGenerator { } } +impl Drop for PyGenerator { + fn drop(&mut self) { + self.inner.frame().clear_generator(); + } +} + pub fn init(ctx: &Context) { PyGenerator::extend_class(ctx, ctx.types.generator_type); } diff --git a/crates/vm/src/builtins/genericalias.rs b/crates/vm/src/builtins/genericalias.rs index e9150e4c088..21034e08f0e 100644 --- a/crates/vm/src/builtins/genericalias.rs +++ b/crates/vm/src/builtins/genericalias.rs @@ -235,11 +235,11 @@ impl PyGenericAlias { Err(vm.new_type_error("issubclass() argument 2 cannot be a parameterized generic")) } - fn __ror__(zelf: PyObjectRef, other: PyObjectRef, vm: &VirtualMachine) -> PyObjectRef { + fn __ror__(zelf: PyObjectRef, other: PyObjectRef, vm: &VirtualMachine) -> PyResult { type_::or_(other, zelf, vm) } - fn __or__(zelf: PyObjectRef, other: PyObjectRef, vm: &VirtualMachine) -> PyObjectRef { + fn __or__(zelf: PyObjectRef, other: PyObjectRef, vm: &VirtualMachine) -> PyResult { type_::or_(zelf, other, vm) } } @@ -509,7 +509,7 @@ impl AsMapping for PyGenericAlias { impl AsNumber for PyGenericAlias { fn as_number() -> &'static PyNumberMethods { static AS_NUMBER: PyNumberMethods = PyNumberMethods { - or: Some(|a, b, vm| Ok(PyGenericAlias::__or__(a.to_owned(), b.to_owned(), vm))), + or: Some(|a, b, vm| PyGenericAlias::__or__(a.to_owned(), b.to_owned(), vm)), ..PyNumberMethods::NOT_IMPLEMENTED }; &AS_NUMBER diff --git a/crates/vm/src/builtins/int.rs b/crates/vm/src/builtins/int.rs index 9b74b66e3a7..273199f69ba 100644 --- a/crates/vm/src/builtins/int.rs +++ b/crates/vm/src/builtins/int.rs @@ -12,7 +12,8 @@ use crate::{ }, convert::{IntoPyException, ToPyObject, ToPyResult}, function::{ - ArgByteOrder, ArgIntoBool, FuncArgs, OptionalArg, PyArithmeticValue, PyComparisonValue, + ArgByteOrder, ArgIntoBool, FuncArgs, OptionalArg, OptionalOption, PyArithmeticValue, + PyComparisonValue, }, protocol::{PyNumberMethods, handle_bytes_to_int_err}, types::{AsNumber, Comparable, Constructor, Hashable, PyComparisonOp, Representable}, @@ -286,6 +287,13 @@ impl PyInt { where I: PrimInt + TryFrom<&'a BigInt>, { + // TODO: Python 3.14+: ValueError for negative int to unsigned type + // See stdlib_socket.py socket.htonl(-1) + // + // if I::min_value() == I::zero() && self.as_bigint().sign() == Sign::Minus { + // return Err(vm.new_value_error("Cannot convert negative int".to_owned())); + // } + I::try_from(self.as_bigint()).map_err(|_| { vm.new_overflow_error(format!( "Python int too large to convert to Rust {}", @@ -381,10 +389,10 @@ impl PyInt { #[pymethod] fn __round__( zelf: PyRef<Self>, - ndigits: OptionalArg<PyIntRef>, + ndigits: OptionalOption<PyIntRef>, vm: &VirtualMachine, ) -> PyResult<PyRef<Self>> { - if let OptionalArg::Present(ndigits) = ndigits { + if let Some(ndigits) = ndigits.flatten() { let ndigits = ndigits.as_bigint(); // round(12345, -2) == 12300 // If precision >= 0, then any integer is already rounded correctly @@ -436,9 +444,13 @@ impl PyInt { } #[pymethod] - fn __format__(&self, spec: PyStrRef, vm: &VirtualMachine) -> PyResult<String> { + fn __format__(zelf: &Py<Self>, spec: PyStrRef, vm: &VirtualMachine) -> PyResult<String> { + // Empty format spec on a subclass: equivalent to str(self) + if spec.is_empty() && !zelf.class().is(vm.ctx.types.int_type) { + return Ok(zelf.as_object().str(vm)?.as_str().to_owned()); + } FormatSpec::parse(spec.as_str()) - .and_then(|format_spec| format_spec.format_int(&self.value)) + .and_then(|format_spec| format_spec.format_int(&zelf.value)) .map_err(|err| err.into_pyexception(vm)) } diff --git a/crates/vm/src/builtins/interpolation.rs b/crates/vm/src/builtins/interpolation.rs new file mode 100644 index 00000000000..d8127c151e3 --- /dev/null +++ b/crates/vm/src/builtins/interpolation.rs @@ -0,0 +1,213 @@ +use super::{PyStr, PyStrRef, PyTupleRef, PyType, tuple::IntoPyTuple}; +use crate::{ + AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, + class::PyClassImpl, + common::hash::PyHash, + convert::ToPyObject, + function::{OptionalArg, PyComparisonValue}, + types::{Comparable, Constructor, Hashable, PyComparisonOp, Representable}, +}; + +/// Interpolation object for t-strings (PEP 750). +/// +/// Represents an interpolated expression within a template string. +#[pyclass(module = "string.templatelib", name = "Interpolation")] +#[derive(Debug, Clone)] +pub struct PyInterpolation { + pub value: PyObjectRef, + pub expression: PyStrRef, + pub conversion: PyObjectRef, // None or 's', 'r', 'a' + pub format_spec: PyStrRef, +} + +impl PyPayload for PyInterpolation { + #[inline] + fn class(ctx: &Context) -> &'static Py<PyType> { + ctx.types.interpolation_type + } +} + +impl PyInterpolation { + pub fn new( + value: PyObjectRef, + expression: PyStrRef, + conversion: PyObjectRef, + format_spec: PyStrRef, + vm: &VirtualMachine, + ) -> PyResult<Self> { + // Validate conversion like _PyInterpolation_Build does + let is_valid = vm.is_none(&conversion) + || conversion + .downcast_ref::<PyStr>() + .is_some_and(|s| matches!(s.as_str(), "s" | "r" | "a")); + if !is_valid { + return Err(vm.new_exception_msg( + vm.ctx.exceptions.system_error.to_owned(), + "Interpolation() argument 'conversion' must be one of 's', 'a' or 'r'".to_owned(), + )); + } + Ok(Self { + value, + expression, + conversion, + format_spec, + }) + } +} + +impl Constructor for PyInterpolation { + type Args = InterpolationArgs; + + fn py_new(_cls: &Py<PyType>, args: Self::Args, vm: &VirtualMachine) -> PyResult<Self> { + let conversion: PyObjectRef = if let Some(s) = args.conversion { + let s_str = s.as_str(); + if s_str.len() != 1 || !matches!(s_str.chars().next(), Some('s' | 'r' | 'a')) { + return Err(vm.new_value_error( + "Interpolation() argument 'conversion' must be one of 's', 'a' or 'r'", + )); + } + s.into() + } else { + vm.ctx.none() + }; + + let expression = args + .expression + .unwrap_or_else(|| vm.ctx.empty_str.to_owned()); + let format_spec = args + .format_spec + .unwrap_or_else(|| vm.ctx.empty_str.to_owned()); + + Ok(PyInterpolation { + value: args.value, + expression, + conversion, + format_spec, + }) + } +} + +#[derive(FromArgs)] +pub struct InterpolationArgs { + #[pyarg(positional)] + value: PyObjectRef, + #[pyarg(any, optional)] + expression: OptionalArg<PyStrRef>, + #[pyarg( + any, + optional, + error_msg = "Interpolation() argument 'conversion' must be str or None" + )] + conversion: Option<PyStrRef>, + #[pyarg(any, optional)] + format_spec: OptionalArg<PyStrRef>, +} + +#[pyclass(with(Constructor, Comparable, Hashable, Representable))] +impl PyInterpolation { + #[pyattr] + fn __match_args__(ctx: &Context) -> PyTupleRef { + ctx.new_tuple(vec![ + ctx.intern_str("value").to_owned().into(), + ctx.intern_str("expression").to_owned().into(), + ctx.intern_str("conversion").to_owned().into(), + ctx.intern_str("format_spec").to_owned().into(), + ]) + } + + #[pygetset] + fn value(&self) -> PyObjectRef { + self.value.clone() + } + + #[pygetset] + fn expression(&self) -> PyStrRef { + self.expression.clone() + } + + #[pygetset] + fn conversion(&self) -> PyObjectRef { + self.conversion.clone() + } + + #[pygetset] + fn format_spec(&self) -> PyStrRef { + self.format_spec.clone() + } + + #[pymethod] + fn __reduce__(zelf: PyRef<Self>, vm: &VirtualMachine) -> PyTupleRef { + let cls = zelf.class().to_owned(); + let args = ( + zelf.value.clone(), + zelf.expression.clone(), + zelf.conversion.clone(), + zelf.format_spec.clone(), + ); + (cls, args.to_pyobject(vm)).into_pytuple(vm) + } +} + +impl Comparable for PyInterpolation { + fn cmp( + zelf: &Py<Self>, + other: &PyObject, + op: PyComparisonOp, + vm: &VirtualMachine, + ) -> PyResult<PyComparisonValue> { + op.eq_only(|| { + let other = class_or_notimplemented!(Self, other); + + let eq = vm.bool_eq(&zelf.value, &other.value)? + && vm.bool_eq(zelf.expression.as_object(), other.expression.as_object())? + && vm.bool_eq(&zelf.conversion, &other.conversion)? + && vm.bool_eq(zelf.format_spec.as_object(), other.format_spec.as_object())?; + + Ok(eq.into()) + }) + } +} + +impl Hashable for PyInterpolation { + fn hash(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<PyHash> { + // Hash based on (value, expression, conversion, format_spec) + let value_hash = zelf.value.hash(vm)?; + let expr_hash = zelf.expression.as_object().hash(vm)?; + let conv_hash = zelf.conversion.hash(vm)?; + let spec_hash = zelf.format_spec.as_object().hash(vm)?; + + // Combine hashes + Ok(value_hash + .wrapping_add(expr_hash.wrapping_mul(3)) + .wrapping_add(conv_hash.wrapping_mul(5)) + .wrapping_add(spec_hash.wrapping_mul(7))) + } +} + +impl Representable for PyInterpolation { + #[inline] + fn repr_str(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<String> { + let value_repr = zelf.value.repr(vm)?; + let expr_repr = zelf.expression.repr(vm)?; + + let conv_str = if vm.is_none(&zelf.conversion) { + "None".to_owned() + } else { + zelf.conversion.repr(vm)?.as_str().to_owned() + }; + + let spec_repr = zelf.format_spec.repr(vm)?; + + Ok(format!( + "Interpolation({}, {}, {}, {})", + value_repr.as_str(), + expr_repr.as_str(), + conv_str, + spec_repr.as_str() + )) + } +} + +pub fn init(context: &Context) { + PyInterpolation::extend_class(context, context.types.interpolation_type); +} diff --git a/crates/vm/src/builtins/list.rs b/crates/vm/src/builtins/list.rs index 02475ee12b6..9c961d2929d 100644 --- a/crates/vm/src/builtins/list.rs +++ b/crates/vm/src/builtins/list.rs @@ -3,6 +3,7 @@ use crate::atomic_func; use crate::common::lock::{ PyMappedRwLockReadGuard, PyMutex, PyRwLock, PyRwLockReadGuard, PyRwLockWriteGuard, }; +use crate::object::{Traverse, TraverseFn}; use crate::{ AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, class::PyClassImpl, @@ -23,7 +24,7 @@ use crate::{ use alloc::fmt; use core::ops::DerefMut; -#[pyclass(module = false, name = "list", unhashable = true, traverse)] +#[pyclass(module = false, name = "list", unhashable = true, traverse = "manual")] #[derive(Default)] pub struct PyList { elements: PyRwLock<Vec<PyObjectRef>>, @@ -50,6 +51,22 @@ impl FromIterator<PyObjectRef> for PyList { } } +// SAFETY: Traverse properly visits all owned PyObjectRefs +unsafe impl Traverse for PyList { + fn traverse(&self, traverse_fn: &mut TraverseFn<'_>) { + self.elements.traverse(traverse_fn); + } + + fn clear(&mut self, out: &mut Vec<PyObjectRef>) { + // During GC, we use interior mutability to access elements. + // This is safe because during GC collection, the object is unreachable + // and no other code should be accessing it. + if let Some(mut guard) = self.elements.try_write() { + out.extend(guard.drain(..)); + } + } +} + impl PyPayload for PyList { #[inline] fn class(ctx: &Context) -> &'static Py<PyType> { @@ -505,12 +522,17 @@ fn do_sort( key_func: Option<PyObjectRef>, reverse: bool, ) -> PyResult<()> { - let op = if reverse { - PyComparisonOp::Lt - } else { - PyComparisonOp::Gt + // CPython uses __lt__ for all comparisons in sort. + // try_sort_by_gt expects is_gt(a, b) = true when a should come AFTER b. + let cmp = |a: &PyObjectRef, b: &PyObjectRef| { + if reverse { + // Descending: a comes after b when a < b + a.rich_compare_bool(b, PyComparisonOp::Lt, vm) + } else { + // Ascending: a comes after b when b < a + b.rich_compare_bool(a, PyComparisonOp::Lt, vm) + } }; - let cmp = |a: &PyObjectRef, b: &PyObjectRef| a.rich_compare_bool(b, op, vm); if let Some(ref key_func) = key_func { let mut items = values diff --git a/crates/vm/src/builtins/mappingproxy.rs b/crates/vm/src/builtins/mappingproxy.rs index f7fb64fa6ab..1036dcfdaf9 100644 --- a/crates/vm/src/builtins/mappingproxy.rs +++ b/crates/vm/src/builtins/mappingproxy.rs @@ -3,13 +3,14 @@ use crate::{ AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, atomic_func, class::PyClassImpl, + common::hash, convert::ToPyObject, function::{ArgMapping, OptionalArg, PyComparisonValue}, object::{Traverse, TraverseFn}, protocol::{PyMappingMethods, PyNumberMethods, PySequenceMethods}, types::{ - AsMapping, AsNumber, AsSequence, Comparable, Constructor, Iterable, PyComparisonOp, - Representable, + AsMapping, AsNumber, AsSequence, Comparable, Constructor, Hashable, Iterable, + PyComparisonOp, Representable, }, }; use std::sync::LazyLock; @@ -83,6 +84,7 @@ impl Constructor for PyMappingProxy { Constructor, AsSequence, Comparable, + Hashable, AsNumber, Representable ))] @@ -215,6 +217,15 @@ impl Comparable for PyMappingProxy { } } +impl Hashable for PyMappingProxy { + #[inline] + fn hash(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<hash::PyHash> { + // Delegate hash to the underlying mapping + let obj = zelf.to_object(vm)?; + obj.hash(vm) + } +} + impl AsMapping for PyMappingProxy { fn as_mapping() -> &'static PyMappingMethods { static AS_MAPPING: LazyLock<PyMappingMethods> = LazyLock::new(|| PyMappingMethods { diff --git a/crates/vm/src/builtins/mod.rs b/crates/vm/src/builtins/mod.rs index 6f379a4babf..994a4f9c758 100644 --- a/crates/vm/src/builtins/mod.rs +++ b/crates/vm/src/builtins/mod.rs @@ -36,6 +36,8 @@ pub(crate) mod getset; pub use getset::PyGetSet; pub(crate) mod int; pub use int::{PyInt, PyIntRef}; +pub(crate) mod interpolation; +pub use interpolation::PyInterpolation; pub(crate) mod iter; pub use iter::*; pub(crate) mod list; @@ -47,7 +49,7 @@ pub use mappingproxy::PyMappingProxy; pub(crate) mod memory; pub use memory::PyMemoryView; pub(crate) mod module; -pub use module::{PyModule, PyModuleDef}; +pub use module::{PyModule, PyModuleDef, PyModuleSlots}; pub(crate) mod namespace; pub use namespace::PyNamespace; pub(crate) mod object; @@ -76,6 +78,8 @@ pub(crate) mod slice; pub use slice::{PyEllipsis, PySlice}; pub(crate) mod staticmethod; pub use staticmethod::PyStaticMethod; +pub(crate) mod template; +pub use template::{PyTemplate, PyTemplateIter}; pub(crate) mod traceback; pub use traceback::PyTraceback; pub(crate) mod tuple; @@ -88,7 +92,7 @@ pub(crate) mod zip; pub use zip::PyZip; #[path = "union.rs"] pub(crate) mod union_; -pub use union_::PyUnion; +pub use union_::{PyUnion, make_union}; pub(crate) mod descriptor; pub use float::try_to_bigint as try_f64_to_bigint; diff --git a/crates/vm/src/builtins/module.rs b/crates/vm/src/builtins/module.rs index faa6e4813fd..60ffb1487fa 100644 --- a/crates/vm/src/builtins/module.rs +++ b/crates/vm/src/builtins/module.rs @@ -4,7 +4,7 @@ use crate::{ builtins::{PyStrInterned, pystr::AsPyStr}, class::PyClassImpl, convert::ToPyObject, - function::{FuncArgs, PyMethodDef}, + function::{FuncArgs, PyMethodDef, PySetterValue}, types::{GetAttr, Initializer, Representable}, }; @@ -41,6 +41,46 @@ impl core::fmt::Debug for PyModuleSlots { } } +impl PyModuleDef { + /// Create a module from this definition (Phase 1 of multi-phase init). + /// + /// This performs: + /// 1. Create module object (using create slot if provided) + /// 2. Initialize module dict from def + /// 3. Add methods to module + /// + /// Does NOT add to sys.modules or call exec slot. + pub fn create_module(&'static self, vm: &VirtualMachine) -> PyResult<PyRef<PyModule>> { + use crate::PyPayload; + + // Create module (use create slot if provided, else default creation) + let module = if let Some(create) = self.slots.create { + // Custom module creation + let spec = vm.ctx.new_str(self.name.as_str()); + create(vm, spec.as_object(), self)? + } else { + // Default module creation + PyModule::from_def(self).into_ref(&vm.ctx) + }; + + // Initialize module dict and methods + PyModule::__init_dict_from_def(vm, &module); + module.__init_methods(vm)?; + + Ok(module) + } + + /// Execute the module's exec slot (Phase 2 of multi-phase init). + /// + /// Calls the exec slot if present. Returns Ok(()) if no exec slot. + pub fn exec_module(&'static self, vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { + if let Some(exec) = self.slots.exec { + exec(vm, module)?; + } + Ok(()) + } +} + #[allow(clippy::new_without_default)] // avoid Default implementation #[pyclass(module = false, name = "module")] #[derive(Debug)] @@ -182,6 +222,116 @@ impl PyModule { let attrs = dict.into_iter().map(|(k, _v)| k).collect(); Ok(attrs) } + + #[pygetset] + fn __annotate__(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<PyObjectRef> { + let dict = zelf.dict(); + // Get __annotate__ from dict; if not present, insert None and return it + // See: module_get_annotate() + if let Some(annotate) = dict.get_item_opt(identifier!(vm, __annotate__), vm)? { + Ok(annotate) + } else { + let none = vm.ctx.none(); + dict.set_item(identifier!(vm, __annotate__), none.clone(), vm)?; + Ok(none) + } + } + + #[pygetset(setter)] + fn set___annotate__( + zelf: &Py<Self>, + value: PySetterValue, + vm: &VirtualMachine, + ) -> PyResult<()> { + match value { + PySetterValue::Assign(value) => { + if !vm.is_none(&value) && !value.is_callable() { + return Err(vm.new_type_error("__annotate__ must be callable or None")); + } + let dict = zelf.dict(); + dict.set_item(identifier!(vm, __annotate__), value.clone(), vm)?; + // Clear __annotations__ if value is not None + if !vm.is_none(&value) { + dict.del_item(identifier!(vm, __annotations__), vm).ok(); + } + Ok(()) + } + PySetterValue::Delete => Err(vm.new_type_error("cannot delete __annotate__ attribute")), + } + } + + #[pygetset] + fn __annotations__(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<PyObjectRef> { + let dict = zelf.dict(); + + // Check if __annotations__ is already in dict (explicitly set) + if let Some(annotations) = dict.get_item_opt(identifier!(vm, __annotations__), vm)? { + return Ok(annotations); + } + + // Check if module is initializing + let is_initializing = Self::is_initializing(&dict, vm); + + // PEP 649: Get __annotate__ and call it if callable + let annotations = if let Some(annotate) = + dict.get_item_opt(identifier!(vm, __annotate__), vm)? + && annotate.is_callable() + { + // Call __annotate__(1) where 1 is FORMAT_VALUE + let result = annotate.call((1i32,), vm)?; + if !result.class().is(vm.ctx.types.dict_type) { + return Err(vm.new_type_error(format!( + "__annotate__ returned non-dict of type '{}'", + result.class().name() + ))); + } + result + } else { + vm.ctx.new_dict().into() + }; + + // Cache result unless module is initializing + if !is_initializing { + dict.set_item(identifier!(vm, __annotations__), annotations.clone(), vm)?; + } + + Ok(annotations) + } + + /// Check if module is initializing via __spec__._initializing + fn is_initializing(dict: &PyDictRef, vm: &VirtualMachine) -> bool { + if let Ok(Some(spec)) = dict.get_item_opt(vm.ctx.intern_str("__spec__"), vm) + && let Ok(initializing) = spec.get_attr(vm.ctx.intern_str("_initializing"), vm) + { + return initializing.try_to_bool(vm).unwrap_or(false); + } + false + } + + #[pygetset(setter)] + fn set___annotations__( + zelf: &Py<Self>, + value: PySetterValue, + vm: &VirtualMachine, + ) -> PyResult<()> { + let dict = zelf.dict(); + match value { + PySetterValue::Assign(value) => { + dict.set_item(identifier!(vm, __annotations__), value, vm)?; + // Clear __annotate__ from dict + dict.del_item(identifier!(vm, __annotate__), vm).ok(); + Ok(()) + } + PySetterValue::Delete => { + if dict.del_item(identifier!(vm, __annotations__), vm).is_err() { + return Err(vm.new_attribute_error("__annotations__".to_owned())); + } + // Also clear __annotate__ + dict.del_item(identifier!(vm, __annotate__), vm).ok(); + Ok(()) + } + } + } } impl Initializer for PyModule { diff --git a/crates/vm/src/builtins/namespace.rs b/crates/vm/src/builtins/namespace.rs index 03969c35e7b..2cc1693302a 100644 --- a/crates/vm/src/builtins/namespace.rs +++ b/crates/vm/src/builtins/namespace.rs @@ -42,15 +42,76 @@ impl PyNamespace { ); result.into_pytuple(vm) } + + #[pymethod] + fn __replace__(zelf: PyObjectRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult { + if !args.args.is_empty() { + return Err(vm.new_type_error("__replace__() takes no positional arguments")); + } + + // Create a new instance of the same type + let cls: PyObjectRef = zelf.class().to_owned().into(); + let result = cls.call((), vm)?; + + // Copy the current namespace dict to the new instance + let src_dict = zelf.dict().unwrap(); + let dst_dict = result.dict().unwrap(); + for (key, value) in src_dict { + dst_dict.set_item(&*key, value, vm)?; + } + + // Update with the provided kwargs + for (name, value) in args.kwargs { + let name = vm.ctx.new_str(name); + result.set_attr(&name, value, vm)?; + } + + Ok(result) + } } impl Initializer for PyNamespace { type Args = FuncArgs; fn init(zelf: PyRef<Self>, args: Self::Args, vm: &VirtualMachine) -> PyResult<()> { - if !args.args.is_empty() { - return Err(vm.new_type_error("no positional arguments expected")); + // SimpleNamespace accepts 0 or 1 positional argument (a mapping) + if args.args.len() > 1 { + return Err(vm.new_type_error(format!( + "{} expected at most 1 positional argument, got {}", + zelf.class().name(), + args.args.len() + ))); } + + // If there's a positional argument, treat it as a mapping + if let Some(mapping) = args.args.first() { + // Convert to dict if not already + let dict: PyRef<PyDict> = if let Some(d) = mapping.downcast_ref::<PyDict>() { + d.to_owned() + } else { + // Call dict() on the mapping + let dict_type: PyObjectRef = vm.ctx.types.dict_type.to_owned().into(); + dict_type + .call((mapping.clone(),), vm)? + .downcast() + .map_err(|_| vm.new_type_error("dict() did not return a dict"))? + }; + + // Validate keys are strings and set attributes + for (key, value) in dict.into_iter() { + let key_str = key + .downcast_ref::<crate::builtins::PyStr>() + .ok_or_else(|| { + vm.new_type_error(format!( + "keywords must be strings, not '{}'", + key.class().name() + )) + })?; + zelf.as_object().set_attr(key_str, value, vm)?; + } + } + + // Apply keyword arguments (these override positional mapping values) for (name, value) in args.kwargs { let name = vm.ctx.new_str(name); zelf.as_object().set_attr(&name, value, vm)?; diff --git a/crates/vm/src/builtins/object.rs b/crates/vm/src/builtins/object.rs index 6f072542547..eb9d226acb4 100644 --- a/crates/vm/src/builtins/object.rs +++ b/crates/vm/src/builtins/object.rs @@ -184,15 +184,12 @@ fn type_slot_names(typ: &Py<PyType>, vm: &VirtualMachine) -> PyResult<Option<sup Ok(result) } -// object_getstate_default in CPython +// object_getstate_default fn object_getstate_default(obj: &PyObject, required: bool, vm: &VirtualMachine) -> PyResult { - // TODO: itemsize - // if required && obj.class().slots.itemsize > 0 { - // return vm.new_type_error(format!( - // "cannot pickle {:.200} objects", - // obj.class().name() - // )); - // } + // Check itemsize + if required && obj.class().slots.itemsize > 0 { + return Err(vm.new_type_error(format!("cannot pickle {:.200} objects", obj.class().name()))); + } let state = if obj.dict().is_none_or(|d| d.is_empty()) { vm.ctx.none() @@ -208,22 +205,36 @@ fn object_getstate_default(obj: &PyObject, required: bool, vm: &VirtualMachine) type_slot_names(obj.class(), vm).map_err(|_| vm.new_type_error("cannot pickle object"))?; if required { - let mut basicsize = obj.class().slots.basicsize; - // if obj.class().slots.dict_offset > 0 - // && !obj.class().slots.flags.has_feature(PyTypeFlags::MANAGED_DICT) - // { - // basicsize += std::mem::size_of::<PyObjectRef>(); - // } - // if obj.class().slots.weaklist_offset > 0 { - // basicsize += std::mem::size_of::<PyObjectRef>(); - // } + // Start with PyBaseObject_Type's basicsize + let mut basicsize = vm.ctx.types.object_type.slots.basicsize; + + // Add __dict__ size if type has dict + if obj.class().slots.flags.has_feature(PyTypeFlags::HAS_DICT) { + basicsize += core::mem::size_of::<PyObjectRef>(); + } + + // Add __weakref__ size if type has weakref support + let has_weakref = if let Some(ref ext) = obj.class().heaptype_ext { + match &ext.slots { + None => true, // Heap type without __slots__ has automatic weakref + Some(slots) => slots.iter().any(|s| s.as_str() == "__weakref__"), + } + } else { + let weakref_name = vm.ctx.intern_str("__weakref__"); + obj.class().attributes.read().contains_key(weakref_name) + }; + if has_weakref { + basicsize += core::mem::size_of::<PyObjectRef>(); + } + + // Add slots size if let Some(ref slot_names) = slot_names { basicsize += core::mem::size_of::<PyObjectRef>() * slot_names.__len__(); } + + // Fail if actual type's basicsize > expected basicsize if obj.class().slots.basicsize > basicsize { - return Err( - vm.new_type_error(format!("cannot pickle {:.200} object", obj.class().name())) - ); + return Err(vm.new_type_error(format!("cannot pickle '{}' object", obj.class().name()))); } } @@ -233,6 +244,12 @@ fn object_getstate_default(obj: &PyObject, required: bool, vm: &VirtualMachine) let slots = vm.ctx.new_dict(); for i in 0..slot_names_len { let borrowed_names = slot_names.borrow_vec(); + // Check if slotnames changed during iteration + if borrowed_names.len() != slot_names_len { + return Err(vm.new_runtime_error( + "__slotnames__ changed size during iteration".to_owned(), + )); + } let name = borrowed_names[i].downcast_ref::<PyStr>().unwrap(); let Ok(value) = obj.get_attr(name, vm) else { continue; @@ -249,7 +266,7 @@ fn object_getstate_default(obj: &PyObject, required: bool, vm: &VirtualMachine) Ok(state) } -// object_getstate in CPython +// object_getstate // fn object_getstate( // obj: &PyObject, // required: bool, @@ -550,11 +567,183 @@ pub fn init(ctx: &Context) { PyBaseObject::extend_class(ctx, ctx.types.object_type); } +/// Get arguments for __new__ from __getnewargs_ex__ or __getnewargs__ +/// Returns (args, kwargs) tuple where either can be None +fn get_new_arguments( + obj: &PyObject, + vm: &VirtualMachine, +) -> PyResult<(Option<super::PyTupleRef>, Option<super::PyDictRef>)> { + // First try __getnewargs_ex__ + if let Some(getnewargs_ex) = vm.get_special_method(obj, identifier!(vm, __getnewargs_ex__))? { + let newargs = getnewargs_ex.invoke((), vm)?; + + let newargs_tuple: PyRef<super::PyTuple> = newargs.downcast().map_err(|obj| { + vm.new_type_error(format!( + "__getnewargs_ex__ should return a tuple, not '{}'", + obj.class().name() + )) + })?; + + if newargs_tuple.len() != 2 { + return Err(vm.new_value_error(format!( + "__getnewargs_ex__ should return a tuple of length 2, not {}", + newargs_tuple.len() + ))); + } + + let args = newargs_tuple.as_slice()[0].clone(); + let kwargs = newargs_tuple.as_slice()[1].clone(); + + let args_tuple: PyRef<super::PyTuple> = args.downcast().map_err(|obj| { + vm.new_type_error(format!( + "first item of the tuple returned by __getnewargs_ex__ must be a tuple, not '{}'", + obj.class().name() + )) + })?; + + let kwargs_dict: PyRef<super::PyDict> = kwargs.downcast().map_err(|obj| { + vm.new_type_error(format!( + "second item of the tuple returned by __getnewargs_ex__ must be a dict, not '{}'", + obj.class().name() + )) + })?; + + return Ok((Some(args_tuple), Some(kwargs_dict))); + } + + // Fall back to __getnewargs__ + if let Some(getnewargs) = vm.get_special_method(obj, identifier!(vm, __getnewargs__))? { + let args = getnewargs.invoke((), vm)?; + + let args_tuple: PyRef<super::PyTuple> = args.downcast().map_err(|obj| { + vm.new_type_error(format!( + "__getnewargs__ should return a tuple, not '{}'", + obj.class().name() + )) + })?; + + return Ok((Some(args_tuple), None)); + } + + // No __getnewargs_ex__ or __getnewargs__ + Ok((None, None)) +} + +/// Check if __getstate__ is overridden by comparing with object.__getstate__ +fn is_getstate_overridden(obj: &PyObject, vm: &VirtualMachine) -> bool { + let obj_cls = obj.class(); + let object_type = vm.ctx.types.object_type; + + // If the class is object itself, not overridden + if obj_cls.is(object_type) { + return false; + } + + // Check if __getstate__ in the MRO comes from object or elsewhere + // If the type has its own __getstate__, it's overridden + if let Some(getstate) = obj_cls.get_attr(identifier!(vm, __getstate__)) + && let Some(obj_getstate) = object_type.get_attr(identifier!(vm, __getstate__)) + { + return !getstate.is(&obj_getstate); + } + false +} + +/// object_getstate - calls __getstate__ method or default implementation +fn object_getstate(obj: &PyObject, required: bool, vm: &VirtualMachine) -> PyResult { + // If __getstate__ is not overridden, use the default implementation with required flag + if !is_getstate_overridden(obj, vm) { + return object_getstate_default(obj, required, vm); + } + + // __getstate__ is overridden, call it without required + let getstate = obj.get_attr(identifier!(vm, __getstate__), vm)?; + getstate.call((), vm) +} + +/// Get list items iterator if obj is a list (or subclass), None iterator otherwise +fn get_items_iter(obj: &PyObjectRef, vm: &VirtualMachine) -> PyResult<(PyObjectRef, PyObjectRef)> { + let listitems: PyObjectRef = if obj.fast_isinstance(vm.ctx.types.list_type) { + obj.get_iter(vm)?.into() + } else { + vm.ctx.none() + }; + + let dictitems: PyObjectRef = if obj.fast_isinstance(vm.ctx.types.dict_type) { + let items = vm.call_method(obj, "items", ())?; + items.get_iter(vm)?.into() + } else { + vm.ctx.none() + }; + + Ok((listitems, dictitems)) +} + +/// reduce_newobj - creates reduce tuple for protocol >= 2 +fn reduce_newobj(obj: PyObjectRef, vm: &VirtualMachine) -> PyResult { + // Check if type has tp_new + let cls = obj.class(); + if cls.slots.new.load().is_none() { + return Err(vm.new_type_error(format!("cannot pickle '{}' object", cls.name()))); + } + + let (args, kwargs) = get_new_arguments(&obj, vm)?; + + let copyreg = vm.import("copyreg", 0)?; + + let has_args = args.is_some(); + + let (newobj, newargs): (PyObjectRef, PyObjectRef) = if kwargs.is_none() + || kwargs.as_ref().is_some_and(|k| k.is_empty()) + { + // Use copyreg.__newobj__ + let newobj = copyreg.get_attr("__newobj__", vm)?; + + let args_vec: Vec<PyObjectRef> = args.map(|a| a.as_slice().to_vec()).unwrap_or_default(); + + // Create (cls, *args) tuple + let mut newargs_vec: Vec<PyObjectRef> = vec![cls.to_owned().into()]; + newargs_vec.extend(args_vec); + let newargs = vm.ctx.new_tuple(newargs_vec); + + (newobj, newargs.into()) + } else { + // args == NULL with non-empty kwargs is BadInternalCall + let Some(args) = args else { + return Err(vm.new_system_error("bad internal call".to_owned())); + }; + // Use copyreg.__newobj_ex__ + let newobj = copyreg.get_attr("__newobj_ex__", vm)?; + let args_tuple: PyObjectRef = args.into(); + let kwargs_dict: PyObjectRef = kwargs + .map(|k| k.into()) + .unwrap_or_else(|| vm.ctx.new_dict().into()); + + let newargs = vm + .ctx + .new_tuple(vec![cls.to_owned().into(), args_tuple, kwargs_dict]); + (newobj, newargs.into()) + }; + + // Determine if state is required + // required = !(has_args || is_list || is_dict) + let is_list = obj.fast_isinstance(vm.ctx.types.list_type); + let is_dict = obj.fast_isinstance(vm.ctx.types.dict_type); + let required = !(has_args || is_list || is_dict); + + let state = object_getstate(&obj, required, vm)?; + + let (listitems, dictitems) = get_items_iter(&obj, vm)?; + + let result = vm + .ctx + .new_tuple(vec![newobj, newargs, state, listitems, dictitems]); + Ok(result.into()) +} + fn common_reduce(obj: PyObjectRef, proto: usize, vm: &VirtualMachine) -> PyResult { if proto >= 2 { - let reducelib = vm.import("__reducelib", 0)?; - let reduce_2 = reducelib.get_attr("reduce_2", vm)?; - reduce_2.call((obj,), vm) + reduce_newobj(obj, vm) } else { let copyreg = vm.import("copyreg", 0)?; let reduce_ex = copyreg.get_attr("_reduce_ex", vm)?; diff --git a/crates/vm/src/builtins/set.rs b/crates/vm/src/builtins/set.rs index 1c0268e2ed7..c3473809734 100644 --- a/crates/vm/src/builtins/set.rs +++ b/crates/vm/src/builtins/set.rs @@ -24,6 +24,7 @@ use crate::{ vm::VirtualMachine, }; use alloc::fmt; +use core::borrow::Borrow; use core::ops::Deref; use rustpython_common::{ atomic::{Ordering, PyAtomic, Radium}, @@ -719,8 +720,10 @@ impl PySet { } fn __iand__(zelf: PyRef<Self>, set: AnySet, vm: &VirtualMachine) -> PyResult<PyRef<Self>> { - zelf.inner - .intersection_update(core::iter::once(set.into_iterable(vm)?), vm)?; + if !set.is(zelf.as_object()) { + zelf.inner + .intersection_update(core::iter::once(set.into_iterable(vm)?), vm)?; + } Ok(zelf) } @@ -731,8 +734,12 @@ impl PySet { } fn __isub__(zelf: PyRef<Self>, set: AnySet, vm: &VirtualMachine) -> PyResult<PyRef<Self>> { - zelf.inner - .difference_update(set.into_iterable_iter(vm)?, vm)?; + if set.is(zelf.as_object()) { + zelf.inner.clear(); + } else { + zelf.inner + .difference_update(set.into_iterable_iter(vm)?, vm)?; + } Ok(zelf) } @@ -748,8 +755,12 @@ impl PySet { } fn __ixor__(zelf: PyRef<Self>, set: AnySet, vm: &VirtualMachine) -> PyResult<PyRef<Self>> { - zelf.inner - .symmetric_difference_update(set.into_iterable_iter(vm)?, vm)?; + if set.is(zelf.as_object()) { + zelf.inner.clear(); + } else { + zelf.inner + .symmetric_difference_update(set.into_iterable_iter(vm)?, vm)?; + } Ok(zelf) } @@ -1297,6 +1308,13 @@ struct AnySet { object: PyObjectRef, } +impl Borrow<PyObject> for AnySet { + #[inline(always)] + fn borrow(&self) -> &PyObject { + &self.object + } +} + impl AnySet { /// Check if object is a set or frozenset (including subclasses) /// Equivalent to CPython's PyAnySet_Check diff --git a/crates/vm/src/builtins/singletons.rs b/crates/vm/src/builtins/singletons.rs index 61ab1968a45..169104efeb3 100644 --- a/crates/vm/src/builtins/singletons.rs +++ b/crates/vm/src/builtins/singletons.rs @@ -108,11 +108,12 @@ impl PyNotImplemented { impl AsNumber for PyNotImplemented { fn as_number() -> &'static PyNumberMethods { - // TODO: As per https://bugs.python.org/issue35712, using NotImplemented - // in boolean contexts will need to raise a DeprecationWarning in 3.9 - // and, eventually, a TypeError. static AS_NUMBER: PyNumberMethods = PyNumberMethods { - boolean: Some(|_number, _vm| Ok(true)), + boolean: Some(|_number, vm| { + Err(vm.new_type_error( + "NotImplemented should not be used in a boolean context".to_owned(), + )) + }), ..PyNumberMethods::NOT_IMPLEMENTED }; &AS_NUMBER diff --git a/crates/vm/src/builtins/staticmethod.rs b/crates/vm/src/builtins/staticmethod.rs index 5d2474a567c..ac363415a9f 100644 --- a/crates/vm/src/builtins/staticmethod.rs +++ b/crates/vm/src/builtins/staticmethod.rs @@ -3,7 +3,7 @@ use crate::{ Context, Py, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, class::PyClassImpl, common::lock::PyMutex, - function::FuncArgs, + function::{FuncArgs, PySetterValue}, types::{Callable, Constructor, GetDescriptor, Initializer, Representable}, }; @@ -121,6 +121,27 @@ impl PyStaticMethod { self.callable.lock().get_attr("__annotations__", vm) } + #[pygetset(setter)] + fn set___annotations__(&self, value: PySetterValue, vm: &VirtualMachine) -> PyResult<()> { + match value { + PySetterValue::Assign(v) => self.callable.lock().set_attr("__annotations__", v, vm), + PySetterValue::Delete => Ok(()), // Silently ignore delete like CPython + } + } + + #[pygetset] + fn __annotate__(&self, vm: &VirtualMachine) -> PyResult { + self.callable.lock().get_attr("__annotate__", vm) + } + + #[pygetset(setter)] + fn set___annotate__(&self, value: PySetterValue, vm: &VirtualMachine) -> PyResult<()> { + match value { + PySetterValue::Assign(v) => self.callable.lock().set_attr("__annotate__", v, vm), + PySetterValue::Delete => Ok(()), // Silently ignore delete like CPython + } + } + #[pygetset] fn __isabstractmethod__(&self, vm: &VirtualMachine) -> PyObjectRef { match vm.get_attribute_opt(self.callable.lock().clone(), "__isabstractmethod__") { diff --git a/crates/vm/src/builtins/str.rs b/crates/vm/src/builtins/str.rs index 4931a748198..e8ba6eb915e 100644 --- a/crates/vm/src/builtins/str.rs +++ b/crates/vm/src/builtins/str.rs @@ -1924,9 +1924,16 @@ impl fmt::Display for PyUtf8Str { } impl MaybeTraverse for PyUtf8Str { + const HAS_TRAVERSE: bool = true; + const HAS_CLEAR: bool = false; + fn try_traverse(&self, traverse_fn: &mut TraverseFn<'_>) { self.0.try_traverse(traverse_fn); } + + fn try_clear(&mut self, _out: &mut Vec<PyObjectRef>) { + // No clear needed for PyUtf8Str + } } impl PyPayload for PyUtf8Str { @@ -1935,11 +1942,9 @@ impl PyPayload for PyUtf8Str { ctx.types.str_type } - fn payload_type_id() -> core::any::TypeId { - core::any::TypeId::of::<PyStr>() - } + const PAYLOAD_TYPE_ID: core::any::TypeId = core::any::TypeId::of::<PyStr>(); - fn validate_downcastable_from(obj: &PyObject) -> bool { + unsafe fn validate_downcastable_from(obj: &PyObject) -> bool { // SAFETY: we know the object is a PyStr in this context let wtf8 = unsafe { obj.downcast_unchecked_ref::<PyStr>() }; wtf8.is_utf8() diff --git a/crates/vm/src/builtins/super.rs b/crates/vm/src/builtins/super.rs index 893509bc6d3..b7bc3004332 100644 --- a/crates/vm/src/builtins/super.rs +++ b/crates/vm/src/builtins/super.rs @@ -60,8 +60,8 @@ impl Constructor for PySuper { #[derive(FromArgs)] pub struct InitArgs { - #[pyarg(positional, optional)] - py_type: OptionalArg<PyObjectRef>, + #[pyarg(positional, optional, error_msg = "super() argument 1 must be a type")] + py_type: OptionalArg<PyTypeRef>, #[pyarg(positional, optional)] py_obj: OptionalArg<PyObjectRef>, } @@ -75,10 +75,7 @@ impl Initializer for PySuper { vm: &VirtualMachine, ) -> PyResult<()> { // Get the type: - let (typ, obj) = if let OptionalArg::Present(ty_obj) = py_type { - let ty = ty_obj - .downcast::<PyType>() - .map_err(|_| vm.new_type_error("super() argument 1 must be a type"))?; + let (typ, obj) = if let OptionalArg::Present(ty) = py_type { (ty, py_obj.unwrap_or_none(vm)) } else { let frame = vm diff --git a/crates/vm/src/builtins/template.rs b/crates/vm/src/builtins/template.rs new file mode 100644 index 00000000000..2433c9082e5 --- /dev/null +++ b/crates/vm/src/builtins/template.rs @@ -0,0 +1,338 @@ +use super::{PyStr, PyTupleRef, PyType}; +use crate::{ + AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, + atomic_func, + class::PyClassImpl, + function::{FuncArgs, PyComparisonValue}, + protocol::{PyIterReturn, PySequenceMethods}, + types::{ + AsSequence, Comparable, Constructor, IterNext, Iterable, PyComparisonOp, Representable, + SelfIter, + }, +}; +use std::sync::LazyLock; + +use super::interpolation::PyInterpolation; + +/// Template object for t-strings (PEP 750). +/// +/// Represents a template string with interpolated expressions. +#[pyclass(module = "string.templatelib", name = "Template")] +#[derive(Debug, Clone)] +pub struct PyTemplate { + pub strings: PyTupleRef, + pub interpolations: PyTupleRef, +} + +impl PyPayload for PyTemplate { + #[inline] + fn class(ctx: &Context) -> &'static Py<PyType> { + ctx.types.template_type + } +} + +impl PyTemplate { + pub fn new(strings: PyTupleRef, interpolations: PyTupleRef) -> Self { + Self { + strings, + interpolations, + } + } +} + +impl Constructor for PyTemplate { + type Args = FuncArgs; + + fn py_new(_cls: &Py<PyType>, args: Self::Args, vm: &VirtualMachine) -> PyResult<Self> { + if !args.kwargs.is_empty() { + return Err(vm.new_type_error("Template.__new__ only accepts *args arguments")); + } + + let mut strings: Vec<PyObjectRef> = Vec::new(); + let mut interpolations: Vec<PyObjectRef> = Vec::new(); + let mut last_was_str = false; + + for item in args.args.iter() { + if let Ok(s) = item.clone().downcast::<PyStr>() { + if last_was_str { + // Concatenate adjacent strings + if let Some(last) = strings.last_mut() { + let last_str = last.downcast_ref::<PyStr>().unwrap(); + let concatenated = format!("{}{}", last_str.as_str(), s.as_str()); + *last = vm.ctx.new_str(concatenated).into(); + } + } else { + strings.push(s.into()); + } + last_was_str = true; + } else if item.class().is(vm.ctx.types.interpolation_type) { + if !last_was_str { + // Add empty string before interpolation + strings.push(vm.ctx.empty_str.to_owned().into()); + } + interpolations.push(item.clone()); + last_was_str = false; + } else { + return Err(vm.new_type_error(format!( + "Template.__new__ *args need to be of type 'str' or 'Interpolation', got {}", + item.class().name() + ))); + } + } + + if !last_was_str { + // Add trailing empty string + strings.push(vm.ctx.empty_str.to_owned().into()); + } + + Ok(PyTemplate { + strings: vm.ctx.new_tuple(strings), + interpolations: vm.ctx.new_tuple(interpolations), + }) + } +} + +#[pyclass(with(Constructor, Comparable, Iterable, Representable, AsSequence))] +impl PyTemplate { + #[pygetset] + fn strings(&self) -> PyTupleRef { + self.strings.clone() + } + + #[pygetset] + fn interpolations(&self) -> PyTupleRef { + self.interpolations.clone() + } + + #[pygetset] + fn values(&self, vm: &VirtualMachine) -> PyTupleRef { + let values: Vec<PyObjectRef> = self + .interpolations + .iter() + .map(|interp| { + interp + .downcast_ref::<PyInterpolation>() + .map(|i| i.value.clone()) + .unwrap_or_else(|| interp.clone()) + }) + .collect(); + vm.ctx.new_tuple(values) + } + + fn concat(&self, other: &PyObject, vm: &VirtualMachine) -> PyResult<PyRef<Self>> { + let other = other.downcast_ref::<PyTemplate>().ok_or_else(|| { + vm.new_type_error(format!( + "can only concatenate Template (not '{}') to Template", + other.class().name() + )) + })?; + + // Concatenate the two templates + let mut new_strings: Vec<PyObjectRef> = Vec::new(); + let mut new_interps: Vec<PyObjectRef> = Vec::new(); + + // Add all strings from self except the last one + let self_strings_len = self.strings.len(); + for i in 0..self_strings_len.saturating_sub(1) { + new_strings.push(self.strings.get(i).unwrap().clone()); + } + + // Add all interpolations from self + for interp in self.interpolations.iter() { + new_interps.push(interp.clone()); + } + + // Concatenate last string of self with first string of other + let last_self = self + .strings + .get(self_strings_len.saturating_sub(1)) + .and_then(|s| s.downcast_ref::<PyStr>().map(|s| s.as_str().to_owned())) + .unwrap_or_default(); + let first_other = other + .strings + .first() + .and_then(|s| s.downcast_ref::<PyStr>().map(|s| s.as_str().to_owned())) + .unwrap_or_default(); + let concatenated = format!("{}{}", last_self, first_other); + new_strings.push(vm.ctx.new_str(concatenated).into()); + + // Add remaining strings from other (skip first) + for i in 1..other.strings.len() { + new_strings.push(other.strings.get(i).unwrap().clone()); + } + + // Add all interpolations from other + for interp in other.interpolations.iter() { + new_interps.push(interp.clone()); + } + + let template = PyTemplate { + strings: vm.ctx.new_tuple(new_strings), + interpolations: vm.ctx.new_tuple(new_interps), + }; + + Ok(template.into_ref(&vm.ctx)) + } + + fn __add__(&self, other: PyObjectRef, vm: &VirtualMachine) -> PyResult<PyRef<Self>> { + self.concat(&other, vm) + } + + #[pymethod] + fn __reduce__(&self, vm: &VirtualMachine) -> PyResult<PyTupleRef> { + // Import string.templatelib._template_unpickle + // We need to import string first, then get templatelib from it, + // because import("string.templatelib", 0) with empty from_list returns the top-level module + let string_mod = vm.import("string.templatelib", 0)?; + let templatelib = string_mod.get_attr("templatelib", vm)?; + let unpickle_func = templatelib.get_attr("_template_unpickle", vm)?; + + // Return (func, (strings, interpolations)) + let args = vm.ctx.new_tuple(vec![ + self.strings.clone().into(), + self.interpolations.clone().into(), + ]); + Ok(vm.ctx.new_tuple(vec![unpickle_func, args.into()])) + } +} + +impl AsSequence for PyTemplate { + fn as_sequence() -> &'static PySequenceMethods { + static AS_SEQUENCE: LazyLock<PySequenceMethods> = LazyLock::new(|| PySequenceMethods { + concat: atomic_func!(|seq, other, vm| { + let zelf = PyTemplate::sequence_downcast(seq); + zelf.concat(other, vm).map(|t| t.into()) + }), + ..PySequenceMethods::NOT_IMPLEMENTED + }); + &AS_SEQUENCE + } +} + +impl Comparable for PyTemplate { + fn cmp( + zelf: &Py<Self>, + other: &PyObject, + op: PyComparisonOp, + vm: &VirtualMachine, + ) -> PyResult<PyComparisonValue> { + op.eq_only(|| { + let other = class_or_notimplemented!(Self, other); + + let eq = vm.bool_eq(zelf.strings.as_object(), other.strings.as_object())? + && vm.bool_eq( + zelf.interpolations.as_object(), + other.interpolations.as_object(), + )?; + + Ok(eq.into()) + }) + } +} + +impl Iterable for PyTemplate { + fn iter(zelf: PyRef<Self>, vm: &VirtualMachine) -> PyResult { + Ok(PyTemplateIter::new(zelf).into_pyobject(vm)) + } +} + +impl Representable for PyTemplate { + #[inline] + fn repr_str(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<String> { + let mut parts = Vec::new(); + + let strings_len = zelf.strings.len(); + let interps_len = zelf.interpolations.len(); + + for i in 0..strings_len.max(interps_len * 2 + 1) { + if i % 2 == 0 { + // String position + let idx = i / 2; + if idx < strings_len { + let s = zelf.strings.get(idx).unwrap(); + parts.push(s.repr(vm)?.as_str().to_owned()); + } + } else { + // Interpolation position + let idx = i / 2; + if idx < interps_len { + let interp = zelf.interpolations.get(idx).unwrap(); + parts.push(interp.repr(vm)?.as_str().to_owned()); + } + } + } + + Ok(format!("Template({})", parts.join(", "))) + } +} + +/// Iterator for Template objects +#[pyclass(module = "string.templatelib", name = "TemplateIter")] +#[derive(Debug)] +pub struct PyTemplateIter { + template: PyRef<PyTemplate>, + index: std::sync::atomic::AtomicUsize, + from_strings: std::sync::atomic::AtomicBool, +} + +impl PyPayload for PyTemplateIter { + #[inline] + fn class(ctx: &Context) -> &'static Py<PyType> { + ctx.types.template_iter_type + } +} + +impl PyTemplateIter { + fn new(template: PyRef<PyTemplate>) -> Self { + Self { + template, + index: std::sync::atomic::AtomicUsize::new(0), + from_strings: std::sync::atomic::AtomicBool::new(true), + } + } +} + +#[pyclass(with(IterNext, Iterable))] +impl PyTemplateIter {} + +impl SelfIter for PyTemplateIter {} + +impl IterNext for PyTemplateIter { + fn next(zelf: &Py<Self>, _vm: &VirtualMachine) -> PyResult<PyIterReturn> { + use std::sync::atomic::Ordering; + + loop { + let from_strings = zelf.from_strings.load(Ordering::SeqCst); + let index = zelf.index.load(Ordering::SeqCst); + + if from_strings { + if index < zelf.template.strings.len() { + let item = zelf.template.strings.get(index).unwrap(); + zelf.from_strings.store(false, Ordering::SeqCst); + + // Skip empty strings + if let Some(s) = item.downcast_ref::<PyStr>() + && s.as_str().is_empty() + { + continue; + } + return Ok(PyIterReturn::Return(item.clone())); + } else { + return Ok(PyIterReturn::StopIteration(None)); + } + } else if index < zelf.template.interpolations.len() { + let item = zelf.template.interpolations.get(index).unwrap(); + zelf.index.fetch_add(1, Ordering::SeqCst); + zelf.from_strings.store(true, Ordering::SeqCst); + return Ok(PyIterReturn::Return(item.clone())); + } else { + return Ok(PyIterReturn::StopIteration(None)); + } + } + } +} + +pub fn init(context: &Context) { + PyTemplate::extend_class(context, context.types.template_type); + PyTemplateIter::extend_class(context, context.types.template_iter_type); +} diff --git a/crates/vm/src/builtins/tuple.rs b/crates/vm/src/builtins/tuple.rs index f6eff5b91e5..ba296686c73 100644 --- a/crates/vm/src/builtins/tuple.rs +++ b/crates/vm/src/builtins/tuple.rs @@ -3,6 +3,7 @@ use crate::common::{ hash::{PyHash, PyUHash}, lock::PyMutex, }; +use crate::object::{Traverse, TraverseFn}; use crate::{ AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, TryFromObject, atomic_func, @@ -24,7 +25,7 @@ use crate::{ use alloc::fmt; use std::sync::LazyLock; -#[pyclass(module = false, name = "tuple", traverse)] +#[pyclass(module = false, name = "tuple", traverse = "manual")] pub struct PyTuple<R = PyObjectRef> { elements: Box<[R]>, } @@ -36,6 +37,19 @@ impl<R> fmt::Debug for PyTuple<R> { } } +// SAFETY: Traverse properly visits all owned PyObjectRefs +// Note: Only impl for PyTuple<PyObjectRef> (the default) +unsafe impl Traverse for PyTuple { + fn traverse(&self, traverse_fn: &mut TraverseFn<'_>) { + self.elements.traverse(traverse_fn); + } + + fn clear(&mut self, out: &mut Vec<PyObjectRef>) { + let elements = std::mem::take(&mut self.elements); + out.extend(elements.into_vec()); + } +} + impl PyPayload for PyTuple { #[inline] fn class(ctx: &Context) -> &'static Py<PyType> { diff --git a/crates/vm/src/builtins/type.rs b/crates/vm/src/builtins/type.rs index 8660d1f2e27..5e984ff6f3d 100644 --- a/crates/vm/src/builtins/type.rs +++ b/crates/vm/src/builtins/type.rs @@ -1,6 +1,6 @@ use super::{ - PyClassMethod, PyDictRef, PyList, PyStr, PyStrInterned, PyStrRef, PyTupleRef, PyWeak, - mappingproxy::PyMappingProxy, object, union_, + PyClassMethod, PyDictRef, PyList, PyStaticMethod, PyStr, PyStrInterned, PyStrRef, PyTupleRef, + PyWeak, mappingproxy::PyMappingProxy, object, union_, }; use crate::{ AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, TryFromObject, @@ -20,7 +20,6 @@ use crate::{ borrow::BorrowedValue, lock::{PyRwLock, PyRwLockReadGuard}, }, - convert::ToPyResult, function::{FuncArgs, KwArgs, OptionalArg, PyMethodDef, PySetterValue}, object::{Traverse, TraverseFn}, protocol::{PyIterReturn, PyNumberMethods}, @@ -851,6 +850,60 @@ impl PyType { Ok(()) } + #[pygetset] + fn __annotate__(&self, vm: &VirtualMachine) -> PyResult<PyObjectRef> { + if !self.slots.flags.has_feature(PyTypeFlags::HEAPTYPE) { + return Err(vm.new_attribute_error(format!( + "type object '{}' has no attribute '__annotate__'", + self.name() + ))); + } + + let mut attrs = self.attributes.write(); + // First try __annotate__, in case that's been set explicitly + if let Some(annotate) = attrs.get(identifier!(vm, __annotate__)).cloned() { + return Ok(annotate); + } + // Then try __annotate_func__ + if let Some(annotate) = attrs.get(identifier!(vm, __annotate_func__)).cloned() { + // TODO: Apply descriptor tp_descr_get if needed + return Ok(annotate); + } + // Set __annotate_func__ = None and return None + let none = vm.ctx.none(); + attrs.insert(identifier!(vm, __annotate_func__), none.clone()); + Ok(none) + } + + #[pygetset(setter)] + fn set___annotate__(&self, value: PySetterValue, vm: &VirtualMachine) -> PyResult<()> { + let value = match value { + PySetterValue::Delete => { + return Err(vm.new_type_error("cannot delete __annotate__ attribute".to_owned())); + } + PySetterValue::Assign(v) => v, + }; + + if self.slots.flags.has_feature(PyTypeFlags::IMMUTABLETYPE) { + return Err(vm.new_type_error(format!( + "cannot set '__annotate__' attribute of immutable type '{}'", + self.name() + ))); + } + + if !vm.is_none(&value) && !value.is_callable() { + return Err(vm.new_type_error("__annotate__ must be callable or None".to_owned())); + } + + let mut attrs = self.attributes.write(); + // Store to __annotate_func__ + attrs.insert(identifier!(vm, __annotate_func__), value.clone()); + // Always clear cached annotations when __annotate__ is updated + attrs.swap_remove(identifier!(vm, __annotations_cache__)); + + Ok(()) + } + #[pygetset] fn __annotations__(&self, vm: &VirtualMachine) -> PyResult<PyObjectRef> { if !self.slots.flags.has_feature(PyTypeFlags::HEAPTYPE) { @@ -860,20 +913,37 @@ impl PyType { ))); } - let __annotations__ = identifier!(vm, __annotations__); - let annotations = self.attributes.read().get(__annotations__).cloned(); + // First try __annotations__ (e.g. for "from __future__ import annotations") + let attrs = self.attributes.read(); + if let Some(annotations) = attrs.get(identifier!(vm, __annotations__)).cloned() { + return Ok(annotations); + } + // Then try __annotations_cache__ + if let Some(annotations) = attrs.get(identifier!(vm, __annotations_cache__)).cloned() { + return Ok(annotations); + } + drop(attrs); - let annotations = if let Some(annotations) = annotations { - annotations + // Get __annotate__ and call it if callable + let annotate = self.__annotate__(vm)?; + let annotations = if annotate.is_callable() { + // Call __annotate__(1) where 1 is FORMAT_VALUE + let result = annotate.call((1i32,), vm)?; + if !result.class().is(vm.ctx.types.dict_type) { + return Err(vm.new_type_error(format!( + "__annotate__ returned non-dict of type '{}'", + result.class().name() + ))); + } + result } else { - let annotations: PyObjectRef = vm.ctx.new_dict().into(); - let removed = self - .attributes - .write() - .insert(__annotations__, annotations.clone()); - debug_assert!(removed.is_none()); - annotations + vm.ctx.new_dict().into() }; + + // Cache the result in __annotations_cache__ + self.attributes + .write() + .insert(identifier!(vm, __annotations_cache__), annotations.clone()); Ok(annotations) } @@ -886,21 +956,36 @@ impl PyType { ))); } - let __annotations__ = identifier!(vm, __annotations__); - if let Some(value) = value { - self.attributes.write().insert(__annotations__, value); + let mut attrs = self.attributes.write(); + // conditional update based on __annotations__ presence + let has_annotations = attrs.contains_key(identifier!(vm, __annotations__)); + + if has_annotations { + // If __annotations__ is in dict, update it + if let Some(value) = value { + attrs.insert(identifier!(vm, __annotations__), value); + } else if attrs + .swap_remove(identifier!(vm, __annotations__)) + .is_none() + { + return Err(vm.new_attribute_error("__annotations__".to_owned())); + } + // Also clear __annotations_cache__ + attrs.swap_remove(identifier!(vm, __annotations_cache__)); } else { - self.attributes - .read() - .get(__annotations__) - .cloned() - .ok_or_else(|| { - vm.new_attribute_error(format!( - "'{}' object has no attribute '__annotations__'", - self.name() - )) - })?; + // Otherwise update only __annotations_cache__ + if let Some(value) = value { + attrs.insert(identifier!(vm, __annotations_cache__), value); + } else if attrs + .swap_remove(identifier!(vm, __annotations_cache__)) + .is_none() + { + return Err(vm.new_attribute_error("__annotations__".to_owned())); + } } + // Always clear __annotate_func__ and __annotate__ + attrs.swap_remove(identifier!(vm, __annotate_func__)); + attrs.swap_remove(identifier!(vm, __annotate__)); Ok(()) } @@ -954,11 +1039,11 @@ impl PyType { ) } - pub fn __ror__(zelf: PyObjectRef, other: PyObjectRef, vm: &VirtualMachine) -> PyObjectRef { + pub fn __ror__(zelf: PyObjectRef, other: PyObjectRef, vm: &VirtualMachine) -> PyResult { or_(other, zelf, vm) } - pub fn __or__(zelf: PyObjectRef, other: PyObjectRef, vm: &VirtualMachine) -> PyObjectRef { + pub fn __or__(zelf: PyObjectRef, other: PyObjectRef, vm: &VirtualMachine) -> PyResult { or_(zelf, other, vm) } @@ -1166,6 +1251,12 @@ impl Constructor for PyType { *f = PyClassMethod::from(f.clone()).into_pyobject(vm); } + if let Some(f) = attributes.get_mut(identifier!(vm, __new__)) + && f.class().is(vm.ctx.types.function_type) + { + *f = PyStaticMethod::from(f.clone()).into_pyobject(vm); + } + if let Some(current_frame) = vm.current_frame() { let entry = attributes.entry(identifier!(vm, __module__)); if matches!(entry, Entry::Vacant(_)) { @@ -1463,15 +1554,17 @@ impl Constructor for PyType { }) .collect::<PyResult<Vec<_>>>()?; for (obj, name, set_name) in attributes { - set_name.call((typ.clone(), name), vm).map_err(|e| { - let err = vm.new_runtime_error(format!( + set_name.call((typ.clone(), name), vm).inspect_err(|e| { + // PEP 678: Add a note to the original exception instead of wrapping it + // (Python 3.12+, gh-77757) + let note = format!( "Error calling __set_name__ on '{}' instance {} in '{}'", obj.class().name(), name, typ.name() - )); - err.set___cause__(Some(e)); - err + ); + // Ignore result - adding a note is best-effort, the original exception is what matters + drop(vm.call_method(e.as_object(), "add_note", (vm.ctx.new_str(note.as_str()),))); })?; } @@ -1760,7 +1853,7 @@ impl Callable for PyType { impl AsNumber for PyType { fn as_number() -> &'static PyNumberMethods { static AS_NUMBER: PyNumberMethods = PyNumberMethods { - or: Some(|a, b, vm| or_(a.to_owned(), b.to_owned(), vm).to_pyresult(vm)), + or: Some(|a, b, vm| or_(a.to_owned(), b.to_owned(), vm)), ..PyNumberMethods::NOT_IMPLEMENTED }; &AS_NUMBER @@ -1923,9 +2016,9 @@ pub(crate) fn call_slot_new( slot_new(subtype, args, vm) } -pub(crate) fn or_(zelf: PyObjectRef, other: PyObjectRef, vm: &VirtualMachine) -> PyObjectRef { +pub(crate) fn or_(zelf: PyObjectRef, other: PyObjectRef, vm: &VirtualMachine) -> PyResult { if !union_::is_unionable(zelf.clone(), vm) || !union_::is_unionable(other.clone(), vm) { - return vm.ctx.not_implemented(); + return Ok(vm.ctx.not_implemented()); } let tuple = PyTuple::new_ref(vec![zelf, other], &vm.ctx); diff --git a/crates/vm/src/builtins/union.rs b/crates/vm/src/builtins/union.rs index b5e12dcb3c8..9856235ecf4 100644 --- a/crates/vm/src/builtins/union.rs +++ b/crates/vm/src/builtins/union.rs @@ -2,10 +2,10 @@ use super::{genericalias, type_}; use crate::{ AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, atomic_func, - builtins::{PyFrozenSet, PyGenericAlias, PyStr, PyTuple, PyTupleRef, PyType}, + builtins::{PyFrozenSet, PySet, PyStr, PyTuple, PyTupleRef, PyType}, class::PyClassImpl, common::hash, - convert::{ToPyObject, ToPyResult}, + convert::ToPyObject, function::PyComparisonValue, protocol::{PyMappingMethods, PyNumberMethods}, stdlib::typing::TypeAliasType, @@ -16,9 +16,13 @@ use std::sync::LazyLock; const CLS_ATTRS: &[&str] = &["__module__"]; -#[pyclass(module = "types", name = "UnionType", traverse)] +#[pyclass(module = "typing", name = "Union", traverse)] pub struct PyUnion { args: PyTupleRef, + /// Frozenset of hashable args, or None if all args were hashable + hashable_args: Option<PyRef<PyFrozenSet>>, + /// Tuple of initially unhashable args, or None if all args were hashable + unhashable_args: Option<PyTupleRef>, parameters: PyTupleRef, } @@ -36,9 +40,15 @@ impl PyPayload for PyUnion { } impl PyUnion { - pub fn new(args: PyTupleRef, vm: &VirtualMachine) -> Self { - let parameters = make_parameters(&args, vm); - Self { args, parameters } + /// Create a new union from dedup result (internal use) + fn from_components(result: UnionComponents, vm: &VirtualMachine) -> PyResult<Self> { + let parameters = make_parameters(&result.args, vm)?; + Ok(Self { + args: result.args, + hashable_args: result.hashable_args, + unhashable_args: result.unhashable_args, + parameters, + }) } /// Direct access to args field, matching CPython's _Py_union_args @@ -88,10 +98,25 @@ impl PyUnion { } #[pyclass( - flags(BASETYPE), + flags(DISALLOW_INSTANTIATION), with(Hashable, Comparable, AsMapping, AsNumber, Representable) )] impl PyUnion { + #[pygetset] + fn __name__(&self, vm: &VirtualMachine) -> PyObjectRef { + vm.ctx.new_str("Union").into() + } + + #[pygetset] + fn __qualname__(&self, vm: &VirtualMachine) -> PyObjectRef { + vm.ctx.new_str("Union").into() + } + + #[pygetset] + fn __origin__(&self, vm: &VirtualMachine) -> PyObjectRef { + vm.ctx.types.union_type.to_owned().into() + } + #[pygetset] fn __parameters__(&self) -> PyObjectRef { self.parameters.clone().into() @@ -136,17 +161,35 @@ impl PyUnion { } } - fn __or__(zelf: PyObjectRef, other: PyObjectRef, vm: &VirtualMachine) -> PyObjectRef { + fn __or__(zelf: PyObjectRef, other: PyObjectRef, vm: &VirtualMachine) -> PyResult { type_::or_(zelf, other, vm) } + #[pymethod] + fn __mro_entries__(zelf: PyRef<Self>, _args: PyObjectRef, vm: &VirtualMachine) -> PyResult { + Err(vm.new_type_error(format!("Cannot subclass {}", zelf.repr(vm)?))) + } + #[pyclassmethod] fn __class_getitem__( - cls: crate::builtins::PyTypeRef, + _cls: crate::builtins::PyTypeRef, args: PyObjectRef, vm: &VirtualMachine, - ) -> PyGenericAlias { - PyGenericAlias::from_args(cls, args, vm) + ) -> PyResult { + // Convert args to tuple if not already + let args_tuple = if let Some(tuple) = args.downcast_ref::<PyTuple>() { + tuple.to_owned() + } else { + PyTuple::new_ref(vec![args], &vm.ctx) + }; + + // Check for empty union + if args_tuple.is_empty() { + return Err(vm.new_type_error("Cannot create empty Union")); + } + + // Create union using make_union to properly handle None -> NoneType conversion + make_union(&args_tuple, vm) } } @@ -159,9 +202,10 @@ pub fn is_unionable(obj: PyObjectRef, vm: &VirtualMachine) -> bool { || obj.downcast_ref::<TypeAliasType>().is_some() } -fn make_parameters(args: &Py<PyTuple>, vm: &VirtualMachine) -> PyTupleRef { +fn make_parameters(args: &Py<PyTuple>, vm: &VirtualMachine) -> PyResult<PyTupleRef> { let parameters = genericalias::make_parameters(args, vm); - dedup_and_flatten_args(&parameters, vm) + let result = dedup_and_flatten_args(&parameters, vm)?; + Ok(result.args) } fn flatten_args(args: &Py<PyTuple>, vm: &VirtualMachine) -> PyTupleRef { @@ -180,6 +224,12 @@ fn flatten_args(args: &Py<PyTuple>, vm: &VirtualMachine) -> PyTupleRef { flattened_args.extend(pyref.args.iter().cloned()); } else if vm.is_none(arg) { flattened_args.push(vm.ctx.types.none_type.to_owned().into()); + } else if arg.downcast_ref::<PyStr>().is_some() { + // Convert string to ForwardRef + match string_to_forwardref(arg.clone(), vm) { + Ok(fr) => flattened_args.push(fr), + Err(_) => flattened_args.push(arg.clone()), + } } else { flattened_args.push(arg.clone()); }; @@ -188,31 +238,105 @@ fn flatten_args(args: &Py<PyTuple>, vm: &VirtualMachine) -> PyTupleRef { PyTuple::new_ref(flattened_args, &vm.ctx) } -fn dedup_and_flatten_args(args: &Py<PyTuple>, vm: &VirtualMachine) -> PyTupleRef { +fn string_to_forwardref(arg: PyObjectRef, vm: &VirtualMachine) -> PyResult { + // Import annotationlib.ForwardRef and create a ForwardRef + let annotationlib = vm.import("annotationlib", 0)?; + let forwardref_cls = annotationlib.get_attr("ForwardRef", vm)?; + forwardref_cls.call((arg,), vm) +} + +/// Components for creating a PyUnion after deduplication +struct UnionComponents { + /// All unique args in order + args: PyTupleRef, + /// Frozenset of hashable args (for fast equality comparison) + hashable_args: Option<PyRef<PyFrozenSet>>, + /// Tuple of unhashable args at creation time (for hash error message) + unhashable_args: Option<PyTupleRef>, +} + +fn dedup_and_flatten_args(args: &Py<PyTuple>, vm: &VirtualMachine) -> PyResult<UnionComponents> { let args = flatten_args(args, vm); + // Use set-based deduplication like CPython: + // - For hashable elements: use Python's set semantics (hash + equality) + // - For unhashable elements: use equality comparison + // + // This avoids calling __eq__ when hashes differ, matching CPython behavior + // where `int | BadType` doesn't raise even if BadType.__eq__ raises. + let mut new_args: Vec<PyObjectRef> = Vec::with_capacity(args.len()); + + // Track hashable elements using a Python set (uses hash + equality) + let hashable_set = PySet::default().into_ref(&vm.ctx); + let mut hashable_list: Vec<PyObjectRef> = Vec::new(); + let mut unhashable_list: Vec<PyObjectRef> = Vec::new(); + for arg in &*args { - if !new_args.iter().any(|param| { - param - .rich_compare_bool(arg, PyComparisonOp::Eq, vm) - .unwrap_or_default() - }) { - new_args.push(arg.clone()); + // Try to hash the element first + match arg.hash(vm) { + Ok(_) => { + // Element is hashable - use set for deduplication + // Set membership uses hash first, then equality only if hashes match + let contains = vm + .call_method(hashable_set.as_ref(), "__contains__", (arg.clone(),)) + .and_then(|r| r.try_to_bool(vm))?; + if !contains { + hashable_set.add(arg.clone(), vm)?; + hashable_list.push(arg.clone()); + new_args.push(arg.clone()); + } + } + Err(_) => { + // Element is unhashable - use equality comparison + let mut is_duplicate = false; + for existing in &unhashable_list { + match existing.rich_compare_bool(arg, PyComparisonOp::Eq, vm) { + Ok(true) => { + is_duplicate = true; + break; + } + Ok(false) => continue, + Err(e) => return Err(e), + } + } + if !is_duplicate { + unhashable_list.push(arg.clone()); + new_args.push(arg.clone()); + } + } } } new_args.shrink_to_fit(); - PyTuple::new_ref(new_args, &vm.ctx) + // Create hashable_args frozenset if there are hashable elements + let hashable_args = if !hashable_list.is_empty() { + Some(PyFrozenSet::from_iter(vm, hashable_list.into_iter())?.into_ref(&vm.ctx)) + } else { + None + }; + + // Create unhashable_args tuple if there are unhashable elements + let unhashable_args = if !unhashable_list.is_empty() { + Some(PyTuple::new_ref(unhashable_list, &vm.ctx)) + } else { + None + }; + + Ok(UnionComponents { + args: PyTuple::new_ref(new_args, &vm.ctx), + hashable_args, + unhashable_args, + }) } -pub fn make_union(args: &Py<PyTuple>, vm: &VirtualMachine) -> PyObjectRef { - let args = dedup_and_flatten_args(args, vm); - match args.len() { - 1 => args[0].to_owned(), - _ => PyUnion::new(args, vm).to_pyobject(vm), - } +pub fn make_union(args: &Py<PyTuple>, vm: &VirtualMachine) -> PyResult { + let result = dedup_and_flatten_args(args, vm)?; + Ok(match result.args.len() { + 1 => result.args[0].to_owned(), + _ => PyUnion::from_components(result, vm)?.to_pyobject(vm), + }) } impl PyUnion { @@ -224,14 +348,15 @@ impl PyUnion { needle, vm, )?; - let mut res; + let res; if new_args.is_empty() { - res = make_union(&new_args, vm); + res = make_union(&new_args, vm)?; } else { - res = new_args[0].to_owned(); + let mut tmp = new_args[0].to_owned(); for arg in new_args.iter().skip(1) { - res = vm._or(&res, arg)?; + tmp = vm._or(&tmp, arg)?; } + res = tmp; } Ok(res) @@ -254,7 +379,7 @@ impl AsMapping for PyUnion { impl AsNumber for PyUnion { fn as_number() -> &'static PyNumberMethods { static AS_NUMBER: PyNumberMethods = PyNumberMethods { - or: Some(|a, b, vm| PyUnion::__or__(a.to_owned(), b.to_owned(), vm).to_pyresult(vm)), + or: Some(|a, b, vm| PyUnion::__or__(a.to_owned(), b.to_owned(), vm)), ..PyNumberMethods::NOT_IMPLEMENTED }; &AS_NUMBER @@ -270,15 +395,62 @@ impl Comparable for PyUnion { ) -> PyResult<PyComparisonValue> { op.eq_only(|| { let other = class_or_notimplemented!(Self, other); - let a = PyFrozenSet::from_iter(vm, zelf.args.into_iter().cloned())?; - let b = PyFrozenSet::from_iter(vm, other.args.into_iter().cloned())?; - Ok(PyComparisonValue::Implemented( - a.into_pyobject(vm).as_object().rich_compare_bool( - b.into_pyobject(vm).as_object(), - PyComparisonOp::Eq, - vm, - )?, - )) + + // Check if lengths are equal + if zelf.args.len() != other.args.len() { + return Ok(PyComparisonValue::Implemented(false)); + } + + // Fast path: if both unions have all hashable args, compare frozensets directly + // Always use Eq here since eq_only handles Ne by negating the result + if zelf.unhashable_args.is_none() + && other.unhashable_args.is_none() + && let (Some(a), Some(b)) = (&zelf.hashable_args, &other.hashable_args) + { + let eq = a + .as_object() + .rich_compare_bool(b.as_object(), PyComparisonOp::Eq, vm)?; + return Ok(PyComparisonValue::Implemented(eq)); + } + + // Slow path: O(n^2) nested loop comparison for unhashable elements + // Check if all elements in zelf.args are in other.args + for arg_a in &*zelf.args { + let mut found = false; + for arg_b in &*other.args { + match arg_a.rich_compare_bool(arg_b, PyComparisonOp::Eq, vm) { + Ok(true) => { + found = true; + break; + } + Ok(false) => continue, + Err(e) => return Err(e), // Propagate comparison errors + } + } + if !found { + return Ok(PyComparisonValue::Implemented(false)); + } + } + + // Check if all elements in other.args are in zelf.args (for symmetry) + for arg_b in &*other.args { + let mut found = false; + for arg_a in &*zelf.args { + match arg_b.rich_compare_bool(arg_a, PyComparisonOp::Eq, vm) { + Ok(true) => { + found = true; + break; + } + Ok(false) => continue, + Err(e) => return Err(e), // Propagate comparison errors + } + } + if !found { + return Ok(PyComparisonValue::Implemented(false)); + } + } + + Ok(PyComparisonValue::Implemented(true)) }) } } @@ -286,7 +458,36 @@ impl Comparable for PyUnion { impl Hashable for PyUnion { #[inline] fn hash(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<hash::PyHash> { - let set = PyFrozenSet::from_iter(vm, zelf.args.into_iter().cloned())?; + // If there are any unhashable args from creation time, the union is unhashable + if let Some(ref unhashable_args) = zelf.unhashable_args { + let n = unhashable_args.len(); + // Try to hash each previously unhashable arg to get an error + for arg in unhashable_args.iter() { + arg.hash(vm)?; + } + // All previously unhashable args somehow became hashable + // But still raise an error to maintain consistent hashing + return Err(vm.new_type_error(format!( + "union contains {} unhashable element{}", + n, + if n > 1 { "s" } else { "" } + ))); + } + + // If we have a stored frozenset of hashable args, use that + if let Some(ref hashable_args) = zelf.hashable_args { + return PyFrozenSet::hash(hashable_args, vm); + } + + // Fallback: compute hash from args + let mut args_to_hash = Vec::new(); + for arg in &*zelf.args { + match arg.hash(vm) { + Ok(_) => args_to_hash.push(arg.clone()), + Err(e) => return Err(e), + } + } + let set = PyFrozenSet::from_iter(vm, args_to_hash.into_iter())?; PyFrozenSet::hash(&set.into_ref(&vm.ctx), vm) } } diff --git a/crates/vm/src/builtins/weakref.rs b/crates/vm/src/builtins/weakref.rs index 327c0fd1489..19f3f44f071 100644 --- a/crates/vm/src/builtins/weakref.rs +++ b/crates/vm/src/builtins/weakref.rs @@ -15,6 +15,7 @@ use crate::{ pub use crate::object::PyWeak; #[derive(FromArgs)] +#[allow(dead_code)] pub struct WeakNewArgs { #[pyarg(positional)] referent: PyObjectRef, @@ -41,8 +42,20 @@ impl Constructor for PyWeak { type Args = WeakNewArgs; fn slot_new(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult { - let Self::Args { referent, callback } = args.bind(vm)?; - let weak = referent.downgrade_with_typ(callback.into_option(), cls, vm)?; + // PyArg_UnpackTuple: only process positional args, ignore kwargs. + // Subclass __init__ will handle extra kwargs. + let mut positional = args.args.into_iter(); + let referent = positional.next().ok_or_else(|| { + vm.new_type_error("__new__ expected at least 1 argument, got 0".to_owned()) + })?; + let callback = positional.next(); + if let Some(_extra) = positional.next() { + return Err(vm.new_type_error(format!( + "__new__ expected at most 2 arguments, got {}", + 3 + positional.count() + ))); + } + let weak = referent.downgrade_with_typ(callback, cls, vm)?; Ok(weak.into()) } diff --git a/crates/vm/src/bytes_inner.rs b/crates/vm/src/bytes_inner.rs index bb5db442c35..7e1c1c2220c 100644 --- a/crates/vm/src/bytes_inner.rs +++ b/crates/vm/src/bytes_inner.rs @@ -423,12 +423,13 @@ impl PyBytesInner { pub fn fromhex(bytes: &[u8], vm: &VirtualMachine) -> PyResult<Vec<u8>> { let mut iter = bytes.iter().enumerate(); - let mut bytes: Vec<u8> = Vec::with_capacity(bytes.len() / 2); - let i = loop { + let mut result: Vec<u8> = Vec::with_capacity(bytes.len() / 2); + // None means odd number of hex digits, Some(i) means invalid char at position i + let invalid_char: Option<usize> = loop { let (i, &b) = match iter.next() { Some(val) => val, None => { - return Ok(bytes); + return Ok(result); } }; @@ -440,27 +441,49 @@ impl PyBytesInner { b'0'..=b'9' => b - b'0', b'a'..=b'f' => 10 + b - b'a', b'A'..=b'F' => 10 + b - b'A', - _ => break i, + _ => break Some(i), }; let (i, b) = match iter.next() { Some(val) => val, - None => break i + 1, + None => break None, // odd number of hex digits }; let bot = match b { b'0'..=b'9' => b - b'0', b'a'..=b'f' => 10 + b - b'a', b'A'..=b'F' => 10 + b - b'A', - _ => break i, + _ => break Some(i), }; - bytes.push((top << 4) + bot); + result.push((top << 4) + bot); }; - Err(vm.new_value_error(format!( - "non-hexadecimal number found in fromhex() arg at position {i}" - ))) + match invalid_char { + None => Err(vm.new_value_error( + "fromhex() arg must contain an even number of hexadecimal digits".to_owned(), + )), + Some(i) => Err(vm.new_value_error(format!( + "non-hexadecimal number found in fromhex() arg at position {i}" + ))), + } + } + + /// Parse hex string from str or bytes-like object + pub fn fromhex_object(string: PyObjectRef, vm: &VirtualMachine) -> PyResult<Vec<u8>> { + if let Some(s) = string.downcast_ref::<PyStr>() { + Self::fromhex(s.as_bytes(), vm) + } else if let Ok(buffer) = PyBuffer::try_from_borrowed_object(vm, &string) { + let borrowed = buffer.as_contiguous().ok_or_else(|| { + vm.new_buffer_error("fromhex() requires a contiguous buffer".to_owned()) + })?; + Self::fromhex(&borrowed, vm) + } else { + Err(vm.new_type_error(format!( + "fromhex() argument must be str or bytes-like, not {}", + string.class().name() + ))) + } } #[inline] diff --git a/crates/vm/src/class.rs b/crates/vm/src/class.rs index 4590b62d503..a71baf070cd 100644 --- a/crates/vm/src/class.rs +++ b/crates/vm/src/class.rs @@ -28,7 +28,7 @@ pub fn add_operators(class: &'static Py<PyType>, ctx: &Context) { .slots .hash .load() - .is_some_and(|h| h as usize == hash_not_implemented as usize) + .is_some_and(|h| h as usize == hash_not_implemented as *const () as usize) { class.set_attr(ctx.names.__hash__, ctx.none.clone().into()); continue; diff --git a/crates/vm/src/coroutine.rs b/crates/vm/src/coroutine.rs index 19830496e6e..67325283f3a 100644 --- a/crates/vm/src/coroutine.rs +++ b/crates/vm/src/coroutine.rs @@ -5,6 +5,7 @@ use crate::{ exceptions::types::PyBaseException, frame::{ExecutionResult, FrameRef}, function::OptionalArg, + object::{Traverse, TraverseFn}, protocol::PyIterReturn, }; use crossbeam_utils::atomic::AtomicCell; @@ -38,6 +39,15 @@ pub struct Coro { exception: PyMutex<Option<PyBaseExceptionRef>>, // exc_state } +unsafe impl Traverse for Coro { + fn traverse(&self, tracer_fn: &mut TraverseFn<'_>) { + self.frame.traverse(tracer_fn); + self.name.traverse(tracer_fn); + self.qualname.traverse(tracer_fn); + self.exception.traverse(tracer_fn); + } +} + fn gen_name(jen: &PyObject, vm: &VirtualMachine) -> &'static str { let typ = jen.class(); if typ.is(vm.ctx.types.coroutine_type) { @@ -236,6 +246,52 @@ pub fn is_gen_exit(exc: &Py<PyBaseException>, vm: &VirtualMachine) -> bool { exc.fast_isinstance(vm.ctx.exceptions.generator_exit) } +/// Get an awaitable iterator from an object. +/// +/// Returns the object itself if it's a coroutine or iterable coroutine (generator with +/// CO_ITERABLE_COROUTINE flag). Otherwise calls `__await__()` and validates the result. +pub fn get_awaitable_iter(obj: PyObjectRef, vm: &VirtualMachine) -> PyResult { + use crate::builtins::{PyCoroutine, PyGenerator}; + use crate::protocol::PyIter; + + if obj.downcastable::<PyCoroutine>() + || obj.downcast_ref::<PyGenerator>().is_some_and(|g| { + g.as_coro() + .frame() + .code + .flags + .contains(crate::bytecode::CodeFlags::ITERABLE_COROUTINE) + }) + { + return Ok(obj); + } + + if let Some(await_method) = vm.get_method(obj.clone(), identifier!(vm, __await__)) { + let result = await_method?.call((), vm)?; + // __await__() must NOT return a coroutine (PEP 492) + if result.downcastable::<PyCoroutine>() + || result.downcast_ref::<PyGenerator>().is_some_and(|g| { + g.as_coro() + .frame() + .code + .flags + .contains(crate::bytecode::CodeFlags::ITERABLE_COROUTINE) + }) + { + return Err(vm.new_type_error("__await__() returned a coroutine".to_owned())); + } + if !PyIter::check(&result) { + return Err(vm.new_type_error(format!( + "__await__() returned non-iterator of type '{}'", + result.class().name() + ))); + } + return Ok(result); + } + + Err(vm.new_type_error(format!("'{}' object can't be awaited", obj.class().name()))) +} + /// Emit DeprecationWarning for the deprecated 3-argument throw() signature. pub fn warn_deprecated_throw_signature( exc_val: &OptionalArg, diff --git a/crates/vm/src/dict_inner.rs b/crates/vm/src/dict_inner.rs index d57f8be0fe7..f2a379d99a5 100644 --- a/crates/vm/src/dict_inner.rs +++ b/crates/vm/src/dict_inner.rs @@ -553,6 +553,22 @@ impl<T: Clone> Dict<T> { .collect() } + pub fn values(&self) -> Vec<T> { + self.read() + .entries + .iter() + .filter_map(|v| v.as_ref().map(|v| v.value.clone())) + .collect() + } + + pub fn items(&self) -> Vec<(PyObjectRef, T)> { + self.read() + .entries + .iter() + .filter_map(|v| v.as_ref().map(|v| (v.key.clone(), v.value.clone()))) + .collect() + } + pub fn try_fold_keys<Acc, Fold>(&self, init: Acc, f: Fold) -> PyResult<Acc> where Fold: FnMut(Acc, &PyObject) -> PyResult<Acc>, @@ -708,6 +724,17 @@ impl<T: Clone> Dict<T> { + inner.indices.len() * size_of::<i64>() + inner.entries.len() * size_of::<DictEntry<T>>() } + + /// Pop all entries from the dict, returning (key, value) pairs. + /// This is used for circular reference resolution in GC. + /// Requires &mut self to avoid lock contention. + pub fn drain_entries(&mut self) -> impl Iterator<Item = (PyObjectRef, T)> + '_ { + let inner = self.inner.get_mut(); + inner.used = 0; + inner.filled = 0; + inner.indices.iter_mut().for_each(|i| *i = IndexEntry::FREE); + inner.entries.drain(..).flatten().map(|e| (e.key, e.value)) + } } type LookupResult = (IndexEntry, IndexIndex); diff --git a/crates/vm/src/exception_group.rs b/crates/vm/src/exception_group.rs index a55273480f6..7ad27c078af 100644 --- a/crates/vm/src/exception_group.rs +++ b/crates/vm/src/exception_group.rs @@ -348,8 +348,18 @@ pub(super) mod types { impl Initializer for PyBaseExceptionGroup { type Args = FuncArgs; - fn slot_init(_zelf: PyObjectRef, _args: FuncArgs, _vm: &VirtualMachine) -> PyResult<()> { - // No-op: __new__ already set up the correct args (message, exceptions_tuple) + fn slot_init(zelf: PyObjectRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult<()> { + // BaseExceptionGroup_init: no kwargs allowed + if !args.kwargs.is_empty() { + return Err(vm.new_type_error(format!( + "{} does not take keyword arguments", + zelf.class().name() + ))); + } + // Do NOT call PyBaseException::slot_init here. + // slot_new already set args to (message, exceptions_tuple). + // Calling base init would overwrite with original args (message, exceptions_list). + let _ = (zelf, args, vm); Ok(()) } diff --git a/crates/vm/src/exceptions.rs b/crates/vm/src/exceptions.rs index a2b08655822..dc25b6d2c03 100644 --- a/crates/vm/src/exceptions.rs +++ b/crates/vm/src/exceptions.rs @@ -1086,15 +1086,16 @@ fn syntax_error_set_msg( } fn system_exit_code(exc: PyBaseExceptionRef) -> Option<PyObjectRef> { - exc.args.read().first().map(|code| { - match_class!(match code { - ref tup @ PyTuple => match tup.as_slice() { - [x] => x.clone(), - _ => code.clone(), - }, - other => other.clone(), - }) - }) + // SystemExit.code based on args length: + // - size == 0: code is None + // - size == 1: code is args[0] + // - size > 1: code is args (the whole tuple) + let args = exc.args.read(); + match args.len() { + 0 => None, + 1 => Some(args.first().unwrap().clone()), + _ => Some(args.as_object().to_owned()), + } } #[cfg(feature = "serde")] @@ -1255,7 +1256,7 @@ pub(super) mod types { }, convert::ToPyObject, convert::ToPyResult, - function::{ArgBytesLike, FuncArgs}, + function::{ArgBytesLike, FuncArgs, KwArgs}, types::{Constructor, Initializer}, }; use crossbeam_utils::atomic::AtomicCell; @@ -1393,11 +1394,29 @@ pub(super) mod types { pub(super) args: PyRwLock<PyTupleRef>, } - #[pyexception(name, base = PyBaseException, ctx = "system_exit", impl)] + #[pyexception(name, base = PyBaseException, ctx = "system_exit")] #[derive(Debug)] #[repr(transparent)] pub struct PySystemExit(PyBaseException); + // SystemExit_init: has its own __init__ that sets the code attribute + #[pyexception(with(Initializer))] + impl PySystemExit {} + + impl Initializer for PySystemExit { + type Args = FuncArgs; + fn slot_init(zelf: PyObjectRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult<()> { + // Call BaseException_init first (handles args) + PyBaseException::slot_init(zelf, args, vm) + // Note: code is computed dynamically via system_exit_code getter + // so we don't need to set it here explicitly + } + + fn init(_zelf: PyRef<Self>, _args: Self::Args, _vm: &VirtualMachine) -> PyResult<()> { + unreachable!("slot_init is defined") + } + } + #[pyexception(name, base = PyBaseException, ctx = "generator_exit", impl)] #[derive(Debug)] #[repr(transparent)] @@ -1474,16 +1493,25 @@ pub(super) mod types { type Args = FuncArgs; fn slot_init(zelf: PyObjectRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult<()> { - zelf.set_attr( - "name", - vm.unwrap_or_none(args.kwargs.get("name").cloned()), - vm, - )?; - zelf.set_attr( - "obj", - vm.unwrap_or_none(args.kwargs.get("obj").cloned()), - vm, - )?; + // Only 'name' and 'obj' kwargs are allowed + let mut kwargs = args.kwargs.clone(); + let name = kwargs.swap_remove("name"); + let obj = kwargs.swap_remove("obj"); + + // Reject unknown kwargs + if let Some(invalid_key) = kwargs.keys().next() { + return Err(vm.new_type_error(format!( + "AttributeError() got an unexpected keyword argument '{invalid_key}'" + ))); + } + + // Pass args without kwargs to BaseException_init + let base_args = FuncArgs::new(args.args.clone(), KwArgs::default()); + PyBaseException::slot_init(zelf.clone(), base_args, vm)?; + + // Set attributes + zelf.set_attr("name", vm.unwrap_or_none(name), vm)?; + zelf.set_attr("obj", vm.unwrap_or_none(obj), vm)?; Ok(()) } @@ -1529,9 +1557,11 @@ pub(super) mod types { type Args = FuncArgs; fn slot_init(zelf: PyObjectRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult<()> { + // Only 'name', 'path', 'name_from' kwargs are allowed let mut kwargs = args.kwargs.clone(); let name = kwargs.swap_remove("name"); let path = kwargs.swap_remove("path"); + let name_from = kwargs.swap_remove("name_from"); // Check for any remaining invalid keyword arguments if let Some(invalid_key) = kwargs.keys().next() { @@ -1543,6 +1573,7 @@ pub(super) mod types { let dict = zelf.dict().unwrap(); dict.set_item("name", vm.unwrap_or_none(name), vm)?; dict.set_item("path", vm.unwrap_or_none(path), vm)?; + dict.set_item("name_from", vm.unwrap_or_none(name_from), vm)?; PyBaseException::slot_init(zelf, args, vm) } @@ -1592,11 +1623,45 @@ pub(super) mod types { #[repr(transparent)] pub struct PyMemoryError(PyException); - #[pyexception(name, base = PyException, ctx = "name_error", impl)] + #[pyexception(name, base = PyException, ctx = "name_error")] #[derive(Debug)] #[repr(transparent)] pub struct PyNameError(PyException); + // NameError_init: handles the .name. kwarg + #[pyexception(with(Initializer))] + impl PyNameError {} + + impl Initializer for PyNameError { + type Args = FuncArgs; + fn slot_init(zelf: PyObjectRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult<()> { + // Only 'name' kwarg is allowed + let mut kwargs = args.kwargs.clone(); + let name = kwargs.swap_remove("name"); + + // Reject unknown kwargs + if let Some(invalid_key) = kwargs.keys().next() { + return Err(vm.new_type_error(format!( + "NameError() got an unexpected keyword argument '{invalid_key}'" + ))); + } + + // Pass args without kwargs to BaseException_init + let base_args = FuncArgs::new(args.args.clone(), KwArgs::default()); + PyBaseException::slot_init(zelf.clone(), base_args, vm)?; + + // Set name attribute if provided + if let Some(name) = name { + zelf.set_attr("name", name, vm)?; + } + Ok(()) + } + + fn init(_zelf: PyRef<Self>, _args: Self::Args, _vm: &VirtualMachine) -> PyResult<()> { + unreachable!("slot_init is defined") + } + } + #[pyexception(name, base = PyNameError, ctx = "unbound_local_error", impl)] #[derive(Debug)] #[repr(transparent)] @@ -2232,31 +2297,17 @@ pub(super) mod types { } } + // MiddlingExtendsException: inherits __init__ from SyntaxError via MRO #[pyexception( name = "_IncompleteInputError", base = PySyntaxError, - ctx = "incomplete_input_error" + ctx = "incomplete_input_error", + impl )] #[derive(Debug)] #[repr(transparent)] pub struct PyIncompleteInputError(PySyntaxError); - #[pyexception(with(Initializer))] - impl PyIncompleteInputError {} - - impl Initializer for PyIncompleteInputError { - type Args = FuncArgs; - - fn slot_init(zelf: PyObjectRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult<()> { - zelf.set_attr("name", vm.ctx.new_str("SyntaxError"), vm)?; - PySyntaxError::slot_init(zelf, args, vm) - } - - fn init(_zelf: PyRef<Self>, _args: Self::Args, _vm: &VirtualMachine) -> PyResult<()> { - unreachable!("slot_init is defined") - } - } - #[pyexception(name, base = PySyntaxError, ctx = "indentation_error", impl)] #[derive(Debug)] #[repr(transparent)] diff --git a/crates/vm/src/frame.rs b/crates/vm/src/frame.rs index fc56f50fb3d..ced0c07f271 100644 --- a/crates/vm/src/frame.rs +++ b/crates/vm/src/frame.rs @@ -1,17 +1,22 @@ +#[cfg(feature = "flame")] +use crate::bytecode::InstructionMetadata; use crate::{ AsObject, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, TryFromObject, VirtualMachine, builtins::{ PyBaseException, PyBaseExceptionRef, PyCode, PyCoroutine, PyDict, PyDictRef, PyGenerator, - PyList, PySet, PySlice, PyStr, PyStrInterned, PyStrRef, PyTraceback, PyType, + PyInterpolation, PyList, PySet, PySlice, PyStr, PyStrInterned, PyStrRef, PyTemplate, + PyTraceback, PyType, asyncgenerator::PyAsyncGenWrappedValue, function::{PyCell, PyCellRef, PyFunction}, tuple::{PyTuple, PyTupleRef}, }, - bytecode, + bytecode::{self, Instruction}, convert::{IntoObject, ToPyResult}, coroutine::Coro, exceptions::ExceptionCtor, function::{ArgMapping, Either, FuncArgs}, + object::PyAtomicBorrow, + object::{Traverse, TraverseFn}, protocol::{PyIter, PyIterReturn}, scope::Scope, stdlib::{builtins, typing}, @@ -24,6 +29,7 @@ use core::iter::zip; use core::sync::atomic; use indexmap::IndexMap; use itertools::Itertools; + use rustpython_common::{boxvec::BoxVec, lock::PyMutex, wtf8::Wtf8Buf}; use rustpython_compiler_core::SourceLocation; @@ -40,20 +46,13 @@ enum UnwindReason { /// We hit an exception, so unwind any try-except and finally blocks. The exception should be /// on top of the vm exception stack. Raising { exception: PyBaseExceptionRef }, - - // NoWorries, - /// We are unwinding blocks, since we hit break - Break { target: bytecode::Label }, - - /// We are unwinding blocks since we hit a continue statements. - Continue { target: bytecode::Label }, } #[derive(Debug)] struct FrameState { // We need 1 stack per frame /// The main data frame of the stack machine - stack: BoxVec<PyObjectRef>, + stack: BoxVec<Option<PyObjectRef>>, /// index of last instruction ran #[cfg(feature = "threading")] lasti: u32, @@ -64,7 +63,7 @@ type Lasti = atomic::AtomicU32; #[cfg(not(feature = "threading"))] type Lasti = core::cell::Cell<u32>; -#[pyclass(module = false, name = "frame")] +#[pyclass(module = false, name = "frame", traverse = "manual")] pub struct Frame { pub code: PyRef<PyCode>, pub func_obj: Option<PyObjectRef>, @@ -85,7 +84,12 @@ pub struct Frame { // member pub trace_lines: PyMutex<bool>, + pub trace_opcodes: PyMutex<bool>, pub temporary_refs: PyMutex<Vec<PyObjectRef>>, + /// Back-reference to owning generator/coroutine/async generator. + /// Borrowed reference (not ref-counted) to avoid Generator↔Frame cycle. + /// Cleared by the generator's Drop impl. + pub generator: PyAtomicBorrow, } impl PyPayload for Frame { @@ -95,6 +99,28 @@ impl PyPayload for Frame { } } +unsafe impl Traverse for FrameState { + fn traverse(&self, tracer_fn: &mut TraverseFn<'_>) { + self.stack.traverse(tracer_fn); + } +} + +unsafe impl Traverse for Frame { + fn traverse(&self, tracer_fn: &mut TraverseFn<'_>) { + self.code.traverse(tracer_fn); + self.func_obj.traverse(tracer_fn); + self.fastlocals.traverse(tracer_fn); + self.cells_frees.traverse(tracer_fn); + self.locals.traverse(tracer_fn); + self.globals.traverse(tracer_fn); + self.builtins.traverse(tracer_fn); + self.trace.traverse(tracer_fn); + self.state.traverse(tracer_fn); + self.temporary_refs.traverse(tracer_fn); + // generator is a borrowed reference, not traversed + } +} + // Running a frame can result in one of the below: pub enum ExecutionResult { Return(PyObjectRef), @@ -113,10 +139,24 @@ impl Frame { func_obj: Option<PyObjectRef>, vm: &VirtualMachine, ) -> Self { - let cells_frees = core::iter::repeat_with(|| PyCell::default().into_ref(&vm.ctx)) - .take(code.cellvars.len()) - .chain(closure.iter().cloned()) - .collect(); + let nlocals = code.varnames.len(); + let num_cells = code.cellvars.len(); + let nfrees = closure.len(); + + let cells_frees: Box<[PyCellRef]> = + core::iter::repeat_with(|| PyCell::default().into_ref(&vm.ctx)) + .take(num_cells) + .chain(closure.iter().cloned()) + .collect(); + + // Extend fastlocals to include varnames + cellvars + freevars (localsplus) + let total_locals = nlocals + num_cells + nfrees; + let mut fastlocals_vec: Vec<Option<PyObjectRef>> = vec![None; total_locals]; + + // Store cell objects at cellvars and freevars positions + for (i, cell) in cells_frees.iter().enumerate() { + fastlocals_vec[nlocals + i] = Some(cell.clone().into()); + } let state = FrameState { stack: BoxVec::new(code.max_stackdepth as usize), @@ -125,7 +165,7 @@ impl Frame { }; Self { - fastlocals: PyMutex::new(vec![None; code.varnames.len()].into_boxed_slice()), + fastlocals: PyMutex::new(fastlocals_vec.into_boxed_slice()), cells_frees, locals: scope.locals, globals: scope.globals, @@ -136,10 +176,23 @@ impl Frame { state: PyMutex::new(state), trace: PyMutex::new(vm.ctx.none()), trace_lines: PyMutex::new(true), + trace_opcodes: PyMutex::new(false), temporary_refs: PyMutex::new(vec![]), + generator: PyAtomicBorrow::new(), } } + /// Store a borrowed back-reference to the owning generator/coroutine. + /// The caller must ensure the generator outlives the frame. + pub fn set_generator(&self, generator: &PyObject) { + self.generator.store(generator); + } + + /// Clear the generator back-reference. Called when the generator is finalized. + pub fn clear_generator(&self) { + self.generator.clear(); + } + pub fn current_location(&self) -> SourceLocation { self.code.locations[self.lasti() as usize - 1].0 } @@ -186,7 +239,7 @@ impl Frame { Ok(()) }; map_to_dict(&code.cellvars, &self.cells_frees)?; - if code.flags.contains(bytecode::CodeFlags::IS_OPTIMIZED) { + if code.flags.contains(bytecode::CodeFlags::OPTIMIZED) { map_to_dict(&code.freevars, &self.cells_frees[code.cellvars.len()..])?; } } @@ -241,7 +294,21 @@ impl Py<Frame> { } pub fn yield_from_target(&self) -> Option<PyObjectRef> { - self.with_exec(|exec| exec.yield_from_target().map(PyObject::to_owned)) + // Use try_lock to avoid deadlock when the frame is currently executing. + // A running coroutine has no yield-from target. + let mut state = self.state.try_lock()?; + let exec = ExecutingFrame { + code: &self.code, + fastlocals: &self.fastlocals, + cells_frees: &self.cells_frees, + locals: &self.locals, + globals: &self.globals, + builtins: &self.builtins, + lasti: &self.lasti, + object: self, + state: &mut state, + }; + exec.yield_from_target().map(PyObject::to_owned) } pub fn is_internal_frame(&self) -> bool { @@ -387,15 +454,15 @@ impl ExecutingFrame<'_> { } // Check if this is a RERAISE instruction - // Both Instruction::Raise { kind: Reraise/ReraiseFromStack } and - // Instruction::Reraise are reraise operations that should not add + // Both AnyInstruction::Raise { kind: Reraise/ReraiseFromStack } and + // AnyInstruction::Reraise are reraise operations that should not add // new traceback entries let is_reraise = match op { - bytecode::Instruction::Raise { kind } => matches!( + Instruction::RaiseVarargs { kind } => matches!( kind.get(arg), bytecode::RaiseKind::BareRaise | bytecode::RaiseKind::ReraiseFromStack ), - bytecode::Instruction::Reraise { .. } => true, + Instruction::Reraise { .. } => true, _ => false, }; @@ -439,13 +506,13 @@ impl ExecutingFrame<'_> { let lasti = self.lasti() as usize; if let Some(unit) = self.code.instructions.get(lasti) { match &unit.op { - bytecode::Instruction::Send { .. } => return Some(self.top_value()), - bytecode::Instruction::Resume { .. } => { + Instruction::Send { .. } => return Some(self.top_value()), + Instruction::Resume { .. } => { // Check if previous instruction was YIELD_VALUE with arg >= 1 // This indicates yield-from/await context if lasti > 0 && let Some(prev_unit) = self.code.instructions.get(lasti - 1) - && let bytecode::Instruction::YieldValue { .. } = &prev_unit.op + && let Instruction::YieldValue { .. } = &prev_unit.op { // YIELD_VALUE arg: 0 = direct yield, >= 1 = yield-from/await // OpArgByte.0 is the raw byte value @@ -550,7 +617,7 @@ impl ExecutingFrame<'_> { #[inline(always)] fn execute_instruction( &mut self, - instruction: bytecode::Instruction, + instruction: Instruction, arg: bytecode::OpArg, extend_arg: &mut bool, vm: &VirtualMachine, @@ -584,77 +651,37 @@ impl ExecutingFrame<'_> { } match instruction { - bytecode::Instruction::BeforeAsyncWith => { - let mgr = self.pop_value(); - let error_string = || -> String { - format!( - "'{:.200}' object does not support the asynchronous context manager protocol", - mgr.class().name(), - ) - }; - - let aenter_res = vm - .get_special_method(&mgr, identifier!(vm, __aenter__))? - .ok_or_else(|| vm.new_type_error(error_string()))? - .invoke((), vm)?; - let aexit = mgr - .get_attr(identifier!(vm, __aexit__), vm) - .map_err(|_exc| { - vm.new_type_error({ - format!("{} (missed __aexit__ method)", error_string()) - }) - })?; - self.push_value(aexit); - self.push_value(aenter_res); - - Ok(None) - } - bytecode::Instruction::BinaryOp { op } => self.execute_bin_op(vm, op.get(arg)), - bytecode::Instruction::BinarySubscript => { - let key = self.pop_value(); + Instruction::BinaryOp { op } => self.execute_bin_op(vm, op.get(arg)), + // TODO: In CPython, this does in-place unicode concatenation when + // refcount is 1. Falls back to regular iadd for now. + Instruction::BinaryOpInplaceAddUnicode => { + self.execute_bin_op(vm, bytecode::BinaryOperator::InplaceAdd) + } + Instruction::BinarySlice => { + // Stack: [container, start, stop] -> [result] + let stop = self.pop_value(); + let start = self.pop_value(); let container = self.pop_value(); - self.state - .stack - .push(container.get_item(key.as_object(), vm)?); - Ok(None) - } - - bytecode::Instruction::Break { target } => self.unwind_blocks( - vm, - UnwindReason::Break { - target: target.get(arg), - }, - ), - bytecode::Instruction::BuildListFromTuples { size } => { - // SAFETY: compiler guarantees `size` tuples are on the stack - let elements = unsafe { self.flatten_tuples(size.get(arg) as usize) }; - let list_obj = vm.ctx.new_list(elements); - self.push_value(list_obj.into()); + let slice: PyObjectRef = PySlice { + start: Some(start), + stop, + step: None, + } + .into_ref(&vm.ctx) + .into(); + let result = container.get_item(&*slice, vm)?; + self.push_value(result); Ok(None) } - bytecode::Instruction::BuildList { size } => { - let elements = self.pop_multiple(size.get(arg) as usize).collect(); + Instruction::BuildList { size } => { + let sz = size.get(arg) as usize; + let elements = self.pop_multiple(sz).collect(); let list_obj = vm.ctx.new_list(elements); self.push_value(list_obj.into()); Ok(None) } - bytecode::Instruction::BuildMapForCall { size } => { - self.execute_build_map_for_call(vm, size.get(arg)) - } - bytecode::Instruction::BuildMap { size } => self.execute_build_map(vm, size.get(arg)), - bytecode::Instruction::BuildSetFromTuples { size } => { - let set = PySet::default().into_ref(&vm.ctx); - for element in self.pop_multiple(size.get(arg) as usize) { - // SAFETY: trust compiler - let tup = unsafe { element.downcast_unchecked::<PyTuple>() }; - for item in tup.iter() { - set.add(item.clone(), vm)?; - } - } - self.push_value(set.into()); - Ok(None) - } - bytecode::Instruction::BuildSet { size } => { + Instruction::BuildMap { size } => self.execute_build_map(vm, size.get(arg)), + Instruction::BuildSet { size } => { let set = PySet::default().into_ref(&vm.ctx); for element in self.pop_multiple(size.get(arg) as usize) { set.add(element, vm)?; @@ -662,11 +689,9 @@ impl ExecutingFrame<'_> { self.push_value(set.into()); Ok(None) } - bytecode::Instruction::BuildSlice { argc } => { - self.execute_build_slice(vm, argc.get(arg)) - } + Instruction::BuildSlice { argc } => self.execute_build_slice(vm, argc.get(arg)), /* - bytecode::Instruction::ToBool => { + Instruction::ToBool => { dbg!("Shouldn't be called outside of match statements for now") let value = self.pop_value(); // call __bool__ @@ -675,86 +700,118 @@ impl ExecutingFrame<'_> { Ok(None) } */ - bytecode::Instruction::BuildString { size } => { - let s = self + Instruction::BuildString { size } => { + let s: Wtf8Buf = self .pop_multiple(size.get(arg) as usize) - .as_slice() - .iter() - .map(|pyobj| pyobj.downcast_ref::<PyStr>().unwrap()) - .collect::<Wtf8Buf>(); - let str_obj = vm.ctx.new_str(s); - self.push_value(str_obj.into()); - Ok(None) - } - bytecode::Instruction::BuildTupleFromIter => { - if !self.top_value().class().is(vm.ctx.types.tuple_type) { - let elements: Vec<_> = self.pop_value().try_to_value(vm)?; - let list_obj = vm.ctx.new_tuple(elements); - self.push_value(list_obj.into()); - } + .map(|pyobj| pyobj.downcast::<PyStr>().unwrap()) + .collect(); + self.push_value(vm.ctx.new_str(s).into()); Ok(None) } - bytecode::Instruction::BuildTupleFromTuples { size } => { - // SAFETY: compiler guarantees `size` tuples are on the stack - let elements = unsafe { self.flatten_tuples(size.get(arg) as usize) }; + Instruction::BuildTuple { size } => { + let elements = self.pop_multiple(size.get(arg) as usize).collect(); let list_obj = vm.ctx.new_tuple(elements); self.push_value(list_obj.into()); Ok(None) } - bytecode::Instruction::BuildTuple { size } => { - let elements = self.pop_multiple(size.get(arg) as usize).collect(); - let list_obj = vm.ctx.new_tuple(elements); - self.push_value(list_obj.into()); + Instruction::BuildTemplate => { + // Stack: [strings_tuple, interpolations_tuple] -> [template] + let interpolations = self.pop_value(); + let strings = self.pop_value(); + + let strings = strings + .downcast::<PyTuple>() + .map_err(|_| vm.new_type_error("BUILD_TEMPLATE expected tuple for strings"))?; + let interpolations = interpolations.downcast::<PyTuple>().map_err(|_| { + vm.new_type_error("BUILD_TEMPLATE expected tuple for interpolations") + })?; + + let template = PyTemplate::new(strings, interpolations); + self.push_value(template.into_pyobject(vm)); + Ok(None) + } + Instruction::BuildInterpolation { oparg } => { + // oparg encoding: (conversion << 2) | has_format_spec + // Stack: [value, expression_str, (format_spec)?] -> [interpolation] + let oparg_val = oparg.get(arg); + let has_format_spec = (oparg_val & 1) != 0; + let conversion_code = oparg_val >> 2; + + let format_spec = if has_format_spec { + self.pop_value().downcast::<PyStr>().map_err(|_| { + vm.new_type_error("BUILD_INTERPOLATION expected str for format_spec") + })? + } else { + vm.ctx.empty_str.to_owned() + }; + + let expression = self.pop_value().downcast::<PyStr>().map_err(|_| { + vm.new_type_error("BUILD_INTERPOLATION expected str for expression") + })?; + let value = self.pop_value(); + + // conversion: 0=None, 1=Str, 2=Repr, 3=Ascii + let conversion: PyObjectRef = match conversion_code { + 0 => vm.ctx.none(), + 1 => vm.ctx.new_str("s").into(), + 2 => vm.ctx.new_str("r").into(), + 3 => vm.ctx.new_str("a").into(), + _ => vm.ctx.none(), // should not happen + }; + + let interpolation = + PyInterpolation::new(value, expression, conversion, format_spec, vm)?; + self.push_value(interpolation.into_pyobject(vm)); Ok(None) } - bytecode::Instruction::CallFunctionEx { has_kwargs } => { - let args = self.collect_ex_args(vm, has_kwargs.get(arg))?; + Instruction::Call { nargs } => { + // Stack: [callable, self_or_null, arg1, ..., argN] + let args = self.collect_positional_args(nargs.get(arg)); self.execute_call(args, vm) } - bytecode::Instruction::CallFunctionKeyword { nargs } => { + Instruction::CallKw { nargs } => { + // Stack: [callable, self_or_null, arg1, ..., argN, kwarg_names] let args = self.collect_keyword_args(nargs.get(arg)); self.execute_call(args, vm) } - bytecode::Instruction::CallFunctionPositional { nargs } => { - let args = self.collect_positional_args(nargs.get(arg)); + Instruction::CallFunctionEx => { + // Stack: [callable, self_or_null, args_tuple, kwargs_or_null] + let args = self.collect_ex_args(vm)?; self.execute_call(args, vm) } - bytecode::Instruction::CallIntrinsic1 { func } => { + Instruction::CallIntrinsic1 { func } => { let value = self.pop_value(); let result = self.call_intrinsic_1(func.get(arg), value, vm)?; self.push_value(result); Ok(None) } - bytecode::Instruction::CallIntrinsic2 { func } => { + Instruction::CallIntrinsic2 { func } => { let value2 = self.pop_value(); let value1 = self.pop_value(); let result = self.call_intrinsic_2(func.get(arg), value1, value2, vm)?; self.push_value(result); Ok(None) } - bytecode::Instruction::CallMethodEx { has_kwargs } => { - let args = self.collect_ex_args(vm, has_kwargs.get(arg))?; - self.execute_method_call(args, vm) - } - bytecode::Instruction::CallMethodKeyword { nargs } => { - let args = self.collect_keyword_args(nargs.get(arg)); - self.execute_method_call(args, vm) - } - bytecode::Instruction::CallMethodPositional { nargs } => { - let args = self.collect_positional_args(nargs.get(arg)); - self.execute_method_call(args, vm) - } - bytecode::Instruction::CheckEgMatch => { + Instruction::CheckEgMatch => { let match_type = self.pop_value(); let exc_value = self.pop_value(); let (rest, matched) = crate::exceptions::exception_group_match(&exc_value, &match_type, vm)?; + + // Set matched exception as current exception (if not None) + // This mirrors CPython's PyErr_SetHandledException(match_o) in CHECK_EG_MATCH + if !vm.is_none(&matched) + && let Some(exc) = matched.downcast_ref::<PyBaseException>() + { + vm.set_exception(Some(exc.to_owned())); + } + self.push_value(rest); self.push_value(matched); Ok(None) } - bytecode::Instruction::CompareOperation { op } => self.execute_compare(vm, op.get(arg)), - bytecode::Instruction::ContainsOp(invert) => { + Instruction::CompareOp { op } => self.execute_compare(vm, op.get(arg)), + Instruction::ContainsOp(invert) => { let b = self.pop_value(); let a = self.pop_value(); @@ -765,17 +822,10 @@ impl ExecutingFrame<'_> { self.push_value(vm.ctx.new_bool(value).into()); Ok(None) } - bytecode::Instruction::Continue { target } => self.unwind_blocks( - vm, - UnwindReason::Continue { - target: target.get(arg), - }, - ), - - bytecode::Instruction::ConvertValue { oparg: conversion } => { + Instruction::ConvertValue { oparg: conversion } => { self.convert_value(conversion.get(arg), vm) } - bytecode::Instruction::CopyItem { index } => { + Instruction::Copy { index } => { // CopyItem { index: 1 } copies TOS // CopyItem { index: 2 } copies second from top // This is 1-indexed to match CPython @@ -788,15 +838,19 @@ impl ExecutingFrame<'_> { panic!("CopyItem: stack underflow"); } let value = self.state.stack[stack_len - idx].clone(); - self.push_value(value); + self.push_value_opt(value); + Ok(None) + } + Instruction::CopyFreeVars { .. } => { + // Free vars are already set up at frame creation time in RustPython Ok(None) } - bytecode::Instruction::DeleteAttr { idx } => self.delete_attr(vm, idx.get(arg)), - bytecode::Instruction::DeleteDeref(i) => { + Instruction::DeleteAttr { idx } => self.delete_attr(vm, idx.get(arg)), + Instruction::DeleteDeref(i) => { self.cells_frees[i.get(arg) as usize].set(None); Ok(None) } - bytecode::Instruction::DeleteFast(idx) => { + Instruction::DeleteFast(idx) => { let mut fastlocals = self.fastlocals.lock(); let idx = idx.get(arg) as usize; if fastlocals[idx].is_none() { @@ -811,7 +865,7 @@ impl ExecutingFrame<'_> { fastlocals[idx] = None; Ok(None) } - bytecode::Instruction::DeleteGlobal(idx) => { + Instruction::DeleteGlobal(idx) => { let name = self.code.names[idx.get(arg) as usize]; match self.globals.del_item(name, vm) { Ok(()) => {} @@ -822,7 +876,7 @@ impl ExecutingFrame<'_> { } Ok(None) } - bytecode::Instruction::DeleteLocal(idx) => { + Instruction::DeleteName(idx) => { let name = self.code.names[idx.get(arg) as usize]; let res = self.locals.mapping().ass_subscript(name, None, vm); @@ -835,8 +889,8 @@ impl ExecutingFrame<'_> { } Ok(None) } - bytecode::Instruction::DeleteSubscript => self.execute_delete_subscript(vm), - bytecode::Instruction::DictUpdate { index } => { + Instruction::DeleteSubscr => self.execute_delete_subscript(vm), + Instruction::DictUpdate { index } => { // Stack before: [..., dict, ..., source] (source at TOS) // Stack after: [..., dict, ...] (source consumed) // The dict to update is at position TOS-i (before popping source) @@ -872,10 +926,52 @@ impl ExecutingFrame<'_> { dict.merge_object(source, vm)?; Ok(None) } - bytecode::Instruction::EndAsyncFor => { - // END_ASYNC_FOR pops (awaitable, exc) from stack - // Stack: [awaitable, exc] -> [] - // exception_unwind pushes exception to stack before jumping to handler + Instruction::DictMerge { index } => { + let source = self.pop_value(); + let idx = index.get(arg); + + // Get the dict to merge into (same logic as DICT_UPDATE) + let dict_ref = if idx <= 1 { + self.top_value() + } else { + self.nth_value(idx - 1) + }; + + let dict: &Py<PyDict> = unsafe { dict_ref.downcast_unchecked_ref() }; + + // Check if source is a mapping + if vm + .get_method(source.clone(), vm.ctx.intern_str("keys")) + .is_none() + { + return Err(vm.new_type_error(format!( + "'{}' object is not a mapping", + source.class().name() + ))); + } + + // Check for duplicate keys + let keys_iter = vm.call_method(&source, "keys", ())?; + for key in keys_iter.try_to_value::<Vec<PyObjectRef>>(vm)? { + if key.downcast_ref::<PyStr>().is_none() { + return Err(vm.new_type_error("keywords must be strings".to_owned())); + } + if dict.contains_key(&*key, vm) { + let key_repr = key.repr(vm)?; + return Err(vm.new_type_error(format!( + "got multiple values for keyword argument {}", + key_repr.as_str() + ))); + } + let value = vm.call_method(&source, "__getitem__", (key.clone(),))?; + dict.set_item(&*key, value, vm)?; + } + Ok(None) + } + Instruction::EndAsyncFor => { + // Pops (awaitable, exc) from stack. + // If exc is StopAsyncIteration, clears it (normal loop end). + // Otherwise re-raises. let exc = self.pop_value(); let _awaitable = self.pop_value(); @@ -892,19 +988,19 @@ impl ExecutingFrame<'_> { Err(exc) } } - bytecode::Instruction::ExtendedArg => { + Instruction::ExtendedArg => { *extend_arg = true; Ok(None) } - bytecode::Instruction::ForIter { target } => self.execute_for_iter(vm, target.get(arg)), - bytecode::Instruction::FormatSimple => { + Instruction::ForIter { target } => self.execute_for_iter(vm, target.get(arg)), + Instruction::FormatSimple => { let value = self.pop_value(); let formatted = vm.format(&value, vm.ctx.new_str(""))?; self.push_value(formatted.into()); Ok(None) } - bytecode::Instruction::FormatWithSpec => { + Instruction::FormatWithSpec => { let spec = self.pop_value(); let value = self.pop_value(); let formatted = vm.format(&value, spec.downcast::<PyStr>().unwrap())?; @@ -912,13 +1008,13 @@ impl ExecutingFrame<'_> { Ok(None) } - bytecode::Instruction::GetAIter => { + Instruction::GetAIter => { let aiterable = self.pop_value(); let aiter = vm.call_special_method(&aiterable, identifier!(vm, __aiter__), ())?; self.push_value(aiter); Ok(None) } - bytecode::Instruction::GetANext => { + Instruction::GetANext => { #[cfg(debug_assertions)] // remove when GetANext is fully implemented let orig_stack_len = self.state.stack.len(); @@ -963,49 +1059,56 @@ impl ExecutingFrame<'_> { debug_assert_eq!(orig_stack_len + 1, self.state.stack.len()); Ok(None) } - bytecode::Instruction::GetAwaitable => { - use crate::protocol::PyIter; + Instruction::GetAwaitable { arg: oparg } => { + let iterable = self.pop_value(); - let awaited_obj = self.pop_value(); - let awaitable = if let Some(coro) = awaited_obj.downcast_ref::<PyCoroutine>() { - // _PyGen_yf() check - detect if coroutine is already being awaited elsewhere - if coro.as_coro().frame().yield_from_target().is_some() { - return Err( - vm.new_runtime_error("coroutine is being awaited already".to_owned()) - ); - } - awaited_obj - } else { - let await_method = vm.get_method_or_type_error( - awaited_obj.clone(), - identifier!(vm, __await__), - || { - format!( - "object {} can't be used in 'await' expression", - awaited_obj.class().name(), - ) - }, - )?; - let result = await_method.call((), vm)?; - // Check that __await__ returned an iterator - if !PyIter::check(&result) { - return Err(vm.new_type_error(format!( - "__await__() returned non-iterator of type '{}'", - result.class().name() - ))); + let iter = match crate::coroutine::get_awaitable_iter(iterable.clone(), vm) { + Ok(iter) => iter, + Err(e) => { + // _PyEval_FormatAwaitableError: override error for async with + // when the type doesn't have __await__ + let oparg_val = oparg.get(arg); + if vm + .get_method(iterable.clone(), identifier!(vm, __await__)) + .is_none() + { + if oparg_val == 1 { + return Err(vm.new_type_error(format!( + "'async with' received an object from __aenter__ \ + that does not implement __await__: {}", + iterable.class().name() + ))); + } else if oparg_val == 2 { + return Err(vm.new_type_error(format!( + "'async with' received an object from __aexit__ \ + that does not implement __await__: {}", + iterable.class().name() + ))); + } + } + return Err(e); } - result }; - self.push_value(awaitable); + + // Check if coroutine is already being awaited + if let Some(coro) = iter.downcast_ref::<PyCoroutine>() + && coro.as_coro().frame().yield_from_target().is_some() + { + return Err( + vm.new_runtime_error("coroutine is being awaited already".to_owned()) + ); + } + + self.push_value(iter); Ok(None) } - bytecode::Instruction::GetIter => { + Instruction::GetIter => { let iterated_obj = self.pop_value(); let iter_obj = iterated_obj.get_iter(vm)?; self.push_value(iter_obj.into()); Ok(None) } - bytecode::Instruction::GetYieldFromIter => { + Instruction::GetYieldFromIter => { // GET_YIELD_FROM_ITER: prepare iterator for yield from // If iterable is a coroutine, ensure we're in a coroutine context // If iterable is a generator, use it directly @@ -1013,11 +1116,9 @@ impl ExecutingFrame<'_> { let iterable = self.pop_value(); let iter = if iterable.class().is(vm.ctx.types.coroutine_type) { // Coroutine requires CO_COROUTINE or CO_ITERABLE_COROUTINE flag - if !self - .code - .flags - .intersects(bytecode::CodeFlags::IS_COROUTINE) - { + if !self.code.flags.intersects( + bytecode::CodeFlags::COROUTINE | bytecode::CodeFlags::ITERABLE_COROUTINE, + ) { return Err(vm.new_type_error( "cannot 'yield from' a coroutine object in a non-coroutine generator" .to_owned(), @@ -1034,23 +1135,23 @@ impl ExecutingFrame<'_> { self.push_value(iter); Ok(None) } - bytecode::Instruction::GetLen => { + Instruction::GetLen => { // STACK.append(len(STACK[-1])) let obj = self.top_value(); let len = obj.length(vm)?; self.push_value(vm.ctx.new_int(len).into()); Ok(None) } - bytecode::Instruction::ImportFrom { idx } => { + Instruction::ImportFrom { idx } => { let obj = self.import_from(vm, idx.get(arg))?; self.push_value(obj); Ok(None) } - bytecode::Instruction::ImportName { idx } => { + Instruction::ImportName { idx } => { self.import(vm, Some(self.code.names[idx.get(arg) as usize]))?; Ok(None) } - bytecode::Instruction::IsOp(invert) => { + Instruction::IsOp(invert) => { let b = self.pop_value(); let a = self.pop_value(); let res = a.is(&b); @@ -1062,40 +1163,19 @@ impl ExecutingFrame<'_> { self.push_value(vm.ctx.new_bool(value).into()); Ok(None) } - bytecode::Instruction::JumpIfFalseOrPop { target } => { - self.jump_if_or_pop(vm, target.get(arg), false) - } - bytecode::Instruction::JumpIfNotExcMatch(target) => { - let b = self.pop_value(); - let a = self.pop_value(); - if let Some(tuple_of_exceptions) = b.downcast_ref::<PyTuple>() { - for exception in tuple_of_exceptions { - if !exception - .is_subclass(vm.ctx.exceptions.base_exception_type.into(), vm)? - { - return Err(vm.new_type_error( - "catching classes that do not inherit from BaseException is not allowed", - )); - } - } - } else if !b.is_subclass(vm.ctx.exceptions.base_exception_type.into(), vm)? { - return Err(vm.new_type_error( - "catching classes that do not inherit from BaseException is not allowed", - )); - } - - let value = a.is_instance(&b, vm)?; - self.push_value(vm.ctx.new_bool(value).into()); - self.pop_jump_if(vm, target.get(arg), false) + Instruction::JumpForward { target } => { + self.jump(target.get(arg)); + Ok(None) } - bytecode::Instruction::JumpIfTrueOrPop { target } => { - self.jump_if_or_pop(vm, target.get(arg), true) + Instruction::JumpBackward { target } => { + self.jump(target.get(arg)); + Ok(None) } - bytecode::Instruction::Jump { target } => { + Instruction::JumpBackwardNoInterrupt { target } => { self.jump(target.get(arg)); Ok(None) } - bytecode::Instruction::ListAppend { i } => { + Instruction::ListAppend { i } => { let item = self.pop_value(); let obj = self.nth_value(i.get(arg)); let list: &Py<PyList> = unsafe { @@ -1105,19 +1185,47 @@ impl ExecutingFrame<'_> { list.append(item); Ok(None) } - bytecode::Instruction::LoadAttr { idx } => self.load_attr(vm, idx.get(arg)), - bytecode::Instruction::LoadBuildClass => { + Instruction::ListExtend { i } => { + let iterable = self.pop_value(); + let obj = self.nth_value(i.get(arg)); + let list: &Py<PyList> = unsafe { + // SAFETY: compiler guarantees correct type + obj.downcast_unchecked_ref() + }; + list.extend(iterable, vm)?; + Ok(None) + } + Instruction::LoadAttr { idx } => self.load_attr(vm, idx.get(arg)), + Instruction::LoadSuperAttr { arg: idx } => self.load_super_attr(vm, idx.get(arg)), + Instruction::LoadBuildClass => { self.push_value(vm.builtins.get_attr(identifier!(vm, __build_class__), vm)?); Ok(None) } - bytecode::Instruction::LoadClassDeref(i) => { + Instruction::LoadLocals => { + // Push the locals dict onto the stack + let locals = self.locals.clone().into_object(); + self.push_value(locals); + Ok(None) + } + Instruction::LoadFromDictOrDeref(i) => { + // Pop dict from stack (locals or classdict depending on context) + let class_dict = self.pop_value(); let i = i.get(arg) as usize; let name = if i < self.code.cellvars.len() { self.code.cellvars[i] } else { self.code.freevars[i - self.code.cellvars.len()] }; - let value = self.locals.mapping().subscript(name, vm).ok(); + // Only treat KeyError as "not found", propagate other exceptions + let value = if let Some(dict_obj) = class_dict.downcast_ref::<PyDict>() { + dict_obj.get_item_opt(name, vm)? + } else { + match class_dict.get_item(name, vm) { + Ok(v) => Some(v), + Err(e) if e.fast_isinstance(vm.ctx.exceptions.key_error) => None, + Err(e) => return Err(e), + } + }; self.push_value(match value { Some(v) => v, None => self.cells_frees[i] @@ -1126,24 +1234,64 @@ impl ExecutingFrame<'_> { }); Ok(None) } - bytecode::Instruction::LoadClosure(i) => { - let value = self.cells_frees[i.get(arg) as usize].clone(); - self.push_value(value.into()); + Instruction::LoadFromDictOrGlobals(idx) => { + // PEP 649: Pop dict from stack (classdict), check there first, then globals + let dict = self.pop_value(); + let name = self.code.names[idx.get(arg) as usize]; + + // Only treat KeyError as "not found", propagate other exceptions + let value = if let Some(dict_obj) = dict.downcast_ref::<PyDict>() { + dict_obj.get_item_opt(name, vm)? + } else { + // Not an exact dict, use mapping protocol + match dict.get_item(name, vm) { + Ok(v) => Some(v), + Err(e) if e.fast_isinstance(vm.ctx.exceptions.key_error) => None, + Err(e) => return Err(e), + } + }; + + self.push_value(match value { + Some(v) => v, + None => self.load_global_or_builtin(name, vm)?, + }); Ok(None) } - bytecode::Instruction::LoadConst { idx } => { + Instruction::LoadConst { idx } => { self.push_value(self.code.constants[idx.get(arg) as usize].clone().into()); Ok(None) } - bytecode::Instruction::LoadDeref(i) => { - let i = i.get(arg) as usize; - let x = self.cells_frees[i] + Instruction::LoadCommonConstant { idx } => { + use bytecode::CommonConstant; + let value = match idx.get(arg) { + CommonConstant::AssertionError => { + vm.ctx.exceptions.assertion_error.to_owned().into() + } + CommonConstant::NotImplementedError => { + vm.ctx.exceptions.not_implemented_error.to_owned().into() + } + CommonConstant::BuiltinTuple => vm.ctx.types.tuple_type.to_owned().into(), + CommonConstant::BuiltinAll => vm.builtins.get_attr("all", vm)?, + CommonConstant::BuiltinAny => vm.builtins.get_attr("any", vm)?, + }; + self.push_value(value); + Ok(None) + } + Instruction::LoadSmallInt { idx } => { + // Push small integer (-5..=256) directly without constant table lookup + let value = vm.ctx.new_int(idx.get(arg) as i32); + self.push_value(value.into()); + Ok(None) + } + Instruction::LoadDeref(i) => { + let idx = i.get(arg) as usize; + let x = self.cells_frees[idx] .get() - .ok_or_else(|| self.unbound_cell_exception(i, vm))?; + .ok_or_else(|| self.unbound_cell_exception(idx, vm))?; self.push_value(x); Ok(None) } - bytecode::Instruction::LoadFast(idx) => { + Instruction::LoadFast(idx) => { #[cold] fn reference_error( varname: &'static PyStrInterned, @@ -1161,7 +1309,7 @@ impl ExecutingFrame<'_> { self.push_value(x); Ok(None) } - bytecode::Instruction::LoadFastAndClear(idx) => { + Instruction::LoadFastAndClear(idx) => { // Load value and clear the slot (for inlined comprehensions) // If slot is empty, push None (not an error - variable may not exist yet) let idx = idx.get(arg) as usize; @@ -1171,28 +1319,110 @@ impl ExecutingFrame<'_> { self.push_value(x); Ok(None) } - bytecode::Instruction::LoadGlobal(idx) => { - let name = &self.code.names[idx.get(arg) as usize]; - let x = self.load_global_or_builtin(name, vm)?; + Instruction::LoadFastCheck(idx) => { + // Same as LoadFast but explicitly checks for unbound locals + // (LoadFast in RustPython already does this check) + let idx = idx.get(arg) as usize; + let x = self.fastlocals.lock()[idx].clone().ok_or_else(|| { + vm.new_exception_msg( + vm.ctx.exceptions.unbound_local_error.to_owned(), + format!( + "local variable '{}' referenced before assignment", + self.code.varnames[idx] + ), + ) + })?; self.push_value(x); Ok(None) } - bytecode::Instruction::LoadMethod { idx } => { - let obj = self.pop_value(); - let method_name = self.code.names[idx.get(arg) as usize]; - let method = PyMethod::get(obj, method_name, vm)?; - let (target, is_method, func) = match method { - PyMethod::Function { target, func } => (target, true, func), - PyMethod::Attribute(val) => (vm.ctx.none(), false, val), - }; - // TODO: figure out a better way to communicate PyMethod::Attribute - CPython uses - // target==NULL, maybe we could use a sentinel value or something? - self.push_value(target); - self.push_value(vm.ctx.new_bool(is_method).into()); - self.push_value(func); + Instruction::LoadFastLoadFast { arg: packed } => { + // Load two local variables at once + // oparg encoding: (idx1 << 4) | idx2 + let oparg = packed.get(arg); + let idx1 = (oparg >> 4) as usize; + let idx2 = (oparg & 15) as usize; + let fastlocals = self.fastlocals.lock(); + let x1 = fastlocals[idx1].clone().ok_or_else(|| { + vm.new_exception_msg( + vm.ctx.exceptions.unbound_local_error.to_owned(), + format!( + "local variable '{}' referenced before assignment", + self.code.varnames[idx1] + ), + ) + })?; + let x2 = fastlocals[idx2].clone().ok_or_else(|| { + vm.new_exception_msg( + vm.ctx.exceptions.unbound_local_error.to_owned(), + format!( + "local variable '{}' referenced before assignment", + self.code.varnames[idx2] + ), + ) + })?; + drop(fastlocals); + self.push_value(x1); + self.push_value(x2); + Ok(None) + } + // TODO: Implement true borrow optimization (skip Arc::clone). + // CPython's LOAD_FAST_BORROW uses PyStackRef_Borrow to avoid refcount + // increment for values that are consumed within the same basic block. + // Possible approaches: + // - Store raw pointers with careful lifetime management + // - Add a "borrowed" variant to stack slots + // - Use arena allocation for short-lived stack values + // Currently this just clones like LoadFast. + Instruction::LoadFastBorrow(idx) => { + let idx = idx.get(arg) as usize; + let x = self.fastlocals.lock()[idx].clone().ok_or_else(|| { + vm.new_exception_msg( + vm.ctx.exceptions.unbound_local_error.to_owned(), + format!( + "local variable '{}' referenced before assignment", + self.code.varnames[idx] + ), + ) + })?; + self.push_value(x); Ok(None) } - bytecode::Instruction::LoadNameAny(idx) => { + // TODO: Same as LoadFastBorrow - implement true borrow optimization. + Instruction::LoadFastBorrowLoadFastBorrow { arg: packed } => { + let oparg = packed.get(arg); + let idx1 = (oparg >> 4) as usize; + let idx2 = (oparg & 15) as usize; + let fastlocals = self.fastlocals.lock(); + let x1 = fastlocals[idx1].clone().ok_or_else(|| { + vm.new_exception_msg( + vm.ctx.exceptions.unbound_local_error.to_owned(), + format!( + "local variable '{}' referenced before assignment", + self.code.varnames[idx1] + ), + ) + })?; + let x2 = fastlocals[idx2].clone().ok_or_else(|| { + vm.new_exception_msg( + vm.ctx.exceptions.unbound_local_error.to_owned(), + format!( + "local variable '{}' referenced before assignment", + self.code.varnames[idx2] + ), + ) + })?; + drop(fastlocals); + self.push_value(x1); + self.push_value(x2); + Ok(None) + } + Instruction::LoadGlobal(idx) => { + let name = &self.code.names[idx.get(arg) as usize]; + let x = self.load_global_or_builtin(name, vm)?; + self.push_value(x); + Ok(None) + } + Instruction::LoadName(idx) => { let name = self.code.names[idx.get(arg) as usize]; let result = self.locals.mapping().subscript(name, vm); match result { @@ -1204,8 +1434,46 @@ impl ExecutingFrame<'_> { } Ok(None) } - bytecode::Instruction::MakeFunction => self.execute_make_function(vm), - bytecode::Instruction::MapAdd { i } => { + Instruction::LoadSpecial { method } => { + // Stack effect: 0 (replaces TOS with bound method) + // Input: [..., obj] + // Output: [..., bound_method] + use crate::vm::PyMethod; + use bytecode::SpecialMethod; + + let obj = self.pop_value(); + let method_name = match method.get(arg) { + SpecialMethod::Enter => identifier!(vm, __enter__), + SpecialMethod::Exit => identifier!(vm, __exit__), + SpecialMethod::AEnter => identifier!(vm, __aenter__), + SpecialMethod::AExit => identifier!(vm, __aexit__), + }; + + let bound = match vm.get_special_method(&obj, method_name)? { + Some(PyMethod::Function { target, func }) => { + // Create bound method: PyBoundMethod(object=target, function=func) + crate::builtins::PyBoundMethod::new(target, func) + .into_ref(&vm.ctx) + .into() + } + Some(PyMethod::Attribute(bound)) => bound, + None => { + return Err(vm.new_type_error(format!( + "'{}' object does not support the context manager protocol (missed {} method)", + obj.class().name(), + method_name + ))); + } + }; + self.push_value(bound); + Ok(None) + } + Instruction::MakeFunction => self.execute_make_function(vm), + Instruction::MakeCell(_) => { + // Cell creation is handled at frame creation time in RustPython + Ok(None) + } + Instruction::MapAdd { i } => { let value = self.pop_value(); let key = self.pop_value(); let obj = self.nth_value(i.get(arg)); @@ -1216,7 +1484,7 @@ impl ExecutingFrame<'_> { dict.set_item(&*key, value, vm)?; Ok(None) } - bytecode::Instruction::MatchClass(nargs) => { + Instruction::MatchClass(nargs) => { // STACK[-1] is a tuple of keyword attribute names, STACK[-2] is the class being matched against, and STACK[-3] is the match subject. // nargs is the number of positional sub-patterns. let kwd_attrs = self.pop_value(); @@ -1337,7 +1605,7 @@ impl ExecutingFrame<'_> { } Ok(None) } - bytecode::Instruction::MatchKeys => { + Instruction::MatchKeys => { // MATCH_KEYS doesn't pop subject and keys, only reads them let keys_tuple = self.top_value(); // stack[-1] let subject = self.nth_value(1); // stack[-2] @@ -1403,7 +1671,7 @@ impl ExecutingFrame<'_> { } Ok(None) } - bytecode::Instruction::MatchMapping => { + Instruction::MatchMapping => { // Pop and push back the subject to keep it on stack let subject = self.pop_value(); @@ -1414,7 +1682,7 @@ impl ExecutingFrame<'_> { self.push_value(vm.ctx.new_bool(is_mapping).into()); Ok(None) } - bytecode::Instruction::MatchSequence => { + Instruction::MatchSequence => { // Pop and push back the subject to keep it on stack let subject = self.pop_value(); @@ -1425,10 +1693,23 @@ impl ExecutingFrame<'_> { self.push_value(vm.ctx.new_bool(is_sequence).into()); Ok(None) } - bytecode::Instruction::Nop => Ok(None), - // PopBlock is now a pseudo-instruction - exception table handles this - bytecode::Instruction::PopBlock => Ok(None), - bytecode::Instruction::PopException => { + Instruction::Nop => Ok(None), + // NOT_TAKEN is a branch prediction hint - functionally a NOP + Instruction::NotTaken => Ok(None), + // Instrumented version of NOT_TAKEN - NOP without monitoring + Instruction::InstrumentedNotTaken => Ok(None), + // CACHE is used by adaptive interpreter for inline caching - NOP for us + Instruction::Cache => Ok(None), + Instruction::ReturnGenerator => { + // In RustPython, generators/coroutines are created in function.rs + // before the frame starts executing. The RETURN_GENERATOR instruction + // pushes None so that the following POP_TOP has something to consume. + // This matches CPython's semantics where the sent value (None for first call) + // is on the stack when the generator resumes. + self.push_value(vm.ctx.none()); + Ok(None) + } + Instruction::PopExcept => { // Pop prev_exc from value stack and restore it let prev_exc = self.pop_value(); if vm.is_none(&prev_exc) { @@ -1446,19 +1727,44 @@ impl ExecutingFrame<'_> { Ok(None) } - bytecode::Instruction::PopJumpIfFalse { target } => { - self.pop_jump_if(vm, target.get(arg), false) + Instruction::PopJumpIfFalse { target } => self.pop_jump_if(vm, target.get(arg), false), + Instruction::PopJumpIfTrue { target } => self.pop_jump_if(vm, target.get(arg), true), + Instruction::PopJumpIfNone { target } => { + let value = self.pop_value(); + if vm.is_none(&value) { + self.jump(target.get(arg)); + } + Ok(None) } - bytecode::Instruction::PopJumpIfTrue { target } => { - self.pop_jump_if(vm, target.get(arg), true) + Instruction::PopJumpIfNotNone { target } => { + let value = self.pop_value(); + if !vm.is_none(&value) { + self.jump(target.get(arg)); + } + Ok(None) } - bytecode::Instruction::PopTop => { + Instruction::PopTop => { // Pop value from stack and ignore. self.pop_value(); Ok(None) } - bytecode::Instruction::Raise { kind } => self.execute_raise(vm, kind.get(arg)), - bytecode::Instruction::Resume { arg: resume_arg } => { + Instruction::EndFor => { + // Pop the next value from stack (cleanup after loop body) + self.pop_value(); + Ok(None) + } + Instruction::PopIter => { + // Pop the iterator from stack (end of for loop) + self.pop_value(); + Ok(None) + } + Instruction::PushNull => { + // Push NULL for self_or_null slot in call protocol + self.push_null(); + Ok(None) + } + Instruction::RaiseVarargs { kind } => self.execute_raise(vm, kind.get(arg)), + Instruction::Resume { arg: resume_arg } => { // Resume execution after yield, await, or at function start // In CPython, this checks instrumentation and eval breaker // For now, we just check for signals/interrupts @@ -1470,20 +1776,11 @@ impl ExecutingFrame<'_> { // } Ok(None) } - bytecode::Instruction::ReturnConst { idx } => { - let value = self.code.constants[idx.get(arg) as usize].clone().into(); - self.unwind_blocks(vm, UnwindReason::Returning { value }) - } - bytecode::Instruction::ReturnValue => { + Instruction::ReturnValue => { let value = self.pop_value(); self.unwind_blocks(vm, UnwindReason::Returning { value }) } - bytecode::Instruction::Reverse { amount } => { - let stack_len = self.state.stack.len(); - self.state.stack[stack_len - amount.get(arg) as usize..stack_len].reverse(); - Ok(None) - } - bytecode::Instruction::SetAdd { i } => { + Instruction::SetAdd { i } => { let item = self.pop_value(); let obj = self.nth_value(i.get(arg)); let set: &Py<PySet> = unsafe { @@ -1493,16 +1790,20 @@ impl ExecutingFrame<'_> { set.add(item, vm)?; Ok(None) } - bytecode::Instruction::SetExcInfo => { - // Set the current exception to TOS (for except* handlers) - // This updates sys.exc_info() so bare 'raise' will reraise the matched exception - let exc = self.top_value(); - if let Some(exc) = exc.downcast_ref::<PyBaseException>() { - vm.set_exception(Some(exc.to_owned())); + Instruction::SetUpdate { i } => { + let iterable = self.pop_value(); + let obj = self.nth_value(i.get(arg)); + let set: &Py<PySet> = unsafe { + // SAFETY: compiler guarantees correct type + obj.downcast_unchecked_ref() + }; + let iter = PyIter::try_from_object(vm, iterable)?; + while let PyIterReturn::Return(item) = iter.next(vm)? { + set.add(item, vm)?; } Ok(None) } - bytecode::Instruction::PushExcInfo => { + Instruction::PushExcInfo => { // Stack: [exc] -> [prev_exc, exc] let exc = self.pop_value(); let prev_exc = vm @@ -1519,17 +1820,33 @@ impl ExecutingFrame<'_> { self.push_value(exc); Ok(None) } - bytecode::Instruction::CheckExcMatch => { + Instruction::CheckExcMatch => { // Stack: [exc, type] -> [exc, bool] let exc_type = self.pop_value(); let exc = self.top_value(); - // Validate that exc_type is valid for exception matching + // Validate that exc_type inherits from BaseException + if let Some(tuple_of_exceptions) = exc_type.downcast_ref::<PyTuple>() { + for exception in tuple_of_exceptions { + if !exception + .is_subclass(vm.ctx.exceptions.base_exception_type.into(), vm)? + { + return Err(vm.new_type_error( + "catching classes that do not inherit from BaseException is not allowed", + )); + } + } + } else if !exc_type.is_subclass(vm.ctx.exceptions.base_exception_type.into(), vm)? { + return Err(vm.new_type_error( + "catching classes that do not inherit from BaseException is not allowed", + )); + } + let result = exc.is_instance(&exc_type, vm)?; self.push_value(vm.ctx.new_bool(result).into()); Ok(None) } - bytecode::Instruction::Reraise { depth } => { + Instruction::Reraise { depth } => { // inst(RERAISE, (values[oparg], exc -- values[oparg])) // // Stack layout: [values..., exc] where len(values) == oparg @@ -1556,51 +1873,22 @@ impl ExecutingFrame<'_> { Err(exc) } } - bytecode::Instruction::SetFunctionAttribute { attr } => { + Instruction::SetFunctionAttribute { attr } => { self.execute_set_function_attribute(vm, attr.get(arg)) } - bytecode::Instruction::SetupAnnotation => self.setup_annotations(vm), - bytecode::Instruction::BeforeWith => { - // TOS: context_manager - // Result: [..., __exit__, __enter__ result] - let context_manager = self.pop_value(); - let error_string = || -> String { - format!( - "'{:.200}' object does not support the context manager protocol", - context_manager.class().name(), - ) - }; - - // Get __exit__ first (before calling __enter__) - let exit = context_manager - .get_attr(identifier!(vm, __exit__), vm) - .map_err(|_exc| { - vm.new_type_error(format!("{} (missed __exit__ method)", error_string())) - })?; - - // Get and call __enter__ - let enter_res = vm - .get_special_method(&context_manager, identifier!(vm, __enter__))? - .ok_or_else(|| vm.new_type_error(error_string()))? - .invoke((), vm)?; - - // Push __exit__ first, then enter result - self.push_value(exit); - self.push_value(enter_res); - Ok(None) - } - bytecode::Instruction::StoreAttr { idx } => self.store_attr(vm, idx.get(arg)), - bytecode::Instruction::StoreDeref(i) => { + Instruction::SetupAnnotations => self.setup_annotations(vm), + Instruction::StoreAttr { idx } => self.store_attr(vm, idx.get(arg)), + Instruction::StoreDeref(i) => { let value = self.pop_value(); self.cells_frees[i.get(arg) as usize].set(Some(value)); Ok(None) } - bytecode::Instruction::StoreFast(idx) => { + Instruction::StoreFast(idx) => { let value = self.pop_value(); self.fastlocals.lock()[idx.get(arg) as usize] = Some(value); Ok(None) } - bytecode::Instruction::StoreFastLoadFast { + Instruction::StoreFastLoadFast { store_idx, load_idx, } => { @@ -1615,21 +1903,51 @@ impl ExecutingFrame<'_> { self.push_value(load_value); Ok(None) } - bytecode::Instruction::StoreGlobal(idx) => { + Instruction::StoreFastStoreFast { arg: packed } => { + // Store two values to two local variables at once + // STORE_FAST idx1 executes first: pops TOS -> locals[idx1] + // STORE_FAST idx2 executes second: pops new TOS -> locals[idx2] + // oparg encoding: (idx1 << 4) | idx2 + let oparg = packed.get(arg); + let idx1 = (oparg >> 4) as usize; + let idx2 = (oparg & 15) as usize; + let value1 = self.pop_value(); // TOS -> idx1 + let value2 = self.pop_value(); // second -> idx2 + let mut fastlocals = self.fastlocals.lock(); + fastlocals[idx1] = Some(value1); + fastlocals[idx2] = Some(value2); + Ok(None) + } + Instruction::StoreGlobal(idx) => { let value = self.pop_value(); self.globals .set_item(self.code.names[idx.get(arg) as usize], value, vm)?; Ok(None) } - bytecode::Instruction::StoreLocal(idx) => { + Instruction::StoreName(idx) => { let name = self.code.names[idx.get(arg) as usize]; let value = self.pop_value(); self.locals.mapping().ass_subscript(name, Some(value), vm)?; Ok(None) } - bytecode::Instruction::StoreSubscript => self.execute_store_subscript(vm), - bytecode::Instruction::Subscript => self.execute_subscript(vm), - bytecode::Instruction::Swap { index } => { + Instruction::StoreSlice => { + // Stack: [value, container, start, stop] -> [] + let stop = self.pop_value(); + let start = self.pop_value(); + let container = self.pop_value(); + let value = self.pop_value(); + let slice: PyObjectRef = PySlice { + start: Some(start), + stop, + step: None, + } + .into_ref(&vm.ctx) + .into(); + container.set_item(&*slice, value, vm)?; + Ok(None) + } + Instruction::StoreSubscr => self.execute_store_subscript(vm), + Instruction::Swap { index } => { let len = self.state.stack.len(); debug_assert!(len > 0, "stack underflow in SWAP"); let i = len - 1; // TOS index @@ -1646,27 +1964,28 @@ impl ExecutingFrame<'_> { self.state.stack.swap(i, j); Ok(None) } - bytecode::Instruction::ToBool => { + Instruction::ToBool => { let obj = self.pop_value(); let bool_val = obj.try_to_bool(vm)?; self.push_value(vm.ctx.new_bool(bool_val).into()); Ok(None) } - bytecode::Instruction::UnpackEx { args } => { + Instruction::UnpackEx { args } => { let args = args.get(arg); self.execute_unpack_ex(vm, args.before, args.after) } - bytecode::Instruction::UnpackSequence { size } => { - self.unpack_sequence(size.get(arg), vm) - } - bytecode::Instruction::WithExceptStart => { + Instruction::UnpackSequence { size } => self.unpack_sequence(size.get(arg), vm), + Instruction::WithExceptStart => { // Stack: [..., __exit__, lasti, prev_exc, exc] // Call __exit__(type, value, tb) and push result // __exit__ is at TOS-3 (below lasti, prev_exc, and exc) let exc = vm.current_exception(); let stack_len = self.state.stack.len(); - let exit = self.state.stack[stack_len - 4].clone(); + let exit = expect_unchecked( + self.state.stack[stack_len - 4].clone(), + "WithExceptStart: __exit__ is NULL", + ); let args = if let Some(ref exc) = exc { vm.split_exception(exc.clone()) @@ -1679,35 +1998,33 @@ impl ExecutingFrame<'_> { Ok(None) } - bytecode::Instruction::YieldValue { arg: oparg } => { + Instruction::YieldValue { arg: oparg } => { let value = self.pop_value(); // arg=0: direct yield (wrapped for async generators) // arg=1: yield from await/yield-from (NOT wrapped) let wrap = oparg.get(arg) == 0; - let value = if wrap && self.code.flags.contains(bytecode::CodeFlags::IS_COROUTINE) { + let value = if wrap && self.code.flags.contains(bytecode::CodeFlags::COROUTINE) { PyAsyncGenWrappedValue(value).into_pyobject(vm) } else { value }; Ok(Some(ExecutionResult::Yield(value))) } - bytecode::Instruction::Send { target } => { - // Stack: (receiver, value) -> (receiver, retval) - // On StopIteration: replace value with stop value and jump to target + Instruction::Send { target } => { + // (receiver, v -- receiver, retval) + // Pops v, sends it to receiver. On yield, pushes retval + // (so stack = [..., receiver, retval]). On return/StopIteration, + // also pushes retval and jumps to END_SEND which will pop receiver. let exit_label = target.get(arg); let val = self.pop_value(); let receiver = self.top_value(); match self._send(receiver, val, vm)? { PyIterReturn::Return(value) => { - // Value yielded, push it back for YIELD_VALUE - // Stack: (receiver, retval) self.push_value(value); Ok(None) } PyIterReturn::StopIteration(value) => { - // StopIteration: replace top with stop value, jump to exit - // Stack: (receiver, value) - receiver stays, v replaced let value = vm.unwrap_or_none(value); self.push_value(value); self.jump(exit_label); @@ -1715,7 +2032,7 @@ impl ExecutingFrame<'_> { } } } - bytecode::Instruction::EndSend => { + Instruction::EndSend => { // Stack: (receiver, value) -> (value) // Pops receiver, leaves value let value = self.pop_value(); @@ -1723,7 +2040,18 @@ impl ExecutingFrame<'_> { self.push_value(value); Ok(None) } - bytecode::Instruction::CleanupThrow => { + Instruction::ExitInitCheck => { + // Check that __init__ returned None + let should_be_none = self.pop_value(); + if !vm.is_none(&should_be_none) { + return Err(vm.new_type_error(format!( + "__init__() should return None, not '{}'", + should_be_none.class().name() + ))); + } + Ok(None) + } + Instruction::CleanupThrow => { // CLEANUP_THROW: (sub_iter, last_sent_val, exc) -> (None, value) OR re-raise // If StopIteration: pop all 3, extract value, push (None, value) // Otherwise: pop all 3, return Err(exc) for unwind_blocks to handle @@ -1761,68 +2089,26 @@ impl ExecutingFrame<'_> { .map_err(|_| vm.new_type_error("exception expected".to_owned()))?; Err(exc) } - bytecode::Instruction::UnaryInvert => { + Instruction::UnaryInvert => { let a = self.pop_value(); let value = vm._invert(&a)?; self.push_value(value); Ok(None) } - bytecode::Instruction::UnaryNegative => { + Instruction::UnaryNegative => { let a = self.pop_value(); let value = vm._neg(&a)?; self.push_value(value); Ok(None) } - bytecode::Instruction::UnaryNot => { + Instruction::UnaryNot => { let obj = self.pop_value(); let value = obj.try_to_bool(vm)?; self.push_value(vm.ctx.new_bool(!value).into()); Ok(None) } - // Placeholder/dummy instructions - these should never be executed - bytecode::Instruction::Cache - | bytecode::Instruction::Reserved3 - | bytecode::Instruction::Reserved17 - | bytecode::Instruction::BinarySlice - | bytecode::Instruction::EndFor - | bytecode::Instruction::ExitInitCheck - | bytecode::Instruction::InterpreterExit - | bytecode::Instruction::LoadAssertionError - | bytecode::Instruction::LoadLocals - | bytecode::Instruction::PushNull - | bytecode::Instruction::ReturnGenerator - | bytecode::Instruction::StoreSlice - | bytecode::Instruction::BuildConstKeyMap { .. } - | bytecode::Instruction::CopyFreeVars { .. } - | bytecode::Instruction::DictMerge { .. } - | bytecode::Instruction::EnterExecutor { .. } - | bytecode::Instruction::JumpBackward { .. } - | bytecode::Instruction::JumpBackwardNoInterrupt { .. } - | bytecode::Instruction::JumpForward { .. } - | bytecode::Instruction::ListExtend { .. } - | bytecode::Instruction::LoadFastCheck(_) - | bytecode::Instruction::LoadFastLoadFast { .. } - | bytecode::Instruction::LoadFromDictOrDeref(_) - | bytecode::Instruction::LoadFromDictOrGlobals(_) - | bytecode::Instruction::LoadSuperAttr { .. } - | bytecode::Instruction::MakeCell(_) - | bytecode::Instruction::PopJumpIfNone { .. } - | bytecode::Instruction::PopJumpIfNotNone { .. } - | bytecode::Instruction::SetUpdate { .. } - | bytecode::Instruction::StoreFastStoreFast { .. } - | bytecode::Instruction::Reserved140 - | bytecode::Instruction::Reserved141 - | bytecode::Instruction::Reserved142 - | bytecode::Instruction::Reserved143 - | bytecode::Instruction::Reserved144 - | bytecode::Instruction::Reserved145 - | bytecode::Instruction::Reserved146 - | bytecode::Instruction::Reserved147 - | bytecode::Instruction::Reserved148 => { - unreachable!( - "placeholder instruction should not be executed: {:?}", - instruction - ) + _ => { + unreachable!("{instruction:?} instruction should not be executed") } } } @@ -1836,16 +2122,6 @@ impl ExecutingFrame<'_> { }) } - unsafe fn flatten_tuples(&mut self, size: usize) -> Vec<PyObjectRef> { - let mut elements = Vec::new(); - for tup in self.pop_multiple(size) { - // SAFETY: caller ensures that the elements are tuples - let tup = unsafe { tup.downcast_unchecked::<PyTuple>() }; - elements.extend(tup.iter().cloned()); - } - elements - } - #[cfg_attr(feature = "flame-it", flame("Frame"))] fn import(&mut self, vm: &VirtualMachine, module_name: Option<&Py<PyStr>>) -> PyResult<()> { let module_name = module_name.unwrap_or(vm.ctx.empty_str); @@ -1951,7 +2227,7 @@ impl ExecutingFrame<'_> { { // 1. Pop stack to entry.depth while self.state.stack.len() > entry.depth as usize { - self.pop_value(); + self.state.stack.pop(); } // 2. If push_lasti=true (SETUP_CLEANUP), push lasti before exception @@ -2011,22 +2287,9 @@ impl ExecutingFrame<'_> { drop(fastlocals); Ok(Some(ExecutionResult::Return(value))) } - UnwindReason::Break { target } | UnwindReason::Continue { target } => { - // Break/continue: jump to the target label - self.jump(target); - Ok(None) - } } } - fn execute_subscript(&mut self, vm: &VirtualMachine) -> FrameResult { - let b_ref = self.pop_value(); - let a_ref = self.pop_value(); - let value = a_ref.get_item(&*b_ref, vm)?; - self.push_value(value); - Ok(None) - } - fn execute_store_subscript(&mut self, vm: &VirtualMachine) -> FrameResult { let idx = self.pop_value(); let obj = self.pop_value(); @@ -2053,35 +2316,6 @@ impl ExecutingFrame<'_> { Ok(None) } - fn execute_build_map_for_call(&mut self, vm: &VirtualMachine, size: u32) -> FrameResult { - let size = size as usize; - let map_obj = vm.ctx.new_dict(); - for obj in self.pop_multiple(size) { - // Use keys() method for all mapping objects to preserve order - Self::iterate_mapping_keys(vm, &obj, "keyword argument", |key| { - // Check for keyword argument restrictions - if key.downcast_ref::<PyStr>().is_none() { - return Err(vm.new_type_error("keywords must be strings")); - } - if map_obj.contains_key(&*key, vm) { - let key_repr = &key.repr(vm)?; - let msg = format!( - "got multiple values for keyword argument {}", - key_repr.as_str() - ); - return Err(vm.new_type_error(msg)); - } - - let value = obj.get_item(&*key, vm)?; - map_obj.set_item(&*key, value, vm)?; - Ok(()) - })?; - } - - self.push_value(map_obj.into()); - Ok(None) - } - fn execute_build_slice( &mut self, vm: &VirtualMachine, @@ -2125,9 +2359,9 @@ impl ExecutingFrame<'_> { FuncArgs::with_kwargs_names(args, kwarg_names) } - fn collect_ex_args(&mut self, vm: &VirtualMachine, has_kwargs: bool) -> PyResult<FuncArgs> { - let kwargs = if has_kwargs { - let kw_obj = self.pop_value(); + fn collect_ex_args(&mut self, vm: &VirtualMachine) -> PyResult<FuncArgs> { + let kwargs_or_null = self.pop_value_opt(); + let kwargs = if let Some(kw_obj) = kwargs_or_null { let mut kwargs = IndexMap::new(); // Use keys() method for all mapping objects to preserve order @@ -2174,31 +2408,25 @@ impl ExecutingFrame<'_> { #[inline] fn execute_call(&mut self, args: FuncArgs, vm: &VirtualMachine) -> FrameResult { - let func_ref = self.pop_value(); - let value = func_ref.call(args, vm)?; - self.push_value(value); - Ok(None) - } - - #[inline] - fn execute_method_call(&mut self, args: FuncArgs, vm: &VirtualMachine) -> FrameResult { - let func = self.pop_value(); - let is_method = self.pop_value().is(&vm.ctx.true_value); - let target = self.pop_value(); - - // TODO: It was PyMethod before #4873. Check if it's correct. - let func = if is_method { - if let Some(descr_get) = func.class().slots.descr_get.load() { - let cls = target.class().to_owned().into(); - descr_get(func, Some(target), Some(cls), vm)? - } else { - func + // Stack: [callable, self_or_null, ...] + let self_or_null = self.pop_value_opt(); // Option<PyObjectRef> + let callable = self.pop_value(); + + // If self_or_null is Some (not NULL), prepend it to args + let final_args = if let Some(self_val) = self_or_null { + // Method call: prepend self to args + let mut all_args = vec![self_val]; + all_args.extend(args.args); + FuncArgs { + args: all_args, + kwargs: args.kwargs, } } else { - drop(target); // should be None - func + // Regular attribute call: self_or_null is NULL + args }; - let value = func.call(args, vm)?; + + let value = callable.call(final_args, vm)?; self.push_value(value); Ok(None) } @@ -2295,14 +2523,16 @@ impl ExecutingFrame<'_> { // Elements on stack from right-to-left: self.state .stack - .extend(elements.drain(before + middle..).rev()); + .extend(elements.drain(before + middle..).rev().map(Some)); let middle_elements = elements.drain(before..).collect(); let t = vm.ctx.new_list(middle_elements); self.push_value(t.into()); // Lastly the first reversed values: - self.state.stack.extend(elements.into_iter().rev()); + self.state + .stack + .extend(elements.into_iter().rev().map(Some)); Ok(None) } @@ -2329,23 +2559,6 @@ impl ExecutingFrame<'_> { Ok(None) } - #[inline] - fn jump_if_or_pop( - &mut self, - vm: &VirtualMachine, - target: bytecode::Label, - flag: bool, - ) -> FrameResult { - let obj = self.top_value(); - let value = obj.to_owned().try_to_bool(vm)?; - if value == flag { - self.jump(target); - } else { - self.pop_value(); - } - Ok(None) - } - /// The top of stack contains the iterator, lets push it forward fn execute_for_iter(&mut self, vm: &VirtualMachine, target: bytecode::Label) -> FrameResult { let top_of_stack = PyIter::new(self.top_value()); @@ -2358,15 +2571,30 @@ impl ExecutingFrame<'_> { Ok(None) } Ok(PyIterReturn::StopIteration(_)) => { - // Pop iterator from stack: - self.pop_value(); - - // End of for loop - self.jump(target); + // Check if target instruction is END_FOR (CPython 3.14 pattern) + // If so, skip it and jump to target + 1 instruction (POP_ITER) + let target_idx = target.0 as usize; + let jump_target = if let Some(unit) = self.code.instructions.get(target_idx) { + if matches!(unit.op, bytecode::Instruction::EndFor) + && matches!( + self.code.instructions.get(target_idx + 1).map(|u| &u.op), + Some(bytecode::Instruction::PopIter) + ) + { + // Skip END_FOR, jump to POP_ITER + bytecode::Label(target.0 + 1) + } else { + // Legacy pattern: jump directly to target (POP_TOP/POP_ITER) + target + } + } else { + target + }; + self.jump(jump_target); Ok(None) } Err(next_error) => { - // Pop iterator from stack: + // On error, pop iterator and propagate self.pop_value(); Err(next_error) } @@ -2395,8 +2623,8 @@ impl ExecutingFrame<'_> { // Stack: [..., attr_value, func] -> [..., func] // Stack order: func is at -1, attr_value is at -2 - let func = self.pop_value(); - let attr_value = self.replace_top(func); + let func = self.pop_value_opt(); + let attr_value = expect_unchecked(self.replace_top(func), "attr_value must not be null"); let func = self.top_value(); // Get the function reference and call the new method @@ -2448,6 +2676,7 @@ impl ExecutingFrame<'_> { bytecode::BinaryOperator::InplaceXor => vm._ixor(a_ref, b_ref), bytecode::BinaryOperator::InplaceOr => vm._ior(a_ref, b_ref), bytecode::BinaryOperator::InplaceAnd => vm._iand(a_ref, b_ref), + bytecode::BinaryOperator::Subscr => a_ref.get_item(b_ref.as_object(), vm), }?; self.push_value(value); @@ -2492,7 +2721,10 @@ impl ExecutingFrame<'_> { })?; let msg = match elements.len().cmp(&(size as usize)) { core::cmp::Ordering::Equal => { - self.state.stack.extend(elements.into_iter().rev()); + // Wrap each element in Some() for Option<PyObjectRef> stack + self.state + .stack + .extend(elements.into_iter().rev().map(Some)); return Ok(None); } core::cmp::Ordering::Greater => { @@ -2553,11 +2785,68 @@ impl ExecutingFrame<'_> { Ok(None) } - fn load_attr(&mut self, vm: &VirtualMachine, attr: bytecode::NameIdx) -> FrameResult { - let attr_name = self.code.names[attr as usize]; + fn load_attr(&mut self, vm: &VirtualMachine, oparg: u32) -> FrameResult { + let (name_idx, is_method) = bytecode::decode_load_attr_arg(oparg); + let attr_name = self.code.names[name_idx as usize]; let parent = self.pop_value(); - let obj = parent.get_attr(attr_name, vm)?; - self.push_value(obj); + + if is_method { + // Method call: push [method, self_or_null] + let method = PyMethod::get(parent.clone(), attr_name, vm)?; + match method { + PyMethod::Function { target: _, func } => { + self.push_value(func); + self.push_value(parent); + } + PyMethod::Attribute(val) => { + self.push_value(val); + self.push_null(); + } + } + } else { + // Regular attribute access + let obj = parent.get_attr(attr_name, vm)?; + self.push_value(obj); + } + Ok(None) + } + + fn load_super_attr(&mut self, vm: &VirtualMachine, oparg: u32) -> FrameResult { + let (name_idx, load_method, has_class) = bytecode::decode_load_super_attr_arg(oparg); + let attr_name = self.code.names[name_idx as usize]; + + // Stack layout (bottom to top): [super, class, self] + // Pop in LIFO order: self, class, super + let self_obj = self.pop_value(); + let class = self.pop_value(); + let global_super = self.pop_value(); + + // Create super object - pass args based on has_class flag + // When super is shadowed, has_class=false means call with 0 args + let super_obj = if has_class { + global_super.call((class.clone(), self_obj.clone()), vm)? + } else { + global_super.call((), vm)? + }; + + if load_method { + // Method load: push [method, self_or_null] + let method = PyMethod::get(super_obj, attr_name, vm)?; + match method { + PyMethod::Function { target: _, func } => { + self.push_value(func); + self.push_value(self_obj); + } + PyMethod::Attribute(val) => { + self.push_value(val); + self.push_null(); + } + } + } else { + // Regular attribute access + let obj = super_obj.get_attr(attr_name, vm)?; + self.push_value(obj); + } Ok(None) } @@ -2579,13 +2868,8 @@ impl ExecutingFrame<'_> { // Block stack functions removed - exception table handles all exception/cleanup #[inline] - #[track_caller] // not a real track_caller but push_value is not very useful - fn push_value(&mut self, obj: PyObjectRef) { - // eprintln!( - // "push_value {} / len: {} +1", - // obj.class().name(), - // self.state.stack.len() - // ); + #[track_caller] // not a real track_caller but push_value is less useful for debugging + fn push_value_opt(&mut self, obj: Option<PyObjectRef>) { match self.state.stack.try_push(obj) { Ok(()) => {} Err(_e) => self.fatal("tried to push value onto stack but overflowed max_stackdepth"), @@ -2593,21 +2877,34 @@ impl ExecutingFrame<'_> { } #[inline] - #[track_caller] // not a real track_caller but pop_value is not very useful - fn pop_value(&mut self) -> PyObjectRef { + #[track_caller] + fn push_value(&mut self, obj: PyObjectRef) { + self.push_value_opt(Some(obj)); + } + + #[inline] + fn push_null(&mut self) { + self.push_value_opt(None); + } + + /// Pop a value from the stack, returning None if the stack slot is NULL + #[inline] + fn pop_value_opt(&mut self) -> Option<PyObjectRef> { match self.state.stack.pop() { - Some(x) => { - // eprintln!( - // "pop_value {} / len: {}", - // x.class().name(), - // self.state.stack.len() - // ); - x - } - None => self.fatal("tried to pop value but there was nothing on the stack"), + Some(slot) => slot, // slot is Option<PyObjectRef> + None => self.fatal("tried to pop from empty stack"), } } + #[inline] + #[track_caller] + fn pop_value(&mut self) -> PyObjectRef { + expect_unchecked( + self.pop_value_opt(), + "pop value but null found. This is a compiler bug.", + ) + } + fn call_intrinsic_1( &mut self, func: bytecode::IntrinsicFunction1, @@ -2690,6 +2987,30 @@ impl ExecutingFrame<'_> { .map_err(|_| vm.new_type_error("LIST_TO_TUPLE expects a list"))?; Ok(vm.ctx.new_tuple(list.borrow_vec().to_vec()).into()) } + bytecode::IntrinsicFunction1::StopIterationError => { + // Convert StopIteration to RuntimeError + // Used to ensure async generators don't raise StopIteration directly + // _PyGen_FetchStopIterationValue + // Use fast_isinstance to handle subclasses of StopIteration + if arg.fast_isinstance(vm.ctx.exceptions.stop_iteration) { + Err(vm.new_runtime_error("coroutine raised StopIteration")) + } else { + // If not StopIteration, just re-raise the original exception + Err(arg.downcast().unwrap_or_else(|obj| { + vm.new_runtime_error(format!( + "unexpected exception type: {:?}", + obj.class() + )) + })) + } + } + bytecode::IntrinsicFunction1::AsyncGenWrap => { + // Wrap value for async generator + // Creates an AsyncGenWrappedValue + Ok(crate::builtins::asyncgenerator::PyAsyncGenWrappedValue(arg) + .into_ref(&vm.ctx) + .into()) + } } } @@ -2733,7 +3054,8 @@ impl ExecutingFrame<'_> { } } - fn pop_multiple(&mut self, count: usize) -> crate::common::boxvec::Drain<'_, PyObjectRef> { + /// Pop multiple values from the stack. Panics if any slot is NULL. + fn pop_multiple(&mut self, count: usize) -> impl ExactSizeIterator<Item = PyObjectRef> + '_ { let stack_len = self.state.stack.len(); if count > stack_len { let instr = self.code.instructions.get(self.lasti() as usize); @@ -2750,21 +3072,24 @@ impl ExecutingFrame<'_> { self.code.source_path ); } - self.state.stack.drain(stack_len - count..) + self.state.stack.drain(stack_len - count..).map(|obj| { + expect_unchecked(obj, "pop_multiple but null found. This is a compiler bug.") + }) } #[inline] - fn replace_top(&mut self, mut top: PyObjectRef) -> PyObjectRef { + fn replace_top(&mut self, mut top: Option<PyObjectRef>) -> Option<PyObjectRef> { let last = self.state.stack.last_mut().unwrap(); - core::mem::swap(&mut top, last); + core::mem::swap(last, &mut top); top } #[inline] - #[track_caller] // not a real track_caller but top_value is not very useful + #[track_caller] fn top_value(&self) -> &PyObject { match &*self.state.stack { - [.., last] => last, + [.., Some(last)] => last, + [.., None] => self.fatal("tried to get top of stack but got NULL"), [] => self.fatal("tried to get top of stack but stack is empty"), } } @@ -2773,7 +3098,10 @@ impl ExecutingFrame<'_> { #[track_caller] fn nth_value(&self, depth: u32) -> &PyObject { let stack = &self.state.stack; - &stack[stack.len() - depth as usize - 1] + match &stack[stack.len() - depth as usize - 1] { + Some(obj) => obj, + None => unsafe { std::hint::unreachable_unchecked() }, + } } #[cold] @@ -2788,11 +3116,17 @@ impl ExecutingFrame<'_> { impl fmt::Debug for Frame { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { let state = self.state.lock(); - let stack_str = state.stack.iter().fold(String::new(), |mut s, elem| { - if elem.downcastable::<Self>() { - s.push_str("\n > {frame}"); - } else { - core::fmt::write(&mut s, format_args!("\n > {elem:?}")).unwrap(); + let stack_str = state.stack.iter().fold(String::new(), |mut s, slot| { + match slot { + Some(elem) if elem.downcastable::<Self>() => { + s.push_str("\n > {frame}"); + } + Some(elem) => { + core::fmt::write(&mut s, format_args!("\n > {elem:?}")).unwrap(); + } + None => { + s.push_str("\n > NULL"); + } } s }); @@ -2819,3 +3153,11 @@ fn is_module_initializing(module: &PyObject, vm: &VirtualMachine) -> bool { }; initializing_attr.try_to_bool(vm).unwrap_or(false) } + +fn expect_unchecked(optional: Option<PyObjectRef>, err_msg: &'static str) -> PyObjectRef { + if cfg!(debug_assertions) { + optional.expect(err_msg) + } else { + unsafe { optional.unwrap_unchecked() } + } +} diff --git a/crates/vm/src/gc_state.rs b/crates/vm/src/gc_state.rs new file mode 100644 index 00000000000..b4f9165ea17 --- /dev/null +++ b/crates/vm/src/gc_state.rs @@ -0,0 +1,482 @@ +//! Garbage Collection State and Algorithm +//! +//! This module implements CPython-compatible generational garbage collection +//! for RustPython, using an intrusive doubly-linked list approach. + +use crate::common::lock::PyMutex; +use crate::{PyObject, PyObjectRef}; +use core::ptr::NonNull; +use core::sync::atomic::{AtomicBool, AtomicU32, AtomicUsize, Ordering}; +use std::collections::HashSet; +use std::sync::{Mutex, RwLock}; + +bitflags::bitflags! { + /// GC debug flags (see Include/internal/pycore_gc.h) + #[derive(Copy, Clone, Debug, Default, PartialEq, Eq)] + pub struct GcDebugFlags: u32 { + /// Print collection statistics + const STATS = 1 << 0; + /// Print collectable objects + const COLLECTABLE = 1 << 1; + /// Print uncollectable objects + const UNCOLLECTABLE = 1 << 2; + /// Save all garbage in gc.garbage + const SAVEALL = 1 << 5; + /// DEBUG_COLLECTABLE | DEBUG_UNCOLLECTABLE | DEBUG_SAVEALL + const LEAK = Self::COLLECTABLE.bits() | Self::UNCOLLECTABLE.bits() | Self::SAVEALL.bits(); + } +} + +/// Statistics for a single generation (gc_generation_stats) +#[derive(Debug, Default, Clone, Copy)] +pub struct GcStats { + pub collections: usize, + pub collected: usize, + pub uncollectable: usize, +} + +/// A single GC generation with intrusive linked list +pub struct GcGeneration { + /// Number of objects in this generation + count: AtomicUsize, + /// Threshold for triggering collection + threshold: AtomicU32, + /// Collection statistics + stats: PyMutex<GcStats>, +} + +impl GcGeneration { + pub const fn new(threshold: u32) -> Self { + Self { + count: AtomicUsize::new(0), + threshold: AtomicU32::new(threshold), + stats: PyMutex::new(GcStats { + collections: 0, + collected: 0, + uncollectable: 0, + }), + } + } + + pub fn count(&self) -> usize { + self.count.load(Ordering::SeqCst) + } + + pub fn threshold(&self) -> u32 { + self.threshold.load(Ordering::SeqCst) + } + + pub fn set_threshold(&self, value: u32) { + self.threshold.store(value, Ordering::SeqCst); + } + + pub fn stats(&self) -> GcStats { + let guard = self.stats.lock(); + GcStats { + collections: guard.collections, + collected: guard.collected, + uncollectable: guard.uncollectable, + } + } + + pub fn update_stats(&self, collected: usize, uncollectable: usize) { + let mut guard = self.stats.lock(); + guard.collections += 1; + guard.collected += collected; + guard.uncollectable += uncollectable; + } +} + +/// Wrapper for raw pointer to make it Send + Sync +#[derive(Clone, Copy, PartialEq, Eq, Hash)] +struct GcObjectPtr(NonNull<PyObject>); + +// SAFETY: We only use this for tracking objects, and proper synchronization is used +unsafe impl Send for GcObjectPtr {} +unsafe impl Sync for GcObjectPtr {} + +/// Global GC state +pub struct GcState { + /// 3 generations (0 = youngest, 2 = oldest) + pub generations: [GcGeneration; 3], + /// Permanent generation (frozen objects) + pub permanent: GcGeneration, + /// GC enabled flag + pub enabled: AtomicBool, + /// Per-generation object tracking (for correct gc_refs algorithm) + /// Objects start in gen0, survivors move to gen1, then gen2 + generation_objects: [RwLock<HashSet<GcObjectPtr>>; 3], + /// Frozen/permanent objects (excluded from normal GC) + permanent_objects: RwLock<HashSet<GcObjectPtr>>, + /// Debug flags + pub debug: AtomicU32, + /// gc.garbage list (uncollectable objects with __del__) + pub garbage: PyMutex<Vec<PyObjectRef>>, + /// gc.callbacks list + pub callbacks: PyMutex<Vec<PyObjectRef>>, + /// Mutex for collection (prevents concurrent collections). + /// Used by collect_inner when the actual collection algorithm is enabled. + #[allow(dead_code)] + collecting: Mutex<()>, + /// Allocation counter for gen0 + alloc_count: AtomicUsize, + /// Registry of all tracked objects (for cycle detection) + tracked_objects: RwLock<HashSet<GcObjectPtr>>, + /// Objects that have been finalized (__del__ already called) + /// Prevents calling __del__ multiple times on resurrected objects + finalized_objects: RwLock<HashSet<GcObjectPtr>>, +} + +// SAFETY: All fields are either inherently Send/Sync (atomics, RwLock, Mutex) or protected by PyMutex. +// PyMutex<Vec<PyObjectRef>> is safe to share/send across threads because access is synchronized. +// PyObjectRef itself is Send, and interior mutability is guarded by the mutex. +unsafe impl Send for GcState {} +unsafe impl Sync for GcState {} + +impl Default for GcState { + fn default() -> Self { + Self::new() + } +} + +impl GcState { + pub fn new() -> Self { + Self { + generations: [ + GcGeneration::new(2000), // young + GcGeneration::new(10), // old[0] + GcGeneration::new(0), // old[1] + ], + permanent: GcGeneration::new(0), + enabled: AtomicBool::new(true), + generation_objects: [ + RwLock::new(HashSet::new()), + RwLock::new(HashSet::new()), + RwLock::new(HashSet::new()), + ], + permanent_objects: RwLock::new(HashSet::new()), + debug: AtomicU32::new(0), + garbage: PyMutex::new(Vec::new()), + callbacks: PyMutex::new(Vec::new()), + collecting: Mutex::new(()), + alloc_count: AtomicUsize::new(0), + tracked_objects: RwLock::new(HashSet::new()), + finalized_objects: RwLock::new(HashSet::new()), + } + } + + /// Check if GC is enabled + pub fn is_enabled(&self) -> bool { + self.enabled.load(Ordering::SeqCst) + } + + /// Enable GC + pub fn enable(&self) { + self.enabled.store(true, Ordering::SeqCst); + } + + /// Disable GC + pub fn disable(&self) { + self.enabled.store(false, Ordering::SeqCst); + } + + /// Get debug flags + pub fn get_debug(&self) -> GcDebugFlags { + GcDebugFlags::from_bits_truncate(self.debug.load(Ordering::SeqCst)) + } + + /// Set debug flags + pub fn set_debug(&self, flags: GcDebugFlags) { + self.debug.store(flags.bits(), Ordering::SeqCst); + } + + /// Get thresholds for all generations + pub fn get_threshold(&self) -> (u32, u32, u32) { + ( + self.generations[0].threshold(), + self.generations[1].threshold(), + self.generations[2].threshold(), + ) + } + + /// Set thresholds + pub fn set_threshold(&self, t0: u32, t1: Option<u32>, t2: Option<u32>) { + self.generations[0].set_threshold(t0); + if let Some(t1) = t1 { + self.generations[1].set_threshold(t1); + } + if let Some(t2) = t2 { + self.generations[2].set_threshold(t2); + } + } + + /// Get counts for all generations + pub fn get_count(&self) -> (usize, usize, usize) { + ( + self.generations[0].count(), + self.generations[1].count(), + self.generations[2].count(), + ) + } + + /// Get statistics for all generations + pub fn get_stats(&self) -> [GcStats; 3] { + [ + self.generations[0].stats(), + self.generations[1].stats(), + self.generations[2].stats(), + ] + } + + /// Track a new object (add to gen0) + /// Called when IS_TRACE objects are created + /// + /// # Safety + /// obj must be a valid pointer to a PyObject + pub unsafe fn track_object(&self, obj: NonNull<PyObject>) { + let gc_ptr = GcObjectPtr(obj); + + // Add to generation 0 tracking first (for correct gc_refs algorithm) + // Only increment count if we successfully add to the set + if let Ok(mut gen0) = self.generation_objects[0].write() + && gen0.insert(gc_ptr) + { + self.generations[0].count.fetch_add(1, Ordering::SeqCst); + self.alloc_count.fetch_add(1, Ordering::SeqCst); + } + + // Also add to global tracking (for get_objects, etc.) + if let Ok(mut tracked) = self.tracked_objects.write() { + tracked.insert(gc_ptr); + } + } + + /// Untrack an object (remove from GC lists) + /// Called when objects are deallocated + /// + /// # Safety + /// obj must be a valid pointer to a PyObject + pub unsafe fn untrack_object(&self, obj: NonNull<PyObject>) { + let gc_ptr = GcObjectPtr(obj); + + // Remove from generation tracking lists and decrement the correct generation's count + for (gen_idx, generation) in self.generation_objects.iter().enumerate() { + if let Ok(mut gen_set) = generation.write() + && gen_set.remove(&gc_ptr) + { + // Decrement count for the generation we removed from + let count = self.generations[gen_idx].count.load(Ordering::SeqCst); + if count > 0 { + self.generations[gen_idx] + .count + .fetch_sub(1, Ordering::SeqCst); + } + break; // Object can only be in one generation + } + } + + // Remove from global tracking + if let Ok(mut tracked) = self.tracked_objects.write() { + tracked.remove(&gc_ptr); + } + + // Remove from permanent tracking + if let Ok(mut permanent) = self.permanent_objects.write() + && permanent.remove(&gc_ptr) + { + let count = self.permanent.count.load(Ordering::SeqCst); + if count > 0 { + self.permanent.count.fetch_sub(1, Ordering::SeqCst); + } + } + + // Remove from finalized set + if let Ok(mut finalized) = self.finalized_objects.write() { + finalized.remove(&gc_ptr); + } + } + + /// Check if an object has been finalized + pub fn is_finalized(&self, obj: NonNull<PyObject>) -> bool { + let gc_ptr = GcObjectPtr(obj); + if let Ok(finalized) = self.finalized_objects.read() { + finalized.contains(&gc_ptr) + } else { + false + } + } + + /// Mark an object as finalized + pub fn mark_finalized(&self, obj: NonNull<PyObject>) { + let gc_ptr = GcObjectPtr(obj); + if let Ok(mut finalized) = self.finalized_objects.write() { + finalized.insert(gc_ptr); + } + } + + /// Get tracked objects (for gc.get_objects) + /// If generation is None, returns all tracked objects. + /// If generation is Some(n), returns objects in generation n only. + pub fn get_objects(&self, generation: Option<i32>) -> Vec<PyObjectRef> { + match generation { + None => { + // Return all tracked objects + if let Ok(tracked) = self.tracked_objects.read() { + tracked + .iter() + .filter_map(|ptr| { + let obj = unsafe { ptr.0.as_ref() }; + if obj.strong_count() > 0 { + Some(obj.to_owned()) + } else { + None + } + }) + .collect() + } else { + Vec::new() + } + } + Some(g) if (0..=2).contains(&g) => { + // Return objects in specific generation + let gen_idx = g as usize; + if let Ok(gen_set) = self.generation_objects[gen_idx].read() { + gen_set + .iter() + .filter_map(|ptr| { + let obj = unsafe { ptr.0.as_ref() }; + if obj.strong_count() > 0 { + Some(obj.to_owned()) + } else { + None + } + }) + .collect() + } else { + Vec::new() + } + } + _ => Vec::new(), + } + } + + /// Check if automatic GC should run and run it if needed. + /// Called after object allocation. + /// Currently a stub — returns false. + pub fn maybe_collect(&self) -> bool { + false + } + + /// Perform garbage collection on the given generation. + /// Returns (collected_count, uncollectable_count). + /// + /// Currently a stub — the actual collection algorithm requires EBR + /// and will be added in a follow-up. + pub fn collect(&self, _generation: usize) -> (usize, usize) { + (0, 0) + } + + /// Force collection even if GC is disabled (for manual gc.collect() calls). + /// Currently a stub. + pub fn collect_force(&self, _generation: usize) -> (usize, usize) { + (0, 0) + } + + /// Get count of frozen objects + pub fn get_freeze_count(&self) -> usize { + self.permanent.count() + } + + /// Freeze all tracked objects (move to permanent generation) + pub fn freeze(&self) { + // Move all objects from gen0-2 to permanent + let mut objects_to_freeze: Vec<GcObjectPtr> = Vec::new(); + + for (gen_idx, generation) in self.generation_objects.iter().enumerate() { + if let Ok(mut gen_set) = generation.write() { + objects_to_freeze.extend(gen_set.drain()); + self.generations[gen_idx].count.store(0, Ordering::SeqCst); + } + } + + // Add to permanent set + if let Ok(mut permanent) = self.permanent_objects.write() { + let count = objects_to_freeze.len(); + for ptr in objects_to_freeze { + permanent.insert(ptr); + } + self.permanent.count.fetch_add(count, Ordering::SeqCst); + } + } + + /// Unfreeze all objects (move from permanent to gen2) + pub fn unfreeze(&self) { + let mut objects_to_unfreeze: Vec<GcObjectPtr> = Vec::new(); + + if let Ok(mut permanent) = self.permanent_objects.write() { + objects_to_unfreeze.extend(permanent.drain()); + self.permanent.count.store(0, Ordering::SeqCst); + } + + // Add to generation 2 + if let Ok(mut gen2) = self.generation_objects[2].write() { + let count = objects_to_unfreeze.len(); + for ptr in objects_to_unfreeze { + gen2.insert(ptr); + } + self.generations[2].count.fetch_add(count, Ordering::SeqCst); + } + } +} + +use std::sync::OnceLock; + +/// Global GC state instance +/// Using a static because GC needs to be accessible from object allocation/deallocation +static GC_STATE: OnceLock<GcState> = OnceLock::new(); + +/// Get a reference to the global GC state +pub fn gc_state() -> &'static GcState { + GC_STATE.get_or_init(GcState::new) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_gc_state_default() { + let state = GcState::new(); + assert!(state.is_enabled()); + assert_eq!(state.get_debug(), GcDebugFlags::empty()); + assert_eq!(state.get_threshold(), (2000, 10, 0)); + assert_eq!(state.get_count(), (0, 0, 0)); + } + + #[test] + fn test_gc_enable_disable() { + let state = GcState::new(); + assert!(state.is_enabled()); + state.disable(); + assert!(!state.is_enabled()); + state.enable(); + assert!(state.is_enabled()); + } + + #[test] + fn test_gc_threshold() { + let state = GcState::new(); + state.set_threshold(100, Some(20), Some(30)); + assert_eq!(state.get_threshold(), (100, 20, 30)); + } + + #[test] + fn test_gc_debug_flags() { + let state = GcState::new(); + state.set_debug(GcDebugFlags::STATS | GcDebugFlags::COLLECTABLE); + assert_eq!( + state.get_debug(), + GcDebugFlags::STATS | GcDebugFlags::COLLECTABLE + ); + } +} diff --git a/crates/vm/src/getpath.rs b/crates/vm/src/getpath.rs index 011d5336873..31fa0617b45 100644 --- a/crates/vm/src/getpath.rs +++ b/crates/vm/src/getpath.rs @@ -174,6 +174,9 @@ pub fn init_path_config(settings: &Settings) -> Paths { paths.module_search_paths = build_module_search_paths(settings, &paths.prefix, &paths.exec_prefix); + // Step 9: Calculate stdlib_dir + paths.stdlib_dir = calculate_stdlib_dir(&paths.prefix); + paths } @@ -301,6 +304,22 @@ fn calculate_base_executable(executable: Option<&PathBuf>, home_dir: &Option<Pat .unwrap_or_default() } +/// Calculate stdlib_dir (sys._stdlib_dir) +/// Returns None if the stdlib directory doesn't exist +fn calculate_stdlib_dir(prefix: &str) -> Option<String> { + #[cfg(not(windows))] + let stdlib_dir = PathBuf::from(prefix).join(platform::stdlib_subdir()); + + #[cfg(windows)] + let stdlib_dir = PathBuf::from(prefix).join(platform::STDLIB_SUBDIR); + + if stdlib_dir.is_dir() { + Some(stdlib_dir.to_string_lossy().into_owned()) + } else { + None + } +} + /// Build the complete module_search_paths (sys.path) fn build_module_search_paths(settings: &Settings, prefix: &str, exec_prefix: &str) -> Vec<String> { let mut paths = Vec::new(); diff --git a/crates/vm/src/import.rs b/crates/vm/src/import.rs index 80ccbe30def..d1b77407ef8 100644 --- a/crates/vm/src/import.rs +++ b/crates/vm/src/import.rs @@ -87,16 +87,36 @@ pub fn import_frozen(vm: &VirtualMachine, module_name: &str) -> PyResult { } pub fn import_builtin(vm: &VirtualMachine, module_name: &str) -> PyResult { - let make_module_func = vm.state.module_inits.get(module_name).ok_or_else(|| { - vm.new_import_error( - format!("Cannot import builtin module {module_name}"), - vm.ctx.new_str(module_name), - ) - })?; - let module = make_module_func(vm); let sys_modules = vm.sys_module.get_attr("modules", vm)?; - sys_modules.set_item(module_name, module.as_object().to_owned(), vm)?; - Ok(module.into()) + + // Check if already in sys.modules (handles recursive imports) + if let Ok(module) = sys_modules.get_item(module_name, vm) { + return Ok(module); + } + + // Try multi-phase init first (preferred for modules that import other modules) + if let Some(&def) = vm.state.module_defs.get(module_name) { + // Phase 1: Create and initialize module + let module = def.create_module(vm)?; + + // Add to sys.modules BEFORE exec (critical for circular import handling) + sys_modules.set_item(module_name, module.clone().into(), vm)?; + + // Phase 2: Call exec slot (can safely import other modules now) + // If exec fails, remove the partially-initialized module from sys.modules + if let Err(e) = def.exec_module(vm, &module) { + let _ = sys_modules.del_item(module_name, vm); + return Err(e); + } + + return Ok(module.into()); + } + + // Module not found in module_defs + Err(vm.new_import_error( + format!("Cannot import builtin module {module_name}"), + vm.ctx.new_str(module_name), + )) } #[cfg(feature = "rustpython-compiler")] diff --git a/crates/vm/src/lib.rs b/crates/vm/src/lib.rs index 3f0eee278a2..a458bc0cbc0 100644 --- a/crates/vm/src/lib.rs +++ b/crates/vm/src/lib.rs @@ -77,6 +77,7 @@ pub mod py_io; #[cfg(feature = "serde")] pub mod py_serde; +pub mod gc_state; pub mod readline; pub mod recursion; pub mod scope; @@ -99,7 +100,7 @@ pub use self::object::{ AsObject, Py, PyAtomicRef, PyExact, PyObject, PyObjectRef, PyPayload, PyRef, PyRefExact, PyResult, PyWeakRef, }; -pub use self::vm::{Context, Interpreter, Settings, VirtualMachine}; +pub use self::vm::{Context, Interpreter, InterpreterBuilder, Settings, VirtualMachine}; pub use rustpython_common as common; pub use rustpython_compiler_core::{bytecode, frozen}; diff --git a/crates/vm/src/object/core.rs b/crates/vm/src/object/core.rs index e904117b06a..99081b8b540 100644 --- a/crates/vm/src/object/core.rs +++ b/crates/vm/src/object/core.rs @@ -19,9 +19,9 @@ use crate::object::traverse_object::PyObjVTable; use crate::{ builtins::{PyDictRef, PyType, PyTypeRef}, common::{ - atomic::{OncePtr, PyAtomic, Radium}, - linked_list::{Link, LinkedList, Pointers}, - lock::{PyMutex, PyMutexGuard, PyRwLock}, + atomic::{Ordering, PyAtomic, Radium}, + linked_list::{Link, Pointers}, + lock::PyRwLock, refcount::RefCount, }, vm::VirtualMachine, @@ -81,8 +81,14 @@ use core::{ #[derive(Debug)] pub(super) struct Erased; -pub(super) unsafe fn drop_dealloc_obj<T: PyPayload>(x: *mut PyObject) { - drop(unsafe { Box::from_raw(x as *mut PyInner<T>) }); +/// Default dealloc: handles __del__, weakref clearing, and memory free. +/// Equivalent to subtype_dealloc in CPython. +pub(super) unsafe fn default_dealloc<T: PyPayload>(obj: *mut PyObject) { + let obj_ref = unsafe { &*(obj as *const PyObject) }; + if let Err(()) = obj_ref.drop_slow_inner() { + return; // resurrected by __del__ + } + drop(unsafe { Box::from_raw(obj as *mut PyInner<T>) }); } pub(super) unsafe fn debug_obj<T: PyPayload + core::fmt::Debug>( x: &PyObject, @@ -93,21 +99,46 @@ pub(super) unsafe fn debug_obj<T: PyPayload + core::fmt::Debug>( } /// Call `try_trace` on payload -pub(super) unsafe fn try_trace_obj<T: PyPayload>(x: &PyObject, tracer_fn: &mut TraverseFn<'_>) { +pub(super) unsafe fn try_traverse_obj<T: PyPayload>(x: &PyObject, tracer_fn: &mut TraverseFn<'_>) { let x = unsafe { &*(x as *const PyObject as *const PyInner<T>) }; let payload = &x.payload; payload.try_traverse(tracer_fn) } +bitflags::bitflags! { + /// GC bits for free-threading support (like ob_gc_bits in Py_GIL_DISABLED) + /// These bits are stored in a separate atomic field for lock-free access. + /// See Include/internal/pycore_gc.h + #[derive(Copy, Clone, Debug, Default)] + pub(crate) struct GcBits: u8 { + /// Tracked by the GC + const TRACKED = 1 << 0; + /// tp_finalize was called (prevents __del__ from being called twice) + const FINALIZED = 1 << 1; + /// Object is unreachable (during GC collection) + const UNREACHABLE = 1 << 2; + /// Object is frozen (immutable) + const FROZEN = 1 << 3; + /// Memory the object references is shared between multiple threads + /// and needs special handling when freeing due to possible in-flight lock-free reads + const SHARED = 1 << 4; + /// Memory of the object itself is shared between multiple threads + /// Objects with this bit that are GC objects will automatically be delay-freed + const SHARED_INLINE = 1 << 5; + /// Use deferred reference counting + const DEFERRED = 1 << 6; + } +} + /// This is an actual python object. It consists of a `typ` which is the /// python class, and carries some rust payload optionally. This rust /// payload can be a rust float or rust int in case of float and int objects. #[repr(C)] pub(super) struct PyInner<T> { pub(super) ref_count: RefCount, - // TODO: move typeid into vtable once TypeId::of is const - pub(super) typeid: TypeId, pub(super) vtable: &'static PyObjVTable, + /// GC bits for free-threading (like ob_gc_bits) + pub(super) gc_bits: PyAtomic<u8>, pub(super) typ: PyAtomicRef<PyType>, // __class__ member pub(super) dict: Option<InstanceDict>, @@ -140,8 +171,77 @@ unsafe impl Traverse for PyObject { } } +// === Stripe lock for weakref list protection (WEAKREF_LIST_LOCK) === + +#[cfg(feature = "threading")] +mod weakref_lock { + use core::sync::atomic::{AtomicU8, Ordering}; + + const NUM_WEAKREF_LOCKS: usize = 64; + + static LOCKS: [AtomicU8; NUM_WEAKREF_LOCKS] = [const { AtomicU8::new(0) }; NUM_WEAKREF_LOCKS]; + + pub(super) struct WeakrefLockGuard { + idx: usize, + } + + impl Drop for WeakrefLockGuard { + fn drop(&mut self) { + LOCKS[self.idx].store(0, Ordering::Release); + } + } + + pub(super) fn lock(addr: usize) -> WeakrefLockGuard { + let idx = (addr >> 4) % NUM_WEAKREF_LOCKS; + loop { + if LOCKS[idx] + .compare_exchange_weak(0, 1, Ordering::Acquire, Ordering::Relaxed) + .is_ok() + { + return WeakrefLockGuard { idx }; + } + core::hint::spin_loop(); + } + } + + /// Reset all weakref stripe locks after fork in child process. + /// Locks held by parent threads would cause infinite spin in the child. + #[cfg(unix)] + pub(crate) fn reset_all_after_fork() { + for lock in &LOCKS { + lock.store(0, Ordering::Release); + } + } +} + +#[cfg(not(feature = "threading"))] +mod weakref_lock { + pub(super) struct WeakrefLockGuard; + + impl Drop for WeakrefLockGuard { + fn drop(&mut self) {} + } + + pub(super) fn lock(_addr: usize) -> WeakrefLockGuard { + WeakrefLockGuard + } +} + +/// Reset weakref stripe locks after fork. Must be called before any +/// Python code runs in the child process. +#[cfg(all(unix, feature = "threading"))] +pub(crate) fn reset_weakref_locks_after_fork() { + weakref_lock::reset_all_after_fork(); +} + +// === WeakRefList: inline on every object (tp_weaklist) === + pub(super) struct WeakRefList { - inner: OncePtr<PyMutex<WeakListInner>>, + /// Head of the intrusive doubly-linked list of weakrefs. + head: PyAtomic<*mut Py<PyWeak>>, + /// Cached generic weakref (no callback, exact weakref type). + /// Matches try_reuse_basic_ref in weakrefobject.c. + generic: PyAtomic<*mut Py<PyWeak>>, } impl fmt::Debug for WeakRefList { @@ -150,33 +250,43 @@ impl fmt::Debug for WeakRefList { } } -struct WeakListInner { - list: LinkedList<WeakLink, Py<PyWeak>>, - generic_weakref: Option<NonNull<Py<PyWeak>>>, - obj: Option<NonNull<PyObject>>, - // one for each live PyWeak with a reference to this, + 1 for the referent object if it's not dead - ref_count: usize, -} +/// Unlink a node from the weakref list. Must be called under stripe lock. +/// +/// # Safety +/// `node` must be a valid pointer to a node currently in the list owned by `wrl`. +unsafe fn unlink_weakref(wrl: &WeakRefList, node: NonNull<Py<PyWeak>>) { + unsafe { + let mut ptrs = WeakLink::pointers(node); + let prev = ptrs.as_ref().get_prev(); + let next = ptrs.as_ref().get_next(); + + if let Some(prev) = prev { + WeakLink::pointers(prev).as_mut().set_next(next); + } else { + // node is the head + wrl.head.store( + next.map_or(ptr::null_mut(), |p| p.as_ptr()), + Ordering::Relaxed, + ); + } + if let Some(next) = next { + WeakLink::pointers(next).as_mut().set_prev(prev); + } -cfg_if::cfg_if! { - if #[cfg(feature = "threading")] { - unsafe impl Send for WeakListInner {} - unsafe impl Sync for WeakListInner {} + ptrs.as_mut().set_prev(None); + ptrs.as_mut().set_next(None); } } impl WeakRefList { pub fn new() -> Self { Self { - inner: OncePtr::new(), + head: Radium::new(ptr::null_mut()), + generic: Radium::new(ptr::null_mut()), } } - /// returns None if there have never been any weakrefs in this list - fn try_lock(&self) -> Option<PyMutexGuard<'_, WeakListInner>> { - self.inner.get().map(|mu| unsafe { mu.as_ref().lock() }) - } - + /// get_or_create_weakref fn add( &self, obj: &PyObject, @@ -186,122 +296,153 @@ impl WeakRefList { dict: Option<PyDictRef>, ) -> PyRef<PyWeak> { let is_generic = cls_is_weakref && callback.is_none(); - let inner_ptr = self.inner.get_or_init(|| { - Box::new(PyMutex::new(WeakListInner { - list: LinkedList::default(), - generic_weakref: None, - obj: Some(NonNull::from(obj)), - ref_count: 1, - })) - }); - let mut inner = unsafe { inner_ptr.as_ref().lock() }; - if is_generic && let Some(generic_weakref) = inner.generic_weakref { - let generic_weakref = unsafe { generic_weakref.as_ref() }; - if generic_weakref.0.ref_count.get() != 0 { - return generic_weakref.to_owned(); + let _lock = weakref_lock::lock(obj as *const PyObject as usize); + + // try_reuse_basic_ref: reuse cached generic weakref + if is_generic { + let generic_ptr = self.generic.load(Ordering::Relaxed); + if !generic_ptr.is_null() { + let generic = unsafe { &*generic_ptr }; + if generic.0.ref_count.safe_inc() { + return unsafe { PyRef::from_raw(generic_ptr) }; + } } } - let obj = PyWeak { + + // Allocate new PyWeak with wr_object pointing to referent + let weak_payload = PyWeak { pointers: Pointers::new(), - parent: inner_ptr, + wr_object: Radium::new(obj as *const PyObject as *mut PyObject), callback: UnsafeCell::new(callback), hash: Radium::new(crate::common::hash::SENTINEL), }; - let weak = PyRef::new_ref(obj, cls, dict); - // SAFETY: we don't actually own the PyObjectWeak's inside `list`, and every time we take - // one out of the list we immediately wrap it in ManuallyDrop or forget it - inner.list.push_front(unsafe { ptr::read(&weak) }); - inner.ref_count += 1; - if is_generic { - inner.generic_weakref = Some(NonNull::from(&*weak)); + let weak = PyRef::new_ref(weak_payload, cls, dict); + + // Insert into linked list under stripe lock + let node_ptr = NonNull::from(&*weak); + unsafe { + let mut ptrs = WeakLink::pointers(node_ptr); + if is_generic { + // Generic ref goes to head (insert_head for basic ref) + let old_head = self.head.load(Ordering::Relaxed); + ptrs.as_mut().set_next(NonNull::new(old_head)); + ptrs.as_mut().set_prev(None); + if let Some(old_head) = NonNull::new(old_head) { + WeakLink::pointers(old_head) + .as_mut() + .set_prev(Some(node_ptr)); + } + self.head.store(node_ptr.as_ptr(), Ordering::Relaxed); + self.generic.store(node_ptr.as_ptr(), Ordering::Relaxed); + } else { + // Non-generic refs go after generic ref (insert_after) + let generic_ptr = self.generic.load(Ordering::Relaxed); + if let Some(after) = NonNull::new(generic_ptr) { + let after_next = WeakLink::pointers(after).as_ref().get_next(); + ptrs.as_mut().set_prev(Some(after)); + ptrs.as_mut().set_next(after_next); + WeakLink::pointers(after).as_mut().set_next(Some(node_ptr)); + if let Some(next) = after_next { + WeakLink::pointers(next).as_mut().set_prev(Some(node_ptr)); + } + } else { + // No generic ref; insert at head + let old_head = self.head.load(Ordering::Relaxed); + ptrs.as_mut().set_next(NonNull::new(old_head)); + ptrs.as_mut().set_prev(None); + if let Some(old_head) = NonNull::new(old_head) { + WeakLink::pointers(old_head) + .as_mut() + .set_prev(Some(node_ptr)); + } + self.head.store(node_ptr.as_ptr(), Ordering::Relaxed); + } + } } + weak } - fn clear(&self) { - let to_dealloc = { - let ptr = match self.inner.get() { - Some(ptr) => ptr, - None => return, - }; - let mut inner = unsafe { ptr.as_ref().lock() }; - inner.obj = None; - // TODO: can be an arrayvec - let mut v = Vec::with_capacity(16); - loop { - let inner2 = &mut *inner; - let iter = inner2 - .list - .drain_filter(|_| true) - .filter_map(|wr| { - // we don't have actual ownership of the reference counts in the list. - // but, now we do want ownership (and so incref these *while the lock - // is held*) to avoid weird things if PyWeakObj::drop happens after - // this but before we reach the loop body below - let wr = ManuallyDrop::new(wr); - - if Some(NonNull::from(&**wr)) == inner2.generic_weakref { - inner2.generic_weakref = None - } - - // if strong_count == 0 there's some reentrancy going on. we don't - // want to call the callback - (wr.as_object().strong_count() > 0).then(|| (*wr).clone()) - }) - .take(16); - v.extend(iter); - if v.is_empty() { - break; + /// PyObject_ClearWeakRefs: clear all weakrefs when the referent dies. + fn clear(&self, obj: &PyObject) { + let obj_addr = obj as *const PyObject as usize; + let mut to_callback: Vec<(PyRef<PyWeak>, PyObjectRef)> = Vec::new(); + + { + let _lock = weakref_lock::lock(obj_addr); + + // Walk the list, collecting weakrefs with callbacks + let mut current = NonNull::new(self.head.load(Ordering::Relaxed)); + while let Some(node) = current { + let next = unsafe { WeakLink::pointers(node).as_ref().get_next() }; + + let wr = unsafe { node.as_ref() }; + + // Set wr_object to null (marks weakref as dead) + wr.0.payload + .wr_object + .store(ptr::null_mut(), Ordering::Relaxed); + + // Unlink from list + unsafe { + let mut ptrs = WeakLink::pointers(node); + ptrs.as_mut().set_prev(None); + ptrs.as_mut().set_next(None); } - PyMutexGuard::unlocked(&mut inner, || { - for wr in v.drain(..) { - let cb = unsafe { wr.callback.get().replace(None) }; - if let Some(cb) = cb { - crate::vm::thread::with_vm(&cb, |vm| { - // TODO: handle unraisable exception - let _ = cb.call((wr.clone(),), vm); - }); - } + + // Collect callback if weakref is still alive (strong_count > 0) + if wr.0.ref_count.get() > 0 { + let cb = unsafe { wr.0.payload.callback.get().replace(None) }; + if let Some(cb) = cb { + to_callback.push((wr.to_owned(), cb)); } - }) + } + + current = next; } - inner.ref_count -= 1; - (inner.ref_count == 0).then_some(ptr) - }; - if let Some(ptr) = to_dealloc { - unsafe { Self::dealloc(ptr) } + + self.head.store(ptr::null_mut(), Ordering::Relaxed); + self.generic.store(ptr::null_mut(), Ordering::Relaxed); } - } - fn count(&self) -> usize { - self.try_lock() - // we assume the object is still alive (and this is only - // called from PyObject::weak_count so it should be) - .map(|inner| inner.ref_count - 1) - .unwrap_or(0) + // Call callbacks without holding the lock + for (wr, cb) in to_callback { + crate::vm::thread::with_vm(&cb, |vm| { + // TODO: handle unraisable exception + let wr_obj: PyObjectRef = wr.clone().into(); + let _ = cb.call((wr_obj,), vm); + }); + } } - unsafe fn dealloc(ptr: NonNull<PyMutex<WeakListInner>>) { - drop(unsafe { Box::from_raw(ptr.as_ptr()) }); + fn count(&self, obj: &PyObject) -> usize { + let _lock = weakref_lock::lock(obj as *const PyObject as usize); + let mut count = 0usize; + let mut current = NonNull::new(self.head.load(Ordering::Relaxed)); + while let Some(node) = current { + if unsafe { node.as_ref() }.0.ref_count.get() > 0 { + count += 1; + } + current = unsafe { WeakLink::pointers(node).as_ref().get_next() }; + } + count } - fn get_weak_references(&self) -> Vec<PyRef<PyWeak>> { - let inner = match self.try_lock() { - Some(inner) => inner, - None => return vec![], - }; - let mut v = Vec::with_capacity(inner.ref_count - 1); - v.extend(inner.iter().map(|wr| wr.to_owned())); + fn get_weak_references(&self, obj: &PyObject) -> Vec<PyRef<PyWeak>> { + let _lock = weakref_lock::lock(obj as *const PyObject as usize); + let mut v = Vec::new(); + let mut current = NonNull::new(self.head.load(Ordering::Relaxed)); + while let Some(node) = current { + let wr = unsafe { node.as_ref() }; + if wr.0.ref_count.get() > 0 { + v.push(wr.to_owned()); + } + current = unsafe { WeakLink::pointers(node).as_ref().get_next() }; + } v } } -impl WeakListInner { - fn iter(&self) -> impl Iterator<Item = &Py<PyWeak>> { - self.list.iter().filter(|wr| wr.0.ref_count.get() > 0) - } -} - impl Default for WeakRefList { fn default() -> Self { Self::new() @@ -321,7 +462,6 @@ unsafe impl Link for WeakLink { #[inline(always)] unsafe fn from_raw(ptr: NonNull<Self::Target>) -> Self::Handle { - // SAFETY: requirements forwarded from caller unsafe { PyRef::from_raw(ptr.as_ptr()) } } @@ -332,12 +472,15 @@ unsafe impl Link for WeakLink { } } +/// PyWeakReference: each weakref holds a direct pointer to its referent. #[pyclass(name = "weakref", module = false)] #[derive(Debug)] pub struct PyWeak { pointers: Pointers<Py<PyWeak>>, - parent: NonNull<PyMutex<WeakListInner>>, - // this is treated as part of parent's mutex - you must hold that lock to access it + /// Direct pointer to the referent object, null when dead. + /// Equivalent to wr_object in PyWeakReference. + wr_object: PyAtomic<*mut PyObject>, + /// Protected by stripe lock (keyed on wr_object address). callback: UnsafeCell<Option<PyObjectRef>>, pub(crate) hash: PyAtomic<crate::common::hash::PyHash>, } @@ -351,43 +494,69 @@ cfg_if::cfg_if! { } impl PyWeak { + /// _PyWeakref_GET_REF: attempt to upgrade the weakref to a strong reference. pub(crate) fn upgrade(&self) -> Option<PyObjectRef> { - let guard = unsafe { self.parent.as_ref().lock() }; - let obj_ptr = guard.obj?; + let obj_ptr = self.wr_object.load(Ordering::Acquire); + if obj_ptr.is_null() { + return None; + } + + let _lock = weakref_lock::lock(obj_ptr as usize); + + // Double-check under lock (clear may have run between our check and lock) + let obj_ptr = self.wr_object.load(Ordering::Relaxed); + if obj_ptr.is_null() { + return None; + } + unsafe { - if !obj_ptr.as_ref().0.ref_count.safe_inc() { + if !(*obj_ptr).0.ref_count.safe_inc() { return None; } - Some(PyObjectRef::from_raw(obj_ptr)) + Some(PyObjectRef::from_raw(NonNull::new_unchecked(obj_ptr))) } } pub(crate) fn is_dead(&self) -> bool { - let guard = unsafe { self.parent.as_ref().lock() }; - guard.obj.is_none() + self.wr_object.load(Ordering::Acquire).is_null() } + /// weakref_dealloc: remove from list if still linked. fn drop_inner(&self) { - let dealloc = { - let mut guard = unsafe { self.parent.as_ref().lock() }; - let offset = std::mem::offset_of!(PyInner<Self>, payload); - let py_inner = (self as *const Self) - .cast::<u8>() - .wrapping_sub(offset) - .cast::<PyInner<Self>>(); - let node_ptr = unsafe { NonNull::new_unchecked(py_inner as *mut Py<Self>) }; - // the list doesn't have ownership over its PyRef<PyWeak>! we're being dropped - // right now so that should be obvious!! - core::mem::forget(unsafe { guard.list.remove(node_ptr) }); - guard.ref_count -= 1; - if Some(node_ptr) == guard.generic_weakref { - guard.generic_weakref = None; - } - guard.ref_count == 0 - }; - if dealloc { - unsafe { WeakRefList::dealloc(self.parent) } + let obj_ptr = self.wr_object.load(Ordering::Acquire); + if obj_ptr.is_null() { + return; // Already cleared by WeakRefList::clear() + } + + let _lock = weakref_lock::lock(obj_ptr as usize); + + // Double-check under lock + let obj_ptr = self.wr_object.load(Ordering::Relaxed); + if obj_ptr.is_null() { + return; // Cleared between our check and lock acquisition + } + + let obj = unsafe { &*obj_ptr }; + let wrl = &obj.0.weak_list; + + // Compute our Py<PyWeak> node pointer from payload address + let offset = std::mem::offset_of!(PyInner<Self>, payload); + let py_inner = (self as *const Self) + .cast::<u8>() + .wrapping_sub(offset) + .cast::<PyInner<Self>>(); + let node_ptr = unsafe { NonNull::new_unchecked(py_inner as *mut Py<Self>) }; + + // Unlink from list + unsafe { unlink_weakref(wrl, node_ptr) }; + + // Update generic cache if this was it + if wrl.generic.load(Ordering::Relaxed) == node_ptr.as_ptr() { + wrl.generic.store(ptr::null_mut(), Ordering::Relaxed); } + + // Mark as dead + self.wr_object.store(ptr::null_mut(), Ordering::Relaxed); } } @@ -395,7 +564,6 @@ impl Drop for PyWeak { #[inline(always)] fn drop(&mut self) { // we do NOT have actual exclusive access! - // no clue if doing this actually reduces chance of UB let me: &Self = self; me.drop_inner(); } @@ -449,8 +617,8 @@ impl<T: PyPayload + core::fmt::Debug> PyInner<T> { let member_count = typ.slots.member_count; Box::new(Self { ref_count: RefCount::new(), - typeid: T::payload_type_id(), vtable: PyObjVTable::of::<T>(), + gc_bits: Radium::new(0), typ: PyAtomicRef::from(typ), dict: dict.map(InstanceDict::new), weak_list: WeakRefList::new(), @@ -633,13 +801,14 @@ impl PyObject { } pub fn get_weak_references(&self) -> Option<Vec<PyRef<PyWeak>>> { - self.weak_ref_list().map(|wrl| wrl.get_weak_references()) + self.weak_ref_list() + .map(|wrl| wrl.get_weak_references(self)) } #[deprecated(note = "use downcastable instead")] #[inline(always)] pub fn payload_is<T: PyPayload>(&self) -> bool { - self.0.typeid == T::payload_type_id() + self.0.vtable.typeid == T::PAYLOAD_TYPE_ID } /// Force to return payload as T. @@ -722,13 +891,13 @@ impl PyObject { #[inline] pub(crate) fn typeid(&self) -> TypeId { - self.0.typeid + self.0.vtable.typeid } /// Check if this object can be downcast to T. #[inline(always)] pub fn downcastable<T: PyPayload>(&self) -> bool { - T::downcastable_from(self) + self.typeid() == T::PAYLOAD_TYPE_ID && unsafe { T::validate_downcastable_from(self) } } /// Attempt to downcast this reference to a subclass. @@ -775,7 +944,7 @@ impl PyObject { #[inline] pub fn weak_count(&self) -> Option<usize> { - self.weak_ref_list().map(|wrl| wrl.count()) + self.weak_ref_list().map(|wrl| wrl.count(self)) } #[inline(always)] @@ -783,6 +952,23 @@ impl PyObject { self } + /// Check if the object has been finalized (__del__ already called). + /// _PyGC_FINALIZED in Py_GIL_DISABLED mode. + #[inline] + pub(crate) fn gc_finalized(&self) -> bool { + GcBits::from_bits_retain(self.0.gc_bits.load(Ordering::Relaxed)).contains(GcBits::FINALIZED) + } + + /// Mark the object as finalized. Should be called before __del__. + /// _PyGC_SET_FINALIZED in Py_GIL_DISABLED mode. + #[inline] + fn set_gc_finalized(&self) { + // Atomic RMW to avoid clobbering other concurrent bit updates. + self.0 + .gc_bits + .fetch_or(GcBits::FINALIZED.bits(), Ordering::Relaxed); + } + #[inline(always)] // the outer function is never inlined fn drop_slow_inner(&self) -> Result<(), ()> { // __del__ is mostly not implemented @@ -793,15 +979,23 @@ impl PyObject { slot_del: fn(&PyObject, &VirtualMachine) -> PyResult<()>, ) -> Result<(), ()> { let ret = crate::vm::thread::with_vm(zelf, |vm| { - zelf.0.ref_count.inc(); + // Temporarily resurrect (0→2) so ref_count stays positive + // during __del__, preventing safe_inc from seeing 0. + zelf.0.ref_count.inc_by(2); + if let Err(e) = slot_del(zelf, vm) { let del_method = zelf.get_class_attr(identifier!(vm, __del__)).unwrap(); vm.run_unraisable(e, None, del_method); } + + // Undo the temporary resurrection. Always remove both + // temporary refs; the second dec returns true only when + // ref_count drops to 0 (no resurrection). + zelf.0.ref_count.dec(); zelf.0.ref_count.dec() }); match ret { - // the decref right above set ref_count back to 0 + // the decref set ref_count back to 0 Some(true) => Ok(()), // we've been resurrected by __del__ Some(false) => Err(()), @@ -812,28 +1006,26 @@ impl PyObject { } } - // CPython-compatible drop implementation + // __del__ should only be called once (like _PyGC_FINALIZED check in GIL_DISABLED) let del = self.class().slots.del.load(); - if let Some(slot_del) = del { + if let Some(slot_del) = del + && !self.gc_finalized() + { + self.set_gc_finalized(); call_slot_del(self, slot_del)?; } if let Some(wrl) = self.weak_ref_list() { - wrl.clear(); + wrl.clear(self); } Ok(()) } - /// Can only be called when ref_count has dropped to zero. `ptr` must be valid + /// _Py_Dealloc: dispatch to type's dealloc #[inline(never)] unsafe fn drop_slow(ptr: NonNull<Self>) { - if let Err(()) = unsafe { ptr.as_ref().drop_slow_inner() } { - // abort drop for whatever reason - return; - } - let drop_dealloc = unsafe { ptr.as_ref().0.vtable.drop_dealloc }; - // call drop only when there are no references in scope - stacked borrows stuff - unsafe { drop_dealloc(ptr.as_ptr()) } + let dealloc = unsafe { ptr.as_ref().0.vtable.dealloc }; + unsafe { dealloc(ptr.as_ptr()) } } /// # Safety @@ -853,6 +1045,25 @@ impl PyObject { pub(crate) fn set_slot(&self, offset: usize, value: Option<PyObjectRef>) { *self.0.slots[offset].write() = value; } + + /// Check if this object is tracked by the garbage collector. + /// Returns true if the object has a trace function or has an instance dict. + pub fn is_gc_tracked(&self) -> bool { + if self.0.vtable.trace.is_some() { + return true; + } + self.0.dict.is_some() + } + + /// Get the referents (objects directly referenced) of this object. + /// Uses the full traverse including dict and slots. + pub fn gc_get_referents(&self) -> Vec<PyObjectRef> { + let mut result = Vec::new(); + self.0.traverse(&mut |child: &PyObject| { + result.push(child.to_owned()); + }); + result + } } impl Borrow<PyObject> for PyObjectRef { @@ -1276,8 +1487,8 @@ pub(crate) fn init_type_hierarchy() -> (PyTypeRef, PyTypeRef, PyTypeRef) { let type_type_ptr = Box::into_raw(Box::new(partially_init!( PyInner::<PyType> { ref_count: RefCount::new(), - typeid: TypeId::of::<PyType>(), vtable: PyObjVTable::of::<PyType>(), + gc_bits: Radium::new(0), dict: None, weak_list: WeakRefList::new(), payload: type_payload, @@ -1288,8 +1499,8 @@ pub(crate) fn init_type_hierarchy() -> (PyTypeRef, PyTypeRef, PyTypeRef) { let object_type_ptr = Box::into_raw(Box::new(partially_init!( PyInner::<PyType> { ref_count: RefCount::new(), - typeid: TypeId::of::<PyType>(), vtable: PyObjVTable::of::<PyType>(), + gc_bits: Radium::new(0), dict: None, weak_list: WeakRefList::new(), payload: object_payload, diff --git a/crates/vm/src/object/ext.rs b/crates/vm/src/object/ext.rs index c1a5f63f85e..0fd251499f1 100644 --- a/crates/vm/src/object/ext.rs +++ b/crates/vm/src/object/ext.rs @@ -463,6 +463,64 @@ impl PyAtomicRef<Option<PyObject>> { } } +/// Atomic borrowed (non-ref-counted) optional reference to a Python object. +/// Unlike `PyAtomicRef`, this does NOT own the reference. +/// The pointed-to object must outlive this reference. +pub struct PyAtomicBorrow { + inner: PyAtomic<*mut u8>, +} + +// Safety: Access patterns ensure the pointed-to object outlives this reference. +// The owner (generator/coroutine) clears this in its Drop impl before deallocation. +unsafe impl Send for PyAtomicBorrow {} +unsafe impl Sync for PyAtomicBorrow {} + +impl PyAtomicBorrow { + pub fn new() -> Self { + Self { + inner: Radium::new(null_mut()), + } + } + + pub fn store(&self, obj: &PyObject) { + let ptr = obj as *const PyObject as *mut u8; + Radium::store(&self.inner, ptr, Ordering::Relaxed); + } + + pub fn load(&self) -> Option<&PyObject> { + let ptr = Radium::load(&self.inner, Ordering::Relaxed); + if ptr.is_null() { + None + } else { + Some(unsafe { &*(ptr as *const PyObject) }) + } + } + + pub fn clear(&self) { + Radium::store(&self.inner, null_mut(), Ordering::Relaxed); + } + + pub fn to_owned(&self) -> Option<PyObjectRef> { + self.load().map(|obj| obj.to_owned()) + } +} + +impl Default for PyAtomicBorrow { + fn default() -> Self { + Self::new() + } +} + +impl fmt::Debug for PyAtomicBorrow { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "PyAtomicBorrow({:?})", + Radium::load(&self.inner, Ordering::Relaxed) + ) + } +} + pub trait AsObject where Self: Borrow<PyObject>, diff --git a/crates/vm/src/object/payload.rs b/crates/vm/src/object/payload.rs index 3a2f42675f7..98c61817568 100644 --- a/crates/vm/src/object/payload.rs +++ b/crates/vm/src/object/payload.rs @@ -26,24 +26,17 @@ pub(crate) fn cold_downcast_type_error( } pub trait PyPayload: MaybeTraverse + PyThreadingConstraint + Sized + 'static { - #[inline] - fn payload_type_id() -> core::any::TypeId { - core::any::TypeId::of::<Self>() - } - - /// # Safety: this function should only be called if `payload_type_id` matches the type of `obj`. - #[inline] - fn downcastable_from(obj: &PyObject) -> bool { - obj.typeid() == Self::payload_type_id() && Self::validate_downcastable_from(obj) - } + const PAYLOAD_TYPE_ID: core::any::TypeId = core::any::TypeId::of::<Self>(); + /// # Safety + /// This function should only be called if `payload_type_id` matches the type of `obj`. #[inline] - fn validate_downcastable_from(_obj: &PyObject) -> bool { + unsafe fn validate_downcastable_from(_obj: &PyObject) -> bool { true } fn try_downcast_from(obj: &PyObject, vm: &VirtualMachine) -> PyResult<()> { - if Self::downcastable_from(obj) { + if obj.downcastable::<Self>() { return Ok(()); } diff --git a/crates/vm/src/object/traverse.rs b/crates/vm/src/object/traverse.rs index 2ce0db41a5e..367076b78e3 100644 --- a/crates/vm/src/object/traverse.rs +++ b/crates/vm/src/object/traverse.rs @@ -12,9 +12,13 @@ pub type TraverseFn<'a> = dyn FnMut(&PyObject) + 'a; /// Every PyObjectPayload impl `MaybeTrace`, which may or may not be traceable pub trait MaybeTraverse { /// if is traceable, will be used by vtable to determine - const IS_TRACE: bool = false; + const HAS_TRAVERSE: bool = false; + /// if has clear implementation for circular reference resolution (tp_clear) + const HAS_CLEAR: bool = false; // if this type is traceable, then call with tracer_fn, default to do nothing fn try_traverse(&self, traverse_fn: &mut TraverseFn<'_>); + // if this type has clear, extract child refs for circular reference resolution (tp_clear) + fn try_clear(&mut self, _out: &mut Vec<PyObjectRef>) {} } /// Type that need traverse it's children should impl [`Traverse`] (not [`MaybeTraverse`]) @@ -28,6 +32,11 @@ pub unsafe trait Traverse { /// /// - _**DO NOT**_ clone a [`PyObjectRef`] or [`PyRef<T>`] in [`Traverse::traverse()`] fn traverse(&self, traverse_fn: &mut TraverseFn<'_>); + + /// Extract all owned child PyObjectRefs for circular reference resolution (tp_clear). + /// Called just before object deallocation to break circular references. + /// Default implementation does nothing. + fn clear(&mut self, _out: &mut Vec<PyObjectRef>) {} } unsafe impl Traverse for PyObjectRef { diff --git a/crates/vm/src/object/traverse_object.rs b/crates/vm/src/object/traverse_object.rs index 075ce5b9513..b297864245e 100644 --- a/crates/vm/src/object/traverse_object.rs +++ b/crates/vm/src/object/traverse_object.rs @@ -1,17 +1,20 @@ use alloc::fmt; +use core::any::TypeId; use crate::{ PyObject, object::{ - Erased, InstanceDict, MaybeTraverse, PyInner, PyObjectPayload, debug_obj, drop_dealloc_obj, - try_trace_obj, + Erased, InstanceDict, MaybeTraverse, PyInner, PyObjectPayload, debug_obj, default_dealloc, + try_traverse_obj, }, }; use super::{Traverse, TraverseFn}; pub(in crate::object) struct PyObjVTable { - pub(in crate::object) drop_dealloc: unsafe fn(*mut PyObject), + pub(in crate::object) typeid: TypeId, + /// dealloc: handles __del__, weakref clearing, and memory free. + pub(in crate::object) dealloc: unsafe fn(*mut PyObject), pub(in crate::object) debug: unsafe fn(&PyObject, &mut fmt::Formatter<'_>) -> fmt::Result, pub(in crate::object) trace: Option<unsafe fn(&PyObject, &mut TraverseFn<'_>)>, } @@ -19,11 +22,12 @@ pub(in crate::object) struct PyObjVTable { impl PyObjVTable { pub const fn of<T: PyObjectPayload>() -> &'static Self { &Self { - drop_dealloc: drop_dealloc_obj::<T>, + typeid: T::PAYLOAD_TYPE_ID, + dealloc: default_dealloc::<T>, debug: debug_obj::<T>, trace: const { - if T::IS_TRACE { - Some(try_trace_obj::<T>) + if T::HAS_TRAVERSE { + Some(try_traverse_obj::<T>) } else { None } @@ -45,7 +49,7 @@ unsafe impl Traverse for PyInner<Erased> { // 2. call vtable's trace function to trace payload // self.typ.trace(tracer_fn); self.dict.traverse(tracer_fn); - // weak_list keeps a *pointer* to a struct for maintenance of weak ref, so no ownership, no trace + // weak_list is inline atomic pointers, no heap allocation, no trace self.slots.traverse(tracer_fn); if let Some(f) = self.vtable.trace { @@ -65,7 +69,7 @@ unsafe impl<T: MaybeTraverse> Traverse for PyInner<T> { // (No need to call vtable's trace function because we already know the type) // self.typ.trace(tracer_fn); self.dict.traverse(tracer_fn); - // weak_list keeps a *pointer* to a struct for maintenance of weak ref, so no ownership, no trace + // weak_list is inline atomic pointers, no heap allocation, no trace self.slots.traverse(tracer_fn); T::try_traverse(&self.payload, tracer_fn); } diff --git a/crates/vm/src/signal.rs b/crates/vm/src/signal.rs index 4a1b84a1521..4aa245ad190 100644 --- a/crates/vm/src/signal.rs +++ b/crates/vm/src/signal.rs @@ -50,6 +50,16 @@ pub(crate) fn set_triggered() { ANY_TRIGGERED.store(true, Ordering::Release); } +/// Reset all signal trigger state after fork in child process. +/// Stale triggers from the parent must not fire in the child. +#[cfg(unix)] +pub(crate) fn clear_after_fork() { + ANY_TRIGGERED.store(false, Ordering::Release); + for trigger in &TRIGGERS { + trigger.store(false, Ordering::Relaxed); + } +} + pub fn assert_in_range(signum: i32, vm: &VirtualMachine) -> PyResult<()> { if (1..NSIG as i32).contains(&signum) { Ok(()) diff --git a/crates/vm/src/stdlib/_abc.rs b/crates/vm/src/stdlib/_abc.rs new file mode 100644 index 00000000000..e04566b4b2d --- /dev/null +++ b/crates/vm/src/stdlib/_abc.rs @@ -0,0 +1,481 @@ +//! Implementation of the `_abc` module. +//! +//! This module provides the C implementation of Abstract Base Classes (ABCs) +//! as defined in PEP 3119. + +pub(crate) use _abc::module_def; + +#[pymodule] +mod _abc { + use crate::{ + AsObject, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, + builtins::{PyFrozenSet, PyList, PySet, PyStr, PyTupleRef, PyTypeRef, PyWeak}, + common::lock::PyRwLock, + convert::ToPyObject, + protocol::PyIterReturn, + types::Constructor, + }; + use std::sync::atomic::{AtomicU64, Ordering}; + + // Global invalidation counter + static ABC_INVALIDATION_COUNTER: AtomicU64 = AtomicU64::new(0); + + fn get_invalidation_counter() -> u64 { + ABC_INVALIDATION_COUNTER.load(Ordering::SeqCst) + } + + fn increment_invalidation_counter() { + ABC_INVALIDATION_COUNTER.fetch_add(1, Ordering::SeqCst); + } + + /// Internal state held by ABC machinery. + #[pyattr] + #[pyclass(name = "_abc_data", module = "_abc")] + #[derive(Debug, PyPayload)] + struct AbcData { + // WeakRef sets for registry and caches + registry: PyRwLock<Option<PyRef<PySet>>>, + cache: PyRwLock<Option<PyRef<PySet>>>, + negative_cache: PyRwLock<Option<PyRef<PySet>>>, + negative_cache_version: AtomicU64, + } + + #[pyclass(with(Constructor))] + impl AbcData { + fn new() -> Self { + AbcData { + registry: PyRwLock::new(None), + cache: PyRwLock::new(None), + negative_cache: PyRwLock::new(None), + negative_cache_version: AtomicU64::new(get_invalidation_counter()), + } + } + + fn get_cache_version(&self) -> u64 { + self.negative_cache_version.load(Ordering::SeqCst) + } + + fn set_cache_version(&self, version: u64) { + self.negative_cache_version.store(version, Ordering::SeqCst); + } + } + + impl Constructor for AbcData { + type Args = (); + + fn py_new( + _cls: &crate::Py<crate::builtins::PyType>, + _args: Self::Args, + _vm: &VirtualMachine, + ) -> PyResult<Self> { + Ok(AbcData::new()) + } + } + + /// Get the _abc_impl attribute from an ABC class + fn get_impl(cls: &PyObject, vm: &VirtualMachine) -> PyResult<PyRef<AbcData>> { + let impl_obj = cls.get_attr("_abc_impl", vm)?; + impl_obj + .downcast::<AbcData>() + .map_err(|_| vm.new_type_error("_abc_impl is set to a wrong type".to_owned())) + } + + /// Check if obj is in the weak set + fn in_weak_set( + set_lock: &PyRwLock<Option<PyRef<PySet>>>, + obj: &PyObject, + vm: &VirtualMachine, + ) -> PyResult<bool> { + let set_opt = set_lock.read(); + let set = match &*set_opt { + Some(s) if !s.elements().is_empty() => s.clone(), + _ => return Ok(false), + }; + drop(set_opt); + + // Create a weak reference to the object + let weak_ref = match obj.downgrade(None, vm) { + Ok(w) => w, + Err(e) => { + // If we can't create a weakref (e.g., TypeError), the object can't be in the set + if e.class().is(vm.ctx.exceptions.type_error) { + return Ok(false); + } + return Err(e); + } + }; + + // Use vm.call_method to call __contains__ + let weak_ref_obj: PyObjectRef = weak_ref.into(); + vm.call_method(set.as_ref(), "__contains__", (weak_ref_obj,))? + .try_to_bool(vm) + } + + /// Add obj to the weak set + fn add_to_weak_set( + set_lock: &PyRwLock<Option<PyRef<PySet>>>, + obj: &PyObject, + vm: &VirtualMachine, + ) -> PyResult<()> { + let mut set_opt = set_lock.write(); + let set = match &*set_opt { + Some(s) => s.clone(), + None => { + let new_set = PySet::default().into_ref(&vm.ctx); + *set_opt = Some(new_set.clone()); + new_set + } + }; + drop(set_opt); + + // Create a weak reference to the object + let weak_ref = obj.downgrade(None, vm)?; + set.add(weak_ref.into(), vm)?; + Ok(()) + } + + /// Returns the current ABC cache token. + #[pyfunction] + fn get_cache_token() -> u64 { + get_invalidation_counter() + } + + /// Compute set of abstract method names. + fn compute_abstract_methods(cls: &PyObject, vm: &VirtualMachine) -> PyResult<()> { + let mut abstracts = Vec::new(); + + // Stage 1: direct abstract methods + let ns = cls.get_attr("__dict__", vm)?; + let items = vm.call_method(&ns, "items", ())?; + let iter = items.get_iter(vm)?; + + while let PyIterReturn::Return(item) = iter.next(vm)? { + let tuple: PyTupleRef = item + .downcast() + .map_err(|_| vm.new_type_error("items() returned non-tuple".to_owned()))?; + let elements = tuple.as_slice(); + if elements.len() != 2 { + return Err( + vm.new_type_error("items() returned item which size is not 2".to_owned()) + ); + } + let key = &elements[0]; + let value = &elements[1]; + + // Check if value has __isabstractmethod__ = True + if let Ok(is_abstract) = value.get_attr("__isabstractmethod__", vm) + && is_abstract.try_to_bool(vm)? + { + abstracts.push(key.clone()); + } + } + + // Stage 2: inherited abstract methods + let bases: PyTupleRef = cls + .get_attr("__bases__", vm)? + .downcast() + .map_err(|_| vm.new_type_error("__bases__ is not a tuple".to_owned()))?; + + for base in bases.iter() { + if let Ok(base_abstracts) = base.get_attr("__abstractmethods__", vm) { + let iter = base_abstracts.get_iter(vm)?; + while let PyIterReturn::Return(key) = iter.next(vm)? { + // Try to get the attribute from cls - key should be a string + if let Some(key_str) = key.downcast_ref::<PyStr>() + && let Some(value) = vm.get_attribute_opt(cls.to_owned(), key_str)? + && let Ok(is_abstract) = value.get_attr("__isabstractmethod__", vm) + && is_abstract.try_to_bool(vm)? + { + abstracts.push(key); + } + } + } + } + + // Set __abstractmethods__ + let abstracts_set = PyFrozenSet::from_iter(vm, abstracts.into_iter())?; + cls.set_attr("__abstractmethods__", abstracts_set.into_pyobject(vm), vm)?; + + Ok(()) + } + + /// Internal ABC helper for class set-up. Should be never used outside abc module. + #[pyfunction] + fn _abc_init(cls: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { + compute_abstract_methods(&cls, vm)?; + + // Set up inheritance registry + let data = AbcData::new(); + cls.set_attr("_abc_impl", data.to_pyobject(vm), vm)?; + + Ok(()) + } + + /// Internal ABC helper for subclass registration. Should be never used outside abc module. + #[pyfunction] + fn _abc_register( + cls: PyObjectRef, + subclass: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult<PyObjectRef> { + // Type check + if !subclass.class().fast_issubclass(vm.ctx.types.type_type) { + return Err(vm.new_type_error("Can only register classes".to_owned())); + } + + // Check if already a subclass + if subclass.is_subclass(&cls, vm)? { + return Ok(subclass); + } + + // Check for cycles + if cls.is_subclass(&subclass, vm)? { + return Err(vm.new_runtime_error("Refusing to create an inheritance cycle".to_owned())); + } + + // Add to registry + let impl_data = get_impl(&cls, vm)?; + add_to_weak_set(&impl_data.registry, &subclass, vm)?; + + // Invalidate negative cache + increment_invalidation_counter(); + + Ok(subclass) + } + + /// Internal ABC helper for instance checks. Should be never used outside abc module. + #[pyfunction] + fn _abc_instancecheck( + cls: PyObjectRef, + instance: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult<PyObjectRef> { + let impl_data = get_impl(&cls, vm)?; + + // Get instance.__class__ + let subclass = instance.get_attr("__class__", vm)?; + + // Check cache + if in_weak_set(&impl_data.cache, &subclass, vm)? { + return Ok(vm.ctx.true_value.clone().into()); + } + + let subtype: PyObjectRef = instance.class().to_owned().into(); + if subtype.is(&subclass) { + let invalidation_counter = get_invalidation_counter(); + if impl_data.get_cache_version() == invalidation_counter + && in_weak_set(&impl_data.negative_cache, &subclass, vm)? + { + return Ok(vm.ctx.false_value.clone().into()); + } + // Fall back to __subclasscheck__ + return vm.call_method(&cls, "__subclasscheck__", (subclass,)); + } + + // Call __subclasscheck__ on subclass + let result = vm.call_method(&cls, "__subclasscheck__", (subclass.clone(),))?; + + match result.clone().try_to_bool(vm) { + Ok(true) => Ok(result), + Ok(false) => { + // Also try with subtype + vm.call_method(&cls, "__subclasscheck__", (subtype,)) + } + Err(e) => Err(e), + } + } + + /// Check if subclass is in registry (recursive) + fn subclasscheck_check_registry( + impl_data: &AbcData, + subclass: &PyObject, + vm: &VirtualMachine, + ) -> PyResult<Option<bool>> { + // Fast path: check if subclass is in weakref directly + if in_weak_set(&impl_data.registry, subclass, vm)? { + return Ok(Some(true)); + } + + let registry_opt = impl_data.registry.read(); + let registry = match &*registry_opt { + Some(s) => s.clone(), + None => return Ok(None), + }; + drop(registry_opt); + + // Make a local copy to protect against concurrent modifications + let registry_copy = PyFrozenSet::from_iter(vm, registry.elements().into_iter())?; + + for weak_ref_obj in registry_copy.elements() { + if let Ok(weak_ref) = weak_ref_obj.downcast::<PyWeak>() + && let Some(rkey) = weak_ref.upgrade() + && subclass.to_owned().is_subclass(&rkey, vm)? + { + add_to_weak_set(&impl_data.cache, subclass, vm)?; + return Ok(Some(true)); + } + } + + Ok(None) + } + + /// Internal ABC helper for subclass checks. Should be never used outside abc module. + #[pyfunction] + fn _abc_subclasscheck( + cls: PyObjectRef, + subclass: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult<bool> { + // Type check + if !subclass.class().fast_issubclass(vm.ctx.types.type_type) { + return Err(vm.new_type_error("issubclass() arg 1 must be a class".to_owned())); + } + + let impl_data = get_impl(&cls, vm)?; + + // 1. Check cache + if in_weak_set(&impl_data.cache, &subclass, vm)? { + return Ok(true); + } + + // 2. Check negative cache; may have to invalidate + let invalidation_counter = get_invalidation_counter(); + if impl_data.get_cache_version() < invalidation_counter { + // Invalidate the negative cache + // Clone set ref and drop lock before calling into VM to avoid reentrancy + let set = impl_data.negative_cache.read().clone(); + if let Some(ref set) = set { + vm.call_method(set.as_ref(), "clear", ())?; + } + impl_data.set_cache_version(invalidation_counter); + } else if in_weak_set(&impl_data.negative_cache, &subclass, vm)? { + return Ok(false); + } + + // 3. Check the subclass hook + let ok = vm.call_method(&cls, "__subclasshook__", (subclass.clone(),))?; + if ok.is(&vm.ctx.true_value) { + add_to_weak_set(&impl_data.cache, &subclass, vm)?; + return Ok(true); + } + if ok.is(&vm.ctx.false_value) { + add_to_weak_set(&impl_data.negative_cache, &subclass, vm)?; + return Ok(false); + } + if !ok.is(&vm.ctx.not_implemented) { + return Err(vm.new_exception_msg( + vm.ctx.exceptions.assertion_error.to_owned(), + "__subclasshook__ must return either False, True, or NotImplemented".to_owned(), + )); + } + + // 4. Check if it's a direct subclass + let subclass_type: PyTypeRef = subclass + .clone() + .downcast() + .map_err(|_| vm.new_type_error("expected a type object".to_owned()))?; + let cls_type: PyTypeRef = cls + .clone() + .downcast() + .map_err(|_| vm.new_type_error("expected a type object".to_owned()))?; + if subclass_type.fast_issubclass(&cls_type) { + add_to_weak_set(&impl_data.cache, &subclass, vm)?; + return Ok(true); + } + + // 5. Check if it's a subclass of a registered class (recursive) + if let Some(result) = subclasscheck_check_registry(&impl_data, &subclass, vm)? { + return Ok(result); + } + + // 6. Check if it's a subclass of a subclass (recursive) + let subclasses: PyRef<PyList> = vm + .call_method(&cls, "__subclasses__", ())? + .downcast() + .map_err(|_| vm.new_type_error("__subclasses__() must return a list".to_owned()))?; + + for scls in subclasses.borrow_vec().iter() { + if subclass.is_subclass(scls, vm)? { + add_to_weak_set(&impl_data.cache, &subclass, vm)?; + return Ok(true); + } + } + + // No dice; update negative cache + add_to_weak_set(&impl_data.negative_cache, &subclass, vm)?; + Ok(false) + } + + /// Internal ABC helper for cache and registry debugging. + #[pyfunction] + fn _get_dump(cls: PyObjectRef, vm: &VirtualMachine) -> PyResult<PyTupleRef> { + let impl_data = get_impl(&cls, vm)?; + + let registry = { + let r = impl_data.registry.read(); + match &*r { + Some(s) => { + // Use copy method to get a shallow copy + vm.call_method(s.as_ref(), "copy", ())? + } + None => PySet::default().to_pyobject(vm), + } + }; + + let cache = { + let c = impl_data.cache.read(); + match &*c { + Some(s) => vm.call_method(s.as_ref(), "copy", ())?, + None => PySet::default().to_pyobject(vm), + } + }; + + let negative_cache = { + let nc = impl_data.negative_cache.read(); + match &*nc { + Some(s) => vm.call_method(s.as_ref(), "copy", ())?, + None => PySet::default().to_pyobject(vm), + } + }; + + let version = impl_data.get_cache_version(); + + Ok(vm.ctx.new_tuple(vec![ + registry, + cache, + negative_cache, + vm.ctx.new_int(version).into(), + ])) + } + + /// Internal ABC helper to reset registry of a given class. + #[pyfunction] + fn _reset_registry(cls: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { + let impl_data = get_impl(&cls, vm)?; + // Clone set ref and drop lock before calling into VM to avoid reentrancy + let set = impl_data.registry.read().clone(); + if let Some(ref set) = set { + vm.call_method(set.as_ref(), "clear", ())?; + } + Ok(()) + } + + /// Internal ABC helper to reset both caches of a given class. + #[pyfunction] + fn _reset_caches(cls: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { + let impl_data = get_impl(&cls, vm)?; + + // Clone set refs and drop locks before calling into VM to avoid reentrancy + let cache = impl_data.cache.read().clone(); + if let Some(ref set) = cache { + vm.call_method(set.as_ref(), "clear", ())?; + } + + let negative_cache = impl_data.negative_cache.read().clone(); + if let Some(ref set) = negative_cache { + vm.call_method(set.as_ref(), "clear", ())?; + } + + Ok(()) + } +} diff --git a/crates/vm/src/stdlib/ast.rs b/crates/vm/src/stdlib/ast.rs index 31aad306f96..cb7bfc289b0 100644 --- a/crates/vm/src/stdlib/ast.rs +++ b/crates/vm/src/stdlib/ast.rs @@ -1,14 +1,16 @@ //! `ast` standard module for abstract syntax trees. + //! //! This module makes use of the parser logic, and translates all ast nodes //! into python ast.AST objects. +pub(crate) use python::_ast::module_def; + mod pyast; use crate::builtins::{PyInt, PyStr}; use crate::stdlib::ast::module::{Mod, ModInteractive}; use crate::stdlib::ast::node::BoxedSlice; -use crate::stdlib::ast::python::_ast; use crate::{ AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyRefExact, PyResult, TryFromObject, VirtualMachine, @@ -19,7 +21,7 @@ use crate::{ convert::ToPyObject, }; use node::Node; -use ruff_python_ast as ruff; +use ruff_python_ast as ast; use ruff_text_size::{Ranged, TextRange, TextSize}; use rustpython_compiler_core::{ LineIndex, OneIndexed, PositionEncoding, SourceFile, SourceFileBuilder, SourceLocation, @@ -283,8 +285,8 @@ pub(crate) fn parse( })? .into_syntax(); let top = match top { - ruff::Mod::Module(m) => Mod::Module(m), - ruff::Mod::Expression(e) => Mod::Expression(e), + ast::Mod::Module(m) => Mod::Module(m), + ast::Mod::Expression(e) => Mod::Expression(e), }; Ok(top.ast_to_object(vm, &source_file)) } @@ -305,13 +307,13 @@ pub(crate) fn compile( let source_file = SourceFileBuilder::new(filename.to_owned(), "".to_owned()).finish(); let ast: Mod = Node::ast_from_object(vm, &source_file, object)?; let ast = match ast { - Mod::Module(m) => ruff::Mod::Module(m), - Mod::Interactive(ModInteractive { range, body }) => ruff::Mod::Module(ruff::ModModule { + Mod::Module(m) => ast::Mod::Module(m), + Mod::Interactive(ModInteractive { range, body }) => ast::Mod::Module(ast::ModModule { node_index: Default::default(), range, body, }), - Mod::Expression(e) => ruff::Mod::Expression(e), + Mod::Expression(e) => ast::Mod::Expression(e), Mod::FunctionType(_) => todo!(), }; // TODO: create a textual representation of the ast @@ -365,9 +367,3 @@ pub const PY_COMPILE_FLAGS_MASK: i32 = PY_COMPILE_FLAG_AST_ONLY | CO_FUTURE_BARRY_AS_BDFL | CO_FUTURE_GENERATOR_STOP | CO_FUTURE_ANNOTATIONS; - -pub fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - let module = _ast::make_module(vm); - pyast::extend_module_nodes(vm, &module); - module -} diff --git a/crates/vm/src/stdlib/ast/argument.rs b/crates/vm/src/stdlib/ast/argument.rs index a13200e6502..626024f5bd6 100644 --- a/crates/vm/src/stdlib/ast/argument.rs +++ b/crates/vm/src/stdlib/ast/argument.rs @@ -3,7 +3,7 @@ use rustpython_compiler_core::SourceFile; pub(super) struct PositionalArguments { pub range: TextRange, - pub args: Box<[ruff::Expr]>, + pub args: Box<[ast::Expr]>, } impl Node for PositionalArguments { @@ -27,7 +27,7 @@ impl Node for PositionalArguments { pub(super) struct KeywordArguments { pub range: TextRange, - pub keywords: Box<[ruff::Keyword]>, + pub keywords: Box<[ast::Keyword]>, } impl Node for KeywordArguments { @@ -53,10 +53,10 @@ impl Node for KeywordArguments { pub(super) fn merge_function_call_arguments( pos_args: PositionalArguments, key_args: KeywordArguments, -) -> ruff::Arguments { +) -> ast::Arguments { let range = pos_args.range.cover(key_args.range); - ruff::Arguments { + ast::Arguments { node_index: Default::default(), range, args: pos_args.args, @@ -65,9 +65,9 @@ pub(super) fn merge_function_call_arguments( } pub(super) fn split_function_call_arguments( - args: ruff::Arguments, + args: ast::Arguments, ) -> (PositionalArguments, KeywordArguments) { - let ruff::Arguments { + let ast::Arguments { node_index: _, range: _, args, @@ -100,13 +100,13 @@ pub(super) fn split_function_call_arguments( } pub(super) fn split_class_def_args( - args: Option<Box<ruff::Arguments>>, + args: Option<Box<ast::Arguments>>, ) -> (Option<PositionalArguments>, Option<KeywordArguments>) { let args = match args { None => return (None, None), Some(args) => *args, }; - let ruff::Arguments { + let ast::Arguments { node_index: _, range: _, args, @@ -141,7 +141,7 @@ pub(super) fn split_class_def_args( pub(super) fn merge_class_def_args( positional_arguments: Option<PositionalArguments>, keyword_arguments: Option<KeywordArguments>, -) -> Option<Box<ruff::Arguments>> { +) -> Option<Box<ast::Arguments>> { if positional_arguments.is_none() && keyword_arguments.is_none() { return None; } @@ -157,7 +157,7 @@ pub(super) fn merge_class_def_args( vec![].into_boxed_slice() }; - Some(Box::new(ruff::Arguments { + Some(Box::new(ast::Arguments { node_index: Default::default(), range: Default::default(), // TODO args, diff --git a/crates/vm/src/stdlib/ast/basic.rs b/crates/vm/src/stdlib/ast/basic.rs index d8565029d6c..612b6144eea 100644 --- a/crates/vm/src/stdlib/ast/basic.rs +++ b/crates/vm/src/stdlib/ast/basic.rs @@ -2,7 +2,7 @@ use super::*; use rustpython_codegen::compile::ruff_int_to_bigint; use rustpython_compiler_core::SourceFile; -impl Node for ruff::Identifier { +impl Node for ast::Identifier { fn ast_to_object(self, vm: &VirtualMachine, _source_file: &SourceFile) -> PyObjectRef { let id = self.as_str(); vm.ctx.new_str(id).into() @@ -18,7 +18,7 @@ impl Node for ruff::Identifier { } } -impl Node for ruff::Int { +impl Node for ast::Int { fn ast_to_object(self, vm: &VirtualMachine, _source_file: &SourceFile) -> PyObjectRef { vm.ctx.new_int(ruff_int_to_bigint(&self).unwrap()).into() } diff --git a/crates/vm/src/stdlib/ast/constant.rs b/crates/vm/src/stdlib/ast/constant.rs index 83b2a7f7015..a6aac224585 100644 --- a/crates/vm/src/stdlib/ast/constant.rs +++ b/crates/vm/src/stdlib/ast/constant.rs @@ -1,6 +1,6 @@ use super::*; use crate::builtins::{PyComplex, PyFrozenSet, PyTuple}; -use ruff::str_prefix::StringLiteralPrefix; +use ast::str_prefix::StringLiteralPrefix; use rustpython_compiler_core::SourceFile; #[derive(Debug)] @@ -22,7 +22,7 @@ impl Constant { } } - pub(super) const fn new_int(value: ruff::Int, range: TextRange) -> Self { + pub(super) const fn new_int(value: ast::Int, range: TextRange) -> Self { Self { range, value: ConstantLiteral::Int(value), @@ -71,7 +71,7 @@ impl Constant { } } - pub(crate) fn into_expr(self) -> ruff::Expr { + pub(crate) fn into_expr(self) -> ast::Expr { constant_to_ruff_expr(self) } } @@ -85,7 +85,7 @@ pub(crate) enum ConstantLiteral { prefix: StringLiteralPrefix, }, Bytes(Box<[u8]>), - Int(ruff::Int), + Int(ast::Int), Tuple(Vec<ConstantLiteral>), FrozenSet(Vec<ConstantLiteral>), Float(f64), @@ -244,48 +244,48 @@ impl Node for ConstantLiteral { } } -fn constant_to_ruff_expr(value: Constant) -> ruff::Expr { +fn constant_to_ruff_expr(value: Constant) -> ast::Expr { let Constant { value, range } = value; match value { - ConstantLiteral::None => ruff::Expr::NoneLiteral(ruff::ExprNoneLiteral { + ConstantLiteral::None => ast::Expr::NoneLiteral(ast::ExprNoneLiteral { node_index: Default::default(), range, }), - ConstantLiteral::Bool(value) => ruff::Expr::BooleanLiteral(ruff::ExprBooleanLiteral { + ConstantLiteral::Bool(value) => ast::Expr::BooleanLiteral(ast::ExprBooleanLiteral { node_index: Default::default(), range, value, }), ConstantLiteral::Str { value, prefix } => { - ruff::Expr::StringLiteral(ruff::ExprStringLiteral { + ast::Expr::StringLiteral(ast::ExprStringLiteral { node_index: Default::default(), range, - value: ruff::StringLiteralValue::single(ruff::StringLiteral { + value: ast::StringLiteralValue::single(ast::StringLiteral { node_index: Default::default(), range, value, - flags: ruff::StringLiteralFlags::empty().with_prefix(prefix), + flags: ast::StringLiteralFlags::empty().with_prefix(prefix), }), }) } ConstantLiteral::Bytes(value) => { - ruff::Expr::BytesLiteral(ruff::ExprBytesLiteral { + ast::Expr::BytesLiteral(ast::ExprBytesLiteral { node_index: Default::default(), range, - value: ruff::BytesLiteralValue::single(ruff::BytesLiteral { + value: ast::BytesLiteralValue::single(ast::BytesLiteral { node_index: Default::default(), range, value, - flags: ruff::BytesLiteralFlags::empty(), // TODO + flags: ast::BytesLiteralFlags::empty(), // TODO }), }) } - ConstantLiteral::Int(value) => ruff::Expr::NumberLiteral(ruff::ExprNumberLiteral { + ConstantLiteral::Int(value) => ast::Expr::NumberLiteral(ast::ExprNumberLiteral { node_index: Default::default(), range, - value: ruff::Number::Int(value), + value: ast::Number::Int(value), }), - ConstantLiteral::Tuple(value) => ruff::Expr::Tuple(ruff::ExprTuple { + ConstantLiteral::Tuple(value) => ast::Expr::Tuple(ast::ExprTuple { node_index: Default::default(), range, elts: value @@ -297,21 +297,21 @@ fn constant_to_ruff_expr(value: Constant) -> ruff::Expr { }) }) .collect(), - ctx: ruff::ExprContext::Load, + ctx: ast::ExprContext::Load, // TODO: Does this matter? parenthesized: true, }), - ConstantLiteral::FrozenSet(value) => ruff::Expr::Call(ruff::ExprCall { + ConstantLiteral::FrozenSet(value) => ast::Expr::Call(ast::ExprCall { node_index: Default::default(), range, // idk lol - func: Box::new(ruff::Expr::Name(ruff::ExprName { + func: Box::new(ast::Expr::Name(ast::ExprName { node_index: Default::default(), range: TextRange::default(), - id: ruff::name::Name::new_static("frozenset"), - ctx: ruff::ExprContext::Load, + id: ast::name::Name::new_static("frozenset"), + ctx: ast::ExprContext::Load, })), - arguments: ruff::Arguments { + arguments: ast::Arguments { node_index: Default::default(), range, args: value @@ -326,19 +326,19 @@ fn constant_to_ruff_expr(value: Constant) -> ruff::Expr { keywords: Box::default(), }, }), - ConstantLiteral::Float(value) => ruff::Expr::NumberLiteral(ruff::ExprNumberLiteral { + ConstantLiteral::Float(value) => ast::Expr::NumberLiteral(ast::ExprNumberLiteral { node_index: Default::default(), range, - value: ruff::Number::Float(value), + value: ast::Number::Float(value), }), ConstantLiteral::Complex { real, imag } => { - ruff::Expr::NumberLiteral(ruff::ExprNumberLiteral { + ast::Expr::NumberLiteral(ast::ExprNumberLiteral { node_index: Default::default(), range, - value: ruff::Number::Complex { real, imag }, + value: ast::Number::Complex { real, imag }, }) } - ConstantLiteral::Ellipsis => ruff::Expr::EllipsisLiteral(ruff::ExprEllipsisLiteral { + ConstantLiteral::Ellipsis => ast::Expr::EllipsisLiteral(ast::ExprEllipsisLiteral { node_index: Default::default(), range, }), @@ -348,17 +348,17 @@ fn constant_to_ruff_expr(value: Constant) -> ruff::Expr { pub(super) fn number_literal_to_object( vm: &VirtualMachine, source_file: &SourceFile, - constant: ruff::ExprNumberLiteral, + constant: ast::ExprNumberLiteral, ) -> PyObjectRef { - let ruff::ExprNumberLiteral { + let ast::ExprNumberLiteral { node_index: _, range, value, } = constant; let c = match value { - ruff::Number::Int(n) => Constant::new_int(n, range), - ruff::Number::Float(n) => Constant::new_float(n, range), - ruff::Number::Complex { real, imag } => Constant::new_complex(real, imag, range), + ast::Number::Int(n) => Constant::new_int(n, range), + ast::Number::Float(n) => Constant::new_float(n, range), + ast::Number::Complex { real, imag } => Constant::new_complex(real, imag, range), }; c.ast_to_object(vm, source_file) } @@ -366,9 +366,9 @@ pub(super) fn number_literal_to_object( pub(super) fn string_literal_to_object( vm: &VirtualMachine, source_file: &SourceFile, - constant: ruff::ExprStringLiteral, + constant: ast::ExprStringLiteral, ) -> PyObjectRef { - let ruff::ExprStringLiteral { + let ast::ExprStringLiteral { node_index: _, range, value, @@ -384,9 +384,9 @@ pub(super) fn string_literal_to_object( pub(super) fn bytes_literal_to_object( vm: &VirtualMachine, source_file: &SourceFile, - constant: ruff::ExprBytesLiteral, + constant: ast::ExprBytesLiteral, ) -> PyObjectRef { - let ruff::ExprBytesLiteral { + let ast::ExprBytesLiteral { node_index: _, range, value, @@ -399,9 +399,9 @@ pub(super) fn bytes_literal_to_object( pub(super) fn boolean_literal_to_object( vm: &VirtualMachine, source_file: &SourceFile, - constant: ruff::ExprBooleanLiteral, + constant: ast::ExprBooleanLiteral, ) -> PyObjectRef { - let ruff::ExprBooleanLiteral { + let ast::ExprBooleanLiteral { node_index: _, range, value, @@ -413,9 +413,9 @@ pub(super) fn boolean_literal_to_object( pub(super) fn none_literal_to_object( vm: &VirtualMachine, source_file: &SourceFile, - constant: ruff::ExprNoneLiteral, + constant: ast::ExprNoneLiteral, ) -> PyObjectRef { - let ruff::ExprNoneLiteral { + let ast::ExprNoneLiteral { node_index: _, range, } = constant; @@ -426,9 +426,9 @@ pub(super) fn none_literal_to_object( pub(super) fn ellipsis_literal_to_object( vm: &VirtualMachine, source_file: &SourceFile, - constant: ruff::ExprEllipsisLiteral, + constant: ast::ExprEllipsisLiteral, ) -> PyObjectRef { - let ruff::ExprEllipsisLiteral { + let ast::ExprEllipsisLiteral { node_index: _, range, } = constant; diff --git a/crates/vm/src/stdlib/ast/elif_else_clause.rs b/crates/vm/src/stdlib/ast/elif_else_clause.rs index e2a8789dd08..b27e956077e 100644 --- a/crates/vm/src/stdlib/ast/elif_else_clause.rs +++ b/crates/vm/src/stdlib/ast/elif_else_clause.rs @@ -2,12 +2,12 @@ use super::*; use rustpython_compiler_core::SourceFile; pub(super) fn ast_to_object( - clause: ruff::ElifElseClause, - mut rest: alloc::vec::IntoIter<ruff::ElifElseClause>, + clause: ast::ElifElseClause, + mut rest: alloc::vec::IntoIter<ast::ElifElseClause>, vm: &VirtualMachine, source_file: &SourceFile, ) -> PyObjectRef { - let ruff::ElifElseClause { + let ast::ElifElseClause { node_index: _, range, test, @@ -48,18 +48,18 @@ pub(super) fn ast_from_object( vm: &VirtualMachine, source_file: &SourceFile, object: PyObjectRef, -) -> PyResult<ruff::StmtIf> { +) -> PyResult<ast::StmtIf> { let test = Node::ast_from_object(vm, source_file, get_node_field(vm, &object, "test", "If")?)?; let body = Node::ast_from_object(vm, source_file, get_node_field(vm, &object, "body", "If")?)?; - let orelse: Vec<ruff::Stmt> = Node::ast_from_object( + let orelse: Vec<ast::Stmt> = Node::ast_from_object( vm, source_file, get_node_field(vm, &object, "orelse", "If")?, )?; let range = range_from_object(vm, source_file, object, "If")?; - let elif_else_clauses = if let [ruff::Stmt::If(_)] = &*orelse { - let Some(ruff::Stmt::If(ruff::StmtIf { + let elif_else_clauses = if let [ast::Stmt::If(_)] = &*orelse { + let Some(ast::Stmt::If(ast::StmtIf { node_index: _, range, test, @@ -71,7 +71,7 @@ pub(super) fn ast_from_object( }; elif_else_clauses.insert( 0, - ruff::ElifElseClause { + ast::ElifElseClause { node_index: Default::default(), range, test: Some(*test), @@ -80,7 +80,7 @@ pub(super) fn ast_from_object( ); elif_else_clauses } else { - vec![ruff::ElifElseClause { + vec![ast::ElifElseClause { node_index: Default::default(), range, test: None, @@ -88,7 +88,7 @@ pub(super) fn ast_from_object( }] }; - Ok(ruff::StmtIf { + Ok(ast::StmtIf { node_index: Default::default(), test, body, diff --git a/crates/vm/src/stdlib/ast/exception.rs b/crates/vm/src/stdlib/ast/exception.rs index b5b3ca2709a..bdb8b7ad9ac 100644 --- a/crates/vm/src/stdlib/ast/exception.rs +++ b/crates/vm/src/stdlib/ast/exception.rs @@ -2,7 +2,7 @@ use super::*; use rustpython_compiler_core::SourceFile; // sum -impl Node for ruff::ExceptHandler { +impl Node for ast::ExceptHandler { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { match self { Self::ExceptHandler(cons) => cons.ast_to_object(vm, source_file), @@ -16,7 +16,7 @@ impl Node for ruff::ExceptHandler { let _cls = _object.class(); Ok( if _cls.is(pyast::NodeExceptHandlerExceptHandler::static_type()) { - Self::ExceptHandler(ruff::ExceptHandlerExceptHandler::ast_from_object( + Self::ExceptHandler(ast::ExceptHandlerExceptHandler::ast_from_object( _vm, source_file, _object, @@ -32,7 +32,7 @@ impl Node for ruff::ExceptHandler { } // constructor -impl Node for ruff::ExceptHandlerExceptHandler { +impl Node for ast::ExceptHandlerExceptHandler { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, diff --git a/crates/vm/src/stdlib/ast/expression.rs b/crates/vm/src/stdlib/ast/expression.rs index 83d77374380..3bf1470795d 100644 --- a/crates/vm/src/stdlib/ast/expression.rs +++ b/crates/vm/src/stdlib/ast/expression.rs @@ -7,7 +7,7 @@ use crate::stdlib::ast::{ use rustpython_compiler_core::SourceFile; // sum -impl Node for ruff::Expr { +impl Node for ast::Expr { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { match self { Self::BoolOp(cons) => cons.ast_to_object(vm, source_file), @@ -36,7 +36,7 @@ impl Node for ruff::Expr { Self::NumberLiteral(cons) => constant::number_literal_to_object(vm, source_file, cons), Self::StringLiteral(cons) => constant::string_literal_to_object(vm, source_file, cons), Self::FString(cons) => string::fstring_to_object(vm, source_file, cons), - Self::TString(_) => unimplemented!(), + Self::TString(cons) => string::tstring_to_object(vm, source_file, cons), Self::BytesLiteral(cons) => constant::bytes_literal_to_object(vm, source_file, cons), Self::BooleanLiteral(cons) => { constant::boolean_literal_to_object(vm, source_file, cons) @@ -59,77 +59,69 @@ impl Node for ruff::Expr { ) -> PyResult<Self> { let cls = object.class(); Ok(if cls.is(pyast::NodeExprBoolOp::static_type()) { - Self::BoolOp(ruff::ExprBoolOp::ast_from_object(vm, source_file, object)?) + Self::BoolOp(ast::ExprBoolOp::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprNamedExpr::static_type()) { - Self::Named(ruff::ExprNamed::ast_from_object(vm, source_file, object)?) + Self::Named(ast::ExprNamed::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprBinOp::static_type()) { - Self::BinOp(ruff::ExprBinOp::ast_from_object(vm, source_file, object)?) + Self::BinOp(ast::ExprBinOp::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprUnaryOp::static_type()) { - Self::UnaryOp(ruff::ExprUnaryOp::ast_from_object(vm, source_file, object)?) + Self::UnaryOp(ast::ExprUnaryOp::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprLambda::static_type()) { - Self::Lambda(ruff::ExprLambda::ast_from_object(vm, source_file, object)?) + Self::Lambda(ast::ExprLambda::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprIfExp::static_type()) { - Self::If(ruff::ExprIf::ast_from_object(vm, source_file, object)?) + Self::If(ast::ExprIf::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprDict::static_type()) { - Self::Dict(ruff::ExprDict::ast_from_object(vm, source_file, object)?) + Self::Dict(ast::ExprDict::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprSet::static_type()) { - Self::Set(ruff::ExprSet::ast_from_object(vm, source_file, object)?) + Self::Set(ast::ExprSet::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprListComp::static_type()) { - Self::ListComp(ruff::ExprListComp::ast_from_object( - vm, - source_file, - object, - )?) + Self::ListComp(ast::ExprListComp::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprSetComp::static_type()) { - Self::SetComp(ruff::ExprSetComp::ast_from_object(vm, source_file, object)?) + Self::SetComp(ast::ExprSetComp::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprDictComp::static_type()) { - Self::DictComp(ruff::ExprDictComp::ast_from_object( - vm, - source_file, - object, - )?) + Self::DictComp(ast::ExprDictComp::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprGeneratorExp::static_type()) { - Self::Generator(ruff::ExprGenerator::ast_from_object( + Self::Generator(ast::ExprGenerator::ast_from_object( vm, source_file, object, )?) } else if cls.is(pyast::NodeExprAwait::static_type()) { - Self::Await(ruff::ExprAwait::ast_from_object(vm, source_file, object)?) + Self::Await(ast::ExprAwait::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprYield::static_type()) { - Self::Yield(ruff::ExprYield::ast_from_object(vm, source_file, object)?) + Self::Yield(ast::ExprYield::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprYieldFrom::static_type()) { - Self::YieldFrom(ruff::ExprYieldFrom::ast_from_object( + Self::YieldFrom(ast::ExprYieldFrom::ast_from_object( vm, source_file, object, )?) } else if cls.is(pyast::NodeExprCompare::static_type()) { - Self::Compare(ruff::ExprCompare::ast_from_object(vm, source_file, object)?) + Self::Compare(ast::ExprCompare::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprCall::static_type()) { - Self::Call(ruff::ExprCall::ast_from_object(vm, source_file, object)?) + Self::Call(ast::ExprCall::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprAttribute::static_type()) { - Self::Attribute(ruff::ExprAttribute::ast_from_object( + Self::Attribute(ast::ExprAttribute::ast_from_object( vm, source_file, object, )?) } else if cls.is(pyast::NodeExprSubscript::static_type()) { - Self::Subscript(ruff::ExprSubscript::ast_from_object( + Self::Subscript(ast::ExprSubscript::ast_from_object( vm, source_file, object, )?) } else if cls.is(pyast::NodeExprStarred::static_type()) { - Self::Starred(ruff::ExprStarred::ast_from_object(vm, source_file, object)?) + Self::Starred(ast::ExprStarred::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprName::static_type()) { - Self::Name(ruff::ExprName::ast_from_object(vm, source_file, object)?) + Self::Name(ast::ExprName::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprList::static_type()) { - Self::List(ruff::ExprList::ast_from_object(vm, source_file, object)?) + Self::List(ast::ExprList::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprTuple::static_type()) { - Self::Tuple(ruff::ExprTuple::ast_from_object(vm, source_file, object)?) + Self::Tuple(ast::ExprTuple::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprSlice::static_type()) { - Self::Slice(ruff::ExprSlice::ast_from_object(vm, source_file, object)?) + Self::Slice(ast::ExprSlice::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeExprConstant::static_type()) { Constant::ast_from_object(vm, source_file, object)?.into_expr() } else if cls.is(pyast::NodeExprJoinedStr::static_type()) { @@ -144,7 +136,7 @@ impl Node for ruff::Expr { } // constructor -impl Node for ruff::ExprBoolOp { +impl Node for ast::ExprBoolOp { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -187,7 +179,7 @@ impl Node for ruff::ExprBoolOp { } // constructor -impl Node for ruff::ExprNamed { +impl Node for ast::ExprNamed { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -230,7 +222,7 @@ impl Node for ruff::ExprNamed { } // constructor -impl Node for ruff::ExprBinOp { +impl Node for ast::ExprBinOp { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -281,7 +273,7 @@ impl Node for ruff::ExprBinOp { } // constructor -impl Node for ruff::ExprUnaryOp { +impl Node for ast::ExprUnaryOp { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -323,7 +315,7 @@ impl Node for ruff::ExprUnaryOp { } // constructor -impl Node for ruff::ExprLambda { +impl Node for ast::ExprLambda { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -366,7 +358,7 @@ impl Node for ruff::ExprLambda { } // constructor -impl Node for ruff::ExprIf { +impl Node for ast::ExprIf { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -417,7 +409,7 @@ impl Node for ruff::ExprIf { } // constructor -impl Node for ruff::ExprDict { +impl Node for ast::ExprDict { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -449,7 +441,7 @@ impl Node for ruff::ExprDict { source_file: &SourceFile, object: PyObjectRef, ) -> PyResult<Self> { - let keys: Vec<Option<ruff::Expr>> = Node::ast_from_object( + let keys: Vec<Option<ast::Expr>> = Node::ast_from_object( vm, source_file, get_node_field(vm, &object, "keys", "Dict")?, @@ -462,7 +454,7 @@ impl Node for ruff::ExprDict { let items = keys .into_iter() .zip(values) - .map(|(key, value)| ruff::DictItem { key, value }) + .map(|(key, value)| ast::DictItem { key, value }) .collect(); Ok(Self { node_index: Default::default(), @@ -473,7 +465,7 @@ impl Node for ruff::ExprDict { } // constructor -impl Node for ruff::ExprSet { +impl Node for ast::ExprSet { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -507,7 +499,7 @@ impl Node for ruff::ExprSet { } // constructor -impl Node for ruff::ExprListComp { +impl Node for ast::ExprListComp { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -550,7 +542,7 @@ impl Node for ruff::ExprListComp { } // constructor -impl Node for ruff::ExprSetComp { +impl Node for ast::ExprSetComp { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -593,7 +585,7 @@ impl Node for ruff::ExprSetComp { } // constructor -impl Node for ruff::ExprDictComp { +impl Node for ast::ExprDictComp { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -644,7 +636,7 @@ impl Node for ruff::ExprDictComp { } // constructor -impl Node for ruff::ExprGenerator { +impl Node for ast::ExprGenerator { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -690,7 +682,7 @@ impl Node for ruff::ExprGenerator { } // constructor -impl Node for ruff::ExprAwait { +impl Node for ast::ExprAwait { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -724,7 +716,7 @@ impl Node for ruff::ExprAwait { } // constructor -impl Node for ruff::ExprYield { +impl Node for ast::ExprYield { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -757,7 +749,7 @@ impl Node for ruff::ExprYield { } // constructor -impl Node for ruff::ExprYieldFrom { +impl Node for ast::ExprYieldFrom { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -792,7 +784,7 @@ impl Node for ruff::ExprYieldFrom { } // constructor -impl Node for ruff::ExprCompare { +impl Node for ast::ExprCompare { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -853,7 +845,7 @@ impl Node for ruff::ExprCompare { } // constructor -impl Node for ruff::ExprCall { +impl Node for ast::ExprCall { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -914,7 +906,7 @@ impl Node for ruff::ExprCall { } // constructor -impl Node for ruff::ExprAttribute { +impl Node for ast::ExprAttribute { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -965,7 +957,7 @@ impl Node for ruff::ExprAttribute { } // constructor -impl Node for ruff::ExprSubscript { +impl Node for ast::ExprSubscript { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -1015,7 +1007,7 @@ impl Node for ruff::ExprSubscript { } // constructor -impl Node for ruff::ExprStarred { +impl Node for ast::ExprStarred { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -1057,7 +1049,7 @@ impl Node for ruff::ExprStarred { } // constructor -impl Node for ruff::ExprName { +impl Node for ast::ExprName { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -1095,7 +1087,7 @@ impl Node for ruff::ExprName { } // constructor -impl Node for ruff::ExprList { +impl Node for ast::ExprList { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -1138,7 +1130,7 @@ impl Node for ruff::ExprList { } // constructor -impl Node for ruff::ExprTuple { +impl Node for ast::ExprTuple { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -1183,7 +1175,7 @@ impl Node for ruff::ExprTuple { } // constructor -impl Node for ruff::ExprSlice { +impl Node for ast::ExprSlice { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -1228,7 +1220,7 @@ impl Node for ruff::ExprSlice { } // sum -impl Node for ruff::ExprContext { +impl Node for ast::ExprContext { fn ast_to_object(self, vm: &VirtualMachine, _source_file: &SourceFile) -> PyObjectRef { let node_type = match self { Self::Load => pyast::NodeExprContextLoad::static_type(), @@ -1266,7 +1258,7 @@ impl Node for ruff::ExprContext { } // product -impl Node for ruff::Comprehension { +impl Node for ast::Comprehension { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, diff --git a/crates/vm/src/stdlib/ast/module.rs b/crates/vm/src/stdlib/ast/module.rs index 6fae8f10a33..78f897b8930 100644 --- a/crates/vm/src/stdlib/ast/module.rs +++ b/crates/vm/src/stdlib/ast/module.rs @@ -18,9 +18,9 @@ use rustpython_compiler_core::SourceFile; /// - `FunctionType`: A function signature with argument and return type /// annotations, representing the type hints of a function (e.g., `def add(x: int, y: int) -> int`). pub(super) enum Mod { - Module(ruff::ModModule), + Module(ast::ModModule), Interactive(ModInteractive), - Expression(ruff::ModExpression), + Expression(ast::ModExpression), FunctionType(ModFunctionType), } @@ -42,11 +42,11 @@ impl Node for Mod { ) -> PyResult<Self> { let cls = object.class(); Ok(if cls.is(pyast::NodeModModule::static_type()) { - Self::Module(ruff::ModModule::ast_from_object(vm, source_file, object)?) + Self::Module(ast::ModModule::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeModInteractive::static_type()) { Self::Interactive(ModInteractive::ast_from_object(vm, source_file, object)?) } else if cls.is(pyast::NodeModExpression::static_type()) { - Self::Expression(ruff::ModExpression::ast_from_object( + Self::Expression(ast::ModExpression::ast_from_object( vm, source_file, object, @@ -63,7 +63,7 @@ impl Node for Mod { } // constructor -impl Node for ruff::ModModule { +impl Node for ast::ModModule { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -113,7 +113,7 @@ impl Node for ruff::ModModule { pub(super) struct ModInteractive { pub(crate) range: TextRange, - pub(crate) body: Vec<ruff::Stmt>, + pub(crate) body: Vec<ast::Stmt>, } // constructor @@ -147,7 +147,7 @@ impl Node for ModInteractive { } // constructor -impl Node for ruff::ModExpression { +impl Node for ast::ModExpression { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -182,8 +182,8 @@ impl Node for ruff::ModExpression { } pub(super) struct ModFunctionType { - pub(crate) argtypes: Box<[ruff::Expr]>, - pub(crate) returns: ruff::Expr, + pub(crate) argtypes: Box<[ast::Expr]>, + pub(crate) returns: ast::Expr, pub(crate) range: TextRange, } diff --git a/crates/vm/src/stdlib/ast/operator.rs b/crates/vm/src/stdlib/ast/operator.rs index c394152da2c..23aa63c7031 100644 --- a/crates/vm/src/stdlib/ast/operator.rs +++ b/crates/vm/src/stdlib/ast/operator.rs @@ -2,7 +2,7 @@ use super::*; use rustpython_compiler_core::SourceFile; // sum -impl Node for ruff::BoolOp { +impl Node for ast::BoolOp { fn ast_to_object(self, vm: &VirtualMachine, _source_file: &SourceFile) -> PyObjectRef { let node_type = match self { Self::And => pyast::NodeBoolOpAnd::static_type(), @@ -34,7 +34,7 @@ impl Node for ruff::BoolOp { } // sum -impl Node for ruff::Operator { +impl Node for ast::Operator { fn ast_to_object(self, vm: &VirtualMachine, _source_file: &SourceFile) -> PyObjectRef { let node_type = match self { Self::Add => pyast::NodeOperatorAdd::static_type(), @@ -99,7 +99,7 @@ impl Node for ruff::Operator { } // sum -impl Node for ruff::UnaryOp { +impl Node for ast::UnaryOp { fn ast_to_object(self, vm: &VirtualMachine, _source_file: &SourceFile) -> PyObjectRef { let node_type = match self { Self::Invert => pyast::NodeUnaryOpInvert::static_type(), @@ -137,7 +137,7 @@ impl Node for ruff::UnaryOp { } // sum -impl Node for ruff::CmpOp { +impl Node for ast::CmpOp { fn ast_to_object(self, vm: &VirtualMachine, _source_file: &SourceFile) -> PyObjectRef { let node_type = match self { Self::Eq => pyast::NodeCmpOpEq::static_type(), diff --git a/crates/vm/src/stdlib/ast/other.rs b/crates/vm/src/stdlib/ast/other.rs index ce7d5fe4807..8a89a740682 100644 --- a/crates/vm/src/stdlib/ast/other.rs +++ b/crates/vm/src/stdlib/ast/other.rs @@ -1,7 +1,7 @@ use super::*; use rustpython_compiler_core::SourceFile; -impl Node for ruff::ConversionFlag { +impl Node for ast::ConversionFlag { fn ast_to_object(self, vm: &VirtualMachine, _source_file: &SourceFile) -> PyObjectRef { vm.ctx.new_int(self as u8).into() } @@ -24,7 +24,7 @@ impl Node for ruff::ConversionFlag { } // /// This is just a string, not strictly an AST node. But it makes AST conversions easier. -impl Node for ruff::name::Name { +impl Node for ast::name::Name { fn ast_to_object(self, vm: &VirtualMachine, _source_file: &SourceFile) -> PyObjectRef { vm.ctx.new_str(self.as_str()).to_pyobject(vm) } @@ -41,9 +41,9 @@ impl Node for ruff::name::Name { } } -impl Node for ruff::Decorator { +impl Node for ast::Decorator { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { - ruff::Expr::ast_to_object(self.expression, vm, source_file) + ast::Expr::ast_to_object(self.expression, vm, source_file) } fn ast_from_object( @@ -51,7 +51,7 @@ impl Node for ruff::Decorator { source_file: &SourceFile, object: PyObjectRef, ) -> PyResult<Self> { - let expression = ruff::Expr::ast_from_object(vm, source_file, object)?; + let expression = ast::Expr::ast_from_object(vm, source_file, object)?; let range = expression.range(); Ok(Self { node_index: Default::default(), @@ -62,7 +62,7 @@ impl Node for ruff::Decorator { } // product -impl Node for ruff::Alias { +impl Node for ast::Alias { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -103,7 +103,7 @@ impl Node for ruff::Alias { } // product -impl Node for ruff::WithItem { +impl Node for ast::WithItem { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, diff --git a/crates/vm/src/stdlib/ast/parameter.rs b/crates/vm/src/stdlib/ast/parameter.rs index 44fcbb2b464..1e411d41ab6 100644 --- a/crates/vm/src/stdlib/ast/parameter.rs +++ b/crates/vm/src/stdlib/ast/parameter.rs @@ -2,7 +2,7 @@ use super::*; use rustpython_compiler_core::SourceFile; // product -impl Node for ruff::Parameters { +impl Node for ast::Parameters { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -101,7 +101,7 @@ impl Node for ruff::Parameters { } // product -impl Node for ruff::Parameter { +impl Node for ast::Parameter { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -156,7 +156,7 @@ impl Node for ruff::Parameter { } // product -impl Node for ruff::Keyword { +impl Node for ast::Keyword { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -197,7 +197,7 @@ impl Node for ruff::Keyword { struct PositionalParameters { pub _range: TextRange, // TODO: Use this - pub args: Box<[ruff::Parameter]>, + pub args: Box<[ast::Parameter]>, } impl Node for PositionalParameters { @@ -220,7 +220,7 @@ impl Node for PositionalParameters { struct KeywordParameters { pub _range: TextRange, // TODO: Use this - pub keywords: Box<[ruff::Parameter]>, + pub keywords: Box<[ast::Parameter]>, } impl Node for KeywordParameters { @@ -243,7 +243,7 @@ impl Node for KeywordParameters { struct ParameterDefaults { pub _range: TextRange, // TODO: Use this - defaults: Box<[Option<Box<ruff::Expr>>]>, + defaults: Box<[Option<Box<ast::Expr>>]>, } impl Node for ParameterDefaults { @@ -265,8 +265,8 @@ impl Node for ParameterDefaults { } fn extract_positional_parameter_defaults( - pos_only_args: Vec<ruff::ParameterWithDefault>, - args: Vec<ruff::ParameterWithDefault>, + pos_only_args: Vec<ast::ParameterWithDefault>, + args: Vec<ast::ParameterWithDefault>, ) -> ( PositionalParameters, PositionalParameters, @@ -325,15 +325,15 @@ fn merge_positional_parameter_defaults( args: PositionalParameters, defaults: ParameterDefaults, ) -> ( - Vec<ruff::ParameterWithDefault>, - Vec<ruff::ParameterWithDefault>, + Vec<ast::ParameterWithDefault>, + Vec<ast::ParameterWithDefault>, ) { let posonlyargs = posonlyargs.args; let args = args.args; let defaults = defaults.defaults; let mut posonlyargs: Vec<_> = <Box<[_]> as IntoIterator>::into_iter(posonlyargs) - .map(|parameter| ruff::ParameterWithDefault { + .map(|parameter| ast::ParameterWithDefault { node_index: Default::default(), range: Default::default(), parameter, @@ -341,7 +341,7 @@ fn merge_positional_parameter_defaults( }) .collect(); let mut args: Vec<_> = <Box<[_]> as IntoIterator>::into_iter(args) - .map(|parameter| ruff::ParameterWithDefault { + .map(|parameter| ast::ParameterWithDefault { node_index: Default::default(), range: Default::default(), parameter, @@ -366,7 +366,7 @@ fn merge_positional_parameter_defaults( } fn extract_keyword_parameter_defaults( - kw_only_args: Vec<ruff::ParameterWithDefault>, + kw_only_args: Vec<ast::ParameterWithDefault>, ) -> (KeywordParameters, ParameterDefaults) { let mut defaults = vec![]; defaults.extend(kw_only_args.iter().map(|item| item.default.clone())); @@ -402,9 +402,9 @@ fn extract_keyword_parameter_defaults( fn merge_keyword_parameter_defaults( kw_only_args: KeywordParameters, defaults: ParameterDefaults, -) -> Vec<ruff::ParameterWithDefault> { +) -> Vec<ast::ParameterWithDefault> { core::iter::zip(kw_only_args.keywords, defaults.defaults) - .map(|(parameter, default)| ruff::ParameterWithDefault { + .map(|(parameter, default)| ast::ParameterWithDefault { node_index: Default::default(), parameter, default, diff --git a/crates/vm/src/stdlib/ast/pattern.rs b/crates/vm/src/stdlib/ast/pattern.rs index d8128cb0622..4531a989cb3 100644 --- a/crates/vm/src/stdlib/ast/pattern.rs +++ b/crates/vm/src/stdlib/ast/pattern.rs @@ -2,7 +2,7 @@ use super::*; use rustpython_compiler_core::SourceFile; // product -impl Node for ruff::MatchCase { +impl Node for ast::MatchCase { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -50,7 +50,7 @@ impl Node for ruff::MatchCase { } // sum -impl Node for ruff::Pattern { +impl Node for ast::Pattern { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { match self { Self::MatchValue(cons) => cons.ast_to_object(vm, source_file), @@ -70,49 +70,49 @@ impl Node for ruff::Pattern { ) -> PyResult<Self> { let _cls = _object.class(); Ok(if _cls.is(pyast::NodePatternMatchValue::static_type()) { - Self::MatchValue(ruff::PatternMatchValue::ast_from_object( + Self::MatchValue(ast::PatternMatchValue::ast_from_object( _vm, source_file, _object, )?) } else if _cls.is(pyast::NodePatternMatchSingleton::static_type()) { - Self::MatchSingleton(ruff::PatternMatchSingleton::ast_from_object( + Self::MatchSingleton(ast::PatternMatchSingleton::ast_from_object( _vm, source_file, _object, )?) } else if _cls.is(pyast::NodePatternMatchSequence::static_type()) { - Self::MatchSequence(ruff::PatternMatchSequence::ast_from_object( + Self::MatchSequence(ast::PatternMatchSequence::ast_from_object( _vm, source_file, _object, )?) } else if _cls.is(pyast::NodePatternMatchMapping::static_type()) { - Self::MatchMapping(ruff::PatternMatchMapping::ast_from_object( + Self::MatchMapping(ast::PatternMatchMapping::ast_from_object( _vm, source_file, _object, )?) } else if _cls.is(pyast::NodePatternMatchClass::static_type()) { - Self::MatchClass(ruff::PatternMatchClass::ast_from_object( + Self::MatchClass(ast::PatternMatchClass::ast_from_object( _vm, source_file, _object, )?) } else if _cls.is(pyast::NodePatternMatchStar::static_type()) { - Self::MatchStar(ruff::PatternMatchStar::ast_from_object( + Self::MatchStar(ast::PatternMatchStar::ast_from_object( _vm, source_file, _object, )?) } else if _cls.is(pyast::NodePatternMatchAs::static_type()) { - Self::MatchAs(ruff::PatternMatchAs::ast_from_object( + Self::MatchAs(ast::PatternMatchAs::ast_from_object( _vm, source_file, _object, )?) } else if _cls.is(pyast::NodePatternMatchOr::static_type()) { - Self::MatchOr(ruff::PatternMatchOr::ast_from_object( + Self::MatchOr(ast::PatternMatchOr::ast_from_object( _vm, source_file, _object, @@ -126,7 +126,7 @@ impl Node for ruff::Pattern { } } // constructor -impl Node for ruff::PatternMatchValue { +impl Node for ast::PatternMatchValue { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -161,7 +161,7 @@ impl Node for ruff::PatternMatchValue { } // constructor -impl Node for ruff::PatternMatchSingleton { +impl Node for ast::PatternMatchSingleton { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -198,12 +198,12 @@ impl Node for ruff::PatternMatchSingleton { } } -impl Node for ruff::Singleton { +impl Node for ast::Singleton { fn ast_to_object(self, vm: &VirtualMachine, _source_file: &SourceFile) -> PyObjectRef { match self { - ruff::Singleton::None => vm.ctx.none(), - ruff::Singleton::True => vm.ctx.new_bool(true).into(), - ruff::Singleton::False => vm.ctx.new_bool(false).into(), + ast::Singleton::None => vm.ctx.none(), + ast::Singleton::True => vm.ctx.new_bool(true).into(), + ast::Singleton::False => vm.ctx.new_bool(false).into(), } } @@ -213,11 +213,11 @@ impl Node for ruff::Singleton { object: PyObjectRef, ) -> PyResult<Self> { if vm.is_none(&object) { - Ok(ruff::Singleton::None) + Ok(ast::Singleton::None) } else if object.is(&vm.ctx.true_value) { - Ok(ruff::Singleton::True) + Ok(ast::Singleton::True) } else if object.is(&vm.ctx.false_value) { - Ok(ruff::Singleton::False) + Ok(ast::Singleton::False) } else { Err(vm.new_value_error(format!( "Expected None, True, or False, got {:?}", @@ -228,7 +228,7 @@ impl Node for ruff::Singleton { } // constructor -impl Node for ruff::PatternMatchSequence { +impl Node for ast::PatternMatchSequence { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -266,7 +266,7 @@ impl Node for ruff::PatternMatchSequence { } // constructor -impl Node for ruff::PatternMatchMapping { +impl Node for ast::PatternMatchMapping { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -318,7 +318,7 @@ impl Node for ruff::PatternMatchMapping { } // constructor -impl Node for ruff::PatternMatchClass { +impl Node for ast::PatternMatchClass { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -377,7 +377,7 @@ impl Node for ruff::PatternMatchClass { get_node_field(vm, &object, "cls", "MatchClass")?, )?, range: range_from_object(vm, source_file, object, "MatchClass")?, - arguments: ruff::PatternArguments { + arguments: ast::PatternArguments { node_index: Default::default(), range: Default::default(), patterns, @@ -387,7 +387,7 @@ impl Node for ruff::PatternMatchClass { } } -struct PatternMatchClassPatterns(Vec<ruff::Pattern>); +struct PatternMatchClassPatterns(Vec<ast::Pattern>); impl Node for PatternMatchClassPatterns { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { @@ -403,7 +403,7 @@ impl Node for PatternMatchClassPatterns { } } -struct PatternMatchClassKeywordAttributes(Vec<ruff::Identifier>); +struct PatternMatchClassKeywordAttributes(Vec<ast::Identifier>); impl Node for PatternMatchClassKeywordAttributes { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { @@ -419,7 +419,7 @@ impl Node for PatternMatchClassKeywordAttributes { } } -struct PatternMatchClassKeywordPatterns(Vec<ruff::Pattern>); +struct PatternMatchClassKeywordPatterns(Vec<ast::Pattern>); impl Node for PatternMatchClassKeywordPatterns { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { @@ -435,7 +435,7 @@ impl Node for PatternMatchClassKeywordPatterns { } } // constructor -impl Node for ruff::PatternMatchStar { +impl Node for ast::PatternMatchStar { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -468,7 +468,7 @@ impl Node for ruff::PatternMatchStar { } // constructor -impl Node for ruff::PatternMatchAs { +impl Node for ast::PatternMatchAs { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -507,7 +507,7 @@ impl Node for ruff::PatternMatchAs { } // constructor -impl Node for ruff::PatternMatchOr { +impl Node for ast::PatternMatchOr { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -541,7 +541,7 @@ impl Node for ruff::PatternMatchOr { } fn split_pattern_match_class( - arguments: ruff::PatternArguments, + arguments: ast::PatternArguments, ) -> ( PatternMatchClassPatterns, PatternMatchClassKeywordAttributes, @@ -562,12 +562,12 @@ fn merge_pattern_match_class( patterns: PatternMatchClassPatterns, kwd_attrs: PatternMatchClassKeywordAttributes, kwd_patterns: PatternMatchClassKeywordPatterns, -) -> (Vec<ruff::Pattern>, Vec<ruff::PatternKeyword>) { +) -> (Vec<ast::Pattern>, Vec<ast::PatternKeyword>) { let keywords = kwd_attrs .0 .into_iter() .zip(kwd_patterns.0) - .map(|(attr, pattern)| ruff::PatternKeyword { + .map(|(attr, pattern)| ast::PatternKeyword { range: Default::default(), node_index: Default::default(), attr, diff --git a/crates/vm/src/stdlib/ast/pyast.rs b/crates/vm/src/stdlib/ast/pyast.rs index e36635fe4b9..2131df29b96 100644 --- a/crates/vm/src/stdlib/ast/pyast.rs +++ b/crates/vm/src/stdlib/ast/pyast.rs @@ -2,6 +2,8 @@ use super::*; use crate::common::ascii; +use crate::function::FuncArgs; +use crate::types::Initializer; macro_rules! impl_node { ( @@ -449,12 +451,68 @@ impl_node!( ); impl_node!( - #[pyclass(module = "_ast", name = "Constant", base = NodeExpr)] - pub(crate) struct NodeExprConstant, - fields: ["value", "kind"], + #[pyclass(module = "_ast", name = "TemplateStr", base = NodeExpr)] + pub(crate) struct NodeExprTemplateStr, + fields: ["values"], + attributes: ["lineno", "col_offset", "end_lineno", "end_col_offset"], +); + +impl_node!( + #[pyclass(module = "_ast", name = "Interpolation", base = NodeExpr)] + pub(crate) struct NodeExprInterpolation, + fields: ["value", "str", "conversion", "format_spec"], attributes: ["lineno", "col_offset", "end_lineno", "end_col_offset"], ); +// NodeExprConstant needs custom Initializer to default kind to None +#[pyclass(module = "_ast", name = "Constant", base = NodeExpr)] +#[repr(transparent)] +pub(crate) struct NodeExprConstant(NodeExpr); + +#[pyclass(flags(HAS_DICT, BASETYPE), with(Initializer))] +impl NodeExprConstant { + #[extend_class] + fn extend_class_with_fields(ctx: &Context, class: &'static Py<PyType>) { + class.set_attr( + identifier!(ctx, _fields), + ctx.new_tuple(vec![ + ctx.new_str(ascii!("value")).into(), + ctx.new_str(ascii!("kind")).into(), + ]) + .into(), + ); + + class.set_attr( + identifier!(ctx, _attributes), + ctx.new_list(vec![ + ctx.new_str(ascii!("lineno")).into(), + ctx.new_str(ascii!("col_offset")).into(), + ctx.new_str(ascii!("end_lineno")).into(), + ctx.new_str(ascii!("end_col_offset")).into(), + ]) + .into(), + ); + } +} + +impl Initializer for NodeExprConstant { + type Args = FuncArgs; + + fn slot_init(zelf: PyObjectRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult<()> { + <NodeAst as Initializer>::slot_init(zelf.clone(), args, vm)?; + // kind defaults to None if not provided + let dict = zelf.as_object().dict().unwrap(); + if !dict.contains_key("kind", vm) { + dict.set_item("kind", vm.ctx.none(), vm)?; + } + Ok(()) + } + + fn init(_zelf: PyRef<Self>, _args: Self::Args, _vm: &VirtualMachine) -> PyResult<()> { + unreachable!("slot_init is defined") + } +} + impl_node!( #[pyclass(module = "_ast", name = "Attribute", base = NodeExpr)] pub(crate) struct NodeExprAttribute, @@ -918,6 +976,8 @@ pub fn extend_module_nodes(vm: &VirtualMachine, module: &Py<PyModule>) { "Call" => NodeExprCall::make_class(&vm.ctx), "FormattedValue" => NodeExprFormattedValue::make_class(&vm.ctx), "JoinedStr" => NodeExprJoinedStr::make_class(&vm.ctx), + "TemplateStr" => NodeExprTemplateStr::make_class(&vm.ctx), + "Interpolation" => NodeExprInterpolation::make_class(&vm.ctx), "Constant" => NodeExprConstant::make_class(&vm.ctx), "Attribute" => NodeExprAttribute::make_class(&vm.ctx), "Subscript" => NodeExprSubscript::make_class(&vm.ctx), diff --git a/crates/vm/src/stdlib/ast/python.rs b/crates/vm/src/stdlib/ast/python.rs index 35fe561527b..6c38b00f9ad 100644 --- a/crates/vm/src/stdlib/ast/python.rs +++ b/crates/vm/src/stdlib/ast/python.rs @@ -80,6 +80,7 @@ pub(crate) mod _ast { } zelf.set_attr(vm.ctx.intern_str(key), value, vm)?; } + Ok(()) } @@ -96,4 +97,13 @@ pub(crate) mod _ast { #[pyattr(name = "PyCF_TYPE_COMMENTS")] use super::PY_CF_TYPE_COMMENTS; + + pub(crate) fn module_exec( + vm: &VirtualMachine, + module: &Py<crate::builtins::PyModule>, + ) -> PyResult<()> { + __module_exec(vm, module); + super::super::pyast::extend_module_nodes(vm, module); + Ok(()) + } } diff --git a/crates/vm/src/stdlib/ast/statement.rs b/crates/vm/src/stdlib/ast/statement.rs index f1d36c52e2e..b7bc692dd2e 100644 --- a/crates/vm/src/stdlib/ast/statement.rs +++ b/crates/vm/src/stdlib/ast/statement.rs @@ -3,7 +3,7 @@ use crate::stdlib::ast::argument::{merge_class_def_args, split_class_def_args}; use rustpython_compiler_core::SourceFile; // sum -impl Node for ruff::Stmt { +impl Node for ast::Stmt { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { match self { Self::FunctionDef(cons) => cons.ast_to_object(vm, source_file), @@ -44,117 +44,93 @@ impl Node for ruff::Stmt { ) -> PyResult<Self> { let _cls = _object.class(); Ok(if _cls.is(pyast::NodeStmtFunctionDef::static_type()) { - Self::FunctionDef(ruff::StmtFunctionDef::ast_from_object( + Self::FunctionDef(ast::StmtFunctionDef::ast_from_object( _vm, source_file, _object, )?) } else if _cls.is(pyast::NodeStmtAsyncFunctionDef::static_type()) { - Self::FunctionDef(ruff::StmtFunctionDef::ast_from_object( + Self::FunctionDef(ast::StmtFunctionDef::ast_from_object( _vm, source_file, _object, )?) } else if _cls.is(pyast::NodeStmtClassDef::static_type()) { - Self::ClassDef(ruff::StmtClassDef::ast_from_object( + Self::ClassDef(ast::StmtClassDef::ast_from_object( _vm, source_file, _object, )?) } else if _cls.is(pyast::NodeStmtReturn::static_type()) { - Self::Return(ruff::StmtReturn::ast_from_object( - _vm, - source_file, - _object, - )?) + Self::Return(ast::StmtReturn::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtDelete::static_type()) { - Self::Delete(ruff::StmtDelete::ast_from_object( - _vm, - source_file, - _object, - )?) + Self::Delete(ast::StmtDelete::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtAssign::static_type()) { - Self::Assign(ruff::StmtAssign::ast_from_object( - _vm, - source_file, - _object, - )?) + Self::Assign(ast::StmtAssign::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtTypeAlias::static_type()) { - Self::TypeAlias(ruff::StmtTypeAlias::ast_from_object( + Self::TypeAlias(ast::StmtTypeAlias::ast_from_object( _vm, source_file, _object, )?) } else if _cls.is(pyast::NodeStmtAugAssign::static_type()) { - Self::AugAssign(ruff::StmtAugAssign::ast_from_object( + Self::AugAssign(ast::StmtAugAssign::ast_from_object( _vm, source_file, _object, )?) } else if _cls.is(pyast::NodeStmtAnnAssign::static_type()) { - Self::AnnAssign(ruff::StmtAnnAssign::ast_from_object( + Self::AnnAssign(ast::StmtAnnAssign::ast_from_object( _vm, source_file, _object, )?) } else if _cls.is(pyast::NodeStmtFor::static_type()) { - Self::For(ruff::StmtFor::ast_from_object(_vm, source_file, _object)?) + Self::For(ast::StmtFor::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtAsyncFor::static_type()) { - Self::For(ruff::StmtFor::ast_from_object(_vm, source_file, _object)?) + Self::For(ast::StmtFor::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtWhile::static_type()) { - Self::While(ruff::StmtWhile::ast_from_object(_vm, source_file, _object)?) + Self::While(ast::StmtWhile::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtIf::static_type()) { - Self::If(ruff::StmtIf::ast_from_object(_vm, source_file, _object)?) + Self::If(ast::StmtIf::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtWith::static_type()) { - Self::With(ruff::StmtWith::ast_from_object(_vm, source_file, _object)?) + Self::With(ast::StmtWith::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtAsyncWith::static_type()) { - Self::With(ruff::StmtWith::ast_from_object(_vm, source_file, _object)?) + Self::With(ast::StmtWith::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtMatch::static_type()) { - Self::Match(ruff::StmtMatch::ast_from_object(_vm, source_file, _object)?) + Self::Match(ast::StmtMatch::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtRaise::static_type()) { - Self::Raise(ruff::StmtRaise::ast_from_object(_vm, source_file, _object)?) + Self::Raise(ast::StmtRaise::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtTry::static_type()) { - Self::Try(ruff::StmtTry::ast_from_object(_vm, source_file, _object)?) + Self::Try(ast::StmtTry::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtTryStar::static_type()) { - Self::Try(ruff::StmtTry::ast_from_object(_vm, source_file, _object)?) + Self::Try(ast::StmtTry::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtAssert::static_type()) { - Self::Assert(ruff::StmtAssert::ast_from_object( - _vm, - source_file, - _object, - )?) + Self::Assert(ast::StmtAssert::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtImport::static_type()) { - Self::Import(ruff::StmtImport::ast_from_object( - _vm, - source_file, - _object, - )?) + Self::Import(ast::StmtImport::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtImportFrom::static_type()) { - Self::ImportFrom(ruff::StmtImportFrom::ast_from_object( + Self::ImportFrom(ast::StmtImportFrom::ast_from_object( _vm, source_file, _object, )?) } else if _cls.is(pyast::NodeStmtGlobal::static_type()) { - Self::Global(ruff::StmtGlobal::ast_from_object( - _vm, - source_file, - _object, - )?) + Self::Global(ast::StmtGlobal::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtNonlocal::static_type()) { - Self::Nonlocal(ruff::StmtNonlocal::ast_from_object( + Self::Nonlocal(ast::StmtNonlocal::ast_from_object( _vm, source_file, _object, )?) } else if _cls.is(pyast::NodeStmtExpr::static_type()) { - Self::Expr(ruff::StmtExpr::ast_from_object(_vm, source_file, _object)?) + Self::Expr(ast::StmtExpr::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtPass::static_type()) { - Self::Pass(ruff::StmtPass::ast_from_object(_vm, source_file, _object)?) + Self::Pass(ast::StmtPass::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtBreak::static_type()) { - Self::Break(ruff::StmtBreak::ast_from_object(_vm, source_file, _object)?) + Self::Break(ast::StmtBreak::ast_from_object(_vm, source_file, _object)?) } else if _cls.is(pyast::NodeStmtContinue::static_type()) { - Self::Continue(ruff::StmtContinue::ast_from_object( + Self::Continue(ast::StmtContinue::ast_from_object( _vm, source_file, _object, @@ -169,7 +145,7 @@ impl Node for ruff::Stmt { } // constructor -impl Node for ruff::StmtFunctionDef { +impl Node for ast::StmtFunctionDef { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -211,7 +187,9 @@ impl Node for ruff::StmtFunctionDef { // .unwrap(); dict.set_item( "type_params", - type_params.ast_to_object(vm, source_file), + type_params + .map(|tp| tp.ast_to_object(vm, source_file)) + .unwrap_or_else(|| vm.ctx.new_list(vec![]).into()), vm, ) .unwrap(); @@ -266,7 +244,7 @@ impl Node for ruff::StmtFunctionDef { } // constructor -impl Node for ruff::StmtClassDef { +impl Node for ast::StmtClassDef { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -284,10 +262,22 @@ impl Node for ruff::StmtClassDef { let dict = node.as_object().dict().unwrap(); dict.set_item("name", name.ast_to_object(_vm, source_file), _vm) .unwrap(); - dict.set_item("bases", bases.ast_to_object(_vm, source_file), _vm) - .unwrap(); - dict.set_item("keywords", keywords.ast_to_object(_vm, source_file), _vm) - .unwrap(); + dict.set_item( + "bases", + bases + .map(|b| b.ast_to_object(_vm, source_file)) + .unwrap_or_else(|| _vm.ctx.new_list(vec![]).into()), + _vm, + ) + .unwrap(); + dict.set_item( + "keywords", + keywords + .map(|k| k.ast_to_object(_vm, source_file)) + .unwrap_or_else(|| _vm.ctx.new_list(vec![]).into()), + _vm, + ) + .unwrap(); dict.set_item("body", body.ast_to_object(_vm, source_file), _vm) .unwrap(); dict.set_item( @@ -298,7 +288,9 @@ impl Node for ruff::StmtClassDef { .unwrap(); dict.set_item( "type_params", - type_params.ast_to_object(_vm, source_file), + type_params + .map(|tp| tp.ast_to_object(_vm, source_file)) + .unwrap_or_else(|| _vm.ctx.new_list(vec![]).into()), _vm, ) .unwrap(); @@ -348,7 +340,7 @@ impl Node for ruff::StmtClassDef { } } // constructor -impl Node for ruff::StmtReturn { +impl Node for ast::StmtReturn { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -379,7 +371,7 @@ impl Node for ruff::StmtReturn { } } // constructor -impl Node for ruff::StmtDelete { +impl Node for ast::StmtDelete { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -413,7 +405,7 @@ impl Node for ruff::StmtDelete { } // constructor -impl Node for ruff::StmtAssign { +impl Node for ast::StmtAssign { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -461,7 +453,7 @@ impl Node for ruff::StmtAssign { } // constructor -impl Node for ruff::StmtTypeAlias { +impl Node for ast::StmtTypeAlias { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -516,7 +508,7 @@ impl Node for ruff::StmtTypeAlias { } // constructor -impl Node for ruff::StmtAugAssign { +impl Node for ast::StmtAugAssign { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -566,7 +558,7 @@ impl Node for ruff::StmtAugAssign { } // constructor -impl Node for ruff::StmtAnnAssign { +impl Node for ast::StmtAnnAssign { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -626,7 +618,7 @@ impl Node for ruff::StmtAnnAssign { } // constructor -impl Node for ruff::StmtFor { +impl Node for ast::StmtFor { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -704,7 +696,7 @@ impl Node for ruff::StmtFor { } // constructor -impl Node for ruff::StmtWhile { +impl Node for ast::StmtWhile { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -754,7 +746,7 @@ impl Node for ruff::StmtWhile { } } // constructor -impl Node for ruff::StmtIf { +impl Node for ast::StmtIf { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -764,7 +756,7 @@ impl Node for ruff::StmtIf { elif_else_clauses, } = self; elif_else_clause::ast_to_object( - ruff::ElifElseClause { + ast::ElifElseClause { node_index: Default::default(), range, test: Some(*test), @@ -784,7 +776,7 @@ impl Node for ruff::StmtIf { } } // constructor -impl Node for ruff::StmtWith { +impl Node for ast::StmtWith { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -844,7 +836,7 @@ impl Node for ruff::StmtWith { } } // constructor -impl Node for ruff::StmtMatch { +impl Node for ast::StmtMatch { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -885,7 +877,7 @@ impl Node for ruff::StmtMatch { } } // constructor -impl Node for ruff::StmtRaise { +impl Node for ast::StmtRaise { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -922,7 +914,7 @@ impl Node for ruff::StmtRaise { } } // constructor -impl Node for ruff::StmtTry { +impl Node for ast::StmtTry { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -996,7 +988,7 @@ impl Node for ruff::StmtTry { } } // constructor -impl Node for ruff::StmtAssert { +impl Node for ast::StmtAssert { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -1035,7 +1027,7 @@ impl Node for ruff::StmtAssert { } } // constructor -impl Node for ruff::StmtImport { +impl Node for ast::StmtImport { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -1068,7 +1060,7 @@ impl Node for ruff::StmtImport { } } // constructor -impl Node for ruff::StmtImportFrom { +impl Node for ast::StmtImportFrom { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -1117,7 +1109,7 @@ impl Node for ruff::StmtImportFrom { } } // constructor -impl Node for ruff::StmtGlobal { +impl Node for ast::StmtGlobal { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -1150,7 +1142,7 @@ impl Node for ruff::StmtGlobal { } } // constructor -impl Node for ruff::StmtNonlocal { +impl Node for ast::StmtNonlocal { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -1183,7 +1175,7 @@ impl Node for ruff::StmtNonlocal { } } // constructor -impl Node for ruff::StmtExpr { +impl Node for ast::StmtExpr { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -1216,7 +1208,7 @@ impl Node for ruff::StmtExpr { } } // constructor -impl Node for ruff::StmtPass { +impl Node for ast::StmtPass { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -1241,7 +1233,7 @@ impl Node for ruff::StmtPass { } } // constructor -impl Node for ruff::StmtBreak { +impl Node for ast::StmtBreak { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -1268,7 +1260,7 @@ impl Node for ruff::StmtBreak { } // constructor -impl Node for ruff::StmtContinue { +impl Node for ast::StmtContinue { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, diff --git a/crates/vm/src/stdlib/ast/string.rs b/crates/vm/src/stdlib/ast/string.rs index ffa5a3a958a..2533fb8c6b9 100644 --- a/crates/vm/src/stdlib/ast/string.rs +++ b/crates/vm/src/stdlib/ast/string.rs @@ -1,76 +1,35 @@ use super::constant::{Constant, ConstantLiteral}; use super::*; -fn ruff_fstring_value_into_iter( - mut fstring_value: ruff::FStringValue, -) -> impl Iterator<Item = ruff::FStringPart> + 'static { - let default = ruff::FStringPart::FString(ruff::FString { +fn ruff_fstring_element_into_iter( + mut fstring_element: ast::InterpolatedStringElements, +) -> impl Iterator<Item = ast::InterpolatedStringElement> { + let default = ast::InterpolatedStringElement::Literal(ast::InterpolatedStringLiteralElement { node_index: Default::default(), range: Default::default(), - elements: Default::default(), - flags: ruff::FStringFlags::empty(), + value: Default::default(), }); - (0..fstring_value.as_slice().len()).map(move |i| { - let tmp = fstring_value.iter_mut().nth(i).unwrap(); - core::mem::replace(tmp, default.clone()) - }) -} - -fn ruff_fstring_element_into_iter( - mut fstring_element: ruff::InterpolatedStringElements, -) -> impl Iterator<Item = ruff::InterpolatedStringElement> + 'static { - let default = - ruff::InterpolatedStringElement::Literal(ruff::InterpolatedStringLiteralElement { - node_index: Default::default(), - range: Default::default(), - value: Default::default(), - }); - (0..fstring_element.into_iter().len()).map(move |i| { - let fstring_element = &mut fstring_element; - let tmp = fstring_element.into_iter().nth(i).unwrap(); - core::mem::replace(tmp, default.clone()) - }) -} - -fn fstring_part_to_joined_str_part(fstring_part: ruff::FStringPart) -> Vec<JoinedStrPart> { - match fstring_part { - ruff::FStringPart::Literal(ruff::StringLiteral { - range, - value, - flags, - node_index: _, - }) => { - vec![JoinedStrPart::Constant(Constant::new_str( - value, - flags.prefix(), - range, - ))] - } - ruff::FStringPart::FString(ruff::FString { - range: _, - elements, - flags: _, // TODO - node_index: _, - }) => ruff_fstring_element_into_iter(elements) - .map(ruff_fstring_element_to_joined_str_part) - .collect(), - } + fstring_element + .iter_mut() + .map(move |elem| core::mem::replace(elem, default.clone())) + .collect::<Vec<_>>() + .into_iter() } fn ruff_fstring_element_to_joined_str_part( - element: ruff::InterpolatedStringElement, + element: ast::InterpolatedStringElement, ) -> JoinedStrPart { match element { - ruff::InterpolatedStringElement::Literal(ruff::InterpolatedStringLiteralElement { + ast::InterpolatedStringElement::Literal(ast::InterpolatedStringLiteralElement { range, value, node_index: _, }) => JoinedStrPart::Constant(Constant::new_str( value, - ruff::str_prefix::StringLiteralPrefix::Empty, + ast::str_prefix::StringLiteralPrefix::Empty, range, )), - ruff::InterpolatedStringElement::Interpolation(ruff::InterpolatedElement { + ast::InterpolatedStringElement::Interpolation(ast::InterpolatedElement { range, expression, debug_text: _, // TODO: What is this? @@ -87,12 +46,12 @@ fn ruff_fstring_element_to_joined_str_part( } fn ruff_format_spec_to_joined_str( - format_spec: Option<Box<ruff::InterpolatedStringFormatSpec>>, + format_spec: Option<Box<ast::InterpolatedStringFormatSpec>>, ) -> Option<Box<JoinedStr>> { match format_spec { None => None, Some(format_spec) => { - let ruff::InterpolatedStringFormatSpec { + let ast::InterpolatedStringFormatSpec { range, elements, node_index: _, @@ -107,36 +66,36 @@ fn ruff_format_spec_to_joined_str( } fn ruff_fstring_element_to_ruff_fstring_part( - element: ruff::InterpolatedStringElement, -) -> ruff::FStringPart { + element: ast::InterpolatedStringElement, +) -> ast::FStringPart { match element { - ruff::InterpolatedStringElement::Literal(value) => { - let ruff::InterpolatedStringLiteralElement { + ast::InterpolatedStringElement::Literal(value) => { + let ast::InterpolatedStringLiteralElement { node_index, range, value, } = value; - ruff::FStringPart::Literal(ruff::StringLiteral { + ast::FStringPart::Literal(ast::StringLiteral { node_index, range, value, - flags: ruff::StringLiteralFlags::empty(), + flags: ast::StringLiteralFlags::empty(), }) } - ruff::InterpolatedStringElement::Interpolation(ruff::InterpolatedElement { + ast::InterpolatedStringElement::Interpolation(ast::InterpolatedElement { range, .. - }) => ruff::FStringPart::FString(ruff::FString { + }) => ast::FStringPart::FString(ast::FString { node_index: Default::default(), range, elements: vec![element].into(), - flags: ruff::FStringFlags::empty(), + flags: ast::FStringFlags::empty(), }), } } fn joined_str_to_ruff_format_spec( joined_str: Option<Box<JoinedStr>>, -) -> Option<Box<ruff::InterpolatedStringFormatSpec>> { +) -> Option<Box<ast::InterpolatedStringFormatSpec>> { match joined_str { None => None, Some(joined_str) => { @@ -144,7 +103,7 @@ fn joined_str_to_ruff_format_spec( let elements: Vec<_> = Box::into_iter(values) .map(joined_str_part_to_ruff_fstring_element) .collect(); - let format_spec = ruff::InterpolatedStringFormatSpec { + let format_spec = ast::InterpolatedStringFormatSpec { node_index: Default::default(), range, elements: elements.into(), @@ -161,32 +120,32 @@ pub(super) struct JoinedStr { } impl JoinedStr { - pub(super) fn into_expr(self) -> ruff::Expr { + pub(super) fn into_expr(self) -> ast::Expr { let Self { range, values } = self; - ruff::Expr::FString(ruff::ExprFString { + ast::Expr::FString(ast::ExprFString { node_index: Default::default(), range: Default::default(), value: match values.len() { // ruff represents an empty fstring like this: - 0 => ruff::FStringValue::single(ruff::FString { + 0 => ast::FStringValue::single(ast::FString { node_index: Default::default(), range, elements: vec![].into(), - flags: ruff::FStringFlags::empty(), + flags: ast::FStringFlags::empty(), }), - 1 => ruff::FStringValue::single( + 1 => ast::FStringValue::single( Box::<[_]>::into_iter(values) .map(joined_str_part_to_ruff_fstring_element) - .map(|element| ruff::FString { + .map(|element| ast::FString { node_index: Default::default(), range, elements: vec![element].into(), - flags: ruff::FStringFlags::empty(), + flags: ast::FStringFlags::empty(), }) .next() .expect("FString has exactly one part"), ), - _ => ruff::FStringValue::concatenated( + _ => ast::FStringValue::concatenated( Box::<[_]>::into_iter(values) .map(joined_str_part_to_ruff_fstring_element) .map(ruff_fstring_element_to_ruff_fstring_part) @@ -197,10 +156,10 @@ impl JoinedStr { } } -fn joined_str_part_to_ruff_fstring_element(part: JoinedStrPart) -> ruff::InterpolatedStringElement { +fn joined_str_part_to_ruff_fstring_element(part: JoinedStrPart) -> ast::InterpolatedStringElement { match part { JoinedStrPart::FormattedValue(value) => { - ruff::InterpolatedStringElement::Interpolation(ruff::InterpolatedElement { + ast::InterpolatedStringElement::Interpolation(ast::InterpolatedElement { node_index: Default::default(), range: value.range, expression: value.value.clone(), @@ -210,7 +169,7 @@ fn joined_str_part_to_ruff_fstring_element(part: JoinedStrPart) -> ruff::Interpo }) } JoinedStrPart::Constant(value) => { - ruff::InterpolatedStringElement::Literal(ruff::InterpolatedStringLiteralElement { + ast::InterpolatedStringElement::Literal(ast::InterpolatedStringLiteralElement { node_index: Default::default(), range: value.range, value: match value.value { @@ -294,8 +253,8 @@ impl Node for JoinedStrPart { #[derive(Debug)] pub(super) struct FormattedValue { - value: Box<ruff::Expr>, - conversion: ruff::ConversionFlag, + value: Box<ast::Expr>, + conversion: ast::ConversionFlag, format_spec: Option<Box<JoinedStr>>, range: TextRange, } @@ -353,17 +312,284 @@ impl Node for FormattedValue { pub(super) fn fstring_to_object( vm: &VirtualMachine, source_file: &SourceFile, - expression: ruff::ExprFString, + expression: ast::ExprFString, ) -> PyObjectRef { - let ruff::ExprFString { + let ast::ExprFString { range, - value, + mut value, node_index: _, } = expression; - let values: Vec<_> = ruff_fstring_value_into_iter(value) - .flat_map(fstring_part_to_joined_str_part) - .collect(); - let values = values.into_boxed_slice(); - let c = JoinedStr { range, values }; + let default_part = ast::FStringPart::FString(ast::FString { + node_index: Default::default(), + range: Default::default(), + elements: Default::default(), + flags: ast::FStringFlags::empty(), + }); + let mut values = Vec::new(); + for i in 0..value.as_slice().len() { + let part = core::mem::replace(value.iter_mut().nth(i).unwrap(), default_part.clone()); + match part { + ast::FStringPart::Literal(ast::StringLiteral { + range, + value, + flags, + node_index: _, + }) => { + values.push(JoinedStrPart::Constant(Constant::new_str( + value, + flags.prefix(), + range, + ))); + } + ast::FStringPart::FString(ast::FString { + range: _, + elements, + flags: _, + node_index: _, + }) => { + for element in ruff_fstring_element_into_iter(elements) { + values.push(ruff_fstring_element_to_joined_str_part(element)); + } + } + } + } + let c = JoinedStr { + range, + values: values.into_boxed_slice(), + }; + c.ast_to_object(vm, source_file) +} + +// ===== TString (Template String) Support ===== + +fn ruff_tstring_element_to_template_str_part( + element: ast::InterpolatedStringElement, + source_file: &SourceFile, +) -> TemplateStrPart { + match element { + ast::InterpolatedStringElement::Literal(ast::InterpolatedStringLiteralElement { + range, + value, + node_index: _, + }) => TemplateStrPart::Constant(Constant::new_str( + value, + ast::str_prefix::StringLiteralPrefix::Empty, + range, + )), + ast::InterpolatedStringElement::Interpolation(ast::InterpolatedElement { + range, + expression, + debug_text, + conversion, + format_spec, + node_index: _, + }) => { + // Get the expression source text for the "str" field + let expr_str = debug_text + .map(|dt| dt.leading.to_string() + &dt.trailing) + .unwrap_or_else(|| source_file.slice(expression.range()).to_string()); + TemplateStrPart::Interpolation(TStringInterpolation { + value: expression, + str: expr_str, + conversion, + format_spec: ruff_format_spec_to_template_str(format_spec, source_file), + range, + }) + } + } +} + +fn ruff_format_spec_to_template_str( + format_spec: Option<Box<ast::InterpolatedStringFormatSpec>>, + source_file: &SourceFile, +) -> Option<Box<TemplateStr>> { + match format_spec { + None => None, + Some(format_spec) => { + let ast::InterpolatedStringFormatSpec { + range, + elements, + node_index: _, + } = *format_spec; + let values: Vec<_> = ruff_fstring_element_into_iter(elements) + .map(|e| ruff_tstring_element_to_template_str_part(e, source_file)) + .collect(); + let values = values.into_boxed_slice(); + Some(Box::new(TemplateStr { range, values })) + } + } +} + +#[derive(Debug)] +pub(super) struct TemplateStr { + pub(super) range: TextRange, + pub(super) values: Box<[TemplateStrPart]>, +} + +// constructor +impl Node for TemplateStr { + fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { + let Self { values, range } = self; + let node = NodeAst + .into_ref_with_type(vm, pyast::NodeExprTemplateStr::static_type().to_owned()) + .unwrap(); + let dict = node.as_object().dict().unwrap(); + dict.set_item( + "values", + BoxedSlice(values).ast_to_object(vm, source_file), + vm, + ) + .unwrap(); + node_add_location(&dict, range, vm, source_file); + node.into() + } + fn ast_from_object( + vm: &VirtualMachine, + source_file: &SourceFile, + object: PyObjectRef, + ) -> PyResult<Self> { + let values: BoxedSlice<_> = Node::ast_from_object( + vm, + source_file, + get_node_field(vm, &object, "values", "TemplateStr")?, + )?; + Ok(Self { + values: values.0, + range: range_from_object(vm, source_file, object, "TemplateStr")?, + }) + } +} + +#[derive(Debug)] +pub(super) enum TemplateStrPart { + Interpolation(TStringInterpolation), + Constant(Constant), +} + +// constructor +impl Node for TemplateStrPart { + fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { + match self { + Self::Interpolation(value) => value.ast_to_object(vm, source_file), + Self::Constant(value) => value.ast_to_object(vm, source_file), + } + } + fn ast_from_object( + vm: &VirtualMachine, + source_file: &SourceFile, + object: PyObjectRef, + ) -> PyResult<Self> { + let cls = object.class(); + if cls.is(pyast::NodeExprInterpolation::static_type()) { + Ok(Self::Interpolation(Node::ast_from_object( + vm, + source_file, + object, + )?)) + } else { + Ok(Self::Constant(Node::ast_from_object( + vm, + source_file, + object, + )?)) + } + } +} + +#[derive(Debug)] +pub(super) struct TStringInterpolation { + value: Box<ast::Expr>, + str: String, + conversion: ast::ConversionFlag, + format_spec: Option<Box<TemplateStr>>, + range: TextRange, +} + +// constructor +impl Node for TStringInterpolation { + fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { + let Self { + value, + str, + conversion, + format_spec, + range, + } = self; + let node = NodeAst + .into_ref_with_type(vm, pyast::NodeExprInterpolation::static_type().to_owned()) + .unwrap(); + let dict = node.as_object().dict().unwrap(); + dict.set_item("value", value.ast_to_object(vm, source_file), vm) + .unwrap(); + dict.set_item("str", vm.ctx.new_str(str).into(), vm) + .unwrap(); + dict.set_item("conversion", conversion.ast_to_object(vm, source_file), vm) + .unwrap(); + dict.set_item( + "format_spec", + format_spec.ast_to_object(vm, source_file), + vm, + ) + .unwrap(); + node_add_location(&dict, range, vm, source_file); + node.into() + } + fn ast_from_object( + vm: &VirtualMachine, + source_file: &SourceFile, + object: PyObjectRef, + ) -> PyResult<Self> { + let str_obj = get_node_field(vm, &object, "str", "Interpolation")?; + let str_val: String = str_obj.try_into_value(vm)?; + Ok(Self { + value: Node::ast_from_object( + vm, + source_file, + get_node_field(vm, &object, "value", "Interpolation")?, + )?, + str: str_val, + conversion: Node::ast_from_object( + vm, + source_file, + get_node_field(vm, &object, "conversion", "Interpolation")?, + )?, + format_spec: get_node_field_opt(vm, &object, "format_spec")? + .map(|obj| Node::ast_from_object(vm, source_file, obj)) + .transpose()?, + range: range_from_object(vm, source_file, object, "Interpolation")?, + }) + } +} + +pub(super) fn tstring_to_object( + vm: &VirtualMachine, + source_file: &SourceFile, + expression: ast::ExprTString, +) -> PyObjectRef { + let ast::ExprTString { + range, + mut value, + node_index: _, + } = expression; + let default_tstring = ast::TString { + node_index: Default::default(), + range: Default::default(), + elements: Default::default(), + flags: ast::TStringFlags::empty(), + }; + let mut values = Vec::new(); + for i in 0..value.as_slice().len() { + let tstring = core::mem::replace(value.iter_mut().nth(i).unwrap(), default_tstring.clone()); + for element in ruff_fstring_element_into_iter(tstring.elements) { + values.push(ruff_tstring_element_to_template_str_part( + element, + source_file, + )); + } + } + let c = TemplateStr { + range, + values: values.into_boxed_slice(), + }; c.ast_to_object(vm, source_file) } diff --git a/crates/vm/src/stdlib/ast/type_parameters.rs b/crates/vm/src/stdlib/ast/type_parameters.rs index 017470f7e64..4801a9a4b28 100644 --- a/crates/vm/src/stdlib/ast/type_parameters.rs +++ b/crates/vm/src/stdlib/ast/type_parameters.rs @@ -1,7 +1,7 @@ use super::*; use rustpython_compiler_core::SourceFile; -impl Node for ruff::TypeParams { +impl Node for ast::TypeParams { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { self.type_params.ast_to_object(vm, source_file) } @@ -11,7 +11,7 @@ impl Node for ruff::TypeParams { _source_file: &SourceFile, _object: PyObjectRef, ) -> PyResult<Self> { - let type_params: Vec<ruff::TypeParam> = Node::ast_from_object(_vm, _source_file, _object)?; + let type_params: Vec<ast::TypeParam> = Node::ast_from_object(_vm, _source_file, _object)?; let range = Option::zip(type_params.first(), type_params.last()) .map(|(first, last)| first.range().cover(last.range())) .unwrap_or_default(); @@ -28,7 +28,7 @@ impl Node for ruff::TypeParams { } // sum -impl Node for ruff::TypeParam { +impl Node for ast::TypeParam { fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { match self { Self::TypeVar(cons) => cons.ast_to_object(vm, source_file), @@ -44,19 +44,19 @@ impl Node for ruff::TypeParam { ) -> PyResult<Self> { let _cls = _object.class(); Ok(if _cls.is(pyast::NodeTypeParamTypeVar::static_type()) { - Self::TypeVar(ruff::TypeParamTypeVar::ast_from_object( + Self::TypeVar(ast::TypeParamTypeVar::ast_from_object( _vm, source_file, _object, )?) } else if _cls.is(pyast::NodeTypeParamParamSpec::static_type()) { - Self::ParamSpec(ruff::TypeParamParamSpec::ast_from_object( + Self::ParamSpec(ast::TypeParamParamSpec::ast_from_object( _vm, source_file, _object, )?) } else if _cls.is(pyast::NodeTypeParamTypeVarTuple::static_type()) { - Self::TypeVarTuple(ruff::TypeParamTypeVarTuple::ast_from_object( + Self::TypeVarTuple(ast::TypeParamTypeVarTuple::ast_from_object( _vm, source_file, _object, @@ -71,7 +71,7 @@ impl Node for ruff::TypeParam { } // constructor -impl Node for ruff::TypeParamTypeVar { +impl Node for ast::TypeParamTypeVar { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -118,7 +118,7 @@ impl Node for ruff::TypeParamTypeVar { } // constructor -impl Node for ruff::TypeParamParamSpec { +impl Node for ast::TypeParamParamSpec { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, @@ -165,7 +165,7 @@ impl Node for ruff::TypeParamParamSpec { } // constructor -impl Node for ruff::TypeParamTypeVarTuple { +impl Node for ast::TypeParamTypeVarTuple { fn ast_to_object(self, _vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef { let Self { node_index: _, diff --git a/crates/vm/src/stdlib/atexit.rs b/crates/vm/src/stdlib/atexit.rs index 2286c36f1db..338fae3b2b7 100644 --- a/crates/vm/src/stdlib/atexit.rs +++ b/crates/vm/src/stdlib/atexit.rs @@ -1,5 +1,5 @@ pub use atexit::_run_exitfuncs; -pub(crate) use atexit::make_module; +pub(crate) use atexit::module_def; #[pymodule] mod atexit { diff --git a/crates/vm/src/stdlib/builtins.rs b/crates/vm/src/stdlib/builtins.rs index b4554bc30aa..1f14f6f5b04 100644 --- a/crates/vm/src/stdlib/builtins.rs +++ b/crates/vm/src/stdlib/builtins.rs @@ -2,7 +2,7 @@ //! //! Implements the list of [builtin Python functions](https://docs.python.org/3/library/builtins.html). use crate::{Py, VirtualMachine, builtins::PyModule, class::PyClassImpl}; -pub(crate) use builtins::{__module_def, DOC}; +pub(crate) use builtins::{DOC, module_def}; pub use builtins::{ascii, print, reversed}; #[pymodule] @@ -1100,7 +1100,8 @@ pub fn init_module(vm: &VirtualMachine, module: &Py<PyModule>) { crate::protocol::VecBuffer::make_class(&vm.ctx); - builtins::extend_module(vm, module).unwrap(); + module.__init_methods(vm).unwrap(); + builtins::module_exec(vm, module).unwrap(); let debug_mode: bool = vm.state.config.settings.optimize == 0; // Create dynamic ExceptionGroup with multiple inheritance (BaseExceptionGroup + Exception) diff --git a/crates/vm/src/stdlib/codecs.rs b/crates/vm/src/stdlib/codecs.rs index 1661eef1750..bc9029cb71a 100644 --- a/crates/vm/src/stdlib/codecs.rs +++ b/crates/vm/src/stdlib/codecs.rs @@ -1,4 +1,4 @@ -pub(crate) use _codecs::make_module; +pub(crate) use _codecs::module_def; #[pymodule] mod _codecs { @@ -705,5 +705,28 @@ mod _codecs { fn utf_16_ex_decode(args: FuncArgs, vm: &VirtualMachine) -> PyResult { delegate_pycodecs!(utf_16_ex_decode, args, vm) } - // TODO: utf-32 functions + #[pyfunction] + fn utf_32_encode(args: FuncArgs, vm: &VirtualMachine) -> PyResult { + delegate_pycodecs!(utf_32_encode, args, vm) + } + #[pyfunction] + fn utf_32_decode(args: FuncArgs, vm: &VirtualMachine) -> PyResult { + delegate_pycodecs!(utf_32_decode, args, vm) + } + #[pyfunction] + fn utf_32_le_encode(args: FuncArgs, vm: &VirtualMachine) -> PyResult { + delegate_pycodecs!(utf_32_le_encode, args, vm) + } + #[pyfunction] + fn utf_32_le_decode(args: FuncArgs, vm: &VirtualMachine) -> PyResult { + delegate_pycodecs!(utf_32_le_decode, args, vm) + } + #[pyfunction] + fn utf_32_be_encode(args: FuncArgs, vm: &VirtualMachine) -> PyResult { + delegate_pycodecs!(utf_32_be_encode, args, vm) + } + #[pyfunction] + fn utf_32_be_decode(args: FuncArgs, vm: &VirtualMachine) -> PyResult { + delegate_pycodecs!(utf_32_be_decode, args, vm) + } } diff --git a/crates/vm/src/stdlib/collections.rs b/crates/vm/src/stdlib/collections.rs index fd48cea0598..0f84db80e74 100644 --- a/crates/vm/src/stdlib/collections.rs +++ b/crates/vm/src/stdlib/collections.rs @@ -1,4 +1,4 @@ -pub(crate) use _collections::make_module; +pub(crate) use _collections::module_def; #[pymodule] mod _collections { diff --git a/crates/vm/src/stdlib/ctypes.rs b/crates/vm/src/stdlib/ctypes.rs index cbe3f5405b6..97255f4384f 100644 --- a/crates/vm/src/stdlib/ctypes.rs +++ b/crates/vm/src/stdlib/ctypes.rs @@ -10,8 +10,8 @@ mod structure; mod union; use crate::{ - AsObject, Py, PyObjectRef, PyRef, PyResult, VirtualMachine, - builtins::{PyModule, PyStr, PyType}, + AsObject, Py, PyObjectRef, PyResult, VirtualMachine, + builtins::{PyStr, PyType}, class::PyClassImpl, types::TypeDataRef, }; @@ -85,32 +85,9 @@ impl PyType { } // Dynamic type check helpers for PyCData -// These check if an object's type's metaclass is a subclass of a specific metaclass +pub(crate) use _ctypes::module_def; -pub(crate) fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - let module = _ctypes::make_module(vm); - let ctx = &vm.ctx; - PyCSimpleType::make_class(ctx); - array::PyCArrayType::make_class(ctx); - pointer::PyCPointerType::make_class(ctx); - structure::PyCStructType::make_class(ctx); - union::PyCUnionType::make_class(ctx); - function::PyCFuncPtrType::make_class(ctx); - extend_module!(vm, &module, { - "_CData" => PyCData::make_class(ctx), - "_SimpleCData" => PyCSimple::make_class(ctx), - "Array" => PyCArray::make_class(ctx), - "CField" => PyCField::make_class(ctx), - "CFuncPtr" => function::PyCFuncPtr::make_class(ctx), - "_Pointer" => PyCPointer::make_class(ctx), - "_pointer_type_cache" => ctx.new_dict(), - "_array_type_cache" => ctx.new_dict(), - "Structure" => PyCStructure::make_class(ctx), - "CThunkObject" => function::PyCThunk::make_class(ctx), - "Union" => PyCUnion::make_class(ctx), - }); - module -} +// These check if an object's type's metaclass is a subclass of a specific metaclass /// Size of long double - platform dependent /// x86_64 macOS/Linux: 16 bytes (80-bit extended + padding) @@ -987,13 +964,13 @@ pub(crate) mod _ctypes { #[pyattr] fn _memmove_addr(_vm: &VirtualMachine) -> usize { let f = libc::memmove; - f as usize + f as *const () as usize } #[pyattr] fn _memset_addr(_vm: &VirtualMachine) -> usize { let f = libc::memset; - f as usize + f as *const () as usize } #[pyattr] @@ -1300,4 +1277,37 @@ pub(crate) mod _ctypes { Ok(S_OK) } + + pub(crate) fn module_exec( + vm: &VirtualMachine, + module: &Py<crate::builtins::PyModule>, + ) -> PyResult<()> { + use super::*; + + __module_exec(vm, module); + + let ctx = &vm.ctx; + PyCSimpleType::make_class(ctx); + array::PyCArrayType::make_class(ctx); + pointer::PyCPointerType::make_class(ctx); + structure::PyCStructType::make_class(ctx); + union::PyCUnionType::make_class(ctx); + function::PyCFuncPtrType::make_class(ctx); + + extend_module!(vm, module, { + "_CData" => PyCData::make_class(ctx), + "_SimpleCData" => PyCSimple::make_class(ctx), + "Array" => PyCArray::make_class(ctx), + "CField" => PyCField::make_class(ctx), + "CFuncPtr" => function::PyCFuncPtr::make_class(ctx), + "_Pointer" => PyCPointer::make_class(ctx), + "_pointer_type_cache" => ctx.new_dict(), + "_array_type_cache" => ctx.new_dict(), + "Structure" => PyCStructure::make_class(ctx), + "CThunkObject" => function::PyCThunk::make_class(ctx), + "Union" => PyCUnion::make_class(ctx), + }); + + Ok(()) + } } diff --git a/crates/vm/src/stdlib/ctypes/simple.rs b/crates/vm/src/stdlib/ctypes/simple.rs index de13dab2202..b2ae0f7cc5b 100644 --- a/crates/vm/src/stdlib/ctypes/simple.rs +++ b/crates/vm/src/stdlib/ctypes/simple.rs @@ -141,11 +141,12 @@ fn set_primitive(_type_: &str, value: &PyObject, vm: &VirtualMachine) -> PyResul // Handle int specially to check overflow if let Some(int_obj) = value.downcast_ref_if_exact::<PyInt>(vm) { // Check if int can fit in f64 - if int_obj.as_bigint().to_f64().is_some() { + if let Some(f) = int_obj.as_bigint().to_f64() + && f.is_finite() + { return Ok(value.to_owned()); - } else { - return Err(vm.new_overflow_error("int too large to convert to float")); } + return Err(vm.new_overflow_error("int too large to convert to float")); } // __float__ protocol if value.try_float(vm).is_ok() { diff --git a/crates/vm/src/stdlib/errno.rs b/crates/vm/src/stdlib/errno.rs index 7a78ceaea83..8e4efbaafe9 100644 --- a/crates/vm/src/stdlib/errno.rs +++ b/crates/vm/src/stdlib/errno.rs @@ -1,25 +1,26 @@ // spell-checker:disable -use crate::{PyRef, VirtualMachine, builtins::PyModule}; +pub(crate) use errno_mod::module_def; -#[pymodule] -mod errno {} +#[pymodule(name = "errno")] +mod errno_mod { + use crate::{Py, PyResult, VirtualMachine, builtins::PyModule}; -pub fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - let module = errno::make_module(vm); - let errorcode = vm.ctx.new_dict(); - extend_module!(vm, &module, { - "errorcode" => errorcode.clone(), - }); - for (name, code) in ERROR_CODES { - let name = vm.ctx.intern_str(*name); - let code = vm.new_pyobj(*code); - errorcode - .set_item(&*code, name.to_owned().into(), vm) - .unwrap(); - module.set_attr(name, code, vm).unwrap(); + pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { + __module_exec(vm, module); + + let errorcode = vm.ctx.new_dict(); + extend_module!(vm, module, { + "errorcode" => errorcode.clone(), + }); + for (name, code) in super::ERROR_CODES { + let name = vm.ctx.intern_str(*name); + let code = vm.new_pyobj(*code); + errorcode.set_item(&*code, name.to_owned().into(), vm)?; + module.set_attr(name, code, vm)?; + } + Ok(()) } - module } #[cfg(any(unix, windows, target_os = "wasi"))] diff --git a/crates/vm/src/stdlib/functools.rs b/crates/vm/src/stdlib/functools.rs index a59fd48f6b2..6c5c8f2e4c5 100644 --- a/crates/vm/src/stdlib/functools.rs +++ b/crates/vm/src/stdlib/functools.rs @@ -1,31 +1,45 @@ -pub(crate) use _functools::make_module; +pub(crate) use _functools::module_def; #[pymodule] mod _functools { use crate::{ Py, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, - builtins::{PyDict, PyGenericAlias, PyTuple, PyTypeRef}, + builtins::{PyBoundMethod, PyDict, PyGenericAlias, PyTuple, PyType, PyTypeRef}, common::lock::PyRwLock, - function::{FuncArgs, KwArgs, OptionalArg}, + function::{FuncArgs, KwArgs, OptionalOption}, object::AsObject, protocol::PyIter, pyclass, recursion::ReprGuard, - types::{Callable, Constructor, Representable}, + types::{Callable, Constructor, GetDescriptor, Representable}, }; use indexmap::IndexMap; - #[pyfunction] - fn reduce( + #[derive(FromArgs)] + struct ReduceArgs { function: PyObjectRef, iterator: PyIter, - start_value: OptionalArg<PyObjectRef>, - vm: &VirtualMachine, - ) -> PyResult { + #[pyarg(any, optional, name = "initial")] + initial: OptionalOption<PyObjectRef>, + } + + #[pyfunction] + fn reduce(args: ReduceArgs, vm: &VirtualMachine) -> PyResult { + let ReduceArgs { + function, + iterator, + initial, + } = args; let mut iter = iterator.iter_without_hint(vm)?; - let start_value = if let OptionalArg::Present(val) = start_value { - val + // OptionalOption distinguishes between: + // - Missing: no argument provided → use first element from iterator + // - Present(None): explicitly passed None → use None as initial value + // - Present(Some(v)): passed a value → use that value + let start_value = if let Some(val) = initial.into_option() { + // initial was provided (could be None or Some value) + val.unwrap_or_else(|| vm.ctx.none()) } else { + // initial was not provided at all iter.next().transpose()?.ok_or_else(|| { let exc_type = vm.ctx.exceptions.type_error.to_owned(); vm.new_exception_msg( @@ -42,6 +56,72 @@ mod _functools { Ok(accumulator) } + // Placeholder singleton for partial arguments + // The singleton is stored as _instance on the type class + #[pyattr] + #[allow(non_snake_case)] + fn Placeholder(vm: &VirtualMachine) -> PyObjectRef { + let placeholder = PyPlaceholderType.into_pyobject(vm); + // Store the singleton on the type class for slot_new to find + let typ = placeholder.class(); + typ.set_attr(vm.ctx.intern_str("_instance"), placeholder.clone()); + placeholder + } + + #[pyattr] + #[pyclass(name = "_PlaceholderType", module = "functools")] + #[derive(Debug, PyPayload)] + pub struct PyPlaceholderType; + + impl Constructor for PyPlaceholderType { + type Args = FuncArgs; + + fn slot_new(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult { + if !args.args.is_empty() || !args.kwargs.is_empty() { + return Err(vm.new_type_error("_PlaceholderType takes no arguments".to_owned())); + } + // Return the singleton stored on the type class + if let Some(instance) = cls.get_attr(vm.ctx.intern_str("_instance")) { + return Ok(instance); + } + // Fallback: create a new instance (shouldn't happen for base type after module init) + Ok(PyPlaceholderType.into_pyobject(vm)) + } + + fn py_new(_cls: &Py<PyType>, _args: Self::Args, _vm: &VirtualMachine) -> PyResult<Self> { + // This is never called because we override slot_new + Ok(PyPlaceholderType) + } + } + + #[pyclass(with(Constructor, Representable))] + impl PyPlaceholderType { + #[pymethod] + fn __reduce__(&self) -> &'static str { + "Placeholder" + } + + #[pymethod] + fn __init_subclass__(_cls: PyTypeRef, vm: &VirtualMachine) -> PyResult<()> { + Err(vm.new_type_error("cannot subclass '_PlaceholderType'".to_owned())) + } + } + + impl Representable for PyPlaceholderType { + #[inline] + fn repr_str(_zelf: &Py<Self>, _vm: &VirtualMachine) -> PyResult<String> { + Ok("Placeholder".to_owned()) + } + } + + fn is_placeholder(obj: &PyObjectRef) -> bool { + &*obj.class().name() == "_PlaceholderType" + } + + fn count_placeholders(args: &[PyObjectRef]) -> usize { + args.iter().filter(|a| is_placeholder(a)).count() + } + #[pyattr] #[pyclass(name = "partial", module = "functools")] #[derive(Debug, PyPayload)] @@ -54,9 +134,13 @@ mod _functools { func: PyObjectRef, args: PyRef<PyTuple>, keywords: PyRef<PyDict>, + phcount: usize, } - #[pyclass(with(Constructor, Callable, Representable), flags(BASETYPE, HAS_DICT))] + #[pyclass( + with(Constructor, Callable, GetDescriptor, Representable), + flags(BASETYPE, HAS_DICT) + )] impl PyPartial { #[pygetset] fn func(&self) -> PyObjectRef { @@ -157,6 +241,13 @@ mod _functools { } }; + // Validate no trailing placeholders + let args_slice = args_tuple.as_slice(); + if !args_slice.is_empty() && is_placeholder(args_slice.last().unwrap()) { + return Err(vm.new_type_error("trailing Placeholders are not allowed".to_owned())); + } + let phcount = count_placeholders(args_slice); + // Actually update the state let mut inner = zelf.inner.write(); inner.func = func.clone(); @@ -165,6 +256,7 @@ mod _functools { // Handle keywords - keep the original type inner.keywords = keywords_dict; + inner.phcount = phcount; // Update __dict__ if provided let Some(instance_dict) = zelf.as_object().dict() else { @@ -218,17 +310,54 @@ mod _functools { return Err(vm.new_type_error("the first argument must be callable")); } + // Check for placeholders in kwargs + for (key, value) in &args.kwargs { + if is_placeholder(value) { + return Err(vm.new_type_error(format!( + "Placeholder cannot be passed as a keyword argument to partial(). \ + Did you mean partial(..., {}=Placeholder, ...)(value)?", + key + ))); + } + } + // Handle nested partial objects let (final_func, final_args, final_keywords) = if let Some(partial) = func.downcast_ref::<Self>() { let inner = partial.inner.read(); - let mut combined_args = inner.args.as_slice().to_vec(); - combined_args.extend_from_slice(args_slice); - (inner.func.clone(), combined_args, inner.keywords.clone()) + let stored_args = inner.args.as_slice(); + + // Merge placeholders: replace placeholders in stored_args with new args + let mut merged_args = Vec::with_capacity(stored_args.len() + args_slice.len()); + let mut new_args_iter = args_slice.iter(); + + for stored_arg in stored_args { + if is_placeholder(stored_arg) { + // Replace placeholder with next new arg, or keep placeholder + if let Some(new_arg) = new_args_iter.next() { + merged_args.push(new_arg.clone()); + } else { + merged_args.push(stored_arg.clone()); + } + } else { + merged_args.push(stored_arg.clone()); + } + } + // Append remaining new args + merged_args.extend(new_args_iter.cloned()); + + (inner.func.clone(), merged_args, inner.keywords.clone()) } else { (func.clone(), args_slice.to_vec(), vm.ctx.new_dict()) }; + // Trailing placeholders are not allowed + if !final_args.is_empty() && is_placeholder(final_args.last().unwrap()) { + return Err(vm.new_type_error("trailing Placeholders are not allowed".to_owned())); + } + + let phcount = count_placeholders(&final_args); + // Add new keywords for (key, value) in args.kwargs { final_keywords.set_item(vm.ctx.intern_str(key.as_str()), value, vm)?; @@ -239,6 +368,7 @@ mod _functools { func: final_func, args: vm.ctx.new_tuple(final_args), keywords: final_keywords, + phcount, }), }) } @@ -249,17 +379,44 @@ mod _functools { fn call(zelf: &Py<Self>, args: FuncArgs, vm: &VirtualMachine) -> PyResult { // Clone and release lock before calling Python code to prevent deadlock - let (func, stored_args, keywords) = { + let (func, stored_args, keywords, phcount) = { let inner = zelf.inner.read(); ( inner.func.clone(), inner.args.clone(), inner.keywords.clone(), + inner.phcount, ) }; - let mut combined_args = stored_args.as_slice().to_vec(); - combined_args.extend_from_slice(&args.args); + // Check if we have enough args to fill placeholders + if phcount > 0 && args.args.len() < phcount { + return Err(vm.new_type_error(format!( + "missing positional arguments in 'partial' call; expected at least {}, got {}", + phcount, + args.args.len() + ))); + } + + // Build combined args, replacing placeholders + let mut combined_args = Vec::with_capacity(stored_args.len() + args.args.len()); + let mut new_args_iter = args.args.iter(); + + for stored_arg in stored_args.as_slice() { + if is_placeholder(stored_arg) { + // Replace placeholder with next new arg + if let Some(new_arg) = new_args_iter.next() { + combined_args.push(new_arg.clone()); + } else { + // This shouldn't happen if phcount check passed + combined_args.push(stored_arg.clone()); + } + } else { + combined_args.push(stored_arg.clone()); + } + } + // Append remaining new args + combined_args.extend(new_args_iter.cloned()); // Merge keywords from self.keywords and args.kwargs let mut final_kwargs = IndexMap::new(); @@ -281,6 +438,21 @@ mod _functools { } } + impl GetDescriptor for PyPartial { + fn descr_get( + zelf: PyObjectRef, + obj: Option<PyObjectRef>, + _cls: Option<PyObjectRef>, + vm: &VirtualMachine, + ) -> PyResult { + let obj = match obj { + Some(obj) if !vm.is_none(&obj) => obj, + _ => return Ok(zelf), + }; + Ok(PyBoundMethod::new(obj, zelf).into_ref(&vm.ctx).into()) + } + } + impl Representable for PyPartial { #[inline] fn repr_str(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<String> { diff --git a/crates/vm/src/stdlib/gc.rs b/crates/vm/src/stdlib/gc.rs new file mode 100644 index 00000000000..82b0c68bd9e --- /dev/null +++ b/crates/vm/src/stdlib/gc.rs @@ -0,0 +1,265 @@ +pub(crate) use gc::module_def; + +#[pymodule] +mod gc { + use crate::{ + PyObjectRef, PyResult, VirtualMachine, + builtins::PyListRef, + function::{FuncArgs, OptionalArg}, + gc_state, + }; + + // Debug flag constants + #[pyattr] + const DEBUG_STATS: u32 = gc_state::GcDebugFlags::STATS.bits(); + #[pyattr] + const DEBUG_COLLECTABLE: u32 = gc_state::GcDebugFlags::COLLECTABLE.bits(); + #[pyattr] + const DEBUG_UNCOLLECTABLE: u32 = gc_state::GcDebugFlags::UNCOLLECTABLE.bits(); + #[pyattr] + const DEBUG_SAVEALL: u32 = gc_state::GcDebugFlags::SAVEALL.bits(); + #[pyattr] + const DEBUG_LEAK: u32 = gc_state::GcDebugFlags::LEAK.bits(); + + /// Enable automatic garbage collection. + #[pyfunction] + fn enable() { + gc_state::gc_state().enable(); + } + + /// Disable automatic garbage collection. + #[pyfunction] + fn disable() { + gc_state::gc_state().disable(); + } + + /// Return true if automatic gc is enabled. + #[pyfunction] + fn isenabled() -> bool { + gc_state::gc_state().is_enabled() + } + + /// Run a garbage collection. Returns the number of unreachable objects found. + #[derive(FromArgs)] + struct CollectArgs { + #[pyarg(any, optional)] + generation: OptionalArg<i32>, + } + + #[pyfunction] + fn collect(args: CollectArgs, vm: &VirtualMachine) -> PyResult<i32> { + let generation = args.generation; + let generation_num = generation.unwrap_or(2); + if !(0..=2).contains(&generation_num) { + return Err(vm.new_value_error("invalid generation".to_owned())); + } + + // Invoke callbacks with "start" phase + invoke_callbacks(vm, "start", generation_num as usize, 0, 0); + + // Manual gc.collect() should run even if GC is disabled + let gc = gc_state::gc_state(); + let (collected, uncollectable) = gc.collect_force(generation_num as usize); + + // Move objects from gc_state.garbage to vm.ctx.gc_garbage (for DEBUG_SAVEALL) + { + let mut state_garbage = gc.garbage.lock(); + if !state_garbage.is_empty() { + let py_garbage = &vm.ctx.gc_garbage; + let mut garbage_vec = py_garbage.borrow_vec_mut(); + for obj in state_garbage.drain(..) { + garbage_vec.push(obj); + } + } + } + + // Invoke callbacks with "stop" phase + invoke_callbacks( + vm, + "stop", + generation_num as usize, + collected, + uncollectable, + ); + + Ok(collected as i32) + } + + /// Return the current collection thresholds as a tuple. + #[pyfunction] + fn get_threshold(vm: &VirtualMachine) -> PyObjectRef { + let (t0, t1, t2) = gc_state::gc_state().get_threshold(); + vm.ctx + .new_tuple(vec![ + vm.ctx.new_int(t0).into(), + vm.ctx.new_int(t1).into(), + vm.ctx.new_int(t2).into(), + ]) + .into() + } + + /// Set the collection thresholds. + #[pyfunction] + fn set_threshold(threshold0: u32, threshold1: OptionalArg<u32>, threshold2: OptionalArg<u32>) { + gc_state::gc_state().set_threshold( + threshold0, + threshold1.into_option(), + threshold2.into_option(), + ); + } + + /// Return the current collection counts as a tuple. + #[pyfunction] + fn get_count(vm: &VirtualMachine) -> PyObjectRef { + let (c0, c1, c2) = gc_state::gc_state().get_count(); + vm.ctx + .new_tuple(vec![ + vm.ctx.new_int(c0).into(), + vm.ctx.new_int(c1).into(), + vm.ctx.new_int(c2).into(), + ]) + .into() + } + + /// Return the current debugging flags. + #[pyfunction] + fn get_debug() -> u32 { + gc_state::gc_state().get_debug().bits() + } + + /// Set the debugging flags. + #[pyfunction] + fn set_debug(flags: u32) { + gc_state::gc_state().set_debug(gc_state::GcDebugFlags::from_bits_truncate(flags)); + } + + /// Return a list of per-generation gc stats. + #[pyfunction] + fn get_stats(vm: &VirtualMachine) -> PyResult<PyListRef> { + let stats = gc_state::gc_state().get_stats(); + let mut result = Vec::with_capacity(3); + + for stat in stats.iter() { + let dict = vm.ctx.new_dict(); + dict.set_item("collections", vm.ctx.new_int(stat.collections).into(), vm)?; + dict.set_item("collected", vm.ctx.new_int(stat.collected).into(), vm)?; + dict.set_item( + "uncollectable", + vm.ctx.new_int(stat.uncollectable).into(), + vm, + )?; + result.push(dict.into()); + } + + Ok(vm.ctx.new_list(result)) + } + + /// Return the list of objects tracked by the collector. + #[derive(FromArgs)] + struct GetObjectsArgs { + #[pyarg(any, optional)] + generation: OptionalArg<Option<i32>>, + } + + #[pyfunction] + fn get_objects(args: GetObjectsArgs, vm: &VirtualMachine) -> PyResult<PyListRef> { + let generation_opt = args.generation.flatten(); + if let Some(g) = generation_opt + && !(0..=2).contains(&g) + { + return Err(vm.new_value_error(format!("generation must be in range(0, 3), not {}", g))); + } + let objects = gc_state::gc_state().get_objects(generation_opt); + Ok(vm.ctx.new_list(objects)) + } + + /// Return the list of objects directly referred to by any of the arguments. + #[pyfunction] + fn get_referents(args: FuncArgs, vm: &VirtualMachine) -> PyListRef { + let mut result = Vec::new(); + + for obj in args.args { + // Use the gc_get_referents method to get references + result.extend(obj.gc_get_referents()); + } + + vm.ctx.new_list(result) + } + + /// Return the list of objects that directly refer to any of the arguments. + #[pyfunction] + fn get_referrers(args: FuncArgs, vm: &VirtualMachine) -> PyListRef { + // This is expensive: we need to scan all tracked objects + // For now, return an empty list (would need full object tracking to implement) + let _ = args; + vm.ctx.new_list(vec![]) + } + + /// Return True if the object is tracked by the garbage collector. + #[pyfunction] + fn is_tracked(obj: PyObjectRef) -> bool { + // An object is tracked if it has IS_TRACE = true (has a trace function) + obj.is_gc_tracked() + } + + /// Return True if the object has been finalized by the garbage collector. + #[pyfunction] + fn is_finalized(obj: PyObjectRef) -> bool { + obj.gc_finalized() + } + + /// Freeze all objects tracked by gc. + #[pyfunction] + fn freeze() { + gc_state::gc_state().freeze(); + } + + /// Unfreeze all objects in the permanent generation. + #[pyfunction] + fn unfreeze() { + gc_state::gc_state().unfreeze(); + } + + /// Return the number of objects in the permanent generation. + #[pyfunction] + fn get_freeze_count() -> usize { + gc_state::gc_state().get_freeze_count() + } + + /// gc.garbage - list of uncollectable objects + #[pyattr] + fn garbage(vm: &VirtualMachine) -> PyListRef { + vm.ctx.gc_garbage.clone() + } + + /// gc.callbacks - list of callbacks to be invoked + #[pyattr] + fn callbacks(vm: &VirtualMachine) -> PyListRef { + vm.ctx.gc_callbacks.clone() + } + + /// Helper function to invoke GC callbacks + fn invoke_callbacks( + vm: &VirtualMachine, + phase: &str, + generation: usize, + collected: usize, + uncollectable: usize, + ) { + let callbacks_list = &vm.ctx.gc_callbacks; + let callbacks: Vec<PyObjectRef> = callbacks_list.borrow_vec().to_vec(); + if callbacks.is_empty() { + return; + } + + let phase_str: PyObjectRef = vm.ctx.new_str(phase).into(); + let info = vm.ctx.new_dict(); + let _ = info.set_item("generation", vm.ctx.new_int(generation).into(), vm); + let _ = info.set_item("collected", vm.ctx.new_int(collected).into(), vm); + let _ = info.set_item("uncollectable", vm.ctx.new_int(uncollectable).into(), vm); + + for callback in callbacks { + let _ = callback.call((phase_str.clone(), info.clone()), vm); + } + } +} diff --git a/crates/vm/src/stdlib/imp.rs b/crates/vm/src/stdlib/imp.rs index ad2cede975c..1c78e835a2d 100644 --- a/crates/vm/src/stdlib/imp.rs +++ b/crates/vm/src/stdlib/imp.rs @@ -1,6 +1,6 @@ use crate::frozen::FrozenModule; use crate::{VirtualMachine, builtins::PyBaseExceptionRef}; -pub(crate) use _imp::make_module; +pub(crate) use _imp::module_def; pub use crate::vm::resolve_frozen_alias; @@ -84,7 +84,7 @@ fn find_frozen(name: &str, vm: &VirtualMachine) -> Result<FrozenModule, FrozenEr #[pymodule(with(lock))] mod _imp { use crate::{ - PyObjectRef, PyRef, PyResult, VirtualMachine, + PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, builtins::{PyBytesRef, PyCode, PyMemoryView, PyModule, PyStrRef}, function::OptionalArg, import, version, @@ -106,7 +106,7 @@ mod _imp { #[pyfunction] fn is_builtin(name: PyStrRef, vm: &VirtualMachine) -> bool { - vm.state.module_inits.contains_key(name.as_str()) + vm.state.module_defs.contains_key(name.as_str()) } #[pyfunction] @@ -119,19 +119,44 @@ mod _imp { let sys_modules = vm.sys_module.get_attr("modules", vm).unwrap(); let name: PyStrRef = spec.get_attr("name", vm)?.try_into_value(vm)?; - let module = if let Ok(module) = sys_modules.get_item(&*name, vm) { - module - } else if let Some(make_module_func) = vm.state.module_inits.get(name.as_str()) { - make_module_func(vm).into() - } else { - vm.ctx.none() - }; - Ok(module) + // Check sys.modules first + if let Ok(module) = sys_modules.get_item(&*name, vm) { + return Ok(module); + } + + // Try multi-phase init modules first (they need special handling) + if let Some(&def) = vm.state.module_defs.get(name.as_str()) { + // Phase 1: Create module (use create slot if provided, else default creation) + let module = if let Some(create) = def.slots.create { + // Custom module creation + create(vm, &spec, def)? + } else { + // Default module creation + PyModule::from_def(def).into_ref(&vm.ctx) + }; + + // Initialize module dict and methods + // Corresponds to PyModule_FromDefAndSpec: md_def, _add_methods_to_object, PyModule_SetDocString + PyModule::__init_dict_from_def(vm, &module); + module.__init_methods(vm)?; + + // Add to sys.modules BEFORE exec (critical for circular import handling) + sys_modules.set_item(&*name, module.clone().into(), vm)?; + + // Phase 2: Call exec slot (can safely import other modules now) + if let Some(exec) = def.slots.exec { + exec(vm, &module)?; + } + + return Ok(module.into()); + } + + Ok(vm.ctx.none()) } #[pyfunction] fn exec_builtin(_mod: PyRef<PyModule>) -> i32 { - // TODO: Should we do something here? + // For multi-phase init modules, exec is already called in create_builtin 0 } diff --git a/crates/vm/src/stdlib/io.rs b/crates/vm/src/stdlib/io.rs index 54a38ef20e6..b98de6a87a5 100644 --- a/crates/vm/src/stdlib/io.rs +++ b/crates/vm/src/stdlib/io.rs @@ -1,6 +1,8 @@ /* * I/O core tools. */ +pub(crate) use _io::module_def; + cfg_if::cfg_if! { if #[cfg(any(not(target_arch = "wasm32"), target_os = "wasi"))] { use crate::common::crt_fd::Offset; @@ -19,7 +21,7 @@ cfg_if::cfg_if! { } use crate::{ - PyObjectRef, PyRef, PyResult, TryFromObject, VirtualMachine, + PyObjectRef, PyResult, TryFromObject, VirtualMachine, builtins::{PyBaseExceptionRef, PyModule}, common::os::ErrorExt, convert::{IntoPyException, ToPyException}, @@ -89,23 +91,6 @@ impl IntoPyException for std::io::Error { } } -pub(crate) fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - let ctx = &vm.ctx; - - let module = _io::make_module(vm); - - #[cfg(any(not(target_arch = "wasm32"), target_os = "wasi"))] - fileio::extend_module(vm, &module).unwrap(); - - let unsupported_operation = _io::unsupported_operation().to_owned(); - extend_module!(vm, &module, { - "UnsupportedOperation" => unsupported_operation, - "BlockingIOError" => ctx.exceptions.blocking_io_error.to_owned(), - }); - - module -} - // not used on all platforms #[derive(Copy, Clone)] #[repr(transparent)] @@ -158,8 +143,8 @@ mod _io { AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, TryFromBorrowedObject, TryFromObject, builtins::{ - PyBaseExceptionRef, PyBool, PyByteArray, PyBytes, PyBytesRef, PyMemoryView, PyStr, - PyStrRef, PyTuple, PyTupleRef, PyType, PyTypeRef, PyUtf8StrRef, + PyBaseExceptionRef, PyBool, PyByteArray, PyBytes, PyBytesRef, PyDict, PyMemoryView, + PyStr, PyStrRef, PyTuple, PyTupleRef, PyType, PyTypeRef, PyUtf8StrRef, }, class::StaticType, common::lock::{ @@ -4077,6 +4062,71 @@ mod _io { const fn line_buffering(&self) -> bool { false } + + #[pymethod] + fn __getstate__(zelf: PyRef<Self>, vm: &VirtualMachine) -> PyResult<PyTupleRef> { + let buffer = zelf.buffer(vm)?; + let content = Wtf8Buf::from_bytes(buffer.getvalue()) + .map_err(|_| vm.new_value_error("Error Retrieving Value"))?; + let pos = buffer.tell(); + drop(buffer); + + // Get __dict__ if it exists and is non-empty + let dict_obj: PyObjectRef = match zelf.as_object().dict() { + Some(d) if !d.is_empty() => d.into(), + _ => vm.ctx.none(), + }; + + // Return (content, newline, position, dict) + // TODO: store actual newline setting when it's implemented + Ok(vm.ctx.new_tuple(vec![ + vm.ctx.new_str(content).into(), + vm.ctx.new_str("\n").into(), + vm.ctx.new_int(pos).into(), + dict_obj, + ])) + } + + #[pymethod] + fn __setstate__(zelf: PyRef<Self>, state: PyTupleRef, vm: &VirtualMachine) -> PyResult<()> { + // Check closed state first (like CHECK_CLOSED) + if zelf.closed.load() { + return Err(vm.new_value_error("__setstate__ on closed file")); + } + if state.len() != 4 { + return Err(vm.new_type_error(format!( + "__setstate__ argument should be 4-tuple, got {}", + state.len() + ))); + } + + let content: PyStrRef = state[0].clone().try_into_value(vm)?; + // state[1] is newline - TODO: use when newline handling is implemented + let pos: u64 = state[2].clone().try_into_value(vm)?; + let dict = &state[3]; + + // Set content and position + let raw_bytes = content.as_bytes().to_vec(); + let mut buffer = zelf.buffer.write(); + *buffer = BufferedIO::new(Cursor::new(raw_bytes)); + buffer + .seek(SeekFrom::Start(pos)) + .map_err(|err| os_err(vm, err))?; + drop(buffer); + + // Set __dict__ if provided + if !vm.is_none(dict) { + let dict_ref: PyRef<PyDict> = dict.clone().try_into_value(vm)?; + if let Some(obj_dict) = zelf.as_object().dict() { + obj_dict.clear(); + for (key, value) in dict_ref.into_iter() { + obj_dict.set_item(&*key, value, vm)?; + } + } + } + + Ok(()) + } } #[pyattr] @@ -4225,6 +4275,65 @@ mod _io { self.closed.store(true); Ok(()) } + + #[pymethod] + fn __getstate__(zelf: PyRef<Self>, vm: &VirtualMachine) -> PyResult<PyTupleRef> { + let buffer = zelf.buffer(vm)?; + let content = buffer.getvalue(); + let pos = buffer.tell(); + drop(buffer); + + // Get __dict__ if it exists and is non-empty + let dict_obj: PyObjectRef = match zelf.as_object().dict() { + Some(d) if !d.is_empty() => d.into(), + _ => vm.ctx.none(), + }; + + // Return (content, position, dict) + Ok(vm.ctx.new_tuple(vec![ + vm.ctx.new_bytes(content).into(), + vm.ctx.new_int(pos).into(), + dict_obj, + ])) + } + + #[pymethod] + fn __setstate__(zelf: PyRef<Self>, state: PyTupleRef, vm: &VirtualMachine) -> PyResult<()> { + if zelf.closed.load() { + return Err(vm.new_value_error("__setstate__ on closed file")); + } + if state.len() != 3 { + return Err(vm.new_type_error(format!( + "__setstate__ argument should be 3-tuple, got {}", + state.len() + ))); + } + + let content: PyBytesRef = state[0].clone().try_into_value(vm)?; + let pos: u64 = state[1].clone().try_into_value(vm)?; + let dict = &state[2]; + + // Check exports and set content (like CHECK_EXPORTS) + let mut buffer = zelf.try_resizable(vm)?; + *buffer = BufferedIO::new(Cursor::new(content.as_bytes().to_vec())); + buffer + .seek(SeekFrom::Start(pos)) + .map_err(|err| os_err(vm, err))?; + drop(buffer); + + // Set __dict__ if provided + if !vm.is_none(dict) { + let dict_ref: PyRef<PyDict> = dict.clone().try_into_value(vm)?; + if let Some(obj_dict) = zelf.as_object().dict() { + obj_dict.clear(); + for (key, value) in dict_ref.into_iter() { + obj_dict.set_item(&*key, value, vm)?; + } + } + } + + Ok(()) + } } #[pyclass] @@ -4619,8 +4728,23 @@ mod _io { assert_eq!(buffered.getvalue(), data); } } -} + pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { + // Call auto-generated initialization first + __module_exec(vm, module); + + // Initialize FileIO types on non-WASM platforms + #[cfg(any(not(target_arch = "wasm32"), target_os = "wasi"))] + super::fileio::module_exec(vm, module)?; + + let unsupported_operation = unsupported_operation().to_owned(); + extend_module!(vm, module, { + "UnsupportedOperation" => unsupported_operation, + "BlockingIOError" => vm.ctx.exceptions.blocking_io_error.to_owned(), + }); + Ok(()) + } +} // disable FileIO on WASM #[cfg(any(not(target_arch = "wasm32"), target_os = "wasi"))] #[pymodule] diff --git a/crates/vm/src/stdlib/itertools.rs b/crates/vm/src/stdlib/itertools.rs index 1eedbde7a22..5a3cfee391b 100644 --- a/crates/vm/src/stdlib/itertools.rs +++ b/crates/vm/src/stdlib/itertools.rs @@ -1,24 +1,19 @@ -pub(crate) use decl::make_module; +pub(crate) use decl::module_def; #[pymodule(name = "itertools")] mod decl { - use crate::stdlib::itertools::decl::int::get_value; use crate::{ - AsObject, Py, PyObjectRef, PyPayload, PyRef, PyResult, PyWeakRef, TryFromObject, - VirtualMachine, - builtins::{ - PyGenericAlias, PyInt, PyIntRef, PyList, PyTuple, PyTupleRef, PyType, PyTypeRef, int, - tuple::IntoPyTuple, - }, + AsObject, Py, PyObjectRef, PyPayload, PyRef, PyResult, PyWeakRef, VirtualMachine, + builtins::{PyGenericAlias, PyInt, PyIntRef, PyList, PyTuple, PyType, PyTypeRef, int}, common::{ lock::{PyMutex, PyRwLock, PyRwLockWriteGuard}, rc::PyRc, }, convert::ToPyObject, - function::{ArgCallable, ArgIntoBool, FuncArgs, OptionalArg, OptionalOption, PosArgs}, + function::{ArgCallable, FuncArgs, OptionalArg, OptionalOption, PosArgs}, protocol::{PyIter, PyIterReturn, PyNumber}, raise_if_stop, - stdlib::{sys, warnings}, + stdlib::sys, types::{Constructor, IterNext, Iterable, Representable, SelfIter}, }; use crossbeam_utils::atomic::AtomicCell; @@ -28,15 +23,6 @@ mod decl { use alloc::fmt; use num_traits::{Signed, ToPrimitive}; - fn pickle_deprecation(vm: &VirtualMachine) -> PyResult<()> { - warnings::warn( - vm.ctx.exceptions.deprecation_warning, - "Itertool pickle/copy/deepcopy support will be removed in a Python 3.14.".to_owned(), - 1, - vm, - ) - } - #[pyattr] #[pyclass(name = "chain")] #[derive(Debug, PyPayload)] @@ -79,55 +65,6 @@ mod decl { ) -> PyGenericAlias { PyGenericAlias::from_args(cls, args, vm) } - - #[pymethod] - fn __reduce__(zelf: PyRef<Self>, vm: &VirtualMachine) -> PyResult<PyTupleRef> { - pickle_deprecation(vm)?; - let source = zelf.source.read().clone(); - let active = zelf.active.read().clone(); - let cls = zelf.class().to_owned(); - let empty_tuple = vm.ctx.empty_tuple.clone(); - let reduced = match source { - Some(source) => match active { - Some(active) => vm.new_tuple((cls, empty_tuple, (source, active))), - None => vm.new_tuple((cls, empty_tuple, (source,))), - }, - None => vm.new_tuple((cls, empty_tuple)), - }; - Ok(reduced) - } - - #[pymethod] - fn __setstate__(zelf: PyRef<Self>, state: PyTupleRef, vm: &VirtualMachine) -> PyResult<()> { - let args = state.as_slice(); - if args.is_empty() { - return Err(vm.new_type_error("function takes at least 1 arguments (0 given)")); - } - if args.len() > 2 { - return Err(vm.new_type_error(format!( - "function takes at most 2 arguments ({} given)", - args.len() - ))); - } - let source = &args[0]; - if args.len() == 1 { - if !PyIter::check(source.as_ref()) { - return Err(vm.new_type_error("Arguments must be iterators.")); - } - *zelf.source.write() = source.to_owned().try_into_value(vm)?; - return Ok(()); - } - let active = &args[1]; - - if !PyIter::check(source.as_ref()) || !PyIter::check(active.as_ref()) { - return Err(vm.new_type_error("Arguments must be iterators.")); - } - let mut source_lock = zelf.source.write(); - let mut active_lock = zelf.active.write(); - *source_lock = source.to_owned().try_into_value(vm)?; - *active_lock = active.to_owned().try_into_value(vm)?; - Ok(()) - } } impl SelfIter for PyItertoolsChain {} @@ -209,16 +146,7 @@ mod decl { } #[pyclass(with(IterNext, Iterable, Constructor), flags(BASETYPE))] - impl PyItertoolsCompress { - #[pymethod] - fn __reduce__(zelf: PyRef<Self>, vm: &VirtualMachine) -> (PyTypeRef, (PyIter, PyIter)) { - let _ = pickle_deprecation(vm); - ( - zelf.class().to_owned(), - (zelf.data.clone(), zelf.selectors.clone()), - ) - } - } + impl PyItertoolsCompress {} impl SelfIter for PyItertoolsCompress {} @@ -275,16 +203,7 @@ mod decl { } #[pyclass(with(IterNext, Iterable, Constructor, Representable))] - impl PyItertoolsCount { - // TODO: Implement this - // if (lz->cnt == PY_SSIZE_T_MAX) - // return Py_BuildValue("0(00)", Py_TYPE(lz), lz->long_cnt, lz->long_step); - #[pymethod] - fn __reduce__(zelf: PyRef<Self>, vm: &VirtualMachine) -> (PyTypeRef, (PyObjectRef,)) { - let _ = pickle_deprecation(vm); - (zelf.class().to_owned(), (zelf.cur.read().clone(),)) - } - } + impl PyItertoolsCount {} impl SelfIter for PyItertoolsCount {} @@ -406,16 +325,6 @@ mod decl { .ok_or_else(|| vm.new_type_error("length of unsized object."))?; Ok(*times.read()) } - - #[pymethod] - fn __reduce__(zelf: PyRef<Self>, vm: &VirtualMachine) -> PyResult<PyTupleRef> { - pickle_deprecation(vm)?; - let cls = zelf.class().to_owned(); - Ok(match zelf.times { - Some(ref times) => vm.new_tuple((cls, (zelf.object.clone(), *times.read()))), - None => vm.new_tuple((cls, (zelf.object.clone(),))), - }) - } } impl SelfIter for PyItertoolsRepeat {} @@ -474,19 +383,7 @@ mod decl { } #[pyclass(with(IterNext, Iterable, Constructor), flags(BASETYPE))] - impl PyItertoolsStarmap { - #[pymethod] - fn __reduce__( - zelf: PyRef<Self>, - vm: &VirtualMachine, - ) -> (PyTypeRef, (PyObjectRef, PyIter)) { - let _ = pickle_deprecation(vm); - ( - zelf.class().to_owned(), - (zelf.function.clone(), zelf.iterable.clone()), - ) - } - } + impl PyItertoolsStarmap {} impl SelfIter for PyItertoolsStarmap {} @@ -541,31 +438,7 @@ mod decl { } #[pyclass(with(IterNext, Iterable, Constructor), flags(BASETYPE))] - impl PyItertoolsTakewhile { - #[pymethod] - fn __reduce__( - zelf: PyRef<Self>, - vm: &VirtualMachine, - ) -> (PyTypeRef, (PyObjectRef, PyIter), u32) { - let _ = pickle_deprecation(vm); - ( - zelf.class().to_owned(), - (zelf.predicate.clone(), zelf.iterable.clone()), - zelf.stop_flag.load() as _, - ) - } - #[pymethod] - fn __setstate__( - zelf: PyRef<Self>, - state: PyObjectRef, - vm: &VirtualMachine, - ) -> PyResult<()> { - if let Ok(obj) = ArgIntoBool::try_from_object(vm, state) { - zelf.stop_flag.store(obj.into_bool()); - } - Ok(()) - } - } + impl PyItertoolsTakewhile {} impl SelfIter for PyItertoolsTakewhile {} @@ -627,32 +500,7 @@ mod decl { } #[pyclass(with(IterNext, Iterable, Constructor), flags(BASETYPE))] - impl PyItertoolsDropwhile { - #[pymethod] - fn __reduce__( - zelf: PyRef<Self>, - vm: &VirtualMachine, - ) -> (PyTypeRef, (PyObjectRef, PyIter), u32) { - let _ = pickle_deprecation(vm); - ( - zelf.class().to_owned(), - (zelf.predicate.clone().into(), zelf.iterable.clone()), - (zelf.start_flag.load() as _), - ) - } - - #[pymethod] - fn __setstate__( - zelf: PyRef<Self>, - state: PyObjectRef, - vm: &VirtualMachine, - ) -> PyResult<()> { - if let Ok(obj) = ArgIntoBool::try_from_object(vm, state) { - zelf.start_flag.store(obj.into_bool()); - } - Ok(()) - } - } + impl PyItertoolsDropwhile {} impl SelfIter for PyItertoolsDropwhile {} @@ -942,38 +790,6 @@ mod decl { .into_ref_with_type(vm, cls) .map(Into::into) } - - #[pymethod] - fn __reduce__(zelf: PyRef<Self>, vm: &VirtualMachine) -> PyResult<PyTupleRef> { - pickle_deprecation(vm)?; - let cls = zelf.class().to_owned(); - let itr = zelf.iterable.clone(); - let cur = zelf.cur.take(); - let next = zelf.next.take(); - let step = zelf.step; - match zelf.stop { - Some(stop) => Ok(vm.new_tuple((cls, (itr, next, stop, step), (cur,)))), - _ => Ok(vm.new_tuple((cls, (itr, next, vm.new_pyobj(()), step), (cur,)))), - } - } - - #[pymethod] - fn __setstate__(zelf: PyRef<Self>, state: PyTupleRef, vm: &VirtualMachine) -> PyResult<()> { - let args = state.as_slice(); - if args.len() != 1 { - return Err(vm.new_type_error(format!( - "function takes exactly 1 argument ({} given)", - args.len() - ))); - } - let cur = &args[0]; - if let Ok(cur) = cur.try_to_value(vm) { - zelf.cur.store(cur); - } else { - return Err(vm.new_type_error("Argument must be usize.")); - } - Ok(()) - } } impl SelfIter for PyItertoolsIslice {} @@ -1037,19 +853,7 @@ mod decl { } #[pyclass(with(IterNext, Iterable, Constructor), flags(BASETYPE))] - impl PyItertoolsFilterFalse { - #[pymethod] - fn __reduce__( - zelf: PyRef<Self>, - vm: &VirtualMachine, - ) -> (PyTypeRef, (PyObjectRef, PyIter)) { - let _ = pickle_deprecation(vm); - ( - zelf.class().to_owned(), - (zelf.predicate.clone(), zelf.iterable.clone()), - ) - } - } + impl PyItertoolsFilterFalse {} impl SelfIter for PyItertoolsFilterFalse {} @@ -1106,59 +910,7 @@ mod decl { } #[pyclass(with(IterNext, Iterable, Constructor))] - impl PyItertoolsAccumulate { - #[pymethod] - fn __setstate__( - zelf: PyRef<Self>, - state: PyObjectRef, - _vm: &VirtualMachine, - ) -> PyResult<()> { - *zelf.acc_value.write() = Some(state); - Ok(()) - } - - #[pymethod] - fn __reduce__(zelf: PyRef<Self>, vm: &VirtualMachine) -> PyTupleRef { - let _ = pickle_deprecation(vm); - let class = zelf.class().to_owned(); - let bin_op = zelf.bin_op.clone(); - let it = zelf.iterable.clone(); - let acc_value = zelf.acc_value.read().clone(); - if let Some(initial) = &zelf.initial { - let chain_args = PyList::from(vec![initial.clone(), it.to_pyobject(vm)]); - let chain = PyItertoolsChain { - source: PyRwLock::new(Some(chain_args.to_pyobject(vm).get_iter(vm).unwrap())), - active: PyRwLock::new(None), - }; - let tup = vm.new_tuple((chain, bin_op)); - return vm.new_tuple((class, tup, acc_value)); - } - match acc_value { - Some(obj) if obj.is(&vm.ctx.none) => { - let chain_args = PyList::from(vec![]); - let chain = PyItertoolsChain { - source: PyRwLock::new(Some( - chain_args.to_pyobject(vm).get_iter(vm).unwrap(), - )), - active: PyRwLock::new(None), - } - .into_pyobject(vm); - let acc = Self { - iterable: PyIter::new(chain), - bin_op, - initial: None, - acc_value: PyRwLock::new(None), - }; - let tup = vm.new_tuple((acc, 1, None::<PyObjectRef>)); - let islice_cls = PyItertoolsIslice::class(&vm.ctx).to_owned(); - return vm.new_tuple((islice_cls, tup)); - } - _ => {} - } - let tup = vm.new_tuple((it, bin_op)); - vm.new_tuple((class, tup, acc_value)) - } - } + impl PyItertoolsAccumulate {} impl SelfIter for PyItertoolsAccumulate {} @@ -1359,58 +1111,6 @@ mod decl { self.cur.store(idxs.len() - 1); } } - - #[pymethod] - fn __setstate__(zelf: PyRef<Self>, state: PyTupleRef, vm: &VirtualMachine) -> PyResult<()> { - let args = state.as_slice(); - if args.len() != zelf.pools.len() { - return Err(vm.new_type_error("Invalid number of arguments")); - } - let mut idxs: PyRwLockWriteGuard<'_, Vec<usize>> = zelf.idxs.write(); - idxs.clear(); - for s in 0..args.len() { - let index = get_value(state.get(s).unwrap()).to_usize().unwrap(); - let pool_size = zelf.pools.get(s).unwrap().len(); - if pool_size == 0 { - zelf.stop.store(true); - return Ok(()); - } - if index >= pool_size { - idxs.push(pool_size - 1); - } else { - idxs.push(index); - } - } - zelf.stop.store(false); - Ok(()) - } - - #[pymethod] - fn __reduce__(zelf: PyRef<Self>, vm: &VirtualMachine) -> PyTupleRef { - let _ = pickle_deprecation(vm); - let class = zelf.class().to_owned(); - - if zelf.stop.load() { - return vm.new_tuple((class, (vm.ctx.empty_tuple.clone(),))); - } - - let mut pools: Vec<PyObjectRef> = Vec::new(); - for element in &zelf.pools { - pools.push(element.clone().into_pytuple(vm).into()); - } - - let mut indices: Vec<PyObjectRef> = Vec::new(); - - for item in &zelf.idxs.read()[..] { - indices.push(vm.new_pyobj(*item)); - } - - vm.new_tuple(( - class, - pools.clone().into_pytuple(vm), - indices.into_pytuple(vm), - )) - } } impl SelfIter for PyItertoolsProduct {} @@ -1492,36 +1192,7 @@ mod decl { } #[pyclass(with(IterNext, Iterable, Constructor))] - impl PyItertoolsCombinations { - #[pymethod] - fn __reduce__(zelf: PyRef<Self>, vm: &VirtualMachine) -> PyTupleRef { - let _ = pickle_deprecation(vm); - let r = zelf.r.load(); - - let class = zelf.class().to_owned(); - - if zelf.exhausted.load() { - return vm.new_tuple(( - class, - vm.new_tuple((vm.ctx.empty_tuple.clone(), vm.ctx.new_int(r))), - )); - } - - let tup = vm.new_tuple((zelf.pool.clone().into_pytuple(vm), vm.ctx.new_int(r))); - - if zelf.result.read().is_none() { - vm.new_tuple((class, tup)) - } else { - let mut indices: Vec<PyObjectRef> = Vec::new(); - - for item in &zelf.indices.read()[..r] { - indices.push(vm.new_pyobj(*item)); - } - - vm.new_tuple((class, tup, indices.into_pytuple(vm))) - } - } - } + impl PyItertoolsCombinations {} impl SelfIter for PyItertoolsCombinations {} impl IterNext for PyItertoolsCombinations { @@ -1730,16 +1401,7 @@ mod decl { } #[pyclass(with(IterNext, Iterable, Constructor))] - impl PyItertoolsPermutations { - #[pymethod] - fn __reduce__(zelf: PyRef<Self>, vm: &VirtualMachine) -> PyRef<PyTuple> { - let _ = pickle_deprecation(vm); - vm.new_tuple(( - zelf.class().to_owned(), - vm.new_tuple((zelf.pool.clone(), vm.ctx.new_int(zelf.r.load()))), - )) - } - } + impl PyItertoolsPermutations {} impl SelfIter for PyItertoolsPermutations {} @@ -1846,32 +1508,7 @@ mod decl { } #[pyclass(with(IterNext, Iterable, Constructor))] - impl PyItertoolsZipLongest { - #[pymethod] - fn __reduce__(zelf: PyRef<Self>, vm: &VirtualMachine) -> PyResult<PyTupleRef> { - pickle_deprecation(vm)?; - let args: Vec<PyObjectRef> = zelf - .iterators - .iter() - .map(|i| i.clone().to_pyobject(vm)) - .collect(); - Ok(vm.new_tuple(( - zelf.class().to_owned(), - vm.new_tuple(args), - zelf.fillvalue.read().to_owned(), - ))) - } - - #[pymethod] - fn __setstate__( - zelf: PyRef<Self>, - state: PyObjectRef, - _vm: &VirtualMachine, - ) -> PyResult<()> { - *zelf.fillvalue.write() = state; - Ok(()) - } - } + impl PyItertoolsZipLongest {} impl SelfIter for PyItertoolsZipLongest {} diff --git a/crates/vm/src/stdlib/marshal.rs b/crates/vm/src/stdlib/marshal.rs index aced9e48773..cf7abe65194 100644 --- a/crates/vm/src/stdlib/marshal.rs +++ b/crates/vm/src/stdlib/marshal.rs @@ -1,5 +1,5 @@ // spell-checker:ignore pyfrozen pycomplex -pub(crate) use decl::make_module; +pub(crate) use decl::module_def; #[pymodule(name = "marshal")] mod decl { diff --git a/crates/vm/src/stdlib/mod.rs b/crates/vm/src/stdlib/mod.rs index e46f333a28b..1d401e50e02 100644 --- a/crates/vm/src/stdlib/mod.rs +++ b/crates/vm/src/stdlib/mod.rs @@ -1,3 +1,4 @@ +mod _abc; #[cfg(feature = "ast")] pub(crate) mod ast; pub mod atexit; @@ -6,6 +7,7 @@ mod codecs; mod collections; pub mod errno; mod functools; +mod gc; mod imp; pub mod io; mod itertools; @@ -61,94 +63,67 @@ mod winapi; #[cfg(windows)] mod winreg; -use crate::{PyRef, VirtualMachine, builtins::PyModule}; -use alloc::borrow::Cow; -use std::collections::HashMap; +use crate::{Context, builtins::PyModuleDef}; -pub type StdlibInitFunc = Box<py_dyn_fn!(dyn Fn(&VirtualMachine) -> PyRef<PyModule>)>; -pub type StdlibMap = HashMap<Cow<'static, str>, StdlibInitFunc, ahash::RandomState>; - -pub fn get_module_inits() -> StdlibMap { - macro_rules! modules { - { - $( - #[cfg($cfg:meta)] - { $( $key:expr => $val:expr),* $(,)? } - )* - } => {{ - let modules = [ - $( - $(#[cfg($cfg)] (Cow::<'static, str>::from($key), Box::new($val) as StdlibInitFunc),)* - )* - ]; - modules.into_iter().collect() - }}; - } - modules! { - #[cfg(all())] - { - "atexit" => atexit::make_module, - "_codecs" => codecs::make_module, - "_collections" => collections::make_module, - "errno" => errno::make_module, - "_functools" => functools::make_module, - "itertools" => itertools::make_module, - "_io" => io::make_module, - "marshal" => marshal::make_module, - "_operator" => operator::make_module, - "_signal" => signal::make_module, - "_sre" => sre::make_module, - "_stat" => stat::make_module, - "_sysconfig" => sysconfig::make_module, - "_string" => string::make_module, - "time" => time::make_module, - "_typing" => typing::make_module, - "_weakref" => weakref::make_module, - "_imp" => imp::make_module, - "_warnings" => warnings::make_module, - sys::sysconfigdata_name() => sysconfigdata::make_module, - } - // parser related modules: +/// Returns module definitions for multi-phase init modules. +/// +/// These modules use multi-phase initialization pattern: +/// 1. Create module from def and add to sys.modules +/// 2. Call exec slot (can safely import other modules without circular import issues) +pub fn builtin_module_defs(ctx: &Context) -> Vec<&'static PyModuleDef> { + vec![ + _abc::module_def(ctx), #[cfg(feature = "ast")] - { - "_ast" => ast::make_module, - } - // compiler related modules: - #[cfg(feature = "compiler")] - { - "_symtable" => symtable::make_module, - } + ast::module_def(ctx), + atexit::module_def(ctx), + codecs::module_def(ctx), + collections::module_def(ctx), + #[cfg(all( + any(target_os = "linux", target_os = "macos", target_os = "windows"), + not(any(target_env = "musl", target_env = "sgx")) + ))] + ctypes::module_def(ctx), + errno::module_def(ctx), + functools::module_def(ctx), + gc::module_def(ctx), + imp::module_def(ctx), + io::module_def(ctx), + itertools::module_def(ctx), + marshal::module_def(ctx), + #[cfg(windows)] + msvcrt::module_def(ctx), + #[cfg(windows)] + nt::module_def(ctx), + operator::module_def(ctx), #[cfg(any(unix, target_os = "wasi"))] - { - "posix" => posix::make_module, - // "fcntl" => fcntl::make_module, - } - #[cfg(feature = "threading")] - { - "_thread" => thread::make_module, - } - // Unix-only + posix::module_def(ctx), + #[cfg(all( + any(not(target_arch = "wasm32"), target_os = "wasi"), + not(any(unix, windows)) + ))] + posix::module_def(ctx), #[cfg(all( unix, not(any(target_os = "ios", target_os = "wasi", target_os = "redox")) ))] - { - "pwd" => pwd::make_module, - } - // Windows-only + pwd::module_def(ctx), + signal::module_def(ctx), + sre::module_def(ctx), + stat::module_def(ctx), + string::module_def(ctx), + #[cfg(feature = "compiler")] + symtable::module_def(ctx), + sysconfigdata::module_def(ctx), + sysconfig::module_def(ctx), + #[cfg(feature = "threading")] + thread::module_def(ctx), + time::module_def(ctx), + typing::module_def(ctx), + warnings::module_def(ctx), + weakref::module_def(ctx), #[cfg(windows)] - { - "nt" => nt::make_module, - "msvcrt" => msvcrt::make_module, - "_winapi" => winapi::make_module, - "winreg" => winreg::make_module, - } - #[cfg(all( - any(target_os = "linux", target_os = "macos", target_os = "windows"), - not(any(target_env = "musl", target_env = "sgx")) - ))] - { - "_ctypes" => ctypes::make_module, - } - } + winapi::module_def(ctx), + #[cfg(windows)] + winreg::module_def(ctx), + ] } diff --git a/crates/vm/src/stdlib/msvcrt.rs b/crates/vm/src/stdlib/msvcrt.rs index cf0dac2c9db..9986db3ff8c 100644 --- a/crates/vm/src/stdlib/msvcrt.rs +++ b/crates/vm/src/stdlib/msvcrt.rs @@ -31,8 +31,25 @@ mod msvcrt { fn _getwche() -> u32; fn _putch(c: u32) -> i32; fn _putwch(c: u16) -> u32; + fn _ungetch(c: i32) -> i32; + fn _ungetwch(c: u32) -> u32; + fn _locking(fd: i32, mode: i32, nbytes: i64) -> i32; + fn _heapmin() -> i32; + fn _kbhit() -> i32; } + // Locking mode constants + #[pyattr] + const LK_UNLCK: i32 = 0; // Unlock + #[pyattr] + const LK_LOCK: i32 = 1; // Lock (blocking) + #[pyattr] + const LK_NBLCK: i32 = 2; // Non-blocking lock + #[pyattr] + const LK_RLCK: i32 = 3; // Lock for reading (same as LK_LOCK) + #[pyattr] + const LK_NBRLCK: i32 = 4; // Non-blocking lock for reading (same as LK_NBLCK) + #[pyfunction] fn getch() -> Vec<u8> { let c = unsafe { _getch() }; @@ -73,6 +90,60 @@ mod msvcrt { Ok(()) } + #[pyfunction] + fn ungetch(b: PyRef<PyBytes>, vm: &VirtualMachine) -> PyResult<()> { + let &c = b.as_bytes().iter().exactly_one().map_err(|_| { + vm.new_type_error("ungetch() argument must be a byte string of length 1") + })?; + let ret = unsafe { suppress_iph!(_ungetch(c as i32)) }; + if ret == -1 { + // EOF returned means the buffer is full + Err(vm.new_os_error(libc::ENOSPC)) + } else { + Ok(()) + } + } + + #[pyfunction] + fn ungetwch(s: PyStrRef, vm: &VirtualMachine) -> PyResult<()> { + let c = + s.as_str().chars().exactly_one().map_err(|_| { + vm.new_type_error("ungetwch() argument must be a string of length 1") + })?; + let ret = unsafe { suppress_iph!(_ungetwch(c as u32)) }; + if ret == 0xFFFF { + // WEOF returned means the buffer is full + Err(vm.new_os_error(libc::ENOSPC)) + } else { + Ok(()) + } + } + + #[pyfunction] + fn kbhit() -> i32 { + unsafe { _kbhit() } + } + + #[pyfunction] + fn locking(fd: i32, mode: i32, nbytes: i64, vm: &VirtualMachine) -> PyResult<()> { + let ret = unsafe { suppress_iph!(_locking(fd, mode, nbytes)) }; + if ret == -1 { + Err(vm.new_last_errno_error()) + } else { + Ok(()) + } + } + + #[pyfunction] + fn heapmin(vm: &VirtualMachine) -> PyResult<()> { + let ret = unsafe { suppress_iph!(_heapmin()) }; + if ret == -1 { + Err(vm.new_last_errno_error()) + } else { + Ok(()) + } + } + unsafe extern "C" { fn _setmode(fd: crt_fd::Borrowed<'_>, flags: i32) -> i32; } diff --git a/crates/vm/src/stdlib/nt.rs b/crates/vm/src/stdlib/nt.rs index a32959808c0..c8c699fb3fa 100644 --- a/crates/vm/src/stdlib/nt.rs +++ b/crates/vm/src/stdlib/nt.rs @@ -1,19 +1,12 @@ // spell-checker:disable -use crate::{PyRef, VirtualMachine, builtins::PyModule}; - +pub(crate) use module::module_def; pub use module::raw_set_handle_inheritable; -pub(crate) fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - let module = module::make_module(vm); - super::os::extend_module(vm, &module); - module -} - #[pymodule(name = "nt", with(super::os::_os))] pub(crate) mod module { use crate::{ - PyResult, TryFromObject, VirtualMachine, + Py, PyResult, TryFromObject, VirtualMachine, builtins::{PyBaseExceptionRef, PyDictRef, PyListRef, PyStrRef, PyTupleRef}, common::{crt_fd, suppress_iph, windows::ToWideString}, convert::ToPyException, @@ -1923,4 +1916,13 @@ pub(crate) mod module { pub(crate) fn support_funcs() -> Vec<SupportFunc> { Vec::new() } + + pub(crate) fn module_exec( + vm: &VirtualMachine, + module: &Py<crate::builtins::PyModule>, + ) -> PyResult<()> { + __module_exec(vm, module); + super::super::os::module_exec(vm, module)?; + Ok(()) + } } diff --git a/crates/vm/src/stdlib/operator.rs b/crates/vm/src/stdlib/operator.rs index 7877ddb0114..c5fa03a12d6 100644 --- a/crates/vm/src/stdlib/operator.rs +++ b/crates/vm/src/stdlib/operator.rs @@ -1,4 +1,4 @@ -pub(crate) use _operator::make_module; +pub(crate) use _operator::module_def; #[pymodule] mod _operator { diff --git a/crates/vm/src/stdlib/os.rs b/crates/vm/src/stdlib/os.rs index 416f26018cb..95836e38337 100644 --- a/crates/vm/src/stdlib/os.rs +++ b/crates/vm/src/stdlib/os.rs @@ -149,6 +149,8 @@ pub(super) mod _os { use super::{DirFd, FollowSymlinks, SupportFunc}; #[cfg(windows)] use crate::common::windows::ToWideString; + #[cfg(any(unix, windows))] + use crate::utils::ToCString; use crate::{ AsObject, Py, PyObjectRef, PyPayload, PyRef, PyResult, TryFromObject, builtins::{ @@ -168,7 +170,6 @@ pub(super) mod _os { protocol::PyIterReturn, recursion::ReprGuard, types::{IterNext, Iterable, PyStructSequence, Representable, SelfIter}, - utils::ToCString, vm::VirtualMachine, }; use core::time::Duration; @@ -906,9 +907,11 @@ pub(super) mod _os { #[pyarg(any, default)] #[pystruct_sequence(skip)] pub st_blocks: i64, + #[cfg(windows)] #[pyarg(any, default)] #[pystruct_sequence(skip)] pub st_reparse_tag: u32, + #[cfg(windows)] #[pyarg(any, default)] #[pystruct_sequence(skip)] pub st_file_attributes: u32, @@ -918,7 +921,7 @@ pub(super) mod _os { fn from_stat(stat: &StatStruct, vm: &VirtualMachine) -> Self { let (atime, mtime, ctime); #[cfg(any(unix, windows))] - #[cfg(not(target_os = "netbsd"))] + #[cfg(not(any(target_os = "netbsd", target_os = "wasi")))] { atime = (stat.st_atime, stat.st_atime_nsec); mtime = (stat.st_mtime, stat.st_mtime_nsec); @@ -943,13 +946,8 @@ pub(super) mod _os { #[cfg(windows)] let st_reparse_tag = stat.st_reparse_tag; - #[cfg(not(windows))] - let st_reparse_tag = 0; - #[cfg(windows)] let st_file_attributes = stat.st_file_attributes; - #[cfg(not(windows))] - let st_file_attributes = 0; // On Windows, combine st_ino and st_ino_high into a 128-bit value // like _pystat_l128_from_l64_l64 @@ -986,7 +984,9 @@ pub(super) mod _os { st_blksize, #[cfg(not(windows))] st_blocks, + #[cfg(windows)] st_reparse_tag, + #[cfg(windows)] st_file_attributes, } } @@ -1268,13 +1268,64 @@ pub(super) mod _os { } #[pyfunction] - fn link(src: OsPath, dst: OsPath, vm: &VirtualMachine) -> PyResult<()> { - fs::hard_link(&src.path, &dst.path).map_err(|err| { - let builder = err.to_os_error_builder(vm); - let builder = builder.filename(src.filename(vm)); - let builder = builder.filename2(dst.filename(vm)); - builder.build(vm).upcast() - }) + fn link( + src: OsPath, + dst: OsPath, + follow_symlinks: FollowSymlinks, + vm: &VirtualMachine, + ) -> PyResult<()> { + #[cfg(unix)] + { + use std::os::unix::ffi::OsStrExt; + let src_cstr = std::ffi::CString::new(src.path.as_os_str().as_bytes()) + .map_err(|_| vm.new_value_error("embedded null byte"))?; + let dst_cstr = std::ffi::CString::new(dst.path.as_os_str().as_bytes()) + .map_err(|_| vm.new_value_error("embedded null byte"))?; + + let flags = if follow_symlinks.0 { + libc::AT_SYMLINK_FOLLOW + } else { + 0 + }; + + let ret = unsafe { + libc::linkat( + libc::AT_FDCWD, + src_cstr.as_ptr(), + libc::AT_FDCWD, + dst_cstr.as_ptr(), + flags, + ) + }; + + if ret != 0 { + let err = std::io::Error::last_os_error(); + let builder = err.to_os_error_builder(vm); + let builder = builder.filename(src.filename(vm)); + let builder = builder.filename2(dst.filename(vm)); + return Err(builder.build(vm).upcast()); + } + + Ok(()) + } + + #[cfg(not(unix))] + { + // On non-Unix platforms, ignore follow_symlinks if it's the default value + // or raise NotImplementedError if explicitly set to False + if !follow_symlinks.0 { + return Err(vm.new_not_implemented_error( + "link: follow_symlinks unavailable on this platform", + )); + } + + fs::hard_link(&src.path, &dst.path).map_err(|err| { + let builder = err.to_os_error_builder(vm); + let builder = builder.filename(src.filename(vm)); + let builder = builder.filename2(dst.filename(vm)); + builder.build(vm).upcast() + }) + } } #[cfg(any(unix, windows))] @@ -1842,6 +1893,7 @@ pub(super) mod _os { SupportFunc::new("access", Some(false), Some(false), None), SupportFunc::new("chdir", None, Some(false), Some(false)), // chflags Some, None Some + SupportFunc::new("link", Some(false), Some(false), Some(cfg!(unix))), SupportFunc::new("listdir", Some(LISTDIR_FD), Some(false), Some(false)), SupportFunc::new("mkdir", Some(false), Some(MKDIR_DIR_FD), Some(false)), // mkfifo Some Some None @@ -1897,21 +1949,21 @@ impl SupportFunc { } } -pub fn extend_module(vm: &VirtualMachine, module: &Py<PyModule>) { +pub fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { let support_funcs = _os::support_funcs(); let supports_fd = PySet::default().into_ref(&vm.ctx); let supports_dir_fd = PySet::default().into_ref(&vm.ctx); let supports_follow_symlinks = PySet::default().into_ref(&vm.ctx); for support in support_funcs { - let func_obj = module.get_attr(support.name, vm).unwrap(); + let func_obj = module.get_attr(support.name, vm)?; if support.fd.unwrap_or(false) { - supports_fd.clone().add(func_obj.clone(), vm).unwrap(); + supports_fd.clone().add(func_obj.clone(), vm)?; } if support.dir_fd.unwrap_or(false) { - supports_dir_fd.clone().add(func_obj.clone(), vm).unwrap(); + supports_dir_fd.clone().add(func_obj.clone(), vm)?; } if support.follow_symlinks.unwrap_or(false) { - supports_follow_symlinks.clone().add(func_obj, vm).unwrap(); + supports_follow_symlinks.clone().add(func_obj, vm)?; } } @@ -1921,6 +1973,8 @@ pub fn extend_module(vm: &VirtualMachine, module: &Py<PyModule>) { "supports_follow_symlinks" => supports_follow_symlinks, "error" => vm.ctx.exceptions.os_error.to_owned(), }); + + Ok(()) } #[cfg(not(windows))] diff --git a/crates/vm/src/stdlib/posix.rs b/crates/vm/src/stdlib/posix.rs index 65a659790b2..5bbfef0f93b 100644 --- a/crates/vm/src/stdlib/posix.rs +++ b/crates/vm/src/stdlib/posix.rs @@ -1,8 +1,9 @@ // spell-checker:disable -use crate::{PyRef, VirtualMachine, builtins::PyModule}; use std::os::fd::BorrowedFd; +pub(crate) use module::module_def; + pub fn set_inheritable(fd: BorrowedFd<'_>, inheritable: bool) -> nix::Result<()> { use nix::fcntl; let flags = fcntl::FdFlag::from_bits_truncate(fcntl::fcntl(fd, fcntl::FcntlArg::F_GETFD)?); @@ -14,12 +15,6 @@ pub fn set_inheritable(fd: BorrowedFd<'_>, inheritable: bool) -> nix::Result<()> Ok(()) } -pub(crate) fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - let module = module::make_module(vm); - super::os::extend_module(vm, &module); - module -} - #[pymodule(name = "posix", with(super::os::_os))] pub mod module { use crate::{ @@ -46,6 +41,10 @@ pub mod module { }; use strum_macros::{EnumIter, EnumString}; + #[cfg(target_os = "android")] + #[pyattr] + use libc::{SCHED_DEADLINE, SCHED_NORMAL}; + #[cfg(target_os = "freebsd")] #[pyattr] use libc::{MFD_HUGE_MASK, SF_MNOWAIT, SF_NOCACHE, SF_NODISKIO, SF_SYNC}; @@ -661,6 +660,19 @@ pub mod module { } fn py_os_after_fork_child(vm: &VirtualMachine) { + // Reset low-level state before any Python code runs in the child. + // Signal triggers from the parent must not fire in the child. + crate::signal::clear_after_fork(); + crate::stdlib::signal::_signal::clear_wakeup_fd_after_fork(); + + // Reset weakref stripe locks that may have been held during fork. + #[cfg(feature = "threading")] + crate::object::reset_weakref_locks_after_fork(); + + // Mark all other threads as done before running Python callbacks + #[cfg(feature = "threading")] + crate::stdlib::thread::after_fork_child(vm); + let after_forkers_child: Vec<PyObjectRef> = vm.state.after_forkers_child.lock().clone(); run_at_forkers(after_forkers_child, false, vm); } @@ -670,8 +682,64 @@ pub mod module { run_at_forkers(after_forkers_parent, false, vm); } + /// Warn if forking from a multi-threaded process + fn warn_if_multi_threaded(name: &str, vm: &VirtualMachine) { + // Only check threading if it was already imported + // Avoid vm.import() which can execute arbitrary Python code in the fork path + let threading = match vm + .sys_module + .get_attr("modules", vm) + .and_then(|m| m.get_item("threading", vm)) + { + Ok(m) => m, + Err(_) => return, + }; + let active = threading.get_attr("_active", vm).ok(); + let limbo = threading.get_attr("_limbo", vm).ok(); + + let count_dict = |obj: Option<crate::PyObjectRef>| -> usize { + obj.and_then(|o| o.length_opt(vm)) + .and_then(|r| r.ok()) + .unwrap_or(0) + }; + + let num_threads = count_dict(active) + count_dict(limbo); + if num_threads > 1 { + // Use Python warnings module to ensure filters are applied correctly + let Ok(warnings) = vm.import("warnings", 0) else { + return; + }; + let Ok(warn_fn) = warnings.get_attr("warn", vm) else { + return; + }; + + let pid = unsafe { libc::getpid() }; + let msg = format!( + "This process (pid={}) is multi-threaded, use of {}() may lead to deadlocks in the child.", + pid, name + ); + + // Call warnings.warn(message, DeprecationWarning, stacklevel=2) + // stacklevel=2 to point to the caller of fork() + let args = crate::function::FuncArgs::new( + vec![ + vm.ctx.new_str(msg).into(), + vm.ctx.exceptions.deprecation_warning.as_object().to_owned(), + ], + crate::function::KwArgs::new( + [("stacklevel".to_owned(), vm.ctx.new_int(2).into())] + .into_iter() + .collect(), + ), + ); + let _ = warn_fn.call(args, vm); + } + } + #[pyfunction] fn fork(vm: &VirtualMachine) -> i32 { + warn_if_multi_threaded("fork", vm); + let pid: i32; py_os_before_fork(vm); unsafe { @@ -1757,6 +1825,23 @@ pub mod module { libc::WTERMSIG(status) } + #[cfg(target_os = "linux")] + #[pyfunction] + fn pidfd_open( + pid: libc::pid_t, + flags: OptionalArg<u32>, + vm: &VirtualMachine, + ) -> PyResult<OwnedFd> { + let flags = flags.unwrap_or(0); + let fd = unsafe { libc::syscall(libc::SYS_pidfd_open, pid, flags) as libc::c_long }; + if fd == -1 { + Err(vm.new_last_errno_error()) + } else { + // Safety: syscall returns a new owned file descriptor. + Ok(unsafe { OwnedFd::from_raw_fd(fd as libc::c_int) }) + } + } + #[pyfunction] fn waitpid(pid: libc::pid_t, opt: i32, vm: &VirtualMachine) -> PyResult<(libc::pid_t, i32)> { let mut status = 0; @@ -2501,7 +2586,14 @@ pub mod module { headers, trailers, ); - res.map_err(|err| err.into_pyexception(vm))?; + // On macOS, sendfile can return EAGAIN even when some bytes were written. + // In that case, we should return the number of bytes written rather than + // raising an exception. Only raise an error if no bytes were written. + if let Err(err) = res + && written == 0 + { + return Err(err.into_pyexception(vm)); + } Ok(vm.ctx.new_int(written as u64).into()) } @@ -2528,4 +2620,13 @@ pub mod module { } Ok(buf) } + + pub(crate) fn module_exec( + vm: &VirtualMachine, + module: &Py<crate::builtins::PyModule>, + ) -> PyResult<()> { + __module_exec(vm, module); + super::super::os::module_exec(vm, module)?; + Ok(()) + } } diff --git a/crates/vm/src/stdlib/posix_compat.rs b/crates/vm/src/stdlib/posix_compat.rs index 0fabd6a1f45..89d3d94d7b2 100644 --- a/crates/vm/src/stdlib/posix_compat.rs +++ b/crates/vm/src/stdlib/posix_compat.rs @@ -1,18 +1,13 @@ // spell-checker:disable //! `posix` compatible module for `not(any(unix, windows))` -use crate::{PyRef, VirtualMachine, builtins::PyModule}; -pub(crate) fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - let module = module::make_module(vm); - super::os::extend_module(vm, &module); - module -} +pub(crate) use module::module_def; #[pymodule(name = "posix", with(super::os::_os))] pub(crate) mod module { use crate::{ - PyObjectRef, PyResult, VirtualMachine, + Py, PyObjectRef, PyResult, VirtualMachine, builtins::PyStrRef, convert::IntoPyException, ospath::OsPath, @@ -71,4 +66,13 @@ pub(crate) mod module { pub(crate) fn support_funcs() -> Vec<SupportFunc> { Vec::new() } + + pub(crate) fn module_exec( + vm: &VirtualMachine, + module: &Py<crate::builtins::PyModule>, + ) -> PyResult<()> { + __module_exec(vm, module); + super::super::os::module_exec(vm, module)?; + Ok(()) + } } diff --git a/crates/vm/src/stdlib/pwd.rs b/crates/vm/src/stdlib/pwd.rs index 6405ed7be91..e8aee608cc7 100644 --- a/crates/vm/src/stdlib/pwd.rs +++ b/crates/vm/src/stdlib/pwd.rs @@ -1,6 +1,6 @@ // spell-checker:disable -pub(crate) use pwd::make_module; +pub(crate) use pwd::module_def; #[pymodule] mod pwd { diff --git a/crates/vm/src/stdlib/signal.rs b/crates/vm/src/stdlib/signal.rs index dd0d9a7a96f..1a73c454b8f 100644 --- a/crates/vm/src/stdlib/signal.rs +++ b/crates/vm/src/stdlib/signal.rs @@ -1,24 +1,12 @@ // spell-checker:disable -use crate::{PyRef, VirtualMachine, builtins::PyModule}; - -pub(crate) fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - let module = _signal::make_module(vm); - - #[cfg(any(unix, windows))] - _signal::init_signal_handlers(&module, vm); - - module -} +pub(crate) use _signal::module_def; #[pymodule] pub(crate) mod _signal { #[cfg(any(unix, windows))] - use crate::{ - Py, - convert::{IntoPyException, TryFromBorrowedObject}, - }; - use crate::{PyObjectRef, PyResult, VirtualMachine, signal}; + use crate::convert::{IntoPyException, TryFromBorrowedObject}; + use crate::{Py, PyObjectRef, PyResult, VirtualMachine, signal}; #[cfg(unix)] use crate::{ builtins::PyTypeRef, @@ -113,6 +101,14 @@ pub(crate) mod _signal { #[pyattr] pub use libc::{SIGABRT, SIGFPE, SIGILL, SIGINT, SIGSEGV, SIGTERM}; + // Windows-specific control events for GenerateConsoleCtrlEvent + #[cfg(windows)] + #[pyattr] + const CTRL_C_EVENT: u32 = 0; + #[cfg(windows)] + #[pyattr] + const CTRL_BREAK_EVENT: u32 = 1; + #[cfg(unix)] #[pyattr] use libc::{ @@ -214,7 +210,7 @@ pub(crate) mod _signal { match usize::try_from_borrowed_object(vm, &handler).ok() { Some(SIG_DFL) => SIG_DFL, Some(SIG_IGN) => SIG_IGN, - None if handler.is_callable() => run_signal as sighandler_t, + None if handler.is_callable() => run_signal as *const () as sighandler_t, _ => return Err(vm.new_type_error( "signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object", )), @@ -431,6 +427,40 @@ pub(crate) mod _signal { } } + #[cfg(target_os = "linux")] + #[pyfunction] + fn pidfd_send_signal( + pidfd: i32, + sig: i32, + siginfo: OptionalArg<PyObjectRef>, + flags: OptionalArg<u32>, + vm: &VirtualMachine, + ) -> PyResult<()> { + signal::assert_in_range(sig, vm)?; + if let OptionalArg::Present(obj) = siginfo + && !vm.is_none(&obj) + { + return Err(vm.new_type_error("siginfo must be None".to_owned())); + } + + let flags = flags.unwrap_or(0); + let ret = unsafe { + libc::syscall( + libc::SYS_pidfd_send_signal, + pidfd, + sig, + std::ptr::null::<libc::siginfo_t>(), + flags, + ) as libc::c_long + }; + + if ret == -1 { + Err(vm.new_last_errno_error()) + } else { + Ok(()) + } + } + #[cfg(all(unix, not(target_os = "redox")))] #[pyfunction(name = "siginterrupt")] fn py_siginterrupt(signum: i32, flag: i32, vm: &VirtualMachine) -> PyResult<()> { @@ -648,4 +678,23 @@ pub(crate) mod _signal { // TODO: handle _res < 1, support warn_on_full_buffer } } + + /// Reset wakeup fd after fork in child process. + /// The child must not write to the parent's wakeup fd. + #[cfg(unix)] + pub(crate) fn clear_wakeup_fd_after_fork() { + WAKEUP.store(INVALID_WAKEUP, Ordering::Relaxed); + } + + pub(crate) fn module_exec( + vm: &VirtualMachine, + module: &Py<crate::builtins::PyModule>, + ) -> PyResult<()> { + __module_exec(vm, module); + + #[cfg(any(unix, windows))] + init_signal_handlers(module, vm); + + Ok(()) + } } diff --git a/crates/vm/src/stdlib/sre.rs b/crates/vm/src/stdlib/sre.rs index 04ff0695b6b..692b9754214 100644 --- a/crates/vm/src/stdlib/sre.rs +++ b/crates/vm/src/stdlib/sre.rs @@ -1,4 +1,4 @@ -pub(crate) use _sre::make_module; +pub(crate) use _sre::module_def; #[pymodule] mod _sre { diff --git a/crates/vm/src/stdlib/stat.rs b/crates/vm/src/stdlib/stat.rs index 90f045fcaf0..44b55628d6f 100644 --- a/crates/vm/src/stdlib/stat.rs +++ b/crates/vm/src/stdlib/stat.rs @@ -1,7 +1,7 @@ -use crate::{PyRef, VirtualMachine, builtins::PyModule}; +pub(crate) use _stat::module_def; #[pymodule] -mod stat { +mod _stat { // Use libc::mode_t for Mode to match the system's definition #[cfg(unix)] type Mode = libc::mode_t; @@ -522,7 +522,3 @@ mod stat { result } } - -pub fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - stat::make_module(vm) -} diff --git a/crates/vm/src/stdlib/string.rs b/crates/vm/src/stdlib/string.rs index a9911f3d45f..c620f2a40e2 100644 --- a/crates/vm/src/stdlib/string.rs +++ b/crates/vm/src/stdlib/string.rs @@ -1,7 +1,7 @@ /* String builtin module */ -pub(crate) use _string::make_module; +pub(crate) use _string::module_def; #[pymodule] mod _string { diff --git a/crates/vm/src/stdlib/symtable.rs b/crates/vm/src/stdlib/symtable.rs index 51c5c8e47ea..881c5519843 100644 --- a/crates/vm/src/stdlib/symtable.rs +++ b/crates/vm/src/stdlib/symtable.rs @@ -1,7 +1,7 @@ -pub(crate) use symtable::make_module; +pub(crate) use _symtable::module_def; #[pymodule] -mod symtable { +mod _symtable { use crate::{ PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, builtins::{PyDictRef, PyStrRef}, diff --git a/crates/vm/src/stdlib/sys.rs b/crates/vm/src/stdlib/sys.rs index 8295da9486d..9c0f84650bc 100644 --- a/crates/vm/src/stdlib/sys.rs +++ b/crates/vm/src/stdlib/sys.rs @@ -1,8 +1,30 @@ use crate::{Py, PyResult, VirtualMachine, builtins::PyModule, convert::ToPyObject}; -pub(crate) use sys::{ - __module_def, DOC, MAXSIZE, RUST_MULTIARCH, UnraisableHookArgsData, multiarch, -}; +pub(crate) use sys::{DOC, MAXSIZE, RUST_MULTIARCH, UnraisableHookArgsData, module_def, multiarch}; + +#[pymodule(name = "_jit")] +mod sys_jit { + /// Return True if the current Python executable supports JIT compilation, + /// and False otherwise. + #[pyfunction] + const fn is_available() -> bool { + false // RustPython has no JIT + } + + /// Return True if JIT compilation is enabled for the current Python process, + /// and False otherwise. + #[pyfunction] + const fn is_enabled() -> bool { + false // RustPython has no JIT + } + + /// Return True if the topmost Python frame is currently executing JIT code, + /// and False otherwise. + #[pyfunction] + const fn is_active() -> bool { + false // RustPython has no JIT + } +} #[pymodule] mod sys { @@ -52,8 +74,11 @@ mod sys { #[pyattr(name = "_rustpython_debugbuild")] const RUSTPYTHON_DEBUGBUILD: bool = cfg!(debug_assertions); + #[cfg(not(windows))] #[pyattr(name = "abiflags")] - pub(crate) const ABIFLAGS: &str = "t"; // 't' for free-threaded (no GIL) + const ABIFLAGS_ATTR: &str = "t"; // 't' for free-threaded (no GIL) + // Internal constant used for sysconfigdata_name + pub const ABIFLAGS: &str = "t"; #[pyattr(name = "api_version")] const API_VERSION: u32 = 0x0; // what C api? #[pyattr(name = "copyright")] @@ -69,7 +94,7 @@ mod sys { #[pyattr(name = "maxunicode")] const MAXUNICODE: u32 = core::char::MAX as u32; #[pyattr(name = "platform")] - pub(crate) const PLATFORM: &str = { + pub const PLATFORM: &str = { cfg_if::cfg_if! { if #[cfg(target_os = "linux")] { "linux" @@ -121,6 +146,10 @@ mod sys { fn platlibdir(_vm: &VirtualMachine) -> &'static str { option_env!("RUSTPYTHON_PLATLIBDIR").unwrap_or("lib") } + #[pyattr] + fn _stdlib_dir(vm: &VirtualMachine) -> PyObjectRef { + vm.state.config.paths.stdlib_dir.clone().to_pyobject(vm) + } // alphabetical order with segments of pyattr and others @@ -137,9 +166,11 @@ mod sys { #[pyattr] fn builtin_module_names(vm: &VirtualMachine) -> PyTupleRef { - let mut module_names: Vec<_> = vm.state.module_inits.keys().cloned().collect(); - module_names.push("sys".into()); - module_names.push("builtins".into()); + let mut module_names: Vec<String> = + vm.state.module_defs.keys().map(|&s| s.to_owned()).collect(); + module_names.push("sys".to_owned()); + module_names.push("builtins".to_owned()); + module_names.sort(); vm.ctx.new_tuple( module_names @@ -503,6 +534,7 @@ mod sys { "_multiarch" => ctx.new_str(multiarch()), "version" => version_info(vm), "hexversion" => ctx.new_int(version::VERSION_HEX), + "supports_isolated_interpreters" => ctx.new_bool(false), }) } @@ -604,6 +636,12 @@ mod sys { false // RustPython has no GIL (like free-threaded Python) } + /// Return True if remote debugging is enabled, False otherwise. + #[pyfunction] + const fn is_remote_debug_enabled() -> bool { + false // RustPython does not support remote debugging + } + #[pyfunction] fn exit(code: OptionalArg<PyObjectRef>, vm: &VirtualMachine) -> PyResult { let code = code.unwrap_or_none(vm); @@ -815,6 +853,32 @@ mod sys { } } + /// Return a dictionary mapping each thread's identifier to the topmost stack frame + /// currently active in that thread at the time the function is called. + #[cfg(feature = "threading")] + #[pyfunction] + fn _current_frames(vm: &VirtualMachine) -> PyResult<PyDictRef> { + use crate::AsObject; + use crate::stdlib::thread::get_all_current_frames; + + let frames = get_all_current_frames(vm); + let dict = vm.ctx.new_dict(); + + for (thread_id, frame) in frames { + let key = vm.ctx.new_int(thread_id); + dict.set_item(key.as_object(), frame.into(), vm)?; + } + + Ok(dict) + } + + /// Stub for non-threading builds - returns empty dict + #[cfg(not(feature = "threading"))] + #[pyfunction] + fn _current_frames(vm: &VirtualMachine) -> PyResult<PyDictRef> { + Ok(vm.ctx.new_dict()) + } + #[pyfunction] fn gettrace(vm: &VirtualMachine) -> PyObjectRef { vm.trace_func.borrow().clone() @@ -1107,6 +1171,22 @@ mod sys { update_use_tracing(vm); } + #[pyfunction] + fn _settraceallthreads(tracefunc: PyObjectRef, vm: &VirtualMachine) { + let func = (!vm.is_none(&tracefunc)).then(|| tracefunc.clone()); + *vm.state.global_trace_func.lock() = func; + vm.trace_func.replace(tracefunc); + update_use_tracing(vm); + } + + #[pyfunction] + fn _setprofileallthreads(profilefunc: PyObjectRef, vm: &VirtualMachine) { + let func = (!vm.is_none(&profilefunc)).then(|| profilefunc.clone()); + *vm.state.global_profile_func.lock() = func; + vm.profile_func.replace(profilefunc); + update_use_tracing(vm); + } + #[cfg(feature = "threading")] #[pyattr] fn thread_info(vm: &VirtualMachine) -> PyTupleRef { @@ -1139,6 +1219,32 @@ mod sys { crate::vm::thread::COROUTINE_ORIGIN_TRACKING_DEPTH.get() as i32 } + #[pyfunction] + fn _clear_type_descriptors(type_obj: PyTypeRef, vm: &VirtualMachine) -> PyResult<()> { + use crate::types::PyTypeFlags; + + // Check if type is immutable + if type_obj.slots.flags.has_feature(PyTypeFlags::IMMUTABLETYPE) { + return Err(vm.new_type_error("argument is immutable".to_owned())); + } + + let mut attributes = type_obj.attributes.write(); + + // Remove __dict__ descriptor if present + attributes.swap_remove(identifier!(vm, __dict__)); + + // Remove __weakref__ descriptor if present + attributes.swap_remove(identifier!(vm, __weakref__)); + + drop(attributes); + + // Update slots to notify subclasses and recalculate cached values + type_obj.update_slot::<true>(identifier!(vm, __dict__), &vm.ctx); + type_obj.update_slot::<true>(identifier!(vm, __weakref__), &vm.ctx); + + Ok(()) + } + #[pyfunction] fn getswitchinterval(vm: &VirtualMachine) -> f64 { // Return the stored switch interval @@ -1187,10 +1293,10 @@ mod sys { } if let Some(finalizer) = args.finalizer.into_option() { - crate::vm::thread::ASYNC_GEN_FINALIZER.set(finalizer); + *vm.async_gen_finalizer.borrow_mut() = finalizer; } if let Some(firstiter) = args.firstiter.into_option() { - crate::vm::thread::ASYNC_GEN_FIRSTITER.set(firstiter); + *vm.async_gen_firstiter.borrow_mut() = firstiter; } Ok(()) @@ -1212,12 +1318,8 @@ mod sys { #[pyfunction] fn get_asyncgen_hooks(vm: &VirtualMachine) -> AsyncgenHooksData { AsyncgenHooksData { - firstiter: crate::vm::thread::ASYNC_GEN_FIRSTITER - .with_borrow(Clone::clone) - .to_pyobject(vm), - finalizer: crate::vm::thread::ASYNC_GEN_FINALIZER - .with_borrow(Clone::clone) - .to_pyobject(vm), + firstiter: vm.async_gen_firstiter.borrow().clone().to_pyobject(vm), + finalizer: vm.async_gen_finalizer.borrow().clone().to_pyobject(vm), } } @@ -1263,6 +1365,10 @@ mod sys { safe_path: bool, /// -X warn_default_encoding, PYTHONWARNDEFAULTENCODING warn_default_encoding: u8, + /// -X thread_inherit_context, whether new threads inherit context from parent + thread_inherit_context: bool, + /// -X context_aware_warnings, whether warnings are context aware + context_aware_warnings: bool, } impl FlagsData { @@ -1286,6 +1392,8 @@ mod sys { int_max_str_digits: settings.int_max_str_digits, safe_path: settings.safe_path, warn_default_encoding: settings.warn_default_encoding as u8, + thread_inherit_context: settings.thread_inherit_context, + context_aware_warnings: settings.context_aware_warnings, } } } @@ -1501,7 +1609,8 @@ mod sys { } pub(crate) fn init_module(vm: &VirtualMachine, module: &Py<PyModule>, builtins: &Py<PyModule>) { - sys::extend_module(vm, module).unwrap(); + module.__init_methods(vm).unwrap(); + sys::module_exec(vm, module).unwrap(); let modules = vm.ctx.new_dict(); modules @@ -1510,9 +1619,15 @@ pub(crate) fn init_module(vm: &VirtualMachine, module: &Py<PyModule>, builtins: modules .set_item("builtins", builtins.to_owned().into(), vm) .unwrap(); + + // Create sys._jit submodule + let jit_def = sys_jit::module_def(&vm.ctx); + let jit_module = jit_def.create_module(vm).unwrap(); + extend_module!(vm, module, { "__doc__" => sys::DOC.to_owned().to_pyobject(vm), "modules" => modules, + "_jit" => jit_module, }); } diff --git a/crates/vm/src/stdlib/sysconfig.rs b/crates/vm/src/stdlib/sysconfig.rs index df5b7100a90..724e1bcf979 100644 --- a/crates/vm/src/stdlib/sysconfig.rs +++ b/crates/vm/src/stdlib/sysconfig.rs @@ -1,7 +1,7 @@ -pub(crate) use sysconfig::make_module; +pub(crate) use _sysconfig::module_def; -#[pymodule(name = "_sysconfig")] -pub(crate) mod sysconfig { +#[pymodule] +pub(crate) mod _sysconfig { use crate::{VirtualMachine, builtins::PyDictRef, convert::ToPyObject}; #[pyfunction] diff --git a/crates/vm/src/stdlib/sysconfigdata.rs b/crates/vm/src/stdlib/sysconfigdata.rs index 99228024d2d..a10745a8cad 100644 --- a/crates/vm/src/stdlib/sysconfigdata.rs +++ b/crates/vm/src/stdlib/sysconfigdata.rs @@ -1,11 +1,29 @@ // spell-checker: words LDSHARED ARFLAGS CPPFLAGS CCSHARED BASECFLAGS BLDSHARED -pub(crate) use _sysconfigdata::make_module; +pub(crate) use _sysconfigdata::module_def; #[pymodule] -pub(crate) mod _sysconfigdata { - use crate::stdlib::sys::{RUST_MULTIARCH, multiarch}; - use crate::{VirtualMachine, builtins::PyDictRef, convert::ToPyObject}; +mod _sysconfigdata { + use crate::stdlib::sys::{RUST_MULTIARCH, multiarch, sysconfigdata_name}; + use crate::{ + Py, PyResult, VirtualMachine, + builtins::{PyDictRef, PyModule}, + convert::ToPyObject, + }; + + fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { + // Set build_time_vars attribute + let build_time_vars = build_time_vars(vm); + module.set_attr("build_time_vars", build_time_vars, vm)?; + + // Ensure the module is registered under the platform-specific name + // (import_builtin() already handles this, but double-check for safety) + let sys_modules = vm.sys_module.get_attr("modules", vm)?; + let sysconfigdata_name = sysconfigdata_name(); + sys_modules.set_item(sysconfigdata_name.as_str(), module.to_owned().into(), vm)?; + + Ok(()) + } #[pyattr] fn build_time_vars(vm: &VirtualMachine) -> PyDictRef { diff --git a/crates/vm/src/stdlib/thread.rs b/crates/vm/src/stdlib/thread.rs index b7f00537c26..22457b3f17f 100644 --- a/crates/vm/src/stdlib/thread.rs +++ b/crates/vm/src/stdlib/thread.rs @@ -1,13 +1,18 @@ //! Implementation of the _thread module +#[cfg(unix)] +pub(crate) use _thread::after_fork_child; #[cfg_attr(target_arch = "wasm32", allow(unused_imports))] -pub(crate) use _thread::{RawRMutex, make_module}; +pub(crate) use _thread::{ + CurrentFrameSlot, HandleEntry, RawRMutex, ShutdownEntry, get_all_current_frames, get_ident, + init_main_thread_ident, module_def, +}; #[pymodule] pub(crate) mod _thread { use crate::{ AsObject, Py, PyPayload, PyRef, PyResult, VirtualMachine, - builtins::{PyDictRef, PyStr, PyTupleRef, PyType, PyTypeRef}, - convert::ToPyException, + builtins::{PyDictRef, PyStr, PyStrRef, PyTupleRef, PyType, PyTypeRef}, + frame::FrameRef, function::{ArgCallable, Either, FuncArgs, KwArgs, OptionalArg, PySetterValue}, types::{Constructor, GetAttr, Representable, SetAttr}, }; @@ -19,7 +24,6 @@ pub(crate) mod _thread { lock_api::{RawMutex as RawMutexT, RawMutexTimed, RawReentrantMutex}, }; use std::thread; - use thread_local::ThreadLocal; // PYTHREAD_NAME: show current thread name pub const PYTHREAD_NAME: Option<&str> = { @@ -110,7 +114,8 @@ pub(crate) mod _thread { } #[pyattr(name = "LockType")] - #[pyclass(module = "thread", name = "lock")] + #[pyattr(name = "lock")] + #[pyclass(module = "_thread", name = "lock")] #[derive(PyPayload)] struct Lock { mu: RawMutex, @@ -172,10 +177,10 @@ pub(crate) mod _thread { } impl Constructor for Lock { - type Args = FuncArgs; + type Args = (); - fn py_new(_cls: &Py<PyType>, _args: Self::Args, vm: &VirtualMachine) -> PyResult<Self> { - Err(vm.new_type_error("cannot create '_thread.lock' instances")) + fn py_new(_cls: &Py<PyType>, _args: Self::Args, _vm: &VirtualMachine) -> PyResult<Self> { + Ok(Self { mu: RawMutex::INIT }) } } @@ -188,7 +193,7 @@ pub(crate) mod _thread { pub type RawRMutex = RawReentrantMutex<RawMutex, RawThreadId>; #[pyattr] - #[pyclass(module = "thread", name = "RLock")] + #[pyclass(module = "_thread", name = "RLock")] #[derive(PyPayload)] struct RLock { mu: RawRMutex, @@ -201,7 +206,7 @@ pub(crate) mod _thread { } } - #[pyclass(with(Representable))] + #[pyclass(with(Representable), flags(BASETYPE))] impl RLock { #[pyslot] fn slot_new(cls: PyTypeRef, _args: FuncArgs, vm: &VirtualMachine) -> PyResult { @@ -249,13 +254,19 @@ pub(crate) mod _thread { } self.count.store(0, core::sync::atomic::Ordering::Relaxed); let new_mut = RawRMutex::INIT; - - let old_mutex: AtomicCell<&RawRMutex> = AtomicCell::new(&self.mu); - old_mutex.swap(&new_mut); + unsafe { + let old_mutex: &AtomicCell<RawRMutex> = core::mem::transmute(&self.mu); + old_mutex.swap(new_mut); + } Ok(()) } + #[pymethod] + fn locked(&self) -> bool { + self.mu.is_locked() + } + #[pymethod] fn _is_owned(&self) -> bool { self.mu.is_owned_by_current_thread() @@ -283,12 +294,69 @@ pub(crate) mod _thread { } } + /// Get thread identity - uses pthread_self() on Unix for fork compatibility + #[pyfunction] + pub fn get_ident() -> u64 { + current_thread_id() + } + + /// Set the name of the current thread #[pyfunction] - fn get_ident() -> u64 { - thread_to_id(&thread::current()) + fn set_name(name: PyStrRef) { + #[cfg(target_os = "linux")] + { + use std::ffi::CString; + if let Ok(c_name) = CString::new(name.as_str()) { + // pthread_setname_np on Linux has a 16-byte limit including null terminator + // TODO: Potential UTF-8 boundary issue when truncating thread name on Linux. + // https://github.com/RustPython/RustPython/pull/6726/changes#r2689379171 + let truncated = if c_name.as_bytes().len() > 15 { + CString::new(&c_name.as_bytes()[..15]).unwrap_or(c_name) + } else { + c_name + }; + unsafe { + libc::pthread_setname_np(libc::pthread_self(), truncated.as_ptr()); + } + } + } + #[cfg(target_os = "macos")] + { + use std::ffi::CString; + if let Ok(c_name) = CString::new(name.as_str()) { + unsafe { + libc::pthread_setname_np(c_name.as_ptr()); + } + } + } + #[cfg(windows)] + { + // Windows doesn't have a simple pthread_setname_np equivalent + // SetThreadDescription requires Windows 10+ + let _ = name; + } + #[cfg(not(any(target_os = "linux", target_os = "macos", windows)))] + { + let _ = name; + } + } + + /// Get OS-level thread ID (pthread_self on Unix) + /// This is important for fork compatibility - the ID must remain stable after fork + #[cfg(unix)] + fn current_thread_id() -> u64 { + // pthread_self() like CPython for fork compatibility + unsafe { libc::pthread_self() as u64 } + } + + #[cfg(not(unix))] + fn current_thread_id() -> u64 { + thread_to_rust_id(&thread::current()) } - fn thread_to_id(t: &thread::Thread) -> u64 { + /// Convert Rust thread to ID (used for non-unix platforms) + #[cfg(not(unix))] + fn thread_to_rust_id(t: &thread::Thread) -> u64 { use core::hash::{Hash, Hasher}; struct U64Hash { v: Option<u64>, @@ -304,13 +372,25 @@ pub(crate) mod _thread { self.v.expect("should have written a u64") } } - // TODO: use id.as_u64() once it's stable, until then, ThreadId is just a wrapper - // around NonZeroU64, so this should work (?) let mut h = U64Hash { v: None }; t.id().hash(&mut h); h.finish() } + /// Get thread ID for a given thread handle (used by start_new_thread) + fn thread_to_id(handle: &thread::JoinHandle<()>) -> u64 { + #[cfg(unix)] + { + // On Unix, use pthread ID from the handle + use std::os::unix::thread::JoinHandleExt; + handle.as_pthread_t() as u64 + } + #[cfg(not(unix))] + { + thread_to_rust_id(handle.thread()) + } + } + #[pyfunction] const fn allocate_lock() -> Lock { Lock { mu: RawMutex::INIT } @@ -341,14 +421,14 @@ pub(crate) mod _thread { vm.new_thread() .make_spawn_func(move |vm| run_thread(func, args, vm)), ) - .map(|handle| { - vm.state.thread_count.fetch_add(1); - thread_to_id(handle.thread()) - }) - .map_err(|err| err.to_pyexception(vm)) + .map(|handle| thread_to_id(&handle)) + .map_err(|err| vm.new_runtime_error(format!("can't start new thread: {err}"))) } fn run_thread(func: ArgCallable, args: FuncArgs, vm: &VirtualMachine) { + // Increment thread count when thread actually starts executing + vm.state.thread_count.fetch_add(1); + match func.invoke(args, vm) { Ok(_obj) => {} Err(e) if e.fast_isinstance(vm.ctx.exceptions.system_exit) => {} @@ -365,9 +445,24 @@ pub(crate) mod _thread { unsafe { lock.mu.unlock() }; } } + // Clean up thread-local storage while VM context is still active + // This ensures __del__ methods are called properly + cleanup_thread_local_data(); + // Clean up frame tracking + crate::vm::thread::cleanup_current_thread_frames(vm); vm.state.thread_count.fetch_sub(1); } + /// Clean up thread-local data for the current thread. + /// This triggers __del__ on objects stored in thread-local variables. + fn cleanup_thread_local_data() { + // Take all guards - this will trigger LocalGuard::drop for each, + // which removes the thread's dict from each Local instance + LOCAL_GUARDS.with(|guards| { + guards.borrow_mut().clear(); + }); + } + #[cfg(not(target_arch = "wasm32"))] #[pyfunction] fn interrupt_main(signum: OptionalArg<i32>, vm: &VirtualMachine) -> PyResult<()> { @@ -400,6 +495,107 @@ pub(crate) mod _thread { vm.state.thread_count.load() } + #[pyfunction] + fn daemon_threads_allowed() -> bool { + // RustPython always allows daemon threads + true + } + + // Registry for non-daemon threads that need to be joined at shutdown + pub type ShutdownEntry = ( + std::sync::Weak<parking_lot::Mutex<ThreadHandleInner>>, + std::sync::Weak<(parking_lot::Mutex<bool>, parking_lot::Condvar)>, + ); + + #[pyfunction] + fn _shutdown(vm: &VirtualMachine) { + // Wait for all non-daemon threads to finish + let current_ident = get_ident(); + + loop { + // Find a thread that's not finished and not the current thread + let handle_to_join = { + let mut handles = vm.state.shutdown_handles.lock(); + // Clean up finished entries + handles.retain(|(inner_weak, _): &ShutdownEntry| { + inner_weak.upgrade().is_some_and(|inner| { + let guard = inner.lock(); + guard.state != ThreadHandleState::Done && guard.ident != current_ident + }) + }); + + // Find first unfinished handle + handles + .iter() + .find_map(|(inner_weak, done_event_weak): &ShutdownEntry| { + let inner = inner_weak.upgrade()?; + let done_event = done_event_weak.upgrade()?; + let guard = inner.lock(); + if guard.state != ThreadHandleState::Done && guard.ident != current_ident { + Some((inner.clone(), done_event.clone())) + } else { + None + } + }) + }; + + match handle_to_join { + Some((_, done_event)) => { + // Wait for this thread to finish (infinite timeout) + // Only check done flag to avoid lock ordering issues + // (done_event lock vs inner lock) + let (lock, cvar) = &*done_event; + let mut done = lock.lock(); + while !*done { + cvar.wait(&mut done); + } + } + None => break, // No more threads to wait on + } + } + } + + /// Add a non-daemon thread handle to the shutdown registry + fn add_to_shutdown_handles( + vm: &VirtualMachine, + inner: &std::sync::Arc<parking_lot::Mutex<ThreadHandleInner>>, + done_event: &std::sync::Arc<(parking_lot::Mutex<bool>, parking_lot::Condvar)>, + ) { + let mut handles = vm.state.shutdown_handles.lock(); + handles.push(( + std::sync::Arc::downgrade(inner), + std::sync::Arc::downgrade(done_event), + )); + } + + #[pyfunction] + fn _make_thread_handle(ident: u64, vm: &VirtualMachine) -> PyRef<ThreadHandle> { + let handle = ThreadHandle::new(vm); + { + let mut inner = handle.inner.lock(); + inner.ident = ident; + inner.state = ThreadHandleState::Running; + } + handle.into_ref(&vm.ctx) + } + + #[pyfunction] + fn _get_main_thread_ident(vm: &VirtualMachine) -> u64 { + vm.state.main_thread_ident.load() + } + + #[pyfunction] + fn _is_main_interpreter() -> bool { + // RustPython only has one interpreter + true + } + + /// Initialize the main thread ident. Should be called once at interpreter startup. + pub fn init_main_thread_ident(vm: &VirtualMachine) { + let ident = get_ident(); + vm.state.main_thread_ident.store(ident); + } + /// ExceptHookArgs - simple class to hold exception hook arguments /// This allows threading.py to import _excepthook and _ExceptHookArgs from _thread #[pyattr] @@ -532,23 +728,92 @@ pub(crate) mod _thread { Ok(()) } + // Thread-local storage for cleanup guards + // When a thread terminates, the guard is dropped, which triggers cleanup + thread_local! { + static LOCAL_GUARDS: std::cell::RefCell<Vec<LocalGuard>> = const { std::cell::RefCell::new(Vec::new()) }; + } + + // Guard that removes thread-local data when dropped + struct LocalGuard { + local: std::sync::Weak<LocalData>, + thread_id: std::thread::ThreadId, + } + + impl Drop for LocalGuard { + fn drop(&mut self) { + if let Some(local_data) = self.local.upgrade() { + // Remove from map while holding the lock, but drop the value + // outside the lock to prevent deadlock if __del__ accesses _local + let removed = local_data.data.lock().remove(&self.thread_id); + drop(removed); + } + } + } + + // Shared data structure for Local + struct LocalData { + data: parking_lot::Mutex<std::collections::HashMap<std::thread::ThreadId, PyDictRef>>, + } + + impl std::fmt::Debug for LocalData { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("LocalData").finish_non_exhaustive() + } + } + #[pyattr] - #[pyclass(module = "thread", name = "_local")] + #[pyclass(module = "_thread", name = "_local")] #[derive(Debug, PyPayload)] struct Local { - data: ThreadLocal<PyDictRef>, + inner: std::sync::Arc<LocalData>, } #[pyclass(with(GetAttr, SetAttr), flags(BASETYPE))] impl Local { fn l_dict(&self, vm: &VirtualMachine) -> PyDictRef { - self.data.get_or(|| vm.ctx.new_dict()).clone() + let thread_id = std::thread::current().id(); + + // Fast path: check if dict exists under lock + if let Some(dict) = self.inner.data.lock().get(&thread_id).cloned() { + return dict; + } + + // Slow path: allocate dict outside lock to reduce lock hold time + let new_dict = vm.ctx.new_dict(); + + // Insert with double-check to handle races + let mut data = self.inner.data.lock(); + use std::collections::hash_map::Entry; + let (dict, need_guard) = match data.entry(thread_id) { + Entry::Occupied(e) => (e.get().clone(), false), + Entry::Vacant(e) => { + e.insert(new_dict.clone()); + (new_dict, true) + } + }; + drop(data); // Release lock before TLS access + + // Register cleanup guard only if we inserted a new entry + if need_guard { + let guard = LocalGuard { + local: std::sync::Arc::downgrade(&self.inner), + thread_id, + }; + LOCAL_GUARDS.with(|guards| { + guards.borrow_mut().push(guard); + }); + } + + dict } #[pyslot] fn slot_new(cls: PyTypeRef, _args: FuncArgs, vm: &VirtualMachine) -> PyResult { Self { - data: ThreadLocal::new(), + inner: std::sync::Arc::new(LocalData { + data: parking_lot::Mutex::new(std::collections::HashMap::new()), + }), } .into_ref_with_type(vm, cls) .map(Into::into) @@ -597,4 +862,401 @@ pub(crate) mod _thread { } } } + + // Registry of all ThreadHandles for fork cleanup + // Stores weak references so handles can be garbage collected normally + pub type HandleEntry = ( + std::sync::Weak<parking_lot::Mutex<ThreadHandleInner>>, + std::sync::Weak<(parking_lot::Mutex<bool>, parking_lot::Condvar)>, + ); + + // Re-export type from vm::thread for PyGlobalState + pub use crate::vm::thread::CurrentFrameSlot; + + /// Get all threads' current frames. Used by sys._current_frames(). + pub fn get_all_current_frames(vm: &VirtualMachine) -> Vec<(u64, FrameRef)> { + let registry = vm.state.thread_frames.lock(); + registry + .iter() + .filter_map(|(id, slot)| slot.lock().clone().map(|f| (*id, f))) + .collect() + } + + /// Called after fork() in child process to mark all other threads as done. + /// This prevents join() from hanging on threads that don't exist in the child. + #[cfg(unix)] + pub fn after_fork_child(vm: &VirtualMachine) { + let current_ident = get_ident(); + + // Update main thread ident - after fork, the current thread becomes the main thread + vm.state.main_thread_ident.store(current_ident); + + // Reinitialize frame slot for current thread + crate::vm::thread::reinit_frame_slot_after_fork(vm); + + // Clean up thread handles if we can acquire the lock. + // Use try_lock because the mutex might have been held during fork. + // If we can't acquire it, just skip - the child process will work + // correctly with new handles it creates. + if let Some(mut handles) = vm.state.thread_handles.try_lock() { + // Clean up dead weak refs and mark non-current threads as done + handles.retain(|(inner_weak, done_event_weak): &HandleEntry| { + let Some(inner) = inner_weak.upgrade() else { + return false; // Remove dead entries + }; + let Some(done_event) = done_event_weak.upgrade() else { + return false; + }; + + // Try to lock the inner state - skip if we can't + let Some(mut inner_guard) = inner.try_lock() else { + return false; + }; + + // Skip current thread and not-started threads + if inner_guard.ident == current_ident { + return true; + } + if inner_guard.state == ThreadHandleState::NotStarted { + return true; + } + + // Mark as done and notify waiters + inner_guard.state = ThreadHandleState::Done; + inner_guard.join_handle = None; // Can't join OS thread from child + drop(inner_guard); + + // Try to notify waiters - skip if we can't acquire the lock + let (lock, cvar) = &*done_event; + if let Some(mut done) = lock.try_lock() { + *done = true; + cvar.notify_all(); + } + + true + }); + } + + // Clean up shutdown_handles as well. + // This is critical to prevent _shutdown() from waiting on threads + // that don't exist in the child process after fork. + if let Some(mut handles) = vm.state.shutdown_handles.try_lock() { + // Mark all non-current threads as done in shutdown_handles + handles.retain(|(inner_weak, done_event_weak): &ShutdownEntry| { + let Some(inner) = inner_weak.upgrade() else { + return false; // Remove dead entries + }; + let Some(done_event) = done_event_weak.upgrade() else { + return false; + }; + + // Try to lock the inner state - skip if we can't + let Some(mut inner_guard) = inner.try_lock() else { + return false; + }; + + // Skip current thread + if inner_guard.ident == current_ident { + return true; + } + + // Keep handles for threads that have not been started yet. + // They are safe to start in the child process. + if inner_guard.state == ThreadHandleState::NotStarted { + return true; + } + + // Mark as done so _shutdown() won't wait on it + inner_guard.state = ThreadHandleState::Done; + drop(inner_guard); + + // Notify waiters + let (lock, cvar) = &*done_event; + if let Some(mut done) = lock.try_lock() { + *done = true; + cvar.notify_all(); + } + + false // Remove from shutdown_handles - these threads don't exist in child + }); + } + } + + // Thread handle state enum + #[derive(Debug, Clone, Copy, PartialEq, Eq)] + pub enum ThreadHandleState { + NotStarted, + Starting, + Running, + Done, + } + + // Internal shared state for thread handle + pub struct ThreadHandleInner { + pub state: ThreadHandleState, + pub ident: u64, + pub join_handle: Option<thread::JoinHandle<()>>, + pub joining: bool, // True if a thread is currently joining + pub joined: bool, // Track if join has completed + } + + impl fmt::Debug for ThreadHandleInner { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("ThreadHandleInner") + .field("state", &self.state) + .field("ident", &self.ident) + .field("join_handle", &self.join_handle.is_some()) + .field("joining", &self.joining) + .field("joined", &self.joined) + .finish() + } + } + + /// _ThreadHandle - handle for joinable threads + #[pyattr] + #[pyclass(module = "_thread", name = "_ThreadHandle")] + #[derive(Debug, PyPayload)] + struct ThreadHandle { + inner: std::sync::Arc<parking_lot::Mutex<ThreadHandleInner>>, + // Event to signal thread completion (for timed join support) + done_event: std::sync::Arc<(parking_lot::Mutex<bool>, parking_lot::Condvar)>, + } + + #[pyclass] + impl ThreadHandle { + fn new(vm: &VirtualMachine) -> Self { + let inner = std::sync::Arc::new(parking_lot::Mutex::new(ThreadHandleInner { + state: ThreadHandleState::NotStarted, + ident: 0, + join_handle: None, + joining: false, + joined: false, + })); + let done_event = + std::sync::Arc::new((parking_lot::Mutex::new(false), parking_lot::Condvar::new())); + + // Register in global registry for fork cleanup + vm.state.thread_handles.lock().push(( + std::sync::Arc::downgrade(&inner), + std::sync::Arc::downgrade(&done_event), + )); + + Self { inner, done_event } + } + + #[pygetset] + fn ident(&self) -> u64 { + self.inner.lock().ident + } + + #[pymethod] + fn is_done(&self) -> bool { + self.inner.lock().state == ThreadHandleState::Done + } + + #[pymethod] + fn _set_done(&self) { + self.inner.lock().state = ThreadHandleState::Done; + // Signal waiting threads that this thread is done + let (lock, cvar) = &*self.done_event; + *lock.lock() = true; + cvar.notify_all(); + } + + #[pymethod] + fn join( + &self, + timeout: OptionalArg<Option<Either<f64, i64>>>, + vm: &VirtualMachine, + ) -> PyResult<()> { + // Convert timeout to Duration (None or negative = infinite wait) + let timeout_duration = match timeout.flatten() { + Some(Either::A(t)) if t >= 0.0 => Some(Duration::from_secs_f64(t)), + Some(Either::B(t)) if t >= 0 => Some(Duration::from_secs(t as u64)), + _ => None, + }; + + // Check for self-join first + { + let inner = self.inner.lock(); + let current_ident = get_ident(); + if inner.ident == current_ident && inner.state == ThreadHandleState::Running { + return Err(vm.new_runtime_error("cannot join current thread".to_owned())); + } + } + + // Wait for thread completion using Condvar (supports timeout) + // Loop to handle spurious wakeups + let (lock, cvar) = &*self.done_event; + let mut done = lock.lock(); + + while !*done { + if let Some(timeout) = timeout_duration { + let result = cvar.wait_for(&mut done, timeout); + if result.timed_out() && !*done { + // Timeout occurred and done is still false + return Ok(()); + } + } else { + // Infinite wait + cvar.wait(&mut done); + } + } + drop(done); + + // Thread is done, now perform cleanup + let join_handle = { + let mut inner = self.inner.lock(); + + // If already joined, return immediately (idempotent) + if inner.joined { + return Ok(()); + } + + // If another thread is already joining, wait for them to finish + if inner.joining { + drop(inner); + // Wait on done_event + let (lock, cvar) = &*self.done_event; + let mut done = lock.lock(); + while !*done { + cvar.wait(&mut done); + } + return Ok(()); + } + + // Mark that we're joining + inner.joining = true; + + // Take the join handle if available + inner.join_handle.take() + }; + + // Perform the actual join outside the lock + if let Some(handle) = join_handle { + // Ignore the result - panics in spawned threads are already handled + let _ = handle.join(); + } + + // Mark as joined and clear joining flag + { + let mut inner = self.inner.lock(); + inner.joined = true; + inner.joining = false; + } + + Ok(()) + } + + #[pyslot] + fn slot_new(cls: PyTypeRef, _args: FuncArgs, vm: &VirtualMachine) -> PyResult { + ThreadHandle::new(vm) + .into_ref_with_type(vm, cls) + .map(Into::into) + } + } + + #[derive(FromArgs)] + struct StartJoinableThreadArgs { + #[pyarg(positional)] + function: ArgCallable, + #[pyarg(any, optional)] + handle: OptionalArg<PyRef<ThreadHandle>>, + #[pyarg(any, default = true)] + daemon: bool, + } + + #[pyfunction] + fn start_joinable_thread( + args: StartJoinableThreadArgs, + vm: &VirtualMachine, + ) -> PyResult<PyRef<ThreadHandle>> { + let handle = match args.handle { + OptionalArg::Present(h) => h, + OptionalArg::Missing => ThreadHandle::new(vm).into_ref(&vm.ctx), + }; + + // Mark as starting + handle.inner.lock().state = ThreadHandleState::Starting; + + // Add non-daemon threads to shutdown registry so _shutdown() will wait for them + if !args.daemon { + add_to_shutdown_handles(vm, &handle.inner, &handle.done_event); + } + + let func = args.function; + let handle_clone = handle.clone(); + let inner_clone = handle.inner.clone(); + let done_event_clone = handle.done_event.clone(); + + let mut thread_builder = thread::Builder::new(); + let stacksize = vm.state.stacksize.load(); + if stacksize != 0 { + thread_builder = thread_builder.stack_size(stacksize); + } + + let join_handle = thread_builder + .spawn(vm.new_thread().make_spawn_func(move |vm| { + // Set ident and mark as running + { + let mut inner = inner_clone.lock(); + inner.ident = get_ident(); + inner.state = ThreadHandleState::Running; + } + + // Ensure cleanup happens even if the function panics + let inner_for_cleanup = inner_clone.clone(); + let done_event_for_cleanup = done_event_clone.clone(); + let vm_state = vm.state.clone(); + scopeguard::defer! { + // Mark as done + inner_for_cleanup.lock().state = ThreadHandleState::Done; + + // Handle sentinels + for lock in SENTINELS.take() { + if lock.mu.is_locked() { + unsafe { lock.mu.unlock() }; + } + } + + // Clean up thread-local data while VM context is still active + cleanup_thread_local_data(); + + // Clean up frame tracking + crate::vm::thread::cleanup_current_thread_frames(vm); + + vm_state.thread_count.fetch_sub(1); + + // Signal waiting threads that this thread is done + // This must be LAST to ensure all cleanup is complete before join() returns + { + let (lock, cvar) = &*done_event_for_cleanup; + *lock.lock() = true; + cvar.notify_all(); + } + } + + // Increment thread count when thread actually starts executing + vm_state.thread_count.fetch_add(1); + + // Run the function + match func.invoke((), vm) { + Ok(_) => {} + Err(e) if e.fast_isinstance(vm.ctx.exceptions.system_exit) => {} + Err(exc) => { + vm.run_unraisable( + exc, + Some("Exception ignored in thread started by".to_owned()), + func.into(), + ); + } + } + })) + .map_err(|err| vm.new_runtime_error(format!("can't start new thread: {err}")))?; + + // Store the join handle + handle.inner.lock().join_handle = Some(join_handle); + + Ok(handle_clone) + } } diff --git a/crates/vm/src/stdlib/time.rs b/crates/vm/src/stdlib/time.rs index 97d60ae98a1..ccd80402bf7 100644 --- a/crates/vm/src/stdlib/time.rs +++ b/crates/vm/src/stdlib/time.rs @@ -3,18 +3,10 @@ // See also: // https://docs.python.org/3/library/time.html -use crate::{PyRef, VirtualMachine, builtins::PyModule}; pub use decl::time; -pub(crate) fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - #[cfg(not(target_env = "msvc"))] - #[cfg(not(target_arch = "wasm32"))] - unsafe { - c_tzset() - }; - decl::make_module(vm) -} +pub(crate) use decl::module_def; #[cfg(not(target_env = "msvc"))] #[cfg(not(target_arch = "wasm32"))] @@ -34,7 +26,7 @@ unsafe extern "C" { #[pymodule(name = "time", with(platform))] mod decl { use crate::{ - AsObject, PyObjectRef, PyResult, VirtualMachine, + AsObject, Py, PyObjectRef, PyResult, VirtualMachine, builtins::{PyStrRef, PyTypeRef, PyUtf8StrRef}, function::{Either, FuncArgs, OptionalArg}, types::{PyStructSequence, struct_sequence_new}, @@ -197,6 +189,23 @@ mod decl { // unsafe { super::_tzset() }; // } + #[cfg(not(target_env = "msvc"))] + #[cfg(not(target_arch = "wasm32"))] + #[pyattr] + fn altzone(_vm: &VirtualMachine) -> core::ffi::c_long { + // TODO: RUSTPYTHON; Add support for using the C altzone + unsafe { super::c_timezone - 3600 } + } + + #[cfg(target_env = "msvc")] + #[cfg(not(target_arch = "wasm32"))] + #[pyattr] + fn altzone(_vm: &VirtualMachine) -> i32 { + let info = get_tz_info(); + // https://users.rust-lang.org/t/accessing-tzname-and-similar-constants-in-windows/125771/3 + (info.Bias + info.StandardBias) * 60 - 3600 + } + #[cfg(not(target_env = "msvc"))] #[cfg(not(target_arch = "wasm32"))] #[pyattr] @@ -441,19 +450,6 @@ mod decl { )) } - // same as the get_process_time impl for most unixes - #[cfg(all(target_arch = "wasm32", target_os = "wasi"))] - pub(super) fn get_process_time(vm: &VirtualMachine) -> PyResult<Duration> { - let time: libc::timespec = unsafe { - let mut time = std::mem::MaybeUninit::uninit(); - if libc::clock_gettime(libc::CLOCK_PROCESS_CPUTIME_ID, time.as_mut_ptr()) == -1 { - return Err(vm.new_os_error("Failed to get clock time".to_owned())); - } - time.assume_init() - }; - Ok(Duration::new(time.tv_sec as u64, time.tv_nsec as u32)) - } - #[cfg(not(any( windows, target_os = "macos", @@ -466,7 +462,7 @@ mod decl { target_os = "solaris", target_os = "openbsd", target_os = "redox", - all(target_arch = "wasm32", not(target_os = "unknown")) + all(target_arch = "wasm32", target_os = "emscripten") )))] fn get_process_time(vm: &VirtualMachine) -> PyResult<Duration> { Err(vm.new_not_implemented_error("process time unsupported in this system")) @@ -577,6 +573,20 @@ mod decl { #[allow(unused_imports)] use super::platform::*; + + pub(crate) fn module_exec( + vm: &VirtualMachine, + module: &Py<crate::builtins::PyModule>, + ) -> PyResult<()> { + #[cfg(not(target_env = "msvc"))] + #[cfg(not(target_arch = "wasm32"))] + unsafe { + super::c_tzset() + }; + + __module_exec(vm, module); + Ok(()) + } } #[cfg(unix)] @@ -601,6 +611,7 @@ mod platform { target_os = "netbsd", target_os = "solaris", target_os = "openbsd", + target_os = "wasi", )))] #[pyattr] use libc::CLOCK_PROCESS_CPUTIME_ID; diff --git a/crates/vm/src/stdlib/typevar.rs b/crates/vm/src/stdlib/typevar.rs index 65249bfd075..36f2b170023 100644 --- a/crates/vm/src/stdlib/typevar.rs +++ b/crates/vm/src/stdlib/typevar.rs @@ -1,1038 +1,1054 @@ // spell-checker:ignore typevarobject funcobj -use crate::{ - AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, - builtins::{PyTupleRef, PyType, PyTypeRef, pystr::AsPyStr}, - common::lock::PyMutex, - function::{FuncArgs, IntoFuncArgs, PyComparisonValue}, - protocol::PyNumberMethods, - types::{AsNumber, Comparable, Constructor, Iterable, PyComparisonOp, Representable}, -}; - -pub(crate) fn _call_typing_func_object<'a>( - vm: &VirtualMachine, - func_name: impl AsPyStr<'a>, - args: impl IntoFuncArgs, -) -> PyResult { - let module = vm.import("typing", 0)?; - let func = module.get_attr(func_name.as_pystr(&vm.ctx), vm)?; - func.call(args, vm) -} -fn type_check(arg: PyObjectRef, msg: &str, vm: &VirtualMachine) -> PyResult<PyObjectRef> { - // Calling typing.py here leads to bootstrapping problems - if vm.is_none(&arg) { - return Ok(arg.class().to_owned().into()); - } - let message_str: PyObjectRef = vm.ctx.new_str(msg).into(); - _call_typing_func_object(vm, "_type_check", (arg, message_str)) -} +pub use typevar::*; + +#[pymodule(sub)] +pub(crate) mod typevar { + use crate::{ + AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, + builtins::{PyTuple, PyTupleRef, PyType, PyTypeRef, make_union, pystr::AsPyStr}, + common::lock::PyMutex, + function::{FuncArgs, IntoFuncArgs, PyComparisonValue}, + protocol::PyNumberMethods, + types::{AsNumber, Comparable, Constructor, Iterable, PyComparisonOp, Representable}, + }; -/// Get the module of the caller frame, similar to CPython's caller() function. -/// Returns the module name or None if not found. -/// -/// Note: CPython's implementation (in typevarobject.c) gets the module from the -/// frame's function object using PyFunction_GetModule(f->f_funcobj). However, -/// RustPython's Frame doesn't store a reference to the function object, so we -/// get the module name from the frame's globals dictionary instead. -fn caller(vm: &VirtualMachine) -> Option<PyObjectRef> { - let frame = vm.current_frame()?; - - // In RustPython, we get the module name from frame's globals - // This is similar to CPython's sys._getframe().f_globals.get('__name__') - frame.globals.get_item("__name__", vm).ok() -} + pub(crate) fn _call_typing_func_object<'a>( + vm: &VirtualMachine, + func_name: impl AsPyStr<'a>, + args: impl IntoFuncArgs, + ) -> PyResult { + let module = vm.import("typing", 0)?; + let func = module.get_attr(func_name.as_pystr(&vm.ctx), vm)?; + func.call(args, vm) + } -/// Set __module__ attribute for an object based on the caller's module. -/// This follows CPython's behavior for TypeVar and similar objects. -fn set_module_from_caller(obj: &PyObject, vm: &VirtualMachine) -> PyResult<()> { - // Note: CPython gets module from frame->f_funcobj, but RustPython's Frame - // architecture is different - we use globals['__name__'] instead - if let Some(module_name) = caller(vm) { - // Special handling for certain module names - if let Ok(name_str) = module_name.str(vm) { - let name = name_str.as_str(); - // CPython sets __module__ to None for builtins and <...> modules - // Also set to None for exec contexts (no __name__ in globals means exec) - if name == "builtins" || name.starts_with('<') { - // Don't set __module__ attribute at all (CPython behavior) - // This allows the typing module to handle it - return Ok(()); + fn type_check(arg: PyObjectRef, msg: &str, vm: &VirtualMachine) -> PyResult<PyObjectRef> { + // Calling typing.py here leads to bootstrapping problems + if vm.is_none(&arg) { + return Ok(arg.class().to_owned().into()); + } + let message_str: PyObjectRef = vm.ctx.new_str(msg).into(); + _call_typing_func_object(vm, "_type_check", (arg, message_str)) + } + + /// Get the module of the caller frame, similar to CPython's caller() function. + /// Returns the module name or None if not found. + /// + /// Note: CPython's implementation (in typevarobject.c) gets the module from the + /// frame's function object using PyFunction_GetModule(f->f_funcobj). However, + /// RustPython's Frame doesn't store a reference to the function object, so we + /// get the module name from the frame's globals dictionary instead. + fn caller(vm: &VirtualMachine) -> Option<PyObjectRef> { + let frame = vm.current_frame()?; + + // In RustPython, we get the module name from frame's globals + // This is similar to CPython's sys._getframe().f_globals.get('__name__') + frame.globals.get_item("__name__", vm).ok() + } + + /// Set __module__ attribute for an object based on the caller's module. + /// This follows CPython's behavior for TypeVar and similar objects. + fn set_module_from_caller(obj: &PyObject, vm: &VirtualMachine) -> PyResult<()> { + // Note: CPython gets module from frame->f_funcobj, but RustPython's Frame + // architecture is different - we use globals['__name__'] instead + if let Some(module_name) = caller(vm) { + // Special handling for certain module names + if let Ok(name_str) = module_name.str(vm) { + let name = name_str.as_str(); + // CPython sets __module__ to None for builtins and <...> modules + // Also set to None for exec contexts (no __name__ in globals means exec) + if name == "builtins" || name.starts_with('<') { + // Don't set __module__ attribute at all (CPython behavior) + // This allows the typing module to handle it + return Ok(()); + } } + obj.set_attr("__module__", module_name, vm)?; + } else { + // If no module name is found (e.g., in exec context), set __module__ to None + obj.set_attr("__module__", vm.ctx.none(), vm)?; } - obj.set_attr("__module__", module_name, vm)?; - } else { - // If no module name is found (e.g., in exec context), set __module__ to None - obj.set_attr("__module__", vm.ctx.none(), vm)?; + Ok(()) } - Ok(()) -} -#[pyclass(name = "TypeVar", module = "typing")] -#[derive(Debug, PyPayload)] -#[allow(dead_code)] -pub struct TypeVar { - name: PyObjectRef, // TODO PyStrRef? - bound: parking_lot::Mutex<PyObjectRef>, - evaluate_bound: PyObjectRef, - constraints: parking_lot::Mutex<PyObjectRef>, - evaluate_constraints: PyObjectRef, - default_value: parking_lot::Mutex<PyObjectRef>, - evaluate_default: PyMutex<PyObjectRef>, - covariant: bool, - contravariant: bool, - infer_variance: bool, -} -#[pyclass(flags(HAS_DICT), with(AsNumber, Constructor, Representable))] -impl TypeVar { - #[pymethod] - fn __mro_entries__(&self, _bases: PyObjectRef, vm: &VirtualMachine) -> PyResult { - Err(vm.new_type_error("Cannot subclass an instance of TypeVar")) - } - - #[pygetset] - fn __name__(&self) -> PyObjectRef { - self.name.clone() - } + #[pyattr] + #[pyclass(name = "TypeVar", module = "typing")] + #[derive(Debug, PyPayload)] + #[allow(dead_code)] + pub struct TypeVar { + name: PyObjectRef, // TODO PyStrRef? + bound: parking_lot::Mutex<PyObjectRef>, + evaluate_bound: PyObjectRef, + constraints: parking_lot::Mutex<PyObjectRef>, + evaluate_constraints: PyObjectRef, + default_value: parking_lot::Mutex<PyObjectRef>, + evaluate_default: PyMutex<PyObjectRef>, + covariant: bool, + contravariant: bool, + infer_variance: bool, + } + #[pyclass(flags(HAS_DICT), with(AsNumber, Constructor, Representable))] + impl TypeVar { + #[pymethod] + fn __mro_entries__(&self, _bases: PyObjectRef, vm: &VirtualMachine) -> PyResult { + Err(vm.new_type_error("Cannot subclass an instance of TypeVar")) + } - #[pygetset] - fn __constraints__(&self, vm: &VirtualMachine) -> PyResult { - let mut constraints = self.constraints.lock(); - if !vm.is_none(&constraints) { - return Ok(constraints.clone()); + #[pygetset] + fn __name__(&self) -> PyObjectRef { + self.name.clone() } - let r = if !vm.is_none(&self.evaluate_constraints) { - *constraints = self.evaluate_constraints.call((), vm)?; - constraints.clone() - } else { - vm.ctx.empty_tuple.clone().into() - }; - Ok(r) - } - #[pygetset] - fn __bound__(&self, vm: &VirtualMachine) -> PyResult { - let mut bound = self.bound.lock(); - if !vm.is_none(&bound) { - return Ok(bound.clone()); + #[pygetset] + fn __constraints__(&self, vm: &VirtualMachine) -> PyResult { + let mut constraints = self.constraints.lock(); + if !vm.is_none(&constraints) { + return Ok(constraints.clone()); + } + let r = if !vm.is_none(&self.evaluate_constraints) { + *constraints = self.evaluate_constraints.call((), vm)?; + constraints.clone() + } else { + vm.ctx.empty_tuple.clone().into() + }; + Ok(r) } - let r = if !vm.is_none(&self.evaluate_bound) { - *bound = self.evaluate_bound.call((), vm)?; - bound.clone() - } else { - vm.ctx.none() - }; - Ok(r) - } - #[pygetset] - const fn __covariant__(&self) -> bool { - self.covariant - } + #[pygetset] + fn __bound__(&self, vm: &VirtualMachine) -> PyResult { + let mut bound = self.bound.lock(); + if !vm.is_none(&bound) { + return Ok(bound.clone()); + } + let r = if !vm.is_none(&self.evaluate_bound) { + *bound = self.evaluate_bound.call((), vm)?; + bound.clone() + } else { + vm.ctx.none() + }; + Ok(r) + } - #[pygetset] - const fn __contravariant__(&self) -> bool { - self.contravariant - } + #[pygetset] + const fn __covariant__(&self) -> bool { + self.covariant + } - #[pygetset] - const fn __infer_variance__(&self) -> bool { - self.infer_variance - } + #[pygetset] + const fn __contravariant__(&self) -> bool { + self.contravariant + } - #[pygetset] - fn __default__(&self, vm: &VirtualMachine) -> PyResult { - let mut default_value = self.default_value.lock(); - // Check if default_value is NoDefault (not just None) - if !default_value.is(&vm.ctx.typing_no_default) { - return Ok(default_value.clone()); - } - let evaluate_default = self.evaluate_default.lock(); - if !vm.is_none(&evaluate_default) { - *default_value = evaluate_default.call((), vm)?; - Ok(default_value.clone()) - } else { - // Return NoDefault singleton - Ok(vm.ctx.typing_no_default.clone().into()) + #[pygetset] + const fn __infer_variance__(&self) -> bool { + self.infer_variance } - } - #[pymethod] - fn __typing_subst__( - zelf: crate::PyRef<Self>, - arg: PyObjectRef, - vm: &VirtualMachine, - ) -> PyResult { - let self_obj: PyObjectRef = zelf.into(); - _call_typing_func_object(vm, "_typevar_subst", (self_obj, arg)) - } + #[pygetset] + fn __default__(&self, vm: &VirtualMachine) -> PyResult { + let mut default_value = self.default_value.lock(); + // Check if default_value is NoDefault (not just None) + if !default_value.is(&vm.ctx.typing_no_default) { + return Ok(default_value.clone()); + } + let evaluate_default = self.evaluate_default.lock(); + if !vm.is_none(&evaluate_default) { + *default_value = evaluate_default.call((), vm)?; + Ok(default_value.clone()) + } else { + // Return NoDefault singleton + Ok(vm.ctx.typing_no_default.clone().into()) + } + } - #[pymethod] - fn __reduce__(&self) -> PyObjectRef { - self.name.clone() - } + #[pymethod] + fn __typing_subst__( + zelf: crate::PyRef<Self>, + arg: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult { + let self_obj: PyObjectRef = zelf.into(); + _call_typing_func_object(vm, "_typevar_subst", (self_obj, arg)) + } - #[pymethod] - fn has_default(&self, vm: &VirtualMachine) -> bool { - if !vm.is_none(&self.evaluate_default.lock()) { - return true; + #[pymethod] + fn __reduce__(&self) -> PyObjectRef { + self.name.clone() } - let default_value = self.default_value.lock(); - // Check if default_value is not NoDefault - !default_value.is(&vm.ctx.typing_no_default) - } - #[pymethod] - fn __typing_prepare_subst__( - zelf: crate::PyRef<Self>, - alias: PyObjectRef, - args: PyObjectRef, - vm: &VirtualMachine, - ) -> PyResult { - // Convert args to tuple if needed - let args_tuple = if let Ok(tuple) = args.try_to_ref::<rustpython_vm::builtins::PyTuple>(vm) - { - tuple - } else { - return Ok(args); - }; + #[pymethod] + fn has_default(&self, vm: &VirtualMachine) -> bool { + if !vm.is_none(&self.evaluate_default.lock()) { + return true; + } + let default_value = self.default_value.lock(); + // Check if default_value is not NoDefault + !default_value.is(&vm.ctx.typing_no_default) + } - // Get alias.__parameters__ - let parameters = alias.get_attr(identifier!(vm, __parameters__), vm)?; - let params_tuple: PyTupleRef = parameters.try_into_value(vm)?; - - // Find our index in parameters - let self_obj: PyObjectRef = zelf.to_owned().into(); - let param_index = params_tuple.iter().position(|p| p.is(&self_obj)); - - if let Some(index) = param_index { - // Check if we have enough arguments - if args_tuple.len() <= index && zelf.has_default(vm) { - // Need to add default value - let mut new_args: Vec<PyObjectRef> = args_tuple.iter().cloned().collect(); - - // Add default value at the correct position - while new_args.len() <= index { - // For the current parameter, add its default - if new_args.len() == index { - let default_val = zelf.__default__(vm)?; - new_args.push(default_val); - } else { - // This shouldn't happen in well-formed code - break; + #[pymethod] + fn __typing_prepare_subst__( + zelf: crate::PyRef<Self>, + alias: PyObjectRef, + args: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult { + // Convert args to tuple if needed + let args_tuple = + if let Ok(tuple) = args.try_to_ref::<rustpython_vm::builtins::PyTuple>(vm) { + tuple + } else { + return Ok(args); + }; + + // Get alias.__parameters__ + let parameters = alias.get_attr(identifier!(vm, __parameters__), vm)?; + let params_tuple: PyTupleRef = parameters.try_into_value(vm)?; + + // Find our index in parameters + let self_obj: PyObjectRef = zelf.to_owned().into(); + let param_index = params_tuple.iter().position(|p| p.is(&self_obj)); + + if let Some(index) = param_index { + // Check if we have enough arguments + if args_tuple.len() <= index && zelf.has_default(vm) { + // Need to add default value + let mut new_args: Vec<PyObjectRef> = args_tuple.iter().cloned().collect(); + + // Add default value at the correct position + while new_args.len() <= index { + // For the current parameter, add its default + if new_args.len() == index { + let default_val = zelf.__default__(vm)?; + new_args.push(default_val); + } else { + // This shouldn't happen in well-formed code + break; + } } - } - return Ok(rustpython_vm::builtins::PyTuple::new_ref(new_args, &vm.ctx).into()); + return Ok(rustpython_vm::builtins::PyTuple::new_ref(new_args, &vm.ctx).into()); + } } - } - // No changes needed - Ok(args) + // No changes needed + Ok(args) + } } -} -impl Representable for TypeVar { - #[inline(always)] - fn repr_str(zelf: &crate::Py<Self>, vm: &VirtualMachine) -> PyResult<String> { - let name = zelf.name.str(vm)?; - let repr = if zelf.covariant { - format!("+{name}") - } else if zelf.contravariant { - format!("-{name}") - } else { - format!("~{name}") - }; - Ok(repr) + impl Representable for TypeVar { + #[inline(always)] + fn repr_str(zelf: &crate::Py<Self>, vm: &VirtualMachine) -> PyResult<String> { + let name = zelf.name.str(vm)?; + let repr = if zelf.covariant { + format!("+{name}") + } else if zelf.contravariant { + format!("-{name}") + } else { + format!("~{name}") + }; + Ok(repr) + } } -} -impl AsNumber for TypeVar { - fn as_number() -> &'static PyNumberMethods { - static AS_NUMBER: PyNumberMethods = PyNumberMethods { - or: Some(|a, b, vm| { - _call_typing_func_object(vm, "_make_union", (a.to_owned(), b.to_owned())) - }), - ..PyNumberMethods::NOT_IMPLEMENTED - }; - &AS_NUMBER + impl AsNumber for TypeVar { + fn as_number() -> &'static PyNumberMethods { + static AS_NUMBER: PyNumberMethods = PyNumberMethods { + or: Some(|a, b, vm| { + let args = PyTuple::new_ref(vec![a.to_owned(), b.to_owned()], &vm.ctx); + make_union(&args, vm) + }), + ..PyNumberMethods::NOT_IMPLEMENTED + }; + &AS_NUMBER + } } -} -impl Constructor for TypeVar { - type Args = FuncArgs; + impl Constructor for TypeVar { + type Args = FuncArgs; - fn slot_new(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult { - let typevar = <Self as Constructor>::py_new(&cls, args, vm)?; - let obj = typevar.into_ref_with_type(vm, cls)?; - let obj_ref: PyObjectRef = obj.into(); - set_module_from_caller(&obj_ref, vm)?; - Ok(obj_ref) - } + fn slot_new(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult { + let typevar = <Self as Constructor>::py_new(&cls, args, vm)?; + let obj = typevar.into_ref_with_type(vm, cls)?; + let obj_ref: PyObjectRef = obj.into(); + set_module_from_caller(&obj_ref, vm)?; + Ok(obj_ref) + } - fn py_new(_cls: &Py<PyType>, args: Self::Args, vm: &VirtualMachine) -> PyResult<Self> { - let mut kwargs = args.kwargs; - // Parse arguments manually - let (name, constraints) = if args.args.is_empty() { - // Check if name is provided as keyword argument - if let Some(name) = kwargs.swap_remove("name") { - (name, vec![]) + fn py_new(_cls: &Py<PyType>, args: Self::Args, vm: &VirtualMachine) -> PyResult<Self> { + let mut kwargs = args.kwargs; + // Parse arguments manually + let (name, constraints) = if args.args.is_empty() { + // Check if name is provided as keyword argument + if let Some(name) = kwargs.swap_remove("name") { + (name, vec![]) + } else { + return Err( + vm.new_type_error("TypeVar() missing required argument: 'name' (pos 1)") + ); + } + } else if args.args.len() == 1 { + (args.args[0].clone(), vec![]) } else { - return Err( - vm.new_type_error("TypeVar() missing required argument: 'name' (pos 1)") - ); + let name = args.args[0].clone(); + let constraints = args.args[1..].to_vec(); + (name, constraints) + }; + + let bound = kwargs.swap_remove("bound"); + let covariant = kwargs + .swap_remove("covariant") + .map(|v| v.try_to_bool(vm)) + .transpose()? + .unwrap_or(false); + let contravariant = kwargs + .swap_remove("contravariant") + .map(|v| v.try_to_bool(vm)) + .transpose()? + .unwrap_or(false); + let infer_variance = kwargs + .swap_remove("infer_variance") + .map(|v| v.try_to_bool(vm)) + .transpose()? + .unwrap_or(false); + let default = kwargs.swap_remove("default"); + + // Check for unexpected keyword arguments + if !kwargs.is_empty() { + let unexpected_keys: Vec<String> = kwargs.keys().map(|s| s.to_string()).collect(); + return Err(vm.new_type_error(format!( + "TypeVar() got unexpected keyword argument(s): {}", + unexpected_keys.join(", ") + ))); } - } else if args.args.len() == 1 { - (args.args[0].clone(), vec![]) - } else { - let name = args.args[0].clone(); - let constraints = args.args[1..].to_vec(); - (name, constraints) - }; - let bound = kwargs.swap_remove("bound"); - let covariant = kwargs - .swap_remove("covariant") - .map(|v| v.try_to_bool(vm)) - .transpose()? - .unwrap_or(false); - let contravariant = kwargs - .swap_remove("contravariant") - .map(|v| v.try_to_bool(vm)) - .transpose()? - .unwrap_or(false); - let infer_variance = kwargs - .swap_remove("infer_variance") - .map(|v| v.try_to_bool(vm)) - .transpose()? - .unwrap_or(false); - let default = kwargs.swap_remove("default"); - - // Check for unexpected keyword arguments - if !kwargs.is_empty() { - let unexpected_keys: Vec<String> = kwargs.keys().map(|s| s.to_string()).collect(); - return Err(vm.new_type_error(format!( - "TypeVar() got unexpected keyword argument(s): {}", - unexpected_keys.join(", ") - ))); - } - - // Check for invalid combinations - if covariant && contravariant { - return Err(vm.new_value_error("Bivariant type variables are not supported.")); - } - - if infer_variance && (covariant || contravariant) { - return Err(vm.new_value_error("Variance cannot be specified with infer_variance")); - } - - // Handle constraints and bound - let (constraints_obj, evaluate_constraints) = if !constraints.is_empty() { - // Check for single constraint - if constraints.len() == 1 { - return Err(vm.new_type_error("A single constraint is not allowed")); + // Check for invalid combinations + if covariant && contravariant { + return Err(vm.new_value_error("Bivariant type variables are not supported.")); } - if bound.is_some() { - return Err(vm.new_type_error("Constraints cannot be used with bound")); - } - let constraints_tuple = vm.ctx.new_tuple(constraints); - (constraints_tuple.clone().into(), constraints_tuple.into()) - } else { - (vm.ctx.none(), vm.ctx.none()) - }; - // Handle bound - let (bound_obj, evaluate_bound) = if let Some(bound) = bound { - if vm.is_none(&bound) { - (vm.ctx.none(), vm.ctx.none()) - } else { - // Type check the bound - let bound = type_check(bound, "Bound must be a type.", vm)?; - (bound, vm.ctx.none()) + if infer_variance && (covariant || contravariant) { + return Err(vm.new_value_error("Variance cannot be specified with infer_variance")); } - } else { - (vm.ctx.none(), vm.ctx.none()) - }; - // Handle default value - let (default_value, evaluate_default) = if let Some(default) = default { - (default, vm.ctx.none()) - } else { - // If no default provided, use NoDefault singleton - (vm.ctx.typing_no_default.clone().into(), vm.ctx.none()) - }; + // Handle constraints and bound + let (constraints_obj, evaluate_constraints) = if !constraints.is_empty() { + // Check for single constraint + if constraints.len() == 1 { + return Err(vm.new_type_error("A single constraint is not allowed")); + } + if bound.is_some() { + return Err(vm.new_type_error("Constraints cannot be used with bound")); + } + let constraints_tuple = vm.ctx.new_tuple(constraints); + (constraints_tuple.clone().into(), constraints_tuple.into()) + } else { + (vm.ctx.none(), vm.ctx.none()) + }; - Ok(Self { - name, - bound: parking_lot::Mutex::new(bound_obj), - evaluate_bound, - constraints: parking_lot::Mutex::new(constraints_obj), - evaluate_constraints, - default_value: parking_lot::Mutex::new(default_value), - evaluate_default: PyMutex::new(evaluate_default), - covariant, - contravariant, - infer_variance, - }) - } -} + // Handle bound + let (bound_obj, evaluate_bound) = if let Some(bound) = bound { + if vm.is_none(&bound) { + (vm.ctx.none(), vm.ctx.none()) + } else { + // Type check the bound + let bound = type_check(bound, "Bound must be a type.", vm)?; + (bound, vm.ctx.none()) + } + } else { + (vm.ctx.none(), vm.ctx.none()) + }; -impl TypeVar { - pub fn new( - vm: &VirtualMachine, - name: PyObjectRef, - evaluate_bound: PyObjectRef, - evaluate_constraints: PyObjectRef, - ) -> Self { - Self { - name, - bound: parking_lot::Mutex::new(vm.ctx.none()), - evaluate_bound, - constraints: parking_lot::Mutex::new(vm.ctx.none()), - evaluate_constraints, - default_value: parking_lot::Mutex::new(vm.ctx.typing_no_default.clone().into()), - evaluate_default: PyMutex::new(vm.ctx.none()), - covariant: false, - contravariant: false, - infer_variance: false, + // Handle default value + let (default_value, evaluate_default) = if let Some(default) = default { + (default, vm.ctx.none()) + } else { + // If no default provided, use NoDefault singleton + (vm.ctx.typing_no_default.clone().into(), vm.ctx.none()) + }; + + Ok(Self { + name, + bound: parking_lot::Mutex::new(bound_obj), + evaluate_bound, + constraints: parking_lot::Mutex::new(constraints_obj), + evaluate_constraints, + default_value: parking_lot::Mutex::new(default_value), + evaluate_default: PyMutex::new(evaluate_default), + covariant, + contravariant, + infer_variance, + }) } } -} -#[pyclass(name = "ParamSpec", module = "typing")] -#[derive(Debug, PyPayload)] -#[allow(dead_code)] -pub struct ParamSpec { - name: PyObjectRef, - bound: Option<PyObjectRef>, - default_value: PyObjectRef, - evaluate_default: PyMutex<PyObjectRef>, - covariant: bool, - contravariant: bool, - infer_variance: bool, -} - -#[pyclass(flags(HAS_DICT), with(AsNumber, Constructor, Representable))] -impl ParamSpec { - #[pymethod] - fn __mro_entries__(&self, _bases: PyObjectRef, vm: &VirtualMachine) -> PyResult { - Err(vm.new_type_error("Cannot subclass an instance of ParamSpec")) + impl TypeVar { + pub fn new( + vm: &VirtualMachine, + name: PyObjectRef, + evaluate_bound: PyObjectRef, + evaluate_constraints: PyObjectRef, + ) -> Self { + Self { + name, + bound: parking_lot::Mutex::new(vm.ctx.none()), + evaluate_bound, + constraints: parking_lot::Mutex::new(vm.ctx.none()), + evaluate_constraints, + default_value: parking_lot::Mutex::new(vm.ctx.typing_no_default.clone().into()), + evaluate_default: PyMutex::new(vm.ctx.none()), + covariant: false, + contravariant: false, + infer_variance: false, + } + } } - #[pygetset] - fn __name__(&self) -> PyObjectRef { - self.name.clone() - } + #[pyattr] + #[pyclass(name = "ParamSpec", module = "typing")] + #[derive(Debug, PyPayload)] + #[allow(dead_code)] + pub struct ParamSpec { + name: PyObjectRef, + bound: Option<PyObjectRef>, + default_value: PyObjectRef, + evaluate_default: PyMutex<PyObjectRef>, + covariant: bool, + contravariant: bool, + infer_variance: bool, + } + + #[pyclass(flags(HAS_DICT), with(AsNumber, Constructor, Representable))] + impl ParamSpec { + #[pymethod] + fn __mro_entries__(&self, _bases: PyObjectRef, vm: &VirtualMachine) -> PyResult { + Err(vm.new_type_error("Cannot subclass an instance of ParamSpec")) + } - #[pygetset] - fn args(zelf: crate::PyRef<Self>, vm: &VirtualMachine) -> PyResult { - let self_obj: PyObjectRef = zelf.into(); - let psa = ParamSpecArgs { - __origin__: self_obj, - }; - Ok(psa.into_ref(&vm.ctx).into()) - } + #[pygetset] + fn __name__(&self) -> PyObjectRef { + self.name.clone() + } - #[pygetset] - fn kwargs(zelf: crate::PyRef<Self>, vm: &VirtualMachine) -> PyResult { - let self_obj: PyObjectRef = zelf.into(); - let psk = ParamSpecKwargs { - __origin__: self_obj, - }; - Ok(psk.into_ref(&vm.ctx).into()) - } + #[pygetset] + fn args(zelf: crate::PyRef<Self>, vm: &VirtualMachine) -> PyResult { + let self_obj: PyObjectRef = zelf.into(); + let psa = ParamSpecArgs { + __origin__: self_obj, + }; + Ok(psa.into_ref(&vm.ctx).into()) + } - #[pygetset] - fn __bound__(&self, vm: &VirtualMachine) -> PyObjectRef { - if let Some(bound) = self.bound.clone() { - return bound; + #[pygetset] + fn kwargs(zelf: crate::PyRef<Self>, vm: &VirtualMachine) -> PyResult { + let self_obj: PyObjectRef = zelf.into(); + let psk = ParamSpecKwargs { + __origin__: self_obj, + }; + Ok(psk.into_ref(&vm.ctx).into()) } - vm.ctx.none() - } - #[pygetset] - const fn __covariant__(&self) -> bool { - self.covariant - } + #[pygetset] + fn __bound__(&self, vm: &VirtualMachine) -> PyObjectRef { + if let Some(bound) = self.bound.clone() { + return bound; + } + vm.ctx.none() + } - #[pygetset] - const fn __contravariant__(&self) -> bool { - self.contravariant - } + #[pygetset] + const fn __covariant__(&self) -> bool { + self.covariant + } - #[pygetset] - const fn __infer_variance__(&self) -> bool { - self.infer_variance - } + #[pygetset] + const fn __contravariant__(&self) -> bool { + self.contravariant + } - #[pygetset] - fn __default__(&self, vm: &VirtualMachine) -> PyResult { - // Check if default_value is NoDefault (not just None) - if !self.default_value.is(&vm.ctx.typing_no_default) { - return Ok(self.default_value.clone()); + #[pygetset] + const fn __infer_variance__(&self) -> bool { + self.infer_variance } - // handle evaluate_default - let evaluate_default = self.evaluate_default.lock(); - if !vm.is_none(&evaluate_default) { - let default_value = evaluate_default.call((), vm)?; - return Ok(default_value); + + #[pygetset] + fn __default__(&self, vm: &VirtualMachine) -> PyResult { + // Check if default_value is NoDefault (not just None) + if !self.default_value.is(&vm.ctx.typing_no_default) { + return Ok(self.default_value.clone()); + } + // handle evaluate_default + let evaluate_default = self.evaluate_default.lock(); + if !vm.is_none(&evaluate_default) { + let default_value = evaluate_default.call((), vm)?; + return Ok(default_value); + } + // Return NoDefault singleton + Ok(vm.ctx.typing_no_default.clone().into()) } - // Return NoDefault singleton - Ok(vm.ctx.typing_no_default.clone().into()) - } - #[pygetset] - fn evaluate_default(&self, _vm: &VirtualMachine) -> PyObjectRef { - self.evaluate_default.lock().clone() - } + #[pygetset] + fn evaluate_default(&self, _vm: &VirtualMachine) -> PyObjectRef { + self.evaluate_default.lock().clone() + } - #[pymethod] - fn __reduce__(&self) -> PyResult { - Ok(self.name.clone()) - } + #[pymethod] + fn __reduce__(&self) -> PyResult { + Ok(self.name.clone()) + } - #[pymethod] - fn has_default(&self, vm: &VirtualMachine) -> bool { - if !vm.is_none(&self.evaluate_default.lock()) { - return true; + #[pymethod] + fn has_default(&self, vm: &VirtualMachine) -> bool { + if !vm.is_none(&self.evaluate_default.lock()) { + return true; + } + // Check if default_value is not NoDefault + !self.default_value.is(&vm.ctx.typing_no_default) } - // Check if default_value is not NoDefault - !self.default_value.is(&vm.ctx.typing_no_default) - } - #[pymethod] - fn __typing_subst__( - zelf: crate::PyRef<Self>, - arg: PyObjectRef, - vm: &VirtualMachine, - ) -> PyResult { - let self_obj: PyObjectRef = zelf.into(); - _call_typing_func_object(vm, "_paramspec_subst", (self_obj, arg)) - } + #[pymethod] + fn __typing_subst__( + zelf: crate::PyRef<Self>, + arg: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult { + let self_obj: PyObjectRef = zelf.into(); + _call_typing_func_object(vm, "_paramspec_subst", (self_obj, arg)) + } - #[pymethod] - fn __typing_prepare_subst__( - zelf: crate::PyRef<Self>, - alias: PyObjectRef, - args: PyObjectRef, - vm: &VirtualMachine, - ) -> PyResult { - let self_obj: PyObjectRef = zelf.into(); - _call_typing_func_object(vm, "_paramspec_prepare_subst", (self_obj, alias, args)) + #[pymethod] + fn __typing_prepare_subst__( + zelf: crate::PyRef<Self>, + alias: PyObjectRef, + args: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult { + let self_obj: PyObjectRef = zelf.into(); + _call_typing_func_object(vm, "_paramspec_prepare_subst", (self_obj, alias, args)) + } } -} -impl AsNumber for ParamSpec { - fn as_number() -> &'static PyNumberMethods { - static AS_NUMBER: PyNumberMethods = PyNumberMethods { - or: Some(|a, b, vm| { - _call_typing_func_object(vm, "_make_union", (a.to_owned(), b.to_owned())) - }), - ..PyNumberMethods::NOT_IMPLEMENTED - }; - &AS_NUMBER + impl AsNumber for ParamSpec { + fn as_number() -> &'static PyNumberMethods { + static AS_NUMBER: PyNumberMethods = PyNumberMethods { + or: Some(|a, b, vm| { + let args = PyTuple::new_ref(vec![a.to_owned(), b.to_owned()], &vm.ctx); + make_union(&args, vm) + }), + ..PyNumberMethods::NOT_IMPLEMENTED + }; + &AS_NUMBER + } } -} -impl Constructor for ParamSpec { - type Args = FuncArgs; + impl Constructor for ParamSpec { + type Args = FuncArgs; - fn slot_new(cls: PyTypeRef, args: Self::Args, vm: &VirtualMachine) -> PyResult { - let mut kwargs = args.kwargs; - // Parse arguments manually - let name = if args.args.is_empty() { - // Check if name is provided as keyword argument - if let Some(name) = kwargs.swap_remove("name") { - name + fn slot_new(cls: PyTypeRef, args: Self::Args, vm: &VirtualMachine) -> PyResult { + let mut kwargs = args.kwargs; + // Parse arguments manually + let name = if args.args.is_empty() { + // Check if name is provided as keyword argument + if let Some(name) = kwargs.swap_remove("name") { + name + } else { + return Err( + vm.new_type_error("ParamSpec() missing required argument: 'name' (pos 1)") + ); + } + } else if args.args.len() == 1 { + args.args[0].clone() } else { - return Err( - vm.new_type_error("ParamSpec() missing required argument: 'name' (pos 1)") - ); + return Err(vm.new_type_error("ParamSpec() takes at most 1 positional argument")); + }; + + let bound = kwargs.swap_remove("bound"); + let covariant = kwargs + .swap_remove("covariant") + .map(|v| v.try_to_bool(vm)) + .transpose()? + .unwrap_or(false); + let contravariant = kwargs + .swap_remove("contravariant") + .map(|v| v.try_to_bool(vm)) + .transpose()? + .unwrap_or(false); + let infer_variance = kwargs + .swap_remove("infer_variance") + .map(|v| v.try_to_bool(vm)) + .transpose()? + .unwrap_or(false); + let default = kwargs.swap_remove("default"); + + // Check for unexpected keyword arguments + if !kwargs.is_empty() { + let unexpected_keys: Vec<String> = kwargs.keys().map(|s| s.to_string()).collect(); + return Err(vm.new_type_error(format!( + "ParamSpec() got unexpected keyword argument(s): {}", + unexpected_keys.join(", ") + ))); } - } else if args.args.len() == 1 { - args.args[0].clone() - } else { - return Err(vm.new_type_error("ParamSpec() takes at most 1 positional argument")); - }; - let bound = kwargs.swap_remove("bound"); - let covariant = kwargs - .swap_remove("covariant") - .map(|v| v.try_to_bool(vm)) - .transpose()? - .unwrap_or(false); - let contravariant = kwargs - .swap_remove("contravariant") - .map(|v| v.try_to_bool(vm)) - .transpose()? - .unwrap_or(false); - let infer_variance = kwargs - .swap_remove("infer_variance") - .map(|v| v.try_to_bool(vm)) - .transpose()? - .unwrap_or(false); - let default = kwargs.swap_remove("default"); - - // Check for unexpected keyword arguments - if !kwargs.is_empty() { - let unexpected_keys: Vec<String> = kwargs.keys().map(|s| s.to_string()).collect(); - return Err(vm.new_type_error(format!( - "ParamSpec() got unexpected keyword argument(s): {}", - unexpected_keys.join(", ") - ))); - } - - // Check for invalid combinations - if covariant && contravariant { - return Err(vm.new_value_error("Bivariant type variables are not supported.")); - } - - if infer_variance && (covariant || contravariant) { - return Err(vm.new_value_error("Variance cannot be specified with infer_variance")); - } - - // Handle default value - let default_value = default.unwrap_or_else(|| vm.ctx.typing_no_default.clone().into()); - - let paramspec = Self { - name, - bound, - default_value, - evaluate_default: PyMutex::new(vm.ctx.none()), - covariant, - contravariant, - infer_variance, - }; + // Check for invalid combinations + if covariant && contravariant { + return Err(vm.new_value_error("Bivariant type variables are not supported.")); + } - let obj = paramspec.into_ref_with_type(vm, cls)?; - let obj_ref: PyObjectRef = obj.into(); - set_module_from_caller(&obj_ref, vm)?; - Ok(obj_ref) - } + if infer_variance && (covariant || contravariant) { + return Err(vm.new_value_error("Variance cannot be specified with infer_variance")); + } - fn py_new(_cls: &Py<PyType>, _args: Self::Args, _vm: &VirtualMachine) -> PyResult<Self> { - unimplemented!("use slot_new") - } -} + // Handle default value + let default_value = default.unwrap_or_else(|| vm.ctx.typing_no_default.clone().into()); + + let paramspec = Self { + name, + bound, + default_value, + evaluate_default: PyMutex::new(vm.ctx.none()), + covariant, + contravariant, + infer_variance, + }; + + let obj = paramspec.into_ref_with_type(vm, cls)?; + let obj_ref: PyObjectRef = obj.into(); + set_module_from_caller(&obj_ref, vm)?; + Ok(obj_ref) + } -impl Representable for ParamSpec { - #[inline(always)] - fn repr_str(zelf: &crate::Py<Self>, vm: &VirtualMachine) -> PyResult<String> { - let name = zelf.__name__().str(vm)?; - Ok(format!("~{name}")) + fn py_new(_cls: &Py<PyType>, _args: Self::Args, _vm: &VirtualMachine) -> PyResult<Self> { + unimplemented!("use slot_new") + } } -} -impl ParamSpec { - pub fn new(name: PyObjectRef, vm: &VirtualMachine) -> Self { - Self { - name, - bound: None, - default_value: vm.ctx.typing_no_default.clone().into(), - evaluate_default: PyMutex::new(vm.ctx.none()), - covariant: false, - contravariant: false, - infer_variance: false, + impl Representable for ParamSpec { + #[inline(always)] + fn repr_str(zelf: &crate::Py<Self>, vm: &VirtualMachine) -> PyResult<String> { + let name = zelf.__name__().str(vm)?; + Ok(format!("~{name}")) } } -} -#[pyclass(name = "TypeVarTuple", module = "typing")] -#[derive(Debug, PyPayload)] -#[allow(dead_code)] -pub struct TypeVarTuple { - name: PyObjectRef, - default_value: parking_lot::Mutex<PyObjectRef>, - evaluate_default: PyMutex<PyObjectRef>, -} -#[pyclass(flags(HAS_DICT), with(Constructor, Representable, Iterable))] -impl TypeVarTuple { - #[pygetset] - fn __name__(&self) -> PyObjectRef { - self.name.clone() + impl ParamSpec { + pub fn new(name: PyObjectRef, vm: &VirtualMachine) -> Self { + Self { + name, + bound: None, + default_value: vm.ctx.typing_no_default.clone().into(), + evaluate_default: PyMutex::new(vm.ctx.none()), + covariant: false, + contravariant: false, + infer_variance: false, + } + } } - #[pygetset] - fn __default__(&self, vm: &VirtualMachine) -> PyResult { - let mut default_value = self.default_value.lock(); - // Check if default_value is NoDefault (not just None) - if !default_value.is(&vm.ctx.typing_no_default) { - return Ok(default_value.clone()); - } - let evaluate_default = self.evaluate_default.lock(); - if !vm.is_none(&evaluate_default) { - *default_value = evaluate_default.call((), vm)?; - Ok(default_value.clone()) - } else { - // Return NoDefault singleton - Ok(vm.ctx.typing_no_default.clone().into()) + #[pyattr] + #[pyclass(name = "TypeVarTuple", module = "typing")] + #[derive(Debug, PyPayload)] + #[allow(dead_code)] + pub struct TypeVarTuple { + name: PyObjectRef, + default_value: parking_lot::Mutex<PyObjectRef>, + evaluate_default: PyMutex<PyObjectRef>, + } + #[pyclass(flags(HAS_DICT), with(Constructor, Representable, Iterable))] + impl TypeVarTuple { + #[pygetset] + fn __name__(&self) -> PyObjectRef { + self.name.clone() + } + + #[pygetset] + fn __default__(&self, vm: &VirtualMachine) -> PyResult { + let mut default_value = self.default_value.lock(); + // Check if default_value is NoDefault (not just None) + if !default_value.is(&vm.ctx.typing_no_default) { + return Ok(default_value.clone()); + } + let evaluate_default = self.evaluate_default.lock(); + if !vm.is_none(&evaluate_default) { + *default_value = evaluate_default.call((), vm)?; + Ok(default_value.clone()) + } else { + // Return NoDefault singleton + Ok(vm.ctx.typing_no_default.clone().into()) + } } - } - #[pymethod] - fn has_default(&self, vm: &VirtualMachine) -> bool { - if !vm.is_none(&self.evaluate_default.lock()) { - return true; + #[pymethod] + fn has_default(&self, vm: &VirtualMachine) -> bool { + if !vm.is_none(&self.evaluate_default.lock()) { + return true; + } + let default_value = self.default_value.lock(); + // Check if default_value is not NoDefault + !default_value.is(&vm.ctx.typing_no_default) } - let default_value = self.default_value.lock(); - // Check if default_value is not NoDefault - !default_value.is(&vm.ctx.typing_no_default) - } - #[pymethod] - fn __reduce__(&self) -> PyObjectRef { - self.name.clone() - } + #[pymethod] + fn __reduce__(&self) -> PyObjectRef { + self.name.clone() + } - #[pymethod] - fn __mro_entries__(&self, _bases: PyObjectRef, vm: &VirtualMachine) -> PyResult { - Err(vm.new_type_error("Cannot subclass an instance of TypeVarTuple")) - } + #[pymethod] + fn __mro_entries__(&self, _bases: PyObjectRef, vm: &VirtualMachine) -> PyResult { + Err(vm.new_type_error("Cannot subclass an instance of TypeVarTuple")) + } - #[pymethod] - fn __typing_subst__(&self, _arg: PyObjectRef, vm: &VirtualMachine) -> PyResult { - Err(vm.new_type_error("Substitution of bare TypeVarTuple is not supported")) - } + #[pymethod] + fn __typing_subst__(&self, _arg: PyObjectRef, vm: &VirtualMachine) -> PyResult { + Err(vm.new_type_error("Substitution of bare TypeVarTuple is not supported")) + } - #[pymethod] - fn __typing_prepare_subst__( - zelf: crate::PyRef<Self>, - alias: PyObjectRef, - args: PyObjectRef, - vm: &VirtualMachine, - ) -> PyResult { - let self_obj: PyObjectRef = zelf.into(); - _call_typing_func_object(vm, "_typevartuple_prepare_subst", (self_obj, alias, args)) + #[pymethod] + fn __typing_prepare_subst__( + zelf: crate::PyRef<Self>, + alias: PyObjectRef, + args: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult { + let self_obj: PyObjectRef = zelf.into(); + _call_typing_func_object(vm, "_typevartuple_prepare_subst", (self_obj, alias, args)) + } } -} -impl Iterable for TypeVarTuple { - fn iter(zelf: PyRef<Self>, vm: &VirtualMachine) -> PyResult { - // When unpacking TypeVarTuple with *, return [Unpack[self]] - // This is how CPython handles Generic[*Ts] - let typing = vm.import("typing", 0)?; - let unpack = typing.get_attr("Unpack", vm)?; - let zelf_obj: PyObjectRef = zelf.into(); - let unpacked = vm.call_method(&unpack, "__getitem__", (zelf_obj,))?; - let list = vm.ctx.new_list(vec![unpacked]); - let list_obj: PyObjectRef = list.into(); - vm.call_method(&list_obj, "__iter__", ()) + impl Iterable for TypeVarTuple { + fn iter(zelf: PyRef<Self>, vm: &VirtualMachine) -> PyResult { + // When unpacking TypeVarTuple with *, return [Unpack[self]] + // This is how CPython handles Generic[*Ts] + let typing = vm.import("typing", 0)?; + let unpack = typing.get_attr("Unpack", vm)?; + let zelf_obj: PyObjectRef = zelf.into(); + let unpacked = vm.call_method(&unpack, "__getitem__", (zelf_obj,))?; + let list = vm.ctx.new_list(vec![unpacked]); + let list_obj: PyObjectRef = list.into(); + vm.call_method(&list_obj, "__iter__", ()) + } } -} -impl Constructor for TypeVarTuple { - type Args = FuncArgs; + impl Constructor for TypeVarTuple { + type Args = FuncArgs; - fn slot_new(cls: PyTypeRef, args: Self::Args, vm: &VirtualMachine) -> PyResult { - let mut kwargs = args.kwargs; - // Parse arguments manually - let name = if args.args.is_empty() { - // Check if name is provided as keyword argument - if let Some(name) = kwargs.swap_remove("name") { - name + fn slot_new(cls: PyTypeRef, args: Self::Args, vm: &VirtualMachine) -> PyResult { + let mut kwargs = args.kwargs; + // Parse arguments manually + let name = if args.args.is_empty() { + // Check if name is provided as keyword argument + if let Some(name) = kwargs.swap_remove("name") { + name + } else { + return Err(vm.new_type_error( + "TypeVarTuple() missing required argument: 'name' (pos 1)", + )); + } + } else if args.args.len() == 1 { + args.args[0].clone() } else { - return Err( - vm.new_type_error("TypeVarTuple() missing required argument: 'name' (pos 1)") - ); + return Err(vm.new_type_error("TypeVarTuple() takes at most 1 positional argument")); + }; + + let default = kwargs.swap_remove("default"); + + // Check for unexpected keyword arguments + if !kwargs.is_empty() { + let unexpected_keys: Vec<String> = kwargs.keys().map(|s| s.to_string()).collect(); + return Err(vm.new_type_error(format!( + "TypeVarTuple() got unexpected keyword argument(s): {}", + unexpected_keys.join(", ") + ))); } - } else if args.args.len() == 1 { - args.args[0].clone() - } else { - return Err(vm.new_type_error("TypeVarTuple() takes at most 1 positional argument")); - }; - let default = kwargs.swap_remove("default"); - - // Check for unexpected keyword arguments - if !kwargs.is_empty() { - let unexpected_keys: Vec<String> = kwargs.keys().map(|s| s.to_string()).collect(); - return Err(vm.new_type_error(format!( - "TypeVarTuple() got unexpected keyword argument(s): {}", - unexpected_keys.join(", ") - ))); + // Handle default value + let (default_value, evaluate_default) = if let Some(default) = default { + (default, vm.ctx.none()) + } else { + // If no default provided, use NoDefault singleton + (vm.ctx.typing_no_default.clone().into(), vm.ctx.none()) + }; + + let typevartuple = Self { + name, + default_value: parking_lot::Mutex::new(default_value), + evaluate_default: PyMutex::new(evaluate_default), + }; + + let obj = typevartuple.into_ref_with_type(vm, cls)?; + let obj_ref: PyObjectRef = obj.into(); + set_module_from_caller(&obj_ref, vm)?; + Ok(obj_ref) } - // Handle default value - let (default_value, evaluate_default) = if let Some(default) = default { - (default, vm.ctx.none()) - } else { - // If no default provided, use NoDefault singleton - (vm.ctx.typing_no_default.clone().into(), vm.ctx.none()) - }; - - let typevartuple = Self { - name, - default_value: parking_lot::Mutex::new(default_value), - evaluate_default: PyMutex::new(evaluate_default), - }; - - let obj = typevartuple.into_ref_with_type(vm, cls)?; - let obj_ref: PyObjectRef = obj.into(); - set_module_from_caller(&obj_ref, vm)?; - Ok(obj_ref) + fn py_new(_cls: &Py<PyType>, _args: Self::Args, _vm: &VirtualMachine) -> PyResult<Self> { + unimplemented!("use slot_new") + } } - fn py_new(_cls: &Py<PyType>, _args: Self::Args, _vm: &VirtualMachine) -> PyResult<Self> { - unimplemented!("use slot_new") + impl Representable for TypeVarTuple { + #[inline(always)] + fn repr_str(zelf: &crate::Py<Self>, vm: &VirtualMachine) -> PyResult<String> { + let name = zelf.name.str(vm)?; + Ok(name.to_string()) + } } -} -impl Representable for TypeVarTuple { - #[inline(always)] - fn repr_str(zelf: &crate::Py<Self>, vm: &VirtualMachine) -> PyResult<String> { - let name = zelf.name.str(vm)?; - Ok(name.to_string()) - } -} - -impl TypeVarTuple { - pub fn new(name: PyObjectRef, vm: &VirtualMachine) -> Self { - Self { - name, - default_value: parking_lot::Mutex::new(vm.ctx.typing_no_default.clone().into()), - evaluate_default: PyMutex::new(vm.ctx.none()), + impl TypeVarTuple { + pub fn new(name: PyObjectRef, vm: &VirtualMachine) -> Self { + Self { + name, + default_value: parking_lot::Mutex::new(vm.ctx.typing_no_default.clone().into()), + evaluate_default: PyMutex::new(vm.ctx.none()), + } } } -} -#[pyclass(name = "ParamSpecArgs", module = "typing")] -#[derive(Debug, PyPayload)] -#[allow(dead_code)] -pub struct ParamSpecArgs { - __origin__: PyObjectRef, -} -#[pyclass(with(Constructor, Representable, Comparable))] -impl ParamSpecArgs { - #[pymethod] - fn __mro_entries__(&self, _bases: PyObjectRef, vm: &VirtualMachine) -> PyResult { - Err(vm.new_type_error("Cannot subclass an instance of ParamSpecArgs")) - } + #[pyattr] + #[pyclass(name = "ParamSpecArgs", module = "typing")] + #[derive(Debug, PyPayload)] + #[allow(dead_code)] + pub struct ParamSpecArgs { + __origin__: PyObjectRef, + } + #[pyclass(with(Constructor, Representable, Comparable))] + impl ParamSpecArgs { + #[pymethod] + fn __mro_entries__(&self, _bases: PyObjectRef, vm: &VirtualMachine) -> PyResult { + Err(vm.new_type_error("Cannot subclass an instance of ParamSpecArgs")) + } - #[pygetset] - fn __origin__(&self) -> PyObjectRef { - self.__origin__.clone() + #[pygetset] + fn __origin__(&self) -> PyObjectRef { + self.__origin__.clone() + } } -} -impl Constructor for ParamSpecArgs { - type Args = (PyObjectRef,); + impl Constructor for ParamSpecArgs { + type Args = (PyObjectRef,); - fn py_new(_cls: &Py<PyType>, args: Self::Args, _vm: &VirtualMachine) -> PyResult<Self> { - let origin = args.0; - Ok(Self { __origin__: origin }) - } -} - -impl Representable for ParamSpecArgs { - #[inline(always)] - fn repr_str(zelf: &crate::Py<Self>, vm: &VirtualMachine) -> PyResult<String> { - // Check if origin is a ParamSpec - if let Ok(name) = zelf.__origin__.get_attr("__name__", vm) { - return Ok(format!("{name}.args", name = name.str(vm)?)); + fn py_new(_cls: &Py<PyType>, args: Self::Args, _vm: &VirtualMachine) -> PyResult<Self> { + let origin = args.0; + Ok(Self { __origin__: origin }) } - Ok(format!("{:?}.args", zelf.__origin__)) } -} -impl Comparable for ParamSpecArgs { - fn cmp( - zelf: &crate::Py<Self>, - other: &PyObject, - op: PyComparisonOp, - vm: &VirtualMachine, - ) -> PyResult<PyComparisonValue> { - fn eq( - zelf: &crate::Py<ParamSpecArgs>, - other: PyObjectRef, - _vm: &VirtualMachine, - ) -> PyResult<bool> { - // First check if other is also ParamSpecArgs - if let Ok(other_args) = other.downcast::<ParamSpecArgs>() { - // Check if they have the same origin - return Ok(zelf.__origin__.is(&other_args.__origin__)); + impl Representable for ParamSpecArgs { + #[inline(always)] + fn repr_str(zelf: &crate::Py<Self>, vm: &VirtualMachine) -> PyResult<String> { + // Check if origin is a ParamSpec + if let Ok(name) = zelf.__origin__.get_attr("__name__", vm) { + return Ok(format!("{name}.args", name = name.str(vm)?)); } - Ok(false) + Ok(format!("{:?}.args", zelf.__origin__)) } - match op { - PyComparisonOp::Eq => { - if let Ok(result) = eq(zelf, other.to_owned(), vm) { - Ok(result.into()) - } else { - Ok(PyComparisonValue::NotImplemented) + } + + impl Comparable for ParamSpecArgs { + fn cmp( + zelf: &crate::Py<Self>, + other: &PyObject, + op: PyComparisonOp, + vm: &VirtualMachine, + ) -> PyResult<PyComparisonValue> { + fn eq( + zelf: &crate::Py<ParamSpecArgs>, + other: PyObjectRef, + _vm: &VirtualMachine, + ) -> PyResult<bool> { + // First check if other is also ParamSpecArgs + if let Ok(other_args) = other.downcast::<ParamSpecArgs>() { + // Check if they have the same origin + return Ok(zelf.__origin__.is(&other_args.__origin__)); } + Ok(false) } - PyComparisonOp::Ne => { - if let Ok(result) = eq(zelf, other.to_owned(), vm) { - Ok((!result).into()) - } else { - Ok(PyComparisonValue::NotImplemented) + match op { + PyComparisonOp::Eq => { + if let Ok(result) = eq(zelf, other.to_owned(), vm) { + Ok(result.into()) + } else { + Ok(PyComparisonValue::NotImplemented) + } + } + PyComparisonOp::Ne => { + if let Ok(result) = eq(zelf, other.to_owned(), vm) { + Ok((!result).into()) + } else { + Ok(PyComparisonValue::NotImplemented) + } } + _ => Ok(PyComparisonValue::NotImplemented), } - _ => Ok(PyComparisonValue::NotImplemented), } } -} -#[pyclass(name = "ParamSpecKwargs", module = "typing")] -#[derive(Debug, PyPayload)] -#[allow(dead_code)] -pub struct ParamSpecKwargs { - __origin__: PyObjectRef, -} -#[pyclass(with(Constructor, Representable, Comparable))] -impl ParamSpecKwargs { - #[pymethod] - fn __mro_entries__(&self, _bases: PyObjectRef, vm: &VirtualMachine) -> PyResult { - Err(vm.new_type_error("Cannot subclass an instance of ParamSpecKwargs")) - } + #[pyattr] + #[pyclass(name = "ParamSpecKwargs", module = "typing")] + #[derive(Debug, PyPayload)] + #[allow(dead_code)] + pub struct ParamSpecKwargs { + __origin__: PyObjectRef, + } + #[pyclass(with(Constructor, Representable, Comparable))] + impl ParamSpecKwargs { + #[pymethod] + fn __mro_entries__(&self, _bases: PyObjectRef, vm: &VirtualMachine) -> PyResult { + Err(vm.new_type_error("Cannot subclass an instance of ParamSpecKwargs")) + } - #[pygetset] - fn __origin__(&self) -> PyObjectRef { - self.__origin__.clone() + #[pygetset] + fn __origin__(&self) -> PyObjectRef { + self.__origin__.clone() + } } -} - -impl Constructor for ParamSpecKwargs { - type Args = (PyObjectRef,); - fn py_new(_cls: &Py<PyType>, args: Self::Args, _vm: &VirtualMachine) -> PyResult<Self> { - let origin = args.0; - Ok(Self { __origin__: origin }) - } -} + impl Constructor for ParamSpecKwargs { + type Args = (PyObjectRef,); -impl Representable for ParamSpecKwargs { - #[inline(always)] - fn repr_str(zelf: &crate::Py<Self>, vm: &VirtualMachine) -> PyResult<String> { - // Check if origin is a ParamSpec - if let Ok(name) = zelf.__origin__.get_attr("__name__", vm) { - return Ok(format!("{name}.kwargs", name = name.str(vm)?)); + fn py_new(_cls: &Py<PyType>, args: Self::Args, _vm: &VirtualMachine) -> PyResult<Self> { + let origin = args.0; + Ok(Self { __origin__: origin }) } - Ok(format!("{:?}.kwargs", zelf.__origin__)) } -} -impl Comparable for ParamSpecKwargs { - fn cmp( - zelf: &crate::Py<Self>, - other: &PyObject, - op: PyComparisonOp, - vm: &VirtualMachine, - ) -> PyResult<PyComparisonValue> { - fn eq( - zelf: &crate::Py<ParamSpecKwargs>, - other: PyObjectRef, - _vm: &VirtualMachine, - ) -> PyResult<bool> { - // First check if other is also ParamSpecKwargs - if let Ok(other_kwargs) = other.downcast::<ParamSpecKwargs>() { - // Check if they have the same origin - return Ok(zelf.__origin__.is(&other_kwargs.__origin__)); + impl Representable for ParamSpecKwargs { + #[inline(always)] + fn repr_str(zelf: &crate::Py<Self>, vm: &VirtualMachine) -> PyResult<String> { + // Check if origin is a ParamSpec + if let Ok(name) = zelf.__origin__.get_attr("__name__", vm) { + return Ok(format!("{name}.kwargs", name = name.str(vm)?)); } - Ok(false) + Ok(format!("{:?}.kwargs", zelf.__origin__)) } - match op { - PyComparisonOp::Eq => { - if let Ok(result) = eq(zelf, other.to_owned(), vm) { - Ok(result.into()) - } else { - Ok(PyComparisonValue::NotImplemented) + } + + impl Comparable for ParamSpecKwargs { + fn cmp( + zelf: &crate::Py<Self>, + other: &PyObject, + op: PyComparisonOp, + vm: &VirtualMachine, + ) -> PyResult<PyComparisonValue> { + fn eq( + zelf: &crate::Py<ParamSpecKwargs>, + other: PyObjectRef, + _vm: &VirtualMachine, + ) -> PyResult<bool> { + // First check if other is also ParamSpecKwargs + if let Ok(other_kwargs) = other.downcast::<ParamSpecKwargs>() { + // Check if they have the same origin + return Ok(zelf.__origin__.is(&other_kwargs.__origin__)); } + Ok(false) } - PyComparisonOp::Ne => { - if let Ok(result) = eq(zelf, other.to_owned(), vm) { - Ok((!result).into()) - } else { - Ok(PyComparisonValue::NotImplemented) + match op { + PyComparisonOp::Eq => { + if let Ok(result) = eq(zelf, other.to_owned(), vm) { + Ok(result.into()) + } else { + Ok(PyComparisonValue::NotImplemented) + } + } + PyComparisonOp::Ne => { + if let Ok(result) = eq(zelf, other.to_owned(), vm) { + Ok((!result).into()) + } else { + Ok(PyComparisonValue::NotImplemented) + } } + _ => Ok(PyComparisonValue::NotImplemented), } - _ => Ok(PyComparisonValue::NotImplemented), } } -} -/// Helper function to call typing module functions with cls as first argument -/// Similar to CPython's call_typing_args_kwargs -fn call_typing_args_kwargs( - name: &'static str, - cls: PyTypeRef, - args: FuncArgs, - vm: &VirtualMachine, -) -> PyResult { - let typing = vm.import("typing", 0)?; - let func = typing.get_attr(name, vm)?; - - // Prepare arguments: (cls, *args) - let mut call_args = vec![cls.into()]; - call_args.extend(args.args); - - // Call with prepared args and original kwargs - let func_args = FuncArgs { - args: call_args, - kwargs: args.kwargs, - }; + /// Helper function to call typing module functions with cls as first argument + /// Similar to CPython's call_typing_args_kwargs + fn call_typing_args_kwargs( + name: &'static str, + cls: PyTypeRef, + args: FuncArgs, + vm: &VirtualMachine, + ) -> PyResult { + let typing = vm.import("typing", 0)?; + let func = typing.get_attr(name, vm)?; - func.call(func_args, vm) -} + // Prepare arguments: (cls, *args) + let mut call_args = vec![cls.into()]; + call_args.extend(args.args); -#[pyclass(name = "Generic", module = "typing")] -#[derive(Debug, PyPayload)] -#[allow(dead_code)] -pub struct Generic {} + // Call with prepared args and original kwargs + let func_args = FuncArgs { + args: call_args, + kwargs: args.kwargs, + }; -#[pyclass(flags(BASETYPE))] -impl Generic { - #[pyattr] - fn __slots__(ctx: &Context) -> PyTupleRef { - ctx.empty_tuple.clone() + func.call(func_args, vm) } - #[pyclassmethod] - fn __class_getitem__(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult { - call_typing_args_kwargs("_generic_class_getitem", cls, args, vm) - } + #[pyattr] + #[pyclass(name = "Generic", module = "typing")] + #[derive(Debug, PyPayload)] + #[allow(dead_code)] + pub struct Generic; + + #[pyclass(flags(BASETYPE))] + impl Generic { + #[pyattr] + fn __slots__(ctx: &Context) -> PyTupleRef { + ctx.empty_tuple.clone() + } - #[pyclassmethod] - fn __init_subclass__(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult { - call_typing_args_kwargs("_generic_init_subclass", cls, args, vm) - } -} + #[pyclassmethod] + fn __class_getitem__(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult { + call_typing_args_kwargs("_generic_class_getitem", cls, args, vm) + } -/// Sets the default value for a type parameter, equivalent to CPython's _Py_set_typeparam_default -/// This is used by the CALL_INTRINSIC_2 SetTypeparamDefault instruction -pub fn set_typeparam_default( - type_param: PyObjectRef, - evaluate_default: PyObjectRef, - vm: &VirtualMachine, -) -> PyResult { - // Inner function to handle common pattern of setting evaluate_default - fn try_set_default<T>( - obj: &PyObject, - evaluate_default: &PyObject, - get_field: impl FnOnce(&T) -> &PyMutex<PyObjectRef>, - ) -> bool - where - T: PyPayload, - { - if let Some(typed_obj) = obj.downcast_ref::<T>() { - *get_field(typed_obj).lock() = evaluate_default.to_owned(); - true - } else { - false + #[pyclassmethod] + fn __init_subclass__(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult { + call_typing_args_kwargs("_generic_init_subclass", cls, args, vm) } } - // Try each type parameter type - if try_set_default::<TypeVar>(&type_param, &evaluate_default, |tv| &tv.evaluate_default) - || try_set_default::<ParamSpec>(&type_param, &evaluate_default, |ps| &ps.evaluate_default) - || try_set_default::<TypeVarTuple>(&type_param, &evaluate_default, |tvt| { - &tvt.evaluate_default - }) - { - Ok(type_param) - } else { - Err(vm.new_type_error(format!( - "Expected a type param, got {}", - type_param.class().name() - ))) + /// Sets the default value for a type parameter, equivalent to CPython's _Py_set_typeparam_default + /// This is used by the CALL_INTRINSIC_2 SetTypeparamDefault instruction + pub fn set_typeparam_default( + type_param: PyObjectRef, + evaluate_default: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult { + // Inner function to handle common pattern of setting evaluate_default + fn try_set_default<T>( + obj: &PyObject, + evaluate_default: &PyObject, + get_field: impl FnOnce(&T) -> &PyMutex<PyObjectRef>, + ) -> bool + where + T: PyPayload, + { + if let Some(typed_obj) = obj.downcast_ref::<T>() { + *get_field(typed_obj).lock() = evaluate_default.to_owned(); + true + } else { + false + } + } + + // Try each type parameter type + if try_set_default::<TypeVar>(&type_param, &evaluate_default, |tv| &tv.evaluate_default) + || try_set_default::<ParamSpec>(&type_param, &evaluate_default, |ps| { + &ps.evaluate_default + }) + || try_set_default::<TypeVarTuple>(&type_param, &evaluate_default, |tvt| { + &tvt.evaluate_default + }) + { + Ok(type_param) + } else { + Err(vm.new_type_error(format!( + "Expected a type param, got {}", + type_param.class().name() + ))) + } } } diff --git a/crates/vm/src/stdlib/typing.rs b/crates/vm/src/stdlib/typing.rs index f11acce3490..6938bca8bbb 100644 --- a/crates/vm/src/stdlib/typing.rs +++ b/crates/vm/src/stdlib/typing.rs @@ -1,10 +1,11 @@ // spell-checker:ignore typevarobject funcobj -use crate::{Context, PyPayload, PyRef, VirtualMachine, class::PyClassImpl, stdlib::PyModule}; +use crate::{Context, class::PyClassImpl}; pub use crate::stdlib::typevar::{ Generic, ParamSpec, ParamSpecArgs, ParamSpecKwargs, TypeVar, TypeVarTuple, set_typeparam_default, }; +pub(crate) use decl::module_def; pub use decl::*; /// Initialize typing types (call extend_class) @@ -12,32 +13,11 @@ pub fn init(ctx: &Context) { NoDefault::extend_class(ctx, ctx.types.typing_no_default_type); } -pub(crate) fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - let module = decl::make_module(vm); - TypeVar::make_class(&vm.ctx); - ParamSpec::make_class(&vm.ctx); - TypeVarTuple::make_class(&vm.ctx); - ParamSpecArgs::make_class(&vm.ctx); - ParamSpecKwargs::make_class(&vm.ctx); - Generic::make_class(&vm.ctx); - extend_module!(vm, &module, { - "NoDefault" => vm.ctx.typing_no_default.clone(), - "TypeVar" => TypeVar::class(&vm.ctx).to_owned(), - "ParamSpec" => ParamSpec::class(&vm.ctx).to_owned(), - "TypeVarTuple" => TypeVarTuple::class(&vm.ctx).to_owned(), - "ParamSpecArgs" => ParamSpecArgs::class(&vm.ctx).to_owned(), - "ParamSpecKwargs" => ParamSpecKwargs::class(&vm.ctx).to_owned(), - "Generic" => Generic::class(&vm.ctx).to_owned(), - }); - module -} - -#[pymodule(name = "_typing")] +#[pymodule(name = "_typing", with(super::typevar::typevar))] pub(crate) mod decl { use crate::{ Py, PyObjectRef, PyPayload, PyResult, VirtualMachine, builtins::{PyStrRef, PyTupleRef, PyType, PyTypeRef, pystr::AsPyStr, type_}, - convert::ToPyResult, function::{FuncArgs, IntoFuncArgs}, protocol::PyNumberMethods, types::{AsNumber, Constructor, Representable}, @@ -188,22 +168,24 @@ pub(crate) mod decl { impl AsNumber for TypeAliasType { fn as_number() -> &'static PyNumberMethods { static AS_NUMBER: PyNumberMethods = PyNumberMethods { - or: Some(|a, b, vm| type_::or_(a.to_owned(), b.to_owned(), vm).to_pyresult(vm)), + or: Some(|a, b, vm| type_::or_(a.to_owned(), b.to_owned(), vm)), ..PyNumberMethods::NOT_IMPLEMENTED }; &AS_NUMBER } } - // impl AsMapping for Generic { - // fn as_mapping() -> &'static PyMappingMethods { - // static AS_MAPPING: Lazy<PyMappingMethods> = Lazy::new(|| PyMappingMethods { - // subscript: atomic_func!(|mapping, needle, vm| { - // call_typing_func_object(vm, "_GenericAlias", (mapping.obj, needle)) - // }), - // ..PyMappingMethods::NOT_IMPLEMENTED - // }); - // &AS_MAPPING - // } - // } + pub(crate) fn module_exec( + vm: &VirtualMachine, + module: &Py<crate::builtins::PyModule>, + ) -> PyResult<()> { + __module_exec(vm, module); + + extend_module!(vm, module, { + "NoDefault" => vm.ctx.typing_no_default.clone(), + "Union" => vm.ctx.types.union_type.to_owned(), + }); + + Ok(()) + } } diff --git a/crates/vm/src/stdlib/warnings.rs b/crates/vm/src/stdlib/warnings.rs index 2d61c3b571f..198df07d6c0 100644 --- a/crates/vm/src/stdlib/warnings.rs +++ b/crates/vm/src/stdlib/warnings.rs @@ -1,4 +1,4 @@ -pub(crate) use _warnings::make_module; +pub(crate) use _warnings::module_def; use crate::{Py, PyResult, VirtualMachine, builtins::PyType}; @@ -12,7 +12,7 @@ pub fn warn( if let Ok(module) = vm.import("warnings", 0) && let Ok(func) = module.get_attr("warn", vm) { - let _ = func.call((message, category.to_owned(), stack_level), vm); + func.call((message, category.to_owned(), stack_level), vm)?; } Ok(()) } diff --git a/crates/vm/src/stdlib/weakref.rs b/crates/vm/src/stdlib/weakref.rs index bedfad9abbd..e7e030b2b01 100644 --- a/crates/vm/src/stdlib/weakref.rs +++ b/crates/vm/src/stdlib/weakref.rs @@ -4,7 +4,7 @@ //! - [python weakref module](https://docs.python.org/3/library/weakref.html) //! - [rust weak struct](https://doc.rust-lang.org/std/rc/struct.Weak.html) //! -pub(crate) use _weakref::make_module; +pub(crate) use _weakref::module_def; #[pymodule] mod _weakref { diff --git a/crates/vm/src/stdlib/winapi.rs b/crates/vm/src/stdlib/winapi.rs index 7279f9776e2..c1fe32aadfc 100644 --- a/crates/vm/src/stdlib/winapi.rs +++ b/crates/vm/src/stdlib/winapi.rs @@ -1,20 +1,21 @@ // spell-checker:disable #![allow(non_snake_case)] -pub(crate) use _winapi::make_module; +pub(crate) use _winapi::module_def; #[pymodule] mod _winapi { use crate::{ - PyObjectRef, PyResult, TryFromObject, VirtualMachine, + Py, PyObjectRef, PyPayload, PyResult, TryFromObject, VirtualMachine, builtins::PyStrRef, - common::windows::ToWideString, + common::{lock::PyMutex, windows::ToWideString}, convert::{ToPyException, ToPyResult}, function::{ArgMapping, ArgSequence, OptionalArg}, + types::Constructor, windows::{WinHandle, WindowsSysResult}, }; use std::ptr::{null, null_mut}; - use windows_sys::Win32::Foundation::{INVALID_HANDLE_VALUE, MAX_PATH}; + use windows_sys::Win32::Foundation::{HANDLE, INVALID_HANDLE_VALUE, MAX_PATH}; #[pyattr] use windows_sys::Win32::{ @@ -31,19 +32,57 @@ mod _winapi { LCMAP_TRADITIONAL_CHINESE, LCMAP_UPPERCASE, }, Storage::FileSystem::{ - COPY_FILE_ALLOW_DECRYPTED_DESTINATION, COPY_FILE_COPY_SYMLINK, - COPY_FILE_FAIL_IF_EXISTS, COPY_FILE_NO_BUFFERING, COPY_FILE_NO_OFFLOAD, - COPY_FILE_OPEN_SOURCE_FOR_WRITE, COPY_FILE_REQUEST_COMPRESSED_TRAFFIC, - COPY_FILE_REQUEST_SECURITY_PRIVILEGES, COPY_FILE_RESTARTABLE, - COPY_FILE_RESUME_FROM_PAUSE, COPYFILE2_CALLBACK_CHUNK_FINISHED, - COPYFILE2_CALLBACK_CHUNK_STARTED, COPYFILE2_CALLBACK_ERROR, - COPYFILE2_CALLBACK_POLL_CONTINUE, COPYFILE2_CALLBACK_STREAM_FINISHED, - COPYFILE2_CALLBACK_STREAM_STARTED, COPYFILE2_PROGRESS_CANCEL, - COPYFILE2_PROGRESS_CONTINUE, COPYFILE2_PROGRESS_PAUSE, COPYFILE2_PROGRESS_QUIET, - COPYFILE2_PROGRESS_STOP, FILE_FLAG_FIRST_PIPE_INSTANCE, FILE_FLAG_OVERLAPPED, - FILE_GENERIC_READ, FILE_GENERIC_WRITE, FILE_TYPE_CHAR, FILE_TYPE_DISK, FILE_TYPE_PIPE, - FILE_TYPE_REMOTE, FILE_TYPE_UNKNOWN, OPEN_EXISTING, PIPE_ACCESS_DUPLEX, - PIPE_ACCESS_INBOUND, SYNCHRONIZE, + COPY_FILE_ALLOW_DECRYPTED_DESTINATION, + COPY_FILE_COPY_SYMLINK, + COPY_FILE_FAIL_IF_EXISTS, + COPY_FILE_NO_BUFFERING, + COPY_FILE_NO_OFFLOAD, + COPY_FILE_OPEN_SOURCE_FOR_WRITE, + COPY_FILE_REQUEST_COMPRESSED_TRAFFIC, + COPY_FILE_REQUEST_SECURITY_PRIVILEGES, + COPY_FILE_RESTARTABLE, + COPY_FILE_RESUME_FROM_PAUSE, + COPYFILE2_CALLBACK_CHUNK_FINISHED, + COPYFILE2_CALLBACK_CHUNK_STARTED, + COPYFILE2_CALLBACK_ERROR, + COPYFILE2_CALLBACK_POLL_CONTINUE, + COPYFILE2_CALLBACK_STREAM_FINISHED, + COPYFILE2_CALLBACK_STREAM_STARTED, + COPYFILE2_PROGRESS_CANCEL, + COPYFILE2_PROGRESS_CONTINUE, + COPYFILE2_PROGRESS_PAUSE, + COPYFILE2_PROGRESS_QUIET, + COPYFILE2_PROGRESS_STOP, + CREATE_ALWAYS, + // CreateFile constants + CREATE_NEW, + FILE_ATTRIBUTE_NORMAL, + FILE_FLAG_BACKUP_SEMANTICS, + FILE_FLAG_DELETE_ON_CLOSE, + FILE_FLAG_FIRST_PIPE_INSTANCE, + FILE_FLAG_NO_BUFFERING, + FILE_FLAG_OPEN_REPARSE_POINT, + FILE_FLAG_OVERLAPPED, + FILE_FLAG_POSIX_SEMANTICS, + FILE_FLAG_RANDOM_ACCESS, + FILE_FLAG_SEQUENTIAL_SCAN, + FILE_FLAG_WRITE_THROUGH, + FILE_GENERIC_READ, + FILE_GENERIC_WRITE, + FILE_SHARE_DELETE, + FILE_SHARE_READ, + FILE_SHARE_WRITE, + FILE_TYPE_CHAR, + FILE_TYPE_DISK, + FILE_TYPE_PIPE, + FILE_TYPE_REMOTE, + FILE_TYPE_UNKNOWN, + OPEN_ALWAYS, + OPEN_EXISTING, + PIPE_ACCESS_DUPLEX, + PIPE_ACCESS_INBOUND, + SYNCHRONIZE, + TRUNCATE_EXISTING, }, System::{ Console::{STD_ERROR_HANDLE, STD_INPUT_HANDLE, STD_OUTPUT_HANDLE}, @@ -55,6 +94,7 @@ mod _winapi { SEC_LARGE_PAGES, SEC_NOCACHE, SEC_RESERVE, SEC_WRITECOMBINE, }, Pipes::{ + NMPWAIT_NOWAIT, NMPWAIT_USE_DEFAULT_WAIT, NMPWAIT_WAIT_FOREVER, PIPE_READMODE_MESSAGE, PIPE_TYPE_MESSAGE, PIPE_UNLIMITED_INSTANCES, PIPE_WAIT, }, SystemServices::LOCALE_NAME_MAX_LENGTH, @@ -78,6 +118,42 @@ mod _winapi { WindowsSysResult(unsafe { windows_sys::Win32::Foundation::CloseHandle(handle.0) }) } + /// CreateFile - Create or open a file or I/O device. + #[pyfunction] + #[allow(clippy::too_many_arguments)] + fn CreateFile( + file_name: PyStrRef, + desired_access: u32, + share_mode: u32, + _security_attributes: PyObjectRef, // Always NULL (0) + creation_disposition: u32, + flags_and_attributes: u32, + _template_file: PyObjectRef, // Always NULL (0) + vm: &VirtualMachine, + ) -> PyResult<WinHandle> { + use windows_sys::Win32::Storage::FileSystem::CreateFileW; + + let file_name_wide = file_name.as_wtf8().to_wide_with_nul(); + + let handle = unsafe { + CreateFileW( + file_name_wide.as_ptr(), + desired_access, + share_mode, + null(), + creation_disposition, + flags_and_attributes, + null_mut(), + ) + }; + + if handle == INVALID_HANDLE_VALUE { + return Err(vm.new_last_os_error()); + } + + Ok(WinHandle(handle)) + } + #[pyfunction] fn GetStdHandle( std_handle: windows_sys::Win32::System::Console::STD_HANDLE, @@ -466,7 +542,15 @@ mod _winapi { } #[pyfunction] - fn WaitForSingleObject(h: WinHandle, ms: u32, vm: &VirtualMachine) -> PyResult<u32> { + fn WaitForSingleObject(h: WinHandle, ms: i64, vm: &VirtualMachine) -> PyResult<u32> { + // Negative values (e.g., -1) map to INFINITE (0xFFFFFFFF) + let ms = if ms < 0 { + windows_sys::Win32::System::Threading::INFINITE + } else if ms > u32::MAX as i64 { + return Err(vm.new_overflow_error("timeout value is too large".to_owned())); + } else { + ms as u32 + }; let ret = unsafe { windows_sys::Win32::System::Threading::WaitForSingleObject(h.0, ms) }; if ret == windows_sys::Win32::Foundation::WAIT_FAILED { Err(vm.new_last_os_error()) @@ -475,6 +559,48 @@ mod _winapi { } } + #[pyfunction] + fn WaitForMultipleObjects( + handle_seq: ArgSequence<isize>, + wait_all: bool, + milliseconds: u32, + vm: &VirtualMachine, + ) -> PyResult<u32> { + use windows_sys::Win32::Foundation::WAIT_FAILED; + use windows_sys::Win32::System::Threading::WaitForMultipleObjects as WinWaitForMultipleObjects; + + let handles: Vec<HANDLE> = handle_seq + .into_vec() + .into_iter() + .map(|h| h as HANDLE) + .collect(); + + if handles.is_empty() { + return Err(vm.new_value_error("handle_seq must not be empty".to_owned())); + } + + if handles.len() > 64 { + return Err( + vm.new_value_error("WaitForMultipleObjects supports at most 64 handles".to_owned()) + ); + } + + let ret = unsafe { + WinWaitForMultipleObjects( + handles.len() as u32, + handles.as_ptr(), + if wait_all { 1 } else { 0 }, + milliseconds, + ) + }; + + if ret == WAIT_FAILED { + Err(vm.new_last_os_error()) + } else { + Ok(ret) + } + } + #[pyfunction] fn GetExitCodeProcess(h: WinHandle, vm: &VirtualMachine) -> PyResult<u32> { unsafe { @@ -679,6 +805,229 @@ mod _winapi { Ok(WinHandle(handle)) } + // ==================== Overlapped class ==================== + // Used for asynchronous I/O operations (ConnectNamedPipe, ReadFile, WriteFile) + + #[pyattr] + #[pyclass(name = "Overlapped", module = "_winapi")] + #[derive(Debug, PyPayload)] + struct Overlapped { + inner: PyMutex<OverlappedInner>, + } + + struct OverlappedInner { + overlapped: windows_sys::Win32::System::IO::OVERLAPPED, + handle: HANDLE, + pending: bool, + completed: bool, + read_buffer: Option<Vec<u8>>, + } + + impl std::fmt::Debug for OverlappedInner { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("OverlappedInner") + .field("handle", &self.handle) + .field("pending", &self.pending) + .field("completed", &self.completed) + .finish() + } + } + + unsafe impl Sync for OverlappedInner {} + unsafe impl Send for OverlappedInner {} + + #[pyclass(with(Constructor))] + impl Overlapped { + fn new_with_handle(handle: HANDLE) -> Self { + use windows_sys::Win32::System::Threading::CreateEventW; + + let event = unsafe { CreateEventW(null(), 1, 0, null()) }; + let mut overlapped: windows_sys::Win32::System::IO::OVERLAPPED = + unsafe { std::mem::zeroed() }; + overlapped.hEvent = event; + + Overlapped { + inner: PyMutex::new(OverlappedInner { + overlapped, + handle, + pending: false, + completed: false, + read_buffer: None, + }), + } + } + + #[pymethod] + fn GetOverlappedResult(&self, wait: bool, vm: &VirtualMachine) -> PyResult<u32> { + use windows_sys::Win32::Foundation::{ERROR_IO_PENDING, GetLastError}; + use windows_sys::Win32::System::IO::GetOverlappedResult; + + let mut inner = self.inner.lock(); + + // If operation was already completed synchronously (e.g., ERROR_PIPE_CONNECTED), + // return immediately without calling GetOverlappedResult + if inner.completed && !inner.pending { + return Ok(0); + } + + let mut transferred: u32 = 0; + + let ret = unsafe { + GetOverlappedResult( + inner.handle, + &inner.overlapped, + &mut transferred, + if wait { 1 } else { 0 }, + ) + }; + + if ret == 0 { + let err = unsafe { GetLastError() }; + if err == ERROR_IO_PENDING { + return Err(std::io::Error::from_raw_os_error(err as i32).to_pyexception(vm)); + } + return Err(std::io::Error::from_raw_os_error(err as i32).to_pyexception(vm)); + } + + inner.completed = true; + inner.pending = false; + Ok(transferred) + } + + #[pymethod] + fn getbuffer(&self, vm: &VirtualMachine) -> PyResult<Option<PyObjectRef>> { + let inner = self.inner.lock(); + if !inner.completed { + return Err(vm.new_value_error("operation not completed".to_owned())); + } + Ok(inner + .read_buffer + .as_ref() + .map(|buf| vm.ctx.new_bytes(buf.clone()).into())) + } + + #[pymethod] + fn cancel(&self, vm: &VirtualMachine) -> PyResult<()> { + use windows_sys::Win32::System::IO::CancelIoEx; + + let inner = self.inner.lock(); + if !inner.pending { + return Ok(()); + } + + let ret = unsafe { CancelIoEx(inner.handle, &inner.overlapped) }; + if ret == 0 { + let err = unsafe { windows_sys::Win32::Foundation::GetLastError() }; + // ERROR_NOT_FOUND means operation already completed + if err != windows_sys::Win32::Foundation::ERROR_NOT_FOUND { + return Err(std::io::Error::from_raw_os_error(err as i32).to_pyexception(vm)); + } + } + Ok(()) + } + + #[pygetset] + fn event(&self) -> isize { + let inner = self.inner.lock(); + inner.overlapped.hEvent as isize + } + } + + impl Constructor for Overlapped { + type Args = (); + + fn py_new( + _cls: &Py<crate::builtins::PyType>, + _args: Self::Args, + _vm: &VirtualMachine, + ) -> PyResult<Self> { + Ok(Overlapped::new_with_handle(null_mut())) + } + } + + impl Drop for OverlappedInner { + fn drop(&mut self) { + use windows_sys::Win32::Foundation::CloseHandle; + if !self.overlapped.hEvent.is_null() { + unsafe { CloseHandle(self.overlapped.hEvent) }; + } + } + } + + /// ConnectNamedPipe - Wait for a client to connect to a named pipe + #[derive(FromArgs)] + struct ConnectNamedPipeArgs { + #[pyarg(positional)] + handle: WinHandle, + #[pyarg(named, optional)] + overlapped: OptionalArg<bool>, + } + + #[pyfunction] + fn ConnectNamedPipe(args: ConnectNamedPipeArgs, vm: &VirtualMachine) -> PyResult<PyObjectRef> { + use windows_sys::Win32::Foundation::{ + ERROR_IO_PENDING, ERROR_PIPE_CONNECTED, GetLastError, + }; + + let handle = args.handle; + let use_overlapped = args.overlapped.unwrap_or(false); + + if use_overlapped { + // Overlapped (async) mode + let ov = Overlapped::new_with_handle(handle.0); + + let ret = { + let mut inner = ov.inner.lock(); + unsafe { + windows_sys::Win32::System::Pipes::ConnectNamedPipe( + handle.0, + &mut inner.overlapped, + ) + } + }; + + if ret != 0 { + // Connected immediately + let mut inner = ov.inner.lock(); + inner.completed = true; + } else { + let err = unsafe { GetLastError() }; + match err { + ERROR_IO_PENDING => { + let mut inner = ov.inner.lock(); + inner.pending = true; + } + ERROR_PIPE_CONNECTED => { + // Client was already connected + let mut inner = ov.inner.lock(); + inner.completed = true; + } + _ => { + return Err( + std::io::Error::from_raw_os_error(err as i32).to_pyexception(vm) + ); + } + } + } + + Ok(ov.into_pyobject(vm)) + } else { + // Synchronous mode + let ret = unsafe { + windows_sys::Win32::System::Pipes::ConnectNamedPipe(handle.0, null_mut()) + }; + + if ret == 0 { + let err = unsafe { GetLastError() }; + if err != ERROR_PIPE_CONNECTED { + return Err(std::io::Error::from_raw_os_error(err as i32).to_pyexception(vm)); + } + } + + Ok(vm.ctx.none()) + } + } + /// Helper for GetShortPathName and GetLongPathName fn get_path_name_impl( path: &PyStrRef, diff --git a/crates/vm/src/stdlib/winreg.rs b/crates/vm/src/stdlib/winreg.rs index 28fa2e9b74c..53a4fd0d556 100644 --- a/crates/vm/src/stdlib/winreg.rs +++ b/crates/vm/src/stdlib/winreg.rs @@ -1,11 +1,7 @@ // spell-checker:disable #![allow(non_snake_case)] -use crate::{PyRef, VirtualMachine, builtins::PyModule}; - -pub(crate) fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { - winreg::make_module(vm) -} +pub(crate) use winreg::module_def; #[pymodule] mod winreg { diff --git a/crates/vm/src/types/slot.rs b/crates/vm/src/types/slot.rs index b637bfc40b6..2f45d9dcff1 100644 --- a/crates/vm/src/types/slot.rs +++ b/crates/vm/src/types/slot.rs @@ -483,7 +483,7 @@ fn hash_wrapper(zelf: &PyObject, vm: &VirtualMachine) -> PyResult<PyHash> { /// Marks a type as unhashable. Similar to PyObject_HashNotImplemented in CPython pub fn hash_not_implemented(zelf: &PyObject, vm: &VirtualMachine) -> PyResult<PyHash> { - Err(vm.new_type_error(format!("unhashable type: {}", zelf.class().name()))) + Err(vm.new_type_error(format!("unhashable type: '{}'", zelf.class().name()))) } fn call_wrapper(zelf: &PyObject, args: FuncArgs, vm: &VirtualMachine) -> PyResult { diff --git a/crates/vm/src/types/zoo.rs b/crates/vm/src/types/zoo.rs index c4aa7258c07..9fa4e777a1a 100644 --- a/crates/vm/src/types/zoo.rs +++ b/crates/vm/src/types/zoo.rs @@ -3,9 +3,9 @@ use crate::{ builtins::{ asyncgenerator, bool_, builtin_func, bytearray, bytes, classmethod, code, complex, coroutine, descriptor, dict, enumerate, filter, float, frame, function, generator, - genericalias, getset, int, iter, list, map, mappingproxy, memory, module, namespace, - object, property, pystr, range, set, singletons, slice, staticmethod, super_, traceback, - tuple, + genericalias, getset, int, interpolation, iter, list, map, mappingproxy, memory, module, + namespace, object, property, pystr, range, set, singletons, slice, staticmethod, super_, + template, traceback, tuple, type_::{self, PyType}, union_, weakproxy, weakref, zip, }, @@ -93,6 +93,9 @@ pub struct TypeZoo { pub not_implemented_type: &'static Py<PyType>, pub generic_alias_type: &'static Py<PyType>, pub union_type: &'static Py<PyType>, + pub interpolation_type: &'static Py<PyType>, + pub template_type: &'static Py<PyType>, + pub template_iter_type: &'static Py<PyType>, pub member_descriptor_type: &'static Py<PyType>, pub wrapper_descriptor_type: &'static Py<PyType>, pub method_wrapper_type: &'static Py<PyType>, @@ -188,6 +191,9 @@ impl TypeZoo { not_implemented_type: singletons::PyNotImplemented::init_builtin_type(), generic_alias_type: genericalias::PyGenericAlias::init_builtin_type(), union_type: union_::PyUnion::init_builtin_type(), + interpolation_type: interpolation::PyInterpolation::init_builtin_type(), + template_type: template::PyTemplate::init_builtin_type(), + template_iter_type: template::PyTemplateIter::init_builtin_type(), member_descriptor_type: descriptor::PyMemberDescriptor::init_builtin_type(), wrapper_descriptor_type: descriptor::PyWrapper::init_builtin_type(), method_wrapper_type: descriptor::PyMethodWrapper::init_builtin_type(), @@ -243,6 +249,8 @@ impl TypeZoo { traceback::init(context); genericalias::init(context); union_::init(context); + interpolation::init(context); + template::init(context); descriptor::init(context); crate::stdlib::typing::init(context); } diff --git a/crates/vm/src/version.rs b/crates/vm/src/version.rs index 2b5e82a8d76..a75a6f47de6 100644 --- a/crates/vm/src/version.rs +++ b/crates/vm/src/version.rs @@ -4,9 +4,9 @@ use chrono::{Local, prelude::DateTime}; use core::time::Duration; use std::time::UNIX_EPOCH; -// = 3.13.0alpha +// = 3.14.0alpha pub const MAJOR: usize = 3; -pub const MINOR: usize = 13; +pub const MINOR: usize = 14; pub const MICRO: usize = 0; pub const RELEASELEVEL: &str = "alpha"; pub const RELEASELEVEL_N: usize = 0xA; @@ -129,7 +129,7 @@ pub fn get_git_datetime() -> String { } // Must be aligned to Lib/importlib/_bootstrap_external.py -pub const PYC_MAGIC_NUMBER: u16 = 2997; +pub const PYC_MAGIC_NUMBER: u16 = 2996; // CPython format: magic_number | ('\r' << 16) | ('\n' << 24) // This protects against text-mode file reads diff --git a/crates/vm/src/vm/context.rs b/crates/vm/src/vm/context.rs index b12352f6eee..34ad66c53c0 100644 --- a/crates/vm/src/vm/context.rs +++ b/crates/vm/src/vm/context.rs @@ -51,6 +51,10 @@ pub struct Context { pub(crate) string_pool: StringPool, pub(crate) slot_new_wrapper: PyMethodDef, pub names: ConstName, + + // GC module state (callbacks and garbage lists) + pub gc_callbacks: PyListRef, + pub gc_garbage: PyListRef, } macro_rules! declare_const_name { @@ -91,7 +95,10 @@ declare_const_name! { __all__, __and__, __anext__, + __annotate__, + __annotate_func__, __annotations__, + __annotations_cache__, __args__, __await__, __bases__, @@ -135,6 +142,7 @@ declare_const_name! { __getformat__, __getitem__, __getnewargs__, + __getnewargs_ex__, __getstate__, __gt__, __hash__, @@ -230,6 +238,7 @@ declare_const_name! { __typing_is_unpacked_typevartuple__, __typing_prepare_subst__, __typing_unpacked_tuple_args__, + __weakref__, __xor__, // common names @@ -328,6 +337,11 @@ impl Context { let empty_str = unsafe { string_pool.intern("", types.str_type.to_owned()) }; let empty_bytes = create_object(PyBytes::from(Vec::new()), types.bytes_type); + + // GC callbacks and garbage lists + let gc_callbacks = PyRef::new_ref(PyList::default(), types.list_type.to_owned(), None); + let gc_garbage = PyRef::new_ref(PyList::default(), types.list_type.to_owned(), None); + Self { true_value, false_value, @@ -347,6 +361,9 @@ impl Context { string_pool, slot_new_wrapper, names, + + gc_callbacks, + gc_garbage, } } diff --git a/crates/vm/src/vm/interpreter.rs b/crates/vm/src/vm/interpreter.rs index 8d37ad6c840..9fcc11d7f42 100644 --- a/crates/vm/src/vm/interpreter.rs +++ b/crates/vm/src/vm/interpreter.rs @@ -1,7 +1,276 @@ -use super::{Context, PyConfig, VirtualMachine, setting::Settings, thread}; -use crate::{PyResult, getpath, stdlib::atexit, vm::PyBaseExceptionRef}; +use super::{Context, PyConfig, PyGlobalState, VirtualMachine, setting::Settings, thread}; +use crate::{ + PyResult, builtins, common::rc::PyRc, frozen::FrozenModule, getpath, py_freeze, stdlib::atexit, + vm::PyBaseExceptionRef, +}; use core::sync::atomic::Ordering; +type InitFunc = Box<dyn FnOnce(&mut VirtualMachine)>; + +/// Configuration builder for constructing an Interpreter. +/// +/// This is the preferred way to configure and create an interpreter with custom modules. +/// Modules must be registered before the interpreter is built, +/// similar to CPython's `PyImport_AppendInittab` which must be called before `Py_Initialize`. +/// +/// # Example +/// ``` +/// use rustpython_vm::Interpreter; +/// +/// let builder = Interpreter::builder(Default::default()); +/// // In practice, add stdlib: builder.add_native_modules(&stdlib_module_defs(&builder.ctx)) +/// let interp = builder.build(); +/// ``` +pub struct InterpreterBuilder { + settings: Settings, + pub ctx: PyRc<Context>, + module_defs: Vec<&'static builtins::PyModuleDef>, + frozen_modules: Vec<(&'static str, FrozenModule)>, + init_hooks: Vec<InitFunc>, +} + +/// Private helper to initialize a VM with settings, context, and custom initialization. +fn initialize_main_vm<F>( + settings: Settings, + ctx: PyRc<Context>, + module_defs: Vec<&'static builtins::PyModuleDef>, + frozen_modules: Vec<(&'static str, FrozenModule)>, + init_hooks: Vec<InitFunc>, + init: F, +) -> (VirtualMachine, PyRc<PyGlobalState>) +where + F: FnOnce(&mut VirtualMachine), +{ + use crate::codecs::CodecsRegistry; + use crate::common::hash::HashSecret; + use crate::common::lock::PyMutex; + use crate::warn::WarningsState; + use crossbeam_utils::atomic::AtomicCell; + use std::sync::atomic::AtomicBool; + + let paths = getpath::init_path_config(&settings); + let config = PyConfig::new(settings, paths); + + crate::types::TypeZoo::extend(&ctx); + crate::exceptions::ExceptionZoo::extend(&ctx); + + // Build module_defs map from builtin modules + additional modules + let mut all_module_defs: std::collections::BTreeMap< + &'static str, + &'static builtins::PyModuleDef, + > = crate::stdlib::builtin_module_defs(&ctx) + .into_iter() + .chain(module_defs) + .map(|def| (def.name.as_str(), def)) + .collect(); + + // Register sysconfigdata under platform-specific name as well + if let Some(&sysconfigdata_def) = all_module_defs.get("_sysconfigdata") { + use std::sync::OnceLock; + static SYSCONFIGDATA_NAME: OnceLock<&'static str> = OnceLock::new(); + let leaked_name = *SYSCONFIGDATA_NAME.get_or_init(|| { + let name = crate::stdlib::sys::sysconfigdata_name(); + Box::leak(name.into_boxed_str()) + }); + all_module_defs.insert(leaked_name, sysconfigdata_def); + } + + // Create hash secret + let seed = match config.settings.hash_seed { + Some(seed) => seed, + None => super::process_hash_secret_seed(), + }; + let hash_secret = HashSecret::new(seed); + + // Create codec registry and warnings state + let codec_registry = CodecsRegistry::new(&ctx); + let warnings = WarningsState::init_state(&ctx); + + // Create int_max_str_digits + let int_max_str_digits = AtomicCell::new(match config.settings.int_max_str_digits { + -1 => 4300, + other => other, + } as usize); + + // Initialize frozen modules (core + user-provided) + let mut frozen: std::collections::HashMap<&'static str, FrozenModule, ahash::RandomState> = + core_frozen_inits().collect(); + frozen.extend(frozen_modules); + + // Create PyGlobalState + let global_state = PyRc::new(PyGlobalState { + config, + module_defs: all_module_defs, + frozen, + stacksize: AtomicCell::new(0), + thread_count: AtomicCell::new(0), + hash_secret, + atexit_funcs: PyMutex::default(), + codec_registry, + finalizing: AtomicBool::new(false), + warnings, + override_frozen_modules: AtomicCell::new(0), + before_forkers: PyMutex::default(), + after_forkers_child: PyMutex::default(), + after_forkers_parent: PyMutex::default(), + int_max_str_digits, + switch_interval: AtomicCell::new(0.005), + global_trace_func: PyMutex::default(), + global_profile_func: PyMutex::default(), + #[cfg(feature = "threading")] + main_thread_ident: AtomicCell::new(0), + #[cfg(feature = "threading")] + thread_frames: parking_lot::Mutex::new(std::collections::HashMap::new()), + #[cfg(feature = "threading")] + thread_handles: parking_lot::Mutex::new(Vec::new()), + #[cfg(feature = "threading")] + shutdown_handles: parking_lot::Mutex::new(Vec::new()), + }); + + // Create VM with the global state + // Note: Don't clone here - init_hooks need exclusive access to mutate state + let mut vm = VirtualMachine::new(ctx, global_state); + + // Execute initialization hooks (can mutate vm.state) + for hook in init_hooks { + hook(&mut vm); + } + + // Call custom init function (can mutate vm.state) + init(&mut vm); + + vm.initialize(); + + // Clone global_state for Interpreter after all initialization is done + let global_state = vm.state.clone(); + (vm, global_state) +} + +impl InterpreterBuilder { + /// Create a new interpreter configuration with default settings. + pub fn new() -> Self { + Self { + settings: Settings::default(), + ctx: Context::genesis().clone(), + module_defs: Vec::new(), + frozen_modules: Vec::new(), + init_hooks: Vec::new(), + } + } + + /// Set custom settings for the interpreter. + /// + /// If called multiple times, only the last settings will be used. + pub fn settings(mut self, settings: Settings) -> Self { + self.settings = settings; + self + } + + /// Add a single native module definition. + /// + /// # Example + /// ``` + /// use rustpython_vm::{Interpreter, builtins::PyModuleDef}; + /// + /// let builder = Interpreter::builder(Default::default()); + /// // Note: In practice, use module_def from your #[pymodule] + /// // let def = mymodule::module_def(&builder.ctx); + /// // let interp = builder.add_native_module(def).build(); + /// let interp = builder.build(); + /// ``` + pub fn add_native_module(self, def: &'static builtins::PyModuleDef) -> Self { + self.add_native_modules(&[def]) + } + + /// Add multiple native module definitions. + /// + /// # Example + /// ``` + /// use rustpython_vm::Interpreter; + /// + /// let builder = Interpreter::builder(Default::default()); + /// // In practice, use module_defs from rustpython_stdlib: + /// // let defs = rustpython_stdlib::stdlib_module_defs(&builder.ctx); + /// // let interp = builder.add_native_modules(&defs).build(); + /// let interp = builder.build(); + /// ``` + pub fn add_native_modules(mut self, defs: &[&'static builtins::PyModuleDef]) -> Self { + self.module_defs.extend_from_slice(defs); + self + } + + /// Add a custom initialization hook. + /// + /// Hooks are executed in the order they are added during interpreter creation. + /// This function will be called after modules are registered but before + /// the VM is initialized, allowing for additional customization. + /// + /// # Example + /// ``` + /// use rustpython_vm::Interpreter; + /// + /// let interp = Interpreter::builder(Default::default()) + /// .init_hook(|vm| { + /// // Custom initialization + /// }) + /// .build(); + /// ``` + pub fn init_hook<F>(mut self, init: F) -> Self + where + F: FnOnce(&mut VirtualMachine) + 'static, + { + self.init_hooks.push(Box::new(init)); + self + } + + /// Add frozen modules to the interpreter. + /// + /// Frozen modules are Python modules compiled into the binary. + /// This method accepts any iterator of (name, FrozenModule) pairs. + /// + /// # Example + /// ``` + /// use rustpython_vm::Interpreter; + /// + /// let interp = Interpreter::builder(Default::default()) + /// // In practice: .add_frozen_modules(rustpython_pylib::FROZEN_STDLIB) + /// .build(); + /// ``` + pub fn add_frozen_modules<I>(mut self, frozen: I) -> Self + where + I: IntoIterator<Item = (&'static str, FrozenModule)>, + { + self.frozen_modules.extend(frozen); + self + } + + /// Build the interpreter. + /// + /// This consumes the configuration and returns a fully initialized Interpreter. + pub fn build(self) -> Interpreter { + let (vm, global_state) = initialize_main_vm( + self.settings, + self.ctx, + self.module_defs, + self.frozen_modules, + self.init_hooks, + |_| {}, // No additional init needed + ); + Interpreter { global_state, vm } + } + + /// Alias for `build()` for compatibility with the `interpreter()` pattern. + pub fn interpreter(self) -> Interpreter { + self.build() + } +} + +impl Default for InterpreterBuilder { + fn default() -> Self { + Self::new() + } +} + /// The general interface for the VM /// /// # Examples @@ -21,43 +290,49 @@ use core::sync::atomic::Ordering; /// }); /// ``` pub struct Interpreter { + pub global_state: PyRc<PyGlobalState>, vm: VirtualMachine, } impl Interpreter { + /// Create a new interpreter configuration builder. + /// + /// # Example + /// ``` + /// use rustpython_vm::Interpreter; + /// + /// let builder = Interpreter::builder(Default::default()); + /// // In practice, add stdlib: builder.add_native_modules(&stdlib_module_defs(&builder.ctx)) + /// let interp = builder.build(); + /// ``` + pub fn builder(settings: Settings) -> InterpreterBuilder { + InterpreterBuilder::new().settings(settings) + } + /// This is a bare unit to build up an interpreter without the standard library. - /// To create an interpreter with the standard library with the `rustpython` crate, use `rustpython::InterpreterConfig`. + /// To create an interpreter with the standard library with the `rustpython` crate, use `rustpython::InterpreterBuilder`. /// To create an interpreter without the `rustpython` crate, but only with `rustpython-vm`, - /// try to build one from the source code of `InterpreterConfig`. It will not be a one-liner but it also will not be too hard. + /// try to build one from the source code of `InterpreterBuilder`. It will not be a one-liner but it also will not be too hard. pub fn without_stdlib(settings: Settings) -> Self { Self::with_init(settings, |_| {}) } /// Create with initialize function taking mutable vm reference. - /// ``` - /// use rustpython_vm::Interpreter; - /// Interpreter::with_init(Default::default(), |vm| { - /// // put this line to add stdlib to the vm - /// // vm.add_native_modules(rustpython_stdlib::get_module_inits()); - /// }).enter(|vm| { - /// vm.run_code_string(vm.new_scope_with_builtins(), "print(1)", "<...>".to_owned()); - /// }); - /// ``` + /// + /// Note: This is a legacy API. To add stdlib, use `Interpreter::builder()` instead. pub fn with_init<F>(settings: Settings, init: F) -> Self where F: FnOnce(&mut VirtualMachine), { - // Compute path configuration from settings - let paths = getpath::init_path_config(&settings); - let config = PyConfig::new(settings, paths); - - let ctx = Context::genesis(); - crate::types::TypeZoo::extend(ctx); - crate::exceptions::ExceptionZoo::extend(ctx); - let mut vm = VirtualMachine::new(config, ctx.clone()); - init(&mut vm); - vm.initialize(); - Self { vm } + let (vm, global_state) = initialize_main_vm( + settings, + Context::genesis().clone(), + Vec::new(), // No module_defs + Vec::new(), // No frozen_modules + Vec::new(), // No init_hooks + init, + ); + Self { global_state, vm } } /// Run a function with the main virtual machine and return a PyResult of the result. @@ -110,10 +385,13 @@ impl Interpreter { /// Finalize vm and turns an exception to exit code. /// - /// Finalization steps including 4 steps: + /// Finalization steps (matching Py_FinalizeEx): /// 1. Flush stdout and stderr. /// 1. Handle exit exception and turn it to exit code. + /// 1. Wait for thread shutdown (call threading._shutdown). + /// 1. Mark vm as finalizing. /// 1. Run atexit exit functions. + /// 1. Finalize modules (clear module dicts in reverse import order). /// 1. Mark vm as finalized. /// /// Note that calling `finalize` is not necessary by purpose though. @@ -128,10 +406,29 @@ impl Interpreter { 0 }; - atexit::_run_exitfuncs(vm); + // Wait for thread shutdown - call threading._shutdown() if available. + // This waits for all non-daemon threads to complete. + // threading module may not be imported, so ignore import errors. + if let Ok(threading) = vm.import("threading", 0) + && let Ok(shutdown) = threading.get_attr("_shutdown", vm) + && let Err(e) = shutdown.call((), vm) + { + vm.run_unraisable( + e, + Some("Exception ignored in threading shutdown".to_owned()), + threading, + ); + } + // Mark as finalizing AFTER thread shutdown vm.state.finalizing.store(true, Ordering::Release); + // Run atexit exit functions + atexit::_run_exitfuncs(vm); + + // Finalize modules: clear module dicts in reverse import order + vm.finalize_modules(); + vm.flush_std(); exit_code @@ -139,6 +436,44 @@ impl Interpreter { } } +fn core_frozen_inits() -> impl Iterator<Item = (&'static str, FrozenModule)> { + let iter = core::iter::empty(); + macro_rules! ext_modules { + ($iter:ident, $($t:tt)*) => { + let $iter = $iter.chain(py_freeze!($($t)*)); + }; + } + + // keep as example but use file one now + // ext_modules!( + // iter, + // source = "initialized = True; print(\"Hello world!\")\n", + // module_name = "__hello__", + // ); + + // Python modules that the vm calls into, but are not actually part of the stdlib. They could + // in theory be implemented in Rust, but are easiest to do in Python for one reason or another. + // Includes _importlib_bootstrap and _importlib_bootstrap_external + ext_modules!( + iter, + dir = "../../Lib/python_builtins", + crate_name = "rustpython_compiler_core" + ); + + // core stdlib Python modules that the vm calls into, but are still used in Python + // application code, e.g. copyreg + // FIXME: Initializing core_modules here results duplicated frozen module generation for core_modules. + // We need a way to initialize this modules for both `Interpreter::without_stdlib()` and `InterpreterBuilder::new().init_stdlib().interpreter()` + // #[cfg(not(feature = "freeze-stdlib"))] + ext_modules!( + iter, + dir = "../../Lib/core_modules", + crate_name = "rustpython_compiler_core" + ); + + iter +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/vm/src/vm/method.rs b/crates/vm/src/vm/method.rs index ba323391488..5f47c8b8c5b 100644 --- a/crates/vm/src/vm/method.rs +++ b/crates/vm/src/vm/method.rs @@ -3,8 +3,8 @@ use super::VirtualMachine; use crate::{ - builtins::{PyBaseObject, PyStr, PyStrInterned}, - function::IntoFuncArgs, + builtins::{PyBaseObject, PyStr, PyStrInterned, descriptor::PyMethodDescriptor}, + function::{IntoFuncArgs, PyMethodFlags}, object::{AsObject, Py, PyObject, PyObjectRef, PyResult}, types::PyTypeFlags, }; @@ -22,7 +22,7 @@ impl PyMethod { pub fn get(obj: PyObjectRef, name: &Py<PyStr>, vm: &VirtualMachine) -> PyResult<Self> { let cls = obj.class(); let getattro = cls.slots.getattro.load().unwrap(); - if getattro as usize != PyBaseObject::getattro as usize { + if getattro as usize != PyBaseObject::getattro as *const () as usize { return obj.get_attr(name, vm).map(Self::Attribute); } @@ -38,8 +38,15 @@ impl PyMethod { .flags .has_feature(PyTypeFlags::METHOD_DESCRIPTOR) { - is_method = true; - None + // For classmethods, we need descr_get to convert instance to class + if let Some(method_descr) = descr.downcast_ref::<PyMethodDescriptor>() + && method_descr.method.flags.contains(PyMethodFlags::CLASS) + { + descr_cls.slots.descr_get.load() + } else { + is_method = true; + None + } } else { let descr_get = descr_cls.slots.descr_get.load(); if let Some(descr_get) = descr_get diff --git a/crates/vm/src/vm/mod.rs b/crates/vm/src/vm/mod.rs index d2172a43a00..48b5655a9eb 100644 --- a/crates/vm/src/vm/mod.rs +++ b/crates/vm/src/vm/mod.rs @@ -19,8 +19,12 @@ mod vm_ops; use crate::{ AsObject, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, builtins::{ - PyBaseExceptionRef, PyDict, PyDictRef, PyInt, PyList, PyModule, PyStr, PyStrInterned, - PyStrRef, PyTypeRef, code::PyCode, pystr::AsPyStr, tuple::PyTuple, + self, PyBaseExceptionRef, PyDict, PyDictRef, PyInt, PyList, PyModule, PyStr, PyStrInterned, + PyStrRef, PyTypeRef, PyWeak, + code::PyCode, + dict::{PyDictItems, PyDictKeys, PyDictValues}, + pystr::AsPyStr, + tuple::PyTuple, }, codecs::CodecsRegistry, common::{hash::HashSecret, lock::PyMutex, rc::PyRc}, @@ -52,7 +56,7 @@ use std::{ }; pub use context::Context; -pub use interpreter::Interpreter; +pub use interpreter::{Interpreter, InterpreterBuilder}; pub(crate) use method::PyMethod; pub use setting::{CheckHashPycsMode, Paths, PyConfig, Settings}; @@ -82,6 +86,16 @@ pub struct VirtualMachine { pub state: PyRc<PyGlobalState>, pub initialized: bool, recursion_depth: Cell<usize>, + /// C stack soft limit for detecting stack overflow (like c_stack_soft_limit) + c_stack_soft_limit: Cell<usize>, + /// Async generator firstiter hook (per-thread, set via sys.set_asyncgen_hooks) + pub async_gen_firstiter: RefCell<Option<PyObjectRef>>, + /// Async generator finalizer hook (per-thread, set via sys.set_asyncgen_hooks) + pub async_gen_finalizer: RefCell<Option<PyObjectRef>>, + /// Current running asyncio event loop for this thread + pub asyncio_running_loop: RefCell<Option<PyObjectRef>>, + /// Current running asyncio task for this thread + pub asyncio_running_task: RefCell<Option<PyObjectRef>>, } #[derive(Debug, Default)] @@ -92,7 +106,7 @@ struct ExceptionStack { pub struct PyGlobalState { pub config: PyConfig, - pub module_inits: stdlib::StdlibMap, + pub module_defs: std::collections::BTreeMap<&'static str, &'static builtins::PyModuleDef>, pub frozen: HashMap<&'static str, FrozenModule, ahash::RandomState>, pub stacksize: AtomicCell<usize>, pub thread_count: AtomicCell<usize>, @@ -107,6 +121,22 @@ pub struct PyGlobalState { pub after_forkers_parent: PyMutex<Vec<PyObjectRef>>, pub int_max_str_digits: AtomicCell<usize>, pub switch_interval: AtomicCell<f64>, + /// Global trace function for all threads (set by sys._settraceallthreads) + pub global_trace_func: PyMutex<Option<PyObjectRef>>, + /// Global profile function for all threads (set by sys._setprofileallthreads) + pub global_profile_func: PyMutex<Option<PyObjectRef>>, + /// Main thread identifier (pthread_self on Unix) + #[cfg(feature = "threading")] + pub main_thread_ident: AtomicCell<u64>, + /// Registry of all threads' current frames for sys._current_frames() + #[cfg(feature = "threading")] + pub thread_frames: parking_lot::Mutex<HashMap<u64, stdlib::thread::CurrentFrameSlot>>, + /// Registry of all ThreadHandles for fork cleanup + #[cfg(feature = "threading")] + pub thread_handles: parking_lot::Mutex<Vec<stdlib::thread::HandleEntry>>, + /// Registry for non-daemon threads that need to be joined at shutdown + #[cfg(feature = "threading")] + pub shutdown_handles: parking_lot::Mutex<Vec<stdlib::thread::ShutdownEntry>>, } pub fn process_hash_secret_seed() -> u32 { @@ -118,7 +148,7 @@ pub fn process_hash_secret_seed() -> u32 { impl VirtualMachine { /// Create a new `VirtualMachine` structure. - fn new(config: PyConfig, ctx: PyRc<Context>) -> Self { + pub(crate) fn new(ctx: PyRc<Context>, state: PyRc<PyGlobalState>) -> Self { flame_guard!("new VirtualMachine"); // make a new module without access to the vm; doesn't @@ -132,8 +162,8 @@ impl VirtualMachine { }; // Hard-core modules: - let builtins = new_module(stdlib::builtins::__module_def(&ctx)); - let sys_module = new_module(stdlib::sys::__module_def(&ctx)); + let builtins = new_module(stdlib::builtins::module_def(&ctx)); + let sys_module = new_module(stdlib::sys::module_def(&ctx)); let import_func = ctx.none(); let profile_func = RefCell::new(ctx.none()); @@ -143,23 +173,7 @@ impl VirtualMachine { const { RefCell::new([const { None }; signal::NSIG]) }, )); - let module_inits = stdlib::get_module_inits(); - - let seed = match config.settings.hash_seed { - Some(seed) => seed, - None => process_hash_secret_seed(), - }; - let hash_secret = HashSecret::new(seed); - - let codec_registry = CodecsRegistry::new(&ctx); - - let warnings = WarningsState::init_state(&ctx); - - let int_max_str_digits = AtomicCell::new(match config.settings.int_max_str_digits { - -1 => 4300, - other => other, - } as usize); - let mut vm = Self { + let vm = Self { builtins, sys_module, ctx, @@ -174,26 +188,14 @@ impl VirtualMachine { signal_handlers, signal_rx: None, repr_guards: RefCell::default(), - state: PyRc::new(PyGlobalState { - config, - module_inits, - frozen: HashMap::default(), - stacksize: AtomicCell::new(0), - thread_count: AtomicCell::new(0), - hash_secret, - atexit_funcs: PyMutex::default(), - codec_registry, - finalizing: AtomicBool::new(false), - warnings, - override_frozen_modules: AtomicCell::new(0), - before_forkers: PyMutex::default(), - after_forkers_child: PyMutex::default(), - after_forkers_parent: PyMutex::default(), - int_max_str_digits, - switch_interval: AtomicCell::new(0.005), - }), + state, initialized: false, recursion_depth: Cell::new(0), + c_stack_soft_limit: Cell::new(Self::calculate_c_stack_soft_limit()), + async_gen_firstiter: RefCell::new(None), + async_gen_finalizer: RefCell::new(None), + asyncio_running_loop: RefCell::new(None), + asyncio_running_task: RefCell::new(None), }; if vm.state.hash_secret.hash_str("") @@ -206,9 +208,6 @@ impl VirtualMachine { panic!("Interpreters in same process must share the hash seed"); } - let frozen = core_frozen_inits().collect(); - PyRc::get_mut(&mut vm.state).unwrap().frozen = frozen; - vm.builtins.init_dict( vm.ctx.intern_str("builtins"), Some(vm.ctx.intern_str(stdlib::builtins::DOC.unwrap()).to_owned()), @@ -236,7 +235,7 @@ impl VirtualMachine { }); let guide_message = if cfg!(feature = "freeze-stdlib") { - "`rustpython_pylib` maybe not set while using `freeze-stdlib` feature. Try using `rustpython::InterpreterConfig::init_stdlib` or manually call `vm.add_frozen(rustpython_pylib::FROZEN_STDLIB)` in `rustpython_vm::Interpreter::with_init`." + "`rustpython_pylib` may not be set while using `freeze-stdlib` feature. Try using `rustpython::InterpreterBuilder::init_stdlib` or manually call `builder.add_frozen_modules(rustpython_pylib::FROZEN_STDLIB)` in `rustpython_vm::Interpreter::builder()`." } else if !env_set { "Neither RUSTPYTHONPATH nor PYTHONPATH is set. Try setting one of them to the stdlib directory." } else if path_contains_env { @@ -299,6 +298,10 @@ impl VirtualMachine { panic!("Double Initialize Error"); } + // Initialize main thread ident before any threading operations + #[cfg(feature = "threading")] + stdlib::thread::init_main_thread_ident(self); + stdlib::builtins::init_module(self, &self.builtins); stdlib::sys::init_module(self, &self.sys_module, &self.builtins); @@ -325,6 +328,7 @@ impl VirtualMachine { Some(if write { "wb" } else { "rb" }), crate::stdlib::io::OpenArgs { buffering: if unbuffered { 0 } else { -1 }, + closefd: false, ..Default::default() }, self, @@ -427,34 +431,6 @@ impl VirtualMachine { self.initialized = true; } - fn state_mut(&mut self) -> &mut PyGlobalState { - PyRc::get_mut(&mut self.state) - .expect("there should not be multiple threads while a user has a mut ref to a vm") - } - - /// Can only be used in the initialization closure passed to [`Interpreter::with_init`] - pub fn add_native_module<S>(&mut self, name: S, module: stdlib::StdlibInitFunc) - where - S: Into<Cow<'static, str>>, - { - self.state_mut().module_inits.insert(name.into(), module); - } - - pub fn add_native_modules<I>(&mut self, iter: I) - where - I: IntoIterator<Item = (Cow<'static, str>, stdlib::StdlibInitFunc)>, - { - self.state_mut().module_inits.extend(iter); - } - - /// Can only be used in the initialization closure passed to [`Interpreter::with_init`] - pub fn add_frozen<I>(&mut self, frozen: I) - where - I: IntoIterator<Item = (&'static str, FrozenModule)>, - { - self.state_mut().frozen.extend(frozen); - } - /// Set the custom signal channel for the interpreter pub fn set_user_signal_channel(&mut self, signal_rx: signal::UserSignalReceiver) { self.signal_rx = Some(signal_rx); @@ -483,7 +459,7 @@ impl VirtualMachine { /// ```no_run /// use rustpython_vm::Interpreter; /// Interpreter::without_stdlib(Default::default()).enter(|vm| { - /// let bytes = std::fs::read("__pycache__/<input>.rustpython-313.pyc").unwrap(); + /// let bytes = std::fs::read("__pycache__/<input>.rustpython-314.pyc").unwrap(); /// let main_scope = vm.new_scope_with_main().unwrap(); /// vm.run_pyc_bytes(&bytes, main_scope); /// }); @@ -510,6 +486,18 @@ impl VirtualMachine { #[cold] pub fn run_unraisable(&self, e: PyBaseExceptionRef, msg: Option<String>, object: PyObjectRef) { + // During interpreter finalization, sys.unraisablehook may not be available, + // but we still need to report exceptions (especially from atexit callbacks). + // Write directly to stderr like PyErr_FormatUnraisable. + if self + .state + .finalizing + .load(std::sync::atomic::Ordering::Acquire) + { + self.write_unraisable_to_stderr(&e, msg.as_deref(), &object); + return; + } + let sys_module = self.import("sys", 0).unwrap(); let unraisablehook = sys_module.get_attr("unraisablehook", self).unwrap(); @@ -528,6 +516,57 @@ impl VirtualMachine { } } + /// Write unraisable exception to stderr during finalization. + /// Similar to _PyErr_WriteUnraisableDefaultHook in CPython. + fn write_unraisable_to_stderr( + &self, + e: &PyBaseExceptionRef, + msg: Option<&str>, + object: &PyObjectRef, + ) { + // Get stderr once and reuse it + let stderr = crate::stdlib::sys::get_stderr(self).ok(); + + let write_to_stderr = |s: &str, stderr: &Option<PyObjectRef>, vm: &VirtualMachine| { + if let Some(stderr) = stderr { + let _ = vm.call_method(stderr, "write", (s.to_owned(),)); + } else { + eprint!("{}", s); + } + }; + + // Format: "Exception ignored {msg} {object_repr}\n" + if let Some(msg) = msg { + write_to_stderr(&format!("Exception ignored {}", msg), &stderr, self); + } else { + write_to_stderr("Exception ignored in: ", &stderr, self); + } + + if let Ok(repr) = object.repr(self) { + write_to_stderr(&format!("{}\n", repr.as_str()), &stderr, self); + } else { + write_to_stderr("<object repr failed>\n", &stderr, self); + } + + // Write exception type and message + let exc_type_name = e.class().name(); + if let Ok(exc_str) = e.as_object().str(self) { + let exc_str = exc_str.as_str(); + if exc_str.is_empty() { + write_to_stderr(&format!("{}\n", exc_type_name), &stderr, self); + } else { + write_to_stderr(&format!("{}: {}\n", exc_type_name, exc_str), &stderr, self); + } + } else { + write_to_stderr(&format!("{}\n", exc_type_name), &stderr, self); + } + + // Flush stderr to ensure output is visible + if let Some(ref stderr) = stderr { + let _ = self.call_method(stderr, "flush", ()); + } + } + #[inline(always)] pub fn run_frame(&self, frame: FrameRef) -> PyResult { match self.with_frame(frame, |f| f.run(self))? { @@ -582,15 +621,311 @@ impl VirtualMachine { } } + /// Clear module references during shutdown. + /// Follows the same phased algorithm as pylifecycle.c finalize_modules(): + /// no hardcoded module names, reverse import order, only builtins/sys last. + pub fn finalize_modules(&self) { + // Phase 1: Set special sys/builtins attributes to None, restore stdio + self.finalize_modules_delete_special(); + + // Phase 2: Remove all modules from sys.modules (set values to None), + // and collect weakrefs to modules preserving import order. + // Also keeps strong refs (module_refs) to prevent premature deallocation. + // CPython uses _PyGC_CollectNoFail() here to collect __globals__ cycles; + // since RustPython has no working GC, we keep modules alive through + // Phase 4 so their dicts can be explicitly cleared. + let (module_weakrefs, module_refs) = self.finalize_remove_modules(); + + // Phase 3: Clear sys.modules dict + self.finalize_clear_modules_dict(); + + // Phase 4: Clear module dicts in reverse import order using 2-pass algorithm. + // All modules are still alive (held by module_refs), so all weakrefs are valid. + // This breaks __globals__ cycles: dict entries set to None → functions freed → + // __globals__ refs dropped → dict refcount decreases. + self.finalize_clear_module_dicts(&module_weakrefs); + + // Drop strong refs → modules freed with already-cleared dicts. + // No __globals__ cycles remain (broken by Phase 4). + drop(module_refs); + + // Phase 5: Clear sys and builtins dicts last + self.finalize_clear_sys_builtins_dict(); + } + + /// Phase 1: Set special sys attributes to None and restore stdio. + fn finalize_modules_delete_special(&self) { + let none = self.ctx.none(); + let sys_dict = self.sys_module.dict(); + + // Set special sys attributes to None + for attr in &[ + "path", + "argv", + "ps1", + "ps2", + "last_exc", + "last_type", + "last_value", + "last_traceback", + "path_importer_cache", + "meta_path", + "path_hooks", + ] { + let _ = sys_dict.set_item(*attr, none.clone(), self); + } + + // Restore stdin/stdout/stderr from __stdin__/__stdout__/__stderr__ + for (std_name, dunder_name) in &[ + ("stdin", "__stdin__"), + ("stdout", "__stdout__"), + ("stderr", "__stderr__"), + ] { + let restored = sys_dict + .get_item_opt(*dunder_name, self) + .ok() + .flatten() + .unwrap_or_else(|| none.clone()); + let _ = sys_dict.set_item(*std_name, restored, self); + } + + // builtins._ = None + let _ = self.builtins.dict().set_item("_", none, self); + } + + /// Phase 2: Set all sys.modules values to None and collect weakrefs to modules. + /// Returns (weakrefs for Phase 4, strong refs to keep modules alive). + fn finalize_remove_modules(&self) -> (Vec<(String, PyRef<PyWeak>)>, Vec<PyObjectRef>) { + let mut module_weakrefs = Vec::new(); + let mut module_refs = Vec::new(); + + let Ok(modules) = self.sys_module.get_attr(identifier!(self, modules), self) else { + return (module_weakrefs, module_refs); + }; + let Some(modules_dict) = modules.downcast_ref::<PyDict>() else { + return (module_weakrefs, module_refs); + }; + + let none = self.ctx.none(); + let items: Vec<_> = modules_dict.into_iter().collect(); + + for (key, value) in items { + let name = key + .downcast_ref::<PyStr>() + .map(|s| s.as_str().to_owned()) + .unwrap_or_default(); + + // Save weakref and strong ref to module for later clearing + if value.downcast_ref::<PyModule>().is_some() { + if let Ok(weak) = value.downgrade(None, self) { + module_weakrefs.push((name, weak)); + } + module_refs.push(value.clone()); + } + + // Set the value to None in sys.modules + let _ = modules_dict.set_item(&*key, none.clone(), self); + } + + (module_weakrefs, module_refs) + } + + /// Phase 3: Clear sys.modules dict. + fn finalize_clear_modules_dict(&self) { + if let Ok(modules) = self.sys_module.get_attr(identifier!(self, modules), self) + && let Some(modules_dict) = modules.downcast_ref::<PyDict>() + { + modules_dict.clear(); + } + } + + /// Phase 4: Clear module dicts. + /// Without GC, only clear __main__ — other modules' __del__ handlers + /// need their globals intact. CPython can clear ALL module dicts because + /// _PyGC_CollectNoFail() finalizes cycle-participating objects beforehand. + fn finalize_clear_module_dicts(&self, module_weakrefs: &[(String, PyRef<PyWeak>)]) { + for (name, weakref) in module_weakrefs.iter().rev() { + // Only clear __main__ — user objects with __del__ get finalized + // while other modules' globals remain intact for their __del__ handlers. + if name != "__main__" { + continue; + } + + let Some(module_obj) = weakref.upgrade() else { + continue; + }; + let Some(module) = module_obj.downcast_ref::<PyModule>() else { + continue; + }; + + Self::module_clear_dict(&module.dict(), self); + } + } + + /// 2-pass module dict clearing (_PyModule_ClearDict algorithm). + /// Pass 1: Set names starting with '_' (except __builtins__) to None. + /// Pass 2: Set all remaining names (except __builtins__) to None. + pub(crate) fn module_clear_dict(dict: &Py<PyDict>, vm: &VirtualMachine) { + let none = vm.ctx.none(); + + // Pass 1: names starting with '_' (except __builtins__) + for (key, value) in dict.into_iter().collect::<Vec<_>>() { + if vm.is_none(&value) { + continue; + } + if let Some(key_str) = key.downcast_ref::<PyStr>() { + let name = key_str.as_str(); + if name.starts_with('_') && name != "__builtins__" && name != "__spec__" { + let _ = dict.set_item(name, none.clone(), vm); + } + } + } + + // Pass 2: all remaining (except __builtins__) + for (key, value) in dict.into_iter().collect::<Vec<_>>() { + if vm.is_none(&value) { + continue; + } + if let Some(key_str) = key.downcast_ref::<PyStr>() + && key_str.as_str() != "__builtins__" + && key_str.as_str() != "__spec__" + { + let _ = dict.set_item(key_str.as_str(), none.clone(), vm); + } + } + } + + /// Phase 5: Clear sys and builtins dicts last. + fn finalize_clear_sys_builtins_dict(&self) { + Self::module_clear_dict(&self.sys_module.dict(), self); + Self::module_clear_dict(&self.builtins.dict(), self); + } + pub fn current_recursion_depth(&self) -> usize { self.recursion_depth.get() } + /// Stack margin bytes (like _PyOS_STACK_MARGIN_BYTES). + /// 2048 * sizeof(void*) = 16KB for 64-bit. + const STACK_MARGIN_BYTES: usize = 2048 * std::mem::size_of::<usize>(); + + /// Get the stack boundaries using platform-specific APIs. + /// Returns (base, top) where base is the lowest address and top is the highest. + #[cfg(all(not(miri), windows))] + fn get_stack_bounds() -> (usize, usize) { + use windows_sys::Win32::System::Threading::{ + GetCurrentThreadStackLimits, SetThreadStackGuarantee, + }; + let mut low: usize = 0; + let mut high: usize = 0; + unsafe { + GetCurrentThreadStackLimits(&mut low as *mut usize, &mut high as *mut usize); + // Add the guaranteed stack space (reserved for exception handling) + let mut guarantee: u32 = 0; + SetThreadStackGuarantee(&mut guarantee); + low += guarantee as usize; + } + (low, high) + } + + /// Get stack boundaries on non-Windows platforms. + /// Falls back to estimating based on current stack pointer. + #[cfg(all(not(miri), not(windows)))] + fn get_stack_bounds() -> (usize, usize) { + // Use pthread_attr_getstack on platforms that support it + #[cfg(any(target_os = "linux", target_os = "android"))] + { + use libc::{ + pthread_attr_destroy, pthread_attr_getstack, pthread_attr_t, pthread_getattr_np, + pthread_self, + }; + let mut attr: pthread_attr_t = unsafe { std::mem::zeroed() }; + unsafe { + if pthread_getattr_np(pthread_self(), &mut attr) == 0 { + let mut stack_addr: *mut libc::c_void = std::ptr::null_mut(); + let mut stack_size: libc::size_t = 0; + if pthread_attr_getstack(&attr, &mut stack_addr, &mut stack_size) == 0 { + pthread_attr_destroy(&mut attr); + let base = stack_addr as usize; + let top = base + stack_size; + return (base, top); + } + pthread_attr_destroy(&mut attr); + } + } + } + + #[cfg(target_os = "macos")] + { + use libc::{pthread_get_stackaddr_np, pthread_get_stacksize_np, pthread_self}; + unsafe { + let thread = pthread_self(); + let stack_top = pthread_get_stackaddr_np(thread) as usize; + let stack_size = pthread_get_stacksize_np(thread); + let stack_base = stack_top - stack_size; + return (stack_base, stack_top); + } + } + + // Fallback: estimate based on current SP and a default stack size + #[allow(unreachable_code)] + { + let current_sp = psm::stack_pointer() as usize; + // Assume 8MB stack, estimate base + let estimated_size = 8 * 1024 * 1024; + let base = current_sp.saturating_sub(estimated_size); + let top = current_sp + 1024 * 1024; // Assume we're not at the very top + (base, top) + } + } + + /// Calculate the C stack soft limit based on actual stack boundaries. + /// soft_limit = base + 2 * margin (for downward-growing stacks) + #[cfg(not(miri))] + fn calculate_c_stack_soft_limit() -> usize { + let (base, _top) = Self::get_stack_bounds(); + // Soft limit is 2 margins above the base + base + Self::STACK_MARGIN_BYTES * 2 + } + + /// Miri doesn't support inline assembly, so disable C stack checking. + #[cfg(miri)] + fn calculate_c_stack_soft_limit() -> usize { + 0 + } + + /// Check if we're near the C stack limit (like _Py_MakeRecCheck). + /// Returns true only when stack pointer is in the "danger zone" between + /// soft_limit and hard_limit (soft_limit - 2*margin). + #[cfg(not(miri))] + #[inline(always)] + fn check_c_stack_overflow(&self) -> bool { + let current_sp = psm::stack_pointer() as usize; + let soft_limit = self.c_stack_soft_limit.get(); + // Stack grows downward: check if we're below soft limit but above hard limit + // This matches CPython's _Py_MakeRecCheck behavior + current_sp < soft_limit + && current_sp >= soft_limit.saturating_sub(Self::STACK_MARGIN_BYTES * 2) + } + + /// Miri doesn't support inline assembly, so always return false. + #[cfg(miri)] + #[inline(always)] + fn check_c_stack_overflow(&self) -> bool { + false + } + /// Used to run the body of a (possibly) recursive function. It will raise a /// RecursionError if recursive functions are nested far too many times, /// preventing a stack overflow. pub fn with_recursion<R, F: FnOnce() -> PyResult<R>>(&self, _where: &str, f: F) -> PyResult<R> { self.check_recursive_call(_where)?; + + // Native stack guard: check C stack like _Py_MakeRecCheck + if self.check_c_stack_overflow() { + return Err(self.new_recursion_error(_where.to_string())); + } + self.recursion_depth.set(self.recursion_depth.get() + 1); let result = f(); self.recursion_depth.set(self.recursion_depth.get() - 1); @@ -604,6 +939,9 @@ impl VirtualMachine { ) -> PyResult<R> { self.with_recursion("", || { self.frames.borrow_mut().push(frame.clone()); + // Update the current frame slot for sys._current_frames() + #[cfg(feature = "threading")] + crate::vm::thread::update_current_frame(Some(frame.clone())); // Push a new exception context for frame isolation // Each frame starts with no active exception (None) // This prevents exceptions from leaking between function calls @@ -613,6 +951,9 @@ impl VirtualMachine { self.pop_exception(); // defer dec frame let _popped = self.frames.borrow_mut().pop(); + // Update the frame slot to the new top frame (or None if empty) + #[cfg(feature = "threading")] + crate::vm::thread::update_current_frame(self.frames.borrow().last().cloned()); result }) } @@ -766,6 +1107,29 @@ impl VirtualMachine { } else if cls.is(self.ctx.types.list_type) { list_borrow = value.downcast_ref::<PyList>().unwrap().borrow_vec(); &list_borrow + } else if cls.is(self.ctx.types.dict_keys_type) { + // Atomic snapshot of dict keys - prevents race condition during iteration + let keys = value.downcast_ref::<PyDictKeys>().unwrap().dict.keys_vec(); + return keys.into_iter().map(func).collect(); + } else if cls.is(self.ctx.types.dict_values_type) { + // Atomic snapshot of dict values - prevents race condition during iteration + let values = value + .downcast_ref::<PyDictValues>() + .unwrap() + .dict + .values_vec(); + return values.into_iter().map(func).collect(); + } else if cls.is(self.ctx.types.dict_items_type) { + // Atomic snapshot of dict items - prevents race condition during iteration + let items = value + .downcast_ref::<PyDictItems>() + .unwrap() + .dict + .items_vec(); + return items + .into_iter() + .map(|(k, v)| func(self.ctx.new_tuple(vec![k, v]).into())) + .collect(); } else { return self.map_py_iter(value, func); }; @@ -1130,89 +1494,52 @@ pub fn resolve_frozen_alias(name: &str) -> &str { } } -fn core_frozen_inits() -> impl Iterator<Item = (&'static str, FrozenModule)> { - let iter = core::iter::empty(); - macro_rules! ext_modules { - ($iter:ident, $($t:tt)*) => { - let $iter = $iter.chain(py_freeze!($($t)*)); - }; - } - - // keep as example but use file one now - // ext_modules!( - // iter, - // source = "initialized = True; print(\"Hello world!\")\n", - // module_name = "__hello__", - // ); - - // Python modules that the vm calls into, but are not actually part of the stdlib. They could - // in theory be implemented in Rust, but are easiest to do in Python for one reason or another. - // Includes _importlib_bootstrap and _importlib_bootstrap_external - ext_modules!( - iter, - dir = "./Lib/python_builtins", - crate_name = "rustpython_compiler_core" - ); - - // core stdlib Python modules that the vm calls into, but are still used in Python - // application code, e.g. copyreg - // FIXME: Initializing core_modules here results duplicated frozen module generation for core_modules. - // We need a way to initialize this modules for both `Interpreter::without_stdlib()` and `InterpreterConfig::new().init_stdlib().interpreter()` - // #[cfg(not(feature = "freeze-stdlib"))] - ext_modules!( - iter, - dir = "./Lib/core_modules", - crate_name = "rustpython_compiler_core" - ); - - iter -} - #[test] fn test_nested_frozen() { use rustpython_vm as vm; - vm::Interpreter::with_init(Default::default(), |vm| { - // vm.add_native_modules(rustpython_stdlib::get_module_inits()); - vm.add_frozen(rustpython_vm::py_freeze!( - dir = "../../extra_tests/snippets" - )); - }) - .enter(|vm| { - let scope = vm.new_scope_with_builtins(); - - let source = "from dir_module.dir_module_inner import value2"; - let code_obj = vm - .compile(source, vm::compiler::Mode::Exec, "<embedded>".to_owned()) - .map_err(|err| vm.new_syntax_error(&err, Some(source))) - .unwrap(); - - if let Err(e) = vm.run_code_obj(code_obj, scope) { - vm.print_exception(e); - panic!(); - } - }) + vm::Interpreter::builder(Default::default()) + .add_frozen_modules(rustpython_vm::py_freeze!( + dir = "../../../../extra_tests/snippets" + )) + .build() + .enter(|vm| { + let scope = vm.new_scope_with_builtins(); + + let source = "from dir_module.dir_module_inner import value2"; + let code_obj = vm + .compile(source, vm::compiler::Mode::Exec, "<embedded>".to_owned()) + .map_err(|err| vm.new_syntax_error(&err, Some(source))) + .unwrap(); + + if let Err(e) = vm.run_code_obj(code_obj, scope) { + vm.print_exception(e); + panic!(); + } + }) } #[test] fn frozen_origname_matches() { use rustpython_vm as vm; - vm::Interpreter::with_init(Default::default(), |_vm| {}).enter(|vm| { - let check = |name, expected| { - let module = import::import_frozen(vm, name).unwrap(); - let origname: PyStrRef = module - .get_attr("__origname__", vm) - .unwrap() - .try_into_value(vm) - .unwrap(); - assert_eq!(origname.as_str(), expected); - }; + vm::Interpreter::builder(Default::default()) + .build() + .enter(|vm| { + let check = |name, expected| { + let module = import::import_frozen(vm, name).unwrap(); + let origname: PyStrRef = module + .get_attr("__origname__", vm) + .unwrap() + .try_into_value(vm) + .unwrap(); + assert_eq!(origname.as_str(), expected); + }; - check("_frozen_importlib", "importlib._bootstrap"); - check( - "_frozen_importlib_external", - "importlib._bootstrap_external", - ); - }); + check("_frozen_importlib", "importlib._bootstrap"); + check( + "_frozen_importlib_external", + "importlib._bootstrap_external", + ); + }); } diff --git a/crates/vm/src/vm/setting.rs b/crates/vm/src/vm/setting.rs index 9be21b4484e..1a5ef9efa8f 100644 --- a/crates/vm/src/vm/setting.rs +++ b/crates/vm/src/vm/setting.rs @@ -16,6 +16,8 @@ pub struct Paths { pub exec_prefix: String, /// sys.base_exec_prefix pub base_exec_prefix: String, + /// sys._stdlib_dir + pub stdlib_dir: Option<String>, /// Computed module_search_paths (complete sys.path) pub module_search_paths: Vec<String>, } @@ -88,6 +90,12 @@ pub struct Settings { /// -X warn_default_encoding, PYTHONWARNDEFAULTENCODING pub warn_default_encoding: bool, + /// -X thread_inherit_context, whether new threads inherit context from parent + pub thread_inherit_context: bool, + + /// -X context_aware_warnings, whether warnings are context aware + pub context_aware_warnings: bool, + /// -i pub inspect: bool, @@ -190,6 +198,8 @@ impl Default for Settings { isolated: false, dev_mode: false, warn_default_encoding: false, + thread_inherit_context: false, + context_aware_warnings: false, warnoptions: vec![], path_list: vec![], argv: vec![], diff --git a/crates/vm/src/vm/thread.rs b/crates/vm/src/vm/thread.rs index 7e8f0f87e56..fb8621d1526 100644 --- a/crates/vm/src/vm/thread.rs +++ b/crates/vm/src/vm/thread.rs @@ -1,17 +1,27 @@ -use crate::{AsObject, PyObject, PyObjectRef, VirtualMachine}; +#[cfg(feature = "threading")] +use crate::frame::FrameRef; +use crate::{AsObject, PyObject, VirtualMachine}; use core::{ cell::{Cell, RefCell}, ptr::NonNull, }; use itertools::Itertools; +#[cfg(feature = "threading")] +use std::sync::Arc; use std::thread_local; +/// Type for current frame slot - shared between threads for sys._current_frames() +#[cfg(feature = "threading")] +pub type CurrentFrameSlot = Arc<parking_lot::Mutex<Option<FrameRef>>>; + thread_local! { pub(super) static VM_STACK: RefCell<Vec<NonNull<VirtualMachine>>> = Vec::with_capacity(1).into(); pub(crate) static COROUTINE_ORIGIN_TRACKING_DEPTH: Cell<u32> = const { Cell::new(0) }; - pub(crate) static ASYNC_GEN_FINALIZER: RefCell<Option<PyObjectRef>> = const { RefCell::new(None) }; - pub(crate) static ASYNC_GEN_FIRSTITER: RefCell<Option<PyObjectRef>> = const { RefCell::new(None) }; + + /// Current thread's frame slot for sys._current_frames() + #[cfg(feature = "threading")] + static CURRENT_FRAME_SLOT: RefCell<Option<CurrentFrameSlot>> = const { RefCell::new(None) }; } scoped_tls::scoped_thread_local!(static VM_CURRENT: VirtualMachine); @@ -26,11 +36,74 @@ pub fn with_current_vm<R>(f: impl FnOnce(&VirtualMachine) -> R) -> R { pub fn enter_vm<R>(vm: &VirtualMachine, f: impl FnOnce() -> R) -> R { VM_STACK.with(|vms| { vms.borrow_mut().push(vm.into()); + + // Initialize frame slot for this thread if not already done + #[cfg(feature = "threading")] + init_frame_slot_if_needed(vm); + scopeguard::defer! { vms.borrow_mut().pop(); } VM_CURRENT.set(vm, f) }) } +/// Initialize frame slot for current thread if not already initialized. +/// Called automatically by enter_vm(). +#[cfg(feature = "threading")] +fn init_frame_slot_if_needed(vm: &VirtualMachine) { + CURRENT_FRAME_SLOT.with(|slot| { + if slot.borrow().is_none() { + let thread_id = crate::stdlib::thread::get_ident(); + let new_slot = Arc::new(parking_lot::Mutex::new(None)); + vm.state + .thread_frames + .lock() + .insert(thread_id, new_slot.clone()); + *slot.borrow_mut() = Some(new_slot); + } + }); +} + +/// Update the current thread's frame. Called when frames are pushed/popped. +/// This is a hot path - uses only thread-local storage, no locks. +#[cfg(feature = "threading")] +pub fn update_current_frame(frame: Option<FrameRef>) { + CURRENT_FRAME_SLOT.with(|slot| { + if let Some(s) = slot.borrow().as_ref() { + *s.lock() = frame; + } + }); +} + +/// Cleanup frame tracking for the current thread. Called at thread exit. +#[cfg(feature = "threading")] +pub fn cleanup_current_thread_frames(vm: &VirtualMachine) { + let thread_id = crate::stdlib::thread::get_ident(); + vm.state.thread_frames.lock().remove(&thread_id); + CURRENT_FRAME_SLOT.with(|s| { + *s.borrow_mut() = None; + }); +} + +/// Reinitialize frame slot after fork. Called in child process. +/// Creates a fresh slot and registers it for the current thread. +#[cfg(feature = "threading")] +pub fn reinit_frame_slot_after_fork(vm: &VirtualMachine) { + let current_ident = crate::stdlib::thread::get_ident(); + let new_slot = Arc::new(parking_lot::Mutex::new(None)); + + // Try to update the global registry. If we can't get the lock + // (parent thread might have been holding it during fork), skip. + if let Some(mut registry) = vm.state.thread_frames.try_lock() { + registry.clear(); + registry.insert(current_ident, new_slot.clone()); + } + + // Always update thread-local to point to the new slot + CURRENT_FRAME_SLOT.with(|s| { + *s.borrow_mut() = Some(new_slot); + }); +} + pub fn with_vm<F, R>(obj: &PyObject, f: F) -> Option<R> where F: Fn(&VirtualMachine) -> R, @@ -139,6 +212,10 @@ impl VirtualMachine { /// specific guaranteed behavior. #[cfg(feature = "threading")] pub fn new_thread(&self) -> ThreadedVirtualMachine { + let global_trace = self.state.global_trace_func.lock().clone(); + let global_profile = self.state.global_profile_func.lock().clone(); + let use_tracing = global_trace.is_some() || global_profile.is_some(); + let vm = Self { builtins: self.builtins.clone(), sys_module: self.sys_module.clone(), @@ -147,9 +224,9 @@ impl VirtualMachine { wasm_id: self.wasm_id.clone(), exceptions: RefCell::default(), import_func: self.import_func.clone(), - profile_func: RefCell::new(self.ctx.none()), - trace_func: RefCell::new(self.ctx.none()), - use_tracing: Cell::new(false), + profile_func: RefCell::new(global_profile.unwrap_or_else(|| self.ctx.none())), + trace_func: RefCell::new(global_trace.unwrap_or_else(|| self.ctx.none())), + use_tracing: Cell::new(use_tracing), recursion_limit: self.recursion_limit.clone(), signal_handlers: None, signal_rx: None, @@ -157,6 +234,11 @@ impl VirtualMachine { state: self.state.clone(), initialized: self.initialized, recursion_depth: Cell::new(0), + c_stack_soft_limit: Cell::new(VirtualMachine::calculate_c_stack_soft_limit()), + async_gen_firstiter: RefCell::new(None), + async_gen_finalizer: RefCell::new(None), + asyncio_running_loop: RefCell::new(None), + asyncio_running_task: RefCell::new(None), }; ThreadedVirtualMachine { vm } } diff --git a/crates/vm/src/vm/vm_new.rs b/crates/vm/src/vm/vm_new.rs index 119444be75c..8fc8108b6a3 100644 --- a/crates/vm/src/vm/vm_new.rs +++ b/crates/vm/src/vm/vm_new.rs @@ -65,10 +65,6 @@ impl VirtualMachine { pub fn new_scope_with_main(&self) -> PyResult<Scope> { let scope = self.new_scope_with_builtins(); let main_module = self.new_module("__main__", scope.globals.clone(), None); - main_module - .dict() - .set_item("__annotations__", self.ctx.new_dict().into(), self) - .expect("Failed to initialize __main__.__annotations__"); self.sys_module.get_attr("modules", self)?.set_item( "__main__", diff --git a/crates/vm/src/vm/vm_ops.rs b/crates/vm/src/vm/vm_ops.rs index 1d466984377..1a362d67bed 100644 --- a/crates/vm/src/vm/vm_ops.rs +++ b/crates/vm/src/vm/vm_ops.rs @@ -380,8 +380,10 @@ impl VirtualMachine { if !result.is(&self.ctx.not_implemented) { return Ok(result); } - if let Ok(seq_a) = a.try_sequence(self) { - let result = seq_a.concat(b, self)?; + // Check if concat slot is available directly, matching PyNumber_Add behavior + let seq = a.sequence_unchecked(); + if let Some(f) = seq.slots().concat.load() { + let result = f(seq, b, self)?; if !result.is(&self.ctx.not_implemented) { return Ok(result); } @@ -394,8 +396,11 @@ impl VirtualMachine { if !result.is(&self.ctx.not_implemented) { return Ok(result); } - if let Ok(seq_a) = a.try_sequence(self) { - let result = seq_a.inplace_concat(b, self)?; + // Check inplace_concat or concat slot directly, matching PyNumber_InPlaceAdd behavior + let seq = a.sequence_unchecked(); + let slots = seq.slots(); + if let Some(f) = slots.inplace_concat.load().or_else(|| slots.concat.load()) { + let result = f(seq, b, self)?; if !result.is(&self.ctx.not_implemented) { return Ok(result); } diff --git a/crates/vm/src/warn.rs b/crates/vm/src/warn.rs index b7249da19fe..3ec75090b4e 100644 --- a/crates/vm/src/warn.rs +++ b/crates/vm/src/warn.rs @@ -178,8 +178,10 @@ fn already_warned( Some(version_obj) if version_obj.try_int(vm).is_ok() || version_obj.is(&filters_version) => { - let already_warned = registry.get_item(key.as_ref(), vm)?; - if already_warned.is_true(vm)? { + // Use .ok() to handle KeyError when key doesn't exist (like Python's dict.get()) + if let Ok(already_warned) = registry.get_item(key.as_ref(), vm) + && already_warned.is_true(vm)? + { return Ok(true); } } diff --git a/crates/wasm/src/browser_module.rs b/crates/wasm/src/browser_module.rs index 9b6219cab6f..d1eecce28a9 100644 --- a/crates/wasm/src/browser_module.rs +++ b/crates/wasm/src/browser_module.rs @@ -1,6 +1,4 @@ -use rustpython_vm::VirtualMachine; - -pub(crate) use _browser::make_module; +pub(crate) use _browser::module_def; #[pymodule] mod _browser { @@ -257,8 +255,3 @@ mod _browser { Ok(PyPromise::from_future(future).into_pyobject(vm)) } } - -pub fn setup_browser_module(vm: &mut VirtualMachine) { - vm.add_native_module("_browser".to_owned(), Box::new(make_module)); - vm.add_frozen(py_freeze!(dir = "Lib")); -} diff --git a/crates/wasm/src/js_module.rs b/crates/wasm/src/js_module.rs index 11d23bdf6d8..750e85994a1 100644 --- a/crates/wasm/src/js_module.rs +++ b/crates/wasm/src/js_module.rs @@ -1,5 +1,4 @@ pub(crate) use _js::{PyJsValue, PyPromise}; -use rustpython_vm::VirtualMachine; #[pymodule] mod _js { @@ -621,8 +620,4 @@ mod _js { } } -pub(crate) use _js::make_module; - -pub fn setup_js_module(vm: &mut VirtualMachine) { - vm.add_native_module("_js".to_owned(), Box::new(make_module)); -} +pub(crate) use _js::module_def; diff --git a/crates/wasm/src/lib.rs b/crates/wasm/src/lib.rs index 64595fd63ab..99668df2855 100644 --- a/crates/wasm/src/lib.rs +++ b/crates/wasm/src/lib.rs @@ -11,7 +11,6 @@ extern crate rustpython_vm; use js_sys::{Reflect, WebAssembly::RuntimeError}; use std::panic; -pub use vm_class::add_init_func; pub(crate) use vm_class::weak_vm; use wasm_bindgen::prelude::*; diff --git a/crates/wasm/src/vm_class.rs b/crates/wasm/src/vm_class.rs index 0339a47821e..4ad347a19e1 100644 --- a/crates/wasm/src/vm_class.rs +++ b/crates/wasm/src/vm_class.rs @@ -1,5 +1,5 @@ use crate::{ - browser_module::setup_browser_module, + browser_module, convert::{self, PyResultExt}, js_module, wasm_builtins, }; @@ -7,10 +7,8 @@ use alloc::rc::{Rc, Weak}; use core::cell::RefCell; use js_sys::{Object, TypeError}; use rustpython_vm::{ - Interpreter, PyObjectRef, PyPayload, PyRef, PyResult, Settings, VirtualMachine, - builtins::{PyModule, PyWeak}, - compiler::Mode, - scope::Scope, + Interpreter, PyObjectRef, PyRef, PyResult, Settings, VirtualMachine, builtins::PyWeak, + compiler::Mode, scope::Scope, }; use std::collections::HashMap; use wasm_bindgen::prelude::*; @@ -24,69 +22,67 @@ pub(crate) struct StoredVirtualMachine { } #[pymodule] -mod _window {} - -fn init_window_module(vm: &VirtualMachine) -> PyRef<PyModule> { - let module = _window::make_module(vm); - - extend_module!(vm, &module, { - "window" => js_module::PyJsValue::new(wasm_builtins::window()).into_ref(&vm.ctx), - }); - - module +mod _window { + use super::{js_module, wasm_builtins}; + use rustpython_vm::{Py, PyPayload, PyResult, VirtualMachine, builtins::PyModule}; + + pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { + __module_exec(vm, module); + extend_module!(vm, module, { + "window" => js_module::PyJsValue::new(wasm_builtins::window()).into_ref(&vm.ctx), + }); + Ok(()) + } } impl StoredVirtualMachine { fn new(id: String, inject_browser_module: bool) -> StoredVirtualMachine { - let mut scope = None; let mut settings = Settings::default(); settings.allow_external_library = false; - let interp = Interpreter::with_init(settings, |vm| { - #[cfg(feature = "freeze-stdlib")] - vm.add_native_modules(rustpython_stdlib::get_module_inits()); - #[cfg(feature = "freeze-stdlib")] - vm.add_frozen(rustpython_pylib::FROZEN_STDLIB); + let mut builder = Interpreter::builder(settings); + + #[cfg(feature = "freeze-stdlib")] + { + let defs = rustpython_stdlib::stdlib_module_defs(&builder.ctx); + builder = builder + .add_native_modules(&defs) + .add_frozen_modules(rustpython_pylib::FROZEN_STDLIB); + } - vm.wasm_id = Some(id); + // Add wasm-specific modules + let js_def = js_module::module_def(&builder.ctx); + builder = builder.add_native_module(js_def); - js_module::setup_js_module(vm); - if inject_browser_module { - vm.add_native_module("_window".to_owned(), Box::new(init_window_module)); - setup_browser_module(vm); - } + if inject_browser_module { + let window_def = _window::module_def(&builder.ctx); + let browser_def = browser_module::module_def(&builder.ctx); + builder = builder + .add_native_modules(&[window_def, browser_def]) + .add_frozen_modules(rustpython_vm::py_freeze!(dir = "../Lib")); + } - VM_INIT_FUNCS.with_borrow(|funcs| { - for f in funcs { - f(vm) - } - }); + let interp = builder + .init_hook(move |vm| { + vm.wasm_id = Some(id); + }) + .build(); - scope = Some(vm.new_scope_with_builtins()); - }); + let scope = interp.enter(|vm| vm.new_scope_with_builtins()); StoredVirtualMachine { interp, - scope: scope.unwrap(), + scope, held_objects: RefCell::new(Vec::new()), } } } -/// Add a hook to add builtins or frozen modules to the RustPython VirtualMachine while it's -/// initializing. -pub fn add_init_func(f: fn(&mut VirtualMachine)) { - VM_INIT_FUNCS.with_borrow_mut(|funcs| funcs.push(f)) -} - // It's fine that it's thread local, since WASM doesn't even have threads yet. thread_local! // probably gets compiled down to a normal-ish static variable, like Atomic* types do: // https://rustwasm.github.io/2018/10/24/multithreading-rust-and-wasm.html#atomic-instructions thread_local! { static STORED_VMS: RefCell<HashMap<String, Rc<StoredVirtualMachine>>> = RefCell::default(); - static VM_INIT_FUNCS: RefCell<Vec<fn(&mut VirtualMachine)>> = const { - RefCell::new(Vec::new()) - }; } pub fn get_vm_id(vm: &VirtualMachine) -> &str { @@ -106,6 +102,7 @@ pub(crate) fn weak_vm(vm: &VirtualMachine) -> Weak<StoredVirtualMachine> { STORED_VMS.with_borrow(|vms| Rc::downgrade(vms.get(id).expect("VirtualMachine is not valid"))) } +#[derive(Clone, Copy)] #[wasm_bindgen(js_name = vmStore)] pub struct VMStore; diff --git a/demo_closures.py b/demo_closures.py deleted file mode 100644 index 0ed673a94fd..00000000000 --- a/demo_closures.py +++ /dev/null @@ -1,12 +0,0 @@ -def foo(x): - def bar(z): - return z + x - - return bar - - -f = foo(9) -g = foo(10) - -print(f(2)) -print(g(2)) diff --git a/example_projects/frozen_stdlib/src/main.rs b/example_projects/frozen_stdlib/src/main.rs index 2688d2164a2..8ff316faed8 100644 --- a/example_projects/frozen_stdlib/src/main.rs +++ b/example_projects/frozen_stdlib/src/main.rs @@ -1,9 +1,10 @@ // spell-checker:ignore aheui -/// Setting up a project with a frozen stdlib can be done *either* by using `rustpython::InterpreterConfig` or `rustpython_vm::Interpreter::with_init`. -/// See each function for example. -/// -/// See also: `aheui-rust.md` for freezing your own package. +//! Setting up a project with a frozen stdlib can be done *either* by using `rustpython::InterpreterBuilder` or `rustpython_vm::Interpreter::builder`. +//! See each function for example. +//! +//! See also: `aheui-rust.md` for freezing your own package. +use rustpython::InterpreterBuilderExt; use rustpython_vm::{PyResult, VirtualMachine}; fn run(keyword: &str, vm: &VirtualMachine) -> PyResult<()> { @@ -17,21 +18,19 @@ fn run(keyword: &str, vm: &VirtualMachine) -> PyResult<()> { } fn interpreter_with_config() { - let interpreter = rustpython::InterpreterConfig::new() + let interpreter = rustpython::InterpreterBuilder::new() .init_stdlib() .interpreter(); // Use interpreter.enter to reuse the same interpreter later - interpreter.run(|vm| run("rustpython::InterpreterConfig", vm)); + interpreter.run(|vm| run("rustpython::InterpreterBuilder", vm)); } fn interpreter_with_vm() { - let interpreter = rustpython_vm::Interpreter::with_init(Default::default(), |vm| { - // This is unintuitive, but the stdlib is out of the vm crate. - // Any suggestion to improve this is welcome. - vm.add_frozen(rustpython_pylib::FROZEN_STDLIB); - }); + let interpreter = rustpython_vm::Interpreter::builder(Default::default()) + .add_frozen_modules(rustpython_pylib::FROZEN_STDLIB) + .build(); // Use interpreter.enter to reuse the same interpreter later - interpreter.run(|vm| run("rustpython_vm::Interpreter::with_init", vm)); + interpreter.run(|vm| run("rustpython_vm::Interpreter::builder", vm)); } fn main() { diff --git a/examples/call_between_rust_and_python.rs b/examples/call_between_rust_and_python.rs index dee17058475..3a7c0ce610e 100644 --- a/examples/call_between_rust_and_python.rs +++ b/examples/call_between_rust_and_python.rs @@ -1,17 +1,12 @@ +use rustpython::InterpreterBuilderExt; use rustpython::vm::{ PyObject, PyPayload, PyResult, TryFromBorrowedObject, VirtualMachine, pyclass, pymodule, }; pub fn main() { - let interp = rustpython::InterpreterConfig::new() - .init_stdlib() - .init_hook(Box::new(|vm| { - vm.add_native_module( - "rust_py_module".to_owned(), - Box::new(rust_py_module::make_module), - ); - })) - .interpreter(); + let builder = rustpython::Interpreter::builder(Default::default()); + let def = rust_py_module::module_def(&builder.ctx); + let interp = builder.init_stdlib().add_native_module(def).build(); interp.enter(|vm| { vm.insert_sys_path(vm.new_pyobj("examples")) diff --git a/examples/dis.rs b/examples/dis.rs index 1ca350603f9..0b6190dde3c 100644 --- a/examples/dis.rs +++ b/examples/dis.rs @@ -60,7 +60,7 @@ fn main() -> Result<(), lexopt::Error> { for script in &scripts { if script.exists() && script.is_file() { - let res = display_script(script, mode, opts.clone(), expand_code_objects); + let res = display_script(script, mode, opts, expand_code_objects); if let Err(e) = res { error!("Error while compiling {script:?}: {e}"); } diff --git a/examples/freeze/main.rs b/examples/freeze/main.rs index 48991129073..ab26a2bc808 100644 --- a/examples/freeze/main.rs +++ b/examples/freeze/main.rs @@ -7,9 +7,8 @@ fn main() -> vm::PyResult<()> { fn run(vm: &vm::VirtualMachine) -> vm::PyResult<()> { let scope = vm.new_scope_with_builtins(); - // the file parameter is relative to the directory where the crate's Cargo.toml is located, see $CARGO_MANIFEST_DIR: - // https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-crates - let module = vm::py_compile!(file = "examples/freeze/freeze.py"); + // the file parameter is relative to the current file. + let module = vm::py_compile!(file = "freeze.py"); let res = vm.run_code_obj(vm.ctx.new_code(module), scope); diff --git a/examples/generator.rs b/examples/generator.rs index 27733a1913d..55841c767a1 100644 --- a/examples/generator.rs +++ b/examples/generator.rs @@ -42,9 +42,9 @@ gen() } fn main() -> ExitCode { - let interp = vm::Interpreter::with_init(Default::default(), |vm| { - vm.add_native_modules(rustpython_stdlib::get_module_inits()); - }); + let builder = vm::Interpreter::builder(Default::default()); + let defs = rustpython_stdlib::stdlib_module_defs(&builder.ctx); + let interp = builder.add_native_modules(&defs).build(); let result = py_main(&interp); vm::common::os::exit_code(interp.run(|_vm| result)) } diff --git a/examples/package_embed.rs b/examples/package_embed.rs index e82e71f5ceb..bb2f29e3f5f 100644 --- a/examples/package_embed.rs +++ b/examples/package_embed.rs @@ -19,9 +19,9 @@ fn main() -> ExitCode { // Add standard library path let mut settings = vm::Settings::default(); settings.path_list.push("Lib".to_owned()); - let interp = vm::Interpreter::with_init(settings, |vm| { - vm.add_native_modules(rustpython_stdlib::get_module_inits()); - }); + let builder = vm::Interpreter::builder(settings); + let defs = rustpython_stdlib::stdlib_module_defs(&builder.ctx); + let interp = builder.add_native_modules(&defs).build(); let result = py_main(&interp); let result = result.map(|result| { println!("name: {result}"); diff --git a/extra_tests/custom_text_test_runner.py b/extra_tests/custom_text_test_runner.py index 018121f0da4..afec493a66c 100644 --- a/extra_tests/custom_text_test_runner.py +++ b/extra_tests/custom_text_test_runner.py @@ -112,7 +112,7 @@ def __call__(self, data_list, totals=None): def get_function_args(func_ref): try: - return [p for p in inspect.getargspec(func_ref).args if p != "self"] + return [p for p in inspect.getfullargspec(func_ref).args if p != "self"] except: return None diff --git a/extra_tests/jsontests.py b/extra_tests/jsontests.py index c1f92509fe7..f3213ac09d1 100644 --- a/extra_tests/jsontests.py +++ b/extra_tests/jsontests.py @@ -2,7 +2,7 @@ import unittest from custom_text_test_runner import CustomTextTestRunner as Runner -from test.libregrtest.runtest import findtests +from test.libregrtest.findtests import findtests testnames = findtests() # idk why this fixes the hanging, if it does diff --git a/extra_tests/snippets/builtin_bool.py b/extra_tests/snippets/builtin_bool.py index 6b6b4e0e08b..902ed0cced0 100644 --- a/extra_tests/snippets/builtin_bool.py +++ b/extra_tests/snippets/builtin_bool.py @@ -18,7 +18,9 @@ assert bool(1) is True assert bool({}) is False -assert bool(NotImplemented) is True +# NotImplemented cannot be used in a boolean context (Python 3.14+) +with assert_raises(TypeError): + bool(NotImplemented) assert bool(...) is True if not 1: diff --git a/extra_tests/snippets/builtin_bytearray.py b/extra_tests/snippets/builtin_bytearray.py index 0b7e419390e..ee11e913ff2 100644 --- a/extra_tests/snippets/builtin_bytearray.py +++ b/extra_tests/snippets/builtin_bytearray.py @@ -153,16 +153,41 @@ class B(bytearray): # # hex from hex assert bytearray([0, 1, 9, 23, 90, 234]).hex() == "000109175aea" -bytearray.fromhex("62 6c7a 34350a ") == b"blz45\n" +# fromhex with str +assert bytearray.fromhex("62 6c7a 34350a ") == b"blz45\n" + +# fromhex with bytes +assert bytearray.fromhex(b"62 6c7a 34350a ") == b"blz45\n" +assert bytearray.fromhex(b"B9 01EF") == b"\xb9\x01\xef" + +# fromhex with bytearray (bytes-like object) +assert bytearray.fromhex(bytearray(b"4142")) == b"AB" + +# fromhex with memoryview (bytes-like object) +assert bytearray.fromhex(memoryview(b"4142")) == b"AB" + +# fromhex error: non-hexadecimal character try: bytearray.fromhex("62 a 21") except ValueError as e: - str(e) == "non-hexadecimal number found in fromhex() arg at position 4" + assert str(e) == "non-hexadecimal number found in fromhex() arg at position 4" try: bytearray.fromhex("6Z2") except ValueError as e: - str(e) == "non-hexadecimal number found in fromhex() arg at position 1" + assert str(e) == "non-hexadecimal number found in fromhex() arg at position 1" + +# fromhex error: odd number of hex digits +try: + bytearray.fromhex("abc") +except ValueError as e: + assert str(e) == "fromhex() arg must contain an even number of hexadecimal digits" + +# fromhex error: wrong type with assert_raises(TypeError): + bytearray.fromhex(123) + +# fromhex with bytes containing invalid hex raises ValueError +with assert_raises(ValueError): bytearray.fromhex(b"hhjjk") # center assert [bytearray(b"koki").center(i, b"|") for i in range(3, 10)] == [ diff --git a/extra_tests/snippets/builtin_bytes.py b/extra_tests/snippets/builtin_bytes.py index 4cb743baa6f..2cb4c317f49 100644 --- a/extra_tests/snippets/builtin_bytes.py +++ b/extra_tests/snippets/builtin_bytes.py @@ -137,16 +137,41 @@ # hex from hex assert bytes([0, 1, 9, 23, 90, 234]).hex() == "000109175aea" -bytes.fromhex("62 6c7a 34350a ") == b"blz45\n" +# fromhex with str +assert bytes.fromhex("62 6c7a 34350a ") == b"blz45\n" + +# fromhex with bytes +assert bytes.fromhex(b"62 6c7a 34350a ") == b"blz45\n" +assert bytes.fromhex(b"B9 01EF") == b"\xb9\x01\xef" + +# fromhex with bytearray (bytes-like object) +assert bytes.fromhex(bytearray(b"4142")) == b"AB" + +# fromhex with memoryview (bytes-like object) +assert bytes.fromhex(memoryview(b"4142")) == b"AB" + +# fromhex error: non-hexadecimal character try: bytes.fromhex("62 a 21") except ValueError as e: - str(e) == "non-hexadecimal number found in fromhex() arg at position 4" + assert str(e) == "non-hexadecimal number found in fromhex() arg at position 4" try: bytes.fromhex("6Z2") except ValueError as e: - str(e) == "non-hexadecimal number found in fromhex() arg at position 1" + assert str(e) == "non-hexadecimal number found in fromhex() arg at position 1" + +# fromhex error: odd number of hex digits +try: + bytes.fromhex("abc") +except ValueError as e: + assert str(e) == "fromhex() arg must contain an even number of hexadecimal digits" + +# fromhex error: wrong type with assert_raises(TypeError): + bytes.fromhex(123) + +# fromhex with bytes containing invalid hex raises ValueError +with assert_raises(ValueError): bytes.fromhex(b"hhjjk") # center assert [b"koki".center(i, b"|") for i in range(3, 10)] == [ diff --git a/extra_tests/snippets/builtin_exceptions.py b/extra_tests/snippets/builtin_exceptions.py index 246be3b8fda..8879e130bc2 100644 --- a/extra_tests/snippets/builtin_exceptions.py +++ b/extra_tests/snippets/builtin_exceptions.py @@ -239,12 +239,10 @@ class SubError(MyError): raise e except MyError as exc: # It was a segmentation fault before, will print info to stdout: - if platform.python_implementation() == "RustPython": - # For some reason `CPython` hangs on this code: - sys.excepthook(type(exc), exc, exc.__traceback__) - assert isinstance(exc, MyError) - assert exc.__cause__ is None - assert exc.__context__ is e + sys.excepthook(type(exc), exc, exc.__traceback__) + assert isinstance(exc, MyError) + assert exc.__cause__ is None + assert exc.__context__ is e # Regression to @@ -255,26 +253,42 @@ class SubError(MyError): assert BaseException.__init__.__qualname__ == "BaseException.__init__" assert BaseException().__dict__ == {} +# Exception inherits __init__ from BaseException assert Exception.__new__.__qualname__ == "Exception.__new__", ( Exception.__new__.__qualname__ ) -assert Exception.__init__.__qualname__ == "Exception.__init__", ( +assert Exception.__init__.__qualname__ == "BaseException.__init__", ( Exception.__init__.__qualname__ ) assert Exception().__dict__ == {} -# Extends `BaseException`, simple: +# Extends `BaseException`, simple - inherits __init__ from BaseException: assert KeyboardInterrupt.__new__.__qualname__ == "KeyboardInterrupt.__new__", ( KeyboardInterrupt.__new__.__qualname__ ) -assert KeyboardInterrupt.__init__.__qualname__ == "KeyboardInterrupt.__init__" +assert KeyboardInterrupt.__init__.__qualname__ == "BaseException.__init__" assert KeyboardInterrupt().__dict__ == {} -# Extends `Exception`, simple: +# Extends `BaseException`, complex - has its own __init__: +# SystemExit_init sets self.code based on args length +assert SystemExit.__init__.__qualname__ == "SystemExit.__init__" +assert SystemExit.__dict__.get("__init__") is not None, ( + "SystemExit must have its own __init__" +) +assert SystemExit.__init__ is not BaseException.__init__ +assert SystemExit().__dict__ == {} +# SystemExit.code behavior: +assert SystemExit().code is None +assert SystemExit(1).code == 1 +assert SystemExit(1, 2).code == (1, 2) +assert SystemExit(1, 2, 3).code == (1, 2, 3) + + +# Extends `Exception`, simple - inherits __init__ from BaseException: assert TypeError.__new__.__qualname__ == "TypeError.__new__" -assert TypeError.__init__.__qualname__ == "TypeError.__init__" +assert TypeError.__init__.__qualname__ == "BaseException.__init__" assert TypeError().__dict__ == {} @@ -356,7 +370,8 @@ class SubError(MyError): # Custom `__new__` and `__init__`: assert ImportError.__init__.__qualname__ == "ImportError.__init__" assert ImportError(name="a").name == "a" -assert ModuleNotFoundError.__init__.__qualname__ == "ModuleNotFoundError.__init__" +# ModuleNotFoundError inherits __init__ from ImportError via MRO (MiddlingExtendsException) +assert ModuleNotFoundError.__init__.__qualname__ == "ImportError.__init__" assert ModuleNotFoundError(name="a").name == "a" diff --git a/extra_tests/snippets/builtin_int.py b/extra_tests/snippets/builtin_int.py index bc3cd5fd996..aab24cbb4cc 100644 --- a/extra_tests/snippets/builtin_int.py +++ b/extra_tests/snippets/builtin_int.py @@ -318,8 +318,9 @@ def __int__(self): assert isinstance((1).__round__(0), int) assert (0).__round__(0) == 0 assert (1).__round__(0) == 1 -assert_raises(TypeError, lambda: (0).__round__(None)) -assert_raises(TypeError, lambda: (1).__round__(None)) +# Python 3.14+: __round__(None) is now allowed, same as __round__() +assert (0).__round__(None) == 0 +assert (1).__round__(None) == 1 assert_raises(TypeError, lambda: (0).__round__(0.0)) assert_raises(TypeError, lambda: (1).__round__(0.0)) diff --git a/extra_tests/snippets/builtin_list.py b/extra_tests/snippets/builtin_list.py index 3e6bb8fc943..d4afbffa1cb 100644 --- a/extra_tests/snippets/builtin_list.py +++ b/extra_tests/snippets/builtin_list.py @@ -270,6 +270,55 @@ def __gt__(self, other): lst.sort(key=C) assert lst == [1, 2, 3, 4, 5] + +# Test that sorted() uses __lt__ (not __gt__) for comparisons. +# Track which comparison method is actually called during sort. +class TrackComparison: + lt_calls = 0 + gt_calls = 0 + + def __init__(self, value): + self.value = value + + def __lt__(self, other): + TrackComparison.lt_calls += 1 + return self.value < other.value + + def __gt__(self, other): + TrackComparison.gt_calls += 1 + return self.value > other.value + + +# Reset and test sorted() +TrackComparison.lt_calls = 0 +TrackComparison.gt_calls = 0 +items = [TrackComparison(3), TrackComparison(1), TrackComparison(2)] +sorted(items) +assert TrackComparison.lt_calls > 0, "sorted() should call __lt__" +assert TrackComparison.gt_calls == 0, ( + f"sorted() should not call __gt__, but it was called {TrackComparison.gt_calls} times" +) + +# Reset and test list.sort() +TrackComparison.lt_calls = 0 +TrackComparison.gt_calls = 0 +items = [TrackComparison(3), TrackComparison(1), TrackComparison(2)] +items.sort() +assert TrackComparison.lt_calls > 0, "list.sort() should call __lt__" +assert TrackComparison.gt_calls == 0, ( + f"list.sort() should not call __gt__, but it was called {TrackComparison.gt_calls} times" +) + +# Reset and test sorted(reverse=True) - should still use __lt__, not __gt__ +TrackComparison.lt_calls = 0 +TrackComparison.gt_calls = 0 +items = [TrackComparison(3), TrackComparison(1), TrackComparison(2)] +sorted(items, reverse=True) +assert TrackComparison.lt_calls > 0, "sorted(reverse=True) should call __lt__" +assert TrackComparison.gt_calls == 0, ( + f"sorted(reverse=True) should not call __gt__, but it was called {TrackComparison.gt_calls} times" +) + lst = [5, 1, 2, 3, 4] diff --git a/extra_tests/snippets/builtin_set.py b/extra_tests/snippets/builtin_set.py index 1b2f6ff0968..950875ea09a 100644 --- a/extra_tests/snippets/builtin_set.py +++ b/extra_tests/snippets/builtin_set.py @@ -200,6 +200,18 @@ class S(set): with assert_raises(TypeError): a &= [1, 2, 3] +a = set([1, 2, 3]) +a &= a +assert a == set([1, 2, 3]) + +a = set([1, 2, 3]) +a -= a +assert a == set() + +a = set([1, 2, 3]) +a ^= a +assert a == set() + a = set([1, 2, 3]) a.difference_update([3, 4, 5]) assert a == set([1, 2]) diff --git a/extra_tests/snippets/builtin_type.py b/extra_tests/snippets/builtin_type.py index 67269e694c0..8cb0a09a215 100644 --- a/extra_tests/snippets/builtin_type.py +++ b/extra_tests/snippets/builtin_type.py @@ -584,6 +584,7 @@ def __new__(cls, *args, **kwargs): assert ClassWithNew().__new__.__qualname__ == "ClassWithNew.__new__" assert ClassWithNew.__new__.__name__ == "__new__" assert ClassWithNew().__new__.__name__ == "__new__" +assert isinstance(ClassWithNew.__dict__.get("__new__"), staticmethod) assert ClassWithNew.N.__new__.__qualname__ == "ClassWithNew.N.__new__" assert ClassWithNew().N.__new__.__qualname__ == "ClassWithNew.N.__new__" @@ -593,6 +594,7 @@ def __new__(cls, *args, **kwargs): assert ClassWithNew().N().__new__.__qualname__ == "ClassWithNew.N.__new__" assert ClassWithNew.N().__new__.__name__ == "__new__" assert ClassWithNew().N().__new__.__name__ == "__new__" +assert isinstance(ClassWithNew.N.__dict__.get("__new__"), staticmethod) # Regression to: diff --git a/extra_tests/snippets/builtins_module.py b/extra_tests/snippets/builtins_module.py index 6dea94d8d77..bf762425c89 100644 --- a/extra_tests/snippets/builtins_module.py +++ b/extra_tests/snippets/builtins_module.py @@ -22,6 +22,17 @@ exec("", namespace) assert namespace["__builtins__"] == __builtins__.__dict__ + +# function.__builtins__ should be a dict, not a module +# See: https://docs.python.org/3/reference/datamodel.html +def test_func(): + pass + + +assert isinstance(test_func.__builtins__, dict), ( + f"function.__builtins__ should be dict, got {type(test_func.__builtins__)}" +) + # with assert_raises(NameError): # exec('print(__builtins__)', {'__builtins__': {}}) diff --git a/extra_tests/snippets/code_co_consts.py b/extra_tests/snippets/code_co_consts.py index 58355652682..13f76a0d13e 100644 --- a/extra_tests/snippets/code_co_consts.py +++ b/extra_tests/snippets/code_co_consts.py @@ -1,39 +1,112 @@ +""" +Test co_consts behavior for Python 3.14+ + +In Python 3.14+: +- Functions with docstrings have the docstring as co_consts[0] +- CO_HAS_DOCSTRING flag (0x4000000) indicates docstring presence +- Functions without docstrings do NOT have None added as placeholder for docstring + +Note: Other constants (small integers, code objects, etc.) may still appear in co_consts +depending on optimization level. This test focuses on docstring behavior. +""" + + +# Test function with docstring - docstring should be co_consts[0] +def with_doc(): + """This is a docstring""" + return 1 + + +assert with_doc.__code__.co_consts[0] == "This is a docstring", ( + with_doc.__code__.co_consts +) +assert with_doc.__doc__ == "This is a docstring" +# Check CO_HAS_DOCSTRING flag (0x4000000) +assert with_doc.__code__.co_flags & 0x4000000, hex(with_doc.__code__.co_flags) + + +# Test function without docstring - should NOT have HAS_DOCSTRING flag +def no_doc(): + return 1 + + +assert not (no_doc.__code__.co_flags & 0x4000000), hex(no_doc.__code__.co_flags) +assert no_doc.__doc__ is None + + +# Test async function with docstring from asyncio import sleep -def f(): - def g(): - return 1 +async def async_with_doc(): + """Async docstring""" + await sleep(1) + return 1 - assert g.__code__.co_consts[0] == None - return 2 +assert async_with_doc.__code__.co_consts[0] == "Async docstring", ( + async_with_doc.__code__.co_consts +) +assert async_with_doc.__doc__ == "Async docstring" +assert async_with_doc.__code__.co_flags & 0x4000000 -assert f.__code__.co_consts[0] == None +# Test async function without docstring +async def async_no_doc(): + await sleep(1) + return 1 + + +assert not (async_no_doc.__code__.co_flags & 0x4000000) +assert async_no_doc.__doc__ is None -def generator(): + +# Test generator with docstring +def gen_with_doc(): + """Generator docstring""" yield 1 yield 2 -assert generator().gi_code.co_consts[0] == None +assert gen_with_doc.__code__.co_consts[0] == "Generator docstring" +assert gen_with_doc.__doc__ == "Generator docstring" +assert gen_with_doc.__code__.co_flags & 0x4000000 -async def async_f(): - await sleep(1) - return 1 +# Test generator without docstring +def gen_no_doc(): + yield 1 + yield 2 + +assert not (gen_no_doc.__code__.co_flags & 0x4000000) +assert gen_no_doc.__doc__ is None -assert async_f.__code__.co_consts[0] == None +# Test lambda - cannot have docstring lambda_f = lambda: 0 -assert lambda_f.__code__.co_consts[0] == None +assert not (lambda_f.__code__.co_flags & 0x4000000) +assert lambda_f.__doc__ is None + + +# Test class method with docstring +class cls_with_doc: + def method(): + """Method docstring""" + return 1 + +assert cls_with_doc.method.__code__.co_consts[0] == "Method docstring" +assert cls_with_doc.method.__doc__ == "Method docstring" -class cls: - def f(): + +# Test class method without docstring +class cls_no_doc: + def method(): return 1 -assert cls().f.__code__.co_consts[0] == None +assert not (cls_no_doc.method.__code__.co_flags & 0x4000000) +assert cls_no_doc.method.__doc__ is None + +print("All co_consts tests passed!") diff --git a/extra_tests/snippets/example_interactive.py b/extra_tests/snippets/example_interactive.py index f9484f15dcf..5958dd11707 100644 --- a/extra_tests/snippets/example_interactive.py +++ b/extra_tests/snippets/example_interactive.py @@ -4,7 +4,7 @@ def f(x, y, *args, power=1, **kwargs): - print("Constant String", 2, None, (2, 4)) + print("Constant String", 256, None, (2, 4)) assert code_class == type(c1) z = x * y return z**power @@ -19,7 +19,7 @@ def f(x, y, *args, power=1, **kwargs): # assert isinstance(c2.co_code, bytes) assert "Constant String" in c2.co_consts, c2.co_consts print(c2.co_consts) -assert 2 in c2.co_consts, c2.co_consts +assert 256 in c2.co_consts, c2.co_consts assert "example_interactive.py" in c2.co_filename assert c2.co_firstlineno == 6, str(c2.co_firstlineno) # assert isinstance(c2.co_flags, int) # 'OPTIMIZED, NEWLOCALS, NOFREE' diff --git a/extra_tests/snippets/stdlib_io.py b/extra_tests/snippets/stdlib_io.py index 722886d34ee..7c473908295 100644 --- a/extra_tests/snippets/stdlib_io.py +++ b/extra_tests/snippets/stdlib_io.py @@ -10,15 +10,15 @@ result = bb.read() -assert len(result) <= 8 * 1024 +assert len(result) <= 16 * 1024 assert len(result) >= 0 assert isinstance(result, bytes) with FileIO("README.md") as fio: res = fio.read() - assert len(result) <= 8 * 1024 - assert len(result) >= 0 - assert isinstance(result, bytes) + assert len(res) <= 16 * 1024 + assert len(res) >= 0 + assert isinstance(res, bytes) fd = os.open("README.md", os.O_RDONLY) diff --git a/extra_tests/snippets/stdlib_itertools.py b/extra_tests/snippets/stdlib_itertools.py index 4d2e9f6e1f7..ce7a494713a 100644 --- a/extra_tests/snippets/stdlib_itertools.py +++ b/extra_tests/snippets/stdlib_itertools.py @@ -1,5 +1,4 @@ import itertools -import pickle from testutils import assert_raises @@ -181,10 +180,6 @@ # itertools.takewhile tests -def underten(x): - return x < 10 - - from itertools import takewhile as tw t = tw(lambda n: n < 5, [1, 2, 5, 1, 3]) @@ -226,70 +221,6 @@ def underten(x): with assert_raises(StopIteration): next(t) -it = tw(underten, [1, 3, 5, 20, 2, 4, 6, 8]) -assert ( - pickle.dumps(it, 0) - == b"citertools\ntakewhile\np0\n(c__main__\nunderten\np1\nc__builtin__\niter\np2\n((lp3\nI1\naI3\naI5\naI20\naI2\naI4\naI6\naI8\natp4\nRp5\nI0\nbtp6\nRp7\nI0\nb." -) -assert ( - pickle.dumps(it, 1) - == b"citertools\ntakewhile\nq\x00(c__main__\nunderten\nq\x01c__builtin__\niter\nq\x02(]q\x03(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08etq\x04Rq\x05K\x00btq\x06Rq\x07K\x00b." -) -assert ( - pickle.dumps(it, 2) - == b"\x80\x02citertools\ntakewhile\nq\x00c__main__\nunderten\nq\x01c__builtin__\niter\nq\x02]q\x03(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08e\x85q\x04Rq\x05K\x00b\x86q\x06Rq\x07K\x00b." -) -assert ( - pickle.dumps(it, 3) - == b"\x80\x03citertools\ntakewhile\nq\x00c__main__\nunderten\nq\x01cbuiltins\niter\nq\x02]q\x03(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08e\x85q\x04Rq\x05K\x00b\x86q\x06Rq\x07K\x00b." -) -assert ( - pickle.dumps(it, 4) - == b"\x80\x04\x95i\x00\x00\x00\x00\x00\x00\x00\x8c\titertools\x94\x8c\ttakewhile\x94\x93\x94\x8c\x08__main__\x94\x8c\x08underten\x94\x93\x94\x8c\x08builtins\x94\x8c\x04iter\x94\x93\x94]\x94(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08e\x85\x94R\x94K\x00b\x86\x94R\x94K\x00b." -) -assert ( - pickle.dumps(it, 5) - == b"\x80\x05\x95i\x00\x00\x00\x00\x00\x00\x00\x8c\titertools\x94\x8c\ttakewhile\x94\x93\x94\x8c\x08__main__\x94\x8c\x08underten\x94\x93\x94\x8c\x08builtins\x94\x8c\x04iter\x94\x93\x94]\x94(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08e\x85\x94R\x94K\x00b\x86\x94R\x94K\x00b." -) -next(it) -next(it) -next(it) -try: - next(it) -except StopIteration: - pass -assert ( - pickle.dumps(it, 0) - == b"citertools\ntakewhile\np0\n(c__main__\nunderten\np1\nc__builtin__\niter\np2\n((lp3\nI1\naI3\naI5\naI20\naI2\naI4\naI6\naI8\natp4\nRp5\nI4\nbtp6\nRp7\nI1\nb." -) -assert ( - pickle.dumps(it, 1) - == b"citertools\ntakewhile\nq\x00(c__main__\nunderten\nq\x01c__builtin__\niter\nq\x02(]q\x03(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08etq\x04Rq\x05K\x04btq\x06Rq\x07K\x01b." -) -assert ( - pickle.dumps(it, 2) - == b"\x80\x02citertools\ntakewhile\nq\x00c__main__\nunderten\nq\x01c__builtin__\niter\nq\x02]q\x03(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08e\x85q\x04Rq\x05K\x04b\x86q\x06Rq\x07K\x01b." -) -assert ( - pickle.dumps(it, 3) - == b"\x80\x03citertools\ntakewhile\nq\x00c__main__\nunderten\nq\x01cbuiltins\niter\nq\x02]q\x03(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08e\x85q\x04Rq\x05K\x04b\x86q\x06Rq\x07K\x01b." -) -assert ( - pickle.dumps(it, 4) - == b"\x80\x04\x95i\x00\x00\x00\x00\x00\x00\x00\x8c\titertools\x94\x8c\ttakewhile\x94\x93\x94\x8c\x08__main__\x94\x8c\x08underten\x94\x93\x94\x8c\x08builtins\x94\x8c\x04iter\x94\x93\x94]\x94(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08e\x85\x94R\x94K\x04b\x86\x94R\x94K\x01b." -) -assert ( - pickle.dumps(it, 5) - == b"\x80\x05\x95i\x00\x00\x00\x00\x00\x00\x00\x8c\titertools\x94\x8c\ttakewhile\x94\x93\x94\x8c\x08__main__\x94\x8c\x08underten\x94\x93\x94\x8c\x08builtins\x94\x8c\x04iter\x94\x93\x94]\x94(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08e\x85\x94R\x94K\x04b\x86\x94R\x94K\x01b." -) -for proto in range(pickle.HIGHEST_PROTOCOL + 1): - try: - next(pickle.loads(pickle.dumps(it, proto))) - assert False - except StopIteration: - pass - - # itertools.islice tests @@ -297,40 +228,28 @@ def assert_matches_seq(it, seq): assert list(it) == list(seq) -def test_islice_pickle(it): - for p in range(pickle.HIGHEST_PROTOCOL + 1): - it == pickle.loads(pickle.dumps(it, p)) - - i = itertools.islice it = i([1, 2, 3, 4, 5], 3) assert_matches_seq(it, [1, 2, 3]) -test_islice_pickle(it) it = i([0.5, 1, 1.5, 2, 2.5, 3, 4, 5], 1, 6, 2) assert_matches_seq(it, [1, 2, 3]) -test_islice_pickle(it) it = i([1, 2], None) assert_matches_seq(it, [1, 2]) -test_islice_pickle(it) it = i([1, 2, 3], None, None, None) assert_matches_seq(it, [1, 2, 3]) -test_islice_pickle(it) it = i([1, 2, 3], 1, None, None) assert_matches_seq(it, [2, 3]) -test_islice_pickle(it) it = i([1, 2, 3], None, 2, None) assert_matches_seq(it, [1, 2]) -test_islice_pickle(it) it = i([1, 2, 3], None, None, 3) assert_matches_seq(it, [1]) -test_islice_pickle(it) # itertools.filterfalse it = itertools.filterfalse(lambda x: x % 2, range(10)) @@ -359,59 +278,6 @@ def test_islice_pickle(it): with assert_raises(StopIteration): next(it) -it = itertools.dropwhile(underten, [1, 3, 5, 20, 2, 4, 6, 8]) -assert ( - pickle.dumps(it, 0) - == b"citertools\ndropwhile\np0\n(c__main__\nunderten\np1\nc__builtin__\niter\np2\n((lp3\nI1\naI3\naI5\naI20\naI2\naI4\naI6\naI8\natp4\nRp5\nI0\nbtp6\nRp7\nI0\nb." -) -assert ( - pickle.dumps(it, 1) - == b"citertools\ndropwhile\nq\x00(c__main__\nunderten\nq\x01c__builtin__\niter\nq\x02(]q\x03(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08etq\x04Rq\x05K\x00btq\x06Rq\x07K\x00b." -) -assert ( - pickle.dumps(it, 2) - == b"\x80\x02citertools\ndropwhile\nq\x00c__main__\nunderten\nq\x01c__builtin__\niter\nq\x02]q\x03(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08e\x85q\x04Rq\x05K\x00b\x86q\x06Rq\x07K\x00b." -) -assert ( - pickle.dumps(it, 3) - == b"\x80\x03citertools\ndropwhile\nq\x00c__main__\nunderten\nq\x01cbuiltins\niter\nq\x02]q\x03(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08e\x85q\x04Rq\x05K\x00b\x86q\x06Rq\x07K\x00b." -) -assert ( - pickle.dumps(it, 4) - == b"\x80\x04\x95i\x00\x00\x00\x00\x00\x00\x00\x8c\titertools\x94\x8c\tdropwhile\x94\x93\x94\x8c\x08__main__\x94\x8c\x08underten\x94\x93\x94\x8c\x08builtins\x94\x8c\x04iter\x94\x93\x94]\x94(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08e\x85\x94R\x94K\x00b\x86\x94R\x94K\x00b." -) -assert ( - pickle.dumps(it, 5) - == b"\x80\x05\x95i\x00\x00\x00\x00\x00\x00\x00\x8c\titertools\x94\x8c\tdropwhile\x94\x93\x94\x8c\x08__main__\x94\x8c\x08underten\x94\x93\x94\x8c\x08builtins\x94\x8c\x04iter\x94\x93\x94]\x94(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08e\x85\x94R\x94K\x00b\x86\x94R\x94K\x00b." -) -next(it) -assert ( - pickle.dumps(it, 0) - == b"citertools\ndropwhile\np0\n(c__main__\nunderten\np1\nc__builtin__\niter\np2\n((lp3\nI1\naI3\naI5\naI20\naI2\naI4\naI6\naI8\natp4\nRp5\nI4\nbtp6\nRp7\nI1\nb." -) -assert ( - pickle.dumps(it, 1) - == b"citertools\ndropwhile\nq\x00(c__main__\nunderten\nq\x01c__builtin__\niter\nq\x02(]q\x03(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08etq\x04Rq\x05K\x04btq\x06Rq\x07K\x01b." -) -assert ( - pickle.dumps(it, 2) - == b"\x80\x02citertools\ndropwhile\nq\x00c__main__\nunderten\nq\x01c__builtin__\niter\nq\x02]q\x03(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08e\x85q\x04Rq\x05K\x04b\x86q\x06Rq\x07K\x01b." -) -assert ( - pickle.dumps(it, 3) - == b"\x80\x03citertools\ndropwhile\nq\x00c__main__\nunderten\nq\x01cbuiltins\niter\nq\x02]q\x03(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08e\x85q\x04Rq\x05K\x04b\x86q\x06Rq\x07K\x01b." -) -assert ( - pickle.dumps(it, 4) - == b"\x80\x04\x95i\x00\x00\x00\x00\x00\x00\x00\x8c\titertools\x94\x8c\tdropwhile\x94\x93\x94\x8c\x08__main__\x94\x8c\x08underten\x94\x93\x94\x8c\x08builtins\x94\x8c\x04iter\x94\x93\x94]\x94(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08e\x85\x94R\x94K\x04b\x86\x94R\x94K\x01b." -) -assert ( - pickle.dumps(it, 5) - == b"\x80\x05\x95i\x00\x00\x00\x00\x00\x00\x00\x8c\titertools\x94\x8c\tdropwhile\x94\x93\x94\x8c\x08__main__\x94\x8c\x08underten\x94\x93\x94\x8c\x08builtins\x94\x8c\x04iter\x94\x93\x94]\x94(K\x01K\x03K\x05K\x14K\x02K\x04K\x06K\x08e\x85\x94R\x94K\x04b\x86\x94R\x94K\x01b." -) -for proto in range(pickle.HIGHEST_PROTOCOL + 1): - assert next(pickle.loads(pickle.dumps(it, proto))) == 2 - # itertools.accumulate it = itertools.accumulate([6, 3, 7, 1, 0, 9, 8, 8]) diff --git a/extra_tests/snippets/stdlib_socket.py b/extra_tests/snippets/stdlib_socket.py index b49fdcf08c2..3f56d2b926e 100644 --- a/extra_tests/snippets/stdlib_socket.py +++ b/extra_tests/snippets/stdlib_socket.py @@ -131,8 +131,9 @@ with assert_raises(OSError): socket.inet_aton("test") -with assert_raises(OverflowError): - socket.htonl(-1) +# TODO: RUSTPYTHON +# with assert_raises(ValueError): +# socket.htonl(-1) assert socket.htonl(0) == 0 assert socket.htonl(10) == 167772160 diff --git a/extra_tests/snippets/stdlib_typing.py b/extra_tests/snippets/stdlib_typing.py index 681790abd0b..07348945842 100644 --- a/extra_tests/snippets/stdlib_typing.py +++ b/extra_tests/snippets/stdlib_typing.py @@ -12,3 +12,26 @@ def abort_signal_handler( # Ensure PEP 604 unions work with typing.Callable aliases. TracebackFilter = bool | Callable[[int], int] + + +# Test that Union/Optional in function parameter annotations work correctly. +# This tests that annotation scopes can access global implicit symbols (like Union) +# that are imported at module level but not explicitly bound in the function scope. +# Regression test for: rich +from typing import Optional, Union + + +def function_with_union_param(x: Optional[Union[int, str]] = None) -> None: + pass + + +class ClassWithUnionParams: + def __init__( + self, + color: Optional[Union[str, int]] = None, + bold: Optional[bool] = None, + ) -> None: + pass + + def method(self, value: Union[int, float]) -> Union[str, bytes]: + return str(value) diff --git a/extra_tests/snippets/syntax_assignment.py b/extra_tests/snippets/syntax_assignment.py index 8635dc5d795..851558a9db0 100644 --- a/extra_tests/snippets/syntax_assignment.py +++ b/extra_tests/snippets/syntax_assignment.py @@ -59,7 +59,18 @@ def g(): assert a == 1337 assert b == False -assert __annotations__['a'] == bool +# PEP 649: In Python 3.14, __annotations__ is not automatically defined at module level +# Accessing it raises NameError +from testutils import assert_raises + +with assert_raises(NameError): + __annotations__ + +# Use __annotate__ to get annotations (PEP 649) +assert callable(__annotate__) +annotations = __annotate__(1) # 1 = FORMAT_VALUE +assert annotations['a'] == bool +assert annotations['b'] == bool n = 0 diff --git a/extra_tests/snippets/syntax_forbidden_name.py b/extra_tests/snippets/syntax_forbidden_name.py index 2e114fe8800..3bd8148436e 100644 --- a/extra_tests/snippets/syntax_forbidden_name.py +++ b/extra_tests/snippets/syntax_forbidden_name.py @@ -21,6 +21,12 @@ def raisesSyntaxError(parse_stmt, exec_stmt=None): raisesSyntaxError("", "del __debug__") raisesSyntaxError("", "(a, __debug__, c) = (1, 2, 3)") raisesSyntaxError("", "(a, *__debug__, c) = (1, 2, 3)") +raisesSyntaxError("", "__debug__ : int") +raisesSyntaxError("", "__debug__ : int = 1") -# TODO: -# raisesSyntaxError("", "__debug__ : int") +# Import statements +raisesSyntaxError("import sys as __debug__") +raisesSyntaxError("from sys import path as __debug__") + +# Comprehension iteration targets +raisesSyntaxError("[x for __debug__ in range(5)]") diff --git a/extra_tests/snippets/syntax_function2.py b/extra_tests/snippets/syntax_function2.py index d0901af6a14..4a04acd51c1 100644 --- a/extra_tests/snippets/syntax_function2.py +++ b/extra_tests/snippets/syntax_function2.py @@ -80,6 +80,7 @@ def nested(): def f7(): + # PEP 649: annotations are deferred, so void is not evaluated at definition time try: def t() -> void: # noqa: F821 pass @@ -87,7 +88,7 @@ def t() -> void: # noqa: F821 return True return False -assert f7() +assert not f7() # PEP 649: no NameError because annotation is deferred def f8() -> int: diff --git a/crawl_sourcecode.py b/scripts/crawl_sourcecode.py similarity index 100% rename from crawl_sourcecode.py rename to scripts/crawl_sourcecode.py diff --git a/scripts/fix_test.py b/scripts/fix_test.py deleted file mode 100644 index 53b10d63834..00000000000 --- a/scripts/fix_test.py +++ /dev/null @@ -1,192 +0,0 @@ -""" -An automated script to mark failures in python test suite. -It adds @unittest.expectedFailure to the test functions that are failing in RustPython, but not in CPython. -As well as marking the test with a TODO comment. - -How to use: -1. Copy a specific test from the CPython repository to the RustPython repository. -2. Remove all unexpected failures from the test and skip the tests that hang. -3. Build RustPython: cargo build --release -4. Run from the project root: - - For single-file tests: python ./scripts/fix_test.py --path ./Lib/test/test_venv.py - - For package tests: python ./scripts/fix_test.py --path ./Lib/test/test_inspect/test_inspect.py -5. Verify: cargo run --release -- -m test test_venv (should pass with expected failures) -6. Actually fix the tests marked with # TODO: RUSTPYTHON -""" - -import argparse -import ast -import itertools -import platform -import sys -from pathlib import Path - - -def parse_args(): - parser = argparse.ArgumentParser(description="Fix test.") - parser.add_argument("--path", type=Path, help="Path to test file") - parser.add_argument("--force", action="store_true", help="Force modification") - parser.add_argument( - "--platform", action="store_true", help="Platform specific failure" - ) - - args = parser.parse_args() - return args - - -class Test: - name: str = "" - path: str = "" - result: str = "" - - def __str__(self): - return f"Test(name={self.name}, path={self.path}, result={self.result})" - - -class TestResult: - tests_result: str = "" - tests = [] - stdout = "" - - def __str__(self): - return f"TestResult(tests_result={self.tests_result},tests={len(self.tests)})" - - -def parse_results(result): - lines = result.stdout.splitlines() - test_results = TestResult() - test_results.stdout = result.stdout - in_test_results = False - for line in lines: - if line == "Run tests sequentially": - in_test_results = True - elif line.startswith("-----------"): - in_test_results = False - if in_test_results and " ... " in line: - line = line.strip() - # Skip lines that don't look like test results - if line.startswith("tests") or line.startswith("["): - continue - # Parse: "test_name (path) [subtest] ... RESULT" - parts = line.split(" ... ") - if len(parts) >= 2: - test_info = parts[0] - result_str = parts[-1].lower() - # Only process FAIL or ERROR - if result_str not in ("fail", "error"): - continue - # Extract test name (first word) - first_space = test_info.find(" ") - if first_space > 0: - test = Test() - test.name = test_info[:first_space] - # Extract path from (path) - rest = test_info[first_space:].strip() - if rest.startswith("("): - end_paren = rest.find(")") - if end_paren > 0: - test.path = rest[1:end_paren] - test.result = result_str - test_results.tests.append(test) - elif "== Tests result: " in line: - res = line.split("== Tests result: ")[1] - res = res.split(" ")[0] - test_results.tests_result = res - return test_results - - -def path_to_test(path) -> list[str]: - # path format: test.module_name[.submodule].ClassName.test_method - # We need [ClassName, test_method] - always the last 2 elements - parts = path.split(".") - return parts[-2:] # Get class name and method name - - -def find_test_lineno(file: str, test: list[str]) -> tuple[int, int] | None: - """Find the line number and column offset of a test function. - Returns (lineno, col_offset) or None if not found. - """ - a = ast.parse(file) - for key, node in ast.iter_fields(a): - if key == "body": - for n in node: - match n: - case ast.ClassDef(): - if len(test) == 2 and test[0] == n.name: - for fn in n.body: - match fn: - case ast.FunctionDef() | ast.AsyncFunctionDef(): - if fn.name == test[-1]: - return (fn.lineno, fn.col_offset) - case ast.FunctionDef() | ast.AsyncFunctionDef(): - if n.name == test[0] and len(test) == 1: - return (n.lineno, n.col_offset) - return None - - -def apply_modifications(file: str, modifications: list[tuple[int, int]]) -> str: - """Apply all modifications in reverse order to avoid line number offset issues.""" - lines = file.splitlines() - fixture = "@unittest.expectedFailure" - # Sort by line number in descending order - modifications.sort(key=lambda x: x[0], reverse=True) - for lineno, col_offset in modifications: - indent = " " * col_offset - lines.insert(lineno - 1, indent + fixture) - lines.insert(lineno - 1, indent + "# TODO: RUSTPYTHON") - return "\n".join(lines) - - -def run_test(test_name): - print(f"Running test: {test_name}") - rustpython_location = "./target/release/rustpython" - if sys.platform == "win32": - rustpython_location += ".exe" - - import subprocess - - result = subprocess.run( - [rustpython_location, "-m", "test", "-v", test_name], - capture_output=True, - text=True, - ) - return parse_results(result) - - -if __name__ == "__main__": - args = parse_args() - test_path = args.path.resolve() - if not test_path.exists(): - print(f"Error: File not found: {test_path}") - sys.exit(1) - # Detect package tests (e.g., test_ctypes/test_random_things.py) - if test_path.parent.name.startswith("test_"): - test_name = f"{test_path.parent.name}.{test_path.stem}" - else: - test_name = test_path.stem - tests = run_test(test_name) - f = test_path.read_text(encoding="utf-8") - - # Collect all modifications first (with deduplication for subtests) - modifications = [] - seen_tests = set() # Track (class_name, method_name) to avoid duplicates - for test in tests.tests: - if test.result == "fail" or test.result == "error": - test_parts = path_to_test(test.path) - test_key = tuple(test_parts) - if test_key in seen_tests: - continue # Skip duplicate (same test, different subtest) - seen_tests.add(test_key) - location = find_test_lineno(f, test_parts) - if location: - print(f"Modifying test: {test.name} at line {location[0]}") - modifications.append(location) - else: - print(f"Warning: Could not find test: {test.name} ({test_parts})") - - # Apply all modifications in reverse order - if modifications: - f = apply_modifications(f, modifications) - test_path.write_text(f, encoding="utf-8") - - print(f"Modified {len(modifications)} tests") diff --git a/scripts/generate_opcode_metadata.py b/scripts/generate_opcode_metadata.py index 9c440d52568..42fb55a7c01 100644 --- a/scripts/generate_opcode_metadata.py +++ b/scripts/generate_opcode_metadata.py @@ -1,209 +1,61 @@ -"""Generate _opcode_metadata.py for RustPython bytecode. +""" +Generate Lib/_opcode_metadata.py for RustPython bytecode. This file generates opcode metadata that is compatible with CPython 3.13. -RustPython's Instruction enum is now ordered to match CPython opcode numbers exactly. """ +import itertools +import pathlib import re +import typing + +ROOT = pathlib.Path(__file__).parents[1] +BYTECODE_FILE = ( + ROOT / "crates" / "compiler-core" / "src" / "bytecode" / "instruction.rs" +) +OPCODE_METADATA_FILE = ROOT / "Lib" / "_opcode_metadata.py" + + +class Opcode(typing.NamedTuple): + rust_name: str + id: int + + @property + def cpython_name(self) -> str: + name = re.sub(r"(?<=[a-z0-9])([A-Z])", r"_\1", self.rust_name) + return re.sub(r"(\D)(\d+)$", r"\1_\2", name).upper() + + @classmethod + def from_str(cls, body: str): + raw_variants = re.split(r"(\d+),", body.strip()) + raw_variants.remove("") + for raw_name, raw_id in itertools.batched(raw_variants, 2): + name = re.findall(r"\b[A-Z][A-Za-z]*\d*\b(?=\s*[\({=])", raw_name)[0] + yield cls(rust_name=name.strip(), id=int(raw_id)) -# Read the bytecode.rs file to get instruction names -with open("crates/compiler-core/src/bytecode.rs", "r") as f: - content = f.read() - -# Find the Instruction enum -match = re.search(r"pub enum Instruction \{(.+?)\n\}", content, re.DOTALL) -if not match: - raise ValueError("Could not find Instruction enum") - -enum_body = match.group(1) - -# Extract variant names -variants = [] -for line in enum_body.split("\n"): - if line.strip().startswith("///") or line.strip().startswith("//"): - continue - m = re.match(r"^\s+([A-Z][a-zA-Z0-9]*)", line) - if m: - variants.append(m.group(1)) - -print(f"Found {len(variants)} instruction variants") - -# Map RustPython variant names to CPython-compatible names -# The opcode number is the index in the Instruction enum -name_mapping = { - # Dummy/placeholder instructions - "Cache": "CACHE", - "Reserved3": "RESERVED", - "Reserved17": "RESERVED", - "Reserved141": "RESERVED", - "Reserved142": "RESERVED", - "Reserved143": "RESERVED", - "Reserved144": "RESERVED", - "Reserved145": "RESERVED", - "Reserved146": "RESERVED", - "Reserved147": "RESERVED", - "Reserved148": "RESERVED", - "BinarySlice": "BINARY_SLICE", - "EndFor": "END_FOR", - "ExitInitCheck": "EXIT_INIT_CHECK", - "GetYieldFromIter": "GET_YIELD_FROM_ITER", - "InterpreterExit": "INTERPRETER_EXIT", - "LoadAssertionError": "LOAD_ASSERTION_ERROR", - "LoadLocals": "LOAD_LOCALS", - "PushNull": "PUSH_NULL", - "ReturnGenerator": "RETURN_GENERATOR", - "StoreSlice": "STORE_SLICE", - "UnaryInvert": "UNARY_INVERT", - "UnaryNegative": "UNARY_NEGATIVE", - "UnaryNot": "UNARY_NOT", - "BuildConstKeyMap": "BUILD_CONST_KEY_MAP", - "CopyFreeVars": "COPY_FREE_VARS", - "DictMerge": "DICT_MERGE", - "EnterExecutor": "ENTER_EXECUTOR", - "JumpBackward": "JUMP_BACKWARD", - "JumpBackwardNoInterrupt": "JUMP_BACKWARD_NO_INTERRUPT", - "JumpForward": "JUMP_FORWARD", - "ListExtend": "LIST_EXTEND", - "LoadFastCheck": "LOAD_FAST_CHECK", - "LoadFastLoadFast": "LOAD_FAST_LOAD_FAST", - "LoadFromDictOrDeref": "LOAD_FROM_DICT_OR_DEREF", - "LoadFromDictOrGlobals": "LOAD_FROM_DICT_OR_GLOBALS", - "LoadSuperAttr": "LOAD_SUPER_ATTR", - "MakeCell": "MAKE_CELL", - "PopJumpIfNone": "POP_JUMP_IF_NONE", - "PopJumpIfNotNone": "POP_JUMP_IF_NOT_NONE", - "SetUpdate": "SET_UPDATE", - "StoreFastStoreFast": "STORE_FAST_STORE_FAST", - # Real instructions - "BeforeAsyncWith": "BEFORE_ASYNC_WITH", - "BeforeWith": "BEFORE_WITH", - "BinaryOp": "BINARY_OP", - "BinarySubscript": "BINARY_SUBSCR", - "Break": "BREAK", - "BuildList": "BUILD_LIST", - "BuildListFromTuples": "BUILD_LIST_UNPACK", - "BuildMap": "BUILD_MAP", - "BuildMapForCall": "BUILD_MAP_FOR_CALL", - "BuildSet": "BUILD_SET", - "BuildSetFromTuples": "BUILD_SET_UNPACK", - "BuildSlice": "BUILD_SLICE", - "BuildString": "BUILD_STRING", - "BuildTuple": "BUILD_TUPLE", - "BuildTupleFromIter": "BUILD_TUPLE_ITER", - "BuildTupleFromTuples": "BUILD_TUPLE_UNPACK", - "CallFunctionEx": "CALL_FUNCTION_EX", - "CallFunctionKeyword": "CALL_KW", - "CallFunctionPositional": "CALL", - "CallIntrinsic1": "CALL_INTRINSIC_1", - "CallIntrinsic2": "CALL_INTRINSIC_2", - "CallMethodEx": "CALL_METHOD_EX", - "CallMethodKeyword": "CALL_METHOD_KW", - "CallMethodPositional": "CALL_METHOD", - "CheckEgMatch": "CHECK_EG_MATCH", - "CheckExcMatch": "CHECK_EXC_MATCH", - "CleanupThrow": "CLEANUP_THROW", - "CompareOperation": "COMPARE_OP", - "ContainsOp": "CONTAINS_OP", - "Continue": "CONTINUE", - "ConvertValue": "CONVERT_VALUE", - "CopyItem": "COPY", - "DeleteAttr": "DELETE_ATTR", - "DeleteDeref": "DELETE_DEREF", - "DeleteFast": "DELETE_FAST", - "DeleteGlobal": "DELETE_GLOBAL", - "DeleteLocal": "DELETE_NAME", - "DeleteSubscript": "DELETE_SUBSCR", - "DictUpdate": "DICT_UPDATE", - "EndAsyncFor": "END_ASYNC_FOR", - "EndSend": "END_SEND", - "ExtendedArg": "EXTENDED_ARG", - "ForIter": "FOR_ITER", - "FormatSimple": "FORMAT_SIMPLE", - "FormatWithSpec": "FORMAT_WITH_SPEC", - "GetAIter": "GET_AITER", - "GetANext": "GET_ANEXT", - "GetAwaitable": "GET_AWAITABLE", - "GetIter": "GET_ITER", - "GetLen": "GET_LEN", - "ImportFrom": "IMPORT_FROM", - "ImportName": "IMPORT_NAME", - "IsOp": "IS_OP", - "Jump": "JUMP", - "JumpIfFalseOrPop": "JUMP_IF_FALSE_OR_POP", - "JumpIfNotExcMatch": "JUMP_IF_NOT_EXC_MATCH", - "JumpIfTrueOrPop": "JUMP_IF_TRUE_OR_POP", - "ListAppend": "LIST_APPEND", - "LoadAttr": "LOAD_ATTR", - "LoadBuildClass": "LOAD_BUILD_CLASS", - "LoadClassDeref": "LOAD_CLASSDEREF", - "LoadClosure": "LOAD_CLOSURE", - "LoadConst": "LOAD_CONST", - "LoadDeref": "LOAD_DEREF", - "LoadFast": "LOAD_FAST", - "LoadFastAndClear": "LOAD_FAST_AND_CLEAR", - "LoadGlobal": "LOAD_GLOBAL", - "LoadMethod": "LOAD_METHOD", - "LoadNameAny": "LOAD_NAME", - "MakeFunction": "MAKE_FUNCTION", - "MapAdd": "MAP_ADD", - "MatchClass": "MATCH_CLASS", - "MatchKeys": "MATCH_KEYS", - "MatchMapping": "MATCH_MAPPING", - "MatchSequence": "MATCH_SEQUENCE", - "Nop": "NOP", - "PopBlock": "POP_BLOCK", - "PopException": "POP_EXCEPT", - "PopJumpIfFalse": "POP_JUMP_IF_FALSE", - "PopJumpIfTrue": "POP_JUMP_IF_TRUE", - "PopTop": "POP_TOP", - "PushExcInfo": "PUSH_EXC_INFO", - "Raise": "RAISE_VARARGS", - "Reraise": "RERAISE", - "Resume": "RESUME", - "ReturnConst": "RETURN_CONST", - "ReturnValue": "RETURN_VALUE", - "Reverse": "REVERSE", - "Send": "SEND", - "SetAdd": "SET_ADD", - "SetExcInfo": "SET_EXC_INFO", - "SetFunctionAttribute": "SET_FUNCTION_ATTRIBUTE", - "SetupAnnotation": "SETUP_ANNOTATIONS", - "StoreAttr": "STORE_ATTR", - "StoreDeref": "STORE_DEREF", - "StoreFast": "STORE_FAST", - "StoreFastLoadFast": "STORE_FAST_LOAD_FAST", - "StoreGlobal": "STORE_GLOBAL", - "StoreLocal": "STORE_NAME", - "StoreSubscript": "STORE_SUBSCR", - "Subscript": "SUBSCRIPT", - "Swap": "SWAP", - "ToBool": "TO_BOOL", - "UnaryOperation": "UNARY_OP", - "UnpackEx": "UNPACK_EX", - "UnpackSequence": "UNPACK_SEQUENCE", - "WithExceptStart": "WITH_EXCEPT_START", - "YieldValue": "YIELD_VALUE", -} - -# Build opmap with RustPython instruction indices -opmap = {} -rust_to_cpython_name = {} -for i, variant in enumerate(variants): - cpython_name = name_mapping.get(variant, variant.upper()) - # Skip adding duplicates (RESERVED appears multiple times) - if cpython_name == "RESERVED": - # Use unique names for reserved slots - cpython_name = f"RESERVED_{i}" - if cpython_name not in opmap: - opmap[cpython_name] = i - rust_to_cpython_name[variant] = cpython_name - - -# Find specific instruction indices for categorization -def find_opcode(cpython_name): - return opmap.get(cpython_name, -1) + def __lt__(self, other: typing.Self) -> bool: + return self.id < other.id +def extract_enum_body(contents: str, enum_name: str) -> str: + res = re.search(f"pub enum {enum_name} " + r"\{(.+?)\n\}", contents, re.DOTALL) + if not res: + raise ValueError(f"Could not find {enum_name} enum") + + return "\n".join( + line.split("//")[0].strip() # Remove any comment. i.e. "foo // some comment" + for line in res.group(1).splitlines() + if not line.strip().startswith("//") # Ignore comment lines + ) + + +contents = BYTECODE_FILE.read_text(encoding="utf-8") +enum_body = "\n".join( + extract_enum_body(contents, enum_name) + for enum_name in ("Instruction", "PseudoInstruction") +) +opcodes = list(Opcode.from_str(enum_body)) + # Generate the output file output = """# This file is generated by scripts/generate_opcode_metadata.py # for RustPython bytecode format (CPython 3.13 compatible opcode numbers). @@ -216,8 +68,8 @@ def find_opcode(cpython_name): opmap = { """ -for name, num in sorted(opmap.items(), key=lambda x: x[1]): - output += f" '{name}': {num},\n" +for opcode in sorted(opcodes): + output += f" '{opcode.cpython_name}': {opcode.id},\n" output += """} @@ -226,21 +78,4 @@ def find_opcode(cpython_name): MIN_INSTRUMENTED_OPCODE = 236 """ -with open("Lib/_opcode_metadata.py", "w") as f: - f.write(output) - -print("Generated Lib/_opcode_metadata.py") -print("\nKey opcode indices (matching CPython 3.13):") -print(f" CACHE = {find_opcode('CACHE')} (expected: 0)") -print(f" BEFORE_ASYNC_WITH = {find_opcode('BEFORE_ASYNC_WITH')} (expected: 1)") -print(f" BINARY_SUBSCR = {find_opcode('BINARY_SUBSCR')} (expected: 5)") -print( - f" WITH_EXCEPT_START = {find_opcode('WITH_EXCEPT_START')} (expected: 44, HAVE_ARGUMENT)" -) -print(f" BINARY_OP = {find_opcode('BINARY_OP')} (expected: 45)") -print(f" LOAD_CONST = {find_opcode('LOAD_CONST')} (expected: 83)") -print(f" LOAD_FAST = {find_opcode('LOAD_FAST')} (expected: 85)") -print(f" LOAD_GLOBAL = {find_opcode('LOAD_GLOBAL')} (expected: 91)") -print(f" STORE_FAST = {find_opcode('STORE_FAST')} (expected: 110)") -print(f" YIELD_VALUE = {find_opcode('YIELD_VALUE')} (expected: 118)") -print(f" RESUME = {find_opcode('RESUME')} (expected: 149)") +OPCODE_METADATA_FILE.write_text(output, encoding="utf-8") diff --git a/scripts/libc_posix.py b/scripts/libc_posix.py index 73f082a0658..be375aebe8f 100644 --- a/scripts/libc_posix.py +++ b/scripts/libc_posix.py @@ -13,7 +13,7 @@ ) # TODO: Exclude matches if they have `(` after (those are functions) -LIBC_VERSION = "0.2.177" +LIBC_VERSION = "0.2.180" EXCLUDE = frozenset( { @@ -96,7 +96,7 @@ def format_groups(groups: dict) -> "Iterator[tuple[str, str]]": def main(): wanted_consts = get_consts( - "https://docs.python.org/3.13/library/os.html", # Should we read from https://github.com/python/cpython/blob/bcee1c322115c581da27600f2ae55e5439c027eb/Modules/posixmodule.c#L17023 instead? + "https://docs.python.org/3.14/library/os.html", # Should we read from https://github.com/python/cpython/blob/bcee1c322115c581da27600f2ae55e5439c027eb/Modules/posixmodule.c#L17023 instead? pattern=OS_CONSTS_PAT, ) available = { diff --git a/scripts/update_lib/.gitignore b/scripts/update_lib/.gitignore new file mode 100644 index 00000000000..ceddaa37f12 --- /dev/null +++ b/scripts/update_lib/.gitignore @@ -0,0 +1 @@ +.cache/ diff --git a/scripts/update_lib/__init__.py b/scripts/update_lib/__init__.py new file mode 100644 index 00000000000..ccb2628d6a4 --- /dev/null +++ b/scripts/update_lib/__init__.py @@ -0,0 +1,37 @@ +""" +Library for updating Python test files with RustPython-specific patches. +""" + +from .patch_spec import ( + COMMENT, + DEFAULT_INDENT, + UT, + PatchEntry, + Patches, + PatchSpec, + UtMethod, + apply_patches, + build_patch_dict, + extract_patches, + iter_patches, + iter_tests, + patches_from_json, + patches_to_json, +) + +__all__ = [ + "COMMENT", + "DEFAULT_INDENT", + "UT", + "Patches", + "PatchEntry", + "PatchSpec", + "UtMethod", + "apply_patches", + "build_patch_dict", + "extract_patches", + "iter_patches", + "iter_tests", + "patches_from_json", + "patches_to_json", +] diff --git a/scripts/update_lib/__main__.py b/scripts/update_lib/__main__.py new file mode 100644 index 00000000000..49399db6f43 --- /dev/null +++ b/scripts/update_lib/__main__.py @@ -0,0 +1,104 @@ +#!/usr/bin/env python +""" +Update library tools for RustPython. + +Usage: + python scripts/update_lib quick cpython/Lib/test/test_foo.py + python scripts/update_lib copy-lib cpython/Lib/dataclasses.py + python scripts/update_lib migrate cpython/Lib/test/test_foo.py + python scripts/update_lib patches --from Lib/test/foo.py --to cpython/Lib/test/foo.py + python scripts/update_lib auto-mark Lib/test/test_foo.py +""" + +import argparse +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent)) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description="Update library tools for RustPython", + ) + + subparsers = parser.add_subparsers(dest="command", required=True) + + subparsers.add_parser( + "quick", + help="Quick update: patch + auto-mark (recommended)", + add_help=False, + ) + subparsers.add_parser( + "migrate", + help="Migrate test file(s) from CPython, preserving RustPython markers", + add_help=False, + ) + subparsers.add_parser( + "patches", + help="Patch management (extract/apply patches between files)", + add_help=False, + ) + subparsers.add_parser( + "auto-mark", + help="Run tests and auto-mark failures with @expectedFailure", + add_help=False, + ) + subparsers.add_parser( + "copy-lib", + help="Copy library file/directory from CPython (delete existing first)", + add_help=False, + ) + subparsers.add_parser( + "deps", + help="Show dependency information for a module", + add_help=False, + ) + subparsers.add_parser( + "todo", + help="Show prioritized list of modules to update", + add_help=False, + ) + + args, remaining = parser.parse_known_args(argv) + + if args.command == "quick": + from update_lib.cmd_quick import main as quick_main + + return quick_main(remaining) + + if args.command == "copy-lib": + from update_lib.cmd_copy_lib import main as copy_lib_main + + return copy_lib_main(remaining) + + if args.command == "migrate": + from update_lib.cmd_migrate import main as migrate_main + + return migrate_main(remaining) + + if args.command == "patches": + from update_lib.cmd_patches import main as patches_main + + return patches_main(remaining) + + if args.command == "auto-mark": + from update_lib.cmd_auto_mark import main as cmd_auto_mark_main + + return cmd_auto_mark_main(remaining) + + if args.command == "deps": + from update_lib.cmd_deps import main as cmd_deps_main + + return cmd_deps_main(remaining) + + if args.command == "todo": + from update_lib.cmd_todo import main as cmd_todo_main + + return cmd_todo_main(remaining) + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/update_lib/cmd_auto_mark.py b/scripts/update_lib/cmd_auto_mark.py new file mode 100644 index 00000000000..c77cbf300f1 --- /dev/null +++ b/scripts/update_lib/cmd_auto_mark.py @@ -0,0 +1,1044 @@ +#!/usr/bin/env python +""" +Auto-mark test failures in Python test suite. + +This module provides functions to: +- Run tests with RustPython and parse results +- Extract test names from test file paths +- Mark failing tests with @unittest.expectedFailure +- Remove expectedFailure from tests that now pass +""" + +import ast +import pathlib +import re +import subprocess +import sys +from dataclasses import dataclass, field + +sys.path.insert(0, str(pathlib.Path(__file__).parent.parent)) + +from update_lib import COMMENT, PatchSpec, UtMethod, apply_patches +from update_lib.file_utils import get_test_module_name + + +class TestRunError(Exception): + """Raised when test run fails entirely (e.g., import error, crash).""" + + pass + + +@dataclass +class Test: + name: str = "" + path: str = "" + result: str = "" + error_message: str = "" + + +@dataclass +class TestResult: + tests_result: str = "" + tests: list[Test] = field(default_factory=list) + unexpected_successes: list[Test] = field(default_factory=list) + stdout: str = "" + + +def run_test(test_name: str, skip_build: bool = False) -> TestResult: + """ + Run a test with RustPython and return parsed results. + + Args: + test_name: Test module name (e.g., "test_foo" or "test_ctypes.test_bar") + skip_build: If True, use pre-built binary instead of cargo run + + Returns: + TestResult with parsed test results + """ + if skip_build: + cmd = ["./target/release/rustpython"] + if sys.platform == "win32": + cmd = ["./target/release/rustpython.exe"] + else: + cmd = ["cargo", "run", "--release", "--"] + + result = subprocess.run( + cmd + ["-m", "test", "-v", "-u", "all", "--slowest", test_name], + stdout=subprocess.PIPE, # Capture stdout for parsing + stderr=None, # Let stderr pass through to terminal + text=True, + ) + return parse_results(result) + + +def _try_parse_test_info(test_info: str) -> tuple[str, str] | None: + """Try to extract (name, path) from 'test_name (path)' or 'test_name (path) [subtest]'.""" + first_space = test_info.find(" ") + if first_space > 0: + name = test_info[:first_space] + rest = test_info[first_space:].strip() + if rest.startswith("("): + end_paren = rest.find(")") + if end_paren > 0: + return name, rest[1:end_paren] + return None + + +def parse_results(result: subprocess.CompletedProcess) -> TestResult: + """Parse subprocess result into TestResult.""" + lines = result.stdout.splitlines() + test_results = TestResult() + test_results.stdout = result.stdout + in_test_results = False + # For multiline format: "test_name (path)\ndocstring ... RESULT" + pending_test_info = None + + for line in lines: + if re.search(r"Run \d+ tests? sequentially", line): + in_test_results = True + elif "== Tests result: " in line: + in_test_results = False + + if in_test_results and " ... " in line: + stripped = line.strip() + # Skip lines that don't look like test results + if stripped.startswith("tests") or stripped.startswith("["): + pending_test_info = None + continue + # Parse: "test_name (path) [subtest] ... RESULT" + parts = stripped.split(" ... ") + if len(parts) >= 2: + test_info = parts[0] + result_str = parts[-1].lower() + # Only process FAIL or ERROR + if result_str not in ("fail", "error"): + pending_test_info = None + continue + # Try parsing from this line (single-line format) + parsed = _try_parse_test_info(test_info) + if not parsed and pending_test_info: + # Multiline format: previous line had test_name (path) + parsed = _try_parse_test_info(pending_test_info) + if parsed: + test = Test() + test.name, test.path = parsed + test.result = result_str + test_results.tests.append(test) + pending_test_info = None + + elif in_test_results: + # Track test info for multiline format: + # test_name (path) + # docstring ... RESULT + stripped = line.strip() + if ( + stripped + and "(" in stripped + and stripped.endswith(")") + and ":" not in stripped.split("(")[0] + ): + pending_test_info = stripped + else: + pending_test_info = None + + # Also check for Tests result on non-" ... " lines + if "== Tests result: " in line: + res = line.split("== Tests result: ")[1] + res = res.split(" ")[0] + test_results.tests_result = res + + elif "== Tests result: " in line: + res = line.split("== Tests result: ")[1] + res = res.split(" ")[0] + test_results.tests_result = res + + # Parse: "UNEXPECTED SUCCESS: test_name (path)" + if line.startswith("UNEXPECTED SUCCESS: "): + rest = line[len("UNEXPECTED SUCCESS: ") :] + # Format: "test_name (path)" + first_space = rest.find(" ") + if first_space > 0: + test = Test() + test.name = rest[:first_space] + path_part = rest[first_space:].strip() + if path_part.startswith("(") and path_part.endswith(")"): + test.path = path_part[1:-1] + test.result = "unexpected_success" + test_results.unexpected_successes.append(test) + + # Parse error details to extract error messages + _parse_error_details(test_results, lines) + + return test_results + + +def _parse_error_details(test_results: TestResult, lines: list[str]) -> None: + """Parse error details section to extract error messages for each test.""" + # Build a lookup dict for tests by (name, path) + test_lookup: dict[tuple[str, str], Test] = {} + for test in test_results.tests: + test_lookup[(test.name, test.path)] = test + + # Parse error detail blocks + # Format: + # ====================================================================== + # FAIL: test_name (path) + # ---------------------------------------------------------------------- + # Traceback (most recent call last): + # ... + # AssertionError: message + # + # ====================================================================== + i = 0 + while i < len(lines): + line = lines[i] + # Look for FAIL: or ERROR: header + if line.startswith(("FAIL: ", "ERROR: ")): + # Parse: "FAIL: test_name (path)" or "ERROR: test_name (path)" + header = line.split(": ", 1)[1] if ": " in line else "" + first_space = header.find(" ") + if first_space > 0: + test_name = header[:first_space] + path_part = header[first_space:].strip() + if path_part.startswith("(") and path_part.endswith(")"): + test_path = path_part[1:-1] + + # Find the last non-empty line before the next separator or end + error_lines = [] + i += 1 + # Skip the separator line + if i < len(lines) and lines[i].startswith("-----"): + i += 1 + + # Collect lines until the next separator or end + while i < len(lines): + current = lines[i] + if current.startswith("=====") or current.startswith("-----"): + break + error_lines.append(current) + i += 1 + + # Find the last non-empty line (the error message) + error_message = "" + for err_line in reversed(error_lines): + stripped = err_line.strip() + if stripped: + error_message = stripped + break + + # Update the test with the error message + if (test_name, test_path) in test_lookup: + test_lookup[ + (test_name, test_path) + ].error_message = error_message + + continue + i += 1 + + +def path_to_test_parts(path: str) -> list[str]: + """ + Extract [ClassName, method_name] from test path. + + Args: + path: Test path like "test.module_name.ClassName.test_method" + + Returns: + [ClassName, method_name] - last 2 elements + """ + parts = path.split(".") + return parts[-2:] + + +def _expand_stripped_to_children( + contents: str, + stripped_tests: set[tuple[str, str]], + all_failing_tests: set[tuple[str, str]], +) -> set[tuple[str, str]]: + """Find child-class failures that correspond to stripped parent-class markers. + + When ``strip_reasonless_expected_failures`` removes a marker from a parent + (mixin) class, test failures are reported against the concrete subclasses, + not the parent itself. This function maps those child failures back so + they get re-marked (and later consolidated to the parent by + ``_consolidate_to_parent``). + + Returns the set of ``(class, method)`` pairs from *all_failing_tests* that + should be re-marked. + """ + # Direct matches (stripped test itself is a concrete TestCase) + result = stripped_tests & all_failing_tests + + unmatched = stripped_tests - all_failing_tests + if not unmatched: + return result + + tree = ast.parse(contents) + class_bases, class_methods = _build_inheritance_info(tree) + + for parent_cls, method_name in unmatched: + if method_name not in class_methods.get(parent_cls, set()): + continue + for cls in _find_all_inheritors( + parent_cls, method_name, class_bases, class_methods + ): + if (cls, method_name) in all_failing_tests: + result.add((cls, method_name)) + + return result + + +def _consolidate_to_parent( + contents: str, + failing_tests: set[tuple[str, str]], + error_messages: dict[tuple[str, str], str] | None = None, +) -> tuple[set[tuple[str, str]], dict[tuple[str, str], str] | None]: + """Move failures to the parent class when ALL inheritors fail. + + If every concrete subclass that inherits a method from a parent class + appears in *failing_tests*, replace those per-subclass entries with a + single entry on the parent. This avoids creating redundant super-call + overrides in every child. + + Returns: + (consolidated_failing_tests, consolidated_error_messages) + """ + tree = ast.parse(contents) + class_bases, class_methods = _build_inheritance_info(tree) + + # Group by (defining_parent, method) → set of failing children + from collections import defaultdict + + groups: dict[tuple[str, str], set[str]] = defaultdict(set) + for class_name, method_name in failing_tests: + defining = _find_method_definition( + class_name, method_name, class_bases, class_methods + ) + if defining and defining != class_name: + groups[(defining, method_name)].add(class_name) + + if not groups: + return failing_tests, error_messages + + result = set(failing_tests) + new_error_messages = dict(error_messages) if error_messages else {} + + for (parent, method_name), failing_children in groups.items(): + all_inheritors = _find_all_inheritors( + parent, method_name, class_bases, class_methods + ) + + if all_inheritors and failing_children >= all_inheritors: + # All inheritors fail → mark on parent instead + children_keys = {(child, method_name) for child in failing_children} + result -= children_keys + result.add((parent, method_name)) + # Pick any child's error message for the parent + if new_error_messages: + for child in failing_children: + msg = new_error_messages.pop((child, method_name), "") + if msg: + new_error_messages[(parent, method_name)] = msg + + return result, new_error_messages or error_messages + + +def build_patches( + test_parts_set: set[tuple[str, str]], + error_messages: dict[tuple[str, str], str] | None = None, +) -> dict: + """Convert failing tests to patch format.""" + patches = {} + error_messages = error_messages or {} + for class_name, method_name in sorted(test_parts_set): + if class_name not in patches: + patches[class_name] = {} + reason = error_messages.get((class_name, method_name), "") + patches[class_name][method_name] = [ + PatchSpec(UtMethod.ExpectedFailure, None, reason) + ] + return patches + + +def _is_super_call_only(func_node: ast.FunctionDef | ast.AsyncFunctionDef) -> bool: + """Check if the method body is just 'return super().method_name()' or 'return await super().method_name()'.""" + if len(func_node.body) != 1: + return False + stmt = func_node.body[0] + if not isinstance(stmt, ast.Return) or stmt.value is None: + return False + call = stmt.value + # Unwrap await for async methods + if isinstance(call, ast.Await): + call = call.value + if not isinstance(call, ast.Call): + return False + if not isinstance(call.func, ast.Attribute): + return False + # Verify the method name matches + if call.func.attr != func_node.name: + return False + super_call = call.func.value + if not isinstance(super_call, ast.Call): + return False + if not isinstance(super_call.func, ast.Name) or super_call.func.id != "super": + return False + return True + + +def _method_removal_range( + func_node: ast.FunctionDef | ast.AsyncFunctionDef, lines: list[str] +) -> range: + """Line range covering an entire method including decorators and a preceding COMMENT line.""" + first = ( + func_node.decorator_list[0].lineno - 1 + if func_node.decorator_list + else func_node.lineno - 1 + ) + if ( + first > 0 + and lines[first - 1].strip().startswith("#") + and COMMENT in lines[first - 1] + ): + first -= 1 + # Also remove a preceding blank line to avoid double-blanks after removal + if first > 0 and not lines[first - 1].strip(): + first -= 1 + return range(first, func_node.end_lineno) + + +def _build_inheritance_info(tree: ast.Module) -> tuple[dict, dict]: + """ + Build inheritance information from AST. + + Returns: + class_bases: dict[str, list[str]] - parent classes for each class + class_methods: dict[str, set[str]] - methods directly defined in each class + """ + all_classes = { + node.name for node in ast.walk(tree) if isinstance(node, ast.ClassDef) + } + class_bases = {} + class_methods = {} + + for node in ast.walk(tree): + if isinstance(node, ast.ClassDef): + bases = [ + base.id + for base in node.bases + if isinstance(base, ast.Name) and base.id in all_classes + ] + class_bases[node.name] = bases + methods = { + item.name + for item in node.body + if isinstance(item, (ast.FunctionDef, ast.AsyncFunctionDef)) + } + class_methods[node.name] = methods + + return class_bases, class_methods + + +def _find_method_definition( + class_name: str, method_name: str, class_bases: dict, class_methods: dict +) -> str | None: + """Find the class where a method is actually defined (BFS).""" + if method_name in class_methods.get(class_name, set()): + return class_name + + visited = set() + queue = list(class_bases.get(class_name, [])) + + while queue: + current = queue.pop(0) + if current in visited: + continue + visited.add(current) + + if method_name in class_methods.get(current, set()): + return current + queue.extend(class_bases.get(current, [])) + + return None + + +def _find_all_inheritors( + parent: str, method_name: str, class_bases: dict, class_methods: dict +) -> set[str]: + """Find all classes that inherit *method_name* from *parent* (not overriding it).""" + return { + cls + for cls in class_bases + if cls != parent + and method_name not in class_methods.get(cls, set()) + and _find_method_definition(cls, method_name, class_bases, class_methods) + == parent + } + + +def remove_expected_failures( + contents: str, tests_to_remove: set[tuple[str, str]] +) -> str: + """Remove @unittest.expectedFailure decorators from tests that now pass.""" + if not tests_to_remove: + return contents + + tree = ast.parse(contents) + lines = contents.splitlines() + lines_to_remove = set() + + class_bases, class_methods = _build_inheritance_info(tree) + + resolved_tests = set() + for class_name, method_name in tests_to_remove: + defining_class = _find_method_definition( + class_name, method_name, class_bases, class_methods + ) + if defining_class: + resolved_tests.add((defining_class, method_name)) + + for node in ast.walk(tree): + if not isinstance(node, ast.ClassDef): + continue + class_name = node.name + for item in node.body: + if not isinstance(item, (ast.FunctionDef, ast.AsyncFunctionDef)): + continue + method_name = item.name + if (class_name, method_name) not in resolved_tests: + continue + + remove_entire_method = _is_super_call_only(item) + + if remove_entire_method: + lines_to_remove.update(_method_removal_range(item, lines)) + else: + for dec in item.decorator_list: + dec_line = dec.lineno - 1 + line_content = lines[dec_line] + + if "expectedFailure" not in line_content: + continue + + has_comment_on_line = COMMENT in line_content + has_comment_before = ( + dec_line > 0 + and lines[dec_line - 1].strip().startswith("#") + and COMMENT in lines[dec_line - 1] + ) + has_comment_after = ( + dec_line + 1 < len(lines) + and lines[dec_line + 1].strip().startswith("#") + and COMMENT not in lines[dec_line + 1] + ) + + if has_comment_on_line or has_comment_before: + lines_to_remove.add(dec_line) + if has_comment_before: + lines_to_remove.add(dec_line - 1) + if has_comment_after and has_comment_on_line: + lines_to_remove.add(dec_line + 1) + + for line_idx in sorted(lines_to_remove, reverse=True): + del lines[line_idx] + + return "\n".join(lines) + "\n" if lines else "" + + +def collect_test_changes( + results: TestResult, + module_prefix: str | None = None, +) -> tuple[set[tuple[str, str]], set[tuple[str, str]], dict[tuple[str, str], str]]: + """ + Collect failing tests and unexpected successes from test results. + + Args: + results: TestResult from run_test() + module_prefix: If set, only collect tests whose path starts with this prefix + + Returns: + (failing_tests, unexpected_successes, error_messages) + - failing_tests: set of (class_name, method_name) tuples + - unexpected_successes: set of (class_name, method_name) tuples + - error_messages: dict mapping (class_name, method_name) to error message + """ + failing_tests = set() + error_messages: dict[tuple[str, str], str] = {} + for test in results.tests: + if test.result in ("fail", "error"): + if module_prefix and not test.path.startswith(module_prefix): + continue + test_parts = path_to_test_parts(test.path) + if len(test_parts) == 2: + key = tuple(test_parts) + failing_tests.add(key) + if test.error_message: + error_messages[key] = test.error_message + + unexpected_successes = set() + for test in results.unexpected_successes: + if module_prefix and not test.path.startswith(module_prefix): + continue + test_parts = path_to_test_parts(test.path) + if len(test_parts) == 2: + unexpected_successes.add(tuple(test_parts)) + + return failing_tests, unexpected_successes, error_messages + + +def apply_test_changes( + contents: str, + failing_tests: set[tuple[str, str]], + unexpected_successes: set[tuple[str, str]], + error_messages: dict[tuple[str, str], str] | None = None, +) -> str: + """ + Apply test changes to content. + + Args: + contents: File content + failing_tests: Set of (class_name, method_name) to mark as expectedFailure + unexpected_successes: Set of (class_name, method_name) to remove expectedFailure + error_messages: Dict mapping (class_name, method_name) to error message + + Returns: + Modified content + """ + if unexpected_successes: + contents = remove_expected_failures(contents, unexpected_successes) + + if failing_tests: + failing_tests, error_messages = _consolidate_to_parent( + contents, failing_tests, error_messages + ) + patches = build_patches(failing_tests, error_messages) + contents = apply_patches(contents, patches) + + return contents + + +def strip_reasonless_expected_failures( + contents: str, +) -> tuple[str, set[tuple[str, str]]]: + """Strip @expectedFailure decorators that have no failure reason. + + Markers like ``@unittest.expectedFailure # TODO: RUSTPYTHON`` (without a + reason after the semicolon) are removed so the tests fail normally during + the next test run and error messages can be captured. + + Returns: + (modified_contents, stripped_tests) where stripped_tests is a set of + (class_name, method_name) tuples whose markers were removed. + """ + tree = ast.parse(contents) + lines = contents.splitlines() + stripped_tests: set[tuple[str, str]] = set() + lines_to_remove: set[int] = set() + + for node in ast.walk(tree): + if not isinstance(node, ast.ClassDef): + continue + for item in node.body: + if not isinstance(item, (ast.FunctionDef, ast.AsyncFunctionDef)): + continue + for dec in item.decorator_list: + dec_line = dec.lineno - 1 + line_content = lines[dec_line] + + if "expectedFailure" not in line_content: + continue + + has_comment_on_line = COMMENT in line_content + has_comment_before = ( + dec_line > 0 + and lines[dec_line - 1].strip().startswith("#") + and COMMENT in lines[dec_line - 1] + ) + + if not has_comment_on_line and not has_comment_before: + continue # not our marker + + # Check if there's a reason (on either the decorator or before) + for check_line in ( + line_content, + lines[dec_line - 1] if has_comment_before else "", + ): + match = re.search(rf"{COMMENT}(.*)", check_line) + if match and match.group(1).strip(";:, "): + break # has a reason, keep it + else: + # No reason found — strip this decorator + stripped_tests.add((node.name, item.name)) + + if _is_super_call_only(item): + # Remove entire super-call override (the method + # exists only to apply the decorator; without it + # the override is pointless and blocks parent + # consolidation) + lines_to_remove.update(_method_removal_range(item, lines)) + else: + lines_to_remove.add(dec_line) + + if has_comment_before: + lines_to_remove.add(dec_line - 1) + + # Also remove a reason-comment on the line after (old format) + if ( + has_comment_on_line + and dec_line + 1 < len(lines) + and lines[dec_line + 1].strip().startswith("#") + and COMMENT not in lines[dec_line + 1] + ): + lines_to_remove.add(dec_line + 1) + + if not lines_to_remove: + return contents, stripped_tests + + for idx in sorted(lines_to_remove, reverse=True): + del lines[idx] + + return "\n".join(lines) + "\n" if lines else "", stripped_tests + + +def extract_test_methods(contents: str) -> set[tuple[str, str]]: + """ + Extract all test method names from file contents. + + Returns: + Set of (class_name, method_name) tuples + """ + from update_lib.file_utils import safe_parse_ast + from update_lib.patch_spec import iter_tests + + tree = safe_parse_ast(contents) + if tree is None: + return set() + + return {(cls_node.name, fn_node.name) for cls_node, fn_node in iter_tests(tree)} + + +def auto_mark_file( + test_path: pathlib.Path, + mark_failure: bool = False, + verbose: bool = True, + original_methods: set[tuple[str, str]] | None = None, + skip_build: bool = False, +) -> tuple[int, int, int]: + """ + Run tests and auto-mark failures in a test file. + + Args: + test_path: Path to the test file + mark_failure: If True, add @expectedFailure to ALL failing tests + verbose: Print progress messages + original_methods: If provided, only auto-mark failures for NEW methods + (methods not in original_methods) even without mark_failure. + Failures in existing methods are treated as regressions. + + Returns: + (num_failures_added, num_successes_removed, num_regressions) + """ + test_path = pathlib.Path(test_path).resolve() + if not test_path.exists(): + raise FileNotFoundError(f"File not found: {test_path}") + + # Strip reason-less markers so those tests fail normally and we capture + # their error messages during the test run. + contents = test_path.read_text(encoding="utf-8") + original_contents = contents + contents, stripped_tests = strip_reasonless_expected_failures(contents) + if stripped_tests: + test_path.write_text(contents, encoding="utf-8") + + test_name = get_test_module_name(test_path) + if verbose: + print(f"Running test: {test_name}") + + results = run_test(test_name, skip_build=skip_build) + + # Check if test run failed entirely (e.g., import error, crash) + if ( + not results.tests_result + and not results.tests + and not results.unexpected_successes + ): + # Restore original contents before raising + if stripped_tests: + test_path.write_text(original_contents, encoding="utf-8") + raise TestRunError( + f"Test run failed for {test_name}. " + f"Output: {results.stdout[-500:] if results.stdout else '(no output)'}" + ) + + # If the run crashed (incomplete), restore original file so that markers + # for tests that never ran are preserved. Only observed results will be + # re-applied below. + if not results.tests_result and stripped_tests: + test_path.write_text(original_contents, encoding="utf-8") + stripped_tests = set() + + contents = test_path.read_text(encoding="utf-8") + + all_failing_tests, unexpected_successes, error_messages = collect_test_changes( + results + ) + + # Determine which failures to mark + if mark_failure: + failing_tests = all_failing_tests + elif original_methods is not None: + # Smart mode: only mark NEW test failures (not regressions) + current_methods = extract_test_methods(contents) + new_methods = current_methods - original_methods + failing_tests = {t for t in all_failing_tests if t in new_methods} + else: + failing_tests = set() + + # Re-mark stripped tests that still fail (to restore markers with reasons). + # Uses inheritance expansion: if a parent marker was stripped, child + # failures are included so _consolidate_to_parent can re-mark the parent. + failing_tests |= _expand_stripped_to_children( + contents, stripped_tests, all_failing_tests + ) + + regressions = all_failing_tests - failing_tests + + if verbose: + for class_name, method_name in failing_tests: + label = "(new test)" if original_methods is not None else "" + err_msg = error_messages.get((class_name, method_name), "") + err_hint = f" - {err_msg}" if err_msg else "" + print( + f"Marking as failing {label}: {class_name}.{method_name}{err_hint}".replace( + " ", " " + ) + ) + for class_name, method_name in unexpected_successes: + print(f"Removing expectedFailure: {class_name}.{method_name}") + + contents = apply_test_changes( + contents, failing_tests, unexpected_successes, error_messages + ) + + if failing_tests or unexpected_successes: + test_path.write_text(contents, encoding="utf-8") + + # Show hints about unmarked failures + if verbose: + unmarked_failures = all_failing_tests - failing_tests + if unmarked_failures: + print( + f"Hint: {len(unmarked_failures)} failing tests can be marked with --mark-failure; " + "but review first and do not blindly mark them all" + ) + for class_name, method_name in sorted(unmarked_failures): + err_msg = error_messages.get((class_name, method_name), "") + err_hint = f" - {err_msg}" if err_msg else "" + print(f" {class_name}.{method_name}{err_hint}") + + return len(failing_tests), len(unexpected_successes), len(regressions) + + +def auto_mark_directory( + test_dir: pathlib.Path, + mark_failure: bool = False, + verbose: bool = True, + original_methods_per_file: dict[pathlib.Path, set[tuple[str, str]]] | None = None, + skip_build: bool = False, +) -> tuple[int, int, int]: + """ + Run tests and auto-mark failures in a test directory. + + Runs the test once for the whole directory, then applies results to each file. + + Args: + test_dir: Path to the test directory + mark_failure: If True, add @expectedFailure to ALL failing tests + verbose: Print progress messages + original_methods_per_file: If provided, only auto-mark failures for NEW methods + even without mark_failure. Dict maps file path to + set of (class_name, method_name) tuples. + + Returns: + (num_failures_added, num_successes_removed, num_regressions) + """ + test_dir = pathlib.Path(test_dir).resolve() + if not test_dir.exists(): + raise FileNotFoundError(f"Directory not found: {test_dir}") + if not test_dir.is_dir(): + raise ValueError(f"Not a directory: {test_dir}") + + # Get all .py files in directory + test_files = sorted(test_dir.glob("**/*.py")) + + # Strip reason-less markers from ALL files before running tests so those + # tests fail normally and we capture their error messages. + stripped_per_file: dict[pathlib.Path, set[tuple[str, str]]] = {} + original_per_file: dict[pathlib.Path, str] = {} + for test_file in test_files: + contents = test_file.read_text(encoding="utf-8") + stripped_contents, stripped = strip_reasonless_expected_failures(contents) + if stripped: + original_per_file[test_file] = contents + test_file.write_text(stripped_contents, encoding="utf-8") + stripped_per_file[test_file] = stripped + + test_name = get_test_module_name(test_dir) + if verbose: + print(f"Running test: {test_name}") + + results = run_test(test_name, skip_build=skip_build) + + # Check if test run failed entirely (e.g., import error, crash) + if ( + not results.tests_result + and not results.tests + and not results.unexpected_successes + ): + # Restore original contents before raising + for fpath, original in original_per_file.items(): + fpath.write_text(original, encoding="utf-8") + raise TestRunError( + f"Test run failed for {test_name}. " + f"Output: {results.stdout[-500:] if results.stdout else '(no output)'}" + ) + + # If the run crashed (incomplete), restore original files so that markers + # for tests that never ran are preserved. + if not results.tests_result and original_per_file: + for fpath, original in original_per_file.items(): + fpath.write_text(original, encoding="utf-8") + stripped_per_file.clear() + + total_added = 0 + total_removed = 0 + total_regressions = 0 + all_regressions: list[tuple[str, str, str, str]] = [] + + for test_file in test_files: + # Get module prefix for this file (e.g., "test_inspect.test_inspect") + module_prefix = get_test_module_name(test_file) + # For __init__.py, the test path doesn't include "__init__" + if module_prefix.endswith(".__init__"): + module_prefix = module_prefix[:-9] # Remove ".__init__" + + all_failing_tests, unexpected_successes, error_messages = collect_test_changes( + results, module_prefix="test." + module_prefix + "." + ) + + # Determine which failures to mark + if mark_failure: + failing_tests = all_failing_tests + elif original_methods_per_file is not None: + # Smart mode: only mark NEW test failures + contents = test_file.read_text(encoding="utf-8") + current_methods = extract_test_methods(contents) + original_methods = original_methods_per_file.get(test_file, set()) + new_methods = current_methods - original_methods + failing_tests = {t for t in all_failing_tests if t in new_methods} + else: + failing_tests = set() + + # Re-mark stripped tests that still fail (restore markers with reasons). + # Uses inheritance expansion for parent→child mapping. + stripped = stripped_per_file.get(test_file, set()) + if stripped: + file_contents = test_file.read_text(encoding="utf-8") + failing_tests |= _expand_stripped_to_children( + file_contents, stripped, all_failing_tests + ) + + regressions = all_failing_tests - failing_tests + + if failing_tests or unexpected_successes: + if verbose: + for class_name, method_name in failing_tests: + label = ( + "(new test)" if original_methods_per_file is not None else "" + ) + err_msg = error_messages.get((class_name, method_name), "") + err_hint = f" - {err_msg}" if err_msg else "" + print( + f" {test_file.name}: Marking as failing {label}: {class_name}.{method_name}{err_hint}".replace( + " :", ":" + ) + ) + for class_name, method_name in unexpected_successes: + print( + f" {test_file.name}: Removing expectedFailure: {class_name}.{method_name}" + ) + + contents = test_file.read_text(encoding="utf-8") + contents = apply_test_changes( + contents, failing_tests, unexpected_successes, error_messages + ) + test_file.write_text(contents, encoding="utf-8") + + # Collect regressions with error messages for later reporting + for class_name, method_name in regressions: + err_msg = error_messages.get((class_name, method_name), "") + all_regressions.append((test_file.name, class_name, method_name, err_msg)) + + total_added += len(failing_tests) + total_removed += len(unexpected_successes) + total_regressions += len(regressions) + + # Show hints about unmarked failures + if verbose and total_regressions > 0: + print( + f"Hint: {total_regressions} failing tests can be marked with --mark-failure; " + "but review first and do not blindly mark them all" + ) + for file_name, class_name, method_name, err_msg in sorted(all_regressions): + err_hint = f" - {err_msg}" if err_msg else "" + print(f" {file_name}: {class_name}.{method_name}{err_hint}") + + return total_added, total_removed, total_regressions + + +def main(argv: list[str] | None = None) -> int: + import argparse + + parser = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument( + "path", + type=pathlib.Path, + help="Path to test file or directory (e.g., Lib/test/test_foo.py or Lib/test/test_foo/)", + ) + parser.add_argument( + "--mark-failure", + action="store_true", + help="Also add @expectedFailure to failing tests (default: only remove unexpected successes)", + ) + parser.add_argument( + "--build", + action=argparse.BooleanOptionalAction, + default=True, + help="Build with cargo (default: enabled)", + ) + + args = parser.parse_args(argv) + + try: + if args.path.is_dir(): + num_added, num_removed, _ = auto_mark_directory( + args.path, mark_failure=args.mark_failure, skip_build=not args.build + ) + else: + num_added, num_removed, _ = auto_mark_file( + args.path, mark_failure=args.mark_failure, skip_build=not args.build + ) + if args.mark_failure: + print(f"Added expectedFailure to {num_added} tests") + print(f"Removed expectedFailure from {num_removed} tests") + return 0 + except (FileNotFoundError, ValueError) as e: + print(f"Error: {e}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/update_lib/cmd_copy_lib.py b/scripts/update_lib/cmd_copy_lib.py new file mode 100644 index 00000000000..1b16497fc83 --- /dev/null +++ b/scripts/update_lib/cmd_copy_lib.py @@ -0,0 +1,111 @@ +#!/usr/bin/env python +""" +Copy library files from CPython. + +Usage: + # Single file + python scripts/update_lib copy-lib cpython/Lib/dataclasses.py + + # Directory + python scripts/update_lib copy-lib cpython/Lib/json +""" + +import argparse +import pathlib +import shutil +import sys + + +def _copy_single( + src_path: pathlib.Path, + lib_path: pathlib.Path, + verbose: bool = True, +) -> None: + """Copy a single file or directory.""" + # Remove existing file/directory + if lib_path.exists(): + if lib_path.is_dir(): + if verbose: + print(f"Removing directory: {lib_path}") + shutil.rmtree(lib_path) + else: + if verbose: + print(f"Removing file: {lib_path}") + lib_path.unlink() + + # Copy + if src_path.is_dir(): + if verbose: + print(f"Copying directory: {src_path} -> {lib_path}") + lib_path.parent.mkdir(parents=True, exist_ok=True) + shutil.copytree(src_path, lib_path) + else: + if verbose: + print(f"Copying file: {src_path} -> {lib_path}") + lib_path.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(src_path, lib_path) + + +def copy_lib( + src_path: pathlib.Path, + verbose: bool = True, +) -> None: + """ + Copy library file or directory from CPython. + + Also copies additional files if defined in DEPENDENCIES table. + + Args: + src_path: Source path (e.g., cpython/Lib/dataclasses.py or cpython/Lib/json) + verbose: Print progress messages + """ + from update_lib.deps import get_lib_paths + from update_lib.file_utils import parse_lib_path + + # Extract module name and cpython prefix from path + path_str = str(src_path).replace("\\", "/") + if "/Lib/" not in path_str: + raise ValueError(f"Path must contain '/Lib/' (got: {src_path})") + + cpython_prefix, after_lib = path_str.split("/Lib/", 1) + # Get module name (first component, without .py) + name = after_lib.split("/")[0] + if name.endswith(".py"): + name = name[:-3] + + # Get all paths to copy from DEPENDENCIES table + all_src_paths = get_lib_paths(name, cpython_prefix) + + # Copy each file + for src in all_src_paths: + if src.exists(): + lib_path = parse_lib_path(src) + _copy_single(src, lib_path, verbose) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument( + "path", + type=pathlib.Path, + help="Source path containing /Lib/ (e.g., cpython/Lib/dataclasses.py)", + ) + + args = parser.parse_args(argv) + + try: + copy_lib(args.path) + return 0 + except ValueError as e: + print(f"Error: {e}", file=sys.stderr) + return 1 + except FileNotFoundError as e: + print(f"Error: {e}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/update_lib/cmd_deps.py b/scripts/update_lib/cmd_deps.py new file mode 100644 index 00000000000..affb4b3609c --- /dev/null +++ b/scripts/update_lib/cmd_deps.py @@ -0,0 +1,458 @@ +#!/usr/bin/env python +""" +Show dependency information for a module. + +Usage: + python scripts/update_lib deps dis + python scripts/update_lib deps dataclasses + python scripts/update_lib deps dis --depth 2 + python scripts/update_lib deps all # Show all modules' dependencies +""" + +import argparse +import pathlib +import sys + +sys.path.insert(0, str(pathlib.Path(__file__).parent.parent)) + + +def get_all_modules(cpython_prefix: str) -> list[str]: + """Get all top-level module names from cpython/Lib/. + + Includes private modules (_*) that are not hard_deps of other modules. + + Returns: + Sorted list of module names (without .py extension) + """ + from update_lib.deps import resolve_hard_dep_parent + + lib_dir = pathlib.Path(cpython_prefix) / "Lib" + if not lib_dir.exists(): + return [] + + modules = set() + for entry in lib_dir.iterdir(): + # Skip hidden files + if entry.name.startswith("."): + continue + # Skip test directory + if entry.name == "test": + continue + + if entry.is_file() and entry.suffix == ".py": + name = entry.stem + elif entry.is_dir() and (entry / "__init__.py").exists(): + name = entry.name + else: + continue + + # Skip modules that are hard_deps of other modules + # e.g., _pydatetime is a hard_dep of datetime, pydoc_data is a hard_dep of pydoc + if resolve_hard_dep_parent(name, cpython_prefix) is not None: + continue + + modules.add(name) + + return sorted(modules) + + +def format_deps_tree( + cpython_prefix: str, + lib_prefix: str, + max_depth: int, + *, + name: str | None = None, + soft_deps: set[str] | None = None, + hard_deps: set[str] | None = None, + _depth: int = 0, + _visited: set[str] | None = None, + _indent: str = "", +) -> list[str]: + """Format soft dependencies as a tree with up-to-date status. + + Args: + cpython_prefix: CPython directory prefix + lib_prefix: Local Lib directory prefix + max_depth: Maximum recursion depth + name: Module name (used to compute deps if soft_deps not provided) + soft_deps: Pre-computed soft dependencies (optional) + hard_deps: Hard dependencies to show under the module (root level only) + _depth: Current depth (internal) + _visited: Already visited modules (internal) + _indent: Current indentation (internal) + + Returns: + List of formatted lines + """ + from update_lib.deps import ( + get_lib_paths, + get_rust_deps, + get_soft_deps, + is_up_to_date, + ) + + lines = [] + + if _visited is None: + _visited = set() + + # Compute deps from name if not provided + if soft_deps is None: + soft_deps = get_soft_deps(name, cpython_prefix) if name else set() + + soft_deps = sorted(soft_deps) + + if not soft_deps and not hard_deps: + return lines + + # Separate up-to-date and outdated modules + up_to_date_deps = [] + outdated_deps = [] + dup_deps = [] + + for dep in soft_deps: + # Skip if library doesn't exist in cpython + lib_paths = get_lib_paths(dep, cpython_prefix) + if not any(p.exists() for p in lib_paths): + continue + + up_to_date = is_up_to_date(dep, cpython_prefix, lib_prefix) + if up_to_date: + # Up-to-date modules collected compactly, no dup tracking needed + up_to_date_deps.append(dep) + elif dep in _visited: + # Only track dup for outdated modules + dup_deps.append(dep) + else: + outdated_deps.append(dep) + + # Show outdated modules with expansion + for dep in outdated_deps: + dep_native = get_rust_deps(dep, cpython_prefix) + native_suffix = ( + f" (native: {', '.join(sorted(dep_native))})" if dep_native else "" + ) + lines.append(f"{_indent}- [ ] {dep}{native_suffix}") + _visited.add(dep) + + # Show hard_deps under this module (only at root level, i.e., when hard_deps is provided) + if hard_deps and dep in soft_deps: + for hd in sorted(hard_deps): + hd_up_to_date = is_up_to_date(hd, cpython_prefix, lib_prefix) + hd_marker = "[x]" if hd_up_to_date else "[ ]" + lines.append(f"{_indent} - {hd_marker} {hd}") + hard_deps = None # Only show once + + # Recurse if within depth limit + if _depth < max_depth - 1: + lines.extend( + format_deps_tree( + cpython_prefix, + lib_prefix, + max_depth, + name=dep, + _depth=_depth + 1, + _visited=_visited, + _indent=_indent + " ", + ) + ) + + # Show duplicates compactly (only for outdated) + if dup_deps: + lines.append(f"{_indent}- [ ] {', '.join(dup_deps)}") + + # Show up-to-date modules compactly on one line + if up_to_date_deps: + lines.append(f"{_indent}- [x] {', '.join(up_to_date_deps)}") + + return lines + + +def format_deps( + name: str, + cpython_prefix: str, + lib_prefix: str, + max_depth: int = 10, + _visited: set[str] | None = None, +) -> list[str]: + """Format all dependency information for a module. + + Args: + name: Module name + cpython_prefix: CPython directory prefix + lib_prefix: Local Lib directory prefix + max_depth: Maximum recursion depth + _visited: Shared visited set for deduplication across modules + + Returns: + List of formatted lines + """ + from update_lib.deps import ( + DEPENDENCIES, + count_test_todos, + find_dependent_tests_tree, + get_lib_paths, + get_test_paths, + is_path_synced, + is_test_up_to_date, + resolve_hard_dep_parent, + ) + + if _visited is None: + _visited = set() + + lines = [] + + # Resolve test_ prefix to module (e.g., test_pydoc -> pydoc) + if name.startswith("test_"): + module_name = name[5:] # strip "test_" + lines.append(f"(redirecting {name} -> {module_name})") + name = module_name + + # Resolve hard_dep to parent module (e.g., pydoc_data -> pydoc) + parent = resolve_hard_dep_parent(name, cpython_prefix) + if parent: + lines.append(f"(redirecting {name} -> {parent})") + name = parent + + # lib paths (only show existing) + lib_paths = get_lib_paths(name, cpython_prefix) + existing_lib_paths = [p for p in lib_paths if p.exists()] + for p in existing_lib_paths: + synced = is_path_synced(p, cpython_prefix, lib_prefix) + marker = "[x]" if synced else "[ ]" + lines.append(f"{marker} lib: {p}") + + # test paths (only show existing) + test_paths = get_test_paths(name, cpython_prefix) + existing_test_paths = [p for p in test_paths if p.exists()] + for p in existing_test_paths: + test_name = p.stem if p.is_file() else p.name + synced = is_test_up_to_date(test_name, cpython_prefix, lib_prefix) + marker = "[x]" if synced else "[ ]" + todo_count = count_test_todos(test_name, lib_prefix) + todo_suffix = f" (TODO: {todo_count})" if todo_count > 0 else "" + lines.append(f"{marker} test: {p}{todo_suffix}") + + # If no lib or test paths exist, module doesn't exist + if not existing_lib_paths and not existing_test_paths: + lines.append(f"(module '{name}' not found)") + return lines + + # Collect all hard_deps (explicit from DEPENDENCIES + implicit from lib_paths) + dep_info = DEPENDENCIES.get(name, {}) + explicit_hard_deps = dep_info.get("hard_deps", []) + + # Get implicit hard_deps from lib_paths (e.g., _pydecimal.py for decimal) + all_hard_deps = set() + for hd in explicit_hard_deps: + # Remove .py extension if present + all_hard_deps.add(hd[:-3] if hd.endswith(".py") else hd) + + for p in existing_lib_paths: + dep_name = p.stem if p.is_file() else p.name + if dep_name != name: # Skip the main module itself + all_hard_deps.add(dep_name) + + lines.append("\ndependencies:") + lines.extend( + format_deps_tree( + cpython_prefix, + lib_prefix, + max_depth, + soft_deps={name}, + _visited=_visited, + hard_deps=all_hard_deps, + ) + ) + + # Show dependent tests as tree (depth 2: module + direct importers + their importers) + tree = find_dependent_tests_tree(name, lib_prefix=lib_prefix, max_depth=2) + lines.extend(_format_dependent_tests_tree(tree, cpython_prefix, lib_prefix)) + + return lines + + +def _format_dependent_tests_tree( + tree: dict, + cpython_prefix: str, + lib_prefix: str, + indent: str = "", +) -> list[str]: + """Format dependent tests tree for display.""" + from update_lib.deps import is_up_to_date + + lines = [] + module = tree["module"] + tests = tree["tests"] + children = tree["children"] + + if indent == "": + # Root level + # Count total tests in tree + def count_tests(t: dict) -> int: + total = len(t.get("tests", [])) + for c in t.get("children", []): + total += count_tests(c) + return total + + total = count_tests(tree) + if total == 0 and not children: + lines.append(f"\ndependent tests: (no tests depend on {module})") + return lines + lines.append(f"\ndependent tests: ({total} tests)") + + # Check if module is up-to-date + synced = is_up_to_date(module.split(".")[0], cpython_prefix, lib_prefix) + marker = "[x]" if synced else "[ ]" + + # Format this node + if tests: + test_str = " ".join(tests) + if indent == "": + lines.append(f"- {marker} {module}: {test_str}") + else: + lines.append(f"{indent}- {marker} {module}: {test_str}") + elif indent != "" and children: + # Has children but no direct tests + lines.append(f"{indent}- {marker} {module}:") + + # Format children + child_indent = indent + " " if indent else " " + for child in children: + lines.extend( + _format_dependent_tests_tree( + child, cpython_prefix, lib_prefix, child_indent + ) + ) + + return lines + + +def _resolve_module_name( + name: str, + cpython_prefix: str, + lib_prefix: str, +) -> list[str]: + """Resolve module name through redirects. + + Returns a list of module names (usually 1, but test support files may expand to multiple). + """ + import pathlib + + from update_lib.deps import ( + _build_test_import_graph, + get_lib_paths, + get_test_paths, + resolve_hard_dep_parent, + resolve_test_to_lib, + ) + + # Resolve test to library group (e.g., test_urllib2 -> urllib) + if name.startswith("test_"): + lib_group = resolve_test_to_lib(name) + if lib_group: + return [lib_group] + name = name[5:] + + # Resolve hard_dep to parent + parent = resolve_hard_dep_parent(name, cpython_prefix) + if parent: + return [parent] + + # Check if it's a valid module + lib_paths = get_lib_paths(name, cpython_prefix) + test_paths = get_test_paths(name, cpython_prefix) + if any(p.exists() for p in lib_paths) or any(p.exists() for p in test_paths): + return [name] + + # Check for test support files (e.g., string_tests -> bytes, str, userstring) + test_support_path = pathlib.Path(cpython_prefix) / "Lib" / "test" / f"{name}.py" + if test_support_path.exists(): + test_dir = pathlib.Path(lib_prefix) / "test" + if test_dir.exists(): + import_graph, _ = _build_test_import_graph(test_dir) + importing_tests = [] + for file_key, imports in import_graph.items(): + if name in imports and file_key.startswith("test_"): + importing_tests.append(file_key) + if importing_tests: + # Resolve test names to module names (test_bytes -> bytes) + return sorted(set(t[5:] for t in importing_tests)) + + return [name] + + +def show_deps( + names: list[str], + cpython_prefix: str, + lib_prefix: str, + max_depth: int = 10, +) -> None: + """Show all dependency information for modules.""" + # Expand "all" to all module names + expanded_names = [] + for name in names: + if name == "all": + expanded_names.extend(get_all_modules(cpython_prefix)) + else: + expanded_names.append(name) + + # Resolve and deduplicate names (preserving order) + seen: set[str] = set() + resolved_names: list[str] = [] + for name in expanded_names: + for resolved in _resolve_module_name(name, cpython_prefix, lib_prefix): + if resolved not in seen: + seen.add(resolved) + resolved_names.append(resolved) + + # Shared visited set across all modules + visited: set[str] = set() + + for i, name in enumerate(resolved_names): + if i > 0: + print() # blank line between modules + for line in format_deps(name, cpython_prefix, lib_prefix, max_depth, visited): + print(line) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument( + "names", + nargs="+", + help="Module names (e.g., dis, dataclasses) or 'all' for all modules", + ) + parser.add_argument( + "--cpython", + default="cpython", + help="CPython directory prefix (default: cpython)", + ) + parser.add_argument( + "--lib", + default="Lib", + help="Local Lib directory prefix (default: Lib)", + ) + parser.add_argument( + "--depth", + type=int, + default=10, + help="Maximum recursion depth for soft_deps tree (default: 10)", + ) + + args = parser.parse_args(argv) + + try: + show_deps(args.names, args.cpython, args.lib, args.depth) + return 0 + except Exception as e: + print(f"Error: {e}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/update_lib/cmd_migrate.py b/scripts/update_lib/cmd_migrate.py new file mode 100644 index 00000000000..77292831cea --- /dev/null +++ b/scripts/update_lib/cmd_migrate.py @@ -0,0 +1,142 @@ +#!/usr/bin/env python +""" +Migrate test file(s) from CPython, preserving RustPython markers. + +Usage: + python scripts/update_lib migrate cpython/Lib/test/test_foo.py + +This will: + 1. Extract patches from Lib/test/test_foo.py (if exists) + 2. Apply them to cpython/Lib/test/test_foo.py + 3. Write result to Lib/test/test_foo.py +""" + +import argparse +import pathlib +import sys + +sys.path.insert(0, str(pathlib.Path(__file__).parent.parent)) + +from update_lib.file_utils import parse_lib_path + + +def patch_single_content( + src_path: pathlib.Path, + lib_path: pathlib.Path, +) -> str: + """ + Patch content without writing to disk. + + Args: + src_path: Source file path (e.g., cpython/Lib/test/foo.py) + lib_path: Lib path to extract patches from (e.g., Lib/test/foo.py) + + Returns: + The patched content. + """ + from update_lib import apply_patches, extract_patches + + # Extract patches from existing file (if exists) + if lib_path.exists(): + patches = extract_patches(lib_path.read_text(encoding="utf-8")) + else: + patches = {} + + # Apply patches to source content + src_content = src_path.read_text(encoding="utf-8") + return apply_patches(src_content, patches) + + +def patch_file( + src_path: pathlib.Path, + lib_path: pathlib.Path | None = None, + verbose: bool = True, +) -> None: + """ + Patch a single file from source to lib. + + Args: + src_path: Source file path (e.g., cpython/Lib/test/foo.py) + lib_path: Target lib path. If None, derived from src_path. + verbose: Print progress messages + """ + if lib_path is None: + lib_path = parse_lib_path(src_path) + + if lib_path.exists(): + if verbose: + print(f"Patching: {src_path} -> {lib_path}") + content = patch_single_content(src_path, lib_path) + else: + if verbose: + print(f"Copying: {src_path} -> {lib_path}") + content = src_path.read_text(encoding="utf-8") + + lib_path.parent.mkdir(parents=True, exist_ok=True) + lib_path.write_text(content, encoding="utf-8") + + +def patch_directory( + src_dir: pathlib.Path, + lib_dir: pathlib.Path | None = None, + verbose: bool = True, +) -> None: + """ + Patch all files in a directory from source to lib. + + Args: + src_dir: Source directory path (e.g., cpython/Lib/test/test_foo/) + lib_dir: Target lib directory. If None, derived from src_dir. + verbose: Print progress messages + """ + if lib_dir is None: + lib_dir = parse_lib_path(src_dir) + + src_files = sorted(src_dir.glob("**/*.py")) + + for src_file in src_files: + rel_path = src_file.relative_to(src_dir) + lib_file = lib_dir / rel_path + + if lib_file.exists(): + if verbose: + print(f"Patching: {src_file} -> {lib_file}") + content = patch_single_content(src_file, lib_file) + else: + if verbose: + print(f"Copying: {src_file} -> {lib_file}") + content = src_file.read_text(encoding="utf-8") + + lib_file.parent.mkdir(parents=True, exist_ok=True) + lib_file.write_text(content, encoding="utf-8") + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument( + "path", + type=pathlib.Path, + help="Source path containing /Lib/ (file or directory)", + ) + + args = parser.parse_args(argv) + + try: + if args.path.is_dir(): + patch_directory(args.path) + else: + patch_file(args.path) + return 0 + except ValueError as e: + print(f"Error: {e}", file=sys.stderr) + return 1 + except FileNotFoundError as e: + print(f"Error: {e}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/update_lib/cmd_patches.py b/scripts/update_lib/cmd_patches.py new file mode 100644 index 00000000000..67ebf1822b7 --- /dev/null +++ b/scripts/update_lib/cmd_patches.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python +""" +Patch management for test files. + +Usage: + # Extract patches from one file and apply to another + python scripts/update_lib patches --from Lib/test/foo.py --to cpython/Lib/test/foo.py + + # Show patches as JSON + python scripts/update_lib patches --from Lib/test/foo.py --show-patches + + # Apply patches from JSON file + python scripts/update_lib patches -p patches.json --to Lib/test/foo.py +""" + +import argparse +import json +import pathlib +import sys + +sys.path.insert(0, str(pathlib.Path(__file__).parent.parent)) + + +def write_output(data: str, dest: str) -> None: + if dest == "-": + print(data, end="") + return + + with open(dest, "w") as fd: + fd.write(data) + + +def main(argv: list[str] | None = None) -> int: + from update_lib import ( + apply_patches, + extract_patches, + patches_from_json, + patches_to_json, + ) + + parser = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + + patches_group = parser.add_mutually_exclusive_group(required=True) + patches_group.add_argument( + "-p", + "--patches", + type=pathlib.Path, + help="File path to file containing patches in a JSON format", + ) + patches_group.add_argument( + "--from", + dest="gather_from", + type=pathlib.Path, + help="File to gather patches from", + ) + + group = parser.add_mutually_exclusive_group(required=False) + group.add_argument( + "--to", + type=pathlib.Path, + help="File to apply patches to", + ) + group.add_argument( + "--show-patches", + action="store_true", + help="Show the patches and exit", + ) + + parser.add_argument( + "-o", + "--output", + default="-", + help="Output file. Set to '-' for stdout", + ) + + args = parser.parse_args(argv) + + # Validate required arguments + if args.to is None and not args.show_patches: + parser.error("--to or --show-patches is required") + + try: + if args.patches: + patches = patches_from_json(json.loads(args.patches.read_text())) + else: + patches = extract_patches(args.gather_from.read_text()) + + if args.show_patches: + output = json.dumps(patches_to_json(patches), indent=4) + "\n" + write_output(output, args.output) + return 0 + + patched = apply_patches(args.to.read_text(), patches) + write_output(patched, args.output) + return 0 + + except FileNotFoundError as e: + print(f"Error: {e}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/update_lib/cmd_quick.py b/scripts/update_lib/cmd_quick.py new file mode 100644 index 00000000000..a56d8f57de8 --- /dev/null +++ b/scripts/update_lib/cmd_quick.py @@ -0,0 +1,467 @@ +#!/usr/bin/env python +""" +Quick update for test files from CPython. + +Usage: + # Library + test: copy lib, then patch + auto-mark test + commit + python scripts/update_lib quick cpython/Lib/dataclasses.py + + # Shortcut: just the module name + python scripts/update_lib quick dataclasses + + # Test file: patch + auto-mark + python scripts/update_lib quick cpython/Lib/test/test_foo.py + + # Test file: migrate only + python scripts/update_lib quick cpython/Lib/test/test_foo.py --no-auto-mark + + # Test file: auto-mark only (Lib/ path implies --no-migrate) + python scripts/update_lib quick Lib/test/test_foo.py + + # Directory: patch all + auto-mark all + python scripts/update_lib quick cpython/Lib/test/test_dataclasses/ + + # Skip git commit + python scripts/update_lib quick dataclasses --no-commit +""" + +import argparse +import pathlib +import sys + +sys.path.insert(0, str(pathlib.Path(__file__).parent.parent)) + +from update_lib.deps import DEPENDENCIES, get_test_paths +from update_lib.file_utils import ( + construct_lib_path, + get_cpython_dir, + get_module_name, + get_test_files, + is_lib_path, + is_test_path, + lib_to_test_path, + parse_lib_path, + resolve_module_path, + safe_read_text, +) + + +def collect_original_methods( + lib_path: pathlib.Path, +) -> set[tuple[str, str]] | dict[pathlib.Path, set[tuple[str, str]]] | None: + """ + Collect original test methods from lib path before patching. + + Returns: + - For file: set of (class_name, method_name) or None if file doesn't exist + - For directory: dict mapping file path to set of methods, or None if dir doesn't exist + """ + from update_lib.cmd_auto_mark import extract_test_methods + + if not lib_path.exists(): + return None + + if lib_path.is_file(): + content = safe_read_text(lib_path) + return extract_test_methods(content) if content else set() + else: + result = {} + for lib_file in get_test_files(lib_path): + content = safe_read_text(lib_file) + if content: + result[lib_file.resolve()] = extract_test_methods(content) + return result + + +def quick( + src_path: pathlib.Path, + no_migrate: bool = False, + no_auto_mark: bool = False, + mark_failure: bool = False, + verbose: bool = True, + skip_build: bool = False, +) -> None: + """ + Process a file or directory: migrate + auto-mark. + + Args: + src_path: Source path (file or directory) + no_migrate: Skip migration step + no_auto_mark: Skip auto-mark step + mark_failure: Add @expectedFailure to ALL failing tests + verbose: Print progress messages + skip_build: Skip cargo build, use pre-built binary + """ + from update_lib.cmd_auto_mark import auto_mark_directory, auto_mark_file + from update_lib.cmd_migrate import patch_directory, patch_file + + # Determine lib_path and whether to migrate + if is_lib_path(src_path): + no_migrate = True + lib_path = src_path + else: + lib_path = parse_lib_path(src_path) + + is_dir = src_path.is_dir() + + # Capture original test methods before migration (for smart auto-mark) + original_methods = collect_original_methods(lib_path) + + # Step 1: Migrate + if not no_migrate: + if is_dir: + patch_directory(src_path, lib_path, verbose=verbose) + else: + patch_file(src_path, lib_path, verbose=verbose) + + # Step 1.5: Handle test dependencies + from update_lib.deps import get_test_dependencies + + test_deps = get_test_dependencies(src_path) + + # Migrate dependency files + for dep_src in test_deps["hard_deps"]: + dep_lib = parse_lib_path(dep_src) + if verbose: + print(f"Migrating dependency: {dep_src.name}") + if dep_src.is_dir(): + patch_directory(dep_src, dep_lib, verbose=False) + else: + patch_file(dep_src, dep_lib, verbose=False) + + # Copy data directories (no migration) + import shutil + + for data_src in test_deps["data"]: + data_lib = parse_lib_path(data_src) + if verbose: + print(f"Copying data: {data_src.name}") + if data_lib.exists(): + if data_lib.is_dir(): + shutil.rmtree(data_lib) + else: + data_lib.unlink() + if data_src.is_dir(): + shutil.copytree(data_src, data_lib) + else: + data_lib.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(data_src, data_lib) + + # Step 2: Auto-mark + if not no_auto_mark: + if not lib_path.exists(): + raise FileNotFoundError(f"Path not found: {lib_path}") + + if is_dir: + num_added, num_removed, _ = auto_mark_directory( + lib_path, + mark_failure=mark_failure, + verbose=verbose, + original_methods_per_file=original_methods, + skip_build=skip_build, + ) + else: + num_added, num_removed, _ = auto_mark_file( + lib_path, + mark_failure=mark_failure, + verbose=verbose, + original_methods=original_methods, + skip_build=skip_build, + ) + + if verbose: + if num_added: + print(f"Added expectedFailure to {num_added} tests") + print(f"Removed expectedFailure from {num_removed} tests") + + +def get_cpython_version(cpython_dir: pathlib.Path) -> str: + """Get CPython version from git tag.""" + import subprocess + + result = subprocess.run( + ["git", "describe", "--tags"], + cwd=cpython_dir, + capture_output=True, + text=True, + check=True, + ) + return result.stdout.strip() + + +def git_commit( + name: str, + lib_path: pathlib.Path | None, + test_paths: list[pathlib.Path] | pathlib.Path | None, + cpython_dir: pathlib.Path, + hard_deps: list[pathlib.Path] | None = None, + verbose: bool = True, +) -> bool: + """Commit changes with CPython author. + + Args: + name: Module name (e.g., "dataclasses") + lib_path: Path to library file/directory (or None) + test_paths: Path(s) to test file/directory (or None) + cpython_dir: Path to cpython directory + hard_deps: Path(s) to hard dependency files (or None) + verbose: Print progress messages + + Returns: + True if commit was created, False otherwise + """ + import subprocess + + # Normalize test_paths to list + if test_paths is None: + test_paths = [] + elif isinstance(test_paths, pathlib.Path): + test_paths = [test_paths] + + # Normalize hard_deps to list + if hard_deps is None: + hard_deps = [] + + # Stage changes + paths_to_add = [] + if lib_path and lib_path.exists(): + paths_to_add.append(str(lib_path)) + for test_path in test_paths: + if test_path and test_path.exists(): + paths_to_add.append(str(test_path)) + for dep_path in hard_deps: + if dep_path and dep_path.exists(): + paths_to_add.append(str(dep_path)) + + if not paths_to_add: + return False + + version = get_cpython_version(cpython_dir) + subprocess.run(["git", "add"] + paths_to_add, check=True) + + # Check if there are staged changes + result = subprocess.run( + ["git", "diff", "--cached", "--quiet"], + capture_output=True, + ) + if result.returncode == 0: + if verbose: + print("No changes to commit") + return False + + # Commit with CPython author + message = f"Update {name} from {version}" + subprocess.run( + [ + "git", + "commit", + "--author", + "CPython Developers <>", + "-m", + message, + ], + check=True, + ) + if verbose: + print(f"Committed: {message}") + return True + + +def _expand_shortcut(path: pathlib.Path) -> pathlib.Path: + """Expand simple name to cpython/Lib path if it exists. + + Examples: + dataclasses -> cpython/Lib/dataclasses.py (if exists) + json -> cpython/Lib/json/ (if exists) + test_types -> cpython/Lib/test/test_types.py (if exists) + regrtest -> cpython/Lib/test/libregrtest (from DEPENDENCIES) + """ + # Only expand if it's a simple name (no path separators) and doesn't exist + if "/" in str(path) or path.exists(): + return path + + name = str(path) + + # Check DEPENDENCIES table for path overrides (e.g., regrtest) + from update_lib.deps import DEPENDENCIES + + if name in DEPENDENCIES and "lib" in DEPENDENCIES[name]: + lib_paths = DEPENDENCIES[name]["lib"] + if lib_paths: + override_path = construct_lib_path("cpython", lib_paths[0]) + if override_path.exists(): + return override_path + + # Test shortcut: test_foo -> cpython/Lib/test/test_foo + if name.startswith("test_"): + resolved = resolve_module_path(f"test/{name}", "cpython", prefer="dir") + if resolved.exists(): + return resolved + + # Library shortcut: foo -> cpython/Lib/foo + resolved = resolve_module_path(name, "cpython", prefer="file") + if resolved.exists(): + return resolved + + # Extension module shortcut: winreg -> cpython/Lib/test/test_winreg + # For C/Rust extension modules that have no Python source but have tests + resolved = resolve_module_path(f"test/test_{name}", "cpython", prefer="dir") + if resolved.exists(): + return resolved + + # Return original (will likely fail later with a clear error) + return path + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument( + "path", + type=pathlib.Path, + help="Source path (file or directory)", + ) + parser.add_argument( + "--copy", + action=argparse.BooleanOptionalAction, + default=True, + help="Copy library file (default: enabled, implied disabled if test path)", + ) + parser.add_argument( + "--migrate", + action=argparse.BooleanOptionalAction, + default=True, + help="Migrate test file (default: enabled, implied disabled if Lib/ path)", + ) + parser.add_argument( + "--auto-mark", + action=argparse.BooleanOptionalAction, + default=True, + help="Auto-mark test failures (default: enabled)", + ) + parser.add_argument( + "--mark-failure", + action="store_true", + help="Add @expectedFailure to failing tests", + ) + parser.add_argument( + "--commit", + action=argparse.BooleanOptionalAction, + default=True, + help="Create git commit (default: enabled)", + ) + parser.add_argument( + "--build", + action=argparse.BooleanOptionalAction, + default=True, + help="Build with cargo (default: enabled)", + ) + + args = parser.parse_args(argv) + + try: + src_path = args.path + + # Shortcut: expand simple name to cpython/Lib path + src_path = _expand_shortcut(src_path) + original_src = src_path # Keep for commit + + # Track library path for commit + lib_file_path = None + test_path = None + hard_deps_for_commit = [] + + # If it's a library path (not test path), do copy_lib first + if not is_test_path(src_path): + # Get library destination path for commit + lib_file_path = parse_lib_path(src_path) + + if args.copy: + from update_lib.cmd_copy_lib import copy_lib + + copy_lib(src_path) + + # Get all test paths from DEPENDENCIES (or fall back to default) + module_name = get_module_name(original_src) + cpython_dir = get_cpython_dir(original_src) + test_src_paths = get_test_paths(module_name, str(cpython_dir)) + + # Fall back to default test path if DEPENDENCIES has no entry + if not test_src_paths: + default_test = lib_to_test_path(original_src) + if default_test.exists(): + test_src_paths = (default_test,) + + # Collect hard dependencies for commit + lib_deps = DEPENDENCIES.get(module_name, {}) + for dep_name in lib_deps.get("hard_deps", []): + dep_lib_path = construct_lib_path("Lib", dep_name) + if dep_lib_path.exists(): + hard_deps_for_commit.append(dep_lib_path) + + # Process all test paths + test_paths_for_commit = [] + for test_src in test_src_paths: + if not test_src.exists(): + print(f"Warning: Test path does not exist: {test_src}") + continue + + test_lib_path = parse_lib_path(test_src) + test_paths_for_commit.append(test_lib_path) + + quick( + test_src, + no_migrate=not args.migrate, + no_auto_mark=not args.auto_mark, + mark_failure=args.mark_failure, + skip_build=not args.build, + ) + + test_paths = test_paths_for_commit + else: + # It's a test path - process single test + test_path = ( + parse_lib_path(src_path) if not is_lib_path(src_path) else src_path + ) + + quick( + src_path, + no_migrate=not args.migrate, + no_auto_mark=not args.auto_mark, + mark_failure=args.mark_failure, + skip_build=not args.build, + ) + test_paths = [test_path] + + # Step 3: Git commit + if args.commit: + cpython_dir = get_cpython_dir(original_src) + git_commit( + get_module_name(original_src), + lib_file_path, + test_paths, + cpython_dir, + hard_deps=hard_deps_for_commit, + ) + + return 0 + except ValueError as e: + print(f"Error: {e}", file=sys.stderr) + return 1 + except FileNotFoundError as e: + print(f"Error: {e}", file=sys.stderr) + return 1 + except Exception as e: + # Handle TestRunError with a clean message + from update_lib.cmd_auto_mark import TestRunError + + if isinstance(e, TestRunError): + print(f"Error: {e}", file=sys.stderr) + return 1 + raise + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/update_lib/cmd_todo.py b/scripts/update_lib/cmd_todo.py new file mode 100644 index 00000000000..87099aa5422 --- /dev/null +++ b/scripts/update_lib/cmd_todo.py @@ -0,0 +1,651 @@ +#!/usr/bin/env python +""" +Show prioritized list of modules to update. + +Usage: + python scripts/update_lib todo + python scripts/update_lib todo --limit 20 +""" + +import argparse +import pathlib +import sys + +sys.path.insert(0, str(pathlib.Path(__file__).parent.parent)) + +from update_lib.deps import ( + count_test_todos, + is_test_tracked, + is_test_up_to_date, +) + + +def compute_todo_list( + cpython_prefix: str, + lib_prefix: str, + include_done: bool = False, +) -> list[dict]: + """Compute prioritized list of modules to update. + + Scoring: + - Modules with no pylib dependencies: score = -1 + - Modules with pylib dependencies: score = count of NOT up-to-date deps + + Sorting (ascending by score): + 1. More reverse dependencies (modules depending on this) = higher priority + 2. Fewer native dependencies = higher priority + + Returns: + List of dicts with module info, sorted by priority + """ + from update_lib.cmd_deps import get_all_modules + from update_lib.deps import ( + get_all_hard_deps, + get_rust_deps, + get_soft_deps, + is_up_to_date, + ) + + all_modules = get_all_modules(cpython_prefix) + + # Build dependency data for all modules + module_data = {} + for name in all_modules: + soft_deps = get_soft_deps(name, cpython_prefix) + native_deps = get_rust_deps(name, cpython_prefix) + up_to_date = is_up_to_date(name, cpython_prefix, lib_prefix) + + # Get hard_deps and check their status + hard_deps = get_all_hard_deps(name, cpython_prefix) + hard_deps_status = { + hd: is_up_to_date(hd, cpython_prefix, lib_prefix) for hd in hard_deps + } + + module_data[name] = { + "name": name, + "soft_deps": soft_deps, + "native_deps": native_deps, + "up_to_date": up_to_date, + "hard_deps_status": hard_deps_status, + } + + # Build reverse dependency map: who depends on this module + reverse_deps: dict[str, set[str]] = {name: set() for name in all_modules} + for name, data in module_data.items(): + for dep in data["soft_deps"]: + if dep in reverse_deps: + reverse_deps[dep].add(name) + + # Compute scores and filter + result = [] + for name, data in module_data.items(): + hard_deps_status = data["hard_deps_status"] + has_outdated_hard_deps = any(not ok for ok in hard_deps_status.values()) + + # Include if: not up-to-date, or has outdated hard_deps, or --done + if data["up_to_date"] and not has_outdated_hard_deps and not include_done: + continue + + soft_deps = data["soft_deps"] + if not soft_deps: + # No pylib dependencies + score = -1 + total_deps = 0 + else: + # Count NOT up-to-date dependencies + score = sum( + 1 + for dep in soft_deps + if dep in module_data and not module_data[dep]["up_to_date"] + ) + total_deps = len(soft_deps) + + result.append( + { + "name": name, + "score": score, + "total_deps": total_deps, + "reverse_deps": reverse_deps[name], + "reverse_deps_count": len(reverse_deps[name]), + "native_deps_count": len(data["native_deps"]), + "native_deps": data["native_deps"], + "soft_deps": soft_deps, + "up_to_date": data["up_to_date"], + "hard_deps_status": hard_deps_status, + } + ) + + # Sort by: + # 1. score (ascending) - fewer outstanding deps first + # 2. reverse_deps_count (descending) - more dependents first + # 3. native_deps_count (ascending) - fewer native deps first + result.sort( + key=lambda x: ( + x["score"], + -x["reverse_deps_count"], + x["native_deps_count"], + ) + ) + + return result + + +def get_all_tests(cpython_prefix: str) -> list[str]: + """Get all test module names from cpython/Lib/test/. + + Returns: + Sorted list of test names (e.g., ["test_abc", "test_dis", ...]) + """ + test_dir = pathlib.Path(cpython_prefix) / "Lib" / "test" + if not test_dir.exists(): + return [] + + tests = set() + for entry in test_dir.iterdir(): + # Skip private/internal and special directories + if entry.name.startswith(("_", ".")): + continue + # Skip non-test items + if not entry.name.startswith("test_"): + continue + + if entry.is_file() and entry.suffix == ".py": + tests.add(entry.stem) + elif entry.is_dir() and (entry / "__init__.py").exists(): + tests.add(entry.name) + + return sorted(tests) + + +def get_untracked_files( + cpython_prefix: str, + lib_prefix: str, +) -> list[str]: + """Get files that exist in cpython/Lib but not in our Lib. + + Excludes files that belong to tracked modules (shown in library todo) + and hard_deps of those modules. + Includes all file types (.py, .txt, .pem, .json, etc.) + + Returns: + Sorted list of relative paths (e.g., ["foo.py", "data/file.txt"]) + """ + from update_lib.cmd_deps import get_all_modules + from update_lib.deps import resolve_hard_dep_parent + + cpython_lib = pathlib.Path(cpython_prefix) / "Lib" + local_lib = pathlib.Path(lib_prefix) + + if not cpython_lib.exists(): + return [] + + # Get tracked modules (shown in library todo) + tracked_modules = set(get_all_modules(cpython_prefix)) + + untracked = [] + + for cpython_file in cpython_lib.rglob("*"): + # Skip directories + if cpython_file.is_dir(): + continue + + # Get relative path from Lib/ + rel_path = cpython_file.relative_to(cpython_lib) + + # Skip test/ directory (handled separately by test todo) + if rel_path.parts and rel_path.parts[0] == "test": + continue + + # Check if file belongs to a tracked module + # e.g., idlelib/Icons/idle.gif -> module "idlelib" + # e.g., foo.py -> module "foo" + first_part = rel_path.parts[0] + if first_part.endswith(".py"): + module_name = first_part[:-3] # Remove .py + else: + module_name = first_part + + if module_name in tracked_modules: + continue + + # Check if this is a hard_dep of a tracked module + if resolve_hard_dep_parent(module_name, cpython_prefix) is not None: + continue + + # Check if exists in local lib + local_file = local_lib / rel_path + if not local_file.exists(): + untracked.append(str(rel_path)) + + return sorted(untracked) + + +def get_original_files( + cpython_prefix: str, + lib_prefix: str, +) -> list[str]: + """Get top-level files/modules that exist in our Lib but not in cpython/Lib. + + These are RustPython-original files that don't come from CPython. + Modules that exist in cpython are handled by the library todo (even if + they have additional local files), so they are excluded here. + Excludes test/ directory (handled separately). + + Returns: + Sorted list of top-level names (e.g., ["_dummy_thread.py"]) + """ + cpython_lib = pathlib.Path(cpython_prefix) / "Lib" + local_lib = pathlib.Path(lib_prefix) + + if not local_lib.exists(): + return [] + + original = [] + + # Only check top-level entries + for entry in local_lib.iterdir(): + name = entry.name + + # Skip hidden files and __pycache__ + if name.startswith(".") or name == "__pycache__": + continue + + # Skip test/ directory (handled separately) + if name == "test": + continue + + # Skip site-packages (not a module) + if name == "site-packages": + continue + + # Only include if it doesn't exist in cpython at all + cpython_entry = cpython_lib / name + if not cpython_entry.exists(): + original.append(name) + + return sorted(original) + + +def _build_test_to_lib_map( + cpython_prefix: str, +) -> tuple[dict[str, str], dict[str, list[str]]]: + """Build reverse mapping from test name to library name using DEPENDENCIES. + + Returns: + Tuple of: + - Dict mapping test_name -> lib_name (e.g., "test_htmlparser" -> "html") + - Dict mapping lib_name -> ordered list of test_names + """ + import pathlib + + from update_lib.deps import DEPENDENCIES + + test_to_lib = {} + lib_test_order: dict[str, list[str]] = {} + for lib_name, dep_info in DEPENDENCIES.items(): + if "test" not in dep_info: + continue + lib_test_order[lib_name] = [] + for test_path in dep_info["test"]: + # test_path is like "test_htmlparser.py" or "test_multiprocessing_fork" + path = pathlib.Path(test_path) + if path.suffix == ".py": + test_name = path.stem + else: + test_name = path.name + test_to_lib[test_name] = lib_name + lib_test_order[lib_name].append(test_name) + + return test_to_lib, lib_test_order + + +def compute_test_todo_list( + cpython_prefix: str, + lib_prefix: str, + include_done: bool = False, + lib_status: dict[str, bool] | None = None, +) -> list[dict]: + """Compute prioritized list of tests to update. + + Scoring: + - If corresponding lib is up-to-date: score = 0 (ready) + - If no corresponding lib: score = 1 (independent) + - If corresponding lib is NOT up-to-date: score = 2 (wait for lib) + + Returns: + List of dicts with test info, sorted by priority + """ + all_tests = get_all_tests(cpython_prefix) + test_to_lib, lib_test_order = _build_test_to_lib_map(cpython_prefix) + + result = [] + for test_name in all_tests: + up_to_date = is_test_up_to_date(test_name, cpython_prefix, lib_prefix) + + if up_to_date and not include_done: + continue + + tracked = is_test_tracked(test_name, cpython_prefix, lib_prefix) + + # Check DEPENDENCIES mapping first, then fall back to simple extraction + if test_name in test_to_lib: + lib_name = test_to_lib[test_name] + # Get order from DEPENDENCIES + test_order = lib_test_order[lib_name].index(test_name) + else: + # Extract lib name from test name (test_foo -> foo) + lib_name = test_name.removeprefix("test_") + test_order = 0 # Default order for tests not in DEPENDENCIES + + # Check if corresponding lib is up-to-date + # Scoring: 0 = lib ready (highest priority), 1 = no lib, 2 = lib pending + if lib_status and lib_name in lib_status: + lib_up_to_date = lib_status[lib_name] + if lib_up_to_date: + score = 0 # Lib is ready, can update test + else: + score = 2 # Wait for lib first + else: + score = 1 # No corresponding lib (independent test) + + todo_count = count_test_todos(test_name, lib_prefix) if tracked else 0 + + result.append( + { + "name": test_name, + "lib_name": lib_name, + "score": score, + "up_to_date": up_to_date, + "tracked": tracked, + "todo_count": todo_count, + "test_order": test_order, + } + ) + + # Sort by score (ascending) + result.sort(key=lambda x: x["score"]) + + return result + + +def _format_test_suffix(item: dict) -> str: + """Format suffix for test item (TODO count or untracked).""" + tracked = item.get("tracked", True) + if not tracked: + return " (untracked)" + todo_count = item.get("todo_count", 0) + if todo_count > 0: + return f" ({todo_count} TODO)" + return "" + + +def format_test_todo_list( + todo_list: list[dict], + limit: int | None = None, +) -> list[str]: + """Format test todo list for display. + + Groups tests by lib_name. If multiple tests share the same lib_name, + the first test is shown as the primary and others are indented below it. + """ + lines = [] + + if limit: + todo_list = todo_list[:limit] + + # Group by lib_name + grouped: dict[str, list[dict]] = {} + for item in todo_list: + lib_name = item.get("lib_name", item["name"]) + if lib_name not in grouped: + grouped[lib_name] = [] + grouped[lib_name].append(item) + + # Sort each group by test_order (from DEPENDENCIES) + for tests in grouped.values(): + tests.sort(key=lambda x: x.get("test_order", 0)) + + for lib_name, tests in grouped.items(): + # First test is the primary + primary = tests[0] + done_mark = "[x]" if primary["up_to_date"] else "[ ]" + suffix = _format_test_suffix(primary) + lines.append(f"- {done_mark} {primary['name']}{suffix}") + + # Rest are indented + for item in tests[1:]: + done_mark = "[x]" if item["up_to_date"] else "[ ]" + suffix = _format_test_suffix(item) + lines.append(f" - {done_mark} {item['name']}{suffix}") + + return lines + + +def format_todo_list( + todo_list: list[dict], + test_by_lib: dict[str, list[dict]] | None = None, + limit: int | None = None, + verbose: bool = False, +) -> list[str]: + """Format todo list for display. + + Args: + todo_list: List from compute_todo_list() + test_by_lib: Dict mapping lib_name -> list of test infos (optional) + limit: Maximum number of items to show + verbose: Show detailed dependency information + + Returns: + List of formatted lines + """ + lines = [] + + if limit: + todo_list = todo_list[:limit] + + for item in todo_list: + name = item["name"] + score = item["score"] + total_deps = item["total_deps"] + rev_count = item["reverse_deps_count"] + + done_mark = "[x]" if item["up_to_date"] else "[ ]" + + if score == -1: + score_str = "no deps" + else: + score_str = f"{score}/{total_deps} deps" + + rev_str = f"{rev_count} dependents" if rev_count else "" + + parts = ["-", done_mark, f"[{score_str}]", f"`{name}`"] + if rev_str: + parts.append(f"({rev_str})") + + lines.append(" ".join(parts)) + + # Show hard_deps: + # - Normal mode: only show if lib is up-to-date but hard_deps are not + # - Verbose mode: always show all hard_deps with their status + hard_deps_status = item.get("hard_deps_status", {}) + if verbose and hard_deps_status: + for hd in sorted(hard_deps_status.keys()): + hd_mark = "[x]" if hard_deps_status[hd] else "[ ]" + lines.append(f" - {hd_mark} {hd} (hard_dep)") + elif item["up_to_date"]: + for hd, ok in sorted(hard_deps_status.items()): + if not ok: + lines.append(f" - [ ] {hd} (hard_dep)") + + # Show corresponding tests if exist + if test_by_lib and name in test_by_lib: + for test_info in test_by_lib[name]: + test_done_mark = "[x]" if test_info["up_to_date"] else "[ ]" + suffix = _format_test_suffix(test_info) + lines.append(f" - {test_done_mark} {test_info['name']}{suffix}") + + # Verbose mode: show detailed dependency info + if verbose: + if item["reverse_deps"]: + lines.append(f" dependents: {', '.join(sorted(item['reverse_deps']))}") + if item["soft_deps"]: + lines.append(f" python: {', '.join(sorted(item['soft_deps']))}") + if item["native_deps"]: + lines.append(f" native: {', '.join(sorted(item['native_deps']))}") + + return lines + + +def format_all_todo( + cpython_prefix: str, + lib_prefix: str, + limit: int | None = None, + include_done: bool = False, + verbose: bool = False, +) -> list[str]: + """Format prioritized list of modules and tests to update. + + Returns: + List of formatted lines + """ + from update_lib.cmd_deps import get_all_modules + from update_lib.deps import is_up_to_date + + lines = [] + + # Build lib status map for test scoring + lib_status = {} + for name in get_all_modules(cpython_prefix): + lib_status[name] = is_up_to_date(name, cpython_prefix, lib_prefix) + + # Compute test todo (always include all to find libs with pending tests) + test_todo = compute_test_todo_list( + cpython_prefix, lib_prefix, include_done=True, lib_status=lib_status + ) + + # Build test_by_lib map (only for tests with corresponding lib) + test_by_lib: dict[str, list[dict]] = {} + no_lib_tests = [] + # Set of libs that have pending tests + libs_with_pending_tests = set() + for test in test_todo: + if test["score"] == 1: # no lib + if not test["up_to_date"] or include_done: + no_lib_tests.append(test) + else: + lib_name = test["lib_name"] + if lib_name not in test_by_lib: + test_by_lib[lib_name] = [] + test_by_lib[lib_name].append(test) + if not test["up_to_date"]: + libs_with_pending_tests.add(lib_name) + + # Sort each lib's tests by test_order (from DEPENDENCIES) + for tests in test_by_lib.values(): + tests.sort(key=lambda x: x.get("test_order", 0)) + + # Compute lib todo - include libs with pending tests even if lib is done + lib_todo_base = compute_todo_list(cpython_prefix, lib_prefix, include_done=True) + + # Filter lib todo: include if lib is not done OR has pending test + lib_todo = [] + for item in lib_todo_base: + lib_not_done = not item["up_to_date"] + has_pending_test = item["name"] in libs_with_pending_tests + + if include_done or lib_not_done or has_pending_test: + lib_todo.append(item) + + # Format lib todo with embedded tests + lines.extend(format_todo_list(lib_todo, test_by_lib, limit, verbose)) + + # Format "no lib" tests separately if any + if no_lib_tests: + lines.append("") + lines.append("## Standalone Tests") + lines.extend(format_test_todo_list(no_lib_tests, limit)) + + # Format untracked files (in cpython but not in our Lib) + untracked = get_untracked_files(cpython_prefix, lib_prefix) + if untracked: + lines.append("") + lines.append("## Untracked Files") + display_untracked = untracked[:limit] if limit else untracked + for path in display_untracked: + lines.append(f"- {path}") + if limit and len(untracked) > limit: + lines.append(f" ... and {len(untracked) - limit} more") + + # Format original files (in our Lib but not in cpython) + original = get_original_files(cpython_prefix, lib_prefix) + if original: + lines.append("") + lines.append("## Original Files") + display_original = original[:limit] if limit else original + for path in display_original: + lines.append(f"- {path}") + if limit and len(original) > limit: + lines.append(f" ... and {len(original) - limit} more") + + return lines + + +def show_todo( + cpython_prefix: str, + lib_prefix: str, + limit: int | None = None, + include_done: bool = False, + verbose: bool = False, +) -> None: + """Show prioritized list of modules and tests to update.""" + for line in format_all_todo( + cpython_prefix, lib_prefix, limit, include_done, verbose + ): + print(line) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument( + "--cpython", + default="cpython", + help="CPython directory prefix (default: cpython)", + ) + parser.add_argument( + "--lib", + default="Lib", + help="Local Lib directory prefix (default: Lib)", + ) + parser.add_argument( + "--limit", + type=int, + default=None, + help="Maximum number of items to show", + ) + parser.add_argument( + "--done", + action="store_true", + help="Include already up-to-date modules", + ) + parser.add_argument( + "--verbose", + "-v", + action="store_true", + help="Show detailed dependency information", + ) + + args = parser.parse_args(argv) + + try: + show_todo(args.cpython, args.lib, args.limit, args.done, args.verbose) + return 0 + except Exception as e: + print(f"Error: {e}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/update_lib/deps.py b/scripts/update_lib/deps.py new file mode 100644 index 00000000000..7acffe88d0b --- /dev/null +++ b/scripts/update_lib/deps.py @@ -0,0 +1,1486 @@ +""" +Dependency resolution for library updates. + +Handles: +- Irregular library paths (e.g., libregrtest at Lib/test/libregrtest/) +- Library dependencies (e.g., datetime requires _pydatetime) +- Test dependencies (auto-detected from 'from test import ...') +""" + +import ast +import functools +import pathlib +import re +import shelve +import subprocess + +from update_lib.file_utils import ( + _dircmp_is_same, + compare_dir_contents, + compare_file_contents, + compare_paths, + construct_lib_path, + cpython_to_local_path, + read_python_files, + resolve_module_path, + resolve_test_path, + safe_parse_ast, + safe_read_text, +) + +# === Import parsing utilities === + + +def _extract_top_level_code(content: str) -> str: + """Extract only top-level code from Python content for faster parsing.""" + def_idx = content.find("\ndef ") + class_idx = content.find("\nclass ") + + indices = [i for i in (def_idx, class_idx) if i != -1] + if indices: + content = content[: min(indices)] + return content.rstrip("\n") + + +_FROM_TEST_IMPORT_RE = re.compile(r"^from test import (.+)", re.MULTILINE) +_FROM_TEST_DOT_RE = re.compile(r"^from test\.(\w+)", re.MULTILINE) +_IMPORT_TEST_DOT_RE = re.compile(r"^import test\.(\w+)", re.MULTILINE) + + +def parse_test_imports(content: str) -> set[str]: + """Parse test file content and extract test package dependencies.""" + content = _extract_top_level_code(content) + imports = set() + + for match in _FROM_TEST_IMPORT_RE.finditer(content): + import_list = match.group(1) + for part in import_list.split(","): + name = part.split()[0].strip() + if name and name not in ("support", "__init__"): + imports.add(name) + + for match in _FROM_TEST_DOT_RE.finditer(content): + dep = match.group(1) + if dep not in ("support", "__init__"): + imports.add(dep) + + for match in _IMPORT_TEST_DOT_RE.finditer(content): + dep = match.group(1) + if dep not in ("support", "__init__"): + imports.add(dep) + + return imports + + +_IMPORT_RE = re.compile(r"^import\s+(\w[\w.]*)", re.MULTILINE) +_FROM_IMPORT_RE = re.compile(r"^from\s+(\w[\w.]*)\s+import", re.MULTILINE) + + +def parse_lib_imports(content: str) -> set[str]: + """Parse library file and extract all imported module names.""" + imports = set() + + for match in _IMPORT_RE.finditer(content): + imports.add(match.group(1)) + + for match in _FROM_IMPORT_RE.finditer(content): + imports.add(match.group(1)) + + return imports + + +# === TODO marker utilities === + +TODO_MARKER = "TODO: RUSTPYTHON" + + +def filter_rustpython_todo(content: str) -> str: + """Remove lines containing RustPython TODO markers.""" + lines = content.splitlines(keepends=True) + filtered = [line for line in lines if TODO_MARKER not in line] + return "".join(filtered) + + +def count_rustpython_todo(content: str) -> int: + """Count lines containing RustPython TODO markers.""" + return sum(1 for line in content.splitlines() if TODO_MARKER in line) + + +def count_todo_in_path(path: pathlib.Path) -> int: + """Count RustPython TODO markers in a file or directory of .py files.""" + if path.is_file(): + content = safe_read_text(path) + return count_rustpython_todo(content) if content else 0 + + total = 0 + for _, content in read_python_files(path): + total += count_rustpython_todo(content) + return total + + +# === Test utilities === + + +def _get_cpython_test_path(test_name: str, cpython_prefix: str) -> pathlib.Path | None: + """Return the CPython test path for a test name, or None if missing.""" + cpython_path = resolve_test_path(test_name, cpython_prefix, prefer="dir") + return cpython_path if cpython_path.exists() else None + + +def _get_local_test_path( + cpython_test_path: pathlib.Path, lib_prefix: str +) -> pathlib.Path: + """Return the local Lib/test path matching a CPython test path.""" + return pathlib.Path(lib_prefix) / "test" / cpython_test_path.name + + +def is_test_tracked(test_name: str, cpython_prefix: str, lib_prefix: str) -> bool: + """Check if a test exists in the local Lib/test.""" + cpython_path = _get_cpython_test_path(test_name, cpython_prefix) + if cpython_path is None: + return True + local_path = _get_local_test_path(cpython_path, lib_prefix) + return local_path.exists() + + +def is_test_up_to_date(test_name: str, cpython_prefix: str, lib_prefix: str) -> bool: + """Check if a test is up-to-date, ignoring RustPython TODO markers.""" + cpython_path = _get_cpython_test_path(test_name, cpython_prefix) + if cpython_path is None: + return True + + local_path = _get_local_test_path(cpython_path, lib_prefix) + if not local_path.exists(): + return False + + if cpython_path.is_file(): + return compare_file_contents( + cpython_path, local_path, local_filter=filter_rustpython_todo + ) + + return compare_dir_contents( + cpython_path, local_path, local_filter=filter_rustpython_todo + ) + + +def count_test_todos(test_name: str, lib_prefix: str) -> int: + """Count RustPython TODO markers in a test file/directory.""" + local_dir = pathlib.Path(lib_prefix) / "test" / test_name + local_file = pathlib.Path(lib_prefix) / "test" / f"{test_name}.py" + + if local_dir.exists(): + return count_todo_in_path(local_dir) + if local_file.exists(): + return count_todo_in_path(local_file) + return 0 + + +# === Cross-process cache using shelve === + + +def _get_cpython_version(cpython_prefix: str) -> str: + """Get CPython version from git tag for cache namespace.""" + try: + result = subprocess.run( + ["git", "describe", "--tags", "--abbrev=0"], + cwd=cpython_prefix, + capture_output=True, + text=True, + ) + if result.returncode == 0: + return result.stdout.strip() + except Exception: + pass + return "unknown" + + +def _get_cache_path() -> str: + """Get cache file path (without extension - shelve adds its own).""" + cache_dir = pathlib.Path(__file__).parent / ".cache" + cache_dir.mkdir(parents=True, exist_ok=True) + return str(cache_dir / "import_graph_cache") + + +def clear_import_graph_caches() -> None: + """Clear in-process import graph caches (for testing).""" + if "_test_import_graph_cache" in globals(): + globals()["_test_import_graph_cache"].clear() + if "_lib_import_graph_cache" in globals(): + globals()["_lib_import_graph_cache"].clear() + + +# Manual dependency table for irregular cases +# Format: "name" -> {"lib": [...], "test": [...], "data": [...], "hard_deps": [...]} +# - lib: override default path (default: name.py or name/) +# - hard_deps: additional files to copy alongside the main module +DEPENDENCIES = { + # regrtest is in Lib/test/libregrtest/, not Lib/libregrtest/ + "regrtest": { + "lib": ["test/libregrtest"], + "test": ["test_regrtest"], + "data": ["test/regrtestdata"], + }, + # Rust-implemented modules (no lib file, only test) + "int": { + "lib": [], + "hard_deps": ["_pylong.py"], + "test": [ + "test_int.py", + "test_long.py", + "test_int_literal.py", + ], + }, + "exception": { + "lib": [], + "test": [ + "test_exceptions.py", + "test_baseexception.py", + "test_except_star.py", + "test_exception_group.py", + "test_exception_hierarchy.py", + "test_exception_variations.py", + ], + }, + "dict": { + "lib": [], + "test": [ + "test_dict.py", + "test_dictcomps.py", + "test_dictviews.py", + "test_userdict.py", + ], + }, + "list": { + "lib": [], + "test": [ + "test_list.py", + "test_listcomps.py", + "test_userlist.py", + ], + }, + "__future__": { + "test": [ + "test___future__.py", + "test_future_stmt.py", + ], + }, + "site": { + "hard_deps": ["_sitebuiltins.py"], + }, + "opcode": { + "hard_deps": ["_opcode_metadata.py"], + "test": [ + "test_opcode.py", + "test__opcode.py", + "test_opcodes.py", + ], + }, + "pickle": { + "hard_deps": ["_compat_pickle.py"], + "test": [ + "test_pickle.py", + "test_picklebuffer.py", + "test_pickletools.py", + ], + }, + "re": { + "hard_deps": ["sre_compile.py", "sre_constants.py", "sre_parse.py"], + }, + "weakref": { + "hard_deps": ["_weakrefset.py"], + "test": [ + "test_weakref.py", + "test_weakset.py", + ], + }, + "codecs": { + "test": [ + "test_codecs.py", + "test_codeccallbacks.py", + "test_codecencodings_cn.py", + "test_codecencodings_hk.py", + "test_codecencodings_iso2022.py", + "test_codecencodings_jp.py", + "test_codecencodings_kr.py", + "test_codecencodings_tw.py", + "test_codecmaps_cn.py", + "test_codecmaps_hk.py", + "test_codecmaps_jp.py", + "test_codecmaps_kr.py", + "test_codecmaps_tw.py", + "test_charmapcodec.py", + "test_multibytecodec.py", + ], + }, + # Non-pattern hard_deps (can't be auto-detected) + "ast": { + "hard_deps": ["_ast_unparse.py"], + "test": [ + "test_ast.py", + "test_unparse.py", + "test_type_comments.py", + ], + }, + # Data directories + "pydoc": { + "hard_deps": ["pydoc_data"], + }, + "turtle": { + "hard_deps": ["turtledemo"], + }, + "sysconfig": { + "hard_deps": ["_aix_support.py", "_osx_support.py"], + "test": [ + "test_sysconfig.py", + "test__osx_support.py", + ], + }, + "tkinter": { + "test": [ + "test_tkinter", + "test_ttk", + "test_ttk_textonly.py", + "test_tcl.py", + "test_idle", + ], + }, + # Test support library (like regrtest) + "support": { + "lib": ["test/support"], + "data": ["test/wheeldata"], + "test": [ + "test_support.py", + "test_script_helper.py", + ], + }, + # test_htmlparser tests html.parser + "html": { + "hard_deps": ["_markupbase.py"], + "test": ["test_html.py", "test_htmlparser.py"], + }, + "xml": { + "test": [ + "test_xml_etree.py", + "test_xml_etree_c.py", + "test_minidom.py", + "test_pulldom.py", + "test_pyexpat.py", + "test_sax.py", + "test_xml_dom_minicompat.py", + "test_xml_dom_xmlbuilder.py", + ], + }, + "multiprocessing": { + "test": [ + "test_multiprocessing_fork", + "test_multiprocessing_forkserver", + "test_multiprocessing_spawn", + "test_multiprocessing_main_handling.py", + ], + }, + "urllib": { + "test": [ + "test_urllib.py", + "test_urllib2.py", + "test_urllib2_localnet.py", + "test_urllib2net.py", + "test_urllibnet.py", + "test_urlparse.py", + "test_urllib_response.py", + "test_robotparser.py", + ], + }, + "collections": { + "hard_deps": ["_collections_abc.py"], + "test": [ + "test_collections.py", + "test_deque.py", + "test_defaultdict.py", + "test_ordered_dict.py", + ], + }, + "http": { + "test": [ + "test_httplib.py", + "test_http_cookiejar.py", + "test_http_cookies.py", + "test_httpservers.py", + ], + }, + "unicode": { + "lib": [], + "test": [ + "test_unicodedata.py", + "test_unicode_file.py", + "test_unicode_file_functions.py", + "test_unicode_identifiers.py", + "test_ucn.py", + ], + }, + "typing": { + "test": [ + "test_typing.py", + "test_type_aliases.py", + "test_type_annotations.py", + "test_type_params.py", + "test_genericalias.py", + ], + }, + "unpack": { + "lib": [], + "test": [ + "test_unpack.py", + "test_unpack_ex.py", + ], + }, + "zipimport": { + "test": [ + "test_zipimport.py", + "test_zipimport_support.py", + ], + }, + "time": { + "lib": [], + "test": [ + "test_time.py", + "test_strftime.py", + ], + }, + "sys": { + "lib": [], + "test": [ + "test_sys.py", + "test_syslog.py", + "test_sys_setprofile.py", + "test_sys_settrace.py", + ], + }, + "str": { + "lib": [], + "test": [ + "test_str.py", + "test_fstring.py", + "test_string_literals.py", + ], + }, + "thread": { + "lib": [], + "test": [ + "test_thread.py", + "test_thread_local_bytecode.py", + "test_threadsignals.py", + ], + }, + "threading": { + "hard_deps": ["_threading_local.py"], + "test": [ + "test_threading.py", + "test_threadedtempfile.py", + "test_threading_local.py", + ], + }, + "class": { + "lib": [], + "test": [ + "test_class.py", + "test_genericclass.py", + "test_subclassinit.py", + ], + }, + "generator": { + "lib": [], + "test": [ + "test_generators.py", + "test_genexps.py", + "test_generator_stop.py", + "test_yield_from.py", + ], + }, + "descr": { + "lib": [], + "test": [ + "test_descr.py", + "test_descrtut.py", + ], + }, + "code": { + "test": [ + "test_code_module.py", + ], + }, + "contextlib": { + "test": [ + "test_contextlib.py", + "test_contextlib_async.py", + ], + }, + "io": { + "test": [ + "test_io.py", + "test_bufio.py", + "test_fileio.py", + "test_memoryio.py", + ], + }, + "dbm": { + "test": [ + "test_dbm.py", + "test_dbm_dumb.py", + "test_dbm_gnu.py", + "test_dbm_ndbm.py", + "test_dbm_sqlite3.py", + ], + }, + "datetime": { + "hard_deps": ["_strptime.py"], + "test": [ + "test_datetime.py", + "test_strptime.py", + ], + }, + "locale": { + "test": [ + "test_locale.py", + "test__locale.py", + ], + }, + "numbers": { + "test": [ + "test_numbers.py", + "test_abstract_numbers.py", + ], + }, + "file": { + "lib": [], + "test": [ + "test_file.py", + "test_largefile.py", + ], + }, + "fcntl": { + "lib": [], + "test": [ + "test_fcntl.py", + "test_ioctl.py", + ], + }, + "select": { + "lib": [], + "test": [ + "test_select.py", + "test_poll.py", + ], + }, + "xmlrpc": { + "test": [ + "test_xmlrpc.py", + "test_docxmlrpc.py", + ], + }, + "ctypes": { + "test": [ + "test_ctypes", + "test_stable_abi_ctypes.py", + ], + }, + # Grouped tests for modules without custom lib paths + "compile": { + "lib": [], + "test": [ + "test_compile.py", + "test_compiler_assemble.py", + "test_compiler_codegen.py", + "test_peepholer.py", + ], + }, + "math": { + "lib": [], + "test": [ + "test_math.py", + "test_math_property.py", + ], + }, + "float": { + "lib": [], + "test": [ + "test_float.py", + "test_strtod.py", + ], + }, + "zipfile": { + "test": [ + "test_zipfile.py", + "test_zipfile64.py", + ], + }, + "smtplib": { + "test": [ + "test_smtplib.py", + "test_smtpnet.py", + ], + }, + "profile": { + "test": [ + "test_profile.py", + "test_cprofile.py", + ], + }, + "string": { + "test": [ + "test_string.py", + "test_userstring.py", + ], + }, + "os": { + "test": [ + "test_os.py", + "test_popen.py", + ], + }, + "pyrepl": { + "test": [ + "test_pyrepl", + "test_repl.py", + ], + }, + "concurrent": { + "test": [ + "test_concurrent_futures", + "test_interpreters", + "test__interpreters.py", + "test__interpchannels.py", + "test_crossinterp.py", + ], + }, +} + + +def resolve_hard_dep_parent(name: str, cpython_prefix: str) -> str | None: + """Resolve a hard_dep name to its parent module. + + Only returns a parent if the file is actually tracked: + - Explicitly listed in DEPENDENCIES as a hard_dep + - Or auto-detected _py{module}.py pattern where the parent module exists + + Args: + name: Module or file name (with or without .py extension) + cpython_prefix: CPython directory prefix + + Returns: + Parent module name if found and tracked, None otherwise + """ + # Normalize: remove .py extension if present + if name.endswith(".py"): + name = name[:-3] + + # Check DEPENDENCIES table first (explicit hard_deps) + for module_name, dep_info in DEPENDENCIES.items(): + hard_deps = dep_info.get("hard_deps", []) + for dep in hard_deps: + # Normalize dep: remove .py extension + dep_normalized = dep[:-3] if dep.endswith(".py") else dep + if dep_normalized == name: + return module_name + + # Auto-detect _py{module} or _py_{module} patterns + # Only if the parent module actually exists + if name.startswith("_py"): + if name.startswith("_py_"): + # _py_abc -> abc + parent = name[4:] + else: + # _pydatetime -> datetime + parent = name[3:] + + # Verify the parent module exists + lib_dir = pathlib.Path(cpython_prefix) / "Lib" + parent_file = lib_dir / f"{parent}.py" + parent_dir = lib_dir / parent + if parent_file.exists() or ( + parent_dir.exists() and (parent_dir / "__init__.py").exists() + ): + return parent + + return None + + +def resolve_test_to_lib(test_name: str) -> str | None: + """Resolve a test name to its library group from DEPENDENCIES. + + Args: + test_name: Test name with or without test_ prefix (e.g., "test_urllib2" or "urllib2") + + Returns: + Library name if test belongs to a group, None otherwise + """ + # Normalize: add test_ prefix if not present + if not test_name.startswith("test_"): + test_name = f"test_{test_name}" + + for lib_name, dep_info in DEPENDENCIES.items(): + tests = dep_info.get("test", []) + for test_path in tests: + # test_path is like "test_urllib2.py" or "test_multiprocessing_fork" + path_stem = test_path[:-3] if test_path.endswith(".py") else test_path + if path_stem == test_name: + return lib_name + + return None + + +# Test-specific dependencies (only when auto-detection isn't enough) +# - hard_deps: files to migrate (tightly coupled, must be migrated together) +# - data: directories to copy without migration +TEST_DEPENDENCIES = { + # Audio tests + "test_winsound": { + "data": ["audiodata"], + }, + "test_wave": { + "data": ["audiodata"], + }, + "audiotests": { + "data": ["audiodata"], + }, + # Archive tests + "test_tarfile": { + "data": ["archivetestdata"], + }, + "test_zipfile": { + "data": ["archivetestdata"], + }, + # Config tests + "test_configparser": { + "data": ["configdata"], + }, + "test_config": { + "data": ["configdata"], + }, + # Other data directories + "test_decimal": { + "data": ["decimaltestdata"], + }, + "test_dtrace": { + "data": ["dtracedata"], + }, + "test_math": { + "data": ["mathdata"], + }, + "test_ssl": { + "data": ["certdata"], + }, + "test_subprocess": { + "data": ["subprocessdata"], + }, + "test_tkinter": { + "data": ["tkinterdata"], + }, + "test_tokenize": { + "data": ["tokenizedata"], + }, + "test_type_annotations": { + "data": ["typinganndata"], + }, + "test_zipimport": { + "data": ["zipimport_data"], + }, + # XML tests share xmltestdata + "test_xml_etree": { + "data": ["xmltestdata"], + }, + "test_pulldom": { + "data": ["xmltestdata"], + }, + "test_sax": { + "data": ["xmltestdata"], + }, + "test_minidom": { + "data": ["xmltestdata"], + }, + # Multibytecodec support needs cjkencodings + "multibytecodec_support": { + "data": ["cjkencodings"], + }, + # i18n + "i18n_helper": { + "data": ["translationdata"], + }, + # wheeldata is used by test_makefile and support + "test_makefile": { + "data": ["wheeldata"], + }, +} + + +@functools.cache +def get_lib_paths(name: str, cpython_prefix: str) -> tuple[pathlib.Path, ...]: + """Get all library paths for a module. + + Args: + name: Module name (e.g., "datetime", "libregrtest") + cpython_prefix: CPython directory prefix + + Returns: + Tuple of paths to copy + """ + dep_info = DEPENDENCIES.get(name, {}) + + # Get main lib path (override or default) + if "lib" in dep_info: + paths = [construct_lib_path(cpython_prefix, p) for p in dep_info["lib"]] + else: + # Default: try file first, then directory + paths = [resolve_module_path(name, cpython_prefix, prefer="file")] + + # Add hard_deps from DEPENDENCIES + for dep in dep_info.get("hard_deps", []): + paths.append(construct_lib_path(cpython_prefix, dep)) + + # Auto-detect _py{module}.py or _py_{module}.py patterns + for pattern in [f"_py{name}.py", f"_py_{name}.py"]: + auto_path = construct_lib_path(cpython_prefix, pattern) + if auto_path.exists() and auto_path not in paths: + paths.append(auto_path) + + return tuple(paths) + + +def get_all_hard_deps(name: str, cpython_prefix: str) -> list[str]: + """Get all hard_deps for a module (explicit + auto-detected). + + Args: + name: Module name (e.g., "decimal", "datetime") + cpython_prefix: CPython directory prefix + + Returns: + List of hard_dep names (without .py extension) + """ + dep_info = DEPENDENCIES.get(name, {}) + hard_deps = set() + + # Explicit hard_deps from DEPENDENCIES + for hd in dep_info.get("hard_deps", []): + # Remove .py extension if present + hard_deps.add(hd[:-3] if hd.endswith(".py") else hd) + + # Auto-detect _py{module}.py or _py_{module}.py patterns + for pattern in [f"_py{name}.py", f"_py_{name}.py"]: + auto_path = construct_lib_path(cpython_prefix, pattern) + if auto_path.exists(): + hard_deps.add(auto_path.stem) + + return sorted(hard_deps) + + +@functools.cache +def get_test_paths(name: str, cpython_prefix: str) -> tuple[pathlib.Path, ...]: + """Get all test paths for a module. + + Args: + name: Module name (e.g., "datetime", "libregrtest") + cpython_prefix: CPython directory prefix + + Returns: + Tuple of test paths + """ + if name in DEPENDENCIES and "test" in DEPENDENCIES[name]: + return tuple( + construct_lib_path(cpython_prefix, f"test/{p}") + for p in DEPENDENCIES[name]["test"] + ) + + # Default: try directory first, then file + return (resolve_module_path(f"test/test_{name}", cpython_prefix, prefer="dir"),) + + +@functools.cache +def get_all_imports(name: str, cpython_prefix: str) -> frozenset[str]: + """Get all imports from a library file. + + Args: + name: Module name + cpython_prefix: CPython directory prefix + + Returns: + Frozenset of all imported module names + """ + all_imports = set() + for lib_path in get_lib_paths(name, cpython_prefix): + if lib_path.exists(): + for _, content in read_python_files(lib_path): + all_imports.update(parse_lib_imports(content)) + + # Remove self + all_imports.discard(name) + return frozenset(all_imports) + + +@functools.cache +def get_soft_deps(name: str, cpython_prefix: str) -> frozenset[str]: + """Get soft dependencies by parsing imports from library file. + + Args: + name: Module name + cpython_prefix: CPython directory prefix + + Returns: + Frozenset of imported stdlib module names (those that exist in cpython/Lib/) + """ + all_imports = get_all_imports(name, cpython_prefix) + + # Filter: only include modules that exist in cpython/Lib/ + stdlib_deps = set() + for imp in all_imports: + module_path = resolve_module_path(imp, cpython_prefix) + if module_path.exists(): + stdlib_deps.add(imp) + + return frozenset(stdlib_deps) + + +@functools.cache +def get_rust_deps(name: str, cpython_prefix: str) -> frozenset[str]: + """Get Rust/C dependencies (imports that don't exist in cpython/Lib/). + + Args: + name: Module name + cpython_prefix: CPython directory prefix + + Returns: + Frozenset of imported module names that are built-in or C extensions + """ + all_imports = get_all_imports(name, cpython_prefix) + soft_deps = get_soft_deps(name, cpython_prefix) + return frozenset(all_imports - soft_deps) + + +def is_path_synced( + cpython_path: pathlib.Path, + cpython_prefix: str, + lib_prefix: str, +) -> bool: + """Check if a CPython path is synced with local. + + Args: + cpython_path: Path in CPython directory + cpython_prefix: CPython directory prefix + lib_prefix: Local Lib directory prefix + + Returns: + True if synced, False otherwise + """ + local_path = cpython_to_local_path(cpython_path, cpython_prefix, lib_prefix) + if local_path is None: + return False + return compare_paths(cpython_path, local_path) + + +@functools.cache +def is_up_to_date(name: str, cpython_prefix: str, lib_prefix: str) -> bool: + """Check if a module is up-to-date by comparing files. + + Args: + name: Module name + cpython_prefix: CPython directory prefix + lib_prefix: Local Lib directory prefix + + Returns: + True if all files match, False otherwise + """ + lib_paths = get_lib_paths(name, cpython_prefix) + + found_any = False + for cpython_path in lib_paths: + if not cpython_path.exists(): + continue + + found_any = True + + # Convert cpython path to local path + # cpython/Lib/foo.py -> Lib/foo.py + rel_path = cpython_path.relative_to(cpython_prefix) + local_path = pathlib.Path(lib_prefix) / rel_path.relative_to("Lib") + + if not compare_paths(cpython_path, local_path): + return False + + if not found_any: + dep_info = DEPENDENCIES.get(name, {}) + if dep_info.get("lib") == []: + return True + return found_any + + +def get_test_dependencies( + test_path: pathlib.Path, +) -> dict[str, list[pathlib.Path]]: + """Get test dependencies by parsing imports. + + Args: + test_path: Path to test file or directory + + Returns: + Dict with "hard_deps" (files to migrate) and "data" (dirs to copy) + """ + result = {"hard_deps": [], "data": []} + + if not test_path.exists(): + return result + + # Parse all files for imports (auto-detect deps) + all_imports = set() + for _, content in read_python_files(test_path): + all_imports.update(parse_test_imports(content)) + + # Also add manual dependencies from TEST_DEPENDENCIES + test_name = test_path.stem if test_path.is_file() else test_path.name + manual_deps = TEST_DEPENDENCIES.get(test_name, {}) + if "hard_deps" in manual_deps: + all_imports.update(manual_deps["hard_deps"]) + + # Convert imports to paths (deps) + for imp in all_imports: + # Check if it's a test file (test_*) or support module + if imp.startswith("test_"): + # It's a test, resolve to test path + dep_path = test_path.parent / f"{imp}.py" + if not dep_path.exists(): + dep_path = test_path.parent / imp + else: + # Support module like string_tests, lock_tests, encoded_modules + # Check file first, then directory + dep_path = test_path.parent / f"{imp}.py" + if not dep_path.exists(): + dep_path = test_path.parent / imp + + if dep_path.exists() and dep_path not in result["hard_deps"]: + result["hard_deps"].append(dep_path) + + # Add data paths from manual table (for the test file itself) + if "data" in manual_deps: + for data_name in manual_deps["data"]: + data_path = test_path.parent / data_name + if data_path.exists() and data_path not in result["data"]: + result["data"].append(data_path) + + # Also add data from auto-detected deps' TEST_DEPENDENCIES + # e.g., test_codecencodings_kr -> multibytecodec_support -> cjkencodings + for imp in all_imports: + dep_info = TEST_DEPENDENCIES.get(imp, {}) + if "data" in dep_info: + for data_name in dep_info["data"]: + data_path = test_path.parent / data_name + if data_path.exists() and data_path not in result["data"]: + result["data"].append(data_path) + + return result + + +def _parse_test_submodule_imports(content: str) -> dict[str, set[str]]: + """Parse 'from test.X import Y' to get submodule imports. + + Args: + content: Python file content + + Returns: + Dict mapping submodule (e.g., "test_bar") -> set of imported names (e.g., {"helper"}) + """ + tree = safe_parse_ast(content) + if tree is None: + return {} + + result: dict[str, set[str]] = {} + for node in ast.walk(tree): + if isinstance(node, ast.ImportFrom): + if node.module and node.module.startswith("test."): + # from test.test_bar import helper -> test_bar: {helper} + parts = node.module.split(".") + if len(parts) >= 2: + submodule = parts[1] + if submodule not in ("support", "__init__"): + if submodule not in result: + result[submodule] = set() + for alias in node.names: + result[submodule].add(alias.name) + + return result + + +_test_import_graph_cache: dict[ + str, tuple[dict[str, set[str]], dict[str, set[str]]] +] = {} + + +def _is_standard_lib_path(path: str) -> bool: + """Check if path is the standard Lib directory (not a temp dir).""" + if "/tmp" in path.lower() or "/var/folders" in path.lower(): + return False + return ( + path == "Lib/test" + or path.endswith("/Lib/test") + or path == "Lib" + or path.endswith("/Lib") + ) + + +def _build_test_import_graph( + test_dir: pathlib.Path, +) -> tuple[dict[str, set[str]], dict[str, set[str]]]: + """Build import graphs for files within test directory (recursive). + + Uses cross-process shelve cache based on CPython version. + + Args: + test_dir: Path to Lib/test/ directory + + Returns: + Tuple of: + - Dict mapping relative path (without .py) -> set of test modules it imports + - Dict mapping relative path (without .py) -> set of all lib imports + """ + # In-process cache + cache_key = str(test_dir) + if cache_key in _test_import_graph_cache: + return _test_import_graph_cache[cache_key] + + # Cross-process cache (only for standard Lib/test directory) + use_file_cache = _is_standard_lib_path(cache_key) + if use_file_cache: + version = _get_cpython_version("cpython") + shelve_key = f"test_import_graph:{version}" + try: + with shelve.open(_get_cache_path()) as db: + if shelve_key in db: + import_graph, lib_imports_graph = db[shelve_key] + _test_import_graph_cache[cache_key] = ( + import_graph, + lib_imports_graph, + ) + return import_graph, lib_imports_graph + except Exception: + pass + + # Build from scratch + import_graph: dict[str, set[str]] = {} + lib_imports_graph: dict[str, set[str]] = {} + + for py_file in test_dir.glob("**/*.py"): + content = safe_read_text(py_file) + if content is None: + continue + + imports = set() + imports.update(parse_test_imports(content)) + all_imports = parse_lib_imports(content) + + for imp in all_imports: + if (py_file.parent / f"{imp}.py").exists(): + imports.add(imp) + if (test_dir / f"{imp}.py").exists(): + imports.add(imp) + + submodule_imports = _parse_test_submodule_imports(content) + for submodule, imported_names in submodule_imports.items(): + submodule_dir = test_dir / submodule + if submodule_dir.is_dir(): + for name in imported_names: + if (submodule_dir / f"{name}.py").exists(): + imports.add(name) + + rel_path = py_file.relative_to(test_dir) + key = str(rel_path.with_suffix("")) + import_graph[key] = imports + lib_imports_graph[key] = all_imports + + # Save to cross-process cache + if use_file_cache: + try: + with shelve.open(_get_cache_path()) as db: + db[shelve_key] = (import_graph, lib_imports_graph) + except Exception: + pass + _test_import_graph_cache[cache_key] = (import_graph, lib_imports_graph) + + return import_graph, lib_imports_graph + + +_lib_import_graph_cache: dict[str, dict[str, set[str]]] = {} + + +def _build_lib_import_graph(lib_prefix: str) -> dict[str, set[str]]: + """Build import graph for Lib modules (full module paths like urllib.request). + + Uses cross-process shelve cache based on CPython version. + + Args: + lib_prefix: RustPython Lib directory + + Returns: + Dict mapping full_module_path -> set of modules it imports + """ + # In-process cache + if lib_prefix in _lib_import_graph_cache: + return _lib_import_graph_cache[lib_prefix] + + # Cross-process cache (only for standard Lib directory) + use_file_cache = _is_standard_lib_path(lib_prefix) + if use_file_cache: + version = _get_cpython_version("cpython") + shelve_key = f"lib_import_graph:{version}" + try: + with shelve.open(_get_cache_path()) as db: + if shelve_key in db: + import_graph = db[shelve_key] + _lib_import_graph_cache[lib_prefix] = import_graph + return import_graph + except Exception: + pass + + # Build from scratch + lib_dir = pathlib.Path(lib_prefix) + if not lib_dir.exists(): + return {} + + import_graph: dict[str, set[str]] = {} + + for entry in lib_dir.iterdir(): + if entry.name.startswith(("_", ".")): + continue + if entry.name == "test": + continue + + if entry.is_file() and entry.suffix == ".py": + content = safe_read_text(entry) + if content: + imports = parse_lib_imports(content) + imports.discard(entry.stem) + import_graph[entry.stem] = imports + elif entry.is_dir() and (entry / "__init__.py").exists(): + for py_file in entry.glob("**/*.py"): + content = safe_read_text(py_file) + if content: + imports = parse_lib_imports(content) + rel_path = py_file.relative_to(lib_dir) + if rel_path.name == "__init__.py": + full_name = str(rel_path.parent).replace("/", ".") + else: + full_name = str(rel_path.with_suffix("")).replace("/", ".") + imports.discard(full_name.split(".")[0]) + import_graph[full_name] = imports + + # Save to cross-process cache + if use_file_cache: + try: + with shelve.open(_get_cache_path()) as db: + db[shelve_key] = import_graph + except Exception: + pass + _lib_import_graph_cache[lib_prefix] = import_graph + + return import_graph + + +def _get_lib_modules_importing( + module_name: str, lib_import_graph: dict[str, set[str]] +) -> set[str]: + """Find Lib modules (full paths) that import module_name or any of its submodules.""" + importers: set[str] = set() + target_top = module_name.split(".")[0] + + for full_path, imports in lib_import_graph.items(): + if full_path.split(".")[0] == target_top: + continue # Skip same package + # Match if module imports target OR any submodule of target + # e.g., for "xml": match imports of "xml", "xml.parsers", "xml.etree.ElementTree" + matches = any( + imp == module_name or imp.startswith(module_name + ".") for imp in imports + ) + if matches: + importers.add(full_path) + + return importers + + +def _consolidate_submodules( + modules: set[str], threshold: int = 3 +) -> dict[str, set[str]]: + """Consolidate submodules if count exceeds threshold. + + Args: + modules: Set of full module paths (e.g., {"urllib.request", "urllib.parse", "xml.dom", "xml.sax"}) + threshold: If submodules > threshold, consolidate to parent + + Returns: + Dict mapping display_name -> set of original module paths + e.g., {"urllib.request": {"urllib.request"}, "xml": {"xml.dom", "xml.sax", "xml.etree", "xml.parsers"}} + """ + # Group by top-level package + by_package: dict[str, set[str]] = {} + for mod in modules: + parts = mod.split(".") + top = parts[0] + if top not in by_package: + by_package[top] = set() + by_package[top].add(mod) + + result: dict[str, set[str]] = {} + for top, submods in by_package.items(): + if len(submods) > threshold: + # Consolidate to top-level + result[top] = submods + else: + # Keep individual + for mod in submods: + result[mod] = {mod} + + return result + + +# Modules that are used everywhere - show but don't expand their dependents +_BLOCKLIST_MODULES = frozenset( + { + "unittest", + "test.support", + "support", + "doctest", + "typing", + "abc", + "collections.abc", + "functools", + "itertools", + "operator", + "contextlib", + "warnings", + "types", + "enum", + "re", + "io", + "os", + "sys", + } +) + + +def find_dependent_tests_tree( + module_name: str, + lib_prefix: str, + max_depth: int = 1, + _depth: int = 0, + _visited_tests: set[str] | None = None, + _visited_modules: set[str] | None = None, +) -> dict: + """Find dependent tests in a tree structure. + + Args: + module_name: Module to search for (e.g., "ftplib") + lib_prefix: RustPython Lib directory + max_depth: Maximum depth to recurse (default 1 = show direct + 1 level of Lib deps) + + Returns: + Dict with structure: + { + "module": "ftplib", + "tests": ["test_ftplib", "test_urllib2"], # Direct importers + "children": [ + {"module": "urllib.request", "tests": [...], "children": []}, + ... + ] + } + """ + lib_dir = pathlib.Path(lib_prefix) + test_dir = lib_dir / "test" + + if _visited_tests is None: + _visited_tests = set() + if _visited_modules is None: + _visited_modules = set() + + # Build graphs + test_import_graph, test_lib_imports = _build_test_import_graph(test_dir) + lib_import_graph = _build_lib_import_graph(lib_prefix) + + # Find tests that directly import this module + target_top = module_name.split(".")[0] + direct_tests: set[str] = set() + for file_key, imports in test_lib_imports.items(): + if file_key in _visited_tests: + continue + # Match exact module OR any child submodule + # e.g., "xml" matches imports of "xml", "xml.parsers", "xml.etree.ElementTree" + # but "collections._defaultdict" only matches "collections._defaultdict" (no children) + matches = any( + imp == module_name or imp.startswith(module_name + ".") for imp in imports + ) + if matches: + # Check if it's a test file + if pathlib.Path(file_key).name.startswith("test_"): + direct_tests.add(file_key) + _visited_tests.add(file_key) + + # Consolidate test names (test_sqlite3/test_dbapi -> test_sqlite3) + consolidated_tests = {_consolidate_file_key(t) for t in direct_tests} + + # Mark this module as visited (cycle detection) + _visited_modules.add(module_name) + _visited_modules.add(target_top) + + children = [] + # Check blocklist and depth limit + should_expand = ( + _depth < max_depth + and module_name not in _BLOCKLIST_MODULES + and target_top not in _BLOCKLIST_MODULES + ) + + if should_expand: + # Find Lib modules that import this module + lib_importers = _get_lib_modules_importing(module_name, lib_import_graph) + + # Skip already visited modules (cycle detection) and blocklisted modules + lib_importers = { + m + for m in lib_importers + if m not in _visited_modules + and m.split(".")[0] not in _visited_modules + and m not in _BLOCKLIST_MODULES + and m.split(".")[0] not in _BLOCKLIST_MODULES + } + + # Consolidate submodules (xml.dom, xml.sax, xml.etree -> xml if > 3) + consolidated_libs = _consolidate_submodules(lib_importers, threshold=3) + + # Build children + for display_name, original_mods in sorted(consolidated_libs.items()): + child = find_dependent_tests_tree( + display_name, + lib_prefix, + max_depth, + _depth + 1, + _visited_tests, + _visited_modules, + ) + if child["tests"] or child["children"]: + children.append(child) + + return { + "module": module_name, + "tests": sorted(consolidated_tests), + "children": children, + } + + +def _consolidate_file_key(file_key: str) -> str: + """Consolidate file_key to test name. + + Args: + file_key: Relative path without .py (e.g., "test_foo", "test_bar/test_sub") + + Returns: + Consolidated test name: + - "test_foo" for "test_foo" + - "test_sqlite3" for "test_sqlite3/test_dbapi" + """ + parts = pathlib.Path(file_key).parts + if len(parts) == 1: + return parts[0] + return parts[0] diff --git a/scripts/update_lib/file_utils.py b/scripts/update_lib/file_utils.py new file mode 100644 index 00000000000..cb86ee2e664 --- /dev/null +++ b/scripts/update_lib/file_utils.py @@ -0,0 +1,289 @@ +""" +File utilities for update_lib. + +This module provides functions for: +- Safe file reading with error handling +- Safe AST parsing with error handling +- Iterating over Python files +- Parsing and converting library paths +- Detecting test paths vs library paths +- Comparing files or directories for equality +""" + +from __future__ import annotations + +import ast +import filecmp +import pathlib +from collections.abc import Callable, Iterator + +# === I/O utilities === + + +def safe_read_text(path: pathlib.Path) -> str | None: + """Read file content with UTF-8 encoding, returning None on error.""" + try: + return path.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError): + return None + + +def safe_parse_ast(content: str) -> ast.Module | None: + """Parse Python content into AST, returning None on syntax error.""" + try: + return ast.parse(content) + except SyntaxError: + return None + + +def iter_python_files(path: pathlib.Path) -> Iterator[pathlib.Path]: + """Yield Python files from a file or directory.""" + if path.is_file(): + yield path + else: + yield from path.glob("**/*.py") + + +def read_python_files(path: pathlib.Path) -> Iterator[tuple[pathlib.Path, str]]: + """Read all Python files from a path, yielding (path, content) pairs.""" + for py_file in iter_python_files(path): + content = safe_read_text(py_file) + if content is not None: + yield py_file, content + + +# === Path utilities === + + +def parse_lib_path(path: pathlib.Path | str) -> pathlib.Path: + """ + Extract the Lib/... portion from a path containing /Lib/. + + Example: + parse_lib_path("cpython/Lib/test/foo.py") -> Path("Lib/test/foo.py") + """ + path_str = str(path).replace("\\", "/") + lib_marker = "/Lib/" + + if lib_marker not in path_str: + raise ValueError(f"Path must contain '/Lib/' or '\\Lib\\' (got: {path})") + + idx = path_str.index(lib_marker) + return pathlib.Path(path_str[idx + 1 :]) + + +def is_lib_path(path: pathlib.Path) -> bool: + """Check if path starts with Lib/""" + path_str = str(path).replace("\\", "/") + return path_str.startswith("Lib/") or path_str.startswith("./Lib/") + + +def is_test_path(path: pathlib.Path) -> bool: + """Check if path is a test path (contains /Lib/test/ or starts with Lib/test/)""" + path_str = str(path).replace("\\", "/") + return "/Lib/test/" in path_str or path_str.startswith("Lib/test/") + + +def lib_to_test_path(src_path: pathlib.Path) -> pathlib.Path: + """ + Convert library path to test path. + + Examples: + cpython/Lib/dataclasses.py -> cpython/Lib/test/test_dataclasses/ + cpython/Lib/json/__init__.py -> cpython/Lib/test/test_json/ + """ + path_str = str(src_path).replace("\\", "/") + lib_marker = "/Lib/" + + if lib_marker in path_str: + lib_path = parse_lib_path(src_path) + lib_name = lib_path.stem if lib_path.suffix == ".py" else lib_path.name + if lib_name == "__init__": + lib_name = lib_path.parent.name + prefix = path_str[: path_str.index(lib_marker)] + dir_path = pathlib.Path(f"{prefix}/Lib/test/test_{lib_name}/") + if dir_path.exists(): + return dir_path + file_path = pathlib.Path(f"{prefix}/Lib/test/test_{lib_name}.py") + if file_path.exists(): + return file_path + return dir_path + else: + lib_name = src_path.stem if src_path.suffix == ".py" else src_path.name + if lib_name == "__init__": + lib_name = src_path.parent.name + dir_path = pathlib.Path(f"Lib/test/test_{lib_name}/") + if dir_path.exists(): + return dir_path + file_path = pathlib.Path(f"Lib/test/test_{lib_name}.py") + if file_path.exists(): + return file_path + return dir_path + + +def get_test_files(path: pathlib.Path) -> list[pathlib.Path]: + """Get all .py test files in a path (file or directory).""" + if path.is_file(): + return [path] + return sorted(path.glob("**/*.py")) + + +def get_test_module_name(test_path: pathlib.Path) -> str: + """ + Extract test module name from a test file path. + + Examples: + Lib/test/test_foo.py -> test_foo + Lib/test/test_ctypes/test_bar.py -> test_ctypes.test_bar + """ + test_path = pathlib.Path(test_path) + if test_path.parent.name.startswith("test_"): + return f"{test_path.parent.name}.{test_path.stem}" + return test_path.stem + + +def resolve_module_path( + name: str, prefix: str = "cpython", prefer: str = "file" +) -> pathlib.Path: + """ + Resolve module path, trying file or directory. + + Args: + name: Module name (e.g., "dataclasses", "json") + prefix: CPython directory prefix + prefer: "file" to try .py first, "dir" to try directory first + """ + file_path = pathlib.Path(f"{prefix}/Lib/{name}.py") + dir_path = pathlib.Path(f"{prefix}/Lib/{name}") + + if prefer == "file": + if file_path.exists(): + return file_path + if dir_path.exists(): + return dir_path + return file_path + else: + if dir_path.exists(): + return dir_path + if file_path.exists(): + return file_path + return dir_path + + +def construct_lib_path(prefix: str, *parts: str) -> pathlib.Path: + """Build a path under prefix/Lib/.""" + return pathlib.Path(prefix) / "Lib" / pathlib.Path(*parts) + + +def resolve_test_path( + test_name: str, prefix: str = "cpython", prefer: str = "dir" +) -> pathlib.Path: + """Resolve a test module path under Lib/test/.""" + return resolve_module_path(f"test/{test_name}", prefix, prefer=prefer) + + +def cpython_to_local_path( + cpython_path: pathlib.Path, + cpython_prefix: str, + lib_prefix: str, +) -> pathlib.Path | None: + """Convert CPython path to local Lib path.""" + try: + rel_path = cpython_path.relative_to(cpython_prefix) + return pathlib.Path(lib_prefix) / rel_path.relative_to("Lib") + except ValueError: + return None + + +def get_module_name(path: pathlib.Path) -> str: + """Extract module name from path, handling __init__.py.""" + if path.suffix == ".py": + name = path.stem + if name == "__init__": + return path.parent.name + return name + return path.name + + +def get_cpython_dir(src_path: pathlib.Path) -> pathlib.Path: + """Extract CPython directory from a path containing /Lib/.""" + path_str = str(src_path).replace("\\", "/") + lib_marker = "/Lib/" + if lib_marker in path_str: + idx = path_str.index(lib_marker) + return pathlib.Path(path_str[:idx]) + return pathlib.Path("cpython") + + +# === Comparison utilities === + + +def _dircmp_is_same(dcmp: filecmp.dircmp) -> bool: + """Recursively check if two directories are identical.""" + if dcmp.diff_files or dcmp.left_only or dcmp.right_only: + return False + + for subdir in dcmp.subdirs.values(): + if not _dircmp_is_same(subdir): + return False + + return True + + +def compare_paths(cpython_path: pathlib.Path, local_path: pathlib.Path) -> bool: + """Compare a CPython path with a local path (file or directory).""" + if not local_path.exists(): + return False + + if cpython_path.is_file(): + return filecmp.cmp(cpython_path, local_path, shallow=False) + + dcmp = filecmp.dircmp(cpython_path, local_path) + return _dircmp_is_same(dcmp) + + +def compare_file_contents( + cpython_path: pathlib.Path, + local_path: pathlib.Path, + *, + local_filter: Callable[[str], str] | None = None, + encoding: str = "utf-8", +) -> bool: + """Compare two files as text, optionally filtering local content.""" + try: + cpython_content = cpython_path.read_text(encoding=encoding) + local_content = local_path.read_text(encoding=encoding) + except (OSError, UnicodeDecodeError): + return False + + if local_filter is not None: + local_content = local_filter(local_content) + + return cpython_content == local_content + + +def compare_dir_contents( + cpython_dir: pathlib.Path, + local_dir: pathlib.Path, + *, + pattern: str = "*.py", + local_filter: Callable[[str], str] | None = None, + encoding: str = "utf-8", +) -> bool: + """Compare directory contents for matching files and text.""" + cpython_files = {f.relative_to(cpython_dir) for f in cpython_dir.rglob(pattern)} + local_files = {f.relative_to(local_dir) for f in local_dir.rglob(pattern)} + + if cpython_files != local_files: + return False + + for rel_path in cpython_files: + if not compare_file_contents( + cpython_dir / rel_path, + local_dir / rel_path, + local_filter=local_filter, + encoding=encoding, + ): + return False + + return True diff --git a/scripts/lib_updater.py b/scripts/update_lib/patch_spec.py old mode 100755 new mode 100644 similarity index 54% rename from scripts/lib_updater.py rename to scripts/update_lib/patch_spec.py index 8573705dd15..d27d2e22fa7 --- a/scripts/lib_updater.py +++ b/scripts/update_lib/patch_spec.py @@ -1,37 +1,14 @@ -#!/usr/bin/env python -__doc__ = """ -This tool helps with updating test files from CPython. +""" +Low-level module for converting between test files and JSON patches. -Examples --------- -To move the patches found in `Lib/test/foo.py` to ` ~/cpython/Lib/test/foo.py` then write the contents back to `Lib/test/foo.py` +This module handles: +- Extracting patches from test files (file -> JSON) +- Applying patches to test files (JSON -> file) +""" ->>> ./{fname} --from Lib/test/foo.py --to ~/cpython/Lib/test/foo.py -o Lib/test/foo.py - -You can run the same command without `-o` to override the `--from` path: - ->>> ./{fname} --from Lib/test/foo.py --to ~/cpython/Lib/test/foo.py - -To get a baseline of patches, you can alter the patches file with your favorite tool/script/etc and then reapply it with: - ->>> ./{fname} --from Lib/test/foo.py --show-patches -o my_patches.json - -(By default the output is set to print to stdout). - -When you want to apply your own patches: - ->>> ./{fname} -p my_patches.json --to Lib/test/foo.py -""".format(fname=__import__("os").path.basename(__file__)) - - -import argparse import ast import collections import enum -import json -import pathlib -import re -import sys import textwrap import typing @@ -105,13 +82,43 @@ def as_ast_node(self) -> ast.Attribute | ast.Call: def as_decorator(self) -> str: unparsed = ast.unparse(self.as_ast_node()) + # ast.unparse uses single quotes; convert to double quotes for ruff compatibility + unparsed = _single_to_double_quotes(unparsed) if not self.ut_method.has_args(): - unparsed = f"{unparsed} # {self._reason}" + unparsed = f"{unparsed} # {self._reason}" return f"@{unparsed}" +def _single_to_double_quotes(s: str) -> str: + """Convert single-quoted strings to double-quoted strings. + + Falls back to original if conversion breaks the AST equivalence. + """ + import re + + def replace_string(match: re.Match) -> str: + content = match.group(1) + # Unescape single quotes and escape double quotes + content = content.replace("\\'", "'").replace('"', '\\"') + return f'"{content}"' + + # Match single-quoted strings (handles escaped single quotes inside) + converted = re.sub(r"'((?:[^'\\]|\\.)*)'", replace_string, s) + + # Verify: parse converted and unparse should equal original + try: + converted_ast = ast.parse(converted, mode="eval") + if ast.unparse(converted_ast) == s: + return converted + except SyntaxError: + pass + + # Fall back to original if conversion failed + return s + + class PatchEntry(typing.NamedTuple): """ Stores patch metadata. @@ -131,9 +138,12 @@ class PatchEntry(typing.NamedTuple): spec: PatchSpec @classmethod - def iter_patch_entires( + def iter_patch_entries( cls, tree: ast.Module, lines: list[str] ) -> "Iterator[typing.Self]": + import re + import sys + for cls_node, fn_node in iter_tests(tree): parent_class = cls_node.name for dec_node in fn_node.decorator_list: @@ -214,7 +224,7 @@ def iter_tests( def iter_patches(contents: str) -> "Iterator[PatchEntry]": lines = contents.splitlines() tree = ast.parse(contents) - yield from PatchEntry.iter_patch_entires(tree, lines) + yield from PatchEntry.iter_patch_entries(tree, lines) def build_patch_dict(it: "Iterator[PatchEntry]") -> Patches: @@ -225,12 +235,40 @@ def build_patch_dict(it: "Iterator[PatchEntry]") -> Patches: return {k: dict(v) for k, v in patches.items()} -def iter_patch_lines(tree: ast.Module, patches: Patches) -> "Iterator[tuple[int, str]]": - cache = {} # Used in phase 2. Stores the end line location of a class name. +def extract_patches(contents: str) -> Patches: + """Extract patches from file contents and return as dict.""" + return build_patch_dict(iter_patches(contents)) + + +def _iter_patch_lines( + tree: ast.Module, patches: Patches +) -> "Iterator[tuple[int, str]]": + import sys + + # Build cache of all classes (for Phase 2 to find classes without methods) + cache = {} + # Build per-class set of async method names (for Phase 2 to generate correct override) + async_methods: dict[str, set[str]] = {} + # Track class bases for inherited async method lookup + class_bases: dict[str, list[str]] = {} + all_classes = {node.name for node in tree.body if isinstance(node, ast.ClassDef)} + for node in tree.body: + if isinstance(node, ast.ClassDef): + cache[node.name] = node.end_lineno + class_bases[node.name] = [ + base.id + for base in node.bases + if isinstance(base, ast.Name) and base.id in all_classes + ] + cls_async: set[str] = set() + for item in node.body: + if isinstance(item, ast.AsyncFunctionDef): + cls_async.add(item.name) + if cls_async: + async_methods[node.name] = cls_async # Phase 1: Iterate and mark existing tests for cls_node, fn_node in iter_tests(tree): - cache[cls_node.name] = cls_node.end_lineno specs = patches.get(cls_node.name, {}).pop(fn_node.name, None) if not specs: continue @@ -243,16 +281,36 @@ def iter_patch_lines(tree: ast.Module, patches: Patches) -> "Iterator[tuple[int, patch_lines = "\n".join(spec.as_decorator() for spec in specs) yield (lineno - 1, textwrap.indent(patch_lines, indent)) - # Phase 2: Iterate and mark inhereted tests - for cls_name, tests in patches.items(): + # Phase 2: Iterate and mark inherited tests + for cls_name, tests in sorted(patches.items()): lineno = cache.get(cls_name) if not lineno: print(f"WARNING: {cls_name} does not exist in remote file", file=sys.stderr) continue - for test_name, specs in tests.items(): + for test_name, specs in sorted(tests.items()): decorators = "\n".join(spec.as_decorator() for spec in specs) - patch_lines = f""" + # Check current class and ancestors for async method + is_async = False + queue = [cls_name] + visited: set[str] = set() + while queue: + cur = queue.pop(0) + if cur in visited: + continue + visited.add(cur) + if test_name in async_methods.get(cur, set()): + is_async = True + break + queue.extend(class_bases.get(cur, [])) + if is_async: + patch_lines = f""" +{decorators} +async def {test_name}(self): +{DEFAULT_INDENT}return await super().{test_name}() +""".rstrip() + else: + patch_lines = f""" {decorators} def {test_name}(self): {DEFAULT_INDENT}return super().{test_name}() @@ -260,12 +318,53 @@ def {test_name}(self): yield (lineno, textwrap.indent(patch_lines, DEFAULT_INDENT)) +def _has_unittest_import(tree: ast.Module) -> bool: + """Check if 'import unittest' is already present in the file.""" + for node in tree.body: + if isinstance(node, ast.Import): + for alias in node.names: + if alias.name == UT and alias.asname is None: + return True + return False + + +def _find_import_insert_line(tree: ast.Module) -> int: + """Find the line number after the last import statement.""" + last_import_line = None + for node in tree.body: + if isinstance(node, (ast.Import, ast.ImportFrom)): + last_import_line = node.end_lineno or node.lineno + if last_import_line is not None: + return last_import_line + # No imports found - insert after module docstring if present, else at top + if ( + tree.body + and isinstance(tree.body[0], ast.Expr) + and isinstance(tree.body[0].value, ast.Constant) + and isinstance(tree.body[0].value.value, str) + ): + return tree.body[0].end_lineno or tree.body[0].lineno + return 0 + + def apply_patches(contents: str, patches: Patches) -> str: + """Apply patches to file contents and return modified contents.""" tree = ast.parse(contents) lines = contents.splitlines() - modifications = list(iter_patch_lines(tree, patches)) - # Going in reverse to not distrupt the line offset + modifications = list(_iter_patch_lines(tree, patches)) + + # If we have modifications and unittest is not imported, add it + if modifications and not _has_unittest_import(tree): + import_line = _find_import_insert_line(tree) + modifications.append( + ( + import_line, + "\nimport unittest # XXX: RUSTPYTHON; importing to be able to skip tests", + ) + ) + + # Going in reverse to not disrupt the line offset for lineno, patch in sorted(modifications, reverse=True): lines.insert(lineno, patch) @@ -273,80 +372,26 @@ def apply_patches(contents: str, patches: Patches) -> str: return f"{joined}\n" -def write_output(data: str, dest: str) -> None: - if dest == "-": - print(data, end="") - return - - with open(dest, "w") as fd: - fd.write(data) - - -def build_argparse() -> argparse.ArgumentParser: - parser = argparse.ArgumentParser( - description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter - ) - - patches_group = parser.add_mutually_exclusive_group(required=True) - patches_group.add_argument( - "-p", - "--patches", - help="File path to file containing patches in a JSON format", - type=pathlib.Path, - ) - patches_group.add_argument( - "--from", - help="File to gather patches from", - dest="gather_from", - type=pathlib.Path, - ) - - group = parser.add_mutually_exclusive_group(required=True) - group.add_argument( - "--to", - help="File to apply patches to", - type=pathlib.Path, - ) - group.add_argument( - "--show-patches", action="store_true", help="Show the patches and exit" - ) - - parser.add_argument( - "-o", "--output", default="-", help="Output file. Set to '-' for stdout" - ) - - return parser - - -if __name__ == "__main__": - parser = build_argparse() - args = parser.parse_args() - - if args.patches: - patches = { - cls_name: { - test_name: [ - PatchSpec(**spec)._replace(ut_method=UtMethod(spec["ut_method"])) - for spec in specs - ] - for test_name, specs in tests.items() - } - for cls_name, tests in json.loads(args.patches.read_text()).items() +def patches_to_json(patches: Patches) -> dict: + """Convert patches to JSON-serializable dict.""" + return { + cls_name: { + test_name: [spec._asdict() for spec in specs] + for test_name, specs in tests.items() } - else: - patches = build_patch_dict(iter_patches(args.gather_from.read_text())) - - if args.show_patches: - patches = { - cls_name: { - test_name: [spec._asdict() for spec in specs] - for test_name, specs in tests.items() - } - for cls_name, tests in patches.items() + for cls_name, tests in patches.items() + } + + +def patches_from_json(data: dict) -> Patches: + """Convert JSON dict back to Patches.""" + return { + cls_name: { + test_name: [ + PatchSpec(**spec)._replace(ut_method=UtMethod(spec["ut_method"])) + for spec in specs + ] + for test_name, specs in tests.items() } - output = json.dumps(patches, indent=4) + "\n" - write_output(output, args.output) - sys.exit(0) - - patched = apply_patches(args.to.read_text(), patches) - write_output(patched, args.output) + for cls_name, tests in data.items() + } diff --git a/scripts/update_lib/tests/__init__.py b/scripts/update_lib/tests/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/scripts/update_lib/tests/test_auto_mark.py b/scripts/update_lib/tests/test_auto_mark.py new file mode 100644 index 00000000000..ce89b0f9918 --- /dev/null +++ b/scripts/update_lib/tests/test_auto_mark.py @@ -0,0 +1,1085 @@ +"""Tests for auto_mark.py - test result parsing and auto-marking.""" + +import ast +import pathlib +import subprocess +import tempfile +import unittest +from unittest import mock + +from update_lib.cmd_auto_mark import ( + Test, + TestResult, + TestRunError, + _expand_stripped_to_children, + _is_super_call_only, + apply_test_changes, + auto_mark_directory, + auto_mark_file, + collect_test_changes, + extract_test_methods, + parse_results, + path_to_test_parts, + remove_expected_failures, + strip_reasonless_expected_failures, +) +from update_lib.patch_spec import COMMENT + + +def _make_result(stdout: str) -> subprocess.CompletedProcess: + return subprocess.CompletedProcess( + args=["test"], returncode=0, stdout=stdout, stderr="" + ) + + +# -- fixtures shared across inheritance-aware tests -- + +BASE_TWO_CHILDREN = """import unittest + +class Base: + def test_foo(self): + pass + +class ChildA(Base, unittest.TestCase): + pass + +class ChildB(Base, unittest.TestCase): + pass +""" + +BASE_TWO_CHILDREN_ONE_OVERRIDE = """import unittest + +class Base: + def test_foo(self): + pass + +class ChildA(Base, unittest.TestCase): + pass + +class ChildB(Base, unittest.TestCase): + def test_foo(self): + # own implementation + pass +""" + + +class TestParseResults(unittest.TestCase): + """Tests for parse_results function.""" + + def test_parse_fail_and_error(self): + """FAIL and ERROR are collected; ok is ignored.""" + stdout = """\ +Run 3 tests sequentially +test_one (test.test_example.TestA.test_one) ... FAIL +test_two (test.test_example.TestA.test_two) ... ok +test_three (test.test_example.TestB.test_three) ... ERROR +----------- +""" + result = parse_results(_make_result(stdout)) + self.assertEqual(len(result.tests), 2) + by_name = {t.name: t for t in result.tests} + self.assertEqual(by_name["test_one"].path, "test.test_example.TestA.test_one") + self.assertEqual(by_name["test_one"].result, "fail") + self.assertEqual(by_name["test_three"].result, "error") + + def test_parse_unexpected_success(self): + stdout = """\ +Run 1 tests sequentially +test_foo (test.test_example.TestClass.test_foo) ... unexpected success +----------- +UNEXPECTED SUCCESS: test_foo (test.test_example.TestClass.test_foo) +""" + result = parse_results(_make_result(stdout)) + self.assertEqual(len(result.unexpected_successes), 1) + self.assertEqual(result.unexpected_successes[0].name, "test_foo") + self.assertEqual( + result.unexpected_successes[0].path, "test.test_example.TestClass.test_foo" + ) + + def test_parse_tests_result(self): + result = parse_results(_make_result("== Tests result: FAILURE ==\n")) + self.assertEqual(result.tests_result, "FAILURE") + + def test_parse_crashed_run_no_tests_result(self): + """Test results are still parsed when the runner crashes (no Tests result line).""" + stdout = """\ +Run 1 test sequentially in a single process +0:00:00 [1/1] test_ast +test_foo (test.test_ast.test_ast.TestA.test_foo) ... FAIL +test_bar (test.test_ast.test_ast.TestA.test_bar) ... ok +test_baz (test.test_ast.test_ast.TestB.test_baz) ... ERROR +""" + result = parse_results(_make_result(stdout)) + self.assertEqual(result.tests_result, "") + self.assertEqual(len(result.tests), 2) + names = {t.name for t in result.tests} + self.assertIn("test_foo", names) + self.assertIn("test_baz", names) + + def test_parse_crashed_run_has_unexpected_success(self): + """Unexpected successes are parsed even without Tests result line.""" + stdout = """\ +Run 1 test sequentially in a single process +0:00:00 [1/1] test_ast +test_foo (test.test_ast.test_ast.TestA.test_foo) ... unexpected success +UNEXPECTED SUCCESS: test_foo (test.test_ast.test_ast.TestA.test_foo) +""" + result = parse_results(_make_result(stdout)) + self.assertEqual(result.tests_result, "") + self.assertEqual(len(result.unexpected_successes), 1) + + def test_parse_error_messages(self): + """Single and multiple error messages are parsed from tracebacks.""" + stdout = """\ +Run 2 tests sequentially +test_foo (test.test_example.TestClass.test_foo) ... FAIL +test_bar (test.test_example.TestClass.test_bar) ... ERROR +----------- +====================================================================== +FAIL: test_foo (test.test_example.TestClass.test_foo) +---------------------------------------------------------------------- +Traceback (most recent call last): + File "test.py", line 10, in test_foo + self.assertEqual(1, 2) +AssertionError: 1 != 2 + +====================================================================== +ERROR: test_bar (test.test_example.TestClass.test_bar) +---------------------------------------------------------------------- +Traceback (most recent call last): + File "test.py", line 20, in test_bar + raise ValueError("oops") +ValueError: oops + +====================================================================== +""" + result = parse_results(_make_result(stdout)) + by_name = {t.name: t for t in result.tests} + self.assertEqual(by_name["test_foo"].error_message, "AssertionError: 1 != 2") + self.assertEqual(by_name["test_bar"].error_message, "ValueError: oops") + + def test_parse_directory_test_multiple_submodules(self): + """Failures across submodule boundaries are all detected.""" + stdout = """\ +Run 3 tests sequentially +0:00:00 [ 1/3] test_asyncio.test_buffered_proto +test_ok (test.test_asyncio.test_buffered_proto.TestProto.test_ok) ... ok + +---------------------------------------------------------------------- +Ran 1 tests in 0.1s + +OK + +0:00:01 [ 2/3] test_asyncio.test_events +test_create (test.test_asyncio.test_events.TestEvents.test_create) ... FAIL + +---------------------------------------------------------------------- +Ran 1 tests in 0.2s + +FAILED (failures=1) + +0:00:02 [ 3/3] test_asyncio.test_tasks +test_gather (test.test_asyncio.test_tasks.TestTasks.test_gather) ... ERROR + +---------------------------------------------------------------------- +Ran 1 tests in 0.3s + +FAILED (errors=1) + +== Tests result: FAILURE == +""" + result = parse_results(_make_result(stdout)) + self.assertEqual(len(result.tests), 2) + names = {t.name for t in result.tests} + self.assertIn("test_create", names) + self.assertIn("test_gather", names) + self.assertEqual(result.tests_result, "FAILURE") + + def test_parse_multiline_test_with_docstring(self): + """Two-line output (test_name + docstring ... RESULT) is handled.""" + stdout = """\ +Run 3 tests sequentially +test_ok (test.test_example.TestClass.test_ok) ... ok +test_with_doc (test.test_example.TestClass.test_with_doc) +Test that something works ... ERROR +test_normal_fail (test.test_example.TestClass.test_normal_fail) ... FAIL +""" + result = parse_results(_make_result(stdout)) + self.assertEqual(len(result.tests), 2) + names = {t.name for t in result.tests} + self.assertIn("test_with_doc", names) + self.assertIn("test_normal_fail", names) + test_doc = next(t for t in result.tests if t.name == "test_with_doc") + self.assertEqual(test_doc.path, "test.test_example.TestClass.test_with_doc") + self.assertEqual(test_doc.result, "error") + + +class TestPathToTestParts(unittest.TestCase): + def test_simple_path(self): + self.assertEqual( + path_to_test_parts("test.test_foo.TestClass.test_method"), + ["TestClass", "test_method"], + ) + + def test_nested_path(self): + self.assertEqual( + path_to_test_parts("test.test_foo.test_bar.TestClass.test_method"), + ["TestClass", "test_method"], + ) + + +class TestCollectTestChanges(unittest.TestCase): + def test_collect_failures_and_error_messages(self): + """Failures and error messages are collected; empty messages are omitted.""" + results = TestResult() + results.tests = [ + Test( + name="test_foo", + path="test.test_example.TestClass.test_foo", + result="fail", + error_message="AssertionError: 1 != 2", + ), + Test( + name="test_bar", + path="test.test_example.TestClass.test_bar", + result="error", + error_message="", + ), + ] + failing, successes, error_messages = collect_test_changes(results) + + self.assertEqual( + failing, {("TestClass", "test_foo"), ("TestClass", "test_bar")} + ) + self.assertEqual(successes, set()) + self.assertEqual(len(error_messages), 1) + self.assertEqual( + error_messages[("TestClass", "test_foo")], "AssertionError: 1 != 2" + ) + + def test_collect_unexpected_successes(self): + results = TestResult() + results.unexpected_successes = [ + Test( + name="test_foo", + path="test.test_example.TestClass.test_foo", + result="unexpected_success", + ), + ] + _, successes, _ = collect_test_changes(results) + self.assertEqual(successes, {("TestClass", "test_foo")}) + + def test_module_prefix_filtering(self): + """Prefix filters with both short and 'test.' prefix formats.""" + results = TestResult() + results.tests = [ + Test(name="test_foo", path="test_a.TestClass.test_foo", result="fail"), + Test( + name="test_bar", + path="test.test_dataclasses.TestCase.test_bar", + result="fail", + ), + Test( + name="test_baz", + path="test.test_other.TestOther.test_baz", + result="fail", + ), + ] + failing_a, _, _ = collect_test_changes(results, module_prefix="test_a.") + self.assertEqual(failing_a, {("TestClass", "test_foo")}) + + failing_dc, _, _ = collect_test_changes( + results, module_prefix="test.test_dataclasses." + ) + self.assertEqual(failing_dc, {("TestCase", "test_bar")}) + + def test_collect_init_module_matching(self): + """__init__.py tests match after stripping .__init__ from the prefix.""" + results = TestResult() + results.tests = [ + Test( + name="test_field_repr", + path="test.test_dataclasses.TestCase.test_field_repr", + result="fail", + ), + ] + module_prefix = "test_dataclasses.__init__" + if module_prefix.endswith(".__init__"): + module_prefix = module_prefix[:-9] + module_prefix = "test." + module_prefix + "." + + failing, _, _ = collect_test_changes(results, module_prefix=module_prefix) + self.assertEqual(failing, {("TestCase", "test_field_repr")}) + + +class TestExtractTestMethods(unittest.TestCase): + def test_extract_methods(self): + """Extracts from single and multiple classes.""" + code = """ +class TestA(unittest.TestCase): + def test_a(self): + pass + +class TestB(unittest.TestCase): + def test_b(self): + pass +""" + methods = extract_test_methods(code) + self.assertEqual(methods, {("TestA", "test_a"), ("TestB", "test_b")}) + + def test_extract_syntax_error_returns_empty(self): + self.assertEqual(extract_test_methods("this is not valid python {"), set()) + + +class TestRemoveExpectedFailures(unittest.TestCase): + def test_remove_comment_before(self): + code = f"""import unittest + +class TestFoo(unittest.TestCase): + # {COMMENT} + @unittest.expectedFailure + def test_one(self): + pass +""" + result = remove_expected_failures(code, {("TestFoo", "test_one")}) + self.assertNotIn("@unittest.expectedFailure", result) + self.assertIn("def test_one(self):", result) + + def test_remove_inline_comment(self): + code = f"""import unittest + +class TestFoo(unittest.TestCase): + @unittest.expectedFailure # {COMMENT} + def test_one(self): + pass +""" + result = remove_expected_failures(code, {("TestFoo", "test_one")}) + self.assertNotIn("@unittest.expectedFailure", result) + + def test_remove_super_call_method(self): + """Super-call-only override is removed entirely (sync).""" + code = f"""import unittest + +class TestFoo(unittest.TestCase): + # {COMMENT} + @unittest.expectedFailure + def test_one(self): + return super().test_one() +""" + result = remove_expected_failures(code, {("TestFoo", "test_one")}) + self.assertNotIn("def test_one", result) + + def test_remove_async_super_call_override(self): + """Super-call-only override is removed entirely (async).""" + code = f"""import unittest + +class BaseTest: + async def test_async_one(self): + pass + +class TestChild(BaseTest, unittest.TestCase): + # {COMMENT} + @unittest.expectedFailure + async def test_async_one(self): + return await super().test_async_one() +""" + result = remove_expected_failures(code, {("TestChild", "test_async_one")}) + self.assertNotIn("return await super().test_async_one()", result) + self.assertNotIn("@unittest.expectedFailure", result) + self.assertIn("class TestChild", result) + self.assertIn("async def test_async_one(self):", result) + + def test_remove_with_comment_after(self): + """Reason comment on the line after the decorator is also removed.""" + code = f"""import unittest + +class TestFoo(unittest.TestCase): + @unittest.expectedFailure # {COMMENT} + # RuntimeError: something went wrong + def test_one(self): + pass +""" + result = remove_expected_failures(code, {("TestFoo", "test_one")}) + self.assertNotIn("@unittest.expectedFailure", result) + self.assertNotIn("RuntimeError: something went wrong", result) + self.assertIn("def test_one(self):", result) + + def test_no_removal_without_comment(self): + """Decorators without our COMMENT marker are left untouched.""" + code = """import unittest + +class TestFoo(unittest.TestCase): + @unittest.expectedFailure + def test_one(self): + pass +""" + result = remove_expected_failures(code, {("TestFoo", "test_one")}) + self.assertIn("@unittest.expectedFailure", result) + + +class TestStripReasonlessExpectedFailures(unittest.TestCase): + def test_strip_reason_formats(self): + """Strips both inline-comment and comment-before formats when no reason.""" + for label, code in [ + ( + "inline", + f"""import unittest + +class TestFoo(unittest.TestCase): + @unittest.expectedFailure # {COMMENT} + def test_one(self): + pass +""", + ), + ( + "comment-before", + f"""import unittest + +class TestFoo(unittest.TestCase): + # {COMMENT} + @unittest.expectedFailure + def test_one(self): + pass +""", + ), + ]: + with self.subTest(label): + result, stripped = strip_reasonless_expected_failures(code) + self.assertNotIn("@unittest.expectedFailure", result) + self.assertIn("def test_one(self):", result) + self.assertEqual(stripped, {("TestFoo", "test_one")}) + + def test_keep_with_reason(self): + code = f"""import unittest + +class TestFoo(unittest.TestCase): + @unittest.expectedFailure # {COMMENT}; AssertionError: 1 != 2 + def test_one(self): + pass +""" + result, stripped = strip_reasonless_expected_failures(code) + self.assertIn("@unittest.expectedFailure", result) + self.assertEqual(stripped, set()) + + def test_strip_with_comment_after(self): + """Old-format reason comment on the next line is also removed.""" + code = f"""import unittest + +class TestFoo(unittest.TestCase): + @unittest.expectedFailure # {COMMENT} + # RuntimeError: something went wrong + def test_one(self): + pass +""" + result, stripped = strip_reasonless_expected_failures(code) + self.assertNotIn("RuntimeError", result) + self.assertIn("def test_one(self):", result) + self.assertEqual(stripped, {("TestFoo", "test_one")}) + + def test_strip_super_call_override(self): + """Super-call overrides are removed entirely (both comment formats).""" + for label, code in [ + ( + "comment-before", + f"""import unittest + +class _BaseTests: + def test_foo(self): + pass + +class TestChild(_BaseTests, unittest.TestCase): + # {COMMENT} + @unittest.expectedFailure + def test_foo(self): + return super().test_foo() +""", + ), + ( + "inline", + f"""import unittest + +class _BaseTests: + def test_foo(self): + pass + +class TestChild(_BaseTests, unittest.TestCase): + @unittest.expectedFailure # {COMMENT} + def test_foo(self): + return super().test_foo() +""", + ), + ]: + with self.subTest(label): + result, stripped = strip_reasonless_expected_failures(code) + self.assertNotIn("return super().test_foo()", result) + self.assertNotIn("@unittest.expectedFailure", result) + self.assertEqual(stripped, {("TestChild", "test_foo")}) + self.assertIn("class _BaseTests:", result) + + def test_no_strip_without_comment(self): + """Markers without our COMMENT are NOT stripped.""" + code = """import unittest + +class TestFoo(unittest.TestCase): + @unittest.expectedFailure + def test_one(self): + pass +""" + result, stripped = strip_reasonless_expected_failures(code) + self.assertIn("@unittest.expectedFailure", result) + self.assertEqual(stripped, set()) + + def test_mixed_with_and_without_reason(self): + code = f"""import unittest + +class TestFoo(unittest.TestCase): + @unittest.expectedFailure # {COMMENT} + def test_no_reason(self): + pass + + @unittest.expectedFailure # {COMMENT}; has a reason + def test_has_reason(self): + pass +""" + result, stripped = strip_reasonless_expected_failures(code) + self.assertEqual(stripped, {("TestFoo", "test_no_reason")}) + self.assertIn("has a reason", result) + self.assertEqual(result.count("@unittest.expectedFailure"), 1) + + +class TestExpandStrippedToChildren(unittest.TestCase): + def test_parent_to_children(self): + """Parent stripped → all/partial failing children returned.""" + stripped = {("Base", "test_foo")} + all_children = {("ChildA", "test_foo"), ("ChildB", "test_foo")} + + # All children fail + result = _expand_stripped_to_children(BASE_TWO_CHILDREN, stripped, all_children) + self.assertEqual(result, all_children) + + # Only one child fails + partial = {("ChildA", "test_foo")} + result = _expand_stripped_to_children(BASE_TWO_CHILDREN, stripped, partial) + self.assertEqual(result, partial) + + def test_direct_match(self): + code = """import unittest + +class TestFoo(unittest.TestCase): + def test_one(self): + pass +""" + s = {("TestFoo", "test_one")} + self.assertEqual(_expand_stripped_to_children(code, s, s), s) + + def test_child_with_own_override_excluded(self): + stripped = {("Base", "test_foo")} + all_failing = {("ChildA", "test_foo"), ("ChildB", "test_foo")} + result = _expand_stripped_to_children( + BASE_TWO_CHILDREN_ONE_OVERRIDE, stripped, all_failing + ) + # ChildA inherits → included; ChildB has own method → excluded + self.assertEqual(result, {("ChildA", "test_foo")}) + + +class TestApplyTestChanges(unittest.TestCase): + def test_apply_failing_tests(self): + code = """import unittest + +class TestFoo(unittest.TestCase): + def test_one(self): + pass +""" + result = apply_test_changes(code, {("TestFoo", "test_one")}, set()) + self.assertIn("@unittest.expectedFailure", result) + self.assertIn(COMMENT, result) + + def test_apply_removes_unexpected_success(self): + code = f"""import unittest + +class TestFoo(unittest.TestCase): + # {COMMENT} + @unittest.expectedFailure + def test_one(self): + pass +""" + result = apply_test_changes(code, set(), {("TestFoo", "test_one")}) + self.assertNotIn("@unittest.expectedFailure", result) + self.assertIn("def test_one(self):", result) + + def test_apply_both_changes(self): + code = f"""import unittest + +class TestFoo(unittest.TestCase): + def test_one(self): + pass + + # {COMMENT} + @unittest.expectedFailure + def test_two(self): + pass +""" + result = apply_test_changes( + code, {("TestFoo", "test_one")}, {("TestFoo", "test_two")} + ) + self.assertEqual(result.count("@unittest.expectedFailure"), 1) + + def test_apply_with_error_message(self): + code = """import unittest + +class TestFoo(unittest.TestCase): + def test_one(self): + pass +""" + result = apply_test_changes( + code, + {("TestFoo", "test_one")}, + set(), + {("TestFoo", "test_one"): "AssertionError: 1 != 2"}, + ) + self.assertIn("AssertionError: 1 != 2", result) + self.assertIn(COMMENT, result) + + +class TestConsolidateToParent(unittest.TestCase): + def test_all_children_fail_marks_parent_with_message(self): + """All subclasses fail → marks parent; error message is transferred.""" + failing = {("ChildA", "test_foo"), ("ChildB", "test_foo")} + error_messages = {("ChildA", "test_foo"): "RuntimeError: boom"} + result = apply_test_changes(BASE_TWO_CHILDREN, failing, set(), error_messages) + + self.assertEqual(result.count("@unittest.expectedFailure"), 1) + self.assertNotIn("return super()", result) + self.assertIn("RuntimeError: boom", result) + + def test_partial_children_fail_marks_children(self): + result = apply_test_changes(BASE_TWO_CHILDREN, {("ChildA", "test_foo")}, set()) + self.assertIn("return super().test_foo()", result) + self.assertEqual(result.count("@unittest.expectedFailure"), 1) + + def test_child_with_own_override_not_consolidated(self): + failing = {("ChildA", "test_foo"), ("ChildB", "test_foo")} + result = apply_test_changes(BASE_TWO_CHILDREN_ONE_OVERRIDE, failing, set()) + self.assertEqual(result.count("@unittest.expectedFailure"), 2) + + def test_strip_then_consolidate_restores_parent_marker(self): + """End-to-end: strip parent marker → child failures → re-mark on parent.""" + code = f"""import unittest + +class _BaseTests: + @unittest.expectedFailure # {COMMENT} + def test_foo(self): + pass + +class ChildA(_BaseTests, unittest.TestCase): + pass + +class ChildB(_BaseTests, unittest.TestCase): + pass +""" + stripped_code, stripped_tests = strip_reasonless_expected_failures(code) + self.assertEqual(stripped_tests, {("_BaseTests", "test_foo")}) + + all_failing = {("ChildA", "test_foo"), ("ChildB", "test_foo")} + error_messages = {("ChildA", "test_foo"): "RuntimeError: boom"} + + to_remark = _expand_stripped_to_children( + stripped_code, stripped_tests, all_failing + ) + self.assertEqual(to_remark, all_failing) + + result = apply_test_changes(stripped_code, to_remark, set(), error_messages) + self.assertIn("RuntimeError: boom", result) + self.assertEqual(result.count("@unittest.expectedFailure"), 1) + self.assertNotIn("return super()", result) + + +class TestSmartAutoMarkFiltering(unittest.TestCase): + """Tests for smart auto-mark filtering (new tests vs regressions).""" + + @staticmethod + def _filter(all_failing, original, current): + new = current - original + to_mark = {t for t in all_failing if t in new} + return to_mark, all_failing - to_mark + + def test_new_vs_regression(self): + """New failures are marked; existing (regression) failures are not.""" + original = {("TestFoo", "test_old1"), ("TestFoo", "test_old2")} + current = original | {("TestFoo", "test_new1"), ("TestFoo", "test_new2")} + all_failing = {("TestFoo", "test_old1"), ("TestFoo", "test_new1")} + + to_mark, regressions = self._filter(all_failing, original, current) + self.assertEqual(to_mark, {("TestFoo", "test_new1")}) + self.assertEqual(regressions, {("TestFoo", "test_old1")}) + + # Edge: all new → all marked + to_mark, regressions = self._filter(all_failing, set(), current) + self.assertEqual(to_mark, all_failing) + self.assertEqual(regressions, set()) + + # Edge: all old → nothing marked + to_mark, regressions = self._filter(all_failing, current, current) + self.assertEqual(to_mark, set()) + self.assertEqual(regressions, all_failing) + + def test_filters_across_classes(self): + original = {("TestA", "test_a"), ("TestB", "test_b")} + current = original | {("TestA", "test_new_a"), ("TestC", "test_c")} + all_failing = { + ("TestA", "test_a"), # regression + ("TestA", "test_new_a"), # new + ("TestC", "test_c"), # new (new class) + } + to_mark, regressions = self._filter(all_failing, original, current) + self.assertEqual(to_mark, {("TestA", "test_new_a"), ("TestC", "test_c")}) + self.assertEqual(regressions, {("TestA", "test_a")}) + + +class TestIsSuperCallOnly(unittest.TestCase): + @staticmethod + def _parse_method(code): + tree = ast.parse(code) + for node in ast.walk(tree): + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): + return node + return None + + def test_sync(self): + cases = [ + ("return super().test_one()", True), + ("return super().test_two()", False), # mismatched name + ("pass", False), # regular body + ("x = 1\n return super().test_one()", False), # multiple stmts + ] + for body, expected in cases: + with self.subTest(body=body): + code = f""" +class Foo: + def test_one(self): + {body} +""" + self.assertEqual( + _is_super_call_only(self._parse_method(code)), expected + ) + + def test_async(self): + cases = [ + ("return await super().test_one()", True), + ("return await super().test_two()", False), + ("return super().test_one()", True), # sync call in async method + ] + for body, expected in cases: + with self.subTest(body=body): + code = f""" +class Foo: + async def test_one(self): + {body} +""" + self.assertEqual( + _is_super_call_only(self._parse_method(code)), expected + ) + + +class TestAutoMarkFileWithCrashedRun(unittest.TestCase): + """auto_mark_file should process partial results when test runner crashes.""" + + CRASHED_STDOUT = """\ +Run 1 test sequentially in a single process +0:00:00 [1/1] test_example +test_foo (test.test_example.TestA.test_foo) ... FAIL +test_bar (test.test_example.TestA.test_bar) ... ok +====================================================================== +FAIL: test_foo (test.test_example.TestA.test_foo) +---------------------------------------------------------------------- +Traceback (most recent call last): + File "test.py", line 10, in test_foo + self.assertEqual(1, 2) +AssertionError: 1 != 2 +""" + + def test_auto_mark_file_crashed_run(self): + """auto_mark_file processes results even when tests_result is empty (crash).""" + test_code = f"""import unittest + +class TestA(unittest.TestCase): + def test_foo(self): + pass + + def test_bar(self): + pass +""" + with tempfile.TemporaryDirectory() as tmpdir: + test_file = pathlib.Path(tmpdir) / "test_example.py" + test_file.write_text(test_code) + + mock_result = TestResult() + mock_result.tests_result = "" + mock_result.tests = [ + Test( + name="test_foo", + path="test.test_example.TestA.test_foo", + result="fail", + error_message="AssertionError: 1 != 2", + ), + ] + + with mock.patch( + "update_lib.cmd_auto_mark.run_test", return_value=mock_result + ): + added, removed, regressions = auto_mark_file( + test_file, mark_failure=True, verbose=False + ) + + self.assertEqual(added, 1) + contents = test_file.read_text() + self.assertIn("expectedFailure", contents) + + def test_auto_mark_file_no_results_at_all_raises(self): + """auto_mark_file raises TestRunError when there are zero parsed results.""" + test_code = """import unittest + +class TestA(unittest.TestCase): + def test_foo(self): + pass +""" + with tempfile.TemporaryDirectory() as tmpdir: + test_file = pathlib.Path(tmpdir) / "test_example.py" + test_file.write_text(test_code) + + mock_result = TestResult() + mock_result.tests_result = "" + mock_result.tests = [] + mock_result.stdout = "some crash output" + + with mock.patch( + "update_lib.cmd_auto_mark.run_test", return_value=mock_result + ): + with self.assertRaises(TestRunError): + auto_mark_file(test_file, verbose=False) + + +class TestAutoMarkDirectoryWithCrashedRun(unittest.TestCase): + """auto_mark_directory should process partial results when test runner crashes.""" + + def test_auto_mark_directory_crashed_run(self): + """auto_mark_directory processes results even when tests_result is empty.""" + test_code = f"""import unittest + +class TestA(unittest.TestCase): + def test_foo(self): + pass +""" + with tempfile.TemporaryDirectory() as tmpdir: + test_dir = pathlib.Path(tmpdir) / "test_example" + test_dir.mkdir() + test_file = test_dir / "test_sub.py" + test_file.write_text(test_code) + + mock_result = TestResult() + mock_result.tests_result = "" + mock_result.tests = [ + Test( + name="test_foo", + path="test.test_example.test_sub.TestA.test_foo", + result="fail", + error_message="AssertionError: oops", + ), + ] + + with ( + mock.patch( + "update_lib.cmd_auto_mark.run_test", return_value=mock_result + ), + mock.patch( + "update_lib.cmd_auto_mark.get_test_module_name", + side_effect=lambda p: ( + "test_example" if p == test_dir else "test_example.test_sub" + ), + ), + ): + added, removed, regressions = auto_mark_directory( + test_dir, mark_failure=True, verbose=False + ) + + self.assertEqual(added, 1) + contents = test_file.read_text() + self.assertIn("expectedFailure", contents) + + def test_auto_mark_directory_no_results_raises(self): + """auto_mark_directory raises TestRunError when zero results.""" + with tempfile.TemporaryDirectory() as tmpdir: + test_dir = pathlib.Path(tmpdir) / "test_example" + test_dir.mkdir() + test_file = test_dir / "test_sub.py" + test_file.write_text("import unittest\n") + + mock_result = TestResult() + mock_result.tests_result = "" + mock_result.tests = [] + mock_result.stdout = "crash" + + with ( + mock.patch( + "update_lib.cmd_auto_mark.run_test", return_value=mock_result + ), + mock.patch( + "update_lib.cmd_auto_mark.get_test_module_name", + return_value="test_example", + ), + ): + with self.assertRaises(TestRunError): + auto_mark_directory(test_dir, verbose=False) + + +class TestAutoMarkFileRestoresOnCrash(unittest.TestCase): + """Stripped markers must be restored when the test runner crashes.""" + + def test_stripped_markers_restored_when_crash(self): + """Markers stripped before run must be restored for unobserved tests on crash.""" + test_code = f"""\ +import unittest + +class TestA(unittest.TestCase): + @unittest.expectedFailure # {COMMENT} + def test_foo(self): + pass + + @unittest.expectedFailure # {COMMENT} + def test_bar(self): + pass + + @unittest.expectedFailure # {COMMENT} + def test_baz(self): + pass +""" + with tempfile.TemporaryDirectory() as tmpdir: + test_file = pathlib.Path(tmpdir) / "test_example.py" + test_file.write_text(test_code) + + # Simulate a crashed run that only observed test_foo (failed) + # test_bar and test_baz never ran due to crash + mock_result = TestResult() + mock_result.tests_result = "" # no Tests result line (crash) + mock_result.tests = [ + Test( + name="test_foo", + path="test.test_example.TestA.test_foo", + result="fail", + error_message="AssertionError: 1 != 2", + ), + ] + + with mock.patch( + "update_lib.cmd_auto_mark.run_test", return_value=mock_result + ): + auto_mark_file(test_file, verbose=False) + + contents = test_file.read_text() + # test_bar and test_baz were not observed — their markers must be restored + self.assertIn("def test_bar", contents) + self.assertIn("def test_baz", contents) + # Count expectedFailure markers: all 3 should be present + self.assertEqual(contents.count("expectedFailure"), 3, contents) + + def test_stripped_markers_removed_when_complete_run(self): + """Markers are properly removed when the run completes normally.""" + test_code = f"""\ +import unittest + +class TestA(unittest.TestCase): + @unittest.expectedFailure # {COMMENT} + def test_foo(self): + pass + + @unittest.expectedFailure # {COMMENT} + def test_bar(self): + pass +""" + with tempfile.TemporaryDirectory() as tmpdir: + test_file = pathlib.Path(tmpdir) / "test_example.py" + test_file.write_text(test_code) + + # Simulate a complete run where test_foo fails but test_bar passes + mock_result = TestResult() + mock_result.tests_result = "FAILURE" # normal completion + mock_result.tests = [ + Test( + name="test_foo", + path="test.test_example.TestA.test_foo", + result="fail", + error_message="AssertionError", + ), + ] + # test_bar passes → shows as unexpected success + mock_result.unexpected_successes = [ + Test( + name="test_bar", + path="test.test_example.TestA.test_bar", + result="unexpected success", + ), + ] + + with mock.patch( + "update_lib.cmd_auto_mark.run_test", return_value=mock_result + ): + auto_mark_file(test_file, verbose=False) + + contents = test_file.read_text() + # test_foo should still have marker (re-added) + self.assertEqual(contents.count("expectedFailure"), 1, contents) + self.assertIn("def test_foo", contents) + + +class TestAutoMarkDirectoryRestoresOnCrash(unittest.TestCase): + """Stripped markers must be restored for directory runs that crash.""" + + def test_stripped_markers_restored_when_crash(self): + test_code = f"""\ +import unittest + +class TestA(unittest.TestCase): + @unittest.expectedFailure # {COMMENT} + def test_foo(self): + pass + + @unittest.expectedFailure # {COMMENT} + def test_bar(self): + pass +""" + with tempfile.TemporaryDirectory() as tmpdir: + test_dir = pathlib.Path(tmpdir) / "test_example" + test_dir.mkdir() + test_file = test_dir / "test_sub.py" + test_file.write_text(test_code) + + mock_result = TestResult() + mock_result.tests_result = "" # crash + mock_result.tests = [ + Test( + name="test_foo", + path="test.test_example.test_sub.TestA.test_foo", + result="fail", + ), + ] + + with ( + mock.patch( + "update_lib.cmd_auto_mark.run_test", return_value=mock_result + ), + mock.patch( + "update_lib.cmd_auto_mark.get_test_module_name", + side_effect=lambda p: ( + "test_example" if p == test_dir else "test_example.test_sub" + ), + ), + ): + auto_mark_directory(test_dir, verbose=False) + + contents = test_file.read_text() + # Both markers must be present (unobserved test_bar restored) + self.assertEqual(contents.count("expectedFailure"), 2, contents) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/update_lib/tests/test_copy_lib.py b/scripts/update_lib/tests/test_copy_lib.py new file mode 100644 index 00000000000..aca00cb18f3 --- /dev/null +++ b/scripts/update_lib/tests/test_copy_lib.py @@ -0,0 +1,75 @@ +"""Tests for copy_lib.py - library copying with dependencies.""" + +import pathlib +import tempfile +import unittest + + +class TestCopySingle(unittest.TestCase): + """Tests for _copy_single helper function.""" + + def test_copies_file(self): + """Test copying a single file.""" + from update_lib.cmd_copy_lib import _copy_single + + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + + src = tmpdir / "source.py" + src.write_text("content") + dst = tmpdir / "dest.py" + + _copy_single(src, dst, verbose=False) + + self.assertTrue(dst.exists()) + self.assertEqual(dst.read_text(), "content") + + def test_copies_directory(self): + """Test copying a directory.""" + from update_lib.cmd_copy_lib import _copy_single + + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + + src = tmpdir / "source_dir" + src.mkdir() + (src / "file.py").write_text("content") + dst = tmpdir / "dest_dir" + + _copy_single(src, dst, verbose=False) + + self.assertTrue(dst.exists()) + self.assertTrue((dst / "file.py").exists()) + + def test_removes_existing_before_copy(self): + """Test that existing destination is removed before copy.""" + from update_lib.cmd_copy_lib import _copy_single + + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + + src = tmpdir / "source.py" + src.write_text("new content") + dst = tmpdir / "dest.py" + dst.write_text("old content") + + _copy_single(src, dst, verbose=False) + + self.assertEqual(dst.read_text(), "new content") + + +class TestCopyLib(unittest.TestCase): + """Tests for copy_lib function.""" + + def test_raises_on_path_without_lib(self): + """Test that copy_lib raises ValueError when path doesn't contain /Lib/.""" + from update_lib.cmd_copy_lib import copy_lib + + with self.assertRaises(ValueError) as ctx: + copy_lib(pathlib.Path("some/path/without/lib.py")) + + self.assertIn("/Lib/", str(ctx.exception)) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/update_lib/tests/test_deps.py b/scripts/update_lib/tests/test_deps.py new file mode 100644 index 00000000000..d97af2867aa --- /dev/null +++ b/scripts/update_lib/tests/test_deps.py @@ -0,0 +1,394 @@ +"""Tests for deps.py - dependency resolution.""" + +import pathlib +import tempfile +import unittest + +from update_lib.deps import ( + get_lib_paths, + get_soft_deps, + get_test_dependencies, + get_test_paths, + parse_lib_imports, + parse_test_imports, +) + + +class TestParseTestImports(unittest.TestCase): + """Tests for parse_test_imports function.""" + + def test_from_test_import(self): + """Test parsing 'from test import foo'.""" + code = """ +from test import string_tests +from test import lock_tests, other_tests +""" + imports = parse_test_imports(code) + self.assertEqual(imports, {"string_tests", "lock_tests", "other_tests"}) + + def test_from_test_dot_module(self): + """Test parsing 'from test.foo import bar'.""" + code = """ +from test.string_tests import CommonTest +from test.support import verbose +""" + imports = parse_test_imports(code) + self.assertEqual(imports, {"string_tests"}) # support is excluded + + def test_excludes_support(self): + """Test that 'support' is excluded.""" + code = """ +from test import support +from test.support import verbose +""" + imports = parse_test_imports(code) + self.assertEqual(imports, set()) + + def test_regular_imports_ignored(self): + """Test that regular imports are ignored.""" + code = """ +import os +from collections import defaultdict +from . import helper +""" + imports = parse_test_imports(code) + self.assertEqual(imports, set()) + + def test_syntax_error_returns_empty(self): + """Test that syntax errors return empty set.""" + code = "this is not valid python {" + imports = parse_test_imports(code) + self.assertEqual(imports, set()) + + +class TestGetLibPaths(unittest.TestCase): + """Tests for get_lib_paths function.""" + + def test_auto_detect_py_module(self): + """Test auto-detection of _py{module}.py pattern.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + lib_dir = tmpdir / "Lib" + lib_dir.mkdir() + (lib_dir / "mymodule.py").write_text("# mymodule") + (lib_dir / "_pymymodule.py").write_text("# _pymymodule") + + paths = get_lib_paths("mymodule", str(tmpdir)) + self.assertEqual(len(paths), 2) + self.assertIn(tmpdir / "Lib" / "mymodule.py", paths) + self.assertIn(tmpdir / "Lib" / "_pymymodule.py", paths) + + def test_default_file(self): + """Test default to .py file.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + lib_dir = tmpdir / "Lib" + lib_dir.mkdir() + (lib_dir / "foo.py").write_text("# foo") + + paths = get_lib_paths("foo", str(tmpdir)) + self.assertEqual(paths, (tmpdir / "Lib" / "foo.py",)) + + def test_default_directory(self): + """Test default to directory when file doesn't exist.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + lib_dir = tmpdir / "Lib" + lib_dir.mkdir() + (lib_dir / "foo").mkdir() + + paths = get_lib_paths("foo", str(tmpdir)) + self.assertEqual(paths, (tmpdir / "Lib" / "foo",)) + + +class TestGetTestPaths(unittest.TestCase): + """Tests for get_test_paths function.""" + + def test_known_dependency(self): + """Test test with known path override.""" + paths = get_test_paths("regrtest", "cpython") + self.assertEqual(len(paths), 1) + self.assertEqual(paths[0], pathlib.Path("cpython/Lib/test/test_regrtest")) + + def test_default_directory(self): + """Test default to test_name/ directory.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + test_dir = tmpdir / "Lib" / "test" + test_dir.mkdir(parents=True) + (test_dir / "test_foo").mkdir() + + paths = get_test_paths("foo", str(tmpdir)) + self.assertEqual(paths, (tmpdir / "Lib" / "test" / "test_foo",)) + + def test_default_file(self): + """Test fallback to test_name.py file.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + test_dir = tmpdir / "Lib" / "test" + test_dir.mkdir(parents=True) + (test_dir / "test_foo.py").write_text("# test") + + paths = get_test_paths("foo", str(tmpdir)) + self.assertEqual(paths, (tmpdir / "Lib" / "test" / "test_foo.py",)) + + +class TestGetTestDependencies(unittest.TestCase): + """Tests for get_test_dependencies function.""" + + def test_parse_file_imports(self): + """Test parsing imports from test file.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + test_dir = tmpdir / "test" + test_dir.mkdir() + + # Create test file with import + test_file = test_dir / "test_foo.py" + test_file.write_text(""" +from test import string_tests + +class TestFoo: + pass +""") + # Create the dependency file + (test_dir / "string_tests.py").write_text("# string tests") + + result = get_test_dependencies(test_file) + self.assertEqual(len(result["hard_deps"]), 1) + self.assertEqual(result["hard_deps"][0], test_dir / "string_tests.py") + self.assertEqual(result["data"], []) + + def test_nonexistent_path(self): + """Test nonexistent path returns empty.""" + result = get_test_dependencies(pathlib.Path("/nonexistent/path")) + self.assertEqual(result, {"hard_deps": [], "data": []}) + + def test_transitive_data_dependency(self): + """Test that data deps are resolved transitively. + + Chain: test_codecencodings_kr -> multibytecodec_support -> cjkencodings + """ + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + test_dir = tmpdir / "test" + test_dir.mkdir() + + # Create test_codecencodings_kr.py that imports multibytecodec_support + test_file = test_dir / "test_codecencodings_kr.py" + test_file.write_text(""" +from test import multibytecodec_support + +class TestKR: + pass +""") + # Create multibytecodec_support.py (the intermediate dependency) + (test_dir / "multibytecodec_support.py").write_text("# support module") + + # Create cjkencodings directory (the data dependency of multibytecodec_support) + (test_dir / "cjkencodings").mkdir() + + result = get_test_dependencies(test_file) + + # Should find multibytecodec_support.py as a hard_dep + self.assertEqual(len(result["hard_deps"]), 1) + self.assertEqual( + result["hard_deps"][0], test_dir / "multibytecodec_support.py" + ) + + # Should find cjkencodings as data (from multibytecodec_support's TEST_DEPENDENCIES) + self.assertEqual(len(result["data"]), 1) + self.assertEqual(result["data"][0], test_dir / "cjkencodings") + + +class TestParseLibImports(unittest.TestCase): + """Tests for parse_lib_imports function.""" + + def test_import_statement(self): + """Test parsing 'import foo'.""" + code = """ +import os +import sys +import collections.abc +""" + imports = parse_lib_imports(code) + self.assertEqual(imports, {"os", "sys", "collections.abc"}) + + def test_from_import(self): + """Test parsing 'from foo import bar'.""" + code = """ +from os import path +from collections.abc import Mapping +from typing import Optional +""" + imports = parse_lib_imports(code) + self.assertEqual(imports, {"os", "collections.abc", "typing"}) + + def test_mixed_imports(self): + """Test mixed import styles.""" + code = """ +import sys +from os import path +from collections import defaultdict +import functools +""" + imports = parse_lib_imports(code) + self.assertEqual(imports, {"sys", "os", "collections", "functools"}) + + def test_syntax_error_returns_empty(self): + """Test that syntax errors return empty set.""" + code = "this is not valid python {" + imports = parse_lib_imports(code) + self.assertEqual(imports, set()) + + def test_relative_import_skipped(self): + """Test that relative imports (no module) are skipped.""" + code = """ +from . import foo +from .. import bar +""" + imports = parse_lib_imports(code) + self.assertEqual(imports, set()) + + +class TestGetSoftDeps(unittest.TestCase): + """Tests for get_soft_deps function.""" + + def test_with_temp_files(self): + """Test soft deps detection with temp files.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + lib_dir = tmpdir / "Lib" + lib_dir.mkdir() + + # Create a module that imports another module + (lib_dir / "foo.py").write_text(""" +import bar +from baz import something +""") + # Create the imported modules + (lib_dir / "bar.py").write_text("# bar module") + (lib_dir / "baz.py").write_text("# baz module") + + soft_deps = get_soft_deps("foo", str(tmpdir)) + self.assertEqual(soft_deps, {"bar", "baz"}) + + def test_skips_self(self): + """Test that module doesn't include itself in soft_deps.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + lib_dir = tmpdir / "Lib" + lib_dir.mkdir() + + # Create a module that imports itself (circular) + (lib_dir / "foo.py").write_text(""" +import foo +import bar +""") + (lib_dir / "bar.py").write_text("# bar module") + + soft_deps = get_soft_deps("foo", str(tmpdir)) + self.assertNotIn("foo", soft_deps) + self.assertIn("bar", soft_deps) + + def test_filters_nonexistent(self): + """Test that nonexistent modules are filtered out.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + lib_dir = tmpdir / "Lib" + lib_dir.mkdir() + + # Create a module that imports nonexistent module + (lib_dir / "foo.py").write_text(""" +import bar +import nonexistent +""") + (lib_dir / "bar.py").write_text("# bar module") + # nonexistent.py is NOT created + + soft_deps = get_soft_deps("foo", str(tmpdir)) + self.assertEqual(soft_deps, {"bar"}) + + +class TestDircmpIsSame(unittest.TestCase): + """Tests for _dircmp_is_same function.""" + + def test_identical_directories(self): + """Test that identical directories return True.""" + import filecmp + + from update_lib.deps import _dircmp_is_same + + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + dir1 = tmpdir / "dir1" + dir2 = tmpdir / "dir2" + dir1.mkdir() + dir2.mkdir() + + (dir1 / "file.py").write_text("content") + (dir2 / "file.py").write_text("content") + + dcmp = filecmp.dircmp(dir1, dir2) + self.assertTrue(_dircmp_is_same(dcmp)) + + def test_different_files(self): + """Test that directories with different files return False.""" + import filecmp + + from update_lib.deps import _dircmp_is_same + + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + dir1 = tmpdir / "dir1" + dir2 = tmpdir / "dir2" + dir1.mkdir() + dir2.mkdir() + + (dir1 / "file.py").write_text("content1") + (dir2 / "file.py").write_text("content2") + + dcmp = filecmp.dircmp(dir1, dir2) + self.assertFalse(_dircmp_is_same(dcmp)) + + def test_nested_identical(self): + """Test that nested identical directories return True.""" + import filecmp + + from update_lib.deps import _dircmp_is_same + + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + dir1 = tmpdir / "dir1" + dir2 = tmpdir / "dir2" + (dir1 / "sub").mkdir(parents=True) + (dir2 / "sub").mkdir(parents=True) + + (dir1 / "sub" / "file.py").write_text("content") + (dir2 / "sub" / "file.py").write_text("content") + + dcmp = filecmp.dircmp(dir1, dir2) + self.assertTrue(_dircmp_is_same(dcmp)) + + def test_nested_different(self): + """Test that nested directories with differences return False.""" + import filecmp + + from update_lib.deps import _dircmp_is_same + + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + dir1 = tmpdir / "dir1" + dir2 = tmpdir / "dir2" + (dir1 / "sub").mkdir(parents=True) + (dir2 / "sub").mkdir(parents=True) + + (dir1 / "sub" / "file.py").write_text("content1") + (dir2 / "sub" / "file.py").write_text("content2") + + dcmp = filecmp.dircmp(dir1, dir2) + self.assertFalse(_dircmp_is_same(dcmp)) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/update_lib/tests/test_migrate.py b/scripts/update_lib/tests/test_migrate.py new file mode 100644 index 00000000000..0cc247ba841 --- /dev/null +++ b/scripts/update_lib/tests/test_migrate.py @@ -0,0 +1,196 @@ +"""Tests for migrate.py - file migration operations.""" + +import pathlib +import tempfile +import unittest + +from update_lib.cmd_migrate import ( + patch_directory, + patch_file, + patch_single_content, +) +from update_lib.patch_spec import COMMENT + + +class TestPatchSingleContent(unittest.TestCase): + """Tests for patch_single_content function.""" + + def test_patch_with_no_existing_file(self): + """Test patching when lib file doesn't exist.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + + # Create source file + src_path = tmpdir / "src.py" + src_path.write_text("""import unittest + +class TestFoo(unittest.TestCase): + def test_one(self): + pass +""") + + # Non-existent lib path + lib_path = tmpdir / "lib.py" + + result = patch_single_content(src_path, lib_path) + + # Should return source content unchanged + self.assertIn("def test_one(self):", result) + self.assertNotIn(COMMENT, result) + + def test_patch_with_existing_patches(self): + """Test patching preserves existing patches.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + + # Create source file (new version) + src_path = tmpdir / "src.py" + src_path.write_text("""import unittest + +class TestFoo(unittest.TestCase): + def test_one(self): + pass + + def test_two(self): + pass +""") + + # Create lib file with existing patch + lib_path = tmpdir / "lib.py" + lib_path.write_text(f"""import unittest + +class TestFoo(unittest.TestCase): + # {COMMENT} + @unittest.expectedFailure + def test_one(self): + pass +""") + + result = patch_single_content(src_path, lib_path) + + # Should have patch on test_one + self.assertIn("@unittest.expectedFailure", result) + self.assertIn(COMMENT, result) + # Should have test_two from source + self.assertIn("def test_two(self):", result) + + +class TestPatchFile(unittest.TestCase): + """Tests for patch_file function.""" + + def test_patch_file_creates_output(self): + """Test that patch_file writes output file.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + + # Create source file + src_path = tmpdir / "src.py" + src_path.write_text("""import unittest + +class TestFoo(unittest.TestCase): + def test_one(self): + pass +""") + + # Output path + lib_path = tmpdir / "Lib" / "test.py" + + patch_file(src_path, lib_path, verbose=False) + + # File should exist + self.assertTrue(lib_path.exists()) + content = lib_path.read_text() + self.assertIn("def test_one(self):", content) + + def test_patch_file_preserves_patches(self): + """Test that patch_file preserves existing patches.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + + # Create source file + src_path = tmpdir / "src.py" + src_path.write_text("""import unittest + +class TestFoo(unittest.TestCase): + def test_one(self): + pass +""") + + # Create existing lib file with patch + lib_path = tmpdir / "lib.py" + lib_path.write_text(f"""import unittest + +class TestFoo(unittest.TestCase): + # {COMMENT} + @unittest.expectedFailure + def test_one(self): + pass +""") + + patch_file(src_path, lib_path, verbose=False) + + content = lib_path.read_text() + self.assertIn("@unittest.expectedFailure", content) + + +class TestPatchDirectory(unittest.TestCase): + """Tests for patch_directory function.""" + + def test_patch_directory_all_files(self): + """Test that patch_directory processes all .py files.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + + # Create source directory with files + src_dir = tmpdir / "src" + src_dir.mkdir() + (src_dir / "test_a.py").write_text("# test_a") + (src_dir / "test_b.py").write_text("# test_b") + (src_dir / "subdir").mkdir() + (src_dir / "subdir" / "test_c.py").write_text("# test_c") + + # Output directory + lib_dir = tmpdir / "lib" + + patch_directory(src_dir, lib_dir, verbose=False) + + # All files should exist + self.assertTrue((lib_dir / "test_a.py").exists()) + self.assertTrue((lib_dir / "test_b.py").exists()) + self.assertTrue((lib_dir / "subdir" / "test_c.py").exists()) + + def test_patch_directory_preserves_patches(self): + """Test that patch_directory preserves patches in existing files.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + + # Create source directory + src_dir = tmpdir / "src" + src_dir.mkdir() + (src_dir / "test_a.py").write_text("""import unittest + +class TestA(unittest.TestCase): + def test_one(self): + pass +""") + + # Create lib directory with patched file + lib_dir = tmpdir / "lib" + lib_dir.mkdir() + (lib_dir / "test_a.py").write_text(f"""import unittest + +class TestA(unittest.TestCase): + # {COMMENT} + @unittest.expectedFailure + def test_one(self): + pass +""") + + patch_directory(src_dir, lib_dir, verbose=False) + + content = (lib_dir / "test_a.py").read_text() + self.assertIn("@unittest.expectedFailure", content) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/update_lib/tests/test_patch_spec.py b/scripts/update_lib/tests/test_patch_spec.py new file mode 100644 index 00000000000..798bd851b3c --- /dev/null +++ b/scripts/update_lib/tests/test_patch_spec.py @@ -0,0 +1,362 @@ +"""Tests for patch_spec.py - core patch extraction and application.""" + +import ast +import unittest + +from update_lib.patch_spec import ( + COMMENT, + PatchSpec, + UtMethod, + _find_import_insert_line, + apply_patches, + extract_patches, + iter_tests, +) + + +class TestIterTests(unittest.TestCase): + """Tests for iter_tests function.""" + + def test_iter_tests_simple(self): + """Test iterating over test methods in a class.""" + code = """ +class TestFoo(unittest.TestCase): + def test_one(self): + pass + + def test_two(self): + pass +""" + tree = ast.parse(code) + results = list(iter_tests(tree)) + self.assertEqual(len(results), 2) + self.assertEqual(results[0][0].name, "TestFoo") + self.assertEqual(results[0][1].name, "test_one") + self.assertEqual(results[1][1].name, "test_two") + + def test_iter_tests_multiple_classes(self): + """Test iterating over multiple test classes.""" + code = """ +class TestFoo(unittest.TestCase): + def test_foo(self): + pass + +class TestBar(unittest.TestCase): + def test_bar(self): + pass +""" + tree = ast.parse(code) + results = list(iter_tests(tree)) + self.assertEqual(len(results), 2) + self.assertEqual(results[0][0].name, "TestFoo") + self.assertEqual(results[1][0].name, "TestBar") + + def test_iter_tests_async(self): + """Test iterating over async test methods.""" + code = """ +class TestAsync(unittest.TestCase): + async def test_async(self): + pass +""" + tree = ast.parse(code) + results = list(iter_tests(tree)) + self.assertEqual(len(results), 1) + self.assertEqual(results[0][1].name, "test_async") + + +class TestExtractPatches(unittest.TestCase): + """Tests for extract_patches function.""" + + def test_extract_expected_failure(self): + """Test extracting @unittest.expectedFailure decorator.""" + code = f""" +class TestFoo(unittest.TestCase): + # {COMMENT} + @unittest.expectedFailure + def test_one(self): + pass +""" + patches = extract_patches(code) + self.assertIn("TestFoo", patches) + self.assertIn("test_one", patches["TestFoo"]) + specs = patches["TestFoo"]["test_one"] + self.assertEqual(len(specs), 1) + self.assertEqual(specs[0].ut_method, UtMethod.ExpectedFailure) + + def test_extract_expected_failure_inline_comment(self): + """Test extracting expectedFailure with inline comment.""" + code = f""" +class TestFoo(unittest.TestCase): + @unittest.expectedFailure # {COMMENT} + def test_one(self): + pass +""" + patches = extract_patches(code) + self.assertIn("TestFoo", patches) + self.assertIn("test_one", patches["TestFoo"]) + + def test_extract_skip_with_reason(self): + """Test extracting @unittest.skip with reason.""" + code = f''' +class TestFoo(unittest.TestCase): + @unittest.skip("{COMMENT}; not implemented") + def test_one(self): + pass +''' + patches = extract_patches(code) + self.assertIn("TestFoo", patches) + specs = patches["TestFoo"]["test_one"] + self.assertEqual(specs[0].ut_method, UtMethod.Skip) + self.assertIn("not implemented", specs[0].reason) + + def test_extract_skip_if(self): + """Test extracting @unittest.skipIf decorator.""" + code = f''' +class TestFoo(unittest.TestCase): + @unittest.skipIf(sys.platform == "win32", "{COMMENT}; windows issue") + def test_one(self): + pass +''' + patches = extract_patches(code) + specs = patches["TestFoo"]["test_one"] + self.assertEqual(specs[0].ut_method, UtMethod.SkipIf) + # ast.unparse normalizes quotes to single quotes + self.assertIn("sys.platform", specs[0].cond) + self.assertIn("win32", specs[0].cond) + + def test_no_patches_without_comment(self): + """Test that decorators without COMMENT are not extracted.""" + code = """ +class TestFoo(unittest.TestCase): + @unittest.expectedFailure + def test_one(self): + pass +""" + patches = extract_patches(code) + self.assertEqual(patches, {}) + + def test_multiple_patches_same_method(self): + """Test extracting multiple decorators on same method.""" + code = f''' +class TestFoo(unittest.TestCase): + # {COMMENT} + @unittest.expectedFailure + @unittest.skip("{COMMENT}; reason") + def test_one(self): + pass +''' + patches = extract_patches(code) + specs = patches["TestFoo"]["test_one"] + self.assertEqual(len(specs), 2) + + +class TestApplyPatches(unittest.TestCase): + """Tests for apply_patches function.""" + + def test_apply_expected_failure(self): + """Test applying @unittest.expectedFailure.""" + code = """import unittest + +class TestFoo(unittest.TestCase): + def test_one(self): + pass +""" + patches = { + "TestFoo": {"test_one": [PatchSpec(UtMethod.ExpectedFailure, None, "")]} + } + result = apply_patches(code, patches) + self.assertIn("@unittest.expectedFailure", result) + self.assertIn(COMMENT, result) + + def test_apply_skip_with_reason(self): + """Test applying @unittest.skip with reason.""" + code = """import unittest + +class TestFoo(unittest.TestCase): + def test_one(self): + pass +""" + patches = { + "TestFoo": {"test_one": [PatchSpec(UtMethod.Skip, None, "not ready")]} + } + result = apply_patches(code, patches) + self.assertIn("@unittest.skip", result) + self.assertIn("not ready", result) + + def test_apply_skip_if(self): + """Test applying @unittest.skipIf.""" + code = """import unittest + +class TestFoo(unittest.TestCase): + def test_one(self): + pass +""" + patches = { + "TestFoo": { + "test_one": [ + PatchSpec(UtMethod.SkipIf, "sys.platform == 'win32'", "windows") + ] + } + } + result = apply_patches(code, patches) + self.assertIn("@unittest.skipIf", result) + self.assertIn('sys.platform == "win32"', result) + + def test_apply_preserves_existing_decorators(self): + """Test that existing decorators are preserved.""" + code = """import unittest + +class TestFoo(unittest.TestCase): + @some_decorator + def test_one(self): + pass +""" + patches = { + "TestFoo": {"test_one": [PatchSpec(UtMethod.ExpectedFailure, None, "")]} + } + result = apply_patches(code, patches) + self.assertIn("@some_decorator", result) + self.assertIn("@unittest.expectedFailure", result) + + def test_apply_inherited_method(self): + """Test applying patch to inherited method (creates override).""" + code = """import unittest + +class TestFoo(unittest.TestCase): + pass +""" + patches = { + "TestFoo": { + "test_inherited": [PatchSpec(UtMethod.ExpectedFailure, None, "")] + } + } + result = apply_patches(code, patches) + self.assertIn("def test_inherited(self):", result) + self.assertIn("return super().test_inherited()", result) + + def test_apply_adds_unittest_import(self): + """Test that unittest import is added if missing.""" + code = """import sys + +class TestFoo: + def test_one(self): + pass +""" + patches = { + "TestFoo": {"test_one": [PatchSpec(UtMethod.ExpectedFailure, None, "")]} + } + result = apply_patches(code, patches) + # Should add unittest import after existing imports + self.assertIn("import unittest", result) + + def test_apply_no_duplicate_import(self): + """Test that unittest import is not duplicated.""" + code = """import unittest + +class TestFoo(unittest.TestCase): + def test_one(self): + pass +""" + patches = { + "TestFoo": {"test_one": [PatchSpec(UtMethod.ExpectedFailure, None, "")]} + } + result = apply_patches(code, patches) + # Count occurrences of 'import unittest' + count = result.count("import unittest") + self.assertEqual(count, 1) + + +class TestPatchSpec(unittest.TestCase): + """Tests for PatchSpec class.""" + + def test_as_decorator_expected_failure(self): + """Test generating expectedFailure decorator string.""" + spec = PatchSpec(UtMethod.ExpectedFailure, None, "reason") + decorator = spec.as_decorator() + self.assertIn("@unittest.expectedFailure", decorator) + self.assertIn(COMMENT, decorator) + self.assertIn("reason", decorator) + + def test_as_decorator_skip(self): + """Test generating skip decorator string.""" + spec = PatchSpec(UtMethod.Skip, None, "not ready") + decorator = spec.as_decorator() + self.assertIn("@unittest.skip", decorator) + self.assertIn("not ready", decorator) + + def test_as_decorator_skip_if(self): + """Test generating skipIf decorator string.""" + spec = PatchSpec(UtMethod.SkipIf, "condition", "reason") + decorator = spec.as_decorator() + self.assertIn("@unittest.skipIf", decorator) + self.assertIn("condition", decorator) + + +class TestRoundTrip(unittest.TestCase): + """Tests for extract -> apply round trip.""" + + def test_round_trip_expected_failure(self): + """Test that extracted patches can be re-applied.""" + original = f"""import unittest + +class TestFoo(unittest.TestCase): + # {COMMENT} + @unittest.expectedFailure + def test_one(self): + pass +""" + # Extract patches + patches = extract_patches(original) + + # Apply to clean code + clean = """import unittest + +class TestFoo(unittest.TestCase): + def test_one(self): + pass +""" + result = apply_patches(clean, patches) + + # Should have the decorator + self.assertIn("@unittest.expectedFailure", result) + self.assertIn(COMMENT, result) + + +class TestFindImportInsertLine(unittest.TestCase): + """Tests for _find_import_insert_line function.""" + + def test_with_imports(self): + """Test finding line after imports.""" + code = """import os +import sys + +class Foo: + pass +""" + tree = ast.parse(code) + line = _find_import_insert_line(tree) + self.assertEqual(line, 2) + + def test_no_imports_with_docstring(self): + """Test fallback to after docstring when no imports.""" + code = '''"""Module docstring.""" + +class Foo: + pass +''' + tree = ast.parse(code) + line = _find_import_insert_line(tree) + self.assertEqual(line, 1) + + def test_no_imports_no_docstring(self): + """Test fallback to line 0 when no imports and no docstring.""" + code = """class Foo: + pass +""" + tree = ast.parse(code) + line = _find_import_insert_line(tree) + self.assertEqual(line, 0) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/update_lib/tests/test_path.py b/scripts/update_lib/tests/test_path.py new file mode 100644 index 00000000000..f2dcdcf8f05 --- /dev/null +++ b/scripts/update_lib/tests/test_path.py @@ -0,0 +1,224 @@ +"""Tests for path.py - path utilities.""" + +import pathlib +import tempfile +import unittest + +from update_lib.file_utils import ( + get_test_files, + get_test_module_name, + is_lib_path, + is_test_path, + lib_to_test_path, + parse_lib_path, +) + + +class TestParseLibPath(unittest.TestCase): + """Tests for parse_lib_path function.""" + + def test_parse_cpython_path(self): + """Test parsing cpython/Lib/... path.""" + result = parse_lib_path("cpython/Lib/test/test_foo.py") + self.assertEqual(result, pathlib.Path("Lib/test/test_foo.py")) + + def test_parse_nested_path(self): + """Test parsing deeply nested path.""" + result = parse_lib_path("/home/user/cpython/Lib/test/test_foo/test_bar.py") + self.assertEqual(result, pathlib.Path("Lib/test/test_foo/test_bar.py")) + + def test_parse_windows_path(self): + """Test parsing Windows-style path.""" + result = parse_lib_path("C:\\cpython\\Lib\\test\\test_foo.py") + self.assertEqual(result, pathlib.Path("Lib/test/test_foo.py")) + + def test_parse_directory(self): + """Test parsing directory path.""" + result = parse_lib_path("cpython/Lib/test/test_json/") + self.assertEqual(result, pathlib.Path("Lib/test/test_json/")) + + def test_parse_no_lib_raises(self): + """Test that path without /Lib/ raises ValueError.""" + with self.assertRaises(ValueError) as ctx: + parse_lib_path("some/random/path.py") + self.assertIn("/Lib/", str(ctx.exception)) + + +class TestIsLibPath(unittest.TestCase): + """Tests for is_lib_path function.""" + + def test_lib_path(self): + """Test detecting Lib/ path.""" + self.assertTrue(is_lib_path(pathlib.Path("Lib/test/test_foo.py"))) + self.assertTrue(is_lib_path(pathlib.Path("./Lib/test/test_foo.py"))) + + def test_cpython_path_not_lib(self): + """Test that cpython/Lib/ is not detected as lib path.""" + self.assertFalse(is_lib_path(pathlib.Path("cpython/Lib/test/test_foo.py"))) + + def test_random_path_not_lib(self): + """Test that random path is not lib path.""" + self.assertFalse(is_lib_path(pathlib.Path("some/other/path.py"))) + + +class TestIsTestPath(unittest.TestCase): + """Tests for is_test_path function.""" + + def test_cpython_test_path(self): + """Test detecting cpython test path.""" + self.assertTrue(is_test_path(pathlib.Path("cpython/Lib/test/test_foo.py"))) + + def test_lib_test_path(self): + """Test detecting Lib/test path.""" + self.assertTrue(is_test_path(pathlib.Path("Lib/test/test_foo.py"))) + + def test_library_path_not_test(self): + """Test that library path (not test) is not test path.""" + self.assertFalse(is_test_path(pathlib.Path("cpython/Lib/dataclasses.py"))) + self.assertFalse(is_test_path(pathlib.Path("Lib/dataclasses.py"))) + + +class TestLibToTestPath(unittest.TestCase): + """Tests for lib_to_test_path function.""" + + def test_prefers_directory_over_file(self): + """Test that directory is preferred when both exist.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + # Create structure: tmpdir/Lib/foo.py, tmpdir/Lib/test/test_foo/, tmpdir/Lib/test/test_foo.py + lib_dir = tmpdir / "Lib" + lib_dir.mkdir() + (lib_dir / "foo.py").write_text("# lib") + test_dir = lib_dir / "test" + test_dir.mkdir() + (test_dir / "test_foo").mkdir() + (test_dir / "test_foo.py").write_text("# test file") + + result = lib_to_test_path(tmpdir / "Lib" / "foo.py") + # Should prefer directory + self.assertEqual(result, tmpdir / "Lib" / "test" / "test_foo/") + + def test_falls_back_to_file(self): + """Test that file is used when directory doesn't exist.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + # Create structure: tmpdir/Lib/foo.py, tmpdir/Lib/test/test_foo.py (no directory) + lib_dir = tmpdir / "Lib" + lib_dir.mkdir() + (lib_dir / "foo.py").write_text("# lib") + test_dir = lib_dir / "test" + test_dir.mkdir() + (test_dir / "test_foo.py").write_text("# test file") + + result = lib_to_test_path(tmpdir / "Lib" / "foo.py") + # Should fall back to file + self.assertEqual(result, tmpdir / "Lib" / "test" / "test_foo.py") + + def test_defaults_to_directory_when_neither_exists(self): + """Test that directory path is returned when neither exists.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + lib_dir = tmpdir / "Lib" + lib_dir.mkdir() + (lib_dir / "foo.py").write_text("# lib") + test_dir = lib_dir / "test" + test_dir.mkdir() + # Neither test_foo/ nor test_foo.py exists + + result = lib_to_test_path(tmpdir / "Lib" / "foo.py") + # Should default to directory + self.assertEqual(result, tmpdir / "Lib" / "test" / "test_foo/") + + def test_lib_path_prefers_directory(self): + """Test Lib/ path prefers directory when it exists.""" + # This test uses actual Lib/ paths, checking current behavior + # When neither exists, defaults to directory + result = lib_to_test_path(pathlib.Path("Lib/nonexistent_module.py")) + self.assertEqual(result, pathlib.Path("Lib/test/test_nonexistent_module/")) + + def test_init_py_uses_parent_name(self): + """Test __init__.py uses parent directory name.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + # Create structure: tmpdir/Lib/json/__init__.py + lib_dir = tmpdir / "Lib" + lib_dir.mkdir() + json_dir = lib_dir / "json" + json_dir.mkdir() + (json_dir / "__init__.py").write_text("# json init") + test_dir = lib_dir / "test" + test_dir.mkdir() + + result = lib_to_test_path(tmpdir / "Lib" / "json" / "__init__.py") + # Should use "json" not "__init__" + self.assertEqual(result, tmpdir / "Lib" / "test" / "test_json/") + + def test_init_py_lib_path_uses_parent_name(self): + """Test __init__.py with Lib/ path uses parent directory name.""" + result = lib_to_test_path(pathlib.Path("Lib/json/__init__.py")) + # Should use "json" not "__init__" + self.assertEqual(result, pathlib.Path("Lib/test/test_json/")) + + +class TestGetTestFiles(unittest.TestCase): + """Tests for get_test_files function.""" + + def test_single_file(self): + """Test getting single file.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + test_file = tmpdir / "test.py" + test_file.write_text("# test") + + files = get_test_files(test_file) + self.assertEqual(len(files), 1) + self.assertEqual(files[0], test_file) + + def test_directory(self): + """Test getting all .py files from directory.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + (tmpdir / "test_a.py").write_text("# a") + (tmpdir / "test_b.py").write_text("# b") + (tmpdir / "not_python.txt").write_text("# not python") + + files = get_test_files(tmpdir) + self.assertEqual(len(files), 2) + names = [f.name for f in files] + self.assertIn("test_a.py", names) + self.assertIn("test_b.py", names) + + def test_nested_directory(self): + """Test getting .py files from nested directory.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + (tmpdir / "test_a.py").write_text("# a") + subdir = tmpdir / "subdir" + subdir.mkdir() + (subdir / "test_b.py").write_text("# b") + + files = get_test_files(tmpdir) + self.assertEqual(len(files), 2) + + +class TestTestNameFromPath(unittest.TestCase): + """Tests for get_test_module_name function.""" + + def test_simple_test_file(self): + """Test extracting name from simple test file.""" + path = pathlib.Path("Lib/test/test_foo.py") + self.assertEqual(get_test_module_name(path), "test_foo") + + def test_nested_test_file(self): + """Test extracting name from nested test directory.""" + path = pathlib.Path("Lib/test/test_ctypes/test_bar.py") + self.assertEqual(get_test_module_name(path), "test_ctypes.test_bar") + + def test_test_directory(self): + """Test extracting name from test directory.""" + path = pathlib.Path("Lib/test/test_json") + self.assertEqual(get_test_module_name(path), "test_json") + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/update_lib/tests/test_quick.py b/scripts/update_lib/tests/test_quick.py new file mode 100644 index 00000000000..f0262eebd04 --- /dev/null +++ b/scripts/update_lib/tests/test_quick.py @@ -0,0 +1,287 @@ +"""Tests for quick.py - quick update functionality.""" + +import pathlib +import tempfile +import unittest +from unittest.mock import patch + +from update_lib.cmd_quick import ( + _expand_shortcut, + collect_original_methods, + get_cpython_dir, + git_commit, +) +from update_lib.file_utils import lib_to_test_path + + +class TestGetCpythonDir(unittest.TestCase): + """Tests for get_cpython_dir function.""" + + def test_extract_from_full_path(self): + """Test extracting cpython dir from full path.""" + path = pathlib.Path("cpython/Lib/dataclasses.py") + result = get_cpython_dir(path) + self.assertEqual(result, pathlib.Path("cpython")) + + def test_extract_from_absolute_path(self): + """Test extracting cpython dir from absolute path.""" + path = pathlib.Path("/some/path/cpython/Lib/test/test_foo.py") + result = get_cpython_dir(path) + self.assertEqual(result, pathlib.Path("/some/path/cpython")) + + def test_shortcut_defaults_to_cpython(self): + """Test that shortcut (no /Lib/) defaults to 'cpython'.""" + path = pathlib.Path("dataclasses") + result = get_cpython_dir(path) + self.assertEqual(result, pathlib.Path("cpython")) + + +class TestExpandShortcut(unittest.TestCase): + """Tests for _expand_shortcut function.""" + + def test_expand_shortcut_to_test_path_integration(self): + """Test that expanded shortcut works with lib_to_test_path. + + This tests the fix for the bug where args.path was used instead of + the expanded src_path when calling lib_to_test_path. + """ + # Simulate the flow in main(): + # 1. User provides "dataclasses" + # 2. _expand_shortcut converts to "cpython/Lib/dataclasses.py" + # 3. lib_to_test_path should receive the expanded path, not original + + original_path = pathlib.Path("dataclasses") + expanded_path = _expand_shortcut(original_path) + + # If cpython/Lib/dataclasses.py exists, it should be expanded + if expanded_path != original_path: + # The expanded path should work with lib_to_test_path + test_path = lib_to_test_path(expanded_path) + # Should return a valid test path, not raise an error + self.assertTrue(str(test_path).startswith("cpython/Lib/test/")) + + # The original unexpanded path would fail or give wrong result + # This is what the bug was - using args.path instead of src_path + + def test_expand_shortcut_file(self): + """Test expanding a simple name to file path.""" + # This test checks the shortcut works when file exists + path = pathlib.Path("dataclasses") + result = _expand_shortcut(path) + + expected_file = pathlib.Path("cpython/Lib/dataclasses.py") + expected_dir = pathlib.Path("cpython/Lib/dataclasses") + + if expected_file.exists(): + self.assertEqual(result, expected_file) + elif expected_dir.exists(): + self.assertEqual(result, expected_dir) + else: + # If neither exists, should return original + self.assertEqual(result, path) + + def test_expand_shortcut_already_full_path(self): + """Test that full paths are not modified.""" + path = pathlib.Path("cpython/Lib/dataclasses.py") + result = _expand_shortcut(path) + self.assertEqual(result, path) + + def test_expand_shortcut_nonexistent(self): + """Test that nonexistent names are returned as-is.""" + path = pathlib.Path("nonexistent_module_xyz") + result = _expand_shortcut(path) + self.assertEqual(result, path) + + def test_expand_shortcut_uses_dependencies_table(self): + """Test that _expand_shortcut uses DEPENDENCIES table for overrides.""" + from update_lib.deps import DEPENDENCIES + + # regrtest has lib override in DEPENDENCIES + self.assertIn("regrtest", DEPENDENCIES) + self.assertIn("lib", DEPENDENCIES["regrtest"]) + + # _expand_shortcut should use this override when path exists + path = pathlib.Path("regrtest") + expected = pathlib.Path("cpython/Lib/test/libregrtest") + + # Only test expansion if cpython checkout exists + if expected.exists(): + result = _expand_shortcut(path) + self.assertEqual( + result, expected, "_expand_shortcut should expand 'regrtest'" + ) + + +class TestCollectOriginalMethods(unittest.TestCase): + """Tests for collect_original_methods function.""" + + def test_collect_from_file(self): + """Test collecting methods from single file.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + test_file = tmpdir / "test.py" + test_file.write_text(""" +class TestFoo: + def test_one(self): + pass + + def test_two(self): + pass +""") + + methods = collect_original_methods(test_file) + self.assertIsInstance(methods, set) + self.assertEqual(len(methods), 2) + self.assertIn(("TestFoo", "test_one"), methods) + self.assertIn(("TestFoo", "test_two"), methods) + + def test_collect_from_directory(self): + """Test collecting methods from directory.""" + with tempfile.TemporaryDirectory() as tmpdir: + tmpdir = pathlib.Path(tmpdir) + (tmpdir / "test_a.py").write_text(""" +class TestA: + def test_a(self): + pass +""") + (tmpdir / "test_b.py").write_text(""" +class TestB: + def test_b(self): + pass +""") + + methods = collect_original_methods(tmpdir) + self.assertIsInstance(methods, dict) + self.assertEqual(len(methods), 2) + + +class TestGitCommit(unittest.TestCase): + """Tests for git_commit function.""" + + @patch("subprocess.run") + @patch("update_lib.cmd_quick.get_cpython_version") + def test_none_lib_path_not_added(self, mock_version, mock_run): + """Test that None lib_path doesn't add '.' to git.""" + mock_version.return_value = "v3.14.0" + mock_run.return_value.returncode = 1 # Has changes + + with tempfile.TemporaryDirectory() as tmpdir: + test_file = pathlib.Path(tmpdir) / "test.py" + test_file.write_text("# test") + + git_commit("test", None, test_file, pathlib.Path("cpython"), verbose=False) + + # Check git add was called with only test_file, not "." + add_call = mock_run.call_args_list[0] + self.assertIn(str(test_file), add_call[0][0]) + self.assertNotIn(".", add_call[0][0][2:]) # Skip "git" and "add" + + @patch("subprocess.run") + @patch("update_lib.cmd_quick.get_cpython_version") + def test_none_test_path_not_added(self, mock_version, mock_run): + """Test that None test_path doesn't add '.' to git.""" + mock_version.return_value = "v3.14.0" + mock_run.return_value.returncode = 1 + + with tempfile.TemporaryDirectory() as tmpdir: + lib_file = pathlib.Path(tmpdir) / "lib.py" + lib_file.write_text("# lib") + + git_commit("lib", lib_file, None, pathlib.Path("cpython"), verbose=False) + + add_call = mock_run.call_args_list[0] + self.assertIn(str(lib_file), add_call[0][0]) + self.assertNotIn(".", add_call[0][0][2:]) + + def test_both_none_returns_false(self): + """Test that both paths None returns False without git operations.""" + # No mocking needed - should return early before any subprocess calls + result = git_commit("test", None, None, pathlib.Path("cpython"), verbose=False) + self.assertFalse(result) + + @patch("subprocess.run") + @patch("update_lib.cmd_quick.get_cpython_version") + def test_hard_deps_are_added(self, mock_version, mock_run): + """Test that hard_deps are included in git commit.""" + mock_version.return_value = "v3.14.0" + mock_run.return_value.returncode = 1 # Has changes + + with tempfile.TemporaryDirectory() as tmpdir: + lib_file = pathlib.Path(tmpdir) / "lib.py" + lib_file.write_text("# lib") + test_file = pathlib.Path(tmpdir) / "test.py" + test_file.write_text("# test") + dep_file = pathlib.Path(tmpdir) / "_dep.py" + dep_file.write_text("# dep") + + git_commit( + "test", + lib_file, + test_file, + pathlib.Path("cpython"), + hard_deps=[dep_file], + verbose=False, + ) + + # Check git add was called with all three files + add_call = mock_run.call_args_list[0] + add_args = add_call[0][0] + self.assertIn(str(lib_file), add_args) + self.assertIn(str(test_file), add_args) + self.assertIn(str(dep_file), add_args) + + @patch("subprocess.run") + @patch("update_lib.cmd_quick.get_cpython_version") + def test_nonexistent_hard_deps_not_added(self, mock_version, mock_run): + """Test that nonexistent hard_deps don't cause errors.""" + mock_version.return_value = "v3.14.0" + mock_run.return_value.returncode = 1 # Has changes + + with tempfile.TemporaryDirectory() as tmpdir: + lib_file = pathlib.Path(tmpdir) / "lib.py" + lib_file.write_text("# lib") + nonexistent_dep = pathlib.Path(tmpdir) / "nonexistent.py" + + git_commit( + "test", + lib_file, + None, + pathlib.Path("cpython"), + hard_deps=[nonexistent_dep], + verbose=False, + ) + + # Check git add was called with only lib_file + add_call = mock_run.call_args_list[0] + add_args = add_call[0][0] + self.assertIn(str(lib_file), add_args) + self.assertNotIn(str(nonexistent_dep), add_args) + + +class TestQuickTestRunFailure(unittest.TestCase): + """Tests for quick() behavior when test run fails.""" + + @patch("update_lib.cmd_auto_mark.run_test") + def test_auto_mark_raises_on_test_run_failure(self, mock_run_test): + """Test that auto_mark_file raises when test run fails entirely.""" + from update_lib.cmd_auto_mark import TestResult, TestRunError, auto_mark_file + + # Simulate test runner crash (empty tests_result) + mock_run_test.return_value = TestResult( + tests_result="", tests=[], stdout="crash" + ) + + with tempfile.TemporaryDirectory() as tmpdir: + # Create a fake test file with Lib/test structure + lib_test_dir = pathlib.Path(tmpdir) / "Lib" / "test" + lib_test_dir.mkdir(parents=True) + test_file = lib_test_dir / "test_foo.py" + test_file.write_text("import unittest\nclass Test(unittest.TestCase): pass") + + # auto_mark_file should raise TestRunError + with self.assertRaises(TestRunError): + auto_mark_file(test_file) + + +if __name__ == "__main__": + unittest.main() diff --git a/whats_left.py b/scripts/whats_left.py similarity index 92% rename from whats_left.py rename to scripts/whats_left.py index c5b0be6eadc..00db9a0ac5c 100755 --- a/whats_left.py +++ b/scripts/whats_left.py @@ -1,6 +1,6 @@ #!/usr/bin/env -S python3 -I # /// script -# requires-python = ">=3.13" +# requires-python = ">=3.14" # /// # This script generates Lib/snippets/whats_left_data.py with these variables defined: @@ -29,7 +29,7 @@ if not sys.flags.isolated: print("running without -I option.") - print("python -I whats_left.py") + print("python -I scripts/whats_left.py") exit(1) GENERATED_FILE = "extra_tests/not_impl.py" @@ -37,9 +37,9 @@ implementation = platform.python_implementation() if implementation != "CPython": sys.exit(f"whats_left.py must be run under CPython, got {implementation} instead") -if sys.version_info[:2] < (3, 13): +if sys.version_info[:2] < (3, 14): sys.exit( - f"whats_left.py must be run under CPython 3.13 or newer, got {implementation} {sys.version} instead. If you have uv, try `uv run python -I whats_left.py` to select a proper Python interpreter easier." + f"whats_left.py must be run under CPython 3.14 or newer, got {implementation} {sys.version} instead. If you have uv, try `uv run python -I scripts/whats_left.py` to select a proper Python interpreter easier." ) @@ -60,6 +60,11 @@ def parse_args(): action="store_true", help="print output as JSON (instead of line by line)", ) + parser.add_argument( + "--no-default-features", + action="store_true", + help="disable default features when building RustPython", + ) parser.add_argument( "--features", action="store", @@ -195,6 +200,9 @@ def gen_methods(): typ = eval(typ_code) attrs = [] for attr in dir(typ): + # Skip attributes in dir() but not actually accessible (e.g., descriptor that raises) + if not hasattr(typ, attr): + continue if attr_is_not_inherited(typ, attr): attrs.append((attr, extra_info(getattr(typ, attr)))) methods[typ.__name__] = (typ_code, extra_info(typ), attrs) @@ -438,19 +446,23 @@ def remove_one_indent(s): f.write(output + "\n") -subprocess.run( - ["cargo", "build", "--release", f"--features={args.features}"], check=True -) +cargo_build_command = ["cargo", "build", "--release"] +if args.no_default_features: + cargo_build_command.append("--no-default-features") +if args.features: + cargo_build_command.extend(["--features", args.features]) + +subprocess.run(cargo_build_command, check=True) + +cargo_run_command = ["cargo", "run", "--release"] +if args.no_default_features: + cargo_run_command.append("--no-default-features") +if args.features: + cargo_run_command.extend(["--features", args.features]) +cargo_run_command.extend(["-q", "--", GENERATED_FILE]) + result = subprocess.run( - [ - "cargo", - "run", - "--release", - f"--features={args.features}", - "-q", - "--", - GENERATED_FILE, - ], + cargo_run_command, env={**os.environ.copy(), "RUSTPYTHONPATH": "Lib"}, text=True, capture_output=True, diff --git a/src/interpreter.rs b/src/interpreter.rs index 51667e724f1..b9ee2dbbc44 100644 --- a/src/interpreter.rs +++ b/src/interpreter.rs @@ -1,140 +1,54 @@ -use rustpython_vm::{Interpreter, PyRef, Settings, VirtualMachine, builtins::PyModule}; +use rustpython_vm::InterpreterBuilder; -pub type InitHook = Box<dyn FnOnce(&mut VirtualMachine)>; - -/// The convenient way to create [rustpython_vm::Interpreter] with stdlib and other stuffs. -/// -/// Basic usage: -/// ``` -/// let interpreter = rustpython::InterpreterConfig::new() -/// .init_stdlib() -/// .interpreter(); -/// ``` -/// -/// To override [rustpython_vm::Settings]: -/// ``` -/// use rustpython_vm::Settings; -/// // Override your settings here. -/// let mut settings = Settings::default(); -/// settings.debug = 1; -/// // You may want to add paths to `rustpython_vm::Settings::path_list` to allow import python libraries. -/// settings.path_list.push("Lib".to_owned()); // add standard library directory -/// settings.path_list.push("".to_owned()); // add current working directory -/// let interpreter = rustpython::InterpreterConfig::new() -/// .settings(settings) -/// .interpreter(); -/// ``` -/// -/// To add native modules: -/// ``` -/// use rustpython_vm::pymodule; -/// -/// #[pymodule] -/// mod your_module {} -/// -/// let interpreter = rustpython::InterpreterConfig::new() -/// .init_stdlib() -/// .add_native_module( -/// "your_module_name".to_owned(), -/// your_module::make_module, -/// ) -/// .interpreter(); -/// ``` -#[derive(Default)] -pub struct InterpreterConfig { - settings: Option<Settings>, - init_hooks: Vec<InitHook>, -} - -impl InterpreterConfig { - /// Creates a new interpreter configuration with default settings. - pub fn new() -> Self { - Self::default() - } - - /// Builds the interpreter with the current configuration. - pub fn interpreter(self) -> Interpreter { - let settings = self.settings.unwrap_or_default(); - Interpreter::with_init(settings, |vm| { - for hook in self.init_hooks { - hook(vm); - } - }) - } - - /// Sets custom settings for the interpreter. - /// - /// If called multiple times, only the last settings will be used. - pub fn settings(mut self, settings: Settings) -> Self { - self.settings = Some(settings); - self - } - - /// Adds a custom initialization hook. - /// - /// Hooks are executed in the order they are added during interpreter creation. - pub fn init_hook(mut self, hook: InitHook) -> Self { - self.init_hooks.push(hook); - self - } - - /// Adds a native module to the interpreter. - pub fn add_native_module( - self, - name: String, - make_module: fn(&VirtualMachine) -> PyRef<PyModule>, - ) -> Self { - self.init_hook(Box::new(move |vm| { - vm.add_native_module(name, Box::new(make_module)) - })) - } - - /// Initializes the Python standard library. +/// Extension trait for InterpreterBuilder to add rustpython-specific functionality. +pub trait InterpreterBuilderExt { + /// Initialize the Python standard library. /// /// Requires the `stdlib` feature to be enabled. #[cfg(feature = "stdlib")] - pub fn init_stdlib(self) -> Self { - self.init_hook(Box::new(init_stdlib)) - } + fn init_stdlib(self) -> Self; } -/// Initializes all standard library modules for the given VM. -#[cfg(feature = "stdlib")] -pub fn init_stdlib(vm: &mut VirtualMachine) { - vm.add_native_modules(rustpython_stdlib::get_module_inits()); +impl InterpreterBuilderExt for InterpreterBuilder { + #[cfg(feature = "stdlib")] + fn init_stdlib(self) -> Self { + let defs = rustpython_stdlib::stdlib_module_defs(&self.ctx); + let builder = self.add_native_modules(&defs); + + #[cfg(feature = "freeze-stdlib")] + let builder = builder + .add_frozen_modules(rustpython_pylib::FROZEN_STDLIB) + .init_hook(set_frozen_stdlib_dir); - #[cfg(feature = "freeze-stdlib")] - setup_frozen_stdlib(vm); + #[cfg(not(feature = "freeze-stdlib"))] + let builder = builder.init_hook(setup_dynamic_stdlib); - #[cfg(not(feature = "freeze-stdlib"))] - setup_dynamic_stdlib(vm); + builder + } } -/// Setup frozen standard library (compiled into the binary) +/// Set stdlib_dir for frozen standard library #[cfg(all(feature = "stdlib", feature = "freeze-stdlib"))] -fn setup_frozen_stdlib(vm: &mut VirtualMachine) { - vm.add_frozen(rustpython_pylib::FROZEN_STDLIB); +fn set_frozen_stdlib_dir(vm: &mut crate::VirtualMachine) { + use rustpython_vm::common::rc::PyRc; - // FIXME: Remove this hack once sys._stdlib_dir is properly implemented - // or _frozen_importlib doesn't depend on it anymore. - assert!(vm.sys_module.get_attr("_stdlib_dir", vm).is_err()); - vm.sys_module - .set_attr( - "_stdlib_dir", - vm.new_pyobj(rustpython_pylib::LIB_PATH.to_owned()), - vm, - ) - .unwrap(); + let state = PyRc::get_mut(&mut vm.state).unwrap(); + state.config.paths.stdlib_dir = Some(rustpython_pylib::LIB_PATH.to_owned()); } /// Setup dynamic standard library loading from filesystem #[cfg(all(feature = "stdlib", not(feature = "freeze-stdlib")))] -fn setup_dynamic_stdlib(vm: &mut VirtualMachine) { +fn setup_dynamic_stdlib(vm: &mut crate::VirtualMachine) { use rustpython_vm::common::rc::PyRc; let state = PyRc::get_mut(&mut vm.state).unwrap(); let paths = collect_stdlib_paths(); + // Set stdlib_dir to the first stdlib path if available + if let Some(first_path) = paths.first() { + state.config.paths.stdlib_dir = Some(first_path.clone()); + } + // Insert at the beginning so stdlib comes before user paths for path in paths.into_iter().rev() { state.config.paths.module_search_paths.insert(0, path); diff --git a/src/lib.rs b/src/lib.rs index c6de6520de8..b73725a0fe2 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -1,23 +1,29 @@ //! This is the `rustpython` binary. If you're looking to embed RustPython into your application, //! you're likely looking for the [`rustpython_vm`] crate. //! -//! You can install `rustpython` with `cargo install rustpython`, or if you'd like to inject your -//! own native modules you can make a binary crate that depends on the `rustpython` crate (and +//! You can install `rustpython` with `cargo install rustpython`. If you'd like to inject your +//! own native modules, you can make a binary crate that depends on the `rustpython` crate (and //! probably [`rustpython_vm`], too), and make a `main.rs` that looks like: //! //! ```no_run +//! use rustpython::{InterpreterBuilder, InterpreterBuilderExt}; //! use rustpython_vm::{pymodule, py_freeze}; -//! fn main() { -//! rustpython::run(|vm| { -//! vm.add_native_module("my_mod".to_owned(), Box::new(my_mod::make_module)); -//! vm.add_frozen(py_freeze!(source = "def foo(): pass", module_name = "other_thing")); -//! }); +//! +//! fn main() -> std::process::ExitCode { +//! let builder = InterpreterBuilder::new().init_stdlib(); +//! // Add a native module using builder.ctx +//! let my_mod_def = my_mod::module_def(&builder.ctx); +//! let builder = builder +//! .add_native_module(my_mod_def) +//! // Add a frozen module +//! .add_frozen_modules(py_freeze!(source = "def foo(): pass", module_name = "other_thing")); +//! +//! rustpython::run(builder) //! } //! //! #[pymodule] //! mod my_mod { //! use rustpython_vm::builtins::PyStrRef; -//TODO: use rustpython_vm::prelude::*; //! //! #[pyfunction] //! fn do_thing(x: i32) -> i32 { @@ -35,8 +41,9 @@ //! //! The binary will have all the standard arguments of a python interpreter (including a REPL!) but //! it will have your modules loaded into the vm. +//! +//! See [`rustpython_derive`](../rustpython_derive/index.html) crate for documentation on macros used in the example above. -#![cfg_attr(all(target_os = "wasi", target_env = "p2"), feature(wasip2))] #![allow(clippy::needless_doctest_main)] #[macro_use] @@ -54,8 +61,8 @@ use std::env; use std::io::IsTerminal; use std::process::ExitCode; -pub use interpreter::InterpreterConfig; -pub use rustpython_vm as vm; +pub use interpreter::InterpreterBuilderExt; +pub use rustpython_vm::{self as vm, Interpreter, InterpreterBuilder}; pub use settings::{InstallPipMode, RunMode, parse_opts}; pub use shell::run_shell; @@ -69,7 +76,11 @@ compile_error!( /// The main cli of the `rustpython` interpreter. This function will return `std::process::ExitCode` /// based on the return code of the python code ran through the cli. -pub fn run(init: impl FnOnce(&mut VirtualMachine) + 'static) -> ExitCode { +/// +/// **Note**: This function provides no way to further initialize the VM after the builder is applied. +/// All VM initialization (adding native modules, init hooks, etc.) must be done through the +/// [`InterpreterBuilder`] parameter before calling this function. +pub fn run(mut builder: InterpreterBuilder) -> ExitCode { env_logger::init(); // NOTE: This is not a WASI convention. But it will be convenient since POSIX shell always defines it. @@ -101,14 +112,9 @@ pub fn run(init: impl FnOnce(&mut VirtualMachine) + 'static) -> ExitCode { } } - let mut config = InterpreterConfig::new().settings(settings); - #[cfg(feature = "stdlib")] - { - config = config.init_stdlib(); - } - config = config.init_hook(Box::new(init)); + builder = builder.settings(settings); - let interp = config.interpreter(); + let interp = builder.interpreter(); let exitcode = interp.run(move |vm| run_rustpython(vm, run_mode)); rustpython_vm::common::os::exit_code(exitcode) @@ -208,6 +214,12 @@ fn run_rustpython(vm: &VirtualMachine, run_mode: RunMode) -> PyResult<()> { let scope = vm.new_scope_with_main()?; + // Initialize warnings module to process sys.warnoptions + // _PyWarnings_Init() + if vm.import("warnings", 0).is_err() { + warn!("Failed to import warnings module"); + } + // Import site first, before setting sys.path[0] // This matches CPython's behavior where site.removeduppaths() runs // before sys.path[0] is set, preventing '' from being converted to cwd @@ -276,11 +288,11 @@ fn run_rustpython(vm: &VirtualMachine, run_mode: RunMode) -> PyResult<()> { } RunMode::Repl => Ok(()), }; - if is_repl || vm.state.config.settings.inspect { - shell::run_shell(vm, scope)?; + let result = if is_repl || vm.state.config.settings.inspect { + shell::run_shell(vm, scope) } else { - res?; - } + res + }; #[cfg(feature = "flame-it")] { @@ -289,7 +301,8 @@ fn run_rustpython(vm: &VirtualMachine, run_mode: RunMode) -> PyResult<()> { error!("Error writing profile information: {}", e); } } - Ok(()) + + result } #[cfg(feature = "flame-it")] @@ -344,7 +357,7 @@ mod tests { use rustpython_vm::Interpreter; fn interpreter() -> Interpreter { - InterpreterConfig::new().init_stdlib().interpreter() + InterpreterBuilder::new().init_stdlib().interpreter() } #[test] diff --git a/src/main.rs b/src/main.rs index e88ea40f3df..3953b9dacfe 100644 --- a/src/main.rs +++ b/src/main.rs @@ -1,3 +1,10 @@ +use rustpython::{InterpreterBuilder, InterpreterBuilderExt}; + pub fn main() -> std::process::ExitCode { - rustpython::run(|_vm| {}) + let mut config = InterpreterBuilder::new(); + #[cfg(feature = "stdlib")] + { + config = config.init_stdlib(); + } + rustpython::run(config) } diff --git a/src/settings.rs b/src/settings.rs index 7dd3c1a7714..1847e22c2d4 100644 --- a/src/settings.rs +++ b/src/settings.rs @@ -285,6 +285,20 @@ pub fn parse_opts() -> Result<(Settings, RunMode), lexopt::Error> { } }; } + "thread_inherit_context" => { + settings.thread_inherit_context = match value { + Some("1") => true, + Some("0") => false, + _ => { + error!( + "Fatal Python error: config_init_thread_inherit_context: \ + -X thread_inherit_context=n: n is missing or invalid\n\ + Python runtime state: preinitialized" + ); + std::process::exit(1); + } + }; + } _ => {} } (name, value.map(str::to_owned)) @@ -297,6 +311,20 @@ pub fn parse_opts() -> Result<(Settings, RunMode), lexopt::Error> { if env_bool("PYTHONNODEBUGRANGES") { settings.code_debug_ranges = false; } + if let Some(val) = get_env("PYTHON_THREAD_INHERIT_CONTEXT") { + settings.thread_inherit_context = match val.to_str() { + Some("1") => true, + Some("0") => false, + _ => { + error!( + "Fatal Python error: config_init_thread_inherit_context: \ + PYTHON_THREAD_INHERIT_CONTEXT=N: N is missing or invalid\n\ + Python runtime state: preinitialized" + ); + std::process::exit(1); + } + }; + } // Parse PYTHONIOENCODING=encoding[:errors] if let Some(val) = get_env("PYTHONIOENCODING") @@ -379,8 +407,10 @@ pub(crate) use env::split_paths; pub(crate) fn split_paths<T: AsRef<std::ffi::OsStr> + ?Sized>( s: &T, ) -> impl Iterator<Item = std::path::PathBuf> + '_ { - use std::os::wasi::ffi::OsStrExt; - let s = s.as_ref().as_bytes(); - s.split(|b| *b == b':') - .map(|x| std::ffi::OsStr::from_bytes(x).to_owned().into()) + let s = s.as_ref().as_encoded_bytes(); + s.split(|b| *b == b':').map(|x| { + unsafe { std::ffi::OsStr::from_encoded_bytes_unchecked(x) } + .to_owned() + .into() + }) } diff --git a/wasm/demo/package-lock.json b/wasm/demo/package-lock.json index eb6957b3c54..699168925db 100644 --- a/wasm/demo/package-lock.json +++ b/wasm/demo/package-lock.json @@ -775,8 +775,7 @@ "version": "5.5.0", "resolved": "https://registry.npmjs.org/@xterm/xterm/-/xterm-5.5.0.tgz", "integrity": "sha512-hqJHYaQb5OptNunnyAnkHyM8aCjZ1MEIDTQu1iIbbTD/xops91NB5yq1ZK/dC2JDbVWtF23zUtl9JE2NqwT87A==", - "license": "MIT", - "peer": true + "license": "MIT" }, "node_modules/@xtuc/ieee754": { "version": "1.2.0", @@ -832,7 +831,6 @@ "integrity": "sha512-B/gBuNg5SiMTrPkC+A2+cW0RszwxYmn6VYxB/inlBStS5nx6xHIt/ehKRhIMhqusl7a8LjQoZnjCs5vhwxOQ1g==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "fast-deep-equal": "^3.1.3", "fast-uri": "^3.0.1", @@ -1151,7 +1149,6 @@ } ], "license": "MIT", - "peer": true, "dependencies": { "caniuse-lite": "^1.0.30001688", "electron-to-chromium": "^1.5.73", @@ -3241,9 +3238,9 @@ } }, "node_modules/lodash": { - "version": "4.17.21", - "resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz", - "integrity": "sha512-v2kDEe57lecTulaDIuNTPy3Ry4gLGJ6Z1O3vE1krgXZNrsQ+LFTGHVxVjcXPs17LhbZVGedAJv8XZ1tvj5FvSg==", + "version": "4.17.23", + "resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.23.tgz", + "integrity": "sha512-LgVTMpQtIopCi79SJeDiP0TfWi5CNEc/L/aRdTh3yIvmZXTnheWpKjSZhnvMl8iXbC1tFg9gdHHDMLoV7CnG+w==", "dev": true, "license": "MIT" }, @@ -3843,7 +3840,6 @@ } ], "license": "MIT", - "peer": true, "dependencies": { "nanoid": "^3.3.8", "picocolors": "^1.1.1", @@ -5102,8 +5098,7 @@ "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", "dev": true, - "license": "0BSD", - "peer": true + "license": "0BSD" }, "node_modules/type-fest": { "version": "2.19.0", @@ -5287,7 +5282,6 @@ "integrity": "sha512-UFynvx+gM44Gv9qFgj0acCQK2VE1CtdfwFdimkapco3hlPCJ/zeq73n2yVKimVbtm+TnApIugGhLJnkU6gjYXA==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "@types/eslint-scope": "^3.7.7", "@types/estree": "^1.0.6", @@ -5335,7 +5329,6 @@ "integrity": "sha512-MfwFQ6SfwinsUVi0rNJm7rHZ31GyTcpVE5pgVA3hwFRb7COD4TzjUUwhGWKfO50+xdc2MQPuEBBJoqIMGt3JDw==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "@discoveryjs/json-ext": "^0.6.1", "@webpack-cli/configtest": "^3.0.1",