The Wayback Machine - https://web.archive.org/web/20201019100052/https://github.com/python/typing/pull/443
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add typing extensions #443

Merged
merged 23 commits into from Jul 20, 2017
Merged

Conversation

@Michael0x2a
Copy link
Contributor

@Michael0x2a Michael0x2a commented Jun 27, 2017

This pull request adds a typing_extensions subproject to typing. The typing_extensions module backports any new additions to typing for Python 3.5+ users who are using older versions of typing that were bundled with their standard library (and so can't update to the latest versions).

See #435 for more motivation and additional context.

@Michael0x2a Michael0x2a changed the title Add typing extensions Add typing extensions [WIP] Jun 27, 2017
@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Jun 27, 2017

I don't think you need all these old versions bundled in tests. I think we should just properly configure Travis (it supports Python micro releases like 3.5.2 etc. (like in mypy's .travis.yml), and specific versions of typing to be installed before tests)

@Michael0x2a
Copy link
Contributor Author

@Michael0x2a Michael0x2a commented Jun 27, 2017

@ilevkivskyi I guess my only concern with that is that it becomes harder to do sanity checks locally -- having to constantly push to run tests seems a bit annoying to me.

@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Jun 27, 2017

having to constantly push to run tests seems a bit annoying to me.

You can still keep these copies locally (without tracking them in git). But I would prefer a properly configured .travis.yml to test with actual Python versions (note that Python runtime changes itself between versions, not only the stdlib, there are few places in typing where this is actually important).

@Michael0x2a Michael0x2a force-pushed the Michael0x2a:add-typing-extensions branch from 124768e to 6928977 Jun 27, 2017
T_contra = typing.TypeVar('T_contra', contravariant=True) # Ditto contravariant.


def _gorg(a):

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Jun 27, 2017
Collaborator

Note that _gorg and _geqv will be removed soon after Python 3.6.2 is released, see PR #439.

This comment has been minimized.

@Michael0x2a

Michael0x2a Jun 27, 2017
Author Contributor

Sounds good -- once this pull request gets accepted, I can prepare another pull request to add those changes and leave it pending just like #439 (or whatever other course of action you think is best).

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Jul 7, 2017
Collaborator

or whatever other course of action you think is best

OK, let's just not forget about this.

@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Jun 27, 2017

Concerning the test failure, we decided to stop supporting Python 3.2 (it reached EoL a year ago), so that you can just skip testing on Python 3.2.

@Michael0x2a
Copy link
Contributor Author

@Michael0x2a Michael0x2a commented Jun 27, 2017

Update: I removed run_tests.py and the associated machinery in favor of just using Travis, as suggested. The only remaining blocker is that we can't actually install typing on Python 3.2. I figured it'd be best to make a separate pull request for that though: #444

Once that's accepted, I can rebase, but this pull request should otherwise be ready for review (?).

Edit: lol, sniped

@Michael0x2a Michael0x2a changed the title Add typing extensions [WIP] Add typing extensions Jun 27, 2017
@Michael0x2a Michael0x2a force-pushed the Michael0x2a:add-typing-extensions branch from 6928977 to 15cbbfc Jun 27, 2017
@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Jun 27, 2017

I can review this, but I am not sure I will be able to review before week-end.

@ilevkivskyi ilevkivskyi self-requested a review Jun 27, 2017
Copy link
Collaborator

@ilevkivskyi ilevkivskyi left a comment

Sorry for waiting with this, the PR looks very good. Thanks for work! I just have mostly minor comments.

In general I am eager to merge this soon, so that we can put stuff from mypy_extensions here as well. And Protocol+@runtime :-)

author='Guido van Rossum, Jukka Lehtosalo, Lukasz Langa, Michael Lee',
author_email='jukka.lehtosalo@iki.fi',
# TODO: Change URL
url='https://github.com/michael0x2a/typing_extensions',

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Jul 7, 2017
Collaborator

The link should be to the main typing repo.

This comment has been minimized.

@Michael0x2a

Michael0x2a Jul 7, 2017
Author Contributor

Whoops, forgot to change that.

'''

classifiers = [
'Development Status :: 5 - Production/Stable',

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Jul 7, 2017
Collaborator

This is far from stable :-) I would put alpha here.

This comment has been minimized.

@Michael0x2a

Michael0x2a Jul 7, 2017
Author Contributor

Done!

tox.ini Outdated
src/test_typing.py,
typing_extensions/src_py2/test_typing_extensions.py,
typing_extensions/src_py3/test_typing_extensions.py,
# test_data should contain an exact snapshot of the standard

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Jul 7, 2017
Collaborator

Do we still need this? I thought we agreed that we could leave this out and just use updated Travis config.

This comment has been minimized.

@Michael0x2a

Michael0x2a Jul 7, 2017
Author Contributor

Same thing -- forgot to remove it.

.travis.yml Outdated
@@ -14,5 +18,8 @@ install:

script:
- export PYTHONPATH=`python -c "import sys; print('python2' if sys.version.startswith('2') else 'src')"`; py.test $PYTHONPATH
- python setup.py install;

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Jul 7, 2017
Collaborator

I am just curious, why this was not necessary before and is necessary now?

This comment has been minimized.

@Michael0x2a

Michael0x2a Jul 7, 2017
Author Contributor

The problem is that the typing_extensions depends on typing being installed for Python 3.4 and below since we try and re-export as much as possible from the typing module. That said, I've tried adjusting travis so it should install typing only for Python 3.4 and below -- we'll see if I did it correctly once the tests are done.

To run tests, navigate into the appropriate source directory and run
`test_typing_extensions.py`. You will also need to install the latest
version of `typing` if you are using a version of Python that does not
include `typing` as a part of the standard library.

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Jul 7, 2017
Collaborator

Maybe we could add a requirements.txt or test_requirements.txt for this?

This comment has been minimized.

@Michael0x2a

Michael0x2a Jul 7, 2017
Author Contributor

I added it to the test_requirements.txt file for the typing repo. If you prefer, I can leave that one alone and add a separate test_requirements.txt and requirements.txt file to the typing_extensions folder.

Since typing_extensions is basically a separate project, maybe it makes sense for it to have its own requirement files, but it does feel a little redundant...

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Jul 18, 2017
Collaborator

I would actually prefer separate requirements for typing_extensions.



if __name__ == '__main__':
# main(argv=[sys.argv[0]] + sys.argv[2:])

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Jul 7, 2017
Collaborator

Is this comment necessary? Why do we need this pattern?

This comment has been minimized.

@Michael0x2a

Michael0x2a Jul 7, 2017
Author Contributor

This was a holdover from the code I previously had for running tests, which we removed. I removed the comment here + in the Python 2 tests.

try:
import collections.abc as collections_abc
except ImportError:
import collections as collections_abc # Fallback for PY3.2.

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Jul 7, 2017
Collaborator

Again, this is not needed.

This comment has been minimized.

@Michael0x2a

Michael0x2a Jul 7, 2017
Author Contributor

Also removed.

_generic_new = typing._generic_new
else:
def _generic_new(base_cls, cls, *args, **kwargs):
return base_cls.__new__(cls, *args, **kwargs)

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Jul 7, 2017
Collaborator

I am not 100% sure this is a safe substitution. Do you have a test for this? IIRC, the point was that __init__ was not called at instantiation of subscripted generics.

This comment has been minimized.

@Michael0x2a

Michael0x2a Jul 7, 2017
Author Contributor

I don't know if I necessarily have tests for specifically this -- I was mostly copying what the typing module was already doing and assumed the behavior was correct. For example, in Python 3.5.0 - 3.5.2, the List type was defined like this:

class List(list, MutableSequence[T], extra=list):
    def __new__(cls, *args, **kwds):
        if _geqv(cls, List):
            raise TypeError("Type List cannot be instantiated; "
                            "use list() instead")
        return list.__new__(cls, *args, **kwds)

In Python 3.5.3+, it was defined like this:

class List(list, MutableSequence[T], extra=list):
    __slots__ = ()

    def __new__(cls, *args, **kwds):
        if _geqv(cls, List):
            raise TypeError("Type List cannot be instantiated; "
                            "use list() instead")
        return _generic_new(list, cls, *args, **kwds)

I'm also not sure if this substitution is 100% correct, but if it's broken, it'll at least be consistently broken in the same way the rest of the typing module was in Python 3.5.0 - 3.5.2. Defining the function this way also helped me reduce redundancy since I can define things like Deque just once instead of twice.

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Jul 18, 2017
Collaborator

be consistently broken in the same way the rest of the typing module was in Python 3.5.0 - 3.5.2

OK, but then please mention this in a docstring/comment.

'NewType',
'overload',
'Text',
'TYPE_CHECKING',

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Jul 7, 2017
Collaborator

Again, I am not sure about GenericMeta (see comment above).

T_contra = typing.TypeVar('T_contra', contravariant=True) # Ditto contravariant.


def _gorg(a):

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Jul 7, 2017
Collaborator

or whatever other course of action you think is best

OK, let's just not forget about this.

@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Jul 7, 2017

Actually, I have one more suggestion. Maybe edit description to reflect the fact that experimental things that are going to appear in typing later will appear here first (like TypedDict, argspecs, and Protocol).

@Michael0x2a
Copy link
Contributor Author

@Michael0x2a Michael0x2a commented Jul 7, 2017

@ilevkivskyi Thanks for the review! I'll start going through your comments and making fixes in a moment.

@gvanrossum
Copy link
Member

@gvanrossum gvanrossum commented Jul 7, 2017

TYPING_V3 = sys.version_info >= (3, 5, 3)
TYPING_V4 = sys.version_info >= (3, 6, 1)
TYPING_LATEST = sys.version_info[:3] < (3, 5, 0)
TYPING_V2 = TYPING_LATEST or sys.version_info[:3] >= (3, 5, 1)

This comment has been minimized.

@JelleZijlstra

JelleZijlstra Jul 7, 2017
Contributor

I think this would be clearer as TYPING_3_5_1 or similar, because "v2" is not a terminology used elsewhere.

This comment has been minimized.

@Michael0x2a

Michael0x2a Jul 7, 2017
Author Contributor

Ok, done.

@Michael0x2a
Copy link
Contributor Author

@Michael0x2a Michael0x2a commented Jul 7, 2017

@ilevkivskyi -- Ok, I should be ready for a second round of review. I tried adding links to relevant pull requests and issues for the typing version-specific if statements, but found it quite hard to do so in general, so unfortunately most of them are still undocumented :(.

Michael0x2a added a commit to Michael0x2a/typeshed that referenced this pull request Jul 8, 2017
See latest comment in python/typing#443
for justification.
@Michael0x2a
Copy link
Contributor Author

@Michael0x2a Michael0x2a commented Jul 8, 2017

Just as an FYI, I decided to remove typing_extensions.Collection -- in retrospect, I think attempting to backport that type was a mistake.

In order to correctly backport that type, I basically need to inject typing_extensions.Collection into the middle of an existing inheritance hierarchy, which I don't think is possible to do cleanly, if at all. More concretely, the following code doesn't typecheck in Python 2 and older versions of Python 3 (the typing stubs have typing.Sequence subclass typing.Collection only some of the time):

from mypy_extensions import Collection

def foo(x):
    # type: Collection[int] -> int
    return sum(x)

foo([1, 2, 3])

I think this is potentially repairable at least at typechecking-time with some very careful surgery on the typing stubs, but doing that seems questionable to me. It also wouldn't fix any runtime isinstance or issubclass checks people might try doing.

(More specifically, Python 3's typing stubs has typing.Sequence always subclass a phantom _Collection type, which is only publicly exposed as typing.Collection for Python 3.6+. We could add a similar phantom type to the Python 2 typing stubs, then have the typing_extension re-export typing._Collection as typing_extensions.Collection. Users would then need to use an up-to-date version of typeshed.)

Michael0x2a added 8 commits Jun 27, 2017
This pull request adds a 'typing_exensions' subproject to 'typing'. The
'typing_extensions' module backports any new additions to 'typing'
for Python 3.5+ users who are using older versions of 'typing' that were
bundled with their standard library (and so can't update to the latest
versions).

See #435 for motivation and
additional context.
Michael0x2a added 14 commits Jun 27, 2017
In retrospect, attempting to backport typing.Collection was a mistake.
It's difficult in practice to use it in a consistently meaningful way
because the typing module deliberately does not have types like Sequence
or List subclass anything resembling the Collections type for older
versions of Python.

This means that the following code will not typecheck on Python 2
and on older versions of Python 3:

    from mypy_extensions import Collection

    def foo(x):
        # type: Collection[int] -> int
        return sum(x)

    foo([1, 2, 3])

...because `List` is not defined to be a subclass of `Collection` in
Python 2 and Python 3.x - 3.5 at runtime and within the typing stubs
in typeshed.

I think this is potentially repairable with some careful surgery on
the typing stubs, but doing that seems highly questionable to me,
so I'm thinking attempting to backport `Collection` is out-of-scope
for this pull request and should be handled later (or potentially
never).
Incorporates change made by #439,
which was recently merged.
@Michael0x2a Michael0x2a force-pushed the Michael0x2a:add-typing-extensions branch from eb24e8f to 11c0fcf Jul 17, 2017
@Michael0x2a
Copy link
Contributor Author

@Michael0x2a Michael0x2a commented Jul 17, 2017

@ilevkivskyi -- as an update, I incorporated the changes you just pushed to master earlier today (removing _gorg and _geqv and making other speed improvements).

@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Jul 17, 2017

@Michael0x2a thanks for continuing to work on this! I will try to make the second review today or tomorrow.

@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Jul 18, 2017

@Michael0x2a
I responded to your previous comments, continuing with review now.

@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Jul 18, 2017

Concerning Collection , I would still add it. Maybe not in this PR, but after you make an initial PR to typeshed with typing_extensions stubs.

Copy link
Collaborator

@ilevkivskyi ilevkivskyi left a comment

OK, just three more minor comments.


The ``typing_extensions`` module contains both backports of changes made to
the ``typing`` module since Python 3.5.0 as well as experimental types that
will be eventually added to the ``typing`` module once stabilized.

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Jul 18, 2017
Collaborator

I think you forgot to mention that it is provisional until Python 3.7.

self.assertIsSubclass(collections.Counter, Counter)
self.assertIs(type(Counter()), collections.Counter)
self.assertIs(type(Counter[T]()), collections.Counter)
self.assertIs(type(Counter[int]()), collections.Counter)

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Jul 18, 2017
Collaborator

I actually wanted to also test that Counter.__bases__ are already erased (before instantiation).

import typing_extensions
import collections.abc as collections_abc

TYPING_LATEST = sys.version_info[:3] < (3, 5, 0)

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Jul 18, 2017
Collaborator

I am not sure I follow the logic here, shouldn't this be TYPING_OLDEST? Or this is the version of typing you "emulate"? Could you please add a comment here?

- Reworded README/description in setup.py to include note about
  typing's provisional status ending in Python 3.7.
- Made the README/description in setup.py a little shorter in general
- Added some explanatory comments
- Added a test to check Counter.__bases__ is erased in Python 2.7
@Michael0x2a
Copy link
Contributor Author

@Michael0x2a Michael0x2a commented Jul 20, 2017

@ilevkivskyi -- ok, I've addressed your new comments and am ready for the 3rd round of review!

Regarding your comment regarding Counter.__bases__: I think I added a test for that in the Python 3 tests in response to one of your comments from the first round of review, but forgot to add the same test for the Python 2 tests. I copied the same check over.

That said, I want to make sure I actually understand both comments correctly -- are the tests I added what you were looking for, or should I need to add in more checks there?

Copy link
Collaborator

@ilevkivskyi ilevkivskyi left a comment

Maybe there are some small things left but it will be easier to fix them in subsequent smaller PRs, so that I am going to merge this soon. I think the next step is to sumbit typing_extensions stubs to typeshed and probably to "teach" mypy to interact with them (I could imagine some typing types are special-cased in mypy).

@Michael0x2a
Copy link
Contributor Author

@Michael0x2a Michael0x2a commented Jul 20, 2017

@ilevkivskyi stubs for typeshed can be found here: python/typeshed#1471

I'll work on making changes to mypy itself next. iirc it contains some special-casing and tests specifically for mypy_extensions so I'll probably start by duplicating a subset of those for typing_extensions.

@ilevkivskyi ilevkivskyi merged commit 59c0987 into python:master Jul 20, 2017
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Michael0x2a added a commit to Michael0x2a/typeshed that referenced this pull request Jul 20, 2017
See latest comment in python/typing#443
for justification.
Michael0x2a added a commit to Michael0x2a/typeshed that referenced this pull request Jul 21, 2017
See latest comment in python/typing#443
for justification.
@Michael0x2a Michael0x2a deleted the Michael0x2a:add-typing-extensions branch Nov 17, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.
Morty Proxy This is a proxified and sanitized view of the page, visit original site.