29. Python Developer FAQ¶

29.2.1.1. Where can I learn about the version control system used, Mercurial (hg)?¶

Mercurial‘s (also known as hg) official web site is at https://www.mercurial-scm.org/. A book on Mercurial published by O’Reilly Media, Mercurial: The Definitive Guide, is available for free online. Another resource is Hg Init: a Mercurial tutorial by Joel Spolsky.

With Mercurial installed, you can run the help tool that comes with Mercurial to get help:

hg help

The man page for hg provides a quick refresher on the details of various commands, but doesn’t provide any guidance on overall workflow.

29.2.1.2. I already know how to use Git, can I use that instead?¶

While the main workflow for core developers requires Mercurial, if you just want to generate patches with git diff and post them to the issue tracker, there is a semi-offical read-only git mirror of the main CPython repository. To create a local clone based on this mirror rather than the main repository:

git clone git://github.com/python/cpython

The mirror’s master branch tracks the main repository’s default branch, while the maintenance branch names (2.7, 3.4, etc) are mapped directly.

Please only use this approach if you’re already an experienced Git user and don’t require assistance with the specifics of version control commands. All other parts of this developer’s guide assume the use of Mercurial for local version control.

29.2.1.3. What do I need to use Mercurial?¶

Host hg.python.org
  Compression yes

ssh = TortoisePlink.exe -ssh -2 -C -i C:\path\to\yourkey.ppk

29.2.1.4. What’s a working copy? What’s a repository?¶

Mercurial is a “distributed” version control system. This means that each participant, even casual contributors, download a complete copy (called a clone, since it is obtained by calling hg clone) of the central repository which can be treated as a stand-alone repository for all purposes. That copy is called in the FAQ the local repository, to differentiate with any remote repository you might also interact with.

But you don’t modify files directly in the local repository; Mercurial doesn’t allow for it. You modify files in what’s called the working copy associated with your local repository: you also run compilations and tests there. Once you are satisfied with your changes, you can commit them; committing records the changes as a new revision in the local repository.

Changes in your local repository don’t get automatically shared with the rest of the world. Mercurial ensures that you have to do so explicitly (this allows you to experiment quite freely with multiple branches of development, all on your private computer). The main commands for doing so are hg pull and hg push.

29.2.1.5. Which branches are in my local repository?¶

Typing hg branches displays the open branches in your local repository:

$ hg branches
default                    93085:030fda7b1de8
2.7                        93060:7ba47bbfe38d
3.5                        99283:4d5417444961 (inactive)
3.4                        93082:5fd481150b35 (inactive)
3.3                        93079:cda907a02a80 (inactive)
3.2                        92975:eac54f7a8018 (inactive)

29.2.1.6. Why are some branches marked “inactive”?¶

Assuming you get the following output:

$ hg branches
default                    93085:030fda7b1de8
3.5                        99283:4d5417444961 (inactive)

This means all changesets in the “3.5” branch have been merged into the “default” branch (or any other branch, if such exists).

29.2.1.7. Which branch is currently checked out in my working copy?¶

Use:

$ hg branch
default

Or to get more information:

$ hg summary
parent: 68026:f12ef116dd10 tip
 In FTP.close() method, make sure to also close the socket object, not only the file.
branch: default
commit: (clean)
update: (current)

29.2.1.8. How do I switch between branches inside my working copy?¶

Simply use hg update to checkout another branch in the current directory:

$ hg branch
default
$ hg update 3.6
86 files updated, 0 files merged, 11 files removed, 0 files unresolved
$ hg branch
3.6

Adding the -v option to hg update will list all updated files.

Note that, due to some previously built executables being used as a part of the build process, you may sometimes run into issues when attempting to switch between Python 2.x and Python 3.x branches. In these cases, it is best to run a make distclean to ensure that all previously built files are removed.

29.2.1.9. I want to keep a separate working copy per development branch, is it possible?¶

There are two ways:

1. Use the “share extension” as described in the Multiple Clones Approach section;
2. Create several clones of your local repository;

If you want to use the second way, you can do:

$ hg clone cpython py36
updating to branch default
3434 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd py35
$ hg update 3.6
86 files updated, 0 files merged, 11 files removed, 0 files unresolved

The current branch in a working copy is “sticky”: if you pull in some new changes, hg update will update to the head of the current branch.

29.2.1.10. How do I link my local repository to a particular remote repository?¶

Your local repository is linked by default to the remote repository it was cloned from. If you created it from scratch, however, it is not linked to any remote repository. In .hg/hgrc file for the local repository, add or modify the following section:

[paths]
default = ssh://hg@hg.python.org/devguide

This example is for a local repository that mirrors the devguide repository on hg.python.org. The same approach works for other remote repositories.

Anywhere that <remote repository> is used in the commands in this FAQ, hg will use the default remote repository if you omit the parameter.

29.2.1.11. How do I create a shorthand alias for a remote repository?¶

In your global .hgrc file add a section similar to the following:

[paths]
dg = ssh://hg@hg.python.org/devguide

This example creates a dg alias for the devguide repository on hg.python.org. This allows “dg” to be entered instead of the full URL for commands taking a repository argument (e.g. hg pull dg or hg outgoing dg).

Anywhere that <remote repository> is used in the commands in this FAQ, hg should accept an alias in place of a complete remote URL.

29.2.1.12. How do I compare my local repository to a remote repository?¶

To display the list of changes that are in your local repository, but not in the remote, use:

hg outgoing <remote repository>

This is the list of changes that will be sent if you call hg push <remote repository>. It does not include any uncommitted changes in your working copy!

Conversely, for the list of changes that are in the remote repository but not in the local, use:

hg incoming <remote repository>

This is the list of changes that will be retrieved if you call hg pull <remote repository>.

29.2.1.13. How do I update my local repository to be in sync with a remote repository?¶

Run:

hg pull <remote repository>

from the repository you wish to pull the latest changes into. Most of the time, that repository is a clone of the repository you want to pull from, so you can simply type:

hg pull

This doesn’t update your working copy, though. See below:

29.2.1.14. How do I update my working copy with the latest changes?¶

Do:

hg update

This will update your working copy with the latest changes on the current branch. If you had uncommitted changes in your working copy, they will be merged in.

If you find yourself typing often hg pull followed by hg update, be aware that you can combine them in a single command:

hg pull -u

29.2.1.15. How do I apply a patch?¶

If you want to try out or review a patch generated using Mercurial, do:

patch -p1 < somework.patch

This will apply the changes in your working copy without committing them. If the patch was not created by Mercurial (for example, a patch created by Subversion and thus lacking any a/b directory prefixes in the patch), replace -p1 with -p0.

If the patch contains renames, deletions or copies, and you intend committing it after your review, you might prefer using:

hg import --no-commit somework.patch

If you want to work on the patch using mq (Mercurial Queues), type instead:

hg qimport somework.patch

This will create a patch in your queue with a name that matches the filename. You can use the -n argument to specify a different name. To have the patch applied to the working copy, type:

hg qpush

Finally, to delete the patch, first un-apply it if necessary using hg qpop, then do:

hg qdelete somework.patch

29.2.1.16. How do I solve conflicts when applying a patch fails?¶

The standard patch command, as well as hg import, will produce unhelpful *.rej files when it fails applying parts of a patch. We suggest you try the mpatch utility, which can help resolve a number of common causes of patch rejects.

To make use of mpatch transparent, you can define a shell alias in one of your startup files. For example, if you want it to open the kdiff3 merge program to fix failing patch hunks:

alias patch='mpatch --merge=kdiff3'

or if you want it to automatically solve conflicts by using heuristics:

alias patch='mpatch --auto --no-merge'

29.2.1.17. How do I add a file or directory to the repository?¶

Simply specify the path to the file or directory to add and run:

hg add PATH

If PATH is a directory, Mercurial will recursively add any files in that directory and its descendants.

If you want Mercurial to figure out by itself which files should be added and/or removed, just run:

hg addremove

Be careful though, as it might add some files that are not desired in the repository (such as build products, cache files, or other data).

You will then need to run hg commit (as discussed below) to commit the file(s) to your local repository.

29.2.1.18. What’s the best way to split a file into several files?¶

To split a file into several files (e.g. a module converted to a package or a long doc file divided in two separate documents) use hg copy:

hg copy module.rst module2.rst

and then remove the parts that are not necessary from module.rst and module2.rst. This allows Mercurial to know that the content of module2.rst used to be in module.rst, and will make subsequent merges easier. If necessary, you can also use hg copy several times.

If you simply create module2.rst, add it with hg add, and copy part of the content from module.rst, Mercurial won’t know that the two file are related.

29.2.1.19. How do I delete a file or directory in the repository?¶

Specify the path to be removed with:

hg remove PATH

This will remove the file or the directory from your working copy; you will have to commit your changes for the removal to be recorded in your local repository.

29.2.1.20. What files are modified in my working copy?¶

Running:

hg status

will list any pending changes in the working copy. These changes will get committed to the local repository if you issue an hg commit without specifying any path.

Some key indicators that can appear in the first column of output are:

A	Scheduled to be added
R	Scheduled to be removed
M	Modified locally
?	Not under version control

If you want a line-by-line listing of the differences, use:

hg diff

29.2.1.21. How do I revert a file I have modified back to the version in the repository?¶

Running:

hg revert PATH

will revert PATH to its version in the repository, throwing away any changes you made locally. If you run:

hg revert -a

from the root of your working copy it will recursively restore everything to match up with the repository.

29.2.1.22. How do I find out who edited or what revision changed a line last?¶

You want:

hg annotate PATH

This will output to stdout every line of the file along with which revision last modified that line. When you have the revision number, it is then easy to display it in detail.

29.2.1.23. How can I see a list of log messages for a file or specific revision?¶

To see the history of changes for a specific file, run:

hg log -v [PATH]

That will list all messages of revisions which modified the file specified in PATH. If PATH is omitted, all revisions are listed.

If you want to display line-by-line differences for each revision as well, add the -p option:

hg log -vp [PATH]

If you want to view the differences for a specific revision, run:

hg log -vp -r <revision number>

29.2.1.24. How can I see the changeset graph in my repository?¶

In Mercurial repositories, changesets don’t form a simple list, but rather a graph: every changeset has one or two parents (it’s called a merge changeset in the latter case), and can have any number of children.

The graphlog extension is very useful for examining the structure of the changeset graph. It is bundled with Mercurial.

Graphical tools, such as TortoiseHG, will display the changeset graph by default.

29.2.1.25. How do I update to a specific release tag?¶

Run:

hg tags

to get a list of tags. To update your working copy to a specific tag, use:

hg update <tag>

29.2.1.26. How do I find which changeset introduced a bug or regression?¶

hg bisect, as the name indicates, helps you do a bisection of a range of changesets.

You need two changesets to start the search: one that is “good” (doesn’t have the bug), and one that is “bad” (has the bug). Usually, you have just noticed the bug in your working copy, so you can start with:

hg bisect --bad

Then you must update to a previous changeset that doesn’t have the bug. You can conveniently choose a faraway changeset (for example a former release), and check that it is indeed “good”. Then type:

hg bisect --good

Mercurial will automatically bisect so as to narrow the range of possible culprits, until a single changeset is isolated. Each time Mercurial presents you with a new changeset, re-compile Python and run the offending test, for example:

make -j2
./python -m test -uall test_sometest

Then, type either hg bisect --good or hg bisect --bad depending on whether the test succeeded or failed.

29.2.1.27. How come feature XYZ isn’t available in Mercurial?¶

Mercurial comes with many bundled extensions which can be explicitly enabled. You can get a list of them by typing hg help extensions. Some of these extensions, such as color, can prettify output; others, such as fetch or graphlog, add new Mercurial commands.

There are also many configuration options to tweak various aspects of the command line and other Mercurial behaviour; typing man hgrc displays their documentation inside your terminal.

In the end, please refer to the Mercurial wiki, especially the pages about extensions (including third-party ones) and the tips and tricks.

29.2.2.1. How do I commit a change to a file?¶

To commit any changes to a file (which includes adding a new file or deleting an existing one), you use the command:

hg commit [PATH]

PATH is optional: if it is omitted, all changes in your working copy will be committed to the local repository. When you commit, be sure that all changes are desired by reviewing them first; also, when making commits that you intend to push to public repositories, you should not commit together unrelated changes.

To abort a commit that you are in the middle of, leave the message empty (i.e., close the text editor without adding any text for the message). Mercurial will then abort the commit operation so that you can try again later.

Once a change is committed to your local repository, it is still only visible by you. This means you are free to experiment with as many local commits you feel like.

29.2.2.2. How do I solve merge conflicts?¶

The easiest way is to install KDiff3 — Mercurial will open it automatically in case of conflicts, and you can then use it to solve the conflicts and save the resulting file(s). KDiff3 will also take care of marking the conflicts as resolved.

If you don’t use a merge tool, you can use hg resolve --list to list the conflicting files, resolve the conflicts manually, and the use hg resolve --mark <file path> to mark these conflicts as resolved. You can also use hg resolve -am to mark all the conflicts as resolved.

29.2.2.3. How do I make a null merge?¶

If you committed something (e.g. on 3.6) that shouldn’t be ported on newer branches (e.g. on default), you have to do a null merge:

cd 3.x
hg merge 3.6
hg revert -ar default
hg resolve -am  # needed only if the merge created conflicts
hg ci -m '#12345: null merge with 3.6.'

Before committing, hg status should list all the merged files as M, but hg diff should produce no output. This will record the merge without actually changing the content of the files.

29.2.2.4. I got “abort: push creates new remote heads!” while pushing, what do I do?¶

If you see this message while pushing, it means that you committed something on a clone that was not up to date, thus creating a new head. This usually happens for two reasons:

1. You forgot to run hg pull and/or hg up before committing;
2. Someone else pushed on the main repo just before you, causing a push race;

First of all you should pull the new changesets using hg pull. Then you can use hg heads to see which branches have multiple heads.

If only one branch has multiple heads, you can do:

cd default
hg heads .
hg up csid-of-the-other-head
hg merge
hg ci -m 'Merge heads.'

hg heads . will show you the two heads of the current branch: the one you pulled and the one you created with your commit (you can also specify a branch with hg heads <branch>). While not strictly necessary, it is highly recommended to switch to the other head before merging. This way you will be merging only your changeset with the rest, and in case of conflicts it will be a lot easier.

If more than one branch has multiple heads, you have to repeat these steps for each branch. Since this creates new changesets, you will also have to merge them between branches. For example, if both 3.6 and default have multiple heads, you should first merge heads in 3.6, then merge heads in default, and finally merge 3.6 with default using hg merge 3.6 as usual.

In order to avoid this, you should always remember to pull and update before committing.

29.2.2.5. How do I undo the changes made in a recent commit?¶

First, this should not happen if you take the habit of reviewing changes before committing them.

In any case, run:

hg backout <revision number>

This will modify your working copy so that all changes in <revision number> (including added or deleted files) are undone. You then need to commit these changes so that the backout gets permanently recorded.

29.3.1.1. UNIX¶

Run:

ssh-keygen -t ed25519

This will generate two files; your public key and your private key. The public key is in file ending in .pub.

29.3.1.2. Windows¶

Use PuTTYgen to generate your public key. Choose the “SSH-2 RSA” radio button, set 4096 as the key size, choose a password, and save the private key to a file. Copy the section with the public key (using Alt-P) to a file; that file now has your public key.

29.3.2.1. UNIX¶

Use ssh-agent and ssh-add to register your private key with SSH for your current session. The simplest solution, though, is to use KeyChain, which is a shell script that will handle ssh-agent and ssh-add for you once per login instead of per session.

29.3.2.2. Windows¶

The Pageant program is bundled with TortoiseHg. You can find it in its installation directory (usually C:\Program Files (x86)\TortoiseHg\); you can also download it separately.

Running Pageant will prevent you from having to type your password constantly. If you add a shortcut to Pageant to your Autostart group and edit the shortcut so that the command line includes an argument to your private key then Pageant will load the key every time you log in.