Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

gh-117587: Add C implementation of os.path.abspath #117855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 87 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
2475af9
Speedup `posixpath.abspath()` for relative paths
nineteendo Apr 12, 2024
807b494
Fix cwd length for byte paths & bus error
nineteendo Apr 12, 2024
aa3ed0c
Merge branch 'python:main' into speedup-posixpath.abspath
nineteendo Apr 12, 2024
fa50a4c
📜🤖 Added by blurb_it.
blurb-it[bot] Apr 13, 2024
f502dd1
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 13, 2024
716f594
Fix `_Py_normpath_and_size` when `size=-1`
nineteendo Apr 13, 2024
cc3a59c
Update Modules/posixmodule.c
nineteendo Apr 13, 2024
f5f2c7b
Update Python/fileutils.c
nineteendo Apr 13, 2024
a1bc125
fastcall & unicode type
nineteendo Apr 13, 2024
ae2c040
Revert changes to `os__path_normpath_impl`
nineteendo Apr 14, 2024
95773d9
simply `os.fsdecode(os.getcwdb())`
nineteendo Apr 14, 2024
b77a15d
1st attempt at full C implementation
nineteendo Apr 14, 2024
f39ab83
Merge branch 'speedup-posixpath.abspath' of https://github.com/ninete…
nineteendo Apr 14, 2024
c63518b
Update Modules/posixmodule.c
nineteendo Apr 14, 2024
94b12c2
2nd attempt at full C implementation
nineteendo Apr 14, 2024
c6c62d2
Merge branch 'speedup-posixpath.abspath' of https://github.com/ninete…
nineteendo Apr 14, 2024
51c9541
3rd attempt at full C implementation
nineteendo Apr 14, 2024
ea0b4d1
Fix tests
nineteendo Apr 14, 2024
8d435d6
Fix tests attempt 2
nineteendo Apr 14, 2024
7057f3b
special case for '' & '.'
nineteendo Apr 15, 2024
fff561e
follow pep 7
nineteendo Apr 15, 2024
15c8f0d
check type with argument clinic
nineteendo Apr 15, 2024
34858a8
Fix memory leak & variable names
nineteendo Apr 16, 2024
21cd24a
Use `PyMem_RawFree`
nineteendo Apr 16, 2024
1113ec3
Update Modules/posixmodule.c
nineteendo Apr 16, 2024
7cca80e
Update Modules/posixmodule.c
nineteendo Apr 16, 2024
0f1f28b
Merge branch 'main' into speedup-posixpath.abspath
eryksun Apr 16, 2024
1313ceb
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 16, 2024
7b6727c
Update Modules/posixmodule.c
nineteendo Apr 17, 2024
97516f6
narrow scoping
nineteendo Apr 17, 2024
7e87ccd
Free `cwd_buf` for all exits
nineteendo Apr 17, 2024
60c99be
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 17, 2024
c06d34f
rename `start` to `prefix_len`
nineteendo Apr 17, 2024
ba14b7d
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 18, 2024
3d2836e
test different cache key
nineteendo Apr 21, 2024
9fcedb5
Merge branch 'main' into speedup-posixpath.abspath
hugovk Apr 23, 2024
111df43
Add C implementation of `ntpath.abspath()`
nineteendo Apr 23, 2024
a538218
Replace with `nt._path_abspath`
nineteendo Apr 23, 2024
d36379d
Update Lib/posixpath.py
nineteendo Apr 23, 2024
6e645f1
Remove fallback
nineteendo Apr 24, 2024
8fd2984
Update Modules/posixmodule.c
nineteendo Apr 24, 2024
4d8eb1d
Follow pep 7
nineteendo Apr 24, 2024
c470123
Check for embedded null characters
nineteendo Apr 25, 2024
3a9af3f
Stricter check for embedded null characters
nineteendo Apr 25, 2024
329692b
Merge branch 'main' into speedup-os.path
nineteendo Apr 25, 2024
9fbc62e
Support qualified referencing
nineteendo Apr 25, 2024
d83a580
Add additional test
nineteendo Apr 25, 2024
08b9674
null in cwd?
nineteendo Apr 25, 2024
6eb1442
Add Windows 11 check
nineteendo Apr 25, 2024
b7f9df6
Add additional test
nineteendo Apr 25, 2024
b127f05
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 26, 2024
3ad0d7f
Fix drive relative paths
nineteendo Apr 26, 2024
549fa4e
Handle all qualified referencing
nineteendo Apr 27, 2024
a768039
Fix ambiguous operation
nineteendo Apr 27, 2024
3339c19
Special case for '.'
nineteendo Apr 27, 2024
0a40599
Handle drive relative paths
nineteendo Apr 28, 2024
2315bbb
Add tests
nineteendo Apr 28, 2024
bbbba83
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 28, 2024
6bac80e
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 30, 2024
b3d04ac
Direct C call for `nt.isdevdrive()`
nineteendo May 4, 2024
415f831
Merge branch 'main' into speedup-posixpath.abspath
nineteendo May 4, 2024
9c55537
Revert "Direct C call for `nt.isdevdrive()`"
nineteendo May 4, 2024
2bf2cd3
Revert "Revert "Direct C call for `nt.isdevdrive()`""
nineteendo May 4, 2024
824f0d2
Restore lost code
nineteendo May 4, 2024
5850c3b
Zero-valued flag enum has no name
nineteendo May 9, 2024
eb26e22
Revert "Zero-valued flag enum has no name"
nineteendo May 9, 2024
bfb235c
Merge branch 'main' into speedup-posixpath.abspath
nineteendo May 27, 2024
d89e875
Direct C call for `posixpath.abspath()`
nineteendo May 28, 2024
66c057f
Optimisation
nineteendo May 28, 2024
8c1e11c
Use root directory for other drives
nineteendo May 28, 2024
50f5882
Fix drive relative paths
nineteendo May 28, 2024
74d7431
Simplify implementation
nineteendo May 28, 2024
807f83f
Update Lib/ntpath.py
nineteendo May 28, 2024
b19bbeb
Improve readability
nineteendo May 28, 2024
f934977
Check embedded null manually
nineteendo May 28, 2024
cf01b90
Revert fallback code on POSIX
nineteendo May 29, 2024
29f9cfc
Update Lib/ntpath.py
nineteendo May 29, 2024
374a39e
Update Lib/test/test_ntpath.py
nineteendo May 29, 2024
adc708d
Apply suggestions from code review
nineteendo May 29, 2024
4d0289a
Narrow scope
nineteendo May 29, 2024
fcffc53
Improve comments
nineteendo May 29, 2024
1ebe5f4
Just check if path is absolute
nineteendo May 29, 2024
e047f03
Narrow scope of `cwd_buf`
nineteendo May 30, 2024
625fc79
Update Python/fileutils.c
nineteendo May 30, 2024
fc3f227
Free memory before calling the `PyErr` APIs
nineteendo May 30, 2024
54c7021
Apply suggestions from code review
nineteendo May 30, 2024
dc66ef9
Use explicit return values
nineteendo May 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Revert "Revert "Direct C call for nt.isdevdrive()""
This reverts commit 9c55537.
  • Loading branch information
nineteendo committed May 4, 2024
commit 2bf2cd3459d0f5a5497e424532793fed6feb46c6
14 changes: 1 addition & 13 deletions 14 Lib/ntpath.py
Original file line number Diff line number Diff line change
Expand Up @@ -933,19 +933,7 @@ def commonpath(paths):
from nt import _path_isfile as isfile
from nt import _path_islink as islink
from nt import _path_exists as exists
from nt import _path_isdevdrive as isdevdrive
except ImportError:
# Use genericpath.* as imported above
pass


try:
from nt import _path_isdevdrive
def isdevdrive(path):
"""Determines whether the specified path is on a Windows Dev Drive."""
try:
return _path_isdevdrive(abspath(path))
except OSError:
return False
except ImportError:
# Use genericpath.isdevdrive as imported above
pass
2 changes: 2 additions & 0 deletions 2 Lib/test/test_ntpath.py
Original file line number Diff line number Diff line change
Expand Up @@ -1135,6 +1135,8 @@ def test_fast_paths_in_use(self):
self.assertFalse(inspect.isfunction(os.path.islink))
self.assertTrue(os.path.exists is nt._path_exists)
self.assertFalse(inspect.isfunction(os.path.exists))
self.assertTrue(os.path.isdevdrive is nt._path_isdevdrive)
self.assertFalse(inspect.isfunction(os.path.isdevdrive))

@unittest.skipIf(os.name != 'nt', "Dev Drives only exist on Win32")
def test_isdevdrive(self):
Expand Down
15 changes: 5 additions & 10 deletions 15 Modules/clinic/posixmodule.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

235 changes: 209 additions & 26 deletions 235 Modules/posixmodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -4723,16 +4723,38 @@ os_listmounts_impl(PyObject *module, path_t *volume)
/*[clinic input]
os._path_isdevdrive

path: path_t
path: object

Determines whether the specified path is on a Windows Dev Drive.

[clinic start generated code]*/

static PyObject *
os__path_isdevdrive_impl(PyObject *module, path_t *path)
/*[clinic end generated code: output=1f437ea6677433a2 input=ee83e4996a48e23d]*/
os__path_isdevdrive_impl(PyObject *module, PyObject *path)
/*[clinic end generated code: output=4f5a02fe80648dab input=cb26f0f438f49cb6]*/
{
path_t _path = PATH_T_INITIALIZE("isdevdrive", "path", 0, 0);
if (!path_converter(path, &_path)) {
path_cleanup(&_path);
if (PyErr_ExceptionMatches(PyExc_ValueError)) {
PyErr_Clear();
Py_RETURN_FALSE;
}
return NULL;
}

PyObject *abs_obj = posix_abspath((wchar_t *)_path.wide, _path.length, 0);
if (!abs_obj) {
path_cleanup(&_path);
return NULL;
}
wchar_t *abs_buf = PyUnicode_AsWideCharString(abs_obj, NULL);
Py_DECREF(abs_obj);
if (!abs_buf) {
path_cleanup(&_path);
return NULL;
}

#ifndef PERSISTENT_VOLUME_STATE_DEV_VOLUME
/* This flag will be documented at
https://learn.microsoft.com/windows-hardware/drivers/ddi/ntifs/ns-ntifs-_file_fs_persistent_volume_information
Expand All @@ -4746,7 +4768,7 @@ os__path_isdevdrive_impl(PyObject *module, path_t *path)
wchar_t volume[MAX_PATH];

Py_BEGIN_ALLOW_THREADS
if (!GetVolumePathNameW(path->wide, volume, MAX_PATH)) {
if (!GetVolumePathNameW(abs_buf, volume, MAX_PATH)) {
/* invalid path of some kind */
/* Note that this also includes the case where a volume is mounted
in a path longer than 260 characters. This is likely to be rare
Expand Down Expand Up @@ -4796,6 +4818,8 @@ os__path_isdevdrive_impl(PyObject *module, path_t *path)
}
Py_END_ALLOW_THREADS

path_cleanup(&_path);
PyMem_RawFree(abs_buf);
if (err) {
PyErr_SetFromWindowsErr(err);
return NULL;
Expand Down Expand Up @@ -5508,38 +5532,158 @@ os__path_normpath_impl(PyObject *module, PyObject *path)
return result;
}

/*[clinic input]
os._path_abspath
#ifdef MS_WINDOWS

path: unicode
/
/* We centralise SECURITY_ATTRIBUTE initialization based around
templates that will probably mostly match common POSIX mode settings.
The _Py_SECURITY_ATTRIBUTE_DATA structure contains temporary data, as
a constructed SECURITY_ATTRIBUTE structure typically refers to memory
that has to be alive while it's being used.

Make path absolute.
[clinic start generated code]*/
Typical use will look like:
SECURITY_ATTRIBUTES *pSecAttr = NULL;
struct _Py_SECURITY_ATTRIBUTE_DATA secAttrData;
int error, error2;

Py_BEGIN_ALLOW_THREADS
switch (mode) {
case 0x1C0: // 0o700
error = initializeMkdir700SecurityAttributes(&pSecAttr, &secAttrData);
break;
...
default:
error = initializeDefaultSecurityAttributes(&pSecAttr, &secAttrData);
break;
}

if (!error) {
// do operation, passing pSecAttr
}

// Unconditionally clear secAttrData.
error2 = clearSecurityAttributes(&pSecAttr, &secAttrData);
if (!error) {
error = error2;
}
Py_END_ALLOW_THREADS

if (error) {
PyErr_SetFromWindowsErr(error);
return NULL;
}
*/

struct _Py_SECURITY_ATTRIBUTE_DATA {
SECURITY_ATTRIBUTES securityAttributes;
PACL acl;
SECURITY_DESCRIPTOR sd;
EXPLICIT_ACCESS_W ea[4];
char sid[64];
};

static int
initializeDefaultSecurityAttributes(
PSECURITY_ATTRIBUTES *securityAttributes,
struct _Py_SECURITY_ATTRIBUTE_DATA *data
) {
assert(securityAttributes);
assert(data);
*securityAttributes = NULL;
memset(data, 0, sizeof(*data));
return 0;
}

static int
initializeMkdir700SecurityAttributes(
PSECURITY_ATTRIBUTES *securityAttributes,
struct _Py_SECURITY_ATTRIBUTE_DATA *data
) {
assert(securityAttributes);
assert(data);
*securityAttributes = NULL;
memset(data, 0, sizeof(*data));

if (!InitializeSecurityDescriptor(&data->sd, SECURITY_DESCRIPTOR_REVISION)
|| !SetSecurityDescriptorGroup(&data->sd, NULL, TRUE)) {
return GetLastError();
}

int use_alias = 0;
DWORD cbSid = sizeof(data->sid);
if (!CreateWellKnownSid(WinCreatorOwnerRightsSid, NULL, (PSID)data->sid, &cbSid)) {
use_alias = 1;
}

data->securityAttributes.nLength = sizeof(SECURITY_ATTRIBUTES);
data->ea[0].grfAccessPermissions = GENERIC_ALL;
data->ea[0].grfAccessMode = SET_ACCESS;
data->ea[0].grfInheritance = SUB_CONTAINERS_AND_OBJECTS_INHERIT;
if (use_alias) {
data->ea[0].Trustee.TrusteeForm = TRUSTEE_IS_NAME;
data->ea[0].Trustee.TrusteeType = TRUSTEE_IS_ALIAS;
data->ea[0].Trustee.ptstrName = L"CURRENT_USER";
} else {
nineteendo marked this conversation as resolved.
Show resolved Hide resolved
data->ea[0].Trustee.TrusteeForm = TRUSTEE_IS_SID;
data->ea[0].Trustee.TrusteeType = TRUSTEE_IS_WELL_KNOWN_GROUP;
data->ea[0].Trustee.ptstrName = (LPWCH)(SID*)data->sid;
}

data->ea[1].grfAccessPermissions = GENERIC_ALL;
data->ea[1].grfAccessMode = SET_ACCESS;
data->ea[1].grfInheritance = SUB_CONTAINERS_AND_OBJECTS_INHERIT;
data->ea[1].Trustee.TrusteeForm = TRUSTEE_IS_NAME;
data->ea[1].Trustee.TrusteeType = TRUSTEE_IS_ALIAS;
data->ea[1].Trustee.ptstrName = L"SYSTEM";

data->ea[2].grfAccessPermissions = GENERIC_ALL;
data->ea[2].grfAccessMode = SET_ACCESS;
data->ea[2].grfInheritance = SUB_CONTAINERS_AND_OBJECTS_INHERIT;
data->ea[2].Trustee.TrusteeForm = TRUSTEE_IS_NAME;
data->ea[2].Trustee.TrusteeType = TRUSTEE_IS_ALIAS;
data->ea[2].Trustee.ptstrName = L"ADMINISTRATORS";

int r = SetEntriesInAclW(3, data->ea, NULL, &data->acl);
if (r) {
return r;
}
if (!SetSecurityDescriptorDacl(&data->sd, TRUE, data->acl, FALSE)) {
return GetLastError();
}
data->securityAttributes.lpSecurityDescriptor = &data->sd;
*securityAttributes = &data->securityAttributes;
return 0;
}

static int
clearSecurityAttributes(
PSECURITY_ATTRIBUTES *securityAttributes,
struct _Py_SECURITY_ATTRIBUTE_DATA *data
) {
assert(securityAttributes);
assert(data);
*securityAttributes = NULL;
if (data->acl) {
if (LocalFree((void *)data->acl)) {
return GetLastError();
}
}
return 0;
}

#endif

static PyObject *
os__path_abspath_impl(PyObject *module, PyObject *path)
/*[clinic end generated code: output=b58956d662b60be0 input=577ecb3473d22113]*/
posix_abspath(wchar_t *path_buf, Py_ssize_t path_len, int use_bytes)
{
Py_ssize_t path_len, abs_len;
Py_ssize_t abs_len;
wchar_t *abs, *abs_buf = NULL, *cwd_buf = NULL;
nineteendo marked this conversation as resolved.
Show resolved Hide resolved
PyObject *result = NULL;

wchar_t *path_buf = PyUnicode_AsWideCharString(path, &path_len);
if (!path_buf) {
goto exit;
}

#ifdef MS_WINDOWS
nineteendo marked this conversation as resolved.
Show resolved Hide resolved
if (wcslen(path_buf) != path_len) {
PyErr_Format(PyExc_ValueError,
"_path_abspath: embedded null character in path");
goto exit;
}
// Preserve `.\` for qualified referencing
abs = _Py_normpath_and_size(path_buf, path_len, 0, &abs_len, 1);
if (abs_len == 0 || (abs_len == 1 && abs[0] == L'.')) {
result = posix_getcwd(0);
result = posix_getcwd(use_bytes);
goto exit;
}
if (_PyOS_getfullpathname(abs, &abs_buf) < 0) {
Expand All @@ -5550,7 +5694,7 @@ os__path_abspath_impl(PyObject *module, PyObject *path)
abs_len = wcslen(abs_buf);
#else
if (path_len == 0 || (path_len == 1 && path_buf[0] == L'.')) {
result = posix_getcwd(0);
result = posix_getcwd(use_bytes);
goto exit;
}

Expand Down Expand Up @@ -5597,14 +5741,53 @@ os__path_abspath_impl(PyObject *module, PyObject *path)
#endif

result = PyUnicode_FromWideChar(abs, abs_len);
if (use_bytes) {
Py_SETREF(result, PyUnicode_EncodeFSDefault(result));
}

exit:
PyMem_Free(path_buf);
PyMem_Free(cwd_buf);
PyMem_RawFree(abs_buf);
return result;
}


/*[clinic input]
os._path_abspath

path: unicode
/

Make path absolute.
nineteendo marked this conversation as resolved.
Show resolved Hide resolved
[clinic start generated code]*/

static PyObject *
os__path_abspath_impl(PyObject *module, PyObject *path)
/*[clinic end generated code: output=b58956d662b60be0 input=577ecb3473d22113]*/
{
Py_ssize_t path_len;
PyObject *result = NULL;

wchar_t *path_buf = PyUnicode_AsWideCharString(path, &path_len);
if (!path_buf) {
goto exit;
}

#ifdef MS_WINDOWS
if (wcslen(path_buf) != path_len) {
PyErr_Format(PyExc_ValueError,
"_path_abspath: embedded null character in path");
goto exit;
}
#endif

result = posix_abspath(path_buf, path_len, 0);

exit:
PyMem_Free(path_buf);
return result;
}

/*[clinic input]
os.mkdir

Expand Down
Morty Proxy This is a proxified and sanitized view of the page, visit original site.