Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

gh-102255: Improve build support on xbox #102256

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 104 commits into from
Mar 9, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
104 commits
Select commit Hold shift + click to select a range
58ab04a
better support building on non desktop windows systems
maxbachmann Feb 24, 2023
66a68bd
stub posix apis
maxbachmann Feb 25, 2023
26f5a44
add basic isabs implementation
maxbachmann Feb 25, 2023
68a1f67
implement isxfile
maxbachmann Feb 25, 2023
492bac0
fix static linking
maxbachmann Feb 25, 2023
26b4b24
fix guards
maxbachmann Feb 25, 2023
467bf40
add MS_XBOX
maxbachmann Feb 25, 2023
e734492
cleanup
maxbachmann Feb 25, 2023
fad23bc
use winsock2 on xbox
maxbachmann Feb 26, 2023
b6b28ec
stub winreg
maxbachmann Feb 26, 2023
cdbb1f2
rename to MS_GAMES
maxbachmann Feb 26, 2023
e7622cc
fix guard
maxbachmann Feb 26, 2023
7812891
add missing semicolon
maxbachmann Feb 26, 2023
3b005f4
remove windows console
maxbachmann Feb 26, 2023
fa526b6
do not read version from kernel32.dll
maxbachmann Feb 26, 2023
36887bc
apply part of code review
maxbachmann Feb 26, 2023
6fe3018
reduce diff
maxbachmann Feb 26, 2023
8499940
regenerate argument clinic
maxbachmann Feb 26, 2023
69f0a2c
include winioctl
maxbachmann Feb 26, 2023
f43ce54
exclude dll_directory
maxbachmann Feb 26, 2023
43cf0a5
patch mscvrtmodule
maxbachmann Feb 26, 2023
8f717ea
patch _winapi
maxbachmann Feb 26, 2023
29d5b65
📜🤖 Added by blurb_it.
blurb-it[bot] Feb 26, 2023
8a73500
patch out tzset
maxbachmann Feb 26, 2023
92ae5ef
patch support_wsa_no_inherit
maxbachmann Feb 26, 2023
f27fe95
patch scoketmodule
maxbachmann Feb 26, 2023
f15d9e2
PathCchCombineEx is unavailable on xbox
maxbachmann Feb 26, 2023
d816baa
patch out remaining funcs
maxbachmann Feb 26, 2023
2ba0a87
Merge branch 'main' into xbox
maxbachmann Feb 26, 2023
ed17e03
apply code review
maxbachmann Feb 26, 2023
c360b4a
cleanup
maxbachmann Feb 26, 2023
3b5b119
remove unneded defined
maxbachmann Feb 26, 2023
975191a
regenerate argument clinic
maxbachmann Feb 26, 2023
91a95f2
remove unused function
maxbachmann Feb 26, 2023
3be5fa7
use processsnapshot api
maxbachmann Feb 26, 2023
6bd1207
simplify implementation
maxbachmann Feb 26, 2023
ca98366
simplify implementation
maxbachmann Feb 26, 2023
fac3431
follow pep7
maxbachmann Feb 26, 2023
c1bf0e2
follow pep7
maxbachmann Feb 26, 2023
ace99f6
Update Modules/posixmodule.c
maxbachmann Feb 26, 2023
e83d398
remove duplicated comment
maxbachmann Feb 26, 2023
76ea310
test error message as well
maxbachmann Feb 26, 2023
2f1d19c
fix condition
maxbachmann Feb 26, 2023
559199f
use sys._base_executable
maxbachmann Feb 26, 2023
b119bf5
do not build mmap on unsupported platforms
maxbachmann Feb 26, 2023
1dd69e8
Apply suggestions from code review
maxbachmann Feb 27, 2023
a1e6dba
add inheritable back
maxbachmann Feb 27, 2023
7a45988
Update Python/fileutils.c
maxbachmann Feb 27, 2023
846f8eb
Update Python/fileutils.c
maxbachmann Feb 27, 2023
a56f215
add back some winreg APIs
maxbachmann Feb 27, 2023
191cab2
regenerate argument clinic
maxbachmann Feb 27, 2023
144280e
do not name platform xbox
maxbachmann Feb 27, 2023
ca9f7d9
exclude functions
maxbachmann Feb 27, 2023
30c0818
implement without nt._path_splitroot
maxbachmann Feb 27, 2023
6ebb05b
Update Modules/socketmodule.c
maxbachmann Feb 27, 2023
fc5c642
apply some code review change requests
maxbachmann Feb 27, 2023
5f286a7
get rid of support_wsa_no_inherit
maxbachmann Feb 27, 2023
3be5e3c
implement _path_splitroot
maxbachmann Feb 27, 2023
d9e1663
Merge branch 'main' into xbox
maxbachmann Feb 28, 2023
2a2a8fd
fix revert of changes to _bootstrap_external
maxbachmann Feb 28, 2023
165286d
reimplement winreg_QueryValue_impl and winreg_SetValue_impl
maxbachmann Feb 28, 2023
1413849
Update Python/fileutils.c
maxbachmann Feb 28, 2023
6465d73
move skip root to own function
maxbachmann Feb 28, 2023
3f0c6f0
cleanup
maxbachmann Feb 28, 2023
8f3a262
better handle joinfile on xbox
maxbachmann Feb 28, 2023
6b8074e
cleanup
maxbachmann Feb 28, 2023
99b6a1d
Update Modules/socketmodule.c
maxbachmann Feb 28, 2023
d0e548d
define missing flags on non desktop builds
maxbachmann Feb 28, 2023
c569a18
set inherit in WASSocketW
maxbachmann Feb 28, 2023
3e86c9e
add back dynamic loading
maxbachmann Feb 28, 2023
d5cb524
more cleanup
maxbachmann Feb 28, 2023
63267d8
rename platform defines
maxbachmann Feb 28, 2023
79f2bfe
add missing inherit flag
maxbachmann Feb 28, 2023
fc6b942
fix guard
maxbachmann Feb 28, 2023
05e9859
use api partititions
maxbachmann Feb 28, 2023
9769662
fix guards
maxbachmann Feb 28, 2023
8f12bf8
add missing include
maxbachmann Mar 1, 2023
509d0f5
Merge branch 'main' into xbox
maxbachmann Mar 1, 2023
f20fea8
apply code review
maxbachmann Mar 1, 2023
e33a42f
move replacement functions into fileutils
maxbachmann Mar 1, 2023
556a8e1
disable dynamic load of pythoncore for static lib
maxbachmann Mar 1, 2023
aa8964b
Merge branch 'main' into xbox
maxbachmann Mar 1, 2023
74258ca
use core dll for windows specific
maxbachmann Mar 1, 2023
686da59
Update Python/fileutils.c
maxbachmann Mar 1, 2023
b151f7f
apply code review
maxbachmann Mar 1, 2023
39d2d13
rename to PathCchCombineEx
maxbachmann Mar 1, 2023
2895563
define for all BUILD_CORE
maxbachmann Mar 1, 2023
8e3d3c5
cleanup
maxbachmann Mar 1, 2023
659d1a3
Update Modules/_randommodule.c
maxbachmann Mar 1, 2023
e9ff60e
add MS_WINDOWS_CRT_DESKTOP
maxbachmann Mar 2, 2023
32c4886
Update Modules/posixmodule.c
maxbachmann Mar 2, 2023
2f1fe40
Update Modules/posixmodule.c
maxbachmann Mar 2, 2023
5854e91
revert MS_WINDOWS_CRT_DESKTOP
maxbachmann Mar 2, 2023
d3d2e1c
use MS_WINDOWS_SYSTEM
maxbachmann Mar 3, 2023
b1ffce9
Apply suggestions from code review
maxbachmann Mar 3, 2023
0b4d407
Update Modules/socketmodule.c
maxbachmann Mar 3, 2023
d9976dc
handle failing malloc
maxbachmann Mar 3, 2023
427de79
Update Modules/posixmodule.c
maxbachmann Mar 4, 2023
603cea7
Update Python/fileutils.c
maxbachmann Mar 6, 2023
6db80a3
Update Include/internal/pycore_fileutils.h
maxbachmann Mar 6, 2023
4d78066
Update Include/internal/pycore_fileutils.h
maxbachmann Mar 6, 2023
65cdd56
Update Python/fileutils.c
maxbachmann Mar 6, 2023
35217ab
Update Python/fileutils.c
maxbachmann Mar 7, 2023
331b1f4
dynamically import from api-ms-win-core-path-l1-1-0.dll
maxbachmann Mar 9, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
better handle joinfile on xbox
  • Loading branch information
maxbachmann committed Feb 28, 2023
commit 8f3a2629722166bb329428b9f6f5cb3f0a5ae614
2 changes: 1 addition & 1 deletion 2 Modules/posixmodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -4442,7 +4442,7 @@ win32_games_skip_root(wchar_t* path)
return path + wcslen(path);
}
end = wcschr(end + 1, '\\');
return (!end) ? wcslen(path) : end + 1;
return (!end) ? path + wcslen(path) : end + 1;
}
/* absolute / relative path with drive, e.g. C: or C:\ */
else if (isalpha(path[0]) && path[1] == L':') {
Expand Down
92 changes: 91 additions & 1 deletion 92 Python/fileutils.c
Original file line number Diff line number Diff line change
Expand Up @@ -2117,17 +2117,107 @@ _Py_abspath(const wchar_t *path, wchar_t **abspath_p)
#endif
}

#ifdef MS_WINDOWS_GAMES
static wchar_t*
win32_games_skip_root(wchar_t* path)
{
if (path[0] == '\\') {
/* relative path with root e.g. \Windows */
if (path[1] != '\\') {
return path + 1;
}

/* UNC drives e.g. \\server\share or \\?\UNC\server\share */
wchar_t *end = path + 2;
if (!wcsnicmp(end, L"?\\UNC\\", 6)) {
end += 6;
}

end = wcschr(end, '\\');
if (!end) {
return path + wcslen(path);
}
end = wcschr(end + 1, '\\');
return (!end) ? path + wcslen(path) : end + 1;
}
/* absolute / relative path with drive, e.g. C: or C:\ */
else if (isalpha(path[0]) && path[1] == ':') {
return (path[2] == '\\') ? path + 3 : path + 2;
}

/* relative path */
return NULL;
}

// The Windows Games API partition does not provide PathCchCombineEx
// so we need our own implementation
static int
win32_games_join_relfile(wchar_t *buffer, size_t bufsize,
maxbachmann marked this conversation as resolved.
Show resolved Hide resolved
const wchar_t *dirname, const wchar_t *relfile)
{
if ((isalpha(relfile[0]) && relfile[1] == ':') ||
(relfile[0] == '\\' && relfile[1] == '\\'))
{
dirname = relfile;
relfile = NULL;
}

size_t dir_len = wcslen(dirname);
size_t file_len = relfile ? wcslen(relfile) : 0;
/* path is at max dirname + filename + backslash + \0 */
size_t new_len = dir_len + file_len + 2;
if (bufsize >= MAXPATHLEN || new_len > bufsize) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The value of bufsize isn't limited. The result length may be limited. If PATHCCH_ALLOW_LONG_PATHS isn't set in flags, and new_len >= MAX_PATH (not MAXPATHLEN), the returned error should be HRESULT_FROM_WIN32(ERROR_FILENAME_EXCED_RANGE).

Else if new_len is greater than bufsize, the returned error should be HRESULT_FROM_WIN32(ERROR_INSUFFICIENT_BUFFER).

PATHCCH_ALLOW_LONG_PATHS also determines whether to convert to or from an extended path, depending on whether the current process supports long paths (Python 3.6+ does). But it's okay to omit that behavior for GAMES.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah, we shouldn't have references to MAXPATHLEN in here.

I'm not so concerned about error codes. They're only going to be reported to users, and really we ought to be getting the buffer right ourselves before calling.

I think it's fine for this implementation to assume PATHCCH_ALLOW_LONG_PATHS. Don't worry about mimicking every possible behaviour of the original.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (bufsize >= MAXPATHLEN || new_len > bufsize) {
if (new_len > bufsize) {

Don't worry about mimicking every possible behaviour of the original.

I agree, if getting the correct behavior is excessively complicated with no tangible benefit. What about my other comment on this function? The way PathCchStripToRoot() is used, when relfile is a rooted path, leads to incorrect and inconsistent behavior. If I know I'm combining a rooted path, I may use a smaller buffer because I know that most of dirname won't be copied -- just the drive part. The current implementation requires using a buffer that's big enough to hold dirname, relname and a joining backslash. OTOH, if relname has a drive or UNC share and thus replaces dirname, then the length of the passed-in dirname is ignored.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Firstly, the suggested change is good.

I'm not particularly worried about the case where relname would shorten the overall drive. It seems something you could take advantage of, but generally we don't control either half of the paths that will come in here, which means the caller is going to have to assume worst case of plain-old concatenation.

Maybe just add a comment like /* assuming worst case result length, so our caller should as well */.

The only reasonable alternative (without complicating everything) would involve modifying the result buffer before potentially returning an error, which I think is worse. I'm pretty sure the original API uses temporary buffers internally, but there's no real need to go there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I would assume the implementation to use something like PathAllocCombine internally

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we really wanted to I could come up with an implementation that behaves closer to PathCchCombineEx. I was simply not sure it was worth the complexity (I kind of hope that at some point microsoft makes these APIs available on the xbox, since there does not appear to be any inherent reason to provide them on every platform except the xbox)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only reasonable alternative (without complicating everything) would involve modifying the result buffer before potentially returning an error, which I think is worse. I'm pretty sure the original API uses temporary buffers internally, but there's no real need to go there.

It would calculate dir_len based on PathCchSkipRoot() if relfile begins with a backslash. Then it's the same initial test based on new_len = dir_len + file_len + 2, requiring new_len < bufsize. Copying dirname would now use wcsncpy(buffer, dirname, dir_len), and PathCchStripToRoot() would not have to be implemented. So you get better behavior AND it's simpler overall.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hold off on any more changes on this function for a day or two - I'm having some interesting conversations with colleagues right now. Might have a more suitable approach (or at least more information about it).

return -1;
}

size_t combined_length = dir_len;
wcscpy(buffer, dirname);
if (!relfile || !relfile[0]) {
if(wcsncmp(buffer, L"\\\\?\\", 4)) {
buffer += 4;
}
if (isalpha(buffer[0]) && buffer[1] == ':') {
if (!buffer[2]) {
buffer[2] = '\\';
buffer[3] = '\0';
}
}
}
else {
if (relfile[0] == '\\' && relfile[1] != '\\')
{
wchar_t* root_end = win32_games_skip_root(buffer);
if (root_end) {
*root_end = '\0';
}
combined_length = root_end - buffer;
relfile++;
}
if (combined_length && buffer[combined_length - 1] != '\\') {
buffer[combined_length++] = '\\';
buffer[combined_length] = '\0';
}
wcscat(buffer, relfile);
}
return 0;
}
#endif /* MS_WINDOWS_GAMES */

// The caller must ensure "buffer" is big enough.
static int
join_relfile(wchar_t *buffer, size_t bufsize,
const wchar_t *dirname, const wchar_t *relfile)
{
#if defined(MS_WINDOWS) && !defined(MS_WINDOWS_GAMES)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if dirname is "G:\eggs" and relfile is the relative rooted path "\spam"? This should be joined as "G:\spam", not as "G:\eggs\spam". relfile could also be a relative drive path such as "T:spam", which should join as "T:spam" because it's a different drive, not as "G:\eggs\T:spam". Code that's meant for POSIX doesn't handle such cases. Maybe that's good enough.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mostly selected this implementation for it's simplicity. On the xbox you should probably simply work with absolute paths, so I considered it as good enough for now. That said a better implementation would certainly not hurt.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On all platforms you should work with absolute paths :)

You're likely in control of your development process well enough to ensure it, but we need to account for broader expectations than that when it comes into core.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm I guess on Windows/Linux we do use paths relative to the binary we ship. On Xbox basically every application has their "own" filesystem, so you really use absolute paths everywhere. I guess that might be the reason why they do not provide the API there.

Anyway I know to little about WIndows file path concatenation to come up with a better implementation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The POSIX code is okay as long as it's always a simple relative path such as "eggs\foo.py". One minor annoyance is if dirname ends in a forward slash, such as "T:/spam/". It will add SEP to join it as "T:/spam/\eggs\foo.py". Such a path still works, so it's not a technical problem.

Copy link
Contributor Author

@maxbachmann maxbachmann Feb 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added an implementation which should be able to handle more of these cases on the xbox.

#ifdef MS_WINDOWS_GAMES
return win32_games_join_relfile(buffer, bufsize, dirname, relfile)
#else
if (FAILED(PathCchCombineEx(buffer, bufsize, dirname, relfile,
PATHCCH_ALLOW_LONG_PATHS))) {
return -1;
}
return 0;
#endif
#else
assert(!_Py_isabs(relfile));
size_t dirlen = wcslen(dirname);
Expand All @@ -2151,8 +2241,8 @@ join_relfile(wchar_t *buffer, size_t bufsize,
}
wcscpy(&buffer[relstart], relfile);
}
#endif
return 0;
#endif
}

/* Join the two paths together, like os.path.join(). Return NULL
Expand Down
Morty Proxy This is a proxified and sanitized view of the page, visit original site.