Add Python extension using pocketpy#124
Add Python extension using pocketpy#124hcarty wants to merge 8 commits intoorx:masterorx/orx:masterfrom hcarty:pocketpy-python-extensionhcarty/orx:pocketpy-python-extensionCopy head branch name to clipboard
Conversation
588a49c to
979f215
Compare
hcarty
left a comment
There was a problem hiding this comment.
Adding some review comments.
| @@ -0,0 +1,2 @@ | ||
| [tool.pyright] |
There was a problem hiding this comment.
This is used by pyright, a type checker for Python. There are no *.py sources for the core engine bindings so the type checker will warn or error on some settings otherwise.
There was a problem hiding this comment.
Can't this be initialized programmatically instead?
There was a problem hiding this comment.
I can remove the configuration file. It's something that was more necessary when I started the extension, but the tooling has advanced since then and people may want to use other tooling for their own projects.
There was a problem hiding this comment.
By that I meant, could this be expressed in an orx config file instead, and set programmatically when starting the extension?
This way we keep all configuration stuff consistent across engine & extensions.
There was a problem hiding this comment.
This is pocketpy's header. The only modification from upstream is the reference to orx's memory allocation functions.
There was a problem hiding this comment.
This is pocketpy's implementation. It is taken directly from upstream without modification.
There was a problem hiding this comment.
This should be moved inside include/extensions, and included directly from orxPy.h.
There was a problem hiding this comment.
I started with that but it breaks compilation of C++ projects since the code does not compile as C++.
There was a problem hiding this comment.
Ah I see, in that case let's move it inside the newly added extensions subdirectory.
There was a problem hiding this comment.
Very nice change - thank you! I moved src/pocketpy/ into src/extensions/pocketpy/.
| ] | ||
| [+python | ||
| // Initialize Python support | ||
| orxPy_InitVM(); |
There was a problem hiding this comment.
Initialize the pocketpy virtual machine. This does not execute any game-specific Python code. It does define all of the orx.* engine module bindings.
There was a problem hiding this comment.
This calls the user's/game's orx.init function, called from the init project's main source file.
There was a problem hiding this comment.
Shouldn't this call also be inside orxExtensions.h, in InitExtensions()?
Is there a reason why it is separate from orxPy_InitVM()?
There was a problem hiding this comment.
I started with this combined as well, but if a project defines its own modules from C then those need to be defined before any game Python code is run. Splitting the two initializes the VM early, then gives space for custom C bindings and module definitions.
There was a problem hiding this comment.
Does the VM rely on any orx API? If not, we could move its initialization to the BootStrap function instead.
I'd rather not have any extension code outside of orxExtensions.h as much as possible as it breaks adding/removing extensions (the main source file can't be touched once created).
There was a problem hiding this comment.
Short version of all the text below: I think we can make this change. I have some minor concerns around more advanced use cases described below. If it's ok for me to change the InitExtensions return type then I think my main concerns would be addressed.
The call to orxPy_Init here is like the call to orxPy_Update and orxPy_CameraUpdate earlier in the file. They each call Python functions defined by game code at the engine-appropriate time. I've tested locally and orxPy_Init could be called from InitExtensions if orxPy_InitVM is called from BootstrapExtensions.
I have some concerns with that change:
InitExtensionsdoesn't return a status value. Would it be ok to haveInitExtensionsreturnorxSTATUSrather thanvoid? In the PR's current state a failure duringorxInit_Pywill stop the game from starting up. I think that's a good feature to have but I'm not sure if others would agree.- It obfuscates where the game's Python code is loaded and the user's
orx.initcallback function is called. I think that's ok. The Python extension is kind of like Scroll in that it can be used to change where primary game logic is run (C vs C++ vs C++/Scroll vs Python). However the Python extension uses the existing C/C++/Scroll engine callbacks for bootstrap/init/update/camera update rather than defining its own. - It is less clear where a user's custom C to Python bindings should be added. If
orxPy_Initis called fromInitExtensionsthen those modules must be defined before the call toInitExtensions. That should work as long as those module definitions do not rely on other extensions being initialized first. Custom module definitions shouldn't require that - initialization should happen when the game logic is initialized. I think that's ok as well.
A bit of extra context on the extension's two init functions:
orxPy_InitVM initializes the pocketpy VM and registers the orx.* Python modules with that VM. It does not run any Python code or call any orx APIs. It could be called from BootstrapExtensions.
orxPy_Init loads and runs the user's [=name].py code, sets up any references or bindings between user code and the pocketpy VM, and calls the user-defined Python init function if one has been defined. Arbitrary orx and Python code can be run at this point.
There was a problem hiding this comment.
I'll have to check orxPy.h, but shouldn't it no rely on Scroll at all and be orthogonal to it, so as to have a single code flow no matter what the kind of project is?
97aba70 to
c5a7d73
Compare
6b8e77b to
7710397
Compare
95acc75 to
c251a20
Compare
ba19856 to
369a643
Compare
369a643 to
0aa62c0
Compare
0aa62c0 to
ac40c04
Compare
dc1b963 to
8db24ea
Compare
8db24ea to
638c2bb
Compare
feda00f to
20af680
Compare
20af680 to
d84797c
Compare
d84797c to
529e69d
Compare
9471b2d to
2032939
Compare
af16e1c to
a2a66a0
Compare
41e2226 to
c71a53d
Compare
hcarty
left a comment
There was a problem hiding this comment.
Added a few more small review comments.
| configuration {"*Profile*"} | ||
| targetsuffix ("p") | ||
| defines {"__orxPROFILER__"} | ||
| defines {"__orxPROFILER__"[+python ,"NDEBUG"]} |
There was a problem hiding this comment.
NDEBUG signals to pocketpy that this is not a debug build.
There was a problem hiding this comment.
You can add a separate defines line for Python, it might make it slightly more easy to read.
Or just decompose the existing one on multiple lines.
| orx.on_camera_update = camera_update | ||
|
|
||
| # Setup engine callbacks for miniscroll | ||
| miniscroll.setup(init, update) |
There was a problem hiding this comment.
miniscroll has its own registration logic to simplify what needs to be done by the user.
There was a problem hiding this comment.
Should we give it a more obvious name maybe, like PyScroll?
There was a problem hiding this comment.
I like PyScroll! I started with "mini" as the prefix because of its relatively limited scope, but I think PyScroll is better and disambiguates well with Scroll.
c71a53d to
689ae8a
Compare
- Based on pocketpy (https://pocketpy.dev/)
689ae8a to
01fe4ae
Compare
Wouldn't
Love that part! :) Additionally, would there be any benefit to expose orx's new task worker pool to the Python environment? |
|
|
||
| // clang-format off | ||
|
|
||
| #define PK_VERSION "2.1.0" |
There was a problem hiding this comment.
Minor alignment issue. Is this file auto-generated?
There was a problem hiding this comment.
This file comes from the pocketpy project. pocketpy generates the combined pocketpy.h and pocketpy.c files from individual header and source files as part of the release process.
There was a problem hiding this comment.
I'll have to check orxPy.h, but shouldn't it no rely on Scroll at all and be orthogonal to it, so as to have a single code flow no matter what the kind of project is?
| @@ -0,0 +1,2 @@ | ||
| [tool.pyright] |
There was a problem hiding this comment.
By that I meant, could this be expressed in an orx config file instead, and set programmatically when starting the extension?
This way we keep all configuration stuff consistent across engine & extensions.
| [+inspector | ||
| OnCreate = Inspector.RegisterObject ^] | ||
| [+python | ||
| Input = @ |
There was a problem hiding this comment.
Why not use the TriggerList here as well?
There was a problem hiding this comment.
miniscroll/pyscroll does not support the same . and other prefixes that Scroll does, so it can't match the behavior the scroll provides with the TriggerList. It's something I could probably add, but last time I looked there were some features Scroll relies on for this that were not bound by orxpy.
There was a problem hiding this comment.
Ah I see, so the inputs are handled entirely in Python code then?
Fine by me, but I do find the data-only input handling handy for simple situations and I would recommend implementing something similar to what's inside ScrollBase (it's not much code).
There was a problem hiding this comment.
so the inputs are handled entirely in Python code then?
Somewhat - it uses a similar approach to what is done in Scroll, but not as refined. This is run for each orx object registered with pyscroll/miniscroll on each update:
# Input triggers
for input_name in mini.input_names:
if orx.input.has_been_activated(input_name):
mini.o.fire_trigger("Input", refinement=[input_name])The mini.input_names list is populated when the object is created, based on the input set associated with the object, similar to what's done by Scroll but captured on object creation rather than looking up the inputs at runtime.
I agree the additional functionality and flexibility provided by Scroll is worth having here. I'll work on that.
There was a problem hiding this comment.
You can add a separate defines line for Python, it might make it slightly more easy to read.
Or just decompose the existing one on multiple lines.
I decomposed the existing one on multiple lines and am happy to change the approach further if you prefer.
| ] | ||
| [+python | ||
| // Call Python update | ||
| [-scroll orxPy_Update(_pstClockInfo, orxNULL);][+scroll orxPy_Update(&_rstClockInfo, orxNULL);]] |
There was a problem hiding this comment.
if the only difference is the first parameter, you can have the -scroll/+scroll only for that parameter instead, for brevity.
| ] | ||
| [+python | ||
| // Call Python camera update | ||
| [-scroll orxPy_CameraUpdate(_pstClockInfo, orxNULL);][+scroll orxPy_CameraUpdate(&_rstClockInfo, orxNULL);]] |
There was a problem hiding this comment.
Similar suggestion as before: use -scroll/+scroll only around the first parameter.
|
|
||
| ] | ||
| [+scroll | ||
| [-python |
There was a problem hiding this comment.
All the -python blocks could be merged in a single one and the callbacks registered at the very end instead.
- PR style feedback
I think that's reasonable. I went back and forth in my own head a lot about the naming for this. pyscroll is probably a better name. I currently use
I do too! That's the main incentive I had to try this out in the first place. No separate runtime to worry about, or binary compatibility issues.
There may be, but I don't think it's needed at this point. pocketpy supports multiple VMs running in parallel, with one VM per thread. It uses C11 threading, manages its own threads, and that does work with this extension. None of the orx modules are available from any VM other than the main VM though, to avoid issues with the config system and object creation. The primitives pocketpy provides for communication and tasking between VMs are very minimal so I eventually want to add some friendly wrappers on top. Those may be modeled after the task worker pool. |
Thinking about this more, I don't think Maybe something like |
- Renamed miniscroll to pyscroll - Renamed Mini class to Base
|
I renamed |
- Add orx.input.has_new_status (orxInput_HasNewStatus) - orx.object.fire_trigger returns event success/failure status - pyscroll now more closely matches Scroll's config-based input trigger handling
|
d2afc53 brings pyscroll's config-based input handling more inline with what Scroll supports, including |
- Fix pyscroll input trigger event handling - Add orx.input.get_next binding orxInput_GetNext
iarwain
left a comment
There was a problem hiding this comment.
Thanks for the changes. Let's merge it post 1.17 release!
- De-conflict Scroll and pyscroll input handling by not registering the Logo object with pyscroll in a fresh init project with both +python and +scroll - Refactor pyscroll to be a bit cleaner and more friendly to common Python idioms - Added and expanded doc comments for pyscroll - Update to latest pocketpy
|
Thank you for the review! I pushed that refactor that I mentioned on discord in d72da6a. I'm happy to have this merged after the 1.17 release! |
There was a problem hiding this comment.
It's super minor, but given this already follows most of orx's coding conventions, would you mind going the extra mile and prefixing function parameters with _?
For example:
orxCHAR *orxPy_ReadSourceFromMain(const orxSTRING _zPath, int *_piDataSize)
There was a problem hiding this comment.
Certainly - apologies for the inconsistency for those function parameter names. I'll fix that and push an updated commit in a few minutes.
There was a problem hiding this comment.
Style fixes are in 687f589. There were a lot! Thank you for calling the naming mismatch out. I'd like this to feel like orx as much as possible.
There was a problem hiding this comment.
Thanks for taking care of it!
- Fix function parameter names to match orx's coding conventions
- Bugfix in pyscroll's shader parameter handling
This adds Python bindings to orx through an extension, using https://pocketpy.dev/ to provide a fully embedded Python runtime.
To quote the pocketpy website:
Extension features
+pythonoption to orx's project init toolinit,update, andexitengine callbacks and several event handlers.orx.*module namespace, including:orxObject_*inorx.objectorxConfig_*inorx.configorxInput_*inorx.inputorxVector_*inorx.vectororxClock_ComputeDTinorx.clockorxCommand_Evaluate(WithGUID)inorx.commandorxLOGasorx.log*.pyifiles for each of the provided Python modules to improve support for editor integration like VSCode's Python extension and support Python type checkers like pyright and mypyPython.Execcommand to run Python code from commandspyscrollmodule to bind per-object behaviorsorxResourcemodule for loading*.pysources. Python modules can be included from any supported resource location, including bundles when thebundleextension is used.