Skip to content

Revise Python and JVM docs audience split #779

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
graalvmbot wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Revise Python and JVM docs audience split #779

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/site/01-jvm-developers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -366,6 +366,12 @@ try (Context context = GraalPyResources.contextBuilder().build()) {
<div class="guides__topics">Use GraalPy with Maven or Gradle in a Java SE Application</div>
</a>
</div>
<div class="guides__card">
<img src='{{ "/assets/img/downloads/miscellaneous-book.svg" | relative_url }}' alt="book icon">
<a href="https://github.com/graalvm/graal-languages-demos/blob/main/graalpy/graalpy-freeze-dependencies-guide/" target="_blank">
<div class="guides__topics">Freeze Transitive Python Dependencies for Reproducible Builds</div>
</a>
</div>
<div class="guides__card">
<img src='{{ "/assets/img/downloads/miscellaneous-book.svg" | relative_url }}' alt="book icon">
<a href="https://github.com/graalvm/graal-languages-demos/blob/main/graalpy/graalpy-spring-boot-guide/" target="_blank">
Expand All @@ -380,6 +386,12 @@ try (Context context = GraalPyResources.contextBuilder().build()) {
<div class="guides__topics">Use GraalPy with Micronaut</div>
</a>
</div>
<div class="guides__card">
<img src='{{ "/assets/img/downloads/miscellaneous-book.svg" | relative_url }}' alt="book icon">
<a href="https://github.com/graalvm/graal-languages-demos/blob/main/graalpy/graalpy-custom-venv-guide/" target="_blank">
<div class="guides__topics">Manually Install Python Packages</div>
</a>
</div>
<div class="guides__card">
<img src='{{ "/assets/img/downloads/miscellaneous-book.svg" | relative_url }}' alt="book icon">
<a href="https://github.com/graalvm/graal-languages-demos/blob/main/graalpy/graalpy-native-extensions-guide/" target="_blank">
Expand All @@ -400,6 +412,12 @@ try (Context context = GraalPyResources.contextBuilder().build()) {
<h3 class="truffle__subtitle">Demos</h3>
<div class="guides__row">
<div class="guides__column">
<div class="guides__card">
<img src='{{ "/assets/img/downloads/miscellaneous-book.svg" | relative_url }}' alt="book icon">
<a href="https://github.com/graalvm/graal-languages-demos/tree/main/graalpy/graalpy-scripts-debug-guide" target="_blank">
<div class="guides__topics">Debug Python Scripts in VS Code with GraalPy</div>
</a>
</div>
<div class="guides__card">
<img src='{{ "/assets/img/downloads/miscellaneous-book.svg" | relative_url }}' alt="book icon">
<a href="https://github.com/graalvm/graal-languages-demos/tree/main/graalpy/graalpy-openai-starter/" target="_blank">
Expand Down
52 changes: 0 additions & 52 deletions docs/site/01-python-developers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,12 +249,6 @@ docker run --rm ghcr.io/graalvm/graalpy-community:{{ site.language_version }} py
<h3 id="guides" class="truffle__subtitle">Guides</h3>
<div class="guides__row">
<div class="guides__column">
<div class="guides__card">
<img src='{{ "/assets/img/downloads/miscellaneous-book.svg" | relative_url }}' alt="book icon">
<a href="https://github.com/graalvm/graal-languages-demos/blob/main/graalpy/graalpy-freeze-dependencies-guide/" target="_blank">
<div class="guides__topics">Freeze Transitive Python Dependencies for Reproducible Builds</div>
</a>
</div>
<div class="guides__card">
<img src='{{ "/assets/img/downloads/miscellaneous-book.svg" | relative_url }}' alt="book icon">
<a href="{{ '/python-developers/docs/' | relative_url }}">
Expand All @@ -275,52 +269,6 @@ docker run --rm ghcr.io/graalvm/graalpy-community:{{ site.language_version }} py
<div class="guides__topics">Use GraalPy as a Standalone Python Runtime</div>
</a>
</div>
<div class="guides__card">
<img src='{{ "/assets/img/downloads/miscellaneous-book.svg" | relative_url }}' alt="book icon">
<a href="https://github.com/graalvm/graal-languages-demos/blob/main/graalpy/graalpy-custom-venv-guide/" target="_blank">
<div class="guides__topics">Manually Install Python Packages</div>
</a>
</div>
</div>
</div>
</div>
</div>
</div>
</section>

<section class="boxes">
<div class="wrapper">
<div class="guides">
<div class="container guides-box build all">
<h3 class="truffle__subtitle">Demos</h3>
<div class="guides__row">
<div class="guides__column">
<div class="guides__card">
<img src='{{ "/assets/img/downloads/miscellaneous-book.svg" | relative_url }}' alt="book icon">
<a href="https://github.com/graalvm/graal-languages-demos/tree/main/graalpy/graalpy-scripts-debug-guide" target="_blank">
<div class="guides__topics">Debug Python Scripts in VS Code with GraalPy</div>
</a>
</div>
<div class="guides__card">
<img src='{{ "/assets/img/downloads/miscellaneous-book.svg" | relative_url }}' alt="book icon">
<a href="https://github.com/graalvm/graal-languages-demos/tree/main/graalpy/graalpy-jbang-qrcode" target="_blank">
<div class="guides__topics">Generate QR Codes with the Python qrcode Package</div>
</a>
</div>
</div>
<div class="guides__column">
<div class="guides__card">
<img src='{{ "/assets/img/downloads/miscellaneous-book.svg" | relative_url }}' alt="book icon">
<a href="https://github.com/graalvm/graal-languages-demos/tree/main/graalpy/graalpy-openai-starter" target="_blank">
<div class="guides__topics">Run the Official OpenAI Python SDK with GraalPy</div>
</a>
</div>
<div class="guides__card">
<img src='{{ "/assets/img/downloads/miscellaneous-book.svg" | relative_url }}' alt="book icon">
<a href="https://github.com/graalvm/graal-languages-demos/tree/main/graalpy/graalpy-custom-venv-guide" target="_blank">
<div class="guides__topics">Install Packages in a Custom GraalPy Virtual Environment</div>
</a>
</div>
</div>
</div>
</div>
Expand Down
1 change: 1 addition & 0 deletions docs/site/02-jvm-developers-docs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ These guides cover everything you need to know:
{% gfm_docs ../user/Embedding-Getting-Started.md %}
{% gfm_docs ../user/Embedding-Build-Tools.md %}
{% gfm_docs ../user/Embedding-Permissions.md %}
{% gfm_docs ../user/Embedding-Native-Extensions.md %}
{% gfm_docs ../user/Interoperability.md %}
{% gfm_docs ../user/Native-Images-with-Python.md %}
{% gfm_docs ../user/Python-on-JVM.md %}
Expand Down
9 changes: 1 addition & 8 deletions docs/site/02-python-developers-docs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,19 +19,12 @@ You get better performance, the ability to compile to native binaries, and acces
These guides cover everything you need to know:

{% gfm_docs ../user/Standalone-Getting-Started.md %}
{% gfm_docs ../user/Python-Runtime.md %}
{% gfm_docs ../user/Python-Standalone-Applications.md %}
{% gfm_docs ../user/Native-Extensions.md %}
{% gfm_docs ../user/Interoperability.md %}
{% gfm_docs ../user/Performance.md %}
{% gfm_docs ../user/Tooling.md %}

<h3 id="python-context-options">
<a href="#python-context-options" class="anchor-link">Python Context Options</a>
</h3>
Below are the options you can set on contexts for GraalPy.
{% python_options ../../graalpython/com.oracle.graal.python/src/com/oracle/graal/python/runtime/PythonOptions.java %}

{% gfm_docs ../user/Test-Tiers.md %}
{% gfm_docs ../user/Troubleshooting.md %}

{% copy_assets ../user/assets %}
45 changes: 45 additions & 0 deletions docs/user/Embedding-Native-Extensions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Embedding Limitations for Native Extensions

Python native extensions run by default as native binaries, with full access to the underlying system.
This has a few implications:

1. Native code is entirely unrestricted and can circumvent any security protections Truffle or the JVM may provide.
2. Native data structures are not subject to the Java GC and the combination of them with Java data structures may lead to increased memory pressure or memory leaks.
3. Native libraries generally cannot be loaded multiple times into the same process, and they may contain global state that cannot be safely reset.

## Full Native Access

The Context API allows you to set options such as `allowIO`, `allowHostAccess`, `allowThreads`, and more on created contexts.
To use Python native extensions on GraalPy, the `allowNativeAccess` option must be set to `true`, but this opens the door to full native access.
This means that while Python code may be denied access to the host file system, thread or subprocess creation, and more, the native extension is under no such restriction.

## Memory Management

Python C extensions, like the CPython reference implementation, use reference counting for memory management.
This is fundamentally incompatible with JVM GCs.

Java objects may end up being referenced from native data structures that the JVM cannot trace, so to avoid crashing, GraalPy keeps such Java objects strongly referenced.
To avoid memory leaks, GraalPy implements a cycle detector that regularly traces references between Java objects and native objects that have crossed between the two worlds and cleans up strong references that are no longer needed.

On the other side, reference-counted native extension objects may end up being referenced from Java objects, and in this case GraalPy bumps their reference count to make them unreclaimable.
Any such references to native extension objects are registered with a `java.lang.ref.WeakReference`, and when the JVM GC has collected the owning Java object, the reference count of the native object is reduced again.

Both of these mechanisms together mean there is additional delay between objects becoming unreachable and their memory being reclaimed when compared to the CPython implementation.
This can manifest in increased memory usage when running C extensions.
You can tweak the context options `python.BackgroundGCTaskInterval`, `python.BackgroundGCTaskThreshold`, and `BackgroundGCTaskMinimum` to mitigate this.
They control the minimum interval between cycle detections, how much RSS memory must have increased since the last time to trigger the cycle detector, and the absolute minimum RSS under which no cycle detection should be done.
You can also manually trigger the detector with the Python `gc.collect()` call.

## Multi-Context and Native Libraries

Using C extensions in multiple contexts is only possible on Linux for now, and many C extensions still have issues in this mode.
You should test your applications thoroughly if you want to use this feature.
There are many possibilities for native code to sidestep the library isolation through other process-wide global state, corrupting the state and leading to incorrect results or crashing.
The implementation also relies on `venv` to work, even if you are not using external packages.

To support creating multiple GraalPy contexts that access native modules within the same JVM or Native Image, GraalPy isolates them from each other.
The current strategy for this is to copy the libraries and modify them such that the dynamic library loader of the operating system will isolate them.
To do this, all GraalPy contexts in the same process, not just those in the same engine, must set the `python.IsolateNativeModules` option to `true`.
You should test your applications thoroughly if you want to use this feature, as there are many possibilities for native code to sidestep the library isolation through other process-wide global state.

For more details on this, see [our implementation details](https://github.com/oracle/graalpython/blob/master/docs/contributor/IMPLEMENTATION_DETAILS.md#c-extension-copying).
2 changes: 1 addition & 1 deletion docs/user/Embedding-Permissions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,4 +128,4 @@ Context context = Context.newBuilder("python")
Python native extensions run by default as native binaries with full access to the underlying system.
This means they bypass the security controls described above.

For more information about limitations when embedding native extensions, see [Embedding limitations](Native-Extensions.md#embedding-limitations).
For more information about limitations when embedding native extensions, see [Embedding limitations](Embedding-Native-Extensions.md).
2 changes: 2 additions & 0 deletions docs/user/Interoperability.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ This means that you can use other languages' objects and functions directly from
This interoperability works in both directions.
Python can call other languages, and other languages can call Python code.

> **Note for Python users:** The Java interoperability sections below require a JVM-based GraalPy runtime with the relevant Java classes on the classpath. The later `polyglot` sections describe the more general cross-language APIs for working with other Graal languages.

## Call Java from Python

Java is the host language of the JVM and runs the GraalPy interpreter itself.
Expand Down
46 changes: 0 additions & 46 deletions docs/user/Native-Extensions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,49 +7,3 @@ Packages that use the native API must be built and installed with GraalPy, and t
For best results, it is crucial that you only use the `pip` command that comes preinstalled in GraalPy virtual environments to install packages.
The version of `pip` shipped with GraalPy applies additional patches to packages upon installation to fix known compatibility issues and it is preconfigured to use an additional repository from graalvm.org where we publish a selection of prebuilt wheels for GraalPy.
Please do not update `pip` or use alternative tools such as `uv`.

## Embedding limitations

Python native extensions run by default as native binaries, with full access to the underlying system.
This has a few implications:

1. Native code is entirely unrestricted and can circumvent any security protections Truffle or the JVM may provide.
2. Native data structures are not subject to the Java GC and the combination of them with Java data structures may lead to increased memory pressure or memory leaks.
3. Native libraries generally cannot be loaded multiple times into the same process, and they may contain global state that cannot be safely reset.

### Full Native Access

The Context API allows to set options such as `allowIO`, `allowHostAccess`, `allowThreads` and more on the created contexts.
To use Python native extensions on GraalPy, the `allowNativeAccess` option must be set to true, but this opens the door to full native access.
This means that while Python code may be denied access to the host file system, thread- or subprocess creation, and more, the native extension is under no such restriction.

### Memory Management

Python C extensions, like the CPython reference implementation, use reference counting for memory management.
This is fundamentally incompatible with JVM GCs.

Java objects may end up being referenced from native data structures which the JVM cannot trace, so to avoid crashing, GraalPy keeps such Java objects strongly referenced.
To avoid memory leaks, GraalPy implements a cycle detector that regularly traces references between Java objects and native objects that have crossed between the two worlds and cleans up strong references that are no longer needed.

On the other side, reference-counted native extension objects may end up being referenced from Java objects, and in this case GraalPy bumps their reference count to make them unreclaimable.
Any such references to native extension objects are registered with a `java.lang.ref.WeakReference` and when the JVM GC has collected the owning Java object, the reference count of the native object is reduced again.

Both of these mechanisms together mean there is additional delay between objects becoming unreachable and their memory being reclaimed when compared to the CPython implementation.
This can manifest in increased memory usage when running C extensions.
You can tweak the Context options `python.BackgroundGCTaskInterval`, `python.BackgroundGCTaskThreshold`, and `BackgroundGCTaskMinimum` to mitigate this.
They control the minimum interval between cycle detections, how much RSS memory must have increased since the last time to trigger the cycle detector, and the absolute minimum RSS under which no cycle detection should be done.
You can also manually trigger the detector with the Python `gc.collect()` call.

### Multi-Context and Native Libraries

Using C extensions in multiple contexts is only possible on Linux for now, and many C extensions still have issues in this mode.
You should test your applications thoroughly if you want to use this feature.
There are many possibilities for native code to sidestep the library isolation through other process-wide global state, corrupting the state and leading to incorrect results or crashing.
The implementation also relies on `venv` to work, even if you are not using external packages.

To support creating multiple GraalPy contexts that access native modules within the same JVM or Native Image, we need to isolate them from each other.
The current strategy for this is to copy the libraries and modify them such that the dynamic library loader of the operating system will isolate them for us.
To do this, all GraalPy contexts in the same process (not just those in the same engine!) must set the `python.IsolateNativeModules` option to `true`.
You should test your applications thoroughly if you want to use this feature, as there are many possiblities for native code to sidestep the library isolation through other process-wide global state.

For more details on this, see [our implementation details](https://github.com/oracle/graalpython/blob/master/docs/contributor/IMPLEMENTATION_DETAILS.md#c-extension-copying).
Loading
Loading