gh-149101: Implement PEP 788

[ZeroIntensity]

Hugo has graciously given me permission to backport this if we don't make the May 5th deadline, but let's try to get this done in time!

I will write a full tutorial and migration guide once this is merged; I want to first make sure that this lands before the beta freeze.

• Issue: Implement PEP 788 -- Protecting the C API from Interpreter Finalization #149101

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #32469002 | 📁 Comparing bc78c10 against main (40dc61a)

  🔍 Preview build  

25 files changed · ± 25 modified

± Modified

• contents.html
• download.html
• genindex-A.html
• genindex-B.html
• genindex-C.html
• genindex-E.html
• genindex-H.html
• genindex-I.html
• genindex-M.html
• genindex-P.html
• and 15 more...

[encukou]

Thanks for adding these!

I'll send notes for Doc/ now; code review coming up.

[encukou]

Maybe use the term for “have to wait”?

Suggested change

	When a guard is held, a thread attempting to finalize the interpreter will
	have to wait until the guard is closed before threads can be blocked.
	When a guard is held, a thread attempting to finalize the interpreter will
	block until the guard is closed before starting the finalization.

[ZeroIntensity]

Not opposed to the change, but I haven't been using "the" before "finalization" in other places. Was that intentional?

[encukou]

Is this relevant for the user? Allocation looks like an implementation detail here.

[ZeroIntensity]

Not sure -- I added it thinking about debugging (e.g., if you're trying to find a missing PyInterpreterGuard_Close call, you can just look for unfreed PyInterpreterGuard * pointers).

[encukou]

Consider not repeating complex information; it can leave the reader wondering whether it actually is the same.
Only repeat the warning.

Suggested change

	The interpreter referenced by *view* will be implicitly guarded. The
	guard will be released upon the corresponding :c:func:`PyThreadState_Release`
	call.
	On success, this function will return the thread state that was previously attached.
	If no thread state was previously attached, this returns a non-``NULL`` sentinel
	value. The behavior of whether this function creates a thread state is
	equivalent to that of :c:func:`PyThreadState_Ensure`.
	To visualize, function is roughly equivalent to the following:
	.. code-block:: c
	PyThreadState *
	PyThreadState_EnsureFromView(PyInterpreterView *view)
	{
	assert(view != NULL);
	PyInterpreterGuard *guard = PyInterpreterGuard_FromView(view);
	if (guard == NULL) {
	return NULL;
	}
	PyThreadState *tstate = PyThreadState_Ensure(guard);
	if (tstate == NULL) {
	PyInterpreterGuard_Close(guard);
	return NULL;
	}
	if (tstate->guard == NULL) {
	tstate->guard = guard;
	} else {
	PyInterpreterGuard_Close(guard);
	}
	return tstate;
	}
	On success, the interpreter referenced by *view* will be implicitly guarded;
	the guard will be released upon the corresponding :c:func:`PyThreadState_Release`
	call.
	Otherwise, the behavior and return value are the same as for
	:c:func:`PyThreadState_Ensure`.
	Note that the returned pointer is not necessarily a valid
	:c:type:`!PyThreadState` pointer.

[encukou]

Again, leave implementation details to the implementation -- and the PEP :)

Suggested change

	This function will decrement an internal counter on the attached thread state. If
	this counter ever reaches below zero, this function emits a fatal error (via
	:c:func:`Py_FatalError`).
	If the attached thread state is owned by ``PyThreadState_Ensure``, then the
	attached thread state will be deallocated and deleted upon the internal counter
	reaching zero. Otherwise, nothing happens when the counter reaches zero.
	If *tstate* is non-``NULL``, it will be attached upon returning.
	If *tstate* indicates that no prior thread state was attached, there will be
	no attached thread state upon returning.
	To visualize, this function is roughly equivalent to the following:
	.. code-block:: c
	void
	PyThreadState_Release(PyThreadState *old_tstate)
	{
	PyThreadState *current_tstate = PyThreadState_Get();
	assert(old_tstate != NULL);
	assert(current_tstate != NULL);
	assert(current_tstate->ensure_counter > 0);
	if (--current_tstate->ensure_counter > 0) {
	// There are remaining PyThreadState_Ensure() calls
	// for this thread state.
	return;
	}
	assert(current_tstate->ensure_counter == 0);
	if (old_tstate == NO_TSTATE_SENTINEL) {
	// No thread state was attached prior the PyThreadState_Ensure()
	// call. So, we can just destroy the current thread state and return.
	assert(current_tstate->owned_by_pythreadstate_ensure);
	PyThreadState_Clear(current_tstate);
	PyThreadState_DeleteCurrent();
	return;
	}
	if (tstate->guard != NULL) {
	PyInterpreterGuard_Close(tstate->guard);
	return;
	}
	if (tstate->owned_by_pythreadstate_ensure) {
	// The attached thread state was created by the initial PyThreadState_Ensure()
	// call. It's our job to destroy it.
	PyThreadState_Clear(current_tstate);
	PyThreadState_DeleteCurrent();
	}
	PyThreadState_Swap(old_tstate);
	}

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @ZeroIntensity for commit bc78c10 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F149116%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.