Feature/free threaded

[serge-sans-paille]

No description provided.

[serge-sans-paille]

@rgommers does that look good to you?

[rgommers]

That was fast! CI and C++ code look good, will test with SciPy to make sure it works.

Main question I have is about the lack of a --freethreading-compatible CLI flag to pythran, to make this opt-in (as I suggested in gh-2258). I'm fairly sure that for SciPy we don't have Pythran kernels that are thread-unsafe, but it is possible to write for example a function that modifies its input arguments, right? That kind of mutation tends to be implicitly protected by the GIL, and without it may have race conditions. That is the reason that Cython and F2py make adding Py_MOD_GIL_NOT_USED in generated C code opt-in, so the user can think about this.

[rgommers]

The C++ code does work as advertised in SciPy. And testing this allowed me to get past the Pythran modules and find the next place to go fix the same thing, so this testing was already quite helpful.

[rgommers]

Just stumbled across https://pythran.readthedocs.io/en/latest/MANUAL.html#thread-safety in the user manual. Is that still up to date, and if so should the generated code default to the behavior enabled by the -DTHREAD_SAFE_REF_COUNT flag under a free-threaded interpreter?

[rgommers]

I think I actually like that the added flag in the .cfg file defaults to true. That makes it quite easy to start using this, and the user with odd code that turns out to be unsafe can still opt out.

[serge-sans-paille]

On Wed, Nov 27, 2024 at 10:52:51AM -0800, Ralf Gommers wrote: Just stumbled across [1]https://pythran.readthedocs.io/en/latest/MANUAL.html# thread-safety in the user manual. Is that still up to date, and if so should the generated code default to the behavior enabled by the -DTHREAD_SAFE_REF_COUNT flag under a free-threaded interpreter?

I don't think so. THREAD_SAFE_REF_COUNT is for pythran-owned reference counter on a shared pointer, it does not interact with any python memory.

[rgommers]

Okay, testing done - all happy! Perhaps this still needs an entry in the docs, so the new option shows up at https://pythran.readthedocs.io/en/latest/MANUAL.html#customizing-your-pythranrc? Other than that it LGTM. 🚀

[paugier]

but it is possible to write for example a function that modifies its input arguments

Just a question about this remark. It seems to me that it is quite common for efficient numerical kernel to modify inplace numpy arrays (a very simple example here).

Will I have to call pythran with --config backend.freethreading_compatible=false in this case ?

[rgommers]

@paugier only if vector_product in your example is a public function in a package, because then you have to guard against the user sharing the input array over multiple threads when they use the threading module in their own code. Such code is rare though, and typically already buggy (there's a reason you have a WARNING comment there). If it's within your own code base and you don't use threading yourself, that code should be safe.

[rgommers]

Thanks again @serge-sans-paille! Do you have any plans for a new release with this included? For the first RC for SciPy 1.15.0 I can point at master, but ideally we'd have this for the final release in January. (if not possible, not the end of the world though)

[rgommers]

@serge-sans-paille friendly ping on the question above. Also, will this go into 0.18.0 or can it be part of 0.17.1?

[serge-sans-paille]

Can be part of a 0.17.1. Do you think we can sleak in #2257 as well?

[rgommers]

Great! I think I can get something done for gh-2257. Let me at least try to get a PR up this week that works with SciPy; if review shows it's nontrivial to get it right then we probably should aim for 0.18.0 instead.