Support for Python 3.14 Free Threading (no GIL)

[echo-cho-ho-o]

What is the feature or improvement you would like to see?
Can I get a timeline for supporting Python 3.14 Free Threading (--disable-gil)?

Use cases for the feature
igraph is a dependency in our project that has been migrated to Python 3.14t

References
https://astral.sh/blog/python-3.14#free-threaded-python

[ntamas]

Unfortunately the internal C igraph library is not thread-safe so it's hard to make the Python interface thread-safe, which would be the first step towards officially supporting free-threaded Python.

There are two issues with the C core that need attention:

1. There's a global stack of temporarily allocated objects that igraph uses to clean up after itself when an error happens (see details here). igraph itself can be configured in a way that it uses thread-local storage for the error stack so that would probably solve the issue but no one has really tested it yet.
2. Running two igraph functions concurrently in two threads such that they both modify the same graph is definitely not going to work. This could probably be solved by having a mutex in each of the constructed Graph objects on the Python side and then locking the mutex when an igraph function is called on that graph (and unlocking it when the call returns), but since the code of the entire Python interface is hand-written, it would be very tedious and error-prone to implement this by hand.

All in all, I don't really see an easy way to add support for free-threaded Python in my spare time. There are two ways ahead:

• We could try to apply for funding from somewhere to add support for free-threaded Python.
• We could try to apply for funding to focus our resources on the development of https://github.com/igraph/python-igraph-ctypes , which would be an alternative Python interface to igraph with full type annotations, no C glue layer (we use ctypes instead), zero-copy NumPy support for data exchange and mostly auto-generated code. The fact that we auto-generate large parts of the bindings would mean that we could add support for locking by modifying the code generator instead of having to modify each of the exposed functions one by one.

[ilan-gold]

Is there an appetite for something more focused?

Specifically, I am interested in https://igraph.org/python/api/0.9.11/igraph.Graph.html#community_leiden with multiple resolutions. Do you see any route forward to being able to parallelize this operation? Of course just using a ThreadPoolExecutor doesn't cut it because, as you said, there is the concurrent write issue on the graph itself which prevents releasing the GIL (even those AFAIK, this operation itself is "safe" from a mutability perspective).

Is there any way we could pass in a list of numbers to resolution_parameter perhaps and have each resolution done in parallel? It seems like this is a fairly common use case - searching over multiple resolution. Thanks!

[szhorvat]

This can't be done just for one function. What is not thread-safe is the error handling mechanism used by igraph, which also include the interruption mechanism (i.e. that you can stop a calculation without breaking into internal state).

Making this thread safe would of course be very valuable, but it's not a simple thing to do, and in fact it's not even clear to me how to best do it. It needs to be researched and designed carefully. There is in fact no library of comparable complexity that I know of that supports both parallelization and interruptability. The only way we could attack this problem is if we manage to obtain funding specifically for this project.

Is there any way we could pass in a list of numbers to resolution_parameter perhaps and have each resolution done in parallel?

Personally, I rarely use igraph from Python, so I can't give a certain answer, but I'm wondering if the multiprocessing package can help. Any solution that runs several processes in parallel can work in principle. The question is how to share the graph object between the processes and whether transferring the data between processes, in whichever way multiprocessing does it, is too costly to make this worthwhile. Another drawback is that there will need to be as many copies of the graph in memory as the number of processes, so memory use will be high. However, if your goal is to run Leiden with different resolution parameters, this may not be an issue. Parallelization is already worthwhile for much smaller graphs than what would cause a memory issue.

I will say that I did do what you are looking for in Mathematica (run community detection with different resolutions), which uses distinct processes for parallelization, and can transfer the graph efficiently enough between the processes that it's absolutely worthwhile to parallelize.

If you try this out have have good success with multiprocessing, please do report back.

[ilan-gold]

Thanks for the reply!

Could you share your experience there with Mathematica? I naively tried this with multiprocessing and only got a 2x speedup, independent of process count. I also tried creating rust bindings to the C library and unsafeing my way to parallelized runnings of leiden (interrupt be damned) but also only got a ~2X speedup, also independent of thread count.

So I'm curious what your experience was and if this sounds familiar. Memory bottleneck maybe? My graphs are generally ~10-75MB.