I tried to use the multiproc capability in the clean3d (although the code is at the moment fast enough that this is not really needed).
I noticed that even if I set --ncores 10 I can see only 2 jobs running (probably 1 worker and the main code). If I don't set --chunk I assumes it uses 1 and it goes extremely slow, probably due to overhead in the multiproc. When I use --chunk 1000 I somewhat reach the same performance of the non-multithread case. So in my case I could not really spawn the desired number of threads.
I also would:
- add a log that states that the code is in "multiproc" mode with N cpus and M chunksize
- add to the doc that one has to set the chunk size to a reasonably high number
I tried to use the multiproc capability in the clean3d (although the code is at the moment fast enough that this is not really needed).
I noticed that even if I set --ncores 10 I can see only 2 jobs running (probably 1 worker and the main code). If I don't set --chunk I assumes it uses 1 and it goes extremely slow, probably due to overhead in the multiproc. When I use --chunk 1000 I somewhat reach the same performance of the non-multithread case. So in my case I could not really spawn the desired number of threads.
I also would: