Is juliacall supporting python multiprocessing?

[markNZed]

I would like to use juliacall in a python script that is using multiprocessing. So far I get seg faults. Is there an example of how to do this ? Thanks

[cjdoris]

I've never tried multiprocessing but there's no obvious reason it shouldn't work. Can you please post some minimal code that crashes and the full error message?

[markNZed]

It does seem to be working. I needed to import inside each of the worker process e.g.

    def initialize(self):
        from juliacall import Main as jl
        jl.seval("using Pkg; Pkg.activate(\"..\"); Pkg.instantiate(); using ARTime")

[cjdoris]

Great!

[VietTralala]

For me this does not work. I can call jl.seval from the main process however in the worker process I always get the following error and the worker process crashes

python: /buildworker/worker/package_linux64/build/src/debuginfo.cpp:1634: void register_eh_frames(uint8_t*, size_t): Assertion `end_ip != 0' failed.

Without the multiprocessing Pool I can call the julia script just fine. What can I do to further debug this?

Here is my minimal example:

import os
os.environ['PYTHON_JULIAPKG_EXE'] = "/home/user/.juliaup/bin/julia"
os.environ['PYTHON_JULIAPKG_OFFLINE'] = 'yes'
os.environ['PYTHON_JULIAPKG_PROJECT'] = '/home/user/julia/environments/v1.6/'

from juliacall import Main as jl, convert as jlconvert

from multiprocessing import Pool
from tqdm import tqdm

import ipdb

def init_worker():
    import os
    os.environ['PYTHON_JULIAPKG_EXE'] = "/home/user/juliaup/bin/julia"
    os.environ['PYTHON_JULIAPKG_OFFLINE'] = 'yes'
    os.environ['PYTHON_JULIAPKG_PROJECT'] = '/home/user/.julia/environments/v1.6/'

    from juliacall import Main as jl, convert as jlconvert

    print('in init_worker()...')
    jl.seval('using Pkg')
    jl.seval('Pkg.status()')
    print('...done')

def compute(jobid):
    print(f'in main({jobid})...')

    jl.seval('include("test_julia_simple.jl")')
    print('...done')
    return

def main():
    njobs = 10
    #start pool with init_worker() as initializer
    
    with Pool(2, initializer=init_worker) as p, tqdm(total=njobs) as pbar:
        res = []
        for jid in range(njobs):
            res.append(p.apply_async(compute, (jid,)))
        for r in res:
            r.get()
            pbar.update(1)


if __name__ == "__main__":
    main()

And the julia script test_julia_simple.jl

for i in 1:10
    println(i)
end
1+2

---

additional info:

$ python --version
Python 3.9.7
$ pip freeze | grep julia
juliacall==0.9.10
juliapkg==0.1.9

$ julia --version
The latest version of Julia in the `1.6` channel is 1.6.7+0.x64.linux.gnu. You currently have `1.6.6+0~x64` installed. Run:                           

  juliaup update

to install Julia 1.6.7+0.x64.linux.gnu and update the `1.6` channel to that version.                                                                  
julia version 1.6.6

not sure if this is related but the error message is nearly identical
JuliaLang/julia#44969

[cjdoris]

I imagine you need to import juliacall within the compute function too.

[VietTralala]

I imagine you need to import juliacall within the compute function too.

I tried what you suggested but the script still fails with the same error message.

# the new compute function with inner imports:
def compute(jobid):
    import os
    os.environ['PYTHON_JULIAPKG_EXE'] = "/system/user/tran/.juliaup/bin/julia"
    os.environ['PYTHON_JULIAPKG_OFFLINE'] = 'yes'
    os.environ['PYTHON_JULIAPKG_PROJECT'] = '/system/user/tran/.julia/environments/v1.6/'
    from juliacall import Main as jl, convert as jlconvert

    print(f'in main({jobid})...')

    jl.seval('include("test_julia_simple.jl")')
    print('...done')
    return

[markNZed]

I did get this working but it was quite some time ago. One option would be to strip down what I did and use that as a "working" starting point to figure out what the difference in your situation is. You can find the details on the package I built at https://github.com/markNZed/ARTimeNAB.jl and the code dealing with difference processes is over here https://github.com/numenta/NAB/blob/master/nab/detectors/ARTime/ARTime_detector.py

[ifmelchor]

see https://stackoverflow.com/questions/74438358/error-while-targeting-a-julia-function-into-multiprocessing-process-of-python/74490051#74490051