Skip to content

Memory errors #766

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
oesteban opened this issue Oct 17, 2017 · 20 comments · Fixed by #788
Closed

Memory errors #766

oesteban opened this issue Oct 17, 2017 · 20 comments · Fixed by #788
Labels
bug memory
Milestone
Phase II

Comments

@oesteban
Copy link
Member

oesteban commented Oct 17, 2017 •
edited
Loading

The node that crashed the most times was mri_coreg followed by autorecon. Worth looking into the requirements these nodes need.

Node: _autorecon31
Working directory: /scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC05_wf/anat_preproc_wf/surface_recon_wf/autorecon_resume_wf/autorecon3/mapflow/
_autorecon31

Node inputs:

T1_files = <undefined>
T2_file = <undefined>
args = <undefined>
big_ventricles = <undefined>
brainstem = <undefined>
directive = autorecon3
environ = {}
expert = <undefined>
flags = <undefined>
hemi = rh
hippocampal_subfields_T1 = <undefined>
hippocampal_subfields_T2 = <undefined>
hires = <undefined>
ignore_exception = False
mprage = <undefined>
mri_aparc2aseg = <undefined>
mri_ca_label = <undefined>
mri_ca_normalize = <undefined>
mri_ca_register = <undefined>
mri_edit_wm_with_aseg = <undefined>
mri_em_register = <undefined>
mri_fill = <undefined>
mri_mask = <undefined>
mri_normalize = <undefined>
mri_pretess = <undefined>
mri_remove_neck = <undefined>
mri_segment = <undefined>
mri_segstats = <undefined>
mri_tessellate = <undefined>
mri_watershed = <undefined>
mris_anatomical_stats = <undefined>
mris_ca_label = <undefined>
mris_fix_topology = <undefined>
mris_inflate = <undefined>
mris_make_surfaces = <undefined>
mris_register = <undefined>
mris_smooth = <undefined>
mris_sphere = <undefined>
mris_surf2vol = <undefined>
mrisp_paint = <undefined>
openmp = 8
parallel = <undefined>
subject_id = sub-MSC05
subjects_dir = /scratch/03763/oesteban/fmriprep-phase1/out/ds000224/freesurfer
talairach = <undefined>
terminal_output = <undefined>
use_T2 = True
xopts = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 51, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 407, in run
    self._run_interface()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 517, in _run_interface
    self._result = self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1088, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1653, in _run_interface
    runtime.dependencies = get_dependencies(executable_name, runtime.environ)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1467, in get_dependencies
    env=environ)
  File "/usr/local/miniconda/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/usr/local/miniconda/lib/python3.6/subprocess.py", line 1260, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory
@oesteban oesteban added the bug label Oct 17, 2017
@oesteban oesteban added this to the Phase II milestone Oct 17, 2017
@oesteban
Copy link
Member Author 

Node: fmriprep_wf.single_subject_MSC08_wf.func_preproc_ses_func01_task_motor_run_02_wf.bold_reg_wf.bbreg_wf.mri_coreg
Working directory: /scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC08_wf/func_preproc_ses_func01_task_motor_run_02_wf/bold_reg_wf/bbreg_wf/mri_co
reg

Node inputs:

args = <undefined>
brute_force_limit = <undefined>
brute_force_samples = <undefined>
compress_report = auto
conform_reference = <undefined>
dof = 9
environ = {'SUBJECTS_DIR': '/opt/freesurfer/subjects'}
ftol = 0.0001
generate_report = True
ignore_exception = False
initial_rotation = <undefined>
initial_scale = <undefined>
initial_shear = <undefined>
initial_translation = <undefined>
linmintol = 0.01
max_iters = <undefined>
no_brute_force = <undefined>
no_coord_dithering = <undefined>
...skipping...
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 51, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 407, in run
    self._run_interface()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 517, in _run_interface
    self._result = self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/freesurfer/base.py", line 162, in run
    return super(FSCommand, self).run(**inputs)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1088, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/common/report.py", line 53, in _run_interface
    ReportCapableInterface, self)._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1653, in _run_interface
    runtime.dependencies = get_dependencies(executable_name, runtime.environ)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1467, in get_dependencies
    env=environ)
  File "/usr/local/miniconda/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/usr/local/miniconda/lib/python3.6/subprocess.py", line 1260, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

@oesteban
Copy link
Member Author 

Working directory: /scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_03_task_localizer_run_02_wf/bold_mni_trans_wf/bold_to_mni_
transform

Node inputs:

args = <undefined>
default_value = 0.0
dimension = <undefined>
environ = {}
float = True
ignore_exception = False
input_image = ['/scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_03_task_localizer_run_02_wf/bold_hmc_wf/split/vol0000.nii.gz'
, '/scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_03_task_localizer_run_02_wf/bold_hmc_wf/split/vol0001.nii.gz', '/scratch/0
3763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_03_task_localizer_run_02_wf/bold_hmc_wf/split/vol0002.nii.gz', '/scratch/03763/oesteban
/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_03_task_localizer_run_02_wf/bold_hmc_wf/split/vol0003.nii.gz', '/scratch/03763/oesteban/fmriprep-pha
se1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_03_task_localizer_run_02_wf/bold_hmc_wf/split/vol0004.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds00
0232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_03_task_localizer_run_02_wf/bold_hmc_wf/split/vol0005.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep
_wf/single_subject_09_wf/func_preproc_ses_03_task_localizer_run_02_wf/bold_hmc_wf/split/vol0006.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_su
bject_09_wf/func_preproc_ses_03_task_localizer_run_02_wf/bold_hmc_wf/split/vol0007.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/f
unc_preproc_ses_03_task_localizer_run_02_wf/bold_hmc_wf/split/vol0008.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_s
es_03_task_localizer_run_02_wf/bold_hmc_wf/split/vol0009.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_03_task_lo
calizer_run_02_wf/bold_hmc_wf/split/vol0010.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_03_task_localizer_run_0
2_wf/bold_hmc_wf/split/vol0011.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_03_task_localizer_run_02_wf/bold_hmc
...skipping...
    self._run_interface()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 517, in _run_interface
    self._result = self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1088, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/itk.py", line 140, in _run_interface
    pool = Pool(processes=num_threads, maxtasksperchild=100)
  File "/usr/local/miniconda/lib/python3.6/multiprocessing/context.py", line 119, in Pool
    context=self.get_context())
  File "/usr/local/miniconda/lib/python3.6/multiprocessing/pool.py", line 168, in __init__
    self._repopulate_pool()
  File "/usr/local/miniconda/lib/python3.6/multiprocessing/pool.py", line 233, in _repopulate_pool
    w.start()
  File "/usr/local/miniconda/lib/python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/usr/local/miniconda/lib/python3.6/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/usr/local/miniconda/lib/python3.6/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/usr/local/miniconda/lib/python3.6/multiprocessing/popen_fork.py", line 67, in _launch
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

@oesteban
Copy link
Member Author 

Node: fmriprep_wf.single_subject_MSC08_wf.anat_preproc_wf.surface_recon_wf.autorecon_resume_wf.recon_report
Working directory: /scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC08_wf/anat_preproc_wf/surface_recon_wf/autorecon_resume_wf/recon_report

Node inputs:

T1_files = <undefined>
T2_file = <undefined>
args = <undefined>
big_ventricles = <undefined>
brainstem = <undefined>
compress_report = auto
directive = autorecon3
environ = {}
expert = <undefined>
flags = <undefined>
generate_report = True
hemi = <undefined>
hippocampal_subfields_T1 = <undefined>
hippocampal_subfields_T2 = <undefined>
hires = <undefined>
ignore_exception = False
mprage = <undefined>
mri_aparc2aseg = <undefined>
mri_ca_label = <undefined>
mri_ca_normalize = <undefined>
mri_ca_register = <undefined>
mri_edit_wm_with_aseg = <undefined>
mri_em_register = <undefined>
mri_fill = <undefined>
mri_mask = <undefined>
mri_normalize = <undefined>
mri_pretess = <undefined>
mri_remove_neck = <undefined>
mri_segment = <undefined>
mri_segstats = <undefined>
mri_tessellate = <undefined>
mri_watershed = <undefined>
mris_anatomical_stats = <undefined>
mris_ca_label = <undefined>
mris_fix_topology = <undefined>
mris_inflate = <undefined>
mris_make_surfaces = <undefined>
mris_register = <undefined>
mris_smooth = <undefined>
mris_sphere = <undefined>
mris_surf2vol = <undefined>
mrisp_paint = <undefined>
openmp = <undefined>
out_report = report.html
parallel = <undefined>
subject_id = sub-MSC08
subjects_dir = /scratch/03763/oesteban/fmriprep-phase1/out/ds000224/freesurfer
talairach = <undefined>
terminal_output = <undefined>
use_T2 = <undefined>
Node: fmriprep_wf.single_subject_MSC08_wf.anat_preproc_wf.surface_recon_wf.autorecon_resume_wf.recon_report
Working directory: /scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC08_wf/anat_preproc_wf/surface_recon_wf/autorecon_resume_wf/recon_report

Node inputs:

T1_files = <undefined>
T2_file = <undefined>
args = <undefined>
big_ventricles = <undefined>
brainstem = <undefined>
compress_report = auto
directive = autorecon3
environ = {}
expert = <undefined>
flags = <undefined>
generate_report = True
hemi = <undefined>
hippocampal_subfields_T1 = <undefined>
hippocampal_subfields_T2 = <undefined>
hires = <undefined>
ignore_exception = False
mprage = <undefined>
mri_aparc2aseg = <undefined>
mri_ca_label = <undefined>
mri_ca_normalize = <undefined>
mri_ca_register = <undefined>
mri_edit_wm_with_aseg = <undefined>
mri_em_register = <undefined>
mri_fill = <undefined>
mri_mask = <undefined>
mri_normalize = <undefined>
mri_pretess = <undefined>
mri_remove_neck = <undefined>
mri_segment = <undefined>
mri_segstats = <undefined>
mri_tessellate = <undefined>
mri_watershed = <undefined>
mris_anatomical_stats = <undefined>
mris_ca_label = <undefined>
mris_fix_topology = <undefined>
mris_inflate = <undefined>
mris_make_surfaces = <undefined>
mris_register = <undefined>
mris_smooth = <undefined>
mris_sphere = <undefined>
mris_surf2vol = <undefined>
mrisp_paint = <undefined>
openmp = <undefined>
out_report = report.html
parallel = <undefined>
subject_id = sub-MSC08
subjects_dir = /scratch/03763/oesteban/fmriprep-phase1/out/ds000224/freesurfer
talairach = <undefined>
terminal_output = <undefined>
use_T2 = <undefined>
xopts = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 51, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 407, in run
    self._run_interface()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 517, in _run_interface
    self._result = self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1088, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/common/report.py", line 53, in _run_interface
    ReportCapableInterface, self)._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1653, in _run_interface
    runtime.dependencies = get_dependencies(executable_name, runtime.environ)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1467, in get_dependencies
    env=environ)
  File "/usr/local/miniconda/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/usr/local/miniconda/lib/python3.6/subprocess.py", line 1260, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

@oesteban
Copy link
Member Author 

Node: fmriprep_wf.single_subject_MSC09_wf.func_preproc_ses_func09_task_motor_run_02_wf.bold_reg_wf.bold_to_t1w_transform
Working directory: /scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_reg_wf/bold_to_t1w_tra
nsform

Node inputs:

args = <undefined>
default_value = 0.0
dimension = <undefined>
environ = {}
float = True
ignore_exception = False
input_image = ['/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0000.nii.
gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0001.nii.gz', '/scr
atch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0002.nii.gz', '/scratch/03763
/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0003.nii.gz', '/scratch/03763/oesteban/
fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0004.nii.gz', '/scratch/03763/oesteban/fmriprep-p
hase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0005.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work
/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0006.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/
fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0007.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_w
f/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0008.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_s
ubject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0009.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC
09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0010.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func
_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0011.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_s
es_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0012.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_
task_motor_run_02_wf/bold_hmc_wf/split/vol0013.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor
_run_02_wf/bold_hmc_wf/split/vol0014.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf
/bold_hmc_wf/split/vol0015.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_
wf/split/vol0016.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/v
ol0017.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0018.nii
.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0019.nii.gz', '/sc
ratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0020.nii.gz', '/scratch/0376
3/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0021.nii.gz', '/scratch/03763/oesteban
/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0022.nii.gz', '/scratch/03763/oesteban/fmriprep-
phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0023.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/wor
k/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0024.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224
/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0025.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_
wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0026.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_
subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0027.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MS
C09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0028.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/fun
c_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0029.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_
ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0030.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09
_task_motor_run_02_wf/bold_hmc_wf/split/vol0031.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_moto
r_run_02_wf/bold_hmc_wf/split/vol0032.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_w
f/bold_hmc_wf/split/vol0033.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc
_wf/split/vol0034.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/
vol0035.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0036.ni
i.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0037.nii.gz', '/s
cratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0038.nii.gz', '/scratch/037
63/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0039.nii.gz', '/scratch/03763/oesteba
n/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0040.nii.gz', '/scratch/03763/oesteban/fmriprep
-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0041.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/wo
rk/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0042.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds00022
4/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0043.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep
_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0044.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single
_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0045.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_M
SC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0046.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/fu
nc_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0047.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc
_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0048.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func0
9_task_motor_run_02_wf/bold_hmc_wf/split/vol0049.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_mot
or_run_02_wf/bold_hmc_wf/split/vol0050.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_
wf/bold_hmc_wf/split/vol0051.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hm
c_wf/split/vol0052.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split
/vol0053.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0054.n
ii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0055.nii.gz', '/
scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0056.nii.gz', '/scratch/03
763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0057.nii.gz', '/scratch/03763/oesteb
an/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0058.nii.gz', '/scratch/03763/oesteban/fmripre
p-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0059.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/w
ork/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0060.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds0002
24/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0061.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmripre
p_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0062.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/singl
e_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0063.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_
MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0064.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/f
unc_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0065.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_prepro
c_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0066.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func
09_task_motor_run_02_wf/bold_hmc_wf/split/vol0067.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_mo
tor_run_02_wf/bold_hmc_wf/split/vol0068.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02
_wf/bold_hmc_wf/split/vol0069.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_h
mc_wf/split/vol0070.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/spli
t/vol0071.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0072.
nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0073.nii.gz', '
/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0074.nii.gz', '/scratch/0
3763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0075.nii.gz', '/scratch/03763/oeste
ban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0076.nii.gz', '/scratch/03763/oesteban/fmripr
ep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0077.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/
work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0078.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000
224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0079.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmripr
ep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0080.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/sing
le_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0081.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject
_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0082.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/
func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0083.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_prepr
oc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0084.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_fun
c09_task_motor_run_02_wf/bold_hmc_wf/split/vol0085.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_m
otor_run_02_wf/bold_hmc_wf/split/vol0086.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_0
2_wf/bold_hmc_wf/split/vol0087.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_
hmc_wf/split/vol0088.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/spl
it/vol0089.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0090
.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0091.nii.gz',
'/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0092.nii.gz', '/scratch/
03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0093.nii.gz', '/scratch/03763/oest
eban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0094.nii.gz', '/scratch/03763/oesteban/fmrip
rep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0095.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1
/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0096.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds00
0224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0097.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmrip
rep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0098.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/sin
gle_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0099.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subjec
t_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0100.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf
/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0101.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_prep
roc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/split/vol0102.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_fu
nc09_task_motor_run_02_wf/bold_hmc_wf/split/vol0103.nii.gz']
input_image_type = <undefined>
interpolation = LanczosWindowedSinc
interpolation_parameters = <undefined>
invert_transform_flags = <undefined>
num_threads = 8
out_postfix = _trans
output_image = <undefined>
print_out_composite_warp_file = <undefined>
reference_image = /scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_reg_wf/gen_ref/highres0
01_BrainExtractionBrain_reference.nii.gz
save_cmd = True
terminal_output = <undefined>
transforms = ['/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_reg_wf/bbreg_wf/fsl2itk_fw
d/affine.txt', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/nonlinear_sdc_wf/syn/ants_susc
eptibility0Warp.nii.gz', '/scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_motor_run_02_wf/bold_hmc_wf/fsl2itk/
mat2itk.txt']

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 51, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 407, in run
    self._run_interface()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 517, in _run_interface
    self._result = self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1088, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/itk.py", line 140, in _run_interface
    pool = Pool(processes=num_threads, maxtasksperchild=100)
  File "/usr/local/miniconda/lib/python3.6/multiprocessing/context.py", line 119, in Pool
    context=self.get_context())
  File "/usr/local/miniconda/lib/python3.6/multiprocessing/pool.py", line 168, in __init__
    self._repopulate_pool()
  File "/usr/local/miniconda/lib/python3.6/multiprocessing/pool.py", line 233, in _repopulate_pool
    w.start()
  File "/usr/local/miniconda/lib/python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/usr/local/miniconda/lib/python3.6/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/usr/local/miniconda/lib/python3.6/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/usr/local/miniconda/lib/python3.6/multiprocessing/popen_fork.py", line 67, in _launch
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

@oesteban
Copy link
Member Author 

Node: fmriprep_wf.single_subject_MSC09_wf.func_preproc_ses_func09_task_memoryfaces_wf.bold_reg_wf.bbreg_wf.bbregister
Working directory: /scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_memoryfaces_wf/bold_reg_wf/bbreg_wf/bbregis
ter

Node inputs:

args = <undefined>
compress_report = auto
contrast_type = t2
dof = 9
environ = {'SUBJECTS_DIR': '/opt/freesurfer/subjects'}
epi_mask = <undefined>
fsldof = <undefined>
generate_report = True
ignore_exception = False
init = <undefined>
init_cost_file = <undefined>
init_reg_file = /scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_memoryfaces_wf/bold_reg_wf/bbreg_wf/mri_coreg/
registration.lta
intermediate_file = <undefined>
out_fsl_file = <undefined>
out_lta_file = True
out_reg_file = <undefined>
out_report = report.svg
reg_frame = <undefined>
reg_middle_frame = <undefined>
registered_file = True
source_file = /scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_memoryfaces_wf/nonlinear_sdc_wf/skullstrip_bold_
wf/apply_mask/ants_susceptibility_Warped_masked.nii.gz
spm_nifti = <undefined>
subject_id = sub-MSC09
subjects_dir = /scratch/03763/oesteban/fmriprep-phase1/out/ds000224/freesurfer
terminal_output = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 51, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 407, in run
    self._run_interface()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 517, in _run_interface
    self._result = self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/freesurfer/base.py", line 162, in run
    return super(FSCommand, self).run(**inputs)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1088, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/common/report.py", line 73, in _run_interface
    self._generate_report()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/common/report.py", line 176, in _generate_report
    compress=self.inputs.compress_report),
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/viz/utils.py", line 379, in plot_registration
    svg = extract_svg(display, compress=compress)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/viz/utils.py", line 206, in extract_svg
    image_svg = svg_compress(image_svg, compress)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/viz/utils.py", line 171, in svg_compress
    stdout=subprocess.PIPE, check=True).stdout
  File "/usr/local/miniconda/lib/python3.6/subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/miniconda/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/usr/local/miniconda/lib/python3.6/subprocess.py", line 1260, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

@oesteban
Copy link
Member Author 

Node: _lta_ras2ras0
Working directory: /scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_memorywords_wf/bold_reg_wf/bbreg_wf/lta_ras
2ras/mapflow/_lta_ras2ras0

Node inputs:

args = <undefined>
environ = {}
ignore_exception = False
in_fsl = <undefined>
in_itk = <undefined>
in_lta = /scratch/03763/oesteban/fmriprep-phase1/work/ds000224/fmriprep_wf/single_subject_MSC09_wf/func_preproc_ses_func09_task_memorywords_wf/bold_reg_wf/bbreg_wf/bbregister/ants_s
usceptibility_Warped_masked_bbreg_sub-MSC09.lta
in_mni = <undefined>
in_niftyreg = <undefined>
in_reg = <undefined>
invert = <undefined>
ltavox2vox = <undefined>
out_fsl = <undefined>
out_itk = <undefined>
out_lta = True
out_mni = <undefined>
out_reg = <undefined>
source_file = <undefined>
target_conform = <undefined>
target_file = <undefined>
terminal_output = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 51, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 407, in run
    self._run_interface()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 517, in _run_interface
    self._result = self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1088, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1653, in _run_interface
    runtime.dependencies = get_dependencies(executable_name, runtime.environ)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1467, in get_dependencies
    env=environ)
  File "/usr/local/miniconda/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/usr/local/miniconda/lib/python3.6/subprocess.py", line 1260, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

@oesteban
Copy link
Member Author 

Node: fmriprep_wf.single_subject_09_wf.func_preproc_ses_04_task_localizer_run_01_wf.bold_confounds_wf.dvars
Working directory: /scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_04_task_localizer_run_01_wf/bold_confounds_wf/dvars

Node inputs:

figdpi = 100
figformat = png
figsize = (11.7, 2.3)
ignore_exception = False
in_file = /scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_04_task_localizer_run_01_wf/bold_reg_wf/merge/vol0000_xform-00000_m
erged.nii.gz
in_mask = /scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_04_task_localizer_run_01_wf/bold_reg_wf/mask_t1w_tfm/ants_susceptib
ility_Warped_brain_mask_maths_trans.nii.gz
intensity_normalization = 1000.0
remove_zerovariance = True
save_all = True
save_nstd = False
save_plot = False
save_std = True
save_vxstd = False
series_tr = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 51, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 407, in run
    self._run_interface()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 517, in _run_interface
    self._result = self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1088, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/algorithms/confounds.py", line 143, in _run_interface
    intensity_normalization=self.inputs.intensity_normalization)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/algorithms/confounds.py", line 771, in compute_dvars
    mfunc = (mfunc / np.median(mfunc)) * intensity_normalization
MemoryError

@oesteban
Copy link
Member Author 

Node: fmriprep_wf.single_subject_09_wf.func_preproc_ses_04_task_localizer_run_02_wf.bold_confounds_wf.tcompcor
Working directory: /scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_04_task_localizer_run_02_wf/bold_confounds_wf/tcompcor

Node inputs:

components_file = tcompcor.tsv
compress_report = auto
generate_report = True
header_prefix = <undefined>
high_pass_cutoff = 128.0
ignore_exception = False
ignore_initial_volumes = 0
mask_files = ['/scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_04_task_localizer_run_02_wf/bold_confounds_wf/csf_roi/eroded_m
ask.nii.gz']
mask_index = <undefined>
merge_method = <undefined>
num_components = 6
out_report = report.html
percentile_threshold = 0.05
pre_filter = cosine
realigned_file = /scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_04_task_localizer_run_02_wf/bold_reg_wf/merge/vol0000_xform-
00000_merged.nii.gz
regress_poly_degree = 1
repetition_time = 1.06
save_pre_filter = True
use_regress_poly = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 51, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 407, in run
    self._run_interface()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 517, in _run_interface
    self._result = self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1088, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/common/report.py", line 53, in _run_interface
    ReportCapableInterface, self)._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/algorithms/confounds.py", line 449, in _run_interface
    mask_images = self._process_masks(mask_images, imgseries.get_data())
  File "/home1/03763/oesteban/.local/lib/python3.6/site-packages/nibabel/spatialimages.py", line 572, in get_data
    data = np.asanyarray(self._dataobj)
  File "/usr/local/miniconda/lib/python3.6/site-packages/numpy/core/numeric.py", line 583, in asanyarray
    return array(a, dtype, copy=False, order=order, subok=True)
  File "/home1/03763/oesteban/.local/lib/python3.6/site-packages/nibabel/arrayproxy.py", line 145, in __array__
    raw_data = self.get_unscaled()
  File "/home1/03763/oesteban/.local/lib/python3.6/site-packages/nibabel/arrayproxy.py", line 140, in get_unscaled
    mmap=self._mmap)
  File "/home1/03763/oesteban/.local/lib/python3.6/site-packages/nibabel/volumeutils.py", line 524, in array_from_file
    n_read = infile.readinto(data_bytes)
  File "/usr/local/miniconda/lib/python3.6/gzip.py", line 276, in read
    return self._buffer.read(size)
MemoryError

@oesteban
Copy link
Member Author 

Node: fmriprep_wf.single_subject_09_wf.func_preproc_ses_04_task_localizer_run_01_wf.bold_confounds_wf.signals
Working directory: /scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_04_task_localizer_run_01_wf/bold_confounds_wf/signals

Node inputs:

class_labels = ['WhiteMatter', 'GlobalSignal']
detrend = True
ignore_exception = False
in_file = /scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_04_task_localizer_run_01_wf/bold_reg_wf/merge/vol0000_xform-00000_m
erged.nii.gz
incl_shared_variance = True
include_global = False
label_files = ['/scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_04_task_localizer_run_01_wf/bold_confounds_wf/concat_rois/con
cat.nii.gz']
out_file = signals.tsv

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 51, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 407, in run
    self._run_interface()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 517, in _run_interface
    self._result = self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1088, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/nilearn.py", line 83, in _run_interface
    signals.append(masker.fit_transform(self.inputs.in_file))
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/input_data/nifti_maps_masker.py", line 214, in fit_transform
    return self.fit().transform(imgs, confounds=confounds)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/input_data/base_masker.py", line 176, in transform
    return self.transform_single_imgs(imgs, confounds)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/input_data/nifti_maps_masker.py", line 320, in transform_single_imgs
    verbose=self.verbose)
  File "/usr/local/miniconda/lib/python3.6/site-packages/sklearn/externals/joblib/memory.py", line 283, in __call__
    return self.func(*args, **kwargs)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/input_data/base_masker.py", line 66, in filter_and_extract
    imgs = _utils.check_niimg(imgs, atleast_4d=True, ensure_ndim=4)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/_utils/niimg_conversions.py", line 271, in check_niimg
    niimg = load_niimg(niimg, dtype=dtype)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/_utils/niimg.py", line 116, in load_niimg
    dtype = _get_target_dtype(niimg.get_data().dtype, dtype)
  File "/home1/03763/oesteban/.local/lib/python3.6/site-packages/nibabel/spatialimages.py", line 572, in get_data
    data = np.asanyarray(self._dataobj)
  File "/usr/local/miniconda/lib/python3.6/site-packages/numpy/core/numeric.py", line 583, in asanyarray
    return array(a, dtype, copy=False, order=order, subok=True)
  File "/home1/03763/oesteban/.local/lib/python3.6/site-packages/nibabel/arrayproxy.py", line 145, in __array__
    raw_data = self.get_unscaled()
  File "/home1/03763/oesteban/.local/lib/python3.6/site-packages/nibabel/arrayproxy.py", line 140, in get_unscaled
    mmap=self._mmap)
  File "/home1/03763/oesteban/.local/lib/python3.6/site-packages/nibabel/volumeutils.py", line 524, in array_from_file
    n_read = infile.readinto(data_bytes)
  File "/usr/local/miniconda/lib/python3.6/gzip.py", line 276, in read
    return self._buffer.read(size)
MemoryError

@oesteban
Copy link
Member Author 

Node: fmriprep_wf.single_subject_09_wf.func_preproc_ses_03_task_localizer_run_02_wf.bold_mni_trans_wf.mask_mni_tfm
Working directory: /scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_03_task_localizer_run_02_wf/bold_mni_trans_wf/mask_mni_tfm

Node inputs:

args = <undefined>
default_value = 0.0
dimension = <undefined>
environ = {'NSLOTS': '1'}
float = True
ignore_exception = False
input_image = /scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_03_task_localizer_run_02_wf/nonlinear_sdc_wf/skullstrip_bold_wf
/combine_masks/ants_susceptibility_Warped_brain_mask_maths.nii.gz
input_image_type = <undefined>
interpolation = NearestNeighbor
interpolation_parameters = <undefined>
invert_transform_flags = <undefined>
num_threads = 1
out_postfix = _trans
output_image = <undefined>
print_out_composite_warp_file = <undefined>
reference_image = /scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_03_task_localizer_run_02_wf/bold_mni_trans_wf/gen_ref/1mm_T
1_reference.nii.gz
terminal_output = <undefined>
transforms = ['/scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/anat_preproc_wf/t1_2_mni/ants_t1_to_mniComposite.h5', '/scratch/03763/oesteban/
fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_03_task_localizer_run_02_wf/bold_reg_wf/bbreg_wf/fsl2itk_fwd/affine.txt']

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 51, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 407, in run
    self._run_interface()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 517, in _run_interface
    self._result = self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1088, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/fixes.py", line 24, in _run_interface
    runtime, correct_return_codes)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1653, in _run_interface
    runtime.dependencies = get_dependencies(executable_name, runtime.environ)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1467, in get_dependencies
    env=environ)
  File "/usr/local/miniconda/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/usr/local/miniconda/lib/python3.6/subprocess.py", line 1260, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

@oesteban
Copy link
Member Author 

Node: fmriprep_wf.single_subject_09_wf.func_preproc_ses_04_task_localizer_run_01_wf.bold_surf_wf.sampler
Working directory: /scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_04_task_localizer_run_01_wf/bold_surf_wf/_hemi_lh/sampler

Node inputs:

apply_rot = <undefined>
apply_trans = <undefined>
args = <undefined>
cortex_mask = True
environ = {'SUBJECTS_DIR': '/opt/freesurfer/subjects'}
fix_tk_reg = <undefined>
float2int_method = <undefined>
frame = <undefined>
hemi = <undefined>
hits_file = <undefined>
hits_type = <undefined>
ico_order = <undefined>
ignore_exception = False
interp_method = trilinear
mask_label = <undefined>
mni152reg = <undefined>
no_reshape = <undefined>
out_file = <undefined>
out_type = gii
override_reg_subj = <undefined>
projection_stem = <undefined>
reference_file = <undefined>
reg_file = <undefined>
reg_header = True
reshape = <undefined>
reshape_slices = <undefined>
sampling_method = average
sampling_range = (0.0, 1.0, 0.2)
sampling_units = frac
scale_input = <undefined>
smooth_surf = <undefined>
smooth_vol = <undefined>
source_file = ['/scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_04_task_localizer_run_01_wf/bold_surf_wf/rename_src/mapflow/_
rename_src0/fsaverage5.nii.gz']
subject_id = <undefined>
subjects_dir = /opt/freesurfer/subjects
surf_reg = <undefined>
surface = <undefined>
target_subject = ['fsaverage5']
terminal_output = <undefined>
vox_file = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 51, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 407, in run
    self._run_interface()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 1326, in _run_interface
    updatehash=updatehash))
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 1232, in _collate_results
    (self.name, '\n'.join(msg)))
Exception: Subnodes of node: sampler failed:
Subnode 0 failed
Error:
Command:
mri_vol2surf --cortex --hemi lh --interp trilinear --o /scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_04_task_localizer_run_
01_wf/bold_surf_wf/_hemi_lh/sampler/mapflow/_sampler0/lh.fsaverage5.gii  --regheader sub-09 --projfrac-avg 0.000 1.000 0.200 --mov /scratch/03763/oesteban/fmriprep-phase1/work/ds000
232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_04_task_localizer_run_01_wf/bold_surf_wf/rename_src/mapflow/_rename_src0/fsaverage5.nii.gz --trgsubject fsaverage5
Standard output:
srcvol = /scratch/03763/oesteban/fmriprep-phase1/work/ds000232/fmriprep_wf/single_subject_09_wf/func_preproc_ses_04_task_localizer_run_01_wf/bold_surf_wf/rename_src/mapflow/_rename_
src0/fsaverage5.nii.gz
srcreg unspecified
srcregold = 0
srcwarp unspecified
surf = white
hemi = lh
trgsubject = fsaverage5
surfreg = sphere.reg
ProjFrac = 0.5
thickness = thickness
reshape = 0
interp = trilinear
float2int = round
GetProjMax = 0
INFO: float2int code = 0
niiRead(): NIFTI_UNITS_UNKNOWN, assuming mm

Standard error:
MRIallocSequence(128, 101, 132,380): could not allocate 51712 bytes for 31378th slice

Cannot allocate memory

Return code: 254

@oesteban
Copy link
Member Author

Some memory issues still persist:

171018-06:05:51,88 workflow ERROR:
         Node signals failed to run on host nid00625.
171018-06:05:51,89 workflow ERROR:
         Saving crash info to /scratch/03763/oesteban/fmriprep-phase1/out/ds000232/fmriprep/sub-09/log/20171018-043323_44a30a1d-c637-474d-9cd2-d372b1b12107/crash-20171018-060551-oesteban-signals-da27074e-2003-
4a55-901e-3454753dbae3.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 51, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 407, in run
    self._run_interface()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 517, in _run_interface
    self._result = self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1088, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/nilearn.py", line 83, in _run_interface
    signals.append(masker.fit_transform(self.inputs.in_file))
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/input_data/nifti_maps_masker.py", line 214, in fit_transform
    return self.fit().transform(imgs, confounds=confounds)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/input_data/base_masker.py", line 176, in transform
    return self.transform_single_imgs(imgs, confounds)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/input_data/nifti_maps_masker.py", line 254, in transform_single_imgs
    imgs_ = _utils.check_niimg_4d(imgs)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/_utils/niimg_conversions.py", line 361, in check_niimg_4d
    dtype=dtype)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/_utils/niimg_conversions.py", line 271, in check_niimg
    niimg = load_niimg(niimg, dtype=dtype)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/_utils/niimg.py", line 116, in load_niimg
    dtype = _get_target_dtype(niimg.get_data().dtype, dtype)
  File "/home1/03763/oesteban/.local/lib/python3.6/site-packages/nibabel/spatialimages.py", line 572, in get_data
    data = np.asanyarray(self._dataobj)
  File "/usr/local/miniconda/lib/python3.6/site-packages/numpy/core/numeric.py", line 583, in asanyarray
    return array(a, dtype, copy=False, order=order, subok=True)
  File "/home1/03763/oesteban/.local/lib/python3.6/site-packages/nibabel/arrayproxy.py", line 145, in __array__
    raw_data = self.get_unscaled()
  File "/home1/03763/oesteban/.local/lib/python3.6/site-packages/nibabel/arrayproxy.py", line 140, in get_unscaled
  File "/home1/03763/oesteban/.local/lib/python3.6/site-packages/nibabel/arrayproxy.py", line 140, in get_unscaled
    mmap=self._mmap)
  File "/home1/03763/oesteban/.local/lib/python3.6/site-packages/nibabel/volumeutils.py", line 524, in array_from_file
    n_read = infile.readinto(data_bytes)
  File "/usr/local/miniconda/lib/python3.6/gzip.py", line 276, in read
    return self._buffer.read(size)
MemoryError

This memory error seems totally related to one on sampler (since they are executed close together and both were using the same compute node).

@effigies
Copy link
Member

Worth considering packaging indexed_gzip, which should help reduce the memory footprint of these reads on large BOLD series. See nipy/nibabel#552.

@oesteban
Copy link
Member Author

Would it be just plug and play? Or we need to revise nibabel's loads?

@effigies
Copy link
Member

Unfortunately, it had to be kept from being fully plug and play, because it clashed with an existing behavior of closing and releasing filehandles, to keep the OS from exhausting them, which apparently Gael ran into some years back.

To enable it (if available) nibabel.arrayproxy.KEEP_FILE_OPEN_DEFAULT needs to be set to 'auto'.

I would at to the top of run.py:

import nibabel
nibabel.arrayproxy.KEEP_FILE_OPEN_DEFAULT = 'auto'

I'm not sure how well that context will be passed when running function nodes, but it should work fine for anything that doesn't have to be pickled. We may want to do some tests.

@oesteban
Copy link
Member Author

oesteban commented Oct 23, 2017 •
edited
Loading

Thanks a lot. Before going down that road, I think I've found the biggest issue here.

For ds000232/ sub-05_ses-04_task-main_run-01 we have the following:

  • Original file: 250MB, int16, grid of 102x102x30x940
  • Merged (bold_reg_wf) file: 5GB (uncompressed), float32, grid of 128x129x97x940.

I think we should calculate confounds (dvars, signals) on the original matrix size and possibly try float16?

The aftermath is related to https://neurostars.org/t/why-are-functional-data-files-output-by-fmriprep-in-t1w-space-so-much-larger/796/4?u=oesteban . We should definitely keep the original pixel resolution in T1w space.

I would unfold this memory issue in two:

  • Calculate confounds on the original space (and maybe cast to the same type-width, ergo float16 in this case)
  • When the T1w is selected as target space, keep the original resolution of the BOLD signal (do not conform to 2mm x 2mm x 2mm).

WDYT @chrisfilo, @effigies ?

@chrisgorgo
Copy link
Contributor

@oesteban we are never conforming to 2x2x2 resolution - the original BOLD resolution is always used. See my comments here. I believe this is a field of view issue - I proposed a solution here.

@effigies
Copy link
Member

Maybe a sensible approach for cropping BOLD series is to inflate the mask (1-2cm in all directions?), and crop that. fslroi would probably be the best tool for that in the short term. (Cropping should be in nibabel in the next release, but that's not always the fastest process.)

@oesteban
Copy link
Member Author

oesteban commented Oct 23, 2017 •
edited
Loading

Ok, yes, I see that the resolution is the same. Only the extent of the volume is extended to that of the reference (T1w) image.

Well, that answers to the second question, when writing output files. I'm opening an issue to address that. (we have #722 open).

I'll first address that one, because we probably don't need to worry about the first one (calculate confounds on the original space) since we are applying all transforms at once. It should be equivalent.

@chrisgorgo
Copy link
Contributor

@effigies FYI cropping is implemented in nilearn. One downside of this approach is that each output file might have different field of view (dimensions in voxels) - across participants that is. Not sure if it matters for T1w space outputs though.

oesteban added a commit to oesteban/fmriprep that referenced this issue Oct 25, 2017
@oesteban
[ENH] Use indexed_gzip with nibabel
2c55a9d 
This PR implements this suggestion nipreps#766 (comment)

If we find this useful, it is worth considering to add a new nipype config that leverages indexed_gzip with nibabel.

Hopefully, this is the last piece of several PRs to fix nipreps#766
@oesteban oesteban mentioned this issue Oct 25, 2017
Merged
@dPys dPys mentioned this issue Jun 20, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug memory
Projects
None yet
Milestone
Phase II
Development

Successfully merging a pull request may close this issue.

[ENH] Use indexed_gzip with nibabel oesteban/fmriprep
3 participants
@effigies @chrisgorgo @oesteban