This is a combination of 3 commits.

doc: misc setup and doc changes

maint: remove unnecessary files

Some reorganization, and rethinking the main workflow. Still not runnable via argparse.

fix eddy_quad bindings

update fork

remove slice timing req

update Dockerfile

fix imports in qc

update MANIFEST.in

update bids.py

update bids.py

update Dockerfile to include new eddy

turn off cnr_maps with eddy

update Dockerfile with fsl 5.0.11

add nlsam denoising to dockerfile

add nlsam denoising option

add nlsam denoising option

fix nlsam, add eddy args

fix nlsam, add eddy args

create new cli skeleton

create new cli skeleton

create new cli skeleton

add support for multiple sessions and multi-file concatenation

add meta-wf handling

reorder workflow stages in light of recommendations from Maximov et al. 2019, Ades-Aron et al. 2018(a), Ades-Aron et al. 2018(b)

rename final dwi output to make it identifiable

adjust ART norm thresh

split noise estimation stage and denoising

tweak cores/mem alloc

liter eddy

fix verbosity, refine eddy memory reqs

save final bval/bvec as txt

update Dockerfile, reduce size

fix bug with eddy_quad base_name

workflow end-end completion

update Dockerfile

tweak mem restrictions

update mem restrictions

fine-tune resource handling

fine-tune resource consumption

more resource handling fine-tuning

refine resource handling

add support to drop outliers from sigma if 4d

openMP handling

squash/merge final pre-meeting

add sigma switch to drop_outliers_fn

fix sigma

fix sigma issue

change os.rename to shutil.copy to handle linking across mounted filesystems

resource handling

remove array passing to alleviate memory/pickling errors

update eddy mem reqs

fix topup odd slice option

update eddy mem reqs
fix b02b0.cfg bug for odd slices

simplify b02b0.cnf

specify handling for second-level modeling in eddy if data is detected as hemispheric

specify handling for second-level modeling in eddy if data is detected as hemispheric

FIX: send empty str to 'args' trait of an EddyInputSpec

empty str to eddy args for non-reverse-phase

empty str to eddy args for non-reverse-phase: PR

fix slm str trait bug

adjust .cnf hyperparameter window

adjust .cnf hyperparameter window

Author: josephmje

.dockerignore

@@ -39,4 +39,4 @@ out/
 .zenodo.json
 .travis.yml
 .readthedocs.yml
-CONTRIBUTING.md
+CONTRIBUTING.rst

.github/config.yml

@@ -5,7 +5,7 @@ newPRWelcomeComment: >
   Thanks for opening this pull request!
   We have detected this is the first time for you to contribute
   to *dMRIPrep*.
-  Please check out our [contributing guidelines](https://github.com/nipreps/dmriprep/blob/master/CONTRIBUTING.md).
+  Please check out our [contributing guidelines](https://github.com/nipreps/dmriprep/blob/master/CONTRIBUTING.rst).
 
   We invite you to list yourself as a *dMRIPrep* contributor, so if your name
   is not already mentioned, please modify the

.gitignore

@@ -103,6 +103,3 @@ ENV/
 
 # Mac OS nonsense:
 .DS_Store
-
-# miscellaneous
-references/

.zenodo.json

@@ -3,6 +3,11 @@
   "title": "dMRIPrep: a robust preprocessing pipeline for diffusion MRI",
   "description": "<p>dMRIPrep is a robust and easy-to-use pipeline for preprocessing of diverse dMRI data. The transparent workflow dispenses of manual intervention, thereby ensuring the reproducibility of the results.</p>",
   "creators": [
+    {
+      "affiliation": "University of Texas at Austin",
+      "name": "Pisner, Derek",
+      "orcid": "0000-0002-1228-0201"
+    },
     {
       "affiliation": "Stanford University",
       "name": "Lerma-Usabiaga, Garikoitz",

CONTRIBUTING.rst

@@ -4,74 +4,6 @@
 Contributing
 ============
 
-.. code-block:: console
-
-    dmriprep
-    ├── CHANGES.rst
-    ├── CONTRIBUTING.rst
-    ├── dmriprep
-    │   ├── __about__.py
-    │   ├── cli.py
-    │   ├── data
-    │   │   ├── boilerplate.bib
-    |   |   ├── eddy_parameters
-    │   │   └── ENIGMA_targets
-    │   ├── due.py
-    │   ├── __init__.py
-    │   ├── interfaces
-    │   │   ├── fmap.py
-    │   │   ├── __init__.py
-    │   │   └── reportlets.py
-    │   ├── qc.py
-    │   ├── utils
-    │   │   ├── bids.py
-    │   │   ├── hemisphere.py
-    │   │   └── __init__.py
-    │   ├── _version.py
-    │   ├── viz
-    │   │   ├── __init__.py
-    │   │   └── utils.py
-    │   └── workflows
-    │       ├── anatomical.py
-    │       ├── base.py
-    │       ├── dwi
-    │       │   ├── __init__.py
-    │       │   ├── artifacts.py
-    │       │   ├── base.py
-    │       │   ├── eddy.py
-    │       │   ├── outputs.py
-    │       │   ├── qc.py
-    │       │   ├── tensor.py
-    │       │   └── util.py
-    │       ├── fieldmap
-    │       │   ├── __init__.py
-    │       │   ├── ants.py
-    │       │   ├── base.py
-    │       │   ├── brainsuite.py
-    │       │   ├── fmap.py
-    │       │   ├── pepolar.py
-    │       │   ├── phasediff.py
-    │       │   └── synb0.py
-    │       └── __init__.py
-    ├── Dockerfile
-    ├── docs
-    ├── LICENSE
-    ├── long_description.rst
-    ├── Makefile
-    ├── MANIFEST.in
-    ├── pyproject.toml
-    ├── README.rst
-    ├── REFERENCES.rst
-    ├── setup.cfg
-    ├── setup.py
-    ├── tests
-    │   ├── __init__.py
-    │   ├── test_dmriprep.py
-    │   └── test_utils.py
-    ├── tox.ini
-    ├── update_changes.sh
-    └── versioneer.py
-
 We love contributions! dmriprep is open source, built on open source,
 and we'd love to have you hang out in our community.
 

Dockerfile

@@ -2,7 +2,7 @@
 FROM ubuntu:xenial-20161213
 
 # Pre-cache neurodebian key
-COPY .docker/neurodebian.gpg /usr/local/etc/neurodebian.gpg
+COPY docker/files/neurodebian.gpg /usr/local/etc/neurodebian.gpg
 
 # Prepare environment
 RUN apt-get update && \
@@ -16,6 +16,10 @@ RUN apt-get update && \
                     autoconf \
                     libtool \
                     pkg-config \
+		    vim \
+		    zip \
+		    unzip \
+		    wget \
                     git && \
     curl -sL https://deb.nodesource.com/setup_10.x | bash - && \
     apt-get install -y --no-install-recommends \
@@ -27,43 +31,13 @@ RUN curl -o pandoc-2.2.2.1-1-amd64.deb -sSL "https://github.com/jgm/pandoc/relea
     dpkg -i pandoc-2.2.2.1-1-amd64.deb && \
     rm pandoc-2.2.2.1-1-amd64.deb
 
-# Installing freesurfer
-RUN curl -sSL https://surfer.nmr.mgh.harvard.edu/pub/dist/freesurfer/6.0.1/freesurfer-Linux-centos6_x86_64-stable-pub-v6.0.1.tar.gz | tar zxv --no-same-owner -C /opt \
-    --exclude='freesurfer/diffusion' \
-    --exclude='freesurfer/docs' \
-    --exclude='freesurfer/fsfast' \
-    --exclude='freesurfer/lib/cuda' \
-    --exclude='freesurfer/lib/qt' \
-    --exclude='freesurfer/matlab' \
-    --exclude='freesurfer/mni/share/man' \
-    --exclude='freesurfer/subjects/fsaverage_sym' \
-    --exclude='freesurfer/subjects/fsaverage3' \
-    --exclude='freesurfer/subjects/fsaverage4' \
-    --exclude='freesurfer/subjects/cvs_avg35' \
-    --exclude='freesurfer/subjects/cvs_avg35_inMNI152' \
-    --exclude='freesurfer/subjects/bert' \
-    --exclude='freesurfer/subjects/lh.EC_average' \
-    --exclude='freesurfer/subjects/rh.EC_average' \
-    --exclude='freesurfer/subjects/sample-*.mgz' \
-    --exclude='freesurfer/subjects/V1_average' \
-    --exclude='freesurfer/trctrain'
-
 ENV FSL_DIR="/usr/share/fsl/5.0" \
     OS="Linux" \
     FS_OVERRIDE=0 \
     FIX_VERTEX_AREA="" \
-    FSF_OUTPUT_FORMAT="nii.gz" \
-    FREESURFER_HOME="/opt/freesurfer"
-ENV SUBJECTS_DIR="$FREESURFER_HOME/subjects" \
-    FUNCTIONALS_DIR="$FREESURFER_HOME/sessions" \
-    MNI_DIR="$FREESURFER_HOME/mni" \
-    LOCAL_DIR="$FREESURFER_HOME/local" \
-    MINC_BIN_DIR="$FREESURFER_HOME/mni/bin" \
-    MINC_LIB_DIR="$FREESURFER_HOME/mni/lib" \
-    MNI_DATAPATH="$FREESURFER_HOME/mni/data"
+    FSF_OUTPUT_FORMAT="nii.gz"
 ENV PERL5LIB="$MINC_LIB_DIR/perl5/5.8.5" \
-    MNI_PERL5LIB="$MINC_LIB_DIR/perl5/5.8.5" \
-    PATH="$FREESURFER_HOME/bin:$FSFAST_HOME/bin:$FREESURFER_HOME/tktools:$MINC_BIN_DIR:$PATH"
+    MNI_PERL5LIB="$MINC_LIB_DIR/perl5/5.8.5"
 
 # Installing Neurodebian packages (FSL, AFNI, git)
 RUN curl -sSL "http://neuro.debian.net/lists/$( lsb_release -c | cut -f2 ).us-ca.full" >> /etc/apt/sources.list.d/neurodebian.sources.list && \
@@ -72,32 +46,63 @@ RUN curl -sSL "http://neuro.debian.net/lists/$( lsb_release -c | cut -f2 ).us-ca
 
 RUN apt-get update && \
     apt-get install -y --no-install-recommends \
-                    fsl-core=5.0.9-5~nd16.04+1 \
-                    fsl-mni152-templates=5.0.7-2 \
                     afni=16.2.07~dfsg.1-5~nd16.04+1 \
-                    convert3d \
                     git-annex-standalone && \
     apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
 
-ENV FSLDIR="/usr/share/fsl/5.0" \
-    FSLOUTPUTTYPE="NIFTI_GZ" \
-    FSLMULTIFILEQUIT="TRUE" \
-    POSSUMDIR="/usr/share/fsl/5.0" \
-    LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH" \
-    FSLTCLSH="/usr/bin/tclsh" \
-    FSLWISH="/usr/bin/wish" \
-    AFNI_MODELPATH="/usr/lib/afni/models" \
-    AFNI_IMSAVE_WARNINGS="NO" \
-    AFNI_TTATLAS_DATASET="/usr/share/afni/atlases" \
-    AFNI_PLUGINPATH="/usr/lib/afni/plugins"
-ENV PATH="/usr/lib/fsl/5.0:/usr/lib/afni/bin:$PATH"
+ENV FSLDIR="/opt/fsl-6.0.1" \
+    PATH="/opt/fsl-6.0.1/bin:$PATH" \
+    FSLOUTPUTTYPE="NIFTI_GZ"
+RUN apt-get update -qq \
+    && apt-get install -y -q --no-install-recommends \
+           bc \
+           dc \
+           file \
+           libfontconfig1 \
+           libfreetype6 \
+           libgl1-mesa-dev \
+           libglu1-mesa-dev \
+           libgomp1 \
+           libice6 \
+           libxcursor1 \
+           libxft2 \
+           libxinerama1 \
+           libxrandr2 \
+           libxrender1 \
+           libxt6 \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/* \
+    && echo "Downloading FSL ..." \
+    && wget -q http://fsl.fmrib.ox.ac.uk/fsldownloads/fslinstaller.py \
+    && chmod 775 fslinstaller.py
+RUN /fslinstaller.py -d /opt/fsl-6.0.1 -V 6.0.1 -q
+
+RUN rm -rf /opt/fsl-6.0.1/data \
+    && rm -rf /opt/fsl-6.0.1/bin/FSLeyes* \
+    && rm -rf /opt/fsl-6.0.1/src \
+    && rm -rf /opt/fsl-6.0.1/extras/src \
+    && rm -rf /opt/fsl-6.0.1/doc \
+    && rm -rf /opt/fsl-6.0.1/bin/fslview.app \
+    && rm -rf /opt/fsl-6.0.1/data/atlases \
+    && rm -rf /opt/fsl-6.0.1/data/first \
+    && rm -rf /opt/fsl-6.0.1/data/mist \
+    && rm -rf /opt/fsl-6.0.1/data/possum
 
 # Installing ANTs 2.2.0 (NeuroDocker build)
 ENV ANTSPATH=/usr/lib/ants
 RUN mkdir -p $ANTSPATH && \
     curl -sSL "https://dl.dropbox.com/s/2f4sui1z6lcgyek/ANTs-Linux-centos5_x86_64-v2.2.0-0740f91.tar.gz" \
     | tar -xzC $ANTSPATH --strip-components 1
-ENV PATH=$ANTSPATH:$PATH
+
+ENV AFNI_INSTALLDIR=/usr/lib/afni \
+    PATH=${PATH}:/usr/lib/afni/bin \
+    AFNI_PLUGINPATH=/usr/lib/afni/plugins \
+    AFNI_MODELPATH=/usr/lib/afni/models \
+    AFNI_TTATLAS_DATASET=/usr/share/afni/atlases \
+    AFNI_IMSAVE_WARNINGS=NO \
+    FSLOUTPUTTYPE=NIFTI_GZ \
+    PATH=$ANTSPATH:$PATH \
+    ANTS_VERSION=2.2.0
 
 # Create a shared $HOME directory
 RUN useradd -m -s /bin/bash -G users dmriprep
@@ -112,14 +117,6 @@ RUN npm install -g svgo
 # Installing bids-validator
 RUN npm install -g bids-validator@1.2.3
 
-# Installing and setting up ICA_AROMA
-RUN mkdir -p /opt/ICA-AROMA && \
-  curl -sSL "https://github.com/maartenmennes/ICA-AROMA/archive/v0.4.4-beta.tar.gz" \
-  | tar -xzC /opt/ICA-AROMA --strip-components 1 && \
-  chmod +x /opt/ICA-AROMA/ICA_AROMA.py
-
-ENV PATH=/opt/ICA-AROMA:$PATH
-
 # Installing and setting up miniconda
 RUN curl -sSLO https://repo.continuum.io/miniconda/Miniconda3-4.5.11-Linux-x86_64.sh && \
     bash Miniconda3-4.5.11-Linux-x86_64.sh -b -p /usr/local/miniconda && \
@@ -147,6 +144,7 @@ RUN conda install -y python=3.7.1 \
                      graphviz=2.40.1 \
                      traits=4.6.0 \
                      zlib; sync && \
+		     cython && \
     chmod -R a+rX /usr/local/miniconda; sync && \
     chmod +x /usr/local/miniconda/bin/*; sync && \
     conda build purge-all; sync && \
@@ -161,26 +159,33 @@ ENV MKL_NUM_THREADS=1 \
 RUN python -c "from matplotlib import font_manager" && \
     sed -i 's/\(backend *: \).*$/\1Agg/g' $( python -c "import matplotlib; print(matplotlib.matplotlib_fname())" )
 
-# Precaching atlases
-RUN pip install --no-cache-dir "templateflow>=0.4.0,<0.5.0a0" && \
-    python -c "from templateflow import api as tfapi; \
-               tfapi.get('MNI152NLin6Asym', atlas=None, extension=['.nii', '.nii.gz']); \
-               tfapi.get('MNI152NLin2009cAsym', atlas=None, extension=['.nii', '.nii.gz']); \
-               tfapi.get('OASIS30ANTs', extension=['.nii', '.nii.gz']);" && \
-    find $HOME/.cache/templateflow -type d -exec chmod go=u {} + && \
-    find $HOME/.cache/templateflow -type f -exec chmod go=u {} +
+RUN pip install --upgrade pip
+
+RUN apt-get update && apt-get install -y \
+  gfortran \
+  liblapack-dev \
+  libopenblas-dev
+
+RUN pip install ipython cython parse
 
 # Installing DMRIPREP
-COPY . /src/dmriprep
-ARG VERSION
-# Force static versioning within container
-RUN echo "${VERSION}" > /src/dmriprep/dmriprep/VERSION && \
-    echo "include dmriprep/VERSION" >> /src/dmriprep/MANIFEST.in && \
-    pip install --no-cache-dir "/src/dmriprep[all]"
+RUN git clone -b nipreps https://github.com/dPys/dmriprep.git dmriprep && \
+    cd dmriprep && \
+    python setup.py install
+
+RUN pip install ipython cython parse
+
+RUN pip install --no-cache-dir https://github.com/samuelstjean/nlsam/archive/master.zip
 
 RUN find $HOME -type d -exec chmod go=u {} + && \
     find $HOME -type f -exec chmod go=u {} +
 
+RUN mkdir /inputs && \
+    chmod -R 777 /inputs
+
+RUN mkdir /outputs && \
+    chmod -R 777 /outputs
+
 ENV IS_DOCKER_8395080871=1
 
 RUN ldconfig

MANIFEST.in

@@ -5,3 +5,20 @@ include LICENSE
 # versioneer
 include versioneer.py
 include dmriprep/_version.py
+recursive-include * *.nii.gz
+recursive-include * *.html
+recursive-include * *.js
+recursive-include * *.json
+recursive-include * *.txt
+recursive-include * *.csv
+recursive-include dmriprep/viz *
+recursive-include dmriprep/utils *
+recursive-include dmriprep/interfaces *
+recursive-include dmriprep/data *
+recursive-include dmriprep/config *
+recursive-include dmriprep/cli *
+recursive-include tests *
+recursive-exclude * __pycache__
+recursive-exclude * *.py[co]
+recursive-include * *.yaml
+recursive-include docs *.rst conf.py Makefile make.bat *.jpg *.png *.gif

Makefile

@@ -87,8 +87,8 @@ dist: clean ## builds source and wheel package
 install: clean ## install the package to the active Python's site-packages
 	python setup.py install
 
-docker-dev: ## build the development environment
-	docker build -t dmriprep:dev -f docker/Dockerfile docker/.
-
-docker: docker-dev
-	docker build -t dmriprep:prod -f Dockerfile-dmriprep .
+docker: docker
+	docker build --rm -t nipreps/dmriprep:latest \
+		--build-arg BUILD_DATE=`date -u +"%Y-%m-%dT%H:%M:%SZ"` \
+		--build-arg VCS_REF=`git rev-parse --short HEAD` \
+		--build-arg VERSION=$( python get_version.py ) .