From 7c8f2a66c93034a91a12a1cf7b8f82fb0112bb0e Mon Sep 17 00:00:00 2001 From: Matthew Archer Date: Sun, 25 Sep 2022 14:57:31 +0100 Subject: [PATCH 1/7] add docs for building detectree2 --- docs/source/installation.rst | 108 ++++++++++++++++++++++++++++++++++- setup.py | 11 ++++ 2 files changed, 117 insertions(+), 2 deletions(-) diff --git a/docs/source/installation.rst b/docs/source/installation.rst index 46936c1d..f485eec8 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation.rst @@ -1,5 +1,109 @@ -Installation ------------- +******************* +Building Detectree2 +******************* + +============================== +Getting up and running quickly +============================== + +To get up and running quickly it is possible to install detectree2 and its dependencies with Conda. Simply do:: + + conda install -c ma595 detectree2 + + + +========= +Using pip +========= + +It is easy to install detectree2 on your own system. Simply pip install and all dependencies will be installed automatically. + + See workflow `python-app.yaml `_ workflow for a working CPU deployment. + +First install ``pytorch``, ``torchvision`` and ``torchaudio`` (compatible versions https://pypi.org/project/torchvision/). Follow `https://pytorch.org/get-started/locally/`_ to get compatible version for your system. Below we run through the process with pip but the conda approach is equally valid. + +This can be done inside ``virtualenv`` (if root access is unavailable):: + + python3 -m venv ./venv # (check version of python is sufficiently high >=3.7, required by detectron2) + . venv/bin/activate + pip install --upgrade pip + pip install wheel + pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113 + +Then point to preinstalled GDAL header files (not necessary if include directory is already in your path):: + + export CPLUS_INCLUDE_PATH=/usr/include/gdal + export C_INCLUDE_PATH=/usr/include/gdal + +then:: + + git clone git@github.com:PatBall1/detectree2.git + cd detectree2 + pip install . # (add -e flag to allow editable installs) + +On other systems the process is more involved especially if root access is not available. + + +.. todo:: + + * https://detectron2.readthedocs.io/en/latest/tutorials/install.html + * http://www.tekroi.in/detectron2/projects/DensePose/setup.py + * https://stackoverflow.com/questions/66738473/installing-pytorch-with-cuda-in-setup-py + + +Possible issues with GDAL +------------------------- +GDAL presents a number of complexities. The issue is covered in `gdal/issue `_ We must point to the location of the preinstalled GDAL headers, and the GDAL version must match the pip package version. https://github.com/OSGeo/gdal/issues/2293 +For instance, on my cluster:: + + gdal-config -v # gives 3.0.4 + +So this means we must install the corresponding pip version: ``GDAL==3.0.4`` or lower. To avoid problems, the version of the system GDAL should be higher than the version bindings. +.. + https://gis.stackexchange.com/questions/188639/adding-gdal-as-dependency-to-python-package +In the event that GDAL does not exist on the system, install it as so (assuming root access):: + + sudo apt install libgdal-dev gdal-bin + + + + + +=============================== +Building Detectree2 using Conda +================================ +Many of the aforementioned complexities with pip, GDAL and detectron2 can be solved with Conda. This is currently working for python 3.9.13, in branch `matt/conda `_. The most important file is `environment.yaml `_ which specifies the required dependencies. + +Install miniconda, and source (usually ``~/.miniconda/bin/activate`` if not in ``.bashrc`` already). Begin by installing ``mamba``:: + + conda install mamba -c conda-forge + +And then create the detectree2 environment: + + mamba env create -f envrironment.yaml + mamba activate detectree2env + +Alternatively we may use a conda lock file which has transitive dependencies pinned. This improves reproducibility.:: + + mamba create --name detectree2env --file conda-linux-64.lock + +and if we modify our environment, we can update the lock file as so:: + + conda-lock -k explicit --conda mamba + +and then update conda packages based on the regenerated lock file:: + + mamba update --file conda-linux-64.lock + +The downside of this approach is that it takes much longer to install compared to pip, even with Mamba's improved dependency resolution. + +.. todo:: + + * Determine how this can be integrated into current pip install without breaking ``colab`` pip deployment. + * Investigate use of poetry as it is easier to package a distribution. But detectron2 is not PEP517 compliant. + * It is possible to combine Conda and Poetry, where Conda is used for packages like GDAL / detectron2 / openCV. + + To use detectree2, first install it using: diff --git a/setup.py b/setup.py index d61f3aae..d693c873 100644 --- a/setup.py +++ b/setup.py @@ -1,5 +1,16 @@ from setuptools import find_packages, setup +try: + import torch # noqa: F401 +except ImportError: + raise Exception(""" +You must install PyTorch prior to installing DensePose: +pip install torch + +For more information: + https://pytorch.org/get-started/locally/ + """) + setup( name="detectree2", version="0.0.1", From e3fcf12a751556a94448be0ce7a3d7f0695c770d Mon Sep 17 00:00:00 2001 From: Matthew Archer Date: Sun, 25 Sep 2022 15:02:08 +0100 Subject: [PATCH 2/7] Change exception comment to detectree2 --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index d693c873..06084acb 100644 --- a/setup.py +++ b/setup.py @@ -4,7 +4,7 @@ import torch # noqa: F401 except ImportError: raise Exception(""" -You must install PyTorch prior to installing DensePose: +You must install PyTorch prior to installing Detectree2: pip install torch For more information: From 416c80652b93f7f7217cc4caf9fe263fe5abd968 Mon Sep 17 00:00:00 2001 From: Matthew Archer Date: Sun, 25 Sep 2022 15:42:53 +0100 Subject: [PATCH 3/7] additional dependency --- docs/source/installation.rst | 4 +++- setup.py | 5 +++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/source/installation.rst b/docs/source/installation.rst index f485eec8..7c6536a3 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation.rst @@ -6,7 +6,7 @@ Building Detectree2 Getting up and running quickly ============================== -To get up and running quickly it is possible to install detectree2 and its dependencies with Conda. Simply do:: +For users wanting to run detectree2 only it is possible to install detectree2 and its dependencies with Conda. Simply do:: conda install -c ma595 detectree2 @@ -35,6 +35,8 @@ Then point to preinstalled GDAL header files (not necessary if include directory export CPLUS_INCLUDE_PATH=/usr/include/gdal export C_INCLUDE_PATH=/usr/include/gdal +Depending on the version of setuptools installed, it may be necessary to downgrade to install GDAL. This may in the process break pytorch. + then:: git clone git@github.com:PatBall1/detectree2.git diff --git a/setup.py b/setup.py index 06084acb..c65b353a 100644 --- a/setup.py +++ b/setup.py @@ -29,7 +29,12 @@ "pypng", "pygeos", "geopandas", +<<<<<<< HEAD "rasterio==1.3.2", +======= + "opencv-python", + "rasterio==1.2.10", +>>>>>>> b8a1ce3 (additional dependency) "fiona", "pycrs", "descartes", From 7d0f5539c88abb0195ff47a5022cc36b79638842 Mon Sep 17 00:00:00 2001 From: Matthew Archer Date: Sun, 25 Sep 2022 16:06:52 +0100 Subject: [PATCH 4/7] save file this time --- setup.py | 4 ---- 1 file changed, 4 deletions(-) diff --git a/setup.py b/setup.py index c65b353a..e69c5246 100644 --- a/setup.py +++ b/setup.py @@ -29,12 +29,8 @@ "pypng", "pygeos", "geopandas", -<<<<<<< HEAD - "rasterio==1.3.2", -======= "opencv-python", "rasterio==1.2.10", ->>>>>>> b8a1ce3 (additional dependency) "fiona", "pycrs", "descartes", From 746b762771ae3bf562926176c8fade1ec4e21cc9 Mon Sep 17 00:00:00 2001 From: Matthew Archer Date: Sun, 25 Sep 2022 16:07:33 +0100 Subject: [PATCH 5/7] bump rasterio --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index e69c5246..d92da2ed 100644 --- a/setup.py +++ b/setup.py @@ -30,7 +30,7 @@ "pygeos", "geopandas", "opencv-python", - "rasterio==1.2.10", + "rasterio==1.3.2", "fiona", "pycrs", "descartes", From 4c6ed6e0180f35755bb8f87bc8b66b2f9f38fc9d Mon Sep 17 00:00:00 2001 From: Matthew Archer Date: Tue, 4 Oct 2022 12:42:49 +0100 Subject: [PATCH 6/7] added docs + updated dependencies --- docs/source/installation.rst | 43 ++++++++++++++++++++---------------- setup.py | 2 +- 2 files changed, 25 insertions(+), 20 deletions(-) diff --git a/docs/source/installation.rst b/docs/source/installation.rst index 7c6536a3..421b7ebe 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation.rst @@ -6,45 +6,50 @@ Building Detectree2 Getting up and running quickly ============================== -For users wanting to run detectree2 only it is possible to install detectree2 and its dependencies with Conda. Simply do:: +For users wanting to run detectree2 only it is possible to install detectree2 and its dependencies with Conda. This assumes CUDA 11.1 compatibility. Simply do:: conda install -c ma595 detectree2 - - ========= Using pip ========= -It is easy to install detectree2 on your own system. Simply pip install and all dependencies will be installed automatically. - - See workflow `python-app.yaml `_ workflow for a working CPU deployment. +It is easy to install detectree2 on your own system. -First install ``pytorch``, ``torchvision`` and ``torchaudio`` (compatible versions https://pypi.org/project/torchvision/). Follow `https://pytorch.org/get-started/locally/`_ to get compatible version for your system. Below we run through the process with pip but the conda approach is equally valid. +First install ``pytorch``, ``torchvision`` and ``torchaudio`` (compatible versions https://pypi.org/project/torchvision/). Follow `https://pytorch.org/get-started/locally/`_ to get compatible version for your system (with or without GPU support). Below we run through the process with pip. By default, ``pip install torch torchvision``, will install compatible versions for CPU and GPU with CUDA 10.2 support. It is possible to point to pre-built wheels that are compatible with your system: This can be done inside ``virtualenv`` (if root access is unavailable):: - python3 -m venv ./venv # (check version of python is sufficiently high >=3.7, required by detectron2) - . venv/bin/activate - pip install --upgrade pip - pip install wheel - pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113 + python3 -m venv ./venv # (check version of python is sufficiently high >=3.7, required by detectron2) + . venv/bin/activate + pip install --upgrade pip + pip install wheel + pip install torch==1.10 torchvision==0.11.1 -f https://download.pytorch.org/whl/cu111/torch_stable.html + +Then install detectron2 from a wheel:: + + pip install detectron2==0.6 -f https://dl.fbaipublicfiles.com/detectron2/wheels/index.html + +Or from source:: + + python -m pip install 'git+https://github.com/facebookresearch/detectron2.git' Then point to preinstalled GDAL header files (not necessary if include directory is already in your path):: - export CPLUS_INCLUDE_PATH=/usr/include/gdal - export C_INCLUDE_PATH=/usr/include/gdal + export CPLUS_INCLUDE_PATH=/usr/include/gdal + export C_INCLUDE_PATH=/usr/include/gdal Depending on the version of setuptools installed, it may be necessary to downgrade to install GDAL. This may in the process break pytorch. -then:: +Then simply pip install and all remaining dependencies will be installed automatically:: - git clone git@github.com:PatBall1/detectree2.git - cd detectree2 - pip install . # (add -e flag to allow editable installs) + git clone git@github.com:PatBall1/detectree2.git + cd detectree2 + pip install . # (add -e flag to allow editable installs) On other systems the process is more involved especially if root access is not available. +See workflow `python-ci.yaml `_ workflow for a working CPU deployment. .. todo:: @@ -111,4 +116,4 @@ To use detectree2, first install it using: .. code-block:: console - (.venv) $ pip install detectree2 \ No newline at end of file + (.venv) $ pip install detectree2 diff --git a/setup.py b/setup.py index d92da2ed..53e4482d 100644 --- a/setup.py +++ b/setup.py @@ -30,7 +30,7 @@ "pygeos", "geopandas", "opencv-python", - "rasterio==1.3.2", + "rasterio==1.3a3", "fiona", "pycrs", "descartes", From 3f2c3d82606a64dc36165018482442c076f26e4c Mon Sep 17 00:00:00 2001 From: Matthew Archer Date: Tue, 4 Oct 2022 12:53:14 +0100 Subject: [PATCH 7/7] added docs --- docs/source/installation.rst | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/source/installation.rst b/docs/source/installation.rst index 421b7ebe..437aeedb 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation.rst @@ -14,9 +14,9 @@ For users wanting to run detectree2 only it is possible to install detectree2 an Using pip ========= -It is easy to install detectree2 on your own system. +It is easy to install detectree2 on your own system with Pip. -First install ``pytorch``, ``torchvision`` and ``torchaudio`` (compatible versions https://pypi.org/project/torchvision/). Follow `https://pytorch.org/get-started/locally/`_ to get compatible version for your system (with or without GPU support). Below we run through the process with pip. By default, ``pip install torch torchvision``, will install compatible versions for CPU and GPU with CUDA 10.2 support. It is possible to point to pre-built wheels that are compatible with your system: +First install ``pytorch``, and ``torchvision``. Follow `https://pytorch.org/get-started/locally/`_ to get compatible version for your system (with or without GPU support). By default, ``pip install torch torchvision``, will install compatible versions for CPU and GPU with CUDA 10.2 support. It is possible to point to pre-built wheels that are compatible with your system / GPU see `https://docs.nvidia.com/deeplearning/cudnn/support-matrix/index.html`_. Below we point to the CUDA 11.1 wheel. This can be done inside ``virtualenv`` (if root access is unavailable):: @@ -26,11 +26,11 @@ This can be done inside ``virtualenv`` (if root access is unavailable):: pip install wheel pip install torch==1.10 torchvision==0.11.1 -f https://download.pytorch.org/whl/cu111/torch_stable.html -Then install detectron2 from a wheel:: +Then install detectron2 either from a wheel:: pip install detectron2==0.6 -f https://dl.fbaipublicfiles.com/detectron2/wheels/index.html -Or from source:: +Or from source (for unsupported CUDA versions):: python -m pip install 'git+https://github.com/facebookresearch/detectron2.git' @@ -47,6 +47,10 @@ Then simply pip install and all remaining dependencies will be installed automat cd detectree2 pip install . # (add -e flag to allow editable installs) +Or:: + + pip install detectree2 + On other systems the process is more involved especially if root access is not available. See workflow `python-ci.yaml `_ workflow for a working CPU deployment.