Skip to content
This repository was archived by the owner on Apr 28, 2023. It is now read-only.

Commit acb27de

Browse files
prigoyalprigoyal
committed
Add a missing deps to docs and allow building c2 from conda package
1 parent e15ee0a commit acb27de

File tree

10 files changed

+234
-75
lines changed

10 files changed

+234
-75
lines changed

‎CMakeLists.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -214,7 +214,7 @@ include(cmake/GetGitRevisionDescription.cmake)
214214

215215
################################################################################
216216

217-
# gflag
217+
# gflags
218218
message("find_library(GFLAGS_LIBRARIES gflags")
219219
find_library(GFLAGS_LIBRARIES gflags PATHS ${CMAKE_INSTALL_PREFIX} PATH_SUFFIXES lib lib64)
220220
message(STATUS "Found gflags: ${GFLAGS_LIBRARIES}")

‎build.sh‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -416,9 +416,14 @@ if ! test -z $aten || ! test -z $all; then
416416
fi
417417

418418
if ! test -z $caffe2 || ! test -z $all ; then
419-
if [ "$WITH_CAFFE2" == "ON" ]; then
420-
install_caffe2
421-
fi
419+
if [ "$WITH_CAFFE2" == "ON" ]; then
420+
if [[ $(find $CONDA_PREFIX -name libcaffe2_gpu.so) ]]; then
421+
echo "caffe2 found"
422+
else
423+
echo "no files found"
424+
install_caffe2
425+
fi
426+
fi
422427
fi
423428

424429
if ! test -z $isl || ! test -z $all; then

‎docs/source/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ Tensor Comprehensions provides framework-Agnostic Abstractions for High-Performa
3838
installation_conda_dep
3939
installation_conda
4040
installation_non_conda
41+
installation_caffe2_integration
4142

4243
.. toctree::
4344
:maxdepth: 1

‎docs/source/installation_caffe2_integration.rst‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
Using Tensor Comprehensions with Caffe2
2+
=======================================
3+
4+
We provide very basic integration of Tensor Comprehensions with Caffe2. Follow
5+
the instructions below on how to build Tensor Comprehensions so that Caffe2 works
6+
fine with it. We only support :code:`conda` environment build for this right now.
7+
8+
Step 1: Install system dependencies
9+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10+
11+
.. code-block:: bash
12+
13+
$ apt-get update
14+
$ apt-get install -y wget cmake git build-essential unzip curl automake build-essential realpath libtool software-properties-common
15+
16+
17+
Step 2: Setup gcc / g++
18+
^^^^^^^^^^^^^^^^^^^^^^^
19+
20+
First, check your gcc/g++ and make sure they are in system path, somewhere like :code:`/usr/bin`. Also, check your gcc/g++ version. Currently, TC officially supports :code:`gcc 4.8` when installing using conda packages.
21+
22+
.. code-block:: bash
23+
24+
$ which gcc
25+
$ which g++
26+
27+
If you don't have correct gcc, g++, follow the instructions below, otherwise skip:
28+
29+
.. code-block:: bash
30+
31+
$ add-apt-repository ppa:ubuntu-toolchain-r/test
32+
$ apt-get update
33+
$ apt-get install -y --no-install-recommends libcilkrts5 gcc-4.8 g++-4.8
34+
35+
*Optionally*, you can configure the gcc versions available on your machine and set up higher priority for the version you want.
36+
37+
.. code-block:: bash
38+
39+
$ update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 50
40+
$ update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 50
41+
42+
43+
Step 3: Install Anaconda3
44+
^^^^^^^^^^^^^^^^^^^^^^^^^^
45+
In order to contribute to TC python/C++ API, you need to install TC from source. For this,
46+
:code:`anaconda3` is required. Install :code:`anaconda3` by following the instructions below:
47+
48+
.. code-block:: bash
49+
50+
$ cd $HOME
51+
$ wget https://repo.continuum.io/archive/Anaconda3-5.0.1-Linux-x86_64.sh -O anaconda3.sh
52+
$ chmod +x anaconda3.sh
53+
$ ./anaconda3.sh -b -p $HOME/anaconda3 && rm anaconda3.sh
54+
55+
Now add :code:`anaconda3` to your :code:`PATH` so that you can use it. For that run the following command:
56+
57+
.. code-block:: bash
58+
59+
$ export PATH=$HOME/anaconda3/bin:$PATH
60+
61+
Now, verify your conda installation and check the version:
62+
63+
.. code-block:: bash
64+
65+
$ which conda
66+
67+
This command should print the path of your conda bin. If it doesn't, make sure conda is in your :code:`PATH`.
68+
69+
Step 4: Get CUDA and CUDNN
70+
^^^^^^^^^^^^^^^^^^^^^^^^^^
71+
In order to build TC, you also need to have :code:`CUDA` and :code:`CUDNN`. If you already have it
72+
you can just export the :code:`PATH`, :code:`LD_LIBRARY_PATH` (see the end of this step). If you don't have CUDA/CUDNN, then follow the instructions below:
73+
74+
First, install :code:`CUDA` Toolkit v8.0 (skip if you have it):
75+
76+
.. code-block:: bash
77+
78+
$ CUDA_REPO_PKG="cuda-repo-ubuntu1604_8.0.61-1_amd64.deb"
79+
$ wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/${CUDA_REPO_PKG}
80+
$ dpkg -i ${CUDA_REPO_PKG}
81+
$ rm -f ${CUDA_REPO_PKG}
82+
$ apt-get update
83+
$ apt-get -y install cuda
84+
85+
Now, Install cuDNN v6.0 (skip if you have it already):
86+
87+
.. code-block:: bash
88+
89+
$ CUDNN_TAR_FILE="cudnn-8.0-linux-x64-v6.0.tgz"
90+
$ wget http://developer.download.nvidia.com/compute/redist/cudnn/v6.0/${CUDNN_TAR_FILE}
91+
$ tar -xzvf ${CUDNN_TAR_FILE}
92+
$ cp -P cuda/include/cudnn.h /usr/local/cuda/include
93+
$ cp -P cuda/lib64/libcudnn* /usr/local/cuda/lib64/
94+
$ chmod a+r /usr/local/cuda/lib64/libcudnn*
95+
96+
.. note::
97+
98+
Please use :code:`sudo` to run the command that might fail with permission issues. Otherwise, run
99+
the commands as is.
100+
101+
Set environment variables:
102+
103+
.. code-block:: bash
104+
105+
$ export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/targets/x86_64-linux/lib/stubs/:$LD_LIBRARY_PATH
106+
$ export PATH=/usr/local/bin:/usr/local/cuda/bin:$PATH
107+
108+
.. _conda_dep_install_tc:
109+
110+
Step 5: Install TC with Caffe2
111+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
112+
113+
We ship conda packages for most of TC dependencies like :code:`clang+llvm`, :code:`glog`,
114+
:code:`gflags`, :code:`protobuf3`, :code:`halide`. We will directly install the
115+
conda packages of TC dependencies and then build TC.
116+
117+
.. code-block:: bash
118+
119+
$ conda create -y --name tc-build-conda python=3.6
120+
$ source activate tc-build-conda
121+
$ conda install -y -c prigoyal llvm-tapir50 isl-tc gflags glog caffe2 protobuf
122+
$ conda install -y -c pytorch pytorch
123+
$ cd $HOME && git clone https://github.com/facebookresearch/TensorComprehensions.git --recursive
124+
$ cd TensorComprehensions && git submodule update --init --recursive
125+
$ BUILD_TYPE=Release INSTALL_PREFIX=$CONDA_PREFIX CLANG_PREFIX=$(llvm-config --prefix) ./build.sh --all
126+
127+
.. note::
128+
Please also make sure that you don't have gflags or glog in your system path. Those might conflict with the TC gflags/glog.
129+
130+
131+
Step 6: Run TC Caffe2 Python test
132+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
133+
134+
.. code-block:: bash
135+
136+
$ cd $HOME/TensorComprehensions
137+
$ python test_python/test_c2.py # if you have GPU

‎docs/source/installation_conda.rst‎

Lines changed: 6 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ If you don't have correct gcc, g++, follow the instructions below, otherwise ski
4040
4141
$ add-apt-repository ppa:ubuntu-toolchain-r/test
4242
$ apt-get update
43+
$ apt-get install -y --no-install-recommends libcilkrts5
4344
$ apt-get install -y --no-install-recommends gcc-5 g++-5
4445
4546
*Optionally*, you can configure the gcc versions available on your machine and set up higher priority for the version you want.
@@ -190,8 +191,7 @@ Now, you need to install TC from source. For installing TC from source, checkout
190191
.. code-block:: bash
191192
192193
$ cd $HOME && git clone https://github.com/facebookresearch/TensorComprehensions.git --recursive
193-
$ cd TensorComprehensions
194-
$ git submodule update --init --recursive
194+
$ cd TensorComprehensions && git submodule update --init --recursive
195195
$ conda install -y pyyaml
196196
$ export TC_DIR=$(pwd)
197197
$ BUILD_TYPE=Release PYTHON=$(which python3) WITH_CAFFE2=OFF CLANG_PREFIX=$HOME/clang+llvm-tapir5.0 ./build.sh --all
@@ -212,25 +212,12 @@ Step 8: Verify TC installation
212212
213213
Build with Basic Caffe2 Integration
214214
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
215-
1. If you want to install TC with Caffe2 as well, run the following:
216215

217-
.. code-block:: bash
218-
219-
$ BUILD_TYPE=Release PYTHON=$(which python3) WITH_PYTHON_C2=OFF CLANG_PREFIX=$HOME/clang+llvm-tapir5.0 ./build.sh --all
220-
221-
222-
.. note::
223-
224-
This turns off the Caffe2 python build. If you want to turn on the Caffe2 python build, see next step:
225-
226-
2. For installing python binaries as well of Caffe2 with TC:
216+
If you want to install TC with Caffe2 as well, run the following:
227217

228218
.. code-block:: bash
229219
230-
$ BUILD_TYPE=Release PYTHON=$(which python3) WITH_PYTHON_C2=ON CLANG_PREFIX=$HOME/clang+llvm-tapir5.0 ./build.sh --all
231-
232-
.. note::
233-
234-
Caffe2 doesn't provide support for pip/conda at the moment and this means in order to use the caffe2 python, you might need to set $PYTHONPATH. Normally, it could be :code:`${TC_DIR}/third-party-install/`
220+
$ conda install -y -c prigoyal caffe2
221+
$ BUILD_TYPE=Release PYTHON=$(which python3) CLANG_PREFIX=$HOME/clang+llvm-tapir5.0 ./build.sh --all
235222
236-
However, please check caffe2 official instructions `here <https://caffe2.ai/docs/getting-started.html?platform=mac&configuration=compile#test-the-caffe2-installation>`_ . TC doesn't yet provide support for caffe2 python usage.
223+
Now, you have the TC bindings with Caffe2 built as well and and you write python examples for TC in caffe2.

‎docs/source/installation_conda_dep.rst‎

Lines changed: 24 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ Step 1: Install system dependencies
1313
.. code-block:: bash
1414
1515
$ apt-get update
16-
$ apt-get install -y wget cmake git build-essential unzip curl automake build-essential realpath libtool
16+
$ apt-get install -y wget cmake git build-essential unzip curl automake build-essential realpath libtool software-properties-common
1717
1818
.. note::
1919

@@ -38,7 +38,7 @@ If you don't have correct gcc, g++, follow the instructions below, otherwise ski
3838
3939
$ add-apt-repository ppa:ubuntu-toolchain-r/test
4040
$ apt-get update
41-
$ apt-get install -y --no-install-recommends gcc-4.8 g++-4.8
41+
$ apt-get install -y --no-install-recommends libcilkrts5 gcc-4.8 g++-4.8
4242
4343
*Optionally*, you can configure the gcc versions available on your machine and set up higher priority for the version you want.
4444

@@ -47,37 +47,8 @@ If you don't have correct gcc, g++, follow the instructions below, otherwise ski
4747
$ update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 50
4848
$ update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 50
4949
50-
Step 3: Get Protobuf3.4
51-
^^^^^^^^^^^^^^^^^^^^^^^
52-
53-
TC officially support protobuf3.4 at the moment. Please follow the below instructions
54-
to install the protobuf.
55-
56-
.. note::
57-
58-
Anaconda3 also has a protobuf3 available but that might not be compatible with TC. So we recommend following the below instructions to install Protobuf3.4
59-
60-
.. code-block:: bash
61-
62-
$ export CC=$(which gcc)
63-
$ export CXX=$(which g++)
64-
$ mkdir -p /tmp/proto-install && cd /tmp/proto-install
65-
$ wget --quiet https://github.com/google/protobuf/archive/v3.4.0.zip -O proto.zip && unzip -qq proto.zip -d .
66-
$ cd protobuf-3.4.0 && ./autogen.sh && ./configure && make -j 8 && make install && ldconfig
67-
68-
.. note::
69-
70-
Please use :code:`sudo` to run the command that might fail with permission issues. Otherwise, run
71-
the commands as is.
72-
7350
74-
Now check your proto version by running:
75-
76-
.. code-block:: bash
77-
78-
$ protoc --version
79-
80-
Step 4: Install Anaconda3
51+
Step 3: Install Anaconda3
8152
^^^^^^^^^^^^^^^^^^^^^^^^^^
8253
In order to contribute to TC python/C++ API, you need to install TC from source. For this,
8354
:code:`anaconda3` is required. Install :code:`anaconda3` by following the instructions below:
@@ -103,7 +74,7 @@ Now, verify your conda installation and check the version:
10374
10475
This command should print the path of your conda bin. If it doesn't, make sure conda is in your :code:`PATH`.
10576

106-
Step 5: Get CUDA and CUDNN
77+
Step 4: Get CUDA and CUDNN
10778
^^^^^^^^^^^^^^^^^^^^^^^^^^
10879
In order to build TC, you also need to have :code:`CUDA` and :code:`CUDNN`. If you already have it
10980
you can just export the :code:`PATH`, :code:`LD_LIBRARY_PATH` (see the end of this step). If you don't have CUDA/CUDNN, then follow the instructions below:
@@ -144,7 +115,7 @@ Set environment variables:
144115
145116
.. _conda_dep_install_tc:
146117

147-
Step 6: Install TC
118+
Step 5: Install TC
148119
^^^^^^^^^^^^^^^^^^
149120

150121
We ship conda packages for most of TC dependencies like :code:`clang+llvm`, :code:`glog`,
@@ -153,23 +124,35 @@ conda packages of TC dependencies and then build TC.
153124

154125
.. code-block:: bash
155126
156-
$ conda create -y --name tc-build-conda python=3.6
157-
$ source activate tc-build-conda
158-
$ conda install -y -c prigoyal llvm-tapir50 isl-tc gflags glog
127+
$ conda create -y --name tc-build-conda python=3.6 && source activate tc-build-conda
128+
$ conda install -y -c prigoyal llvm-tapir50 isl-tc gflags glog protobuf
159129
$ conda install -y -c pytorch pytorch
160130
$ cd $HOME && git clone https://github.com/facebookresearch/TensorComprehensions.git --recursive
161-
$ cd TensorComprehensions
162-
$ git submodule update --init --recursive
131+
$ cd TensorComprehensions && git submodule update --init --recursive
163132
$ BUILD_TYPE=Release INSTALL_PREFIX=$CONDA_PREFIX WITH_CAFFE2=OFF CLANG_PREFIX=$(llvm-config --prefix) ./build.sh --all
164133
165134
.. note::
166-
Please also make sure that you don't have gflags or glog in your system path. Those might conflict with the TC gflags/glog.
135+
Please also make sure that you don't have :code:`gflags` or :code:`glog` in your system path. Those might conflict with the TC gflags/glog.
167136

168-
Step 7: Verify TC installation
137+
138+
Step 6: Verify TC installation
169139
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
170140

171141
.. code-block:: bash
172142
173143
$ cd $HOME/TensorComprehensions
174144
$ ./test.sh # if you have GPU
175145
$ ./test_cpu.sh # if you have only CPU
146+
147+
148+
Build with Basic Caffe2 Integration
149+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
150+
151+
If you want to install TC with Caffe2 as well, run the following:
152+
153+
.. code-block:: bash
154+
155+
$ conda install -y -c prigoyal caffe2
156+
$ BUILD_TYPE=Release INSTALL_PREFIX=$CONDA_PREFIX CLANG_PREFIX=$(llvm-config --prefix) ./build.sh --all
157+
158+
Now, you have the TC bindings with Caffe2 built as well and and you write python examples for TC in caffe2.

‎include/tc/c2/tc_op.h‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -37,13 +37,15 @@ class TcOp : public Operator<Context> {
3737
public:
3838
TcOp(const OperatorDef& operator_def, Workspace* ws)
3939
: caffe2::Operator<Context>(operator_def, ws),
40+
tc_(OperatorBase::GetSingleArgument<std::string>("tcDef", "ERROR")),
41+
tcName_(
42+
OperatorBase::GetSingleArgument<std::string>("tcName", "ERROR")),
4043
mappingOptions_(tc::MappingOptions::makeNaiveMappingOptions()),
4144
gradMappingOptions_(tc::MappingOptions::makeNaiveMappingOptions()) {
42-
tc_ = OperatorBase::GetSingleArgument<std::string>("def", "ERROR");
43-
gradTc_ = OperatorBase::GetSingleArgument<std::string>("grad_def", "ERROR");
44-
tcName_ = OperatorBase::GetSingleArgument<std::string>("name", "ERROR");
45+
gradTc_ =
46+
OperatorBase::GetSingleArgument<std::string>("tcGradDef", "ERROR");
4547
gradTcName_ =
46-
OperatorBase::GetSingleArgument<std::string>("grad_name", "ERROR");
48+
OperatorBase::GetSingleArgument<std::string>("tcGradName", "ERROR");
4749
profile_ = OperatorBase::GetSingleArgument<bool>("profile", false);
4850
ArgumentHelper args(operator_def);
4951
if (args.HasArgument("mappingOptions")) {

‎test/test_caffe2.cc‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -637,8 +637,8 @@ def fun(float(B, N, M) X, float(B, M, K) Y) -> (Z)
637637

638638
Workspace w_test;
639639
init_ws(w_test);
640-
Argument tcArg = MakeArgument<string>("def", tc);
641-
Argument tcNameArg = MakeArgument<string>("name", "fun");
640+
Argument tcArg = MakeArgument<string>("tcDef", tc);
641+
Argument tcNameArg = MakeArgument<string>("tcName", "fun");
642642
MappingOptions options = tc::makeBaseCliStrategy()
643643
.tile({1})
644644
.mapToThreads({128})
@@ -686,8 +686,8 @@ def fun(float(N) X, int(A,B) I) -> (Z) {
686686
}
687687
)TC";
688688

689-
Argument tcArg = MakeArgument<string>("def", tc);
690-
Argument tcNameArg = MakeArgument<string>("name", "fun");
689+
Argument tcArg = MakeArgument<string>("tcDef", tc);
690+
Argument tcNameArg = MakeArgument<string>("tcName", "fun");
691691
MappingOptions options = tc::makeBaseCliStrategy()
692692
.tile({1})
693693
.mapToThreads({128})
@@ -726,8 +726,8 @@ def fun(float(B, N, M) X) -> (Z) {
726726

727727
Workspace w_test;
728728
init_ws(w_test);
729-
Argument tcArg = MakeArgument<string>("def", tc);
730-
Argument tcNameArg = MakeArgument<string>("name", "fun");
729+
Argument tcArg = MakeArgument<string>("tcDef", tc);
730+
Argument tcNameArg = MakeArgument<string>("tcName", "fun");
731731
MappingOptions options = tc::makeBaseCliStrategy()
732732
.tile({1})
733733
.mapToThreads({128})

0 commit comments

Comments
 (0)