# Compiling MXNet

# Install Prerequisites

Deps.

```bash id=af47fbe7-3023-4e9e-a560-485078c5376e
apt-get -qq update
apt-get install --no-install-recommends \
  software-properties-common apt-transport-https \
  build-essential cmake libjemalloc-dev \
  libatlas-base-dev liblapack-dev liblapacke-dev libopenblas-dev libopencv-dev \
  libcurl4-openssl-dev libzmq3-dev ninja-build libhdf5-dev libomp-dev
apt-get clean
rm -r /var/lib/apt/lists/* # Clear package list so it isn't stale
```

Install base Python.

```bash id=089a9622-a797-4922-918d-1e6cac5e7896
[reference][nextjournal#reference#729fa78f-66b7-41d8-9eae-91d4c3fc3ddb][reference][nextjournal#reference#bc6fb2c4-aa15-4ee7-95a8-a85edf563daa]
```

Install other Python stuff.  We'll use the Intel channel because it has all of the MKL stuff, and most of the other things we need.

```bash id=b3b89741-7e00-4846-ac6a-e9907e5441fb
conda install \
  setuptools six nose boto3 \
  pylint=2.1.1 'requests<2.19.0,>=2.18.4' \
  'numpy<=1.15.2,>=1.8.2' scipy=1.0.1 h5py=2.8.0
conda clean -tipsy

pip install cpplint==1.3.0 nose-timer

ldconfig
```

Install Maven.

```bash id=ad714557-6b68-49a6-bef7-40ea207e64ec
MAV_VER="3.6.0"
FILE="apache-maven-${MAV_VER}-bin.tar.gz"

wget --progress=bar:force -O $FILE \
  http://ftp.naz.com/apache/maven/maven-3/${MAV_VER}/binaries/$FILE

tar -zxf $FILE -C /usr/local
rm $FILE

ln -sf /usr/local/apache-maven-${MAV_VER} /usr/local/maven

/usr/local/maven/bin/mvn --version
```

# Compile

Get MXNet source.

```bash id=0553e588-7c09-4a1f-8f34-ed2d25fb673c
rm -rf /mxnet
git clone --recursive https://github.com/apache/incubator-mxnet /mxnet
cd mxnet
git checkout v1.4.x
git submodule update --init --recursive
```

The compilation.  Needs absurd amounts of RAM (over 10 GB), takes over three hours.  We edit the cmake defs to set the minimum CPU to the Sandy Bridge family.  Also set a bunch of options—of special interest are:

* USE_OLDCMAKECUDA because a newer cmake build setup fails with CUDA
* CUDA_ARCH_\* set which 'Compute Capability' identifiers we compile for—the current setting builds binary compatability for K80, P4, P100, and V100 GPUs, and JIT capability for the entire Kepler, Pascal, and Volta families
* USE_SSE is set off, because we do not need that specifier when we set -march
* IS_CONTAINER_BUILD makes things less picky about whether or not a GPU is currently available

```bash id=03242ebb-eaa6-4106-b2a6-3edcfce5b75b
ls /usr/local/cuda/bin
```

```bash id=fb9697ac-288d-4b42-b3ad-f89f8b34ec3d
cd /mxnet

sed -i 's#30 35 50 52 60 61 70 75#37 60 61 70#' Makefile

make -j5 \
  USE_CUDA=1 USE_CUDA_PATH="/usr/local/cuda" USE_CUDNN=1 \
  USE_OPENMP=1 USE_MKLDNN=1 USE_OPENCV=1 USE_F16C=1 \
  USE_JEMALLOC=1 USE_THREADED_ENGINE=1 \
  ADD_CFLAGS="-march=sandybridge -mtune=generic" \
  ADD_LDFLAGS="-L/usr/local/cuda/lib64/stubs" \
  USE_LAPACK=1 USE_LAPACK_PATH="/usr/lib/x86_64-linux-gnu" USE_BLAS="openblas" \
  VERBOSE=1
```

```bash id=f14f7b7c-14a6-4c71-92e2-9fb5642604c4
cd /mxnet

export PATH="/usr/local/maven/bin:$PATH:/mxnet/lib:/opt/conda/lib"

make scalapkg USE_LAPACK_PATH="/usr/lib/x86_64-linux-gnu" VERBOSE=1
```

```bash id=348a0696-8d7c-4a2d-b1e9-c87b44d95dc1
cd /mxnet

sed -i 's#"-mtune=generic"#"-march=sandybridge -mtune=generic"#' CMakeLists.txt

rm -rf build; mkdir build; cd build

cmake \
  -DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs" \
  -DCMAKE_SHARED_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs" \
  -DUSE_OLDCMAKECUDA=ON -DUSE_CUDA=ON -DUSE_NCCL=ON -DCUDA_ARCH_NAME="Manual" \
  -DCUDA_ARCH_BIN="3.7 6.0 6.1 7.0" -DCUDA_ARCH_PTX="3.0 6.0 7.0" \
  -DUSE_SSE=OFF -DBUILD_CPP_EXAMPLES=OFF -DIS_CONTAINER_BUILD=TRUE ..

make -j5 --no-print-directory VERBOSE=1
```

```bash id=dafca3ac-54fe-4015-a934-e7cd25041e53
cd /mxnet/lib
tar -zcf /results/mxnetlibs.tar.gz *
```

[mxnetlibs.tar.gz][nextjournal#output#dafca3ac-54fe-4015-a934-e7cd25041e53#mxnetlibs.tar.gz]

```bash id=d35413ea-45b1-491a-9fea-84f1988ed459
cp [reference][nextjournal#reference#4a55a914-9d6c-4e3f-9157-34f5f77a8ccc] /mxnetlibs.tar.gz
```

[nextjournal#reference#729fa78f-66b7-41d8-9eae-91d4c3fc3ddb]:
<#nextjournal#reference#729fa78f-66b7-41d8-9eae-91d4c3fc3ddb>

[nextjournal#reference#bc6fb2c4-aa15-4ee7-95a8-a85edf563daa]:
<#nextjournal#reference#bc6fb2c4-aa15-4ee7-95a8-a85edf563daa>

[nextjournal#output#dafca3ac-54fe-4015-a934-e7cd25041e53#mxnetlibs.tar.gz]:
<https://nextjournal.com/data/QmXJVdSuMS14wc87ZHg9iQBnrgB1GkYLcviNLZXuG7jovd?content-type=application/gzip&node-id=dafca3ac-54fe-4015-a934-e7cd25041e53&filename=mxnetlibs.tar.gz&node-kind=output>

[nextjournal#reference#4a55a914-9d6c-4e3f-9157-34f5f77a8ccc]:
<#nextjournal#reference#4a55a914-9d6c-4e3f-9157-34f5f77a8ccc>

<details id="com.nextjournal.article">
<summary>This notebook was exported from <a href="https://nextjournal.com/a/KZbMJaxwwDnC3RtYphw3q?change-id=CN3mqXen8Ji6Ts8PcoQfLK">https://nextjournal.com/a/KZbMJaxwwDnC3RtYphw3q?change-id=CN3mqXen8Ji6Ts8PcoQfLK</a></summary>

```edn nextjournal-metadata
{:article
 {:settings nil,
  :nodes
  {"03242ebb-eaa6-4106-b2a6-3edcfce5b75b"
   {:compute-ref #uuid "06c2bd2b-f268-40a0-9d5d-eef2f9eed3fe",
    :exec-duration 855,
    :id "03242ebb-eaa6-4106-b2a6-3edcfce5b75b",
    :kind "code",
    :output-log-lines {:stdout 16},
    :runtime [:runtime "40ca8824-6d79-4033-9d49-d1bf8538b5bf"]},
   "0553e588-7c09-4a1f-8f34-ed2d25fb673c"
   {:compute-ref #uuid "02dee176-821e-453f-a29a-b87e065c0e3b",
    :exec-duration 35875,
    :id "0553e588-7c09-4a1f-8f34-ed2d25fb673c",
    :kind "code",
    :output-log-lines {:stdout 54},
    :runtime [:runtime "40ca8824-6d79-4033-9d49-d1bf8538b5bf"]},
   "089a9622-a797-4922-918d-1e6cac5e7896"
   {:compute-ref #uuid "0efab2bf-8043-415a-b61e-a8e1361f2017",
    :exec-duration 60535,
    :id "089a9622-a797-4922-918d-1e6cac5e7896",
    :kind "code",
    :output-log-lines {:stdout 228},
    :runtime [:runtime "95d95862-3002-4f9c-a1a3-be66bca57ff8"]},
   "348a0696-8d7c-4a2d-b1e9-c87b44d95dc1"
   {:compute-ref #uuid "891ecd8f-a6c7-4c60-a231-04b6096704c3",
    :exec-duration 9143412,
    :id "348a0696-8d7c-4a2d-b1e9-c87b44d95dc1",
    :kind "code",
    :output-log-lines {:stdout 54},
    :runtime [:runtime "40ca8824-6d79-4033-9d49-d1bf8538b5bf"]},
   "40ca8824-6d79-4033-9d49-d1bf8538b5bf"
   {:environment [:environment "95d95862-3002-4f9c-a1a3-be66bca57ff8"],
    :environment? true,
    :id "40ca8824-6d79-4033-9d49-d1bf8538b5bf",
    :kind "runtime",
    :language "bash",
    :name "MXNet Compile",
    :resources {:machine-type "n1-highmem-4"},
    :type :nextjournal,
    :runtime/checkpoint? true},
   "4a55a914-9d6c-4e3f-9157-34f5f77a8ccc"
   {:id "4a55a914-9d6c-4e3f-9157-34f5f77a8ccc",
    :kind "reference",
    :link
    [:output
     "dafca3ac-54fe-4015-a934-e7cd25041e53"
     "mxnetlibs.tar.gz"]},
   "729fa78f-66b7-41d8-9eae-91d4c3fc3ddb"
   {:id "729fa78f-66b7-41d8-9eae-91d4c3fc3ddb",
    :kind "reference",
    :link
    [:output
     {:article/nextjournal.id
      #uuid "5b45e08b-5b96-413e-84ed-f03b5b65bd66",
      :change/nextjournal.id
      #uuid "5bedbf77-553c-4476-8417-77d2d832aaf1",
      :node/id "3cc976d4-8020-41a9-b2d8-2bdc2710310d"}
     nil]},
   "95d95862-3002-4f9c-a1a3-be66bca57ff8"
   {:environment
    [:environment
     {:article/nextjournal.id
      #uuid "5b45eb52-bad4-413d-9d7f-b2b573a25322",
      :change/nextjournal.id
      #uuid "5bf981e8-5a86-4c9e-b604-76cf893ffa37",
      :node/id "0ae15688-6f6a-40e2-a4fa-52d81371f733"}],
    :environment? true,
    :id "95d95862-3002-4f9c-a1a3-be66bca57ff8",
    :kind "runtime",
    :language "clojure",
    :name "Prereqs",
    :resources {:machine-type "n1-standard-2"},
    :type :nextjournal,
    :docker/environment-image
    "eu.gcr.io/nextjournal-com/environment@sha256:1e4110269c82a73cc2ee6488de65285d80edf9379273d317aa49561586c32aad",
    :runtime/environment-variables
    [{:name "PATH",
      :value
      "/usr/local/maven/bin:/opt/conda/bin:/usr/local/graalvm/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"}]},
   "ad714557-6b68-49a6-bef7-40ea207e64ec"
   {:id "ad714557-6b68-49a6-bef7-40ea207e64ec",
    :kind "code",
    :runtime [:runtime "40ca8824-6d79-4033-9d49-d1bf8538b5bf"]},
   "af47fbe7-3023-4e9e-a560-485078c5376e"
   {:compute-ref #uuid "bbf8d725-1f41-4836-9f9f-4d354532c107",
    :exec-duration 103856,
    :id "af47fbe7-3023-4e9e-a560-485078c5376e",
    :kind "code",
    :output-log-lines {:stdout 2144},
    :runtime [:runtime "95d95862-3002-4f9c-a1a3-be66bca57ff8"]},
   "b3b89741-7e00-4846-ac6a-e9907e5441fb"
   {:compute-ref #uuid "055357eb-78fe-406c-9f74-878d761ef310",
    :exec-duration 116363,
    :id "b3b89741-7e00-4846-ac6a-e9907e5441fb",
    :kind "code",
    :output-log-lines {:stdout 230},
    :runtime [:runtime "95d95862-3002-4f9c-a1a3-be66bca57ff8"]},
   "bc6fb2c4-aa15-4ee7-95a8-a85edf563daa"
   {:id "bc6fb2c4-aa15-4ee7-95a8-a85edf563daa",
    :kind "reference",
    :link
    [:output
     {:article/nextjournal.id
      #uuid "5b45e08b-5b96-413e-84ed-f03b5b65bd66",
      :change/nextjournal.id
      #uuid "5bedbf77-553c-4476-8417-77d2d832aaf1",
      :node/id "3c052b77-1240-4dc0-a9f9-9607eab2156f"}
     nil]},
   "bd65c37c-2d1c-48c0-8975-c8267f369aaa"
   {:environment
    [:environment
     {:article/nextjournal.id
      #uuid "5b45dad0-dfdf-4576-9b8c-f90892e74c94",
      :change/nextjournal.id
      #uuid "5bf80f64-4f61-4bb8-992e-aa552608e59f",
      :node/id "01204d2e-3c51-4802-88f3-a128e4d85a3e"}],
    :environment? true,
    :id "bd65c37c-2d1c-48c0-8975-c8267f369aaa",
    :kind "runtime",
    :language "bash",
    :name "Compiled MXNet Libs",
    :type :nextjournal,
    :docker/environment-image
    "eu.gcr.io/nextjournal-com/environment@sha256:8ca6db451ebf818170ace6f378b41508858e430c36fd4de2da05d300ad53a0ae"},
   "d35413ea-45b1-491a-9fea-84f1988ed459"
   {:compute-ref #uuid "52b5e86b-a700-4196-aef6-5b31bb3393a4",
    :exec-duration 454,
    :id "d35413ea-45b1-491a-9fea-84f1988ed459",
    :kind "code",
    :output-log-lines {},
    :runtime [:runtime "bd65c37c-2d1c-48c0-8975-c8267f369aaa"]},
   "dafca3ac-54fe-4015-a934-e7cd25041e53"
   {:compute-ref #uuid "f113ffac-d073-41a6-9be5-44afc06f5a16",
    :exec-duration 47843,
    :id "dafca3ac-54fe-4015-a934-e7cd25041e53",
    :kind "code",
    :output-log-lines {},
    :runtime [:runtime "40ca8824-6d79-4033-9d49-d1bf8538b5bf"]},
   "f14f7b7c-14a6-4c71-92e2-9fb5642604c4"
   {:compute-ref #uuid "99eff219-a4d4-4275-a6be-d23f9690801c",
    :exec-duration 685,
    :id "f14f7b7c-14a6-4c71-92e2-9fb5642604c4",
    :kind "code",
    :output-log-lines {:stdout 10},
    :runtime [:runtime "40ca8824-6d79-4033-9d49-d1bf8538b5bf"]},
   "fb9697ac-288d-4b42-b3ad-f89f8b34ec3d"
   {:compute-ref #uuid "324c2354-e8d0-461e-b33a-c4acc705dace",
    :exec-duration 17503,
    :id "fb9697ac-288d-4b42-b3ad-f89f8b34ec3d",
    :kind "code",
    :output-log-lines {:stdout 8},
    :runtime [:runtime "40ca8824-6d79-4033-9d49-d1bf8538b5bf"]}},
  :nextjournal/id #uuid "0297793a-1d09-4c66-a4e5-42a6cc574f8c",
  :article/change
  {:nextjournal/id #uuid "5c04da77-252c-48a8-bc24-f8db791c7410"}}}

```
</details>