Docs build improvements

CONTRIBUTORS.md

@@ -174,3 +174,4 @@ List of Contributors
 * [Hang Zhang](http://hangzh.com)
 * [Kou Ding](https://github.com/chinakook)
 * [Istvan Fehervari](https://github.com/ifeherva)
+* [Aaron Markham](https://github.com/aaronmarkham)

docs/README.md

@@ -8,9 +8,21 @@ MXNet Documentation Website is built with [Sphinx](http://www.sphinx-doc.org) an
 
 ## How to Build the MXNet Website for Development and QA
 
-Using `make docs` from the MXNet root is the quickest way to generate the MXNet API docs and the website. This method automatically generates each API, [except the Perl and R APIs](#other-build-processes).
+Using `make docs` from the MXNet root is the quickest way to generate the MXNet API docs and the website, as long as you already have all of the dependencies installed. This method automatically generates each API, [except the Perl and R APIs](#other-build-processes).
 
-* [Dependencies](https://github.com/apache/incubator-mxnet/tree/master/docs/build_version_doc#dependencies) - required before you do any building of the docs
+**Easy docs setup for Ubuntu:** Run the following on Ubuntu 16.04 to install all MXNet and docs dependencies and to build MXNet from source. Then issue the `make docs` command from the source root to build the docs.
+
+```bash
+git clone --recursive https://github.com/apache/incubator-mxnet.git mxnet
+cd mxnet/docs/build_version_doc
+./setup_docs_ubuntu.sh
+cd ../../
+make docs USE_OPENMP=1
+```
+
+For more information on each API's documentation dependencies, how to serve the docs, or how to build the full website with each legacy MXNet version, refer to the following links:
+
+* [Dependencies](https://github.com/apache/incubator-mxnet/tree/master/docs/build_version_doc#dependencies) - required before you build the docs
 * [Developer Build Instructions](https://github.com/apache/incubator-mxnet/tree/master/docs/build_version_doc#developer-instructions) - build your local branch
 * [Full Site Build Instructions](https://github.com/apache/incubator-mxnet/tree/master/docs/build_version_doc#full-website-build) - build the latest commits to the official branches
 
@@ -37,12 +49,12 @@ The host repo is hooked with [Apache gitbox](https://gitbox.apache.org/repos/asf
 ### Process for Running the Docs Build Job
 
 1. Login to [Jenkins](http://jenkins.mxnet-ci.amazon-ml.com/).
-1. View the pipeline currently called `website build pipeline`.
+1. View the pipeline currently called `restricted website build`.
 1. Click `Build with Parameters`.
 1. Use the defaults, or change the domain to be your staging server's IP/DNS web address.
 1. Wait about 20-30 minutes while it builds the full site.
 1. On your staging server, clone the [mxnet site repo](https://github.com/apache/incubator-mxnet-site.git).
-1. When you ran `website build pipeline` it followed up with website build - test publish which pushed the changes to the incubator-mxnet-site repo.
+1. When you ran `restricted website build` it followed up with `restricted website publish` which pushed the changes to the incubator-mxnet-site repo.
 1. Make sure you git pull if you had already cloned the site repo before this first run-through.
 1. Copy the files to your webroot. For more info on this see the developer instructions for docs build.
 1. Preview the site on your staging server. Note, use the domain default before you try to use this for production, but using your own is helpful for QA'ing the site.

docs/build_version_doc/README.md

@@ -11,12 +11,98 @@ This folder contains a variety of scripts to generate the MXNet.io website as we
 * [Dockerfile](Dockerfile) - has all dependencies needed to build and update MXNet.io's static html
 * [update_all_version.sh](update_all_version.sh) - takes the output of `build_all_version.sh` then uses `AddVersion.py` and `AddPackageLink.py` to update the static html
 
+## Dependencies
+
+The `make docs` process triggers several different documentation functions. Most of this logic is found in [mxdoc.py](https://github.com/apache/incubator-mxnet/blob/master/docs/mxdoc.py). If you review this file you will see several functions which will generate different API documentation sets. For example, they are called in this way:
+
+```python
+app.connect("builder-inited", generate_doxygen)
+app.connect("builder-inited", build_scala_docs)
+app.connect("builder-inited", build_clojure_docs)
+# app.connect("builder-inited", build_r_docs)
+```
+
+These four are mentioned here in dependencies since you will need to install several extra packages beyond just Sphinx and its collection of plugins and packages. You can, if you wish, simply comment out any of these lines to skip generating the related docs, thereby eliminating several dependencies, and reducing complexity in your builds.
+
+Note: If you review the file, you will see that the function to generate the R docs is commented out by default. Remove this comment mark to generate R docs manually.
+
+Covering each one, here are the related dependencies.
+
+* core MXNet dependencies - you need to build MXNet from source before generating the docs. Once you have accomplished this, you have further dependencies to consider for docs.
+
+* `generate_doxygen` - this generates the Python and C++ APIs.
+
+To enable this manually, run the following (on Ubuntu). Similar packages can be found for other operating systems, but pay attention to versions. Newer versions have been found to have incompatibilities between packages.
+
+```bash
+apt-get install \
+    doxygen \
+    pandoc
+
+pip install --upgrade pip && pip install \
+    beautifulsoup4 \
+    breathe \
+    CommonMark==0.5.4 \
+    h5py \
+    mock==1.0.1 \
+    pypandoc \
+    recommonmark==0.4.0 \
+    sphinx==1.5.6
+```
+
+* `build_scala_docs` - this generates the MXNet-Scala package & Scala API docs.
+
+To enable this manually, run the following:
+
+```bash
+apt-get install software-properties-common
+apt-get update
+apt-get install openjdk-8-jdk
+apt-get install openjdk-8-jre
+
+echo "deb https://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list
+apt-key add sbt.gpg
+apt-get update && apt-get install \
+    maven \
+    sbt \
+    scala
+```
+
+* `build_clojure_docs` - this generates the Clojure API docs.
+
+To enable this manually, run the following:
+
+```bash
+wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein
+chmod 775 lein
+sudo cp lein /usr/local/bin
+```
+
+* `build_r_docs` - this generates the R API docs in a PDF file found at `/docs/api/r/mxnet-r-reference-manual.pdf`.
+
+To enable this manually, run the following, and be sure to re-enable the `build_r_docs` function in `mxdoc.py` before you run `make docs`:
+
+```bash
+apt-get install \
+    libcairo2-dev \
+    libcurl4-openssl-dev \
+    libssl-dev \
+    libxml2-dev \
+    libxt-dev \
+    r-base \
+    r-base-dev
+
+Rscript -e "install.packages('devtools', repo = 'https://cran.rstudio.com')"
+cd R-package
+Rscript -e "library(devtools); library(methods); options(repos=c(CRAN='https://cran.rstudio.com')); install_deps(dependencies = TRUE)"
+```
+
 
 ## Setting Up a Docs Dev Server
 
-Running docs builds locally on a Mac is not recommended. For these instructions, you will use an Ubuntu machine.
+Running docs builds locally on a Mac is not recommended. See [#10858](https://github.com/apache/incubator-mxnet/issues/10858) for workarounds.
 
-This flow has been tested on a vanilla Ubuntu 16.04 cloud instance on AWS.
+For these instructions, you will use an Ubuntu machine. This flow has been tested on a vanilla Ubuntu 16.04 cloud instance on AWS.
 
 **Step 1:** Spin up your Ubuntu server and SSH in.
 
@@ -30,29 +116,26 @@ source mxnet_docs/bin/activate
 
 **Note:** Using a Python 2.7 environment is required to build older versions of the docs that have Python 3 incompatibilities. If you're only building the latest or version 1.0.0+, then you may use a Python 3 environment.
 
-**Step 3:** Clone the repo.
+**Step 3:** Clone the repo or your own fork of the repo.
 
 ```bash
 git clone --recursive https://github.com/apache/incubator-mxnet.git mxnet
-cd mxnet
+cd mxnet/docs/build_version_doc
 ```
 
-**Step 4:** Install dependencies.
+**Step 4:** Install dependencies and build MXNet.
 
-These scripts will install the dependencies for you.
+This script will install the dependencies for you and build MXNet from source.
 
 ```bash
-./ci/docker/install/ubuntu_core.sh
-./ci/docker/install/ubuntu_python.sh
-./ci/docker/install/ubuntu_scala.sh
-./ci/docker/install/ubuntu_docs.sh
+./setup_docs_ubuntu.sh
 ```
 
 **Step 5a:** Make the docs.
 
 Here you have two options (recommended for most situations):
 
-* Build this current branch with the following:
+* Change branches first, or build master with the following:
 
 ```bash
 cd mxnet

docs/build_version_doc/requirements.txt

@@ -0,0 +1,15 @@
+beautifulsoup4
+breathe
+cpplint==1.3.0
+CommonMark==0.5.4
+h5py==2.8.0rc1
+mock==1.0.1
+nose
+nose-timer
+numpy<1.15.0,>=1.8.2
+pylint==1.8.3
+pypandoc
+recommonmark==0.4.0 
+requests<2.19.0,>=2.18.4
+scipy==1.0.1
+sphinx==1.5.6

docs/build_version_doc/setup_docs_ubuntu.sh

@@ -17,45 +17,126 @@
 # specific language governing permissions and limitations
 # under the License.
 
+######################################################################
+# This script installs MXNet with required dependencies on a Ubuntu Machine.
+# Tested on Ubuntu 16.04+ distro.
+# Important Maintenance Instructions:
+#    Align changes with CI in /ci/docker/install/ubuntu_core.sh
+#    and ubuntu_python.sh
+######################################################################
 
-# If you need to build <= v0.12.0 then use a Python 2 environment
-# mxdoc.py - a sphinx extension, was not Python 3 compatible in the old versions
-# source activate mxnet_p27
+set -ex
+cd "$(dirname "$0")"
 
-# Install dependencies
 sudo apt-get update
+echo "Installing MXNet core dependencies..."
 sudo apt-get install -y \
     apt-transport-https \
+    build-essential \
     ca-certificates \
+    cmake \
     curl \
-    doxygen \
     git \
+    libatlas-base-dev \
+    libcurl4-openssl-dev \
     libjemalloc-dev \
-    pandoc \
-    software-properties-common
-
-pip install --user \
-    beautifulsoup4 \
-    breathe \
-    CommonMark==0.5.4 \
-    h5py \
-    mock==1.0.1 \
-    pypandoc \
-    recommonmark==0.4.0 \
-    sphinx==1.5.6
-
-# Setup scala
-echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
-sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
-sudo apt-get update
+    liblapack-dev \
+    libopenblas-dev \
+    libopencv-dev \
+    libzmq3-dev \
+    ninja-build \
+    python-dev \
+    python3-dev \
+    software-properties-common \
+    sudo \
+    unzip \
+    virtualenv \
+    wget
+
+
+echo "Installing MXNet docs dependencies..."
 sudo apt-get install -y \
-  sbt \
-  scala
+    doxygen \
+    pandoc
+
+
+echo "Installing Clojure dependencies..."
+wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein
+chmod 775 lein
+sudo cp lein /usr/local/bin
+
+
+echo "Installing R dependencies..."
+# install libraries for mxnet's r package on ubuntu
+sudo sh -c 'echo "deb http://cran.rstudio.com/bin/linux/ubuntu trusty/" >> /etc/apt/sources.list'
+
+key=E084DAB9
+
+sudo sh -c 'gpg --keyserver keyserver.ubuntu.com --recv-key $key || \
+    gpg --keyserver keyserver.pgp.com --recv-keys $key || \
+    gpg --keyserver ha.pool.sks-keyservers.net --recv-keys $key ;'
+
+# Installing the latest version (3.3+) that is compatible with MXNet
+sudo add-apt-repository 'deb [arch=amd64,i386] https://cran.rstudio.com/bin/linux/ubuntu xenial/'
+sudo apt-key add ../../ci/docker/install/sbt.gpg
+
+sudo apt-get update
+sudo apt-get install -y --allow-unauthenticated \
+    libcairo2-dev \
+    libssl-dev \
+    libxml2-dev \
+    libxt-dev \
+    r-base \
+    r-base-dev
+
+
+echo "Installing Scala dependencies..."
+sudo apt-get install -y software-properties-common
+sudo apt-get update
+sudo apt-get install -y openjdk-8-jdk
+sudo apt-get install -y openjdk-8-jre
+
+sudo sh -c 'echo "deb https://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list'
+# ubuntu keyserver is very flaky
+#apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
+#apt-key adv --keyserver keys.gnupg.net --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
+sudo apt-key add ../../ci/docker/install/sbt.gpg
+sudo apt-get update && sudo apt-get install -y \
+    maven \
+    sbt \
+    scala
+
+
+wget -nv https://bootstrap.pypa.io/get-pip.py
+echo "Installing for Python 3..."
+sudo python3 get-pip.py
+pip3 install --user -r requirements.txt
+echo "Installing for Python 2..."
+sudo python2 get-pip.py
+pip2 install --user -r requirements.txt
 
-# Optionally setup Apache2
+
+cd ../../
+
+echo "Checking for GPUs..."
+gpu_install=$(which nvidia-smi | wc -l)
+if [ "$gpu_install" = "0" ]; then
+    make_params="USE_OPENCV=1 USE_BLAS=openblas"
+    echo "nvidia-smi not found. Installing in CPU-only mode with these build flags: $make_params"
+else
+    make_params="USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1"
+    echo "nvidia-smi found! Installing with CUDA and cuDNN support with these build flags: $make_params"
+fi
+
+echo "Building MXNet core. This can take few minutes..."
+make -j $(nproc) $make_params
+
+
+# Setup Apache2
+echo "Setting up Apache web server..."
 sudo apt-get install -y apache2
 sudo ufw allow 'Apache Full'
-# turn on mod_rewrite
+# Turn on mod_rewrite
 sudo a2enmod rewrite
 
 echo 'To enable redirects you need to edit /etc/apache2/apache2.conf '
@@ -64,5 +145,6 @@ echo '       AllowOverride all '
 echo '--> Then restart apache with: '
 echo '       sudo systemctl restart apache2'
 
+
 # Cleanup
 sudo apt autoremove -y