Update CI documendation

Update documentation

Update ssh-key documentation

update azure documentation

Clean pr

Update documenation

Rephrase documentation

update documentation

Co-authored-by: Jaladh Singhal <jaladh-singhal@users.noreply.github.com>

Update documentation

azure-pipelines/doc-build.yml

@@ -22,15 +22,15 @@ jobs:
   steps:
 # When setting up the azure pipeline for the first time, generate a ssh key locally
 # and add the generated public key to github as a deploy key through forked_repo_home/settings/deploy_key.
-# Use steps listed from: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/install-ssh-key?view=azure-devops#example-setup-using-github.
+# Use steps listed from: https://developer.github.com/v3/guides/managing-deploy-keys/#deploy-keys.
 
 # Secure files stored in the azure server are encryped and again decrypted by the azure task that uses the file.
 # Download a secure file to a temporary location in the virtual machine.
   - task: DownloadSecureFile@1
     inputs: 
       secureFile: 'id_azure_rsa'
 # Make sure you've added the generated private key file (named 'id_azure_rsa' here) to library & authorize it for all pipelines.
-# by using: https://docs.microsoft.com/en-us/azure/devops/pipelines/library/secure-files?view=azure-devops#how-do-i-authorize-a-secure-file-for-use-in-all-pipelines
+# by using: https://docs.microsoft.com/en-us/azure/devops/pipelines/library/secure-files?view=azure-devops#how-do-i-authorize-a-secure-file-for-use-in-all-pipelines.
 
 # Install an SSH key prior to a build or release.
 # This is needed to give azure access to deploy to github.
@@ -43,8 +43,9 @@ jobs:
      sshPublicKey: $(public_key)
      #sshPassphrase: $(Agent.TempDirectory)  # Optional - leave empty if it was left empty while generating the key.
      sshKeySecureFile: 'id_azure_rsa'
+# For more details, see: http://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/install-ssh-key?view=azure-devops#example-setup-using-github.
 # You can mask hostName & sshPublicKey values under secret variables (i.e. gh_host & public_key here) in your azure pipeline build page.
-# Follow these steps: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#secret-variables
+# Follow these steps: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#secret-variables.
 
 # ##vso[task.prependpath] is a built-in logging command to add paths.
 # Add conda to the list of paths available from the terminal.

deploy_docs.sh

@@ -15,15 +15,15 @@ SSH_REPO=${REPO/https:\/\/github.com\//git@github.com:}
 SHA=`git rev-parse --verify HEAD`
 
 # Clone the existing gh-pages for this repo into out/
-# Create a new empty branch if gh-pages doesn't exist yet (should only happen on first deply)
 git clone $REPO out
 cd out
 
+# Create a new empty branch if gh-pages doesn't exist yet (should only happen on first deploy)
 git checkout $TARGET_BRANCH || git checkout --orphan $TARGET_BRANCH
-cd ..
 
 # Clean out existing contents
-rm -rf out/sphinx_docs/ || exit 0
+git rm -rf . || exit 0
+cd ..
 
 # Pull from SOURCE_BRANCH again
 git pull $SSH_REPO $SOURCE_BRANCH
@@ -33,8 +33,8 @@ cd docs
 make html
 cd ../  
 
-mkdir -p out/sphinx_docs/
-mv -f docs/_build/html/* out/sphinx_docs/
+# Move it to the gh-pages branch
+mv -f docs/_build/html/* out/
 touch out/.nojekyll
 
 # Now let's go have some fun with the cloned repo

docs/workflow/azure_links.inc

@@ -0,0 +1,15 @@
+
+.. azure stuff
+.. _Azure Devops: http://azure.microsoft.com/en-us/services/devops/?nav=mi
+.. _Azure ssh-task: http://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/install-ssh-key?view=azure-devops#example-setup-using-github
+.. _Azure secure files: http://docs.microsoft.com/en-us/azure/devops/pipelines/library/secure-files?view=azure-devops#how-do-i-authorize-a-secure-file-for-use-in-all-pipelines
+.. _Azure variables: http://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#secret-variables
+.. _Azure secure files: http://docs.microsoft.com/en-us/azure/devops/pipelines/library/secure-files?view=azure-devops#how-do-i-authorize-a-secure-file-for-use-in-all-pipelines
+.. _Azure task: http://github.com/microsoft/azure-pipelines-tasks
+
+.. general stuff
+.. _Continuous Integration: https://en.wikipedia.org/wiki/Continuous_integration
+
+.. |emdash| unicode:: U+02014
+
+.. vim: ft=rstS

docs/workflow/continuous_integration.rst

@@ -0,0 +1,229 @@
+**********************
+Continuous Integration
+**********************
+
+We use a so-called `Continuous Integration`_ workflow with TARDIS. 
+This means that each time a change is proposed (via pull-request) 
+or a proposed change is merged into the main master branch a service will download the version 
+and execute all the unit tests and integration tests offered with TARDIS. This helps 
+us detect bugs immediately. 
+
+The following pages explain how we setup automated debugging on TARDIS through a remote 
+cloud service, called Azure, hosted by Visual Studio Team Services. This is done by 
+testing, building, and securely deploying this documentation to gh-pages. As 
+a developer, one should be familiar with how to smoothly run tests and record the 
+documentation. 
+
+
+Testing and Documentation
+=========================
+
+Testing
+-------
+
+To test TARDIS, we activate the designated environment, fetch the most recent reference data, install
+TARDIS, and use the python app tester, pytest::
+
+    $ sh ci-helpers/fetch_reference_data.sh
+    $ source activate tardis
+    $ python setup.py build_ext --inplace
+    $ pytest tardis --tardis-refdata=$(ref.data.home) --test-run-title="TARDIS test" --cov=tardis --cov-report=xml --cov-report=html
+
+The build_ext command is for accomdating C/C++ extentions. The --inplace command is to ensure that the build is in the source directory.
+
+See https://docs.pytest.org/en/latest/ for more information on pytest.
+
+Documentation
+-------------
+
+The documentation is built using Sphinx from the master branch. We use Restructured Text (reST) to encode the 
+required files, which are then converted into html links, that are then pushed to the root of the 
+gh-pages branch. 
+
+See https://pythonhosted.org/an_example_pypi_project/sphinx.html and https://help.github.com/en/categories/github-pages-basics
+for more information.
+
+Setting up a secure pathway to github
+=====================================
+
+.. _install-ssh-key:
+
+Installing a ssh-key
+--------------------
+
+When setting up the Azure pipeline for the first time, generate a ssh key locally::
+
+    $ ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
+
+Follow along with the command prompt, and you can neglect adding a passphrase due to Azure's added security.
+
+For more details, see `git key install`_.
+
+Adding it as a deploy key
+-------------------------
+
+After you install the key, you must add it as a deploy key to grant access to push to your repository.
+
+To do this, go to GitHub_
+    - Go to your profile
+    - Go to your desired repository
+    - Open the Settings
+    - Click on Deploy keys and copy the contents of your public key (The default name is id_rsa.pub).
+
+For explicit details, see `git deploy key`_.
+
+Adding your key locally and copying the known host name
+-------------------------------------------------------
+
+If you wish to deploy the documentation locally to gh-pages, you must add the generated key on your computer::
+
+    $ eval "$(ssh-agent -s)"
+    $ ssh-add ~/.ssh/id_rsa (Or whatever you called your key)
+    $ ssh-agent -k
+
+Copy the saved known host, as we will need it for installing the key on Azure.
+It should look something like (should look something like [1]As3..=ssh-rsa ..) and will be the last line added to::
+
+     ~/.ssh/known_hosts 
+
+
+Setting up Azure services 
+=========================
+
+The first step is to visit `Azure Devops`_ and create an account.
+
+Adding the key to Azure's secure files
+--------------------------------------
+
+Upload the secure key to Azure's library to download it in the script.
+
+In the library tab, select the **Secure files** tab, upload the file, and authorize it for use in all pipelines.
+As seen here:
+  .. image:: images/secure_file.png
+
+See also: `Azure secure files`_   
+
+One must use the DownloadSecureFile@1 `Azure task`_ to download the file onto the virtual machine when the pipeline runs.
+After which, one must use the InstallSSHKey@0 `Azure task`_ to add the ssh key.
+
+Setting up the YAML file to deploy
+----------------------------------
+
+YAML is short for "YAML Ain't Markup Language", as it is intented to be a simple way to write script
+that is standard for all programing languages. It is the file that communicates directly with the
+pipeline. 
+
+In the script, specify which branches you want to trigger for continuous deployment 
+and/or applicable for pull requests. Otherwise, it triggers all of them::
+
+    trigger:
+    - branch_name
+
+    pr:
+    - branch_name
+
+It follows the following hierarchy:
+
+Pipeline hierarchy
+^^^^^^^^^^^^^^^^^^
+
+.. graphviz::
+
+  digraph {
+    a -> b -> c -> d -> e
+    c -> f
+    a [label="Stages",shape=circle,fillcolor="white",style="filled"];
+    b [label="Stage",shape=circle,fillcolor="white",style="filled"];
+    c [label="Jobs", shape="circle", fillcolor="white", style="filled"]
+    d [label="Job", shape="circle", fillcolor="white", style="filled"]
+    e [label="Steps", shape="circle", fillcolor="white", style="filled"]
+    f [label="Task", shape="circle", fillcolor="white", style="filled"]
+  }
+
+A job is referred to as the most basic building block that the pipeline runs, using a single agent, which 
+is composed of steps of script.
+A task is a predefined script with a definitive purpose, such as downloading the secure file or installing a ssh-key.
+
+To download and add the ssh-key, prepare the scripts as::
+
+      - task: DownloadSecureFile@1
+        inputs: 
+          secureFile: 'id_azure_rsa'
+
+Secure files stored in the Azure server are encryped and again decrypted by the Azure task that uses the file.
+
+Download a secure file to a temporary location in the virtual machine::
+
+      - task: InstallSSHKey@0
+        inputs:
+          hostName: $(gh_host)
+          sshPublicKey: $(public_key)
+          #sshPassphrase: # Optional - leave empty if it was left empty while generating the key.
+          sshKeySecureFile: 'id_azure_rsa'
+
+
+hostName is the line that was copied in `Adding your key locally and copying the known host name`_.
+
+sshPublicKey should be a string value of what is inside your .pub file (i.e: rsa-key Axddd... username@server).
+
+sshKeySecureFile is the downloaded secure file you generated, you can reference directly as shown.
+
+For more details, see `Azure ssh-task`_
+
+To define variables in the script, one can do so using key-value pairs. For example::
+
+    variables:
+      system.debug: 'true'
+
+To define secret variables, or variables outside the script, one must navigate to variables after
+selecting the three dots on the top right while editing that pipeline, as seen here:  
+
+  .. image:: images/variables.png
+
+After defining the variable, one can encrypt it using this lock symbol:
+
+  .. image:: images/lock.png
+
+Variables are referenced as $(variable_name), as seen in the InstallSSHKey@0 task in the hostName and sshPublicKey inputs.
+
+Azure provides a list of agent hosts that can run the pipeline on a virtual machine. In our pipelines, we
+use the vm_Images: Ubuntu 16.04 and macOs-10.13.
+
+In a job, you can list a single vm as::
+
+      pool:
+        vmImage: "Ubuntu 16.04"
+
+If you are using a self hosted agent (see `Installing and running a self hosted agent` for more details)::
+
+      pool:
+        name: "agent_pool_name"
+
+Or if you prefer to use multiple virtual machines and specity the maximum that can run at the same time::
+
+      strategy:
+        matrix: 
+          linux:
+            vmImage: "Ubuntu 16.04"
+            conda: '/usr/share/miniconda'
+          mac:
+            vm_Image: 'macOS-10.13'
+            miniconda.url: 'http://repo.continuum.io/miniconda/Miniconda2-latest-mac-x86_64.sh'
+        maxParallel: 4
+
+
+
+Additional references
+--------------------
+
+https://thomas-cokelaer.info/tutorials/sphinx/rest_syntax.html#inline-markup-and-special-characters-e-g-bold-italic-verbatim
+
+https://docs.microsoft.com/en-us/azure/devops/pipelines/?view=azure-devops
+
+https://yaml.org/
+
+https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema
+
+
+.. include:: git_links.inc
+.. include:: azure_links.inc

docs/workflow/development_workflow.rst

@@ -20,4 +20,8 @@ to the Astropy team for designing it.
 
     update_refdata
 
+.. toctree::
+
+    continuous_integration
+
 

docs/workflow/git_links.inc

@@ -48,11 +48,12 @@
 .. _linux git workflow: http://www.mail-archive.com/dri-devel@lists.sourceforge.net/msg39091.html
 .. _git parable: http://tom.preston-werner.com/2009/05/19/the-git-parable.html
 .. _git foundation: http://matthew-brett.github.com/pydagogue/foundation.html
+.. _git key install: http://help.github.com/en/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent
+.. _git deploy key: https://developer.github.com/v3/guides/managing-deploy-keys/#deploy-keys
 .. _deleting master on github: http://matthew-brett.github.com/pydagogue/gh_delete_master.html
 .. _rebase without tears: http://matthew-brett.github.com/pydagogue/rebase_without_tears.html
 .. _resolving a merge: http://schacon.github.com/git/user-manual.html#resolving-a-merge
 .. _ipython git workflow: http://mail.scipy.org/pipermail/ipython-dev/2010-October/006746.html
-
 .. other stuff
 .. _python: http://www.python.org
 

docs/workflow/links.inc

@@ -2,3 +2,4 @@
 .. include:: known_projects.inc
 .. include:: this_project.inc
 .. include:: git_links.inc
+.. include:: azure_links.inc