[AIRFLOW-2832] Lint and resolve inconsistencies in Markdown files (#3670)

Clean up the Markdown files and make the formatting consistent

Author: tedmiston

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,33 +1,34 @@
 Make sure you have checked _all_ steps below.
 
-### JIRA
-- [ ] My PR addresses the following [Airflow JIRA](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
-    - https://issues.apache.org/jira/browse/AIRFLOW-XXX
-    - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a JIRA issue.
+### Jira
 
+- [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
+  - https://issues.apache.org/jira/browse/AIRFLOW-XXX
+  - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 
 ### Description
-- [ ] Here are some details about my PR, including screenshots of any UI changes:
 
+- [ ] Here are some details about my PR, including screenshots of any UI changes:
 
 ### Tests
-- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
 
+- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
 
 ### Commits
-- [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
-    1. Subject is separated from body by a blank line
-    2. Subject is limited to 50 characters
-    3. Subject does not end with a period
-    4. Subject uses the imperative mood ("add", not "adding")
-    5. Body wraps at 72 characters
-    6. Body explains "what" and "why", not "how"
 
+- [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
+  1. Subject is separated from body by a blank line
+  1. Subject is limited to 50 characters (not including Jira issue reference)
+  1. Subject does not end with a period
+  1. Subject uses the imperative mood ("add", not "adding")
+  1. Body wraps at 72 characters
+  1. Body explains "what" and "why", not "how"
 
 ### Documentation
-- [ ] In case of new functionality, my PR adds documentation that describes how to use it.
-    - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
 
+- [ ] In case of new functionality, my PR adds documentation that describes how to use it.
+  - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
 
 ### Code Quality
+
 - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

CONTRIBUTING.md

@@ -3,22 +3,21 @@
 Contributions are welcome and are greatly appreciated! Every
 little bit helps, and credit will always be given.
 
-
-# Table of Contents
-  * [TOC](#table-of-contents)
-  * [Types of Contributions](#types-of-contributions)
-      - [Report Bugs](#report-bugs)
-      - [Fix Bugs](#fix-bugs)
-      - [Implement Features](#implement-features)
-      - [Improve Documentation](#improve-documentation)
-      - [Submit Feedback](#submit-feedback)
-  * [Documentation](#documentation)
-  * [Development and Testing](#development-and-testing)
-      - [Setting up a development environment](#setting-up-a-development-environment)
-      - [Pull requests guidelines](#pull-request-guidelines)
-      - [Testing Locally](#testing-locally)
-  * [Changing the Metadata Database](#changing-the-metadata-database)
-
+## Table of Contents
+
+- [TOC](#table-of-contents)
+- [Types of Contributions](#types-of-contributions)
+  - [Report Bugs](#report-bugs)
+  - [Fix Bugs](#fix-bugs)
+  - [Implement Features](#implement-features)
+  - [Improve Documentation](#improve-documentation)
+  - [Submit Feedback](#submit-feedback)
+- [Documentation](#documentation)
+- [Development and Testing](#development-and-testing)
+  - [Setting up a development environment](#setting-up-a-development-environment)
+  - [Pull requests guidelines](#pull-request-guidelines)
+  - [Testing Locally](#testing-locally)
+- [Changing the Metadata Database](#changing-the-metadata-database)
 
 ## Types of Contributions
 
@@ -55,11 +54,9 @@ The best way to send feedback is to open an issue on [Apache Jira](https://issue
 
 If you are proposing a feature:
 
--   Explain in detail how it would work.
--   Keep the scope as narrow as possible, to make it easier to
-    implement.
--   Remember that this is a volunteer-driven project, and that
-    contributions are welcome :)
+- Explain in detail how it would work.
+- Keep the scope as narrow as possible, to make it easier to implement.
+- Remember that this is a volunteer-driven project, and that contributions are welcome :)
 
 ## Documentation
 
@@ -68,11 +65,15 @@ The latest API documentation is usually available
 you need to have set up an Airflow development environment (see below). Also
 install the `doc` extra.
 
-    pip install -e .[doc]
+```
+pip install -e .[doc]
+```
 
 Generate the documentation by running:
 
-    cd docs && ./build.sh
+```
+cd docs && ./build.sh
+```
 
 Only a subset of the API reference documentation builds. Install additional
 extras to build the full API reference.
@@ -122,11 +123,13 @@ Please install python(2.7.x or 3.4.x), mysql, and libxml by using system-level p
 managers like yum, apt-get for Linux, or homebrew for Mac OS at first.
 It is usually best to work in a virtualenv and tox. Install development requirements:
 
-    cd $AIRFLOW_HOME
-    virtualenv env
-    source env/bin/activate
-    pip install -e .[devel]
-    tox
+```
+cd $AIRFLOW_HOME
+virtualenv env
+source env/bin/activate
+pip install -e .[devel]
+tox
+```
 
 Feel free to customize based on the extras available in [setup.py](./setup.py)
 
@@ -135,52 +138,31 @@ Feel free to customize based on the extras available in [setup.py](./setup.py)
 Before you submit a pull request from your forked repo, check that it
 meets these guidelines:
 
-1. The pull request should include tests, either as doctests, unit tests, or
-both. The airflow repo uses [Travis CI](https://travis-ci.org/apache/incubator-airflow)
-to run the tests and [codecov](https://codecov.io/gh/apache/incubator-airflow)
-to track coverage. You can set up both for free on your fork. It will
-help you making sure you do not break the build with your PR and that you help
-increase coverage.
-2. Please [rebase your fork](http://stackoverflow.com/a/7244456/1110993),
-squash commits, and resolve all conflicts.
-3. Every pull request should have an associated
-[JIRA](https://issues.apache.org/jira/browse/AIRFLOW/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
-The JIRA link should also be contained in the PR description.
-4. Preface your commit's subject & PR's title with **[AIRFLOW-XXX]**
-where *XXX* is the JIRA number. We compose release notes (i.e. for Airflow releases) from all commit titles in a release.
-By placing the JIRA number in the commit title and hence in the release notes,
-Airflow users can look into JIRA and Github PRs for more details about a particular change.
-5. Add an [Apache License](http://www.apache.org/legal/src-headers.html)
- header to all new files
-6. If the pull request adds functionality, the docs should be updated as part
-of the same PR. Doc string are often sufficient.  Make sure to follow the
-Sphinx compatible standards.
-7. The pull request should work for Python 2.7 and 3.4. If you need help
-writing code that works in both Python 2 and 3, see the documentation at the
-[Python-Future project](http://python-future.org) (the future package is an
-Airflow requirement and should be used where possible).
-8. As Airflow grows as a project, we try to enforce a more consistent
-style and try to follow the Python community guidelines. We track this
-using [landscape.io](https://landscape.io/github/apache/incubator-airflow/),
-which you can setup on your fork as well to check before you submit your
-PR. We currently enforce most [PEP8](https://www.python.org/dev/peps/pep-0008/)
-and a few other linting rules. It is usually a good idea to lint locally
-as well using [flake8](https://flake8.readthedocs.org/en/latest/)
-using `flake8 airflow tests`. `git diff upstream/master -u -- "*.py" | flake8 --diff` will return any changed files in your branch that require linting.
-9. Please read this excellent [article](http://chris.beams.io/posts/git-commit/) on
-commit messages and adhere to them. It makes the lives of those who
-come after you a lot easier.
+1. The pull request should include tests, either as doctests, unit tests, or both. The airflow repo uses [Travis CI](https://travis-ci.org/apache/incubator-airflow) to run the tests and [codecov](https://codecov.io/gh/apache/incubator-airflow) to track coverage. You can set up both for free on your fork. It will help you making sure you do not break the build with your PR and that you help increase coverage.
+1. Please [rebase your fork](http://stackoverflow.com/a/7244456/1110993), squash commits, and resolve all conflicts.
+1. Every pull request should have an associated [JIRA](https://issues.apache.org/jira/browse/AIRFLOW/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). The JIRA link should also be contained in the PR description.
+1. Preface your commit's subject & PR's title with **[AIRFLOW-XXX]** where *XXX* is the JIRA number. We compose release notes (i.e. for Airflow releases) from all commit titles in a release. By placing the JIRA number in the commit title and hence in the release notes, Airflow users can look into JIRA and Github PRs for more details about a particular change.
+1. Add an [Apache License](http://www.apache.org/legal/src-headers.html) header to all new files
+1. If the pull request adds functionality, the docs should be updated as part of the same PR. Doc string are often sufficient.  Make sure to follow the Sphinx compatible standards.
+1. The pull request should work for Python 2.7 and 3.4. If you need help writing code that works in both Python 2 and 3, see the documentation at the [Python-Future project](http://python-future.org) (the future package is an Airflow requirement and should be used where possible).
+1. As Airflow grows as a project, we try to enforce a more consistent style and try to follow the Python community guidelines. We track this using [landscape.io](https://landscape.io/github/apache/incubator-airflow/), which you can setup on your fork as well to check before you submit your PR. We currently enforce most [PEP8](https://www.python.org/dev/peps/pep-0008/) and a few other linting rules. It is usually a good idea to lint locally as well using [flake8](https://flake8.readthedocs.org/en/latest/) using `flake8 airflow tests`. `git diff upstream/master -u -- "*.py" | flake8 --diff` will return any changed files in your branch that require linting.
+1. Please read this excellent [article](http://chris.beams.io/posts/git-commit/) on commit messages and adhere to them. It makes the lives of those who come after you a lot easier.
 
 ### Testing locally
 
 #### TL;DR
+
 Tests can then be run with (see also the [Running unit tests](#running-unit-tests) section below):
 
-    ./run_unit_tests.sh
+```
+./run_unit_tests.sh
+```
 
 Individual test files can be run with:
 
-    nosetests [path to file]
+```
+nosetests [path to file]
+```
 
 #### Running unit tests
 
@@ -251,13 +233,16 @@ While these may be phased out over time, these packages are currently not
 managed with npm.
 
 ### Node/npm versions
+
 Make sure you are using recent versions of node and npm. No problems have been found with node>=8.11.3 and npm>=6.1.3
 
 ### Using npm to generate bundled files
 
 #### npm
+
 First, npm must be available in your environment. If it is not you can run the following commands
 (taken from [this source](https://gist.github.com/DanHerbert/9520689))
+
 ```
 brew install node --without-npm
 echo prefix=~/.npm-packages >> ~/.npmrc
@@ -266,11 +251,13 @@ curl -L https://www.npmjs.com/install.sh | sh
 
 The final step is to add `~/.npm-packages/bin` to your `PATH` so commands you install globally are usable.
 Add something like this to your `.bashrc` file, then `source ~/.bashrc` to reflect the change.
+
 ```
 export PATH="$HOME/.npm-packages/bin:$PATH"
 ```
 
 #### npm packages
+
 To install third party libraries defined in `package.json`, run the
 following within the `airflow/www_rbac/` directory which will install them in a
 new `node_modules/` folder within `www_rbac/`.
@@ -296,13 +283,13 @@ npm run dev
 
 #### Upgrading npm packages
 
-Should you add or upgrade a npm package, which involves changing `package.json`, you'll need to re-run `npm install` 
+Should you add or upgrade a npm package, which involves changing `package.json`, you'll need to re-run `npm install`
 and push the newly generated `package-lock.json` file so we get the reproducible build.
 
 #### Javascript Style Guide
 
-We try to enforce a more consistent style and try to follow the JS community guidelines. 
-Once you add or modify any javascript code in the project, please make sure it follows the guidelines 
+We try to enforce a more consistent style and try to follow the JS community guidelines.
+Once you add or modify any javascript code in the project, please make sure it follows the guidelines
 defined in [Airbnb JavaScript Style Guide](https://github.com/airbnb/javascript).
 Apache Airflow uses [ESLint](https://eslint.org/) as a tool for identifying and reporting on patterns in JavaScript,
 which can be used by running any of the following commands.
@@ -311,7 +298,6 @@ which can be used by running any of the following commands.
 # Check JS code in .js and .html files, and report any errors/warnings
 npm run lint
 
-# Check JS code in .js and .html files, report any errors/warnings and fix them if possible 
+# Check JS code in .js and .html files, report any errors/warnings and fix them if possible
 npm run lint:fix
 ```
- 

README.md

@@ -23,6 +23,7 @@ makes it easy to visualize pipelines running in production,
 monitor progress, and troubleshoot issues when needed.
 
 ## Getting started
+
 Please visit the Airflow Platform documentation (latest **stable** release) for help with [installing Airflow](https://airflow.incubator.apache.org/installation.html), getting a [quick start](https://airflow.incubator.apache.org/start.html), or a more complete [tutorial](https://airflow.incubator.apache.org/tutorial.html).
 
 Documentation of GitHub master (latest development branch): [ReadTheDocs Documentation](https://airflow.readthedocs.io/en/latest/)
@@ -54,22 +55,28 @@ unit of work and continuity.
 ## User Interface
 
 - **DAGs**: Overview of all DAGs in your environment.
-![](/docs/img/dags.png)
+
+  ![](/docs/img/dags.png)
 
 - **Tree View**: Tree representation of a DAG that spans across time.
-![](/docs/img/tree.png)
+
+  ![](/docs/img/tree.png)
 
 - **Graph View**: Visualization of a DAG's dependencies and their current status for a specific run.
-![](/docs/img/graph.png)
+
+  ![](/docs/img/graph.png)
 
 - **Task Duration**: Total time spent on different tasks over time.
-![](/docs/img/duration.png)
+
+  ![](/docs/img/duration.png)
 
 - **Gantt View**: Duration and overlap of a DAG.
-![](/docs/img/gantt.png)
+
+  ![](/docs/img/gantt.png)
 
 - **Code View**:  Quick way to view source code of a DAG.
-![](/docs/img/code.png)
+
+  ![](/docs/img/code.png)
 
 ## Who uses Airflow?
 
@@ -79,7 +86,7 @@ if you may.
 
 Committers:
 
-* Refer to [Committers](https://cwiki.apache.org/confluence/display/AIRFLOW/Committers)
+- Refer to [Committers](https://cwiki.apache.org/confluence/display/AIRFLOW/Committers)
 
 Currently **officially** using Airflow:
 
@@ -268,8 +275,7 @@ Currently **officially** using Airflow:
 
 ## Links
 
-
-* [Documentation](https://airflow.incubator.apache.org/)
-* [Chat](https://gitter.im/apache/incubator-airflow)
-* [Apache Airflow Incubation Status](http://incubator.apache.org/projects/airflow.html)
-* [More](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links)
+- [Documentation](https://airflow.incubator.apache.org/)
+- [Chat](https://gitter.im/apache/incubator-airflow)
+- [Apache Airflow Incubation Status](http://incubator.apache.org/projects/airflow.html)
+- [More](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links)

TODO.md

@@ -1,30 +1,35 @@
 #### Roadmap items
+
 * UI page answering "Why isn't this task instance running?"
 * Attempt removing DagBag caching for the web server
 * Distributed scheduler (supervisors)
-    * Get the supervisors to run sensors (as opposed to each sensor taking a slot)
-    * Improve DagBag differential refresh
-    * Pickle all the THINGS! supervisors maintains fresh, versioned pickles in the database as they monitor for change
+  * Get the supervisors to run sensors (as opposed to each sensor taking a slot)
+  * Improve DagBag differential refresh
+  * Pickle all the THINGS! supervisors maintains fresh, versioned pickles in the database as they monitor for change
 * Pre-prod running off of master
 * Containment / YarnExecutor / Docker?
 * Get s3 logs
 * Test and migrate to use beeline instead of the Hive CLI
 * Run Hive / Hadoop / HDFS tests in Travis-CI
 
 #### UI
+
 * Backfill form
 * Better task filtering int duration and landing time charts (operator toggle, task regex, uncheck all button)
 * Add templating to adhoc queries
 
 #### Backend
+
 * Add a run_only_latest flag to BaseOperator, runs only most recent task instance where deps are met
 * Raise errors when setting dependencies on task in foreign DAGs
 * Add an is_test flag to the run context
 
 #### Wishlist
+
 * Pause flag at the task level
 * Increase unit test coverage
 * Stats logging interface with support for stats and sqlalchemy to collect detailed information from the scheduler and dag processing times
 
 #### Other
-* deprecate TimeSensor
+
+* Deprecate TimeSensor