[AIRFLOW-3049] Add extra operations for Mongo hook

[dlebech]

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-3049

Description

• Here are some details about my PR, including screenshots of any UI changes:

This PR adds six extra operations to the Mongo hook: update, replace and delete as well as a *_many version for each of those.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

• test_update_one
• test_update_one_with_upsert
• test_update_many
• test_update_many_with_upsert
• test_replace_one
• test_replace_one_with_filter
• test_replace_one_with_upsert
• test_replace_many
• test_replace_many_with_upsert
• test_delete_one
• test_delete_many

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

• Passes git diff upstream/master -u -- "*.py" | flake8 --diff

[dlebech]

The Travis build seems to have failed for one of the environments. It runs on my own branch though: https://travis-ci.org/dlebech/incubator-airflow/builds/427786725

[Fokko]

Maybe call this one just kwargs as well, to make it congruent with the other calls.

[Fokko]

The upsert and collation are missing from the docstring.

[dlebech]

@Fokko thanks for the feedback. I've updated the parameter name and added tweaks to the documentation.

[Fokko]

LGTM, one minor think. Recently we've moved to a Docker-compose based CI pipeline. Instead of mocking the requests, there is also a real mongo instance running. What are your opinions on mocking, or talking to a real mongo instance?

[dlebech]

@Fokko Thanks!

Regarding mocking vs real instance, I'm not sure I'm in the best position to answer that since I basically just built this PR upon the pre-existing setup, so I cannot say why mongomock was originally chosen.

But it looks like a pretty good library for this kind of thing, and since Travis is already taking a very long time to run, I assume mocking speeds thing up a bit in this case. I don't know how much that matters, but at least from my perspective, it feels like mocking is an ok thing to do 🙂

Perhaps the original author @andscoop has a better opinion than mine?

[andscoop]

@dlebech @Fokko When I was originally writing unit tests, I don't believe that we were using docker-compose, or if we were I was not aware of it. I'm not familiar with all the reasons for moving to docker-compose, but idk if I see a compelling reason to move away from mocking the responses in the hook as all responses are known before hand.

[ron819]

@Fokko is this ready to merge?

[Fokko]

Thanks, @dlebech. Merging to master!