Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source MongoDB fetch authorized collections only #9238

Merged
merged 30 commits into from
Jan 13, 2022
Merged

Source MongoDB fetch authorized collections only #9238

Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
93d53ac
fix for jdk 17
Dec 15, 2021
ede8d38
Merge branch 'master' of github.com:airbytehq/airbyte
Dec 15, 2021
27be3fe
Merge branch 'master' of github.com:airbytehq/airbyte
Dec 15, 2021
ca75033
Merge branch 'master' of github.com:airbytehq/airbyte
Dec 16, 2021
72ab46f
Merge branch 'master' of github.com:airbytehq/airbyte
Dec 16, 2021
aec3384
Merge branch 'master' of github.com:airbytehq/airbyte
Dec 16, 2021
7efd5aa
Merge branch 'master' of github.com:airbytehq/airbyte
Dec 20, 2021
6a773f9
Merge branch 'master' of github.com:airbytehq/airbyte
Dec 21, 2021
fa18537
Merge branch 'master' of github.com:airbytehq/airbyte
Dec 21, 2021
b0ba37b
Merge branch 'master' of github.com:airbytehq/airbyte
Dec 22, 2021
fa31a0d
Merge branch 'master' of github.com:airbytehq/airbyte
Dec 23, 2021
67e0bd6
Merge branch 'master' of github.com:airbytehq/airbyte
Dec 24, 2021
ec0d1bd
Merge branch 'master' of github.com:airbytehq/airbyte
Dec 28, 2021
23598ec
Merge branch 'master' of github.com:airbytehq/airbyte
Dec 28, 2021
59d63e6
Merge branch 'master' of github.com:airbytehq/airbyte
Dec 30, 2021
4d10720
Source MongoDB show authorized collections
Dec 31, 2021
e2cb62c
Merge branch 'master' of github.com:airbytehq/airbyte
Jan 10, 2022
f622758
add javadoc
Jan 10, 2022
aae25d6
fixed checkstyle
Jan 10, 2022
632583a
Merge branch 'master' of github.com:airbytehq/airbyte
Jan 10, 2022
2bc76ce
Merge branch 'master' of github.com:airbytehq/airbyte
Jan 11, 2022
0556234
add CHANGELOG
Jan 11, 2022
131c45f
Merge branch 'master' into vmaltsev/8752-source-mongodb-show-authoriz…
Jan 11, 2022
a1c94e1
fix checkstyle
Jan 11, 2022
a51ec08
refactoring
Jan 12, 2022
45dfc2d
bump version anf fix checkstyle
Jan 12, 2022
9baf2e1
Merge branch 'master' of github.com:airbytehq/airbyte
Jan 12, 2022
9942d14
Merge branch 'master' into vmaltsev/8752-source-mongodb-show-authoriz…
Jan 12, 2022
ed4f1ba
Merge branch 'master' of github.com:airbytehq/airbyte
Jan 13, 2022
6d073b7
Merge branch 'master' into vmaltsev/8752-source-mongodb-show-authoriz…
Jan 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-mongodb-v2/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ dependencies {
implementation files(project(':airbyte-integrations:bases:base-java').airbyteDocker.outputs)
implementation project(':airbyte-integrations:connectors:source-relational-db')

implementation 'org.mongodb:mongodb-driver-sync:4.3.0'
implementation 'org.mongodb:mongodb-driver-sync:4.4.0'

testImplementation 'org.testcontainers:mongodb:1.15.3'

Expand Down
25 changes: 22 additions & 3 deletions ...source-mongodb-v2/src/main/java/io.airbyte.integrations.source.mongodb/MongoDbSource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ public List<CheckedConsumer<MongoDatabase, Exception>> getCheckOperations(final
throws Exception {
final List<CheckedConsumer<MongoDatabase, Exception>> checkList = new ArrayList<>();
checkList.add(database -> {
if (database.getCollectionNames().isEmpty()) {
if (getAuthorizedCollections(database).isEmpty()) {
throw new Exception("Unable to execute any operation on the source!");
} else {
LOGGER.info("The source passed the basic operation test!");
Expand All @@ -114,7 +114,7 @@ protected List<TableInfo<CommonField<BsonType>>> discoverInternal(final MongoDat
throws Exception {
final List<TableInfo<CommonField<BsonType>>> tableInfos = new ArrayList<>();

for (final String collectionName : database.getCollectionNames()) {
for (final String collectionName : getAuthorizedCollections(database)) {
final MongoCollection<Document> collection = database.getCollection(collectionName);
final Map<String, BsonType> uniqueFields = MongoUtils.getUniqueFields(collection);

Expand All @@ -135,7 +135,26 @@ protected List<TableInfo<CommonField<BsonType>>> discoverInternal(final MongoDat
return tableInfos;
}

@Override
private Set<String> getAuthorizedCollections(MongoDatabase database) {
/* db.runCommand ({listCollections: 1.0, authorizedCollections: true, nameOnly: true })
the command returns only those collections for which the user has privileges.
For example, if a user has find action on specific collections, the command returns only those collections;
or, if a user has find or any other action, on the database resource, the command lists all collections in the database.
*/
Document document = database.getDatabase().runCommand(new Document("listCollections", 1)
.append("authorizedCollections", true).append("nameOnly", true));

return document.toBsonDocument()
.get("cursor").asDocument()
.getArray("firstBatch")
.stream()
.filter(bsonValue -> bsonValue.asDocument().getString("type").getValue().equals("collection"))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick .The listCollections command can take in a filter parameter to filter on any field in the returned document directly.

Reference: https://docs.mongodb.com/manual/reference/command/listCollections/#definition

I think if you add the filter parameter, you won't need to do the filter here:

Document document = database.getDatabase().runCommand(new Document("listCollections", 1)
  .append("authorizedCollections", true)
  .append("nameOnly", true))
  .append("filter", "{ 'type': 'collection' }")

I am not 100% sure this is the correct syntax though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick .The listCollections command can take in a filter parameter to filter on any field in the returned document directly.

Reference: https://docs.mongodb.com/manual/reference/command/listCollections/#definition

I think if you add the filter parameter, you won't need to do the filter here:

Document document = database.getDatabase().runCommand(new Document("listCollections", 1)
  .append("authorizedCollections", true)
  .append("nameOnly", true))
  .append("filter", "{ 'type': 'collection' }")

I am not 100% sure this is the correct syntax though.

added filter into mongo shell command

.map(bsonValue -> bsonValue.asDocument().getString("name").getValue())
.collect(Collectors.toSet());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great)


}

@Override
protected List<TableInfo<CommonField<BsonType>>> discoverInternal(final MongoDatabase database, final String schema) throws Exception {
// MondoDb doesn't support schemas
return discoverInternal(database);
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/mongodb-v2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ For more information regarding configuration parameters, please see [MongoDb Doc

| Version | Date | Pull Request | Subject |
| :--- | :--- | :--- | :--- |
| 0.1.11 | 2022-01-10 | [9238](https://github.com/airbytehq/airbyte/pull/9238) | Return only those collections for which the user has privileges |
| 0.1.10 | 2021-12-30 | [9202](https://github.com/airbytehq/airbyte/pull/9202) | Update connector fields title/description |
| 0.1.9 | 2021-12-07 | [8491](https://github.com/airbytehq/airbyte/pull/8491) | Configure 10000 limit doc reading during Discovery step |
| 0.1.8 | 2021-11-29 | [8306](https://github.com/airbytehq/airbyte/pull/8306) | Added milliseconds for date format for cursor |
Expand Down