Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only migrate active and disable connection #9454

Merged
merged 3 commits into from
Jan 13, 2022
Merged

Only migrate active and disable connection #9454

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
90f011c
Only migrate active and disable connection
benmoriceau Jan 12, 2022
1fb68de
format
benmoriceau Jan 12, 2022
b0614f3
Merge branch 'master' of github.com:airbytehq/airbyte into bmoric/mig…
benmoriceau Jan 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-scheduler/app/src/main/java/io/airbyte/scheduler/app/SchedulerApp.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ public void start() throws IOException {
// anymore.
cleanupZombies(jobPersistence, jobNotifier);

LOGGER.error("Start running the old scheduler");
LOGGER.info("Start running the old scheduler");
scheduleJobsPool.scheduleWithFixedDelay(
() -> {
MDC.setContextMap(mdc);
Expand Down
5 changes: 4 additions & 1 deletion airbyte-server/src/main/java/io/airbyte/server/ServerApp.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
import io.airbyte.commons.version.AirbyteVersion;
import io.airbyte.config.Configs;
import io.airbyte.config.EnvConfigs;
import io.airbyte.config.StandardSync.Status;
import io.airbyte.config.helpers.LogClientSingleton;
import io.airbyte.config.init.YamlSeedConfigPersistence;
import io.airbyte.config.persistence.ConfigNotFoundException;
Expand Down Expand Up @@ -235,7 +236,9 @@ private static void migrateExistingConnection(final ConfigRepository configRepos
throws JsonValidationException, ConfigNotFoundException, IOException {
LOGGER.info("Start migration to the new scheduler...");
final Set<UUID> connectionIds =
configRepository.listStandardSyncs().stream().map(standardSync -> standardSync.getConnectionId()).collect(Collectors.toSet());
configRepository.listStandardSyncs().stream()
.filter(standardSync -> standardSync.getStatus() == Status.ACTIVE || standardSync.getStatus() == Status.INACTIVE)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the reasoning behind filtering down to only ACTIVE and INACTIVE?

Copy link
Contributor Author

@benmoriceau benmoriceau Jan 12, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACTIVE = enabled connections
INACTIVE = disabled connections
DEPRECATED = deleted connections

We need to start the enabled ones for an obvious reason. The disabled ones need to be started too so it can be re-enable later on. The workflow will await forever if it is disabled (like if it was a manual workflow).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that makes sense to me. Though we should probably get a confirmation from @jrhizor to be sure

.map(standardSync -> standardSync.getConnectionId()).collect(Collectors.toSet());
temporalWorkerRunFactory.migrateSyncIfNeeded(connectionIds);
LOGGER.info("Done migrating to the new scheduler...");
}
Expand Down