Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add metadata to segment tracking #8872

Merged
merged 6 commits into from
Jan 14, 2022
Merged

Add metadata to segment tracking #8872

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
f50dd2e
Add metadata to segment tracking
ChristopheDuong Dec 17, 2021
65735b8
Add sync start time
ChristopheDuong Dec 17, 2021
2f08fb8
Merge remote-tracking branch 'origin/master' into add-tracking-metadata
ChristopheDuong Jan 5, 2022
799af89
Merge remote-tracking branch 'origin/master' into add-tracking-metadata
ChristopheDuong Jan 12, 2022
f21130d
Fix test and format code
ChristopheDuong Jan 14, 2022
ae5e856
Merge remote-tracking branch 'origin/master' into add-tracking-metadata
ChristopheDuong Jan 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions airbyte-analytics/src/main/java/io/airbyte/analytics/SegmentTrackingClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
import com.segment.analytics.messages.IdentifyMessage;
import com.segment.analytics.messages.TrackMessage;
import io.airbyte.config.StandardWorkspace;
import java.time.Instant;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
Expand Down Expand Up @@ -43,6 +44,7 @@ public class SegmentTrackingClient implements TrackingClient {
private static final String SEGMENT_WRITE_KEY = "7UDdp5K55CyiGgsauOr2pNNujGvmhaeu";
private static final String AIRBYTE_VERSION_KEY = "airbyte_version";
private static final String AIRBYTE_ROLE = "airbyte_role";
private static final String AIRBYTE_TRACKED_AT = "tracked_at";

// Analytics is threadsafe.
private final Analytics analytics;
Expand Down Expand Up @@ -116,6 +118,7 @@ public void track(final UUID workspaceId, final String action, final Map<String,
// Always add these traits.
mapCopy.put(AIRBYTE_VERSION_KEY, trackingIdentity.getAirbyteVersion().serialize());
mapCopy.put(CUSTOMER_ID_KEY, trackingIdentity.getCustomerId());
mapCopy.put(AIRBYTE_TRACKED_AT, Instant.now().toString());
if (!metadata.isEmpty()) {
trackingIdentity.getEmail().ifPresent(email -> mapCopy.put("email", email));
}
Expand Down
3 changes: 3 additions & 0 deletions ...sistence/src/main/java/io/airbyte/scheduler/persistence/job_tracker/TrackingMetadata.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ public static ImmutableMap<String, Object> generateDestinationDefinitionMetadata
final Builder<String, Object> metadata = ImmutableMap.builder();
metadata.put("connector_destination", destinationDefinition.getName());
metadata.put("connector_destination_definition_id", destinationDefinition.getDestinationDefinitionId());
metadata.put("connector_destination_docker_repository", destinationDefinition.getDockerRepository());
final String imageTag = destinationDefinition.getDockerImageTag();
if (!Strings.isEmpty(imageTag)) {
metadata.put("connector_destination_version", imageTag);
Expand All @@ -77,6 +78,7 @@ public static ImmutableMap<String, Object> generateSourceDefinitionMetadata(fina
final Builder<String, Object> metadata = ImmutableMap.builder();
metadata.put("connector_source", sourceDefinition.getName());
metadata.put("connector_source_definition_id", sourceDefinition.getSourceDefinitionId());
metadata.put("connector_source_docker_repository", sourceDefinition.getDockerRepository());
final String imageTag = sourceDefinition.getDockerImageTag();
if (!Strings.isEmpty(imageTag)) {
metadata.put("connector_source_version", imageTag);
Expand All @@ -94,6 +96,7 @@ public static ImmutableMap<String, Object> generateJobAttemptMetadata(final Job
final JobOutput jobOutput = lastAttempt.getOutput().get();
if (jobOutput.getSync() != null) {
final StandardSyncSummary syncSummary = jobOutput.getSync().getStandardSyncSummary();
metadata.put("sync_start_time", syncSummary.getStartTime());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we also add end time?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duration is computed using the end time and we now have start time so we can always re-compute the reverse.
Let's leave it out to make the amount of data transiting through segment a little smaller

metadata.put("duration", Math.round((syncSummary.getEndTime() - syncSummary.getStartTime()) / 1000.0));
metadata.put("volume_mb", syncSummary.getBytesSynced());
metadata.put("volume_rows", syncSummary.getRecordsSynced());
Expand Down