|
| 1 | +--- |
| 2 | +title: "Managing data products via the CLI" |
| 3 | +description: "Use the 'snowplow-cli data-products' command to manage your data products." |
| 4 | +sidebar_label: "Using the CLI" |
| 5 | +sidebar_position: 999 |
| 6 | +--- |
| 7 | +```mdx-code-block |
| 8 | +import Tabs from '@theme/Tabs'; |
| 9 | +import TabItem from '@theme/TabItem'; |
| 10 | +``` |
| 11 | +The `data-products` subcommand of [Snowplow CLI](/docs/data-product-studio/snowplow-cli/index.md) provides a collection of functionality to ease the integration of custom development and publishing workflows. |
| 12 | +## Snowplow CLI Prerequisites |
| 13 | +Installed and configured [Snowplow CLI](/docs/data-product-studio/snowplow-cli/index.md) |
| 14 | +## Available commands |
| 15 | +### Creating data product |
| 16 | +```bash |
| 17 | +./snowplow-cli dp generate --data-product my-data-product |
| 18 | +``` |
| 19 | +This command creates a minimal data product template in a new file `./data-products/my-data-product.yaml`. |
| 20 | +### Creating source application |
| 21 | +```bash |
| 22 | +./snowplow-cli dp generate --source-app my-source-app |
| 23 | +``` |
| 24 | +This command creates a minimal source application template in a new file `./data-products/source-apps/my-source-app.yaml`. |
| 25 | +### Creating event specification |
| 26 | +To create an event specification, you need to modify the existing data-product file and add an event specification object. Here's a minimal example: |
| 27 | +```yaml title="./data-products/test-cli.yaml" |
| 28 | +apiVersion: v1 |
| 29 | +resourceType: data-product |
| 30 | +resourceName: 3d3059c4-d29b-4979-a973-43f7070e1dd0 |
| 31 | +data: |
| 32 | + name: test-cli |
| 33 | + sourceApplications: [] |
| 34 | + eventSpecifications: |
| 35 | + - resourceName: 11d881cd-316e-4286-b5d4-fe7aebf56fca |
| 36 | + name: test |
| 37 | + event: |
| 38 | + source: iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0 |
| 39 | +``` |
| 40 | +:::caution Warning |
| 41 | +The `source` fields of events and entities must refer to a deployed data structure. Referring to a locally created data structure is not yet supported. |
| 42 | +::: |
| 43 | +### Linking data product to a source application |
| 44 | +To link a data product to a source application, provide a list of references to the source application files in the `data.sourceApplications` field. Here's an example: |
| 45 | +```yaml title="./data-products/test-cli.yaml" |
| 46 | +apiVersion: v1 |
| 47 | +resourceType: data-product |
| 48 | +resourceName: 3d3059c4-d29b-4979-a973-43f7070e1dd0 |
| 49 | +data: |
| 50 | + name: test-cli |
| 51 | + sourceApplications: |
| 52 | + - $ref: ./source-apps/my-source-app.yaml |
| 53 | +``` |
| 54 | +### Modifying the event specifications source applications |
| 55 | +By default event specifications inherit all the source applications of the data product. If you want to customise it, you can use the `excludedSourceApplications` in the event specification description to remove a given source application from an event specification. |
| 56 | +```yaml title="./data-products/test-cli.yaml" |
| 57 | +apiVersion: v1 |
| 58 | +resourceType: data-product |
| 59 | +resourceName: 3d3059c4-d29b-4979-a973-43f7070e1dd0 |
| 60 | +data: |
| 61 | + name: test-cli |
| 62 | + sourceApplications: |
| 63 | + - $ref: ./source-apps/generic.yaml |
| 64 | + - $ref: ./source-apps/specific.yaml |
| 65 | + eventSpecifications: |
| 66 | + - resourceName: 11d881cd-316e-4286-b5d4-fe7aebf56fca |
| 67 | + name: All source apps |
| 68 | + event: |
| 69 | + source: iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0 |
| 70 | + - resourceName: b9c994a0-03b2-479c-b1cf-7d25c3adc572 |
| 71 | + name: Not quite everything |
| 72 | + excludedSourceApplications: |
| 73 | + - $ref: ./source-apps/specific.yaml |
| 74 | + event: |
| 75 | + source: iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0 |
| 76 | +``` |
| 77 | +In this example event specification `All source apps` is related to both `generic` and `specific` source apps, but event specification `Not quite everything` is related only to the `generic` source application. |
| 78 | +### Downloading data products, event specifications and source apps |
| 79 | +```bash |
| 80 | +./snowplow-cli dp download |
| 81 | +``` |
| 82 | +This command retrieves all organization data products, event specifications, and source applications. By default, it creates a folder named `data-products` in your current working directory. You can specify a different folder name as an argument if needed. |
| 83 | +The command creates the following structure: |
| 84 | +- A main `data-products` folder containing your data product files |
| 85 | +- A `source-apps` subfolder containing source application definitions |
| 86 | +- Event specifications embedded within their related data product files. |
| 87 | +### Validating data products, event specifications and source applications |
| 88 | +```bash |
| 89 | +./snowplow-cli dp validate |
| 90 | +``` |
| 91 | +This command scans all files under `./data-products` and validates them using the BDP console. It checks: |
| 92 | +1. Whether each file is in a valid format (YAML/JSON) with correctly formatted fields |
| 93 | +2. Whether all source application references in the data product files are valid |
| 94 | +3. Whether event specification rules are compatible with their schemas |
| 95 | +If validation fails, the command displays the errors in the console and exits with status code 1. |
| 96 | +### Publishing data products, event specifications and source applications |
| 97 | +```bash |
| 98 | +./snowplow-cli dp publish |
| 99 | +``` |
| 100 | +This command locates all files under `./data-products`, validates them, and publishes them to the BDP console. |
0 commit comments