Merge pull request #91 from dbt-labs/dave/small-tweaks

Dave/small tweaks

Author: joellabes

README.md

@@ -3,12 +3,19 @@
 Useful macros when performing data audits
 
 # Contents
-* [compare_relations](#compare_relations-source)
-* [compare_queries](#compare_queries-source)
-* [compare_column_values](#compare_column_values-source)
-* [compare_relation_columns](#compare_relation_columns-source)
-* [compare_all_columns](#compare_all_columns-source)
-* [compare_column_values_verbose](#compare_column_values_verbose-source)
+- [dbt-audit-helper](#dbt-audit-helper)
+- [Contents](#contents)
+- [Installation instructions](#installation-instructions)
+- [Macros](#macros)
+  - [compare\_relations (source)](#compare_relations-source)
+  - [compare\_queries (source)](#compare_queries-source)
+  - [compare\_column\_values (source)](#compare_column_values-source)
+    - [Usage:](#usage)
+    - [Advanced usage - dbt Cloud:](#advanced-usage---dbt-cloud)
+  - [compare\_relation\_columns (source)](#compare_relation_columns-source)
+  - [compare\_all\_columns (source)](#compare_all_columns-source)
+    - [Usage:](#usage-1)
+      - [Arguments:](#arguments)
 
 # Installation instructions
 New to dbt packages? Read more about them [here](https://docs.getdbt.com/docs/building-a-dbt-project/package-management/).
@@ -71,6 +78,7 @@ Arguments:
   results for row-by-row validation.
 * `summarize` (optional): Allows you to switch between a summary or detailed view
   of the compared data. Accepts `true` or `false` values. Defaults to `true`.
+* `limit` (optional): Allows you to limit the number of rows returned when summarize=False. Defaults to `None` (no limit).
 
 ## compare_queries ([source](macros/compare_queries.sql))
 Super similar to `compare_relations`, except it takes two select statements. This macro is useful when:
@@ -107,8 +115,13 @@ Super similar to `compare_relations`, except it takes two select statements. Thi
 ```
 
 Arguments:
-* `summarize` (optional): Allows you to switch between a summary or detaied view
-  of the compared data. Accepts `true` or `false` vaules. Defaults to `true`.
+
+* `a_query` and `b_query`: The queries you want to compare.
+* `exclude_columns` (optional): Any columns you wish to exclude from the
+  validation.
+* `summarize` (optional): Allows you to switch between a summary or detailed view
+  of the compared data. Accepts `true` or `false` values. Defaults to `true`.
+* `limit` (optional): Allows you to limit the number of rows returned when summarize=False. Defaults to `null` (no limit).
 
 ## compare_column_values ([source](macros/compare_column_values.sql))
 This macro will return a query, that, when executed, compares a column across
@@ -159,6 +172,7 @@ number of your records don't match.
 **Usage notes:**
 * `primary_key` must be a unique key in both tables, otherwise the join won't
 work as expected.
+* `emojis` is a boolean argument that defaults to `true` and displays ✅, 🤷 and ❌ for easier visual scanning. If you don't want to include emojis in the output, set it to `false`.
 
 
 ### Advanced usage - dbt Cloud:
@@ -202,19 +216,18 @@ The ``.print_table()`` function is not compatible with dbt Cloud so an adjustmen
 This macro will return a query, that, when executed, compares the ordinal_position
 and data_types of columns in two [Relations](https://docs.getdbt.com/docs/api-variable#section-relation).
 
-| column_name | a_ordinal_position | b_ordinal_position | a_data_type       | b_data_type       |
-|-------------|--------------------|--------------------|-------------------|-------------------|
-| order_id    | 1                  | 1                  | integer           | integer           |
-| customer_id | 2                  | 2                  | integer           | integer           |
-| order_date  | 3                  | 3                  | timestamp         | date              |
-| status      | 4                  | 5                  | character varying | character varying |
-| amount      | 5                  | 4                  | bigint            | bigint            |
-
+| column_name | a_ordinal_position | b_ordinal_position | a_data_type       | b_data_type       | has_ordinal_position_match | has_data_type_match | in_a_only | in_b_only | in_both |
+|-------------|--------------------|--------------------|-------------------|-------------------| -------------------------- | ------------------- | --------- | --------- | ------- |
+| order_id    | 1                  | 1                  | integer           | integer           |                       True |                True |     False |     False |    True |
+| customer_id | 2                  | 2                  | integer           | integer           |                       True |                True |     False |     False |    True |
+| order_date  | 3                  | 3                  | timestamp         | date              |                       True |               False |     False |     False |    True |
+| status      | 4                  | 5                  | character varying | character varying |                      False |                True |     False |     False |    True |
+| amount      | 5                  | 4                  | bigint            | bigint            |                      False |                True |     False |     False |    True |
 
 This is especially useful in two situations:
 1. Comparing a new version of a relation with an old one, to make sure that the
 structure is the same
-2. Helping figure out why a `union` of two relations won't work (often because
+1. Helping figure out why a `union` of two relations won't work (often because
 the data types are different)
 
 For example, in the above result set, we can see that `status` and `amount` have

macros/compare_column_values.sql

@@ -1,8 +1,8 @@
-{% macro compare_column_values(a_query, b_query, primary_key, column_to_compare) -%}
-  {{ return(adapter.dispatch('compare_column_values', 'audit_helper')(a_query, b_query, primary_key, column_to_compare)) }}
+{% macro compare_column_values(a_query, b_query, primary_key, column_to_compare, emojis=True, a_relation_name='a', b_relation_name='b') -%}
+  {{ return(adapter.dispatch('compare_column_values', 'audit_helper')(a_query, b_query, primary_key, column_to_compare, emojis, a_relation_name, b_relation_name)) }}
 {%- endmacro %}
 
-{% macro default__compare_column_values(a_query, b_query, primary_key, column_to_compare) -%}
+{% macro default__compare_column_values(a_query, b_query, primary_key, column_to_compare, emojis, a_relation_name, b_relation_name) -%}
 with a_query as (
     {{ a_query }}
 ),
@@ -17,13 +17,13 @@ joined as (
         a_query.{{ column_to_compare }} as a_query_value,
         b_query.{{ column_to_compare }} as b_query_value,
         case
-            when a_query.{{ column_to_compare }} = b_query.{{ column_to_compare }} then '✅: perfect match'
-            when a_query.{{ column_to_compare }} is null and b_query.{{ column_to_compare }} is null then '✅: both are null'
-            when a_query.{{ primary_key }} is null then '🤷: ‍missing from a'
-            when b_query.{{ primary_key }} is null then '🤷: missing from b'
-            when a_query.{{ column_to_compare }} is null then '🤷: value is null in a only'
-            when b_query.{{ column_to_compare }} is null then '🤷: value is null in b only'
-            when a_query.{{ column_to_compare }} != b_query.{{ column_to_compare }} then '🙅: ‍values do not match'
+            when a_query.{{ column_to_compare }} = b_query.{{ column_to_compare }} then '{% if emojis %}✅: {% endif %}perfect match'
+            when a_query.{{ column_to_compare }} is null and b_query.{{ column_to_compare }} is null then '{% if emojis %}✅: {% endif %}both are null'
+            when a_query.{{ primary_key }} is null then '{% if emojis %}🤷: {% endif %}missing from {{ a_relation_name }}'
+            when b_query.{{ primary_key }} is null then '{% if emojis %}🤷: {% endif %}missing from {{ b_relation_name }}'
+            when a_query.{{ column_to_compare }} is null then '{% if emojis %}🤷: {% endif %}value is null in {{ a_relation_name }} only'
+            when b_query.{{ column_to_compare }} is null then '{% if emojis %}🤷: {% endif %}value is null in {{ b_relation_name }} only'
+            when a_query.{{ column_to_compare }} != b_query.{{ column_to_compare }} then '{% if emojis %}❌: {% endif %}‍values do not match'
             else 'unknown' -- this should never happen
         end as match_status,
         case

macros/compare_queries.sql

@@ -1,8 +1,8 @@
-{% macro compare_queries(a_query, b_query, primary_key=None, summarize=true) -%}
-  {{ return(adapter.dispatch('compare_queries', 'audit_helper')(a_query, b_query, primary_key, summarize)) }}
+{% macro compare_queries(a_query, b_query, primary_key=None, summarize=true, limit=None) -%}
+  {{ return(adapter.dispatch('compare_queries', 'audit_helper')(a_query, b_query, primary_key, summarize, limit)) }}
 {%- endmacro %}
 
-{% macro default__compare_queries(a_query, b_query, primary_key=None, summarize=true) %}
+{% macro default__compare_queries(a_query, b_query, primary_key=None, summarize=true, limit=None) %}
 
 with a as (
 
@@ -106,5 +106,9 @@ final as (
 {%- endif %}
 
 select * from final
+{%- if limit and not summarize %}
+limit {{ limit }}
+{%- endif %}
+
 
 {% endmacro %}

macros/compare_relation_columns.sql

@@ -19,7 +19,10 @@ select
     a_cols.data_type as a_data_type,
     b_cols.data_type as b_data_type,
     coalesce(a_cols.ordinal_position = b_cols.ordinal_position, false) as has_ordinal_position_match,
-    coalesce(a_cols.data_type = b_cols.data_type, false) as has_data_type_match
+    coalesce(a_cols.data_type = b_cols.data_type, false) as has_data_type_match,
+    a_cols.data_type is not null and b_cols.data_type is null as in_a_only,
+    b_cols.data_type is not null and a_cols.data_type is null as in_b_only,
+    b_cols.data_type is not null and a_cols.data_type is not null as in_both
 from a_cols
 full outer join b_cols using (column_name)
 order by coalesce(a_cols.ordinal_position, b_cols.ordinal_position)

macros/compare_relations.sql

@@ -1,4 +1,4 @@
-{% macro compare_relations(a_relation, b_relation, exclude_columns=[], primary_key=None, summarize=true) %}
+{% macro compare_relations(a_relation, b_relation, exclude_columns=[], primary_key=None, summarize=true, limit=None) %}
 
 {% set column_names = dbt_utils.get_filtered_columns_in_relation(from=a_relation, except=exclude_columns) %}
 
@@ -29,6 +29,6 @@ select
 from {{ b_relation }}
 {% endset %}
 
-{{ audit_helper.compare_queries(a_query, b_query, primary_key, summarize) }}
+{{ audit_helper.compare_queries(a_query, b_query, primary_key, summarize, limit) }}
 
 {% endmacro %}