Skip to content

Commit 45bd4ad

Browse files
joellabesfivetran-joemarkiewiczdatadersKamitenshigrahamwetzler
authored
utils 0.8.1 (#502)
* Fix/timestamp withought timezone (#458) * timestamp and changelog updates * changelog fix * Add context for why change to no timezone Co-authored-by: Joel Labes <joel.labes@dbtlabs.com> * also ignore dbt_packages (#463) * also ignore dbt_packages * Update CHANGELOG.md * Update CHANGELOG.md Co-authored-by: Joel Labes <joel.labes@dbtlabs.com> * date_spine: transform comment to jinja (#462) * Have union_relations raise exception when include parameter results in no columns (#473) * Raise exception if no columns in column_superset * Add relation names to compiler error message * Add `union_relations` fix to changelog * Added case for handling postgres foreign tables... (#476) * Add link for fewer_rows_than schema test in docs (#465) * Added case for handling postgres foreign tables (tables which are external to current database and are imported into current database from remote data stores by using Foreign Data Wrappers functionallity). * Reworked getting of postges table_type. * Added needed changes to CHANGELOG. Co-authored-by: José Coto <jlcoto@users.noreply.github.com> Co-authored-by: Taras Stetsiak <tstetsiak@health-union.com> * Enhance usability of star macro by only generating column aliases when prefix and/or suffix is specified (#468) * The star macro should only produce column aliases when there is either a prefix or suffix specified. * Enhanced the readme for the star macro. * Add new integration test Co-authored-by: Nick Perrott <nperrott@roiti.com> Co-authored-by: Josh Elston-Green Co-authored-by: Joel Labes <joel.labes@dbtlabs.com> * fix: extra brace typo in insert_by_period_materialization (#480) * Support quoted column names in sequential_values test (#479) * Add any value (#501) * Add link for fewer_rows_than schema test in docs (#465) * Update get_query_results_as_dict example to demonstrate accessing columnar results as dictionary values (#474) * Update get_qu ery_results_as_dict example to demonstrate accessing columnar results as dictionary values * Use slugify in example * Fix slugify example with dbt_utils. package prefix Co-authored-by: Elize Papineau <elize.papineau@dbtlabs.com> * Add note about not_null_where deprecation to Readme (#477) * Add note about not_null_where deprecation to Readme * Add docs to unique_where test * Update pull_request_template.md to reference `main` vs `master` (#496) * Correct coalesce -> concatenation typo (#495) * add any_value cross-db macro * Missing colon in test * Update CHANGELOG.md Co-authored-by: José Coto <jlcoto@users.noreply.github.com> Co-authored-by: Elize Papineau <elizepapineau@gmail.com> Co-authored-by: Elize Papineau <elize.papineau@dbtlabs.com> Co-authored-by: Joe Ste.Marie <stemarie.joe@gmail.com> Co-authored-by: Niall Woodward <niall@niallrees.com> * Fix changelog * Second take at fixing pivot to allow single quotes (#503) * fix pivot : in pivoted column value, single quote must be escaped (on postgresql) else ex. syntax error near : when color = 'blue's' * patched expected * single quote escape : added dispatched version of the macro to support bigquery & snowflake * second backslash to escape in Jinja, change case of test file columns Let's see if other databases allow this * explicitly list columns to compare * different tests for snowflake and others * specific comparison seed * Don't quote identifiers for apostrophe, to avoid BQ and SF problems * Whitespace management for macros * Update CHANGELOG.md Co-authored-by: Marc Dutoo <marc.dutoo@gmail.com> * Add bool or cross db (#504) * Create bool_or cross-db func * Forgot a comma * Update CHANGELOG.md * Code review tweaks Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com> Co-authored-by: Anders <swanson.anders@gmail.com> Co-authored-by: Mikaël Simarik <mikael.simarik@gmail.com> Co-authored-by: Graham Wetzler <graham@wetzler.dev> Co-authored-by: Taras <32882370+Aesthet@users.noreply.github.com> Co-authored-by: José Coto <jlcoto@users.noreply.github.com> Co-authored-by: Taras Stetsiak <tstetsiak@health-union.com> Co-authored-by: nickperrott <46330920+nickperrott@users.noreply.github.com> Co-authored-by: Nick Perrott <nperrott@roiti.com> Co-authored-by: Ted Conbeer <tconbeer@users.noreply.github.com> Co-authored-by: Armand Duijn <armandduijn@users.noreply.github.com> Co-authored-by: Elize Papineau <elizepapineau@gmail.com> Co-authored-by: Elize Papineau <elize.papineau@dbtlabs.com> Co-authored-by: Joe Ste.Marie <stemarie.joe@gmail.com> Co-authored-by: Niall Woodward <niall@niallrees.com> Co-authored-by: Marc Dutoo <marc.dutoo@gmail.com>
1 parent 4ef456e commit 45bd4ad

30 files changed

+257
-24
lines changed

‎.gitignore

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11

22
target/
33
dbt_modules/
4+
dbt_packages/
45
logs/
5-
venv/
6+
venv/

‎CHANGELOG.md

+28
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,31 @@
1+
# dbt-utils v0.8.1
2+
3+
## New features
4+
- A cross-database implementation of `any_value()` ([#497](https://github.com/dbt-labs/dbt-utils/issues/497), [#501](https://github.com/dbt-labs/dbt-utils/pull/501))
5+
- A cross-database implementation of `bool_or()` ([#504](https://github.com/dbt-labs/dbt-utils/pull/504))
6+
7+
## Under the hood
8+
- also ignore `dbt_packages/` directory [#463](https://github.com/dbt-labs/dbt-utils/pull/463)
9+
- Remove block comments to make date_spine macro compatible with the Athena connector ([#462](https://github.com/dbt-labs/dbt-utils/pull/462))
10+
11+
## Fixes
12+
- `type_timestamp` macro now explicitly casts postgres and redshift warehouse timestamp data types as `timestamp without time zone`, to be consistent with Snowflake behaviour (`timestamp_ntz`).
13+
- `union_relations` macro will now raise an exception if the use of `include` or `exclude` results in no columns ([#473](https://github.com/dbt-labs/dbt-utils/pull/473), [#266](https://github.com/dbt-labs/dbt-utils/issues/266)).
14+
- `get_relations_by_pattern()` works with foreign data wrappers on Postgres again. ([#357](https://github.com/dbt-labs/dbt-utils/issues/357), [#476](https://github.com/dbt-labs/dbt-utils/pull/476))
15+
- `star()` will only alias columns if a prefix/suffix is provided, to allow the unmodified output to still be used in `group by` clauses etc. [#468](https://github.com/dbt-labs/dbt-utils/pull/468)
16+
- The `sequential_values` test is now compatible with quoted columns [#479](https://github.com/dbt-labs/dbt-utils/pull/479)
17+
- `pivot()` escapes values containing apostrophes [#503](https://github.com/dbt-labs/dbt-utils/pull/503)
18+
19+
## Contributors:
20+
- [grahamwetzler](https://github.com/grahamwetzler) (#473)
21+
- [Aesthet](https://github.com/Aesthet) (#476)
22+
- [Kamitenshi](https://github.com/Kamitenshi) (#462)
23+
- [nickperrott](https://github.com/nickperrott) (#468)
24+
- [jelstongreen](https://github.com/jelstongreen) (#468)
25+
- [armandduijn](https://github.com/armandduijn) (#479)
26+
- [mdutoo](https://github.com/mdutoo) (#503)
27+
28+
129
# dbt-utils v0.8.0
230
## 🚨 Breaking changes
331
- dbt ONE POINT OH is here! This version of dbt-utils requires _any_ version (minor and patch) of v1, which means far less need for compatibility releases in the future.

‎README.md

+3-1
Original file line numberDiff line numberDiff line change
@@ -742,7 +742,9 @@ group by 1,2,3
742742
```
743743

744744
#### star ([source](macros/sql/star.sql))
745-
This macro generates a list of all fields that exist in the `from` relation, excluding any fields listed in the `except` argument. The construction is identical to `select * from {{ref('my_model')}}`, replacing star (`*`) with the star macro. This macro also has an optional `relation_alias` argument that will prefix all generated fields with an alias (`relation_alias`.`field_name`). The macro also has optional `prefix` and `suffix` arguments, which will be appropriately concatenated to each field name in the output (`prefix` ~ `field_name` ~ `suffix`).
745+
This macro generates a comma-separated list of all fields that exist in the `from` relation, excluding any fields listed in the `except` argument. The construction is identical to `select * from {{ref('my_model')}}`, replacing star (`*`) with the star macro. This macro also has an optional `relation_alias` argument that will prefix all generated fields with an alias (`relation_alias`.`field_name`).
746+
747+
The macro also has optional `prefix` and `suffix` arguments. When one or both are provided, they will be concatenated onto each field's alias in the output (`prefix` ~ `field_name` ~ `suffix`). NB: This prevents the output from being used in any context other than a select statement.
746748

747749
**Usage:**
748750
```sql

‎integration_tests/data/cross_db/data_any_value_expected.csv

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
key_name,static_col,num_rows
2+
abc,dbt,2
3+
jkl,dbt,3
4+
xyz,test,1

‎integration_tests/data/cross_db/data_bool_or.csv

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
key,val1,val2
2+
abc,1,1
3+
abc,1,0
4+
def,1,0
5+
hij,1,1
6+
hij,1,
7+
klm,1,0
8+
klm,1,

‎integration_tests/data/cross_db/data_bool_or_expected.csv

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
key,value
2+
abc,true
3+
def,false
4+
hij,true
5+
klm,false

‎integration_tests/data/sql/data_pivot.csv

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
size,color
22
S,red
33
S,blue
4-
M,red
4+
S,blue's
5+
M,red

‎integration_tests/data/sql/data_pivot_expected.csv

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
size,red,blue
22
S,1,1
3-
M,1,0
3+
M,1,0

‎integration_tests/data/sql/data_pivot_expected_apostrophe.csv

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
size,red,blue,blues
2+
S,1,1,1
3+
M,1,0,0

‎integration_tests/data/sql/data_star_aggregate.csv

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
group_field_1,group_field_2,value_field
2+
a,b,1
3+
a,b,2
4+
c,d,3
5+
c,e,4

‎integration_tests/data/sql/data_star_aggregate_expected.csv

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
group_field_1,group_field_2,value_field_sum
2+
a,b,3
3+
c,d,3
4+
c,e,4

‎integration_tests/dbt_project.yml

-1
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,6 @@ seeds:
6060
# this.incorporate() to hardcode the node's type as otherwise dbt doesn't know it yet
6161
+post-hook: "{% do adapter.drop_relation(this.incorporate(type='table')) %}"
6262

63-
6463
schema_tests:
6564
data_test_sequential_timestamps:
6665
+column_types:

‎integration_tests/models/cross_db_utils/schema.yml

+10
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,16 @@
11
version: 2
22

33
models:
4+
- name: test_any_value
5+
tests:
6+
- dbt_utils.equality:
7+
compare_model: ref('data_any_value_expected')
8+
9+
- name: test_bool_or
10+
tests:
11+
- dbt_utils.equality:
12+
compare_model: ref('data_bool_or_expected')
13+
414
- name: test_concat
515
tests:
616
- assert_equal:

‎integration_tests/models/cross_db_utils/test_any_value.sql

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
with some_model as (
2+
select 1 as id, 'abc' as key_name, 'dbt' as static_col union all
3+
select 2 as id, 'abc' as key_name, 'dbt' as static_col union all
4+
select 3 as id, 'jkl' as key_name, 'dbt' as static_col union all
5+
select 4 as id, 'jkl' as key_name, 'dbt' as static_col union all
6+
select 5 as id, 'jkl' as key_name, 'dbt' as static_col union all
7+
select 6 as id, 'xyz' as key_name, 'test' as static_col
8+
),
9+
10+
final as (
11+
select
12+
key_name,
13+
{{ dbt_utils.any_value('static_col') }} as static_col,
14+
count(id) as num_rows
15+
from some_model
16+
group by key_name
17+
)
18+
19+
select * from final

‎integration_tests/models/cross_db_utils/test_bool_or.sql

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
select
2+
key,
3+
{{ dbt_utils.bool_or('val1 = val2') }} as value
4+
from {{ ref('data_bool_or' )}}
5+
group by key

‎integration_tests/models/sql/schema.yml

+10
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,11 @@ models:
8585
tests:
8686
- dbt_utils.equality:
8787
compare_model: ref('data_pivot_expected')
88+
89+
- name: test_pivot_apostrophe
90+
tests:
91+
- dbt_utils.equality:
92+
compare_model: ref('data_pivot_expected_apostrophe')
8893

8994
- name: test_unpivot_original_api
9095
tests:
@@ -111,6 +116,11 @@ models:
111116
- dbt_utils.equality:
112117
compare_model: ref('data_star_prefix_suffix_expected')
113118

119+
- name: test_star_aggregate
120+
tests:
121+
- dbt_utils.equality:
122+
compare_model: ref('data_star_aggregate_expected')
123+
114124
- name: test_surrogate_key
115125
tests:
116126
- assert_equal:

‎integration_tests/models/sql/test_pivot_apostrophe.sql

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
2+
-- TODO: How do we make this work nicely on Snowflake too?
3+
4+
{% if target.type == 'snowflake' %}
5+
{% set column_values = ['RED', 'BLUE', "BLUE'S"] %}
6+
{% set cmp = 'ilike' %}
7+
{% else %}
8+
{% set column_values = ['red', 'blue', "blue's"] %}
9+
{% set cmp = '=' %}
10+
{% endif %}
11+
12+
select
13+
size,
14+
{{ dbt_utils.pivot('color', column_values, cmp=cmp, quote_identifiers=False) }}
15+
16+
from {{ ref('data_pivot') }}
17+
group by size

‎integration_tests/models/sql/test_star_aggregate.sql

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
/*This test checks that column aliases aren't applied unless there's a prefix/suffix necessary, to ensure that GROUP BYs keep working*/
2+
3+
{% set selected_columns = dbt_utils.star(from=ref('data_star_aggregate'), except=['value_field']) %}
4+
5+
with data as (
6+
7+
select
8+
{{ selected_columns }},
9+
sum(value_field) as value_field_sum
10+
11+
from {{ ref('data_star_aggregate') }}
12+
group by {{ selected_columns }}
13+
14+
)
15+
16+
select * from data

‎macros/cross_db_utils/any_value.sql

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
{% macro any_value(expression) -%}
2+
{{ return(adapter.dispatch('any_value', 'dbt_utils') (expression)) }}
3+
{% endmacro %}
4+
5+
6+
{% macro default__any_value(expression) -%}
7+
8+
any_value({{ expression }})
9+
10+
{%- endmacro %}
11+
12+
13+
{% macro postgres__any_value(expression) -%}
14+
{#- /*Postgres doesn't support any_value, so we're using min() to get the same result*/ -#}
15+
min({{ expression }})
16+
17+
{%- endmacro %}

‎macros/cross_db_utils/bool_or.sql

+24
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{% macro bool_or(expression) -%}
2+
{{ return(adapter.dispatch('bool_or', 'dbt_utils') (expression)) }}
3+
{% endmacro %}
4+
5+
6+
{% macro default__bool_or(expression) -%}
7+
8+
bool_or({{ expression }})
9+
10+
{%- endmacro %}
11+
12+
13+
{% macro snowflake__bool_or(expression) -%}
14+
15+
boolor_agg({{ expression }})
16+
17+
{%- endmacro %}
18+
19+
20+
{% macro bigquery__bool_or(expression) -%}
21+
22+
logical_or({{ expression }})
23+
24+
{%- endmacro %}

‎macros/cross_db_utils/datatypes.sql

+4
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,10 @@
3232
timestamp
3333
{% endmacro %}
3434

35+
{% macro postgres__type_timestamp() %}
36+
timestamp without time zone
37+
{% endmacro %}
38+
3539
{% macro snowflake__type_timestamp() %}
3640
timestamp_ntz
3741
{% endmacro %}

‎macros/cross_db_utils/escape_single_quotes.sql

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{% macro escape_single_quotes(expression) %}
2+
{{ return(adapter.dispatch('escape_single_quotes', 'dbt_utils') (expression)) }}
3+
{% endmacro %}
4+
5+
{# /*Default to replacing a single apostrophe with two apostrophes: they're -> they''re*/ #}
6+
{% macro default__escape_single_quotes(expression) -%}
7+
{{ expression | replace("'","''") }}
8+
{%- endmacro %}
9+
10+
{# /*Snowflake uses a single backslash: they're -> they\'re. The second backslash is to escape it from Jinja */ #}
11+
{% macro snowflake__escape_single_quotes(expression) -%}
12+
{{ expression | replace("'", "\\'") }}
13+
{%- endmacro %}
14+
15+
{# /*BigQuery uses a single backslash: they're -> they\'re. The second backslash is to escape it from Jinja */ #}
16+
{% macro bigquery__escape_single_quotes(expression) -%}
17+
{{ expression | replace("'", "\\'") }}
18+
{%- endmacro %}

‎macros/materializations/insert_by_period_materialization.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@
5353
{% materialization insert_by_period, default -%}
5454
{%- set timestamp_field = config.require('timestamp_field') -%}
5555
{%- set start_date = config.require('start_date') -%}
56-
{%- set stop_date = config.get('stop_date') or '' -%}}
56+
{%- set stop_date = config.get('stop_date') or '' -%}
5757
{%- set period = config.get('period') or 'week' -%}
5858

5959
{%- if sql.find('__PERIOD_FILTER__') == -1 -%}

‎macros/schema_tests/sequential_values.sql

+5-3
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,15 @@
66

77
{% macro default__test_sequential_values(model, column_name, interval=1, datepart=None) %}
88

9+
{% set previous_column_name = "previous_" ~ dbt_utils.slugify(column_name) %}
10+
911
with windowed as (
1012

1113
select
1214
{{ column_name }},
1315
lag({{ column_name }}) over (
1416
order by {{ column_name }}
15-
) as previous_{{ column_name }}
17+
) as {{ previous_column_name }}
1618
from {{ model }}
1719
),
1820

@@ -21,9 +23,9 @@ validation_errors as (
2123
*
2224
from windowed
2325
{% if datepart %}
24-
where not(cast({{ column_name }} as {{ dbt_utils.type_timestamp() }})= cast({{ dbt_utils.dateadd(datepart, interval, 'previous_' + column_name) }} as {{ dbt_utils.type_timestamp() }}))
26+
where not(cast({{ column_name }} as {{ dbt_utils.type_timestamp() }})= cast({{ dbt_utils.dateadd(datepart, interval, previous_column_name) }} as {{ dbt_utils.type_timestamp() }}))
2527
{% else %}
26-
where not({{ column_name }} = previous_{{ column_name }} + {{ interval }})
28+
where not({{ column_name }} = {{ previous_column_name }} + {{ interval }})
2729
{% endif %}
2830
)
2931

‎macros/sql/date_spine.sql

+3-4
Original file line numberDiff line numberDiff line change
@@ -29,16 +29,15 @@
2929

3030
{% macro default__date_spine(datepart, start_date, end_date) %}
3131

32-
/*
33-
call as follows:
32+
33+
{# call as follows:
3434

3535
date_spine(
3636
"day",
3737
"to_date('01/01/2016', 'mm/dd/yyyy')",
3838
"dateadd(week, 1, current_date)"
39-
)
39+
) #}
4040

41-
*/
4241

4342
with rawdata as (
4443

‎macros/sql/get_table_types_sql.sql

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{%- macro get_table_types_sql() -%}
2+
{{ return(adapter.dispatch('get_table_types_sql', 'dbt_utils')()) }}
3+
{%- endmacro -%}
4+
5+
{% macro default__get_table_types_sql() %}
6+
case table_type
7+
when 'BASE TABLE' then 'table'
8+
when 'EXTERNAL TABLE' then 'external'
9+
when 'MATERIALIZED VIEW' then 'materializedview'
10+
else lower(table_type)
11+
end as "table_type"
12+
{% endmacro %}
13+
14+
15+
{% macro postgres__get_table_types_sql() %}
16+
case table_type
17+
when 'BASE TABLE' then 'table'
18+
when 'FOREIGN' then 'external'
19+
when 'MATERIALIZED VIEW' then 'materializedview'
20+
else lower(table_type)
21+
end as "table_type"
22+
{% endmacro %}

‎macros/sql/get_tables_by_pattern_sql.sql

+1-6
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,7 @@
88
select distinct
99
table_schema as "table_schema",
1010
table_name as "table_name",
11-
case table_type
12-
when 'BASE TABLE' then 'table'
13-
when 'EXTERNAL TABLE' then 'external'
14-
when 'MATERIALIZED VIEW' then 'materializedview'
15-
else lower(table_type)
16-
end as "table_type"
11+
{{ dbt_utils.get_table_types_sql() }}
1712
from {{ database }}.information_schema.tables
1813
where table_schema ilike '{{ schema_pattern }}'
1914
and table_name ilike '{{ table_pattern }}'

0 commit comments

Comments
 (0)