Skip to content

Commit 5f396cc

Browse files
jtcohen6jtcohen6
authored
Merge pull request #267 from fishtown-analytics/feature/0-18-adapter-dispatch
Feature: use adapter.dispatch
2 parents f13f9b5 + 9760ba1 commit 5f396cc

31 files changed

+131
-76
lines changed

‎CHANGELOG.md

+6
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,12 @@
44

55
## Features
66

7+
* Switch usage of `adapter_macro` to `adapter.dispatch`, and define `dbt_utils_dispatch_list`,
8+
enabling users of community-supported database plugins to add or override macro implementations
9+
specific to their database (#267)
10+
* Use `add_ephemeral_prefix` instead of hard-coding a string literal, to support
11+
database adapters that use different prefixes (#267)
12+
713
## Quality of life
814

915
# dbt-utils v0.5.1

‎README.md

+36
Original file line numberDiff line numberDiff line change
@@ -777,6 +777,42 @@ We welcome contributions to this repo! To contribute a new feature or a fix, ple
777777

778778
----
779779

780+
### Dispatch macros
781+
782+
**Note:** This is primarily relevant to users and maintainers of community-supported
783+
database plugins. If you use Postgres, Redshift, Snowflake, or Bigquery, this likely
784+
does not apply to you.
785+
786+
dbt v0.18.0 introduces `adapter.dispatch()`, a reliable way to define different implementations of the same macro
787+
across different databases.
788+
789+
All dispatched macros in `dbt_utils` have an override setting: a `var` named
790+
`dbt_utils_dispatch_list` that accepts a list of package names. If you set this
791+
variable in your project, when dbt searches for implementations of a dispatched
792+
`dbt_utils` macro, it will search through your listed packages _before_ using
793+
the implementations defined in `dbt_utils`.
794+
795+
Set the variable:
796+
```yml
797+
vars:
798+
dbt_utils_dispatch_list:
799+
- first_package_to_search # likely the name of your root project
800+
- second_package_to_search # likely an "add-on" package, such as spark_utils
801+
# dbt_utils is always the last place searched
802+
```
803+
804+
When running on Spark, if dbt needs to dispatch `dbt_utils.datediff`, it will search for the following in order:
805+
```
806+
first_package_to_search.spark__datediff
807+
first_package_to_search.default__datediff
808+
second_package_to_search.spark__datediff
809+
second_package_to_search.default__datediff
810+
dbt_utils.spark__datediff
811+
dbt_utils.default__datediff
812+
```
813+
814+
----
815+
780816
### Getting started with dbt
781817

782818
- [What is dbt]?

‎dbt_project.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
name: 'dbt_utils'
22
version: '0.1.0'
33

4-
require-dbt-version: [">=0.17.0", "<0.18.0"]
4+
require-dbt-version: [">=0.18.0", "<0.19.0"]
55
config-version: 2
66

77
target-path: "target"

‎integration_tests/models/datetime/test_date_spine.sql

+1-2
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,7 @@
99
with date_spine as (
1010

1111
{% if target.type == 'postgres' %}
12-
{{ log("WARNING: Not testing - datediff macro is unsupported on Postgres", info=True) }}
13-
select * from {{ ref('data_date_spine') }}
12+
{{ dbt_utils.date_spine("day", "'2018-01-01'::date", "'2018-01-10'::date") }}
1413

1514
{% elif target.type == 'bigquery' %}
1615
select cast(date_day as date) as date_day

‎integration_tests/models/sql/test_get_relations_by_pattern.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
{{ config(materialized = 'table') }}
22

3-
{% set relations = dbt_utils.get_relations_by_pattern(target.schema, 'data_events_') %}
3+
{% set relations = dbt_utils.get_relations_by_pattern(target.schema, 'data_events_%') %}
44
{{ dbt_utils.union_relations(relations) }}

‎macros/cross_db_utils/_get_utils_namespaces.sql

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{% macro _get_utils_namespaces() %}
2+
{% set override_namespaces = var('dbt_utils_dispatch_list', []) %}
3+
{% do return(override_namespaces + ['dbt_utils']) %}
4+
{% endmacro %}

‎macros/cross_db_utils/_is_ephemeral.sql

+3-2
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
{% macro _is_ephemeral(obj, macro) %}
22
{%- if obj.is_cte -%}
3-
{% if obj.name.startswith('__dbt__CTE__') %}
4-
{% set model_name = obj.name[12:] %}
3+
{% set ephemeral_prefix = api.Relation.add_ephemeral_prefix('') %}
4+
{% if obj.name.startswith(ephemeral_prefix) %}
5+
{% set model_name = obj.name[(ephemeral_prefix|length):] %}
56
{% else %}
67
{% set model_name = obj.name %}
78
{%- endif -%}

‎macros/cross_db_utils/concat.sql

+5-5
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,23 @@
1+
2+
13
{% macro concat(fields) -%}
2-
{{ adapter_macro('dbt_utils.concat', fields) }}
4+
{{ adapter.dispatch('concat', packages = dbt_utils._get_utils_namespaces())(fields) }}
35
{%- endmacro %}
46

5-
67
{% macro default__concat(fields) -%}
78
concat({{ fields|join(', ') }})
89
{%- endmacro %}
910

10-
1111
{% macro alternative_concat(fields) %}
1212
{{ fields|join(' || ') }}
1313
{% endmacro %}
1414

1515

1616
{% macro redshift__concat(fields) %}
17-
{{dbt_utils.alternative_concat(fields)}}
17+
{{ dbt_utils.alternative_concat(fields) }}
1818
{% endmacro %}
1919

2020

2121
{% macro snowflake__concat(fields) %}
22-
{{dbt_utils.alternative_concat(fields)}}
22+
{{ dbt_utils.alternative_concat(fields) }}
2323
{% endmacro %}

‎macros/cross_db_utils/current_timestamp.sql

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{% macro current_timestamp() -%}
2-
{{ adapter_macro('dbt_utils.current_timestamp') }}
2+
{{ adapter.dispatch('current_timestamp', packages = dbt_utils._get_utils_namespaces())() }}
33
{%- endmacro %}
44

55
{% macro default__current_timestamp() %}
@@ -17,7 +17,7 @@
1717

1818

1919
{% macro current_timestamp_in_utc() -%}
20-
{{ adapter_macro('dbt_utils.current_timestamp_in_utc') }}
20+
{{ adapter.dispatch('current_timestamp_in_utc', packages = dbt_utils._get_utils_namespaces())() }}
2121
{%- endmacro %}
2222

2323
{% macro default__current_timestamp_in_utc() %}

‎macros/cross_db_utils/datatypes.sql

+6-6
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{# string ------------------------------------------------- #}
22

33
{%- macro type_string() -%}
4-
{{ adapter_macro('dbt_utils.type_string') }}
4+
{{ adapter.dispatch('type_string', packages = dbt_utils._get_utils_namespaces())() }}
55
{%- endmacro -%}
66

77
{% macro default__type_string() %}
@@ -25,7 +25,7 @@
2525
{# timestamp ------------------------------------------------- #}
2626

2727
{%- macro type_timestamp() -%}
28-
{{ adapter_macro('dbt_utils.type_timestamp') }}
28+
{{ adapter.dispatch('type_timestamp', packages = dbt_utils._get_utils_namespaces())() }}
2929
{%- endmacro -%}
3030

3131
{% macro default__type_timestamp() %}
@@ -40,7 +40,7 @@
4040
{# float ------------------------------------------------- #}
4141

4242
{%- macro type_float() -%}
43-
{{ adapter_macro('dbt_utils.type_float') }}
43+
{{ adapter.dispatch('type_float', packages = dbt_utils._get_utils_namespaces())() }}
4444
{%- endmacro -%}
4545

4646
{% macro default__type_float() %}
@@ -54,7 +54,7 @@
5454
{# numeric ------------------------------------------------ #}
5555

5656
{%- macro type_numeric() -%}
57-
{{ adapter_macro('dbt_utils.type_numeric') }}
57+
{{ adapter.dispatch('type_numeric', packages = dbt_utils._get_utils_namespaces())() }}
5858
{%- endmacro -%}
5959

6060
{% macro default__type_numeric() %}
@@ -69,7 +69,7 @@
6969
{# bigint ------------------------------------------------- #}
7070

7171
{%- macro type_bigint() -%}
72-
{{ adapter_macro('dbt_utils.type_bigint') }}
72+
{{ adapter.dispatch('type_bigint', packages = dbt_utils._get_utils_namespaces())() }}
7373
{%- endmacro -%}
7474

7575
{% macro default__type_bigint() %}
@@ -83,7 +83,7 @@
8383
{# int ------------------------------------------------- #}
8484

8585
{%- macro type_int() -%}
86-
{{ adapter_macro('dbt_utils.type_int') }}
86+
{{ adapter.dispatch('type_int', packages = dbt_utils._get_utils_namespaces())() }}
8787
{%- endmacro -%}
8888

8989
{% macro default__type_int() %}

‎macros/cross_db_utils/date_trunc.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{% macro date_trunc(datepart, date) -%}
2-
{{ adapter_macro('dbt_utils.date_trunc', datepart, date) }}
2+
{{ adapter.dispatch('date_trunc', packages = dbt_utils._get_utils_namespaces()) (datepart, date) }}
33
{%- endmacro %}
44

55
{% macro default__date_trunc(datepart, date) %}

‎macros/cross_db_utils/dateadd.sql

+1-2
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{% macro dateadd(datepart, interval, from_date_or_timestamp) %}
2-
{{ adapter_macro('dbt_utils.dateadd', datepart, interval, from_date_or_timestamp) }}
2+
{{ adapter.dispatch('dateadd', packages = dbt_utils._get_utils_namespaces())(datepart, interval, from_date_or_timestamp) }}
33
{% endmacro %}
44

55

@@ -23,7 +23,6 @@
2323

2424
{% endmacro %}
2525

26-
2726
{% macro postgres__dateadd(datepart, interval, from_date_or_timestamp) %}
2827

2928
{{ from_date_or_timestamp }} + ((interval '1 {{ datepart }}') * ({{ interval }}))

‎macros/cross_db_utils/datediff.sql

+1-2
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{% macro datediff(first_date, second_date, datepart) %}
2-
{{ adapter_macro('dbt_utils.datediff', first_date, second_date, datepart) }}
2+
{{ adapter.dispatch('datediff', packages = dbt_utils._get_utils_namespaces())(first_date, second_date, datepart) }}
33
{% endmacro %}
44

55

@@ -24,7 +24,6 @@
2424

2525
{% endmacro %}
2626

27-
2827
{% macro postgres__datediff(first_date, second_date, datepart) %}
2928

3029
{% if datepart == 'year' %}

‎macros/cross_db_utils/except.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{% macro except() %}
2-
{{ adapter_macro('dbt_utils.except') }}
2+
{{ adapter.dispatch('except', packages = dbt_utils._get_utils_namespaces())() }}
33
{% endmacro %}
44

55

‎macros/cross_db_utils/hash.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{% macro hash(field) -%}
2-
{{ adapter_macro('dbt_utils.hash', field) }}
2+
{{ adapter.dispatch('hash', packages = dbt_utils._get_utils_namespaces()) (field) }}
33
{%- endmacro %}
44

55

‎macros/cross_db_utils/identifier.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
Use `adapter.quote` instead. The {}.{} model triggered this warning. \
55
'.format(model.package_name, model.name) -%}
66
{%- do exceptions.warn(error_message) -%}
7-
{{ adapter_macro('dbt_utils.identifier', value) }}
7+
{{ adapter.dispatch('identifier', packages = dbt_utils._get_utils_namespaces()) (value) }}
88
{% endmacro %}
99

1010
{% macro default__identifier(value) -%}

‎macros/cross_db_utils/intersect.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{% macro intersect() %}
2-
{{ adapter_macro('dbt_utils.intersect') }}
2+
{{ adapter.dispatch('intersect', packages = dbt_utils._get_utils_namespaces())() }}
33
{% endmacro %}
44

55

‎macros/cross_db_utils/last_day.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ testing is required to validate that it will work on other dateparts.
44
*/
55

66
{% macro last_day(date, datepart) %}
7-
{{ adapter_macro('dbt_utils.last_day', date, datepart) }}
7+
{{ adapter.dispatch('last_day', packages = dbt_utils._get_utils_namespaces()) (date, datepart) }}
88
{% endmacro %}
99

1010

‎macros/cross_db_utils/length.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{% macro length(expression) -%}
2-
{{ adapter_macro('dbt_utils.length', expression) }}
2+
{{ adapter.dispatch('length', packages = dbt_utils._get_utils_namespaces()) (expression) }}
33
{% endmacro %}
44

55

‎macros/cross_db_utils/literal.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11

22
{%- macro string_literal(value) -%}
3-
{{ adapter_macro('dbt_utils.string_literal', value) }}
3+
{{ adapter.dispatch('string_literal', packages = dbt_utils._get_utils_namespaces()) (value) }}
44
{%- endmacro -%}
55

66
{% macro default__string_literal(value) -%}

‎macros/cross_db_utils/position.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{% macro position(substring_text, string_text) -%}
2-
{{ adapter_macro('dbt_utils.position', substring_text, string_text) }}
2+
{{ adapter.dispatch('position', packages = dbt_utils._get_utils_namespaces()) (substring_text, string_text) }}
33
{% endmacro %}
44

55

‎macros/cross_db_utils/replace.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{% macro replace(field, old_chars, new_chars) -%}
2-
{{ adapter_macro('dbt_utils.replace', field, old_chars, new_chars) }}
2+
{{ adapter.dispatch('replace', packages = dbt_utils._get_utils_namespaces()) (field, old_chars, new_chars) }}
33
{% endmacro %}
44

55

‎macros/cross_db_utils/right.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{% macro right(string_text, length_expression) -%}
2-
{{ adapter_macro('dbt_utils.right', string_text, length_expression) }}
2+
{{ adapter.dispatch('right', packages = dbt_utils._get_utils_namespaces()) (string_text, length_expression) }}
33
{% endmacro %}
44

55
{% macro default__right(string_text, length_expression) %}

‎macros/cross_db_utils/safe_cast.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{% macro safe_cast(field, type) %}
2-
{{ adapter_macro('dbt_utils.safe_cast', field, type) }}
2+
{{ adapter.dispatch('safe_cast', packages = dbt_utils._get_utils_namespaces()) (field, type) }}
33
{% endmacro %}
44

55

‎macros/cross_db_utils/split_part.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{% macro split_part(string_text, delimiter_text, part_number) %}
2-
{{ adapter_macro('dbt_utils.split_part', string_text, delimiter_text, part_number) }}
2+
{{ adapter.dispatch('split_part', packages = dbt_utils._get_utils_namespaces()) (string_text, delimiter_text, part_number) }}
33
{% endmacro %}
44

55

‎macros/cross_db_utils/width_bucket.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{% macro width_bucket(expr, min_value, max_value, num_buckets) %}
2-
{{ adapter_macro('dbt_utils.width_bucket', expr, min_value, max_value, num_buckets) }}
2+
{{ adapter.dispatch('width_bucket', packages = dbt_utils._get_utils_namespaces()) (expr, min_value, max_value, num_buckets) }}
33
{% endmacro %}
44

55

‎macros/sql/get_relations_by_pattern.sql

+14-2
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
{%- call statement('get_tables', fetch_result=True) %}
44

5-
{{ dbt_utils.get_tables_by_pattern(schema_pattern, table_pattern, exclude, database) }}
5+
{{ dbt_utils.get_tables_by_pattern_sql(schema_pattern, table_pattern, exclude, database) }}
66

77
{%- endcall -%}
88

@@ -20,4 +20,16 @@
2020
{{ return([]) }}
2121
{%- endif -%}
2222

23-
{% endmacro %}
23+
{% endmacro %}
24+
25+
{% macro get_tables_by_pattern(schema_pattern, table_pattern, exclude='', database=target.database) %}
26+
{%- set error_message = '
27+
Warning: the `get_tables_by_pattern` macro is no longer supported and will be deprecated in a future release of dbt-utils. \
28+
Use the `get_relations_by_prefix` macro instead. \
29+
The {}.{} model triggered this warning. \
30+
'.format(model.package_name, model.name) -%}
31+
{%- do exceptions.warn(error_message) -%}
32+
33+
{{ return(dbt_utils.get_relations_by_pattern(schema_pattern, table_pattern, exclude='', database=target.database)) }}
34+
35+
{% endmacro %}

‎macros/sql/get_tables_by_pattern.sql

-12
This file was deleted.

‎macros/sql/get_tables_by_pattern_sql.sql

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{% macro get_tables_by_pattern_sql(schema_pattern, table_pattern, exclude='', database=target.database) %}
2+
{{ adapter.dispatch('get_tables_by_pattern_sql', packages = dbt_utils._get_utils_namespaces())
3+
(schema_pattern, table_pattern, exclude='', database=target.database) }}
4+
{% endmacro %}
5+
6+
{% macro default__get_tables_by_pattern_sql(schema_pattern, table_pattern, exclude='', database=target.database) %}
7+
8+
select distinct
9+
table_schema as "table_schema", table_name as "table_name"
10+
from {{database}}.information_schema.tables
11+
where table_schema ilike '{{ schema_pattern }}'
12+
and table_name ilike '{{ table_pattern }}'
13+
and table_name not ilike '{{ exclude }}'
14+
15+
{% endmacro %}
16+
17+
18+
{% macro bigquery__get_tables_by_pattern_sql(schema_pattern, table_pattern, exclude='', database=target.database) %}
19+
20+
select distinct
21+
table_schema, table_name
22+
23+
from {{adapter.quote(database)}}.{{schema}}.INFORMATION_SCHEMA.TABLES
24+
where table_schema = '{{schema_pattern}}'
25+
and lower(table_name) like lower ('{{table_pattern}}')
26+
and lower(table_name) not like lower ('{{exclude}}')
27+
28+
{% endmacro %}

0 commit comments

Comments
 (0)