Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery Not Expanding Source Variables #92

Open
1 of 6 tasks
bassrock opened this issue Nov 21, 2024 · 0 comments
Open
1 of 6 tasks

BigQuery Not Expanding Source Variables #92

bassrock opened this issue Nov 21, 2024 · 0 comments
Labels
status:needs_triage Needs maintainer triage. type:bug Bugs or weaknesses. The issue has to contain steps to reproduce.

Comments

@bassrock
Copy link

Describe the bug

Using a variable or Jinja syntax in the variable for: snowplow__database or snowplow__atomic_schema does not expand when used with BigQuery.

Steps to reproduce

Using bigquery, within the DBT config put:

  snowplow_unified:
    snowplow__database: "{{ env_var('DBT_DATBASE', 'default') }}"
    snowplow__atomic_schema:  "{{ env_var('DBT_SCHEMA', 'default') }}"

NOTE: This does work for Snowflake, but not BigQuery.

Expected results

The models to execute successfully using the env_var that is used.

Actual results

I receive the following error:

22:18:45 6 of 11 START sql table model default.dbt_scratch.snowplow_unified_base_events_this_run  [RUN]
22:18:46 Unhandled error while executing 
400 GET https://bigquery.googleapis.com/bigquery/v2/projects/%7B%7B%20env_var('DBT_DATABASE',%20'default')%20%7D%7D/datasets/%7B%7B%20env_var('DBT_SCHEMA',%20'default')%20%7D%7D/tables/events?prettyPrint=false: Invalid resource name projects/{{ env_var('DBT_DATABASE', 'default') }}; Project id: {{ env_var('DBT_DATABASE', 'default') }}
22:18:46 6 of 11 ERROR creating sql table model default.dbt_scratch.snowplow_unified_base_events_this_run  [ERROR in 1.42s]

Upon investigation, the base macro is returniogn the right sql. It looks like the error is within the get_context field macros when they call source('atomic', 'events`) . Because the underlying source file is not expanding the snowplow variables.

System information

The contents of your packages.yml file:

packages:
  - package: snowplow/snowplow_unified
    version: 0.4.5
  - package: snowplow/snowplow_utils
    version: 0.16.8
  - package: snowplow/snowplow_normalize
    version: 0.3.5

Which database are you using dbt with?

  • postgres
  • redshift
  • bigquery
  • snowflake
  • databricks
  • other (specify: ____________)

The output of dbt --version:

dbt Cloud CLI - 0.38.22 (1183c2abdb6003083b0fa91fcd89cd5feb25f9f7 2024-11-20T15:49:01Z)

The operating system you're using:
Mac OS Sequoia

The output of python --version:
Python 3.8.18

@bassrock bassrock added the type:bug Bugs or weaknesses. The issue has to contain steps to reproduce. label Nov 21, 2024
@github-actions github-actions bot added the status:needs_triage Needs maintainer triage. label Nov 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status:needs_triage Needs maintainer triage. type:bug Bugs or weaknesses. The issue has to contain steps to reproduce.
Projects
None yet
Development

No branches or pull requests

1 participant