Skip to content

Commit

Permalink
Prepare for release
Browse files Browse the repository at this point in the history
  • Loading branch information
agnessnowplow committed Sep 2, 2022
1 parent 22fe5e1 commit 7b2c528
Show file tree
Hide file tree
Showing 30 changed files with 50 additions and 42 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,7 @@
"meta": {
"versions": {
"test_suite_version": "1.1.1",
"redshift_model_version": "1.3.0"
"redshift_model_version": "1.3.1"
},
"great_expectations.__version__": "0.12.0"
}
Expand Down
2 changes: 1 addition & 1 deletion .test/great_expectations/expectations/web/v1/metadata.json
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@
"meta": {
"versions": {
"test_suite_version": "1.1.1",
"redshift_model_version": "1.3.0",
"redshift_model_version": "1.3.1",
"bigquery_model_version": "1.0.3",
"snowflake_model_version": "1.0.1"
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
"meta": {
"versions": {
"test_suite_version": "1.1.1",
"redshift_model_version": "1.3.0",
"redshift_model_version": "1.3.1",
"bigquery_model_version": "1.0.3",
"snowflake_model_version": "1.0.1"
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -225,7 +225,7 @@
"meta": {
"versions": {
"test_suite_version": "1.1.1",
"redshift_model_version": "1.3.0",
"redshift_model_version": "1.3.1",
"bigquery_model_version": "1.0.3",
"snowflake_model_version": "1.0.1"
},
Expand Down
2 changes: 1 addition & 1 deletion .test/great_expectations/expectations/web/v1/sessions.json
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@
"meta": {
"versions": {
"test_suite_version": "1.1.1",
"redshift_model_version": "1.3.0",
"redshift_model_version": "1.3.1",
"bigquery_model_version": "1.0.3",
"snowflake_model_version": "1.0.1"
},
Expand Down
2 changes: 1 addition & 1 deletion .test/great_expectations/expectations/web/v1/users.json
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@
"meta": {
"versions": {
"test_suite_version": "1.1.1",
"redshift_model_version": "1.3.0",
"redshift_model_version": "1.3.1",
"bigquery_model_version": "1.0.3",
"snowflake_model_version": "1.0.1"
},
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
Redshift Web Version 1.3.1 (2022-09-02)
--------------------------------------
Redshift Web: load_tstamp missing from table definition (#135)

Maintenance release (2022-07-21)
--------------------------------
Add migrations for previous releases (#133)
Expand Down
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ The Snowplow atomic data acts as an immutable log of all the actions that occurr

# Try Snowplow

This repo contains data models which are relevant to users who already have a full Snowplow pipeline running (which can be done Open Source or via [Snowplow BDP](https://snowplowanalytics.com/snowplow-bdp/)).
This repo contains data models which are relevant to users who already have a full Snowplow pipeline running (which can be done Open Source or via [Snowplow BDP](https://snowplow.io/snowplow-bdp/)).

If you don't have a pipeline yet, you might be interested in finding out what Snowplow can do by setting up [Try Snowplow](https://try.snowplowanalytics.com/?utm_source=github&utm_medium=post&utm_campaign=try-snowplow).

Expand Down Expand Up @@ -66,7 +66,7 @@ For detail on using the helper scripts, see the README in `.scripts/`

### Snowplow BDP

Snowplow BDP customers can configure jobs for SQL-runner in production via configuration files. [See our docs site for details on doing so](https://docs.snowplowanalytics.com/docs/modeling-your-data/configuring-and-running-data-models-via-snowplow-bdp/). The `configs/datamodeling.json` file in each model is an example configuration for the standard model. The `configs/example_with_custom.json` file is an example configuration with a customization.
Snowplow BDP customers can configure jobs for SQL-runner in production via configuration files. [See our docs site for details on doing so](https://docs.snowplow.io/docs/modeling-your-data/configuring-and-running-data-models-via-snowplow-bdp/). The `configs/datamodeling.json` file in each model is an example configuration for the standard model. The `configs/example_with_custom.json` file is an example configuration with a customization.

### Open Source

Expand Down Expand Up @@ -103,12 +103,12 @@ limitations under the License.

[license]: http://www.apache.org/licenses/LICENSE-2.0
[license-image]: http://img.shields.io/badge/license-Apache--2-blue.svg?style=flat
[tracker-classificiation]: https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/tracker-maintenance-classification/
[tracker-classificiation]: https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/tracker-maintenance-classification/
[actively-maintained]: https://img.shields.io/static/v1?style=flat&label=Snowplow&message=Actively%20Maintained&color=6638b8&labelColor=9ba0aa&logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAMAAAAoLQ9TAAAAeFBMVEVMaXGXANeYANeXANZbAJmXANeUANSQAM+XANeMAMpaAJhZAJeZANiXANaXANaOAM2WANVnAKWXANZ9ALtmAKVaAJmXANZaAJlXAJZdAJxaAJlZAJdbAJlbAJmQAM+UANKZANhhAJ+EAL+BAL9oAKZnAKVjAKF1ALNBd8J1AAAAKHRSTlMAa1hWXyteBTQJIEwRgUh2JjJon21wcBgNfmc+JlOBQjwezWF2l5dXzkW3/wAAAHpJREFUeNokhQOCA1EAxTL85hi7dXv/E5YPCYBq5DeN4pcqV1XbtW/xTVMIMAZE0cBHEaZhBmIQwCFofeprPUHqjmD/+7peztd62dWQRkvrQayXkn01f/gWp2CrxfjY7rcZ5V7DEMDQgmEozFpZqLUYDsNwOqbnMLwPAJEwCopZxKttAAAAAElFTkSuQmCC

[tracker-docs]: https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/
[docs-what-is-dm]: https://docs.snowplowanalytics.com/docs/modeling-your-data/what-is-data-modeling/
[docs-data-models]: https://docs.snowplowanalytics.com/docs/modeling-your-data/
[tracker-docs]: https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/
[docs-what-is-dm]: https://docs.snowplow.io/docs/modeling-your-data/what-is-data-modeling/
[docs-data-models]: https://docs.snowplow.io/docs/modeling-your-data/

[sql-runner]: https://github.com/snowplow/sql-runner
[sql-runner-github]: https://github.com/snowplow/sql-runner/releases/
8 changes: 4 additions & 4 deletions mobile/v1/bigquery/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ To customise the model, we recommend following the guidance found in the README

### Prerequisites

[SQL-runner](https://github.com/snowplow/sql-runner) must be installed, and a dataset of mobile events from either the Snowplow [iOS tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/objective-c-tracker/) or [Android tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/android-tracker/) must be available in the database. The session context and screen view events most both be enabled for the mobile model to run.
[SQL-runner](https://github.com/snowplow/sql-runner) must be installed, and a dataset of mobile events from either the Snowplow [iOS tracker](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/objective-c-tracker/) or [Android tracker](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/android-tracker/) must be available in the database. The session context and screen view events most both be enabled for the mobile model to run.

### Configuration

Expand Down Expand Up @@ -120,7 +120,7 @@ The `{{.scratch_schema}}.mobile_events_this_run` table contains all events relev

If there is a requirement that a custom module consumes data _more frequently than the screen views module for example_, the `{{.scratch_schema}}.mobile_events_this_run` table may be used for this purpose.

The `{{.scratch_schema}}.mobile_events_staged` table is incrementally updated to contain all events relevant to any run of the base module _since the last time the sessions module consumed it_ (ie since the last time the `99-sessions-complete.yml.tmpl` has run). This allows one to run the base module more frequently than the subsequent modules (if, for example, a custom module reads from events_this_run).
The `{{.scratch_schema}}.mobile_events_staged` table is incrementally updated to contain all events relevant to any run of the base module _since the last time the sessions module consumed it_ (ie since the last time the `99-sessions-complete.yml.tmpl` has run). This allows one to run the base module more frequently than the subsequent modules (if, for example, a custom module reads from events_this_run).

Detail on configuring the base module's playbook can be found [in the relevant playbook directory's README](sql-runner/playbooks/standard/01-base).

Expand Down Expand Up @@ -214,15 +214,15 @@ Detail on configuring the users module's playbook can be found [in the relevant

While the model is configured by default to run the entire way through, i.e. from the base module through to the users module, it is possible to run each module independently. For instance one could run the screen views module hourly while only running the sessions module daily. To do so you should run hourly all modules up to and including the desired module i.e. the base and screen view modules. The sessions module can then be run on a daily schedule. A few points to note:

- It is only when the sessions module is run that the `{{.scratch_schema}}.mobile_events_staged` is truncated. As a result, the hourly runs of the screen views module will both process new events data as well as re-process data stored in `mobile_events_staged` since the last time the sessions module ran.
- It is only when the sessions module is run that the `{{.scratch_schema}}.mobile_events_staged` is truncated. As a result, the hourly runs of the screen views module will both process new events data as well as re-process data stored in `mobile_events_staged` since the last time the sessions module ran.
- Prior to running sessions module ensure that all input modules have been run i.e. base, screen views and _any enabled optional modules_. This ensures all the inputs are up to date and in-sync.

### Incomplete Runs

It is not a requirement to run every module. For example you may decide you do not need sessions or users data and only want screen view data. To do so:

- Set `stage_next` to `False` and `:ends_run:` to true in the screen views module. See the [README](sql-runner/playbooks/02-screen-views) for more details.
- Run all modules up to and including the screen views module.
- Run all modules up to and including the screen views module.
- Ensure that the sessions 'complete' playbook, `99-sessions-complete.yml.tmpl`, is the last step in the run. This playbook includes the truncation of the `mobile_events_staged` table. Without this truncation _each subsequent run will re-process data severely impacting performance._

## A note on duplicates
Expand Down
8 changes: 4 additions & 4 deletions mobile/v1/redshift/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ To customise the model, we recommend following the guidance found in the README

### Prerequisites

[SQL-runner](https://github.com/snowplow/sql-runner) must be installed, and a dataset of mobile events from either the Snowplow [iOS tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/objective-c-tracker/) or [Android tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/android-tracker/) must be available in the database. The session context and screen view events most both be enabled for the mobile model to run.
[SQL-runner](https://github.com/snowplow/sql-runner) must be installed, and a dataset of mobile events from either the Snowplow [iOS tracker](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/objective-c-tracker/) or [Android tracker](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/android-tracker/) must be available in the database. The session context and screen view events most both be enabled for the mobile model to run.

### Configuration

Expand Down Expand Up @@ -122,7 +122,7 @@ The `{{.scratch_schema}}.mobile_events_this_run` table contains all events relev

If there is a requirement that a custom module consumes data _more frequently than the screen views module for example_, the `{{.scratch_schema}}.mobile_events_this_run` table may be used for this purpose.

The `{{.scratch_schema}}.mobile_events_staged` table is incrementally updated to contain all events relevant to any run of the base module _since the last time the sessions module consumed it_ (ie since the last time the `99-sessions-complete.yml.tmpl` has run). This allows one to run the base module more frequently than the subsequent modules (if, for example, a custom module reads from events_this_run).
The `{{.scratch_schema}}.mobile_events_staged` table is incrementally updated to contain all events relevant to any run of the base module _since the last time the sessions module consumed it_ (ie since the last time the `99-sessions-complete.yml.tmpl` has run). This allows one to run the base module more frequently than the subsequent modules (if, for example, a custom module reads from events_this_run).

Detail on configuring the base module's playbook can be found [in the relevant playbook directory's README](sql-runner/playbooks/standard/01-base).

Expand Down Expand Up @@ -216,15 +216,15 @@ Detail on configuring the users module's playbook can be found [in the relevant

While the model is configured by default to run the entire way through, i.e. from the base module through to the users module, it is possible to run each module independently. For instance one could run the screen views module hourly while only running the sessions module daily. To do so you should run hourly all modules up to and including the desired module i.e. the base and screen view modules. The sessions module can then be run on a daily schedule. A few points to note:

- It is only when the sessions module is run that the `{{.scratch_schema}}.mobile_events_staged` is truncated. As a result, the hourly runs of the screen views module will both process new events data as well as re-process data stored in `mobile_events_staged` since the last time the sessions module ran.
- It is only when the sessions module is run that the `{{.scratch_schema}}.mobile_events_staged` is truncated. As a result, the hourly runs of the screen views module will both process new events data as well as re-process data stored in `mobile_events_staged` since the last time the sessions module ran.
- Prior to running sessions module ensure that all input modules have been run i.e. base, screen views and _any enabled optional modules_. This ensures all the inputs are up to date and in-sync.

### Incomplete Runs

It is not a requirement to run every module. For example you may decide you do not need sessions or users data and only want screen view data. To do so:

- Set `stage_next` to `False` and `:ends_run:` to true in the screen views module. See the [README](sql-runner/playbooks/02-screen-views) for more details.
- Run all modules up to and including the screen views module.
- Run all modules up to and including the screen views module.
- Ensure that the sessions 'complete' playbook, `99-sessions-complete.yml.tmpl`, is the last step in the run. This playbook includes the truncation of the `mobile_events_staged` table. Without this truncation _each subsequent run will re-process data severely impacting performance._

## A note on duplicates
Expand Down
8 changes: 4 additions & 4 deletions mobile/v1/snowflake/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ To customise the model, we recommend following the guidance found in the README

### Prerequisites

[SQL-runner 0.9.3](https://github.com/snowplow/sql-runner) or later must be installed, and a dataset of mobile events from either the Snowplow [iOS tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/objective-c-tracker/) or [Android tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/android-tracker/) must be available in the database. The session context and screen view events most both be enabled for the mobile model to run.
[SQL-runner 0.9.3](https://github.com/snowplow/sql-runner) or later must be installed, and a dataset of mobile events from either the Snowplow [iOS tracker](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/objective-c-tracker/) or [Android tracker](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/android-tracker/) must be available in the database. The session context and screen view events most both be enabled for the mobile model to run.

### Configuration

Expand Down Expand Up @@ -120,7 +120,7 @@ The `{{.scratch_schema}}.mobile_events_this_run` table contains all events relev

If there is a requirement that a custom module consumes data _more frequently than the screen views module for example_, the `{{.scratch_schema}}.mobile_events_this_run` table may be used for this purpose.

The `{{.scratch_schema}}.mobile_events_staged` table is incrementally updated to contain all events relevant to any run of the base module _since the last time the sessions module consumed it_ (ie since the last time the `99-sessions-complete.yml.tmpl` has run). This allows one to run the base module more frequently than the subsequent modules (if, for example, a custom module reads from events_this_run).
The `{{.scratch_schema}}.mobile_events_staged` table is incrementally updated to contain all events relevant to any run of the base module _since the last time the sessions module consumed it_ (ie since the last time the `99-sessions-complete.yml.tmpl` has run). This allows one to run the base module more frequently than the subsequent modules (if, for example, a custom module reads from events_this_run).

Detail on configuring the base module's playbook can be found [in the relevant playbook directory's README](sql-runner/playbooks/standard/01-base).

Expand Down Expand Up @@ -214,15 +214,15 @@ Detail on configuring the users module's playbook can be found [in the relevant

While the model is configured by default to run the entire way through, i.e. from the base module through to the users module, it is possible to run each module independently. For instance one could run the screen views module hourly while only running the sessions module daily. To do so you should run hourly all modules up to and including the desired module i.e. the base and screen view modules. The sessions module can then be run on a daily schedule. A few points to note:

- It is only when the sessions module is run that the `{{.scratch_schema}}.mobile_events_staged` is truncated. As a result, the hourly runs of the screen views module will both process new events data as well as re-process data stored in `mobile_events_staged` since the last time the sessions module ran.
- It is only when the sessions module is run that the `{{.scratch_schema}}.mobile_events_staged` is truncated. As a result, the hourly runs of the screen views module will both process new events data as well as re-process data stored in `mobile_events_staged` since the last time the sessions module ran.
- Prior to running sessions module ensure that all input modules have been run i.e. base, screen views and _any enabled optional modules_. This ensures all the inputs are up to date and in-sync.

### Incomplete Runs

It is not a requirement to run every module. For example you may decide you do not need sessions or users data and only want screen view data. To do so:

- Set `stage_next` to `False` and `:ends_run:` to true in the screen views module. See the [README](sql-runner/playbooks/02-screen-views) for more details.
- Run all modules up to and including the screen views module.
- Run all modules up to and including the screen views module.
- Ensure that the sessions 'complete' playbook, `99-sessions-complete.yml.tmpl`, is the last step in the run. This playbook includes the truncation of the `mobile_events_staged` table. Without this truncation _each subsequent run will re-process data severely impacting performance._

## A note on duplicates
Expand Down
2 changes: 1 addition & 1 deletion web/v1/bigquery/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ To customise the model, we recommend following the guidance found in the README

### Prerequisites

[SQL-runner](https://github.com/snowplow/sql-runner) must be installed, and a dataset of web events from the [Snowplow Javascript tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-tracker/) must be available in the database.
[SQL-runner](https://github.com/snowplow/sql-runner) must be installed, and a dataset of web events from the [Snowplow Javascript tracker](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/) must be available in the database.

### Configuration

Expand Down
4 changes: 4 additions & 0 deletions web/v1/redshift/CHANGELOG
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
Version 1.3.1 (2022-09-02)
--------------------------
Redshift Web: load_tstamp missing from table definition (#135)

Version 1.3.0 (2022-06-07)
--------------------------
Redshift web: Change SORTKEY encoding to RAW (#129) (thanks @mark-walle!)
Expand Down
Loading

0 comments on commit 7b2c528

Please sign in to comment.