Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using runtime variable for stream name #376

Open
RINDLU opened this issue Sep 12, 2024 · 0 comments
Open

Using runtime variable for stream name #376

RINDLU opened this issue Sep 12, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@RINDLU
Copy link

RINDLU commented Sep 12, 2024

It would be great if it is possible to make the stream name dynamic.
For example in this case, {YYYY}-{MM}-{DD} is used to define the stream.
See the replication yaml file below:

source: S3
target: SNOWFLAKE

defaults:
mode: truncate

streams:
test/{YYYY}-{MM}-{DD}/SITE_20*.csv:
object: 'testschema.site'
single: true

When running, the variables are not resolved.
This feature will allow to dynamically get the files everyday from the right current folder.

@RINDLU RINDLU added the enhancement New feature or request label Sep 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant