Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add configuration option to support reading changes from Beginning of Time #450

Open
ryankingston opened this issue Feb 17, 2022 · 2 comments
Labels
feature-request New feature or request

Comments

@ryankingston
Copy link

When configuring my Cosmos DB Source connector it seems that the connector is processing changes from the last time it was monitored by a source task. I would like process these changes from the beginning of time in my Cosmos DB instance, but I can't seem to find a solution for this. During configuration it seems the only two options for the "Use latest offset" attribute is processing from the last recorded offset or process from the last time it was monitored by a source task, nothing around beginning of time. Any suggestions here?

Note: Using Conduktor as the Kafka Platform.

@ryancrawcour
Copy link
Contributor

ryancrawcour commented Feb 17, 2022

correct, this doesn't appear to be supported today.
we could add startFromBeginning config option or something similar to how Functions handles this in the Cosmos DB trigger.

for now, you could look in to deleting the lease collection in Cosmos DB and letting the connector recreate a new leases collection. I believe this will have the same effect as telling the connector to "start from the beginning"

@ryancrawcour ryancrawcour added the feature-request New feature or request label Feb 17, 2022
@ryancrawcour ryancrawcour changed the title Resetting Cosmos DB Source Connector Offsets to Beginning of Time Add configuration option to support reading changes from Beginning of Time Feb 17, 2022
@ryancrawcour
Copy link
Contributor

ryancrawcour commented Feb 17, 2022

another option would be to add a withStartTime configuration option that would allow a user to set a specific start date and time. to read from beginning of the life of the container you would set the config value to the min valid Java date & time.

similar to https://docs.microsoft.com/en-us/azure/cosmos-db/sql/change-feed-processor#reading-from-the-beginning

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants