You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When configuring my Cosmos DB Source connector it seems that the connector is processing changes from the last time it was monitored by a source task. I would like process these changes from the beginning of time in my Cosmos DB instance, but I can't seem to find a solution for this. During configuration it seems the only two options for the "Use latest offset" attribute is processing from the last recorded offset or process from the last time it was monitored by a source task, nothing around beginning of time. Any suggestions here?
Note: Using Conduktor as the Kafka Platform.
The text was updated successfully, but these errors were encountered:
correct, this doesn't appear to be supported today.
we could add startFromBeginning config option or something similar to how Functions handles this in the Cosmos DB trigger.
for now, you could look in to deleting the lease collection in Cosmos DB and letting the connector recreate a new leases collection. I believe this will have the same effect as telling the connector to "start from the beginning"
ryancrawcour
changed the title
Resetting Cosmos DB Source Connector Offsets to Beginning of Time
Add configuration option to support reading changes from Beginning of Time
Feb 17, 2022
another option would be to add a withStartTime configuration option that would allow a user to set a specific start date and time. to read from beginning of the life of the container you would set the config value to the min valid Java date & time.
When configuring my Cosmos DB Source connector it seems that the connector is processing changes from the last time it was monitored by a source task. I would like process these changes from the beginning of time in my Cosmos DB instance, but I can't seem to find a solution for this. During configuration it seems the only two options for the "Use latest offset" attribute is processing from the last recorded offset or process from the last time it was monitored by a source task, nothing around beginning of time. Any suggestions here?
Note: Using Conduktor as the Kafka Platform.
The text was updated successfully, but these errors were encountered: