Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for keying messages by id and partition key #456

Open
3 tasks
cacodev opened this issue Apr 14, 2022 · 1 comment
Open
3 tasks

Support for keying messages by id and partition key #456

cacodev opened this issue Apr 14, 2022 · 1 comment

Comments

@cacodev
Copy link

cacodev commented Apr 14, 2022

## Problem Statement
I want to stream my cosmos data to a Kafka topic. I need the Kafka messages to be keyed with my document id plus the partition key. Having my topic messages keyed this way will allow for other processes to easily join my topic to other topics in my organization.

## Proposed Solution
My ideal solution would be to do this using the connector transforms (SMTs). This post seems to suggest that it is not possible, however, because the message is handed off to the transform as a string, not a JSON structure: #159 (comment)

## Alternative Solutions
An alternate solution would be to alter the message key options to select the Id and Partition key by field name.
image

## Additional context
I was very frustrated to learn that I could not do this with SMTs. I spent a lot of time debugging this and comparing with other connectors in my org. At the very least - can a note or warning be added to the documentation to save someone else the heart burn?

Next Steps

  • Team consensus to proceed
  • Schedule Design Session
  • Complete Design Review
@jmorg98
Copy link

jmorg98 commented May 15, 2023

+1 to this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants