Kafka integration #983
Kafka integration #983
Conversation
# Conflicts: # Cargo.toml
|
Kafka-console-producer: Data: |
| Ok(()) | ||
| } | ||
|
|
||
| pub async fn setup_integration() -> Result<JoinHandle<()>, KafkaError> { |
There was a problem hiding this comment.
It should be fine to drop the handle if I am not wrong, why are we returning it?
server/Cargo.toml
Outdated
| @@ -106,6 +106,8 @@ path-clean = "1.0.1" | |||
| prost = "0.13.3" | |||
| prometheus-parse = "0.2.5" | |||
| sha2 = "0.10.8" | |||
| kafka = "0.10.0" | |||
There was a problem hiding this comment.
Using https://github.com/fede1024/rust-rdkafka instead of kafka-rust could allow the implementation to be async
|
@TomBebb can you please document the code a bit more, this will help us understand what we are going to maintain in the long run 😅 The code here is lacking comments, especially regarding the functions and what they are doing as well as the use of good const(The env variables should be named in const as the deepsource action points out) that describes what they are doing. Thank you in advance! |
| loop { | ||
| while let Some(curr) = stream.next().await { | ||
| let msg = curr.unwrap(); | ||
| ingest_message(&stream_name, msg).await.unwrap(); | ||
| } | ||
| } |
There was a problem hiding this comment.
while let inside a loop? Also, why the unwrap is not after stream.next() and instead on the returned value when ref says to do otherwise?
| @@ -14,4 +14,5 @@ parseable | |||
| parseable_* | |||
| parseable-env-secret | |||
| cache | |||
| .idea | |||
There was a problem hiding this comment.
please remove unrelated change
| Ok(()) | ||
| } | ||
|
|
||
| pub async fn setup_integration() -> Result<JoinHandle<()>, KafkaError> { |
There was a problem hiding this comment.
| pub async fn setup_integration() -> Result<JoinHandle<()>, KafkaError> { | |
| pub async fn setup_integration() -> Result<(), KafkaError> { |
| log::info!("Setup kafka integration for {stream_name}"); | ||
| create_stream_if_not_exists(&stream_name, &StreamType::UserDefined.to_string()).await?; | ||
|
|
||
| let res = task::spawn(async move { |
There was a problem hiding this comment.
| let res = task::spawn(async move { | |
| task::spawn(async move { |
| } | ||
|
|
||
| pub async fn setup_integration() -> Result<JoinHandle<()>, KafkaError> { | ||
| let my_res = if let Ok(stream_name) = env::var("KAFKA_TOPIC") { |
There was a problem hiding this comment.
| let my_res = if let Ok(stream_name) = env::var("KAFKA_TOPIC") { | |
| if let Ok(stream_name) = env::var("KAFKA_TOPIC") { |
| } else { | ||
| task::spawn_blocking(|| {}) | ||
| }; |
There was a problem hiding this comment.
| } else { | |
| task::spawn_blocking(|| {}) | |
| }; | |
| } |
| } else { | ||
| task::spawn_blocking(|| {}) | ||
| }; | ||
| Ok(my_res) |
There was a problem hiding this comment.
| Ok(my_res) | |
| Ok(()) |
/claim #936
Fixes #936
Description
Integrates with Kakfa, using its crate.
This is toggled on/off via the KAFKA_TOPIC environment variable.
This uses a background polling iterator to loop through messages, and ingesting messages to the stream with the name of the KAFKA_TOPIC.
Covers all possible config options via environment variables.
This PR has: