Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bad performance when executing kafka-producer-perf-test.sh #162

Open
lawrence-zhang opened this issue Mar 20, 2024 · 3 comments
Open

bad performance when executing kafka-producer-perf-test.sh #162

lawrence-zhang opened this issue Mar 20, 2024 · 3 comments

Comments

@lawrence-zhang
Copy link

Hi I got bad performance with kafka-proxy.
currently we have 6 kafka brokers on our EKS and its has plaintext sasl. 3 kafka-proxy with SSL. the configuration as below
- server
- --debug-enable
- --log-level=debug
- --bootstrap-server-mapping=osskafka-kafka-0.osskafka-kafka-headless.infra.svc.cluster.local:9091,0.0.0.0:12687,kafka-proxy.hk.xxxx.dev.net:12687
- --bootstrap-server-mapping=osskafka-kafka-1.osskafka-kafka-headless.infra.svc.cluster.local:9091,0.0.0.0:12688,kafka-proxy.hk.xxxx.dev.net:12688
- --bootstrap-server-mapping=osskafka-kafka-2.osskafka-kafka-headless.infra.svc.cluster.local:9091,0.0.0.0:12689,kafka-proxy.hk.xxxx.dev.net:12689
- --bootstrap-server-mapping=osskafka-kafka-3.osskafka-kafka-headless.infra.svc.cluster.local:9091,0.0.0.0:12690,kafka-proxy.hk.xxxx.dev.net:12690
- --bootstrap-server-mapping=osskafka-kafka-4.osskafka-kafka-headless.infra.svc.cluster.local:9091,0.0.0.0:12691,kafka-proxy.hk.xxxx.dev.net:12691
- --bootstrap-server-mapping=osskafka-kafka-5.osskafka-kafka-headless.infra.svc.cluster.local:9091,0.0.0.0:12692,kafka-proxy.hk.xxxx.dev.net:12692
- --proxy-request-buffer-size=327680000
- --proxy-response-buffer-size=327680000
- --proxy-listener-read-buffer-size=327680000
- --proxy-listener-write-buffer-size=1310720000
- --kafka-connection-read-buffer-size=1310720000
- --kafka-connection-write-buffer-size=327680000
- --dynamic-listeners-disable
- --proxy-listener-tls-enable
- --proxy-listener-ca-chain-cert-file=/etc/kafka-proxy/secrets/ca.crt
- --proxy-listener-cert-file=/etc/kafka-proxy/secrets/server.crt
- --proxy-listener-key-file=/etc/kafka-proxy/secrets/private.key
- --proxy-listener-key-password=XXXX
- --kafka-max-open-requests=50000
- --producer-acks-0-disabled
- --kafka-keep-alive=30m

we have expose endpoint with kafka-proxy.hk.xxxx.dev.net:12692
now we can use kafka-topics, kafka-console-producer and kafka-console-consumer normally. but when we executing performance testing, it is bad performance from pod inside.

org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120001 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120001 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120001 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120001 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120001 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120001 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120001 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120001 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation
2128 records sent, 17.622459 records/sec (0.03 MB/sec), 161.69 ms avg latency, 463.00 ms max latency, 163 ms 50th, 195 ms 95th, 205 ms 99th, 208 ms 99.9th.
@cobolbaby
Copy link

Where is the bottleneck?

@lawrence-zhang
Copy link
Author

if testing using kafka broker endpoints, performance is good. I don't know how to adjust configure to enhance performance

@lawrence-zhang
Copy link
Author

I can't figure out the bottleneck.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lawrence-zhang @cobolbaby