Tools for Tests
The open-source scripts of the Kafka client can be used for Kafka producer and consumer performance testing. Test results are displayed mainly based on the size of messages sent per second (MB/second) and the number of messages sent per second (records/second).
Kafka producer test script: $KAFKA_HOME/bin/kafka-producer-perf-test.sh
Kafka consumer test script: $KAFKA_HOME/bin/kafka-consumer-perf-test.sh
Testing Commands
Note
ckafka vip:vport in the following sample commands should be replaced by the actual IP address and port assigned for your instance.
Production testing command example:
bin/kafka-producer-perf-test.sh
--topic test
--num-records 123
--record-size 1000
--producer-props bootstrap.servers= ckafka vip : port
--throughput 20000
Consumption testing command example:
bin/kafka-consumer-perf-test.sh
--topic test
--new-consumer
--fetch-size 10000
--messages 1000
--broker-list bootstrap.servers=ckafka vip : port
Suggestions for Tests
It is recommended to create three or more partitions to increase throughput. This is because there must be at least three TDMQ for CKafka (CKafka) cluster nodes at the backend. If only one partition is created, it will be distributed in a single broker, which will affect CKafka performance.
As messages in each CKafka partition are ordered, the production performance will be affected if there are too many partitions. It is recommended that the number of partitions should not exceed 6 according to actual load testing.
It is necessary to simulate concurrency with multiple clients to ensure the effect of load testing. It is recommended that you use multiple servers as load testing clients (producers) and start multiple load testing programs on each server to increase concurrency. To avoid high load on testing servers, it is also recommended that you start one producer every second rather than all producers simultaneously.