Expired messages are not deleted promptly.
The message deletion mechanism of Kafka may cause a problem where expired messages are not deleted promptly in certain business scenarios. You may feel confused if you are unfamiliar with this mechanism. The specific problem is as follows:
Here, the message timestamps in partition 0 and partition 7 are obviously different, and expired messages in partition 0 are not deleted promptly as shown below:
Kafka data is stored in three dimensions: topic, partition, and data segment. The message data deletion conditions are as follows:
For some reasons, messages are written unevenly and concentrated in a certain partition (such as partition 7), and some other partitions (such as partition 0) have only little data. In this case, the size of the data segment in partition 0 does not reach 1 GB, so no new segments are generated; however, there is data within the retention period in the entire data segment. Therefore, messages in partition 0 will not be deleted.