tencent cloud


Message Idempotency

Last updated: 2024-06-28 11:31:36
    Idempotency is a key to the design of distributed systems. If idempotency is not taken into account, a message may be consumed repeatedly in case of a business processing failure, which doesn't meet your business expectations. To avoid such an exception, the consumer of a message queue needs to ensure the idempotency of the received messages based on the unique business key.

    What Is Message Idempotency


    The consumption of a message is deemed idempotent if consuming the message multiple times yields the same result as consuming it only once and the duplicate consumption of the same message doesn't have any negative impact on the business system.

    Scenario example

    Take a bank's payment system as an example. After the consumer consumes a deduction message, the payment system will deduct an amount (say, 1 USD in this example) for the order. If the consumer consumes that deduction message again due to network instability or other reasons, yet only 1 USD is finally deducted and the user has only one deduction record, then the deduction operation is valid because the amount hasn't been deducted multiple times. In this scenario, the whole consumption process is idempotent.

    Use Cases

    Message duplication caused by message sending

    After a message is sent by a producer and is successfully received and persistently stored by the server, it will be resent if the producer hasn’t received the server's acknowledgment. This happens when a client restart or momentary network disconnection prevents the server and client from communicating. In this case, the same message will be sent twice to the customer, but with different IDs.

    Message duplication caused by message consumption

    After a message is consumed by a consumer for related business, it will be consumed again if the consumer fails to send the acknowledgment to the server due to network exception. In this case, the same message will be consumed twice by the consumer, with the same ID.


    We can see from the above use cases that there are two message duplication scenarios:
    The same message but with different IDs;
    The same message with the same ID.
    Therefore, we recommend that you use the unique business identifier instead of the message ID to implement idempotency. For example, in the payment scenario, you can use the order ID to implement idempotency. After a message is successfully consumed, the order ID of a business can be used to determine whether the business has been processed.

    Sample code

    public static class Order {
    public String orderId;
    public String orderData;
    Producer<Order> producer = client.newProducer(Schema.AVRO(Order.class)).create();
    producer.newMessage().value(new Order("orderid-12345678", "orderData")).send();
    Consumer<Order> consumer = client.newConsumer(Schema.AVRO(Order.class)).subscribe();
    Order order = consumer.receive().getValue();
    String key = order.orderId;
    After obtaining the unique business identifier orderId, deduplicate it.

    Common Deduplication Methods

    Deduplicating data in the database

    You can filter duplicate data by adding a deduplication table or unique index in the database. This allows you to ensure idempotency on the business side.
    For example, if you need to write order flow messages to an order log table, you can use the order ID or the modification timestamp as the unique index.
    When the consumer consumes the same message multiple times, the order log table will be written each time, but only the first time takes effect because of the unique index. This implements idempotency to ensure that the consumption result is the same even in case of repeated consumption.

    Setting the globally unique message/task ID

    The call trace ID can also be set as a globally unique ID. The producer can add a unique ID to each message when producing messages. The consumer can set a key (which corresponds to the unique ID) in the cache to identify the consumed message. In this way, the consumer can determine whether the message has been consumed when consuming messages.
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support