- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- Getting Started
- Developer Guide
- Operation Guide
- Best Practices
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Cluster APIs
- Namespace APIs
- Topic APIs
- Message APIs
- CMQ Message APIs
- CMQ Management APIs
- UnbindCmqDeadLetter
- RewindCmqQueue
- ModifyCmqTopicAttribute
- ModifyCmqSubscriptionAttribute
- ModifyCmqQueueAttribute
- DescribeCmqTopics
- DescribeCmqTopicDetail
- DescribeCmqSubscriptionDetail
- DescribeCmqQueues
- DescribeCmqQueueDetail
- DescribeCmqDeadLetterSourceQueues
- DeleteCmqTopic
- DeleteCmqSubscribe
- DeleteCmqQueue
- CreateCmqTopic
- CreateCmqSubscribe
- CreateCmqQueue
- Other APIs
- Environment Role APIs
- TDMQ for RabbitMQ APIs
- RocketMQ APIs
- DescribeRocketMQVipInstanceDetail
- DescribeRocketMQMsg
- ModifyRocketMQTopic
- ModifyRocketMQNamespace
- ModifyRocketMQGroup
- ModifyRocketMQCluster
- DescribeRocketMQTopics
- DescribeRocketMQNamespaces
- DescribeRocketMQGroups
- DescribeRocketMQClusters
- DescribeRocketMQCluster
- DeleteRocketMQTopic
- DeleteRocketMQGroup
- DeleteRocketMQCluster
- CreateRocketMQTopic
- CreateRocketMQNamespace
- CreateRocketMQGroup
- CreateRocketMQCluster
- TDMQ for RocketMQ APIs
- Production and Consumption APIs
- Data Types
- Error Codes
- SDK Documentation
- FAQs
- TDMQ Policy
- Service Level Agreement
- General References
- Contact Us
- Glossary
- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- Getting Started
- Developer Guide
- Operation Guide
- Best Practices
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Cluster APIs
- Namespace APIs
- Topic APIs
- Message APIs
- CMQ Message APIs
- CMQ Management APIs
- UnbindCmqDeadLetter
- RewindCmqQueue
- ModifyCmqTopicAttribute
- ModifyCmqSubscriptionAttribute
- ModifyCmqQueueAttribute
- DescribeCmqTopics
- DescribeCmqTopicDetail
- DescribeCmqSubscriptionDetail
- DescribeCmqQueues
- DescribeCmqQueueDetail
- DescribeCmqDeadLetterSourceQueues
- DeleteCmqTopic
- DeleteCmqSubscribe
- DeleteCmqQueue
- CreateCmqTopic
- CreateCmqSubscribe
- CreateCmqQueue
- Other APIs
- Environment Role APIs
- TDMQ for RabbitMQ APIs
- RocketMQ APIs
- DescribeRocketMQVipInstanceDetail
- DescribeRocketMQMsg
- ModifyRocketMQTopic
- ModifyRocketMQNamespace
- ModifyRocketMQGroup
- ModifyRocketMQCluster
- DescribeRocketMQTopics
- DescribeRocketMQNamespaces
- DescribeRocketMQGroups
- DescribeRocketMQClusters
- DescribeRocketMQCluster
- DeleteRocketMQTopic
- DeleteRocketMQGroup
- DeleteRocketMQCluster
- CreateRocketMQTopic
- CreateRocketMQNamespace
- CreateRocketMQGroup
- CreateRocketMQCluster
- TDMQ for RocketMQ APIs
- Production and Consumption APIs
- Data Types
- Error Codes
- SDK Documentation
- FAQs
- TDMQ Policy
- Service Level Agreement
- General References
- Contact Us
- Glossary
Idempotency is a key to the design of distributed systems. If idempotency is not taken into account, a message may be consumed repeatedly in case of a business processing failure, which doesn't meet your business expectations. To avoid such an exception, the consumer of a message queue needs to ensure the idempotency of the received messages based on the unique business key.
What Is Message Idempotency
Definition
The consumption of a message is deemed idempotent if consuming the message multiple times yields the same result as consuming it only once and the duplicate consumption of the same message doesn't have any negative impact on the business system.
Scenario example
Take a bank's payment system as an example. After the consumer consumes a deduction message, the payment system will deduct an amount (say, 1 USD in this example) for the order. If the consumer consumes that deduction message again due to network instability or other reasons, yet only 1 USD is finally deducted and the user has only one deduction record, then the deduction operation is valid because the amount hasn't been deducted multiple times. In this scenario, the whole consumption process is idempotent.
Use Cases
Message duplication caused by message sending
After a message is sent by a producer and is successfully received and persistently stored by the server, it will be resent if the producer hasn’t received the server's acknowledgment. This happens when a client restart or momentary network disconnection prevents the server and client from communicating. In this case, the same message will be sent twice to the customer, but with different IDs.
Message duplication caused by message consumption
After a message is consumed by a consumer for related business, it will be consumed again if the consumer fails to send the acknowledgment to the server due to network exception. In this case, the same message will be consumed twice by the consumer, with the same ID.
Solutions
We can see from the above use cases that there are two message duplication scenarios:
The same message but with different IDs;
The same message with the same ID.
Therefore, we recommend that you use the unique business identifier instead of the message ID to implement idempotency. For example, in the payment scenario, you can use the order ID to implement idempotency. After a message is successfully consumed, the order ID of a business can be used to determine whether the business has been processed.
Sample code
public static class Order {public String orderId;public String orderData;}
Producer:
Producer<Order> producer = client.newProducer(Schema.AVRO(Order.class)).create();producer.newMessage().value(new Order("orderid-12345678", "orderData")).send();
Consumer:
Consumer<Order> consumer = client.newConsumer(Schema.AVRO(Order.class)).subscribe();Order order = consumer.receive().getValue();String key = order.orderId;
After obtaining the unique business identifier
orderId, deduplicate it.Common Deduplication Methods
Deduplicating data in the database
You can filter duplicate data by adding a deduplication table or unique index in the database. This allows you to ensure idempotency on the business side.
For example, if you need to write order flow messages to an order log table, you can use the order ID or the modification timestamp as the unique index.
When the consumer consumes the same message multiple times, the order log table will be written each time, but only the first time takes effect because of the unique index. This implements idempotency to ensure that the consumption result is the same even in case of repeated consumption.
Setting the globally unique message/task ID
The call trace ID can also be set as a globally unique ID. The producer can add a unique ID to each message when producing messages. The consumer can set a key (which corresponds to the unique ID) in the cache to identify the consumed message. In this way, the consumer can determine whether the message has been consumed when consuming messages.
Yes
No
Was this page helpful?