tencent cloud

Message Retry and Dead Letter Mechanisms
Download PDF
Last updated:2025-12-24 16:44:29
Message Retry and Dead Letter Mechanisms
Last updated: 2025-12-24 16:44:29
Download PDF
In message scenarios, messages may fail to be sent, be backlogged, or not be consumed properly. To address these scenarios, TDMQ for Apache Pulsar provides the message retry and dead letter mechanisms.

Automatic Retries

The automatic retry topic is designed to ensure that messages are consumed normally. If no normal response is received after a message is consumed by a consumer for the first time, it enters the retry topic. When the number of retries reaches a threshold, no retry is performed, and the message is delivered to the dead letter topic.
When messages enter the dead letter queue, it indicates that TDMQ for Apache Pulsar no longer automatically processes the messages. At this point, manual intervention is generally required to handle these messages. You can write a dedicated client to subscribe to the dead letter topic to process such messages.

Relevant Concepts

Retry topic: A retry topic corresponds to a subscription name (the unique identifier of a subscriber group), and it exists in the form of a topic in TDMQ for Apache Pulsar. When you create a subscription in the console and enable Automatically Create Retry & Dead Letter Queues, the system automatically creates a retry topic. This topic autonomously implements the message retry mechanism.
The topic is named as follows:
2.9.2 version cluster: [Topic name]-[Subscription name]-RETRY
2.7.2 version cluster: [Subscription name]-RETRY
2.6.1 Version Cluster: [Subscription name]-retry

How It Works

When a created consumer subscribes to a topic using a specific subscription name in Shared mode and the enableRetry property is enabled, the retry queue corresponding to the subscription name is automatically subscribed to.
When consumption fails and the consumer.reconsumeLater API is called, the client internally checks the number of message retries. If the number of message retries reaches the maximum limit, the message is delivered to the dead letter queue. (Messages delivered to the dead letter queue are not automatically consumed. If required, users create additional consumers for consumption.) If the number of message retries does not reach the maximum limit, the message is delivered to the retry queue. The retry interval is implemented through delayed messages. Actually, a delayed message is delivered to the retry queue, and the delay time is specified by the user in reconsumeLater.
Note
Only the Shared mode (including the Key-Shared mode) supports the automatic retry and dead letter mechanisms.
If the subscription mode is Exclusive or Failover, the specified retry interval does not take effect, and retries occur immediately. This is because the retry interval is implemented through the delayed message feature, which is not supported in Exclusive or Failover mode.
Note that the client version should be consistent with the cluster version. In this way, the client can accurately identify the automatically created retry and dead letter queues.
When a token is used to access the retry/dead letter queue, you need to grant the message production permission to the role used by the consumer.
A Java client is used as an example here. A subscription sub1 is created for topic1, and the client subscribes to topic1 using the subscription name sub1 and enables enableRetry, as shown in the following figure:
Consumer consumer = client.newConsumer()
    .topic("persistent://1******30/my-ns/topic1")
    .subscriptionType(SubscriptionType.Shared)//Only the Shared mode supports the retry and dead letter mechanisms.
    .enableRetry(true)
    .subscriptionName("sub1")
    .subscribe();
Subscription to sub1 by topic1 forms a delivery mode with a retry mechanism. sub1 automatically subscribes to the retry topic automatically created during initial subscription creation (available in the topic list of the console). When no acknowledgment from the consumer is received for a message in topic1 for the first time, the message is automatically delivered to the retry topic. Since the consumer automatically subscribes to this topic, the message will be re-consumed based on specific retry rules. If consumption fails after the maximum number of retries is reached, the message is delivered to the corresponding dead letter queue for manual handling.
Note
If the subscription is automatically created by the client, you can log in to the console and choose Topic > More > View Subscriptions to go to the Consumption Management page and manually recreate the retry and dead letter queues. 

Custom Parameter Settings

The retry and dead letter parameters are configured for TDMQ for Apache Pulsar by default. The details are as follows:
2.9.2 Version Cluster
2.7.2 Version Cluster
2.6.1 Version Cluster
The number of retries is set to 16. (After 16 retries, a message is delivered to the dead letter queue at the 17th retry.)
The retry queue is specified as [Topic name]-[Subscription name]-RETRY.
The dead letter queue is specified as [Topic name]-[Subscription name]-DLQ.
The number of retries is set to 16. (After 16 retries, a message is delivered to the dead letter queue at the 17th retry.)
The retry queue is specified as [Subscription name]-RETRY.
The dead letter queue is specified as [Subscription name]-DLQ.
The number of retries is set to 16. (After 16 retries, a message is delivered to the dead letter queue at the 17th retry.)
The retry queue is specified as [Subscription name]-retry.
The dead letter queue is specified as [Subscription name]-dlq.
To customize these parameters, use the deadLetterPolicy API. The code is as follows:
Consumer<byte[]> consumer = pulsarClient.newConsumer()
    .topic("persistent://pulsar-****")
    .subscriptionName("sub1")
    .subscriptionType(SubscriptionType.Shared)
    .enableRetry(true)//Enable retry consumption.
    .deadLetterPolicy(DeadLetterPolicy.builder()
          .maxRedeliverCount(maxRedeliveryCount)//Specify the maximum number of retries.
          .retryLetterTopic("persistent://my-property/my-ns/sub1-retry")//Specify the retry queue.
          .deadLetterTopic("persistent://my-property/my-ns/sub1-dlq")//Specify the dead letter queue.
          .build())
    .subscribe();

Retry Rules

Retry rules are implemented using the reconsumeLater API. Three modes are available.
 //Specify any delay time.
 consumer.reconsumeLater(msg, 1000L, TimeUnit.MILLISECONDS);
 //Specify the delay level.
 consumer.reconsumeLater(msg, 1);
 //Use level-based incremental increase.
 consumer.reconsumeLater(msg);
First mode (specifying any delay time): The second parameter specifies the delay time, and the third parameter specifies the time unit. The value ranges of delay time and delayed messages are the same, ranging from 1 to 864,000 seconds.
Second mode (specifying any delay level, only available to existing users of the Tencent Cloud SDK): The implementation effect is basically the same as that of the first mode, and it is more convenient to manage the delay duration in a distributed system. The delay level is explained as follows:
1.1 The second parameter in reconsumeLater(msg, 1) is the message level.
1.2 By default, MESSAGE_DELAYLEVEL = "1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h". This constant determines the delay time of each level. For example, level 1 corresponds to 1s, and level 3 corresponds to 10s. If the default value does not meet actual business requirements, users can customize it.
Third mode (level-based increment increase, only available to existing users of the Tencent Cloud SDK): Unlike the previous two modes, this mode implements a backoff retry mechanism. After the first failure, the retry interval is 1 second. After the second failure, it becomes 5 seconds. The interval increases progressively with each attempt. The specific interval is determined by the MESSAGE_DELAYLEVEL constant introduced in the second mode. This retry mechanism is more practical in actual business scenarios. If consumption fails, services are not restored immediately, typically, making this progressive retry method more reasonable.
Note:
If you are using the Pulsar community SDK, the delay level and level-based increment increase modes are not supported.

Message Properties of Retried Messages

A retried message carries the following properties:
{
  REAL_TOPIC="persistent://my-property/my-ns/test, 
  ORIGIN_MESSAGE_ID=314:28:-1, 
  RETRY_TOPIC="persistent://my-property/my-ns/my-subscription-retry, 
  RECONSUMETIMES=16
}
REAL_TOPIC: original topic.
ORIGIN_MESSAGE_ID: ID of the originally produced message.
RETRY_TOPIC: retry topic.
RECONSUMETIMES: number of message retries.

How to Obtain the Number of Retries

msg.getProperties().get("RECONSUMETIMES")
Note:
The number of retries obtained through the msg.getRedeliveryCount() API is the number of retries under the negativeAcknowledge retry mechanism.

Message ID Transfer of Retried Messages

The following figure shows the message ID transfer process. You can analyze related logs based on this rule.
Original consumption: msgid=1:1:0:1
First retry: msgid=2:1:-1
Second retry: msgid=2:2:-1
Third retry: msgid=2:3:-1
.......
16th retry: msgid=2:16:0:1
Written to dead letter queue at the 17th retry: msgid=3:1:-1

Complete Sample Code

The message retry feature needs to be enabled (enableRetry set to true) for the retry topic (-RETRY) in the consumer first. It is disabled by default. Messages are sent to the retry topic only when the reconsumeLater() API is called.
The consumer.reconsumeLater() API needs to be called for the dead letter topic (-DLQ). After reconsumeLater is executed, the original message in the topic is acknowledged and transferred to the retry topic. When the number of retries reaches the upper limit, the message is transferred to the dead letter queue. The PulsarClient automatically subscribes to the retry topic. However, messages in the dead letter queue are not automatically subscribed to and should be manually subscribed to by users.
The following sample code is used to implement the complete message retry mechanism using TDMQ for Apache Pulsar. It is for reference by developers.
Subscription topic
Consumer<byte[]> consumer1 = client.newConsumer()
        .topic("persistent://pulsar-****")
        .subscriptionName("my-subscription")
        .subscriptionType(SubscriptionType.Shared)
        .enableRetry(true)//Enable retry consumption.
        //.deadLetterPolicy(DeadLetterPolicy.builder()
        //         .maxRedeliverCount(maxRedeliveryCount)
        //         .retryLetterTopic("persistent://my-property/my-ns/my-subscription-retry")//Specify the retry queue.
        //         .deadLetterTopic("persistent://my-property/my-ns/my-subscription-dlq")//Specify the dead letter queue.
        //         .build())
        .subscribe();
Consumption execution
while (true) {
      Message msg = consumer.receive();
      try {
            // Do something with the message
            System.out.printf("Message received: %s", new String(msg.getData()));
            // Acknowledge the message so that it can be deleted by the message broker
            consumer.acknowledge(msg);
      } catch (Exception e) {
            // select reconsume policy
            consumer.reconsumeLater(msg, 1000L, TimeUnit.MILLISECONDS);
            //consumer.reconsumeLater(msg, 1);
            //consumer.reconsumeLater(msg);
      }
}

Active Retries

When a client fails to consume a message and wishes to consume it again, the consumer can call the negativeAcknowledge API. The message will be re-obtained after a certain period. The interval for re-obtaining the message can be specified by negativeAckRedeliveryDelay in the consumer.

Implementation Mechanism Overview

The client caches the message IDs of negativeAcknowledge messages and periodically scans the list of negativeAcknowledge messages internally (scanning interval: negativeAckRedeliveryDelay x 1/3). When the time specified by negativeAckRedeliveryDelay reaches, the client informs the server to redeliver the message to retry. After receiving the redelivery request from the client, the server repushes the message to the client.
Note:
1. The actual time to receive the message again may exceed 1/3 of the time specified by negativeAckRedeliveryDelay, which is related to the implementation logic of the client.
2. In this mode, no new message is generated.
The following Java sample code is for an active retry.
Consumer<byte[]> consumer = client.newConsumer()
        .topic("persistent://pulsar-****")
        .subscriptionName("my-subscription")
        .subscriptionType(SubscriptionType.Shared)
        // 1 minute by default.
        .negativeAckRedeliveryDelay(1, TimeUnit.MINUTES)
        .subscribe();


while (true) {
    Message msg = consumer.receive();
    try {
        // Do something with the message
        System.out.printf("Message received: %s", new String(msg.getData()));
        // Acknowledge the message so that it can be deleted by the message broker
        consumer.acknowledge(msg);
    } catch (Exception e) {
        // Message failed to process, redeliver later
        consumer.negativeAcknowledge(msg);
    }
}

Must-Knows

1. Under the negativeAcknowledge retry mechanism, messages to retry remain unacknowledged on the server.
2. Under the negativeAcknowledge retry mechanism, there is no maximum number of retries by default. However, this can be implemented by configuring the maximum number of retries and the dead letter queue. In this mode, when a message has been retried for a specified number of times, it will be delivered to the dead letter queue.
Consumer<byte[]> consumer = client.newConsumer()
        .topic("persistent://pulsar-****")
        .subscriptionName("my-subscription")
        .subscriptionType(SubscriptionType.Shared)
        // 1 minute by default.
        .negativeAckRedeliveryDelay(1, TimeUnit.MINUTES)
        .deadLetterPolicy(DeadLetterPolicy.builder()
             .maxRedeliverCount(5)//Specify the maximum number of retries.
             .deadLetterTopic("persistent://my-property/my-ns/sub1-dlq")//Specify the dead letter queue.
             .build())        
        .subscribe();


while (true) {
    Message msg = consumer.receive();
    try {
        // Do something with the message
        System.out.printf("Message received: %s", new String(msg.getData()));
        // Acknowledge the message so that it can be deleted by the message broker
        consumer.acknowledge(msg);
    } catch (Exception e) {
        // Message failed to process, redeliver later
        consumer.negativeAcknowledge(msg);
    }
}
3. Under the negativeAcknowledge retry mechanism, the number of retries can be obtained from msg.getRedeliveryCount(). Note that if all consumers under a subscription go offline, the number of message retries will be reset to 0. (Typical scenario: When only one consumer exists under a subscription, the number of message retries will be reset to 0 after the consumer is restarted.)
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback