tencent cloud

Message Filtering
Download PDF
Last updated: 2025-07-15 17:46:17
Message Filtering
Last updated: 2025-07-15 17:46:17
Download PDF
This document describes the features, application scenarios, and usage instructions of message filtering in TDMQ for RocketMQ.

Feature Description

Message filtering indicates that messages are filtered by the message attribute configured by the message producer when the producer sends messages to the topic. The consumer that subscribes to the topic can filter messages based on their attributes so that only eligible messages are delivered to the consumer for consumption.
If a consumer configures no filter conditions when subscribing to a topic, no matter whether filter attributes are configured during message sending, all messages in the topic will be delivered to the consumer for consumption.

Use Cases

Generally, messages with the same business attributes are stored in the same topic. For example, when an order transaction topic contains messages of order placements, payments, and deliveries. These messages are sent to the same transaction statement topic. To obtain a certain type of message, users can filter it on the client.
Billing system: just subscribe to payment messages.
Logistics system: just subscribe to logistics messages.
Inventory system: just subscribe to order messages.


Instructions

Currently, message filtering mainly supports two types of filtering: SQL filtering and TAG filtering. The core logic involves setting custom fields when sending messages, then specifying the corresponding filter expression when subscribing via a consumption group. Messages are filtered on the server side before being consumed by the group.
Filtering by Tag
Filtering by SQL

Sending Messages

Note:
During message sending, tags must be clearly specified for each message.
 String tag = "yourMessageTagA";
        final Message message = provider.newMessageBuilder()
            // Set topic for the current message.
            .setTopic(topic)
            // Message secondary classifier of message besides topic.
            .setTag(tag)
            // Key(s) of the message, another way to mark message besides message id.
            .setKeys("yourMessageKey-1c151062f96e")
            .setBody(body)
            .build();

Subscribing to Messages

Subscribing to all tags:
If a consumer wants to subscribe to all types of messages under a topic, an asterisk (*) can be used to represent all tags.
  String consumerGroup = "yourConsumerGroup";
        String topic = "yourTopic";
        String tag = "*";
        FilterExpression filterExpression = new FilterExpression(tag, FilterExpressionType.TAG);
        // In most case, you don't need to create too many consumers, singleton pattern is recommended.
        PushConsumer pushConsumer = provider.newPushConsumerBuilder()
            .setClientConfiguration(clientConfiguration)
            // Set the consumer group name.
            .setConsumerGroup(consumerGroup)
            // Set the subscription for the consumer.
            .setSubscriptionExpressions(Collections.singletonMap(topic, filterExpression))
            .setMessageListener(messageView -> {
                // Handle the received message and return consume result.
                log.info("Consume message={}", messageView);
                return ConsumeResult.SUCCESS;
            })
            .build();
Subscribing to one tag:
If a consumer wants to subscribe to a certain type of messages under a topic, the tag should be specified clearly.
  String consumerGroup = "yourConsumerGroup";
        String topic = "yourTopic";
        String tag = "TAGA";
        FilterExpression filterExpression = new FilterExpression(tag, FilterExpressionType.TAG);
        // In most case, you don't need to create too many consumers, singleton pattern is recommended.
        PushConsumer pushConsumer = provider.newPushConsumerBuilder()
            .setClientConfiguration(clientConfiguration)
            // Set the consumer group name.
            .setConsumerGroup(consumerGroup)
            // Set the subscription for the consumer.
            .setSubscriptionExpressions(Collections.singletonMap(topic, filterExpression))
            .setMessageListener(messageView -> {
                // Handle the received message and return consume result.
                log.info("Consume message={}", messageView);
                return ConsumeResult.SUCCESS;
            })
            .build();
Subscribing to multiple tags:
If a consumer wants to subscribe to multiple types of messages under a topic, two vertical bars (||) should be added between the two tags for separation.
 String consumerGroup = "yourConsumerGroup";
        String topic = "yourTopic";
        String tag = "TAGA || TAGB";
        FilterExpression filterExpression = new FilterExpression(tag, FilterExpressionType.TAG);
        // In most case, you don't need to create too many consumers, singleton pattern is recommended.
        PushConsumer pushConsumer = provider.newPushConsumerBuilder()
            .setClientConfiguration(clientConfiguration)
            // Set the consumer group name.
            .setConsumerGroup(consumerGroup)
            // Set the subscription for the consumer.
            .setSubscriptionExpressions(Collections.singletonMap(topic, filterExpression))
            .setMessageListener(messageView -> {
                // Handle the received message and return consume result.
                log.info("Consume message={}", messageView);
                return ConsumeResult.SUCCESS;
            })
            .build();

Sending Messages

The message sending code here is basically the same as the code for sending simple messages. A message is allowed to carry multiple user-defined attributes when you construct the message body.
final Message message = provider.newMessageBuilder()
            // Set topic for the current message.
            .setTopic(topic)
            // Message secondary classifier of message besides topic.
            // Key(s) of the message, another way to mark message besides message id.
            .setKeys("yourMessageKey-1c151062f96e")
            .setBody(body)
            // Some information for SQL filtering
            .addProperty("key1", "value1")
            .build();

Subscribing to Messages

The message consumption code here is basically the same as the code for consuming simple messages. However, a message needs to be carried with the corresponding SQL expression when being subscribed to.
  String consumerGroup = "yourConsumerGroup";
        String topic = "yourTopic";
        String sql = "key1 IS NOT NULL AND key1='value1'";
        // SQL expression
        FilterExpression filterExpression = new FilterExpression(sql, FilterExpressionType.SQL92);
        // If all is subscribed to
        //FilterExpression filterExpression = FilterExpression.SUB_ALL;
        // In most case, you don't need to create too many consumers, singleton pattern is recommended.
        PushConsumer pushConsumer = provider.newPushConsumerBuilder()
            .setClientConfiguration(clientConfiguration)
            // Set the consumer group name.
            .setConsumerGroup(consumerGroup)
            // Set the subscription for the consumer.
            .setSubscriptionExpressions(Collections.singletonMap(topic, filterExpression))
            .setMessageListener(messageView -> {
                // Handle the received message and return consume result.
                log.info("Consume message={}", messageView);
                return ConsumeResult.SUCCESS;
            })
            .build();
Note:
The preceding sections provide introduction to the use instructions of message publishing and subscription. For more operations, see GitHub Demo or RocketMQ Official Documentation.

Use Limits

Since SQL attribute filtering involves producers defining message attributes and consumers setting SQL filter conditions, the computation may yield different results. Therefore, the server's processing method is as follows:
Exception handling: If the expression calculation of the filter condition throws an exception, the message is filtered by default and will not be delivered to consumers. For example, comparing values of numeric and non-numeric types.
Null value handling: If the expression calculation of the filter condition is null or not a boolean type (true and false), the message is filtered by default and will not be delivered to consumers. For example, when sending a message, if a certain attribute does not exist and the filter condition directly uses this attribute during subscription, the expression calculation result will be null.
Type mismatch handling: If the message custom attribute is floating-point but the filter condition uses an integer to judge, the message is filtered by default and will not be delivered to consumers.
Although this approach is flexible, it is not recommended to set too many values in the message header because the total size limit of message header properties is 32K, and the built-in ones already occupy a significant portion. If it becomes excessively long, it may cause message sending or consumption to be abnormal.

Usage recommendations

Divide topics and tags reasonably.
From the message filtering mechanism and the principles behind topics, you can see that business messages can be filtered based on topics or by tags and attributes within a topic. For the selection of split methods, note the following problems.
Message type consistency: Different types of messages, such as sequential messaging and standard messages, require the use of different topics for splitting and cannot be grouped by Tag.
Whether the business domain is the same: Messages from different business domains and departments should be split into different topics. For example, logistics messages and payment messages should use two different topics. For logistics messages within the same topic, ordinary logistics messages and urgent logistics messages can be distinguished by different Tags.
Message scale and severity consistency: If there is a huge difference in message scale or severity, different topics should be used to isolate and split them.


Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback