プライバシーポリシー
データプライバシーとセキュリティ契約
Message Trace
フォーカスモード
フォントサイズ
Introduction
The message trace feature provided by TDMQ for MQTT fully connects upstream and downstream business information, helping you better troubleshoot abnormal information and locate issues. It records the entire lifecycle of a message from when it is sent by a producer to the server and then delivered by the server to a consumer. This article will introduce the core concepts of message tracing, the definitions of key parameters, and focus on how to use trace data for observability analysis to quickly locate typical failures such as "messages not received" and "consumption delays".
What Message Trace Is
Definition
The message trace refers to the complete chain of information formed by aggregating detailed data from various relevant nodes throughout the entire process of a message being sent from a producer to the TDMQ for MQTT server and then consumed by a consumer. This includes timestamps, execution results, producer IP address, consumer IP address, and other critical details. If message delivery does not occur as expected, you can query the message trace to rapidly analyze and locate issues, enabling timely business recovery.
Composition
A complete MQTT message trace consists of the following two parts:
Message Production Trace: Recording the process of producers sending messages to the Broker.
Core focus: whether the producer side successfully sent the message, and whether the server successfully received it.
Message Consumption Trace: Recording the process of the Broker delivering messages to all subscribed consumers.
Core focus: which consumers have subscribed to the topic; whether the message was successfully delivered to the client; and whether the client returned a confirmation (ACK).
Message Trace Use Cases
The message queue for MQTT enables querying key data during message delivery processes. By viewing trace information, you can quickly obtain the current status of workflow processing, helping you promptly locate abnormal issues.
The typical application scenarios for message trace are as follows:
Scenario 1: Checking whether a specific message was successfully sent or consumed.
Confirm whether the message was successfully sent by the device.
Confirm whether the server has received and persisted the message.
Confirm whether the consumer is online and included in the subscription list.
Scenario 2: Checking whether the consumption process is functioning properly.
Compare the "server receipt time" with the "delivery time to consumers".
Check whether the QoS interaction process is complete (for example, whether QoS 1 is missing PUBACK).
Scenario 3: Viewing the identity information of producers and consumers to further investigate the cause of the issue.
Check the Client ID and IP address of the actual message sender and receiver to identify issues such as unauthorized connections or Client ID reuse.
Troubleshooting Consumption Exceptions via Message Trace Practice
In actual business operations, the most common issue is "the device sent a message, but the App/server did not receive it." You can follow these steps to diagnose the issue using message trace combined with client events:
Path One: End-to-End Troubleshooting from the Message Perspective
Applicable scenarios: When a specific message (Message ID) or approximate sending time is known, and you need to confirm which devices consumed this message or why it was not consumed.
Entry: Console > Message Query > Message Trace.
View the Message Trace of a Specific Message
Go to the message query page, specify the target cluster, Topic (enter the accurate Topic Name or Topic Filter used during subscription), and time range, then search for the target message:
Select a specific Message ID and click Details to view detailed message trace.
In end-to-end troubleshooting from the message perspective, judgments and investigations can be made based on the following phenomena:
Check "Message Production"
Symptom | Judgment | Troubleshooting/Solution Suggestions |
The message trace cannot be found. | The message failed to reach the server. | 1. Check whether the device network is connected. 2. Check whether the sending exception is caught in the device-side code. 3. Confirm whether the Topic sent by the device exactly matches the queried Topic. |
Request result: Failure | The server refused to receive the message. | 1. Check whether the device has publish permissions for this Topic (ACL). 2. Check whether the message body size exceeds the server-side limit. |
Request result: Success | Production is normal. | Producer normal. Otherwise, issues may lie in the consumption path. |
Check "Message Consumption"
Symptom | Judgment | Troubleshooting/Solution Suggestions |
The target Client-ID cannot be found in the list. | The consumer is malfunctioning, preventing message consumption. | 1. Check the Topic Filter: Can the consumer's subscription rule (wildcards) match the current message's Topic? 2. Check the subscription status: Confirm whether the consumer is in an online subscription state (or has established a persistent session) at the time of message delivery. |
The list contains a Client ID, but the request result: failure. | The push from the cloud failed, resulting in the consumer not receiving the queue messages. | 1. Check whether the consumer device is offline. 2. Check the consumer's network status (weak network environment). |
Last Push Time >> Production Time (Significant time difference) | Consumption backlog/latency | 1. The consumer's processing logic is too slow, blocking the MQTT thread. 2. Frequent client connection and disconnection causes messages to queue up on the server side. 3. If shared subscription is used, check whether the number of consumers in the shared subscription group is reasonable. |
QoS Packet not closed-loop | QoS acknowledgment exception | Message trace details the request result and time for each request Packet during message production and consumption. QoS 0 Normal process: The server pushes PUBLISH. If the request fails, it may be caused by an unstable TCP connection. QoS 1 message Normal process: The server pushes PUBLISH -> the client replies with PUBACK. If there is only PUBLISH but no PUBACK, meaning the client received the message but did not return an acknowledgment to the server. The server will consider the transmission failed and continuously retry, causing message duplication or congestion. QoS 2 message Normal process: PUBLISH -> PUBREC -> PUBREL -> PUBCOMP. When the process is interrupted at any intermediate step, it can usually be determined that the loss of acknowledgment packets is caused by an unstable network link. |
Path Two: Client Perspective Troubleshooting
Applicable scenarios: Focused on specific devices (for example, Device A reports failure to receive commands or unsuccessful data reporting), requiring inspection of all send/receive records for that device.
Entry: Console > Client > Click a specific Client ID > Client Message Trace Tab.
View Client Message Trace
Go to the Client Management page and search for the corresponding Client ID.
Click Details to view the corresponding client's basic information and Session details.
Within the Tab, you can select to view the client message trace.
In this view, the system filters out all traces related to the device, including messages it sends (as a producer) and receives (as a consumer).
Producer (Producer): Represents the record of this device reporting data to the cloud.
Consumer (Consumer): Represents the record of the cloud delivering commands/pushing messages to this device.
Producer Issue
1. Filter or locate records where "Client Type" is Producer to confirm whether the message was successfully delivered to the MQTT instance.
2. Unable to locate eligible "Producer" records. Check whether the parameters in the producer code are correct and whether the network request initiated by the device was successful.
If it is confirmed through the above methods that the message has indeed been stored by the MQTT server-side, you can focus on troubleshooting whether there are any abnormalities in the subsequent message consumption process.
Consumer Issue
If you suspect that the device is not receiving commands, please filter or locate records where "Client Type" is Consumer:
No "Consumer" records in the list.
Conclusion: The cloud has never pushed messages to this device.
Possible reasons: The device may have gone offline (the Broker stops pushing messages if it considers the device offline), or the publisher sent the message to an incorrect Topic (resulting in a mismatch with the device's subscription).
Records exist, but clicking View Details revealed no ACK.
Conclusion: The cloud delivered the message, and the device (potentially) received it, but the device froze and failed to respond with an ACK.
Recommendation: Check whether the device-side business logic is blocking the MQTT client thread.
Appendix: Parameter Descriptions for Message Trace
Message Production Parameters (Uplink)
Parameter | Description | Troubleshooting Key Points |
Request time | The time point at which the publisher client (Producer) sends a request Packet to the server. | Message production start time. |
Request Packet Type | Request Packet Type: PUBLISH: Publish message PUBACK: Publish Acknowledgement, an acknowledgment response to PUBLISH packets at QoS level 1. PUBREC: Publish Received, the first acknowledgment at QoS level 2, indicating that a QoS 2 message is received. PUBREL: Publish Released, the second step at QoS level 2. It is the sender's response to PUBREC, indicating a request to release the message that was preliminarily acknowledged earlier. PUBCOMP: Publish Complete, the final acknowledgment at QoS level 2, indicating that the entire message delivery process for the QoS 2 message is fully completed. | PUBLISH: Sends a message. PUBACK: QoS 1 acknowledgment. |
client ID | The unique identifier of the client that published this message, used to trace the source of the message. | Used to verify the source and exclude "dirty data". |
QoS | The Quality of Service (QoS) level of a message determines the reliability and assurance of message delivery, including 0 (at most once), 1 (at least once), and 2 (exactly once). | The reliability mechanism for message delivery. |
Request result. | Server processing result for this message publish request. | Success: Server received. Failure: Send rejected. |
Message Consumption Parameters (Downlink)
Parameter | Description | Troubleshooting Key Points |
client ID | The unique identifier of the target client to which the server delivers this message. | If the expected ID is not displayed here, it indicates that the device subscription was not successful. |
QoS | The Quality of Service (QoS) level of a message determines the reliability and assurance of message delivery, including 0 (at most once), 1 (at least once), and 2 (exactly once). | The reliability mechanism for message delivery. |
Last Push Time | The time when the server last pushed this message to the client. | If this time is continuously updated, it indicates that the server is retrying the delivery. |
Request result. | The final result of the server delivering this message. | Success: Closed loop. Failure: Expand the details to see which Packet it is stuck at. |
Clicking the right-pointing triangle below the client ID allows you to view the request details of the server pushing this message.
Parameter | Description | Troubleshooting Key Points |
Request time | The time when the server sends the request Packet to the target client (Consumer). | Message Consumption Start Time |
Request Packet Type | Request Packet Type: PUBLISH: Publish message PUBACK: Publish Acknowledgement, an acknowledgment response to PUBLISH packets at QoS level 1. PUBREC: Publish Received, the first acknowledgment at QoS level 2, indicating that a QoS 2 message is received. PUBREL: Publish Released, the second step at QoS level 2. It is the sender's response to PUBREC, indicating a request to release the message that was preliminarily acknowledged earlier. PUBCOMP: Publish Complete, the final acknowledgment at QoS level 2, indicating that the entire message delivery process for the QoS 2 message is fully completed. | By combining the production time, the end-to-end latency can be calculated. |
Request result. | The processing result of the client for this message delivery request. | Focus on observing whether there are ACK-type packets returned. |
フィードバック