As the microservices architecture has gradually become popular, accurately locating the complete lifecycle of a request in complex and large-scale systems has increasingly become one of the biggest pain points faced by R&D personnel. Taking the self - test process in R&D as an example, developers usually hope that after initiating an HTTP/RPC test request, they can obtain the context information of the entire test request in a simple way. This usually includes relevant upstream and downstream links, the method stacks of internal requests in each service, and data such as logs printed on the link. The demand for integrated monitoring of indicators - links - logs is urgent.
Tencent Cloud Application Performance Management (APM), as a full - link monitoring platform that itself includes indicators - links, has been integrated with Tencent Cloud Log Service (CLS), realizing integrated monitoring of indicators - links - logs. When viewing link details, associated information can be viewed correspondingly, completing the troubleshooting process from links to logs. Prerequisites
Make sure that the application has been connected to Application Performance Monitoring (APM), and refer to the CLS Getting Started Guide to report business logs to Tencent Cloud Log Service (CLS). Note:
To ensure the reliability of your log data and use the log service more efficiently, it is recommended that you use the CLS - optimized API/Agent to upload structured logs.
Submit via API/SDK. For details, see the contents section in Uploading Logs. Operation Steps
Step 1: Output the TraceID To the Log
To achieve correlation query between link and log, you need to output the TraceID to the log. For applications accessing APM in the following two ways, the TraceID can be automatically injected into the log without modifying the code:
For applications accessing APM in other ways, please refer to the corresponding access documentation, obtain the TraceID in the code, and output the TraceID in the business log.
Step 2: Configure the Association Between the APM Application and the CLS Topic
2. In the left-side menu bar, choose Application Performance Monitoring > System Configuration, and go to the Workspace Configuration page.
3. Select the corresponding business system, and in the Log Association section, click Edit.
4. Configure log correlation information.
Enable Association Logs.
Select the corresponding Log Region, Log Set, and Log Topic, and specify the fields containing TraceID and SpanID.
5. After configuration is complete, click OK.
Step 3: Query the Business Logs Corresponding To the Link
2. In the left-side menu bar, choose Application Performance Monitoring > Distributed Tracing, and go to the Trace Search page.
3. Select the corresponding business system, specify the query conditions, query the link invocation, and click TraceID to enter the Trace details page.
4. In the Trace Logs and Span Logs tabs, you can query the log information corresponding to this trace.
Print TraceID and SpanID to logs
Tencent Cloud Enhanced OpenTelemetry Java Agent
For log components that meet the requirements, you can inject TraceID and SpanID by modifying the pattern in the log configuration file. In the log configuration file, TraceID and SpanID are injected using the %X{trace_id} and %X{span_id} fields respectively.
|
Log4j | 1.2+ |
Log4j2 | 2.7+ |
Logback | 1.0+ |
Log4j
log4j.properties file modification example:
log4j.appender.warn.layout=org.apache.log4j.PatternLayout
log4j.appender.warn.layout.ConversionPattern=trace_id: %X{trace_id} span_id: %X{span_id} - %m%n
Log4j2
Log4j2 supports configuration files in multiple formats, including XML, JSON, YAML, and Properties. Taking the XML format as an example, there are multiple ways to write the Pattern. Please refer to one of the existing patterns:
<PatternLayout pattern="trace_id: %X{trace_id} span_id: %X{span_id} - %m%n"/>
<PatternLayout>
<Pattern>trace_id: %X{trace_id} span_id: %X{span_id} - %m%n</Pattern>
</PatternLayout>
<Layout type="PatternLayout" pattern="trace_id: %X{trace_id} span_id: %X{span_id} - %m%n"/>
Logback
logback.xml file modification example:
<encoder>
<pattern>trace_id: %X{trace_id} span_id: %X{span_id} %msg%n</pattern>
</encoder>
SkyWalking Java Agent
Please obtain the operational guidance for injecting TraceID and SpanID into logs from the SkyWalking official website based on the type of log component:
Tencent Cloud OpenTelemetry-Go Agent
For log components that meet the requirements, TraceID and SpanID can be injected without any additional operations. If the probe detects the presence of an active Span, it will inject TraceID and SpanID into the logs using trace_id and span_id as markers respectively.
|
Zap | go.uber.org/zap/zapcore | 1.20.0+ |
Logrus | github.com/sirupsen/logrus | 1.5.0+ |
Zerolog | github.com/rs/zerolog | 1.10.0+ |
go-kit/log | github.com/go-kit/log | 0.1.0+ |
Go standard library log | log | No limit |
Go standard library slog | log/slog | No limit |
Taking Zap as an example, the generated log content after TraceID and SpanID are injected is as shown below:
{"level":"info","msg":"this is info message","trace_id":"38e6910973e1c0c11351e502329d2d72","span_id":"6f4efff78445ea61"}
OpenTelemetry-Python
For the built-in logging library and log components that make underlying calls to the built-in logging library, you can achieve the injection of TraceID and SpanID by adding the OTEL_PYTHON_LOG_CORRELATION environment variable.
OTEL_PYTHON_LOG_CORRELATION=true
tapm-instrument \\
--traces_exporter otlp \\
--metrics_exporter otlp \\
--logs_exporter none \\
--service_name python-web-test \\
--resource_attributes "token=xxxxx" \\
--exporter_otlp_endpoint xxxxxx \\
python main.py
The generated log content after log correlation is enabled is as shown below:
2026-02-25 19:51:25,069 INFO [__main__] [main.py:19] [trace_id=c20f528fa45eebc2cbdae9e341ced39a span_id=fa84ffb16a978ab6 resource.service.name=python-web-test trace_sampled=True]
OpenTelemetry-JS
For log components that meet the requirements, TraceID and SpanID can be injected without any additional operations. If the probe detects the presence of an active Span, it will inject TraceID and SpanID into the logs using trace_id and span_id as markers respectively.
|
Winston | github.com/winstonjs/winston | 1.0.0+ |
Pino | github.com/pinojs/pino | 5.14.0+ |
Bunyan | trentm/node-bunyan | 1.0.0+ |
When we take Bunyan as an example, the generated log content after injecting TraceID and SpanID is as shown below:
INFO: this is info message: GET / | trace_id="e793e3edafb93f98a8a395c13096a26f" span_id="9564de488477c6a8" trace_flags="01"
Please note that if you customize the log formatting in the Winston framework, you need to refer to the following code to manually append TraceID and SpanID to the log content:
const logger = winston.createLogger({
level: "info",
format: winston.format.combine(
winston.format.timestamp(),
winston.format.printf(({ timestamp, level, message, trace_id, span_id }) => {
return `${timestamp} [${level.toUpperCase()}] [trace_id=${trace_id} span_id=${span_id}]: ${message}`;
})
),
transports: [new winston.transports.Console()],
});
OpenTelemetry-dotnet
For log components that meet the requirements, the injection of TraceID and SpanID can be achieved by modifying the pattern in the log configuration file.
Log4Net
In the log configuration file, TraceID and SpanID are injected using the %property{trace_id} and %property{span_id} fields respectively.
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%date [%level] %logger - %message trace_id=%property{trace_id} span_id=%property{span_id} %newline" />
</layout>
NLog
In the log configuration file, TraceID and SpanID are injected using the ${event-properties:TraceId} and span_id=${event-properties:SpanId} fields respectively.
<target xsi:type="File" name="file" fileName="logs/app.log"
layout="${longdate} [${level:uppercase=true}] ${logger} - ${message} trace_id=${event-properties:TraceId} span_id=${event-properties:SpanId}" />
Other solutions
For other solutions, use the OpenTelemetry SDK to obtain TraceID and SpanID from the context of the trace and append them to the log content. For example, for applications integrated via the OpenTelemetry-Go SDK, you can refer to the following code snippet to print TraceID and SpanID to the logs:
func logWithTrace(ctx context.Context, format string, args ...interface{}) {
span := trace.SpanFromContext(ctx)
sc := span.SpanContext()
traceID := sc.TraceID().String()
spanID := sc.SpanID().String()
msg := fmt.Sprintf(format, args...)
log.Printf("[trace_id: %s span_id: %s] %s", traceID, spanID, msg)
}