tencent cloud

Cloud Log Service

Release Notes and Announcements
Release Notes
Announcements
User Guide
Product Introduction
Overview
Features
Available Regions
Limits
Concepts
Service Regions and Service Providers
Purchase Guide
Billing Overview
Product Pricing
Pay-as-You-Go
Billing
Cleaning up CLS resources
Cost Optimization
FAQs
Getting Started
Getting Started in 1 Minute
Getting Started Guide
Quickly Trying out CLS with Demo
Operation Guide
Resource Management
Permission Management
Log Collection
Metric Collection
Log Storage
Metric Storage
Search and Analysis (Log Topic)
Search and Analysis (Metric Topic)
Dashboard
Data Processing documents
Shipping and Consumption
Monitoring Alarm
Cloud Insight
Independent DataSight console
Historical Documentation
Practical Tutorial
Log Collection
Search and Analysis
Dashboard
Monitoring Alarm
Shipping and Consumption
Cost Optimization
Developer Guide
Embedding CLS Console
CLS Connection to Grafana
API Documentation
History
Introduction
API Category
Making API Requests
Topic Management APIs
Log Set Management APIs
Index APIs
Topic Partition APIs
Machine Group APIs
Collection Configuration APIs
Log APIs
Metric APIs
Alarm Policy APIs
Data Processing APIs
Kafka Protocol Consumption APIs
CKafka Shipping Task APIs
Kafka Data Subscription APIs
COS Shipping Task APIs
SCF Delivery Task APIs
Scheduled SQL Analysis APIs
COS Data Import Task APIs
Data Types
Error Codes
FAQs
Health Check
Collection
Log Search
Others
CLS Service Level Agreement
CLS Policy
Privacy Policy
Data Processing And Security Agreement
Contact Us
Glossary
DocumentationCloud Log ServiceOperation GuideMetric CollectionConverting Logs to Metrics

Converting Logs to Metrics

PDF
Focus Mode
Font Size
Last updated: 2026-02-12 15:13:08
Original logs stored in the log topic can be analyzed using SQL to get metrics. For example, use the following SQL to get the number of error requests, where errorRequestCount is a metric:
status:>=400 | select count(*) as errorRequestCount
This way of obtaining metrics is highly flexible. You can readily get new metrics or change metric calculation rules by adjusting search conditions and SQL statements. However, it depends on the complete storage of original logs in the log topic Standard Storage. If you need to query the trend of metrics in the last 60 days, the original logs should be stored for at least 60 days, which brings a large data storage cost.
When metrics are relatively fixed, you can convert original logs into metrics for storage to reduce the log retention days (or you can use log settlement and shipping logs to Cloud Object Storage (COS) to store historical logs at low cost), thereby lowering overall costs. Besides, since the number of data entries for converted metrics is generally much less than the number of original log entries, and metric topics are optimized for time-series data queries, querying converted metrics typically delivers better performance, enabling faster metric queries and dashboard drawing.

Conversion Logic

Cloud Log Service (CLS) converts original logs into metrics through scheduled SQL analysis. For example, if there are 10 log entries in a minute, CLS converts these 10 entries into metrics through scheduled SQL analysis.
Original logs:
{"requestTime":"2020/12/30 15:35:46","method":"POST","handler":"/messages","timeCost":"45","status":"200"}
{"requestTime":"2020/12/30 15:35:46","method":"POST","handler":"/users","timeCost":"23","status":"200"}
{"requestTime":"2020/12/30 15:35:47","method":"POST","handler":"/messages","timeCost":"57","status":"200"}
{"requestTime":"2020/12/30 15:35:47","method":"POST","handler":"/users","timeCost":"25","status":"200"}
{"requestTime":"2020/12/30 15:35:47","method":"POST","handler":"/users","timeCost":"689","status":"200"}
{"requestTime":"2020/12/30 15:35:48","method":"POST","handler":"/users","timeCost":"87","status":"200"}
{"requestTime":"2020/12/30 15:35:48","method":"POST","handler":"/users","timeCost":"34","status":"200"}
{"requestTime":"2020/12/30 15:35:48","method":"POST","handler":"/users","timeCost":"698","status":"200"}
{"requestTime":"2020/12/30 15:35:48","method":"POST","handler":"/messages","timeCost":"25","status":"200"}
{"requestTime":"2020/12/30 15:35:48","method":"POST","handler":"/messages","timeCost":"78","status":"200"}
Scheduled SQL:
* | select count(*) as requests_total,method,handler group by method,handler order by requests_total desc limit 10000
Convert logs to the following metrics:
requests_total{method="POST", handler="/messages"} 4
requests_total{method="POST", handler="/users"} 6
A scheduled SQL task can run periodically, for example, once per minute. Each time, it converts the most recent original logs from the last minute, automatically converting them into minute-level metrics.

Prerequisites

Logs have been reported to the log topic, and standard storage is used. For low-frequency storage, SQL is not supported, so logs cannot be converted to metrics.
The log topic has configured index.

Operation Steps

Step 1: Adding Scheduled SQL Analysis

1. Log in to the CLS console.
2. In the left sidebar, choose Scheduled SQL analysis, and click Create to create a task.
3. In Basic Configuration, fill in the following information:
Configuration Item
Description
Source Log Topic
The input for the scheduled SQL task, namely, the log topic where the original logs to be converted into metrics are located.
Target topic
The output of the scheduled SQL task, select metric topic to store the converted metrics in the specified metric topic. If there is no metric topic currently, you can create a metric topic.
Query Statement
Use SQL statements to statistically analyze original logs to get metrics, for example:
* | select count(*) as requests_total,method,handler group by method,handler order by requests_total desc limit 10000
Metric Name
Select the field that needs to be the metric name from the SQL statement results. For example, in the example above, requests_total is the metric name, and its value is the metric value.
Metric Dimensions
Select the fields that need to be metric dimensions from the SQL statement results. For example, in the example above, method and handler are metric dimensions.
4. Click Next, and fill in the following information in Scheduling configuration:
Configuration Item
Description
Duration
The time range for querying logs. For example, query log data from 2023-01-01 00:00:00 to 2023-03-31 00:00:00. An empty end time means the task will continue to run.
It is recommended to set the end time to empty to continuously convert newly collected logs into metrics.
Scheduling cycle
Periodic query, meaning logs are queried once every X minutes. The value ranges from 1 to 1,440 minutes.
It is recommended to run it every 1 minute to convert logs into metrics more promptly.
Query Time Window
Time window for query statements.
It is recommended to use @m-1m ～ @m, the last minute. With the scheduling cycle of 1 minute, this is equivalent to converting the logs of that minute into metrics every minute.
SQL execution delayed
The delay time for query execution, set in Advanced Settings. The default value to 60 seconds. There is usually a delay in log index generation, and queries cannot be performed before the index is generated. Therefore, it is recommended to set a 60-second delay for querying, by which time the index is already generated (99.9% of the index data will be generated within 5 seconds).
5. Click OK to save the configuration.

Step 2: Querying Generated Metrics

1. In the left sidebar, choose Search and Analysis, select the region where the metric topic is located at the top, select Metric Topic as the subject type, and then select the target metric topic.
2. Enter the metric name in the input box to query metrics.
Previous Topic: Metrics ReportingNext Topic: Storage Class Overview

Help and Support

Was this page helpful?

Help us improve! Rate your documentation experience in 5 mins.

Feedback