A log topic is a basic unit for log data collection, storage, search and analysis on the CLS platform. The massive amounts of logs collected are managed by log topic. For example, you can configure log collection rules and storage time, search for and analyze logs, and download, consume, and ship logs by log topic.
A log topic usually corresponds to an application/service. It is recommended to collect logs of the same application/service from different servers to the same log topic. For example, a payment service (payService) is deployed on dozens of machines and has two types of logs: access logs (access_log) and error logs (error_log). You can create two log topics, payService_access_log_topic and payService_error_log_topic, to collect the two types of logs on the machines respectively. You can use these two log topics to centrally search for and analyze all logs on the machines.
Log topics and applications/services are not strictly in one-to-one mapping relationships. If the log structures of two services are similar, and the logs of the two services need to be analyzed in a centralized manner, you can report the logs of the two services to the same log topic.
A logset is a class of log topics and can contain multiple log topics. A logset itself does not store any log data, but just makes it easier for users to manage log topics.
A logset usually corresponds to a project/business in a company. You are advised to add the log topics of multiple applications/businesses of a project/business to the same logset. For example, if a company's e-commerce project contains multiple services (payService, userService, stockService, etc.), you can create a logset named e_commerce_logset and add the log topics of these services to this logset. In this way, when the company has multiple projects, the project personnel only need to view the log topics in the logset corresponding to the project, without interference from the log topics of other projects.
The logset to which a log topic belongs can be specified when the log topic is created and cannot be changed once specified.
As shown in the figure above, the company has two departments:
To use CLS to monitor the logs of the applications of the company, you can create the following logsets and log topics:
|e_commerce_logset||payService_access_log_topic||Department: Department A|
|e_commerce_logset||payService_error_log_topic||Department: Department A|
|e_commerce_logset||userService_access_log_topic||Department: Department A|
|e_commerce_logset||userService_error_log_topic||Department: Department A|
|e_commerce_logset||stockService_access_log_topic||Department: Department A|
|e_commerce_logset||stockService_error_log_topic||Department: Department A|
|e_commerce_logset||......||Department: Department A|
|gameApplet_logset||gameApplet_nginx_log_topic||Department: Department B|
|socialApplet_logset||socialApplet_nginx_log_topic||Department: Department B|
The labels are used to identify the departments to which the logsets and log topics belong. Combined with permission policies, the personnel in each department can view the data of their own department only.
For department A, the e_commerce_logset logset covers the log topics of all e-commerce services. If other projects are added later, you only need to create a logset.
For department B, although the current technical architecture is relatively simple, with only two types of logs in total, two logsets are created, each with only one log topic, for the following purposes: