tencent cloud

Feedback

Using COS to Store Tencent Cloud Product Logs

Last updated: 2024-03-25 15:28:24

    Overview

    The process of using Tencent Cloud products generates a massive number of logs, which can be used to help you analyze and make decisions for your business. For your convenience, you can store these logs persistently in COS, and then easily access them for analysis by using various APIs, SDKs or tools.
    COS log storage offers the following benefits:
    Persistent storage: COS provides stable, persistent storage that allows you to store logs in COS at a very low cost. You can access your logs anytime, anywhere for business analysis or decision-making.
    COS Select: This feature allows for the simple extraction of logs stored on COS. You can combine log fields to make it easier to extract the information you need and reduce data download traffic.
    Data analysis: You can analyze one or more COS logs using Sparkling, which then can be used to facilitate and provide a basis for important business decisions.

    Shipping Logs

    You can ship your Tencent Cloud product logs to COS in one of the following two ways:
    Use the Tencent Cloud log shipping feature in which logs are directly shipped to COS. Products such as COS and CloudAudit (CA) support directly shipping logs to COS.
    Use the CLS log shipping feature in which logs are first shipped to CLS and then transferred to COS for persistent storage.
    The table below shows the particular log shipping method or methods currently supported by various Tencent Cloud products:
    Product
    Supports Direct Shipping to COS
    Supports Shipping to CLS
    CA
    Yes
    No
    CLB
    No
    Yes
    CKafka
    Yes
    No
    API Gateway
    No
    Yes
    SCF
    No
    Yes
    TKE
    No
    Yes
    CSS
    No
    Yes
    TCB
    No
    Yes (however, TCB does not support shipping logs to COS through CLS)
    COS
    Yes
    Yes for accounts on the allowlist. You can contact us to add your account to the allowlist.

    Directly shipping logs to COS

    The Tencent Cloud products outlined below allow you to ship logs directly to COS by setting log shipping rules as instructed in the product-specific documentation.
    Product
    Log Shipping Document
    Log Shipping Interval
    Log Shipping Path
    CA
    10-15 min
    cloudaudit/customprefix/timestamp
    CKafka
    5-60 min
    You can specify the interval
    instance id/topic id/timestamp
    COS
    5 min
    You can specify the path prefix. We recommend specifying an identifiable path, e.g. cos_bucketname_access_log/timestamp.
    Note:
    CKafka supports shipping the message data that it generates. However, to get logs on certain actions such as the creation of CKafka instances, you need to ship the CA logs.

    Using CLS to ship logs to COS

    The Tencent Cloud products outlined below allow you to ship logs directly to CLS for search and analysis. With CLS, you can further ship the logs to COS for persistent storage, thus reducing your costs and allowing for convenient offline analysis.
    Product
    Documentation
    API Gateway
    TKE
    CSS
    Click here to view
    CLS can ship the following three types of logs to COS:
    CSV-formatted logs: Logs formatted with comma-separated values. For more information, see Shipping CSV-Formatted Logs.
    JSON-formatted logs: Logs in JSON format. For more information, see Shipping JSON-Formatted Logs.
    Original logs: Logs in their original format, including those with full text in a single line, full text in multi-lines, and comma-separated values (for certain products). For more information, see Shipping Original Logs.
    To ship logs from CLS to COS, you need to do the following:
    1. Choose a Tencent Cloud product that meets your business needs and configure a logset and log topic as instructed in the log shipping documentation provided above to connect your business data to CLS.
    2. With your particular business needs in mind, select the most suitable log type to ship to COS. We recommend that you use the product name as the path prefix to differentiate product logs, for example, you could use the following name for TKE logs: TKE_tkeid_log/timestamp.
    3. Once the shipping rule is configured, you can configure COS event notifications for file uploads through SCF. You can perform further operations on logs shipped to COS based on these event notifications. For more information, see Event Notifications.

    Log Analysis

    Downloading logs locally for offline analysis

    You can download logs to the local file system through the Tencent Cloud console, or by using various SDKs, APIs, and tools. To do so, follow the instructions in one of the following documents and replace each file path in the code with your log storage path:

    Analyzing logs using COS Select

    You have the option to extract and analyze COS logs stored in CSV or JSON format directly by using the COS Select feature. It enables you to query desired log fields, which largely reduces the number of logs transferred by COS, thus lowering usage costs and improving data access speed. For more information on COS Select, see Select Overview.
    Currently, you can access COS Select using the console or APIs.
    Method
    Instructions
    Console
    API
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support