tencent cloud

Feedback

Raw Log Shipping

Last updated: 2022-10-13 15:12:38

    Overview

    This document describes how to ship the collected raw log to COS. Currently, only logs collected in a single line, in multiple lines, or with certain separators are supported in this scenario.

    Notes

    You can configure CSV shipping to COS to ship certain raw logs. The following table describes the options of CSV shipping to COS but doesn't apply to JSON or Parquet shipping.

    Log Collection Format Support for Raw Log Shipping User Raw Log CLS Storage Format Format for Shipping to COS
    Full text in a single line Yes. For more information, see Logs collected in a single line. yourlog __CONTENT__:yourlog yourlog
    Full text in multiple lines Yes. For more information, see Logs collected in multiple lines. yourlog __CONTENT__:yourlog yourlog
    Separator (CSV) format It depends. For more information, see CSV format. Only space, tab, comma, semicolon, and vertical bar are supported as raw log separators. V1 separator V2 separator V3 separator...Vn
    For example, V1,V2,V3...Vn
    K1:V1 K2:V2 K3:V3...Kn:Vn V1 separator V2 separator V3 separator...Vn
    For example, V1,V2,V3...Vn
    JSON format No K1:V1 K2:V2 K3:V3...Kn:Vn K1:V1 K2:V2 K3:V3...Kn:Vn V1,V2,V3...Vn differs from the original JSON.
    Full regex No V11V22V33...Vnn K1:V1 K2:V2 K3:V3...Kn:Vn V1,V2,V3...Vn differs from the raw log.

    Directions

    Log collected in a single line or multiple lines

    For logs collected in a single line or multiple lines, you can configure parameters based on CSV shipping to implement raw log shipping.

    1. Complete the first step of Basic Configuration as instructed in CSV Shipping.
    2. Set Shipping Format to CSV, retain the __CONTENT__ field, delete other fields, set Separator to Space, set Escape Character to Space, set Invalid Field Filling to None, and disable Key in First Line.
      The parameters are described as follows:
      Configuration ItemDescriptionRemarks
      Key __CONTENT__ For full text in a single line or multiple lines, __CONTENT__ is used as the default key, and the raw log is used as the value. When the raw log is shipped, only the __CONTENT__ field is retained.
      Separator Space Set **Separator** to **Space** for the full text in a single line or multiple lines.
      Escape Character None To prevent the raw log from being modified due to escape characters, set **Escape Character** to **None**.
      Invalid Field Filling None Set **Invalid Field Filling** to **None**.
      Key in First Line Disabled You don't need to add a description of the field name in the first line of the CSV file for raw log shipping.
    3. Click OK to enable shipping.

    Logs collected in CSV format

    Note:

    CSV shipping supports limited types of separators, including space, tab, comma, semicolon, and vertical bar. Therefore, you can ship log data in the raw format only when the separator in the raw log content is supported by CSV shipping.

    1. Complete the first step of Basic Configuration as instructed in CSV Shipping.
    2. Set Shipping Format to CSV. During field configuration, you need to delete CLS metadata fields.

    The parameters are described as follows:

    Configuration Item Value Description
    Key Keys Only the user field is retained.
    Separator A value selected from the drop-down list Select the separator of the raw log content. If separators are different, raw log shipping is not supported. Currently, only space, tab, comma, semicolon, and vertical bar are supported.
    Escape Character None To prevent the raw log from being modified due to escape characters, set **Escape Character** to **None**.
    Invalid Field Filling None Set **Invalid Field Filling** to **None**.
    Key in First Line Disabled You don't need to add a description of the field name in the first line of the CSV file for raw log shipping.
    3. Click **OK** to enable shipping.
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support