tencent cloud


Dumping Logs to ES with SCF + CLS

Last updated: 2023-05-05 15:44:04


    This document describes how to use SCF to dump CLS logs to ES. CLS is mainly used for log collection, and SCF mainly provides node computing capabilities for data processing. For the data processing flowchart, see Function Processing Overview.


    Creating logset and topic

    1. Log in to the CLS console, and click Log Topic on the left sidebar.
    2. On the logset management page, select the region of the logset at the top.
    3. Click Create Log Topic and enter relevant information in the Create Logset pop-up window:
    Log Topic Name: Enter project_test for example.
    Logset Name: Enter nginx for example.
    4. Click OK.
    5. The log topic is successfully added, and you will be redirected to the log topic management page.

    Creating an SCF function

    1. Log in to the SCF console and enter the Functions page.
    2. At the top of the Functions page, select the Beijing region and click Create to go to the function creating page and configure the following parameters:
    Creation method: Select Template.
    Fuzzy search: Enter CLSToElasticsearch and search.
    3. You can click Learn more in a template to view its details or download the template. Click Next to enter the Function configuration page.
    4. In the Basic configurations section, the function name has been automatically generated and can be modified as needed. Follow the prompts to configure environment variables and execution role.
    Environment variable: Add the following environment variables and configure them.
    ES service address
    ES username, which is elastic by default.
    ES password.
    Execution Role: Select Enable and click Configure and use SCF template role.
    5. In Network configuration, select Enable for VPC, and select the same VPC and subnet as Elasticsearch.
    6. Keep the default configuration and click Complete to complete the function creation.

    Configuring CLS triggers

    1. Log in to the CLS console and click Logset on the left sidebar.
    2. Locate the existing logset you want, and click View under Operation on the right to go to the logset details page.
    3. On the log topic details page, select Function Processing and click Create. Add the created function in the Function Processing pop-up window.
    The main parameter information is as follows. Use the default values for the remaining configuration items.
    Namespace: Select the function namespace.
    Function name: Select the function created in the Creating SCF function step.
    Alias: Select a function alias.
    Max waiting time: Configure the longest waiting time for a single event pull. Default value: 60s.

    Testing the function

    1. Download the log file in the test sample, extract demo-scf1.txt, and import it to the source CLS service.
    2. Go to the SCF console to view the execution result. Select the Log Query tab on the function details page to view the printed log information.
    3. Log in to the ES console to view the data dumping and processing result.
    You can write specific data processing methods as needed.
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support