tencent cloud


ETL Log Processing

Last updated: 2022-05-18 15:01:54


    This document describes how to use SCF to process CLS logs. CLS is mainly used for log collection, while SCF mainly provides node computing capabilities.
    The data flow is as follows:


    You have logged in to the CLS console.


    Creating log topic

    Create two log topics as instructed in Managing Log Topic.


    As both the source and destination of data ETL are CLS, you need to create at least two topics.

    Creating SCF function

    Create a function as instructed in Creating and Testing Function.
    The main parameters are as follows:

    • Region: Select the Beijing region.
    • Function Name: Enter "CLSdemo".
    • Creation Method: Click Template and select the CLSLogETL template.

    You should select the VPC and subnet of CLS for the created function on the Function Configuration page.

    Configuring CLS trigger

    1. Log in to the CLS console.
    2. On the left sidebar, click Log Topic to enter the log topic management page.
    3. Find the log topic you just created and click its ID/name to enter the log topic details page.
    4. On the log topic details page, select the Function Processing tab and click Create.
    5. In the Function Processing pop-up window, add the created function and click OK.
      The main parameter information is as follows. Use the default values for the remaining configuration items.
      • Namespace: Select the function namespace.
      • Function Name: Select the function created in the Creating SCF function step.
      • Alias: Select a function alias.
      • Batch Window: Configure the longest waiting time for a single event pull. Default value: 60s.

    Testing function

    1. Download the log file in the test sample, extract demo-scf1.txt, and import it to the source CLS service.

    2. Switch to the SCF console to view the execution result.
      On the function details page, select the Log Query tab to view the printed log information.

    3. Switch to the target CLS service to view the data processing result.


      You can write specific data processing methods as needed.

    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support