tencent cloud


SCF + CDWPG for CKafka Data Import

Last updated: 2022-01-23 16:11:24


    SCF provides a serverless execution environment for companies and developers. For more information, see SCF.

    A common use case of CDWPG is to syncing messages from the messaging middleware for analysis. This document describes a convenient method where an SCF function imports data from Kafka to CDWPG in real time, eliminating your need to maintain any services.


    • This function currently can use CKafka as the data source and does not support self-built Kafka.
    • This function currently can write data into only one table in CDWPG as the target. If you want to write data into multiple tables, create the corresponding function for each table as follows:


    Step 1. Create a function

    In the SCF console, select Function Service > Create. On the Create Function page, select Phython3.6 as the runtime environment, search for the keyword "ckafka" in Fuzzy search, select the template function Load CKafka Data into CDW, and click Next.

    On the Function Configuration page, complete the settings in Environment Configuration and Network Configuration in Advanced Configuration as follows:

    • Environment Configuration
      • Memory: set the memory based on the actual running status, which is 128 MB by default. If it is insufficient during data import, you should increase it.
      • Environment variables:
        Parameter Required Description
        DB_DATABASE Yes Database name.
        DB_HOST Yes If the function is deployed in a VPC and in the same subnet as CDWPG, you can enter the private IP of CDWPG; otherwise, enter the public IP and configure an allowlist.
        DB_USER Yes Username.
        DB_PASSWORD Yes User password.
        DB_SCHEMA Yes Schema name. If it is not specified during table creation, it will be `public` in general.
        DB_TABLE Yes Table name.
        DB_PORT No CDWPG port, which is 5436 by default.
        MSG_SEPARATOR_ASCII No ASCII code of the data delimiter in CKafka, which is 39 (comma) by default. As commas usually show up in the business data, we recommend you set this parameter to 11 (vertical bar).
        MSG_NULL No NULL value of CKafka consumption. The default value is `\N`
        REPLACE_0X00 No Whether to replace "0x00" in strings. The default value is 0 (1 indicates to replace).
        ENABLE_DEBUG No Whether to print error records. The default value is 0 (1 indicates to print).
        ENABLE_COS No Whether to dump unwritten records to COS. The default value is 0 (1 indicates to dump).
        COS_SECRET_ID No `secret_id` for COS access. If `ENABLE_COS` is 1, this field is required.
        COS_SECRET_KEY No `secret_key` for COS access. If `ENABLE_COS` is 1, this field is required.
        COS_BUCKET No COS bucket name. If `ENABLE_COS` is 1, this field is required.
        STATMENT_TIMEOUT No Query timeout period, which is 50 seconds by default.
    • Network Configuration
      • VPC: we recommend you activate VPC and set the same VPC and subnet values as those of the CDWPG instance.

        The corresponding values in CDWPG are as shown below:
      • Public Network Access: we recommend you also enable this option.

    Step 2. Configure a trigger

    In the Function Service list in the SCF console, click the name of the newly created function to enter the function details page and click Trigger Management > Create a Trigger on the left to create a trigger. Here, set CKafka trigger for Trigger Method as shown below:

    For more information on trigger parameter configuration, see CKafka Trigger Description.

    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support