This document introduces a free-of-maintenance approach to import Kafka data to Cloud Data Warehouse PostgreSQL instances by using SCF.
Cloud Data Warehouse PostgreSQL (CDWPG) can sync messages from the messaging middleware for analysis.
In the SCF console, select Functions > Create. In the Create page, enter ckafka and CDW in the Fuzzy search field, complete the settings and click Next.
On the Function configuration page, complete the settings in Environment configuration and Network configuration in Advanced configuration as follows:
|DB_HOST||Supported||If the function is deployed in a VPC and in the same subnet as CDWPG, you can enter the private IP of CDWPG; otherwise, enter the public IP and configure an allowlist.|
|DB_SCHEMA||Supported||Schema name. If it is not specified during table creation, it will be `public` in general.|
|DB_PORT||No||CDWPG port, which is 5436 by default.|
|MSG_SEPARATOR_ASCII||No||ASCII code of the data delimiter in CKafka, which is 39 (comma) by default. As commas usually show up in the business data, we recommend you set this parameter to 11 (vertical bar).|
|MSG_NULL||No||NULL value of CKafka consumption. The default value is `\N`|
|REPLACE_0X00||No||Whether to replace "0x00" in strings. The default value is 0 (1 indicates to replace).|
|ENABLE_DEBUG||No||Whether to print error records. The default value is 0 (1 indicates to print).|
|ENABLE_COS||No||Whether to dump unwritten records to COS. The default value is 0 (1 indicates to dump).|
|COS_SECRET_ID||No||`secret_id` for COS access. If `ENABLE_COS` is 1, this field is required.|
|COS_SECRET_KEY||No||`secret_key` for COS access. If `ENABLE_COS` is 1, this field is required.|
|COS_BUCKET||No||COS bucket name. If `ENABLE_COS` is 1, this field is required.|
|STATMENT_TIMEOUT||No||Query timeout period, which is 50 seconds by default.|
In the Functions list in the SCF console, click the name of the newly created function to enter the function details page and click Trigger management > Create trigger on the left to create a trigger. Here, set CKafka trigger for Trigger method.
For details of trigger settings, see CKafka Trigger Description.
Contact our sales team or business advisors to help your business.
Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.
Was this page helpful?