tencent cloud

DataHub is a CKafka service module that connects data sources to data processing systems. It preprocesses data obtained from business data sources and then distributes the output data to offline/online processing systems. In this way, it decouples these systems from business data sources.

DataHub supports accessing different types of data generated by various data sources for unified management and distribution to downstream offline/online processing systems, forming a clear data flow channel.

Data Source

DataHub supports the following types of data sources: actively reported data, services, and logs.

• Actively reported data: HTTP data can be actively reported to CKafka. For detailed directions, see Reporting over HTTP.
• Asynchronously pulled data - services: MongoDB, CLS, and DTS are supported.
  • MongoDB: CKafka can collect data from MongoDB. For detailed directions, see MongoDB.
  • DTS: CKafka can collect data from DTS. For detailed directions, see DTS.

Note：

You can configure log data source access in the shipper's console instead of the CKafka console.

Data Access

Data access consists of two parts:

Active reporting

An SDK is provided. The process is as shown below:

Async pull

Complete productized configuration GUIs are offered for service, log, and API data sources, eliminating your need to care about the underlying implementation.