tencent cloud

Data Lake Compute

Release Notes
Product Introduction
Overview
Strengths
Use Cases
Purchase Guide
Billing Overview
Refund
Payment Overdue
Configuration Adjustment Fees
Getting Started
Complete Process for New User Activation
DLC Data Import Guide
Quick Start with Data Analytics in Data Lake Compute
Quick Start with Permission Management in Data Lake Compute
Quick Start with Partition Table
Enabling Data Optimization
Cross-Source Analysis of EMR Hive Data
Standard Engine Configuration Guide
Configuring Data Access Policy
Operation Guide
Console Operation Introduction
Development Guide
Runtime Environment
SparkJar Job Development Guide
PySpark Job Development Guide
Query Performance Optimization Guide
UDF Function Development Guide
System Restraints
Client Access
JDBC Access
TDLC Command Line Interface Tool Access
Third-party Software Linkage
Python Access
Practical Tutorial
Accessing DLC Data with Power BI
Table Creation Practice
Using Apache Airflow to Schedule DLC Engine to Submit Tasks
Direct Query of DLC Internal Storage with StarRocks
Spark cost optimization practice
DATA + AI
Using DLC to Analyze CLS Logs
Using Role SSO to Access DLC
Resource-Level Authentication Guide
Implementing Tencent Cloud TCHouse-D Read and Write Operations in DLC
DLC Native Table
SQL Statement
SuperSQL Statement
Overview of Standard Spark Statement
Overview of Standard Presto Statement
Reserved Words
API Documentation
History
Introduction
API Category
Making API Requests
Data Table APIs
Task APIs
Metadata APIs
Service Configuration APIs
Permission Management APIs
Database APIs
Data Source Connection APIs
Data Optimization APIs
Data Engine APIs
Resource Group for the Standard Engine APIs
Data Types
Error Codes
General Reference
Error Codes
Quotas and limits
Operation Guide on Connecting Third-Party Software to DLC
FAQs
FAQs on Permissions
FAQs on Engines
FAQs on Features
FAQs on Spark Jobs
DLC Policy
Privacy Policy
Data Privacy And Security Agreement
Service Level Agreement
Contact Us
DocumentationData Lake ComputeSQL StatementSuperSQL StatementIceberg Table StatementProcedure

Procedure

PDF
Focus Mode
Font Size
Last updated: 2024-08-07 17:29:12

Description

Supported engine: SparkSQL
Applicable table: External and native Iceberg tables

Basic Statement

CALL catalog_name.system.procedure_name(arg_name_2 => arg_2, arg_name_1 => arg_1);

CALL catalog_name.system.procedure_name(arg_1, arg_2, ... arg_n);

Snapshot Management

rollback_to_snapshot

Roll back the snapshot to a specified version. Input parameters: table name and version number.
CALL `DataLakeCatalog`.`system`.rollback_to_snapshot('validation.dempts', 1);

rollback_to_timestamp

Roll back the snapshot to a specified timestamp. Input parameters: table name and timestamp
CALL `DataLakeCatalog`.`system`.rollback_to_timestamp('validation.dempts', TIMESTAMP '2022-08-11 19:49:43.224');

set_current_snapshot

Set the current snapshot version. Input parameters: table name and version number.
CALL `DataLakeCatalog`.`system`.set_current_snapshot('validation.dempts', 1);

cherrypick_snapshot

Cherrypick from a specified snapshot version to the current snapshot.
CALL `DataLakeCatalog`.`system`.cherrypick_snapshot('validation.dempts', 1);
CALL `DataLakeCatalog`.`system`.cherrypick_snapshot(snapshot_id => 1, table => 'my_table' )

Metadata Management

expire_snapshots

Clean up expired snapshots to reduce the number of small files.
CALL `Catalog`.`system`.expire_snapshots(table_name, [older_than], [retain_last], [max_concurrent_deletes], [stream_results]);
Example:
CALL `DataLakeCatalog`.`system`.expire_snapshots('validation.dempts', TIMESTAMP '2021-06-30 00:00:00.000', 100);

remove_orphan_files

Remove metadata files that are no longer referenced.
CALL `Catalog`.`system`.remove_orphan_files(table_name, [older_than], [location], [dry_run], [max_concurrent_deletes]);
Example:
CALL `DataLakeCatalog`.`system`.remove_orphan_files(`table`=>'validation.dempts', dry_run=>TRUE);
CALL `DataLakeCatalog`.`system`.remove_orphan_files(`table`=>'validation.dempts', `location`=>'cosn://channingdata-1305424723/example2/');
CALL `DataLakeCatalog`.`system`.remove_orphan_files('validation.dempts', TIMESTAMP '2022-07-10 17:25:19.000');

remove_orphan_files

Remove metadata files that are no longer referenced.
CALL `Catalog`.`system`.remove_orphan_files(table_name, [older_than], [location], [dry_run], [max_concurrent_deletes]);
Example:
CALL `DataLakeCatalog`.`system`.remove_orphan_files(`table`=>'validation.dempts', dry_run=>TRUE);
CALL `DataLakeCatalog`.`system`.remove_orphan_files(`table`=>'validation.dempts', `location`=>'cosn://channingdata-1305424723/example2/');
CALL `DataLakeCatalog`.`system`.remove_orphan_files('validation.dempts', TIMESTAMP '2022-07-10 17:25:19.000');

rewrite_data_files

Merge and rewrite data files, i.e. merging small data files.
CALL `Catalog`.`system`.rewrite_data_files(table_name, [strategy], [sort_order], [options], [where]);
Example:
CALL `DataLakeCatalog`.`system`.rewrite_data_files('validation.dempts');
CALL `DataLakeCatalog`.`system`.rewrite_data_files(`table`=>'validation.dempts', `strategy`=>'sort', `sort_order`=>'id DESC NULLS LAST,data ASC NULLS FIRST');
CALL `DataLakeCatalog`.`system`.rewrite_data_files(`table`=>'validation.dempts', `options`=>map('min-input-files','2'));
CALL `DataLakeCatalog`.`system`.rewrite_data_files(`table`=>'validation.dempts', `where`=>'id = 3 and data = "foo"');

rewrite_manifests

Merge and rewrite manifests files.
CALL `Catalog`.`system`.rewrite_manifests(table_name, [using_caching]);
Example:
CALL `DataLakeCatalog`.`system`.rewrite_manifests('validation.dempts');
CALL `DataLakeCatalog`.`system`.rewrite_manifests('validation.dempts', FALSE);

ancestors_of

Get the lineage information of a snapshot.
CALL `Catalog`.`system`.ancestors_of(table_name, [snapshot_id]);
Example:
CALL `DataLakeCatalog`.`system`.ancestors_of('validation.dempts');
CALL `DataLakeCatalog`.`system`.ancestors_of('validation.dempts', 1);

Data Table Migration Management

Note
The original table must be a Hive table or Spark table.

snapshot

Create lightweight temporary tables based on the original tables. The temporary tables directly reuse the snapshots of the original tables.
CALL `Catalog`.`system`.snapshot(source_table, table, [location], [properties]);
Example:
CALL `DataLakeCatalog`.`system`.snapshot('validation.table_01', 'validation.snap');
CALL `DataLakeCatalog`.`system`.snapshot('validation.table_01', 'validation.snap2', 'cosn://channingdata-1305424723/example3/');

migrate

Update and replace table attributes.
CALL `Catalog`.`system`.migrate(table, [properties]);
Example:
CALL `DataLakeCatalog`.`system`.migrate('validation.table_01');
CALL `DataLakeCatalog`.`system`.migrate('validation.table_01', map('data', 'name'));

add_files

Load data files directly from Hive, and you can specify data files to a specific partition.
CALL `Catalog`.`system`.add_files(table, source_table, [partition_filter]);
Example:
CALL `DataLakeCatalog`.`system`.add_files(`table`=>'validation.table_02', `source_table`=>'validation.table_01');
CALL `DataLakeCatalog`.`system`.add_files(`table`=>'validation.table_02', `source_table`=>'validation.table_01', `partition_filter`=>map('part_col', 'A'));


Previous Topic: DQL StatementNext Topic: Differences in Statement Between Iceberg External Tables and Native Tables

Help and Support

Was this page helpful?

Help us improve! Rate your documentation experience in 5 mins.

Feedback