- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Task APIs
- CreateSparkSessionBatchSQL
- CancelSparkSessionBatchSQL
- DescribeSparkSessionBatchSqlLog
- ModifySparkAppBatch
- CancelTask
- CreateResultDownload
- CreateSparkAppTask
- CreateTask
- CreateTasks
- DeleteSparkApp
- DescribeEngineUsageInfo
- DescribeResultDownload
- DescribeSparkAppTasks
- DescribeTaskResult
- DescribeTasks
- CreateSparkApp
- DescribeSparkAppJob
- DescribeSparkAppJobs
- ModifySparkApp
- Table APIs
- Metadata APIs
- Service Configuration APIs
- Permission Management APIs
- Data Source Connection APIs
- Database APIs
- Data Types
- Error Codes
- DLC Policy
- Service Level Agreement
- Contact Us
- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Task APIs
- CreateSparkSessionBatchSQL
- CancelSparkSessionBatchSQL
- DescribeSparkSessionBatchSqlLog
- ModifySparkAppBatch
- CancelTask
- CreateResultDownload
- CreateSparkAppTask
- CreateTask
- CreateTasks
- DeleteSparkApp
- DescribeEngineUsageInfo
- DescribeResultDownload
- DescribeSparkAppTasks
- DescribeTaskResult
- DescribeTasks
- CreateSparkApp
- DescribeSparkAppJob
- DescribeSparkAppJobs
- ModifySparkApp
- Table APIs
- Metadata APIs
- Service Configuration APIs
- Permission Management APIs
- Data Source Connection APIs
- Database APIs
- Data Types
- Error Codes
- DLC Policy
- Service Level Agreement
- Contact Us
Preparations
Before creating a data job, you need to configure the CAM role arn to secure the data access from the data job. For detailed directions, see Configuring Data Access Policy.
Directions
- Log in to the Data Lake Compute console and select Data job on the left sidebar.
- Click Create job.
Configure parameters as follows:
| Parameter | Description |
|---|---|
| Job name | It can contain up to 40 letters, digits, and underscores. |
| Job type | In batch: Batch data jobs based on Spark JAR In flow: Flow data jobs based on Spark Streaming |
| Data source connection | Data source for In batch data jobs. Currently, it can only be CKafka, which needs to be configured in advanced in Job configuration. |
| Data engine | It can be a Spark job data engine for which you have the permission. If you select Data source, you can only select a data engine connected to the data source. |
| Program package | The JAR format is supported. You can select a local file of up to 5 MB in size or a file in COS. If the local file exceeds 5 MB, upload it to COS for use. You can directly enter a COS path. |
| Dependency JAR resource | The JAR format is supported. You can select multiple resources. You can select a local file of up to 5 MB in size or a file in COS. If the local file exceeds 5 MB, upload it to COS for use. You can directly enter multiple COS paths and separate them by semicolon. |
| Dependency file resource | You can select a local file of up to 5 MB in size or a file in COS. If the local file exceeds 5 MB, upload it to COS for use. You can directly enter multiple COS paths and separate them by semicolon. |
| CAM role arn | The data access policy configured in Job configuration, which specifies the scope of data accessible to a data job. For more information, see Configuring Data Access Policy. |
| Main class | JAR package parameter in the main class. Separate multiple parameters by space. |
| Job parameter | -config information of the job, which starts with spark. in the format of k=v. Separate multiple parameters by line break.Example: spark.network.timeout=120s |
| Resource configuration | The engine resources that can be configured with the data job, the number of which cannot exceed the specifications of the selected data engine. Resource description: 1 CU ≈ 1-core 4 GB MEM Billable CUs = executor resource * executor quantity + driver resource Pay-as-you-go data engines are billed by the billable CUs. |
- After configuring the parameters, click Save.
Yes
No
Was this page helpful?