Overview
You can configure the JobManager spec, TaskManager spec, and default operator parallelism in Job parameters > Resources. In the Resources section, you can configure compute resources suitable for your job.
Stream Compute Service provides three CU specs: 0.5 CU, 1 CU, and 2 CUs. The JobManager spec and TaskManager spec can be set to different values. 0.5 CU represents fine-grained resources, which are available only to some new clusters by default. The JobManager and TaskManager spec options are not displayed for existing clusters. If you need to configure these two options for an existing cluster, please contact us. CUs used by a job
CUs used by job = JobManager spec + TaskManager spec x maximum parallelism of all operators in job.
You can adjust the specs and the default operator parallelism based on the maximum number of CUs available to the job given in the Resources section. For example, the figure below shows that a maximum of 8 CUs are available to the current job, so if you set the TaskManager spec to 2 CUs, the default operator parallelism can be set to 3 at most.
Resources for scenarios with low resource consumption
Some jobs may have no data sync most of the time, and using 1-CU resources causes a waste of resources. For such jobs, you can adjust the JobManager and TaskManager specs to 0.5 CU (1 CU in total) and the default operator parallelism to 1, reducing waste.
FAQs
1. What are fine-grained resources?
Fine-grained resources refer to resources whose compute unit specs can be smaller than 1 CU (1 core and 4 GB memory). Stream Compute Service currently provides three CU specs: 0.5 CU, 1 CU, and 2 CUs. The JobManager and TaskManager specs can be set to different values.
2. Why does the actual job parallelism fail to reach the maximum parallelism of the job?
If fine-grained resources are used, there is a remote possibility that resource fragmentation affects the job running, leading to an actual parallelism of the job smaller than the maximum one. In this case, you can select appropriate resource specs to avoid resource fragmentation as much as possible. If you need any help, please contact us.