Each Flink job has an attribute called maximum parallelism (MaxParallelism). It determines the maximum degree of parallelism and specifies the upper limit for dynamic scaling.
This parameter involves the most underlying state allocation logic in Flink, so it cannot be changed once set. If you must change this parameter (you want to scale out the cluster to CUs exceeding MaxParallelism, for example), Flink has to discard the current runtime state of the job and restart it.
In other words, when the job is recovered from checkpoint (a checkpoint taken when Checkpoint or Savepoint is triggered), the new maximum parallelism of operators specified cannot exceed this value. Otherwise, Flink will raise an exception and stop starting the job. If the Flink job is not started from checkpoint, it can still be properly started.
Parameter description
By default, this configuration option has been set in Job parameters > Advanced parameters of a new job draft, and you do not need to change it. If you delete this configuration option, the maximum parallelism defaults to 2048.
Note
For jobs created before April 14, 2021 (the date when the Flink version was updated), the maximum parallelism defaults to 128. To increase their maximum parallelism, you need to manually change the value of this parameter, and the system will discard the existing runtime state and restart the jobs.
In most cases, you can ignore the setting of the maximum parallelism. You need to set this parameter only when the maximum parallelism of any operator in the job exceeds MaxParallelism, or when you want to explicitly restrict the maximum scaling capabilities of the job.
Notes
The minimum value of pipeline.max-parallelism is the maximum parallelism of all operators in the job. For example, if a job has 5 operators, whose parallelism is [1, 5, 100, 2, 2], respectively, the minimum value of pipeline.max-parallelism allowed is 100.
In Stream Compute Service, the maximum value of pipeline.max-parallelism is fixed to 16384. However, we recommend you maintain pipeline.max-parallelism at 2048 or lower to avoid unnecessary runtime overhead or reduced processing capabilities of the job.