Scheduling Mode — spark.scheduler.mode
Spark Property
Scheduling Mode (aka order task policy or scheduling policy or scheduling order) defines a policy to sort tasks in order for execution.
The scheduling mode schedulingMode
attribute is a part of the TaskScheduler Contract.
The only implementation of the TaskScheduler
contract in Spark — TaskSchedulerImpl — uses spark.scheduler.mode setting to configure schedulingMode
that is merely used to set up the rootPool attribute (with FIFO
being the default). It happens when TaskSchedulerImpl
is initialized.
There are three acceptable scheduling modes:
-
FIFO
with no pools but a single top-level unnamed pool with elements being TaskSetManager objects; lower priority gets Schedulable sooner or earlier stage wins. -
FAIR
with a hierarchy ofSchedulable
(sub)pools with the rootPool at the top. -
NONE (not used)
Note
|
Out of three possible SchedulingMode policies only FIFO and FAIR modes are supported by TaskSchedulerImpl.
|
Note
|
After the root pool is initialized, the scheduling mode is no longer relevant (since the Schedulable that represents the root pool is fully set up). The root pool is later used when |
Note
|
The root pool is a Schedulable . Refer to Schedulable.
|
Monitoring FAIR Scheduling Mode using Spark UI
Caution
|
FIXME Describe me… |