Fair Scheduler Pool Details Page
The Fair Scheduler Pool Details page shows information about a Schedulable pool and is only available when a Spark application uses the FAIR scheduling mode (which is controlled by spark.scheduler.mode setting).
Figure 1. Details Page for production Pool
PoolPage renders a page under /pool URL and requires one request parameter poolname that is the name of the pool to display, e.g. http://localhost:4040/stages/pool/?poolname=production. It is made up of two tables: Summary (with the details of the pool) and Active Stages (with the active stages in the pool).
PoolPage is a part of Stages tab.
PoolPage uses the parent’s SparkContext to access information about the pool and JobProgressListener for active stages in the pool (sorted by submissionTime in descending order by default).
Summary Table
The Summary table shows the details of a Schedulable pool.
Figure 2. Summary for production Pool
It uses the following columns:
-
Pool Name
-
Minimum Share
-
Pool Weight
-
Active Stages - the number of the active stages in a
Schedulablepool. -
Running Tasks
-
SchedulingMode
All the columns are the attributes of a Schedulable but the number of active stages which is calculated using the list of active stages of a pool (from the parent’s JobProgressListener).
Active Stages Table
The Active Stages table shows the active stages in a pool.
Figure 3. Active Stages for production Pool
It uses the following columns:
-
Stage Id
-
(optional) Pool Name - only available when in FAIR scheduling mode.
-
Description
-
Submitted
-
Duration
-
Tasks: Succeeded/Total
-
Input — Bytes and records read from Hadoop or from Spark storage.
-
Output — Bytes and records written to Hadoop.
-
Shuffle Read — Total shuffle bytes and records read (includes both data read locally and data read from remote executors).
-
Shuffle Write — Bytes and records written to disk in order to be read by a shuffle in a future stage.
The table uses JobProgressListener for information per stage in the pool.