Stages for All Jobs Page
AllStagesPage is a web page (section) that is registered with the Stages tab that displays all stages in a Spark application - active, pending, completed, and failed stages with their count.
Figure 1. Stages Tab in web UI for FAIR scheduling mode (with pools only)
In FAIR scheduling mode you have access to the table showing the scheduler pools as well as the pool names per stage.
|
Note
|
Pool names are calculated using SparkContext.getAllPools. |
Internally, AllStagesPage is a WebUIPage with access to the parent Stages tab and more importantly the JobProgressListener to have access to current state of the entire Spark application.
Rendering AllStagesPage (render method)
render(request: HttpServletRequest): Seq[Node]render generates a HTML page to display in a web browser.
It uses the parent’s JobProgressListener to know about:
-
active stages (as
activeStages) -
pending stages (as
pendingStages) -
completed stages (as
completedStages) -
failed stages (as
failedStages) -
the number of completed stages (as
numCompletedStages) -
the number of failed stages (as
numFailedStages)
|
Note
|
Stage information is available as StageInfo object. |
There are 4 different tables for the different states of stages - active, pending, completed, and failed. They are displayed only when there are stages in a given state.
Figure 2. Stages Tab in web UI for FAIR scheduling mode (with pools and stages)
You could also notice "retry" for stage when it was retried.
|
Caution
|
FIXME A screenshot |