Web UI — Spark Application’s Web Console

Web UI (aka Application UI or webUI or Spark UI) is the web interface of a running Spark application to monitor and inspect Spark job executions in a web browser.

spark webui jobs.png
Figure 1. Welcome page - Jobs page

Every SparkContext launches its own instance of Web UI which is available at http://[driver]:4040 by default (the port can be changed using spark.ui.port setting) and will increase if this port is already taken (until an open port is found).

web UI comes with the following tabs (which may not all be visible at once as they are lazily created on demand, e.g. Streaming tab):

  1. Jobs

  2. Stages

  3. Storage with RDD size and memory use

  4. Environment

  5. Executors

  6. SQL

Tip
 You can use the web UI after the application has finished by persisting events using EventLoggingListener and using Spark History Server.
Note
 All the information that is displayed in web UI is available thanks to JobProgressListener and other SparkListeners. One could say that web UI is a web layer to Spark listeners.

Settings

Table 1. Spark Properties
Spark Property Default Value Description

spark.ui.enabled

true

The flag to control whether the web UI is started (true) or not (false).

spark.ui.port

4040

The port web UI binds to.

If multiple SparkContexts attempt to run on the same host (it is not possible to have two or more Spark contexts on a single JVM, though), they will bind to successive ports beginning with spark.ui.port.

spark.ui.killEnabled

true

The flag to control whether you can kill stages in web UI (true) or not (false).

spark.ui.retainedDeadExecutors

100

The maximum number of entries in executorToTaskSummary (in ExecutorsListener) and deadExecutorStorageStatus (in StorageStatusListener) internal registries.

results matching ""

    No results matching ""