Executors Tab

Executors tab in web UI shows …​

spark webui executors.png
Figure 1. Executors Tab in web UI (local mode)

ExecutorsTab uses ExecutorsListener to collect information about executors in a Spark application.

The title of the tab is Executors.

You can access the Executors tab under /executors URL, e.g. http://localhost:4040/executors.

Note

What’s interesting in how Storage Memory is displayed in the Executors tab is that the default UnifiedMemoryManager calculates the maximum memory in a way that is different from what the page displays (using the custom JavaScript formatBytes function in utils.js).

// local mode with spark.driver.memory 2g
// ./bin/spark-shell --conf spark.driver.memory=2g
// UnifiedMemoryManager reports 912MB
// You can see it after enabling INFO messages for BlockManagerMasterEndpoint

INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.8:54503 with 912.3 MB RAM, BlockManagerId(driver, 192.168.1.8, 54503, None)

// custom JavaScript `formatBytes` function (from utils.js) reports...956.6MB
// See https://github.com/apache/spark/blob/master/core/src/main/resources/org/apache/spark/ui/static/utils.js#L40-L48
def formatBytes(bytes: Double) = {
  val k = 1000
  val i = math.floor(math.log(bytes) / math.log(k))
  val maxMemoryWebUI = bytes / math.pow(k, i)
  f"$maxMemoryWebUI%1.1f"
}
scala> println(formatBytes(maxMemory))
956.6

ExecutorsPage

ExecutorsPage is a WebUIPage.

Caution
 FIXME

getExecInfo Method

getExecInfo(
  listener: ExecutorsListener,
  statusId: Int,
  isActive: Boolean): ExecutorSummary

getExecInfo creates a ExecutorSummary.

Caution
 FIXME
Note
 getExecInfo is used when…​FIXME

ExecutorThreadDumpPage

ExecutorThreadDumpPage is enabled or disabled using spark.ui.threadDumpsEnabled setting.

Settings

spark.ui.threadDumpsEnabled

spark.ui.threadDumpsEnabled (default: true) is to enable (true) or disable (false) ExecutorThreadDumpPage.

results matching ""

    No results matching ""