BlockManagerSource — Metrics Source for BlockManager

BlockManagerSource is the metrics Source for BlockManager.

BlockManagerSource is registered under the name BlockManager (when SparkContext is created).

Table 1. BlockManagerSource’s Metrics
Name Type Description

memory.maxMem_MB

long

Requests BlockManagerMaster for storage status (for every BlockManager) and sums up their maximum memory limit.

memory.remainingMem_MB

long

Requests BlockManagerMaster for storage status (for every BlockManager) and sums up their memory remaining.

memory.memUsed_MB

long

Requests BlockManagerMaster for storage status (for every BlockManager) and sums up their memory used.

disk.diskSpaceUsed_MB

long

Requests BlockManagerMaster for storage status (for every BlockManager) and sums up their disk space used.

You can access the BlockManagerSource metrics using the web UI’s port (as spark.ui.port property).

$ http --follow http://localhost:4040/metrics/json \
  | jq '.gauges | keys | .[] | select(test(".driver.BlockManager"; "g"))'
"local-1488272192549.driver.BlockManager.disk.diskSpaceUsed_MB"
"local-1488272192549.driver.BlockManager.memory.maxMem_MB"
"local-1488272192549.driver.BlockManager.memory.memUsed_MB"
"local-1488272192549.driver.BlockManager.memory.remainingMem_MB"

results matching ""

    No results matching ""