TaskMetrics

TaskMetrics is a collection of metrics tracked during execution of a Task.

TaskMetrics uses accumulators to represent the metrics and offers "increment" methods to increment them.

Note
 The local values of the accumulators for a task (as accumulated while the task runs) are sent from the executor to the driver when the task completes (and DAGScheduler re-creates TaskMetrics).
Table 1. Metrics
Property Name Type Description

_memoryBytesSpilled

internal.metrics.memoryBytesSpilled

LongAccumulator

Used in memoryBytesSpilled, incMemoryBytesSpilled

_updatedBlockStatuses

internal.metrics.updatedBlockStatuses

CollectionAccumulator[(BlockId, BlockStatus)]

Used in updatedBlockStatuses, recording updated BlockStatus for a Block, setUpdatedBlockStatuses
Table 2. TaskMetrics’s Internal Registries and Counters
Name Description

nameToAccums

Internal accumulators indexed by their names.

Used when TaskMetrics re-creates TaskMetrics from AccumulatorV2s, …​FIXME

NOTE: nameToAccums is a transient and lazy value.

internalAccums

Collection of internal AccumulatorV2 objects.

Used when…​FIXME

NOTE: internalAccums is a transient and lazy value.

externalAccums

Collection of external AccumulatorV2 objects.

Used when TaskMetrics re-creates TaskMetrics from AccumulatorV2s, …​FIXME

NOTE: externalAccums is a transient and lazy value.

accumulators Method

Caution
 FIXME

mergeShuffleReadMetrics Method

Caution
 FIXME

memoryBytesSpilled Method

Caution
 FIXME

updatedBlockStatuses Method

Caution
 FIXME

setExecutorCpuTime Method

Caution
 FIXME

setResultSerializationTime Method

Caution
 FIXME

setJvmGCTime Method

Caution
 FIXME

setExecutorRunTime Method

Caution
 FIXME

setExecutorDeserializeCpuTime Method

Caution
 FIXME

setExecutorDeserializeTime Method

Caution
 FIXME

setUpdatedBlockStatuses Method

Caution
 FIXME

Re-Creating TaskMetrics From AccumulatorV2s — fromAccumulators Method

fromAccumulators(accums: Seq[AccumulatorV2[_, _]]): TaskMetrics

fromAccumulators creates a new TaskMetrics and registers accums as internal and external task metrics (using nameToAccums internal registry).

Internally, fromAccumulators creates a new TaskMetrics. It then splits accums into internal and external task metrics collections (using nameToAccums internal registry).

For every internal task metrics, fromAccumulators finds the metrics in nameToAccums internal registry (of the new TaskMetrics instance), copies metadata, and merges state.

In the end, fromAccumulators adds the external accumulators to the new TaskMetrics instance.

Note
 fromAccumulators is used exclusively when DAGScheduler gets notified that a task has finished (and re-creates TaskMetrics).

Recording Memory Bytes Spilled — incMemoryBytesSpilled Method

incMemoryBytesSpilled(v: Long): Unit

incMemoryBytesSpilled adds v to _memoryBytesSpilled task metrics.

Note

incMemoryBytesSpilled is used when:

  1. Aggregator updates task metrics

  2. CoGroupedRDD computes a Partition

  3. BlockStoreShuffleReader reads combined key-value records for a reduce task

  4. ShuffleExternalSorter frees execution memory by spilling to disk

  5. ExternalSorter writes the records into a temporary partitioned file in the disk store

  6. UnsafeExternalSorter spills current records due to memory pressure

  7. SpillableIterator spills records to disk

  8. JsonProtocol creates TaskMetrics from JSON

Recording Updated BlockStatus For Block — incUpdatedBlockStatuses Method

incUpdatedBlockStatuses(v: (BlockId, BlockStatus)): Unit

incUpdatedBlockStatuses adds v in _updatedBlockStatuses internal registry.

Note
 incUpdatedBlockStatuses is used exclusively when BlockManager does addUpdatedBlockStatusToTaskMetrics.

Registering Internal Accumulators — register Method

register(sc: SparkContext): Unit

register registers the internal accumulators (from nameToAccums internal registry) with countFailedValues enabled (true).

Note
 register is used exclusively when Stage is requested for its new attempt.

results matching ""

    No results matching ""