Spark Listeners — Intercepting Events from Spark Scheduler

SparkListener is a mechanism in Spark to intercept events from the Spark scheduler that are emitted over the course of execution of a Spark application.

SparkListener extends SparkListenerInterface with all the callback methods being no-op/do-nothing.

Spark relies on SparkListeners internally to manage communication between internal components in the distributed environment for a Spark application, e.g. web UI, event persistence (for History Server), dynamic allocation of executors, keeping track of executors (using HeartbeatReceiver) and others.

You can develop your own custom SparkListener and register it using SparkContext.addSparkListener method or spark.extraListeners Spark property.

With SparkListener you can focus on Spark events of your liking and process a subset of all scheduling events.

Tip	Developing a custom SparkListener is an excellent introduction to low-level details of Spark’s Execution Model. Check out the exercise Developing Custom SparkListener to monitor DAGScheduler in Scala.

Tip	Enable `INFO` logging level for `org.apache.spark.SparkContext` logger to see when custom Spark listeners are registered. `INFO SparkContext: Registered listener org.apache.spark.scheduler.StatsReportListener` See `SparkContext` — Entry Point to Spark (Core).

SparkListenerInterface — Internal Contract for Spark Listeners

SparkListenerInterface is an private[spark] contract for Spark listeners to intercept events from the Spark scheduler.

Note	SparkListener and SparkFirehoseListener Spark listeners are direct implementations of `SparkListenerInterface` contract to help developing more sophisticated Spark listeners.

Table 1. SparkListenerInterface Methods (in alphabetical order)
Method	Event	Reason
`onApplicationEnd`	`SparkListenerApplicationEnd`	`SparkContext` does `postApplicationEnd`
`onApplicationStart`	`SparkListenerApplicationStart`	`SparkContext` does `postApplicationStart`
`onBlockManagerAdded`	`SparkListenerBlockManagerAdded`	`BlockManagerMasterEndpoint` has registered a `BlockManager`.
`onBlockManagerRemoved`	`SparkListenerBlockManagerRemoved`	`BlockManagerMasterEndpoint` has removed a `BlockManager` (which is when…FIXME)
`onBlockUpdated`	`SparkListenerBlockUpdated`	`BlockManagerMasterEndpoint` receives a `UpdateBlockInfo` message (which is when a `BlockManager` reports a block status update to driver).
`onEnvironmentUpdate`	`SparkListenerEnvironmentUpdate`	`SparkContext` does `postEnvironmentUpdate`.
`onExecutorMetricsUpdate`	`SparkListenerExecutorMetricsUpdate`
`onExecutorAdded`	`SparkListenerExecutorAdded`	`DriverEndpoint` RPC endpoint (of `CoarseGrainedSchedulerBackend`) receives `RegisterExecutor` message, `MesosFineGrainedSchedulerBackend` does `resourceOffers`, and `LocalSchedulerBackendEndpoint` starts.
`onExecutorBlacklisted`	`SparkListenerExecutorBlacklisted`	FIXME
`onExecutorRemoved`	`SparkListenerExecutorRemoved`	`DriverEndpoint` RPC endpoint (of `CoarseGrainedSchedulerBackend`) does removeExecutor and `MesosFineGrainedSchedulerBackend` does `removeExecutor`.
`onExecutorUnblacklisted`	`SparkListenerExecutorUnblacklisted`	FIXME
`onJobEnd`	`SparkListenerJobEnd`	`DAGScheduler` does `cleanUpAfterSchedulerStop`, `handleTaskCompletion`, `failJobAndIndependentStages`, and markMapStageJobAsFinished.
`onJobStart`	`SparkListenerJobStart`	`DAGScheduler` handles JobSubmitted and MapStageSubmitted messages
`onNodeBlacklisted`	`SparkListenerNodeBlacklisted`	FIXME
`onNodeUnblacklisted`	`SparkListenerNodeUnblacklisted`	FIXME
`onStageCompleted`	`SparkListenerStageCompleted`	`DAGScheduler` marks a stage as finished.
`onStageSubmitted`	`SparkListenerStageSubmitted`	`DAGScheduler` submits the missing tasks of a stage (in a Spark job).
`onTaskEnd`	`SparkListenerTaskEnd`	`DAGScheduler` handles a task completion
`onTaskGettingResult`	`SparkListenerTaskGettingResult`	`DAGScheduler` handles `GettingResultEvent` event
`onTaskStart`	`SparkListenerTaskStart`	`DAGScheduler` is informed that a task is about to start.
`onUnpersistRDD`	`SparkListenerUnpersistRDD`	`SparkContext` unpersists an RDD, i.e. removes RDD blocks from `BlockManagerMaster` (that can be triggered explicitly or implicitly).
`onOtherEvent`	`SparkListenerEvent`

Built-In Spark Listeners

Table 2. Built-In Spark Listeners
Spark Listener	Description
EventLoggingListener	Logs JSON-encoded events to a file that can later be read by History Server
StatsReportListener
`SparkFirehoseListener`	Allows users to receive all SparkListenerEvent events by overriding the single `onEvent` method only.
ExecutorAllocationListener
HeartbeatReceiver
StreamingJobProgressListener
ExecutorsListener	Prepares information for Executors tab in web UI
StorageStatusListener, RDDOperationGraphListener, EnvironmentListener, BlockStatusListener and StorageListener	For web UI
`SpillListener`
`ApplicationEventListener`
StreamingQueryListenerBus
SQLListener / SQLHistoryListener	Support for History Server
StreamingListenerBus
JobProgressListener

SparkListener — Intercepting Events from Spark Scheduler

Spark Listeners — Intercepting Events from Spark Scheduler

SparkListenerInterface — Internal Contract for Spark Listeners

Built-In Spark Listeners

results matching ""

No results matching ""