SparkListenerBus — Internal Contract for Spark Event Buses

SparkListenerBus is a private[spark] ListenerBus for SparkListenerInterface listeners that process SparkListenerEvent events.

SparkListenerBus comes with a custom doPostEvent method that simply relays SparkListenerEvent events to appropriate SparkListenerInterface methods.

Note	There are two custom `SparkListenerBus` listeners: LiveListenerBus and ReplayListenerBus.

Table 1. SparkListenerEvent to SparkListenerInterface’s Method "mapping"
SparkListenerEvent	SparkListenerInterface’s Method
`SparkListenerStageSubmitted`	`onStageSubmitted`
`SparkListenerStageCompleted`	`onStageCompleted`
`SparkListenerJobStart`	`onJobStart`
`SparkListenerJobEnd`	`onJobEnd`
`SparkListenerJobEnd`	`onJobEnd`
`SparkListenerTaskStart`	`onTaskStart`
`SparkListenerTaskGettingResult`	`onTaskGettingResult`
SparkListenerTaskEnd	`onTaskEnd`
`SparkListenerEnvironmentUpdate`	`onEnvironmentUpdate`
`SparkListenerBlockManagerAdded`	`onBlockManagerAdded`
`SparkListenerBlockManagerRemoved`	`onBlockManagerRemoved`
`SparkListenerUnpersistRDD`	`onUnpersistRDD`
`SparkListenerApplicationStart`	`onApplicationStart`
`SparkListenerApplicationEnd`	`onApplicationEnd`
`SparkListenerExecutorMetricsUpdate`	`onExecutorMetricsUpdate`
`SparkListenerExecutorAdded`	`onExecutorAdded`
`SparkListenerExecutorRemoved`	`onExecutorRemoved`
`SparkListenerBlockUpdated`	`onBlockUpdated`
`SparkListenerLogStart`	event ignored
other event types	`onOtherEvent`

`ListenerBus` Event Bus Contract

ListenerBus[L <: AnyRef, E]

ListenerBus is an event bus that post events (of type E) to all registered listeners (of type L).

It manages listeners of type L, i.e. it can add to and remove listeners from an internal listeners collection.

addListener(listener: L): Unit
removeListener(listener: L): Unit

It can post events of type E to all registered listeners (using postToAll method). It simply iterates over the internal listeners collection and executes the abstract doPostEvent method.

doPostEvent(listener: L, event: E): Unit

Note	`doPostEvent` is provided by more specialized `ListenerBus` event buses.

In case of exception while posting an event to a listener you should see the following ERROR message in the logs and the exception.

ERROR Listener [listener] threw an exception

Note	There are three custom `ListenerBus` listeners: SparkListenerBus, StreamingQueryListenerBus, and StreamingListenerBus.

Tip	Enable `ERROR` logging level for `org.apache.spark.util.ListenerBus` logger to see what happens inside. Add the following line to `conf/log4j.properties`: `log4j.logger.org.apache.spark.util.ListenerBus=ERROR` Refer to Logging.

SparkListenerBus — Internal Contract for Spark Event Buses