ReplayListenerBus

ReplayListenerBus is a custom SparkListenerBus that can replay JSON-encoded SparkListenerEvent events.

Note
ReplayListenerBus is used in FsHistoryProvider.
Note
ReplayListenerBus is a private[spark] class in org.apache.spark.scheduler package.

Replaying JSON-encoded SparkListenerEvents from Stream — replay Method

replay(
  logData: InputStream,
  sourceName: String,
  maybeTruncated: Boolean = false): Unit

replay reads JSON-encoded SparkListenerEvent events from logData (one event per line) and posts them to all registered SparkListenerInterface listeners.

Note
replay uses jackson from json4s library to parse the AST for JSON.

When there is an exception parsing a JSON event, you may see the following WARN message in the logs (for the last line) or a JsonParseException.

WARN Got JsonParseException from log file $sourceName at line [lineNumber], the file might not have finished writing cleanly.

Any other non-IO exceptions end up with the following ERROR messages in the logs:

ERROR Exception parsing Spark event log: [sourceName]
ERROR Malformed line #[lineNumber]: [currentLine]
Note
The sourceName input argument is only used for messages.

results matching ""

    No results matching ""