replay(
logData: InputStream,
sourceName: String,
maybeTruncated: Boolean = false): Unit
ReplayListenerBus
ReplayListenerBus
is a custom SparkListenerBus that can replay JSON-encoded SparkListenerEvent
events.
Note
|
ReplayListenerBus is used in FsHistoryProvider.
|
Note
|
ReplayListenerBus is a private[spark] class in org.apache.spark.scheduler package.
|
Replaying JSON-encoded SparkListenerEvents from Stream — replay
Method
replay
reads JSON-encoded SparkListenerEvent events from logData
(one event per line) and posts them to all registered SparkListenerInterface listeners.
Note
|
replay uses jackson from json4s library to parse the AST for JSON.
|
When there is an exception parsing a JSON event, you may see the following WARN message in the logs (for the last line) or a JsonParseException
.
WARN Got JsonParseException from log file $sourceName at line [lineNumber], the file might not have finished writing cleanly.
Any other non-IO exceptions end up with the following ERROR messages in the logs:
ERROR Exception parsing Spark event log: [sourceName]
ERROR Malformed line #[lineNumber]: [currentLine]
Note
|
The sourceName input argument is only used for messages.
|