log4j.logger.org.apache.spark.deploy.history.FsHistoryProvider=DEBUG
FsHistoryProvider
FsHistoryProvider
is the default application history provider for HistoryServer. It uses SparkConf and Clock
objects for its operation.
Tip
|
Enable Add the following line to Refer to Logging. |
ApplicationHistoryProvider
ApplicationHistoryProvider
tracks the history of Spark applications with their Spark UIs. It can be stopped and write events to a stream.
It is an abstract class.
ApplicationHistoryProvider Contract
Every ApplicationHistoryProvider
offers the following:
-
getListing
to return a list of all known applications.getListing(): Iterable[ApplicationHistoryInfo]
-
getAppUI
to return Spark UI for an application.getAppUI(appId: String, attemptId: Option[String]): Option[LoadedAppUI]
-
stop
to stop the instance.stop(): Unit
-
getConfig
to return configuration of…FIXMEgetConfig(): Map[String, String] = Map()
-
writeEventLogs
to write events to a stream.writeEventLogs(appId: String, attemptId: Option[String], zipStream: ZipOutputStream): Unit