FsHistoryProvider

FsHistoryProvider is the default application history provider for HistoryServer. It uses SparkConf and Clock objects for its operation.

Tip

Enable INFO or DEBUG logging levels for org.apache.spark.deploy.history.FsHistoryProvider logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.deploy.history.FsHistoryProvider=DEBUG

Refer to Logging.

ApplicationHistoryProvider

ApplicationHistoryProvider tracks the history of Spark applications with their Spark UIs. It can be stopped and write events to a stream.

It is an abstract class.

ApplicationHistoryProvider Contract

Every ApplicationHistoryProvider offers the following:

  • getListing to return a list of all known applications.

    getListing(): Iterable[ApplicationHistoryInfo]
  • getAppUI to return Spark UI for an application.

    getAppUI(appId: String, attemptId: Option[String]): Option[LoadedAppUI]
  • stop to stop the instance.

    stop(): Unit
  • getConfig to return configuration of…​FIXME

    getConfig(): Map[String, String] = Map()
  • writeEventLogs to write events to a stream.

    writeEventLogs(appId: String, attemptId: Option[String], zipStream: ZipOutputStream): Unit

results matching ""

    No results matching ""