SQLHistoryListener

SQLHistoryListener is a custom SQLListener for History Server. It attaches SQL tab to History Server’s web UI only when the first SparkListenerSQLExecutionStart arrives and shuts onExecutorMetricsUpdate off. It also handles ends of tasks in a slightly different way.

Note
Support for SQL UI in History Server was added in SPARK-11206 Support SQL UI on the history server.
Caution
FIXME Add the link to the JIRA.

onOtherEvent

onOtherEvent(event: SparkListenerEvent): Unit

When SparkListenerSQLExecutionStart event comes, onOtherEvent attaches SQL tab to web UI and passes the call to the parent SQLListener.

onTaskEnd

Caution
FIXME

Creating SQLHistoryListener Instance

SQLHistoryListener is created using a (private[sql]) SQLHistoryListenerFactory class (which is SparkHistoryListenerFactory).

The SQLHistoryListenerFactory class is registered when SparkUI creates a web UI for History Server as a Java service in META-INF/services/org.apache.spark.scheduler.SparkHistoryListenerFactory:

org.apache.spark.sql.execution.ui.SQLHistoryListenerFactory
Note
Loading the service uses Java’s ServiceLoader.load method.

onExecutorMetricsUpdate

onExecutorMetricsUpdate does nothing.

results matching ""

    No results matching ""