log4j.logger.org.apache.spark.ui.SparkUI=INFO
SparkUI
SparkUI
represents the web UI for a Spark application and Spark History Server. It is created and bound when SparkContext
is created (with spark.ui.enabled enabled).
Note
|
The only difference between SparkUI for a Spark application and Spark History Server is that…FIXME
|
When started, SparkUI
binds to appUIAddress address that you can control using SPARK_PUBLIC_DNS
environment variable or spark.driver.host Spark property.
Name | Description |
---|---|
Tip
|
Enable Add the following line to Refer to Logging. |
attachTab
Method
Caution
|
FIXME |
Creating SparkUI
Instance
class SparkUI (
val sc: Option[SparkContext],
val conf: SparkConf,
securityManager: SecurityManager,
val environmentListener: EnvironmentListener,
val storageStatusListener: StorageStatusListener,
val executorsListener: ExecutorsListener,
val jobProgressListener: JobProgressListener,
val storageListener: StorageListener,
val operationGraphListener: RDDOperationGraphListener,
var appName: String,
val basePath: String,
val startTime: Long)
extends WebUI(securityManager,
securityManager.getSSLOptions("ui"), SparkUI.getUIPort(conf),
conf, basePath, "SparkUI")
When executed, SparkUI
creates a StagesTab and initializes the tabs and handlers in web UI.
Note
|
SparkUI is created when SparkContext is created (with spark.ui.enabled enabled). SparkUI gets the references to the owning SparkContext and the other properties, i.e. SparkConf, LiveListenerBus Event Bus, JobProgressListener, SecurityManager , appName , and startTime .
|
Assigning Unique Identifier of Spark Application — setAppId
Method
setAppId(id: String): Unit
setAppId
sets the internal appId.
Note
|
setAppId is used exclusively when SparkContext is initialized.
|
Attaching Tabs and Context Handlers — initialize
Method
initialize(): Unit
initialize
attaches the following tabs:
initialize
also attaches ServletContextHandler
handlers:
-
/static
to serve static files fromorg/apache/spark/ui/static
directory (on CLASSPATH). -
Redirecting
/
to/jobs/
(so Jobs tab is the first tab when you open web UI). -
Serving
/api
context path (withorg.apache.spark.status.api.v1
provider package) using ApiRootResource. -
Redirecting
/stages/stage/kill
to/stages/
Note
|
initialize is a part of the WebUI Contract and is executed when SparkUI is created.
|
Stopping SparkUI
— stop
Method
stop(): Unit
stop
stops the HTTP server and prints the following INFO message to the logs:
INFO SparkUI: Stopped Spark web UI at [appUIAddress]
Note
|
appUIAddress in the above INFO message is the result of appUIAddress method.
|
appUIAddress
Method
appUIAddress: String
appUIAddress
returns the entire URL of a Spark application’s web UI, including http://
scheme.
Internally, appUIAddress
uses appUIHostPort.
getSparkUser
Method
getSparkUser: String
getSparkUser
returns the name of the user a Spark application runs as.
Internally, getSparkUser
requests user.name
System property from EnvironmentListener Spark listener.
Note
|
getSparkUser is only used to display the user name in Spark Jobs page
|
createLiveUI
Method
createLiveUI(
sc: SparkContext,
conf: SparkConf,
listenerBus: SparkListenerBus,
jobProgressListener: JobProgressListener,
securityManager: SecurityManager,
appName: String,
startTime: Long): SparkUI
createLiveUI
creates a SparkUI
for a live running Spark application.
Internally, createLiveUI
simply forwards the call to create.
Note
|
createLiveUI is called when SparkContext is created (and spark.ui.enabled is enabled).
|
createHistoryUI
Method
Caution
|
FIXME |
create
Factory Method
create(
sc: Option[SparkContext],
conf: SparkConf,
listenerBus: SparkListenerBus,
securityManager: SecurityManager,
appName: String,
basePath: String = "",
jobProgressListener: Option[JobProgressListener] = None,
startTime: Long): SparkUI
create
creates a SparkUI
and is responsible for registering SparkListeners for SparkUI
.
Note
|
create creates a web UI for a running Spark application and Spark History Server.
|
Internally, create
registers the following SparkListeners with the input listenerBus
.
create
then creates a SparkUI
.
appUIHostPort
Method
appUIHostPort: String
appUIHostPort
returns the Spark application’s web UI which is the public hostname and port, excluding the scheme.
Note
|
appUIAddress uses appUIHostPort and adds http:// scheme.
|
getAppName
Method
getAppName: String
getAppName
returns the name of the Spark application (of a SparkUI
instance).
Note
|
getAppName is used when SparkUITab is requested the application’s name.
|
SparkUITab
— Custom WebUITab
SparkUITab
is a private[spark]
custom WebUITab
that defines one method only, i.e. appName
.
appName: String
appName
returns the application’s name.