List<String> buildJavaCommand(String extraClassPath)
AbstractCommandBuilder
AbstractCommandBuilder
is the base command builder for SparkSubmitCommandBuilder and SparkClassCommandBuilder
specialized command builders.
AbstractCommandBuilder
expects that command builders define buildCommand
.
Method | Description |
---|---|
|
The only abstract method that subclasses have to define. |
Loads the configuration file for a Spark application, be it the user-specified properties file or |
buildJavaCommand
Internal Method
buildJavaCommand
builds the Java command for a Spark application (which is a collection of elements with the path to java
executable, JVM options from java-opts
file, and a class path).
If javaHome
is set, buildJavaCommand
adds [javaHome]/bin/java
to the result Java command. Otherwise, it uses JAVA_HOME
or, when no earlier checks succeeded, falls through to java.home
Java’s system property.
Caution
|
FIXME Who sets javaHome internal property and when?
|
buildJavaCommand
loads extra Java options from the java-opts
file in configuration directory if the file exists and adds them to the result Java command.
Eventually, buildJavaCommand
builds the class path (with the extra class path if non-empty) and adds it as -cp
to the result Java command.
buildClassPath
method
List<String> buildClassPath(String appClassPath)
buildClassPath
builds the classpath for a Spark application.
Note
|
Directories always end up with the OS-specific file separator at the end of their paths. |
buildClassPath
adds the following in that order:
-
SPARK_CLASSPATH
environment variable -
The input
appClassPath
-
(only with
SPARK_PREPEND_CLASSES
set orSPARK_TESTING
being1
) Locally compiled Spark classes inclasses
,test-classes
and Core’s jars.CautionFIXME Elaborate on "locally compiled Spark classes". -
(only with
SPARK_SQL_TESTING
being1
) …CautionFIXME Elaborate on the SQL testing case -
HADOOP_CONF_DIR
environment variable -
YARN_CONF_DIR
environment variable -
SPARK_DIST_CLASSPATH
environment variable
Note
|
childEnv is queried first before System properties. It is always empty for AbstractCommandBuilder (and SparkSubmitCommandBuilder , too).
|
Loading Properties File — loadPropertiesFile
Internal Method
Properties loadPropertiesFile()
loadPropertiesFile
is a part of AbstractCommandBuilder
private API that loads Spark settings from a properties file (when specified on the command line) or spark-defaults.conf in the configuration directory.
It loads the settings from the following files starting from the first and checking every location until the first properties file is found:
-
propertiesFile
(if specified using--properties-file
command-line option or set byAbstractCommandBuilder.setPropertiesFile
). -
[SPARK_CONF_DIR]/spark-defaults.conf
-
[SPARK_HOME]/conf/spark-defaults.conf
Note
|
loadPropertiesFile reads a properties file using UTF-8 .
|
Spark’s Configuration Directory — getConfDir
Internal Method
AbstractCommandBuilder
uses getConfDir
to compute the current configuration directory of a Spark application.
It uses SPARK_CONF_DIR
(from childEnv
which is always empty anyway or as a environment variable) and falls through to [SPARK_HOME]/conf
(with SPARK_HOME
from getSparkHome
internal method).
Spark’s Home Directory — getSparkHome
Internal Method
AbstractCommandBuilder
uses getSparkHome
to compute Spark’s home directory for a Spark application.
It uses SPARK_HOME
(from childEnv
which is always empty anyway or as a environment variable).
If SPARK_HOME
is not set, Spark throws a IllegalStateException
:
Spark home not found; set it explicitly or use the SPARK_HOME environment variable.