log4j.logger.org.apache.spark.deploy.yarn.YarnSparkHadoopUtil=DEBUG
YarnSparkHadoopUtil
YarnSparkHadoopUtil
is…FIXME
YarnSparkHadoopUtil
can only be created when SPARK_YARN_MODE flag is enabled.
Note
|
YarnSparkHadoopUtil belongs to org.apache.spark.deploy.yarn package.
|
Tip
|
Enable Add the following line to Refer to Logging. |
startCredentialUpdater
Method
Caution
|
FIXME |
Getting YarnSparkHadoopUtil Instance — get
Method
Caution
|
FIXME |
addPathToEnvironment
Method
addPathToEnvironment(env: HashMap[String, String], key: String, value: String): Unit
Caution
|
FIXME |
startExecutorDelegationTokenRenewer
Caution
|
FIXME |
stopExecutorDelegationTokenRenewer
Caution
|
FIXME |
getApplicationAclsForYarn
Method
Caution
|
FIXME |
Resolving Environment Variable — expandEnvironment
Method
expandEnvironment(environment: Environment): String
expandEnvironment
resolves environment
variable using YARN’s Environment.$
or Environment.$$
methods (depending on the version of Hadoop used).
Computing YARN’s ContainerId — getContainerId
Method
getContainerId: ContainerId
getContainerId
is a private[spark]
method that gets YARN’s ContainerId
from the YARN environment variable ApplicationConstants.Environment.CONTAINER_ID
and converts it to the return object using YARN’s ConverterUtils.toContainerId
.
Calculating Initial Number of Executors — getInitialTargetExecutorNumber
Method
getInitialTargetExecutorNumber(conf: SparkConf, numExecutors: Int = 2): Int
getInitialTargetExecutorNumber
calculates the initial number of executors for Spark on YARN. It varies by whether dynamic allocation is enabled or not.
Note
|
The default number of executors (aka DEFAULT_NUMBER_EXECUTORS ) is 2 .
|
With dynamic allocation enabled, getInitialTargetExecutorNumber
is spark.dynamicAllocation.initialExecutors or spark.dynamicAllocation.minExecutors to fall back to 0
if the others are undefined.
With dynamic allocation disabled, getInitialTargetExecutorNumber
is the value of spark.executor.instances property or SPARK_EXECUTOR_INSTANCES
environment variable, or the default value (of the input parameter numExecutors
) 2
.
Note
|
getInitialTargetExecutorNumber is used to calculate totalExpectedExecutors to start Spark on YARN in client or cluster modes.
|