YarnSparkHadoopUtil

YarnSparkHadoopUtil is…​FIXME

YarnSparkHadoopUtil can only be created when SPARK_YARN_MODE flag is enabled.

Note
YarnSparkHadoopUtil belongs to org.apache.spark.deploy.yarn package.
Tip

Enable DEBUG logging level for org.apache.spark.deploy.yarn.YarnSparkHadoopUtil logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.deploy.yarn.YarnSparkHadoopUtil=DEBUG

Refer to Logging.

startCredentialUpdater Method

Caution
FIXME

Getting YarnSparkHadoopUtil Instance — get Method

Caution
FIXME

addPathToEnvironment Method

addPathToEnvironment(env: HashMap[String, String], key: String, value: String): Unit
Caution
FIXME

startExecutorDelegationTokenRenewer

Caution
FIXME

stopExecutorDelegationTokenRenewer

Caution
FIXME

getApplicationAclsForYarn Method

Caution
FIXME

MEMORY_OVERHEAD_FACTOR

MEMORY_OVERHEAD_FACTOR is a constant that equals to 10% for memory overhead.

MEMORY_OVERHEAD_MIN

MEMORY_OVERHEAD_MIN is a constant that equals to 384L for memory overhead.

Resolving Environment Variable — expandEnvironment Method

expandEnvironment(environment: Environment): String

expandEnvironment resolves environment variable using YARN’s Environment.$ or Environment.$$ methods (depending on the version of Hadoop used).

Computing YARN’s ContainerId — getContainerId Method

getContainerId: ContainerId

getContainerId is a private[spark] method that gets YARN’s ContainerId from the YARN environment variable ApplicationConstants.Environment.CONTAINER_ID and converts it to the return object using YARN’s ConverterUtils.toContainerId.

Calculating Initial Number of Executors — getInitialTargetExecutorNumber Method

getInitialTargetExecutorNumber(conf: SparkConf, numExecutors: Int = 2): Int

getInitialTargetExecutorNumber calculates the initial number of executors for Spark on YARN. It varies by whether dynamic allocation is enabled or not.

Note
The default number of executors (aka DEFAULT_NUMBER_EXECUTORS) is 2.

With dynamic allocation enabled, getInitialTargetExecutorNumber is spark.dynamicAllocation.initialExecutors or spark.dynamicAllocation.minExecutors to fall back to 0 if the others are undefined.

With dynamic allocation disabled, getInitialTargetExecutorNumber is the value of spark.executor.instances property or SPARK_EXECUTOR_INSTANCES environment variable, or the default value (of the input parameter numExecutors) 2.

Note
getInitialTargetExecutorNumber is used to calculate totalExpectedExecutors to start Spark on YARN in client or cluster modes.

results matching ""

    No results matching ""