YarnRMClient

YarnRMClient is responsible for registering and unregistering a Spark application (in the form of ApplicationMaster) with YARN ResourceManager (and hence RM in the name).

spark yarn YarnRMClient register.png
Figure 1. Registering ApplicationMaster with YARN ResourceManager

YarnRMClient is just a wrapper for AMRMClient[ContainerRequest] that is started when registering ApplicationMaster (and never stopped explicitly!).

YarnRMClient tracks the application attempt identifiers and the maximum number of attempts to register ApplicationMaster.

Table 1. YarnRMClient’s Internal Registries and Counters
Name Description

amClient

AMRMClient using ContainerRequest for YARN ResourceManager.

Created (initialized and started) when YarnRMClient registers ApplicationMaster.

Used when YarnRMClient creates a YarnAllocator (after registering ApplicationMaster) and to unregister ApplicationMaster.

uiHistoryAddress

registered

Flag to say whether YarnRMClient is connected to YARN ResourceManager (i.e. true) or not. Disabled by default. Used when unregistering ApplicationMaster.
Tip

Enable INFO logging level for org.apache.spark.deploy.yarn.YarnRMClient logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.deploy.yarn.YarnRMClient=INFO

Refer to Logging.

Registering ApplicationMaster with YARN ResourceManager (and Creating YarnAllocator) — register Method

register(
  driverUrl: String,
  driverRef: RpcEndpointRef,
  conf: YarnConfiguration,
  sparkConf: SparkConf,
  uiAddress: String,
  uiHistoryAddress: String,
  securityMgr: SecurityManager,
  localResources: Map[String, LocalResource]): YarnAllocator

register creates a AMRMClient, initializes it (using the input YarnConfiguration) and starts immediately.

Note
 AMRMClient is used in YARN to register an application’s ApplicationMaster with the YARN ResourceManager.
Tip

register connects to YARN ResourceManager using the following design pattern:

val amClient: AMRMClient[ContainerRequest] = AMRMClient.createAMRMClient()
amClient.init(conf)
amClient.start()

register saves the input uiHistoryAddress as uiHistoryAddress.

You should see the following INFO message in the logs (in stderr in YARN):

INFO YarnRMClient: Registering the ApplicationMaster

register then uses AMRMClient to register the Spark application’s ApplicationMaster (using the local hostname, the port 0 and the input uiAddress).

Note
 The input uiAddress is the web UI of the Spark application and is specified using the SparkContext (when the application runs in cluster deploy mode) or using spark.driver.appUIAddress property.

registered flag is enabled.

In the end, register creates a new YarnAllocator (using the input parameters of register and the internal AMRMClient).

Note
 register is used exclusively when ApplicationMaster registers itself with the YARN ResourceManager.

Unregistering ApplicationMaster from YARN ResourceManager — unregister Method

unregister(status: FinalApplicationStatus, diagnostics: String = ""): Unit

unregister unregisters the ApplicationMaster of a Spark application.

It basically checks that ApplicationMaster is registered and only when it is requests the internal AMRMClient to unregister.

unregister is called when ApplicationMaster wants to unregister.

Maximum Number of Attempts to Register ApplicationMaster — getMaxRegAttempts Method

getMaxRegAttempts(sparkConf: SparkConf, yarnConf: YarnConfiguration): Int

getMaxRegAttempts uses SparkConf and YARN’s YarnConfiguration to read configuration settings and return the maximum number of application attempts before ApplicationMaster registration with YARN is considered unsuccessful (and so the Spark application).

It reads YARN’s yarn.resourcemanager.am.max-attempts (available as YarnConfiguration.RM_AM_MAX_ATTEMPTS) or falls back to YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS (which is 2).

The return value is the minimum of the configuration settings of YARN and Spark.

Getting ApplicationAttemptId of Spark Application — getAttemptId Method

getAttemptId(): ApplicationAttemptId

getAttemptId returns YARN’s ApplicationAttemptId (of the Spark application to which the container was assigned).

Internally, it uses YARN-specific methods like ConverterUtils.toContainerId and ContainerId.getApplicationAttemptId.

getAmIpFilterParams Method

Caution
 FIXME

results matching ""

    No results matching ""