YarnClusterManager — ExternalClusterManager for YARN

YarnClusterManager is the only currently known ExternalClusterManager in Spark. It creates a TaskScheduler and a SchedulerBackend for YARN.

canCreate Method

YarnClusterManager can handle the yarn master URL only.

createTaskScheduler Method

createTaskScheduler creates a YarnClusterScheduler for cluster deploy mode and a YarnScheduler for client deploy mode.

It throws a SparkException for unknown deploy modes.

Unknown deploy mode '[deployMode]' for Yarn

createSchedulerBackend Method

createSchedulerBackend creates a YarnClusterSchedulerBackend for cluster deploy mode and a YarnClientSchedulerBackend for client deploy mode.

It throws a SparkException for unknown deploy modes.

Unknown deploy mode '[deployMode]' for Yarn

Initializing YarnClusterManager — initialize Method

initialize simply initializes the input TaskSchedulerImpl.

results matching ""

    No results matching ""