Powered by GitBook

ExecutorAllocationClient

ExecutorAllocationClient is a contract for clients to communicate with a cluster manager to request or kill executors.

ExecutorAllocationClient Contract

trait ExecutorAllocationClient {
  def getExecutorIds(): Seq[String]
  def requestTotalExecutors(numExecutors: Int, localityAwareTasks: Int, hostToLocalTaskCount: Map[String, Int]): Boolean
  def requestExecutors(numAdditionalExecutors: Int): Boolean
  def killExecutor(executorId: String): Boolean
  def killExecutors(executorIds: Seq[String]): Seq[String]
  def killExecutorsOnHost(host: String): Boolean
}

Note	`ExecutorAllocationClient` is a `private[spark]` contract.

Table 1. ExecutorAllocationClient Contract
Method	Description
`getExecutorIds`	Finds identifiers of the executors in use. Used when `SparkContext` calculates the executors in use and also when Spark Streaming manages executors.
`requestTotalExecutors`	Updates the cluster manager with the exact number of executors desired. It returns whether the request has been acknowledged by the cluster manager (`true`) or not (`false`). Used when: `SparkContext` requests executors (for coarse-grained scheduler backends only). `ExecutorAllocationManager` starts, does updateAndSyncNumExecutorsTarget, and addExecutors. Streaming `ExecutorAllocationManager` requests executors. `YarnSchedulerBackend` stops.
`requestExecutors`	Requests additional executors from a cluster manager and returns whether the request has been acknowledged by the cluster manager (`true`) or not (`false`). Used when `SparkContext` requests additional executors (for coarse-grained scheduler backends only).
`killExecutor`	Requests a cluster manager to kill a single executor that is no longer in use and returns whether the request has been acknowledged by the cluster manager (`true`) or not (`false`). The default implementation simply calls killExecutors (with a single-element collection of executors to kill). Used when: `ExecutorAllocationManager` removes an executor. `SparkContext` is requested to kill executors. Streaming `ExecutorAllocationManager` is requested to kill executors.
`killExecutors`	Requests that a cluster manager to kill one or many executors that are no longer in use and returns whether the request has been acknowledged by the cluster manager (`true`) or not (`false`). Interestingly, it is only used for killExecutor.
`killExecutorsOnHost`	Used exclusively when `BlacklistTracker` kills blacklisted executors.

results matching ""

No results matching ""