LocalSchedulerBackend
LocalSchedulerBackend is a scheduler backend and a ExecutorBackend for Spark local run mode.
LocalSchedulerBackend acts as a "cluster manager" for local mode to offer resources on the single worker it manages, i.e. it calls TaskSchedulerImpl.resourceOffers(offers) with offers being a single-element collection with WorkerOffer.
|
Note
|
WorkerOffer represents a resource offer with CPU cores available on an executor. |
When an executor sends task status updates (using ExecutorBackend.statusUpdate), they are passed along as StatusUpdate to LocalEndpoint.
Figure 1. Task status updates flow in local mode
When LocalSchedulerBackend starts up, it registers a new RpcEndpoint called LocalSchedulerBackendEndpoint that is backed by LocalEndpoint. This is announced on LiveListenerBus as driver (using SparkListenerExecutorAdded message).
The application ids are in the format of local-[current time millis].
It communicates with LocalEndpoint using RPC messages.
The default parallelism is controlled using spark.default.parallelism property.