INFO YarnClusterScheduler: Created YarnClusterScheduler
YarnClusterScheduler — TaskScheduler for Cluster Deploy Mode
YarnClusterScheduler
is the TaskScheduler for Spark on YARN in cluster deploy mode.
It is a custom YarnScheduler that makes sure that appropriate initialization of ApplicationMaster is performed, i.e. SparkContext is initialized and stopped.
While being created, you should see the following INFO message in the logs:
Tip
|
Enable Add the following line to
Refer to Logging. |
postStartHook
Callback
postStartHook
calls ApplicationMaster.sparkContextInitialized before the parent’s postStartHook
.
You should see the following INFO message in the logs:
INFO YarnClusterScheduler: YarnClusterScheduler.postStartHook done
Stopping YarnClusterScheduler (stop method)
stop
calls the parent’s stop
followed by ApplicationMaster.sparkContextStopped.