MapOutputTrackerMasterEndpoint

MapOutputTrackerMasterEndpoint is a RpcEndpoint for MapOutputTrackerMaster.

MapOutputTrackerMasterEndpoint handles the following messages:

Tip	Enable `INFO` or `DEBUG` logging levels for `org.apache.spark.MapOutputTrackerMasterEndpoint` logger to see what happens in `MapOutputTrackerMasterEndpoint`. Add the following line to `conf/log4j.properties`: `log4j.logger.org.apache.spark.MapOutputTrackerMasterEndpoint=DEBUG` Refer to Logging.

MapOutputTrackerMasterEndpoint takes the following when created:

When created, you should see the following DEBUG message in the logs:

DEBUG init

Note	`MapOutputTrackerMasterEndpoint` is created when `SparkEnv` is created for the driver and executors.

GetMapOutputStatuses(shuffleId: Int)
extends MapOutputTrackerMessage

When GetMapOutputStatuses arrives, MapOutputTrackerMasterEndpoint reads the host and the port of the sender.

You should see the following INFO message in the logs:

INFO Asked to send map output locations for shuffle [shuffleId] to [hostPort]

MapOutputTrackerMasterEndpoint posts a GetMapOutputMessage to MapOutputTrackerMaster (with shuffleId and the current RpcCallContext).

Note	`GetMapOutputStatuses` is posted when `MapOutputTracker` fetches shuffle map outputs remotely.

StopMapOutputTracker
extends MapOutputTrackerMessage

When StopMapOutputTracker arrives, you should see the following INFO message in the logs:

INFO MapOutputTrackerMasterEndpoint stopped!

MapOutputTrackerMasterEndpoint confirms the request (by replying true) and stops itself (and stops accepting messages).

Note	`StopMapOutputTracker` is posted when `MapOutputTrackerMaster` stops.

results matching ""