NettyBlockRpcServer

NettyBlockRpcServer is a RpcHandler (i.e. a handler for sendRPC() messages sent by TransportClients) that handles BlockTransferMessage messages for NettyBlockTransferService.

NettyBlockRpcServer uses OneForOneStreamManager as the internal StreamManager.

Table 1. `NettyBlockRpcServer` Messages
Message	Behaviour
OpenBlocks	Obtaining local blocks and registering them with the internal OneForOneStreamManager.
UploadBlock	Deserializes a block and stores it in BlockDataManager.

Tip	Enable `TRACE` logging level to see received messages in the logs.

Tip	Enable `TRACE` logging level for `org.apache.spark.network.netty.NettyBlockRpcServer` logger to see what happens inside. Add the following line to `conf/log4j.properties`: `log4j.logger.org.apache.spark.network.netty.NettyBlockRpcServer=TRACE` Refer to Logging.

Creating NettyBlockRpcServer Instance

class NettyBlockRpcServer(
  appId: String,
  serializer: Serializer,
  blockManager: BlockDataManager)
extends RpcHandler

When created, NettyBlockRpcServer gets the application id (appId) and a Serializer and a BlockDataManager.

Note	`NettyBlockRpcServer` is created when `NettyBlockTransferService` is initialized.

NettyBlockRpcServer merely creates the internal instance of OneForOneStreamManager.

Note	As a `RpcHandler`, `NettyBlockRpcServer` uses the `OneForOneStreamManager` for `getStreamManager` (which is a part of the `RpcHandler` contract).

Obtaining Local Blocks and Registering with Internal `OneForOneStreamManager` — `OpenBlocks` Message Handler

When OpenBlocks arrives, NettyBlockRpcServer requests block data (from BlockDataManager) for every block id in the message. The block data is a collection of ManagedBuffer for every block id in the incoming message.

Note	`BlockDataManager` is given when `NettyBlockRpcServer` is created.

NettyBlockRpcServer then registers a stream of ManagedBuffers (for the blocks) with the internal StreamManager under streamId.

Note	The internal `StreamManager` is OneForOneStreamManager and is created when `NettyBlockRpcServer` is created.

You should see the following TRACE message in the logs:

TRACE NettyBlockRpcServer: Registered streamId [streamId]  with [size] buffers

In the end, NettyBlockRpcServer responds with a StreamHandle (with the streamId and the number of blocks). The response is serialized as a ByteBuffer.

Deserializing Block and Storing in `BlockDataManager` — `UploadBlock` Message Handler

When UploadBlock arrives, NettyBlockRpcServer deserializes the metadata of the input message to get the StorageLevel and ClassTag of the block being uploaded.

Note	`metadata` is serialized before `NettyBlockTransferService` sends a `UploadBlock` message (using the internal `JavaSerializer`) that is given as `serializer` when `NettyBlockRpcServer` is created.

NettyBlockRpcServer creates a BlockId for the block id and requests the BlockDataManager to store the block.

Note	The `BlockDataManager` is passed in when `NettyBlockRpcServer` is created.

In the end, NettyBlockRpcServer responds with a 0-capacity ByteBuffer.

Note	`UploadBlock` is sent when `NettyBlockTransferService` uploads a block.

NettyBlockRpcServer