BlockTransferService — Pluggable Block Transfers

BlockTransferService is a contract for ShuffleClients that can fetch and upload blocks synchronously or asynchronously.

Note
 BlockTransferService is a private[spark] abstract class.
Note
 NettyBlockTransferService is the only available implementation of BlockTransferService Contract.
Note
 BlockTransferService was introduced in SPARK-3019 Pluggable block transfer interface (BlockTransferService) and is available since Spark 1.2.0.

BlockTransferService Contract

Every BlockTransferService offers the following:

  • init that accepts BlockDataManager for storing or fetching blocks. It is assumed that the method is called before a BlockTransferService service is considered fully operational.

    init(blockDataManager: BlockDataManager): Unit

  • port the service listens to.

    port: Int

  • hostName the service listens to.

    hostName: String

  • uploadBlock to upload a block (of ManagedBuffer identified by BlockId) to a remote hostname and port.

    uploadBlock(
  hostname: String,
  port: Int,
  execId: String,
  blockId: BlockId,
  blockData: ManagedBuffer,
  level: StorageLevel,
  classTag: ClassTag[_]): Future[Unit]

  • Synchronous (and hence blocking) fetchBlockSync to fetch one block blockId (that corresponds to the ShuffleClient parent’s asynchronous fetchBlocks).

    fetchBlockSync(
  host: String,
  port: Int,
  execId: String,
  blockId: String): ManagedBuffer

    fetchBlockSync is a mere wrapper around fetchBlocks to fetch one blockId block that waits until the fetch finishes.

uploadBlockSync Method

uploadBlockSync(
  hostname: String,
  port: Int,
  execId: String,
  blockId: BlockId,
  blockData: ManagedBuffer,
  level: StorageLevel,
  classTag: ClassTag[_]): Unit

uploadBlockSync is a mere blocking wrapper around uploadBlock that waits until the upload finishes.

Note
 uploadBlockSync is only executed when BlockManager replicates a block to another node(s) (i.e. when a replication level is greater than 1).

results matching ""

    No results matching ""