BroadcastManager

Broadcast Manager (BroadcastManager) is a Spark service to manage broadcast variables in Spark. It is created for a Spark application when SparkContext is initialized and is a simple wrapper around BroadcastFactory.

BroadcastManager tracks the number of broadcast variables in a Spark application (using the internal field nextBroadcastId).

The idea is to transfer values used in transformations from a driver to executors in a most effective way so they are copied once and used many times by tasks (rather than being copied every time a task is launched).

When initialized, BroadcastManager creates an instance of TorrentBroadcastFactory.

stop Method

Caution
FIXME

Creating BroadcastManager Instance

Caution
FIXME

Initializing BroadcastManager — initialize Internal Method

initialize(): Unit
Note
initialize is executed only once (when BroadcastManager is created) and controlled by the internal initialized flag.

newBroadcast Method

newBroadcast[T](value_ : T, isLocal: Boolean): Broadcast[T]

newBroadcast simply requests the current BroadcastFactory for a new broadcast variable.

Note
The BroadcastFactory is created when BroadcastManager is initialized.
Note
newBroadcast is executed for SparkContext.broadcast method and when MapOutputTracker serializes MapStatuses.

Settings

Table 1. Settings
Name Default value Description

spark.broadcast.blockSize

4m

The size of a block (in kB when unit not specified).

Used when TorrentBroadcast stores brodcast blocks to BlockManager.

spark.broadcast.compress

true

The flag to enable compression.

Refer to CompressionCodec.

Used when TorrentBroadcast is created and later when it stores broadcast blocks to BlockManager. Also in SerializerManager.

results matching ""

    No results matching ""