initialize(): Unit
BroadcastManager
Broadcast Manager (BroadcastManager
) is a Spark service to manage broadcast variables in Spark. It is created for a Spark application when SparkContext is initialized and is a simple wrapper around BroadcastFactory.
BroadcastManager
tracks the number of broadcast variables in a Spark application (using the internal field nextBroadcastId
).
The idea is to transfer values used in transformations from a driver to executors in a most effective way so they are copied once and used many times by tasks (rather than being copied every time a task is launched).
When initialized, BroadcastManager
creates an instance of TorrentBroadcastFactory.
stop
Method
Caution
|
FIXME |
Creating BroadcastManager
Instance
Caution
|
FIXME |
Initializing BroadcastManager — initialize
Internal Method
Note
|
initialize is executed only once (when BroadcastManager is created) and controlled by the internal initialized flag.
|
newBroadcast
Method
newBroadcast[T](value_ : T, isLocal: Boolean): Broadcast[T]
newBroadcast
simply requests the current BroadcastFactory
for a new broadcast variable.
Note
|
The BroadcastFactory is created when BroadcastManager is initialized.
|
Note
|
newBroadcast is executed for SparkContext.broadcast method and when MapOutputTracker serializes MapStatuses .
|
Settings
Name | Default value | Description |
---|---|---|
|
The size of a block (in kB when unit not specified). Used when |
|
|
The flag to enable compression. Refer to CompressionCodec. Used when |