acquireStorageMemory(blockId: BlockId, numBytes: Long, memoryMode: MemoryMode): BooleanMemoryManager — Memory Management System
MemoryManager is an abstract base memory manager to manage shared memory for execution and storage.
Execution memory is used for computation in shuffles, joins, sorts and aggregations.
Storage memory is used for caching and propagating internal data across the nodes in a cluster.
A MemoryManager is created when SparkEnv is created (one per JVM) and can be one of the two possible implementations:
-
UnifiedMemoryManager — the default memory manager since Spark 1.6.
-
StaticMemoryManager(legacy)
|
Note
|
org.apache.spark.memory.MemoryManager is a private[spark] Scala trait in Spark.
|
MemoryManager Contract
Every MemoryManager obeys the following contract:
acquireStorageMemory
acquireStorageMemory
|
Caution
|
FIXME |
acquireStorageMemory is used in MemoryStore to put bytes.
maxOnHeapStorageMemory Attribute
maxOnHeapStorageMemory: LongmaxOnHeapStorageMemory is the total amount of memory available for storage, in bytes. It can vary over time.
|
Caution
|
FIXME Where is this used? |
It is used in MemoryStore to ??? and BlockManager to ???
tungstenMemoryMode
tungstenMemoryMode informs others whether Spark works in OFF_HEAP or ON_HEAP memory mode.
It uses spark.memory.offHeap.enabled (default: false), spark.memory.offHeap.size (default: 0), and org.apache.spark.unsafe.Platform.unaligned before OFF_HEAP is assumed.
|
Caution
|
FIXME Describe org.apache.spark.unsafe.Platform.unaligned.
|