log4j.logger.org.apache.spark.storage.memory.MemoryStore=DEBUG
MemoryStore
Memory store (MemoryStore
) manages blocks.
MemoryStore
requires SparkConf, BlockInfoManager, SerializerManager, MemoryManager and BlockEvictionHandler
.
Name | Description |
---|---|
Collection of …FIXME
NOTE: |
Caution
|
FIXME Where are these dependencies used? |
Caution
|
FIXME Where is the MemoryStore created? What params provided?
|
Note
|
MemoryStore is a private[spark] class.
|
Tip
|
Enable Add the following line to Refer to Logging. |
releaseUnrollMemoryForThisTask
Method
Caution
|
FIXME |
Is Block Available? — contains
Method
contains(blockId: BlockId): Boolean
contains
returns true
when the internal entries registry contains blockId
.
putIteratorAsBytes
Method
putIteratorAsBytes[T](
blockId: BlockId,
values: Iterator[T],
classTag: ClassTag[T],
memoryMode: MemoryMode): Either[PartiallySerializedBlock[T], Long]
putIteratorAsBytes
tries to put the blockId
block in memory store as bytes.
Caution
|
FIXME |
putIteratorAsValues
Method
putIteratorAsValues[T](
blockId: BlockId,
values: Iterator[T],
classTag: ClassTag[T]): Either[PartiallyUnrolledIterator[T], Long]
putIteratorAsValues
tries to put the blockId
block in memory store as values
.
Note
|
putIteratorAsValues is a private[storage] method.
|
Note
|
is called when BlockManager stores bytes of a block or iterator of values of a block or when attempting to cache spilled values read from disk.
|
Evicting Blocks to Free Space
Caution
|
FIXME |
Removing Block
Caution
|
FIXME |
Acquiring Storage Memory for Blocks — putBytes
Method
putBytes[T](
blockId: BlockId,
size: Long,
memoryMode: MemoryMode,
_bytes: () => ChunkedByteBuffer): Boolean
putBytes
requests storage memory for blockId
from MemoryManager
and registers the block in entries internal registry.
Internally, putBytes
first makes sure that blockId
block has not been registered already in entries internal registry.
putBytes
then requests size
memory for the blockId
block in a given memoryMode
from the current MemoryManager
.
Note
|
|
If successful, putBytes
"materializes" _bytes
byte buffer and makes sure that the size is exactly size
. It then registers a SerializedMemoryEntry
(for the bytes and memoryMode
) for blockId
in the internal entries registry.
You should see the following INFO message in the logs:
INFO Block [blockId] stored as bytes in memory (estimated size [size], free [bytes])
putBytes
returns true
only after blockId
was successfully registered in the internal entries registry.