write(kvBytes: Array[Byte], offs: Int, len: Int): Unit
DiskBlockObjectWriter
DiskBlockObjectWriter
is a java.io.OutputStream that BlockManager
offers for writing blocks to disk.
Whenever DiskBlockObjectWriter
is requested to write a key-value pair, it makes sure that the underlying output streams are open.
DiskBlockObjectWriter
can be in the following states (that match the state of the underlying output streams):
-
Initialized
-
Open
-
Closed
Name | Description |
---|---|
Internal flag…FIXME Used when…FIXME |
|
Internal flag…FIXME Used when…FIXME |
|
Internal flag…FIXME Used when…FIXME |
|
Used when…FIXME |
|
Used when…FIXME |
|
Used when…FIXME |
|
Used when…FIXME |
|
Used when…FIXME |
Note
|
DiskBlockObjectWriter is a private[spark] class.
|
updateBytesWritten
Method
Caution
|
FIXME |
initialize
Method
Caution
|
FIXME |
Writing Bytes (From Byte Array Starting From Offset) — write
Method
write
…FIXME
Caution
|
FIXME |
recordWritten
Method
Caution
|
FIXME |
close
Method
Caution
|
FIXME |
Creating DiskBlockObjectWriter Instance
DiskBlockObjectWriter
takes the following when created:
-
file
-
serializerManager
— SerializerManager -
serializerInstance
— SerializerInstance -
bufferSize
-
syncWrites
flag -
writeMetrics
— ShuffleWriteMetrics -
blockId
— BlockId
DiskBlockObjectWriter
initializes the internal registries and counters.
Writing Key-Value Pair — write
Method
write(key: Any, value: Any): Unit
Before writing, write
opens the stream unless already open.
write
then writes the key
first followed by writing the value
.
In the end, write
recordWritten.
Note
|
write is used when BypassMergeSortShuffleWriter writes records and in ExternalAppendOnlyMap , ExternalSorter and WritablePartitionedPairCollection .
|
Opening DiskBlockObjectWriter — open
Method
open(): DiskBlockObjectWriter
open
opens DiskBlockObjectWriter
, i.e. initializes and re-sets bs and objOut internal output streams.
Internally, open
makes sure that DiskBlockObjectWriter
is not closed (i.e. hasBeenClosed flag is disabled). If it was, open
throws a IllegalStateException
:
Writer already closed. Cannot be reopened.
Unless DiskBlockObjectWriter
has already been initialized (i.e. initialized flag is enabled), open
initializes it (and turns initialized flag on).
Regardless of whether DiskBlockObjectWriter
was already initialized or not, open
requests SerializerManager
to wrap mcs
output stream for encryption and compression (for blockId) and sets it as bs.
Note
|
open uses SerializerManager that was specified when DiskBlockObjectWriter was created
|
open
requests SerializerInstance
to serialize bs
output stream and sets it as objOut.
Note
|
open uses SerializerInstance that was specified when DiskBlockObjectWriter was created
|
In the end, open
turns streamOpen flag on.
Note
|
open is used exclusively when DiskBlockObjectWriter writes a key-value pair or bytes from a specified byte array but the stream is not open yet.
|