DirectTaskResult[T](
var valueBytes: ByteBuffer,
var accumUpdates: Seq[AccumulatorV2[_, _]])
extends TaskResult[T] with Externalizable
TaskResults — DirectTaskResult and IndirectTaskResult
TaskResult
models a task result. It has exactly two concrete implementations:
-
DirectTaskResult is the
TaskResult
to be serialized and sent over the wire to the driver together with the result bytes and accumulators. -
IndirectTaskResult is the
TaskResult
that is just a pointer to a task result in aBlockManager
.
The decision of the concrete TaskResult
is made when a TaskRunner
finishes running a task and checks the size of the result.
Note
|
The types are private[spark] .
|
DirectTaskResult
Task Result
DirectTaskResult
is the TaskResult of running a task (that is later returned serialized to the driver) when the size of the task’s result is smaller than spark.driver.maxResultSize and spark.task.maxDirectResultSize (or spark.rpc.message.maxSize whatever is smaller).
Note
|
DirectTaskResult is Java’s java.io.Externalizable.
|
IndirectTaskResult
Task Result
IndirectTaskResult[T](blockId: BlockId, size: Int)
extends TaskResult[T] with Serializable
IndirectTaskResult
is a TaskResult that…
Note
|
IndirectTaskResult is Java’s java.io.Serializable.
|