log4j.logger.org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArray=INFO
ExternalAppendOnlyUnsafeRowArray — Append-Only Array for UnsafeRows (with Disk Spill Threshold)
ExternalAppendOnlyUnsafeRowArray
is an append-only array for UnsafeRows that spills content to disk when a predefined spill threshold of rows is reached.
Note
|
Choosing a proper spill threshold of rows is a performance optimization. |
ExternalAppendOnlyUnsafeRowArray
is created when:
-
WindowExec
physical operator is executed (and creates an internal buffer for window frames) -
WindowFunctionFrame
is prepared -
SortMergeJoinExec
physical operator is executed (and creates aRowIterator
for INNER and CROSS joins) and forgetBufferedMatches
-
SortMergeJoinScanner
creates an internalbufferedMatches
-
UnsafeCartesianRDD
is computed
Name | Description |
---|---|
Used when…FIXME |
|
Can grow up to numRowsSpillThreshold rows (i.e. new Used when…FIXME |
|
Used when…FIXME |
|
Used when…FIXME |
|
Used when…FIXME |
|
Used when…FIXME |
Tip
|
Enable Add the following line to Refer to Logging. |
generateIterator
Method
generateIterator(): Iterator[UnsafeRow]
generateIterator(startIndex: Int): Iterator[UnsafeRow]
Caution
|
FIXME |
add
Method
add(unsafeRow: UnsafeRow): Unit
Caution
|
FIXME |
Note
|
|
Creating ExternalAppendOnlyUnsafeRowArray Instance
ExternalAppendOnlyUnsafeRowArray
takes the following when created:
ExternalAppendOnlyUnsafeRowArray
initializes the internal registries and counters.