Window Frame [f] must match the required frame [frame]
ResolveWindowFrame Logical Evaluation Rule
ResolveWindowFrame
is a logical evaluation rule that Spark SQL’s logical query plan analyzer uses to validate and resolve WindowExpression Catalyst logical expressions.
ResolveWindowFrame
is a part of Resolution fixed-point batch of rules.
ResolveWindowFrame
takes a logical plan and does the following:
-
Makes sure that the window frame of a
WindowFunction
is unspecified or matches theSpecifiedWindowFrame
of the WindowSpecDefinition expression.Reports a
AnalysisException
when the frames do not match: -
Copies the frame specification of
WindowFunction
to WindowSpecDefinition -
Creates a new
SpecifiedWindowFrame
forWindowExpression
with the resolved Catalyst expression andUnspecifiedFrame
Note
|
ResolveWindowFrame is a Scala object inside Analyzer class.
|
import import org.apache.spark.sql.expressions.Window
// cume_dist requires ordered windows
val q = spark.
range(5).
withColumn("cume_dist", cume_dist() over Window.orderBy("id"))
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
val planBefore: LogicalPlan = q.queryExecution.logical
// Before ResolveWindowFrame
scala> println(planBefore.numberedTreeString)
00 'Project [*, cume_dist() windowspecdefinition('id ASC NULLS FIRST, UnspecifiedFrame) AS cume_dist#39]
01 +- Range (0, 5, step=1, splits=Some(8))
import spark.sessionState.analyzer.ResolveWindowFrame
val planAfter = ResolveWindowFrame.apply(plan)
// After ResolveWindowFrame
scala> println(planAfter.numberedTreeString)
00 'Project [*, cume_dist() windowspecdefinition('id ASC NULLS FIRST, RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS cume_dist#31]
01 +- Range (0, 5, step=1, splits=Some(8))