Powered by GitBook

WindowExpression Unevaluable Expression

WindowExpression is an unevaluable expression that contains the Catalyst expressions of a window function and WindowSpecDefinition in a query plan after Analyzer resolves UnresolvedWindowExpressions.

import org.apache.spark.sql.catalyst.expressions.WindowExpression
// relation - Dataset as a table to query
val table = spark.emptyDataset[Int]

scala> val windowExpr = table
  .selectExpr("count() OVER (PARTITION BY value) AS count")
  .queryExecution
  .logical      (1)
  .expressions
  .toList(0)
  .children(0)
  .asInstanceOf[WindowExpression]
windowExpr: org.apache.spark.sql.catalyst.expressions.WindowExpression = 'count() windowspecdefinition('value, UnspecifiedFrame)

scala> windowExpr.sql
res2: String = count() OVER (PARTITION BY `value` UnspecifiedFrame)

Use sqlParser directly as in WithWindowDefinition Example

Note	`WindowExpression` is used in ExtractWindowExpressions, ResolveWindowOrder and ResolveWindowFrame logical evaluation rules.

Note	`WindowExpression` is also used in `Analyzer` for analysis validation for the following checks: FIXME…

Note	`WindowExpression` is used in NullPropagation optimization.

Table 1. WindowExpression’s Properties (in alphabetical order)
Name	Description
`children`	Collection of two expressions, i.e. windowFunction and WindowSpecDefinition, for which `WindowExpression` was created.
`dataType`	DataType of windowFunction
`foldable`	Whether or not windowFunction is foldable.
`nullable`	Whether or not windowFunction is nullable.
`sql`	`"[windowFunction].sql OVER [windowSpec].sql"`
`toString`	`"[windowFunction] [windowSpec]"`

UnresolvedWindowExpression Unevaluable Expression — WindowExpression With Unresolved Window Specification Reference

UnresolvedWindowExpression is an unevaluable expression (i.e. with no support for eval and doGenCode methods).

UnresolvedWindowExpression is created to represent a child expression and WindowSpecReference (with an identifier for the window reference) when AstBuilder parses a function evaluated in a windowed context with a WindowSpecReference.

UnresolvedWindowExpression is resolved to a WindowExpression when Analyzer resolves UnresolvedWindowExpressions.

import spark.sessionState.sqlParser

scala> sqlParser.parseExpression("foo() OVER windowSpecRef")
res1: org.apache.spark.sql.catalyst.expressions.Expression = unresolvedwindowexpression('foo(), WindowSpecReference(windowSpecRef))

Table 2. UnresolvedWindowExpression’s Properties (in alphabetical order)
Name	Description
`dataType`	Reports a `UnresolvedException`
`foldable`	Reports a `UnresolvedException`
`nullable`	Reports a `UnresolvedException`
`resolved`	Disabled (i.e. `false`)

results matching ""

No results matching ""