GenerateUnsafeProjection

GenerateUnsafeProjection is a CodeGenerator for converting Catalyst expressions to UnsafeProjection.

GenerateUnsafeProjection: Seq[Expression] => UnsafeProjection

Tip

Enable DEBUG logging level for org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjection logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjection=DEBUG

Refer to Logging.

Creating ExprCode for Catalyst Expressions — `createCode` Method

Caution

FIXME

`generate` Method

generate(
  expressions: Seq[Expression],
  subexpressionEliminationEnabled: Boolean): UnsafeProjection

generate creates a UnsafeProjection with expressions canonicalized.

Note	`generate` is used when `UnsafeProjection` factory object creates a `UnsafeProjection`.

`canonicalize` Method

canonicalize(in: Seq[Expression]): Seq[Expression]

canonicalize removes unnecessary Alias expressions.

Internally, canonicalize uses ExpressionCanonicalizer rule executor (that in turn uses just one CleanExpressions expression rule).

`create` Method

create(
  expressions: Seq[Expression],
  subexpressionEliminationEnabled: Boolean): UnsafeProjection
create(references: Seq[Expression]): UnsafeProjection (1)

Calls the former create with subexpressionEliminationEnabled disabled

create first creates a CodegenContext and an ExprCode for the input expressions that is converted to a Java source code (as CodeAndComment).

You should see the following DEBUG message in the logs:

DEBUG GenerateUnsafeProjection: code for [expressions]:
[code]

Tip	Enable `DEBUG` logging level for `org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator` logger to see the message above. See CodeGenerator.

create requests CodeGenerator to compile the Java source code into a GeneratedClass.

You should see the following INFO message in the logs:

INFO CodeGenerator: Code generated in [time] ms

Tip	Enable `INFO` logging level for `org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator` logger to see the message above. See CodeGenerator.

create passes references into the GeneratedClass that eventually becomes the final UnsafeProjection.

Note	(Single-argument) `create` is a part of CodeGenerator Contract.

GenerateUnsafeProjection