val spark: SparkSession = ...
spark.sessionState.sqlParserSparkSqlParser — Default SQL Parser
SparkSqlParser is the default parser of the SQL statements supported in Spark SQL with the astBuilder as SparkSqlAstBuilder and support for variable substitution.
|
Note
|
Spark SQL supports SQL statements as described in SqlBase.g4 ANTLR grammar. |
SparkSqlParser is available as sqlParser of a SessionState.
SparkSqlParser is used to translate an expression to its corresponding Column in the following:
-
expr function
-
selectExpr method (of
Dataset) -
filter method (of
Dataset) -
where method (of
Dataset)
scala> expr("token = 'hello'")
16/07/07 18:32:53 INFO SparkSqlParser: Parsing command: token = 'hello'
res0: org.apache.spark.sql.Column = (token = hello)SparkSqlParser is used to parse table strings into their corresponding table identifiers in the following:
-
tablemethods in DataFrameReader and SparkSession -
insertInto and saveAsTable methods of
DataFrameWriter -
createExternalTableandrefreshTablemethods of Catalog (and SessionState)
SparkSqlParser is used to translate a SQL text to its corresponding LogicalPlan in sql method in SparkSession.
|
Tip
|
Enable Add the following line to Refer to Logging. |
Variable Substitution
|
Caution
|
FIXME See SparkSqlParser and substitutor.
|