import org.apache.spark.sql.types.StructType
scala> val struct = new StructType().add("a", "int")
struct: org.apache.spark.sql.types.StructType = StructType(StructField(a,IntegerType,true))
scala> val asInt = expr("token = 'hello'").cast("int")
asInt: org.apache.spark.sql.Column = CAST((token = hello) AS INT)CatalystSqlParser — DataTypes and StructTypes Parser
CatalystSqlParser is a AbstractSqlParser with AstBuilder as the required astBuilder.
CatalystSqlParser is used to translate DataTypes from their canonical string representation (e.g. when adding fields to a schema or casting column to a different data type) or StructTypes.
When parsing, you should see INFO messages in the logs:
INFO CatalystSqlParser: Parsing command: intIt is also used in HiveClientImpl (when converting columns from Hive to Spark) and in OrcFileOperator (when inferring the schema for ORC files).
|
Tip
|
Enable Add the following line to Refer to Logging. |