SessionState

SessionState is the state separation layer between Spark SQL sessions, including SQL configuration, tables, functions, UDFs, SQL parser, and everything else that depends on a SQLConf.

You can access SessionState of a SparkSession through sessionState property.

val spark: SparkSession = ...
spark.sessionState

SessionState is created when…​FIXME

Table 1. SessionState’s (Lazily-Initialized) Attributes (in alphabetical order)
Name Type Description

analyzer

Analyzer

FIXME

Used when…​FIXME

catalog

SessionCatalog

Manages tables and databases.

Used when…​FIXME

conf

SQLConf

FIXME

Used when…​FIXME

experimentalMethods

ExperimentalMethods

FIXME

Used when…​FIXME

functionRegistry

FunctionRegistry

FIXME

Used when…​FIXME

functionResourceLoader

FunctionResourceLoader

FIXME

Used when…​FIXME

listenerManager

ExecutionListenerManager

FIXME

Used when…​FIXME

optimizer

Optimizer

Logical query plan optimizer

Used exclusively when QueryExecution creates an optimized logical plan.

planner

SparkPlanner

FIXME

Used when…​FIXME

resourceLoader

SessionResourceLoader

FIXME

Used when…​FIXME

sqlParser

ParserInterface

FIXME

Used when…​FIXME

streamingQueryManager

StreamingQueryManager

FIXME

Used when…​FIXME

udfRegistration

UDFRegistration

Interface to register user-defined functions.

Used when…​FIXME
Note
 SessionState is a private[sql] class and, given the package org.apache.spark.sql.internal, SessionState should be considered internal.

Creating SessionState Instance

SessionState takes the following when created:

SessionState initializes the attributes.

apply Factory Methods

Caution
 FIXME 
apply(sparkSession: SparkSession): SessionState (1)
apply(sparkSession: SparkSession, sqlConf: SQLConf): SessionState

  1. Passes sparkSession to the other apply with a new SQLConf

Note
 apply is used when SparkSession is requested for SessionState.

clone Method

Caution
 FIXME
Note
 clone is used when…​

createAnalyzer Internal Method

createAnalyzer(
  sparkSession: SparkSession,
  catalog: SessionCatalog,
  sqlConf: SQLConf): Analyzer

createAnalyzer creates a logical query plan Analyzer with rules specific to a non-Hive SessionState.

Table 2. Analyzer’s Evaluation Rules for non-Hive SessionState (in the order of execution)
Method Rules Description

extendedResolutionRules

FindDataSourceTable

Replaces InsertIntoTable (with CatalogRelation) and CatalogRelation logical plans with LogicalRelation.

ResolveSQLOnFile

postHocResolutionRules

PreprocessTableCreation

PreprocessTableInsertion

DataSourceAnalysis

extendedCheckRules

PreWriteCheck

HiveOnlyCheck
Note
 createAnalyzer is used when SessionState is created or cloned.

Executing Logical Plan — executePlan Method

executePlan(plan: LogicalPlan): QueryExecution

executePlan executes the input LogicalPlan to produce a QueryExecution in the current SparkSession.

refreshTable Method

refreshTable is…​

addJar Method

addJar is…​

analyze Method

analyze is…​

Creating New Hadoop Configuration — newHadoopConf Method

newHadoopConf(): Configuration

newHadoopConf returns Hadoop’s Configuration that it builds using SparkContext.hadoopConfiguration (through SparkSession) with all configuration settings added.

Note
 newHadoopConf is used by ScriptTransformation, ParquetRelation, StateStoreRDD, and SessionState itself, and few other places.
Caution
 FIXME What is ScriptTransformation? StateStoreRDD?

results matching ""

    No results matching ""