SparkPlanner — Query Planner with no Hive Support

Note	`SparkPlanner` is expected to plan (aka generate) at least one physical plan for a logical plan.

SparkPlanner is available as planner of a SessionState.

val spark: SparkSession = ...
spark.sessionState.planner

Table 1. SparkPlanner’s Execution Planning Strategies (in execution order)
SparkStrategy	Description
`ExperimentalMethods`'s extraStrategies
extraPlanningStrategies	Extension point for extra planning strategies
FileSourceStrategy
DataSourceStrategy
`SpecialLimits`
Aggregation
JoinSelection
InMemoryScans
BasicOperators

Note	`SparkPlanner` extends SparkStrategies abstract class.

SparkPlanner takes the following when created:

Note	`SparkPlanner` is created in: BaseSessionStateBuilder `HiveSessionStateBuilder` Structured Streaming’s `IncrementalExecution`

extraPlanningStrategies: Seq[Strategy] = Nil

extraPlanningStrategies is an extension point to register extra planning strategies with the query planner.

Note	`extraPlanningStrategies` are executed after extraStrategies.

Note	`extraPlanningStrategies` is used when `SparkPlanner` is requested for planning strategies. `extraPlanningStrategies` is overriden in the `SessionState` builders — BaseSessionStateBuilder and `HiveSessionStateBuilder`.

collectPlaceholders(plan: SparkPlan): Seq[(SparkPlan, LogicalPlan)]

collectPlaceholders collects all PlanLater physical operators in the plan physical plan.

Note	`collectPlaceholders` is a part of QueryPlanner Contract.

prunePlans(plans: Iterator[SparkPlan]): Iterator[SparkPlan]

prunePlans gives the input plans physical plans back (i.e. with no changes).

Note	`prunePlans` is a part of QueryPlanner Contract to remove somehow "bad" plans.

Caution

Note	`pruneFilterProject` is a helper method used exclusively in InMemoryScans and `HiveTableScans` execution planning strategies.

results matching ""