JdbcRelationProvider

JdbcRelationProvider is a CreatableRelationProvider and RelationProvider that handles data sources for jdbc format.

val table = spark.read.jdbc(...)

// or in a more verbose way
val table = spark.read.format("jdbc").load(...)

Creating JDBCRelation — createRelation Method (from RelationProvider)

createRelation(
  sqlContext: SQLContext,
  parameters: Map[String, String]): BaseRelation

createRelation creates a JDBCPartitioningInfo (using JDBCOptions and the input parameters that correspond to Options for JDBC Data Source).

Note
createRelation uses partitionColumn, lowerBound, upperBound and numPartitions.

In the end, createRelation creates a JDBCRelation using column partitions (and JDBCOptions).

Note
createRelation is a part of RelationProvider Contract.

Creating JDBCRelation After Preparing Table in Database — createRelation Method (from CreatableRelationProvider)

createRelation(
  sqlContext: SQLContext,
  mode: SaveMode,
  parameters: Map[String, String],
  df: DataFrame): BaseRelation

Internally, createRelation creates a JDBCOptions (from the input parameters).

createRelation reads caseSensitiveAnalysis (using the input sqlContext).

createRelation checks whether the table (given dbtable and url options in the input parameters) exists.

Note
createRelation uses a database-specific JdbcDialect to check whether a table exists.

createRelation branches off per whether the table already exists in the database or not.

If the table does not exist, createRelation creates the table (by executing CREATE TABLE with createTableColumnTypes and createTableOptions options from the input parameters) and saves the records to the database in a single transaction.

If however the table does exist, createRelation branches off per SaveMode (see the following createRelation and SaveMode).

Table 1. createRelation and SaveMode (in alphabetical order)
Name Description

Append

Saves the records to the table.

ErrorIfExists

Throws a AnalysisException with the message:

Table or view '[table]' already exists. SaveMode: ErrorIfExists.

Ignore

Does nothing.

Overwrite

Truncates or drops the table

Note
createRelation truncates the table only when truncate option is enabled and isCascadingTruncateTable is disabled.

In the end, createRelation closes the JDBC connection to the database and creates a JDBCRelation.

Note
createRelation is a part of CreatableRelationProvider Contract.

results matching ""

    No results matching ""