ClientDistributedCacheManager

ClientDistributedCacheManager is a mere wrapper to hold the collection of cache-related resource entries CacheEntry (as distCacheEntries) to add resources to and later update Spark configuration with files to distribute.

Caution
 FIXME What is a resource? Is this a file only?

Adding Cache-Related Resource (addResource method)

addResource(
  fs: FileSystem,
  conf: Configuration,
  destPath: Path,
  localResources: HashMap[String, LocalResource],
  resourceType: LocalResourceType,
  link: String,
  statCache: Map[URI, FileStatus],
  appMasterOnly: Boolean = false): Unit

Updating Spark Configuration with Resources to Distribute (updateConfiguration method)

updateConfiguration(conf: SparkConf): Unit

updateConfiguration sets the following internal Spark configuration settings in the input conf Spark configuration:

It uses the internal distCacheEntries with resources to distribute.

Note
 It is later used in ApplicationMaster when it prepares local resources.

results matching ""

    No results matching ""