MetricsConfig — Metrics System Configuration

MetricsConfig is the configuration of the MetricsSystem (i.e. metrics sources and sinks).

MetricsConfig uses metrics.properties as the default metrics configuration file that can however be changed using spark.metrics.conf property. The file is first loaded from the path directly before using Spark’s CLASSPATH.

MetricsConfig lets you also configure the metrics configuration using spark.metrics.conf.-prefixed Spark properties.

MetricsConfig makes sure that the default metrics properties are always defined.

Table 1. MetricsConfig’s Default Metrics Properties
Name	Description
`*.sink.servlet.class`	`org.apache.spark.metrics.sink.MetricsServlet`
`*.sink.servlet.path`	`/metrics/json`
`master.sink.servlet.path`	`/metrics/master/json`
`applications.sink.servlet.path`	`/metrics/applications/json`

Note	The order of precedence of metrics configuration settings is as follows: Default metrics properties spark.metrics.conf or `metrics.properties` configuration file `spark.metrics.conf.`-prefixed Spark properties

MetricsConfig is created when MetricsSystem is created.

Table 2. MetricsConfig’s Internal Registries and Counters
Name	Description
`properties`	java.util.Properties with metrics properties Used to initialize per-subsystem’s perInstanceSubProperties.
`perInstanceSubProperties`	Lookup table of metrics properties per subsystem

Creating MetricsConfig Instance

MetricsConfig takes the following when created:

SparkConf

MetricsConfig initializes the internal registries and counters.

Initializing MetricsConfig — `initialize` Method

initialize(): Unit

initialize sets the default properties and loads the properties from the configuration file (that is defined using spark.metrics.conf Spark property).

initialize takes all Spark properties that start with spark.metrics.conf. prefix from SparkConf and adds them to properties (without the prefix).

In the end, initialize splits configuration per Spark subsystem with the default configuration (denoted as *) assigned to the configured subsystems afterwards.

Note	`initialize` accepts `*` (star) for the default configuration or any combination of lower- and upper-case letters for Spark subsystem names.

Note	`initialize` is used exclusively when `MetricsSystem` is created.

`setDefaultProperties` Internal Method

setDefaultProperties(prop: Properties): Unit

setDefaultProperties sets the default properties (in the input prop).

Note	`setDefaultProperties` is used exclusively when `MetricsConfig` is initialized.

Loading Custom Metrics Configuration File or metrics.properties — `loadPropertiesFromFile` Method

loadPropertiesFromFile(path: Option[String]): Unit

loadPropertiesFromFile tries to open the input path file (if defined) or the default metrics configuration file metrics.properties (on CLASSPATH).

If either file is available, loadPropertiesFromFile loads the properties (to properties registry).

In case of exceptions, you should see the following ERROR message in the logs followed by the exception.

ERROR Error loading configuration file [file]

Note	`loadPropertiesFromFile` is used exclusively when `MetricsConfig` is initialized.

Grouping Properties Per Subsystem — `subProperties` Method

subProperties(prop: Properties, regex: Regex): mutable.HashMap[String, Properties]

subProperties takes prop properties and destructures keys given regex. subProperties takes the matching prefix (of a key per regex) and uses it as a new key with the value(s) being the matching suffix(es).

driver.hello.world => (driver, (hello.world))

Note	`subProperties` is used when `MetricsConfig` is initialized (to apply the default metrics configuration) and when `MetricsSystem` registers metrics sources and sinks.

Settings

Table 3. Spark Properties
Spark Property	Default Value	Description
`spark.metrics.conf`	`metrics.properties`	The metrics configuration file.

MetricsConfig — Metrics System Configuration