Usage

Splunk App for Corda relies on having direct access to the environment where your Corda nodes are running. As such, you'll need to modify the start up command for your Corda nodes. The startup command is illustrated here. Take note of the new dependencies you'll have, mentioned above. You can find an example of how to obtain and install them here.

In addition to the two java agents, make sure you install the following on your corda node host:
- collectd
- Splunk Universal Forwarder

Refer to the example node dockerfile for how to accomplish the above, or if you already have Splunk App for Infrastructure installed on your Splunk instance, follow the instructions in the App to add more data sources.

Corda Logging Spec (IMPORTANT!)

Splunk App for Corda relies on the Corda node logging in JSON. One of the most important updates you'll need to make to your Corda node startup is to specify a new logging configuration for Log4j2. You can see the full command here. Specifically, you'll find the logging config here, and will specify it like this:

java -Dlog4j.configurationFile=logging.xml -jar corda.jar

Logging arbitrary data

We have implemented a pattern for logging arbitrary data that is illustrated in the CorDapp used in the Docker example. This involves using ThreadContext to put additional strings into your logging context.

You'll need to:
- include log4j2 core in your project.
- import ThreadContext in your workflow.
- put values into your logger ThreadContext.
- remove values from ThreadContext when done.

Splunk considerations

Specify your Splunk host in the following locations:
- On your node host, in the Universal Forwarder outputs.conf
- In the Otel Collector config, here and here.

Specify your Splunk HEC token in the following locations:
- In the Otel Collector config, here and here.

On your Splunk instance

You'll need to add 3 indexes to your Splunk installation.
- name: corda, type: event
- name: traces, type: event
- name: metrics, type: metric

You will want to install our Splunk Apps and Add-ons to make sense of all of the new data being sent to your instance. You can read more about infrastructure monitoring using our Infrastructure Monitoring Tooling here and IT Essentials here.

Architecture / Getting Data In

If you're installing manually in your environment, please consider GDI architecture outline below.

Splunk Configuration

• props.conf
• You'll need to add an entry in /opt/splunk/etc/system/local/props.conf to tell splunk where to find the timestamp in the Corda JSON logs.
• Relevant files: props.conf

Corda Environment

• Corda Log4j2 JSON config
  • Splunk App for Corda relies on the Corda node logging in the JSON format. You will need to specify a log4j2 config to the corda node at runtime.
  • Relevant files: logging.xml, start_node.sh
• OpenTelemetry JavaAgent
  • The OpenTelemetry JavaAgent (otel-agent.jar) exposes full-fidelity trace data from Corda (operation, timespan). Examples of this include database performance traces and custom traces in CorDapps. The Otel Agent relies on environment variables for configuration and connection info for the Otel Colletor that should be supplied at runtime.
  • Relevant files: fetch_javaagents.sh, start_node.sh (note the OTEL_* environment variables)
• Prometheus JMX JavaAgent
  • The Prometheus JMX JavaAgent (prometheus-jmx-agent.jar) exposes the JMX metrics from Corda in Prometheus format. The Prometheus JMX JavaAgent relies on a local yaml file for configuration that should be available at runtime. This is mainly formatting rules.
  • Relevant files: fetch_javaagents.sh, jmx_prometheus_config.yaml, start_node.sh

Additional Services

• Splunk Universal Forwarder
  • The Splunk Universal Forwarder forwards specified log files from the Corda node to Splunk on the port 9997.
  • Relevant files: install_collectd_and_splunk_uf.sh, fwdr_inputs.conf, start_splunk_uf.sh
  • Note: Ignore the collectd install if you do not want to collect and send host machine metrics.
• OpenTelemetry Collector
  • The OpenTelemetry Colector (Otel Collector) collects metrics and traces from the OpenTelemetry JavaAgent and the Prometheus JMX JavaAgent running in the JVM with Corda, on ports 4317 and 8080 respectively. The Otel collector sends to Splunk via HEC on port 8088.
  • Relevant files: install_otel_collector.sh, otel-collector-config.yml, start_otel_collector.sh

Examples

We've provided a fully functioning example of everything outlined above, and you can learn more in the Docker example's README.