The recommendations to configure Splunk instances that are running on K8S using the Splunk operator is by bundling .conf files in Splunk apps.
Unfortunately some things can’t be configured with Splunk apps, e.g. adding a user. Then you have two options:
Adding the user manually via CLI or UI and therefore not having the configuration in Git OR Deploying a second instance in K8S that is doing the configuration for the first instance Here’s the deployment YAML for this second option:...
Currently, the Splunk K8S operator doesn’t support injecting passwords from Kubernetes secrets into the Splunk config. The workaround is to store a complete configuration file as a secret, see https://github.com/splunk/splunk-operator/issues/657
As configuration files can be quite large, I created a small bash script that is using template files for the configuration and filling in the secrets based on user input. You can create the K8S secret containing the configuration by calling:...
After having discussions about how to use Gitops with Splunk, I created this template to store the complete configuration (apps and system configuration) of a Splunk installation running on Kubernetes in Git:
https://github.com/marcusschiesser/splunk-gitops.
Having the complete configuration in a single repository is an important prerequisite to enjoy the benefits of GitOps.
To ensure that this template is running with future versions of Splunk, it is solely extending Splunk Docker images according to the docker-splunk configuration....