The recommendations to configure Splunk instances that are running on K8S using the Splunk operator is by bundling .conf files in Splunk apps.
Unfortunately some things can’t be configured with Splunk apps, e.g. adding a user. Then you have two options:
Adding the user manually via CLI or UI and therefore not having the configuration in Git OR Deploying a second instance in K8S that is doing the configuration for the first instance Here’s the deployment YAML for this second option:...
K8S doesn’t allow to share secrets between namespaces for security reasons.
It might be acceptable from a security point of view to share secrets nevertheless - then a common workaround is to copy secrets between namespaces. One pattern is to keep a template of the secret in the default namespace and copy it from there to other namespaces.
Here’s a version of copying secrets between namespaces that also works if the destination secret already exists - in this case it’s just updated with the data from the source:...
Currently, the Splunk K8S operator doesn’t support injecting passwords from Kubernetes secrets into the Splunk config. The workaround is to store a complete configuration file as a secret, see https://github.com/splunk/splunk-operator/issues/657
As configuration files can be quite large, I created a small bash script that is using template files for the configuration and filling in the secrets based on user input. You can create the K8S secret containing the configuration by calling:...
After having discussions about how to use Gitops with Splunk, I created this template to store the complete configuration (apps and system configuration) of a Splunk installation running on Kubernetes in Git:
https://github.com/marcusschiesser/splunk-gitops.
Having the complete configuration in a single repository is an important prerequisite to enjoy the benefits of GitOps.
To ensure that this template is running with future versions of Splunk, it is solely extending Splunk Docker images according to the docker-splunk configuration....
I needed an EKS cluster that supports Persistent Volumes with Access Mode ReadWriteMany.
Therefore I updated my EKS Terraform template to support adding the EFS CSI driver if needed.
Usage is pretty simple, you’ll just have to add enable_efs = true to your Terraform cluster definition.
This will also create an EFS file system (storage will by provisioned dynamically depending on the requested persistent volume).
You’ll just have to retrieve the file system ID from your AWS console and use it to create a new K8S storage class:...