After having discussions about how to use Gitops with Splunk, I created this template to store the complete configuration (apps and system configuration) of a Splunk installation running on Kubernetes in Git:
https://github.com/marcusschiesser/splunk-gitops.
Having the complete configuration in a single repository is an important prerequisite to enjoy the benefits of GitOps.
To ensure that this template is running with future versions of Splunk, it is solely extending Splunk Docker images according to the docker-splunk configuration....
I needed an EKS cluster that supports Persistent Volumes with Access Mode ReadWriteMany.
Therefore I updated my EKS Terraform template to support adding the EFS CSI driver if needed.
Usage is pretty simple, you’ll just have to add enable_efs = true to your Terraform cluster definition.
This will also create an EFS file system (storage will by provisioned dynamically depending on the requested persistent volume).
You’ll just have to retrieve the file system ID from your AWS console and use it to create a new K8S storage class:...
Usually, you’ll use the Splunk Operator to run Splunk on K8S. There are some use cases where you might want to run Splunk without the operator though.
As with any deployment, it’s good practice then to add liveness probes to restart Splunk if it’s not healthy anymore.
Furthermore, as the Splunk container needs about one minute to startup, I’ll recommend adding a readiness probe. This ensures that no traffic is sent to a pod as long as Splunk hasn’t been fully started yet....
If you’re running saved searches in Splunk as subsearches inside of the map command, they are bound by the subsearch limitation.
This is an alternative command that doesn’t have this limitation as it starts a new job for each subsearch.
To use it, instead of calling:
| makeresults | map test You’re using:
| makeresults | mapsearch search=test Missing the full flexibility of map, the command also passes each event’s values as input parameters to each called saved search....
Wonder how to use the ISO8601 format in Splunk?
Simplified extended ISO8601 is for example used in Javascript’s toISOString function.
It’s a great way (readable and to timezone agnostic) to exchange timestamps between Splunk and Splunk Apps.
Here’s how it’s done: