Enterprise Security Health Check: Getting the Most out of Your Splunk Subscription

At BlueVoyant, we believe in leveraging all customer-purchased technologies to their fullest extent possible, including Splunk itself. Our Splunk customers have a long legacy developing and using Splunk Enterprise Security and we know how to maximize their security investment — due to our expertise and long-standing relationship with the company.

BlueVoyant’s Enterprise Security Health Check leverages best practices, advanced detection methods, and scalable designs to relevant cybersecurity frameworks like MITRE and Information Technology Infrastructure Library (ITIL), providing the most functional Splunk Enterprise Security (ES) deployment possible.

What do we look for?

• Performant and meaningful searches. With Splunk license structure changes and the move to cloud workload, performances and execution costs should be heavily scrutinized. Our health checks help consolidate searches, ensure they’re relevant, and maximize their performance. This allows for more critical detections to be added while managing costs and expectations.

• MITRE Mappings. Not sure what coverage you have? Mapping security use cases to security detections helps ensure complete and total coverage of your threat landscape. This starts with documentation and enabling the correct tags for the best reporting. Showing your existing MITRE coverage provides your leadership security investment visualization and also helps your team chart where your security roadmap should go next.

• Content development lifecycle. As new and growing threats are ever present, a documented and repeatable lifecycle for creating and deploying accurate content is critical. This process enables easy audits to ensure the correct use cases are being deployed and all teams are aware of the capability. We believe a security operations center (SOC) should never be surprised by a new alert that has no runbook applied.

• Evaluate your ES frameworks. ES is a collection of frameworks that enables a SOC to build, triage, and respond to known alerts. These frameworks require integrations into Active Directory, CMDB, and Threat Intelligence feeds. A third-party review of these integrations for accuracy and efficiencies can help you decide where to invest next.

Here are a few examples of findings we typically run across during our health checks:

Scenario 1: The Zombie Search

One customer was using ES as their primary security information and event management (SIEM) with alerts firing into the Incident Review. During an incident, a few additional correlation searches — which is a search type that looks for defined patterns — were built to support the SOC for detection and escalation purposes. After the incident was completed, these searches were never removed. In addition to continually running, none of the original data was present any longer. Searches consumed the CPU and memory from the Search Head Cluster with no gain to the SOC. Thus zombie searches like this cost the team valuable space on their Splunk license.

Scenario 2: Who’s Seeing That Alert?

Correlation searches can send an email to a distro or person when fired. During a content evaluation, we found 50% of the correlation searches had email addresses for outside parties assigned to them. This was found not to be fraudulent but a result of analyst turnover and no auditing of the correlation searches. We provided a recommendation to remove all the emails and set up additional correlation searches to detect if an analyst added an email address to any correlation searches.

Scenario 3: Don’t Just Allow Anyone!

Allowlisting can be difficult to manage and maintain. We have found several times that the use of OR statements in security protocol language (SPL) is a useful but dangerous tool because it is difficult to audit and review.

One time an allowlist was configured with so many entries that it removed 90% of the environment from the detection. To solve this, we applied a time_entry and retention fields to all allowlist lookups.

Another search ran over all the lookups and removed any entries that were set based on the time it was entered and the retention specification. If the alert fired and was deemed a false positive, another entry was made in the lookup table.

BlueVoyant was able to solve these issues by making sure these searches were always checked and verified through normal processes.

These are just a handful of things we check for and review when working without clients on their ES stack. We focus on making things as efficient as possible, so our customers get the most value out of their security tools.

If you are interested in learning more, contact our sales team to start the conversation.

Matthew Gonter is vice president of technology for Concanon, a BlueVoyant company. Concanon is a global professional services and big data solutions consultancy that specializes in Splunk.