Beyond the Hype of AI Agents in the SOC

While the entire year of 2025 and the year prior, and inevitably the year(s) after, were and will be dominated by hype around AI and Agents, few areas, especially within cybersecurity, have seen as much buzz as SecOps.

Security operations (SecOps) and the SOC have been a key target for aspiring Agentic AI startups, industry leaders, investors, practitioners, and researchers, who are building on years of challenges, including alert fatigue, burnout, and the constant barrage of cybersecurity incidents.

But, below the marketing hype, fundraising, slide decks, and promises of an Agentic SOC panacea, where do we truly stand?

Up until now, data to help analyze this question has been limited, which is why I am really excited to dive into and cover the recent Cloud Security Alliance (CSA) publication “Beyond the Hype: A Benchmark Study of AI Agents in the SOC.”

---

---

The report dives into quantitative results to bring realism to discussions about AI Agents in the SOC, while also covering the attitudes and perceptions of this emerging Agentic AI use case, as well as feedback and discussions on where the industry is headed as this space matures.

To explore this space, CSA partnered with Dropzone AI, a leading vendor focusing on AI SOC analysts and autonomous investigation and response.

So, let’s take a look and see what they found!

---

---

Key Takeaways

I love digging into the details, but I know some folks also want the tl;dr, so let’s start with the key takeaways. They utilized simulated SOC environments and alert escalations to look at the impact of AI SOC agents on security analyst performance.

This makes sense, given that much of the hope and promise of AI Agents in the SOC focuses on helping to deal with alert fatigue, improving metrics such as mean-time-to-detection/response (MTTD/MTTR), and facilitating more effective SOC operations.

The key question is, can AI Agents improve a SOC analyst's speed, accuracy, and consistency over traditional methods and toil?

To answer this question, CSA and Dropzone conducted a study involving nearly 150 participants in various SOC and IR roles and experience levels. Some participants conducted manual activities, while others used the AI-assisted platform.

As it turns out, Agentic AI SOC platforms aren’t just hype, their use improves a SOC analyst's accuracy and speed, and also leads to participants viewing AI in cyber investigations more favorably after the experience.

And we’re not talking about trivial improvements in performance, but exponential ones, as those using the AI-assisted platform saw 45-61% faster with AI. Not only did they conduct investigations more quickly, but those using the AI-assisted platform also produced more robust and detailed reports.

This means Agentic AI SOC platforms, such as Dropzone, can help mitigate SOC fatigue for practitioners while also improving organizational security outcomes in terms of driving down investigation timelines and making reporting more insightful and actionable as well.

Another notable takeaway was that the use of the AI-assisted SOC platform consistently led to overwhelmingly positive perceptions among those involved in its use.

This is important from multiple perspectives, both from that of the practitioner, who are often skeptical of new technologies and methodologies, especially in the security community where we tend to be risk adverse and late adopters, but also for those building these Agentic AI SOC platforms either as a startup, incumbent or investor, as user perceptions and experience have a significant impact on potential adoption and usage.

These perceptions stood out when those involved in the study provided feedback on the AI platform and frequently used words that align with not only the user experience (e.g., helpful and insightful) but also with many of the marketing claims we see associated with Agentic AI SOC platforms and vendors (e.g., efficiency and time/cost savings).

Diving Deeper

Now that we’ve touched on the key takeaways and findings, let’s dig a bit deeper into the publication and study to gather more details. One thing I found interesting about the report is that it focused on “AI Augmentation”, meaning it was looking to boost the performance of SOC Analysts, which is much different than replacement, and instead is looking at making the practitioners more effective by leveraging AI and Agents, as opposed to being replaced by them.

To explore the use of AI Agents in the SOC, the study involved several different and common scenarios, such as alerts related to AWS S3 buckets and others tied to Microsoft Entra ID failed login alerts. These are logical examples to use for the exercise, as alerts and incidents involving data storage, credential compromise, and abuse are commonly associated with security incidents and malicious activities.

The manual group, those without access to the Dropzone AI Platform, still had tools at their disposal, particularly CSP native services such as AWS’s GuardDuty and Microsoft’s Sentinel.

Who participated?

Now, before folks tune out from the results, suspecting it just involved junior participants, that isn’t the case at all. In fact, the majority of participants had several years, and even up to a decade, of experience as a SOC analyst.

Now, remember that the study involved examining key metrics, including accuracy, speed, completeness, and response length. These are relevant both for the quality of investigating alerts and mitigating organizational risks.

Let’s take a look at accuracy first. They defined this as whether the participants correctly identified if the investigation required further action based on the scenario. In both scenarios, those using the AI-assisted platform had improved accuracy, which is essential given that SOC analysts historically struggle with burnout, fatigue, limited bandwidth, understaffing, and more, so knowing when to commit to further investigation and avoid wasted effort and increased cognitive load is key.

Speed is another key factor, as it is crucial for limiting the blast radius of incidents, mitigating lateral movement, enabling quick intervention, reducing the potential impact, and more. As anticipated, those using the AI-assisted platform saw significant speed performance compared to those using manual methods (which are the traditional and most common methodologies currently).

The performance differences were substantial as well, with AI-assisted users investigating the alerts in scenario one at 58 minutes versus 1 hour and 45 minutes, and in 30 minutes compared to 1 hour and 18 minutes in scenario two, compared to those using manual methods. In cybersecurity, where minutes matter and minimizing dwell time is crucial to disrupt data exfiltration, lateral movement, and other threats, these are exponential gains worth noting.

How about the completeness of the investigation, which is akin to saying the quality of the investigations, including gathering the correct data, analyzing it, and providing actionable steps for remediation? The study used a seven-core criteria for the investigations, aligned with the analytical steps expected for the scenarios involved, and captured them as:

• 1 = Clearly addressed

• 0.5 = Implied/partially addressed

• 0 = Not addressed

As seen below, those in the manual group scored lower on completeness across both scenarios. As the report itself notes, this problem becomes increasingly exacerbated over time, resulting in numerous alerts, notifications, investigations, and potential incidents, as SOC analysts become fatigued and the quality of their investigations tends to decline. Those in the AI-assisted category were able to maintain their thoroughness over time, demonstrating that investigative quality can be sustained, and these AI tools can mitigate challenges such as fatigue and burnout for those SOC practitioners working without the AI tooling.

This improved quality is in addition to the enhanced speed and accuracy we discussed above. To build on this insight to highlight the potential impact of alert fatigue, the study also looked at the average response length in words of those using the AI-assisted platform and those investigating the alerts manually, and the group doing it manually saw their investigation step length drop by 27% and their conclusions drop by 20%.

This implies that those with AI assistance were able to sustain more robust responses and documentation of their investigations, potentially capturing more valuable details for their organizations.

Attitudes Towards AI in Security

A key aspect of the potential widespread adoption of AI-assisted SOC platforms involves not just improved outcomes but the perceptions and experiences of the practitioners involved. This includes the practitioners' attitudes toward AI in security more broadly. As I mentioned above, the cyber community has historically been risk-averse and reluctant to be early adopters of emerging technologies.

This is a topic I have addressed in articles such as “Security’s AI-Driven Dilemma.” In that article, I made the case that cybersecurity faces a dilemma in being traditionally a laggard and a late adopter of emerging technologies. With AI, we face the dilemma of perpetuating our slow adoption curve, or leaning in like developers and attackers, or risk falling further behind on the various systemic security issues we struggle with, including in the SOC.

The study in this case with CSA had participants rank their attitudes towards AI in security, and surprisingly, there was a favorable view towards AI in security, with an overall average score of 8.3.

The study continued by having the participants using the AI assistance evaluate the AI SOC agent after the investigations and looked at overall satisfaction, perceived usability and the potential for adoption within SOC environments, including recommending the tool to others. The results were very positive, including:

• An average score of 8.7 to recommend AI-enabled platforms to their peers in SOC or IR roles, and included a NPS of 53, which is a “strong” rating in terms of user satisfaction

• The majority reporting noticeable benefits in their investigation speed with the AI-assisted platform, with 53% saying it significantly sped up their investigation and 41% saying it somewhat sped it up

  

  The potentially most significant takeaway when it comes to perceptions of AI in security is the fact that those using the AI SOC agent walked away with 94% of them saying they view AI more positively than they did before the study.

  

  Closing Thoughts

  As I mentioned leading into this article, AI is poised to transform our digital environments. From fundamentally changing the way modern software is written, the way attacker workflows and TTPs are conducted, and, as this study demonstrates, the way SecOps and SOC activities are performed.

  We already know organizations historically struggle with factors such as alert fatigue, burnout, lengthy MTTD/MTTR, and other factors that expose organizations to risk.

  This study and publication from the Cloud Security Alliance and Dropzone offers excellent insights into the potential value of AI-assisted SOC platforms, including enhanced investigation activities, streamlined processes, mitigation of organizational risks, and improved SOC analyst experiences. This helps take the hype around the Agentic SOC from hype to demonstration.