Leveraging an AI Security Framework
Leveraging an AI Security Framework
An overview of Databricks AI Security Framework
Welcome to Resilient Cyber!
If you’re interested in FREE content around AppSec, DevSecOps, Software Supply Chain and more, be sure to hit the “Subscribe” button below.
Join 5,000+ other readers ensuring a more secure digital ecosystem.
If you’re interested in Vulnerability Management, you can check out my upcoming book “Effective Vulnerability Management: Managing Risk in the Vulnerable Digital Ecosystem” on Amazon. It is focused on infusing efficiency into risk mitigation practices by optimizing resource use with the latest best practices in vulnerability management.
As the industry continues the push to adopt artificial intelligence (AI), security practitioners continue to race to keep up with the businesses explosive rate of adoption. Security is seeking to walk the delicate balance between being a business enabler and not blocking innovation but also implementing sound governance and empowering the business to make risk informed decisions around technology use and integration.
Luckily, there are several resources coming from industry leaders such as the Cloud Security Alliance (CSA), OWASP and now Databricks when it comes to AI security.
For those who are interested, I have previously covered other related resources, such as the OWASP LLM AI Cybersecurity & Governance Checklist and OWASP AI Exchange and interviewed OWASP AI Exchange creator and industry AI Security leader Rob van der Veer on the Resilient Cyber Show here.
In this article we will take a high-level look at the recently released Databricks AI Security Framework (DASF).
DASF was recently published as a comprehensive 80+ page whitepaper, and is intended for not only security practitioners but also AI professionals such as in machine learning, engineering and data science, as well as governance leaders, risk officers and business counterparts interested and concerned about AI security.
Model Types
The paper opens by defining the three broad types of models used in AI. They include predictive ML models, state of the art open models and external models. Predictive models include examples like PyTorch and HuggingFace, state of the art open models include examples such as Llama-2-70B-chat and are foundational to LLM’s and used for fine-tuning of LLM’s. Lastly, external models, or third-party services include foundation models such as OpenAI GPT, Anthropic and others.
AI System Components and Their Risks
The paper advances to discussing the common AI system components and their associated risks. This includes everything from raw data, data prep, cataloging, developing, evaluating and managing models and performing inference requests and responses.
The paper catalogs the number of risks associated with each aspect of the AI system components.
DASF lays out 55 different technical security risks across the 12 components they identify which are commonly deployed by their customers. Each system component is not only associated with potential security risks but also system stages as well, which align with the diagram above.
Let’s briefly touch on the various components, their risks and the stage of the SDLC they would be part of. Since the components are grouped by system stage, we will use that to orient the discussion and cover the associated components and risks therein.
Data Operations
The first system stage identified is Data Operations. This includes the following components and associated risks:
This system stage boasts the highest number of potential security risks, including risks such as insufficient access controls, poor data quality, ineffective encryption, lack of logging and data poisoning among many others.
This stage involves raw data, which the DASF emphasizes i the foundation that all AI functionality builds upon. If the data is compromised in any fashion, it has downstream consequences on all other aspects of AI systems and outputs.
There also needs to be controls in place to securely handle data preparation, properly safeguarding of datasets and properly cataloging and governing the broader data operations.
Model Operations
Model operations is the next system stage identified, which includes the components and risks:
15 risks are identified in the DSAF which apply to model operations, including model drift, malicious libraries, ML supply chain vulnerabilities and model theft.
Any nefarious activities impacting a model can impact outputs. There are also supply chain security concerns, such as the potential for malicious libraries that can impact systems, data and models. Given the widespread use of third-party libraries and open source, this risk is of particular concern.
The DASF also calls out the significant investment required to train ML systems, especially LLM’s. Facing model theft can be a significant risk, erode competitive advantage and squander resource investments.
This emphasizes the need to properly secure ML algorithms, the model building and training process and also the management of models themselves.
Model Deployment and Serving
The model deployment and serving stage includes the components and risks:
Risks called out at this stage include prompt injection, model breakouts, LLM hallucinations and accidental exposure of unauthorized data to models.
Prompt Injection is a particular attack technique that has gained a lot of visibility, and involves users injecting text aimed at altering the behavior of an LLM. It can also be used to bypass safeguards and cause damage.
Hallucinations are another risk that have been in the headlines, due to LLM’s generating incorrect or false outputs or even leaking sensitive data that wasn’t intended to be shared.
Lastly, the risk of exposure of unauthorized data to models is catching attention, as LLM’s and GenAI become a key part of modern organizational infrastructure and software environments accidental exposure of data to models can pose significant risks to organizations impacting proprietary and confidential data.
Operations and Platform
Rounding out the system stages identified in DSAF is Operations and Platform. The components identified and risks here include:
Various risks impact this stage, such as a lack of repeatable standards, poor vulnerability management, a lack of penetration testing and incident response, subpar SDLC and lack of compliance.
It’s worth noting many of these risks apply to many digital enterprise environments and apply to broader application security and governance as well.
Additionally, the specific mention of a lack of compliance highlights the increasingly tense intersection between the prolific growth of AI and increased pushes for regulatory compliance.
Organizations already historically struggle with several of the items highlighted, and the increased potential risks of AI emphasize the need for maturity and rigor in traditional activities like vulnerability management, incident response and implementing a secure software development lifecycle.
Moving Forward
The Databricks DASF represents a robust and comprehensive approach to cataloging the risks and security considerations associated with AI and AI systems.
AI adoption continues to grow rapidly and the DASF represents an excellent resource that diverse stakeholders across organizations can leverage to ensure a secure and compliant approach to AI adoption.
Databricks is a pioneer in this space, leading with their AI security risk workshops, thought leadership and amazing platform to help organizations securely use AI.
For those interested in a deeper dive of the DASF, I strongly recommend checking out the actual paper as it provides and incredibly comprehensive walkthrough of the various lifecycle stages, components and associated risks as well as how the Databricks industry leading platform helps address them for customers.