AI-Driven Risk-Aware Software Development
The Current Landscape
Diff Risk Score (DRS) is an AI-powered tool developed at Meta to predict the chance that a code change might cause a production incident, known as a SEV. It uses a fine-tuned Llama large language model (LLM) to analyze code changes and related metadata, producing a risk score and flagging risky code snippets.
DRS supports various features that improve product quality, developer efficiency, and computational resource management. One major benefit has been the elimination of extensive code freezes, allowing developers to ship code in periods they previously couldn’t, with minimal negative impact on users or business.
Why Risk Awareness Matters
Software development involves significant risk, especially for complex and large-scale products like those Meta operates. Protecting user experience and business outcomes requires effective risk management tools.
Applying AI to software development processes helps identify potential risks early. DRS was created to predict whether code changes might cause severe incidents, enabling workflows that reduce risk while maintaining productivity.
For instance, during sensitive times like the Cyber 5 holiday shopping week, teams used to implement strict code freezes to avoid incidents. While this improved reliability, it also reduced productivity considerably. DRS enabled a more balanced approach, allowing low-risk changes to proceed during freeze periods, thus protecting users and business while improving development speed.
During a major partner event in 2024, over 10,000 code changes were accepted during a freeze period with minimal production issues, demonstrating how risk-aware development enables ongoing innovation. This approach also reduces the engineering effort spent on incident management.
DRS is just the beginning. Meta now explores 19 different use cases for risk tools, showing the broad potential of risk-aware software development.
Future Directions
The success of DRS has led to new risk-aware features across Meta’s development lifecycle, from planning to post-release monitoring. This growth also inspired the creation of the Risk Awareness Platform (RAP), which offers risk analysis APIs and tool integrations.
- More Risk-Aware Features: The platform currently supports optimizing builds and tests, improving reliability, selecting code reviewers, and analyzing release risks. The goal is to increase innovation speed within reliability limits, expanding the role of risk models and data.
- Configuration Change Risk: Beyond code changes, configuration changes also cause many incidents. RAP now includes models predicting configuration risk, an area still early in research but expected to power new feature sets soon.
- Automated Risk Mitigation: Instead of just flagging risky changes, future AI agents could proactively suggest or create risk-reducing edits for both code in motion (pull requests) and existing code, as well as configuration changes.
- Explainable AI for Risk Scores: Showing clear, natural language explanations for risk assessments aims to help engineers understand and address risks effectively. This feedback loop will improve both risk models and user trust.
Looking Ahead
Risk-aware development is evolving quickly, with practical impacts on product quality and developer productivity. Continued collaboration and learning across the industry will be key to advancing these tools.
Acknowledgements
Special thanks to the many team members and leaders who contributed to the success of DRS at Meta, including Rui Abreu, David Amsallem, Parveen Bansal, Kaavya Chinniah, Brian Ellis, James Everingham, Peng Fan, Ford Garberson, Jun Ge, Kelly Hirano, Kosay Jabre, David Khavari, Sahil Kumar, Ajay Lingapuram, Yalin Liu, Audris Mockus, Megh Mehta, Vijayaraghavan Murali, Venus Montes, Aishwarya Girish Paraspatki, Akshay Patel, Brandon Reznicek, Peter C Rigby, Maher Saba, Babak Shakibi, Roy Shen, Gursharan Singh, Matt Steiner, Weiyan Sun, Ryan Tracy, Siri Uppalapati, and Nachiappan Nagappan.
Your membership also unlocks:
