A little over a year ago, GitLab acquired UnReview, a machine learning-based solution for automatically identifying relevant code reviewers and distributing review workloads and knowledge. Our goal is to integrate UnReview’s ML-powered code review features throughout GitLab, the One DevOps Platform. We checked in with Taylor McCaslin, principal product manager, ModelOps, at GitLab, to find out the impact UnReview has had so far and what comes next.
The idea of applying machine learning to code review was already underway at GitLab before the UnReview acquisition. What was it about ML/AI and automation that seemed a good fit for the code review process? How did the UnReview acquisition affect that strategy?
The acquisition of UnReview gave GitLab a practical way to get started with a really focused value proposition that was obvious to the platform. ML/AI is a lot more than just having a useful algorithm. UnReview and its team gave GitLab talent with experience building MLOps pipelines and working with production DataOps workflows. As a source code management (SCM) and continuous integration (CI) platform, MLOps and DataOps are key ambitions for our ModelOps stage. UnReview is the foundational anchor of our AI Assisted group, and we anticipate developing more ML-powered features with the base that we’ve built integrating UnReview into our One DevOps platform. If it’s something you manually set today within GitLab, we’ll consider suggestions and automations: suggested labels, assignees, issue relationships, etc. You can learn more about our plans on our AI Assisted direction page.
You’re invited! Join us on June 23rd for the GitLab 15 launch event with DevOps guru Gene Kim and several GitLab leaders. They’ll show you what they see for the future of DevOps and The One DevOps Platform.
There were three specific objectives with the UnReview project when you first started:
- Eliminate the time wasted manually searching for an appropriate code reviewer to review code changes.
- Make optimum recommendations that consider the reviewers’ experience and optimize the review load across the team, which additionally facilitates knowledge sharing.
- Provide analytics on the state of code review in the project, explaining why a particular code reviewer is recommended.
Have you had to change or add to these in any way?
We now have Suggested Reviewers running for external beta customers as well as dogfooding it internally. We’ve learned a lot about what makes a good code reviewer. Some of the obvious things like context with the changed files and history of committing to that area of code are obvious. But there are less obvious things like what type of code someone has experience with (front-end or back-end).
We’re finding the concept of recency interesting: the idea that people who more recently interacted with files and functions may be better suited to review the code. Also, people leave companies, and that’s usually not something that can be inferred by the source graph, so we’re working on merging additional GitLab activity data with the recommendation engine.
In addition, we’re thinking a lot about bias in our recommendations. For example, a senior engineer likely has the most commits across a project, but we don’t always want to recommend a senior engineer. The more we work with the algorithm and recommendations, the more nuanced we find it.
Not every organization does code review the same way, so we’re considering building different models for those that have no process versus organizations that have very rigid and hierarchical reviewer requirements. We also have to consider how recommendations interact with other features of the platform like code owners, maintainer roles, and commit access.
We’ve never been more excited about the potential of machine learning within GitLab. Some of the feedback we’ve had from beta customers are “this feels like magic” and that honestly encapsulates what we’re going for. Sometimes the right code reviewer is just a feeling that you can’t quite put your finger on. Through data and a little bit of magic, we may see Suggested Reviewers help speed up workflows, and cut down on back and forth and wasted time trying to find someone to do a great review of your code.
Introducing ML-powered features can come with challenges, especially being GitLab’s first data science feature. Can you speak to some of those challenges and how the team overcame them?
It has been about a year since we closed the transaction. During that time period we’ve introduced a lot of new concepts to GitLab. Access to real-time data within the feature with DataOps extraction and cleaning of platform activity data. We have an end-to-end MLOps pipeline running 100% within GitLab CI that extracts, builds, trains, and deploys the UnReview model, and new observability metrics to know if the whole system is working. These are all foundational concepts that we’ve had to build from the ground up.
Also, we’ve introduced Python to the GitLab tech stack and have to develop new engineering standards and hiring interview practices to find the right talent for this team. We’re now turning the corner of this foundational work and I anticipate that relatively soon we’ll release Suggested Reviewers fully integrated with the platform and UI.
Milestones have been part of the way we’ve sliced up the integration work. We have a variety of internal milestones we’ve been tracking against, including porting the model into GitLab SCM and CI, building the Dataops and MLOps pipelines, and internal and external customer betas. It’s helpful to have these milestones to know what’s most important at any given time and not to get overwhelmed with all the moving pieces. We’re paving a new path with ML-powered features at GitLab, and once we’re done we’ll have a repeatable process and template to replicate over and over with new data science-powered features.
What has been the most surprising thing you’ve encountered or learned since UnReview first debuted?
Code Reviewers are foundational to the software development lifecycle. We thought this would be a really straightforward feature, but it turns out people REALLY care about recommendations. People hate bad suggestions so when the recommendations are wrong, the feedback is fast and furious. But when it’s right, it feels like magic. That really surprised me how positively people respond to a great suggestion.
A lot of GitLab users have asked me what our success metric is for Suggested Reviewers. It should just feel like magic. Maybe you don’t know why someone was chosen, but you just feel they were the right person to review the change. And hopefully that leads to a more thoughtful code review, reduces the back and forth of trying to find someone to review your code, and ultimately creates a better experience end-to-end. A lot of engineers dread code reviews; we want to change that. I hope Suggested Reviewers can take the pain out of the experience and make it something engineers look forward to. That’s the feeling we’re trying to create with our recommendations. Obvious but magic.
What’s next for UnReview specifically and DevOps code review more generally? Where do you see the next big advances happening?
We’re just scratching the surface. There are so many opportunities for recommendations and automations across the platform. We have a lot of data at GitLab, from the source graph, contribution history, CI builds, test logs, security scans, and deployment data. We believe all of this can be integrated together. I’m particularly excited about what we’re calling Intelligent Code Security. The idea is that we will be able to look at your source code as you’re writing it, analyze it for security vulnerabilities, and not only suggest fixes to common security flaws, but also apply that change, run your CI, confirm the build succeeds, confirm the vulnerability was resolved, and possibly even deploy that change, all automatically.
Imagine the future where your code gets more secure automatically while you sleep. That sounds wild, but we have the data to power a feature like this in the future. Suggested Reviewers is just the beginning. We haven’t seen many DevOps platforms fully embrace the data, code, and activity data that they have in a material way. I think we’ll see a lot more in this space moving forward as development platforms identify the massive opportunities to drive efficiencies and remove the frustrating parts of software development from the process.
