Study: Gumdrop

Finding ways to make developers' lives better is incredibly important to us. Our GUMDROP study looks to see improvements to GitHub Copilot’s code suggestions by improving model context.
COMPLETE
Cartoon cat in a spacesuit joyfully floating in a colorful cosmic scene with planets, stars, and vibrant clouds.
Speaker presenting a research paper titled 'Improving Performance of Commercially Available AI Products in a Multi-Agent Configuration' to an audience.
Presented at GitHub Universe 2024 by Cory Hymel
First page of a research paper titled 'Improving Performance of Commercially Available AI Products in a Multi-Agent Configuration' with author names and affiliations.

Improving Performance of Commercially Available AI Products in a Multi-Agent Configuration

Abstract: In recent years, with the rapid advancement of large language models (LLMs), multi-agent systems have become increasingly capable of practical application. At the same time, the software development industry has developed a number of new AI-powered tools that improve the software development lifecycle (SDLC).

Academically, much attention has been paid to the role of multi-agent systems in the SDLC. And, while single-agent systems have frequently been examined in real-world applications, we have seen comparatively few real-world examples of publicly available commercial tools working together in a multi-agent system with measurable improvements.

In this experiment we test context sharing between Crowdbotics PRD AI, a tool for generating software requirements using AI, and GitHub Copilot, an AI pair-programming tool. By sharing business requirements from PRD AI, we improve the code suggestion capabilities of GitHub Copilot by 13.8% and developer task success rate by 24.5% -- demonstrating a real-world example of commercially-available AI systems working together with improved outcomes.
Logo of arXiv with a stylized red and gray 'X' between 'ar' and 'iv'.
Read

Contributors:

Crowdbotics company logo featuring a stylized arrow inside a square followed by the text Crowdbotics.Microsoft logo with four colored squares in red, green, blue, and yellow next to the company name in gray text.GitHub logo text in black.
Speaker presenting on requirements encoding to an audience with a large screen showing a flowchart beginning with 'Idea'.Man giving a presentation on requirements encoding with a diagram on a large screen and audience seated in front.Man in light shirt presenting statistical tables about GitHub Copilot's feature development success and code suggestion acceptance rate to an audience.Bald man with a beard wearing a light shirt speaking and gesturing during a presentation in front of an audience.Speaker presenting the Crowdbotics GitHub Copilot extension to an audience in a conference room.Man presenting in front of a purple screen displaying '78% Fail due to poor requirement management' with audience silhouettes in the foreground.