CodeSignal, which makes skills assessment and AI-powered learning tools, recently released an interesting new benchmark study on the performance of AI code assistance against human developers. The big headline is that many models are outperforming the average developer and are starting to catch up to the top developers. However, this irresistible clickbait was backed up by much more than a headline, and there are some very tangible takeaways.
How CodeSignal Evaluates Developers
Before we get into the results, let’s talk about the methodology. CodeSignal makes skills testing and evaluation tools for developers. So if you were asked to take a skills assessment during a hiring process, it may have come from CodeSignal. Often these tests require you to actually write code (say, 40 to 60 lines), and to answer important questions about the development process. CodeSignal now has a dataset of 500,000 developers who have taken the test. This rich set of data allows the company to have a good feel for developer skills, areas of competency and the like.
In this case, the assessment gave a bunch of LLMs the same test to see how they compared to humans. But what was so interesting was that CodeSignal also measured LLM efficacy using different numbers of examples. In the LLM prompt engineering world, this is called “few-shot” engineering (as opposed to “many-shot”), and it’s a valuable way for models to deliver more precise results. To some extent, trialing a coding task based on just a few examples also mimics how developers learn, because when they get stuck they will seek examples from peers or Google.
This is a particularly intriguing test given that it is (a) not vendor-sponsored, and (b) built on a huge set of control data cultivated over years. These are the results:
The results suggest that three shots (examples) yielded the most optimal results for the LLMs. It was not known how many “shots” humans took, since they got as many shots as they wanted and their tests were constrained by an overall time limit. Here is the three-shot LLM data compared to human benchmarks:
Key Takeaways
Going through the results—and drawing from my own long experience leading software development efforts—has led me to a few conclusions.
- There’s a big difference between small and large models, with one exception. For the most part, smaller models earned lower scores. To put it another way, it seems that experience and skill matter to the models just like they do to distinguish between average and top human developers. However, it also should be noted that the new (small) OpenAI o1-Mini model was a notable exception—so something may be changing here.
- I myself have tried some few-shot prompt-engineering exercises in both academic and professional settings. I have also talked with many people on this topic. The biggest takeaway I have is that if the goal is to create some Java code, for example, I can now either do prompt engineering using few-shot examples or I can just write in Java. Both seem to take a similar amount of time. So I guess the real question is not which is better, but which is better based on my skills. If I am a good prompt engineer and a lousy Java developer—or vice versa—the choice is going to be clear.
- While this test creates a very interesting discussion point, the output raises an issue that’s even more profound. It’s not humans versus AI; it’s humans using AI to augment themselves. So the ideal result would be a line that shows Top Candidates collaborating with AI (such as with agents), creating a new top line that is twice as long as the one for Top Candidates. But to achieve that type of result, developers will need to cultivate the new skill set of working with AI. The good news is that CodeSignal can help with that. It has a new toolset called the AI-Assisted Coding Framework that can help developers transition to working with AI to accelerate their skills and results.
When I saw the headline for the CodeSignal article, I just knew I had to read it, but I was pleasantly surprised to see the methodology and results. The protocol made sense to me and was able to paint a picture of how far we have come, while also suggesting how far we can still go as we learn to harness AI. It is also a reminder that while the big stewards of AI have a distinct capital advantage, the only way AI will become real in both the consumer and the enterprise space is through efforts—like those from CodeSignal—that establish best practices and pragmatic approaches.
Moor Insights & Strategy provides or has provided paid services to technology companies, like all tech industry research and analyst firms. These services include research, analysis, advising, consulting, benchmarking, acquisition matchmaking and video and speaking sponsorships. Of the companies mentioned in this article, Moor Insights & Strategy currently has (or has had) a paid business relationship with Google.
