Real example · Build apps (vibe coding)
Give Devin 20 real coding tasks and count what works
What they did
Three engineers at Answer.AI spent a month handing Devin more than 20 tasks: new projects, research, reading existing code and changing existing projects.
How it went
They counted 3 successes, 14 failures and 3 inconclusive results, and could not predict which tasks would succeed. Wins included pulling Notion data into Google Sheets; failures included a Railway deployment where it invented unsupported features. The Braintrust integration was finished with Cursor instead.
Worth knowing
A January 2025 test of an early Devin; the team says the tasks it could do were small enough to do faster by hand. Cognition's own later reports show a higher merge rate on narrow work.
Try it yourself with Devin
Replace anything in [brackets] with your details.
Task for the coding agent: in the repo [repo], [one narrow change, e.g. add input validation to function X]. Success means: tests in [test folder] pass and no other files change. If you hit an error you cannot fix in two attempts, stop and tell me what you tried instead of adding workarounds.Read the original
Thoughts On A Month With Devin (Answer.AI) ↗
answer.ai · · Company blog
Attributed to the source linked below; not our own test.
Share Share on X Share on WhatsApp
