Rendered at 20:09:57 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
mNovak 20 hours ago [-]
Given the $10k price tag for tokens and high rate of bugs (several per minute) they mention, it'd be very interesting to see this experiment run with cheaper models too.
I wonder if we get to a world where a full repo sweep like this is a default Github action after commit.
1over137 19 hours ago [-]
Most C/C++ projects I know don't even run tests with ASan/TSan/UBSan before each commit/merge.
Quarrel 17 hours ago [-]
and in the meantime, just a sweep of the committed code (or the to-be-committed code for lots of us) and the code it interacts with, is increasingly catching lots of problems.
rurban 13 hours ago [-]
Boy, I told DeepSeek V4 Flash Free to find compiler bugs, esp. with the gcc torture test suite, and it did find plenty. For free. To fix them in my compiler it did cost about $40 or so. Cooperate guys just have too much budget in their hands to throw at the rich.
jlebar 19 hours ago [-]
Author here; I'm happy to answer questions, take criticism, etc etc.
ebiederm 17 hours ago [-]
Thank you for posting this.
I had heard LLMs were finding a lot of bugs very quickly and now I can see what that looks like from a user perspective.
eqvinox 17 hours ago [-]
> Codex and I collaboratively wrote a fuzzer.
Why are you using phrasing that equates AI and humans? You used Codex to write a fuzzer. It didn't decide to join you.
derdi 13 hours ago [-]
Why are you using phrasing that equates AI and humans? Codex isn't in a position to decide whether to do work.
Traubenfuchs 10 hours ago [-]
I wonder how much damage all those countless bugs caused in real life.
Does MRI low level code produce wrong images? Do some kind of unexpected http connection quirks happen? Does (LL)M inference produce randomly wrong and non reproduceable output? Graphical artifacts in video games? Application crashes that happen once every billionth request? Security vulnerabilities? Race conditions?
I wonder if we get to a world where a full repo sweep like this is a default Github action after commit.
I had heard LLMs were finding a lot of bugs very quickly and now I can see what that looks like from a user perspective.
Why are you using phrasing that equates AI and humans? You used Codex to write a fuzzer. It didn't decide to join you.
Does MRI low level code produce wrong images? Do some kind of unexpected http connection quirks happen? Does (LL)M inference produce randomly wrong and non reproduceable output? Graphical artifacts in video games? Application crashes that happen once every billionth request? Security vulnerabilities? Race conditions?