A standard protocol to handle and discard low-effort, AI-Generated pull requests (opens in new tab)

It would be nice to have some kind of forever patch mode on these git forges, where my fork (which, let's say, is a one line change) gets rebased on top of the original repo periodically.

Good idea, though I'm not sure how to enforce it. You can ask an AI for that and then rewrite it in your own words.

Indeed. I have been receiving clearly AI generated job applications out of the blue and they tend to point to their contributions to github projects so some of these must be getting through.

Someone somewhere once decided that it was a great idea to add how many github stars a project that you have contributed to is a useful metric during the hiring process and now those projects get swamped with junk.

"I can do math really fast"

"okay, what's 137*243"

"132,498"

"not even close"

"but it was fast"

I think the idea is you stick a link to this page in your PR-closed comment.

While I am with you on hoping, someone shamelessly PRing slop just is not going to feel shame when one of their efforts fail. It’s like being mean to a phone scammer, they just hang up and do it again

The bigger issue is that that kind of statement is highly manipulative, and indicates someone who is playing politics instead of focusing on results.

The better response is to call the bluff, something along the lines of: "Running an open-source project is quite time consuming. Please don't waste our time with emotional manipulation to get your way. Instead, take the time to understand why your LLM-generated pull request is not useful. You can start by understanding that we have access to LLMs too, and realize that a significant amount of work needs to happen after an LLM proposes changes."

Aside from anything else, you have good engineering instincts, and I wish more people in the industry were like you.

Here’s what you could do if you somehow found yourself with an LLM-generated change to a codebase implementing a feature you want, and you wanted to do the most do expedite the implementation of that feature without disrespecting and alienating maintainers:

1. Go through all changes, understand what changed and how it solves the problem.

2. Armed with that understanding, write (by hand) a high-level summary of what can be done (and why) to implement your feature.

3. Write a regular feature request, and include that summary in it (as an appendix).

Not long ago I found myself on the receiving end of a couple of LLM-generated PRs and partly LLM-generated issue descriptions with purported solutions. Both were a bit of a waste of time.

The worst about the PRs is when you cannot engage in a good-faith, succint and quick “why” sort of discussion with the submitter as you are going through changes. Also, when PR fails to notice a large-scale pre-existing pattern I would want to follow to reduce mental overhead and instead writes something completely new, I have to discard it.

For issues and feature requests, there was some “investigation” submitter thought would be helpful to me. It ended up a bit misleading, and at the same time I noticed that people may want to spend the same total amount of effort on writing it up, except so now part of that effort goes towards their interaction with some LLM. So, I asked to just focus on describing the issue from their human perspective—if they feel like they have extra time and energy, they should put more into that instead.

If it happens at work, I obviously still get paid to handle this, but I would have to deprioritise submissions from people who ignore my requests.

> but I did not feel that I knew the codebase enough to be able to actually assess the correctness of the change.

The good engineering approach is to verify that the change is correct. More prompts for the AI does nothing, instead play with the code, try to break it, write more tests yourself.

> I did not feel that I knew the codebase enough to be able to actually assess the correctness of the change.

> I want to do good engineering, not produce slop, but for 1 min of prompting, 5 mins of tidying, and 30 mins of review, we might save 2 days of eng time.

I don't really understand where do "2 days of engineering time" come from.

What exactly would prevent someone who does know the codebase do "1 min of prompting, 5 mins of tidying, and 30 mins of review" but then actually understand if changes make sense or not?

More general question: why do so many slopposters act like they are the only ones who have access to a genAI tool? Trust me, I also have access to all this stuff, so if I wanted to read a bunch of LLM-slop I could easily go and prompt it myself, there is no need to send it to me.

Related link: https://claytonwramsey.com/blog/prompt/ (hn discussion: https://news.ycombinator.com/item?id=43888803 )

Do you use the library? if yes, test it in prod or even staging with your patch, then submit the review

To be entirely fair "sorta working, solving a problem but not really all that great for the rest of the codebase" PRs are human thing too.

The problem is AI generating it en masse, and frankly most people put far less effort that even your first paragraph and blindly push stuff they have not even read let alone understood

> Interestingly, the pro-AI folks almost universally doubled down and said that I should use AI more to gain more confidence – ask how can I test it, how can we verify it, etc – to move my confidence instead of changing how review works.

Well, it's not terrible at just getting your bearings in the codebase, the most productive use I got out of it is treating it as "turbo grep" to look around existing codebases and figure out things

> Interestingly, the pro-AI folks almost universally doubled down and said that I should use AI more to gain more confidence – ask how can I test it, how can we verify it, etc – to move my confidence instead of changing how review works.

I think this is a good suggestion, and it's what I usually do. If - at work - Claude generated something I'm not fully understanding already, and if what has generated works as expected when experimentally tested, I ask it "why did you put this? what is this construct for? how you will this handle this edge case?" and specifically tell it to not modify anything, just answer the question. This way I can process its output "at human speed" and actually make it mine.

>I thought that was an interesting idea that I hadn't pushed enough, so I spent a further hour or so prompting around ways to gain confidence, throughout which the AI "fixed" so many things to "improve" the code that I completely lost all confidence in the change because there were clearly things that were needed and things that weren't, and disentangling them was going to be way more work than starting from scratch.

I feel this so much. In my opinion, all of the debate around accepting AI generated stuff can be boiled down to one attribute, which is effort. Personally, I really dislike AI generated videos and blogs for example, and will actively avoid them because I believe I "deserve more effort".

similarly for AI generated PRs, I roll my eyes when I see an AI PR, and I'm quicker to dismiss it as opposed to a human written one. In my opinion, if the maintainers cannot hold the human accountable for the AI generated code, then it shouldn't be accepted. This involves asking questions, and expecting the human to respond.

I don't know if we should gatekeep based on effort or not. Obviously the downside is, you reduce the "features shipped" metric a lot if you expect the human to put in the same amount of effort, or a comparable amount of effort as they would've done otherwise. Despite the downside, I'm still pro gatekeeping based on effort (It doesn't help that most of the people trying to convince otherwise are using the very same low effort methods that they're trying to convince us to accept). But, as in most things, one must keep an open mind.

> but I did not feel that I knew the codebase enough to be able to actually assess the correctness of the change.

> I want to do good engineering, not produce slop, but for [...]

IFF this is true, you can already stop. This will never be good engineering. Guess and check, which is what your describing, you're letting the statistical probability machine make a prediction, and then instead of verifying it, you're assuming the tests will check your work for you. That's ... something, but it's not good engineering.

> That has to be worth something.

if it was so easy, why hasn't someone else done it already? Perhaps the cost value, in the code base you don't understand isn't actually worth that specific something?

> I could see a few ways forward:

> Send it, but be clear that it came from AI, I don't know if it works, and ask the reviewers to pay special attention to it because of that...

so, off load all the hard work on to the maintainers? Where's that 2 days of eng time your claiming in that case?

> Or Send it as normal, because it passes tests/linters, and review should be the same regardless of author or provenance.

guess, and check; is not good engineering.

> Interestingly, the pro-AI folks almost universally doubled down and said that I should use AI more to gain more confidence – ask how can I test it, how can we verify it, etc – to move my confidence instead of changing how review works.

the pro-ai groups are pro AI? I wouldn't call that interesting. What did the Anti-AI groups suggest?

> the AI "fixed" so many things to "improve" the code that I completely lost all confidence in the change because there were clearly things that were needed and things that weren't, and disentangling them was going to be way more work than starting from scratch.

Yeah, that's the problem with AI isn't it? It's not selling anything of significant value... it's selling false confidence in something of minimal value... but only with a lot of additional work from someone who understands the project. Work that you already pointed out, can only be off loaded to the maintainers who understand the code base...

General follow up question... if AI is writing all the PRs, what happens when eventually no one understands the code base?

I would expect nothing less from the BOFH Task Force.

LLM already did rm -rf the brain of posters of those PRs...

Around 1990 I attended ISO/JTC1 meetings generating standards for data communication. I still recall my surprise over the heated arguments over these words between the UK and the US delegations. (I'm from Denmark). In particular 'shall' and 'should' meant different things in English and American languages. ISO's first standard, ISO 1, states that ISO Standards shall be written in English so we had to do that, US delegation too. Similarly Scott Bradner stated in RFC 2219 how American conventions should be followed for future IETF STDs.

So I'm confident that the word 'shall' has a strong meaning in English; whether it has too in American legalese I cannot tell.

On (possibly weak) counterpoint that I can offer is that in some languages, “must not” is a false friend, easily misinterpreted as “is not required to” (“it is not the case that they must”).

Right. I think when these appear in some documentation related to computing, they should also mention whether it is using these words in compliance with RFC 2119 or RFC 6919.

Must is a strict requirement, no flexibility. Shall is a recommendation or a duty, you should do it. You must put gas in the car to drive it. You shall get an oil change every 6000 miles.

Many legal documents use "may" to say you must. That's why i hate legalese...

I didn't read it as this, what signs do you see?

Resurrecting proof-of-work for pull requests just trades spam for compute and turns open source into a contest to see who can rent the most cloud CPU.

A more useful approach is verifiable signals: require GPG-signed commits or mandate a CI job that produces a reproducible build and signs the artifact via GitHub Actions or a pre-receive hook before the PR can be merged. Making verification mandatory will cut bot noise, but it adds operational cost in key management and onboarding, and pure hashcash-style proofs only push attackers to cheap cloud farms while making honest contributors miserable.

Hashcash: https://en.wikipedia.org/wiki/Hashcash

CAPTCHA: https://en.wikipedia.org/wiki/CAPTCHA

Proof of intelligence might be better.

As if we need wasting even more power

To quote TFA: "...outputs strictly designed to farm green squares on github, grind out baseless bug bounties, artificially inflate sprint velocity, or maliciously comply with corporate KPI metrics".

My best guess: to show on their resume, in the hope it helps to land a job.

If there were any reasonable way to do something like this, I would love to see it.

Not necessarily a bond to be paid back when accepted, but rather, something to ensure against AI. "If you assert this is not AI, insert $10. If a substantial number of people think your submission is AI, you lose the $10."

If you know what you're doing, you can achieve good results with more or less any tool, including a properly-wielded coding agent. The problem is people who _don't_ know what they're doing.

> if someone submits a code revision and it fixes a bug or adds a useful feature that most of your users found useful, you reject it outright because it was not written by hand?

If they didn't read it, then neither will I, otherwise we have this weird arms race where you submit 200 PRs per day to 200 different projects, wasting 1hr of each project, 200 hrs total, while incurring only 8hrs of your time.

If your PR took less time to create and submit than it takes the maintainer to read, then you didn't read your own PR!

Your PR time is writing time + reading time. The maintainer time is reading time only, albeit more carefully.

I advise you read the article, it gives many specific examples of things that qualify for such treatment:

> A 600-word commit message or sprawling theoretical essay explaining a profound paradigm shift for a single typo correction or theoretical bug.

> Importing a completely nonexistent, hallucinated library called utils.helpers and hoping no one would notice.

There's plenty more. All pretty egregious

Yep, communication is pretty cool.

Quis custodiet ipsos custodes?

If the problem is that we don't trust people who use AI without understanding its output, and we base the gate-keeping on tests that are written on AI, then how can we trust that output?

That's not helpful, because:

1: LLMs can write awful tests

2: LLMs can write very useful code, especially when they are working in well-understood areas.

Which comes down to understanding that the LLM is a tool, and it's the job of the programmer to know how to use the LLM and evaluate its output.

Are the PRs not accompanied by test cases? Do the README changes not document the expected benefit?