The prompt would also maybe be better if it encouraged variety in diagrams. For somethings, a flow chart would fit better than a sequence diagram (e.g., a durable state machine workflow written using AWS Step Functions).
I don’t think the outright dismissal of AI is smart. (And, OP, I don’t mean to imply that you are doing that. I mean this generally.)
I also suspect people who level these criticisms have never really used a frontier LLM.
Feeding in a whole codebase that I’m familiar with, and hearing the LLM give good answers about its purpose and implementation from a completely cold read is very impressive.
Even if the LLM never writes a line of code - this is still valuable, because helping humans understand software faster means you can help humans write software faster.
Cold read ability for this particular tool is still an open question. As others have mentioned, a lot of the example tutorials are for very popular codebases that are probably well-represented in the language model's training data. I'm personally going to test it on my private, undocumented repos.
IMHO, Ai text additions are generally not valuable and I assume, until proven wrong, that Ai text provides little to no value.
I have seen so many startups fold after they made some ai product that on the surface level appeared impressive but provided no substantial value.
Now, I will be impressed by the ai that can remove code without affecting the product.
I'll just wait for a winner to shake out and learn that one. I've gotten tired of trying AIs only to get slop.
Honestly, I wonder if I'm living in some parallel universe, because my experience is that "most engineers" are far from that position. The reactions I'm seeing are either "AI is the future" or "I have serious objections to and/or problems with AI".
If you're calling the latter group "the outright dismissal of AI", I would disagree. If I had to call it the outright dismissal of anything, it would be of AI hype.
> I also suspect people who level these criticisms have never really used a frontier LLM.
It's possible. At my workplace, we did a trial of an LLM-based bot that would generate summaries for our GitHub PRs. I have no idea whether it's a "frontier" LLM or not, but I came out of that trial equally impressed, disappointed, and terrified.
Impressed, because its summaries got so many details right. I could immediately see the use for a tool like that: even when the PR author provides a summary of the PR, it's often hard to figure out where to start looking at the PR and in which order to go through changes. The bulleted list of changes from the bot's summary was incredibly useful, especially because it was almost always correct.
Disappointed, because it would often get the most important thing wrong. For the very first PR that I made, it got the whole list of changes right, but the explanation of what the PR did was the opposite of the truth. I made a change to make certain behavior disabled by default and added an option to enable it for testing purposes, and the bot claimed that the behavior was impossible before this change and the PR made it possible if you used this option.
Terrified, because I can see how alluring it is for people to think that they can replace critical thinking with AI. Maybe it's my borderline burnout speaking, but I can easily imagine the future where the pressure from above to be more "efficient" and to reduce costs brings us to the point where we start trusting faulty AI and the small mistakes start accumulating to the point where great damage is done to millions of people.
> Even if the LLM never writes a line of code - this is still valuable, because helping humans understand software faster means you can help humans write software faster.
I have my doubts about this. Yes, if we get an AI that is reliable and doesn't make these mistakes, it can help us understand software faster, as long as we're willing to make the effort to actually understand it, rather than delegating to the AI's understanding.
What I mean by that is that there are different levels of understanding. How deep do you dive before you decide it's "deep enough" and trust what the AI said? This is even more important if you start also using the AI to write the code and not just read it. Now you have even less motivation to understand the code, because you don't have to learn something that you will use to write your own code.
I'll keep learning how to use LLMs, because it's necessary, but I'm very worried about what we seem to want from them. I can't think of any previous technological advance that aimed to replace human critical thinking and creativity. Why are we even pursuing efficiency if it isn't to give us more time and freedom to be creative?
It sounds like the tool (as it's currently set up) may not actually be that effective at writing tutorial-style content in particular. Tutorials [1] are usually heavily action-oriented and take you from a specific start point to a specific end point to help you get hands-on experience in some skill. Some technical writers argue that there should be no theory whatsoever in tutorials. However, it's probably easy to tweak the prompts to get more action-oriented content with less conceptual explanation (and exclamation marks).
Sometimes it explains things like I am a child and sometimes it doesn't explain things well enough. I think fixing this just by a simple prompt change won't work - it may fix it in one part and make things worse in the other part. This is a problem which I have with LLM: you can fine-tune the prompt for a specific case but I find it difficult to write a universally-working prompt. The problem seems to be LLM "does not understand my intents", like it can't deduce what I need and "proactively" help. It follows requirements from the prompt but the prompt has to (and can't) handle all situations. I am getting tired of LLM.
Ensure the tone is welcoming and easy for a newcomer to understand{tone_note}.
- Output only the Markdown content for this chapter.
Now, directly provide a super beginner-friendly Markdown output (DON'T need ```markdown``` tags)
So just a change here might do the trick if you’re interested.
But I wonder how Gemini would manage different levels. From my take (mostly edtech and not in English) it’s really hard to tone the answer properly and not just have a black and white (5 year old vs expert talk) answer. Anyone has advice on that?
This really shows off that the simple node graph, shared storage and utilities patterns you have defined in your PocketFlow framework are useful for helping the AI translate your documented design into (mostly) working code.
Impressive project!
See design doc https://github.com/The-Pocket/Tutorial-Codebase-Knowledge/bl...
I changed it to use this line:
api_key=os.getenv("GEMINI_API_KEY", "your-api_key")
and I changed it to use a different model:
model = os.getenv("GEMINI_MODEL", "gemini-2.5-pro-preview-03-25")
I used this on a few projects from my employer:
- https://github.com/prime-framework/prime-mvc a largish open source MVC java framework my company uses. I'm not overly familiar with this, though I've read a lot of code written in this framework.
- https://github.com/FusionAuth/fusionauth-quickstart-ruby-on-... a smaller example application I reviewed and am quite familiar with.
- https://github.com/fusionauth/fusionauth-jwt a JWT java library that I've used but not contributed to.
Overall thoughts:
Lots of exclamation points.
Thorough overview, including of some things that were not application specific (rails routing).
Great analogies. Seems to lean on them pretty heavily.
Didn't see any inaccuracies in the tutorials I reviewed.
Pretty amazing overall!
https://github.com/mooreds/prime-mvc-tutorial
https://github.com/mooreds/railsquickstart-tutorial
https://github.com/mooreds/fusionauth-jwt-tutorial/
Other than renaming the index.md file to README.md and modifying it slightly, I made no changes.
Edit: added note that there are examples in the original link.
Like at least one other person in the comments mentioned, I would like a slightly different tone.
Perhaps good feature would be a "style template", that can be chosen to match your preferred writing style.
I may submit a PR though not if it takes a lot of time.
I bet in the next few months we'll be getting dynamic, personalized documentation for every library!! Good times
from ollama import chat, ChatResponse
def call_llm(prompt, use_cache: bool = True, model="phi4") -> str: response: ChatResponse = chat( model=model, messages=[{ 'role': 'user', 'content': prompt, }] ) return response.message.content
The tutorial on requests looks uncanny for being generated with no prior context. The use cases and examples it gives are too specific. It is making up terminology, for concepts that are not mentioned once in the repository, like "functional api" and "hooks checkpoints". There must be thousands of tutorials on requests online that every AI was already trained on. How do we know that it is not using them?
I did however archive the wiki that it generated for the project I work on: https://web.archive.org/web/20240815184418/wiki.mutable.ai/g...
(The images aren't working. I believe those were auto-generated class inheritance or dependency diagrams.)
* The first paragraph is pretty good.
* The second paragraph is incorrect to call pw_rpc the "core" of Pigweed. That implies that you must always use pw_rpc and that all other modules depend on it, which is not true.
* The subsequent descriptions of modules all seemed decent, IIRC.
* The big issue is that the wiki is just a grab bag summary of different parts of the codebase. It doesn't feel coherent. And it doesn't mention the other 100+ modules that the Pigweed codebase contains.
When working on a big codebase, I imagine that tools like mutable.ai and Pocket Flow will need specific instruction on what aspects of the codebase to document.
https://news.ycombinator.com/item?id=42542512
The latter is what this thread claims ^
I think it should be possible to extract some more useful usage patterns by poking into related unit tests. How to use should be what matters to most tutorial readers.
Here is an example feature map for the Spot robot SDK from Boston Dynamics with 100k lines of code: https://product-map.ai/app/public?url=https://github.com/bos...
You built in in one afternoon? I need to figure out these mythical abilities.
I've thought about this idea few weeks back but could not figure out how to implement it.
Amazing job OP
You can tell that because simonw writes quite heavily-documented code an the logic is pretty straightforward, it helps the model a lot!
https://github.com/Florents-Tselai/Tutorial-Codebase-Knowled...
https://github.com/Florents-Tselai/Tutorial-Codebase-Knowled...
An LLM can't magically figure out your motivation behind doing something a certain way.
It’s a pretty foolproof way for smart political operators to get out of a relatively dreary - but high leverage - task.
AI doesn’t complain. It just writes it. Makes the whole task a lot faster when a human is a reviewer for correctness instead of an author and reviewer.
Can see some finetuning after generation being required, but assuming you know your own codebase that's not an issue anyway.
Put in postgres or redis codebase, get a good understanding and get going to contribute.
The burden of understanding still is with the engineers. All you would get is some (partially inaccurate at places) good overview of where to look for.
I think this could be solved with an “assume the reader knows …” part of the prompt.
Definitely looks like ELI5 writing there, but many technical documents assume too much knowledge (especially implicit knowledge of the context) so even though I’m not a fan of this section either, I’m not so quick to dismiss it as having no value.
* Previously, it was simply infeasible for most codebases to get a decent tutorial for one reason or another. E.g. the codebase is someone's side project and they don't have the time or energy to maintain docs, let alone a tutorial, which is widely regarded as one of the most labor-intensive types of docs.
* It's always been hard to persuade businesses to hire more technical writers because it's perenially hard to connect our work to the bottom or top line.
* We may actually see more demand for technical writers because it's now more feasible (and expected) for software projects of all types to have decent docs. The key future skill would be knowing how to orchestrate ML tools to produce (and update) docs.
(But I'm also under no delusion: it definitely possible for TWs to go the way of the dodo bird and animatronics professionals.)
I think I have a very good way to evaluate this "turn GitHub codebases into easy tutorials" tool but it'll take me a few days to write up. I'll post my first impressions to https://technicalwriting.dev
P.S. there has been a flurry of recent YC startups focused on automating docs. I think it's a tough space. The market is very fragmented. Because docs are such a widespread and common need I imagine that a lot of the best practices will get commoditized and open sourced (exactly like Pocket Flow is doing here)
You might get two or three tutorials built for yourself inside the free 25/day limit, depending on how many chapters it needs.
Do you have examples of LLMs running tutorials you can share?
I haven't used it, but it looks like it's in the same space and I've been curious about it for a while.
I've tried my own homebrew solutions, creating embedding databases by having something like aider or simonw's llm make an ingests json from every function, then using it as a rag in qdrant to do an architecture document, then using that to do contextual inline function commenting and make a doxygen then using all of that once again as an mcp with playwright to hook that up through roo.
It's a weird pipeline and it's been ok, not great but ok.
I'm looking into perplexica as part of the chain, mostly as a negation tool
PS: The generated "design documents" are 2k+ lines long. This seems like a great way to exceed quotas.
DEFAULT_INCLUDE_PATTERNS = { ".py", ".js", ".jsx", ".ts", ".tsx", ".go", ".java", ".pyi", ".pyx", ".c", ".cc", ".cpp", ".h", ".md", ".rst", "Dockerfile", "Makefile", ".yaml", ".yml", } DEFAULT_EXCLUDE_PATTERNS = { "test", "tests/", "docs/", "examples/", "v1/", "dist/", "build/", "experimental/", "deprecated/", "legacy/", ".git/", ".github/", ".next/", ".vscode/", "obj/", "bin/", "node_modules/", ".log" }
Looks inside
REST API calls
The hype is that AI isn’t a tool but the developer.
Thanks buddy! this will be very helpful !!
It seems a trifle... overexcited at times.
I wonder why all examples are from projects with great docs already so it doesn't even need to read the actual code.
True
> It's more like you wrote a prompt.
False
> I wonder why all examples are from projects with great docs already so it doesn't even need to read the actual code.
False.
This: https://github.com/browser-use/browser-use/tree/main/browser...
Became this: https://the-pocket.github.io/Tutorial-Codebase-Knowledge/Bro...
If you trained a model from scratch to do this I would say you "built an AI", but if you're just calling existing models in a loop then you didn't build an AI. You just wrote some prompts and loops and did some RAG. Which isn't building an AI and isn't particularly novel.
> look inside
> it’s a ChatGPT wrapper
With the rise of AI understanding software will become relatively easy
Do you have anything in mind? Are you familiar enough with any of those codebases to suggest something useful?
The task will be much more interesting if there is not a good existing tutorial that the LLM may have trained on.
OS kernel: tutorial on how to write a driver?
OpenZFS: ?
https://github.com/openzfs/zfs/graphs/contributors
I would have preferred to see what would have been generated without my guidance, but since you asked:
* Explanations of how each sub-component is organized and works would be useful.
* Explanations of the modern disk format (an updated ZFS disk format specification) would be useful.
* Explanations of how the more complex features are implemented (e.g. encryption, raid-z expansion, draid) would be interesting.
Basically, making guides that aid development by avoiding a need to read everything line by line would be useful (the ZFS disk format specification, while old, is an excellent example of this). I have spent years doing ZFS development, and there are parts of ZFS codebase that I do not yet understand. This is true for practically all contributors. Having guides that avoid the need for developers to learn the hard way would be useful. Certain historical bugs might have been avoided had we had such guides.
As for the others, LLVM could use improved documentation on how to make plugins. A guide to the various optimization passes would also be useful. Then there is the architecture in general which would be nice to have documented. Documentation for various esoteric features of both FreeBSD and Linux would be useful. I could continue, but I the whole point of having a LLM do this sort of work is to avoid needing myself or someone else to spend time thinking about these things.
# Use the Session as a context manager
with requests.Session() as s:
s.get('https://httpbin.org/cookies/set/contextcookie/abc')
response = s.get(url) # ???
print("Cookies sent within 'with' block:", response.json())
I may miss the error, but could you elaborate where it is?
I tried this for some very small decompilation projects, and it was cute at best.
Then I sent it a boot loader. I should have posted it on a ceral box for better results.
Is someone going to suggest that I check the dissembly into git hub and watch it make a tutorial?