chenxi9649

Hi HN, my nephew is 11, and I'm planning on giving him my old laptop.

However, I don't want him to JUST play games 24/7 but for him to do something productive like learning how to code. And I'll meet with him once per week to check up onhim.

My dilemma is that AI has completely changed how I personally code(and learn things), and all of the existing courses/curriculums for kids just feel a bit archaic...

Obviously, I can't just tell my nephew to go talk to chatgpt all day long and learn, without a project plan/target, he's gonna drift off(as I would too). I'm also thinking, since my main goal here is for him to just learn, maybe we should even just start with basic math/science instead of straight into coding?

I don't have any kids myself, so I'm curious to learn how all of the parents on HN are educating their kids today! Anything is helpful :)

I have a question for those who deeply understand LLMs.

From what I understand, the leap from GPT-2 to GPT-3 was mostly about scaling - more compute, more data. GPT-3 to 4 probably followed the same path.

But in the year and a half since GPT-4, LLMs have gotten significantly better, especially the smaller ones. I'm consistently impressed by models like Claude 3.5 Sonnet, despite us supposedly reaching scaling limits.

What's driving these improvements? Is it thousands of small optimizations in data cleaning, training, and prompting? Or am I just deep enough in tech now that I'm noticing subtle changes more? Really curious to hear from people who understand the technical internals here.

Recently, we saw Etched that basically implemented transformers in verilog. Before that was Groq, which had some specialized hardware for working with tensors. Both of which can deliver 10-100x faster inference speeds than your 8xH100s.

I'm vaguely aware of the fact that training is a lot more complex than inferencing, so the fruits of further optimizing hardware are not as low hanging nor rewarding.

But nevertheless, I'm curious to hear a bit more about the details from folks that work in those fields. Is it worth it? Is it possible? What's the complexity here?