undefined | Better HN

0 pointssfmike3mo ago0 comments

Everything is still based on 4 4o still right? is a new model training just too expensive? They can consult deepseek team maybe for cost constrained new models.

0 comments

elgatolopez3mo ago

Where did you get that from? Cutoff date says august 2025. Looks like a newly pretrained model

FergusArgyll3mo ago

> This stands in sharp contrast to rivals: OpenAI’s leading researchers have not completed a successful full-scale pre-training run that was broadly deployed for a new frontier model since GPT-4o in May 2024, highlighting the significant technical hurdle that Google’s TPU fleet has managed to overcome.

- https://newsletter.semianalysis.com/p/tpuv7-google-takes-a-s...

It's also plainly obvious from using it. The "Broadly deployed" qualifier is presumably referring to 4.5

ric2b3mo ago

How is that a technical hurdle if they obviously were able to do it before?

It's probably just a question of cost/benefit analysis, it's very expensive to do, so the benefits need to be significant.

SparkyMcUnicorn3mo ago

If the pretraining rumors are true, they're probably using continued pretraining on the older weights. Right?

1 more reply

verdverm3mo ago

Apparently they have not had a successful pre training run in 1.5 years

fouronnes33mo ago

I want to read a short scify story set in 2150 about how, mysteriously, no one has been able to train a better LLM for 125 years. The binary weights are studied with unbelievably advanced quantum computers but no one can really train a new AI from scratch. This starts cults, wars and legends and ultimately (by the third book) leads to the main protagonist learning to code by hand, something that no human left alive still knows how to do. Could this be the secret to making a new AI from scratch, more than a century later?

WhyOhWhyQ3mo ago

There's a scifi short story about a janitor who knows how to do basic arithmetic and becomes the most important person in the world when some disaster happens. Of course after things get set up again due to his expertise, he becomes low status again.

1 more reply

verdverm3mo ago

You can ask 2025 Ai to write such a book, it's happy to comply and may or may not actually write the book

https://www.pcgamer.com/software/ai/i-have-been-fooled-reddi...

ssl-33mo ago

Sounds good.

Might sell better with the protagonist learning iron age leatherworking, with hides tanned from cows that were grown within earshot, as part of a process of finding the real root of the reason for why any of us ever came to be in the first place. This realization process culminates in the formation of a global, unified steampunk BDSM movement and a wealth of new diseases, and then: Zombies.

(That's the end. Zombies are always the end.)

2 more replies

georgefrowny3mo ago

An software version of Asimov's Holmes-Ginsbook device? https://sfwritersworkshop.org/node/1232

I feel like there was a similar one about software, but it might have been mathematics (also Asimov: The Feeling of Power)

barrenko3mo ago

Monsieur, if I may offer a vaaaguely similar story on how things may progress https://www.owlposting.com/p/a-body-most-amenable-to-experim...

armenarmen3mo ago

I’d read it!

ijl3mo ago

What kind of issues could prevent a company with such resources from that?

verdverm3mo ago

Drama if I had to pick the symptom most visible from the outside.

A lot of talent left OpenAI around that time, most notably in this regard would be Ilya in May '24. Remember that time Ilya and the board ousted Sam only to reverse it almost immediately?

https://arstechnica.com/information-technology/2024/05/chief...

Wowfunhappy3mo ago

I thought whenever the knowledge cutoff increased that meant they’d trained a new model, I guess that’s completely wrong?

rockinghigh3mo ago

They add new data to the existing base model via continuous pre-training. You save on pre-training, the next token prediction task, but still have to re-run mid and post training stages like context length extension, supervised fine tuning, reinforcement learning, safety alignment ...

astrange3mo ago

Continuous pretraining has issues because it starts forgetting the older stuff. There is some research into other approaches.

brokencode3mo ago

Typically I think, but you could pre-train your previous model on new data too.

I don’t think it’s publicly known for sure how different the models really are. You can improve a lot just by improving the post-training set.

catigula3mo ago

The irony is that Deepseek is still running with a distilled 4o model.

blovescoffee3mo ago

Source?

j / k navigate · click thread line to collapse

0 comments

elgatolopez3mo ago

Where did you get that from? Cutoff date says august 2025. Looks like a newly pretrained model

FergusArgyll3mo ago

- https://newsletter.semianalysis.com/p/tpuv7-google-takes-a-s...

It's also plainly obvious from using it. The "Broadly deployed" qualifier is presumably referring to 4.5

ric2b3mo ago

How is that a technical hurdle if they obviously were able to do it before?

It's probably just a question of cost/benefit analysis, it's very expensive to do, so the benefits need to be significant.

SparkyMcUnicorn3mo ago

If the pretraining rumors are true, they're probably using continued pretraining on the older weights. Right?

1 more reply

verdverm3mo ago

Apparently they have not had a successful pre training run in 1.5 years

fouronnes33mo ago

WhyOhWhyQ3mo ago

1 more reply

verdverm3mo ago

You can ask 2025 Ai to write such a book, it's happy to comply and may or may not actually write the book

https://www.pcgamer.com/software/ai/i-have-been-fooled-reddi...

ssl-33mo ago

Sounds good.

(That's the end. Zombies are always the end.)

2 more replies

georgefrowny3mo ago

An software version of Asimov's Holmes-Ginsbook device? https://sfwritersworkshop.org/node/1232

I feel like there was a similar one about software, but it might have been mathematics (also Asimov: The Feeling of Power)

barrenko3mo ago

Monsieur, if I may offer a vaaaguely similar story on how things may progress https://www.owlposting.com/p/a-body-most-amenable-to-experim...

armenarmen3mo ago

I’d read it!

ijl3mo ago

What kind of issues could prevent a company with such resources from that?

verdverm3mo ago

Drama if I had to pick the symptom most visible from the outside.

A lot of talent left OpenAI around that time, most notably in this regard would be Ilya in May '24. Remember that time Ilya and the board ousted Sam only to reverse it almost immediately?

https://arstechnica.com/information-technology/2024/05/chief...

Wowfunhappy3mo ago

I thought whenever the knowledge cutoff increased that meant they’d trained a new model, I guess that’s completely wrong?

rockinghigh3mo ago

astrange3mo ago

Continuous pretraining has issues because it starts forgetting the older stuff. There is some research into other approaches.

brokencode3mo ago

Typically I think, but you could pre-train your previous model on new data too.

I don’t think it’s publicly known for sure how different the models really are. You can improve a lot just by improving the post-training set.

catigula3mo ago

The irony is that Deepseek is still running with a distilled 4o model.

blovescoffee3mo ago

Source?

j / k navigate · click thread line to collapse