undefined | Better HN

0 pointsisotypic1y ago0 comments

Why does pretraining or not matter in the ISPD 2023 paper? The circuit_training repo, as noted in the rebuttal of the rebuttal by the ISPD 2023 paper authors, claims training from scratch is "comparable or better" than fine-tuning the pre-trained model. So no matter your opinion on the importance of the pretraining step, this result isn't replicable, at which point the ball is in Google's court to release code/checkpoints to show otherwise.

0 comments

negativeonehalf1y ago

The quick-start guide in the repo that said you don't have to pre-train for the sample test case, meaning that you can validate your setup without pre-training. That does not mean you don't need to pre-train! Again, the paper talks at length about the importance of pre-training.

marcinzm1y ago

This is what the repo says:

>Results >Ariane RISC-V CPU >View the full details of the Ariane experiment on our details page. With this code we are able to get comparable or better results training from scratch as fine-tuning a pre-trained model.

The paper includes a graph showing that it takes longer for Ariane to train without pre-training however the results in the end are the same.

negativeonehalf1y ago

See their ISPD 2022 paper, which goes into more detail about the value of pre-training (e.g. Figure 7): https://dl.acm.org/doi/pdf/10.1145/3505170.3511478

Sometimes training from scratch is able to match the results of pre-training, given ~5X more time to converge. Other times, though, it never does as well as a pre-trained model, converging to a worse final result.

This isn't too surprising -- the whole point of the method is to be able to learn from experience.

anna-gabriella1y ago

That does not mean you need to pre-train either. Common sense, no?

j / k navigate · click thread line to collapse

0 pointsisotypic1y ago0 comments

0 comments

negativeonehalf1y ago

marcinzm1y ago

This is what the repo says:

The paper includes a graph showing that it takes longer for Ariane to train without pre-training however the results in the end are the same.

negativeonehalf1y ago

See their ISPD 2022 paper, which goes into more detail about the value of pre-training (e.g. Figure 7): https://dl.acm.org/doi/pdf/10.1145/3505170.3511478

This isn't too surprising -- the whole point of the method is to be able to learn from experience.

anna-gabriella1y ago

That does not mean you need to pre-train either. Common sense, no?

j / k navigate · click thread line to collapse