undefined | Better HN

0 pointsaledalgrande4y ago0 comments

Great hacks, although you have to be aware of the trade-offs:

1. if you choose the wrong subset, you'll find a non optimum local min

2. still risk dead ends when expanding the model and lengthen the time to finding that out

3. a lot of public models are made from inaccurate datasets, so beware

Overall you have to start somewhere though, and your points are still valid.

0 comments

m_ke4y ago

1. The small subset is to test that your training pipeline works and converges near 0 loss.

2. Sure, but for most new hacks like mixup, randaugment and etc the results usually transfer over. Problem with deep learning is that most of the new results don't replicate so it's good to have a way to quickly validate things.

3. The lower level features are usually pretty data agnostic and transfer well to new tasks.

aabaker994y ago

1. Gradient descent almost always finds a non optimum local min (it is not guaranteed to find a global min).

agnosticmantis4y ago

Isn’t the current best practice to train highly over-parametrized models to zero training error? That’d be a global optima, no?

Unless we’re talking about the optima of test error.

aabaker994y ago

If you find a zero in a non negative function, I would call that a global minima, yes.

aledalgrandeOP4y ago

Yeah but depending on the data you might have even worse results, selecting the right subset to be representative is really important.

jhgb4y ago

Would a random sample be representative? Statistically this seems to be the case for any large N. In fact it's not clear to me that any other sample would be more representative.

1 more reply

j / k navigate · click thread line to collapse

0 pointsaledalgrande4y ago0 comments

Great hacks, although you have to be aware of the trade-offs:

1. if you choose the wrong subset, you'll find a non optimum local min

2. still risk dead ends when expanding the model and lengthen the time to finding that out

3. a lot of public models are made from inaccurate datasets, so beware

Overall you have to start somewhere though, and your points are still valid.

0 comments

m_ke4y ago

1. The small subset is to test that your training pipeline works and converges near 0 loss.

3. The lower level features are usually pretty data agnostic and transfer well to new tasks.

aabaker994y ago

1. Gradient descent almost always finds a non optimum local min (it is not guaranteed to find a global min).

agnosticmantis4y ago

Isn’t the current best practice to train highly over-parametrized models to zero training error? That’d be a global optima, no?

Unless we’re talking about the optima of test error.

aabaker994y ago

If you find a zero in a non negative function, I would call that a global minima, yes.

aledalgrandeOP4y ago

Yeah but depending on the data you might have even worse results, selecting the right subset to be representative is really important.

jhgb4y ago

Would a random sample be representative? Statistically this seems to be the case for any large N. In fact it's not clear to me that any other sample would be more representative.

1 more reply

j / k navigate · click thread line to collapse