undefined | Better HN

0 pointsSemanticStrengh3y ago0 comments

This is not the same.. The masks are randomized and lossy. Although yes there is potential for a transformer specially trained to segment prefixes/affixes/suffixes, it might augment some of its encoding abilities, see e.g spanbert for a related example of opportunity.

0 comments

MauranKilom3y ago

What do you mean with "lossy"? What information is being lost? Or do you just mean that there isn't necessarily a unique way to encode a given string?

SemanticStrenghOP3y ago

I mean that information is being lost https://arxiv.org/abs/1906.08237 See xlnet for the rethoric https://www.microsoft.com/en-us/research/publication/mpnet-m... Or mpnet which attempt to combine the best of both worlds information wise but still find that masked modeling is much less useful than autoregressive.

j / k navigate · click thread line to collapse

0 pointsSemanticStrengh3y ago0 comments

0 comments

MauranKilom3y ago

What do you mean with "lossy"? What information is being lost? Or do you just mean that there isn't necessarily a unique way to encode a given string?

SemanticStrenghOP3y ago

j / k navigate · click thread line to collapse