Serde has an interface that allows failing. That one should fail. There is also another that panics, and AFAIK it will automatically panic on any parser that fails.
Do not try to handle huge values, do not pretend your parser is total, and do not pretend it's a correct value.
If you want to create an specialized parser that handles huge numbers, that's great. But any general one must fail on them.