The other idea was a bit more theoretical: If you know only a handful tokens are valid, then calculating the logits of the other tokens in the forward pass is wasteful as they won't affect the sampling process. However, it's probably not worthe the cost to optimize that as it only affects the last layer and might be mostly amortized by SIMD parallel processing anyway.

I think there are lots of boilerplate sequences like '":{' or '":[' or '", "', etc - though they might already be compressed into a single token if the tokenizer was trained on enough JSON.

There are also situations where the schema would only allow a specific field name as the next token, e.g. if it was the only remaining valid and required field, or if fields have to be output in a specific order.