Cool, makes sense.
Yeah, have you considered maybe looking into just running it on embeddings [1], instead of the imagery itself?
Would save on most of the inference cost, at the cost of flexibility (i.e. you are locked into whatever embeddings have been created).
[1] https://developers.google.com/earth-engine/datasets/catalog/...