No, state refresh is justifiably slow, it's a lot of API calls, which can only be done so quickly, and it can be skipped (with care) if desired.
Once you eliminate that with `-refresh=false` most of the time is taken building the graph. For our largest config that can take almost 5 minutes for each run, and throwing hardware at it doesn't really help; it doesn't seem to make much use of more than one core. Some TF version updates improve it, but more often the performance gets worse.
Recent relevant issues:
https://github.com/hashicorp/terraform/issues/27523
https://github.com/hashicorp/terraform/issues/26355