In this case I would guess the issue is not with speed but feedback. The user did something, nothing changed, so they thought it was broken.
Instead of the delay you could add a toast or some text with near the button indicating that the action actually happened.
