This assumption has unfortunately led to countless security issues, at least in the past. The nosniff header (see
https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...), was created because of this and should be added.
While this probably works, you should also add a restrictive CSP (using the sandbox directive).
Forcing the download (via Content-Disposition header) would likely be even better, but it is annoying for users.