I think that usually when people say “algebra over a ring” they assume that ring to be commutative, so that the word “bilinear” in “bilinear multiplication” is useful. It’s possible to define an algebra over a non-commutative ring as a bimodule (rather than left module or right module) equipped with a bilinear multiplication, but I have rarely seen this used.
The definition the parent poster used (or intended to use, but wrote the wrong way around, I believe) was that an algebra over a non-commutative ring is just an algebra over its commutative centre. (In which case, we’re still really just talking about algebras over commutative rings).