Mathematicians are comfortable with the idea that a matrix can be over any
set, in other words matrix is a mapping from the cartesian product of two intervals [0,N_i) to a given set S. Of course to define matrix multiplication you need at least a ring, but a matrix doesn’t
have to define such an operation, and there are many alternative matrix products, for example you can define the Kronecker product with just a matrix over a group. Or no product at all for example a matrix over the set {Red,Yellow,Blue}.
Tensors require some algebraic structure, usually a vector space.