One of my old physics professors taught us to think of tensors as "arrays with units." If it's a vector/matrix/higher dimensional array but has physical units, it's probably a tensor. The fact that it has units means it represents something physical which must obey additional constraints (like the coordinate system transformation rule).
Obviously they are not. One is a linear operator, the other is a data structure for implementing computations using that operator. This description extends to all tensors.
It's like saying "queues are not just lists". That is true and also neither insightful nor helpful.
I don't see it as mystifying or complicated, what am I missing?
The coordinate change stuff that physicists talk about stems from observing that a matrix can be used to represent some tensors, but the rule for changing basis changes along with the kind of tensor. So if M is a matrix which represents a linear map and P is a matrix whose columns are basis vectors, then PMP^{-1} is the same linear map as M but in basis P; if on the other hand the matrix M represents a bilinear form as opposed to a linear map, then the basis change formula is actually PMP^T, where we use the matrix transpose. Sylvester's Law Of Inertia is then a non-trivial observation about matrix representations of bilinear forms.
Physicists conflate a tensor with its representation in some coordinate system. Then they show how changing the coordinate system changes the coordinates. This point of view does provide some concrete intuition, though, so it's not all bad. By a coordinate system, I mean a linear basis.
Hope that helps.
The OP also emphasizes the abstract interpretation as providing more intuition than the coordinate transformation rule.
Rather, mathematicians that complain about the physicist's approach just haven't advanced far enough in their studies to understand how vector bundles are associated to the frame bundle ;)
Physicists' tensors = generalization of arrays with units; have to transform according to certain coordinate laws.
Mathematicians' tensors = generalization of arrays, transformation rules don't matter.
That's definitely inaccurate, at least it doesn't match what I think of as tensors in mathematics.
In mathematics, tensors are the most general result of a bilinear opteration. This does imply that they transform according to certain laws: if you represent the tensor using some particular basis, that basis can be expressed in the original vector spaces you multiplied, and choosing a different basis for your vector spaces results in a different basis for your tensors.
By "most general bilinear operation" I am talking about what is expressed in category theory as a universal property... with a morphism that preserves bilinear maps.
Tensors can be over multiple vector spaces or a single vector space (in which case it's typically implied that it's over the vector space and its dual). When you use a vector space and its dual, I believe you get the kind of tensor that physicists deal with, and all of the same properties. Note that while vector spaces and their dual may seem to be equivalent at first glance (and they are isomorphic in finite-dimensional cases), both mathematicians and physicists must know that they have different structure and transform differently.
Something that will throw you off is that mathematicians often like to use category theory and "point free" reasoning where you talk about vector spaces and tensor products in terms of things like objects and morphisms, and often avoid talking about actual vectors and tensors. Physicists talk about tensors using much more concrete terms and specify coordinate systems for them. It can require some insight in order to figure out that mathematicians and physicists are actually talking about the same thing, and figure out how to translate what a physicist says about a tensor to what a mathematician says.
