Yup. Interestingly, actual speakers work either way too. Reverse the polarity on them and you can use them as a crappy microphone.

You observe leaves shaking when there is wind (a correlation). To test for causality you need to make an intervention. You still the leaves - the wind keeps going. You block the wind - the leaves stop shaking. There clearly is an asymmetry here. The physical equations may be symmetric, but this doesn't justify calling a microphone "speaker".