The part that often ends up being tricky is the fact that I need to send data from processor A to processor B. And I want to send as little data as possible. So one of the first sources of bugs is that when I do my gather-scatter I make a mistake mapping a value to a coordinate. In shared-memory you never have to do this mapping back and forth, so its not an issue.
Next issue is related to the fact that I don't want to ever block waiting for data. There are a variety of models for handling this. I can do a non-blocking receive, and do some work waiting for the data to arrive. This is often another source bugs as people will often do work that depends on the new data, but they chug along without it. Add the new data when they get it, and alas their computation is already hosed.
And the last common error in this case is handing the data off to the wrong object (or processor) or being confused as to which data you're receiving at any given point in time.
Now all of these can be handled by simply being careful, and using some good programming practices. But they are just simple, if not grossly naive, examples of issues you have with traditional message passing that don't exist in shared memory.
