I actually went through a few adapters to find one that worked, because some of them (e.g. the one Apple makes) actually only wire up a DAC inside the phone to the physical 3.5mm connector. The Palm phone doesn't have a DAC though ...
Maybe the one I have (by a company called Baseus, which I've never heard of before) just sucks and it shouldn't be drawing as much power as it does. Thanks for the link to that video, might be some good alternative in there!