As for which library, they provide mostly the same functionalty at the bottom but differ in the high level feature set and how they're organized. My take:
* jQuery - Most popular, best for 'ajax sprinkes' apps but less great for full-on applications
* prototype - Prototype hacking outside of ES5 feature backfilling is evil
* mootools - The community cares a lot about their visuals but I got burned with a number of bugs caused by them taking shortcuts for size. This was 4 years ago so the kinks are probably worked out, but there are other options so meh.
* yui3 - My favorite. Built around a selector api, fine grained module system, custom event based, and the UI layer is very much about progressive enhancement. The downside is that there's not a lot of community support.
* sproutcore - the absolute best data layer, UI layer wants to own the page
* dojo/yui2/closure - I don't like these because I think they're too heavy or too much like java
My specific problem with jQuery is that every app over 1k lines or so tends to go spaghetti. The module/event systems in dojo and yui and the data binding in sproutcore guides you to a more modular system which makes for a more pleasant application building experience. There isn't really a technical reason you couldn't do it with jQuery but I've never run across an elegant, modular jQuery codebase. If anybody has one, I'd love to see how it's structured.
Your mileage may vary, by the way, but I think it cuts about 90% of the pain out of AJAX development... leaving 90% to go.
That being said, you'll find much more people fluent in jQuery or Prototype.js than in MooTools.
I'm also specifically annoyed at how it tries to modify the HTMLElement prototype, since that's not allowed in IE < 8, and caused some weird bugs for me (sure, it should be mutable everywhere, but let's be realistic: IE < 8 has a huge market share, and should have code consistency. After all, that's what JS frameworks are for, amongst other things).
I used to use prototype.js and once I used jQuery I never looked back.
Props to prototype.js though as it was one of the first standard libs that did a really good job abstracting away the crappy parts of JS (browser deps).
Whatever you want to do - you can do in both - it all depends on the size. I find Prototype better for big projects where you can use it's class system. It feels like Ruby as well.
jQuery on the other hand is small and gets the job done, but without a plan on how to structure your code things can get really messy.