Yeah, pdroid as in its Android 2.3 form is wont work. Before OpenPdroid, I had to manually merge changes into the ASOP source by hand and do a little tweaking. You should be all good with takju. I have a Verizon Galaxy Nexus (toro) and build AOSP fine with that.
All the proprietary binaries are on Google's site[1] for Nexus devices or can be pulled from your device under /vendor/ (in the past, google sometimes failed to put up a few like the camera or bluetooth so had to get those via your device or a trusted repository). Alternatively, you can pull them from a repository maintained by a some of the more well known Android ROM modders using the directions I put up here (https://github.com/yareally/android_proprietary_files). Also you'll have to get a google apps package (or alternatively build it yourself) if you compile AOSP since you won't have access to the Android Market and such otherwise (stock AOSP resembles what you get in the emulator). A premade google apps package I would trust (and use myself when I don't want to build my own) can be located on goo.im[2]. They'll say Cyanogenmod, but they'll work for stock AOSP just the same.
[1] https://developers.google.com/android/nexus/drivers
[2] http://goo.im/gapps
