Reddit mentions of 16GB Combo Memory MS Pro Duo FOR Sony PSP & Sony Cybershot Digital Camera & Sony Webbie

You'll want one (or more) of these paired with a couple of 16GB microSD cards. It costs about $25 to get 32GB of space as opposed to about $75 for a 32GB MS Pro Duo card.

You can also get this which is just one of those adapters with two 16GB microSD cards included. The availability and prices change on Amazon all the time, so just search for "ms pro duo microsd" to see what the current best deals are.

You can get the adapter by itself from here

and here's a very high rated option comes with 2x 8gb micro sd cards and a micro sd to sd adapter

Also check Amazon for used PSPs. Just last week I found and bought a 2001 for $45. It's in great shape and works perfectly. A couple weeks ago, there was a 2001 for $60 including a 4GB memory stick, charger, case and 2 games. Be patient, check a few times a day and you'll find a good deal.

For games, GameStop seems to be clearing out PSP stock, so they're dirt cheap. Amazon also has good deals on used PSP games, and the owner gets more of your money than if you buy from GS. If you spend more than $10 on anything, you're overpaying.

Finally, if you don't have a memorystick, you can get a MicroSD to MS ProDuo adapter and 16GB of SD cards (2x8GB) for about $15.
http://www.amazon.com/Memory-Cybershot-Digital-Camera-Webbie/dp/B0038EWH0G/ref=sr_1_11?ie=UTF8&qid=1334320756&sr=8-11 Don't waste your money on the proprietary MS BS.

This is my first PSP and I'm loving it. You're in for a good time.

Here's one I picked up recently.

https://www.amazon.com/gp/product/B0038EWH0G/ref=oh_aui_detailpage_o01_s00?ie=UTF8&psc=1

Enough other devices I own use MicroSD and don't need ultra-fast read/write that I was more than happy to get a couple cheap 8GB cards with the adapter.

This is the one I have and works good

16GB Combo Memory MS Pro Duo FOR... https://www.amazon.com/dp/B0038EWH0G?ref=ppx_pop_mob_ap_share