Reddit mentions of MS PRO DUO Adapter. Converts Two MicroSD or MicroSDHC Cards To MS PRO DUO. Compatible with PSP. Support up to 32GB

Then I'd recommend a Sony PSP. Preferably the 3000 or Go models (though a 2000 works fine, it just has less vibrant colors; avoid the 1000 and the European-only E1000 Street since they have less hardware capabilities). Just hack the PSP with ProCFW using this guide (you'll need a valid memory card) and then it can run a bunch of different emulators; it does GB/GBC/GBA especially well. You can get the emulators I use here, and put them in your memory card at ?:\PSP\GAME (where ? is your PSP's drive letter after connecting it to your computer with a miniUSB cable like the old flip phones, not microUSB or USB type C like a smartphone). Homer's RIN is for GB/GBC games and has cleaner scaling than anything else, but has compatibility probs with a few games. EmuMaster (which plays GB/GBC and also Sega GameGear/MasterSystem) will handle some of the ones that RIN can't, but has ugly bilinear scaling. TempGBA is just the best GBA emulator you can get for the system and nothing else compares (it'll even play the Golden Sun games, with very slight frameskip in the overworld). You'll have to find/dump your own roms, though.

Also, you can get a microSD to MS Pro Duo adapter for cheap instead of paying up the arse for an official memory card (Warning: Doesn't work with the Go since it takes a Memory Stick Micro (M2) instead). Make sure your microSD cards aren't too high of a capacity, though; a PSP only officially supports up to 32GB total (that is, 2x16GB in the adapter I linked; these should work perfectly), and my friend got his working with 2x32GB Class-10 SanDisk cards, but I hear certain brands won't work at that capacity and we haven't found anything above those 2x32GB cards that work at all (they are seen as 'damaged' and unable to be formatted in the PSP). Not that you'll need that much space for GB/GBC/GBA games anyway... :P

On the other hand, most cheap smartphones nowadays can run the My Boy! and My OldBoy! emulators just fine, even for 'problem' games like Golden Sun, so getting a cheap tracfone and a decent bluetooth controller might be an even better idea than going for an aging PSP, and still cheaper than most of those clone consoles. Of course, that comes with Android's audio latency problems, and you won't be able to use save states in those emulators unless you get the paid versions (which are $4.99 and $3.99, respectively) but I think it'd still be better than getting any of the clone systems you mention, especially since Android emulators are so easily updated. Note that there are also other free GB/GBC/GBA emulators for Android, I just know from personal experience that the MyBoy ones work well...

A bit of indirect advice:

Size: Memory Stick cards (Pro Duo specifically) come in 4, 8, 16, and 32GB variants, so you'll most likely be looking for a 32GB card.

Speed: There are a variety of speeds out there, depending on the manufacturer, but as the PSP can't read faster than about 10MB/s anyway, you don't need to worry about high-end MS cards that are really designed for digital cameras and computers.

Brands: I would recommend going with a Sony or SanDisk card [1]. However, as with a lot of memory media these days you need to be extremely careful of who you buy from. Flash memory is one of the most commonly counterfeited products on the market and it's very easy for legitimate companies to inadvertently stock counterfeit cards. So whatever card you decide to buy, make sure you test it and see that it performs at the approximate speed indicated and that you have the full amount of space as described on the packaging.

Alternatives to expensive cards: There are also micro-SD to Memory Stick Pro Duo adapters (though they may not be reliable using two cards in tandem instead of one). I've never tried it out on a PSP, but supposedly they work and buying two cheaper micro-SD cards and an adapter might be cheaper and easier to replace in the event that something happens (because ultimately all of these small electronics aren't going to be perfectly reliable).

[1] Sony has more stringent requirements about where they source their memory from, so Sony-branded memory should be above average. SanDisk makes their own memory so they can pick from the best price/performance of their own inventory without having to rely on a memory maker to supply them with reliable parts.

I have a question though, would he ever consider using this MS PRO DUO adapter so that we could use 2 micro SDs? This is great though, I am definitely interested in this.

I have this memory card adapter with 32+64GB SD cards.

The regular charger and Sony battery that comes with the PSP should be good enough. Most batteries you find are aftermarket junk and lots of people have bad luck with them.

I also have one of these coming in the mail soon. It fits 2000 and 3000 models only.

Here is the one I used along with 2 64gb cards. I fuckin love it.

One thing to remember, put one card in the a slot at a time, format that card through your PSP, then format the other card in that same slot you previously used. I think after all of that, I put both cards in, and had to do one more format. Good to go!