If you've ever checked out /r/OriginalXbox there were some pretty neat hardware mods like LCD screens on the front of the console that display current playing content.
The PS2 can play games off of an SD card with MX4SIO through the memory card slot, through USB, through SMB with a Raspberry Pi, and through an internal HDD. You don't even need a working disc drive or a hard mod.
The PS Vita has a rocking homebrew scene.
The 3DS had neat custom themes and being able to use QR codes to download games is pretty nifty.
There's also SBC Gaming devices like the RG350 and Retroid Pocket 3+ that can play a wide array of systems.
Basically I have a hard time narrowing it down. What about you? Is there one console or device you keep coming back to?