It is in Morrow County.
The closest towns are Hepner and Spray. It is definitely in Eastern Oregon. You can take 84 most of the way there or go through Condon, which is a whole lot more scenic. I don't recall exactly how long it took to get there from Portland, but it takes a while (4 hours?).
Morrow County Parks
There is camping or you can rent these crazy little A frame cabins that feature bunks, electric light/heat and not much else. Gas grills on the porch. Apparently they were taken from that crazy cult that took over Antelope (the one with all the Rolls Royces).
I've only gone with the ATV, but I know there are some play areas just for 4x4s including a big rock garden. The Land Rover guys go there a lot.