Wombat State Forest has plenty of what you seek, if you are coming from Melbourne and turn off the Western Freeway to go through to Blackwood and Woodend there are heaps of secluded but accessable dirt roads with suitable camping areas dotted through them. There is a proper campsite near Blackwood itself on the Lerderderg River with toilets but I don't really recommend it, the river is dry and firewood is hard to find, if you want secluded I'd just choose one of the other side roads and see what you can find.
The Lerderderg Gorge Park itself has some nice mountainous areas but I'm not sure how accessable it is nowadays. If you go to the Gorge itself which is opposite Bacchus Marsh on the Western Freeway, be prepared to do some hiking before you find your campsite because they don't allow campers at that end of the gorge anymore. There are creeks there but no rivers.
If you're willing to do some hiking Wilsons Prom is good, from memory you'd have to hike 4 hours to get to the first secluded beach area but it's well worth it if you chose that one. Not sure if you need to book for that.
I've been recommended the Cathedral Ranges too for good camping and there is a river there too, but I'm not sure how good the swimming is. It used to be only deep in smallish spots and that may have changed for the worst since the drought.
It's early days for me starting out again. Ask me again in another year hopefully I'll have plenty more suggestions by then.