Cheap is how I started. Ironically hi_bud_gl helped me too, I will explain in a minute.

The way I see it is you can do two things for bare minimum:

Shavette or cheap straight.

Either way get the van der Hagen set from Walmart. The brush is stiff and loses bristles, but after +/- 20 or 30 uses the bristles get split ends, stop shedding and start softenign up (this is good). $10 (I still use the bowl, and the brush from time to time when I need something a little stiffer than badger)

Then get a cheap strop you can cut up- risk on ebay, or reliable from Kenrup (the filly model) either way, $20. (go with Kenrups filly)

Then get your blade- a cheap shavette option is getting a thinning razor and blades for it- some use DE's others you have to find hiarshaper blades for- ebay abounds though. You can get the razor and blades for like $10-$15 total if you look around.
The alternative is a not well known ebay razor that is stained (not rusted) or otherwise cosmetically (but not functionally) impaired. Send it out for honing (someone might offer to hone it free) and you're out $20-$40 bucks total, depending.
Or you can buy shave ready from the classifieds (also $20-$40 depending).

I personally got an ugly blade of the bay and hi_bud_gl offered to hone it for me. So I got the VDH set ($10) a basic basic strop ($20) and a razor ($15) for a total of $45. I also have a dubl duck hair shaper and blades that I use for camping and what not which I got for $5 and blades run $.10 to $1 depending the quantity and shipping and stuff.