You can pick up a VDH kit with bowl, soap, and brush in most pharmacy sections for around $12, and it's a good kit to start out with. The brush will lose a few hairs for the first several shaves, but it's what a lot of guys use (myself included), especially to start out. The bowl is great for holding your soap and building lather. The soap is pretty mild too.
You can also dig around antique stores, flea markets, and swap meets to find razors, cups, brushes, and other odds and ends, though Buyer Beware; some of these vendors think that because it's old, it's gold!