+1 to the advice above. A good rule of thumb is to start with the highest grit possible. As Thisisclog said, lower grits (for example 80, 120, even 220) will leave distinct scratches that you'll want to remove at a later stage with higher grit. And that can take a LOT of work to do. So if you can, best to avoid making the scratches in the first place. I will typically start at around 400 or higher just to see how it works, and only go to a lower grit if absolutely necessary.

If it is active (i.e. red) rust, much of it can be removed with a razor blade. A single-edge disposable blade is great for this and it should go without saying that a straight razor blade should *never* be used in that way!

There are other techniques that can help depending on the type of rust you're dealing with. If you post some pics, we might be able to give some more suggestions better tailored to your situation.

Good luck