Originally Posted by JerseyLawyer
The only problem with that response is that in the "real world" it's likely to be followed by, "Please step out of the car, sir." The question is always whether you're willing to go to court to vindicate your rights, or whether it's easier to just give in and get it over with.
Also remember that if the police have a reasonable suspicion that "criminal activity is afoot" they can briefly detain you to investigate, and if they have probable cause to believe you're guilty of any crime, they can arrest you. So, I don't know, if you're parked in a handicapped space, look out. :P They can talk to you, and ask for your consent to a search, with no suspicion at all. If they've arrested you, they might be able to search your car, either as a search incident to lawful arrest or under the automobile exception. New Jersey has limited the former with regards to cars, but it depends on your jurisdiction.
So, like many things, "it depends." I wouldn't consent to the search either, but then realize there's at least some chance they will do it anyway.