I think so.
In our Garrison, we have a former Marine who served in Iraq and knows first hand what things look like when they're used for combat in the desert. He's always telling me that white plastic would not look like the way we've painted ours to look. I would have to agree with him, but I think think that some humidity would escape from the inside of our armor, and create 'wet spots' on the armor where sand/dirt would accumulate. I would also think even with casual wear, there would be a light dusting.
One thing to keep in mind, is that while it sounds a little backward - less IS more. The human eye can detect the slightest blemishes and notices when a Stormtrooper is dirty compared to filthy.
Right there with you. Trying to attain that "middle ground" myself... Maybe we shoudl come up with the "Sewer Trooper" designation for those who are overly grungy..