I'm the ward exec. secretary. I have a ragged goatee, I get my hair cut every 3 months, and I wear colored shirts, and I almost never wear a suit coat/sportsjacket (esp. in the summer).
Nobody has said anything to me directly. Certainly many members have probably wondered why I have my calling.