Wash your towels in hot water with a cup of vinegar, and then run again in hot water with a half-cup of baking soda. That will strip your towels from all of that residue and mildew smell and will actually leave them feeling fluffy and smelling fresh. (Do not add laundry detergent to either wash. Just once with vinegar and once with baking soda.) good to know for the future.
Okay, this stuff is AMAZING! DIY shower cleaner -- vinegar and dish detergent. Don't worry about the brand or color. We used clear store brand detergent and it was GREAT! NO SCRUBBING required. As long as you can tolerate the smell :)