Do your thing. It is YOUR life. Don't EVER feel sorry for doing exactly what it is that you want to with YOUR life. Let people talk if they want to. Who cares. It's not their life you are living. Real friends, family, and partners will be on your side and encourage you. Anyone else is just a waste of time to worry about, so simply don't! The inner happiness you will find from doing this is unlike anything else.