Even inside the fandom, being 'furry' means something different to each person. For some, it's an appreciation for art and subject matter that is not human-centered. Others enjoy the occasional escape from their daily lives through role-play or by wearing a suit that shows the world another part of themselves. Yet others might believe that their 'furry' form is their true form; that their human body is some sort of mistake.
I don't think that I have ever started a conversation with a furry related subject. Not once have I worn any suit, clothing, tail or other item to identify myself as one. It's not my style. I also don't use bumper stickers and, unless it's a gift, refuse to wear advertising on my clothing.
That said, my family and closest friends know this about me. I have gone as far as to tell them that though I am content within this body, I would much rather be a dragon.
This all happened over time; one person at a time. These people got to know me as a person first, before some stereotypical label could cast a shadow on their opinion of me. For them, it's just an additional eccentricity to add to the list.
Honestly, worrying about people viewing me as 'furry' seems trivial to me. Maybe that's because I survived being labeled "homosexual/faggot/queer. . ." at a time and place where people were publicly frowned on, harassed, arrested, beaten, and/or killed for living this life.
This is one of those cases where the fear of exposure might be more stressful in the long run than than any repercussions from just being yourself. If one is part of a strongly conservative family or community, it would be wise to approach the subject more carefully.