Y'know this question really comes to a head when you're in Japan, you're a guy, and you're buying the latest issue of Ribon so that you can read the latest chapter of Aishiteruze Baby and get the cute little Yuzuyu alarm clock that comes free with it (no I don't actually use the clock, but it was a nice souvenier). Actually in the anime club that I'm part of it's very normal for us guys to be obsessed with shoujo romance shows. Hana Yori Dango is one of our favorites (and the ratio of guys to girls in our club is about 15 - 1) as is Aishiteruze Baby, and Uchuu no Stellvia used to be a big hit too. Mostly I'd say the only anime that are strictly gender specific are things like Gravitation... guys just shouldn't touch that series, or Fake. On the flip-side, girls probably shouldn't watch the ultra-guy-ish stuff with all kinds of fanservice. Other than that most anime is up in the air as far as who should or should not be expected to watch it. I personally choose my anime based on how good the story is and whether or not there's at least one cute character involved.
Of course this doesn't mean everyone should go around telling non-otakus that their favorite show is about a high-schooler trying to raise a five year old...