I too strugled with this question ever since I discovered my affinity for shemales. And I don't think theres is a straight answer. I found some consolation in knowing that there are others like me out there.
On a side note and related to this, throughout history there have been many accounts of straight men that induleged in sex with other men. Greeks for one were well renovened as "boy lovers". Also I reccount watching a documentary on History channel "Taboo sexuallity" and a segment about a "holyday" in India when straight married men would go out en masse and have sex with trannies, and this is perfectly socially acceptable.
In conclusion, it's like 'whatever floats your boat' thing.
/rant.