Unless we're talking fantasy/sci-fi, I don't see the difference from a third-party perspective. If you draw a Venn diagram they're almost a 100% overlap. Whatever percentage is remaining is, like you said, likely political or based on feelings (psychological). Both could be considered a 'trap' from a third party's perspective. If you're attracted to either, I'd probably assume you're gay or have latent homosexual feelings - which is PERFECTLY FINE. This thread just reminded me of the Family Guy episode about Quagmire's Dad.
Based on the existence of things like Chimera syndrome and studies that have been done, it is far more likely biological than psychological, because we know twin fetuses can be absorbed by one another in the womb, even if they are opposite sexed, and we know trans people's brain structure falls in between that of the male and female brain thanks to said studies. What happens when the brain of one sex is in the body of the opposite? Probably, gender identity issues like that we see in trans people.
As for sexuality, it's not really "gay or straight." After all, there's a whole spectrum of sexualities out there. I, for instance, am attracted to feminine qualities, regardless of sex. Femboys (or traps), attractive. Trans women, attractive. Cis women, attractive. Twinks? Not as much. Trans men? If they're femboys (yes, there are feminine trans men), then yes, otherwise, not really. Likewise, very masculine women and cis men aren't attractive to me. Thick facial and body hair? Too masculine, complete turn off. Bodybuilder women? Starts hitting that too masculine mark for me, so turn off, even though muscular women are a yes otherwise.
I could practically write a thesis on this subject, but doing so would
definitely be crossing the "no politics" rule, and be 100% off thread topic (I acknowledge I'm straddling the line on that one already with this).