I'm in that 'middle ground' - and planning on staying in it lol. I'm not going to tell you its perfect, but there's nothing inevitable about the 'worst case' scenario horror stories either. Everyone is different, and depending on the level of need for treatments (some like me need less if you already have a moderately 'girly' looking form to start with) and the dose you require of whatever combin\tion - plus whether or not you choose castration - all have an effect on the outcome with individuals.
There is nothing certain about your 'cock shrivelling to almost nothing' - mine hasnt and neither have those of a couple of other (older) women I know. It is tru that the testicles are more often affected, although of course if they are still there they are still producing testosterone.
Psychological factors can't be ignored either - and play a big part in desire and response in all people, not only TS. My sexual response remains adequate, not so many spontaneous erections but it works when it has to. Talk it over with a lot of docs and other TS - its mainly all about balance.
And good luck!