I wondered if enjoying both things may be a problem!!! If I really, really had to make a choice I suppose it would be the latter although I have thoroughly enjoyed my limited time on friends 'scopes (some years ago, other side of the world) just looking around our tiny universe etc.
I've been an avid photographer for years now so my trusty DSLR is like my third arm, hence the reason for using it, or at least wanting to use it and to cut down on any other unnecessary expense.
In all honesty my line of thought was "I'm certain that I'm going to enjoy it so why not buy a reasonable setup to begin with then I wouldn't have the urge to 'upgrade' after a very short time and, in effect, spend even more money".
Hmmm, let me find some 2nd hand gear somewhere...