I have ridden XC and trail bikes, down dh lines. Lets just say it ended when the xc bike need constant part replacements, wheels etc. Then the trial bike ended, when I snapped the frame in 2. Xc was a ht and the trail bike was dual sus.
Still, it can be done, and there is no limit in riding an XC bike on any trail. It is about how comfortable and the amount of fun you want to have.
Op, home you many happy miles with whatever you decide on.