I've worn my Apple Watch Ultra on everything except Outlaw Run and WildFire and it hasn't gone off.
I typically don't take my phone on rides.
I wonder if the feature works while it's not paired? I know all the Ultras have cell connectivity on their own, so if that is set up as part of your plan, it should work all by itself.
(Edit to add: I mean, if triggered. It's good that it's not being triggered.)