> This of course assumes you believe in time travel, which I don't.

When you're writing a story with magic in it,  one thing you have to do
is set down your rules for how magic works and stick with it.

This lends your imaginary magic credibility and versimilitude.

That doesn't mean magic is real - it's a literary device

So is time travel.