1 trick to do this is by using Sadhu’s Astral Body Explosion knockdown effect. The knockdown effect can move the tower around.
But, from your description
Even if the opponent were using Sadhu’s Astral Body Explosion knockdown to move your tower to their base, they would need to do that at least (roughly) 30+ times and it would take a while to do that unless if they had multiple Sadhus and utilized Chronomancer’s Pass.