You know when you get stuck on a problem that you know isn't so difficult but your brain just can't figure out? Yuuuuup.
After too long, this exists:
It's not perfect - can't think of a way of setting position boundaries that transition with scaling. But this thing kinda works enough for now at least. (And I think the zoom origin problem is solved!)