I'm no expert or anything like, but I would say the stress would be worse than just leaving him.I've never left him before so bit worried.
My nephew is really big and awkward and I worry he may step on him, so I need to make sure he doesn't actually go near him other option is find somewhere that will take him for a couple of days but that could be quite stressful for him.
He won't starve, probably just sulk and sleep like Tidgy does.
He doesn't flip, does he?