I don't ever remember having a bank holiday the same week as terms starts although I'm sure it must have happened.I think it's really meanie, mean-mean starting back straight after the bank holiday.
Your holiday and the kids minus a day, is spoiled thinking you've got to go back straight after.
In my day we'd start on the Thursday if that happened, or the following week.