Yes, you need to be worried. Look at his eyes. He's either in a lot of pain or he's really sick. Take him to the vet. The bump is not in the right place to be an ear abscess, but it looks like an abscess. The vet will need to lance it and get the tortoise on an antibiotic regimen.
Vet ASAP. His eyes are a real clear sign of something bad. Also, maybe it's just the way your hand is holding him, but it looks like his slants upward, which could mean he has MDB... (Metabolic Bone Disorder)
Yes. The sheer fact that he's lethargic when being picked up and not even fighting is a sign he's unwell. I also think the abscess we're talking about is th elump/bump thing in the lower cheek region? Does his entire head look somewhat swollen to anyone else? the hop of franklin's head is more flat and right above each eye is a slightly raised bump (where the top if the eye socket is), this guys head seems really rounded to me... or is that a normal variation in torts?