In a true playoff system you would be right. But that’s not what we have. In a system predicated on “picking” the objectively assumed 4 best teams at the time, yes it would.
The problem is this isn’t a real playoff. It’s a year end ranking with a 4 team tournament tacked on. A true playoff would include, at the minimum, all FBS conference champions.