The problem is that countless organizations planned around an afternoon football game, with group dinners in the evening. For many fraternities and sororities, I imagine it was their biggest alumni event of the year. I know it always was for K∆P.
There's just no other time of the weekend a group could reliably have an event like that. Friday dinner? Too many people won't be in town yet. Sunday brunch? Too many people will be hitting the road early. (Or will be hung over.) Brunch is too informal, anyway, and people tend to swing through and grab a bagel rather than relax for a couple of hours, meet people, and so forth. Mid-afternoon Saturday? Well, all the tailgates start at 3 -- and, for some reason, the "Greek" tailgates seem to be off in their own parking lot, not next to the stadium where everyone expects to find tailgating.
Of course, there are loads of people who have no interest in the football game, but since it's the cornerstone of any Homecoming weekend, making people choose between the game or your satellite event either a) wouldn't work or b) would spark resentment.
And Alumni Affairs is freaking out that one of their early-afternoon events, a Student-Alumni Networking gathering, has loads of students signed up but hardly any alums. Why am I not surprised? It's the sort of event I might be persuaded to come to in a vacuum, but they've so heavily scheduled the weekend that I'm keeping myself free so I can check stuff out.