My unofficial rationale may be more philosophically appealing. Reverse mathematics shows us that you can do an awful lot of math using only the natural numbers and sets thereof. It *also* shows us that you can do a lot of math using weaker inductive assumptions; it highlights five interesting "levels" of induction (the celebrated RCA, WKL, ACA, ATR, and Pi11CA theories) and Simpson's book hints that a few weaker systems like EFA might turn out to be of similar interest. So I personally see induction over the ordinals in ZFC as one of many points on a spectrum of induction that's bounded below by RCA (or maybe EFA) and unbounded above by Godel's incompleteness theorem.
