Letterman has 2-3 years left?

It was just an off handed comment on Lettermen when Albert Brooks mentioned that this was Dave's last week. An inaudible gasp seemed to come over the audience, but it was just a joke. However, later on in the interview Letterman said he had "2 to 3 years left"... and then repeated this sentence a few minutes later. This could just be an off hand remark made in conversation with the joking Brooks, but perhaps not. It's been more than 15 years since Letterman took the helm at CBS's Late Show. This was after he left Late Night at NBC when he was passed up for the Tonight Show by NBC executives who favoured, of course, Jay Leno.