With just a few days remaining before David Letterman airs his final Late Show broadcast, longtime bandleader Paul Shaffer sat down with Billboard to reminisce about spending more than three decades as half of a legendary late-night duo.

As he quickly points out in the interview, the earliest broadcasts of their first show together, NBC's Late Night With David Letterman, were a low-budget, on-the-fly affair — and although budgets and production techniques eventually expanded to add some extra polish to the viewing experience, he still recalls those shoestring performances fondly.

"Paul Simon was singing a beautiful song of his called 'The Late Great Johnny Ace,'" recalled Shaffer. "He was singing it with one guitar, and a string broke or something and he just came to a stop. In my ear, I hear the assistant director pushing the applause button. He got the audience to applaud in the middle of the song because the music came to a stop. They went to commercial and when we came back, Simon finished the song. It was absolutely honest and natural. Nowadays they would stop the tape and start again."

Asked to present a list of "Hot 5" performers from Letterman shows over the years, Shaffer came up with an eclectic bunch that includes James Brown ("I've never experienced anything like that before or since"), Supertramp saxophone player John Helliwell ("He was the first guy to sit in with us, and it was his idea. We were tickled"), singer-songwriter Carole King ("Her first appearance was huge for me since I'm such a fan. She's such a nice lady"), Sly Stone ("We rented him a clavinet. It didn't work. He hooked up a wah-wah pedal and played it anyway") and Eric Clapton ("We met him backstage at his show and he asked if he could sit in. That was monumental").

And while we already know who Letterman's final musical guest will be, Shaffer admits he and the rest of the staff are avoiding focusing on the last episode. Having played alongside just about everyone in the music business, he puts his years of late night service at the top of a distinguished career. Calling it "the most idyllic work situation I’ll ever be in," he says, "I hope to do a bit of everything. But I know I won't be able to do it all at once, the way I was able to on this show."

See 2015’s Biggest Rock News Stories

25 Interesting Rock Movie Facts