Zverev, 21, who is seen as the main threat to Rafael Nadal's expected coronation as champion for an 11th time, endured a nightmare outing on Court Philippe Chatrier before claiming victory after nearly four hours of play.
In the fourth set, his opponent served for the victory.
The match will resume Monday. Both times, he proved steadier and sturdier than Damir Dzumhur, a Bosnian ranked 29th and seeded 26th, and Zverev eventually prevailed 6-2, 3-6, 4-6, 7-6 (3), 7-5 in 3 hours, 54 minutes.
The Williams-Williams duo last competed at a major tournament together at Wimbledon in 2016. "I don't know what could I say after a match like this and a defeat like this".
2016 men's champion Novak Djokovic and women's Australian Open victor Caroline Wozniacki have both advanced, while men's fourth seed Grigor Dimitrov and women's fourth seed Elina Svitolina have been bundled out. But Grand Slam success has been elusive.
More likely, he won't fuss too much over the details, including that Dzumhur entered the French Open with a losing record this year.
The win notches up the 33rd tour-level victory for Zverev this year, more than any other player.More news: NBC News’ ‘Today’ Show Grills Bill Clinton About Monica Lewinsky Affair
4th seeded Ukrainian Elina Svitolina left the tournament on the third round.
Watch highlights of Maria Sharapova's win over Karolina Pliskova.
The victory was his first on Court Philippe Chatrier.
The 25th-seeded Kontaveit was unable to exert any pressure on the 10th-seeded Stephens with her error-strewn game.
Kontaveit made 21 unforced errors and hit just six clean winners.
Bosnian Damir Dzumhur felt sad after a tight French Open third round loss.
But having won two of the first three sets, Dzumhur was caught up in the moment in the fourth set when he charged to collect a ball dropping from the sky. "I was feeling fine physically, so for me that gives me a lot of confidence going deep into the fifth set, and long matches on this kind of surface". But like Zverev, things became rather tougher for him in the second round.More news: Memorial Tournament 2018: Final results, leaderboard (Bryson DeChambeau winner)
Thiem made the semifinals past year and in 2016. He was playing with Alexander Zverev at Roland Garros.
Paris: Alexander Zverev's appetite for five-set marathons showed no signs of shrinking and even blisters on his toes could not stop the German from storming into his first major quarterfinal with a 4-6, 7-6 (4), 2-6, 6-3, 6-3 French Open win over Karen Khachanov on Sunday.
Zverev has climbed to number three in the world rankings despite never previously having gone beyond the fourth round of a Grand Slam, achieved at Wimbledon a year ago.
At 21, Zverev is the youngest quarterfinalist at Roland Garros since Juan Martin del Potro in 2009.
Her feat means the 13th-seeded American has now reached the last eight at every major.
She has not dropped a set yet and will face U.S. Open runner-up Madison Keys in the fourth round. Serving for the match, Zverez was up 30-0 but dropped the next two points for 30-all.
Dzumhur was up against it when Zverev secured an early break in the second, but the 27-year-old warmed to the task under grey skies and deservedly got back on serve at 3-3.More news: Neymar scores in return; Brazil tops Croatia