She also said she would stand down as chancellor once the current legislative term ends in 2021.
In the turbulent times of Trump, Brexit and multiple global crises, Merkel came to be hailed overseas as the new "leader of the free world", a label she was quick to reject as "absurd".
She now governs Germany in a "grand coalition" of what traditionally have been the country's biggest parties - the CDU, Bavaria's conservative Christian Social Union, and the center-left Social Democrats. The CSU also suffered a setback in Sunday's Hesse elections. "Angela Merkel knows best what to do, and now she has decided".
She said she would not seek reelection as chancellor.
And in both Bavaria and Hesse, the anti-immigrant Alternative For Germany (AfD) party vote surged, benefiting from voters unimpressed by Merkel's 2015 decision to allow more than one million asylum seekers to settle in Germany.More news: Obama urges Wisconsin Democrats not to be ‘bamboozled’
The other risk for Merkel is that her SPD coalition partners come third in Hesse, which is home to financial hub Frankfurt. She had won four consecutive terms.
Earlier this year, she appeared to have given her blessing to Kramp-Karrenbauer, 56, the onetime leader of the west German state of Saarland and now the CDU's general secretary.
Ms Merkel, 64, has been the CDU's chairwoman since 2000 and chancellor since 2005. In projections throughout the evening, the outgoing CDU-Green coalition was forecast to win 61 seats out of 121, securing a razor-thin majority.
The dismal performance poured further scorn over Merkel's authority as a leader of the CDU.
The instability has further eroded the credibility of the conservatives and the SPD in the eyes of German voters, who are increasingly turning to smaller parties on the right and on the left.
Two weeks ago, two of the federal governing parties - the Christian Social Union, the Bavaria-only sister to Merkel's CDU, and the Social Democrats - were battered in a state election in neighboring Bavaria.More news: Why was Khashoggi killed? The Saudi and Turkish explanations
"Angela Merkel knows quite clearly that she is living through the last months or through the previous year of her chancellorship", Werner Patzelt, a politics professor at the Technical University in Dresden, told NBC News.
The Frankfurter Allgemeine Zeitung said on Sunday it would be "a mistake" for Merkel to cling to power.
While the CDU remained the largest party in the election, which was held in the central state of Hesse, results were down 10% from the previous election.
Increasing numbers of SPD members have been calling for the party to quit government and lick its wounds in opposition, as it is presently polling below AfD nationwide, at 15 percent to the far-right's 16 percent.
Merkel's first step towards the exit door is likely to send ripples across the European Union, where she has served a beacon of stability as bloc grapples with multiple global crises, Brexit and an unpredictable ally in the White House.More news: 14 killed as Jordan flash flood sweeps away students on trip