The spot gold price was quoted at $1,295.00-1,295.30 per oz as of 04:33 BST, up just $0.95 on the previous session's close.
Both New York Federal Reserve Bank President William Dudley and Chicago Fed President Charles Evans said they believe the US economy's fundamentals are sound, and both said they support gradual rate hikes ahead.
The US dollar got a solid lift after Ms Yellen said "persistently easy monetary policy" could lead to developments that could have "adverse implications for financial stability", in a speech. This is while the Pound-to-Dollar rate was seen 0.29% lower at 1.3438.
The S&P 500 index closed up 0.01% Tuesday after briefly climbing above the 2,500 mark, earlier during the United States trading day.
The biggest focus for the market for Wednesday is the announcement of a tax plan by the U.S. administration and Republicans in Congress.More news: Bayern winger warns Liverpool target against leaving PSG
The rise in US yields was stoked further by remarks from US President Donald Trump who said and lawmakers were working to pass a big tax cut for the middle class. Traders had no details on the plan, but speculated it might raise the federal deficit and increase government borrowing.
Each one presented valid arguments for their cases but Janet Yellen's speech before the 59th Annual Meeting of the National Association for Business Economics in Cleveland, Ohio proved why she's chair of the US Federal Reserve.
It was also interesting to note that Fed officials worry that keeping rates lower allows little room for stimulus when another economic slowdown hits.
Cape Town - The rand lost more than 1.5% against the U.S. dollar on Wednesday, breaking through the level of R13.50/$ for the first time in nearly three months as the greenback rallied on rate hike hopes. But even with uncertainty about the possible reasons for the low rate of inflation - from misjudging the strength of the labor market to the impact of foreign competition on the global supply chain - Yellen said the Fed "should be wary of moving too gradually".
Also scheduled for release at 10:00 a.m. ET is a report on US consumer confidence, which is expected to show it fell to 120 in September from 122.9 in August.More news: Arsenal 2 West Brom 0: Tony Pulis accused Alexis Sanchez of 'cheating'
"The divergence between USA and euro zone data surprises is closing rapidly and while the once popular dollar parity forecasts still look ridiculous, the euro looks past its peak", said Sean Maher, an independent macro-strategist based in London.
Dow e-minis were down 10 points, or 0.04 percent, with 22,892 contracts changing hands. German 10-year benchmark yields rose 6 basis points to 0.47 percent.
CURRENCIES: The dollar rose to 112.41 yen from 112.23 yen.
-The euro sank 0.5 percent to $1.1792, the weakest in five weeks.
-West Texas Intermediate crude fell 0.5 percent to $51.94 a barrel.More news: Backed Militia Says It Has Recaptured Syria's Biggest Gas Field from ISIS