With the scores level at 1-1 and the clock winding down, the ball came to Phil Foden in the United box and although the English midfielder had no control over it, Rodri stepped in to power a half-volley into the top corner.
Champions City continued their perfect start as they moved on to nine points, two ahead of West Ham United, Tottenham Hotspur and Arsenal.
City striker Erling Haaland missed a first-half penalty given for handball as his spot-kick hit the post.
But the Norwegian made amends just after the hour-mark when he opened the scoring with a close-range header from a cross by Jack Grealish.
Sheffield United equalised in the 85th minute when the hosts capitalised on a defensive error from Kyle Walker’s botched backheel clearance and Jayden Bogle fired an angled shot through the crowd and beat City keeper Ederson.
Just as United looked to have gained their first point on their return to the top flight, Paul Heckingbottom’s side were left heartbroken when Rodri fired home the winner.
City manager Pep Guardiola was not on the touchline at Bramall Lane having undergone back surgery but they got the job done despite seven minutes being added on as the home side searched for an equaliser.
United are 17th in the standings without a point from three games.
(Reporting by Rohith Nair in Bengaluru, editing by Ed Osmond)