They were level at the break after Callum Johnson’s first goal for the club cancelled out Famara Diedhiou’s opener.
And the Blues looked the more likely side to score a winner as the seconds ticked down, only for a fine Chris Martin strike to settle the tie in favour of the hosts.
Kenny Jackett made two changes from the side that won at Hull to reach the League One summit before Christmas.
Michael Jacobs made his first start for more than two months, with Ronan Curtis left out as a precaution after feeling unwell.
And with Cam Pring being recalled from his Robins loan spell, the fit-again Lee Brown returned at left-back.
The visitors were first to threaten, but Johnson’s cross was just too high for Ryan Williams, allowing Adrian Mariappa to head the ball safely into the arms of Dan Bentley.
Their opponents went closer at the other end, with Craig MacGillivray preventing Martin from finding the net from an acute angle.
And there were half-chances at either end, as both sides looked confident, without managing to find a clinical touch.
That changed on 19 minutes, however, as Diedhiou played a neat one-two with Hakeeb Adelakun before bending home a neat left-footed finish.
The Blues tried to hit back and carved out an opening following some good combination work down the right by Marcus Harness and Johnson.
A cross from the latter was nodded down by Jacobs for Marquis, who spun neatly and saw his shot blocked.
It appeared as though Bristol City would be taking their lead into the interval, only for a leveller to arrive in stoppage-time.
Cannon received a return pass from Harness and surged into the box, cutting the ball back for JOHNSON to arrow home.
Bristol City 1
Jackett made a change at the break, with Rasmus Nicolaisen – who was on a yellow card – being replaced by Jack Whatmough.
The Robins were quicker out of the traps, though, and Tommy Rowe fired a warning shot with an effort from the edge of the box that cleared the crossbar.
But Pompey soon threatened themselves and after a fierce Harness strike was blocked by Tomas Kalas, Tom Naylor’s placed follow-up was saved by Bentley.
And a red-shirted defender then got in the way of another attempt from Harness, who had been teed-up by Williams.
It was then the turn of Bristol City to look dangerous, with Antoine Semenyo spinning sharply in the box to meet Rowe’s delivery and hit a shot that was smothered by MacGillivray.
Pompey’s second substitution followed on 65 minutes, as Ellis Harrison came on to lead the line in place of Marquis.
And the newcomer immediately looked lively, scuffing an attempt wide and then playing a part in a neat move that ended with Harness unable to dig out a shot inside the area.
The visitors were starting to cause a lot of problems and a jinking run by Naylor ended with the skipper forcing a save from Bentley.
And the keeper was soon called into action again to deny both Cannon and Jacobs in quick succession.
He was certainly the much busier of the two stoppers, although a Harrison header lacked enough power to cause too much trouble.
But then, out of nothing, Bristol City restored their lead – and there was little MacGillivray could do to prevent a Martin rocket finding the top corner.
Jordy Hiwula was thrown on for the closing stages, as the Blues fought to take the tie into extra-time.
And he had a chance to equalise right at the death, only for that man Bentley to come to the Robins’ rescue once more.
Bristol City (4-4-2): Bentley; Mariappa, Kalas (c), Mawson (Moore 66), Rowe; Semenyo, Nagy (Massengo 77), Bakinson (Vyner 81), Adelakun (Wells 66); Diedhiou, Martin
Goals: Diedhiou 19, Martin 83
Subs not used: O’Leary, Towler, Edwards, Pearson, Bell
Pompey (4-2-3-1): MacGillivray; Johnson, Raggett, Nicolaisen (Whatmough 46), Brown; Naylor (c), Cannon; Harness, Williams, Jacobs (Hiwula 87); Marquis (Harrison 65)
Goals: Johnson 45+1
Booked: Nicolaisen, Brown, Naylor
Subs not used: Turnbull, Mnoga, Bolton, Downing, Morris, Close
Referee: Leigh Doughty