A Gareth Evans penalty set them on the way in the early stages and there were plenty of chances to add to the lead.
Ronan Curtis eventually headed home to calm some nerves, only for Alex Rodman’s strike to ensure a nervy finale.
And, with just seconds remaining, a corner struck Ross McCrorie, the bar and Craig MacGillivray before crossing the line to leave the Blues heartbroken.
Kenny Jackett made one change from the side that had beaten Lincoln on home soil earlier in the week.
It came on the left of the forward line, where Marcus Harness dropped to the bench to make way for Curtis.
Former Rovers hero Lee Brown kept his spot in defence, but Ellis Harrison had to be content with a place among the substitutes.
There were only seconds on the clock when Sean Raggett received a booking for a late challenge on Victor Adeboyejo.
But it was otherwise an encouraging opening for the visitors, who looked a constant threat on the break.
Ryan Williams teed-up Ben Close for an early effort that was blocked by Ed Upson and the deadlock was broken soon after.
Evans’ cross was met by the head of Curtis and the ball flicked off Raggett’s shoulder and came back off the post.
But John Marquis had received a shove from Luke Leahy that was spotted by referee Darren Handley.
He pointed straight to the spot and EVANS made no mistake, slamming home a penalty that left Anssi Jaakkola with no chance.
The keeper managed to prevent Williams from doubling Pompey’s lead soon after, while a teasing delivery from McCrorie was agonisingly over the heads of both Marquis and Curtis.
MacGillivray had to palm away a corner at the other end, but the counter-attack that followed led to three great chances in quick succession.
Curtis sent Marquis racing clear, only for Abu Ogogo to fling himself bravely into the path of the striker’s shot.
But the hosts could not get rid of the ball and crosses from Evans and Brown were met by Burgess and Raggett respectively, with Jaakkola making decent saves on each occasion.
Rovers had seen more of the ball, but not done much with it and their first decent chance arrived on 26 minutes.
Upson did well to keep possession in the middle of the park and drive forward, before drilling a low effort inches the past.
Pompey were still looking the more threatening side and when a Curtis cross missed Marquis, the ball appeared to be dipping under the bar, forcing Jaakkola into action.
They received a second booking before the break, although McCrorie was furious to be penalised for simulation after appearing to be tripped by Alfie Kilgour.
There was a corner for the visitors to deal with in stoppage-time and Marquis was back to block Ollie Clarke’s attempt.
Bristol Rovers 0
The home side – and their supporters – cried for a penalty straight after the restart when Leahy’s shot hit McCrorie in the box.
But the Blues loanee was far too close to get out the way and replays showed that it hit him on the head.
There was then a nervy moment when MacGillivray could not keep hold of Adeboyejo’s cross, although the keeper immediately leapt on the loose ball.
Pompey were still trying to do damage on the break, with one promising move breaking down when Marquis strayed offside from Curtis’ pass.
The hosts were certainly providing more of a threat since the break, though, and a diving Ogogo could not quite get on the end of Rodman’s centre.
An Evans free-kick caused problems at the opposite end of the pitch, although Rovers were eventually able to scramble the ball behind.
The visitors calmed things down with a spell of possession and might have gone further ahead, only for skipper Tony Craig to bravely get in the way of Close’s fierce volley.
And they did create some breathing space on 70 minutes, when a Marquis delivery was firmly headed home by CURTIS from close range.
Pompey’s counter-attacking credentials were still impressive and a low Williams cross had to be cut out to deny Marquis a simple tap-in.
But they were dealt a blow soon after, with Rodman reducing the hosts’ deficit after a free-kick had not been properly dealt with.
That meant there were plenty of nerves among the large collection of away fans for the closing stages of the contest.
Rovers brought former Fratton winger Kyle Bennett on to try add more potency to their attack, but it looked like Pompey would comfortably see the game out.
Right at the death, however, Liam Sercombe whipped in a corner and the ball struck McCrorie, ricocheting off the bar and the back of MacGillivray’s head before nestling in the net.
Bristol Rovers (3-5-2): Jaakkola; Davies, Craig (c) (Little 80), Kilgour; Rodman, Clarke (Sercombe 71), Upson (Bennett 89), Ogogo, Leahy; Adeboyejo, Smith
Goals: Rodman 78, MacGillivray 90+3 (og)
Booked: Upson, Ogogo
Subs not used: van Stappershoef, Kelly, Menayese, Nichols
Pompey (4-2-3-1): MacGillivray; McCrorie, Burgess, Raggett, Brown; Naylor (c), Close; Williams, Evans, Curtis; Marquis (Harrison 90+1)
Goals: Evans 9 (pen), Curtis 70
Booked: Raggett, McCrorie, Curtis, Burgess
Subs not used: Bass, Bolton, Downing, Haunstrup, Cannon, Harness
Referee: Darren Handley
Attendance: 8,648 (1,404 Pompey fans)