Gareth Evans netted for the second game in succession to give them an interval lead at the Memorial Stadium.
But Chris Lines then levelled from the penalty spot and was sent off in the space of three second half minutes.
It looked like the spoils would be shared, only for Curtis to scramble a shot home late on and secure another three-point haul for the Blues.
Kenny Jackett unsurprisingly stuck with the same side that beat Oxford in such impressive fashion at the weekend.
There was a familiar face in the opposition line-up, with Kyle Bennett starting against his former club.
The game began in a cagey fashion, as both sides were limited to long-range efforts that failed to find the target.
Tom Naylor, Ben Thompson and Evans had their radars off for the visitors, while it was Ed Upson and Bennett for Rovers.
But Liam Sercombe really should have done better when Bennett’s cross found him unmarked in the box and he screwed over the bar.
Matt Clarke then did well to hook the ball to safety and prevent Stefan Payne from getting a shot away.
Those opportunities came during a spell of pressure for the hosts, but instead it was the Blues that broke the deadlock on 32 minutes.
A long ball up the pitch was flicked on first by Oli Hawkins and then Curtis, with EVANS collecting it and lashing home a rocket of a shot.
Rovers came close to levelling before the break, with MacGillivray getting down to his right to push clear a low Sercombe drive.
Anton Walkes was also called into action to block efforts from Ollie Clarke and Upson, ensuring Pompey went in ahead at the break.
Bristol Rovers 0
It was Pompey who threatened first following the interval, but when Jamal Lowe’s cross was headed clear, Naylor was unable to keep his shot down.
There was then a break in play while Clarke received treatment for a cut above his eye and the Blues defender returned to the pitch with a bandaged head.
Rovers then had a great chance to equalise when MacGillivray palmed clear a curler from Tom Nichols, but Sercombe could only send the loose ball into the side netting.
Rovers were on top and after having penalty appeals turned down when Sercombe’s shot hit Thompson, an effort from Nichols had to be blocked by Clarke.
Pompey saw that nervy period out, however, and began to look more of a threat at the other end of the pitch.
Evans came close to his second goal of the evening after a surging run across the edge of the box, but drilled narrowly wide with his left foot.
But then came a couple of moments of controversy, as Rovers sub Lines scored from the spot before being sent off.
It was his corner that referee Graham Salisbury decided Curtis had deliberately handled, with the midfielder just about squeezing his penalty past MacGillivray’s outstretched arm and into the bottom corner.
Brett Pitman was immediately introduced from the bench and was caught by the studs of Lines, whose reckless challenge earned him a red card.
It was a hectic ending, with goals looking likely to go in at either end of the pitch, perhaps thanks to some tiring limbs.
But it was the Blues celebrating on 87 minutes, when CURTIS capitalised on Pitman’s flick from a Thompson pass and just about managed to force a shot over the line.
The visitors were then indebted to MacGillivray, who made a stunning save to deny Bennett, while Pitman cleared off the line from the resulting corner.
Bristol Rovers (4-4-2): Smith; Leadbitter, Lockyer (c), Craig, J.Clarke; Bennett, Upson (Lines 69), O.Clarke, Sercombe; Payne (Jakubiak 69), Nichols (Rodman 79)
Goals: Lines 76 (pen)
Sent off: Lines
Subs not used: Andre Jr, Broadbent, Mensah Jr, Reilly
Pompey (4-3-3): MacGillivray; Walkes, Whatmough, Clarke, Brown; B.Thompson, Evans (c), Naylor; Lowe, Hawkins (Pitman 77), Curtis (Haunstrup 90)
Goals: Evans 32, Curtis 87
Subs not used: McGee, Burgess, Close, Wheeler, Chaplin
Referee: Graham Salisbury
Attendance: 9,073 (1,137 Pompey fans)