The visitors were heading out of the competition before the Republic of Ireland international’s late heroics.
George Hirst had nodded them into an early lead in Devon, only for Matt Jay to hit an immediate leveller.
The Blues then failed to make the most of their first half dominance and were made to pay when Archie Collins struck for the Grecians after the break.
But Curtis headed home with time running out and appeared to have secured a penalty shoot-out, only to then find the net again in stoppage-time.
Pompey will now look to make it through to the quarter-finals when they head to Cambridge on Tuesday night.
Danny Cowley made four changes from the side that drew at the Abbey Stadium in the league at the start of the week.
John Marquis was handed a starting berth up front, while Lee Brown – who wore the captain’s armband – Alex Bass and Miguel Azeez all returned.
Curtis and Reeco Hackett both dropped to the bench, while Gavin Bazunu and Shaun Williams were not included in the squad.
Also named among the substitutes were teenage keeper Toby Steward and his academy team-mate Harry Jewitt-White.
The contest certainly got off to a lively start and the visitors took an early lead when Marcus Harness lifted the ball towards the back post and HIRST rose highest to head into the bottom corner.
But their advantage lasted just 68 seconds, as Exeter went straight up the pitch to level from kick-off.
Jay picked up the ball and found it all too easy to shake off a couple of challenges before expertly guiding a shot past Bass.
From that point on, though, it was pretty much one-way traffic, as the Blues were totally dominant in possession.
But the struggle was to make the most of it and a number of key openings were wasted in the search for a second goal.
Pompey’s high-pressing game was causing plenty of problems and there were a few occasions when possession was stolen and the situation not capitalised on.
The lead should have been retaken when Harness released Marquis down the left and the striker saw his shot blocked.
Hirst tried to convert the loose ball, but was left swiping at thin air, as the Grecians breathed a sigh of relief.
The striker soon had another opportunity after being fed by Kieron Freeman following a patient spell of passing, spinning neatly and seeing his attempt deflected just past the post.
And he twice tried to get on the end of Brown deliveries in quick succession, but could not make a firm contact on each occasion.
Pompey maintained a constant vigil in their hosts’ half as the seconds ticked down, with Exeter defending resolutely to keep them at bay.
Exeter City 1
It was the home side who were first to threaten when the action restarted, but Bass was equal to Jay’s shot after Owura Edwards had teed the ball up perfectly for his team-mate.
And they had another decent chance moments later, with Josh Key’s teasing cross somehow evading everyone as it made its way across the Blues box.
Pompey started to settle again and looked to impose themselves, although Azeez’s pass for Marquis in the area ricocheted off the striker and landed clear.
Cowley then introduced a raft of fresh legs by making a quartet of substitutions just past the hour mark.
Curtis, Hackett, Michael Jacobs and the fit-again Louis Thompson all came on, as Marquis, Hirst, Harness and Azeez made way.
Exeter were still looking more threatening than before the break, though, and Connor Ogilvie got in the way of Key’s effort before Bass beat Jay to Jake Caprice’s cross.
And they did take the lead on 76 minutes, as Key got forward and laid the ball off for Collins, whose shot deflected off Freeman to sneak into the net.
The home side tried to see out the closing stages and were doing a good job of protecting their slender lead.
There was a late opportunity for Pompey, but Cameron Dawson dealt with Hackett’s header after Joe Morrell had lifted the ball into the mixer.
Jewitt-White injected some youthful vigour into the Blues for the final few minutes, but it was a more experienced player who drew them level.
Jacobs swung in a free-kick from the right and CURTIS timed his leap perfectly to head the ball past Dawson.
He might have had another just a few moments later, but Hackett’s cross was just too high for him to head on target.
CURTIS did grab the winner deep into stoppage-time, though, firing into the net after Hackett had raced across goal and found the forward.
Exeter (3-4-3): Dawson; Sweeney (Kite 46), Stubbs, Diabate; Key, Collins, Taylor (Dieng 55), Caprice; Seymour (Brown 55), Jay (c) (Coley 79), Edwards (Amond 71)
Goals: Jay 6, Collins 76
Subs not used: Brown, Ray
Pompey (3-4-1-2): Bass; Freeman, Raggett, Ogilvie; Romeo, Morrell (Jewitt-White 88), Azeez (Thompson 63), Brown (c); Harness (Jacobs 63); Marquis (Curtis 63), Hirst (Hackett 63)
Goals: Hirst 5, Curtis 89, 90+3
Booked: Morrell, Raggett, Curtis
Subs not used: Steward, Downing
Referee: Paul Howard
Attendance: 2,146 (507 Pompey fans)