Jack Diamond slotted home in the final minute of stoppage-time to seal progress for the League Two side.
Luke Armstrong had earlier given them the lead just before the break, only for Ellis Harrison to immediately equalise.
Danny Cowley made four changes from the side that dramatically won at Gillingham in the league the previous week.
Michael Jacobs was rewarded for his late Priestfield winner with a start, while Harrison, Miguel Azeez and Sean Raggett also returned.
George Hirst and Joe Morrell both missed out with a non-Covid related illness, while there were four academy players among the nine substitutes.
There was a minute’s applause before kick-off in memory of promotion-winning manager Frank Burrows, who passed away recently at the age of 77.
An open first half saw both sides launch plenty of quick attacks, although there was not much to work either keeper.
Mark Oxley’s best moment between the Harrogate sticks came in the opening stages, as Marcus Harness forced a smart save after Jacobs’ run had been halted.
The forward had already nodded wide from a Mahlon Romeo cross, while a Shaun Williams corner just evaded the head of Raggett at the far post.
Pompey were passing the ball around nicely and one such passage of play ended with Romeo drilling the ball narrowly wide.
Harness also went close with an effort that was deflected behind, following some good work involving Harrison, Jacobs and Azeez.
And another precise corner from Williams then picked out Azeez, whose shot grazed the top of the net.
But Harrogate were causing problems of their own and it was not completely against the run of play when they took the lead just before the break.
A pass from Williams was intercepted and the ball worked to Armstrong, who neatly bent the ball around Gavin Bazunu’s grasp and into the net.
But the visitors’ lead lasted a matter of moment, as HARRISON was alert when a Jacobs run was halted and slotted into the net.
Harrogate Town 1
Raggett had to make a key block once the action restarted after Diamond beat Kieron Freeman in the centre of the pitch and charged towards goal.
There was a change of personnel and system just before the hour mark, with Freeman making way for Lee Brown and the Blues utilising a back four.
Harrison soon had a chance to score when he met a deep Williams free-kick, but could only guide his header into Oxley’s arms.
That was the striker’s final piece of action, though, as he was immediately replaced up front by Curtis.
Pompey were almost made to pay for some slack defending on 69 minutes, as Lloyd Kerry was allowed time to pick up a loose ball before seeing his shot deflected behind off Connor Ogilvie.
Cowley’s final throw of the dice was an attacking one, as Hackett was brought off for a fit-again John Marquis.
But it was Harrogate who almost scored seconds later, as Armstrong diverted a George Thomson effort inches past the post.
Referee Simon Mather was not endearing himself to the Fratton faithful with a couple of strange decisions, including showing a yellow card to Ogilvie for a clean tackle.
The Blues tried to apply plenty of pressure as the contest entered its closing stages and committed men forward.
But they were almost caught out on the break, with Armstrong’s effort blocked by Brown and Alex Pattison hitting the loose ball wastefully wide.
Raggett then got a foot to Diamond’s shot, affording Bazunu a simple save as Harrogate again countered quickly.
It looked as though the sides would be heading to Wetherby Road for a replay, only for Lewis Page to square the ball for a stretching Diamond to convert right at the death.
The result saw Pompey’s nine-match unbeaten run come to an end, while Harrogate can look forward to a spot in the third round for the first time.
Pompey (3-4-1-2): Bazunu; Freeman (Brown 57), Raggett (c), Ogilvie; Romeo, Azeez, Williams, Hackett (Marquis 75); Jacobs; Harness, Harrison (Curtis 60)
Goals: Harrison 45+1
Booked: Azeez, Williams, Ogilvie
Subs not used: Bass, Hughes, Jewitt-White, Payce, Gassan, Gifford
Harrogate Town (4-2-3-1): Oxley; Sheron, Burrell, Hall, Page; Falkingham (c), Kerry; Thomson, Pattison, Diamond; Armstrong
Goals: Armstrong 44
Booked: Kerry, Burrell, Pattison, Hall
Subs not used: Cracknell, Power, Orsi, Muldoon
Referee: Simon Mather
Attendance: 7,857 (121 away fans)