It seemed like the sides would be heading to Moss Lane for a replay, as the National League North outfit battled valiantly from behind.
Ben Close’s superb volley had opened the scoring, only for Josh Hancock to grab a late leveller from the spot.
The Blues looked to have been punished for not seeing the game off, but they were rescued by substitute Pitman at the death.
Kenny Jackett made one change from the side that beat Rotherham in League One earlier in the week.
It came in the attacking wide positions, with Ryan Williams dropping to the bench and Marcus Harness taking his spot on the right.
Pompey started brightly and when Close dug out a fine cross towards the far post, Ronan Curtis was waiting to nod narrowly past the post.
John Marquis then met a Harness delivery and his goal-bound header struck James Jones and was diverted wide, although the officials bizarrely failed to give a corner.
The hosts produced plenty of sloppy moments during the first half, with Altrincham looking dangerous going forward.
They thought a penalty should have been awarded when Jordan Hulme went down under Oli Hawkins’ challenge on 11 minutes.
Referee Tom Nield waved their appeals away and Hawkins felt the striker had gone down too easily, while their opponents were furious at the decision.
Marquis was struggling to get into the game, but sent one great delivery across the face of goal that agonisingly evaded everyone.
Anton Walkes then stepped up to steal possession and find Ellis Harrison, who burst forward and fired against the base of the post.
Curtis was next to try his luck, turning on the edge of the area and dragging a shot narrowly wide.
Some patient passing play then ended with a fierce volley from Brandon Haunstrup that rose over the crossbar.
And when Harrison tried to send Harness clear through on goal, it took a timely challenge from Andy White to deny him.
Both sides then had excellent opportunities to break the deadlock in the closing stages of the first half.
Curtis did brilliantly to win possession after losing the ball, but could not bring it under control and Hulme drilled a shot against the post.
Harrison then met Lee Brown’s delivery at the other end of the pitch, drawing an excellent save from Tony Thompson.
Jackett made a double change at the start of the second half, with Curtis and Marquis withdrawn for Williams and Pitman.
But the Blues were still looking well below their best and were relieved to see an unmarked Jake Moult fire wastefully wide.
Craig MacGillivray was then called into action to beat away an angled effort from White, as the Robins looked to punish their higher-ranked hosts.
It was Pompey who finally broke the deadlock just before the hour mark, however, courtesy of another fine strike from CLOSE.
The ball sat up nicely for the midfielder after Harness’ cross had not been properly cleared and he was left alone to hit a crisp volley into the net.
He might had had another moments later, but Thompson kept out his angled drive, only to then collect a back-pass.
That meant the rare sight of a free-kick inside the box, with the wall doing its job to block Pitman’s attempt from Brown’s lay-off.
It was Pompey’s turn to appeal for a penalty when Williams went down under Shaun Densmore’s challenge on 67 minutes, although again it was waved away by Nield.
Williams then launched a quick break up the pitch and slipped a fine pass across for Harness, but Thompson again came to the Robins’ rescue with a fine save from point-blank range.
And when Haunstrup tried to pass the ball into the bottom corner, it was deflected just past the post.
But Altrincham were always in the contest while they only trailed by one and a delivery from the left was deflected onto the roof of the net.
The scores were level on 83 minutes, with Christian Burgess penalised for a clumsy challenge on Hulme inside the box.
This time Nield did point to the spot and although MacGillivray went the right way, Hancock’s penalty was too powerful.
The Blues tried to hit back and Pitman did well to get an acrobatic effort on target, with the ball flying straight at Thompson.
But there was a sense of joy and relief in the final minute of stoppage-time, as Harness’ corner was flicked on by Harrison and PITMAN stooped to head home at the far post.
Pompey (4-2-3-1): MacGillivray; Haunstrup, Hawkins, Burgess, Brown (c); Walkes, Close; Harness, Marquis (Pitman 46), Curtis (Williams 46); Harrison
Goals: Close 56, Pitman 90+4
Booked: Harness, Williams
Subs not used: Bass, Bolton, Raggett, Cannon, Evans
Altrincham (4-2-3-1): Thompson; Densmore, Jones, Hannigan, White; Williams (Mullarkey 84), Moult (c); Peers (Harrop 80), Hancock, Johnston (Hampson 88); Hulme
Goals: Hancock 83 (pen)
Subs not used: Torrance, Richman
Referee: Tom Nield
Attendance: 8,539 (622 away fans)