Brett Pitman came back to haunt his former club with a goal in each half, the second coming from the penalty spot.
Ronan Curtis reduced the deficit and the Blues pressed for a leveller, only for Tyer Smith to wrap things up for the struggling Robins at the death.
Danny Cowley made two changes from the side that were narrowly beaten at MK Dons at the weekend.
Ben Close recovered from a groin problem to replace Harvey White in the centre of midfield, while Rasmus Nicolaisen came in for Jack Whatmough – who begins a four-match ban – at the back.
It was a nervy start from the visitors, who looked nervy at the back and sloppy in possession during the opening stages.
But they had the first decent chance of the contest on 13 minutes, as Marcus Harness met Callum Johnson’s corner with a fine header that was turned over by Lee Camp.
Craig MacGillivray was soon called into action at the other end, getting down to his right to push clear a Scott Twine free-kick.
He could do nothing to prevent Swindon from breaking the deadlock moments later, though, after the Blues had lost possession.
Their opponents took advantage in clinical fashion, as Jack Payne laid the ball off for Pitman to lash into the net.
Pompey began to enjoy more possession as the half progressed, without really threatening to equalise.
One fine Johnson pass over the top played Marquis through and although the striker made a nice first touch, his second saw the ball run wastefully out of play.
A low Harness ball towards the striker at the back post was cut out by Anthony Grant, as the visitors finished the half on top.
But the Robins dealt with a couple of Charlie Daniels corners to ensure they would go in ahead at the interval.
Swindon Town 1
Pompey showed plenty of attacking intent once the action restarted, but were continuing to struggle in the final third.
Curtis – who was the most dangerous player going forward – flashed a shot wide, while another Daniels corner came to nothing.
They almost levelled on 55 minutes after Tom Naylor sent a perfectly timed lofted pass over the Robins back-line.
Marquis collected it well, but his shot was blocked by Dominic Thompson and Camp saved the follow-up from Harness.
Swindon went straight up the other end and came close to doubling their advantage, with MacGillivray coming to the rescue after Smith had squared the ball to Pitman.
And the home side did go further ahead just past the hour mark, with Daniels conceding a penalty for the second time in as many games.
He was booked for bundling over Smith, who had taken a heavy touch following fine work from Pitman.
Referee Ben Toner pointed to the spot and Pitman made no mistake, slamming the ball right into the bottom corner.
Pompey were trying to fight back and a fizzing strike from Curtis had to be palmed behind by Camp after Ben Close had laid the ball into his path.
George Byers then came on for Daniels, with skipper Naylor dropping into a three-man defensive line, while Sean Raggett soon made way for James Bolton.
Andy Cannon also replaced Ben Close for the closing stages, as the visitors continued to control possession.
And they had a lifeline on 82 minutes, as CURTIS controlled a threaded Marquis pass and bundled the ball home.
They came close to a leveller seconds later, with Byers playing a neat one-two with Curtis and drawing a smart stop from Camp.
Cannon was next to go close, with a low drive that crept inches past the post, as the Blues increased the pressure.
But Swindon wrapped things up in stoppage-time, as Smith stole possession, advanced forwards and neatly found the bottom corner.
Swindon (4-4-2): Camp; Garrick, A.Grant, Grounds, Thompson; Payne, Palmer, Missilou (Hope 79), Twine; Pitman (c) (Broadbent 90), Smith
Goals: Pitman 17, 61 (pen), Smith 90+1
Subs not used: Matthews, Baudry, Curran, Omotoye, J.Grant
Pompey (4-2-3-1): MacGillivray; Johnson, Raggett (Bolton 74), Nicolaisen, Daniels (Byers 70); Naylor (c), Close (Cannon 81); Harness, Williams, Curtis; Marquis
Goals: Curtis 82
Booked: Daniels, Nicolaisen, Cannon
Subs not used: Ward, Downing, White, Stanley
Referee: Ben Toner