Conor Chaplin came off the bench to fire the hosts in front and seemingly back to the top of the League Two table.
But, just seconds after leaving the bench, Williams levelled to burst the Blues bubble and send everyone home deflated.
Paul Cook made one change from the side that won at Newport on Saturday.
Rommy Boco came in for his first Pompey start wide on the right, with Kal Naismith dropping to the bench.
The visitors started brightly, with Aaron McCarey and Christian Burgess forced to deal with some dangerous crosses.
But the Blues had chances of their own in a very open first half and Jesse Joronen got down quickly to keep out Gary Roberts’ drive following a quick counter-attack.
The keeper was called into action again moments later and only just got his hand to a long-range Boco effort that was headed for the bottom corner.
It was Stevenage who were next to threaten on 17 minutes, as Tom Pett flashed a dangerous ball across the face of goal before Arnand Gnanduillet headed wide at the far post.
Pompey began to dominate as the half progressed, however, with a series of pin-point crosses from Ben Davies causing Boro plenty of problems.
One of those was met by the livewire Boco, whose stooping header lacked enough power to really trouble Joronen.
The right-back also had a penalty appeal turned down when, after shaping to pass, he went down under Jerome Okimo’s challenge.
It was Roberts who should have broken the deadlock on 32 minutes, firing into the side netting after Matt Tubbs had taken advantage of Mark Hughes’ slip.
And the midfielder then whipped in a cross that was headed back into the danger zone by Boco, with Hughes only just getting there before Tubbs could pounce.
Boro came flying out of the traps at the start of the second half and they should have forged ahead on 48 minutes.
A fine passing move cut open the Blues back-line, but with just McCarey to beat, Dipo Akinyemi fired wide.
And it was Pompey wondering why they had not scored moments later, following a fine surging run from Enda Stevens.
The left-back picked out Roberts, whose low pass across the face of goal should have been turned home by Tubbs at the far post.
The hosts were, by now, looking the much more dangerous side and they created another decent opening just before the hour mark.
And the chance owed a lot to the perseverance of Davies, who managed to lay the ball off for Michael Doyle, with the skipper’s pile-driver well saved by Joronen.
Cook then made his first substitution, with Caolan Lavery coming on for his home debut in place of Tubbs.
The loan signing did brilliantly to control McCarey’s long clearance on 66 minutes and feed Bennett, whose fierce drive was saved by Joronen.
He then had another attempt after bursting down the centre, but slipped as he shot and the ball rolled tamely into the keeper’s hands.
A second change followed on 71 minutes, with the tiring Boco making way for Chaplin’s fresh legs.
And it was the teenager who finally found a route through to goal on 77 minutes following a spell of intense Pompey pressure.
Adam Webster was the creator, bursting forward and sending a pass down the channel which Jamie McCombe missed, allowing CHAPLIN to beat Joronen at his near post.
Stevenage sent men forward in search of an equaliser and Tom Pett fired narrowly over the crossbar.
But it was the Blues who looked likely to score in the closing stages, with Chaplin sending a curling effort just past the post.
However, the hosts were left with their heads in their hands deep into stoppage-time when substitute Williams met Charlie Lee’s long throw and his header looped over McCarey and into the net.
Pompey (4-2-3-1): McCarey; Davies, Burgess, Webster, Stevens; Barton (Atangana 79), Doyle (c); Boco (Chaplin 71), Roberts, Bennett; Tubbs (Lavery 61)
Goals: Chaplin 77
Subs not used: Murphy, Clarke, Hollands, Naismith
Stevenage (4-1-4-1): Joronen; Franks, McCombe, Hughes (c), Okimo; Tonge; Kennedy (Lee 72), Parrett (Williams 90+1), Pett, Akinyemi (Conlon 57); Gnanduillet
Goals: Williams 90+3
Subs not used: Day, Wells, Jebb, Loza
Referee: Andy Woolmer
Attendance: 14,900 (133 away fans)