Rovers exited the FA Cup at the Preliminary Round stage for the second successive season as they surrendered a two goal lead at Wordsworth Drive.
Having survived a first minute scare as keeper, Kyle Phillips, produced a fine save to deny Raith Plant, Rovers went ahead in the 5th minute as skipper, Dan Cleverley, sent over a right footed free kick from the right and just inside the Taunton half, which James Billing nodded down to Marcus Mapstone who stabbed the ball home at the second attempt.
After a period where neither side threatened Rovers doubled their lead in the 18th minute as James Vincent, making his debut, went away down the right and crossed for Cleverley whose shot was blocked but fell kindly to Vincent who fired home the rebound.
Having dominated the early exchanges and looking comfortable Rovers sat back and gradually the hosts began to take control although Kris Miller did go close for the visitors on 34 minutes firing over from a narrow angle.
Taunton were now enjoying more possession and territorial advantage but hardly threatened until a minute before the interval when Rovers’ full back, Stuart Tovey, was adjudged to have brought down Craig Herrod just inside the box and home skipper, Simon Ingram made no mistake from the penalty spot sending Phillips the wrong way.
After the break it was all Taunton as Phillips did well to save a curling shot from Ingram and then had to race from his goal and deflect a shot from James Blake.
It was no surprise when the hosts equalised, but there was an element of good fortune to it, as on 63 minutes a long range effort from Ingram took a wicked deflection, which left Phillips stranded.
Rovers did respond and felt they should have had a penalty minutes later when Miller went down in the box. However it always looked as though the home side were more likely to snatch the win and having missed a great opportunity a minute earlier when Herrod fired over from a Rodney Marsh cross, the come back was completed in the 85th minute.
The winner came with a long ball forward, which at full stretch was superbly controlled by Herrod, who then advanced before slipping the ball past Phillips and into the net.
Rovers nearly took the tie to extra time when in injury time Mapstone, now pushed forward, sent in a shot which was pushed away by Taunton keeper, Connor Sidley-Adams to ensure it was his team that visit Gillingham in the next round.
Rovers: Kyle Phillips, Stuart Tovey, Lee Davidson, Marcus Mapstone, Scott Brice, James Vincent (Josh Jeffries79 mins), Craig Loxton, Kris Miller, Dave Stone (Nick McCootie 59 mins), Dan Cleverley (C), James Billing
Sub not used: Nick Bunyard.
Rovers Man of the Match: Marcus Mapstone.
Referee: Adam Penwell
Assts: SavvasYianni / Kristian Gould
Attendance: 154