For the sixth successive home game Rovers fell behind and, despite a late goal from Liam Mandeville, were unable to claw anything back from the game.
Ferguson said: “We’re giving away games we should not be losing.
“I don’t think my goalkeeper’s had much to do at all.
“It was too easy for them. They’ve not had to work for it. Too easy. We’re giving too many goals away. Too sloppy.
“We had opportunities in the first half and we needed to take them.
“Again we’ve come back late on but we’re letting in far too many sloppy goals.
“It is a feeling of déjà-vu and it has to stop,” he added.
“We have to get back to the consistency we were showing, when we had a real momentum about us.”
Walsall’s victory capped a happy Rovers return for former boss Sean O’Driscoll, who was in the away dugout at the Keepmoat Stadium for the first time since being sacked by Doncaster in 2011.
After the game the Saddlers’ boss revealed his knowledge of the Keepmoat pitch played a part in his half time team-talk.
O’Driscoll said: “I thought we were organised and disciplined.
“I said at half time to them if we can make Doncaster’s back four defend, the pitch is as lively as ever.
“It’s a difficult pitch to play silky football on.
“I thought we did that really well and got the ball into the box pretty quickly when we got it wide.
“We were comfortable at 0-0 and even more comfortable at 2-0 but credit to them for getting back into it and putting us under pressure.”