Police were called to Somerset Road in Hyde Park following an incident near the street’s junction with Apley Road.
The street was cordoned off in an area outside Leo’s Mini Market.
There have been unconfimed reports that the man was stabbed inside the shop.
A South Yorkshire Police spokesman said: “Police were called at around 11.22am this morning to reports that a man had been stabbed in Somerset Road.
“Officers attended the scene and discovered a man in his 60s with a stab wound to his abdomen.
"He was taken to hospital via ambulance where he remains in a serious condition.
“An investigation has been launched and officers remain at the scene whilst they conduct their enquiries.”
The area has been taped off with numerous emergency service vehicles in attendance.
Anyone with information can contact police on 101.