Firefighters said the fire is thought to have broken out in the emergency room of the hospital in Miryang on Friday morning.
Officials said there are around 79 people injured, eight of those in a critical condition.
Yonhap news agency reported the death toll stood at 41 people.
During the early stages of the rescue operation, fire chief Choi Man-Wu said: "Two nurses said they had seen fire suddenly erupting in the emergency room.
"The victims came both from the hospital and the nursing home. Some died on their way to another hospital."
Sejong Hospital also has a nursing home for the elderly, and there were 194 people in the building when the fire broke out, according to police.
Most of those who died were on the first and second floors of the six-storey building, and none died of burns.
According to AFP, Mr Man-Wu said all patients had been brought out of the building, but there had been a delay getting patients out of the intensive care unit, where firefighters had to wait for medical staff to supervise the process.
Survivors were carried out wrapped in blankets, and images show the firefighters tackling the dark smoke as the fire engulfed the building.
In a press conference, Mr Man-Wu said: "About 90 patients at the care hospital cannot move by themselves.
"Therefore, we primarily rescued all 94 patients [at the care hospital] and completed their transfers [to another hospital].
"Yet, there were about 100 patients at Sejong Hospital - in front of the care hospital - and among those hundred patients, some of them were in the intensive care unit on the third floor. The 15 patients in the intensive care unit were wearing oxygen masks."
The fire broke out at 7.35am local time (10.30pm GMT) and was extinguished by 10.26am (1.30am GMT), officials said.
South Korean President Moon Jae-In held an emergency meeting after the fire to discuss response measures.
The cause of the fire is under investigation.
It is one of the deadliest fires to sweep South Korea in recent months.
It comes just a month after 29 people were killed in a blaze at a fitness club in Jecheon, which was blamed on insufficient emergency exits and flammable finishing materials.