In this Aug 6, 2020, file photo, Japanese Prime Minister Shinzo Abe walks off after delivering a speech during a ceremony to mark the 75th anniversary of the bombing at the Hiroshima Peace Memorial Park, in Hiroshima, western Japan.  (EUGENE HOSHIKO / AP)

TOKYO - Japanese Prime Minister Shinzo Abe is scheduled to return to work on Wednesday after a three-day summer break, during which he visited a hospital in Tokyo on Monday amid concerns from inside his own camp over his physical condition.

The prime minister's visit to Keio University Hospital on Monday fueled rumors about his deteriorating health

It is the first time for the Japanese leader not to visit his support base of Yamaguchi prefecture or his vacation home in Yamanashi prefecture since he returned to power in 2012. According to local media, he has told people close to him that the COVID-19 outbreak has forced him to change his routine summer visits.

The prime minister's visit to Keio University Hospital on Monday fueled rumors about his deteriorating health, while a number of people around him have expressed concerns about his exhaustion from continuously working.

ALSO READ: Pandemic derails Abe's plan to revive regional Japan with tourism

"I believe the prime minister has worked restlessly and has overused his body," said Natsuo Yamaguchi, the head of Komeito Party, the junior partner of the ruling coalition.

According to local media, Akira Amari, the tax policy chief of the ruling Liberal Democratic Party and others have suggested that Abe extend his summer break. However, the idea was rejected by the prime minister.

Abe, 65, during his first tenure as prime minister, which started in late September 2006, abruptly stepped down from his post in 2007 due to chronic ulcerative colitis, an intestinal disease.

Although some local media have recently run reports about his possible deteriorating health condition, top government spokesperson Chief Cabinet Secretary Yoshihide Suga denied Abe is in poor health, stating, "I see the prime minister every day, and I think he has no health problems at all as he has been carrying out his duties smoothly." 

READ MORE: Virus cases are surging in Japan, and Abe may be bowing out