Though it is not certain, this garden is believed to have been built from 1688 to 1703 by a lord of the Hitachi-Kasama domain, Honjo Inabanokami Munesuke, younger brother of Kensho-in, mother of the fifth Tokugawa Shogun Tsunayoshi. The garden had a strolling style with the level of water drawn from the Sumida River rising and has falling with the tides. This garden was remodeled in 1894 by Zenjiro Yasuda, founder of the Yasuda financial group. It was granted to Tokyo City according to his dying wish in 1922 and has opened to the public since 1927. This is the first Japanese-style garden opened thanks to a contribution of a private philanthropist.

Unfortunately this garden was destroyed by the Great Kanto Earthquake and World War II. It was also ruined by pollution of the Sumida River. After the jurisdiction of the garden was transferred from the Metropolis of Tokyo to Sumida City in 1969, it underwent an overall renovation and newly opened to the public in 1971.

Since this garden successfully retains an appearance described in literature of the Meiji period up to the present, it is one of the typical gardens of this period comparable to Kiyosumi Gardens.