The building was built in 1861 during the reign of Napoleon III. It originally housed tennis courts; the name is from the precursor of tennis, the jeu de paume.
It was used from 1940 to 1944 to store Jewish cultural property looted by the Nazi regime in France.
Before 1986, it contained the Mus? du Jeu de Paume, which held many important impressionist works now in the Mus? d'Orsay.