Originally from Sauve in Languedoc, the Greffulhe family left France at the time of the Revocation of the Edict of Nantes. After taking refuge in Geneva, they moved to Amsterdam, where they founded a business under the name J. J. Greffulhe et Cie. In May 1789, Louis Greffulhe (1741-1810) relocated to Paris, and in association with Jacques-Marc Montz, under the name Greffulhe, Montz et Co., took over the business of the Girardot et Haller bank, founded at the beginning of the century and with which Necker had been associated. The bank was dissolved in 1793, and Louis Greffulhe left for England to set up a new banking house in London. The Greffulhe family returned to France under the the reign of Napoleon. Jean-Louis (1774-1820), son of Louis Greffulhe, later allied himself with the Bourbons and eventually acquired the Château de Bois-Boudran near Fontenailles in the Seine-et-Marne department. In 1816, Jean-Louis was one of a group of French bankers associated with Baring and Hope to issue loans to aid the French royalists. Two years later he received the title of Count from Louis XVIII. Jean-Louis and his wife Célestine Gabrielle de Vintimille du Luc (1787-1862) had two sons: Charles (1814-1888), who inherited his business and the Bois-Boudran property and, around 1867, purchased the hôtels at 8 and 10 rue d'Astorg, and Henri (1815-1879). Charles Greffulhe married Félicité Pauline de La Rochefoucauld (1824-1911) on April 29, 1846.
They had three children: Henri (1848-1932); Jeanne (1850-1891) who married Auguste, duc d'Arenberg; and Louise (1852-1932) who married Robert des Acres de l'Aigle, marquis de L'Aigle.

In 1878, Henri married Elisabeth de Riquet, princesse de Caraman-Chimay (1860-1952). Elisabeth was a renowned beauty and the uncontested queen of the salons of the Faubourg Saint-Germain. She was idolized by Proust, among others, and famously served as the inspiration for the Duchesse de Guermantes in his novel A la Recherche du Temps Perdu. Widely respected as an arbiter of taste, she launched a fashion for greyhound racing, was a patron of Diaghilev’s Ballets Russes and promoted many other artists in high society which included Rodin, Moreau and, in particular, Whistler. The comte and comtesse shared their time between their Parisian hôtel on the rue d’Astorg and the château de Bois-Boudran, Seine-et-Marne. It is not certain, however, which of the two Greffuhle residences was home to the the present bergère. The Greffuhle collections were partly dispersed several years after Henry’s death in 1932, while the rest was bequeathed to the couple’s only child Élaine (1882-1958), wife of Armand de Gramont (1879-1962), after Elisabeth’s death.