The building was formerly a ‘byre’ (old English for cowshed) belonging to Milton Street Farm. The Farm belonged to the Dumbrell Family who owned it until the 21st Centry. The whole farm compromised of the farmhouse, the hay barn, the granary and the byre before the farm was spilt up into separate properties. Parts of the original Byre walls date back to the mid-13th century and is likely to have housed a plough team of oxen. These oxen teams survived for much longer on the South Downs than in other areas.
For a time, the farm was occupied by the painter, biographer and collector Sir Roland Penrose (1900-1984) and his wife the famous model and war photographer Lee Miller. Picasso visited the house in the 1960s.