The castle was originally built on land granted to Odo de Barri, a Norman knight, at the end of the 11th century. Initially he constructed a Motte-and-bailey on the site which had a wooden keep defended by a palisade and earthworks embankments. But it was Odo's son, William de Barri, who began work on the stone fortification in the early part of the 12th century.



In 1146 Gerald of Wales, the great twelfth century scholar known as Geraldus Cambrensis was born at the castle. As the fourth son of William de Barri, he was related on his mothers's side, to the legendary Welsh princess Nest verch Rhys. Gerald wrote of his birthplace:



In all the broad lands of Wales, Manorbier is the most pleasant place by far.