Every valley has a lord and all of them distrust outsiders. The majority of the population is descended from the First Men. The Andals tried to conquer Crackclaw Point, but were defeated by their lack of knowledge of the area. The Darklyn kings of Duskendale tried to impose themselves on the houses of the Point, as did the Celtigars of Crab Isle. They were both beaten off. Eventually, the Andals learned that what they could not conquer with swords they could through marriage.
When not fighting would-be conquerors, internecine fighting between the houses was commonplace. Champions rising from local noble families would impose peace on the Point but it never survived their lifetime; these included Ser Clarence Crabb, the Brothers Brune, and Lord Lucifer Hardy.