Castres-en-Albigenses was a dependence of the Viscount of Albi. The Viscounts of Albi granted Castres a city charter establishing a commune with the city, headed by consuls. During the Albigensian Crusade, the city quickly surrendered to Simon de Montfort, who gave it to his brother Guy de Montfort.