Crowd behavior analysis is a challenging task in computer vision, mainly due to the high complexity of the interactions between groups and individuals. We establish the first baseline of crowd characterization with an extensive evaluation on shallow and deep methods. This characterization is expected to be useful in multiple crowd analysis circumstances, we present a new deep architecture for crowd characterization and demonstrate its application in the context of anomaly classification.