Q&A: New Tracking Techniques Reveal Unexpected Social Behaviors

Annegret Falkner, a neuroscientist at Princeton University, estimates she spent hundreds of hours of her postdoc staring at videos of mice, manually tracking their every move. But those tedious days may be largely behind her, thanks to new technologies for automatically tracking animals’ behavior. These techniques employ advances in machine learning and computer vision to parse the structure of animals’ behavior. The result is a quantitative description of behavior that detects features that may be missed by human observers. Researchers combine movement tracking with neural recording to decipher how the brain produces behavior.

Initial studies employing these techniques focused on individual animals. (For more on automated tracking techniques, see: “Decoding Body Language Reveals How the Brain Organizes Behavior” and “New Dataset Explores Neuronal Basis of Behavior.”) But researchers are now starting to use these technologies to look at social behavior — pairs or groups of animals interacting.

Social activities, such as mating, fighting or nesting, are essential for survival and often make up the bulk of animals’ behavioral repertoire. But because social interactions are complex and often difficult to control, our understanding of their neural dynamics has lagged behind that of solitary tasks. An animal’s social history, state of mind and hormone levels all come into play when it’s defending its territory or searching for a mate.

To help drive the field forward, Falkner and her colleague Ann Kennedy, a postdoc in David Anderson’s lab at the California Institute of Technology, organized a workshop at the Cosyne conference in Portugal in March, titled “Quantifying social behaviors: Computational challenges and experimental pitfalls.” A slew of researchers shared their newest techniques and results, as well as their thoughts on the field’s biggest challenges.

Falkner talked with the Simons Collaboration on the Global Brain (SCGB) about the promise and challenges of studying social behavior, and some of the highlights from the workshop. An edited version of the conversation follows.

Why is it so important to study social behavior?

It’s such a vital part of existence. Social behavior comprises a large proportion of what an animal does. A lot of an animal’s behaviors are in service of social needs, such as caring for young, finding a mate or defending territory. You could argue that the whole brain exists in service of social needs.

What’s interesting about social behavior is it seems to have proprietary modulatory systems — evolution has put into place systems to generate and reinforce these behaviors. They are ancient functions of the brain that do not require the cortex and are evolutionarily conserved from fish to mammals.

Given that social behavior is so essential, why has it taken so long to study it in this way?

It’s only in the past 5 to 10 years that people have explored recording from the deep brain structures that govern social behavior. There has also been a lot of progress on tools for automated behavioral tracking, such as DeepLabCut and MoSeq, which helps to quantify behavior.

What inspired you to organize this workshop?

I think the time is right for it. The field has moved a long way in a short time, from the state of the art being manual annotation to being able to systematize behavioral quantification across labs. Now we have a handle on what people are talking about when talking about social behaviors.

Moreover, we can now record the neural dynamics during social behaviors. So people who do more straight systems neuroscience are starting to become more interested. We can quantify behavior in a more precise way and use that to look at dynamics and predict which signals are tied to certain behaviors. Ann [Kennedy] and I had a good time talking about these things over the past 18 months and watching the field move in the way we wanted it to go. We thought it would be a fun excuse to get people in a room together.

Why is social behavior more challenging to study than other types of behavior?

For one, it occurs on a variety of timescales. Some behavioral changes occur on the millisecond to second timescale based on things that happen in the environment, such as a direct interaction with a potential mate or competitor. Neuromodulators, such as neuropeptides and sex hormones, which control socially motivated behaviors, act on a timescale of hours to days. Other, longer-term interactions, such as social learning, occur on timescales we don’t even know. Given that diversity, thinking about the appropriate timescale to quantify behavior has been a real challenge. Having tools that allow us to record longitudinally from social behavior circuits will be useful to stitch together disparate timescales.

It’s also difficult to quantify behavior when no two instantiations of a behavior are identical. For example, Ann and I have both struggled with how to define an attack, because no pair of mouse attacks are ever identical. Looking for statistical descriptions of groups of behaviors might be a good solution to this.

What other factors complicate analysis of social behavior?

Another challenge with social behavior is that an animal’s previous experience has a profound effect on behavior. This is the case for lots of behaviors but is especially true for social behavior. We struggled with how to provide a quantitative description of this experience, which could include how much sexual or fighting experience they have had and how they were housed. How much can we expect people to report? It’s frustrating reading the literature and not being able to interpret the data because you don’t know anything about the history of the animal. The more people we have working on social behavior, the more we will run into this interpretability problem.

The International Brain Lab has had a lot of success in trying to control every variable going into an experiment so that the results are interpretable. It would be nice to have a social version of the IBL task. Do we get the same neural response if we can control all those conditions? But maybe we just don’t know enough about social behavior at this point. Maybe we just need to let everyone do their own thing and worry about collapsing that space later.

How are people using new technologies to study social behavior at different timescales?

Taiga Abe, a graduate student with John Cunningham at Columbia University and Ioana Carcea at Rutgers University, gave a nice demonstration at the workshop of how it’s possible to pull complex behaviors you’re not even looking for out of video data. Starting with DeepLabCut, they built an algorithm to track the emergence of maternal shepherding behaviors, in which female mice herd their pups into the nest. In these videos, you see how these behaviors evolve across many hours as mothers recruit females to help care for pups. In doing this, they found unexpected pre–shepherding behaviors, in which naive female mice (that have never had pups) trail the mother. This behavior then transitions into shepherding behaviors across time.

What other interesting social behaviors are people studying with these new quantitative methods?

Adam Calhoun, a postdoc in Mala Murthy’s lab at Princeton University, talked about using generalized linear (GLM) and hidden Markov models to describe how flies shift courtship strategies. He developed models in which animals can exist in different internal states. They show a different distribution of behavior depending on the internal state. I think this will be useful for a lot of people in thinking about how internal state affects behavior. (For more on this research, see: “How Fruit Flies Woo a Mate.”)

Oren Forkosh, a postdoc in Iain Couzin’s lab at the Max Planck Institute in Germany, is on the opposite end of the spectrum, trying to develop a quantitative way to describe an animal’s personality — particularly aspects of personality that should be persistent. He looked for behavioral features that don’t change when animals move from group to group. They found that these features cluster into domains they think map onto aspects of mouse personality. He cautioned not to think about the domains as corresponding to submission or aggression, but more like clusters of traits. An alpha mouse is not always alpha. But an animal might show something like ‘boldness’ systematically in different groups.

Gordon Berman of Emory University made the physicist’s case for why modeling the behavior of multiple animals is a more difficult problem than modeling interactions between molecules or even flocks of animals, where the exact details of any particular interaction can be averaged out. This is not the case for pairs of animals. Two are tough because there is interesting structure existing on multiple timescales.

Bob Datta of Harvard University also had some interesting new ideas of how to benchmark new models of behavior. He created a common database of mouse behavior that tracks behavioral changes in animals that have been given different kinds of drugs. You can ask how well a model is doing by asking how well it predicts the drug the animal has been given.

What do you hope to see next?

Now that we can record over longer timescales, what we really need is new methods for interpreting complex data that occur over multiple timescales.