Research
The research in our laboratory is funded by grants from the National Science Foundation and the National Institutes of Health. The overall aim of our research program is to determine how infants use their vision and hearing to perceive their world and to learn about its meaningful attributes. To this end, we investigate when and how various basic perceptual skills develop in infancy and how they provide the foundation for the development of basic cognitive abilities. We are particularly interested in the way babies and young children use their vision and hearing to figure out what the various objects and people in their world do, why they do it, and what their action means. For example, in some of our studies we investigate whether babies are able to perceive rhythm as a basic perceptual property. Rhythm is an important property of music and language. For example, as babies are learning language, their biggest problem is to figure out where words end and new ones begin. In an ongoing speech stream, the rhythmic pattern of language gives a young baby a particularly useful way to figure out where the words begin and end. In addition, when babies interact with adults and siblings, they get to hear and see speech and, thus, learn language. One of the questions that we have addressed in our studies is whether the synchrony between the movement of the lips and voice is important to infants' perception of other people and their speech. In some recent studies we have been investigating whether babies and children are able to perceive, learn, and discriminate sequences. Speech as well as most events around us consist of strings of words or actions that are arranged as structured sequences. In order for us to perceive the meanings that are created by the sequential structure, we must be able to perceive that structure. Our studies investigate the development of this critical perceptual and cognitive ability in early life. Finally, in our most recent work, we have uncovered some rather surprising facts about perceptual development: as we get older some of our perceptual abilities actually decline rather than improve. If you would like more details about these various studies, please read below:
Perceptual Development in Infancy
Auditory-Visual Integration
Our world is usually multisensory in nature. In other words, the people and the inanimate objects that comprise our perceptual world are usually specified concurrently by separate and distinct sensations. We can see them, hear them, and sometimes feel, smell, and taste them as well. Thus, our world is a veritable cornucopia of sensory experiences. This, however, is the very problem: how do we organize this cornucopia into meaningfully unified experiences? Fortunately, the human brain has evolved as an organ that not only receives and process this cornucopia of sensations but also unifies them into meaningful experiences that transcend the sensory modality which specifies. Thus, rather than perceiving faces and voices of the people that we interact with in our everyday world as separate features of our social world, we experience them as attributes of particular people. This ability to unify disparate sensory experiences into meaningful wholes is fundamental to adaptive functioning in the perceptual cognitive, and social domains. Therefore, as might be expected, it emerges in infancy and this laboratory was one of the first to discover when some basic intersensory integration abilities emerge in infancy and then characterize how these abilities change over early human development. There are two main cues that enable us to perform intersensory integration: time and space. As events unfold over time, their visual and auditory attributes correspond in terms of synchrony, duration, tempo, and rhythmical patterning relations. Our research investigates when in infancy the ability to respond to audiovisual events as unified entities first emerges, how it changes over early development, and how it might facilitate early learning and the acquisition of basic cognitive skills (e.g., sequence learning). A recent article describes one of our studies that investigated infant perception of audio-visual synchrony relations and the way that short-term learning can affect this ability.
Sequence Learning
Many
events in our world occur over time. Music, dance, typing on a
keyboard, and language all illustrate the importance of
stringing together a series of elements into specific sequences
that have different meanings depending on the way the elements
are arranged. In music, a given series of notes can give rise to
very different melodies depending on the way they are arranged.
In language, the meaning of the phrase - the boy hit the tree -
has clear meaning, whereas the phrase - the tree hit the boy -
is meaningless. Finally, the behaviors of people around us
consist of strings of actions whose specific meaning depends, in
part, on the specific arrangement of their actions over time.
Indeed, we are very good at extracting and interpreting other
peoples' intentions from the sequential structure of their
actions. Given the fundamental importance of sequences for both
the production of behavior and its interpretation, it is
important to study how sequence learning skills emerge in early
human development. Currently, we are investigating the
developmental emergence of specific types of sequence learning
skills in infancy. The picture on the left shows a baby watching
as 4 different objects move and make sounds on the computer
screen. After the baby learns a particular ordering of the
objects and their sounds we then change their ordering and
measure the baby's visual attention to determine if he can
detect the change. Here is a recent article that reports on a series of studies in which we asked whether young infants are able to learn sequential relations among objects and their associated sounds and the basis for such learning (i.e., do they learn them on the basis of ordinal information or statistical relations).
Perceptual Narrowing
According to conventional wisdom, as we get older - from infancy up to adulthood - our perceptual and cognitive abilities improve While that is generally true, a growing body of evidence has demonstrated that this is not always the case. This evidence now makes it clear that as we grow we also become specialized for the processing of native kinds of information (e.g., the speech sounds belonging to our own language as opposed to those belonging to other languages and human faces as opposed to the faces of other species). As we do so, we become experts at perceiving native as opposed to non-native perceptual information but at a cost. The cost is that our ability to respond to non-native perceptual declines. This process is known as perceptual narrowing and is the result of the selective experience that we have in the specific environment that we grow up in.
One major focus of the research in our laboratory investigates the process of perceptual narrowing with the specific aim of wanting to better understand how perceptual narrowing contributes to our ability to perceive and integrate the multisensory attributes of our environment. In particular, we are interested in how infants come to understand the most important source of communication: the vocalizing face. We have conducted several studies in collaboration with colleagues at Princeton University, McGill University (Canada), Universities of Barcelona and Pompeu Fabra (Spain), and the University of Padova (Italy) to shed light on the development of face-voice integration in human infants and vervet monkeys.
Decline in Responsiveness to Non-Native Faces and Vocalizations
In the first study in this line of investigation, published in the Proceedings of the National Academy of Sciences, we showed for the first time that infants' ability to perceive non-native faces and vocalizations as part-and-parcel of the same unitary event actually declines during the first year of life even though this ability does not decline when infants see and hear human faces and vocalizations. That is, prior studies have shown that infants between 2 and 12 months of age are equally good at matching human faces and vocalizations. For example, if you show side-by-side faces of the same person articulating the vowels /a/ and /i/ and, at the same time, play either vowel through a centrally placed speaker, babies will look longer at the matching visual articulation of the audible vowel. This indicates that babies can perceive the unity of the visible and audible attributes of speech. We wondered how broad this ability to make audio-visual matches is and whether it might change in development. As babies grow they acquire increasingly greater experience with human faces and vocalizations but do not have many experiences with the faces and vocalizations of other species. If the effects of that experience are important for the ultimate perceptual abilities of infants then it might be that this selective experience with human faces and vocalizations over time might lead to a decline in the ability to integrate unfamiliar facial and vocal information. To test this hypothesis, we used the intersensory matching task and showed infants side-by-side movies of a monkey face producing two types of vocalizations that monkeys typically produce (either a 'coo' or a 'grunt') and at the same time played one of these audible calls through a speaker located between the two faces and measured the amount of time that infants looked at each face while listening to the audible call.
|
"coo" |
"grunt" |
|
|
We found that younger babies looked longer at the matching face but that the older infants did not indicating that the younger but not older infants integrated the visual and auditory information. This intriguing finding indicates that young infants are broadly tuned and sensitive to the facial gestures and accompanying vocalizations of different species but that as they grow, due to specific and selective experience with human faces and vocalizations, they becomes specialized for them and, as they do, their intersensory integration abilities actually decline. Put differently, as infants become experts at perceiving human faces and vocalizations their initially broadly tuned perceptual abilities narrow in scope.
Matching of Monkey Faces & Vocalizations in Newborn Infants
More recently, we have conducted a follow-up study to determine whether the broad intersensory perceptual tuning that we discovered in 4-6 month-old infants in our initial study described above is present right at birth or whether it emerges after birth. Thus, we repeated the initial study but this time with newborn infants. As expected, we found that newborns also can match monkey audible and visible calls. To read more about this study, please click here.
Decline in Responsiveness to Non-Native Audiovisual Speech
In another one of our follow-up studies, also published in the
Proceedings of the National Academy of Sciences, we have
shown for the first time that intersensory perceptual narrowing
also occurs in infant response to audiovisual speech. We used
the intersensory matching technique once again with a minor
modification. First infants watched side-by-side videos of the
same person silently and repeatedly uttering a /ba/ syllable on
one computer monitor and a /va/ syllable on the other monitor.
Results
of this test showed that the infants did not prefer one syllable
over the other. Then, we allowed the infants to listen to either
the /ba/ or the /va/ syllable for 45 seconds. After they
finished listening to the syllable, we once again tested the
infants' preferences for the visual syllables by showing the
silent versions of the side-by-side syllables. We tested 6- and
11-month-old Spanish-learning and English-learning infants and
expected that that Spanish-learning infants would exhibit a
decline in responsiveness because the /v/ sound does not exist
in the Spanish language but that the English-learning infants
would not. The results were consistent with our prediction. We
found that 6-month-old Spanish-learning infants showed a clear
preference for the visible syllable that matched the audible
syllable that they had just heard - indicating that they
perceived them as belonging together - but that the 11-month-old
Spanish-learning infants did not. This failure of the older
infants to integrate the audible and visible speech information
suggests that the older Spanish-learning infants’ greater
experience with the Spanish language reduced their sensitivity
to the visible and audible sounds of other languages. In
contrast to the decline in the ability to integrate auditory and
visual speech in Spanish-learning infants, we found that both 6-
and 11-month-old English-learning infants successfully matched
the audible and visible syllables. In addition, we confirmed
that the decline in the perception of non-native audiovisual
speech persists into adulthood by showing that Spanish adults
were unable to integrate the same syllables that were presented
to the infants but that English adults easily did. In sum, these
results extend our previous results and demonstrate that
intersensory perceptual narrowing is a general phenomenon that
plays a critical role in the development of audiovisual speech
responsiveness.
Development of Selective Attention to Talking Faces & Audiovisual Objects
In this line of research we are investigating when and how visual attention changes across the first year of life in response to talking faces and moving/sounding objects. As infants grow, their perceptual and cognitive abilities change rapidly and, as a result, they become increasingly more sophisticated in their preferences. For example, as infants grow, they quickly become experts in perceiving faces and the vocalizations that they produce. As they do so, they shift their attention to those features of vocalizing faces that are most important from the standpoint of communication and social interaction.
We utilize state-of-the-art eye tracking technology to characterize the shifting patterns of audi
ovisual attention across infancy and beyond. The picture to the right shows a baby seated in front of a computer monitor displaying a woman talking. During this time, we monitor the baby's eye movements and eye tracker device (it can be seen below the video monitor). In our latest studies of infant response to talking human faces, we have discovered that as infants grow, their attention shifts from an initial primary focus on the eyes at 4 months of age to a primary focus on the mouth at 8 & 10 months of age when infants are exposed to someone talking in their native language. We found the same developmental pattern in infants exposed to a nonnative language and, in addition, we found that even at 12 months infants look more at the mouth. We believe that the shift from the eyes to the mouth and then back to the eyes in infants exposed to their native langauge reflects the emergence of speech production capacity. That is, as infants begin to learn how to talk they become very interested in their social partners' mouths and begin to lip read to learn how to produce their native speech forms. The cease to lip read by around 12 months of age and begin to shift to the eyes to access the many social cues available there unless they are exposed to a foreign language in which case they continue to lip read to disambiguate what is now unfamilar to them. You may read more about these findings in our paper published in the Proceedings of the National Academy of Sciences.
Perceptual & Cognitive Development in Pre-School Children
Our
studies with preschoolers are an extension of our infant studies.
Their purpose is to investigate how the basic sequential perception
skills that emerge during infancy become transformed in childhood
and become incorporated into higher-level cognitive abilities that
permit children to learn and understand sequences. As children enter
the preschool years, they are faced with increasing demands to learn
and understand all sorts of sequences. For example, they need to
learn the proper sequence of actions involved in putting on their
shoes, the proper finger movements to play a particular musical
piece on a piano, or the proper way to string together a series of
different words to produce a correct and meaningful sentence. These
various abilities require that they correctly perceive the
particular sequence first and that they then understand the overall
structure of the sequence. Although preschoolers appear to exhibit
an impressive understanding of order in some situations (e.g., when
they engage in certain social routines), they have considerable
difficulty understanding order in other situations (when looking at
simple shapes that are presented one at a time on a computer
screen). In our laboratory, we are interested in the various factors
that influence young children's understanding of sequences and their
ability to perceive the specific serial ordering of the elements
that comprise those sequences. Currently, we are investigating the
idea that certain factors (such as familiarity of the context and
the presence of meaningful goals or intentions) play an important
role in the development of a basic understanding of serial order.
Please click here to learn about the methods we employ to investigate the above issues