Psycholinguistics/Gesture

GESTURE

Introduction

Gesture appears not to be deliberate, but occurs during a large portion of speech.

In the words of Edward Sapir , gestures are "an elaborate and secret code that is written nowhere, known to none, and understood by all." It is extraordinary that humans have such a complicated, underlying communication system parallel with verbal speech that is so taken for granted.

The study of speech production must not be limited to verbal production; after all, speech as a whole includes lexical, emotional, semantic, phonological, syntactic, and motoric/gestural aspects. This non-verbal expression or gestural communication is largely unconscious, but makes up an important aspect of language production.

Are gesture and language really so interconnected? The evidence strongly points toward "yes"; though we may not notice our own actions, gesture appears to serve many functions, including more efficient and better communication of meaning, facilitated speech production and assisted language education. Even from a young age gesture is integral to language formation and its production. Clearly, the relationship between gesture and spoken language is more reciprocal and significant than one might intuitively think.

Types of Gestures

Iconic	These gestures are recognized as hand signals that convey meaning in both physical form and manner that co-occur with verbal meaning. Example: In this photo the man is describing a camera lens verbally, while simultaneously representing the size and shape of the lens with his hand.
Metaphoric	Metaphorics, or metaphorical gestures, are true to their name in that they are hand symbols that represent abstract ideas that are impossible to represent directly. Example: A fist motion upwards and twisted to the left may stand for the idea of freedom.
Beat	Unlike iconic gestures, beats are hand movements that emphasize the spoken discourse itself, the function of speech, and convey minimal or no extra information. Example: Up-and-down or back-and-forth hand movements that coincide with spoken clauses, breaks, or sentence ends.
Deictic	These are limited to pointing gestures that occur at the beginning of conversation or at the beginning of a new topic to reference a real or imagined target. Example: Someone may point off into the distance to reference a place or time.
Cohesive	These gestures incorporate iconics as a way to physically connect different parts of a narrative. Example: To connect a previous statement with a current one, someone may flick the hand backwards from the lap towards the ear.	A cohesive gesture.
Emblem	Emblems have direct meaning of their own and require no linguistic reinforcement. Typically they do not co-occur with speech as other gesture types do. Emblems tend to be culture specific, whereas co-speech gestures have interesting similarities across cultures. Example: In North America the thumbs-up gesture is usually interpreted as "good" or "ready". In many cultures, however, it is considered an obscene gesture.

Roles of Gesture in Speech Production

Information Communication

There are two main hypotheses of what role gesture plays in speech, one of which is to aid in the communication of meaning or information and to allow the transfer of supplementary information other than that presented in linguistic output. For example, when verbally describing how large a structure is, a person may also concurrently make hand motions that detail the shape. This hypothesis is also known as the Mutually Adaptive Modalities hypothesis, or MAM.

Research suggests that gesturing is beneficial for communication. When one is faced with ambiguity in speech, possibly due to background noise, incomprehensible requests, or unintelligible speech, a listener tends to rely on gestural input to provide extra information. In cases of when speech is comprehensible, studies have shown that participants still exhibit a more accurate understanding of instructions or narratives when speech is paired with gestures.

Wu and Coulson (2007) found that when participants were shown pictures congruent with both speech and gesture, smaller N300 and N400 Event-Related Potentials were elicited than when the pictures were only congruent with speech. In other words, pictures were easier to understand when gestures were combined with speech. This indicates that, corresponding with previous research, co-speech gestures allow for the better interpretation of spoken meaning.

In addition, co-speech gesture has been shown to facilitate the learning of new words in a foreign language , and young children may be able to understand the meaning of novel verbs when presented with gestural information and no speech.

So it appears that gesture could not only relieve speech ambiguity and supply additional meaning, but it could also aid in the understanding of novel words in both familiar and foreign languages.

This gesture could fulfill the role of information communication between speakers, but it could also act to aid this man in lexical access as he speaks.

Lexical Access

The second hypothesis of why we use gesture is to facilitate lexical access, or in different terms, to aid working memory in word retrieval during speech. When we speak of gesture assisted speech it becomes necessary to mention lexical affiliates; these are defined as the word a specific gesture is hypothesized to facilitate.

The reasoning behind the lexical access premise comes from studies that have found a link between cognitive tasks and motor movements. For instance, when told to imagine the shape of an inverted “S”, one may furrow their brow, change their gaze, and gesticulate. Physical movement has also been found to occur during mental arithmetic , the consideration of visual presentations , and during the course of memory processes. With this connection in mind, further research has demonstrated that motor movements may perform a supporting function for a few cognitive tasks .

During the course of speech it is well documented that gesturing occurs , and knowing that speech is a cognitive process, it stands to reason that there is another motive for co-speech gesture than just information communication; gesture may also assist lexical access. Krauss (1998) proposed that because human memory represents knowledge using many different dimensions (e.g. visuo-spatial and motoric), and the same memory may be stored in multiple domains, gesticulation (which reflects spatio-dynamic features), may aid in the activation of other memory formats, therefore assisting the lexical access.

Wesp, Hesse, and Keutmann (2001) also suggested that gestures help to maintain spatial representations in working memory; they found that when participants were asked to describe a painting from memory, they tended to gesture almost twice as much compared to when describing a painting in their immediate environment. In another study, this was further supported when participants exhibited more muscular activity in their dominant arm during memory retrieval of concrete and spatial words, i.e. those that can be more expressed in visual format.

A study by Morrel-Samuels and Krauss (1992) indicates that gestures occur extremely near to their lexical affiliates, and a gesture's duration is closely related to how long it takes a speaker to retrieve a word. Conjointly, it appears that more lexical gestures occur during spontaneous compared to rehearsed speech, indicating that gestures may be more prevalent in more hesitant speakers, and are performed to facilitate word retrieval. Continuing this line of research, Rauscher, Krauss, and Chen (1996) showed that when gesture was constrained, participants spoke more slowly, but only when the content of speech was spatial in nature; this supports the idea that gestures may maintain spatial representations and assist in memory retrieval.

These results have been further supported by Pine, Bird, and Kirk (2007) when they found that, during a naming task, children could name more words correctly and complete more 'tip-of-the-tongue' tasks when allowed to gesture than when their movements were restricted. Altogether, this research supports the lexical access hypothesis, and suggests that gestures are also important for speech production as well as information communication.

These two hypotheses have the most support and evidence, but a few newer ideas include that gesture may work to decrease cognitive load during language processing , and aid conceptual processes before speech. It is possible that gesture fills all of these roles, contributing in an important, yet subtle way to spoken language.

Relationship of Gesture with Speech

Neuropsychological Evidence

We know that gesture most likely serves a function of assisting lexical retrieval and/or communicating information, but speech also has other inherent connections with gesture. According to the evidence, one of the primary roles of gesture is lexical retrieval, so it would follow that those with neuropsychological damage would have this word access affected. Correspondingly, research on stroke patients has provided evidence that those with anomic aphasia (deficits in word naming and retrieval abilities with normal comprehension), exhibit more gestures during a narration than typical patients or those with visuo-spatial deficits. In addition, those with Wernicke's aphasia (deficits in comprehension with satisfactory levels of object naming and repetition) do not gesture as much as those with anomia or conduction aphasia (deficits in object naming and repetition with alright comprehension).

Rose and Douglas (2001) conducted a study in which they had patients with aphasia participate in a naming task; when instructed to point, visualize, and gesture, those with phonological access, storage, or encoding difficulties displayed significant improvement with iconic gesture. This research indicates that an underlying neurological connection between speech and gesture may be at work, and gestures may aid in the recovery of patients with aphasia due to their ability to promote lexical access.

Development

This adult and infant are participating in joint attention.

Gesture and spoken language appear to be highly interrelated in the brain; and this translates to the importance of gestural role in adult speech, but gesture also plays a central role during language development.

Gestures usually begin early, before speech, and are accordingly referred to as prelinguistic gestures; most infants begin gesturing between 6 and 10 months of age as a critical communication method. Another important step in language is learning the appropriate response to parental gestures, and this occurs around 9 and 12 months of age.

Zammit and Schafer (2010) found that the use of iconic gestures by mothers, controlling for maternal labeling and volubility, reliably predicted noun acquisition and comprehension. This indicates that joint attention between parents and infants serves to facilitate language acquisition. It appears that because gestures are synchronized with speech, it aids infants in deducing the meaning behind spoken words.

Up until verbal communication takes over, infants too use their own individual gestures to facilitate communication with parents; as children grow older, gestures remain, but take on a more subtle role. Gestures used by infants and children are referred to as deictic (pointing, references gestures) and representational (both referential and semantic content, e.g. waving goodbye or opening and closing the hand to refer to a fish. Colletta, Pellenq, and le Guidetti (2010) found a strong correlation between age, narrative complexity, and gesture use. This supports the hypothesis that as we age not only does our linguistic discourse grow more complex, but our co-speech gestures develop as well, from only deictic and representational to more complex gesticulation, in the context of narrative and social activity.

The link between cognition, language, and gestures is further expanded by other investigators, including Fenson and Ramsey (1980) , who found that along with the first two word combinations, two or more gestures combined into one functional movement occur near the same time, during Piaget's final stage of sensorimotor intelligence, around 20 months of age. Furthermore, significant correlations have been found between multi-word speech and multi-combinational gestures. This indicates that as speech progresses through development, so does gesticulation in a similar fashion.

So, gestures appear to be necessary in the development of language through parental input and social interaction, and as speech develops and becomes more complex throughout life, so does gesture.

Gesture - Inclusive Models

Comprehensive models of speech production incorporate speaking with co-speech gestures, in an attempt to understand language; here are the three main models:

The McNeill Integrated Model

In the McNeill integrated model, it has been proposed that gestures share a computational, or comprehension, stage with speech, thereby developing simultaneously as a part of the same underlying psychological processes to respond to external agents concurrently. This psychological source of a synchronous verbal gestural unit is called a 'growth point'. Additionally, because they occur together, gestures and speech most likely have the same or similar semantic information to communicate; this is especially true for iconic gestures. The focus is placed on iconic gestures, because they are visuo-spatial in nature and the Integration model concentrates on spatial information. This hypothesis would predict that when communicating in a noisy room, and when spatial information is being communicated over the phone, more gestures (mostly iconic with the phone) would be used.

McNeill (1985) illustrated his model by comparing the gestures and speech of five different people on describing a narrative involving a character entering and climbing up inside a drainpipe. On inspection, not only was the recorded spoken language similar, but so were the iconic gestures:

“...he tries going up inside of the drainpipe...” while the hand rises and points upward.

“...he tries climbing up in the drainspout of the building...” while the hand rises and starts to point up.

“...and he goes up through the pipe this time...” while the hand rises and the fingers form a basket.

“...this time he tries to go up inside the raingutter...” while the hand points and rises quickly.

“...as he tries climbing up the rain barrel...” while the hand flexes back and then upwards.

As these examples demonstrate, gestures occurred at the same point as speech, and described the same semantic information (spatial in nature). Furthermore, complex multi-combinational gestures tend to have pauses between semantic units which correspond with speech. This indicates that gestures mirror speech, and it is especially poignant since gestures, like language, are similar between speakers.

In this model, there are slight changes in meaning during co-speech gestures, and taken together with concurrent spoken word, a single complex meaning is derived from a lone cognitive growth point. An implication of this model is that imagistic thinking, aided through gesture, plays a central role is speech and its conceptualization.

The Sketch Model

The Sketch Model is a processing model for gesture and speech developed by Jan Peter de Ruiter (2006) . It is based on the gesture-inclusive McNeill model of language, in that it incorporates the idea of simultaneous growth points (here referred to as a conceptualization stage) and the hypothesis of shared semantic information into a more specific representation.

This model attempts to account for evidence for growth points provided by McNeill by proposing a single cognitive ‘conceptualizer’ that synchronizes gesture with speech plans and subsequently leads to overt production of both at once.

The Krauss Autonomous Model

The Krauss model differs from the previous ones in that it sees gesture and speech as non-integrated; in this way, gestures may serve to communicate some meaning, but their primary role is to facilitate lexical retrieval. So this model focuses on the ‘lexical access’ role of gestures while the McNeill and Sketch models focus on the ‘information communication’ role.

The Krauss and McNeill models are similar in that they both assume gestures occur very close to their lexical affiliates and may share a common origin at some point; in addition, they both concentrate on the idea that gestures more so reflect spatio-dynamic, or visuo-spatial, features of speech. Contrarily, this model assumes that there is no shared semantic conceptualization stage. This is based on evidence from tip-of-the-tongue (TOT) tasks like that of Frick-Horbury and Guttentag (1998) ; this study restricted the participant ability to gesture during TOT tasks and found that preventing gesticulation interfered with word retrieval. This has been further supported by others such as Pine, Bird, and Kirk (2007). Retrieval failures in these TOT tasks were mostly phonological so it was inferred that not only do gestures and speech not share a conceptualization stage, gestures must reflect phonological more than semantic encoding.

It was concluded that instead of gesture and speech being related at a semantic level during a conceptualization stage, it is much more likely that the influence of gestures on speech is with their ability to aid word retrieval in a spatio-dynamic fashion during phonological encoding. So in that way, speech and gesture diverge and run parallel to each other from the same point in working memory to meet at the overt production point. Some gestures may be intended for information communication purposes (such as emblematics), but most are solely produced for lexical retrieval.

This is a summary table of the major similar and differing aspects between gesture- inclusive models.

Conclusion

Finally, research into language must include gesticulation in its repertoire; as one studies the role of gesticulation in speech it becomes more and more apparent of just how important it really is: gesture serves to help the development of language in children, aid communication of information, facilitate lexical access and possibly much more. In this, gesture acts in a subtle, yet complex manner to reinforce human communication and bring us closer together.

Learning Exercises

Quiz Yourself! 1 This Gesture-Inclusive Model is based on the lexical access hypothesis The Krauss Autonomous Model. The Sketch Model. The Mcneill Integrated Model. 2 This is defined as the word gesticulation is hypothesized to facilitate: 3 During early development, gesture appears to be: Integral to communication and language development. Useful for early noun facilitation, but not crucial for communication or language. A natural instinct with no real use. 4 What type of gesture would this be classified as? Deictic Beat Cohesive This is not a gesture 5 Knowing that gesture aids lexical access, which aphasia patient would gesture the least? One with: Broca's Aphasia. Wernicke's Aphasia. Anomic Aphasia. Conduction Aphasia. 6 When Wu and Coulson (2007) found that smaller N300 and N400 ERP's were elicited when pictures were congruent with both speech and gesture,than when the pictures were only congruent with speech, they inferred that: Co-speech gesture aids lexical access. Co-speech gesture allows for better interpretation of meaning. Co-speech gesture aids both access and interpretation simultaneously. None of the above. 7 What type of gesture would this be classified as? Deictic Iconic Cohesive Metaphoric This is not a gesture Further Exercises 1) For this exercise you will need a pencil, paper, and the help of a friend: describe the following video to them with your hands held behind your back and see how long it takes them to understand what you mean, and successfully replicate it on paper. Had you been able to gesture, how long do you suppose it would have taken for them to understand and draw it? Which role of gesture do you suppose this provides evidence for? Roller coaster vertical loop 2) Observe someone in normal conversation. Make note of the types of gestures they use and in what context they use them. David McNeill (1985) postualted that most gestures occur for or in visuo-spatial contexts. Have your observations supported this? Why do you think they have or have not? 3) Describe iconic gestures and how they differ from the other types of gestures. Why would these gestures, and not necessarily others, aid the noun facilitation of children? Which linguistic model incorporating gesture could use this as evidence for their hypothesis, and why? 4) What are the characteristic difficulties of those with anomia (a type of aphasia)? Design a hypothetical therapy to help improve the speech deficits of people with anomia. Which role of co-speech gesture does your therapy focus on? Why do you think it would work and what evidence is there that would support your therapy? 5) Using your memory only fill in the following table with "YES" or "NO" answers and check your them against the table above. 6) Watch the following video of a speech and make note of the gestures used. Observe how difficult it can be to differentiate between gestures because of how fast and effortlessly they're made. Do you believe that, with regard to this speech, her gestures more so reflect lexical access or information communication? May their be another reason or use in this context for her gestures?