Dubbing, subtitling, transportation and the redundancy effect
We have done a number of trials where either translators, video dubbing, or captions were used to assist witnesses for whom English was not a first language and, in one amusing instance, for a man whose Southern U.S. regional dialect was so thick that mock jurors wondered aloud if he was even speaking English. Oddly, jurors thought highly of him–they simply couldn’t understand him. It was a very complex technical case and the hard-to-comprehend witness was a field engineer (highly capable but ‘pure country’, and not well-educated). Jurors were sure they would value his opinions, if they could understand him.
Recently, in the midst of trial, the question was raised as to whether there was research directly speaking to whether it was better to use subtitle captioning on witness videos routinely or whether it was better to use them very sparingly. We were not aware of research specifically on this issue and checked around a bit to see if anyone else was (and they were not) so made the following recommendations:
Don’t use captioning routinely,
Do use them for witnesses who are hard to understand or require a translator, and,
Do use them for your witnesses with distracting non-verbal habits so the listener attention is less intently focused on the odd things the witness is doing while speaking.
But the question itself made us wonder about what recent research has been done in this area. We reviewed the multimedia research area and identified a few research-based principles that we will act on in the future until someone out there does some research on the impact of these different efforts on how witnesses are perceived. (Hint: This would be easy and useful research for some graduate student out there!)
Is the potential juror a foreign film devotee? This may seem like an odd question for voir dire, but it has to do with comfort reading subtitles. If you are using subtitling/captions, it might help you to have jurors who are comfortable switching their attention back and forth between the images and the text. [And, conversely, you may want to avoid those who are comfortable with this activity depending on your goal with the witness.] As true as this tip is, there are very few venues where you’ll run into many folks who say “yes” to this question, and they are undoubtedly different on a variety of levels.
Do you want to downplay the impact of visual information? This is an example of a witness/party with very distracting nonverbal behaviors, like perhaps, putting his fist in his mouth. Research has demonstrated that when subtitles are present, we automatically read. Further, “subtitles appear to exert a detrimental effect on visual information processing but a facilitating one on linguistic information processing” (Lavaur & Bairstow, 2011). In other words, if your witness has a very distracting nonverbal habit (in a recent project one of our mock jurors was convinced a witness was repeatedly but subtly “showing the middle finger” to the questioning attorney), it might be a good idea to use subtitles and reduce the amount of attention on the visual image itself.
Is dubbing or subtitling going to reduce ‘transportation’? We’ve written about the importance of getting caught up in a story before–that experience is often referred to as ‘transportation’. A study in Switzerland (Wissmath, Weibel & Groner, 2009) looked at multiple effects of enjoyment depending on whether films were subtitled or dubbed and found no difference in the level of story involvement when viewers either read subtitles or had voices dubbed with their own language to allow understanding. On the other hand, Switzerland is a unique country with four national languages (French, Italian, German and Romansch). In contrast, we have people from Chicago who can’t understand witnesses from the bayous of Louisiana. Sigh.
We would also say that while there is no difference between subtitling and dubbing, the experience of the viewer is more thorough and immersing when they are listening to a language with which they are fluent. This is also consistent with what these researchers found. We have often had mock jurors complain that they found it hard to get a sense of a witness when they relied on translation, dubbing or subtitling. They indicated wanting to appreciate nonverbal differences in both tone and emotional expression that made it harder to ‘know’ the speaker. They would identify larger issues–like “she seems nervous” but were less comfortable saying they found a witness dishonest because they were missing so much information they were wary of making finer distinctions. (This is inconsistent with earlier work we’ve blogged about indicating we see people with accents as less honest than those without accents.)
Is the redundancy effect relevant? The redundancy effect (Sweller, et al., 2011) is the reality many of us have experienced when a speaker reads a PowerPoint slide presentation to us. When we see and hear the same information, our retention actually decreases due to the redundancy effect. Here’s a more formal definition:
“The redundancy effect refers to the phenomenon in instruction where learning is hindered when additional information is presented to learners compared to the presentation of less information. It can take one of two forms.
First, when identical information is given in two or more forms such as pictures and words or text in both written and audio form. If one of these forms is redundant, eliminating it may enhance learning.
Second, when additional information is given in order to enhance or elaborate information, for example, a full text and a summarized text. If the elaborations in the full text are redundant, then the elimination of the additional information may result in enhanced learning.”
So, if a witness video is subtitled when their speech is easily understood, will the observer retain less information about the content? Redundancy effect says yes–because of how the eye is automatically drawn to the subtitle text. That’s why you subtitle someone who is nonverbally distracting so they are less closely observed visually when the eye is automatically drawn to the text. You cannot look at the visual image and at the subtitled text at the same time and so you miss information. You also may be more intent on whether the subtitles are accurate or paraphrased and therefore miss even more visual information.
This is hardly an exhaustive list but we looked pretty intently for research specifically testing this simple (and highly relevant) activity on viewer impressions of witnesses to no avail. It does, however, give us some guidance on this question. But more than that, it tells us that some actual research focused on witness impressions would be very useful.
For the fourth year in a row we have been honored with recognition from the ABA via inclusion in their 2013 list of the Top 100 legal blogs in the country. We work hard to blog consistently even when inundated with work and would appreciate your vote for us at the Blawg 100 site under the LITIGATION category. You will have to register your email just so you can’t vote 47 times. There are many worthwhile law blogs on this list so take some time to peruse. Thanks! Doug and Rita
Lavaur JM, & Bairstow D (2011). Languages on the screen: is film comprehension related to the viewers’ fluency level and to the language in the subtitles? International Journal of Psychology, 46 (6), 455-62 PMID: 22046988
Sweller, J.,, Ayres, P.,, & Kalyuga, S. (2011). Chapter 11: The Redundancy Effect. Cognitive Load Theory. DOI: 10.1007/978-1-4419-8126-4_11
Wissmath, B., Weibel, D., & Groner, R. (2009). Dubbing or subtitling? Effects on spatial presence, transportation, flow and enjoyment. Journal of Media Psychology, 21 (3) DOI: 10.1027/1864-1188.8.131.52