Skip to main content

An international collaborative replication study

Will and Timothy are joined by guest Dr. Mario Baldassari, for a chat about an international collaboration to replicate a previously published study.

In this chat format, we gather regular authors and guests in Slack and have a moderated conversation, guided by prompts and questions selected in advance. Participants get to respond to each other's points, make comments, and ask each other questions in real-time.

The transcript has been lightly edited.



Will Crozier 🐙
Welcome to another Exercise in Exceptions chat! Today we’re joined by Dr. Mario Baldassari to talk about an issue that isn’t directly related to psych and law, but science in general. Mario was recently a part of a team that did an internationally collaborative replication – that is, a large team of researchers across the world ran a previously-published psychology study, to see if it still worked (or how far the effects generalized).

Big international collaborations that produce replications are a bit of a new feature of scientific research, and Timothy and I have never done one before - so we wanted to get a first-hand account of how they work, and the implications for psychology research going forward.

Thanks for joining us, Mario! Care to introduce yourself?

Mario
Hi Timothy and Will, happy to be with you. I’m Mario Baldassari, soon-to-be Adjunct Professor at Trinity University and current and future Adjunct at St. Mary’s University, both in San Antonio, Texas. My research focuses primarily on eyewitness identification, but I have worked on other projects within the realm of memory such as other race effects in face recognition and false memory (which is what we’re here to talk about).

Will Crozier 🐙
A fellow psych and law researcher!

Mario
I grew up in Indiana, did my PhD in Victoria, BC. I use he/his pronouns. And yes, P&L is my bread and butter. I'm very excited to teach Psych & Law at Trinity this fall!

Will Crozier 🐙
It's definitely fun to teach your specialty psych class.

As always, Timothy is here today. How's summer treating you in the far reaches of northern Europe? Are you enjoying the 110°F+ heat indexes we're witnessing back here in the states?

rabbitsnore (Timothy J. Luke)
You may be having a heatwave there, but here, the sun sets around 10 or 11 at night during the summer -- which is gradually driving me insane.

Mario
Oh wow, how early does it rise?

rabbitsnore
It starts getting light out around 4am. By 6am, it's fully bright.

Mario
Yeesh. We whined about weeks in a row of 100% gray days in Victoria, but only 5 hours of dark must have its own psychological effects.

Will Crozier 🐙
*schedules a future blog post about Timothy's health and sunlight*

I've given a bit of an intro to replications and international collabs, but before we get into what that process is like, I think we should talk a bit about the original study. Basically, you picked a study that had already been published, and decided to run it again. Can you tell us a bit about the original study?

Mario
Yes, and I will start by saying that I served as a cog in a much bigger machine. To call this “my” project in any way would be disrespectful to the actual drivers of the project. So I am happy to describe my experience, but I want to be clear before we start that I don’t have delusions of grandeur here. I was fortunate to be involved in a cool thing driven mostly by Dr. Kazuo Mori (who goes by Kaz most of the time).

The original paper was by Dr. Maryanne Garry and colleagues, published in 2008. It’s based on the MORI technique, developed by Dr. Kaz Mori in Japan and first tested on a Western sample by Dr. Garry and her colleagues in New Zealand. Dr. Mori developed a method in which two participants sit side by side and watch a video on a projection screen. Under some cover story, they both wear sunglasses though they are in a dark room. The sunglasses are polarized, and there are secretly two differently polarized projectors sending different images onto the same screen. Participants only see the image projected by the projector polarized in the same direction as their glasses. The brilliance of the method is that participants have almost no reason to believe that they watched different videos from each other. After the video(s), participants discuss several questions about details from the video together and finish with an individual multiple choice test of those and other details. In the end, those who discussed differing details with their partner are less likely to correctly report the detail from their own video.

MORI, by the way, is an acronym he coined in a 2007 paper. Stands for Manipulation of Overlapping Rivalrous Images.

rabbitsnore
That procedure is amazingly cool.

Mario
After the original project in Japan, Kaz got funding to send machinery around the world for people to replicate it. I would venture that this replication project is the culmination of that effort, which must have been about a decade ago by now. My lab-mate in Victoria, Dr. Tanjeem Azad, did a MORI procedure for her Masters thesis.

rabbitsnore
Does that sort of effect map onto a phenomenon we might see in the real world? On its face, it seems pretty far removed from everyday life.

Mario
Ah, good question. Yes, the idea is that people might sometimes witness events/crimes from different angles and thus see or be unable to see different details. It is also possible for two people to see an event from the same angle and nonetheless encode different details, resulting in different memories of the event. The critical question in either of these instances is: If one of the people remembers correctly and the other incorrectly, is the correct person able to maintain their memory after conversation with the incorrect person, or will their memory be infected with misleading information?

Will Crozier 🐙
Sounds like a really good method to see how two eyewitnesses, with different memory of the same event, can affect each other - without arousing suspicion as to what you're doing

Mario
So the procedure can seem artificial, but it is intended (by my understanding) to be a way to create a situation in which two witnesses have different memories. If we just showed pairs of people the same videos until they remembered different details, we'd be in the lab forever.
Of course, in 2018, not everyone is fooled. Even if they don't immediately figure out the polarization thing, there are suspicious participants. Typically those who are close enough to correct in debriefing are removed from the dataset before analysis.

rabbitsnore
Even with those practical challenges, it seems like a clever way to model a common (and problematic) eyewitness situation in the lab.

Mario
Agreed

Will Crozier 🐙
Also sounds like a really fun study to run.

Mario
Yes! Much of the data collection was done by my superstar research assistant, Nicole Laird, who finished her degree this spring. Woo!

Will Crozier 🐙
As psychologists, we often hear that we need to run "new" studies - essentially, "that's been done before, do something new." You ran a study that was nearly identical to the original paper, right? What's the point of doing that - what are you hoping to accomplish?

Mario
For sure, and novelty is always a helpful thing in a journal submission. In the past few years, though, news of the so-called “replication crisis” has gotten more and more attention in scientific and popular news sources. As psychologists have attempted to replicate or extend major findings, we have found that certain effects can be difficult to reproduce despite fame or influence of the original article. Additionally, a replication that generates a null finding can be very difficult to publish. Some journals are designing workarounds for this problem, which are called Registered Replications. For an RR, authors submit the plan and literature review to the journal before they even collect data. The editors offer some comments and, if they see fit, will offer to accept the paper whatever the results are. Then the data are collected and the paper is written, and comments after the fact are only regarding the presentation of the information, not the utility of the dataset in the literature.

And so, the point of this replication I am involved with is to find out if the effect found by Mori originally and Garry et al. in 2008 would replicate at all and worldwide. We had the additional hypothesis that collectivist countries might show a larger effect than individualists.

rabbitsnore
Did you find what you were looking for?

That question sounds a bit more existential than I intended it to be. Heh.

More broadly, what did you find?

Mario
[Pastes link to U2 video]

One of the advantages of work like this is that the motivation to come up with a particular finding is considerably weakened. As long as we gathered enough evidence to make a strong statement on the presence *or absence* of the effect, we will have found what we were looking for. Our data show a small effect across all the countries, and we ended up not quite having enough data to make a strong statement on the collectivist/individualist hypothesis.
But it seems from the data we do have that university undergraduates in collectivist countries are not much more likely, if at all, than students in individualist countries to change their answers on a memory test due to misinformation from a partner who appears to have watched the same video.

Of course, drawing a hard line between these two country types is difficult.

rabbitsnore
The basic effect from the original study replicated, though? Although it was a smaller effect than originally found.

Mario
Yes, a smaller effect size than that of Garry et al. (2008)

rabbitsnore
Even if it's smaller, I've got to say that I'm heartened that it replicated. In a world in which well-known effects have been failing to replicate, it's a bit reassuring when there's a successful replication. Of course, it would be informative and important to know if it didn't replicate. But there's a part of me that's relieved when things actually work out more or less the way we thought.

Mario
Yes, I think the answer is almost always that The Effect (general term) is a bit more complicated than we would like to think. If papers presenting new effects focused boundary conditions rather than presenting the largest effect size possible, we would have a clearer idea of what was credible. In this vein, I like Dr. Simine Vazire's renaming of the replication crisis to the Credibility Revolution.

rabbitsnore
Yeah, I like that too. Nelson, Simmons, and Simonsohn have called it "psychology's renaissance" which I like as well. It's a more positive -- and I think productive -- framing of the situation.

Will Crozier 🐙
What was the motivation to do this paper? Anything specific?

Mario
I think that the specific motivation to try to replicate Garry et al. comes from Kaz. I don't want to speak for him, but I think he sent all these MORI setups around the world with this exact collaboration in mind. "I've come up with this cool thing, it works for me, let's see if it works for more people." If it does, then maybe we could expand the method to ask other interesting questions about the situations in which someone might be more or less-influenced by a co-witness. Some of this research is already out there, and full confidence in the procedure allows that to continue to expand.

rabbitsnore
That seems like exactly the kind of mentality a scientist ought to have about replication.

Will Crozier 🐙
Yeah, sometimes it's easy for us to think, "Oh, I discovered this thing; it's going to be really important!" Being able to take a step back and say, "Let's make sure it works in other situations," is really admirable, I think

Mario
He's also a really nice guy!

rabbitsnore
It also seems like a lot of people are nervous about whether their findings will be replicate, rather than being more purely motivated to learn about the robustness of the results and the boundary conditions.

Mario
Understandably so, when our livelihoods can feel dependent on having hot findings. One of the nice things about the Credibility Revolution is that young scientists are earning respect by doing things like replications or publishing with open data and code, rather than having a fancy new effect.

rabbitsnore
I totally agree.

One of the first things you said was that you were a cog in a much bigger machine for this project. Who was involved? And what was it like working with so many collaborators? I imagine it could be pretty different from working just with your local lab.

Mario
The author list on the manuscript is extensive: Hiroshi Ito, Krystian Barzykowski, Magdalena Grzesik, Sami Gülgöz, Ceren Gürdere, Steve M. J. Janssen, Jessie Khor, Harriet Rowthorn, Kimberley A. Wade, Karlos Luna, Pedro B. Albuquerque, Devvarta Kumar, Arman Deep Singh, William Cecconello, Sara Cadavid Espinha, Nicole Laird, Mario J. Baldassari, D. Stephen Lindsay, & Kazuo Mori.

I’m sure that most of the work came from the leading authors and Kaz himself, as they did the organizing and planning. When we came on board, there was an OSF project from which we downloaded the materials. We then all discussed the particulars of the procedure in an email chain and went off to collect data. It was nice to have methodological details pre-determined. When we ran into a question that would have previously necessitated careful consideration, we just referenced the plan or outsourced the question to Kaz.

Once all the data were in, the whole group divided the work of initial draft writing and data analysis. It was the first time I got to hear what my data said without doing the analyses myself, which was nice. Then drafts started going around. I did one major edit myself, and it was definitely clear that several voices had been combined. I tried to pare down the verbiage to keep things simple. At times, the editing process felt a little like peer review, as some folks disagreed on what conclusions we could legitimately reach from the data. I think that resulted in a better product than any one lab could have produced.

Will Crozier 🐙
That sounds a bit different than how we normally do research in small teams of 2-4.

Did you ever have any 20-person skype meetings? or were issues mostly resolvable with emails and track changes?

Mario
Yes, at UVic the deal was usually that Steve and I would conceive a project and put a team of 1-5 RA's on data collection, then I would synthesize and analyze the data and the two of us would write it up. A much smaller setup.

I wasn't aware of any major meetings, it was all email and tracked changes in Word. I wouldn't be surprised if some folks spoke on the phone, but we are so spread around the world that any major meetings would have been basically impossible without excluding someone (or waking them up at 2am).

rabbitsnore
Given that this was different from the typical "cozier" setup of working with a small group of people, did you learn anything from the process of working on this project?

Mario
I think the project opened up new questions for me. I'm curious about pre-prints, as the editing process had that peer-review feel. I wonder whether receiving more detailed, outside feedback prior to peer review might strengthen the chance of publishing a manuscript. But otherwise, large scale collaboration is definitely not simple and I would say is a bit of an army mentality. You do as ordered and the boss is the boss, otherwise we won’t get where we’re going. I wonder how difficult it would be to get one going. I might try one day. The positivity coming through in everyone's emails was very nice to see. Reminders that we have a worldwide and mostly very excellent community are always good to have.

rabbitsnore
That makes sense. Intuitively, a project of this scale would demand a great deal of organization and teamwork to get off the ground.

This chat's not about my work, but I had my first preprint experience recently, and I quite liked it.

Will Crozier 🐙
I'm considering putting up a pre-print myself

Mario
The part I don't understand is how you get people to look at your preprint. You put it up and advertise it and... hope someone sees? Or did you send to people you'd like to read it?

rabbitsnore
A bit of both, in my case. I sent it to the labs I work with (here at Gothenburg and back at John Jay), and I tweeted about it. It's not as if I'm a big deal on Twitter (or anywhere else for that matter), so it isn't as if a ton of people read it and sent me comments -- but it was far more feedback than I would normally get on a paper prior to submission. The various feedback gave me a really good sense of how to improve the paper and what parts of the paper were working well. And the feedback took a lot of different shapes.

Sometimes people would send me emails with loads of commentary -- like a peer review you'd get when submitting to a journal. Sometimes people would tweet brief comments to me.

Mario
Another example of the awesome community we have.

rabbitsnore
Indeed! One of the nice aspects about it was that it was clear that the people sending me comments wanted the paper to be better. Sometimes peer reviews can feel sort of... How do I put this nicely? It doesn't always feel as if reviewers want you to perform well. But this was a totally different experience -- entirely positive and helpful.

Mario
That sounds great. My only thought now is "Why not?"

rabbitsnore
There are questions about whether preprints jeopardize the integrity of the peer review process by making it clear who the author is -- but I think most of the time that's not going to be too much of a problem.

Mario
I am in the review process on a paper for which it is obviously us because we cite the paper upon which we're building, and it actually made the feedback better.

Will Crozier 🐙
I suspect people can usually piece together who the primary authors are on a paper anyways
may as well be up front about it? I think there's some room for empirical work on biases in publishing

rabbitsnore
Sometimes perhaps anonymity is useful, but I suspect those cases are rare. It's generally been my experience that openness enhances the process.

Mario
An interesting empirical question, no doubt.

rabbitsnore
Spoken like a proper scientist. Heh.

Mario
A lab coat spontaneously appeared on me as I hit "Send."

rabbitsnore
Getting back to the replication -- after having done this once, do you think you'd do a project like this again?

Mario
Absolutely! These large scale, international collaborations make claims that few other papers are equipped to make. And making connections through such a project is a great start at further collaborating down the road. I wouldn’t hesitate to ask any of my co-authors about working together again.

Will Crozier 🐙
What would you think if someone did a replication of one of your studies?

Like, looking at it from the other point of view - someone says, "Hey, this is interesting. I wonder if I can replicate it." Would you want to be involved? Notified before they did it?

Mario
Would be an honor! It's exciting to imagine coming up with something important enough that somebody else would want to try to do it, too.

It seems like the best way to run one of these big replication projects is to involve the original authors, so that they can address little, unpredictable methodological questions in the same way they did the first time. It's a shame when a big replication comes out and the original authors dismiss it because of some small inconsistency.

Will Crozier 🐙
Yeah, there seems to be a bit of that happening now…

But could involving the original authors inject bias in some way though? How much challenging/questioning did Kaz do?

Mario
Kaz was specific with regard to procedural detail, but he gave up main analyses to other folks (Steve Janssen did lots, @jidenteki_kioku on Twitter)

rabbitsnore
Seems to me that it's the procedural detail that might be the most important for the original authors to comment on. That's where they're likely to have specific insight.

Overall, my impression is that this was an awesome project -- not to mention a terrific research experience. Mario, do you have any last comments before we wrap?

Mario
Yes, I think this was the ideal scenario for a large, international collaboration to replicate a well-known finding. All the people are great, and I look forward to seeing the paper come out. Two last notes: The aforementioned Dr. Steve Janssen posted a great tweet thread about the project, and the Open Science Framework project is public at https://osf.io/j5f82/ for those interested in learning more details.

Will Crozier 🐙
I'm looking forward to reading it! Thanks for stopping by!

Mario
Thanks for having me. It's nice to chat research during a quiet summer between adjunct gigs. Hope to see you both in Cape Cod next summer!

References

Garry, M., French, L., Kinzett, T., & Mori, K. (2008). Eyewitness memory following discussion: Using the MORI technique with a western sample. Applied Cognitive Psychology, 22, 431–439. doi:10.1002/acp.1376

Nelson, L. D., Simmons, J., & Simonsohn, U. (2018). Psychology's Renaissance. Annual Review of Psychology69, 511-534

Comments

  1. I have a fancy for your posts. They are so great and useful. You deserve a pat on the back for your sharing. I wish you can update articles often. I hope I could read them everyday. Your posts offer advice to me. Thanks very much! facebook chat box, facebook inbox, https://autoketing.com

    ReplyDelete
  2. I think that I'm into your pieces of writing. I wait for your posts every week. I learn many new interesting things from the articles. The information is essential for me. I want to thank you for sharing. juegos gogy
    , motox3m3 games , games abcya3 for kids

    ReplyDelete
  3. These articles are exactly what I need. It is very nice of you to share your understanding. I have learned interesting things. I have a liking for your posts. Please, upload more and more posts. geometry dash , superhexio , pool 8 ball

    ReplyDelete
  4. These articles are exactly what I need. It is very nice of you to share your understanding. I have learned interesting things. I have a liking for your posts. Please, upload more and more posts. I want to know more about this related topics. Thank you for the wonderful sharing. They are useful pieces of advice.currency converter shopify app , shopify free discount app , Product discount master , 15 best apps on shopify

    ReplyDelete
  5. food games
    basketball free game
    soccer free online
    These articles are exactly what I need. It is very nice of you to share your understanding. I have learned interesting things.

    ReplyDelete
  6. You have make some good information post,its amazing article provide such an help full information provide i have share with my friends keep doing great job
    miniclip com 2019, a10 kids, Jogos 2019

    ReplyDelete
  7. Playing the free online gambling club amusements not simply enable you to build up the aptitudes or enhance the same yet additionally enable the students to take in the guidelines and controls of the diversions and inspire them to have a superior knowledge of the distinctive procedures and the methodologies that are to be utilized while playing the online clubhouse.
    2player best free online games
    friv Games to play
    io jogos for school

    ReplyDelete
  8. Having lights set for you is also a great way to establish a working relationship with me as a magical practitioner. It involves very little financial investment, and you can get a good idea of how I manage my cases, get a feel for my working style, and also get a good taste of my skills as a prophet, seer, and diviner.
    These articles are exactly what I need. It is very nice of you to share your understanding. I have learned interesting things. I have a liking for your posts. Please, upload more and more posts. I want to know more about this related topics. Thank you for the wonderful sharing. They are useful pieces of advice.currency converter shopify app ,
    games free 2019
    friv jogos 2019
    jogos 360 online game

    ReplyDelete

  9. Pokemon Magikarp Jump Online
    Championship 2016
    Hyper Vault
    I have a mission that I’m just now working on, and I have been at the look out for such information

    ReplyDelete
  10. This is a great source of information, I will keep track of your posts and share them with everyone. I admire the person who wrote this post, you are so talented, hope you will promote them and become more successful.
    pbs Games for girls, jogos friv 2019, kizi unblocked

    ReplyDelete
  11. The sedentary lifestyle has made way for a lot of diseases. personal trainer near me

    ReplyDelete

Post a Comment

Popular posts from this blog

You have the right to remain silent....unless you are free to leave

Guest post by Fabiana Alceste.

“You have the right to remain silent. Anything you say can and will be used against you in a court of law. You have the right to an attorney. If you cannot afford an attorney, one will be provided for you.” These words have likely been burned into our collective consciousness by the ubiquitous and fantastic TV series Law & Order (in my case, Law & Order: SVU). These words are often spoken as Olivia Benson and Elliot Stabler handcuff the main suspect of the episode—the last words trail off as they walk offscreen and the show cuts to a commercial break. We know this scene, and we know the words…but why do these warnings exist and for whom are they intended?
According to the U.S. Supreme Court, custodial interrogations are sometimes characterized by the use of police intimidation, trickery or other psychological tactics, a restriction of one’s personal liberty, and a feeling of compulsion to speak against one’s will1. Recognizing this, in 1966’s infam…

We find the defendant....

If you’ve ever heard the jury foreman give a jury’s decision, you’ll notice that they never say the defendant is “innocent.” Instead, they’ll conclude that the defendant is “not guilty.” Although the terms may sound interchangeable, “not guilty” and “innocent” actually mean two different things. “Not guilty” is a legal conclusion, whereas “innocent” means the person didn’t commit the crime. For example, think of O.J. Simpson. In a court of law, he was found “not guilty” of killing his ex-wife – but if you ask the majority of Americans, they’ll tell you he’s not innocent. Here, we’re going to talk a bit about the distinction between the two conclusions – and why it matters to psychology and law researchers*.

When we say that “not guilty” is a legal decision, we mean that it’s a decision based on criteria described by laws – namely in the U.S., if the prosecutor can convince the jury of the defendant’s guilt. This is based partially on the amount of evidence that points toward the defend…