this post was submitted on 11 Jun 2023
1385 points (100.0% liked)
Technology
37737 readers
585 users here now
A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.
Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.
Subcommunities on Beehaw:
This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
No.
Strictly speaking, COPPA prevents Reddit from collecting information from users under the age of 13. While there are no explicit guarantees that a person on the site isn't 13 or older, and also recognizing that age of majority is typically 18, then in a general literal sense yes there are minors involved, in so far as the activity discussed is research.
A research intervention is an intervention in so far as it intersects with the participant. A drug trial that lasts for 10 weeks total but only gives doses to participants for 2 weeks (and then results monitoring during, 2-weeks post, and 4-weeks post) does not mean that the "intervention" lasts 10 weeks.
In most practical terms, running an experiment for 2-3 weeks is very common to collect sufficient data. However, the intervention itself may be quite brief (for example, a short 45-minute interview with a participant would be the "intervention" for a 2-3 week long study interviewing physicians on their concerns about organizational capacity for change).
For a repeated measures experiment, the intervention usually involves the actual experiment encounter and maybe some additional time between them.
For the case of A/B testing, usually the "intervention" in this case is the A/B test as it applies to the user at the moment, and not the entire duration of when the testing is taking place for all participants.
Once again, to iterate, you are equating the inability to view the content of a website without logging into an account with such substantial emotional and psychological harm that is comparable to being verbally derided in public, for a week, shamed on a public channel, and/or comparable situations. You are not going to convince an IRB that being able to view the content of a website without logging in, then subsequently going to a different community to ask for help, and then hypothetically ruminate about the matter for weeks, is going to exceed the kind of everyday ordinary harm to qualify a risk level above minimal risk.
Deception goes beyond simply "lying" to or "not informing" the participant. Duke University gives some good considerations here:
Additionally, a waiver for utilizing deception in research has to:
Satisfying #1 and #2 in an UI/UX A/B testing regime that Reddit used here is pretty easy. You are specifically hung up on the implicit harms involved but in reality they are of no particular serious concern.
#3 is particularly interesting, because effectively what that means is you need to demonstrate deception is necessary in experimental design for the experiment to actually work. If you are seeing whether a person will interact more with a site or not if they are blocked from seeing content without logging into an account, telling them ahead of time could already bias the outcome. This is in very specific consideration that #1 and #2 are already met: being unable to view a website without logging into an account is not anything more than minimal risk. And even then, it is important to emphasize that the failure to disclose the research hypothesis to counter the demand effect is NOT deception.
The kind of UI/UX A/B testing Reddit employed in this specific instance is absolutely not equivocal to the Asch conformity studies.
To be very clear here, when YOU operate under the belief that being unable to view the content of the website, then posting about it elsewhere and potentially being ridiculed for it is sufficient of a bar to meet beyond minimal risk, then we have very different definitions of what "minimal risk" entails. Since we cannot come to consensus on this particular topic, and instead you've gone so far as to associate this kind of activity's harm of equivalent to the Asch conformity studies is frankly ludicrous.
If we cannot agree on this, then so be it. However, I will repeat (with added finality): YOU running the same A/B experiment Reddit is doing on an university-sanctioned website will NOT get you run out of the university by the IRB. In a real-world scenario. you would likely discuss this experiment with an administrator or similar at your department, then maybe send an e-mail off to the IRB for any clarification, would likely have some back-and-forth, and then would ultimately receive a determination that it is exempt (at worst), or not considered human subjects research at all. I can see a few circumstances where such an effort might merit an expedited review, but to do this would involve some torturous twisting of the situation that could be easily avoided (for example, running this testing on a page that has instructions for performing the Heimlich maneuver).
The Asch conformity study experiments are absolutely not equivalent to doing A/B testing on a wide scale with regards to how users interact with a website when presented with a pop-up preventing further interaction without logging in.
Edit: I would add that, with regards to "harm", IRBs usually don't contend themselves excessively with imaginative processes leading to catastrophic harm. Some kinds of harm are obvious (for example, disclosing that someone has a degenerative life disease discovered incidentally during clinical sequencing), and others are very much in the ballpark of "it could happen, but if you make reasonable accommodation it's not a big deal", such as being concerned that someone might risk choking to death while being a participant on a blind taste test of different beverages using tiny samples. Allergies are real and participants should be informed about these risks ahead of time; potentially choking on a small sip of Coke is good to note but is not going to register on even the participant's radar.
Thank you for agreeing that Asch conformity experiment falls under human experiment ethics considerations and informed consent requirements! I am surprised though that you consider the reddit experiment unlike the Asch experiment (I personally see reddit as the worse one actually!). Could you explain how in your mind, Asch does carry a risk of causing harm in a way that reddit does not? Asch is just looking at a bunch of lines on a page after all! How can a bunch of lines cause harm? I also find it odd your cavalier attitude towards the word "should".
If "should" isn't prescriptive, why even have any "should" statements in our guidelines if you are just going to ignore them all? And yet again your link disagrees with you:
Reddit never had any intention to ameliorate the deception, to debrief the participants, or to give them an opportunity to delete their experiment records after the fact. Reddit never implemented alternative practical research methods like opt-in studies. Are you seriously arguing that because the page says "should" and not "must", reddit was perfectly ethical simply not doing any of this at all? This isn't some RFC, this is normal people language!
Or if you are saying this wasn't deception, then why link to the entire Duke deception page at all? The only relevant sentence here to you is:
And it only refers to the disclosing the research hypothesis itself, not the very fact that you are taking part in some experiment! And you agree that it is not impossible to perform usage experiments without informing participants in advance (I brought up Firefox as a better example alternative), it is just more laborious. Moreover, reddit did engage in actual deception beyond simply keeping the fact of the experiment secret:
What happened went beyond that. If you read the reddit OP:
This isn't some simple A/B testing of things like text size or link color. This isn't like Facebook or Instagram blocking everyone equally from seeing communities without logging in. OP was logged in. Reddit lied to them saying they were not logged in when they were. Reddit lied to them saying there is no way to log in. Reddit lied to them saying the only way to see the content was to download the app. This is the deception part. This is the part that's similar to Asch and the people in the room with you lying that they are participants like you. You think you are in a normal situation but you are not. You've been singled out and no one believes you.