Talk:WikiJournal of Medicine/Does the packaging of health information affect the assessment of its reliability? A randomized controlled trial protocol

From Wikiversity
Jump to navigation Jump to search

WikiJournal of Medicine
Open access • Publication charge free • Public peer review • Wikipedia-integrated

WikiJournal of Medicine is an open-access, free-to-publish, Wikipedia-integrated academic journal for Medical and Biomedical topics. <seo title=" WJM, WikiJMed, Wiki.J.Med., WikiJMed, Wikiversity Journal of Medicine, WikiJournal Medicine, Wikipedia Medicine, Wikipedia medical journal, WikiMed, Wikimedicine, Wikimedical, Medicine, Biomedicine, Free to publish, Open access, Open-access, Non-profit, online journal, Public peer review "/>

<meta name='citation_doi' value='10.15347/WJM/2021.001'>

Article information

Authors: Leela Raj[a][i], Denise A. Smith[a][ii] , James Heilman[b]

See author information ▼
  1. 1.0 1.1 McMaster University
  2. University of British Columbia
  1. rajl2@mcmaster.ca
  2. dsmith@mcmaster.ca

 

Plagiarism check

Pass. The WMF's plagiarism check tool points out that 'violation is unlikely' with 13.0% confidence. Please see the results here [1]. This does not require any action. Thanks Rwatson1955 (discusscontribs) 16:25, 29 October 2020 (UTC)[reply]

First peer review


Review by David R Thompson , Queen's University Belfast
These assessment comments were submitted on , and refer to this previous version of the article

This is a clearly-written, well-designed RCT protocol addressing a novel topic - whether the packaging of health information affects the assessment of its reliability. I commend the authors for designing a protocol to examine an issue that is assuming increasing importance among academics, health professionals, students and the public. The proposed study is a factorial double-blind RCT of an original Wikipedia health information article versus a BMJ literature review using a modified 10-item version (rather than the original 16-item version) of the DISCERN tool. Participants will be 336 faculty (physicians, medical residents and medical residents) from 4 Canadian university medical schools who will be randomized to one of 4 study arms. The primary outcome will be whether a statistically significant difference in the DISCERN scores exists, which could indicate how health information is packaged influences its quality assessment. Secondary outcomes will be whether participants recognized one or both of the articles and whether the order of reading them affected their grading.

Comments
Major

as the sample will include a variety of medical faculty (physicians, residents, students) it is unclear whether years of experience (and the discipline) will be taken into account in the analysis of outcomes. One might predict that senior physicians, those in practice and research or those with an MD + PhD would perform better than undergraduate students or newly graduated ones. I see in Appendix B that the pre-participation survey asks for such information, but it is unclear to me how this will be taken into account when assessing the 4 groups.

Minor

under Design in Arm 3 'as' is missing - it should read '...Wikipedia article formatted as a...'. -

Response

3 November 2020‎

Dear David, Thank you for taking the time to review our submission and provide helpful feedback. Below you will find our responses to your two comments:

Minor

This typo has been resolved. Thank you for your keen eye.

Major

You're absolutely right to point out that it is unclear whether years of experience and/or discipline will be taken into account in the analysis of outcomes. This was certainly the intention behind the collection of information about participants' experience (level of practice) in the pre-participation survey. We have made the following revisions to the protocol:

  • Added an additional secondary outcome to measure as noted below in italics:

Secondary Outcome Measures

  1. The academic backgrounds and expertise of our participants may result in previous knowledge or familiarity with articles used in this study. Chances of this occurrence can not be eliminated and must be considered in analysis of the study data. Therefore, our secondary outcome will address the potential unblinding of participants by using a short questionnaire (Appendix B) to determine whether subjects recognized one or both of the articles from previous readings. We will also determine whether the order of reading of the modified articles impacted grading by participants.T
  2. The academic backgrounds and expertise of our participants could potentially influence the scores recorded with the DISCERN tool. Chances of this cannot be eliminated. We will analyze the study data to determine whether there is a relationship between participants' expertise (level of practice) and the score they assign to their articles with the DISCERN tool.

Secondary assessment

  • Study subject questionnaire (see: Appendix C): Descriptive statistics using SPSS
    • Does the order in which articles are assess affect the outcome?
  • Study subject questionnaire (see: Appendix B): Inferential statistics using SPSS
    • Is there a relationship between the participants level of practice and the score they assign to each article using the DISCERN tool?

Second peer review


Review by anonymous peer reviewer , a track record of over 10 papers about Wikipedia in prime outlets, including medical ones
These assessment comments were submitted on , and refer to this previous version of the article

This is a clearly written, important paper with an important and well-argued goal.

My main suggestion would be to explore a little bit more the relations between Wikipedia and Academia in the literature review section. There is a clear gap between the perceived value and the actual value, and it is worth studying a bit more. Additionally, given the prominence of Wikipedia as a source of medical information for lay population, it would be useful to comment on the relations between medical journals' coverage in Wikipedia and their professional standing.

Response

3 November 2020‎

Dear reviewer 2, Thanks very much for the time and effort you have devoted to reviewing our protocol and providing your feedback. Our response to your suggestion to explore the relations between Wikipedia and Academia in the literature review is below:

I am happy to engage with you in further discussion, but based on the purpose of this study, my co-authors and I don’t see the benefit in reviewing the literature in the context of the relationship between academic and Wikipedia. While this is certainly an important conversation, and one I am personally quite interested in, the RCT planned here will not address the relationship between Wikipedia and academia. The focus of this study is to measure whether bias can influence the results from use of the DISCERN tool. Here, Wikipedia is a medium. We talk about it in the introduction, but the focus of this study isn’t Wikipedia, it’s DISCERN. I hope this response provides you with satisfactory clarity with respect to why the conversation about academic and Wikipedia was not included in our review of the literature.

Peer review 3


Review by Graeme Drummond Smith , Caritas Institute of Higher Education
These assessment comments were submitted on , and refer to this previous version of the article

In general, this protocol paper is well written. It appears accurate, balanced and it certainly is accessible for the reader. Clear rationale is provided for the study. However, I could not find a clearly stated research question/hypothesis making it rather difficult to determine whether a single double blind randomized controlled trial is appropriate. There are several other (minor) points that I would like to make: It is not fully clear what is meant by 'modification' to DISCERN. I believe this could be clearer. Perhaps a little bit more could be said about what exactly is involved in the 'one supervised session'. The protocol could be enhanced with additional information about the control of order of the articles. How will consent be obtained? I am not sure about the relevance of the statement about 'number of clinical faculty not being being included'. What exactly will be contained in the the participants packs In Appendix B, should there be an 'other' option? Otherwise, material in the protocol paper seems accurate. I have followed the general review guidance for this paper, as I could not find specific guidance for protocol papers.

Response

2 March 2021

Dear reviewer 3, Thanks very much for the time and effort you have devoted to reviewing our protocol and providing your valuable feedback. Please find a summary of your comments, and our responses, below:

Could not find a clearly stated research question/hypothesis making it rather difficult to determine whether a single double blind randomized controlled trial is appropriate
  • Added to introduction
Not fully clear what is meant by 'modification' to DISCERN
  • We have added text to the introduction to indicate how the DISCERN instrument was modified and why
What exactly is involved in the 'one supervised session'
  • Added to Methods and Design - Design: The session will be supervised by one of the co-investigators who will ensure that participants do not have access to any outside materials while completing the study intervention.
Additional information about the control of order of the articles
  • Added to Methods and Design - Design: we indicated which arms will be used for the control/experiment and elaborated on why the order in which the articles must be read and evaluated allows us to control for the possibility of a sequence effect.
How will consent be obtained
  • Added to participants and recruitment section: Individuals who wish to take part in the study will be required to fill and sign a consent form prior to attending the supervised session. Consent forms will be stored by the co-investigators. Participants will be able to withdraw their consent at any time prior to data analysis.
Relevance of the statement about 'number of clinical faculty not being being included'
  • This is not relevant. Sentence removed.
What exactly will be contained in the the participants packs In Appendix B, should there be an 'other' option?
  • Provided more detail in the Intervention section re: what is in the packets.
  • Added "Other (please specify):" to pre-participation questionnaire

Thank you again for taking the time to review our protocol.

Peer review 4


Review by Richard Gray , La Trobe University
These assessment comments were submitted on , and refer to this previous version of the article

I did struggle to work out the aim of the trial, I think this can be more clearly expressed. As far as I can determine the study aim is to test if readers judgement about the quality of health information is biased by the source of that information. In this trial the hypothesis the authors are seeking to test is that participants rate the same information more highly if it is presented as being published in the BMJ rather than Wikipedia. The question is relevant and will be of interest to a broad clinical audience. I did wonder if there was a sub experiment that the authors should carry out which is asking some participants to just read the text (not formatted as BMJ or Wikipedia) and ask them to guess the source (ie, does the writing style give away where it was published). I was a little confused about the assumptions of the sample size calculation and the randomization process. It seemed to me that the authors are proposing a four arm trial but in the randomization section of the paper it appears that the authors are conducting a two arm study. Greater clarity, I think, is required. By using the randbetween function in excel to generate two equal group sizes I think there is risk that researchers will be able to work out group allocation. This paper by Schulz and Grimes explains the issue. Finally, it was not clear to me how or why the DISCERN measure was amended; some further explanation from the authors would be appreciated.

Response

2 March 2021

Dear reviewer 4, Thanks very much for the time and effort you have devoted to reviewing our protocol and providing your valuable feedback. Please find a summary of your comments, and our responses, below:

Aim of the trial can be more clearly expressed
  • This has been updated, based on Reviewer #3's comments. Have added to introduction.
Sub experiment that the authors should carry out which is asking some participants to just read the text (not formatted as BMJ or Wikipedia) and ask them to guess the source (ie, does the writing style give away where it was published)?
  • The authors understand what Reviewer #4 is trying to suggest. The authors have decided that the suggestion is unfortunately out of the scope of our study. While it could certainly be an informative experiment, there are many other ways we could examine our research question and the current protocol indicates our selection of one way.
Authors are proposing a four arm trial but in the randomization section of the paper it appears that the authors are conducting a two arm study
  • Thank your for noticing this! This was a byproduct of missing something as we worked through initial study designed. Corrected.
By using the randbetween function in excel to generate two equal group sizes I think there is risk that researchers will be able to work out group allocation
  • Thanks for this observation. We have addressed your concerns re: group allocation by updating the section on Randomization
Unclear how or why the DISCERN measure was amended
  • We have added text to the introduction to indicate how the DISCERN instrument was modified and why

Thank you again for taking the time to review our protocol.

Editorial comment


Comments by Roger Watson ,
These editorial comments were submitted on , and refer to this previous version of the article

Before I can finally accept this could I ask you to indicate: 'Author contributions'; 'Funding'; and 'Acknowledgements'. Rwatson1955 (discusscontribs) 14:51, 12 March 2021 (UTC)[reply]

@Rwatson1955:: The document has been updated as per your request. Thank you. Mcbrarian (discusscontribs) 18:53, 18 March 2021 (UTC)[reply]

Hi and thanks but I don't understand the comment: "Articles used for the purpose of this study were selected in order to ensure minimal involvement of study authors." which is under Conflicts of Interest. I was hoping to see a separate sub-heading on 'Author contributions' which ascribed: Design of study; Preparation and revision of manuscript; Approval of submitted manuscript, and you put author initials next to each of these (NB: all initials must go next to the 3rd point. Thanks, Roger Rwatson1955 (discusscontribs) 13:48, 19 March 2021 (UTC)[reply]
Hi @Rwatson1955,
I have added the Author contributions sub-heading as per your request. I also revised the statement "Articles used for the purpose of this study were selected in order to ensure minimal involvement of study authors" as I see now that it is vague and unclear. The statement has been update to be specific, direct and clear. Let me know if this helps. Mcbrarian (discusscontribs) 16:04, 23 March 2021 (UTC)[reply]