WikiJournal Preprints/Does the packaging of health information affect the assessment of its reliability? A randomized controlled trial protocol

From Wikiversity
Jump to navigation Jump to search

WikiJournal Preprints logo.svg

WikiJournal Preprints
Open access • Publication charge free • Public peer review


WikiJournal User Group is a publishing group of open-access, free-to-publish, Wikipedia-integrated academic journals. <seo title=" Wikiversity Journal User Group, WikiJournal Free to publish, Open access, Open-access, Non-profit, online journal, Public peer review "/>

<meta name='citation_doi' value=>

Article information

Authors: Leela Raj[a][i], Denise Smith[a]ORCID iD.svg , James Heilman[b]ORCID iD.svg 

Leela Raj; Denise Smith; James Heilman, "Does the packaging of health information affect the assessment of its reliability? A randomized controlled trial protocol", WikiJournal preprints, Wikidata Q100657786




Abstract

Background

Wikipedia is frequently used as a source of health information. However, the quality of its content varies widely across articles. The DISCERN tool is a brief questionnaire developed in 1996 by the Division of Public Health and Primary Health Care of the Institute of Health Sciences of the University of Oxford. They claim it provides users with a valid and reliable way of assessing the quality of written information. However, the DISCERN instrument’s reliability in measuring the quality of online health information, particularly whether or not its scores are affected by reader biases about specific publication sources, has not yet been explored.

Methods

This study is a double-blind randomized assessment of a Wikipedia article versus a BMJ literature review using a modified version of the DISCERN tool. Participants will include physicians and medical residents from four university campuses in Ontario and British Columbia and will be randomized into one of four study arms. Inferential statistics tests (paired T-test, multi-level ordinal regression, and one-way ANOVA) will be conducted with the data collected from the study.

Outcomes

The primary outcome of this study will be to determine whether a statistically significant difference in DISCERN scores exists, which could suggest whether or not how health information is packaged influences how it is assessed for quality.

Plain Language Summary

The internet, and in particular Wikipedia, is an important way for professionals, students and the public to obtain health information. For this reason, the DISCERN tool was developed in 1996 to help users assess the quality of the health information they find.

The ability of DISCERN to measure the quality of online health information has been supported with research, but the role of bias has not necessarily been accounted for. Does how the information is packaged influence how the information itself is evaluated? This study will compare the scores assigned to articles in their original format to the same articles in a modified format in order to determine whether the DISCERN tool is able to overcome bias.

A significant difference in ratings between original and inverted articles will suggest that the DISCERN tool lacks the ability to overcome bias related to how health information is packaged.

Background[edit | edit source]

Introduction[edit | edit source]

The internet is a crucial source of health information for health practitioners, students and the public. In the online information landscape, Wikipedia, a widely accessible and free encyclopedia, stands out as one of the most frequently consulted sources of online health information. Despite its high frequency of usage, the health content on Wikipedia varies widely in quality. Thus, it is important for all consumers of Wikipedia's health content to consider the quality of a specific article prior to applying its content.

This is specifically the aim of the DISCERN tool, which was first developed in 1996 by the Division of Public Health and Primary Health Care of the Institute of Health Sciences of the University of Oxford. DISCERN is a brief questionnaire that provides users with a valid and reliable way of assessing the quality of written information.[1] The original tool is comprised of fifteen questions that address targeted aspects of the article. The sixteenth question asks for the respondent's overall impression of the publication's quality. The DISCERN instrument has experienced success as a tool to measure the quality of online health information, as demonstrated by its application in 218 published studies including quality assessments of Wikipedia articles.

Nevertheless, the accuracy of DISCERN when it is used in a blinded study has not yet been evaluated. Therefore, it remains unknown whether the use of DISCERN to evaluate health information is affected by reader biases about specific publication sources. In light of this, this study will conduct a double-blind randomized assessment of a Wikipedia article versus a BMJ literature review using a modified version of the DISCERN tool. Participants, including physicians and medical residents, will be asked to evaluate and compare, either the quality of both articles in their original formats, or both articles in inverted formats. Differences between article scoring between both groups will allow the authors to determine DISCERN's ability to overcome bias related to the article's publication source.

Literature review[edit | edit source]

The quality of Wikipedia's health content has received the vast majority of the academic attention paid to Wikipedia in the context of its usage as a health information resource. The reports of Wikipedia's quality in the academic literature generally focus on Wikipedia's suitability for: patients or the general health consumer, students in health sciences, or professionals in the field of health and wellness.

To date, topics  included in the assessment of Wikipedia's content for patient education or consumer health include gastroenterology,[2] nephrology,[3] cancer,[4][5][6][7] autoimmune disorders,[8] medicinal drugs[9][10][11][12][13] or herbal supplements,[14] pathology informatics,[15] surgery,[16][17] toxicology,[18][19] nutrition,[20][21][22] complementary and alternative medicine,[23] hearing loss,[24] and mental health or the brain.[25][26] These assessments assess readability, reliability, and accuracy or completeness and specifically discuss their findings in relation to the public consumer or patient. Of those that include results – some conference proceedings do not – there is some agreement that Wikipedia is suitable for patients and a 2010 study finds that while Wikipedia is not necessarily the superior resource, it is the preferred resource.[5]

There is strong evidence in the literature that students enrolled in health and medicine programs are highly likely to use or have used Wikipedia to supplement their education. Herbert, et al (2015) present evidence that suggests most medical students use Wikipedia at a moderate or high rate (67%), but this investigation reports a response rate of 21% so the findings cannot be generalized.[27] Judd and Kennedy (2011) find that medical students used Google in 69% of biomedical sessions in a computer lab and Wikipedia in 51% of those same sessions.[28] While the study notes an interesting trajectory in which students' reliance on Wikipedia decreases each year from first year to third year, actual Wikipedia usage remains prominent throughout students' progression throughout the curriculum. At Queen's University[29] and USCF.[30] Wikipedia is used in formal education as a learning tool for evidence based medicine. Overall however, there is a lack of consensus in the literature about Wikipedia's suitability for health education. Some studies conclude Wikipedia is suitable for students[31][32] while many conclude it is not.[33][34][35][36][37]

A minority of evaluations of Wikipedia's health content consider its suitability for health care workers and the outcomes of these studies is also inconsistent. Park, Masupe, Joseph, et al (2016) report that Botswanan health care workers' perceptions of Wikipedia's quality is divisive at best. Further, participants in the Botswana study indicated Wikipedia's medical content as valuable simply because it is freely available and, through a now defunct relationship with telecommunications companies, remained accessible when internet access was lost.[38] However, the ability to access Wikipedia offline has been unavailable since 2018.[39] As a surgical reference Wikipedia is found to be accurate, albeit incomplete, and an appropriate resource.[40] Conversely, the drug information on Wikipedia is deemed variable in comparison to Micromedex and therefore considered inappropriate drug reference for professionals.[14]

Methods and design[edit | edit source]

Design[edit | edit source]

This is a factorial double-blind randomized controlled trial to determine if how an article is packaged affects the score it receives when the DISCERN tool is used to evaluate its reliability and quality. The study will involve four intervention arms:

  • Arm 1: will use DISCERN to evaluate an original BMJ article first and an original Wikipedia article second
  • Arm 2: will use DISCERN to evaluate an original Wikipedia article first and an original BMJ article second
  • Arm 3: will use DISCERN to evaluate a BMJ article formatted as a Wikipedia article first and a Wikipedia article formatted as a BMJ article second
  • Arm 4: will use DISCERN to evaluate a Wikipedia article formatted as a BMJ article first and a BMJ article formatted as a Wikipedia article second

Controlling the order in which the articles are read will allow the researchers to detect any potential of sequence effect impacting the study results. The study involves four Canadian medical schools including three in Ontario and one in British Columbia allowing for recruitment of medical faculty and students possessing the relevant backgrounds of knowledge and experience to complete the study intervention. Consenting participants will be asked to attend one supervised session (organized in their home institution) to complete the study intervention.

Settings[edit | edit source]

This study will be conducted on four university campuses in Ontario and British Columbia that include a medical school and that are also within a reasonable proximity of the researchers' home campuses to facilitate in-person administration of participants' packets. Such institutions include:

  1. Michael G. DeGroote School of Medicine: McMaster University (Hamilton, ON)
  2. Faculty of Medicine, University of Toronto (Toronto, ON)
  3. Schulich School of Medicine & Dentistry. Western University (London, ON)
  4. Faculty of Medicine, University of British Columbia (Vancouver, BC

Participants and recruitment[edit | edit source]

Participants will include faculty from the four medical institutions listed above. Participant recruitment will be done through a combination of a purposive approach, directly contacting individuals responding to inclusion criteria through e-mail or by telephone, and through study advertisement, using paper and electronic posters.

Sample size[edit | edit source]

The four medical schools included in this study report an approximate cumulative 4,770 full-and part-time faculty members. The number of clinical faculty was not included because they were not consistently reported by each institution.

Table 1. Number of faculty members reported by medical schools included in study
Michael G. DeGroote School of Medicine: McMaster University (Hamilton, ON) >700[41]
Department of Medicine: University of Toronto (Toronto, ON) 800[42]
Schulich School of Medicine & Dentistry: Western University (London, ON) 2,681[43]
Faculty of Medicine: University of British Columbia (Vancouver, BC) 589[44]
Total 4,770

In order to achieve the desired confidence level of 90% and a margin of error of 5%, the authors will randomly select 336 participants from the pool of faculty members recruited for the study. In the event that more than 336 participants were not recruited, the authors will use a convenience sampling method until at least 336 participants have been recruited.

The estimated sample size to produce statistically significant results was calculated using the following formula:

Sample Size = (Distribution of 50%) / ((Margin of Error 5% / Confidence Level Score 90%)2)

        = (0.5 x (1-0.5)) / ((0.05/1.9)2)

        = 0.25 / 0.0006925

        = 361.01

True Sample = (Sample Size x Population) / (Sample Size + Population – 1)

       = (361.01... x 4,770) / (361.01... + 4,770 - 1)

       = 1722021.6606… / 5130.0108

       = 335.6760

Randomization[edit | edit source]

Each recruited participant will be assigned a unique identification number. Each ID number will be recorded in an encrypted, password protected MS Excel spreadsheet. The participants' names and contact information and their corresponding participant ID will be kept in a separate, encrypted and password protected MS Excel spreadsheet. Once the recruitment phase is complete, the authors will use the RANDBETWEEN function in MSExcel to randomly select 336 participants. If more than 336 participants are not recruited, the authors will employ a convenience sampling method until 336 participants have been recruited.

Using the RANDBETWEEN function in MSExcel, half (168) of the participants will be randomly selected for Arm 1 and half will be randomly selected for Arm 2. All packets for each participant will be labeled with only their unique participant ID number.

Interventions[edit | edit source]

Eligible and consenting participants will be randomized into one of four arms as outlined in the study design.

Participants will be required to attend a 30 to 60 minute session supervised by study investigators (JH, DS or LR) during which they will be tasked with reading the article to which they have been allocated and subsequently completing the DISCERN questionnaire. All study materials including articles and questionnaires will be collected by investigators at the end of the time allocated to completion. Following collection of the article and DISCERN questionnaire, participants will be asked to complete a short additional questionnaire inquiring about their prior knowledge of the article.

Proposed outcome measures[edit | edit source]

Primary[edit | edit source]

The modified DISCERN instrument is composed of 10 questions covering the depth of content, scientific accuracy, completeness, justification or evidence given, and readability grade. Users respond to each question of the tool using a scale from 1 to 5 where 1 represents  serious or extensive shortcomings while a score of 3 signifies potentially important but not serious shortcomings and a score of 5 constitutes minimal shortcomings. Final grade of the article is determined through a composite score of the 10 questions. The primary outcome of our study is the difference in scores between the original articles when compared to each other, with the difference in scores of the modified articles when compared to each other. If the difference in scores between both original articles is not significantly different from the difference in score between both modified articles, it may be concluded that DISCERN is not effective in overcoming article sourcing bias.

Secondary[edit | edit source]

The academic backgrounds and expertise of our participants may result in previous knowledge or familiarity with articles used in this study. Chances of this occurrence can not be eliminated and must be considered in analysis of the study data. Therefore, our secondary outcome will address the potential unblinding of participants by using a short questionnaire (Appendix B) to determine whether subjects recognized one or both of the articles from previous readings. We will also determine whether the order of reading of the modified articles impacted grading by participants.T

Data collection, storage, and analysis[edit | edit source]

Primary assessment[edit | edit source]

A modified version of the DISCERN instrument will be used to collect data from participants (see: Appendix A). All responses to each DISCERN questionnaire will be entered into SPSS and separated into four groups: BMJ as BMJ, BMJ as WP, WP as WP, WP as BMJ.

The following inferential statistical tests may be conducted with the collected data:

  1. Paired T-test to determine whether the mean difference in individual DISCERN scores between results in Arms 1 and 2 are statistically different from the results in Arms 3 and 4. This test will not consider the effect that sequence may have on the DISCERN scores for each article
  2. Multi-level ordinal regression using all four arms to determine whether the order in which the two articles are read by participants potentially influenced their assessment of each article.
  3. One-Way ANOVA to determine whether the difference between DISCERN scores within each Arms 1 and 2 is statistically significant from Arms 3 and 4.

The following hypotheses will be tested:

Null hypothesis A: There is no difference in DISCERN scores between the BMJ article as a BMJ article and the Wikipedia article as a Wikipedia article

  • Data types: independent variable (nominal data): document name; dependent variable (ratio data): DISCERN score
  • Null hypothesis B: There is no difference in DISCERN scores between the WP → BMJ article and the BMJ → to WP article
    • Data types: independent variable (nominal data): document name; dependent variable (ratio data): DISCERN score
  • Null hypothesis C: There is no difference in DISCERN scores between the WP → BMJ article and the BMJ as BMJ article
    • Data types: independent variable (nominal data): document name; dependent variable (ratio data): DISCERN score
  • Null hypothesis D: There is no difference in DISCERN scores between the BMJ → WP article and the WP as WP article
    • Data types: independent variable (nominal data): document name; dependent variable (ratio data): DISCERN score

Secondary assessment[edit | edit source]

  • Study subject questionnaire (see: Appendix B and Appendix C): Descriptive statistics using SPSS
  • Does the order in which the articles are assessed affect the outcome?

Appendix A[edit | edit source]

Modified DISCERN Instrument for article evaluation[edit | edit source]

Are the aims and objectives of the topic clearly stated at the beginning of the article?

No Partially Yes
1 2 3 4 5

Does the articles cover the needed subtitles and key concepts related to the topic?

No Partially Yes
1 2 3 4 5

Is the information provided throughout the article scientifically correct and in agreement with current valid resources and textbooks?

No Partially Yes
1 2 3 4 5

Is the article neutral and not based on personal views?

No Partially Yes
1 2 3 4 5

Is the article balanced and unbiased?

No Partially Yes
1 2 3 4 5

Is it clear what sources of information were used to compile the publication (references, links to professional web sites?)

No Partially Yes
1 2 3 4 5

Has the article been regularly updated and amended?

No Partially Yes
1 2 3 4 5

Are there key areas in the article that are completed and do not need further addition?

No Partially Yes
1 2 3 4 5

Do the images, figures, and tables provided in the article support the information given and enhance the understanding of points raised?

No Partially Yes
1 2 3 4 5

What is your overall rating of the whole article as a source of information to medical students?

Serious or extensive shortcomings Potentially important but not serious shortcomings Minimal shortcomings
1 2 3 4 5

Appendix B[edit | edit source]

Pre-participation survey[edit | edit source]

(1) What is your current level of practice?

(a) Undergraduate medical student (select this option if you are a current student in an MD program, or an MD graduate who has not yet passed your license exam)

(b) Resident

(c) MD, post-residency

(d) MD + additional education (e.g. PhD, Masters)

(2) If applicable, what is your specialty (in practice or research or both): _________________________

Appendix C[edit | edit source]

Post-participation questionnaire[edit | edit source]

To be completed once you have completed the DISCERN instrument:

(1) Were either of the two articles you read familiar to you in any way? (Yes/No)

If you answered “NO” to question one (1) above, you may return the questionnaire to the investigator now.

If you answered “Yes” to the above question, please answer the two questions below.

(2) If you answered yes to question one (1) above, which article(s) were familiar to you with respect to their content:

  • Both articles
  • The Wikipedia article only
  • The BMJ article only

(3) Which article did you read first (e.g. Wikipedia or BMJ)?

Additional information[edit | edit source]

Author contributions[edit | edit source]

Competing interests[edit | edit source]

DS and JH are Wikipedians. They research Wikipedia and contribute to its content. Articles used for the purpose of this study were selected in order to ensure minimal involvement of study authors.

Ethics statement[edit | edit source]

This protocol has been approved by the Hamilton Integrated Research Ethics Board (HIREB) under project ID 8228.

Acknowledgements[edit | edit source]

Funding[edit | edit source]

References[edit | edit source]

  1. "Welcome to DISCERN". DISCERN Online. DISCERN Project. Retrieved 2020-09-09.
  2. Czarnecka-Kujawa, Kasia; Abdalian, Rupert; Grover, Samir C. (2008-04). "M1042 The Quality of Open Access and Open Source Internet Material in Gastroenterology: Is Wikipedia Appropriate for Knowledge Transfer to Patients?". Gastroenterology 134 (4): A-325–A-326. doi:10.1016/S0016-5085(08)61518-8. https://linkinghub.elsevier.com/retrieve/pii/S0016508508615188. 
  3. Thomas, Garry R.; Eng, Lawson; de Wolff, Jacob F.; Grover, Samir C. (2013-03). "An Evaluation of Wikipedia as a Resource for Patient Education in Nephrology". Seminars in Dialysis 26 (2): 159–163. doi:10.1111/sdi.12059. http://doi.wiley.com/10.1111/sdi.12059. 
  4. Qureishi, A.; Sharma, A. (2012). "Can Wikipedia replace traditional patient information leaflets? Comparing the internet to official patient information resources in thyroid cancer". European Surgery - Acta Chirurgica Austriaca 44 (Suppl. 247): 35. doi:10.1007/s10353-012-0163-y. 
  5. 5.0 5.1 Leithner, A.; Maurer-Ertl, W.; Glehr, M.; Friesenbichler, J.; Leithner, K.; Windhager, R. (2010). "Wikipedia and osteosarcoma: a trustworthy patients' information?". Journal of the American Medical Informatics Association 17 (4): 373–374. doi:10.1136/jamia.2010.004507. PMID 20595302. https://academic.oup.com/jamia/article-lookup/doi/10.1136/jamia.2010.004507. 
  6. Rajagopalan, Malolan S.; Khanna, Vineet K.; Leiter, Yaacov; Stott, Meghan; Showalter, Timothy N.; Dicker, Adam P.; Lawrence, Yaacov R. (2011). "Patient-Oriented Cancer Information on the Internet: A Comparison of Wikipedia and a Professionally Maintained Database". Journal of Oncology Practice 7 (5): 319–323. doi:10.1200/JOP.2010.000209. PMID 22211130. http://ascopubs.org/doi/10.1200/JOP.2010.000209. 
  7. Biggs, T. C.; Jayakody, N.; Best, K.; King, E. V. (2018-06). "Quality of online otolaryngology health information". The Journal of Laryngology & Otology 132 (6): 560–563. doi:10.1017/S0022215118000774. ISSN 0022-2151. https://www.cambridge.org/core/product/identifier/S0022215118000774/type/journal_article. 
  8. Watad, Abdulla; Bragazzi, Nicola Luigi; Brigo, Francesco; Sharif, Kassem; Amital, Howard; McGonagle, Dennis; Shoenfeld, Yehuda; Adawi, Mohammad (2017-07-18). "Readability of Wikipedia Pages on Autoimmune Disorders: Systematic Quantitative Assessment". Journal of Medical Internet Research 19 (7): e260. doi:10.2196/jmir.8225. ISSN 1438-8871. PMID 28720555. PMC PMC5539385. http://www.jmir.org/2017/7/e260/. 
  9. Clauson, Kevin A; Polen, Hyla H; Boulos, Maged N Kamel; Dzenowagis, Joan H (2008-12). "Scope, Completeness, and Accuracy of Drug Information in Wikipedia". Annals of Pharmacotherapy 42 (12): 1814–1821. doi:10.1345/aph.1L474. ISSN 1060-0280. http://journals.sagepub.com/doi/10.1345/aph.1L474. 
  10. Hunter, Julia Alexandra; Lee, Taehoon; Persaud, Navindra (2018-07-02). "A comparison of the content and primary literature support for online medication information provided by Lexicomp and Wikipedia". Journal of the Medical Library Association 106 (3). doi:10.5195/JMLA.2018.256. ISSN 1558-9439. PMID 29962913. PMC PMC6013145. http://jmla.pitt.edu/ojs/jmla/article/view/256. 
  11. Koppen, Laura; Phillips, Jennifer; Papageorgiou, Renee (2015-07). "Analysis of reference sources used in drug-related Wikipedia articles". Journal of the Medical Library Association 103 (3): 140–144. doi:10.3163/1536-5050.103.3.007. ISSN 1536-5050. PMID 26213506. PMC PMC4511054. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4511054/. 
  12. Lavsa, Stacey M.; Corman, Shelby L.; Culley, Colleen M.; Pummer, Tara L. (2011-04). "Reliability of Wikipedia as a medication information source for pharmacy students". Currents in Pharmacy Teaching and Learning 3 (2): 154–158. doi:10.1016/j.cptl.2011.01.007. https://linkinghub.elsevier.com/retrieve/pii/S1877129711000086. 
  13. Reilly, Timothy; Jackson, William; Berger, Victoria; Candelario, Danielle (2017-03). "Accuracy and completeness of drug information in Wikipedia medication monographs". Journal of the American Pharmacists Association 57 (2): 193–196.e1. doi:10.1016/j.japh.2016.10.007. https://linkinghub.elsevier.com/retrieve/pii/S1544319116308652. 
  14. 14.0 14.1 Phillips, Jennifer; Lam, Connie; Palmisano, Lisa (2014-07). "Analysis of the accuracy and readability of herbal supplement information on Wikipedia". Journal of the American Pharmacists Association 54 (4): 406–414. doi:10.1331/JAPhA.2014.13181. https://linkinghub.elsevier.com/retrieve/pii/S1544319115302156. 
  15. Kim, Ji Yeon; Gudewicz, Thomas M.; Dighe, Anand S.; Gilbertson, John R. (2010). "The pathology informatics curriculum wiki: Harnessing the power of user-generated content". Journal of Pathology Informatics 1 (1): 10. doi:10.4103/2153-3539.65428. ISSN 2153-3539. PMID 20805963. PMC PMC2929539. http://www.jpathinformatics.org/text.asp?2010/1/1/10/65428. 
  16. Devgan, Lara; Powe, Neil; Blakey, Brittony; Makary, Martin (2007-09). "Wiki-Surgery? Internal validity of Wikipedia as a medical and surgical reference". Journal of the American College of Surgeons 205 (3): S76–S77. doi:10.1016/j.jamcollsurg.2007.06.190. https://linkinghub.elsevier.com/retrieve/pii/S1072751507009520. 
  17. Modiri, Omeed; Guha, Daipayan; Alotaibi, Naif M.; Ibrahim, George M.; Lipsman, Nir; Fallah, Aria (2018-03). "Readability and quality of wikipedia pages on neurosurgical topics". Clinical Neurology and Neurosurgery 166: 66–70. doi:10.1016/j.clineuro.2018.01.021. https://linkinghub.elsevier.com/retrieve/pii/S0303846718300271. 
  18. Ayes, K. B.; Bardsley, C. H. (2010). "Wikipedia Information for Toxicologic Emergencies: How Reliable Is It?". Clinical Toxicology 48 (6): 635. doi:10.3109/15563650.2010.493290. ISSN 1556-3650. http://www.tandfonline.com/doi/full/10.3109/15563650.2010.493290. 
  19. Ayes, K. B.; Bardsley, C. H.; Jantz, J. M.; Frederick, W. A. (2011). "Wikipedia Information for Toxicologic Emergencies Involving Household Products, Plants and Envenomations: How reliable is it?". Clinical Toxicology 49 (6): 609. doi:10.3109/15563650.2011.598695. http://www.tandfonline.com/doi/full/10.3109/15563650.2011.598695. 
  20. Messner, Marcus; DiStaso, Marcia W.; Jin, Yan; Meganck, Shana; Sherman, Scott; Norton, Sally (2014). "Influencing public opinion from corn syrup to obesity: A longitudinal analysis of the references for nutritional entries on Wikipedia". First Monday 19 (11). doi:10.5210/fm.v19i11.4823. https://journals.uic.edu/ojs/index.php/fm/article/view/4823. 
  21. Sanz-Valero, J.; Cabrera-Hernández, L.; Wanden-Berghe, C.; Culebras-Fernandez, J.M. (2013-09). "PP188-MON The popularization of food and nutritional sciences: Wikipedia versus a general encyclopedia". Clinical Nutrition 32 (Suppl. 1): S192. doi:10.1016/S0261-5614(13)60499-9. https://linkinghub.elsevier.com/retrieve/pii/S0261561413604999. 
  22. Sanz-Valero, J.; Wanden-Berghe, C.; Guardiola-Wanden-Berghe, R. (2012-09). "PP154-MON Nutrition and metabolism in Wikipedia: Presence and adequacy of English and Spanish terminology". Clinical Nutrition Supplements 7 (1): 198–199. doi:10.1016/S1744-1161(12)70493-3. https://linkinghub.elsevier.com/retrieve/pii/S1744116112704933. 
  23. Koo, Malcolm (2014). "Complementary and Alternative Medicine on Wikipedia: Opportunities for Improvement". Evidence-Based Complementary and Alternative Medicine 2014: 1–4. doi:10.1155/2014/105186. ISSN 1741-427X. PMID 24864148. PMC PMC4016830. http://www.hindawi.com/journals/ecam/2014/105186/. 
  24. Simpson, Andrea; Le, Michelle; Malicka, Alicja N. (2018-10-02). "The Accuracy and Readability of Wikipedia Articles on Hearing Loss". Journal of Consumer Health on the Internet 22 (4): 323–336. doi:10.1080/15398285.2018.1542251. ISSN 1539-8285. https://www.tandfonline.com/doi/full/10.1080/15398285.2018.1542251. 
  25. Reavley, N. J.; Mackinnon, A. J.; Morgan, A. J.; Alvarez-Jimenez, M.; Hetrick, S. E.; Killackey, E.; Nelson, B.; Purcell, R. et al. (2012-08). "Quality of information sources about mental disorders: a comparison of Wikipedia with centrally controlled web and printed sources". Psychological Medicine 42 (8): 1753–1762. doi:10.1017/S003329171100287X. ISSN 0033-2917. https://www.cambridge.org/core/product/identifier/S003329171100287X/type/journal_article. 
  26. Stankus, Tony; Spiegel, Sarah E. (2010-08-31). "Wikipedia, Scholarpedia, and References to Journals in the Brain and Behavioral Sciences: A Comparison of Cited Sources and Recommended Readings in Matching Free Online Encyclopedia Entries". Science & Technology Libraries 29 (3): 258–265. doi:10.1080/0194262X.2010.497711. ISSN 0194-262X. https://doi.org/10.1080/0194262X.2010.497711. 
  27. Herbert, Verena G; Frings, Andreas; Rehatschek, Herwig; Richard, Gisbert; Leithner, Andreas (2015-12). "Wikipedia – challenges and new horizons in enhancing medical education". BMC Medical Education 15 (1): 32. doi:10.1186/s12909-015-0309-2. ISSN 1472-6920. PMID 25879421. PMC PMC4384304. https://bmcmededuc.biomedcentral.com/articles/10.1186/s12909-015-0309-2. 
  28. Judd, Terry; Kennedy, Gregor (2011-03). "Expediency-based practice? Medical students' reliance on Google and Wikipedia for biomedical inquiries: Students' reliance on Google and Wikipedia". British Journal of Educational Technology 42 (2): 351–360. doi:10.1111/j.1467-8535.2009.01019.x. http://doi.wiley.com/10.1111/j.1467-8535.2009.01019.x. 
  29. Murray, Heather; Walker, Melanie; Maggio, Lauren; Dawson, Jennifer (2018-06). "24 Wikipedia medical page editing as a platform to teach evidence-based medicine". Oral Sessions 23: A12–A13. doi:10.1136/bmjebm-2018-111024.24. http://ebm.bmj.com/lookup/doi/10.1136/bmjebm-2018-111024.24. 
  30. Azzam, Amin; Bresler, David; Leon, Armando; Maggio, Lauren; Whitaker, Evans; Heilman, James; Orlowitz, Jake; Swisher, Valerie et al. (2017-02). "Why Medical Schools Should Embrace Wikipedia: Final-Year Medical Student Contributions to Wikipedia Articles for Academic Credit at One School". Academic Medicine 92 (2): 194–200. doi:10.1097/ACM.0000000000001381. ISSN 1040-2446. PMID 27627633. PMC PMC5265689. http://journals.lww.com/00001888-201702000-00022. 
  31. Schweitzer, N. J. (2008-04-15). "Wikipedia and Psychology: Coverage of Concepts and Its Use by Undergraduate Students". Teaching of Psychology 35 (2): 81–85. doi:10.1080/00986280802004594. ISSN 0098-6283. http://www.tandfonline.com/doi/abs/10.1080/00986280802004594. 
  32. Haigh, Carol A. (2011-02). "Wikipedia as an evidence source for nursing and healthcare students". Nurse Education Today 31 (2): 135–139. doi:10.1016/j.nedt.2010.05.004. https://linkinghub.elsevier.com/retrieve/pii/S0260691710000924. 
  33. Azer, Samy A. (2014-02). "Evaluation of gastroenterology and hepatology articles on Wikipedia: Are they suitable as learning resources for medical students?". European Journal of Gastroenterology & Hepatology 26 (2): 155–163. doi:10.1097/MEG.0000000000000003. ISSN 0954-691X. http://journals.lww.com/00042737-201402000-00004. 
  34. Azer, Samy A.; AlSwaidan, Nourah M.; Alshwairikh, Lama A.; AlShammari, Jumana M. (2015-10). "Accuracy and readability of cardiovascular entries on Wikipedia: are they reliable learning resources for medical students?". BMJ Open 5 (10): e008187. doi:10.1136/bmjopen-2015-008187. ISSN 2044-6055. PMID 26443650. PMC PMC4606442. http://bmjopen.bmj.com/lookup/doi/10.1136/bmjopen-2015-008187. 
  35. Antivalle, M.; Battellino, M.; Ditto, M.C.; Varisco, V.; Chevallard, M.; Rigamonti, F.; Batticciotto, A.; Atzeni, F. et al. (2014-06). "SAT0585 Evaluation of Wikipedia Rheumatology Articles as A Learning Resource for Medical Students". Annals of the Rheumatic Diseases 73 (Suppl. 2): 801.3–802. doi:10.1136/annrheumdis-2014-eular.5610. ISSN 0003-4967. http://ard.bmj.com/lookup/doi/10.1136/annrheumdis-2014-eular.5610. 
  36. Jetty, Prasad; Yacob, Michael M.; Lotfi, Shamim (2014-11). "The Wikipedia Medical Student: Comparing the Quality of Vascular Surgery Topics Across Two Commonly Used Educational Resources". Journal of Vascular Surgery 60 (5): 1404. doi:10.1016/j.jvs.2014.08.030. https://linkinghub.elsevier.com/retrieve/pii/S0741521414015559. 
  37. Azer, Samy A. (2015-03). "Is Wikipedia a reliable learning resource for medical students? Evaluating respiratory topics". Advances in Physiology Education 39 (1): 5–14. doi:10.1152/advan.00110.2014. ISSN 1043-4046. https://www.physiology.org/doi/10.1152/advan.00110.2014. 
  38. Park, Elizabeth; Masupe, Tiny; Joseph, Joseph; Ho-Foster, Ari; Chavez, Afton; Jammalamadugu, Swetha; Marek, Andrew; Arumala, Ruth et al. (2016-11). "Information needs of Botswana health care workers and perceptions of wikipedia". International Journal of Medical Informatics 95: 8–16. doi:10.1016/j.ijmedinf.2016.07.013. https://linkinghub.elsevier.com/retrieve/pii/S1386505616301745. 
  39. "Wikipedia Zero". Wikimedia Foundation Governance Wiki. Wikimedia Foundation. 2018-10-05. Retrieved 2020-09-23.
  40. Devgan, Lara; Powe, Neil; Blakey, Brittony; Makary, Martin (2007-09). "Wiki-Surgery? Internal validity of Wikipedia as a medical and surgical reference". Journal of the American College of Surgeons 205 (3): S76–S77. doi:10.1016/j.jamcollsurg.2007.06.190. ISSN 1072-7515. https://doi.org/10.1016/j.jamcollsurg.2007.06.190. 
  41. "Welcome from the Chair". Department of Medicine. McMaster University. Archived from the original on 2020-06-29. Retrieved 2020-09-09.
  42. "Welcome to the Department of Medicine, University of Toronto". Department of Medicine. University of Toronto. 2016-03-04. Retrieved 2020-09-09.
  43. "Facts and Figures". Schulich School of Medicine & Dentistry. Western University. Retrieved 2020-09-09.
  44. "Faculty of Medicine". UBC Research + Innovation. University of British Columbia. Archived from the original on 2020-04-14. Retrieved 2019-12-13.