User:JWSchmidt/Blog/11 March 2007
IRC meeting about research
The IRC meeting about research held on 10 March 2007 attracted about 30 people to the #wikiversity chat channel. (Note: there are some links below to specific sections of the research policy pages at the Wikiversity multi-lingual hub, however the links do not actually take you right to the correct sections of those pages.)
Wikiversity has an education-oriented mission that leads some participants to explore how to move beyond the Wikimedia Foundation Neutral point of view (NPOV) policy and the Wikipedia:No original research (NOR) restriction. In particular, the Wikiversity community was asked to develop research guidelines during the first six months of the project (email discussion group post, by Board member Anthere; special projects committee resolution).
Issues raised in the IRC discussion
Scope of research. An attempt was made to start with the scope of research activities that will be allowed within Wikiversity. One issue raised was the relationship between publishing of research results and the conduct of research at Wikiversity. Rayc mentioned a rather extreme way to define research, "only things that will eventually be published be considered research. everything else would be a learning project." Rayc suggested a second category, "casual research", as being, "something that is not going to be published." CQ (yeoman) mentioned the idea of, "classifications or ratings for research intensity... for instance 'formal methods' for academically peer-reviewed projects at one end but allowing 'informal or casual' research for the rest of us." The only distinction between types of research projects in the current proposed policy is for projects that would require review by an IRB (Institutional Review Board). Wikiversity would not have an IRB, but it might make sense to have a "research review board" that could deal with any Wikiversity research issues that might come to involve an IRB from a bricks-and-mortar research institution. The term "research review board" was selected in anticipation of a future time when communities of Wikiversity scholars might begin to explore formal peer-reviewed publishing within Wikiversity, but the proposed research policies leave that for the future. For now, the proposed research review board would be an advisory board that would help the whole Wikiversity community deal with any problems arising in on-going research projects. User:Cormaggio stated the current major goal for Wikiversity research, "a place where people can learn about the practice of doing good research".
Another topic that arose early in the IRC-based discussion was the question of to what extent research policy can be set for all Wikiversity projects. The Wikimedia Foundation Board of Trustees originally requested that "as much as possible" there be one set of rules for all of Wikiversity rather than have each language-specific Wikiversity website doing their own thing. The largest objection to uniform policy seemed to come from Rayc, "uniform policies don't seem to have taken hold in wikipedia. What is the benfit?" However, as far as I can tell, NPOV and NOR are universal policies in Wikipedia projects. If Wikiversity is going to allow original research and editing outside of the restrictions of traditional NPOV policy, then there can be a single Wikiversity-wide approach to doing so. I think it is clear that the Board does not want to get into a situation where each time a small community starts a new Wikiversity website for a particular language, they have to re-invent rules for research activities just for that language.
A second issue related to the scope of Wikiversity research was if the education-oriented mission restricts the scope of research or if Wikiversity is open to all types of research. The current policy proposal starts with the idea that "The scope of research at Wikiversity is limited to research activities that promote learning and the goals of the Wikimedia Foundation." I think it would be possible to fit many kinds of research into the Wikiversity mission, but it also makes sense to include the idea that: "At times, it may be useful to amend the Wikiversity research guidelines so as to explicitly exclude some types of fringe research if they disrupt Wikiversity or distract the community from its educational mission." I think that sentence probably addresses a concern raised by Gmaxwell, "Do you agree that if we wanted to we could define a policy that said 'we will reject whatever our community dislikes'?" If someone brings some fringe research topic to Wikiversity and it causes more conflict and disruption than education, then the community will certainly be able to restrict that area of research.
The research policy proposal includes a list of categories of research projects that are not suitable for Wikiversity.
Restrictions on the scope of research imposed by ethical guidelines
Detecting and eliminating bogus research
A problem that has been made clear through past experience at the Wikipedia project is that wiki projects become targets for people who want to post bogus ideas. The policies that require citation of verifiable sources and the NPOV and NOR policies help to defend Wikipedia against large amounts of potential trouble from pushers of fringe ideas and scam artists who enjoy posting bogus ideas. What is not clear to most people is the role played by experts in helping a wiki community identify bogus ideas and keep them out of a wiki website. The familiar problems with identifying and removing bogus wiki content will be larger problems within Wikiversity research projects where the protections used at Wikipedia will be removed. The proposed research guidelines say, "some forms of bogus and unethical research can only be recognized by experts" and calls for Wikiversity to have a system (research review board) that promotes the use of expert knowledge to detect and correct problems in research methods.
Historybuff (HistoryOnTheRoad) made the point that, "A uniform approach will cut down on 'forum shopping' that an extreme group does to try to find a WV that will accept it's screwball theory." User:WiseWoman made the point that, "what one group finds 'well-documented and cited' is considered to be 'non-verified' in another." Somehow we need to be clear about the distinction between large issues that need to be stated explicitly in Wikiversity-wide policy and small issues that need to be handled by smaller communities of participants at the level of editing individual wiki webpages and research projects. The current proposal says, "Wikiversity does not exist as a platform to support advocates of particular political movements, religious ideologies or scientific, legal or historical theories." That is a general policy statement that allows Wikiversity communities a basis in policy for blocking unwelcome groups from using Wikiversity as a soap box for their propaganda. The details of determining what is "unwelcome" or a "screwball theory" is left up to individual wiki communities. We do not try to micromanage and provide a list of every specific research project that would not be welcome within Wikiversity. Such details are "fine-tuning" that will take place outside of the policies when the general principles in the policies are applied by the Wikiversity community.
The research policy proposal identifies two levels of "fine-tuning"; conventional wiki editing processes that provide for continual peer review of wiki content and, in addition, something new. "Conventional wiki editing" includes all of the tools that have been developed by editors at Wikipedia, such as the Wikipedia processes by which community members use templates to flag problems on wiki pages and then use of discussions to decide how to correct problems, and, if needed, delete pages that cannot be corrected. At Wikipedia, decisions about page deletion often come down to counting the number of verifiable citations that can be found to support the importance and continued existence of a page. When the Wikiversity community gets involved with discussion of research projects, it will not be possible to just count existing published sources that support the research. When dealing with the production of new research results, you need to make use of the experience of participants who are familiar with the problems that can arise from bogus research methods. The proposed research review board is a suggestion for how to create a process that would help the community make use of existing expertise in the community for dealing with problems that will arise in research projects.
For the types of research projects that are appearing within Wikiversity (see the examples at scope of research), there are no great controversies and a formal research review board is not needed. However, it is easy to imagine future situations that will benefit from a formal research review board. There will be "crank research" placed on Wikiversity pages, and Wikiversity will benefit from a system that encourages participants with expertise to help identify and correct/remove bogus research. Eventually, some Wikiversity participants will want to experiment with formal peer review of research results. The proposed research review board could gracefully evolve into a way for Wikiversity to incorporate a formal peer review process.
Gmaxwell mentioned the idea that, "it's usually best to wait until there is a problem before deciding on a policy to avoid problems, rather than trying to pre-solve issues which may never arise and working with conjecture rather than fact". I think that attitude is fairly common among wiki participants and it can be applied to the whole process of creating policy for research projects within Wikiversity. _sj_ suggested that, "the 'review board' should be a process and perhaps a page, not a group of users". I imagine that Wikiversity might come to have a "process" such as Wikipedia:Articles for deletion, but I'm more interested in a process that would focus on identifying methodological problems in research projects and guiding the community towards correcting problems that arise in research projects. The proposed research review board is designed to "help the community verify the methods used in research projects and deal with any potential problems". The distinction between "a process" and "a group of users" seems like an artificial distinction, particularly when Wikiversity will be a community in which some people have previous experience in dealing with research problems while others are trying to learn how to do research. Why not allow the community to have a process that explicitly calls upon more experienced members of the community to take a leadership role in the process?
It is natural for people to see words like "review board" and start imagining a select group that will impose its will. Anyone who reads the proposed research review board policy will see that it is essentially a proposal for an advisory board designed to guide community discussions. Mike42 expressed such concerns as, "If 'we' is an elected group, then people will see them as above the rules." This is why the review board policy says, "Becoming a Referee means taking on extra duties and responsibilities in service of the Wikiversity community. In particular, Referees are expected to apply their expertise and experience to help guide the community in peer review of research methods and practices used in Wikiversity research projects. Referees never demand action based on their 'expert opinion', rather they use their expertise to help provide explanations that guide the community to consensus."
JW: I think that this is an absolutely essential piece to the puzzle if we are to have any sustainable credibility as a valid source of reference. The ideal would lie in a WSERB (Wikiversity Services Ethics Review Board) coupled with a Peer Review Process. That being that there be a pair or team working together before any item is "Accepted" into the Wikiversity. There could be an interim phase where the submission is "pending" pending editorial or peer review and acceptance. Not any different than we utilize to publish in a journal. I feel quite adamantly about this because the Wikiversity is public domain and the information enclosed herein may benefit all. However, interpretation of this knowledge may result in negative impacts to the users. It wasnt that long ago In Canada that this credibility was an issue. That was of the major Canadian news networks airing a clip of a boat in high seas and distress that was on YouTube and mistakenly labelled as occurring off Newfoundland. In reality it was no where near the Canadian coast and this had negative impacts on the shipping company in question.
What is published in Wikiversity should be valid, accurate, credible and replicatable. We ultimately need depth and breadth. Challenging ideas while labelled challenging or contoversial I feel should be included as long as a disclaimer is attached. If we are using this service clinically we are able to make an educated and informed decision as long as conflicts of interest and diclaimers for "off-label" uses (no matter what area or service) are included.
Thanks for all of your dedication to the cause and enriching minds across the miles and the years. Joel Lamoure 00:14, 16 March 2007 (UTC)
"credibility as a valid source" <-- Why should anyone trust the information they find at Wikiversity? Sure, you could make Wikiversity a closed project where every editor and edit is under strict control. But Wikiversity is using the Wikipedia model of openness and inclusiveness. We invite everyone to edit. Wikiversity is a wiki where participants can "learn by doing" and, in particular, learn by editing webpages. So if anyone can edit, why trust the content? We rely on some rules that shape the culture of Wikiversity. In particular, we demand that information on Wikiversity pages be supported by citation of verifiable sources. But what happens when we follow our interests and start exploring new domains of knowledge beyond what can be supported by previously published information? When we move beyond the safety of Wikiversity:Cite sources, how can we keep Wikiversity safe from errors and fraud and hoaxes?
Wikipedia has always made use of experts and the special skills they have, but Wikipedia has never turned the expertise of editors into "a big deal". As Wikiversity moves towards allowing original research projects, it is clear that those projects can benefit from the experience of experts who have already learned how to do good research in various subject areas. How can we encourage the participation of experts and make use of that participation to provide the Wikiversity project with credibility? And how can we attract and efficiently utilize experts while still keeping Wikiversity an open and inclusive project where non-experts feel welcome and eager to participate? I hope we can continue to engineer a wiki culture to promote the idea of, "From each according to his abilities, to each according to his needs."
--JWSchmidt 01:21, 16 March 2007 (UTC)
- Gmaxwell mentioned the idea that, "it's usually best to wait until there is a problem before deciding on a policy to avoid problems, rather than trying to pre-solve issues which may never arise and working with conjecture rather than fact".
- This was from the experience of the English Wikipedia. However, if I am not mistaken, it took Larry Singer some effort to formulate the NPOV policy originally, and the English Wikipedia had Jimbo Wales and then the ArbComm to protect it. Core policies should not wait.Hillgentleman|Talk17:34, 23 March 2007 (UTC)]