Become a Reviewer


	Information for Reviewers
		JESSE Reviewer's Guide Background Science and education communities agree that peer review of scholarly research is a fundamental requirement for establishing credibility and acceptance in academia. Publication in the peer reviewed literature documents the accomplishments of a researcher, and offers a level of prestige to the authors. Most importantly, peer review serves the broader community by establishing a rubric for applying critical thinking and the scientific process to the extension of knowledge. The peer review and editorial function assures the basic reliability and quality of the reviewed work, which serves as a basis for extension by others. JESSE implements both peer review and moderated peer commentary for learning resources. Peer review vets a resource using a defined set of criteria embraced by the broader scientific community but applied by a small group of experts during the review itself. Peer commentary extends the discourse to the community at large, including the actual users of the resource, and encourages feedback and comments on how the material was actually used, its effectiveness in the classroom, and suggestions for supplements or improvements. Peer commentary relies upon grassroots community participation to be effective. JESSE implements an innovative open approach to the peer review process to foster meaningful community interactions, partnerships and mentoring. The Editors and potential reviewers will first preview the learning resource before agreeing to the open review. If the review is to be open, reviewers are asked to communicate directly with the author in reaching agreement regarding the acceptability for publication. The Editor will mediate the discussion as necessary and will be responsible for the final decision regarding publication. The purpose of the JESSE review is to confirm the acceptability of a resource for publication following the review criteria set forth. The open peer review results in a final commentary from the reviewers and the author regarding the resource and its implementation for specific audiences. For example, a comment could be: "This resource provides an excellent introduction at the freshman undergraduate level to satellite remote sensing of net primary productivity, but does not address the physics of EM scattering and reflection." These kinds of comments will be available online with the reviewed resource, easing the burden on faculty who wish to determine in advance whether or not a resource is appropriate for a specific lesson plan or curriculum. The open review fosters a new spirit of cooperation and community participation in defining and vetting the body of educational resources for Earth system educators. Reviewers and authors as mentors and mentees (or vice versa) become partners in creating quality content that can be shared with the community. The demands upon a reviewer increase as a result of the open review process, but the reciprocal rewards of such a cooperative effort will be considerable. Whenever possible, JESSE seeks reviewers who will include the resource in their own course offerings as a benefit to the reviewers and their students. Experienced faculty can share their knowledge of what works in the classroom with younger faculty, and innovative pedagogies and learning tools can be introduced in new contexts. JESSE Review Criteria Resources will be reviewed by seven criteria that are also adopted by the Digital Library for Earth System Education: 1. Scientific Accuracy / Credibility Scientific accuracy and credibility of a resource is critical. The information conveyed in a resource must be based upon recognized scientific principles and results, or empirical evidence observed directly. Resources dealing with scientific uncertainty are welcome, but should clearly describe the state of current knowledge and the nature of the uncertainty. Resources must avoid philosophical "spin" and should present facts openly with an unbiased manner. Scientific credibility and accuracy will be determined by reviewers intimately familiar with the science being described, and in most cases be practicing scientists themselves. 2. Pedagogical effectiveness The pedagogical effectiveness of a learning resource must be considered in the context of how the object is to be used, the level of the audience to be reached, and the background of the teacher presenting the material. Pedagogies should be adapted to different learning styles, and a review of pedagogical effectiveness becomes an assessment of the match between the pedagogy presented and the intended audience. Some resources will lend themselves to be delivered using multiple pedagogies. The purpose of this element of the review is to categorize which pedagogies are being used and comment on their relative effectiveness for the target audience. This pedagogical categorization will be available as part of the metadata, searchable by the user. It is unlikely that any pedagogy will be categorically incorrect, but the pedagogy chosen will be noted, with respect to strengths, limitations and appropriateness for the target audience. The range of pedagogies applicable to Earth system education and their effectiveness among learning styles and audiences will be a welcome topic within JESSE. 3. Ease of use for students and faculty Resources which are readily utilized and self explanatory are likely to be in demand by more people than complex learning resources with special system requirements, installation procedures or user caveats. This criterion will assess the simplicity of working with the learning resource, and note any special requirements or problems. Complicated objects may well require special operating systems, conditions and user inputs. The end user needs to be aware of these requirements in trying to adopt them for classroom use. Here again the peer commentary portion of JESSE will augment the original reviews on this topic, and users can comment on the ease of use of a learning resource via peer commentary. 4. Well documented The resource must be complete and reviewable as submitted. Data file formats must be documented, a bibliography or reference list included if applicable, as well as a one paragraph synopsis of the resource, and statements regarding the grade level of the resource, intended audience, intended use (e.g. classroom, lab, student project, etc.) and intended duration of the exercise. Documentation should also list ancillary data required for the exercise (e.g. a specific textbook, or data set, image, observation, etc.). If applicable, the appropriate National Science Education Standards or AAAS Benchmarks addressed by the learning resource should be referenced. 5. Importance / Significance The importance or significance of a resource for educators is difficult to measure before making it available to the community and determining the degree of acceptance. A resource can deal with an important or significant problem, and still be of minimal value for classroom educational purposes, and vice versa. The developer of the resource will be required to state his/her perceived significance of the resource, which will be provided for the reviewer and then included in documenting metadata. Following publication, community response to the resource as indicated by the peer commentary received will be used to measure the resource’s utility for education. 6. Inspirational, motivational for students Some learning resources will motivate and inspire students more than others, and this review criteria will gauge the level and depth of student engagement and long term effect. Visually appealing layout, topical relevance and engaging content will contribute to high marks for this criterion. Reviewers will be asked to judge the perceived inspirational, motivational value, but direct student feedback via peer commentary is most important. 7. Robustness / Sustainability Some learning resources are fragile, requiring specific operating system requirements, plug-ins or specialized software. These resources may be difficult to maintain and should be noted. Authors will be asked to identify any special system requirements that impact this criterion, and reviewers will note their experiences. The interactive open JESSE review process will assist authors to identify software or process flaws in the products, though the review itself is not to be a substitute for thorough beta testing of a product. Beta testing is essential before the formal review of the resource. JESSE Review Process By agreeing to be an open reviewer for JESSE, you should recognize that your review is not anonymous, and that you will be in direct contact with the authors. You will be contributing directly to the authors' efforts to create or revise a resource deemed suitable for publication by JESSE. A summary of reviewer comments (not necessarily attributed directly to you) will be made available with the resource when published, and you will be identified as one of the reviewers. Provision can be made for anonymous involvement in this process if necessary. However, we hope that reviewers will choose to identify themselves since this is likely to foster a better spirit of open discussion with the author, other reviewers, and the wider interdisciplinary community who participate with their comments after the resource is published. In this early stage, JESSE also conducts a traditional anonymous review in parallel with the open review to validate and ensure credibility of the open review approach. Anonymous reviewers use the same review criteria, but without the D3E or contact with each other or the authors. JESSE has adopted the Digital Document Discourse Environment (D3E) as the web based tool for managing the review discussion. D3E was developed by Tammy Sumner and Simon Buckingham Shum (http://d3e.open.ac.uk/general/d3e-chi98/) and is being used to manage the review process for another online journal - the Journal of Interactive Media in Education (JIME) (http://www-jime.open.ac.uk/). D3E builds upon the HyperNews threaded discussion software. From a reviewer's perspective, the process is as follows: 1. The resource is posted in the JESSE article area as a private (though not password protected) link. This link will be provided to the authors, reviewers and editors, but not distributed broadly. Other e-journals have found this minimal level of security (i.e. private link distributed only to those with a need to know) adequate for the purpose of this kind of review. Read the submission, and try any interactive demonstrations or links that the author has provided. In some cases the resource is better reviewed directly from CD or web-site than in the D3E window. The D3E tries to present the resource in a format that is as close to what a user will experience as possible. D3E works well for text-only resources, but complicated multimedia resources or web sites may only be outlined by the D3E. 2. Formulate your comments. Try to organize your comments under the seven JESSE criteria: Scientific accuracy / credibility Pedagogical effectiveness Ease of use Adequacy of documentation Importance / significance Motivational value Robustness / sustainability Post your comments in the appropriate D3E thread for discussion with the other reviewers and authors. All postings to the web site will be automatically sent by email to those subscribed to the discussion. Try to be as descriptive as possible in your message titles. Please place section-specific comments under the relevant number heading (e.g. 1.3 or 6.0). There is no need to comment on every section heading. These threads are created for your convenience and to help focus the discussion if a specific content area needs work. Comments about the PROCESS are welcome under the heading "Review Logistics". Use the thread "General Comments" for broad questions about the resource. You may find it useful to view a summary of all comments using the "Go to top of review debate" link when in a specific thread. You may wish to use your normal word processor from which you can paste text into the website. Specific suggestions for clarifying concepts or writing are welcomed. Reviewers should consider themselves to be advisors and consultants to the author (in fact, in situations which require significant rework, the author may wish to consider adding contributing reviewers as co-authors, in which case additional reviewers will be found. While this example is not expected to represent the norm, it exemplifies a new model of collaboration and community involvement that JESSE is trying to foster). You can of course send private comments to the Editors or authors via normal e-mail, but we perceive the strength of the JESSE review to be in the open and constructive exchange of comments, criticisms and suggestions. 3. At the end of the agreed upon review period, the editors will poll the reviewers for their recommendation as to the resource, and what changes if any need to be made before the resource can be published. If the Editors determine that the paper is broadly acceptable, they will work with the authors to establish a timeline for receiving changes and publication of the resource. A summary of reviewer comments and suggestions will be created, as well as a moderated peer commentary discussion area soliciting public comments about the resource, its use in the classroom, suggestions for authors, etc. 4. At this point your obligations as a reviewer are completed. However, we hope that you will be interested enough in the resource and the process to join in the open peer commentary that follows, if the submission has been judged broadly acceptable.