[Humanist] 24.837 DHQ to automated peer-review
Humanist Discussion Group
willard.mccarty at mccarty.org.uk
Fri Apr 1 09:01:09 CEST 2011
Humanist Discussion Group, Vol. 24, No. 837.
Centre for Computing in the Humanities, King's College London
Submit to: humanist at lists.digitalhumanities.org
Date: Thu, 31 Mar 2011 12:42:35 -0400
From: Julia Flanders <julia_flanders at brown.edu>
Subject: DHQ moves to all-automated peer review
Digital Humanities Quarterly is pleased to announce a move to Peer Review with Advanced Technology (PRAT): a new computational peer review system based on text analysis.
This new approach to peer review will enable DHQ to process vastly increased numbers of submissions, speeding up time to publication and ensuring consistency of review criteria, compared with results from human peer reviewers.
This move comes at a time when peer review mechanisms are coming under close scrutiny, as journals and other online publications experiment with alternatives to traditional peer review. DHQ has considered a variety of models, including crowd-sourced peer review based on reader comments, but we have determined that these require too great an investment of time and create unacceptable delays in production. Our new text-analysis-based method will eliminate human reviewers altogether and assess submissions based on a set of measures that may include:
1. Stylistic similarity to articles in the highest tier of citation rankings in a selected set of major journals, using state-of-the art measures that focus on vocabulary choice and patterns of clause construction.
2. Average and maximum word length, sentence length, and clause length. Submissions will be scored in all three categories and the results will be compared with a proprietary DHQ readability index to determine the submission's suitability for publication. The highest-scoring articles will be published in a special "Expert" column.
3. Frequency of high-value technical terminology. Terms from specific domains will be weighted dynamically, based on current measures of usage of those terms in high-ranking peer journals, to ensure that DHQ's published articles keep pace with the changing patterns of technical jargon in the broader community. For special issues, terms from specific topic areas will be given extra weight to ensure that submissions are related to the issue's subject area.
4. Rate of citation of the most highly cited references in digital humanities. Submissions that cite the most highly cited references (or articles citing those) will be more highly ranked. Over time, we anticipate that this mechanism if properly used could result in the DHQ article corpus achieving complete self-referentiality.
The new system goes into effect with the special issue "From Lemons to Lemonade: Learning from project failures in the Digital Humanities", to be published on April 1.
Best wishes, Julia, Melissa, and Wendell
General Editors, DHQ
More information about the Humanist