[Humanist] 27.395 peer review

Humanist Discussion Group willard.mccarty at mccarty.org.uk
Thu Oct 3 07:52:20 CEST 2013


                 Humanist Discussion Group, Vol. 27, No. 395.
            Department of Digital Humanities, King's College London
                       www.digitalhumanities.org/humanist
                Submit to: humanist at lists.digitalhumanities.org



        Date: Wed, 02 Oct 2013 11:06:54 -0400
        From: "David L. Hoover" <david.hoover at nyu.edu>
        Subject: Re:  27.391 peer review
        In-Reply-To: <20131002062659.7AFE13A14 at digitalhumanities.org>


Dear James, Barbara, and all,

The single blind process we have now acknowledges the fact that the 
number of submissions has probably reached the point where we can't 
check to assure that each one has been made as anonymous as possible. It 
also seems to me to be based on the view that blind reviews are more 
likely to be candid than ones where the author can see who wrote the 
review. It has the negative characteristic that authors with high status 
or a penchant for revenge may have abstracts accepted that are of 
significantly lower quality than those by unknown authors. And, as 
Barbara suggests, blind reviews allow for the possibility of negative 
bias and allow the reviewer to hide behind the anonymity of the process.

A totally open process does, as Barbara says, require that reviewers own 
their own views, but it has the corresponding negative characteristic 
that reviewers may be intimidated by some authors, and may, in general, 
tend to write bland, mid-range reviews that make the program committee's 
job much more difficult. Again, authors with strong reputations may have 
relatively weak abstracts accepted. I have sometimes given an author 
whose talks are always of the highest quality a set of rankings that are 
not completely justified by the quality of the abstract alone, but I 
think this is, in fact, a justifiable practice. Further, it assures that 
those who want to reward their friends and punish their enemies can do 
so, as long as they do it openly (though it won't necessarily be obvious 
that they are doing it).

The process last year and this, by allowing authors to respond to the 
reviews, seems to respond to some of the weaknesses of the single blind 
review, by allowing the author to counter an unfair or biased 
assessment. Allowing reviewers to opt out of reviewing some papers also 
seems like a positive step, and it at least assures that reviewers don't 
work on abstracts that they have no interest in. I am not as sure about 
the benefits of allowing reviewers to bid on papers, as this is also 
obviously open to abuse and collusion (If you give my paper a great 
review, I'll do the same for yours; oh, that jerk has an abstract this 
year, I'll fix her; he did me a good turn, so I'm going to give his 
crappy abstract a high score).

All the processes have competing weaknesses and strengths. In a perfect 
world, I would prefer a double-blind process with an opportunity for the 
author to respond, but I realize that is probably not be possible. This 
issue seems like one that should be taken up by the entire community and 
discussed at AGM's and in the Conference Coordinating Committee.

David Hoover

On 10/2/2013 2:26 AM, Humanist Discussion Group wrote:
>                   Humanist Discussion Group, Vol. 27, No. 391.
>              Department of Digital Humanities, King's College London
>                         www.digitalhumanities.org/humanist
>                  Submit to: humanist at lists.digitalhumanities.org
>
>
>
>          Date: Tue, 1 Oct 2013 20:47:47 +0100
>          From: "James O'Sullivan" <josullivan.c at gmail.com>
>          Subject: Re:  27.389 peer review
>          In-Reply-To: <20131001053222.F33D130A2 at digitalhumanities.org>
>
>
> Dear Barbara,
>
> There's various open models, but I admit that I've only ever had experience
> with blind processes. The majority of them, as you have pointed out, are
> based on the idea that reviewer identities are also visible. I agree with
> your position that we require a system where "every individual takes
> responsibility for what they write". Whether or not this would be feasible
> for the ADHO, I don't know, but it's a position that should certainly be
> explored.
>
> The consensus seems to be that reviewers are capable of identifying
> authors. But I wonder as the popularity of the discipline grows, will this
> continue to be the case? Where reviewers can identify authors, there will
> always be doubt in relation to a selection process, and as unrealistic as
> it might seem to achieve a completely fair process, surely it is in the
> best interest of the discipline and community at large to at least try an
> implement a system that strives to offer either complete anonymity or
> transparency?
>
> To my mind there simply isn't any advantage to a single blind review
> process. Ok, perhaps reviewers can identify authors, but why not take
> measures to at least make this more difficult? Or, take the converse
> approach as Barbara suggests, and ensure that reviewers are as visible as
> authors, so as to ensure transparency across the board?
>
> It's certainly food for thought and something that I think should be
> discussed in a more official capacity by the powers that be.
>
> Thanks to all for the informed responses - a pleasure as always!
>
> Best regards,
> James
>
> On Tue, Oct 1, 2013 at 6:32 AM, Humanist Discussion Group <
> willard.mccarty at mccarty.org.uk> wrote:
>
>>                   Humanist Discussion Group, Vol. 27, No. 389.
>>              Department of Digital Humanities, King's College London
>>                         www.digitalhumanities.org/humanist
>>                  Submit to: humanist at lists.digitalhumanities.org
>>
>>
>>
>>          Date: Mon, 30 Sep 2013 09:51:11 -0600
>>          From: Barbara Bordalejo <bb268 at nyu.edu>
>>          Subject: Re:  27.387 peer review
>>          In-Reply-To: <20130930053844.B6C8930A0 at digitalhumanities.org>
>>
>>
>> Hi James,
>>
>> I think you make some very good points, but I wish you had elaborated more
>> on the "open process" you suggest. I have some experience editing a journal
>> for which we used blind peer review, a process that I was responsible for
>> setting up. The problem, as Martin has already pointed out, is that
>> sometimes it is very evident who wrote a paper. A further problem is that
>> some reviewers seem to "enjoy" to discover who wrote what or to confirm a
>> suspicion about the identity of an author. So, when these factors are taken
>> into account it is very difficult to sustain a "blind" process (although we
>> might want to pretend that everything is fine). Double blind review does
>> not help much either, as it is generally carried out by the same group of
>> established academics.
>>
>> I am not sure what you had in mind with the open process, but what I would
>> like to see is the end of a blind review process that is only in place to
>> protect the reviewers or even to allow people to exert petty vengeances or
>> advance themselves. My preference would be a completely open system in
>> which every individual takes responsibility for what they write (positive
>> or negative) about anyone else's work. If we could implement a completely
>> open process reviewers would have to be much more mindful about what they
>> write and would have to question themselves all the time: are my comments
>> constructive and can they lead to an improvement of this piece? Is *my own
>> argument* supported by the evidence? Am I being fair? Is there a conflict
>> of interests here? Are their personal feelings involved in my review? Will
>> the majority of my colleagues see my objections as fair?
>>
>> When I write a review (or an article, or even a message to this or other
>> distribution lists) I ask myself whether if this text were to be available
>> in future years I would feel OK about it. I want to be able to sign my
>> writings knowing that I have tried to do my best, but that my best is not
>> achieved by preventing others to take part in the community or by being
>> destructive or negative just for the sake of it.
>>
>> The academic world would be much better if we asked scholars to own their
>> words.
>>
>> Best,
>>
>> BB





More information about the Humanist mailing list