Dear Georg, Anne and I had a long meeting on Friday and we looked at a bunch of submissions around the threshold. I don't have our notes with me, but if we didn't include those papers that you mention, that's probably because we performed our own oral "meta-review"... and we put on the list only those that required an extra pair of eyes. Anne and I are meeting next tomorrow to handle changes to submission types (i.e. paper to poster etc.) and will be in touch about those separately. All best, Toma -- Belgrade Center for Digital Humanities http://humanistika.org

20.02.2023, в 12:10, Georg Vogeler написал(а):

Dear Anne, dear Toma, dear colleagues,

this sounds like a good procedure. I wonder hot to deal with proposals which reviewers suggested to accept in a different format (e.g. 506, 591, 436, 563, 523).

And I wonder, if we should include proposals with a wide range but close below the threshold into our considerations (e.g. 659, 669, 656, 131).

Any thoughts about this?

Best

Georg

Am 20.02.2023 um 10:58 schrieb Anne Baillot:

...

Dear members of the PC, in the next stage of preparations for DH2023 – finalizing the list of accepted submissions – the role of the PC will be essential. We will need your help with some specific tasks, so please read on. To access all the submissions and the reviews, you should log in to ConfTool and then click on “Manage Submissions and Reviews” > “Online Forum for the Program Committee”. Quantitative cutoffs for acceptance Keeping in mind that we would like to have around 6 parallel sessions in the main program, we are currently proposing to accept the following submissions: * Long presentations: submissions with a grade equal or above 80,5 * Short presentations: submissions with a grade equal or above 76,7 (except one rejection for an author who submitted only their CV and didn’t provide an actual abstract in the response phase although he was asked to) * Panels: submissions with a grade equal or above 85,7 * Workshops: submissions with a grade equal or above 85 This would give us enough content to fill around 6 parallel sessions. Grading spans This purely quantitative approach, however, seems to us to be only half satisfactory considering the very wide spans there can be between two reviews for the same paper (you can see the span in Conftool in the second column from the left) - see picture below: For some submissions whose score is within the ranges mentioned above or slightly below and that have a particularly big span between reviews, we would like to ask you to provide meta-reviews. Meta-reviews Providing a meta-review consists in having a look at the abstract and at the reviews and suggesting a decision (accept/not accept) with one sentence of explanation. The goal is not that you do an additional review, but that you provide an assessment in these cases where the quality of the submission does not seem to be unequivocal. You can do this “meta-review” directly in the forum - neither authors nor reviewers will have access to it, and your judgment will remain within the PC. When in the forum, the right column leads to a “comment” area in which you can both provide an assessment and a justification for it in a textbox. We have prepared a sheet with a list of submission IDs that need such a meta-review and would be extremely grateful if you could: 1. put your name next to 1-2 submissions; 2. do the meta-reviews for your self-assigned submissions by March 1st. This should not be as work-intensive as our reviewing effort in December. Self-assignments should be random and on a first-come, first-served basis. We are not looking for specific expertise, just common sense and an objective, external point of view in regard to the quality of the submissions, the available reviews and, in some cases, authors’ responses. Here is the link to the list: https://docs.google.com/spreadsheets/d/1ra35OvOzSIo1CsBygq5CdHfy-Qt2LuizsYIF...https://docs.google.com/spreadsheets/d/1ra35OvOzSIo1CsBygq5CdHfy-Qt2LuizsYIF... This concerns all submission types but posters, which we will deal with and inform you about separately. We thank you in advance for your help with this work step too! - and will keep you posted on the overall progress of the selection process. All best, Anne and Toma -- Anne Baillot Professeure en Etudes Germaniques Faculté de Lettres, Langues et Sciences Humaines Département d'allemand Université du Mans Avenue Olivier Messiaen F-72085 LE MANS Cedex 9 CV et publications:https://cv.archives-ouvertes.fr/annebaillot Sur twitter: @AnneBaillot _______________________________________________ PC2023 mailing list PC2023@lists.digitalhumanities.org http://lists.lists.digitalhumanities.org/mailman/listinfo/pc2023

-- Prof. Dr. Georg Vogeler

Professur für Digital Humanities - Zentrum für Informationsmodellierung Universität Graz A-8010 Graz | Elisabethstraße 59/III Tel. +43 316 380 8033 <http://informationsmodellierung.uni-graz.at http://informationsmodellierung.uni-graz.at/> - <http://gams.uni-graz.at http://gams.uni-graz.at/> https://online.uni-graz.at/kfu_online/wbForschungsportal.cbShowPortal?pPerso...

Institut für Dokumentologie und Editorik e.V. <http://www.i-d-e.de http://www.i-d-e.de/> International Center for Archival Research ICARus <http://www.icar-us.eu http://www.icar-us.eu/> _______________________________________________ PC2023 mailing list PC2023@lists.digitalhumanities.org mailto:PC2023@lists.digitalhumanities.org http://lists.lists.digitalhumanities.org/mailman/listinfo/pc2023

Hello all, Thanks again for all of your work getting us to this point in the conference planning process. I know it's intense. In addition to Richard’s observation about wide-ranging review scores, I’d like to offer a statistical perspective. My calculations could be off, but based on the number of submissions received in ConfTool and the cutoff proposed, the rate of acceptance appears to look like this: Long presentations: 78 out of 234 (33%) Short presentations: 100 out of 292 (34%) Panels: 14 out of 43 (32.5%) Workshops: 15 out of 33 (45%) By comparison, an average of acceptance rates from the 2015-2019 conferences looks like this: Long presentations: 42% Short presentations: 46% Panels: 76% Workshops: 64% Note that in some ways these acceptance rates are complicated because of the location and venue size of each conference and the options for submission types (in some cases ‘panels’ included other formats, like ‘forums’). Certainly it is the prerogative of the PC to set the standard for the academic program, and that is a very complex calculus. It is worth considering, though, that there is a direct connection between conference acceptance and conference attendance (especially now, with funding so scarce). In the bid that Georg and Walter submitted, I believe there was room for 8 parallel sessions. Is that still the case? Is there room to expand the number of concurrent sessions, or is the maximum number of available rooms now 6? I ask to better understand if the cap on parallel sessions is a logistical one. Thanks again for all that you are doing! and best wishes, Diane On Mon, Feb 20, 2023 at 7:09 PM Richard Tsai wrote:

Dear Anne, Toma, pc colleagues,

I wanted to share my thoughts on the inclusion of papers with only one extreme low score in the conference. In my view, even though these papers may have received one extreme low score, they may still have other strengths and valuable contributions to the field that make them worth considering for inclusion. I believe that it is important to evaluate each submission holistically and not base our decisions solely on one review. Thank you for your consideration.

Best,

Richard

On Mon, Feb 20, 2023 at 9:25 PM Toma Tasovac wrote:

...

Dear Georg,

Anne and I had a long meeting on Friday and we looked at a bunch of submissions around the threshold. I don't have our notes with me, but if we didn't include those papers that you mention, that's probably because we performed our own oral "meta-review"... and we put on the list only those that required an extra pair of eyes.

Anne and I are meeting next tomorrow to handle changes to submission types (i.e. paper to poster etc.) and will be in touch about those separately.

All best, Toma

-- ====================================================== Richard Tzong-Han Tsai

The President of the Board of Directors of the Taiwanese Association for Digital Humanities (TADH) Third Term CEO, Center for GIS, RCHSS, Academia Sinica Taiwan Professor, Department of Computer Science and Information Engineering, National Central University Tel.: +886-3-4227151 ext. 35203 Fax.: +886-3-4222681 Email: thtsai@g.ncu.edu.tw _______________________________________________ PC2023 mailing list PC2023@lists.digitalhumanities.org http://lists.lists.digitalhumanities.org/mailman/listinfo/pc2023

-- Diane Jakacki, Ph.D. Digital Scholarship Coordinator Affiliate Faculty in Comparative & Digital Humanities Bucknell University diane.jakacki@bucknell.edu (she/her) Fulbright Canada Research Chair in Digital Humanities, 2022-3 Principal Investigator, LAB Cooperative and REED London Online Executive Board Chair-Elect, ADHO Chair, TEI-C Executive Board

Short practical notice: We could accommodate up to 9 parallel tracks: https://docs.google.com/spreadsheets/d/1NNlnffSBvl-prg1RhslRPwVx8saUAm7lNXS5... Which does not mean that we have to: scholarly quality is certainly the main criteria. Best Georg Am 21.02.2023 um 02:40 schrieb Diane Jakacki:

Hello all,

Thanks again for all of your work getting us to this point in the conference planning process. I know it's intense.

In addition to Richard’s observation about wide-ranging review scores, I’d like to offer a statistical perspective. My calculations could be off, but based on the number of submissions received in ConfTool and the cutoff proposed, the rate of acceptance appears to look like this:

Long presentations: 78 out of 234 (33%)

Short presentations: 100 out of 292 (34%)

Panels: 14 out of 43 (32.5%)

Workshops: 15 out of 33 (45%)

By comparison, an average of acceptance rates from the 2015-2019 conferences looks like this:

Long presentations: 42%

Short presentations: 46%

Panels: 76%

Workshops: 64%

Note that in some ways these acceptance rates are complicated because of the location and venue size of each conference and the options for submission types (in some cases ‘panels’ included other formats, like ‘forums’). Certainly it is the prerogative of the PC to set the standard for the academic program, and that is a very complex calculus. It is worth considering, though, that there is a direct connection between conference acceptance and conference attendance (especially now, with funding so scarce). In the bid that Georg and Walter submitted, I believe there was room for 8 parallel sessions. Is that still the case? Is there room to expand the number of concurrent sessions, or is the maximum number of available rooms now 6? I ask to better understand if the cap on parallel sessions is a logistical one.

Thanks again for all that you are doing! and best wishes,

Diane

On Mon, Feb 20, 2023 at 7:09 PM Richard Tsai wrote:

...

Dear Anne, Toma, pc colleagues,

I wanted to share my thoughts on the inclusion of papers with only one extreme low score in the conference. In my view, even though these papers may have received one extreme low score, they may still have other strengths and valuable contributions to the field that make them worth considering for inclusion. I believe that it is important to evaluate each submission holistically and not base our decisions solely on one review. Thank you for your consideration.

Best,

Richard

On Mon, Feb 20, 2023 at 9:25 PM Toma Tasovac wrote:

...

Dear Georg,

Anne and I had a long meeting on Friday and we looked at a bunch of submissions around the threshold. I don't have our notes with me, but if we didn't include those papers that you mention, that's probably because we performed our own oral "meta-review"... and we put on the list only those that required an extra pair of eyes.

Anne and I are meeting next tomorrow to handle changes to submission types (i.e. paper to poster etc.) and will be in touch about those separately.

All best, Toma

-- ====================================================== Richard Tzong-Han Tsai

The President of the Board of Directors of the Taiwanese Association for Digital Humanities (TADH) Third Term CEO, Center for GIS, RCHSS, Academia Sinica Taiwan Professor, Department of Computer Science and Information Engineering, National Central University Tel.: +886-3-4227151 ext. 35203 Fax.: +886-3-4222681 Email: thtsai@g.ncu.edu.tw _______________________________________________ PC2023 mailing list PC2023@lists.digitalhumanities.org http://lists.lists.digitalhumanities.org/mailman/listinfo/pc2023

_______________________________________________ PC2023 mailing list PC2023@lists.digitalhumanities.org http://lists.lists.digitalhumanities.org/mailman/listinfo/pc2023

-- Prof. Dr. Georg Vogeler Professur für Digital Humanities - Zentrum für Informationsmodellierung Universität Graz A-8010 Graz | Elisabethstraße 59/III Tel. +43 316 380 8033 http://informationsmodellierung.uni-graz.at - http://gams.uni-graz.at https://online.uni-graz.at/kfu_online/wbForschungsportal.cbShowPortal?pPerso... Institut für Dokumentologie und Editorik e.V. http://www.i-d-e.de International Center for Archival Research ICARus http://www.icar-us.eu

Dear members of the PC, We have learned today that there will be no Buza Prize Award at this year’s conference. Since that frees up a late (6 pm) slot in the conference schedule, this – while meaning that we are sadly missing out on an opportunity to celebrate the achievements of one of our colleagues – presents us with the opportunity to accept additional contributions by moving the Poster Presentation session originally scheduled for Wednesday afternoon (4-5.30 pm) to the time-slot originally reserved for the Buza Prize lecture at 6 pm (and possibly combining it with a drinks reception). Upon consultation with Anne and Toma, we would therefor propose to accept 3 additional long papers, 20 additional short papers and 2 additional panels (filling up 7 additional sessions from 4-5:30 pm on Wednesday), so the updated numbers would be: * Long Papers: 108 submissions accepted with a cut-off at 77,3 points (45,96%) * Short Papers: 125 accepted with cut-off at 73,3 (42,81) * Posters: 112 accepted with cut-off at 64,0 (71,79%) * Panels: 16 accepted with cut-off at 85,0 (37,21%) * Workshops: 25 accepted with cut-off at 79,7 (75,75%) * The overall acceptance rate (with workshops) is 50,86%; without workshops 49,72% Please let us know if you have any objections to this by tomorrow at 10 am CET, because we want to send the acceptance confirmations tomorrow at noon at the latest. Kind regards, Walter (on behalf of Anne and Toma)

Dear Anne, Toma, and the PC, Anne and Toma, thank you for all of the hard work you’ve been doing. I know you have made the decision to accept proposals numerically, but I urge you not to claim that this is “fair”: instead, we are delegating the work of creating the conference program to the ConfTool algorithm that assigns reviewers and to reviewers who have, as you noted, wildly variant expectations for what constitutes a good score for the conference. The Conference Protocolshttps://adho.org/conference/conference-protocol/ note that “the PC oversees the academic program”—but this year, the program will not be shaped by the PC. Basing acceptance entirely on numbers, furthermore, does not uphold ADHO’s commitment to Equity, Diversity, and Inclusionhttps://adho.org/inclusion/ because, for instance, the program committee can’t advocate for inclusion of a panel on indigenous data sovereignty, for instance, that might have been lowly ranked because of a reviewer’s unconscious bias. (For DH2020, we had instances of entire fields of studies being dismissed by reviewers that resulted in low rankings.) Moreover, numerical acceptance means that, in future years, we might see more of the “100 with no explanation” phenomenon that you point to (or its opposite), because people will think that an extreme number is more useful than any meaningful feedback on a proposal. I understand that the PC Chairs have made this decision already. If you choose to be transparent about the conference process, however, I would avoid rhetoric about “fairness.” I know organizing this conference is a monumental task and that reading the hundreds of submissions and thousands of reviews takes time, so again, thank you. Laura Dr. Laura Estill Associate Professor of English St. Francis Xavier University Tier 2 Canada Research Chair in Digital Humanities From: PC2023 on behalf of Michael Eberle Sinatra Date: Tuesday, March 7, 2023 at 12:08 PM To: pc2023@lists.digitalhumanities.org Subject: Re: [Pc2023] Final selection of submissions Dear Anne and Toma and all the hard working members of the PC, on behalf of the CCC, I would like to thank you for all your intellectual contribution and time devoted to this crucial component of our annual conference. I look forward to what promises to be a wonderful event and hope to see as many of you as possible in person in Graz! Best wishes, Michael Michael E. Sinatrahttps://www.michaelsinatra.org | Professeur titulaire et directeur Centre de recherche interuniversitaire sur les humanités numériques (CRIHNhttp://crihn.org) Responsable du Groupe de Recherche sur les Éditions critiques en contexte Numérique (GRENhttps://gren.openum.ca) Responsable de l’option doctorale en Humanités numériqueshttps://llm.umontreal.ca/programmes-cours/humanites-numeriques/doctorat-en-l..., du microprogrammehttps://admission.umontreal.ca/programmes/microprogramme-de-2e-cycle-en-huma... et de la mineurehttps://llm.umontreal.ca/programmes-cours/humanites-numeriques/mineure-en-hu... en HN, Université de Montréal Co-Chair centerNet, An international network of digital humanities centers On Mar 7, 2023, at 5:00 AM, Anne Baillot wrote: Dear members of the PC, based on your meta-reviews (for which we are very grateful), our ongoing discussions with the LOs regarding the location and availability of rooms, as well as further discussions on quality insurance, we have decided to shift our cut-off in several submission categories. This will make it possible to accommodate a wider range of papers: * Long Papers: 105 submissions accepted with a cut-off at 78,3 points (44,68%) * Short Papers: 105 accepted with cut-off at 76,7 (35,96) * Posters: 112 accepted with cut-off at 64,0 (71,79%) * Panels: 14 accepted with cut-off at 85,7 (32,5%) * Workshops: 25 accepted with cut-off at 79,7 (75,75%) The overall acceptance rate (with workshops) is 47,56%; without workshops 46,28% This will translate into 7 parallel sessions during the conference. Accepting more submissions than that would get us into the territory where we wouldn’t feel comfortable with the kind of quality level that we’d like to see at DH2023. While this is, in the end, a purely numerical approach, we have decided to opt for it after long discussions on the fairness of the results. We balanced the flaws of the review process for quite a while (wide spans, extremely harsh reviews, 100 point reviews with no explanation of the grade) etc. but couldn’t find a qualitative solution that would be realistically accomplishable for the PC without introducing new challenges: for instance, cherry-picking submissions solely on the basis of a wide span for additional consideration would privilege those submissions over similarly graded submissions with a smaller grade span. We have gathered a few suggestions on how to improve the review quality; we will pass them on to the next PC. This is a work in progress into which the community can (and will) certainly be able to contribute to. We have all learned a lot from this year’s take on Open Identities in the review process… one thing that we have learned for sure is that it does not necessarily make the PC’s life easier. We will be communicating the selection results to the authors via conftool on Wednesday (March 8th). We hope the final selection meets your approval and would like to thank you again for your efficient support in the process! We are currently in touch with the keynotes to discuss the topics of their talks and will let you know about that as soon as they are final. We will also get back to you regarding the best reviewer award in a couple of weeks. All best, Anne and Toma _______________________________________________ PC2023 mailing list PC2023@lists.digitalhumanities.org http://lists.lists.digitalhumanities.org/mailman/listinfo/pc2023