[Humanist] 27.384 help: graders to evaluate system outputs

Humanist Discussion Group willard.mccarty at mccarty.org.uk
Sat Sep 28 09:02:10 CEST 2013


                 Humanist Discussion Group, Vol. 27, No. 384.
            Department of Digital Humanities, King's College London
                       www.digitalhumanities.org/humanist
                Submit to: humanist at lists.digitalhumanities.org



        Date: Fri, 27 Sep 2013 20:19:00 +0000
        From: "Downie, J Stephen" <jdownie at illinois.edu>
        Subject: Be a Hero: MIREX 2013 AMS and SMS graders needed (time sensitive)

Dear Colleagues:

The Music Information Retrieval Evaluation eXchange (MIREX) 2013 Audio Music Similarity (AMS) and Symbolic Melodic Similarity (SMS) tasks both require graders to evaluate system outputs.

For more information about MIREX, please see:
http://www.music-ir.org/mirex/wiki/MIREX_HOME

We will need two sets of graders, one for each task.

For AMS, we need *50* graders (50 queries, 1 query list per grader).

For SMS, we need *6* graders as there are only 6 query lists to be evaluated.

We are under the usual time constraints this year. ISMIR 2013 begins 4 November. We need to have all the MIREX results calculated and posted by our 21 October target date (fingers crossed)!

We hope to open the Evalutron 6000 (E6K) v.2 grading system by Tuesday, 8 October. To meet our 21 October (MIREX results release date) goal, we must have all the AMS and SMS similarity grades entered into the E6K by Sunday, 17 October. 

YES, THIS GIVES US LESS THAN TWO WEEKS OF EVALUATION!

So, if you are kind enough to sign up to be a grader, PLEASE understand that we REALLY need you complete your assigned grading by 17 October.

If you are a SMS or AMS participant, we ask that you do what you can to encourage adults over 18 years of age to be graders. If you are an AMS or SMS participant, please chip in and be a grader.

We have set up two sign up forms.

The SMS sign up form is found at:
http://bit.ly/1fNJD1s

The AMS sign up form is found at:
http://bit.ly/1aw16ra

Once we get our lists put together, we will send out instructions to the graders on what they need to do and where to go to do the online evaluations.

If you have any questions, please contact <mirex at imirsel.org> or <jdownie at illinois.edu>

Thank you very much everybody.

Cheers, 
Stephen

**********************************************************
   "Research funding makes the world a better place"
**********************************************************
J. Stephen Downie, PhD
Associate Dean for Research
Professor
Graduate School of Library and Information Science
University of Illinois at Urbana-Champaign
[Vox/Voicemail] (217) 649-3839





More information about the Humanist mailing list