Judging Process

At the conclusion of the entry period Australian Event Awards Pty Ltd will compile the entries and prepare for online judging by the Industry Judging Panel.

The judging process is explained below. To download the judging process as a PDF, click here.

BACKGROUND

• Australian Event Awards Pty Ltd ABN 85 154 434 358 (Management Company) is an Australian Private Company incorporated in 2011 to manage, market and administer the Australian Event Awards (The Awards) – the pinnacle awards program for the Australian Events Industry.
• Entries into the The Awards for 2012 are open to events staged in Australia and achievements by Australians during the period from 1 July 2011 to 30 June 2012.
• Entries for the The Awards for 2012 will open on June 1, 2012 and close on 23 July 2012.
• This document outlines the Judging process which will be used for the 2012 Awards.
• This document will be reviewed prior to the 2013 Awards.

THE JUDGING PANEL

• The Awards are judged and awards made by the members of the Independent Industry Judging Panel (IIJP).
• The members of the Management Company appoint two co-chairs of the IIJP (Co-Chairs).
• The key role of the Co-Chairs is to appoint all other judges by unanimous agreement and to resolve issues with the judging process or questions from the judges
• The IIJP is divided into two sub-panels:
  • a Finalist Selection Panel (FSP) and
  • a Winner Selection Panel (WSP)
• Members of the IIJP can be appointed as a member of either sub-panel or both sub-panels by the Co-Chairs.

 

JUDGING PROCESS

Part 1: Finalist Selection

Scoring

• Each entry is assigned to at least two members of the FSP (FSP Judges) for scoring.
• Each FSP Judge checks the entries assigned to them and confirms that he/she does not have any real or perceived conflicts of interest in relation to the assigned entries.
• If a member of the FSP identifies any real or perceived conflicts of interest in relation to the assigned entries, the Judge must immediate notify the Management Company and the Management Company must reallocate the judging of the entry to a non-conflicted judge.
• Members of the FSP are provided with a scoring guide such as the one below for the purposes of calibration:

 

 

• Each FSP Judge assesses the entries assigned to them, scoring each entry against the criteria for its Category and providing comments to assist entrants with future entries.
• As a FSP Judge assesses further entries, they may wish to amend scores from previously-judged entries for consistency. This can be done at any time up until the close of the judging period.

 

Resolution of Judges Issues

• At any time during the scoring process, a member of the FSP may refer to the Co-Chairs any entry that the Judge considers:
  • may be inappropriate to enter its Category because it does not meet the published Entry Requirements; or
  • is not assessable in its Category for another reason nominated by the member.
• Upon referral of any entry above, the Co-Chairs may:
  • Offer the Entrant the opportunity to have the entry considered under a different Category nominated by either the Entrant or the Co-Chairs; or
  • Rule that the Entry is not able to be assessed in any Category.

 

Minimum Raw Score

• To progress to the next stage in the assessment of entries, an entry must achieve the minimum weighted average raw score.
• The weighted average raw scoreis calculated by:
  • adding the raw score from each judge for each criterion
  • multiplying by the weighting of each criterion
  • adding the resulting weighted score for each criterion
  • then dividing by the number of judges for that entry

resulting in a single weighted average raw score for that entry. This table is illustrated below:

 

 

• A minimum weighted average raw score is set by the Co-Chairs to establish a minimum quality standard that entries much achieve to be eligible to become a Finalist
• Entries that do not attain the minimum weighted average raw score are removed from the pool.

 

Score Moderation

• Scores for the remaining entries are moderated using an academic scaling algorithm to remove any bias that may have been introduced by the use of different judges across the pool.
• The result is a score out of 100 for each entry.

 

Automatic Finalists:

• The Entries that receive the top three scores in each category are Automatic Finalists.

 

Finalist Review:

• Each member of the FSP is advised of the Automatic Finalists and members are invited to nominate any entries that are not Automatic Finalists that they believe, against the published Category Criteria, should be considered by the Winner Selection Panel (WSP).
• Such a nomination must by supported by a statement from the nominating FSP Judge that:
  • gives reasons for the nomination against the Category Criteria; and
  • puts forward the position that, in relation to the Category Criteria, the entry cannot be reasonably distinguished from the Automatic Finalists on the basis of quality.
• To make such a nomination, the entry must:
  • Have been scored by the nominating judge
  • Meet the minimum raw score
• Entries nominated for review (Review Entries) are made available to a Review Panel made up of all judges that scored that Category (other than the nominating judge), and the Co-Chairs of Judges.
• In any Category where there is a Review Entry, Members of the Review Panel have the opportunity to re-read the Automatic Finalist entries together with any Review Entries. Members of the review panel may, in relation to each Review Entry:
  • support the nominating judge’s view that, in relation to the criteria, the entry should be considered by the WSP
  • reject the nominating judge’s view that, in relation to the criteria, the entry should be considered by the WSP
  • abstain from voting on the nomination (this is compulsory if the member of the Review Panel is conflicted in relation to any of the entries nominated for review or any of the Automatic Finalists for the Category)
• If any Review Entry receives:
  • at least one vote of support or nominations from two judges; and
  • no votes of rejection

the Review Entry will be become a Review Finalist for the Category.

 

Finalist Announcement

• At the end of the Finalist Review, the Automatic Finalists together with any Review Finalists for each Category will become the Category Finalists for the Category and the list of Category Finalists will be released.
• From this point, no distinction will be drawn between Automatic Finalists and Review Finalists.

  

Part 2: Winner Selection for all Categories Except Entertainer of the Year, Australian Event of the Year and Lifetime Achievement

Category Allocation

• Each member of the Winner Selection Panel (WSP) is allocated a number of Categories to review and coordinate.
• Each member of the WSP checks the Category Finalists in the allocated Categories and confirms that he/she does not have any real or perceived conflicts of interest in relation to the entries.
• If a member of the WSP identifies any real or perceived conflicts of interest in relation to any Category Finalists in an allocated Category, the Judge must immediately notify the Management Company and the Management Company must reallocate the Category to a non-conflicted member of the WSP.
• Once the confirmation above is made, the member of the WSP becomes the Category Coordinator for each category they’ve been assigned.
  • This means they are responsible for undertaking detailed review of the Finalist entries in each of their categories prior to Final Judging Day. They will represent the categories assigned to them and, for each category, lead the process described below.

 

Final Judging Day

• All members of the WSP attend the final judging day.
• Where a member of the WSP becomes unavailable for the final judging day, the Management Company must assign any Categories for which the unavailable member was Coordinator to another member of the WSP or to one of the Co-Chairs. This person then becomes the Category Coordinator for the assigned Category.

 

Sub-Panel Review

• The WSP breaks in into sub-panels.
• Each sub-panel is responsible for the in-depth review of every Finalist in the categories allocated to each member of the sub-panel
• For each Category represented in the sub-panel, the following process is followed:
  • The Category Coordinator introduces the Category Finalists
  • Each member of the sub-panel confirms that he/she does not have any real or perceived conflicts of interest in relation to any of the Category Finalists
  • Where a member of the sub-panel has any real or perceived conflicts of interest in relation to any of the Category Finalists, the sub-panel member must remove themselves from all discussion and voting regarding that Category
  • The Category Coordinator leads a discussion of the Category Finalists with regard to the Criteria.
  • Supplementary material submitted by entrants is also reviewed by the sub-panel at this stage. Entrants must limit supplementary material to what could be reasonably reviewed by the sub-panel within 5 minutes.
  • The sub-panel may contact a person nominated by the entrant if they feel further information or clarification is required.
  • The sub-panel decides on a recommendation for the Winner, based on the Category Criteria that will be presented to the full panel.

 

Full Panel Review

• The full panel convenes to consider the recommendations of each sub-panel and to vote on the Winner of each Category.
• The final consideration of each Category proceeds as follows:
  • The Category Coordinator introduces the Category Finalists
  • Each member of the full panel confirms that he/she does not have any real or perceived conflicts of interest in relation to the Category Finalists
  • Where a member of the sub-panel has any real or perceived conflicts of interest in relation to any of the Category Finalists, the sub-panel member must remove themselves from all discussion and voting regarding the Category
  • The Category Coordinator presents the sub-panel’s assessment of the Finalists, and the recommendation for the Winner of that Category
  • If requested by any member of the WSP or the Co-Chairs, discussion may take place, covering any questions or concerns that the WSP may have regarding the Category Finalists.
  • The Winning Entry is then selected from the Category Finalists by secret ballot in which each member of the WSP casts a single vote nominating their preference for the winner against the Category Criteria.
  • Votes are sealed in an envelope to be counted at a later date using the 'first past the post' method. The results are not made known to the members of the IIJP.
  • Should there be a tie, the entry with the highest weighted average raw score will be declared the winner.

 

Part 3: Winner Selection for Australian Event of the Year

One Step Process

• Only the winners of each ‘Best Event’ category are eligible for Australian Event of the Year
• On final judging day, all members of the WSP are presented with a ballot paper listing the finalists of all ‘Best Event’ categories, as the winners of each category are not released to the WSP at the time of voting
• Following discussion, each member votes for a minimum of 8 entries, numbering them in order of preference.
• Ballots are sealed in an envelope to be counted by the Management Company at a later date. The results will not be made known to the members of the IIJP. The winner is declared by preferential voting (as explained below).
• If the winner of the ballot did not also win its ‘Best Event’ category, then it is not eligible for Australian Event of the Year and will be disqualified. The entry with the next highest number of preferences (that has also won its Best Event category) will become the winner.

 

Part 4: Winner Selection for the Lifetime Achievement Award

One Step Process

• All members of the WSP will have had the opportunity to review entries via the web Portal prior to the judging day.
• On final judging day, a short discussion may take place covering any questions or concerns that any member of the WSP may have regarding the entries.
• The Co-Chairs of Judges ask for a vote by show of hands on the question, “can you separate the entrants to find one winner”. If a majority cannot, then a vote is held for “can you separate the entrants to find one or two winners.” Again, if a majority cannot then “one, two or three winners” and so on until a majority of the panel (50%+1) agree on how many awards will be made. During this process there should be no discussions over who would be the winner(s), the decision is how many winners there should be.
• Once the majority have expressed the number of winners that they can separate to, everyone votes knowing they are choosing (say) 2 winners (or whatever the vote in part c decides).
• Written votes are cast and nominees are numbered preferentially. If it is decided that two winners will be named then votes are numbered 1,1,2,3,4, etc. For three winners 1,1,1,2,3,4, etc.
• Ballots are sealed in an envelope to be counted by the Management Company at a later date. The winner is declared by preferential voting (as described below) for a single winner. For multiple winners, the following preferential voting method will be used to ensure equal weighting for each vote:

 

• Each judge’s ballot paper is identified by a unique code (not linked to their name so that anonymity is maintained)
• Multiple copies of the  ballot papers (two for two winners, three for three winners etc) are created for each judge code, each showing only the first preference from that judge
• The original ballot papers are set aside, and the copied vote papers are distributed amongst candidates.
•  The lowest candidate is eliminated the same way as explained below (under Preferential Voting)
• When a vote paper (showing only the first preference) is eliminated,  it is replaced by that judge’s original ballot paper (showing all remaining preferences) and the preferences are distributed as listed
• Should another vote paper from the same judge be eliminated, it is removed from the pool, because their remaining preferences on the original ballot paper are already elsewhere in the pool and therefore still carried through  
• Should the winner not wish to accept the Award (ie they did not have prior knowledge of the nomination and do not wish to be recognised in this way) then all votes will be recounted with that person’s preferences redistributed.

 

Part 5: Winner Selection for Entertainer of the Year

Audience Vote

• The Category Finalists will be determined by the process explained in Part 1
• The Finalists will not be discussed on Final Judging Day
• The Entertainer of the Year Finalists will perform at the Australian Event Awards Industry Night of Nights
• The winner will be determined by Audience Vote, both those in the room and watching the performances online via web streaming
• Measures will be in place to encourage those voting online to do so, only after watching all the performances

 

UNFORSEEN CIRCUMSTANCES

• Should a ruling be required on a matter not covered in this document, the Co-Chairs will make a unanimous ruling on that matter.
• The Co-Chairs of Judges may take advice from judges or others as they see fit.
• Generally, resolutions to unforeseen circumstances are documented in the form of a revision to this document ahead of the awards in the following year.

 

PREFERENTIAL VOTING

The Australian Event Awards use instant runoff voting (IRV) form of preferential voting to decide the outcome of the Lifetime Achievement Award and the Australian Event of the Year Award.

Instant Runoff Voting Explained

•  Each voter ranks the list of candidates in order of preference. The voter marks a '1' beside the most preferred candidate, a '2' beside the second-most preferred, and so forth, in ascending order. The mechanics of the process are the same regardless of how many candidates the voter ranks, and how many are left unranked.
• In the initial count, the first preference of each voter is counted and used to order the candidates. Each first preference counts as one vote for the appropriate candidate. Once all the first preferences are counted, if one candidate holds a majority, that candidate wins. Otherwise the candidate who holds the fewest first preferences is eliminated. If there is an exact tie for last place in numbers of votes, tie-breaking rules determine which candidate to eliminate (see below).
• The second preferences of ballots assigned to eliminated candidates are recounted and assigned to one of the remaining candidates, potentially producing a different candidate ordering. The process then repeats until one candidate achieves a majority. Ballots that 'exhaust' all their preferences (all its ranked candidates are eliminated) are discarded.

 

Tie-Breaking Rules

• In the event that two or more candidates have the lowest number of votes, the following procedures take place:
  • If the tied candidates combined have fewer votes than the next highest candidate, the entire tied set are eliminated at once. Their preferences are then redistributed.
  • If  the tied candidates combined do not have fewer votes than the next highest candidate, then the candidate with the least number of first preference votes is eliminated. If there is a tie in first preference votes then the candidate with the least number of second preferences is eliminated. This process continues until eighth preferences have been counted
• If a tie still exists, the candidate with the lowest moderated FSP score is eliminated

 

APPEALS

• The Awards does not have an appeals process. All decisions by the IIJP are final but the IIJP welcomes feedback and encourages constructive criticism aimed at improving The Awards.