Submission and Evaluation

Submission

The Cup participants shall submit their programs as Java source code, creating a single executable, which can be called with filename(s) for the input dataset, and a filename for writing the program results into a file. Your program must extend the base classes defined in COMSET. No modifications to the base classes are allowed. You may only use the libraries provided by COMSET and Java SE. If you think there is a specific library that is essential for your submission, please write an email to the mailing list for approval. Libraries should be open source. Using libraries or tools other than those provided by COMSET and Java SE or not previously approved will lead to immediate disqualification. COMSET is licensed under an MIT license. The participants of GISCUP'20 shall license their CUP submissions under an MIT license as well.

Submission Portal

Participants shall submit their solution as a single .zip file via EasyChair:

https://easychair.org/conferences/?conf=giscup20

The submission must contain:

  • The original source code and all dependencies (submission of the source code is mandatory to ensure originality of the submitted work).
  • A readme.txt file containing information on how to compile and run the submitted code. You may include a brief description of the main idea behind your submission.
  • A contact.txt file containing the full name, email addresses, and affiliations of the authors.

Evaluation System

The submissions will be evaluated on an up to date server-grade machine. You can expect a multicore CPU and enough RAM to hold the dataset in memory.

Allowed libraries / tools

  • COMSET (mandatory)
  • Java SE

Additional libraries can be requested by mailing the contest chairs.

Evaluation Criteria

We will randomly choose multiple days from the time period covered by the training dataset. The COMSET simulator will be run for each of the chosen days independently starting from 8am until 9pm. The average of the outcomes of all runs will be used for evaluation. As an example for evaluating the agent utilization scenario, we will use the average search time as the primary evaluation metric. For breaking ties, we will first look at the average wait time, and then the expiration percentage (see section 2 for the definition of these metrics). Finally, we break ties on code stability, quality, and readability.

CUP Forum

Challenges similar to the one posted here often lead to questions concerning various details of the challenge and evaluation rules. We will use a public Google Group for communicating any questions that can be answered as well as the corresponding answers to all interested people. Therefore, please try to pose questions only in a form that does not leak too much of your ideas.

https://groups.google.com/forum/#!forum/giscup-2020

We will also post notifications for important updates to the challenge to this group.