First Workshop on
ROC Analysis in AI
Valencia, Spain, 22 August, 2004

hold within ECAI’2004, the European Conference on Artificial Intelligence.

Followed by ROCML'05 and ROCML'06.

Brief description

Receiver Operating Characteristic Analysis (ROC Analysis) is related in a direct and natural way to cost/benefit analysis of diagnostic decision making. Widely used in medicine for many decades, it has been introduced relatively recently in some areas of artificial intelligence: machine learning, multiagent systems, intelligent decision support and expert systems. In this context, ROC analysis provides tools to select possibly optimal models and to discard suboptimal ones independently from (and prior to specifying) the cost context or the class distribution. Furthermore, the Area Under the ROC Curve (AUC) has been shown to be a better evaluation measure than accuracy in contexts with variable misclassification costs and/or imbalanced datasets. AUC is also the standard measure when using classifiers to rank examples, and, hence, is used in applications where ranking is crucial, such as campaign design, model combination, collaboration strategies, and co-learning.

Nevertheless, there are many open questions and some limitations that hamper a broader use and applicability of ROC analysis. Its connections with other evaluation measures is not yet clarified completely, its incorporation in decision support and expert systems technology just envisaged, its use for improving the decisions of (communities of) intelligent agents unexplored, and its use in data mining still below its full potential. Among the limitations of ROC analysis, an important one, despite some recent progress, is its possible but difficult extension for more than two classes. 

One of the main goals of the workshop is to foster the cross-fertilisation of ideas and applications with related areas in artificial intelligence. Consequently, the presentations and discussions will be open to a broad list of topics (not exhaustive):

• Adaptation of classical learning methods for improving AUC instead of accuracy.
• Agent selection/ranking in multiagent systems.
• Alternative ROC representations.
• Alternatives to AUC measures (such as AUC*).
• Analysis of the performance of learners or decision systems based on AUC.
• Applications of ROC Analysis in Expert Systems.
• Classifier Evaluation.
• Co-learning (collaborative learning) and ROC Analysis.
• Consensus and Collaborative/Distributed Decision Making.
• Constraint Satisfaction Methods for ROC Analysis.
• Cost-sensitive Learning.
• Decision Networks and Probabilistic Reasoning with ROC Analysis.
• Expert Systems, Knowledge Bases and ROC Analysis.
• Inductive Logic Programming and ROC Analysis.
• Multi-class ROC on top of binary ROC.
• Oversampling and ROC analysis.
• Precision & Recall measures in Information Retrieval.
• Ranking Actions. Applications in Planning and Robotics with variable contexts.
• Reinforcement Learning and ROC analysis.
• ROC Analysis for Model Building and Modification.
• ROC Analysis for Descriptive Data Mining (association rules, subgroup discovery).
• Soft classifiers and Probability Estimators.
• Software Packages and Efficient Implementations (Convex Hull).
• Use of ROC analysis and measures as fitness and selection criteria in evolutionary techniques.
• Working with Imbalanced datasets.

This is a one-day workshop (see schedule), with sessions consisting of short paper presentations, devoting an important share of time to informal discussion and interaction between the participants. The workshop will be closed with an open discussion about more promising open problems and research areas of ROC analysis, continuation of the workshop, future related events, etc. 

Papers will be selected by the program committee according to the quality of the submission and its relevance to the workshop topics. All accepted papers will be gathered in printed form and distributed to registered attendees as workshop notes.

The publication of a selected set of papers for a special volume or a journal issue is considered, but this will depend on the success and overall results of the workshop.

Accepted Papers 

List of accepted papers

Schedule

Important Dates

• Call for papers: 15 January 2004
• Deadline for the workshop papers: 26 April 2004 
• Notification of acceptance: 21 May 2004
• Camera Ready copies: 4 June 2004
• Workshop dates: 22 August 2004.
  

• Hendrik Blockeel, K.U.Leuven, Belgium.
• Stephan Dreiseitl FHS Hagenberg, Austria.
  
• Tom Fawcett, HP Labs, Palo Alto, CA, USA.
  
• Cèsar Ferri, Technical University of Valencia, Spain.
• Peter Flach, University of Bristol, UK.
• Johannes Fürnkranz, TU Darmstadt, Germany .
• José Hernández-Orallo, Technical University of Valencia, Spain.
• Nicolas Lachiche, University of Strasbourg, France.
• Charles Ling, University of Western Ontario, London, Ontario, Canada
• Maarten van Someren, University of Amsterdam, The Netherlands.
• Francesco Tortorella, University of Cassino, Italy.

Workshop Organizing Committee

• Cèsar Ferri, Technical University of Valencia, Spain.
• Peter Flach, University of Bristol, UK.
• José Hernández-Orallo, Technical University of Valencia, Spain (contact person) .
• Nicolas Lachiche, University of Strasbourg, France.

Submission Guidelines

Potential participants are invited to submit short papers (no longer than 10 pages), which may be in the form of a technical paper, a position paper (e.g. highlighting open problems or new applications of ROC analysis), an overview of their research or a software demonstration.

Authors should submit their papers electronically (PDF or PS format) to the contact person (jorallo@dsic.upv.es). It is recommended to submit papers using the final camera-ready ECAI 2004 conference paper style, including author names.

Those wishing to attend the meeting without submitting a paper should send a short statement of interest to the contact person describing their work or interest in the area.

Some relevants links (introduction to ROC analysis, software, etc.)

• The ROC Convex Hull: software and papers, mantained by Tom Fawcett.
• Receiver Operating Characteristic (ROC) Literature Research: ROC papers and articles, mantained by Kelly Zou.
  
• ROC Analysis Software.
• JROCFITi: A ROC Analysis Applet.
• ROCOn: A tool for visualising ROC graphs.
• Very introductory slides on ROC Analysis.
• Tutorial on "The Many Faces of ROC Analysis in Machine Learning" by Peter Flach