The full conference program with a detailed schedule will be released at a later date. In the meantime, information on tutorials and other events will be published here progressively. Please follow our social media or RSS feed for updates.

Tutorial Day: Monday 30 November

ALTA 2026 will host two tutorials on the first day of the workshop. Details of both tutorials are below.

Morning tutorial

Title: Transfer Learning in Clinical NLP

Presenters: Distinguished Professor Karin Verspoor, Dr David Martinez Iraola, Wei Han (RMIT University)

Abstract: Machine learning models often require large, labelled datasets, making high-quality annotation costly and time-consuming. Transfer learning addresses this challenge by reusing knowledge learned from related tasks, datasets, or models to improve learning in low-resource settings. In NLP, transfer learning enables models to reuse linguistic knowledge across tasks, but differences in domains, text types, and data distributions still limit model generalisation. These challenges are particularly significant in clinical NLP, where annotation is labour-intensive and requires specialist expertise. Clinical text also varies across institutions, specialities, document types, and patient groups. Differences in terminology, abbreviations, and writing styles further reduce model generalisation. In this tutorial, we will first introduce transfer learning and its development in NLP, together with representative transfer learning methods. After that we will talk about the transfer shifts between source and target clinical settings, why transfer is important in clinical NLP, and the main challenges for its application. Hands-on exercises with accessible clinical datasets will allow participants to implement and evaluate several transfer approaches. Finally, the tutorial will discuss clinical-specific transfer learning challenges and future research directions.

Tutorial Overview:

  1. Session 1: Foundations and Methods of Clinical NLP Transfer Learning (1h)
    • Part 1: Transfer Learning in NLP: Introduces transfer learning, and explains how transfer has developed in NLP. Introduces representative methods used for transfer learning in NLP.
    • Part 2: Transfer Learning Scenarios in Clinical NLP: Explains why transfer learning is specifically valuable for clinical NLP, and gives an overview of transfer scenarios from source to target in clinical settings.
  2. Session 2: Practical Clinical Transfer Learning (1.5h)
    • Part 3: Hands-on Clinical Transfer Learning Exercise: Applies representative transfer learning methods to real-world clinical datasets and evaluates their effectiveness under realistic source-to-target shifts.
    • Part 4: Challenges and Future Directions

Pre-requisite: Basic knowledge of machine learning and NLP will help participants follow the technical content. No prior experience in clinical NLP or transfer learning is required. Some hands-on exercises will use datasets hosted on PhysioNet, which requires prior credentialed access through their website. There will also be publicly available datasets that can be accessed without credentials.

Afternoon tutorial

Title: Evaluating AI Agents with LLM-as-a-Judge

Presenters: Parvin Zarei Eskikand (Senior Data Scientist, SEEK), Terrence Szymanski (Head of Generative AI, SEEK), Sara Mirabi (Deakin University), Yelaman Abdullin (Macquarie University)

Abstract: AI agents extend large language models by planning, using tools, and interacting with users and other systems. These capabilities make agents useful across a growing range of applications, but also make their behaviour harder to evaluate. Unlike conventional NLP systems, an agent may produce a plausible final answer while choosing the wrong tool, supplying incorrect parameters, or following an unreliable sequence of steps. Evaluating agentic systems therefore requires methods that consider both outcomes and behaviour. Deterministic checks can verify properties such as tool selection, parameters, and task completion, while model-based evaluation can assess qualities that are difficult to capture with fixed rules. LLM-as-a-judge offers a flexible approach in which a language model applies an evaluation rubric to an agent’s outputs or trajectory, but its reliability depends on the criteria, prompts, scales, and test scenarios used. In this tutorial, we will introduce AI agents and the distinctive challenges involved in evaluating them. We will examine deterministic and non-deterministic evaluation approaches, discuss when LLM judges are useful, and show how evaluation criteria and rubrics can be designed for different domains and agent scenarios. We will also discuss how agent evaluation is approached in industry, including how prompts and evaluation assets are organised and versioned, different ways teams manage evaluation alongside application development, and how evaluation fits into broader testing, deployment, monitoring, and review workflows. Examples will include tool-using and conversational agents, drawing on perspectives from current research and real-world industry practice. Hands-on exercises will allow participants to design an evaluation rubric, create an LLM-as-a-judge prompt, and apply it to outputs from a practical agent scenario. Participants will run the judge across test cases, compare results, and explore considerations such as reliability, robustness, and cost. The tutorial will conclude with limitations, practical guidance, industry lessons, and open research directions for trustworthy evaluation of agentic systems.

Tutorial Overview:

  1. Session 1: Foundations of AI Agent Evaluation (1.5h)
    • Part 1: AI Agents and Evaluation Challenges: Introduces AI agents, common application scenarios and tool use, and explains why evaluating agentic behaviour requires more than assessing the final response. We will also introduce the practical evaluation lifecycle used in industry: where prompts, test cases, datasets, rubrics, and evaluation code live; different approaches to managing these assets alongside application code or through external tools; versioning and reproducibility; and how evaluation supports development, testing, deployment, and production monitoring.
    • Part 2: LLM-as-a-Judge Methods: Compares deterministic and non-deterministic evaluation, introduces LLM-as-a-judge, and discusses criteria, rubrics, rating scales, prompting approaches, and domain-specific considerations. We will connect these methods to industry practice, including how teams choose between deterministic checks, LLM judges, and human review; how offline evaluation relates to online monitoring; and practical trade-offs between quality, latency, reliability, and cost.
  2. Session 2: Practical Evaluation of AI Agents (1.5h)
    • Part 3: Hands-on Agent Evaluation Exercise: Participants develop a rubric and LLM judge for a practical tool-using agent scenario, run the judge over test cases, and analyse outcomes including tool choice, parameters, and response quality.
    • Part 4: Reliability, Limitations, and Future Directions: Examines judge consistency, robustness, cost, and common failure modes, before discussing good practice, lessons from real-world deployment, and future research in trustworthy agent evaluation.

Pre-requisite: Basic familiarity with machine learning, NLP, and large language models will help participants follow the technical content. No prior experience with AI agents or LLM-as-a-judge is required. Participants should bring a laptop for the hands-on exercises. Access to the required LLM will be arranged and confirmed before the tutorial.

Main Conference: 1–2 December

The main conference program will be announced at a later date.