Fairness and Explainability in Large Language Models: NHR SW Second LLM Workshop

We cordially invite you to our Introduction to Fairness and Explainability in Large Language Models NHR SW Second LLM workshop organized by Saarland University, with Prof. Anne Lauscher from the University of Hamburg, as our guest speaker. This hands-on workshop will take place at Saarland University, Campus Saarbrücken on October 31st, 11:00 - 17:00.

This course is designed for researchers and practitioners looking to deepen their understanding of fairness, inclusivity, and explainability in the context of LLMs, providing both theoretical insights and hands-on experience. In a series of talks we will discuss the role of sociocultural factors in LLMs, highlighting the need for more inclusive and fair models, address evaluating LLMs, emphasizing multidimensional goals like robustness and fairness, and present insights into language adapters, explaining how they adapt models to new languages while preserving core structures. During the practical session, participants will work with SHAP to explore model explainability across tasks in tree ensembles, computer vision, and language models.

This workshop offers a comprehensive exploration of the social, technical, and practical dimensions of LLMs, empowering participants to approach the development and evaluation of language models with a more nuanced and responsible perspective.

Date and Time: October 31st, 11:00 AM – 5:00 PM

Location: Saarland University (in-person) and online (details will be shared in a subsequent message)

Registration form: https://forms.office.com/e/U5gD5zWrDp

Evaluating Large Language Models — Choosing Goals and Measuring Them

by Vagrant Gautam

Abstract:

Before the rapid mainstream adoption of systems like ChatGPT, the technology that backs them—language models—were primarily evaluated on how well they modelled language. As language models have gotten larger and better, they are now used for various tasks, including answering questions, generating text, etc. Using examples from my work, I will show how these tasks require a multidimensional view of performance that goes beyond accuracy to include goals such as humility, robustness, and fairness. I will discuss the tensions that come with selecting these goals and conceptualizing and operationalizing them via datasets and metrics. As decisions about model evaluation have increasing societal impact, I will close with my thoughts on how we might democratically select goals with input from those affected by such technologies.

The Hidden Space of Transformer Language Adapters

by Jesujoba Alabi

Abstract:

We analyze the operation of transformer language adapters, which are small modules trained on top of a frozen language model to adapt its predictions to new target languages. We show that adapted predictions mostly evolve in the source language the model was trained on, while the target language becomes pronounced only in the very last layers of the model. Moreover, the adaptation process is gradual and distributed across layers, where it is possible to skip small groups of adapters without decreasing adaptation performance. Last, we show that adapters operate on top of the model’s frozen representation space while largely preserving its structure, rather than on an ‘isolated’ subspace. Our findings provide a deeper view into the adaptation process of language models to new languages, showcasing the constraints imposed on it by the underlying model and introduce practical implications to enhance its efficiency.

Keynote session: Effective, Fair, and Inclusive Large Language Models — On the Role of Social Factors

by Professor Anne Lauscher

Abstract:

Recently, language technology has seen tremendous advancements due to the development and use of large language models, large machine learning models pre-trained on large amounts of textual data. However, while how humans express themselves in language and how they perceive language is largely driven by their individual sociodemographic and sociocultural backgrounds, language models still only partially account for these social aspects. In this talk, I argue that we should consider these social factors more when researching and applying LLMs. Concretely, I will discuss some of our recent works relating to effectiveness, fairness, and inclusiveness, which will illustrate the critical role of social factors in natural language processing.

Practical session : Model Explainability in practice

by Israel A. Azime and Paloma García de Herreros García

In this section we will explore SHAP (SHapley Additive exPlanations) which is a game theoretic approach to explain the output of any machine learning model. Using SHAP we can explain decisions made with my machine learning models. For this practical session we will work with the following examples and see how to explain the output of each type of exercise.

• Explainability on Tree ensemble models.

  • Tabular data practical task

• Explainability in Computer Vision

  • MNIST image classification practice task

• Explainability in Language Models

  • Emotion classification explainability