We do not need to fire off a bazooka if we can just use small language models

Aaron Maladry is a postdoctoral researcher at the LT3 research group at the University of Ghent. He is a linguist by training and a computational researcher. He joined the European Centre of Excellence in AI for Digital Humanities as a visiting researcher. In this interview, he discusses irony detection with large language models, syntactic parsing of spoken data and his favourite running route in Ljubljana.

You do research on irony. How would you explain it?

With irony, we generally try to use language that expresses something other than the literal meaning of what is said. So when you say, “I hope my paper gets rejected,” you intuitively know that it is very unlikely someone would mean this genuinely. You then infer from the context, “Oh, that person is making a joke and actually means the opposite.” This is a typical example of verbal irony.

However, there are other types, such as situational irony, which mostly occur in situations that defy expectations. For instance, if firefighters rush out to extinguish a burning building and, at the same time, their own fire station burns down, that would be situational irony, because you would not expect this to happen to them. With situational irony, the irony emerges from the context, not from linguistic expression.

What is the practical relevance of detecting irony?

Irony detection and exploration is, in itself, a fun linguistic project. However, there are also direct practical applications. Irony and figurative language remain limitations for many semantic tasks such as sentiment analysis or emotion detection, and they are particularly relevant to areas like hate speech or toxicity detection, where people express hateful or problematic opinions in a joking way. As humans, we interpret this as offensive, but such content can remain undetected by automatic filtering systems, which is a significant problem.

This affects, for example, social media content moderation and customer reviews, or situations where you have a livestream and people are posting messages in real time, and you want a system that quickly detects ironic or humorous hate speech and prevents it from being posted immediately.

Can language models detect irony?

They seem to perform well on the datasets we currently have, but there is a major question about when they work and when they do not. Most curated datasets for irony detection contain very clear, exaggerated examples of irony. Much work has been done using soaps and TV series, where one character consistently speaks in an ironic way. The types of irony modelled in computational research often do not reflect the genuinely difficult cases that occur in real-world hate speech or everyday conversations. It is definitely a field that we should explore further, really looking for irony in the wild, where it is only a side component.

How do you use AI in your work?

I use AI as a tool, and I put a lot of effort into evaluation. I think there is more to computational modelling than predicting scores. We also carry out a great deal of manual analysis, and we should treat AI critically and approach it with a healthy degree of scepticism. Does it really work in the way we want it to?

What did you do during your visiting research stay at AI4DH?

I am working on syntax for spoken language. The project I am employed on focuses on predicting syntactic labels for spoken dialect data in Dutch. We are collaborating on Slovene, as researchers here are also working on syntactic parsing of spoken data, and we are benchmarking a variety of models against existing approaches to see what works well for different languages. We found that researchers often adopt one pipeline and then keep using it, while in another language people use a different pipeline and never really compare the methods directly. It may be that one tool is simply better, and it is useful to know that in advance. At present, no one is benchmarking spoken syntax.

What does it mean to syntactically parse spoken data?

There are several components to this. If you have a sentence, each word first needs to be labelled: is it a noun, a verb, and so on; and syntactic parsing then identifies, for example, the noun that is the subject of a particular verb. Doing this with spoken language presents specific difficulties, including many repairs. If you make a mistake and then correct yourself, how should this be labelled? If you use a lot of interjections, which are common in spoken language, it becomes more interesting, because when people speak they do not follow grammatical rules as strictly.

We have a hypothesis that if you were able to build a model for Slovene and Dutch, the use of interjections and repairs would be shared across languages. We also hypothesise that the characteristics of spoken languages are quite similar across languages, but we have no concrete evidence yet.

What are the current trends in your field of research?

Many people are using generative models, and this is where the field of syntactic labelling is somewhat of an exception, because the generative decoder models we have now do not work as well in contexts where every word has to be labelled. For syntactic models we are still using slightly old-school types of tools. They are also a bit more interpretable, which is an advantage. We do not need to fire off a bazooka if we can just use small language models.

In tools for syntactic parsing, you have to predict which word depends on which other word and which type of label belongs to it. Complex model graphs are used here and are often more useful than generative models. You can also make clever use of linguistic information. One task is linguistic feature prediction: if you know that a noun can be a subject and a verb can have a tense, and that a noun cannot have a tense, you can constrain the model; you do not have to teach it everything. There are rules you can use to constrain your model so that it does not predict things that are not logical. One of Kaja Dobrovoljc’s research papers showed that by adding these rule-based shortcuts as defaults, you do not allow the model to make certain mistakes. This is a real advantage in this area of the syntactic component of NLP.

What is your favourite running route in Ljubljana?

I have a good one! I start in the city centre and run to Tivoli Park, detour to Jesenkova pot, then go around through Mostec and back through the forest.