Abstract

The paper presents an overview of our submission to the fifth edition of the CheckThat! Lab challenge. Specifically, our team participated in subtasks 1A, 1B and 1C under task 1, which aimed to identify relevant claims in tweets. More specifically, the three subtasks deal with evaluating the check-worthiness, presence of verifiable facts and presence of harmful content in the tweets, respectively. The lab provided us datasets for the three subtasks in multiple languages including English, Dutch and Bulgarian. A total of 14, 10 and 12 teams participated in the subtasks 1A, 1B and 1C respectively, out of which we ranked 9th, 2nd and 2nd, respectively. In this paper, we discuss our methodology for the subtasks, which includes data augmentation to increase the size of our training dataset, followed by preprocessing of the tweets and feature extraction for the tweets from the Twitter API to gain more data points that can help gauge the credibility and/or authenticity of the tweet. Finally, we discuss the structure of our Multimodal model which uses numerical and categorical features in addition to the textual data from tweets.

The problem

The CheckThat! Lab at CLEF 2022 targets disinformation on Twitter. Task 1 asks systems to identify relevant claims in tweets through three binary subtasks: 1A, is the tweet worth fact-checking; 1B, does it contain a verifiable factual claim; 1C, is it harmful to society. Subtasks 1A and 1C are scored by F1 on the positive class and 1B by accuracy. We submitted for the English test sets.

The datasets are small (2,122 English training tweets for 1A, 3,324 for 1B and 3,323 for 1C), mostly about COVID-19, and imbalanced. They also contain only the tweet text and topic, even though whether a claim is check-worthy or harmful depends on who is posting it and how far it has spread.

Approach

Preprocessing. We remove hashtags, URLs and @mentions, and replace numeric values with the token “number” so the model can pick up on the presence of a figure, which is a strong signal of verifiability, without memorising specific values. Because a link may still matter, the presence of a URL is kept as a categorical feature.

Data augmentation by translation. We translate the Bulgarian and Dutch training sets into English with the googletrans library and append them to the English data, choosing only languages with similar scripts and semantics to preserve information. This grows the training sets to 4,916 tweets for 1A, 7,984 for 1B and 7,977 for 1C.

Feature extraction through the Twitter API. For each tweet we fetch numerical features (author’s followers, following and post counts; the tweet’s likes and retweets) and categorical features (whether the author is verified; whether the tweet contains a URL). Missing values, for deleted tweets or private accounts, default to 0. The intuition is that a claim about vaccines from an account with political authority is more check-worthy than the same words from an anonymous account.

Multimodal model. The text is encoded by bert-base-uncased, and the categorical and numerical features pass through separate MLPs whose outputs are concatenated with the transformer output before the classification layer: m = x || MLP(c) || MLP(n). We build this with the Multimodal-Toolkit for transformers with tabular data, using the individual_mlps_on_cat_and_numerical_feats_then_concat combine method, dropout 0.1, an MLP division ratio of 4, batch size 8, and 1 epoch per model on a Google Colab GPU.

Tweet text preprocessed Categorical verified, has_url Numerical followers, likes, RTs features via Twitter API BERT bert-base-uncased MLP MLP(c) MLP MLP(n) Concatenate m = x ‖ MLP(c) ‖ MLP(n) fully connected layers Claim / no claim 1A check-worthy 1B verifiable 1C harmful one model per subtask
Each tweet becomes three inputs: the preprocessed text for BERT, and categorical and numerical features from the Twitter API for two small MLPs; their outputs are concatenated and fully connected layers make the binary prediction.

Example

The preprocessing step, on the tweet shown in Figure 4 of the paper:

Input (raw tweet)

"There are now 10 new confirmed cases of #COVID19. Total now at 20, according to DOH | via @kristinesabillo https://twitter.com/kristinesabillo"

Output (text passed to BERT)

"There are now number new confirmed cases of . Total now at number, according to DOH | via"
Hashtags, URLs and @mentions are removed and numeric values become the token "number", so the model learns that a figure is present without memorising the value. The removed URL survives as the categorical feature has_url = 1.

The feature-extraction step, on a tweet from the Subtask 1A training set (Figure 5 of the paper):

Input (tweet and author)

A tweet from the Subtask 1A training set and the features fetched for it through the Twitter API.

Output (tabular features)

likes 17478, rts 14455, followers 958555, following 64763, posts 329424, verified 1, url 1. The same words posted by someone with strong political authority and a large audience are more likely to be check-worthy than from an anonymous account, which is what these features let the model see.

Results

On the English test sets of CheckThat! 2022 Task 1:

Subtask Metric Asatya (ours) Rank Winning score
1A: check-worthiness F1 (positive class) 0.500 9 of 13 0.698
1B: verifiable factual claim Accuracy 0.749 2 of 10 0.761
1C: harmful tweet F1 (positive class) 0.361 2 of 11 0.397

Source: Section 6 and Tables 5 to 7 of the paper (official leaderboard, English test sets). Higher is better.

Top of the leaderboards for the two subtasks where we ranked second:

Team (Subtask 1B) Accuracy Team (Subtask 1C) F1
Team_PoliMi-FlatEarthers 0.761 nicuBuliga 0.397
Asatya 0.749 Asatya 0.361
Team_NLP&IR@UNED 0.725 asavchev 0.361
asavchev 0.713 Team_NLP&IR@UNED 0.347
nicuBuliga 0.709 mkutlu 0.329
random baseline 0.494 random baseline 0.200

Source: Tables 6 and 7 of the paper (top five teams plus the random baseline). Higher is better.

  • Subtask 1B: accuracy 0.749, rank 2 of the 10 teams that submitted, 0.012 behind the winner.
  • Subtask 1C: F1 0.361, rank 2 of the 11 teams that submitted, and well clear of the random baseline at 0.200.
  • Subtask 1A: F1 0.500, rank 9 of the 13 teams that submitted. Verifiability correlates with numbers and objective facts and harmfulness has recurring patterns, both of which the features and the “number” token capture directly; 1B and 1C also had the larger augmented training sets (7,984 and 7,977 tweets versus 4,916 for 1A).

Training data after translation-based augmentation:

Subtask English Bulgarian (translated) Dutch (translated) Total
1A 2,122 1,871 923 4,916
1B 3,324 2,710 1,950 7,984
1C 3,323 2,708 1,946 7,977

Source: Table 4 of the paper. Augmentation grows the training sets by 231%, 240% and 240%.

Resources