Abstract

Event detection, specifically in the socio-political domain, has posed a long-standing challenge to researchers in the NLP domain. Therefore, the creation of automated techniques that perform classification of the large amounts of accessible data on the Internet becomes imperative. This paper is a summary of the efforts we made in participating in Task 1 of CASE 2022. We use state-of-art multilingual BERT (mBERT) with further fine-tuning to perform document classification in English, Portuguese, Spanish, Urdu, Hindi, Turkish and Mandarin. In the document classification subtask, we were able to achieve F1 scores of 0.8062, 0.6445, 0.7302, 0.5671, 0.6555, 0.7545 and 0.6702 in English, Spanish, Portuguese, Hindi, Urdu, Mandarin and Turkish respectively achieving a rank of 5 in English and 7 on the remaining language tasks.

The problem

Subtask 1 of CASE 2022 Task 1 is document-level binary classification: given a news article, decide whether it reports a protest event that has happened or is ongoing. Scheduled events, rumours and speculation count as negative. The metric is macro-F1.

The data makes this harder than a standard text classification task. Training data exists only for English (9,324 documents), Spanish (1,000) and Portuguese (1,487), and in each the positive class is a minority (positive ratios of 0.205, 0.131 and 0.132). Hindi, Urdu, Turkish and Mandarin have test sets but no training data at all, so they are a zero-shot setting. A model trained naively would lean toward the negative class and have nothing to learn from for four of the seven languages.

Approach

Data augmentation by translation. We treat the three available datasets as a resource for each other. Positive samples from Spanish and Portuguese are translated into English and added to the English set. The Spanish and Portuguese sets are extended with the full English set and each other’s data, translated into the target language. For Hindi, Urdu, Mandarin and Turkish, the final augmented English set is translated into each language to create training data where none existed. Translation used the googletrans library. After augmentation the English, Hindi, Urdu, Mandarin and Turkish sets have 9,652 documents, Spanish 10,521 and Portuguese 10,942, with the positive class now the majority (for example 7,412 positive vs 2,240 negative in the English set).

Fine-tuning multilingual BERT. Our classifier is bert-base-multilingual-cased, trained on Wikipedia in 104 languages with a shared 110k WordPiece vocabulary (12 layers, 768 hidden dimensions, 12 heads, 110M parameters). We stack a dropout layer and a dense layer on the [CLS] representation, with a two-neuron sigmoid output and a decision threshold of 0.62 for the positive class.

The classifier: mBERT encodes the tokenised document, and the [CLS] feature vector passes through dropout and a fully connected layer with sigmoid activation to predict event or no event.

Training details. Models were built in Keras with HuggingFace transformers and trained on a Google Colab GPU with binary cross-entropy and Adam. We tried learning rates of 1e-5, 3e-5 and 5e-5 and found 3e-5 best, with a maximum length of 512 tokens, batch size 6, dropout 0.2 and 4 epochs.

Training corpora English 9,324 docs Spanish 1,000 docs Portuguese 1,487 positives 13–21% no HI / UR / ZH / TR Translation augmentation positives shared across EN / ES / PT augmented EN → HI, UR, ZH, TR sets mBERT fine-tuning bert-base- multilingual-cased [CLS] → dropout → dense, sigmoid threshold 0.62 Document classification protest event reported? yes / no evaluated on seven test sets, four of them zero-shot English Spanish Portuguese Hindi Urdu Mandarin Turkish
Positive samples are translated across the English, Spanish and Portuguese corpora, the augmented English set is translated into the four zero-shot languages, and one mBERT classifier is fine-tuned per language to decide whether a document reports a protest event.

Example

Two worked examples of the augmentation recipe, with the sizes reported in the paper.

Input (few-shot language)

Spanish training set as provided: 1,000 documents with a positive ratio of 0.131.

Output (augmented training set)

Original Spanish data + the full English set (both classes) translated into Spanish + Portuguese samples translated into Spanish = 10,521 documents, 8,281 positive and 2,240 negative. The model fine-tuned on this set scores a macro-F1 of 0.6445 on the 671-document Spanish test set.

Input (zero-shot language)

Hindi: a 268-document test set and no training data at all.

Output (augmented training set)

The final augmented English set (9,652 documents: 7,412 positive, 2,240 negative) translated into Hindi with googletrans. The same recipe produces the Urdu, Mandarin and Turkish training sets. The Hindi model scores a macro-F1 of 0.5671; Mandarin, built the same way, reaches 0.7545.

Results

Macro-F1 of our system on the CASE 2022 Subtask 1 test sets, alongside the data each model was trained on:

Language Original train docs Train docs after augmentation Test docs Macro-F1 Rank
English 9,324 9,652 3,871 0.8062 5
Mandarin none (zero-shot) 9,652 300 0.7545 7
Portuguese 1,487 10,942 671 0.7302 7
Turkish none (zero-shot) 9,652 300 0.6702 7
Urdu none (zero-shot) 9,652 299 0.6555 7
Spanish 1,000 10,521 671 0.6445 7
Hindi none (zero-shot) 9,652 268 0.5671 7

Source: Tables 1, 3 and 4 of the paper. Metric is macro-F1 on the official test sets. Higher is better.

  • English is the strongest result at 0.8062 macro-F1 (rank 5), consistent with multilingual BERT’s strongest contextualisation being in English.
  • The zero-shot languages are competitive: Mandarin (0.7545) and Turkish (0.6702) score above Spanish, which has 1,000 native training documents, so translated training data is a workable substitute when none exists.
  • A single recipe (translate, fine-tune bert-base-multilingual-cased, threshold at 0.62) covers all seven evaluation languages.

Label distribution of the training sets after augmentation:

Training set Negative (label 0) Positive (label 1) Total Positive ratio before augmentation
English, Hindi, Urdu, Mandarin, Turkish 2,240 7,412 9,652 0.205 (English)
Spanish 2,240 8,281 10,521 0.131
Portuguese 2,240 8,702 10,942 0.132

Source: Table 3 and Section 3.2 of the paper. Augmentation turns the positive class from a 13 to 21 percent minority into the majority.

Resources