ACLM: A Selective-Denoising based Generative Data Augmentation Approach for Low-Resource Complex NER

Abstract

Complex Named Entity Recognition (NER) is the task of detecting linguistically complex named entities in low-context text. In this paper, we present ACLM (Attention-map aware keyword selection for Conditional Language Model fine-tuning), a novel data augmentation approach, based on conditional generation, to address the data scarcity problem in low-resource complex NER. ACLM alleviates the context-entity mismatch issue, a problem existing NER data augmentation techniques suffer from and often generates incoherent augmentations by placing complex named entities in the wrong context. ACLM builds on BART and is optimized on a novel text reconstruction or denoising task: we use selective masking (aided by attention maps) to retain the named entities and certain keywords in the input sentence that provide contextually relevant additional knowledge or hints about the named entities. Compared with other data augmentation strategies, ACLM can generate more diverse and coherent augmentations preserving the true word sense of complex entities in the sentence. We demonstrate the effectiveness of ACLM both qualitatively and quantitatively on monolingual, cross-lingual, and multilingual complex NER across various low-resource settings. ACLM outperforms all our neural baselines by a significant margin (1%-36%). In addition, we demonstrate the application of ACLM to other domains that suffer from data scarcity (e.g., biomedical). In practice, ACLM generates more effective and factual augmentations for these domains than prior methods.

The problem

Standard NER benchmarks such as CoNLL 2003 test “easy” proper-name entities in well-formed news text. Complex NER benchmarks like MultiCoNER instead contain short, low-context sentences with emerging and semantically ambiguous entities (movie titles in comments, product names, political groups). We found that a state-of-the-art NER model drops by 23% when moved from CoNLL 2003 to MultiCoNER, and by 31.8% in a low-resource setting with only 500 training samples.

Data augmentation is the usual remedy for scarce labels, but existing NER augmenters break down on complex entities. Swapping entities between sentences or replacing them with synonyms leads to context-entity mismatch (the name of a book dropped into a sentence about a movie), and fine-tuned language models struggle to invent new context around entities they have never seen. The resulting incoherent or non-factual augmentations hurt learning, especially in knowledge-sensitive domains such as biomedical NER.

Approach

ACLM reframes augmentation as conditional generation: keep the entities, keep the few words that explain them, and ask a denoising model to rewrite everything else. A sentence becomes a template through a four-step process:

• Keyword selection. A XLM-RoBERTa NER model fine-tuned only on gold data provides attention maps. Summing attention over the last four layers, we pick the top p% of non-entity tokens that the entities attend to most, ignoring punctuation, stop words, and other entities.
• Selective masking. Every non-entity token outside the keyword set is replaced by a mask token, and contiguous masks are collapsed.
• Labelled sequence linearization. Label tokens are inserted before and after each entity so the model sees entity types and boundaries during fine-tuning and generation.
• Dynamic masking. At every training and generation round, a masking rate sampled from a Gaussian hides a small, random subset of the keywords, which increases context and length diversity.

ACLM is built on mBART-50-large and fine-tuned to reconstruct the original sentence from its template, a selective version of the denoising objective BART is pre-trained on. At generation time, each training sentence is corrupted R times to produce R augmentations, with top-k sampling and beam search for diversity. Post-processing removes augmentations that are too similar to the original sentence and strips the label tokens, and the augmented data is concatenated with the gold data to fine-tune the NER model.

mixner adds a second source of diversity: during generation, the template of a sentence is concatenated with the template of a semantically similar sentence (retrieved with multilingual Sentence-BERT), so ACLM produces sentences that combine context and entities from both.

Example

The worked example from Figure 1 of the paper (MultiCoNER, English). Keywords selected from the attention map are enemy, infantry, retired, swarmed, hidden ground, attack; the only entity is royal artillery (GRP).

[M] enemy [M] infantry [M] retired [M] enemy swarmed [M] hidden ground [M]
<b-grp> royal <b-grp> <i-grp> artillery <i-grp> guns [M] attack [M]

[M] enemy [M] infantry [M] swarmed [M] hidden ground [M]
<b-grp> royal <b-grp> <i-grp> artillery <i-grp> guns [M]

A second example from Figure 3 compares augmenters on a sentence about a game studio, where entity swapping and random word replacement break either coherence or factuality:

Results

All experiments use MultiCoNER across 10 languages (English, Bengali, Hindi, German, Spanish, Korean, Dutch, Russian, Turkish, Chinese) with 100, 200, 500, and 1000 gold training sentences, reporting micro-F1 averaged over 3 seeds. Baselines include Gold-Only, LwTR, DAGA, MulDA, and MELM.

Source: Table 1 of the paper (Avg column). Micro-F1 on the MultiCoNER test sets, averaged over the 10 languages and 3 seeds. Higher is better.

Source: Table 1 of the paper (right half), 500 gold English sentences, zero-shot evaluation on the target language. Higher is better.

Source: Table 4 of the paper. F1 on news (CoNLL 2003), biomedical (BC2GM, NCBI Disease), and science (TDMSci) NER with 500 gold sentences. Higher is better.

• Monolingual complex NER: ACLM achieves the best result in every language and every low-resource setting, with absolute gains of 1.5%-22% over the neural baselines MELM and DAGA.
• Multilingual complex NER: in the combined 10-language setting ACLM averages 51.40 / 53.45 / 55.90 / 58.27 F1 at 100 / 200 / 500 / 1000 sentences per language, versus 46.86 / 51.42 / 53.66 / 57.78 for Gold-Only (Table 2).
• Generation quality: at 500 gold sentences ACLM has the lowest GPT-2 perplexity of the compared augmenters (57.68 vs. 82.31 for MELM and 129.35 for LwTR) and the highest non-entity diversity (41.16 vs. 0.0 and 16.22) and length diversity (5.82 vs. 0.0 and 0.0) (Table 3).
• Other domains: ACLM has the best average F1 at both 200 (67.93) and 500 (72.24) gold sentences, and the best score on every dataset except CoNLL 2003 at 500, where LwTR leads.

MultiCoNER at a glance

Source: Table 12 of the paper. Six entity classes: PER, LOC, GRP, CORP, CW (creative work), PROD. The low-resource experiments sample 100-1000 sentences from each training split.

Resources

• Paper: ACL Anthology · arXiv:2306.00928 · Hugging Face papers
• Code: github.com/Sreyan88/ACLM
• Dataset: MultiCoNER (Malmasi et al., 2022)
• Related: CoSyn, a companion project from the same group on implicit hate speech
• All publications

Quick start

From the repository README: install the requirements, then run the training script with the language, dataset size, seed, masking rate, and number of generations.

pip install -r requirements.txt
cd ./src
# sh train_dynamic_multilingual.sh <language> <language label> <size of dataset> <flair batch size> <seed> <masking rate> <number of generations>
sh train_dynamic_multilingual.sh zh zh_CN 100 8 42 0.3 5
# with mixner
sh train_dynamic_multilingual_mixner.sh zh zh_CN 100 8 42 0.3 5