Abstract

Disfluency, though originating from human spoken utterances, is primarily studied as a uni-modal text-based Natural Language Processing (NLP) task. Based on early-fusion and self-attention-based multimodal interaction between text and acoustic modalities, in this paper, we propose a novel multimodal architecture for disfluency detection from individual utterances. Our architecture leverages a multimodal dynamic fusion network that adds minimal parameters over an existing text encoder commonly used in prior art to leverage the prosodic and acoustic cues hidden in speech. Through experiments, we show that our proposed model achieves state-of-the-art results on the widely used English Switchboard for disfluency detection and outperforms prior unimodal and multimodal systems in literature by a significant margin. In addition, we make a thorough qualitative analysis and show that, unlike text-only systems, which suffer from spurious correlations in the data, our system overcomes this problem through additional cues from speech signals.

The problem

When people speak, they repeat, restart, and correct themselves. A disfluency has a reparandum (the part to remove), an optional interregnum such as “uh”, and a repair. Detecting and removing reparandums matters because language understanding systems trained on fluent text are easily misled by disfluent input.

Most state-of-the-art systems on the Switchboard corpus tag disfluencies from transcripts alone. Text carries rich semantics and syntax, but disfluency originates in speech, and cues such as prosody, pitch and stutter are ignored. The few systems that did use speech relied on hand-engineered acoustic features and simple concatenation with text features, which cannot capture fine-grained interactions between the two modalities.

Approach

Contextualised representations. The transcript is encoded with BERT-base, giving a 768-dimensional embedding per token. The raw audio is encoded with wav2vec 2.0 (the robust wav2vec 2.0-large fine-tuned on Switchboard), giving a 768-dimensional embedding per frame. No hand-crafted acoustic features are needed.

Multimodal Interaction Module (MMI). Three Cross-Modal Encoder (CME) blocks, each a transformer layer built around cross-modal attention, align the two streams:

  • Speech-aware word representations: block A uses the speech embeddings as queries and token embeddings as keys and values; block B then uses the original tokens as queries over that output, so the result R is indexed by word rather than by frame.
  • Word-aware speech representations: block C uses the token embeddings as queries and the speech embeddings as keys and values, giving Q.
  • Acoustic gate: a sigmoid gate computed from [R; Q] scales Q so that redundant or noisy speech frames contribute less.

Span classification. R and Q are concatenated into the final representation M. Instead of tagging tokens, the model enumerates candidate spans up to a maximum length and classifies each as disfluent or fluent from the concatenation of its start-token representation, end-token representation and a learned length embedding, followed by the heuristic decoding of prior span-classification work.

Training. Models are implemented in PyTorch with HuggingFace checkpoints and trained for 20 epochs with batch size 32 and Adam at a learning rate of 1e-5. Data follows the standard Switchboard split, with lowercasing and removal of punctuation and partial words.

Transcript tokens w1 … wM Audio raw waveform BERT-base 768-d per token wav2vec 2.0 768-d per frame Cross-modal encoders A, B: speech-aware word representation R C: word-aware speech representation Q cross-modal attention Dynamic fusion gate σ([R;Q]) M = [R ; g⊙Q] Span extraction start, end, length emb. disfluent? queries ↔ keys
MDFN encodes the transcript with BERT and the audio with wav2vec 2.0, aligns the two streams with three cross-modal encoders, fuses them through an acoustic gate, and classifies candidate spans as fluent or disfluent.

Example

Disfluencies come in five types. In the paper’s notation the reparandum sits before the “+”, an optional interregnum in braces, and the repair after it; detection means finding the reparandum spans.

Input (disfluent utterances, Switchboard)

Repair: "[i do + i] ski yes"
Repetition: "but [i + i] grew up with cats"
Restart: "[you were + {uh}] he was waiting for what again"
Deletion: "[i that it just +] you know it's absolutely devastating"
Substitution: "the pen was kept [under + over] the table"

Output (spans to remove)

The bracketed reparandum before each "+" is the disfluent span: i do, i, you were, i that it just, under.

Where the acoustic modality pays off: two test utterances from the paper’s qualitative analysis that the text-only state of the art marks as disfluent but that are fluent according to the ground truth.

Input

1. "i finally got impaneled on one case on my next to the last day"
2. "and that is that money tends to stick where it lands first"

Output

Text-only Span Classification BERT-GCN: flags "on my" in 1 and "that is" in 2 as disfluent, reading the repeated "on" and "that" as repetitions.
MDFN (text + speech): both utterances fluent, matching the ground truth. The confidence and tone of the speaker in the audio reveal that these are ordinary, if loosely grammatical, sentences rather than restarts.

Results

Evaluation on the English Switchboard test set under the IO tagging scheme:

Model Precision Recall F1
Self-trained 87.5 93.8 90.6
EGBC 95.7 88.3 91.8
BERT fine-tune 94.7 89.8 92.2
BERT-CRF-Aux 94.6 91.2 92.9
ELECTRA-CRF-Aux 94.8 91.6 93.1
Span Classification BERT-GCN 95.2 93.2 94.2
BERT span classifier (text-only baseline) 95.1 93.0 94.1
MDFN (ours) 92.8 98.7 95.7

Source: Table 2 of the paper (8 of its 13 rows; earlier systems such as Semi-CRF, Bi-LSTM, attention-based and transition-based models score 85.4 to 87.5 F1). Higher is better.

  • MDFN reaches 95.7 F1, 1.5 points above the previous state of the art (Span Classification BERT-GCN, 94.2) and 1.6 points above the text-only span-classification BERT baseline built with the same span head.
  • Recall on disfluent spans rises to 98.7, the highest in the table by 4.9 points, which is where the acoustic cues contribute most.
  • The multimodal interaction module adds only three cross-modal encoder blocks and a gate on top of the text encoder used by prior work, with no hand-crafted acoustic features.
  • Qualitative analysis shows the text-only model flags spurious repetitions (for example a repeated “on” or “that” in grammatically loose sentences) as disfluencies, while MDFN uses the speaker’s tone and confidence to correctly call them fluent.

Resources