Abstract

It is established that social media is not only a possible cause of mental health disorders, but a strong indicator. Great predictive information is contained in both text posts and images posted, which can be exploited by classification models. In this work, we evaluate and compare a number of different approaches to the detection of depression from social media activity. We use publicly available twitter posts and profile and background images as our predictive features. We test classical machine learning approaches, sequential models such as LSTMs, and Convolutional Neural Networks. Additionally, we implement and test a modality fusion model which fuses textual and image-based features to achieve greater accuracy. This fusion model outperforms the best textual and image models tested by a full 17.36 percentage points and 35.99 percentage points respectively, indicating that the information contained in text and images is complementary and is best exploited in conjunction.

The problem

Posts on Twitter and Reddit are an established indicator of depression, and most detection work encodes text into fixed-length vectors and applies classifiers such as logistic regression or random forests. Profile and background images have been used less often, though they too carry information about mental health.

Two questions were open when we did this work: whether richer text representations (Word2Vec, Doc2Vec) and sequential models (LSTMs, GRUs) help, and whether fusing text with images gives a better classifier than either modality alone. This paper runs both comparisons on a large, newly collected dataset.

Approach

  • Dataset. We collected tweets from 1 January 2017 to 1 June 2022 matching self-diagnosis patterns such as “i have/was (just) (been) diagnosed with depression”. After removing retweets, duplicates, users with fewer than 100 tweets, and users without a profile or background image, the Diagnosis group has 2,970 users and 6.1 million tweets. A Control group was built from tweets containing “the” on a single day (1 June 2022), filtered the same way with overlapping users removed, giving 2,273 users and 4.6 million tweets. Profile and background images were resized to 512x512. Data was anonymised and is not released.
  • Text features and models. After tokenisation, stop-word removal, lemmatisation, and stemming, we compare TF-IDF character 2/4-grams and word 1/2/3-grams with logistic regression, ridge classifiers, gradient boosted trees, random forests, and a two-hidden-layer ANN (256 and 32 neurons, dropout 0.2). We also train the ANN on Doc2Vec vectors and a three-layer bidirectional LSTM with 1D convolution on Word2Vec (skip-gram) embeddings.
  • Image models. EfficientNetV2 backbones (B0, S, B3, M) pretrained on ImageNet, frozen, with a single sigmoid dense layer fine-tuned for 20 epochs.
  • Modality fusion. Following Gallo et al. (2020), an early-fusion model concatenates the 1024-unit Doc2Vec vector of a tweet with the 1280-unit flattened EfficientNetV2 feature map of the author’s profile image (and separately the background image), giving 2304-unit inputs to a two-hidden-layer ANN trained for 50 epochs.
Tweet text tokenised, stemmed 10M+ tweets Profile image or background image 512 x 512 Doc2Vec document embedding 1024 units EfficientNetV2 ImageNet weights, frozen pooled + flattened: 1280 concat 2304 units Dense classifier 256 -> 32 -> 2 softmax, dropout 0.2 Adam, lr 0.001 50 epochs, batch 32 Depression / control 99.3% accuracy F1 95.61, AUC 93.2
Each tweet is embedded by Doc2Vec and the author's profile (or background) image by a frozen EfficientNetV2; the concatenated 2304-unit vector is classified by a small dense network, and the two modalities together far outperform either alone.

Results

Type Model Feature Accuracy F1 AUC
Classical Logistic regression Character 4-gram 62.90% 63.13% 68.24%
Classical Gradient boosted trees Word 1-gram 70.46% 68.40% 72.83%
Classical Random forest Character 4-gram 71.14% 69.03% 72.40%
Classical ANN Character 4-gram 81.94% 92.10% 83.43%
Classical ANN Doc2Vec vectors 64.30% 63.32% 70.47%
Sequential Bidirectional LSTM Word2Vec vectors 64.93% 65.04% 71.04%
Image EfficientNetV2B0 Images 59.88% 62.12% 57.76%
Image EfficientNetV2M Images 63.31% 68.50% 63.10%
Fusion CNN + ANN Doc2Vec + EfficientNetV2B0 feature map 99.30% 95.61% 93.20%

Source: Table 1 of the paper; the best feature set is shown for each classical model. Held-out validation split (1% of the more than 10 million tweets). Higher is better.

  • Text-only models. The ANN on character 4-grams is the best text-only model at 81.94% accuracy (F1 92.1%), followed by the ANN on character 2-grams at 81.3%. Richer embeddings on their own do not beat TF-IDF here: the ANN on Doc2Vec vectors reaches 64.3% and the bidirectional LSTM on Word2Vec embeddings 64.93% (63.01% with pre-processed text).
  • Image-only models. The best CNN, EfficientNetV2M, reaches 63.31% accuracy, comparable to or better than several text-only models, confirming that profile and background images carry substantial signal.
  • Modality fusion. Doc2Vec features plus the EfficientNetV2B0 feature map reach 99.3% accuracy, 95.61% F1, and 93.2% AUC, 17.36 percentage points above the best text model and 35.99 above the best image model. The information in the two modalities is complementary and is best exploited together.

Dataset

Group Users Tweets Images Collection rule
Diagnosis 2,970 6.1 million profile + background per user self-reported depression diagnosis, 1 Jan 2017 to 1 Jun 2022
Control 2,273 4.6 million profile + background per user tweets containing “the” on 1 Jun 2022, no overlap with Diagnosis

Source: the Dataset section of the paper. Both groups drop retweets, duplicates, users with fewer than 100 tweets, and users without a profile or background image; images are resized to 512x512. In total over 10 million tweets and roughly 10,000 images. The data were anonymised and kept private.

Resources